2023-05-31T19:32:20.756 INFO:root:teuthology version: 0.0.1.dev111+gfa97ddd 2023-05-31T19:32:20.757 DEBUG:teuthology.run:Teuthology command: teuthology --description rgw/multisite/{clusters frontend/beast ignore-pg-availability notify omap_limits overrides realms/two-zonegroup supported-random-distro$/{centos_8} tasks/test_multi} --verbose --owner scheduled_yuriw@teuthology --archive /home/teuthworker/archive/yuriw-2023-05-31_19:25:20-rgw-reef-release-distro-default-smithi/7292285 --name yuriw-2023-05-31_19:25:20-rgw-reef-release-distro-default-smithi -- /home/teuthworker/archive/yuriw-2023-05-31_19:25:20-rgw-reef-release-distro-default-smithi/7292285/orig.config.yaml 2023-05-31T19:32:20.793 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2023-05-31T19:32:20.880 INFO:teuthology.run:Config: archive_path: /home/teuthworker/archive/yuriw-2023-05-31_19:25:20-rgw-reef-release-distro-default-smithi/7292285 branch: reef-release description: rgw/multisite/{clusters frontend/beast ignore-pg-availability notify omap_limits overrides realms/two-zonegroup supported-random-distro$/{centos_8} tasks/test_multi} email: yweinste@redhat.com first_in_suite: false job_id: '7292285' kernel: kdb: true sha1: distro last_in_suite: false machine_type: smithi name: yuriw-2023-05-31_19:25:20-rgw-reef-release-distro-default-smithi no_nested_subset: false nuke-on-error: true os_type: centos os_version: 8.stream overrides: admin_socket: branch: reef-release ceph: conf: client: debug rgw: 20 debug rgw sync: 20 rgw crypt require ssl: false rgw crypt s3 kms backend: testing rgw crypt s3 kms encryption keys: testkey-1=YmluCmJvb3N0CmJvb3N0LWJ1aWxkCmNlcGguY29uZgo= rgw curl low speed time: 300 rgw data log num shards: 4 rgw md log max shards: 4 rgw sync data inject err probability: 0.1 rgw sync log trim interval: 0 rgw sync meta inject err probability: 0.1 rgw sync obj etag verify: true setgroup: ceph setuser: ceph client.0: rgw data notify interval msec: 0 mgr: debug mgr: 20 debug ms: 1 mon: debug mon: 20 debug ms: 1 debug paxos: 20 osd: debug ms: 1 debug osd: 20 rocksdb delete range threshold: 0 osd.0: osd_max_omap_entries_per_request: 10 osd.1: osd_max_omap_entries_per_request: 1000 osd.2: osd_max_omap_entries_per_request: 10000 flavor: default log-ignorelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) - \(PG_AVAILABILITY\) - \(PG_DEGRADED\) - \(POOL_APP_NOT_ENABLED\) sha1: 4a02f3f496d9039326c49bf1fbe140388cd2f619 wait-for-scrub: false ceph-deploy: conf: client: log file: /var/log/ceph/ceph-$name.$pid.log mon: osd default pool size: 2 install: ceph: flavor: default sha1: 4a02f3f496d9039326c49bf1fbe140388cd2f619 rgw: compression type: random frontend: beast rgw-multisite: realm: is default: true name: test-realm zonegroups: - enabled_features: - resharding endpoints: - c1.client.0 is_default: true is_master: true name: a zones: - endpoints: - c1.client.0 is_default: true is_master: true name: a1 - endpoints: - c1.client.1 name: a2 - enabled_features: - resharding endpoints: - c2.client.0 is_default: true name: b zones: - endpoints: - c2.client.0 is_default: true is_master: true name: b1 - endpoints: - c2.client.1 name: b2 rgw-multisite-tests: args: - tests.py selinux: whitelist: - scontext=system_u:system_r:logrotate_t:s0 workunit: branch: reef-release sha1: 2870bf456d3115490c153abce9d47ac5c043b65e owner: scheduled_yuriw@teuthology priority: 71 repo: https://github.com/ceph/ceph-ci.git roles: - - c1.mon.a - c1.mgr.x - c1.osd.0 - c1.osd.1 - c1.osd.2 - c1.client.0 - c1.client.1 - - c2.mon.a - c2.mgr.x - c2.osd.0 - c2.osd.1 - c2.osd.2 - c2.client.0 - c2.client.1 seed: 2752 sha1: 4a02f3f496d9039326c49bf1fbe140388cd2f619 sleep_before_teardown: 0 subset: 111/12000 suite: rgw suite_branch: reef-release suite_path: /home/teuthworker/src/github.com_ceph_ceph-c_2870bf456d3115490c153abce9d47ac5c043b65e/qa suite_relpath: qa suite_repo: https://github.com/ceph/ceph-ci.git suite_sha1: 2870bf456d3115490c153abce9d47ac5c043b65e targets: smithi099.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDox97V1ZbGiXYfIam0QsAKCOossAxUlL++9Y+qMtFgf7WbqpNcOYHYWWtH1ov+9qROYmTCRNQIFX+NkdHU8Q2Wbwg5LFP43nv89monL09OSkXGOUGTF6MF+MDeWU/gCRrcDxxs4QnoWTTbcd4HtqMzUprCFR5KLeO4XJSkTCxXlVJC1I2F9x9uGkv5VM4gB0ZVUeoVsATmS79+GLwNLH2DhXh4JQJL9HKgC9FN03CerL+Ka6cdcu3Wc6rSelCxMx0Ead4Vsre8il1J8PfCPvcko7HAkfp2XPu3avJXTKUTKYrF05s3uZq027l5T6VrlavgVtfGsyB8Ez4YDolVQOoGGfF2I99aVh9LAYc+J8KOY1Qh5KaILCCmrMI74XRPrqPrg7phDNIDNwjAxPcmoKpNDR6ALert5o+mM8onphQ/yRxisjwMH/bNXE63CdbjgkIOm2VJ2g/k1Ncf4lBxUPSx3eF+ERlKFdboN8S567n9hWRVwbSdGmzQf3AsOb38Hlk= smithi112.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBFfCJhk9QMgDL70wZsKQnRvWaVVb4spsBOWpIa4TPw/+clTUBIAM2haG88c+/+tNQhpaYNEiHo4bsY1a+mmIK9A= tasks: - install: null - ceph: cluster: c1 - ceph: cluster: c2 - rgw: c1.client.0: port: 8000 c1.client.1: port: 8001 c2.client.0: port: 8000 c2.client.1: port: 8001 - rgw-multisite: null - rgw-multisite-tests: config: reconfigure_delay: 60 teuthology: fragments_dropped: [] meta: {} postmerge: [] teuthology_branch: main teuthology_sha1: fa97ddd17d12e6d74b49196948f66e54fa009130 timestamp: 2023-05-31_19:25:20 tube: smithi user: yuriw verbose: true worker_log: /home/teuthworker/archive/worker_logs/dispatcher.smithi.23459 2023-05-31T19:32:20.881 INFO:teuthology.run:suite_path is set to /home/teuthworker/src/github.com_ceph_ceph-c_2870bf456d3115490c153abce9d47ac5c043b65e/qa; will attempt to use it 2023-05-31T19:32:20.881 INFO:teuthology.run:Found tasks at /home/teuthworker/src/github.com_ceph_ceph-c_2870bf456d3115490c153abce9d47ac5c043b65e/qa/tasks 2023-05-31T19:32:20.882 INFO:teuthology.run_tasks:Running task internal.check_packages... 2023-05-31T19:32:20.884 INFO:teuthology.task.internal:Checking packages... 2023-05-31T19:32:20.916 INFO:teuthology.task.internal:Checking packages for os_type 'centos', flavor 'default' and ceph hash '4a02f3f496d9039326c49bf1fbe140388cd2f619' 2023-05-31T19:32:20.916 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2023-05-31T19:32:20.917 INFO:teuthology.packaging:ref: None 2023-05-31T19:32:20.917 INFO:teuthology.packaging:tag: None 2023-05-31T19:32:20.917 INFO:teuthology.packaging:branch: reef-release 2023-05-31T19:32:20.918 INFO:teuthology.packaging:sha1: 4a02f3f496d9039326c49bf1fbe140388cd2f619 2023-05-31T19:32:20.918 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F8%2Fx86_64&ref=reef-release 2023-05-31T19:32:21.063 INFO:teuthology.task.internal:Found packages for ceph version 18.0.0-3488.g2870bf45 2023-05-31T19:32:21.064 INFO:teuthology.run_tasks:Running task internal.buildpackages_prep... 2023-05-31T19:32:21.069 INFO:teuthology.task.internal:no buildpackages task found 2023-05-31T19:32:21.070 INFO:teuthology.run_tasks:Running task internal.save_config... 2023-05-31T19:32:21.080 INFO:teuthology.task.internal:Saving configuration 2023-05-31T19:32:21.095 INFO:teuthology.run_tasks:Running task internal.check_lock... 2023-05-31T19:32:21.101 INFO:teuthology.task.internal.check_lock:Checking locks... 2023-05-31T19:32:21.129 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi099.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/yuriw-2023-05-31_19:25:20-rgw-reef-release-distro-default-smithi/7292285', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '8', 'arch': 'x86_64', 'locked': True, 'locked_since': '2023-05-31 19:27:13.562019', 'locked_by': 'scheduled_yuriw@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBBT+8m3VeJE1EnaoeX2K6/4NrcXkJHmAIDqY3cjRcIHP1gre8XKxaFODjL9z/kuMQPxwtIwy3e9fLkCPItpXuSE='} 2023-05-31T19:32:21.156 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi112.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/yuriw-2023-05-31_19:25:20-rgw-reef-release-distro-default-smithi/7292285', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '8', 'arch': 'x86_64', 'locked': True, 'locked_since': '2023-05-31 19:27:13.563506', 'locked_by': 'scheduled_yuriw@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBFfCJhk9QMgDL70wZsKQnRvWaVVb4spsBOWpIa4TPw/+clTUBIAM2haG88c+/+tNQhpaYNEiHo4bsY1a+mmIK9A='} 2023-05-31T19:32:21.156 INFO:teuthology.run_tasks:Running task internal.add_remotes... 2023-05-31T19:32:21.163 INFO:teuthology.task.internal:roles: ubuntu@smithi099.front.sepia.ceph.com - ['c1.mon.a', 'c1.mgr.x', 'c1.osd.0', 'c1.osd.1', 'c1.osd.2', 'c1.client.0', 'c1.client.1'] 2023-05-31T19:32:21.164 INFO:teuthology.task.internal:roles: ubuntu@smithi112.front.sepia.ceph.com - ['c2.mon.a', 'c2.mgr.x', 'c2.osd.0', 'c2.osd.1', 'c2.osd.2', 'c2.client.0', 'c2.client.1'] 2023-05-31T19:32:21.164 INFO:teuthology.run_tasks:Running task console_log... 2023-05-31T19:32:21.244 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x7f29b6aced30>, signals=[15]) 2023-05-31T19:32:21.246 INFO:teuthology.run_tasks:Running task internal.connect... 2023-05-31T19:32:21.252 INFO:teuthology.task.internal:Opening connections... 2023-05-31T19:32:21.253 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi099.front.sepia.ceph.com 2023-05-31T19:32:21.254 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi099.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-05-31T19:32:21.324 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi112.front.sepia.ceph.com 2023-05-31T19:32:21.325 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi112.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-05-31T19:32:21.389 INFO:teuthology.run_tasks:Running task internal.push_inventory... 2023-05-31T19:32:21.400 DEBUG:teuthology.orchestra.run.smithi099:> uname -m 2023-05-31T19:32:21.421 INFO:teuthology.orchestra.run.smithi099.stdout:x86_64 2023-05-31T19:32:21.422 DEBUG:teuthology.orchestra.run.smithi099:> cat /etc/os-release 2023-05-31T19:32:21.482 INFO:teuthology.orchestra.run.smithi099.stdout:NAME="CentOS Stream" 2023-05-31T19:32:21.483 INFO:teuthology.orchestra.run.smithi099.stdout:VERSION="8" 2023-05-31T19:32:21.483 INFO:teuthology.orchestra.run.smithi099.stdout:ID="centos" 2023-05-31T19:32:21.483 INFO:teuthology.orchestra.run.smithi099.stdout:ID_LIKE="rhel fedora" 2023-05-31T19:32:21.483 INFO:teuthology.orchestra.run.smithi099.stdout:VERSION_ID="8" 2023-05-31T19:32:21.483 INFO:teuthology.orchestra.run.smithi099.stdout:PLATFORM_ID="platform:el8" 2023-05-31T19:32:21.483 INFO:teuthology.orchestra.run.smithi099.stdout:PRETTY_NAME="CentOS Stream 8" 2023-05-31T19:32:21.483 INFO:teuthology.orchestra.run.smithi099.stdout:ANSI_COLOR="0;31" 2023-05-31T19:32:21.484 INFO:teuthology.orchestra.run.smithi099.stdout:CPE_NAME="cpe:/o:centos:centos:8" 2023-05-31T19:32:21.484 INFO:teuthology.orchestra.run.smithi099.stdout:HOME_URL="https://centos.org/" 2023-05-31T19:32:21.484 INFO:teuthology.orchestra.run.smithi099.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2023-05-31T19:32:21.484 INFO:teuthology.orchestra.run.smithi099.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 8" 2023-05-31T19:32:21.484 INFO:teuthology.orchestra.run.smithi099.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2023-05-31T19:32:21.485 INFO:teuthology.lock.ops:Updating smithi099.front.sepia.ceph.com on lock server 2023-05-31T19:32:21.508 DEBUG:teuthology.orchestra.run.smithi112:> uname -m 2023-05-31T19:32:21.530 INFO:teuthology.orchestra.run.smithi112.stdout:x86_64 2023-05-31T19:32:21.531 DEBUG:teuthology.orchestra.run.smithi112:> cat /etc/os-release 2023-05-31T19:32:21.591 INFO:teuthology.orchestra.run.smithi112.stdout:NAME="CentOS Stream" 2023-05-31T19:32:21.592 INFO:teuthology.orchestra.run.smithi112.stdout:VERSION="8" 2023-05-31T19:32:21.592 INFO:teuthology.orchestra.run.smithi112.stdout:ID="centos" 2023-05-31T19:32:21.592 INFO:teuthology.orchestra.run.smithi112.stdout:ID_LIKE="rhel fedora" 2023-05-31T19:32:21.592 INFO:teuthology.orchestra.run.smithi112.stdout:VERSION_ID="8" 2023-05-31T19:32:21.592 INFO:teuthology.orchestra.run.smithi112.stdout:PLATFORM_ID="platform:el8" 2023-05-31T19:32:21.593 INFO:teuthology.orchestra.run.smithi112.stdout:PRETTY_NAME="CentOS Stream 8" 2023-05-31T19:32:21.593 INFO:teuthology.orchestra.run.smithi112.stdout:ANSI_COLOR="0;31" 2023-05-31T19:32:21.593 INFO:teuthology.orchestra.run.smithi112.stdout:CPE_NAME="cpe:/o:centos:centos:8" 2023-05-31T19:32:21.593 INFO:teuthology.orchestra.run.smithi112.stdout:HOME_URL="https://centos.org/" 2023-05-31T19:32:21.593 INFO:teuthology.orchestra.run.smithi112.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2023-05-31T19:32:21.593 INFO:teuthology.orchestra.run.smithi112.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 8" 2023-05-31T19:32:21.593 INFO:teuthology.orchestra.run.smithi112.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2023-05-31T19:32:21.594 INFO:teuthology.lock.ops:Updating smithi112.front.sepia.ceph.com on lock server 2023-05-31T19:32:21.616 INFO:teuthology.run_tasks:Running task internal.serialize_remote_roles... 2023-05-31T19:32:21.625 INFO:teuthology.run_tasks:Running task internal.check_conflict... 2023-05-31T19:32:21.630 INFO:teuthology.task.internal:Checking for old test directory... 2023-05-31T19:32:21.631 DEBUG:teuthology.orchestra.run.smithi099:> test '!' -e /home/ubuntu/cephtest 2023-05-31T19:32:21.633 DEBUG:teuthology.orchestra.run.smithi112:> test '!' -e /home/ubuntu/cephtest 2023-05-31T19:32:21.652 INFO:teuthology.run_tasks:Running task internal.check_ceph_data... 2023-05-31T19:32:21.659 INFO:teuthology.task.internal:Checking for non-empty /var/lib/ceph... 2023-05-31T19:32:21.659 DEBUG:teuthology.orchestra.run.smithi099:> test -z $(ls -A /var/lib/ceph) 2023-05-31T19:32:21.693 DEBUG:teuthology.orchestra.run.smithi112:> test -z $(ls -A /var/lib/ceph) 2023-05-31T19:32:21.720 INFO:teuthology.run_tasks:Running task internal.vm_setup... 2023-05-31T19:32:21.778 INFO:teuthology.run_tasks:Running task kernel... 2023-05-31T19:32:21.793 INFO:teuthology.task.kernel:normalize config orig: {'kdb': True, 'sha1': 'distro'} 2023-05-31T19:32:21.794 INFO:teuthology.task.kernel:config {'c1.mon.a': {'kdb': True, 'sha1': 'distro'}, 'c2.mon.a': {'kdb': True, 'sha1': 'distro'}}, timeout 300 2023-05-31T19:32:21.794 DEBUG:teuthology.orchestra.run.smithi099:> test -f /run/.containerenv -o -f /.dockerenv 2023-05-31T19:32:21.795 DEBUG:teuthology.orchestra.run.smithi112:> test -f /run/.containerenv -o -f /.dockerenv 2023-05-31T19:32:21.814 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-05-31T19:32:21.815 DEBUG:teuthology.orchestra.run.smithi099:> uname -r 2023-05-31T19:32:21.816 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-05-31T19:32:21.816 DEBUG:teuthology.orchestra.run.smithi112:> uname -r 2023-05-31T19:32:21.876 INFO:teuthology.orchestra.run.smithi099.stdout:4.18.0-489.el8.x86_64 2023-05-31T19:32:21.876 INFO:teuthology.task.kernel:Running kernel on smithi099: 4.18.0-489.el8.x86_64 2023-05-31T19:32:21.876 DEBUG:teuthology.orchestra.run.smithi099:> sudo yum install -y kernel 2023-05-31T19:32:21.877 INFO:teuthology.orchestra.run.smithi112.stdout:4.18.0-489.el8.x86_64 2023-05-31T19:32:21.878 INFO:teuthology.task.kernel:Running kernel on smithi112: 4.18.0-489.el8.x86_64 2023-05-31T19:32:21.878 DEBUG:teuthology.orchestra.run.smithi112:> sudo yum install -y kernel 2023-05-31T19:32:25.650 INFO:teuthology.orchestra.run.smithi099.stdout:CentOS Stream 8 - AppStream 26 MB/s | 29 MB 00:01 2023-05-31T19:32:26.203 INFO:teuthology.orchestra.run.smithi112.stdout:CentOS Stream 8 - AppStream 17 MB/s | 29 MB 00:01 2023-05-31T19:32:32.475 INFO:teuthology.orchestra.run.smithi099.stdout:CentOS Stream 8 - BaseOS 22 MB/s | 36 MB 00:01 2023-05-31T19:32:33.037 INFO:teuthology.orchestra.run.smithi112.stdout:CentOS Stream 8 - BaseOS 22 MB/s | 36 MB 00:01 2023-05-31T19:32:38.091 INFO:teuthology.orchestra.run.smithi099.stdout:CentOS Stream 8 - Extras 50 kB/s | 18 kB 00:00 2023-05-31T19:32:38.383 INFO:teuthology.orchestra.run.smithi099.stdout:CentOS Stream 8 - Extras common packages 49 kB/s | 6.6 kB 00:00 2023-05-31T19:32:38.504 INFO:teuthology.orchestra.run.smithi112.stdout:CentOS Stream 8 - Extras 79 kB/s | 18 kB 00:00 2023-05-31T19:32:38.804 INFO:teuthology.orchestra.run.smithi112.stdout:CentOS Stream 8 - Extras common packages 46 kB/s | 6.6 kB 00:00 2023-05-31T19:32:40.760 INFO:teuthology.orchestra.run.smithi099.stdout:Copr repo for python3-asyncssh owned by ceph 1.5 kB/s | 3.5 kB 00:02 2023-05-31T19:32:41.472 INFO:teuthology.orchestra.run.smithi112.stdout:Copr repo for python3-asyncssh owned by ceph 1.4 kB/s | 3.5 kB 00:02 2023-05-31T19:32:42.712 INFO:teuthology.orchestra.run.smithi099.stdout:Extra Packages for Enterprise Linux 7.7 MB/s | 14 MB 00:01 2023-05-31T19:32:45.917 INFO:teuthology.orchestra.run.smithi112.stdout:Extra Packages for Enterprise Linux 3.2 MB/s | 14 MB 00:04 2023-05-31T19:32:46.143 INFO:teuthology.orchestra.run.smithi099.stdout:lab-extras 464 kB/s | 24 kB 00:00 2023-05-31T19:32:49.125 INFO:teuthology.orchestra.run.smithi099.stdout:Package kernel-4.18.0-348.el8.x86_64 is already installed. 2023-05-31T19:32:49.125 INFO:teuthology.orchestra.run.smithi099.stdout:Package kernel-4.18.0-489.el8.x86_64 is already installed. 2023-05-31T19:32:49.298 INFO:teuthology.orchestra.run.smithi112.stdout:lab-extras 495 kB/s | 24 kB 00:00 2023-05-31T19:32:49.330 INFO:teuthology.orchestra.run.smithi099.stdout:Dependencies resolved. 2023-05-31T19:32:49.331 INFO:teuthology.orchestra.run.smithi099.stdout:================================================================================ 2023-05-31T19:32:49.331 INFO:teuthology.orchestra.run.smithi099.stdout: Package Architecture Version Repository Size 2023-05-31T19:32:49.331 INFO:teuthology.orchestra.run.smithi099.stdout:================================================================================ 2023-05-31T19:32:49.331 INFO:teuthology.orchestra.run.smithi099.stdout:Installing: 2023-05-31T19:32:49.332 INFO:teuthology.orchestra.run.smithi099.stdout: kernel x86_64 4.18.0-492.el8 baseos 9.5 M 2023-05-31T19:32:49.332 INFO:teuthology.orchestra.run.smithi099.stdout:Installing dependencies: 2023-05-31T19:32:49.332 INFO:teuthology.orchestra.run.smithi099.stdout: kernel-core x86_64 4.18.0-492.el8 baseos 42 M 2023-05-31T19:32:49.332 INFO:teuthology.orchestra.run.smithi099.stdout: kernel-modules x86_64 4.18.0-492.el8 baseos 35 M 2023-05-31T19:32:49.332 INFO:teuthology.orchestra.run.smithi099.stdout: 2023-05-31T19:32:49.332 INFO:teuthology.orchestra.run.smithi099.stdout:Transaction Summary 2023-05-31T19:32:49.333 INFO:teuthology.orchestra.run.smithi099.stdout:================================================================================ 2023-05-31T19:32:49.333 INFO:teuthology.orchestra.run.smithi099.stdout:Install 3 Packages 2023-05-31T19:32:49.333 INFO:teuthology.orchestra.run.smithi099.stdout: 2023-05-31T19:32:49.333 INFO:teuthology.orchestra.run.smithi099.stdout:Total download size: 87 M 2023-05-31T19:32:49.334 INFO:teuthology.orchestra.run.smithi099.stdout:Installed size: 95 M 2023-05-31T19:32:49.334 INFO:teuthology.orchestra.run.smithi099.stdout:Downloading Packages: 2023-05-31T19:32:50.361 INFO:teuthology.orchestra.run.smithi099.stdout:(1/3): kernel-4.18.0-492.el8.x86_64.rpm 11 MB/s | 9.5 MB 00:00 2023-05-31T19:32:51.619 INFO:teuthology.orchestra.run.smithi099.stdout:(2/3): kernel-modules-4.18.0-492.el8.x86_64.rpm 17 MB/s | 35 MB 00:02 2023-05-31T19:32:52.020 INFO:teuthology.orchestra.run.smithi099.stdout:(3/3): kernel-core-4.18.0-492.el8.x86_64.rpm 17 MB/s | 42 MB 00:02 2023-05-31T19:32:52.020 INFO:teuthology.orchestra.run.smithi099.stdout:-------------------------------------------------------------------------------- 2023-05-31T19:32:52.021 INFO:teuthology.orchestra.run.smithi099.stdout:Total 32 MB/s | 87 MB 00:02 2023-05-31T19:32:52.338 INFO:teuthology.orchestra.run.smithi112.stdout:Package kernel-4.18.0-348.el8.x86_64 is already installed. 2023-05-31T19:32:52.338 INFO:teuthology.orchestra.run.smithi112.stdout:Package kernel-4.18.0-489.el8.x86_64 is already installed. 2023-05-31T19:32:52.553 INFO:teuthology.orchestra.run.smithi112.stdout:Dependencies resolved. 2023-05-31T19:32:52.553 INFO:teuthology.orchestra.run.smithi112.stdout:================================================================================ 2023-05-31T19:32:52.553 INFO:teuthology.orchestra.run.smithi112.stdout: Package Architecture Version Repository Size 2023-05-31T19:32:52.554 INFO:teuthology.orchestra.run.smithi112.stdout:================================================================================ 2023-05-31T19:32:52.554 INFO:teuthology.orchestra.run.smithi112.stdout:Installing: 2023-05-31T19:32:52.554 INFO:teuthology.orchestra.run.smithi112.stdout: kernel x86_64 4.18.0-492.el8 baseos 9.5 M 2023-05-31T19:32:52.554 INFO:teuthology.orchestra.run.smithi112.stdout:Installing dependencies: 2023-05-31T19:32:52.554 INFO:teuthology.orchestra.run.smithi112.stdout: kernel-core x86_64 4.18.0-492.el8 baseos 42 M 2023-05-31T19:32:52.554 INFO:teuthology.orchestra.run.smithi112.stdout: kernel-modules x86_64 4.18.0-492.el8 baseos 35 M 2023-05-31T19:32:52.554 INFO:teuthology.orchestra.run.smithi112.stdout: 2023-05-31T19:32:52.555 INFO:teuthology.orchestra.run.smithi112.stdout:Transaction Summary 2023-05-31T19:32:52.555 INFO:teuthology.orchestra.run.smithi112.stdout:================================================================================ 2023-05-31T19:32:52.555 INFO:teuthology.orchestra.run.smithi112.stdout:Install 3 Packages 2023-05-31T19:32:52.555 INFO:teuthology.orchestra.run.smithi112.stdout: 2023-05-31T19:32:52.556 INFO:teuthology.orchestra.run.smithi112.stdout:Total download size: 87 M 2023-05-31T19:32:52.556 INFO:teuthology.orchestra.run.smithi112.stdout:Installed size: 95 M 2023-05-31T19:32:52.556 INFO:teuthology.orchestra.run.smithi112.stdout:Downloading Packages: 2023-05-31T19:32:52.677 INFO:teuthology.orchestra.run.smithi099.stdout:Running transaction check 2023-05-31T19:32:53.108 INFO:teuthology.orchestra.run.smithi099.stdout:Transaction check succeeded. 2023-05-31T19:32:53.108 INFO:teuthology.orchestra.run.smithi099.stdout:Running transaction test 2023-05-31T19:32:53.391 INFO:teuthology.orchestra.run.smithi112.stdout:(1/3): kernel-4.18.0-492.el8.x86_64.rpm 13 MB/s | 9.5 MB 00:00 2023-05-31T19:32:54.300 INFO:teuthology.orchestra.run.smithi099.stdout:Transaction test succeeded. 2023-05-31T19:32:54.324 INFO:teuthology.orchestra.run.smithi099.stdout:Running transaction 2023-05-31T19:32:54.515 INFO:teuthology.orchestra.run.smithi112.stdout:(2/3): kernel-modules-4.18.0-492.el8.x86_64.rpm 19 MB/s | 35 MB 00:01 2023-05-31T19:32:55.109 INFO:teuthology.orchestra.run.smithi112.stdout:(3/3): kernel-core-4.18.0-492.el8.x86_64.rpm 17 MB/s | 42 MB 00:02 2023-05-31T19:32:55.109 INFO:teuthology.orchestra.run.smithi112.stdout:-------------------------------------------------------------------------------- 2023-05-31T19:32:55.110 INFO:teuthology.orchestra.run.smithi112.stdout:Total 34 MB/s | 87 MB 00:02 2023-05-31T19:32:55.767 INFO:teuthology.orchestra.run.smithi112.stdout:Running transaction check 2023-05-31T19:32:56.197 INFO:teuthology.orchestra.run.smithi112.stdout:Transaction check succeeded. 2023-05-31T19:32:56.197 INFO:teuthology.orchestra.run.smithi112.stdout:Running transaction test 2023-05-31T19:32:57.072 INFO:teuthology.orchestra.run.smithi099.stdout: Preparing : 1/1 2023-05-31T19:32:57.393 INFO:teuthology.orchestra.run.smithi112.stdout:Transaction test succeeded. 2023-05-31T19:32:57.426 INFO:teuthology.orchestra.run.smithi112.stdout:Running transaction 2023-05-31T19:32:57.576 INFO:teuthology.orchestra.run.smithi099.stdout: Installing : kernel-core-4.18.0-492.el8.x86_64 1/3 2023-05-31T19:33:00.159 INFO:teuthology.orchestra.run.smithi112.stdout: Preparing : 1/1 2023-05-31T19:33:00.704 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : kernel-core-4.18.0-492.el8.x86_64 1/3 2023-05-31T19:33:01.178 INFO:teuthology.orchestra.run.smithi099.stdout: Running scriptlet: kernel-core-4.18.0-492.el8.x86_64 1/3 2023-05-31T19:33:01.592 INFO:teuthology.orchestra.run.smithi099.stdout: Installing : kernel-modules-4.18.0-492.el8.x86_64 2/3 2023-05-31T19:33:04.273 INFO:teuthology.orchestra.run.smithi112.stdout: Running scriptlet: kernel-core-4.18.0-492.el8.x86_64 1/3 2023-05-31T19:33:04.670 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : kernel-modules-4.18.0-492.el8.x86_64 2/3 2023-05-31T19:33:04.932 INFO:teuthology.orchestra.run.smithi099.stdout: Running scriptlet: kernel-modules-4.18.0-492.el8.x86_64 2/3 2023-05-31T19:33:05.208 INFO:teuthology.orchestra.run.smithi099.stdout: Installing : kernel-4.18.0-492.el8.x86_64 3/3 2023-05-31T19:33:08.013 INFO:teuthology.orchestra.run.smithi112.stdout: Running scriptlet: kernel-modules-4.18.0-492.el8.x86_64 2/3 2023-05-31T19:33:08.279 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : kernel-4.18.0-492.el8.x86_64 3/3 2023-05-31T19:33:26.993 INFO:teuthology.orchestra.run.smithi099.stdout: Running scriptlet: kernel-core-4.18.0-492.el8.x86_64 3/3 2023-05-31T19:33:27.016 INFO:teuthology.orchestra.run.smithi099.stdout: Running scriptlet: kernel-modules-4.18.0-492.el8.x86_64 3/3 2023-05-31T19:33:27.363 INFO:teuthology.orchestra.run.smithi099.stdout: Running scriptlet: kernel-4.18.0-492.el8.x86_64 3/3 2023-05-31T19:33:27.363 INFO:teuthology.orchestra.run.smithi099.stdout: Verifying : kernel-4.18.0-492.el8.x86_64 1/3 2023-05-31T19:33:27.363 INFO:teuthology.orchestra.run.smithi099.stdout: Verifying : kernel-core-4.18.0-492.el8.x86_64 2/3 2023-05-31T19:33:28.024 INFO:teuthology.orchestra.run.smithi099.stdout: Verifying : kernel-modules-4.18.0-492.el8.x86_64 3/3 2023-05-31T19:33:28.025 INFO:teuthology.orchestra.run.smithi099.stdout: 2023-05-31T19:33:28.025 INFO:teuthology.orchestra.run.smithi099.stdout:Installed: 2023-05-31T19:33:28.025 INFO:teuthology.orchestra.run.smithi099.stdout: kernel-4.18.0-492.el8.x86_64 kernel-core-4.18.0-492.el8.x86_64 2023-05-31T19:33:28.025 INFO:teuthology.orchestra.run.smithi099.stdout: kernel-modules-4.18.0-492.el8.x86_64 2023-05-31T19:33:28.025 INFO:teuthology.orchestra.run.smithi099.stdout: 2023-05-31T19:33:28.026 INFO:teuthology.orchestra.run.smithi099.stdout:Complete! 2023-05-31T19:33:28.160 DEBUG:teuthology.orchestra.run.smithi099:> rpm -q kernel | sort -rV | head -n 1 2023-05-31T19:33:28.375 INFO:teuthology.orchestra.run.smithi099.stdout:kernel-4.18.0-492.el8.x86_64 2023-05-31T19:33:28.376 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 4.18.0-492.el8.x86_64 2023-05-31T19:33:28.376 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 4.18.0-489.el8.x86_64 Expected: 4.18.0-492.el8.x86_64 2023-05-31T19:33:28.376 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2023-05-31T19:33:28.377 DEBUG:teuthology.task.kernel:src is distro, skipping download 2023-05-31T19:33:28.377 INFO:teuthology.task.kernel:Installing distro kernel on c1.mon.a... 2023-05-31T19:33:28.377 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi099.front.sepia.ceph.com, path=None, version=distro) 2023-05-31T19:33:28.377 DEBUG:teuthology.orchestra.run.smithi099:> rpm -q kernel | sort -rV | head -n 1 2023-05-31T19:33:28.581 INFO:teuthology.orchestra.run.smithi099.stdout:kernel-4.18.0-492.el8.x86_64 2023-05-31T19:33:28.581 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 4.18.0-492.el8.x86_64 2023-05-31T19:33:28.582 DEBUG:teuthology.orchestra.run.smithi099:> sudo rpm -qi grub2-tools 2023-05-31T19:33:28.616 INFO:teuthology.orchestra.run.smithi099.stdout:Name : grub2-tools 2023-05-31T19:33:28.616 INFO:teuthology.orchestra.run.smithi099.stdout:Epoch : 1 2023-05-31T19:33:28.616 INFO:teuthology.orchestra.run.smithi099.stdout:Version : 2.02 2023-05-31T19:33:28.617 INFO:teuthology.orchestra.run.smithi099.stdout:Release : 129.el8 2023-05-31T19:33:28.617 INFO:teuthology.orchestra.run.smithi099.stdout:Architecture: x86_64 2023-05-31T19:33:28.618 INFO:teuthology.orchestra.run.smithi099.stdout:Install Date: Thu 04 May 2023 10:17:48 PM UTC 2023-05-31T19:33:28.618 INFO:teuthology.orchestra.run.smithi099.stdout:Group : System Environment/Base 2023-05-31T19:33:28.618 INFO:teuthology.orchestra.run.smithi099.stdout:Size : 9265586 2023-05-31T19:33:28.619 INFO:teuthology.orchestra.run.smithi099.stdout:License : GPLv3+ 2023-05-31T19:33:28.619 INFO:teuthology.orchestra.run.smithi099.stdout:Signature : RSA/SHA256, Tue 19 Jul 2022 02:52:06 PM UTC, Key ID 05b555b38483c65d 2023-05-31T19:33:28.619 INFO:teuthology.orchestra.run.smithi099.stdout:Source RPM : grub2-2.02-129.el8.src.rpm 2023-05-31T19:33:28.620 INFO:teuthology.orchestra.run.smithi099.stdout:Build Date : Tue 19 Jul 2022 02:30:39 PM UTC 2023-05-31T19:33:28.620 INFO:teuthology.orchestra.run.smithi099.stdout:Build Host : kbuilder.bsys.centos.org 2023-05-31T19:33:28.620 INFO:teuthology.orchestra.run.smithi099.stdout:Relocations : (not relocatable) 2023-05-31T19:33:28.620 INFO:teuthology.orchestra.run.smithi099.stdout:Packager : CentOS BuildSystem 2023-05-31T19:33:28.621 INFO:teuthology.orchestra.run.smithi099.stdout:Vendor : CentOS 2023-05-31T19:33:28.621 INFO:teuthology.orchestra.run.smithi099.stdout:URL : http://www.gnu.org/software/grub/ 2023-05-31T19:33:28.621 INFO:teuthology.orchestra.run.smithi099.stdout:Summary : Support tools for GRUB. 2023-05-31T19:33:28.621 INFO:teuthology.orchestra.run.smithi099.stdout:Description : 2023-05-31T19:33:28.622 INFO:teuthology.orchestra.run.smithi099.stdout: 2023-05-31T19:33:28.622 INFO:teuthology.orchestra.run.smithi099.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2023-05-31T19:33:28.622 INFO:teuthology.orchestra.run.smithi099.stdout:customizable bootloader with modular architecture. It supports a rich 2023-05-31T19:33:28.622 INFO:teuthology.orchestra.run.smithi099.stdout:variety of kernel formats, file systems, computer architectures and 2023-05-31T19:33:28.623 INFO:teuthology.orchestra.run.smithi099.stdout:hardware devices. 2023-05-31T19:33:28.623 INFO:teuthology.orchestra.run.smithi099.stdout: 2023-05-31T19:33:28.623 INFO:teuthology.orchestra.run.smithi099.stdout:This subpackage provides tools for support of all platforms. 2023-05-31T19:33:28.625 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2023-05-31T19:33:28.625 INFO:teuthology.task.kernel:Updating grub on smithi099 to boot 4.18.0-492.el8.x86_64 2023-05-31T19:33:28.626 DEBUG:teuthology.orchestra.run.smithi099:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2023-05-31T19:33:29.917 INFO:teuthology.orchestra.run.smithi112.stdout: Running scriptlet: kernel-core-4.18.0-492.el8.x86_64 3/3 2023-05-31T19:33:29.940 INFO:teuthology.orchestra.run.smithi112.stdout: Running scriptlet: kernel-modules-4.18.0-492.el8.x86_64 3/3 2023-05-31T19:33:30.283 INFO:teuthology.orchestra.run.smithi112.stdout: Running scriptlet: kernel-4.18.0-492.el8.x86_64 3/3 2023-05-31T19:33:30.284 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : kernel-4.18.0-492.el8.x86_64 1/3 2023-05-31T19:33:30.284 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : kernel-core-4.18.0-492.el8.x86_64 2/3 2023-05-31T19:33:30.318 INFO:teuthology.orchestra.run.smithi099.stderr:Generating grub configuration file ... 2023-05-31T19:33:30.986 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : kernel-modules-4.18.0-492.el8.x86_64 3/3 2023-05-31T19:33:30.986 INFO:teuthology.orchestra.run.smithi112.stdout: 2023-05-31T19:33:30.987 INFO:teuthology.orchestra.run.smithi112.stdout:Installed: 2023-05-31T19:33:30.987 INFO:teuthology.orchestra.run.smithi112.stdout: kernel-4.18.0-492.el8.x86_64 kernel-core-4.18.0-492.el8.x86_64 2023-05-31T19:33:30.987 INFO:teuthology.orchestra.run.smithi112.stdout: kernel-modules-4.18.0-492.el8.x86_64 2023-05-31T19:33:30.988 INFO:teuthology.orchestra.run.smithi112.stdout: 2023-05-31T19:33:30.988 INFO:teuthology.orchestra.run.smithi112.stdout:Complete! 2023-05-31T19:33:31.144 DEBUG:teuthology.orchestra.run.smithi112:> rpm -q kernel | sort -rV | head -n 1 2023-05-31T19:33:31.356 INFO:teuthology.orchestra.run.smithi112.stdout:kernel-4.18.0-492.el8.x86_64 2023-05-31T19:33:31.357 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 4.18.0-492.el8.x86_64 2023-05-31T19:33:31.357 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 4.18.0-489.el8.x86_64 Expected: 4.18.0-492.el8.x86_64 2023-05-31T19:33:31.357 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2023-05-31T19:33:31.357 DEBUG:teuthology.task.kernel:src is distro, skipping download 2023-05-31T19:33:31.357 INFO:teuthology.task.kernel:Installing distro kernel on c2.mon.a... 2023-05-31T19:33:31.358 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi112.front.sepia.ceph.com, path=None, version=distro) 2023-05-31T19:33:31.358 DEBUG:teuthology.orchestra.run.smithi112:> rpm -q kernel | sort -rV | head -n 1 2023-05-31T19:33:31.561 INFO:teuthology.orchestra.run.smithi112.stdout:kernel-4.18.0-492.el8.x86_64 2023-05-31T19:33:31.561 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 4.18.0-492.el8.x86_64 2023-05-31T19:33:31.562 DEBUG:teuthology.orchestra.run.smithi112:> sudo rpm -qi grub2-tools 2023-05-31T19:33:31.595 INFO:teuthology.orchestra.run.smithi112.stdout:Name : grub2-tools 2023-05-31T19:33:31.595 INFO:teuthology.orchestra.run.smithi112.stdout:Epoch : 1 2023-05-31T19:33:31.596 INFO:teuthology.orchestra.run.smithi112.stdout:Version : 2.02 2023-05-31T19:33:31.596 INFO:teuthology.orchestra.run.smithi112.stdout:Release : 129.el8 2023-05-31T19:33:31.596 INFO:teuthology.orchestra.run.smithi112.stdout:Architecture: x86_64 2023-05-31T19:33:31.597 INFO:teuthology.orchestra.run.smithi112.stdout:Install Date: Thu 04 May 2023 10:17:48 PM UTC 2023-05-31T19:33:31.597 INFO:teuthology.orchestra.run.smithi112.stdout:Group : System Environment/Base 2023-05-31T19:33:31.597 INFO:teuthology.orchestra.run.smithi112.stdout:Size : 9265586 2023-05-31T19:33:31.597 INFO:teuthology.orchestra.run.smithi112.stdout:License : GPLv3+ 2023-05-31T19:33:31.598 INFO:teuthology.orchestra.run.smithi112.stdout:Signature : RSA/SHA256, Tue 19 Jul 2022 02:52:06 PM UTC, Key ID 05b555b38483c65d 2023-05-31T19:33:31.598 INFO:teuthology.orchestra.run.smithi112.stdout:Source RPM : grub2-2.02-129.el8.src.rpm 2023-05-31T19:33:31.598 INFO:teuthology.orchestra.run.smithi112.stdout:Build Date : Tue 19 Jul 2022 02:30:39 PM UTC 2023-05-31T19:33:31.598 INFO:teuthology.orchestra.run.smithi112.stdout:Build Host : kbuilder.bsys.centos.org 2023-05-31T19:33:31.598 INFO:teuthology.orchestra.run.smithi112.stdout:Relocations : (not relocatable) 2023-05-31T19:33:31.598 INFO:teuthology.orchestra.run.smithi112.stdout:Packager : CentOS BuildSystem 2023-05-31T19:33:31.598 INFO:teuthology.orchestra.run.smithi112.stdout:Vendor : CentOS 2023-05-31T19:33:31.598 INFO:teuthology.orchestra.run.smithi112.stdout:URL : http://www.gnu.org/software/grub/ 2023-05-31T19:33:31.599 INFO:teuthology.orchestra.run.smithi112.stdout:Summary : Support tools for GRUB. 2023-05-31T19:33:31.599 INFO:teuthology.orchestra.run.smithi112.stdout:Description : 2023-05-31T19:33:31.599 INFO:teuthology.orchestra.run.smithi112.stdout: 2023-05-31T19:33:31.599 INFO:teuthology.orchestra.run.smithi112.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2023-05-31T19:33:31.599 INFO:teuthology.orchestra.run.smithi112.stdout:customizable bootloader with modular architecture. It supports a rich 2023-05-31T19:33:31.599 INFO:teuthology.orchestra.run.smithi112.stdout:variety of kernel formats, file systems, computer architectures and 2023-05-31T19:33:31.599 INFO:teuthology.orchestra.run.smithi112.stdout:hardware devices. 2023-05-31T19:33:31.600 INFO:teuthology.orchestra.run.smithi112.stdout: 2023-05-31T19:33:31.600 INFO:teuthology.orchestra.run.smithi112.stdout:This subpackage provides tools for support of all platforms. 2023-05-31T19:33:31.601 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2023-05-31T19:33:31.602 INFO:teuthology.task.kernel:Updating grub on smithi112 to boot 4.18.0-492.el8.x86_64 2023-05-31T19:33:31.602 DEBUG:teuthology.orchestra.run.smithi112:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2023-05-31T19:33:32.669 INFO:teuthology.orchestra.run.smithi099.stderr:done 2023-05-31T19:33:32.670 DEBUG:teuthology.orchestra.run.smithi099:> mktemp 2023-05-31T19:33:32.696 INFO:teuthology.orchestra.run.smithi099.stdout:/tmp/tmp.E4fKVg9Ya7 2023-05-31T19:33:32.696 DEBUG:teuthology.orchestra.run.smithi099:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.E4fKVg9Ya7 2023-05-31T19:33:32.774 DEBUG:teuthology.orchestra.run.smithi099:> sudo chmod 0666 /tmp/tmp.E4fKVg9Ya7 2023-05-31T19:33:32.919 DEBUG:teuthology.orchestra.remote:smithi099:/tmp/tmp.E4fKVg9Ya7 is 6KB 2023-05-31T19:33:32.942 DEBUG:teuthology.orchestra.run.smithi099:> rm -fr /tmp/tmp.E4fKVg9Ya7 2023-05-31T19:33:32.960 DEBUG:teuthology.orchestra.run.smithi099:> sudo /bin/ls /boot/loader/entries || true 2023-05-31T19:33:33.025 INFO:teuthology.orchestra.run.smithi099.stdout:a1b29d6c289747bdb22930598d39874a-0-rescue.conf 2023-05-31T19:33:33.025 INFO:teuthology.orchestra.run.smithi099.stdout:a1b29d6c289747bdb22930598d39874a-4.18.0-348.el8.x86_64.conf 2023-05-31T19:33:33.025 INFO:teuthology.orchestra.run.smithi099.stdout:a1b29d6c289747bdb22930598d39874a-4.18.0-489.el8.x86_64.conf 2023-05-31T19:33:33.025 INFO:teuthology.orchestra.run.smithi099.stdout:a1b29d6c289747bdb22930598d39874a-4.18.0-492.el8.x86_64.conf 2023-05-31T19:33:33.026 DEBUG:teuthology.orchestra.run.smithi099:> sudo grub2-set-default a1b29d6c289747bdb22930598d39874a-4.18.0-492.el8.x86_64 2023-05-31T19:33:33.175 DEBUG:teuthology.orchestra.run.smithi099:> sudo shutdown -r now 2023-05-31T19:33:33.186 INFO:teuthology.orchestra.run.smithi112.stderr:Generating grub configuration file ... 2023-05-31T19:33:35.491 INFO:teuthology.orchestra.run.smithi112.stderr:done 2023-05-31T19:33:35.492 DEBUG:teuthology.orchestra.run.smithi112:> mktemp 2023-05-31T19:33:35.515 INFO:teuthology.orchestra.run.smithi112.stdout:/tmp/tmp.xR5FODiUJ2 2023-05-31T19:33:35.515 DEBUG:teuthology.orchestra.run.smithi112:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.xR5FODiUJ2 2023-05-31T19:33:35.590 DEBUG:teuthology.orchestra.run.smithi112:> sudo chmod 0666 /tmp/tmp.xR5FODiUJ2 2023-05-31T19:33:35.736 DEBUG:teuthology.orchestra.remote:smithi112:/tmp/tmp.xR5FODiUJ2 is 6KB 2023-05-31T19:33:35.783 DEBUG:teuthology.orchestra.run.smithi112:> rm -fr /tmp/tmp.xR5FODiUJ2 2023-05-31T19:33:35.800 DEBUG:teuthology.orchestra.run.smithi112:> sudo /bin/ls /boot/loader/entries || true 2023-05-31T19:33:35.878 INFO:teuthology.orchestra.run.smithi112.stdout:a1b29d6c289747bdb22930598d39874a-0-rescue.conf 2023-05-31T19:33:35.878 INFO:teuthology.orchestra.run.smithi112.stdout:a1b29d6c289747bdb22930598d39874a-4.18.0-348.el8.x86_64.conf 2023-05-31T19:33:35.878 INFO:teuthology.orchestra.run.smithi112.stdout:a1b29d6c289747bdb22930598d39874a-4.18.0-489.el8.x86_64.conf 2023-05-31T19:33:35.879 INFO:teuthology.orchestra.run.smithi112.stdout:a1b29d6c289747bdb22930598d39874a-4.18.0-492.el8.x86_64.conf 2023-05-31T19:33:35.879 DEBUG:teuthology.orchestra.run.smithi112:> sudo grub2-set-default a1b29d6c289747bdb22930598d39874a-4.18.0-492.el8.x86_64 2023-05-31T19:33:36.044 DEBUG:teuthology.orchestra.run.smithi112:> sudo shutdown -r now 2023-05-31T19:34:03.204 INFO:teuthology.task.kernel:Checking client c1.mon.a for new kernel version... 2023-05-31T19:34:03.204 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi099.front.sepia.ceph.com' 2023-05-31T19:34:03.205 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi099.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-05-31T19:34:06.048 INFO:teuthology.task.kernel:Checking client c2.mon.a for new kernel version... 2023-05-31T19:34:06.048 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi112.front.sepia.ceph.com' 2023-05-31T19:34:06.049 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi112.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-05-31T19:35:03.264 DEBUG:teuthology.orchestra.remote:timed out 2023-05-31T19:35:06.052 DEBUG:teuthology.orchestra.remote:timed out 2023-05-31T19:35:33.292 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi099.front.sepia.ceph.com' 2023-05-31T19:35:33.293 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi099.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-05-31T19:35:33.638 DEBUG:teuthology.orchestra.run.smithi099:> true 2023-05-31T19:35:33.892 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi099.front.sepia.ceph.com' 2023-05-31T19:35:33.892 INFO:teuthology.task.kernel:Checking kernel version of c1.mon.a, want "4.18.0-492.el8.x86_64"... 2023-05-31T19:35:33.893 DEBUG:teuthology.orchestra.run.smithi099:> uname -r 2023-05-31T19:35:33.912 INFO:teuthology.orchestra.run.smithi099.stdout:4.18.0-492.el8.x86_64 2023-05-31T19:35:33.913 DEBUG:teuthology.task.kernel:current kernel version is 4.18.0-492.el8.x86_64 vs 4.18.0-492.el8.x86_64 2023-05-31T19:35:33.913 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2023-05-31T19:35:33.913 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2023-05-31T19:35:34.915 INFO:teuthology.task.kernel:Enabling kdb on c1.mon.a... 2023-05-31T19:35:34.915 DEBUG:teuthology.orchestra.run.smithi099:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2023-05-31T19:35:34.985 INFO:teuthology.orchestra.run.smithi099.stdout:ttyS1 2023-05-31T19:35:34.999 DEBUG:teuthology.parallel:result is None 2023-05-31T19:35:36.054 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi112.front.sepia.ceph.com' 2023-05-31T19:35:36.055 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi112.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-05-31T19:35:36.392 DEBUG:teuthology.orchestra.run.smithi112:> true 2023-05-31T19:35:36.676 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi112.front.sepia.ceph.com' 2023-05-31T19:35:36.677 INFO:teuthology.task.kernel:Checking kernel version of c2.mon.a, want "4.18.0-492.el8.x86_64"... 2023-05-31T19:35:36.677 DEBUG:teuthology.orchestra.run.smithi112:> uname -r 2023-05-31T19:35:36.696 INFO:teuthology.orchestra.run.smithi112.stdout:4.18.0-492.el8.x86_64 2023-05-31T19:35:36.696 DEBUG:teuthology.task.kernel:current kernel version is 4.18.0-492.el8.x86_64 vs 4.18.0-492.el8.x86_64 2023-05-31T19:35:36.697 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2023-05-31T19:35:36.697 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2023-05-31T19:35:37.698 INFO:teuthology.task.kernel:Enabling kdb on c2.mon.a... 2023-05-31T19:35:37.698 DEBUG:teuthology.orchestra.run.smithi112:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2023-05-31T19:35:37.768 INFO:teuthology.orchestra.run.smithi112.stdout:ttyS1 2023-05-31T19:35:37.781 DEBUG:teuthology.parallel:result is None 2023-05-31T19:35:37.782 INFO:teuthology.run_tasks:Running task internal.base... 2023-05-31T19:35:37.790 INFO:teuthology.task.internal:Creating test directory... 2023-05-31T19:35:37.790 DEBUG:teuthology.orchestra.run.smithi099:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2023-05-31T19:35:37.793 DEBUG:teuthology.orchestra.run.smithi112:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2023-05-31T19:35:37.816 INFO:teuthology.run_tasks:Running task internal.archive_upload... 2023-05-31T19:35:37.823 INFO:teuthology.run_tasks:Running task internal.archive... 2023-05-31T19:35:37.830 INFO:teuthology.task.internal:Creating archive directory... 2023-05-31T19:35:37.830 DEBUG:teuthology.orchestra.run.smithi099:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2023-05-31T19:35:37.857 DEBUG:teuthology.orchestra.run.smithi112:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2023-05-31T19:35:37.899 INFO:teuthology.run_tasks:Running task internal.coredump... 2023-05-31T19:35:37.906 INFO:teuthology.task.internal:Enabling coredump saving... 2023-05-31T19:35:37.906 DEBUG:teuthology.orchestra.run.smithi099:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2023-05-31T19:35:37.928 DEBUG:teuthology.orchestra.run.smithi112:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2023-05-31T19:35:37.974 INFO:teuthology.orchestra.run.smithi112.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2023-05-31T19:35:37.978 INFO:teuthology.orchestra.run.smithi099.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2023-05-31T19:35:37.985 INFO:teuthology.orchestra.run.smithi112.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2023-05-31T19:35:37.989 INFO:teuthology.orchestra.run.smithi099.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2023-05-31T19:35:37.991 INFO:teuthology.run_tasks:Running task internal.sudo... 2023-05-31T19:35:38.001 INFO:teuthology.task.internal:Configuring sudo... 2023-05-31T19:35:38.002 DEBUG:teuthology.orchestra.run.smithi099:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2023-05-31T19:35:38.004 DEBUG:teuthology.orchestra.run.smithi112:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2023-05-31T19:35:38.056 INFO:teuthology.run_tasks:Running task internal.syslog... 2023-05-31T19:35:38.069 INFO:teuthology.task.internal.syslog:Starting syslog monitoring... 2023-05-31T19:35:38.070 DEBUG:teuthology.orchestra.run.smithi099:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2023-05-31T19:35:38.075 DEBUG:teuthology.orchestra.run.smithi112:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2023-05-31T19:35:38.116 DEBUG:teuthology.orchestra.run.smithi099:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2023-05-31T19:35:38.184 DEBUG:teuthology.orchestra.run.smithi099:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2023-05-31T19:35:38.251 DEBUG:teuthology.orchestra.run.smithi099:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2023-05-31T19:35:38.308 DEBUG:teuthology.orchestra.run.smithi099:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2023-05-31T19:35:38.343 DEBUG:teuthology.orchestra.run.smithi099:> set -ex 2023-05-31T19:35:38.343 DEBUG:teuthology.orchestra.run.smithi099:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2023-05-31T19:35:38.411 DEBUG:teuthology.orchestra.run.smithi112:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2023-05-31T19:35:38.465 DEBUG:teuthology.orchestra.run.smithi112:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2023-05-31T19:35:38.518 DEBUG:teuthology.orchestra.run.smithi112:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2023-05-31T19:35:38.608 DEBUG:teuthology.orchestra.run.smithi112:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2023-05-31T19:35:38.650 DEBUG:teuthology.orchestra.run.smithi112:> set -ex 2023-05-31T19:35:38.651 DEBUG:teuthology.orchestra.run.smithi112:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2023-05-31T19:35:38.720 DEBUG:teuthology.orchestra.run.smithi099:> sudo service rsyslog restart 2023-05-31T19:35:38.723 DEBUG:teuthology.orchestra.run.smithi112:> sudo service rsyslog restart 2023-05-31T19:35:38.775 INFO:teuthology.orchestra.run.smithi099.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2023-05-31T19:35:38.821 INFO:teuthology.orchestra.run.smithi112.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2023-05-31T19:35:39.301 INFO:teuthology.run_tasks:Running task internal.timer... 2023-05-31T19:35:39.315 INFO:teuthology.task.internal:Starting timer... 2023-05-31T19:35:39.315 INFO:teuthology.run_tasks:Running task pcp... 2023-05-31T19:35:39.330 INFO:teuthology.run_tasks:Running task selinux... 2023-05-31T19:35:39.338 DEBUG:teuthology.task:Applying overrides for task selinux: {'whitelist': ['scontext=system_u:system_r:logrotate_t:s0']} 2023-05-31T19:35:39.339 DEBUG:teuthology.orchestra.run.smithi099:> sudo service auditd rotate 2023-05-31T19:35:39.404 INFO:teuthology.orchestra.run.smithi099.stdout:Rotating logs: 2023-05-31T19:35:39.406 DEBUG:teuthology.orchestra.run.smithi112:> sudo service auditd rotate 2023-05-31T19:35:39.497 INFO:teuthology.orchestra.run.smithi112.stdout:Rotating logs: 2023-05-31T19:35:39.499 DEBUG:teuthology.task.selinux:Getting current SELinux state 2023-05-31T19:35:39.500 DEBUG:teuthology.orchestra.run.smithi099:> /usr/sbin/getenforce 2023-05-31T19:35:39.560 INFO:teuthology.orchestra.run.smithi099.stdout:Permissive 2023-05-31T19:35:39.561 DEBUG:teuthology.orchestra.run.smithi112:> /usr/sbin/getenforce 2023-05-31T19:35:39.591 INFO:teuthology.orchestra.run.smithi112.stdout:Permissive 2023-05-31T19:35:39.591 DEBUG:teuthology.task.selinux:Existing SELinux modes: {'ubuntu@smithi099.front.sepia.ceph.com': 'permissive', 'ubuntu@smithi112.front.sepia.ceph.com': 'permissive'} 2023-05-31T19:35:39.592 DEBUG:teuthology.orchestra.run.smithi099:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2023-05-31T19:35:39.619 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-05-31T19:35:39.620 DEBUG:teuthology.orchestra.run.smithi112:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2023-05-31T19:35:39.661 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-05-31T19:35:39.661 INFO:teuthology.task.selinux:Putting SELinux into permissive mode 2023-05-31T19:35:39.662 DEBUG:teuthology.orchestra.run.smithi099:> sudo /usr/sbin/setenforce permissive 2023-05-31T19:35:39.689 DEBUG:teuthology.orchestra.run.smithi112:> sudo /usr/sbin/setenforce permissive 2023-05-31T19:35:39.729 INFO:teuthology.run_tasks:Running task ansible.cephlab... 2023-05-31T19:35:39.742 DEBUG:teuthology.repo_utils:Setting repo remote to https://git.ceph.com/ceph-cm-ansible.git 2023-05-31T19:35:39.749 INFO:teuthology.repo_utils:Fetching main from origin 2023-05-31T19:35:39.877 INFO:teuthology.repo_utils:Resetting repo at /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main to origin/main 2023-05-31T19:35:39.902 INFO:teuthology.task.ansible:Playbook: [{'import_playbook': 'ansible_managed.yml'}, {'import_playbook': 'teuthology.yml'}, {'hosts': 'testnodes', 'tasks': [{'set_fact': {'ran_from_cephlab_playbook': True}}]}, {'import_playbook': 'testnodes.yml'}, {'import_playbook': 'container-host.yml'}, {'import_playbook': 'cobbler.yml'}, {'import_playbook': 'paddles.yml'}, {'import_playbook': 'pulpito.yml'}, {'hosts': 'testnodes', 'become': True, 'tasks': [{'name': 'Touch /ceph-qa-ready', 'file': {'path': '/ceph-qa-ready', 'state': 'touch'}, 'when': 'ran_from_cephlab_playbook|bool'}]}] 2023-05-31T19:35:39.904 DEBUG:teuthology.task.ansible:Running ansible-playbook -v --extra-vars '{"ansible_ssh_user": "ubuntu"}' -i /etc/ansible/hosts --limit smithi099.front.sepia.ceph.com,smithi112.front.sepia.ceph.com /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main/cephlab.yml 2023-05-31T19:42:21.135 DEBUG:teuthology.task.ansible:Reconnecting to [Remote(name='ubuntu@smithi099.front.sepia.ceph.com'), Remote(name='ubuntu@smithi112.front.sepia.ceph.com')] 2023-05-31T19:42:21.139 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi099.front.sepia.ceph.com' 2023-05-31T19:42:21.140 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi099.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-05-31T19:42:21.209 DEBUG:teuthology.orchestra.run.smithi099:> true 2023-05-31T19:42:21.280 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi112.front.sepia.ceph.com' 2023-05-31T19:42:21.281 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi112.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-05-31T19:42:21.349 DEBUG:teuthology.orchestra.run.smithi112:> true 2023-05-31T19:42:21.379 INFO:teuthology.run_tasks:Running task clock... 2023-05-31T19:42:21.396 INFO:teuthology.task.clock:Syncing clocks and checking initial clock skew... 2023-05-31T19:42:21.396 INFO:teuthology.orchestra.run:Running command with timeout 360 2023-05-31T19:42:21.397 DEBUG:teuthology.orchestra.run.smithi099:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2023-05-31T19:42:21.399 INFO:teuthology.orchestra.run:Running command with timeout 360 2023-05-31T19:42:21.400 DEBUG:teuthology.orchestra.run.smithi112:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2023-05-31T19:42:21.429 INFO:teuthology.orchestra.run.smithi099.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2023-05-31T19:42:21.443 INFO:teuthology.orchestra.run.smithi099.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2023-05-31T19:42:21.454 INFO:teuthology.orchestra.run.smithi112.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2023-05-31T19:42:21.469 INFO:teuthology.orchestra.run.smithi112.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2023-05-31T19:42:21.504 INFO:teuthology.orchestra.run.smithi099.stderr:sudo: ntpd: command not found 2023-05-31T19:42:21.515 INFO:teuthology.orchestra.run.smithi099.stdout:506 Cannot talk to daemon 2023-05-31T19:42:21.529 INFO:teuthology.orchestra.run.smithi099.stderr:Failed to start ntp.service: Unit ntp.service not found. 2023-05-31T19:42:21.535 INFO:teuthology.orchestra.run.smithi112.stderr:sudo: ntpd: command not found 2023-05-31T19:42:21.542 INFO:teuthology.orchestra.run.smithi099.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2023-05-31T19:42:21.547 INFO:teuthology.orchestra.run.smithi112.stdout:506 Cannot talk to daemon 2023-05-31T19:42:21.561 INFO:teuthology.orchestra.run.smithi112.stderr:Failed to start ntp.service: Unit ntp.service not found. 2023-05-31T19:42:21.576 INFO:teuthology.orchestra.run.smithi112.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2023-05-31T19:42:21.616 INFO:teuthology.orchestra.run.smithi099.stderr:bash: ntpq: command not found 2023-05-31T19:42:21.620 INFO:teuthology.orchestra.run.smithi099.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2023-05-31T19:42:21.620 INFO:teuthology.orchestra.run.smithi099.stdout:=============================================================================== 2023-05-31T19:42:21.620 INFO:teuthology.orchestra.run.smithi099.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2023-05-31T19:42:21.620 INFO:teuthology.orchestra.run.smithi099.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2023-05-31T19:42:21.621 INFO:teuthology.orchestra.run.smithi099.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2023-05-31T19:42:21.621 INFO:teuthology.orchestra.run.smithi099.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2023-05-31T19:42:21.653 INFO:teuthology.orchestra.run.smithi112.stderr:bash: ntpq: command not found 2023-05-31T19:42:21.657 INFO:teuthology.orchestra.run.smithi112.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2023-05-31T19:42:21.657 INFO:teuthology.orchestra.run.smithi112.stdout:=============================================================================== 2023-05-31T19:42:21.657 INFO:teuthology.orchestra.run.smithi112.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2023-05-31T19:42:21.657 INFO:teuthology.orchestra.run.smithi112.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2023-05-31T19:42:21.657 INFO:teuthology.orchestra.run.smithi112.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2023-05-31T19:42:21.658 INFO:teuthology.orchestra.run.smithi112.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2023-05-31T19:42:21.659 INFO:teuthology.run_tasks:Running task install... 2023-05-31T19:42:21.667 DEBUG:teuthology.task.install:project ceph 2023-05-31T19:42:21.667 DEBUG:teuthology.task.install:INSTALL overrides: {'ceph': {'flavor': 'default', 'sha1': '4a02f3f496d9039326c49bf1fbe140388cd2f619'}} 2023-05-31T19:42:21.667 DEBUG:teuthology.task.install:config {'flavor': 'default', 'sha1': '4a02f3f496d9039326c49bf1fbe140388cd2f619'} 2023-05-31T19:42:21.668 INFO:teuthology.task.install:Using flavor: default 2023-05-31T19:42:21.673 DEBUG:teuthology.task.install:Package list is: {'deb': ['ceph', 'cephadm', 'ceph-mds', 'ceph-mgr', 'ceph-common', 'ceph-fuse', 'ceph-test', 'ceph-volume', 'radosgw', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'libcephfs2', 'libcephfs-dev', 'librados2', 'librbd1', 'rbd-fuse'], 'rpm': ['ceph-radosgw', 'ceph-test', 'ceph', 'ceph-base', 'cephadm', 'ceph-immutable-object-cache', 'ceph-mgr', 'ceph-mgr-dashboard', 'ceph-mgr-diskprediction-local', 'ceph-mgr-rook', 'ceph-mgr-cephadm', 'ceph-fuse', 'ceph-volume', 'librados-devel', 'libcephfs2', 'libcephfs-devel', 'librados2', 'librbd1', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'rbd-fuse', 'rbd-mirror', 'rbd-nbd']} 2023-05-31T19:42:21.674 INFO:teuthology.task.install:extra packages: [] 2023-05-31T19:42:21.675 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': [], 'extra_system_packages': [], 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': '4a02f3f496d9039326c49bf1fbe140388cd2f619', 'tag': None, 'wait_for_package': False} 2023-05-31T19:42:21.676 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F8%2Fx86_64&sha1=4a02f3f496d9039326c49bf1fbe140388cd2f619 2023-05-31T19:42:21.681 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': [], 'extra_system_packages': [], 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': '4a02f3f496d9039326c49bf1fbe140388cd2f619', 'tag': None, 'wait_for_package': False} 2023-05-31T19:42:21.682 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F8%2Fx86_64&sha1=4a02f3f496d9039326c49bf1fbe140388cd2f619 2023-05-31T19:42:21.900 INFO:teuthology.task.install.rpm:Pulling from https://1.chacra.ceph.com/r/ceph/reef-release/4a02f3f496d9039326c49bf1fbe140388cd2f619/centos/8/flavors/default/ 2023-05-31T19:42:21.900 INFO:teuthology.task.install.rpm:Package version is 18.0.0-3487.g4a02f3f4 2023-05-31T19:42:21.910 INFO:teuthology.task.install.rpm:Pulling from https://1.chacra.ceph.com/r/ceph/reef-release/4a02f3f496d9039326c49bf1fbe140388cd2f619/centos/8/flavors/default/ 2023-05-31T19:42:21.910 INFO:teuthology.task.install.rpm:Package version is 18.0.0-3487.g4a02f3f4 2023-05-31T19:42:22.070 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://1.chacra.ceph.com/r/ceph/reef-release/4a02f3f496d9039326c49bf1fbe140388cd2f619/centos/8/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://1.chacra.ceph.com/r/ceph/reef-release/4a02f3f496d9039326c49bf1fbe140388cd2f619/centos/8/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://1.chacra.ceph.com/r/ceph/reef-release/4a02f3f496d9039326c49bf1fbe140388cd2f619/centos/8/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2023-05-31T19:42:22.070 DEBUG:teuthology.orchestra.run.smithi099:> set -ex 2023-05-31T19:42:22.071 DEBUG:teuthology.orchestra.run.smithi099:> sudo dd of=/etc/yum.repos.d/ceph.repo 2023-05-31T19:42:22.076 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://1.chacra.ceph.com/r/ceph/reef-release/4a02f3f496d9039326c49bf1fbe140388cd2f619/centos/8/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://1.chacra.ceph.com/r/ceph/reef-release/4a02f3f496d9039326c49bf1fbe140388cd2f619/centos/8/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://1.chacra.ceph.com/r/ceph/reef-release/4a02f3f496d9039326c49bf1fbe140388cd2f619/centos/8/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2023-05-31T19:42:22.076 DEBUG:teuthology.orchestra.run.smithi112:> set -ex 2023-05-31T19:42:22.077 DEBUG:teuthology.orchestra.run.smithi112:> sudo dd of=/etc/yum.repos.d/ceph.repo 2023-05-31T19:42:22.101 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd on remote rpm x86_64 2023-05-31T19:42:22.101 DEBUG:teuthology.orchestra.run.smithi099:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/4a02f3f496d9039326c49bf1fbe140388cd2f619/;g' /etc/yum.repos.d/ceph.repo ; fi 2023-05-31T19:42:22.105 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd on remote rpm x86_64 2023-05-31T19:42:22.105 DEBUG:teuthology.orchestra.run.smithi112:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/4a02f3f496d9039326c49bf1fbe140388cd2f619/;g' /etc/yum.repos.d/ceph.repo ; fi 2023-05-31T19:42:22.170 DEBUG:teuthology.orchestra.run.smithi099:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2023-05-31T19:42:22.173 DEBUG:teuthology.orchestra.run.smithi112:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2023-05-31T19:42:22.249 DEBUG:teuthology.orchestra.run.smithi099:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2023-05-31T19:42:22.250 DEBUG:teuthology.orchestra.run.smithi112:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2023-05-31T19:42:22.315 INFO:teuthology.orchestra.run.smithi099.stdout:check_obsoletes = 1 2023-05-31T19:42:22.316 DEBUG:teuthology.orchestra.run.smithi099:> sudo yum clean all 2023-05-31T19:42:22.317 INFO:teuthology.orchestra.run.smithi112.stdout:check_obsoletes = 1 2023-05-31T19:42:22.318 DEBUG:teuthology.orchestra.run.smithi112:> sudo yum clean all 2023-05-31T19:42:22.584 INFO:teuthology.orchestra.run.smithi099.stdout:55 files removed 2023-05-31T19:42:22.586 INFO:teuthology.orchestra.run.smithi112.stdout:55 files removed 2023-05-31T19:42:22.611 DEBUG:teuthology.orchestra.run.smithi099:> sudo yum -y install ceph-radosgw 2023-05-31T19:42:22.614 DEBUG:teuthology.orchestra.run.smithi112:> sudo yum -y install ceph-radosgw 2023-05-31T19:42:23.360 INFO:teuthology.orchestra.run.smithi099.stdout:ceph packages for x86_64 229 kB/s | 80 kB 00:00 2023-05-31T19:42:23.476 INFO:teuthology.orchestra.run.smithi112.stdout:ceph packages for x86_64 173 kB/s | 80 kB 00:00 2023-05-31T19:42:23.653 INFO:teuthology.orchestra.run.smithi099.stdout:ceph noarch packages 37 kB/s | 9.4 kB 00:00 2023-05-31T19:42:23.777 INFO:teuthology.orchestra.run.smithi112.stdout:ceph noarch packages 35 kB/s | 9.4 kB 00:00 2023-05-31T19:42:24.043 INFO:teuthology.orchestra.run.smithi099.stdout:ceph source packages 4.1 kB/s | 1.5 kB 00:00 2023-05-31T19:42:24.069 INFO:teuthology.orchestra.run.smithi112.stdout:ceph source packages 5.6 kB/s | 1.5 kB 00:00 2023-05-31T19:42:25.636 INFO:teuthology.orchestra.run.smithi112.stdout:CentOS Stream 8 - AppStream 19 MB/s | 29 MB 00:01 2023-05-31T19:42:25.646 INFO:teuthology.orchestra.run.smithi099.stdout:CentOS Stream 8 - AppStream 18 MB/s | 29 MB 00:01 2023-05-31T19:42:32.586 INFO:teuthology.orchestra.run.smithi112.stdout:CentOS Stream 8 - BaseOS 20 MB/s | 36 MB 00:01 2023-05-31T19:42:32.670 INFO:teuthology.orchestra.run.smithi099.stdout:CentOS Stream 8 - BaseOS 20 MB/s | 36 MB 00:01 2023-05-31T19:42:38.170 INFO:teuthology.orchestra.run.smithi112.stdout:CentOS Stream 8 - Extras 53 kB/s | 18 kB 00:00 2023-05-31T19:42:38.279 INFO:teuthology.orchestra.run.smithi099.stdout:CentOS Stream 8 - Extras 50 kB/s | 18 kB 00:00 2023-05-31T19:42:38.463 INFO:teuthology.orchestra.run.smithi112.stdout:CentOS Stream 8 - Extras common packages 49 kB/s | 6.6 kB 00:00 2023-05-31T19:42:38.571 INFO:teuthology.orchestra.run.smithi099.stdout:CentOS Stream 8 - Extras common packages 49 kB/s | 6.6 kB 00:00 2023-05-31T19:42:41.073 INFO:teuthology.orchestra.run.smithi112.stdout:Copr repo for python3-asyncssh owned by ceph 1.4 kB/s | 3.5 kB 00:02 2023-05-31T19:42:41.205 INFO:teuthology.orchestra.run.smithi099.stdout:Copr repo for python3-asyncssh owned by ceph 1.4 kB/s | 3.5 kB 00:02 2023-05-31T19:42:43.098 INFO:teuthology.orchestra.run.smithi112.stdout:Extra Packages for Enterprise Linux 7.4 MB/s | 14 MB 00:01 2023-05-31T19:42:46.507 INFO:teuthology.orchestra.run.smithi112.stdout:lab-extras 324 kB/s | 24 kB 00:00 2023-05-31T19:42:49.987 INFO:teuthology.orchestra.run.smithi112.stdout:Dependencies resolved. 2023-05-31T19:42:49.990 INFO:teuthology.orchestra.run.smithi112.stdout:================================================================================ 2023-05-31T19:42:49.990 INFO:teuthology.orchestra.run.smithi112.stdout: Package Arch Version Repository Size 2023-05-31T19:42:49.990 INFO:teuthology.orchestra.run.smithi112.stdout:================================================================================ 2023-05-31T19:42:49.991 INFO:teuthology.orchestra.run.smithi112.stdout:Installing: 2023-05-31T19:42:49.991 INFO:teuthology.orchestra.run.smithi112.stdout: ceph-radosgw x86_64 2:18.0.0-3487.g4a02f3f4.el8 ceph 13 M 2023-05-31T19:42:49.991 INFO:teuthology.orchestra.run.smithi112.stdout:Upgrading: 2023-05-31T19:42:49.991 INFO:teuthology.orchestra.run.smithi112.stdout: librados2 x86_64 2:18.0.0-3487.g4a02f3f4.el8 ceph 3.7 M 2023-05-31T19:42:49.991 INFO:teuthology.orchestra.run.smithi112.stdout: librbd1 x86_64 2:18.0.0-3487.g4a02f3f4.el8 ceph 3.6 M 2023-05-31T19:42:49.991 INFO:teuthology.orchestra.run.smithi112.stdout:Installing dependencies: 2023-05-31T19:42:49.991 INFO:teuthology.orchestra.run.smithi112.stdout: ceph-base x86_64 2:18.0.0-3487.g4a02f3f4.el8 ceph 5.4 M 2023-05-31T19:42:49.992 INFO:teuthology.orchestra.run.smithi112.stdout: ceph-common x86_64 2:18.0.0-3487.g4a02f3f4.el8 ceph 22 M 2023-05-31T19:42:49.992 INFO:teuthology.orchestra.run.smithi112.stdout: ceph-selinux x86_64 2:18.0.0-3487.g4a02f3f4.el8 ceph 24 k 2023-05-31T19:42:49.992 INFO:teuthology.orchestra.run.smithi112.stdout: gperftools-libs x86_64 1:2.7-9.el8 epel 306 k 2023-05-31T19:42:49.992 INFO:teuthology.orchestra.run.smithi112.stdout: libcephfs2 x86_64 2:18.0.0-3487.g4a02f3f4.el8 ceph 767 k 2023-05-31T19:42:49.992 INFO:teuthology.orchestra.run.smithi112.stdout: liboath x86_64 2.6.2-3.el8 epel 59 k 2023-05-31T19:42:49.992 INFO:teuthology.orchestra.run.smithi112.stdout: librabbitmq x86_64 0.9.0-3.el8 baseos 47 k 2023-05-31T19:42:49.992 INFO:teuthology.orchestra.run.smithi112.stdout: libradosstriper1 x86_64 2:18.0.0-3487.g4a02f3f4.el8 ceph 474 k 2023-05-31T19:42:49.993 INFO:teuthology.orchestra.run.smithi112.stdout: librdkafka x86_64 0.11.4-3.el8 appstream 354 k 2023-05-31T19:42:49.993 INFO:teuthology.orchestra.run.smithi112.stdout: librgw2 x86_64 2:18.0.0-3487.g4a02f3f4.el8 ceph 6.5 M 2023-05-31T19:42:49.993 INFO:teuthology.orchestra.run.smithi112.stdout: libunwind x86_64 1.3.1-3.el8 epel 75 k 2023-05-31T19:42:49.993 INFO:teuthology.orchestra.run.smithi112.stdout: lttng-ust x86_64 2.8.1-11.el8 appstream 259 k 2023-05-31T19:42:49.993 INFO:teuthology.orchestra.run.smithi112.stdout: python3-ceph-argparse x86_64 2:18.0.0-3487.g4a02f3f4.el8 ceph 45 k 2023-05-31T19:42:49.993 INFO:teuthology.orchestra.run.smithi112.stdout: python3-ceph-common x86_64 2:18.0.0-3487.g4a02f3f4.el8 ceph 119 k 2023-05-31T19:42:49.993 INFO:teuthology.orchestra.run.smithi112.stdout: python3-cephfs x86_64 2:18.0.0-3487.g4a02f3f4.el8 ceph 190 k 2023-05-31T19:42:49.994 INFO:teuthology.orchestra.run.smithi112.stdout: python3-prettytable noarch 0.7.2-14.el8 appstream 44 k 2023-05-31T19:42:49.994 INFO:teuthology.orchestra.run.smithi112.stdout: python3-rados x86_64 2:18.0.0-3487.g4a02f3f4.el8 ceph 371 k 2023-05-31T19:42:49.994 INFO:teuthology.orchestra.run.smithi112.stdout: python3-rbd x86_64 2:18.0.0-3487.g4a02f3f4.el8 ceph 365 k 2023-05-31T19:42:49.994 INFO:teuthology.orchestra.run.smithi112.stdout: python3-rgw x86_64 2:18.0.0-3487.g4a02f3f4.el8 ceph 106 k 2023-05-31T19:42:49.994 INFO:teuthology.orchestra.run.smithi112.stdout: thrift x86_64 0.13.0-2.el8 epel 1.7 M 2023-05-31T19:42:49.994 INFO:teuthology.orchestra.run.smithi112.stdout: 2023-05-31T19:42:49.994 INFO:teuthology.orchestra.run.smithi112.stdout:Transaction Summary 2023-05-31T19:42:49.994 INFO:teuthology.orchestra.run.smithi112.stdout:================================================================================ 2023-05-31T19:42:49.995 INFO:teuthology.orchestra.run.smithi112.stdout:Install 21 Packages 2023-05-31T19:42:49.995 INFO:teuthology.orchestra.run.smithi112.stdout:Upgrade 2 Packages 2023-05-31T19:42:49.995 INFO:teuthology.orchestra.run.smithi112.stdout: 2023-05-31T19:42:49.995 INFO:teuthology.orchestra.run.smithi112.stdout:Total download size: 59 M 2023-05-31T19:42:49.996 INFO:teuthology.orchestra.run.smithi112.stdout:Downloading Packages: 2023-05-31T19:42:51.450 INFO:teuthology.orchestra.run.smithi112.stdout:(1/23): ceph-base-18.0.0-3487.g4a02f3f4.el8.x86 5.1 MB/s | 5.4 MB 00:01 2023-05-31T19:42:51.999 INFO:teuthology.orchestra.run.smithi112.stdout:(2/23): ceph-selinux-18.0.0-3487.g4a02f3f4.el8. 44 kB/s | 24 kB 00:00 2023-05-31T19:42:52.150 INFO:teuthology.orchestra.run.smithi112.stdout:(3/23): libcephfs2-18.0.0-3487.g4a02f3f4.el8.x8 4.9 MB/s | 767 kB 00:00 2023-05-31T19:42:52.309 INFO:teuthology.orchestra.run.smithi112.stdout:(4/23): libradosstriper1-18.0.0-3487.g4a02f3f4. 2.9 MB/s | 474 kB 00:00 2023-05-31T19:42:52.667 INFO:teuthology.orchestra.run.smithi112.stdout:(5/23): librgw2-18.0.0-3487.g4a02f3f4.el8.x86_6 18 MB/s | 6.5 MB 00:00 2023-05-31T19:42:52.709 INFO:teuthology.orchestra.run.smithi112.stdout:(6/23): python3-ceph-argparse-18.0.0-3487.g4a02 1.1 MB/s | 45 kB 00:00 2023-05-31T19:42:52.752 INFO:teuthology.orchestra.run.smithi112.stdout:(7/23): python3-ceph-common-18.0.0-3487.g4a02f3 2.8 MB/s | 119 kB 00:00 2023-05-31T19:42:52.802 INFO:teuthology.orchestra.run.smithi112.stdout:(8/23): python3-cephfs-18.0.0-3487.g4a02f3f4.el 3.7 MB/s | 190 kB 00:00 2023-05-31T19:42:52.893 INFO:teuthology.orchestra.run.smithi112.stdout:(9/23): python3-rados-18.0.0-3487.g4a02f3f4.el8 4.0 MB/s | 371 kB 00:00 2023-05-31T19:42:52.994 INFO:teuthology.orchestra.run.smithi112.stdout:(10/23): python3-rbd-18.0.0-3487.g4a02f3f4.el8. 3.6 MB/s | 365 kB 00:00 2023-05-31T19:42:53.061 INFO:teuthology.orchestra.run.smithi112.stdout:(11/23): python3-rgw-18.0.0-3487.g4a02f3f4.el8. 1.6 MB/s | 106 kB 00:00 2023-05-31T19:42:53.486 INFO:teuthology.orchestra.run.smithi112.stdout:(12/23): librdkafka-0.11.4-3.el8.x86_64.rpm 833 kB/s | 354 kB 00:00 2023-05-31T19:42:53.587 INFO:teuthology.orchestra.run.smithi112.stdout:(13/23): lttng-ust-2.8.1-11.el8.x86_64.rpm 2.5 MB/s | 259 kB 00:00 2023-05-31T19:42:53.628 INFO:teuthology.orchestra.run.smithi112.stdout:(14/23): python3-prettytable-0.7.2-14.el8.noarc 1.0 MB/s | 44 kB 00:00 2023-05-31T19:42:53.803 INFO:teuthology.orchestra.run.smithi112.stdout:(15/23): librabbitmq-0.9.0-3.el8.x86_64.rpm 268 kB/s | 47 kB 00:00 2023-05-31T19:42:53.996 INFO:teuthology.orchestra.run.smithi112.stdout:(16/23): gperftools-libs-2.7-9.el8.x86_64.rpm 1.6 MB/s | 306 kB 00:00 2023-05-31T19:42:54.071 INFO:teuthology.orchestra.run.smithi112.stdout:(17/23): liboath-2.6.2-3.el8.x86_64.rpm 788 kB/s | 59 kB 00:00 2023-05-31T19:42:54.155 INFO:teuthology.orchestra.run.smithi112.stdout:(18/23): libunwind-1.3.1-3.el8.x86_64.rpm 897 kB/s | 75 kB 00:00 2023-05-31T19:42:54.838 INFO:teuthology.orchestra.run.smithi112.stdout:(19/23): thrift-0.13.0-2.el8.x86_64.rpm 2.5 MB/s | 1.7 MB 00:00 2023-05-31T19:42:55.225 INFO:teuthology.orchestra.run.smithi112.stdout:(20/23): librados2-18.0.0-3487.g4a02f3f4.el8.x8 9.5 MB/s | 3.7 MB 00:00 2023-05-31T19:42:55.439 INFO:teuthology.orchestra.run.smithi112.stdout:(21/23): librbd1-18.0.0-3487.g4a02f3f4.el8.x86_ 17 MB/s | 3.6 MB 00:00 2023-05-31T19:42:55.868 INFO:teuthology.orchestra.run.smithi112.stdout:(22/23): ceph-common-18.0.0-3487.g4a02f3f4.el8. 3.9 MB/s | 22 MB 00:05 2023-05-31T19:42:57.290 INFO:teuthology.orchestra.run.smithi099.stdout:Extra Packages for Enterprise Linux 893 kB/s | 14 MB 00:15 2023-05-31T19:42:57.389 INFO:teuthology.orchestra.run.smithi112.stdout:(23/23): ceph-radosgw-18.0.0-3487.g4a02f3f4.el8 1.9 MB/s | 13 MB 00:06 2023-05-31T19:42:57.393 INFO:teuthology.orchestra.run.smithi112.stdout:-------------------------------------------------------------------------------- 2023-05-31T19:42:57.393 INFO:teuthology.orchestra.run.smithi112.stdout:Total 8.0 MB/s | 59 MB 00:07 2023-05-31T19:42:57.434 INFO:teuthology.orchestra.run.smithi112.stdout:Running transaction check 2023-05-31T19:42:57.480 INFO:teuthology.orchestra.run.smithi112.stdout:Transaction check succeeded. 2023-05-31T19:42:57.480 INFO:teuthology.orchestra.run.smithi112.stdout:Running transaction test 2023-05-31T19:42:58.085 INFO:teuthology.orchestra.run.smithi112.stdout:Transaction test succeeded. 2023-05-31T19:42:58.088 INFO:teuthology.orchestra.run.smithi112.stdout:Running transaction 2023-05-31T19:42:59.310 INFO:teuthology.orchestra.run.smithi112.stdout: Preparing : 1/1 2023-05-31T19:42:59.540 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : thrift-0.13.0-2.el8.x86_64 1/25 2023-05-31T19:42:59.771 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : liboath-2.6.2-3.el8.x86_64 2/25 2023-05-31T19:42:59.967 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : lttng-ust-2.8.1-11.el8.x86_64 3/25 2023-05-31T19:43:00.444 INFO:teuthology.orchestra.run.smithi112.stdout: Running scriptlet: lttng-ust-2.8.1-11.el8.x86_64 3/25 2023-05-31T19:43:00.712 INFO:teuthology.orchestra.run.smithi099.stdout:lab-extras 443 kB/s | 24 kB 00:00 2023-05-31T19:43:00.759 INFO:teuthology.orchestra.run.smithi112.stdout: Upgrading : librados2-2:18.0.0-3487.g4a02f3f4.el8.x86_64 4/25 2023-05-31T19:43:00.857 INFO:teuthology.orchestra.run.smithi112.stdout: Running scriptlet: librados2-2:18.0.0-3487.g4a02f3f4.el8.x86_64 4/25 2023-05-31T19:43:01.165 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : python3-rados-2:18.0.0-3487.g4a02f3f4.el8.x86_64 5/25 2023-05-31T19:43:01.392 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : libcephfs2-2:18.0.0-3487.g4a02f3f4.el8.x86_64 6/25 2023-05-31T19:43:01.774 INFO:teuthology.orchestra.run.smithi112.stdout: Running scriptlet: libcephfs2-2:18.0.0-3487.g4a02f3f4.el8.x86_64 6/25 2023-05-31T19:43:02.008 INFO:teuthology.orchestra.run.smithi112.stdout: Upgrading : librbd1-2:18.0.0-3487.g4a02f3f4.el8.x86_64 7/25 2023-05-31T19:43:02.071 INFO:teuthology.orchestra.run.smithi112.stdout: Running scriptlet: librbd1-2:18.0.0-3487.g4a02f3f4.el8.x86_64 7/25 2023-05-31T19:43:02.306 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : librabbitmq-0.9.0-3.el8.x86_64 8/25 2023-05-31T19:43:02.508 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : librdkafka-0.11.4-3.el8.x86_64 9/25 2023-05-31T19:43:02.575 INFO:teuthology.orchestra.run.smithi112.stdout: Running scriptlet: librdkafka-0.11.4-3.el8.x86_64 9/25 2023-05-31T19:43:02.744 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : python3-ceph-argparse-2:18.0.0-3487.g4a02f3f4.el8. 10/25 2023-05-31T19:43:02.999 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : python3-cephfs-2:18.0.0-3487.g4a02f3f4.el8.x86_64 11/25 2023-05-31T19:43:03.259 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : python3-rbd-2:18.0.0-3487.g4a02f3f4.el8.x86_64 12/25 2023-05-31T19:43:03.450 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : libradosstriper1-2:18.0.0-3487.g4a02f3f4.el8.x86_6 13/25 2023-05-31T19:43:03.520 INFO:teuthology.orchestra.run.smithi112.stdout: Running scriptlet: libradosstriper1-2:18.0.0-3487.g4a02f3f4.el8.x86_6 13/25 2023-05-31T19:43:03.785 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : libunwind-1.3.1-3.el8.x86_64 14/25 2023-05-31T19:43:04.149 INFO:teuthology.orchestra.run.smithi099.stdout:Dependencies resolved. 2023-05-31T19:43:04.151 INFO:teuthology.orchestra.run.smithi099.stdout:================================================================================ 2023-05-31T19:43:04.152 INFO:teuthology.orchestra.run.smithi099.stdout: Package Arch Version Repository Size 2023-05-31T19:43:04.152 INFO:teuthology.orchestra.run.smithi099.stdout:================================================================================ 2023-05-31T19:43:04.152 INFO:teuthology.orchestra.run.smithi099.stdout:Installing: 2023-05-31T19:43:04.152 INFO:teuthology.orchestra.run.smithi099.stdout: ceph-radosgw x86_64 2:18.0.0-3487.g4a02f3f4.el8 ceph 13 M 2023-05-31T19:43:04.152 INFO:teuthology.orchestra.run.smithi099.stdout:Upgrading: 2023-05-31T19:43:04.152 INFO:teuthology.orchestra.run.smithi099.stdout: librados2 x86_64 2:18.0.0-3487.g4a02f3f4.el8 ceph 3.7 M 2023-05-31T19:43:04.152 INFO:teuthology.orchestra.run.smithi099.stdout: librbd1 x86_64 2:18.0.0-3487.g4a02f3f4.el8 ceph 3.6 M 2023-05-31T19:43:04.153 INFO:teuthology.orchestra.run.smithi099.stdout:Installing dependencies: 2023-05-31T19:43:04.153 INFO:teuthology.orchestra.run.smithi099.stdout: ceph-base x86_64 2:18.0.0-3487.g4a02f3f4.el8 ceph 5.4 M 2023-05-31T19:43:04.153 INFO:teuthology.orchestra.run.smithi099.stdout: ceph-common x86_64 2:18.0.0-3487.g4a02f3f4.el8 ceph 22 M 2023-05-31T19:43:04.153 INFO:teuthology.orchestra.run.smithi099.stdout: ceph-selinux x86_64 2:18.0.0-3487.g4a02f3f4.el8 ceph 24 k 2023-05-31T19:43:04.153 INFO:teuthology.orchestra.run.smithi099.stdout: gperftools-libs x86_64 1:2.7-9.el8 epel 306 k 2023-05-31T19:43:04.153 INFO:teuthology.orchestra.run.smithi099.stdout: libcephfs2 x86_64 2:18.0.0-3487.g4a02f3f4.el8 ceph 767 k 2023-05-31T19:43:04.153 INFO:teuthology.orchestra.run.smithi099.stdout: liboath x86_64 2.6.2-3.el8 epel 59 k 2023-05-31T19:43:04.154 INFO:teuthology.orchestra.run.smithi099.stdout: librabbitmq x86_64 0.9.0-3.el8 baseos 47 k 2023-05-31T19:43:04.154 INFO:teuthology.orchestra.run.smithi099.stdout: libradosstriper1 x86_64 2:18.0.0-3487.g4a02f3f4.el8 ceph 474 k 2023-05-31T19:43:04.154 INFO:teuthology.orchestra.run.smithi099.stdout: librdkafka x86_64 0.11.4-3.el8 appstream 354 k 2023-05-31T19:43:04.154 INFO:teuthology.orchestra.run.smithi099.stdout: librgw2 x86_64 2:18.0.0-3487.g4a02f3f4.el8 ceph 6.5 M 2023-05-31T19:43:04.154 INFO:teuthology.orchestra.run.smithi099.stdout: libunwind x86_64 1.3.1-3.el8 epel 75 k 2023-05-31T19:43:04.154 INFO:teuthology.orchestra.run.smithi099.stdout: lttng-ust x86_64 2.8.1-11.el8 appstream 259 k 2023-05-31T19:43:04.154 INFO:teuthology.orchestra.run.smithi099.stdout: python3-ceph-argparse x86_64 2:18.0.0-3487.g4a02f3f4.el8 ceph 45 k 2023-05-31T19:43:04.155 INFO:teuthology.orchestra.run.smithi099.stdout: python3-ceph-common x86_64 2:18.0.0-3487.g4a02f3f4.el8 ceph 119 k 2023-05-31T19:43:04.155 INFO:teuthology.orchestra.run.smithi099.stdout: python3-cephfs x86_64 2:18.0.0-3487.g4a02f3f4.el8 ceph 190 k 2023-05-31T19:43:04.155 INFO:teuthology.orchestra.run.smithi099.stdout: python3-prettytable noarch 0.7.2-14.el8 appstream 44 k 2023-05-31T19:43:04.155 INFO:teuthology.orchestra.run.smithi099.stdout: python3-rados x86_64 2:18.0.0-3487.g4a02f3f4.el8 ceph 371 k 2023-05-31T19:43:04.155 INFO:teuthology.orchestra.run.smithi099.stdout: python3-rbd x86_64 2:18.0.0-3487.g4a02f3f4.el8 ceph 365 k 2023-05-31T19:43:04.155 INFO:teuthology.orchestra.run.smithi099.stdout: python3-rgw x86_64 2:18.0.0-3487.g4a02f3f4.el8 ceph 106 k 2023-05-31T19:43:04.155 INFO:teuthology.orchestra.run.smithi099.stdout: thrift x86_64 0.13.0-2.el8 epel 1.7 M 2023-05-31T19:43:04.155 INFO:teuthology.orchestra.run.smithi099.stdout: 2023-05-31T19:43:04.156 INFO:teuthology.orchestra.run.smithi099.stdout:Transaction Summary 2023-05-31T19:43:04.156 INFO:teuthology.orchestra.run.smithi099.stdout:================================================================================ 2023-05-31T19:43:04.156 INFO:teuthology.orchestra.run.smithi099.stdout:Install 21 Packages 2023-05-31T19:43:04.156 INFO:teuthology.orchestra.run.smithi099.stdout:Upgrade 2 Packages 2023-05-31T19:43:04.156 INFO:teuthology.orchestra.run.smithi099.stdout: 2023-05-31T19:43:04.157 INFO:teuthology.orchestra.run.smithi099.stdout:Total download size: 59 M 2023-05-31T19:43:04.157 INFO:teuthology.orchestra.run.smithi099.stdout:Downloading Packages: 2023-05-31T19:43:04.597 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : gperftools-libs-1:2.7-9.el8.x86_64 15/25 2023-05-31T19:43:04.800 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : librgw2-2:18.0.0-3487.g4a02f3f4.el8.x86_64 16/25 2023-05-31T19:43:04.878 INFO:teuthology.orchestra.run.smithi112.stdout: Running scriptlet: librgw2-2:18.0.0-3487.g4a02f3f4.el8.x86_64 16/25 2023-05-31T19:43:05.116 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : python3-rgw-2:18.0.0-3487.g4a02f3f4.el8.x86_64 17/25 2023-05-31T19:43:05.263 INFO:teuthology.orchestra.run.smithi099.stdout:(1/23): ceph-base-18.0.0-3487.g4a02f3f4.el8.x86 6.5 MB/s | 5.4 MB 00:00 2023-05-31T19:43:05.313 INFO:teuthology.orchestra.run.smithi099.stdout:(2/23): ceph-selinux-18.0.0-3487.g4a02f3f4.el8. 489 kB/s | 24 kB 00:00 2023-05-31T19:43:05.336 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : python3-prettytable-0.7.2-14.el8.noarch 18/25 2023-05-31T19:43:05.438 INFO:teuthology.orchestra.run.smithi099.stdout:(3/23): libcephfs2-18.0.0-3487.g4a02f3f4.el8.x8 6.0 MB/s | 767 kB 00:00 2023-05-31T19:43:05.547 INFO:teuthology.orchestra.run.smithi099.stdout:(4/23): libradosstriper1-18.0.0-3487.g4a02f3f4. 4.3 MB/s | 474 kB 00:00 2023-05-31T19:43:05.575 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : python3-ceph-common-2:18.0.0-3487.g4a02f3f4.el8.x8 19/25 2023-05-31T19:43:05.972 INFO:teuthology.orchestra.run.smithi099.stdout:(5/23): ceph-radosgw-18.0.0-3487.g4a02f3f4.el8. 8.5 MB/s | 13 MB 00:01 2023-05-31T19:43:06.023 INFO:teuthology.orchestra.run.smithi099.stdout:(6/23): python3-ceph-argparse-18.0.0-3487.g4a02 900 kB/s | 45 kB 00:00 2023-05-31T19:43:06.090 INFO:teuthology.orchestra.run.smithi099.stdout:(7/23): python3-ceph-common-18.0.0-3487.g4a02f3 1.7 MB/s | 119 kB 00:00 2023-05-31T19:43:06.156 INFO:teuthology.orchestra.run.smithi099.stdout:(8/23): python3-cephfs-18.0.0-3487.g4a02f3f4.el 2.8 MB/s | 190 kB 00:00 2023-05-31T19:43:06.249 INFO:teuthology.orchestra.run.smithi099.stdout:(9/23): librgw2-18.0.0-3487.g4a02f3f4.el8.x86_6 9.3 MB/s | 6.5 MB 00:00 2023-05-31T19:43:06.274 INFO:teuthology.orchestra.run.smithi099.stdout:(10/23): python3-rados-18.0.0-3487.g4a02f3f4.el 3.1 MB/s | 371 kB 00:00 2023-05-31T19:43:06.300 INFO:teuthology.orchestra.run.smithi099.stdout:(11/23): python3-rbd-18.0.0-3487.g4a02f3f4.el8. 7.2 MB/s | 365 kB 00:00 2023-05-31T19:43:06.324 INFO:teuthology.orchestra.run.smithi099.stdout:(12/23): python3-rgw-18.0.0-3487.g4a02f3f4.el8. 2.1 MB/s | 106 kB 00:00 2023-05-31T19:43:06.467 INFO:teuthology.orchestra.run.smithi099.stdout:(13/23): lttng-ust-2.8.1-11.el8.x86_64.rpm 1.8 MB/s | 259 kB 00:00 2023-05-31T19:43:06.492 INFO:teuthology.orchestra.run.smithi099.stdout:(14/23): librdkafka-0.11.4-3.el8.x86_64.rpm 1.8 MB/s | 354 kB 00:00 2023-05-31T19:43:06.517 INFO:teuthology.orchestra.run.smithi099.stdout:(15/23): python3-prettytable-0.7.2-14.el8.noarc 876 kB/s | 44 kB 00:00 2023-05-31T19:43:06.701 INFO:teuthology.orchestra.run.smithi099.stdout:(16/23): librabbitmq-0.9.0-3.el8.x86_64.rpm 225 kB/s | 47 kB 00:00 2023-05-31T19:43:07.326 INFO:teuthology.orchestra.run.smithi099.stdout:(17/23): ceph-common-18.0.0-3487.g4a02f3f4.el8. 7.5 MB/s | 22 MB 00:02 2023-05-31T19:43:07.355 INFO:teuthology.orchestra.run.smithi099.stdout:(18/23): liboath-2.6.2-3.el8.x86_64.rpm 90 kB/s | 59 kB 00:00 2023-05-31T19:43:07.476 INFO:teuthology.orchestra.run.smithi099.stdout:(19/23): gperftools-libs-2.7-9.el8.x86_64.rpm 319 kB/s | 306 kB 00:00 2023-05-31T19:43:07.785 INFO:teuthology.orchestra.run.smithi099.stdout:(20/23): librados2-18.0.0-3487.g4a02f3f4.el8.x8 12 MB/s | 3.7 MB 00:00 2023-05-31T19:43:07.977 INFO:teuthology.orchestra.run.smithi099.stdout:(21/23): librbd1-18.0.0-3487.g4a02f3f4.el8.x86_ 19 MB/s | 3.6 MB 00:00 2023-05-31T19:43:08.002 INFO:teuthology.orchestra.run.smithi099.stdout:(22/23): libunwind-1.3.1-3.el8.x86_64.rpm 111 kB/s | 75 kB 00:00 2023-05-31T19:43:08.177 INFO:teuthology.orchestra.run.smithi112.stdout: Running scriptlet: ceph-common-2:18.0.0-3487.g4a02f3f4.el8.x86_64 20/25 2023-05-31T19:43:08.392 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : ceph-common-2:18.0.0-3487.g4a02f3f4.el8.x86_64 20/25 2023-05-31T19:43:08.988 INFO:teuthology.orchestra.run.smithi112.stdout: Running scriptlet: ceph-common-2:18.0.0-3487.g4a02f3f4.el8.x86_64 20/25 2023-05-31T19:43:09.280 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : ceph-base-2:18.0.0-3487.g4a02f3f4.el8.x86_64 21/25 2023-05-31T19:43:09.405 INFO:teuthology.orchestra.run.smithi112.stdout: Running scriptlet: ceph-base-2:18.0.0-3487.g4a02f3f4.el8.x86_64 21/25 2023-05-31T19:43:09.566 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : ceph-selinux-2:18.0.0-3487.g4a02f3f4.el8.x86_64 22/25 2023-05-31T19:43:10.044 INFO:teuthology.orchestra.run.smithi099.stdout:(23/23): thrift-0.13.0-2.el8.x86_64.rpm 660 kB/s | 1.7 MB 00:02 2023-05-31T19:43:10.050 INFO:teuthology.orchestra.run.smithi099.stdout:-------------------------------------------------------------------------------- 2023-05-31T19:43:10.050 INFO:teuthology.orchestra.run.smithi099.stdout:Total 10 MB/s | 59 MB 00:05 2023-05-31T19:43:10.090 INFO:teuthology.orchestra.run.smithi099.stdout:Running transaction check 2023-05-31T19:43:10.137 INFO:teuthology.orchestra.run.smithi099.stdout:Transaction check succeeded. 2023-05-31T19:43:10.137 INFO:teuthology.orchestra.run.smithi099.stdout:Running transaction test 2023-05-31T19:43:10.711 INFO:teuthology.orchestra.run.smithi099.stdout:Transaction test succeeded. 2023-05-31T19:43:10.714 INFO:teuthology.orchestra.run.smithi099.stdout:Running transaction 2023-05-31T19:43:11.897 INFO:teuthology.orchestra.run.smithi099.stdout: Preparing : 1/1 2023-05-31T19:43:12.130 INFO:teuthology.orchestra.run.smithi099.stdout: Installing : thrift-0.13.0-2.el8.x86_64 1/25 2023-05-31T19:43:12.375 INFO:teuthology.orchestra.run.smithi099.stdout: Installing : liboath-2.6.2-3.el8.x86_64 2/25 2023-05-31T19:43:12.598 INFO:teuthology.orchestra.run.smithi099.stdout: Installing : lttng-ust-2.8.1-11.el8.x86_64 3/25 2023-05-31T19:43:13.004 INFO:teuthology.orchestra.run.smithi099.stdout: Running scriptlet: lttng-ust-2.8.1-11.el8.x86_64 3/25 2023-05-31T19:43:13.215 INFO:teuthology.orchestra.run.smithi099.stdout: Upgrading : librados2-2:18.0.0-3487.g4a02f3f4.el8.x86_64 4/25 2023-05-31T19:43:13.312 INFO:teuthology.orchestra.run.smithi099.stdout: Running scriptlet: librados2-2:18.0.0-3487.g4a02f3f4.el8.x86_64 4/25 2023-05-31T19:43:13.562 INFO:teuthology.orchestra.run.smithi099.stdout: Installing : python3-rados-2:18.0.0-3487.g4a02f3f4.el8.x86_64 5/25 2023-05-31T19:43:13.748 INFO:teuthology.orchestra.run.smithi099.stdout: Installing : libcephfs2-2:18.0.0-3487.g4a02f3f4.el8.x86_64 6/25 2023-05-31T19:43:14.129 INFO:teuthology.orchestra.run.smithi099.stdout: Running scriptlet: libcephfs2-2:18.0.0-3487.g4a02f3f4.el8.x86_64 6/25 2023-05-31T19:43:14.340 INFO:teuthology.orchestra.run.smithi099.stdout: Upgrading : librbd1-2:18.0.0-3487.g4a02f3f4.el8.x86_64 7/25 2023-05-31T19:43:14.410 INFO:teuthology.orchestra.run.smithi099.stdout: Running scriptlet: librbd1-2:18.0.0-3487.g4a02f3f4.el8.x86_64 7/25 2023-05-31T19:43:14.646 INFO:teuthology.orchestra.run.smithi099.stdout: Installing : librabbitmq-0.9.0-3.el8.x86_64 8/25 2023-05-31T19:43:14.839 INFO:teuthology.orchestra.run.smithi099.stdout: Installing : librdkafka-0.11.4-3.el8.x86_64 9/25 2023-05-31T19:43:14.905 INFO:teuthology.orchestra.run.smithi099.stdout: Running scriptlet: librdkafka-0.11.4-3.el8.x86_64 9/25 2023-05-31T19:43:15.100 INFO:teuthology.orchestra.run.smithi099.stdout: Installing : python3-ceph-argparse-2:18.0.0-3487.g4a02f3f4.el8. 10/25 2023-05-31T19:43:15.335 INFO:teuthology.orchestra.run.smithi099.stdout: Installing : python3-cephfs-2:18.0.0-3487.g4a02f3f4.el8.x86_64 11/25 2023-05-31T19:43:15.723 INFO:teuthology.orchestra.run.smithi099.stdout: Installing : python3-rbd-2:18.0.0-3487.g4a02f3f4.el8.x86_64 12/25 2023-05-31T19:43:15.881 INFO:teuthology.orchestra.run.smithi099.stdout: Installing : libradosstriper1-2:18.0.0-3487.g4a02f3f4.el8.x86_6 13/25 2023-05-31T19:43:15.953 INFO:teuthology.orchestra.run.smithi099.stdout: Running scriptlet: libradosstriper1-2:18.0.0-3487.g4a02f3f4.el8.x86_6 13/25 2023-05-31T19:43:16.223 INFO:teuthology.orchestra.run.smithi099.stdout: Installing : libunwind-1.3.1-3.el8.x86_64 14/25 2023-05-31T19:43:17.010 INFO:teuthology.orchestra.run.smithi099.stdout: Installing : gperftools-libs-1:2.7-9.el8.x86_64 15/25 2023-05-31T19:43:17.197 INFO:teuthology.orchestra.run.smithi099.stdout: Installing : librgw2-2:18.0.0-3487.g4a02f3f4.el8.x86_64 16/25 2023-05-31T19:43:17.276 INFO:teuthology.orchestra.run.smithi099.stdout: Running scriptlet: librgw2-2:18.0.0-3487.g4a02f3f4.el8.x86_64 16/25 2023-05-31T19:43:17.463 INFO:teuthology.orchestra.run.smithi099.stdout: Installing : python3-rgw-2:18.0.0-3487.g4a02f3f4.el8.x86_64 17/25 2023-05-31T19:43:17.680 INFO:teuthology.orchestra.run.smithi099.stdout: Installing : python3-prettytable-0.7.2-14.el8.noarch 18/25 2023-05-31T19:43:17.881 INFO:teuthology.orchestra.run.smithi099.stdout: Installing : python3-ceph-common-2:18.0.0-3487.g4a02f3f4.el8.x8 19/25 2023-05-31T19:43:20.406 INFO:teuthology.orchestra.run.smithi099.stdout: Running scriptlet: ceph-common-2:18.0.0-3487.g4a02f3f4.el8.x86_64 20/25 2023-05-31T19:43:20.647 INFO:teuthology.orchestra.run.smithi099.stdout: Installing : ceph-common-2:18.0.0-3487.g4a02f3f4.el8.x86_64 20/25 2023-05-31T19:43:21.246 INFO:teuthology.orchestra.run.smithi099.stdout: Running scriptlet: ceph-common-2:18.0.0-3487.g4a02f3f4.el8.x86_64 20/25 2023-05-31T19:43:21.519 INFO:teuthology.orchestra.run.smithi099.stdout: Installing : ceph-base-2:18.0.0-3487.g4a02f3f4.el8.x86_64 21/25 2023-05-31T19:43:21.595 INFO:teuthology.orchestra.run.smithi099.stdout: Running scriptlet: ceph-base-2:18.0.0-3487.g4a02f3f4.el8.x86_64 21/25 2023-05-31T19:43:21.763 INFO:teuthology.orchestra.run.smithi099.stdout: Installing : ceph-selinux-2:18.0.0-3487.g4a02f3f4.el8.x86_64 22/25 2023-05-31T19:43:28.447 INFO:teuthology.orchestra.run.smithi112.stdout: Running scriptlet: ceph-selinux-2:18.0.0-3487.g4a02f3f4.el8.x86_64 22/25 2023-05-31T19:43:28.447 INFO:teuthology.orchestra.run.smithi112.stdout:skipping the directory /sys 2023-05-31T19:43:28.448 INFO:teuthology.orchestra.run.smithi112.stdout:skipping the directory /proc 2023-05-31T19:43:28.448 INFO:teuthology.orchestra.run.smithi112.stdout:skipping the directory /dev 2023-05-31T19:43:28.448 INFO:teuthology.orchestra.run.smithi112.stdout:skipping the directory /run 2023-05-31T19:43:28.448 INFO:teuthology.orchestra.run.smithi112.stdout:skipping the directory /mnt 2023-05-31T19:43:28.449 INFO:teuthology.orchestra.run.smithi112.stdout:skipping the directory /var/tmp 2023-05-31T19:43:28.449 INFO:teuthology.orchestra.run.smithi112.stdout:skipping the directory /home 2023-05-31T19:43:28.449 INFO:teuthology.orchestra.run.smithi112.stdout:skipping the directory /tmp 2023-05-31T19:43:28.450 INFO:teuthology.orchestra.run.smithi112.stdout:skipping the directory /dev 2023-05-31T19:43:28.450 INFO:teuthology.orchestra.run.smithi112.stdout: 2023-05-31T19:43:29.860 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : ceph-radosgw-2:18.0.0-3487.g4a02f3f4.el8.x86_64 23/25 2023-05-31T19:43:29.881 INFO:teuthology.orchestra.run.smithi112.stdout: Running scriptlet: ceph-radosgw-2:18.0.0-3487.g4a02f3f4.el8.x86_64 23/25 2023-05-31T19:43:29.881 INFO:teuthology.orchestra.run.smithi112.stdout: Cleanup : librbd1-1:12.2.7-9.el8.x86_64 24/25 2023-05-31T19:43:30.099 INFO:teuthology.orchestra.run.smithi112.stdout: Running scriptlet: librbd1-1:12.2.7-9.el8.x86_64 24/25 2023-05-31T19:43:30.099 INFO:teuthology.orchestra.run.smithi112.stdout: Cleanup : librados2-1:12.2.7-9.el8.x86_64 25/25 2023-05-31T19:43:30.826 INFO:teuthology.orchestra.run.smithi112.stdout: Running scriptlet: librados2-1:12.2.7-9.el8.x86_64 25/25 2023-05-31T19:43:30.826 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : ceph-base-2:18.0.0-3487.g4a02f3f4.el8.x86_64 1/25 2023-05-31T19:43:30.826 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : ceph-common-2:18.0.0-3487.g4a02f3f4.el8.x86_64 2/25 2023-05-31T19:43:30.826 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : ceph-radosgw-2:18.0.0-3487.g4a02f3f4.el8.x86_64 3/25 2023-05-31T19:43:30.827 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : ceph-selinux-2:18.0.0-3487.g4a02f3f4.el8.x86_64 4/25 2023-05-31T19:43:30.827 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : libcephfs2-2:18.0.0-3487.g4a02f3f4.el8.x86_64 5/25 2023-05-31T19:43:30.827 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : libradosstriper1-2:18.0.0-3487.g4a02f3f4.el8.x86_6 6/25 2023-05-31T19:43:30.827 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : librgw2-2:18.0.0-3487.g4a02f3f4.el8.x86_64 7/25 2023-05-31T19:43:30.827 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : python3-ceph-argparse-2:18.0.0-3487.g4a02f3f4.el8. 8/25 2023-05-31T19:43:30.827 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : python3-ceph-common-2:18.0.0-3487.g4a02f3f4.el8.x8 9/25 2023-05-31T19:43:30.828 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : python3-cephfs-2:18.0.0-3487.g4a02f3f4.el8.x86_64 10/25 2023-05-31T19:43:30.828 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : python3-rados-2:18.0.0-3487.g4a02f3f4.el8.x86_64 11/25 2023-05-31T19:43:30.828 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : python3-rbd-2:18.0.0-3487.g4a02f3f4.el8.x86_64 12/25 2023-05-31T19:43:30.828 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : python3-rgw-2:18.0.0-3487.g4a02f3f4.el8.x86_64 13/25 2023-05-31T19:43:30.828 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : librdkafka-0.11.4-3.el8.x86_64 14/25 2023-05-31T19:43:30.828 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : lttng-ust-2.8.1-11.el8.x86_64 15/25 2023-05-31T19:43:30.828 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : python3-prettytable-0.7.2-14.el8.noarch 16/25 2023-05-31T19:43:30.829 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : librabbitmq-0.9.0-3.el8.x86_64 17/25 2023-05-31T19:43:30.829 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : gperftools-libs-1:2.7-9.el8.x86_64 18/25 2023-05-31T19:43:30.829 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : liboath-2.6.2-3.el8.x86_64 19/25 2023-05-31T19:43:30.829 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : libunwind-1.3.1-3.el8.x86_64 20/25 2023-05-31T19:43:30.830 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : thrift-0.13.0-2.el8.x86_64 21/25 2023-05-31T19:43:30.830 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : librados2-2:18.0.0-3487.g4a02f3f4.el8.x86_64 22/25 2023-05-31T19:43:30.830 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : librados2-1:12.2.7-9.el8.x86_64 23/25 2023-05-31T19:43:30.830 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : librbd1-2:18.0.0-3487.g4a02f3f4.el8.x86_64 24/25 2023-05-31T19:43:31.897 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : librbd1-1:12.2.7-9.el8.x86_64 25/25 2023-05-31T19:43:31.897 INFO:teuthology.orchestra.run.smithi112.stdout: 2023-05-31T19:43:31.897 INFO:teuthology.orchestra.run.smithi112.stdout:Upgraded: 2023-05-31T19:43:31.898 INFO:teuthology.orchestra.run.smithi112.stdout: librados2-2:18.0.0-3487.g4a02f3f4.el8.x86_64 2023-05-31T19:43:31.898 INFO:teuthology.orchestra.run.smithi112.stdout: librbd1-2:18.0.0-3487.g4a02f3f4.el8.x86_64 2023-05-31T19:43:31.898 INFO:teuthology.orchestra.run.smithi112.stdout:Installed: 2023-05-31T19:43:31.898 INFO:teuthology.orchestra.run.smithi112.stdout: ceph-base-2:18.0.0-3487.g4a02f3f4.el8.x86_64 2023-05-31T19:43:31.898 INFO:teuthology.orchestra.run.smithi112.stdout: ceph-common-2:18.0.0-3487.g4a02f3f4.el8.x86_64 2023-05-31T19:43:31.898 INFO:teuthology.orchestra.run.smithi112.stdout: ceph-radosgw-2:18.0.0-3487.g4a02f3f4.el8.x86_64 2023-05-31T19:43:31.898 INFO:teuthology.orchestra.run.smithi112.stdout: ceph-selinux-2:18.0.0-3487.g4a02f3f4.el8.x86_64 2023-05-31T19:43:31.899 INFO:teuthology.orchestra.run.smithi112.stdout: gperftools-libs-1:2.7-9.el8.x86_64 2023-05-31T19:43:31.899 INFO:teuthology.orchestra.run.smithi112.stdout: libcephfs2-2:18.0.0-3487.g4a02f3f4.el8.x86_64 2023-05-31T19:43:31.899 INFO:teuthology.orchestra.run.smithi112.stdout: liboath-2.6.2-3.el8.x86_64 2023-05-31T19:43:31.899 INFO:teuthology.orchestra.run.smithi112.stdout: librabbitmq-0.9.0-3.el8.x86_64 2023-05-31T19:43:31.899 INFO:teuthology.orchestra.run.smithi112.stdout: libradosstriper1-2:18.0.0-3487.g4a02f3f4.el8.x86_64 2023-05-31T19:43:31.899 INFO:teuthology.orchestra.run.smithi112.stdout: librdkafka-0.11.4-3.el8.x86_64 2023-05-31T19:43:31.899 INFO:teuthology.orchestra.run.smithi112.stdout: librgw2-2:18.0.0-3487.g4a02f3f4.el8.x86_64 2023-05-31T19:43:31.899 INFO:teuthology.orchestra.run.smithi112.stdout: libunwind-1.3.1-3.el8.x86_64 2023-05-31T19:43:31.900 INFO:teuthology.orchestra.run.smithi112.stdout: lttng-ust-2.8.1-11.el8.x86_64 2023-05-31T19:43:31.900 INFO:teuthology.orchestra.run.smithi112.stdout: python3-ceph-argparse-2:18.0.0-3487.g4a02f3f4.el8.x86_64 2023-05-31T19:43:31.900 INFO:teuthology.orchestra.run.smithi112.stdout: python3-ceph-common-2:18.0.0-3487.g4a02f3f4.el8.x86_64 2023-05-31T19:43:31.900 INFO:teuthology.orchestra.run.smithi112.stdout: python3-cephfs-2:18.0.0-3487.g4a02f3f4.el8.x86_64 2023-05-31T19:43:31.900 INFO:teuthology.orchestra.run.smithi112.stdout: python3-prettytable-0.7.2-14.el8.noarch 2023-05-31T19:43:31.900 INFO:teuthology.orchestra.run.smithi112.stdout: python3-rados-2:18.0.0-3487.g4a02f3f4.el8.x86_64 2023-05-31T19:43:31.900 INFO:teuthology.orchestra.run.smithi112.stdout: python3-rbd-2:18.0.0-3487.g4a02f3f4.el8.x86_64 2023-05-31T19:43:31.901 INFO:teuthology.orchestra.run.smithi112.stdout: python3-rgw-2:18.0.0-3487.g4a02f3f4.el8.x86_64 2023-05-31T19:43:31.901 INFO:teuthology.orchestra.run.smithi112.stdout: thrift-0.13.0-2.el8.x86_64 2023-05-31T19:43:31.901 INFO:teuthology.orchestra.run.smithi112.stdout: 2023-05-31T19:43:31.901 INFO:teuthology.orchestra.run.smithi112.stdout:Complete! 2023-05-31T19:43:32.062 DEBUG:teuthology.orchestra.run.smithi112:> sudo yum -y install ceph-test 2023-05-31T19:43:32.839 INFO:teuthology.orchestra.run.smithi112.stdout:Last metadata expiration check: 0:00:46 ago on Wed 31 May 2023 07:42:46 PM UTC. 2023-05-31T19:43:33.965 INFO:teuthology.orchestra.run.smithi112.stdout:Dependencies resolved. 2023-05-31T19:43:33.967 INFO:teuthology.orchestra.run.smithi112.stdout:================================================================================ 2023-05-31T19:43:33.967 INFO:teuthology.orchestra.run.smithi112.stdout: Package Arch Version Repository Size 2023-05-31T19:43:33.968 INFO:teuthology.orchestra.run.smithi112.stdout:================================================================================ 2023-05-31T19:43:33.968 INFO:teuthology.orchestra.run.smithi112.stdout:Installing: 2023-05-31T19:43:33.968 INFO:teuthology.orchestra.run.smithi112.stdout: ceph-test x86_64 2:18.0.0-3487.g4a02f3f4.el8 ceph 47 M 2023-05-31T19:43:33.968 INFO:teuthology.orchestra.run.smithi112.stdout:Installing dependencies: 2023-05-31T19:43:33.969 INFO:teuthology.orchestra.run.smithi112.stdout: jq x86_64 1.6-7.el8 appstream 202 k 2023-05-31T19:43:33.969 INFO:teuthology.orchestra.run.smithi112.stdout: libcephsqlite x86_64 2:18.0.0-3487.g4a02f3f4.el8 ceph 137 k 2023-05-31T19:43:33.969 INFO:teuthology.orchestra.run.smithi112.stdout: oniguruma x86_64 6.8.2-2.el8 appstream 187 k 2023-05-31T19:43:33.970 INFO:teuthology.orchestra.run.smithi112.stdout: socat x86_64 1.7.4.1-1.el8 appstream 323 k 2023-05-31T19:43:33.970 INFO:teuthology.orchestra.run.smithi112.stdout: xmlstarlet x86_64 1.6.1-20.el8 appstream 71 k 2023-05-31T19:43:33.970 INFO:teuthology.orchestra.run.smithi112.stdout: 2023-05-31T19:43:33.970 INFO:teuthology.orchestra.run.smithi112.stdout:Transaction Summary 2023-05-31T19:43:33.970 INFO:teuthology.orchestra.run.smithi112.stdout:================================================================================ 2023-05-31T19:43:33.970 INFO:teuthology.orchestra.run.smithi112.stdout:Install 6 Packages 2023-05-31T19:43:33.971 INFO:teuthology.orchestra.run.smithi112.stdout: 2023-05-31T19:43:33.971 INFO:teuthology.orchestra.run.smithi112.stdout:Total download size: 47 M 2023-05-31T19:43:33.972 INFO:teuthology.orchestra.run.smithi112.stdout:Installed size: 210 M 2023-05-31T19:43:33.972 INFO:teuthology.orchestra.run.smithi112.stdout:Downloading Packages: 2023-05-31T19:43:34.345 INFO:teuthology.orchestra.run.smithi112.stdout:(1/6): jq-1.6-7.el8.x86_64.rpm 1.2 MB/s | 202 kB 00:00 2023-05-31T19:43:34.420 INFO:teuthology.orchestra.run.smithi112.stdout:(2/6): oniguruma-6.8.2-2.el8.x86_64.rpm 2.4 MB/s | 187 kB 00:00 2023-05-31T19:43:34.530 INFO:teuthology.orchestra.run.smithi112.stdout:(3/6): socat-1.7.4.1-1.el8.x86_64.rpm 2.9 MB/s | 323 kB 00:00 2023-05-31T19:43:34.563 INFO:teuthology.orchestra.run.smithi112.stdout:(4/6): libcephsqlite-18.0.0-3487.g4a02f3f4.el8. 357 kB/s | 137 kB 00:00 2023-05-31T19:43:34.595 INFO:teuthology.orchestra.run.smithi112.stdout:(5/6): xmlstarlet-1.6.1-20.el8.x86_64.rpm 1.1 MB/s | 71 kB 00:00 2023-05-31T19:43:36.682 INFO:teuthology.orchestra.run.smithi112.stdout:(6/6): ceph-test-18.0.0-3487.g4a02f3f4.el8.x86_ 19 MB/s | 47 MB 00:02 2023-05-31T19:43:36.685 INFO:teuthology.orchestra.run.smithi112.stdout:-------------------------------------------------------------------------------- 2023-05-31T19:43:36.685 INFO:teuthology.orchestra.run.smithi112.stdout:Total 17 MB/s | 47 MB 00:02 2023-05-31T19:43:36.726 INFO:teuthology.orchestra.run.smithi112.stdout:Running transaction check 2023-05-31T19:43:36.748 INFO:teuthology.orchestra.run.smithi112.stdout:Transaction check succeeded. 2023-05-31T19:43:36.748 INFO:teuthology.orchestra.run.smithi112.stdout:Running transaction test 2023-05-31T19:43:37.053 INFO:teuthology.orchestra.run.smithi112.stdout:Transaction test succeeded. 2023-05-31T19:43:37.056 INFO:teuthology.orchestra.run.smithi112.stdout:Running transaction 2023-05-31T19:43:37.545 INFO:teuthology.orchestra.run.smithi112.stdout: Preparing : 1/1 2023-05-31T19:43:37.757 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : xmlstarlet-1.6.1-20.el8.x86_64 1/6 2023-05-31T19:43:37.957 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : socat-1.7.4.1-1.el8.x86_64 2/6 2023-05-31T19:43:38.070 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : oniguruma-6.8.2-2.el8.x86_64 3/6 2023-05-31T19:43:38.136 INFO:teuthology.orchestra.run.smithi112.stdout: Running scriptlet: oniguruma-6.8.2-2.el8.x86_64 3/6 2023-05-31T19:43:38.341 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : jq-1.6-7.el8.x86_64 4/6 2023-05-31T19:43:38.529 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : libcephsqlite-2:18.0.0-3487.g4a02f3f4.el8.x86_64 5/6 2023-05-31T19:43:40.579 INFO:teuthology.orchestra.run.smithi099.stdout: Running scriptlet: ceph-selinux-2:18.0.0-3487.g4a02f3f4.el8.x86_64 22/25 2023-05-31T19:43:40.579 INFO:teuthology.orchestra.run.smithi099.stdout:skipping the directory /sys 2023-05-31T19:43:40.580 INFO:teuthology.orchestra.run.smithi099.stdout:skipping the directory /proc 2023-05-31T19:43:40.580 INFO:teuthology.orchestra.run.smithi099.stdout:skipping the directory /dev 2023-05-31T19:43:40.580 INFO:teuthology.orchestra.run.smithi099.stdout:skipping the directory /run 2023-05-31T19:43:40.580 INFO:teuthology.orchestra.run.smithi099.stdout:skipping the directory /mnt 2023-05-31T19:43:40.580 INFO:teuthology.orchestra.run.smithi099.stdout:skipping the directory /var/tmp 2023-05-31T19:43:40.580 INFO:teuthology.orchestra.run.smithi099.stdout:skipping the directory /home 2023-05-31T19:43:40.581 INFO:teuthology.orchestra.run.smithi099.stdout:skipping the directory /tmp 2023-05-31T19:43:40.581 INFO:teuthology.orchestra.run.smithi099.stdout:skipping the directory /dev 2023-05-31T19:43:40.581 INFO:teuthology.orchestra.run.smithi099.stdout: 2023-05-31T19:43:41.989 INFO:teuthology.orchestra.run.smithi099.stdout: Installing : ceph-radosgw-2:18.0.0-3487.g4a02f3f4.el8.x86_64 23/25 2023-05-31T19:43:42.009 INFO:teuthology.orchestra.run.smithi099.stdout: Running scriptlet: ceph-radosgw-2:18.0.0-3487.g4a02f3f4.el8.x86_64 23/25 2023-05-31T19:43:42.010 INFO:teuthology.orchestra.run.smithi099.stdout: Cleanup : librbd1-1:12.2.7-9.el8.x86_64 24/25 2023-05-31T19:43:42.211 INFO:teuthology.orchestra.run.smithi099.stdout: Running scriptlet: librbd1-1:12.2.7-9.el8.x86_64 24/25 2023-05-31T19:43:42.211 INFO:teuthology.orchestra.run.smithi099.stdout: Cleanup : librados2-1:12.2.7-9.el8.x86_64 25/25 2023-05-31T19:43:42.927 INFO:teuthology.orchestra.run.smithi099.stdout: Running scriptlet: librados2-1:12.2.7-9.el8.x86_64 25/25 2023-05-31T19:43:42.928 INFO:teuthology.orchestra.run.smithi099.stdout: Verifying : ceph-base-2:18.0.0-3487.g4a02f3f4.el8.x86_64 1/25 2023-05-31T19:43:42.928 INFO:teuthology.orchestra.run.smithi099.stdout: Verifying : ceph-common-2:18.0.0-3487.g4a02f3f4.el8.x86_64 2/25 2023-05-31T19:43:42.928 INFO:teuthology.orchestra.run.smithi099.stdout: Verifying : ceph-radosgw-2:18.0.0-3487.g4a02f3f4.el8.x86_64 3/25 2023-05-31T19:43:42.928 INFO:teuthology.orchestra.run.smithi099.stdout: Verifying : ceph-selinux-2:18.0.0-3487.g4a02f3f4.el8.x86_64 4/25 2023-05-31T19:43:42.928 INFO:teuthology.orchestra.run.smithi099.stdout: Verifying : libcephfs2-2:18.0.0-3487.g4a02f3f4.el8.x86_64 5/25 2023-05-31T19:43:42.929 INFO:teuthology.orchestra.run.smithi099.stdout: Verifying : libradosstriper1-2:18.0.0-3487.g4a02f3f4.el8.x86_6 6/25 2023-05-31T19:43:42.929 INFO:teuthology.orchestra.run.smithi099.stdout: Verifying : librgw2-2:18.0.0-3487.g4a02f3f4.el8.x86_64 7/25 2023-05-31T19:43:42.929 INFO:teuthology.orchestra.run.smithi099.stdout: Verifying : python3-ceph-argparse-2:18.0.0-3487.g4a02f3f4.el8. 8/25 2023-05-31T19:43:42.929 INFO:teuthology.orchestra.run.smithi099.stdout: Verifying : python3-ceph-common-2:18.0.0-3487.g4a02f3f4.el8.x8 9/25 2023-05-31T19:43:42.929 INFO:teuthology.orchestra.run.smithi099.stdout: Verifying : python3-cephfs-2:18.0.0-3487.g4a02f3f4.el8.x86_64 10/25 2023-05-31T19:43:42.929 INFO:teuthology.orchestra.run.smithi099.stdout: Verifying : python3-rados-2:18.0.0-3487.g4a02f3f4.el8.x86_64 11/25 2023-05-31T19:43:42.930 INFO:teuthology.orchestra.run.smithi099.stdout: Verifying : python3-rbd-2:18.0.0-3487.g4a02f3f4.el8.x86_64 12/25 2023-05-31T19:43:42.930 INFO:teuthology.orchestra.run.smithi099.stdout: Verifying : python3-rgw-2:18.0.0-3487.g4a02f3f4.el8.x86_64 13/25 2023-05-31T19:43:42.930 INFO:teuthology.orchestra.run.smithi099.stdout: Verifying : librdkafka-0.11.4-3.el8.x86_64 14/25 2023-05-31T19:43:42.930 INFO:teuthology.orchestra.run.smithi099.stdout: Verifying : lttng-ust-2.8.1-11.el8.x86_64 15/25 2023-05-31T19:43:42.930 INFO:teuthology.orchestra.run.smithi099.stdout: Verifying : python3-prettytable-0.7.2-14.el8.noarch 16/25 2023-05-31T19:43:42.930 INFO:teuthology.orchestra.run.smithi099.stdout: Verifying : librabbitmq-0.9.0-3.el8.x86_64 17/25 2023-05-31T19:43:42.930 INFO:teuthology.orchestra.run.smithi099.stdout: Verifying : gperftools-libs-1:2.7-9.el8.x86_64 18/25 2023-05-31T19:43:42.930 INFO:teuthology.orchestra.run.smithi099.stdout: Verifying : liboath-2.6.2-3.el8.x86_64 19/25 2023-05-31T19:43:42.931 INFO:teuthology.orchestra.run.smithi099.stdout: Verifying : libunwind-1.3.1-3.el8.x86_64 20/25 2023-05-31T19:43:42.931 INFO:teuthology.orchestra.run.smithi099.stdout: Verifying : thrift-0.13.0-2.el8.x86_64 21/25 2023-05-31T19:43:42.931 INFO:teuthology.orchestra.run.smithi099.stdout: Verifying : librados2-2:18.0.0-3487.g4a02f3f4.el8.x86_64 22/25 2023-05-31T19:43:42.931 INFO:teuthology.orchestra.run.smithi099.stdout: Verifying : librados2-1:12.2.7-9.el8.x86_64 23/25 2023-05-31T19:43:42.931 INFO:teuthology.orchestra.run.smithi099.stdout: Verifying : librbd1-2:18.0.0-3487.g4a02f3f4.el8.x86_64 24/25 2023-05-31T19:43:43.052 INFO:teuthology.orchestra.run.smithi112.stdout: Running scriptlet: libcephsqlite-2:18.0.0-3487.g4a02f3f4.el8.x86_64 5/6 2023-05-31T19:43:43.302 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : ceph-test-2:18.0.0-3487.g4a02f3f4.el8.x86_64 6/6 2023-05-31T19:43:43.563 INFO:teuthology.orchestra.run.smithi112.stdout: Running scriptlet: ceph-test-2:18.0.0-3487.g4a02f3f4.el8.x86_64 6/6 2023-05-31T19:43:43.563 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : ceph-test-2:18.0.0-3487.g4a02f3f4.el8.x86_64 1/6 2023-05-31T19:43:43.563 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : libcephsqlite-2:18.0.0-3487.g4a02f3f4.el8.x86_64 2/6 2023-05-31T19:43:43.564 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : jq-1.6-7.el8.x86_64 3/6 2023-05-31T19:43:43.564 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : oniguruma-6.8.2-2.el8.x86_64 4/6 2023-05-31T19:43:43.564 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : socat-1.7.4.1-1.el8.x86_64 5/6 2023-05-31T19:43:43.956 INFO:teuthology.orchestra.run.smithi099.stdout: Verifying : librbd1-1:12.2.7-9.el8.x86_64 25/25 2023-05-31T19:43:43.956 INFO:teuthology.orchestra.run.smithi099.stdout: 2023-05-31T19:43:43.956 INFO:teuthology.orchestra.run.smithi099.stdout:Upgraded: 2023-05-31T19:43:43.956 INFO:teuthology.orchestra.run.smithi099.stdout: librados2-2:18.0.0-3487.g4a02f3f4.el8.x86_64 2023-05-31T19:43:43.957 INFO:teuthology.orchestra.run.smithi099.stdout: librbd1-2:18.0.0-3487.g4a02f3f4.el8.x86_64 2023-05-31T19:43:43.957 INFO:teuthology.orchestra.run.smithi099.stdout:Installed: 2023-05-31T19:43:43.957 INFO:teuthology.orchestra.run.smithi099.stdout: ceph-base-2:18.0.0-3487.g4a02f3f4.el8.x86_64 2023-05-31T19:43:43.957 INFO:teuthology.orchestra.run.smithi099.stdout: ceph-common-2:18.0.0-3487.g4a02f3f4.el8.x86_64 2023-05-31T19:43:43.957 INFO:teuthology.orchestra.run.smithi099.stdout: ceph-radosgw-2:18.0.0-3487.g4a02f3f4.el8.x86_64 2023-05-31T19:43:43.957 INFO:teuthology.orchestra.run.smithi099.stdout: ceph-selinux-2:18.0.0-3487.g4a02f3f4.el8.x86_64 2023-05-31T19:43:43.957 INFO:teuthology.orchestra.run.smithi099.stdout: gperftools-libs-1:2.7-9.el8.x86_64 2023-05-31T19:43:43.958 INFO:teuthology.orchestra.run.smithi099.stdout: libcephfs2-2:18.0.0-3487.g4a02f3f4.el8.x86_64 2023-05-31T19:43:43.958 INFO:teuthology.orchestra.run.smithi099.stdout: liboath-2.6.2-3.el8.x86_64 2023-05-31T19:43:43.958 INFO:teuthology.orchestra.run.smithi099.stdout: librabbitmq-0.9.0-3.el8.x86_64 2023-05-31T19:43:43.958 INFO:teuthology.orchestra.run.smithi099.stdout: libradosstriper1-2:18.0.0-3487.g4a02f3f4.el8.x86_64 2023-05-31T19:43:43.958 INFO:teuthology.orchestra.run.smithi099.stdout: librdkafka-0.11.4-3.el8.x86_64 2023-05-31T19:43:43.958 INFO:teuthology.orchestra.run.smithi099.stdout: librgw2-2:18.0.0-3487.g4a02f3f4.el8.x86_64 2023-05-31T19:43:43.958 INFO:teuthology.orchestra.run.smithi099.stdout: libunwind-1.3.1-3.el8.x86_64 2023-05-31T19:43:43.959 INFO:teuthology.orchestra.run.smithi099.stdout: lttng-ust-2.8.1-11.el8.x86_64 2023-05-31T19:43:43.959 INFO:teuthology.orchestra.run.smithi099.stdout: python3-ceph-argparse-2:18.0.0-3487.g4a02f3f4.el8.x86_64 2023-05-31T19:43:43.959 INFO:teuthology.orchestra.run.smithi099.stdout: python3-ceph-common-2:18.0.0-3487.g4a02f3f4.el8.x86_64 2023-05-31T19:43:43.959 INFO:teuthology.orchestra.run.smithi099.stdout: python3-cephfs-2:18.0.0-3487.g4a02f3f4.el8.x86_64 2023-05-31T19:43:43.959 INFO:teuthology.orchestra.run.smithi099.stdout: python3-prettytable-0.7.2-14.el8.noarch 2023-05-31T19:43:43.959 INFO:teuthology.orchestra.run.smithi099.stdout: python3-rados-2:18.0.0-3487.g4a02f3f4.el8.x86_64 2023-05-31T19:43:43.959 INFO:teuthology.orchestra.run.smithi099.stdout: python3-rbd-2:18.0.0-3487.g4a02f3f4.el8.x86_64 2023-05-31T19:43:43.960 INFO:teuthology.orchestra.run.smithi099.stdout: python3-rgw-2:18.0.0-3487.g4a02f3f4.el8.x86_64 2023-05-31T19:43:43.960 INFO:teuthology.orchestra.run.smithi099.stdout: thrift-0.13.0-2.el8.x86_64 2023-05-31T19:43:43.960 INFO:teuthology.orchestra.run.smithi099.stdout: 2023-05-31T19:43:43.960 INFO:teuthology.orchestra.run.smithi099.stdout:Complete! 2023-05-31T19:43:44.091 DEBUG:teuthology.orchestra.run.smithi099:> sudo yum -y install ceph-test 2023-05-31T19:43:44.157 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : xmlstarlet-1.6.1-20.el8.x86_64 6/6 2023-05-31T19:43:44.158 INFO:teuthology.orchestra.run.smithi112.stdout: 2023-05-31T19:43:44.158 INFO:teuthology.orchestra.run.smithi112.stdout:Installed: 2023-05-31T19:43:44.158 INFO:teuthology.orchestra.run.smithi112.stdout: ceph-test-2:18.0.0-3487.g4a02f3f4.el8.x86_64 2023-05-31T19:43:44.158 INFO:teuthology.orchestra.run.smithi112.stdout: jq-1.6-7.el8.x86_64 2023-05-31T19:43:44.158 INFO:teuthology.orchestra.run.smithi112.stdout: libcephsqlite-2:18.0.0-3487.g4a02f3f4.el8.x86_64 2023-05-31T19:43:44.158 INFO:teuthology.orchestra.run.smithi112.stdout: oniguruma-6.8.2-2.el8.x86_64 2023-05-31T19:43:44.159 INFO:teuthology.orchestra.run.smithi112.stdout: socat-1.7.4.1-1.el8.x86_64 2023-05-31T19:43:44.159 INFO:teuthology.orchestra.run.smithi112.stdout: xmlstarlet-1.6.1-20.el8.x86_64 2023-05-31T19:43:44.159 INFO:teuthology.orchestra.run.smithi112.stdout: 2023-05-31T19:43:44.159 INFO:teuthology.orchestra.run.smithi112.stdout:Complete! 2023-05-31T19:43:44.368 DEBUG:teuthology.orchestra.run.smithi112:> sudo yum -y install ceph 2023-05-31T19:43:44.859 INFO:teuthology.orchestra.run.smithi099.stdout:Last metadata expiration check: 0:00:44 ago on Wed 31 May 2023 07:43:00 PM UTC. 2023-05-31T19:43:45.098 INFO:teuthology.orchestra.run.smithi112.stdout:Last metadata expiration check: 0:00:59 ago on Wed 31 May 2023 07:42:46 PM UTC. 2023-05-31T19:43:45.962 INFO:teuthology.orchestra.run.smithi099.stdout:Dependencies resolved. 2023-05-31T19:43:45.963 INFO:teuthology.orchestra.run.smithi099.stdout:================================================================================ 2023-05-31T19:43:45.964 INFO:teuthology.orchestra.run.smithi099.stdout: Package Arch Version Repository Size 2023-05-31T19:43:45.964 INFO:teuthology.orchestra.run.smithi099.stdout:================================================================================ 2023-05-31T19:43:45.964 INFO:teuthology.orchestra.run.smithi099.stdout:Installing: 2023-05-31T19:43:45.964 INFO:teuthology.orchestra.run.smithi099.stdout: ceph-test x86_64 2:18.0.0-3487.g4a02f3f4.el8 ceph 47 M 2023-05-31T19:43:45.965 INFO:teuthology.orchestra.run.smithi099.stdout:Installing dependencies: 2023-05-31T19:43:45.965 INFO:teuthology.orchestra.run.smithi099.stdout: jq x86_64 1.6-7.el8 appstream 202 k 2023-05-31T19:43:45.965 INFO:teuthology.orchestra.run.smithi099.stdout: libcephsqlite x86_64 2:18.0.0-3487.g4a02f3f4.el8 ceph 137 k 2023-05-31T19:43:45.965 INFO:teuthology.orchestra.run.smithi099.stdout: oniguruma x86_64 6.8.2-2.el8 appstream 187 k 2023-05-31T19:43:45.965 INFO:teuthology.orchestra.run.smithi099.stdout: socat x86_64 1.7.4.1-1.el8 appstream 323 k 2023-05-31T19:43:45.965 INFO:teuthology.orchestra.run.smithi099.stdout: xmlstarlet x86_64 1.6.1-20.el8 appstream 71 k 2023-05-31T19:43:45.965 INFO:teuthology.orchestra.run.smithi099.stdout: 2023-05-31T19:43:45.966 INFO:teuthology.orchestra.run.smithi099.stdout:Transaction Summary 2023-05-31T19:43:45.966 INFO:teuthology.orchestra.run.smithi099.stdout:================================================================================ 2023-05-31T19:43:45.966 INFO:teuthology.orchestra.run.smithi099.stdout:Install 6 Packages 2023-05-31T19:43:45.967 INFO:teuthology.orchestra.run.smithi099.stdout: 2023-05-31T19:43:45.967 INFO:teuthology.orchestra.run.smithi099.stdout:Total download size: 47 M 2023-05-31T19:43:45.968 INFO:teuthology.orchestra.run.smithi099.stdout:Installed size: 210 M 2023-05-31T19:43:45.968 INFO:teuthology.orchestra.run.smithi099.stdout:Downloading Packages: 2023-05-31T19:43:46.222 INFO:teuthology.orchestra.run.smithi112.stdout:Dependencies resolved. 2023-05-31T19:43:46.226 INFO:teuthology.orchestra.run.smithi112.stdout:================================================================================ 2023-05-31T19:43:46.226 INFO:teuthology.orchestra.run.smithi112.stdout: Package Arch Version Repository Size 2023-05-31T19:43:46.226 INFO:teuthology.orchestra.run.smithi112.stdout:================================================================================ 2023-05-31T19:43:46.226 INFO:teuthology.orchestra.run.smithi112.stdout:Installing: 2023-05-31T19:43:46.227 INFO:teuthology.orchestra.run.smithi112.stdout: ceph x86_64 2:18.0.0-3487.g4a02f3f4.el8 ceph 6.4 k 2023-05-31T19:43:46.227 INFO:teuthology.orchestra.run.smithi112.stdout:Installing dependencies: 2023-05-31T19:43:46.227 INFO:teuthology.orchestra.run.smithi112.stdout: ceph-mds x86_64 2:18.0.0-3487.g4a02f3f4.el8 ceph 2.3 M 2023-05-31T19:43:46.228 INFO:teuthology.orchestra.run.smithi112.stdout: ceph-mgr x86_64 2:18.0.0-3487.g4a02f3f4.el8 ceph 1.5 M 2023-05-31T19:43:46.228 INFO:teuthology.orchestra.run.smithi112.stdout: ceph-mgr-modules-core noarch 2:18.0.0-3487.g4a02f3f4.el8 ceph-noarch 240 k 2023-05-31T19:43:46.228 INFO:teuthology.orchestra.run.smithi112.stdout: ceph-mon x86_64 2:18.0.0-3487.g4a02f3f4.el8 ceph 4.6 M 2023-05-31T19:43:46.228 INFO:teuthology.orchestra.run.smithi112.stdout: ceph-osd x86_64 2:18.0.0-3487.g4a02f3f4.el8 ceph 18 M 2023-05-31T19:43:46.228 INFO:teuthology.orchestra.run.smithi112.stdout: python3-bcrypt x86_64 3.1.6-2.el8.1 epel 44 k 2023-05-31T19:43:46.229 INFO:teuthology.orchestra.run.smithi112.stdout: python3-beautifulsoup4 noarch 4.6.3-2.el8.1 epel 185 k 2023-05-31T19:43:46.229 INFO:teuthology.orchestra.run.smithi112.stdout: python3-cffi x86_64 1.11.5-5.el8 baseos 237 k 2023-05-31T19:43:46.229 INFO:teuthology.orchestra.run.smithi112.stdout: python3-cheroot noarch 8.5.2-1.el8 epel 173 k 2023-05-31T19:43:46.229 INFO:teuthology.orchestra.run.smithi112.stdout: python3-cherrypy noarch 18.4.0-1.el8 epel 384 k 2023-05-31T19:43:46.229 INFO:teuthology.orchestra.run.smithi112.stdout: python3-cryptography x86_64 3.2.1-5.el8 baseos 559 k 2023-05-31T19:43:46.229 INFO:teuthology.orchestra.run.smithi112.stdout: python3-jaraco noarch 6.2-6.el8 epel 11 k 2023-05-31T19:43:46.229 INFO:teuthology.orchestra.run.smithi112.stdout: python3-jaraco-functools noarch 2.0-4.el8 epel 18 k 2023-05-31T19:43:46.230 INFO:teuthology.orchestra.run.smithi112.stdout: python3-logutils noarch 0.3.5-11.el8 epel 49 k 2023-05-31T19:43:46.230 INFO:teuthology.orchestra.run.smithi112.stdout: python3-mako noarch 1.0.6-14.el8 appstream 157 k 2023-05-31T19:43:46.230 INFO:teuthology.orchestra.run.smithi112.stdout: python3-markupsafe x86_64 0.23-19.el8 appstream 39 k 2023-05-31T19:43:46.230 INFO:teuthology.orchestra.run.smithi112.stdout: python3-more-itertools noarch 7.2.0-3.el8 epel 59 k 2023-05-31T19:43:46.230 INFO:teuthology.orchestra.run.smithi112.stdout: python3-pecan noarch 1.3.2-9.el8 epel 283 k 2023-05-31T19:43:46.231 INFO:teuthology.orchestra.run.smithi112.stdout: python3-portend noarch 2.6-1.el8 epel 16 k 2023-05-31T19:43:46.231 INFO:teuthology.orchestra.run.smithi112.stdout: python3-pyOpenSSL noarch 19.0.0-1.el8 appstream 103 k 2023-05-31T19:43:46.231 INFO:teuthology.orchestra.run.smithi112.stdout: python3-pycparser noarch 2.14-14.el8 baseos 109 k 2023-05-31T19:43:46.231 INFO:teuthology.orchestra.run.smithi112.stdout: python3-pytz noarch 2017.2-9.el8 appstream 54 k 2023-05-31T19:43:46.231 INFO:teuthology.orchestra.run.smithi112.stdout: python3-simplegeneric noarch 0.8.1-17.el8 epel 19 k 2023-05-31T19:43:46.231 INFO:teuthology.orchestra.run.smithi112.stdout: python3-singledispatch noarch 3.4.0.3-18.el8 epel 24 k 2023-05-31T19:43:46.231 INFO:teuthology.orchestra.run.smithi112.stdout: python3-tempora noarch 1.14.1-5.el8 epel 29 k 2023-05-31T19:43:46.231 INFO:teuthology.orchestra.run.smithi112.stdout: python3-trustme noarch 0.6.0-4.el8 epel 27 k 2023-05-31T19:43:46.232 INFO:teuthology.orchestra.run.smithi112.stdout: python3-waitress noarch 1.4.3-1.el8 epel 255 k 2023-05-31T19:43:46.232 INFO:teuthology.orchestra.run.smithi112.stdout: python3-webob noarch 1.8.5-1.el8.1 epel 251 k 2023-05-31T19:43:46.232 INFO:teuthology.orchestra.run.smithi112.stdout: python3-webtest noarch 2.0.33-1.el8 epel 85 k 2023-05-31T19:43:46.232 INFO:teuthology.orchestra.run.smithi112.stdout: python3-werkzeug noarch 0.12.2-4.el8 appstream 457 k 2023-05-31T19:43:46.232 INFO:teuthology.orchestra.run.smithi112.stdout: python3-zc-lockfile noarch 2.0-2.el8 epel 23 k 2023-05-31T19:43:46.232 INFO:teuthology.orchestra.run.smithi112.stdout: 2023-05-31T19:43:46.233 INFO:teuthology.orchestra.run.smithi112.stdout:Transaction Summary 2023-05-31T19:43:46.233 INFO:teuthology.orchestra.run.smithi112.stdout:================================================================================ 2023-05-31T19:43:46.233 INFO:teuthology.orchestra.run.smithi112.stdout:Install 32 Packages 2023-05-31T19:43:46.233 INFO:teuthology.orchestra.run.smithi112.stdout: 2023-05-31T19:43:46.234 INFO:teuthology.orchestra.run.smithi112.stdout:Total download size: 30 M 2023-05-31T19:43:46.234 INFO:teuthology.orchestra.run.smithi112.stdout:Installed size: 119 M 2023-05-31T19:43:46.234 INFO:teuthology.orchestra.run.smithi112.stdout:Downloading Packages: 2023-05-31T19:43:46.274 INFO:teuthology.orchestra.run.smithi099.stdout:(1/6): jq-1.6-7.el8.x86_64.rpm 1.5 MB/s | 202 kB 00:00 2023-05-31T19:43:46.316 INFO:teuthology.orchestra.run.smithi099.stdout:(2/6): oniguruma-6.8.2-2.el8.x86_64.rpm 4.4 MB/s | 187 kB 00:00 2023-05-31T19:43:46.342 INFO:teuthology.orchestra.run.smithi099.stdout:(3/6): libcephsqlite-18.0.0-3487.g4a02f3f4.el8. 700 kB/s | 137 kB 00:00 2023-05-31T19:43:46.400 INFO:teuthology.orchestra.run.smithi099.stdout:(4/6): socat-1.7.4.1-1.el8.x86_64.rpm 3.7 MB/s | 323 kB 00:00 2023-05-31T19:43:46.467 INFO:teuthology.orchestra.run.smithi099.stdout:(5/6): xmlstarlet-1.6.1-20.el8.x86_64.rpm 564 kB/s | 71 kB 00:00 2023-05-31T19:43:47.019 INFO:teuthology.orchestra.run.smithi112.stdout:(1/32): ceph-18.0.0-3487.g4a02f3f4.el8.x86_64.r 24 kB/s | 6.4 kB 00:00 2023-05-31T19:43:47.284 INFO:teuthology.orchestra.run.smithi112.stdout:(2/32): ceph-mds-18.0.0-3487.g4a02f3f4.el8.x86_ 4.2 MB/s | 2.3 MB 00:00 2023-05-31T19:43:48.026 INFO:teuthology.orchestra.run.smithi112.stdout:(3/32): ceph-mgr-18.0.0-3487.g4a02f3f4.el8.x86_ 1.2 MB/s | 1.5 MB 00:01 2023-05-31T19:43:48.232 INFO:teuthology.orchestra.run.smithi112.stdout:(4/32): ceph-mgr-modules-core-18.0.0-3487.g4a02 1.1 MB/s | 240 kB 00:00 2023-05-31T19:43:48.454 INFO:teuthology.orchestra.run.smithi112.stdout:(5/32): ceph-osd-18.0.0-3487.g4a02f3f4.el8.x86_ 15 MB/s | 18 MB 00:01 2023-05-31T19:43:48.554 INFO:teuthology.orchestra.run.smithi112.stdout:(6/32): python3-markupsafe-0.23-19.el8.x86_64.r 389 kB/s | 39 kB 00:00 2023-05-31T19:43:48.580 INFO:teuthology.orchestra.run.smithi112.stdout:(7/32): python3-mako-1.0.6-14.el8.noarch.rpm 452 kB/s | 157 kB 00:00 2023-05-31T19:43:48.621 INFO:teuthology.orchestra.run.smithi112.stdout:(8/32): python3-pytz-2017.2-9.el8.noarch.rpm 1.3 MB/s | 54 kB 00:00 2023-05-31T19:43:48.647 INFO:teuthology.orchestra.run.smithi112.stdout:(9/32): python3-pyOpenSSL-19.0.0-1.el8.noarch.r 1.1 MB/s | 103 kB 00:00 2023-05-31T19:43:48.756 INFO:teuthology.orchestra.run.smithi112.stdout:(10/32): python3-werkzeug-0.12.2-4.el8.noarch.r 3.3 MB/s | 457 kB 00:00 2023-05-31T19:43:48.889 INFO:teuthology.orchestra.run.smithi112.stdout:(11/32): python3-cffi-1.11.5-5.el8.x86_64.rpm 983 kB/s | 237 kB 00:00 2023-05-31T19:43:49.064 INFO:teuthology.orchestra.run.smithi112.stdout:(12/32): ceph-mon-18.0.0-3487.g4a02f3f4.el8.x86 2.3 MB/s | 4.6 MB 00:02 2023-05-31T19:43:49.109 INFO:teuthology.orchestra.run.smithi112.stdout:(13/32): python3-cryptography-3.2.1-5.el8.x86_6 1.5 MB/s | 559 kB 00:00 2023-05-31T19:43:49.140 INFO:teuthology.orchestra.run.smithi112.stdout:(14/32): python3-pycparser-2.14-14.el8.noarch.r 435 kB/s | 109 kB 00:00 2023-05-31T19:43:49.332 INFO:teuthology.orchestra.run.smithi112.stdout:(15/32): python3-bcrypt-3.1.6-2.el8.1.x86_64.rp 163 kB/s | 44 kB 00:00 2023-05-31T19:43:49.424 INFO:teuthology.orchestra.run.smithi112.stdout:(16/32): python3-beautifulsoup4-4.6.3-2.el8.1.n 589 kB/s | 185 kB 00:00 2023-05-31T19:43:49.449 INFO:teuthology.orchestra.run.smithi112.stdout:(17/32): python3-cheroot-8.5.2-1.el8.noarch.rpm 559 kB/s | 173 kB 00:00 2023-05-31T19:43:49.541 INFO:teuthology.orchestra.run.smithi112.stdout:(18/32): python3-jaraco-functools-2.0-4.el8.noa 192 kB/s | 18 kB 00:00 2023-05-31T19:43:49.566 INFO:teuthology.orchestra.run.smithi112.stdout:(19/32): python3-cherrypy-18.4.0-1.el8.noarch.r 1.6 MB/s | 384 kB 00:00 2023-05-31T19:43:49.617 INFO:teuthology.orchestra.run.smithi112.stdout:(20/32): python3-logutils-0.3.5-11.el8.noarch.r 646 kB/s | 49 kB 00:00 2023-05-31T19:43:49.642 INFO:teuthology.orchestra.run.smithi112.stdout:(21/32): python3-more-itertools-7.2.0-3.el8.noa 781 kB/s | 59 kB 00:00 2023-05-31T19:43:49.742 INFO:teuthology.orchestra.run.smithi112.stdout:(22/32): python3-portend-2.6-1.el8.noarch.rpm 156 kB/s | 16 kB 00:00 2023-05-31T19:43:49.759 INFO:teuthology.orchestra.run.smithi112.stdout:(23/32): python3-jaraco-6.2-6.el8.noarch.rpm 33 kB/s | 11 kB 00:00 2023-05-31T19:43:49.784 INFO:teuthology.orchestra.run.smithi112.stdout:(24/32): python3-pecan-1.3.2-9.el8.noarch.rpm 1.7 MB/s | 283 kB 00:00 2023-05-31T19:43:49.818 INFO:teuthology.orchestra.run.smithi112.stdout:(25/32): python3-simplegeneric-0.8.1-17.el8.noa 248 kB/s | 19 kB 00:00 2023-05-31T19:43:49.843 INFO:teuthology.orchestra.run.smithi112.stdout:(26/32): python3-singledispatch-3.4.0.3-18.el8. 285 kB/s | 24 kB 00:00 2023-05-31T19:43:49.869 INFO:teuthology.orchestra.run.smithi112.stdout:(27/32): python3-tempora-1.14.1-5.el8.noarch.rp 349 kB/s | 29 kB 00:00 2023-05-31T19:43:49.910 INFO:teuthology.orchestra.run.smithi112.stdout:(28/32): python3-trustme-0.6.0-4.el8.noarch.rpm 299 kB/s | 27 kB 00:00 2023-05-31T19:43:50.002 INFO:teuthology.orchestra.run.smithi112.stdout:(29/32): python3-webob-1.8.5-1.el8.1.noarch.rpm 1.8 MB/s | 251 kB 00:00 2023-05-31T19:43:50.027 INFO:teuthology.orchestra.run.smithi112.stdout:(30/32): python3-webtest-2.0.33-1.el8.noarch.rp 730 kB/s | 85 kB 00:00 2023-05-31T19:43:50.053 INFO:teuthology.orchestra.run.smithi112.stdout:(31/32): python3-waitress-1.4.3-1.el8.noarch.rp 1.2 MB/s | 255 kB 00:00 2023-05-31T19:43:50.095 INFO:teuthology.orchestra.run.smithi112.stdout:(32/32): python3-zc-lockfile-2.0-2.el8.noarch.r 250 kB/s | 23 kB 00:00 2023-05-31T19:43:50.097 INFO:teuthology.orchestra.run.smithi112.stdout:-------------------------------------------------------------------------------- 2023-05-31T19:43:50.097 INFO:teuthology.orchestra.run.smithi112.stdout:Total 7.7 MB/s | 30 MB 00:03 2023-05-31T19:43:50.180 INFO:teuthology.orchestra.run.smithi112.stdout:Running transaction check 2023-05-31T19:43:50.219 INFO:teuthology.orchestra.run.smithi112.stdout:Transaction check succeeded. 2023-05-31T19:43:50.219 INFO:teuthology.orchestra.run.smithi112.stdout:Running transaction test 2023-05-31T19:43:50.541 INFO:teuthology.orchestra.run.smithi112.stdout:Transaction test succeeded. 2023-05-31T19:43:50.544 INFO:teuthology.orchestra.run.smithi112.stdout:Running transaction 2023-05-31T19:43:51.728 INFO:teuthology.orchestra.run.smithi112.stdout: Preparing : 1/1 2023-05-31T19:43:51.983 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : python3-more-itertools-7.2.0-3.el8.noarch 1/32 2023-05-31T19:43:52.235 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : python3-webob-1.8.5-1.el8.1.noarch 2/32 2023-05-31T19:43:52.519 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : python3-zc-lockfile-2.0-2.el8.noarch 3/32 2023-05-31T19:43:52.759 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : python3-waitress-1.4.3-1.el8.noarch 4/32 2023-05-31T19:43:52.969 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : python3-singledispatch-3.4.0.3-18.el8.noarch 5/32 2023-05-31T19:43:53.188 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : python3-simplegeneric-0.8.1-17.el8.noarch 6/32 2023-05-31T19:43:53.413 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : python3-logutils-0.3.5-11.el8.noarch 7/32 2023-05-31T19:43:53.641 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : python3-jaraco-6.2-6.el8.noarch 8/32 2023-05-31T19:43:53.934 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : python3-jaraco-functools-2.0-4.el8.noarch 9/32 2023-05-31T19:43:54.188 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : python3-beautifulsoup4-4.6.3-2.el8.1.noarch 10/32 2023-05-31T19:43:54.417 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : python3-webtest-2.0.33-1.el8.noarch 11/32 2023-05-31T19:43:54.661 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : python3-pycparser-2.14-14.el8.noarch 12/32 2023-05-31T19:43:55.066 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : python3-cffi-1.11.5-5.el8.x86_64 13/32 2023-05-31T19:43:55.321 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : python3-cryptography-3.2.1-5.el8.x86_64 14/32 2023-05-31T19:43:55.509 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : python3-pyOpenSSL-19.0.0-1.el8.noarch 15/32 2023-05-31T19:43:55.776 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : python3-trustme-0.6.0-4.el8.noarch 16/32 2023-05-31T19:43:56.011 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : python3-cheroot-8.5.2-1.el8.noarch 17/32 2023-05-31T19:43:56.213 INFO:teuthology.orchestra.run.smithi099.stdout:(6/6): ceph-test-18.0.0-3487.g4a02f3f4.el8.x86_ 4.6 MB/s | 47 MB 00:10 2023-05-31T19:43:56.215 INFO:teuthology.orchestra.run.smithi099.stdout:-------------------------------------------------------------------------------- 2023-05-31T19:43:56.216 INFO:teuthology.orchestra.run.smithi099.stdout:Total 4.6 MB/s | 47 MB 00:10 2023-05-31T19:43:56.256 INFO:teuthology.orchestra.run.smithi099.stdout:Running transaction check 2023-05-31T19:43:56.277 INFO:teuthology.orchestra.run.smithi099.stdout:Transaction check succeeded. 2023-05-31T19:43:56.278 INFO:teuthology.orchestra.run.smithi099.stdout:Running transaction test 2023-05-31T19:43:56.361 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : python3-bcrypt-3.1.6-2.el8.1.x86_64 18/32 2023-05-31T19:43:56.565 INFO:teuthology.orchestra.run.smithi099.stdout:Transaction test succeeded. 2023-05-31T19:43:56.568 INFO:teuthology.orchestra.run.smithi099.stdout:Running transaction 2023-05-31T19:43:56.582 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : python3-werkzeug-0.12.2-4.el8.noarch 19/32 2023-05-31T19:43:56.804 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : python3-pytz-2017.2-9.el8.noarch 20/32 2023-05-31T19:43:57.006 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : python3-tempora-1.14.1-5.el8.noarch 21/32 2023-05-31T19:43:57.062 INFO:teuthology.orchestra.run.smithi099.stdout: Preparing : 1/1 2023-05-31T19:43:57.311 INFO:teuthology.orchestra.run.smithi099.stdout: Installing : xmlstarlet-1.6.1-20.el8.x86_64 1/6 2023-05-31T19:43:57.330 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : python3-portend-2.6-1.el8.noarch 22/32 2023-05-31T19:43:57.483 INFO:teuthology.orchestra.run.smithi099.stdout: Installing : socat-1.7.4.1-1.el8.x86_64 2/6 2023-05-31T19:43:57.572 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : python3-cherrypy-18.4.0-1.el8.noarch 23/32 2023-05-31T19:43:57.596 INFO:teuthology.orchestra.run.smithi099.stdout: Installing : oniguruma-6.8.2-2.el8.x86_64 3/6 2023-05-31T19:43:57.677 INFO:teuthology.orchestra.run.smithi099.stdout: Running scriptlet: oniguruma-6.8.2-2.el8.x86_64 3/6 2023-05-31T19:43:57.805 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : python3-markupsafe-0.23-19.el8.x86_64 24/32 2023-05-31T19:43:57.901 INFO:teuthology.orchestra.run.smithi099.stdout: Installing : jq-1.6-7.el8.x86_64 4/6 2023-05-31T19:43:58.072 INFO:teuthology.orchestra.run.smithi099.stdout: Installing : libcephsqlite-2:18.0.0-3487.g4a02f3f4.el8.x86_64 5/6 2023-05-31T19:43:58.167 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : python3-mako-1.0.6-14.el8.noarch 25/32 2023-05-31T19:43:58.490 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : python3-pecan-1.3.2-9.el8.noarch 26/32 2023-05-31T19:43:58.860 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : ceph-mgr-modules-core-2:18.0.0-3487.g4a02f3f4.el8. 27/32 2023-05-31T19:43:59.057 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : ceph-mgr-2:18.0.0-3487.g4a02f3f4.el8.x86_64 28/32 2023-05-31T19:44:00.625 INFO:teuthology.orchestra.run.smithi112.stdout: Running scriptlet: ceph-mgr-2:18.0.0-3487.g4a02f3f4.el8.x86_64 28/32 2023-05-31T19:44:00.856 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : ceph-osd-2:18.0.0-3487.g4a02f3f4.el8.x86_64 29/32 2023-05-31T19:44:01.321 INFO:teuthology.orchestra.run.smithi112.stdout: Running scriptlet: ceph-osd-2:18.0.0-3487.g4a02f3f4.el8.x86_64 29/32 2023-05-31T19:44:01.484 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : ceph-mon-2:18.0.0-3487.g4a02f3f4.el8.x86_64 30/32 2023-05-31T19:44:01.697 INFO:teuthology.orchestra.run.smithi112.stdout: Running scriptlet: ceph-mon-2:18.0.0-3487.g4a02f3f4.el8.x86_64 30/32 2023-05-31T19:44:01.851 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : ceph-mds-2:18.0.0-3487.g4a02f3f4.el8.x86_64 31/32 2023-05-31T19:44:01.868 INFO:teuthology.orchestra.run.smithi112.stdout: Running scriptlet: ceph-mds-2:18.0.0-3487.g4a02f3f4.el8.x86_64 31/32 2023-05-31T19:44:02.030 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : ceph-2:18.0.0-3487.g4a02f3f4.el8.x86_64 32/32 2023-05-31T19:44:02.433 INFO:teuthology.orchestra.run.smithi112.stdout: Running scriptlet: ceph-2:18.0.0-3487.g4a02f3f4.el8.x86_64 32/32 2023-05-31T19:44:02.433 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : ceph-2:18.0.0-3487.g4a02f3f4.el8.x86_64 1/32 2023-05-31T19:44:02.433 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : ceph-mds-2:18.0.0-3487.g4a02f3f4.el8.x86_64 2/32 2023-05-31T19:44:02.433 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : ceph-mgr-2:18.0.0-3487.g4a02f3f4.el8.x86_64 3/32 2023-05-31T19:44:02.434 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : ceph-mon-2:18.0.0-3487.g4a02f3f4.el8.x86_64 4/32 2023-05-31T19:44:02.434 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : ceph-osd-2:18.0.0-3487.g4a02f3f4.el8.x86_64 5/32 2023-05-31T19:44:02.434 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : ceph-mgr-modules-core-2:18.0.0-3487.g4a02f3f4.el8. 6/32 2023-05-31T19:44:02.434 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : python3-mako-1.0.6-14.el8.noarch 7/32 2023-05-31T19:44:02.434 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : python3-markupsafe-0.23-19.el8.x86_64 8/32 2023-05-31T19:44:02.434 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : python3-pyOpenSSL-19.0.0-1.el8.noarch 9/32 2023-05-31T19:44:02.435 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : python3-pytz-2017.2-9.el8.noarch 10/32 2023-05-31T19:44:02.435 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : python3-werkzeug-0.12.2-4.el8.noarch 11/32 2023-05-31T19:44:02.435 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : python3-cffi-1.11.5-5.el8.x86_64 12/32 2023-05-31T19:44:02.435 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : python3-cryptography-3.2.1-5.el8.x86_64 13/32 2023-05-31T19:44:02.435 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : python3-pycparser-2.14-14.el8.noarch 14/32 2023-05-31T19:44:02.435 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : python3-bcrypt-3.1.6-2.el8.1.x86_64 15/32 2023-05-31T19:44:02.435 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : python3-beautifulsoup4-4.6.3-2.el8.1.noarch 16/32 2023-05-31T19:44:02.435 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : python3-cheroot-8.5.2-1.el8.noarch 17/32 2023-05-31T19:44:02.436 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : python3-cherrypy-18.4.0-1.el8.noarch 18/32 2023-05-31T19:44:02.436 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : python3-jaraco-6.2-6.el8.noarch 19/32 2023-05-31T19:44:02.436 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : python3-jaraco-functools-2.0-4.el8.noarch 20/32 2023-05-31T19:44:02.436 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : python3-logutils-0.3.5-11.el8.noarch 21/32 2023-05-31T19:44:02.436 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : python3-more-itertools-7.2.0-3.el8.noarch 22/32 2023-05-31T19:44:02.436 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : python3-pecan-1.3.2-9.el8.noarch 23/32 2023-05-31T19:44:02.436 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : python3-portend-2.6-1.el8.noarch 24/32 2023-05-31T19:44:02.437 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : python3-simplegeneric-0.8.1-17.el8.noarch 25/32 2023-05-31T19:44:02.437 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : python3-singledispatch-3.4.0.3-18.el8.noarch 26/32 2023-05-31T19:44:02.437 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : python3-tempora-1.14.1-5.el8.noarch 27/32 2023-05-31T19:44:02.437 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : python3-trustme-0.6.0-4.el8.noarch 28/32 2023-05-31T19:44:02.438 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : python3-waitress-1.4.3-1.el8.noarch 29/32 2023-05-31T19:44:02.438 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : python3-webob-1.8.5-1.el8.1.noarch 30/32 2023-05-31T19:44:02.438 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : python3-webtest-2.0.33-1.el8.noarch 31/32 2023-05-31T19:44:02.584 INFO:teuthology.orchestra.run.smithi099.stdout: Running scriptlet: libcephsqlite-2:18.0.0-3487.g4a02f3f4.el8.x86_64 5/6 2023-05-31T19:44:02.836 INFO:teuthology.orchestra.run.smithi099.stdout: Installing : ceph-test-2:18.0.0-3487.g4a02f3f4.el8.x86_64 6/6 2023-05-31T19:44:03.121 INFO:teuthology.orchestra.run.smithi099.stdout: Running scriptlet: ceph-test-2:18.0.0-3487.g4a02f3f4.el8.x86_64 6/6 2023-05-31T19:44:03.121 INFO:teuthology.orchestra.run.smithi099.stdout: Verifying : ceph-test-2:18.0.0-3487.g4a02f3f4.el8.x86_64 1/6 2023-05-31T19:44:03.121 INFO:teuthology.orchestra.run.smithi099.stdout: Verifying : libcephsqlite-2:18.0.0-3487.g4a02f3f4.el8.x86_64 2/6 2023-05-31T19:44:03.121 INFO:teuthology.orchestra.run.smithi099.stdout: Verifying : jq-1.6-7.el8.x86_64 3/6 2023-05-31T19:44:03.121 INFO:teuthology.orchestra.run.smithi099.stdout: Verifying : oniguruma-6.8.2-2.el8.x86_64 4/6 2023-05-31T19:44:03.122 INFO:teuthology.orchestra.run.smithi099.stdout: Verifying : socat-1.7.4.1-1.el8.x86_64 5/6 2023-05-31T19:44:03.710 INFO:teuthology.orchestra.run.smithi099.stdout: Verifying : xmlstarlet-1.6.1-20.el8.x86_64 6/6 2023-05-31T19:44:03.710 INFO:teuthology.orchestra.run.smithi099.stdout: 2023-05-31T19:44:03.710 INFO:teuthology.orchestra.run.smithi099.stdout:Installed: 2023-05-31T19:44:03.711 INFO:teuthology.orchestra.run.smithi099.stdout: ceph-test-2:18.0.0-3487.g4a02f3f4.el8.x86_64 2023-05-31T19:44:03.711 INFO:teuthology.orchestra.run.smithi099.stdout: jq-1.6-7.el8.x86_64 2023-05-31T19:44:03.711 INFO:teuthology.orchestra.run.smithi099.stdout: libcephsqlite-2:18.0.0-3487.g4a02f3f4.el8.x86_64 2023-05-31T19:44:03.712 INFO:teuthology.orchestra.run.smithi099.stdout: oniguruma-6.8.2-2.el8.x86_64 2023-05-31T19:44:03.712 INFO:teuthology.orchestra.run.smithi099.stdout: socat-1.7.4.1-1.el8.x86_64 2023-05-31T19:44:03.712 INFO:teuthology.orchestra.run.smithi099.stdout: xmlstarlet-1.6.1-20.el8.x86_64 2023-05-31T19:44:03.713 INFO:teuthology.orchestra.run.smithi099.stdout: 2023-05-31T19:44:03.713 INFO:teuthology.orchestra.run.smithi099.stdout:Complete! 2023-05-31T19:44:03.728 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : python3-zc-lockfile-2.0-2.el8.noarch 32/32 2023-05-31T19:44:03.728 INFO:teuthology.orchestra.run.smithi112.stdout: 2023-05-31T19:44:03.729 INFO:teuthology.orchestra.run.smithi112.stdout:Installed: 2023-05-31T19:44:03.729 INFO:teuthology.orchestra.run.smithi112.stdout: ceph-2:18.0.0-3487.g4a02f3f4.el8.x86_64 2023-05-31T19:44:03.729 INFO:teuthology.orchestra.run.smithi112.stdout: ceph-mds-2:18.0.0-3487.g4a02f3f4.el8.x86_64 2023-05-31T19:44:03.729 INFO:teuthology.orchestra.run.smithi112.stdout: ceph-mgr-2:18.0.0-3487.g4a02f3f4.el8.x86_64 2023-05-31T19:44:03.729 INFO:teuthology.orchestra.run.smithi112.stdout: ceph-mgr-modules-core-2:18.0.0-3487.g4a02f3f4.el8.noarch 2023-05-31T19:44:03.730 INFO:teuthology.orchestra.run.smithi112.stdout: ceph-mon-2:18.0.0-3487.g4a02f3f4.el8.x86_64 2023-05-31T19:44:03.730 INFO:teuthology.orchestra.run.smithi112.stdout: ceph-osd-2:18.0.0-3487.g4a02f3f4.el8.x86_64 2023-05-31T19:44:03.730 INFO:teuthology.orchestra.run.smithi112.stdout: python3-bcrypt-3.1.6-2.el8.1.x86_64 2023-05-31T19:44:03.730 INFO:teuthology.orchestra.run.smithi112.stdout: python3-beautifulsoup4-4.6.3-2.el8.1.noarch 2023-05-31T19:44:03.730 INFO:teuthology.orchestra.run.smithi112.stdout: python3-cffi-1.11.5-5.el8.x86_64 2023-05-31T19:44:03.730 INFO:teuthology.orchestra.run.smithi112.stdout: python3-cheroot-8.5.2-1.el8.noarch 2023-05-31T19:44:03.731 INFO:teuthology.orchestra.run.smithi112.stdout: python3-cherrypy-18.4.0-1.el8.noarch 2023-05-31T19:44:03.731 INFO:teuthology.orchestra.run.smithi112.stdout: python3-cryptography-3.2.1-5.el8.x86_64 2023-05-31T19:44:03.731 INFO:teuthology.orchestra.run.smithi112.stdout: python3-jaraco-6.2-6.el8.noarch 2023-05-31T19:44:03.731 INFO:teuthology.orchestra.run.smithi112.stdout: python3-jaraco-functools-2.0-4.el8.noarch 2023-05-31T19:44:03.731 INFO:teuthology.orchestra.run.smithi112.stdout: python3-logutils-0.3.5-11.el8.noarch 2023-05-31T19:44:03.732 INFO:teuthology.orchestra.run.smithi112.stdout: python3-mako-1.0.6-14.el8.noarch 2023-05-31T19:44:03.732 INFO:teuthology.orchestra.run.smithi112.stdout: python3-markupsafe-0.23-19.el8.x86_64 2023-05-31T19:44:03.732 INFO:teuthology.orchestra.run.smithi112.stdout: python3-more-itertools-7.2.0-3.el8.noarch 2023-05-31T19:44:03.732 INFO:teuthology.orchestra.run.smithi112.stdout: python3-pecan-1.3.2-9.el8.noarch 2023-05-31T19:44:03.733 INFO:teuthology.orchestra.run.smithi112.stdout: python3-portend-2.6-1.el8.noarch 2023-05-31T19:44:03.733 INFO:teuthology.orchestra.run.smithi112.stdout: python3-pyOpenSSL-19.0.0-1.el8.noarch 2023-05-31T19:44:03.733 INFO:teuthology.orchestra.run.smithi112.stdout: python3-pycparser-2.14-14.el8.noarch 2023-05-31T19:44:03.733 INFO:teuthology.orchestra.run.smithi112.stdout: python3-pytz-2017.2-9.el8.noarch 2023-05-31T19:44:03.733 INFO:teuthology.orchestra.run.smithi112.stdout: python3-simplegeneric-0.8.1-17.el8.noarch 2023-05-31T19:44:03.733 INFO:teuthology.orchestra.run.smithi112.stdout: python3-singledispatch-3.4.0.3-18.el8.noarch 2023-05-31T19:44:03.733 INFO:teuthology.orchestra.run.smithi112.stdout: python3-tempora-1.14.1-5.el8.noarch 2023-05-31T19:44:03.734 INFO:teuthology.orchestra.run.smithi112.stdout: python3-trustme-0.6.0-4.el8.noarch 2023-05-31T19:44:03.734 INFO:teuthology.orchestra.run.smithi112.stdout: python3-waitress-1.4.3-1.el8.noarch 2023-05-31T19:44:03.734 INFO:teuthology.orchestra.run.smithi112.stdout: python3-webob-1.8.5-1.el8.1.noarch 2023-05-31T19:44:03.734 INFO:teuthology.orchestra.run.smithi112.stdout: python3-webtest-2.0.33-1.el8.noarch 2023-05-31T19:44:03.734 INFO:teuthology.orchestra.run.smithi112.stdout: python3-werkzeug-0.12.2-4.el8.noarch 2023-05-31T19:44:03.734 INFO:teuthology.orchestra.run.smithi112.stdout: python3-zc-lockfile-2.0-2.el8.noarch 2023-05-31T19:44:03.734 INFO:teuthology.orchestra.run.smithi112.stdout: 2023-05-31T19:44:03.735 INFO:teuthology.orchestra.run.smithi112.stdout:Complete! 2023-05-31T19:44:03.911 DEBUG:teuthology.orchestra.run.smithi099:> sudo yum -y install ceph 2023-05-31T19:44:04.641 INFO:teuthology.orchestra.run.smithi099.stdout:Last metadata expiration check: 0:01:04 ago on Wed 31 May 2023 07:43:00 PM UTC. 2023-05-31T19:44:04.670 DEBUG:teuthology.orchestra.run.smithi112:> sudo yum -y install ceph-base 2023-05-31T19:44:05.404 INFO:teuthology.orchestra.run.smithi112.stdout:Last metadata expiration check: 0:01:19 ago on Wed 31 May 2023 07:42:46 PM UTC. 2023-05-31T19:44:05.764 INFO:teuthology.orchestra.run.smithi099.stdout:Dependencies resolved. 2023-05-31T19:44:05.767 INFO:teuthology.orchestra.run.smithi099.stdout:================================================================================ 2023-05-31T19:44:05.767 INFO:teuthology.orchestra.run.smithi099.stdout: Package Arch Version Repository Size 2023-05-31T19:44:05.767 INFO:teuthology.orchestra.run.smithi099.stdout:================================================================================ 2023-05-31T19:44:05.768 INFO:teuthology.orchestra.run.smithi099.stdout:Installing: 2023-05-31T19:44:05.768 INFO:teuthology.orchestra.run.smithi099.stdout: ceph x86_64 2:18.0.0-3487.g4a02f3f4.el8 ceph 6.4 k 2023-05-31T19:44:05.768 INFO:teuthology.orchestra.run.smithi099.stdout:Installing dependencies: 2023-05-31T19:44:05.768 INFO:teuthology.orchestra.run.smithi099.stdout: ceph-mds x86_64 2:18.0.0-3487.g4a02f3f4.el8 ceph 2.3 M 2023-05-31T19:44:05.768 INFO:teuthology.orchestra.run.smithi099.stdout: ceph-mgr x86_64 2:18.0.0-3487.g4a02f3f4.el8 ceph 1.5 M 2023-05-31T19:44:05.768 INFO:teuthology.orchestra.run.smithi099.stdout: ceph-mgr-modules-core noarch 2:18.0.0-3487.g4a02f3f4.el8 ceph-noarch 240 k 2023-05-31T19:44:05.768 INFO:teuthology.orchestra.run.smithi099.stdout: ceph-mon x86_64 2:18.0.0-3487.g4a02f3f4.el8 ceph 4.6 M 2023-05-31T19:44:05.769 INFO:teuthology.orchestra.run.smithi099.stdout: ceph-osd x86_64 2:18.0.0-3487.g4a02f3f4.el8 ceph 18 M 2023-05-31T19:44:05.769 INFO:teuthology.orchestra.run.smithi099.stdout: python3-bcrypt x86_64 3.1.6-2.el8.1 epel 44 k 2023-05-31T19:44:05.769 INFO:teuthology.orchestra.run.smithi099.stdout: python3-beautifulsoup4 noarch 4.6.3-2.el8.1 epel 185 k 2023-05-31T19:44:05.769 INFO:teuthology.orchestra.run.smithi099.stdout: python3-cffi x86_64 1.11.5-5.el8 baseos 237 k 2023-05-31T19:44:05.769 INFO:teuthology.orchestra.run.smithi099.stdout: python3-cheroot noarch 8.5.2-1.el8 epel 173 k 2023-05-31T19:44:05.769 INFO:teuthology.orchestra.run.smithi099.stdout: python3-cherrypy noarch 18.4.0-1.el8 epel 384 k 2023-05-31T19:44:05.769 INFO:teuthology.orchestra.run.smithi099.stdout: python3-cryptography x86_64 3.2.1-5.el8 baseos 559 k 2023-05-31T19:44:05.769 INFO:teuthology.orchestra.run.smithi099.stdout: python3-jaraco noarch 6.2-6.el8 epel 11 k 2023-05-31T19:44:05.770 INFO:teuthology.orchestra.run.smithi099.stdout: python3-jaraco-functools noarch 2.0-4.el8 epel 18 k 2023-05-31T19:44:05.770 INFO:teuthology.orchestra.run.smithi099.stdout: python3-logutils noarch 0.3.5-11.el8 epel 49 k 2023-05-31T19:44:05.770 INFO:teuthology.orchestra.run.smithi099.stdout: python3-mako noarch 1.0.6-14.el8 appstream 157 k 2023-05-31T19:44:05.770 INFO:teuthology.orchestra.run.smithi099.stdout: python3-markupsafe x86_64 0.23-19.el8 appstream 39 k 2023-05-31T19:44:05.770 INFO:teuthology.orchestra.run.smithi099.stdout: python3-more-itertools noarch 7.2.0-3.el8 epel 59 k 2023-05-31T19:44:05.771 INFO:teuthology.orchestra.run.smithi099.stdout: python3-pecan noarch 1.3.2-9.el8 epel 283 k 2023-05-31T19:44:05.771 INFO:teuthology.orchestra.run.smithi099.stdout: python3-portend noarch 2.6-1.el8 epel 16 k 2023-05-31T19:44:05.771 INFO:teuthology.orchestra.run.smithi099.stdout: python3-pyOpenSSL noarch 19.0.0-1.el8 appstream 103 k 2023-05-31T19:44:05.771 INFO:teuthology.orchestra.run.smithi099.stdout: python3-pycparser noarch 2.14-14.el8 baseos 109 k 2023-05-31T19:44:05.771 INFO:teuthology.orchestra.run.smithi099.stdout: python3-pytz noarch 2017.2-9.el8 appstream 54 k 2023-05-31T19:44:05.771 INFO:teuthology.orchestra.run.smithi099.stdout: python3-simplegeneric noarch 0.8.1-17.el8 epel 19 k 2023-05-31T19:44:05.771 INFO:teuthology.orchestra.run.smithi099.stdout: python3-singledispatch noarch 3.4.0.3-18.el8 epel 24 k 2023-05-31T19:44:05.772 INFO:teuthology.orchestra.run.smithi099.stdout: python3-tempora noarch 1.14.1-5.el8 epel 29 k 2023-05-31T19:44:05.772 INFO:teuthology.orchestra.run.smithi099.stdout: python3-trustme noarch 0.6.0-4.el8 epel 27 k 2023-05-31T19:44:05.772 INFO:teuthology.orchestra.run.smithi099.stdout: python3-waitress noarch 1.4.3-1.el8 epel 255 k 2023-05-31T19:44:05.772 INFO:teuthology.orchestra.run.smithi099.stdout: python3-webob noarch 1.8.5-1.el8.1 epel 251 k 2023-05-31T19:44:05.772 INFO:teuthology.orchestra.run.smithi099.stdout: python3-webtest noarch 2.0.33-1.el8 epel 85 k 2023-05-31T19:44:05.772 INFO:teuthology.orchestra.run.smithi099.stdout: python3-werkzeug noarch 0.12.2-4.el8 appstream 457 k 2023-05-31T19:44:05.772 INFO:teuthology.orchestra.run.smithi099.stdout: python3-zc-lockfile noarch 2.0-2.el8 epel 23 k 2023-05-31T19:44:05.772 INFO:teuthology.orchestra.run.smithi099.stdout: 2023-05-31T19:44:05.773 INFO:teuthology.orchestra.run.smithi099.stdout:Transaction Summary 2023-05-31T19:44:05.773 INFO:teuthology.orchestra.run.smithi099.stdout:================================================================================ 2023-05-31T19:44:05.773 INFO:teuthology.orchestra.run.smithi099.stdout:Install 32 Packages 2023-05-31T19:44:05.773 INFO:teuthology.orchestra.run.smithi099.stdout: 2023-05-31T19:44:05.773 INFO:teuthology.orchestra.run.smithi099.stdout:Total download size: 30 M 2023-05-31T19:44:05.774 INFO:teuthology.orchestra.run.smithi099.stdout:Installed size: 119 M 2023-05-31T19:44:05.774 INFO:teuthology.orchestra.run.smithi099.stdout:Downloading Packages: 2023-05-31T19:44:06.224 INFO:teuthology.orchestra.run.smithi099.stdout:(1/32): ceph-18.0.0-3487.g4a02f3f4.el8.x86_64.r 36 kB/s | 6.4 kB 00:00 2023-05-31T19:44:06.324 INFO:teuthology.orchestra.run.smithi099.stdout:(2/32): ceph-mgr-18.0.0-3487.g4a02f3f4.el8.x86_ 5.6 MB/s | 1.5 MB 00:00 2023-05-31T19:44:06.378 INFO:teuthology.orchestra.run.smithi112.stdout:Package ceph-base-2:18.0.0-3487.g4a02f3f4.el8.x86_64 is already installed. 2023-05-31T19:44:06.482 INFO:teuthology.orchestra.run.smithi099.stdout:(3/32): ceph-mds-18.0.0-3487.g4a02f3f4.el8.x86_ 5.2 MB/s | 2.3 MB 00:00 2023-05-31T19:44:06.517 INFO:teuthology.orchestra.run.smithi112.stdout:Dependencies resolved. 2023-05-31T19:44:06.518 INFO:teuthology.orchestra.run.smithi112.stdout:Nothing to do. 2023-05-31T19:44:06.518 INFO:teuthology.orchestra.run.smithi112.stdout:Complete! 2023-05-31T19:44:06.558 INFO:teuthology.orchestra.run.smithi099.stdout:(4/32): ceph-mgr-modules-core-18.0.0-3487.g4a02 3.1 MB/s | 240 kB 00:00 2023-05-31T19:44:06.606 DEBUG:teuthology.orchestra.run.smithi112:> sudo yum -y install cephadm 2023-05-31T19:44:06.658 INFO:teuthology.orchestra.run.smithi099.stdout:(5/32): python3-mako-1.0.6-14.el8.noarch.rpm 1.5 MB/s | 157 kB 00:00 2023-05-31T19:44:06.691 INFO:teuthology.orchestra.run.smithi099.stdout:(6/32): python3-markupsafe-0.23-19.el8.x86_64.r 1.1 MB/s | 39 kB 00:00 2023-05-31T19:44:06.733 INFO:teuthology.orchestra.run.smithi099.stdout:(7/32): python3-pyOpenSSL-19.0.0-1.el8.noarch.r 2.4 MB/s | 103 kB 00:00 2023-05-31T19:44:06.775 INFO:teuthology.orchestra.run.smithi099.stdout:(8/32): python3-pytz-2017.2-9.el8.noarch.rpm 1.3 MB/s | 54 kB 00:00 2023-05-31T19:44:06.834 INFO:teuthology.orchestra.run.smithi099.stdout:(9/32): python3-werkzeug-0.12.2-4.el8.noarch.rp 7.7 MB/s | 457 kB 00:00 2023-05-31T19:44:07.084 INFO:teuthology.orchestra.run.smithi099.stdout:(10/32): python3-cffi-1.11.5-5.el8.x86_64.rpm 951 kB/s | 237 kB 00:00 2023-05-31T19:44:07.260 INFO:teuthology.orchestra.run.smithi099.stdout:(11/32): ceph-osd-18.0.0-3487.g4a02f3f4.el8.x86 19 MB/s | 18 MB 00:00 2023-05-31T19:44:07.328 INFO:teuthology.orchestra.run.smithi112.stdout:Last metadata expiration check: 0:01:21 ago on Wed 31 May 2023 07:42:46 PM UTC. 2023-05-31T19:44:07.468 INFO:teuthology.orchestra.run.smithi099.stdout:(12/32): python3-pycparser-2.14-14.el8.noarch.r 524 kB/s | 109 kB 00:00 2023-05-31T19:44:07.527 INFO:teuthology.orchestra.run.smithi099.stdout:(13/32): python3-cryptography-3.2.1-5.el8.x86_6 1.2 MB/s | 559 kB 00:00 2023-05-31T19:44:07.627 INFO:teuthology.orchestra.run.smithi099.stdout:(14/32): ceph-mon-18.0.0-3487.g4a02f3f4.el8.x86 3.3 MB/s | 4.6 MB 00:01 2023-05-31T19:44:07.652 INFO:teuthology.orchestra.run.smithi099.stdout:(15/32): python3-bcrypt-3.1.6-2.el8.1.x86_64.rp 237 kB/s | 44 kB 00:00 2023-05-31T19:44:07.686 INFO:teuthology.orchestra.run.smithi099.stdout:(16/32): python3-beautifulsoup4-4.6.3-2.el8.1.n 1.1 MB/s | 185 kB 00:00 2023-05-31T19:44:07.728 INFO:teuthology.orchestra.run.smithi099.stdout:(17/32): python3-jaraco-6.2-6.el8.noarch.rpm 267 kB/s | 11 kB 00:00 2023-05-31T19:44:07.770 INFO:teuthology.orchestra.run.smithi099.stdout:(18/32): python3-jaraco-functools-2.0-4.el8.noa 424 kB/s | 18 kB 00:00 2023-05-31T19:44:07.787 INFO:teuthology.orchestra.run.smithi099.stdout:(19/32): python3-cheroot-8.5.2-1.el8.noarch.rpm 1.1 MB/s | 173 kB 00:00 2023-05-31T19:44:07.812 INFO:teuthology.orchestra.run.smithi099.stdout:(20/32): python3-cherrypy-18.4.0-1.el8.noarch.r 2.4 MB/s | 384 kB 00:00 2023-05-31T19:44:07.837 INFO:teuthology.orchestra.run.smithi099.stdout:(21/32): python3-logutils-0.3.5-11.el8.noarch.r 725 kB/s | 49 kB 00:00 2023-05-31T19:44:07.863 INFO:teuthology.orchestra.run.smithi099.stdout:(22/32): python3-more-itertools-7.2.0-3.el8.noa 777 kB/s | 59 kB 00:00 2023-05-31T19:44:07.880 INFO:teuthology.orchestra.run.smithi099.stdout:(23/32): python3-portend-2.6-1.el8.noarch.rpm 370 kB/s | 16 kB 00:00 2023-05-31T19:44:07.905 INFO:teuthology.orchestra.run.smithi099.stdout:(24/32): python3-pecan-1.3.2-9.el8.noarch.rpm 3.0 MB/s | 283 kB 00:00 2023-05-31T19:44:07.923 INFO:teuthology.orchestra.run.smithi099.stdout:(25/32): python3-simplegeneric-0.8.1-17.el8.noa 319 kB/s | 19 kB 00:00 2023-05-31T19:44:07.939 INFO:teuthology.orchestra.run.smithi099.stdout:(26/32): python3-singledispatch-3.4.0.3-18.el8. 406 kB/s | 24 kB 00:00 2023-05-31T19:44:07.956 INFO:teuthology.orchestra.run.smithi099.stdout:(27/32): python3-tempora-1.14.1-5.el8.noarch.rp 581 kB/s | 29 kB 00:00 2023-05-31T19:44:08.023 INFO:teuthology.orchestra.run.smithi099.stdout:(28/32): python3-trustme-0.6.0-4.el8.noarch.rpm 330 kB/s | 27 kB 00:00 2023-05-31T19:44:08.048 INFO:teuthology.orchestra.run.smithi099.stdout:(29/32): python3-waitress-1.4.3-1.el8.noarch.rp 2.3 MB/s | 255 kB 00:00 2023-05-31T19:44:08.073 INFO:teuthology.orchestra.run.smithi099.stdout:(30/32): python3-webob-1.8.5-1.el8.1.noarch.rpm 2.1 MB/s | 251 kB 00:00 2023-05-31T19:44:08.098 INFO:teuthology.orchestra.run.smithi099.stdout:(31/32): python3-zc-lockfile-2.0-2.el8.noarch.r 459 kB/s | 23 kB 00:00 2023-05-31T19:44:08.123 INFO:teuthology.orchestra.run.smithi099.stdout:(32/32): python3-webtest-2.0.33-1.el8.noarch.rp 850 kB/s | 85 kB 00:00 2023-05-31T19:44:08.126 INFO:teuthology.orchestra.run.smithi099.stdout:-------------------------------------------------------------------------------- 2023-05-31T19:44:08.126 INFO:teuthology.orchestra.run.smithi099.stdout:Total 13 MB/s | 30 MB 00:02 2023-05-31T19:44:08.209 INFO:teuthology.orchestra.run.smithi099.stdout:Running transaction check 2023-05-31T19:44:08.247 INFO:teuthology.orchestra.run.smithi099.stdout:Transaction check succeeded. 2023-05-31T19:44:08.247 INFO:teuthology.orchestra.run.smithi099.stdout:Running transaction test 2023-05-31T19:44:08.446 INFO:teuthology.orchestra.run.smithi112.stdout:Dependencies resolved. 2023-05-31T19:44:08.446 INFO:teuthology.orchestra.run.smithi112.stdout:================================================================================ 2023-05-31T19:44:08.447 INFO:teuthology.orchestra.run.smithi112.stdout: Package Arch Version Repository Size 2023-05-31T19:44:08.447 INFO:teuthology.orchestra.run.smithi112.stdout:================================================================================ 2023-05-31T19:44:08.447 INFO:teuthology.orchestra.run.smithi112.stdout:Installing: 2023-05-31T19:44:08.447 INFO:teuthology.orchestra.run.smithi112.stdout: cephadm noarch 2:18.0.0-3487.g4a02f3f4.el8 ceph-noarch 160 k 2023-05-31T19:44:08.447 INFO:teuthology.orchestra.run.smithi112.stdout: 2023-05-31T19:44:08.447 INFO:teuthology.orchestra.run.smithi112.stdout:Transaction Summary 2023-05-31T19:44:08.447 INFO:teuthology.orchestra.run.smithi112.stdout:================================================================================ 2023-05-31T19:44:08.448 INFO:teuthology.orchestra.run.smithi112.stdout:Install 1 Package 2023-05-31T19:44:08.448 INFO:teuthology.orchestra.run.smithi112.stdout: 2023-05-31T19:44:08.448 INFO:teuthology.orchestra.run.smithi112.stdout:Total download size: 160 k 2023-05-31T19:44:08.448 INFO:teuthology.orchestra.run.smithi112.stdout:Installed size: 639 k 2023-05-31T19:44:08.449 INFO:teuthology.orchestra.run.smithi112.stdout:Downloading Packages: 2023-05-31T19:44:08.549 INFO:teuthology.orchestra.run.smithi099.stdout:Transaction test succeeded. 2023-05-31T19:44:08.552 INFO:teuthology.orchestra.run.smithi099.stdout:Running transaction 2023-05-31T19:44:08.691 INFO:teuthology.orchestra.run.smithi112.stdout:cephadm-18.0.0-3487.g4a02f3f4.el8.noarch.rpm 658 kB/s | 160 kB 00:00 2023-05-31T19:44:08.692 INFO:teuthology.orchestra.run.smithi112.stdout:-------------------------------------------------------------------------------- 2023-05-31T19:44:08.692 INFO:teuthology.orchestra.run.smithi112.stdout:Total 654 kB/s | 160 kB 00:00 2023-05-31T19:44:08.693 INFO:teuthology.orchestra.run.smithi112.stdout:Running transaction check 2023-05-31T19:44:08.700 INFO:teuthology.orchestra.run.smithi112.stdout:Transaction check succeeded. 2023-05-31T19:44:08.700 INFO:teuthology.orchestra.run.smithi112.stdout:Running transaction test 2023-05-31T19:44:08.702 INFO:teuthology.orchestra.run.smithi112.stdout:Transaction test succeeded. 2023-05-31T19:44:08.705 INFO:teuthology.orchestra.run.smithi112.stdout:Running transaction 2023-05-31T19:44:08.749 INFO:teuthology.orchestra.run.smithi112.stdout: Preparing : 1/1 2023-05-31T19:44:09.306 INFO:teuthology.orchestra.run.smithi112.stdout: Running scriptlet: cephadm-2:18.0.0-3487.g4a02f3f4.el8.noarch 1/1 2023-05-31T19:44:09.484 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : cephadm-2:18.0.0-3487.g4a02f3f4.el8.noarch 1/1 2023-05-31T19:44:09.665 INFO:teuthology.orchestra.run.smithi099.stdout: Preparing : 1/1 2023-05-31T19:44:09.708 INFO:teuthology.orchestra.run.smithi112.stdout: Running scriptlet: cephadm-2:18.0.0-3487.g4a02f3f4.el8.noarch 1/1 2023-05-31T19:44:09.925 INFO:teuthology.orchestra.run.smithi099.stdout: Installing : python3-more-itertools-7.2.0-3.el8.noarch 1/32 2023-05-31T19:44:10.177 INFO:teuthology.orchestra.run.smithi099.stdout: Installing : python3-webob-1.8.5-1.el8.1.noarch 2/32 2023-05-31T19:44:10.419 INFO:teuthology.orchestra.run.smithi099.stdout: Installing : python3-zc-lockfile-2.0-2.el8.noarch 3/32 2023-05-31T19:44:10.756 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : cephadm-2:18.0.0-3487.g4a02f3f4.el8.noarch 1/1 2023-05-31T19:44:10.757 INFO:teuthology.orchestra.run.smithi112.stdout: 2023-05-31T19:44:10.757 INFO:teuthology.orchestra.run.smithi112.stdout:Installed: 2023-05-31T19:44:10.757 INFO:teuthology.orchestra.run.smithi112.stdout: cephadm-2:18.0.0-3487.g4a02f3f4.el8.noarch 2023-05-31T19:44:10.757 INFO:teuthology.orchestra.run.smithi112.stdout: 2023-05-31T19:44:10.757 INFO:teuthology.orchestra.run.smithi112.stdout:Complete! 2023-05-31T19:44:10.951 INFO:teuthology.orchestra.run.smithi099.stdout: Installing : python3-waitress-1.4.3-1.el8.noarch 4/32 2023-05-31T19:44:11.132 INFO:teuthology.orchestra.run.smithi099.stdout: Installing : python3-singledispatch-3.4.0.3-18.el8.noarch 5/32 2023-05-31T19:44:11.330 INFO:teuthology.orchestra.run.smithi099.stdout: Installing : python3-simplegeneric-0.8.1-17.el8.noarch 6/32 2023-05-31T19:44:11.530 INFO:teuthology.orchestra.run.smithi099.stdout: Installing : python3-logutils-0.3.5-11.el8.noarch 7/32 2023-05-31T19:44:11.708 INFO:teuthology.orchestra.run.smithi099.stdout: Installing : python3-jaraco-6.2-6.el8.noarch 8/32 2023-05-31T19:44:11.874 DEBUG:teuthology.orchestra.run.smithi112:> sudo yum -y install ceph-immutable-object-cache 2023-05-31T19:44:11.943 INFO:teuthology.orchestra.run.smithi099.stdout: Installing : python3-jaraco-functools-2.0-4.el8.noarch 9/32 2023-05-31T19:44:12.179 INFO:teuthology.orchestra.run.smithi099.stdout: Installing : python3-beautifulsoup4-4.6.3-2.el8.1.noarch 10/32 2023-05-31T19:44:12.393 INFO:teuthology.orchestra.run.smithi099.stdout: Installing : python3-webtest-2.0.33-1.el8.noarch 11/32 2023-05-31T19:44:12.615 INFO:teuthology.orchestra.run.smithi112.stdout:Last metadata expiration check: 0:01:26 ago on Wed 31 May 2023 07:42:46 PM UTC. 2023-05-31T19:44:12.634 INFO:teuthology.orchestra.run.smithi099.stdout: Installing : python3-pycparser-2.14-14.el8.noarch 12/32 2023-05-31T19:44:13.029 INFO:teuthology.orchestra.run.smithi099.stdout: Installing : python3-cffi-1.11.5-5.el8.x86_64 13/32 2023-05-31T19:44:13.304 INFO:teuthology.orchestra.run.smithi099.stdout: Installing : python3-cryptography-3.2.1-5.el8.x86_64 14/32 2023-05-31T19:44:13.484 INFO:teuthology.orchestra.run.smithi099.stdout: Installing : python3-pyOpenSSL-19.0.0-1.el8.noarch 15/32 2023-05-31T19:44:13.701 INFO:teuthology.orchestra.run.smithi099.stdout: Installing : python3-trustme-0.6.0-4.el8.noarch 16/32 2023-05-31T19:44:13.728 INFO:teuthology.orchestra.run.smithi112.stdout:Dependencies resolved. 2023-05-31T19:44:13.729 INFO:teuthology.orchestra.run.smithi112.stdout:================================================================================ 2023-05-31T19:44:13.730 INFO:teuthology.orchestra.run.smithi112.stdout: Package Arch Version Repo Size 2023-05-31T19:44:13.730 INFO:teuthology.orchestra.run.smithi112.stdout:================================================================================ 2023-05-31T19:44:13.731 INFO:teuthology.orchestra.run.smithi112.stdout:Installing: 2023-05-31T19:44:13.731 INFO:teuthology.orchestra.run.smithi112.stdout: ceph-immutable-object-cache x86_64 2:18.0.0-3487.g4a02f3f4.el8 ceph 165 k 2023-05-31T19:44:13.731 INFO:teuthology.orchestra.run.smithi112.stdout: 2023-05-31T19:44:13.732 INFO:teuthology.orchestra.run.smithi112.stdout:Transaction Summary 2023-05-31T19:44:13.732 INFO:teuthology.orchestra.run.smithi112.stdout:================================================================================ 2023-05-31T19:44:13.732 INFO:teuthology.orchestra.run.smithi112.stdout:Install 1 Package 2023-05-31T19:44:13.733 INFO:teuthology.orchestra.run.smithi112.stdout: 2023-05-31T19:44:13.733 INFO:teuthology.orchestra.run.smithi112.stdout:Total download size: 165 k 2023-05-31T19:44:13.733 INFO:teuthology.orchestra.run.smithi112.stdout:Installed size: 503 k 2023-05-31T19:44:13.733 INFO:teuthology.orchestra.run.smithi112.stdout:Downloading Packages: 2023-05-31T19:44:13.935 INFO:teuthology.orchestra.run.smithi099.stdout: Installing : python3-cheroot-8.5.2-1.el8.noarch 17/32 2023-05-31T19:44:14.038 INFO:teuthology.orchestra.run.smithi112.stdout:ceph-immutable-object-cache-18.0.0-3487.g4a02f3 537 kB/s | 165 kB 00:00 2023-05-31T19:44:14.038 INFO:teuthology.orchestra.run.smithi112.stdout:-------------------------------------------------------------------------------- 2023-05-31T19:44:14.038 INFO:teuthology.orchestra.run.smithi112.stdout:Total 535 kB/s | 165 kB 00:00 2023-05-31T19:44:14.039 INFO:teuthology.orchestra.run.smithi112.stdout:Running transaction check 2023-05-31T19:44:14.050 INFO:teuthology.orchestra.run.smithi112.stdout:Transaction check succeeded. 2023-05-31T19:44:14.051 INFO:teuthology.orchestra.run.smithi112.stdout:Running transaction test 2023-05-31T19:44:14.097 INFO:teuthology.orchestra.run.smithi112.stdout:Transaction test succeeded. 2023-05-31T19:44:14.100 INFO:teuthology.orchestra.run.smithi112.stdout:Running transaction 2023-05-31T19:44:14.212 INFO:teuthology.orchestra.run.smithi112.stdout: Preparing : 1/1 2023-05-31T19:44:14.228 INFO:teuthology.orchestra.run.smithi099.stdout: Installing : python3-bcrypt-3.1.6-2.el8.1.x86_64 18/32 2023-05-31T19:44:14.399 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : ceph-immutable-object-cache-2:18.0.0-3487.g4a02f3f4. 1/1 2023-05-31T19:44:14.440 INFO:teuthology.orchestra.run.smithi099.stdout: Installing : python3-werkzeug-0.12.2-4.el8.noarch 19/32 2023-05-31T19:44:14.662 INFO:teuthology.orchestra.run.smithi099.stdout: Installing : python3-pytz-2017.2-9.el8.noarch 20/32 2023-05-31T19:44:14.805 INFO:teuthology.orchestra.run.smithi112.stdout: Running scriptlet: ceph-immutable-object-cache-2:18.0.0-3487.g4a02f3f4. 1/1 2023-05-31T19:44:14.831 INFO:teuthology.orchestra.run.smithi099.stdout: Installing : python3-tempora-1.14.1-5.el8.noarch 21/32 2023-05-31T19:44:15.140 INFO:teuthology.orchestra.run.smithi099.stdout: Installing : python3-portend-2.6-1.el8.noarch 22/32 2023-05-31T19:44:15.406 INFO:teuthology.orchestra.run.smithi099.stdout: Installing : python3-cherrypy-18.4.0-1.el8.noarch 23/32 2023-05-31T19:44:15.407 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : ceph-immutable-object-cache-2:18.0.0-3487.g4a02f3f4. 1/1 2023-05-31T19:44:15.407 INFO:teuthology.orchestra.run.smithi112.stdout: 2023-05-31T19:44:15.407 INFO:teuthology.orchestra.run.smithi112.stdout:Installed: 2023-05-31T19:44:15.407 INFO:teuthology.orchestra.run.smithi112.stdout: ceph-immutable-object-cache-2:18.0.0-3487.g4a02f3f4.el8.x86_64 2023-05-31T19:44:15.407 INFO:teuthology.orchestra.run.smithi112.stdout: 2023-05-31T19:44:15.408 INFO:teuthology.orchestra.run.smithi112.stdout:Complete! 2023-05-31T19:44:15.598 DEBUG:teuthology.orchestra.run.smithi112:> sudo yum -y install ceph-mgr 2023-05-31T19:44:15.637 INFO:teuthology.orchestra.run.smithi099.stdout: Installing : python3-markupsafe-0.23-19.el8.x86_64 24/32 2023-05-31T19:44:15.930 INFO:teuthology.orchestra.run.smithi099.stdout: Installing : python3-mako-1.0.6-14.el8.noarch 25/32 2023-05-31T19:44:16.267 INFO:teuthology.orchestra.run.smithi099.stdout: Installing : python3-pecan-1.3.2-9.el8.noarch 26/32 2023-05-31T19:44:16.334 INFO:teuthology.orchestra.run.smithi112.stdout:Last metadata expiration check: 0:01:30 ago on Wed 31 May 2023 07:42:46 PM UTC. 2023-05-31T19:44:16.845 INFO:teuthology.orchestra.run.smithi099.stdout: Installing : ceph-mgr-modules-core-2:18.0.0-3487.g4a02f3f4.el8. 27/32 2023-05-31T19:44:17.056 INFO:teuthology.orchestra.run.smithi099.stdout: Installing : ceph-mgr-2:18.0.0-3487.g4a02f3f4.el8.x86_64 28/32 2023-05-31T19:44:17.298 INFO:teuthology.orchestra.run.smithi112.stdout:Package ceph-mgr-2:18.0.0-3487.g4a02f3f4.el8.x86_64 is already installed. 2023-05-31T19:44:17.437 INFO:teuthology.orchestra.run.smithi112.stdout:Dependencies resolved. 2023-05-31T19:44:17.438 INFO:teuthology.orchestra.run.smithi112.stdout:Nothing to do. 2023-05-31T19:44:17.439 INFO:teuthology.orchestra.run.smithi112.stdout:Complete! 2023-05-31T19:44:17.633 DEBUG:teuthology.orchestra.run.smithi112:> sudo yum -y install ceph-mgr-dashboard 2023-05-31T19:44:18.359 INFO:teuthology.orchestra.run.smithi112.stdout:Last metadata expiration check: 0:01:32 ago on Wed 31 May 2023 07:42:46 PM UTC. 2023-05-31T19:44:18.630 INFO:teuthology.orchestra.run.smithi099.stdout: Running scriptlet: ceph-mgr-2:18.0.0-3487.g4a02f3f4.el8.x86_64 28/32 2023-05-31T19:44:18.839 INFO:teuthology.orchestra.run.smithi099.stdout: Installing : ceph-osd-2:18.0.0-3487.g4a02f3f4.el8.x86_64 29/32 2023-05-31T19:44:19.301 INFO:teuthology.orchestra.run.smithi099.stdout: Running scriptlet: ceph-osd-2:18.0.0-3487.g4a02f3f4.el8.x86_64 29/32 2023-05-31T19:44:19.479 INFO:teuthology.orchestra.run.smithi112.stdout:Dependencies resolved. 2023-05-31T19:44:19.480 INFO:teuthology.orchestra.run.smithi112.stdout:================================================================================ 2023-05-31T19:44:19.480 INFO:teuthology.orchestra.run.smithi112.stdout: Package Arch Version Repository Size 2023-05-31T19:44:19.480 INFO:teuthology.orchestra.run.smithi112.stdout:================================================================================ 2023-05-31T19:44:19.480 INFO:teuthology.orchestra.run.smithi112.stdout:Installing: 2023-05-31T19:44:19.481 INFO:teuthology.orchestra.run.smithi112.stdout: ceph-mgr-dashboard noarch 2:18.0.0-3487.g4a02f3f4.el8 ceph-noarch 1.7 M 2023-05-31T19:44:19.481 INFO:teuthology.orchestra.run.smithi112.stdout:Installing dependencies: 2023-05-31T19:44:19.481 INFO:teuthology.orchestra.run.smithi112.stdout: ceph-grafana-dashboards noarch 2:18.0.0-3487.g4a02f3f4.el8 ceph-noarch 23 k 2023-05-31T19:44:19.481 INFO:teuthology.orchestra.run.smithi112.stdout: ceph-prometheus-alerts noarch 2:18.0.0-3487.g4a02f3f4.el8 ceph-noarch 15 k 2023-05-31T19:44:19.481 INFO:teuthology.orchestra.run.smithi112.stdout: python3-jwt noarch 1.6.1-2.el8 baseos 43 k 2023-05-31T19:44:19.481 INFO:teuthology.orchestra.run.smithi112.stdout: python3-repoze-lru noarch 0.7-6.el8 epel 34 k 2023-05-31T19:44:19.481 INFO:teuthology.orchestra.run.smithi112.stdout: python3-routes noarch 2.4.1-12.el8 epel 196 k 2023-05-31T19:44:19.482 INFO:teuthology.orchestra.run.smithi112.stdout: 2023-05-31T19:44:19.482 INFO:teuthology.orchestra.run.smithi112.stdout:Transaction Summary 2023-05-31T19:44:19.482 INFO:teuthology.orchestra.run.smithi112.stdout:================================================================================ 2023-05-31T19:44:19.482 INFO:teuthology.orchestra.run.smithi112.stdout:Install 6 Packages 2023-05-31T19:44:19.482 INFO:teuthology.orchestra.run.smithi112.stdout: 2023-05-31T19:44:19.483 INFO:teuthology.orchestra.run.smithi112.stdout:Total download size: 2.0 M 2023-05-31T19:44:19.483 INFO:teuthology.orchestra.run.smithi112.stdout:Installed size: 8.0 M 2023-05-31T19:44:19.483 INFO:teuthology.orchestra.run.smithi112.stdout:Downloading Packages: 2023-05-31T19:44:19.517 INFO:teuthology.orchestra.run.smithi099.stdout: Installing : ceph-mon-2:18.0.0-3487.g4a02f3f4.el8.x86_64 30/32 2023-05-31T19:44:19.730 INFO:teuthology.orchestra.run.smithi099.stdout: Running scriptlet: ceph-mon-2:18.0.0-3487.g4a02f3f4.el8.x86_64 30/32 2023-05-31T19:44:19.875 INFO:teuthology.orchestra.run.smithi099.stdout: Installing : ceph-mds-2:18.0.0-3487.g4a02f3f4.el8.x86_64 31/32 2023-05-31T19:44:19.893 INFO:teuthology.orchestra.run.smithi099.stdout: Running scriptlet: ceph-mds-2:18.0.0-3487.g4a02f3f4.el8.x86_64 31/32 2023-05-31T19:44:20.060 INFO:teuthology.orchestra.run.smithi099.stdout: Installing : ceph-2:18.0.0-3487.g4a02f3f4.el8.x86_64 32/32 2023-05-31T19:44:20.223 INFO:teuthology.orchestra.run.smithi112.stdout:(1/6): ceph-prometheus-alerts-18.0.0-3487.g4a02 46 kB/s | 15 kB 00:00 2023-05-31T19:44:20.264 INFO:teuthology.orchestra.run.smithi112.stdout:(2/6): ceph-grafana-dashboards-18.0.0-3487.g4a0 64 kB/s | 23 kB 00:00 2023-05-31T19:44:20.398 INFO:teuthology.orchestra.run.smithi112.stdout:(3/6): python3-jwt-1.6.1-2.el8.noarch.rpm 243 kB/s | 43 kB 00:00 2023-05-31T19:44:20.449 INFO:teuthology.orchestra.run.smithi112.stdout:(4/6): python3-repoze-lru-0.7-6.el8.noarch.rpm 183 kB/s | 34 kB 00:00 2023-05-31T19:44:20.473 INFO:teuthology.orchestra.run.smithi099.stdout: Running scriptlet: ceph-2:18.0.0-3487.g4a02f3f4.el8.x86_64 32/32 2023-05-31T19:44:20.473 INFO:teuthology.orchestra.run.smithi099.stdout: Verifying : ceph-2:18.0.0-3487.g4a02f3f4.el8.x86_64 1/32 2023-05-31T19:44:20.473 INFO:teuthology.orchestra.run.smithi099.stdout: Verifying : ceph-mds-2:18.0.0-3487.g4a02f3f4.el8.x86_64 2/32 2023-05-31T19:44:20.473 INFO:teuthology.orchestra.run.smithi099.stdout: Verifying : ceph-mgr-2:18.0.0-3487.g4a02f3f4.el8.x86_64 3/32 2023-05-31T19:44:20.473 INFO:teuthology.orchestra.run.smithi099.stdout: Verifying : ceph-mon-2:18.0.0-3487.g4a02f3f4.el8.x86_64 4/32 2023-05-31T19:44:20.474 INFO:teuthology.orchestra.run.smithi099.stdout: Verifying : ceph-osd-2:18.0.0-3487.g4a02f3f4.el8.x86_64 5/32 2023-05-31T19:44:20.474 INFO:teuthology.orchestra.run.smithi099.stdout: Verifying : ceph-mgr-modules-core-2:18.0.0-3487.g4a02f3f4.el8. 6/32 2023-05-31T19:44:20.474 INFO:teuthology.orchestra.run.smithi099.stdout: Verifying : python3-mako-1.0.6-14.el8.noarch 7/32 2023-05-31T19:44:20.474 INFO:teuthology.orchestra.run.smithi099.stdout: Verifying : python3-markupsafe-0.23-19.el8.x86_64 8/32 2023-05-31T19:44:20.474 INFO:teuthology.orchestra.run.smithi099.stdout: Verifying : python3-pyOpenSSL-19.0.0-1.el8.noarch 9/32 2023-05-31T19:44:20.474 INFO:teuthology.orchestra.run.smithi099.stdout: Verifying : python3-pytz-2017.2-9.el8.noarch 10/32 2023-05-31T19:44:20.474 INFO:teuthology.orchestra.run.smithi099.stdout: Verifying : python3-werkzeug-0.12.2-4.el8.noarch 11/32 2023-05-31T19:44:20.475 INFO:teuthology.orchestra.run.smithi099.stdout: Verifying : python3-cffi-1.11.5-5.el8.x86_64 12/32 2023-05-31T19:44:20.475 INFO:teuthology.orchestra.run.smithi099.stdout: Verifying : python3-cryptography-3.2.1-5.el8.x86_64 13/32 2023-05-31T19:44:20.475 INFO:teuthology.orchestra.run.smithi099.stdout: Verifying : python3-pycparser-2.14-14.el8.noarch 14/32 2023-05-31T19:44:20.475 INFO:teuthology.orchestra.run.smithi099.stdout: Verifying : python3-bcrypt-3.1.6-2.el8.1.x86_64 15/32 2023-05-31T19:44:20.475 INFO:teuthology.orchestra.run.smithi099.stdout: Verifying : python3-beautifulsoup4-4.6.3-2.el8.1.noarch 16/32 2023-05-31T19:44:20.475 INFO:teuthology.orchestra.run.smithi099.stdout: Verifying : python3-cheroot-8.5.2-1.el8.noarch 17/32 2023-05-31T19:44:20.475 INFO:teuthology.orchestra.run.smithi099.stdout: Verifying : python3-cherrypy-18.4.0-1.el8.noarch 18/32 2023-05-31T19:44:20.476 INFO:teuthology.orchestra.run.smithi099.stdout: Verifying : python3-jaraco-6.2-6.el8.noarch 19/32 2023-05-31T19:44:20.476 INFO:teuthology.orchestra.run.smithi099.stdout: Verifying : python3-jaraco-functools-2.0-4.el8.noarch 20/32 2023-05-31T19:44:20.476 INFO:teuthology.orchestra.run.smithi099.stdout: Verifying : python3-logutils-0.3.5-11.el8.noarch 21/32 2023-05-31T19:44:20.476 INFO:teuthology.orchestra.run.smithi099.stdout: Verifying : python3-more-itertools-7.2.0-3.el8.noarch 22/32 2023-05-31T19:44:20.477 INFO:teuthology.orchestra.run.smithi099.stdout: Verifying : python3-pecan-1.3.2-9.el8.noarch 23/32 2023-05-31T19:44:20.477 INFO:teuthology.orchestra.run.smithi099.stdout: Verifying : python3-portend-2.6-1.el8.noarch 24/32 2023-05-31T19:44:20.477 INFO:teuthology.orchestra.run.smithi099.stdout: Verifying : python3-simplegeneric-0.8.1-17.el8.noarch 25/32 2023-05-31T19:44:20.477 INFO:teuthology.orchestra.run.smithi099.stdout: Verifying : python3-singledispatch-3.4.0.3-18.el8.noarch 26/32 2023-05-31T19:44:20.478 INFO:teuthology.orchestra.run.smithi099.stdout: Verifying : python3-tempora-1.14.1-5.el8.noarch 27/32 2023-05-31T19:44:20.478 INFO:teuthology.orchestra.run.smithi099.stdout: Verifying : python3-trustme-0.6.0-4.el8.noarch 28/32 2023-05-31T19:44:20.478 INFO:teuthology.orchestra.run.smithi099.stdout: Verifying : python3-waitress-1.4.3-1.el8.noarch 29/32 2023-05-31T19:44:20.479 INFO:teuthology.orchestra.run.smithi099.stdout: Verifying : python3-webob-1.8.5-1.el8.1.noarch 30/32 2023-05-31T19:44:20.479 INFO:teuthology.orchestra.run.smithi099.stdout: Verifying : python3-webtest-2.0.33-1.el8.noarch 31/32 2023-05-31T19:44:20.757 INFO:teuthology.orchestra.run.smithi112.stdout:(5/6): python3-routes-2.4.1-12.el8.noarch.rpm 548 kB/s | 196 kB 00:00 2023-05-31T19:44:21.441 INFO:teuthology.orchestra.run.smithi112.stdout:(6/6): ceph-mgr-dashboard-18.0.0-3487.g4a02f3f4 1.1 MB/s | 1.7 MB 00:01 2023-05-31T19:44:21.445 INFO:teuthology.orchestra.run.smithi112.stdout:-------------------------------------------------------------------------------- 2023-05-31T19:44:21.445 INFO:teuthology.orchestra.run.smithi112.stdout:Total 1.0 MB/s | 2.0 MB 00:01 2023-05-31T19:44:21.457 INFO:teuthology.orchestra.run.smithi112.stdout:Running transaction check 2023-05-31T19:44:21.467 INFO:teuthology.orchestra.run.smithi112.stdout:Transaction check succeeded. 2023-05-31T19:44:21.467 INFO:teuthology.orchestra.run.smithi112.stdout:Running transaction test 2023-05-31T19:44:21.550 INFO:teuthology.orchestra.run.smithi112.stdout:Transaction test succeeded. 2023-05-31T19:44:21.553 INFO:teuthology.orchestra.run.smithi112.stdout:Running transaction 2023-05-31T19:44:21.559 INFO:teuthology.orchestra.run.smithi099.stdout: Verifying : python3-zc-lockfile-2.0-2.el8.noarch 32/32 2023-05-31T19:44:21.559 INFO:teuthology.orchestra.run.smithi099.stdout: 2023-05-31T19:44:21.559 INFO:teuthology.orchestra.run.smithi099.stdout:Installed: 2023-05-31T19:44:21.559 INFO:teuthology.orchestra.run.smithi099.stdout: ceph-2:18.0.0-3487.g4a02f3f4.el8.x86_64 2023-05-31T19:44:21.559 INFO:teuthology.orchestra.run.smithi099.stdout: ceph-mds-2:18.0.0-3487.g4a02f3f4.el8.x86_64 2023-05-31T19:44:21.560 INFO:teuthology.orchestra.run.smithi099.stdout: ceph-mgr-2:18.0.0-3487.g4a02f3f4.el8.x86_64 2023-05-31T19:44:21.560 INFO:teuthology.orchestra.run.smithi099.stdout: ceph-mgr-modules-core-2:18.0.0-3487.g4a02f3f4.el8.noarch 2023-05-31T19:44:21.560 INFO:teuthology.orchestra.run.smithi099.stdout: ceph-mon-2:18.0.0-3487.g4a02f3f4.el8.x86_64 2023-05-31T19:44:21.560 INFO:teuthology.orchestra.run.smithi099.stdout: ceph-osd-2:18.0.0-3487.g4a02f3f4.el8.x86_64 2023-05-31T19:44:21.560 INFO:teuthology.orchestra.run.smithi099.stdout: python3-bcrypt-3.1.6-2.el8.1.x86_64 2023-05-31T19:44:21.560 INFO:teuthology.orchestra.run.smithi099.stdout: python3-beautifulsoup4-4.6.3-2.el8.1.noarch 2023-05-31T19:44:21.560 INFO:teuthology.orchestra.run.smithi099.stdout: python3-cffi-1.11.5-5.el8.x86_64 2023-05-31T19:44:21.561 INFO:teuthology.orchestra.run.smithi099.stdout: python3-cheroot-8.5.2-1.el8.noarch 2023-05-31T19:44:21.561 INFO:teuthology.orchestra.run.smithi099.stdout: python3-cherrypy-18.4.0-1.el8.noarch 2023-05-31T19:44:21.561 INFO:teuthology.orchestra.run.smithi099.stdout: python3-cryptography-3.2.1-5.el8.x86_64 2023-05-31T19:44:21.561 INFO:teuthology.orchestra.run.smithi099.stdout: python3-jaraco-6.2-6.el8.noarch 2023-05-31T19:44:21.561 INFO:teuthology.orchestra.run.smithi099.stdout: python3-jaraco-functools-2.0-4.el8.noarch 2023-05-31T19:44:21.561 INFO:teuthology.orchestra.run.smithi099.stdout: python3-logutils-0.3.5-11.el8.noarch 2023-05-31T19:44:21.561 INFO:teuthology.orchestra.run.smithi099.stdout: python3-mako-1.0.6-14.el8.noarch 2023-05-31T19:44:21.561 INFO:teuthology.orchestra.run.smithi099.stdout: python3-markupsafe-0.23-19.el8.x86_64 2023-05-31T19:44:21.562 INFO:teuthology.orchestra.run.smithi099.stdout: python3-more-itertools-7.2.0-3.el8.noarch 2023-05-31T19:44:21.562 INFO:teuthology.orchestra.run.smithi099.stdout: python3-pecan-1.3.2-9.el8.noarch 2023-05-31T19:44:21.562 INFO:teuthology.orchestra.run.smithi099.stdout: python3-portend-2.6-1.el8.noarch 2023-05-31T19:44:21.562 INFO:teuthology.orchestra.run.smithi099.stdout: python3-pyOpenSSL-19.0.0-1.el8.noarch 2023-05-31T19:44:21.562 INFO:teuthology.orchestra.run.smithi099.stdout: python3-pycparser-2.14-14.el8.noarch 2023-05-31T19:44:21.562 INFO:teuthology.orchestra.run.smithi099.stdout: python3-pytz-2017.2-9.el8.noarch 2023-05-31T19:44:21.562 INFO:teuthology.orchestra.run.smithi099.stdout: python3-simplegeneric-0.8.1-17.el8.noarch 2023-05-31T19:44:21.563 INFO:teuthology.orchestra.run.smithi099.stdout: python3-singledispatch-3.4.0.3-18.el8.noarch 2023-05-31T19:44:21.563 INFO:teuthology.orchestra.run.smithi099.stdout: python3-tempora-1.14.1-5.el8.noarch 2023-05-31T19:44:21.563 INFO:teuthology.orchestra.run.smithi099.stdout: python3-trustme-0.6.0-4.el8.noarch 2023-05-31T19:44:21.563 INFO:teuthology.orchestra.run.smithi099.stdout: python3-waitress-1.4.3-1.el8.noarch 2023-05-31T19:44:21.563 INFO:teuthology.orchestra.run.smithi099.stdout: python3-webob-1.8.5-1.el8.1.noarch 2023-05-31T19:44:21.563 INFO:teuthology.orchestra.run.smithi099.stdout: python3-webtest-2.0.33-1.el8.noarch 2023-05-31T19:44:21.563 INFO:teuthology.orchestra.run.smithi099.stdout: python3-werkzeug-0.12.2-4.el8.noarch 2023-05-31T19:44:21.563 INFO:teuthology.orchestra.run.smithi099.stdout: python3-zc-lockfile-2.0-2.el8.noarch 2023-05-31T19:44:21.564 INFO:teuthology.orchestra.run.smithi099.stdout: 2023-05-31T19:44:21.564 INFO:teuthology.orchestra.run.smithi099.stdout:Complete! 2023-05-31T19:44:21.815 DEBUG:teuthology.orchestra.run.smithi099:> sudo yum -y install ceph-base 2023-05-31T19:44:21.839 INFO:teuthology.orchestra.run.smithi112.stdout: Preparing : 1/1 2023-05-31T19:44:22.071 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : python3-repoze-lru-0.7-6.el8.noarch 1/6 2023-05-31T19:44:22.341 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : python3-routes-2.4.1-12.el8.noarch 2/6 2023-05-31T19:44:22.542 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : python3-jwt-1.6.1-2.el8.noarch 3/6 2023-05-31T19:44:22.550 INFO:teuthology.orchestra.run.smithi099.stdout:Last metadata expiration check: 0:01:22 ago on Wed 31 May 2023 07:43:00 PM UTC. 2023-05-31T19:44:22.663 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : ceph-prometheus-alerts-2:18.0.0-3487.g4a02f3f4.el8.n 4/6 2023-05-31T19:44:23.073 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : ceph-grafana-dashboards-2:18.0.0-3487.g4a02f3f4.el8. 5/6 2023-05-31T19:44:23.303 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : ceph-mgr-dashboard-2:18.0.0-3487.g4a02f3f4.el8.noarc 6/6 2023-05-31T19:44:23.519 INFO:teuthology.orchestra.run.smithi099.stdout:Package ceph-base-2:18.0.0-3487.g4a02f3f4.el8.x86_64 is already installed. 2023-05-31T19:44:23.619 INFO:teuthology.orchestra.run.smithi112.stdout: Running scriptlet: ceph-mgr-dashboard-2:18.0.0-3487.g4a02f3f4.el8.noarc 6/6 2023-05-31T19:44:23.619 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : ceph-grafana-dashboards-2:18.0.0-3487.g4a02f3f4.el8. 1/6 2023-05-31T19:44:23.619 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : ceph-mgr-dashboard-2:18.0.0-3487.g4a02f3f4.el8.noarc 2/6 2023-05-31T19:44:23.619 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : ceph-prometheus-alerts-2:18.0.0-3487.g4a02f3f4.el8.n 3/6 2023-05-31T19:44:23.620 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : python3-jwt-1.6.1-2.el8.noarch 4/6 2023-05-31T19:44:23.620 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : python3-repoze-lru-0.7-6.el8.noarch 5/6 2023-05-31T19:44:23.659 INFO:teuthology.orchestra.run.smithi099.stdout:Dependencies resolved. 2023-05-31T19:44:23.660 INFO:teuthology.orchestra.run.smithi099.stdout:Nothing to do. 2023-05-31T19:44:23.660 INFO:teuthology.orchestra.run.smithi099.stdout:Complete! 2023-05-31T19:44:23.743 DEBUG:teuthology.orchestra.run.smithi099:> sudo yum -y install cephadm 2023-05-31T19:44:24.259 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : python3-routes-2.4.1-12.el8.noarch 6/6 2023-05-31T19:44:24.260 INFO:teuthology.orchestra.run.smithi112.stdout: 2023-05-31T19:44:24.260 INFO:teuthology.orchestra.run.smithi112.stdout:Installed: 2023-05-31T19:44:24.260 INFO:teuthology.orchestra.run.smithi112.stdout: ceph-grafana-dashboards-2:18.0.0-3487.g4a02f3f4.el8.noarch 2023-05-31T19:44:24.260 INFO:teuthology.orchestra.run.smithi112.stdout: ceph-mgr-dashboard-2:18.0.0-3487.g4a02f3f4.el8.noarch 2023-05-31T19:44:24.260 INFO:teuthology.orchestra.run.smithi112.stdout: ceph-prometheus-alerts-2:18.0.0-3487.g4a02f3f4.el8.noarch 2023-05-31T19:44:24.261 INFO:teuthology.orchestra.run.smithi112.stdout: python3-jwt-1.6.1-2.el8.noarch 2023-05-31T19:44:24.261 INFO:teuthology.orchestra.run.smithi112.stdout: python3-repoze-lru-0.7-6.el8.noarch 2023-05-31T19:44:24.261 INFO:teuthology.orchestra.run.smithi112.stdout: python3-routes-2.4.1-12.el8.noarch 2023-05-31T19:44:24.261 INFO:teuthology.orchestra.run.smithi112.stdout: 2023-05-31T19:44:24.261 INFO:teuthology.orchestra.run.smithi112.stdout:Complete! 2023-05-31T19:44:24.389 DEBUG:teuthology.orchestra.run.smithi112:> sudo yum -y install ceph-mgr-diskprediction-local 2023-05-31T19:44:24.468 INFO:teuthology.orchestra.run.smithi099.stdout:Last metadata expiration check: 0:01:24 ago on Wed 31 May 2023 07:43:00 PM UTC. 2023-05-31T19:44:25.114 INFO:teuthology.orchestra.run.smithi112.stdout:Last metadata expiration check: 0:01:39 ago on Wed 31 May 2023 07:42:46 PM UTC. 2023-05-31T19:44:25.589 INFO:teuthology.orchestra.run.smithi099.stdout:Dependencies resolved. 2023-05-31T19:44:25.590 INFO:teuthology.orchestra.run.smithi099.stdout:================================================================================ 2023-05-31T19:44:25.590 INFO:teuthology.orchestra.run.smithi099.stdout: Package Arch Version Repository Size 2023-05-31T19:44:25.591 INFO:teuthology.orchestra.run.smithi099.stdout:================================================================================ 2023-05-31T19:44:25.591 INFO:teuthology.orchestra.run.smithi099.stdout:Installing: 2023-05-31T19:44:25.591 INFO:teuthology.orchestra.run.smithi099.stdout: cephadm noarch 2:18.0.0-3487.g4a02f3f4.el8 ceph-noarch 160 k 2023-05-31T19:44:25.592 INFO:teuthology.orchestra.run.smithi099.stdout: 2023-05-31T19:44:25.592 INFO:teuthology.orchestra.run.smithi099.stdout:Transaction Summary 2023-05-31T19:44:25.592 INFO:teuthology.orchestra.run.smithi099.stdout:================================================================================ 2023-05-31T19:44:25.593 INFO:teuthology.orchestra.run.smithi099.stdout:Install 1 Package 2023-05-31T19:44:25.593 INFO:teuthology.orchestra.run.smithi099.stdout: 2023-05-31T19:44:25.593 INFO:teuthology.orchestra.run.smithi099.stdout:Total download size: 160 k 2023-05-31T19:44:25.594 INFO:teuthology.orchestra.run.smithi099.stdout:Installed size: 639 k 2023-05-31T19:44:25.594 INFO:teuthology.orchestra.run.smithi099.stdout:Downloading Packages: 2023-05-31T19:44:26.220 INFO:teuthology.orchestra.run.smithi099.stdout:cephadm-18.0.0-3487.g4a02f3f4.el8.noarch.rpm 254 kB/s | 160 kB 00:00 2023-05-31T19:44:26.221 INFO:teuthology.orchestra.run.smithi099.stdout:-------------------------------------------------------------------------------- 2023-05-31T19:44:26.221 INFO:teuthology.orchestra.run.smithi099.stdout:Total 253 kB/s | 160 kB 00:00 2023-05-31T19:44:26.222 INFO:teuthology.orchestra.run.smithi099.stdout:Running transaction check 2023-05-31T19:44:26.229 INFO:teuthology.orchestra.run.smithi099.stdout:Transaction check succeeded. 2023-05-31T19:44:26.230 INFO:teuthology.orchestra.run.smithi099.stdout:Running transaction test 2023-05-31T19:44:26.232 INFO:teuthology.orchestra.run.smithi099.stdout:Transaction test succeeded. 2023-05-31T19:44:26.233 INFO:teuthology.orchestra.run.smithi112.stdout:Dependencies resolved. 2023-05-31T19:44:26.235 INFO:teuthology.orchestra.run.smithi112.stdout:================================================================================================ 2023-05-31T19:44:26.235 INFO:teuthology.orchestra.run.smithi112.stdout: Package Arch Version Repository Size 2023-05-31T19:44:26.235 INFO:teuthology.orchestra.run.smithi112.stdout:================================================================================================ 2023-05-31T19:44:26.236 INFO:teuthology.orchestra.run.smithi112.stdout:Installing: 2023-05-31T19:44:26.236 INFO:teuthology.orchestra.run.smithi112.stdout: ceph-mgr-diskprediction-local noarch 2:18.0.0-3487.g4a02f3f4.el8 ceph-noarch 7.4 M 2023-05-31T19:44:26.236 INFO:teuthology.orchestra.run.smithi112.stdout:Installing dependencies: 2023-05-31T19:44:26.237 INFO:teuthology.orchestra.run.smithi112.stdout: libgfortran x86_64 8.5.0-19.el8 baseos 645 k 2023-05-31T19:44:26.237 INFO:teuthology.orchestra.run.smithi112.stdout: libquadmath x86_64 8.5.0-19.el8 baseos 172 k 2023-05-31T19:44:26.237 INFO:teuthology.orchestra.run.smithi112.stdout: openblas x86_64 0.3.15-4.el8 appstream 4.7 M 2023-05-31T19:44:26.237 INFO:teuthology.orchestra.run.smithi112.stdout: openblas-threads x86_64 0.3.15-4.el8 appstream 4.9 M 2023-05-31T19:44:26.237 INFO:teuthology.orchestra.run.smithi112.stdout: python3-numpy x86_64 1:1.14.3-10.el8 appstream 3.7 M 2023-05-31T19:44:26.237 INFO:teuthology.orchestra.run.smithi112.stdout: python3-numpy-f2py x86_64 1:1.14.3-10.el8 appstream 225 k 2023-05-31T19:44:26.238 INFO:teuthology.orchestra.run.smithi112.stdout: python3-scipy x86_64 1.0.0-21.module_el8.5.0+771+e5d9a225 appstream 14 M 2023-05-31T19:44:26.238 INFO:teuthology.orchestra.run.smithi112.stdout: 2023-05-31T19:44:26.238 INFO:teuthology.orchestra.run.smithi112.stdout:Transaction Summary 2023-05-31T19:44:26.238 INFO:teuthology.orchestra.run.smithi112.stdout:================================================================================================ 2023-05-31T19:44:26.238 INFO:teuthology.orchestra.run.smithi112.stdout:Install 8 Packages 2023-05-31T19:44:26.238 INFO:teuthology.orchestra.run.smithi112.stdout: 2023-05-31T19:44:26.239 INFO:teuthology.orchestra.run.smithi099.stdout:Running transaction 2023-05-31T19:44:26.239 INFO:teuthology.orchestra.run.smithi112.stdout:Total download size: 35 M 2023-05-31T19:44:26.239 INFO:teuthology.orchestra.run.smithi112.stdout:Installed size: 210 M 2023-05-31T19:44:26.240 INFO:teuthology.orchestra.run.smithi112.stdout:Downloading Packages: 2023-05-31T19:44:26.277 INFO:teuthology.orchestra.run.smithi099.stdout: Preparing : 1/1 2023-05-31T19:44:26.660 INFO:teuthology.orchestra.run.smithi112.stdout:(1/8): openblas-0.3.15-4.el8.x86_64.rpm 14 MB/s | 4.7 MB 00:00 2023-05-31T19:44:26.739 INFO:teuthology.orchestra.run.smithi112.stdout:(2/8): openblas-threads-0.3.15-4.el8.x86_64.rpm 12 MB/s | 4.9 MB 00:00 2023-05-31T19:44:26.757 INFO:teuthology.orchestra.run.smithi099.stdout: Running scriptlet: cephadm-2:18.0.0-3487.g4a02f3f4.el8.noarch 1/1 2023-05-31T19:44:26.789 INFO:teuthology.orchestra.run.smithi112.stdout:(3/8): python3-numpy-f2py-1.14.3-10.el8.x86_64. 4.4 MB/s | 225 kB 00:00 2023-05-31T19:44:26.848 INFO:teuthology.orchestra.run.smithi112.stdout:(4/8): python3-numpy-1.14.3-10.el8.x86_64.rpm 20 MB/s | 3.7 MB 00:00 2023-05-31T19:44:26.933 INFO:teuthology.orchestra.run.smithi099.stdout: Installing : cephadm-2:18.0.0-3487.g4a02f3f4.el8.noarch 1/1 2023-05-31T19:44:27.140 INFO:teuthology.orchestra.run.smithi112.stdout:(5/8): libgfortran-8.5.0-19.el8.x86_64.rpm 2.2 MB/s | 645 kB 00:00 2023-05-31T19:44:27.158 INFO:teuthology.orchestra.run.smithi099.stdout: Running scriptlet: cephadm-2:18.0.0-3487.g4a02f3f4.el8.noarch 1/1 2023-05-31T19:44:27.348 INFO:teuthology.orchestra.run.smithi112.stdout:(6/8): libquadmath-8.5.0-19.el8.x86_64.rpm 827 kB/s | 172 kB 00:00 2023-05-31T19:44:27.671 INFO:teuthology.orchestra.run.smithi099.stdout: Verifying : cephadm-2:18.0.0-3487.g4a02f3f4.el8.noarch 1/1 2023-05-31T19:44:27.672 INFO:teuthology.orchestra.run.smithi099.stdout: 2023-05-31T19:44:27.672 INFO:teuthology.orchestra.run.smithi099.stdout:Installed: 2023-05-31T19:44:27.672 INFO:teuthology.orchestra.run.smithi099.stdout: cephadm-2:18.0.0-3487.g4a02f3f4.el8.noarch 2023-05-31T19:44:27.672 INFO:teuthology.orchestra.run.smithi099.stdout: 2023-05-31T19:44:27.672 INFO:teuthology.orchestra.run.smithi099.stdout:Complete! 2023-05-31T19:44:27.857 DEBUG:teuthology.orchestra.run.smithi099:> sudo yum -y install ceph-immutable-object-cache 2023-05-31T19:44:28.013 INFO:teuthology.orchestra.run.smithi112.stdout:(7/8): python3-scipy-1.0.0-21.module_el8.5.0+77 11 MB/s | 14 MB 00:01 2023-05-31T19:44:28.592 INFO:teuthology.orchestra.run.smithi099.stdout:Last metadata expiration check: 0:01:28 ago on Wed 31 May 2023 07:43:00 PM UTC. 2023-05-31T19:44:29.724 INFO:teuthology.orchestra.run.smithi099.stdout:Dependencies resolved. 2023-05-31T19:44:29.725 INFO:teuthology.orchestra.run.smithi099.stdout:================================================================================ 2023-05-31T19:44:29.725 INFO:teuthology.orchestra.run.smithi099.stdout: Package Arch Version Repo Size 2023-05-31T19:44:29.725 INFO:teuthology.orchestra.run.smithi099.stdout:================================================================================ 2023-05-31T19:44:29.725 INFO:teuthology.orchestra.run.smithi099.stdout:Installing: 2023-05-31T19:44:29.726 INFO:teuthology.orchestra.run.smithi099.stdout: ceph-immutable-object-cache x86_64 2:18.0.0-3487.g4a02f3f4.el8 ceph 165 k 2023-05-31T19:44:29.726 INFO:teuthology.orchestra.run.smithi099.stdout: 2023-05-31T19:44:29.726 INFO:teuthology.orchestra.run.smithi099.stdout:Transaction Summary 2023-05-31T19:44:29.726 INFO:teuthology.orchestra.run.smithi099.stdout:================================================================================ 2023-05-31T19:44:29.726 INFO:teuthology.orchestra.run.smithi099.stdout:Install 1 Package 2023-05-31T19:44:29.726 INFO:teuthology.orchestra.run.smithi099.stdout: 2023-05-31T19:44:29.727 INFO:teuthology.orchestra.run.smithi099.stdout:Total download size: 165 k 2023-05-31T19:44:29.727 INFO:teuthology.orchestra.run.smithi099.stdout:Installed size: 503 k 2023-05-31T19:44:29.727 INFO:teuthology.orchestra.run.smithi099.stdout:Downloading Packages: 2023-05-31T19:44:30.257 INFO:teuthology.orchestra.run.smithi099.stdout:ceph-immutable-object-cache-18.0.0-3487.g4a02f3 311 kB/s | 165 kB 00:00 2023-05-31T19:44:30.258 INFO:teuthology.orchestra.run.smithi099.stdout:-------------------------------------------------------------------------------- 2023-05-31T19:44:30.258 INFO:teuthology.orchestra.run.smithi099.stdout:Total 311 kB/s | 165 kB 00:00 2023-05-31T19:44:30.258 INFO:teuthology.orchestra.run.smithi099.stdout:Running transaction check 2023-05-31T19:44:30.270 INFO:teuthology.orchestra.run.smithi099.stdout:Transaction check succeeded. 2023-05-31T19:44:30.270 INFO:teuthology.orchestra.run.smithi099.stdout:Running transaction test 2023-05-31T19:44:30.319 INFO:teuthology.orchestra.run.smithi099.stdout:Transaction test succeeded. 2023-05-31T19:44:30.322 INFO:teuthology.orchestra.run.smithi099.stdout:Running transaction 2023-05-31T19:44:30.433 INFO:teuthology.orchestra.run.smithi099.stdout: Preparing : 1/1 2023-05-31T19:44:30.589 INFO:teuthology.orchestra.run.smithi099.stdout: Installing : ceph-immutable-object-cache-2:18.0.0-3487.g4a02f3f4. 1/1 2023-05-31T19:44:31.005 INFO:teuthology.orchestra.run.smithi099.stdout: Running scriptlet: ceph-immutable-object-cache-2:18.0.0-3487.g4a02f3f4. 1/1 2023-05-31T19:44:31.561 INFO:teuthology.orchestra.run.smithi099.stdout: Verifying : ceph-immutable-object-cache-2:18.0.0-3487.g4a02f3f4. 1/1 2023-05-31T19:44:31.561 INFO:teuthology.orchestra.run.smithi099.stdout: 2023-05-31T19:44:31.562 INFO:teuthology.orchestra.run.smithi099.stdout:Installed: 2023-05-31T19:44:31.562 INFO:teuthology.orchestra.run.smithi099.stdout: ceph-immutable-object-cache-2:18.0.0-3487.g4a02f3f4.el8.x86_64 2023-05-31T19:44:31.562 INFO:teuthology.orchestra.run.smithi099.stdout: 2023-05-31T19:44:31.562 INFO:teuthology.orchestra.run.smithi099.stdout:Complete! 2023-05-31T19:44:31.746 DEBUG:teuthology.orchestra.run.smithi099:> sudo yum -y install ceph-mgr 2023-05-31T19:44:32.025 INFO:teuthology.orchestra.run.smithi112.stdout:(8/8): ceph-mgr-diskprediction-local-18.0.0-348 1.3 MB/s | 7.4 MB 00:05 2023-05-31T19:44:32.026 INFO:teuthology.orchestra.run.smithi112.stdout:-------------------------------------------------------------------------------- 2023-05-31T19:44:32.026 INFO:teuthology.orchestra.run.smithi112.stdout:Total 6.1 MB/s | 35 MB 00:05 2023-05-31T19:44:32.262 INFO:teuthology.orchestra.run.smithi112.stdout:Running transaction check 2023-05-31T19:44:32.280 INFO:teuthology.orchestra.run.smithi112.stdout:Transaction check succeeded. 2023-05-31T19:44:32.280 INFO:teuthology.orchestra.run.smithi112.stdout:Running transaction test 2023-05-31T19:44:32.488 INFO:teuthology.orchestra.run.smithi099.stdout:Last metadata expiration check: 0:01:32 ago on Wed 31 May 2023 07:43:00 PM UTC. 2023-05-31T19:44:32.686 INFO:teuthology.orchestra.run.smithi112.stdout:Transaction test succeeded. 2023-05-31T19:44:32.689 INFO:teuthology.orchestra.run.smithi112.stdout:Running transaction 2023-05-31T19:44:33.274 INFO:teuthology.orchestra.run.smithi112.stdout: Preparing : 1/1 2023-05-31T19:44:33.405 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : libquadmath-8.5.0-19.el8.x86_64 1/8 2023-05-31T19:44:33.474 INFO:teuthology.orchestra.run.smithi099.stdout:Package ceph-mgr-2:18.0.0-3487.g4a02f3f4.el8.x86_64 is already installed. 2023-05-31T19:44:33.614 INFO:teuthology.orchestra.run.smithi099.stdout:Dependencies resolved. 2023-05-31T19:44:33.615 INFO:teuthology.orchestra.run.smithi099.stdout:Nothing to do. 2023-05-31T19:44:33.615 INFO:teuthology.orchestra.run.smithi099.stdout:Complete! 2023-05-31T19:44:33.648 INFO:teuthology.orchestra.run.smithi112.stdout: Running scriptlet: libquadmath-8.5.0-19.el8.x86_64 1/8 2023-05-31T19:44:34.000 DEBUG:teuthology.orchestra.run.smithi099:> sudo yum -y install ceph-mgr-dashboard 2023-05-31T19:44:34.721 INFO:teuthology.orchestra.run.smithi099.stdout:Last metadata expiration check: 0:01:34 ago on Wed 31 May 2023 07:43:00 PM UTC. 2023-05-31T19:44:34.815 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : libgfortran-8.5.0-19.el8.x86_64 2/8 2023-05-31T19:44:35.428 INFO:teuthology.orchestra.run.smithi112.stdout: Running scriptlet: libgfortran-8.5.0-19.el8.x86_64 2/8 2023-05-31T19:44:35.596 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : openblas-0.3.15-4.el8.x86_64 3/8 2023-05-31T19:44:35.836 INFO:teuthology.orchestra.run.smithi099.stdout:Dependencies resolved. 2023-05-31T19:44:35.837 INFO:teuthology.orchestra.run.smithi099.stdout:================================================================================ 2023-05-31T19:44:35.838 INFO:teuthology.orchestra.run.smithi099.stdout: Package Arch Version Repository Size 2023-05-31T19:44:35.838 INFO:teuthology.orchestra.run.smithi099.stdout:================================================================================ 2023-05-31T19:44:35.838 INFO:teuthology.orchestra.run.smithi099.stdout:Installing: 2023-05-31T19:44:35.838 INFO:teuthology.orchestra.run.smithi099.stdout: ceph-mgr-dashboard noarch 2:18.0.0-3487.g4a02f3f4.el8 ceph-noarch 1.7 M 2023-05-31T19:44:35.838 INFO:teuthology.orchestra.run.smithi099.stdout:Installing dependencies: 2023-05-31T19:44:35.838 INFO:teuthology.orchestra.run.smithi099.stdout: ceph-grafana-dashboards noarch 2:18.0.0-3487.g4a02f3f4.el8 ceph-noarch 23 k 2023-05-31T19:44:35.839 INFO:teuthology.orchestra.run.smithi099.stdout: ceph-prometheus-alerts noarch 2:18.0.0-3487.g4a02f3f4.el8 ceph-noarch 15 k 2023-05-31T19:44:35.839 INFO:teuthology.orchestra.run.smithi099.stdout: python3-jwt noarch 1.6.1-2.el8 baseos 43 k 2023-05-31T19:44:35.839 INFO:teuthology.orchestra.run.smithi099.stdout: python3-repoze-lru noarch 0.7-6.el8 epel 34 k 2023-05-31T19:44:35.839 INFO:teuthology.orchestra.run.smithi099.stdout: python3-routes noarch 2.4.1-12.el8 epel 196 k 2023-05-31T19:44:35.839 INFO:teuthology.orchestra.run.smithi099.stdout: 2023-05-31T19:44:35.839 INFO:teuthology.orchestra.run.smithi099.stdout:Transaction Summary 2023-05-31T19:44:35.839 INFO:teuthology.orchestra.run.smithi099.stdout:================================================================================ 2023-05-31T19:44:35.839 INFO:teuthology.orchestra.run.smithi099.stdout:Install 6 Packages 2023-05-31T19:44:35.840 INFO:teuthology.orchestra.run.smithi099.stdout: 2023-05-31T19:44:35.840 INFO:teuthology.orchestra.run.smithi099.stdout:Total download size: 2.0 M 2023-05-31T19:44:35.840 INFO:teuthology.orchestra.run.smithi099.stdout:Installed size: 8.0 M 2023-05-31T19:44:35.841 INFO:teuthology.orchestra.run.smithi099.stdout:Downloading Packages: 2023-05-31T19:44:36.193 INFO:teuthology.orchestra.run.smithi112.stdout: Running scriptlet: openblas-0.3.15-4.el8.x86_64 3/8 2023-05-31T19:44:36.351 INFO:teuthology.orchestra.run.smithi099.stdout:(1/6): ceph-prometheus-alerts-18.0.0-3487.g4a02 35 kB/s | 15 kB 00:00 2023-05-31T19:44:36.363 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : openblas-threads-0.3.15-4.el8.x86_64 4/8 2023-05-31T19:44:36.410 INFO:teuthology.orchestra.run.smithi099.stdout:(2/6): ceph-grafana-dashboards-18.0.0-3487.g4a0 49 kB/s | 23 kB 00:00 2023-05-31T19:44:36.543 INFO:teuthology.orchestra.run.smithi099.stdout:(3/6): python3-repoze-lru-0.7-6.el8.noarch.rpm 252 kB/s | 34 kB 00:00 2023-05-31T19:44:36.577 INFO:teuthology.orchestra.run.smithi099.stdout:(4/6): python3-jwt-1.6.1-2.el8.noarch.rpm 189 kB/s | 43 kB 00:00 2023-05-31T19:44:36.677 INFO:teuthology.orchestra.run.smithi099.stdout:(5/6): python3-routes-2.4.1-12.el8.noarch.rpm 1.4 MB/s | 196 kB 00:00 2023-05-31T19:44:37.051 INFO:teuthology.orchestra.run.smithi112.stdout: Running scriptlet: openblas-threads-0.3.15-4.el8.x86_64 4/8 2023-05-31T19:44:37.383 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : python3-numpy-1:1.14.3-10.el8.x86_64 5/8 2023-05-31T19:44:37.621 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : python3-numpy-f2py-1:1.14.3-10.el8.x86_64 6/8 2023-05-31T19:44:38.310 INFO:teuthology.orchestra.run.smithi099.stdout:(6/6): ceph-mgr-dashboard-18.0.0-3487.g4a02f3f4 731 kB/s | 1.7 MB 00:02 2023-05-31T19:44:38.313 INFO:teuthology.orchestra.run.smithi099.stdout:-------------------------------------------------------------------------------- 2023-05-31T19:44:38.313 INFO:teuthology.orchestra.run.smithi099.stdout:Total 826 kB/s | 2.0 MB 00:02 2023-05-31T19:44:38.325 INFO:teuthology.orchestra.run.smithi099.stdout:Running transaction check 2023-05-31T19:44:38.335 INFO:teuthology.orchestra.run.smithi099.stdout:Transaction check succeeded. 2023-05-31T19:44:38.335 INFO:teuthology.orchestra.run.smithi099.stdout:Running transaction test 2023-05-31T19:44:38.424 INFO:teuthology.orchestra.run.smithi099.stdout:Transaction test succeeded. 2023-05-31T19:44:38.427 INFO:teuthology.orchestra.run.smithi099.stdout:Running transaction 2023-05-31T19:44:38.682 INFO:teuthology.orchestra.run.smithi099.stdout: Preparing : 1/1 2023-05-31T19:44:38.943 INFO:teuthology.orchestra.run.smithi099.stdout: Installing : python3-repoze-lru-0.7-6.el8.noarch 1/6 2023-05-31T19:44:39.172 INFO:teuthology.orchestra.run.smithi099.stdout: Installing : python3-routes-2.4.1-12.el8.noarch 2/6 2023-05-31T19:44:39.356 INFO:teuthology.orchestra.run.smithi099.stdout: Installing : python3-jwt-1.6.1-2.el8.noarch 3/6 2023-05-31T19:44:39.398 INFO:teuthology.orchestra.run.smithi112.stdout: Running scriptlet: python3-numpy-f2py-1:1.14.3-10.el8.x86_64 6/8 2023-05-31T19:44:39.485 INFO:teuthology.orchestra.run.smithi099.stdout: Installing : ceph-prometheus-alerts-2:18.0.0-3487.g4a02f3f4.el8.n 4/6 2023-05-31T19:44:39.824 INFO:teuthology.orchestra.run.smithi099.stdout: Installing : ceph-grafana-dashboards-2:18.0.0-3487.g4a02f3f4.el8. 5/6 2023-05-31T19:44:40.017 INFO:teuthology.orchestra.run.smithi099.stdout: Installing : ceph-mgr-dashboard-2:18.0.0-3487.g4a02f3f4.el8.noarc 6/6 2023-05-31T19:44:40.292 INFO:teuthology.orchestra.run.smithi099.stdout: Running scriptlet: ceph-mgr-dashboard-2:18.0.0-3487.g4a02f3f4.el8.noarc 6/6 2023-05-31T19:44:40.292 INFO:teuthology.orchestra.run.smithi099.stdout: Verifying : ceph-grafana-dashboards-2:18.0.0-3487.g4a02f3f4.el8. 1/6 2023-05-31T19:44:40.293 INFO:teuthology.orchestra.run.smithi099.stdout: Verifying : ceph-mgr-dashboard-2:18.0.0-3487.g4a02f3f4.el8.noarc 2/6 2023-05-31T19:44:40.293 INFO:teuthology.orchestra.run.smithi099.stdout: Verifying : ceph-prometheus-alerts-2:18.0.0-3487.g4a02f3f4.el8.n 3/6 2023-05-31T19:44:40.293 INFO:teuthology.orchestra.run.smithi099.stdout: Verifying : python3-jwt-1.6.1-2.el8.noarch 4/6 2023-05-31T19:44:40.294 INFO:teuthology.orchestra.run.smithi099.stdout: Verifying : python3-repoze-lru-0.7-6.el8.noarch 5/6 2023-05-31T19:44:40.642 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : python3-scipy-1.0.0-21.module_el8.5.0+771+e5d9a225.x 7/8 2023-05-31T19:44:40.860 INFO:teuthology.orchestra.run.smithi099.stdout: Verifying : python3-routes-2.4.1-12.el8.noarch 6/6 2023-05-31T19:44:40.860 INFO:teuthology.orchestra.run.smithi099.stdout: 2023-05-31T19:44:40.860 INFO:teuthology.orchestra.run.smithi099.stdout:Installed: 2023-05-31T19:44:40.861 INFO:teuthology.orchestra.run.smithi099.stdout: ceph-grafana-dashboards-2:18.0.0-3487.g4a02f3f4.el8.noarch 2023-05-31T19:44:40.861 INFO:teuthology.orchestra.run.smithi099.stdout: ceph-mgr-dashboard-2:18.0.0-3487.g4a02f3f4.el8.noarch 2023-05-31T19:44:40.861 INFO:teuthology.orchestra.run.smithi099.stdout: ceph-prometheus-alerts-2:18.0.0-3487.g4a02f3f4.el8.noarch 2023-05-31T19:44:40.861 INFO:teuthology.orchestra.run.smithi099.stdout: python3-jwt-1.6.1-2.el8.noarch 2023-05-31T19:44:40.861 INFO:teuthology.orchestra.run.smithi099.stdout: python3-repoze-lru-0.7-6.el8.noarch 2023-05-31T19:44:40.861 INFO:teuthology.orchestra.run.smithi099.stdout: python3-routes-2.4.1-12.el8.noarch 2023-05-31T19:44:40.861 INFO:teuthology.orchestra.run.smithi099.stdout: 2023-05-31T19:44:40.862 INFO:teuthology.orchestra.run.smithi099.stdout:Complete! 2023-05-31T19:44:40.862 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : ceph-mgr-diskprediction-local-2:18.0.0-3487.g4a02f3f 8/8 2023-05-31T19:44:40.976 DEBUG:teuthology.orchestra.run.smithi099:> sudo yum -y install ceph-mgr-diskprediction-local 2023-05-31T19:44:41.160 INFO:teuthology.orchestra.run.smithi112.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:18.0.0-3487.g4a02f3f 8/8 2023-05-31T19:44:41.160 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : ceph-mgr-diskprediction-local-2:18.0.0-3487.g4a02f3f 1/8 2023-05-31T19:44:41.160 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : openblas-0.3.15-4.el8.x86_64 2/8 2023-05-31T19:44:41.161 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : openblas-threads-0.3.15-4.el8.x86_64 3/8 2023-05-31T19:44:41.161 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : python3-numpy-1:1.14.3-10.el8.x86_64 4/8 2023-05-31T19:44:41.161 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : python3-numpy-f2py-1:1.14.3-10.el8.x86_64 5/8 2023-05-31T19:44:41.161 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : python3-scipy-1.0.0-21.module_el8.5.0+771+e5d9a225.x 6/8 2023-05-31T19:44:41.162 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : libgfortran-8.5.0-19.el8.x86_64 7/8 2023-05-31T19:44:41.703 INFO:teuthology.orchestra.run.smithi099.stdout:Last metadata expiration check: 0:01:41 ago on Wed 31 May 2023 07:43:00 PM UTC. 2023-05-31T19:44:41.722 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : libquadmath-8.5.0-19.el8.x86_64 8/8 2023-05-31T19:44:41.722 INFO:teuthology.orchestra.run.smithi112.stdout: 2023-05-31T19:44:41.723 INFO:teuthology.orchestra.run.smithi112.stdout:Installed: 2023-05-31T19:44:41.723 INFO:teuthology.orchestra.run.smithi112.stdout: ceph-mgr-diskprediction-local-2:18.0.0-3487.g4a02f3f4.el8.noarch 2023-05-31T19:44:41.724 INFO:teuthology.orchestra.run.smithi112.stdout: libgfortran-8.5.0-19.el8.x86_64 2023-05-31T19:44:41.724 INFO:teuthology.orchestra.run.smithi112.stdout: libquadmath-8.5.0-19.el8.x86_64 2023-05-31T19:44:41.724 INFO:teuthology.orchestra.run.smithi112.stdout: openblas-0.3.15-4.el8.x86_64 2023-05-31T19:44:41.724 INFO:teuthology.orchestra.run.smithi112.stdout: openblas-threads-0.3.15-4.el8.x86_64 2023-05-31T19:44:41.725 INFO:teuthology.orchestra.run.smithi112.stdout: python3-numpy-1:1.14.3-10.el8.x86_64 2023-05-31T19:44:41.725 INFO:teuthology.orchestra.run.smithi112.stdout: python3-numpy-f2py-1:1.14.3-10.el8.x86_64 2023-05-31T19:44:41.725 INFO:teuthology.orchestra.run.smithi112.stdout: python3-scipy-1.0.0-21.module_el8.5.0+771+e5d9a225.x86_64 2023-05-31T19:44:41.725 INFO:teuthology.orchestra.run.smithi112.stdout: 2023-05-31T19:44:41.725 INFO:teuthology.orchestra.run.smithi112.stdout:Complete! 2023-05-31T19:44:41.839 DEBUG:teuthology.orchestra.run.smithi112:> sudo yum -y install ceph-mgr-rook 2023-05-31T19:44:42.564 INFO:teuthology.orchestra.run.smithi112.stdout:Last metadata expiration check: 0:01:56 ago on Wed 31 May 2023 07:42:46 PM UTC. 2023-05-31T19:44:42.809 INFO:teuthology.orchestra.run.smithi099.stdout:Dependencies resolved. 2023-05-31T19:44:42.810 INFO:teuthology.orchestra.run.smithi099.stdout:================================================================================================ 2023-05-31T19:44:42.811 INFO:teuthology.orchestra.run.smithi099.stdout: Package Arch Version Repository Size 2023-05-31T19:44:42.811 INFO:teuthology.orchestra.run.smithi099.stdout:================================================================================================ 2023-05-31T19:44:42.811 INFO:teuthology.orchestra.run.smithi099.stdout:Installing: 2023-05-31T19:44:42.811 INFO:teuthology.orchestra.run.smithi099.stdout: ceph-mgr-diskprediction-local noarch 2:18.0.0-3487.g4a02f3f4.el8 ceph-noarch 7.4 M 2023-05-31T19:44:42.811 INFO:teuthology.orchestra.run.smithi099.stdout:Installing dependencies: 2023-05-31T19:44:42.812 INFO:teuthology.orchestra.run.smithi099.stdout: libgfortran x86_64 8.5.0-19.el8 baseos 645 k 2023-05-31T19:44:42.812 INFO:teuthology.orchestra.run.smithi099.stdout: libquadmath x86_64 8.5.0-19.el8 baseos 172 k 2023-05-31T19:44:42.812 INFO:teuthology.orchestra.run.smithi099.stdout: openblas x86_64 0.3.15-4.el8 appstream 4.7 M 2023-05-31T19:44:42.812 INFO:teuthology.orchestra.run.smithi099.stdout: openblas-threads x86_64 0.3.15-4.el8 appstream 4.9 M 2023-05-31T19:44:42.812 INFO:teuthology.orchestra.run.smithi099.stdout: python3-numpy x86_64 1:1.14.3-10.el8 appstream 3.7 M 2023-05-31T19:44:42.812 INFO:teuthology.orchestra.run.smithi099.stdout: python3-numpy-f2py x86_64 1:1.14.3-10.el8 appstream 225 k 2023-05-31T19:44:42.813 INFO:teuthology.orchestra.run.smithi099.stdout: python3-scipy x86_64 1.0.0-21.module_el8.5.0+771+e5d9a225 appstream 14 M 2023-05-31T19:44:42.813 INFO:teuthology.orchestra.run.smithi099.stdout: 2023-05-31T19:44:42.813 INFO:teuthology.orchestra.run.smithi099.stdout:Transaction Summary 2023-05-31T19:44:42.813 INFO:teuthology.orchestra.run.smithi099.stdout:================================================================================================ 2023-05-31T19:44:42.813 INFO:teuthology.orchestra.run.smithi099.stdout:Install 8 Packages 2023-05-31T19:44:42.813 INFO:teuthology.orchestra.run.smithi099.stdout: 2023-05-31T19:44:42.814 INFO:teuthology.orchestra.run.smithi099.stdout:Total download size: 35 M 2023-05-31T19:44:42.814 INFO:teuthology.orchestra.run.smithi099.stdout:Installed size: 210 M 2023-05-31T19:44:42.814 INFO:teuthology.orchestra.run.smithi099.stdout:Downloading Packages: 2023-05-31T19:44:43.516 INFO:teuthology.orchestra.run.smithi099.stdout:(1/8): openblas-0.3.15-4.el8.x86_64.rpm 15 MB/s | 4.7 MB 00:00 2023-05-31T19:44:43.591 INFO:teuthology.orchestra.run.smithi099.stdout:(2/8): openblas-threads-0.3.15-4.el8.x86_64.rpm 12 MB/s | 4.9 MB 00:00 2023-05-31T19:44:43.642 INFO:teuthology.orchestra.run.smithi099.stdout:(3/8): python3-numpy-f2py-1.14.3-10.el8.x86_64. 4.4 MB/s | 225 kB 00:00 2023-05-31T19:44:43.673 INFO:teuthology.orchestra.run.smithi112.stdout:Dependencies resolved. 2023-05-31T19:44:43.675 INFO:teuthology.orchestra.run.smithi112.stdout:================================================================================ 2023-05-31T19:44:43.675 INFO:teuthology.orchestra.run.smithi112.stdout: Package Arch Version Repository Size 2023-05-31T19:44:43.675 INFO:teuthology.orchestra.run.smithi112.stdout:================================================================================ 2023-05-31T19:44:43.675 INFO:teuthology.orchestra.run.smithi112.stdout:Installing: 2023-05-31T19:44:43.675 INFO:teuthology.orchestra.run.smithi112.stdout: ceph-mgr-rook noarch 2:18.0.0-3487.g4a02f3f4.el8 ceph-noarch 47 k 2023-05-31T19:44:43.676 INFO:teuthology.orchestra.run.smithi112.stdout:Installing dependencies: 2023-05-31T19:44:43.676 INFO:teuthology.orchestra.run.smithi112.stdout: python3-cachetools noarch 3.1.1-4.el8 epel 33 k 2023-05-31T19:44:43.676 INFO:teuthology.orchestra.run.smithi112.stdout: python3-certifi noarch 2018.10.15-7.el8 epel 16 k 2023-05-31T19:44:43.676 INFO:teuthology.orchestra.run.smithi112.stdout: python3-google-auth noarch 1:1.1.1-10.el8 epel 81 k 2023-05-31T19:44:43.676 INFO:teuthology.orchestra.run.smithi112.stdout: python3-jsonpatch noarch 1.21-2.el8 appstream 27 k 2023-05-31T19:44:43.677 INFO:teuthology.orchestra.run.smithi112.stdout: python3-jsonpointer noarch 1.10-11.el8 appstream 20 k 2023-05-31T19:44:43.677 INFO:teuthology.orchestra.run.smithi112.stdout: python3-kubernetes noarch 1:11.0.0-6.el8 epel 1.2 M 2023-05-31T19:44:43.677 INFO:teuthology.orchestra.run.smithi112.stdout: python3-oauthlib noarch 2.1.0-1.el8 baseos 155 k 2023-05-31T19:44:43.677 INFO:teuthology.orchestra.run.smithi112.stdout: python3-pyasn1 noarch 0.3.7-6.el8 appstream 126 k 2023-05-31T19:44:43.677 INFO:teuthology.orchestra.run.smithi112.stdout: python3-pyasn1-modules noarch 0.3.7-6.el8 appstream 110 k 2023-05-31T19:44:43.677 INFO:teuthology.orchestra.run.smithi112.stdout: python3-requests-oauthlib noarch 1.0.0-1.el8 baseos 43 k 2023-05-31T19:44:43.677 INFO:teuthology.orchestra.run.smithi112.stdout: python3-rsa noarch 4.8-1.el8 epel 61 k 2023-05-31T19:44:43.678 INFO:teuthology.orchestra.run.smithi112.stdout: python3-websocket-client noarch 0.56.0-5.el8 epel 61 k 2023-05-31T19:44:43.678 INFO:teuthology.orchestra.run.smithi112.stdout: 2023-05-31T19:44:43.678 INFO:teuthology.orchestra.run.smithi112.stdout:Transaction Summary 2023-05-31T19:44:43.678 INFO:teuthology.orchestra.run.smithi112.stdout:================================================================================ 2023-05-31T19:44:43.678 INFO:teuthology.orchestra.run.smithi112.stdout:Install 13 Packages 2023-05-31T19:44:43.679 INFO:teuthology.orchestra.run.smithi112.stdout: 2023-05-31T19:44:43.680 INFO:teuthology.orchestra.run.smithi112.stdout:Total download size: 1.9 M 2023-05-31T19:44:43.680 INFO:teuthology.orchestra.run.smithi112.stdout:Installed size: 25 M 2023-05-31T19:44:43.681 INFO:teuthology.orchestra.run.smithi112.stdout:Downloading Packages: 2023-05-31T19:44:43.700 INFO:teuthology.orchestra.run.smithi099.stdout:(4/8): python3-numpy-1.14.3-10.el8.x86_64.rpm 20 MB/s | 3.7 MB 00:00 2023-05-31T19:44:44.085 INFO:teuthology.orchestra.run.smithi099.stdout:(5/8): libgfortran-8.5.0-19.el8.x86_64.rpm 1.6 MB/s | 645 kB 00:00 2023-05-31T19:44:44.089 INFO:teuthology.orchestra.run.smithi112.stdout:(1/13): python3-jsonpatch-1.21-2.el8.noarch.rpm 327 kB/s | 27 kB 00:00 2023-05-31T19:44:44.106 INFO:teuthology.orchestra.run.smithi112.stdout:(2/13): python3-jsonpointer-1.10-11.el8.noarch. 203 kB/s | 20 kB 00:00 2023-05-31T19:44:44.181 INFO:teuthology.orchestra.run.smithi112.stdout:(3/13): python3-pyasn1-0.3.7-6.el8.noarch.rpm 1.3 MB/s | 126 kB 00:00 2023-05-31T19:44:44.207 INFO:teuthology.orchestra.run.smithi112.stdout:(4/13): python3-pyasn1-modules-0.3.7-6.el8.noar 1.1 MB/s | 110 kB 00:00 2023-05-31T19:44:44.248 INFO:teuthology.orchestra.run.smithi099.stdout:(6/8): python3-scipy-1.0.0-21.module_el8.5.0+77 23 MB/s | 14 MB 00:00 2023-05-31T19:44:44.349 INFO:teuthology.orchestra.run.smithi112.stdout:(5/13): ceph-mgr-rook-18.0.0-3487.g4a02f3f4.el8 139 kB/s | 47 kB 00:00 2023-05-31T19:44:44.365 INFO:teuthology.orchestra.run.smithi112.stdout:(6/13): python3-requests-oauthlib-1.0.0-1.el8.n 268 kB/s | 43 kB 00:00 2023-05-31T19:44:44.382 INFO:teuthology.orchestra.run.smithi112.stdout:(7/13): python3-oauthlib-2.1.0-1.el8.noarch.rpm 771 kB/s | 155 kB 00:00 2023-05-31T19:44:44.419 INFO:teuthology.orchestra.run.smithi099.stdout:(7/8): libquadmath-8.5.0-19.el8.x86_64.rpm 517 kB/s | 172 kB 00:00 2023-05-31T19:44:44.491 INFO:teuthology.orchestra.run.smithi112.stdout:(8/13): python3-certifi-2018.10.15-7.el8.noarch 130 kB/s | 16 kB 00:00 2023-05-31T19:44:44.508 INFO:teuthology.orchestra.run.smithi112.stdout:(9/13): python3-cachetools-3.1.1-4.el8.noarch.r 207 kB/s | 33 kB 00:00 2023-05-31T19:44:44.584 INFO:teuthology.orchestra.run.smithi112.stdout:(10/13): python3-google-auth-1.1.1-10.el8.noarc 405 kB/s | 81 kB 00:00 2023-05-31T19:44:44.601 INFO:teuthology.orchestra.run.smithi112.stdout:(11/13): python3-rsa-4.8-1.el8.noarch.rpm 668 kB/s | 61 kB 00:00 2023-05-31T19:44:44.617 INFO:teuthology.orchestra.run.smithi112.stdout:(12/13): python3-websocket-client-0.56.0-5.el8. 1.8 MB/s | 61 kB 00:00 2023-05-31T19:44:44.793 INFO:teuthology.orchestra.run.smithi112.stdout:(13/13): python3-kubernetes-11.0.0-6.el8.noarch 3.9 MB/s | 1.2 MB 00:00 2023-05-31T19:44:44.795 INFO:teuthology.orchestra.run.smithi112.stdout:-------------------------------------------------------------------------------- 2023-05-31T19:44:44.796 INFO:teuthology.orchestra.run.smithi112.stdout:Total 1.7 MB/s | 1.9 MB 00:01 2023-05-31T19:44:44.853 INFO:teuthology.orchestra.run.smithi112.stdout:Running transaction check 2023-05-31T19:44:44.868 INFO:teuthology.orchestra.run.smithi112.stdout:Transaction check succeeded. 2023-05-31T19:44:44.868 INFO:teuthology.orchestra.run.smithi112.stdout:Running transaction test 2023-05-31T19:44:44.953 INFO:teuthology.orchestra.run.smithi112.stdout:Transaction test succeeded. 2023-05-31T19:44:44.956 INFO:teuthology.orchestra.run.smithi112.stdout:Running transaction 2023-05-31T19:44:45.472 INFO:teuthology.orchestra.run.smithi112.stdout: Preparing : 1/1 2023-05-31T19:44:45.717 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : python3-pyasn1-0.3.7-6.el8.noarch 1/13 2023-05-31T19:44:45.794 INFO:teuthology.orchestra.run.smithi099.stdout:(8/8): ceph-mgr-diskprediction-local-18.0.0-348 2.8 MB/s | 7.4 MB 00:02 2023-05-31T19:44:45.795 INFO:teuthology.orchestra.run.smithi099.stdout:-------------------------------------------------------------------------------- 2023-05-31T19:44:45.795 INFO:teuthology.orchestra.run.smithi099.stdout:Total 12 MB/s | 35 MB 00:02 2023-05-31T19:44:45.934 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : python3-pyasn1-modules-0.3.7-6.el8.noarch 2/13 2023-05-31T19:44:46.027 INFO:teuthology.orchestra.run.smithi099.stdout:Running transaction check 2023-05-31T19:44:46.046 INFO:teuthology.orchestra.run.smithi099.stdout:Transaction check succeeded. 2023-05-31T19:44:46.046 INFO:teuthology.orchestra.run.smithi099.stdout:Running transaction test 2023-05-31T19:44:46.234 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : python3-rsa-4.8-1.el8.noarch 3/13 2023-05-31T19:44:46.445 INFO:teuthology.orchestra.run.smithi099.stdout:Transaction test succeeded. 2023-05-31T19:44:46.448 INFO:teuthology.orchestra.run.smithi099.stdout:Running transaction 2023-05-31T19:44:46.486 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : python3-websocket-client-0.56.0-5.el8.noarch 4/13 2023-05-31T19:44:46.686 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : python3-certifi-2018.10.15-7.el8.noarch 5/13 2023-05-31T19:44:46.912 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : python3-cachetools-3.1.1-4.el8.noarch 6/13 2023-05-31T19:44:46.996 INFO:teuthology.orchestra.run.smithi099.stdout: Preparing : 1/1 2023-05-31T19:44:47.127 INFO:teuthology.orchestra.run.smithi099.stdout: Installing : libquadmath-8.5.0-19.el8.x86_64 1/8 2023-05-31T19:44:47.179 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : python3-google-auth-1:1.1.1-10.el8.noarch 7/13 2023-05-31T19:44:47.289 INFO:teuthology.orchestra.run.smithi099.stdout: Running scriptlet: libquadmath-8.5.0-19.el8.x86_64 1/8 2023-05-31T19:44:47.409 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : python3-oauthlib-2.1.0-1.el8.noarch 8/13 2023-05-31T19:44:47.419 INFO:teuthology.orchestra.run.smithi099.stdout: Installing : libgfortran-8.5.0-19.el8.x86_64 2/8 2023-05-31T19:44:48.000 INFO:teuthology.orchestra.run.smithi099.stdout: Running scriptlet: libgfortran-8.5.0-19.el8.x86_64 2/8 2023-05-31T19:44:48.211 INFO:teuthology.orchestra.run.smithi099.stdout: Installing : openblas-0.3.15-4.el8.x86_64 3/8 2023-05-31T19:44:48.590 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : python3-requests-oauthlib-1.0.0-1.el8.noarch 9/13 2023-05-31T19:44:48.827 INFO:teuthology.orchestra.run.smithi099.stdout: Running scriptlet: openblas-0.3.15-4.el8.x86_64 3/8 2023-05-31T19:44:48.868 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : python3-kubernetes-1:11.0.0-6.el8.noarch 10/13 2023-05-31T19:44:48.977 INFO:teuthology.orchestra.run.smithi099.stdout: Installing : openblas-threads-0.3.15-4.el8.x86_64 4/8 2023-05-31T19:44:49.094 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : python3-jsonpointer-1.10-11.el8.noarch 11/13 2023-05-31T19:44:49.264 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : python3-jsonpatch-1.21-2.el8.noarch 12/13 2023-05-31T19:44:49.461 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : ceph-mgr-rook-2:18.0.0-3487.g4a02f3f4.el8.noarch 13/13 2023-05-31T19:44:49.747 INFO:teuthology.orchestra.run.smithi112.stdout: Running scriptlet: ceph-mgr-rook-2:18.0.0-3487.g4a02f3f4.el8.noarch 13/13 2023-05-31T19:44:49.747 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : ceph-mgr-rook-2:18.0.0-3487.g4a02f3f4.el8.noarch 1/13 2023-05-31T19:44:49.747 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : python3-jsonpatch-1.21-2.el8.noarch 2/13 2023-05-31T19:44:49.747 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : python3-jsonpointer-1.10-11.el8.noarch 3/13 2023-05-31T19:44:49.747 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : python3-pyasn1-0.3.7-6.el8.noarch 4/13 2023-05-31T19:44:49.748 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : python3-pyasn1-modules-0.3.7-6.el8.noarch 5/13 2023-05-31T19:44:49.748 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : python3-oauthlib-2.1.0-1.el8.noarch 6/13 2023-05-31T19:44:49.748 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : python3-requests-oauthlib-1.0.0-1.el8.noarch 7/13 2023-05-31T19:44:49.748 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : python3-cachetools-3.1.1-4.el8.noarch 8/13 2023-05-31T19:44:49.748 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : python3-certifi-2018.10.15-7.el8.noarch 9/13 2023-05-31T19:44:49.748 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : python3-google-auth-1:1.1.1-10.el8.noarch 10/13 2023-05-31T19:44:49.748 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : python3-kubernetes-1:11.0.0-6.el8.noarch 11/13 2023-05-31T19:44:49.749 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : python3-rsa-4.8-1.el8.noarch 12/13 2023-05-31T19:44:49.894 INFO:teuthology.orchestra.run.smithi099.stdout: Running scriptlet: openblas-threads-0.3.15-4.el8.x86_64 4/8 2023-05-31T19:44:50.288 INFO:teuthology.orchestra.run.smithi099.stdout: Installing : python3-numpy-1:1.14.3-10.el8.x86_64 5/8 2023-05-31T19:44:50.426 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : python3-websocket-client-0.56.0-5.el8.noarch 13/13 2023-05-31T19:44:50.426 INFO:teuthology.orchestra.run.smithi112.stdout: 2023-05-31T19:44:50.426 INFO:teuthology.orchestra.run.smithi112.stdout:Installed: 2023-05-31T19:44:50.426 INFO:teuthology.orchestra.run.smithi112.stdout: ceph-mgr-rook-2:18.0.0-3487.g4a02f3f4.el8.noarch 2023-05-31T19:44:50.427 INFO:teuthology.orchestra.run.smithi112.stdout: python3-cachetools-3.1.1-4.el8.noarch 2023-05-31T19:44:50.427 INFO:teuthology.orchestra.run.smithi112.stdout: python3-certifi-2018.10.15-7.el8.noarch 2023-05-31T19:44:50.427 INFO:teuthology.orchestra.run.smithi112.stdout: python3-google-auth-1:1.1.1-10.el8.noarch 2023-05-31T19:44:50.427 INFO:teuthology.orchestra.run.smithi112.stdout: python3-jsonpatch-1.21-2.el8.noarch 2023-05-31T19:44:50.427 INFO:teuthology.orchestra.run.smithi112.stdout: python3-jsonpointer-1.10-11.el8.noarch 2023-05-31T19:44:50.427 INFO:teuthology.orchestra.run.smithi112.stdout: python3-kubernetes-1:11.0.0-6.el8.noarch 2023-05-31T19:44:50.427 INFO:teuthology.orchestra.run.smithi112.stdout: python3-oauthlib-2.1.0-1.el8.noarch 2023-05-31T19:44:50.427 INFO:teuthology.orchestra.run.smithi112.stdout: python3-pyasn1-0.3.7-6.el8.noarch 2023-05-31T19:44:50.428 INFO:teuthology.orchestra.run.smithi112.stdout: python3-pyasn1-modules-0.3.7-6.el8.noarch 2023-05-31T19:44:50.428 INFO:teuthology.orchestra.run.smithi112.stdout: python3-requests-oauthlib-1.0.0-1.el8.noarch 2023-05-31T19:44:50.428 INFO:teuthology.orchestra.run.smithi112.stdout: python3-rsa-4.8-1.el8.noarch 2023-05-31T19:44:50.428 INFO:teuthology.orchestra.run.smithi112.stdout: python3-websocket-client-0.56.0-5.el8.noarch 2023-05-31T19:44:50.429 INFO:teuthology.orchestra.run.smithi112.stdout: 2023-05-31T19:44:50.429 INFO:teuthology.orchestra.run.smithi112.stdout:Complete! 2023-05-31T19:44:50.501 INFO:teuthology.orchestra.run.smithi099.stdout: Installing : python3-numpy-f2py-1:1.14.3-10.el8.x86_64 6/8 2023-05-31T19:44:50.561 DEBUG:teuthology.orchestra.run.smithi112:> sudo yum -y install ceph-mgr-cephadm 2023-05-31T19:44:51.295 INFO:teuthology.orchestra.run.smithi112.stdout:Last metadata expiration check: 0:02:05 ago on Wed 31 May 2023 07:42:46 PM UTC. 2023-05-31T19:44:52.280 INFO:teuthology.orchestra.run.smithi099.stdout: Running scriptlet: python3-numpy-f2py-1:1.14.3-10.el8.x86_64 6/8 2023-05-31T19:44:52.410 INFO:teuthology.orchestra.run.smithi112.stdout:Dependencies resolved. 2023-05-31T19:44:52.411 INFO:teuthology.orchestra.run.smithi112.stdout:================================================================================ 2023-05-31T19:44:52.411 INFO:teuthology.orchestra.run.smithi112.stdout: Package Arch Version Repository Size 2023-05-31T19:44:52.411 INFO:teuthology.orchestra.run.smithi112.stdout:================================================================================ 2023-05-31T19:44:52.411 INFO:teuthology.orchestra.run.smithi112.stdout:Installing: 2023-05-31T19:44:52.412 INFO:teuthology.orchestra.run.smithi112.stdout: ceph-mgr-cephadm noarch 2:18.0.0-3487.g4a02f3f4.el8 ceph-noarch 127 k 2023-05-31T19:44:52.412 INFO:teuthology.orchestra.run.smithi112.stdout:Installing dependencies: 2023-05-31T19:44:52.412 INFO:teuthology.orchestra.run.smithi112.stdout: python3-asyncssh noarch 2.7.0-2.el8 epel 476 k 2023-05-31T19:44:52.412 INFO:teuthology.orchestra.run.smithi112.stdout: python3-babel noarch 2.5.1-7.el8 appstream 4.8 M 2023-05-31T19:44:52.413 INFO:teuthology.orchestra.run.smithi112.stdout: python3-jinja2 noarch 2.10.1-3.el8 appstream 538 k 2023-05-31T19:44:52.413 INFO:teuthology.orchestra.run.smithi112.stdout: python3-natsort noarch 7.1.1-2.el8 epel 60 k 2023-05-31T19:44:52.413 INFO:teuthology.orchestra.run.smithi112.stdout: 2023-05-31T19:44:52.413 INFO:teuthology.orchestra.run.smithi112.stdout:Transaction Summary 2023-05-31T19:44:52.413 INFO:teuthology.orchestra.run.smithi112.stdout:================================================================================ 2023-05-31T19:44:52.413 INFO:teuthology.orchestra.run.smithi112.stdout:Install 5 Packages 2023-05-31T19:44:52.413 INFO:teuthology.orchestra.run.smithi112.stdout: 2023-05-31T19:44:52.414 INFO:teuthology.orchestra.run.smithi112.stdout:Total download size: 6.0 M 2023-05-31T19:44:52.414 INFO:teuthology.orchestra.run.smithi112.stdout:Installed size: 26 M 2023-05-31T19:44:52.414 INFO:teuthology.orchestra.run.smithi112.stdout:Downloading Packages: 2023-05-31T19:44:52.683 INFO:teuthology.orchestra.run.smithi112.stdout:(1/5): ceph-mgr-cephadm-18.0.0-3487.g4a02f3f4.e 679 kB/s | 127 kB 00:00 2023-05-31T19:44:52.866 INFO:teuthology.orchestra.run.smithi112.stdout:(2/5): python3-jinja2-2.10.1-3.el8.noarch.rpm 1.4 MB/s | 538 kB 00:00 2023-05-31T19:44:53.225 INFO:teuthology.orchestra.run.smithi112.stdout:(3/5): python3-natsort-7.1.1-2.el8.noarch.rpm 168 kB/s | 60 kB 00:00 2023-05-31T19:44:53.433 INFO:teuthology.orchestra.run.smithi112.stdout:(4/5): python3-asyncssh-2.7.0-2.el8.noarch.rpm 634 kB/s | 476 kB 00:00 2023-05-31T19:44:53.474 INFO:teuthology.orchestra.run.smithi099.stdout: Installing : python3-scipy-1.0.0-21.module_el8.5.0+771+e5d9a225.x 7/8 2023-05-31T19:44:53.509 INFO:teuthology.orchestra.run.smithi112.stdout:(5/5): python3-babel-2.5.1-7.el8.noarch.rpm 4.7 MB/s | 4.8 MB 00:01 2023-05-31T19:44:53.511 INFO:teuthology.orchestra.run.smithi112.stdout:-------------------------------------------------------------------------------- 2023-05-31T19:44:53.511 INFO:teuthology.orchestra.run.smithi112.stdout:Total 5.4 MB/s | 6.0 MB 00:01 2023-05-31T19:44:53.562 INFO:teuthology.orchestra.run.smithi112.stdout:Running transaction check 2023-05-31T19:44:53.573 INFO:teuthology.orchestra.run.smithi112.stdout:Transaction check succeeded. 2023-05-31T19:44:53.573 INFO:teuthology.orchestra.run.smithi112.stdout:Running transaction test 2023-05-31T19:44:53.675 INFO:teuthology.orchestra.run.smithi099.stdout: Installing : ceph-mgr-diskprediction-local-2:18.0.0-3487.g4a02f3f 8/8 2023-05-31T19:44:53.677 INFO:teuthology.orchestra.run.smithi112.stdout:Transaction test succeeded. 2023-05-31T19:44:53.680 INFO:teuthology.orchestra.run.smithi112.stdout:Running transaction 2023-05-31T19:44:53.960 INFO:teuthology.orchestra.run.smithi112.stdout: Preparing : 1/1 2023-05-31T19:44:54.017 INFO:teuthology.orchestra.run.smithi099.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:18.0.0-3487.g4a02f3f 8/8 2023-05-31T19:44:54.018 INFO:teuthology.orchestra.run.smithi099.stdout: Verifying : ceph-mgr-diskprediction-local-2:18.0.0-3487.g4a02f3f 1/8 2023-05-31T19:44:54.018 INFO:teuthology.orchestra.run.smithi099.stdout: Verifying : openblas-0.3.15-4.el8.x86_64 2/8 2023-05-31T19:44:54.018 INFO:teuthology.orchestra.run.smithi099.stdout: Verifying : openblas-threads-0.3.15-4.el8.x86_64 3/8 2023-05-31T19:44:54.019 INFO:teuthology.orchestra.run.smithi099.stdout: Verifying : python3-numpy-1:1.14.3-10.el8.x86_64 4/8 2023-05-31T19:44:54.019 INFO:teuthology.orchestra.run.smithi099.stdout: Verifying : python3-numpy-f2py-1:1.14.3-10.el8.x86_64 5/8 2023-05-31T19:44:54.019 INFO:teuthology.orchestra.run.smithi099.stdout: Verifying : python3-scipy-1.0.0-21.module_el8.5.0+771+e5d9a225.x 6/8 2023-05-31T19:44:54.019 INFO:teuthology.orchestra.run.smithi099.stdout: Verifying : libgfortran-8.5.0-19.el8.x86_64 7/8 2023-05-31T19:44:54.291 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : python3-natsort-7.1.1-2.el8.noarch 1/5 2023-05-31T19:44:54.593 INFO:teuthology.orchestra.run.smithi099.stdout: Verifying : libquadmath-8.5.0-19.el8.x86_64 8/8 2023-05-31T19:44:54.593 INFO:teuthology.orchestra.run.smithi099.stdout: 2023-05-31T19:44:54.594 INFO:teuthology.orchestra.run.smithi099.stdout:Installed: 2023-05-31T19:44:54.594 INFO:teuthology.orchestra.run.smithi099.stdout: ceph-mgr-diskprediction-local-2:18.0.0-3487.g4a02f3f4.el8.noarch 2023-05-31T19:44:54.594 INFO:teuthology.orchestra.run.smithi099.stdout: libgfortran-8.5.0-19.el8.x86_64 2023-05-31T19:44:54.594 INFO:teuthology.orchestra.run.smithi099.stdout: libquadmath-8.5.0-19.el8.x86_64 2023-05-31T19:44:54.594 INFO:teuthology.orchestra.run.smithi099.stdout: openblas-0.3.15-4.el8.x86_64 2023-05-31T19:44:54.595 INFO:teuthology.orchestra.run.smithi099.stdout: openblas-threads-0.3.15-4.el8.x86_64 2023-05-31T19:44:54.595 INFO:teuthology.orchestra.run.smithi099.stdout: python3-numpy-1:1.14.3-10.el8.x86_64 2023-05-31T19:44:54.595 INFO:teuthology.orchestra.run.smithi099.stdout: python3-numpy-f2py-1:1.14.3-10.el8.x86_64 2023-05-31T19:44:54.595 INFO:teuthology.orchestra.run.smithi099.stdout: python3-scipy-1.0.0-21.module_el8.5.0+771+e5d9a225.x86_64 2023-05-31T19:44:54.595 INFO:teuthology.orchestra.run.smithi099.stdout: 2023-05-31T19:44:54.595 INFO:teuthology.orchestra.run.smithi099.stdout:Complete! 2023-05-31T19:44:54.703 DEBUG:teuthology.orchestra.run.smithi099:> sudo yum -y install ceph-mgr-rook 2023-05-31T19:44:55.231 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : python3-asyncssh-2.7.0-2.el8.noarch 2/5 2023-05-31T19:44:55.428 INFO:teuthology.orchestra.run.smithi099.stdout:Last metadata expiration check: 0:01:55 ago on Wed 31 May 2023 07:43:00 PM UTC. 2023-05-31T19:44:55.606 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : python3-babel-2.5.1-7.el8.noarch 3/5 2023-05-31T19:44:55.882 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : python3-jinja2-2.10.1-3.el8.noarch 4/5 2023-05-31T19:44:56.102 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : ceph-mgr-cephadm-2:18.0.0-3487.g4a02f3f4.el8.noarch 5/5 2023-05-31T19:44:56.387 INFO:teuthology.orchestra.run.smithi112.stdout: Running scriptlet: ceph-mgr-cephadm-2:18.0.0-3487.g4a02f3f4.el8.noarch 5/5 2023-05-31T19:44:56.387 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : ceph-mgr-cephadm-2:18.0.0-3487.g4a02f3f4.el8.noarch 1/5 2023-05-31T19:44:56.387 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : python3-babel-2.5.1-7.el8.noarch 2/5 2023-05-31T19:44:56.387 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : python3-jinja2-2.10.1-3.el8.noarch 3/5 2023-05-31T19:44:56.388 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : python3-asyncssh-2.7.0-2.el8.noarch 4/5 2023-05-31T19:44:56.538 INFO:teuthology.orchestra.run.smithi099.stdout:Dependencies resolved. 2023-05-31T19:44:56.540 INFO:teuthology.orchestra.run.smithi099.stdout:================================================================================ 2023-05-31T19:44:56.540 INFO:teuthology.orchestra.run.smithi099.stdout: Package Arch Version Repository Size 2023-05-31T19:44:56.540 INFO:teuthology.orchestra.run.smithi099.stdout:================================================================================ 2023-05-31T19:44:56.540 INFO:teuthology.orchestra.run.smithi099.stdout:Installing: 2023-05-31T19:44:56.540 INFO:teuthology.orchestra.run.smithi099.stdout: ceph-mgr-rook noarch 2:18.0.0-3487.g4a02f3f4.el8 ceph-noarch 47 k 2023-05-31T19:44:56.541 INFO:teuthology.orchestra.run.smithi099.stdout:Installing dependencies: 2023-05-31T19:44:56.541 INFO:teuthology.orchestra.run.smithi099.stdout: python3-cachetools noarch 3.1.1-4.el8 epel 33 k 2023-05-31T19:44:56.541 INFO:teuthology.orchestra.run.smithi099.stdout: python3-certifi noarch 2018.10.15-7.el8 epel 16 k 2023-05-31T19:44:56.541 INFO:teuthology.orchestra.run.smithi099.stdout: python3-google-auth noarch 1:1.1.1-10.el8 epel 81 k 2023-05-31T19:44:56.541 INFO:teuthology.orchestra.run.smithi099.stdout: python3-jsonpatch noarch 1.21-2.el8 appstream 27 k 2023-05-31T19:44:56.541 INFO:teuthology.orchestra.run.smithi099.stdout: python3-jsonpointer noarch 1.10-11.el8 appstream 20 k 2023-05-31T19:44:56.542 INFO:teuthology.orchestra.run.smithi099.stdout: python3-kubernetes noarch 1:11.0.0-6.el8 epel 1.2 M 2023-05-31T19:44:56.542 INFO:teuthology.orchestra.run.smithi099.stdout: python3-oauthlib noarch 2.1.0-1.el8 baseos 155 k 2023-05-31T19:44:56.542 INFO:teuthology.orchestra.run.smithi099.stdout: python3-pyasn1 noarch 0.3.7-6.el8 appstream 126 k 2023-05-31T19:44:56.542 INFO:teuthology.orchestra.run.smithi099.stdout: python3-pyasn1-modules noarch 0.3.7-6.el8 appstream 110 k 2023-05-31T19:44:56.542 INFO:teuthology.orchestra.run.smithi099.stdout: python3-requests-oauthlib noarch 1.0.0-1.el8 baseos 43 k 2023-05-31T19:44:56.542 INFO:teuthology.orchestra.run.smithi099.stdout: python3-rsa noarch 4.8-1.el8 epel 61 k 2023-05-31T19:44:56.542 INFO:teuthology.orchestra.run.smithi099.stdout: python3-websocket-client noarch 0.56.0-5.el8 epel 61 k 2023-05-31T19:44:56.543 INFO:teuthology.orchestra.run.smithi099.stdout: 2023-05-31T19:44:56.543 INFO:teuthology.orchestra.run.smithi099.stdout:Transaction Summary 2023-05-31T19:44:56.543 INFO:teuthology.orchestra.run.smithi099.stdout:================================================================================ 2023-05-31T19:44:56.543 INFO:teuthology.orchestra.run.smithi099.stdout:Install 13 Packages 2023-05-31T19:44:56.543 INFO:teuthology.orchestra.run.smithi099.stdout: 2023-05-31T19:44:56.544 INFO:teuthology.orchestra.run.smithi099.stdout:Total download size: 1.9 M 2023-05-31T19:44:56.544 INFO:teuthology.orchestra.run.smithi099.stdout:Installed size: 25 M 2023-05-31T19:44:56.544 INFO:teuthology.orchestra.run.smithi099.stdout:Downloading Packages: 2023-05-31T19:44:56.948 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : python3-natsort-7.1.1-2.el8.noarch 5/5 2023-05-31T19:44:56.948 INFO:teuthology.orchestra.run.smithi112.stdout: 2023-05-31T19:44:56.948 INFO:teuthology.orchestra.run.smithi112.stdout:Installed: 2023-05-31T19:44:56.948 INFO:teuthology.orchestra.run.smithi112.stdout: ceph-mgr-cephadm-2:18.0.0-3487.g4a02f3f4.el8.noarch 2023-05-31T19:44:56.948 INFO:teuthology.orchestra.run.smithi112.stdout: python3-asyncssh-2.7.0-2.el8.noarch 2023-05-31T19:44:56.949 INFO:teuthology.orchestra.run.smithi112.stdout: python3-babel-2.5.1-7.el8.noarch 2023-05-31T19:44:56.949 INFO:teuthology.orchestra.run.smithi112.stdout: python3-jinja2-2.10.1-3.el8.noarch 2023-05-31T19:44:56.949 INFO:teuthology.orchestra.run.smithi112.stdout: python3-natsort-7.1.1-2.el8.noarch 2023-05-31T19:44:56.949 INFO:teuthology.orchestra.run.smithi112.stdout: 2023-05-31T19:44:56.950 INFO:teuthology.orchestra.run.smithi112.stdout:Complete! 2023-05-31T19:44:57.008 INFO:teuthology.orchestra.run.smithi099.stdout:(1/13): python3-jsonpatch-1.21-2.el8.noarch.rpm 301 kB/s | 27 kB 00:00 2023-05-31T19:44:57.025 INFO:teuthology.orchestra.run.smithi099.stdout:(2/13): python3-jsonpointer-1.10-11.el8.noarch. 190 kB/s | 20 kB 00:00 2023-05-31T19:44:57.059 DEBUG:teuthology.orchestra.run.smithi112:> sudo yum -y install ceph-fuse 2023-05-31T19:44:57.092 INFO:teuthology.orchestra.run.smithi099.stdout:(3/13): python3-pyasn1-0.3.7-6.el8.noarch.rpm 1.9 MB/s | 126 kB 00:00 2023-05-31T19:44:57.117 INFO:teuthology.orchestra.run.smithi099.stdout:(4/13): python3-pyasn1-modules-0.3.7-6.el8.noar 1.2 MB/s | 110 kB 00:00 2023-05-31T19:44:57.192 INFO:teuthology.orchestra.run.smithi099.stdout:(5/13): ceph-mgr-rook-18.0.0-3487.g4a02f3f4.el8 174 kB/s | 47 kB 00:00 2023-05-31T19:44:57.284 INFO:teuthology.orchestra.run.smithi099.stdout:(6/13): python3-requests-oauthlib-1.0.0-1.el8.n 255 kB/s | 43 kB 00:00 2023-05-31T19:44:57.326 INFO:teuthology.orchestra.run.smithi099.stdout:(7/13): python3-oauthlib-2.1.0-1.el8.noarch.rpm 662 kB/s | 155 kB 00:00 2023-05-31T19:44:57.576 INFO:teuthology.orchestra.run.smithi099.stdout:(8/13): python3-cachetools-3.1.1-4.el8.noarch.r 86 kB/s | 33 kB 00:00 2023-05-31T19:44:57.647 INFO:teuthology.orchestra.run.smithi099.stdout:(9/13): python3-certifi-2018.10.15-7.el8.noarch 46 kB/s | 16 kB 00:00 2023-05-31T19:44:57.784 INFO:teuthology.orchestra.run.smithi112.stdout:Last metadata expiration check: 0:02:11 ago on Wed 31 May 2023 07:42:46 PM UTC. 2023-05-31T19:44:57.785 INFO:teuthology.orchestra.run.smithi099.stdout:(10/13): python3-google-auth-1.1.1-10.el8.noarc 177 kB/s | 81 kB 00:00 2023-05-31T19:44:57.868 INFO:teuthology.orchestra.run.smithi099.stdout:(11/13): python3-rsa-4.8-1.el8.noarch.rpm 278 kB/s | 61 kB 00:00 2023-05-31T19:44:57.910 INFO:teuthology.orchestra.run.smithi099.stdout:(12/13): python3-websocket-client-0.56.0-5.el8. 490 kB/s | 61 kB 00:00 2023-05-31T19:44:58.077 INFO:teuthology.orchestra.run.smithi099.stdout:(13/13): python3-kubernetes-11.0.0-6.el8.noarch 2.3 MB/s | 1.2 MB 00:00 2023-05-31T19:44:58.080 INFO:teuthology.orchestra.run.smithi099.stdout:-------------------------------------------------------------------------------- 2023-05-31T19:44:58.080 INFO:teuthology.orchestra.run.smithi099.stdout:Total 1.3 MB/s | 1.9 MB 00:01 2023-05-31T19:44:58.139 INFO:teuthology.orchestra.run.smithi099.stdout:Running transaction check 2023-05-31T19:44:58.154 INFO:teuthology.orchestra.run.smithi099.stdout:Transaction check succeeded. 2023-05-31T19:44:58.155 INFO:teuthology.orchestra.run.smithi099.stdout:Running transaction test 2023-05-31T19:44:58.242 INFO:teuthology.orchestra.run.smithi099.stdout:Transaction test succeeded. 2023-05-31T19:44:58.245 INFO:teuthology.orchestra.run.smithi099.stdout:Running transaction 2023-05-31T19:44:58.733 INFO:teuthology.orchestra.run.smithi099.stdout: Preparing : 1/1 2023-05-31T19:44:58.895 INFO:teuthology.orchestra.run.smithi112.stdout:Dependencies resolved. 2023-05-31T19:44:58.895 INFO:teuthology.orchestra.run.smithi112.stdout:================================================================================ 2023-05-31T19:44:58.896 INFO:teuthology.orchestra.run.smithi112.stdout: Package Arch Version Repository Size 2023-05-31T19:44:58.896 INFO:teuthology.orchestra.run.smithi112.stdout:================================================================================ 2023-05-31T19:44:58.896 INFO:teuthology.orchestra.run.smithi112.stdout:Installing: 2023-05-31T19:44:58.896 INFO:teuthology.orchestra.run.smithi112.stdout: ceph-fuse x86_64 2:18.0.0-3487.g4a02f3f4.el8 ceph 845 k 2023-05-31T19:44:58.896 INFO:teuthology.orchestra.run.smithi112.stdout:Installing dependencies: 2023-05-31T19:44:58.897 INFO:teuthology.orchestra.run.smithi112.stdout: fuse x86_64 2.9.7-17.el8 baseos 83 k 2023-05-31T19:44:58.897 INFO:teuthology.orchestra.run.smithi112.stdout: 2023-05-31T19:44:58.897 INFO:teuthology.orchestra.run.smithi112.stdout:Transaction Summary 2023-05-31T19:44:58.897 INFO:teuthology.orchestra.run.smithi112.stdout:================================================================================ 2023-05-31T19:44:58.897 INFO:teuthology.orchestra.run.smithi112.stdout:Install 2 Packages 2023-05-31T19:44:58.897 INFO:teuthology.orchestra.run.smithi112.stdout: 2023-05-31T19:44:58.898 INFO:teuthology.orchestra.run.smithi112.stdout:Total download size: 928 k 2023-05-31T19:44:58.898 INFO:teuthology.orchestra.run.smithi112.stdout:Installed size: 3.0 M 2023-05-31T19:44:58.898 INFO:teuthology.orchestra.run.smithi112.stdout:Downloading Packages: 2023-05-31T19:44:58.973 INFO:teuthology.orchestra.run.smithi099.stdout: Installing : python3-pyasn1-0.3.7-6.el8.noarch 1/13 2023-05-31T19:44:59.134 INFO:teuthology.orchestra.run.smithi112.stdout:(1/2): fuse-2.9.7-17.el8.x86_64.rpm 429 kB/s | 83 kB 00:00 2023-05-31T19:44:59.200 INFO:teuthology.orchestra.run.smithi099.stdout: Installing : python3-pyasn1-modules-0.3.7-6.el8.noarch 2/13 2023-05-31T19:44:59.367 INFO:teuthology.orchestra.run.smithi112.stdout:(2/2): ceph-fuse-18.0.0-3487.g4a02f3f4.el8.x86_ 1.9 MB/s | 845 kB 00:00 2023-05-31T19:44:59.368 INFO:teuthology.orchestra.run.smithi112.stdout:-------------------------------------------------------------------------------- 2023-05-31T19:44:59.368 INFO:teuthology.orchestra.run.smithi112.stdout:Total 1.9 MB/s | 928 kB 00:00 2023-05-31T19:44:59.378 INFO:teuthology.orchestra.run.smithi112.stdout:Running transaction check 2023-05-31T19:44:59.400 INFO:teuthology.orchestra.run.smithi112.stdout:Transaction check succeeded. 2023-05-31T19:44:59.400 INFO:teuthology.orchestra.run.smithi112.stdout:Running transaction test 2023-05-31T19:44:59.449 INFO:teuthology.orchestra.run.smithi099.stdout: Installing : python3-rsa-4.8-1.el8.noarch 3/13 2023-05-31T19:44:59.484 INFO:teuthology.orchestra.run.smithi112.stdout:Transaction test succeeded. 2023-05-31T19:44:59.486 INFO:teuthology.orchestra.run.smithi112.stdout:Running transaction 2023-05-31T19:44:59.652 INFO:teuthology.orchestra.run.smithi112.stdout: Preparing : 1/1 2023-05-31T19:44:59.731 INFO:teuthology.orchestra.run.smithi099.stdout: Installing : python3-websocket-client-0.56.0-5.el8.noarch 4/13 2023-05-31T19:44:59.920 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : fuse-2.9.7-17.el8.x86_64 1/2 2023-05-31T19:44:59.922 INFO:teuthology.orchestra.run.smithi099.stdout: Installing : python3-certifi-2018.10.15-7.el8.noarch 5/13 2023-05-31T19:45:00.118 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : ceph-fuse-2:18.0.0-3487.g4a02f3f4.el8.x86_64 2/2 2023-05-31T19:45:00.191 INFO:teuthology.orchestra.run.smithi099.stdout: Installing : python3-cachetools-3.1.1-4.el8.noarch 6/13 2023-05-31T19:45:00.533 INFO:teuthology.orchestra.run.smithi112.stdout: Running scriptlet: ceph-fuse-2:18.0.0-3487.g4a02f3f4.el8.x86_64 2/2 2023-05-31T19:45:00.533 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : ceph-fuse-2:18.0.0-3487.g4a02f3f4.el8.x86_64 1/2 2023-05-31T19:45:00.884 INFO:teuthology.orchestra.run.smithi099.stdout: Installing : python3-google-auth-1:1.1.1-10.el8.noarch 7/13 2023-05-31T19:45:01.088 INFO:teuthology.orchestra.run.smithi099.stdout: Installing : python3-oauthlib-2.1.0-1.el8.noarch 8/13 2023-05-31T19:45:01.094 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : fuse-2.9.7-17.el8.x86_64 2/2 2023-05-31T19:45:01.095 INFO:teuthology.orchestra.run.smithi112.stdout: 2023-05-31T19:45:01.095 INFO:teuthology.orchestra.run.smithi112.stdout:Installed: 2023-05-31T19:45:01.095 INFO:teuthology.orchestra.run.smithi112.stdout: ceph-fuse-2:18.0.0-3487.g4a02f3f4.el8.x86_64 fuse-2.9.7-17.el8.x86_64 2023-05-31T19:45:01.095 INFO:teuthology.orchestra.run.smithi112.stdout: 2023-05-31T19:45:01.095 INFO:teuthology.orchestra.run.smithi112.stdout:Complete! 2023-05-31T19:45:01.282 DEBUG:teuthology.orchestra.run.smithi112:> sudo yum -y install ceph-volume 2023-05-31T19:45:02.021 INFO:teuthology.orchestra.run.smithi112.stdout:Last metadata expiration check: 0:02:16 ago on Wed 31 May 2023 07:42:46 PM UTC. 2023-05-31T19:45:02.140 INFO:teuthology.orchestra.run.smithi099.stdout: Installing : python3-requests-oauthlib-1.0.0-1.el8.noarch 9/13 2023-05-31T19:45:02.405 INFO:teuthology.orchestra.run.smithi099.stdout: Installing : python3-kubernetes-1:11.0.0-6.el8.noarch 10/13 2023-05-31T19:45:02.639 INFO:teuthology.orchestra.run.smithi099.stdout: Installing : python3-jsonpointer-1.10-11.el8.noarch 11/13 2023-05-31T19:45:02.833 INFO:teuthology.orchestra.run.smithi099.stdout: Installing : python3-jsonpatch-1.21-2.el8.noarch 12/13 2023-05-31T19:45:02.990 INFO:teuthology.orchestra.run.smithi099.stdout: Installing : ceph-mgr-rook-2:18.0.0-3487.g4a02f3f4.el8.noarch 13/13 2023-05-31T19:45:03.139 INFO:teuthology.orchestra.run.smithi112.stdout:Dependencies resolved. 2023-05-31T19:45:03.140 INFO:teuthology.orchestra.run.smithi112.stdout:================================================================================ 2023-05-31T19:45:03.140 INFO:teuthology.orchestra.run.smithi112.stdout: Package Arch Version Repository Size 2023-05-31T19:45:03.140 INFO:teuthology.orchestra.run.smithi112.stdout:================================================================================ 2023-05-31T19:45:03.140 INFO:teuthology.orchestra.run.smithi112.stdout:Installing: 2023-05-31T19:45:03.141 INFO:teuthology.orchestra.run.smithi112.stdout: ceph-volume noarch 2:18.0.0-3487.g4a02f3f4.el8 ceph-noarch 264 k 2023-05-31T19:45:03.141 INFO:teuthology.orchestra.run.smithi112.stdout: 2023-05-31T19:45:03.141 INFO:teuthology.orchestra.run.smithi112.stdout:Transaction Summary 2023-05-31T19:45:03.141 INFO:teuthology.orchestra.run.smithi112.stdout:================================================================================ 2023-05-31T19:45:03.141 INFO:teuthology.orchestra.run.smithi112.stdout:Install 1 Package 2023-05-31T19:45:03.141 INFO:teuthology.orchestra.run.smithi112.stdout: 2023-05-31T19:45:03.141 INFO:teuthology.orchestra.run.smithi112.stdout:Total download size: 264 k 2023-05-31T19:45:03.141 INFO:teuthology.orchestra.run.smithi112.stdout:Installed size: 1.3 M 2023-05-31T19:45:03.142 INFO:teuthology.orchestra.run.smithi112.stdout:Downloading Packages: 2023-05-31T19:45:03.294 INFO:teuthology.orchestra.run.smithi099.stdout: Running scriptlet: ceph-mgr-rook-2:18.0.0-3487.g4a02f3f4.el8.noarch 13/13 2023-05-31T19:45:03.294 INFO:teuthology.orchestra.run.smithi099.stdout: Verifying : ceph-mgr-rook-2:18.0.0-3487.g4a02f3f4.el8.noarch 1/13 2023-05-31T19:45:03.295 INFO:teuthology.orchestra.run.smithi099.stdout: Verifying : python3-jsonpatch-1.21-2.el8.noarch 2/13 2023-05-31T19:45:03.295 INFO:teuthology.orchestra.run.smithi099.stdout: Verifying : python3-jsonpointer-1.10-11.el8.noarch 3/13 2023-05-31T19:45:03.295 INFO:teuthology.orchestra.run.smithi099.stdout: Verifying : python3-pyasn1-0.3.7-6.el8.noarch 4/13 2023-05-31T19:45:03.295 INFO:teuthology.orchestra.run.smithi099.stdout: Verifying : python3-pyasn1-modules-0.3.7-6.el8.noarch 5/13 2023-05-31T19:45:03.295 INFO:teuthology.orchestra.run.smithi099.stdout: Verifying : python3-oauthlib-2.1.0-1.el8.noarch 6/13 2023-05-31T19:45:03.295 INFO:teuthology.orchestra.run.smithi099.stdout: Verifying : python3-requests-oauthlib-1.0.0-1.el8.noarch 7/13 2023-05-31T19:45:03.295 INFO:teuthology.orchestra.run.smithi099.stdout: Verifying : python3-cachetools-3.1.1-4.el8.noarch 8/13 2023-05-31T19:45:03.296 INFO:teuthology.orchestra.run.smithi099.stdout: Verifying : python3-certifi-2018.10.15-7.el8.noarch 9/13 2023-05-31T19:45:03.296 INFO:teuthology.orchestra.run.smithi099.stdout: Verifying : python3-google-auth-1:1.1.1-10.el8.noarch 10/13 2023-05-31T19:45:03.296 INFO:teuthology.orchestra.run.smithi099.stdout: Verifying : python3-kubernetes-1:11.0.0-6.el8.noarch 11/13 2023-05-31T19:45:03.296 INFO:teuthology.orchestra.run.smithi099.stdout: Verifying : python3-rsa-4.8-1.el8.noarch 12/13 2023-05-31T19:45:03.503 INFO:teuthology.orchestra.run.smithi112.stdout:ceph-volume-18.0.0-3487.g4a02f3f4.el8.noarch.rp 728 kB/s | 264 kB 00:00 2023-05-31T19:45:03.504 INFO:teuthology.orchestra.run.smithi112.stdout:-------------------------------------------------------------------------------- 2023-05-31T19:45:03.504 INFO:teuthology.orchestra.run.smithi112.stdout:Total 725 kB/s | 264 kB 00:00 2023-05-31T19:45:03.505 INFO:teuthology.orchestra.run.smithi112.stdout:Running transaction check 2023-05-31T19:45:03.515 INFO:teuthology.orchestra.run.smithi112.stdout:Transaction check succeeded. 2023-05-31T19:45:03.515 INFO:teuthology.orchestra.run.smithi112.stdout:Running transaction test 2023-05-31T19:45:03.534 INFO:teuthology.orchestra.run.smithi112.stdout:Transaction test succeeded. 2023-05-31T19:45:03.537 INFO:teuthology.orchestra.run.smithi112.stdout:Running transaction 2023-05-31T19:45:03.708 INFO:teuthology.orchestra.run.smithi112.stdout: Preparing : 1/1 2023-05-31T19:45:03.914 INFO:teuthology.orchestra.run.smithi099.stdout: Verifying : python3-websocket-client-0.56.0-5.el8.noarch 13/13 2023-05-31T19:45:03.915 INFO:teuthology.orchestra.run.smithi099.stdout: 2023-05-31T19:45:03.915 INFO:teuthology.orchestra.run.smithi099.stdout:Installed: 2023-05-31T19:45:03.915 INFO:teuthology.orchestra.run.smithi099.stdout: ceph-mgr-rook-2:18.0.0-3487.g4a02f3f4.el8.noarch 2023-05-31T19:45:03.916 INFO:teuthology.orchestra.run.smithi099.stdout: python3-cachetools-3.1.1-4.el8.noarch 2023-05-31T19:45:03.916 INFO:teuthology.orchestra.run.smithi099.stdout: python3-certifi-2018.10.15-7.el8.noarch 2023-05-31T19:45:03.916 INFO:teuthology.orchestra.run.smithi099.stdout: python3-google-auth-1:1.1.1-10.el8.noarch 2023-05-31T19:45:03.917 INFO:teuthology.orchestra.run.smithi099.stdout: python3-jsonpatch-1.21-2.el8.noarch 2023-05-31T19:45:03.917 INFO:teuthology.orchestra.run.smithi099.stdout: python3-jsonpointer-1.10-11.el8.noarch 2023-05-31T19:45:03.917 INFO:teuthology.orchestra.run.smithi099.stdout: python3-kubernetes-1:11.0.0-6.el8.noarch 2023-05-31T19:45:03.917 INFO:teuthology.orchestra.run.smithi099.stdout: python3-oauthlib-2.1.0-1.el8.noarch 2023-05-31T19:45:03.918 INFO:teuthology.orchestra.run.smithi099.stdout: python3-pyasn1-0.3.7-6.el8.noarch 2023-05-31T19:45:03.918 INFO:teuthology.orchestra.run.smithi099.stdout: python3-pyasn1-modules-0.3.7-6.el8.noarch 2023-05-31T19:45:03.919 INFO:teuthology.orchestra.run.smithi099.stdout: python3-requests-oauthlib-1.0.0-1.el8.noarch 2023-05-31T19:45:03.919 INFO:teuthology.orchestra.run.smithi099.stdout: python3-rsa-4.8-1.el8.noarch 2023-05-31T19:45:03.919 INFO:teuthology.orchestra.run.smithi099.stdout: python3-websocket-client-0.56.0-5.el8.noarch 2023-05-31T19:45:03.920 INFO:teuthology.orchestra.run.smithi099.stdout: 2023-05-31T19:45:03.920 INFO:teuthology.orchestra.run.smithi099.stdout:Complete! 2023-05-31T19:45:04.001 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : ceph-volume-2:18.0.0-3487.g4a02f3f4.el8.noarch 1/1 2023-05-31T19:45:04.035 DEBUG:teuthology.orchestra.run.smithi099:> sudo yum -y install ceph-mgr-cephadm 2023-05-31T19:45:04.412 INFO:teuthology.orchestra.run.smithi112.stdout: Running scriptlet: ceph-volume-2:18.0.0-3487.g4a02f3f4.el8.noarch 1/1 2023-05-31T19:45:04.766 INFO:teuthology.orchestra.run.smithi099.stdout:Last metadata expiration check: 0:02:04 ago on Wed 31 May 2023 07:43:00 PM UTC. 2023-05-31T19:45:04.945 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : ceph-volume-2:18.0.0-3487.g4a02f3f4.el8.noarch 1/1 2023-05-31T19:45:04.945 INFO:teuthology.orchestra.run.smithi112.stdout: 2023-05-31T19:45:04.945 INFO:teuthology.orchestra.run.smithi112.stdout:Installed: 2023-05-31T19:45:04.945 INFO:teuthology.orchestra.run.smithi112.stdout: ceph-volume-2:18.0.0-3487.g4a02f3f4.el8.noarch 2023-05-31T19:45:04.945 INFO:teuthology.orchestra.run.smithi112.stdout: 2023-05-31T19:45:04.946 INFO:teuthology.orchestra.run.smithi112.stdout:Complete! 2023-05-31T19:45:05.136 DEBUG:teuthology.orchestra.run.smithi112:> sudo yum -y install librados-devel 2023-05-31T19:45:05.867 INFO:teuthology.orchestra.run.smithi112.stdout:Last metadata expiration check: 0:02:19 ago on Wed 31 May 2023 07:42:46 PM UTC. 2023-05-31T19:45:05.886 INFO:teuthology.orchestra.run.smithi099.stdout:Dependencies resolved. 2023-05-31T19:45:05.887 INFO:teuthology.orchestra.run.smithi099.stdout:================================================================================ 2023-05-31T19:45:05.887 INFO:teuthology.orchestra.run.smithi099.stdout: Package Arch Version Repository Size 2023-05-31T19:45:05.887 INFO:teuthology.orchestra.run.smithi099.stdout:================================================================================ 2023-05-31T19:45:05.887 INFO:teuthology.orchestra.run.smithi099.stdout:Installing: 2023-05-31T19:45:05.887 INFO:teuthology.orchestra.run.smithi099.stdout: ceph-mgr-cephadm noarch 2:18.0.0-3487.g4a02f3f4.el8 ceph-noarch 127 k 2023-05-31T19:45:05.887 INFO:teuthology.orchestra.run.smithi099.stdout:Installing dependencies: 2023-05-31T19:45:05.888 INFO:teuthology.orchestra.run.smithi099.stdout: python3-asyncssh noarch 2.7.0-2.el8 epel 476 k 2023-05-31T19:45:05.888 INFO:teuthology.orchestra.run.smithi099.stdout: python3-babel noarch 2.5.1-7.el8 appstream 4.8 M 2023-05-31T19:45:05.888 INFO:teuthology.orchestra.run.smithi099.stdout: python3-jinja2 noarch 2.10.1-3.el8 appstream 538 k 2023-05-31T19:45:05.888 INFO:teuthology.orchestra.run.smithi099.stdout: python3-natsort noarch 7.1.1-2.el8 epel 60 k 2023-05-31T19:45:05.888 INFO:teuthology.orchestra.run.smithi099.stdout: 2023-05-31T19:45:05.888 INFO:teuthology.orchestra.run.smithi099.stdout:Transaction Summary 2023-05-31T19:45:05.888 INFO:teuthology.orchestra.run.smithi099.stdout:================================================================================ 2023-05-31T19:45:05.889 INFO:teuthology.orchestra.run.smithi099.stdout:Install 5 Packages 2023-05-31T19:45:05.889 INFO:teuthology.orchestra.run.smithi099.stdout: 2023-05-31T19:45:05.889 INFO:teuthology.orchestra.run.smithi099.stdout:Total download size: 6.0 M 2023-05-31T19:45:05.889 INFO:teuthology.orchestra.run.smithi099.stdout:Installed size: 26 M 2023-05-31T19:45:05.890 INFO:teuthology.orchestra.run.smithi099.stdout:Downloading Packages: 2023-05-31T19:45:06.826 INFO:teuthology.orchestra.run.smithi099.stdout:(1/5): python3-jinja2-2.10.1-3.el8.noarch.rpm 3.6 MB/s | 538 kB 00:00 2023-05-31T19:45:06.851 INFO:teuthology.orchestra.run.smithi099.stdout:(2/5): ceph-mgr-cephadm-18.0.0-3487.g4a02f3f4.e 745 kB/s | 127 kB 00:00 2023-05-31T19:45:06.982 INFO:teuthology.orchestra.run.smithi112.stdout:Dependencies resolved. 2023-05-31T19:45:06.984 INFO:teuthology.orchestra.run.smithi112.stdout:================================================================================ 2023-05-31T19:45:06.984 INFO:teuthology.orchestra.run.smithi112.stdout: Package Arch Version Repo Size 2023-05-31T19:45:06.985 INFO:teuthology.orchestra.run.smithi112.stdout:================================================================================ 2023-05-31T19:45:06.985 INFO:teuthology.orchestra.run.smithi112.stdout:Installing: 2023-05-31T19:45:06.985 INFO:teuthology.orchestra.run.smithi112.stdout: librados-devel x86_64 2:18.0.0-3487.g4a02f3f4.el8 ceph 139 k 2023-05-31T19:45:06.985 INFO:teuthology.orchestra.run.smithi112.stdout: 2023-05-31T19:45:06.985 INFO:teuthology.orchestra.run.smithi112.stdout:Transaction Summary 2023-05-31T19:45:06.986 INFO:teuthology.orchestra.run.smithi112.stdout:================================================================================ 2023-05-31T19:45:06.986 INFO:teuthology.orchestra.run.smithi112.stdout:Install 1 Package 2023-05-31T19:45:06.986 INFO:teuthology.orchestra.run.smithi112.stdout: 2023-05-31T19:45:06.986 INFO:teuthology.orchestra.run.smithi112.stdout:Total download size: 139 k 2023-05-31T19:45:06.987 INFO:teuthology.orchestra.run.smithi112.stdout:Installed size: 512 k 2023-05-31T19:45:06.987 INFO:teuthology.orchestra.run.smithi112.stdout:Downloading Packages: 2023-05-31T19:45:07.168 INFO:teuthology.orchestra.run.smithi099.stdout:(3/5): python3-babel-2.5.1-7.el8.noarch.rpm 9.8 MB/s | 4.8 MB 00:00 2023-05-31T19:45:07.207 INFO:teuthology.orchestra.run.smithi112.stdout:librados-devel-18.0.0-3487.g4a02f3f4.el8.x86_64 624 kB/s | 139 kB 00:00 2023-05-31T19:45:07.208 INFO:teuthology.orchestra.run.smithi112.stdout:-------------------------------------------------------------------------------- 2023-05-31T19:45:07.208 INFO:teuthology.orchestra.run.smithi112.stdout:Total 620 kB/s | 139 kB 00:00 2023-05-31T19:45:07.209 INFO:teuthology.orchestra.run.smithi112.stdout:Running transaction check 2023-05-31T19:45:07.213 INFO:teuthology.orchestra.run.smithi112.stdout:Transaction check succeeded. 2023-05-31T19:45:07.214 INFO:teuthology.orchestra.run.smithi112.stdout:Running transaction test 2023-05-31T19:45:07.262 INFO:teuthology.orchestra.run.smithi112.stdout:Transaction test succeeded. 2023-05-31T19:45:07.265 INFO:teuthology.orchestra.run.smithi112.stdout:Running transaction 2023-05-31T19:45:07.376 INFO:teuthology.orchestra.run.smithi112.stdout: Preparing : 1/1 2023-05-31T19:45:07.608 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : librados-devel-2:18.0.0-3487.g4a02f3f4.el8.x86_64 1/1 2023-05-31T19:45:07.901 INFO:teuthology.orchestra.run.smithi112.stdout: Running scriptlet: librados-devel-2:18.0.0-3487.g4a02f3f4.el8.x86_64 1/1 2023-05-31T19:45:08.404 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : librados-devel-2:18.0.0-3487.g4a02f3f4.el8.x86_64 1/1 2023-05-31T19:45:08.404 INFO:teuthology.orchestra.run.smithi112.stdout: 2023-05-31T19:45:08.404 INFO:teuthology.orchestra.run.smithi112.stdout:Installed: 2023-05-31T19:45:08.404 INFO:teuthology.orchestra.run.smithi112.stdout: librados-devel-2:18.0.0-3487.g4a02f3f4.el8.x86_64 2023-05-31T19:45:08.404 INFO:teuthology.orchestra.run.smithi112.stdout: 2023-05-31T19:45:08.405 INFO:teuthology.orchestra.run.smithi112.stdout:Complete! 2023-05-31T19:45:08.606 DEBUG:teuthology.orchestra.run.smithi112:> sudo yum -y install libcephfs2 2023-05-31T19:45:09.340 INFO:teuthology.orchestra.run.smithi112.stdout:Last metadata expiration check: 0:02:23 ago on Wed 31 May 2023 07:42:46 PM UTC. 2023-05-31T19:45:10.327 INFO:teuthology.orchestra.run.smithi112.stdout:Package libcephfs2-2:18.0.0-3487.g4a02f3f4.el8.x86_64 is already installed. 2023-05-31T19:45:10.467 INFO:teuthology.orchestra.run.smithi112.stdout:Dependencies resolved. 2023-05-31T19:45:10.468 INFO:teuthology.orchestra.run.smithi112.stdout:Nothing to do. 2023-05-31T19:45:10.468 INFO:teuthology.orchestra.run.smithi112.stdout:Complete! 2023-05-31T19:45:10.565 DEBUG:teuthology.orchestra.run.smithi112:> sudo yum -y install libcephfs-devel 2023-05-31T19:45:11.288 INFO:teuthology.orchestra.run.smithi112.stdout:Last metadata expiration check: 0:02:25 ago on Wed 31 May 2023 07:42:46 PM UTC. 2023-05-31T19:45:12.410 INFO:teuthology.orchestra.run.smithi112.stdout:Dependencies resolved. 2023-05-31T19:45:12.411 INFO:teuthology.orchestra.run.smithi112.stdout:================================================================================ 2023-05-31T19:45:12.411 INFO:teuthology.orchestra.run.smithi112.stdout: Package Arch Version Repo Size 2023-05-31T19:45:12.411 INFO:teuthology.orchestra.run.smithi112.stdout:================================================================================ 2023-05-31T19:45:12.412 INFO:teuthology.orchestra.run.smithi112.stdout:Installing: 2023-05-31T19:45:12.412 INFO:teuthology.orchestra.run.smithi112.stdout: libcephfs-devel x86_64 2:18.0.0-3487.g4a02f3f4.el8 ceph 30 k 2023-05-31T19:45:12.412 INFO:teuthology.orchestra.run.smithi112.stdout: 2023-05-31T19:45:12.412 INFO:teuthology.orchestra.run.smithi112.stdout:Transaction Summary 2023-05-31T19:45:12.412 INFO:teuthology.orchestra.run.smithi112.stdout:================================================================================ 2023-05-31T19:45:12.412 INFO:teuthology.orchestra.run.smithi112.stdout:Install 1 Package 2023-05-31T19:45:12.413 INFO:teuthology.orchestra.run.smithi112.stdout: 2023-05-31T19:45:12.414 INFO:teuthology.orchestra.run.smithi112.stdout:Total download size: 30 k 2023-05-31T19:45:12.414 INFO:teuthology.orchestra.run.smithi112.stdout:Installed size: 137 k 2023-05-31T19:45:12.414 INFO:teuthology.orchestra.run.smithi112.stdout:Downloading Packages: 2023-05-31T19:45:13.180 INFO:teuthology.orchestra.run.smithi112.stdout:libcephfs-devel-18.0.0-3487.g4a02f3f4.el8.x86_6 40 kB/s | 30 kB 00:00 2023-05-31T19:45:13.181 INFO:teuthology.orchestra.run.smithi112.stdout:-------------------------------------------------------------------------------- 2023-05-31T19:45:13.181 INFO:teuthology.orchestra.run.smithi112.stdout:Total 40 kB/s | 30 kB 00:00 2023-05-31T19:45:13.182 INFO:teuthology.orchestra.run.smithi112.stdout:Running transaction check 2023-05-31T19:45:13.186 INFO:teuthology.orchestra.run.smithi112.stdout:Transaction check succeeded. 2023-05-31T19:45:13.186 INFO:teuthology.orchestra.run.smithi112.stdout:Running transaction test 2023-05-31T19:45:13.235 INFO:teuthology.orchestra.run.smithi112.stdout:Transaction test succeeded. 2023-05-31T19:45:13.238 INFO:teuthology.orchestra.run.smithi112.stdout:Running transaction 2023-05-31T19:45:13.299 INFO:teuthology.orchestra.run.smithi112.stdout: Preparing : 1/1 2023-05-31T19:45:13.490 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : libcephfs-devel-2:18.0.0-3487.g4a02f3f4.el8.x86_64 1/1 2023-05-31T19:45:13.769 INFO:teuthology.orchestra.run.smithi112.stdout: Running scriptlet: libcephfs-devel-2:18.0.0-3487.g4a02f3f4.el8.x86_64 1/1 2023-05-31T19:45:14.295 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : libcephfs-devel-2:18.0.0-3487.g4a02f3f4.el8.x86_64 1/1 2023-05-31T19:45:14.295 INFO:teuthology.orchestra.run.smithi112.stdout: 2023-05-31T19:45:14.295 INFO:teuthology.orchestra.run.smithi112.stdout:Installed: 2023-05-31T19:45:14.295 INFO:teuthology.orchestra.run.smithi112.stdout: libcephfs-devel-2:18.0.0-3487.g4a02f3f4.el8.x86_64 2023-05-31T19:45:14.296 INFO:teuthology.orchestra.run.smithi112.stdout: 2023-05-31T19:45:14.296 INFO:teuthology.orchestra.run.smithi112.stdout:Complete! 2023-05-31T19:45:14.397 DEBUG:teuthology.orchestra.run.smithi112:> sudo yum -y install librados2 2023-05-31T19:45:15.132 INFO:teuthology.orchestra.run.smithi112.stdout:Last metadata expiration check: 0:02:29 ago on Wed 31 May 2023 07:42:46 PM UTC. 2023-05-31T19:45:16.111 INFO:teuthology.orchestra.run.smithi112.stdout:Package librados2-2:18.0.0-3487.g4a02f3f4.el8.x86_64 is already installed. 2023-05-31T19:45:16.250 INFO:teuthology.orchestra.run.smithi112.stdout:Dependencies resolved. 2023-05-31T19:45:16.251 INFO:teuthology.orchestra.run.smithi112.stdout:Nothing to do. 2023-05-31T19:45:16.251 INFO:teuthology.orchestra.run.smithi112.stdout:Complete! 2023-05-31T19:45:16.346 DEBUG:teuthology.orchestra.run.smithi112:> sudo yum -y install librbd1 2023-05-31T19:45:17.074 INFO:teuthology.orchestra.run.smithi112.stdout:Last metadata expiration check: 0:02:31 ago on Wed 31 May 2023 07:42:46 PM UTC. 2023-05-31T19:45:18.050 INFO:teuthology.orchestra.run.smithi112.stdout:Package librbd1-2:18.0.0-3487.g4a02f3f4.el8.x86_64 is already installed. 2023-05-31T19:45:18.189 INFO:teuthology.orchestra.run.smithi112.stdout:Dependencies resolved. 2023-05-31T19:45:18.191 INFO:teuthology.orchestra.run.smithi112.stdout:Nothing to do. 2023-05-31T19:45:18.191 INFO:teuthology.orchestra.run.smithi112.stdout:Complete! 2023-05-31T19:45:18.283 DEBUG:teuthology.orchestra.run.smithi112:> sudo yum -y install python3-rados 2023-05-31T19:45:19.005 INFO:teuthology.orchestra.run.smithi112.stdout:Last metadata expiration check: 0:02:32 ago on Wed 31 May 2023 07:42:46 PM UTC. 2023-05-31T19:45:19.980 INFO:teuthology.orchestra.run.smithi112.stdout:Package python3-rados-2:18.0.0-3487.g4a02f3f4.el8.x86_64 is already installed. 2023-05-31T19:45:20.121 INFO:teuthology.orchestra.run.smithi112.stdout:Dependencies resolved. 2023-05-31T19:45:20.122 INFO:teuthology.orchestra.run.smithi112.stdout:Nothing to do. 2023-05-31T19:45:20.123 INFO:teuthology.orchestra.run.smithi112.stdout:Complete! 2023-05-31T19:45:20.215 DEBUG:teuthology.orchestra.run.smithi112:> sudo yum -y install python3-rgw 2023-05-31T19:45:20.953 INFO:teuthology.orchestra.run.smithi112.stdout:Last metadata expiration check: 0:02:34 ago on Wed 31 May 2023 07:42:46 PM UTC. 2023-05-31T19:45:21.914 INFO:teuthology.orchestra.run.smithi112.stdout:Package python3-rgw-2:18.0.0-3487.g4a02f3f4.el8.x86_64 is already installed. 2023-05-31T19:45:22.056 INFO:teuthology.orchestra.run.smithi112.stdout:Dependencies resolved. 2023-05-31T19:45:22.058 INFO:teuthology.orchestra.run.smithi112.stdout:Nothing to do. 2023-05-31T19:45:22.058 INFO:teuthology.orchestra.run.smithi112.stdout:Complete! 2023-05-31T19:45:22.150 DEBUG:teuthology.orchestra.run.smithi112:> sudo yum -y install python3-cephfs 2023-05-31T19:45:22.875 INFO:teuthology.orchestra.run.smithi112.stdout:Last metadata expiration check: 0:02:36 ago on Wed 31 May 2023 07:42:46 PM UTC. 2023-05-31T19:45:23.852 INFO:teuthology.orchestra.run.smithi112.stdout:Package python3-cephfs-2:18.0.0-3487.g4a02f3f4.el8.x86_64 is already installed. 2023-05-31T19:45:23.992 INFO:teuthology.orchestra.run.smithi112.stdout:Dependencies resolved. 2023-05-31T19:45:23.993 INFO:teuthology.orchestra.run.smithi112.stdout:Nothing to do. 2023-05-31T19:45:23.993 INFO:teuthology.orchestra.run.smithi112.stdout:Complete! 2023-05-31T19:45:24.087 DEBUG:teuthology.orchestra.run.smithi112:> sudo yum -y install python3-rbd 2023-05-31T19:45:24.820 INFO:teuthology.orchestra.run.smithi112.stdout:Last metadata expiration check: 0:02:38 ago on Wed 31 May 2023 07:42:46 PM UTC. 2023-05-31T19:45:25.804 INFO:teuthology.orchestra.run.smithi112.stdout:Package python3-rbd-2:18.0.0-3487.g4a02f3f4.el8.x86_64 is already installed. 2023-05-31T19:45:25.944 INFO:teuthology.orchestra.run.smithi112.stdout:Dependencies resolved. 2023-05-31T19:45:25.945 INFO:teuthology.orchestra.run.smithi112.stdout:Nothing to do. 2023-05-31T19:45:25.945 INFO:teuthology.orchestra.run.smithi112.stdout:Complete! 2023-05-31T19:45:26.052 DEBUG:teuthology.orchestra.run.smithi112:> sudo yum -y install rbd-fuse 2023-05-31T19:45:26.781 INFO:teuthology.orchestra.run.smithi112.stdout:Last metadata expiration check: 0:02:40 ago on Wed 31 May 2023 07:42:46 PM UTC. 2023-05-31T19:45:27.889 INFO:teuthology.orchestra.run.smithi112.stdout:Dependencies resolved. 2023-05-31T19:45:27.890 INFO:teuthology.orchestra.run.smithi112.stdout:================================================================================ 2023-05-31T19:45:27.891 INFO:teuthology.orchestra.run.smithi112.stdout: Package Architecture Version Repository Size 2023-05-31T19:45:27.891 INFO:teuthology.orchestra.run.smithi112.stdout:================================================================================ 2023-05-31T19:45:27.892 INFO:teuthology.orchestra.run.smithi112.stdout:Installing: 2023-05-31T19:45:27.892 INFO:teuthology.orchestra.run.smithi112.stdout: rbd-fuse x86_64 2:18.0.0-3487.g4a02f3f4.el8 ceph 94 k 2023-05-31T19:45:27.892 INFO:teuthology.orchestra.run.smithi112.stdout: 2023-05-31T19:45:27.893 INFO:teuthology.orchestra.run.smithi112.stdout:Transaction Summary 2023-05-31T19:45:27.893 INFO:teuthology.orchestra.run.smithi112.stdout:================================================================================ 2023-05-31T19:45:27.894 INFO:teuthology.orchestra.run.smithi112.stdout:Install 1 Package 2023-05-31T19:45:27.894 INFO:teuthology.orchestra.run.smithi112.stdout: 2023-05-31T19:45:27.895 INFO:teuthology.orchestra.run.smithi112.stdout:Total download size: 94 k 2023-05-31T19:45:27.895 INFO:teuthology.orchestra.run.smithi112.stdout:Installed size: 280 k 2023-05-31T19:45:27.895 INFO:teuthology.orchestra.run.smithi112.stdout:Downloading Packages: 2023-05-31T19:45:28.193 INFO:teuthology.orchestra.run.smithi112.stdout:rbd-fuse-18.0.0-3487.g4a02f3f4.el8.x86_64.rpm 313 kB/s | 94 kB 00:00 2023-05-31T19:45:28.193 INFO:teuthology.orchestra.run.smithi112.stdout:-------------------------------------------------------------------------------- 2023-05-31T19:45:28.193 INFO:teuthology.orchestra.run.smithi112.stdout:Total 311 kB/s | 94 kB 00:00 2023-05-31T19:45:28.194 INFO:teuthology.orchestra.run.smithi112.stdout:Running transaction check 2023-05-31T19:45:28.204 INFO:teuthology.orchestra.run.smithi112.stdout:Transaction check succeeded. 2023-05-31T19:45:28.204 INFO:teuthology.orchestra.run.smithi112.stdout:Running transaction test 2023-05-31T19:45:28.254 INFO:teuthology.orchestra.run.smithi112.stdout:Transaction test succeeded. 2023-05-31T19:45:28.257 INFO:teuthology.orchestra.run.smithi112.stdout:Running transaction 2023-05-31T19:45:28.356 INFO:teuthology.orchestra.run.smithi112.stdout: Preparing : 1/1 2023-05-31T19:45:28.555 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : rbd-fuse-2:18.0.0-3487.g4a02f3f4.el8.x86_64 1/1 2023-05-31T19:45:28.868 INFO:teuthology.orchestra.run.smithi112.stdout: Running scriptlet: rbd-fuse-2:18.0.0-3487.g4a02f3f4.el8.x86_64 1/1 2023-05-31T19:45:29.410 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : rbd-fuse-2:18.0.0-3487.g4a02f3f4.el8.x86_64 1/1 2023-05-31T19:45:29.410 INFO:teuthology.orchestra.run.smithi112.stdout: 2023-05-31T19:45:29.410 INFO:teuthology.orchestra.run.smithi112.stdout:Installed: 2023-05-31T19:45:29.410 INFO:teuthology.orchestra.run.smithi112.stdout: rbd-fuse-2:18.0.0-3487.g4a02f3f4.el8.x86_64 2023-05-31T19:45:29.411 INFO:teuthology.orchestra.run.smithi112.stdout: 2023-05-31T19:45:29.411 INFO:teuthology.orchestra.run.smithi112.stdout:Complete! 2023-05-31T19:45:29.847 DEBUG:teuthology.orchestra.run.smithi112:> sudo yum -y install rbd-mirror 2023-05-31T19:45:30.595 INFO:teuthology.orchestra.run.smithi112.stdout:Last metadata expiration check: 0:02:44 ago on Wed 31 May 2023 07:42:46 PM UTC. 2023-05-31T19:45:31.711 INFO:teuthology.orchestra.run.smithi112.stdout:Dependencies resolved. 2023-05-31T19:45:31.712 INFO:teuthology.orchestra.run.smithi112.stdout:================================================================================ 2023-05-31T19:45:31.712 INFO:teuthology.orchestra.run.smithi112.stdout: Package Arch Version Repo Size 2023-05-31T19:45:31.712 INFO:teuthology.orchestra.run.smithi112.stdout:================================================================================ 2023-05-31T19:45:31.712 INFO:teuthology.orchestra.run.smithi112.stdout:Installing: 2023-05-31T19:45:31.713 INFO:teuthology.orchestra.run.smithi112.stdout: rbd-mirror x86_64 2:18.0.0-3487.g4a02f3f4.el8 ceph 3.4 M 2023-05-31T19:45:31.713 INFO:teuthology.orchestra.run.smithi112.stdout: 2023-05-31T19:45:31.713 INFO:teuthology.orchestra.run.smithi112.stdout:Transaction Summary 2023-05-31T19:45:31.713 INFO:teuthology.orchestra.run.smithi112.stdout:================================================================================ 2023-05-31T19:45:31.713 INFO:teuthology.orchestra.run.smithi112.stdout:Install 1 Package 2023-05-31T19:45:31.713 INFO:teuthology.orchestra.run.smithi112.stdout: 2023-05-31T19:45:31.714 INFO:teuthology.orchestra.run.smithi112.stdout:Total download size: 3.4 M 2023-05-31T19:45:31.714 INFO:teuthology.orchestra.run.smithi112.stdout:Installed size: 16 M 2023-05-31T19:45:31.714 INFO:teuthology.orchestra.run.smithi112.stdout:Downloading Packages: 2023-05-31T19:45:32.489 INFO:teuthology.orchestra.run.smithi112.stdout:rbd-mirror-18.0.0-3487.g4a02f3f4.el8.x86_64.rpm 4.4 MB/s | 3.4 MB 00:00 2023-05-31T19:45:32.490 INFO:teuthology.orchestra.run.smithi112.stdout:-------------------------------------------------------------------------------- 2023-05-31T19:45:32.490 INFO:teuthology.orchestra.run.smithi112.stdout:Total 4.4 MB/s | 3.4 MB 00:00 2023-05-31T19:45:32.490 INFO:teuthology.orchestra.run.smithi112.stdout:Running transaction check 2023-05-31T19:45:32.503 INFO:teuthology.orchestra.run.smithi112.stdout:Transaction check succeeded. 2023-05-31T19:45:32.503 INFO:teuthology.orchestra.run.smithi112.stdout:Running transaction test 2023-05-31T19:45:32.564 INFO:teuthology.orchestra.run.smithi112.stdout:Transaction test succeeded. 2023-05-31T19:45:32.613 INFO:teuthology.orchestra.run.smithi112.stdout:Running transaction 2023-05-31T19:45:33.002 INFO:teuthology.orchestra.run.smithi112.stdout: Preparing : 1/1 2023-05-31T19:45:33.189 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : rbd-mirror-2:18.0.0-3487.g4a02f3f4.el8.x86_64 1/1 2023-05-31T19:45:33.632 INFO:teuthology.orchestra.run.smithi112.stdout: Running scriptlet: rbd-mirror-2:18.0.0-3487.g4a02f3f4.el8.x86_64 1/1 2023-05-31T19:45:34.163 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : rbd-mirror-2:18.0.0-3487.g4a02f3f4.el8.x86_64 1/1 2023-05-31T19:45:34.163 INFO:teuthology.orchestra.run.smithi112.stdout: 2023-05-31T19:45:34.164 INFO:teuthology.orchestra.run.smithi112.stdout:Installed: 2023-05-31T19:45:34.164 INFO:teuthology.orchestra.run.smithi112.stdout: rbd-mirror-2:18.0.0-3487.g4a02f3f4.el8.x86_64 2023-05-31T19:45:34.164 INFO:teuthology.orchestra.run.smithi112.stdout: 2023-05-31T19:45:34.164 INFO:teuthology.orchestra.run.smithi112.stdout:Complete! 2023-05-31T19:45:34.356 DEBUG:teuthology.orchestra.run.smithi112:> sudo yum -y install rbd-nbd 2023-05-31T19:45:35.091 INFO:teuthology.orchestra.run.smithi112.stdout:Last metadata expiration check: 0:02:49 ago on Wed 31 May 2023 07:42:46 PM UTC. 2023-05-31T19:45:36.212 INFO:teuthology.orchestra.run.smithi112.stdout:Dependencies resolved. 2023-05-31T19:45:36.212 INFO:teuthology.orchestra.run.smithi112.stdout:================================================================================ 2023-05-31T19:45:36.212 INFO:teuthology.orchestra.run.smithi112.stdout: Package Architecture Version Repository Size 2023-05-31T19:45:36.212 INFO:teuthology.orchestra.run.smithi112.stdout:================================================================================ 2023-05-31T19:45:36.213 INFO:teuthology.orchestra.run.smithi112.stdout:Installing: 2023-05-31T19:45:36.213 INFO:teuthology.orchestra.run.smithi112.stdout: rbd-nbd x86_64 2:18.0.0-3487.g4a02f3f4.el8 ceph 179 k 2023-05-31T19:45:36.213 INFO:teuthology.orchestra.run.smithi112.stdout: 2023-05-31T19:45:36.213 INFO:teuthology.orchestra.run.smithi112.stdout:Transaction Summary 2023-05-31T19:45:36.213 INFO:teuthology.orchestra.run.smithi112.stdout:================================================================================ 2023-05-31T19:45:36.213 INFO:teuthology.orchestra.run.smithi112.stdout:Install 1 Package 2023-05-31T19:45:36.213 INFO:teuthology.orchestra.run.smithi112.stdout: 2023-05-31T19:45:36.214 INFO:teuthology.orchestra.run.smithi112.stdout:Total download size: 179 k 2023-05-31T19:45:36.214 INFO:teuthology.orchestra.run.smithi112.stdout:Installed size: 568 k 2023-05-31T19:45:36.214 INFO:teuthology.orchestra.run.smithi112.stdout:Downloading Packages: 2023-05-31T19:45:36.408 INFO:teuthology.orchestra.run.smithi112.stdout:rbd-nbd-18.0.0-3487.g4a02f3f4.el8.x86_64.rpm 919 kB/s | 179 kB 00:00 2023-05-31T19:45:36.408 INFO:teuthology.orchestra.run.smithi112.stdout:-------------------------------------------------------------------------------- 2023-05-31T19:45:36.409 INFO:teuthology.orchestra.run.smithi112.stdout:Total 913 kB/s | 179 kB 00:00 2023-05-31T19:45:36.409 INFO:teuthology.orchestra.run.smithi112.stdout:Running transaction check 2023-05-31T19:45:36.420 INFO:teuthology.orchestra.run.smithi112.stdout:Transaction check succeeded. 2023-05-31T19:45:36.422 INFO:teuthology.orchestra.run.smithi112.stdout:Running transaction test 2023-05-31T19:45:36.470 INFO:teuthology.orchestra.run.smithi112.stdout:Transaction test succeeded. 2023-05-31T19:45:36.473 INFO:teuthology.orchestra.run.smithi112.stdout:Running transaction 2023-05-31T19:45:36.607 INFO:teuthology.orchestra.run.smithi112.stdout: Preparing : 1/1 2023-05-31T19:45:36.829 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : rbd-nbd-2:18.0.0-3487.g4a02f3f4.el8.x86_64 1/1 2023-05-31T19:45:37.116 INFO:teuthology.orchestra.run.smithi112.stdout: Running scriptlet: rbd-nbd-2:18.0.0-3487.g4a02f3f4.el8.x86_64 1/1 2023-05-31T19:45:37.687 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : rbd-nbd-2:18.0.0-3487.g4a02f3f4.el8.x86_64 1/1 2023-05-31T19:45:37.688 INFO:teuthology.orchestra.run.smithi112.stdout: 2023-05-31T19:45:37.688 INFO:teuthology.orchestra.run.smithi112.stdout:Installed: 2023-05-31T19:45:37.688 INFO:teuthology.orchestra.run.smithi112.stdout: rbd-nbd-2:18.0.0-3487.g4a02f3f4.el8.x86_64 2023-05-31T19:45:37.688 INFO:teuthology.orchestra.run.smithi112.stdout: 2023-05-31T19:45:37.689 INFO:teuthology.orchestra.run.smithi112.stdout:Complete! 2023-05-31T19:45:37.870 DEBUG:teuthology.parallel:result is None 2023-05-31T19:47:15.930 INFO:teuthology.orchestra.run.smithi099.stdout:[MIRROR] python3-asyncssh-2.7.0-2.el8.noarch.rpm: Curl error (7): Couldn't connect to server for http://mirror.es.its.nyu.edu/epel/8/Everything/x86_64/Packages/p/python3-asyncssh-2.7.0-2.el8.noarch.rpm [Failed to connect to mirror.es.its.nyu.edu port 80: Connection timed out] 2023-05-31T19:47:15.930 INFO:teuthology.orchestra.run.smithi099.stdout:[MIRROR] python3-natsort-7.1.1-2.el8.noarch.rpm: Curl error (7): Couldn't connect to server for http://mirror.es.its.nyu.edu/epel/8/Everything/x86_64/Packages/p/python3-natsort-7.1.1-2.el8.noarch.rpm [Failed to connect to mirror.es.its.nyu.edu port 80: Connection timed out] 2023-05-31T19:47:16.090 INFO:teuthology.orchestra.run.smithi099.stdout:(4/5): python3-natsort-7.1.1-2.el8.noarch.rpm 477 B/s | 60 kB 02:09 2023-05-31T19:47:16.207 INFO:teuthology.orchestra.run.smithi099.stdout:(5/5): python3-asyncssh-2.7.0-2.el8.noarch.rpm 3.7 kB/s | 476 kB 02:09 2023-05-31T19:47:16.208 INFO:teuthology.orchestra.run.smithi099.stdout:-------------------------------------------------------------------------------- 2023-05-31T19:47:16.208 INFO:teuthology.orchestra.run.smithi099.stdout:Total 47 kB/s | 6.0 MB 02:10 2023-05-31T19:47:16.258 INFO:teuthology.orchestra.run.smithi099.stdout:Running transaction check 2023-05-31T19:47:16.270 INFO:teuthology.orchestra.run.smithi099.stdout:Transaction check succeeded. 2023-05-31T19:47:16.270 INFO:teuthology.orchestra.run.smithi099.stdout:Running transaction test 2023-05-31T19:47:16.382 INFO:teuthology.orchestra.run.smithi099.stdout:Transaction test succeeded. 2023-05-31T19:47:16.386 INFO:teuthology.orchestra.run.smithi099.stdout:Running transaction 2023-05-31T19:47:16.640 INFO:teuthology.orchestra.run.smithi099.stdout: Preparing : 1/1 2023-05-31T19:47:17.012 INFO:teuthology.orchestra.run.smithi099.stdout: Installing : python3-natsort-7.1.1-2.el8.noarch 1/5 2023-05-31T19:47:18.000 INFO:teuthology.orchestra.run.smithi099.stdout: Installing : python3-asyncssh-2.7.0-2.el8.noarch 2/5 2023-05-31T19:47:18.377 INFO:teuthology.orchestra.run.smithi099.stdout: Installing : python3-babel-2.5.1-7.el8.noarch 3/5 2023-05-31T19:47:18.600 INFO:teuthology.orchestra.run.smithi099.stdout: Installing : python3-jinja2-2.10.1-3.el8.noarch 4/5 2023-05-31T19:47:18.801 INFO:teuthology.orchestra.run.smithi099.stdout: Installing : ceph-mgr-cephadm-2:18.0.0-3487.g4a02f3f4.el8.noarch 5/5 2023-05-31T19:47:19.084 INFO:teuthology.orchestra.run.smithi099.stdout: Running scriptlet: ceph-mgr-cephadm-2:18.0.0-3487.g4a02f3f4.el8.noarch 5/5 2023-05-31T19:47:19.084 INFO:teuthology.orchestra.run.smithi099.stdout: Verifying : ceph-mgr-cephadm-2:18.0.0-3487.g4a02f3f4.el8.noarch 1/5 2023-05-31T19:47:19.085 INFO:teuthology.orchestra.run.smithi099.stdout: Verifying : python3-babel-2.5.1-7.el8.noarch 2/5 2023-05-31T19:47:19.085 INFO:teuthology.orchestra.run.smithi099.stdout: Verifying : python3-jinja2-2.10.1-3.el8.noarch 3/5 2023-05-31T19:47:19.085 INFO:teuthology.orchestra.run.smithi099.stdout: Verifying : python3-asyncssh-2.7.0-2.el8.noarch 4/5 2023-05-31T19:47:19.664 INFO:teuthology.orchestra.run.smithi099.stdout: Verifying : python3-natsort-7.1.1-2.el8.noarch 5/5 2023-05-31T19:47:19.664 INFO:teuthology.orchestra.run.smithi099.stdout: 2023-05-31T19:47:19.664 INFO:teuthology.orchestra.run.smithi099.stdout:Installed: 2023-05-31T19:47:19.664 INFO:teuthology.orchestra.run.smithi099.stdout: ceph-mgr-cephadm-2:18.0.0-3487.g4a02f3f4.el8.noarch 2023-05-31T19:47:19.665 INFO:teuthology.orchestra.run.smithi099.stdout: python3-asyncssh-2.7.0-2.el8.noarch 2023-05-31T19:47:19.665 INFO:teuthology.orchestra.run.smithi099.stdout: python3-babel-2.5.1-7.el8.noarch 2023-05-31T19:47:19.665 INFO:teuthology.orchestra.run.smithi099.stdout: python3-jinja2-2.10.1-3.el8.noarch 2023-05-31T19:47:19.665 INFO:teuthology.orchestra.run.smithi099.stdout: python3-natsort-7.1.1-2.el8.noarch 2023-05-31T19:47:19.665 INFO:teuthology.orchestra.run.smithi099.stdout: 2023-05-31T19:47:19.665 INFO:teuthology.orchestra.run.smithi099.stdout:Complete! 2023-05-31T19:47:19.775 DEBUG:teuthology.orchestra.run.smithi099:> sudo yum -y install ceph-fuse 2023-05-31T19:47:20.495 INFO:teuthology.orchestra.run.smithi099.stdout:Last metadata expiration check: 0:04:20 ago on Wed 31 May 2023 07:43:00 PM UTC. 2023-05-31T19:47:21.612 INFO:teuthology.orchestra.run.smithi099.stdout:Dependencies resolved. 2023-05-31T19:47:21.613 INFO:teuthology.orchestra.run.smithi099.stdout:================================================================================ 2023-05-31T19:47:21.613 INFO:teuthology.orchestra.run.smithi099.stdout: Package Arch Version Repository Size 2023-05-31T19:47:21.613 INFO:teuthology.orchestra.run.smithi099.stdout:================================================================================ 2023-05-31T19:47:21.613 INFO:teuthology.orchestra.run.smithi099.stdout:Installing: 2023-05-31T19:47:21.614 INFO:teuthology.orchestra.run.smithi099.stdout: ceph-fuse x86_64 2:18.0.0-3487.g4a02f3f4.el8 ceph 845 k 2023-05-31T19:47:21.614 INFO:teuthology.orchestra.run.smithi099.stdout:Installing dependencies: 2023-05-31T19:47:21.614 INFO:teuthology.orchestra.run.smithi099.stdout: fuse x86_64 2.9.7-17.el8 baseos 83 k 2023-05-31T19:47:21.614 INFO:teuthology.orchestra.run.smithi099.stdout: 2023-05-31T19:47:21.614 INFO:teuthology.orchestra.run.smithi099.stdout:Transaction Summary 2023-05-31T19:47:21.614 INFO:teuthology.orchestra.run.smithi099.stdout:================================================================================ 2023-05-31T19:47:21.614 INFO:teuthology.orchestra.run.smithi099.stdout:Install 2 Packages 2023-05-31T19:47:21.614 INFO:teuthology.orchestra.run.smithi099.stdout: 2023-05-31T19:47:21.615 INFO:teuthology.orchestra.run.smithi099.stdout:Total download size: 928 k 2023-05-31T19:47:21.615 INFO:teuthology.orchestra.run.smithi099.stdout:Installed size: 3.0 M 2023-05-31T19:47:21.615 INFO:teuthology.orchestra.run.smithi099.stdout:Downloading Packages: 2023-05-31T19:47:21.847 INFO:teuthology.orchestra.run.smithi099.stdout:(1/2): fuse-2.9.7-17.el8.x86_64.rpm 449 kB/s | 83 kB 00:00 2023-05-31T19:47:22.489 INFO:teuthology.orchestra.run.smithi099.stdout:(2/2): ceph-fuse-18.0.0-3487.g4a02f3f4.el8.x86_ 1.0 MB/s | 845 kB 00:00 2023-05-31T19:47:22.489 INFO:teuthology.orchestra.run.smithi099.stdout:-------------------------------------------------------------------------------- 2023-05-31T19:47:22.490 INFO:teuthology.orchestra.run.smithi099.stdout:Total 1.0 MB/s | 928 kB 00:00 2023-05-31T19:47:22.501 INFO:teuthology.orchestra.run.smithi099.stdout:Running transaction check 2023-05-31T19:47:22.522 INFO:teuthology.orchestra.run.smithi099.stdout:Transaction check succeeded. 2023-05-31T19:47:22.523 INFO:teuthology.orchestra.run.smithi099.stdout:Running transaction test 2023-05-31T19:47:22.608 INFO:teuthology.orchestra.run.smithi099.stdout:Transaction test succeeded. 2023-05-31T19:47:22.610 INFO:teuthology.orchestra.run.smithi099.stdout:Running transaction 2023-05-31T19:47:22.782 INFO:teuthology.orchestra.run.smithi099.stdout: Preparing : 1/1 2023-05-31T19:47:23.057 INFO:teuthology.orchestra.run.smithi099.stdout: Installing : fuse-2.9.7-17.el8.x86_64 1/2 2023-05-31T19:47:23.308 INFO:teuthology.orchestra.run.smithi099.stdout: Installing : ceph-fuse-2:18.0.0-3487.g4a02f3f4.el8.x86_64 2/2 2023-05-31T19:47:23.701 INFO:teuthology.orchestra.run.smithi099.stdout: Running scriptlet: ceph-fuse-2:18.0.0-3487.g4a02f3f4.el8.x86_64 2/2 2023-05-31T19:47:23.701 INFO:teuthology.orchestra.run.smithi099.stdout: Verifying : ceph-fuse-2:18.0.0-3487.g4a02f3f4.el8.x86_64 1/2 2023-05-31T19:47:24.240 INFO:teuthology.orchestra.run.smithi099.stdout: Verifying : fuse-2.9.7-17.el8.x86_64 2/2 2023-05-31T19:47:24.240 INFO:teuthology.orchestra.run.smithi099.stdout: 2023-05-31T19:47:24.240 INFO:teuthology.orchestra.run.smithi099.stdout:Installed: 2023-05-31T19:47:24.240 INFO:teuthology.orchestra.run.smithi099.stdout: ceph-fuse-2:18.0.0-3487.g4a02f3f4.el8.x86_64 fuse-2.9.7-17.el8.x86_64 2023-05-31T19:47:24.240 INFO:teuthology.orchestra.run.smithi099.stdout: 2023-05-31T19:47:24.240 INFO:teuthology.orchestra.run.smithi099.stdout:Complete! 2023-05-31T19:47:24.445 DEBUG:teuthology.orchestra.run.smithi099:> sudo yum -y install ceph-volume 2023-05-31T19:47:25.180 INFO:teuthology.orchestra.run.smithi099.stdout:Last metadata expiration check: 0:04:25 ago on Wed 31 May 2023 07:43:00 PM UTC. 2023-05-31T19:47:26.296 INFO:teuthology.orchestra.run.smithi099.stdout:Dependencies resolved. 2023-05-31T19:47:26.296 INFO:teuthology.orchestra.run.smithi099.stdout:================================================================================ 2023-05-31T19:47:26.296 INFO:teuthology.orchestra.run.smithi099.stdout: Package Arch Version Repository Size 2023-05-31T19:47:26.297 INFO:teuthology.orchestra.run.smithi099.stdout:================================================================================ 2023-05-31T19:47:26.297 INFO:teuthology.orchestra.run.smithi099.stdout:Installing: 2023-05-31T19:47:26.297 INFO:teuthology.orchestra.run.smithi099.stdout: ceph-volume noarch 2:18.0.0-3487.g4a02f3f4.el8 ceph-noarch 264 k 2023-05-31T19:47:26.298 INFO:teuthology.orchestra.run.smithi099.stdout: 2023-05-31T19:47:26.298 INFO:teuthology.orchestra.run.smithi099.stdout:Transaction Summary 2023-05-31T19:47:26.298 INFO:teuthology.orchestra.run.smithi099.stdout:================================================================================ 2023-05-31T19:47:26.298 INFO:teuthology.orchestra.run.smithi099.stdout:Install 1 Package 2023-05-31T19:47:26.299 INFO:teuthology.orchestra.run.smithi099.stdout: 2023-05-31T19:47:26.299 INFO:teuthology.orchestra.run.smithi099.stdout:Total download size: 264 k 2023-05-31T19:47:26.299 INFO:teuthology.orchestra.run.smithi099.stdout:Installed size: 1.3 M 2023-05-31T19:47:26.299 INFO:teuthology.orchestra.run.smithi099.stdout:Downloading Packages: 2023-05-31T19:47:26.951 INFO:teuthology.orchestra.run.smithi099.stdout:ceph-volume-18.0.0-3487.g4a02f3f4.el8.noarch.rp 403 kB/s | 264 kB 00:00 2023-05-31T19:47:26.951 INFO:teuthology.orchestra.run.smithi099.stdout:-------------------------------------------------------------------------------- 2023-05-31T19:47:26.952 INFO:teuthology.orchestra.run.smithi099.stdout:Total 402 kB/s | 264 kB 00:00 2023-05-31T19:47:26.952 INFO:teuthology.orchestra.run.smithi099.stdout:Running transaction check 2023-05-31T19:47:26.962 INFO:teuthology.orchestra.run.smithi099.stdout:Transaction check succeeded. 2023-05-31T19:47:26.963 INFO:teuthology.orchestra.run.smithi099.stdout:Running transaction test 2023-05-31T19:47:26.989 INFO:teuthology.orchestra.run.smithi099.stdout:Transaction test succeeded. 2023-05-31T19:47:26.994 INFO:teuthology.orchestra.run.smithi099.stdout:Running transaction 2023-05-31T19:47:27.186 INFO:teuthology.orchestra.run.smithi099.stdout: Preparing : 1/1 2023-05-31T19:47:27.441 INFO:teuthology.orchestra.run.smithi099.stdout: Installing : ceph-volume-2:18.0.0-3487.g4a02f3f4.el8.noarch 1/1 2023-05-31T19:47:27.874 INFO:teuthology.orchestra.run.smithi099.stdout: Running scriptlet: ceph-volume-2:18.0.0-3487.g4a02f3f4.el8.noarch 1/1 2023-05-31T19:47:28.429 INFO:teuthology.orchestra.run.smithi099.stdout: Verifying : ceph-volume-2:18.0.0-3487.g4a02f3f4.el8.noarch 1/1 2023-05-31T19:47:28.429 INFO:teuthology.orchestra.run.smithi099.stdout: 2023-05-31T19:47:28.429 INFO:teuthology.orchestra.run.smithi099.stdout:Installed: 2023-05-31T19:47:28.430 INFO:teuthology.orchestra.run.smithi099.stdout: ceph-volume-2:18.0.0-3487.g4a02f3f4.el8.noarch 2023-05-31T19:47:28.430 INFO:teuthology.orchestra.run.smithi099.stdout: 2023-05-31T19:47:28.430 INFO:teuthology.orchestra.run.smithi099.stdout:Complete! 2023-05-31T19:47:28.605 DEBUG:teuthology.orchestra.run.smithi099:> sudo yum -y install librados-devel 2023-05-31T19:47:29.342 INFO:teuthology.orchestra.run.smithi099.stdout:Last metadata expiration check: 0:04:29 ago on Wed 31 May 2023 07:43:00 PM UTC. 2023-05-31T19:47:30.458 INFO:teuthology.orchestra.run.smithi099.stdout:Dependencies resolved. 2023-05-31T19:47:30.458 INFO:teuthology.orchestra.run.smithi099.stdout:================================================================================ 2023-05-31T19:47:30.459 INFO:teuthology.orchestra.run.smithi099.stdout: Package Arch Version Repo Size 2023-05-31T19:47:30.459 INFO:teuthology.orchestra.run.smithi099.stdout:================================================================================ 2023-05-31T19:47:30.459 INFO:teuthology.orchestra.run.smithi099.stdout:Installing: 2023-05-31T19:47:30.459 INFO:teuthology.orchestra.run.smithi099.stdout: librados-devel x86_64 2:18.0.0-3487.g4a02f3f4.el8 ceph 139 k 2023-05-31T19:47:30.459 INFO:teuthology.orchestra.run.smithi099.stdout: 2023-05-31T19:47:30.459 INFO:teuthology.orchestra.run.smithi099.stdout:Transaction Summary 2023-05-31T19:47:30.459 INFO:teuthology.orchestra.run.smithi099.stdout:================================================================================ 2023-05-31T19:47:30.460 INFO:teuthology.orchestra.run.smithi099.stdout:Install 1 Package 2023-05-31T19:47:30.460 INFO:teuthology.orchestra.run.smithi099.stdout: 2023-05-31T19:47:30.460 INFO:teuthology.orchestra.run.smithi099.stdout:Total download size: 139 k 2023-05-31T19:47:30.461 INFO:teuthology.orchestra.run.smithi099.stdout:Installed size: 512 k 2023-05-31T19:47:30.461 INFO:teuthology.orchestra.run.smithi099.stdout:Downloading Packages: 2023-05-31T19:47:30.637 INFO:teuthology.orchestra.run.smithi099.stdout:librados-devel-18.0.0-3487.g4a02f3f4.el8.x86_64 783 kB/s | 139 kB 00:00 2023-05-31T19:47:30.638 INFO:teuthology.orchestra.run.smithi099.stdout:-------------------------------------------------------------------------------- 2023-05-31T19:47:30.639 INFO:teuthology.orchestra.run.smithi099.stdout:Total 778 kB/s | 139 kB 00:00 2023-05-31T19:47:30.639 INFO:teuthology.orchestra.run.smithi099.stdout:Running transaction check 2023-05-31T19:47:30.644 INFO:teuthology.orchestra.run.smithi099.stdout:Transaction check succeeded. 2023-05-31T19:47:30.644 INFO:teuthology.orchestra.run.smithi099.stdout:Running transaction test 2023-05-31T19:47:30.693 INFO:teuthology.orchestra.run.smithi099.stdout:Transaction test succeeded. 2023-05-31T19:47:30.695 INFO:teuthology.orchestra.run.smithi099.stdout:Running transaction 2023-05-31T19:47:30.800 INFO:teuthology.orchestra.run.smithi099.stdout: Preparing : 1/1 2023-05-31T19:47:31.098 INFO:teuthology.orchestra.run.smithi099.stdout: Installing : librados-devel-2:18.0.0-3487.g4a02f3f4.el8.x86_64 1/1 2023-05-31T19:47:31.391 INFO:teuthology.orchestra.run.smithi099.stdout: Running scriptlet: librados-devel-2:18.0.0-3487.g4a02f3f4.el8.x86_64 1/1 2023-05-31T19:47:31.906 INFO:teuthology.orchestra.run.smithi099.stdout: Verifying : librados-devel-2:18.0.0-3487.g4a02f3f4.el8.x86_64 1/1 2023-05-31T19:47:31.906 INFO:teuthology.orchestra.run.smithi099.stdout: 2023-05-31T19:47:31.906 INFO:teuthology.orchestra.run.smithi099.stdout:Installed: 2023-05-31T19:47:31.906 INFO:teuthology.orchestra.run.smithi099.stdout: librados-devel-2:18.0.0-3487.g4a02f3f4.el8.x86_64 2023-05-31T19:47:31.907 INFO:teuthology.orchestra.run.smithi099.stdout: 2023-05-31T19:47:31.907 INFO:teuthology.orchestra.run.smithi099.stdout:Complete! 2023-05-31T19:47:32.107 DEBUG:teuthology.orchestra.run.smithi099:> sudo yum -y install libcephfs2 2023-05-31T19:47:32.846 INFO:teuthology.orchestra.run.smithi099.stdout:Last metadata expiration check: 0:04:32 ago on Wed 31 May 2023 07:43:00 PM UTC. 2023-05-31T19:47:33.825 INFO:teuthology.orchestra.run.smithi099.stdout:Package libcephfs2-2:18.0.0-3487.g4a02f3f4.el8.x86_64 is already installed. 2023-05-31T19:47:33.965 INFO:teuthology.orchestra.run.smithi099.stdout:Dependencies resolved. 2023-05-31T19:47:33.966 INFO:teuthology.orchestra.run.smithi099.stdout:Nothing to do. 2023-05-31T19:47:33.966 INFO:teuthology.orchestra.run.smithi099.stdout:Complete! 2023-05-31T19:47:34.057 DEBUG:teuthology.orchestra.run.smithi099:> sudo yum -y install libcephfs-devel 2023-05-31T19:47:34.784 INFO:teuthology.orchestra.run.smithi099.stdout:Last metadata expiration check: 0:04:34 ago on Wed 31 May 2023 07:43:00 PM UTC. 2023-05-31T19:47:35.895 INFO:teuthology.orchestra.run.smithi099.stdout:Dependencies resolved. 2023-05-31T19:47:35.897 INFO:teuthology.orchestra.run.smithi099.stdout:================================================================================ 2023-05-31T19:47:35.898 INFO:teuthology.orchestra.run.smithi099.stdout: Package Arch Version Repo Size 2023-05-31T19:47:35.898 INFO:teuthology.orchestra.run.smithi099.stdout:================================================================================ 2023-05-31T19:47:35.899 INFO:teuthology.orchestra.run.smithi099.stdout:Installing: 2023-05-31T19:47:35.899 INFO:teuthology.orchestra.run.smithi099.stdout: libcephfs-devel x86_64 2:18.0.0-3487.g4a02f3f4.el8 ceph 30 k 2023-05-31T19:47:35.899 INFO:teuthology.orchestra.run.smithi099.stdout: 2023-05-31T19:47:35.899 INFO:teuthology.orchestra.run.smithi099.stdout:Transaction Summary 2023-05-31T19:47:35.899 INFO:teuthology.orchestra.run.smithi099.stdout:================================================================================ 2023-05-31T19:47:35.899 INFO:teuthology.orchestra.run.smithi099.stdout:Install 1 Package 2023-05-31T19:47:35.899 INFO:teuthology.orchestra.run.smithi099.stdout: 2023-05-31T19:47:35.900 INFO:teuthology.orchestra.run.smithi099.stdout:Total download size: 30 k 2023-05-31T19:47:35.900 INFO:teuthology.orchestra.run.smithi099.stdout:Installed size: 137 k 2023-05-31T19:47:35.900 INFO:teuthology.orchestra.run.smithi099.stdout:Downloading Packages: 2023-05-31T19:47:36.173 INFO:teuthology.orchestra.run.smithi099.stdout:libcephfs-devel-18.0.0-3487.g4a02f3f4.el8.x86_6 111 kB/s | 30 kB 00:00 2023-05-31T19:47:36.174 INFO:teuthology.orchestra.run.smithi099.stdout:-------------------------------------------------------------------------------- 2023-05-31T19:47:36.175 INFO:teuthology.orchestra.run.smithi099.stdout:Total 110 kB/s | 30 kB 00:00 2023-05-31T19:47:36.175 INFO:teuthology.orchestra.run.smithi099.stdout:Running transaction check 2023-05-31T19:47:36.178 INFO:teuthology.orchestra.run.smithi099.stdout:Transaction check succeeded. 2023-05-31T19:47:36.178 INFO:teuthology.orchestra.run.smithi099.stdout:Running transaction test 2023-05-31T19:47:36.202 INFO:teuthology.orchestra.run.smithi099.stdout:Transaction test succeeded. 2023-05-31T19:47:36.205 INFO:teuthology.orchestra.run.smithi099.stdout:Running transaction 2023-05-31T19:47:36.267 INFO:teuthology.orchestra.run.smithi099.stdout: Preparing : 1/1 2023-05-31T19:47:36.488 INFO:teuthology.orchestra.run.smithi099.stdout: Installing : libcephfs-devel-2:18.0.0-3487.g4a02f3f4.el8.x86_64 1/1 2023-05-31T19:47:36.755 INFO:teuthology.orchestra.run.smithi099.stdout: Running scriptlet: libcephfs-devel-2:18.0.0-3487.g4a02f3f4.el8.x86_64 1/1 2023-05-31T19:47:37.257 INFO:teuthology.orchestra.run.smithi099.stdout: Verifying : libcephfs-devel-2:18.0.0-3487.g4a02f3f4.el8.x86_64 1/1 2023-05-31T19:47:37.258 INFO:teuthology.orchestra.run.smithi099.stdout: 2023-05-31T19:47:37.258 INFO:teuthology.orchestra.run.smithi099.stdout:Installed: 2023-05-31T19:47:37.258 INFO:teuthology.orchestra.run.smithi099.stdout: libcephfs-devel-2:18.0.0-3487.g4a02f3f4.el8.x86_64 2023-05-31T19:47:37.258 INFO:teuthology.orchestra.run.smithi099.stdout: 2023-05-31T19:47:37.258 INFO:teuthology.orchestra.run.smithi099.stdout:Complete! 2023-05-31T19:47:37.352 DEBUG:teuthology.orchestra.run.smithi099:> sudo yum -y install librados2 2023-05-31T19:47:38.079 INFO:teuthology.orchestra.run.smithi099.stdout:Last metadata expiration check: 0:04:38 ago on Wed 31 May 2023 07:43:00 PM UTC. 2023-05-31T19:47:39.054 INFO:teuthology.orchestra.run.smithi099.stdout:Package librados2-2:18.0.0-3487.g4a02f3f4.el8.x86_64 is already installed. 2023-05-31T19:47:39.194 INFO:teuthology.orchestra.run.smithi099.stdout:Dependencies resolved. 2023-05-31T19:47:39.195 INFO:teuthology.orchestra.run.smithi099.stdout:Nothing to do. 2023-05-31T19:47:39.195 INFO:teuthology.orchestra.run.smithi099.stdout:Complete! 2023-05-31T19:47:39.292 DEBUG:teuthology.orchestra.run.smithi099:> sudo yum -y install librbd1 2023-05-31T19:47:40.020 INFO:teuthology.orchestra.run.smithi099.stdout:Last metadata expiration check: 0:04:40 ago on Wed 31 May 2023 07:43:00 PM UTC. 2023-05-31T19:47:40.989 INFO:teuthology.orchestra.run.smithi099.stdout:Package librbd1-2:18.0.0-3487.g4a02f3f4.el8.x86_64 is already installed. 2023-05-31T19:47:41.128 INFO:teuthology.orchestra.run.smithi099.stdout:Dependencies resolved. 2023-05-31T19:47:41.129 INFO:teuthology.orchestra.run.smithi099.stdout:Nothing to do. 2023-05-31T19:47:41.129 INFO:teuthology.orchestra.run.smithi099.stdout:Complete! 2023-05-31T19:47:41.222 DEBUG:teuthology.orchestra.run.smithi099:> sudo yum -y install python3-rados 2023-05-31T19:47:41.948 INFO:teuthology.orchestra.run.smithi099.stdout:Last metadata expiration check: 0:04:41 ago on Wed 31 May 2023 07:43:00 PM UTC. 2023-05-31T19:47:42.928 INFO:teuthology.orchestra.run.smithi099.stdout:Package python3-rados-2:18.0.0-3487.g4a02f3f4.el8.x86_64 is already installed. 2023-05-31T19:47:43.070 INFO:teuthology.orchestra.run.smithi099.stdout:Dependencies resolved. 2023-05-31T19:47:43.072 INFO:teuthology.orchestra.run.smithi099.stdout:Nothing to do. 2023-05-31T19:47:43.072 INFO:teuthology.orchestra.run.smithi099.stdout:Complete! 2023-05-31T19:47:43.167 DEBUG:teuthology.orchestra.run.smithi099:> sudo yum -y install python3-rgw 2023-05-31T19:47:43.894 INFO:teuthology.orchestra.run.smithi099.stdout:Last metadata expiration check: 0:04:43 ago on Wed 31 May 2023 07:43:00 PM UTC. 2023-05-31T19:47:44.863 INFO:teuthology.orchestra.run.smithi099.stdout:Package python3-rgw-2:18.0.0-3487.g4a02f3f4.el8.x86_64 is already installed. 2023-05-31T19:47:45.003 INFO:teuthology.orchestra.run.smithi099.stdout:Dependencies resolved. 2023-05-31T19:47:45.004 INFO:teuthology.orchestra.run.smithi099.stdout:Nothing to do. 2023-05-31T19:47:45.004 INFO:teuthology.orchestra.run.smithi099.stdout:Complete! 2023-05-31T19:47:45.099 DEBUG:teuthology.orchestra.run.smithi099:> sudo yum -y install python3-cephfs 2023-05-31T19:47:45.825 INFO:teuthology.orchestra.run.smithi099.stdout:Last metadata expiration check: 0:04:45 ago on Wed 31 May 2023 07:43:00 PM UTC. 2023-05-31T19:47:46.799 INFO:teuthology.orchestra.run.smithi099.stdout:Package python3-cephfs-2:18.0.0-3487.g4a02f3f4.el8.x86_64 is already installed. 2023-05-31T19:47:46.939 INFO:teuthology.orchestra.run.smithi099.stdout:Dependencies resolved. 2023-05-31T19:47:46.940 INFO:teuthology.orchestra.run.smithi099.stdout:Nothing to do. 2023-05-31T19:47:46.941 INFO:teuthology.orchestra.run.smithi099.stdout:Complete! 2023-05-31T19:47:47.103 DEBUG:teuthology.orchestra.run.smithi099:> sudo yum -y install python3-rbd 2023-05-31T19:47:47.826 INFO:teuthology.orchestra.run.smithi099.stdout:Last metadata expiration check: 0:04:47 ago on Wed 31 May 2023 07:43:00 PM UTC. 2023-05-31T19:47:48.796 INFO:teuthology.orchestra.run.smithi099.stdout:Package python3-rbd-2:18.0.0-3487.g4a02f3f4.el8.x86_64 is already installed. 2023-05-31T19:47:48.935 INFO:teuthology.orchestra.run.smithi099.stdout:Dependencies resolved. 2023-05-31T19:47:48.936 INFO:teuthology.orchestra.run.smithi099.stdout:Nothing to do. 2023-05-31T19:47:48.937 INFO:teuthology.orchestra.run.smithi099.stdout:Complete! 2023-05-31T19:47:49.022 DEBUG:teuthology.orchestra.run.smithi099:> sudo yum -y install rbd-fuse 2023-05-31T19:47:49.744 INFO:teuthology.orchestra.run.smithi099.stdout:Last metadata expiration check: 0:04:49 ago on Wed 31 May 2023 07:43:00 PM UTC. 2023-05-31T19:47:50.854 INFO:teuthology.orchestra.run.smithi099.stdout:Dependencies resolved. 2023-05-31T19:47:50.893 INFO:teuthology.orchestra.run.smithi099.stdout:================================================================================ 2023-05-31T19:47:50.894 INFO:teuthology.orchestra.run.smithi099.stdout: Package Architecture Version Repository Size 2023-05-31T19:47:50.894 INFO:teuthology.orchestra.run.smithi099.stdout:================================================================================ 2023-05-31T19:47:50.894 INFO:teuthology.orchestra.run.smithi099.stdout:Installing: 2023-05-31T19:47:50.894 INFO:teuthology.orchestra.run.smithi099.stdout: rbd-fuse x86_64 2:18.0.0-3487.g4a02f3f4.el8 ceph 94 k 2023-05-31T19:47:50.894 INFO:teuthology.orchestra.run.smithi099.stdout: 2023-05-31T19:47:50.894 INFO:teuthology.orchestra.run.smithi099.stdout:Transaction Summary 2023-05-31T19:47:50.894 INFO:teuthology.orchestra.run.smithi099.stdout:================================================================================ 2023-05-31T19:47:50.895 INFO:teuthology.orchestra.run.smithi099.stdout:Install 1 Package 2023-05-31T19:47:50.895 INFO:teuthology.orchestra.run.smithi099.stdout: 2023-05-31T19:47:50.895 INFO:teuthology.orchestra.run.smithi099.stdout:Total download size: 94 k 2023-05-31T19:47:50.895 INFO:teuthology.orchestra.run.smithi099.stdout:Installed size: 280 k 2023-05-31T19:47:50.895 INFO:teuthology.orchestra.run.smithi099.stdout:Downloading Packages: 2023-05-31T19:47:51.140 INFO:teuthology.orchestra.run.smithi099.stdout:rbd-fuse-18.0.0-3487.g4a02f3f4.el8.x86_64.rpm 332 kB/s | 94 kB 00:00 2023-05-31T19:47:51.140 INFO:teuthology.orchestra.run.smithi099.stdout:-------------------------------------------------------------------------------- 2023-05-31T19:47:51.141 INFO:teuthology.orchestra.run.smithi099.stdout:Total 331 kB/s | 94 kB 00:00 2023-05-31T19:47:51.141 INFO:teuthology.orchestra.run.smithi099.stdout:Running transaction check 2023-05-31T19:47:51.151 INFO:teuthology.orchestra.run.smithi099.stdout:Transaction check succeeded. 2023-05-31T19:47:51.152 INFO:teuthology.orchestra.run.smithi099.stdout:Running transaction test 2023-05-31T19:47:51.201 INFO:teuthology.orchestra.run.smithi099.stdout:Transaction test succeeded. 2023-05-31T19:47:51.204 INFO:teuthology.orchestra.run.smithi099.stdout:Running transaction 2023-05-31T19:47:51.307 INFO:teuthology.orchestra.run.smithi099.stdout: Preparing : 1/1 2023-05-31T19:47:51.501 INFO:teuthology.orchestra.run.smithi099.stdout: Installing : rbd-fuse-2:18.0.0-3487.g4a02f3f4.el8.x86_64 1/1 2023-05-31T19:47:51.802 INFO:teuthology.orchestra.run.smithi099.stdout: Running scriptlet: rbd-fuse-2:18.0.0-3487.g4a02f3f4.el8.x86_64 1/1 2023-05-31T19:47:52.353 INFO:teuthology.orchestra.run.smithi099.stdout: Verifying : rbd-fuse-2:18.0.0-3487.g4a02f3f4.el8.x86_64 1/1 2023-05-31T19:47:52.353 INFO:teuthology.orchestra.run.smithi099.stdout: 2023-05-31T19:47:52.353 INFO:teuthology.orchestra.run.smithi099.stdout:Installed: 2023-05-31T19:47:52.354 INFO:teuthology.orchestra.run.smithi099.stdout: rbd-fuse-2:18.0.0-3487.g4a02f3f4.el8.x86_64 2023-05-31T19:47:52.354 INFO:teuthology.orchestra.run.smithi099.stdout: 2023-05-31T19:47:52.354 INFO:teuthology.orchestra.run.smithi099.stdout:Complete! 2023-05-31T19:47:52.828 DEBUG:teuthology.orchestra.run.smithi099:> sudo yum -y install rbd-mirror 2023-05-31T19:47:53.573 INFO:teuthology.orchestra.run.smithi099.stdout:Last metadata expiration check: 0:04:53 ago on Wed 31 May 2023 07:43:00 PM UTC. 2023-05-31T19:47:54.695 INFO:teuthology.orchestra.run.smithi099.stdout:Dependencies resolved. 2023-05-31T19:47:54.696 INFO:teuthology.orchestra.run.smithi099.stdout:================================================================================ 2023-05-31T19:47:54.696 INFO:teuthology.orchestra.run.smithi099.stdout: Package Arch Version Repo Size 2023-05-31T19:47:54.696 INFO:teuthology.orchestra.run.smithi099.stdout:================================================================================ 2023-05-31T19:47:54.696 INFO:teuthology.orchestra.run.smithi099.stdout:Installing: 2023-05-31T19:47:54.696 INFO:teuthology.orchestra.run.smithi099.stdout: rbd-mirror x86_64 2:18.0.0-3487.g4a02f3f4.el8 ceph 3.4 M 2023-05-31T19:47:54.696 INFO:teuthology.orchestra.run.smithi099.stdout: 2023-05-31T19:47:54.697 INFO:teuthology.orchestra.run.smithi099.stdout:Transaction Summary 2023-05-31T19:47:54.697 INFO:teuthology.orchestra.run.smithi099.stdout:================================================================================ 2023-05-31T19:47:54.697 INFO:teuthology.orchestra.run.smithi099.stdout:Install 1 Package 2023-05-31T19:47:54.697 INFO:teuthology.orchestra.run.smithi099.stdout: 2023-05-31T19:47:54.697 INFO:teuthology.orchestra.run.smithi099.stdout:Total download size: 3.4 M 2023-05-31T19:47:54.698 INFO:teuthology.orchestra.run.smithi099.stdout:Installed size: 16 M 2023-05-31T19:47:54.698 INFO:teuthology.orchestra.run.smithi099.stdout:Downloading Packages: 2023-05-31T19:47:55.062 INFO:teuthology.orchestra.run.smithi099.stdout:rbd-mirror-18.0.0-3487.g4a02f3f4.el8.x86_64.rpm 9.4 MB/s | 3.4 MB 00:00 2023-05-31T19:47:55.063 INFO:teuthology.orchestra.run.smithi099.stdout:-------------------------------------------------------------------------------- 2023-05-31T19:47:55.063 INFO:teuthology.orchestra.run.smithi099.stdout:Total 9.4 MB/s | 3.4 MB 00:00 2023-05-31T19:47:55.064 INFO:teuthology.orchestra.run.smithi099.stdout:Running transaction check 2023-05-31T19:47:55.076 INFO:teuthology.orchestra.run.smithi099.stdout:Transaction check succeeded. 2023-05-31T19:47:55.076 INFO:teuthology.orchestra.run.smithi099.stdout:Running transaction test 2023-05-31T19:47:55.137 INFO:teuthology.orchestra.run.smithi099.stdout:Transaction test succeeded. 2023-05-31T19:47:55.140 INFO:teuthology.orchestra.run.smithi099.stdout:Running transaction 2023-05-31T19:47:55.577 INFO:teuthology.orchestra.run.smithi099.stdout: Preparing : 1/1 2023-05-31T19:47:55.793 INFO:teuthology.orchestra.run.smithi099.stdout: Installing : rbd-mirror-2:18.0.0-3487.g4a02f3f4.el8.x86_64 1/1 2023-05-31T19:47:56.192 INFO:teuthology.orchestra.run.smithi099.stdout: Running scriptlet: rbd-mirror-2:18.0.0-3487.g4a02f3f4.el8.x86_64 1/1 2023-05-31T19:47:56.748 INFO:teuthology.orchestra.run.smithi099.stdout: Verifying : rbd-mirror-2:18.0.0-3487.g4a02f3f4.el8.x86_64 1/1 2023-05-31T19:47:56.748 INFO:teuthology.orchestra.run.smithi099.stdout: 2023-05-31T19:47:56.748 INFO:teuthology.orchestra.run.smithi099.stdout:Installed: 2023-05-31T19:47:56.749 INFO:teuthology.orchestra.run.smithi099.stdout: rbd-mirror-2:18.0.0-3487.g4a02f3f4.el8.x86_64 2023-05-31T19:47:56.749 INFO:teuthology.orchestra.run.smithi099.stdout: 2023-05-31T19:47:56.749 INFO:teuthology.orchestra.run.smithi099.stdout:Complete! 2023-05-31T19:47:56.934 DEBUG:teuthology.orchestra.run.smithi099:> sudo yum -y install rbd-nbd 2023-05-31T19:47:57.671 INFO:teuthology.orchestra.run.smithi099.stdout:Last metadata expiration check: 0:04:57 ago on Wed 31 May 2023 07:43:00 PM UTC. 2023-05-31T19:47:58.790 INFO:teuthology.orchestra.run.smithi099.stdout:Dependencies resolved. 2023-05-31T19:47:58.791 INFO:teuthology.orchestra.run.smithi099.stdout:================================================================================ 2023-05-31T19:47:58.791 INFO:teuthology.orchestra.run.smithi099.stdout: Package Architecture Version Repository Size 2023-05-31T19:47:58.791 INFO:teuthology.orchestra.run.smithi099.stdout:================================================================================ 2023-05-31T19:47:58.791 INFO:teuthology.orchestra.run.smithi099.stdout:Installing: 2023-05-31T19:47:58.792 INFO:teuthology.orchestra.run.smithi099.stdout: rbd-nbd x86_64 2:18.0.0-3487.g4a02f3f4.el8 ceph 179 k 2023-05-31T19:47:58.792 INFO:teuthology.orchestra.run.smithi099.stdout: 2023-05-31T19:47:58.792 INFO:teuthology.orchestra.run.smithi099.stdout:Transaction Summary 2023-05-31T19:47:58.792 INFO:teuthology.orchestra.run.smithi099.stdout:================================================================================ 2023-05-31T19:47:58.792 INFO:teuthology.orchestra.run.smithi099.stdout:Install 1 Package 2023-05-31T19:47:58.792 INFO:teuthology.orchestra.run.smithi099.stdout: 2023-05-31T19:47:58.793 INFO:teuthology.orchestra.run.smithi099.stdout:Total download size: 179 k 2023-05-31T19:47:58.793 INFO:teuthology.orchestra.run.smithi099.stdout:Installed size: 568 k 2023-05-31T19:47:58.793 INFO:teuthology.orchestra.run.smithi099.stdout:Downloading Packages: 2023-05-31T19:47:59.107 INFO:teuthology.orchestra.run.smithi099.stdout:rbd-nbd-18.0.0-3487.g4a02f3f4.el8.x86_64.rpm 571 kB/s | 179 kB 00:00 2023-05-31T19:47:59.107 INFO:teuthology.orchestra.run.smithi099.stdout:-------------------------------------------------------------------------------- 2023-05-31T19:47:59.107 INFO:teuthology.orchestra.run.smithi099.stdout:Total 569 kB/s | 179 kB 00:00 2023-05-31T19:47:59.107 INFO:teuthology.orchestra.run.smithi099.stdout:Running transaction check 2023-05-31T19:47:59.118 INFO:teuthology.orchestra.run.smithi099.stdout:Transaction check succeeded. 2023-05-31T19:47:59.118 INFO:teuthology.orchestra.run.smithi099.stdout:Running transaction test 2023-05-31T19:47:59.166 INFO:teuthology.orchestra.run.smithi099.stdout:Transaction test succeeded. 2023-05-31T19:47:59.171 INFO:teuthology.orchestra.run.smithi099.stdout:Running transaction 2023-05-31T19:47:59.289 INFO:teuthology.orchestra.run.smithi099.stdout: Preparing : 1/1 2023-05-31T19:47:59.476 INFO:teuthology.orchestra.run.smithi099.stdout: Installing : rbd-nbd-2:18.0.0-3487.g4a02f3f4.el8.x86_64 1/1 2023-05-31T19:47:59.768 INFO:teuthology.orchestra.run.smithi099.stdout: Running scriptlet: rbd-nbd-2:18.0.0-3487.g4a02f3f4.el8.x86_64 1/1 2023-05-31T19:48:00.298 INFO:teuthology.orchestra.run.smithi099.stdout: Verifying : rbd-nbd-2:18.0.0-3487.g4a02f3f4.el8.x86_64 1/1 2023-05-31T19:48:00.298 INFO:teuthology.orchestra.run.smithi099.stdout: 2023-05-31T19:48:00.298 INFO:teuthology.orchestra.run.smithi099.stdout:Installed: 2023-05-31T19:48:00.299 INFO:teuthology.orchestra.run.smithi099.stdout: rbd-nbd-2:18.0.0-3487.g4a02f3f4.el8.x86_64 2023-05-31T19:48:00.299 INFO:teuthology.orchestra.run.smithi099.stdout: 2023-05-31T19:48:00.299 INFO:teuthology.orchestra.run.smithi099.stdout:Complete! 2023-05-31T19:48:00.481 DEBUG:teuthology.parallel:result is None 2023-05-31T19:48:00.482 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F8%2Fx86_64&sha1=4a02f3f496d9039326c49bf1fbe140388cd2f619 2023-05-31T19:48:00.704 DEBUG:teuthology.orchestra.run.smithi099:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2023-05-31T19:48:00.728 INFO:teuthology.orchestra.run.smithi099.stdout:18.0.0-3487.g4a02f3f4.el8 2023-05-31T19:48:00.729 INFO:teuthology.packaging:The installed version of ceph is 18.0.0-3487.g4a02f3f4.el8 2023-05-31T19:48:00.730 INFO:teuthology.task.install:The correct ceph version 18.0.0-3487.g4a02f3f4 is installed. 2023-05-31T19:48:00.730 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F8%2Fx86_64&sha1=4a02f3f496d9039326c49bf1fbe140388cd2f619 2023-05-31T19:48:00.870 DEBUG:teuthology.orchestra.run.smithi112:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2023-05-31T19:48:00.899 INFO:teuthology.orchestra.run.smithi112.stdout:18.0.0-3487.g4a02f3f4.el8 2023-05-31T19:48:00.899 INFO:teuthology.packaging:The installed version of ceph is 18.0.0-3487.g4a02f3f4.el8 2023-05-31T19:48:00.900 INFO:teuthology.task.install:The correct ceph version 18.0.0-3487.g4a02f3f4 is installed. 2023-05-31T19:48:00.900 INFO:teuthology.task.install.util:Shipping valgrind.supp... 2023-05-31T19:48:00.901 DEBUG:teuthology.orchestra.run.smithi099:> set -ex 2023-05-31T19:48:00.901 DEBUG:teuthology.orchestra.run.smithi099:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2023-05-31T19:48:00.931 DEBUG:teuthology.orchestra.run.smithi112:> set -ex 2023-05-31T19:48:00.931 DEBUG:teuthology.orchestra.run.smithi112:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2023-05-31T19:48:00.986 INFO:teuthology.task.install.util:Shipping 'daemon-helper'... 2023-05-31T19:48:00.987 DEBUG:teuthology.orchestra.run.smithi099:> set -ex 2023-05-31T19:48:00.987 DEBUG:teuthology.orchestra.run.smithi099:> sudo dd of=/usr/bin/daemon-helper 2023-05-31T19:48:01.017 DEBUG:teuthology.orchestra.run.smithi099:> sudo chmod a=rx -- /usr/bin/daemon-helper 2023-05-31T19:48:01.086 DEBUG:teuthology.orchestra.run.smithi112:> set -ex 2023-05-31T19:48:01.087 DEBUG:teuthology.orchestra.run.smithi112:> sudo dd of=/usr/bin/daemon-helper 2023-05-31T19:48:01.121 DEBUG:teuthology.orchestra.run.smithi112:> sudo chmod a=rx -- /usr/bin/daemon-helper 2023-05-31T19:48:01.189 INFO:teuthology.task.install.util:Shipping 'adjust-ulimits'... 2023-05-31T19:48:01.190 DEBUG:teuthology.orchestra.run.smithi099:> set -ex 2023-05-31T19:48:01.190 DEBUG:teuthology.orchestra.run.smithi099:> sudo dd of=/usr/bin/adjust-ulimits 2023-05-31T19:48:01.216 DEBUG:teuthology.orchestra.run.smithi099:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2023-05-31T19:48:01.284 DEBUG:teuthology.orchestra.run.smithi112:> set -ex 2023-05-31T19:48:01.284 DEBUG:teuthology.orchestra.run.smithi112:> sudo dd of=/usr/bin/adjust-ulimits 2023-05-31T19:48:01.314 DEBUG:teuthology.orchestra.run.smithi112:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2023-05-31T19:48:01.385 INFO:teuthology.run_tasks:Running task ceph... 2023-05-31T19:48:01.459 INFO:tasks.ceph:Making ceph log dir writeable by non-root... 2023-05-31T19:48:01.460 DEBUG:teuthology.orchestra.run.smithi099:> sudo chmod 777 /var/log/ceph 2023-05-31T19:48:01.464 DEBUG:teuthology.orchestra.run.smithi112:> sudo chmod 777 /var/log/ceph 2023-05-31T19:48:01.496 INFO:tasks.ceph:Disabling ceph logrotate... 2023-05-31T19:48:01.497 DEBUG:teuthology.orchestra.run.smithi099:> sudo rm -f -- /etc/logrotate.d/ceph 2023-05-31T19:48:01.542 DEBUG:teuthology.orchestra.run.smithi112:> sudo rm -f -- /etc/logrotate.d/ceph 2023-05-31T19:48:01.573 INFO:tasks.ceph:Creating extra log directories... 2023-05-31T19:48:01.574 DEBUG:teuthology.orchestra.run.smithi099:> sudo install -d -m0777 -- /var/log/ceph/valgrind /var/log/ceph/profiling-logger 2023-05-31T19:48:01.599 DEBUG:teuthology.orchestra.run.smithi112:> sudo install -d -m0777 -- /var/log/ceph/valgrind /var/log/ceph/profiling-logger 2023-05-31T19:48:01.652 INFO:tasks.ceph:Creating ceph cluster c1... 2023-05-31T19:48:01.653 INFO:tasks.ceph:config {'conf': {'client': {'debug rgw': 20, 'debug rgw sync': 20, 'rgw crypt require ssl': False, 'rgw crypt s3 kms backend': 'testing', 'rgw crypt s3 kms encryption keys': 'testkey-1=YmluCmJvb3N0CmJvb3N0LWJ1aWxkCmNlcGguY29uZgo=', 'rgw curl low speed time': 300, 'rgw data log num shards': 4, 'rgw md log max shards': 4, 'rgw sync data inject err probability': 0.1, 'rgw sync log trim interval': 0, 'rgw sync meta inject err probability': 0.1, 'rgw sync obj etag verify': True, 'setgroup': 'ceph', 'setuser': 'ceph'}, 'client.0': {'rgw data notify interval msec': 0}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20}, 'osd': {'debug ms': 1, 'debug osd': 20, 'rocksdb delete range threshold': 0}, 'osd.0': {'osd_max_omap_entries_per_request': 10}, 'osd.1': {'osd_max_omap_entries_per_request': 1000}, 'osd.2': {'osd_max_omap_entries_per_request': 10000}}, 'fs': 'xfs', 'mkfs_options': None, 'mount_options': None, 'skip_mgr_daemons': False, 'log_ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(PG_AVAILABILITY\\)', '\\(PG_DEGRADED\\)', '\\(POOL_APP_NOT_ENABLED\\)'], 'cpu_profile': set(), 'cluster': 'c1', 'mon_bind_msgr2': True, 'mon_bind_addrvec': True} 2023-05-31T19:48:01.653 INFO:tasks.ceph:ctx.config {'archive_path': '/home/teuthworker/archive/yuriw-2023-05-31_19:25:20-rgw-reef-release-distro-default-smithi/7292285', 'branch': 'reef-release', 'description': 'rgw/multisite/{clusters frontend/beast ignore-pg-availability notify omap_limits overrides realms/two-zonegroup supported-random-distro$/{centos_8} tasks/test_multi}', 'email': 'yweinste@redhat.com', 'first_in_suite': False, 'job_id': '7292285', 'kernel': {'kdb': True, 'sha1': 'distro'}, 'last_in_suite': False, 'machine_type': 'smithi', 'name': 'yuriw-2023-05-31_19:25:20-rgw-reef-release-distro-default-smithi', 'no_nested_subset': False, 'nuke-on-error': True, 'os_type': 'centos', 'os_version': '8.stream', 'overrides': {'admin_socket': {'branch': 'reef-release'}, 'ceph': {'conf': {'client': {'debug rgw': 20, 'debug rgw sync': 20, 'rgw crypt require ssl': False, 'rgw crypt s3 kms backend': 'testing', 'rgw crypt s3 kms encryption keys': 'testkey-1=YmluCmJvb3N0CmJvb3N0LWJ1aWxkCmNlcGguY29uZgo=', 'rgw curl low speed time': 300, 'rgw data log num shards': 4, 'rgw md log max shards': 4, 'rgw sync data inject err probability': 0.1, 'rgw sync log trim interval': 0, 'rgw sync meta inject err probability': 0.1, 'rgw sync obj etag verify': True, 'setgroup': 'ceph', 'setuser': 'ceph'}, 'client.0': {'rgw data notify interval msec': 0}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20}, 'osd': {'debug ms': 1, 'debug osd': 20, 'rocksdb delete range threshold': 0}, 'osd.0': {'osd_max_omap_entries_per_request': 10}, 'osd.1': {'osd_max_omap_entries_per_request': 1000}, 'osd.2': {'osd_max_omap_entries_per_request': 10000}}, 'flavor': 'default', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(PG_AVAILABILITY\\)', '\\(PG_DEGRADED\\)', '\\(POOL_APP_NOT_ENABLED\\)'], 'sha1': '4a02f3f496d9039326c49bf1fbe140388cd2f619', 'wait-for-scrub': False}, 'ceph-deploy': {'conf': {'client': {'log file': '/var/log/ceph/ceph-$name.$pid.log'}, 'mon': {'osd default pool size': 2}}}, 'install': {'ceph': {'flavor': 'default', 'sha1': '4a02f3f496d9039326c49bf1fbe140388cd2f619'}}, 'rgw': {'compression type': 'random', 'frontend': 'beast'}, 'rgw-multisite': {'realm': {'is default': True, 'name': 'test-realm'}, 'zonegroups': [{'enabled_features': ['resharding'], 'endpoints': ['c1.client.0'], 'is_default': True, 'is_master': True, 'name': 'a', 'zones': [{'endpoints': ['c1.client.0'], 'is_default': True, 'is_master': True, 'name': 'a1'}, {'endpoints': ['c1.client.1'], 'name': 'a2'}]}, {'enabled_features': ['resharding'], 'endpoints': ['c2.client.0'], 'is_default': True, 'name': 'b', 'zones': [{'endpoints': ['c2.client.0'], 'is_default': True, 'is_master': True, 'name': 'b1'}, {'endpoints': ['c2.client.1'], 'name': 'b2'}]}]}, 'rgw-multisite-tests': {'args': ['tests.py']}, 'selinux': {'whitelist': ['scontext=system_u:system_r:logrotate_t:s0']}, 'workunit': {'branch': 'reef-release', 'sha1': '2870bf456d3115490c153abce9d47ac5c043b65e'}}, 'owner': 'scheduled_yuriw@teuthology', 'priority': 71, 'repo': 'https://github.com/ceph/ceph-ci.git', 'roles': [['c1.mon.a', 'c1.mgr.x', 'c1.osd.0', 'c1.osd.1', 'c1.osd.2', 'c1.client.0', 'c1.client.1'], ['c2.mon.a', 'c2.mgr.x', 'c2.osd.0', 'c2.osd.1', 'c2.osd.2', 'c2.client.0', 'c2.client.1']], 'seed': 2752, 'sha1': '4a02f3f496d9039326c49bf1fbe140388cd2f619', 'sleep_before_teardown': 0, 'subset': '111/12000', 'suite': 'rgw', 'suite_branch': 'reef-release', 'suite_path': '/home/teuthworker/src/github.com_ceph_ceph-c_2870bf456d3115490c153abce9d47ac5c043b65e/qa', 'suite_relpath': 'qa', 'suite_repo': 'https://github.com/ceph/ceph-ci.git', 'suite_sha1': '2870bf456d3115490c153abce9d47ac5c043b65e', 'targets': {'smithi099.front.sepia.ceph.com': 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDox97V1ZbGiXYfIam0QsAKCOossAxUlL++9Y+qMtFgf7WbqpNcOYHYWWtH1ov+9qROYmTCRNQIFX+NkdHU8Q2Wbwg5LFP43nv89monL09OSkXGOUGTF6MF+MDeWU/gCRrcDxxs4QnoWTTbcd4HtqMzUprCFR5KLeO4XJSkTCxXlVJC1I2F9x9uGkv5VM4gB0ZVUeoVsATmS79+GLwNLH2DhXh4JQJL9HKgC9FN03CerL+Ka6cdcu3Wc6rSelCxMx0Ead4Vsre8il1J8PfCPvcko7HAkfp2XPu3avJXTKUTKYrF05s3uZq027l5T6VrlavgVtfGsyB8Ez4YDolVQOoGGfF2I99aVh9LAYc+J8KOY1Qh5KaILCCmrMI74XRPrqPrg7phDNIDNwjAxPcmoKpNDR6ALert5o+mM8onphQ/yRxisjwMH/bNXE63CdbjgkIOm2VJ2g/k1Ncf4lBxUPSx3eF+ERlKFdboN8S567n9hWRVwbSdGmzQf3AsOb38Hlk=', 'smithi112.front.sepia.ceph.com': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBFfCJhk9QMgDL70wZsKQnRvWaVVb4spsBOWpIa4TPw/+clTUBIAM2haG88c+/+tNQhpaYNEiHo4bsY1a+mmIK9A='}, 'tasks': [{'internal.check_packages': None}, {'internal.buildpackages_prep': None}, {'internal.save_config': None}, {'internal.check_lock': None}, {'internal.add_remotes': None}, {'console_log': None}, {'internal.connect': None}, {'internal.push_inventory': None}, {'internal.serialize_remote_roles': None}, {'internal.check_conflict': None}, {'internal.check_ceph_data': None}, {'internal.vm_setup': None}, {'kernel': {'kdb': True, 'sha1': 'distro'}}, {'internal.base': None}, {'internal.archive_upload': None}, {'internal.archive': None}, {'internal.coredump': None}, {'internal.sudo': None}, {'internal.syslog': None}, {'internal.timer': None}, {'pcp': None}, {'selinux': None}, {'ansible.cephlab': None}, {'clock': None}, {'install': None}, {'ceph': {'cluster': 'c1', 'conf': {'client': {'debug rgw': 20, 'debug rgw sync': 20, 'rgw crypt require ssl': False, 'rgw crypt s3 kms backend': 'testing', 'rgw crypt s3 kms encryption keys': 'testkey-1=YmluCmJvb3N0CmJvb3N0LWJ1aWxkCmNlcGguY29uZgo=', 'rgw curl low speed time': 300, 'rgw data log num shards': 4, 'rgw md log max shards': 4, 'rgw sync data inject err probability': 0.1, 'rgw sync log trim interval': 0, 'rgw sync meta inject err probability': 0.1, 'rgw sync obj etag verify': True, 'setgroup': 'ceph', 'setuser': 'ceph'}, 'client.0': {'rgw data notify interval msec': 0}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20}, 'osd': {'debug ms': 1, 'debug osd': 20, 'rocksdb delete range threshold': 0}, 'osd.0': {'osd_max_omap_entries_per_request': 10}, 'osd.1': {'osd_max_omap_entries_per_request': 1000}, 'osd.2': {'osd_max_omap_entries_per_request': 10000}}, 'flavor': 'default', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(PG_AVAILABILITY\\)', '\\(PG_DEGRADED\\)', '\\(POOL_APP_NOT_ENABLED\\)'], 'sha1': '4a02f3f496d9039326c49bf1fbe140388cd2f619', 'wait-for-scrub': False}}, {'ceph': {'cluster': 'c2'}}, {'rgw': {'c1.client.0': {'port': 8000}, 'c1.client.1': {'port': 8001}, 'c2.client.0': {'port': 8000}, 'c2.client.1': {'port': 8001}}}, {'rgw-multisite': None}, {'rgw-multisite-tests': {'config': {'reconfigure_delay': 60}}}], 'teuthology': {'fragments_dropped': [], 'meta': {}, 'postmerge': []}, 'teuthology_branch': 'main', 'teuthology_sha1': 'fa97ddd17d12e6d74b49196948f66e54fa009130', 'timestamp': '2023-05-31_19:25:20', 'tube': 'smithi', 'user': 'yuriw', 'verbose': True, 'worker_log': '/home/teuthworker/archive/worker_logs/dispatcher.smithi.23459'} 2023-05-31T19:48:01.654 DEBUG:teuthology.orchestra.run.smithi099:> install -d -m0755 -- /home/ubuntu/cephtest/c1.data 2023-05-31T19:48:01.667 DEBUG:teuthology.orchestra.run.smithi112:> install -d -m0755 -- /home/ubuntu/cephtest/c1.data 2023-05-31T19:48:01.702 DEBUG:teuthology.orchestra.run.smithi099:> sudo install -d -m0777 -- /var/run/ceph 2023-05-31T19:48:01.726 DEBUG:teuthology.orchestra.run.smithi112:> sudo install -d -m0777 -- /var/run/ceph 2023-05-31T19:48:01.770 DEBUG:teuthology.orchestra.run.smithi099:> set -ex 2023-05-31T19:48:01.771 DEBUG:teuthology.orchestra.run.smithi099:> dd if=/scratch_devs of=/dev/stdout 2023-05-31T19:48:01.814 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2023-05-31T19:48:01.814 DEBUG:teuthology.orchestra.run.smithi099:> stat /dev/vg_nvme/lv_1 2023-05-31T19:48:01.871 INFO:teuthology.orchestra.run.smithi099.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2023-05-31T19:48:01.872 INFO:teuthology.orchestra.run.smithi099.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-05-31T19:48:01.872 INFO:teuthology.orchestra.run.smithi099.stdout:Device: 6h/6d Inode: 371890 Links: 1 2023-05-31T19:48:01.873 INFO:teuthology.orchestra.run.smithi099.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-05-31T19:48:01.874 INFO:teuthology.orchestra.run.smithi099.stdout:Context: system_u:object_r:device_t:s0 2023-05-31T19:48:01.874 INFO:teuthology.orchestra.run.smithi099.stdout:Access: 2023-05-31 19:47:55.941990607 +0000 2023-05-31T19:48:01.874 INFO:teuthology.orchestra.run.smithi099.stdout:Modify: 2023-05-31 19:41:50.609986758 +0000 2023-05-31T19:48:01.874 INFO:teuthology.orchestra.run.smithi099.stdout:Change: 2023-05-31 19:41:50.609986758 +0000 2023-05-31T19:48:01.874 INFO:teuthology.orchestra.run.smithi099.stdout: Birth: - 2023-05-31T19:48:01.875 DEBUG:teuthology.orchestra.run.smithi099:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2023-05-31T19:48:01.938 INFO:teuthology.orchestra.run.smithi099.stderr:1+0 records in 2023-05-31T19:48:01.938 INFO:teuthology.orchestra.run.smithi099.stderr:1+0 records out 2023-05-31T19:48:01.938 INFO:teuthology.orchestra.run.smithi099.stderr:512 bytes copied, 0.00025474 s, 2.0 MB/s 2023-05-31T19:48:01.939 DEBUG:teuthology.orchestra.run.smithi099:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2023-05-31T19:48:02.000 DEBUG:teuthology.orchestra.run.smithi099:> stat /dev/vg_nvme/lv_2 2023-05-31T19:48:02.059 INFO:teuthology.orchestra.run.smithi099.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2023-05-31T19:48:02.060 INFO:teuthology.orchestra.run.smithi099.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-05-31T19:48:02.060 INFO:teuthology.orchestra.run.smithi099.stdout:Device: 6h/6d Inode: 366930 Links: 1 2023-05-31T19:48:02.060 INFO:teuthology.orchestra.run.smithi099.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-05-31T19:48:02.060 INFO:teuthology.orchestra.run.smithi099.stdout:Context: system_u:object_r:device_t:s0 2023-05-31T19:48:02.060 INFO:teuthology.orchestra.run.smithi099.stdout:Access: 2023-05-31 19:47:55.941990607 +0000 2023-05-31T19:48:02.060 INFO:teuthology.orchestra.run.smithi099.stdout:Modify: 2023-05-31 19:41:51.134974347 +0000 2023-05-31T19:48:02.061 INFO:teuthology.orchestra.run.smithi099.stdout:Change: 2023-05-31 19:41:51.134974347 +0000 2023-05-31T19:48:02.061 INFO:teuthology.orchestra.run.smithi099.stdout: Birth: - 2023-05-31T19:48:02.061 DEBUG:teuthology.orchestra.run.smithi099:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2023-05-31T19:48:02.125 INFO:teuthology.orchestra.run.smithi099.stderr:1+0 records in 2023-05-31T19:48:02.126 INFO:teuthology.orchestra.run.smithi099.stderr:1+0 records out 2023-05-31T19:48:02.126 INFO:teuthology.orchestra.run.smithi099.stderr:512 bytes copied, 0.000223343 s, 2.3 MB/s 2023-05-31T19:48:02.127 DEBUG:teuthology.orchestra.run.smithi099:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2023-05-31T19:48:02.191 DEBUG:teuthology.orchestra.run.smithi099:> stat /dev/vg_nvme/lv_3 2023-05-31T19:48:02.253 INFO:teuthology.orchestra.run.smithi099.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2023-05-31T19:48:02.253 INFO:teuthology.orchestra.run.smithi099.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-05-31T19:48:02.253 INFO:teuthology.orchestra.run.smithi099.stdout:Device: 6h/6d Inode: 367932 Links: 1 2023-05-31T19:48:02.253 INFO:teuthology.orchestra.run.smithi099.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-05-31T19:48:02.254 INFO:teuthology.orchestra.run.smithi099.stdout:Context: system_u:object_r:device_t:s0 2023-05-31T19:48:02.254 INFO:teuthology.orchestra.run.smithi099.stdout:Access: 2023-05-31 19:47:55.942990584 +0000 2023-05-31T19:48:02.254 INFO:teuthology.orchestra.run.smithi099.stdout:Modify: 2023-05-31 19:41:51.672961628 +0000 2023-05-31T19:48:02.254 INFO:teuthology.orchestra.run.smithi099.stdout:Change: 2023-05-31 19:41:51.672961628 +0000 2023-05-31T19:48:02.254 INFO:teuthology.orchestra.run.smithi099.stdout: Birth: - 2023-05-31T19:48:02.254 DEBUG:teuthology.orchestra.run.smithi099:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2023-05-31T19:48:02.320 INFO:teuthology.orchestra.run.smithi099.stderr:1+0 records in 2023-05-31T19:48:02.320 INFO:teuthology.orchestra.run.smithi099.stderr:1+0 records out 2023-05-31T19:48:02.321 INFO:teuthology.orchestra.run.smithi099.stderr:512 bytes copied, 0.000220718 s, 2.3 MB/s 2023-05-31T19:48:02.321 DEBUG:teuthology.orchestra.run.smithi099:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2023-05-31T19:48:02.381 DEBUG:teuthology.orchestra.run.smithi099:> stat /dev/vg_nvme/lv_4 2023-05-31T19:48:02.441 INFO:teuthology.orchestra.run.smithi099.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2023-05-31T19:48:02.441 INFO:teuthology.orchestra.run.smithi099.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-05-31T19:48:02.441 INFO:teuthology.orchestra.run.smithi099.stdout:Device: 6h/6d Inode: 370037 Links: 1 2023-05-31T19:48:02.441 INFO:teuthology.orchestra.run.smithi099.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-05-31T19:48:02.441 INFO:teuthology.orchestra.run.smithi099.stdout:Context: system_u:object_r:device_t:s0 2023-05-31T19:48:02.441 INFO:teuthology.orchestra.run.smithi099.stdout:Access: 2023-05-31 19:47:55.942990584 +0000 2023-05-31T19:48:02.443 INFO:teuthology.orchestra.run.smithi099.stdout:Modify: 2023-05-31 19:41:52.199949169 +0000 2023-05-31T19:48:02.444 INFO:teuthology.orchestra.run.smithi099.stdout:Change: 2023-05-31 19:41:52.199949169 +0000 2023-05-31T19:48:02.444 INFO:teuthology.orchestra.run.smithi099.stdout: Birth: - 2023-05-31T19:48:02.445 DEBUG:teuthology.orchestra.run.smithi099:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2023-05-31T19:48:02.507 INFO:teuthology.orchestra.run.smithi099.stderr:1+0 records in 2023-05-31T19:48:02.507 INFO:teuthology.orchestra.run.smithi099.stderr:1+0 records out 2023-05-31T19:48:02.507 INFO:teuthology.orchestra.run.smithi099.stderr:512 bytes copied, 0.00024289 s, 2.1 MB/s 2023-05-31T19:48:02.509 DEBUG:teuthology.orchestra.run.smithi099:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2023-05-31T19:48:02.569 INFO:tasks.ceph:osd dev map: {'c1.osd.0': '/dev/vg_nvme/lv_1', 'c1.osd.1': '/dev/vg_nvme/lv_2', 'c1.osd.2': '/dev/vg_nvme/lv_3'} 2023-05-31T19:48:02.569 INFO:tasks.ceph:remote_to_roles_to_devs: {Remote(name='ubuntu@smithi099.front.sepia.ceph.com'): {'c1.osd.0': '/dev/vg_nvme/lv_1', 'c1.osd.1': '/dev/vg_nvme/lv_2', 'c1.osd.2': '/dev/vg_nvme/lv_3'}} 2023-05-31T19:48:02.569 INFO:tasks.ceph:Generating config... 2023-05-31T19:48:02.571 INFO:tasks.ceph:[client] debug rgw = 20 2023-05-31T19:48:02.571 INFO:tasks.ceph:[client] debug rgw sync = 20 2023-05-31T19:48:02.571 INFO:tasks.ceph:[client] rgw crypt require ssl = False 2023-05-31T19:48:02.572 INFO:tasks.ceph:[client] rgw crypt s3 kms backend = testing 2023-05-31T19:48:02.572 INFO:tasks.ceph:[client] rgw crypt s3 kms encryption keys = testkey-1=YmluCmJvb3N0CmJvb3N0LWJ1aWxkCmNlcGguY29uZgo= 2023-05-31T19:48:02.572 INFO:tasks.ceph:[client] rgw curl low speed time = 300 2023-05-31T19:48:02.572 INFO:tasks.ceph:[client] rgw data log num shards = 4 2023-05-31T19:48:02.572 INFO:tasks.ceph:[client] rgw md log max shards = 4 2023-05-31T19:48:02.573 INFO:tasks.ceph:[client] rgw sync data inject err probability = 0.1 2023-05-31T19:48:02.573 INFO:tasks.ceph:[client] rgw sync log trim interval = 0 2023-05-31T19:48:02.573 INFO:tasks.ceph:[client] rgw sync meta inject err probability = 0.1 2023-05-31T19:48:02.574 INFO:tasks.ceph:[client] rgw sync obj etag verify = True 2023-05-31T19:48:02.574 INFO:tasks.ceph:[client] setgroup = ceph 2023-05-31T19:48:02.574 INFO:tasks.ceph:[client] setuser = ceph 2023-05-31T19:48:02.575 INFO:tasks.ceph:[client.0] rgw data notify interval msec = 0 2023-05-31T19:48:02.575 INFO:tasks.ceph:[mgr] debug mgr = 20 2023-05-31T19:48:02.576 INFO:tasks.ceph:[mgr] debug ms = 1 2023-05-31T19:48:02.576 INFO:tasks.ceph:[mon] debug mon = 20 2023-05-31T19:48:02.576 INFO:tasks.ceph:[mon] debug ms = 1 2023-05-31T19:48:02.576 INFO:tasks.ceph:[mon] debug paxos = 20 2023-05-31T19:48:02.576 INFO:tasks.ceph:[osd] debug ms = 1 2023-05-31T19:48:02.577 INFO:tasks.ceph:[osd] debug osd = 20 2023-05-31T19:48:02.577 INFO:tasks.ceph:[osd] rocksdb delete range threshold = 0 2023-05-31T19:48:02.577 INFO:tasks.ceph:[osd.0] osd_max_omap_entries_per_request = 10 2023-05-31T19:48:02.577 INFO:tasks.ceph:[osd.1] osd_max_omap_entries_per_request = 1000 2023-05-31T19:48:02.577 INFO:tasks.ceph:[osd.2] osd_max_omap_entries_per_request = 10000 2023-05-31T19:48:02.578 INFO:tasks.ceph:Setting up c1.mon.a... 2023-05-31T19:48:02.578 DEBUG:teuthology.orchestra.run.smithi099:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring /etc/ceph/c1.keyring 2023-05-31T19:48:02.663 INFO:teuthology.orchestra.run.smithi099.stdout:creating /etc/ceph/c1.keyring 2023-05-31T19:48:02.665 DEBUG:teuthology.orchestra.run.smithi099:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=mon. /etc/ceph/c1.keyring 2023-05-31T19:48:02.710 DEBUG:teuthology.orchestra.run.smithi099:> sudo chmod 0644 /etc/ceph/c1.keyring 2023-05-31T19:48:02.738 DEBUG:tasks.ceph:Ceph mon addresses: [('c1.mon.a', '172.21.15.99')] 2023-05-31T19:48:02.742 DEBUG:tasks.ceph:writing out conf {'global': {'chdir': '', 'pid file': '/var/run/ceph/$cluster-$name.pid', 'auth supported': 'cephx', 'filestore xattr use omap': 'true', 'mon clock drift allowed': '1.000', 'osd crush chooseleaf type': '0', 'auth debug': 'true', 'ms die on old message': 'true', 'ms die on bug': 'true', 'mon max pg per osd': '10000', 'mon pg warn max object skew': '0', 'osd_pool_default_pg_autoscale_mode': 'off', 'osd pool default size': '2', 'mon osd allow primary affinity': 'true', 'mon osd allow pg remap': 'true', 'mon warn on legacy crush tunables': 'false', 'mon warn on crush straw calc version zero': 'false', 'mon warn on no sortbitwise': 'false', 'mon warn on osd down out interval zero': 'false', 'mon warn on too few osds': 'false', 'mon_warn_on_pool_pg_num_not_power_of_two': 'false', 'mon_warn_on_pool_no_redundancy': 'false', 'mon_allow_pool_size_one': 'true', 'osd pool default erasure code profile': 'plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd', 'osd default data pool replay window': '5', 'mon allow pool delete': 'true', 'mon cluster log file level': 'debug', 'debug asserts on shutdown': 'true', 'mon health detail to clog': 'false', 'mon host': '172.21.15.99'}, 'osd': {'osd journal size': '100', 'osd scrub load threshold': '5.0', 'osd scrub max interval': '600', 'osd mclock profile': 'high_recovery_ops', 'osd recover clone overlap': 'true', 'osd recovery max chunk': '1048576', 'osd debug shutdown': 'true', 'osd debug op order': 'true', 'osd debug verify stray on activate': 'true', 'osd open classes on start': 'true', 'osd debug pg log writeout': 'true', 'osd deep scrub update digest min age': '30', 'osd map max advance': '10', 'journal zero on create': 'true', 'filestore ondisk finisher threads': '3', 'filestore apply finisher threads': '3', 'bdev debug aio': 'true', 'osd debug misdirected ops': 'true', 'debug ms': 1, 'debug osd': 20, 'rocksdb delete range threshold': 0}, 'mgr': {'debug ms': 1, 'debug mgr': 20, 'debug mon': '20', 'debug auth': '20', 'mon reweight min pgs per osd': '4', 'mon reweight min bytes per osd': '10', 'mgr/telemetry/nag': 'false'}, 'mon': {'debug ms': 1, 'debug mon': 20, 'debug paxos': 20, 'debug auth': '20', 'mon data avail warn': '5', 'mon mgr mkfs grace': '240', 'mon reweight min pgs per osd': '4', 'mon osd reporter subtree level': 'osd', 'mon osd prime pg temp': 'true', 'mon reweight min bytes per osd': '10', 'auth mon ticket ttl': '660', 'auth service ticket ttl': '240', 'mon_warn_on_insecure_global_id_reclaim': 'false', 'mon_warn_on_insecure_global_id_reclaim_allowed': 'false', 'mon_down_mkfs_grace': '2m', 'mon_warn_on_filestore_osds': 'false'}, 'client': {'rgw cache enabled': 'true', 'rgw enable ops log': 'true', 'rgw enable usage log': 'true', 'log file': '/var/log/ceph/$cluster-$name.$pid.log', 'admin socket': '/var/run/ceph/$cluster-$name.$pid.asok', 'debug rgw': 20, 'debug rgw sync': 20, 'rgw crypt require ssl': False, 'rgw crypt s3 kms backend': 'testing', 'rgw crypt s3 kms encryption keys': 'testkey-1=YmluCmJvb3N0CmJvb3N0LWJ1aWxkCmNlcGguY29uZgo=', 'rgw curl low speed time': 300, 'rgw data log num shards': 4, 'rgw md log max shards': 4, 'rgw sync data inject err probability': 0.1, 'rgw sync log trim interval': 0, 'rgw sync meta inject err probability': 0.1, 'rgw sync obj etag verify': True, 'setgroup': 'ceph', 'setuser': 'ceph'}, 'mon.a': {}, 'client.0': {'rgw data notify interval msec': 0}, 'osd.0': {'osd_max_omap_entries_per_request': 10}, 'osd.1': {'osd_max_omap_entries_per_request': 1000}, 'osd.2': {'osd_max_omap_entries_per_request': 10000}} 2023-05-31T19:48:02.743 DEBUG:teuthology.orchestra.run.smithi099:> set -ex 2023-05-31T19:48:02.744 DEBUG:teuthology.orchestra.run.smithi099:> dd of=/home/ubuntu/cephtest/ceph.tmp.conf 2023-05-31T19:48:02.797 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage monmaptool -c /home/ubuntu/cephtest/ceph.tmp.conf --create --clobber --enable-all-features --add a 172.21.15.99 --print /home/ubuntu/cephtest/c1.monmap 2023-05-31T19:48:02.879 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2023-05-31T19:48:02.880 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2023-05-31T19:48:02.880 INFO:teuthology.orchestra.run.smithi099.stdout:monmaptool: monmap file /home/ubuntu/cephtest/c1.monmap 2023-05-31T19:48:02.881 INFO:teuthology.orchestra.run.smithi099.stdout:monmaptool: generated fsid 7c5f80fb-3341-4f1e-be16-38b17ec1b3d0 2023-05-31T19:48:02.881 INFO:teuthology.orchestra.run.smithi099.stdout:setting min_mon_release = pacific 2023-05-31T19:48:02.881 INFO:teuthology.orchestra.run.smithi099.stdout:epoch 0 2023-05-31T19:48:02.881 INFO:teuthology.orchestra.run.smithi099.stdout:fsid 7c5f80fb-3341-4f1e-be16-38b17ec1b3d0 2023-05-31T19:48:02.881 INFO:teuthology.orchestra.run.smithi099.stdout:last_changed 2023-05-31T19:48:02.875801+0000 2023-05-31T19:48:02.881 INFO:teuthology.orchestra.run.smithi099.stdout:created 2023-05-31T19:48:02.875801+0000 2023-05-31T19:48:02.882 INFO:teuthology.orchestra.run.smithi099.stdout:min_mon_release 16 (pacific) 2023-05-31T19:48:02.882 INFO:teuthology.orchestra.run.smithi099.stdout:election_strategy: 1 2023-05-31T19:48:02.882 INFO:teuthology.orchestra.run.smithi099.stdout:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2023-05-31T19:48:02.882 INFO:teuthology.orchestra.run.smithi099.stdout:monmaptool: writing epoch 0 to /home/ubuntu/cephtest/c1.monmap (1 monitors) 2023-05-31T19:48:02.883 DEBUG:teuthology.orchestra.run.smithi099:> rm -- /home/ubuntu/cephtest/ceph.tmp.conf 2023-05-31T19:48:02.941 INFO:tasks.ceph:Writing /etc/ceph/c1.conf for FSID 7c5f80fb-3341-4f1e-be16-38b17ec1b3d0... 2023-05-31T19:48:02.942 DEBUG:teuthology.orchestra.run.smithi099:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/c1.conf && sudo chmod 0644 /etc/ceph/c1.conf > /dev/null 2023-05-31T19:48:02.983 DEBUG:teuthology.orchestra.run.smithi112:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/c1.conf && sudo chmod 0644 /etc/ceph/c1.conf > /dev/null 2023-05-31T19:48:03.026 INFO:teuthology.orchestra.run.smithi099.stdout:[global] 2023-05-31T19:48:03.026 INFO:teuthology.orchestra.run.smithi099.stdout: chdir = "" 2023-05-31T19:48:03.026 INFO:teuthology.orchestra.run.smithi099.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2023-05-31T19:48:03.026 INFO:teuthology.orchestra.run.smithi099.stdout: auth supported = cephx 2023-05-31T19:48:03.026 INFO:teuthology.orchestra.run.smithi099.stdout: 2023-05-31T19:48:03.027 INFO:teuthology.orchestra.run.smithi099.stdout: filestore xattr use omap = true 2023-05-31T19:48:03.027 INFO:teuthology.orchestra.run.smithi099.stdout: 2023-05-31T19:48:03.027 INFO:teuthology.orchestra.run.smithi099.stdout: mon clock drift allowed = 1.000 2023-05-31T19:48:03.027 INFO:teuthology.orchestra.run.smithi099.stdout: 2023-05-31T19:48:03.027 INFO:teuthology.orchestra.run.smithi099.stdout: osd crush chooseleaf type = 0 2023-05-31T19:48:03.027 INFO:teuthology.orchestra.run.smithi099.stdout: auth debug = true 2023-05-31T19:48:03.027 INFO:teuthology.orchestra.run.smithi099.stdout: 2023-05-31T19:48:03.028 INFO:teuthology.orchestra.run.smithi099.stdout: ms die on old message = true 2023-05-31T19:48:03.028 INFO:teuthology.orchestra.run.smithi099.stdout: ms die on bug = true 2023-05-31T19:48:03.028 INFO:teuthology.orchestra.run.smithi099.stdout: 2023-05-31T19:48:03.028 INFO:teuthology.orchestra.run.smithi099.stdout: mon max pg per osd = 10000 # >= luminous 2023-05-31T19:48:03.028 INFO:teuthology.orchestra.run.smithi099.stdout: mon pg warn max object skew = 0 2023-05-31T19:48:03.028 INFO:teuthology.orchestra.run.smithi099.stdout: 2023-05-31T19:48:03.029 INFO:teuthology.orchestra.run.smithi099.stdout: # disable pg_autoscaler by default for new pools 2023-05-31T19:48:03.029 INFO:teuthology.orchestra.run.smithi099.stdout: osd_pool_default_pg_autoscale_mode = off 2023-05-31T19:48:03.029 INFO:teuthology.orchestra.run.smithi099.stdout: 2023-05-31T19:48:03.029 INFO:teuthology.orchestra.run.smithi099.stdout: osd pool default size = 2 2023-05-31T19:48:03.029 INFO:teuthology.orchestra.run.smithi099.stdout: 2023-05-31T19:48:03.029 INFO:teuthology.orchestra.run.smithi099.stdout: mon osd allow primary affinity = true 2023-05-31T19:48:03.029 INFO:teuthology.orchestra.run.smithi099.stdout: mon osd allow pg remap = true 2023-05-31T19:48:03.030 INFO:teuthology.orchestra.run.smithi099.stdout: mon warn on legacy crush tunables = false 2023-05-31T19:48:03.030 INFO:teuthology.orchestra.run.smithi099.stdout: mon warn on crush straw calc version zero = false 2023-05-31T19:48:03.030 INFO:teuthology.orchestra.run.smithi099.stdout: mon warn on no sortbitwise = false 2023-05-31T19:48:03.030 INFO:teuthology.orchestra.run.smithi099.stdout: mon warn on osd down out interval zero = false 2023-05-31T19:48:03.030 INFO:teuthology.orchestra.run.smithi099.stdout: mon warn on too few osds = false 2023-05-31T19:48:03.030 INFO:teuthology.orchestra.run.smithi099.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2023-05-31T19:48:03.030 INFO:teuthology.orchestra.run.smithi099.stdout: mon_warn_on_pool_no_redundancy = false 2023-05-31T19:48:03.031 INFO:teuthology.orchestra.run.smithi099.stdout: mon_allow_pool_size_one = true 2023-05-31T19:48:03.031 INFO:teuthology.orchestra.run.smithi099.stdout: 2023-05-31T19:48:03.031 INFO:teuthology.orchestra.run.smithi099.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2023-05-31T19:48:03.031 INFO:teuthology.orchestra.run.smithi099.stdout: 2023-05-31T19:48:03.031 INFO:teuthology.orchestra.run.smithi099.stdout: osd default data pool replay window = 5 2023-05-31T19:48:03.031 INFO:teuthology.orchestra.run.smithi099.stdout: 2023-05-31T19:48:03.031 INFO:teuthology.orchestra.run.smithi099.stdout: mon allow pool delete = true 2023-05-31T19:48:03.031 INFO:teuthology.orchestra.run.smithi099.stdout: 2023-05-31T19:48:03.032 INFO:teuthology.orchestra.run.smithi099.stdout: mon cluster log file level = debug 2023-05-31T19:48:03.032 INFO:teuthology.orchestra.run.smithi099.stdout: debug asserts on shutdown = true 2023-05-31T19:48:03.032 INFO:teuthology.orchestra.run.smithi099.stdout: mon health detail to clog = false 2023-05-31T19:48:03.032 INFO:teuthology.orchestra.run.smithi099.stdout: mon host = 172.21.15.99 2023-05-31T19:48:03.032 INFO:teuthology.orchestra.run.smithi099.stdout: fsid = 7c5f80fb-3341-4f1e-be16-38b17ec1b3d0 2023-05-31T19:48:03.032 INFO:teuthology.orchestra.run.smithi099.stdout: 2023-05-31T19:48:03.032 INFO:teuthology.orchestra.run.smithi099.stdout:[osd] 2023-05-31T19:48:03.032 INFO:teuthology.orchestra.run.smithi099.stdout: osd journal size = 100 2023-05-31T19:48:03.033 INFO:teuthology.orchestra.run.smithi099.stdout: 2023-05-31T19:48:03.033 INFO:teuthology.orchestra.run.smithi099.stdout: osd scrub load threshold = 5.0 2023-05-31T19:48:03.033 INFO:teuthology.orchestra.run.smithi099.stdout: osd scrub max interval = 600 2023-05-31T19:48:03.033 INFO:teuthology.orchestra.run.smithi099.stdout: osd mclock profile = high_recovery_ops 2023-05-31T19:48:03.033 INFO:teuthology.orchestra.run.smithi099.stdout: 2023-05-31T19:48:03.033 INFO:teuthology.orchestra.run.smithi099.stdout: osd recover clone overlap = true 2023-05-31T19:48:03.033 INFO:teuthology.orchestra.run.smithi099.stdout: osd recovery max chunk = 1048576 2023-05-31T19:48:03.034 INFO:teuthology.orchestra.run.smithi099.stdout: 2023-05-31T19:48:03.034 INFO:teuthology.orchestra.run.smithi099.stdout: osd debug shutdown = true 2023-05-31T19:48:03.034 INFO:teuthology.orchestra.run.smithi099.stdout: osd debug op order = true 2023-05-31T19:48:03.034 INFO:teuthology.orchestra.run.smithi099.stdout: osd debug verify stray on activate = true 2023-05-31T19:48:03.034 INFO:teuthology.orchestra.run.smithi099.stdout: 2023-05-31T19:48:03.034 INFO:teuthology.orchestra.run.smithi099.stdout: osd open classes on start = true 2023-05-31T19:48:03.035 INFO:teuthology.orchestra.run.smithi099.stdout: osd debug pg log writeout = true 2023-05-31T19:48:03.035 INFO:teuthology.orchestra.run.smithi099.stdout: 2023-05-31T19:48:03.035 INFO:teuthology.orchestra.run.smithi099.stdout: osd deep scrub update digest min age = 30 2023-05-31T19:48:03.035 INFO:teuthology.orchestra.run.smithi099.stdout: 2023-05-31T19:48:03.035 INFO:teuthology.orchestra.run.smithi099.stdout: osd map max advance = 10 2023-05-31T19:48:03.035 INFO:teuthology.orchestra.run.smithi099.stdout: 2023-05-31T19:48:03.035 INFO:teuthology.orchestra.run.smithi099.stdout: journal zero on create = true 2023-05-31T19:48:03.035 INFO:teuthology.orchestra.run.smithi099.stdout: 2023-05-31T19:48:03.036 INFO:teuthology.orchestra.run.smithi099.stdout: filestore ondisk finisher threads = 3 2023-05-31T19:48:03.036 INFO:teuthology.orchestra.run.smithi099.stdout: filestore apply finisher threads = 3 2023-05-31T19:48:03.036 INFO:teuthology.orchestra.run.smithi099.stdout: 2023-05-31T19:48:03.036 INFO:teuthology.orchestra.run.smithi099.stdout: bdev debug aio = true 2023-05-31T19:48:03.036 INFO:teuthology.orchestra.run.smithi099.stdout: osd debug misdirected ops = true 2023-05-31T19:48:03.036 INFO:teuthology.orchestra.run.smithi099.stdout: debug ms = 1 2023-05-31T19:48:03.036 INFO:teuthology.orchestra.run.smithi099.stdout: debug osd = 20 2023-05-31T19:48:03.037 INFO:teuthology.orchestra.run.smithi099.stdout: rocksdb delete range threshold = 0 2023-05-31T19:48:03.037 INFO:teuthology.orchestra.run.smithi099.stdout: 2023-05-31T19:48:03.037 INFO:teuthology.orchestra.run.smithi099.stdout:[mgr] 2023-05-31T19:48:03.037 INFO:teuthology.orchestra.run.smithi099.stdout: debug ms = 1 2023-05-31T19:48:03.037 INFO:teuthology.orchestra.run.smithi099.stdout: debug mgr = 20 2023-05-31T19:48:03.037 INFO:teuthology.orchestra.run.smithi099.stdout: debug mon = 20 2023-05-31T19:48:03.037 INFO:teuthology.orchestra.run.smithi099.stdout: debug auth = 20 2023-05-31T19:48:03.037 INFO:teuthology.orchestra.run.smithi099.stdout: mon reweight min pgs per osd = 4 2023-05-31T19:48:03.038 INFO:teuthology.orchestra.run.smithi099.stdout: mon reweight min bytes per osd = 10 2023-05-31T19:48:03.038 INFO:teuthology.orchestra.run.smithi099.stdout: mgr/telemetry/nag = false 2023-05-31T19:48:03.038 INFO:teuthology.orchestra.run.smithi099.stdout: 2023-05-31T19:48:03.038 INFO:teuthology.orchestra.run.smithi099.stdout:[mon] 2023-05-31T19:48:03.038 INFO:teuthology.orchestra.run.smithi099.stdout: debug ms = 1 2023-05-31T19:48:03.038 INFO:teuthology.orchestra.run.smithi099.stdout: debug mon = 20 2023-05-31T19:48:03.038 INFO:teuthology.orchestra.run.smithi099.stdout: debug paxos = 20 2023-05-31T19:48:03.038 INFO:teuthology.orchestra.run.smithi099.stdout: debug auth = 20 2023-05-31T19:48:03.039 INFO:teuthology.orchestra.run.smithi099.stdout: mon data avail warn = 5 2023-05-31T19:48:03.039 INFO:teuthology.orchestra.run.smithi099.stdout: mon mgr mkfs grace = 240 2023-05-31T19:48:03.039 INFO:teuthology.orchestra.run.smithi099.stdout: mon reweight min pgs per osd = 4 2023-05-31T19:48:03.039 INFO:teuthology.orchestra.run.smithi099.stdout: mon osd reporter subtree level = osd 2023-05-31T19:48:03.039 INFO:teuthology.orchestra.run.smithi099.stdout: mon osd prime pg temp = true 2023-05-31T19:48:03.039 INFO:teuthology.orchestra.run.smithi099.stdout: mon reweight min bytes per osd = 10 2023-05-31T19:48:03.039 INFO:teuthology.orchestra.run.smithi099.stdout: 2023-05-31T19:48:03.039 INFO:teuthology.orchestra.run.smithi099.stdout: # rotate auth tickets quickly to exercise renewal paths 2023-05-31T19:48:03.040 INFO:teuthology.orchestra.run.smithi099.stdout: auth mon ticket ttl = 660 # 11m 2023-05-31T19:48:03.040 INFO:teuthology.orchestra.run.smithi099.stdout: auth service ticket ttl = 240 # 4m 2023-05-31T19:48:03.040 INFO:teuthology.orchestra.run.smithi099.stdout: 2023-05-31T19:48:03.040 INFO:teuthology.orchestra.run.smithi099.stdout: # don't complain about insecure global_id in the test suite 2023-05-31T19:48:03.040 INFO:teuthology.orchestra.run.smithi099.stdout: mon_warn_on_insecure_global_id_reclaim = false 2023-05-31T19:48:03.040 INFO:teuthology.orchestra.run.smithi099.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2023-05-31T19:48:03.040 INFO:teuthology.orchestra.run.smithi099.stdout: 2023-05-31T19:48:03.041 INFO:teuthology.orchestra.run.smithi099.stdout: # 1m isn't quite enough 2023-05-31T19:48:03.041 INFO:teuthology.orchestra.run.smithi099.stdout: mon_down_mkfs_grace = 2m 2023-05-31T19:48:03.041 INFO:teuthology.orchestra.run.smithi099.stdout: 2023-05-31T19:48:03.041 INFO:teuthology.orchestra.run.smithi099.stdout: mon_warn_on_filestore_osds = false 2023-05-31T19:48:03.041 INFO:teuthology.orchestra.run.smithi099.stdout: 2023-05-31T19:48:03.041 INFO:teuthology.orchestra.run.smithi099.stdout:[client] 2023-05-31T19:48:03.041 INFO:teuthology.orchestra.run.smithi099.stdout: rgw cache enabled = true 2023-05-31T19:48:03.041 INFO:teuthology.orchestra.run.smithi099.stdout: rgw enable ops log = true 2023-05-31T19:48:03.042 INFO:teuthology.orchestra.run.smithi099.stdout: rgw enable usage log = true 2023-05-31T19:48:03.042 INFO:teuthology.orchestra.run.smithi099.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2023-05-31T19:48:03.042 INFO:teuthology.orchestra.run.smithi099.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2023-05-31T19:48:03.042 INFO:teuthology.orchestra.run.smithi099.stdout: debug rgw = 20 2023-05-31T19:48:03.042 INFO:teuthology.orchestra.run.smithi099.stdout: debug rgw sync = 20 2023-05-31T19:48:03.042 INFO:teuthology.orchestra.run.smithi099.stdout: rgw crypt require ssl = False 2023-05-31T19:48:03.042 INFO:teuthology.orchestra.run.smithi099.stdout: rgw crypt s3 kms backend = testing 2023-05-31T19:48:03.042 INFO:teuthology.orchestra.run.smithi099.stdout: rgw crypt s3 kms encryption keys = testkey-1=YmluCmJvb3N0CmJvb3N0LWJ1aWxkCmNlcGguY29uZgo= 2023-05-31T19:48:03.043 INFO:teuthology.orchestra.run.smithi099.stdout: rgw curl low speed time = 300 2023-05-31T19:48:03.043 INFO:teuthology.orchestra.run.smithi099.stdout: rgw data log num shards = 4 2023-05-31T19:48:03.043 INFO:teuthology.orchestra.run.smithi099.stdout: rgw md log max shards = 4 2023-05-31T19:48:03.043 INFO:teuthology.orchestra.run.smithi099.stdout: rgw sync data inject err probability = 0.1 2023-05-31T19:48:03.043 INFO:teuthology.orchestra.run.smithi099.stdout: rgw sync log trim interval = 0 2023-05-31T19:48:03.043 INFO:teuthology.orchestra.run.smithi099.stdout: rgw sync meta inject err probability = 0.1 2023-05-31T19:48:03.043 INFO:teuthology.orchestra.run.smithi099.stdout: rgw sync obj etag verify = True 2023-05-31T19:48:03.044 INFO:teuthology.orchestra.run.smithi099.stdout: setgroup = ceph 2023-05-31T19:48:03.044 INFO:teuthology.orchestra.run.smithi099.stdout: setuser = ceph 2023-05-31T19:48:03.044 INFO:teuthology.orchestra.run.smithi099.stdout:[mon.a] 2023-05-31T19:48:03.044 INFO:teuthology.orchestra.run.smithi099.stdout:[client.0] 2023-05-31T19:48:03.044 INFO:teuthology.orchestra.run.smithi099.stdout: rgw data notify interval msec = 0 2023-05-31T19:48:03.044 INFO:teuthology.orchestra.run.smithi099.stdout:[osd.0] 2023-05-31T19:48:03.044 INFO:teuthology.orchestra.run.smithi099.stdout: osd_max_omap_entries_per_request = 10 2023-05-31T19:48:03.044 INFO:teuthology.orchestra.run.smithi099.stdout:[osd.1] 2023-05-31T19:48:03.045 INFO:teuthology.orchestra.run.smithi099.stdout: osd_max_omap_entries_per_request = 1000 2023-05-31T19:48:03.045 INFO:teuthology.orchestra.run.smithi099.stdout:[osd.2] 2023-05-31T19:48:03.045 INFO:teuthology.orchestra.run.smithi099.stdout: osd_max_omap_entries_per_request = 10000 2023-05-31T19:48:03.046 INFO:teuthology.orchestra.run.smithi112.stdout:[global] 2023-05-31T19:48:03.046 INFO:teuthology.orchestra.run.smithi112.stdout: chdir = "" 2023-05-31T19:48:03.046 INFO:teuthology.orchestra.run.smithi112.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2023-05-31T19:48:03.046 INFO:teuthology.orchestra.run.smithi112.stdout: auth supported = cephx 2023-05-31T19:48:03.046 INFO:teuthology.orchestra.run.smithi112.stdout: 2023-05-31T19:48:03.046 INFO:teuthology.orchestra.run.smithi112.stdout: filestore xattr use omap = true 2023-05-31T19:48:03.047 INFO:teuthology.orchestra.run.smithi112.stdout: 2023-05-31T19:48:03.048 INFO:teuthology.orchestra.run.smithi112.stdout: mon clock drift allowed = 1.000 2023-05-31T19:48:03.048 INFO:teuthology.orchestra.run.smithi112.stdout: 2023-05-31T19:48:03.048 INFO:teuthology.orchestra.run.smithi112.stdout: osd crush chooseleaf type = 0 2023-05-31T19:48:03.048 INFO:teuthology.orchestra.run.smithi112.stdout: auth debug = true 2023-05-31T19:48:03.048 INFO:teuthology.orchestra.run.smithi112.stdout: 2023-05-31T19:48:03.048 INFO:teuthology.orchestra.run.smithi112.stdout: ms die on old message = true 2023-05-31T19:48:03.048 INFO:teuthology.orchestra.run.smithi112.stdout: ms die on bug = true 2023-05-31T19:48:03.049 INFO:teuthology.orchestra.run.smithi112.stdout: 2023-05-31T19:48:03.049 INFO:teuthology.orchestra.run.smithi112.stdout: mon max pg per osd = 10000 # >= luminous 2023-05-31T19:48:03.049 INFO:teuthology.orchestra.run.smithi112.stdout: mon pg warn max object skew = 0 2023-05-31T19:48:03.049 INFO:teuthology.orchestra.run.smithi112.stdout: 2023-05-31T19:48:03.049 INFO:teuthology.orchestra.run.smithi112.stdout: # disable pg_autoscaler by default for new pools 2023-05-31T19:48:03.049 INFO:teuthology.orchestra.run.smithi112.stdout: osd_pool_default_pg_autoscale_mode = off 2023-05-31T19:48:03.049 INFO:teuthology.orchestra.run.smithi112.stdout: 2023-05-31T19:48:03.049 INFO:teuthology.orchestra.run.smithi112.stdout: osd pool default size = 2 2023-05-31T19:48:03.050 INFO:teuthology.orchestra.run.smithi112.stdout: 2023-05-31T19:48:03.050 INFO:teuthology.orchestra.run.smithi112.stdout: mon osd allow primary affinity = true 2023-05-31T19:48:03.050 INFO:teuthology.orchestra.run.smithi112.stdout: mon osd allow pg remap = true 2023-05-31T19:48:03.050 INFO:teuthology.orchestra.run.smithi112.stdout: mon warn on legacy crush tunables = false 2023-05-31T19:48:03.050 INFO:teuthology.orchestra.run.smithi112.stdout: mon warn on crush straw calc version zero = false 2023-05-31T19:48:03.050 INFO:teuthology.orchestra.run.smithi112.stdout: mon warn on no sortbitwise = false 2023-05-31T19:48:03.050 INFO:teuthology.orchestra.run.smithi112.stdout: mon warn on osd down out interval zero = false 2023-05-31T19:48:03.050 INFO:teuthology.orchestra.run.smithi112.stdout: mon warn on too few osds = false 2023-05-31T19:48:03.051 INFO:teuthology.orchestra.run.smithi112.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2023-05-31T19:48:03.051 INFO:teuthology.orchestra.run.smithi112.stdout: mon_warn_on_pool_no_redundancy = false 2023-05-31T19:48:03.051 INFO:teuthology.orchestra.run.smithi112.stdout: mon_allow_pool_size_one = true 2023-05-31T19:48:03.051 INFO:teuthology.orchestra.run.smithi112.stdout: 2023-05-31T19:48:03.051 INFO:teuthology.orchestra.run.smithi112.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2023-05-31T19:48:03.051 INFO:teuthology.orchestra.run.smithi112.stdout: 2023-05-31T19:48:03.051 INFO:teuthology.orchestra.run.smithi112.stdout: osd default data pool replay window = 5 2023-05-31T19:48:03.052 INFO:teuthology.orchestra.run.smithi112.stdout: 2023-05-31T19:48:03.052 INFO:teuthology.orchestra.run.smithi112.stdout: mon allow pool delete = true 2023-05-31T19:48:03.052 INFO:teuthology.orchestra.run.smithi112.stdout: 2023-05-31T19:48:03.052 INFO:teuthology.orchestra.run.smithi112.stdout: mon cluster log file level = debug 2023-05-31T19:48:03.052 INFO:teuthology.orchestra.run.smithi112.stdout: debug asserts on shutdown = true 2023-05-31T19:48:03.052 INFO:teuthology.orchestra.run.smithi112.stdout: mon health detail to clog = false 2023-05-31T19:48:03.052 INFO:teuthology.orchestra.run.smithi112.stdout: mon host = 172.21.15.99 2023-05-31T19:48:03.053 INFO:teuthology.orchestra.run.smithi112.stdout: fsid = 7c5f80fb-3341-4f1e-be16-38b17ec1b3d0 2023-05-31T19:48:03.053 INFO:teuthology.orchestra.run.smithi112.stdout: 2023-05-31T19:48:03.053 INFO:teuthology.orchestra.run.smithi112.stdout:[osd] 2023-05-31T19:48:03.053 INFO:teuthology.orchestra.run.smithi112.stdout: osd journal size = 100 2023-05-31T19:48:03.053 INFO:teuthology.orchestra.run.smithi112.stdout: 2023-05-31T19:48:03.053 INFO:teuthology.orchestra.run.smithi112.stdout: osd scrub load threshold = 5.0 2023-05-31T19:48:03.053 INFO:teuthology.orchestra.run.smithi112.stdout: osd scrub max interval = 600 2023-05-31T19:48:03.053 INFO:teuthology.orchestra.run.smithi112.stdout: osd mclock profile = high_recovery_ops 2023-05-31T19:48:03.054 INFO:teuthology.orchestra.run.smithi112.stdout: 2023-05-31T19:48:03.054 INFO:teuthology.orchestra.run.smithi112.stdout: osd recover clone overlap = true 2023-05-31T19:48:03.054 INFO:teuthology.orchestra.run.smithi112.stdout: osd recovery max chunk = 1048576 2023-05-31T19:48:03.054 INFO:teuthology.orchestra.run.smithi112.stdout: 2023-05-31T19:48:03.054 INFO:teuthology.orchestra.run.smithi112.stdout: osd debug shutdown = true 2023-05-31T19:48:03.054 INFO:teuthology.orchestra.run.smithi112.stdout: osd debug op order = true 2023-05-31T19:48:03.054 INFO:teuthology.orchestra.run.smithi112.stdout: osd debug verify stray on activate = true 2023-05-31T19:48:03.054 INFO:teuthology.orchestra.run.smithi112.stdout: 2023-05-31T19:48:03.055 INFO:teuthology.orchestra.run.smithi112.stdout: osd open classes on start = true 2023-05-31T19:48:03.055 INFO:teuthology.orchestra.run.smithi112.stdout: osd debug pg log writeout = true 2023-05-31T19:48:03.055 INFO:teuthology.orchestra.run.smithi112.stdout: 2023-05-31T19:48:03.055 INFO:teuthology.orchestra.run.smithi112.stdout: osd deep scrub update digest min age = 30 2023-05-31T19:48:03.055 INFO:teuthology.orchestra.run.smithi112.stdout: 2023-05-31T19:48:03.055 INFO:teuthology.orchestra.run.smithi112.stdout: osd map max advance = 10 2023-05-31T19:48:03.055 INFO:teuthology.orchestra.run.smithi112.stdout: 2023-05-31T19:48:03.055 INFO:teuthology.orchestra.run.smithi112.stdout: journal zero on create = true 2023-05-31T19:48:03.056 INFO:teuthology.orchestra.run.smithi112.stdout: 2023-05-31T19:48:03.056 INFO:teuthology.orchestra.run.smithi112.stdout: filestore ondisk finisher threads = 3 2023-05-31T19:48:03.056 INFO:teuthology.orchestra.run.smithi112.stdout: filestore apply finisher threads = 3 2023-05-31T19:48:03.056 INFO:teuthology.orchestra.run.smithi112.stdout: 2023-05-31T19:48:03.056 INFO:teuthology.orchestra.run.smithi112.stdout: bdev debug aio = true 2023-05-31T19:48:03.056 INFO:teuthology.orchestra.run.smithi112.stdout: osd debug misdirected ops = true 2023-05-31T19:48:03.056 INFO:teuthology.orchestra.run.smithi112.stdout: debug ms = 1 2023-05-31T19:48:03.057 INFO:teuthology.orchestra.run.smithi112.stdout: debug osd = 20 2023-05-31T19:48:03.057 INFO:teuthology.orchestra.run.smithi112.stdout: rocksdb delete range threshold = 0 2023-05-31T19:48:03.057 INFO:teuthology.orchestra.run.smithi112.stdout: 2023-05-31T19:48:03.057 INFO:teuthology.orchestra.run.smithi112.stdout:[mgr] 2023-05-31T19:48:03.057 INFO:teuthology.orchestra.run.smithi112.stdout: debug ms = 1 2023-05-31T19:48:03.057 INFO:teuthology.orchestra.run.smithi112.stdout: debug mgr = 20 2023-05-31T19:48:03.057 INFO:teuthology.orchestra.run.smithi112.stdout: debug mon = 20 2023-05-31T19:48:03.057 INFO:teuthology.orchestra.run.smithi112.stdout: debug auth = 20 2023-05-31T19:48:03.058 INFO:teuthology.orchestra.run.smithi112.stdout: mon reweight min pgs per osd = 4 2023-05-31T19:48:03.058 INFO:teuthology.orchestra.run.smithi112.stdout: mon reweight min bytes per osd = 10 2023-05-31T19:48:03.058 INFO:teuthology.orchestra.run.smithi112.stdout: mgr/telemetry/nag = false 2023-05-31T19:48:03.058 INFO:teuthology.orchestra.run.smithi112.stdout: 2023-05-31T19:48:03.058 INFO:teuthology.orchestra.run.smithi112.stdout:[mon] 2023-05-31T19:48:03.058 INFO:teuthology.orchestra.run.smithi112.stdout: debug ms = 1 2023-05-31T19:48:03.058 INFO:teuthology.orchestra.run.smithi112.stdout: debug mon = 20 2023-05-31T19:48:03.058 INFO:teuthology.orchestra.run.smithi112.stdout: debug paxos = 20 2023-05-31T19:48:03.059 INFO:teuthology.orchestra.run.smithi112.stdout: debug auth = 20 2023-05-31T19:48:03.059 INFO:teuthology.orchestra.run.smithi112.stdout: mon data avail warn = 5 2023-05-31T19:48:03.059 INFO:teuthology.orchestra.run.smithi112.stdout: mon mgr mkfs grace = 240 2023-05-31T19:48:03.059 INFO:teuthology.orchestra.run.smithi112.stdout: mon reweight min pgs per osd = 4 2023-05-31T19:48:03.059 INFO:teuthology.orchestra.run.smithi112.stdout: mon osd reporter subtree level = osd 2023-05-31T19:48:03.059 INFO:teuthology.orchestra.run.smithi112.stdout: mon osd prime pg temp = true 2023-05-31T19:48:03.059 INFO:teuthology.orchestra.run.smithi112.stdout: mon reweight min bytes per osd = 10 2023-05-31T19:48:03.059 INFO:teuthology.orchestra.run.smithi112.stdout: 2023-05-31T19:48:03.060 INFO:teuthology.orchestra.run.smithi112.stdout: # rotate auth tickets quickly to exercise renewal paths 2023-05-31T19:48:03.060 INFO:teuthology.orchestra.run.smithi112.stdout: auth mon ticket ttl = 660 # 11m 2023-05-31T19:48:03.060 INFO:teuthology.orchestra.run.smithi112.stdout: auth service ticket ttl = 240 # 4m 2023-05-31T19:48:03.060 INFO:teuthology.orchestra.run.smithi112.stdout: 2023-05-31T19:48:03.060 INFO:teuthology.orchestra.run.smithi112.stdout: # don't complain about insecure global_id in the test suite 2023-05-31T19:48:03.060 INFO:teuthology.orchestra.run.smithi112.stdout: mon_warn_on_insecure_global_id_reclaim = false 2023-05-31T19:48:03.060 INFO:teuthology.orchestra.run.smithi112.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2023-05-31T19:48:03.061 INFO:teuthology.orchestra.run.smithi112.stdout: 2023-05-31T19:48:03.061 INFO:teuthology.orchestra.run.smithi112.stdout: # 1m isn't quite enough 2023-05-31T19:48:03.061 INFO:teuthology.orchestra.run.smithi112.stdout: mon_down_mkfs_grace = 2m 2023-05-31T19:48:03.061 INFO:teuthology.orchestra.run.smithi112.stdout: 2023-05-31T19:48:03.061 INFO:teuthology.orchestra.run.smithi112.stdout: mon_warn_on_filestore_osds = false 2023-05-31T19:48:03.061 INFO:teuthology.orchestra.run.smithi112.stdout: 2023-05-31T19:48:03.061 INFO:teuthology.orchestra.run.smithi112.stdout:[client] 2023-05-31T19:48:03.061 INFO:teuthology.orchestra.run.smithi112.stdout: rgw cache enabled = true 2023-05-31T19:48:03.062 INFO:teuthology.orchestra.run.smithi112.stdout: rgw enable ops log = true 2023-05-31T19:48:03.062 INFO:teuthology.orchestra.run.smithi112.stdout: rgw enable usage log = true 2023-05-31T19:48:03.062 INFO:teuthology.orchestra.run.smithi112.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2023-05-31T19:48:03.062 INFO:teuthology.orchestra.run.smithi112.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2023-05-31T19:48:03.062 INFO:teuthology.orchestra.run.smithi112.stdout: debug rgw = 20 2023-05-31T19:48:03.062 INFO:teuthology.orchestra.run.smithi112.stdout: debug rgw sync = 20 2023-05-31T19:48:03.062 INFO:teuthology.orchestra.run.smithi112.stdout: rgw crypt require ssl = False 2023-05-31T19:48:03.062 INFO:teuthology.orchestra.run.smithi112.stdout: rgw crypt s3 kms backend = testing 2023-05-31T19:48:03.063 INFO:teuthology.orchestra.run.smithi112.stdout: rgw crypt s3 kms encryption keys = testkey-1=YmluCmJvb3N0CmJvb3N0LWJ1aWxkCmNlcGguY29uZgo= 2023-05-31T19:48:03.063 INFO:teuthology.orchestra.run.smithi112.stdout: rgw curl low speed time = 300 2023-05-31T19:48:03.063 INFO:teuthology.orchestra.run.smithi112.stdout: rgw data log num shards = 4 2023-05-31T19:48:03.063 INFO:teuthology.orchestra.run.smithi112.stdout: rgw md log max shards = 4 2023-05-31T19:48:03.063 INFO:teuthology.orchestra.run.smithi112.stdout: rgw sync data inject err probability = 0.1 2023-05-31T19:48:03.063 INFO:teuthology.orchestra.run.smithi112.stdout: rgw sync log trim interval = 0 2023-05-31T19:48:03.063 INFO:teuthology.orchestra.run.smithi112.stdout: rgw sync meta inject err probability = 0.1 2023-05-31T19:48:03.064 INFO:teuthology.orchestra.run.smithi112.stdout: rgw sync obj etag verify = True 2023-05-31T19:48:03.064 INFO:teuthology.orchestra.run.smithi112.stdout: setgroup = ceph 2023-05-31T19:48:03.064 INFO:teuthology.orchestra.run.smithi112.stdout: setuser = ceph 2023-05-31T19:48:03.064 INFO:teuthology.orchestra.run.smithi112.stdout:[mon.a] 2023-05-31T19:48:03.064 INFO:teuthology.orchestra.run.smithi112.stdout:[client.0] 2023-05-31T19:48:03.064 INFO:teuthology.orchestra.run.smithi112.stdout: rgw data notify interval msec = 0 2023-05-31T19:48:03.064 INFO:teuthology.orchestra.run.smithi112.stdout:[osd.0] 2023-05-31T19:48:03.064 INFO:teuthology.orchestra.run.smithi112.stdout: osd_max_omap_entries_per_request = 10 2023-05-31T19:48:03.065 INFO:teuthology.orchestra.run.smithi112.stdout:[osd.1] 2023-05-31T19:48:03.065 INFO:teuthology.orchestra.run.smithi112.stdout: osd_max_omap_entries_per_request = 1000 2023-05-31T19:48:03.065 INFO:teuthology.orchestra.run.smithi112.stdout:[osd.2] 2023-05-31T19:48:03.065 INFO:teuthology.orchestra.run.smithi112.stdout: osd_max_omap_entries_per_request = 10000 2023-05-31T19:48:03.066 INFO:tasks.ceph:Creating admin key on c1.mon.a... 2023-05-31T19:48:03.067 DEBUG:teuthology.orchestra.run.smithi099:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=client.admin --cap mon 'allow *' --cap osd 'allow *' --cap mds 'allow *' --cap mgr 'allow *' /etc/ceph/c1.keyring 2023-05-31T19:48:03.163 INFO:tasks.ceph:Copying monmap to all nodes... 2023-05-31T19:48:03.164 DEBUG:teuthology.orchestra.run.smithi099:> set -ex 2023-05-31T19:48:03.164 DEBUG:teuthology.orchestra.run.smithi099:> dd if=/etc/ceph/c1.keyring of=/dev/stdout 2023-05-31T19:48:03.182 DEBUG:teuthology.orchestra.run.smithi099:> set -ex 2023-05-31T19:48:03.182 DEBUG:teuthology.orchestra.run.smithi099:> dd if=/home/ubuntu/cephtest/c1.monmap of=/dev/stdout 2023-05-31T19:48:03.240 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi099.front.sepia.ceph.com 2023-05-31T19:48:03.240 DEBUG:teuthology.orchestra.run.smithi099:> set -ex 2023-05-31T19:48:03.241 DEBUG:teuthology.orchestra.run.smithi099:> sudo dd of=/etc/ceph/c1.keyring 2023-05-31T19:48:03.241 DEBUG:teuthology.orchestra.run.smithi099:> sudo chmod 0644 /etc/ceph/c1.keyring 2023-05-31T19:48:03.316 DEBUG:teuthology.orchestra.run.smithi099:> set -ex 2023-05-31T19:48:03.317 DEBUG:teuthology.orchestra.run.smithi099:> dd of=/home/ubuntu/cephtest/c1.monmap 2023-05-31T19:48:03.375 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi112.front.sepia.ceph.com 2023-05-31T19:48:03.376 DEBUG:teuthology.orchestra.run.smithi112:> set -ex 2023-05-31T19:48:03.377 DEBUG:teuthology.orchestra.run.smithi112:> sudo dd of=/etc/ceph/c1.keyring 2023-05-31T19:48:03.377 DEBUG:teuthology.orchestra.run.smithi112:> sudo chmod 0644 /etc/ceph/c1.keyring 2023-05-31T19:48:03.424 DEBUG:teuthology.orchestra.run.smithi112:> set -ex 2023-05-31T19:48:03.424 DEBUG:teuthology.orchestra.run.smithi112:> dd of=/home/ubuntu/cephtest/c1.monmap 2023-05-31T19:48:03.484 INFO:tasks.ceph:Setting up mon nodes... 2023-05-31T19:48:03.485 INFO:tasks.ceph:Setting up mgr nodes... 2023-05-31T19:48:03.485 DEBUG:teuthology.orchestra.run.smithi099:> sudo mkdir -p /var/lib/ceph/mgr/c1-x && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mgr.x /var/lib/ceph/mgr/c1-x/keyring 2023-05-31T19:48:03.540 INFO:teuthology.orchestra.run.smithi099.stdout:creating /var/lib/ceph/mgr/c1-x/keyring 2023-05-31T19:48:03.542 INFO:tasks.ceph:Setting up mds nodes... 2023-05-31T19:48:03.542 INFO:tasks.ceph_client:Setting up client nodes... 2023-05-31T19:48:03.543 DEBUG:teuthology.orchestra.run.smithi099:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.0 /etc/ceph/c1.client.0.keyring && sudo chmod 0644 /etc/ceph/c1.client.0.keyring 2023-05-31T19:48:03.586 INFO:teuthology.orchestra.run.smithi099.stdout:creating /etc/ceph/c1.client.0.keyring 2023-05-31T19:48:03.598 DEBUG:teuthology.orchestra.run.smithi099:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.1 /etc/ceph/c1.client.1.keyring && sudo chmod 0644 /etc/ceph/c1.client.1.keyring 2023-05-31T19:48:03.642 INFO:teuthology.orchestra.run.smithi099.stdout:creating /etc/ceph/c1.client.1.keyring 2023-05-31T19:48:03.654 INFO:tasks.ceph:Running mkfs on osd nodes... 2023-05-31T19:48:03.655 INFO:tasks.ceph:ctx.disk_config.remote_to_roles_to_dev: {Remote(name='ubuntu@smithi099.front.sepia.ceph.com'): {'c1.osd.0': '/dev/vg_nvme/lv_1', 'c1.osd.1': '/dev/vg_nvme/lv_2', 'c1.osd.2': '/dev/vg_nvme/lv_3'}} 2023-05-31T19:48:03.655 DEBUG:teuthology.orchestra.run.smithi099:> sudo mkdir -p /var/lib/ceph/osd/c1-0 2023-05-31T19:48:03.682 INFO:tasks.ceph:roles_to_devs: {'c1.osd.0': '/dev/vg_nvme/lv_1', 'c1.osd.1': '/dev/vg_nvme/lv_2', 'c1.osd.2': '/dev/vg_nvme/lv_3'} 2023-05-31T19:48:03.682 INFO:tasks.ceph:role: c1.osd.0 2023-05-31T19:48:03.682 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_1 on ubuntu@smithi099.front.sepia.ceph.com 2023-05-31T19:48:03.683 DEBUG:teuthology.orchestra.run.smithi099:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_1 2023-05-31T19:48:03.757 INFO:teuthology.orchestra.run.smithi099.stdout:meta-data=/dev/vg_nvme/lv_1 isize=2048 agcount=4, agsize=5859072 blks 2023-05-31T19:48:03.757 INFO:teuthology.orchestra.run.smithi099.stdout: = sectsz=512 attr=2, projid32bit=1 2023-05-31T19:48:03.757 INFO:teuthology.orchestra.run.smithi099.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2023-05-31T19:48:03.757 INFO:teuthology.orchestra.run.smithi099.stdout: = reflink=1 bigtime=0 inobtcount=0 2023-05-31T19:48:03.758 INFO:teuthology.orchestra.run.smithi099.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2023-05-31T19:48:03.758 INFO:teuthology.orchestra.run.smithi099.stdout: = sunit=0 swidth=0 blks 2023-05-31T19:48:03.758 INFO:teuthology.orchestra.run.smithi099.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2023-05-31T19:48:03.758 INFO:teuthology.orchestra.run.smithi099.stdout:log =internal log bsize=4096 blocks=11443, version=2 2023-05-31T19:48:03.758 INFO:teuthology.orchestra.run.smithi099.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2023-05-31T19:48:03.758 INFO:teuthology.orchestra.run.smithi099.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2023-05-31T19:48:04.187 INFO:teuthology.orchestra.run.smithi099.stdout:Discarding blocks...Done. 2023-05-31T19:48:04.188 INFO:tasks.ceph:mount /dev/vg_nvme/lv_1 on ubuntu@smithi099.front.sepia.ceph.com -o noatime 2023-05-31T19:48:04.189 DEBUG:teuthology.orchestra.run.smithi099:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_1 /var/lib/ceph/osd/c1-0 2023-05-31T19:48:04.364 DEBUG:teuthology.orchestra.run.smithi099:> sudo /sbin/restorecon /var/lib/ceph/osd/c1-0 2023-05-31T19:48:04.401 DEBUG:teuthology.orchestra.run.smithi099:> sudo mkdir -p /var/lib/ceph/osd/c1-1 2023-05-31T19:48:04.468 INFO:tasks.ceph:roles_to_devs: {'c1.osd.0': '/dev/vg_nvme/lv_1', 'c1.osd.1': '/dev/vg_nvme/lv_2', 'c1.osd.2': '/dev/vg_nvme/lv_3'} 2023-05-31T19:48:04.469 INFO:tasks.ceph:role: c1.osd.1 2023-05-31T19:48:04.469 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_2 on ubuntu@smithi099.front.sepia.ceph.com 2023-05-31T19:48:04.469 DEBUG:teuthology.orchestra.run.smithi099:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_2 2023-05-31T19:48:04.542 INFO:teuthology.orchestra.run.smithi099.stdout:meta-data=/dev/vg_nvme/lv_2 isize=2048 agcount=4, agsize=5859072 blks 2023-05-31T19:48:04.542 INFO:teuthology.orchestra.run.smithi099.stdout: = sectsz=512 attr=2, projid32bit=1 2023-05-31T19:48:04.543 INFO:teuthology.orchestra.run.smithi099.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2023-05-31T19:48:04.543 INFO:teuthology.orchestra.run.smithi099.stdout: = reflink=1 bigtime=0 inobtcount=0 2023-05-31T19:48:04.543 INFO:teuthology.orchestra.run.smithi099.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2023-05-31T19:48:04.543 INFO:teuthology.orchestra.run.smithi099.stdout: = sunit=0 swidth=0 blks 2023-05-31T19:48:04.544 INFO:teuthology.orchestra.run.smithi099.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2023-05-31T19:48:04.544 INFO:teuthology.orchestra.run.smithi099.stdout:log =internal log bsize=4096 blocks=11443, version=2 2023-05-31T19:48:04.544 INFO:teuthology.orchestra.run.smithi099.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2023-05-31T19:48:04.544 INFO:teuthology.orchestra.run.smithi099.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2023-05-31T19:48:04.970 INFO:teuthology.orchestra.run.smithi099.stdout:Discarding blocks...Done. 2023-05-31T19:48:04.972 INFO:tasks.ceph:mount /dev/vg_nvme/lv_2 on ubuntu@smithi099.front.sepia.ceph.com -o noatime 2023-05-31T19:48:04.972 DEBUG:teuthology.orchestra.run.smithi099:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_2 /var/lib/ceph/osd/c1-1 2023-05-31T19:48:05.081 DEBUG:teuthology.orchestra.run.smithi099:> sudo /sbin/restorecon /var/lib/ceph/osd/c1-1 2023-05-31T19:48:05.113 DEBUG:teuthology.orchestra.run.smithi099:> sudo mkdir -p /var/lib/ceph/osd/c1-2 2023-05-31T19:48:05.180 INFO:tasks.ceph:roles_to_devs: {'c1.osd.0': '/dev/vg_nvme/lv_1', 'c1.osd.1': '/dev/vg_nvme/lv_2', 'c1.osd.2': '/dev/vg_nvme/lv_3'} 2023-05-31T19:48:05.180 INFO:tasks.ceph:role: c1.osd.2 2023-05-31T19:48:05.181 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_3 on ubuntu@smithi099.front.sepia.ceph.com 2023-05-31T19:48:05.181 DEBUG:teuthology.orchestra.run.smithi099:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_3 2023-05-31T19:48:05.254 INFO:teuthology.orchestra.run.smithi099.stdout:meta-data=/dev/vg_nvme/lv_3 isize=2048 agcount=4, agsize=5859072 blks 2023-05-31T19:48:05.255 INFO:teuthology.orchestra.run.smithi099.stdout: = sectsz=512 attr=2, projid32bit=1 2023-05-31T19:48:05.255 INFO:teuthology.orchestra.run.smithi099.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2023-05-31T19:48:05.255 INFO:teuthology.orchestra.run.smithi099.stdout: = reflink=1 bigtime=0 inobtcount=0 2023-05-31T19:48:05.255 INFO:teuthology.orchestra.run.smithi099.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2023-05-31T19:48:05.255 INFO:teuthology.orchestra.run.smithi099.stdout: = sunit=0 swidth=0 blks 2023-05-31T19:48:05.255 INFO:teuthology.orchestra.run.smithi099.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2023-05-31T19:48:05.256 INFO:teuthology.orchestra.run.smithi099.stdout:log =internal log bsize=4096 blocks=11443, version=2 2023-05-31T19:48:05.256 INFO:teuthology.orchestra.run.smithi099.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2023-05-31T19:48:05.256 INFO:teuthology.orchestra.run.smithi099.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2023-05-31T19:48:05.701 INFO:teuthology.orchestra.run.smithi099.stdout:Discarding blocks...Done. 2023-05-31T19:48:05.703 INFO:tasks.ceph:mount /dev/vg_nvme/lv_3 on ubuntu@smithi099.front.sepia.ceph.com -o noatime 2023-05-31T19:48:05.703 DEBUG:teuthology.orchestra.run.smithi099:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_3 /var/lib/ceph/osd/c1-2 2023-05-31T19:48:05.856 DEBUG:teuthology.orchestra.run.smithi099:> sudo /sbin/restorecon /var/lib/ceph/osd/c1-2 2023-05-31T19:48:05.892 DEBUG:teuthology.orchestra.run.smithi099:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster c1 --mkfs --mkkey -i 0 --monmap /home/ubuntu/cephtest/c1.monmap 2023-05-31T19:48:05.995 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T19:48:05.989+0000 7f288d44f640 -1 auth: error reading file: /var/lib/ceph/osd/c1-0/keyring: can't open /var/lib/ceph/osd/c1-0/keyring: (2) No such file or directory 2023-05-31T19:48:05.997 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T19:48:05.991+0000 7f288d44f640 -1 created new key in keyring /var/lib/ceph/osd/c1-0/keyring 2023-05-31T19:48:05.998 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T19:48:05.991+0000 7f288d44f640 -1 bluestore(/var/lib/ceph/osd/c1-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/c1-0/block: (2) No such file or directory 2023-05-31T19:48:05.998 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T19:48:05.991+0000 7f288d44f640 -1 bluestore(/var/lib/ceph/osd/c1-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/c1-0/block: (2) No such file or directory 2023-05-31T19:48:05.998 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T19:48:05.991+0000 7f288d44f640 -1 bluestore(/var/lib/ceph/osd/c1-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/c1-0/block: (2) No such file or directory 2023-05-31T19:48:06.002 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T19:48:05.996+0000 7f288d44f640 -1 bluestore(/var/lib/ceph/osd/c1-0) _read_fsid unparsable uuid 2023-05-31T19:48:08.085 DEBUG:teuthology.orchestra.run.smithi099:> sudo chown -R ceph:ceph /var/lib/ceph/osd/c1-0 2023-05-31T19:48:08.121 DEBUG:teuthology.orchestra.run.smithi099:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster c1 --mkfs --mkkey -i 1 --monmap /home/ubuntu/cephtest/c1.monmap 2023-05-31T19:48:08.215 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T19:48:08.209+0000 7f6be73cf640 -1 auth: error reading file: /var/lib/ceph/osd/c1-1/keyring: can't open /var/lib/ceph/osd/c1-1/keyring: (2) No such file or directory 2023-05-31T19:48:08.216 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T19:48:08.209+0000 7f6be73cf640 -1 created new key in keyring /var/lib/ceph/osd/c1-1/keyring 2023-05-31T19:48:08.216 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T19:48:08.209+0000 7f6be73cf640 -1 bluestore(/var/lib/ceph/osd/c1-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/c1-1/block: (2) No such file or directory 2023-05-31T19:48:08.218 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T19:48:08.209+0000 7f6be73cf640 -1 bluestore(/var/lib/ceph/osd/c1-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/c1-1/block: (2) No such file or directory 2023-05-31T19:48:08.218 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T19:48:08.209+0000 7f6be73cf640 -1 bluestore(/var/lib/ceph/osd/c1-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/c1-1/block: (2) No such file or directory 2023-05-31T19:48:08.218 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T19:48:08.210+0000 7f6be73cf640 -1 bluestore(/var/lib/ceph/osd/c1-1) _read_fsid unparsable uuid 2023-05-31T19:48:10.294 DEBUG:teuthology.orchestra.run.smithi099:> sudo chown -R ceph:ceph /var/lib/ceph/osd/c1-1 2023-05-31T19:48:10.333 DEBUG:teuthology.orchestra.run.smithi099:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster c1 --mkfs --mkkey -i 2 --monmap /home/ubuntu/cephtest/c1.monmap 2023-05-31T19:48:10.426 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T19:48:10.419+0000 7f972d501640 -1 auth: error reading file: /var/lib/ceph/osd/c1-2/keyring: can't open /var/lib/ceph/osd/c1-2/keyring: (2) No such file or directory 2023-05-31T19:48:10.427 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T19:48:10.420+0000 7f972d501640 -1 created new key in keyring /var/lib/ceph/osd/c1-2/keyring 2023-05-31T19:48:10.427 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T19:48:10.420+0000 7f972d501640 -1 bluestore(/var/lib/ceph/osd/c1-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/c1-2/block: (2) No such file or directory 2023-05-31T19:48:10.427 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T19:48:10.420+0000 7f972d501640 -1 bluestore(/var/lib/ceph/osd/c1-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/c1-2/block: (2) No such file or directory 2023-05-31T19:48:10.427 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T19:48:10.420+0000 7f972d501640 -1 bluestore(/var/lib/ceph/osd/c1-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/c1-2/block: (2) No such file or directory 2023-05-31T19:48:10.427 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T19:48:10.420+0000 7f972d501640 -1 bluestore(/var/lib/ceph/osd/c1-2) _read_fsid unparsable uuid 2023-05-31T19:48:12.505 DEBUG:teuthology.orchestra.run.smithi099:> sudo chown -R ceph:ceph /var/lib/ceph/osd/c1-2 2023-05-31T19:48:12.538 INFO:tasks.ceph:Reading keys from all nodes... 2023-05-31T19:48:12.538 DEBUG:teuthology.orchestra.run.smithi099:> set -ex 2023-05-31T19:48:12.539 DEBUG:teuthology.orchestra.run.smithi099:> sudo dd if=/var/lib/ceph/mgr/c1-x/keyring of=/dev/stdout 2023-05-31T19:48:12.609 DEBUG:teuthology.orchestra.run.smithi099:> set -ex 2023-05-31T19:48:12.609 DEBUG:teuthology.orchestra.run.smithi099:> sudo dd if=/var/lib/ceph/osd/c1-0/keyring of=/dev/stdout 2023-05-31T19:48:12.679 DEBUG:teuthology.orchestra.run.smithi099:> set -ex 2023-05-31T19:48:12.680 DEBUG:teuthology.orchestra.run.smithi099:> sudo dd if=/var/lib/ceph/osd/c1-1/keyring of=/dev/stdout 2023-05-31T19:48:12.749 DEBUG:teuthology.orchestra.run.smithi099:> set -ex 2023-05-31T19:48:12.750 DEBUG:teuthology.orchestra.run.smithi099:> sudo dd if=/var/lib/ceph/osd/c1-2/keyring of=/dev/stdout 2023-05-31T19:48:12.818 DEBUG:teuthology.orchestra.run.smithi099:> set -ex 2023-05-31T19:48:12.819 DEBUG:teuthology.orchestra.run.smithi099:> dd if=/etc/ceph/c1.client.0.keyring of=/dev/stdout 2023-05-31T19:48:12.877 DEBUG:teuthology.orchestra.run.smithi099:> set -ex 2023-05-31T19:48:12.877 DEBUG:teuthology.orchestra.run.smithi099:> dd if=/etc/ceph/c1.client.1.keyring of=/dev/stdout 2023-05-31T19:48:12.935 INFO:tasks.ceph:Adding keys to all mons... 2023-05-31T19:48:12.936 DEBUG:teuthology.orchestra.run.smithi099:> sudo tee -a /etc/ceph/c1.keyring 2023-05-31T19:48:13.002 INFO:teuthology.orchestra.run.smithi099.stdout:[mgr.x] 2023-05-31T19:48:13.002 INFO:teuthology.orchestra.run.smithi099.stdout: key = AQBzpHdk2V/tHxAAQwdKcppkZdj+a41EWOMrPg== 2023-05-31T19:48:13.002 INFO:teuthology.orchestra.run.smithi099.stdout:[osd.0] 2023-05-31T19:48:13.003 INFO:teuthology.orchestra.run.smithi099.stdout: key = AQB1pHdkQoIPOxAAz0jUEvk4bcSmyZtTNntwaw== 2023-05-31T19:48:13.003 INFO:teuthology.orchestra.run.smithi099.stdout:[osd.1] 2023-05-31T19:48:13.003 INFO:teuthology.orchestra.run.smithi099.stdout: key = AQB4pHdkPsOTDBAAC5i2AIJAjI6TEyQexiP9Ow== 2023-05-31T19:48:13.003 INFO:teuthology.orchestra.run.smithi099.stdout:[osd.2] 2023-05-31T19:48:13.003 INFO:teuthology.orchestra.run.smithi099.stdout: key = AQB6pHdkmvYdGRAAccD9xVgF81QkdRrSZGmvTw== 2023-05-31T19:48:13.003 INFO:teuthology.orchestra.run.smithi099.stdout:[client.0] 2023-05-31T19:48:13.003 INFO:teuthology.orchestra.run.smithi099.stdout: key = AQBzpHdkPba0IhAAesZpROFmTyzhrBALS4UoCw== 2023-05-31T19:48:13.004 INFO:teuthology.orchestra.run.smithi099.stdout:[client.1] 2023-05-31T19:48:13.004 INFO:teuthology.orchestra.run.smithi099.stdout: key = AQBzpHdkWU0HJhAA9TFDYZvmXSpF+J7c/+qodg== 2023-05-31T19:48:13.005 DEBUG:teuthology.orchestra.run.smithi099:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/c1.keyring --name=mgr.x --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2023-05-31T19:48:13.100 DEBUG:teuthology.orchestra.run.smithi099:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/c1.keyring --name=osd.0 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-05-31T19:48:13.154 DEBUG:teuthology.orchestra.run.smithi099:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/c1.keyring --name=osd.1 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-05-31T19:48:13.208 DEBUG:teuthology.orchestra.run.smithi099:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/c1.keyring --name=osd.2 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-05-31T19:48:13.263 DEBUG:teuthology.orchestra.run.smithi099:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/c1.keyring --name=client.0 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2023-05-31T19:48:13.319 DEBUG:teuthology.orchestra.run.smithi099:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/c1.keyring --name=client.1 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2023-05-31T19:48:13.376 INFO:tasks.ceph:Running mkfs on mon nodes... 2023-05-31T19:48:13.378 DEBUG:teuthology.orchestra.run.smithi099:> sudo mkdir -p /var/lib/ceph/mon/c1-a 2023-05-31T19:48:13.406 DEBUG:teuthology.orchestra.run.smithi099:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster c1 --mkfs -i a --monmap /home/ubuntu/cephtest/c1.monmap --keyring /etc/ceph/c1.keyring 2023-05-31T19:48:13.505 DEBUG:teuthology.orchestra.run.smithi099:> sudo chown -R ceph:ceph /var/lib/ceph/mon/c1-a 2023-05-31T19:48:13.537 DEBUG:teuthology.orchestra.run.smithi099:> rm -- /home/ubuntu/cephtest/c1.monmap 2023-05-31T19:48:13.595 INFO:tasks.ceph:Starting mon daemons in cluster c1... 2023-05-31T19:48:13.596 INFO:tasks.ceph.c1.mon.a:Restarting daemon 2023-05-31T19:48:13.597 DEBUG:teuthology.orchestra.run.smithi099:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster c1 -i a 2023-05-31T19:48:13.638 INFO:tasks.ceph.c1.mon.a:Started 2023-05-31T19:48:13.638 INFO:tasks.ceph:Starting mgr daemons in cluster c1... 2023-05-31T19:48:13.639 INFO:tasks.ceph.c1.mgr.x:Restarting daemon 2023-05-31T19:48:13.639 DEBUG:teuthology.orchestra.run.smithi099:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mgr -f --cluster c1 -i x 2023-05-31T19:48:13.641 INFO:tasks.ceph.c1.mgr.x:Started 2023-05-31T19:48:13.641 INFO:tasks.ceph:Setting crush tunables to default 2023-05-31T19:48:13.641 DEBUG:teuthology.orchestra.run.smithi099:> sudo ceph --cluster c1 osd crush tunables default 2023-05-31T19:48:13.852 INFO:teuthology.orchestra.run.smithi099.stderr:adjusted tunables profile to default 2023-05-31T19:48:13.863 INFO:tasks.ceph:check_enable_crimson: False 2023-05-31T19:48:13.864 INFO:tasks.ceph:Starting osd daemons in cluster c1... 2023-05-31T19:48:13.864 DEBUG:teuthology.orchestra.run.smithi099:> set -ex 2023-05-31T19:48:13.865 DEBUG:teuthology.orchestra.run.smithi099:> sudo dd if=/var/lib/ceph/osd/c1-0/fsid of=/dev/stdout 2023-05-31T19:48:13.892 DEBUG:teuthology.orchestra.run.smithi099:> set -ex 2023-05-31T19:48:13.892 DEBUG:teuthology.orchestra.run.smithi099:> sudo dd if=/var/lib/ceph/osd/c1-1/fsid of=/dev/stdout 2023-05-31T19:48:13.960 DEBUG:teuthology.orchestra.run.smithi099:> set -ex 2023-05-31T19:48:13.961 DEBUG:teuthology.orchestra.run.smithi099:> sudo dd if=/var/lib/ceph/osd/c1-2/fsid of=/dev/stdout 2023-05-31T19:48:14.030 DEBUG:teuthology.orchestra.run.smithi099:> sudo ceph --cluster c1 osd new dd28a15c-0353-4391-9f6d-89f552ed805a 0 2023-05-31T19:48:14.045 INFO:tasks.ceph.c1.mgr.x.smithi099.stderr:2023-05-31T19:48:14.039+0000 7fc189534200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2023-05-31T19:48:14.226 INFO:tasks.ceph.c1.mgr.x.smithi099.stderr:2023-05-31T19:48:14.220+0000 7fc189534200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2023-05-31T19:48:14.289 INFO:teuthology.orchestra.run.smithi099.stdout:0 2023-05-31T19:48:14.302 DEBUG:teuthology.orchestra.run.smithi099:> sudo ceph --cluster c1 osd new d5355dba-8098-40a3-a773-5e2fea80d126 1 2023-05-31T19:48:14.514 INFO:teuthology.orchestra.run.smithi099.stdout:1 2023-05-31T19:48:14.527 DEBUG:teuthology.orchestra.run.smithi099:> sudo ceph --cluster c1 osd new cf901f24-2d75-4638-9694-d6ec95c56541 2 2023-05-31T19:48:14.574 INFO:tasks.ceph.c1.mgr.x.smithi099.stderr:2023-05-31T19:48:14.568+0000 7fc189534200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2023-05-31T19:48:14.742 INFO:teuthology.orchestra.run.smithi099.stdout:2 2023-05-31T19:48:14.754 INFO:tasks.ceph.c1.osd.0:Restarting daemon 2023-05-31T19:48:14.755 DEBUG:teuthology.orchestra.run.smithi099:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster c1 -i 0 2023-05-31T19:48:14.757 INFO:tasks.ceph.c1.osd.0:Started 2023-05-31T19:48:14.758 INFO:tasks.ceph.c1.osd.1:Restarting daemon 2023-05-31T19:48:14.758 DEBUG:teuthology.orchestra.run.smithi099:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster c1 -i 1 2023-05-31T19:48:14.760 INFO:tasks.ceph.c1.osd.1:Started 2023-05-31T19:48:14.761 INFO:tasks.ceph.c1.osd.2:Restarting daemon 2023-05-31T19:48:14.761 DEBUG:teuthology.orchestra.run.smithi099:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster c1 -i 2 2023-05-31T19:48:14.763 INFO:tasks.ceph.c1.osd.2:Started 2023-05-31T19:48:14.763 DEBUG:teuthology.orchestra.run.smithi099:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c1 osd dump --format=json 2023-05-31T19:48:14.981 INFO:teuthology.orchestra.run.smithi099.stdout: 2023-05-31T19:48:14.982 INFO:teuthology.orchestra.run.smithi099.stdout:{"epoch":5,"fsid":"7c5f80fb-3341-4f1e-be16-38b17ec1b3d0","created":"2023-05-31T19:48:13.760704+0000","modified":"2023-05-31T19:48:14.735974+0000","last_up_change":"0.000000","last_in_change":"2023-05-31T19:48:14.735974+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"dd28a15c-0353-4391-9f6d-89f552ed805a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"d5355dba-8098-40a3-a773-5e2fea80d126","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"cf901f24-2d75-4638-9694-d6ec95c56541","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-05-31T19:48:14.997 INFO:tasks.ceph.ceph_manager.c1:[] 2023-05-31T19:48:14.998 INFO:tasks.ceph:Waiting for OSDs to come up 2023-05-31T19:48:15.105 INFO:tasks.ceph.c1.mgr.x.smithi099.stderr:2023-05-31T19:48:15.099+0000 7fc189534200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2023-05-31T19:48:15.299 INFO:tasks.ceph.c1.mgr.x.smithi099.stderr:2023-05-31T19:48:15.293+0000 7fc189534200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2023-05-31T19:48:15.301 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster c1 osd dump --format=json 2023-05-31T19:48:15.371 INFO:tasks.ceph.c1.osd.2.smithi099.stderr:2023-05-31T19:48:15.365+0000 7f143be61640 -1 Falling back to public interface 2023-05-31T19:48:15.378 INFO:tasks.ceph.c1.osd.1.smithi099.stderr:2023-05-31T19:48:15.372+0000 7fa447983640 -1 Falling back to public interface 2023-05-31T19:48:15.379 INFO:tasks.ceph.c1.osd.0.smithi099.stderr:2023-05-31T19:48:15.372+0000 7fb04312a640 -1 Falling back to public interface 2023-05-31T19:48:15.497 INFO:teuthology.misc.health.smithi099.stdout: 2023-05-31T19:48:15.497 INFO:teuthology.misc.health.smithi099.stdout:{"epoch":5,"fsid":"7c5f80fb-3341-4f1e-be16-38b17ec1b3d0","created":"2023-05-31T19:48:13.760704+0000","modified":"2023-05-31T19:48:14.735974+0000","last_up_change":"0.000000","last_in_change":"2023-05-31T19:48:14.735974+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"dd28a15c-0353-4391-9f6d-89f552ed805a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"d5355dba-8098-40a3-a773-5e2fea80d126","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"cf901f24-2d75-4638-9694-d6ec95c56541","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-05-31T19:48:15.510 DEBUG:teuthology.misc:0 of 3 OSDs are up 2023-05-31T19:48:15.685 INFO:tasks.ceph.c1.mgr.x.smithi099.stderr:2023-05-31T19:48:15.678+0000 7fc189534200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2023-05-31T19:48:16.343 INFO:tasks.ceph.c1.mgr.x.smithi099.stderr:2023-05-31T19:48:16.332+0000 7fc189534200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2023-05-31T19:48:16.410 INFO:tasks.ceph.c1.mgr.x.smithi099.stderr:2023-05-31T19:48:16.404+0000 7fc189534200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2023-05-31T19:48:16.544 INFO:tasks.ceph.c1.mgr.x.smithi099.stderr:2023-05-31T19:48:16.538+0000 7fc189534200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2023-05-31T19:48:16.614 INFO:tasks.ceph.c1.mgr.x.smithi099.stderr:2023-05-31T19:48:16.608+0000 7fc189534200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2023-05-31T19:48:16.793 INFO:tasks.ceph.c1.mgr.x.smithi099.stderr:2023-05-31T19:48:16.787+0000 7fc189534200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2023-05-31T19:48:16.871 INFO:tasks.ceph.c1.mgr.x.smithi099.stderr:2023-05-31T19:48:16.865+0000 7fc189534200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2023-05-31T19:48:16.948 INFO:tasks.ceph.c1.mgr.x.smithi099.stderr:2023-05-31T19:48:16.941+0000 7fc189534200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2023-05-31T19:48:17.040 INFO:tasks.ceph.c1.mgr.x.smithi099.stderr:2023-05-31T19:48:17.034+0000 7fc189534200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2023-05-31T19:48:17.219 INFO:tasks.ceph.c1.mgr.x.smithi099.stderr:2023-05-31T19:48:17.213+0000 7fc189534200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2023-05-31T19:48:17.305 INFO:tasks.ceph.c1.mgr.x.smithi099.stderr:2023-05-31T19:48:17.295+0000 7fc189534200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2023-05-31T19:48:17.590 INFO:tasks.ceph.c1.mgr.x.smithi099.stderr:2023-05-31T19:48:17.583+0000 7fc189534200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2023-05-31T19:48:17.660 INFO:tasks.ceph.c1.mgr.x.smithi099.stderr:2023-05-31T19:48:17.654+0000 7fc189534200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2023-05-31T19:48:18.096 INFO:tasks.ceph.c1.osd.1.smithi099.stderr:2023-05-31T19:48:18.090+0000 7fa447983640 -1 osd.1 0 log_to_monitors true 2023-05-31T19:48:18.123 INFO:tasks.ceph.c1.osd.0.smithi099.stderr:2023-05-31T19:48:18.117+0000 7fb04312a640 -1 osd.0 0 log_to_monitors true 2023-05-31T19:48:18.123 INFO:tasks.ceph.c1.osd.2.smithi099.stderr:2023-05-31T19:48:18.117+0000 7f143be61640 -1 osd.2 0 log_to_monitors true 2023-05-31T19:48:18.186 INFO:tasks.ceph.c1.mgr.x.smithi099.stderr:2023-05-31T19:48:18.181+0000 7fc189534200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2023-05-31T19:48:18.499 INFO:tasks.ceph.c1.mgr.x.smithi099.stderr:2023-05-31T19:48:18.494+0000 7fc189534200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2023-05-31T19:48:18.642 INFO:tasks.ceph.c1.mgr.x.smithi099.stderr:2023-05-31T19:48:18.637+0000 7fc189534200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2023-05-31T19:48:18.996 INFO:tasks.ceph.c1.mgr.x.smithi099.stderr:2023-05-31T19:48:18.991+0000 7fc189534200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2023-05-31T19:48:19.142 INFO:tasks.ceph.c1.mgr.x.smithi099.stderr:2023-05-31T19:48:19.137+0000 7fc189534200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2023-05-31T19:48:19.316 INFO:tasks.ceph.c1.mgr.x.smithi099.stderr:2023-05-31T19:48:19.311+0000 7fc189534200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2023-05-31T19:48:19.655 INFO:tasks.ceph.c1.mgr.x.smithi099.stderr:2023-05-31T19:48:19.650+0000 7fc189534200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2023-05-31T19:48:20.382 INFO:tasks.ceph.c1.osd.1.smithi099.stderr:2023-05-31T19:48:20.376+0000 7fa43bd24700 -1 osd.1 0 waiting for initial osdmap 2023-05-31T19:48:20.386 INFO:tasks.ceph.c1.osd.2.smithi099.stderr:2023-05-31T19:48:20.381+0000 7f1430202700 -1 osd.2 0 waiting for initial osdmap 2023-05-31T19:48:20.388 INFO:tasks.ceph.c1.osd.0.smithi099.stderr:2023-05-31T19:48:20.382+0000 7fb037ccc700 -1 osd.0 0 waiting for initial osdmap 2023-05-31T19:48:20.392 INFO:tasks.ceph.c1.osd.1.smithi099.stderr:2023-05-31T19:48:20.387+0000 7fa4338fe700 -1 osd.1 7 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2023-05-31T19:48:20.402 INFO:tasks.ceph.c1.osd.0.smithi099.stderr:2023-05-31T19:48:20.397+0000 7fb02f0a5700 -1 osd.0 7 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2023-05-31T19:48:20.411 INFO:tasks.ceph.c1.osd.2.smithi099.stderr:2023-05-31T19:48:20.406+0000 7f1427ddc700 -1 osd.2 7 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2023-05-31T19:48:20.719 INFO:tasks.ceph.c1.mgr.x.smithi099.stderr:2023-05-31T19:48:20.714+0000 7fc14a950700 -1 mgr.server handle_report got status from non-daemon mon.a 2023-05-31T19:48:21.814 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster c1 osd dump --format=json 2023-05-31T19:48:22.134 INFO:teuthology.misc.health.smithi099.stdout: 2023-05-31T19:48:22.135 INFO:teuthology.misc.health.smithi099.stdout:{"epoch":8,"fsid":"7c5f80fb-3341-4f1e-be16-38b17ec1b3d0","created":"2023-05-31T19:48:13.760704+0000","modified":"2023-05-31T19:48:21.385600+0000","last_up_change":"2023-05-31T19:48:21.385600+0000","last_in_change":"2023-05-31T19:48:14.735974+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"dd28a15c-0353-4391-9f6d-89f552ed805a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6809","nonce":104675},{"type":"v1","addr":"172.21.15.99:6811","nonce":104675}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6813","nonce":104675},{"type":"v1","addr":"172.21.15.99:6815","nonce":104675}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6822","nonce":104675},{"type":"v1","addr":"172.21.15.99:6823","nonce":104675}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6818","nonce":104675},{"type":"v1","addr":"172.21.15.99:6820","nonce":104675}]},"public_addr":"172.21.15.99:6811/104675","cluster_addr":"172.21.15.99:6815/104675","heartbeat_back_addr":"172.21.15.99:6823/104675","heartbeat_front_addr":"172.21.15.99:6820/104675","state":["exists","up"]},{"osd":1,"uuid":"d5355dba-8098-40a3-a773-5e2fea80d126","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6808","nonce":104674},{"type":"v1","addr":"172.21.15.99:6810","nonce":104674}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6812","nonce":104674},{"type":"v1","addr":"172.21.15.99:6814","nonce":104674}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6819","nonce":104674},{"type":"v1","addr":"172.21.15.99:6821","nonce":104674}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6816","nonce":104674},{"type":"v1","addr":"172.21.15.99:6817","nonce":104674}]},"public_addr":"172.21.15.99:6810/104674","cluster_addr":"172.21.15.99:6814/104674","heartbeat_back_addr":"172.21.15.99:6821/104674","heartbeat_front_addr":"172.21.15.99:6817/104674","state":["exists","up"]},{"osd":2,"uuid":"cf901f24-2d75-4638-9694-d6ec95c56541","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6800","nonce":104673},{"type":"v1","addr":"172.21.15.99:6801","nonce":104673}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6802","nonce":104673},{"type":"v1","addr":"172.21.15.99:6803","nonce":104673}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6806","nonce":104673},{"type":"v1","addr":"172.21.15.99:6807","nonce":104673}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6804","nonce":104673},{"type":"v1","addr":"172.21.15.99:6805","nonce":104673}]},"public_addr":"172.21.15.99:6801/104673","cluster_addr":"172.21.15.99:6803/104673","heartbeat_back_addr":"172.21.15.99:6807/104673","heartbeat_front_addr":"172.21.15.99:6805/104673","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-05-31T19:48:22.147 DEBUG:teuthology.misc:3 of 3 OSDs are up 2023-05-31T19:48:22.147 INFO:tasks.ceph:Creating RBD pool 2023-05-31T19:48:22.147 DEBUG:teuthology.orchestra.run.smithi099:> sudo ceph --cluster c1 osd pool create rbd 8 2023-05-31T19:48:22.729 INFO:teuthology.orchestra.run.smithi099.stderr:pool 'rbd' created 2023-05-31T19:48:22.741 DEBUG:teuthology.orchestra.run.smithi099:> sudo ceph --cluster c1 osd pool application enable rbd rbd --yes-i-really-mean-it 2023-05-31T19:48:23.731 INFO:teuthology.orchestra.run.smithi099.stderr:enabled application 'rbd' on pool 'rbd' 2023-05-31T19:48:23.745 INFO:tasks.ceph:Starting mds daemons in cluster c1... 2023-05-31T19:48:23.746 INFO:tasks.ceph:Waiting until c1 daemons up and pgs clean... 2023-05-31T19:48:23.746 INFO:tasks.ceph.ceph_manager.c1:waiting for mgr available 2023-05-31T19:48:23.746 DEBUG:teuthology.orchestra.run.smithi099:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c1 mgr dump --format=json 2023-05-31T19:48:23.747 INFO:tasks.daemonwatchdog.daemon_watchdog:watchdog starting 2023-05-31T19:48:24.169 INFO:teuthology.orchestra.run.smithi099.stdout: 2023-05-31T19:48:24.184 INFO:teuthology.orchestra.run.smithi099.stdout:{"epoch":4,"active_gid":4101,"active_name":"x","active_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6824","nonce":104261},{"type":"v1","addr":"172.21.15.99:6825","nonce":104261}]},"active_addr":"172.21.15.99:6825/104261","active_change":"2023-05-31T19:48:19.663232+0000","active_mgr_features":4540138322906710015,"available":true,"standbys":[],"modules":["iostat","nfs","restful"],"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","upmap"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"alertmanager_web_password":{"name":"alertmanager_web_password","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"Alertmanager web password","long_desc":"","tags":[],"see_also":[]},"alertmanager_web_user":{"name":"alertmanager_web_user","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"Alertmanager web user","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/ceph-grafana:9.4.7","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"secs","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"prometheus_web_password":{"name":"prometheus_web_password","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"Prometheus web password","long_desc":"","tags":[],"see_also":[]},"prometheus_web_user":{"name":"prometheus_web_user","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"Prometheus web user","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"noautoscale":{"name":"noautoscale","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"global autoscale flag","long_desc":"Option to turn on/off the autoscaler for all pools","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"drive_group_interval":{"name":"drive_group_interval","type":"float","level":"advanced","flags":0,"default_value":"300.0","min":"","max":"","enum_allowed":[],"desc":"interval in seconds between re-application of applied drive_groups","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}],"services":{},"always_on_modules":{"octopus":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"pacific":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"quincy":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"reef":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"]},"last_failure_osd_epoch":0,"active_clients":[{"name":"libcephsqlite","addrvec":[{"type":"v2","addr":"172.21.15.99:0","nonce":1419483827}]},{"name":"rbd_support","addrvec":[{"type":"v2","addr":"172.21.15.99:0","nonce":4125722521}]},{"name":"volumes","addrvec":[{"type":"v2","addr":"172.21.15.99:0","nonce":3406182677}]}]} 2023-05-31T19:48:24.187 INFO:tasks.ceph.ceph_manager.c1:mgr available! 2023-05-31T19:48:24.187 INFO:tasks.ceph.ceph_manager.c1:waiting for all up 2023-05-31T19:48:24.187 DEBUG:teuthology.orchestra.run.smithi099:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c1 osd dump --format=json 2023-05-31T19:48:24.507 INFO:teuthology.orchestra.run.smithi099.stdout: 2023-05-31T19:48:24.508 INFO:teuthology.orchestra.run.smithi099.stdout:{"epoch":10,"fsid":"7c5f80fb-3341-4f1e-be16-38b17ec1b3d0","created":"2023-05-31T19:48:13.760704+0000","modified":"2023-05-31T19:48:23.716935+0000","last_up_change":"2023-05-31T19:48:21.385600+0000","last_in_change":"2023-05-31T19:48:14.735974+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-05-31T19:48:21.752614+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"10","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":2.9900000095367432,"score_stable":2.9900000095367432,"optimal_score":0.67000001668930054,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2023-05-31T19:48:22.476145+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"10","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"dd28a15c-0353-4391-9f6d-89f552ed805a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":9,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6809","nonce":104675},{"type":"v1","addr":"172.21.15.99:6811","nonce":104675}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6813","nonce":104675},{"type":"v1","addr":"172.21.15.99:6815","nonce":104675}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6822","nonce":104675},{"type":"v1","addr":"172.21.15.99:6823","nonce":104675}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6818","nonce":104675},{"type":"v1","addr":"172.21.15.99:6820","nonce":104675}]},"public_addr":"172.21.15.99:6811/104675","cluster_addr":"172.21.15.99:6815/104675","heartbeat_back_addr":"172.21.15.99:6823/104675","heartbeat_front_addr":"172.21.15.99:6820/104675","state":["exists","up"]},{"osd":1,"uuid":"d5355dba-8098-40a3-a773-5e2fea80d126","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":9,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6808","nonce":104674},{"type":"v1","addr":"172.21.15.99:6810","nonce":104674}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6812","nonce":104674},{"type":"v1","addr":"172.21.15.99:6814","nonce":104674}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6819","nonce":104674},{"type":"v1","addr":"172.21.15.99:6821","nonce":104674}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6816","nonce":104674},{"type":"v1","addr":"172.21.15.99:6817","nonce":104674}]},"public_addr":"172.21.15.99:6810/104674","cluster_addr":"172.21.15.99:6814/104674","heartbeat_back_addr":"172.21.15.99:6821/104674","heartbeat_front_addr":"172.21.15.99:6817/104674","state":["exists","up"]},{"osd":2,"uuid":"cf901f24-2d75-4638-9694-d6ec95c56541","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":9,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6800","nonce":104673},{"type":"v1","addr":"172.21.15.99:6801","nonce":104673}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6802","nonce":104673},{"type":"v1","addr":"172.21.15.99:6803","nonce":104673}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6806","nonce":104673},{"type":"v1","addr":"172.21.15.99:6807","nonce":104673}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6804","nonce":104673},{"type":"v1","addr":"172.21.15.99:6805","nonce":104673}]},"public_addr":"172.21.15.99:6801/104673","cluster_addr":"172.21.15.99:6803/104673","heartbeat_back_addr":"172.21.15.99:6807/104673","heartbeat_front_addr":"172.21.15.99:6805/104673","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-05-31T19:48:19.073918+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-05-31T19:48:19.138136+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-05-31T19:48:19.090488+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-05-31T19:48:24.520 INFO:tasks.ceph.ceph_manager.c1:all up! 2023-05-31T19:48:24.520 DEBUG:teuthology.orchestra.run.smithi099:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c1 osd dump --format=json 2023-05-31T19:48:24.837 INFO:teuthology.orchestra.run.smithi099.stdout: 2023-05-31T19:48:24.837 INFO:teuthology.orchestra.run.smithi099.stdout:{"epoch":11,"fsid":"7c5f80fb-3341-4f1e-be16-38b17ec1b3d0","created":"2023-05-31T19:48:13.760704+0000","modified":"2023-05-31T19:48:24.722836+0000","last_up_change":"2023-05-31T19:48:21.385600+0000","last_in_change":"2023-05-31T19:48:14.735974+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-05-31T19:48:21.752614+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"11","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":2.9900000095367432,"score_stable":2.9900000095367432,"optimal_score":0.67000001668930054,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2023-05-31T19:48:22.476145+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"11","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"dd28a15c-0353-4391-9f6d-89f552ed805a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":9,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6809","nonce":104675},{"type":"v1","addr":"172.21.15.99:6811","nonce":104675}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6813","nonce":104675},{"type":"v1","addr":"172.21.15.99:6815","nonce":104675}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6822","nonce":104675},{"type":"v1","addr":"172.21.15.99:6823","nonce":104675}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6818","nonce":104675},{"type":"v1","addr":"172.21.15.99:6820","nonce":104675}]},"public_addr":"172.21.15.99:6811/104675","cluster_addr":"172.21.15.99:6815/104675","heartbeat_back_addr":"172.21.15.99:6823/104675","heartbeat_front_addr":"172.21.15.99:6820/104675","state":["exists","up"]},{"osd":1,"uuid":"d5355dba-8098-40a3-a773-5e2fea80d126","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":9,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6808","nonce":104674},{"type":"v1","addr":"172.21.15.99:6810","nonce":104674}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6812","nonce":104674},{"type":"v1","addr":"172.21.15.99:6814","nonce":104674}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6819","nonce":104674},{"type":"v1","addr":"172.21.15.99:6821","nonce":104674}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6816","nonce":104674},{"type":"v1","addr":"172.21.15.99:6817","nonce":104674}]},"public_addr":"172.21.15.99:6810/104674","cluster_addr":"172.21.15.99:6814/104674","heartbeat_back_addr":"172.21.15.99:6821/104674","heartbeat_front_addr":"172.21.15.99:6817/104674","state":["exists","up"]},{"osd":2,"uuid":"cf901f24-2d75-4638-9694-d6ec95c56541","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":9,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6800","nonce":104673},{"type":"v1","addr":"172.21.15.99:6801","nonce":104673}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6802","nonce":104673},{"type":"v1","addr":"172.21.15.99:6803","nonce":104673}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6806","nonce":104673},{"type":"v1","addr":"172.21.15.99:6807","nonce":104673}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6804","nonce":104673},{"type":"v1","addr":"172.21.15.99:6805","nonce":104673}]},"public_addr":"172.21.15.99:6801/104673","cluster_addr":"172.21.15.99:6803/104673","heartbeat_back_addr":"172.21.15.99:6807/104673","heartbeat_front_addr":"172.21.15.99:6805/104673","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-05-31T19:48:19.073918+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-05-31T19:48:19.138136+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-05-31T19:48:19.090488+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-05-31T19:48:24.851 DEBUG:teuthology.orchestra.run.smithi099:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c1 tell osd.0 flush_pg_stats 2023-05-31T19:48:24.852 DEBUG:teuthology.orchestra.run.smithi099:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c1 tell osd.1 flush_pg_stats 2023-05-31T19:48:24.852 DEBUG:teuthology.orchestra.run.smithi099:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c1 tell osd.2 flush_pg_stats 2023-05-31T19:48:25.022 INFO:teuthology.orchestra.run.smithi099.stdout:34359738370 2023-05-31T19:48:25.022 DEBUG:teuthology.orchestra.run.smithi099:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c1 osd last-stat-seq osd.0 2023-05-31T19:48:25.046 INFO:teuthology.orchestra.run.smithi099.stdout:34359738370 2023-05-31T19:48:25.047 DEBUG:teuthology.orchestra.run.smithi099:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c1 osd last-stat-seq osd.1 2023-05-31T19:48:25.047 INFO:teuthology.orchestra.run.smithi099.stdout:34359738370 2023-05-31T19:48:25.048 DEBUG:teuthology.orchestra.run.smithi099:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c1 osd last-stat-seq osd.2 2023-05-31T19:48:25.380 INFO:teuthology.orchestra.run.smithi099.stdout:34359738369 2023-05-31T19:48:25.394 INFO:tasks.ceph.ceph_manager.c1:need seq 34359738370 got 34359738369 for osd.0 2023-05-31T19:48:25.426 INFO:teuthology.orchestra.run.smithi099.stdout:34359738369 2023-05-31T19:48:25.436 INFO:teuthology.orchestra.run.smithi099.stdout:34359738369 2023-05-31T19:48:25.441 INFO:tasks.ceph.ceph_manager.c1:need seq 34359738370 got 34359738369 for osd.2 2023-05-31T19:48:25.450 INFO:tasks.ceph.ceph_manager.c1:need seq 34359738370 got 34359738369 for osd.1 2023-05-31T19:48:26.395 DEBUG:teuthology.orchestra.run.smithi099:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c1 osd last-stat-seq osd.0 2023-05-31T19:48:26.442 DEBUG:teuthology.orchestra.run.smithi099:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c1 osd last-stat-seq osd.2 2023-05-31T19:48:26.451 DEBUG:teuthology.orchestra.run.smithi099:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c1 osd last-stat-seq osd.1 2023-05-31T19:48:26.747 INFO:teuthology.orchestra.run.smithi099.stdout:34359738370 2023-05-31T19:48:26.761 INFO:tasks.ceph.ceph_manager.c1:need seq 34359738370 got 34359738370 for osd.0 2023-05-31T19:48:26.761 DEBUG:teuthology.parallel:result is None 2023-05-31T19:48:26.786 INFO:teuthology.orchestra.run.smithi099.stdout:34359738370 2023-05-31T19:48:26.799 INFO:tasks.ceph.ceph_manager.c1:need seq 34359738370 got 34359738370 for osd.2 2023-05-31T19:48:26.800 DEBUG:teuthology.parallel:result is None 2023-05-31T19:48:26.827 INFO:teuthology.orchestra.run.smithi099.stdout:34359738370 2023-05-31T19:48:26.841 INFO:tasks.ceph.ceph_manager.c1:need seq 34359738370 got 34359738370 for osd.1 2023-05-31T19:48:26.842 DEBUG:teuthology.parallel:result is None 2023-05-31T19:48:26.843 INFO:tasks.ceph.ceph_manager.c1:waiting for clean 2023-05-31T19:48:26.843 DEBUG:teuthology.orchestra.run.smithi099:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c1 pg dump --format=json 2023-05-31T19:48:27.193 INFO:teuthology.orchestra.run.smithi099.stdout: 2023-05-31T19:48:27.196 INFO:teuthology.orchestra.run.smithi099.stderr:dumped all 2023-05-31T19:48:27.206 INFO:teuthology.orchestra.run.smithi099.stdout:{"pg_ready":true,"pg_map":{"version":9,"stamp":"2023-05-31T19:48:25.711441+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":6,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":314572800,"kb_used":24944,"kb_used_data":1432,"kb_used_omap":0,"kb_used_meta":23488,"kb_avail":314547856,"statfs":{"total":322122547200,"available":322097004544,"internally_reserved":0,"allocated":1466368,"data_stored":1252324,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":24051712},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"0.000000"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":10,"reported_epoch":11,"state":"active+clean","last_fresh":"2023-05-31T19:48:24.724930+0000","last_change":"2023-05-31T19:48:23.728893+0000","last_active":"2023-05-31T19:48:24.724930+0000","last_peered":"2023-05-31T19:48:24.724930+0000","last_clean":"2023-05-31T19:48:24.724930+0000","last_became_active":"2023-05-31T19:48:23.728633+0000","last_became_peered":"2023-05-31T19:48:23.728633+0000","last_unstale":"2023-05-31T19:48:24.724930+0000","last_undegraded":"2023-05-31T19:48:24.724930+0000","last_fullsized":"2023-05-31T19:48:24.724930+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-31T19:48:22.713746+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-31T19:48:22.713746+0000","last_clean_scrub_stamp":"2023-05-31T19:48:22.713746+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-01T21:00:43.468297+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":10,"reported_epoch":11,"state":"active+clean","last_fresh":"2023-05-31T19:48:24.724906+0000","last_change":"2023-05-31T19:48:23.728965+0000","last_active":"2023-05-31T19:48:24.724906+0000","last_peered":"2023-05-31T19:48:24.724906+0000","last_clean":"2023-05-31T19:48:24.724906+0000","last_became_active":"2023-05-31T19:48:23.728496+0000","last_became_peered":"2023-05-31T19:48:23.728496+0000","last_unstale":"2023-05-31T19:48:24.724906+0000","last_undegraded":"2023-05-31T19:48:24.724906+0000","last_fullsized":"2023-05-31T19:48:24.724906+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-31T19:48:22.713746+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-31T19:48:22.713746+0000","last_clean_scrub_stamp":"2023-05-31T19:48:22.713746+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-01T22:31:01.359974+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":10,"reported_epoch":11,"state":"active+clean","last_fresh":"2023-05-31T19:48:24.724888+0000","last_change":"2023-05-31T19:48:23.726504+0000","last_active":"2023-05-31T19:48:24.724888+0000","last_peered":"2023-05-31T19:48:24.724888+0000","last_clean":"2023-05-31T19:48:24.724888+0000","last_became_active":"2023-05-31T19:48:23.726227+0000","last_became_peered":"2023-05-31T19:48:23.726227+0000","last_unstale":"2023-05-31T19:48:24.724888+0000","last_undegraded":"2023-05-31T19:48:24.724888+0000","last_fullsized":"2023-05-31T19:48:24.724888+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-31T19:48:22.713746+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-31T19:48:22.713746+0000","last_clean_scrub_stamp":"2023-05-31T19:48:22.713746+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-02T02:09:21.030368+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":9,"reported_epoch":10,"state":"active+clean","last_fresh":"2023-05-31T19:48:23.730770+0000","last_change":"2023-05-31T19:48:23.725911+0000","last_active":"2023-05-31T19:48:23.730770+0000","last_peered":"2023-05-31T19:48:23.730770+0000","last_clean":"2023-05-31T19:48:23.730770+0000","last_became_active":"2023-05-31T19:48:23.725649+0000","last_became_peered":"2023-05-31T19:48:23.725649+0000","last_unstale":"2023-05-31T19:48:23.730770+0000","last_undegraded":"2023-05-31T19:48:23.730770+0000","last_fullsized":"2023-05-31T19:48:23.730770+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-31T19:48:22.713746+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-31T19:48:22.713746+0000","last_clean_scrub_stamp":"2023-05-31T19:48:22.713746+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-01T21:41:24.762841+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"10'97","reported_seq":137,"reported_epoch":11,"state":"active+clean","last_fresh":"2023-05-31T19:48:24.724937+0000","last_change":"2023-05-31T19:48:23.728848+0000","last_active":"2023-05-31T19:48:24.724937+0000","last_peered":"2023-05-31T19:48:24.724937+0000","last_clean":"2023-05-31T19:48:24.724937+0000","last_became_active":"2023-05-31T19:48:23.728579+0000","last_became_peered":"2023-05-31T19:48:23.728579+0000","last_unstale":"2023-05-31T19:48:24.724937+0000","last_undegraded":"2023-05-31T19:48:24.724937+0000","last_fullsized":"2023-05-31T19:48:24.724937+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-31T19:48:22.713746+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-31T19:48:22.713746+0000","last_clean_scrub_stamp":"2023-05-31T19:48:22.713746+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-01T22:58:09.119583+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":10,"reported_epoch":11,"state":"active+clean","last_fresh":"2023-05-31T19:48:24.724843+0000","last_change":"2023-05-31T19:48:23.726215+0000","last_active":"2023-05-31T19:48:24.724843+0000","last_peered":"2023-05-31T19:48:24.724843+0000","last_clean":"2023-05-31T19:48:24.724843+0000","last_became_active":"2023-05-31T19:48:23.725976+0000","last_became_peered":"2023-05-31T19:48:23.725976+0000","last_unstale":"2023-05-31T19:48:24.724843+0000","last_undegraded":"2023-05-31T19:48:24.724843+0000","last_fullsized":"2023-05-31T19:48:24.724843+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-31T19:48:22.713746+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-31T19:48:22.713746+0000","last_clean_scrub_stamp":"2023-05-31T19:48:22.713746+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-02T01:49:59.993152+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":9,"reported_epoch":10,"state":"active+clean","last_fresh":"2023-05-31T19:48:23.730757+0000","last_change":"2023-05-31T19:48:23.725521+0000","last_active":"2023-05-31T19:48:23.730757+0000","last_peered":"2023-05-31T19:48:23.730757+0000","last_clean":"2023-05-31T19:48:23.730757+0000","last_became_active":"2023-05-31T19:48:23.725103+0000","last_became_peered":"2023-05-31T19:48:23.725103+0000","last_unstale":"2023-05-31T19:48:23.730757+0000","last_undegraded":"2023-05-31T19:48:23.730757+0000","last_fullsized":"2023-05-31T19:48:23.730757+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-31T19:48:22.713746+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-31T19:48:22.713746+0000","last_clean_scrub_stamp":"2023-05-31T19:48:22.713746+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-01T22:37:14.601905+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":9,"reported_epoch":10,"state":"active+clean","last_fresh":"2023-05-31T19:48:23.728206+0000","last_change":"2023-05-31T19:48:23.724261+0000","last_active":"2023-05-31T19:48:23.728206+0000","last_peered":"2023-05-31T19:48:23.728206+0000","last_clean":"2023-05-31T19:48:23.728206+0000","last_became_active":"2023-05-31T19:48:23.724015+0000","last_became_peered":"2023-05-31T19:48:23.724015+0000","last_unstale":"2023-05-31T19:48:23.728206+0000","last_undegraded":"2023-05-31T19:48:23.728206+0000","last_fullsized":"2023-05-31T19:48:23.728206+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-31T19:48:22.713746+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-31T19:48:22.713746+0000","last_clean_scrub_stamp":"2023-05-31T19:48:22.713746+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-01T21:20:31.261171+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":10,"reported_epoch":11,"state":"active+clean","last_fresh":"2023-05-31T19:48:24.724906+0000","last_change":"2023-05-31T19:48:23.728761+0000","last_active":"2023-05-31T19:48:24.724906+0000","last_peered":"2023-05-31T19:48:24.724906+0000","last_clean":"2023-05-31T19:48:24.724906+0000","last_became_active":"2023-05-31T19:48:23.728515+0000","last_became_peered":"2023-05-31T19:48:23.728515+0000","last_unstale":"2023-05-31T19:48:24.724906+0000","last_undegraded":"2023-05-31T19:48:24.724906+0000","last_fullsized":"2023-05-31T19:48:24.724906+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-31T19:48:22.713746+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-31T19:48:22.713746+0000","last_clean_scrub_stamp":"2023-05-31T19:48:22.713746+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-01T22:06:47.240221+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738370,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":7648,"kb_used_data":88,"kb_used_omap":0,"kb_used_meta":7552,"kb_avail":104849952,"statfs":{"total":107374182400,"available":107366350848,"internally_reserved":0,"allocated":90112,"data_stored":22580,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":7733248},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[]},{"osd":0,"up_from":8,"seq":34359738370,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":8612,"kb_used_data":668,"kb_used_omap":0,"kb_used_meta":7936,"kb_avail":104848988,"statfs":{"total":107374182400,"available":107365363712,"internally_reserved":0,"allocated":684032,"data_stored":612948,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":8126464},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[]},{"osd":1,"up_from":8,"seq":34359738370,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":8684,"kb_used_data":676,"kb_used_omap":0,"kb_used_meta":8000,"kb_avail":104848916,"statfs":{"total":107374182400,"available":107365289984,"internally_reserved":0,"allocated":692224,"data_stored":616796,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":8192000},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.28299999999999997}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.36499999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-05-31T19:48:27.209 DEBUG:teuthology.orchestra.run.smithi099:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c1 pg dump --format=json 2023-05-31T19:48:27.522 INFO:teuthology.orchestra.run.smithi099.stdout: 2023-05-31T19:48:27.524 INFO:teuthology.orchestra.run.smithi099.stderr:dumped all 2023-05-31T19:48:27.535 INFO:teuthology.orchestra.run.smithi099.stdout:{"pg_ready":true,"pg_map":{"version":9,"stamp":"2023-05-31T19:48:25.711441+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":6,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":314572800,"kb_used":24944,"kb_used_data":1432,"kb_used_omap":0,"kb_used_meta":23488,"kb_avail":314547856,"statfs":{"total":322122547200,"available":322097004544,"internally_reserved":0,"allocated":1466368,"data_stored":1252324,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":24051712},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"0.000000"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":10,"reported_epoch":11,"state":"active+clean","last_fresh":"2023-05-31T19:48:24.724930+0000","last_change":"2023-05-31T19:48:23.728893+0000","last_active":"2023-05-31T19:48:24.724930+0000","last_peered":"2023-05-31T19:48:24.724930+0000","last_clean":"2023-05-31T19:48:24.724930+0000","last_became_active":"2023-05-31T19:48:23.728633+0000","last_became_peered":"2023-05-31T19:48:23.728633+0000","last_unstale":"2023-05-31T19:48:24.724930+0000","last_undegraded":"2023-05-31T19:48:24.724930+0000","last_fullsized":"2023-05-31T19:48:24.724930+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-31T19:48:22.713746+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-31T19:48:22.713746+0000","last_clean_scrub_stamp":"2023-05-31T19:48:22.713746+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-01T21:00:43.468297+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":10,"reported_epoch":11,"state":"active+clean","last_fresh":"2023-05-31T19:48:24.724906+0000","last_change":"2023-05-31T19:48:23.728965+0000","last_active":"2023-05-31T19:48:24.724906+0000","last_peered":"2023-05-31T19:48:24.724906+0000","last_clean":"2023-05-31T19:48:24.724906+0000","last_became_active":"2023-05-31T19:48:23.728496+0000","last_became_peered":"2023-05-31T19:48:23.728496+0000","last_unstale":"2023-05-31T19:48:24.724906+0000","last_undegraded":"2023-05-31T19:48:24.724906+0000","last_fullsized":"2023-05-31T19:48:24.724906+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-31T19:48:22.713746+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-31T19:48:22.713746+0000","last_clean_scrub_stamp":"2023-05-31T19:48:22.713746+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-01T22:31:01.359974+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":10,"reported_epoch":11,"state":"active+clean","last_fresh":"2023-05-31T19:48:24.724888+0000","last_change":"2023-05-31T19:48:23.726504+0000","last_active":"2023-05-31T19:48:24.724888+0000","last_peered":"2023-05-31T19:48:24.724888+0000","last_clean":"2023-05-31T19:48:24.724888+0000","last_became_active":"2023-05-31T19:48:23.726227+0000","last_became_peered":"2023-05-31T19:48:23.726227+0000","last_unstale":"2023-05-31T19:48:24.724888+0000","last_undegraded":"2023-05-31T19:48:24.724888+0000","last_fullsized":"2023-05-31T19:48:24.724888+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-31T19:48:22.713746+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-31T19:48:22.713746+0000","last_clean_scrub_stamp":"2023-05-31T19:48:22.713746+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-02T02:09:21.030368+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":9,"reported_epoch":10,"state":"active+clean","last_fresh":"2023-05-31T19:48:23.730770+0000","last_change":"2023-05-31T19:48:23.725911+0000","last_active":"2023-05-31T19:48:23.730770+0000","last_peered":"2023-05-31T19:48:23.730770+0000","last_clean":"2023-05-31T19:48:23.730770+0000","last_became_active":"2023-05-31T19:48:23.725649+0000","last_became_peered":"2023-05-31T19:48:23.725649+0000","last_unstale":"2023-05-31T19:48:23.730770+0000","last_undegraded":"2023-05-31T19:48:23.730770+0000","last_fullsized":"2023-05-31T19:48:23.730770+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-31T19:48:22.713746+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-31T19:48:22.713746+0000","last_clean_scrub_stamp":"2023-05-31T19:48:22.713746+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-01T21:41:24.762841+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"10'97","reported_seq":137,"reported_epoch":11,"state":"active+clean","last_fresh":"2023-05-31T19:48:24.724937+0000","last_change":"2023-05-31T19:48:23.728848+0000","last_active":"2023-05-31T19:48:24.724937+0000","last_peered":"2023-05-31T19:48:24.724937+0000","last_clean":"2023-05-31T19:48:24.724937+0000","last_became_active":"2023-05-31T19:48:23.728579+0000","last_became_peered":"2023-05-31T19:48:23.728579+0000","last_unstale":"2023-05-31T19:48:24.724937+0000","last_undegraded":"2023-05-31T19:48:24.724937+0000","last_fullsized":"2023-05-31T19:48:24.724937+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-31T19:48:22.713746+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-31T19:48:22.713746+0000","last_clean_scrub_stamp":"2023-05-31T19:48:22.713746+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-01T22:58:09.119583+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":10,"reported_epoch":11,"state":"active+clean","last_fresh":"2023-05-31T19:48:24.724843+0000","last_change":"2023-05-31T19:48:23.726215+0000","last_active":"2023-05-31T19:48:24.724843+0000","last_peered":"2023-05-31T19:48:24.724843+0000","last_clean":"2023-05-31T19:48:24.724843+0000","last_became_active":"2023-05-31T19:48:23.725976+0000","last_became_peered":"2023-05-31T19:48:23.725976+0000","last_unstale":"2023-05-31T19:48:24.724843+0000","last_undegraded":"2023-05-31T19:48:24.724843+0000","last_fullsized":"2023-05-31T19:48:24.724843+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-31T19:48:22.713746+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-31T19:48:22.713746+0000","last_clean_scrub_stamp":"2023-05-31T19:48:22.713746+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-02T01:49:59.993152+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":9,"reported_epoch":10,"state":"active+clean","last_fresh":"2023-05-31T19:48:23.730757+0000","last_change":"2023-05-31T19:48:23.725521+0000","last_active":"2023-05-31T19:48:23.730757+0000","last_peered":"2023-05-31T19:48:23.730757+0000","last_clean":"2023-05-31T19:48:23.730757+0000","last_became_active":"2023-05-31T19:48:23.725103+0000","last_became_peered":"2023-05-31T19:48:23.725103+0000","last_unstale":"2023-05-31T19:48:23.730757+0000","last_undegraded":"2023-05-31T19:48:23.730757+0000","last_fullsized":"2023-05-31T19:48:23.730757+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-31T19:48:22.713746+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-31T19:48:22.713746+0000","last_clean_scrub_stamp":"2023-05-31T19:48:22.713746+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-01T22:37:14.601905+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":9,"reported_epoch":10,"state":"active+clean","last_fresh":"2023-05-31T19:48:23.728206+0000","last_change":"2023-05-31T19:48:23.724261+0000","last_active":"2023-05-31T19:48:23.728206+0000","last_peered":"2023-05-31T19:48:23.728206+0000","last_clean":"2023-05-31T19:48:23.728206+0000","last_became_active":"2023-05-31T19:48:23.724015+0000","last_became_peered":"2023-05-31T19:48:23.724015+0000","last_unstale":"2023-05-31T19:48:23.728206+0000","last_undegraded":"2023-05-31T19:48:23.728206+0000","last_fullsized":"2023-05-31T19:48:23.728206+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-31T19:48:22.713746+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-31T19:48:22.713746+0000","last_clean_scrub_stamp":"2023-05-31T19:48:22.713746+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-01T21:20:31.261171+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":10,"reported_epoch":11,"state":"active+clean","last_fresh":"2023-05-31T19:48:24.724906+0000","last_change":"2023-05-31T19:48:23.728761+0000","last_active":"2023-05-31T19:48:24.724906+0000","last_peered":"2023-05-31T19:48:24.724906+0000","last_clean":"2023-05-31T19:48:24.724906+0000","last_became_active":"2023-05-31T19:48:23.728515+0000","last_became_peered":"2023-05-31T19:48:23.728515+0000","last_unstale":"2023-05-31T19:48:24.724906+0000","last_undegraded":"2023-05-31T19:48:24.724906+0000","last_fullsized":"2023-05-31T19:48:24.724906+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-31T19:48:22.713746+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-31T19:48:22.713746+0000","last_clean_scrub_stamp":"2023-05-31T19:48:22.713746+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-01T22:06:47.240221+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738370,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":7648,"kb_used_data":88,"kb_used_omap":0,"kb_used_meta":7552,"kb_avail":104849952,"statfs":{"total":107374182400,"available":107366350848,"internally_reserved":0,"allocated":90112,"data_stored":22580,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":7733248},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[]},{"osd":0,"up_from":8,"seq":34359738370,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":8612,"kb_used_data":668,"kb_used_omap":0,"kb_used_meta":7936,"kb_avail":104848988,"statfs":{"total":107374182400,"available":107365363712,"internally_reserved":0,"allocated":684032,"data_stored":612948,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":8126464},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[]},{"osd":1,"up_from":8,"seq":34359738370,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":8684,"kb_used_data":676,"kb_used_omap":0,"kb_used_meta":8000,"kb_avail":104848916,"statfs":{"total":107374182400,"available":107365289984,"internally_reserved":0,"allocated":692224,"data_stored":616796,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":8192000},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.28299999999999997}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.36499999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-05-31T19:48:27.537 INFO:tasks.ceph.ceph_manager.c1:clean! 2023-05-31T19:48:27.537 INFO:tasks.ceph:Waiting until ceph cluster c1 is healthy... 2023-05-31T19:48:27.538 INFO:tasks.ceph.ceph_manager.c1:wait_until_healthy 2023-05-31T19:48:27.539 DEBUG:teuthology.orchestra.run.smithi099:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c1 health --format=json 2023-05-31T19:48:27.899 INFO:teuthology.orchestra.run.smithi099.stdout: 2023-05-31T19:48:27.901 INFO:teuthology.orchestra.run.smithi099.stdout:{"status":"HEALTH_OK","checks":{},"mutes":[]} 2023-05-31T19:48:27.913 INFO:tasks.ceph.ceph_manager.c1:wait_until_healthy done 2023-05-31T19:48:27.913 INFO:teuthology.run_tasks:Running task ceph... 2023-05-31T19:48:27.926 INFO:tasks.ceph:Creating ceph cluster c2... 2023-05-31T19:48:27.927 INFO:tasks.ceph:config {'conf': {'client': {'debug rgw': 20, 'debug rgw sync': 20, 'rgw crypt require ssl': False, 'rgw crypt s3 kms backend': 'testing', 'rgw crypt s3 kms encryption keys': 'testkey-1=YmluCmJvb3N0CmJvb3N0LWJ1aWxkCmNlcGguY29uZgo=', 'rgw curl low speed time': 300, 'rgw data log num shards': 4, 'rgw md log max shards': 4, 'rgw sync data inject err probability': 0.1, 'rgw sync log trim interval': 0, 'rgw sync meta inject err probability': 0.1, 'rgw sync obj etag verify': True, 'setgroup': 'ceph', 'setuser': 'ceph'}, 'client.0': {'rgw data notify interval msec': 0}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20}, 'osd': {'debug ms': 1, 'debug osd': 20, 'rocksdb delete range threshold': 0}, 'osd.0': {'osd_max_omap_entries_per_request': 10}, 'osd.1': {'osd_max_omap_entries_per_request': 1000}, 'osd.2': {'osd_max_omap_entries_per_request': 10000}}, 'fs': 'xfs', 'mkfs_options': None, 'mount_options': None, 'skip_mgr_daemons': False, 'log_ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(PG_AVAILABILITY\\)', '\\(PG_DEGRADED\\)', '\\(POOL_APP_NOT_ENABLED\\)'], 'cpu_profile': set(), 'cluster': 'c2', 'mon_bind_msgr2': True, 'mon_bind_addrvec': True} 2023-05-31T19:48:27.927 INFO:tasks.ceph:ctx.config {'archive_path': '/home/teuthworker/archive/yuriw-2023-05-31_19:25:20-rgw-reef-release-distro-default-smithi/7292285', 'branch': 'reef-release', 'description': 'rgw/multisite/{clusters frontend/beast ignore-pg-availability notify omap_limits overrides realms/two-zonegroup supported-random-distro$/{centos_8} tasks/test_multi}', 'email': 'yweinste@redhat.com', 'first_in_suite': False, 'job_id': '7292285', 'kernel': {'kdb': True, 'sha1': 'distro'}, 'last_in_suite': False, 'machine_type': 'smithi', 'name': 'yuriw-2023-05-31_19:25:20-rgw-reef-release-distro-default-smithi', 'no_nested_subset': False, 'nuke-on-error': True, 'os_type': 'centos', 'os_version': '8.stream', 'overrides': {'admin_socket': {'branch': 'reef-release'}, 'ceph': {'conf': {'client': {'debug rgw': 20, 'debug rgw sync': 20, 'rgw crypt require ssl': False, 'rgw crypt s3 kms backend': 'testing', 'rgw crypt s3 kms encryption keys': 'testkey-1=YmluCmJvb3N0CmJvb3N0LWJ1aWxkCmNlcGguY29uZgo=', 'rgw curl low speed time': 300, 'rgw data log num shards': 4, 'rgw md log max shards': 4, 'rgw sync data inject err probability': 0.1, 'rgw sync log trim interval': 0, 'rgw sync meta inject err probability': 0.1, 'rgw sync obj etag verify': True, 'setgroup': 'ceph', 'setuser': 'ceph'}, 'client.0': {'rgw data notify interval msec': 0}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20}, 'osd': {'debug ms': 1, 'debug osd': 20, 'rocksdb delete range threshold': 0}, 'osd.0': {'osd_max_omap_entries_per_request': 10}, 'osd.1': {'osd_max_omap_entries_per_request': 1000}, 'osd.2': {'osd_max_omap_entries_per_request': 10000}}, 'flavor': 'default', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(PG_AVAILABILITY\\)', '\\(PG_DEGRADED\\)', '\\(POOL_APP_NOT_ENABLED\\)'], 'sha1': '4a02f3f496d9039326c49bf1fbe140388cd2f619', 'wait-for-scrub': False}, 'ceph-deploy': {'conf': {'client': {'log file': '/var/log/ceph/ceph-$name.$pid.log'}, 'mon': {'osd default pool size': 2}}}, 'install': {'ceph': {'flavor': 'default', 'sha1': '4a02f3f496d9039326c49bf1fbe140388cd2f619'}}, 'rgw': {'compression type': 'random', 'frontend': 'beast'}, 'rgw-multisite': {'realm': {'is default': True, 'name': 'test-realm'}, 'zonegroups': [{'enabled_features': ['resharding'], 'endpoints': ['c1.client.0'], 'is_default': True, 'is_master': True, 'name': 'a', 'zones': [{'endpoints': ['c1.client.0'], 'is_default': True, 'is_master': True, 'name': 'a1'}, {'endpoints': ['c1.client.1'], 'name': 'a2'}]}, {'enabled_features': ['resharding'], 'endpoints': ['c2.client.0'], 'is_default': True, 'name': 'b', 'zones': [{'endpoints': ['c2.client.0'], 'is_default': True, 'is_master': True, 'name': 'b1'}, {'endpoints': ['c2.client.1'], 'name': 'b2'}]}]}, 'rgw-multisite-tests': {'args': ['tests.py']}, 'selinux': {'whitelist': ['scontext=system_u:system_r:logrotate_t:s0']}, 'workunit': {'branch': 'reef-release', 'sha1': '2870bf456d3115490c153abce9d47ac5c043b65e'}}, 'owner': 'scheduled_yuriw@teuthology', 'priority': 71, 'repo': 'https://github.com/ceph/ceph-ci.git', 'roles': [['c1.mon.a', 'c1.mgr.x', 'c1.osd.0', 'c1.osd.1', 'c1.osd.2', 'c1.client.0', 'c1.client.1'], ['c2.mon.a', 'c2.mgr.x', 'c2.osd.0', 'c2.osd.1', 'c2.osd.2', 'c2.client.0', 'c2.client.1']], 'seed': 2752, 'sha1': '4a02f3f496d9039326c49bf1fbe140388cd2f619', 'sleep_before_teardown': 0, 'subset': '111/12000', 'suite': 'rgw', 'suite_branch': 'reef-release', 'suite_path': '/home/teuthworker/src/github.com_ceph_ceph-c_2870bf456d3115490c153abce9d47ac5c043b65e/qa', 'suite_relpath': 'qa', 'suite_repo': 'https://github.com/ceph/ceph-ci.git', 'suite_sha1': '2870bf456d3115490c153abce9d47ac5c043b65e', 'targets': {'smithi099.front.sepia.ceph.com': 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDox97V1ZbGiXYfIam0QsAKCOossAxUlL++9Y+qMtFgf7WbqpNcOYHYWWtH1ov+9qROYmTCRNQIFX+NkdHU8Q2Wbwg5LFP43nv89monL09OSkXGOUGTF6MF+MDeWU/gCRrcDxxs4QnoWTTbcd4HtqMzUprCFR5KLeO4XJSkTCxXlVJC1I2F9x9uGkv5VM4gB0ZVUeoVsATmS79+GLwNLH2DhXh4JQJL9HKgC9FN03CerL+Ka6cdcu3Wc6rSelCxMx0Ead4Vsre8il1J8PfCPvcko7HAkfp2XPu3avJXTKUTKYrF05s3uZq027l5T6VrlavgVtfGsyB8Ez4YDolVQOoGGfF2I99aVh9LAYc+J8KOY1Qh5KaILCCmrMI74XRPrqPrg7phDNIDNwjAxPcmoKpNDR6ALert5o+mM8onphQ/yRxisjwMH/bNXE63CdbjgkIOm2VJ2g/k1Ncf4lBxUPSx3eF+ERlKFdboN8S567n9hWRVwbSdGmzQf3AsOb38Hlk=', 'smithi112.front.sepia.ceph.com': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBFfCJhk9QMgDL70wZsKQnRvWaVVb4spsBOWpIa4TPw/+clTUBIAM2haG88c+/+tNQhpaYNEiHo4bsY1a+mmIK9A='}, 'tasks': [{'internal.check_packages': None}, {'internal.buildpackages_prep': None}, {'internal.save_config': None}, {'internal.check_lock': None}, {'internal.add_remotes': None}, {'console_log': None}, {'internal.connect': None}, {'internal.push_inventory': None}, {'internal.serialize_remote_roles': None}, {'internal.check_conflict': None}, {'internal.check_ceph_data': None}, {'internal.vm_setup': None}, {'kernel': {'kdb': True, 'sha1': 'distro'}}, {'internal.base': None}, {'internal.archive_upload': None}, {'internal.archive': None}, {'internal.coredump': None}, {'internal.sudo': None}, {'internal.syslog': None}, {'internal.timer': None}, {'pcp': None}, {'selinux': None}, {'ansible.cephlab': None}, {'clock': None}, {'install': None}, {'ceph': {'cluster': 'c1', 'conf': {'client': {'debug rgw': 20, 'debug rgw sync': 20, 'rgw crypt require ssl': False, 'rgw crypt s3 kms backend': 'testing', 'rgw crypt s3 kms encryption keys': 'testkey-1=YmluCmJvb3N0CmJvb3N0LWJ1aWxkCmNlcGguY29uZgo=', 'rgw curl low speed time': 300, 'rgw data log num shards': 4, 'rgw md log max shards': 4, 'rgw sync data inject err probability': 0.1, 'rgw sync log trim interval': 0, 'rgw sync meta inject err probability': 0.1, 'rgw sync obj etag verify': True, 'setgroup': 'ceph', 'setuser': 'ceph'}, 'client.0': {'rgw data notify interval msec': 0}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20}, 'osd': {'debug ms': 1, 'debug osd': 20, 'rocksdb delete range threshold': 0}, 'osd.0': {'osd_max_omap_entries_per_request': 10}, 'osd.1': {'osd_max_omap_entries_per_request': 1000}, 'osd.2': {'osd_max_omap_entries_per_request': 10000}}, 'flavor': 'default', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(PG_AVAILABILITY\\)', '\\(PG_DEGRADED\\)', '\\(POOL_APP_NOT_ENABLED\\)'], 'sha1': '4a02f3f496d9039326c49bf1fbe140388cd2f619', 'wait-for-scrub': False}}, {'ceph': {'cluster': 'c2', 'conf': {'client': {'debug rgw': 20, 'debug rgw sync': 20, 'rgw crypt require ssl': False, 'rgw crypt s3 kms backend': 'testing', 'rgw crypt s3 kms encryption keys': 'testkey-1=YmluCmJvb3N0CmJvb3N0LWJ1aWxkCmNlcGguY29uZgo=', 'rgw curl low speed time': 300, 'rgw data log num shards': 4, 'rgw md log max shards': 4, 'rgw sync data inject err probability': 0.1, 'rgw sync log trim interval': 0, 'rgw sync meta inject err probability': 0.1, 'rgw sync obj etag verify': True, 'setgroup': 'ceph', 'setuser': 'ceph'}, 'client.0': {'rgw data notify interval msec': 0}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20}, 'osd': {'debug ms': 1, 'debug osd': 20, 'rocksdb delete range threshold': 0}, 'osd.0': {'osd_max_omap_entries_per_request': 10}, 'osd.1': {'osd_max_omap_entries_per_request': 1000}, 'osd.2': {'osd_max_omap_entries_per_request': 10000}}, 'flavor': 'default', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(PG_AVAILABILITY\\)', '\\(PG_DEGRADED\\)', '\\(POOL_APP_NOT_ENABLED\\)'], 'sha1': '4a02f3f496d9039326c49bf1fbe140388cd2f619', 'wait-for-scrub': False}}, {'rgw': {'c1.client.0': {'port': 8000}, 'c1.client.1': {'port': 8001}, 'c2.client.0': {'port': 8000}, 'c2.client.1': {'port': 8001}}}, {'rgw-multisite': None}, {'rgw-multisite-tests': {'config': {'reconfigure_delay': 60}}}], 'teuthology': {'fragments_dropped': [], 'meta': {}, 'postmerge': []}, 'teuthology_branch': 'main', 'teuthology_sha1': 'fa97ddd17d12e6d74b49196948f66e54fa009130', 'timestamp': '2023-05-31_19:25:20', 'tube': 'smithi', 'user': 'yuriw', 'verbose': True, 'worker_log': '/home/teuthworker/archive/worker_logs/dispatcher.smithi.23459'} 2023-05-31T19:48:27.928 DEBUG:teuthology.orchestra.run.smithi099:> install -d -m0755 -- /home/ubuntu/cephtest/c2.data 2023-05-31T19:48:27.931 DEBUG:teuthology.orchestra.run.smithi112:> install -d -m0755 -- /home/ubuntu/cephtest/c2.data 2023-05-31T19:48:27.954 DEBUG:teuthology.orchestra.run.smithi099:> sudo install -d -m0777 -- /var/run/ceph 2023-05-31T19:48:27.990 DEBUG:teuthology.orchestra.run.smithi112:> sudo install -d -m0777 -- /var/run/ceph 2023-05-31T19:48:28.027 DEBUG:teuthology.orchestra.run.smithi112:> set -ex 2023-05-31T19:48:28.028 DEBUG:teuthology.orchestra.run.smithi112:> dd if=/scratch_devs of=/dev/stdout 2023-05-31T19:48:28.089 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2023-05-31T19:48:28.089 DEBUG:teuthology.orchestra.run.smithi112:> stat /dev/vg_nvme/lv_1 2023-05-31T19:48:28.149 INFO:teuthology.orchestra.run.smithi112.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2023-05-31T19:48:28.149 INFO:teuthology.orchestra.run.smithi112.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-05-31T19:48:28.149 INFO:teuthology.orchestra.run.smithi112.stdout:Device: 6h/6d Inode: 360288 Links: 1 2023-05-31T19:48:28.149 INFO:teuthology.orchestra.run.smithi112.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-05-31T19:48:28.149 INFO:teuthology.orchestra.run.smithi112.stdout:Context: system_u:object_r:device_t:s0 2023-05-31T19:48:28.149 INFO:teuthology.orchestra.run.smithi112.stdout:Access: 2023-05-31 19:45:33.379446870 +0000 2023-05-31T19:48:28.150 INFO:teuthology.orchestra.run.smithi112.stdout:Modify: 2023-05-31 19:41:50.584131701 +0000 2023-05-31T19:48:28.150 INFO:teuthology.orchestra.run.smithi112.stdout:Change: 2023-05-31 19:41:50.584131701 +0000 2023-05-31T19:48:28.150 INFO:teuthology.orchestra.run.smithi112.stdout: Birth: - 2023-05-31T19:48:28.150 DEBUG:teuthology.orchestra.run.smithi112:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2023-05-31T19:48:28.217 INFO:teuthology.orchestra.run.smithi112.stderr:1+0 records in 2023-05-31T19:48:28.218 INFO:teuthology.orchestra.run.smithi112.stderr:1+0 records out 2023-05-31T19:48:28.218 INFO:teuthology.orchestra.run.smithi112.stderr:512 bytes copied, 0.000245347 s, 2.1 MB/s 2023-05-31T19:48:28.219 DEBUG:teuthology.orchestra.run.smithi112:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2023-05-31T19:48:28.281 DEBUG:teuthology.orchestra.run.smithi112:> stat /dev/vg_nvme/lv_2 2023-05-31T19:48:28.340 INFO:teuthology.orchestra.run.smithi112.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2023-05-31T19:48:28.340 INFO:teuthology.orchestra.run.smithi112.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-05-31T19:48:28.340 INFO:teuthology.orchestra.run.smithi112.stdout:Device: 6h/6d Inode: 369586 Links: 1 2023-05-31T19:48:28.340 INFO:teuthology.orchestra.run.smithi112.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-05-31T19:48:28.341 INFO:teuthology.orchestra.run.smithi112.stdout:Context: system_u:object_r:device_t:s0 2023-05-31T19:48:28.341 INFO:teuthology.orchestra.run.smithi112.stdout:Access: 2023-05-31 19:45:33.379446870 +0000 2023-05-31T19:48:28.341 INFO:teuthology.orchestra.run.smithi112.stdout:Modify: 2023-05-31 19:41:51.114117745 +0000 2023-05-31T19:48:28.341 INFO:teuthology.orchestra.run.smithi112.stdout:Change: 2023-05-31 19:41:51.114117745 +0000 2023-05-31T19:48:28.342 INFO:teuthology.orchestra.run.smithi112.stdout: Birth: - 2023-05-31T19:48:28.342 DEBUG:teuthology.orchestra.run.smithi112:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2023-05-31T19:48:28.407 INFO:teuthology.orchestra.run.smithi112.stderr:1+0 records in 2023-05-31T19:48:28.408 INFO:teuthology.orchestra.run.smithi112.stderr:1+0 records out 2023-05-31T19:48:28.408 INFO:teuthology.orchestra.run.smithi112.stderr:512 bytes copied, 0.000229296 s, 2.2 MB/s 2023-05-31T19:48:28.409 DEBUG:teuthology.orchestra.run.smithi112:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2023-05-31T19:48:28.469 DEBUG:teuthology.orchestra.run.smithi112:> stat /dev/vg_nvme/lv_3 2023-05-31T19:48:28.528 INFO:teuthology.orchestra.run.smithi112.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2023-05-31T19:48:28.529 INFO:teuthology.orchestra.run.smithi112.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-05-31T19:48:28.529 INFO:teuthology.orchestra.run.smithi112.stdout:Device: 6h/6d Inode: 363350 Links: 1 2023-05-31T19:48:28.529 INFO:teuthology.orchestra.run.smithi112.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-05-31T19:48:28.530 INFO:teuthology.orchestra.run.smithi112.stdout:Context: system_u:object_r:device_t:s0 2023-05-31T19:48:28.530 INFO:teuthology.orchestra.run.smithi112.stdout:Access: 2023-05-31 19:45:33.380446843 +0000 2023-05-31T19:48:28.530 INFO:teuthology.orchestra.run.smithi112.stdout:Modify: 2023-05-31 19:41:51.684102736 +0000 2023-05-31T19:48:28.530 INFO:teuthology.orchestra.run.smithi112.stdout:Change: 2023-05-31 19:41:51.684102736 +0000 2023-05-31T19:48:28.531 INFO:teuthology.orchestra.run.smithi112.stdout: Birth: - 2023-05-31T19:48:28.531 DEBUG:teuthology.orchestra.run.smithi112:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2023-05-31T19:48:28.595 INFO:teuthology.orchestra.run.smithi112.stderr:1+0 records in 2023-05-31T19:48:28.595 INFO:teuthology.orchestra.run.smithi112.stderr:1+0 records out 2023-05-31T19:48:28.595 INFO:teuthology.orchestra.run.smithi112.stderr:512 bytes copied, 0.000242294 s, 2.1 MB/s 2023-05-31T19:48:28.596 DEBUG:teuthology.orchestra.run.smithi112:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2023-05-31T19:48:28.655 DEBUG:teuthology.orchestra.run.smithi112:> stat /dev/vg_nvme/lv_4 2023-05-31T19:48:28.715 INFO:teuthology.orchestra.run.smithi112.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2023-05-31T19:48:28.715 INFO:teuthology.orchestra.run.smithi112.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-05-31T19:48:28.715 INFO:teuthology.orchestra.run.smithi112.stdout:Device: 6h/6d Inode: 368273 Links: 1 2023-05-31T19:48:28.715 INFO:teuthology.orchestra.run.smithi112.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-05-31T19:48:28.715 INFO:teuthology.orchestra.run.smithi112.stdout:Context: system_u:object_r:device_t:s0 2023-05-31T19:48:28.716 INFO:teuthology.orchestra.run.smithi112.stdout:Access: 2023-05-31 19:45:33.380446843 +0000 2023-05-31T19:48:28.716 INFO:teuthology.orchestra.run.smithi112.stdout:Modify: 2023-05-31 19:41:52.392084094 +0000 2023-05-31T19:48:28.716 INFO:teuthology.orchestra.run.smithi112.stdout:Change: 2023-05-31 19:41:52.392084094 +0000 2023-05-31T19:48:28.716 INFO:teuthology.orchestra.run.smithi112.stdout: Birth: - 2023-05-31T19:48:28.716 DEBUG:teuthology.orchestra.run.smithi112:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2023-05-31T19:48:28.780 INFO:teuthology.orchestra.run.smithi112.stderr:1+0 records in 2023-05-31T19:48:28.781 INFO:teuthology.orchestra.run.smithi112.stderr:1+0 records out 2023-05-31T19:48:28.781 INFO:teuthology.orchestra.run.smithi112.stderr:512 bytes copied, 0.000214924 s, 2.4 MB/s 2023-05-31T19:48:28.782 DEBUG:teuthology.orchestra.run.smithi112:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2023-05-31T19:48:28.841 INFO:tasks.ceph:osd dev map: {'c2.osd.0': '/dev/vg_nvme/lv_1', 'c2.osd.1': '/dev/vg_nvme/lv_2', 'c2.osd.2': '/dev/vg_nvme/lv_3'} 2023-05-31T19:48:28.842 INFO:tasks.ceph:remote_to_roles_to_devs: {Remote(name='ubuntu@smithi112.front.sepia.ceph.com'): {'c2.osd.0': '/dev/vg_nvme/lv_1', 'c2.osd.1': '/dev/vg_nvme/lv_2', 'c2.osd.2': '/dev/vg_nvme/lv_3'}} 2023-05-31T19:48:28.842 INFO:tasks.ceph:Generating config... 2023-05-31T19:48:28.843 INFO:tasks.ceph:[client] debug rgw = 20 2023-05-31T19:48:28.843 INFO:tasks.ceph:[client] debug rgw sync = 20 2023-05-31T19:48:28.844 INFO:tasks.ceph:[client] rgw crypt require ssl = False 2023-05-31T19:48:28.844 INFO:tasks.ceph:[client] rgw crypt s3 kms backend = testing 2023-05-31T19:48:28.844 INFO:tasks.ceph:[client] rgw crypt s3 kms encryption keys = testkey-1=YmluCmJvb3N0CmJvb3N0LWJ1aWxkCmNlcGguY29uZgo= 2023-05-31T19:48:28.844 INFO:tasks.ceph:[client] rgw curl low speed time = 300 2023-05-31T19:48:28.844 INFO:tasks.ceph:[client] rgw data log num shards = 4 2023-05-31T19:48:28.845 INFO:tasks.ceph:[client] rgw md log max shards = 4 2023-05-31T19:48:28.845 INFO:tasks.ceph:[client] rgw sync data inject err probability = 0.1 2023-05-31T19:48:28.845 INFO:tasks.ceph:[client] rgw sync log trim interval = 0 2023-05-31T19:48:28.845 INFO:tasks.ceph:[client] rgw sync meta inject err probability = 0.1 2023-05-31T19:48:28.845 INFO:tasks.ceph:[client] rgw sync obj etag verify = True 2023-05-31T19:48:28.846 INFO:tasks.ceph:[client] setgroup = ceph 2023-05-31T19:48:28.846 INFO:tasks.ceph:[client] setuser = ceph 2023-05-31T19:48:28.846 INFO:tasks.ceph:[client.0] rgw data notify interval msec = 0 2023-05-31T19:48:28.846 INFO:tasks.ceph:[mgr] debug mgr = 20 2023-05-31T19:48:28.846 INFO:tasks.ceph:[mgr] debug ms = 1 2023-05-31T19:48:28.847 INFO:tasks.ceph:[mon] debug mon = 20 2023-05-31T19:48:28.847 INFO:tasks.ceph:[mon] debug ms = 1 2023-05-31T19:48:28.847 INFO:tasks.ceph:[mon] debug paxos = 20 2023-05-31T19:48:28.847 INFO:tasks.ceph:[osd] debug ms = 1 2023-05-31T19:48:28.848 INFO:tasks.ceph:[osd] debug osd = 20 2023-05-31T19:48:28.848 INFO:tasks.ceph:[osd] rocksdb delete range threshold = 0 2023-05-31T19:48:28.848 INFO:tasks.ceph:[osd.0] osd_max_omap_entries_per_request = 10 2023-05-31T19:48:28.848 INFO:tasks.ceph:[osd.1] osd_max_omap_entries_per_request = 1000 2023-05-31T19:48:28.849 INFO:tasks.ceph:[osd.2] osd_max_omap_entries_per_request = 10000 2023-05-31T19:48:28.849 INFO:tasks.ceph:Setting up c2.mon.a... 2023-05-31T19:48:28.850 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring /etc/ceph/c2.keyring 2023-05-31T19:48:28.943 INFO:teuthology.orchestra.run.smithi112.stdout:creating /etc/ceph/c2.keyring 2023-05-31T19:48:28.945 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=mon. /etc/ceph/c2.keyring 2023-05-31T19:48:28.990 DEBUG:teuthology.orchestra.run.smithi112:> sudo chmod 0644 /etc/ceph/c2.keyring 2023-05-31T19:48:29.017 DEBUG:tasks.ceph:Ceph mon addresses: [('c2.mon.a', '172.21.15.112')] 2023-05-31T19:48:29.018 DEBUG:tasks.ceph:writing out conf {'global': {'chdir': '', 'pid file': '/var/run/ceph/$cluster-$name.pid', 'auth supported': 'cephx', 'filestore xattr use omap': 'true', 'mon clock drift allowed': '1.000', 'osd crush chooseleaf type': '0', 'auth debug': 'true', 'ms die on old message': 'true', 'ms die on bug': 'true', 'mon max pg per osd': '10000', 'mon pg warn max object skew': '0', 'osd_pool_default_pg_autoscale_mode': 'off', 'osd pool default size': '2', 'mon osd allow primary affinity': 'true', 'mon osd allow pg remap': 'true', 'mon warn on legacy crush tunables': 'false', 'mon warn on crush straw calc version zero': 'false', 'mon warn on no sortbitwise': 'false', 'mon warn on osd down out interval zero': 'false', 'mon warn on too few osds': 'false', 'mon_warn_on_pool_pg_num_not_power_of_two': 'false', 'mon_warn_on_pool_no_redundancy': 'false', 'mon_allow_pool_size_one': 'true', 'osd pool default erasure code profile': 'plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd', 'osd default data pool replay window': '5', 'mon allow pool delete': 'true', 'mon cluster log file level': 'debug', 'debug asserts on shutdown': 'true', 'mon health detail to clog': 'false', 'mon host': '172.21.15.112'}, 'osd': {'osd journal size': '100', 'osd scrub load threshold': '5.0', 'osd scrub max interval': '600', 'osd mclock profile': 'high_recovery_ops', 'osd recover clone overlap': 'true', 'osd recovery max chunk': '1048576', 'osd debug shutdown': 'true', 'osd debug op order': 'true', 'osd debug verify stray on activate': 'true', 'osd open classes on start': 'true', 'osd debug pg log writeout': 'true', 'osd deep scrub update digest min age': '30', 'osd map max advance': '10', 'journal zero on create': 'true', 'filestore ondisk finisher threads': '3', 'filestore apply finisher threads': '3', 'bdev debug aio': 'true', 'osd debug misdirected ops': 'true', 'debug ms': 1, 'debug osd': 20, 'rocksdb delete range threshold': 0}, 'mgr': {'debug ms': 1, 'debug mgr': 20, 'debug mon': '20', 'debug auth': '20', 'mon reweight min pgs per osd': '4', 'mon reweight min bytes per osd': '10', 'mgr/telemetry/nag': 'false'}, 'mon': {'debug ms': 1, 'debug mon': 20, 'debug paxos': 20, 'debug auth': '20', 'mon data avail warn': '5', 'mon mgr mkfs grace': '240', 'mon reweight min pgs per osd': '4', 'mon osd reporter subtree level': 'osd', 'mon osd prime pg temp': 'true', 'mon reweight min bytes per osd': '10', 'auth mon ticket ttl': '660', 'auth service ticket ttl': '240', 'mon_warn_on_insecure_global_id_reclaim': 'false', 'mon_warn_on_insecure_global_id_reclaim_allowed': 'false', 'mon_down_mkfs_grace': '2m', 'mon_warn_on_filestore_osds': 'false'}, 'client': {'rgw cache enabled': 'true', 'rgw enable ops log': 'true', 'rgw enable usage log': 'true', 'log file': '/var/log/ceph/$cluster-$name.$pid.log', 'admin socket': '/var/run/ceph/$cluster-$name.$pid.asok', 'debug rgw': 20, 'debug rgw sync': 20, 'rgw crypt require ssl': False, 'rgw crypt s3 kms backend': 'testing', 'rgw crypt s3 kms encryption keys': 'testkey-1=YmluCmJvb3N0CmJvb3N0LWJ1aWxkCmNlcGguY29uZgo=', 'rgw curl low speed time': 300, 'rgw data log num shards': 4, 'rgw md log max shards': 4, 'rgw sync data inject err probability': 0.1, 'rgw sync log trim interval': 0, 'rgw sync meta inject err probability': 0.1, 'rgw sync obj etag verify': True, 'setgroup': 'ceph', 'setuser': 'ceph'}, 'mon.a': {}, 'client.0': {'rgw data notify interval msec': 0}, 'osd.0': {'osd_max_omap_entries_per_request': 10}, 'osd.1': {'osd_max_omap_entries_per_request': 1000}, 'osd.2': {'osd_max_omap_entries_per_request': 10000}} 2023-05-31T19:48:29.019 DEBUG:teuthology.orchestra.run.smithi112:> set -ex 2023-05-31T19:48:29.019 DEBUG:teuthology.orchestra.run.smithi112:> dd of=/home/ubuntu/cephtest/ceph.tmp.conf 2023-05-31T19:48:29.076 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage monmaptool -c /home/ubuntu/cephtest/ceph.tmp.conf --create --clobber --enable-all-features --add a 172.21.15.112 --print /home/ubuntu/cephtest/c2.monmap 2023-05-31T19:48:29.158 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-05-31T19:48:29.159 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-05-31T19:48:29.159 INFO:teuthology.orchestra.run.smithi112.stdout:monmaptool: monmap file /home/ubuntu/cephtest/c2.monmap 2023-05-31T19:48:29.159 INFO:teuthology.orchestra.run.smithi112.stdout:monmaptool: generated fsid ff2a32a9-2a3b-4db6-ad15-e0ad1a679cca 2023-05-31T19:48:29.160 INFO:teuthology.orchestra.run.smithi112.stdout:setting min_mon_release = pacific 2023-05-31T19:48:29.160 INFO:teuthology.orchestra.run.smithi112.stdout:epoch 0 2023-05-31T19:48:29.160 INFO:teuthology.orchestra.run.smithi112.stdout:fsid ff2a32a9-2a3b-4db6-ad15-e0ad1a679cca 2023-05-31T19:48:29.161 INFO:teuthology.orchestra.run.smithi112.stdout:last_changed 2023-05-31T19:48:29.154740+0000 2023-05-31T19:48:29.161 INFO:teuthology.orchestra.run.smithi112.stdout:created 2023-05-31T19:48:29.154740+0000 2023-05-31T19:48:29.161 INFO:teuthology.orchestra.run.smithi112.stdout:min_mon_release 16 (pacific) 2023-05-31T19:48:29.161 INFO:teuthology.orchestra.run.smithi112.stdout:election_strategy: 1 2023-05-31T19:48:29.161 INFO:teuthology.orchestra.run.smithi112.stdout:0: [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] mon.a 2023-05-31T19:48:29.161 INFO:teuthology.orchestra.run.smithi112.stdout:monmaptool: writing epoch 0 to /home/ubuntu/cephtest/c2.monmap (1 monitors) 2023-05-31T19:48:29.162 DEBUG:teuthology.orchestra.run.smithi112:> rm -- /home/ubuntu/cephtest/ceph.tmp.conf 2023-05-31T19:48:29.220 INFO:tasks.ceph:Writing /etc/ceph/c2.conf for FSID ff2a32a9-2a3b-4db6-ad15-e0ad1a679cca... 2023-05-31T19:48:29.221 DEBUG:teuthology.orchestra.run.smithi099:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/c2.conf && sudo chmod 0644 /etc/ceph/c2.conf > /dev/null 2023-05-31T19:48:29.224 DEBUG:teuthology.orchestra.run.smithi112:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/c2.conf && sudo chmod 0644 /etc/ceph/c2.conf > /dev/null 2023-05-31T19:48:29.275 INFO:teuthology.orchestra.run.smithi099.stdout:[global] 2023-05-31T19:48:29.276 INFO:teuthology.orchestra.run.smithi099.stdout: chdir = "" 2023-05-31T19:48:29.276 INFO:teuthology.orchestra.run.smithi099.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2023-05-31T19:48:29.276 INFO:teuthology.orchestra.run.smithi099.stdout: auth supported = cephx 2023-05-31T19:48:29.276 INFO:teuthology.orchestra.run.smithi099.stdout: 2023-05-31T19:48:29.277 INFO:teuthology.orchestra.run.smithi099.stdout: filestore xattr use omap = true 2023-05-31T19:48:29.277 INFO:teuthology.orchestra.run.smithi099.stdout: 2023-05-31T19:48:29.277 INFO:teuthology.orchestra.run.smithi099.stdout: mon clock drift allowed = 1.000 2023-05-31T19:48:29.277 INFO:teuthology.orchestra.run.smithi099.stdout: 2023-05-31T19:48:29.277 INFO:teuthology.orchestra.run.smithi099.stdout: osd crush chooseleaf type = 0 2023-05-31T19:48:29.277 INFO:teuthology.orchestra.run.smithi099.stdout: auth debug = true 2023-05-31T19:48:29.277 INFO:teuthology.orchestra.run.smithi099.stdout: 2023-05-31T19:48:29.278 INFO:teuthology.orchestra.run.smithi099.stdout: ms die on old message = true 2023-05-31T19:48:29.278 INFO:teuthology.orchestra.run.smithi099.stdout: ms die on bug = true 2023-05-31T19:48:29.278 INFO:teuthology.orchestra.run.smithi099.stdout: 2023-05-31T19:48:29.278 INFO:teuthology.orchestra.run.smithi099.stdout: mon max pg per osd = 10000 # >= luminous 2023-05-31T19:48:29.278 INFO:teuthology.orchestra.run.smithi099.stdout: mon pg warn max object skew = 0 2023-05-31T19:48:29.278 INFO:teuthology.orchestra.run.smithi099.stdout: 2023-05-31T19:48:29.278 INFO:teuthology.orchestra.run.smithi099.stdout: # disable pg_autoscaler by default for new pools 2023-05-31T19:48:29.279 INFO:teuthology.orchestra.run.smithi099.stdout: osd_pool_default_pg_autoscale_mode = off 2023-05-31T19:48:29.279 INFO:teuthology.orchestra.run.smithi099.stdout: 2023-05-31T19:48:29.279 INFO:teuthology.orchestra.run.smithi099.stdout: osd pool default size = 2 2023-05-31T19:48:29.279 INFO:teuthology.orchestra.run.smithi099.stdout: 2023-05-31T19:48:29.279 INFO:teuthology.orchestra.run.smithi099.stdout: mon osd allow primary affinity = true 2023-05-31T19:48:29.279 INFO:teuthology.orchestra.run.smithi099.stdout: mon osd allow pg remap = true 2023-05-31T19:48:29.279 INFO:teuthology.orchestra.run.smithi099.stdout: mon warn on legacy crush tunables = false 2023-05-31T19:48:29.280 INFO:teuthology.orchestra.run.smithi099.stdout: mon warn on crush straw calc version zero = false 2023-05-31T19:48:29.280 INFO:teuthology.orchestra.run.smithi099.stdout: mon warn on no sortbitwise = false 2023-05-31T19:48:29.280 INFO:teuthology.orchestra.run.smithi099.stdout: mon warn on osd down out interval zero = false 2023-05-31T19:48:29.280 INFO:teuthology.orchestra.run.smithi099.stdout: mon warn on too few osds = false 2023-05-31T19:48:29.280 INFO:teuthology.orchestra.run.smithi099.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2023-05-31T19:48:29.280 INFO:teuthology.orchestra.run.smithi099.stdout: mon_warn_on_pool_no_redundancy = false 2023-05-31T19:48:29.280 INFO:teuthology.orchestra.run.smithi099.stdout: mon_allow_pool_size_one = true 2023-05-31T19:48:29.280 INFO:teuthology.orchestra.run.smithi099.stdout: 2023-05-31T19:48:29.281 INFO:teuthology.orchestra.run.smithi099.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2023-05-31T19:48:29.281 INFO:teuthology.orchestra.run.smithi099.stdout: 2023-05-31T19:48:29.281 INFO:teuthology.orchestra.run.smithi099.stdout: osd default data pool replay window = 5 2023-05-31T19:48:29.281 INFO:teuthology.orchestra.run.smithi099.stdout: 2023-05-31T19:48:29.281 INFO:teuthology.orchestra.run.smithi099.stdout: mon allow pool delete = true 2023-05-31T19:48:29.281 INFO:teuthology.orchestra.run.smithi099.stdout: 2023-05-31T19:48:29.281 INFO:teuthology.orchestra.run.smithi099.stdout: mon cluster log file level = debug 2023-05-31T19:48:29.282 INFO:teuthology.orchestra.run.smithi099.stdout: debug asserts on shutdown = true 2023-05-31T19:48:29.282 INFO:teuthology.orchestra.run.smithi099.stdout: mon health detail to clog = false 2023-05-31T19:48:29.282 INFO:teuthology.orchestra.run.smithi099.stdout: mon host = 172.21.15.112 2023-05-31T19:48:29.282 INFO:teuthology.orchestra.run.smithi099.stdout: fsid = ff2a32a9-2a3b-4db6-ad15-e0ad1a679cca 2023-05-31T19:48:29.282 INFO:teuthology.orchestra.run.smithi099.stdout: 2023-05-31T19:48:29.282 INFO:teuthology.orchestra.run.smithi099.stdout:[osd] 2023-05-31T19:48:29.282 INFO:teuthology.orchestra.run.smithi099.stdout: osd journal size = 100 2023-05-31T19:48:29.282 INFO:teuthology.orchestra.run.smithi099.stdout: 2023-05-31T19:48:29.283 INFO:teuthology.orchestra.run.smithi099.stdout: osd scrub load threshold = 5.0 2023-05-31T19:48:29.283 INFO:teuthology.orchestra.run.smithi099.stdout: osd scrub max interval = 600 2023-05-31T19:48:29.283 INFO:teuthology.orchestra.run.smithi099.stdout: osd mclock profile = high_recovery_ops 2023-05-31T19:48:29.283 INFO:teuthology.orchestra.run.smithi099.stdout: 2023-05-31T19:48:29.283 INFO:teuthology.orchestra.run.smithi099.stdout: osd recover clone overlap = true 2023-05-31T19:48:29.283 INFO:teuthology.orchestra.run.smithi099.stdout: osd recovery max chunk = 1048576 2023-05-31T19:48:29.283 INFO:teuthology.orchestra.run.smithi099.stdout: 2023-05-31T19:48:29.283 INFO:teuthology.orchestra.run.smithi099.stdout: osd debug shutdown = true 2023-05-31T19:48:29.284 INFO:teuthology.orchestra.run.smithi099.stdout: osd debug op order = true 2023-05-31T19:48:29.284 INFO:teuthology.orchestra.run.smithi099.stdout: osd debug verify stray on activate = true 2023-05-31T19:48:29.284 INFO:teuthology.orchestra.run.smithi099.stdout: 2023-05-31T19:48:29.284 INFO:teuthology.orchestra.run.smithi099.stdout: osd open classes on start = true 2023-05-31T19:48:29.284 INFO:teuthology.orchestra.run.smithi099.stdout: osd debug pg log writeout = true 2023-05-31T19:48:29.284 INFO:teuthology.orchestra.run.smithi099.stdout: 2023-05-31T19:48:29.284 INFO:teuthology.orchestra.run.smithi099.stdout: osd deep scrub update digest min age = 30 2023-05-31T19:48:29.284 INFO:teuthology.orchestra.run.smithi099.stdout: 2023-05-31T19:48:29.285 INFO:teuthology.orchestra.run.smithi099.stdout: osd map max advance = 10 2023-05-31T19:48:29.285 INFO:teuthology.orchestra.run.smithi099.stdout: 2023-05-31T19:48:29.285 INFO:teuthology.orchestra.run.smithi099.stdout: journal zero on create = true 2023-05-31T19:48:29.285 INFO:teuthology.orchestra.run.smithi099.stdout: 2023-05-31T19:48:29.285 INFO:teuthology.orchestra.run.smithi099.stdout: filestore ondisk finisher threads = 3 2023-05-31T19:48:29.286 INFO:teuthology.orchestra.run.smithi099.stdout: filestore apply finisher threads = 3 2023-05-31T19:48:29.286 INFO:teuthology.orchestra.run.smithi099.stdout: 2023-05-31T19:48:29.286 INFO:teuthology.orchestra.run.smithi099.stdout: bdev debug aio = true 2023-05-31T19:48:29.286 INFO:teuthology.orchestra.run.smithi099.stdout: osd debug misdirected ops = true 2023-05-31T19:48:29.286 INFO:teuthology.orchestra.run.smithi099.stdout: debug ms = 1 2023-05-31T19:48:29.286 INFO:teuthology.orchestra.run.smithi099.stdout: debug osd = 20 2023-05-31T19:48:29.286 INFO:teuthology.orchestra.run.smithi099.stdout: rocksdb delete range threshold = 0 2023-05-31T19:48:29.286 INFO:teuthology.orchestra.run.smithi099.stdout: 2023-05-31T19:48:29.287 INFO:teuthology.orchestra.run.smithi099.stdout:[mgr] 2023-05-31T19:48:29.287 INFO:teuthology.orchestra.run.smithi099.stdout: debug ms = 1 2023-05-31T19:48:29.287 INFO:teuthology.orchestra.run.smithi099.stdout: debug mgr = 20 2023-05-31T19:48:29.287 INFO:teuthology.orchestra.run.smithi099.stdout: debug mon = 20 2023-05-31T19:48:29.287 INFO:teuthology.orchestra.run.smithi099.stdout: debug auth = 20 2023-05-31T19:48:29.287 INFO:teuthology.orchestra.run.smithi099.stdout: mon reweight min pgs per osd = 4 2023-05-31T19:48:29.287 INFO:teuthology.orchestra.run.smithi099.stdout: mon reweight min bytes per osd = 10 2023-05-31T19:48:29.287 INFO:teuthology.orchestra.run.smithi099.stdout: mgr/telemetry/nag = false 2023-05-31T19:48:29.288 INFO:teuthology.orchestra.run.smithi099.stdout: 2023-05-31T19:48:29.288 INFO:teuthology.orchestra.run.smithi099.stdout:[mon] 2023-05-31T19:48:29.288 INFO:teuthology.orchestra.run.smithi099.stdout: debug ms = 1 2023-05-31T19:48:29.288 INFO:teuthology.orchestra.run.smithi099.stdout: debug mon = 20 2023-05-31T19:48:29.288 INFO:teuthology.orchestra.run.smithi099.stdout: debug paxos = 20 2023-05-31T19:48:29.288 INFO:teuthology.orchestra.run.smithi099.stdout: debug auth = 20 2023-05-31T19:48:29.288 INFO:teuthology.orchestra.run.smithi099.stdout: mon data avail warn = 5 2023-05-31T19:48:29.289 INFO:teuthology.orchestra.run.smithi099.stdout: mon mgr mkfs grace = 240 2023-05-31T19:48:29.289 INFO:teuthology.orchestra.run.smithi099.stdout: mon reweight min pgs per osd = 4 2023-05-31T19:48:29.289 INFO:teuthology.orchestra.run.smithi099.stdout: mon osd reporter subtree level = osd 2023-05-31T19:48:29.289 INFO:teuthology.orchestra.run.smithi099.stdout: mon osd prime pg temp = true 2023-05-31T19:48:29.289 INFO:teuthology.orchestra.run.smithi099.stdout: mon reweight min bytes per osd = 10 2023-05-31T19:48:29.289 INFO:teuthology.orchestra.run.smithi099.stdout: 2023-05-31T19:48:29.289 INFO:teuthology.orchestra.run.smithi099.stdout: # rotate auth tickets quickly to exercise renewal paths 2023-05-31T19:48:29.289 INFO:teuthology.orchestra.run.smithi099.stdout: auth mon ticket ttl = 660 # 11m 2023-05-31T19:48:29.290 INFO:teuthology.orchestra.run.smithi099.stdout: auth service ticket ttl = 240 # 4m 2023-05-31T19:48:29.290 INFO:teuthology.orchestra.run.smithi099.stdout: 2023-05-31T19:48:29.290 INFO:teuthology.orchestra.run.smithi099.stdout: # don't complain about insecure global_id in the test suite 2023-05-31T19:48:29.290 INFO:teuthology.orchestra.run.smithi099.stdout: mon_warn_on_insecure_global_id_reclaim = false 2023-05-31T19:48:29.290 INFO:teuthology.orchestra.run.smithi099.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2023-05-31T19:48:29.290 INFO:teuthology.orchestra.run.smithi099.stdout: 2023-05-31T19:48:29.290 INFO:teuthology.orchestra.run.smithi099.stdout: # 1m isn't quite enough 2023-05-31T19:48:29.290 INFO:teuthology.orchestra.run.smithi099.stdout: mon_down_mkfs_grace = 2m 2023-05-31T19:48:29.291 INFO:teuthology.orchestra.run.smithi099.stdout: 2023-05-31T19:48:29.291 INFO:teuthology.orchestra.run.smithi099.stdout: mon_warn_on_filestore_osds = false 2023-05-31T19:48:29.291 INFO:teuthology.orchestra.run.smithi099.stdout: 2023-05-31T19:48:29.291 INFO:teuthology.orchestra.run.smithi099.stdout:[client] 2023-05-31T19:48:29.291 INFO:teuthology.orchestra.run.smithi099.stdout: rgw cache enabled = true 2023-05-31T19:48:29.291 INFO:teuthology.orchestra.run.smithi099.stdout: rgw enable ops log = true 2023-05-31T19:48:29.291 INFO:teuthology.orchestra.run.smithi099.stdout: rgw enable usage log = true 2023-05-31T19:48:29.291 INFO:teuthology.orchestra.run.smithi099.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2023-05-31T19:48:29.292 INFO:teuthology.orchestra.run.smithi099.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2023-05-31T19:48:29.292 INFO:teuthology.orchestra.run.smithi099.stdout: debug rgw = 20 2023-05-31T19:48:29.292 INFO:teuthology.orchestra.run.smithi099.stdout: debug rgw sync = 20 2023-05-31T19:48:29.292 INFO:teuthology.orchestra.run.smithi099.stdout: rgw crypt require ssl = False 2023-05-31T19:48:29.292 INFO:teuthology.orchestra.run.smithi099.stdout: rgw crypt s3 kms backend = testing 2023-05-31T19:48:29.292 INFO:teuthology.orchestra.run.smithi099.stdout: rgw crypt s3 kms encryption keys = testkey-1=YmluCmJvb3N0CmJvb3N0LWJ1aWxkCmNlcGguY29uZgo= 2023-05-31T19:48:29.292 INFO:teuthology.orchestra.run.smithi099.stdout: rgw curl low speed time = 300 2023-05-31T19:48:29.292 INFO:teuthology.orchestra.run.smithi099.stdout: rgw data log num shards = 4 2023-05-31T19:48:29.293 INFO:teuthology.orchestra.run.smithi099.stdout: rgw md log max shards = 4 2023-05-31T19:48:29.293 INFO:teuthology.orchestra.run.smithi099.stdout: rgw sync data inject err probability = 0.1 2023-05-31T19:48:29.293 INFO:teuthology.orchestra.run.smithi099.stdout: rgw sync log trim interval = 0 2023-05-31T19:48:29.293 INFO:teuthology.orchestra.run.smithi099.stdout: rgw sync meta inject err probability = 0.1 2023-05-31T19:48:29.293 INFO:teuthology.orchestra.run.smithi099.stdout: rgw sync obj etag verify = True 2023-05-31T19:48:29.293 INFO:teuthology.orchestra.run.smithi099.stdout: setgroup = ceph 2023-05-31T19:48:29.293 INFO:teuthology.orchestra.run.smithi099.stdout: setuser = ceph 2023-05-31T19:48:29.293 INFO:teuthology.orchestra.run.smithi099.stdout:[mon.a] 2023-05-31T19:48:29.294 INFO:teuthology.orchestra.run.smithi099.stdout:[client.0] 2023-05-31T19:48:29.294 INFO:teuthology.orchestra.run.smithi099.stdout: rgw data notify interval msec = 0 2023-05-31T19:48:29.294 INFO:teuthology.orchestra.run.smithi099.stdout:[osd.0] 2023-05-31T19:48:29.294 INFO:teuthology.orchestra.run.smithi099.stdout: osd_max_omap_entries_per_request = 10 2023-05-31T19:48:29.294 INFO:teuthology.orchestra.run.smithi099.stdout:[osd.1] 2023-05-31T19:48:29.294 INFO:teuthology.orchestra.run.smithi099.stdout: osd_max_omap_entries_per_request = 1000 2023-05-31T19:48:29.294 INFO:teuthology.orchestra.run.smithi099.stdout:[osd.2] 2023-05-31T19:48:29.294 INFO:teuthology.orchestra.run.smithi099.stdout: osd_max_omap_entries_per_request = 10000 2023-05-31T19:48:29.305 INFO:teuthology.orchestra.run.smithi112.stdout:[global] 2023-05-31T19:48:29.305 INFO:teuthology.orchestra.run.smithi112.stdout: chdir = "" 2023-05-31T19:48:29.305 INFO:teuthology.orchestra.run.smithi112.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2023-05-31T19:48:29.306 INFO:teuthology.orchestra.run.smithi112.stdout: auth supported = cephx 2023-05-31T19:48:29.306 INFO:teuthology.orchestra.run.smithi112.stdout: 2023-05-31T19:48:29.306 INFO:teuthology.orchestra.run.smithi112.stdout: filestore xattr use omap = true 2023-05-31T19:48:29.306 INFO:teuthology.orchestra.run.smithi112.stdout: 2023-05-31T19:48:29.306 INFO:teuthology.orchestra.run.smithi112.stdout: mon clock drift allowed = 1.000 2023-05-31T19:48:29.306 INFO:teuthology.orchestra.run.smithi112.stdout: 2023-05-31T19:48:29.306 INFO:teuthology.orchestra.run.smithi112.stdout: osd crush chooseleaf type = 0 2023-05-31T19:48:29.307 INFO:teuthology.orchestra.run.smithi112.stdout: auth debug = true 2023-05-31T19:48:29.307 INFO:teuthology.orchestra.run.smithi112.stdout: 2023-05-31T19:48:29.307 INFO:teuthology.orchestra.run.smithi112.stdout: ms die on old message = true 2023-05-31T19:48:29.307 INFO:teuthology.orchestra.run.smithi112.stdout: ms die on bug = true 2023-05-31T19:48:29.307 INFO:teuthology.orchestra.run.smithi112.stdout: 2023-05-31T19:48:29.307 INFO:teuthology.orchestra.run.smithi112.stdout: mon max pg per osd = 10000 # >= luminous 2023-05-31T19:48:29.307 INFO:teuthology.orchestra.run.smithi112.stdout: mon pg warn max object skew = 0 2023-05-31T19:48:29.308 INFO:teuthology.orchestra.run.smithi112.stdout: 2023-05-31T19:48:29.308 INFO:teuthology.orchestra.run.smithi112.stdout: # disable pg_autoscaler by default for new pools 2023-05-31T19:48:29.308 INFO:teuthology.orchestra.run.smithi112.stdout: osd_pool_default_pg_autoscale_mode = off 2023-05-31T19:48:29.308 INFO:teuthology.orchestra.run.smithi112.stdout: 2023-05-31T19:48:29.308 INFO:teuthology.orchestra.run.smithi112.stdout: osd pool default size = 2 2023-05-31T19:48:29.308 INFO:teuthology.orchestra.run.smithi112.stdout: 2023-05-31T19:48:29.308 INFO:teuthology.orchestra.run.smithi112.stdout: mon osd allow primary affinity = true 2023-05-31T19:48:29.309 INFO:teuthology.orchestra.run.smithi112.stdout: mon osd allow pg remap = true 2023-05-31T19:48:29.309 INFO:teuthology.orchestra.run.smithi112.stdout: mon warn on legacy crush tunables = false 2023-05-31T19:48:29.309 INFO:teuthology.orchestra.run.smithi112.stdout: mon warn on crush straw calc version zero = false 2023-05-31T19:48:29.309 INFO:teuthology.orchestra.run.smithi112.stdout: mon warn on no sortbitwise = false 2023-05-31T19:48:29.309 INFO:teuthology.orchestra.run.smithi112.stdout: mon warn on osd down out interval zero = false 2023-05-31T19:48:29.309 INFO:teuthology.orchestra.run.smithi112.stdout: mon warn on too few osds = false 2023-05-31T19:48:29.309 INFO:teuthology.orchestra.run.smithi112.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2023-05-31T19:48:29.309 INFO:teuthology.orchestra.run.smithi112.stdout: mon_warn_on_pool_no_redundancy = false 2023-05-31T19:48:29.310 INFO:teuthology.orchestra.run.smithi112.stdout: mon_allow_pool_size_one = true 2023-05-31T19:48:29.310 INFO:teuthology.orchestra.run.smithi112.stdout: 2023-05-31T19:48:29.310 INFO:teuthology.orchestra.run.smithi112.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2023-05-31T19:48:29.310 INFO:teuthology.orchestra.run.smithi112.stdout: 2023-05-31T19:48:29.310 INFO:teuthology.orchestra.run.smithi112.stdout: osd default data pool replay window = 5 2023-05-31T19:48:29.310 INFO:teuthology.orchestra.run.smithi112.stdout: 2023-05-31T19:48:29.310 INFO:teuthology.orchestra.run.smithi112.stdout: mon allow pool delete = true 2023-05-31T19:48:29.310 INFO:teuthology.orchestra.run.smithi112.stdout: 2023-05-31T19:48:29.311 INFO:teuthology.orchestra.run.smithi112.stdout: mon cluster log file level = debug 2023-05-31T19:48:29.311 INFO:teuthology.orchestra.run.smithi112.stdout: debug asserts on shutdown = true 2023-05-31T19:48:29.311 INFO:teuthology.orchestra.run.smithi112.stdout: mon health detail to clog = false 2023-05-31T19:48:29.311 INFO:teuthology.orchestra.run.smithi112.stdout: mon host = 172.21.15.112 2023-05-31T19:48:29.311 INFO:teuthology.orchestra.run.smithi112.stdout: fsid = ff2a32a9-2a3b-4db6-ad15-e0ad1a679cca 2023-05-31T19:48:29.311 INFO:teuthology.orchestra.run.smithi112.stdout: 2023-05-31T19:48:29.311 INFO:teuthology.orchestra.run.smithi112.stdout:[osd] 2023-05-31T19:48:29.311 INFO:teuthology.orchestra.run.smithi112.stdout: osd journal size = 100 2023-05-31T19:48:29.312 INFO:teuthology.orchestra.run.smithi112.stdout: 2023-05-31T19:48:29.312 INFO:teuthology.orchestra.run.smithi112.stdout: osd scrub load threshold = 5.0 2023-05-31T19:48:29.312 INFO:teuthology.orchestra.run.smithi112.stdout: osd scrub max interval = 600 2023-05-31T19:48:29.312 INFO:teuthology.orchestra.run.smithi112.stdout: osd mclock profile = high_recovery_ops 2023-05-31T19:48:29.312 INFO:teuthology.orchestra.run.smithi112.stdout: 2023-05-31T19:48:29.312 INFO:teuthology.orchestra.run.smithi112.stdout: osd recover clone overlap = true 2023-05-31T19:48:29.312 INFO:teuthology.orchestra.run.smithi112.stdout: osd recovery max chunk = 1048576 2023-05-31T19:48:29.313 INFO:teuthology.orchestra.run.smithi112.stdout: 2023-05-31T19:48:29.313 INFO:teuthology.orchestra.run.smithi112.stdout: osd debug shutdown = true 2023-05-31T19:48:29.313 INFO:teuthology.orchestra.run.smithi112.stdout: osd debug op order = true 2023-05-31T19:48:29.313 INFO:teuthology.orchestra.run.smithi112.stdout: osd debug verify stray on activate = true 2023-05-31T19:48:29.313 INFO:teuthology.orchestra.run.smithi112.stdout: 2023-05-31T19:48:29.313 INFO:teuthology.orchestra.run.smithi112.stdout: osd open classes on start = true 2023-05-31T19:48:29.313 INFO:teuthology.orchestra.run.smithi112.stdout: osd debug pg log writeout = true 2023-05-31T19:48:29.313 INFO:teuthology.orchestra.run.smithi112.stdout: 2023-05-31T19:48:29.314 INFO:teuthology.orchestra.run.smithi112.stdout: osd deep scrub update digest min age = 30 2023-05-31T19:48:29.314 INFO:teuthology.orchestra.run.smithi112.stdout: 2023-05-31T19:48:29.314 INFO:teuthology.orchestra.run.smithi112.stdout: osd map max advance = 10 2023-05-31T19:48:29.314 INFO:teuthology.orchestra.run.smithi112.stdout: 2023-05-31T19:48:29.314 INFO:teuthology.orchestra.run.smithi112.stdout: journal zero on create = true 2023-05-31T19:48:29.314 INFO:teuthology.orchestra.run.smithi112.stdout: 2023-05-31T19:48:29.314 INFO:teuthology.orchestra.run.smithi112.stdout: filestore ondisk finisher threads = 3 2023-05-31T19:48:29.314 INFO:teuthology.orchestra.run.smithi112.stdout: filestore apply finisher threads = 3 2023-05-31T19:48:29.315 INFO:teuthology.orchestra.run.smithi112.stdout: 2023-05-31T19:48:29.315 INFO:teuthology.orchestra.run.smithi112.stdout: bdev debug aio = true 2023-05-31T19:48:29.315 INFO:teuthology.orchestra.run.smithi112.stdout: osd debug misdirected ops = true 2023-05-31T19:48:29.315 INFO:teuthology.orchestra.run.smithi112.stdout: debug ms = 1 2023-05-31T19:48:29.315 INFO:teuthology.orchestra.run.smithi112.stdout: debug osd = 20 2023-05-31T19:48:29.315 INFO:teuthology.orchestra.run.smithi112.stdout: rocksdb delete range threshold = 0 2023-05-31T19:48:29.315 INFO:teuthology.orchestra.run.smithi112.stdout: 2023-05-31T19:48:29.315 INFO:teuthology.orchestra.run.smithi112.stdout:[mgr] 2023-05-31T19:48:29.316 INFO:teuthology.orchestra.run.smithi112.stdout: debug ms = 1 2023-05-31T19:48:29.316 INFO:teuthology.orchestra.run.smithi112.stdout: debug mgr = 20 2023-05-31T19:48:29.316 INFO:teuthology.orchestra.run.smithi112.stdout: debug mon = 20 2023-05-31T19:48:29.316 INFO:teuthology.orchestra.run.smithi112.stdout: debug auth = 20 2023-05-31T19:48:29.316 INFO:teuthology.orchestra.run.smithi112.stdout: mon reweight min pgs per osd = 4 2023-05-31T19:48:29.316 INFO:teuthology.orchestra.run.smithi112.stdout: mon reweight min bytes per osd = 10 2023-05-31T19:48:29.316 INFO:teuthology.orchestra.run.smithi112.stdout: mgr/telemetry/nag = false 2023-05-31T19:48:29.317 INFO:teuthology.orchestra.run.smithi112.stdout: 2023-05-31T19:48:29.317 INFO:teuthology.orchestra.run.smithi112.stdout:[mon] 2023-05-31T19:48:29.317 INFO:teuthology.orchestra.run.smithi112.stdout: debug ms = 1 2023-05-31T19:48:29.317 INFO:teuthology.orchestra.run.smithi112.stdout: debug mon = 20 2023-05-31T19:48:29.317 INFO:teuthology.orchestra.run.smithi112.stdout: debug paxos = 20 2023-05-31T19:48:29.317 INFO:teuthology.orchestra.run.smithi112.stdout: debug auth = 20 2023-05-31T19:48:29.317 INFO:teuthology.orchestra.run.smithi112.stdout: mon data avail warn = 5 2023-05-31T19:48:29.317 INFO:teuthology.orchestra.run.smithi112.stdout: mon mgr mkfs grace = 240 2023-05-31T19:48:29.318 INFO:teuthology.orchestra.run.smithi112.stdout: mon reweight min pgs per osd = 4 2023-05-31T19:48:29.318 INFO:teuthology.orchestra.run.smithi112.stdout: mon osd reporter subtree level = osd 2023-05-31T19:48:29.318 INFO:teuthology.orchestra.run.smithi112.stdout: mon osd prime pg temp = true 2023-05-31T19:48:29.318 INFO:teuthology.orchestra.run.smithi112.stdout: mon reweight min bytes per osd = 10 2023-05-31T19:48:29.318 INFO:teuthology.orchestra.run.smithi112.stdout: 2023-05-31T19:48:29.318 INFO:teuthology.orchestra.run.smithi112.stdout: # rotate auth tickets quickly to exercise renewal paths 2023-05-31T19:48:29.318 INFO:teuthology.orchestra.run.smithi112.stdout: auth mon ticket ttl = 660 # 11m 2023-05-31T19:48:29.318 INFO:teuthology.orchestra.run.smithi112.stdout: auth service ticket ttl = 240 # 4m 2023-05-31T19:48:29.319 INFO:teuthology.orchestra.run.smithi112.stdout: 2023-05-31T19:48:29.319 INFO:teuthology.orchestra.run.smithi112.stdout: # don't complain about insecure global_id in the test suite 2023-05-31T19:48:29.319 INFO:teuthology.orchestra.run.smithi112.stdout: mon_warn_on_insecure_global_id_reclaim = false 2023-05-31T19:48:29.319 INFO:teuthology.orchestra.run.smithi112.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2023-05-31T19:48:29.319 INFO:teuthology.orchestra.run.smithi112.stdout: 2023-05-31T19:48:29.319 INFO:teuthology.orchestra.run.smithi112.stdout: # 1m isn't quite enough 2023-05-31T19:48:29.319 INFO:teuthology.orchestra.run.smithi112.stdout: mon_down_mkfs_grace = 2m 2023-05-31T19:48:29.320 INFO:teuthology.orchestra.run.smithi112.stdout: 2023-05-31T19:48:29.320 INFO:teuthology.orchestra.run.smithi112.stdout: mon_warn_on_filestore_osds = false 2023-05-31T19:48:29.320 INFO:teuthology.orchestra.run.smithi112.stdout: 2023-05-31T19:48:29.320 INFO:teuthology.orchestra.run.smithi112.stdout:[client] 2023-05-31T19:48:29.320 INFO:teuthology.orchestra.run.smithi112.stdout: rgw cache enabled = true 2023-05-31T19:48:29.320 INFO:teuthology.orchestra.run.smithi112.stdout: rgw enable ops log = true 2023-05-31T19:48:29.320 INFO:teuthology.orchestra.run.smithi112.stdout: rgw enable usage log = true 2023-05-31T19:48:29.320 INFO:teuthology.orchestra.run.smithi112.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2023-05-31T19:48:29.321 INFO:teuthology.orchestra.run.smithi112.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2023-05-31T19:48:29.321 INFO:teuthology.orchestra.run.smithi112.stdout: debug rgw = 20 2023-05-31T19:48:29.321 INFO:teuthology.orchestra.run.smithi112.stdout: debug rgw sync = 20 2023-05-31T19:48:29.321 INFO:teuthology.orchestra.run.smithi112.stdout: rgw crypt require ssl = False 2023-05-31T19:48:29.321 INFO:teuthology.orchestra.run.smithi112.stdout: rgw crypt s3 kms backend = testing 2023-05-31T19:48:29.321 INFO:teuthology.orchestra.run.smithi112.stdout: rgw crypt s3 kms encryption keys = testkey-1=YmluCmJvb3N0CmJvb3N0LWJ1aWxkCmNlcGguY29uZgo= 2023-05-31T19:48:29.321 INFO:teuthology.orchestra.run.smithi112.stdout: rgw curl low speed time = 300 2023-05-31T19:48:29.322 INFO:teuthology.orchestra.run.smithi112.stdout: rgw data log num shards = 4 2023-05-31T19:48:29.322 INFO:teuthology.orchestra.run.smithi112.stdout: rgw md log max shards = 4 2023-05-31T19:48:29.322 INFO:teuthology.orchestra.run.smithi112.stdout: rgw sync data inject err probability = 0.1 2023-05-31T19:48:29.322 INFO:teuthology.orchestra.run.smithi112.stdout: rgw sync log trim interval = 0 2023-05-31T19:48:29.322 INFO:teuthology.orchestra.run.smithi112.stdout: rgw sync meta inject err probability = 0.1 2023-05-31T19:48:29.322 INFO:teuthology.orchestra.run.smithi112.stdout: rgw sync obj etag verify = True 2023-05-31T19:48:29.322 INFO:teuthology.orchestra.run.smithi112.stdout: setgroup = ceph 2023-05-31T19:48:29.322 INFO:teuthology.orchestra.run.smithi112.stdout: setuser = ceph 2023-05-31T19:48:29.323 INFO:teuthology.orchestra.run.smithi112.stdout:[mon.a] 2023-05-31T19:48:29.323 INFO:teuthology.orchestra.run.smithi112.stdout:[client.0] 2023-05-31T19:48:29.323 INFO:teuthology.orchestra.run.smithi112.stdout: rgw data notify interval msec = 0 2023-05-31T19:48:29.323 INFO:teuthology.orchestra.run.smithi112.stdout:[osd.0] 2023-05-31T19:48:29.323 INFO:teuthology.orchestra.run.smithi112.stdout: osd_max_omap_entries_per_request = 10 2023-05-31T19:48:29.323 INFO:teuthology.orchestra.run.smithi112.stdout:[osd.1] 2023-05-31T19:48:29.323 INFO:teuthology.orchestra.run.smithi112.stdout: osd_max_omap_entries_per_request = 1000 2023-05-31T19:48:29.323 INFO:teuthology.orchestra.run.smithi112.stdout:[osd.2] 2023-05-31T19:48:29.324 INFO:teuthology.orchestra.run.smithi112.stdout: osd_max_omap_entries_per_request = 10000 2023-05-31T19:48:29.325 INFO:tasks.ceph:Creating admin key on c2.mon.a... 2023-05-31T19:48:29.325 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=client.admin --cap mon 'allow *' --cap osd 'allow *' --cap mds 'allow *' --cap mgr 'allow *' /etc/ceph/c2.keyring 2023-05-31T19:48:29.423 INFO:tasks.ceph:Copying monmap to all nodes... 2023-05-31T19:48:29.423 DEBUG:teuthology.orchestra.run.smithi112:> set -ex 2023-05-31T19:48:29.423 DEBUG:teuthology.orchestra.run.smithi112:> dd if=/etc/ceph/c2.keyring of=/dev/stdout 2023-05-31T19:48:29.442 DEBUG:teuthology.orchestra.run.smithi112:> set -ex 2023-05-31T19:48:29.442 DEBUG:teuthology.orchestra.run.smithi112:> dd if=/home/ubuntu/cephtest/c2.monmap of=/dev/stdout 2023-05-31T19:48:29.499 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi099.front.sepia.ceph.com 2023-05-31T19:48:29.500 DEBUG:teuthology.orchestra.run.smithi099:> set -ex 2023-05-31T19:48:29.500 DEBUG:teuthology.orchestra.run.smithi099:> sudo dd of=/etc/ceph/c2.keyring 2023-05-31T19:48:29.500 DEBUG:teuthology.orchestra.run.smithi099:> sudo chmod 0644 /etc/ceph/c2.keyring 2023-05-31T19:48:29.545 DEBUG:teuthology.orchestra.run.smithi099:> set -ex 2023-05-31T19:48:29.545 DEBUG:teuthology.orchestra.run.smithi099:> dd of=/home/ubuntu/cephtest/c2.monmap 2023-05-31T19:48:29.604 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi112.front.sepia.ceph.com 2023-05-31T19:48:29.605 DEBUG:teuthology.orchestra.run.smithi112:> set -ex 2023-05-31T19:48:29.605 DEBUG:teuthology.orchestra.run.smithi112:> sudo dd of=/etc/ceph/c2.keyring 2023-05-31T19:48:29.605 DEBUG:teuthology.orchestra.run.smithi112:> sudo chmod 0644 /etc/ceph/c2.keyring 2023-05-31T19:48:29.641 DEBUG:teuthology.orchestra.run.smithi112:> set -ex 2023-05-31T19:48:29.641 DEBUG:teuthology.orchestra.run.smithi112:> dd of=/home/ubuntu/cephtest/c2.monmap 2023-05-31T19:48:29.699 INFO:tasks.ceph:Setting up mon nodes... 2023-05-31T19:48:29.699 INFO:tasks.ceph:Setting up mgr nodes... 2023-05-31T19:48:29.700 DEBUG:teuthology.orchestra.run.smithi112:> sudo mkdir -p /var/lib/ceph/mgr/c2-x && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mgr.x /var/lib/ceph/mgr/c2-x/keyring 2023-05-31T19:48:29.793 INFO:teuthology.orchestra.run.smithi112.stdout:creating /var/lib/ceph/mgr/c2-x/keyring 2023-05-31T19:48:29.795 INFO:tasks.ceph:Setting up mds nodes... 2023-05-31T19:48:29.796 INFO:tasks.ceph_client:Setting up client nodes... 2023-05-31T19:48:29.796 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.0 /etc/ceph/c2.client.0.keyring && sudo chmod 0644 /etc/ceph/c2.client.0.keyring 2023-05-31T19:48:29.840 INFO:teuthology.orchestra.run.smithi112.stdout:creating /etc/ceph/c2.client.0.keyring 2023-05-31T19:48:29.854 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.1 /etc/ceph/c2.client.1.keyring && sudo chmod 0644 /etc/ceph/c2.client.1.keyring 2023-05-31T19:48:29.937 INFO:teuthology.orchestra.run.smithi112.stdout:creating /etc/ceph/c2.client.1.keyring 2023-05-31T19:48:29.949 INFO:tasks.ceph:Running mkfs on osd nodes... 2023-05-31T19:48:29.950 INFO:tasks.ceph:ctx.disk_config.remote_to_roles_to_dev: {Remote(name='ubuntu@smithi099.front.sepia.ceph.com'): {'c1.osd.0': '/dev/vg_nvme/lv_1', 'c1.osd.1': '/dev/vg_nvme/lv_2', 'c1.osd.2': '/dev/vg_nvme/lv_3'}, Remote(name='ubuntu@smithi112.front.sepia.ceph.com'): {'c2.osd.0': '/dev/vg_nvme/lv_1', 'c2.osd.1': '/dev/vg_nvme/lv_2', 'c2.osd.2': '/dev/vg_nvme/lv_3'}} 2023-05-31T19:48:29.950 DEBUG:teuthology.orchestra.run.smithi112:> sudo mkdir -p /var/lib/ceph/osd/c2-0 2023-05-31T19:48:29.976 INFO:tasks.ceph:roles_to_devs: {'c2.osd.0': '/dev/vg_nvme/lv_1', 'c2.osd.1': '/dev/vg_nvme/lv_2', 'c2.osd.2': '/dev/vg_nvme/lv_3'} 2023-05-31T19:48:29.977 INFO:tasks.ceph:role: c2.osd.0 2023-05-31T19:48:29.977 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_1 on ubuntu@smithi112.front.sepia.ceph.com 2023-05-31T19:48:29.977 DEBUG:teuthology.orchestra.run.smithi112:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_1 2023-05-31T19:48:30.051 INFO:teuthology.orchestra.run.smithi112.stdout:meta-data=/dev/vg_nvme/lv_1 isize=2048 agcount=4, agsize=5859072 blks 2023-05-31T19:48:30.052 INFO:teuthology.orchestra.run.smithi112.stdout: = sectsz=512 attr=2, projid32bit=1 2023-05-31T19:48:30.052 INFO:teuthology.orchestra.run.smithi112.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2023-05-31T19:48:30.052 INFO:teuthology.orchestra.run.smithi112.stdout: = reflink=1 bigtime=0 inobtcount=0 2023-05-31T19:48:30.052 INFO:teuthology.orchestra.run.smithi112.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2023-05-31T19:48:30.052 INFO:teuthology.orchestra.run.smithi112.stdout: = sunit=0 swidth=0 blks 2023-05-31T19:48:30.052 INFO:teuthology.orchestra.run.smithi112.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2023-05-31T19:48:30.053 INFO:teuthology.orchestra.run.smithi112.stdout:log =internal log bsize=4096 blocks=11443, version=2 2023-05-31T19:48:30.053 INFO:teuthology.orchestra.run.smithi112.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2023-05-31T19:48:30.053 INFO:teuthology.orchestra.run.smithi112.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2023-05-31T19:48:30.480 INFO:teuthology.orchestra.run.smithi112.stdout:Discarding blocks...Done. 2023-05-31T19:48:30.482 INFO:tasks.ceph:mount /dev/vg_nvme/lv_1 on ubuntu@smithi112.front.sepia.ceph.com -o noatime 2023-05-31T19:48:30.483 DEBUG:teuthology.orchestra.run.smithi112:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_1 /var/lib/ceph/osd/c2-0 2023-05-31T19:48:30.646 DEBUG:teuthology.orchestra.run.smithi112:> sudo /sbin/restorecon /var/lib/ceph/osd/c2-0 2023-05-31T19:48:30.696 DEBUG:teuthology.orchestra.run.smithi112:> sudo mkdir -p /var/lib/ceph/osd/c2-1 2023-05-31T19:48:30.766 INFO:tasks.ceph:roles_to_devs: {'c2.osd.0': '/dev/vg_nvme/lv_1', 'c2.osd.1': '/dev/vg_nvme/lv_2', 'c2.osd.2': '/dev/vg_nvme/lv_3'} 2023-05-31T19:48:30.766 INFO:tasks.ceph:role: c2.osd.1 2023-05-31T19:48:30.766 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_2 on ubuntu@smithi112.front.sepia.ceph.com 2023-05-31T19:48:30.767 DEBUG:teuthology.orchestra.run.smithi112:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_2 2023-05-31T19:48:30.842 INFO:teuthology.orchestra.run.smithi112.stdout:meta-data=/dev/vg_nvme/lv_2 isize=2048 agcount=4, agsize=5859072 blks 2023-05-31T19:48:30.842 INFO:teuthology.orchestra.run.smithi112.stdout: = sectsz=512 attr=2, projid32bit=1 2023-05-31T19:48:30.843 INFO:teuthology.orchestra.run.smithi112.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2023-05-31T19:48:30.843 INFO:teuthology.orchestra.run.smithi112.stdout: = reflink=1 bigtime=0 inobtcount=0 2023-05-31T19:48:30.843 INFO:teuthology.orchestra.run.smithi112.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2023-05-31T19:48:30.844 INFO:teuthology.orchestra.run.smithi112.stdout: = sunit=0 swidth=0 blks 2023-05-31T19:48:30.844 INFO:teuthology.orchestra.run.smithi112.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2023-05-31T19:48:30.844 INFO:teuthology.orchestra.run.smithi112.stdout:log =internal log bsize=4096 blocks=11443, version=2 2023-05-31T19:48:30.844 INFO:teuthology.orchestra.run.smithi112.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2023-05-31T19:48:30.845 INFO:teuthology.orchestra.run.smithi112.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2023-05-31T19:48:31.269 INFO:teuthology.orchestra.run.smithi112.stdout:Discarding blocks...Done. 2023-05-31T19:48:31.271 INFO:tasks.ceph:mount /dev/vg_nvme/lv_2 on ubuntu@smithi112.front.sepia.ceph.com -o noatime 2023-05-31T19:48:31.272 DEBUG:teuthology.orchestra.run.smithi112:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_2 /var/lib/ceph/osd/c2-1 2023-05-31T19:48:31.386 DEBUG:teuthology.orchestra.run.smithi112:> sudo /sbin/restorecon /var/lib/ceph/osd/c2-1 2023-05-31T19:48:31.423 DEBUG:teuthology.orchestra.run.smithi112:> sudo mkdir -p /var/lib/ceph/osd/c2-2 2023-05-31T19:48:31.491 INFO:tasks.ceph:roles_to_devs: {'c2.osd.0': '/dev/vg_nvme/lv_1', 'c2.osd.1': '/dev/vg_nvme/lv_2', 'c2.osd.2': '/dev/vg_nvme/lv_3'} 2023-05-31T19:48:31.491 INFO:tasks.ceph:role: c2.osd.2 2023-05-31T19:48:31.491 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_3 on ubuntu@smithi112.front.sepia.ceph.com 2023-05-31T19:48:31.492 DEBUG:teuthology.orchestra.run.smithi112:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_3 2023-05-31T19:48:31.566 INFO:teuthology.orchestra.run.smithi112.stdout:meta-data=/dev/vg_nvme/lv_3 isize=2048 agcount=4, agsize=5859072 blks 2023-05-31T19:48:31.566 INFO:teuthology.orchestra.run.smithi112.stdout: = sectsz=512 attr=2, projid32bit=1 2023-05-31T19:48:31.569 INFO:teuthology.orchestra.run.smithi112.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2023-05-31T19:48:31.569 INFO:teuthology.orchestra.run.smithi112.stdout: = reflink=1 bigtime=0 inobtcount=0 2023-05-31T19:48:31.570 INFO:teuthology.orchestra.run.smithi112.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2023-05-31T19:48:31.570 INFO:teuthology.orchestra.run.smithi112.stdout: = sunit=0 swidth=0 blks 2023-05-31T19:48:31.570 INFO:teuthology.orchestra.run.smithi112.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2023-05-31T19:48:31.570 INFO:teuthology.orchestra.run.smithi112.stdout:log =internal log bsize=4096 blocks=11443, version=2 2023-05-31T19:48:31.570 INFO:teuthology.orchestra.run.smithi112.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2023-05-31T19:48:31.571 INFO:teuthology.orchestra.run.smithi112.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2023-05-31T19:48:32.015 INFO:teuthology.orchestra.run.smithi112.stdout:Discarding blocks...Done. 2023-05-31T19:48:32.016 INFO:tasks.ceph:mount /dev/vg_nvme/lv_3 on ubuntu@smithi112.front.sepia.ceph.com -o noatime 2023-05-31T19:48:32.017 DEBUG:teuthology.orchestra.run.smithi112:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_3 /var/lib/ceph/osd/c2-2 2023-05-31T19:48:32.180 DEBUG:teuthology.orchestra.run.smithi112:> sudo /sbin/restorecon /var/lib/ceph/osd/c2-2 2023-05-31T19:48:32.215 DEBUG:teuthology.orchestra.run.smithi112:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster c2 --mkfs --mkkey -i 0 --monmap /home/ubuntu/cephtest/c2.monmap 2023-05-31T19:48:32.323 INFO:teuthology.orchestra.run.smithi112.stderr:2023-05-31T19:48:32.317+0000 7f1bf0edb640 -1 auth: error reading file: /var/lib/ceph/osd/c2-0/keyring: can't open /var/lib/ceph/osd/c2-0/keyring: (2) No such file or directory 2023-05-31T19:48:32.325 INFO:teuthology.orchestra.run.smithi112.stderr:2023-05-31T19:48:32.319+0000 7f1bf0edb640 -1 created new key in keyring /var/lib/ceph/osd/c2-0/keyring 2023-05-31T19:48:32.325 INFO:teuthology.orchestra.run.smithi112.stderr:2023-05-31T19:48:32.319+0000 7f1bf0edb640 -1 bluestore(/var/lib/ceph/osd/c2-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/c2-0/block: (2) No such file or directory 2023-05-31T19:48:32.325 INFO:teuthology.orchestra.run.smithi112.stderr:2023-05-31T19:48:32.319+0000 7f1bf0edb640 -1 bluestore(/var/lib/ceph/osd/c2-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/c2-0/block: (2) No such file or directory 2023-05-31T19:48:32.325 INFO:teuthology.orchestra.run.smithi112.stderr:2023-05-31T19:48:32.319+0000 7f1bf0edb640 -1 bluestore(/var/lib/ceph/osd/c2-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/c2-0/block: (2) No such file or directory 2023-05-31T19:48:32.333 INFO:teuthology.orchestra.run.smithi112.stderr:2023-05-31T19:48:32.326+0000 7f1bf0edb640 -1 bluestore(/var/lib/ceph/osd/c2-0) _read_fsid unparsable uuid 2023-05-31T19:48:34.415 DEBUG:teuthology.orchestra.run.smithi112:> sudo chown -R ceph:ceph /var/lib/ceph/osd/c2-0 2023-05-31T19:48:34.447 DEBUG:teuthology.orchestra.run.smithi112:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster c2 --mkfs --mkkey -i 1 --monmap /home/ubuntu/cephtest/c2.monmap 2023-05-31T19:48:34.541 INFO:teuthology.orchestra.run.smithi112.stderr:2023-05-31T19:48:34.535+0000 7fb3a69af640 -1 auth: error reading file: /var/lib/ceph/osd/c2-1/keyring: can't open /var/lib/ceph/osd/c2-1/keyring: (2) No such file or directory 2023-05-31T19:48:34.542 INFO:teuthology.orchestra.run.smithi112.stderr:2023-05-31T19:48:34.536+0000 7fb3a69af640 -1 created new key in keyring /var/lib/ceph/osd/c2-1/keyring 2023-05-31T19:48:34.542 INFO:teuthology.orchestra.run.smithi112.stderr:2023-05-31T19:48:34.536+0000 7fb3a69af640 -1 bluestore(/var/lib/ceph/osd/c2-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/c2-1/block: (2) No such file or directory 2023-05-31T19:48:34.542 INFO:teuthology.orchestra.run.smithi112.stderr:2023-05-31T19:48:34.536+0000 7fb3a69af640 -1 bluestore(/var/lib/ceph/osd/c2-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/c2-1/block: (2) No such file or directory 2023-05-31T19:48:34.542 INFO:teuthology.orchestra.run.smithi112.stderr:2023-05-31T19:48:34.536+0000 7fb3a69af640 -1 bluestore(/var/lib/ceph/osd/c2-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/c2-1/block: (2) No such file or directory 2023-05-31T19:48:34.543 INFO:teuthology.orchestra.run.smithi112.stderr:2023-05-31T19:48:34.536+0000 7fb3a69af640 -1 bluestore(/var/lib/ceph/osd/c2-1) _read_fsid unparsable uuid 2023-05-31T19:48:36.621 DEBUG:teuthology.orchestra.run.smithi112:> sudo chown -R ceph:ceph /var/lib/ceph/osd/c2-1 2023-05-31T19:48:36.653 DEBUG:teuthology.orchestra.run.smithi112:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster c2 --mkfs --mkkey -i 2 --monmap /home/ubuntu/cephtest/c2.monmap 2023-05-31T19:48:36.746 INFO:teuthology.orchestra.run.smithi112.stderr:2023-05-31T19:48:36.740+0000 7f9c1b9e5640 -1 auth: error reading file: /var/lib/ceph/osd/c2-2/keyring: can't open /var/lib/ceph/osd/c2-2/keyring: (2) No such file or directory 2023-05-31T19:48:36.746 INFO:teuthology.orchestra.run.smithi112.stderr:2023-05-31T19:48:36.740+0000 7f9c1b9e5640 -1 created new key in keyring /var/lib/ceph/osd/c2-2/keyring 2023-05-31T19:48:36.746 INFO:teuthology.orchestra.run.smithi112.stderr:2023-05-31T19:48:36.740+0000 7f9c1b9e5640 -1 bluestore(/var/lib/ceph/osd/c2-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/c2-2/block: (2) No such file or directory 2023-05-31T19:48:36.747 INFO:teuthology.orchestra.run.smithi112.stderr:2023-05-31T19:48:36.740+0000 7f9c1b9e5640 -1 bluestore(/var/lib/ceph/osd/c2-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/c2-2/block: (2) No such file or directory 2023-05-31T19:48:36.747 INFO:teuthology.orchestra.run.smithi112.stderr:2023-05-31T19:48:36.740+0000 7f9c1b9e5640 -1 bluestore(/var/lib/ceph/osd/c2-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/c2-2/block: (2) No such file or directory 2023-05-31T19:48:36.748 INFO:teuthology.orchestra.run.smithi112.stderr:2023-05-31T19:48:36.741+0000 7f9c1b9e5640 -1 bluestore(/var/lib/ceph/osd/c2-2) _read_fsid unparsable uuid 2023-05-31T19:48:38.822 DEBUG:teuthology.orchestra.run.smithi112:> sudo chown -R ceph:ceph /var/lib/ceph/osd/c2-2 2023-05-31T19:48:38.854 INFO:tasks.ceph:Reading keys from all nodes... 2023-05-31T19:48:38.855 DEBUG:teuthology.orchestra.run.smithi112:> set -ex 2023-05-31T19:48:38.855 DEBUG:teuthology.orchestra.run.smithi112:> sudo dd if=/var/lib/ceph/mgr/c2-x/keyring of=/dev/stdout 2023-05-31T19:48:38.925 DEBUG:teuthology.orchestra.run.smithi112:> set -ex 2023-05-31T19:48:38.926 DEBUG:teuthology.orchestra.run.smithi112:> sudo dd if=/var/lib/ceph/osd/c2-0/keyring of=/dev/stdout 2023-05-31T19:48:38.995 DEBUG:teuthology.orchestra.run.smithi112:> set -ex 2023-05-31T19:48:38.996 DEBUG:teuthology.orchestra.run.smithi112:> sudo dd if=/var/lib/ceph/osd/c2-1/keyring of=/dev/stdout 2023-05-31T19:48:39.069 DEBUG:teuthology.orchestra.run.smithi112:> set -ex 2023-05-31T19:48:39.070 DEBUG:teuthology.orchestra.run.smithi112:> sudo dd if=/var/lib/ceph/osd/c2-2/keyring of=/dev/stdout 2023-05-31T19:48:39.138 DEBUG:teuthology.orchestra.run.smithi112:> set -ex 2023-05-31T19:48:39.138 DEBUG:teuthology.orchestra.run.smithi112:> dd if=/etc/ceph/c2.client.0.keyring of=/dev/stdout 2023-05-31T19:48:39.197 DEBUG:teuthology.orchestra.run.smithi112:> set -ex 2023-05-31T19:48:39.197 DEBUG:teuthology.orchestra.run.smithi112:> dd if=/etc/ceph/c2.client.1.keyring of=/dev/stdout 2023-05-31T19:48:39.255 INFO:tasks.ceph:Adding keys to all mons... 2023-05-31T19:48:39.255 DEBUG:teuthology.orchestra.run.smithi112:> sudo tee -a /etc/ceph/c2.keyring 2023-05-31T19:48:39.320 INFO:teuthology.orchestra.run.smithi112.stdout:[mgr.x] 2023-05-31T19:48:39.321 INFO:teuthology.orchestra.run.smithi112.stdout: key = AQCNpHdkf6UBLxAAYDXSE1H3qPontOPGdq/lgA== 2023-05-31T19:48:39.321 INFO:teuthology.orchestra.run.smithi112.stdout:[osd.0] 2023-05-31T19:48:39.321 INFO:teuthology.orchestra.run.smithi112.stdout: key = AQCQpHdkkqH7EhAAUr5L8zfLMY2RZ56OF/t6pg== 2023-05-31T19:48:39.321 INFO:teuthology.orchestra.run.smithi112.stdout:[osd.1] 2023-05-31T19:48:39.321 INFO:teuthology.orchestra.run.smithi112.stdout: key = AQCSpHdkjIwDIBAASc+HmezmZapRcBTNYxb79g== 2023-05-31T19:48:39.322 INFO:teuthology.orchestra.run.smithi112.stdout:[osd.2] 2023-05-31T19:48:39.322 INFO:teuthology.orchestra.run.smithi112.stdout: key = AQCUpHdkROIzLBAAEPTHV39qM5LZgs2umwLQpw== 2023-05-31T19:48:39.322 INFO:teuthology.orchestra.run.smithi112.stdout:[client.0] 2023-05-31T19:48:39.322 INFO:teuthology.orchestra.run.smithi112.stdout: key = AQCNpHdkoTHUMRAADEZWdmCK87PgyqxQzDYSAQ== 2023-05-31T19:48:39.322 INFO:teuthology.orchestra.run.smithi112.stdout:[client.1] 2023-05-31T19:48:39.322 INFO:teuthology.orchestra.run.smithi112.stdout: key = AQCNpHdkNkWiNxAAQDNkazXyb7Bb4NquQ44dWw== 2023-05-31T19:48:39.323 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/c2.keyring --name=mgr.x --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2023-05-31T19:48:39.417 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/c2.keyring --name=osd.0 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-05-31T19:48:39.471 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/c2.keyring --name=osd.1 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-05-31T19:48:39.525 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/c2.keyring --name=osd.2 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-05-31T19:48:39.577 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/c2.keyring --name=client.0 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2023-05-31T19:48:39.634 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/c2.keyring --name=client.1 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2023-05-31T19:48:39.690 INFO:tasks.ceph:Running mkfs on mon nodes... 2023-05-31T19:48:39.692 DEBUG:teuthology.orchestra.run.smithi112:> sudo mkdir -p /var/lib/ceph/mon/c2-a 2023-05-31T19:48:39.719 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster c2 --mkfs -i a --monmap /home/ubuntu/cephtest/c2.monmap --keyring /etc/ceph/c2.keyring 2023-05-31T19:48:39.817 DEBUG:teuthology.orchestra.run.smithi112:> sudo chown -R ceph:ceph /var/lib/ceph/mon/c2-a 2023-05-31T19:48:39.844 DEBUG:teuthology.orchestra.run.smithi112:> rm -- /home/ubuntu/cephtest/c2.monmap 2023-05-31T19:48:39.902 INFO:tasks.ceph:Starting mon daemons in cluster c2... 2023-05-31T19:48:39.903 INFO:tasks.ceph.c2.mon.a:Restarting daemon 2023-05-31T19:48:39.903 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster c2 -i a 2023-05-31T19:48:39.944 INFO:tasks.ceph.c2.mon.a:Started 2023-05-31T19:48:39.945 INFO:tasks.ceph:Starting mgr daemons in cluster c2... 2023-05-31T19:48:39.945 INFO:tasks.ceph.c2.mgr.x:Restarting daemon 2023-05-31T19:48:39.945 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mgr -f --cluster c2 -i x 2023-05-31T19:48:39.947 INFO:tasks.ceph.c2.mgr.x:Started 2023-05-31T19:48:39.947 INFO:tasks.ceph:Setting crush tunables to default 2023-05-31T19:48:39.948 DEBUG:teuthology.orchestra.run.smithi112:> sudo ceph --cluster c2 osd crush tunables default 2023-05-31T19:48:40.153 INFO:teuthology.orchestra.run.smithi112.stderr:adjusted tunables profile to default 2023-05-31T19:48:40.164 INFO:tasks.ceph:check_enable_crimson: False 2023-05-31T19:48:40.165 INFO:tasks.ceph:Starting osd daemons in cluster c2... 2023-05-31T19:48:40.165 DEBUG:teuthology.orchestra.run.smithi112:> set -ex 2023-05-31T19:48:40.165 DEBUG:teuthology.orchestra.run.smithi112:> sudo dd if=/var/lib/ceph/osd/c2-0/fsid of=/dev/stdout 2023-05-31T19:48:40.194 DEBUG:teuthology.orchestra.run.smithi112:> set -ex 2023-05-31T19:48:40.194 DEBUG:teuthology.orchestra.run.smithi112:> sudo dd if=/var/lib/ceph/osd/c2-1/fsid of=/dev/stdout 2023-05-31T19:48:40.262 DEBUG:teuthology.orchestra.run.smithi112:> set -ex 2023-05-31T19:48:40.263 DEBUG:teuthology.orchestra.run.smithi112:> sudo dd if=/var/lib/ceph/osd/c2-2/fsid of=/dev/stdout 2023-05-31T19:48:40.333 DEBUG:teuthology.orchestra.run.smithi112:> sudo ceph --cluster c2 osd new 3dd1dcd9-8411-4cb0-af57-db2a456da14a 0 2023-05-31T19:48:40.359 INFO:tasks.ceph.c2.mgr.x.smithi112.stderr:2023-05-31T19:48:40.353+0000 7f55b996c200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2023-05-31T19:48:40.546 INFO:tasks.ceph.c2.mgr.x.smithi112.stderr:2023-05-31T19:48:40.540+0000 7f55b996c200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2023-05-31T19:48:40.599 INFO:teuthology.orchestra.run.smithi112.stdout:0 2023-05-31T19:48:40.612 DEBUG:teuthology.orchestra.run.smithi112:> sudo ceph --cluster c2 osd new 4aeb5fc8-5101-423c-b419-e0ee5645c2bd 1 2023-05-31T19:48:40.824 INFO:teuthology.orchestra.run.smithi112.stdout:1 2023-05-31T19:48:40.836 DEBUG:teuthology.orchestra.run.smithi112:> sudo ceph --cluster c2 osd new eff79be9-8a30-4254-85f7-b8e812a195d2 2 2023-05-31T19:48:40.917 INFO:tasks.ceph.c2.mgr.x.smithi112.stderr:2023-05-31T19:48:40.911+0000 7f55b996c200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2023-05-31T19:48:41.051 INFO:teuthology.orchestra.run.smithi112.stdout:2 2023-05-31T19:48:41.063 INFO:tasks.ceph.c2.osd.0:Restarting daemon 2023-05-31T19:48:41.063 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster c2 -i 0 2023-05-31T19:48:41.065 INFO:tasks.ceph.c2.osd.0:Started 2023-05-31T19:48:41.065 INFO:tasks.ceph.c2.osd.1:Restarting daemon 2023-05-31T19:48:41.066 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster c2 -i 1 2023-05-31T19:48:41.067 INFO:tasks.ceph.c2.osd.1:Started 2023-05-31T19:48:41.068 INFO:tasks.ceph.c2.osd.2:Restarting daemon 2023-05-31T19:48:41.068 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster c2 -i 2 2023-05-31T19:48:41.070 INFO:tasks.ceph.c2.osd.2:Started 2023-05-31T19:48:41.070 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c2 osd dump --format=json 2023-05-31T19:48:41.289 INFO:teuthology.orchestra.run.smithi112.stdout: 2023-05-31T19:48:41.289 INFO:teuthology.orchestra.run.smithi112.stdout:{"epoch":5,"fsid":"ff2a32a9-2a3b-4db6-ad15-e0ad1a679cca","created":"2023-05-31T19:48:40.051627+0000","modified":"2023-05-31T19:48:41.044890+0000","last_up_change":"0.000000","last_in_change":"2023-05-31T19:48:41.044890+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"3dd1dcd9-8411-4cb0-af57-db2a456da14a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"4aeb5fc8-5101-423c-b419-e0ee5645c2bd","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"eff79be9-8a30-4254-85f7-b8e812a195d2","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-05-31T19:48:41.301 INFO:tasks.ceph.ceph_manager.c2:[] 2023-05-31T19:48:41.302 INFO:tasks.ceph:Waiting for OSDs to come up 2023-05-31T19:48:41.453 INFO:tasks.ceph.c2.mgr.x.smithi112.stderr:2023-05-31T19:48:41.447+0000 7f55b996c200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2023-05-31T19:48:41.604 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster c2 osd dump --format=json 2023-05-31T19:48:41.655 INFO:tasks.ceph.c2.mgr.x.smithi112.stderr:2023-05-31T19:48:41.649+0000 7f55b996c200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2023-05-31T19:48:41.667 INFO:tasks.ceph.c2.osd.0.smithi112.stderr:2023-05-31T19:48:41.661+0000 7fbb9fb18640 -1 Falling back to public interface 2023-05-31T19:48:41.698 INFO:tasks.ceph.c2.osd.1.smithi112.stderr:2023-05-31T19:48:41.692+0000 7ff23b83e640 -1 Falling back to public interface 2023-05-31T19:48:41.699 INFO:tasks.ceph.c2.osd.2.smithi112.stderr:2023-05-31T19:48:41.693+0000 7f1990665640 -1 Falling back to public interface 2023-05-31T19:48:41.801 INFO:teuthology.misc.health.smithi112.stdout: 2023-05-31T19:48:41.801 INFO:teuthology.misc.health.smithi112.stdout:{"epoch":5,"fsid":"ff2a32a9-2a3b-4db6-ad15-e0ad1a679cca","created":"2023-05-31T19:48:40.051627+0000","modified":"2023-05-31T19:48:41.044890+0000","last_up_change":"0.000000","last_in_change":"2023-05-31T19:48:41.044890+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"3dd1dcd9-8411-4cb0-af57-db2a456da14a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"4aeb5fc8-5101-423c-b419-e0ee5645c2bd","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"eff79be9-8a30-4254-85f7-b8e812a195d2","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-05-31T19:48:41.813 DEBUG:teuthology.misc:0 of 3 OSDs are up 2023-05-31T19:48:42.039 INFO:tasks.ceph.c2.mgr.x.smithi112.stderr:2023-05-31T19:48:42.033+0000 7f55b996c200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2023-05-31T19:48:42.704 INFO:tasks.ceph.c2.mgr.x.smithi112.stderr:2023-05-31T19:48:42.694+0000 7f55b996c200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2023-05-31T19:48:42.773 INFO:tasks.ceph.c2.mgr.x.smithi112.stderr:2023-05-31T19:48:42.767+0000 7f55b996c200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2023-05-31T19:48:42.907 INFO:tasks.ceph.c2.mgr.x.smithi112.stderr:2023-05-31T19:48:42.901+0000 7f55b996c200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2023-05-31T19:48:42.977 INFO:tasks.ceph.c2.mgr.x.smithi112.stderr:2023-05-31T19:48:42.971+0000 7f55b996c200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2023-05-31T19:48:43.158 INFO:tasks.ceph.c2.mgr.x.smithi112.stderr:2023-05-31T19:48:43.152+0000 7f55b996c200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2023-05-31T19:48:43.236 INFO:tasks.ceph.c2.mgr.x.smithi112.stderr:2023-05-31T19:48:43.230+0000 7f55b996c200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2023-05-31T19:48:43.313 INFO:tasks.ceph.c2.mgr.x.smithi112.stderr:2023-05-31T19:48:43.307+0000 7f55b996c200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2023-05-31T19:48:43.406 INFO:tasks.ceph.c2.mgr.x.smithi112.stderr:2023-05-31T19:48:43.400+0000 7f55b996c200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2023-05-31T19:48:43.587 INFO:tasks.ceph.c2.mgr.x.smithi112.stderr:2023-05-31T19:48:43.581+0000 7f55b996c200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2023-05-31T19:48:43.668 INFO:tasks.ceph.c2.mgr.x.smithi112.stderr:2023-05-31T19:48:43.663+0000 7f55b996c200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2023-05-31T19:48:43.956 INFO:tasks.ceph.c2.mgr.x.smithi112.stderr:2023-05-31T19:48:43.950+0000 7f55b996c200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2023-05-31T19:48:44.026 INFO:tasks.ceph.c2.mgr.x.smithi112.stderr:2023-05-31T19:48:44.020+0000 7f55b996c200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2023-05-31T19:48:44.423 INFO:tasks.ceph.c2.osd.1.smithi112.stderr:2023-05-31T19:48:44.417+0000 7ff23b83e640 -1 osd.1 0 log_to_monitors true 2023-05-31T19:48:44.424 INFO:tasks.ceph.c2.osd.0.smithi112.stderr:2023-05-31T19:48:44.418+0000 7fbb9fb18640 -1 osd.0 0 log_to_monitors true 2023-05-31T19:48:44.443 INFO:tasks.ceph.c2.osd.2.smithi112.stderr:2023-05-31T19:48:44.437+0000 7f1990665640 -1 osd.2 0 log_to_monitors true 2023-05-31T19:48:44.559 INFO:tasks.ceph.c2.mgr.x.smithi112.stderr:2023-05-31T19:48:44.553+0000 7f55b996c200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2023-05-31T19:48:44.872 INFO:tasks.ceph.c2.mgr.x.smithi112.stderr:2023-05-31T19:48:44.866+0000 7f55b996c200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2023-05-31T19:48:45.014 INFO:tasks.ceph.c2.mgr.x.smithi112.stderr:2023-05-31T19:48:45.008+0000 7f55b996c200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2023-05-31T19:48:45.368 INFO:tasks.ceph.c2.mgr.x.smithi112.stderr:2023-05-31T19:48:45.363+0000 7f55b996c200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2023-05-31T19:48:45.518 INFO:tasks.ceph.c2.mgr.x.smithi112.stderr:2023-05-31T19:48:45.512+0000 7f55b996c200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2023-05-31T19:48:45.690 INFO:tasks.ceph.c2.mgr.x.smithi112.stderr:2023-05-31T19:48:45.684+0000 7f55b996c200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2023-05-31T19:48:46.028 INFO:tasks.ceph.c2.mgr.x.smithi112.stderr:2023-05-31T19:48:46.022+0000 7f55b996c200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2023-05-31T19:48:46.663 INFO:tasks.ceph.c2.osd.1.smithi112.stderr:2023-05-31T19:48:46.657+0000 7ff22fbdf700 -1 osd.1 0 waiting for initial osdmap 2023-05-31T19:48:46.677 INFO:tasks.ceph.c2.osd.1.smithi112.stderr:2023-05-31T19:48:46.671+0000 7ff2277b9700 -1 osd.1 7 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2023-05-31T19:48:46.707 INFO:tasks.ceph.c2.osd.2.smithi112.stderr:2023-05-31T19:48:46.701+0000 7f1984a06700 -1 osd.2 0 waiting for initial osdmap 2023-05-31T19:48:46.715 INFO:tasks.ceph.c2.osd.2.smithi112.stderr:2023-05-31T19:48:46.710+0000 7f197c5e0700 -1 osd.2 7 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2023-05-31T19:48:46.732 INFO:tasks.ceph.c2.osd.0.smithi112.stderr:2023-05-31T19:48:46.727+0000 7fbb93eb9700 -1 osd.0 0 waiting for initial osdmap 2023-05-31T19:48:46.743 INFO:tasks.ceph.c2.osd.0.smithi112.stderr:2023-05-31T19:48:46.738+0000 7fbb8ba93700 -1 osd.0 7 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2023-05-31T19:48:47.101 INFO:tasks.ceph.c2.mgr.x.smithi112.stderr:2023-05-31T19:48:47.096+0000 7f557ad88700 -1 mgr.server handle_report got status from non-daemon mon.a 2023-05-31T19:48:48.120 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster c2 osd dump --format=json 2023-05-31T19:48:48.440 INFO:teuthology.misc.health.smithi112.stdout: 2023-05-31T19:48:48.440 INFO:teuthology.misc.health.smithi112.stdout:{"epoch":8,"fsid":"ff2a32a9-2a3b-4db6-ad15-e0ad1a679cca","created":"2023-05-31T19:48:40.051627+0000","modified":"2023-05-31T19:48:47.659300+0000","last_up_change":"2023-05-31T19:48:47.659300+0000","last_in_change":"2023-05-31T19:48:41.044890+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"3dd1dcd9-8411-4cb0-af57-db2a456da14a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6800","nonce":104807},{"type":"v1","addr":"172.21.15.112:6801","nonce":104807}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6802","nonce":104807},{"type":"v1","addr":"172.21.15.112:6803","nonce":104807}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6806","nonce":104807},{"type":"v1","addr":"172.21.15.112:6807","nonce":104807}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6804","nonce":104807},{"type":"v1","addr":"172.21.15.112:6805","nonce":104807}]},"public_addr":"172.21.15.112:6801/104807","cluster_addr":"172.21.15.112:6803/104807","heartbeat_back_addr":"172.21.15.112:6807/104807","heartbeat_front_addr":"172.21.15.112:6805/104807","state":["exists","up"]},{"osd":1,"uuid":"4aeb5fc8-5101-423c-b419-e0ee5645c2bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6808","nonce":104816},{"type":"v1","addr":"172.21.15.112:6811","nonce":104816}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6813","nonce":104816},{"type":"v1","addr":"172.21.15.112:6815","nonce":104816}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6822","nonce":104816},{"type":"v1","addr":"172.21.15.112:6823","nonce":104816}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6818","nonce":104816},{"type":"v1","addr":"172.21.15.112:6820","nonce":104816}]},"public_addr":"172.21.15.112:6811/104816","cluster_addr":"172.21.15.112:6815/104816","heartbeat_back_addr":"172.21.15.112:6823/104816","heartbeat_front_addr":"172.21.15.112:6820/104816","state":["exists","up"]},{"osd":2,"uuid":"eff79be9-8a30-4254-85f7-b8e812a195d2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6809","nonce":104821},{"type":"v1","addr":"172.21.15.112:6810","nonce":104821}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6812","nonce":104821},{"type":"v1","addr":"172.21.15.112:6814","nonce":104821}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6819","nonce":104821},{"type":"v1","addr":"172.21.15.112:6821","nonce":104821}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6816","nonce":104821},{"type":"v1","addr":"172.21.15.112:6817","nonce":104821}]},"public_addr":"172.21.15.112:6810/104821","cluster_addr":"172.21.15.112:6814/104821","heartbeat_back_addr":"172.21.15.112:6821/104821","heartbeat_front_addr":"172.21.15.112:6817/104821","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-05-31T19:48:48.452 DEBUG:teuthology.misc:3 of 3 OSDs are up 2023-05-31T19:48:48.453 INFO:tasks.ceph:Creating RBD pool 2023-05-31T19:48:48.453 DEBUG:teuthology.orchestra.run.smithi112:> sudo ceph --cluster c2 osd pool create rbd 8 2023-05-31T19:48:49.114 INFO:teuthology.orchestra.run.smithi112.stderr:pool 'rbd' created 2023-05-31T19:48:49.126 DEBUG:teuthology.orchestra.run.smithi112:> sudo ceph --cluster c2 osd pool application enable rbd rbd --yes-i-really-mean-it 2023-05-31T19:48:50.113 INFO:teuthology.orchestra.run.smithi112.stderr:enabled application 'rbd' on pool 'rbd' 2023-05-31T19:48:50.127 INFO:tasks.ceph:Starting mds daemons in cluster c2... 2023-05-31T19:48:50.128 INFO:tasks.ceph:Waiting until c2 daemons up and pgs clean... 2023-05-31T19:48:50.128 INFO:tasks.ceph.ceph_manager.c2:waiting for mgr available 2023-05-31T19:48:50.128 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c2 mgr dump --format=json 2023-05-31T19:48:50.129 INFO:tasks.daemonwatchdog.daemon_watchdog:watchdog starting 2023-05-31T19:48:50.543 INFO:teuthology.orchestra.run.smithi112.stdout: 2023-05-31T19:48:50.556 INFO:teuthology.orchestra.run.smithi112.stdout:{"epoch":4,"active_gid":4102,"active_name":"x","active_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6824","nonce":104395},{"type":"v1","addr":"172.21.15.112:6825","nonce":104395}]},"active_addr":"172.21.15.112:6825/104395","active_change":"2023-05-31T19:48:46.036187+0000","active_mgr_features":4540138322906710015,"available":true,"standbys":[],"modules":["iostat","nfs","restful"],"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","upmap"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"alertmanager_web_password":{"name":"alertmanager_web_password","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"Alertmanager web password","long_desc":"","tags":[],"see_also":[]},"alertmanager_web_user":{"name":"alertmanager_web_user","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"Alertmanager web user","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/ceph-grafana:9.4.7","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"secs","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"prometheus_web_password":{"name":"prometheus_web_password","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"Prometheus web password","long_desc":"","tags":[],"see_also":[]},"prometheus_web_user":{"name":"prometheus_web_user","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"Prometheus web user","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"noautoscale":{"name":"noautoscale","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"global autoscale flag","long_desc":"Option to turn on/off the autoscaler for all pools","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"drive_group_interval":{"name":"drive_group_interval","type":"float","level":"advanced","flags":0,"default_value":"300.0","min":"","max":"","enum_allowed":[],"desc":"interval in seconds between re-application of applied drive_groups","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}],"services":{},"always_on_modules":{"octopus":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"pacific":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"quincy":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"reef":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"]},"last_failure_osd_epoch":0,"active_clients":[{"name":"libcephsqlite","addrvec":[{"type":"v2","addr":"172.21.15.112:0","nonce":831955379}]},{"name":"rbd_support","addrvec":[{"type":"v2","addr":"172.21.15.112:0","nonce":1412026681}]},{"name":"volumes","addrvec":[{"type":"v2","addr":"172.21.15.112:0","nonce":1048909254}]}]} 2023-05-31T19:48:50.559 INFO:tasks.ceph.ceph_manager.c2:mgr available! 2023-05-31T19:48:50.559 INFO:tasks.ceph.ceph_manager.c2:waiting for all up 2023-05-31T19:48:50.559 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c2 osd dump --format=json 2023-05-31T19:48:50.874 INFO:teuthology.orchestra.run.smithi112.stdout: 2023-05-31T19:48:50.875 INFO:teuthology.orchestra.run.smithi112.stdout:{"epoch":10,"fsid":"ff2a32a9-2a3b-4db6-ad15-e0ad1a679cca","created":"2023-05-31T19:48:40.051627+0000","modified":"2023-05-31T19:48:50.101355+0000","last_up_change":"2023-05-31T19:48:47.659300+0000","last_in_change":"2023-05-31T19:48:41.044890+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-05-31T19:48:48.138683+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"10","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":2.9900000095367432,"score_stable":2.9900000095367432,"optimal_score":0.67000001668930054,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2023-05-31T19:48:48.780643+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"10","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"3dd1dcd9-8411-4cb0-af57-db2a456da14a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":9,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6800","nonce":104807},{"type":"v1","addr":"172.21.15.112:6801","nonce":104807}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6802","nonce":104807},{"type":"v1","addr":"172.21.15.112:6803","nonce":104807}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6806","nonce":104807},{"type":"v1","addr":"172.21.15.112:6807","nonce":104807}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6804","nonce":104807},{"type":"v1","addr":"172.21.15.112:6805","nonce":104807}]},"public_addr":"172.21.15.112:6801/104807","cluster_addr":"172.21.15.112:6803/104807","heartbeat_back_addr":"172.21.15.112:6807/104807","heartbeat_front_addr":"172.21.15.112:6805/104807","state":["exists","up"]},{"osd":1,"uuid":"4aeb5fc8-5101-423c-b419-e0ee5645c2bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":9,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6808","nonce":104816},{"type":"v1","addr":"172.21.15.112:6811","nonce":104816}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6813","nonce":104816},{"type":"v1","addr":"172.21.15.112:6815","nonce":104816}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6822","nonce":104816},{"type":"v1","addr":"172.21.15.112:6823","nonce":104816}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6818","nonce":104816},{"type":"v1","addr":"172.21.15.112:6820","nonce":104816}]},"public_addr":"172.21.15.112:6811/104816","cluster_addr":"172.21.15.112:6815/104816","heartbeat_back_addr":"172.21.15.112:6823/104816","heartbeat_front_addr":"172.21.15.112:6820/104816","state":["exists","up"]},{"osd":2,"uuid":"eff79be9-8a30-4254-85f7-b8e812a195d2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":9,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6809","nonce":104821},{"type":"v1","addr":"172.21.15.112:6810","nonce":104821}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6812","nonce":104821},{"type":"v1","addr":"172.21.15.112:6814","nonce":104821}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6819","nonce":104821},{"type":"v1","addr":"172.21.15.112:6821","nonce":104821}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6816","nonce":104821},{"type":"v1","addr":"172.21.15.112:6817","nonce":104821}]},"public_addr":"172.21.15.112:6810/104821","cluster_addr":"172.21.15.112:6814/104821","heartbeat_back_addr":"172.21.15.112:6821/104821","heartbeat_front_addr":"172.21.15.112:6817/104821","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-05-31T19:48:45.437892+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-05-31T19:48:45.399142+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-05-31T19:48:45.474295+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-05-31T19:48:50.890 INFO:tasks.ceph.ceph_manager.c2:all up! 2023-05-31T19:48:50.890 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c2 osd dump --format=json 2023-05-31T19:48:51.210 INFO:teuthology.orchestra.run.smithi112.stdout: 2023-05-31T19:48:51.211 INFO:teuthology.orchestra.run.smithi112.stdout:{"epoch":11,"fsid":"ff2a32a9-2a3b-4db6-ad15-e0ad1a679cca","created":"2023-05-31T19:48:40.051627+0000","modified":"2023-05-31T19:48:51.102293+0000","last_up_change":"2023-05-31T19:48:47.659300+0000","last_in_change":"2023-05-31T19:48:41.044890+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-05-31T19:48:48.138683+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"11","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":2.9900000095367432,"score_stable":2.9900000095367432,"optimal_score":0.67000001668930054,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2023-05-31T19:48:48.780643+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"11","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"3dd1dcd9-8411-4cb0-af57-db2a456da14a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":9,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6800","nonce":104807},{"type":"v1","addr":"172.21.15.112:6801","nonce":104807}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6802","nonce":104807},{"type":"v1","addr":"172.21.15.112:6803","nonce":104807}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6806","nonce":104807},{"type":"v1","addr":"172.21.15.112:6807","nonce":104807}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6804","nonce":104807},{"type":"v1","addr":"172.21.15.112:6805","nonce":104807}]},"public_addr":"172.21.15.112:6801/104807","cluster_addr":"172.21.15.112:6803/104807","heartbeat_back_addr":"172.21.15.112:6807/104807","heartbeat_front_addr":"172.21.15.112:6805/104807","state":["exists","up"]},{"osd":1,"uuid":"4aeb5fc8-5101-423c-b419-e0ee5645c2bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":9,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6808","nonce":104816},{"type":"v1","addr":"172.21.15.112:6811","nonce":104816}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6813","nonce":104816},{"type":"v1","addr":"172.21.15.112:6815","nonce":104816}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6822","nonce":104816},{"type":"v1","addr":"172.21.15.112:6823","nonce":104816}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6818","nonce":104816},{"type":"v1","addr":"172.21.15.112:6820","nonce":104816}]},"public_addr":"172.21.15.112:6811/104816","cluster_addr":"172.21.15.112:6815/104816","heartbeat_back_addr":"172.21.15.112:6823/104816","heartbeat_front_addr":"172.21.15.112:6820/104816","state":["exists","up"]},{"osd":2,"uuid":"eff79be9-8a30-4254-85f7-b8e812a195d2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":9,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6809","nonce":104821},{"type":"v1","addr":"172.21.15.112:6810","nonce":104821}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6812","nonce":104821},{"type":"v1","addr":"172.21.15.112:6814","nonce":104821}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6819","nonce":104821},{"type":"v1","addr":"172.21.15.112:6821","nonce":104821}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6816","nonce":104821},{"type":"v1","addr":"172.21.15.112:6817","nonce":104821}]},"public_addr":"172.21.15.112:6810/104821","cluster_addr":"172.21.15.112:6814/104821","heartbeat_back_addr":"172.21.15.112:6821/104821","heartbeat_front_addr":"172.21.15.112:6817/104821","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-05-31T19:48:45.437892+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-05-31T19:48:45.399142+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-05-31T19:48:45.474295+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-05-31T19:48:51.224 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c2 tell osd.0 flush_pg_stats 2023-05-31T19:48:51.225 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c2 tell osd.1 flush_pg_stats 2023-05-31T19:48:51.225 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c2 tell osd.2 flush_pg_stats 2023-05-31T19:48:51.392 INFO:teuthology.orchestra.run.smithi112.stdout:34359738371 2023-05-31T19:48:51.393 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c2 osd last-stat-seq osd.2 2023-05-31T19:48:51.396 INFO:teuthology.orchestra.run.smithi112.stdout:34359738371 2023-05-31T19:48:51.397 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c2 osd last-stat-seq osd.0 2023-05-31T19:48:51.397 INFO:teuthology.orchestra.run.smithi112.stdout:34359738370 2023-05-31T19:48:51.398 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c2 osd last-stat-seq osd.1 2023-05-31T19:48:51.731 INFO:teuthology.orchestra.run.smithi112.stdout:34359738370 2023-05-31T19:48:51.745 INFO:tasks.ceph.ceph_manager.c2:need seq 34359738371 got 34359738370 for osd.2 2023-05-31T19:48:51.771 INFO:teuthology.orchestra.run.smithi112.stdout:34359738369 2023-05-31T19:48:51.772 INFO:teuthology.orchestra.run.smithi112.stdout:34359738370 2023-05-31T19:48:51.793 INFO:tasks.ceph.ceph_manager.c2:need seq 34359738371 got 34359738370 for osd.0 2023-05-31T19:48:51.794 INFO:tasks.ceph.ceph_manager.c2:need seq 34359738370 got 34359738369 for osd.1 2023-05-31T19:48:52.746 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c2 osd last-stat-seq osd.2 2023-05-31T19:48:52.794 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c2 osd last-stat-seq osd.0 2023-05-31T19:48:52.795 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c2 osd last-stat-seq osd.1 2023-05-31T19:48:53.106 INFO:teuthology.orchestra.run.smithi112.stdout:34359738371 2023-05-31T19:48:53.121 INFO:tasks.ceph.ceph_manager.c2:need seq 34359738371 got 34359738371 for osd.2 2023-05-31T19:48:53.122 DEBUG:teuthology.parallel:result is None 2023-05-31T19:48:53.136 INFO:teuthology.orchestra.run.smithi112.stdout:34359738371 2023-05-31T19:48:53.143 INFO:teuthology.orchestra.run.smithi112.stdout:34359738370 2023-05-31T19:48:53.153 INFO:tasks.ceph.ceph_manager.c2:need seq 34359738371 got 34359738371 for osd.0 2023-05-31T19:48:53.154 DEBUG:teuthology.parallel:result is None 2023-05-31T19:48:53.158 INFO:tasks.ceph.ceph_manager.c2:need seq 34359738370 got 34359738370 for osd.1 2023-05-31T19:48:53.158 DEBUG:teuthology.parallel:result is None 2023-05-31T19:48:53.158 INFO:tasks.ceph.ceph_manager.c2:waiting for clean 2023-05-31T19:48:53.158 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c2 pg dump --format=json 2023-05-31T19:48:53.513 INFO:teuthology.orchestra.run.smithi112.stdout: 2023-05-31T19:48:53.515 INFO:teuthology.orchestra.run.smithi112.stderr:dumped all 2023-05-31T19:48:53.525 INFO:teuthology.orchestra.run.smithi112.stdout:{"pg_ready":true,"pg_map":{"version":9,"stamp":"2023-05-31T19:48:52.096337+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":0,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":314572800,"kb_used":25440,"kb_used_data":1420,"kb_used_omap":0,"kb_used_meta":23424,"kb_avail":314547360,"statfs":{"total":322122547200,"available":322096496640,"internally_reserved":0,"allocated":1454080,"data_stored":1247365,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":23986176},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"0.000000"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":10,"reported_epoch":11,"state":"active+clean","last_fresh":"2023-05-31T19:48:51.106145+0000","last_change":"2023-05-31T19:48:50.115040+0000","last_active":"2023-05-31T19:48:51.106145+0000","last_peered":"2023-05-31T19:48:51.106145+0000","last_clean":"2023-05-31T19:48:51.106145+0000","last_became_active":"2023-05-31T19:48:50.114785+0000","last_became_peered":"2023-05-31T19:48:50.114785+0000","last_unstale":"2023-05-31T19:48:51.106145+0000","last_undegraded":"2023-05-31T19:48:51.106145+0000","last_fullsized":"2023-05-31T19:48:51.106145+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-31T19:48:49.097633+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-31T19:48:49.097633+0000","last_clean_scrub_stamp":"2023-05-31T19:48:49.097633+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-02T06:17:16.040863+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":10,"reported_epoch":11,"state":"active+clean","last_fresh":"2023-05-31T19:48:51.106165+0000","last_change":"2023-05-31T19:48:50.115072+0000","last_active":"2023-05-31T19:48:51.106165+0000","last_peered":"2023-05-31T19:48:51.106165+0000","last_clean":"2023-05-31T19:48:51.106165+0000","last_became_active":"2023-05-31T19:48:50.114831+0000","last_became_peered":"2023-05-31T19:48:50.114831+0000","last_unstale":"2023-05-31T19:48:51.106165+0000","last_undegraded":"2023-05-31T19:48:51.106165+0000","last_fullsized":"2023-05-31T19:48:51.106165+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-31T19:48:49.097633+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-31T19:48:49.097633+0000","last_clean_scrub_stamp":"2023-05-31T19:48:49.097633+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-02T07:15:53.725346+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":10,"reported_epoch":11,"state":"active+clean","last_fresh":"2023-05-31T19:48:51.106044+0000","last_change":"2023-05-31T19:48:50.115042+0000","last_active":"2023-05-31T19:48:51.106044+0000","last_peered":"2023-05-31T19:48:51.106044+0000","last_clean":"2023-05-31T19:48:51.106044+0000","last_became_active":"2023-05-31T19:48:50.114792+0000","last_became_peered":"2023-05-31T19:48:50.114792+0000","last_unstale":"2023-05-31T19:48:51.106044+0000","last_undegraded":"2023-05-31T19:48:51.106044+0000","last_fullsized":"2023-05-31T19:48:51.106044+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-31T19:48:49.097633+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-31T19:48:49.097633+0000","last_clean_scrub_stamp":"2023-05-31T19:48:49.097633+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-02T06:53:44.451425+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":9,"reported_epoch":10,"state":"active+clean","last_fresh":"2023-05-31T19:48:50.117205+0000","last_change":"2023-05-31T19:48:50.112387+0000","last_active":"2023-05-31T19:48:50.117205+0000","last_peered":"2023-05-31T19:48:50.117205+0000","last_clean":"2023-05-31T19:48:50.117205+0000","last_became_active":"2023-05-31T19:48:50.112088+0000","last_became_peered":"2023-05-31T19:48:50.112088+0000","last_unstale":"2023-05-31T19:48:50.117205+0000","last_undegraded":"2023-05-31T19:48:50.117205+0000","last_fullsized":"2023-05-31T19:48:50.117205+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-31T19:48:49.097633+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-31T19:48:49.097633+0000","last_clean_scrub_stamp":"2023-05-31T19:48:49.097633+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-02T04:13:11.477599+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"10'97","reported_seq":137,"reported_epoch":11,"state":"active+clean","last_fresh":"2023-05-31T19:48:51.106004+0000","last_change":"2023-05-31T19:48:50.109933+0000","last_active":"2023-05-31T19:48:51.106004+0000","last_peered":"2023-05-31T19:48:51.106004+0000","last_clean":"2023-05-31T19:48:51.106004+0000","last_became_active":"2023-05-31T19:48:50.109689+0000","last_became_peered":"2023-05-31T19:48:50.109689+0000","last_unstale":"2023-05-31T19:48:51.106004+0000","last_undegraded":"2023-05-31T19:48:51.106004+0000","last_fullsized":"2023-05-31T19:48:51.106004+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-31T19:48:49.097633+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-31T19:48:49.097633+0000","last_clean_scrub_stamp":"2023-05-31T19:48:49.097633+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-02T05:12:35.925537+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":10,"reported_epoch":11,"state":"active+clean","last_fresh":"2023-05-31T19:48:51.106052+0000","last_change":"2023-05-31T19:48:50.110014+0000","last_active":"2023-05-31T19:48:51.106052+0000","last_peered":"2023-05-31T19:48:51.106052+0000","last_clean":"2023-05-31T19:48:51.106052+0000","last_became_active":"2023-05-31T19:48:50.109759+0000","last_became_peered":"2023-05-31T19:48:50.109759+0000","last_unstale":"2023-05-31T19:48:51.106052+0000","last_undegraded":"2023-05-31T19:48:51.106052+0000","last_fullsized":"2023-05-31T19:48:51.106052+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-31T19:48:49.097633+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-31T19:48:49.097633+0000","last_clean_scrub_stamp":"2023-05-31T19:48:49.097633+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-02T06:48:17.785725+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":9,"reported_epoch":10,"state":"active+clean","last_fresh":"2023-05-31T19:48:50.117208+0000","last_change":"2023-05-31T19:48:50.112471+0000","last_active":"2023-05-31T19:48:50.117208+0000","last_peered":"2023-05-31T19:48:50.117208+0000","last_clean":"2023-05-31T19:48:50.117208+0000","last_became_active":"2023-05-31T19:48:50.112109+0000","last_became_peered":"2023-05-31T19:48:50.112109+0000","last_unstale":"2023-05-31T19:48:50.117208+0000","last_undegraded":"2023-05-31T19:48:50.117208+0000","last_fullsized":"2023-05-31T19:48:50.117208+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-31T19:48:49.097633+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-31T19:48:49.097633+0000","last_clean_scrub_stamp":"2023-05-31T19:48:49.097633+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-01T20:27:28.088959+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":9,"reported_epoch":10,"state":"active+clean","last_fresh":"2023-05-31T19:48:50.119952+0000","last_change":"2023-05-31T19:48:50.110025+0000","last_active":"2023-05-31T19:48:50.119952+0000","last_peered":"2023-05-31T19:48:50.119952+0000","last_clean":"2023-05-31T19:48:50.119952+0000","last_became_active":"2023-05-31T19:48:50.109793+0000","last_became_peered":"2023-05-31T19:48:50.109793+0000","last_unstale":"2023-05-31T19:48:50.119952+0000","last_undegraded":"2023-05-31T19:48:50.119952+0000","last_fullsized":"2023-05-31T19:48:50.119952+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-31T19:48:49.097633+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-31T19:48:49.097633+0000","last_clean_scrub_stamp":"2023-05-31T19:48:49.097633+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-02T04:56:57.868061+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":10,"reported_epoch":11,"state":"active+clean","last_fresh":"2023-05-31T19:48:51.106151+0000","last_change":"2023-05-31T19:48:50.115163+0000","last_active":"2023-05-31T19:48:51.106151+0000","last_peered":"2023-05-31T19:48:51.106151+0000","last_clean":"2023-05-31T19:48:51.106151+0000","last_became_active":"2023-05-31T19:48:50.114880+0000","last_became_peered":"2023-05-31T19:48:50.114880+0000","last_unstale":"2023-05-31T19:48:51.106151+0000","last_undegraded":"2023-05-31T19:48:51.106151+0000","last_fullsized":"2023-05-31T19:48:51.106151+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-31T19:48:49.097633+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-31T19:48:49.097633+0000","last_clean_scrub_stamp":"2023-05-31T19:48:49.097633+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-02T02:12:55.654453+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738371,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":7648,"kb_used_data":88,"kb_used_omap":0,"kb_used_meta":7552,"kb_avail":104849952,"statfs":{"total":107374182400,"available":107366350848,"internally_reserved":0,"allocated":90112,"data_stored":22580,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":7733248},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.30399999999999999}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.34499999999999997}]}]},{"osd":0,"up_from":8,"seq":34359738371,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":8924,"kb_used_data":664,"kb_used_omap":0,"kb_used_meta":7936,"kb_avail":104848676,"statfs":{"total":107374182400,"available":107365044224,"internally_reserved":0,"allocated":679936,"data_stored":611837,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":8126464},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.503}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.48499999999999999}]}]},{"osd":1,"up_from":8,"seq":34359738370,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":8868,"kb_used_data":668,"kb_used_omap":0,"kb_used_meta":7936,"kb_avail":104848732,"statfs":{"total":107374182400,"available":107365101568,"internally_reserved":0,"allocated":684032,"data_stored":612948,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":8126464},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.246}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.29299999999999998}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-05-31T19:48:53.528 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c2 pg dump --format=json 2023-05-31T19:48:53.840 INFO:teuthology.orchestra.run.smithi112.stdout: 2023-05-31T19:48:53.843 INFO:teuthology.orchestra.run.smithi112.stderr:dumped all 2023-05-31T19:48:53.854 INFO:teuthology.orchestra.run.smithi112.stdout:{"pg_ready":true,"pg_map":{"version":9,"stamp":"2023-05-31T19:48:52.096337+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":0,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":314572800,"kb_used":25440,"kb_used_data":1420,"kb_used_omap":0,"kb_used_meta":23424,"kb_avail":314547360,"statfs":{"total":322122547200,"available":322096496640,"internally_reserved":0,"allocated":1454080,"data_stored":1247365,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":23986176},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"0.000000"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":10,"reported_epoch":11,"state":"active+clean","last_fresh":"2023-05-31T19:48:51.106145+0000","last_change":"2023-05-31T19:48:50.115040+0000","last_active":"2023-05-31T19:48:51.106145+0000","last_peered":"2023-05-31T19:48:51.106145+0000","last_clean":"2023-05-31T19:48:51.106145+0000","last_became_active":"2023-05-31T19:48:50.114785+0000","last_became_peered":"2023-05-31T19:48:50.114785+0000","last_unstale":"2023-05-31T19:48:51.106145+0000","last_undegraded":"2023-05-31T19:48:51.106145+0000","last_fullsized":"2023-05-31T19:48:51.106145+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-31T19:48:49.097633+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-31T19:48:49.097633+0000","last_clean_scrub_stamp":"2023-05-31T19:48:49.097633+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-02T06:17:16.040863+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":10,"reported_epoch":11,"state":"active+clean","last_fresh":"2023-05-31T19:48:51.106165+0000","last_change":"2023-05-31T19:48:50.115072+0000","last_active":"2023-05-31T19:48:51.106165+0000","last_peered":"2023-05-31T19:48:51.106165+0000","last_clean":"2023-05-31T19:48:51.106165+0000","last_became_active":"2023-05-31T19:48:50.114831+0000","last_became_peered":"2023-05-31T19:48:50.114831+0000","last_unstale":"2023-05-31T19:48:51.106165+0000","last_undegraded":"2023-05-31T19:48:51.106165+0000","last_fullsized":"2023-05-31T19:48:51.106165+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-31T19:48:49.097633+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-31T19:48:49.097633+0000","last_clean_scrub_stamp":"2023-05-31T19:48:49.097633+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-02T07:15:53.725346+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":10,"reported_epoch":11,"state":"active+clean","last_fresh":"2023-05-31T19:48:51.106044+0000","last_change":"2023-05-31T19:48:50.115042+0000","last_active":"2023-05-31T19:48:51.106044+0000","last_peered":"2023-05-31T19:48:51.106044+0000","last_clean":"2023-05-31T19:48:51.106044+0000","last_became_active":"2023-05-31T19:48:50.114792+0000","last_became_peered":"2023-05-31T19:48:50.114792+0000","last_unstale":"2023-05-31T19:48:51.106044+0000","last_undegraded":"2023-05-31T19:48:51.106044+0000","last_fullsized":"2023-05-31T19:48:51.106044+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-31T19:48:49.097633+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-31T19:48:49.097633+0000","last_clean_scrub_stamp":"2023-05-31T19:48:49.097633+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-02T06:53:44.451425+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":9,"reported_epoch":10,"state":"active+clean","last_fresh":"2023-05-31T19:48:50.117205+0000","last_change":"2023-05-31T19:48:50.112387+0000","last_active":"2023-05-31T19:48:50.117205+0000","last_peered":"2023-05-31T19:48:50.117205+0000","last_clean":"2023-05-31T19:48:50.117205+0000","last_became_active":"2023-05-31T19:48:50.112088+0000","last_became_peered":"2023-05-31T19:48:50.112088+0000","last_unstale":"2023-05-31T19:48:50.117205+0000","last_undegraded":"2023-05-31T19:48:50.117205+0000","last_fullsized":"2023-05-31T19:48:50.117205+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-31T19:48:49.097633+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-31T19:48:49.097633+0000","last_clean_scrub_stamp":"2023-05-31T19:48:49.097633+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-02T04:13:11.477599+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"10'97","reported_seq":137,"reported_epoch":11,"state":"active+clean","last_fresh":"2023-05-31T19:48:51.106004+0000","last_change":"2023-05-31T19:48:50.109933+0000","last_active":"2023-05-31T19:48:51.106004+0000","last_peered":"2023-05-31T19:48:51.106004+0000","last_clean":"2023-05-31T19:48:51.106004+0000","last_became_active":"2023-05-31T19:48:50.109689+0000","last_became_peered":"2023-05-31T19:48:50.109689+0000","last_unstale":"2023-05-31T19:48:51.106004+0000","last_undegraded":"2023-05-31T19:48:51.106004+0000","last_fullsized":"2023-05-31T19:48:51.106004+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-31T19:48:49.097633+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-31T19:48:49.097633+0000","last_clean_scrub_stamp":"2023-05-31T19:48:49.097633+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-02T05:12:35.925537+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":10,"reported_epoch":11,"state":"active+clean","last_fresh":"2023-05-31T19:48:51.106052+0000","last_change":"2023-05-31T19:48:50.110014+0000","last_active":"2023-05-31T19:48:51.106052+0000","last_peered":"2023-05-31T19:48:51.106052+0000","last_clean":"2023-05-31T19:48:51.106052+0000","last_became_active":"2023-05-31T19:48:50.109759+0000","last_became_peered":"2023-05-31T19:48:50.109759+0000","last_unstale":"2023-05-31T19:48:51.106052+0000","last_undegraded":"2023-05-31T19:48:51.106052+0000","last_fullsized":"2023-05-31T19:48:51.106052+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-31T19:48:49.097633+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-31T19:48:49.097633+0000","last_clean_scrub_stamp":"2023-05-31T19:48:49.097633+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-02T06:48:17.785725+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":9,"reported_epoch":10,"state":"active+clean","last_fresh":"2023-05-31T19:48:50.117208+0000","last_change":"2023-05-31T19:48:50.112471+0000","last_active":"2023-05-31T19:48:50.117208+0000","last_peered":"2023-05-31T19:48:50.117208+0000","last_clean":"2023-05-31T19:48:50.117208+0000","last_became_active":"2023-05-31T19:48:50.112109+0000","last_became_peered":"2023-05-31T19:48:50.112109+0000","last_unstale":"2023-05-31T19:48:50.117208+0000","last_undegraded":"2023-05-31T19:48:50.117208+0000","last_fullsized":"2023-05-31T19:48:50.117208+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-31T19:48:49.097633+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-31T19:48:49.097633+0000","last_clean_scrub_stamp":"2023-05-31T19:48:49.097633+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-01T20:27:28.088959+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":9,"reported_epoch":10,"state":"active+clean","last_fresh":"2023-05-31T19:48:50.119952+0000","last_change":"2023-05-31T19:48:50.110025+0000","last_active":"2023-05-31T19:48:50.119952+0000","last_peered":"2023-05-31T19:48:50.119952+0000","last_clean":"2023-05-31T19:48:50.119952+0000","last_became_active":"2023-05-31T19:48:50.109793+0000","last_became_peered":"2023-05-31T19:48:50.109793+0000","last_unstale":"2023-05-31T19:48:50.119952+0000","last_undegraded":"2023-05-31T19:48:50.119952+0000","last_fullsized":"2023-05-31T19:48:50.119952+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-31T19:48:49.097633+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-31T19:48:49.097633+0000","last_clean_scrub_stamp":"2023-05-31T19:48:49.097633+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-02T04:56:57.868061+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":10,"reported_epoch":11,"state":"active+clean","last_fresh":"2023-05-31T19:48:51.106151+0000","last_change":"2023-05-31T19:48:50.115163+0000","last_active":"2023-05-31T19:48:51.106151+0000","last_peered":"2023-05-31T19:48:51.106151+0000","last_clean":"2023-05-31T19:48:51.106151+0000","last_became_active":"2023-05-31T19:48:50.114880+0000","last_became_peered":"2023-05-31T19:48:50.114880+0000","last_unstale":"2023-05-31T19:48:51.106151+0000","last_undegraded":"2023-05-31T19:48:51.106151+0000","last_fullsized":"2023-05-31T19:48:51.106151+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-05-31T19:48:49.097633+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-05-31T19:48:49.097633+0000","last_clean_scrub_stamp":"2023-05-31T19:48:49.097633+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-02T02:12:55.654453+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738371,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":7648,"kb_used_data":88,"kb_used_omap":0,"kb_used_meta":7552,"kb_avail":104849952,"statfs":{"total":107374182400,"available":107366350848,"internally_reserved":0,"allocated":90112,"data_stored":22580,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":7733248},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.30399999999999999}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.34499999999999997}]}]},{"osd":0,"up_from":8,"seq":34359738371,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":8924,"kb_used_data":664,"kb_used_omap":0,"kb_used_meta":7936,"kb_avail":104848676,"statfs":{"total":107374182400,"available":107365044224,"internally_reserved":0,"allocated":679936,"data_stored":611837,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":8126464},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.503}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.48499999999999999}]}]},{"osd":1,"up_from":8,"seq":34359738370,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":8868,"kb_used_data":668,"kb_used_omap":0,"kb_used_meta":7936,"kb_avail":104848732,"statfs":{"total":107374182400,"available":107365101568,"internally_reserved":0,"allocated":684032,"data_stored":612948,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":8126464},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.246}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.29299999999999998}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-05-31T19:48:53.856 INFO:tasks.ceph.ceph_manager.c2:clean! 2023-05-31T19:48:53.857 INFO:tasks.ceph:Waiting until ceph cluster c2 is healthy... 2023-05-31T19:48:53.857 INFO:tasks.ceph.ceph_manager.c2:wait_until_healthy 2023-05-31T19:48:53.858 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c2 health --format=json 2023-05-31T19:48:54.213 INFO:teuthology.orchestra.run.smithi112.stdout: 2023-05-31T19:48:54.213 INFO:teuthology.orchestra.run.smithi112.stdout:{"status":"HEALTH_OK","checks":{},"mutes":[]} 2023-05-31T19:48:54.229 INFO:tasks.ceph.ceph_manager.c2:wait_until_healthy done 2023-05-31T19:48:54.229 INFO:teuthology.run_tasks:Running task rgw... 2023-05-31T19:48:54.250 DEBUG:tasks.rgw:config is {'c1.client.0': {'port': 8000}, 'c1.client.1': {'port': 8001}, 'c2.client.0': {'port': 8000}, 'c2.client.1': {'port': 8001}} 2023-05-31T19:48:54.251 DEBUG:tasks.rgw:client list is dict_keys(['c1.client.0', 'c1.client.1', 'c2.client.0', 'c2.client.1']) 2023-05-31T19:48:54.251 INFO:tasks.rgw:Creating data pools 2023-05-31T19:48:54.252 DEBUG:tasks.rgw:Obtaining remote for client c1.client.0 2023-05-31T19:48:54.253 DEBUG:teuthology.orchestra.run.smithi099:> sudo ceph osd pool create default.rgw.buckets.data 64 64 --cluster c1 2023-05-31T19:48:54.781 INFO:teuthology.orchestra.run.smithi099.stderr:pool 'default.rgw.buckets.data' created 2023-05-31T19:48:54.796 DEBUG:teuthology.orchestra.run.smithi099:> sudo ceph osd pool application enable default.rgw.buckets.data rgw --cluster c1 2023-05-31T19:48:55.806 INFO:teuthology.orchestra.run.smithi099.stderr:enabled application 'rgw' on pool 'default.rgw.buckets.data' 2023-05-31T19:48:55.816 DEBUG:teuthology.orchestra.run.smithi099:> sudo ceph osd pool create default.rgw.buckets.index 64 64 --cluster c1 2023-05-31T19:48:56.791 INFO:teuthology.orchestra.run.smithi099.stderr:pool 'default.rgw.buckets.index' created 2023-05-31T19:48:56.805 DEBUG:teuthology.orchestra.run.smithi099:> sudo ceph osd pool application enable default.rgw.buckets.index rgw --cluster c1 2023-05-31T19:48:57.799 INFO:teuthology.orchestra.run.smithi099.stderr:enabled application 'rgw' on pool 'default.rgw.buckets.index' 2023-05-31T19:48:57.820 DEBUG:tasks.rgw:Obtaining remote for client c1.client.1 2023-05-31T19:48:57.820 DEBUG:teuthology.orchestra.run.smithi099:> sudo ceph osd pool create default.rgw.buckets.data 64 64 --cluster c1 2023-05-31T19:48:58.142 INFO:teuthology.orchestra.run.smithi099.stderr:pool 'default.rgw.buckets.data' already exists 2023-05-31T19:48:58.153 DEBUG:teuthology.orchestra.run.smithi099:> sudo ceph osd pool application enable default.rgw.buckets.data rgw --cluster c1 2023-05-31T19:48:58.788 INFO:teuthology.orchestra.run.smithi099.stderr:enabled application 'rgw' on pool 'default.rgw.buckets.data' 2023-05-31T19:48:58.800 DEBUG:teuthology.orchestra.run.smithi099:> sudo ceph osd pool create default.rgw.buckets.index 64 64 --cluster c1 2023-05-31T19:48:59.123 INFO:teuthology.orchestra.run.smithi099.stderr:pool 'default.rgw.buckets.index' already exists 2023-05-31T19:48:59.134 DEBUG:teuthology.orchestra.run.smithi099:> sudo ceph osd pool application enable default.rgw.buckets.index rgw --cluster c1 2023-05-31T19:48:59.789 INFO:teuthology.orchestra.run.smithi099.stderr:enabled application 'rgw' on pool 'default.rgw.buckets.index' 2023-05-31T19:48:59.800 DEBUG:tasks.rgw:Obtaining remote for client c2.client.0 2023-05-31T19:48:59.801 DEBUG:teuthology.orchestra.run.smithi112:> sudo ceph osd pool create default.rgw.buckets.data 64 64 --cluster c2 2023-05-31T19:49:01.047 INFO:teuthology.orchestra.run.smithi112.stderr:pool 'default.rgw.buckets.data' created 2023-05-31T19:49:01.066 DEBUG:teuthology.orchestra.run.smithi112:> sudo ceph osd pool application enable default.rgw.buckets.data rgw --cluster c2 2023-05-31T19:49:02.058 INFO:teuthology.orchestra.run.smithi112.stderr:enabled application 'rgw' on pool 'default.rgw.buckets.data' 2023-05-31T19:49:02.080 DEBUG:teuthology.orchestra.run.smithi112:> sudo ceph osd pool create default.rgw.buckets.index 64 64 --cluster c2 2023-05-31T19:49:03.065 INFO:teuthology.orchestra.run.smithi112.stderr:pool 'default.rgw.buckets.index' created 2023-05-31T19:49:03.074 DEBUG:teuthology.orchestra.run.smithi112:> sudo ceph osd pool application enable default.rgw.buckets.index rgw --cluster c2 2023-05-31T19:49:04.071 INFO:teuthology.orchestra.run.smithi112.stderr:enabled application 'rgw' on pool 'default.rgw.buckets.index' 2023-05-31T19:49:04.090 DEBUG:tasks.rgw:Obtaining remote for client c2.client.1 2023-05-31T19:49:04.090 DEBUG:teuthology.orchestra.run.smithi112:> sudo ceph osd pool create default.rgw.buckets.data 64 64 --cluster c2 2023-05-31T19:49:04.415 INFO:teuthology.orchestra.run.smithi112.stderr:pool 'default.rgw.buckets.data' already exists 2023-05-31T19:49:04.426 DEBUG:teuthology.orchestra.run.smithi112:> sudo ceph osd pool application enable default.rgw.buckets.data rgw --cluster c2 2023-05-31T19:49:05.057 INFO:teuthology.orchestra.run.smithi112.stderr:enabled application 'rgw' on pool 'default.rgw.buckets.data' 2023-05-31T19:49:05.068 DEBUG:teuthology.orchestra.run.smithi112:> sudo ceph osd pool create default.rgw.buckets.index 64 64 --cluster c2 2023-05-31T19:49:05.390 INFO:teuthology.orchestra.run.smithi112.stderr:pool 'default.rgw.buckets.index' already exists 2023-05-31T19:49:05.401 DEBUG:teuthology.orchestra.run.smithi112:> sudo ceph osd pool application enable default.rgw.buckets.index rgw --cluster c2 2023-05-31T19:49:06.059 INFO:teuthology.orchestra.run.smithi112.stderr:enabled application 'rgw' on pool 'default.rgw.buckets.index' 2023-05-31T19:49:06.071 DEBUG:tasks.rgw:Pools created 2023-05-31T19:49:06.072 INFO:tasks.rgw:Configuring compression type = random 2023-05-31T19:49:06.072 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['user', 'list'] 2023-05-31T19:49:06.073 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'user', 'list'] 2023-05-31T19:49:06.074 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 user list 2023-05-31T19:49:06.200 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2023-05-31T19:49:06.201 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2023-05-31T19:49:06.212 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T19:49:06.207+0000 7fe1ad479780 20 rados->read ofs=0 len=0 2023-05-31T19:49:07.811 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T19:49:07.805+0000 7fe1ad479780 20 rados_obj.operate() r=-2 bl.length=0 2023-05-31T19:49:07.811 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T19:49:07.805+0000 7fe1ad479780 20 realm 2023-05-31T19:49:07.812 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T19:49:07.805+0000 7fe1ad479780 20 rados->read ofs=0 len=0 2023-05-31T19:49:07.812 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T19:49:07.806+0000 7fe1ad479780 20 rados_obj.operate() r=-2 bl.length=0 2023-05-31T19:49:07.812 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T19:49:07.806+0000 7fe1ad479780 4 RGWPeriod::init failed to init realm id : (2) No such file or directory 2023-05-31T19:49:07.812 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T19:49:07.806+0000 7fe1ad479780 20 rados->read ofs=0 len=0 2023-05-31T19:49:07.818 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T19:49:07.812+0000 7fe1ad479780 20 rados_obj.operate() r=-2 bl.length=0 2023-05-31T19:49:07.818 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T19:49:07.812+0000 7fe1ad479780 20 rados->read ofs=0 len=0 2023-05-31T19:49:07.818 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T19:49:07.813+0000 7fe1ad479780 20 rados_obj.operate() r=-2 bl.length=0 2023-05-31T19:49:07.819 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T19:49:07.813+0000 7fe1ad479780 20 rados->read ofs=0 len=0 2023-05-31T19:49:07.819 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T19:49:07.813+0000 7fe1ad479780 20 rados_obj.operate() r=-2 bl.length=0 2023-05-31T19:49:07.819 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T19:49:07.813+0000 7fe1ad479780 20 rados->read ofs=0 len=0 2023-05-31T19:49:07.819 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T19:49:07.814+0000 7fe1ad479780 20 rados_obj.operate() r=-2 bl.length=0 2023-05-31T19:49:07.820 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T19:49:07.814+0000 7fe1ad479780 10 Creating default zonegroup 2023-05-31T19:49:07.820 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T19:49:07.814+0000 7fe1ad479780 10 couldn't find old data placement pools config, setting up new ones for the zone 2023-05-31T19:49:07.830 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T19:49:07.825+0000 7fe1ad479780 10 WARNING: driver->list_zones() returned r=-2 2023-05-31T19:49:07.830 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T19:49:07.825+0000 7fe1ad479780 20 rados->read ofs=0 len=0 2023-05-31T19:49:07.830 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T19:49:07.825+0000 7fe1ad479780 20 rados_obj.operate() r=-2 bl.length=0 2023-05-31T19:49:07.836 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T19:49:07.831+0000 7fe1ad479780 20 rados->read ofs=0 len=0 2023-05-31T19:49:07.836 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T19:49:07.831+0000 7fe1ad479780 20 rados_obj.operate() r=-2 bl.length=0 2023-05-31T19:49:07.836 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T19:49:07.831+0000 7fe1ad479780 10 could not read realm id: (2) No such file or directory 2023-05-31T19:49:07.837 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T19:49:07.831+0000 7fe1ad479780 10 WARNING: failed to set zone as default, r=-22 2023-05-31T19:49:07.837 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T19:49:07.831+0000 7fe1ad479780 20 rados->read ofs=0 len=0 2023-05-31T19:49:07.837 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T19:49:07.831+0000 7fe1ad479780 20 rados_obj.operate() r=-2 bl.length=0 2023-05-31T19:49:07.840 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T19:49:07.835+0000 7fe1ad479780 20 rados->read ofs=0 len=0 2023-05-31T19:49:07.840 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T19:49:07.835+0000 7fe1ad479780 20 rados_obj.operate() r=0 bl.length=874 2023-05-31T19:49:07.840 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T19:49:07.835+0000 7fe1ad479780 20 rados->read ofs=0 len=0 2023-05-31T19:49:07.841 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T19:49:07.836+0000 7fe1ad479780 20 rados_obj.operate() r=0 bl.length=398 2023-05-31T19:49:07.841 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T19:49:07.836+0000 7fe1ad479780 10 Using default name default 2023-05-31T19:49:07.841 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T19:49:07.836+0000 7fe1ad479780 20 rados->read ofs=0 len=0 2023-05-31T19:49:07.842 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T19:49:07.837+0000 7fe1ad479780 20 rados_obj.operate() r=0 bl.length=46 2023-05-31T19:49:07.842 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T19:49:07.837+0000 7fe1ad479780 20 rados->read ofs=0 len=0 2023-05-31T19:49:07.842 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T19:49:07.837+0000 7fe1ad479780 20 rados_obj.operate() r=0 bl.length=874 2023-05-31T19:49:07.843 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T19:49:07.837+0000 7fe1ad479780 20 zone default found 2023-05-31T19:49:07.843 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T19:49:07.837+0000 7fe1ad479780 4 Realm: () 2023-05-31T19:49:07.843 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T19:49:07.837+0000 7fe1ad479780 4 ZoneGroup: default (620542c3-c1f7-4de8-bb00-51c7d315194d) 2023-05-31T19:49:07.843 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T19:49:07.837+0000 7fe1ad479780 4 Zone: default (08c1d690-6638-499a-b78d-3e3610481040) 2023-05-31T19:49:07.843 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T19:49:07.837+0000 7fe1ad479780 10 cannot find current period zonegroup using local zonegroup configuration 2023-05-31T19:49:07.843 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T19:49:07.837+0000 7fe1ad479780 20 zonegroup default 2023-05-31T19:49:07.843 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T19:49:07.837+0000 7fe1ad479780 20 rados->read ofs=0 len=0 2023-05-31T19:49:07.844 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T19:49:07.837+0000 7fe1ad479780 20 rados_obj.operate() r=-2 bl.length=0 2023-05-31T19:49:07.844 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T19:49:07.837+0000 7fe1ad479780 20 rados->read ofs=0 len=0 2023-05-31T19:49:09.834 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T19:49:09.829+0000 7fe1ad479780 20 rados_obj.operate() r=-2 bl.length=0 2023-05-31T19:49:09.834 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T19:49:09.829+0000 7fe1ad479780 20 rados->read ofs=0 len=0 2023-05-31T19:49:09.834 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T19:49:09.829+0000 7fe1ad479780 20 rados_obj.operate() r=-2 bl.length=0 2023-05-31T19:49:09.835 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T19:49:09.829+0000 7fe1ad479780 20 started sync module instance, tier type = 2023-05-31T19:49:09.835 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T19:49:09.829+0000 7fe1ad479780 20 started zone id=08c1d690-6638-499a-b78d-3e3610481040 (name=default) with tier type = 2023-05-31T19:49:11.865 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T19:49:11.860+0000 7fe1ad479780 20 add_watcher() i=0 2023-05-31T19:49:11.865 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T19:49:11.860+0000 7fe1ad479780 20 add_watcher() i=1 2023-05-31T19:49:11.866 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T19:49:11.860+0000 7fe1ad479780 20 add_watcher() i=2 2023-05-31T19:49:11.866 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T19:49:11.860+0000 7fe1ad479780 20 add_watcher() i=3 2023-05-31T19:49:11.866 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T19:49:11.860+0000 7fe1ad479780 20 add_watcher() i=4 2023-05-31T19:49:11.866 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T19:49:11.860+0000 7fe1ad479780 20 add_watcher() i=5 2023-05-31T19:49:11.866 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T19:49:11.860+0000 7fe1ad479780 20 add_watcher() i=6 2023-05-31T19:49:11.868 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T19:49:11.863+0000 7fe1ad479780 20 add_watcher() i=7 2023-05-31T19:49:11.868 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T19:49:11.863+0000 7fe1ad479780 2 all 8 watchers are set, enabling cache 2023-05-31T19:49:11.870 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T19:49:11.863+0000 7fe1ad479780 5 tl::expected, obj_version>, boost::system::error_code> logback_generations::read(const DoutPrefixProvider*, optional_yield):410: oid=data_loggenerations_metadata not found 2023-05-31T19:49:11.870 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T19:49:11.863+0000 7fe1ad479780 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):38 probing oid=data_log.0 2023-05-31T19:49:11.870 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T19:49:11.863+0000 7fe1ad479780 20 static int rgw::cls::fifo::FIFO::open(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::string, std::unique_ptr*, optional_yield, std::optional, bool):1168 entering 2023-05-31T19:49:11.871 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T19:49:11.863+0000 7fe1ad479780 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):60: oid=data_log.0 does not exist 2023-05-31T19:49:11.871 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T19:49:11.863+0000 7fe1ad479780 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):38 probing oid=data_log.1 2023-05-31T19:49:11.871 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T19:49:11.863+0000 7fe1ad479780 20 static int rgw::cls::fifo::FIFO::open(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::string, std::unique_ptr*, optional_yield, std::optional, bool):1168 entering 2023-05-31T19:49:11.872 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T19:49:11.864+0000 7fe1ad479780 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):60: oid=data_log.1 does not exist 2023-05-31T19:49:11.872 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T19:49:11.864+0000 7fe1ad479780 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):38 probing oid=data_log.2 2023-05-31T19:49:11.872 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T19:49:11.864+0000 7fe1ad479780 20 static int rgw::cls::fifo::FIFO::open(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::string, std::unique_ptr*, optional_yield, std::optional, bool):1168 entering 2023-05-31T19:49:11.873 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T19:49:11.864+0000 7fe1ad479780 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):60: oid=data_log.2 does not exist 2023-05-31T19:49:11.873 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T19:49:11.864+0000 7fe1ad479780 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):38 probing oid=data_log.3 2023-05-31T19:49:11.873 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T19:49:11.864+0000 7fe1ad479780 20 static int rgw::cls::fifo::FIFO::open(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::string, std::unique_ptr*, optional_yield, std::optional, bool):1168 entering 2023-05-31T19:49:11.873 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T19:49:11.864+0000 7fe1ad479780 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):60: oid=data_log.3 does not exist 2023-05-31T19:49:11.873 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T19:49:11.864+0000 7fe1ad479780 20 static int rgw::cls::fifo::FIFO::create(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::string, std::unique_ptr*, optional_yield, std::optional, std::optional >, bool, uint64_t, uint64_t):1210 entering 2023-05-31T19:49:11.874 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T19:49:11.865+0000 7fe1ad479780 20 static int rgw::cls::fifo::FIFO::open(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::string, std::unique_ptr*, optional_yield, std::optional, bool):1168 entering 2023-05-31T19:49:11.876 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T19:49:11.871+0000 7fe1923f5700 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2023-05-31T19:49:11.876 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T19:49:11.871+0000 7fe1ad479780 20 check_secure_mon_conn(): auth registy supported: methods=[2] modes=[2,1] 2023-05-31T19:49:11.876 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T19:49:11.871+0000 7fe1ad479780 20 check_secure_mon_conn(): mode 1 is insecure 2023-05-31T19:49:14.834 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T19:49:14.829+0000 7fe1ad479780 10 rgw_init_ioctx warning: failed to set recovery_priority on default.rgw.meta 2023-05-31T19:49:14.834 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T19:49:14.829+0000 7fe1ad479780 5 note: GC not initialized 2023-05-31T19:49:14.836 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T19:49:14.830+0000 7fe18dbec700 20 reqs_thread_entry: start 2023-05-31T19:49:14.836 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T19:49:14.830+0000 7fe1ad479780 20 init_complete bucket index max shards: 11 2023-05-31T19:49:14.836 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T19:49:14.830+0000 7fe1ad479780 10 rgw notify: Started notification manager with: 1 workers 2023-05-31T19:49:14.836 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T19:49:14.831+0000 7fe18bbe8700 20 rgw notify: INFO: next queues processing will happen at: Wed May 31 19:49:44 2023 2023-05-31T19:49:14.837 INFO:teuthology.orchestra.run.smithi099.stderr: 2023-05-31T19:49:14.837 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T19:49:14.831+0000 7fe18b3c6700 20 reqs_thread_entry: start 2023-05-31T19:49:14.857 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T19:49:14.853+0000 7fe1923f5700 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2023-05-31T19:49:14.859 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T19:49:14.855+0000 7fe1ad479780 20 remove_watcher() i=0 2023-05-31T19:49:14.860 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T19:49:14.855+0000 7fe1ad479780 2 removed watcher, disabling cache 2023-05-31T19:49:14.860 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T19:49:14.856+0000 7fe1ad479780 20 remove_watcher() i=1 2023-05-31T19:49:14.861 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T19:49:14.857+0000 7fe1ad479780 20 remove_watcher() i=2 2023-05-31T19:49:14.862 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T19:49:14.858+0000 7fe1ad479780 20 remove_watcher() i=3 2023-05-31T19:49:14.863 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T19:49:14.859+0000 7fe1ad479780 20 remove_watcher() i=4 2023-05-31T19:49:14.864 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T19:49:14.860+0000 7fe1ad479780 20 remove_watcher() i=5 2023-05-31T19:49:14.865 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T19:49:14.860+0000 7fe1ad479780 20 remove_watcher() i=6 2023-05-31T19:49:14.868 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T19:49:14.863+0000 7fe1ad479780 20 remove_watcher() i=7 2023-05-31T19:49:14.871 INFO:teuthology.orchestra.run.smithi099.stdout:[] 2023-05-31T19:49:14.873 DEBUG:tasks.util.rgw: json result: [] 2023-05-31T19:49:14.873 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['zone', 'placement', 'modify', '--rgw-zone', 'default', '--placement-id', 'default-placement', '--compression', 'random'] 2023-05-31T19:49:14.873 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'zone', 'placement', 'modify', '--rgw-zone', 'default', '--placement-id', 'default-placement', '--compression', 'random'] 2023-05-31T19:49:14.874 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 zone placement modify --rgw-zone default --placement-id default-placement --compression random 2023-05-31T19:49:14.927 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2023-05-31T19:49:14.928 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2023-05-31T19:49:14.936 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T19:49:14.931+0000 7f2f454e6780 20 check_secure_mon_conn(): auth registy supported: methods=[2] modes=[2,1] 2023-05-31T19:49:14.937 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T19:49:14.931+0000 7f2f454e6780 20 check_secure_mon_conn(): mode 1 is insecure 2023-05-31T19:49:14.940 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T19:49:14.935+0000 7f2f26c5b700 20 reqs_thread_entry: start 2023-05-31T19:49:14.947 INFO:teuthology.orchestra.run.smithi099.stdout:{ 2023-05-31T19:49:14.947 INFO:teuthology.orchestra.run.smithi099.stdout: "id": "08c1d690-6638-499a-b78d-3e3610481040", 2023-05-31T19:49:14.947 INFO:teuthology.orchestra.run.smithi099.stdout: "name": "default", 2023-05-31T19:49:14.948 INFO:teuthology.orchestra.run.smithi099.stdout: "domain_root": "default.rgw.meta:root", 2023-05-31T19:49:14.948 INFO:teuthology.orchestra.run.smithi099.stdout: "control_pool": "default.rgw.control", 2023-05-31T19:49:14.948 INFO:teuthology.orchestra.run.smithi099.stdout: "gc_pool": "default.rgw.log:gc", 2023-05-31T19:49:14.948 INFO:teuthology.orchestra.run.smithi099.stdout: "lc_pool": "default.rgw.log:lc", 2023-05-31T19:49:14.948 INFO:teuthology.orchestra.run.smithi099.stdout: "log_pool": "default.rgw.log", 2023-05-31T19:49:14.948 INFO:teuthology.orchestra.run.smithi099.stdout: "intent_log_pool": "default.rgw.log:intent", 2023-05-31T19:49:14.948 INFO:teuthology.orchestra.run.smithi099.stdout: "usage_log_pool": "default.rgw.log:usage", 2023-05-31T19:49:14.949 INFO:teuthology.orchestra.run.smithi099.stdout: "roles_pool": "default.rgw.meta:roles", 2023-05-31T19:49:14.949 INFO:teuthology.orchestra.run.smithi099.stdout: "reshard_pool": "default.rgw.log:reshard", 2023-05-31T19:49:14.949 INFO:teuthology.orchestra.run.smithi099.stdout: "user_keys_pool": "default.rgw.meta:users.keys", 2023-05-31T19:49:14.949 INFO:teuthology.orchestra.run.smithi099.stdout: "user_email_pool": "default.rgw.meta:users.email", 2023-05-31T19:49:14.949 INFO:teuthology.orchestra.run.smithi099.stdout: "user_swift_pool": "default.rgw.meta:users.swift", 2023-05-31T19:49:14.949 INFO:teuthology.orchestra.run.smithi099.stdout: "user_uid_pool": "default.rgw.meta:users.uid", 2023-05-31T19:49:14.949 INFO:teuthology.orchestra.run.smithi099.stdout: "otp_pool": "default.rgw.otp", 2023-05-31T19:49:14.949 INFO:teuthology.orchestra.run.smithi099.stdout: "system_key": { 2023-05-31T19:49:14.950 INFO:teuthology.orchestra.run.smithi099.stdout: "access_key": "", 2023-05-31T19:49:14.950 INFO:teuthology.orchestra.run.smithi099.stdout: "secret_key": "" 2023-05-31T19:49:14.950 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T19:49:14.950 INFO:teuthology.orchestra.run.smithi099.stdout: "placement_pools": [ 2023-05-31T19:49:14.950 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T19:49:14.950 INFO:teuthology.orchestra.run.smithi099.stdout: "key": "default-placement", 2023-05-31T19:49:14.950 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T19:49:14.951 INFO:teuthology.orchestra.run.smithi099.stdout: "index_pool": "default.rgw.buckets.index", 2023-05-31T19:49:14.951 INFO:teuthology.orchestra.run.smithi099.stdout: "storage_classes": { 2023-05-31T19:49:14.951 INFO:teuthology.orchestra.run.smithi099.stdout: "STANDARD": { 2023-05-31T19:49:14.951 INFO:teuthology.orchestra.run.smithi099.stdout: "data_pool": "default.rgw.buckets.data", 2023-05-31T19:49:14.951 INFO:teuthology.orchestra.run.smithi099.stdout: "compression_type": "random" 2023-05-31T19:49:14.951 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T19:49:14.951 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T19:49:14.952 INFO:teuthology.orchestra.run.smithi099.stdout: "data_extra_pool": "default.rgw.buckets.non-ec", 2023-05-31T19:49:14.952 INFO:teuthology.orchestra.run.smithi099.stdout: "index_type": 0, 2023-05-31T19:49:14.952 INFO:teuthology.orchestra.run.smithi099.stdout: "inline_data": true 2023-05-31T19:49:14.952 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T19:49:14.952 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T19:49:14.952 INFO:teuthology.orchestra.run.smithi099.stdout: ], 2023-05-31T19:49:14.952 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_id": "", 2023-05-31T19:49:14.952 INFO:teuthology.orchestra.run.smithi099.stdout: "notif_pool": "default.rgw.log:notif" 2023-05-31T19:49:14.953 INFO:teuthology.orchestra.run.smithi099.stdout:} 2023-05-31T19:49:14.954 DEBUG:tasks.util.rgw: json result: {'id': '08c1d690-6638-499a-b78d-3e3610481040', 'name': 'default', 'domain_root': 'default.rgw.meta:root', 'control_pool': 'default.rgw.control', 'gc_pool': 'default.rgw.log:gc', 'lc_pool': 'default.rgw.log:lc', 'log_pool': 'default.rgw.log', 'intent_log_pool': 'default.rgw.log:intent', 'usage_log_pool': 'default.rgw.log:usage', 'roles_pool': 'default.rgw.meta:roles', 'reshard_pool': 'default.rgw.log:reshard', 'user_keys_pool': 'default.rgw.meta:users.keys', 'user_email_pool': 'default.rgw.meta:users.email', 'user_swift_pool': 'default.rgw.meta:users.swift', 'user_uid_pool': 'default.rgw.meta:users.uid', 'otp_pool': 'default.rgw.otp', 'system_key': {'access_key': '', 'secret_key': ''}, 'placement_pools': [{'key': 'default-placement', 'val': {'index_pool': 'default.rgw.buckets.index', 'storage_classes': {'STANDARD': {'data_pool': 'default.rgw.buckets.data', 'compression_type': 'random'}}, 'data_extra_pool': 'default.rgw.buckets.non-ec', 'index_type': 0, 'inline_data': True}}], 'realm_id': '', 'notif_pool': 'default.rgw.log:notif'} 2023-05-31T19:49:14.954 INFO:tasks.util.rgw:rgwadmin: c1.client.1 : ['user', 'list'] 2023-05-31T19:49:14.954 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.1', '--cluster', 'c1', 'user', 'list'] 2023-05-31T19:49:14.955 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.1 --cluster c1 user list 2023-05-31T19:49:15.000 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2023-05-31T19:49:15.001 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2023-05-31T19:49:15.009 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T19:49:15.004+0000 7fa45d253780 20 rados->read ofs=0 len=0 2023-05-31T19:49:15.010 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T19:49:15.005+0000 7fa45d253780 20 rados_obj.operate() r=-2 bl.length=0 2023-05-31T19:49:15.011 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T19:49:15.005+0000 7fa45d253780 20 realm 2023-05-31T19:49:15.011 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T19:49:15.005+0000 7fa45d253780 20 rados->read ofs=0 len=0 2023-05-31T19:49:15.011 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T19:49:15.005+0000 7fa45d253780 20 rados_obj.operate() r=-2 bl.length=0 2023-05-31T19:49:15.012 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T19:49:15.005+0000 7fa45d253780 4 RGWPeriod::init failed to init realm id : (2) No such file or directory 2023-05-31T19:49:15.012 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T19:49:15.005+0000 7fa45d253780 20 rados->read ofs=0 len=0 2023-05-31T19:49:15.012 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T19:49:15.005+0000 7fa45d253780 20 rados_obj.operate() r=-2 bl.length=0 2023-05-31T19:49:15.012 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T19:49:15.005+0000 7fa45d253780 20 rados->read ofs=0 len=0 2023-05-31T19:49:15.013 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T19:49:15.005+0000 7fa45d253780 20 rados_obj.operate() r=0 bl.length=46 2023-05-31T19:49:15.013 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T19:49:15.005+0000 7fa45d253780 20 rados->read ofs=0 len=0 2023-05-31T19:49:15.013 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T19:49:15.006+0000 7fa45d253780 20 rados_obj.operate() r=0 bl.length=890 2023-05-31T19:49:15.013 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T19:49:15.006+0000 7fa45d253780 20 searching for the correct realm 2023-05-31T19:49:15.019 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T19:49:15.013+0000 7fa45d253780 20 RGWRados::pool_iterate: got zone_info.08c1d690-6638-499a-b78d-3e3610481040 2023-05-31T19:49:15.019 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T19:49:15.013+0000 7fa45d253780 20 RGWRados::pool_iterate: got zonegroup_info.620542c3-c1f7-4de8-bb00-51c7d315194d 2023-05-31T19:49:15.020 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T19:49:15.013+0000 7fa45d253780 20 RGWRados::pool_iterate: got zone_names.default 2023-05-31T19:49:15.020 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T19:49:15.013+0000 7fa45d253780 20 RGWRados::pool_iterate: got zonegroups_names.default 2023-05-31T19:49:15.020 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T19:49:15.013+0000 7fa45d253780 20 rados->read ofs=0 len=0 2023-05-31T19:49:15.020 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T19:49:15.013+0000 7fa45d253780 20 rados_obj.operate() r=-2 bl.length=0 2023-05-31T19:49:15.020 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T19:49:15.013+0000 7fa45d253780 20 rados->read ofs=0 len=0 2023-05-31T19:49:15.021 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T19:49:15.013+0000 7fa45d253780 20 rados_obj.operate() r=0 bl.length=46 2023-05-31T19:49:15.021 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T19:49:15.013+0000 7fa45d253780 20 rados->read ofs=0 len=0 2023-05-31T19:49:15.021 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T19:49:15.013+0000 7fa45d253780 20 rados_obj.operate() r=0 bl.length=398 2023-05-31T19:49:15.021 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T19:49:15.013+0000 7fa45d253780 20 zone default found 2023-05-31T19:49:15.021 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T19:49:15.013+0000 7fa45d253780 4 Realm: () 2023-05-31T19:49:15.021 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T19:49:15.013+0000 7fa45d253780 4 ZoneGroup: default (620542c3-c1f7-4de8-bb00-51c7d315194d) 2023-05-31T19:49:15.022 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T19:49:15.013+0000 7fa45d253780 4 Zone: default (08c1d690-6638-499a-b78d-3e3610481040) 2023-05-31T19:49:15.022 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T19:49:15.013+0000 7fa45d253780 10 cannot find current period zonegroup using local zonegroup configuration 2023-05-31T19:49:15.022 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T19:49:15.013+0000 7fa45d253780 20 zonegroup default 2023-05-31T19:49:15.022 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T19:49:15.013+0000 7fa45d253780 20 rados->read ofs=0 len=0 2023-05-31T19:49:15.022 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T19:49:15.014+0000 7fa45d253780 20 rados_obj.operate() r=-2 bl.length=0 2023-05-31T19:49:15.022 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T19:49:15.014+0000 7fa45d253780 20 rados->read ofs=0 len=0 2023-05-31T19:49:15.023 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T19:49:15.014+0000 7fa45d253780 20 rados_obj.operate() r=-2 bl.length=0 2023-05-31T19:49:15.023 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T19:49:15.014+0000 7fa45d253780 20 rados->read ofs=0 len=0 2023-05-31T19:49:15.023 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T19:49:15.014+0000 7fa45d253780 20 rados_obj.operate() r=-2 bl.length=0 2023-05-31T19:49:15.023 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T19:49:15.014+0000 7fa45d253780 20 started sync module instance, tier type = 2023-05-31T19:49:15.023 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T19:49:15.014+0000 7fa45d253780 20 started zone id=08c1d690-6638-499a-b78d-3e3610481040 (name=default) with tier type = 2023-05-31T19:49:15.045 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T19:49:15.039+0000 7fa45d253780 20 add_watcher() i=0 2023-05-31T19:49:15.045 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T19:49:15.039+0000 7fa45d253780 20 add_watcher() i=1 2023-05-31T19:49:15.045 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T19:49:15.039+0000 7fa45d253780 20 add_watcher() i=2 2023-05-31T19:49:15.046 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T19:49:15.039+0000 7fa45d253780 20 add_watcher() i=3 2023-05-31T19:49:15.046 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T19:49:15.039+0000 7fa45d253780 20 add_watcher() i=4 2023-05-31T19:49:15.046 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T19:49:15.039+0000 7fa45d253780 20 add_watcher() i=5 2023-05-31T19:49:15.046 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T19:49:15.039+0000 7fa45d253780 20 add_watcher() i=6 2023-05-31T19:49:15.047 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T19:49:15.040+0000 7fa45d253780 20 add_watcher() i=7 2023-05-31T19:49:15.047 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T19:49:15.040+0000 7fa45d253780 2 all 8 watchers are set, enabling cache 2023-05-31T19:49:15.048 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T19:49:15.043+0000 7fa4421cf700 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2023-05-31T19:49:15.048 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T19:49:15.043+0000 7fa45d253780 20 check_secure_mon_conn(): auth registy supported: methods=[2] modes=[2,1] 2023-05-31T19:49:15.049 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T19:49:15.043+0000 7fa45d253780 20 check_secure_mon_conn(): mode 1 is insecure 2023-05-31T19:49:15.052 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T19:49:15.047+0000 7fa45d253780 5 note: GC not initialized 2023-05-31T19:49:15.053 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T19:49:15.048+0000 7fa43d9c6700 20 reqs_thread_entry: start 2023-05-31T19:49:15.053 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T19:49:15.048+0000 7fa45d253780 20 init_complete bucket index max shards: 11 2023-05-31T19:49:15.053 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T19:49:15.048+0000 7fa45d253780 10 rgw notify: Started notification manager with: 1 workers 2023-05-31T19:49:15.053 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T19:49:15.048+0000 7fa43b9c2700 20 rgw notify: INFO: next queues processing will happen at: Wed May 31 19:49:45 2023 2023-05-31T19:49:15.053 INFO:teuthology.orchestra.run.smithi099.stderr: 2023-05-31T19:49:15.054 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T19:49:15.048+0000 7fa43b1a0700 20 reqs_thread_entry: start 2023-05-31T19:49:15.060 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T19:49:15.055+0000 7fa4421cf700 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2023-05-31T19:49:15.063 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T19:49:15.059+0000 7fa45d253780 20 remove_watcher() i=0 2023-05-31T19:49:15.064 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T19:49:15.059+0000 7fa45d253780 2 removed watcher, disabling cache 2023-05-31T19:49:15.064 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T19:49:15.059+0000 7fa45d253780 20 remove_watcher() i=1 2023-05-31T19:49:15.066 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T19:49:15.061+0000 7fa45d253780 20 remove_watcher() i=2 2023-05-31T19:49:15.069 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T19:49:15.064+0000 7fa45d253780 20 remove_watcher() i=3 2023-05-31T19:49:15.072 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T19:49:15.067+0000 7fa45d253780 20 remove_watcher() i=4 2023-05-31T19:49:15.073 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T19:49:15.067+0000 7fa45d253780 20 remove_watcher() i=5 2023-05-31T19:49:15.075 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T19:49:15.070+0000 7fa45d253780 20 remove_watcher() i=6 2023-05-31T19:49:15.076 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T19:49:15.071+0000 7fa45d253780 20 remove_watcher() i=7 2023-05-31T19:49:15.078 INFO:teuthology.orchestra.run.smithi099.stdout:[] 2023-05-31T19:49:15.079 DEBUG:tasks.util.rgw: json result: [] 2023-05-31T19:49:15.080 INFO:tasks.util.rgw:rgwadmin: c1.client.1 : ['zone', 'placement', 'modify', '--rgw-zone', 'default', '--placement-id', 'default-placement', '--compression', 'random'] 2023-05-31T19:49:15.080 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.1', '--cluster', 'c1', 'zone', 'placement', 'modify', '--rgw-zone', 'default', '--placement-id', 'default-placement', '--compression', 'random'] 2023-05-31T19:49:15.080 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.1 --cluster c1 zone placement modify --rgw-zone default --placement-id default-placement --compression random 2023-05-31T19:49:15.131 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2023-05-31T19:49:15.132 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2023-05-31T19:49:15.139 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T19:49:15.134+0000 7fa636720780 20 check_secure_mon_conn(): auth registy supported: methods=[2] modes=[2,1] 2023-05-31T19:49:15.140 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T19:49:15.134+0000 7fa636720780 20 check_secure_mon_conn(): mode 1 is insecure 2023-05-31T19:49:15.142 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T19:49:15.137+0000 7fa617e95700 20 reqs_thread_entry: start 2023-05-31T19:49:15.148 INFO:teuthology.orchestra.run.smithi099.stdout:{ 2023-05-31T19:49:15.149 INFO:teuthology.orchestra.run.smithi099.stdout: "id": "08c1d690-6638-499a-b78d-3e3610481040", 2023-05-31T19:49:15.149 INFO:teuthology.orchestra.run.smithi099.stdout: "name": "default", 2023-05-31T19:49:15.149 INFO:teuthology.orchestra.run.smithi099.stdout: "domain_root": "default.rgw.meta:root", 2023-05-31T19:49:15.149 INFO:teuthology.orchestra.run.smithi099.stdout: "control_pool": "default.rgw.control", 2023-05-31T19:49:15.149 INFO:teuthology.orchestra.run.smithi099.stdout: "gc_pool": "default.rgw.log:gc", 2023-05-31T19:49:15.150 INFO:teuthology.orchestra.run.smithi099.stdout: "lc_pool": "default.rgw.log:lc", 2023-05-31T19:49:15.150 INFO:teuthology.orchestra.run.smithi099.stdout: "log_pool": "default.rgw.log", 2023-05-31T19:49:15.150 INFO:teuthology.orchestra.run.smithi099.stdout: "intent_log_pool": "default.rgw.log:intent", 2023-05-31T19:49:15.150 INFO:teuthology.orchestra.run.smithi099.stdout: "usage_log_pool": "default.rgw.log:usage", 2023-05-31T19:49:15.150 INFO:teuthology.orchestra.run.smithi099.stdout: "roles_pool": "default.rgw.meta:roles", 2023-05-31T19:49:15.150 INFO:teuthology.orchestra.run.smithi099.stdout: "reshard_pool": "default.rgw.log:reshard", 2023-05-31T19:49:15.150 INFO:teuthology.orchestra.run.smithi099.stdout: "user_keys_pool": "default.rgw.meta:users.keys", 2023-05-31T19:49:15.150 INFO:teuthology.orchestra.run.smithi099.stdout: "user_email_pool": "default.rgw.meta:users.email", 2023-05-31T19:49:15.151 INFO:teuthology.orchestra.run.smithi099.stdout: "user_swift_pool": "default.rgw.meta:users.swift", 2023-05-31T19:49:15.151 INFO:teuthology.orchestra.run.smithi099.stdout: "user_uid_pool": "default.rgw.meta:users.uid", 2023-05-31T19:49:15.151 INFO:teuthology.orchestra.run.smithi099.stdout: "otp_pool": "default.rgw.otp", 2023-05-31T19:49:15.151 INFO:teuthology.orchestra.run.smithi099.stdout: "system_key": { 2023-05-31T19:49:15.151 INFO:teuthology.orchestra.run.smithi099.stdout: "access_key": "", 2023-05-31T19:49:15.151 INFO:teuthology.orchestra.run.smithi099.stdout: "secret_key": "" 2023-05-31T19:49:15.151 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T19:49:15.152 INFO:teuthology.orchestra.run.smithi099.stdout: "placement_pools": [ 2023-05-31T19:49:15.152 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T19:49:15.152 INFO:teuthology.orchestra.run.smithi099.stdout: "key": "default-placement", 2023-05-31T19:49:15.152 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T19:49:15.152 INFO:teuthology.orchestra.run.smithi099.stdout: "index_pool": "default.rgw.buckets.index", 2023-05-31T19:49:15.152 INFO:teuthology.orchestra.run.smithi099.stdout: "storage_classes": { 2023-05-31T19:49:15.153 INFO:teuthology.orchestra.run.smithi099.stdout: "STANDARD": { 2023-05-31T19:49:15.153 INFO:teuthology.orchestra.run.smithi099.stdout: "data_pool": "default.rgw.buckets.data", 2023-05-31T19:49:15.153 INFO:teuthology.orchestra.run.smithi099.stdout: "compression_type": "random" 2023-05-31T19:49:15.153 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T19:49:15.153 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T19:49:15.154 INFO:teuthology.orchestra.run.smithi099.stdout: "data_extra_pool": "default.rgw.buckets.non-ec", 2023-05-31T19:49:15.154 INFO:teuthology.orchestra.run.smithi099.stdout: "index_type": 0, 2023-05-31T19:49:15.154 INFO:teuthology.orchestra.run.smithi099.stdout: "inline_data": true 2023-05-31T19:49:15.154 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T19:49:15.154 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T19:49:15.154 INFO:teuthology.orchestra.run.smithi099.stdout: ], 2023-05-31T19:49:15.154 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_id": "", 2023-05-31T19:49:15.155 INFO:teuthology.orchestra.run.smithi099.stdout: "notif_pool": "default.rgw.log:notif" 2023-05-31T19:49:15.155 INFO:teuthology.orchestra.run.smithi099.stdout:} 2023-05-31T19:49:15.156 DEBUG:tasks.util.rgw: json result: {'id': '08c1d690-6638-499a-b78d-3e3610481040', 'name': 'default', 'domain_root': 'default.rgw.meta:root', 'control_pool': 'default.rgw.control', 'gc_pool': 'default.rgw.log:gc', 'lc_pool': 'default.rgw.log:lc', 'log_pool': 'default.rgw.log', 'intent_log_pool': 'default.rgw.log:intent', 'usage_log_pool': 'default.rgw.log:usage', 'roles_pool': 'default.rgw.meta:roles', 'reshard_pool': 'default.rgw.log:reshard', 'user_keys_pool': 'default.rgw.meta:users.keys', 'user_email_pool': 'default.rgw.meta:users.email', 'user_swift_pool': 'default.rgw.meta:users.swift', 'user_uid_pool': 'default.rgw.meta:users.uid', 'otp_pool': 'default.rgw.otp', 'system_key': {'access_key': '', 'secret_key': ''}, 'placement_pools': [{'key': 'default-placement', 'val': {'index_pool': 'default.rgw.buckets.index', 'storage_classes': {'STANDARD': {'data_pool': 'default.rgw.buckets.data', 'compression_type': 'random'}}, 'data_extra_pool': 'default.rgw.buckets.non-ec', 'index_type': 0, 'inline_data': True}}], 'realm_id': '', 'notif_pool': 'default.rgw.log:notif'} 2023-05-31T19:49:15.156 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['user', 'list'] 2023-05-31T19:49:15.156 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'user', 'list'] 2023-05-31T19:49:15.158 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 user list 2023-05-31T19:49:15.282 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-05-31T19:49:15.283 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-05-31T19:49:15.293 INFO:teuthology.orchestra.run.smithi112.stderr:2023-05-31T19:49:15.288+0000 7f6690662780 20 rados->read ofs=0 len=0 2023-05-31T19:49:17.185 INFO:teuthology.orchestra.run.smithi112.stderr:2023-05-31T19:49:17.178+0000 7f6690662780 20 rados_obj.operate() r=-2 bl.length=0 2023-05-31T19:49:17.185 INFO:teuthology.orchestra.run.smithi112.stderr:2023-05-31T19:49:17.178+0000 7f6690662780 20 realm 2023-05-31T19:49:17.186 INFO:teuthology.orchestra.run.smithi112.stderr:2023-05-31T19:49:17.179+0000 7f6690662780 20 rados->read ofs=0 len=0 2023-05-31T19:49:17.186 INFO:teuthology.orchestra.run.smithi112.stderr:2023-05-31T19:49:17.179+0000 7f6690662780 20 rados_obj.operate() r=-2 bl.length=0 2023-05-31T19:49:17.186 INFO:teuthology.orchestra.run.smithi112.stderr:2023-05-31T19:49:17.179+0000 7f6690662780 4 RGWPeriod::init failed to init realm id : (2) No such file or directory 2023-05-31T19:49:17.186 INFO:teuthology.orchestra.run.smithi112.stderr:2023-05-31T19:49:17.179+0000 7f6690662780 20 rados->read ofs=0 len=0 2023-05-31T19:49:17.186 INFO:teuthology.orchestra.run.smithi112.stderr:2023-05-31T19:49:17.179+0000 7f6690662780 20 rados_obj.operate() r=-2 bl.length=0 2023-05-31T19:49:17.186 INFO:teuthology.orchestra.run.smithi112.stderr:2023-05-31T19:49:17.179+0000 7f6690662780 20 rados->read ofs=0 len=0 2023-05-31T19:49:17.188 INFO:teuthology.orchestra.run.smithi112.stderr:2023-05-31T19:49:17.180+0000 7f6690662780 20 rados_obj.operate() r=-2 bl.length=0 2023-05-31T19:49:17.188 INFO:teuthology.orchestra.run.smithi112.stderr:2023-05-31T19:49:17.180+0000 7f6690662780 20 rados->read ofs=0 len=0 2023-05-31T19:49:17.188 INFO:teuthology.orchestra.run.smithi112.stderr:2023-05-31T19:49:17.180+0000 7f6690662780 20 rados_obj.operate() r=-2 bl.length=0 2023-05-31T19:49:17.188 INFO:teuthology.orchestra.run.smithi112.stderr:2023-05-31T19:49:17.180+0000 7f6690662780 20 rados->read ofs=0 len=0 2023-05-31T19:49:17.188 INFO:teuthology.orchestra.run.smithi112.stderr:2023-05-31T19:49:17.180+0000 7f6690662780 20 rados_obj.operate() r=-2 bl.length=0 2023-05-31T19:49:17.188 INFO:teuthology.orchestra.run.smithi112.stderr:2023-05-31T19:49:17.180+0000 7f6690662780 10 Creating default zonegroup 2023-05-31T19:49:17.189 INFO:teuthology.orchestra.run.smithi112.stderr:2023-05-31T19:49:17.180+0000 7f6690662780 10 couldn't find old data placement pools config, setting up new ones for the zone 2023-05-31T19:49:17.196 INFO:teuthology.orchestra.run.smithi112.stderr:2023-05-31T19:49:17.190+0000 7f6690662780 10 WARNING: driver->list_zones() returned r=-2 2023-05-31T19:49:17.196 INFO:teuthology.orchestra.run.smithi112.stderr:2023-05-31T19:49:17.190+0000 7f6690662780 20 rados->read ofs=0 len=0 2023-05-31T19:49:17.196 INFO:teuthology.orchestra.run.smithi112.stderr:2023-05-31T19:49:17.190+0000 7f6690662780 20 rados_obj.operate() r=-2 bl.length=0 2023-05-31T19:49:17.198 INFO:teuthology.orchestra.run.smithi112.stderr:2023-05-31T19:49:17.192+0000 7f6690662780 20 rados->read ofs=0 len=0 2023-05-31T19:49:17.198 INFO:teuthology.orchestra.run.smithi112.stderr:2023-05-31T19:49:17.192+0000 7f6690662780 20 rados_obj.operate() r=-2 bl.length=0 2023-05-31T19:49:17.198 INFO:teuthology.orchestra.run.smithi112.stderr:2023-05-31T19:49:17.192+0000 7f6690662780 10 could not read realm id: (2) No such file or directory 2023-05-31T19:49:17.199 INFO:teuthology.orchestra.run.smithi112.stderr:2023-05-31T19:49:17.192+0000 7f6690662780 10 WARNING: failed to set zone as default, r=-22 2023-05-31T19:49:17.199 INFO:teuthology.orchestra.run.smithi112.stderr:2023-05-31T19:49:17.192+0000 7f6690662780 20 rados->read ofs=0 len=0 2023-05-31T19:49:17.199 INFO:teuthology.orchestra.run.smithi112.stderr:2023-05-31T19:49:17.193+0000 7f6690662780 20 rados_obj.operate() r=-2 bl.length=0 2023-05-31T19:49:17.202 INFO:teuthology.orchestra.run.smithi112.stderr:2023-05-31T19:49:17.197+0000 7f6690662780 20 rados->read ofs=0 len=0 2023-05-31T19:49:17.203 INFO:teuthology.orchestra.run.smithi112.stderr:2023-05-31T19:49:17.197+0000 7f6690662780 20 rados_obj.operate() r=0 bl.length=874 2023-05-31T19:49:17.203 INFO:teuthology.orchestra.run.smithi112.stderr:2023-05-31T19:49:17.197+0000 7f6690662780 20 rados->read ofs=0 len=0 2023-05-31T19:49:17.203 INFO:teuthology.orchestra.run.smithi112.stderr:2023-05-31T19:49:17.197+0000 7f6690662780 20 rados_obj.operate() r=0 bl.length=398 2023-05-31T19:49:17.204 INFO:teuthology.orchestra.run.smithi112.stderr:2023-05-31T19:49:17.197+0000 7f6690662780 10 Using default name default 2023-05-31T19:49:17.204 INFO:teuthology.orchestra.run.smithi112.stderr:2023-05-31T19:49:17.197+0000 7f6690662780 20 rados->read ofs=0 len=0 2023-05-31T19:49:17.204 INFO:teuthology.orchestra.run.smithi112.stderr:2023-05-31T19:49:17.198+0000 7f6690662780 20 rados_obj.operate() r=0 bl.length=46 2023-05-31T19:49:17.204 INFO:teuthology.orchestra.run.smithi112.stderr:2023-05-31T19:49:17.198+0000 7f6690662780 20 rados->read ofs=0 len=0 2023-05-31T19:49:17.205 INFO:teuthology.orchestra.run.smithi112.stderr:2023-05-31T19:49:17.198+0000 7f6690662780 20 rados_obj.operate() r=0 bl.length=874 2023-05-31T19:49:17.205 INFO:teuthology.orchestra.run.smithi112.stderr:2023-05-31T19:49:17.198+0000 7f6690662780 20 zone default found 2023-05-31T19:49:17.205 INFO:teuthology.orchestra.run.smithi112.stderr:2023-05-31T19:49:17.198+0000 7f6690662780 4 Realm: () 2023-05-31T19:49:17.205 INFO:teuthology.orchestra.run.smithi112.stderr:2023-05-31T19:49:17.198+0000 7f6690662780 4 ZoneGroup: default (39908ee0-a1c5-4a7d-8b2d-28f0fca3c8cb) 2023-05-31T19:49:17.205 INFO:teuthology.orchestra.run.smithi112.stderr:2023-05-31T19:49:17.198+0000 7f6690662780 4 Zone: default (45687fb7-b52f-4449-8e12-6fd6afd32a53) 2023-05-31T19:49:17.205 INFO:teuthology.orchestra.run.smithi112.stderr:2023-05-31T19:49:17.198+0000 7f6690662780 10 cannot find current period zonegroup using local zonegroup configuration 2023-05-31T19:49:17.206 INFO:teuthology.orchestra.run.smithi112.stderr:2023-05-31T19:49:17.198+0000 7f6690662780 20 zonegroup default 2023-05-31T19:49:17.206 INFO:teuthology.orchestra.run.smithi112.stderr:2023-05-31T19:49:17.198+0000 7f6690662780 20 rados->read ofs=0 len=0 2023-05-31T19:49:17.206 INFO:teuthology.orchestra.run.smithi112.stderr:2023-05-31T19:49:17.198+0000 7f6690662780 20 rados_obj.operate() r=-2 bl.length=0 2023-05-31T19:49:17.206 INFO:teuthology.orchestra.run.smithi112.stderr:2023-05-31T19:49:17.198+0000 7f6690662780 20 rados->read ofs=0 len=0 2023-05-31T19:49:19.200 INFO:teuthology.orchestra.run.smithi112.stderr:2023-05-31T19:49:19.194+0000 7f6690662780 20 rados_obj.operate() r=-2 bl.length=0 2023-05-31T19:49:19.201 INFO:teuthology.orchestra.run.smithi112.stderr:2023-05-31T19:49:19.194+0000 7f6690662780 20 rados->read ofs=0 len=0 2023-05-31T19:49:19.201 INFO:teuthology.orchestra.run.smithi112.stderr:2023-05-31T19:49:19.195+0000 7f6690662780 20 rados_obj.operate() r=-2 bl.length=0 2023-05-31T19:49:19.201 INFO:teuthology.orchestra.run.smithi112.stderr:2023-05-31T19:49:19.195+0000 7f6690662780 20 started sync module instance, tier type = 2023-05-31T19:49:19.201 INFO:teuthology.orchestra.run.smithi112.stderr:2023-05-31T19:49:19.195+0000 7f6690662780 20 started zone id=45687fb7-b52f-4449-8e12-6fd6afd32a53 (name=default) with tier type = 2023-05-31T19:49:21.230 INFO:teuthology.orchestra.run.smithi112.stderr:2023-05-31T19:49:21.224+0000 7f6690662780 20 add_watcher() i=0 2023-05-31T19:49:21.230 INFO:teuthology.orchestra.run.smithi112.stderr:2023-05-31T19:49:21.224+0000 7f6690662780 20 add_watcher() i=1 2023-05-31T19:49:21.230 INFO:teuthology.orchestra.run.smithi112.stderr:2023-05-31T19:49:21.224+0000 7f6690662780 20 add_watcher() i=2 2023-05-31T19:49:21.231 INFO:teuthology.orchestra.run.smithi112.stderr:2023-05-31T19:49:21.224+0000 7f6690662780 20 add_watcher() i=3 2023-05-31T19:49:21.231 INFO:teuthology.orchestra.run.smithi112.stderr:2023-05-31T19:49:21.224+0000 7f6690662780 20 add_watcher() i=4 2023-05-31T19:49:21.231 INFO:teuthology.orchestra.run.smithi112.stderr:2023-05-31T19:49:21.224+0000 7f6690662780 20 add_watcher() i=5 2023-05-31T19:49:21.232 INFO:teuthology.orchestra.run.smithi112.stderr:2023-05-31T19:49:21.224+0000 7f6690662780 20 add_watcher() i=6 2023-05-31T19:49:21.233 INFO:teuthology.orchestra.run.smithi112.stderr:2023-05-31T19:49:21.225+0000 7f6690662780 20 add_watcher() i=7 2023-05-31T19:49:21.233 INFO:teuthology.orchestra.run.smithi112.stderr:2023-05-31T19:49:21.225+0000 7f6690662780 2 all 8 watchers are set, enabling cache 2023-05-31T19:49:21.233 INFO:teuthology.orchestra.run.smithi112.stderr:2023-05-31T19:49:21.225+0000 7f6690662780 5 tl::expected, obj_version>, boost::system::error_code> logback_generations::read(const DoutPrefixProvider*, optional_yield):410: oid=data_loggenerations_metadata not found 2023-05-31T19:49:21.233 INFO:teuthology.orchestra.run.smithi112.stderr:2023-05-31T19:49:21.225+0000 7f6690662780 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):38 probing oid=data_log.0 2023-05-31T19:49:21.234 INFO:teuthology.orchestra.run.smithi112.stderr:2023-05-31T19:49:21.225+0000 7f6690662780 20 static int rgw::cls::fifo::FIFO::open(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::string, std::unique_ptr*, optional_yield, std::optional, bool):1168 entering 2023-05-31T19:49:21.234 INFO:teuthology.orchestra.run.smithi112.stderr:2023-05-31T19:49:21.225+0000 7f6690662780 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):60: oid=data_log.0 does not exist 2023-05-31T19:49:21.234 INFO:teuthology.orchestra.run.smithi112.stderr:2023-05-31T19:49:21.225+0000 7f6690662780 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):38 probing oid=data_log.1 2023-05-31T19:49:21.234 INFO:teuthology.orchestra.run.smithi112.stderr:2023-05-31T19:49:21.225+0000 7f6690662780 20 static int rgw::cls::fifo::FIFO::open(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::string, std::unique_ptr*, optional_yield, std::optional, bool):1168 entering 2023-05-31T19:49:21.234 INFO:teuthology.orchestra.run.smithi112.stderr:2023-05-31T19:49:21.226+0000 7f6690662780 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):60: oid=data_log.1 does not exist 2023-05-31T19:49:21.234 INFO:teuthology.orchestra.run.smithi112.stderr:2023-05-31T19:49:21.226+0000 7f6690662780 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):38 probing oid=data_log.2 2023-05-31T19:49:21.234 INFO:teuthology.orchestra.run.smithi112.stderr:2023-05-31T19:49:21.226+0000 7f6690662780 20 static int rgw::cls::fifo::FIFO::open(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::string, std::unique_ptr*, optional_yield, std::optional, bool):1168 entering 2023-05-31T19:49:21.235 INFO:teuthology.orchestra.run.smithi112.stderr:2023-05-31T19:49:21.226+0000 7f6690662780 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):60: oid=data_log.2 does not exist 2023-05-31T19:49:21.235 INFO:teuthology.orchestra.run.smithi112.stderr:2023-05-31T19:49:21.226+0000 7f6690662780 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):38 probing oid=data_log.3 2023-05-31T19:49:21.235 INFO:teuthology.orchestra.run.smithi112.stderr:2023-05-31T19:49:21.226+0000 7f6690662780 20 static int rgw::cls::fifo::FIFO::open(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::string, std::unique_ptr*, optional_yield, std::optional, bool):1168 entering 2023-05-31T19:49:21.235 INFO:teuthology.orchestra.run.smithi112.stderr:2023-05-31T19:49:21.226+0000 7f6690662780 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):60: oid=data_log.3 does not exist 2023-05-31T19:49:21.235 INFO:teuthology.orchestra.run.smithi112.stderr:2023-05-31T19:49:21.226+0000 7f6690662780 20 static int rgw::cls::fifo::FIFO::create(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::string, std::unique_ptr*, optional_yield, std::optional, std::optional >, bool, uint64_t, uint64_t):1210 entering 2023-05-31T19:49:21.235 INFO:teuthology.orchestra.run.smithi112.stderr:2023-05-31T19:49:21.227+0000 7f6690662780 20 static int rgw::cls::fifo::FIFO::open(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::string, std::unique_ptr*, optional_yield, std::optional, bool):1168 entering 2023-05-31T19:49:21.240 INFO:teuthology.orchestra.run.smithi112.stderr:2023-05-31T19:49:21.234+0000 7f66755de700 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2023-05-31T19:49:21.240 INFO:teuthology.orchestra.run.smithi112.stderr:2023-05-31T19:49:21.234+0000 7f6690662780 20 check_secure_mon_conn(): auth registy supported: methods=[2] modes=[2,1] 2023-05-31T19:49:21.240 INFO:teuthology.orchestra.run.smithi112.stderr:2023-05-31T19:49:21.234+0000 7f6690662780 20 check_secure_mon_conn(): mode 1 is insecure 2023-05-31T19:49:24.224 INFO:teuthology.orchestra.run.smithi112.stderr:2023-05-31T19:49:24.218+0000 7f6690662780 10 rgw_init_ioctx warning: failed to set recovery_priority on default.rgw.meta 2023-05-31T19:49:24.224 INFO:teuthology.orchestra.run.smithi112.stderr:2023-05-31T19:49:24.218+0000 7f6690662780 5 note: GC not initialized 2023-05-31T19:49:24.225 INFO:teuthology.orchestra.run.smithi112.stderr:2023-05-31T19:49:24.219+0000 7f6670dd5700 20 reqs_thread_entry: start 2023-05-31T19:49:24.226 INFO:teuthology.orchestra.run.smithi112.stderr:2023-05-31T19:49:24.219+0000 7f6690662780 20 init_complete bucket index max shards: 11 2023-05-31T19:49:24.226 INFO:teuthology.orchestra.run.smithi112.stderr:2023-05-31T19:49:24.220+0000 7f6690662780 10 rgw notify: Started notification manager with: 1 workers 2023-05-31T19:49:24.226 INFO:teuthology.orchestra.run.smithi112.stderr:2023-05-31T19:49:24.220+0000 7f666edd1700 20 rgw notify: INFO: next queues processing will happen at: Wed May 31 19:49:54 2023 2023-05-31T19:49:24.226 INFO:teuthology.orchestra.run.smithi112.stderr: 2023-05-31T19:49:24.228 INFO:teuthology.orchestra.run.smithi112.stderr:2023-05-31T19:49:24.220+0000 7f666e5af700 20 reqs_thread_entry: start 2023-05-31T19:49:24.247 INFO:teuthology.orchestra.run.smithi112.stderr:2023-05-31T19:49:24.241+0000 7f66755de700 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2023-05-31T19:49:24.251 INFO:teuthology.orchestra.run.smithi112.stderr:2023-05-31T19:49:24.245+0000 7f6690662780 20 remove_watcher() i=0 2023-05-31T19:49:24.252 INFO:teuthology.orchestra.run.smithi112.stderr:2023-05-31T19:49:24.245+0000 7f6690662780 2 removed watcher, disabling cache 2023-05-31T19:49:24.253 INFO:teuthology.orchestra.run.smithi112.stderr:2023-05-31T19:49:24.246+0000 7f6690662780 20 remove_watcher() i=1 2023-05-31T19:49:24.255 INFO:teuthology.orchestra.run.smithi112.stderr:2023-05-31T19:49:24.249+0000 7f6690662780 20 remove_watcher() i=2 2023-05-31T19:49:24.256 INFO:teuthology.orchestra.run.smithi112.stderr:2023-05-31T19:49:24.250+0000 7f6690662780 20 remove_watcher() i=3 2023-05-31T19:49:24.258 INFO:teuthology.orchestra.run.smithi112.stderr:2023-05-31T19:49:24.252+0000 7f6690662780 20 remove_watcher() i=4 2023-05-31T19:49:24.259 INFO:teuthology.orchestra.run.smithi112.stderr:2023-05-31T19:49:24.253+0000 7f6690662780 20 remove_watcher() i=5 2023-05-31T19:49:24.262 INFO:teuthology.orchestra.run.smithi112.stderr:2023-05-31T19:49:24.256+0000 7f6690662780 20 remove_watcher() i=6 2023-05-31T19:49:24.263 INFO:teuthology.orchestra.run.smithi112.stderr:2023-05-31T19:49:24.257+0000 7f6690662780 20 remove_watcher() i=7 2023-05-31T19:49:24.267 INFO:teuthology.orchestra.run.smithi112.stdout:[] 2023-05-31T19:49:24.268 DEBUG:tasks.util.rgw: json result: [] 2023-05-31T19:49:24.268 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['zone', 'placement', 'modify', '--rgw-zone', 'default', '--placement-id', 'default-placement', '--compression', 'random'] 2023-05-31T19:49:24.268 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'zone', 'placement', 'modify', '--rgw-zone', 'default', '--placement-id', 'default-placement', '--compression', 'random'] 2023-05-31T19:49:24.269 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 zone placement modify --rgw-zone default --placement-id default-placement --compression random 2023-05-31T19:49:24.322 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-05-31T19:49:24.322 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-05-31T19:49:24.331 INFO:teuthology.orchestra.run.smithi112.stderr:2023-05-31T19:49:24.325+0000 7f7b77109780 20 check_secure_mon_conn(): auth registy supported: methods=[2] modes=[2,1] 2023-05-31T19:49:24.331 INFO:teuthology.orchestra.run.smithi112.stderr:2023-05-31T19:49:24.325+0000 7f7b77109780 20 check_secure_mon_conn(): mode 1 is insecure 2023-05-31T19:49:24.334 INFO:teuthology.orchestra.run.smithi112.stderr:2023-05-31T19:49:24.329+0000 7f7b5887e700 20 reqs_thread_entry: start 2023-05-31T19:49:24.340 INFO:teuthology.orchestra.run.smithi112.stdout:{ 2023-05-31T19:49:24.340 INFO:teuthology.orchestra.run.smithi112.stdout: "id": "45687fb7-b52f-4449-8e12-6fd6afd32a53", 2023-05-31T19:49:24.341 INFO:teuthology.orchestra.run.smithi112.stdout: "name": "default", 2023-05-31T19:49:24.341 INFO:teuthology.orchestra.run.smithi112.stdout: "domain_root": "default.rgw.meta:root", 2023-05-31T19:49:24.342 INFO:teuthology.orchestra.run.smithi112.stdout: "control_pool": "default.rgw.control", 2023-05-31T19:49:24.342 INFO:teuthology.orchestra.run.smithi112.stdout: "gc_pool": "default.rgw.log:gc", 2023-05-31T19:49:24.342 INFO:teuthology.orchestra.run.smithi112.stdout: "lc_pool": "default.rgw.log:lc", 2023-05-31T19:49:24.343 INFO:teuthology.orchestra.run.smithi112.stdout: "log_pool": "default.rgw.log", 2023-05-31T19:49:24.343 INFO:teuthology.orchestra.run.smithi112.stdout: "intent_log_pool": "default.rgw.log:intent", 2023-05-31T19:49:24.343 INFO:teuthology.orchestra.run.smithi112.stdout: "usage_log_pool": "default.rgw.log:usage", 2023-05-31T19:49:24.343 INFO:teuthology.orchestra.run.smithi112.stdout: "roles_pool": "default.rgw.meta:roles", 2023-05-31T19:49:24.344 INFO:teuthology.orchestra.run.smithi112.stdout: "reshard_pool": "default.rgw.log:reshard", 2023-05-31T19:49:24.345 INFO:teuthology.orchestra.run.smithi112.stdout: "user_keys_pool": "default.rgw.meta:users.keys", 2023-05-31T19:49:24.345 INFO:teuthology.orchestra.run.smithi112.stdout: "user_email_pool": "default.rgw.meta:users.email", 2023-05-31T19:49:24.346 INFO:teuthology.orchestra.run.smithi112.stdout: "user_swift_pool": "default.rgw.meta:users.swift", 2023-05-31T19:49:24.346 INFO:teuthology.orchestra.run.smithi112.stdout: "user_uid_pool": "default.rgw.meta:users.uid", 2023-05-31T19:49:24.346 INFO:teuthology.orchestra.run.smithi112.stdout: "otp_pool": "default.rgw.otp", 2023-05-31T19:49:24.346 INFO:teuthology.orchestra.run.smithi112.stdout: "system_key": { 2023-05-31T19:49:24.347 INFO:teuthology.orchestra.run.smithi112.stdout: "access_key": "", 2023-05-31T19:49:24.347 INFO:teuthology.orchestra.run.smithi112.stdout: "secret_key": "" 2023-05-31T19:49:24.347 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-05-31T19:49:24.347 INFO:teuthology.orchestra.run.smithi112.stdout: "placement_pools": [ 2023-05-31T19:49:24.347 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-05-31T19:49:24.348 INFO:teuthology.orchestra.run.smithi112.stdout: "key": "default-placement", 2023-05-31T19:49:24.348 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-05-31T19:49:24.348 INFO:teuthology.orchestra.run.smithi112.stdout: "index_pool": "default.rgw.buckets.index", 2023-05-31T19:49:24.349 INFO:teuthology.orchestra.run.smithi112.stdout: "storage_classes": { 2023-05-31T19:49:24.350 INFO:teuthology.orchestra.run.smithi112.stdout: "STANDARD": { 2023-05-31T19:49:24.352 INFO:teuthology.orchestra.run.smithi112.stdout: "data_pool": "default.rgw.buckets.data", 2023-05-31T19:49:24.352 INFO:teuthology.orchestra.run.smithi112.stdout: "compression_type": "random" 2023-05-31T19:49:24.352 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-05-31T19:49:24.352 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-05-31T19:49:24.352 INFO:teuthology.orchestra.run.smithi112.stdout: "data_extra_pool": "default.rgw.buckets.non-ec", 2023-05-31T19:49:24.353 INFO:teuthology.orchestra.run.smithi112.stdout: "index_type": 0, 2023-05-31T19:49:24.353 INFO:teuthology.orchestra.run.smithi112.stdout: "inline_data": true 2023-05-31T19:49:24.353 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-05-31T19:49:24.353 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-05-31T19:49:24.353 INFO:teuthology.orchestra.run.smithi112.stdout: ], 2023-05-31T19:49:24.353 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_id": "", 2023-05-31T19:49:24.354 INFO:teuthology.orchestra.run.smithi112.stdout: "notif_pool": "default.rgw.log:notif" 2023-05-31T19:49:24.354 INFO:teuthology.orchestra.run.smithi112.stdout:} 2023-05-31T19:49:24.355 DEBUG:tasks.util.rgw: json result: {'id': '45687fb7-b52f-4449-8e12-6fd6afd32a53', 'name': 'default', 'domain_root': 'default.rgw.meta:root', 'control_pool': 'default.rgw.control', 'gc_pool': 'default.rgw.log:gc', 'lc_pool': 'default.rgw.log:lc', 'log_pool': 'default.rgw.log', 'intent_log_pool': 'default.rgw.log:intent', 'usage_log_pool': 'default.rgw.log:usage', 'roles_pool': 'default.rgw.meta:roles', 'reshard_pool': 'default.rgw.log:reshard', 'user_keys_pool': 'default.rgw.meta:users.keys', 'user_email_pool': 'default.rgw.meta:users.email', 'user_swift_pool': 'default.rgw.meta:users.swift', 'user_uid_pool': 'default.rgw.meta:users.uid', 'otp_pool': 'default.rgw.otp', 'system_key': {'access_key': '', 'secret_key': ''}, 'placement_pools': [{'key': 'default-placement', 'val': {'index_pool': 'default.rgw.buckets.index', 'storage_classes': {'STANDARD': {'data_pool': 'default.rgw.buckets.data', 'compression_type': 'random'}}, 'data_extra_pool': 'default.rgw.buckets.non-ec', 'index_type': 0, 'inline_data': True}}], 'realm_id': '', 'notif_pool': 'default.rgw.log:notif'} 2023-05-31T19:49:24.356 INFO:tasks.util.rgw:rgwadmin: c2.client.1 : ['user', 'list'] 2023-05-31T19:49:24.356 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.1', '--cluster', 'c2', 'user', 'list'] 2023-05-31T19:49:24.356 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.1 --cluster c2 user list 2023-05-31T19:49:24.401 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-05-31T19:49:24.402 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-05-31T19:49:24.409 INFO:teuthology.orchestra.run.smithi112.stderr:2023-05-31T19:49:24.403+0000 7fa2aec48780 20 rados->read ofs=0 len=0 2023-05-31T19:49:24.411 INFO:teuthology.orchestra.run.smithi112.stderr:2023-05-31T19:49:24.404+0000 7fa2aec48780 20 rados_obj.operate() r=-2 bl.length=0 2023-05-31T19:49:24.412 INFO:teuthology.orchestra.run.smithi112.stderr:2023-05-31T19:49:24.404+0000 7fa2aec48780 20 realm 2023-05-31T19:49:24.412 INFO:teuthology.orchestra.run.smithi112.stderr:2023-05-31T19:49:24.404+0000 7fa2aec48780 20 rados->read ofs=0 len=0 2023-05-31T19:49:24.412 INFO:teuthology.orchestra.run.smithi112.stderr:2023-05-31T19:49:24.404+0000 7fa2aec48780 20 rados_obj.operate() r=-2 bl.length=0 2023-05-31T19:49:24.412 INFO:teuthology.orchestra.run.smithi112.stderr:2023-05-31T19:49:24.404+0000 7fa2aec48780 4 RGWPeriod::init failed to init realm id : (2) No such file or directory 2023-05-31T19:49:24.412 INFO:teuthology.orchestra.run.smithi112.stderr:2023-05-31T19:49:24.404+0000 7fa2aec48780 20 rados->read ofs=0 len=0 2023-05-31T19:49:24.413 INFO:teuthology.orchestra.run.smithi112.stderr:2023-05-31T19:49:24.404+0000 7fa2aec48780 20 rados_obj.operate() r=-2 bl.length=0 2023-05-31T19:49:24.413 INFO:teuthology.orchestra.run.smithi112.stderr:2023-05-31T19:49:24.404+0000 7fa2aec48780 20 rados->read ofs=0 len=0 2023-05-31T19:49:24.414 INFO:teuthology.orchestra.run.smithi112.stderr:2023-05-31T19:49:24.404+0000 7fa2aec48780 20 rados_obj.operate() r=0 bl.length=46 2023-05-31T19:49:24.414 INFO:teuthology.orchestra.run.smithi112.stderr:2023-05-31T19:49:24.404+0000 7fa2aec48780 20 rados->read ofs=0 len=0 2023-05-31T19:49:24.414 INFO:teuthology.orchestra.run.smithi112.stderr:2023-05-31T19:49:24.405+0000 7fa2aec48780 20 rados_obj.operate() r=0 bl.length=890 2023-05-31T19:49:24.414 INFO:teuthology.orchestra.run.smithi112.stderr:2023-05-31T19:49:24.405+0000 7fa2aec48780 20 searching for the correct realm 2023-05-31T19:49:24.421 INFO:teuthology.orchestra.run.smithi112.stderr:2023-05-31T19:49:24.412+0000 7fa2aec48780 20 RGWRados::pool_iterate: got zone_info.45687fb7-b52f-4449-8e12-6fd6afd32a53 2023-05-31T19:49:24.421 INFO:teuthology.orchestra.run.smithi112.stderr:2023-05-31T19:49:24.412+0000 7fa2aec48780 20 RGWRados::pool_iterate: got zonegroup_info.39908ee0-a1c5-4a7d-8b2d-28f0fca3c8cb 2023-05-31T19:49:24.421 INFO:teuthology.orchestra.run.smithi112.stderr:2023-05-31T19:49:24.412+0000 7fa2aec48780 20 RGWRados::pool_iterate: got zone_names.default 2023-05-31T19:49:24.421 INFO:teuthology.orchestra.run.smithi112.stderr:2023-05-31T19:49:24.412+0000 7fa2aec48780 20 RGWRados::pool_iterate: got zonegroups_names.default 2023-05-31T19:49:24.421 INFO:teuthology.orchestra.run.smithi112.stderr:2023-05-31T19:49:24.412+0000 7fa2aec48780 20 rados->read ofs=0 len=0 2023-05-31T19:49:24.421 INFO:teuthology.orchestra.run.smithi112.stderr:2023-05-31T19:49:24.412+0000 7fa2aec48780 20 rados_obj.operate() r=-2 bl.length=0 2023-05-31T19:49:24.422 INFO:teuthology.orchestra.run.smithi112.stderr:2023-05-31T19:49:24.412+0000 7fa2aec48780 20 rados->read ofs=0 len=0 2023-05-31T19:49:24.422 INFO:teuthology.orchestra.run.smithi112.stderr:2023-05-31T19:49:24.412+0000 7fa2aec48780 20 rados_obj.operate() r=0 bl.length=46 2023-05-31T19:49:24.422 INFO:teuthology.orchestra.run.smithi112.stderr:2023-05-31T19:49:24.412+0000 7fa2aec48780 20 rados->read ofs=0 len=0 2023-05-31T19:49:24.422 INFO:teuthology.orchestra.run.smithi112.stderr:2023-05-31T19:49:24.412+0000 7fa2aec48780 20 rados_obj.operate() r=0 bl.length=398 2023-05-31T19:49:24.422 INFO:teuthology.orchestra.run.smithi112.stderr:2023-05-31T19:49:24.412+0000 7fa2aec48780 20 zone default found 2023-05-31T19:49:24.422 INFO:teuthology.orchestra.run.smithi112.stderr:2023-05-31T19:49:24.412+0000 7fa2aec48780 4 Realm: () 2023-05-31T19:49:24.423 INFO:teuthology.orchestra.run.smithi112.stderr:2023-05-31T19:49:24.412+0000 7fa2aec48780 4 ZoneGroup: default (39908ee0-a1c5-4a7d-8b2d-28f0fca3c8cb) 2023-05-31T19:49:24.423 INFO:teuthology.orchestra.run.smithi112.stderr:2023-05-31T19:49:24.412+0000 7fa2aec48780 4 Zone: default (45687fb7-b52f-4449-8e12-6fd6afd32a53) 2023-05-31T19:49:24.423 INFO:teuthology.orchestra.run.smithi112.stderr:2023-05-31T19:49:24.412+0000 7fa2aec48780 10 cannot find current period zonegroup using local zonegroup configuration 2023-05-31T19:49:24.424 INFO:teuthology.orchestra.run.smithi112.stderr:2023-05-31T19:49:24.412+0000 7fa2aec48780 20 zonegroup default 2023-05-31T19:49:24.424 INFO:teuthology.orchestra.run.smithi112.stderr:2023-05-31T19:49:24.412+0000 7fa2aec48780 20 rados->read ofs=0 len=0 2023-05-31T19:49:24.424 INFO:teuthology.orchestra.run.smithi112.stderr:2023-05-31T19:49:24.412+0000 7fa2aec48780 20 rados_obj.operate() r=-2 bl.length=0 2023-05-31T19:49:24.424 INFO:teuthology.orchestra.run.smithi112.stderr:2023-05-31T19:49:24.412+0000 7fa2aec48780 20 rados->read ofs=0 len=0 2023-05-31T19:49:24.425 INFO:teuthology.orchestra.run.smithi112.stderr:2023-05-31T19:49:24.413+0000 7fa2aec48780 20 rados_obj.operate() r=-2 bl.length=0 2023-05-31T19:49:24.425 INFO:teuthology.orchestra.run.smithi112.stderr:2023-05-31T19:49:24.413+0000 7fa2aec48780 20 rados->read ofs=0 len=0 2023-05-31T19:49:24.425 INFO:teuthology.orchestra.run.smithi112.stderr:2023-05-31T19:49:24.413+0000 7fa2aec48780 20 rados_obj.operate() r=-2 bl.length=0 2023-05-31T19:49:24.425 INFO:teuthology.orchestra.run.smithi112.stderr:2023-05-31T19:49:24.413+0000 7fa2aec48780 20 started sync module instance, tier type = 2023-05-31T19:49:24.426 INFO:teuthology.orchestra.run.smithi112.stderr:2023-05-31T19:49:24.413+0000 7fa2aec48780 20 started zone id=45687fb7-b52f-4449-8e12-6fd6afd32a53 (name=default) with tier type = 2023-05-31T19:49:24.444 INFO:teuthology.orchestra.run.smithi112.stderr:2023-05-31T19:49:24.436+0000 7fa2aec48780 20 add_watcher() i=0 2023-05-31T19:49:24.444 INFO:teuthology.orchestra.run.smithi112.stderr:2023-05-31T19:49:24.436+0000 7fa2aec48780 20 add_watcher() i=1 2023-05-31T19:49:24.445 INFO:teuthology.orchestra.run.smithi112.stderr:2023-05-31T19:49:24.436+0000 7fa2aec48780 20 add_watcher() i=2 2023-05-31T19:49:24.445 INFO:teuthology.orchestra.run.smithi112.stderr:2023-05-31T19:49:24.436+0000 7fa2aec48780 20 add_watcher() i=3 2023-05-31T19:49:24.445 INFO:teuthology.orchestra.run.smithi112.stderr:2023-05-31T19:49:24.436+0000 7fa2aec48780 20 add_watcher() i=4 2023-05-31T19:49:24.445 INFO:teuthology.orchestra.run.smithi112.stderr:2023-05-31T19:49:24.436+0000 7fa2aec48780 20 add_watcher() i=5 2023-05-31T19:49:24.445 INFO:teuthology.orchestra.run.smithi112.stderr:2023-05-31T19:49:24.436+0000 7fa2aec48780 20 add_watcher() i=6 2023-05-31T19:49:24.445 INFO:teuthology.orchestra.run.smithi112.stderr:2023-05-31T19:49:24.437+0000 7fa2aec48780 20 add_watcher() i=7 2023-05-31T19:49:24.446 INFO:teuthology.orchestra.run.smithi112.stderr:2023-05-31T19:49:24.437+0000 7fa2aec48780 2 all 8 watchers are set, enabling cache 2023-05-31T19:49:24.446 INFO:teuthology.orchestra.run.smithi112.stderr:2023-05-31T19:49:24.439+0000 7fa293bc4700 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2023-05-31T19:49:24.446 INFO:teuthology.orchestra.run.smithi112.stderr:2023-05-31T19:49:24.439+0000 7fa2aec48780 20 check_secure_mon_conn(): auth registy supported: methods=[2] modes=[2,1] 2023-05-31T19:49:24.448 INFO:teuthology.orchestra.run.smithi112.stderr:2023-05-31T19:49:24.439+0000 7fa2aec48780 20 check_secure_mon_conn(): mode 1 is insecure 2023-05-31T19:49:24.449 INFO:teuthology.orchestra.run.smithi112.stderr:2023-05-31T19:49:24.443+0000 7fa2aec48780 5 note: GC not initialized 2023-05-31T19:49:24.450 INFO:teuthology.orchestra.run.smithi112.stderr:2023-05-31T19:49:24.444+0000 7fa28f3bb700 20 reqs_thread_entry: start 2023-05-31T19:49:24.450 INFO:teuthology.orchestra.run.smithi112.stderr:2023-05-31T19:49:24.444+0000 7fa2aec48780 20 init_complete bucket index max shards: 11 2023-05-31T19:49:24.451 INFO:teuthology.orchestra.run.smithi112.stderr:2023-05-31T19:49:24.444+0000 7fa2aec48780 10 rgw notify: Started notification manager with: 1 workers 2023-05-31T19:49:24.451 INFO:teuthology.orchestra.run.smithi112.stderr:2023-05-31T19:49:24.444+0000 7fa28d3b7700 20 rgw notify: INFO: next queues processing will happen at: Wed May 31 19:49:54 2023 2023-05-31T19:49:24.451 INFO:teuthology.orchestra.run.smithi112.stderr: 2023-05-31T19:49:24.451 INFO:teuthology.orchestra.run.smithi112.stderr:2023-05-31T19:49:24.444+0000 7fa28cb95700 20 reqs_thread_entry: start 2023-05-31T19:49:24.457 INFO:teuthology.orchestra.run.smithi112.stderr:2023-05-31T19:49:24.451+0000 7fa293bc4700 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2023-05-31T19:49:24.459 INFO:teuthology.orchestra.run.smithi112.stderr:2023-05-31T19:49:24.453+0000 7fa2aec48780 20 remove_watcher() i=0 2023-05-31T19:49:24.459 INFO:teuthology.orchestra.run.smithi112.stderr:2023-05-31T19:49:24.453+0000 7fa2aec48780 2 removed watcher, disabling cache 2023-05-31T19:49:24.460 INFO:teuthology.orchestra.run.smithi112.stderr:2023-05-31T19:49:24.454+0000 7fa2aec48780 20 remove_watcher() i=1 2023-05-31T19:49:24.461 INFO:teuthology.orchestra.run.smithi112.stderr:2023-05-31T19:49:24.454+0000 7fa2aec48780 20 remove_watcher() i=2 2023-05-31T19:49:24.461 INFO:teuthology.orchestra.run.smithi112.stderr:2023-05-31T19:49:24.455+0000 7fa2aec48780 20 remove_watcher() i=3 2023-05-31T19:49:24.462 INFO:teuthology.orchestra.run.smithi112.stderr:2023-05-31T19:49:24.456+0000 7fa2aec48780 20 remove_watcher() i=4 2023-05-31T19:49:24.463 INFO:teuthology.orchestra.run.smithi112.stderr:2023-05-31T19:49:24.457+0000 7fa2aec48780 20 remove_watcher() i=5 2023-05-31T19:49:24.463 INFO:teuthology.orchestra.run.smithi112.stderr:2023-05-31T19:49:24.458+0000 7fa2aec48780 20 remove_watcher() i=6 2023-05-31T19:49:24.464 INFO:teuthology.orchestra.run.smithi112.stderr:2023-05-31T19:49:24.459+0000 7fa2aec48780 20 remove_watcher() i=7 2023-05-31T19:49:24.466 INFO:teuthology.orchestra.run.smithi112.stdout:[] 2023-05-31T19:49:24.469 DEBUG:tasks.util.rgw: json result: [] 2023-05-31T19:49:24.469 INFO:tasks.util.rgw:rgwadmin: c2.client.1 : ['zone', 'placement', 'modify', '--rgw-zone', 'default', '--placement-id', 'default-placement', '--compression', 'random'] 2023-05-31T19:49:24.469 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.1', '--cluster', 'c2', 'zone', 'placement', 'modify', '--rgw-zone', 'default', '--placement-id', 'default-placement', '--compression', 'random'] 2023-05-31T19:49:24.469 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.1 --cluster c2 zone placement modify --rgw-zone default --placement-id default-placement --compression random 2023-05-31T19:49:24.519 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-05-31T19:49:24.519 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-05-31T19:49:24.527 INFO:teuthology.orchestra.run.smithi112.stderr:2023-05-31T19:49:24.521+0000 7f478826e780 20 check_secure_mon_conn(): auth registy supported: methods=[2] modes=[2,1] 2023-05-31T19:49:24.527 INFO:teuthology.orchestra.run.smithi112.stderr:2023-05-31T19:49:24.521+0000 7f478826e780 20 check_secure_mon_conn(): mode 1 is insecure 2023-05-31T19:49:24.530 INFO:teuthology.orchestra.run.smithi112.stderr:2023-05-31T19:49:24.524+0000 7f47699e3700 20 reqs_thread_entry: start 2023-05-31T19:49:24.536 INFO:teuthology.orchestra.run.smithi112.stdout:{ 2023-05-31T19:49:24.536 INFO:teuthology.orchestra.run.smithi112.stdout: "id": "45687fb7-b52f-4449-8e12-6fd6afd32a53", 2023-05-31T19:49:24.536 INFO:teuthology.orchestra.run.smithi112.stdout: "name": "default", 2023-05-31T19:49:24.536 INFO:teuthology.orchestra.run.smithi112.stdout: "domain_root": "default.rgw.meta:root", 2023-05-31T19:49:24.536 INFO:teuthology.orchestra.run.smithi112.stdout: "control_pool": "default.rgw.control", 2023-05-31T19:49:24.537 INFO:teuthology.orchestra.run.smithi112.stdout: "gc_pool": "default.rgw.log:gc", 2023-05-31T19:49:24.537 INFO:teuthology.orchestra.run.smithi112.stdout: "lc_pool": "default.rgw.log:lc", 2023-05-31T19:49:24.537 INFO:teuthology.orchestra.run.smithi112.stdout: "log_pool": "default.rgw.log", 2023-05-31T19:49:24.537 INFO:teuthology.orchestra.run.smithi112.stdout: "intent_log_pool": "default.rgw.log:intent", 2023-05-31T19:49:24.538 INFO:teuthology.orchestra.run.smithi112.stdout: "usage_log_pool": "default.rgw.log:usage", 2023-05-31T19:49:24.538 INFO:teuthology.orchestra.run.smithi112.stdout: "roles_pool": "default.rgw.meta:roles", 2023-05-31T19:49:24.538 INFO:teuthology.orchestra.run.smithi112.stdout: "reshard_pool": "default.rgw.log:reshard", 2023-05-31T19:49:24.538 INFO:teuthology.orchestra.run.smithi112.stdout: "user_keys_pool": "default.rgw.meta:users.keys", 2023-05-31T19:49:24.538 INFO:teuthology.orchestra.run.smithi112.stdout: "user_email_pool": "default.rgw.meta:users.email", 2023-05-31T19:49:24.538 INFO:teuthology.orchestra.run.smithi112.stdout: "user_swift_pool": "default.rgw.meta:users.swift", 2023-05-31T19:49:24.538 INFO:teuthology.orchestra.run.smithi112.stdout: "user_uid_pool": "default.rgw.meta:users.uid", 2023-05-31T19:49:24.539 INFO:teuthology.orchestra.run.smithi112.stdout: "otp_pool": "default.rgw.otp", 2023-05-31T19:49:24.539 INFO:teuthology.orchestra.run.smithi112.stdout: "system_key": { 2023-05-31T19:49:24.539 INFO:teuthology.orchestra.run.smithi112.stdout: "access_key": "", 2023-05-31T19:49:24.539 INFO:teuthology.orchestra.run.smithi112.stdout: "secret_key": "" 2023-05-31T19:49:24.539 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-05-31T19:49:24.539 INFO:teuthology.orchestra.run.smithi112.stdout: "placement_pools": [ 2023-05-31T19:49:24.539 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-05-31T19:49:24.539 INFO:teuthology.orchestra.run.smithi112.stdout: "key": "default-placement", 2023-05-31T19:49:24.540 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-05-31T19:49:24.540 INFO:teuthology.orchestra.run.smithi112.stdout: "index_pool": "default.rgw.buckets.index", 2023-05-31T19:49:24.540 INFO:teuthology.orchestra.run.smithi112.stdout: "storage_classes": { 2023-05-31T19:49:24.540 INFO:teuthology.orchestra.run.smithi112.stdout: "STANDARD": { 2023-05-31T19:49:24.540 INFO:teuthology.orchestra.run.smithi112.stdout: "data_pool": "default.rgw.buckets.data", 2023-05-31T19:49:24.540 INFO:teuthology.orchestra.run.smithi112.stdout: "compression_type": "random" 2023-05-31T19:49:24.540 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-05-31T19:49:24.540 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-05-31T19:49:24.541 INFO:teuthology.orchestra.run.smithi112.stdout: "data_extra_pool": "default.rgw.buckets.non-ec", 2023-05-31T19:49:24.541 INFO:teuthology.orchestra.run.smithi112.stdout: "index_type": 0, 2023-05-31T19:49:24.541 INFO:teuthology.orchestra.run.smithi112.stdout: "inline_data": true 2023-05-31T19:49:24.541 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-05-31T19:49:24.541 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-05-31T19:49:24.541 INFO:teuthology.orchestra.run.smithi112.stdout: ], 2023-05-31T19:49:24.541 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_id": "", 2023-05-31T19:49:24.542 INFO:teuthology.orchestra.run.smithi112.stdout: "notif_pool": "default.rgw.log:notif" 2023-05-31T19:49:24.542 INFO:teuthology.orchestra.run.smithi112.stdout:} 2023-05-31T19:49:24.543 DEBUG:tasks.util.rgw: json result: {'id': '45687fb7-b52f-4449-8e12-6fd6afd32a53', 'name': 'default', 'domain_root': 'default.rgw.meta:root', 'control_pool': 'default.rgw.control', 'gc_pool': 'default.rgw.log:gc', 'lc_pool': 'default.rgw.log:lc', 'log_pool': 'default.rgw.log', 'intent_log_pool': 'default.rgw.log:intent', 'usage_log_pool': 'default.rgw.log:usage', 'roles_pool': 'default.rgw.meta:roles', 'reshard_pool': 'default.rgw.log:reshard', 'user_keys_pool': 'default.rgw.meta:users.keys', 'user_email_pool': 'default.rgw.meta:users.email', 'user_swift_pool': 'default.rgw.meta:users.swift', 'user_uid_pool': 'default.rgw.meta:users.uid', 'otp_pool': 'default.rgw.otp', 'system_key': {'access_key': '', 'secret_key': ''}, 'placement_pools': [{'key': 'default-placement', 'val': {'index_pool': 'default.rgw.buckets.index', 'storage_classes': {'STANDARD': {'data_pool': 'default.rgw.buckets.data', 'compression_type': 'random'}}, 'data_extra_pool': 'default.rgw.buckets.non-ec', 'index_type': 0, 'inline_data': True}}], 'realm_id': '', 'notif_pool': 'default.rgw.log:notif'} 2023-05-31T19:49:24.543 INFO:tasks.rgw:Starting rgw... 2023-05-31T19:49:24.543 INFO:tasks.rgw:rgw c1.client.0 config is {'port': 8000} 2023-05-31T19:49:24.543 INFO:tasks.rgw:Using beast as radosgw frontend 2023-05-31T19:49:24.544 INFO:tasks.rgw.c1.client.0:Restarting daemon 2023-05-31T19:49:24.544 DEBUG:teuthology.orchestra.run.smithi099:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term radosgw --rgw-frontends 'beast port=8000' -n client.0 --cluster c1 -k /etc/ceph/c1.client.0.keyring --log-file /var/log/ceph/rgw.c1.client.0.log --rgw_ops_log_socket_path /home/ubuntu/cephtest/rgw.opslog.c1.client.0.sock --foreground | sudo tee /var/log/ceph/rgw.c1.client.0.stdout 2>&1 2023-05-31T19:49:24.547 INFO:tasks.rgw.c1.client.0:Started 2023-05-31T19:49:24.547 INFO:tasks.rgw:rgw c1.client.1 config is {'port': 8001} 2023-05-31T19:49:24.547 INFO:tasks.rgw:Using beast as radosgw frontend 2023-05-31T19:49:24.548 INFO:tasks.rgw.c1.client.1:Restarting daemon 2023-05-31T19:49:24.548 DEBUG:teuthology.orchestra.run.smithi099:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term radosgw --rgw-frontends 'beast port=8001' -n client.1 --cluster c1 -k /etc/ceph/c1.client.1.keyring --log-file /var/log/ceph/rgw.c1.client.1.log --rgw_ops_log_socket_path /home/ubuntu/cephtest/rgw.opslog.c1.client.1.sock --foreground | sudo tee /var/log/ceph/rgw.c1.client.1.stdout 2>&1 2023-05-31T19:49:24.550 INFO:tasks.rgw.c1.client.1:Started 2023-05-31T19:49:24.551 INFO:tasks.rgw:rgw c2.client.0 config is {'port': 8000} 2023-05-31T19:49:24.551 INFO:tasks.rgw:Using beast as radosgw frontend 2023-05-31T19:49:24.551 INFO:tasks.rgw.c2.client.0:Restarting daemon 2023-05-31T19:49:24.551 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term radosgw --rgw-frontends 'beast port=8000' -n client.0 --cluster c2 -k /etc/ceph/c2.client.0.keyring --log-file /var/log/ceph/rgw.c2.client.0.log --rgw_ops_log_socket_path /home/ubuntu/cephtest/rgw.opslog.c2.client.0.sock --foreground | sudo tee /var/log/ceph/rgw.c2.client.0.stdout 2>&1 2023-05-31T19:49:24.554 INFO:tasks.rgw.c2.client.0:Started 2023-05-31T19:49:24.554 INFO:tasks.rgw:rgw c2.client.1 config is {'port': 8001} 2023-05-31T19:49:24.555 INFO:tasks.rgw:Using beast as radosgw frontend 2023-05-31T19:49:24.555 INFO:tasks.rgw.c2.client.1:Restarting daemon 2023-05-31T19:49:24.555 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term radosgw --rgw-frontends 'beast port=8001' -n client.1 --cluster c2 -k /etc/ceph/c2.client.1.keyring --log-file /var/log/ceph/rgw.c2.client.1.log --rgw_ops_log_socket_path /home/ubuntu/cephtest/rgw.opslog.c2.client.1.sock --foreground | sudo tee /var/log/ceph/rgw.c2.client.1.stdout 2>&1 2023-05-31T19:49:24.557 INFO:tasks.rgw.c2.client.1:Started 2023-05-31T19:49:24.557 INFO:tasks.rgw:Polling c1.client.0 until it starts accepting connections on http://smithi099.front.sepia.ceph.com:8000/ 2023-05-31T19:49:24.557 DEBUG:teuthology.orchestra.run.smithi099:> curl http://smithi099.front.sepia.ceph.com:8000/ 2023-05-31T19:49:24.592 INFO:teuthology.orchestra.run.smithi099.stderr: % Total % Received % Xferd Average Speed Time Time Time Current 2023-05-31T19:49:24.592 INFO:teuthology.orchestra.run.smithi099.stderr: Dload Upload Total Spent Left Speed 2023-05-31T19:49:24.593 DEBUG:teuthology.orchestra.run:got remote process result: 7 2023-05-31T19:49:24.593 INFO:teuthology.orchestra.run.smithi099.stderr: 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0curl: (7) Failed to connect to smithi099.front.sepia.ceph.com port 8000: Connection refused 2023-05-31T19:49:25.595 DEBUG:teuthology.orchestra.run.smithi099:> curl http://smithi099.front.sepia.ceph.com:8000/ 2023-05-31T19:49:25.617 INFO:teuthology.orchestra.run.smithi099.stderr: % Total % Received % Xferd Average Speed Time Time Time Current 2023-05-31T19:49:25.618 INFO:teuthology.orchestra.run.smithi099.stderr: Dload Upload Total Spent Left Speed 2023-05-31T19:49:25.619 INFO:teuthology.orchestra.run.smithi099.stderr: 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 100 214 0 214 0 0 208k 0 --:--:-- --:--:-- --:--:-- 208k 2023-05-31T19:49:25.619 INFO:teuthology.orchestra.run.smithi099.stdout:anonymous 2023-05-31T19:49:25.619 INFO:tasks.rgw:Polling c1.client.1 until it starts accepting connections on http://smithi099.front.sepia.ceph.com:8001/ 2023-05-31T19:49:25.620 DEBUG:teuthology.orchestra.run.smithi099:> curl http://smithi099.front.sepia.ceph.com:8001/ 2023-05-31T19:49:25.681 INFO:teuthology.orchestra.run.smithi099.stderr: % Total % Received % Xferd Average Speed Time Time Time Current 2023-05-31T19:49:25.681 INFO:teuthology.orchestra.run.smithi099.stderr: Dload Upload Total Spent Left Speed 2023-05-31T19:49:25.681 INFO:teuthology.orchestra.run.smithi099.stderr: 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 100 214 0 214 0 0 208k 0 --:--:-- --:--:-- --:--:-- 208k 2023-05-31T19:49:25.682 INFO:teuthology.orchestra.run.smithi099.stdout:anonymous 2023-05-31T19:49:25.683 INFO:tasks.rgw:Polling c2.client.0 until it starts accepting connections on http://smithi112.front.sepia.ceph.com:8000/ 2023-05-31T19:49:25.683 DEBUG:teuthology.orchestra.run.smithi112:> curl http://smithi112.front.sepia.ceph.com:8000/ 2023-05-31T19:49:25.731 INFO:teuthology.orchestra.run.smithi112.stderr: % Total % Received % Xferd Average Speed Time Time Time Current 2023-05-31T19:49:25.731 INFO:teuthology.orchestra.run.smithi112.stderr: Dload Upload Total Spent Left Speed 2023-05-31T19:49:25.731 INFO:teuthology.orchestra.run.smithi112.stderr: 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 100 214 0 214 0 0 208k 0 --:--:-- --:--:-- --:--:-- 208k 2023-05-31T19:49:25.732 INFO:teuthology.orchestra.run.smithi112.stdout:anonymous 2023-05-31T19:49:25.733 INFO:tasks.rgw:Polling c2.client.1 until it starts accepting connections on http://smithi112.front.sepia.ceph.com:8001/ 2023-05-31T19:49:25.733 DEBUG:teuthology.orchestra.run.smithi112:> curl http://smithi112.front.sepia.ceph.com:8001/ 2023-05-31T19:49:25.794 INFO:teuthology.orchestra.run.smithi112.stderr: % Total % Received % Xferd Average Speed Time Time Time Current 2023-05-31T19:49:25.794 INFO:teuthology.orchestra.run.smithi112.stderr: Dload Upload Total Spent Left Speed 2023-05-31T19:49:25.795 INFO:teuthology.orchestra.run.smithi112.stderr: 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 100 214 0 214 0 0 208k 0 --:--:-- --:--:-- --:--:-- 208k 2023-05-31T19:49:25.796 INFO:teuthology.orchestra.run.smithi112.stdout:anonymous 2023-05-31T19:49:25.796 INFO:teuthology.run_tasks:Running task rgw-multisite... 2023-05-31T19:49:25.862 INFO:tasks.rgw_multisite:creating realm.. 2023-05-31T19:49:25.862 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['realm', 'create', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-05-31T19:49:25.863 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'realm', 'create', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-05-31T19:49:25.863 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 realm create --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 2023-05-31T19:49:25.909 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2023-05-31T19:49:25.909 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2023-05-31T19:49:25.939 INFO:teuthology.orchestra.run.smithi099.stdout:{ 2023-05-31T19:49:25.940 INFO:teuthology.orchestra.run.smithi099.stdout: "id": "aaa33de6-515b-4cac-8f1b-3a1a30452117", 2023-05-31T19:49:25.940 INFO:teuthology.orchestra.run.smithi099.stdout: "name": "test-realm", 2023-05-31T19:49:25.940 INFO:teuthology.orchestra.run.smithi099.stdout: "current_period": "629b9ced-80af-41ed-8a6c-f2cfc334cbb4", 2023-05-31T19:49:25.940 INFO:teuthology.orchestra.run.smithi099.stdout: "epoch": 1 2023-05-31T19:49:25.940 INFO:teuthology.orchestra.run.smithi099.stdout:} 2023-05-31T19:49:25.941 INFO:tasks.rgw_multisite:creating master zonegroup.. 2023-05-31T19:49:25.942 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['zonegroup', 'set', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--default', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-05-31T19:49:25.942 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'zonegroup', 'set', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--default', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-05-31T19:49:25.942 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 zonegroup set --rgw-zonegroup a --rgw-realm test-realm --default --cluster c1 --debug-rgw 0 --debug-ms 0 2023-05-31T19:49:25.990 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2023-05-31T19:49:25.991 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2023-05-31T19:49:26.005 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T19:49:25.998+0000 7f605a32f780 -1 old format 2023-05-31T19:49:26.013 INFO:teuthology.orchestra.run.smithi099.stdout:{ 2023-05-31T19:49:26.013 INFO:teuthology.orchestra.run.smithi099.stdout: "id": "a", 2023-05-31T19:49:26.013 INFO:teuthology.orchestra.run.smithi099.stdout: "name": "a", 2023-05-31T19:49:26.013 INFO:teuthology.orchestra.run.smithi099.stdout: "api_name": "", 2023-05-31T19:49:26.013 INFO:teuthology.orchestra.run.smithi099.stdout: "is_master": true, 2023-05-31T19:49:26.013 INFO:teuthology.orchestra.run.smithi099.stdout: "endpoints": [ 2023-05-31T19:49:26.014 INFO:teuthology.orchestra.run.smithi099.stdout: "http://smithi099.front.sepia.ceph.com:8000" 2023-05-31T19:49:26.014 INFO:teuthology.orchestra.run.smithi099.stdout: ], 2023-05-31T19:49:26.014 INFO:teuthology.orchestra.run.smithi099.stdout: "hostnames": [], 2023-05-31T19:49:26.014 INFO:teuthology.orchestra.run.smithi099.stdout: "hostnames_s3website": [], 2023-05-31T19:49:26.014 INFO:teuthology.orchestra.run.smithi099.stdout: "master_zone": "", 2023-05-31T19:49:26.014 INFO:teuthology.orchestra.run.smithi099.stdout: "zones": [], 2023-05-31T19:49:26.014 INFO:teuthology.orchestra.run.smithi099.stdout: "placement_targets": [ 2023-05-31T19:49:26.015 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T19:49:26.015 INFO:teuthology.orchestra.run.smithi099.stdout: "name": "default-placement", 2023-05-31T19:49:26.015 INFO:teuthology.orchestra.run.smithi099.stdout: "tags": [], 2023-05-31T19:49:26.015 INFO:teuthology.orchestra.run.smithi099.stdout: "storage_classes": [] 2023-05-31T19:49:26.015 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T19:49:26.015 INFO:teuthology.orchestra.run.smithi099.stdout: ], 2023-05-31T19:49:26.015 INFO:teuthology.orchestra.run.smithi099.stdout: "default_placement": "default-placement", 2023-05-31T19:49:26.015 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_id": "aaa33de6-515b-4cac-8f1b-3a1a30452117", 2023-05-31T19:49:26.016 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_policy": { 2023-05-31T19:49:26.016 INFO:teuthology.orchestra.run.smithi099.stdout: "groups": [] 2023-05-31T19:49:26.016 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T19:49:26.016 INFO:teuthology.orchestra.run.smithi099.stdout: "enabled_features": [ 2023-05-31T19:49:26.016 INFO:teuthology.orchestra.run.smithi099.stdout: "resharding" 2023-05-31T19:49:26.016 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2023-05-31T19:49:26.016 INFO:teuthology.orchestra.run.smithi099.stdout:} 2023-05-31T19:49:26.017 INFO:tasks.rgw_multisite:creating master zone.. 2023-05-31T19:49:26.018 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['zone', 'create', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--default', '--master', '--access-key', 'RVN19ED3W7FH7E08', '--secret', 'XzQjGe60gbIUStGfs96TKhuPw3IQi3hK', '--endpoints', 'http://smithi099.front.sepia.ceph.com:8000', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-05-31T19:49:26.018 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'zone', 'create', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--default', '--master', '--access-key', 'RVN19ED3W7FH7E08', '--secret', 'XzQjGe60gbIUStGfs96TKhuPw3IQi3hK', '--endpoints', 'http://smithi099.front.sepia.ceph.com:8000', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-05-31T19:49:26.018 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 zone create --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --default --master --access-key RVN19ED3W7FH7E08 --secret XzQjGe60gbIUStGfs96TKhuPw3IQi3hK --endpoints http://smithi099.front.sepia.ceph.com:8000 --cluster c1 --debug-rgw 0 --debug-ms 0 2023-05-31T19:49:26.113 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2023-05-31T19:49:26.113 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2023-05-31T19:49:26.127 INFO:teuthology.orchestra.run.smithi099.stderr:NOTICE: set zone's realm_id=aaa33de6-515b-4cac-8f1b-3a1a30452117 2023-05-31T19:49:26.147 INFO:teuthology.orchestra.run.smithi099.stdout:{ 2023-05-31T19:49:26.147 INFO:teuthology.orchestra.run.smithi099.stdout: "id": "3d163b1e-09c7-47e2-ae87-add4aace3f38", 2023-05-31T19:49:26.147 INFO:teuthology.orchestra.run.smithi099.stdout: "name": "a1", 2023-05-31T19:49:26.147 INFO:teuthology.orchestra.run.smithi099.stdout: "domain_root": "a1.rgw.meta:root", 2023-05-31T19:49:26.147 INFO:teuthology.orchestra.run.smithi099.stdout: "control_pool": "a1.rgw.control", 2023-05-31T19:49:26.147 INFO:teuthology.orchestra.run.smithi099.stdout: "gc_pool": "a1.rgw.log:gc", 2023-05-31T19:49:26.148 INFO:teuthology.orchestra.run.smithi099.stdout: "lc_pool": "a1.rgw.log:lc", 2023-05-31T19:49:26.148 INFO:teuthology.orchestra.run.smithi099.stdout: "log_pool": "a1.rgw.log", 2023-05-31T19:49:26.148 INFO:teuthology.orchestra.run.smithi099.stdout: "intent_log_pool": "a1.rgw.log:intent", 2023-05-31T19:49:26.148 INFO:teuthology.orchestra.run.smithi099.stdout: "usage_log_pool": "a1.rgw.log:usage", 2023-05-31T19:49:26.148 INFO:teuthology.orchestra.run.smithi099.stdout: "roles_pool": "a1.rgw.meta:roles", 2023-05-31T19:49:26.148 INFO:teuthology.orchestra.run.smithi099.stdout: "reshard_pool": "a1.rgw.log:reshard", 2023-05-31T19:49:26.148 INFO:teuthology.orchestra.run.smithi099.stdout: "user_keys_pool": "a1.rgw.meta:users.keys", 2023-05-31T19:49:26.149 INFO:teuthology.orchestra.run.smithi099.stdout: "user_email_pool": "a1.rgw.meta:users.email", 2023-05-31T19:49:26.149 INFO:teuthology.orchestra.run.smithi099.stdout: "user_swift_pool": "a1.rgw.meta:users.swift", 2023-05-31T19:49:26.149 INFO:teuthology.orchestra.run.smithi099.stdout: "user_uid_pool": "a1.rgw.meta:users.uid", 2023-05-31T19:49:26.149 INFO:teuthology.orchestra.run.smithi099.stdout: "otp_pool": "a1.rgw.otp", 2023-05-31T19:49:26.149 INFO:teuthology.orchestra.run.smithi099.stdout: "system_key": { 2023-05-31T19:49:26.149 INFO:teuthology.orchestra.run.smithi099.stdout: "access_key": "RVN19ED3W7FH7E08", 2023-05-31T19:49:26.149 INFO:teuthology.orchestra.run.smithi099.stdout: "secret_key": "XzQjGe60gbIUStGfs96TKhuPw3IQi3hK" 2023-05-31T19:49:26.150 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T19:49:26.150 INFO:teuthology.orchestra.run.smithi099.stdout: "placement_pools": [ 2023-05-31T19:49:26.150 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T19:49:26.150 INFO:teuthology.orchestra.run.smithi099.stdout: "key": "default-placement", 2023-05-31T19:49:26.150 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T19:49:26.150 INFO:teuthology.orchestra.run.smithi099.stdout: "index_pool": "a1.rgw.buckets.index", 2023-05-31T19:49:26.151 INFO:teuthology.orchestra.run.smithi099.stdout: "storage_classes": { 2023-05-31T19:49:26.151 INFO:teuthology.orchestra.run.smithi099.stdout: "STANDARD": { 2023-05-31T19:49:26.151 INFO:teuthology.orchestra.run.smithi099.stdout: "data_pool": "a1.rgw.buckets.data" 2023-05-31T19:49:26.151 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T19:49:26.151 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T19:49:26.151 INFO:teuthology.orchestra.run.smithi099.stdout: "data_extra_pool": "a1.rgw.buckets.non-ec", 2023-05-31T19:49:26.152 INFO:teuthology.orchestra.run.smithi099.stdout: "index_type": 0, 2023-05-31T19:49:26.152 INFO:teuthology.orchestra.run.smithi099.stdout: "inline_data": true 2023-05-31T19:49:26.152 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T19:49:26.153 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T19:49:26.153 INFO:teuthology.orchestra.run.smithi099.stdout: ], 2023-05-31T19:49:26.154 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_id": "aaa33de6-515b-4cac-8f1b-3a1a30452117", 2023-05-31T19:49:26.154 INFO:teuthology.orchestra.run.smithi099.stdout: "notif_pool": "a1.rgw.log:notif" 2023-05-31T19:49:26.154 INFO:teuthology.orchestra.run.smithi099.stdout:} 2023-05-31T19:49:26.156 DEBUG:teuthology.orchestra.run.smithi099:> sudo ceph osd pool create a1.rgw.buckets.data 64 64 --cluster c1 2023-05-31T19:49:26.873 INFO:teuthology.orchestra.run.smithi099.stderr:pool 'a1.rgw.buckets.data' created 2023-05-31T19:49:26.890 DEBUG:teuthology.orchestra.run.smithi099:> sudo ceph osd pool application enable a1.rgw.buckets.data rgw --cluster c1 2023-05-31T19:49:27.911 INFO:teuthology.orchestra.run.smithi099.stderr:enabled application 'rgw' on pool 'a1.rgw.buckets.data' 2023-05-31T19:49:27.918 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['zone', 'placement', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', 'modify', '--placement-id', 'default-placement', '--compression', 'random', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-05-31T19:49:27.918 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'zone', 'placement', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', 'modify', '--placement-id', 'default-placement', '--compression', 'random', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-05-31T19:49:27.919 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 zone placement --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm modify --placement-id default-placement --compression random --cluster c1 --debug-rgw 0 --debug-ms 0 2023-05-31T19:49:27.970 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2023-05-31T19:49:27.970 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2023-05-31T19:49:27.988 INFO:teuthology.orchestra.run.smithi099.stdout:{ 2023-05-31T19:49:27.988 INFO:teuthology.orchestra.run.smithi099.stdout: "id": "3d163b1e-09c7-47e2-ae87-add4aace3f38", 2023-05-31T19:49:27.988 INFO:teuthology.orchestra.run.smithi099.stdout: "name": "a1", 2023-05-31T19:49:27.988 INFO:teuthology.orchestra.run.smithi099.stdout: "domain_root": "a1.rgw.meta:root", 2023-05-31T19:49:27.989 INFO:teuthology.orchestra.run.smithi099.stdout: "control_pool": "a1.rgw.control", 2023-05-31T19:49:27.989 INFO:teuthology.orchestra.run.smithi099.stdout: "gc_pool": "a1.rgw.log:gc", 2023-05-31T19:49:27.989 INFO:teuthology.orchestra.run.smithi099.stdout: "lc_pool": "a1.rgw.log:lc", 2023-05-31T19:49:27.989 INFO:teuthology.orchestra.run.smithi099.stdout: "log_pool": "a1.rgw.log", 2023-05-31T19:49:27.989 INFO:teuthology.orchestra.run.smithi099.stdout: "intent_log_pool": "a1.rgw.log:intent", 2023-05-31T19:49:27.989 INFO:teuthology.orchestra.run.smithi099.stdout: "usage_log_pool": "a1.rgw.log:usage", 2023-05-31T19:49:27.989 INFO:teuthology.orchestra.run.smithi099.stdout: "roles_pool": "a1.rgw.meta:roles", 2023-05-31T19:49:27.990 INFO:teuthology.orchestra.run.smithi099.stdout: "reshard_pool": "a1.rgw.log:reshard", 2023-05-31T19:49:27.990 INFO:teuthology.orchestra.run.smithi099.stdout: "user_keys_pool": "a1.rgw.meta:users.keys", 2023-05-31T19:49:27.990 INFO:teuthology.orchestra.run.smithi099.stdout: "user_email_pool": "a1.rgw.meta:users.email", 2023-05-31T19:49:27.990 INFO:teuthology.orchestra.run.smithi099.stdout: "user_swift_pool": "a1.rgw.meta:users.swift", 2023-05-31T19:49:27.990 INFO:teuthology.orchestra.run.smithi099.stdout: "user_uid_pool": "a1.rgw.meta:users.uid", 2023-05-31T19:49:27.990 INFO:teuthology.orchestra.run.smithi099.stdout: "otp_pool": "a1.rgw.otp", 2023-05-31T19:49:27.990 INFO:teuthology.orchestra.run.smithi099.stdout: "system_key": { 2023-05-31T19:49:27.991 INFO:teuthology.orchestra.run.smithi099.stdout: "access_key": "RVN19ED3W7FH7E08", 2023-05-31T19:49:27.991 INFO:teuthology.orchestra.run.smithi099.stdout: "secret_key": "XzQjGe60gbIUStGfs96TKhuPw3IQi3hK" 2023-05-31T19:49:27.991 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T19:49:27.991 INFO:teuthology.orchestra.run.smithi099.stdout: "placement_pools": [ 2023-05-31T19:49:27.991 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T19:49:27.991 INFO:teuthology.orchestra.run.smithi099.stdout: "key": "default-placement", 2023-05-31T19:49:27.991 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T19:49:27.991 INFO:teuthology.orchestra.run.smithi099.stdout: "index_pool": "a1.rgw.buckets.index", 2023-05-31T19:49:27.992 INFO:teuthology.orchestra.run.smithi099.stdout: "storage_classes": { 2023-05-31T19:49:27.992 INFO:teuthology.orchestra.run.smithi099.stdout: "STANDARD": { 2023-05-31T19:49:27.992 INFO:teuthology.orchestra.run.smithi099.stdout: "data_pool": "a1.rgw.buckets.data", 2023-05-31T19:49:27.992 INFO:teuthology.orchestra.run.smithi099.stdout: "compression_type": "random" 2023-05-31T19:49:27.992 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T19:49:27.993 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T19:49:27.993 INFO:teuthology.orchestra.run.smithi099.stdout: "data_extra_pool": "a1.rgw.buckets.non-ec", 2023-05-31T19:49:27.993 INFO:teuthology.orchestra.run.smithi099.stdout: "index_type": 0, 2023-05-31T19:49:27.993 INFO:teuthology.orchestra.run.smithi099.stdout: "inline_data": true 2023-05-31T19:49:27.993 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T19:49:27.993 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T19:49:27.994 INFO:teuthology.orchestra.run.smithi099.stdout: ], 2023-05-31T19:49:27.994 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_id": "aaa33de6-515b-4cac-8f1b-3a1a30452117", 2023-05-31T19:49:27.994 INFO:teuthology.orchestra.run.smithi099.stdout: "notif_pool": "a1.rgw.log:notif" 2023-05-31T19:49:27.994 INFO:teuthology.orchestra.run.smithi099.stdout:} 2023-05-31T19:49:27.995 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['period', 'update', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-05-31T19:49:27.995 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'period', 'update', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-05-31T19:49:27.996 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 period update --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --commit --cluster c1 --debug-rgw 0 --debug-ms 0 2023-05-31T19:49:28.043 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2023-05-31T19:49:28.044 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2023-05-31T19:49:28.055 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T19:49:28.049+0000 7fbe36d75780 0 period (629b9ced-80af-41ed-8a6c-f2cfc334cbb4 does not have zone 3d163b1e-09c7-47e2-ae87-add4aace3f38 configured 2023-05-31T19:49:34.969 INFO:teuthology.orchestra.run.smithi099.stdout:{ 2023-05-31T19:49:34.970 INFO:teuthology.orchestra.run.smithi099.stdout: "id": "2e6123d2-d695-4350-9a01-61a455bc782b", 2023-05-31T19:49:34.970 INFO:teuthology.orchestra.run.smithi099.stdout: "epoch": 1, 2023-05-31T19:49:34.970 INFO:teuthology.orchestra.run.smithi099.stdout: "predecessor_uuid": "629b9ced-80af-41ed-8a6c-f2cfc334cbb4", 2023-05-31T19:49:34.970 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_status": [], 2023-05-31T19:49:34.971 INFO:teuthology.orchestra.run.smithi099.stdout: "period_map": { 2023-05-31T19:49:34.971 INFO:teuthology.orchestra.run.smithi099.stdout: "id": "2e6123d2-d695-4350-9a01-61a455bc782b", 2023-05-31T19:49:34.971 INFO:teuthology.orchestra.run.smithi099.stdout: "zonegroups": [ 2023-05-31T19:49:34.971 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T19:49:34.971 INFO:teuthology.orchestra.run.smithi099.stdout: "id": "a", 2023-05-31T19:49:34.972 INFO:teuthology.orchestra.run.smithi099.stdout: "name": "a", 2023-05-31T19:49:34.972 INFO:teuthology.orchestra.run.smithi099.stdout: "api_name": "", 2023-05-31T19:49:34.972 INFO:teuthology.orchestra.run.smithi099.stdout: "is_master": true, 2023-05-31T19:49:34.972 INFO:teuthology.orchestra.run.smithi099.stdout: "endpoints": [ 2023-05-31T19:49:34.972 INFO:teuthology.orchestra.run.smithi099.stdout: "http://smithi099.front.sepia.ceph.com:8000" 2023-05-31T19:49:34.973 INFO:teuthology.orchestra.run.smithi099.stdout: ], 2023-05-31T19:49:34.973 INFO:teuthology.orchestra.run.smithi099.stdout: "hostnames": [], 2023-05-31T19:49:34.973 INFO:teuthology.orchestra.run.smithi099.stdout: "hostnames_s3website": [], 2023-05-31T19:49:34.973 INFO:teuthology.orchestra.run.smithi099.stdout: "master_zone": "3d163b1e-09c7-47e2-ae87-add4aace3f38", 2023-05-31T19:49:34.973 INFO:teuthology.orchestra.run.smithi099.stdout: "zones": [ 2023-05-31T19:49:34.973 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T19:49:34.974 INFO:teuthology.orchestra.run.smithi099.stdout: "id": "3d163b1e-09c7-47e2-ae87-add4aace3f38", 2023-05-31T19:49:34.974 INFO:teuthology.orchestra.run.smithi099.stdout: "name": "a1", 2023-05-31T19:49:34.974 INFO:teuthology.orchestra.run.smithi099.stdout: "endpoints": [ 2023-05-31T19:49:34.974 INFO:teuthology.orchestra.run.smithi099.stdout: "http://smithi099.front.sepia.ceph.com:8000" 2023-05-31T19:49:34.974 INFO:teuthology.orchestra.run.smithi099.stdout: ], 2023-05-31T19:49:34.975 INFO:teuthology.orchestra.run.smithi099.stdout: "log_meta": false, 2023-05-31T19:49:34.975 INFO:teuthology.orchestra.run.smithi099.stdout: "log_data": false, 2023-05-31T19:49:34.975 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket_index_max_shards": 11, 2023-05-31T19:49:34.975 INFO:teuthology.orchestra.run.smithi099.stdout: "read_only": false, 2023-05-31T19:49:34.975 INFO:teuthology.orchestra.run.smithi099.stdout: "tier_type": "", 2023-05-31T19:49:34.976 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_from_all": true, 2023-05-31T19:49:34.976 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_from": [], 2023-05-31T19:49:34.976 INFO:teuthology.orchestra.run.smithi099.stdout: "redirect_zone": "", 2023-05-31T19:49:34.976 INFO:teuthology.orchestra.run.smithi099.stdout: "supported_features": [ 2023-05-31T19:49:34.977 INFO:teuthology.orchestra.run.smithi099.stdout: "resharding" 2023-05-31T19:49:34.977 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2023-05-31T19:49:34.977 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T19:49:34.977 INFO:teuthology.orchestra.run.smithi099.stdout: ], 2023-05-31T19:49:34.978 INFO:teuthology.orchestra.run.smithi099.stdout: "placement_targets": [ 2023-05-31T19:49:34.978 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T19:49:34.978 INFO:teuthology.orchestra.run.smithi099.stdout: "name": "default-placement", 2023-05-31T19:49:34.978 INFO:teuthology.orchestra.run.smithi099.stdout: "tags": [], 2023-05-31T19:49:34.979 INFO:teuthology.orchestra.run.smithi099.stdout: "storage_classes": [ 2023-05-31T19:49:34.979 INFO:teuthology.orchestra.run.smithi099.stdout: "STANDARD" 2023-05-31T19:49:34.979 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2023-05-31T19:49:34.979 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T19:49:34.980 INFO:teuthology.orchestra.run.smithi099.stdout: ], 2023-05-31T19:49:34.980 INFO:teuthology.orchestra.run.smithi099.stdout: "default_placement": "default-placement", 2023-05-31T19:49:34.980 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_id": "aaa33de6-515b-4cac-8f1b-3a1a30452117", 2023-05-31T19:49:34.981 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_policy": { 2023-05-31T19:49:34.981 INFO:teuthology.orchestra.run.smithi099.stdout: "groups": [] 2023-05-31T19:49:34.981 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T19:49:34.981 INFO:teuthology.orchestra.run.smithi099.stdout: "enabled_features": [ 2023-05-31T19:49:34.982 INFO:teuthology.orchestra.run.smithi099.stdout: "resharding" 2023-05-31T19:49:34.982 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2023-05-31T19:49:34.982 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T19:49:34.982 INFO:teuthology.orchestra.run.smithi099.stdout: ], 2023-05-31T19:49:34.983 INFO:teuthology.orchestra.run.smithi099.stdout: "short_zone_ids": [ 2023-05-31T19:49:34.983 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T19:49:34.983 INFO:teuthology.orchestra.run.smithi099.stdout: "key": "3d163b1e-09c7-47e2-ae87-add4aace3f38", 2023-05-31T19:49:34.984 INFO:teuthology.orchestra.run.smithi099.stdout: "val": 3101833771 2023-05-31T19:49:34.984 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T19:49:34.984 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2023-05-31T19:49:34.984 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T19:49:34.985 INFO:teuthology.orchestra.run.smithi099.stdout: "master_zonegroup": "a", 2023-05-31T19:49:34.985 INFO:teuthology.orchestra.run.smithi099.stdout: "master_zone": "3d163b1e-09c7-47e2-ae87-add4aace3f38", 2023-05-31T19:49:34.986 INFO:teuthology.orchestra.run.smithi099.stdout: "period_config": { 2023-05-31T19:49:34.986 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket_quota": { 2023-05-31T19:49:34.986 INFO:teuthology.orchestra.run.smithi099.stdout: "enabled": false, 2023-05-31T19:49:34.987 INFO:teuthology.orchestra.run.smithi099.stdout: "check_on_raw": false, 2023-05-31T19:49:34.987 INFO:teuthology.orchestra.run.smithi099.stdout: "max_size": -1, 2023-05-31T19:49:34.987 INFO:teuthology.orchestra.run.smithi099.stdout: "max_size_kb": 0, 2023-05-31T19:49:34.987 INFO:teuthology.orchestra.run.smithi099.stdout: "max_objects": -1 2023-05-31T19:49:34.988 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T19:49:34.989 INFO:teuthology.orchestra.run.smithi099.stdout: "user_quota": { 2023-05-31T19:49:34.989 INFO:teuthology.orchestra.run.smithi099.stdout: "enabled": false, 2023-05-31T19:49:34.990 INFO:teuthology.orchestra.run.smithi099.stdout: "check_on_raw": false, 2023-05-31T19:49:34.990 INFO:teuthology.orchestra.run.smithi099.stdout: "max_size": -1, 2023-05-31T19:49:34.990 INFO:teuthology.orchestra.run.smithi099.stdout: "max_size_kb": 0, 2023-05-31T19:49:34.991 INFO:teuthology.orchestra.run.smithi099.stdout: "max_objects": -1 2023-05-31T19:49:34.991 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T19:49:34.991 INFO:teuthology.orchestra.run.smithi099.stdout: "user_ratelimit": { 2023-05-31T19:49:34.991 INFO:teuthology.orchestra.run.smithi099.stdout: "max_read_ops": 0, 2023-05-31T19:49:34.992 INFO:teuthology.orchestra.run.smithi099.stdout: "max_write_ops": 0, 2023-05-31T19:49:34.992 INFO:teuthology.orchestra.run.smithi099.stdout: "max_read_bytes": 0, 2023-05-31T19:49:34.992 INFO:teuthology.orchestra.run.smithi099.stdout: "max_write_bytes": 0, 2023-05-31T19:49:34.992 INFO:teuthology.orchestra.run.smithi099.stdout: "enabled": false 2023-05-31T19:49:34.993 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T19:49:34.993 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket_ratelimit": { 2023-05-31T19:49:34.993 INFO:teuthology.orchestra.run.smithi099.stdout: "max_read_ops": 0, 2023-05-31T19:49:34.993 INFO:teuthology.orchestra.run.smithi099.stdout: "max_write_ops": 0, 2023-05-31T19:49:34.994 INFO:teuthology.orchestra.run.smithi099.stdout: "max_read_bytes": 0, 2023-05-31T19:49:34.994 INFO:teuthology.orchestra.run.smithi099.stdout: "max_write_bytes": 0, 2023-05-31T19:49:34.994 INFO:teuthology.orchestra.run.smithi099.stdout: "enabled": false 2023-05-31T19:49:34.994 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T19:49:34.995 INFO:teuthology.orchestra.run.smithi099.stdout: "anonymous_ratelimit": { 2023-05-31T19:49:34.995 INFO:teuthology.orchestra.run.smithi099.stdout: "max_read_ops": 0, 2023-05-31T19:49:34.995 INFO:teuthology.orchestra.run.smithi099.stdout: "max_write_ops": 0, 2023-05-31T19:49:34.995 INFO:teuthology.orchestra.run.smithi099.stdout: "max_read_bytes": 0, 2023-05-31T19:49:34.996 INFO:teuthology.orchestra.run.smithi099.stdout: "max_write_bytes": 0, 2023-05-31T19:49:34.996 INFO:teuthology.orchestra.run.smithi099.stdout: "enabled": false 2023-05-31T19:49:34.996 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T19:49:34.997 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T19:49:34.997 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_id": "aaa33de6-515b-4cac-8f1b-3a1a30452117", 2023-05-31T19:49:34.997 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_name": "test-realm", 2023-05-31T19:49:34.997 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2023-05-31T19:49:34.998 INFO:teuthology.orchestra.run.smithi099.stdout:} 2023-05-31T19:49:35.000 DEBUG:tasks.rgw.c1.client.0:waiting for process to exit 2023-05-31T19:49:35.000 INFO:teuthology.orchestra.run:waiting for 300 2023-05-31T19:49:35.001 INFO:tasks.rgw.c1.client.0.smithi099.stdout:2023-05-31T19:49:34.995+0000 7f76718eb700 -1 received signal: Terminated from /usr/bin/python3 /bin/daemon-helper term radosgw --rgw-frontends beast port=8000 -n client.0 --cluster c1 -k /etc/ceph/c1.client.0.keyring --log-file /var/log/ceph/rgw.c1.client.0.log --rgw_ops_log_socket_path /home/ubuntu/cephtest/rgw.opslog.c1.client.0.sock --foreground (PID: 110685) UID: 0 2023-05-31T19:49:35.002 INFO:tasks.rgw.c1.client.0.smithi099.stdout:2023-05-31T19:49:34.995+0000 7f7675b547c0 -1 shutting down 2023-05-31T19:49:36.107 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.0 is failed for ~0s 2023-05-31T19:49:41.102 INFO:tasks.rgw.c1.client.0:Stopped 2023-05-31T19:49:41.103 INFO:tasks.rgw.c1.client.0:Restarting daemon 2023-05-31T19:49:41.103 DEBUG:teuthology.orchestra.run.smithi099:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term radosgw --rgw-frontends 'beast port=8000' -n client.0 --cluster c1 -k /etc/ceph/c1.client.0.keyring --log-file /var/log/ceph/rgw.c1.client.0.log --rgw_ops_log_socket_path /home/ubuntu/cephtest/rgw.opslog.c1.client.0.sock --foreground --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm | sudo tee /var/log/ceph/rgw.c1.client.0.stdout 2>&1 2023-05-31T19:49:41.106 INFO:tasks.rgw.c1.client.0:Started 2023-05-31T19:49:41.106 DEBUG:teuthology.orchestra.run.smithi099:> curl http://smithi099.front.sepia.ceph.com:8000 2023-05-31T19:49:41.132 DEBUG:teuthology.orchestra.run:got remote process result: 7 2023-05-31T19:49:41.132 INFO:teuthology.orchestra.run.smithi099.stderr: % Total % Received % Xferd Average Speed Time Time Time Current 2023-05-31T19:49:41.133 INFO:teuthology.orchestra.run.smithi099.stderr: Dload Upload Total Spent Left Speed 2023-05-31T19:49:41.133 INFO:teuthology.orchestra.run.smithi099.stderr: 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0curl: (7) Failed to connect to smithi099.front.sepia.ceph.com port 8000: Connection refused 2023-05-31T19:49:41.813 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.0 has been restored 2023-05-31T19:49:42.135 DEBUG:teuthology.orchestra.run.smithi099:> curl http://smithi099.front.sepia.ceph.com:8000 2023-05-31T19:49:42.157 INFO:teuthology.orchestra.run.smithi099.stderr: % Total % Received % Xferd Average Speed Time Time Time Current 2023-05-31T19:49:42.158 INFO:teuthology.orchestra.run.smithi099.stderr: Dload Upload Total Spent Left Speed 2023-05-31T19:49:42.160 INFO:teuthology.orchestra.run.smithi099.stderr: 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 100 214 0 214 0 0 208k 0 --:--:-- --:--:-- --:--:-- 208k 2023-05-31T19:49:42.161 INFO:teuthology.orchestra.run.smithi099.stdout:anonymous 2023-05-31T19:49:42.162 INFO:tasks.rgw_multisite:creating admin user.. 2023-05-31T19:49:42.163 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['user', 'create', '--uid', 'realm-admin', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--display-name', 'Realm Admin', '--system', '--access-key', 'RVN19ED3W7FH7E08', '--secret', 'XzQjGe60gbIUStGfs96TKhuPw3IQi3hK', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-05-31T19:49:42.163 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'user', 'create', '--uid', 'realm-admin', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--display-name', 'Realm Admin', '--system', '--access-key', 'RVN19ED3W7FH7E08', '--secret', 'XzQjGe60gbIUStGfs96TKhuPw3IQi3hK', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-05-31T19:49:42.164 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 user create --uid realm-admin --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --display-name 'Realm Admin' --system --access-key RVN19ED3W7FH7E08 --secret XzQjGe60gbIUStGfs96TKhuPw3IQi3hK --cluster c1 --debug-rgw 0 --debug-ms 0 2023-05-31T19:49:42.258 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2023-05-31T19:49:42.258 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2023-05-31T19:49:42.308 INFO:teuthology.orchestra.run.smithi099.stdout:{ 2023-05-31T19:49:42.309 INFO:teuthology.orchestra.run.smithi099.stdout: "user_id": "realm-admin", 2023-05-31T19:49:42.309 INFO:teuthology.orchestra.run.smithi099.stdout: "display_name": "Realm Admin", 2023-05-31T19:49:42.309 INFO:teuthology.orchestra.run.smithi099.stdout: "email": "", 2023-05-31T19:49:42.309 INFO:teuthology.orchestra.run.smithi099.stdout: "suspended": 0, 2023-05-31T19:49:42.309 INFO:teuthology.orchestra.run.smithi099.stdout: "max_buckets": 1000, 2023-05-31T19:49:42.310 INFO:teuthology.orchestra.run.smithi099.stdout: "subusers": [], 2023-05-31T19:49:42.310 INFO:teuthology.orchestra.run.smithi099.stdout: "keys": [ 2023-05-31T19:49:42.310 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T19:49:42.311 INFO:teuthology.orchestra.run.smithi099.stdout: "user": "realm-admin", 2023-05-31T19:49:42.311 INFO:teuthology.orchestra.run.smithi099.stdout: "access_key": "RVN19ED3W7FH7E08", 2023-05-31T19:49:42.311 INFO:teuthology.orchestra.run.smithi099.stdout: "secret_key": "XzQjGe60gbIUStGfs96TKhuPw3IQi3hK" 2023-05-31T19:49:42.311 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T19:49:42.312 INFO:teuthology.orchestra.run.smithi099.stdout: ], 2023-05-31T19:49:42.312 INFO:teuthology.orchestra.run.smithi099.stdout: "swift_keys": [], 2023-05-31T19:49:42.312 INFO:teuthology.orchestra.run.smithi099.stdout: "caps": [], 2023-05-31T19:49:42.313 INFO:teuthology.orchestra.run.smithi099.stdout: "op_mask": "read, write, delete", 2023-05-31T19:49:42.313 INFO:teuthology.orchestra.run.smithi099.stdout: "system": true, 2023-05-31T19:49:42.313 INFO:teuthology.orchestra.run.smithi099.stdout: "default_placement": "", 2023-05-31T19:49:42.314 INFO:teuthology.orchestra.run.smithi099.stdout: "default_storage_class": "", 2023-05-31T19:49:42.314 INFO:teuthology.orchestra.run.smithi099.stdout: "placement_tags": [], 2023-05-31T19:49:42.314 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket_quota": { 2023-05-31T19:49:42.314 INFO:teuthology.orchestra.run.smithi099.stdout: "enabled": false, 2023-05-31T19:49:42.315 INFO:teuthology.orchestra.run.smithi099.stdout: "check_on_raw": false, 2023-05-31T19:49:42.315 INFO:teuthology.orchestra.run.smithi099.stdout: "max_size": -1, 2023-05-31T19:49:42.316 INFO:teuthology.orchestra.run.smithi099.stdout: "max_size_kb": 0, 2023-05-31T19:49:42.316 INFO:teuthology.orchestra.run.smithi099.stdout: "max_objects": -1 2023-05-31T19:49:42.316 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T19:49:42.316 INFO:teuthology.orchestra.run.smithi099.stdout: "user_quota": { 2023-05-31T19:49:42.317 INFO:teuthology.orchestra.run.smithi099.stdout: "enabled": false, 2023-05-31T19:49:42.317 INFO:teuthology.orchestra.run.smithi099.stdout: "check_on_raw": false, 2023-05-31T19:49:42.317 INFO:teuthology.orchestra.run.smithi099.stdout: "max_size": -1, 2023-05-31T19:49:42.317 INFO:teuthology.orchestra.run.smithi099.stdout: "max_size_kb": 0, 2023-05-31T19:49:42.318 INFO:teuthology.orchestra.run.smithi099.stdout: "max_objects": -1 2023-05-31T19:49:42.318 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T19:49:42.318 INFO:teuthology.orchestra.run.smithi099.stdout: "temp_url_keys": [], 2023-05-31T19:49:42.319 INFO:teuthology.orchestra.run.smithi099.stdout: "type": "rgw", 2023-05-31T19:49:42.319 INFO:teuthology.orchestra.run.smithi099.stdout: "mfa_ids": [] 2023-05-31T19:49:42.319 INFO:teuthology.orchestra.run.smithi099.stdout:} 2023-05-31T19:49:42.319 INFO:teuthology.orchestra.run.smithi099.stdout: 2023-05-31T19:49:42.328 INFO:tasks.rgw_multisite:creating zone.. 2023-05-31T19:49:42.328 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['zone', 'create', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--access-key', 'RVN19ED3W7FH7E08', '--secret', 'XzQjGe60gbIUStGfs96TKhuPw3IQi3hK', '--endpoints', 'http://smithi099.front.sepia.ceph.com:8001', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-05-31T19:49:42.329 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'zone', 'create', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--access-key', 'RVN19ED3W7FH7E08', '--secret', 'XzQjGe60gbIUStGfs96TKhuPw3IQi3hK', '--endpoints', 'http://smithi099.front.sepia.ceph.com:8001', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-05-31T19:49:42.329 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 zone create --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --access-key RVN19ED3W7FH7E08 --secret XzQjGe60gbIUStGfs96TKhuPw3IQi3hK --endpoints http://smithi099.front.sepia.ceph.com:8001 --cluster c1 --debug-rgw 0 --debug-ms 0 2023-05-31T19:49:42.382 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2023-05-31T19:49:42.383 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2023-05-31T19:49:42.397 INFO:teuthology.orchestra.run.smithi099.stderr:NOTICE: set zone's realm_id=aaa33de6-515b-4cac-8f1b-3a1a30452117 2023-05-31T19:49:42.416 INFO:teuthology.orchestra.run.smithi099.stdout:{ 2023-05-31T19:49:42.416 INFO:teuthology.orchestra.run.smithi099.stdout: "id": "5d8acec7-7e57-4e56-8b32-e3d6b5036001", 2023-05-31T19:49:42.416 INFO:teuthology.orchestra.run.smithi099.stdout: "name": "a2", 2023-05-31T19:49:42.417 INFO:teuthology.orchestra.run.smithi099.stdout: "domain_root": "a2.rgw.meta:root", 2023-05-31T19:49:42.417 INFO:teuthology.orchestra.run.smithi099.stdout: "control_pool": "a2.rgw.control", 2023-05-31T19:49:42.417 INFO:teuthology.orchestra.run.smithi099.stdout: "gc_pool": "a2.rgw.log:gc", 2023-05-31T19:49:42.417 INFO:teuthology.orchestra.run.smithi099.stdout: "lc_pool": "a2.rgw.log:lc", 2023-05-31T19:49:42.417 INFO:teuthology.orchestra.run.smithi099.stdout: "log_pool": "a2.rgw.log", 2023-05-31T19:49:42.418 INFO:teuthology.orchestra.run.smithi099.stdout: "intent_log_pool": "a2.rgw.log:intent", 2023-05-31T19:49:42.418 INFO:teuthology.orchestra.run.smithi099.stdout: "usage_log_pool": "a2.rgw.log:usage", 2023-05-31T19:49:42.418 INFO:teuthology.orchestra.run.smithi099.stdout: "roles_pool": "a2.rgw.meta:roles", 2023-05-31T19:49:42.418 INFO:teuthology.orchestra.run.smithi099.stdout: "reshard_pool": "a2.rgw.log:reshard", 2023-05-31T19:49:42.418 INFO:teuthology.orchestra.run.smithi099.stdout: "user_keys_pool": "a2.rgw.meta:users.keys", 2023-05-31T19:49:42.419 INFO:teuthology.orchestra.run.smithi099.stdout: "user_email_pool": "a2.rgw.meta:users.email", 2023-05-31T19:49:42.419 INFO:teuthology.orchestra.run.smithi099.stdout: "user_swift_pool": "a2.rgw.meta:users.swift", 2023-05-31T19:49:42.419 INFO:teuthology.orchestra.run.smithi099.stdout: "user_uid_pool": "a2.rgw.meta:users.uid", 2023-05-31T19:49:42.419 INFO:teuthology.orchestra.run.smithi099.stdout: "otp_pool": "a2.rgw.otp", 2023-05-31T19:49:42.419 INFO:teuthology.orchestra.run.smithi099.stdout: "system_key": { 2023-05-31T19:49:42.420 INFO:teuthology.orchestra.run.smithi099.stdout: "access_key": "RVN19ED3W7FH7E08", 2023-05-31T19:49:42.420 INFO:teuthology.orchestra.run.smithi099.stdout: "secret_key": "XzQjGe60gbIUStGfs96TKhuPw3IQi3hK" 2023-05-31T19:49:42.420 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T19:49:42.420 INFO:teuthology.orchestra.run.smithi099.stdout: "placement_pools": [ 2023-05-31T19:49:42.420 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T19:49:42.420 INFO:teuthology.orchestra.run.smithi099.stdout: "key": "default-placement", 2023-05-31T19:49:42.420 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T19:49:42.421 INFO:teuthology.orchestra.run.smithi099.stdout: "index_pool": "a2.rgw.buckets.index", 2023-05-31T19:49:42.421 INFO:teuthology.orchestra.run.smithi099.stdout: "storage_classes": { 2023-05-31T19:49:42.421 INFO:teuthology.orchestra.run.smithi099.stdout: "STANDARD": { 2023-05-31T19:49:42.421 INFO:teuthology.orchestra.run.smithi099.stdout: "data_pool": "a2.rgw.buckets.data" 2023-05-31T19:49:42.421 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T19:49:42.421 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T19:49:42.421 INFO:teuthology.orchestra.run.smithi099.stdout: "data_extra_pool": "a2.rgw.buckets.non-ec", 2023-05-31T19:49:42.422 INFO:teuthology.orchestra.run.smithi099.stdout: "index_type": 0, 2023-05-31T19:49:42.422 INFO:teuthology.orchestra.run.smithi099.stdout: "inline_data": true 2023-05-31T19:49:42.422 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T19:49:42.422 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T19:49:42.422 INFO:teuthology.orchestra.run.smithi099.stdout: ], 2023-05-31T19:49:42.423 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_id": "aaa33de6-515b-4cac-8f1b-3a1a30452117", 2023-05-31T19:49:42.423 INFO:teuthology.orchestra.run.smithi099.stdout: "notif_pool": "a2.rgw.log:notif" 2023-05-31T19:49:42.423 INFO:teuthology.orchestra.run.smithi099.stdout:} 2023-05-31T19:49:42.425 DEBUG:teuthology.orchestra.run.smithi099:> sudo ceph osd pool create a2.rgw.buckets.data 64 64 --cluster c1 2023-05-31T19:49:42.927 INFO:teuthology.orchestra.run.smithi099.stderr:pool 'a2.rgw.buckets.data' created 2023-05-31T19:49:42.947 DEBUG:teuthology.orchestra.run.smithi099:> sudo ceph osd pool application enable a2.rgw.buckets.data rgw --cluster c1 2023-05-31T19:49:43.956 INFO:teuthology.orchestra.run.smithi099.stderr:enabled application 'rgw' on pool 'a2.rgw.buckets.data' 2023-05-31T19:49:43.972 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['zone', 'placement', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', 'modify', '--placement-id', 'default-placement', '--compression', 'random', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-05-31T19:49:43.972 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'zone', 'placement', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', 'modify', '--placement-id', 'default-placement', '--compression', 'random', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-05-31T19:49:43.973 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 zone placement --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm modify --placement-id default-placement --compression random --cluster c1 --debug-rgw 0 --debug-ms 0 2023-05-31T19:49:44.026 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2023-05-31T19:49:44.026 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2023-05-31T19:49:44.044 INFO:teuthology.orchestra.run.smithi099.stdout:{ 2023-05-31T19:49:44.044 INFO:teuthology.orchestra.run.smithi099.stdout: "id": "5d8acec7-7e57-4e56-8b32-e3d6b5036001", 2023-05-31T19:49:44.044 INFO:teuthology.orchestra.run.smithi099.stdout: "name": "a2", 2023-05-31T19:49:44.045 INFO:teuthology.orchestra.run.smithi099.stdout: "domain_root": "a2.rgw.meta:root", 2023-05-31T19:49:44.045 INFO:teuthology.orchestra.run.smithi099.stdout: "control_pool": "a2.rgw.control", 2023-05-31T19:49:44.045 INFO:teuthology.orchestra.run.smithi099.stdout: "gc_pool": "a2.rgw.log:gc", 2023-05-31T19:49:44.045 INFO:teuthology.orchestra.run.smithi099.stdout: "lc_pool": "a2.rgw.log:lc", 2023-05-31T19:49:44.045 INFO:teuthology.orchestra.run.smithi099.stdout: "log_pool": "a2.rgw.log", 2023-05-31T19:49:44.045 INFO:teuthology.orchestra.run.smithi099.stdout: "intent_log_pool": "a2.rgw.log:intent", 2023-05-31T19:49:44.046 INFO:teuthology.orchestra.run.smithi099.stdout: "usage_log_pool": "a2.rgw.log:usage", 2023-05-31T19:49:44.046 INFO:teuthology.orchestra.run.smithi099.stdout: "roles_pool": "a2.rgw.meta:roles", 2023-05-31T19:49:44.046 INFO:teuthology.orchestra.run.smithi099.stdout: "reshard_pool": "a2.rgw.log:reshard", 2023-05-31T19:49:44.046 INFO:teuthology.orchestra.run.smithi099.stdout: "user_keys_pool": "a2.rgw.meta:users.keys", 2023-05-31T19:49:44.046 INFO:teuthology.orchestra.run.smithi099.stdout: "user_email_pool": "a2.rgw.meta:users.email", 2023-05-31T19:49:44.046 INFO:teuthology.orchestra.run.smithi099.stdout: "user_swift_pool": "a2.rgw.meta:users.swift", 2023-05-31T19:49:44.046 INFO:teuthology.orchestra.run.smithi099.stdout: "user_uid_pool": "a2.rgw.meta:users.uid", 2023-05-31T19:49:44.047 INFO:teuthology.orchestra.run.smithi099.stdout: "otp_pool": "a2.rgw.otp", 2023-05-31T19:49:44.047 INFO:teuthology.orchestra.run.smithi099.stdout: "system_key": { 2023-05-31T19:49:44.047 INFO:teuthology.orchestra.run.smithi099.stdout: "access_key": "RVN19ED3W7FH7E08", 2023-05-31T19:49:44.047 INFO:teuthology.orchestra.run.smithi099.stdout: "secret_key": "XzQjGe60gbIUStGfs96TKhuPw3IQi3hK" 2023-05-31T19:49:44.047 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T19:49:44.047 INFO:teuthology.orchestra.run.smithi099.stdout: "placement_pools": [ 2023-05-31T19:49:44.047 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T19:49:44.047 INFO:teuthology.orchestra.run.smithi099.stdout: "key": "default-placement", 2023-05-31T19:49:44.048 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T19:49:44.048 INFO:teuthology.orchestra.run.smithi099.stdout: "index_pool": "a2.rgw.buckets.index", 2023-05-31T19:49:44.048 INFO:teuthology.orchestra.run.smithi099.stdout: "storage_classes": { 2023-05-31T19:49:44.048 INFO:teuthology.orchestra.run.smithi099.stdout: "STANDARD": { 2023-05-31T19:49:44.048 INFO:teuthology.orchestra.run.smithi099.stdout: "data_pool": "a2.rgw.buckets.data", 2023-05-31T19:49:44.048 INFO:teuthology.orchestra.run.smithi099.stdout: "compression_type": "random" 2023-05-31T19:49:44.048 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T19:49:44.049 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T19:49:44.049 INFO:teuthology.orchestra.run.smithi099.stdout: "data_extra_pool": "a2.rgw.buckets.non-ec", 2023-05-31T19:49:44.049 INFO:teuthology.orchestra.run.smithi099.stdout: "index_type": 0, 2023-05-31T19:49:44.049 INFO:teuthology.orchestra.run.smithi099.stdout: "inline_data": true 2023-05-31T19:49:44.049 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T19:49:44.049 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T19:49:44.049 INFO:teuthology.orchestra.run.smithi099.stdout: ], 2023-05-31T19:49:44.049 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_id": "aaa33de6-515b-4cac-8f1b-3a1a30452117", 2023-05-31T19:49:44.050 INFO:teuthology.orchestra.run.smithi099.stdout: "notif_pool": "a2.rgw.log:notif" 2023-05-31T19:49:44.050 INFO:teuthology.orchestra.run.smithi099.stdout:} 2023-05-31T19:49:44.051 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['period', 'update', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-05-31T19:49:44.051 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'period', 'update', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-05-31T19:49:44.051 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 period update --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --commit --cluster c1 --debug-rgw 0 --debug-ms 0 2023-05-31T19:49:44.099 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2023-05-31T19:49:44.101 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2023-05-31T19:49:44.112 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T19:49:44.105+0000 7f0926036780 0 period (2e6123d2-d695-4350-9a01-61a455bc782b does not have zone 5d8acec7-7e57-4e56-8b32-e3d6b5036001 configured 2023-05-31T19:49:51.005 INFO:teuthology.orchestra.run.smithi099.stderr:Sending period to new master zone 3d163b1e-09c7-47e2-ae87-add4aace3f38 2023-05-31T19:49:51.031 INFO:teuthology.orchestra.run.smithi099.stdout:{ 2023-05-31T19:49:51.031 INFO:teuthology.orchestra.run.smithi099.stdout: "id": "2e6123d2-d695-4350-9a01-61a455bc782b", 2023-05-31T19:49:51.031 INFO:teuthology.orchestra.run.smithi099.stdout: "epoch": 2, 2023-05-31T19:49:51.031 INFO:teuthology.orchestra.run.smithi099.stdout: "predecessor_uuid": "629b9ced-80af-41ed-8a6c-f2cfc334cbb4", 2023-05-31T19:49:51.031 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_status": [], 2023-05-31T19:49:51.032 INFO:teuthology.orchestra.run.smithi099.stdout: "period_map": { 2023-05-31T19:49:51.032 INFO:teuthology.orchestra.run.smithi099.stdout: "id": "2e6123d2-d695-4350-9a01-61a455bc782b", 2023-05-31T19:49:51.032 INFO:teuthology.orchestra.run.smithi099.stdout: "zonegroups": [ 2023-05-31T19:49:51.032 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T19:49:51.032 INFO:teuthology.orchestra.run.smithi099.stdout: "id": "a", 2023-05-31T19:49:51.032 INFO:teuthology.orchestra.run.smithi099.stdout: "name": "a", 2023-05-31T19:49:51.032 INFO:teuthology.orchestra.run.smithi099.stdout: "api_name": "", 2023-05-31T19:49:51.033 INFO:teuthology.orchestra.run.smithi099.stdout: "is_master": true, 2023-05-31T19:49:51.033 INFO:teuthology.orchestra.run.smithi099.stdout: "endpoints": [ 2023-05-31T19:49:51.033 INFO:teuthology.orchestra.run.smithi099.stdout: "http://smithi099.front.sepia.ceph.com:8000" 2023-05-31T19:49:51.033 INFO:teuthology.orchestra.run.smithi099.stdout: ], 2023-05-31T19:49:51.033 INFO:teuthology.orchestra.run.smithi099.stdout: "hostnames": [], 2023-05-31T19:49:51.033 INFO:teuthology.orchestra.run.smithi099.stdout: "hostnames_s3website": [], 2023-05-31T19:49:51.033 INFO:teuthology.orchestra.run.smithi099.stdout: "master_zone": "3d163b1e-09c7-47e2-ae87-add4aace3f38", 2023-05-31T19:49:51.033 INFO:teuthology.orchestra.run.smithi099.stdout: "zones": [ 2023-05-31T19:49:51.034 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T19:49:51.034 INFO:teuthology.orchestra.run.smithi099.stdout: "id": "3d163b1e-09c7-47e2-ae87-add4aace3f38", 2023-05-31T19:49:51.034 INFO:teuthology.orchestra.run.smithi099.stdout: "name": "a1", 2023-05-31T19:49:51.035 INFO:teuthology.orchestra.run.smithi099.stdout: "endpoints": [ 2023-05-31T19:49:51.035 INFO:teuthology.orchestra.run.smithi099.stdout: "http://smithi099.front.sepia.ceph.com:8000" 2023-05-31T19:49:51.035 INFO:teuthology.orchestra.run.smithi099.stdout: ], 2023-05-31T19:49:51.035 INFO:teuthology.orchestra.run.smithi099.stdout: "log_meta": false, 2023-05-31T19:49:51.035 INFO:teuthology.orchestra.run.smithi099.stdout: "log_data": true, 2023-05-31T19:49:51.035 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket_index_max_shards": 11, 2023-05-31T19:49:51.035 INFO:teuthology.orchestra.run.smithi099.stdout: "read_only": false, 2023-05-31T19:49:51.036 INFO:teuthology.orchestra.run.smithi099.stdout: "tier_type": "", 2023-05-31T19:49:51.036 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_from_all": true, 2023-05-31T19:49:51.036 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_from": [], 2023-05-31T19:49:51.036 INFO:teuthology.orchestra.run.smithi099.stdout: "redirect_zone": "", 2023-05-31T19:49:51.036 INFO:teuthology.orchestra.run.smithi099.stdout: "supported_features": [ 2023-05-31T19:49:51.036 INFO:teuthology.orchestra.run.smithi099.stdout: "resharding" 2023-05-31T19:49:51.036 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2023-05-31T19:49:51.037 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T19:49:51.037 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T19:49:51.037 INFO:teuthology.orchestra.run.smithi099.stdout: "id": "5d8acec7-7e57-4e56-8b32-e3d6b5036001", 2023-05-31T19:49:51.037 INFO:teuthology.orchestra.run.smithi099.stdout: "name": "a2", 2023-05-31T19:49:51.037 INFO:teuthology.orchestra.run.smithi099.stdout: "endpoints": [ 2023-05-31T19:49:51.037 INFO:teuthology.orchestra.run.smithi099.stdout: "http://smithi099.front.sepia.ceph.com:8001" 2023-05-31T19:49:51.037 INFO:teuthology.orchestra.run.smithi099.stdout: ], 2023-05-31T19:49:51.037 INFO:teuthology.orchestra.run.smithi099.stdout: "log_meta": false, 2023-05-31T19:49:51.038 INFO:teuthology.orchestra.run.smithi099.stdout: "log_data": true, 2023-05-31T19:49:51.038 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket_index_max_shards": 11, 2023-05-31T19:49:51.038 INFO:teuthology.orchestra.run.smithi099.stdout: "read_only": false, 2023-05-31T19:49:51.038 INFO:teuthology.orchestra.run.smithi099.stdout: "tier_type": "", 2023-05-31T19:49:51.038 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_from_all": true, 2023-05-31T19:49:51.038 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_from": [], 2023-05-31T19:49:51.038 INFO:teuthology.orchestra.run.smithi099.stdout: "redirect_zone": "", 2023-05-31T19:49:51.038 INFO:teuthology.orchestra.run.smithi099.stdout: "supported_features": [ 2023-05-31T19:49:51.039 INFO:teuthology.orchestra.run.smithi099.stdout: "resharding" 2023-05-31T19:49:51.039 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2023-05-31T19:49:51.039 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T19:49:51.039 INFO:teuthology.orchestra.run.smithi099.stdout: ], 2023-05-31T19:49:51.039 INFO:teuthology.orchestra.run.smithi099.stdout: "placement_targets": [ 2023-05-31T19:49:51.039 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T19:49:51.039 INFO:teuthology.orchestra.run.smithi099.stdout: "name": "default-placement", 2023-05-31T19:49:51.040 INFO:teuthology.orchestra.run.smithi099.stdout: "tags": [], 2023-05-31T19:49:51.040 INFO:teuthology.orchestra.run.smithi099.stdout: "storage_classes": [ 2023-05-31T19:49:51.040 INFO:teuthology.orchestra.run.smithi099.stdout: "STANDARD" 2023-05-31T19:49:51.040 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2023-05-31T19:49:51.040 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T19:49:51.040 INFO:teuthology.orchestra.run.smithi099.stdout: ], 2023-05-31T19:49:51.040 INFO:teuthology.orchestra.run.smithi099.stdout: "default_placement": "default-placement", 2023-05-31T19:49:51.041 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_id": "aaa33de6-515b-4cac-8f1b-3a1a30452117", 2023-05-31T19:49:51.041 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_policy": { 2023-05-31T19:49:51.041 INFO:teuthology.orchestra.run.smithi099.stdout: "groups": [] 2023-05-31T19:49:51.041 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T19:49:51.041 INFO:teuthology.orchestra.run.smithi099.stdout: "enabled_features": [ 2023-05-31T19:49:51.041 INFO:teuthology.orchestra.run.smithi099.stdout: "resharding" 2023-05-31T19:49:51.041 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2023-05-31T19:49:51.041 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T19:49:51.042 INFO:teuthology.orchestra.run.smithi099.stdout: ], 2023-05-31T19:49:51.042 INFO:teuthology.orchestra.run.smithi099.stdout: "short_zone_ids": [ 2023-05-31T19:49:51.042 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T19:49:51.042 INFO:teuthology.orchestra.run.smithi099.stdout: "key": "3d163b1e-09c7-47e2-ae87-add4aace3f38", 2023-05-31T19:49:51.042 INFO:teuthology.orchestra.run.smithi099.stdout: "val": 3101833771 2023-05-31T19:49:51.042 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T19:49:51.042 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T19:49:51.042 INFO:teuthology.orchestra.run.smithi099.stdout: "key": "5d8acec7-7e57-4e56-8b32-e3d6b5036001", 2023-05-31T19:49:51.043 INFO:teuthology.orchestra.run.smithi099.stdout: "val": 4255530623 2023-05-31T19:49:51.043 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T19:49:51.043 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2023-05-31T19:49:51.043 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T19:49:51.043 INFO:teuthology.orchestra.run.smithi099.stdout: "master_zonegroup": "a", 2023-05-31T19:49:51.043 INFO:teuthology.orchestra.run.smithi099.stdout: "master_zone": "3d163b1e-09c7-47e2-ae87-add4aace3f38", 2023-05-31T19:49:51.043 INFO:teuthology.orchestra.run.smithi099.stdout: "period_config": { 2023-05-31T19:49:51.043 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket_quota": { 2023-05-31T19:49:51.044 INFO:teuthology.orchestra.run.smithi099.stdout: "enabled": false, 2023-05-31T19:49:51.044 INFO:teuthology.orchestra.run.smithi099.stdout: "check_on_raw": false, 2023-05-31T19:49:51.044 INFO:teuthology.orchestra.run.smithi099.stdout: "max_size": -1, 2023-05-31T19:49:51.044 INFO:teuthology.orchestra.run.smithi099.stdout: "max_size_kb": 0, 2023-05-31T19:49:51.044 INFO:teuthology.orchestra.run.smithi099.stdout: "max_objects": -1 2023-05-31T19:49:51.044 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T19:49:51.044 INFO:teuthology.orchestra.run.smithi099.stdout: "user_quota": { 2023-05-31T19:49:51.045 INFO:teuthology.orchestra.run.smithi099.stdout: "enabled": false, 2023-05-31T19:49:51.045 INFO:teuthology.orchestra.run.smithi099.stdout: "check_on_raw": false, 2023-05-31T19:49:51.045 INFO:teuthology.orchestra.run.smithi099.stdout: "max_size": -1, 2023-05-31T19:49:51.045 INFO:teuthology.orchestra.run.smithi099.stdout: "max_size_kb": 0, 2023-05-31T19:49:51.045 INFO:teuthology.orchestra.run.smithi099.stdout: "max_objects": -1 2023-05-31T19:49:51.045 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T19:49:51.045 INFO:teuthology.orchestra.run.smithi099.stdout: "user_ratelimit": { 2023-05-31T19:49:51.045 INFO:teuthology.orchestra.run.smithi099.stdout: "max_read_ops": 0, 2023-05-31T19:49:51.046 INFO:teuthology.orchestra.run.smithi099.stdout: "max_write_ops": 0, 2023-05-31T19:49:51.046 INFO:teuthology.orchestra.run.smithi099.stdout: "max_read_bytes": 0, 2023-05-31T19:49:51.046 INFO:teuthology.orchestra.run.smithi099.stdout: "max_write_bytes": 0, 2023-05-31T19:49:51.046 INFO:teuthology.orchestra.run.smithi099.stdout: "enabled": false 2023-05-31T19:49:51.046 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T19:49:51.046 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket_ratelimit": { 2023-05-31T19:49:51.046 INFO:teuthology.orchestra.run.smithi099.stdout: "max_read_ops": 0, 2023-05-31T19:49:51.047 INFO:teuthology.orchestra.run.smithi099.stdout: "max_write_ops": 0, 2023-05-31T19:49:51.047 INFO:teuthology.orchestra.run.smithi099.stdout: "max_read_bytes": 0, 2023-05-31T19:49:51.047 INFO:teuthology.orchestra.run.smithi099.stdout: "max_write_bytes": 0, 2023-05-31T19:49:51.047 INFO:teuthology.orchestra.run.smithi099.stdout: "enabled": false 2023-05-31T19:49:51.047 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T19:49:51.047 INFO:teuthology.orchestra.run.smithi099.stdout: "anonymous_ratelimit": { 2023-05-31T19:49:51.051 INFO:teuthology.orchestra.run.smithi099.stdout: "max_read_ops": 0, 2023-05-31T19:49:51.051 INFO:teuthology.orchestra.run.smithi099.stdout: "max_write_ops": 0, 2023-05-31T19:49:51.052 INFO:teuthology.orchestra.run.smithi099.stdout: "max_read_bytes": 0, 2023-05-31T19:49:51.052 INFO:teuthology.orchestra.run.smithi099.stdout: "max_write_bytes": 0, 2023-05-31T19:49:51.052 INFO:teuthology.orchestra.run.smithi099.stdout: "enabled": false 2023-05-31T19:49:51.052 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T19:49:51.052 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T19:49:51.052 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_id": "aaa33de6-515b-4cac-8f1b-3a1a30452117", 2023-05-31T19:49:51.052 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_name": "test-realm", 2023-05-31T19:49:51.053 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2023-05-31T19:49:51.053 INFO:teuthology.orchestra.run.smithi099.stdout:} 2023-05-31T19:49:51.054 DEBUG:tasks.rgw.c1.client.1:waiting for process to exit 2023-05-31T19:49:51.054 INFO:teuthology.orchestra.run:waiting for 300 2023-05-31T19:49:51.055 INFO:tasks.rgw.c1.client.1.smithi099.stdout:2023-05-31T19:49:51.050+0000 7f3472d70700 -1 received signal: Terminated from /usr/bin/python3 /bin/daemon-helper term radosgw --rgw-frontends beast port=8001 -n client.1 --cluster c1 -k /etc/ceph/c1.client.1.keyring --log-file /var/log/ceph/rgw.c1.client.1.log --rgw_ops_log_socket_path /home/ubuntu/cephtest/rgw.opslog.c1.client.1.sock --foreground (PID: 110687) UID: 0 2023-05-31T19:49:51.055 INFO:tasks.rgw.c1.client.1.smithi099.stdout:2023-05-31T19:49:51.050+0000 7f3476fd97c0 -1 shutting down 2023-05-31T19:49:53.125 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.1 is failed for ~0s 2023-05-31T19:49:57.157 INFO:tasks.rgw.c1.client.1:Stopped 2023-05-31T19:49:57.157 INFO:tasks.rgw.c1.client.1:Restarting daemon 2023-05-31T19:49:57.158 DEBUG:teuthology.orchestra.run.smithi099:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term radosgw --rgw-frontends 'beast port=8001' -n client.1 --cluster c1 -k /etc/ceph/c1.client.1.keyring --log-file /var/log/ceph/rgw.c1.client.1.log --rgw_ops_log_socket_path /home/ubuntu/cephtest/rgw.opslog.c1.client.1.sock --foreground --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm | sudo tee /var/log/ceph/rgw.c1.client.1.stdout 2>&1 2023-05-31T19:49:57.161 INFO:tasks.rgw.c1.client.1:Started 2023-05-31T19:49:57.161 DEBUG:teuthology.orchestra.run.smithi099:> curl http://smithi099.front.sepia.ceph.com:8001 2023-05-31T19:49:57.187 DEBUG:teuthology.orchestra.run:got remote process result: 7 2023-05-31T19:49:57.188 INFO:teuthology.orchestra.run.smithi099.stderr: % Total % Received % Xferd Average Speed Time Time Time Current 2023-05-31T19:49:57.188 INFO:teuthology.orchestra.run.smithi099.stderr: Dload Upload Total Spent Left Speed 2023-05-31T19:49:57.189 INFO:teuthology.orchestra.run.smithi099.stderr: 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0curl: (7) Failed to connect to smithi099.front.sepia.ceph.com port 8001: Connection refused 2023-05-31T19:49:58.190 DEBUG:teuthology.orchestra.run.smithi099:> curl http://smithi099.front.sepia.ceph.com:8001 2023-05-31T19:49:58.214 INFO:teuthology.orchestra.run.smithi099.stderr: % Total % Received % Xferd Average Speed Time Time Time Current 2023-05-31T19:49:58.215 INFO:teuthology.orchestra.run.smithi099.stderr: Dload Upload Total Spent Left Speed 2023-05-31T19:49:58.215 INFO:teuthology.orchestra.run.smithi099.stderr: 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 100 214 0 214 0 0 208k 0 --:--:-- --:--:-- --:--:-- 208k 2023-05-31T19:49:58.216 INFO:teuthology.orchestra.run.smithi099.stdout:anonymous 2023-05-31T19:49:58.217 INFO:tasks.rgw_multisite:pulling realm configuration to c2 2023-05-31T19:49:58.218 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['realm', 'pull', '--rgw-realm', 'test-realm', '--url', 'http://smithi099.front.sepia.ceph.com:8000', '--access-key', 'RVN19ED3W7FH7E08', '--secret', 'XzQjGe60gbIUStGfs96TKhuPw3IQi3hK', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2023-05-31T19:49:58.218 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'realm', 'pull', '--rgw-realm', 'test-realm', '--url', 'http://smithi099.front.sepia.ceph.com:8000', '--access-key', 'RVN19ED3W7FH7E08', '--secret', 'XzQjGe60gbIUStGfs96TKhuPw3IQi3hK', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2023-05-31T19:49:58.218 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 realm pull --rgw-realm test-realm --url http://smithi099.front.sepia.ceph.com:8000 --access-key RVN19ED3W7FH7E08 --secret XzQjGe60gbIUStGfs96TKhuPw3IQi3hK --cluster c2 --debug-rgw 0 --debug-ms 0 2023-05-31T19:49:58.279 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-05-31T19:49:58.279 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-05-31T19:49:58.315 INFO:teuthology.orchestra.run.smithi112.stdout:{ 2023-05-31T19:49:58.316 INFO:teuthology.orchestra.run.smithi112.stdout: "id": "aaa33de6-515b-4cac-8f1b-3a1a30452117", 2023-05-31T19:49:58.316 INFO:teuthology.orchestra.run.smithi112.stdout: "name": "test-realm", 2023-05-31T19:49:58.316 INFO:teuthology.orchestra.run.smithi112.stdout: "current_period": "2e6123d2-d695-4350-9a01-61a455bc782b", 2023-05-31T19:49:58.316 INFO:teuthology.orchestra.run.smithi112.stdout: "epoch": 2 2023-05-31T19:49:58.316 INFO:teuthology.orchestra.run.smithi112.stdout:} 2023-05-31T19:49:58.317 INFO:tasks.rgw_multisite:creating zonegroup.. 2023-05-31T19:49:58.318 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['zonegroup', 'set', '--rgw-zonegroup', 'b', '--rgw-realm', 'test-realm', '--default', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2023-05-31T19:49:58.318 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'zonegroup', 'set', '--rgw-zonegroup', 'b', '--rgw-realm', 'test-realm', '--default', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2023-05-31T19:49:58.318 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 zonegroup set --rgw-zonegroup b --rgw-realm test-realm --default --cluster c2 --debug-rgw 0 --debug-ms 0 2023-05-31T19:49:58.367 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-05-31T19:49:58.367 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-05-31T19:49:58.381 INFO:teuthology.orchestra.run.smithi112.stderr:2023-05-31T19:49:58.375+0000 7f6ac9839780 -1 old format 2023-05-31T19:49:58.387 INFO:teuthology.orchestra.run.smithi112.stdout:{ 2023-05-31T19:49:58.387 INFO:teuthology.orchestra.run.smithi112.stdout: "id": "b", 2023-05-31T19:49:58.387 INFO:teuthology.orchestra.run.smithi112.stdout: "name": "b", 2023-05-31T19:49:58.387 INFO:teuthology.orchestra.run.smithi112.stdout: "api_name": "", 2023-05-31T19:49:58.387 INFO:teuthology.orchestra.run.smithi112.stdout: "is_master": false, 2023-05-31T19:49:58.388 INFO:teuthology.orchestra.run.smithi112.stdout: "endpoints": [ 2023-05-31T19:49:58.388 INFO:teuthology.orchestra.run.smithi112.stdout: "http://smithi112.front.sepia.ceph.com:8000" 2023-05-31T19:49:58.388 INFO:teuthology.orchestra.run.smithi112.stdout: ], 2023-05-31T19:49:58.388 INFO:teuthology.orchestra.run.smithi112.stdout: "hostnames": [], 2023-05-31T19:49:58.388 INFO:teuthology.orchestra.run.smithi112.stdout: "hostnames_s3website": [], 2023-05-31T19:49:58.388 INFO:teuthology.orchestra.run.smithi112.stdout: "master_zone": "", 2023-05-31T19:49:58.389 INFO:teuthology.orchestra.run.smithi112.stdout: "zones": [], 2023-05-31T19:49:58.389 INFO:teuthology.orchestra.run.smithi112.stdout: "placement_targets": [ 2023-05-31T19:49:58.389 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-05-31T19:49:58.389 INFO:teuthology.orchestra.run.smithi112.stdout: "name": "default-placement", 2023-05-31T19:49:58.389 INFO:teuthology.orchestra.run.smithi112.stdout: "tags": [], 2023-05-31T19:49:58.389 INFO:teuthology.orchestra.run.smithi112.stdout: "storage_classes": [] 2023-05-31T19:49:58.390 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-05-31T19:49:58.390 INFO:teuthology.orchestra.run.smithi112.stdout: ], 2023-05-31T19:49:58.390 INFO:teuthology.orchestra.run.smithi112.stdout: "default_placement": "default-placement", 2023-05-31T19:49:58.390 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_id": "aaa33de6-515b-4cac-8f1b-3a1a30452117", 2023-05-31T19:49:58.390 INFO:teuthology.orchestra.run.smithi112.stdout: "sync_policy": { 2023-05-31T19:49:58.390 INFO:teuthology.orchestra.run.smithi112.stdout: "groups": [] 2023-05-31T19:49:58.390 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-05-31T19:49:58.391 INFO:teuthology.orchestra.run.smithi112.stdout: "enabled_features": [ 2023-05-31T19:49:58.391 INFO:teuthology.orchestra.run.smithi112.stdout: "resharding" 2023-05-31T19:49:58.391 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2023-05-31T19:49:58.391 INFO:teuthology.orchestra.run.smithi112.stdout:} 2023-05-31T19:49:58.392 INFO:tasks.rgw_multisite:creating zone.. 2023-05-31T19:49:58.393 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['zone', 'create', '--rgw-zone', 'b1', '--rgw-zonegroup', 'b', '--rgw-realm', 'test-realm', '--default', '--master', '--access-key', 'RVN19ED3W7FH7E08', '--secret', 'XzQjGe60gbIUStGfs96TKhuPw3IQi3hK', '--endpoints', 'http://smithi112.front.sepia.ceph.com:8000', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2023-05-31T19:49:58.394 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'zone', 'create', '--rgw-zone', 'b1', '--rgw-zonegroup', 'b', '--rgw-realm', 'test-realm', '--default', '--master', '--access-key', 'RVN19ED3W7FH7E08', '--secret', 'XzQjGe60gbIUStGfs96TKhuPw3IQi3hK', '--endpoints', 'http://smithi112.front.sepia.ceph.com:8000', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2023-05-31T19:49:58.394 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 zone create --rgw-zone b1 --rgw-zonegroup b --rgw-realm test-realm --default --master --access-key RVN19ED3W7FH7E08 --secret XzQjGe60gbIUStGfs96TKhuPw3IQi3hK --endpoints http://smithi112.front.sepia.ceph.com:8000 --cluster c2 --debug-rgw 0 --debug-ms 0 2023-05-31T19:49:58.488 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-05-31T19:49:58.490 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-05-31T19:49:58.504 INFO:teuthology.orchestra.run.smithi112.stderr:NOTICE: set zone's realm_id=aaa33de6-515b-4cac-8f1b-3a1a30452117 2023-05-31T19:49:58.518 INFO:teuthology.orchestra.run.smithi112.stdout:{ 2023-05-31T19:49:58.518 INFO:teuthology.orchestra.run.smithi112.stdout: "id": "f85d7236-2f27-4988-af26-9c66ab280b1b", 2023-05-31T19:49:58.519 INFO:teuthology.orchestra.run.smithi112.stdout: "name": "b1", 2023-05-31T19:49:58.519 INFO:teuthology.orchestra.run.smithi112.stdout: "domain_root": "b1.rgw.meta:root", 2023-05-31T19:49:58.519 INFO:teuthology.orchestra.run.smithi112.stdout: "control_pool": "b1.rgw.control", 2023-05-31T19:49:58.519 INFO:teuthology.orchestra.run.smithi112.stdout: "gc_pool": "b1.rgw.log:gc", 2023-05-31T19:49:58.519 INFO:teuthology.orchestra.run.smithi112.stdout: "lc_pool": "b1.rgw.log:lc", 2023-05-31T19:49:58.519 INFO:teuthology.orchestra.run.smithi112.stdout: "log_pool": "b1.rgw.log", 2023-05-31T19:49:58.520 INFO:teuthology.orchestra.run.smithi112.stdout: "intent_log_pool": "b1.rgw.log:intent", 2023-05-31T19:49:58.520 INFO:teuthology.orchestra.run.smithi112.stdout: "usage_log_pool": "b1.rgw.log:usage", 2023-05-31T19:49:58.520 INFO:teuthology.orchestra.run.smithi112.stdout: "roles_pool": "b1.rgw.meta:roles", 2023-05-31T19:49:58.521 INFO:teuthology.orchestra.run.smithi112.stdout: "reshard_pool": "b1.rgw.log:reshard", 2023-05-31T19:49:58.521 INFO:teuthology.orchestra.run.smithi112.stdout: "user_keys_pool": "b1.rgw.meta:users.keys", 2023-05-31T19:49:58.521 INFO:teuthology.orchestra.run.smithi112.stdout: "user_email_pool": "b1.rgw.meta:users.email", 2023-05-31T19:49:58.521 INFO:teuthology.orchestra.run.smithi112.stdout: "user_swift_pool": "b1.rgw.meta:users.swift", 2023-05-31T19:49:58.521 INFO:teuthology.orchestra.run.smithi112.stdout: "user_uid_pool": "b1.rgw.meta:users.uid", 2023-05-31T19:49:58.521 INFO:teuthology.orchestra.run.smithi112.stdout: "otp_pool": "b1.rgw.otp", 2023-05-31T19:49:58.521 INFO:teuthology.orchestra.run.smithi112.stdout: "system_key": { 2023-05-31T19:49:58.522 INFO:teuthology.orchestra.run.smithi112.stdout: "access_key": "RVN19ED3W7FH7E08", 2023-05-31T19:49:58.522 INFO:teuthology.orchestra.run.smithi112.stdout: "secret_key": "XzQjGe60gbIUStGfs96TKhuPw3IQi3hK" 2023-05-31T19:49:58.522 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-05-31T19:49:58.522 INFO:teuthology.orchestra.run.smithi112.stdout: "placement_pools": [ 2023-05-31T19:49:58.522 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-05-31T19:49:58.522 INFO:teuthology.orchestra.run.smithi112.stdout: "key": "default-placement", 2023-05-31T19:49:58.522 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-05-31T19:49:58.523 INFO:teuthology.orchestra.run.smithi112.stdout: "index_pool": "b1.rgw.buckets.index", 2023-05-31T19:49:58.523 INFO:teuthology.orchestra.run.smithi112.stdout: "storage_classes": { 2023-05-31T19:49:58.523 INFO:teuthology.orchestra.run.smithi112.stdout: "STANDARD": { 2023-05-31T19:49:58.523 INFO:teuthology.orchestra.run.smithi112.stdout: "data_pool": "b1.rgw.buckets.data" 2023-05-31T19:49:58.523 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-05-31T19:49:58.523 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-05-31T19:49:58.523 INFO:teuthology.orchestra.run.smithi112.stdout: "data_extra_pool": "b1.rgw.buckets.non-ec", 2023-05-31T19:49:58.523 INFO:teuthology.orchestra.run.smithi112.stdout: "index_type": 0, 2023-05-31T19:49:58.524 INFO:teuthology.orchestra.run.smithi112.stdout: "inline_data": true 2023-05-31T19:49:58.525 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-05-31T19:49:58.525 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-05-31T19:49:58.525 INFO:teuthology.orchestra.run.smithi112.stdout: ], 2023-05-31T19:49:58.525 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_id": "aaa33de6-515b-4cac-8f1b-3a1a30452117", 2023-05-31T19:49:58.526 INFO:teuthology.orchestra.run.smithi112.stdout: "notif_pool": "b1.rgw.log:notif" 2023-05-31T19:49:58.526 INFO:teuthology.orchestra.run.smithi112.stdout:} 2023-05-31T19:49:58.527 DEBUG:teuthology.orchestra.run.smithi112:> sudo ceph osd pool create b1.rgw.buckets.data 64 64 --cluster c2 2023-05-31T19:49:58.829 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.1 has been restored 2023-05-31T19:49:59.186 INFO:teuthology.orchestra.run.smithi112.stderr:pool 'b1.rgw.buckets.data' created 2023-05-31T19:49:59.203 DEBUG:teuthology.orchestra.run.smithi112:> sudo ceph osd pool application enable b1.rgw.buckets.data rgw --cluster c2 2023-05-31T19:50:00.217 INFO:teuthology.orchestra.run.smithi112.stderr:enabled application 'rgw' on pool 'b1.rgw.buckets.data' 2023-05-31T19:50:00.232 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['zone', 'placement', '--rgw-zone', 'b1', '--rgw-zonegroup', 'b', '--rgw-realm', 'test-realm', 'modify', '--placement-id', 'default-placement', '--compression', 'random', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2023-05-31T19:50:00.233 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'zone', 'placement', '--rgw-zone', 'b1', '--rgw-zonegroup', 'b', '--rgw-realm', 'test-realm', 'modify', '--placement-id', 'default-placement', '--compression', 'random', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2023-05-31T19:50:00.233 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 zone placement --rgw-zone b1 --rgw-zonegroup b --rgw-realm test-realm modify --placement-id default-placement --compression random --cluster c2 --debug-rgw 0 --debug-ms 0 2023-05-31T19:50:00.287 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-05-31T19:50:00.288 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-05-31T19:50:00.306 INFO:teuthology.orchestra.run.smithi112.stdout:{ 2023-05-31T19:50:00.307 INFO:teuthology.orchestra.run.smithi112.stdout: "id": "f85d7236-2f27-4988-af26-9c66ab280b1b", 2023-05-31T19:50:00.307 INFO:teuthology.orchestra.run.smithi112.stdout: "name": "b1", 2023-05-31T19:50:00.307 INFO:teuthology.orchestra.run.smithi112.stdout: "domain_root": "b1.rgw.meta:root", 2023-05-31T19:50:00.307 INFO:teuthology.orchestra.run.smithi112.stdout: "control_pool": "b1.rgw.control", 2023-05-31T19:50:00.308 INFO:teuthology.orchestra.run.smithi112.stdout: "gc_pool": "b1.rgw.log:gc", 2023-05-31T19:50:00.308 INFO:teuthology.orchestra.run.smithi112.stdout: "lc_pool": "b1.rgw.log:lc", 2023-05-31T19:50:00.308 INFO:teuthology.orchestra.run.smithi112.stdout: "log_pool": "b1.rgw.log", 2023-05-31T19:50:00.308 INFO:teuthology.orchestra.run.smithi112.stdout: "intent_log_pool": "b1.rgw.log:intent", 2023-05-31T19:50:00.308 INFO:teuthology.orchestra.run.smithi112.stdout: "usage_log_pool": "b1.rgw.log:usage", 2023-05-31T19:50:00.309 INFO:teuthology.orchestra.run.smithi112.stdout: "roles_pool": "b1.rgw.meta:roles", 2023-05-31T19:50:00.309 INFO:teuthology.orchestra.run.smithi112.stdout: "reshard_pool": "b1.rgw.log:reshard", 2023-05-31T19:50:00.309 INFO:teuthology.orchestra.run.smithi112.stdout: "user_keys_pool": "b1.rgw.meta:users.keys", 2023-05-31T19:50:00.309 INFO:teuthology.orchestra.run.smithi112.stdout: "user_email_pool": "b1.rgw.meta:users.email", 2023-05-31T19:50:00.309 INFO:teuthology.orchestra.run.smithi112.stdout: "user_swift_pool": "b1.rgw.meta:users.swift", 2023-05-31T19:50:00.310 INFO:teuthology.orchestra.run.smithi112.stdout: "user_uid_pool": "b1.rgw.meta:users.uid", 2023-05-31T19:50:00.310 INFO:teuthology.orchestra.run.smithi112.stdout: "otp_pool": "b1.rgw.otp", 2023-05-31T19:50:00.310 INFO:teuthology.orchestra.run.smithi112.stdout: "system_key": { 2023-05-31T19:50:00.310 INFO:teuthology.orchestra.run.smithi112.stdout: "access_key": "RVN19ED3W7FH7E08", 2023-05-31T19:50:00.310 INFO:teuthology.orchestra.run.smithi112.stdout: "secret_key": "XzQjGe60gbIUStGfs96TKhuPw3IQi3hK" 2023-05-31T19:50:00.311 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-05-31T19:50:00.311 INFO:teuthology.orchestra.run.smithi112.stdout: "placement_pools": [ 2023-05-31T19:50:00.311 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-05-31T19:50:00.311 INFO:teuthology.orchestra.run.smithi112.stdout: "key": "default-placement", 2023-05-31T19:50:00.312 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-05-31T19:50:00.312 INFO:teuthology.orchestra.run.smithi112.stdout: "index_pool": "b1.rgw.buckets.index", 2023-05-31T19:50:00.312 INFO:teuthology.orchestra.run.smithi112.stdout: "storage_classes": { 2023-05-31T19:50:00.313 INFO:teuthology.orchestra.run.smithi112.stdout: "STANDARD": { 2023-05-31T19:50:00.313 INFO:teuthology.orchestra.run.smithi112.stdout: "data_pool": "b1.rgw.buckets.data", 2023-05-31T19:50:00.313 INFO:teuthology.orchestra.run.smithi112.stdout: "compression_type": "random" 2023-05-31T19:50:00.313 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-05-31T19:50:00.314 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-05-31T19:50:00.314 INFO:teuthology.orchestra.run.smithi112.stdout: "data_extra_pool": "b1.rgw.buckets.non-ec", 2023-05-31T19:50:00.314 INFO:teuthology.orchestra.run.smithi112.stdout: "index_type": 0, 2023-05-31T19:50:00.314 INFO:teuthology.orchestra.run.smithi112.stdout: "inline_data": true 2023-05-31T19:50:00.314 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-05-31T19:50:00.314 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-05-31T19:50:00.314 INFO:teuthology.orchestra.run.smithi112.stdout: ], 2023-05-31T19:50:00.315 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_id": "aaa33de6-515b-4cac-8f1b-3a1a30452117", 2023-05-31T19:50:00.315 INFO:teuthology.orchestra.run.smithi112.stdout: "notif_pool": "b1.rgw.log:notif" 2023-05-31T19:50:00.315 INFO:teuthology.orchestra.run.smithi112.stdout:} 2023-05-31T19:50:00.316 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['period', 'update', '--rgw-zone', 'b1', '--rgw-zonegroup', 'b', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2023-05-31T19:50:00.316 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'period', 'update', '--rgw-zone', 'b1', '--rgw-zonegroup', 'b', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2023-05-31T19:50:00.317 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 period update --rgw-zone b1 --rgw-zonegroup b --rgw-realm test-realm --commit --cluster c2 --debug-rgw 0 --debug-ms 0 2023-05-31T19:50:00.365 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-05-31T19:50:00.366 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-05-31T19:50:00.377 INFO:teuthology.orchestra.run.smithi112.stderr:2023-05-31T19:50:00.371+0000 7ff0fa684780 0 period (2e6123d2-d695-4350-9a01-61a455bc782b does not have zone f85d7236-2f27-4988-af26-9c66ab280b1b configured 2023-05-31T19:50:07.254 INFO:teuthology.orchestra.run.smithi112.stderr:Sending period to new master zone 3d163b1e-09c7-47e2-ae87-add4aace3f38 2023-05-31T19:50:07.296 INFO:teuthology.orchestra.run.smithi112.stdout:{ 2023-05-31T19:50:07.296 INFO:teuthology.orchestra.run.smithi112.stdout: "id": "2e6123d2-d695-4350-9a01-61a455bc782b", 2023-05-31T19:50:07.297 INFO:teuthology.orchestra.run.smithi112.stdout: "epoch": 3, 2023-05-31T19:50:07.297 INFO:teuthology.orchestra.run.smithi112.stdout: "predecessor_uuid": "629b9ced-80af-41ed-8a6c-f2cfc334cbb4", 2023-05-31T19:50:07.298 INFO:teuthology.orchestra.run.smithi112.stdout: "sync_status": [], 2023-05-31T19:50:07.298 INFO:teuthology.orchestra.run.smithi112.stdout: "period_map": { 2023-05-31T19:50:07.298 INFO:teuthology.orchestra.run.smithi112.stdout: "id": "2e6123d2-d695-4350-9a01-61a455bc782b", 2023-05-31T19:50:07.298 INFO:teuthology.orchestra.run.smithi112.stdout: "zonegroups": [ 2023-05-31T19:50:07.299 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-05-31T19:50:07.299 INFO:teuthology.orchestra.run.smithi112.stdout: "id": "a", 2023-05-31T19:50:07.299 INFO:teuthology.orchestra.run.smithi112.stdout: "name": "a", 2023-05-31T19:50:07.299 INFO:teuthology.orchestra.run.smithi112.stdout: "api_name": "", 2023-05-31T19:50:07.299 INFO:teuthology.orchestra.run.smithi112.stdout: "is_master": true, 2023-05-31T19:50:07.300 INFO:teuthology.orchestra.run.smithi112.stdout: "endpoints": [ 2023-05-31T19:50:07.300 INFO:teuthology.orchestra.run.smithi112.stdout: "http://smithi099.front.sepia.ceph.com:8000" 2023-05-31T19:50:07.300 INFO:teuthology.orchestra.run.smithi112.stdout: ], 2023-05-31T19:50:07.301 INFO:teuthology.orchestra.run.smithi112.stdout: "hostnames": [], 2023-05-31T19:50:07.301 INFO:teuthology.orchestra.run.smithi112.stdout: "hostnames_s3website": [], 2023-05-31T19:50:07.301 INFO:teuthology.orchestra.run.smithi112.stdout: "master_zone": "3d163b1e-09c7-47e2-ae87-add4aace3f38", 2023-05-31T19:50:07.301 INFO:teuthology.orchestra.run.smithi112.stdout: "zones": [ 2023-05-31T19:50:07.302 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-05-31T19:50:07.302 INFO:teuthology.orchestra.run.smithi112.stdout: "id": "3d163b1e-09c7-47e2-ae87-add4aace3f38", 2023-05-31T19:50:07.302 INFO:teuthology.orchestra.run.smithi112.stdout: "name": "a1", 2023-05-31T19:50:07.302 INFO:teuthology.orchestra.run.smithi112.stdout: "endpoints": [ 2023-05-31T19:50:07.302 INFO:teuthology.orchestra.run.smithi112.stdout: "http://smithi099.front.sepia.ceph.com:8000" 2023-05-31T19:50:07.302 INFO:teuthology.orchestra.run.smithi112.stdout: ], 2023-05-31T19:50:07.302 INFO:teuthology.orchestra.run.smithi112.stdout: "log_meta": false, 2023-05-31T19:50:07.303 INFO:teuthology.orchestra.run.smithi112.stdout: "log_data": true, 2023-05-31T19:50:07.303 INFO:teuthology.orchestra.run.smithi112.stdout: "bucket_index_max_shards": 11, 2023-05-31T19:50:07.303 INFO:teuthology.orchestra.run.smithi112.stdout: "read_only": false, 2023-05-31T19:50:07.303 INFO:teuthology.orchestra.run.smithi112.stdout: "tier_type": "", 2023-05-31T19:50:07.303 INFO:teuthology.orchestra.run.smithi112.stdout: "sync_from_all": true, 2023-05-31T19:50:07.303 INFO:teuthology.orchestra.run.smithi112.stdout: "sync_from": [], 2023-05-31T19:50:07.303 INFO:teuthology.orchestra.run.smithi112.stdout: "redirect_zone": "", 2023-05-31T19:50:07.303 INFO:teuthology.orchestra.run.smithi112.stdout: "supported_features": [ 2023-05-31T19:50:07.304 INFO:teuthology.orchestra.run.smithi112.stdout: "resharding" 2023-05-31T19:50:07.304 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2023-05-31T19:50:07.304 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-05-31T19:50:07.304 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-05-31T19:50:07.304 INFO:teuthology.orchestra.run.smithi112.stdout: "id": "5d8acec7-7e57-4e56-8b32-e3d6b5036001", 2023-05-31T19:50:07.304 INFO:teuthology.orchestra.run.smithi112.stdout: "name": "a2", 2023-05-31T19:50:07.304 INFO:teuthology.orchestra.run.smithi112.stdout: "endpoints": [ 2023-05-31T19:50:07.305 INFO:teuthology.orchestra.run.smithi112.stdout: "http://smithi099.front.sepia.ceph.com:8001" 2023-05-31T19:50:07.305 INFO:teuthology.orchestra.run.smithi112.stdout: ], 2023-05-31T19:50:07.305 INFO:teuthology.orchestra.run.smithi112.stdout: "log_meta": false, 2023-05-31T19:50:07.305 INFO:teuthology.orchestra.run.smithi112.stdout: "log_data": true, 2023-05-31T19:50:07.305 INFO:teuthology.orchestra.run.smithi112.stdout: "bucket_index_max_shards": 11, 2023-05-31T19:50:07.305 INFO:teuthology.orchestra.run.smithi112.stdout: "read_only": false, 2023-05-31T19:50:07.305 INFO:teuthology.orchestra.run.smithi112.stdout: "tier_type": "", 2023-05-31T19:50:07.306 INFO:teuthology.orchestra.run.smithi112.stdout: "sync_from_all": true, 2023-05-31T19:50:07.306 INFO:teuthology.orchestra.run.smithi112.stdout: "sync_from": [], 2023-05-31T19:50:07.306 INFO:teuthology.orchestra.run.smithi112.stdout: "redirect_zone": "", 2023-05-31T19:50:07.306 INFO:teuthology.orchestra.run.smithi112.stdout: "supported_features": [ 2023-05-31T19:50:07.306 INFO:teuthology.orchestra.run.smithi112.stdout: "resharding" 2023-05-31T19:50:07.306 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2023-05-31T19:50:07.307 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-05-31T19:50:07.307 INFO:teuthology.orchestra.run.smithi112.stdout: ], 2023-05-31T19:50:07.307 INFO:teuthology.orchestra.run.smithi112.stdout: "placement_targets": [ 2023-05-31T19:50:07.307 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-05-31T19:50:07.308 INFO:teuthology.orchestra.run.smithi112.stdout: "name": "default-placement", 2023-05-31T19:50:07.308 INFO:teuthology.orchestra.run.smithi112.stdout: "tags": [], 2023-05-31T19:50:07.308 INFO:teuthology.orchestra.run.smithi112.stdout: "storage_classes": [ 2023-05-31T19:50:07.308 INFO:teuthology.orchestra.run.smithi112.stdout: "STANDARD" 2023-05-31T19:50:07.309 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2023-05-31T19:50:07.309 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-05-31T19:50:07.309 INFO:teuthology.orchestra.run.smithi112.stdout: ], 2023-05-31T19:50:07.309 INFO:teuthology.orchestra.run.smithi112.stdout: "default_placement": "default-placement", 2023-05-31T19:50:07.309 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_id": "aaa33de6-515b-4cac-8f1b-3a1a30452117", 2023-05-31T19:50:07.310 INFO:teuthology.orchestra.run.smithi112.stdout: "sync_policy": { 2023-05-31T19:50:07.310 INFO:teuthology.orchestra.run.smithi112.stdout: "groups": [] 2023-05-31T19:50:07.310 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-05-31T19:50:07.311 INFO:teuthology.orchestra.run.smithi112.stdout: "enabled_features": [ 2023-05-31T19:50:07.311 INFO:teuthology.orchestra.run.smithi112.stdout: "resharding" 2023-05-31T19:50:07.311 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2023-05-31T19:50:07.311 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-05-31T19:50:07.312 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-05-31T19:50:07.312 INFO:teuthology.orchestra.run.smithi112.stdout: "id": "b", 2023-05-31T19:50:07.312 INFO:teuthology.orchestra.run.smithi112.stdout: "name": "b", 2023-05-31T19:50:07.312 INFO:teuthology.orchestra.run.smithi112.stdout: "api_name": "", 2023-05-31T19:50:07.313 INFO:teuthology.orchestra.run.smithi112.stdout: "is_master": false, 2023-05-31T19:50:07.313 INFO:teuthology.orchestra.run.smithi112.stdout: "endpoints": [ 2023-05-31T19:50:07.313 INFO:teuthology.orchestra.run.smithi112.stdout: "http://smithi112.front.sepia.ceph.com:8000" 2023-05-31T19:50:07.313 INFO:teuthology.orchestra.run.smithi112.stdout: ], 2023-05-31T19:50:07.314 INFO:teuthology.orchestra.run.smithi112.stdout: "hostnames": [], 2023-05-31T19:50:07.314 INFO:teuthology.orchestra.run.smithi112.stdout: "hostnames_s3website": [], 2023-05-31T19:50:07.314 INFO:teuthology.orchestra.run.smithi112.stdout: "master_zone": "f85d7236-2f27-4988-af26-9c66ab280b1b", 2023-05-31T19:50:07.314 INFO:teuthology.orchestra.run.smithi112.stdout: "zones": [ 2023-05-31T19:50:07.315 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-05-31T19:50:07.315 INFO:teuthology.orchestra.run.smithi112.stdout: "id": "f85d7236-2f27-4988-af26-9c66ab280b1b", 2023-05-31T19:50:07.315 INFO:teuthology.orchestra.run.smithi112.stdout: "name": "b1", 2023-05-31T19:50:07.316 INFO:teuthology.orchestra.run.smithi112.stdout: "endpoints": [ 2023-05-31T19:50:07.316 INFO:teuthology.orchestra.run.smithi112.stdout: "http://smithi112.front.sepia.ceph.com:8000" 2023-05-31T19:50:07.316 INFO:teuthology.orchestra.run.smithi112.stdout: ], 2023-05-31T19:50:07.316 INFO:teuthology.orchestra.run.smithi112.stdout: "log_meta": false, 2023-05-31T19:50:07.317 INFO:teuthology.orchestra.run.smithi112.stdout: "log_data": false, 2023-05-31T19:50:07.317 INFO:teuthology.orchestra.run.smithi112.stdout: "bucket_index_max_shards": 11, 2023-05-31T19:50:07.317 INFO:teuthology.orchestra.run.smithi112.stdout: "read_only": false, 2023-05-31T19:50:07.317 INFO:teuthology.orchestra.run.smithi112.stdout: "tier_type": "", 2023-05-31T19:50:07.318 INFO:teuthology.orchestra.run.smithi112.stdout: "sync_from_all": true, 2023-05-31T19:50:07.318 INFO:teuthology.orchestra.run.smithi112.stdout: "sync_from": [], 2023-05-31T19:50:07.318 INFO:teuthology.orchestra.run.smithi112.stdout: "redirect_zone": "", 2023-05-31T19:50:07.318 INFO:teuthology.orchestra.run.smithi112.stdout: "supported_features": [ 2023-05-31T19:50:07.319 INFO:teuthology.orchestra.run.smithi112.stdout: "resharding" 2023-05-31T19:50:07.319 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2023-05-31T19:50:07.319 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-05-31T19:50:07.320 INFO:teuthology.orchestra.run.smithi112.stdout: ], 2023-05-31T19:50:07.320 INFO:teuthology.orchestra.run.smithi112.stdout: "placement_targets": [ 2023-05-31T19:50:07.321 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-05-31T19:50:07.322 INFO:teuthology.orchestra.run.smithi112.stdout: "name": "default-placement", 2023-05-31T19:50:07.322 INFO:teuthology.orchestra.run.smithi112.stdout: "tags": [], 2023-05-31T19:50:07.322 INFO:teuthology.orchestra.run.smithi112.stdout: "storage_classes": [ 2023-05-31T19:50:07.323 INFO:teuthology.orchestra.run.smithi112.stdout: "STANDARD" 2023-05-31T19:50:07.323 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2023-05-31T19:50:07.323 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-05-31T19:50:07.323 INFO:teuthology.orchestra.run.smithi112.stdout: ], 2023-05-31T19:50:07.324 INFO:teuthology.orchestra.run.smithi112.stdout: "default_placement": "default-placement", 2023-05-31T19:50:07.324 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_id": "aaa33de6-515b-4cac-8f1b-3a1a30452117", 2023-05-31T19:50:07.324 INFO:teuthology.orchestra.run.smithi112.stdout: "sync_policy": { 2023-05-31T19:50:07.325 INFO:teuthology.orchestra.run.smithi112.stdout: "groups": [] 2023-05-31T19:50:07.325 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-05-31T19:50:07.326 INFO:teuthology.orchestra.run.smithi112.stdout: "enabled_features": [ 2023-05-31T19:50:07.326 INFO:teuthology.orchestra.run.smithi112.stdout: "resharding" 2023-05-31T19:50:07.326 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2023-05-31T19:50:07.326 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-05-31T19:50:07.327 INFO:teuthology.orchestra.run.smithi112.stdout: ], 2023-05-31T19:50:07.327 INFO:teuthology.orchestra.run.smithi112.stdout: "short_zone_ids": [ 2023-05-31T19:50:07.327 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-05-31T19:50:07.328 INFO:teuthology.orchestra.run.smithi112.stdout: "key": "3d163b1e-09c7-47e2-ae87-add4aace3f38", 2023-05-31T19:50:07.328 INFO:teuthology.orchestra.run.smithi112.stdout: "val": 3101833771 2023-05-31T19:50:07.328 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-05-31T19:50:07.328 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-05-31T19:50:07.329 INFO:teuthology.orchestra.run.smithi112.stdout: "key": "5d8acec7-7e57-4e56-8b32-e3d6b5036001", 2023-05-31T19:50:07.329 INFO:teuthology.orchestra.run.smithi112.stdout: "val": 4255530623 2023-05-31T19:50:07.329 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-05-31T19:50:07.330 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-05-31T19:50:07.330 INFO:teuthology.orchestra.run.smithi112.stdout: "key": "f85d7236-2f27-4988-af26-9c66ab280b1b", 2023-05-31T19:50:07.330 INFO:teuthology.orchestra.run.smithi112.stdout: "val": 233395200 2023-05-31T19:50:07.331 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-05-31T19:50:07.331 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2023-05-31T19:50:07.331 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-05-31T19:50:07.331 INFO:teuthology.orchestra.run.smithi112.stdout: "master_zonegroup": "a", 2023-05-31T19:50:07.332 INFO:teuthology.orchestra.run.smithi112.stdout: "master_zone": "3d163b1e-09c7-47e2-ae87-add4aace3f38", 2023-05-31T19:50:07.333 INFO:teuthology.orchestra.run.smithi112.stdout: "period_config": { 2023-05-31T19:50:07.333 INFO:teuthology.orchestra.run.smithi112.stdout: "bucket_quota": { 2023-05-31T19:50:07.333 INFO:teuthology.orchestra.run.smithi112.stdout: "enabled": false, 2023-05-31T19:50:07.334 INFO:teuthology.orchestra.run.smithi112.stdout: "check_on_raw": false, 2023-05-31T19:50:07.334 INFO:teuthology.orchestra.run.smithi112.stdout: "max_size": -1, 2023-05-31T19:50:07.334 INFO:teuthology.orchestra.run.smithi112.stdout: "max_size_kb": 0, 2023-05-31T19:50:07.334 INFO:teuthology.orchestra.run.smithi112.stdout: "max_objects": -1 2023-05-31T19:50:07.334 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-05-31T19:50:07.335 INFO:teuthology.orchestra.run.smithi112.stdout: "user_quota": { 2023-05-31T19:50:07.335 INFO:teuthology.orchestra.run.smithi112.stdout: "enabled": false, 2023-05-31T19:50:07.335 INFO:teuthology.orchestra.run.smithi112.stdout: "check_on_raw": false, 2023-05-31T19:50:07.335 INFO:teuthology.orchestra.run.smithi112.stdout: "max_size": -1, 2023-05-31T19:50:07.336 INFO:teuthology.orchestra.run.smithi112.stdout: "max_size_kb": 0, 2023-05-31T19:50:07.336 INFO:teuthology.orchestra.run.smithi112.stdout: "max_objects": -1 2023-05-31T19:50:07.336 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-05-31T19:50:07.337 INFO:teuthology.orchestra.run.smithi112.stdout: "user_ratelimit": { 2023-05-31T19:50:07.337 INFO:teuthology.orchestra.run.smithi112.stdout: "max_read_ops": 0, 2023-05-31T19:50:07.337 INFO:teuthology.orchestra.run.smithi112.stdout: "max_write_ops": 0, 2023-05-31T19:50:07.337 INFO:teuthology.orchestra.run.smithi112.stdout: "max_read_bytes": 0, 2023-05-31T19:50:07.337 INFO:teuthology.orchestra.run.smithi112.stdout: "max_write_bytes": 0, 2023-05-31T19:50:07.337 INFO:teuthology.orchestra.run.smithi112.stdout: "enabled": false 2023-05-31T19:50:07.338 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-05-31T19:50:07.338 INFO:teuthology.orchestra.run.smithi112.stdout: "bucket_ratelimit": { 2023-05-31T19:50:07.338 INFO:teuthology.orchestra.run.smithi112.stdout: "max_read_ops": 0, 2023-05-31T19:50:07.338 INFO:teuthology.orchestra.run.smithi112.stdout: "max_write_ops": 0, 2023-05-31T19:50:07.338 INFO:teuthology.orchestra.run.smithi112.stdout: "max_read_bytes": 0, 2023-05-31T19:50:07.338 INFO:teuthology.orchestra.run.smithi112.stdout: "max_write_bytes": 0, 2023-05-31T19:50:07.338 INFO:teuthology.orchestra.run.smithi112.stdout: "enabled": false 2023-05-31T19:50:07.339 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-05-31T19:50:07.339 INFO:teuthology.orchestra.run.smithi112.stdout: "anonymous_ratelimit": { 2023-05-31T19:50:07.339 INFO:teuthology.orchestra.run.smithi112.stdout: "max_read_ops": 0, 2023-05-31T19:50:07.339 INFO:teuthology.orchestra.run.smithi112.stdout: "max_write_ops": 0, 2023-05-31T19:50:07.339 INFO:teuthology.orchestra.run.smithi112.stdout: "max_read_bytes": 0, 2023-05-31T19:50:07.339 INFO:teuthology.orchestra.run.smithi112.stdout: "max_write_bytes": 0, 2023-05-31T19:50:07.339 INFO:teuthology.orchestra.run.smithi112.stdout: "enabled": false 2023-05-31T19:50:07.339 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-05-31T19:50:07.340 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-05-31T19:50:07.340 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_id": "aaa33de6-515b-4cac-8f1b-3a1a30452117", 2023-05-31T19:50:07.340 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_name": "test-realm", 2023-05-31T19:50:07.340 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-05-31T19:50:07.340 INFO:teuthology.orchestra.run.smithi112.stdout:} 2023-05-31T19:50:07.341 DEBUG:tasks.rgw.c2.client.0:waiting for process to exit 2023-05-31T19:50:07.342 INFO:teuthology.orchestra.run:waiting for 300 2023-05-31T19:50:07.343 INFO:tasks.rgw.c2.client.0.smithi112.stdout:2023-05-31T19:50:07.336+0000 7f3b1d571700 -1 received signal: Terminated from /usr/bin/python3 /bin/daemon-helper term radosgw --rgw-frontends beast port=8000 -n client.0 --cluster c2 -k /etc/ceph/c2.client.0.keyring --log-file /var/log/ceph/rgw.c2.client.0.log --rgw_ops_log_socket_path /home/ubuntu/cephtest/rgw.opslog.c2.client.0.sock --foreground (PID: 110543) UID: 0 2023-05-31T19:50:07.343 INFO:tasks.rgw.c2.client.0.smithi112.stdout:2023-05-31T19:50:07.336+0000 7f3b217da7c0 -1 shutting down 2023-05-31T19:50:09.233 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.rgw.client.0 is failed for ~0s 2023-05-31T19:50:13.444 INFO:tasks.rgw.c2.client.0:Stopped 2023-05-31T19:50:13.445 INFO:tasks.rgw.c2.client.0:Restarting daemon 2023-05-31T19:50:13.445 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term radosgw --rgw-frontends 'beast port=8000' -n client.0 --cluster c2 -k /etc/ceph/c2.client.0.keyring --log-file /var/log/ceph/rgw.c2.client.0.log --rgw_ops_log_socket_path /home/ubuntu/cephtest/rgw.opslog.c2.client.0.sock --foreground --rgw-zone b1 --rgw-zonegroup b --rgw-realm test-realm | sudo tee /var/log/ceph/rgw.c2.client.0.stdout 2>&1 2023-05-31T19:50:13.448 INFO:tasks.rgw.c2.client.0:Started 2023-05-31T19:50:13.448 DEBUG:teuthology.orchestra.run.smithi112:> curl http://smithi112.front.sepia.ceph.com:8000 2023-05-31T19:50:13.474 DEBUG:teuthology.orchestra.run:got remote process result: 7 2023-05-31T19:50:13.475 INFO:teuthology.orchestra.run.smithi112.stderr: % Total % Received % Xferd Average Speed Time Time Time Current 2023-05-31T19:50:13.475 INFO:teuthology.orchestra.run.smithi112.stderr: Dload Upload Total Spent Left Speed 2023-05-31T19:50:13.475 INFO:teuthology.orchestra.run.smithi112.stderr: 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0curl: (7) Failed to connect to smithi112.front.sepia.ceph.com port 8000: Connection refused 2023-05-31T19:50:14.476 DEBUG:teuthology.orchestra.run.smithi112:> curl http://smithi112.front.sepia.ceph.com:8000 2023-05-31T19:50:14.499 INFO:teuthology.orchestra.run.smithi112.stderr: % Total % Received % Xferd Average Speed Time Time Time Current 2023-05-31T19:50:14.500 INFO:teuthology.orchestra.run.smithi112.stderr: Dload Upload Total Spent Left Speed 2023-05-31T19:50:14.500 INFO:teuthology.orchestra.run.smithi112.stderr: 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 100 214 0 214 0 0 208k 0 --:--:-- --:--:-- --:--:-- 208k 2023-05-31T19:50:14.501 INFO:teuthology.orchestra.run.smithi112.stdout:anonymous 2023-05-31T19:50:14.501 INFO:tasks.rgw_multisite:pulling realm configuration to c2 2023-05-31T19:50:14.502 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['realm', 'pull', '--rgw-realm', 'test-realm', '--url', 'http://smithi099.front.sepia.ceph.com:8000', '--access-key', 'RVN19ED3W7FH7E08', '--secret', 'XzQjGe60gbIUStGfs96TKhuPw3IQi3hK', '--url', 'http://smithi099.front.sepia.ceph.com:8000', '--access-key', 'RVN19ED3W7FH7E08', '--secret', 'XzQjGe60gbIUStGfs96TKhuPw3IQi3hK', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2023-05-31T19:50:14.502 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'realm', 'pull', '--rgw-realm', 'test-realm', '--url', 'http://smithi099.front.sepia.ceph.com:8000', '--access-key', 'RVN19ED3W7FH7E08', '--secret', 'XzQjGe60gbIUStGfs96TKhuPw3IQi3hK', '--url', 'http://smithi099.front.sepia.ceph.com:8000', '--access-key', 'RVN19ED3W7FH7E08', '--secret', 'XzQjGe60gbIUStGfs96TKhuPw3IQi3hK', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2023-05-31T19:50:14.502 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 realm pull --rgw-realm test-realm --url http://smithi099.front.sepia.ceph.com:8000 --access-key RVN19ED3W7FH7E08 --secret XzQjGe60gbIUStGfs96TKhuPw3IQi3hK --url http://smithi099.front.sepia.ceph.com:8000 --access-key RVN19ED3W7FH7E08 --secret XzQjGe60gbIUStGfs96TKhuPw3IQi3hK --cluster c2 --debug-rgw 0 --debug-ms 0 2023-05-31T19:50:14.599 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-05-31T19:50:14.599 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-05-31T19:50:14.637 INFO:teuthology.orchestra.run.smithi112.stdout:{ 2023-05-31T19:50:14.637 INFO:teuthology.orchestra.run.smithi112.stdout: "id": "aaa33de6-515b-4cac-8f1b-3a1a30452117", 2023-05-31T19:50:14.637 INFO:teuthology.orchestra.run.smithi112.stdout: "name": "test-realm", 2023-05-31T19:50:14.637 INFO:teuthology.orchestra.run.smithi112.stdout: "current_period": "2e6123d2-d695-4350-9a01-61a455bc782b", 2023-05-31T19:50:14.638 INFO:teuthology.orchestra.run.smithi112.stdout: "epoch": 2 2023-05-31T19:50:14.638 INFO:teuthology.orchestra.run.smithi112.stdout:} 2023-05-31T19:50:14.639 INFO:tasks.rgw_multisite:creating zone.. 2023-05-31T19:50:14.639 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['zone', 'create', '--rgw-zone', 'b2', '--rgw-zonegroup', 'b', '--rgw-realm', 'test-realm', '--access-key', 'RVN19ED3W7FH7E08', '--secret', 'XzQjGe60gbIUStGfs96TKhuPw3IQi3hK', '--endpoints', 'http://smithi112.front.sepia.ceph.com:8001', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2023-05-31T19:50:14.639 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'zone', 'create', '--rgw-zone', 'b2', '--rgw-zonegroup', 'b', '--rgw-realm', 'test-realm', '--access-key', 'RVN19ED3W7FH7E08', '--secret', 'XzQjGe60gbIUStGfs96TKhuPw3IQi3hK', '--endpoints', 'http://smithi112.front.sepia.ceph.com:8001', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2023-05-31T19:50:14.640 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 zone create --rgw-zone b2 --rgw-zonegroup b --rgw-realm test-realm --access-key RVN19ED3W7FH7E08 --secret XzQjGe60gbIUStGfs96TKhuPw3IQi3hK --endpoints http://smithi112.front.sepia.ceph.com:8001 --cluster c2 --debug-rgw 0 --debug-ms 0 2023-05-31T19:50:14.693 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-05-31T19:50:14.693 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-05-31T19:50:14.708 INFO:teuthology.orchestra.run.smithi112.stderr:NOTICE: set zone's realm_id=aaa33de6-515b-4cac-8f1b-3a1a30452117 2023-05-31T19:50:14.721 INFO:teuthology.orchestra.run.smithi112.stdout:{ 2023-05-31T19:50:14.722 INFO:teuthology.orchestra.run.smithi112.stdout: "id": "a6a3aab5-bc52-44a4-8543-5c5ea6056acd", 2023-05-31T19:50:14.722 INFO:teuthology.orchestra.run.smithi112.stdout: "name": "b2", 2023-05-31T19:50:14.723 INFO:teuthology.orchestra.run.smithi112.stdout: "domain_root": "b2.rgw.meta:root", 2023-05-31T19:50:14.723 INFO:teuthology.orchestra.run.smithi112.stdout: "control_pool": "b2.rgw.control", 2023-05-31T19:50:14.723 INFO:teuthology.orchestra.run.smithi112.stdout: "gc_pool": "b2.rgw.log:gc", 2023-05-31T19:50:14.724 INFO:teuthology.orchestra.run.smithi112.stdout: "lc_pool": "b2.rgw.log:lc", 2023-05-31T19:50:14.724 INFO:teuthology.orchestra.run.smithi112.stdout: "log_pool": "b2.rgw.log", 2023-05-31T19:50:14.724 INFO:teuthology.orchestra.run.smithi112.stdout: "intent_log_pool": "b2.rgw.log:intent", 2023-05-31T19:50:14.725 INFO:teuthology.orchestra.run.smithi112.stdout: "usage_log_pool": "b2.rgw.log:usage", 2023-05-31T19:50:14.725 INFO:teuthology.orchestra.run.smithi112.stdout: "roles_pool": "b2.rgw.meta:roles", 2023-05-31T19:50:14.725 INFO:teuthology.orchestra.run.smithi112.stdout: "reshard_pool": "b2.rgw.log:reshard", 2023-05-31T19:50:14.725 INFO:teuthology.orchestra.run.smithi112.stdout: "user_keys_pool": "b2.rgw.meta:users.keys", 2023-05-31T19:50:14.726 INFO:teuthology.orchestra.run.smithi112.stdout: "user_email_pool": "b2.rgw.meta:users.email", 2023-05-31T19:50:14.726 INFO:teuthology.orchestra.run.smithi112.stdout: "user_swift_pool": "b2.rgw.meta:users.swift", 2023-05-31T19:50:14.726 INFO:teuthology.orchestra.run.smithi112.stdout: "user_uid_pool": "b2.rgw.meta:users.uid", 2023-05-31T19:50:14.727 INFO:teuthology.orchestra.run.smithi112.stdout: "otp_pool": "b2.rgw.otp", 2023-05-31T19:50:14.727 INFO:teuthology.orchestra.run.smithi112.stdout: "system_key": { 2023-05-31T19:50:14.727 INFO:teuthology.orchestra.run.smithi112.stdout: "access_key": "RVN19ED3W7FH7E08", 2023-05-31T19:50:14.727 INFO:teuthology.orchestra.run.smithi112.stdout: "secret_key": "XzQjGe60gbIUStGfs96TKhuPw3IQi3hK" 2023-05-31T19:50:14.727 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-05-31T19:50:14.727 INFO:teuthology.orchestra.run.smithi112.stdout: "placement_pools": [ 2023-05-31T19:50:14.728 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-05-31T19:50:14.728 INFO:teuthology.orchestra.run.smithi112.stdout: "key": "default-placement", 2023-05-31T19:50:14.728 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-05-31T19:50:14.728 INFO:teuthology.orchestra.run.smithi112.stdout: "index_pool": "b2.rgw.buckets.index", 2023-05-31T19:50:14.729 INFO:teuthology.orchestra.run.smithi112.stdout: "storage_classes": { 2023-05-31T19:50:14.729 INFO:teuthology.orchestra.run.smithi112.stdout: "STANDARD": { 2023-05-31T19:50:14.729 INFO:teuthology.orchestra.run.smithi112.stdout: "data_pool": "b2.rgw.buckets.data" 2023-05-31T19:50:14.729 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-05-31T19:50:14.729 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-05-31T19:50:14.730 INFO:teuthology.orchestra.run.smithi112.stdout: "data_extra_pool": "b2.rgw.buckets.non-ec", 2023-05-31T19:50:14.730 INFO:teuthology.orchestra.run.smithi112.stdout: "index_type": 0, 2023-05-31T19:50:14.731 INFO:teuthology.orchestra.run.smithi112.stdout: "inline_data": true 2023-05-31T19:50:14.731 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-05-31T19:50:14.731 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-05-31T19:50:14.731 INFO:teuthology.orchestra.run.smithi112.stdout: ], 2023-05-31T19:50:14.732 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_id": "aaa33de6-515b-4cac-8f1b-3a1a30452117", 2023-05-31T19:50:14.732 INFO:teuthology.orchestra.run.smithi112.stdout: "notif_pool": "b2.rgw.log:notif" 2023-05-31T19:50:14.732 INFO:teuthology.orchestra.run.smithi112.stdout:} 2023-05-31T19:50:14.733 DEBUG:teuthology.orchestra.run.smithi112:> sudo ceph osd pool create b2.rgw.buckets.data 64 64 --cluster c2 2023-05-31T19:50:14.939 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.rgw.client.0 has been restored 2023-05-31T19:50:16.057 INFO:teuthology.orchestra.run.smithi112.stderr:pool 'b2.rgw.buckets.data' created 2023-05-31T19:50:16.084 DEBUG:teuthology.orchestra.run.smithi112:> sudo ceph osd pool application enable b2.rgw.buckets.data rgw --cluster c2 2023-05-31T19:50:17.077 INFO:teuthology.orchestra.run.smithi112.stderr:enabled application 'rgw' on pool 'b2.rgw.buckets.data' 2023-05-31T19:50:17.100 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['zone', 'placement', '--rgw-zone', 'b2', '--rgw-zonegroup', 'b', '--rgw-realm', 'test-realm', 'modify', '--placement-id', 'default-placement', '--compression', 'random', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2023-05-31T19:50:17.100 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'zone', 'placement', '--rgw-zone', 'b2', '--rgw-zonegroup', 'b', '--rgw-realm', 'test-realm', 'modify', '--placement-id', 'default-placement', '--compression', 'random', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2023-05-31T19:50:17.101 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 zone placement --rgw-zone b2 --rgw-zonegroup b --rgw-realm test-realm modify --placement-id default-placement --compression random --cluster c2 --debug-rgw 0 --debug-ms 0 2023-05-31T19:50:17.155 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-05-31T19:50:17.155 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-05-31T19:50:17.173 INFO:teuthology.orchestra.run.smithi112.stdout:{ 2023-05-31T19:50:17.175 INFO:teuthology.orchestra.run.smithi112.stdout: "id": "a6a3aab5-bc52-44a4-8543-5c5ea6056acd", 2023-05-31T19:50:17.176 INFO:teuthology.orchestra.run.smithi112.stdout: "name": "b2", 2023-05-31T19:50:17.176 INFO:teuthology.orchestra.run.smithi112.stdout: "domain_root": "b2.rgw.meta:root", 2023-05-31T19:50:17.176 INFO:teuthology.orchestra.run.smithi112.stdout: "control_pool": "b2.rgw.control", 2023-05-31T19:50:17.176 INFO:teuthology.orchestra.run.smithi112.stdout: "gc_pool": "b2.rgw.log:gc", 2023-05-31T19:50:17.177 INFO:teuthology.orchestra.run.smithi112.stdout: "lc_pool": "b2.rgw.log:lc", 2023-05-31T19:50:17.177 INFO:teuthology.orchestra.run.smithi112.stdout: "log_pool": "b2.rgw.log", 2023-05-31T19:50:17.178 INFO:teuthology.orchestra.run.smithi112.stdout: "intent_log_pool": "b2.rgw.log:intent", 2023-05-31T19:50:17.178 INFO:teuthology.orchestra.run.smithi112.stdout: "usage_log_pool": "b2.rgw.log:usage", 2023-05-31T19:50:17.179 INFO:teuthology.orchestra.run.smithi112.stdout: "roles_pool": "b2.rgw.meta:roles", 2023-05-31T19:50:17.179 INFO:teuthology.orchestra.run.smithi112.stdout: "reshard_pool": "b2.rgw.log:reshard", 2023-05-31T19:50:17.180 INFO:teuthology.orchestra.run.smithi112.stdout: "user_keys_pool": "b2.rgw.meta:users.keys", 2023-05-31T19:50:17.180 INFO:teuthology.orchestra.run.smithi112.stdout: "user_email_pool": "b2.rgw.meta:users.email", 2023-05-31T19:50:17.180 INFO:teuthology.orchestra.run.smithi112.stdout: "user_swift_pool": "b2.rgw.meta:users.swift", 2023-05-31T19:50:17.180 INFO:teuthology.orchestra.run.smithi112.stdout: "user_uid_pool": "b2.rgw.meta:users.uid", 2023-05-31T19:50:17.181 INFO:teuthology.orchestra.run.smithi112.stdout: "otp_pool": "b2.rgw.otp", 2023-05-31T19:50:17.181 INFO:teuthology.orchestra.run.smithi112.stdout: "system_key": { 2023-05-31T19:50:17.181 INFO:teuthology.orchestra.run.smithi112.stdout: "access_key": "RVN19ED3W7FH7E08", 2023-05-31T19:50:17.183 INFO:teuthology.orchestra.run.smithi112.stdout: "secret_key": "XzQjGe60gbIUStGfs96TKhuPw3IQi3hK" 2023-05-31T19:50:17.183 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-05-31T19:50:17.183 INFO:teuthology.orchestra.run.smithi112.stdout: "placement_pools": [ 2023-05-31T19:50:17.183 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-05-31T19:50:17.183 INFO:teuthology.orchestra.run.smithi112.stdout: "key": "default-placement", 2023-05-31T19:50:17.184 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-05-31T19:50:17.184 INFO:teuthology.orchestra.run.smithi112.stdout: "index_pool": "b2.rgw.buckets.index", 2023-05-31T19:50:17.184 INFO:teuthology.orchestra.run.smithi112.stdout: "storage_classes": { 2023-05-31T19:50:17.184 INFO:teuthology.orchestra.run.smithi112.stdout: "STANDARD": { 2023-05-31T19:50:17.185 INFO:teuthology.orchestra.run.smithi112.stdout: "data_pool": "b2.rgw.buckets.data", 2023-05-31T19:50:17.185 INFO:teuthology.orchestra.run.smithi112.stdout: "compression_type": "random" 2023-05-31T19:50:17.185 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-05-31T19:50:17.186 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-05-31T19:50:17.186 INFO:teuthology.orchestra.run.smithi112.stdout: "data_extra_pool": "b2.rgw.buckets.non-ec", 2023-05-31T19:50:17.186 INFO:teuthology.orchestra.run.smithi112.stdout: "index_type": 0, 2023-05-31T19:50:17.187 INFO:teuthology.orchestra.run.smithi112.stdout: "inline_data": true 2023-05-31T19:50:17.188 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-05-31T19:50:17.189 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-05-31T19:50:17.189 INFO:teuthology.orchestra.run.smithi112.stdout: ], 2023-05-31T19:50:17.189 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_id": "aaa33de6-515b-4cac-8f1b-3a1a30452117", 2023-05-31T19:50:17.189 INFO:teuthology.orchestra.run.smithi112.stdout: "notif_pool": "b2.rgw.log:notif" 2023-05-31T19:50:17.189 INFO:teuthology.orchestra.run.smithi112.stdout:} 2023-05-31T19:50:17.191 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['period', 'update', '--rgw-zone', 'b2', '--rgw-zonegroup', 'b', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2023-05-31T19:50:17.191 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'period', 'update', '--rgw-zone', 'b2', '--rgw-zonegroup', 'b', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2023-05-31T19:50:17.191 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 period update --rgw-zone b2 --rgw-zonegroup b --rgw-realm test-realm --commit --cluster c2 --debug-rgw 0 --debug-ms 0 2023-05-31T19:50:17.240 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-05-31T19:50:17.241 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-05-31T19:50:17.252 INFO:teuthology.orchestra.run.smithi112.stderr:2023-05-31T19:50:17.247+0000 7fd16eeaf780 0 period (2e6123d2-d695-4350-9a01-61a455bc782b does not have zone a6a3aab5-bc52-44a4-8543-5c5ea6056acd configured 2023-05-31T19:50:24.129 INFO:teuthology.orchestra.run.smithi112.stderr:Sending period to new master zone 3d163b1e-09c7-47e2-ae87-add4aace3f38 2023-05-31T19:50:24.167 INFO:teuthology.orchestra.run.smithi112.stdout:{ 2023-05-31T19:50:24.167 INFO:teuthology.orchestra.run.smithi112.stdout: "id": "2e6123d2-d695-4350-9a01-61a455bc782b", 2023-05-31T19:50:24.168 INFO:teuthology.orchestra.run.smithi112.stdout: "epoch": 4, 2023-05-31T19:50:24.168 INFO:teuthology.orchestra.run.smithi112.stdout: "predecessor_uuid": "629b9ced-80af-41ed-8a6c-f2cfc334cbb4", 2023-05-31T19:50:24.168 INFO:teuthology.orchestra.run.smithi112.stdout: "sync_status": [], 2023-05-31T19:50:24.169 INFO:teuthology.orchestra.run.smithi112.stdout: "period_map": { 2023-05-31T19:50:24.170 INFO:teuthology.orchestra.run.smithi112.stdout: "id": "2e6123d2-d695-4350-9a01-61a455bc782b", 2023-05-31T19:50:24.170 INFO:teuthology.orchestra.run.smithi112.stdout: "zonegroups": [ 2023-05-31T19:50:24.170 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-05-31T19:50:24.170 INFO:teuthology.orchestra.run.smithi112.stdout: "id": "a", 2023-05-31T19:50:24.170 INFO:teuthology.orchestra.run.smithi112.stdout: "name": "a", 2023-05-31T19:50:24.170 INFO:teuthology.orchestra.run.smithi112.stdout: "api_name": "", 2023-05-31T19:50:24.171 INFO:teuthology.orchestra.run.smithi112.stdout: "is_master": true, 2023-05-31T19:50:24.171 INFO:teuthology.orchestra.run.smithi112.stdout: "endpoints": [ 2023-05-31T19:50:24.171 INFO:teuthology.orchestra.run.smithi112.stdout: "http://smithi099.front.sepia.ceph.com:8000" 2023-05-31T19:50:24.171 INFO:teuthology.orchestra.run.smithi112.stdout: ], 2023-05-31T19:50:24.171 INFO:teuthology.orchestra.run.smithi112.stdout: "hostnames": [], 2023-05-31T19:50:24.171 INFO:teuthology.orchestra.run.smithi112.stdout: "hostnames_s3website": [], 2023-05-31T19:50:24.171 INFO:teuthology.orchestra.run.smithi112.stdout: "master_zone": "3d163b1e-09c7-47e2-ae87-add4aace3f38", 2023-05-31T19:50:24.172 INFO:teuthology.orchestra.run.smithi112.stdout: "zones": [ 2023-05-31T19:50:24.172 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-05-31T19:50:24.172 INFO:teuthology.orchestra.run.smithi112.stdout: "id": "3d163b1e-09c7-47e2-ae87-add4aace3f38", 2023-05-31T19:50:24.172 INFO:teuthology.orchestra.run.smithi112.stdout: "name": "a1", 2023-05-31T19:50:24.172 INFO:teuthology.orchestra.run.smithi112.stdout: "endpoints": [ 2023-05-31T19:50:24.172 INFO:teuthology.orchestra.run.smithi112.stdout: "http://smithi099.front.sepia.ceph.com:8000" 2023-05-31T19:50:24.172 INFO:teuthology.orchestra.run.smithi112.stdout: ], 2023-05-31T19:50:24.173 INFO:teuthology.orchestra.run.smithi112.stdout: "log_meta": false, 2023-05-31T19:50:24.173 INFO:teuthology.orchestra.run.smithi112.stdout: "log_data": true, 2023-05-31T19:50:24.173 INFO:teuthology.orchestra.run.smithi112.stdout: "bucket_index_max_shards": 11, 2023-05-31T19:50:24.173 INFO:teuthology.orchestra.run.smithi112.stdout: "read_only": false, 2023-05-31T19:50:24.173 INFO:teuthology.orchestra.run.smithi112.stdout: "tier_type": "", 2023-05-31T19:50:24.173 INFO:teuthology.orchestra.run.smithi112.stdout: "sync_from_all": true, 2023-05-31T19:50:24.173 INFO:teuthology.orchestra.run.smithi112.stdout: "sync_from": [], 2023-05-31T19:50:24.173 INFO:teuthology.orchestra.run.smithi112.stdout: "redirect_zone": "", 2023-05-31T19:50:24.174 INFO:teuthology.orchestra.run.smithi112.stdout: "supported_features": [ 2023-05-31T19:50:24.174 INFO:teuthology.orchestra.run.smithi112.stdout: "resharding" 2023-05-31T19:50:24.174 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2023-05-31T19:50:24.174 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-05-31T19:50:24.174 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-05-31T19:50:24.174 INFO:teuthology.orchestra.run.smithi112.stdout: "id": "5d8acec7-7e57-4e56-8b32-e3d6b5036001", 2023-05-31T19:50:24.174 INFO:teuthology.orchestra.run.smithi112.stdout: "name": "a2", 2023-05-31T19:50:24.175 INFO:teuthology.orchestra.run.smithi112.stdout: "endpoints": [ 2023-05-31T19:50:24.175 INFO:teuthology.orchestra.run.smithi112.stdout: "http://smithi099.front.sepia.ceph.com:8001" 2023-05-31T19:50:24.175 INFO:teuthology.orchestra.run.smithi112.stdout: ], 2023-05-31T19:50:24.175 INFO:teuthology.orchestra.run.smithi112.stdout: "log_meta": false, 2023-05-31T19:50:24.175 INFO:teuthology.orchestra.run.smithi112.stdout: "log_data": true, 2023-05-31T19:50:24.175 INFO:teuthology.orchestra.run.smithi112.stdout: "bucket_index_max_shards": 11, 2023-05-31T19:50:24.175 INFO:teuthology.orchestra.run.smithi112.stdout: "read_only": false, 2023-05-31T19:50:24.175 INFO:teuthology.orchestra.run.smithi112.stdout: "tier_type": "", 2023-05-31T19:50:24.176 INFO:teuthology.orchestra.run.smithi112.stdout: "sync_from_all": true, 2023-05-31T19:50:24.176 INFO:teuthology.orchestra.run.smithi112.stdout: "sync_from": [], 2023-05-31T19:50:24.176 INFO:teuthology.orchestra.run.smithi112.stdout: "redirect_zone": "", 2023-05-31T19:50:24.176 INFO:teuthology.orchestra.run.smithi112.stdout: "supported_features": [ 2023-05-31T19:50:24.176 INFO:teuthology.orchestra.run.smithi112.stdout: "resharding" 2023-05-31T19:50:24.176 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2023-05-31T19:50:24.176 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-05-31T19:50:24.177 INFO:teuthology.orchestra.run.smithi112.stdout: ], 2023-05-31T19:50:24.177 INFO:teuthology.orchestra.run.smithi112.stdout: "placement_targets": [ 2023-05-31T19:50:24.177 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-05-31T19:50:24.177 INFO:teuthology.orchestra.run.smithi112.stdout: "name": "default-placement", 2023-05-31T19:50:24.177 INFO:teuthology.orchestra.run.smithi112.stdout: "tags": [], 2023-05-31T19:50:24.177 INFO:teuthology.orchestra.run.smithi112.stdout: "storage_classes": [ 2023-05-31T19:50:24.177 INFO:teuthology.orchestra.run.smithi112.stdout: "STANDARD" 2023-05-31T19:50:24.177 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2023-05-31T19:50:24.178 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-05-31T19:50:24.178 INFO:teuthology.orchestra.run.smithi112.stdout: ], 2023-05-31T19:50:24.178 INFO:teuthology.orchestra.run.smithi112.stdout: "default_placement": "default-placement", 2023-05-31T19:50:24.178 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_id": "aaa33de6-515b-4cac-8f1b-3a1a30452117", 2023-05-31T19:50:24.178 INFO:teuthology.orchestra.run.smithi112.stdout: "sync_policy": { 2023-05-31T19:50:24.178 INFO:teuthology.orchestra.run.smithi112.stdout: "groups": [] 2023-05-31T19:50:24.178 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-05-31T19:50:24.179 INFO:teuthology.orchestra.run.smithi112.stdout: "enabled_features": [ 2023-05-31T19:50:24.179 INFO:teuthology.orchestra.run.smithi112.stdout: "resharding" 2023-05-31T19:50:24.179 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2023-05-31T19:50:24.179 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-05-31T19:50:24.179 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-05-31T19:50:24.179 INFO:teuthology.orchestra.run.smithi112.stdout: "id": "b", 2023-05-31T19:50:24.179 INFO:teuthology.orchestra.run.smithi112.stdout: "name": "b", 2023-05-31T19:50:24.179 INFO:teuthology.orchestra.run.smithi112.stdout: "api_name": "", 2023-05-31T19:50:24.180 INFO:teuthology.orchestra.run.smithi112.stdout: "is_master": false, 2023-05-31T19:50:24.180 INFO:teuthology.orchestra.run.smithi112.stdout: "endpoints": [ 2023-05-31T19:50:24.180 INFO:teuthology.orchestra.run.smithi112.stdout: "http://smithi112.front.sepia.ceph.com:8000" 2023-05-31T19:50:24.180 INFO:teuthology.orchestra.run.smithi112.stdout: ], 2023-05-31T19:50:24.180 INFO:teuthology.orchestra.run.smithi112.stdout: "hostnames": [], 2023-05-31T19:50:24.180 INFO:teuthology.orchestra.run.smithi112.stdout: "hostnames_s3website": [], 2023-05-31T19:50:24.180 INFO:teuthology.orchestra.run.smithi112.stdout: "master_zone": "f85d7236-2f27-4988-af26-9c66ab280b1b", 2023-05-31T19:50:24.181 INFO:teuthology.orchestra.run.smithi112.stdout: "zones": [ 2023-05-31T19:50:24.181 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-05-31T19:50:24.181 INFO:teuthology.orchestra.run.smithi112.stdout: "id": "a6a3aab5-bc52-44a4-8543-5c5ea6056acd", 2023-05-31T19:50:24.181 INFO:teuthology.orchestra.run.smithi112.stdout: "name": "b2", 2023-05-31T19:50:24.181 INFO:teuthology.orchestra.run.smithi112.stdout: "endpoints": [ 2023-05-31T19:50:24.181 INFO:teuthology.orchestra.run.smithi112.stdout: "http://smithi112.front.sepia.ceph.com:8001" 2023-05-31T19:50:24.181 INFO:teuthology.orchestra.run.smithi112.stdout: ], 2023-05-31T19:50:24.181 INFO:teuthology.orchestra.run.smithi112.stdout: "log_meta": false, 2023-05-31T19:50:24.182 INFO:teuthology.orchestra.run.smithi112.stdout: "log_data": true, 2023-05-31T19:50:24.182 INFO:teuthology.orchestra.run.smithi112.stdout: "bucket_index_max_shards": 11, 2023-05-31T19:50:24.182 INFO:teuthology.orchestra.run.smithi112.stdout: "read_only": false, 2023-05-31T19:50:24.182 INFO:teuthology.orchestra.run.smithi112.stdout: "tier_type": "", 2023-05-31T19:50:24.182 INFO:teuthology.orchestra.run.smithi112.stdout: "sync_from_all": true, 2023-05-31T19:50:24.182 INFO:teuthology.orchestra.run.smithi112.stdout: "sync_from": [], 2023-05-31T19:50:24.182 INFO:teuthology.orchestra.run.smithi112.stdout: "redirect_zone": "", 2023-05-31T19:50:24.182 INFO:teuthology.orchestra.run.smithi112.stdout: "supported_features": [ 2023-05-31T19:50:24.183 INFO:teuthology.orchestra.run.smithi112.stdout: "resharding" 2023-05-31T19:50:24.183 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2023-05-31T19:50:24.183 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-05-31T19:50:24.183 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-05-31T19:50:24.183 INFO:teuthology.orchestra.run.smithi112.stdout: "id": "f85d7236-2f27-4988-af26-9c66ab280b1b", 2023-05-31T19:50:24.184 INFO:teuthology.orchestra.run.smithi112.stdout: "name": "b1", 2023-05-31T19:50:24.184 INFO:teuthology.orchestra.run.smithi112.stdout: "endpoints": [ 2023-05-31T19:50:24.184 INFO:teuthology.orchestra.run.smithi112.stdout: "http://smithi112.front.sepia.ceph.com:8000" 2023-05-31T19:50:24.184 INFO:teuthology.orchestra.run.smithi112.stdout: ], 2023-05-31T19:50:24.184 INFO:teuthology.orchestra.run.smithi112.stdout: "log_meta": false, 2023-05-31T19:50:24.185 INFO:teuthology.orchestra.run.smithi112.stdout: "log_data": true, 2023-05-31T19:50:24.185 INFO:teuthology.orchestra.run.smithi112.stdout: "bucket_index_max_shards": 11, 2023-05-31T19:50:24.185 INFO:teuthology.orchestra.run.smithi112.stdout: "read_only": false, 2023-05-31T19:50:24.185 INFO:teuthology.orchestra.run.smithi112.stdout: "tier_type": "", 2023-05-31T19:50:24.185 INFO:teuthology.orchestra.run.smithi112.stdout: "sync_from_all": true, 2023-05-31T19:50:24.185 INFO:teuthology.orchestra.run.smithi112.stdout: "sync_from": [], 2023-05-31T19:50:24.185 INFO:teuthology.orchestra.run.smithi112.stdout: "redirect_zone": "", 2023-05-31T19:50:24.185 INFO:teuthology.orchestra.run.smithi112.stdout: "supported_features": [ 2023-05-31T19:50:24.186 INFO:teuthology.orchestra.run.smithi112.stdout: "resharding" 2023-05-31T19:50:24.186 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2023-05-31T19:50:24.186 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-05-31T19:50:24.186 INFO:teuthology.orchestra.run.smithi112.stdout: ], 2023-05-31T19:50:24.186 INFO:teuthology.orchestra.run.smithi112.stdout: "placement_targets": [ 2023-05-31T19:50:24.186 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-05-31T19:50:24.186 INFO:teuthology.orchestra.run.smithi112.stdout: "name": "default-placement", 2023-05-31T19:50:24.187 INFO:teuthology.orchestra.run.smithi112.stdout: "tags": [], 2023-05-31T19:50:24.187 INFO:teuthology.orchestra.run.smithi112.stdout: "storage_classes": [ 2023-05-31T19:50:24.187 INFO:teuthology.orchestra.run.smithi112.stdout: "STANDARD" 2023-05-31T19:50:24.187 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2023-05-31T19:50:24.187 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-05-31T19:50:24.187 INFO:teuthology.orchestra.run.smithi112.stdout: ], 2023-05-31T19:50:24.187 INFO:teuthology.orchestra.run.smithi112.stdout: "default_placement": "default-placement", 2023-05-31T19:50:24.187 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_id": "aaa33de6-515b-4cac-8f1b-3a1a30452117", 2023-05-31T19:50:24.188 INFO:teuthology.orchestra.run.smithi112.stdout: "sync_policy": { 2023-05-31T19:50:24.188 INFO:teuthology.orchestra.run.smithi112.stdout: "groups": [] 2023-05-31T19:50:24.188 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-05-31T19:50:24.188 INFO:teuthology.orchestra.run.smithi112.stdout: "enabled_features": [ 2023-05-31T19:50:24.188 INFO:teuthology.orchestra.run.smithi112.stdout: "resharding" 2023-05-31T19:50:24.188 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2023-05-31T19:50:24.188 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-05-31T19:50:24.189 INFO:teuthology.orchestra.run.smithi112.stdout: ], 2023-05-31T19:50:24.189 INFO:teuthology.orchestra.run.smithi112.stdout: "short_zone_ids": [ 2023-05-31T19:50:24.189 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-05-31T19:50:24.189 INFO:teuthology.orchestra.run.smithi112.stdout: "key": "3d163b1e-09c7-47e2-ae87-add4aace3f38", 2023-05-31T19:50:24.189 INFO:teuthology.orchestra.run.smithi112.stdout: "val": 3101833771 2023-05-31T19:50:24.189 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-05-31T19:50:24.189 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-05-31T19:50:24.189 INFO:teuthology.orchestra.run.smithi112.stdout: "key": "5d8acec7-7e57-4e56-8b32-e3d6b5036001", 2023-05-31T19:50:24.190 INFO:teuthology.orchestra.run.smithi112.stdout: "val": 4255530623 2023-05-31T19:50:24.190 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-05-31T19:50:24.190 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-05-31T19:50:24.190 INFO:teuthology.orchestra.run.smithi112.stdout: "key": "a6a3aab5-bc52-44a4-8543-5c5ea6056acd", 2023-05-31T19:50:24.190 INFO:teuthology.orchestra.run.smithi112.stdout: "val": 3708073710 2023-05-31T19:50:24.190 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-05-31T19:50:24.190 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-05-31T19:50:24.190 INFO:teuthology.orchestra.run.smithi112.stdout: "key": "f85d7236-2f27-4988-af26-9c66ab280b1b", 2023-05-31T19:50:24.191 INFO:teuthology.orchestra.run.smithi112.stdout: "val": 233395200 2023-05-31T19:50:24.191 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-05-31T19:50:24.191 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2023-05-31T19:50:24.191 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-05-31T19:50:24.191 INFO:teuthology.orchestra.run.smithi112.stdout: "master_zonegroup": "a", 2023-05-31T19:50:24.191 INFO:teuthology.orchestra.run.smithi112.stdout: "master_zone": "3d163b1e-09c7-47e2-ae87-add4aace3f38", 2023-05-31T19:50:24.191 INFO:teuthology.orchestra.run.smithi112.stdout: "period_config": { 2023-05-31T19:50:24.192 INFO:teuthology.orchestra.run.smithi112.stdout: "bucket_quota": { 2023-05-31T19:50:24.192 INFO:teuthology.orchestra.run.smithi112.stdout: "enabled": false, 2023-05-31T19:50:24.192 INFO:teuthology.orchestra.run.smithi112.stdout: "check_on_raw": false, 2023-05-31T19:50:24.192 INFO:teuthology.orchestra.run.smithi112.stdout: "max_size": -1, 2023-05-31T19:50:24.192 INFO:teuthology.orchestra.run.smithi112.stdout: "max_size_kb": 0, 2023-05-31T19:50:24.192 INFO:teuthology.orchestra.run.smithi112.stdout: "max_objects": -1 2023-05-31T19:50:24.192 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-05-31T19:50:24.192 INFO:teuthology.orchestra.run.smithi112.stdout: "user_quota": { 2023-05-31T19:50:24.193 INFO:teuthology.orchestra.run.smithi112.stdout: "enabled": false, 2023-05-31T19:50:24.193 INFO:teuthology.orchestra.run.smithi112.stdout: "check_on_raw": false, 2023-05-31T19:50:24.193 INFO:teuthology.orchestra.run.smithi112.stdout: "max_size": -1, 2023-05-31T19:50:24.193 INFO:teuthology.orchestra.run.smithi112.stdout: "max_size_kb": 0, 2023-05-31T19:50:24.193 INFO:teuthology.orchestra.run.smithi112.stdout: "max_objects": -1 2023-05-31T19:50:24.193 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-05-31T19:50:24.193 INFO:teuthology.orchestra.run.smithi112.stdout: "user_ratelimit": { 2023-05-31T19:50:24.194 INFO:teuthology.orchestra.run.smithi112.stdout: "max_read_ops": 0, 2023-05-31T19:50:24.194 INFO:teuthology.orchestra.run.smithi112.stdout: "max_write_ops": 0, 2023-05-31T19:50:24.194 INFO:teuthology.orchestra.run.smithi112.stdout: "max_read_bytes": 0, 2023-05-31T19:50:24.194 INFO:teuthology.orchestra.run.smithi112.stdout: "max_write_bytes": 0, 2023-05-31T19:50:24.194 INFO:teuthology.orchestra.run.smithi112.stdout: "enabled": false 2023-05-31T19:50:24.194 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-05-31T19:50:24.194 INFO:teuthology.orchestra.run.smithi112.stdout: "bucket_ratelimit": { 2023-05-31T19:50:24.194 INFO:teuthology.orchestra.run.smithi112.stdout: "max_read_ops": 0, 2023-05-31T19:50:24.195 INFO:teuthology.orchestra.run.smithi112.stdout: "max_write_ops": 0, 2023-05-31T19:50:24.195 INFO:teuthology.orchestra.run.smithi112.stdout: "max_read_bytes": 0, 2023-05-31T19:50:24.195 INFO:teuthology.orchestra.run.smithi112.stdout: "max_write_bytes": 0, 2023-05-31T19:50:24.195 INFO:teuthology.orchestra.run.smithi112.stdout: "enabled": false 2023-05-31T19:50:24.195 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-05-31T19:50:24.195 INFO:teuthology.orchestra.run.smithi112.stdout: "anonymous_ratelimit": { 2023-05-31T19:50:24.195 INFO:teuthology.orchestra.run.smithi112.stdout: "max_read_ops": 0, 2023-05-31T19:50:24.195 INFO:teuthology.orchestra.run.smithi112.stdout: "max_write_ops": 0, 2023-05-31T19:50:24.196 INFO:teuthology.orchestra.run.smithi112.stdout: "max_read_bytes": 0, 2023-05-31T19:50:24.196 INFO:teuthology.orchestra.run.smithi112.stdout: "max_write_bytes": 0, 2023-05-31T19:50:24.196 INFO:teuthology.orchestra.run.smithi112.stdout: "enabled": false 2023-05-31T19:50:24.196 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-05-31T19:50:24.196 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-05-31T19:50:24.197 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_id": "aaa33de6-515b-4cac-8f1b-3a1a30452117", 2023-05-31T19:50:24.197 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_name": "test-realm", 2023-05-31T19:50:24.197 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-05-31T19:50:24.197 INFO:teuthology.orchestra.run.smithi112.stdout:} 2023-05-31T19:50:24.198 DEBUG:tasks.rgw.c2.client.1:waiting for process to exit 2023-05-31T19:50:24.198 INFO:teuthology.orchestra.run:waiting for 300 2023-05-31T19:50:24.199 INFO:tasks.rgw.c2.client.1.smithi112.stdout:2023-05-31T19:50:24.193+0000 7f5127aa5700 -1 received signal: Terminated from /usr/bin/python3 /bin/daemon-helper term radosgw --rgw-frontends beast port=8001 -n client.1 --cluster c2 -k /etc/ceph/c2.client.1.keyring --log-file /var/log/ceph/rgw.c2.client.1.log --rgw_ops_log_socket_path /home/ubuntu/cephtest/rgw.opslog.c2.client.1.sock --foreground (PID: 110546) UID: 0 2023-05-31T19:50:24.199 INFO:tasks.rgw.c2.client.1.smithi112.stdout:2023-05-31T19:50:24.193+0000 7f512bd0e7c0 -1 shutting down 2023-05-31T19:50:26.257 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.rgw.client.1 is failed for ~0s 2023-05-31T19:50:30.300 INFO:tasks.rgw.c2.client.1:Stopped 2023-05-31T19:50:30.301 INFO:tasks.rgw.c2.client.1:Restarting daemon 2023-05-31T19:50:30.301 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term radosgw --rgw-frontends 'beast port=8001' -n client.1 --cluster c2 -k /etc/ceph/c2.client.1.keyring --log-file /var/log/ceph/rgw.c2.client.1.log --rgw_ops_log_socket_path /home/ubuntu/cephtest/rgw.opslog.c2.client.1.sock --foreground --rgw-zone b2 --rgw-zonegroup b --rgw-realm test-realm | sudo tee /var/log/ceph/rgw.c2.client.1.stdout 2>&1 2023-05-31T19:50:30.304 INFO:tasks.rgw.c2.client.1:Started 2023-05-31T19:50:30.305 DEBUG:teuthology.orchestra.run.smithi112:> curl http://smithi112.front.sepia.ceph.com:8001 2023-05-31T19:50:30.330 INFO:teuthology.orchestra.run.smithi112.stderr: % Total % Received % Xferd Average Speed Time Time Time Current 2023-05-31T19:50:30.331 INFO:teuthology.orchestra.run.smithi112.stderr: Dload Upload Total Spent Left Speed 2023-05-31T19:50:30.332 DEBUG:teuthology.orchestra.run:got remote process result: 7 2023-05-31T19:50:30.332 INFO:teuthology.orchestra.run.smithi112.stderr: 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0curl: (7) Failed to connect to smithi112.front.sepia.ceph.com port 8001: Connection refused 2023-05-31T19:50:31.333 DEBUG:teuthology.orchestra.run.smithi112:> curl http://smithi112.front.sepia.ceph.com:8001 2023-05-31T19:50:31.357 INFO:teuthology.orchestra.run.smithi112.stderr: % Total % Received % Xferd Average Speed Time Time Time Current 2023-05-31T19:50:31.357 INFO:teuthology.orchestra.run.smithi112.stderr: Dload Upload Total Spent Left Speed 2023-05-31T19:50:31.357 INFO:teuthology.orchestra.run.smithi112.stderr: 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 100 214 0 214 0 0 208k 0 --:--:-- --:--:-- --:--:-- 208k 2023-05-31T19:50:31.358 INFO:teuthology.orchestra.run.smithi112.stdout:anonymous 2023-05-31T19:50:31.358 INFO:tasks.rgw_multisite:rgw multisite configuration completed 2023-05-31T19:50:31.359 INFO:teuthology.run_tasks:Running task rgw-multisite-tests... 2023-05-31T19:50:31.371 INFO:tasks.rgw_multisite_tests:importing tests from /home/teuthworker/src/github.com_ceph_ceph-c_2870bf456d3115490c153abce9d47ac5c043b65e/qa/../src/test/rgw/rgw_multi 2023-05-31T19:50:31.405 INFO:tasks.rgw_multisite_tests:creating test user.. 2023-05-31T19:50:31.406 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['user', 'create', '--uid', 'rgw-multisite-test-user', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--display-name', 'Multisite Test User', '--gen-access-key', '--gen-secret', '--caps', 'roles=*', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-05-31T19:50:31.407 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'user', 'create', '--uid', 'rgw-multisite-test-user', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--display-name', 'Multisite Test User', '--gen-access-key', '--gen-secret', '--caps', 'roles=*', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-05-31T19:50:31.407 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 user create --uid rgw-multisite-test-user --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --display-name 'Multisite Test User' --gen-access-key --gen-secret --caps 'roles=*' --cluster c1 --debug-rgw 0 --debug-ms 0 2023-05-31T19:50:31.471 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2023-05-31T19:50:31.472 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2023-05-31T19:50:31.524 INFO:teuthology.orchestra.run.smithi099.stdout:{ 2023-05-31T19:50:31.525 INFO:teuthology.orchestra.run.smithi099.stdout: "user_id": "rgw-multisite-test-user", 2023-05-31T19:50:31.525 INFO:teuthology.orchestra.run.smithi099.stdout: "display_name": "Multisite Test User", 2023-05-31T19:50:31.525 INFO:teuthology.orchestra.run.smithi099.stdout: "email": "", 2023-05-31T19:50:31.525 INFO:teuthology.orchestra.run.smithi099.stdout: "suspended": 0, 2023-05-31T19:50:31.526 INFO:teuthology.orchestra.run.smithi099.stdout: "max_buckets": 1000, 2023-05-31T19:50:31.526 INFO:teuthology.orchestra.run.smithi099.stdout: "subusers": [], 2023-05-31T19:50:31.526 INFO:teuthology.orchestra.run.smithi099.stdout: "keys": [ 2023-05-31T19:50:31.526 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T19:50:31.526 INFO:teuthology.orchestra.run.smithi099.stdout: "user": "rgw-multisite-test-user", 2023-05-31T19:50:31.526 INFO:teuthology.orchestra.run.smithi099.stdout: "access_key": "Q80H11OLNU1S48PPYL26", 2023-05-31T19:50:31.526 INFO:teuthology.orchestra.run.smithi099.stdout: "secret_key": "jDiBp2Ye6hBaOmushaIpLK5gAc86cS7JCzHKWcIa" 2023-05-31T19:50:31.526 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T19:50:31.527 INFO:teuthology.orchestra.run.smithi099.stdout: ], 2023-05-31T19:50:31.527 INFO:teuthology.orchestra.run.smithi099.stdout: "swift_keys": [], 2023-05-31T19:50:31.527 INFO:teuthology.orchestra.run.smithi099.stdout: "caps": [ 2023-05-31T19:50:31.527 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T19:50:31.527 INFO:teuthology.orchestra.run.smithi099.stdout: "type": "roles", 2023-05-31T19:50:31.527 INFO:teuthology.orchestra.run.smithi099.stdout: "perm": "*" 2023-05-31T19:50:31.527 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T19:50:31.528 INFO:teuthology.orchestra.run.smithi099.stdout: ], 2023-05-31T19:50:31.528 INFO:teuthology.orchestra.run.smithi099.stdout: "op_mask": "read, write, delete", 2023-05-31T19:50:31.528 INFO:teuthology.orchestra.run.smithi099.stdout: "default_placement": "", 2023-05-31T19:50:31.528 INFO:teuthology.orchestra.run.smithi099.stdout: "default_storage_class": "", 2023-05-31T19:50:31.528 INFO:teuthology.orchestra.run.smithi099.stdout: "placement_tags": [], 2023-05-31T19:50:31.528 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket_quota": { 2023-05-31T19:50:31.528 INFO:teuthology.orchestra.run.smithi099.stdout: "enabled": false, 2023-05-31T19:50:31.528 INFO:teuthology.orchestra.run.smithi099.stdout: "check_on_raw": false, 2023-05-31T19:50:31.529 INFO:teuthology.orchestra.run.smithi099.stdout: "max_size": -1, 2023-05-31T19:50:31.529 INFO:teuthology.orchestra.run.smithi099.stdout: "max_size_kb": 0, 2023-05-31T19:50:31.529 INFO:teuthology.orchestra.run.smithi099.stdout: "max_objects": -1 2023-05-31T19:50:31.529 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T19:50:31.529 INFO:teuthology.orchestra.run.smithi099.stdout: "user_quota": { 2023-05-31T19:50:31.529 INFO:teuthology.orchestra.run.smithi099.stdout: "enabled": false, 2023-05-31T19:50:31.529 INFO:teuthology.orchestra.run.smithi099.stdout: "check_on_raw": false, 2023-05-31T19:50:31.529 INFO:teuthology.orchestra.run.smithi099.stdout: "max_size": -1, 2023-05-31T19:50:31.530 INFO:teuthology.orchestra.run.smithi099.stdout: "max_size_kb": 0, 2023-05-31T19:50:31.530 INFO:teuthology.orchestra.run.smithi099.stdout: "max_objects": -1 2023-05-31T19:50:31.530 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T19:50:31.530 INFO:teuthology.orchestra.run.smithi099.stdout: "temp_url_keys": [], 2023-05-31T19:50:31.530 INFO:teuthology.orchestra.run.smithi099.stdout: "type": "rgw", 2023-05-31T19:50:31.530 INFO:teuthology.orchestra.run.smithi099.stdout: "mfa_ids": [] 2023-05-31T19:50:31.531 INFO:teuthology.orchestra.run.smithi099.stdout:} 2023-05-31T19:50:31.531 INFO:teuthology.orchestra.run.smithi099.stdout: 2023-05-31T19:50:31.544 INFO:rgw_multi.tests:meta checkpoint 2023-05-31T19:50:31.546 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-31T19:50:31.546 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-31T19:50:31.547 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-05-31T19:50:31.596 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2023-05-31T19:50:31.596 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2023-05-31T19:50:31.649 INFO:teuthology.orchestra.run.smithi099.stderr:No --period given, using current period=2e6123d2-d695-4350-9a01-61a455bc782b 2023-05-31T19:50:31.660 INFO:teuthology.orchestra.run.smithi099.stdout:[ 2023-05-31T19:50:31.661 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T19:50:31.661 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1685562631.516089_59.1", 2023-05-31T19:50:31.661 INFO:teuthology.orchestra.run.smithi099.stdout: "last_update": "2023-05-31T19:50:31.516089Z" 2023-05-31T19:50:31.661 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T19:50:31.661 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T19:50:31.662 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "", 2023-05-31T19:50:31.662 INFO:teuthology.orchestra.run.smithi099.stdout: "last_update": "0.000000" 2023-05-31T19:50:31.662 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T19:50:31.662 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T19:50:31.662 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "", 2023-05-31T19:50:31.662 INFO:teuthology.orchestra.run.smithi099.stdout: "last_update": "0.000000" 2023-05-31T19:50:31.662 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T19:50:31.662 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T19:50:31.663 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "", 2023-05-31T19:50:31.663 INFO:teuthology.orchestra.run.smithi099.stdout: "last_update": "0.000000" 2023-05-31T19:50:31.663 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T19:50:31.663 INFO:teuthology.orchestra.run.smithi099.stdout:] 2023-05-31T19:50:31.664 INFO:rgw_multi.tests:starting meta checkpoint for zone=a2 2023-05-31T19:50:31.664 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-31T19:50:31.665 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-31T19:50:31.665 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-05-31T19:50:31.714 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2023-05-31T19:50:31.715 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2023-05-31T19:50:31.745 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T19:50:31.740+0000 7fe84977a780 20 RGW-SYNC:meta: read sync status 2023-05-31T19:50:31.745 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T19:50:31.741+0000 7fe84977a780 20 RGW-SYNC:meta: read sync status 2023-05-31T19:50:31.757 INFO:teuthology.orchestra.run.smithi099.stdout:{ 2023-05-31T19:50:31.757 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_status": { 2023-05-31T19:50:31.758 INFO:teuthology.orchestra.run.smithi099.stdout: "info": { 2023-05-31T19:50:31.758 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "sync", 2023-05-31T19:50:31.758 INFO:teuthology.orchestra.run.smithi099.stdout: "num_shards": 4, 2023-05-31T19:50:31.758 INFO:teuthology.orchestra.run.smithi099.stdout: "period": "2e6123d2-d695-4350-9a01-61a455bc782b", 2023-05-31T19:50:31.758 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2023-05-31T19:50:31.758 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T19:50:31.758 INFO:teuthology.orchestra.run.smithi099.stdout: "markers": [ 2023-05-31T19:50:31.759 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T19:50:31.759 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 0, 2023-05-31T19:50:31.759 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T19:50:31.759 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2023-05-31T19:50:31.759 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "", 2023-05-31T19:50:31.759 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T19:50:31.759 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 1, 2023-05-31T19:50:31.760 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T19:50:31.760 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "0.000000", 2023-05-31T19:50:31.760 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2023-05-31T19:50:31.760 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T19:50:31.760 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T19:50:31.761 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T19:50:31.761 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 1, 2023-05-31T19:50:31.761 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T19:50:31.761 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2023-05-31T19:50:31.761 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "", 2023-05-31T19:50:31.761 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T19:50:31.761 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2023-05-31T19:50:31.761 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T19:50:31.762 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "0.000000", 2023-05-31T19:50:31.762 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2023-05-31T19:50:31.762 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T19:50:31.762 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T19:50:31.762 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T19:50:31.762 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 2, 2023-05-31T19:50:31.762 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T19:50:31.762 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2023-05-31T19:50:31.763 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "", 2023-05-31T19:50:31.763 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T19:50:31.763 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2023-05-31T19:50:31.763 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T19:50:31.763 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "0.000000", 2023-05-31T19:50:31.763 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2023-05-31T19:50:31.763 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T19:50:31.763 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T19:50:31.764 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T19:50:31.764 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 3, 2023-05-31T19:50:31.764 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T19:50:31.764 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2023-05-31T19:50:31.764 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "", 2023-05-31T19:50:31.764 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T19:50:31.764 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2023-05-31T19:50:31.765 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T19:50:31.765 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "0.000000", 2023-05-31T19:50:31.765 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2023-05-31T19:50:31.765 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T19:50:31.765 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T19:50:31.765 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2023-05-31T19:50:31.765 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T19:50:31.765 INFO:teuthology.orchestra.run.smithi099.stdout: "full_sync": { 2023-05-31T19:50:31.766 INFO:teuthology.orchestra.run.smithi099.stdout: "total": 1, 2023-05-31T19:50:31.766 INFO:teuthology.orchestra.run.smithi099.stdout: "complete": 1 2023-05-31T19:50:31.766 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T19:50:31.766 INFO:teuthology.orchestra.run.smithi099.stdout: "current_time": "2023-05-31T19:50:31Z" 2023-05-31T19:50:31.766 INFO:teuthology.orchestra.run.smithi099.stdout:} 2023-05-31T19:50:31.767 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1685562631.516089_59.1 target= 2023-05-31T19:50:32.013 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.rgw.client.1 has been restored 2023-05-31T19:50:36.768 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-31T19:50:36.769 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-31T19:50:36.769 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-05-31T19:50:36.826 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2023-05-31T19:50:36.827 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2023-05-31T19:50:36.880 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T19:50:36.875+0000 7f703174f780 20 RGW-SYNC:meta: read sync status 2023-05-31T19:50:36.881 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T19:50:36.876+0000 7f703174f780 20 RGW-SYNC:meta: read sync status 2023-05-31T19:50:36.902 INFO:teuthology.orchestra.run.smithi099.stdout:{ 2023-05-31T19:50:36.902 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_status": { 2023-05-31T19:50:36.903 INFO:teuthology.orchestra.run.smithi099.stdout: "info": { 2023-05-31T19:50:36.903 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "sync", 2023-05-31T19:50:36.903 INFO:teuthology.orchestra.run.smithi099.stdout: "num_shards": 4, 2023-05-31T19:50:36.903 INFO:teuthology.orchestra.run.smithi099.stdout: "period": "2e6123d2-d695-4350-9a01-61a455bc782b", 2023-05-31T19:50:36.904 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2023-05-31T19:50:36.904 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T19:50:36.904 INFO:teuthology.orchestra.run.smithi099.stdout: "markers": [ 2023-05-31T19:50:36.904 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T19:50:36.904 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 0, 2023-05-31T19:50:36.905 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T19:50:36.905 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2023-05-31T19:50:36.905 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "", 2023-05-31T19:50:36.905 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T19:50:36.906 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 1, 2023-05-31T19:50:36.906 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T19:50:36.906 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "0.000000", 2023-05-31T19:50:36.906 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2023-05-31T19:50:36.906 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T19:50:36.906 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T19:50:36.906 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T19:50:36.907 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 1, 2023-05-31T19:50:36.907 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T19:50:36.907 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2023-05-31T19:50:36.907 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "", 2023-05-31T19:50:36.907 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T19:50:36.907 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2023-05-31T19:50:36.907 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T19:50:36.908 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "0.000000", 2023-05-31T19:50:36.908 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2023-05-31T19:50:36.908 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T19:50:36.908 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T19:50:36.908 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T19:50:36.908 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 2, 2023-05-31T19:50:36.909 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T19:50:36.909 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2023-05-31T19:50:36.909 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "", 2023-05-31T19:50:36.909 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T19:50:36.909 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2023-05-31T19:50:36.909 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T19:50:36.910 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "0.000000", 2023-05-31T19:50:36.910 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2023-05-31T19:50:36.910 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T19:50:36.910 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T19:50:36.910 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T19:50:36.910 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 3, 2023-05-31T19:50:36.910 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T19:50:36.911 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2023-05-31T19:50:36.911 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "", 2023-05-31T19:50:36.911 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T19:50:36.911 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2023-05-31T19:50:36.911 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T19:50:36.911 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "0.000000", 2023-05-31T19:50:36.911 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2023-05-31T19:50:36.912 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T19:50:36.912 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T19:50:36.912 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2023-05-31T19:50:36.912 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T19:50:36.912 INFO:teuthology.orchestra.run.smithi099.stdout: "full_sync": { 2023-05-31T19:50:36.912 INFO:teuthology.orchestra.run.smithi099.stdout: "total": 1, 2023-05-31T19:50:36.913 INFO:teuthology.orchestra.run.smithi099.stdout: "complete": 1 2023-05-31T19:50:36.913 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T19:50:36.913 INFO:teuthology.orchestra.run.smithi099.stdout: "current_time": "2023-05-31T19:50:36Z" 2023-05-31T19:50:36.913 INFO:teuthology.orchestra.run.smithi099.stdout:} 2023-05-31T19:50:36.914 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1685562631.516089_59.1 target= 2023-05-31T19:50:41.919 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-31T19:50:41.919 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-31T19:50:41.920 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-05-31T19:50:41.976 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2023-05-31T19:50:41.976 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2023-05-31T19:50:42.008 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T19:50:42.003+0000 7f4e6d61a780 20 RGW-SYNC:meta: read sync status 2023-05-31T19:50:42.009 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T19:50:42.004+0000 7f4e6d61a780 20 RGW-SYNC:meta: read sync status 2023-05-31T19:50:42.020 INFO:teuthology.orchestra.run.smithi099.stdout:{ 2023-05-31T19:50:42.021 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_status": { 2023-05-31T19:50:42.021 INFO:teuthology.orchestra.run.smithi099.stdout: "info": { 2023-05-31T19:50:42.021 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "sync", 2023-05-31T19:50:42.021 INFO:teuthology.orchestra.run.smithi099.stdout: "num_shards": 4, 2023-05-31T19:50:42.021 INFO:teuthology.orchestra.run.smithi099.stdout: "period": "2e6123d2-d695-4350-9a01-61a455bc782b", 2023-05-31T19:50:42.021 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2023-05-31T19:50:42.022 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T19:50:42.022 INFO:teuthology.orchestra.run.smithi099.stdout: "markers": [ 2023-05-31T19:50:42.022 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T19:50:42.022 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 0, 2023-05-31T19:50:42.022 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T19:50:42.022 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2023-05-31T19:50:42.022 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "", 2023-05-31T19:50:42.022 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T19:50:42.023 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 1, 2023-05-31T19:50:42.023 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T19:50:42.023 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "0.000000", 2023-05-31T19:50:42.023 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2023-05-31T19:50:42.023 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T19:50:42.023 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T19:50:42.023 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T19:50:42.024 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 1, 2023-05-31T19:50:42.024 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T19:50:42.024 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2023-05-31T19:50:42.024 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "", 2023-05-31T19:50:42.025 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T19:50:42.025 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2023-05-31T19:50:42.025 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T19:50:42.025 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "0.000000", 2023-05-31T19:50:42.025 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2023-05-31T19:50:42.025 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T19:50:42.025 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T19:50:42.026 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T19:50:42.026 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 2, 2023-05-31T19:50:42.026 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T19:50:42.026 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2023-05-31T19:50:42.026 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "", 2023-05-31T19:50:42.026 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T19:50:42.026 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2023-05-31T19:50:42.026 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T19:50:42.027 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "0.000000", 2023-05-31T19:50:42.027 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2023-05-31T19:50:42.027 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T19:50:42.027 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T19:50:42.027 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T19:50:42.027 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 3, 2023-05-31T19:50:42.027 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T19:50:42.027 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2023-05-31T19:50:42.028 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "", 2023-05-31T19:50:42.028 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T19:50:42.028 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2023-05-31T19:50:42.028 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T19:50:42.028 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "0.000000", 2023-05-31T19:50:42.028 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2023-05-31T19:50:42.028 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T19:50:42.030 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T19:50:42.030 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2023-05-31T19:50:42.031 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T19:50:42.031 INFO:teuthology.orchestra.run.smithi099.stdout: "full_sync": { 2023-05-31T19:50:42.031 INFO:teuthology.orchestra.run.smithi099.stdout: "total": 1, 2023-05-31T19:50:42.031 INFO:teuthology.orchestra.run.smithi099.stdout: "complete": 1 2023-05-31T19:50:42.031 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T19:50:42.031 INFO:teuthology.orchestra.run.smithi099.stdout: "current_time": "2023-05-31T19:50:42Z" 2023-05-31T19:50:42.032 INFO:teuthology.orchestra.run.smithi099.stdout:} 2023-05-31T19:50:42.033 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1685562631.516089_59.1 target= 2023-05-31T19:50:47.034 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-31T19:50:47.035 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-31T19:50:47.035 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-05-31T19:50:47.098 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2023-05-31T19:50:47.098 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2023-05-31T19:50:47.145 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T19:50:47.140+0000 7ff52cc67780 20 RGW-SYNC:meta: read sync status 2023-05-31T19:50:47.146 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T19:50:47.141+0000 7ff52cc67780 20 RGW-SYNC:meta: read sync status 2023-05-31T19:50:47.169 INFO:teuthology.orchestra.run.smithi099.stdout:{ 2023-05-31T19:50:47.169 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_status": { 2023-05-31T19:50:47.169 INFO:teuthology.orchestra.run.smithi099.stdout: "info": { 2023-05-31T19:50:47.169 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "sync", 2023-05-31T19:50:47.170 INFO:teuthology.orchestra.run.smithi099.stdout: "num_shards": 4, 2023-05-31T19:50:47.170 INFO:teuthology.orchestra.run.smithi099.stdout: "period": "2e6123d2-d695-4350-9a01-61a455bc782b", 2023-05-31T19:50:47.170 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2023-05-31T19:50:47.170 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T19:50:47.170 INFO:teuthology.orchestra.run.smithi099.stdout: "markers": [ 2023-05-31T19:50:47.170 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T19:50:47.171 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 0, 2023-05-31T19:50:47.171 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T19:50:47.172 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2023-05-31T19:50:47.172 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "", 2023-05-31T19:50:47.172 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T19:50:47.172 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 1, 2023-05-31T19:50:47.172 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T19:50:47.172 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "0.000000", 2023-05-31T19:50:47.173 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2023-05-31T19:50:47.173 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T19:50:47.173 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T19:50:47.173 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T19:50:47.174 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 1, 2023-05-31T19:50:47.174 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T19:50:47.174 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2023-05-31T19:50:47.174 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "", 2023-05-31T19:50:47.174 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T19:50:47.175 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2023-05-31T19:50:47.175 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T19:50:47.175 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "0.000000", 2023-05-31T19:50:47.175 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2023-05-31T19:50:47.175 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T19:50:47.175 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T19:50:47.175 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T19:50:47.176 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 2, 2023-05-31T19:50:47.176 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T19:50:47.176 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2023-05-31T19:50:47.176 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "", 2023-05-31T19:50:47.176 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T19:50:47.177 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2023-05-31T19:50:47.177 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T19:50:47.177 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "0.000000", 2023-05-31T19:50:47.177 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2023-05-31T19:50:47.177 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T19:50:47.177 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T19:50:47.178 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T19:50:47.178 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 3, 2023-05-31T19:50:47.178 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T19:50:47.178 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2023-05-31T19:50:47.178 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "", 2023-05-31T19:50:47.178 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T19:50:47.178 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2023-05-31T19:50:47.179 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T19:50:47.179 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "0.000000", 2023-05-31T19:50:47.179 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2023-05-31T19:50:47.179 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T19:50:47.179 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T19:50:47.179 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2023-05-31T19:50:47.179 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T19:50:47.180 INFO:teuthology.orchestra.run.smithi099.stdout: "full_sync": { 2023-05-31T19:50:47.180 INFO:teuthology.orchestra.run.smithi099.stdout: "total": 1, 2023-05-31T19:50:47.180 INFO:teuthology.orchestra.run.smithi099.stdout: "complete": 1 2023-05-31T19:50:47.180 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T19:50:47.181 INFO:teuthology.orchestra.run.smithi099.stdout: "current_time": "2023-05-31T19:50:47Z" 2023-05-31T19:50:47.181 INFO:teuthology.orchestra.run.smithi099.stdout:} 2023-05-31T19:50:47.182 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1685562631.516089_59.1 target= 2023-05-31T19:50:52.186 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-31T19:50:52.186 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-31T19:50:52.186 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-05-31T19:50:52.244 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2023-05-31T19:50:52.244 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2023-05-31T19:50:52.301 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T19:50:52.295+0000 7f7c824dd780 20 RGW-SYNC:meta: read sync status 2023-05-31T19:50:52.302 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T19:50:52.296+0000 7f7c824dd780 20 RGW-SYNC:meta: read sync status 2023-05-31T19:50:52.323 INFO:teuthology.orchestra.run.smithi099.stdout:{ 2023-05-31T19:50:52.323 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_status": { 2023-05-31T19:50:52.323 INFO:teuthology.orchestra.run.smithi099.stdout: "info": { 2023-05-31T19:50:52.323 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "sync", 2023-05-31T19:50:52.324 INFO:teuthology.orchestra.run.smithi099.stdout: "num_shards": 4, 2023-05-31T19:50:52.324 INFO:teuthology.orchestra.run.smithi099.stdout: "period": "2e6123d2-d695-4350-9a01-61a455bc782b", 2023-05-31T19:50:52.324 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2023-05-31T19:50:52.324 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T19:50:52.324 INFO:teuthology.orchestra.run.smithi099.stdout: "markers": [ 2023-05-31T19:50:52.325 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T19:50:52.325 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 0, 2023-05-31T19:50:52.325 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T19:50:52.325 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2023-05-31T19:50:52.325 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "", 2023-05-31T19:50:52.326 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T19:50:52.326 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 1, 2023-05-31T19:50:52.326 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T19:50:52.326 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "0.000000", 2023-05-31T19:50:52.326 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2023-05-31T19:50:52.326 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T19:50:52.327 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T19:50:52.327 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T19:50:52.327 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 1, 2023-05-31T19:50:52.327 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T19:50:52.327 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2023-05-31T19:50:52.327 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "", 2023-05-31T19:50:52.327 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T19:50:52.328 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2023-05-31T19:50:52.328 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T19:50:52.328 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "0.000000", 2023-05-31T19:50:52.328 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2023-05-31T19:50:52.328 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T19:50:52.328 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T19:50:52.328 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T19:50:52.329 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 2, 2023-05-31T19:50:52.329 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T19:50:52.329 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2023-05-31T19:50:52.329 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "", 2023-05-31T19:50:52.329 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T19:50:52.330 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2023-05-31T19:50:52.330 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T19:50:52.330 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "0.000000", 2023-05-31T19:50:52.330 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2023-05-31T19:50:52.330 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T19:50:52.330 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T19:50:52.330 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T19:50:52.331 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 3, 2023-05-31T19:50:52.331 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T19:50:52.331 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2023-05-31T19:50:52.331 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "", 2023-05-31T19:50:52.331 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T19:50:52.331 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2023-05-31T19:50:52.331 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T19:50:52.332 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "0.000000", 2023-05-31T19:50:52.332 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2023-05-31T19:50:52.332 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T19:50:52.332 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T19:50:52.332 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2023-05-31T19:50:52.332 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T19:50:52.332 INFO:teuthology.orchestra.run.smithi099.stdout: "full_sync": { 2023-05-31T19:50:52.333 INFO:teuthology.orchestra.run.smithi099.stdout: "total": 1, 2023-05-31T19:50:52.333 INFO:teuthology.orchestra.run.smithi099.stdout: "complete": 1 2023-05-31T19:50:52.333 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T19:50:52.333 INFO:teuthology.orchestra.run.smithi099.stdout: "current_time": "2023-05-31T19:50:52Z" 2023-05-31T19:50:52.333 INFO:teuthology.orchestra.run.smithi099.stdout:} 2023-05-31T19:50:52.334 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1685562631.516089_59.1 target= 2023-05-31T19:50:57.335 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-31T19:50:57.336 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-31T19:50:57.336 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-05-31T19:50:57.393 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2023-05-31T19:50:57.393 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2023-05-31T19:50:57.435 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T19:50:57.429+0000 7fdb279e0780 20 RGW-SYNC:meta: read sync status 2023-05-31T19:50:57.435 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T19:50:57.430+0000 7fdb279e0780 20 RGW-SYNC:meta: read sync status 2023-05-31T19:50:57.457 INFO:teuthology.orchestra.run.smithi099.stdout:{ 2023-05-31T19:50:57.458 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_status": { 2023-05-31T19:50:57.458 INFO:teuthology.orchestra.run.smithi099.stdout: "info": { 2023-05-31T19:50:57.458 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "sync", 2023-05-31T19:50:57.458 INFO:teuthology.orchestra.run.smithi099.stdout: "num_shards": 4, 2023-05-31T19:50:57.458 INFO:teuthology.orchestra.run.smithi099.stdout: "period": "2e6123d2-d695-4350-9a01-61a455bc782b", 2023-05-31T19:50:57.459 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2023-05-31T19:50:57.459 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T19:50:57.459 INFO:teuthology.orchestra.run.smithi099.stdout: "markers": [ 2023-05-31T19:50:57.459 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T19:50:57.459 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 0, 2023-05-31T19:50:57.459 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T19:50:57.459 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2023-05-31T19:50:57.459 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "", 2023-05-31T19:50:57.460 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T19:50:57.460 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 1, 2023-05-31T19:50:57.460 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T19:50:57.460 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "0.000000", 2023-05-31T19:50:57.460 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2023-05-31T19:50:57.460 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T19:50:57.461 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T19:50:57.461 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T19:50:57.461 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 1, 2023-05-31T19:50:57.461 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T19:50:57.461 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2023-05-31T19:50:57.461 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "", 2023-05-31T19:50:57.461 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T19:50:57.461 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2023-05-31T19:50:57.462 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T19:50:57.462 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "0.000000", 2023-05-31T19:50:57.462 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2023-05-31T19:50:57.462 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T19:50:57.462 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T19:50:57.462 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T19:50:57.462 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 2, 2023-05-31T19:50:57.463 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T19:50:57.463 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2023-05-31T19:50:57.463 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "", 2023-05-31T19:50:57.463 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T19:50:57.463 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2023-05-31T19:50:57.463 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T19:50:57.464 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "0.000000", 2023-05-31T19:50:57.464 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2023-05-31T19:50:57.464 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T19:50:57.464 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T19:50:57.464 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T19:50:57.464 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 3, 2023-05-31T19:50:57.464 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T19:50:57.465 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2023-05-31T19:50:57.465 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "", 2023-05-31T19:50:57.465 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T19:50:57.465 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2023-05-31T19:50:57.465 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T19:50:57.465 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "0.000000", 2023-05-31T19:50:57.465 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2023-05-31T19:50:57.465 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T19:50:57.466 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T19:50:57.466 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2023-05-31T19:50:57.466 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T19:50:57.466 INFO:teuthology.orchestra.run.smithi099.stdout: "full_sync": { 2023-05-31T19:50:57.466 INFO:teuthology.orchestra.run.smithi099.stdout: "total": 1, 2023-05-31T19:50:57.466 INFO:teuthology.orchestra.run.smithi099.stdout: "complete": 1 2023-05-31T19:50:57.466 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T19:50:57.466 INFO:teuthology.orchestra.run.smithi099.stdout: "current_time": "2023-05-31T19:50:57Z" 2023-05-31T19:50:57.467 INFO:teuthology.orchestra.run.smithi099.stdout:} 2023-05-31T19:50:57.468 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1685562631.516089_59.1 target= 2023-05-31T19:51:02.468 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-31T19:51:02.469 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-31T19:51:02.469 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-05-31T19:51:02.526 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2023-05-31T19:51:02.526 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2023-05-31T19:51:02.570 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T19:51:02.564+0000 7f1766dcd780 20 RGW-SYNC:meta: read sync status 2023-05-31T19:51:02.571 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T19:51:02.565+0000 7f1766dcd780 20 RGW-SYNC:meta: read sync status 2023-05-31T19:51:02.589 INFO:teuthology.orchestra.run.smithi099.stdout:{ 2023-05-31T19:51:02.589 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_status": { 2023-05-31T19:51:02.590 INFO:teuthology.orchestra.run.smithi099.stdout: "info": { 2023-05-31T19:51:02.590 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "sync", 2023-05-31T19:51:02.590 INFO:teuthology.orchestra.run.smithi099.stdout: "num_shards": 4, 2023-05-31T19:51:02.590 INFO:teuthology.orchestra.run.smithi099.stdout: "period": "2e6123d2-d695-4350-9a01-61a455bc782b", 2023-05-31T19:51:02.590 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2023-05-31T19:51:02.590 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T19:51:02.591 INFO:teuthology.orchestra.run.smithi099.stdout: "markers": [ 2023-05-31T19:51:02.591 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T19:51:02.591 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 0, 2023-05-31T19:51:02.591 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T19:51:02.592 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2023-05-31T19:51:02.592 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "", 2023-05-31T19:51:02.592 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T19:51:02.592 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 1, 2023-05-31T19:51:02.592 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T19:51:02.592 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "0.000000", 2023-05-31T19:51:02.592 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2023-05-31T19:51:02.593 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T19:51:02.593 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T19:51:02.593 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T19:51:02.593 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 1, 2023-05-31T19:51:02.593 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T19:51:02.593 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2023-05-31T19:51:02.593 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "", 2023-05-31T19:51:02.594 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T19:51:02.594 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2023-05-31T19:51:02.594 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T19:51:02.594 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "0.000000", 2023-05-31T19:51:02.594 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2023-05-31T19:51:02.594 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T19:51:02.594 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T19:51:02.595 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T19:51:02.595 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 2, 2023-05-31T19:51:02.595 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T19:51:02.595 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2023-05-31T19:51:02.595 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "", 2023-05-31T19:51:02.595 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T19:51:02.595 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2023-05-31T19:51:02.595 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T19:51:02.596 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "0.000000", 2023-05-31T19:51:02.596 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2023-05-31T19:51:02.596 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T19:51:02.596 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T19:51:02.596 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T19:51:02.596 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 3, 2023-05-31T19:51:02.596 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T19:51:02.597 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2023-05-31T19:51:02.597 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "", 2023-05-31T19:51:02.597 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T19:51:02.597 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2023-05-31T19:51:02.597 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T19:51:02.597 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "0.000000", 2023-05-31T19:51:02.597 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2023-05-31T19:51:02.597 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T19:51:02.598 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T19:51:02.598 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2023-05-31T19:51:02.598 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T19:51:02.598 INFO:teuthology.orchestra.run.smithi099.stdout: "full_sync": { 2023-05-31T19:51:02.598 INFO:teuthology.orchestra.run.smithi099.stdout: "total": 1, 2023-05-31T19:51:02.598 INFO:teuthology.orchestra.run.smithi099.stdout: "complete": 1 2023-05-31T19:51:02.598 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T19:51:02.598 INFO:teuthology.orchestra.run.smithi099.stdout: "current_time": "2023-05-31T19:51:02Z" 2023-05-31T19:51:02.599 INFO:teuthology.orchestra.run.smithi099.stdout:} 2023-05-31T19:51:02.600 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1685562631.516089_59.1 target= 2023-05-31T19:51:07.601 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-31T19:51:07.601 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-31T19:51:07.602 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-05-31T19:51:07.658 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2023-05-31T19:51:07.659 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2023-05-31T19:51:07.693 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T19:51:07.687+0000 7f193db65780 20 RGW-SYNC:meta: read sync status 2023-05-31T19:51:07.694 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T19:51:07.688+0000 7f193db65780 20 RGW-SYNC:meta: read sync status 2023-05-31T19:51:07.705 INFO:teuthology.orchestra.run.smithi099.stdout:{ 2023-05-31T19:51:07.706 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_status": { 2023-05-31T19:51:07.706 INFO:teuthology.orchestra.run.smithi099.stdout: "info": { 2023-05-31T19:51:07.706 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "sync", 2023-05-31T19:51:07.706 INFO:teuthology.orchestra.run.smithi099.stdout: "num_shards": 4, 2023-05-31T19:51:07.706 INFO:teuthology.orchestra.run.smithi099.stdout: "period": "2e6123d2-d695-4350-9a01-61a455bc782b", 2023-05-31T19:51:07.706 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2023-05-31T19:51:07.706 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T19:51:07.707 INFO:teuthology.orchestra.run.smithi099.stdout: "markers": [ 2023-05-31T19:51:07.707 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T19:51:07.707 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 0, 2023-05-31T19:51:07.707 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T19:51:07.707 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2023-05-31T19:51:07.707 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "", 2023-05-31T19:51:07.707 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T19:51:07.708 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 1, 2023-05-31T19:51:07.708 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T19:51:07.708 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "0.000000", 2023-05-31T19:51:07.708 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2023-05-31T19:51:07.708 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T19:51:07.708 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T19:51:07.708 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T19:51:07.709 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 1, 2023-05-31T19:51:07.709 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T19:51:07.709 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2023-05-31T19:51:07.709 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "", 2023-05-31T19:51:07.709 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T19:51:07.709 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2023-05-31T19:51:07.710 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T19:51:07.710 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "0.000000", 2023-05-31T19:51:07.710 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2023-05-31T19:51:07.710 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T19:51:07.710 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T19:51:07.710 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T19:51:07.710 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 2, 2023-05-31T19:51:07.710 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T19:51:07.711 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2023-05-31T19:51:07.711 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "", 2023-05-31T19:51:07.711 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T19:51:07.711 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2023-05-31T19:51:07.711 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T19:51:07.711 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "0.000000", 2023-05-31T19:51:07.711 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2023-05-31T19:51:07.711 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T19:51:07.712 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T19:51:07.712 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T19:51:07.712 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 3, 2023-05-31T19:51:07.712 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T19:51:07.712 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2023-05-31T19:51:07.712 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "", 2023-05-31T19:51:07.712 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T19:51:07.713 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2023-05-31T19:51:07.713 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T19:51:07.713 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "0.000000", 2023-05-31T19:51:07.713 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2023-05-31T19:51:07.713 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T19:51:07.713 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T19:51:07.713 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2023-05-31T19:51:07.713 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T19:51:07.714 INFO:teuthology.orchestra.run.smithi099.stdout: "full_sync": { 2023-05-31T19:51:07.714 INFO:teuthology.orchestra.run.smithi099.stdout: "total": 1, 2023-05-31T19:51:07.714 INFO:teuthology.orchestra.run.smithi099.stdout: "complete": 1 2023-05-31T19:51:07.714 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T19:51:07.715 INFO:teuthology.orchestra.run.smithi099.stdout: "current_time": "2023-05-31T19:51:07Z" 2023-05-31T19:51:07.715 INFO:teuthology.orchestra.run.smithi099.stdout:} 2023-05-31T19:51:07.716 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1685562631.516089_59.1 target= 2023-05-31T19:51:12.717 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-31T19:51:12.718 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-31T19:51:12.719 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-05-31T19:51:12.776 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2023-05-31T19:51:12.777 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2023-05-31T19:51:12.826 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T19:51:12.821+0000 7f7ce3e48780 20 RGW-SYNC:meta: read sync status 2023-05-31T19:51:12.826 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T19:51:12.822+0000 7f7ce3e48780 20 RGW-SYNC:meta: read sync status 2023-05-31T19:51:12.848 INFO:teuthology.orchestra.run.smithi099.stdout:{ 2023-05-31T19:51:12.848 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_status": { 2023-05-31T19:51:12.848 INFO:teuthology.orchestra.run.smithi099.stdout: "info": { 2023-05-31T19:51:12.848 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "sync", 2023-05-31T19:51:12.849 INFO:teuthology.orchestra.run.smithi099.stdout: "num_shards": 4, 2023-05-31T19:51:12.849 INFO:teuthology.orchestra.run.smithi099.stdout: "period": "2e6123d2-d695-4350-9a01-61a455bc782b", 2023-05-31T19:51:12.849 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2023-05-31T19:51:12.849 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T19:51:12.849 INFO:teuthology.orchestra.run.smithi099.stdout: "markers": [ 2023-05-31T19:51:12.849 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T19:51:12.849 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 0, 2023-05-31T19:51:12.849 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T19:51:12.850 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2023-05-31T19:51:12.850 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "", 2023-05-31T19:51:12.850 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T19:51:12.850 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 1, 2023-05-31T19:51:12.850 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T19:51:12.851 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "0.000000", 2023-05-31T19:51:12.851 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2023-05-31T19:51:12.851 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T19:51:12.851 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T19:51:12.851 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T19:51:12.851 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 1, 2023-05-31T19:51:12.852 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T19:51:12.852 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2023-05-31T19:51:12.852 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "", 2023-05-31T19:51:12.852 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T19:51:12.852 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2023-05-31T19:51:12.852 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T19:51:12.852 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "0.000000", 2023-05-31T19:51:12.853 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2023-05-31T19:51:12.853 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T19:51:12.853 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T19:51:12.853 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T19:51:12.853 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 2, 2023-05-31T19:51:12.853 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T19:51:12.853 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2023-05-31T19:51:12.853 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "", 2023-05-31T19:51:12.854 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T19:51:12.854 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2023-05-31T19:51:12.854 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T19:51:12.854 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "0.000000", 2023-05-31T19:51:12.854 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2023-05-31T19:51:12.854 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T19:51:12.854 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T19:51:12.854 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T19:51:12.855 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 3, 2023-05-31T19:51:12.855 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T19:51:12.855 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2023-05-31T19:51:12.855 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "", 2023-05-31T19:51:12.855 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T19:51:12.855 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2023-05-31T19:51:12.855 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T19:51:12.855 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "0.000000", 2023-05-31T19:51:12.856 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2023-05-31T19:51:12.856 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T19:51:12.856 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T19:51:12.856 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2023-05-31T19:51:12.856 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T19:51:12.856 INFO:teuthology.orchestra.run.smithi099.stdout: "full_sync": { 2023-05-31T19:51:12.856 INFO:teuthology.orchestra.run.smithi099.stdout: "total": 1, 2023-05-31T19:51:12.857 INFO:teuthology.orchestra.run.smithi099.stdout: "complete": 1 2023-05-31T19:51:12.857 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T19:51:12.857 INFO:teuthology.orchestra.run.smithi099.stdout: "current_time": "2023-05-31T19:51:12Z" 2023-05-31T19:51:12.857 INFO:teuthology.orchestra.run.smithi099.stdout:} 2023-05-31T19:51:12.858 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1685562631.516089_59.1 target= 2023-05-31T19:51:17.859 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-31T19:51:17.859 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-31T19:51:17.860 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-05-31T19:51:17.916 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2023-05-31T19:51:17.917 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2023-05-31T19:51:17.969 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T19:51:17.964+0000 7ffb5820f780 20 RGW-SYNC:meta: read sync status 2023-05-31T19:51:17.970 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T19:51:17.965+0000 7ffb5820f780 20 RGW-SYNC:meta: read sync status 2023-05-31T19:51:17.988 INFO:teuthology.orchestra.run.smithi099.stdout:{ 2023-05-31T19:51:17.988 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_status": { 2023-05-31T19:51:17.988 INFO:teuthology.orchestra.run.smithi099.stdout: "info": { 2023-05-31T19:51:17.988 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "sync", 2023-05-31T19:51:17.988 INFO:teuthology.orchestra.run.smithi099.stdout: "num_shards": 4, 2023-05-31T19:51:17.989 INFO:teuthology.orchestra.run.smithi099.stdout: "period": "2e6123d2-d695-4350-9a01-61a455bc782b", 2023-05-31T19:51:17.989 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2023-05-31T19:51:17.989 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T19:51:17.989 INFO:teuthology.orchestra.run.smithi099.stdout: "markers": [ 2023-05-31T19:51:17.989 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T19:51:17.989 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 0, 2023-05-31T19:51:17.990 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T19:51:17.990 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2023-05-31T19:51:17.990 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "", 2023-05-31T19:51:17.990 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T19:51:17.990 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 1, 2023-05-31T19:51:17.990 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T19:51:17.990 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "0.000000", 2023-05-31T19:51:17.991 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2023-05-31T19:51:17.991 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T19:51:17.991 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T19:51:17.991 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T19:51:17.992 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 1, 2023-05-31T19:51:17.993 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T19:51:17.994 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2023-05-31T19:51:17.994 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "", 2023-05-31T19:51:17.994 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T19:51:17.994 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2023-05-31T19:51:17.994 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T19:51:17.994 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "0.000000", 2023-05-31T19:51:17.994 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2023-05-31T19:51:17.995 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T19:51:17.995 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T19:51:17.995 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T19:51:17.995 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 2, 2023-05-31T19:51:17.996 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T19:51:17.996 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2023-05-31T19:51:17.996 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "", 2023-05-31T19:51:17.996 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T19:51:17.996 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2023-05-31T19:51:17.996 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T19:51:17.996 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "0.000000", 2023-05-31T19:51:17.997 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2023-05-31T19:51:17.997 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T19:51:17.997 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T19:51:17.997 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T19:51:17.997 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 3, 2023-05-31T19:51:17.997 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T19:51:17.997 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2023-05-31T19:51:17.997 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "", 2023-05-31T19:51:17.998 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T19:51:17.998 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2023-05-31T19:51:17.998 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T19:51:17.998 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "0.000000", 2023-05-31T19:51:17.998 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2023-05-31T19:51:17.998 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T19:51:17.998 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T19:51:17.999 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2023-05-31T19:51:17.999 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T19:51:17.999 INFO:teuthology.orchestra.run.smithi099.stdout: "full_sync": { 2023-05-31T19:51:17.999 INFO:teuthology.orchestra.run.smithi099.stdout: "total": 1, 2023-05-31T19:51:17.999 INFO:teuthology.orchestra.run.smithi099.stdout: "complete": 1 2023-05-31T19:51:17.999 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T19:51:17.999 INFO:teuthology.orchestra.run.smithi099.stdout: "current_time": "2023-05-31T19:51:17Z" 2023-05-31T19:51:17.999 INFO:teuthology.orchestra.run.smithi099.stdout:} 2023-05-31T19:51:18.000 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1685562631.516089_59.1 target= 2023-05-31T19:51:23.002 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-31T19:51:23.002 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-31T19:51:23.002 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-05-31T19:51:23.060 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2023-05-31T19:51:23.061 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2023-05-31T19:51:23.091 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T19:51:23.086+0000 7fe601ad6780 20 RGW-SYNC:meta: read sync status 2023-05-31T19:51:23.092 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T19:51:23.087+0000 7fe601ad6780 20 RGW-SYNC:meta: read sync status 2023-05-31T19:51:23.104 INFO:teuthology.orchestra.run.smithi099.stdout:{ 2023-05-31T19:51:23.105 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_status": { 2023-05-31T19:51:23.105 INFO:teuthology.orchestra.run.smithi099.stdout: "info": { 2023-05-31T19:51:23.106 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "sync", 2023-05-31T19:51:23.106 INFO:teuthology.orchestra.run.smithi099.stdout: "num_shards": 4, 2023-05-31T19:51:23.106 INFO:teuthology.orchestra.run.smithi099.stdout: "period": "2e6123d2-d695-4350-9a01-61a455bc782b", 2023-05-31T19:51:23.106 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2023-05-31T19:51:23.107 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T19:51:23.107 INFO:teuthology.orchestra.run.smithi099.stdout: "markers": [ 2023-05-31T19:51:23.107 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T19:51:23.107 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 0, 2023-05-31T19:51:23.107 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T19:51:23.107 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2023-05-31T19:51:23.108 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "", 2023-05-31T19:51:23.108 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T19:51:23.108 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 1, 2023-05-31T19:51:23.108 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T19:51:23.108 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "0.000000", 2023-05-31T19:51:23.108 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2023-05-31T19:51:23.109 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T19:51:23.109 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T19:51:23.110 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T19:51:23.110 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 1, 2023-05-31T19:51:23.110 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T19:51:23.111 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2023-05-31T19:51:23.111 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "", 2023-05-31T19:51:23.111 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T19:51:23.111 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2023-05-31T19:51:23.112 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T19:51:23.112 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "0.000000", 2023-05-31T19:51:23.112 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2023-05-31T19:51:23.112 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T19:51:23.113 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T19:51:23.113 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T19:51:23.113 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 2, 2023-05-31T19:51:23.113 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T19:51:23.113 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2023-05-31T19:51:23.114 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "", 2023-05-31T19:51:23.114 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T19:51:23.114 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2023-05-31T19:51:23.114 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T19:51:23.114 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "0.000000", 2023-05-31T19:51:23.115 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2023-05-31T19:51:23.115 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T19:51:23.115 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T19:51:23.115 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T19:51:23.116 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 3, 2023-05-31T19:51:23.117 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T19:51:23.117 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2023-05-31T19:51:23.117 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "", 2023-05-31T19:51:23.117 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T19:51:23.117 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2023-05-31T19:51:23.117 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T19:51:23.117 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "0.000000", 2023-05-31T19:51:23.118 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2023-05-31T19:51:23.118 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T19:51:23.118 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T19:51:23.118 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2023-05-31T19:51:23.118 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T19:51:23.118 INFO:teuthology.orchestra.run.smithi099.stdout: "full_sync": { 2023-05-31T19:51:23.118 INFO:teuthology.orchestra.run.smithi099.stdout: "total": 1, 2023-05-31T19:51:23.119 INFO:teuthology.orchestra.run.smithi099.stdout: "complete": 1 2023-05-31T19:51:23.119 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T19:51:23.119 INFO:teuthology.orchestra.run.smithi099.stdout: "current_time": "2023-05-31T19:51:23Z" 2023-05-31T19:51:23.119 INFO:teuthology.orchestra.run.smithi099.stdout:} 2023-05-31T19:51:23.120 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1685562631.516089_59.1 target= 2023-05-31T19:51:28.121 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-31T19:51:28.121 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-31T19:51:28.122 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-05-31T19:51:28.178 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2023-05-31T19:51:28.179 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2023-05-31T19:51:28.229 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T19:51:28.224+0000 7f30c4907780 20 RGW-SYNC:meta: read sync status 2023-05-31T19:51:28.230 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T19:51:28.225+0000 7f30c4907780 20 RGW-SYNC:meta: read sync status 2023-05-31T19:51:28.253 INFO:teuthology.orchestra.run.smithi099.stdout:{ 2023-05-31T19:51:28.253 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_status": { 2023-05-31T19:51:28.253 INFO:teuthology.orchestra.run.smithi099.stdout: "info": { 2023-05-31T19:51:28.253 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "sync", 2023-05-31T19:51:28.254 INFO:teuthology.orchestra.run.smithi099.stdout: "num_shards": 4, 2023-05-31T19:51:28.254 INFO:teuthology.orchestra.run.smithi099.stdout: "period": "2e6123d2-d695-4350-9a01-61a455bc782b", 2023-05-31T19:51:28.254 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2023-05-31T19:51:28.254 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T19:51:28.255 INFO:teuthology.orchestra.run.smithi099.stdout: "markers": [ 2023-05-31T19:51:28.255 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T19:51:28.255 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 0, 2023-05-31T19:51:28.256 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T19:51:28.256 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2023-05-31T19:51:28.256 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "", 2023-05-31T19:51:28.256 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T19:51:28.257 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 1, 2023-05-31T19:51:28.257 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T19:51:28.257 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "0.000000", 2023-05-31T19:51:28.257 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2023-05-31T19:51:28.257 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T19:51:28.257 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T19:51:28.258 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T19:51:28.258 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 1, 2023-05-31T19:51:28.258 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T19:51:28.258 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2023-05-31T19:51:28.258 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "", 2023-05-31T19:51:28.259 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T19:51:28.259 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2023-05-31T19:51:28.259 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T19:51:28.259 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "0.000000", 2023-05-31T19:51:28.259 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2023-05-31T19:51:28.259 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T19:51:28.259 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T19:51:28.259 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T19:51:28.260 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 2, 2023-05-31T19:51:28.260 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T19:51:28.260 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2023-05-31T19:51:28.260 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "", 2023-05-31T19:51:28.260 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T19:51:28.260 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2023-05-31T19:51:28.260 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T19:51:28.261 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "0.000000", 2023-05-31T19:51:28.261 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2023-05-31T19:51:28.261 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T19:51:28.262 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T19:51:28.262 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T19:51:28.262 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 3, 2023-05-31T19:51:28.262 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T19:51:28.262 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2023-05-31T19:51:28.262 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "", 2023-05-31T19:51:28.263 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T19:51:28.263 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2023-05-31T19:51:28.263 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T19:51:28.263 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "0.000000", 2023-05-31T19:51:28.264 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2023-05-31T19:51:28.264 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T19:51:28.264 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T19:51:28.264 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2023-05-31T19:51:28.264 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T19:51:28.264 INFO:teuthology.orchestra.run.smithi099.stdout: "full_sync": { 2023-05-31T19:51:28.264 INFO:teuthology.orchestra.run.smithi099.stdout: "total": 1, 2023-05-31T19:51:28.265 INFO:teuthology.orchestra.run.smithi099.stdout: "complete": 1 2023-05-31T19:51:28.265 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T19:51:28.265 INFO:teuthology.orchestra.run.smithi099.stdout: "current_time": "2023-05-31T19:51:28Z" 2023-05-31T19:51:28.265 INFO:teuthology.orchestra.run.smithi099.stdout:} 2023-05-31T19:51:28.266 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1685562631.516089_59.1 target= 2023-05-31T19:51:33.267 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-31T19:51:33.267 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-31T19:51:33.268 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-05-31T19:51:33.325 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2023-05-31T19:51:33.325 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2023-05-31T19:51:33.360 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T19:51:33.354+0000 7f9dd233b780 20 RGW-SYNC:meta: read sync status 2023-05-31T19:51:33.361 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T19:51:33.355+0000 7f9dd233b780 20 RGW-SYNC:meta: read sync status 2023-05-31T19:51:33.374 INFO:teuthology.orchestra.run.smithi099.stdout:{ 2023-05-31T19:51:33.374 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_status": { 2023-05-31T19:51:33.374 INFO:teuthology.orchestra.run.smithi099.stdout: "info": { 2023-05-31T19:51:33.374 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "sync", 2023-05-31T19:51:33.374 INFO:teuthology.orchestra.run.smithi099.stdout: "num_shards": 4, 2023-05-31T19:51:33.375 INFO:teuthology.orchestra.run.smithi099.stdout: "period": "2e6123d2-d695-4350-9a01-61a455bc782b", 2023-05-31T19:51:33.375 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2023-05-31T19:51:33.375 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T19:51:33.375 INFO:teuthology.orchestra.run.smithi099.stdout: "markers": [ 2023-05-31T19:51:33.375 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T19:51:33.375 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 0, 2023-05-31T19:51:33.375 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T19:51:33.376 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2023-05-31T19:51:33.376 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "", 2023-05-31T19:51:33.376 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T19:51:33.376 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 1, 2023-05-31T19:51:33.376 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T19:51:33.376 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "0.000000", 2023-05-31T19:51:33.376 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2023-05-31T19:51:33.376 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T19:51:33.377 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T19:51:33.377 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T19:51:33.377 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 1, 2023-05-31T19:51:33.377 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T19:51:33.377 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2023-05-31T19:51:33.377 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "", 2023-05-31T19:51:33.377 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T19:51:33.378 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2023-05-31T19:51:33.378 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T19:51:33.378 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "0.000000", 2023-05-31T19:51:33.378 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2023-05-31T19:51:33.378 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T19:51:33.379 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T19:51:33.379 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T19:51:33.379 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 2, 2023-05-31T19:51:33.379 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T19:51:33.379 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2023-05-31T19:51:33.379 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "", 2023-05-31T19:51:33.379 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T19:51:33.379 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2023-05-31T19:51:33.380 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T19:51:33.380 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "0.000000", 2023-05-31T19:51:33.380 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2023-05-31T19:51:33.380 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T19:51:33.380 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T19:51:33.380 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T19:51:33.380 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 3, 2023-05-31T19:51:33.380 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T19:51:33.381 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2023-05-31T19:51:33.381 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "", 2023-05-31T19:51:33.381 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T19:51:33.381 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2023-05-31T19:51:33.381 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T19:51:33.381 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "0.000000", 2023-05-31T19:51:33.381 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2023-05-31T19:51:33.381 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T19:51:33.382 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T19:51:33.382 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2023-05-31T19:51:33.382 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T19:51:33.382 INFO:teuthology.orchestra.run.smithi099.stdout: "full_sync": { 2023-05-31T19:51:33.382 INFO:teuthology.orchestra.run.smithi099.stdout: "total": 1, 2023-05-31T19:51:33.382 INFO:teuthology.orchestra.run.smithi099.stdout: "complete": 1 2023-05-31T19:51:33.382 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T19:51:33.383 INFO:teuthology.orchestra.run.smithi099.stdout: "current_time": "2023-05-31T19:51:33Z" 2023-05-31T19:51:33.383 INFO:teuthology.orchestra.run.smithi099.stdout:} 2023-05-31T19:51:33.384 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1685562631.516089_59.1 target= 2023-05-31T19:51:38.385 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-31T19:51:38.385 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-31T19:51:38.386 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-05-31T19:51:38.443 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2023-05-31T19:51:38.444 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2023-05-31T19:51:38.501 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T19:51:38.495+0000 7fd07e2a4780 20 RGW-SYNC:meta: read sync status 2023-05-31T19:51:38.501 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T19:51:38.496+0000 7fd07e2a4780 20 RGW-SYNC:meta: read sync status 2023-05-31T19:51:38.524 INFO:teuthology.orchestra.run.smithi099.stdout:{ 2023-05-31T19:51:38.524 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_status": { 2023-05-31T19:51:38.524 INFO:teuthology.orchestra.run.smithi099.stdout: "info": { 2023-05-31T19:51:38.525 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "sync", 2023-05-31T19:51:38.525 INFO:teuthology.orchestra.run.smithi099.stdout: "num_shards": 4, 2023-05-31T19:51:38.525 INFO:teuthology.orchestra.run.smithi099.stdout: "period": "2e6123d2-d695-4350-9a01-61a455bc782b", 2023-05-31T19:51:38.525 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2023-05-31T19:51:38.525 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T19:51:38.525 INFO:teuthology.orchestra.run.smithi099.stdout: "markers": [ 2023-05-31T19:51:38.525 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T19:51:38.526 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 0, 2023-05-31T19:51:38.526 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T19:51:38.526 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2023-05-31T19:51:38.526 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "", 2023-05-31T19:51:38.526 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T19:51:38.527 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 1, 2023-05-31T19:51:38.527 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T19:51:38.527 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "0.000000", 2023-05-31T19:51:38.527 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2023-05-31T19:51:38.527 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T19:51:38.527 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T19:51:38.527 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T19:51:38.527 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 1, 2023-05-31T19:51:38.528 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T19:51:38.528 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2023-05-31T19:51:38.528 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "", 2023-05-31T19:51:38.528 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T19:51:38.528 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2023-05-31T19:51:38.528 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T19:51:38.529 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "0.000000", 2023-05-31T19:51:38.529 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2023-05-31T19:51:38.529 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T19:51:38.529 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T19:51:38.529 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T19:51:38.529 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 2, 2023-05-31T19:51:38.529 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T19:51:38.529 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2023-05-31T19:51:38.530 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "", 2023-05-31T19:51:38.530 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T19:51:38.530 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2023-05-31T19:51:38.530 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T19:51:38.530 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "0.000000", 2023-05-31T19:51:38.530 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2023-05-31T19:51:38.530 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T19:51:38.531 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T19:51:38.531 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T19:51:38.531 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 3, 2023-05-31T19:51:38.531 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T19:51:38.532 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2023-05-31T19:51:38.532 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "", 2023-05-31T19:51:38.532 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T19:51:38.532 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2023-05-31T19:51:38.533 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T19:51:38.533 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "0.000000", 2023-05-31T19:51:38.534 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2023-05-31T19:51:38.534 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T19:51:38.534 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T19:51:38.535 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2023-05-31T19:51:38.535 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T19:51:38.535 INFO:teuthology.orchestra.run.smithi099.stdout: "full_sync": { 2023-05-31T19:51:38.536 INFO:teuthology.orchestra.run.smithi099.stdout: "total": 1, 2023-05-31T19:51:38.536 INFO:teuthology.orchestra.run.smithi099.stdout: "complete": 1 2023-05-31T19:51:38.536 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T19:51:38.537 INFO:teuthology.orchestra.run.smithi099.stdout: "current_time": "2023-05-31T19:51:38Z" 2023-05-31T19:51:38.537 INFO:teuthology.orchestra.run.smithi099.stdout:} 2023-05-31T19:51:38.538 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1685562631.516089_59.1 target= 2023-05-31T19:51:43.540 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-31T19:51:43.540 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-31T19:51:43.540 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-05-31T19:51:43.599 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2023-05-31T19:51:43.599 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2023-05-31T19:51:43.645 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T19:51:43.639+0000 7fa9d8d3c780 20 RGW-SYNC:meta: read sync status 2023-05-31T19:51:43.646 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T19:51:43.640+0000 7fa9d8d3c780 20 RGW-SYNC:meta: read sync status 2023-05-31T19:51:43.662 INFO:teuthology.orchestra.run.smithi099.stdout:{ 2023-05-31T19:51:43.662 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_status": { 2023-05-31T19:51:43.662 INFO:teuthology.orchestra.run.smithi099.stdout: "info": { 2023-05-31T19:51:43.662 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "sync", 2023-05-31T19:51:43.662 INFO:teuthology.orchestra.run.smithi099.stdout: "num_shards": 4, 2023-05-31T19:51:43.663 INFO:teuthology.orchestra.run.smithi099.stdout: "period": "2e6123d2-d695-4350-9a01-61a455bc782b", 2023-05-31T19:51:43.663 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2023-05-31T19:51:43.663 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T19:51:43.663 INFO:teuthology.orchestra.run.smithi099.stdout: "markers": [ 2023-05-31T19:51:43.663 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T19:51:43.663 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 0, 2023-05-31T19:51:43.663 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T19:51:43.664 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2023-05-31T19:51:43.664 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "", 2023-05-31T19:51:43.664 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T19:51:43.664 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 1, 2023-05-31T19:51:43.664 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T19:51:43.664 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "0.000000", 2023-05-31T19:51:43.664 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2023-05-31T19:51:43.665 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T19:51:43.665 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T19:51:43.665 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T19:51:43.665 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 1, 2023-05-31T19:51:43.665 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T19:51:43.665 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2023-05-31T19:51:43.665 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "", 2023-05-31T19:51:43.666 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T19:51:43.666 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2023-05-31T19:51:43.666 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T19:51:43.666 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "0.000000", 2023-05-31T19:51:43.666 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2023-05-31T19:51:43.666 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T19:51:43.666 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T19:51:43.666 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T19:51:43.667 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 2, 2023-05-31T19:51:43.667 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T19:51:43.667 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2023-05-31T19:51:43.667 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "", 2023-05-31T19:51:43.667 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T19:51:43.667 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2023-05-31T19:51:43.667 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T19:51:43.668 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "0.000000", 2023-05-31T19:51:43.668 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2023-05-31T19:51:43.668 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T19:51:43.668 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T19:51:43.668 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T19:51:43.669 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 3, 2023-05-31T19:51:43.669 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T19:51:43.669 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2023-05-31T19:51:43.669 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "", 2023-05-31T19:51:43.669 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T19:51:43.669 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2023-05-31T19:51:43.669 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T19:51:43.669 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "0.000000", 2023-05-31T19:51:43.670 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2023-05-31T19:51:43.670 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T19:51:43.670 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T19:51:43.670 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2023-05-31T19:51:43.670 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T19:51:43.670 INFO:teuthology.orchestra.run.smithi099.stdout: "full_sync": { 2023-05-31T19:51:43.670 INFO:teuthology.orchestra.run.smithi099.stdout: "total": 1, 2023-05-31T19:51:43.671 INFO:teuthology.orchestra.run.smithi099.stdout: "complete": 1 2023-05-31T19:51:43.671 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T19:51:43.671 INFO:teuthology.orchestra.run.smithi099.stdout: "current_time": "2023-05-31T19:51:43Z" 2023-05-31T19:51:43.671 INFO:teuthology.orchestra.run.smithi099.stdout:} 2023-05-31T19:51:43.672 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1685562631.516089_59.1 target= 2023-05-31T19:51:48.673 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-31T19:51:48.674 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-31T19:51:48.674 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-05-31T19:51:48.731 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2023-05-31T19:51:48.731 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2023-05-31T19:51:48.764 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T19:51:48.758+0000 7efe46db4780 20 RGW-SYNC:meta: read sync status 2023-05-31T19:51:48.765 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T19:51:48.759+0000 7efe46db4780 20 RGW-SYNC:meta: read sync status 2023-05-31T19:51:48.785 INFO:teuthology.orchestra.run.smithi099.stdout:{ 2023-05-31T19:51:48.785 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_status": { 2023-05-31T19:51:48.785 INFO:teuthology.orchestra.run.smithi099.stdout: "info": { 2023-05-31T19:51:48.785 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "sync", 2023-05-31T19:51:48.785 INFO:teuthology.orchestra.run.smithi099.stdout: "num_shards": 4, 2023-05-31T19:51:48.786 INFO:teuthology.orchestra.run.smithi099.stdout: "period": "2e6123d2-d695-4350-9a01-61a455bc782b", 2023-05-31T19:51:48.786 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2023-05-31T19:51:48.786 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T19:51:48.786 INFO:teuthology.orchestra.run.smithi099.stdout: "markers": [ 2023-05-31T19:51:48.786 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T19:51:48.786 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 0, 2023-05-31T19:51:48.786 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T19:51:48.787 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2023-05-31T19:51:48.787 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "", 2023-05-31T19:51:48.787 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T19:51:48.787 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 1, 2023-05-31T19:51:48.787 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T19:51:48.787 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "0.000000", 2023-05-31T19:51:48.788 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2023-05-31T19:51:48.788 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T19:51:48.788 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T19:51:48.788 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T19:51:48.788 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 1, 2023-05-31T19:51:48.788 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T19:51:48.788 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2023-05-31T19:51:48.789 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "", 2023-05-31T19:51:48.789 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T19:51:48.789 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2023-05-31T19:51:48.789 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T19:51:48.789 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "0.000000", 2023-05-31T19:51:48.790 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2023-05-31T19:51:48.790 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T19:51:48.790 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T19:51:48.790 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T19:51:48.790 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 2, 2023-05-31T19:51:48.790 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T19:51:48.790 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2023-05-31T19:51:48.790 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "", 2023-05-31T19:51:48.791 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T19:51:48.791 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2023-05-31T19:51:48.791 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T19:51:48.791 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "0.000000", 2023-05-31T19:51:48.791 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2023-05-31T19:51:48.791 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T19:51:48.793 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T19:51:48.793 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T19:51:48.793 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 3, 2023-05-31T19:51:48.793 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T19:51:48.793 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2023-05-31T19:51:48.793 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "", 2023-05-31T19:51:48.794 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T19:51:48.794 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2023-05-31T19:51:48.794 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T19:51:48.794 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "0.000000", 2023-05-31T19:51:48.794 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2023-05-31T19:51:48.794 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T19:51:48.794 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T19:51:48.795 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2023-05-31T19:51:48.795 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T19:51:48.795 INFO:teuthology.orchestra.run.smithi099.stdout: "full_sync": { 2023-05-31T19:51:48.795 INFO:teuthology.orchestra.run.smithi099.stdout: "total": 1, 2023-05-31T19:51:48.795 INFO:teuthology.orchestra.run.smithi099.stdout: "complete": 1 2023-05-31T19:51:48.795 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T19:51:48.795 INFO:teuthology.orchestra.run.smithi099.stdout: "current_time": "2023-05-31T19:51:48Z" 2023-05-31T19:51:48.795 INFO:teuthology.orchestra.run.smithi099.stdout:} 2023-05-31T19:51:48.797 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1685562631.516089_59.1 target= 2023-05-31T19:51:53.799 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-31T19:51:53.799 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-31T19:51:53.800 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-05-31T19:51:53.859 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2023-05-31T19:51:53.859 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2023-05-31T19:51:53.913 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T19:51:53.908+0000 7f2026006780 20 RGW-SYNC:meta: read sync status 2023-05-31T19:51:53.914 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T19:51:53.909+0000 7f2026006780 20 RGW-SYNC:meta: read sync status 2023-05-31T19:51:53.936 INFO:teuthology.orchestra.run.smithi099.stdout:{ 2023-05-31T19:51:53.936 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_status": { 2023-05-31T19:51:53.937 INFO:teuthology.orchestra.run.smithi099.stdout: "info": { 2023-05-31T19:51:53.937 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "sync", 2023-05-31T19:51:53.937 INFO:teuthology.orchestra.run.smithi099.stdout: "num_shards": 4, 2023-05-31T19:51:53.937 INFO:teuthology.orchestra.run.smithi099.stdout: "period": "2e6123d2-d695-4350-9a01-61a455bc782b", 2023-05-31T19:51:53.937 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2023-05-31T19:51:53.937 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T19:51:53.937 INFO:teuthology.orchestra.run.smithi099.stdout: "markers": [ 2023-05-31T19:51:53.938 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T19:51:53.938 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 0, 2023-05-31T19:51:53.938 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T19:51:53.938 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2023-05-31T19:51:53.938 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "", 2023-05-31T19:51:53.938 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T19:51:53.938 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 1, 2023-05-31T19:51:53.939 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T19:51:53.939 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "0.000000", 2023-05-31T19:51:53.939 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2023-05-31T19:51:53.939 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T19:51:53.939 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T19:51:53.939 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T19:51:53.939 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 1, 2023-05-31T19:51:53.939 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T19:51:53.940 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2023-05-31T19:51:53.940 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "", 2023-05-31T19:51:53.940 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T19:51:53.940 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2023-05-31T19:51:53.940 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T19:51:53.940 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "0.000000", 2023-05-31T19:51:53.940 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2023-05-31T19:51:53.941 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T19:51:53.941 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T19:51:53.941 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T19:51:53.941 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 2, 2023-05-31T19:51:53.941 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T19:51:53.941 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2023-05-31T19:51:53.941 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "", 2023-05-31T19:51:53.941 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T19:51:53.942 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2023-05-31T19:51:53.942 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T19:51:53.942 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "0.000000", 2023-05-31T19:51:53.942 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2023-05-31T19:51:53.942 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T19:51:53.942 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T19:51:53.942 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T19:51:53.942 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 3, 2023-05-31T19:51:53.943 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T19:51:53.943 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2023-05-31T19:51:53.943 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "", 2023-05-31T19:51:53.943 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T19:51:53.943 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2023-05-31T19:51:53.943 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T19:51:53.943 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "0.000000", 2023-05-31T19:51:53.943 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2023-05-31T19:51:53.944 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T19:51:53.944 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T19:51:53.944 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2023-05-31T19:51:53.944 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T19:51:53.944 INFO:teuthology.orchestra.run.smithi099.stdout: "full_sync": { 2023-05-31T19:51:53.944 INFO:teuthology.orchestra.run.smithi099.stdout: "total": 1, 2023-05-31T19:51:53.944 INFO:teuthology.orchestra.run.smithi099.stdout: "complete": 1 2023-05-31T19:51:53.945 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T19:51:53.945 INFO:teuthology.orchestra.run.smithi099.stdout: "current_time": "2023-05-31T19:51:53Z" 2023-05-31T19:51:53.945 INFO:teuthology.orchestra.run.smithi099.stdout:} 2023-05-31T19:51:53.946 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1685562631.516089_59.1 target= 2023-05-31T19:51:58.947 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-31T19:51:58.947 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-31T19:51:58.948 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-05-31T19:51:59.006 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2023-05-31T19:51:59.007 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2023-05-31T19:51:59.039 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T19:51:59.034+0000 7f428f72f780 20 RGW-SYNC:meta: read sync status 2023-05-31T19:51:59.039 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T19:51:59.035+0000 7f428f72f780 20 RGW-SYNC:meta: read sync status 2023-05-31T19:51:59.056 INFO:teuthology.orchestra.run.smithi099.stdout:{ 2023-05-31T19:51:59.056 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_status": { 2023-05-31T19:51:59.056 INFO:teuthology.orchestra.run.smithi099.stdout: "info": { 2023-05-31T19:51:59.057 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "sync", 2023-05-31T19:51:59.057 INFO:teuthology.orchestra.run.smithi099.stdout: "num_shards": 4, 2023-05-31T19:51:59.058 INFO:teuthology.orchestra.run.smithi099.stdout: "period": "2e6123d2-d695-4350-9a01-61a455bc782b", 2023-05-31T19:51:59.058 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2023-05-31T19:51:59.058 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T19:51:59.058 INFO:teuthology.orchestra.run.smithi099.stdout: "markers": [ 2023-05-31T19:51:59.058 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T19:51:59.058 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 0, 2023-05-31T19:51:59.059 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T19:51:59.059 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2023-05-31T19:51:59.059 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "", 2023-05-31T19:51:59.059 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T19:51:59.059 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 1, 2023-05-31T19:51:59.059 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T19:51:59.060 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "0.000000", 2023-05-31T19:51:59.060 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2023-05-31T19:51:59.060 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T19:51:59.060 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T19:51:59.060 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T19:51:59.060 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 1, 2023-05-31T19:51:59.060 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T19:51:59.061 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2023-05-31T19:51:59.061 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "", 2023-05-31T19:51:59.061 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T19:51:59.061 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2023-05-31T19:51:59.061 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T19:51:59.061 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "0.000000", 2023-05-31T19:51:59.061 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2023-05-31T19:51:59.062 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T19:51:59.062 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T19:51:59.062 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T19:51:59.062 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 2, 2023-05-31T19:51:59.062 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T19:51:59.062 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2023-05-31T19:51:59.062 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "", 2023-05-31T19:51:59.063 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T19:51:59.063 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2023-05-31T19:51:59.063 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T19:51:59.063 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "0.000000", 2023-05-31T19:51:59.063 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2023-05-31T19:51:59.063 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T19:51:59.063 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T19:51:59.063 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T19:51:59.064 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 3, 2023-05-31T19:51:59.064 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T19:51:59.064 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2023-05-31T19:51:59.064 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "", 2023-05-31T19:51:59.064 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T19:51:59.065 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2023-05-31T19:51:59.065 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T19:51:59.065 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "0.000000", 2023-05-31T19:51:59.065 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2023-05-31T19:51:59.065 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T19:51:59.065 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T19:51:59.065 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2023-05-31T19:51:59.066 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T19:51:59.066 INFO:teuthology.orchestra.run.smithi099.stdout: "full_sync": { 2023-05-31T19:51:59.066 INFO:teuthology.orchestra.run.smithi099.stdout: "total": 1, 2023-05-31T19:51:59.066 INFO:teuthology.orchestra.run.smithi099.stdout: "complete": 1 2023-05-31T19:51:59.066 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T19:51:59.067 INFO:teuthology.orchestra.run.smithi099.stdout: "current_time": "2023-05-31T19:51:59Z" 2023-05-31T19:51:59.067 INFO:teuthology.orchestra.run.smithi099.stdout:} 2023-05-31T19:51:59.067 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1685562631.516089_59.1 target= 2023-05-31T19:52:04.068 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-31T19:52:04.068 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-31T19:52:04.069 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-05-31T19:52:04.125 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2023-05-31T19:52:04.125 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2023-05-31T19:52:04.173 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T19:52:04.168+0000 7f9406b00780 20 RGW-SYNC:meta: read sync status 2023-05-31T19:52:04.174 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T19:52:04.169+0000 7f9406b00780 20 RGW-SYNC:meta: read sync status 2023-05-31T19:52:04.194 INFO:teuthology.orchestra.run.smithi099.stdout:{ 2023-05-31T19:52:04.194 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_status": { 2023-05-31T19:52:04.194 INFO:teuthology.orchestra.run.smithi099.stdout: "info": { 2023-05-31T19:52:04.195 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "sync", 2023-05-31T19:52:04.195 INFO:teuthology.orchestra.run.smithi099.stdout: "num_shards": 4, 2023-05-31T19:52:04.195 INFO:teuthology.orchestra.run.smithi099.stdout: "period": "2e6123d2-d695-4350-9a01-61a455bc782b", 2023-05-31T19:52:04.195 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2023-05-31T19:52:04.195 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T19:52:04.195 INFO:teuthology.orchestra.run.smithi099.stdout: "markers": [ 2023-05-31T19:52:04.195 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T19:52:04.195 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 0, 2023-05-31T19:52:04.196 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T19:52:04.196 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2023-05-31T19:52:04.196 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "", 2023-05-31T19:52:04.196 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T19:52:04.196 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 1, 2023-05-31T19:52:04.196 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T19:52:04.196 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "0.000000", 2023-05-31T19:52:04.197 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2023-05-31T19:52:04.197 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T19:52:04.197 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T19:52:04.197 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T19:52:04.197 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 1, 2023-05-31T19:52:04.197 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T19:52:04.197 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2023-05-31T19:52:04.198 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "", 2023-05-31T19:52:04.198 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T19:52:04.198 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2023-05-31T19:52:04.198 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T19:52:04.198 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "0.000000", 2023-05-31T19:52:04.198 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2023-05-31T19:52:04.198 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T19:52:04.198 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T19:52:04.199 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T19:52:04.199 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 2, 2023-05-31T19:52:04.199 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T19:52:04.199 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2023-05-31T19:52:04.199 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "", 2023-05-31T19:52:04.199 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T19:52:04.199 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2023-05-31T19:52:04.200 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T19:52:04.200 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "0.000000", 2023-05-31T19:52:04.200 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2023-05-31T19:52:04.200 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T19:52:04.200 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T19:52:04.200 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T19:52:04.200 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 3, 2023-05-31T19:52:04.200 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T19:52:04.201 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2023-05-31T19:52:04.201 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "", 2023-05-31T19:52:04.201 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T19:52:04.201 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2023-05-31T19:52:04.201 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T19:52:04.201 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "0.000000", 2023-05-31T19:52:04.201 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2023-05-31T19:52:04.201 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T19:52:04.202 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T19:52:04.202 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2023-05-31T19:52:04.202 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T19:52:04.202 INFO:teuthology.orchestra.run.smithi099.stdout: "full_sync": { 2023-05-31T19:52:04.202 INFO:teuthology.orchestra.run.smithi099.stdout: "total": 1, 2023-05-31T19:52:04.202 INFO:teuthology.orchestra.run.smithi099.stdout: "complete": 1 2023-05-31T19:52:04.202 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T19:52:04.203 INFO:teuthology.orchestra.run.smithi099.stdout: "current_time": "2023-05-31T19:52:04Z" 2023-05-31T19:52:04.203 INFO:teuthology.orchestra.run.smithi099.stdout:} 2023-05-31T19:52:04.204 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1685562631.516089_59.1 target= 2023-05-31T19:52:09.205 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-31T19:52:09.206 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-31T19:52:09.206 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-05-31T19:52:09.263 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2023-05-31T19:52:09.263 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2023-05-31T19:52:09.316 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T19:52:09.305+0000 7f4ab0269780 20 RGW-SYNC:meta: read sync status 2023-05-31T19:52:09.317 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T19:52:09.306+0000 7f4ab0269780 20 RGW-SYNC:meta: read sync status 2023-05-31T19:52:09.322 INFO:teuthology.orchestra.run.smithi099.stdout:{ 2023-05-31T19:52:09.323 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_status": { 2023-05-31T19:52:09.323 INFO:teuthology.orchestra.run.smithi099.stdout: "info": { 2023-05-31T19:52:09.323 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "sync", 2023-05-31T19:52:09.323 INFO:teuthology.orchestra.run.smithi099.stdout: "num_shards": 4, 2023-05-31T19:52:09.323 INFO:teuthology.orchestra.run.smithi099.stdout: "period": "2e6123d2-d695-4350-9a01-61a455bc782b", 2023-05-31T19:52:09.323 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2023-05-31T19:52:09.323 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T19:52:09.324 INFO:teuthology.orchestra.run.smithi099.stdout: "markers": [ 2023-05-31T19:52:09.324 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T19:52:09.324 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 0, 2023-05-31T19:52:09.324 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T19:52:09.324 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2023-05-31T19:52:09.324 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "", 2023-05-31T19:52:09.324 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T19:52:09.325 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 1, 2023-05-31T19:52:09.325 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T19:52:09.325 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "0.000000", 2023-05-31T19:52:09.325 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2023-05-31T19:52:09.325 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T19:52:09.325 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T19:52:09.325 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T19:52:09.325 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 1, 2023-05-31T19:52:09.326 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T19:52:09.326 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2023-05-31T19:52:09.326 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "", 2023-05-31T19:52:09.326 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T19:52:09.326 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2023-05-31T19:52:09.326 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T19:52:09.326 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "0.000000", 2023-05-31T19:52:09.327 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2023-05-31T19:52:09.327 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T19:52:09.327 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T19:52:09.327 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T19:52:09.327 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 2, 2023-05-31T19:52:09.327 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T19:52:09.327 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2023-05-31T19:52:09.328 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "", 2023-05-31T19:52:09.329 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T19:52:09.329 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2023-05-31T19:52:09.329 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T19:52:09.329 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "0.000000", 2023-05-31T19:52:09.329 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2023-05-31T19:52:09.329 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T19:52:09.330 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T19:52:09.330 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T19:52:09.330 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 3, 2023-05-31T19:52:09.330 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T19:52:09.330 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2023-05-31T19:52:09.330 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "", 2023-05-31T19:52:09.331 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T19:52:09.331 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2023-05-31T19:52:09.331 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T19:52:09.331 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "0.000000", 2023-05-31T19:52:09.332 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2023-05-31T19:52:09.332 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T19:52:09.332 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T19:52:09.332 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2023-05-31T19:52:09.332 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T19:52:09.332 INFO:teuthology.orchestra.run.smithi099.stdout: "full_sync": { 2023-05-31T19:52:09.332 INFO:teuthology.orchestra.run.smithi099.stdout: "total": 1, 2023-05-31T19:52:09.333 INFO:teuthology.orchestra.run.smithi099.stdout: "complete": 1 2023-05-31T19:52:09.333 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T19:52:09.333 INFO:teuthology.orchestra.run.smithi099.stdout: "current_time": "2023-05-31T19:52:09Z" 2023-05-31T19:52:09.333 INFO:teuthology.orchestra.run.smithi099.stdout:} 2023-05-31T19:52:09.334 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1685562631.516089_59.1 target= 2023-05-31T19:52:14.338 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-31T19:52:14.339 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-31T19:52:14.339 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-05-31T19:52:14.399 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2023-05-31T19:52:14.399 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2023-05-31T19:52:14.433 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T19:52:14.425+0000 7f3884903780 20 RGW-SYNC:meta: read sync status 2023-05-31T19:52:14.433 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T19:52:14.426+0000 7f3884903780 20 RGW-SYNC:meta: read sync status 2023-05-31T19:52:14.445 INFO:teuthology.orchestra.run.smithi099.stdout:{ 2023-05-31T19:52:14.446 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_status": { 2023-05-31T19:52:14.446 INFO:teuthology.orchestra.run.smithi099.stdout: "info": { 2023-05-31T19:52:14.446 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "sync", 2023-05-31T19:52:14.447 INFO:teuthology.orchestra.run.smithi099.stdout: "num_shards": 4, 2023-05-31T19:52:14.447 INFO:teuthology.orchestra.run.smithi099.stdout: "period": "2e6123d2-d695-4350-9a01-61a455bc782b", 2023-05-31T19:52:14.447 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2023-05-31T19:52:14.448 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T19:52:14.448 INFO:teuthology.orchestra.run.smithi099.stdout: "markers": [ 2023-05-31T19:52:14.448 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T19:52:14.449 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 0, 2023-05-31T19:52:14.449 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T19:52:14.449 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2023-05-31T19:52:14.450 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "", 2023-05-31T19:52:14.450 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T19:52:14.450 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 1, 2023-05-31T19:52:14.451 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T19:52:14.451 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "0.000000", 2023-05-31T19:52:14.452 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2023-05-31T19:52:14.452 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T19:52:14.452 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T19:52:14.453 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T19:52:14.453 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 1, 2023-05-31T19:52:14.453 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T19:52:14.453 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2023-05-31T19:52:14.453 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "", 2023-05-31T19:52:14.453 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T19:52:14.453 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2023-05-31T19:52:14.454 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T19:52:14.454 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "0.000000", 2023-05-31T19:52:14.454 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2023-05-31T19:52:14.455 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T19:52:14.455 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T19:52:14.456 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T19:52:14.456 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 2, 2023-05-31T19:52:14.456 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T19:52:14.457 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2023-05-31T19:52:14.457 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "", 2023-05-31T19:52:14.457 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T19:52:14.457 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2023-05-31T19:52:14.458 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T19:52:14.458 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "0.000000", 2023-05-31T19:52:14.458 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2023-05-31T19:52:14.458 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T19:52:14.459 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T19:52:14.459 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T19:52:14.459 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 3, 2023-05-31T19:52:14.459 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T19:52:14.460 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2023-05-31T19:52:14.460 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "", 2023-05-31T19:52:14.460 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T19:52:14.461 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2023-05-31T19:52:14.461 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T19:52:14.461 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "0.000000", 2023-05-31T19:52:14.461 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2023-05-31T19:52:14.462 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T19:52:14.462 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T19:52:14.462 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2023-05-31T19:52:14.462 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T19:52:14.463 INFO:teuthology.orchestra.run.smithi099.stdout: "full_sync": { 2023-05-31T19:52:14.464 INFO:teuthology.orchestra.run.smithi099.stdout: "total": 1, 2023-05-31T19:52:14.464 INFO:teuthology.orchestra.run.smithi099.stdout: "complete": 1 2023-05-31T19:52:14.464 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T19:52:14.465 INFO:teuthology.orchestra.run.smithi099.stdout: "current_time": "2023-05-31T19:52:14Z" 2023-05-31T19:52:14.465 INFO:teuthology.orchestra.run.smithi099.stdout:} 2023-05-31T19:52:14.468 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1685562631.516089_59.1 target= 2023-05-31T19:52:19.469 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-31T19:52:19.470 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-31T19:52:19.471 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-05-31T19:52:19.527 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2023-05-31T19:52:19.528 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2023-05-31T19:52:19.578 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T19:52:19.572+0000 7ff02929a780 20 RGW-SYNC:meta: read sync status 2023-05-31T19:52:19.579 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T19:52:19.573+0000 7ff02929a780 20 RGW-SYNC:meta: read sync status 2023-05-31T19:52:19.599 INFO:teuthology.orchestra.run.smithi099.stdout:{ 2023-05-31T19:52:19.599 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_status": { 2023-05-31T19:52:19.599 INFO:teuthology.orchestra.run.smithi099.stdout: "info": { 2023-05-31T19:52:19.599 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "sync", 2023-05-31T19:52:19.600 INFO:teuthology.orchestra.run.smithi099.stdout: "num_shards": 4, 2023-05-31T19:52:19.600 INFO:teuthology.orchestra.run.smithi099.stdout: "period": "2e6123d2-d695-4350-9a01-61a455bc782b", 2023-05-31T19:52:19.600 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2023-05-31T19:52:19.600 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T19:52:19.600 INFO:teuthology.orchestra.run.smithi099.stdout: "markers": [ 2023-05-31T19:52:19.600 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T19:52:19.601 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 0, 2023-05-31T19:52:19.601 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T19:52:19.601 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2023-05-31T19:52:19.601 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "", 2023-05-31T19:52:19.601 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T19:52:19.601 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 1, 2023-05-31T19:52:19.601 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T19:52:19.601 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "0.000000", 2023-05-31T19:52:19.602 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2023-05-31T19:52:19.602 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T19:52:19.602 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T19:52:19.602 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T19:52:19.603 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 1, 2023-05-31T19:52:19.603 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T19:52:19.603 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2023-05-31T19:52:19.603 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "", 2023-05-31T19:52:19.603 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T19:52:19.603 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2023-05-31T19:52:19.603 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T19:52:19.603 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "0.000000", 2023-05-31T19:52:19.604 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2023-05-31T19:52:19.604 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T19:52:19.604 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T19:52:19.604 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T19:52:19.605 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 2, 2023-05-31T19:52:19.605 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T19:52:19.605 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2023-05-31T19:52:19.605 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "", 2023-05-31T19:52:19.605 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T19:52:19.605 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2023-05-31T19:52:19.605 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T19:52:19.606 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "0.000000", 2023-05-31T19:52:19.606 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2023-05-31T19:52:19.606 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T19:52:19.606 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T19:52:19.606 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T19:52:19.606 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 3, 2023-05-31T19:52:19.607 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T19:52:19.607 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2023-05-31T19:52:19.607 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "", 2023-05-31T19:52:19.607 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T19:52:19.607 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2023-05-31T19:52:19.607 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T19:52:19.607 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "0.000000", 2023-05-31T19:52:19.608 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2023-05-31T19:52:19.608 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T19:52:19.608 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T19:52:19.608 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2023-05-31T19:52:19.608 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T19:52:19.608 INFO:teuthology.orchestra.run.smithi099.stdout: "full_sync": { 2023-05-31T19:52:19.608 INFO:teuthology.orchestra.run.smithi099.stdout: "total": 1, 2023-05-31T19:52:19.608 INFO:teuthology.orchestra.run.smithi099.stdout: "complete": 1 2023-05-31T19:52:19.609 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T19:52:19.609 INFO:teuthology.orchestra.run.smithi099.stdout: "current_time": "2023-05-31T19:52:19Z" 2023-05-31T19:52:19.609 INFO:teuthology.orchestra.run.smithi099.stdout:} 2023-05-31T19:52:19.610 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1685562631.516089_59.1 target= 2023-05-31T19:52:24.611 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-31T19:52:24.612 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-31T19:52:24.612 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-05-31T19:52:24.668 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2023-05-31T19:52:24.669 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2023-05-31T19:52:24.707 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T19:52:24.701+0000 7f19204ca780 20 RGW-SYNC:meta: read sync status 2023-05-31T19:52:24.708 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T19:52:24.702+0000 7f19204ca780 20 RGW-SYNC:meta: read sync status 2023-05-31T19:52:24.720 INFO:teuthology.orchestra.run.smithi099.stdout:{ 2023-05-31T19:52:24.720 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_status": { 2023-05-31T19:52:24.720 INFO:teuthology.orchestra.run.smithi099.stdout: "info": { 2023-05-31T19:52:24.720 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "sync", 2023-05-31T19:52:24.720 INFO:teuthology.orchestra.run.smithi099.stdout: "num_shards": 4, 2023-05-31T19:52:24.721 INFO:teuthology.orchestra.run.smithi099.stdout: "period": "2e6123d2-d695-4350-9a01-61a455bc782b", 2023-05-31T19:52:24.721 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2023-05-31T19:52:24.721 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T19:52:24.721 INFO:teuthology.orchestra.run.smithi099.stdout: "markers": [ 2023-05-31T19:52:24.721 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T19:52:24.721 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 0, 2023-05-31T19:52:24.721 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T19:52:24.722 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2023-05-31T19:52:24.722 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "", 2023-05-31T19:52:24.722 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T19:52:24.722 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 1, 2023-05-31T19:52:24.722 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T19:52:24.722 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "0.000000", 2023-05-31T19:52:24.722 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2023-05-31T19:52:24.723 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T19:52:24.723 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T19:52:24.723 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T19:52:24.723 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 1, 2023-05-31T19:52:24.723 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T19:52:24.723 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2023-05-31T19:52:24.723 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "", 2023-05-31T19:52:24.724 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T19:52:24.724 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2023-05-31T19:52:24.724 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T19:52:24.724 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "0.000000", 2023-05-31T19:52:24.724 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2023-05-31T19:52:24.724 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T19:52:24.724 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T19:52:24.724 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T19:52:24.725 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 2, 2023-05-31T19:52:24.725 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T19:52:24.725 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2023-05-31T19:52:24.725 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "", 2023-05-31T19:52:24.725 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T19:52:24.725 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2023-05-31T19:52:24.725 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T19:52:24.726 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "0.000000", 2023-05-31T19:52:24.726 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2023-05-31T19:52:24.726 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T19:52:24.726 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T19:52:24.726 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T19:52:24.726 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 3, 2023-05-31T19:52:24.726 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T19:52:24.726 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2023-05-31T19:52:24.727 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "", 2023-05-31T19:52:24.727 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T19:52:24.727 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2023-05-31T19:52:24.727 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T19:52:24.727 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "0.000000", 2023-05-31T19:52:24.727 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2023-05-31T19:52:24.727 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T19:52:24.727 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T19:52:24.728 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2023-05-31T19:52:24.728 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T19:52:24.728 INFO:teuthology.orchestra.run.smithi099.stdout: "full_sync": { 2023-05-31T19:52:24.728 INFO:teuthology.orchestra.run.smithi099.stdout: "total": 1, 2023-05-31T19:52:24.728 INFO:teuthology.orchestra.run.smithi099.stdout: "complete": 1 2023-05-31T19:52:24.728 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T19:52:24.728 INFO:teuthology.orchestra.run.smithi099.stdout: "current_time": "2023-05-31T19:52:24Z" 2023-05-31T19:52:24.728 INFO:teuthology.orchestra.run.smithi099.stdout:} 2023-05-31T19:52:24.729 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1685562631.516089_59.1 target= 2023-05-31T19:52:29.731 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-31T19:52:29.731 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-31T19:52:29.731 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-05-31T19:52:29.788 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2023-05-31T19:52:29.789 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2023-05-31T19:52:29.842 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T19:52:29.836+0000 7f2be10a2780 20 RGW-SYNC:meta: read sync status 2023-05-31T19:52:29.843 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T19:52:29.837+0000 7f2be10a2780 20 RGW-SYNC:meta: read sync status 2023-05-31T19:52:29.865 INFO:teuthology.orchestra.run.smithi099.stdout:{ 2023-05-31T19:52:29.865 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_status": { 2023-05-31T19:52:29.866 INFO:teuthology.orchestra.run.smithi099.stdout: "info": { 2023-05-31T19:52:29.866 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "sync", 2023-05-31T19:52:29.866 INFO:teuthology.orchestra.run.smithi099.stdout: "num_shards": 4, 2023-05-31T19:52:29.866 INFO:teuthology.orchestra.run.smithi099.stdout: "period": "2e6123d2-d695-4350-9a01-61a455bc782b", 2023-05-31T19:52:29.867 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2023-05-31T19:52:29.867 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T19:52:29.867 INFO:teuthology.orchestra.run.smithi099.stdout: "markers": [ 2023-05-31T19:52:29.867 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T19:52:29.868 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 0, 2023-05-31T19:52:29.868 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T19:52:29.868 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2023-05-31T19:52:29.868 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1685562631.516089_59.1", 2023-05-31T19:52:29.869 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T19:52:29.869 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 1, 2023-05-31T19:52:29.869 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T19:52:29.870 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2023-05-31T19:50:31.516089Z", 2023-05-31T19:52:29.870 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2023-05-31T19:52:29.870 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T19:52:29.870 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T19:52:29.871 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T19:52:29.871 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 1, 2023-05-31T19:52:29.871 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T19:52:29.872 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2023-05-31T19:52:29.872 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "", 2023-05-31T19:52:29.872 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T19:52:29.872 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2023-05-31T19:52:29.873 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T19:52:29.873 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "0.000000", 2023-05-31T19:52:29.873 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2023-05-31T19:52:29.873 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T19:52:29.874 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T19:52:29.874 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T19:52:29.874 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 2, 2023-05-31T19:52:29.875 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T19:52:29.875 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2023-05-31T19:52:29.875 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "", 2023-05-31T19:52:29.875 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T19:52:29.876 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2023-05-31T19:52:29.876 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T19:52:29.876 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "0.000000", 2023-05-31T19:52:29.877 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2023-05-31T19:52:29.877 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T19:52:29.877 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T19:52:29.877 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T19:52:29.878 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 3, 2023-05-31T19:52:29.878 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T19:52:29.878 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2023-05-31T19:52:29.878 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "", 2023-05-31T19:52:29.879 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T19:52:29.879 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2023-05-31T19:52:29.879 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T19:52:29.879 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "0.000000", 2023-05-31T19:52:29.880 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2023-05-31T19:52:29.880 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T19:52:29.880 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T19:52:29.880 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2023-05-31T19:52:29.880 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T19:52:29.881 INFO:teuthology.orchestra.run.smithi099.stdout: "full_sync": { 2023-05-31T19:52:29.881 INFO:teuthology.orchestra.run.smithi099.stdout: "total": 1, 2023-05-31T19:52:29.881 INFO:teuthology.orchestra.run.smithi099.stdout: "complete": 1 2023-05-31T19:52:29.881 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T19:52:29.882 INFO:teuthology.orchestra.run.smithi099.stdout: "current_time": "2023-05-31T19:52:29Z" 2023-05-31T19:52:29.882 INFO:teuthology.orchestra.run.smithi099.stdout:} 2023-05-31T19:52:29.883 INFO:rgw_multi.tests:finish meta checkpoint for zone=a2 2023-05-31T19:52:29.884 INFO:rgw_multi.tests:starting meta checkpoint for zone=b1 2023-05-31T19:52:29.884 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'b1', '--rgw-zonegroup', 'b', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-31T19:52:29.885 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'b1', '--rgw-zonegroup', 'b', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-31T19:52:29.885 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone b1 --rgw-zonegroup b --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-05-31T19:52:29.942 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-05-31T19:52:29.943 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-05-31T19:52:29.991 INFO:teuthology.orchestra.run.smithi112.stderr:2023-05-31T19:52:29.984+0000 7f9fcd2da780 20 RGW-SYNC:meta: read sync status 2023-05-31T19:52:29.991 INFO:teuthology.orchestra.run.smithi112.stderr:2023-05-31T19:52:29.985+0000 7f9fcd2da780 20 RGW-SYNC:meta: read sync status 2023-05-31T19:52:30.009 INFO:teuthology.orchestra.run.smithi112.stdout:{ 2023-05-31T19:52:30.009 INFO:teuthology.orchestra.run.smithi112.stdout: "sync_status": { 2023-05-31T19:52:30.010 INFO:teuthology.orchestra.run.smithi112.stdout: "info": { 2023-05-31T19:52:30.010 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "sync", 2023-05-31T19:52:30.010 INFO:teuthology.orchestra.run.smithi112.stdout: "num_shards": 4, 2023-05-31T19:52:30.010 INFO:teuthology.orchestra.run.smithi112.stdout: "period": "2e6123d2-d695-4350-9a01-61a455bc782b", 2023-05-31T19:52:30.011 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-05-31T19:52:30.011 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-05-31T19:52:30.011 INFO:teuthology.orchestra.run.smithi112.stdout: "markers": [ 2023-05-31T19:52:30.012 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-05-31T19:52:30.012 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 0, 2023-05-31T19:52:30.012 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-05-31T19:52:30.012 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-05-31T19:52:30.013 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1685562631.516089_59.1", 2023-05-31T19:52:30.013 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-05-31T19:52:30.013 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 1, 2023-05-31T19:52:30.013 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-05-31T19:52:30.014 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-05-31T19:50:31.516089Z", 2023-05-31T19:52:30.014 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-05-31T19:52:30.014 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-05-31T19:52:30.014 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-05-31T19:52:30.015 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-05-31T19:52:30.015 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 1, 2023-05-31T19:52:30.015 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-05-31T19:52:30.016 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-05-31T19:52:30.016 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "", 2023-05-31T19:52:30.017 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-05-31T19:52:30.017 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-05-31T19:52:30.017 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-05-31T19:52:30.018 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "0.000000", 2023-05-31T19:52:30.018 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-05-31T19:52:30.018 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-05-31T19:52:30.018 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-05-31T19:52:30.018 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-05-31T19:52:30.018 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 2, 2023-05-31T19:52:30.018 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-05-31T19:52:30.019 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-05-31T19:52:30.019 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "", 2023-05-31T19:52:30.019 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-05-31T19:52:30.019 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-05-31T19:52:30.019 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-05-31T19:52:30.019 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "0.000000", 2023-05-31T19:52:30.019 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-05-31T19:52:30.019 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-05-31T19:52:30.020 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-05-31T19:52:30.020 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-05-31T19:52:30.020 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 3, 2023-05-31T19:52:30.020 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-05-31T19:52:30.020 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-05-31T19:52:30.020 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "", 2023-05-31T19:52:30.021 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-05-31T19:52:30.021 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-05-31T19:52:30.021 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-05-31T19:52:30.021 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "0.000000", 2023-05-31T19:52:30.021 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-05-31T19:52:30.021 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-05-31T19:52:30.021 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-05-31T19:52:30.021 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2023-05-31T19:52:30.022 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-05-31T19:52:30.022 INFO:teuthology.orchestra.run.smithi112.stdout: "full_sync": { 2023-05-31T19:52:30.022 INFO:teuthology.orchestra.run.smithi112.stdout: "total": 1, 2023-05-31T19:52:30.022 INFO:teuthology.orchestra.run.smithi112.stdout: "complete": 1 2023-05-31T19:52:30.022 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-05-31T19:52:30.022 INFO:teuthology.orchestra.run.smithi112.stdout: "current_time": "2023-05-31T19:52:29Z" 2023-05-31T19:52:30.022 INFO:teuthology.orchestra.run.smithi112.stdout:} 2023-05-31T19:52:30.024 INFO:rgw_multi.tests:finish meta checkpoint for zone=b1 2023-05-31T19:52:30.024 INFO:rgw_multi.tests:starting meta checkpoint for zone=b2 2023-05-31T19:52:30.026 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'b2', '--rgw-zonegroup', 'b', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-31T19:52:30.026 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'b2', '--rgw-zonegroup', 'b', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-31T19:52:30.027 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone b2 --rgw-zonegroup b --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-05-31T19:52:30.079 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-05-31T19:52:30.079 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-05-31T19:52:30.120 INFO:teuthology.orchestra.run.smithi112.stderr:2023-05-31T19:52:30.115+0000 7feb78696780 20 RGW-SYNC:meta: read sync status 2023-05-31T19:52:30.121 INFO:teuthology.orchestra.run.smithi112.stderr:2023-05-31T19:52:30.116+0000 7feb78696780 20 RGW-SYNC:meta: read sync status 2023-05-31T19:52:30.136 INFO:teuthology.orchestra.run.smithi112.stdout:{ 2023-05-31T19:52:30.137 INFO:teuthology.orchestra.run.smithi112.stdout: "sync_status": { 2023-05-31T19:52:30.137 INFO:teuthology.orchestra.run.smithi112.stdout: "info": { 2023-05-31T19:52:30.137 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "sync", 2023-05-31T19:52:30.137 INFO:teuthology.orchestra.run.smithi112.stdout: "num_shards": 4, 2023-05-31T19:52:30.137 INFO:teuthology.orchestra.run.smithi112.stdout: "period": "2e6123d2-d695-4350-9a01-61a455bc782b", 2023-05-31T19:52:30.138 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-05-31T19:52:30.138 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-05-31T19:52:30.138 INFO:teuthology.orchestra.run.smithi112.stdout: "markers": [ 2023-05-31T19:52:30.139 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-05-31T19:52:30.139 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 0, 2023-05-31T19:52:30.139 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-05-31T19:52:30.139 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-05-31T19:52:30.140 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1685562631.516089_59.1", 2023-05-31T19:52:30.140 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-05-31T19:52:30.140 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 1, 2023-05-31T19:52:30.140 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-05-31T19:52:30.141 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-05-31T19:50:31.516089Z", 2023-05-31T19:52:30.141 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-05-31T19:52:30.141 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-05-31T19:52:30.142 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-05-31T19:52:30.142 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-05-31T19:52:30.142 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 1, 2023-05-31T19:52:30.143 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-05-31T19:52:30.143 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-05-31T19:52:30.143 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "", 2023-05-31T19:52:30.143 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-05-31T19:52:30.144 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-05-31T19:52:30.144 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-05-31T19:52:30.144 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "0.000000", 2023-05-31T19:52:30.145 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-05-31T19:52:30.145 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-05-31T19:52:30.145 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-05-31T19:52:30.145 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-05-31T19:52:30.146 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 2, 2023-05-31T19:52:30.146 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-05-31T19:52:30.146 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-05-31T19:52:30.147 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "", 2023-05-31T19:52:30.147 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-05-31T19:52:30.147 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-05-31T19:52:30.147 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-05-31T19:52:30.148 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "0.000000", 2023-05-31T19:52:30.148 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-05-31T19:52:30.148 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-05-31T19:52:30.149 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-05-31T19:52:30.149 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-05-31T19:52:30.149 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 3, 2023-05-31T19:52:30.149 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-05-31T19:52:30.149 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-05-31T19:52:30.149 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "", 2023-05-31T19:52:30.150 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-05-31T19:52:30.150 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-05-31T19:52:30.150 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-05-31T19:52:30.150 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "0.000000", 2023-05-31T19:52:30.150 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-05-31T19:52:30.151 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-05-31T19:52:30.151 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-05-31T19:52:30.151 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2023-05-31T19:52:30.151 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-05-31T19:52:30.151 INFO:teuthology.orchestra.run.smithi112.stdout: "full_sync": { 2023-05-31T19:52:30.151 INFO:teuthology.orchestra.run.smithi112.stdout: "total": 1, 2023-05-31T19:52:30.151 INFO:teuthology.orchestra.run.smithi112.stdout: "complete": 1 2023-05-31T19:52:30.152 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-05-31T19:52:30.152 INFO:teuthology.orchestra.run.smithi112.stdout: "current_time": "2023-05-31T19:52:30Z" 2023-05-31T19:52:30.152 INFO:teuthology.orchestra.run.smithi112.stdout:} 2023-05-31T19:52:30.153 INFO:rgw_multi.tests:finish meta checkpoint for zone=b2 2023-05-31T19:52:30.154 INFO:rgw_multi.tests:meta checkpoint 2023-05-31T19:52:30.154 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-31T19:52:30.154 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-31T19:52:30.155 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-05-31T19:52:30.206 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2023-05-31T19:52:30.207 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2023-05-31T19:52:30.252 INFO:teuthology.orchestra.run.smithi099.stderr:No --period given, using current period=2e6123d2-d695-4350-9a01-61a455bc782b 2023-05-31T19:52:30.263 INFO:teuthology.orchestra.run.smithi099.stdout:[ 2023-05-31T19:52:30.264 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T19:52:30.264 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1685562631.516089_59.1", 2023-05-31T19:52:30.264 INFO:teuthology.orchestra.run.smithi099.stdout: "last_update": "2023-05-31T19:50:31.516089Z" 2023-05-31T19:52:30.265 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T19:52:30.265 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T19:52:30.265 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "", 2023-05-31T19:52:30.266 INFO:teuthology.orchestra.run.smithi099.stdout: "last_update": "0.000000" 2023-05-31T19:52:30.266 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T19:52:30.266 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T19:52:30.266 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "", 2023-05-31T19:52:30.267 INFO:teuthology.orchestra.run.smithi099.stdout: "last_update": "0.000000" 2023-05-31T19:52:30.267 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T19:52:30.267 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T19:52:30.267 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "", 2023-05-31T19:52:30.268 INFO:teuthology.orchestra.run.smithi099.stdout: "last_update": "0.000000" 2023-05-31T19:52:30.268 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T19:52:30.268 INFO:teuthology.orchestra.run.smithi099.stdout:] 2023-05-31T19:52:30.270 INFO:rgw_multi.tests:starting meta checkpoint for zone=a2 2023-05-31T19:52:30.271 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-31T19:52:30.272 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-31T19:52:30.272 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-05-31T19:52:30.321 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2023-05-31T19:52:30.322 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2023-05-31T19:52:30.352 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T19:52:30.346+0000 7fcd59791780 20 RGW-SYNC:meta: read sync status 2023-05-31T19:52:30.352 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T19:52:30.347+0000 7fcd59791780 20 RGW-SYNC:meta: read sync status 2023-05-31T19:52:30.364 INFO:teuthology.orchestra.run.smithi099.stdout:{ 2023-05-31T19:52:30.364 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_status": { 2023-05-31T19:52:30.364 INFO:teuthology.orchestra.run.smithi099.stdout: "info": { 2023-05-31T19:52:30.364 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "sync", 2023-05-31T19:52:30.365 INFO:teuthology.orchestra.run.smithi099.stdout: "num_shards": 4, 2023-05-31T19:52:30.365 INFO:teuthology.orchestra.run.smithi099.stdout: "period": "2e6123d2-d695-4350-9a01-61a455bc782b", 2023-05-31T19:52:30.365 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2023-05-31T19:52:30.366 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T19:52:30.366 INFO:teuthology.orchestra.run.smithi099.stdout: "markers": [ 2023-05-31T19:52:30.366 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T19:52:30.366 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 0, 2023-05-31T19:52:30.367 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T19:52:30.367 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2023-05-31T19:52:30.367 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1685562631.516089_59.1", 2023-05-31T19:52:30.367 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T19:52:30.368 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 1, 2023-05-31T19:52:30.368 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T19:52:30.368 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2023-05-31T19:50:31.516089Z", 2023-05-31T19:52:30.369 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2023-05-31T19:52:30.369 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T19:52:30.369 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T19:52:30.369 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T19:52:30.370 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 1, 2023-05-31T19:52:30.370 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T19:52:30.370 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2023-05-31T19:52:30.370 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "", 2023-05-31T19:52:30.371 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T19:52:30.371 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2023-05-31T19:52:30.371 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T19:52:30.371 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "0.000000", 2023-05-31T19:52:30.372 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2023-05-31T19:52:30.372 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T19:52:30.372 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T19:52:30.373 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T19:52:30.373 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 2, 2023-05-31T19:52:30.373 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T19:52:30.373 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2023-05-31T19:52:30.374 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "", 2023-05-31T19:52:30.374 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T19:52:30.374 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2023-05-31T19:52:30.374 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T19:52:30.375 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "0.000000", 2023-05-31T19:52:30.375 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2023-05-31T19:52:30.375 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T19:52:30.375 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T19:52:30.376 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T19:52:30.376 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 3, 2023-05-31T19:52:30.376 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T19:52:30.377 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2023-05-31T19:52:30.377 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "", 2023-05-31T19:52:30.377 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T19:52:30.377 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2023-05-31T19:52:30.378 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T19:52:30.378 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "0.000000", 2023-05-31T19:52:30.378 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2023-05-31T19:52:30.378 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T19:52:30.379 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T19:52:30.379 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2023-05-31T19:52:30.379 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T19:52:30.379 INFO:teuthology.orchestra.run.smithi099.stdout: "full_sync": { 2023-05-31T19:52:30.380 INFO:teuthology.orchestra.run.smithi099.stdout: "total": 1, 2023-05-31T19:52:30.380 INFO:teuthology.orchestra.run.smithi099.stdout: "complete": 1 2023-05-31T19:52:30.380 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T19:52:30.381 INFO:teuthology.orchestra.run.smithi099.stdout: "current_time": "2023-05-31T19:52:30Z" 2023-05-31T19:52:30.381 INFO:teuthology.orchestra.run.smithi099.stdout:} 2023-05-31T19:52:30.383 INFO:rgw_multi.tests:finish meta checkpoint for zone=a2 2023-05-31T19:52:30.384 INFO:rgw_multi.tests:starting meta checkpoint for zone=b1 2023-05-31T19:52:30.384 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'b1', '--rgw-zonegroup', 'b', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-31T19:52:30.385 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'b1', '--rgw-zonegroup', 'b', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-31T19:52:30.386 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone b1 --rgw-zonegroup b --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-05-31T19:52:30.436 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-05-31T19:52:30.436 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-05-31T19:52:30.466 INFO:teuthology.orchestra.run.smithi112.stderr:2023-05-31T19:52:30.460+0000 7f9af5f00780 20 RGW-SYNC:meta: read sync status 2023-05-31T19:52:30.467 INFO:teuthology.orchestra.run.smithi112.stderr:2023-05-31T19:52:30.461+0000 7f9af5f00780 20 RGW-SYNC:meta: read sync status 2023-05-31T19:52:30.478 INFO:teuthology.orchestra.run.smithi112.stdout:{ 2023-05-31T19:52:30.478 INFO:teuthology.orchestra.run.smithi112.stdout: "sync_status": { 2023-05-31T19:52:30.479 INFO:teuthology.orchestra.run.smithi112.stdout: "info": { 2023-05-31T19:52:30.479 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "sync", 2023-05-31T19:52:30.479 INFO:teuthology.orchestra.run.smithi112.stdout: "num_shards": 4, 2023-05-31T19:52:30.479 INFO:teuthology.orchestra.run.smithi112.stdout: "period": "2e6123d2-d695-4350-9a01-61a455bc782b", 2023-05-31T19:52:30.480 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-05-31T19:52:30.480 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-05-31T19:52:30.480 INFO:teuthology.orchestra.run.smithi112.stdout: "markers": [ 2023-05-31T19:52:30.481 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-05-31T19:52:30.481 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 0, 2023-05-31T19:52:30.481 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-05-31T19:52:30.481 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-05-31T19:52:30.482 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1685562631.516089_59.1", 2023-05-31T19:52:30.482 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-05-31T19:52:30.482 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 1, 2023-05-31T19:52:30.482 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-05-31T19:52:30.483 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-05-31T19:50:31.516089Z", 2023-05-31T19:52:30.483 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-05-31T19:52:30.483 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-05-31T19:52:30.484 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-05-31T19:52:30.484 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-05-31T19:52:30.484 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 1, 2023-05-31T19:52:30.484 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-05-31T19:52:30.485 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-05-31T19:52:30.485 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "", 2023-05-31T19:52:30.485 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-05-31T19:52:30.485 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-05-31T19:52:30.486 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-05-31T19:52:30.486 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "0.000000", 2023-05-31T19:52:30.486 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-05-31T19:52:30.487 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-05-31T19:52:30.487 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-05-31T19:52:30.487 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-05-31T19:52:30.487 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 2, 2023-05-31T19:52:30.488 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-05-31T19:52:30.488 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-05-31T19:52:30.488 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "", 2023-05-31T19:52:30.488 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-05-31T19:52:30.488 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-05-31T19:52:30.488 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-05-31T19:52:30.489 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "0.000000", 2023-05-31T19:52:30.489 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-05-31T19:52:30.489 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-05-31T19:52:30.489 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-05-31T19:52:30.489 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-05-31T19:52:30.489 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 3, 2023-05-31T19:52:30.489 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-05-31T19:52:30.489 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-05-31T19:52:30.490 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "", 2023-05-31T19:52:30.490 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-05-31T19:52:30.490 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-05-31T19:52:30.490 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-05-31T19:52:30.490 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "0.000000", 2023-05-31T19:52:30.490 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-05-31T19:52:30.490 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-05-31T19:52:30.491 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-05-31T19:52:30.491 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2023-05-31T19:52:30.491 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-05-31T19:52:30.491 INFO:teuthology.orchestra.run.smithi112.stdout: "full_sync": { 2023-05-31T19:52:30.491 INFO:teuthology.orchestra.run.smithi112.stdout: "total": 1, 2023-05-31T19:52:30.491 INFO:teuthology.orchestra.run.smithi112.stdout: "complete": 1 2023-05-31T19:52:30.492 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-05-31T19:52:30.492 INFO:teuthology.orchestra.run.smithi112.stdout: "current_time": "2023-05-31T19:52:30Z" 2023-05-31T19:52:30.492 INFO:teuthology.orchestra.run.smithi112.stdout:} 2023-05-31T19:52:30.493 INFO:rgw_multi.tests:finish meta checkpoint for zone=b1 2023-05-31T19:52:30.493 INFO:rgw_multi.tests:starting meta checkpoint for zone=b2 2023-05-31T19:52:30.493 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'b2', '--rgw-zonegroup', 'b', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-31T19:52:30.494 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'b2', '--rgw-zonegroup', 'b', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-31T19:52:30.494 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone b2 --rgw-zonegroup b --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-05-31T19:52:30.543 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-05-31T19:52:30.544 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-05-31T19:52:30.584 INFO:teuthology.orchestra.run.smithi112.stderr:2023-05-31T19:52:30.578+0000 7fc8722ed780 20 RGW-SYNC:meta: read sync status 2023-05-31T19:52:30.585 INFO:teuthology.orchestra.run.smithi112.stderr:2023-05-31T19:52:30.579+0000 7fc8722ed780 20 RGW-SYNC:meta: read sync status 2023-05-31T19:52:30.598 INFO:teuthology.orchestra.run.smithi112.stdout:{ 2023-05-31T19:52:30.598 INFO:teuthology.orchestra.run.smithi112.stdout: "sync_status": { 2023-05-31T19:52:30.598 INFO:teuthology.orchestra.run.smithi112.stdout: "info": { 2023-05-31T19:52:30.598 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "sync", 2023-05-31T19:52:30.598 INFO:teuthology.orchestra.run.smithi112.stdout: "num_shards": 4, 2023-05-31T19:52:30.599 INFO:teuthology.orchestra.run.smithi112.stdout: "period": "2e6123d2-d695-4350-9a01-61a455bc782b", 2023-05-31T19:52:30.599 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-05-31T19:52:30.599 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-05-31T19:52:30.599 INFO:teuthology.orchestra.run.smithi112.stdout: "markers": [ 2023-05-31T19:52:30.599 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-05-31T19:52:30.600 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 0, 2023-05-31T19:52:30.600 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-05-31T19:52:30.600 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-05-31T19:52:30.600 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1685562631.516089_59.1", 2023-05-31T19:52:30.600 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-05-31T19:52:30.600 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 1, 2023-05-31T19:52:30.601 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-05-31T19:52:30.601 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-05-31T19:50:31.516089Z", 2023-05-31T19:52:30.601 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-05-31T19:52:30.601 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-05-31T19:52:30.601 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-05-31T19:52:30.601 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-05-31T19:52:30.601 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 1, 2023-05-31T19:52:30.602 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-05-31T19:52:30.602 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-05-31T19:52:30.602 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "", 2023-05-31T19:52:30.602 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-05-31T19:52:30.602 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-05-31T19:52:30.603 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-05-31T19:52:30.603 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "0.000000", 2023-05-31T19:52:30.603 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-05-31T19:52:30.603 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-05-31T19:52:30.604 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-05-31T19:52:30.604 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-05-31T19:52:30.604 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 2, 2023-05-31T19:52:30.604 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-05-31T19:52:30.605 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-05-31T19:52:30.605 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "", 2023-05-31T19:52:30.606 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-05-31T19:52:30.606 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-05-31T19:52:30.606 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-05-31T19:52:30.607 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "0.000000", 2023-05-31T19:52:30.607 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-05-31T19:52:30.607 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-05-31T19:52:30.607 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-05-31T19:52:30.607 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-05-31T19:52:30.608 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 3, 2023-05-31T19:52:30.608 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-05-31T19:52:30.608 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-05-31T19:52:30.608 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "", 2023-05-31T19:52:30.608 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-05-31T19:52:30.609 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-05-31T19:52:30.609 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-05-31T19:52:30.609 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "0.000000", 2023-05-31T19:52:30.609 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-05-31T19:52:30.610 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-05-31T19:52:30.610 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-05-31T19:52:30.610 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2023-05-31T19:52:30.610 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-05-31T19:52:30.610 INFO:teuthology.orchestra.run.smithi112.stdout: "full_sync": { 2023-05-31T19:52:30.610 INFO:teuthology.orchestra.run.smithi112.stdout: "total": 1, 2023-05-31T19:52:30.611 INFO:teuthology.orchestra.run.smithi112.stdout: "complete": 1 2023-05-31T19:52:30.611 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-05-31T19:52:30.611 INFO:teuthology.orchestra.run.smithi112.stdout: "current_time": "2023-05-31T19:52:30Z" 2023-05-31T19:52:30.611 INFO:teuthology.orchestra.run.smithi112.stdout:} 2023-05-31T19:52:30.612 INFO:rgw_multi.tests:finish meta checkpoint for zone=b2 2023-05-31T19:52:30.612 INFO:tasks.rgw_multisite_tests:running rgw multisite tests on '/home/teuthworker/src/github.com_ceph_ceph-c_2870bf456d3115490c153abce9d47ac5c043b65e/qa/../src/test/rgw/rgw_multi' with args=['tests.py'] 2023-05-31T19:52:30.639 DEBUG:boto:Using access key provided by client. 2023-05-31T19:52:30.641 DEBUG:boto:Using secret key provided by client. 2023-05-31T19:52:30.642 DEBUG:boto:Using access key provided by client. 2023-05-31T19:52:30.643 DEBUG:boto:Using secret key provided by client. 2023-05-31T19:52:30.644 DEBUG:boto:Using access key provided by client. 2023-05-31T19:52:30.645 DEBUG:boto:Using secret key provided by client. 2023-05-31T19:52:30.645 DEBUG:boto:Using access key provided by client. 2023-05-31T19:52:30.646 DEBUG:boto:Using secret key provided by client. 2023-05-31T19:52:30.647 INFO:rgw_multi.tests:create bucket zone=a1 name=pklxkd-1 2023-05-31T19:52:30.647 DEBUG:boto:path=/pklxkd-1/ 2023-05-31T19:52:30.647 DEBUG:boto:auth_path=/pklxkd-1/ 2023-05-31T19:52:30.648 DEBUG:boto:Method: PUT 2023-05-31T19:52:30.648 DEBUG:boto:Path: /pklxkd-1/ 2023-05-31T19:52:30.648 DEBUG:boto:Data: 2023-05-31T19:52:30.649 DEBUG:boto:Headers: {} 2023-05-31T19:52:30.649 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T19:52:30.649 DEBUG:boto:Port: 8000 2023-05-31T19:52:30.650 DEBUG:boto:Params: {} 2023-05-31T19:52:30.650 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2023-05-31T19:52:30.651 DEBUG:boto:Token: None 2023-05-31T19:52:30.651 DEBUG:boto:StringToSign: PUT Wed, 31 May 2023 19:52:30 GMT /pklxkd-1/ 2023-05-31T19:52:30.652 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:pxOX2AteHjb0ZkccW8IxljZ/J8M= 2023-05-31T19:52:30.652 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 19:52:30 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:pxOX2AteHjb0ZkccW8IxljZ/J8M=', 'Content-Length': '0'} 2023-05-31T19:52:32.938 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000aa6655949f627666-006477a57e-110e-a1'), ('Content-Length', '0'), ('Date', 'Wed, 31 May 2023 19:52:32 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T19:52:32.939 INFO:rgw_multi.tests:create bucket zone=a2 name=pklxkd-2 2023-05-31T19:52:32.939 DEBUG:boto:path=/pklxkd-2/ 2023-05-31T19:52:32.940 DEBUG:boto:auth_path=/pklxkd-2/ 2023-05-31T19:52:32.940 DEBUG:boto:Method: PUT 2023-05-31T19:52:32.940 DEBUG:boto:Path: /pklxkd-2/ 2023-05-31T19:52:32.941 DEBUG:boto:Data: 2023-05-31T19:52:32.941 DEBUG:boto:Headers: {} 2023-05-31T19:52:32.941 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T19:52:32.942 DEBUG:boto:Port: 8001 2023-05-31T19:52:32.942 DEBUG:boto:Params: {} 2023-05-31T19:52:32.942 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8001} 2023-05-31T19:52:32.943 DEBUG:boto:Token: None 2023-05-31T19:52:32.943 DEBUG:boto:StringToSign: PUT Wed, 31 May 2023 19:52:32 GMT /pklxkd-2/ 2023-05-31T19:52:32.944 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:MRwmHEiymoT3UOdpGZw2pPI8oXg= 2023-05-31T19:52:32.944 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 19:52:32 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:MRwmHEiymoT3UOdpGZw2pPI8oXg=', 'Content-Length': '0'} 2023-05-31T19:52:35.950 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx000006d597b5365fb190b-006477a580-1112-a2'), ('Content-Length', '0'), ('Date', 'Wed, 31 May 2023 19:52:35 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T19:52:35.951 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-31T19:52:35.952 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-31T19:52:35.952 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-05-31T19:52:36.007 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2023-05-31T19:52:36.008 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2023-05-31T19:52:36.041 INFO:teuthology.orchestra.run.smithi099.stderr:No --period given, using current period=2e6123d2-d695-4350-9a01-61a455bc782b 2023-05-31T19:52:36.053 INFO:teuthology.orchestra.run.smithi099.stdout:[ 2023-05-31T19:52:36.053 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T19:52:36.053 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1685562631.516089_59.1", 2023-05-31T19:52:36.053 INFO:teuthology.orchestra.run.smithi099.stdout: "last_update": "2023-05-31T19:50:31.516089Z" 2023-05-31T19:52:36.054 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T19:52:36.054 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T19:52:36.054 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "", 2023-05-31T19:52:36.054 INFO:teuthology.orchestra.run.smithi099.stdout: "last_update": "0.000000" 2023-05-31T19:52:36.054 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T19:52:36.054 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T19:52:36.054 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1685562752.955777_47.1", 2023-05-31T19:52:36.055 INFO:teuthology.orchestra.run.smithi099.stdout: "last_update": "2023-05-31T19:52:32.955777Z" 2023-05-31T19:52:36.055 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T19:52:36.055 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T19:52:36.055 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "", 2023-05-31T19:52:36.055 INFO:teuthology.orchestra.run.smithi099.stdout: "last_update": "0.000000" 2023-05-31T19:52:36.056 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T19:52:36.056 INFO:teuthology.orchestra.run.smithi099.stdout:] 2023-05-31T19:52:36.057 DEBUG:rgw_multi.tests:master meta markers={0: '1_1685562631.516089_59.1', 1: '', 2: '1_1685562752.955777_47.1', 3: ''} 2023-05-31T19:52:36.057 INFO:rgw_multi.tests:starting meta checkpoint for zone=a2 2023-05-31T19:52:36.058 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-31T19:52:36.058 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-31T19:52:36.058 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-05-31T19:52:36.108 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2023-05-31T19:52:36.108 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2023-05-31T19:52:36.140 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T19:52:36.135+0000 7ff256a89780 20 RGW-SYNC:meta: read sync status 2023-05-31T19:52:36.141 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T19:52:36.136+0000 7ff256a89780 20 RGW-SYNC:meta: read sync status 2023-05-31T19:52:36.161 INFO:teuthology.orchestra.run.smithi099.stdout:{ 2023-05-31T19:52:36.162 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_status": { 2023-05-31T19:52:36.162 INFO:teuthology.orchestra.run.smithi099.stdout: "info": { 2023-05-31T19:52:36.162 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "sync", 2023-05-31T19:52:36.162 INFO:teuthology.orchestra.run.smithi099.stdout: "num_shards": 4, 2023-05-31T19:52:36.163 INFO:teuthology.orchestra.run.smithi099.stdout: "period": "2e6123d2-d695-4350-9a01-61a455bc782b", 2023-05-31T19:52:36.163 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2023-05-31T19:52:36.163 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T19:52:36.163 INFO:teuthology.orchestra.run.smithi099.stdout: "markers": [ 2023-05-31T19:52:36.163 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T19:52:36.164 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 0, 2023-05-31T19:52:36.164 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T19:52:36.164 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2023-05-31T19:52:36.164 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1685562631.516089_59.1", 2023-05-31T19:52:36.164 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T19:52:36.164 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 1, 2023-05-31T19:52:36.165 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T19:52:36.165 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2023-05-31T19:50:31.516089Z", 2023-05-31T19:52:36.165 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2023-05-31T19:52:36.166 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T19:52:36.166 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T19:52:36.166 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T19:52:36.166 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 1, 2023-05-31T19:52:36.167 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T19:52:36.167 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2023-05-31T19:52:36.167 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "", 2023-05-31T19:52:36.167 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T19:52:36.167 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2023-05-31T19:52:36.167 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T19:52:36.168 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "0.000000", 2023-05-31T19:52:36.168 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2023-05-31T19:52:36.168 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T19:52:36.168 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T19:52:36.169 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T19:52:36.169 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 2, 2023-05-31T19:52:36.170 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T19:52:36.170 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2023-05-31T19:52:36.170 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1685562752.913639_40.1", 2023-05-31T19:52:36.170 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T19:52:36.171 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2023-05-31T19:52:36.172 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T19:52:36.172 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2023-05-31T19:52:32.913639Z", 2023-05-31T19:52:36.172 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2023-05-31T19:52:36.172 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T19:52:36.173 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T19:52:36.173 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T19:52:36.173 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 3, 2023-05-31T19:52:36.173 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T19:52:36.173 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2023-05-31T19:52:36.174 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "", 2023-05-31T19:52:36.174 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T19:52:36.174 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2023-05-31T19:52:36.174 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T19:52:36.175 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "0.000000", 2023-05-31T19:52:36.175 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2023-05-31T19:52:36.175 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T19:52:36.176 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T19:52:36.176 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2023-05-31T19:52:36.176 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T19:52:36.176 INFO:teuthology.orchestra.run.smithi099.stdout: "full_sync": { 2023-05-31T19:52:36.176 INFO:teuthology.orchestra.run.smithi099.stdout: "total": 1, 2023-05-31T19:52:36.176 INFO:teuthology.orchestra.run.smithi099.stdout: "complete": 1 2023-05-31T19:52:36.177 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T19:52:36.177 INFO:teuthology.orchestra.run.smithi099.stdout: "current_time": "2023-05-31T19:52:36Z" 2023-05-31T19:52:36.177 INFO:teuthology.orchestra.run.smithi099.stdout:} 2023-05-31T19:52:36.178 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "2e6123d2-d695-4350-9a01-61a455bc782b", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1685562631.516089_59.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-05-31T19:50:31.516089Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1685562752.913639_40.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-05-31T19:52:32.913639Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-05-31T19:52:36Z" } 2023-05-31T19:52:36.179 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1685562631.516089_59.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-05-31T19:50:31.516089Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1685562752.913639_40.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-05-31T19:52:32.913639Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2023-05-31T19:52:36.179 DEBUG:rgw_multi.tests:log_status={0: '1_1685562631.516089_59.1', 1: '', 2: '1_1685562752.955777_47.1', 3: ''} 2023-05-31T19:52:36.179 DEBUG:rgw_multi.tests:sync_status={0: '1_1685562631.516089_59.1', 1: '', 2: '1_1685562752.913639_40.1', 3: ''} 2023-05-31T19:52:36.180 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1685562752.955777_47.1 target=1_1685562752.913639_40.1 2023-05-31T19:52:41.181 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-31T19:52:41.182 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-31T19:52:41.183 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-05-31T19:52:41.240 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2023-05-31T19:52:41.240 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2023-05-31T19:52:41.272 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T19:52:41.267+0000 7fb385f21780 20 RGW-SYNC:meta: read sync status 2023-05-31T19:52:41.273 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T19:52:41.268+0000 7fb385f21780 20 RGW-SYNC:meta: read sync status 2023-05-31T19:52:41.285 INFO:teuthology.orchestra.run.smithi099.stdout:{ 2023-05-31T19:52:41.285 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_status": { 2023-05-31T19:52:41.286 INFO:teuthology.orchestra.run.smithi099.stdout: "info": { 2023-05-31T19:52:41.286 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "sync", 2023-05-31T19:52:41.286 INFO:teuthology.orchestra.run.smithi099.stdout: "num_shards": 4, 2023-05-31T19:52:41.286 INFO:teuthology.orchestra.run.smithi099.stdout: "period": "2e6123d2-d695-4350-9a01-61a455bc782b", 2023-05-31T19:52:41.286 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2023-05-31T19:52:41.287 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T19:52:41.287 INFO:teuthology.orchestra.run.smithi099.stdout: "markers": [ 2023-05-31T19:52:41.287 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T19:52:41.287 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 0, 2023-05-31T19:52:41.287 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T19:52:41.287 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2023-05-31T19:52:41.287 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1685562631.516089_59.1", 2023-05-31T19:52:41.288 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T19:52:41.288 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 1, 2023-05-31T19:52:41.288 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T19:52:41.288 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2023-05-31T19:50:31.516089Z", 2023-05-31T19:52:41.288 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2023-05-31T19:52:41.288 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T19:52:41.289 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T19:52:41.289 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T19:52:41.289 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 1, 2023-05-31T19:52:41.289 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T19:52:41.289 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2023-05-31T19:52:41.290 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "", 2023-05-31T19:52:41.290 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T19:52:41.290 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2023-05-31T19:52:41.290 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T19:52:41.290 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "0.000000", 2023-05-31T19:52:41.291 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2023-05-31T19:52:41.291 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T19:52:41.291 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T19:52:41.291 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T19:52:41.291 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 2, 2023-05-31T19:52:41.291 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T19:52:41.291 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2023-05-31T19:52:41.292 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1685562752.955777_47.1", 2023-05-31T19:52:41.292 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T19:52:41.292 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2023-05-31T19:52:41.292 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T19:52:41.292 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2023-05-31T19:52:32.955777Z", 2023-05-31T19:52:41.292 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2023-05-31T19:52:41.293 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T19:52:41.293 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T19:52:41.293 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T19:52:41.293 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 3, 2023-05-31T19:52:41.293 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T19:52:41.293 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2023-05-31T19:52:41.293 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "", 2023-05-31T19:52:41.294 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T19:52:41.294 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2023-05-31T19:52:41.294 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T19:52:41.294 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "0.000000", 2023-05-31T19:52:41.294 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2023-05-31T19:52:41.294 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T19:52:41.294 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T19:52:41.295 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2023-05-31T19:52:41.295 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T19:52:41.295 INFO:teuthology.orchestra.run.smithi099.stdout: "full_sync": { 2023-05-31T19:52:41.295 INFO:teuthology.orchestra.run.smithi099.stdout: "total": 1, 2023-05-31T19:52:41.295 INFO:teuthology.orchestra.run.smithi099.stdout: "complete": 1 2023-05-31T19:52:41.295 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T19:52:41.295 INFO:teuthology.orchestra.run.smithi099.stdout: "current_time": "2023-05-31T19:52:41Z" 2023-05-31T19:52:41.296 INFO:teuthology.orchestra.run.smithi099.stdout:} 2023-05-31T19:52:41.297 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "2e6123d2-d695-4350-9a01-61a455bc782b", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1685562631.516089_59.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-05-31T19:50:31.516089Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1685562752.955777_47.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-05-31T19:52:32.955777Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-05-31T19:52:41Z" } 2023-05-31T19:52:41.297 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1685562631.516089_59.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-05-31T19:50:31.516089Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1685562752.955777_47.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-05-31T19:52:32.955777Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2023-05-31T19:52:41.298 DEBUG:rgw_multi.tests:log_status={0: '1_1685562631.516089_59.1', 1: '', 2: '1_1685562752.955777_47.1', 3: ''} 2023-05-31T19:52:41.298 DEBUG:rgw_multi.tests:sync_status={0: '1_1685562631.516089_59.1', 1: '', 2: '1_1685562752.955777_47.1', 3: ''} 2023-05-31T19:52:41.298 INFO:rgw_multi.tests:finish meta checkpoint for zone=a2 2023-05-31T19:52:41.299 DEBUG:boto:path=/pklxkd-1/ 2023-05-31T19:52:41.299 DEBUG:boto:auth_path=/pklxkd-1/ 2023-05-31T19:52:41.299 DEBUG:boto:Method: HEAD 2023-05-31T19:52:41.300 DEBUG:boto:Path: /pklxkd-1/ 2023-05-31T19:52:41.300 DEBUG:boto:Data: 2023-05-31T19:52:41.300 DEBUG:boto:Headers: {} 2023-05-31T19:52:41.301 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T19:52:41.301 DEBUG:boto:Port: 8000 2023-05-31T19:52:41.301 DEBUG:boto:Params: {} 2023-05-31T19:52:41.302 DEBUG:boto:Token: None 2023-05-31T19:52:41.302 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 19:52:41 GMT /pklxkd-1/ 2023-05-31T19:52:41.303 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:jVKcfarnRMMwvowjXlggtruz+Cw= 2023-05-31T19:52:41.303 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 19:52:41 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:jVKcfarnRMMwvowjXlggtruz+Cw=', 'Content-Length': '0'} 2023-05-31T19:52:41.305 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000004d83af434ac18a1e-006477a589-110e-a1'), ('Content-Length', '0'), ('Date', 'Wed, 31 May 2023 19:52:41 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T19:52:41.306 DEBUG:boto:path=/pklxkd-2/ 2023-05-31T19:52:41.306 DEBUG:boto:auth_path=/pklxkd-2/ 2023-05-31T19:52:41.306 DEBUG:boto:Method: HEAD 2023-05-31T19:52:41.307 DEBUG:boto:Path: /pklxkd-2/ 2023-05-31T19:52:41.307 DEBUG:boto:Data: 2023-05-31T19:52:41.308 DEBUG:boto:Headers: {} 2023-05-31T19:52:41.308 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T19:52:41.309 DEBUG:boto:Port: 8000 2023-05-31T19:52:41.309 DEBUG:boto:Params: {} 2023-05-31T19:52:41.310 DEBUG:boto:Token: None 2023-05-31T19:52:41.310 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 19:52:41 GMT /pklxkd-2/ 2023-05-31T19:52:41.310 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:lCZfNARyPDCvEoTVTEj7qrNyt1E= 2023-05-31T19:52:41.311 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 19:52:41 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:lCZfNARyPDCvEoTVTEj7qrNyt1E=', 'Content-Length': '0'} 2023-05-31T19:52:41.313 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000fb989a97d13c4c25-006477a589-110e-a1'), ('Content-Length', '0'), ('Date', 'Wed, 31 May 2023 19:52:41 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T19:52:41.313 DEBUG:boto:path=/pklxkd-1/ 2023-05-31T19:52:41.314 DEBUG:boto:auth_path=/pklxkd-1/ 2023-05-31T19:52:41.314 DEBUG:boto:Method: HEAD 2023-05-31T19:52:41.314 DEBUG:boto:Path: /pklxkd-1/ 2023-05-31T19:52:41.315 DEBUG:boto:Data: 2023-05-31T19:52:41.315 DEBUG:boto:Headers: {} 2023-05-31T19:52:41.315 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T19:52:41.316 DEBUG:boto:Port: 8001 2023-05-31T19:52:41.316 DEBUG:boto:Params: {} 2023-05-31T19:52:41.317 DEBUG:boto:Token: None 2023-05-31T19:52:41.317 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 19:52:41 GMT /pklxkd-1/ 2023-05-31T19:52:41.318 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:jVKcfarnRMMwvowjXlggtruz+Cw= 2023-05-31T19:52:41.318 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 19:52:41 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:jVKcfarnRMMwvowjXlggtruz+Cw=', 'Content-Length': '0'} 2023-05-31T19:52:41.320 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000009b752e5be69ae598-006477a589-1112-a2'), ('Content-Length', '0'), ('Date', 'Wed, 31 May 2023 19:52:41 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T19:52:41.321 DEBUG:boto:path=/pklxkd-2/ 2023-05-31T19:52:41.321 DEBUG:boto:auth_path=/pklxkd-2/ 2023-05-31T19:52:41.322 DEBUG:boto:Method: HEAD 2023-05-31T19:52:41.322 DEBUG:boto:Path: /pklxkd-2/ 2023-05-31T19:52:41.322 DEBUG:boto:Data: 2023-05-31T19:52:41.323 DEBUG:boto:Headers: {} 2023-05-31T19:52:41.323 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T19:52:41.323 DEBUG:boto:Port: 8001 2023-05-31T19:52:41.324 DEBUG:boto:Params: {} 2023-05-31T19:52:41.324 DEBUG:boto:Token: None 2023-05-31T19:52:41.325 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 19:52:41 GMT /pklxkd-2/ 2023-05-31T19:52:41.325 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:lCZfNARyPDCvEoTVTEj7qrNyt1E= 2023-05-31T19:52:41.325 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 19:52:41 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:lCZfNARyPDCvEoTVTEj7qrNyt1E=', 'Content-Length': '0'} 2023-05-31T19:52:41.327 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000008633c0edc6b1a259-006477a589-1112-a2'), ('Content-Length', '0'), ('Date', 'Wed, 31 May 2023 19:52:41 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T19:52:41.328 INFO:tasks.rgw_multisite_tests:rgw_multi.tests.test_bucket_create ... ok 2023-05-31T19:52:41.329 INFO:rgw_multi.tests:create bucket zone=a1 name=pklxkd-3 2023-05-31T19:52:41.330 DEBUG:boto:path=/pklxkd-3/ 2023-05-31T19:52:41.330 DEBUG:boto:auth_path=/pklxkd-3/ 2023-05-31T19:52:41.330 DEBUG:boto:Method: PUT 2023-05-31T19:52:41.331 DEBUG:boto:Path: /pklxkd-3/ 2023-05-31T19:52:41.331 DEBUG:boto:Data: 2023-05-31T19:52:41.332 DEBUG:boto:Headers: {} 2023-05-31T19:52:41.332 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T19:52:41.333 DEBUG:boto:Port: 8000 2023-05-31T19:52:41.333 DEBUG:boto:Params: {} 2023-05-31T19:52:41.333 DEBUG:boto:Token: None 2023-05-31T19:52:41.334 DEBUG:boto:StringToSign: PUT Wed, 31 May 2023 19:52:41 GMT /pklxkd-3/ 2023-05-31T19:52:41.334 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:UZSvOF8KbsL7Mbi4nEDtbvb7qKY= 2023-05-31T19:52:41.335 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 19:52:41 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:UZSvOF8KbsL7Mbi4nEDtbvb7qKY=', 'Content-Length': '0'} 2023-05-31T19:52:41.348 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx000008c516bedb9c6169f-006477a589-110e-a1'), ('Content-Length', '0'), ('Date', 'Wed, 31 May 2023 19:52:41 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T19:52:41.348 INFO:rgw_multi.tests:create bucket zone=a2 name=pklxkd-4 2023-05-31T19:52:41.349 DEBUG:boto:path=/pklxkd-4/ 2023-05-31T19:52:41.349 DEBUG:boto:auth_path=/pklxkd-4/ 2023-05-31T19:52:41.350 DEBUG:boto:Method: PUT 2023-05-31T19:52:41.350 DEBUG:boto:Path: /pklxkd-4/ 2023-05-31T19:52:41.350 DEBUG:boto:Data: 2023-05-31T19:52:41.351 DEBUG:boto:Headers: {} 2023-05-31T19:52:41.351 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T19:52:41.351 DEBUG:boto:Port: 8001 2023-05-31T19:52:41.352 DEBUG:boto:Params: {} 2023-05-31T19:52:41.352 DEBUG:boto:Token: None 2023-05-31T19:52:41.353 DEBUG:boto:StringToSign: PUT Wed, 31 May 2023 19:52:41 GMT /pklxkd-4/ 2023-05-31T19:52:41.353 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:zkXFKksoJk0yJCAhJWeBoxncUNM= 2023-05-31T19:52:41.353 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 19:52:41 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:zkXFKksoJk0yJCAhJWeBoxncUNM=', 'Content-Length': '0'} 2023-05-31T19:52:41.390 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000c742a1f6b22751a1-006477a589-1112-a2'), ('Content-Length', '0'), ('Date', 'Wed, 31 May 2023 19:52:41 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T19:52:41.391 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-31T19:52:41.391 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-31T19:52:41.392 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-05-31T19:52:41.442 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2023-05-31T19:52:41.442 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2023-05-31T19:52:41.494 INFO:teuthology.orchestra.run.smithi099.stderr:No --period given, using current period=2e6123d2-d695-4350-9a01-61a455bc782b 2023-05-31T19:52:41.514 INFO:teuthology.orchestra.run.smithi099.stdout:[ 2023-05-31T19:52:41.514 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T19:52:41.514 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1685562631.516089_59.1", 2023-05-31T19:52:41.514 INFO:teuthology.orchestra.run.smithi099.stdout: "last_update": "2023-05-31T19:50:31.516089Z" 2023-05-31T19:52:41.515 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T19:52:41.515 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T19:52:41.515 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "", 2023-05-31T19:52:41.515 INFO:teuthology.orchestra.run.smithi099.stdout: "last_update": "0.000000" 2023-05-31T19:52:41.515 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T19:52:41.515 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T19:52:41.516 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1685562761.369257_55.1", 2023-05-31T19:52:41.516 INFO:teuthology.orchestra.run.smithi099.stdout: "last_update": "2023-05-31T19:52:41.369257Z" 2023-05-31T19:52:41.516 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T19:52:41.516 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T19:52:41.516 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "", 2023-05-31T19:52:41.517 INFO:teuthology.orchestra.run.smithi099.stdout: "last_update": "0.000000" 2023-05-31T19:52:41.517 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T19:52:41.517 INFO:teuthology.orchestra.run.smithi099.stdout:] 2023-05-31T19:52:41.518 DEBUG:rgw_multi.tests:master meta markers={0: '1_1685562631.516089_59.1', 1: '', 2: '1_1685562761.369257_55.1', 3: ''} 2023-05-31T19:52:41.518 INFO:rgw_multi.tests:starting meta checkpoint for zone=a2 2023-05-31T19:52:41.519 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-31T19:52:41.519 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-31T19:52:41.520 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-05-31T19:52:41.570 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2023-05-31T19:52:41.570 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2023-05-31T19:52:41.600 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T19:52:41.595+0000 7f92fe313780 20 RGW-SYNC:meta: read sync status 2023-05-31T19:52:41.601 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T19:52:41.596+0000 7f92fe313780 20 RGW-SYNC:meta: read sync status 2023-05-31T19:52:41.622 INFO:teuthology.orchestra.run.smithi099.stdout:{ 2023-05-31T19:52:41.623 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_status": { 2023-05-31T19:52:41.623 INFO:teuthology.orchestra.run.smithi099.stdout: "info": { 2023-05-31T19:52:41.623 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "sync", 2023-05-31T19:52:41.623 INFO:teuthology.orchestra.run.smithi099.stdout: "num_shards": 4, 2023-05-31T19:52:41.624 INFO:teuthology.orchestra.run.smithi099.stdout: "period": "2e6123d2-d695-4350-9a01-61a455bc782b", 2023-05-31T19:52:41.624 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2023-05-31T19:52:41.624 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T19:52:41.625 INFO:teuthology.orchestra.run.smithi099.stdout: "markers": [ 2023-05-31T19:52:41.625 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T19:52:41.625 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 0, 2023-05-31T19:52:41.626 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T19:52:41.626 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2023-05-31T19:52:41.626 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1685562631.516089_59.1", 2023-05-31T19:52:41.627 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T19:52:41.627 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 1, 2023-05-31T19:52:41.627 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T19:52:41.627 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2023-05-31T19:50:31.516089Z", 2023-05-31T19:52:41.627 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2023-05-31T19:52:41.628 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T19:52:41.628 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T19:52:41.628 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T19:52:41.628 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 1, 2023-05-31T19:52:41.628 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T19:52:41.628 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2023-05-31T19:52:41.629 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "", 2023-05-31T19:52:41.629 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T19:52:41.629 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2023-05-31T19:52:41.629 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T19:52:41.629 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "0.000000", 2023-05-31T19:52:41.629 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2023-05-31T19:52:41.629 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T19:52:41.630 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T19:52:41.630 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T19:52:41.630 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 2, 2023-05-31T19:52:41.630 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T19:52:41.630 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2023-05-31T19:52:41.630 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1685562761.356534_52.1", 2023-05-31T19:52:41.630 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T19:52:41.631 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2023-05-31T19:52:41.631 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T19:52:41.631 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2023-05-31T19:52:41.356534Z", 2023-05-31T19:52:41.631 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2023-05-31T19:52:41.631 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T19:52:41.632 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T19:52:41.632 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T19:52:41.632 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 3, 2023-05-31T19:52:41.632 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T19:52:41.632 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2023-05-31T19:52:41.632 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "", 2023-05-31T19:52:41.633 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T19:52:41.633 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2023-05-31T19:52:41.633 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T19:52:41.633 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "0.000000", 2023-05-31T19:52:41.633 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2023-05-31T19:52:41.633 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T19:52:41.634 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T19:52:41.634 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2023-05-31T19:52:41.634 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T19:52:41.634 INFO:teuthology.orchestra.run.smithi099.stdout: "full_sync": { 2023-05-31T19:52:41.634 INFO:teuthology.orchestra.run.smithi099.stdout: "total": 1, 2023-05-31T19:52:41.634 INFO:teuthology.orchestra.run.smithi099.stdout: "complete": 1 2023-05-31T19:52:41.635 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T19:52:41.635 INFO:teuthology.orchestra.run.smithi099.stdout: "current_time": "2023-05-31T19:52:41Z" 2023-05-31T19:52:41.635 INFO:teuthology.orchestra.run.smithi099.stdout:} 2023-05-31T19:52:41.636 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "2e6123d2-d695-4350-9a01-61a455bc782b", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1685562631.516089_59.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-05-31T19:50:31.516089Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1685562761.356534_52.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-05-31T19:52:41.356534Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-05-31T19:52:41Z" } 2023-05-31T19:52:41.637 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1685562631.516089_59.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-05-31T19:50:31.516089Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1685562761.356534_52.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-05-31T19:52:41.356534Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2023-05-31T19:52:41.637 DEBUG:rgw_multi.tests:log_status={0: '1_1685562631.516089_59.1', 1: '', 2: '1_1685562761.369257_55.1', 3: ''} 2023-05-31T19:52:41.637 DEBUG:rgw_multi.tests:sync_status={0: '1_1685562631.516089_59.1', 1: '', 2: '1_1685562761.356534_52.1', 3: ''} 2023-05-31T19:52:41.638 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1685562761.369257_55.1 target=1_1685562761.356534_52.1 2023-05-31T19:52:46.639 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-31T19:52:46.640 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-31T19:52:46.641 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-05-31T19:52:46.698 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2023-05-31T19:52:46.699 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2023-05-31T19:52:46.745 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T19:52:46.740+0000 7f2ed8d2d780 20 RGW-SYNC:meta: read sync status 2023-05-31T19:52:46.746 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T19:52:46.741+0000 7f2ed8d2d780 20 RGW-SYNC:meta: read sync status 2023-05-31T19:52:46.765 INFO:teuthology.orchestra.run.smithi099.stdout:{ 2023-05-31T19:52:46.766 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_status": { 2023-05-31T19:52:46.766 INFO:teuthology.orchestra.run.smithi099.stdout: "info": { 2023-05-31T19:52:46.766 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "sync", 2023-05-31T19:52:46.766 INFO:teuthology.orchestra.run.smithi099.stdout: "num_shards": 4, 2023-05-31T19:52:46.766 INFO:teuthology.orchestra.run.smithi099.stdout: "period": "2e6123d2-d695-4350-9a01-61a455bc782b", 2023-05-31T19:52:46.767 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2023-05-31T19:52:46.767 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T19:52:46.767 INFO:teuthology.orchestra.run.smithi099.stdout: "markers": [ 2023-05-31T19:52:46.767 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T19:52:46.767 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 0, 2023-05-31T19:52:46.767 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T19:52:46.767 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2023-05-31T19:52:46.768 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1685562631.516089_59.1", 2023-05-31T19:52:46.768 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T19:52:46.768 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 1, 2023-05-31T19:52:46.768 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T19:52:46.769 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2023-05-31T19:50:31.516089Z", 2023-05-31T19:52:46.769 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2023-05-31T19:52:46.769 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T19:52:46.769 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T19:52:46.769 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T19:52:46.769 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 1, 2023-05-31T19:52:46.770 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T19:52:46.770 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2023-05-31T19:52:46.770 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "", 2023-05-31T19:52:46.770 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T19:52:46.770 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2023-05-31T19:52:46.770 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T19:52:46.770 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "0.000000", 2023-05-31T19:52:46.771 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2023-05-31T19:52:46.771 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T19:52:46.771 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T19:52:46.771 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T19:52:46.771 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 2, 2023-05-31T19:52:46.771 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T19:52:46.771 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2023-05-31T19:52:46.772 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1685562761.369257_55.1", 2023-05-31T19:52:46.772 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T19:52:46.772 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2023-05-31T19:52:46.772 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T19:52:46.773 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2023-05-31T19:52:41.369257Z", 2023-05-31T19:52:46.773 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2023-05-31T19:52:46.773 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T19:52:46.773 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T19:52:46.773 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T19:52:46.774 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 3, 2023-05-31T19:52:46.774 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T19:52:46.774 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2023-05-31T19:52:46.774 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "", 2023-05-31T19:52:46.774 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T19:52:46.774 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2023-05-31T19:52:46.774 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T19:52:46.775 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "0.000000", 2023-05-31T19:52:46.775 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2023-05-31T19:52:46.775 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T19:52:46.775 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T19:52:46.775 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2023-05-31T19:52:46.776 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T19:52:46.776 INFO:teuthology.orchestra.run.smithi099.stdout: "full_sync": { 2023-05-31T19:52:46.776 INFO:teuthology.orchestra.run.smithi099.stdout: "total": 1, 2023-05-31T19:52:46.776 INFO:teuthology.orchestra.run.smithi099.stdout: "complete": 1 2023-05-31T19:52:46.776 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T19:52:46.776 INFO:teuthology.orchestra.run.smithi099.stdout: "current_time": "2023-05-31T19:52:46Z" 2023-05-31T19:52:46.776 INFO:teuthology.orchestra.run.smithi099.stdout:} 2023-05-31T19:52:46.777 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "2e6123d2-d695-4350-9a01-61a455bc782b", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1685562631.516089_59.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-05-31T19:50:31.516089Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1685562761.369257_55.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-05-31T19:52:41.369257Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-05-31T19:52:46Z" } 2023-05-31T19:52:46.778 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1685562631.516089_59.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-05-31T19:50:31.516089Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1685562761.369257_55.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-05-31T19:52:41.369257Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2023-05-31T19:52:46.778 DEBUG:rgw_multi.tests:log_status={0: '1_1685562631.516089_59.1', 1: '', 2: '1_1685562761.369257_55.1', 3: ''} 2023-05-31T19:52:46.779 DEBUG:rgw_multi.tests:sync_status={0: '1_1685562631.516089_59.1', 1: '', 2: '1_1685562761.369257_55.1', 3: ''} 2023-05-31T19:52:46.779 INFO:rgw_multi.tests:finish meta checkpoint for zone=a2 2023-05-31T19:52:46.780 DEBUG:boto:path=/pklxkd-3/ 2023-05-31T19:52:46.780 DEBUG:boto:auth_path=/pklxkd-3/ 2023-05-31T19:52:46.780 DEBUG:boto:Method: HEAD 2023-05-31T19:52:46.781 DEBUG:boto:Path: /pklxkd-3/ 2023-05-31T19:52:46.781 DEBUG:boto:Data: 2023-05-31T19:52:46.782 DEBUG:boto:Headers: {} 2023-05-31T19:52:46.782 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T19:52:46.782 DEBUG:boto:Port: 8000 2023-05-31T19:52:46.782 DEBUG:boto:Params: {} 2023-05-31T19:52:46.783 DEBUG:boto:Token: None 2023-05-31T19:52:46.784 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 19:52:46 GMT /pklxkd-3/ 2023-05-31T19:52:46.784 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:+2CVPGZXAItFId4GgbxBqhTLRUw= 2023-05-31T19:52:46.785 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 19:52:46 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:+2CVPGZXAItFId4GgbxBqhTLRUw=', 'Content-Length': '0'} 2023-05-31T19:52:46.787 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000418e357c387b33ae-006477a58e-110e-a1'), ('Content-Length', '0'), ('Date', 'Wed, 31 May 2023 19:52:46 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T19:52:46.787 DEBUG:boto:path=/pklxkd-4/ 2023-05-31T19:52:46.788 DEBUG:boto:auth_path=/pklxkd-4/ 2023-05-31T19:52:46.788 DEBUG:boto:Method: HEAD 2023-05-31T19:52:46.789 DEBUG:boto:Path: /pklxkd-4/ 2023-05-31T19:52:46.789 DEBUG:boto:Data: 2023-05-31T19:52:46.790 DEBUG:boto:Headers: {} 2023-05-31T19:52:46.790 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T19:52:46.790 DEBUG:boto:Port: 8000 2023-05-31T19:52:46.791 DEBUG:boto:Params: {} 2023-05-31T19:52:46.791 DEBUG:boto:Token: None 2023-05-31T19:52:46.792 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 19:52:46 GMT /pklxkd-4/ 2023-05-31T19:52:46.792 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:t0/AA2l1WdFi8d7Umu5jh3dC7fw= 2023-05-31T19:52:46.793 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 19:52:46 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:t0/AA2l1WdFi8d7Umu5jh3dC7fw=', 'Content-Length': '0'} 2023-05-31T19:52:46.795 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000068060c6d2a1d1173-006477a58e-110e-a1'), ('Content-Length', '0'), ('Date', 'Wed, 31 May 2023 19:52:46 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T19:52:46.796 DEBUG:boto:path=/pklxkd-3/ 2023-05-31T19:52:46.796 DEBUG:boto:auth_path=/pklxkd-3/ 2023-05-31T19:52:46.797 DEBUG:boto:Method: HEAD 2023-05-31T19:52:46.797 DEBUG:boto:Path: /pklxkd-3/ 2023-05-31T19:52:46.798 DEBUG:boto:Data: 2023-05-31T19:52:46.798 DEBUG:boto:Headers: {} 2023-05-31T19:52:46.799 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T19:52:46.799 DEBUG:boto:Port: 8001 2023-05-31T19:52:46.800 DEBUG:boto:Params: {} 2023-05-31T19:52:46.800 DEBUG:boto:Token: None 2023-05-31T19:52:46.801 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 19:52:46 GMT /pklxkd-3/ 2023-05-31T19:52:46.802 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:+2CVPGZXAItFId4GgbxBqhTLRUw= 2023-05-31T19:52:46.803 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 19:52:46 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:+2CVPGZXAItFId4GgbxBqhTLRUw=', 'Content-Length': '0'} 2023-05-31T19:52:46.805 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000d75bea1c4961af21-006477a58e-1112-a2'), ('Content-Length', '0'), ('Date', 'Wed, 31 May 2023 19:52:46 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T19:52:46.805 DEBUG:boto:path=/pklxkd-4/ 2023-05-31T19:52:46.806 DEBUG:boto:auth_path=/pklxkd-4/ 2023-05-31T19:52:46.806 DEBUG:boto:Method: HEAD 2023-05-31T19:52:46.807 DEBUG:boto:Path: /pklxkd-4/ 2023-05-31T19:52:46.807 DEBUG:boto:Data: 2023-05-31T19:52:46.807 DEBUG:boto:Headers: {} 2023-05-31T19:52:46.808 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T19:52:46.808 DEBUG:boto:Port: 8001 2023-05-31T19:52:46.809 DEBUG:boto:Params: {} 2023-05-31T19:52:46.809 DEBUG:boto:Token: None 2023-05-31T19:52:46.810 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 19:52:46 GMT /pklxkd-4/ 2023-05-31T19:52:46.810 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:t0/AA2l1WdFi8d7Umu5jh3dC7fw= 2023-05-31T19:52:46.811 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 19:52:46 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:t0/AA2l1WdFi8d7Umu5jh3dC7fw=', 'Content-Length': '0'} 2023-05-31T19:52:46.812 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000a53bea836c55cb30-006477a58e-1112-a2'), ('Content-Length', '0'), ('Date', 'Wed, 31 May 2023 19:52:46 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T19:52:46.813 DEBUG:boto:path=/pklxkd-3/ 2023-05-31T19:52:46.813 DEBUG:boto:auth_path=/pklxkd-3/ 2023-05-31T19:52:46.814 DEBUG:boto:Method: PUT 2023-05-31T19:52:46.814 DEBUG:boto:Path: /pklxkd-3/ 2023-05-31T19:52:46.815 DEBUG:boto:Data: 2023-05-31T19:52:46.815 DEBUG:boto:Headers: {} 2023-05-31T19:52:46.815 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T19:52:46.816 DEBUG:boto:Port: 8000 2023-05-31T19:52:46.816 DEBUG:boto:Params: {} 2023-05-31T19:52:46.816 DEBUG:boto:Token: None 2023-05-31T19:52:46.817 DEBUG:boto:StringToSign: PUT Wed, 31 May 2023 19:52:46 GMT /pklxkd-3/ 2023-05-31T19:52:46.817 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:8804gWZOluJfSad+kguuo7aWMQI= 2023-05-31T19:52:46.818 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 19:52:46 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:8804gWZOluJfSad+kguuo7aWMQI=', 'Content-Length': '0'} 2023-05-31T19:52:46.820 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000b2af6a0de9fc9685-006477a58e-110e-a1'), ('Content-Length', '0'), ('Date', 'Wed, 31 May 2023 19:52:46 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T19:52:46.821 DEBUG:boto:path=/pklxkd-4/ 2023-05-31T19:52:46.821 DEBUG:boto:auth_path=/pklxkd-4/ 2023-05-31T19:52:46.821 DEBUG:boto:Method: PUT 2023-05-31T19:52:46.822 DEBUG:boto:Path: /pklxkd-4/ 2023-05-31T19:52:46.822 DEBUG:boto:Data: 2023-05-31T19:52:46.822 DEBUG:boto:Headers: {} 2023-05-31T19:52:46.823 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T19:52:46.823 DEBUG:boto:Port: 8000 2023-05-31T19:52:46.823 DEBUG:boto:Params: {} 2023-05-31T19:52:46.824 DEBUG:boto:Token: None 2023-05-31T19:52:46.824 DEBUG:boto:StringToSign: PUT Wed, 31 May 2023 19:52:46 GMT /pklxkd-4/ 2023-05-31T19:52:46.825 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:U9Ns5xUFE7V58nQhExDEiTppOjY= 2023-05-31T19:52:46.825 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 19:52:46 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:U9Ns5xUFE7V58nQhExDEiTppOjY=', 'Content-Length': '0'} 2023-05-31T19:52:46.828 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx0000059f68250a5f2814a-006477a58e-110e-a1'), ('Content-Length', '0'), ('Date', 'Wed, 31 May 2023 19:52:46 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T19:52:46.828 DEBUG:boto:path=/pklxkd-3/ 2023-05-31T19:52:46.829 DEBUG:boto:auth_path=/pklxkd-3/ 2023-05-31T19:52:46.829 DEBUG:boto:Method: PUT 2023-05-31T19:52:46.829 DEBUG:boto:Path: /pklxkd-3/ 2023-05-31T19:52:46.830 DEBUG:boto:Data: 2023-05-31T19:52:46.830 DEBUG:boto:Headers: {} 2023-05-31T19:52:46.830 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T19:52:46.831 DEBUG:boto:Port: 8001 2023-05-31T19:52:46.831 DEBUG:boto:Params: {} 2023-05-31T19:52:46.831 DEBUG:boto:Token: None 2023-05-31T19:52:46.832 DEBUG:boto:StringToSign: PUT Wed, 31 May 2023 19:52:46 GMT /pklxkd-3/ 2023-05-31T19:52:46.832 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:8804gWZOluJfSad+kguuo7aWMQI= 2023-05-31T19:52:46.833 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 19:52:46 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:8804gWZOluJfSad+kguuo7aWMQI=', 'Content-Length': '0'} 2023-05-31T19:52:46.838 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000be9fb735d4ee6854-006477a58e-1112-a2'), ('Content-Length', '0'), ('Date', 'Wed, 31 May 2023 19:52:46 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T19:52:46.838 DEBUG:boto:path=/pklxkd-4/ 2023-05-31T19:52:46.839 DEBUG:boto:auth_path=/pklxkd-4/ 2023-05-31T19:52:46.839 DEBUG:boto:Method: PUT 2023-05-31T19:52:46.839 DEBUG:boto:Path: /pklxkd-4/ 2023-05-31T19:52:46.840 DEBUG:boto:Data: 2023-05-31T19:52:46.840 DEBUG:boto:Headers: {} 2023-05-31T19:52:46.841 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T19:52:46.841 DEBUG:boto:Port: 8001 2023-05-31T19:52:46.841 DEBUG:boto:Params: {} 2023-05-31T19:52:46.842 DEBUG:boto:Token: None 2023-05-31T19:52:46.842 DEBUG:boto:StringToSign: PUT Wed, 31 May 2023 19:52:46 GMT /pklxkd-4/ 2023-05-31T19:52:46.842 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:U9Ns5xUFE7V58nQhExDEiTppOjY= 2023-05-31T19:52:46.843 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 19:52:46 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:U9Ns5xUFE7V58nQhExDEiTppOjY=', 'Content-Length': '0'} 2023-05-31T19:52:46.889 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000b5729fe51af72e80-006477a58e-1112-a2'), ('Content-Length', '0'), ('Date', 'Wed, 31 May 2023 19:52:46 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T19:52:46.889 DEBUG:boto:path=/pklxkd-3/ 2023-05-31T19:52:46.890 DEBUG:boto:auth_path=/pklxkd-3/ 2023-05-31T19:52:46.890 DEBUG:boto:Method: HEAD 2023-05-31T19:52:46.890 DEBUG:boto:Path: /pklxkd-3/ 2023-05-31T19:52:46.891 DEBUG:boto:Data: 2023-05-31T19:52:46.891 DEBUG:boto:Headers: {} 2023-05-31T19:52:46.892 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T19:52:46.893 DEBUG:boto:Port: 8000 2023-05-31T19:52:46.893 DEBUG:boto:Params: {} 2023-05-31T19:52:46.894 DEBUG:boto:Token: None 2023-05-31T19:52:46.894 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 19:52:46 GMT /pklxkd-3/ 2023-05-31T19:52:46.895 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:+2CVPGZXAItFId4GgbxBqhTLRUw= 2023-05-31T19:52:46.895 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 19:52:46 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:+2CVPGZXAItFId4GgbxBqhTLRUw=', 'Content-Length': '0'} 2023-05-31T19:52:46.898 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000003579420301d55803-006477a58e-110e-a1'), ('Content-Length', '0'), ('Date', 'Wed, 31 May 2023 19:52:46 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T19:52:46.898 DEBUG:boto:path=/pklxkd-4/ 2023-05-31T19:52:46.899 DEBUG:boto:auth_path=/pklxkd-4/ 2023-05-31T19:52:46.899 DEBUG:boto:Method: HEAD 2023-05-31T19:52:46.899 DEBUG:boto:Path: /pklxkd-4/ 2023-05-31T19:52:46.900 DEBUG:boto:Data: 2023-05-31T19:52:46.900 DEBUG:boto:Headers: {} 2023-05-31T19:52:46.900 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T19:52:46.901 DEBUG:boto:Port: 8000 2023-05-31T19:52:46.901 DEBUG:boto:Params: {} 2023-05-31T19:52:46.901 DEBUG:boto:Token: None 2023-05-31T19:52:46.902 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 19:52:46 GMT /pklxkd-4/ 2023-05-31T19:52:46.903 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:t0/AA2l1WdFi8d7Umu5jh3dC7fw= 2023-05-31T19:52:46.903 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 19:52:46 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:t0/AA2l1WdFi8d7Umu5jh3dC7fw=', 'Content-Length': '0'} 2023-05-31T19:52:46.905 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000194252473d46baba-006477a58e-110e-a1'), ('Content-Length', '0'), ('Date', 'Wed, 31 May 2023 19:52:46 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T19:52:46.906 DEBUG:boto:path=/pklxkd-3/ 2023-05-31T19:52:46.906 DEBUG:boto:auth_path=/pklxkd-3/ 2023-05-31T19:52:46.907 DEBUG:boto:Method: HEAD 2023-05-31T19:52:46.907 DEBUG:boto:Path: /pklxkd-3/ 2023-05-31T19:52:46.907 DEBUG:boto:Data: 2023-05-31T19:52:46.908 DEBUG:boto:Headers: {} 2023-05-31T19:52:46.908 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T19:52:46.908 DEBUG:boto:Port: 8001 2023-05-31T19:52:46.909 DEBUG:boto:Params: {} 2023-05-31T19:52:46.909 DEBUG:boto:Token: None 2023-05-31T19:52:46.910 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 19:52:46 GMT /pklxkd-3/ 2023-05-31T19:52:46.910 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:+2CVPGZXAItFId4GgbxBqhTLRUw= 2023-05-31T19:52:46.910 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 19:52:46 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:+2CVPGZXAItFId4GgbxBqhTLRUw=', 'Content-Length': '0'} 2023-05-31T19:52:46.912 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000aec3dca43e8873db-006477a58e-1112-a2'), ('Content-Length', '0'), ('Date', 'Wed, 31 May 2023 19:52:46 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T19:52:46.913 DEBUG:boto:path=/pklxkd-4/ 2023-05-31T19:52:46.913 DEBUG:boto:auth_path=/pklxkd-4/ 2023-05-31T19:52:46.913 DEBUG:boto:Method: HEAD 2023-05-31T19:52:46.914 DEBUG:boto:Path: /pklxkd-4/ 2023-05-31T19:52:46.914 DEBUG:boto:Data: 2023-05-31T19:52:46.914 DEBUG:boto:Headers: {} 2023-05-31T19:52:46.915 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T19:52:46.915 DEBUG:boto:Port: 8001 2023-05-31T19:52:46.915 DEBUG:boto:Params: {} 2023-05-31T19:52:46.916 DEBUG:boto:Token: None 2023-05-31T19:52:46.916 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 19:52:46 GMT /pklxkd-4/ 2023-05-31T19:52:46.917 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:t0/AA2l1WdFi8d7Umu5jh3dC7fw= 2023-05-31T19:52:46.917 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 19:52:46 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:t0/AA2l1WdFi8d7Umu5jh3dC7fw=', 'Content-Length': '0'} 2023-05-31T19:52:46.919 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000076b735579142f3ce-006477a58e-1112-a2'), ('Content-Length', '0'), ('Date', 'Wed, 31 May 2023 19:52:46 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T19:52:46.920 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-31T19:52:46.920 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-31T19:52:46.920 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-05-31T19:52:46.968 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2023-05-31T19:52:46.969 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2023-05-31T19:52:47.015 INFO:teuthology.orchestra.run.smithi099.stderr:No --period given, using current period=2e6123d2-d695-4350-9a01-61a455bc782b 2023-05-31T19:52:47.032 INFO:teuthology.orchestra.run.smithi099.stdout:[ 2023-05-31T19:52:47.033 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T19:52:47.033 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1685562631.516089_59.1", 2023-05-31T19:52:47.033 INFO:teuthology.orchestra.run.smithi099.stdout: "last_update": "2023-05-31T19:50:31.516089Z" 2023-05-31T19:52:47.033 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T19:52:47.034 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T19:52:47.034 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "", 2023-05-31T19:52:47.034 INFO:teuthology.orchestra.run.smithi099.stdout: "last_update": "0.000000" 2023-05-31T19:52:47.035 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T19:52:47.035 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T19:52:47.035 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1685562761.369257_55.1", 2023-05-31T19:52:47.035 INFO:teuthology.orchestra.run.smithi099.stdout: "last_update": "2023-05-31T19:52:41.369257Z" 2023-05-31T19:52:47.036 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T19:52:47.036 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T19:52:47.036 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "", 2023-05-31T19:52:47.037 INFO:teuthology.orchestra.run.smithi099.stdout: "last_update": "0.000000" 2023-05-31T19:52:47.037 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T19:52:47.037 INFO:teuthology.orchestra.run.smithi099.stdout:] 2023-05-31T19:52:47.039 DEBUG:rgw_multi.tests:master meta markers={0: '1_1685562631.516089_59.1', 1: '', 2: '1_1685562761.369257_55.1', 3: ''} 2023-05-31T19:52:47.039 INFO:rgw_multi.tests:starting meta checkpoint for zone=a2 2023-05-31T19:52:47.039 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-31T19:52:47.040 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-31T19:52:47.040 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-05-31T19:52:47.089 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2023-05-31T19:52:47.089 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2023-05-31T19:52:47.131 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T19:52:47.126+0000 7f47b66cf780 20 RGW-SYNC:meta: read sync status 2023-05-31T19:52:47.131 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T19:52:47.127+0000 7f47b66cf780 20 RGW-SYNC:meta: read sync status 2023-05-31T19:52:47.145 INFO:teuthology.orchestra.run.smithi099.stdout:{ 2023-05-31T19:52:47.145 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_status": { 2023-05-31T19:52:47.145 INFO:teuthology.orchestra.run.smithi099.stdout: "info": { 2023-05-31T19:52:47.145 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "sync", 2023-05-31T19:52:47.146 INFO:teuthology.orchestra.run.smithi099.stdout: "num_shards": 4, 2023-05-31T19:52:47.146 INFO:teuthology.orchestra.run.smithi099.stdout: "period": "2e6123d2-d695-4350-9a01-61a455bc782b", 2023-05-31T19:52:47.146 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2023-05-31T19:52:47.147 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T19:52:47.147 INFO:teuthology.orchestra.run.smithi099.stdout: "markers": [ 2023-05-31T19:52:47.147 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T19:52:47.147 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 0, 2023-05-31T19:52:47.147 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T19:52:47.148 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2023-05-31T19:52:47.148 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1685562631.516089_59.1", 2023-05-31T19:52:47.148 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T19:52:47.148 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 1, 2023-05-31T19:52:47.148 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T19:52:47.149 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2023-05-31T19:50:31.516089Z", 2023-05-31T19:52:47.149 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2023-05-31T19:52:47.149 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T19:52:47.150 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T19:52:47.150 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T19:52:47.150 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 1, 2023-05-31T19:52:47.151 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T19:52:47.151 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2023-05-31T19:52:47.151 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "", 2023-05-31T19:52:47.152 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T19:52:47.152 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2023-05-31T19:52:47.152 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T19:52:47.152 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "0.000000", 2023-05-31T19:52:47.153 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2023-05-31T19:52:47.153 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T19:52:47.153 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T19:52:47.154 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T19:52:47.154 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 2, 2023-05-31T19:52:47.154 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T19:52:47.154 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2023-05-31T19:52:47.155 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1685562761.369257_55.1", 2023-05-31T19:52:47.155 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T19:52:47.155 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2023-05-31T19:52:47.155 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T19:52:47.155 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2023-05-31T19:52:41.369257Z", 2023-05-31T19:52:47.155 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2023-05-31T19:52:47.156 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T19:52:47.156 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T19:52:47.156 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T19:52:47.156 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 3, 2023-05-31T19:52:47.156 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T19:52:47.156 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2023-05-31T19:52:47.157 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "", 2023-05-31T19:52:47.157 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T19:52:47.157 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2023-05-31T19:52:47.157 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T19:52:47.157 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "0.000000", 2023-05-31T19:52:47.157 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2023-05-31T19:52:47.157 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T19:52:47.158 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T19:52:47.158 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2023-05-31T19:52:47.158 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T19:52:47.158 INFO:teuthology.orchestra.run.smithi099.stdout: "full_sync": { 2023-05-31T19:52:47.158 INFO:teuthology.orchestra.run.smithi099.stdout: "total": 1, 2023-05-31T19:52:47.158 INFO:teuthology.orchestra.run.smithi099.stdout: "complete": 1 2023-05-31T19:52:47.158 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T19:52:47.159 INFO:teuthology.orchestra.run.smithi099.stdout: "current_time": "2023-05-31T19:52:47Z" 2023-05-31T19:52:47.159 INFO:teuthology.orchestra.run.smithi099.stdout:} 2023-05-31T19:52:47.160 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "2e6123d2-d695-4350-9a01-61a455bc782b", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1685562631.516089_59.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-05-31T19:50:31.516089Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1685562761.369257_55.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-05-31T19:52:41.369257Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-05-31T19:52:47Z" } 2023-05-31T19:52:47.160 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1685562631.516089_59.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-05-31T19:50:31.516089Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1685562761.369257_55.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-05-31T19:52:41.369257Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2023-05-31T19:52:47.161 DEBUG:rgw_multi.tests:log_status={0: '1_1685562631.516089_59.1', 1: '', 2: '1_1685562761.369257_55.1', 3: ''} 2023-05-31T19:52:47.161 DEBUG:rgw_multi.tests:sync_status={0: '1_1685562631.516089_59.1', 1: '', 2: '1_1685562761.369257_55.1', 3: ''} 2023-05-31T19:52:47.161 INFO:rgw_multi.tests:finish meta checkpoint for zone=a2 2023-05-31T19:52:47.162 DEBUG:boto:path=/pklxkd-3/ 2023-05-31T19:52:47.162 DEBUG:boto:auth_path=/pklxkd-3/ 2023-05-31T19:52:47.162 DEBUG:boto:Method: HEAD 2023-05-31T19:52:47.163 DEBUG:boto:Path: /pklxkd-3/ 2023-05-31T19:52:47.163 DEBUG:boto:Data: 2023-05-31T19:52:47.164 DEBUG:boto:Headers: {} 2023-05-31T19:52:47.164 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T19:52:47.165 DEBUG:boto:Port: 8000 2023-05-31T19:52:47.165 DEBUG:boto:Params: {} 2023-05-31T19:52:47.165 DEBUG:boto:Token: None 2023-05-31T19:52:47.166 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 19:52:47 GMT /pklxkd-3/ 2023-05-31T19:52:47.166 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:7bfA2AqQ3VoKUEJ3Myv/PLGjW7Y= 2023-05-31T19:52:47.167 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 19:52:47 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:7bfA2AqQ3VoKUEJ3Myv/PLGjW7Y=', 'Content-Length': '0'} 2023-05-31T19:52:47.169 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000cdcd7d59a48e6bf5-006477a58f-110e-a1'), ('Content-Length', '0'), ('Date', 'Wed, 31 May 2023 19:52:47 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T19:52:47.169 DEBUG:boto:path=/pklxkd-4/ 2023-05-31T19:52:47.170 DEBUG:boto:auth_path=/pklxkd-4/ 2023-05-31T19:52:47.170 DEBUG:boto:Method: HEAD 2023-05-31T19:52:47.171 DEBUG:boto:Path: /pklxkd-4/ 2023-05-31T19:52:47.171 DEBUG:boto:Data: 2023-05-31T19:52:47.171 DEBUG:boto:Headers: {} 2023-05-31T19:52:47.172 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T19:52:47.172 DEBUG:boto:Port: 8000 2023-05-31T19:52:47.173 DEBUG:boto:Params: {} 2023-05-31T19:52:47.174 DEBUG:boto:Token: None 2023-05-31T19:52:47.174 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 19:52:47 GMT /pklxkd-4/ 2023-05-31T19:52:47.174 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:5PKO58LYc8zYRxP1RMNxWZoSg8s= 2023-05-31T19:52:47.175 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 19:52:47 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:5PKO58LYc8zYRxP1RMNxWZoSg8s=', 'Content-Length': '0'} 2023-05-31T19:52:47.177 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000776c906c934b079a-006477a58f-110e-a1'), ('Content-Length', '0'), ('Date', 'Wed, 31 May 2023 19:52:47 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T19:52:47.177 DEBUG:boto:path=/pklxkd-3/ 2023-05-31T19:52:47.178 DEBUG:boto:auth_path=/pklxkd-3/ 2023-05-31T19:52:47.178 DEBUG:boto:Method: HEAD 2023-05-31T19:52:47.178 DEBUG:boto:Path: /pklxkd-3/ 2023-05-31T19:52:47.179 DEBUG:boto:Data: 2023-05-31T19:52:47.179 DEBUG:boto:Headers: {} 2023-05-31T19:52:47.179 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T19:52:47.180 DEBUG:boto:Port: 8001 2023-05-31T19:52:47.180 DEBUG:boto:Params: {} 2023-05-31T19:52:47.180 DEBUG:boto:Token: None 2023-05-31T19:52:47.181 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 19:52:47 GMT /pklxkd-3/ 2023-05-31T19:52:47.181 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:7bfA2AqQ3VoKUEJ3Myv/PLGjW7Y= 2023-05-31T19:52:47.182 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 19:52:47 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:7bfA2AqQ3VoKUEJ3Myv/PLGjW7Y=', 'Content-Length': '0'} 2023-05-31T19:52:47.184 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000005f8a037d3dafa35f-006477a58f-1112-a2'), ('Content-Length', '0'), ('Date', 'Wed, 31 May 2023 19:52:47 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T19:52:47.184 DEBUG:boto:path=/pklxkd-4/ 2023-05-31T19:52:47.185 DEBUG:boto:auth_path=/pklxkd-4/ 2023-05-31T19:52:47.185 DEBUG:boto:Method: HEAD 2023-05-31T19:52:47.185 DEBUG:boto:Path: /pklxkd-4/ 2023-05-31T19:52:47.186 DEBUG:boto:Data: 2023-05-31T19:52:47.186 DEBUG:boto:Headers: {} 2023-05-31T19:52:47.187 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T19:52:47.187 DEBUG:boto:Port: 8001 2023-05-31T19:52:47.188 DEBUG:boto:Params: {} 2023-05-31T19:52:47.188 DEBUG:boto:Token: None 2023-05-31T19:52:47.189 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 19:52:47 GMT /pklxkd-4/ 2023-05-31T19:52:47.190 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:5PKO58LYc8zYRxP1RMNxWZoSg8s= 2023-05-31T19:52:47.190 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 19:52:47 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:5PKO58LYc8zYRxP1RMNxWZoSg8s=', 'Content-Length': '0'} 2023-05-31T19:52:47.193 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000006c173df7236f0583-006477a58f-1112-a2'), ('Content-Length', '0'), ('Date', 'Wed, 31 May 2023 19:52:47 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T19:52:47.194 INFO:tasks.rgw_multisite_tests:rgw_multi.tests.test_bucket_recreate ... ok 2023-05-31T19:52:47.195 INFO:rgw_multi.tests:create bucket zone=a1 name=pklxkd-5 2023-05-31T19:52:47.196 DEBUG:boto:path=/pklxkd-5/ 2023-05-31T19:52:47.197 DEBUG:boto:auth_path=/pklxkd-5/ 2023-05-31T19:52:47.198 DEBUG:boto:Method: PUT 2023-05-31T19:52:47.198 DEBUG:boto:Path: /pklxkd-5/ 2023-05-31T19:52:47.199 DEBUG:boto:Data: 2023-05-31T19:52:47.199 DEBUG:boto:Headers: {} 2023-05-31T19:52:47.200 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T19:52:47.200 DEBUG:boto:Port: 8000 2023-05-31T19:52:47.200 DEBUG:boto:Params: {} 2023-05-31T19:52:47.201 DEBUG:boto:Token: None 2023-05-31T19:52:47.201 DEBUG:boto:StringToSign: PUT Wed, 31 May 2023 19:52:47 GMT /pklxkd-5/ 2023-05-31T19:52:47.202 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:KOCKQJCcDZSe9FND/IdP7NDuaAk= 2023-05-31T19:52:47.202 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 19:52:47 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:KOCKQJCcDZSe9FND/IdP7NDuaAk=', 'Content-Length': '0'} 2023-05-31T19:52:47.214 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx000007c4d51b6ff2c0ba4-006477a58f-110e-a1'), ('Content-Length', '0'), ('Date', 'Wed, 31 May 2023 19:52:47 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T19:52:47.214 INFO:rgw_multi.tests:create bucket zone=a2 name=pklxkd-6 2023-05-31T19:52:47.215 DEBUG:boto:path=/pklxkd-6/ 2023-05-31T19:52:47.215 DEBUG:boto:auth_path=/pklxkd-6/ 2023-05-31T19:52:47.215 DEBUG:boto:Method: PUT 2023-05-31T19:52:47.216 DEBUG:boto:Path: /pklxkd-6/ 2023-05-31T19:52:47.216 DEBUG:boto:Data: 2023-05-31T19:52:47.217 DEBUG:boto:Headers: {} 2023-05-31T19:52:47.217 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T19:52:47.218 DEBUG:boto:Port: 8001 2023-05-31T19:52:47.218 DEBUG:boto:Params: {} 2023-05-31T19:52:47.218 DEBUG:boto:Token: None 2023-05-31T19:52:47.219 DEBUG:boto:StringToSign: PUT Wed, 31 May 2023 19:52:47 GMT /pklxkd-6/ 2023-05-31T19:52:47.220 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:VAKgS79QygvOmMowJW5ad1V//qU= 2023-05-31T19:52:47.221 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 19:52:47 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:VAKgS79QygvOmMowJW5ad1V//qU=', 'Content-Length': '0'} 2023-05-31T19:52:47.242 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx000006f5462b8dceaeeea-006477a58f-1112-a2'), ('Content-Length', '0'), ('Date', 'Wed, 31 May 2023 19:52:47 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T19:52:47.242 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-31T19:52:47.243 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-31T19:52:47.243 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-05-31T19:52:47.293 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2023-05-31T19:52:47.293 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2023-05-31T19:52:47.331 INFO:teuthology.orchestra.run.smithi099.stderr:No --period given, using current period=2e6123d2-d695-4350-9a01-61a455bc782b 2023-05-31T19:52:47.351 INFO:teuthology.orchestra.run.smithi099.stdout:[ 2023-05-31T19:52:47.351 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T19:52:47.351 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1685562631.516089_59.1", 2023-05-31T19:52:47.352 INFO:teuthology.orchestra.run.smithi099.stdout: "last_update": "2023-05-31T19:50:31.516089Z" 2023-05-31T19:52:47.352 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T19:52:47.352 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T19:52:47.352 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "", 2023-05-31T19:52:47.352 INFO:teuthology.orchestra.run.smithi099.stdout: "last_update": "0.000000" 2023-05-31T19:52:47.353 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T19:52:47.353 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T19:52:47.353 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1685562767.228641_63.1", 2023-05-31T19:52:47.353 INFO:teuthology.orchestra.run.smithi099.stdout: "last_update": "2023-05-31T19:52:47.228641Z" 2023-05-31T19:52:47.353 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T19:52:47.354 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T19:52:47.354 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "", 2023-05-31T19:52:47.354 INFO:teuthology.orchestra.run.smithi099.stdout: "last_update": "0.000000" 2023-05-31T19:52:47.354 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T19:52:47.355 INFO:teuthology.orchestra.run.smithi099.stdout:] 2023-05-31T19:52:47.356 DEBUG:rgw_multi.tests:master meta markers={0: '1_1685562631.516089_59.1', 1: '', 2: '1_1685562767.228641_63.1', 3: ''} 2023-05-31T19:52:47.357 INFO:rgw_multi.tests:starting meta checkpoint for zone=a2 2023-05-31T19:52:47.357 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-31T19:52:47.357 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-31T19:52:47.358 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-05-31T19:52:47.410 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2023-05-31T19:52:47.410 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2023-05-31T19:52:47.457 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T19:52:47.452+0000 7f2821122780 20 RGW-SYNC:meta: read sync status 2023-05-31T19:52:47.458 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T19:52:47.453+0000 7f2821122780 20 RGW-SYNC:meta: read sync status 2023-05-31T19:52:47.469 INFO:teuthology.orchestra.run.smithi099.stdout:{ 2023-05-31T19:52:47.469 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_status": { 2023-05-31T19:52:47.469 INFO:teuthology.orchestra.run.smithi099.stdout: "info": { 2023-05-31T19:52:47.470 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "sync", 2023-05-31T19:52:47.470 INFO:teuthology.orchestra.run.smithi099.stdout: "num_shards": 4, 2023-05-31T19:52:47.470 INFO:teuthology.orchestra.run.smithi099.stdout: "period": "2e6123d2-d695-4350-9a01-61a455bc782b", 2023-05-31T19:52:47.470 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2023-05-31T19:52:47.471 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T19:52:47.471 INFO:teuthology.orchestra.run.smithi099.stdout: "markers": [ 2023-05-31T19:52:47.471 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T19:52:47.472 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 0, 2023-05-31T19:52:47.472 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T19:52:47.472 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2023-05-31T19:52:47.472 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1685562631.516089_59.1", 2023-05-31T19:52:47.473 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T19:52:47.473 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 1, 2023-05-31T19:52:47.473 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T19:52:47.473 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2023-05-31T19:50:31.516089Z", 2023-05-31T19:52:47.473 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2023-05-31T19:52:47.473 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T19:52:47.473 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T19:52:47.474 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T19:52:47.474 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 1, 2023-05-31T19:52:47.474 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T19:52:47.474 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2023-05-31T19:52:47.474 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "", 2023-05-31T19:52:47.475 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T19:52:47.475 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2023-05-31T19:52:47.475 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T19:52:47.475 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "0.000000", 2023-05-31T19:52:47.475 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2023-05-31T19:52:47.475 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T19:52:47.476 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T19:52:47.476 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T19:52:47.477 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 2, 2023-05-31T19:52:47.477 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T19:52:47.477 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2023-05-31T19:52:47.477 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1685562767.228641_63.1", 2023-05-31T19:52:47.477 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T19:52:47.478 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2023-05-31T19:52:47.478 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T19:52:47.478 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2023-05-31T19:52:47.228641Z", 2023-05-31T19:52:47.478 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2023-05-31T19:52:47.478 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T19:52:47.478 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T19:52:47.478 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T19:52:47.479 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 3, 2023-05-31T19:52:47.479 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T19:52:47.479 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2023-05-31T19:52:47.479 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "", 2023-05-31T19:52:47.479 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T19:52:47.479 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2023-05-31T19:52:47.480 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T19:52:47.480 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "0.000000", 2023-05-31T19:52:47.480 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2023-05-31T19:52:47.480 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T19:52:47.480 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T19:52:47.480 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2023-05-31T19:52:47.481 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T19:52:47.481 INFO:teuthology.orchestra.run.smithi099.stdout: "full_sync": { 2023-05-31T19:52:47.481 INFO:teuthology.orchestra.run.smithi099.stdout: "total": 1, 2023-05-31T19:52:47.481 INFO:teuthology.orchestra.run.smithi099.stdout: "complete": 1 2023-05-31T19:52:47.481 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T19:52:47.481 INFO:teuthology.orchestra.run.smithi099.stdout: "current_time": "2023-05-31T19:52:47Z" 2023-05-31T19:52:47.482 INFO:teuthology.orchestra.run.smithi099.stdout:} 2023-05-31T19:52:47.482 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "2e6123d2-d695-4350-9a01-61a455bc782b", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1685562631.516089_59.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-05-31T19:50:31.516089Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1685562767.228641_63.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-05-31T19:52:47.228641Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-05-31T19:52:47Z" } 2023-05-31T19:52:47.483 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1685562631.516089_59.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-05-31T19:50:31.516089Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1685562767.228641_63.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-05-31T19:52:47.228641Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2023-05-31T19:52:47.484 DEBUG:rgw_multi.tests:log_status={0: '1_1685562631.516089_59.1', 1: '', 2: '1_1685562767.228641_63.1', 3: ''} 2023-05-31T19:52:47.484 DEBUG:rgw_multi.tests:sync_status={0: '1_1685562631.516089_59.1', 1: '', 2: '1_1685562767.228641_63.1', 3: ''} 2023-05-31T19:52:47.485 INFO:rgw_multi.tests:finish meta checkpoint for zone=a2 2023-05-31T19:52:47.485 DEBUG:boto:path=/pklxkd-5/ 2023-05-31T19:52:47.486 DEBUG:boto:auth_path=/pklxkd-5/ 2023-05-31T19:52:47.486 DEBUG:boto:Method: HEAD 2023-05-31T19:52:47.486 DEBUG:boto:Path: /pklxkd-5/ 2023-05-31T19:52:47.487 DEBUG:boto:Data: 2023-05-31T19:52:47.487 DEBUG:boto:Headers: {} 2023-05-31T19:52:47.487 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T19:52:47.488 DEBUG:boto:Port: 8000 2023-05-31T19:52:47.488 DEBUG:boto:Params: {} 2023-05-31T19:52:47.489 DEBUG:boto:Token: None 2023-05-31T19:52:47.489 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 19:52:47 GMT /pklxkd-5/ 2023-05-31T19:52:47.490 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:/YO9+uH7i/nj0TSLFkT5GqSQ83I= 2023-05-31T19:52:47.490 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 19:52:47 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:/YO9+uH7i/nj0TSLFkT5GqSQ83I=', 'Content-Length': '0'} 2023-05-31T19:52:47.492 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000aab80d90082757ed-006477a58f-110e-a1'), ('Content-Length', '0'), ('Date', 'Wed, 31 May 2023 19:52:47 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T19:52:47.493 DEBUG:boto:path=/pklxkd-6/ 2023-05-31T19:52:47.493 DEBUG:boto:auth_path=/pklxkd-6/ 2023-05-31T19:52:47.493 DEBUG:boto:Method: HEAD 2023-05-31T19:52:47.494 DEBUG:boto:Path: /pklxkd-6/ 2023-05-31T19:52:47.494 DEBUG:boto:Data: 2023-05-31T19:52:47.494 DEBUG:boto:Headers: {} 2023-05-31T19:52:47.495 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T19:52:47.495 DEBUG:boto:Port: 8000 2023-05-31T19:52:47.495 DEBUG:boto:Params: {} 2023-05-31T19:52:47.496 DEBUG:boto:Token: None 2023-05-31T19:52:47.496 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 19:52:47 GMT /pklxkd-6/ 2023-05-31T19:52:47.497 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:ly/QDa/7M07apOPmHAYVyrE6AiI= 2023-05-31T19:52:47.497 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 19:52:47 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:ly/QDa/7M07apOPmHAYVyrE6AiI=', 'Content-Length': '0'} 2023-05-31T19:52:47.499 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000a0059c1820821844-006477a58f-110e-a1'), ('Content-Length', '0'), ('Date', 'Wed, 31 May 2023 19:52:47 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T19:52:47.500 DEBUG:boto:path=/pklxkd-5/ 2023-05-31T19:52:47.500 DEBUG:boto:auth_path=/pklxkd-5/ 2023-05-31T19:52:47.500 DEBUG:boto:Method: HEAD 2023-05-31T19:52:47.501 DEBUG:boto:Path: /pklxkd-5/ 2023-05-31T19:52:47.501 DEBUG:boto:Data: 2023-05-31T19:52:47.501 DEBUG:boto:Headers: {} 2023-05-31T19:52:47.502 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T19:52:47.502 DEBUG:boto:Port: 8001 2023-05-31T19:52:47.502 DEBUG:boto:Params: {} 2023-05-31T19:52:47.503 DEBUG:boto:Token: None 2023-05-31T19:52:47.503 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 19:52:47 GMT /pklxkd-5/ 2023-05-31T19:52:47.504 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:/YO9+uH7i/nj0TSLFkT5GqSQ83I= 2023-05-31T19:52:47.504 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 19:52:47 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:/YO9+uH7i/nj0TSLFkT5GqSQ83I=', 'Content-Length': '0'} 2023-05-31T19:52:47.506 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000005587ac2a2652a077-006477a58f-1112-a2'), ('Content-Length', '0'), ('Date', 'Wed, 31 May 2023 19:52:47 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T19:52:47.507 DEBUG:boto:path=/pklxkd-6/ 2023-05-31T19:52:47.507 DEBUG:boto:auth_path=/pklxkd-6/ 2023-05-31T19:52:47.508 DEBUG:boto:Method: HEAD 2023-05-31T19:52:47.508 DEBUG:boto:Path: /pklxkd-6/ 2023-05-31T19:52:47.508 DEBUG:boto:Data: 2023-05-31T19:52:47.509 DEBUG:boto:Headers: {} 2023-05-31T19:52:47.509 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T19:52:47.510 DEBUG:boto:Port: 8001 2023-05-31T19:52:47.510 DEBUG:boto:Params: {} 2023-05-31T19:52:47.511 DEBUG:boto:Token: None 2023-05-31T19:52:47.511 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 19:52:47 GMT /pklxkd-6/ 2023-05-31T19:52:47.512 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:ly/QDa/7M07apOPmHAYVyrE6AiI= 2023-05-31T19:52:47.512 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 19:52:47 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:ly/QDa/7M07apOPmHAYVyrE6AiI=', 'Content-Length': '0'} 2023-05-31T19:52:47.515 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000d757200e549a1d10-006477a58f-1112-a2'), ('Content-Length', '0'), ('Date', 'Wed, 31 May 2023 19:52:47 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T19:52:47.515 DEBUG:boto:path=/pklxkd-5/ 2023-05-31T19:52:47.515 DEBUG:boto:auth_path=/pklxkd-5/ 2023-05-31T19:52:47.516 DEBUG:boto:Method: DELETE 2023-05-31T19:52:47.516 DEBUG:boto:Path: /pklxkd-5/ 2023-05-31T19:52:47.516 DEBUG:boto:Data: 2023-05-31T19:52:47.517 DEBUG:boto:Headers: {} 2023-05-31T19:52:47.517 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T19:52:47.517 DEBUG:boto:Port: 8000 2023-05-31T19:52:47.518 DEBUG:boto:Params: {} 2023-05-31T19:52:47.518 DEBUG:boto:Token: None 2023-05-31T19:52:47.519 DEBUG:boto:StringToSign: DELETE Wed, 31 May 2023 19:52:47 GMT /pklxkd-5/ 2023-05-31T19:52:47.519 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:5HXldj7WvbcaT2nvTO8ISEuoQs4= 2023-05-31T19:52:47.519 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 19:52:47 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:5HXldj7WvbcaT2nvTO8ISEuoQs4=', 'Content-Length': '0'} 2023-05-31T19:52:47.545 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000b205fc2b85c7eade-006477a58f-110e-a1'), ('Date', 'Wed, 31 May 2023 19:52:47 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T19:52:47.547 DEBUG:boto:path=/pklxkd-6/ 2023-05-31T19:52:47.547 DEBUG:boto:auth_path=/pklxkd-6/ 2023-05-31T19:52:47.547 DEBUG:boto:Method: DELETE 2023-05-31T19:52:47.548 DEBUG:boto:Path: /pklxkd-6/ 2023-05-31T19:52:47.548 DEBUG:boto:Data: 2023-05-31T19:52:47.548 DEBUG:boto:Headers: {} 2023-05-31T19:52:47.549 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T19:52:47.549 DEBUG:boto:Port: 8001 2023-05-31T19:52:47.549 DEBUG:boto:Params: {} 2023-05-31T19:52:47.550 DEBUG:boto:Token: None 2023-05-31T19:52:47.550 DEBUG:boto:StringToSign: DELETE Wed, 31 May 2023 19:52:47 GMT /pklxkd-6/ 2023-05-31T19:52:47.551 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:FqiphaWBvM7XmruLtLLKzsLadZI= 2023-05-31T19:52:47.551 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 19:52:47 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:FqiphaWBvM7XmruLtLLKzsLadZI=', 'Content-Length': '0'} 2023-05-31T19:52:47.625 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx0000098c23ca28d2f16b4-006477a58f-1112-a2'), ('Date', 'Wed, 31 May 2023 19:52:47 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T19:52:47.626 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-31T19:52:47.626 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-31T19:52:47.626 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-05-31T19:52:47.674 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2023-05-31T19:52:47.675 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2023-05-31T19:52:47.718 INFO:teuthology.orchestra.run.smithi099.stderr:No --period given, using current period=2e6123d2-d695-4350-9a01-61a455bc782b 2023-05-31T19:52:47.735 INFO:teuthology.orchestra.run.smithi099.stdout:[ 2023-05-31T19:52:47.735 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T19:52:47.736 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1685562631.516089_59.1", 2023-05-31T19:52:47.736 INFO:teuthology.orchestra.run.smithi099.stdout: "last_update": "2023-05-31T19:50:31.516089Z" 2023-05-31T19:52:47.736 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T19:52:47.737 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T19:52:47.737 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "", 2023-05-31T19:52:47.737 INFO:teuthology.orchestra.run.smithi099.stdout: "last_update": "0.000000" 2023-05-31T19:52:47.737 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T19:52:47.738 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T19:52:47.738 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1685562767.575580_71.1", 2023-05-31T19:52:47.738 INFO:teuthology.orchestra.run.smithi099.stdout: "last_update": "2023-05-31T19:52:47.575580Z" 2023-05-31T19:52:47.738 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T19:52:47.739 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T19:52:47.739 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "", 2023-05-31T19:52:47.739 INFO:teuthology.orchestra.run.smithi099.stdout: "last_update": "0.000000" 2023-05-31T19:52:47.740 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T19:52:47.740 INFO:teuthology.orchestra.run.smithi099.stdout:] 2023-05-31T19:52:47.741 DEBUG:rgw_multi.tests:master meta markers={0: '1_1685562631.516089_59.1', 1: '', 2: '1_1685562767.575580_71.1', 3: ''} 2023-05-31T19:52:47.742 INFO:rgw_multi.tests:starting meta checkpoint for zone=a2 2023-05-31T19:52:47.743 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-31T19:52:47.743 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-31T19:52:47.744 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-05-31T19:52:47.796 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2023-05-31T19:52:47.796 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2023-05-31T19:52:47.837 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T19:52:47.832+0000 7fad22d6f780 20 RGW-SYNC:meta: read sync status 2023-05-31T19:52:47.842 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T19:52:47.833+0000 7fad22d6f780 20 RGW-SYNC:meta: read sync status 2023-05-31T19:52:47.860 INFO:teuthology.orchestra.run.smithi099.stdout:{ 2023-05-31T19:52:47.860 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_status": { 2023-05-31T19:52:47.860 INFO:teuthology.orchestra.run.smithi099.stdout: "info": { 2023-05-31T19:52:47.861 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "sync", 2023-05-31T19:52:47.861 INFO:teuthology.orchestra.run.smithi099.stdout: "num_shards": 4, 2023-05-31T19:52:47.861 INFO:teuthology.orchestra.run.smithi099.stdout: "period": "2e6123d2-d695-4350-9a01-61a455bc782b", 2023-05-31T19:52:47.861 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2023-05-31T19:52:47.861 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T19:52:47.861 INFO:teuthology.orchestra.run.smithi099.stdout: "markers": [ 2023-05-31T19:52:47.861 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T19:52:47.862 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 0, 2023-05-31T19:52:47.862 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T19:52:47.862 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2023-05-31T19:52:47.862 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1685562631.516089_59.1", 2023-05-31T19:52:47.862 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T19:52:47.863 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 1, 2023-05-31T19:52:47.863 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T19:52:47.863 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2023-05-31T19:50:31.516089Z", 2023-05-31T19:52:47.863 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2023-05-31T19:52:47.863 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T19:52:47.863 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T19:52:47.863 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T19:52:47.864 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 1, 2023-05-31T19:52:47.864 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T19:52:47.864 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2023-05-31T19:52:47.864 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "", 2023-05-31T19:52:47.864 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T19:52:47.864 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2023-05-31T19:52:47.865 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T19:52:47.865 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "0.000000", 2023-05-31T19:52:47.865 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2023-05-31T19:52:47.865 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T19:52:47.865 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T19:52:47.865 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T19:52:47.865 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 2, 2023-05-31T19:52:47.866 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T19:52:47.866 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2023-05-31T19:52:47.866 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1685562767.575580_71.1", 2023-05-31T19:52:47.866 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T19:52:47.866 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2023-05-31T19:52:47.867 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T19:52:47.867 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2023-05-31T19:52:47.575580Z", 2023-05-31T19:52:47.867 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2023-05-31T19:52:47.867 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T19:52:47.868 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T19:52:47.868 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T19:52:47.868 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 3, 2023-05-31T19:52:47.868 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T19:52:47.869 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2023-05-31T19:52:47.869 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "", 2023-05-31T19:52:47.869 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T19:52:47.869 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2023-05-31T19:52:47.869 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T19:52:47.870 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "0.000000", 2023-05-31T19:52:47.870 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2023-05-31T19:52:47.870 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T19:52:47.870 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T19:52:47.870 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2023-05-31T19:52:47.871 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T19:52:47.871 INFO:teuthology.orchestra.run.smithi099.stdout: "full_sync": { 2023-05-31T19:52:47.871 INFO:teuthology.orchestra.run.smithi099.stdout: "total": 1, 2023-05-31T19:52:47.871 INFO:teuthology.orchestra.run.smithi099.stdout: "complete": 1 2023-05-31T19:52:47.871 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T19:52:47.871 INFO:teuthology.orchestra.run.smithi099.stdout: "current_time": "2023-05-31T19:52:47Z" 2023-05-31T19:52:47.871 INFO:teuthology.orchestra.run.smithi099.stdout:} 2023-05-31T19:52:47.872 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "2e6123d2-d695-4350-9a01-61a455bc782b", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1685562631.516089_59.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-05-31T19:50:31.516089Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1685562767.575580_71.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-05-31T19:52:47.575580Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-05-31T19:52:47Z" } 2023-05-31T19:52:47.873 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1685562631.516089_59.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-05-31T19:50:31.516089Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1685562767.575580_71.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-05-31T19:52:47.575580Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2023-05-31T19:52:47.873 DEBUG:rgw_multi.tests:log_status={0: '1_1685562631.516089_59.1', 1: '', 2: '1_1685562767.575580_71.1', 3: ''} 2023-05-31T19:52:47.874 DEBUG:rgw_multi.tests:sync_status={0: '1_1685562631.516089_59.1', 1: '', 2: '1_1685562767.575580_71.1', 3: ''} 2023-05-31T19:52:47.874 INFO:rgw_multi.tests:finish meta checkpoint for zone=a2 2023-05-31T19:52:47.874 DEBUG:boto:path=/pklxkd-5/ 2023-05-31T19:52:47.875 DEBUG:boto:auth_path=/pklxkd-5/ 2023-05-31T19:52:47.875 DEBUG:boto:Method: HEAD 2023-05-31T19:52:47.876 DEBUG:boto:Path: /pklxkd-5/ 2023-05-31T19:52:47.876 DEBUG:boto:Data: 2023-05-31T19:52:47.877 DEBUG:boto:Headers: {} 2023-05-31T19:52:47.877 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T19:52:47.878 DEBUG:boto:Port: 8000 2023-05-31T19:52:47.878 DEBUG:boto:Params: {} 2023-05-31T19:52:47.879 DEBUG:boto:Token: None 2023-05-31T19:52:47.879 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 19:52:47 GMT /pklxkd-5/ 2023-05-31T19:52:47.880 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:/YO9+uH7i/nj0TSLFkT5GqSQ83I= 2023-05-31T19:52:47.880 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 19:52:47 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:/YO9+uH7i/nj0TSLFkT5GqSQ83I=', 'Content-Length': '0'} 2023-05-31T19:52:47.881 DEBUG:boto:Response headers: [('Content-Length', '221'), ('x-amz-request-id', 'tx000007fd5d58c60186b7b-006477a58f-110e-a1'), ('Accept-Ranges', 'bytes'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 19:52:47 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T19:52:47.882 DEBUG:boto:path=/pklxkd-6/ 2023-05-31T19:52:47.882 DEBUG:boto:auth_path=/pklxkd-6/ 2023-05-31T19:52:47.883 DEBUG:boto:Method: HEAD 2023-05-31T19:52:47.883 DEBUG:boto:Path: /pklxkd-6/ 2023-05-31T19:52:47.883 DEBUG:boto:Data: 2023-05-31T19:52:47.884 DEBUG:boto:Headers: {} 2023-05-31T19:52:47.884 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T19:52:47.885 DEBUG:boto:Port: 8000 2023-05-31T19:52:47.885 DEBUG:boto:Params: {} 2023-05-31T19:52:47.886 DEBUG:boto:Token: None 2023-05-31T19:52:47.886 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 19:52:47 GMT /pklxkd-6/ 2023-05-31T19:52:47.886 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:ly/QDa/7M07apOPmHAYVyrE6AiI= 2023-05-31T19:52:47.887 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 19:52:47 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:ly/QDa/7M07apOPmHAYVyrE6AiI=', 'Content-Length': '0'} 2023-05-31T19:52:47.888 DEBUG:boto:Response headers: [('Content-Length', '221'), ('x-amz-request-id', 'tx00000d8d7d61b20f7655d-006477a58f-110e-a1'), ('Accept-Ranges', 'bytes'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 19:52:47 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T19:52:47.889 DEBUG:boto:path=/pklxkd-5/ 2023-05-31T19:52:47.889 DEBUG:boto:auth_path=/pklxkd-5/ 2023-05-31T19:52:47.890 DEBUG:boto:Method: HEAD 2023-05-31T19:52:47.890 DEBUG:boto:Path: /pklxkd-5/ 2023-05-31T19:52:47.891 DEBUG:boto:Data: 2023-05-31T19:52:47.891 DEBUG:boto:Headers: {} 2023-05-31T19:52:47.892 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T19:52:47.893 DEBUG:boto:Port: 8001 2023-05-31T19:52:47.893 DEBUG:boto:Params: {} 2023-05-31T19:52:47.894 DEBUG:boto:Token: None 2023-05-31T19:52:47.895 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 19:52:47 GMT /pklxkd-5/ 2023-05-31T19:52:47.895 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:/YO9+uH7i/nj0TSLFkT5GqSQ83I= 2023-05-31T19:52:47.896 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 19:52:47 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:/YO9+uH7i/nj0TSLFkT5GqSQ83I=', 'Content-Length': '0'} 2023-05-31T19:52:47.897 DEBUG:boto:Response headers: [('Content-Length', '221'), ('x-amz-request-id', 'tx00000a62502aef7150b32-006477a58f-1112-a2'), ('Accept-Ranges', 'bytes'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 19:52:47 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T19:52:47.898 DEBUG:boto:path=/pklxkd-6/ 2023-05-31T19:52:47.898 DEBUG:boto:auth_path=/pklxkd-6/ 2023-05-31T19:52:47.898 DEBUG:boto:Method: HEAD 2023-05-31T19:52:47.899 DEBUG:boto:Path: /pklxkd-6/ 2023-05-31T19:52:47.899 DEBUG:boto:Data: 2023-05-31T19:52:47.899 DEBUG:boto:Headers: {} 2023-05-31T19:52:47.900 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T19:52:47.900 DEBUG:boto:Port: 8001 2023-05-31T19:52:47.900 DEBUG:boto:Params: {} 2023-05-31T19:52:47.901 DEBUG:boto:Token: None 2023-05-31T19:52:47.901 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 19:52:47 GMT /pklxkd-6/ 2023-05-31T19:52:47.902 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:ly/QDa/7M07apOPmHAYVyrE6AiI= 2023-05-31T19:52:47.902 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 19:52:47 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:ly/QDa/7M07apOPmHAYVyrE6AiI=', 'Content-Length': '0'} 2023-05-31T19:52:47.903 DEBUG:boto:Response headers: [('Content-Length', '221'), ('x-amz-request-id', 'tx000002bf7a543ef28420d-006477a58f-1112-a2'), ('Accept-Ranges', 'bytes'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 19:52:47 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T19:52:47.904 INFO:tasks.rgw_multisite_tests:rgw_multi.tests.test_bucket_remove ... ok 2023-05-31T19:52:47.904 INFO:rgw_multi.tests:create bucket zone=a1 name=pklxkd-7 2023-05-31T19:52:47.905 DEBUG:boto:path=/pklxkd-7/ 2023-05-31T19:52:47.905 DEBUG:boto:auth_path=/pklxkd-7/ 2023-05-31T19:52:47.906 DEBUG:boto:Method: PUT 2023-05-31T19:52:47.906 DEBUG:boto:Path: /pklxkd-7/ 2023-05-31T19:52:47.906 DEBUG:boto:Data: 2023-05-31T19:52:47.907 DEBUG:boto:Headers: {} 2023-05-31T19:52:47.907 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T19:52:47.907 DEBUG:boto:Port: 8000 2023-05-31T19:52:47.908 DEBUG:boto:Params: {} 2023-05-31T19:52:47.908 DEBUG:boto:Token: None 2023-05-31T19:52:47.909 DEBUG:boto:StringToSign: PUT Wed, 31 May 2023 19:52:47 GMT /pklxkd-7/ 2023-05-31T19:52:47.909 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:enD1BpfZ8Ac40u9XspS8c55lnl4= 2023-05-31T19:52:47.910 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 19:52:47 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:enD1BpfZ8Ac40u9XspS8c55lnl4=', 'Content-Length': '0'} 2023-05-31T19:52:47.938 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000ff937eab5a78385d-006477a58f-110e-a1'), ('Content-Length', '0'), ('Date', 'Wed, 31 May 2023 19:52:47 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T19:52:47.939 INFO:rgw_multi.tests:create bucket zone=a2 name=pklxkd-8 2023-05-31T19:52:47.939 DEBUG:boto:path=/pklxkd-8/ 2023-05-31T19:52:47.940 DEBUG:boto:auth_path=/pklxkd-8/ 2023-05-31T19:52:47.940 DEBUG:boto:Method: PUT 2023-05-31T19:52:47.941 DEBUG:boto:Path: /pklxkd-8/ 2023-05-31T19:52:47.941 DEBUG:boto:Data: 2023-05-31T19:52:47.942 DEBUG:boto:Headers: {} 2023-05-31T19:52:47.942 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T19:52:47.942 DEBUG:boto:Port: 8001 2023-05-31T19:52:47.943 DEBUG:boto:Params: {} 2023-05-31T19:52:47.943 DEBUG:boto:Token: None 2023-05-31T19:52:47.943 DEBUG:boto:StringToSign: PUT Wed, 31 May 2023 19:52:47 GMT /pklxkd-8/ 2023-05-31T19:52:47.944 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:gf59e1KthzZMCif2LYwtgc3eJRw= 2023-05-31T19:52:47.945 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 19:52:47 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:gf59e1KthzZMCif2LYwtgc3eJRw=', 'Content-Length': '0'} 2023-05-31T19:52:47.986 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000d1494e538248c5cc-006477a58f-1112-a2'), ('Content-Length', '0'), ('Date', 'Wed, 31 May 2023 19:52:47 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T19:52:48.063 DEBUG:boto:path=/pklxkd-7/ 2023-05-31T19:52:48.064 DEBUG:boto:auth_path=/pklxkd-7/ 2023-05-31T19:52:48.064 DEBUG:boto:Method: HEAD 2023-05-31T19:52:48.064 DEBUG:boto:Path: /pklxkd-7/ 2023-05-31T19:52:48.065 DEBUG:boto:Data: 2023-05-31T19:52:48.065 DEBUG:boto:Headers: {} 2023-05-31T19:52:48.065 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T19:52:48.066 DEBUG:boto:Port: 8000 2023-05-31T19:52:48.066 DEBUG:boto:Params: {} 2023-05-31T19:52:48.067 DEBUG:boto:Token: None 2023-05-31T19:52:48.067 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 19:52:48 GMT /pklxkd-7/ 2023-05-31T19:52:48.068 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:lg1RckkP8N68jVLSlmh/MmDojh0= 2023-05-31T19:52:48.068 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 19:52:48 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:lg1RckkP8N68jVLSlmh/MmDojh0=', 'Content-Length': '0'} 2023-05-31T19:52:48.070 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000b5bc7eb37d5541fd-006477a590-110e-a1'), ('Content-Length', '0'), ('Date', 'Wed, 31 May 2023 19:52:48 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T19:52:48.071 DEBUG:boto:path=/pklxkd-7/myobj 2023-05-31T19:52:48.072 DEBUG:boto:auth_path=/pklxkd-7/myobj 2023-05-31T19:52:48.072 DEBUG:boto:Method: PUT 2023-05-31T19:52:48.072 DEBUG:boto:Path: /pklxkd-7/myobj 2023-05-31T19:52:48.073 DEBUG:boto:Data: 2023-05-31T19:52:48.073 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2023-05-31T19:52:48.073 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T19:52:48.074 DEBUG:boto:Port: 8000 2023-05-31T19:52:48.074 DEBUG:boto:Params: {} 2023-05-31T19:52:48.074 DEBUG:boto:Token: None 2023-05-31T19:52:48.075 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Wed, 31 May 2023 19:52:48 GMT /pklxkd-7/myobj 2023-05-31T19:52:48.075 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:Rbo2aRd8ziO3dejnzFbSOjkDy0E= 2023-05-31T19:52:48.076 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Wed, 31 May 2023 19:52:48 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:Rbo2aRd8ziO3dejnzFbSOjkDy0E='} 2023-05-31T19:52:48.088 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000004e62c6b9b43fb51e-006477a590-110e-a1'), ('Date', 'Wed, 31 May 2023 19:52:48 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T19:52:48.088 DEBUG:boto:path=/pklxkd-7/ 2023-05-31T19:52:48.089 DEBUG:boto:auth_path=/pklxkd-7/ 2023-05-31T19:52:48.089 DEBUG:boto:Method: HEAD 2023-05-31T19:52:48.089 DEBUG:boto:Path: /pklxkd-7/ 2023-05-31T19:52:48.090 DEBUG:boto:Data: 2023-05-31T19:52:48.090 DEBUG:boto:Headers: {} 2023-05-31T19:52:48.091 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T19:52:48.091 DEBUG:boto:Port: 8000 2023-05-31T19:52:48.092 DEBUG:boto:Params: {} 2023-05-31T19:52:48.092 DEBUG:boto:Token: None 2023-05-31T19:52:48.093 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 19:52:48 GMT /pklxkd-7/ 2023-05-31T19:52:48.093 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:lg1RckkP8N68jVLSlmh/MmDojh0= 2023-05-31T19:52:48.093 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 19:52:48 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:lg1RckkP8N68jVLSlmh/MmDojh0=', 'Content-Length': '0'} 2023-05-31T19:52:48.096 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '6'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000ab138c01744df99a-006477a590-110e-a1'), ('Content-Length', '0'), ('Date', 'Wed, 31 May 2023 19:52:48 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T19:52:48.096 DEBUG:boto:path=/pklxkd-7/_myobj 2023-05-31T19:52:48.097 DEBUG:boto:auth_path=/pklxkd-7/_myobj 2023-05-31T19:52:48.097 DEBUG:boto:Method: PUT 2023-05-31T19:52:48.097 DEBUG:boto:Path: /pklxkd-7/_myobj 2023-05-31T19:52:48.098 DEBUG:boto:Data: 2023-05-31T19:52:48.098 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2023-05-31T19:52:48.098 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T19:52:48.099 DEBUG:boto:Port: 8000 2023-05-31T19:52:48.099 DEBUG:boto:Params: {} 2023-05-31T19:52:48.099 DEBUG:boto:Token: None 2023-05-31T19:52:48.100 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Wed, 31 May 2023 19:52:48 GMT /pklxkd-7/_myobj 2023-05-31T19:52:48.104 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:RCYLTtQmxIHNHj66XZhdXBEq0so= 2023-05-31T19:52:48.105 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Wed, 31 May 2023 19:52:48 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:RCYLTtQmxIHNHj66XZhdXBEq0so='} 2023-05-31T19:52:48.152 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000033cdedbe5a2fed00-006477a590-110e-a1'), ('Date', 'Wed, 31 May 2023 19:52:48 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T19:52:48.153 DEBUG:boto:path=/pklxkd-7/ 2023-05-31T19:52:48.153 DEBUG:boto:auth_path=/pklxkd-7/ 2023-05-31T19:52:48.154 DEBUG:boto:Method: HEAD 2023-05-31T19:52:48.154 DEBUG:boto:Path: /pklxkd-7/ 2023-05-31T19:52:48.154 DEBUG:boto:Data: 2023-05-31T19:52:48.155 DEBUG:boto:Headers: {} 2023-05-31T19:52:48.155 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T19:52:48.155 DEBUG:boto:Port: 8000 2023-05-31T19:52:48.156 DEBUG:boto:Params: {} 2023-05-31T19:52:48.156 DEBUG:boto:Token: None 2023-05-31T19:52:48.157 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 19:52:48 GMT /pklxkd-7/ 2023-05-31T19:52:48.157 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:lg1RckkP8N68jVLSlmh/MmDojh0= 2023-05-31T19:52:48.158 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 19:52:48 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:lg1RckkP8N68jVLSlmh/MmDojh0=', 'Content-Length': '0'} 2023-05-31T19:52:48.160 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '2'), ('X-RGW-Bytes-Used', '12'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000045439e759fa37dc-006477a590-110e-a1'), ('Content-Length', '0'), ('Date', 'Wed, 31 May 2023 19:52:48 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T19:52:48.161 DEBUG:boto:path=/pklxkd-7/%3A 2023-05-31T19:52:48.161 DEBUG:boto:auth_path=/pklxkd-7/%3A 2023-05-31T19:52:48.161 DEBUG:boto:Method: PUT 2023-05-31T19:52:48.162 DEBUG:boto:Path: /pklxkd-7/%3A 2023-05-31T19:52:48.162 DEBUG:boto:Data: 2023-05-31T19:52:48.162 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2023-05-31T19:52:48.163 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T19:52:48.163 DEBUG:boto:Port: 8000 2023-05-31T19:52:48.163 DEBUG:boto:Params: {} 2023-05-31T19:52:48.164 DEBUG:boto:Token: None 2023-05-31T19:52:48.164 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Wed, 31 May 2023 19:52:48 GMT /pklxkd-7/%3A 2023-05-31T19:52:48.165 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:4UdPC+f5tGBb5OGLSgOKMgluCM8= 2023-05-31T19:52:48.165 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Wed, 31 May 2023 19:52:48 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:4UdPC+f5tGBb5OGLSgOKMgluCM8='} 2023-05-31T19:52:48.183 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000005028dd96953569d2-006477a590-110e-a1'), ('Date', 'Wed, 31 May 2023 19:52:48 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T19:52:48.183 DEBUG:boto:path=/pklxkd-7/ 2023-05-31T19:52:48.184 DEBUG:boto:auth_path=/pklxkd-7/ 2023-05-31T19:52:48.184 DEBUG:boto:Method: HEAD 2023-05-31T19:52:48.184 DEBUG:boto:Path: /pklxkd-7/ 2023-05-31T19:52:48.185 DEBUG:boto:Data: 2023-05-31T19:52:48.185 DEBUG:boto:Headers: {} 2023-05-31T19:52:48.186 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T19:52:48.186 DEBUG:boto:Port: 8000 2023-05-31T19:52:48.186 DEBUG:boto:Params: {} 2023-05-31T19:52:48.187 DEBUG:boto:Token: None 2023-05-31T19:52:48.187 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 19:52:48 GMT /pklxkd-7/ 2023-05-31T19:52:48.187 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:lg1RckkP8N68jVLSlmh/MmDojh0= 2023-05-31T19:52:48.188 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 19:52:48 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:lg1RckkP8N68jVLSlmh/MmDojh0=', 'Content-Length': '0'} 2023-05-31T19:52:48.190 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '3'), ('X-RGW-Bytes-Used', '18'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000b85d62f414a440d4-006477a590-110e-a1'), ('Content-Length', '0'), ('Date', 'Wed, 31 May 2023 19:52:48 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T19:52:48.190 DEBUG:boto:path=/pklxkd-7/%26 2023-05-31T19:52:48.191 DEBUG:boto:auth_path=/pklxkd-7/%26 2023-05-31T19:52:48.191 DEBUG:boto:Method: PUT 2023-05-31T19:52:48.192 DEBUG:boto:Path: /pklxkd-7/%26 2023-05-31T19:52:48.192 DEBUG:boto:Data: 2023-05-31T19:52:48.192 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2023-05-31T19:52:48.193 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T19:52:48.193 DEBUG:boto:Port: 8000 2023-05-31T19:52:48.194 DEBUG:boto:Params: {} 2023-05-31T19:52:48.194 DEBUG:boto:Token: None 2023-05-31T19:52:48.195 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Wed, 31 May 2023 19:52:48 GMT /pklxkd-7/%26 2023-05-31T19:52:48.195 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:hlVtzGFKeHn9urWQty+8OBvcHh4= 2023-05-31T19:52:48.196 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Wed, 31 May 2023 19:52:48 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:hlVtzGFKeHn9urWQty+8OBvcHh4='} 2023-05-31T19:52:48.201 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000a3b75032a109bb37-006477a590-110e-a1'), ('Date', 'Wed, 31 May 2023 19:52:48 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T19:52:48.201 DEBUG:boto:path=/pklxkd-8/ 2023-05-31T19:52:48.202 DEBUG:boto:auth_path=/pklxkd-8/ 2023-05-31T19:52:48.202 DEBUG:boto:Method: HEAD 2023-05-31T19:52:48.203 DEBUG:boto:Path: /pklxkd-8/ 2023-05-31T19:52:48.203 DEBUG:boto:Data: 2023-05-31T19:52:48.203 DEBUG:boto:Headers: {} 2023-05-31T19:52:48.204 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T19:52:48.204 DEBUG:boto:Port: 8001 2023-05-31T19:52:48.205 DEBUG:boto:Params: {} 2023-05-31T19:52:48.205 DEBUG:boto:Token: None 2023-05-31T19:52:48.205 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 19:52:48 GMT /pklxkd-8/ 2023-05-31T19:52:48.206 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:2i2w0mW9GhE2PFTdE0AxMJ5SuXI= 2023-05-31T19:52:48.206 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 19:52:48 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:2i2w0mW9GhE2PFTdE0AxMJ5SuXI=', 'Content-Length': '0'} 2023-05-31T19:52:48.208 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000319a4398472f49c2-006477a590-1112-a2'), ('Content-Length', '0'), ('Date', 'Wed, 31 May 2023 19:52:48 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T19:52:48.209 DEBUG:boto:path=/pklxkd-8/myobj 2023-05-31T19:52:48.209 DEBUG:boto:auth_path=/pklxkd-8/myobj 2023-05-31T19:52:48.210 DEBUG:boto:Method: PUT 2023-05-31T19:52:48.210 DEBUG:boto:Path: /pklxkd-8/myobj 2023-05-31T19:52:48.210 DEBUG:boto:Data: 2023-05-31T19:52:48.211 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2023-05-31T19:52:48.211 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T19:52:48.211 DEBUG:boto:Port: 8001 2023-05-31T19:52:48.212 DEBUG:boto:Params: {} 2023-05-31T19:52:48.212 DEBUG:boto:Token: None 2023-05-31T19:52:48.212 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Wed, 31 May 2023 19:52:48 GMT /pklxkd-8/myobj 2023-05-31T19:52:48.213 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:bUQvoohObmgjnfnNh88MGfsmFQs= 2023-05-31T19:52:48.213 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Wed, 31 May 2023 19:52:48 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:bUQvoohObmgjnfnNh88MGfsmFQs='} 2023-05-31T19:52:48.223 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000d6dfb63796ec7f6f-006477a590-1112-a2'), ('Date', 'Wed, 31 May 2023 19:52:48 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T19:52:48.224 DEBUG:boto:path=/pklxkd-8/ 2023-05-31T19:52:48.224 DEBUG:boto:auth_path=/pklxkd-8/ 2023-05-31T19:52:48.225 DEBUG:boto:Method: HEAD 2023-05-31T19:52:48.225 DEBUG:boto:Path: /pklxkd-8/ 2023-05-31T19:52:48.226 DEBUG:boto:Data: 2023-05-31T19:52:48.227 DEBUG:boto:Headers: {} 2023-05-31T19:52:48.227 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T19:52:48.228 DEBUG:boto:Port: 8001 2023-05-31T19:52:48.228 DEBUG:boto:Params: {} 2023-05-31T19:52:48.228 DEBUG:boto:Token: None 2023-05-31T19:52:48.229 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 19:52:48 GMT /pklxkd-8/ 2023-05-31T19:52:48.229 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:2i2w0mW9GhE2PFTdE0AxMJ5SuXI= 2023-05-31T19:52:48.230 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 19:52:48 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:2i2w0mW9GhE2PFTdE0AxMJ5SuXI=', 'Content-Length': '0'} 2023-05-31T19:52:48.232 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '6'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000e13da4dfa733c047-006477a590-1112-a2'), ('Content-Length', '0'), ('Date', 'Wed, 31 May 2023 19:52:48 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T19:52:48.233 DEBUG:boto:path=/pklxkd-8/_myobj 2023-05-31T19:52:48.233 DEBUG:boto:auth_path=/pklxkd-8/_myobj 2023-05-31T19:52:48.233 DEBUG:boto:Method: PUT 2023-05-31T19:52:48.234 DEBUG:boto:Path: /pklxkd-8/_myobj 2023-05-31T19:52:48.234 DEBUG:boto:Data: 2023-05-31T19:52:48.234 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2023-05-31T19:52:48.235 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T19:52:48.235 DEBUG:boto:Port: 8001 2023-05-31T19:52:48.236 DEBUG:boto:Params: {} 2023-05-31T19:52:48.236 DEBUG:boto:Token: None 2023-05-31T19:52:48.237 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Wed, 31 May 2023 19:52:48 GMT /pklxkd-8/_myobj 2023-05-31T19:52:48.238 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:f8Oauryi6niaW+ZHVNnIfetAPD8= 2023-05-31T19:52:48.238 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Wed, 31 May 2023 19:52:48 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:f8Oauryi6niaW+ZHVNnIfetAPD8='} 2023-05-31T19:52:48.248 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000776a6eb87b6e2d90-006477a590-1112-a2'), ('Date', 'Wed, 31 May 2023 19:52:48 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T19:52:48.249 DEBUG:boto:path=/pklxkd-8/ 2023-05-31T19:52:48.249 DEBUG:boto:auth_path=/pklxkd-8/ 2023-05-31T19:52:48.250 DEBUG:boto:Method: HEAD 2023-05-31T19:52:48.250 DEBUG:boto:Path: /pklxkd-8/ 2023-05-31T19:52:48.251 DEBUG:boto:Data: 2023-05-31T19:52:48.251 DEBUG:boto:Headers: {} 2023-05-31T19:52:48.251 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T19:52:48.252 DEBUG:boto:Port: 8001 2023-05-31T19:52:48.252 DEBUG:boto:Params: {} 2023-05-31T19:52:48.252 DEBUG:boto:Token: None 2023-05-31T19:52:48.253 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 19:52:48 GMT /pklxkd-8/ 2023-05-31T19:52:48.254 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:2i2w0mW9GhE2PFTdE0AxMJ5SuXI= 2023-05-31T19:52:48.254 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 19:52:48 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:2i2w0mW9GhE2PFTdE0AxMJ5SuXI=', 'Content-Length': '0'} 2023-05-31T19:52:48.256 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '2'), ('X-RGW-Bytes-Used', '12'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000910580aebc731fe2-006477a590-1112-a2'), ('Content-Length', '0'), ('Date', 'Wed, 31 May 2023 19:52:48 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T19:52:48.257 DEBUG:boto:path=/pklxkd-8/%3A 2023-05-31T19:52:48.259 DEBUG:boto:auth_path=/pklxkd-8/%3A 2023-05-31T19:52:48.260 DEBUG:boto:Method: PUT 2023-05-31T19:52:48.260 DEBUG:boto:Path: /pklxkd-8/%3A 2023-05-31T19:52:48.260 DEBUG:boto:Data: 2023-05-31T19:52:48.261 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2023-05-31T19:52:48.261 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T19:52:48.261 DEBUG:boto:Port: 8001 2023-05-31T19:52:48.262 DEBUG:boto:Params: {} 2023-05-31T19:52:48.262 DEBUG:boto:Token: None 2023-05-31T19:52:48.263 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Wed, 31 May 2023 19:52:48 GMT /pklxkd-8/%3A 2023-05-31T19:52:48.265 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:D11ibhSlkFo8DSI+NEgZ7bAf0PQ= 2023-05-31T19:52:48.265 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Wed, 31 May 2023 19:52:48 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:D11ibhSlkFo8DSI+NEgZ7bAf0PQ='} 2023-05-31T19:52:48.308 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000008491115a8dd2a72e-006477a590-1112-a2'), ('Date', 'Wed, 31 May 2023 19:52:48 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T19:52:48.309 DEBUG:boto:path=/pklxkd-8/ 2023-05-31T19:52:48.309 DEBUG:boto:auth_path=/pklxkd-8/ 2023-05-31T19:52:48.310 DEBUG:boto:Method: HEAD 2023-05-31T19:52:48.310 DEBUG:boto:Path: /pklxkd-8/ 2023-05-31T19:52:48.310 DEBUG:boto:Data: 2023-05-31T19:52:48.311 DEBUG:boto:Headers: {} 2023-05-31T19:52:48.311 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T19:52:48.312 DEBUG:boto:Port: 8001 2023-05-31T19:52:48.312 DEBUG:boto:Params: {} 2023-05-31T19:52:48.312 DEBUG:boto:Token: None 2023-05-31T19:52:48.313 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 19:52:48 GMT /pklxkd-8/ 2023-05-31T19:52:48.313 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:2i2w0mW9GhE2PFTdE0AxMJ5SuXI= 2023-05-31T19:52:48.314 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 19:52:48 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:2i2w0mW9GhE2PFTdE0AxMJ5SuXI=', 'Content-Length': '0'} 2023-05-31T19:52:48.315 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '3'), ('X-RGW-Bytes-Used', '18'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000141f0932b2ae1808-006477a590-1112-a2'), ('Content-Length', '0'), ('Date', 'Wed, 31 May 2023 19:52:48 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T19:52:48.316 DEBUG:boto:path=/pklxkd-8/%26 2023-05-31T19:52:48.317 DEBUG:boto:auth_path=/pklxkd-8/%26 2023-05-31T19:52:48.317 DEBUG:boto:Method: PUT 2023-05-31T19:52:48.317 DEBUG:boto:Path: /pklxkd-8/%26 2023-05-31T19:52:48.318 DEBUG:boto:Data: 2023-05-31T19:52:48.318 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2023-05-31T19:52:48.318 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T19:52:48.319 DEBUG:boto:Port: 8001 2023-05-31T19:52:48.319 DEBUG:boto:Params: {} 2023-05-31T19:52:48.319 DEBUG:boto:Token: None 2023-05-31T19:52:48.321 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Wed, 31 May 2023 19:52:48 GMT /pklxkd-8/%26 2023-05-31T19:52:48.321 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:JXyWw2JFQDF9YrclRwsfm/0TZ5c= 2023-05-31T19:52:48.322 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Wed, 31 May 2023 19:52:48 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:JXyWw2JFQDF9YrclRwsfm/0TZ5c='} 2023-05-31T19:52:48.326 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000d544942bbda8c3f7-006477a590-1112-a2'), ('Date', 'Wed, 31 May 2023 19:52:48 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T19:52:48.327 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-31T19:52:48.327 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-31T19:52:48.328 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-05-31T19:52:48.376 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2023-05-31T19:52:48.376 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2023-05-31T19:52:48.404 INFO:teuthology.orchestra.run.smithi099.stderr:No --period given, using current period=2e6123d2-d695-4350-9a01-61a455bc782b 2023-05-31T19:52:48.421 INFO:teuthology.orchestra.run.smithi099.stdout:[ 2023-05-31T19:52:48.422 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T19:52:48.422 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1685562631.516089_59.1", 2023-05-31T19:52:48.423 INFO:teuthology.orchestra.run.smithi099.stdout: "last_update": "2023-05-31T19:50:31.516089Z" 2023-05-31T19:52:48.423 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T19:52:48.423 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T19:52:48.423 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "", 2023-05-31T19:52:48.424 INFO:teuthology.orchestra.run.smithi099.stdout: "last_update": "0.000000" 2023-05-31T19:52:48.424 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T19:52:48.424 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T19:52:48.425 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1685562767.961795_79.1", 2023-05-31T19:52:48.425 INFO:teuthology.orchestra.run.smithi099.stdout: "last_update": "2023-05-31T19:52:47.961795Z" 2023-05-31T19:52:48.425 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T19:52:48.426 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T19:52:48.426 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "", 2023-05-31T19:52:48.426 INFO:teuthology.orchestra.run.smithi099.stdout: "last_update": "0.000000" 2023-05-31T19:52:48.427 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T19:52:48.427 INFO:teuthology.orchestra.run.smithi099.stdout:] 2023-05-31T19:52:48.428 DEBUG:rgw_multi.tests:master meta markers={0: '1_1685562631.516089_59.1', 1: '', 2: '1_1685562767.961795_79.1', 3: ''} 2023-05-31T19:52:48.429 INFO:rgw_multi.tests:starting meta checkpoint for zone=a2 2023-05-31T19:52:48.430 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-31T19:52:48.431 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-31T19:52:48.432 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-05-31T19:52:48.482 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2023-05-31T19:52:48.482 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2023-05-31T19:52:48.533 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T19:52:48.527+0000 7f1d13136780 20 RGW-SYNC:meta: read sync status 2023-05-31T19:52:48.533 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T19:52:48.528+0000 7f1d13136780 20 RGW-SYNC:meta: read sync status 2023-05-31T19:52:48.552 INFO:teuthology.orchestra.run.smithi099.stdout:{ 2023-05-31T19:52:48.552 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_status": { 2023-05-31T19:52:48.552 INFO:teuthology.orchestra.run.smithi099.stdout: "info": { 2023-05-31T19:52:48.552 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "sync", 2023-05-31T19:52:48.553 INFO:teuthology.orchestra.run.smithi099.stdout: "num_shards": 4, 2023-05-31T19:52:48.553 INFO:teuthology.orchestra.run.smithi099.stdout: "period": "2e6123d2-d695-4350-9a01-61a455bc782b", 2023-05-31T19:52:48.553 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2023-05-31T19:52:48.553 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T19:52:48.553 INFO:teuthology.orchestra.run.smithi099.stdout: "markers": [ 2023-05-31T19:52:48.553 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T19:52:48.554 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 0, 2023-05-31T19:52:48.554 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T19:52:48.554 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2023-05-31T19:52:48.555 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1685562631.516089_59.1", 2023-05-31T19:52:48.555 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T19:52:48.555 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 1, 2023-05-31T19:52:48.555 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T19:52:48.556 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2023-05-31T19:50:31.516089Z", 2023-05-31T19:52:48.556 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2023-05-31T19:52:48.556 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T19:52:48.557 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T19:52:48.557 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T19:52:48.557 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 1, 2023-05-31T19:52:48.558 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T19:52:48.558 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2023-05-31T19:52:48.558 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "", 2023-05-31T19:52:48.559 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T19:52:48.559 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2023-05-31T19:52:48.559 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T19:52:48.560 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "0.000000", 2023-05-31T19:52:48.560 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2023-05-31T19:52:48.560 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T19:52:48.561 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T19:52:48.561 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T19:52:48.561 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 2, 2023-05-31T19:52:48.561 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T19:52:48.562 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2023-05-31T19:52:48.562 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1685562767.961795_79.1", 2023-05-31T19:52:48.562 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T19:52:48.563 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2023-05-31T19:52:48.563 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T19:52:48.563 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2023-05-31T19:52:47.961795Z", 2023-05-31T19:52:48.563 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2023-05-31T19:52:48.564 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T19:52:48.564 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T19:52:48.564 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T19:52:48.565 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 3, 2023-05-31T19:52:48.565 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T19:52:48.565 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2023-05-31T19:52:48.566 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "", 2023-05-31T19:52:48.566 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T19:52:48.566 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2023-05-31T19:52:48.567 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T19:52:48.568 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "0.000000", 2023-05-31T19:52:48.568 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2023-05-31T19:52:48.569 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T19:52:48.569 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T19:52:48.569 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2023-05-31T19:52:48.569 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T19:52:48.569 INFO:teuthology.orchestra.run.smithi099.stdout: "full_sync": { 2023-05-31T19:52:48.569 INFO:teuthology.orchestra.run.smithi099.stdout: "total": 1, 2023-05-31T19:52:48.570 INFO:teuthology.orchestra.run.smithi099.stdout: "complete": 1 2023-05-31T19:52:48.570 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T19:52:48.570 INFO:teuthology.orchestra.run.smithi099.stdout: "current_time": "2023-05-31T19:52:48Z" 2023-05-31T19:52:48.570 INFO:teuthology.orchestra.run.smithi099.stdout:} 2023-05-31T19:52:48.571 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "2e6123d2-d695-4350-9a01-61a455bc782b", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1685562631.516089_59.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-05-31T19:50:31.516089Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1685562767.961795_79.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-05-31T19:52:47.961795Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-05-31T19:52:48Z" } 2023-05-31T19:52:48.572 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1685562631.516089_59.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-05-31T19:50:31.516089Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1685562767.961795_79.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-05-31T19:52:47.961795Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2023-05-31T19:52:48.573 DEBUG:rgw_multi.tests:log_status={0: '1_1685562631.516089_59.1', 1: '', 2: '1_1685562767.961795_79.1', 3: ''} 2023-05-31T19:52:48.573 DEBUG:rgw_multi.tests:sync_status={0: '1_1685562631.516089_59.1', 1: '', 2: '1_1685562767.961795_79.1', 3: ''} 2023-05-31T19:52:48.573 INFO:rgw_multi.tests:finish meta checkpoint for zone=a2 2023-05-31T19:52:48.574 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'pklxkd-7', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-05-31T19:52:48.574 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'pklxkd-7', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-05-31T19:52:48.575 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket pklxkd-7 --source-zone a1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2023-05-31T19:52:48.632 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2023-05-31T19:52:48.632 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2023-05-31T19:52:48.673 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T19:52:48.668+0000 7f0b4abf9780 1 waiting to reach incremental sync.. 2023-05-31T19:52:53.673 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T19:52:53.668+0000 7f0b4abf9780 1 waiting to reach incremental sync.. 2023-05-31T19:52:58.674 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T19:52:58.668+0000 7f0b4abf9780 1 waiting to reach incremental sync.. 2023-05-31T19:53:03.674 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T19:53:03.669+0000 7f0b4abf9780 1 waiting to reach incremental sync.. 2023-05-31T19:53:08.675 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T19:53:08.669+0000 7f0b4abf9780 1 waiting to reach incremental sync.. 2023-05-31T19:53:13.676 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T19:53:13.670+0000 7f0b4abf9780 1 waiting to reach incremental sync.. 2023-05-31T19:53:18.677 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T19:53:18.672+0000 7f0b4abf9780 1 waiting to reach incremental sync.. 2023-05-31T19:53:23.677 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T19:53:23.672+0000 7f0b4abf9780 1 waiting to reach incremental sync.. 2023-05-31T19:53:28.678 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T19:53:28.673+0000 7f0b4abf9780 1 waiting to reach incremental sync.. 2023-05-31T19:53:33.678 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T19:53:33.673+0000 7f0b4abf9780 1 waiting to reach incremental sync.. 2023-05-31T19:53:38.679 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T19:53:38.674+0000 7f0b4abf9780 1 waiting to reach incremental sync.. 2023-05-31T19:53:43.680 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T19:53:43.674+0000 7f0b4abf9780 1 waiting to reach incremental sync.. 2023-05-31T19:53:48.680 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T19:53:48.675+0000 7f0b4abf9780 1 waiting to reach incremental sync.. 2023-05-31T19:53:53.681 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T19:53:53.675+0000 7f0b4abf9780 1 waiting to reach incremental sync.. 2023-05-31T19:53:58.682 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T19:53:58.676+0000 7f0b4abf9780 1 waiting to reach incremental sync.. 2023-05-31T19:54:03.682 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T19:54:03.677+0000 7f0b4abf9780 1 waiting to reach incremental sync.. 2023-05-31T19:54:08.683 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T19:54:08.678+0000 7f0b4abf9780 1 waiting to reach incremental sync.. 2023-05-31T19:54:13.683 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T19:54:13.678+0000 7f0b4abf9780 1 waiting to reach incremental sync.. 2023-05-31T19:54:18.684 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T19:54:18.679+0000 7f0b4abf9780 1 waiting to reach incremental sync.. 2023-05-31T19:54:23.685 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T19:54:23.679+0000 7f0b4abf9780 1 waiting to reach incremental sync.. 2023-05-31T19:54:28.685 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T19:54:28.679+0000 7f0b4abf9780 1 waiting to reach incremental sync.. 2023-05-31T19:54:33.686 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T19:54:33.680+0000 7f0b4abf9780 1 waiting to reach incremental sync.. 2023-05-31T19:54:38.687 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T19:54:38.681+0000 7f0b4abf9780 1 waiting to reach incremental sync.. 2023-05-31T19:54:43.687 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T19:54:43.682+0000 7f0b4abf9780 1 waiting to reach incremental sync.. 2023-05-31T19:54:48.688 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T19:54:48.683+0000 7f0b4abf9780 1 waiting to reach incremental sync.. 2023-05-31T19:54:53.688 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T19:54:53.683+0000 7f0b4abf9780 1 waiting to reach incremental sync.. 2023-05-31T19:54:58.689 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T19:54:58.684+0000 7f0b4abf9780 1 waiting to reach incremental sync.. 2023-05-31T19:55:03.689 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T19:55:03.685+0000 7f0b4abf9780 1 waiting to reach incremental sync.. 2023-05-31T19:55:08.690 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T19:55:08.684+0000 7f0b4abf9780 1 waiting to reach incremental sync.. 2023-05-31T19:55:13.690 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T19:55:13.685+0000 7f0b4abf9780 1 waiting to reach incremental sync.. 2023-05-31T19:55:18.691 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T19:55:18.685+0000 7f0b4abf9780 1 waiting to reach incremental sync.. 2023-05-31T19:55:23.692 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T19:55:23.686+0000 7f0b4abf9780 1 waiting to reach incremental sync.. 2023-05-31T19:55:28.693 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T19:55:28.688+0000 7f0b4abf9780 1 waiting to reach incremental sync.. 2023-05-31T19:55:33.694 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T19:55:33.689+0000 7f0b4abf9780 1 waiting to reach incremental sync.. 2023-05-31T19:55:38.694 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T19:55:38.689+0000 7f0b4abf9780 1 waiting to reach incremental sync.. 2023-05-31T19:55:43.695 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T19:55:43.690+0000 7f0b4abf9780 1 waiting to reach incremental sync.. 2023-05-31T19:55:48.696 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T19:55:48.690+0000 7f0b4abf9780 1 waiting to reach incremental sync.. 2023-05-31T19:55:53.696 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T19:55:53.690+0000 7f0b4abf9780 1 waiting to reach incremental sync.. 2023-05-31T19:55:58.697 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T19:55:58.691+0000 7f0b4abf9780 1 waiting to reach incremental sync.. 2023-05-31T19:56:03.697 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T19:56:03.691+0000 7f0b4abf9780 1 waiting to reach incremental sync.. 2023-05-31T19:56:08.699 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T19:56:08.693+0000 7f0b4abf9780 1 bucket sync caught up with source: 2023-05-31T19:56:08.700 INFO:teuthology.orchestra.run.smithi099.stderr: local status: [, 00000000001.2.6, , 00000000001.5.6, , , , 00000000001.6.6, 00000000001.4.6, , ] 2023-05-31T19:56:08.700 INFO:teuthology.orchestra.run.smithi099.stderr: remote markers: [, 00000000001.2.6, , 00000000001.5.6, , , , 00000000001.6.6, 00000000001.4.6, , ] 2023-05-31T19:56:08.700 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T19:56:08.693+0000 7f0b4abf9780 0 bucket checkpoint complete 2023-05-31T19:56:08.722 INFO:tasks.rgw_multi.zone_rados:comparing bucket=pklxkd-7 zones={a2, a1} 2023-05-31T19:56:08.722 DEBUG:boto:path=/pklxkd-7/ 2023-05-31T19:56:08.723 DEBUG:boto:auth_path=/pklxkd-7/ 2023-05-31T19:56:08.723 DEBUG:boto:Method: HEAD 2023-05-31T19:56:08.724 DEBUG:boto:Path: /pklxkd-7/ 2023-05-31T19:56:08.724 DEBUG:boto:Data: 2023-05-31T19:56:08.724 DEBUG:boto:Headers: {} 2023-05-31T19:56:08.725 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T19:56:08.725 DEBUG:boto:Port: 8001 2023-05-31T19:56:08.725 DEBUG:boto:Params: {} 2023-05-31T19:56:08.726 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8001} 2023-05-31T19:56:08.726 DEBUG:boto:Token: None 2023-05-31T19:56:08.727 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 19:56:08 GMT /pklxkd-7/ 2023-05-31T19:56:08.727 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:Wj+zMv3xM7+eJFc1AzJgF9uJ1sI= 2023-05-31T19:56:08.728 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 19:56:08 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:Wj+zMv3xM7+eJFc1AzJgF9uJ1sI=', 'Content-Length': '0'} 2023-05-31T19:56:08.732 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '4'), ('X-RGW-Bytes-Used', '24'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000b6e1012e00a7aa12-006477a658-1112-a2'), ('Content-Length', '0'), ('Date', 'Wed, 31 May 2023 19:56:08 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T19:56:08.732 DEBUG:boto:path=/pklxkd-7/ 2023-05-31T19:56:08.733 DEBUG:boto:auth_path=/pklxkd-7/ 2023-05-31T19:56:08.733 DEBUG:boto:Method: HEAD 2023-05-31T19:56:08.733 DEBUG:boto:Path: /pklxkd-7/ 2023-05-31T19:56:08.734 DEBUG:boto:Data: 2023-05-31T19:56:08.734 DEBUG:boto:Headers: {} 2023-05-31T19:56:08.734 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T19:56:08.735 DEBUG:boto:Port: 8000 2023-05-31T19:56:08.735 DEBUG:boto:Params: {} 2023-05-31T19:56:08.735 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2023-05-31T19:56:08.736 DEBUG:boto:Token: None 2023-05-31T19:56:08.736 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 19:56:08 GMT /pklxkd-7/ 2023-05-31T19:56:08.737 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:Wj+zMv3xM7+eJFc1AzJgF9uJ1sI= 2023-05-31T19:56:08.737 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 19:56:08 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:Wj+zMv3xM7+eJFc1AzJgF9uJ1sI=', 'Content-Length': '0'} 2023-05-31T19:56:08.740 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '4'), ('X-RGW-Bytes-Used', '24'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000008d7820e55d914f7e-006477a658-110e-a1'), ('Content-Length', '0'), ('Date', 'Wed, 31 May 2023 19:56:08 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T19:56:08.741 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2023-05-31T19:56:08.742 DEBUG:boto:path=/pklxkd-7/ 2023-05-31T19:56:08.743 DEBUG:boto:auth_path=/pklxkd-7/ 2023-05-31T19:56:08.743 DEBUG:boto:path=/pklxkd-7/?versions&max-keys=999 2023-05-31T19:56:08.744 DEBUG:boto:auth_path=/pklxkd-7/?versions&max-keys=999 2023-05-31T19:56:08.744 DEBUG:boto:Method: GET 2023-05-31T19:56:08.744 DEBUG:boto:Path: /pklxkd-7/?versions&max-keys=999 2023-05-31T19:56:08.745 DEBUG:boto:Data: 2023-05-31T19:56:08.745 DEBUG:boto:Headers: {} 2023-05-31T19:56:08.745 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T19:56:08.746 DEBUG:boto:Port: 8001 2023-05-31T19:56:08.746 DEBUG:boto:Params: {} 2023-05-31T19:56:08.746 DEBUG:boto:Token: None 2023-05-31T19:56:08.747 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 19:56:08 GMT /pklxkd-7/?versions 2023-05-31T19:56:08.747 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:FqyCmiO2UTvVmvfneQhMrDgQomo= 2023-05-31T19:56:08.748 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 19:56:08 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:FqyCmiO2UTvVmvfneQhMrDgQomo=', 'Content-Length': '0'} 2023-05-31T19:56:08.750 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b06b142e4a4a6317-006477a658-1112-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 19:56:08 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T19:56:08.751 DEBUG:boto:b'pklxkd-7999false&nulltrue2023-05-31T19:52:48.193Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal:nulltrue2023-05-31T19:52:48.162Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal_myobjnulltrue2023-05-31T19:52:48.103Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalmyobjnulltrue2023-05-31T19:52:48.073Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T19:56:08.753 DEBUG:tasks.rgw_multi.zone_rados:o=& 2023-05-31T19:56:08.754 DEBUG:tasks.rgw_multi.zone_rados:o=: 2023-05-31T19:56:08.754 DEBUG:tasks.rgw_multi.zone_rados:o=_myobj 2023-05-31T19:56:08.755 DEBUG:tasks.rgw_multi.zone_rados:o=myobj 2023-05-31T19:56:08.755 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2023-05-31T19:56:08.755 DEBUG:boto:path=/pklxkd-7/ 2023-05-31T19:56:08.756 DEBUG:boto:auth_path=/pklxkd-7/ 2023-05-31T19:56:08.756 DEBUG:boto:path=/pklxkd-7/?versions&max-keys=999 2023-05-31T19:56:08.756 DEBUG:boto:auth_path=/pklxkd-7/?versions&max-keys=999 2023-05-31T19:56:08.757 DEBUG:boto:Method: GET 2023-05-31T19:56:08.757 DEBUG:boto:Path: /pklxkd-7/?versions&max-keys=999 2023-05-31T19:56:08.757 DEBUG:boto:Data: 2023-05-31T19:56:08.758 DEBUG:boto:Headers: {} 2023-05-31T19:56:08.758 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T19:56:08.758 DEBUG:boto:Port: 8000 2023-05-31T19:56:08.759 DEBUG:boto:Params: {} 2023-05-31T19:56:08.759 DEBUG:boto:Token: None 2023-05-31T19:56:08.759 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 19:56:08 GMT /pklxkd-7/?versions 2023-05-31T19:56:08.760 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:FqyCmiO2UTvVmvfneQhMrDgQomo= 2023-05-31T19:56:08.760 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 19:56:08 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:FqyCmiO2UTvVmvfneQhMrDgQomo=', 'Content-Length': '0'} 2023-05-31T19:56:08.763 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000002d88c3f025cd2560-006477a658-110e-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 19:56:08 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T19:56:08.763 DEBUG:boto:b'pklxkd-7999false&nulltrue2023-05-31T19:52:48.193Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal:nulltrue2023-05-31T19:52:48.162Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal_myobjnulltrue2023-05-31T19:52:48.103Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalmyobjnulltrue2023-05-31T19:52:48.073Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T19:56:08.764 DEBUG:tasks.rgw_multi.zone_rados:o=& 2023-05-31T19:56:08.765 DEBUG:tasks.rgw_multi.zone_rados:o=: 2023-05-31T19:56:08.765 DEBUG:tasks.rgw_multi.zone_rados:o=_myobj 2023-05-31T19:56:08.765 DEBUG:tasks.rgw_multi.zone_rados:o=myobj 2023-05-31T19:56:08.766 DEBUG:boto:path=/pklxkd-7/ 2023-05-31T19:56:08.766 DEBUG:boto:auth_path=/pklxkd-7/ 2023-05-31T19:56:08.767 DEBUG:boto:path=/pklxkd-7/?versions&max-keys=999 2023-05-31T19:56:08.767 DEBUG:boto:auth_path=/pklxkd-7/?versions&max-keys=999 2023-05-31T19:56:08.767 DEBUG:boto:Method: GET 2023-05-31T19:56:08.768 DEBUG:boto:Path: /pklxkd-7/?versions&max-keys=999 2023-05-31T19:56:08.768 DEBUG:boto:Data: 2023-05-31T19:56:08.768 DEBUG:boto:Headers: {} 2023-05-31T19:56:08.769 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T19:56:08.769 DEBUG:boto:Port: 8001 2023-05-31T19:56:08.769 DEBUG:boto:Params: {} 2023-05-31T19:56:08.770 DEBUG:boto:Token: None 2023-05-31T19:56:08.770 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 19:56:08 GMT /pklxkd-7/?versions 2023-05-31T19:56:08.771 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:FqyCmiO2UTvVmvfneQhMrDgQomo= 2023-05-31T19:56:08.771 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 19:56:08 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:FqyCmiO2UTvVmvfneQhMrDgQomo=', 'Content-Length': '0'} 2023-05-31T19:56:08.774 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000027593352850cb901-006477a658-1112-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 19:56:08 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T19:56:08.774 DEBUG:boto:b'pklxkd-7999false&nulltrue2023-05-31T19:52:48.193Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal:nulltrue2023-05-31T19:52:48.162Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal_myobjnulltrue2023-05-31T19:52:48.103Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalmyobjnulltrue2023-05-31T19:52:48.073Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T19:56:08.775 DEBUG:boto:path=/pklxkd-7/ 2023-05-31T19:56:08.775 DEBUG:boto:auth_path=/pklxkd-7/ 2023-05-31T19:56:08.776 DEBUG:boto:path=/pklxkd-7/?versions&max-keys=999 2023-05-31T19:56:08.776 DEBUG:boto:auth_path=/pklxkd-7/?versions&max-keys=999 2023-05-31T19:56:08.776 DEBUG:boto:Method: GET 2023-05-31T19:56:08.777 DEBUG:boto:Path: /pklxkd-7/?versions&max-keys=999 2023-05-31T19:56:08.777 DEBUG:boto:Data: 2023-05-31T19:56:08.777 DEBUG:boto:Headers: {} 2023-05-31T19:56:08.778 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T19:56:08.778 DEBUG:boto:Port: 8000 2023-05-31T19:56:08.778 DEBUG:boto:Params: {} 2023-05-31T19:56:08.779 DEBUG:boto:Token: None 2023-05-31T19:56:08.779 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 19:56:08 GMT /pklxkd-7/?versions 2023-05-31T19:56:08.780 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:FqyCmiO2UTvVmvfneQhMrDgQomo= 2023-05-31T19:56:08.780 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 19:56:08 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:FqyCmiO2UTvVmvfneQhMrDgQomo=', 'Content-Length': '0'} 2023-05-31T19:56:08.782 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000078703cd83f9623dd-006477a658-110e-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 19:56:08 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T19:56:08.783 DEBUG:boto:b'pklxkd-7999false&nulltrue2023-05-31T19:52:48.193Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal:nulltrue2023-05-31T19:52:48.162Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal_myobjnulltrue2023-05-31T19:52:48.103Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalmyobjnulltrue2023-05-31T19:52:48.073Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T19:56:08.783 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=& 2023-05-31T19:56:08.784 DEBUG:boto:path=/pklxkd-7/%26 2023-05-31T19:56:08.784 DEBUG:boto:auth_path=/pklxkd-7/%26 2023-05-31T19:56:08.785 DEBUG:boto:path=/pklxkd-7/%26?versionId=null 2023-05-31T19:56:08.785 DEBUG:boto:auth_path=/pklxkd-7/%26?versionId=null 2023-05-31T19:56:08.785 DEBUG:boto:Method: GET 2023-05-31T19:56:08.786 DEBUG:boto:Path: /pklxkd-7/%26?versionId=null 2023-05-31T19:56:08.786 DEBUG:boto:Data: 2023-05-31T19:56:08.787 DEBUG:boto:Headers: {} 2023-05-31T19:56:08.787 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T19:56:08.787 DEBUG:boto:Port: 8001 2023-05-31T19:56:08.788 DEBUG:boto:Params: {} 2023-05-31T19:56:08.788 DEBUG:boto:Token: None 2023-05-31T19:56:08.788 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 19:56:08 GMT /pklxkd-7/%26?versionId=null 2023-05-31T19:56:08.789 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:gsROAsJSd89/3CrF0IBzBcsaw28= 2023-05-31T19:56:08.789 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 19:56:08 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:gsROAsJSd89/3CrF0IBzBcsaw28=', 'Content-Length': '0'} 2023-05-31T19:56:08.791 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 19:52:48 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000fba82cc04d87456a-006477a658-1112-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 19:56:08 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T19:56:08.792 DEBUG:boto:path=/pklxkd-7/%26 2023-05-31T19:56:08.792 DEBUG:boto:auth_path=/pklxkd-7/%26 2023-05-31T19:56:08.793 DEBUG:boto:path=/pklxkd-7/%26?versionId=null 2023-05-31T19:56:08.793 DEBUG:boto:auth_path=/pklxkd-7/%26?versionId=null 2023-05-31T19:56:08.793 DEBUG:boto:Method: GET 2023-05-31T19:56:08.794 DEBUG:boto:Path: /pklxkd-7/%26?versionId=null 2023-05-31T19:56:08.794 DEBUG:boto:Data: 2023-05-31T19:56:08.794 DEBUG:boto:Headers: {} 2023-05-31T19:56:08.795 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T19:56:08.795 DEBUG:boto:Port: 8000 2023-05-31T19:56:08.796 DEBUG:boto:Params: {} 2023-05-31T19:56:08.796 DEBUG:boto:Token: None 2023-05-31T19:56:08.796 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 19:56:08 GMT /pklxkd-7/%26?versionId=null 2023-05-31T19:56:08.797 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:gsROAsJSd89/3CrF0IBzBcsaw28= 2023-05-31T19:56:08.797 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 19:56:08 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:gsROAsJSd89/3CrF0IBzBcsaw28=', 'Content-Length': '0'} 2023-05-31T19:56:08.799 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 19:52:48 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000d0fc0db563e33dd1-006477a658-110e-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 19:56:08 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T19:56:08.800 DEBUG:boto:path=/pklxkd-7/%26 2023-05-31T19:56:08.800 DEBUG:boto:auth_path=/pklxkd-7/%26 2023-05-31T19:56:08.800 DEBUG:boto:path=/pklxkd-7/%26?versionId=null 2023-05-31T19:56:08.801 DEBUG:boto:auth_path=/pklxkd-7/%26?versionId=null 2023-05-31T19:56:08.801 DEBUG:boto:Method: HEAD 2023-05-31T19:56:08.801 DEBUG:boto:Path: /pklxkd-7/%26?versionId=null 2023-05-31T19:56:08.802 DEBUG:boto:Data: 2023-05-31T19:56:08.802 DEBUG:boto:Headers: {} 2023-05-31T19:56:08.802 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T19:56:08.803 DEBUG:boto:Port: 8001 2023-05-31T19:56:08.803 DEBUG:boto:Params: {} 2023-05-31T19:56:08.803 DEBUG:boto:Token: None 2023-05-31T19:56:08.804 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 19:56:08 GMT /pklxkd-7/%26?versionId=null 2023-05-31T19:56:08.804 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:AWHIJqxyTVmgG0X5FF1UXnaNzKI= 2023-05-31T19:56:08.805 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 19:56:08 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:AWHIJqxyTVmgG0X5FF1UXnaNzKI=', 'Content-Length': '0'} 2023-05-31T19:56:08.806 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 19:52:48 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000f0d5f28d98fa4e92-006477a658-1112-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 19:56:08 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T19:56:08.807 DEBUG:boto:path=/pklxkd-7/%26 2023-05-31T19:56:08.807 DEBUG:boto:auth_path=/pklxkd-7/%26 2023-05-31T19:56:08.808 DEBUG:boto:path=/pklxkd-7/%26?versionId=null 2023-05-31T19:56:08.808 DEBUG:boto:auth_path=/pklxkd-7/%26?versionId=null 2023-05-31T19:56:08.808 DEBUG:boto:Method: HEAD 2023-05-31T19:56:08.809 DEBUG:boto:Path: /pklxkd-7/%26?versionId=null 2023-05-31T19:56:08.809 DEBUG:boto:Data: 2023-05-31T19:56:08.809 DEBUG:boto:Headers: {} 2023-05-31T19:56:08.810 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T19:56:08.810 DEBUG:boto:Port: 8000 2023-05-31T19:56:08.810 DEBUG:boto:Params: {} 2023-05-31T19:56:08.811 DEBUG:boto:Token: None 2023-05-31T19:56:08.811 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 19:56:08 GMT /pklxkd-7/%26?versionId=null 2023-05-31T19:56:08.811 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:AWHIJqxyTVmgG0X5FF1UXnaNzKI= 2023-05-31T19:56:08.812 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 19:56:08 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:AWHIJqxyTVmgG0X5FF1UXnaNzKI=', 'Content-Length': '0'} 2023-05-31T19:56:08.814 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 19:52:48 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000cc70d6eb569c04c4-006477a658-110e-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 19:56:08 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T19:56:08.814 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=& 2023-05-31T19:56:08.815 DEBUG:boto:path=/pklxkd-7/%26 2023-05-31T19:56:08.815 DEBUG:boto:auth_path=/pklxkd-7/%26 2023-05-31T19:56:08.815 DEBUG:boto:path=/pklxkd-7/%26?versionId=null 2023-05-31T19:56:08.816 DEBUG:boto:auth_path=/pklxkd-7/%26?versionId=null 2023-05-31T19:56:08.816 DEBUG:boto:Method: GET 2023-05-31T19:56:08.816 DEBUG:boto:Path: /pklxkd-7/%26?versionId=null 2023-05-31T19:56:08.817 DEBUG:boto:Data: 2023-05-31T19:56:08.817 DEBUG:boto:Headers: {} 2023-05-31T19:56:08.817 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T19:56:08.818 DEBUG:boto:Port: 8001 2023-05-31T19:56:08.818 DEBUG:boto:Params: {} 2023-05-31T19:56:08.819 DEBUG:boto:Token: None 2023-05-31T19:56:08.819 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 19:56:08 GMT /pklxkd-7/%26?versionId=null 2023-05-31T19:56:08.819 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:gsROAsJSd89/3CrF0IBzBcsaw28= 2023-05-31T19:56:08.820 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 19:56:08 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:gsROAsJSd89/3CrF0IBzBcsaw28=', 'Content-Length': '0'} 2023-05-31T19:56:08.821 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 19:52:48 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000f3cf0bf4ecfba00a-006477a658-1112-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 19:56:08 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T19:56:08.822 DEBUG:boto:path=/pklxkd-7/%26 2023-05-31T19:56:08.822 DEBUG:boto:auth_path=/pklxkd-7/%26 2023-05-31T19:56:08.823 DEBUG:boto:path=/pklxkd-7/%26?versionId=null 2023-05-31T19:56:08.823 DEBUG:boto:auth_path=/pklxkd-7/%26?versionId=null 2023-05-31T19:56:08.823 DEBUG:boto:Method: GET 2023-05-31T19:56:08.824 DEBUG:boto:Path: /pklxkd-7/%26?versionId=null 2023-05-31T19:56:08.824 DEBUG:boto:Data: 2023-05-31T19:56:08.825 DEBUG:boto:Headers: {} 2023-05-31T19:56:08.825 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T19:56:08.825 DEBUG:boto:Port: 8000 2023-05-31T19:56:08.826 DEBUG:boto:Params: {} 2023-05-31T19:56:08.826 DEBUG:boto:Token: None 2023-05-31T19:56:08.826 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 19:56:08 GMT /pklxkd-7/%26?versionId=null 2023-05-31T19:56:08.827 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:gsROAsJSd89/3CrF0IBzBcsaw28= 2023-05-31T19:56:08.827 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 19:56:08 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:gsROAsJSd89/3CrF0IBzBcsaw28=', 'Content-Length': '0'} 2023-05-31T19:56:08.829 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 19:52:48 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000003ce97d50169f2c8c-006477a658-110e-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 19:56:08 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T19:56:08.830 DEBUG:boto:path=/pklxkd-7/ 2023-05-31T19:56:08.830 DEBUG:boto:auth_path=/pklxkd-7/ 2023-05-31T19:56:08.830 DEBUG:boto:path=/pklxkd-7/?prefix=%26 2023-05-31T19:56:08.831 DEBUG:boto:auth_path=/pklxkd-7/?prefix=%26 2023-05-31T19:56:08.831 DEBUG:boto:Method: GET 2023-05-31T19:56:08.831 DEBUG:boto:Path: /pklxkd-7/?prefix=%26 2023-05-31T19:56:08.832 DEBUG:boto:Data: 2023-05-31T19:56:08.832 DEBUG:boto:Headers: {} 2023-05-31T19:56:08.833 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T19:56:08.833 DEBUG:boto:Port: 8001 2023-05-31T19:56:08.833 DEBUG:boto:Params: {} 2023-05-31T19:56:08.834 DEBUG:boto:Token: None 2023-05-31T19:56:08.834 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 19:56:08 GMT /pklxkd-7/ 2023-05-31T19:56:08.834 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:DSBtXYx0ubm9AqsV8wT32ONzsVQ= 2023-05-31T19:56:08.835 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 19:56:08 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:DSBtXYx0ubm9AqsV8wT32ONzsVQ=', 'Content-Length': '0'} 2023-05-31T19:56:08.837 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e9f7c00668e49781-006477a658-1112-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 19:56:08 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T19:56:08.838 DEBUG:boto:b'pklxkd-7&1000false&2023-05-31T19:52:48.193Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T19:56:08.838 DEBUG:boto:path=/pklxkd-7/ 2023-05-31T19:56:08.839 DEBUG:boto:auth_path=/pklxkd-7/ 2023-05-31T19:56:08.839 DEBUG:boto:path=/pklxkd-7/?prefix=%26 2023-05-31T19:56:08.839 DEBUG:boto:auth_path=/pklxkd-7/?prefix=%26 2023-05-31T19:56:08.840 DEBUG:boto:Method: GET 2023-05-31T19:56:08.840 DEBUG:boto:Path: /pklxkd-7/?prefix=%26 2023-05-31T19:56:08.840 DEBUG:boto:Data: 2023-05-31T19:56:08.841 DEBUG:boto:Headers: {} 2023-05-31T19:56:08.841 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T19:56:08.841 DEBUG:boto:Port: 8000 2023-05-31T19:56:08.842 DEBUG:boto:Params: {} 2023-05-31T19:56:08.842 DEBUG:boto:Token: None 2023-05-31T19:56:08.843 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 19:56:08 GMT /pklxkd-7/ 2023-05-31T19:56:08.843 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:DSBtXYx0ubm9AqsV8wT32ONzsVQ= 2023-05-31T19:56:08.843 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 19:56:08 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:DSBtXYx0ubm9AqsV8wT32ONzsVQ=', 'Content-Length': '0'} 2023-05-31T19:56:08.846 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000005872b99946249431-006477a658-110e-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 19:56:08 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T19:56:08.846 DEBUG:boto:b'pklxkd-7&1000false&2023-05-31T19:52:48.193Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T19:56:08.847 DEBUG:boto:path=/pklxkd-7/%26 2023-05-31T19:56:08.847 DEBUG:boto:auth_path=/pklxkd-7/%26 2023-05-31T19:56:08.847 DEBUG:boto:Method: HEAD 2023-05-31T19:56:08.848 DEBUG:boto:Path: /pklxkd-7/%26 2023-05-31T19:56:08.848 DEBUG:boto:Data: 2023-05-31T19:56:08.848 DEBUG:boto:Headers: {} 2023-05-31T19:56:08.849 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T19:56:08.849 DEBUG:boto:Port: 8001 2023-05-31T19:56:08.849 DEBUG:boto:Params: {} 2023-05-31T19:56:08.850 DEBUG:boto:Token: None 2023-05-31T19:56:08.850 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 19:56:08 GMT /pklxkd-7/%26 2023-05-31T19:56:08.851 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:ozm4Wkl5nLEl5G/l5Z1KwD2+2BE= 2023-05-31T19:56:08.851 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 19:56:08 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:ozm4Wkl5nLEl5G/l5Z1KwD2+2BE=', 'Content-Length': '0'} 2023-05-31T19:56:08.853 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 19:52:48 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000005ff0ae9864ebea96-006477a658-1112-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 19:56:08 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T19:56:08.853 DEBUG:boto:path=/pklxkd-7/%26 2023-05-31T19:56:08.854 DEBUG:boto:auth_path=/pklxkd-7/%26 2023-05-31T19:56:08.854 DEBUG:boto:Method: HEAD 2023-05-31T19:56:08.855 DEBUG:boto:Path: /pklxkd-7/%26 2023-05-31T19:56:08.855 DEBUG:boto:Data: 2023-05-31T19:56:08.855 DEBUG:boto:Headers: {} 2023-05-31T19:56:08.855 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T19:56:08.856 DEBUG:boto:Port: 8000 2023-05-31T19:56:08.856 DEBUG:boto:Params: {} 2023-05-31T19:56:08.857 DEBUG:boto:Token: None 2023-05-31T19:56:08.857 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 19:56:08 GMT /pklxkd-7/%26 2023-05-31T19:56:08.857 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:ozm4Wkl5nLEl5G/l5Z1KwD2+2BE= 2023-05-31T19:56:08.858 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 19:56:08 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:ozm4Wkl5nLEl5G/l5Z1KwD2+2BE=', 'Content-Length': '0'} 2023-05-31T19:56:08.859 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 19:52:48 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000d654cbaad8beccc4-006477a658-110e-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 19:56:08 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T19:56:08.860 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=& 2023-05-31T19:56:08.860 DEBUG:boto:path=/pklxkd-7/%26 2023-05-31T19:56:08.861 DEBUG:boto:auth_path=/pklxkd-7/%26 2023-05-31T19:56:08.861 DEBUG:boto:Method: GET 2023-05-31T19:56:08.861 DEBUG:boto:Path: /pklxkd-7/%26 2023-05-31T19:56:08.862 DEBUG:boto:Data: 2023-05-31T19:56:08.862 DEBUG:boto:Headers: {} 2023-05-31T19:56:08.862 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T19:56:08.863 DEBUG:boto:Port: 8001 2023-05-31T19:56:08.863 DEBUG:boto:Params: {} 2023-05-31T19:56:08.863 DEBUG:boto:Token: None 2023-05-31T19:56:08.864 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 19:56:08 GMT /pklxkd-7/%26 2023-05-31T19:56:08.864 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:gKXEjRSarHIHr3F+ilu4FNMSaRY= 2023-05-31T19:56:08.865 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 19:56:08 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:gKXEjRSarHIHr3F+ilu4FNMSaRY=', 'Content-Length': '0'} 2023-05-31T19:56:08.866 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 19:52:48 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000267c961c60d462dd-006477a658-1112-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 19:56:08 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T19:56:08.908 DEBUG:boto:path=/pklxkd-7/%26 2023-05-31T19:56:08.909 DEBUG:boto:auth_path=/pklxkd-7/%26 2023-05-31T19:56:08.909 DEBUG:boto:Method: GET 2023-05-31T19:56:08.909 DEBUG:boto:Path: /pklxkd-7/%26 2023-05-31T19:56:08.910 DEBUG:boto:Data: 2023-05-31T19:56:08.910 DEBUG:boto:Headers: {} 2023-05-31T19:56:08.910 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T19:56:08.911 DEBUG:boto:Port: 8000 2023-05-31T19:56:08.911 DEBUG:boto:Params: {} 2023-05-31T19:56:08.911 DEBUG:boto:Token: None 2023-05-31T19:56:08.912 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 19:56:08 GMT /pklxkd-7/%26 2023-05-31T19:56:08.912 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:gKXEjRSarHIHr3F+ilu4FNMSaRY= 2023-05-31T19:56:08.913 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 19:56:08 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:gKXEjRSarHIHr3F+ilu4FNMSaRY=', 'Content-Length': '0'} 2023-05-31T19:56:08.914 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 19:52:48 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000e0f6c8c1d014c552-006477a658-110e-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 19:56:08 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T19:56:08.915 DEBUG:boto:path=/pklxkd-7/ 2023-05-31T19:56:08.915 DEBUG:boto:auth_path=/pklxkd-7/ 2023-05-31T19:56:08.916 DEBUG:boto:path=/pklxkd-7/?prefix=%26 2023-05-31T19:56:08.916 DEBUG:boto:auth_path=/pklxkd-7/?prefix=%26 2023-05-31T19:56:08.916 DEBUG:boto:Method: GET 2023-05-31T19:56:08.917 DEBUG:boto:Path: /pklxkd-7/?prefix=%26 2023-05-31T19:56:08.917 DEBUG:boto:Data: 2023-05-31T19:56:08.918 DEBUG:boto:Headers: {} 2023-05-31T19:56:08.918 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T19:56:08.918 DEBUG:boto:Port: 8001 2023-05-31T19:56:08.919 DEBUG:boto:Params: {} 2023-05-31T19:56:08.919 DEBUG:boto:Token: None 2023-05-31T19:56:08.919 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 19:56:08 GMT /pklxkd-7/ 2023-05-31T19:56:08.920 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:DSBtXYx0ubm9AqsV8wT32ONzsVQ= 2023-05-31T19:56:08.920 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 19:56:08 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:DSBtXYx0ubm9AqsV8wT32ONzsVQ=', 'Content-Length': '0'} 2023-05-31T19:56:08.922 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f125fbd9a4dee2dc-006477a658-1112-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 19:56:08 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T19:56:08.923 DEBUG:boto:b'pklxkd-7&1000false&2023-05-31T19:52:48.193Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T19:56:08.923 DEBUG:boto:path=/pklxkd-7/ 2023-05-31T19:56:08.924 DEBUG:boto:auth_path=/pklxkd-7/ 2023-05-31T19:56:08.924 DEBUG:boto:path=/pklxkd-7/?prefix=%26 2023-05-31T19:56:08.925 DEBUG:boto:auth_path=/pklxkd-7/?prefix=%26 2023-05-31T19:56:08.925 DEBUG:boto:Method: GET 2023-05-31T19:56:08.925 DEBUG:boto:Path: /pklxkd-7/?prefix=%26 2023-05-31T19:56:08.926 DEBUG:boto:Data: 2023-05-31T19:56:08.926 DEBUG:boto:Headers: {} 2023-05-31T19:56:08.926 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T19:56:08.927 DEBUG:boto:Port: 8000 2023-05-31T19:56:08.927 DEBUG:boto:Params: {} 2023-05-31T19:56:08.927 DEBUG:boto:Token: None 2023-05-31T19:56:08.928 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 19:56:08 GMT /pklxkd-7/ 2023-05-31T19:56:08.928 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:DSBtXYx0ubm9AqsV8wT32ONzsVQ= 2023-05-31T19:56:08.929 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 19:56:08 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:DSBtXYx0ubm9AqsV8wT32ONzsVQ=', 'Content-Length': '0'} 2023-05-31T19:56:08.931 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e7b7bead81b3a678-006477a658-110e-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 19:56:08 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T19:56:08.931 DEBUG:boto:b'pklxkd-7&1000false&2023-05-31T19:52:48.193Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T19:56:08.932 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=: 2023-05-31T19:56:08.932 DEBUG:boto:path=/pklxkd-7/%3A 2023-05-31T19:56:08.933 DEBUG:boto:auth_path=/pklxkd-7/%3A 2023-05-31T19:56:08.933 DEBUG:boto:path=/pklxkd-7/%3A?versionId=null 2023-05-31T19:56:08.933 DEBUG:boto:auth_path=/pklxkd-7/%3A?versionId=null 2023-05-31T19:56:08.934 DEBUG:boto:Method: GET 2023-05-31T19:56:08.934 DEBUG:boto:Path: /pklxkd-7/%3A?versionId=null 2023-05-31T19:56:08.935 DEBUG:boto:Data: 2023-05-31T19:56:08.935 DEBUG:boto:Headers: {} 2023-05-31T19:56:08.935 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T19:56:08.936 DEBUG:boto:Port: 8001 2023-05-31T19:56:08.936 DEBUG:boto:Params: {} 2023-05-31T19:56:08.936 DEBUG:boto:Token: None 2023-05-31T19:56:08.937 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 19:56:08 GMT /pklxkd-7/%3A?versionId=null 2023-05-31T19:56:08.937 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:VtO8WxtLjZJa+OFXpO2xVUb+veY= 2023-05-31T19:56:08.938 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 19:56:08 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:VtO8WxtLjZJa+OFXpO2xVUb+veY=', 'Content-Length': '0'} 2023-05-31T19:56:08.939 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 19:52:48 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000011c1320efe767932-006477a658-1112-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 19:56:08 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T19:56:08.940 DEBUG:boto:path=/pklxkd-7/%3A 2023-05-31T19:56:08.940 DEBUG:boto:auth_path=/pklxkd-7/%3A 2023-05-31T19:56:08.941 DEBUG:boto:path=/pklxkd-7/%3A?versionId=null 2023-05-31T19:56:08.941 DEBUG:boto:auth_path=/pklxkd-7/%3A?versionId=null 2023-05-31T19:56:08.941 DEBUG:boto:Method: GET 2023-05-31T19:56:08.942 DEBUG:boto:Path: /pklxkd-7/%3A?versionId=null 2023-05-31T19:56:08.942 DEBUG:boto:Data: 2023-05-31T19:56:08.942 DEBUG:boto:Headers: {} 2023-05-31T19:56:08.943 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T19:56:08.943 DEBUG:boto:Port: 8000 2023-05-31T19:56:08.944 DEBUG:boto:Params: {} 2023-05-31T19:56:08.944 DEBUG:boto:Token: None 2023-05-31T19:56:08.944 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 19:56:08 GMT /pklxkd-7/%3A?versionId=null 2023-05-31T19:56:08.945 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:VtO8WxtLjZJa+OFXpO2xVUb+veY= 2023-05-31T19:56:08.945 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 19:56:08 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:VtO8WxtLjZJa+OFXpO2xVUb+veY=', 'Content-Length': '0'} 2023-05-31T19:56:08.947 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 19:52:48 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000fc110fa817f594e6-006477a658-110e-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 19:56:08 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T19:56:08.948 DEBUG:boto:path=/pklxkd-7/%3A 2023-05-31T19:56:08.948 DEBUG:boto:auth_path=/pklxkd-7/%3A 2023-05-31T19:56:08.948 DEBUG:boto:path=/pklxkd-7/%3A?versionId=null 2023-05-31T19:56:08.949 DEBUG:boto:auth_path=/pklxkd-7/%3A?versionId=null 2023-05-31T19:56:08.949 DEBUG:boto:Method: HEAD 2023-05-31T19:56:08.949 DEBUG:boto:Path: /pklxkd-7/%3A?versionId=null 2023-05-31T19:56:08.950 DEBUG:boto:Data: 2023-05-31T19:56:08.950 DEBUG:boto:Headers: {} 2023-05-31T19:56:08.950 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T19:56:08.951 DEBUG:boto:Port: 8001 2023-05-31T19:56:08.951 DEBUG:boto:Params: {} 2023-05-31T19:56:08.951 DEBUG:boto:Token: None 2023-05-31T19:56:08.952 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 19:56:08 GMT /pklxkd-7/%3A?versionId=null 2023-05-31T19:56:08.952 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:mp8DryaE36l84Q4o+ttqj/PmCtU= 2023-05-31T19:56:08.953 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 19:56:08 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:mp8DryaE36l84Q4o+ttqj/PmCtU=', 'Content-Length': '0'} 2023-05-31T19:56:08.954 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 19:52:48 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000003aeaa93f9c716c3b-006477a658-1112-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 19:56:08 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T19:56:08.955 DEBUG:boto:path=/pklxkd-7/%3A 2023-05-31T19:56:08.955 DEBUG:boto:auth_path=/pklxkd-7/%3A 2023-05-31T19:56:08.955 DEBUG:boto:path=/pklxkd-7/%3A?versionId=null 2023-05-31T19:56:08.956 DEBUG:boto:auth_path=/pklxkd-7/%3A?versionId=null 2023-05-31T19:56:08.956 DEBUG:boto:Method: HEAD 2023-05-31T19:56:08.956 DEBUG:boto:Path: /pklxkd-7/%3A?versionId=null 2023-05-31T19:56:08.957 DEBUG:boto:Data: 2023-05-31T19:56:08.957 DEBUG:boto:Headers: {} 2023-05-31T19:56:08.957 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T19:56:08.958 DEBUG:boto:Port: 8000 2023-05-31T19:56:08.958 DEBUG:boto:Params: {} 2023-05-31T19:56:08.958 DEBUG:boto:Token: None 2023-05-31T19:56:08.959 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 19:56:08 GMT /pklxkd-7/%3A?versionId=null 2023-05-31T19:56:08.959 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:mp8DryaE36l84Q4o+ttqj/PmCtU= 2023-05-31T19:56:08.959 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 19:56:08 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:mp8DryaE36l84Q4o+ttqj/PmCtU=', 'Content-Length': '0'} 2023-05-31T19:56:08.961 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 19:52:48 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000f2c349791b0ff31f-006477a658-110e-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 19:56:08 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T19:56:08.962 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=: 2023-05-31T19:56:08.962 DEBUG:boto:path=/pklxkd-7/%3A 2023-05-31T19:56:08.962 DEBUG:boto:auth_path=/pklxkd-7/%3A 2023-05-31T19:56:08.963 DEBUG:boto:path=/pklxkd-7/%3A?versionId=null 2023-05-31T19:56:08.963 DEBUG:boto:auth_path=/pklxkd-7/%3A?versionId=null 2023-05-31T19:56:08.963 DEBUG:boto:Method: GET 2023-05-31T19:56:08.964 DEBUG:boto:Path: /pklxkd-7/%3A?versionId=null 2023-05-31T19:56:08.964 DEBUG:boto:Data: 2023-05-31T19:56:08.964 DEBUG:boto:Headers: {} 2023-05-31T19:56:08.965 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T19:56:08.965 DEBUG:boto:Port: 8001 2023-05-31T19:56:08.966 DEBUG:boto:Params: {} 2023-05-31T19:56:08.966 DEBUG:boto:Token: None 2023-05-31T19:56:08.966 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 19:56:08 GMT /pklxkd-7/%3A?versionId=null 2023-05-31T19:56:08.967 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:VtO8WxtLjZJa+OFXpO2xVUb+veY= 2023-05-31T19:56:08.967 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 19:56:08 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:VtO8WxtLjZJa+OFXpO2xVUb+veY=', 'Content-Length': '0'} 2023-05-31T19:56:08.969 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 19:52:48 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000a887e3a7c0c93ef9-006477a658-1112-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 19:56:08 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T19:56:08.970 DEBUG:boto:path=/pklxkd-7/%3A 2023-05-31T19:56:08.970 DEBUG:boto:auth_path=/pklxkd-7/%3A 2023-05-31T19:56:08.970 DEBUG:boto:path=/pklxkd-7/%3A?versionId=null 2023-05-31T19:56:08.971 DEBUG:boto:auth_path=/pklxkd-7/%3A?versionId=null 2023-05-31T19:56:08.971 DEBUG:boto:Method: GET 2023-05-31T19:56:08.971 DEBUG:boto:Path: /pklxkd-7/%3A?versionId=null 2023-05-31T19:56:08.972 DEBUG:boto:Data: 2023-05-31T19:56:08.972 DEBUG:boto:Headers: {} 2023-05-31T19:56:08.972 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T19:56:08.973 DEBUG:boto:Port: 8000 2023-05-31T19:56:08.973 DEBUG:boto:Params: {} 2023-05-31T19:56:08.974 DEBUG:boto:Token: None 2023-05-31T19:56:08.974 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 19:56:08 GMT /pklxkd-7/%3A?versionId=null 2023-05-31T19:56:08.974 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:VtO8WxtLjZJa+OFXpO2xVUb+veY= 2023-05-31T19:56:08.975 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 19:56:08 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:VtO8WxtLjZJa+OFXpO2xVUb+veY=', 'Content-Length': '0'} 2023-05-31T19:56:08.976 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 19:52:48 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000004a4d39ef7ef9a08a-006477a658-110e-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 19:56:08 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T19:56:08.977 DEBUG:boto:path=/pklxkd-7/ 2023-05-31T19:56:08.977 DEBUG:boto:auth_path=/pklxkd-7/ 2023-05-31T19:56:08.978 DEBUG:boto:path=/pklxkd-7/?prefix=%3A 2023-05-31T19:56:08.978 DEBUG:boto:auth_path=/pklxkd-7/?prefix=%3A 2023-05-31T19:56:08.978 DEBUG:boto:Method: GET 2023-05-31T19:56:08.979 DEBUG:boto:Path: /pklxkd-7/?prefix=%3A 2023-05-31T19:56:08.979 DEBUG:boto:Data: 2023-05-31T19:56:08.979 DEBUG:boto:Headers: {} 2023-05-31T19:56:08.980 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T19:56:08.980 DEBUG:boto:Port: 8001 2023-05-31T19:56:08.981 DEBUG:boto:Params: {} 2023-05-31T19:56:08.981 DEBUG:boto:Token: None 2023-05-31T19:56:08.981 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 19:56:08 GMT /pklxkd-7/ 2023-05-31T19:56:08.982 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:DSBtXYx0ubm9AqsV8wT32ONzsVQ= 2023-05-31T19:56:08.982 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 19:56:08 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:DSBtXYx0ubm9AqsV8wT32ONzsVQ=', 'Content-Length': '0'} 2023-05-31T19:56:08.984 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000009244f2839047d5-006477a658-1112-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 19:56:08 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T19:56:08.985 DEBUG:boto:b'pklxkd-7:1000false:2023-05-31T19:52:48.162Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T19:56:08.986 DEBUG:boto:path=/pklxkd-7/ 2023-05-31T19:56:08.986 DEBUG:boto:auth_path=/pklxkd-7/ 2023-05-31T19:56:08.986 DEBUG:boto:path=/pklxkd-7/?prefix=%3A 2023-05-31T19:56:08.987 DEBUG:boto:auth_path=/pklxkd-7/?prefix=%3A 2023-05-31T19:56:08.987 DEBUG:boto:Method: GET 2023-05-31T19:56:08.988 DEBUG:boto:Path: /pklxkd-7/?prefix=%3A 2023-05-31T19:56:08.988 DEBUG:boto:Data: 2023-05-31T19:56:08.988 DEBUG:boto:Headers: {} 2023-05-31T19:56:08.989 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T19:56:08.989 DEBUG:boto:Port: 8000 2023-05-31T19:56:08.989 DEBUG:boto:Params: {} 2023-05-31T19:56:08.990 DEBUG:boto:Token: None 2023-05-31T19:56:08.990 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 19:56:08 GMT /pklxkd-7/ 2023-05-31T19:56:08.991 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:DSBtXYx0ubm9AqsV8wT32ONzsVQ= 2023-05-31T19:56:08.991 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 19:56:08 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:DSBtXYx0ubm9AqsV8wT32ONzsVQ=', 'Content-Length': '0'} 2023-05-31T19:56:08.993 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000013ab1784bc7e3145-006477a658-110e-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 19:56:08 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T19:56:09.036 DEBUG:boto:b'pklxkd-7:1000false:2023-05-31T19:52:48.162Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T19:56:09.037 DEBUG:boto:path=/pklxkd-7/%3A 2023-05-31T19:56:09.037 DEBUG:boto:auth_path=/pklxkd-7/%3A 2023-05-31T19:56:09.038 DEBUG:boto:Method: HEAD 2023-05-31T19:56:09.038 DEBUG:boto:Path: /pklxkd-7/%3A 2023-05-31T19:56:09.038 DEBUG:boto:Data: 2023-05-31T19:56:09.039 DEBUG:boto:Headers: {} 2023-05-31T19:56:09.039 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T19:56:09.039 DEBUG:boto:Port: 8001 2023-05-31T19:56:09.040 DEBUG:boto:Params: {} 2023-05-31T19:56:09.040 DEBUG:boto:Token: None 2023-05-31T19:56:09.040 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 19:56:09 GMT /pklxkd-7/%3A 2023-05-31T19:56:09.041 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:ttyMLpOXVb8a/U+AlYxfWgjQMSo= 2023-05-31T19:56:09.041 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 19:56:09 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:ttyMLpOXVb8a/U+AlYxfWgjQMSo=', 'Content-Length': '0'} 2023-05-31T19:56:09.043 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 19:52:48 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000eac8d050a0a92bbb-006477a659-1112-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 19:56:09 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T19:56:09.044 DEBUG:boto:path=/pklxkd-7/%3A 2023-05-31T19:56:09.044 DEBUG:boto:auth_path=/pklxkd-7/%3A 2023-05-31T19:56:09.044 DEBUG:boto:Method: HEAD 2023-05-31T19:56:09.045 DEBUG:boto:Path: /pklxkd-7/%3A 2023-05-31T19:56:09.045 DEBUG:boto:Data: 2023-05-31T19:56:09.045 DEBUG:boto:Headers: {} 2023-05-31T19:56:09.046 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T19:56:09.046 DEBUG:boto:Port: 8000 2023-05-31T19:56:09.046 DEBUG:boto:Params: {} 2023-05-31T19:56:09.047 DEBUG:boto:Token: None 2023-05-31T19:56:09.047 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 19:56:09 GMT /pklxkd-7/%3A 2023-05-31T19:56:09.048 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:ttyMLpOXVb8a/U+AlYxfWgjQMSo= 2023-05-31T19:56:09.048 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 19:56:09 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:ttyMLpOXVb8a/U+AlYxfWgjQMSo=', 'Content-Length': '0'} 2023-05-31T19:56:09.050 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 19:52:48 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000379c1454f1861be8-006477a659-110e-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 19:56:09 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T19:56:09.050 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=: 2023-05-31T19:56:09.051 DEBUG:boto:path=/pklxkd-7/%3A 2023-05-31T19:56:09.051 DEBUG:boto:auth_path=/pklxkd-7/%3A 2023-05-31T19:56:09.051 DEBUG:boto:Method: GET 2023-05-31T19:56:09.052 DEBUG:boto:Path: /pklxkd-7/%3A 2023-05-31T19:56:09.052 DEBUG:boto:Data: 2023-05-31T19:56:09.052 DEBUG:boto:Headers: {} 2023-05-31T19:56:09.053 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T19:56:09.053 DEBUG:boto:Port: 8001 2023-05-31T19:56:09.053 DEBUG:boto:Params: {} 2023-05-31T19:56:09.054 DEBUG:boto:Token: None 2023-05-31T19:56:09.054 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 19:56:09 GMT /pklxkd-7/%3A 2023-05-31T19:56:09.055 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:KPj75T2NJLM2wIDwMUnRzdkJACw= 2023-05-31T19:56:09.055 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 19:56:09 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:KPj75T2NJLM2wIDwMUnRzdkJACw=', 'Content-Length': '0'} 2023-05-31T19:56:09.057 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 19:52:48 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000009191ae48a04ddb50-006477a659-1112-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 19:56:09 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T19:56:09.057 DEBUG:boto:path=/pklxkd-7/%3A 2023-05-31T19:56:09.058 DEBUG:boto:auth_path=/pklxkd-7/%3A 2023-05-31T19:56:09.058 DEBUG:boto:Method: GET 2023-05-31T19:56:09.059 DEBUG:boto:Path: /pklxkd-7/%3A 2023-05-31T19:56:09.059 DEBUG:boto:Data: 2023-05-31T19:56:09.059 DEBUG:boto:Headers: {} 2023-05-31T19:56:09.060 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T19:56:09.060 DEBUG:boto:Port: 8000 2023-05-31T19:56:09.061 DEBUG:boto:Params: {} 2023-05-31T19:56:09.061 DEBUG:boto:Token: None 2023-05-31T19:56:09.062 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 19:56:09 GMT /pklxkd-7/%3A 2023-05-31T19:56:09.062 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:KPj75T2NJLM2wIDwMUnRzdkJACw= 2023-05-31T19:56:09.063 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 19:56:09 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:KPj75T2NJLM2wIDwMUnRzdkJACw=', 'Content-Length': '0'} 2023-05-31T19:56:09.064 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 19:52:48 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000057d856189898cd48-006477a659-110e-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 19:56:09 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T19:56:09.065 DEBUG:boto:path=/pklxkd-7/ 2023-05-31T19:56:09.066 DEBUG:boto:auth_path=/pklxkd-7/ 2023-05-31T19:56:09.066 DEBUG:boto:path=/pklxkd-7/?prefix=%3A 2023-05-31T19:56:09.066 DEBUG:boto:auth_path=/pklxkd-7/?prefix=%3A 2023-05-31T19:56:09.067 DEBUG:boto:Method: GET 2023-05-31T19:56:09.067 DEBUG:boto:Path: /pklxkd-7/?prefix=%3A 2023-05-31T19:56:09.067 DEBUG:boto:Data: 2023-05-31T19:56:09.068 DEBUG:boto:Headers: {} 2023-05-31T19:56:09.068 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T19:56:09.069 DEBUG:boto:Port: 8001 2023-05-31T19:56:09.069 DEBUG:boto:Params: {} 2023-05-31T19:56:09.069 DEBUG:boto:Token: None 2023-05-31T19:56:09.070 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 19:56:09 GMT /pklxkd-7/ 2023-05-31T19:56:09.070 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:hTDo8we7N3jPGM/woJctQUnCKbo= 2023-05-31T19:56:09.071 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 19:56:09 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:hTDo8we7N3jPGM/woJctQUnCKbo=', 'Content-Length': '0'} 2023-05-31T19:56:09.073 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c9adbdf0346e16bb-006477a659-1112-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 19:56:09 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T19:56:09.116 DEBUG:boto:b'pklxkd-7:1000false:2023-05-31T19:52:48.162Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T19:56:09.117 DEBUG:boto:path=/pklxkd-7/ 2023-05-31T19:56:09.118 DEBUG:boto:auth_path=/pklxkd-7/ 2023-05-31T19:56:09.118 DEBUG:boto:path=/pklxkd-7/?prefix=%3A 2023-05-31T19:56:09.118 DEBUG:boto:auth_path=/pklxkd-7/?prefix=%3A 2023-05-31T19:56:09.119 DEBUG:boto:Method: GET 2023-05-31T19:56:09.119 DEBUG:boto:Path: /pklxkd-7/?prefix=%3A 2023-05-31T19:56:09.119 DEBUG:boto:Data: 2023-05-31T19:56:09.120 DEBUG:boto:Headers: {} 2023-05-31T19:56:09.120 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T19:56:09.120 DEBUG:boto:Port: 8000 2023-05-31T19:56:09.121 DEBUG:boto:Params: {} 2023-05-31T19:56:09.121 DEBUG:boto:Token: None 2023-05-31T19:56:09.122 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 19:56:09 GMT /pklxkd-7/ 2023-05-31T19:56:09.122 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:hTDo8we7N3jPGM/woJctQUnCKbo= 2023-05-31T19:56:09.122 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 19:56:09 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:hTDo8we7N3jPGM/woJctQUnCKbo=', 'Content-Length': '0'} 2023-05-31T19:56:09.125 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000083a8c2056a12f708-006477a659-110e-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 19:56:09 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T19:56:09.125 DEBUG:boto:b'pklxkd-7:1000false:2023-05-31T19:52:48.162Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T19:56:09.126 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=_myobj 2023-05-31T19:56:09.126 DEBUG:boto:path=/pklxkd-7/_myobj 2023-05-31T19:56:09.127 DEBUG:boto:auth_path=/pklxkd-7/_myobj 2023-05-31T19:56:09.127 DEBUG:boto:path=/pklxkd-7/_myobj?versionId=null 2023-05-31T19:56:09.127 DEBUG:boto:auth_path=/pklxkd-7/_myobj?versionId=null 2023-05-31T19:56:09.128 DEBUG:boto:Method: GET 2023-05-31T19:56:09.128 DEBUG:boto:Path: /pklxkd-7/_myobj?versionId=null 2023-05-31T19:56:09.129 DEBUG:boto:Data: 2023-05-31T19:56:09.129 DEBUG:boto:Headers: {} 2023-05-31T19:56:09.129 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T19:56:09.130 DEBUG:boto:Port: 8001 2023-05-31T19:56:09.130 DEBUG:boto:Params: {} 2023-05-31T19:56:09.130 DEBUG:boto:Token: None 2023-05-31T19:56:09.131 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 19:56:09 GMT /pklxkd-7/_myobj?versionId=null 2023-05-31T19:56:09.131 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:lE+YBBZ/1TTyWAmA2UrkYO4uSeI= 2023-05-31T19:56:09.132 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 19:56:09 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:lE+YBBZ/1TTyWAmA2UrkYO4uSeI=', 'Content-Length': '0'} 2023-05-31T19:56:09.133 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 19:52:48 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000009d903b8f5a2e1324-006477a659-1112-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 19:56:09 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T19:56:09.134 DEBUG:boto:path=/pklxkd-7/_myobj 2023-05-31T19:56:09.135 DEBUG:boto:auth_path=/pklxkd-7/_myobj 2023-05-31T19:56:09.135 DEBUG:boto:path=/pklxkd-7/_myobj?versionId=null 2023-05-31T19:56:09.135 DEBUG:boto:auth_path=/pklxkd-7/_myobj?versionId=null 2023-05-31T19:56:09.136 DEBUG:boto:Method: GET 2023-05-31T19:56:09.136 DEBUG:boto:Path: /pklxkd-7/_myobj?versionId=null 2023-05-31T19:56:09.136 DEBUG:boto:Data: 2023-05-31T19:56:09.137 DEBUG:boto:Headers: {} 2023-05-31T19:56:09.137 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T19:56:09.138 DEBUG:boto:Port: 8000 2023-05-31T19:56:09.138 DEBUG:boto:Params: {} 2023-05-31T19:56:09.138 DEBUG:boto:Token: None 2023-05-31T19:56:09.139 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 19:56:09 GMT /pklxkd-7/_myobj?versionId=null 2023-05-31T19:56:09.139 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:lE+YBBZ/1TTyWAmA2UrkYO4uSeI= 2023-05-31T19:56:09.139 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 19:56:09 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:lE+YBBZ/1TTyWAmA2UrkYO4uSeI=', 'Content-Length': '0'} 2023-05-31T19:56:09.141 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 19:52:48 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000002b0b1d38212ac9a2-006477a659-110e-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 19:56:09 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T19:56:09.142 DEBUG:boto:path=/pklxkd-7/_myobj 2023-05-31T19:56:09.142 DEBUG:boto:auth_path=/pklxkd-7/_myobj 2023-05-31T19:56:09.143 DEBUG:boto:path=/pklxkd-7/_myobj?versionId=null 2023-05-31T19:56:09.143 DEBUG:boto:auth_path=/pklxkd-7/_myobj?versionId=null 2023-05-31T19:56:09.143 DEBUG:boto:Method: HEAD 2023-05-31T19:56:09.144 DEBUG:boto:Path: /pklxkd-7/_myobj?versionId=null 2023-05-31T19:56:09.144 DEBUG:boto:Data: 2023-05-31T19:56:09.145 DEBUG:boto:Headers: {} 2023-05-31T19:56:09.145 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T19:56:09.145 DEBUG:boto:Port: 8001 2023-05-31T19:56:09.146 DEBUG:boto:Params: {} 2023-05-31T19:56:09.146 DEBUG:boto:Token: None 2023-05-31T19:56:09.147 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 19:56:09 GMT /pklxkd-7/_myobj?versionId=null 2023-05-31T19:56:09.147 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:8NhchEfq5zGlJ5KlM82r/Upoouk= 2023-05-31T19:56:09.147 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 19:56:09 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:8NhchEfq5zGlJ5KlM82r/Upoouk=', 'Content-Length': '0'} 2023-05-31T19:56:09.149 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 19:52:48 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000003c672e54c289a4d3-006477a659-1112-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 19:56:09 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T19:56:09.150 DEBUG:boto:path=/pklxkd-7/_myobj 2023-05-31T19:56:09.150 DEBUG:boto:auth_path=/pklxkd-7/_myobj 2023-05-31T19:56:09.150 DEBUG:boto:path=/pklxkd-7/_myobj?versionId=null 2023-05-31T19:56:09.151 DEBUG:boto:auth_path=/pklxkd-7/_myobj?versionId=null 2023-05-31T19:56:09.151 DEBUG:boto:Method: HEAD 2023-05-31T19:56:09.151 DEBUG:boto:Path: /pklxkd-7/_myobj?versionId=null 2023-05-31T19:56:09.152 DEBUG:boto:Data: 2023-05-31T19:56:09.152 DEBUG:boto:Headers: {} 2023-05-31T19:56:09.152 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T19:56:09.153 DEBUG:boto:Port: 8000 2023-05-31T19:56:09.153 DEBUG:boto:Params: {} 2023-05-31T19:56:09.154 DEBUG:boto:Token: None 2023-05-31T19:56:09.154 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 19:56:09 GMT /pklxkd-7/_myobj?versionId=null 2023-05-31T19:56:09.154 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:8NhchEfq5zGlJ5KlM82r/Upoouk= 2023-05-31T19:56:09.155 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 19:56:09 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:8NhchEfq5zGlJ5KlM82r/Upoouk=', 'Content-Length': '0'} 2023-05-31T19:56:09.156 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 19:52:48 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000d668a55606de3946-006477a659-110e-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 19:56:09 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T19:56:09.157 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=_myobj 2023-05-31T19:56:09.157 DEBUG:boto:path=/pklxkd-7/_myobj 2023-05-31T19:56:09.158 DEBUG:boto:auth_path=/pklxkd-7/_myobj 2023-05-31T19:56:09.158 DEBUG:boto:path=/pklxkd-7/_myobj?versionId=null 2023-05-31T19:56:09.158 DEBUG:boto:auth_path=/pklxkd-7/_myobj?versionId=null 2023-05-31T19:56:09.159 DEBUG:boto:Method: GET 2023-05-31T19:56:09.159 DEBUG:boto:Path: /pklxkd-7/_myobj?versionId=null 2023-05-31T19:56:09.160 DEBUG:boto:Data: 2023-05-31T19:56:09.160 DEBUG:boto:Headers: {} 2023-05-31T19:56:09.160 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T19:56:09.161 DEBUG:boto:Port: 8001 2023-05-31T19:56:09.161 DEBUG:boto:Params: {} 2023-05-31T19:56:09.161 DEBUG:boto:Token: None 2023-05-31T19:56:09.162 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 19:56:09 GMT /pklxkd-7/_myobj?versionId=null 2023-05-31T19:56:09.162 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:lE+YBBZ/1TTyWAmA2UrkYO4uSeI= 2023-05-31T19:56:09.163 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 19:56:09 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:lE+YBBZ/1TTyWAmA2UrkYO4uSeI=', 'Content-Length': '0'} 2023-05-31T19:56:09.165 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 19:52:48 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000002006fe4e562810b1-006477a659-1112-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 19:56:09 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T19:56:09.165 DEBUG:boto:path=/pklxkd-7/_myobj 2023-05-31T19:56:09.166 DEBUG:boto:auth_path=/pklxkd-7/_myobj 2023-05-31T19:56:09.166 DEBUG:boto:path=/pklxkd-7/_myobj?versionId=null 2023-05-31T19:56:09.166 DEBUG:boto:auth_path=/pklxkd-7/_myobj?versionId=null 2023-05-31T19:56:09.167 DEBUG:boto:Method: GET 2023-05-31T19:56:09.167 DEBUG:boto:Path: /pklxkd-7/_myobj?versionId=null 2023-05-31T19:56:09.167 DEBUG:boto:Data: 2023-05-31T19:56:09.168 DEBUG:boto:Headers: {} 2023-05-31T19:56:09.168 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T19:56:09.168 DEBUG:boto:Port: 8000 2023-05-31T19:56:09.169 DEBUG:boto:Params: {} 2023-05-31T19:56:09.169 DEBUG:boto:Token: None 2023-05-31T19:56:09.170 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 19:56:09 GMT /pklxkd-7/_myobj?versionId=null 2023-05-31T19:56:09.170 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:lE+YBBZ/1TTyWAmA2UrkYO4uSeI= 2023-05-31T19:56:09.170 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 19:56:09 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:lE+YBBZ/1TTyWAmA2UrkYO4uSeI=', 'Content-Length': '0'} 2023-05-31T19:56:09.172 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 19:52:48 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000bb2145261e037dc4-006477a659-110e-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 19:56:09 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T19:56:09.216 DEBUG:boto:path=/pklxkd-7/ 2023-05-31T19:56:09.217 DEBUG:boto:auth_path=/pklxkd-7/ 2023-05-31T19:56:09.217 DEBUG:boto:path=/pklxkd-7/?prefix=_myobj 2023-05-31T19:56:09.217 DEBUG:boto:auth_path=/pklxkd-7/?prefix=_myobj 2023-05-31T19:56:09.218 DEBUG:boto:Method: GET 2023-05-31T19:56:09.218 DEBUG:boto:Path: /pklxkd-7/?prefix=_myobj 2023-05-31T19:56:09.218 DEBUG:boto:Data: 2023-05-31T19:56:09.219 DEBUG:boto:Headers: {} 2023-05-31T19:56:09.219 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T19:56:09.219 DEBUG:boto:Port: 8001 2023-05-31T19:56:09.220 DEBUG:boto:Params: {} 2023-05-31T19:56:09.220 DEBUG:boto:Token: None 2023-05-31T19:56:09.221 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 19:56:09 GMT /pklxkd-7/ 2023-05-31T19:56:09.221 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:hTDo8we7N3jPGM/woJctQUnCKbo= 2023-05-31T19:56:09.222 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 19:56:09 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:hTDo8we7N3jPGM/woJctQUnCKbo=', 'Content-Length': '0'} 2023-05-31T19:56:09.224 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000074126d2e7b341a3e-006477a659-1112-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 19:56:09 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T19:56:09.224 DEBUG:boto:b'pklxkd-7_myobj1000false_myobj2023-05-31T19:52:48.103Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T19:56:09.225 DEBUG:boto:path=/pklxkd-7/ 2023-05-31T19:56:09.225 DEBUG:boto:auth_path=/pklxkd-7/ 2023-05-31T19:56:09.226 DEBUG:boto:path=/pklxkd-7/?prefix=_myobj 2023-05-31T19:56:09.226 DEBUG:boto:auth_path=/pklxkd-7/?prefix=_myobj 2023-05-31T19:56:09.226 DEBUG:boto:Method: GET 2023-05-31T19:56:09.227 DEBUG:boto:Path: /pklxkd-7/?prefix=_myobj 2023-05-31T19:56:09.227 DEBUG:boto:Data: 2023-05-31T19:56:09.228 DEBUG:boto:Headers: {} 2023-05-31T19:56:09.228 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T19:56:09.228 DEBUG:boto:Port: 8000 2023-05-31T19:56:09.229 DEBUG:boto:Params: {} 2023-05-31T19:56:09.229 DEBUG:boto:Token: None 2023-05-31T19:56:09.229 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 19:56:09 GMT /pklxkd-7/ 2023-05-31T19:56:09.230 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:hTDo8we7N3jPGM/woJctQUnCKbo= 2023-05-31T19:56:09.230 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 19:56:09 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:hTDo8we7N3jPGM/woJctQUnCKbo=', 'Content-Length': '0'} 2023-05-31T19:56:09.233 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000690b4a26a436d69e-006477a659-110e-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 19:56:09 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T19:56:09.233 DEBUG:boto:b'pklxkd-7_myobj1000false_myobj2023-05-31T19:52:48.103Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T19:56:09.234 DEBUG:boto:path=/pklxkd-7/_myobj 2023-05-31T19:56:09.234 DEBUG:boto:auth_path=/pklxkd-7/_myobj 2023-05-31T19:56:09.235 DEBUG:boto:Method: HEAD 2023-05-31T19:56:09.235 DEBUG:boto:Path: /pklxkd-7/_myobj 2023-05-31T19:56:09.235 DEBUG:boto:Data: 2023-05-31T19:56:09.236 DEBUG:boto:Headers: {} 2023-05-31T19:56:09.236 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T19:56:09.236 DEBUG:boto:Port: 8001 2023-05-31T19:56:09.237 DEBUG:boto:Params: {} 2023-05-31T19:56:09.237 DEBUG:boto:Token: None 2023-05-31T19:56:09.237 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 19:56:09 GMT /pklxkd-7/_myobj 2023-05-31T19:56:09.238 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:z2tBqWWrYlMUwsD0iJkrtBK4e2M= 2023-05-31T19:56:09.238 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 19:56:09 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:z2tBqWWrYlMUwsD0iJkrtBK4e2M=', 'Content-Length': '0'} 2023-05-31T19:56:09.240 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 19:52:48 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000111f78730dbbfd74-006477a659-1112-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 19:56:09 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T19:56:09.241 DEBUG:boto:path=/pklxkd-7/_myobj 2023-05-31T19:56:09.241 DEBUG:boto:auth_path=/pklxkd-7/_myobj 2023-05-31T19:56:09.241 DEBUG:boto:Method: HEAD 2023-05-31T19:56:09.242 DEBUG:boto:Path: /pklxkd-7/_myobj 2023-05-31T19:56:09.242 DEBUG:boto:Data: 2023-05-31T19:56:09.242 DEBUG:boto:Headers: {} 2023-05-31T19:56:09.243 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T19:56:09.243 DEBUG:boto:Port: 8000 2023-05-31T19:56:09.243 DEBUG:boto:Params: {} 2023-05-31T19:56:09.244 DEBUG:boto:Token: None 2023-05-31T19:56:09.244 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 19:56:09 GMT /pklxkd-7/_myobj 2023-05-31T19:56:09.245 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:z2tBqWWrYlMUwsD0iJkrtBK4e2M= 2023-05-31T19:56:09.245 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 19:56:09 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:z2tBqWWrYlMUwsD0iJkrtBK4e2M=', 'Content-Length': '0'} 2023-05-31T19:56:09.247 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 19:52:48 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000ef703256a034983d-006477a659-110e-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 19:56:09 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T19:56:09.247 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=_myobj 2023-05-31T19:56:09.248 DEBUG:boto:path=/pklxkd-7/_myobj 2023-05-31T19:56:09.248 DEBUG:boto:auth_path=/pklxkd-7/_myobj 2023-05-31T19:56:09.248 DEBUG:boto:Method: GET 2023-05-31T19:56:09.249 DEBUG:boto:Path: /pklxkd-7/_myobj 2023-05-31T19:56:09.249 DEBUG:boto:Data: 2023-05-31T19:56:09.249 DEBUG:boto:Headers: {} 2023-05-31T19:56:09.250 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T19:56:09.250 DEBUG:boto:Port: 8001 2023-05-31T19:56:09.251 DEBUG:boto:Params: {} 2023-05-31T19:56:09.251 DEBUG:boto:Token: None 2023-05-31T19:56:09.251 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 19:56:09 GMT /pklxkd-7/_myobj 2023-05-31T19:56:09.252 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:HUntbnGJOfJfZssOMnvzfhh4aKE= 2023-05-31T19:56:09.252 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 19:56:09 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:HUntbnGJOfJfZssOMnvzfhh4aKE=', 'Content-Length': '0'} 2023-05-31T19:56:09.254 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 19:52:48 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000174f20f2babc38cd-006477a659-1112-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 19:56:09 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T19:56:09.255 DEBUG:boto:path=/pklxkd-7/_myobj 2023-05-31T19:56:09.255 DEBUG:boto:auth_path=/pklxkd-7/_myobj 2023-05-31T19:56:09.255 DEBUG:boto:Method: GET 2023-05-31T19:56:09.256 DEBUG:boto:Path: /pklxkd-7/_myobj 2023-05-31T19:56:09.256 DEBUG:boto:Data: 2023-05-31T19:56:09.257 DEBUG:boto:Headers: {} 2023-05-31T19:56:09.257 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T19:56:09.257 DEBUG:boto:Port: 8000 2023-05-31T19:56:09.258 DEBUG:boto:Params: {} 2023-05-31T19:56:09.258 DEBUG:boto:Token: None 2023-05-31T19:56:09.258 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 19:56:09 GMT /pklxkd-7/_myobj 2023-05-31T19:56:09.259 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:HUntbnGJOfJfZssOMnvzfhh4aKE= 2023-05-31T19:56:09.259 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 19:56:09 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:HUntbnGJOfJfZssOMnvzfhh4aKE=', 'Content-Length': '0'} 2023-05-31T19:56:09.261 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 19:52:48 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000ec4f2bf952672726-006477a659-110e-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 19:56:09 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T19:56:09.262 DEBUG:boto:path=/pklxkd-7/ 2023-05-31T19:56:09.262 DEBUG:boto:auth_path=/pklxkd-7/ 2023-05-31T19:56:09.263 DEBUG:boto:path=/pklxkd-7/?prefix=_myobj 2023-05-31T19:56:09.263 DEBUG:boto:auth_path=/pklxkd-7/?prefix=_myobj 2023-05-31T19:56:09.263 DEBUG:boto:Method: GET 2023-05-31T19:56:09.264 DEBUG:boto:Path: /pklxkd-7/?prefix=_myobj 2023-05-31T19:56:09.264 DEBUG:boto:Data: 2023-05-31T19:56:09.264 DEBUG:boto:Headers: {} 2023-05-31T19:56:09.265 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T19:56:09.265 DEBUG:boto:Port: 8001 2023-05-31T19:56:09.265 DEBUG:boto:Params: {} 2023-05-31T19:56:09.266 DEBUG:boto:Token: None 2023-05-31T19:56:09.266 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 19:56:09 GMT /pklxkd-7/ 2023-05-31T19:56:09.267 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:hTDo8we7N3jPGM/woJctQUnCKbo= 2023-05-31T19:56:09.267 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 19:56:09 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:hTDo8we7N3jPGM/woJctQUnCKbo=', 'Content-Length': '0'} 2023-05-31T19:56:09.269 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ca3244c32bbb0941-006477a659-1112-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 19:56:09 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T19:56:09.270 DEBUG:boto:b'pklxkd-7_myobj1000false_myobj2023-05-31T19:52:48.103Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T19:56:09.270 DEBUG:boto:path=/pklxkd-7/ 2023-05-31T19:56:09.271 DEBUG:boto:auth_path=/pklxkd-7/ 2023-05-31T19:56:09.271 DEBUG:boto:path=/pklxkd-7/?prefix=_myobj 2023-05-31T19:56:09.272 DEBUG:boto:auth_path=/pklxkd-7/?prefix=_myobj 2023-05-31T19:56:09.272 DEBUG:boto:Method: GET 2023-05-31T19:56:09.272 DEBUG:boto:Path: /pklxkd-7/?prefix=_myobj 2023-05-31T19:56:09.273 DEBUG:boto:Data: 2023-05-31T19:56:09.273 DEBUG:boto:Headers: {} 2023-05-31T19:56:09.273 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T19:56:09.274 DEBUG:boto:Port: 8000 2023-05-31T19:56:09.274 DEBUG:boto:Params: {} 2023-05-31T19:56:09.274 DEBUG:boto:Token: None 2023-05-31T19:56:09.275 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 19:56:09 GMT /pklxkd-7/ 2023-05-31T19:56:09.275 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:hTDo8we7N3jPGM/woJctQUnCKbo= 2023-05-31T19:56:09.276 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 19:56:09 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:hTDo8we7N3jPGM/woJctQUnCKbo=', 'Content-Length': '0'} 2023-05-31T19:56:09.278 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000024f039b99afb454e-006477a659-110e-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 19:56:09 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T19:56:09.320 DEBUG:boto:b'pklxkd-7_myobj1000false_myobj2023-05-31T19:52:48.103Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T19:56:09.321 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2023-05-31T19:56:09.321 DEBUG:boto:path=/pklxkd-7/myobj 2023-05-31T19:56:09.322 DEBUG:boto:auth_path=/pklxkd-7/myobj 2023-05-31T19:56:09.322 DEBUG:boto:path=/pklxkd-7/myobj?versionId=null 2023-05-31T19:56:09.322 DEBUG:boto:auth_path=/pklxkd-7/myobj?versionId=null 2023-05-31T19:56:09.323 DEBUG:boto:Method: GET 2023-05-31T19:56:09.323 DEBUG:boto:Path: /pklxkd-7/myobj?versionId=null 2023-05-31T19:56:09.323 DEBUG:boto:Data: 2023-05-31T19:56:09.324 DEBUG:boto:Headers: {} 2023-05-31T19:56:09.324 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T19:56:09.324 DEBUG:boto:Port: 8001 2023-05-31T19:56:09.325 DEBUG:boto:Params: {} 2023-05-31T19:56:09.325 DEBUG:boto:Token: None 2023-05-31T19:56:09.326 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 19:56:09 GMT /pklxkd-7/myobj?versionId=null 2023-05-31T19:56:09.326 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:UhkAElLbfbuiEX2bNjkeFN2wZ4Q= 2023-05-31T19:56:09.326 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 19:56:09 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:UhkAElLbfbuiEX2bNjkeFN2wZ4Q=', 'Content-Length': '0'} 2023-05-31T19:56:09.329 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 19:52:48 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000437fbb0f5fa60b2a-006477a659-1112-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 19:56:09 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T19:56:09.329 DEBUG:boto:path=/pklxkd-7/myobj 2023-05-31T19:56:09.330 DEBUG:boto:auth_path=/pklxkd-7/myobj 2023-05-31T19:56:09.330 DEBUG:boto:path=/pklxkd-7/myobj?versionId=null 2023-05-31T19:56:09.330 DEBUG:boto:auth_path=/pklxkd-7/myobj?versionId=null 2023-05-31T19:56:09.331 DEBUG:boto:Method: GET 2023-05-31T19:56:09.331 DEBUG:boto:Path: /pklxkd-7/myobj?versionId=null 2023-05-31T19:56:09.331 DEBUG:boto:Data: 2023-05-31T19:56:09.332 DEBUG:boto:Headers: {} 2023-05-31T19:56:09.332 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T19:56:09.333 DEBUG:boto:Port: 8000 2023-05-31T19:56:09.333 DEBUG:boto:Params: {} 2023-05-31T19:56:09.333 DEBUG:boto:Token: None 2023-05-31T19:56:09.334 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 19:56:09 GMT /pklxkd-7/myobj?versionId=null 2023-05-31T19:56:09.334 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:UhkAElLbfbuiEX2bNjkeFN2wZ4Q= 2023-05-31T19:56:09.335 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 19:56:09 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:UhkAElLbfbuiEX2bNjkeFN2wZ4Q=', 'Content-Length': '0'} 2023-05-31T19:56:09.336 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 19:52:48 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000cfa5b03e4f18530c-006477a659-110e-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 19:56:09 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T19:56:09.337 DEBUG:boto:path=/pklxkd-7/myobj 2023-05-31T19:56:09.337 DEBUG:boto:auth_path=/pklxkd-7/myobj 2023-05-31T19:56:09.338 DEBUG:boto:path=/pklxkd-7/myobj?versionId=null 2023-05-31T19:56:09.338 DEBUG:boto:auth_path=/pklxkd-7/myobj?versionId=null 2023-05-31T19:56:09.338 DEBUG:boto:Method: HEAD 2023-05-31T19:56:09.339 DEBUG:boto:Path: /pklxkd-7/myobj?versionId=null 2023-05-31T19:56:09.339 DEBUG:boto:Data: 2023-05-31T19:56:09.339 DEBUG:boto:Headers: {} 2023-05-31T19:56:09.340 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T19:56:09.340 DEBUG:boto:Port: 8001 2023-05-31T19:56:09.340 DEBUG:boto:Params: {} 2023-05-31T19:56:09.341 DEBUG:boto:Token: None 2023-05-31T19:56:09.341 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 19:56:09 GMT /pklxkd-7/myobj?versionId=null 2023-05-31T19:56:09.342 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:2ehoWTq8uWxAA8Yq7nuh2hIIaxo= 2023-05-31T19:56:09.342 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 19:56:09 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:2ehoWTq8uWxAA8Yq7nuh2hIIaxo=', 'Content-Length': '0'} 2023-05-31T19:56:09.344 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 19:52:48 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000396ec1cb472fe588-006477a659-1112-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 19:56:09 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T19:56:09.344 DEBUG:boto:path=/pklxkd-7/myobj 2023-05-31T19:56:09.345 DEBUG:boto:auth_path=/pklxkd-7/myobj 2023-05-31T19:56:09.345 DEBUG:boto:path=/pklxkd-7/myobj?versionId=null 2023-05-31T19:56:09.346 DEBUG:boto:auth_path=/pklxkd-7/myobj?versionId=null 2023-05-31T19:56:09.346 DEBUG:boto:Method: HEAD 2023-05-31T19:56:09.346 DEBUG:boto:Path: /pklxkd-7/myobj?versionId=null 2023-05-31T19:56:09.347 DEBUG:boto:Data: 2023-05-31T19:56:09.347 DEBUG:boto:Headers: {} 2023-05-31T19:56:09.347 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T19:56:09.347 DEBUG:boto:Port: 8000 2023-05-31T19:56:09.348 DEBUG:boto:Params: {} 2023-05-31T19:56:09.348 DEBUG:boto:Token: None 2023-05-31T19:56:09.349 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 19:56:09 GMT /pklxkd-7/myobj?versionId=null 2023-05-31T19:56:09.349 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:2ehoWTq8uWxAA8Yq7nuh2hIIaxo= 2023-05-31T19:56:09.349 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 19:56:09 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:2ehoWTq8uWxAA8Yq7nuh2hIIaxo=', 'Content-Length': '0'} 2023-05-31T19:56:09.351 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 19:52:48 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000587e5f13bf301926-006477a659-110e-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 19:56:09 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T19:56:09.352 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2023-05-31T19:56:09.352 DEBUG:boto:path=/pklxkd-7/myobj 2023-05-31T19:56:09.352 DEBUG:boto:auth_path=/pklxkd-7/myobj 2023-05-31T19:56:09.353 DEBUG:boto:path=/pklxkd-7/myobj?versionId=null 2023-05-31T19:56:09.353 DEBUG:boto:auth_path=/pklxkd-7/myobj?versionId=null 2023-05-31T19:56:09.353 DEBUG:boto:Method: GET 2023-05-31T19:56:09.354 DEBUG:boto:Path: /pklxkd-7/myobj?versionId=null 2023-05-31T19:56:09.354 DEBUG:boto:Data: 2023-05-31T19:56:09.355 DEBUG:boto:Headers: {} 2023-05-31T19:56:09.355 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T19:56:09.355 DEBUG:boto:Port: 8001 2023-05-31T19:56:09.356 DEBUG:boto:Params: {} 2023-05-31T19:56:09.356 DEBUG:boto:Token: None 2023-05-31T19:56:09.356 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 19:56:09 GMT /pklxkd-7/myobj?versionId=null 2023-05-31T19:56:09.357 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:UhkAElLbfbuiEX2bNjkeFN2wZ4Q= 2023-05-31T19:56:09.357 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 19:56:09 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:UhkAElLbfbuiEX2bNjkeFN2wZ4Q=', 'Content-Length': '0'} 2023-05-31T19:56:09.359 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 19:52:48 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000007c928a0aff015033-006477a659-1112-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 19:56:09 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T19:56:09.360 DEBUG:boto:path=/pklxkd-7/myobj 2023-05-31T19:56:09.360 DEBUG:boto:auth_path=/pklxkd-7/myobj 2023-05-31T19:56:09.361 DEBUG:boto:path=/pklxkd-7/myobj?versionId=null 2023-05-31T19:56:09.361 DEBUG:boto:auth_path=/pklxkd-7/myobj?versionId=null 2023-05-31T19:56:09.361 DEBUG:boto:Method: GET 2023-05-31T19:56:09.362 DEBUG:boto:Path: /pklxkd-7/myobj?versionId=null 2023-05-31T19:56:09.362 DEBUG:boto:Data: 2023-05-31T19:56:09.362 DEBUG:boto:Headers: {} 2023-05-31T19:56:09.363 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T19:56:09.363 DEBUG:boto:Port: 8000 2023-05-31T19:56:09.363 DEBUG:boto:Params: {} 2023-05-31T19:56:09.364 DEBUG:boto:Token: None 2023-05-31T19:56:09.364 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 19:56:09 GMT /pklxkd-7/myobj?versionId=null 2023-05-31T19:56:09.365 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:UhkAElLbfbuiEX2bNjkeFN2wZ4Q= 2023-05-31T19:56:09.365 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 19:56:09 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:UhkAElLbfbuiEX2bNjkeFN2wZ4Q=', 'Content-Length': '0'} 2023-05-31T19:56:09.367 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 19:52:48 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000007f4c159dbb0fe47d-006477a659-110e-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 19:56:09 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T19:56:09.367 DEBUG:boto:path=/pklxkd-7/ 2023-05-31T19:56:09.368 DEBUG:boto:auth_path=/pklxkd-7/ 2023-05-31T19:56:09.368 DEBUG:boto:path=/pklxkd-7/?prefix=myobj 2023-05-31T19:56:09.369 DEBUG:boto:auth_path=/pklxkd-7/?prefix=myobj 2023-05-31T19:56:09.369 DEBUG:boto:Method: GET 2023-05-31T19:56:09.369 DEBUG:boto:Path: /pklxkd-7/?prefix=myobj 2023-05-31T19:56:09.370 DEBUG:boto:Data: 2023-05-31T19:56:09.370 DEBUG:boto:Headers: {} 2023-05-31T19:56:09.370 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T19:56:09.371 DEBUG:boto:Port: 8001 2023-05-31T19:56:09.371 DEBUG:boto:Params: {} 2023-05-31T19:56:09.371 DEBUG:boto:Token: None 2023-05-31T19:56:09.372 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 19:56:09 GMT /pklxkd-7/ 2023-05-31T19:56:09.372 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:hTDo8we7N3jPGM/woJctQUnCKbo= 2023-05-31T19:56:09.373 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 19:56:09 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:hTDo8we7N3jPGM/woJctQUnCKbo=', 'Content-Length': '0'} 2023-05-31T19:56:09.375 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000019575069816396a3-006477a659-1112-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 19:56:09 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T19:56:09.376 DEBUG:boto:b'pklxkd-7myobj1000falsemyobj2023-05-31T19:52:48.073Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T19:56:09.376 DEBUG:boto:path=/pklxkd-7/ 2023-05-31T19:56:09.377 DEBUG:boto:auth_path=/pklxkd-7/ 2023-05-31T19:56:09.377 DEBUG:boto:path=/pklxkd-7/?prefix=myobj 2023-05-31T19:56:09.377 DEBUG:boto:auth_path=/pklxkd-7/?prefix=myobj 2023-05-31T19:56:09.378 DEBUG:boto:Method: GET 2023-05-31T19:56:09.378 DEBUG:boto:Path: /pklxkd-7/?prefix=myobj 2023-05-31T19:56:09.378 DEBUG:boto:Data: 2023-05-31T19:56:09.379 DEBUG:boto:Headers: {} 2023-05-31T19:56:09.379 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T19:56:09.379 DEBUG:boto:Port: 8000 2023-05-31T19:56:09.380 DEBUG:boto:Params: {} 2023-05-31T19:56:09.380 DEBUG:boto:Token: None 2023-05-31T19:56:09.381 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 19:56:09 GMT /pklxkd-7/ 2023-05-31T19:56:09.381 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:hTDo8we7N3jPGM/woJctQUnCKbo= 2023-05-31T19:56:09.382 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 19:56:09 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:hTDo8we7N3jPGM/woJctQUnCKbo=', 'Content-Length': '0'} 2023-05-31T19:56:09.384 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000008e247cfdc283b3d-006477a659-110e-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 19:56:09 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T19:56:09.384 DEBUG:boto:b'pklxkd-7myobj1000falsemyobj2023-05-31T19:52:48.073Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T19:56:09.385 DEBUG:boto:path=/pklxkd-7/myobj 2023-05-31T19:56:09.385 DEBUG:boto:auth_path=/pklxkd-7/myobj 2023-05-31T19:56:09.386 DEBUG:boto:Method: HEAD 2023-05-31T19:56:09.386 DEBUG:boto:Path: /pklxkd-7/myobj 2023-05-31T19:56:09.386 DEBUG:boto:Data: 2023-05-31T19:56:09.387 DEBUG:boto:Headers: {} 2023-05-31T19:56:09.387 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T19:56:09.387 DEBUG:boto:Port: 8001 2023-05-31T19:56:09.388 DEBUG:boto:Params: {} 2023-05-31T19:56:09.388 DEBUG:boto:Token: None 2023-05-31T19:56:09.388 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 19:56:09 GMT /pklxkd-7/myobj 2023-05-31T19:56:09.389 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:mzw23RVZJkZ5wvL+QWa8Wj9/Nm8= 2023-05-31T19:56:09.389 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 19:56:09 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:mzw23RVZJkZ5wvL+QWa8Wj9/Nm8=', 'Content-Length': '0'} 2023-05-31T19:56:09.391 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 19:52:48 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000017b678533a51dc9c-006477a659-1112-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 19:56:09 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T19:56:09.392 DEBUG:boto:path=/pklxkd-7/myobj 2023-05-31T19:56:09.392 DEBUG:boto:auth_path=/pklxkd-7/myobj 2023-05-31T19:56:09.393 DEBUG:boto:Method: HEAD 2023-05-31T19:56:09.393 DEBUG:boto:Path: /pklxkd-7/myobj 2023-05-31T19:56:09.393 DEBUG:boto:Data: 2023-05-31T19:56:09.394 DEBUG:boto:Headers: {} 2023-05-31T19:56:09.394 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T19:56:09.394 DEBUG:boto:Port: 8000 2023-05-31T19:56:09.395 DEBUG:boto:Params: {} 2023-05-31T19:56:09.395 DEBUG:boto:Token: None 2023-05-31T19:56:09.395 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 19:56:09 GMT /pklxkd-7/myobj 2023-05-31T19:56:09.396 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:mzw23RVZJkZ5wvL+QWa8Wj9/Nm8= 2023-05-31T19:56:09.396 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 19:56:09 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:mzw23RVZJkZ5wvL+QWa8Wj9/Nm8=', 'Content-Length': '0'} 2023-05-31T19:56:09.398 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 19:52:48 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000005cb9232b0a849122-006477a659-110e-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 19:56:09 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T19:56:09.398 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2023-05-31T19:56:09.399 DEBUG:boto:path=/pklxkd-7/myobj 2023-05-31T19:56:09.399 DEBUG:boto:auth_path=/pklxkd-7/myobj 2023-05-31T19:56:09.399 DEBUG:boto:Method: GET 2023-05-31T19:56:09.400 DEBUG:boto:Path: /pklxkd-7/myobj 2023-05-31T19:56:09.400 DEBUG:boto:Data: 2023-05-31T19:56:09.401 DEBUG:boto:Headers: {} 2023-05-31T19:56:09.401 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T19:56:09.401 DEBUG:boto:Port: 8001 2023-05-31T19:56:09.402 DEBUG:boto:Params: {} 2023-05-31T19:56:09.402 DEBUG:boto:Token: None 2023-05-31T19:56:09.402 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 19:56:09 GMT /pklxkd-7/myobj 2023-05-31T19:56:09.403 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:ULLXdnW7aIOf8X+LaVrUAnL9wjk= 2023-05-31T19:56:09.403 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 19:56:09 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:ULLXdnW7aIOf8X+LaVrUAnL9wjk=', 'Content-Length': '0'} 2023-05-31T19:56:09.405 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 19:52:48 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000ecb54fbeb43b4d80-006477a659-1112-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 19:56:09 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T19:56:09.406 DEBUG:boto:path=/pklxkd-7/myobj 2023-05-31T19:56:09.406 DEBUG:boto:auth_path=/pklxkd-7/myobj 2023-05-31T19:56:09.407 DEBUG:boto:Method: GET 2023-05-31T19:56:09.407 DEBUG:boto:Path: /pklxkd-7/myobj 2023-05-31T19:56:09.407 DEBUG:boto:Data: 2023-05-31T19:56:09.408 DEBUG:boto:Headers: {} 2023-05-31T19:56:09.408 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T19:56:09.409 DEBUG:boto:Port: 8000 2023-05-31T19:56:09.409 DEBUG:boto:Params: {} 2023-05-31T19:56:09.409 DEBUG:boto:Token: None 2023-05-31T19:56:09.410 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 19:56:09 GMT /pklxkd-7/myobj 2023-05-31T19:56:09.410 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:ULLXdnW7aIOf8X+LaVrUAnL9wjk= 2023-05-31T19:56:09.411 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 19:56:09 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:ULLXdnW7aIOf8X+LaVrUAnL9wjk=', 'Content-Length': '0'} 2023-05-31T19:56:09.412 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 19:52:48 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000004f808ebb44348cd8-006477a659-110e-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 19:56:09 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T19:56:09.413 DEBUG:boto:path=/pklxkd-7/ 2023-05-31T19:56:09.413 DEBUG:boto:auth_path=/pklxkd-7/ 2023-05-31T19:56:09.414 DEBUG:boto:path=/pklxkd-7/?prefix=myobj 2023-05-31T19:56:09.414 DEBUG:boto:auth_path=/pklxkd-7/?prefix=myobj 2023-05-31T19:56:09.414 DEBUG:boto:Method: GET 2023-05-31T19:56:09.415 DEBUG:boto:Path: /pklxkd-7/?prefix=myobj 2023-05-31T19:56:09.415 DEBUG:boto:Data: 2023-05-31T19:56:09.416 DEBUG:boto:Headers: {} 2023-05-31T19:56:09.416 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T19:56:09.416 DEBUG:boto:Port: 8001 2023-05-31T19:56:09.417 DEBUG:boto:Params: {} 2023-05-31T19:56:09.417 DEBUG:boto:Token: None 2023-05-31T19:56:09.417 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 19:56:09 GMT /pklxkd-7/ 2023-05-31T19:56:09.418 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:hTDo8we7N3jPGM/woJctQUnCKbo= 2023-05-31T19:56:09.418 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 19:56:09 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:hTDo8we7N3jPGM/woJctQUnCKbo=', 'Content-Length': '0'} 2023-05-31T19:56:09.420 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ebcc36ae29069c4c-006477a659-1112-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 19:56:09 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T19:56:09.464 DEBUG:boto:b'pklxkd-7myobj1000falsemyobj2023-05-31T19:52:48.073Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T19:56:09.465 DEBUG:boto:path=/pklxkd-7/ 2023-05-31T19:56:09.465 DEBUG:boto:auth_path=/pklxkd-7/ 2023-05-31T19:56:09.466 DEBUG:boto:path=/pklxkd-7/?prefix=myobj 2023-05-31T19:56:09.466 DEBUG:boto:auth_path=/pklxkd-7/?prefix=myobj 2023-05-31T19:56:09.466 DEBUG:boto:Method: GET 2023-05-31T19:56:09.467 DEBUG:boto:Path: /pklxkd-7/?prefix=myobj 2023-05-31T19:56:09.467 DEBUG:boto:Data: 2023-05-31T19:56:09.467 DEBUG:boto:Headers: {} 2023-05-31T19:56:09.468 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T19:56:09.468 DEBUG:boto:Port: 8000 2023-05-31T19:56:09.468 DEBUG:boto:Params: {} 2023-05-31T19:56:09.469 DEBUG:boto:Token: None 2023-05-31T19:56:09.469 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 19:56:09 GMT /pklxkd-7/ 2023-05-31T19:56:09.470 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:hTDo8we7N3jPGM/woJctQUnCKbo= 2023-05-31T19:56:09.470 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 19:56:09 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:hTDo8we7N3jPGM/woJctQUnCKbo=', 'Content-Length': '0'} 2023-05-31T19:56:09.472 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000007ced1c0d71e6ccc9-006477a659-110e-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 19:56:09 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T19:56:09.473 DEBUG:boto:b'pklxkd-7myobj1000falsemyobj2023-05-31T19:52:48.073Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T19:56:09.474 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=pklxkd-7 zones={a2, a1} 2023-05-31T19:56:09.474 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'pklxkd-8', '--source-zone', 'a2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-05-31T19:56:09.474 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'pklxkd-8', '--source-zone', 'a2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-05-31T19:56:09.475 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket pklxkd-8 --source-zone a2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2023-05-31T19:56:09.538 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2023-05-31T19:56:09.539 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2023-05-31T19:56:09.592 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T19:56:09.588+0000 7f7e52e42780 1 bucket sync caught up with source: 2023-05-31T19:56:09.593 INFO:teuthology.orchestra.run.smithi099.stderr: local status: [, 00000000001.13.6, , 00000000001.7.6, , , , 00000000001.17.6, 00000000001.13.6, , ] 2023-05-31T19:56:09.593 INFO:teuthology.orchestra.run.smithi099.stderr: remote markers: [, 00000000001.13.6, , 00000000001.7.6, , , , 00000000001.17.6, 00000000001.13.6, , ] 2023-05-31T19:56:09.593 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T19:56:09.588+0000 7f7e52e42780 0 bucket checkpoint complete 2023-05-31T19:56:09.614 INFO:tasks.rgw_multi.zone_rados:comparing bucket=pklxkd-8 zones={a1, a2} 2023-05-31T19:56:09.615 DEBUG:boto:path=/pklxkd-8/ 2023-05-31T19:56:09.615 DEBUG:boto:auth_path=/pklxkd-8/ 2023-05-31T19:56:09.615 DEBUG:boto:Method: HEAD 2023-05-31T19:56:09.616 DEBUG:boto:Path: /pklxkd-8/ 2023-05-31T19:56:09.616 DEBUG:boto:Data: 2023-05-31T19:56:09.616 DEBUG:boto:Headers: {} 2023-05-31T19:56:09.617 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T19:56:09.617 DEBUG:boto:Port: 8000 2023-05-31T19:56:09.617 DEBUG:boto:Params: {} 2023-05-31T19:56:09.618 DEBUG:boto:Token: None 2023-05-31T19:56:09.618 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 19:56:09 GMT /pklxkd-8/ 2023-05-31T19:56:09.619 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:HRY93UozdY1SZbl36kGjpsswuqI= 2023-05-31T19:56:09.619 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 19:56:09 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:HRY93UozdY1SZbl36kGjpsswuqI=', 'Content-Length': '0'} 2023-05-31T19:56:09.622 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '4'), ('X-RGW-Bytes-Used', '24'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000040310536a3d7adb1-006477a659-110e-a1'), ('Content-Length', '0'), ('Date', 'Wed, 31 May 2023 19:56:09 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T19:56:09.622 DEBUG:boto:path=/pklxkd-8/ 2023-05-31T19:56:09.622 DEBUG:boto:auth_path=/pklxkd-8/ 2023-05-31T19:56:09.623 DEBUG:boto:Method: HEAD 2023-05-31T19:56:09.623 DEBUG:boto:Path: /pklxkd-8/ 2023-05-31T19:56:09.624 DEBUG:boto:Data: 2023-05-31T19:56:09.624 DEBUG:boto:Headers: {} 2023-05-31T19:56:09.624 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T19:56:09.625 DEBUG:boto:Port: 8001 2023-05-31T19:56:09.625 DEBUG:boto:Params: {} 2023-05-31T19:56:09.625 DEBUG:boto:Token: None 2023-05-31T19:56:09.626 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 19:56:09 GMT /pklxkd-8/ 2023-05-31T19:56:09.626 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:HRY93UozdY1SZbl36kGjpsswuqI= 2023-05-31T19:56:09.627 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 19:56:09 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:HRY93UozdY1SZbl36kGjpsswuqI=', 'Content-Length': '0'} 2023-05-31T19:56:09.629 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '4'), ('X-RGW-Bytes-Used', '24'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000b50a7dc83368bc59-006477a659-1112-a2'), ('Content-Length', '0'), ('Date', 'Wed, 31 May 2023 19:56:09 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T19:56:09.629 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2023-05-31T19:56:09.630 DEBUG:boto:path=/pklxkd-8/ 2023-05-31T19:56:09.630 DEBUG:boto:auth_path=/pklxkd-8/ 2023-05-31T19:56:09.630 DEBUG:boto:path=/pklxkd-8/?versions&max-keys=999 2023-05-31T19:56:09.631 DEBUG:boto:auth_path=/pklxkd-8/?versions&max-keys=999 2023-05-31T19:56:09.631 DEBUG:boto:Method: GET 2023-05-31T19:56:09.631 DEBUG:boto:Path: /pklxkd-8/?versions&max-keys=999 2023-05-31T19:56:09.632 DEBUG:boto:Data: 2023-05-31T19:56:09.632 DEBUG:boto:Headers: {} 2023-05-31T19:56:09.632 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T19:56:09.633 DEBUG:boto:Port: 8000 2023-05-31T19:56:09.633 DEBUG:boto:Params: {} 2023-05-31T19:56:09.634 DEBUG:boto:Token: None 2023-05-31T19:56:09.634 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 19:56:09 GMT /pklxkd-8/?versions 2023-05-31T19:56:09.635 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:yV8h+Q5Bo8wsJ5cFUtVsorG+0e4= 2023-05-31T19:56:09.635 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 19:56:09 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:yV8h+Q5Bo8wsJ5cFUtVsorG+0e4=', 'Content-Length': '0'} 2023-05-31T19:56:09.637 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000006fa9a2621522d11b-006477a659-110e-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 19:56:09 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T19:56:09.638 DEBUG:boto:b'pklxkd-8999false&nulltrue2023-05-31T19:52:48.319Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal:nulltrue2023-05-31T19:52:48.263Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal_myobjnulltrue2023-05-31T19:52:48.236Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalmyobjnulltrue2023-05-31T19:52:48.211Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T19:56:09.638 DEBUG:tasks.rgw_multi.zone_rados:o=& 2023-05-31T19:56:09.639 DEBUG:tasks.rgw_multi.zone_rados:o=: 2023-05-31T19:56:09.639 DEBUG:tasks.rgw_multi.zone_rados:o=_myobj 2023-05-31T19:56:09.640 DEBUG:tasks.rgw_multi.zone_rados:o=myobj 2023-05-31T19:56:09.640 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2023-05-31T19:56:09.640 DEBUG:boto:path=/pklxkd-8/ 2023-05-31T19:56:09.641 DEBUG:boto:auth_path=/pklxkd-8/ 2023-05-31T19:56:09.641 DEBUG:boto:path=/pklxkd-8/?versions&max-keys=999 2023-05-31T19:56:09.641 DEBUG:boto:auth_path=/pklxkd-8/?versions&max-keys=999 2023-05-31T19:56:09.642 DEBUG:boto:Method: GET 2023-05-31T19:56:09.642 DEBUG:boto:Path: /pklxkd-8/?versions&max-keys=999 2023-05-31T19:56:09.642 DEBUG:boto:Data: 2023-05-31T19:56:09.643 DEBUG:boto:Headers: {} 2023-05-31T19:56:09.643 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T19:56:09.643 DEBUG:boto:Port: 8001 2023-05-31T19:56:09.644 DEBUG:boto:Params: {} 2023-05-31T19:56:09.644 DEBUG:boto:Token: None 2023-05-31T19:56:09.645 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 19:56:09 GMT /pklxkd-8/?versions 2023-05-31T19:56:09.645 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:yV8h+Q5Bo8wsJ5cFUtVsorG+0e4= 2023-05-31T19:56:09.645 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 19:56:09 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:yV8h+Q5Bo8wsJ5cFUtVsorG+0e4=', 'Content-Length': '0'} 2023-05-31T19:56:09.648 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ec8741e915d70d1a-006477a659-1112-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 19:56:09 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T19:56:09.648 DEBUG:boto:b'pklxkd-8999false&nulltrue2023-05-31T19:52:48.319Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal:nulltrue2023-05-31T19:52:48.263Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal_myobjnulltrue2023-05-31T19:52:48.236Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalmyobjnulltrue2023-05-31T19:52:48.211Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T19:56:09.649 DEBUG:tasks.rgw_multi.zone_rados:o=& 2023-05-31T19:56:09.649 DEBUG:tasks.rgw_multi.zone_rados:o=: 2023-05-31T19:56:09.650 DEBUG:tasks.rgw_multi.zone_rados:o=_myobj 2023-05-31T19:56:09.650 DEBUG:tasks.rgw_multi.zone_rados:o=myobj 2023-05-31T19:56:09.650 DEBUG:boto:path=/pklxkd-8/ 2023-05-31T19:56:09.651 DEBUG:boto:auth_path=/pklxkd-8/ 2023-05-31T19:56:09.651 DEBUG:boto:path=/pklxkd-8/?versions&max-keys=999 2023-05-31T19:56:09.651 DEBUG:boto:auth_path=/pklxkd-8/?versions&max-keys=999 2023-05-31T19:56:09.652 DEBUG:boto:Method: GET 2023-05-31T19:56:09.652 DEBUG:boto:Path: /pklxkd-8/?versions&max-keys=999 2023-05-31T19:56:09.653 DEBUG:boto:Data: 2023-05-31T19:56:09.653 DEBUG:boto:Headers: {} 2023-05-31T19:56:09.653 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T19:56:09.654 DEBUG:boto:Port: 8000 2023-05-31T19:56:09.654 DEBUG:boto:Params: {} 2023-05-31T19:56:09.655 DEBUG:boto:Token: None 2023-05-31T19:56:09.655 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 19:56:09 GMT /pklxkd-8/?versions 2023-05-31T19:56:09.655 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:yV8h+Q5Bo8wsJ5cFUtVsorG+0e4= 2023-05-31T19:56:09.656 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 19:56:09 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:yV8h+Q5Bo8wsJ5cFUtVsorG+0e4=', 'Content-Length': '0'} 2023-05-31T19:56:09.658 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000008a1267c0546b96aa-006477a659-110e-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 19:56:09 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T19:56:09.658 DEBUG:boto:b'pklxkd-8999false&nulltrue2023-05-31T19:52:48.319Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal:nulltrue2023-05-31T19:52:48.263Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal_myobjnulltrue2023-05-31T19:52:48.236Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalmyobjnulltrue2023-05-31T19:52:48.211Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T19:56:09.659 DEBUG:boto:path=/pklxkd-8/ 2023-05-31T19:56:09.659 DEBUG:boto:auth_path=/pklxkd-8/ 2023-05-31T19:56:09.660 DEBUG:boto:path=/pklxkd-8/?versions&max-keys=999 2023-05-31T19:56:09.660 DEBUG:boto:auth_path=/pklxkd-8/?versions&max-keys=999 2023-05-31T19:56:09.661 DEBUG:boto:Method: GET 2023-05-31T19:56:09.661 DEBUG:boto:Path: /pklxkd-8/?versions&max-keys=999 2023-05-31T19:56:09.661 DEBUG:boto:Data: 2023-05-31T19:56:09.662 DEBUG:boto:Headers: {} 2023-05-31T19:56:09.662 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T19:56:09.662 DEBUG:boto:Port: 8001 2023-05-31T19:56:09.663 DEBUG:boto:Params: {} 2023-05-31T19:56:09.663 DEBUG:boto:Token: None 2023-05-31T19:56:09.664 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 19:56:09 GMT /pklxkd-8/?versions 2023-05-31T19:56:09.664 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:yV8h+Q5Bo8wsJ5cFUtVsorG+0e4= 2023-05-31T19:56:09.664 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 19:56:09 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:yV8h+Q5Bo8wsJ5cFUtVsorG+0e4=', 'Content-Length': '0'} 2023-05-31T19:56:09.666 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000098dc4cbfeaf6d7d6-006477a659-1112-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 19:56:09 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T19:56:09.667 DEBUG:boto:b'pklxkd-8999false&nulltrue2023-05-31T19:52:48.319Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal:nulltrue2023-05-31T19:52:48.263Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal_myobjnulltrue2023-05-31T19:52:48.236Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalmyobjnulltrue2023-05-31T19:52:48.211Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T19:56:09.668 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=& 2023-05-31T19:56:09.668 DEBUG:boto:path=/pklxkd-8/%26 2023-05-31T19:56:09.668 DEBUG:boto:auth_path=/pklxkd-8/%26 2023-05-31T19:56:09.669 DEBUG:boto:path=/pklxkd-8/%26?versionId=null 2023-05-31T19:56:09.669 DEBUG:boto:auth_path=/pklxkd-8/%26?versionId=null 2023-05-31T19:56:09.670 DEBUG:boto:Method: GET 2023-05-31T19:56:09.670 DEBUG:boto:Path: /pklxkd-8/%26?versionId=null 2023-05-31T19:56:09.670 DEBUG:boto:Data: 2023-05-31T19:56:09.671 DEBUG:boto:Headers: {} 2023-05-31T19:56:09.671 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T19:56:09.671 DEBUG:boto:Port: 8000 2023-05-31T19:56:09.672 DEBUG:boto:Params: {} 2023-05-31T19:56:09.672 DEBUG:boto:Token: None 2023-05-31T19:56:09.673 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 19:56:09 GMT /pklxkd-8/%26?versionId=null 2023-05-31T19:56:09.673 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:Bwtey6UgtblJjxoaVXf8YApx4AU= 2023-05-31T19:56:09.673 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 19:56:09 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:Bwtey6UgtblJjxoaVXf8YApx4AU=', 'Content-Length': '0'} 2023-05-31T19:56:09.675 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 19:52:48 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000049f79784941bec44-006477a659-110e-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 19:56:09 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T19:56:09.716 DEBUG:boto:path=/pklxkd-8/%26 2023-05-31T19:56:09.717 DEBUG:boto:auth_path=/pklxkd-8/%26 2023-05-31T19:56:09.717 DEBUG:boto:path=/pklxkd-8/%26?versionId=null 2023-05-31T19:56:09.718 DEBUG:boto:auth_path=/pklxkd-8/%26?versionId=null 2023-05-31T19:56:09.718 DEBUG:boto:Method: GET 2023-05-31T19:56:09.718 DEBUG:boto:Path: /pklxkd-8/%26?versionId=null 2023-05-31T19:56:09.719 DEBUG:boto:Data: 2023-05-31T19:56:09.719 DEBUG:boto:Headers: {} 2023-05-31T19:56:09.719 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T19:56:09.720 DEBUG:boto:Port: 8001 2023-05-31T19:56:09.720 DEBUG:boto:Params: {} 2023-05-31T19:56:09.721 DEBUG:boto:Token: None 2023-05-31T19:56:09.721 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 19:56:09 GMT /pklxkd-8/%26?versionId=null 2023-05-31T19:56:09.721 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:Bwtey6UgtblJjxoaVXf8YApx4AU= 2023-05-31T19:56:09.722 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 19:56:09 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:Bwtey6UgtblJjxoaVXf8YApx4AU=', 'Content-Length': '0'} 2023-05-31T19:56:09.723 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 19:52:48 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000085a8442ebd89d594-006477a659-1112-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 19:56:09 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T19:56:09.724 DEBUG:boto:path=/pklxkd-8/%26 2023-05-31T19:56:09.725 DEBUG:boto:auth_path=/pklxkd-8/%26 2023-05-31T19:56:09.725 DEBUG:boto:path=/pklxkd-8/%26?versionId=null 2023-05-31T19:56:09.725 DEBUG:boto:auth_path=/pklxkd-8/%26?versionId=null 2023-05-31T19:56:09.726 DEBUG:boto:Method: HEAD 2023-05-31T19:56:09.726 DEBUG:boto:Path: /pklxkd-8/%26?versionId=null 2023-05-31T19:56:09.726 DEBUG:boto:Data: 2023-05-31T19:56:09.727 DEBUG:boto:Headers: {} 2023-05-31T19:56:09.727 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T19:56:09.727 DEBUG:boto:Port: 8000 2023-05-31T19:56:09.728 DEBUG:boto:Params: {} 2023-05-31T19:56:09.728 DEBUG:boto:Token: None 2023-05-31T19:56:09.728 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 19:56:09 GMT /pklxkd-8/%26?versionId=null 2023-05-31T19:56:09.729 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:8xks63HK67s15pTVhBFOzIbU9NU= 2023-05-31T19:56:09.729 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 19:56:09 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:8xks63HK67s15pTVhBFOzIbU9NU=', 'Content-Length': '0'} 2023-05-31T19:56:09.731 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 19:52:48 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000000cd1e7511e0b9f36-006477a659-110e-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 19:56:09 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T19:56:09.732 DEBUG:boto:path=/pklxkd-8/%26 2023-05-31T19:56:09.732 DEBUG:boto:auth_path=/pklxkd-8/%26 2023-05-31T19:56:09.732 DEBUG:boto:path=/pklxkd-8/%26?versionId=null 2023-05-31T19:56:09.733 DEBUG:boto:auth_path=/pklxkd-8/%26?versionId=null 2023-05-31T19:56:09.733 DEBUG:boto:Method: HEAD 2023-05-31T19:56:09.733 DEBUG:boto:Path: /pklxkd-8/%26?versionId=null 2023-05-31T19:56:09.734 DEBUG:boto:Data: 2023-05-31T19:56:09.734 DEBUG:boto:Headers: {} 2023-05-31T19:56:09.734 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T19:56:09.735 DEBUG:boto:Port: 8001 2023-05-31T19:56:09.735 DEBUG:boto:Params: {} 2023-05-31T19:56:09.735 DEBUG:boto:Token: None 2023-05-31T19:56:09.736 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 19:56:09 GMT /pklxkd-8/%26?versionId=null 2023-05-31T19:56:09.736 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:8xks63HK67s15pTVhBFOzIbU9NU= 2023-05-31T19:56:09.737 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 19:56:09 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:8xks63HK67s15pTVhBFOzIbU9NU=', 'Content-Length': '0'} 2023-05-31T19:56:09.738 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 19:52:48 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000009d310fc20b418281-006477a659-1112-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 19:56:09 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T19:56:09.739 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=& 2023-05-31T19:56:09.739 DEBUG:boto:path=/pklxkd-8/%26 2023-05-31T19:56:09.740 DEBUG:boto:auth_path=/pklxkd-8/%26 2023-05-31T19:56:09.740 DEBUG:boto:path=/pklxkd-8/%26?versionId=null 2023-05-31T19:56:09.740 DEBUG:boto:auth_path=/pklxkd-8/%26?versionId=null 2023-05-31T19:56:09.741 DEBUG:boto:Method: GET 2023-05-31T19:56:09.741 DEBUG:boto:Path: /pklxkd-8/%26?versionId=null 2023-05-31T19:56:09.741 DEBUG:boto:Data: 2023-05-31T19:56:09.742 DEBUG:boto:Headers: {} 2023-05-31T19:56:09.742 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T19:56:09.742 DEBUG:boto:Port: 8000 2023-05-31T19:56:09.743 DEBUG:boto:Params: {} 2023-05-31T19:56:09.743 DEBUG:boto:Token: None 2023-05-31T19:56:09.744 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 19:56:09 GMT /pklxkd-8/%26?versionId=null 2023-05-31T19:56:09.744 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:Bwtey6UgtblJjxoaVXf8YApx4AU= 2023-05-31T19:56:09.745 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 19:56:09 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:Bwtey6UgtblJjxoaVXf8YApx4AU=', 'Content-Length': '0'} 2023-05-31T19:56:09.746 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 19:52:48 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000007c145a63c825146c-006477a659-110e-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 19:56:09 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T19:56:09.747 DEBUG:boto:path=/pklxkd-8/%26 2023-05-31T19:56:09.747 DEBUG:boto:auth_path=/pklxkd-8/%26 2023-05-31T19:56:09.748 DEBUG:boto:path=/pklxkd-8/%26?versionId=null 2023-05-31T19:56:09.748 DEBUG:boto:auth_path=/pklxkd-8/%26?versionId=null 2023-05-31T19:56:09.748 DEBUG:boto:Method: GET 2023-05-31T19:56:09.749 DEBUG:boto:Path: /pklxkd-8/%26?versionId=null 2023-05-31T19:56:09.749 DEBUG:boto:Data: 2023-05-31T19:56:09.749 DEBUG:boto:Headers: {} 2023-05-31T19:56:09.750 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T19:56:09.750 DEBUG:boto:Port: 8001 2023-05-31T19:56:09.750 DEBUG:boto:Params: {} 2023-05-31T19:56:09.751 DEBUG:boto:Token: None 2023-05-31T19:56:09.751 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 19:56:09 GMT /pklxkd-8/%26?versionId=null 2023-05-31T19:56:09.752 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:Bwtey6UgtblJjxoaVXf8YApx4AU= 2023-05-31T19:56:09.752 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 19:56:09 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:Bwtey6UgtblJjxoaVXf8YApx4AU=', 'Content-Length': '0'} 2023-05-31T19:56:09.754 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 19:52:48 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000e30f29e8e4d7d1be-006477a659-1112-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 19:56:09 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T19:56:09.755 DEBUG:boto:path=/pklxkd-8/ 2023-05-31T19:56:09.755 DEBUG:boto:auth_path=/pklxkd-8/ 2023-05-31T19:56:09.755 DEBUG:boto:path=/pklxkd-8/?prefix=%26 2023-05-31T19:56:09.756 DEBUG:boto:auth_path=/pklxkd-8/?prefix=%26 2023-05-31T19:56:09.756 DEBUG:boto:Method: GET 2023-05-31T19:56:09.756 DEBUG:boto:Path: /pklxkd-8/?prefix=%26 2023-05-31T19:56:09.757 DEBUG:boto:Data: 2023-05-31T19:56:09.757 DEBUG:boto:Headers: {} 2023-05-31T19:56:09.757 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T19:56:09.758 DEBUG:boto:Port: 8000 2023-05-31T19:56:09.758 DEBUG:boto:Params: {} 2023-05-31T19:56:09.759 DEBUG:boto:Token: None 2023-05-31T19:56:09.759 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 19:56:09 GMT /pklxkd-8/ 2023-05-31T19:56:09.759 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:cuApYHAkLLChEuzBtxyexdpBnqA= 2023-05-31T19:56:09.760 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 19:56:09 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:cuApYHAkLLChEuzBtxyexdpBnqA=', 'Content-Length': '0'} 2023-05-31T19:56:09.762 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d16b3b8068755bf1-006477a659-110e-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 19:56:09 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T19:56:09.762 DEBUG:boto:b'pklxkd-8&1000false&2023-05-31T19:52:48.319Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T19:56:09.763 DEBUG:boto:path=/pklxkd-8/ 2023-05-31T19:56:09.764 DEBUG:boto:auth_path=/pklxkd-8/ 2023-05-31T19:56:09.764 DEBUG:boto:path=/pklxkd-8/?prefix=%26 2023-05-31T19:56:09.765 DEBUG:boto:auth_path=/pklxkd-8/?prefix=%26 2023-05-31T19:56:09.765 DEBUG:boto:Method: GET 2023-05-31T19:56:09.766 DEBUG:boto:Path: /pklxkd-8/?prefix=%26 2023-05-31T19:56:09.766 DEBUG:boto:Data: 2023-05-31T19:56:09.766 DEBUG:boto:Headers: {} 2023-05-31T19:56:09.767 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T19:56:09.767 DEBUG:boto:Port: 8001 2023-05-31T19:56:09.768 DEBUG:boto:Params: {} 2023-05-31T19:56:09.768 DEBUG:boto:Token: None 2023-05-31T19:56:09.769 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 19:56:09 GMT /pklxkd-8/ 2023-05-31T19:56:09.769 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:cuApYHAkLLChEuzBtxyexdpBnqA= 2023-05-31T19:56:09.770 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 19:56:09 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:cuApYHAkLLChEuzBtxyexdpBnqA=', 'Content-Length': '0'} 2023-05-31T19:56:09.772 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b0e90125def389a3-006477a659-1112-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 19:56:09 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T19:56:09.773 DEBUG:boto:b'pklxkd-8&1000false&2023-05-31T19:52:48.319Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T19:56:09.773 DEBUG:boto:path=/pklxkd-8/%26 2023-05-31T19:56:09.774 DEBUG:boto:auth_path=/pklxkd-8/%26 2023-05-31T19:56:09.774 DEBUG:boto:Method: HEAD 2023-05-31T19:56:09.774 DEBUG:boto:Path: /pklxkd-8/%26 2023-05-31T19:56:09.775 DEBUG:boto:Data: 2023-05-31T19:56:09.775 DEBUG:boto:Headers: {} 2023-05-31T19:56:09.775 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T19:56:09.776 DEBUG:boto:Port: 8000 2023-05-31T19:56:09.776 DEBUG:boto:Params: {} 2023-05-31T19:56:09.777 DEBUG:boto:Token: None 2023-05-31T19:56:09.777 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 19:56:09 GMT /pklxkd-8/%26 2023-05-31T19:56:09.777 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:MHOEcj69NKIkc5GmVBsGDU16mw8= 2023-05-31T19:56:09.778 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 19:56:09 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:MHOEcj69NKIkc5GmVBsGDU16mw8=', 'Content-Length': '0'} 2023-05-31T19:56:09.779 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 19:52:48 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000003e628712739a4622-006477a659-110e-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 19:56:09 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T19:56:09.780 DEBUG:boto:path=/pklxkd-8/%26 2023-05-31T19:56:09.780 DEBUG:boto:auth_path=/pklxkd-8/%26 2023-05-31T19:56:09.781 DEBUG:boto:Method: HEAD 2023-05-31T19:56:09.781 DEBUG:boto:Path: /pklxkd-8/%26 2023-05-31T19:56:09.781 DEBUG:boto:Data: 2023-05-31T19:56:09.782 DEBUG:boto:Headers: {} 2023-05-31T19:56:09.782 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T19:56:09.782 DEBUG:boto:Port: 8001 2023-05-31T19:56:09.783 DEBUG:boto:Params: {} 2023-05-31T19:56:09.783 DEBUG:boto:Token: None 2023-05-31T19:56:09.784 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 19:56:09 GMT /pklxkd-8/%26 2023-05-31T19:56:09.784 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:MHOEcj69NKIkc5GmVBsGDU16mw8= 2023-05-31T19:56:09.784 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 19:56:09 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:MHOEcj69NKIkc5GmVBsGDU16mw8=', 'Content-Length': '0'} 2023-05-31T19:56:09.786 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 19:52:48 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000639c96c1ec6a46d9-006477a659-1112-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 19:56:09 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T19:56:09.786 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=& 2023-05-31T19:56:09.787 DEBUG:boto:path=/pklxkd-8/%26 2023-05-31T19:56:09.787 DEBUG:boto:auth_path=/pklxkd-8/%26 2023-05-31T19:56:09.787 DEBUG:boto:Method: GET 2023-05-31T19:56:09.788 DEBUG:boto:Path: /pklxkd-8/%26 2023-05-31T19:56:09.788 DEBUG:boto:Data: 2023-05-31T19:56:09.789 DEBUG:boto:Headers: {} 2023-05-31T19:56:09.789 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T19:56:09.789 DEBUG:boto:Port: 8000 2023-05-31T19:56:09.790 DEBUG:boto:Params: {} 2023-05-31T19:56:09.790 DEBUG:boto:Token: None 2023-05-31T19:56:09.791 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 19:56:09 GMT /pklxkd-8/%26 2023-05-31T19:56:09.791 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:p37f5Y7RfN2kcYIkxOslJ2CorEM= 2023-05-31T19:56:09.791 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 19:56:09 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:p37f5Y7RfN2kcYIkxOslJ2CorEM=', 'Content-Length': '0'} 2023-05-31T19:56:09.793 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 19:52:48 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000002a478bf1d54bb6f8-006477a659-110e-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 19:56:09 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T19:56:09.794 DEBUG:boto:path=/pklxkd-8/%26 2023-05-31T19:56:09.794 DEBUG:boto:auth_path=/pklxkd-8/%26 2023-05-31T19:56:09.795 DEBUG:boto:Method: GET 2023-05-31T19:56:09.795 DEBUG:boto:Path: /pklxkd-8/%26 2023-05-31T19:56:09.795 DEBUG:boto:Data: 2023-05-31T19:56:09.796 DEBUG:boto:Headers: {} 2023-05-31T19:56:09.796 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T19:56:09.797 DEBUG:boto:Port: 8001 2023-05-31T19:56:09.797 DEBUG:boto:Params: {} 2023-05-31T19:56:09.797 DEBUG:boto:Token: None 2023-05-31T19:56:09.798 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 19:56:09 GMT /pklxkd-8/%26 2023-05-31T19:56:09.798 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:p37f5Y7RfN2kcYIkxOslJ2CorEM= 2023-05-31T19:56:09.799 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 19:56:09 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:p37f5Y7RfN2kcYIkxOslJ2CorEM=', 'Content-Length': '0'} 2023-05-31T19:56:09.800 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 19:52:48 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000087a0cff8feeadfa3-006477a659-1112-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 19:56:09 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T19:56:09.801 DEBUG:boto:path=/pklxkd-8/ 2023-05-31T19:56:09.801 DEBUG:boto:auth_path=/pklxkd-8/ 2023-05-31T19:56:09.802 DEBUG:boto:path=/pklxkd-8/?prefix=%26 2023-05-31T19:56:09.802 DEBUG:boto:auth_path=/pklxkd-8/?prefix=%26 2023-05-31T19:56:09.802 DEBUG:boto:Method: GET 2023-05-31T19:56:09.803 DEBUG:boto:Path: /pklxkd-8/?prefix=%26 2023-05-31T19:56:09.803 DEBUG:boto:Data: 2023-05-31T19:56:09.803 DEBUG:boto:Headers: {} 2023-05-31T19:56:09.804 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T19:56:09.804 DEBUG:boto:Port: 8000 2023-05-31T19:56:09.804 DEBUG:boto:Params: {} 2023-05-31T19:56:09.805 DEBUG:boto:Token: None 2023-05-31T19:56:09.805 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 19:56:09 GMT /pklxkd-8/ 2023-05-31T19:56:09.806 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:cuApYHAkLLChEuzBtxyexdpBnqA= 2023-05-31T19:56:09.806 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 19:56:09 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:cuApYHAkLLChEuzBtxyexdpBnqA=', 'Content-Length': '0'} 2023-05-31T19:56:09.808 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000093e54049102c5358-006477a659-110e-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 19:56:09 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T19:56:09.809 DEBUG:boto:b'pklxkd-8&1000false&2023-05-31T19:52:48.319Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T19:56:09.810 DEBUG:boto:path=/pklxkd-8/ 2023-05-31T19:56:09.810 DEBUG:boto:auth_path=/pklxkd-8/ 2023-05-31T19:56:09.811 DEBUG:boto:path=/pklxkd-8/?prefix=%26 2023-05-31T19:56:09.811 DEBUG:boto:auth_path=/pklxkd-8/?prefix=%26 2023-05-31T19:56:09.811 DEBUG:boto:Method: GET 2023-05-31T19:56:09.812 DEBUG:boto:Path: /pklxkd-8/?prefix=%26 2023-05-31T19:56:09.812 DEBUG:boto:Data: 2023-05-31T19:56:09.813 DEBUG:boto:Headers: {} 2023-05-31T19:56:09.813 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T19:56:09.813 DEBUG:boto:Port: 8001 2023-05-31T19:56:09.814 DEBUG:boto:Params: {} 2023-05-31T19:56:09.814 DEBUG:boto:Token: None 2023-05-31T19:56:09.814 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 19:56:09 GMT /pklxkd-8/ 2023-05-31T19:56:09.815 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:cuApYHAkLLChEuzBtxyexdpBnqA= 2023-05-31T19:56:09.815 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 19:56:09 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:cuApYHAkLLChEuzBtxyexdpBnqA=', 'Content-Length': '0'} 2023-05-31T19:56:09.818 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b6881e7aeaf488f9-006477a659-1112-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 19:56:09 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T19:56:09.818 DEBUG:boto:b'pklxkd-8&1000false&2023-05-31T19:52:48.319Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T19:56:09.819 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=: 2023-05-31T19:56:09.819 DEBUG:boto:path=/pklxkd-8/%3A 2023-05-31T19:56:09.820 DEBUG:boto:auth_path=/pklxkd-8/%3A 2023-05-31T19:56:09.820 DEBUG:boto:path=/pklxkd-8/%3A?versionId=null 2023-05-31T19:56:09.820 DEBUG:boto:auth_path=/pklxkd-8/%3A?versionId=null 2023-05-31T19:56:09.821 DEBUG:boto:Method: GET 2023-05-31T19:56:09.821 DEBUG:boto:Path: /pklxkd-8/%3A?versionId=null 2023-05-31T19:56:09.821 DEBUG:boto:Data: 2023-05-31T19:56:09.822 DEBUG:boto:Headers: {} 2023-05-31T19:56:09.822 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T19:56:09.823 DEBUG:boto:Port: 8000 2023-05-31T19:56:09.823 DEBUG:boto:Params: {} 2023-05-31T19:56:09.823 DEBUG:boto:Token: None 2023-05-31T19:56:09.824 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 19:56:09 GMT /pklxkd-8/%3A?versionId=null 2023-05-31T19:56:09.824 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:J+SsgnmkRDC/mEuOwdBatuuEb5s= 2023-05-31T19:56:09.825 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 19:56:09 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:J+SsgnmkRDC/mEuOwdBatuuEb5s=', 'Content-Length': '0'} 2023-05-31T19:56:09.826 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 19:52:48 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000000dfc5dd42ef8b6d-006477a659-110e-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 19:56:09 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T19:56:09.827 DEBUG:boto:path=/pklxkd-8/%3A 2023-05-31T19:56:09.827 DEBUG:boto:auth_path=/pklxkd-8/%3A 2023-05-31T19:56:09.828 DEBUG:boto:path=/pklxkd-8/%3A?versionId=null 2023-05-31T19:56:09.828 DEBUG:boto:auth_path=/pklxkd-8/%3A?versionId=null 2023-05-31T19:56:09.828 DEBUG:boto:Method: GET 2023-05-31T19:56:09.829 DEBUG:boto:Path: /pklxkd-8/%3A?versionId=null 2023-05-31T19:56:09.829 DEBUG:boto:Data: 2023-05-31T19:56:09.830 DEBUG:boto:Headers: {} 2023-05-31T19:56:09.830 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T19:56:09.830 DEBUG:boto:Port: 8001 2023-05-31T19:56:09.831 DEBUG:boto:Params: {} 2023-05-31T19:56:09.831 DEBUG:boto:Token: None 2023-05-31T19:56:09.831 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 19:56:09 GMT /pklxkd-8/%3A?versionId=null 2023-05-31T19:56:09.832 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:J+SsgnmkRDC/mEuOwdBatuuEb5s= 2023-05-31T19:56:09.832 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 19:56:09 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:J+SsgnmkRDC/mEuOwdBatuuEb5s=', 'Content-Length': '0'} 2023-05-31T19:56:09.834 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 19:52:48 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000005acc35fe3eaa3acc-006477a659-1112-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 19:56:09 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T19:56:09.876 DEBUG:boto:path=/pklxkd-8/%3A 2023-05-31T19:56:09.877 DEBUG:boto:auth_path=/pklxkd-8/%3A 2023-05-31T19:56:09.877 DEBUG:boto:path=/pklxkd-8/%3A?versionId=null 2023-05-31T19:56:09.877 DEBUG:boto:auth_path=/pklxkd-8/%3A?versionId=null 2023-05-31T19:56:09.878 DEBUG:boto:Method: HEAD 2023-05-31T19:56:09.878 DEBUG:boto:Path: /pklxkd-8/%3A?versionId=null 2023-05-31T19:56:09.878 DEBUG:boto:Data: 2023-05-31T19:56:09.879 DEBUG:boto:Headers: {} 2023-05-31T19:56:09.879 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T19:56:09.879 DEBUG:boto:Port: 8000 2023-05-31T19:56:09.880 DEBUG:boto:Params: {} 2023-05-31T19:56:09.880 DEBUG:boto:Token: None 2023-05-31T19:56:09.881 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 19:56:09 GMT /pklxkd-8/%3A?versionId=null 2023-05-31T19:56:09.881 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:BsLHW3QlvaPaK5dge2lLKJPb5B4= 2023-05-31T19:56:09.881 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 19:56:09 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:BsLHW3QlvaPaK5dge2lLKJPb5B4=', 'Content-Length': '0'} 2023-05-31T19:56:09.883 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 19:52:48 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000003a48815b7327f607-006477a659-110e-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 19:56:09 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T19:56:09.884 DEBUG:boto:path=/pklxkd-8/%3A 2023-05-31T19:56:09.884 DEBUG:boto:auth_path=/pklxkd-8/%3A 2023-05-31T19:56:09.884 DEBUG:boto:path=/pklxkd-8/%3A?versionId=null 2023-05-31T19:56:09.885 DEBUG:boto:auth_path=/pklxkd-8/%3A?versionId=null 2023-05-31T19:56:09.885 DEBUG:boto:Method: HEAD 2023-05-31T19:56:09.885 DEBUG:boto:Path: /pklxkd-8/%3A?versionId=null 2023-05-31T19:56:09.886 DEBUG:boto:Data: 2023-05-31T19:56:09.886 DEBUG:boto:Headers: {} 2023-05-31T19:56:09.886 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T19:56:09.887 DEBUG:boto:Port: 8001 2023-05-31T19:56:09.887 DEBUG:boto:Params: {} 2023-05-31T19:56:09.887 DEBUG:boto:Token: None 2023-05-31T19:56:09.888 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 19:56:09 GMT /pklxkd-8/%3A?versionId=null 2023-05-31T19:56:09.888 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:BsLHW3QlvaPaK5dge2lLKJPb5B4= 2023-05-31T19:56:09.889 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 19:56:09 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:BsLHW3QlvaPaK5dge2lLKJPb5B4=', 'Content-Length': '0'} 2023-05-31T19:56:09.890 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 19:52:48 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000007373c6a41e1f1612-006477a659-1112-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 19:56:09 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T19:56:09.890 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=: 2023-05-31T19:56:09.891 DEBUG:boto:path=/pklxkd-8/%3A 2023-05-31T19:56:09.891 DEBUG:boto:auth_path=/pklxkd-8/%3A 2023-05-31T19:56:09.892 DEBUG:boto:path=/pklxkd-8/%3A?versionId=null 2023-05-31T19:56:09.892 DEBUG:boto:auth_path=/pklxkd-8/%3A?versionId=null 2023-05-31T19:56:09.892 DEBUG:boto:Method: GET 2023-05-31T19:56:09.893 DEBUG:boto:Path: /pklxkd-8/%3A?versionId=null 2023-05-31T19:56:09.893 DEBUG:boto:Data: 2023-05-31T19:56:09.893 DEBUG:boto:Headers: {} 2023-05-31T19:56:09.894 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T19:56:09.894 DEBUG:boto:Port: 8000 2023-05-31T19:56:09.895 DEBUG:boto:Params: {} 2023-05-31T19:56:09.895 DEBUG:boto:Token: None 2023-05-31T19:56:09.895 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 19:56:09 GMT /pklxkd-8/%3A?versionId=null 2023-05-31T19:56:09.896 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:J+SsgnmkRDC/mEuOwdBatuuEb5s= 2023-05-31T19:56:09.896 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 19:56:09 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:J+SsgnmkRDC/mEuOwdBatuuEb5s=', 'Content-Length': '0'} 2023-05-31T19:56:09.898 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 19:52:48 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000725abe52fda13a88-006477a659-110e-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 19:56:09 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T19:56:09.898 DEBUG:boto:path=/pklxkd-8/%3A 2023-05-31T19:56:09.899 DEBUG:boto:auth_path=/pklxkd-8/%3A 2023-05-31T19:56:09.899 DEBUG:boto:path=/pklxkd-8/%3A?versionId=null 2023-05-31T19:56:09.899 DEBUG:boto:auth_path=/pklxkd-8/%3A?versionId=null 2023-05-31T19:56:09.900 DEBUG:boto:Method: GET 2023-05-31T19:56:09.900 DEBUG:boto:Path: /pklxkd-8/%3A?versionId=null 2023-05-31T19:56:09.901 DEBUG:boto:Data: 2023-05-31T19:56:09.901 DEBUG:boto:Headers: {} 2023-05-31T19:56:09.901 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T19:56:09.902 DEBUG:boto:Port: 8001 2023-05-31T19:56:09.902 DEBUG:boto:Params: {} 2023-05-31T19:56:09.902 DEBUG:boto:Token: None 2023-05-31T19:56:09.903 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 19:56:09 GMT /pklxkd-8/%3A?versionId=null 2023-05-31T19:56:09.903 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:J+SsgnmkRDC/mEuOwdBatuuEb5s= 2023-05-31T19:56:09.904 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 19:56:09 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:J+SsgnmkRDC/mEuOwdBatuuEb5s=', 'Content-Length': '0'} 2023-05-31T19:56:09.905 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 19:52:48 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000f4ca530d4affb7a1-006477a659-1112-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 19:56:09 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T19:56:09.906 DEBUG:boto:path=/pklxkd-8/ 2023-05-31T19:56:09.906 DEBUG:boto:auth_path=/pklxkd-8/ 2023-05-31T19:56:09.907 DEBUG:boto:path=/pklxkd-8/?prefix=%3A 2023-05-31T19:56:09.907 DEBUG:boto:auth_path=/pklxkd-8/?prefix=%3A 2023-05-31T19:56:09.907 DEBUG:boto:Method: GET 2023-05-31T19:56:09.908 DEBUG:boto:Path: /pklxkd-8/?prefix=%3A 2023-05-31T19:56:09.908 DEBUG:boto:Data: 2023-05-31T19:56:09.908 DEBUG:boto:Headers: {} 2023-05-31T19:56:09.909 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T19:56:09.909 DEBUG:boto:Port: 8000 2023-05-31T19:56:09.909 DEBUG:boto:Params: {} 2023-05-31T19:56:09.910 DEBUG:boto:Token: None 2023-05-31T19:56:09.910 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 19:56:09 GMT /pklxkd-8/ 2023-05-31T19:56:09.911 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:cuApYHAkLLChEuzBtxyexdpBnqA= 2023-05-31T19:56:09.911 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 19:56:09 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:cuApYHAkLLChEuzBtxyexdpBnqA=', 'Content-Length': '0'} 2023-05-31T19:56:09.913 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000bf2bb5952090ceff-006477a659-110e-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 19:56:09 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T19:56:09.914 DEBUG:boto:b'pklxkd-8:1000false:2023-05-31T19:52:48.263Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T19:56:09.914 DEBUG:boto:path=/pklxkd-8/ 2023-05-31T19:56:09.915 DEBUG:boto:auth_path=/pklxkd-8/ 2023-05-31T19:56:09.915 DEBUG:boto:path=/pklxkd-8/?prefix=%3A 2023-05-31T19:56:09.915 DEBUG:boto:auth_path=/pklxkd-8/?prefix=%3A 2023-05-31T19:56:09.916 DEBUG:boto:Method: GET 2023-05-31T19:56:09.916 DEBUG:boto:Path: /pklxkd-8/?prefix=%3A 2023-05-31T19:56:09.916 DEBUG:boto:Data: 2023-05-31T19:56:09.917 DEBUG:boto:Headers: {} 2023-05-31T19:56:09.917 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T19:56:09.918 DEBUG:boto:Port: 8001 2023-05-31T19:56:09.918 DEBUG:boto:Params: {} 2023-05-31T19:56:09.918 DEBUG:boto:Token: None 2023-05-31T19:56:09.919 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 19:56:09 GMT /pklxkd-8/ 2023-05-31T19:56:09.919 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:cuApYHAkLLChEuzBtxyexdpBnqA= 2023-05-31T19:56:09.919 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 19:56:09 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:cuApYHAkLLChEuzBtxyexdpBnqA=', 'Content-Length': '0'} 2023-05-31T19:56:09.921 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000febd1a9a674a9797-006477a659-1112-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 19:56:09 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T19:56:09.922 DEBUG:boto:b'pklxkd-8:1000false:2023-05-31T19:52:48.263Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T19:56:09.923 DEBUG:boto:path=/pklxkd-8/%3A 2023-05-31T19:56:09.923 DEBUG:boto:auth_path=/pklxkd-8/%3A 2023-05-31T19:56:09.923 DEBUG:boto:Method: HEAD 2023-05-31T19:56:09.924 DEBUG:boto:Path: /pklxkd-8/%3A 2023-05-31T19:56:09.924 DEBUG:boto:Data: 2023-05-31T19:56:09.924 DEBUG:boto:Headers: {} 2023-05-31T19:56:09.925 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T19:56:09.925 DEBUG:boto:Port: 8000 2023-05-31T19:56:09.925 DEBUG:boto:Params: {} 2023-05-31T19:56:09.926 DEBUG:boto:Token: None 2023-05-31T19:56:09.926 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 19:56:09 GMT /pklxkd-8/%3A 2023-05-31T19:56:09.927 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:+cecQPutZOzQqT/BOMwXLLyXjYQ= 2023-05-31T19:56:09.927 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 19:56:09 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:+cecQPutZOzQqT/BOMwXLLyXjYQ=', 'Content-Length': '0'} 2023-05-31T19:56:09.928 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 19:52:48 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000032648af7842d8907-006477a659-110e-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 19:56:09 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T19:56:09.929 DEBUG:boto:path=/pklxkd-8/%3A 2023-05-31T19:56:09.929 DEBUG:boto:auth_path=/pklxkd-8/%3A 2023-05-31T19:56:09.930 DEBUG:boto:Method: HEAD 2023-05-31T19:56:09.930 DEBUG:boto:Path: /pklxkd-8/%3A 2023-05-31T19:56:09.930 DEBUG:boto:Data: 2023-05-31T19:56:09.931 DEBUG:boto:Headers: {} 2023-05-31T19:56:09.931 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T19:56:09.931 DEBUG:boto:Port: 8001 2023-05-31T19:56:09.931 DEBUG:boto:Params: {} 2023-05-31T19:56:09.932 DEBUG:boto:Token: None 2023-05-31T19:56:09.932 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 19:56:09 GMT /pklxkd-8/%3A 2023-05-31T19:56:09.933 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:+cecQPutZOzQqT/BOMwXLLyXjYQ= 2023-05-31T19:56:09.933 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 19:56:09 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:+cecQPutZOzQqT/BOMwXLLyXjYQ=', 'Content-Length': '0'} 2023-05-31T19:56:09.934 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 19:52:48 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000009674c8e7c09330ef-006477a659-1112-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 19:56:09 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T19:56:09.935 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=: 2023-05-31T19:56:09.935 DEBUG:boto:path=/pklxkd-8/%3A 2023-05-31T19:56:09.936 DEBUG:boto:auth_path=/pklxkd-8/%3A 2023-05-31T19:56:09.936 DEBUG:boto:Method: GET 2023-05-31T19:56:09.936 DEBUG:boto:Path: /pklxkd-8/%3A 2023-05-31T19:56:09.937 DEBUG:boto:Data: 2023-05-31T19:56:09.937 DEBUG:boto:Headers: {} 2023-05-31T19:56:09.938 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T19:56:09.938 DEBUG:boto:Port: 8000 2023-05-31T19:56:09.938 DEBUG:boto:Params: {} 2023-05-31T19:56:09.939 DEBUG:boto:Token: None 2023-05-31T19:56:09.939 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 19:56:09 GMT /pklxkd-8/%3A 2023-05-31T19:56:09.939 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:6PeNpE5RjJHpr/w+X2iyIgrJzIw= 2023-05-31T19:56:09.940 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 19:56:09 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:6PeNpE5RjJHpr/w+X2iyIgrJzIw=', 'Content-Length': '0'} 2023-05-31T19:56:09.942 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 19:52:48 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000003bccf92e2cc39394-006477a659-110e-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 19:56:09 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T19:56:09.984 DEBUG:boto:path=/pklxkd-8/%3A 2023-05-31T19:56:09.985 DEBUG:boto:auth_path=/pklxkd-8/%3A 2023-05-31T19:56:09.985 DEBUG:boto:Method: GET 2023-05-31T19:56:09.985 DEBUG:boto:Path: /pklxkd-8/%3A 2023-05-31T19:56:09.986 DEBUG:boto:Data: 2023-05-31T19:56:09.986 DEBUG:boto:Headers: {} 2023-05-31T19:56:09.986 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T19:56:09.987 DEBUG:boto:Port: 8001 2023-05-31T19:56:09.987 DEBUG:boto:Params: {} 2023-05-31T19:56:09.988 DEBUG:boto:Token: None 2023-05-31T19:56:09.988 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 19:56:09 GMT /pklxkd-8/%3A 2023-05-31T19:56:09.988 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:6PeNpE5RjJHpr/w+X2iyIgrJzIw= 2023-05-31T19:56:09.989 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 19:56:09 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:6PeNpE5RjJHpr/w+X2iyIgrJzIw=', 'Content-Length': '0'} 2023-05-31T19:56:09.990 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 19:52:48 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000d04b233828c13c54-006477a659-1112-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 19:56:09 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T19:56:09.991 DEBUG:boto:path=/pklxkd-8/ 2023-05-31T19:56:09.991 DEBUG:boto:auth_path=/pklxkd-8/ 2023-05-31T19:56:09.992 DEBUG:boto:path=/pklxkd-8/?prefix=%3A 2023-05-31T19:56:09.992 DEBUG:boto:auth_path=/pklxkd-8/?prefix=%3A 2023-05-31T19:56:09.993 DEBUG:boto:Method: GET 2023-05-31T19:56:09.993 DEBUG:boto:Path: /pklxkd-8/?prefix=%3A 2023-05-31T19:56:09.993 DEBUG:boto:Data: 2023-05-31T19:56:09.994 DEBUG:boto:Headers: {} 2023-05-31T19:56:09.994 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T19:56:09.994 DEBUG:boto:Port: 8000 2023-05-31T19:56:09.995 DEBUG:boto:Params: {} 2023-05-31T19:56:09.995 DEBUG:boto:Token: None 2023-05-31T19:56:09.995 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 19:56:09 GMT /pklxkd-8/ 2023-05-31T19:56:09.996 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:cuApYHAkLLChEuzBtxyexdpBnqA= 2023-05-31T19:56:09.996 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 19:56:09 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:cuApYHAkLLChEuzBtxyexdpBnqA=', 'Content-Length': '0'} 2023-05-31T19:56:09.998 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000008fb202e7a0f3ac5a-006477a659-110e-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 19:56:09 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T19:56:09.999 DEBUG:boto:b'pklxkd-8:1000false:2023-05-31T19:52:48.263Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T19:56:09.999 DEBUG:boto:path=/pklxkd-8/ 2023-05-31T19:56:10.000 DEBUG:boto:auth_path=/pklxkd-8/ 2023-05-31T19:56:10.000 DEBUG:boto:path=/pklxkd-8/?prefix=%3A 2023-05-31T19:56:10.001 DEBUG:boto:auth_path=/pklxkd-8/?prefix=%3A 2023-05-31T19:56:10.001 DEBUG:boto:Method: GET 2023-05-31T19:56:10.001 DEBUG:boto:Path: /pklxkd-8/?prefix=%3A 2023-05-31T19:56:10.002 DEBUG:boto:Data: 2023-05-31T19:56:10.002 DEBUG:boto:Headers: {} 2023-05-31T19:56:10.002 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T19:56:10.003 DEBUG:boto:Port: 8001 2023-05-31T19:56:10.003 DEBUG:boto:Params: {} 2023-05-31T19:56:10.003 DEBUG:boto:Token: None 2023-05-31T19:56:10.004 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 19:56:10 GMT /pklxkd-8/ 2023-05-31T19:56:10.004 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:SPU0kq56jF2a9Iff2IEhhNichhE= 2023-05-31T19:56:10.005 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 19:56:10 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:SPU0kq56jF2a9Iff2IEhhNichhE=', 'Content-Length': '0'} 2023-05-31T19:56:10.007 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000009e910e22102d324-006477a65a-1112-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 19:56:10 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T19:56:10.008 DEBUG:boto:b'pklxkd-8:1000false:2023-05-31T19:52:48.263Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T19:56:10.008 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=_myobj 2023-05-31T19:56:10.009 DEBUG:boto:path=/pklxkd-8/_myobj 2023-05-31T19:56:10.009 DEBUG:boto:auth_path=/pklxkd-8/_myobj 2023-05-31T19:56:10.009 DEBUG:boto:path=/pklxkd-8/_myobj?versionId=null 2023-05-31T19:56:10.010 DEBUG:boto:auth_path=/pklxkd-8/_myobj?versionId=null 2023-05-31T19:56:10.010 DEBUG:boto:Method: GET 2023-05-31T19:56:10.010 DEBUG:boto:Path: /pklxkd-8/_myobj?versionId=null 2023-05-31T19:56:10.011 DEBUG:boto:Data: 2023-05-31T19:56:10.011 DEBUG:boto:Headers: {} 2023-05-31T19:56:10.011 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T19:56:10.012 DEBUG:boto:Port: 8000 2023-05-31T19:56:10.012 DEBUG:boto:Params: {} 2023-05-31T19:56:10.013 DEBUG:boto:Token: None 2023-05-31T19:56:10.013 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 19:56:10 GMT /pklxkd-8/_myobj?versionId=null 2023-05-31T19:56:10.013 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:tZcwoPQALbPg7AtocXDaGYQUcqo= 2023-05-31T19:56:10.014 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 19:56:10 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:tZcwoPQALbPg7AtocXDaGYQUcqo=', 'Content-Length': '0'} 2023-05-31T19:56:10.016 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 19:52:48 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000005794866251b4a067-006477a65a-110e-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 19:56:10 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T19:56:10.016 DEBUG:boto:path=/pklxkd-8/_myobj 2023-05-31T19:56:10.017 DEBUG:boto:auth_path=/pklxkd-8/_myobj 2023-05-31T19:56:10.017 DEBUG:boto:path=/pklxkd-8/_myobj?versionId=null 2023-05-31T19:56:10.017 DEBUG:boto:auth_path=/pklxkd-8/_myobj?versionId=null 2023-05-31T19:56:10.018 DEBUG:boto:Method: GET 2023-05-31T19:56:10.018 DEBUG:boto:Path: /pklxkd-8/_myobj?versionId=null 2023-05-31T19:56:10.018 DEBUG:boto:Data: 2023-05-31T19:56:10.019 DEBUG:boto:Headers: {} 2023-05-31T19:56:10.019 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T19:56:10.019 DEBUG:boto:Port: 8001 2023-05-31T19:56:10.020 DEBUG:boto:Params: {} 2023-05-31T19:56:10.020 DEBUG:boto:Token: None 2023-05-31T19:56:10.021 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 19:56:10 GMT /pklxkd-8/_myobj?versionId=null 2023-05-31T19:56:10.021 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:tZcwoPQALbPg7AtocXDaGYQUcqo= 2023-05-31T19:56:10.021 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 19:56:10 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:tZcwoPQALbPg7AtocXDaGYQUcqo=', 'Content-Length': '0'} 2023-05-31T19:56:10.023 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 19:52:48 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000004513df3a22fa8237-006477a65a-1112-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 19:56:10 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T19:56:10.023 DEBUG:boto:path=/pklxkd-8/_myobj 2023-05-31T19:56:10.024 DEBUG:boto:auth_path=/pklxkd-8/_myobj 2023-05-31T19:56:10.024 DEBUG:boto:path=/pklxkd-8/_myobj?versionId=null 2023-05-31T19:56:10.024 DEBUG:boto:auth_path=/pklxkd-8/_myobj?versionId=null 2023-05-31T19:56:10.025 DEBUG:boto:Method: HEAD 2023-05-31T19:56:10.025 DEBUG:boto:Path: /pklxkd-8/_myobj?versionId=null 2023-05-31T19:56:10.025 DEBUG:boto:Data: 2023-05-31T19:56:10.026 DEBUG:boto:Headers: {} 2023-05-31T19:56:10.026 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T19:56:10.026 DEBUG:boto:Port: 8000 2023-05-31T19:56:10.027 DEBUG:boto:Params: {} 2023-05-31T19:56:10.027 DEBUG:boto:Token: None 2023-05-31T19:56:10.028 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 19:56:10 GMT /pklxkd-8/_myobj?versionId=null 2023-05-31T19:56:10.028 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:pHEUbM6aqa+e9iz/blaFgor/9AU= 2023-05-31T19:56:10.029 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 19:56:10 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:pHEUbM6aqa+e9iz/blaFgor/9AU=', 'Content-Length': '0'} 2023-05-31T19:56:10.030 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 19:52:48 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000005bc71fa86cc3c4f1-006477a65a-110e-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 19:56:10 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T19:56:10.031 DEBUG:boto:path=/pklxkd-8/_myobj 2023-05-31T19:56:10.031 DEBUG:boto:auth_path=/pklxkd-8/_myobj 2023-05-31T19:56:10.032 DEBUG:boto:path=/pklxkd-8/_myobj?versionId=null 2023-05-31T19:56:10.032 DEBUG:boto:auth_path=/pklxkd-8/_myobj?versionId=null 2023-05-31T19:56:10.032 DEBUG:boto:Method: HEAD 2023-05-31T19:56:10.033 DEBUG:boto:Path: /pklxkd-8/_myobj?versionId=null 2023-05-31T19:56:10.033 DEBUG:boto:Data: 2023-05-31T19:56:10.033 DEBUG:boto:Headers: {} 2023-05-31T19:56:10.034 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T19:56:10.034 DEBUG:boto:Port: 8001 2023-05-31T19:56:10.034 DEBUG:boto:Params: {} 2023-05-31T19:56:10.035 DEBUG:boto:Token: None 2023-05-31T19:56:10.035 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 19:56:10 GMT /pklxkd-8/_myobj?versionId=null 2023-05-31T19:56:10.036 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:pHEUbM6aqa+e9iz/blaFgor/9AU= 2023-05-31T19:56:10.036 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 19:56:10 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:pHEUbM6aqa+e9iz/blaFgor/9AU=', 'Content-Length': '0'} 2023-05-31T19:56:10.037 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 19:52:48 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000362175aec4e54c68-006477a65a-1112-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 19:56:10 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T19:56:10.038 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=_myobj 2023-05-31T19:56:10.038 DEBUG:boto:path=/pklxkd-8/_myobj 2023-05-31T19:56:10.039 DEBUG:boto:auth_path=/pklxkd-8/_myobj 2023-05-31T19:56:10.039 DEBUG:boto:path=/pklxkd-8/_myobj?versionId=null 2023-05-31T19:56:10.039 DEBUG:boto:auth_path=/pklxkd-8/_myobj?versionId=null 2023-05-31T19:56:10.040 DEBUG:boto:Method: GET 2023-05-31T19:56:10.040 DEBUG:boto:Path: /pklxkd-8/_myobj?versionId=null 2023-05-31T19:56:10.041 DEBUG:boto:Data: 2023-05-31T19:56:10.041 DEBUG:boto:Headers: {} 2023-05-31T19:56:10.041 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T19:56:10.042 DEBUG:boto:Port: 8000 2023-05-31T19:56:10.042 DEBUG:boto:Params: {} 2023-05-31T19:56:10.042 DEBUG:boto:Token: None 2023-05-31T19:56:10.043 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 19:56:10 GMT /pklxkd-8/_myobj?versionId=null 2023-05-31T19:56:10.043 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:tZcwoPQALbPg7AtocXDaGYQUcqo= 2023-05-31T19:56:10.044 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 19:56:10 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:tZcwoPQALbPg7AtocXDaGYQUcqo=', 'Content-Length': '0'} 2023-05-31T19:56:10.045 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 19:52:48 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000718eeab6f56d3a9f-006477a65a-110e-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 19:56:10 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T19:56:10.046 DEBUG:boto:path=/pklxkd-8/_myobj 2023-05-31T19:56:10.046 DEBUG:boto:auth_path=/pklxkd-8/_myobj 2023-05-31T19:56:10.047 DEBUG:boto:path=/pklxkd-8/_myobj?versionId=null 2023-05-31T19:56:10.047 DEBUG:boto:auth_path=/pklxkd-8/_myobj?versionId=null 2023-05-31T19:56:10.047 DEBUG:boto:Method: GET 2023-05-31T19:56:10.048 DEBUG:boto:Path: /pklxkd-8/_myobj?versionId=null 2023-05-31T19:56:10.048 DEBUG:boto:Data: 2023-05-31T19:56:10.048 DEBUG:boto:Headers: {} 2023-05-31T19:56:10.049 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T19:56:10.049 DEBUG:boto:Port: 8001 2023-05-31T19:56:10.049 DEBUG:boto:Params: {} 2023-05-31T19:56:10.050 DEBUG:boto:Token: None 2023-05-31T19:56:10.050 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 19:56:10 GMT /pklxkd-8/_myobj?versionId=null 2023-05-31T19:56:10.051 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:tZcwoPQALbPg7AtocXDaGYQUcqo= 2023-05-31T19:56:10.051 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 19:56:10 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:tZcwoPQALbPg7AtocXDaGYQUcqo=', 'Content-Length': '0'} 2023-05-31T19:56:10.053 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 19:52:48 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000cbf25653763d1052-006477a65a-1112-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 19:56:10 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T19:56:10.053 DEBUG:boto:path=/pklxkd-8/ 2023-05-31T19:56:10.054 DEBUG:boto:auth_path=/pklxkd-8/ 2023-05-31T19:56:10.054 DEBUG:boto:path=/pklxkd-8/?prefix=_myobj 2023-05-31T19:56:10.054 DEBUG:boto:auth_path=/pklxkd-8/?prefix=_myobj 2023-05-31T19:56:10.055 DEBUG:boto:Method: GET 2023-05-31T19:56:10.055 DEBUG:boto:Path: /pklxkd-8/?prefix=_myobj 2023-05-31T19:56:10.055 DEBUG:boto:Data: 2023-05-31T19:56:10.056 DEBUG:boto:Headers: {} 2023-05-31T19:56:10.056 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T19:56:10.057 DEBUG:boto:Port: 8000 2023-05-31T19:56:10.057 DEBUG:boto:Params: {} 2023-05-31T19:56:10.057 DEBUG:boto:Token: None 2023-05-31T19:56:10.058 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 19:56:10 GMT /pklxkd-8/ 2023-05-31T19:56:10.058 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:SPU0kq56jF2a9Iff2IEhhNichhE= 2023-05-31T19:56:10.059 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 19:56:10 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:SPU0kq56jF2a9Iff2IEhhNichhE=', 'Content-Length': '0'} 2023-05-31T19:56:10.061 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000abc72690904fd4a8-006477a65a-110e-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 19:56:10 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T19:56:10.061 DEBUG:boto:b'pklxkd-8_myobj1000false_myobj2023-05-31T19:52:48.236Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T19:56:10.062 DEBUG:boto:path=/pklxkd-8/ 2023-05-31T19:56:10.062 DEBUG:boto:auth_path=/pklxkd-8/ 2023-05-31T19:56:10.063 DEBUG:boto:path=/pklxkd-8/?prefix=_myobj 2023-05-31T19:56:10.063 DEBUG:boto:auth_path=/pklxkd-8/?prefix=_myobj 2023-05-31T19:56:10.063 DEBUG:boto:Method: GET 2023-05-31T19:56:10.064 DEBUG:boto:Path: /pklxkd-8/?prefix=_myobj 2023-05-31T19:56:10.064 DEBUG:boto:Data: 2023-05-31T19:56:10.065 DEBUG:boto:Headers: {} 2023-05-31T19:56:10.065 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T19:56:10.065 DEBUG:boto:Port: 8001 2023-05-31T19:56:10.066 DEBUG:boto:Params: {} 2023-05-31T19:56:10.066 DEBUG:boto:Token: None 2023-05-31T19:56:10.066 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 19:56:10 GMT /pklxkd-8/ 2023-05-31T19:56:10.067 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:SPU0kq56jF2a9Iff2IEhhNichhE= 2023-05-31T19:56:10.067 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 19:56:10 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:SPU0kq56jF2a9Iff2IEhhNichhE=', 'Content-Length': '0'} 2023-05-31T19:56:10.069 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000057740575a6565669-006477a65a-1112-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 19:56:10 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T19:56:10.070 DEBUG:boto:b'pklxkd-8_myobj1000false_myobj2023-05-31T19:52:48.236Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T19:56:10.071 DEBUG:boto:path=/pklxkd-8/_myobj 2023-05-31T19:56:10.071 DEBUG:boto:auth_path=/pklxkd-8/_myobj 2023-05-31T19:56:10.071 DEBUG:boto:Method: HEAD 2023-05-31T19:56:10.072 DEBUG:boto:Path: /pklxkd-8/_myobj 2023-05-31T19:56:10.072 DEBUG:boto:Data: 2023-05-31T19:56:10.072 DEBUG:boto:Headers: {} 2023-05-31T19:56:10.073 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T19:56:10.073 DEBUG:boto:Port: 8000 2023-05-31T19:56:10.073 DEBUG:boto:Params: {} 2023-05-31T19:56:10.074 DEBUG:boto:Token: None 2023-05-31T19:56:10.074 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 19:56:10 GMT /pklxkd-8/_myobj 2023-05-31T19:56:10.074 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:W+Gep3FmuSyIkHZi32RHQi63Os4= 2023-05-31T19:56:10.075 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 19:56:10 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:W+Gep3FmuSyIkHZi32RHQi63Os4=', 'Content-Length': '0'} 2023-05-31T19:56:10.076 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 19:52:48 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000007c0a91a3c396cce2-006477a65a-110e-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 19:56:10 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T19:56:10.077 DEBUG:boto:path=/pklxkd-8/_myobj 2023-05-31T19:56:10.077 DEBUG:boto:auth_path=/pklxkd-8/_myobj 2023-05-31T19:56:10.078 DEBUG:boto:Method: HEAD 2023-05-31T19:56:10.078 DEBUG:boto:Path: /pklxkd-8/_myobj 2023-05-31T19:56:10.078 DEBUG:boto:Data: 2023-05-31T19:56:10.079 DEBUG:boto:Headers: {} 2023-05-31T19:56:10.079 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T19:56:10.079 DEBUG:boto:Port: 8001 2023-05-31T19:56:10.080 DEBUG:boto:Params: {} 2023-05-31T19:56:10.080 DEBUG:boto:Token: None 2023-05-31T19:56:10.080 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 19:56:10 GMT /pklxkd-8/_myobj 2023-05-31T19:56:10.081 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:W+Gep3FmuSyIkHZi32RHQi63Os4= 2023-05-31T19:56:10.081 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 19:56:10 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:W+Gep3FmuSyIkHZi32RHQi63Os4=', 'Content-Length': '0'} 2023-05-31T19:56:10.083 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 19:52:48 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000b7b974e1826df753-006477a65a-1112-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 19:56:10 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T19:56:10.083 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=_myobj 2023-05-31T19:56:10.084 DEBUG:boto:path=/pklxkd-8/_myobj 2023-05-31T19:56:10.084 DEBUG:boto:auth_path=/pklxkd-8/_myobj 2023-05-31T19:56:10.085 DEBUG:boto:Method: GET 2023-05-31T19:56:10.085 DEBUG:boto:Path: /pklxkd-8/_myobj 2023-05-31T19:56:10.085 DEBUG:boto:Data: 2023-05-31T19:56:10.086 DEBUG:boto:Headers: {} 2023-05-31T19:56:10.086 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T19:56:10.086 DEBUG:boto:Port: 8000 2023-05-31T19:56:10.087 DEBUG:boto:Params: {} 2023-05-31T19:56:10.087 DEBUG:boto:Token: None 2023-05-31T19:56:10.087 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 19:56:10 GMT /pklxkd-8/_myobj 2023-05-31T19:56:10.088 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:rhJ2kWcT3XxnojH2XRDZIo0Fd5Y= 2023-05-31T19:56:10.088 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 19:56:10 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:rhJ2kWcT3XxnojH2XRDZIo0Fd5Y=', 'Content-Length': '0'} 2023-05-31T19:56:10.090 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 19:52:48 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000095f82b93580b7882-006477a65a-110e-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 19:56:10 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T19:56:10.132 DEBUG:boto:path=/pklxkd-8/_myobj 2023-05-31T19:56:10.133 DEBUG:boto:auth_path=/pklxkd-8/_myobj 2023-05-31T19:56:10.133 DEBUG:boto:Method: GET 2023-05-31T19:56:10.133 DEBUG:boto:Path: /pklxkd-8/_myobj 2023-05-31T19:56:10.134 DEBUG:boto:Data: 2023-05-31T19:56:10.134 DEBUG:boto:Headers: {} 2023-05-31T19:56:10.134 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T19:56:10.135 DEBUG:boto:Port: 8001 2023-05-31T19:56:10.135 DEBUG:boto:Params: {} 2023-05-31T19:56:10.136 DEBUG:boto:Token: None 2023-05-31T19:56:10.136 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 19:56:10 GMT /pklxkd-8/_myobj 2023-05-31T19:56:10.136 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:rhJ2kWcT3XxnojH2XRDZIo0Fd5Y= 2023-05-31T19:56:10.137 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 19:56:10 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:rhJ2kWcT3XxnojH2XRDZIo0Fd5Y=', 'Content-Length': '0'} 2023-05-31T19:56:10.138 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 19:52:48 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000cee2f5e871747368-006477a65a-1112-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 19:56:10 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T19:56:10.139 DEBUG:boto:path=/pklxkd-8/ 2023-05-31T19:56:10.139 DEBUG:boto:auth_path=/pklxkd-8/ 2023-05-31T19:56:10.140 DEBUG:boto:path=/pklxkd-8/?prefix=_myobj 2023-05-31T19:56:10.140 DEBUG:boto:auth_path=/pklxkd-8/?prefix=_myobj 2023-05-31T19:56:10.141 DEBUG:boto:Method: GET 2023-05-31T19:56:10.141 DEBUG:boto:Path: /pklxkd-8/?prefix=_myobj 2023-05-31T19:56:10.141 DEBUG:boto:Data: 2023-05-31T19:56:10.142 DEBUG:boto:Headers: {} 2023-05-31T19:56:10.142 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T19:56:10.142 DEBUG:boto:Port: 8000 2023-05-31T19:56:10.143 DEBUG:boto:Params: {} 2023-05-31T19:56:10.143 DEBUG:boto:Token: None 2023-05-31T19:56:10.143 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 19:56:10 GMT /pklxkd-8/ 2023-05-31T19:56:10.144 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:SPU0kq56jF2a9Iff2IEhhNichhE= 2023-05-31T19:56:10.144 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 19:56:10 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:SPU0kq56jF2a9Iff2IEhhNichhE=', 'Content-Length': '0'} 2023-05-31T19:56:10.146 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ec10d071578a1996-006477a65a-110e-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 19:56:10 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T19:56:10.147 DEBUG:boto:b'pklxkd-8_myobj1000false_myobj2023-05-31T19:52:48.236Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T19:56:10.147 DEBUG:boto:path=/pklxkd-8/ 2023-05-31T19:56:10.148 DEBUG:boto:auth_path=/pklxkd-8/ 2023-05-31T19:56:10.148 DEBUG:boto:path=/pklxkd-8/?prefix=_myobj 2023-05-31T19:56:10.148 DEBUG:boto:auth_path=/pklxkd-8/?prefix=_myobj 2023-05-31T19:56:10.149 DEBUG:boto:Method: GET 2023-05-31T19:56:10.149 DEBUG:boto:Path: /pklxkd-8/?prefix=_myobj 2023-05-31T19:56:10.149 DEBUG:boto:Data: 2023-05-31T19:56:10.150 DEBUG:boto:Headers: {} 2023-05-31T19:56:10.150 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T19:56:10.150 DEBUG:boto:Port: 8001 2023-05-31T19:56:10.151 DEBUG:boto:Params: {} 2023-05-31T19:56:10.151 DEBUG:boto:Token: None 2023-05-31T19:56:10.152 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 19:56:10 GMT /pklxkd-8/ 2023-05-31T19:56:10.152 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:SPU0kq56jF2a9Iff2IEhhNichhE= 2023-05-31T19:56:10.152 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 19:56:10 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:SPU0kq56jF2a9Iff2IEhhNichhE=', 'Content-Length': '0'} 2023-05-31T19:56:10.154 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000aefa73ac3d993463-006477a65a-1112-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 19:56:10 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T19:56:10.155 DEBUG:boto:b'pklxkd-8_myobj1000false_myobj2023-05-31T19:52:48.236Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T19:56:10.155 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2023-05-31T19:56:10.156 DEBUG:boto:path=/pklxkd-8/myobj 2023-05-31T19:56:10.156 DEBUG:boto:auth_path=/pklxkd-8/myobj 2023-05-31T19:56:10.157 DEBUG:boto:path=/pklxkd-8/myobj?versionId=null 2023-05-31T19:56:10.157 DEBUG:boto:auth_path=/pklxkd-8/myobj?versionId=null 2023-05-31T19:56:10.158 DEBUG:boto:Method: GET 2023-05-31T19:56:10.158 DEBUG:boto:Path: /pklxkd-8/myobj?versionId=null 2023-05-31T19:56:10.158 DEBUG:boto:Data: 2023-05-31T19:56:10.159 DEBUG:boto:Headers: {} 2023-05-31T19:56:10.159 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T19:56:10.159 DEBUG:boto:Port: 8000 2023-05-31T19:56:10.160 DEBUG:boto:Params: {} 2023-05-31T19:56:10.160 DEBUG:boto:Token: None 2023-05-31T19:56:10.161 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 19:56:10 GMT /pklxkd-8/myobj?versionId=null 2023-05-31T19:56:10.161 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:xVBWbQBZlGkvcVVCwXDAnhWcr2w= 2023-05-31T19:56:10.161 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 19:56:10 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:xVBWbQBZlGkvcVVCwXDAnhWcr2w=', 'Content-Length': '0'} 2023-05-31T19:56:10.163 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 19:52:48 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000036376e35096bb53a-006477a65a-110e-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 19:56:10 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T19:56:10.164 DEBUG:boto:path=/pklxkd-8/myobj 2023-05-31T19:56:10.164 DEBUG:boto:auth_path=/pklxkd-8/myobj 2023-05-31T19:56:10.164 DEBUG:boto:path=/pklxkd-8/myobj?versionId=null 2023-05-31T19:56:10.165 DEBUG:boto:auth_path=/pklxkd-8/myobj?versionId=null 2023-05-31T19:56:10.165 DEBUG:boto:Method: GET 2023-05-31T19:56:10.165 DEBUG:boto:Path: /pklxkd-8/myobj?versionId=null 2023-05-31T19:56:10.166 DEBUG:boto:Data: 2023-05-31T19:56:10.166 DEBUG:boto:Headers: {} 2023-05-31T19:56:10.166 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T19:56:10.167 DEBUG:boto:Port: 8001 2023-05-31T19:56:10.167 DEBUG:boto:Params: {} 2023-05-31T19:56:10.168 DEBUG:boto:Token: None 2023-05-31T19:56:10.168 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 19:56:10 GMT /pklxkd-8/myobj?versionId=null 2023-05-31T19:56:10.168 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:xVBWbQBZlGkvcVVCwXDAnhWcr2w= 2023-05-31T19:56:10.169 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 19:56:10 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:xVBWbQBZlGkvcVVCwXDAnhWcr2w=', 'Content-Length': '0'} 2023-05-31T19:56:10.171 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 19:52:48 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000005ae24e30136e8db1-006477a65a-1112-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 19:56:10 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T19:56:10.171 DEBUG:boto:path=/pklxkd-8/myobj 2023-05-31T19:56:10.172 DEBUG:boto:auth_path=/pklxkd-8/myobj 2023-05-31T19:56:10.172 DEBUG:boto:path=/pklxkd-8/myobj?versionId=null 2023-05-31T19:56:10.172 DEBUG:boto:auth_path=/pklxkd-8/myobj?versionId=null 2023-05-31T19:56:10.173 DEBUG:boto:Method: HEAD 2023-05-31T19:56:10.173 DEBUG:boto:Path: /pklxkd-8/myobj?versionId=null 2023-05-31T19:56:10.173 DEBUG:boto:Data: 2023-05-31T19:56:10.174 DEBUG:boto:Headers: {} 2023-05-31T19:56:10.174 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T19:56:10.174 DEBUG:boto:Port: 8000 2023-05-31T19:56:10.175 DEBUG:boto:Params: {} 2023-05-31T19:56:10.175 DEBUG:boto:Token: None 2023-05-31T19:56:10.176 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 19:56:10 GMT /pklxkd-8/myobj?versionId=null 2023-05-31T19:56:10.176 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:HAlT0xcfMxJ1Gvz8/vMJOoSe69c= 2023-05-31T19:56:10.176 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 19:56:10 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:HAlT0xcfMxJ1Gvz8/vMJOoSe69c=', 'Content-Length': '0'} 2023-05-31T19:56:10.178 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 19:52:48 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000c7ba1d9af8f63d29-006477a65a-110e-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 19:56:10 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T19:56:10.179 DEBUG:boto:path=/pklxkd-8/myobj 2023-05-31T19:56:10.179 DEBUG:boto:auth_path=/pklxkd-8/myobj 2023-05-31T19:56:10.179 DEBUG:boto:path=/pklxkd-8/myobj?versionId=null 2023-05-31T19:56:10.180 DEBUG:boto:auth_path=/pklxkd-8/myobj?versionId=null 2023-05-31T19:56:10.180 DEBUG:boto:Method: HEAD 2023-05-31T19:56:10.180 DEBUG:boto:Path: /pklxkd-8/myobj?versionId=null 2023-05-31T19:56:10.181 DEBUG:boto:Data: 2023-05-31T19:56:10.181 DEBUG:boto:Headers: {} 2023-05-31T19:56:10.181 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T19:56:10.182 DEBUG:boto:Port: 8001 2023-05-31T19:56:10.182 DEBUG:boto:Params: {} 2023-05-31T19:56:10.182 DEBUG:boto:Token: None 2023-05-31T19:56:10.183 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 19:56:10 GMT /pklxkd-8/myobj?versionId=null 2023-05-31T19:56:10.183 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:HAlT0xcfMxJ1Gvz8/vMJOoSe69c= 2023-05-31T19:56:10.184 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 19:56:10 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:HAlT0xcfMxJ1Gvz8/vMJOoSe69c=', 'Content-Length': '0'} 2023-05-31T19:56:10.185 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 19:52:48 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000643b60d2dbb76366-006477a65a-1112-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 19:56:10 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T19:56:10.186 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2023-05-31T19:56:10.186 DEBUG:boto:path=/pklxkd-8/myobj 2023-05-31T19:56:10.187 DEBUG:boto:auth_path=/pklxkd-8/myobj 2023-05-31T19:56:10.187 DEBUG:boto:path=/pklxkd-8/myobj?versionId=null 2023-05-31T19:56:10.187 DEBUG:boto:auth_path=/pklxkd-8/myobj?versionId=null 2023-05-31T19:56:10.188 DEBUG:boto:Method: GET 2023-05-31T19:56:10.188 DEBUG:boto:Path: /pklxkd-8/myobj?versionId=null 2023-05-31T19:56:10.188 DEBUG:boto:Data: 2023-05-31T19:56:10.189 DEBUG:boto:Headers: {} 2023-05-31T19:56:10.189 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T19:56:10.190 DEBUG:boto:Port: 8000 2023-05-31T19:56:10.190 DEBUG:boto:Params: {} 2023-05-31T19:56:10.190 DEBUG:boto:Token: None 2023-05-31T19:56:10.191 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 19:56:10 GMT /pklxkd-8/myobj?versionId=null 2023-05-31T19:56:10.191 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:xVBWbQBZlGkvcVVCwXDAnhWcr2w= 2023-05-31T19:56:10.192 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 19:56:10 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:xVBWbQBZlGkvcVVCwXDAnhWcr2w=', 'Content-Length': '0'} 2023-05-31T19:56:10.193 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 19:52:48 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000835c82c8cfacb9bd-006477a65a-110e-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 19:56:10 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T19:56:10.194 DEBUG:boto:path=/pklxkd-8/myobj 2023-05-31T19:56:10.194 DEBUG:boto:auth_path=/pklxkd-8/myobj 2023-05-31T19:56:10.194 DEBUG:boto:path=/pklxkd-8/myobj?versionId=null 2023-05-31T19:56:10.195 DEBUG:boto:auth_path=/pklxkd-8/myobj?versionId=null 2023-05-31T19:56:10.195 DEBUG:boto:Method: GET 2023-05-31T19:56:10.196 DEBUG:boto:Path: /pklxkd-8/myobj?versionId=null 2023-05-31T19:56:10.196 DEBUG:boto:Data: 2023-05-31T19:56:10.196 DEBUG:boto:Headers: {} 2023-05-31T19:56:10.197 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T19:56:10.197 DEBUG:boto:Port: 8001 2023-05-31T19:56:10.198 DEBUG:boto:Params: {} 2023-05-31T19:56:10.198 DEBUG:boto:Token: None 2023-05-31T19:56:10.199 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 19:56:10 GMT /pklxkd-8/myobj?versionId=null 2023-05-31T19:56:10.199 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:xVBWbQBZlGkvcVVCwXDAnhWcr2w= 2023-05-31T19:56:10.199 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 19:56:10 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:xVBWbQBZlGkvcVVCwXDAnhWcr2w=', 'Content-Length': '0'} 2023-05-31T19:56:10.201 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 19:52:48 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000000f1c78e8f01a7f80-006477a65a-1112-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 19:56:10 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T19:56:10.202 DEBUG:boto:path=/pklxkd-8/ 2023-05-31T19:56:10.202 DEBUG:boto:auth_path=/pklxkd-8/ 2023-05-31T19:56:10.203 DEBUG:boto:path=/pklxkd-8/?prefix=myobj 2023-05-31T19:56:10.203 DEBUG:boto:auth_path=/pklxkd-8/?prefix=myobj 2023-05-31T19:56:10.203 DEBUG:boto:Method: GET 2023-05-31T19:56:10.204 DEBUG:boto:Path: /pklxkd-8/?prefix=myobj 2023-05-31T19:56:10.204 DEBUG:boto:Data: 2023-05-31T19:56:10.204 DEBUG:boto:Headers: {} 2023-05-31T19:56:10.205 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T19:56:10.205 DEBUG:boto:Port: 8000 2023-05-31T19:56:10.205 DEBUG:boto:Params: {} 2023-05-31T19:56:10.206 DEBUG:boto:Token: None 2023-05-31T19:56:10.206 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 19:56:10 GMT /pklxkd-8/ 2023-05-31T19:56:10.207 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:SPU0kq56jF2a9Iff2IEhhNichhE= 2023-05-31T19:56:10.207 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 19:56:10 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:SPU0kq56jF2a9Iff2IEhhNichhE=', 'Content-Length': '0'} 2023-05-31T19:56:10.209 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000012059d3f951746fd-006477a65a-110e-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 19:56:10 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T19:56:10.209 DEBUG:boto:b'pklxkd-8myobj1000falsemyobj2023-05-31T19:52:48.211Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T19:56:10.210 DEBUG:boto:path=/pklxkd-8/ 2023-05-31T19:56:10.211 DEBUG:boto:auth_path=/pklxkd-8/ 2023-05-31T19:56:10.211 DEBUG:boto:path=/pklxkd-8/?prefix=myobj 2023-05-31T19:56:10.211 DEBUG:boto:auth_path=/pklxkd-8/?prefix=myobj 2023-05-31T19:56:10.212 DEBUG:boto:Method: GET 2023-05-31T19:56:10.212 DEBUG:boto:Path: /pklxkd-8/?prefix=myobj 2023-05-31T19:56:10.212 DEBUG:boto:Data: 2023-05-31T19:56:10.213 DEBUG:boto:Headers: {} 2023-05-31T19:56:10.213 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T19:56:10.213 DEBUG:boto:Port: 8001 2023-05-31T19:56:10.214 DEBUG:boto:Params: {} 2023-05-31T19:56:10.214 DEBUG:boto:Token: None 2023-05-31T19:56:10.215 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 19:56:10 GMT /pklxkd-8/ 2023-05-31T19:56:10.215 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:SPU0kq56jF2a9Iff2IEhhNichhE= 2023-05-31T19:56:10.215 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 19:56:10 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:SPU0kq56jF2a9Iff2IEhhNichhE=', 'Content-Length': '0'} 2023-05-31T19:56:10.217 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000076e235ae9edd9220-006477a65a-1112-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 19:56:10 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T19:56:10.218 DEBUG:boto:b'pklxkd-8myobj1000falsemyobj2023-05-31T19:52:48.211Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T19:56:10.218 DEBUG:boto:path=/pklxkd-8/myobj 2023-05-31T19:56:10.219 DEBUG:boto:auth_path=/pklxkd-8/myobj 2023-05-31T19:56:10.219 DEBUG:boto:Method: HEAD 2023-05-31T19:56:10.219 DEBUG:boto:Path: /pklxkd-8/myobj 2023-05-31T19:56:10.220 DEBUG:boto:Data: 2023-05-31T19:56:10.220 DEBUG:boto:Headers: {} 2023-05-31T19:56:10.220 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T19:56:10.221 DEBUG:boto:Port: 8000 2023-05-31T19:56:10.221 DEBUG:boto:Params: {} 2023-05-31T19:56:10.221 DEBUG:boto:Token: None 2023-05-31T19:56:10.222 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 19:56:10 GMT /pklxkd-8/myobj 2023-05-31T19:56:10.222 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:NflQ+k8fV4XRxJRhLbu6CiP0VD0= 2023-05-31T19:56:10.223 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 19:56:10 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:NflQ+k8fV4XRxJRhLbu6CiP0VD0=', 'Content-Length': '0'} 2023-05-31T19:56:10.224 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 19:52:48 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000099efc92d3344d918-006477a65a-110e-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 19:56:10 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T19:56:10.225 DEBUG:boto:path=/pklxkd-8/myobj 2023-05-31T19:56:10.225 DEBUG:boto:auth_path=/pklxkd-8/myobj 2023-05-31T19:56:10.225 DEBUG:boto:Method: HEAD 2023-05-31T19:56:10.226 DEBUG:boto:Path: /pklxkd-8/myobj 2023-05-31T19:56:10.226 DEBUG:boto:Data: 2023-05-31T19:56:10.226 DEBUG:boto:Headers: {} 2023-05-31T19:56:10.227 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T19:56:10.227 DEBUG:boto:Port: 8001 2023-05-31T19:56:10.227 DEBUG:boto:Params: {} 2023-05-31T19:56:10.228 DEBUG:boto:Token: None 2023-05-31T19:56:10.228 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 19:56:10 GMT /pklxkd-8/myobj 2023-05-31T19:56:10.228 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:NflQ+k8fV4XRxJRhLbu6CiP0VD0= 2023-05-31T19:56:10.229 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 19:56:10 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:NflQ+k8fV4XRxJRhLbu6CiP0VD0=', 'Content-Length': '0'} 2023-05-31T19:56:10.230 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 19:52:48 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000036c97c0cf0b93e40-006477a65a-1112-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 19:56:10 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T19:56:10.231 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2023-05-31T19:56:10.231 DEBUG:boto:path=/pklxkd-8/myobj 2023-05-31T19:56:10.231 DEBUG:boto:auth_path=/pklxkd-8/myobj 2023-05-31T19:56:10.232 DEBUG:boto:Method: GET 2023-05-31T19:56:10.232 DEBUG:boto:Path: /pklxkd-8/myobj 2023-05-31T19:56:10.233 DEBUG:boto:Data: 2023-05-31T19:56:10.233 DEBUG:boto:Headers: {} 2023-05-31T19:56:10.233 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T19:56:10.234 DEBUG:boto:Port: 8000 2023-05-31T19:56:10.234 DEBUG:boto:Params: {} 2023-05-31T19:56:10.234 DEBUG:boto:Token: None 2023-05-31T19:56:10.235 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 19:56:10 GMT /pklxkd-8/myobj 2023-05-31T19:56:10.235 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:36cldlYbYozuSlu8un/oxkBM/3k= 2023-05-31T19:56:10.236 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 19:56:10 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:36cldlYbYozuSlu8un/oxkBM/3k=', 'Content-Length': '0'} 2023-05-31T19:56:10.237 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 19:52:48 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000e0c1e46a2ff1fd51-006477a65a-110e-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 19:56:10 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T19:56:10.238 DEBUG:boto:path=/pklxkd-8/myobj 2023-05-31T19:56:10.238 DEBUG:boto:auth_path=/pklxkd-8/myobj 2023-05-31T19:56:10.238 DEBUG:boto:Method: GET 2023-05-31T19:56:10.239 DEBUG:boto:Path: /pklxkd-8/myobj 2023-05-31T19:56:10.239 DEBUG:boto:Data: 2023-05-31T19:56:10.239 DEBUG:boto:Headers: {} 2023-05-31T19:56:10.240 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T19:56:10.240 DEBUG:boto:Port: 8001 2023-05-31T19:56:10.241 DEBUG:boto:Params: {} 2023-05-31T19:56:10.241 DEBUG:boto:Token: None 2023-05-31T19:56:10.242 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 19:56:10 GMT /pklxkd-8/myobj 2023-05-31T19:56:10.242 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:36cldlYbYozuSlu8un/oxkBM/3k= 2023-05-31T19:56:10.242 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 19:56:10 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:36cldlYbYozuSlu8un/oxkBM/3k=', 'Content-Length': '0'} 2023-05-31T19:56:10.244 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 19:52:48 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000003963cf5729f473f5-006477a65a-1112-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 19:56:10 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T19:56:10.245 DEBUG:boto:path=/pklxkd-8/ 2023-05-31T19:56:10.245 DEBUG:boto:auth_path=/pklxkd-8/ 2023-05-31T19:56:10.246 DEBUG:boto:path=/pklxkd-8/?prefix=myobj 2023-05-31T19:56:10.246 DEBUG:boto:auth_path=/pklxkd-8/?prefix=myobj 2023-05-31T19:56:10.246 DEBUG:boto:Method: GET 2023-05-31T19:56:10.247 DEBUG:boto:Path: /pklxkd-8/?prefix=myobj 2023-05-31T19:56:10.247 DEBUG:boto:Data: 2023-05-31T19:56:10.247 DEBUG:boto:Headers: {} 2023-05-31T19:56:10.248 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T19:56:10.248 DEBUG:boto:Port: 8000 2023-05-31T19:56:10.249 DEBUG:boto:Params: {} 2023-05-31T19:56:10.249 DEBUG:boto:Token: None 2023-05-31T19:56:10.249 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 19:56:10 GMT /pklxkd-8/ 2023-05-31T19:56:10.250 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:SPU0kq56jF2a9Iff2IEhhNichhE= 2023-05-31T19:56:10.250 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 19:56:10 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:SPU0kq56jF2a9Iff2IEhhNichhE=', 'Content-Length': '0'} 2023-05-31T19:56:10.252 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000004743fb44633cc3f-006477a65a-110e-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 19:56:10 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T19:56:10.253 DEBUG:boto:b'pklxkd-8myobj1000falsemyobj2023-05-31T19:52:48.211Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T19:56:10.253 DEBUG:boto:path=/pklxkd-8/ 2023-05-31T19:56:10.254 DEBUG:boto:auth_path=/pklxkd-8/ 2023-05-31T19:56:10.254 DEBUG:boto:path=/pklxkd-8/?prefix=myobj 2023-05-31T19:56:10.254 DEBUG:boto:auth_path=/pklxkd-8/?prefix=myobj 2023-05-31T19:56:10.255 DEBUG:boto:Method: GET 2023-05-31T19:56:10.255 DEBUG:boto:Path: /pklxkd-8/?prefix=myobj 2023-05-31T19:56:10.255 DEBUG:boto:Data: 2023-05-31T19:56:10.256 DEBUG:boto:Headers: {} 2023-05-31T19:56:10.256 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T19:56:10.257 DEBUG:boto:Port: 8001 2023-05-31T19:56:10.257 DEBUG:boto:Params: {} 2023-05-31T19:56:10.257 DEBUG:boto:Token: None 2023-05-31T19:56:10.258 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 19:56:10 GMT /pklxkd-8/ 2023-05-31T19:56:10.258 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:SPU0kq56jF2a9Iff2IEhhNichhE= 2023-05-31T19:56:10.258 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 19:56:10 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:SPU0kq56jF2a9Iff2IEhhNichhE=', 'Content-Length': '0'} 2023-05-31T19:56:10.260 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000001ef141466624d69-006477a65a-1112-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 19:56:10 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T19:56:10.261 DEBUG:boto:b'pklxkd-8myobj1000falsemyobj2023-05-31T19:52:48.211Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T19:56:10.261 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=pklxkd-8 zones={a1, a2} 2023-05-31T19:56:10.262 INFO:tasks.rgw_multisite_tests:rgw_multi.tests.test_object_sync ... ok 2023-05-31T19:56:10.263 INFO:rgw_multi.tests:create bucket zone=a1 name=pklxkd-9 2023-05-31T19:56:10.264 DEBUG:boto:path=/pklxkd-9/ 2023-05-31T19:56:10.264 DEBUG:boto:auth_path=/pklxkd-9/ 2023-05-31T19:56:10.265 DEBUG:boto:Method: PUT 2023-05-31T19:56:10.265 DEBUG:boto:Path: /pklxkd-9/ 2023-05-31T19:56:10.265 DEBUG:boto:Data: 2023-05-31T19:56:10.265 DEBUG:boto:Headers: {} 2023-05-31T19:56:10.266 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T19:56:10.266 DEBUG:boto:Port: 8000 2023-05-31T19:56:10.266 DEBUG:boto:Params: {} 2023-05-31T19:56:10.267 DEBUG:boto:Token: None 2023-05-31T19:56:10.267 DEBUG:boto:StringToSign: PUT Wed, 31 May 2023 19:56:10 GMT /pklxkd-9/ 2023-05-31T19:56:10.268 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:s/cQiTaPi8RmybPCgaJt66XRDQ4= 2023-05-31T19:56:10.268 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 19:56:10 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:s/cQiTaPi8RmybPCgaJt66XRDQ4=', 'Content-Length': '0'} 2023-05-31T19:56:10.280 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx000009504c73a0556a929-006477a65a-110e-a1'), ('Content-Length', '0'), ('Date', 'Wed, 31 May 2023 19:56:10 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T19:56:10.280 INFO:rgw_multi.tests:create bucket zone=a2 name=pklxkd-10 2023-05-31T19:56:10.281 DEBUG:boto:path=/pklxkd-10/ 2023-05-31T19:56:10.281 DEBUG:boto:auth_path=/pklxkd-10/ 2023-05-31T19:56:10.281 DEBUG:boto:Method: PUT 2023-05-31T19:56:10.282 DEBUG:boto:Path: /pklxkd-10/ 2023-05-31T19:56:10.282 DEBUG:boto:Data: 2023-05-31T19:56:10.283 DEBUG:boto:Headers: {} 2023-05-31T19:56:10.283 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T19:56:10.283 DEBUG:boto:Port: 8001 2023-05-31T19:56:10.283 DEBUG:boto:Params: {} 2023-05-31T19:56:10.284 DEBUG:boto:Token: None 2023-05-31T19:56:10.284 DEBUG:boto:StringToSign: PUT Wed, 31 May 2023 19:56:10 GMT /pklxkd-10/ 2023-05-31T19:56:10.285 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:KiCGRNtya6H3h5ZVn4IlIvDFut0= 2023-05-31T19:56:10.285 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 19:56:10 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:KiCGRNtya6H3h5ZVn4IlIvDFut0=', 'Content-Length': '0'} 2023-05-31T19:56:10.305 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx000000dfa6f7eb172460d-006477a65a-1112-a2'), ('Content-Length', '0'), ('Date', 'Wed, 31 May 2023 19:56:10 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T19:56:10.305 DEBUG:boto:path=/pklxkd-9/ 2023-05-31T19:56:10.305 DEBUG:boto:auth_path=/pklxkd-9/ 2023-05-31T19:56:10.306 DEBUG:boto:Method: HEAD 2023-05-31T19:56:10.306 DEBUG:boto:Path: /pklxkd-9/ 2023-05-31T19:56:10.306 DEBUG:boto:Data: 2023-05-31T19:56:10.307 DEBUG:boto:Headers: {} 2023-05-31T19:56:10.307 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T19:56:10.307 DEBUG:boto:Port: 8000 2023-05-31T19:56:10.308 DEBUG:boto:Params: {} 2023-05-31T19:56:10.308 DEBUG:boto:Token: None 2023-05-31T19:56:10.309 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 19:56:10 GMT /pklxkd-9/ 2023-05-31T19:56:10.309 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:u0gyDBeX/cvIYnhx3Beko2gRJXI= 2023-05-31T19:56:10.309 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 19:56:10 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:u0gyDBeX/cvIYnhx3Beko2gRJXI=', 'Content-Length': '0'} 2023-05-31T19:56:10.311 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000f0227a0d0e6fb735-006477a65a-110e-a1'), ('Content-Length', '0'), ('Date', 'Wed, 31 May 2023 19:56:10 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T19:56:10.312 DEBUG:boto:path=/pklxkd-9/myobj 2023-05-31T19:56:10.312 DEBUG:boto:auth_path=/pklxkd-9/myobj 2023-05-31T19:56:10.313 DEBUG:boto:Method: PUT 2023-05-31T19:56:10.313 DEBUG:boto:Path: /pklxkd-9/myobj 2023-05-31T19:56:10.313 DEBUG:boto:Data: 2023-05-31T19:56:10.314 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2023-05-31T19:56:10.314 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T19:56:10.314 DEBUG:boto:Port: 8000 2023-05-31T19:56:10.315 DEBUG:boto:Params: {} 2023-05-31T19:56:10.315 DEBUG:boto:Token: None 2023-05-31T19:56:10.315 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Wed, 31 May 2023 19:56:10 GMT /pklxkd-9/myobj 2023-05-31T19:56:10.316 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:Vl47GkzUgzpz7vnHKSjBs/V0n+w= 2023-05-31T19:56:10.316 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Wed, 31 May 2023 19:56:10 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:Vl47GkzUgzpz7vnHKSjBs/V0n+w='} 2023-05-31T19:56:10.360 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000a2032a8cf3b2e57b-006477a65a-110e-a1'), ('Date', 'Wed, 31 May 2023 19:56:10 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T19:56:10.361 DEBUG:boto:path=/pklxkd-10/ 2023-05-31T19:56:10.361 DEBUG:boto:auth_path=/pklxkd-10/ 2023-05-31T19:56:10.361 DEBUG:boto:Method: HEAD 2023-05-31T19:56:10.362 DEBUG:boto:Path: /pklxkd-10/ 2023-05-31T19:56:10.362 DEBUG:boto:Data: 2023-05-31T19:56:10.362 DEBUG:boto:Headers: {} 2023-05-31T19:56:10.363 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T19:56:10.363 DEBUG:boto:Port: 8001 2023-05-31T19:56:10.363 DEBUG:boto:Params: {} 2023-05-31T19:56:10.364 DEBUG:boto:Token: None 2023-05-31T19:56:10.364 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 19:56:10 GMT /pklxkd-10/ 2023-05-31T19:56:10.364 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:n0v9iqV7YSxg6B47vudyC5l9xLY= 2023-05-31T19:56:10.365 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 19:56:10 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:n0v9iqV7YSxg6B47vudyC5l9xLY=', 'Content-Length': '0'} 2023-05-31T19:56:10.367 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000676e273943eb33cf-006477a65a-1112-a2'), ('Content-Length', '0'), ('Date', 'Wed, 31 May 2023 19:56:10 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T19:56:10.367 DEBUG:boto:path=/pklxkd-10/myobj 2023-05-31T19:56:10.368 DEBUG:boto:auth_path=/pklxkd-10/myobj 2023-05-31T19:56:10.368 DEBUG:boto:Method: PUT 2023-05-31T19:56:10.368 DEBUG:boto:Path: /pklxkd-10/myobj 2023-05-31T19:56:10.369 DEBUG:boto:Data: 2023-05-31T19:56:10.369 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2023-05-31T19:56:10.369 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T19:56:10.370 DEBUG:boto:Port: 8001 2023-05-31T19:56:10.370 DEBUG:boto:Params: {} 2023-05-31T19:56:10.370 DEBUG:boto:Token: None 2023-05-31T19:56:10.371 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Wed, 31 May 2023 19:56:10 GMT /pklxkd-10/myobj 2023-05-31T19:56:10.371 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:aLgItMSmc0cLKo9dzsnXVf5YYjI= 2023-05-31T19:56:10.372 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Wed, 31 May 2023 19:56:10 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:aLgItMSmc0cLKo9dzsnXVf5YYjI='} 2023-05-31T19:56:10.383 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000005478e20ec6a06294-006477a65a-1112-a2'), ('Date', 'Wed, 31 May 2023 19:56:10 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T19:56:10.383 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-31T19:56:10.384 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-31T19:56:10.385 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-05-31T19:56:10.433 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2023-05-31T19:56:10.433 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2023-05-31T19:56:10.489 INFO:teuthology.orchestra.run.smithi099.stderr:No --period given, using current period=2e6123d2-d695-4350-9a01-61a455bc782b 2023-05-31T19:56:10.516 INFO:teuthology.orchestra.run.smithi099.stdout:[ 2023-05-31T19:56:10.516 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T19:56:10.516 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1685562631.516089_59.1", 2023-05-31T19:56:10.516 INFO:teuthology.orchestra.run.smithi099.stdout: "last_update": "2023-05-31T19:50:31.516089Z" 2023-05-31T19:56:10.517 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T19:56:10.517 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T19:56:10.517 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "", 2023-05-31T19:56:10.517 INFO:teuthology.orchestra.run.smithi099.stdout: "last_update": "0.000000" 2023-05-31T19:56:10.517 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T19:56:10.517 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T19:56:10.517 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1685562970.274572_93.1", 2023-05-31T19:56:10.518 INFO:teuthology.orchestra.run.smithi099.stdout: "last_update": "2023-05-31T19:56:10.274572Z" 2023-05-31T19:56:10.518 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T19:56:10.518 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T19:56:10.518 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1685562970.292149_34.1", 2023-05-31T19:56:10.518 INFO:teuthology.orchestra.run.smithi099.stdout: "last_update": "2023-05-31T19:56:10.292149Z" 2023-05-31T19:56:10.518 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T19:56:10.518 INFO:teuthology.orchestra.run.smithi099.stdout:] 2023-05-31T19:56:10.519 DEBUG:rgw_multi.tests:master meta markers={0: '1_1685562631.516089_59.1', 1: '', 2: '1_1685562970.274572_93.1', 3: '1_1685562970.292149_34.1'} 2023-05-31T19:56:10.520 INFO:rgw_multi.tests:starting meta checkpoint for zone=a2 2023-05-31T19:56:10.520 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-31T19:56:10.521 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-31T19:56:10.521 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-05-31T19:56:10.570 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2023-05-31T19:56:10.570 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2023-05-31T19:56:10.607 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T19:56:10.602+0000 7f6ef863f780 20 RGW-SYNC:meta: read sync status 2023-05-31T19:56:10.608 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T19:56:10.603+0000 7f6ef863f780 20 RGW-SYNC:meta: read sync status 2023-05-31T19:56:10.620 INFO:teuthology.orchestra.run.smithi099.stdout:{ 2023-05-31T19:56:10.620 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_status": { 2023-05-31T19:56:10.620 INFO:teuthology.orchestra.run.smithi099.stdout: "info": { 2023-05-31T19:56:10.620 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "sync", 2023-05-31T19:56:10.620 INFO:teuthology.orchestra.run.smithi099.stdout: "num_shards": 4, 2023-05-31T19:56:10.621 INFO:teuthology.orchestra.run.smithi099.stdout: "period": "2e6123d2-d695-4350-9a01-61a455bc782b", 2023-05-31T19:56:10.621 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2023-05-31T19:56:10.621 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T19:56:10.621 INFO:teuthology.orchestra.run.smithi099.stdout: "markers": [ 2023-05-31T19:56:10.621 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T19:56:10.621 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 0, 2023-05-31T19:56:10.622 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T19:56:10.622 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2023-05-31T19:56:10.622 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1685562631.516089_59.1", 2023-05-31T19:56:10.622 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T19:56:10.622 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 1, 2023-05-31T19:56:10.622 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T19:56:10.622 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2023-05-31T19:50:31.516089Z", 2023-05-31T19:56:10.623 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2023-05-31T19:56:10.623 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T19:56:10.623 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T19:56:10.623 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T19:56:10.623 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 1, 2023-05-31T19:56:10.623 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T19:56:10.623 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2023-05-31T19:56:10.624 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "", 2023-05-31T19:56:10.624 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T19:56:10.624 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2023-05-31T19:56:10.624 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T19:56:10.624 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "0.000000", 2023-05-31T19:56:10.625 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2023-05-31T19:56:10.625 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T19:56:10.625 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T19:56:10.625 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T19:56:10.625 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 2, 2023-05-31T19:56:10.625 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T19:56:10.625 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2023-05-31T19:56:10.626 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1685562970.274572_93.1", 2023-05-31T19:56:10.626 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T19:56:10.626 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2023-05-31T19:56:10.626 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T19:56:10.626 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2023-05-31T19:56:10.274572Z", 2023-05-31T19:56:10.626 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2023-05-31T19:56:10.626 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T19:56:10.627 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T19:56:10.627 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T19:56:10.627 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 3, 2023-05-31T19:56:10.627 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T19:56:10.627 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2023-05-31T19:56:10.627 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1685562970.292149_34.1", 2023-05-31T19:56:10.627 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T19:56:10.628 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2023-05-31T19:56:10.628 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T19:56:10.628 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2023-05-31T19:56:10.292149Z", 2023-05-31T19:56:10.628 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2023-05-31T19:56:10.628 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T19:56:10.628 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T19:56:10.628 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2023-05-31T19:56:10.629 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T19:56:10.629 INFO:teuthology.orchestra.run.smithi099.stdout: "full_sync": { 2023-05-31T19:56:10.629 INFO:teuthology.orchestra.run.smithi099.stdout: "total": 1, 2023-05-31T19:56:10.629 INFO:teuthology.orchestra.run.smithi099.stdout: "complete": 1 2023-05-31T19:56:10.629 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T19:56:10.629 INFO:teuthology.orchestra.run.smithi099.stdout: "current_time": "2023-05-31T19:56:10Z" 2023-05-31T19:56:10.629 INFO:teuthology.orchestra.run.smithi099.stdout:} 2023-05-31T19:56:10.630 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "2e6123d2-d695-4350-9a01-61a455bc782b", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1685562631.516089_59.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-05-31T19:50:31.516089Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1685562970.274572_93.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-05-31T19:56:10.274572Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1685562970.292149_34.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-05-31T19:56:10.292149Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-05-31T19:56:10Z" } 2023-05-31T19:56:10.631 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1685562631.516089_59.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-05-31T19:50:31.516089Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1685562970.274572_93.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-05-31T19:56:10.274572Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1685562970.292149_34.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-05-31T19:56:10.292149Z', 'realm_epoch': 2}}] 2023-05-31T19:56:10.631 DEBUG:rgw_multi.tests:log_status={0: '1_1685562631.516089_59.1', 1: '', 2: '1_1685562970.274572_93.1', 3: '1_1685562970.292149_34.1'} 2023-05-31T19:56:10.632 DEBUG:rgw_multi.tests:sync_status={0: '1_1685562631.516089_59.1', 1: '', 2: '1_1685562970.274572_93.1', 3: '1_1685562970.292149_34.1'} 2023-05-31T19:56:10.632 INFO:rgw_multi.tests:finish meta checkpoint for zone=a2 2023-05-31T19:56:10.632 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'pklxkd-9', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-05-31T19:56:10.633 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'pklxkd-9', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-05-31T19:56:10.633 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket pklxkd-9 --source-zone a1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2023-05-31T19:56:10.688 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2023-05-31T19:56:10.688 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2023-05-31T19:56:10.733 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T19:56:10.728+0000 7f3fb9ddc780 1 waiting to reach incremental sync.. 2023-05-31T19:56:15.734 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T19:56:15.729+0000 7f3fb9ddc780 1 waiting to reach incremental sync.. 2023-05-31T19:56:20.734 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T19:56:20.729+0000 7f3fb9ddc780 1 waiting to reach incremental sync.. 2023-05-31T19:56:25.735 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T19:56:25.730+0000 7f3fb9ddc780 1 waiting to reach incremental sync.. 2023-05-31T19:56:30.736 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T19:56:30.731+0000 7f3fb9ddc780 1 bucket sync caught up with source: 2023-05-31T19:56:30.737 INFO:teuthology.orchestra.run.smithi099.stderr: local status: [, 00000000001.8.6, , , , , , , , , ] 2023-05-31T19:56:30.737 INFO:teuthology.orchestra.run.smithi099.stderr: remote markers: [, 00000000001.8.6, , , , , , , , , ] 2023-05-31T19:56:30.737 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T19:56:30.731+0000 7f3fb9ddc780 0 bucket checkpoint complete 2023-05-31T19:56:30.755 INFO:tasks.rgw_multi.zone_rados:comparing bucket=pklxkd-9 zones={a2, a1} 2023-05-31T19:56:30.756 DEBUG:boto:path=/pklxkd-9/ 2023-05-31T19:56:30.756 DEBUG:boto:auth_path=/pklxkd-9/ 2023-05-31T19:56:30.756 DEBUG:boto:Method: HEAD 2023-05-31T19:56:30.757 DEBUG:boto:Path: /pklxkd-9/ 2023-05-31T19:56:30.757 DEBUG:boto:Data: 2023-05-31T19:56:30.757 DEBUG:boto:Headers: {} 2023-05-31T19:56:30.758 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T19:56:30.758 DEBUG:boto:Port: 8001 2023-05-31T19:56:30.758 DEBUG:boto:Params: {} 2023-05-31T19:56:30.759 DEBUG:boto:Token: None 2023-05-31T19:56:30.759 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 19:56:30 GMT /pklxkd-9/ 2023-05-31T19:56:30.760 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:DhYKUMsARxMtMoaxmcR5x8+jhZI= 2023-05-31T19:56:30.760 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 19:56:30 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:DhYKUMsARxMtMoaxmcR5x8+jhZI=', 'Content-Length': '0'} 2023-05-31T19:56:30.763 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '6'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000007156fd8c46e5ec9d-006477a66e-1112-a2'), ('Content-Length', '0'), ('Date', 'Wed, 31 May 2023 19:56:30 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T19:56:30.763 DEBUG:boto:path=/pklxkd-9/ 2023-05-31T19:56:30.763 DEBUG:boto:auth_path=/pklxkd-9/ 2023-05-31T19:56:30.764 DEBUG:boto:Method: HEAD 2023-05-31T19:56:30.764 DEBUG:boto:Path: /pklxkd-9/ 2023-05-31T19:56:30.765 DEBUG:boto:Data: 2023-05-31T19:56:30.765 DEBUG:boto:Headers: {} 2023-05-31T19:56:30.765 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T19:56:30.766 DEBUG:boto:Port: 8000 2023-05-31T19:56:30.766 DEBUG:boto:Params: {} 2023-05-31T19:56:30.766 DEBUG:boto:Token: None 2023-05-31T19:56:30.767 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 19:56:30 GMT /pklxkd-9/ 2023-05-31T19:56:30.767 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:DhYKUMsARxMtMoaxmcR5x8+jhZI= 2023-05-31T19:56:30.767 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 19:56:30 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:DhYKUMsARxMtMoaxmcR5x8+jhZI=', 'Content-Length': '0'} 2023-05-31T19:56:30.770 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '6'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000696acee320bfaafd-006477a66e-110e-a1'), ('Content-Length', '0'), ('Date', 'Wed, 31 May 2023 19:56:30 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T19:56:30.770 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2023-05-31T19:56:30.771 DEBUG:boto:path=/pklxkd-9/ 2023-05-31T19:56:30.771 DEBUG:boto:auth_path=/pklxkd-9/ 2023-05-31T19:56:30.771 DEBUG:boto:path=/pklxkd-9/?versions&max-keys=999 2023-05-31T19:56:30.772 DEBUG:boto:auth_path=/pklxkd-9/?versions&max-keys=999 2023-05-31T19:56:30.772 DEBUG:boto:Method: GET 2023-05-31T19:56:30.772 DEBUG:boto:Path: /pklxkd-9/?versions&max-keys=999 2023-05-31T19:56:30.773 DEBUG:boto:Data: 2023-05-31T19:56:30.773 DEBUG:boto:Headers: {} 2023-05-31T19:56:30.773 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T19:56:30.774 DEBUG:boto:Port: 8001 2023-05-31T19:56:30.774 DEBUG:boto:Params: {} 2023-05-31T19:56:30.774 DEBUG:boto:Token: None 2023-05-31T19:56:30.775 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 19:56:30 GMT /pklxkd-9/?versions 2023-05-31T19:56:30.775 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:qMZ+lHgIqb/0oblgVUwTynaEJsg= 2023-05-31T19:56:30.776 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 19:56:30 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:qMZ+lHgIqb/0oblgVUwTynaEJsg=', 'Content-Length': '0'} 2023-05-31T19:56:30.778 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000bdb81f110f21bf5a-006477a66e-1112-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 19:56:30 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T19:56:30.778 DEBUG:boto:b'pklxkd-9999falsemyobjnulltrue2023-05-31T19:56:10.314Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T19:56:30.779 DEBUG:tasks.rgw_multi.zone_rados:o=myobj 2023-05-31T19:56:30.779 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2023-05-31T19:56:30.780 DEBUG:boto:path=/pklxkd-9/ 2023-05-31T19:56:30.780 DEBUG:boto:auth_path=/pklxkd-9/ 2023-05-31T19:56:30.781 DEBUG:boto:path=/pklxkd-9/?versions&max-keys=999 2023-05-31T19:56:30.781 DEBUG:boto:auth_path=/pklxkd-9/?versions&max-keys=999 2023-05-31T19:56:30.781 DEBUG:boto:Method: GET 2023-05-31T19:56:30.782 DEBUG:boto:Path: /pklxkd-9/?versions&max-keys=999 2023-05-31T19:56:30.782 DEBUG:boto:Data: 2023-05-31T19:56:30.782 DEBUG:boto:Headers: {} 2023-05-31T19:56:30.783 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T19:56:30.783 DEBUG:boto:Port: 8000 2023-05-31T19:56:30.783 DEBUG:boto:Params: {} 2023-05-31T19:56:30.784 DEBUG:boto:Token: None 2023-05-31T19:56:30.784 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 19:56:30 GMT /pklxkd-9/?versions 2023-05-31T19:56:30.784 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:qMZ+lHgIqb/0oblgVUwTynaEJsg= 2023-05-31T19:56:30.785 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 19:56:30 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:qMZ+lHgIqb/0oblgVUwTynaEJsg=', 'Content-Length': '0'} 2023-05-31T19:56:30.787 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d4965b27e41f3c1c-006477a66e-110e-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 19:56:30 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T19:56:30.787 DEBUG:boto:b'pklxkd-9999falsemyobjnulltrue2023-05-31T19:56:10.314Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T19:56:30.788 DEBUG:tasks.rgw_multi.zone_rados:o=myobj 2023-05-31T19:56:30.788 DEBUG:boto:path=/pklxkd-9/ 2023-05-31T19:56:30.789 DEBUG:boto:auth_path=/pklxkd-9/ 2023-05-31T19:56:30.789 DEBUG:boto:path=/pklxkd-9/?versions&max-keys=999 2023-05-31T19:56:30.790 DEBUG:boto:auth_path=/pklxkd-9/?versions&max-keys=999 2023-05-31T19:56:30.790 DEBUG:boto:Method: GET 2023-05-31T19:56:30.790 DEBUG:boto:Path: /pklxkd-9/?versions&max-keys=999 2023-05-31T19:56:30.791 DEBUG:boto:Data: 2023-05-31T19:56:30.791 DEBUG:boto:Headers: {} 2023-05-31T19:56:30.791 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T19:56:30.792 DEBUG:boto:Port: 8001 2023-05-31T19:56:30.792 DEBUG:boto:Params: {} 2023-05-31T19:56:30.792 DEBUG:boto:Token: None 2023-05-31T19:56:30.793 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 19:56:30 GMT /pklxkd-9/?versions 2023-05-31T19:56:30.793 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:qMZ+lHgIqb/0oblgVUwTynaEJsg= 2023-05-31T19:56:30.794 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 19:56:30 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:qMZ+lHgIqb/0oblgVUwTynaEJsg=', 'Content-Length': '0'} 2023-05-31T19:56:30.796 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000039bd73d2a3093284-006477a66e-1112-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 19:56:30 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T19:56:30.796 DEBUG:boto:b'pklxkd-9999falsemyobjnulltrue2023-05-31T19:56:10.314Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T19:56:30.797 DEBUG:boto:path=/pklxkd-9/ 2023-05-31T19:56:30.797 DEBUG:boto:auth_path=/pklxkd-9/ 2023-05-31T19:56:30.798 DEBUG:boto:path=/pklxkd-9/?versions&max-keys=999 2023-05-31T19:56:30.798 DEBUG:boto:auth_path=/pklxkd-9/?versions&max-keys=999 2023-05-31T19:56:30.798 DEBUG:boto:Method: GET 2023-05-31T19:56:30.799 DEBUG:boto:Path: /pklxkd-9/?versions&max-keys=999 2023-05-31T19:56:30.799 DEBUG:boto:Data: 2023-05-31T19:56:30.799 DEBUG:boto:Headers: {} 2023-05-31T19:56:30.800 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T19:56:30.800 DEBUG:boto:Port: 8000 2023-05-31T19:56:30.800 DEBUG:boto:Params: {} 2023-05-31T19:56:30.801 DEBUG:boto:Token: None 2023-05-31T19:56:30.801 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 19:56:30 GMT /pklxkd-9/?versions 2023-05-31T19:56:30.802 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:qMZ+lHgIqb/0oblgVUwTynaEJsg= 2023-05-31T19:56:30.802 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 19:56:30 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:qMZ+lHgIqb/0oblgVUwTynaEJsg=', 'Content-Length': '0'} 2023-05-31T19:56:30.804 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000006d36018cb4691060-006477a66e-110e-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 19:56:30 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T19:56:30.805 DEBUG:boto:b'pklxkd-9999falsemyobjnulltrue2023-05-31T19:56:10.314Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T19:56:30.805 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2023-05-31T19:56:30.806 DEBUG:boto:path=/pklxkd-9/myobj 2023-05-31T19:56:30.806 DEBUG:boto:auth_path=/pklxkd-9/myobj 2023-05-31T19:56:30.807 DEBUG:boto:path=/pklxkd-9/myobj?versionId=null 2023-05-31T19:56:30.807 DEBUG:boto:auth_path=/pklxkd-9/myobj?versionId=null 2023-05-31T19:56:30.807 DEBUG:boto:Method: GET 2023-05-31T19:56:30.808 DEBUG:boto:Path: /pklxkd-9/myobj?versionId=null 2023-05-31T19:56:30.808 DEBUG:boto:Data: 2023-05-31T19:56:30.808 DEBUG:boto:Headers: {} 2023-05-31T19:56:30.809 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T19:56:30.809 DEBUG:boto:Port: 8001 2023-05-31T19:56:30.810 DEBUG:boto:Params: {} 2023-05-31T19:56:30.810 DEBUG:boto:Token: None 2023-05-31T19:56:30.810 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 19:56:30 GMT /pklxkd-9/myobj?versionId=null 2023-05-31T19:56:30.811 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:3u+HwEh6EAo6cL4UzM2tYLGv1lk= 2023-05-31T19:56:30.811 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 19:56:30 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:3u+HwEh6EAo6cL4UzM2tYLGv1lk=', 'Content-Length': '0'} 2023-05-31T19:56:30.813 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 19:56:10 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000006d3a44c8640c33bd-006477a66e-1112-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 19:56:30 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T19:56:30.814 DEBUG:boto:path=/pklxkd-9/myobj 2023-05-31T19:56:30.814 DEBUG:boto:auth_path=/pklxkd-9/myobj 2023-05-31T19:56:30.815 DEBUG:boto:path=/pklxkd-9/myobj?versionId=null 2023-05-31T19:56:30.815 DEBUG:boto:auth_path=/pklxkd-9/myobj?versionId=null 2023-05-31T19:56:30.815 DEBUG:boto:Method: GET 2023-05-31T19:56:30.816 DEBUG:boto:Path: /pklxkd-9/myobj?versionId=null 2023-05-31T19:56:30.816 DEBUG:boto:Data: 2023-05-31T19:56:30.816 DEBUG:boto:Headers: {} 2023-05-31T19:56:30.817 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T19:56:30.817 DEBUG:boto:Port: 8000 2023-05-31T19:56:30.817 DEBUG:boto:Params: {} 2023-05-31T19:56:30.818 DEBUG:boto:Token: None 2023-05-31T19:56:30.818 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 19:56:30 GMT /pklxkd-9/myobj?versionId=null 2023-05-31T19:56:30.819 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:3u+HwEh6EAo6cL4UzM2tYLGv1lk= 2023-05-31T19:56:30.819 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 19:56:30 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:3u+HwEh6EAo6cL4UzM2tYLGv1lk=', 'Content-Length': '0'} 2023-05-31T19:56:30.821 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 19:56:10 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000a59d22dd4845e1f2-006477a66e-110e-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 19:56:30 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T19:56:30.821 DEBUG:boto:path=/pklxkd-9/myobj 2023-05-31T19:56:30.822 DEBUG:boto:auth_path=/pklxkd-9/myobj 2023-05-31T19:56:30.822 DEBUG:boto:path=/pklxkd-9/myobj?versionId=null 2023-05-31T19:56:30.822 DEBUG:boto:auth_path=/pklxkd-9/myobj?versionId=null 2023-05-31T19:56:30.823 DEBUG:boto:Method: HEAD 2023-05-31T19:56:30.823 DEBUG:boto:Path: /pklxkd-9/myobj?versionId=null 2023-05-31T19:56:30.823 DEBUG:boto:Data: 2023-05-31T19:56:30.824 DEBUG:boto:Headers: {} 2023-05-31T19:56:30.824 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T19:56:30.824 DEBUG:boto:Port: 8001 2023-05-31T19:56:30.825 DEBUG:boto:Params: {} 2023-05-31T19:56:30.825 DEBUG:boto:Token: None 2023-05-31T19:56:30.826 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 19:56:30 GMT /pklxkd-9/myobj?versionId=null 2023-05-31T19:56:30.826 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:R++qWijW9Ob+KWDuIf3lb9Wn4Fw= 2023-05-31T19:56:30.826 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 19:56:30 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:R++qWijW9Ob+KWDuIf3lb9Wn4Fw=', 'Content-Length': '0'} 2023-05-31T19:56:30.828 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 19:56:10 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000001d5c1bba1ed0d32d-006477a66e-1112-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 19:56:30 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T19:56:30.828 DEBUG:boto:path=/pklxkd-9/myobj 2023-05-31T19:56:30.829 DEBUG:boto:auth_path=/pklxkd-9/myobj 2023-05-31T19:56:30.829 DEBUG:boto:path=/pklxkd-9/myobj?versionId=null 2023-05-31T19:56:30.829 DEBUG:boto:auth_path=/pklxkd-9/myobj?versionId=null 2023-05-31T19:56:30.830 DEBUG:boto:Method: HEAD 2023-05-31T19:56:30.830 DEBUG:boto:Path: /pklxkd-9/myobj?versionId=null 2023-05-31T19:56:30.830 DEBUG:boto:Data: 2023-05-31T19:56:30.831 DEBUG:boto:Headers: {} 2023-05-31T19:56:30.831 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T19:56:30.831 DEBUG:boto:Port: 8000 2023-05-31T19:56:30.832 DEBUG:boto:Params: {} 2023-05-31T19:56:30.832 DEBUG:boto:Token: None 2023-05-31T19:56:30.833 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 19:56:30 GMT /pklxkd-9/myobj?versionId=null 2023-05-31T19:56:30.833 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:R++qWijW9Ob+KWDuIf3lb9Wn4Fw= 2023-05-31T19:56:30.833 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 19:56:30 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:R++qWijW9Ob+KWDuIf3lb9Wn4Fw=', 'Content-Length': '0'} 2023-05-31T19:56:30.835 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 19:56:10 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000610a39304839c3a7-006477a66e-110e-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 19:56:30 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T19:56:30.836 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2023-05-31T19:56:30.836 DEBUG:boto:path=/pklxkd-9/myobj 2023-05-31T19:56:30.836 DEBUG:boto:auth_path=/pklxkd-9/myobj 2023-05-31T19:56:30.837 DEBUG:boto:path=/pklxkd-9/myobj?versionId=null 2023-05-31T19:56:30.837 DEBUG:boto:auth_path=/pklxkd-9/myobj?versionId=null 2023-05-31T19:56:30.837 DEBUG:boto:Method: GET 2023-05-31T19:56:30.838 DEBUG:boto:Path: /pklxkd-9/myobj?versionId=null 2023-05-31T19:56:30.838 DEBUG:boto:Data: 2023-05-31T19:56:30.838 DEBUG:boto:Headers: {} 2023-05-31T19:56:30.839 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T19:56:30.839 DEBUG:boto:Port: 8001 2023-05-31T19:56:30.839 DEBUG:boto:Params: {} 2023-05-31T19:56:30.840 DEBUG:boto:Token: None 2023-05-31T19:56:30.840 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 19:56:30 GMT /pklxkd-9/myobj?versionId=null 2023-05-31T19:56:30.841 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:3u+HwEh6EAo6cL4UzM2tYLGv1lk= 2023-05-31T19:56:30.841 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 19:56:30 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:3u+HwEh6EAo6cL4UzM2tYLGv1lk=', 'Content-Length': '0'} 2023-05-31T19:56:30.843 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 19:56:10 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000044cc17a453190e91-006477a66e-1112-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 19:56:30 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T19:56:30.884 DEBUG:boto:path=/pklxkd-9/myobj 2023-05-31T19:56:30.885 DEBUG:boto:auth_path=/pklxkd-9/myobj 2023-05-31T19:56:30.885 DEBUG:boto:path=/pklxkd-9/myobj?versionId=null 2023-05-31T19:56:30.885 DEBUG:boto:auth_path=/pklxkd-9/myobj?versionId=null 2023-05-31T19:56:30.886 DEBUG:boto:Method: GET 2023-05-31T19:56:30.886 DEBUG:boto:Path: /pklxkd-9/myobj?versionId=null 2023-05-31T19:56:30.886 DEBUG:boto:Data: 2023-05-31T19:56:30.887 DEBUG:boto:Headers: {} 2023-05-31T19:56:30.887 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T19:56:30.888 DEBUG:boto:Port: 8000 2023-05-31T19:56:30.888 DEBUG:boto:Params: {} 2023-05-31T19:56:30.888 DEBUG:boto:Token: None 2023-05-31T19:56:30.889 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 19:56:30 GMT /pklxkd-9/myobj?versionId=null 2023-05-31T19:56:30.889 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:3u+HwEh6EAo6cL4UzM2tYLGv1lk= 2023-05-31T19:56:30.890 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 19:56:30 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:3u+HwEh6EAo6cL4UzM2tYLGv1lk=', 'Content-Length': '0'} 2023-05-31T19:56:30.891 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 19:56:10 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000052734913d3b9b6f3-006477a66e-110e-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 19:56:30 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T19:56:30.892 DEBUG:boto:path=/pklxkd-9/ 2023-05-31T19:56:30.892 DEBUG:boto:auth_path=/pklxkd-9/ 2023-05-31T19:56:30.893 DEBUG:boto:path=/pklxkd-9/?prefix=myobj 2023-05-31T19:56:30.893 DEBUG:boto:auth_path=/pklxkd-9/?prefix=myobj 2023-05-31T19:56:30.894 DEBUG:boto:Method: GET 2023-05-31T19:56:30.894 DEBUG:boto:Path: /pklxkd-9/?prefix=myobj 2023-05-31T19:56:30.894 DEBUG:boto:Data: 2023-05-31T19:56:30.895 DEBUG:boto:Headers: {} 2023-05-31T19:56:30.895 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T19:56:30.895 DEBUG:boto:Port: 8001 2023-05-31T19:56:30.896 DEBUG:boto:Params: {} 2023-05-31T19:56:30.896 DEBUG:boto:Token: None 2023-05-31T19:56:30.896 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 19:56:30 GMT /pklxkd-9/ 2023-05-31T19:56:30.897 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:cgODEfwBTlk1qWOESEU+TfXUQXM= 2023-05-31T19:56:30.897 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 19:56:30 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:cgODEfwBTlk1qWOESEU+TfXUQXM=', 'Content-Length': '0'} 2023-05-31T19:56:30.900 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e89d2b79aab17e5b-006477a66e-1112-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 19:56:30 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T19:56:30.900 DEBUG:boto:b'pklxkd-9myobj1000falsemyobj2023-05-31T19:56:10.314Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T19:56:30.901 DEBUG:boto:path=/pklxkd-9/ 2023-05-31T19:56:30.901 DEBUG:boto:auth_path=/pklxkd-9/ 2023-05-31T19:56:30.902 DEBUG:boto:path=/pklxkd-9/?prefix=myobj 2023-05-31T19:56:30.902 DEBUG:boto:auth_path=/pklxkd-9/?prefix=myobj 2023-05-31T19:56:30.902 DEBUG:boto:Method: GET 2023-05-31T19:56:30.903 DEBUG:boto:Path: /pklxkd-9/?prefix=myobj 2023-05-31T19:56:30.903 DEBUG:boto:Data: 2023-05-31T19:56:30.903 DEBUG:boto:Headers: {} 2023-05-31T19:56:30.904 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T19:56:30.904 DEBUG:boto:Port: 8000 2023-05-31T19:56:30.905 DEBUG:boto:Params: {} 2023-05-31T19:56:30.905 DEBUG:boto:Token: None 2023-05-31T19:56:30.905 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 19:56:30 GMT /pklxkd-9/ 2023-05-31T19:56:30.906 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:cgODEfwBTlk1qWOESEU+TfXUQXM= 2023-05-31T19:56:30.906 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 19:56:30 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:cgODEfwBTlk1qWOESEU+TfXUQXM=', 'Content-Length': '0'} 2023-05-31T19:56:30.908 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000df5af10a19a58221-006477a66e-110e-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 19:56:30 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T19:56:30.909 DEBUG:boto:b'pklxkd-9myobj1000falsemyobj2023-05-31T19:56:10.314Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T19:56:30.910 DEBUG:boto:path=/pklxkd-9/myobj 2023-05-31T19:56:30.910 DEBUG:boto:auth_path=/pklxkd-9/myobj 2023-05-31T19:56:30.910 DEBUG:boto:Method: HEAD 2023-05-31T19:56:30.911 DEBUG:boto:Path: /pklxkd-9/myobj 2023-05-31T19:56:30.911 DEBUG:boto:Data: 2023-05-31T19:56:30.911 DEBUG:boto:Headers: {} 2023-05-31T19:56:30.912 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T19:56:30.912 DEBUG:boto:Port: 8001 2023-05-31T19:56:30.912 DEBUG:boto:Params: {} 2023-05-31T19:56:30.913 DEBUG:boto:Token: None 2023-05-31T19:56:30.913 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 19:56:30 GMT /pklxkd-9/myobj 2023-05-31T19:56:30.913 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:udBVzf1GREf2Dj+n64S5pvJCU5A= 2023-05-31T19:56:30.914 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 19:56:30 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:udBVzf1GREf2Dj+n64S5pvJCU5A=', 'Content-Length': '0'} 2023-05-31T19:56:30.915 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 19:56:10 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000984d3da98ce3f742-006477a66e-1112-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 19:56:30 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T19:56:30.916 DEBUG:boto:path=/pklxkd-9/myobj 2023-05-31T19:56:30.916 DEBUG:boto:auth_path=/pklxkd-9/myobj 2023-05-31T19:56:30.917 DEBUG:boto:Method: HEAD 2023-05-31T19:56:30.917 DEBUG:boto:Path: /pklxkd-9/myobj 2023-05-31T19:56:30.917 DEBUG:boto:Data: 2023-05-31T19:56:30.918 DEBUG:boto:Headers: {} 2023-05-31T19:56:30.918 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T19:56:30.918 DEBUG:boto:Port: 8000 2023-05-31T19:56:30.919 DEBUG:boto:Params: {} 2023-05-31T19:56:30.919 DEBUG:boto:Token: None 2023-05-31T19:56:30.919 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 19:56:30 GMT /pklxkd-9/myobj 2023-05-31T19:56:30.920 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:udBVzf1GREf2Dj+n64S5pvJCU5A= 2023-05-31T19:56:30.920 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 19:56:30 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:udBVzf1GREf2Dj+n64S5pvJCU5A=', 'Content-Length': '0'} 2023-05-31T19:56:30.922 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 19:56:10 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000ba64d7c57b933bf8-006477a66e-110e-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 19:56:30 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T19:56:30.922 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2023-05-31T19:56:30.923 DEBUG:boto:path=/pklxkd-9/myobj 2023-05-31T19:56:30.923 DEBUG:boto:auth_path=/pklxkd-9/myobj 2023-05-31T19:56:30.923 DEBUG:boto:Method: GET 2023-05-31T19:56:30.924 DEBUG:boto:Path: /pklxkd-9/myobj 2023-05-31T19:56:30.924 DEBUG:boto:Data: 2023-05-31T19:56:30.924 DEBUG:boto:Headers: {} 2023-05-31T19:56:30.925 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T19:56:30.925 DEBUG:boto:Port: 8001 2023-05-31T19:56:30.926 DEBUG:boto:Params: {} 2023-05-31T19:56:30.926 DEBUG:boto:Token: None 2023-05-31T19:56:30.926 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 19:56:30 GMT /pklxkd-9/myobj 2023-05-31T19:56:30.927 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:7E6rTCB8YlrGsNzXBICo37JvTAQ= 2023-05-31T19:56:30.927 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 19:56:30 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:7E6rTCB8YlrGsNzXBICo37JvTAQ=', 'Content-Length': '0'} 2023-05-31T19:56:30.929 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 19:56:10 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000ba28d091b289cc58-006477a66e-1112-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 19:56:30 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T19:56:30.929 DEBUG:boto:path=/pklxkd-9/myobj 2023-05-31T19:56:30.930 DEBUG:boto:auth_path=/pklxkd-9/myobj 2023-05-31T19:56:30.930 DEBUG:boto:Method: GET 2023-05-31T19:56:30.931 DEBUG:boto:Path: /pklxkd-9/myobj 2023-05-31T19:56:30.931 DEBUG:boto:Data: 2023-05-31T19:56:30.931 DEBUG:boto:Headers: {} 2023-05-31T19:56:30.932 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T19:56:30.932 DEBUG:boto:Port: 8000 2023-05-31T19:56:30.932 DEBUG:boto:Params: {} 2023-05-31T19:56:30.933 DEBUG:boto:Token: None 2023-05-31T19:56:30.933 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 19:56:30 GMT /pklxkd-9/myobj 2023-05-31T19:56:30.934 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:7E6rTCB8YlrGsNzXBICo37JvTAQ= 2023-05-31T19:56:30.934 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 19:56:30 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:7E6rTCB8YlrGsNzXBICo37JvTAQ=', 'Content-Length': '0'} 2023-05-31T19:56:30.936 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 19:56:10 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000693dc4e6879e9252-006477a66e-110e-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 19:56:30 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T19:56:30.936 DEBUG:boto:path=/pklxkd-9/ 2023-05-31T19:56:30.937 DEBUG:boto:auth_path=/pklxkd-9/ 2023-05-31T19:56:30.937 DEBUG:boto:path=/pklxkd-9/?prefix=myobj 2023-05-31T19:56:30.937 DEBUG:boto:auth_path=/pklxkd-9/?prefix=myobj 2023-05-31T19:56:30.938 DEBUG:boto:Method: GET 2023-05-31T19:56:30.938 DEBUG:boto:Path: /pklxkd-9/?prefix=myobj 2023-05-31T19:56:30.939 DEBUG:boto:Data: 2023-05-31T19:56:30.939 DEBUG:boto:Headers: {} 2023-05-31T19:56:30.939 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T19:56:30.940 DEBUG:boto:Port: 8001 2023-05-31T19:56:30.940 DEBUG:boto:Params: {} 2023-05-31T19:56:30.940 DEBUG:boto:Token: None 2023-05-31T19:56:30.941 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 19:56:30 GMT /pklxkd-9/ 2023-05-31T19:56:30.941 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:cgODEfwBTlk1qWOESEU+TfXUQXM= 2023-05-31T19:56:30.942 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 19:56:30 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:cgODEfwBTlk1qWOESEU+TfXUQXM=', 'Content-Length': '0'} 2023-05-31T19:56:30.944 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e1b5ba25bcccb8fe-006477a66e-1112-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 19:56:30 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T19:56:30.944 DEBUG:boto:b'pklxkd-9myobj1000falsemyobj2023-05-31T19:56:10.314Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T19:56:30.945 DEBUG:boto:path=/pklxkd-9/ 2023-05-31T19:56:30.945 DEBUG:boto:auth_path=/pklxkd-9/ 2023-05-31T19:56:30.946 DEBUG:boto:path=/pklxkd-9/?prefix=myobj 2023-05-31T19:56:30.946 DEBUG:boto:auth_path=/pklxkd-9/?prefix=myobj 2023-05-31T19:56:30.946 DEBUG:boto:Method: GET 2023-05-31T19:56:30.947 DEBUG:boto:Path: /pklxkd-9/?prefix=myobj 2023-05-31T19:56:30.947 DEBUG:boto:Data: 2023-05-31T19:56:30.948 DEBUG:boto:Headers: {} 2023-05-31T19:56:30.948 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T19:56:30.948 DEBUG:boto:Port: 8000 2023-05-31T19:56:30.949 DEBUG:boto:Params: {} 2023-05-31T19:56:30.949 DEBUG:boto:Token: None 2023-05-31T19:56:30.949 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 19:56:30 GMT /pklxkd-9/ 2023-05-31T19:56:30.950 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:cgODEfwBTlk1qWOESEU+TfXUQXM= 2023-05-31T19:56:30.950 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 19:56:30 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:cgODEfwBTlk1qWOESEU+TfXUQXM=', 'Content-Length': '0'} 2023-05-31T19:56:30.952 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000012c133ff49751b3e-006477a66e-110e-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 19:56:30 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T19:56:30.953 DEBUG:boto:b'pklxkd-9myobj1000falsemyobj2023-05-31T19:56:10.314Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T19:56:30.953 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=pklxkd-9 zones={a2, a1} 2023-05-31T19:56:30.954 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'pklxkd-10', '--source-zone', 'a2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-05-31T19:56:30.954 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'pklxkd-10', '--source-zone', 'a2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-05-31T19:56:30.955 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket pklxkd-10 --source-zone a2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2023-05-31T19:56:31.017 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2023-05-31T19:56:31.017 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2023-05-31T19:56:31.067 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T19:56:31.062+0000 7f9a8f5c2780 1 bucket sync caught up with source: 2023-05-31T19:56:31.068 INFO:teuthology.orchestra.run.smithi099.stderr: local status: [, 00000000001.3.6, , , , , , , , , ] 2023-05-31T19:56:31.068 INFO:teuthology.orchestra.run.smithi099.stderr: remote markers: [, 00000000001.3.6, , , , , , , , , ] 2023-05-31T19:56:31.068 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T19:56:31.062+0000 7f9a8f5c2780 0 bucket checkpoint complete 2023-05-31T19:56:31.092 INFO:tasks.rgw_multi.zone_rados:comparing bucket=pklxkd-10 zones={a1, a2} 2023-05-31T19:56:31.093 DEBUG:boto:path=/pklxkd-10/ 2023-05-31T19:56:31.093 DEBUG:boto:auth_path=/pklxkd-10/ 2023-05-31T19:56:31.094 DEBUG:boto:Method: HEAD 2023-05-31T19:56:31.094 DEBUG:boto:Path: /pklxkd-10/ 2023-05-31T19:56:31.094 DEBUG:boto:Data: 2023-05-31T19:56:31.095 DEBUG:boto:Headers: {} 2023-05-31T19:56:31.095 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T19:56:31.095 DEBUG:boto:Port: 8000 2023-05-31T19:56:31.096 DEBUG:boto:Params: {} 2023-05-31T19:56:31.096 DEBUG:boto:Token: None 2023-05-31T19:56:31.097 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 19:56:31 GMT /pklxkd-10/ 2023-05-31T19:56:31.097 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:ZfjqDtx2+RZO0idxYDsd/bKv+/s= 2023-05-31T19:56:31.097 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 19:56:31 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:ZfjqDtx2+RZO0idxYDsd/bKv+/s=', 'Content-Length': '0'} 2023-05-31T19:56:31.100 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '6'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000006554091acaf4e156-006477a66f-110e-a1'), ('Content-Length', '0'), ('Date', 'Wed, 31 May 2023 19:56:31 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T19:56:31.100 DEBUG:boto:path=/pklxkd-10/ 2023-05-31T19:56:31.101 DEBUG:boto:auth_path=/pklxkd-10/ 2023-05-31T19:56:31.101 DEBUG:boto:Method: HEAD 2023-05-31T19:56:31.102 DEBUG:boto:Path: /pklxkd-10/ 2023-05-31T19:56:31.102 DEBUG:boto:Data: 2023-05-31T19:56:31.102 DEBUG:boto:Headers: {} 2023-05-31T19:56:31.103 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T19:56:31.103 DEBUG:boto:Port: 8001 2023-05-31T19:56:31.103 DEBUG:boto:Params: {} 2023-05-31T19:56:31.104 DEBUG:boto:Token: None 2023-05-31T19:56:31.104 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 19:56:31 GMT /pklxkd-10/ 2023-05-31T19:56:31.104 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:ZfjqDtx2+RZO0idxYDsd/bKv+/s= 2023-05-31T19:56:31.105 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 19:56:31 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:ZfjqDtx2+RZO0idxYDsd/bKv+/s=', 'Content-Length': '0'} 2023-05-31T19:56:31.107 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '6'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000003149de7f0c36039-006477a66f-1112-a2'), ('Content-Length', '0'), ('Date', 'Wed, 31 May 2023 19:56:31 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T19:56:31.107 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2023-05-31T19:56:31.107 DEBUG:boto:path=/pklxkd-10/ 2023-05-31T19:56:31.108 DEBUG:boto:auth_path=/pklxkd-10/ 2023-05-31T19:56:31.108 DEBUG:boto:path=/pklxkd-10/?versions&max-keys=999 2023-05-31T19:56:31.108 DEBUG:boto:auth_path=/pklxkd-10/?versions&max-keys=999 2023-05-31T19:56:31.109 DEBUG:boto:Method: GET 2023-05-31T19:56:31.109 DEBUG:boto:Path: /pklxkd-10/?versions&max-keys=999 2023-05-31T19:56:31.110 DEBUG:boto:Data: 2023-05-31T19:56:31.110 DEBUG:boto:Headers: {} 2023-05-31T19:56:31.110 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T19:56:31.111 DEBUG:boto:Port: 8000 2023-05-31T19:56:31.111 DEBUG:boto:Params: {} 2023-05-31T19:56:31.111 DEBUG:boto:Token: None 2023-05-31T19:56:31.112 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 19:56:31 GMT /pklxkd-10/?versions 2023-05-31T19:56:31.112 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:hhrTj7ftpwxVqr8QWYnwY53Nb0Y= 2023-05-31T19:56:31.112 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 19:56:31 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:hhrTj7ftpwxVqr8QWYnwY53Nb0Y=', 'Content-Length': '0'} 2023-05-31T19:56:31.114 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000247d9f3dd4cd2900-006477a66f-110e-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 19:56:31 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T19:56:31.115 DEBUG:boto:b'pklxkd-10999falsemyobjnulltrue2023-05-31T19:56:10.369Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T19:56:31.115 DEBUG:tasks.rgw_multi.zone_rados:o=myobj 2023-05-31T19:56:31.116 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2023-05-31T19:56:31.116 DEBUG:boto:path=/pklxkd-10/ 2023-05-31T19:56:31.116 DEBUG:boto:auth_path=/pklxkd-10/ 2023-05-31T19:56:31.117 DEBUG:boto:path=/pklxkd-10/?versions&max-keys=999 2023-05-31T19:56:31.117 DEBUG:boto:auth_path=/pklxkd-10/?versions&max-keys=999 2023-05-31T19:56:31.117 DEBUG:boto:Method: GET 2023-05-31T19:56:31.118 DEBUG:boto:Path: /pklxkd-10/?versions&max-keys=999 2023-05-31T19:56:31.118 DEBUG:boto:Data: 2023-05-31T19:56:31.118 DEBUG:boto:Headers: {} 2023-05-31T19:56:31.119 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T19:56:31.119 DEBUG:boto:Port: 8001 2023-05-31T19:56:31.119 DEBUG:boto:Params: {} 2023-05-31T19:56:31.120 DEBUG:boto:Token: None 2023-05-31T19:56:31.120 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 19:56:31 GMT /pklxkd-10/?versions 2023-05-31T19:56:31.121 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:hhrTj7ftpwxVqr8QWYnwY53Nb0Y= 2023-05-31T19:56:31.121 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 19:56:31 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:hhrTj7ftpwxVqr8QWYnwY53Nb0Y=', 'Content-Length': '0'} 2023-05-31T19:56:31.123 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000038e0987edef5cf59-006477a66f-1112-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 19:56:31 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T19:56:31.124 DEBUG:boto:b'pklxkd-10999falsemyobjnulltrue2023-05-31T19:56:10.369Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T19:56:31.124 DEBUG:tasks.rgw_multi.zone_rados:o=myobj 2023-05-31T19:56:31.125 DEBUG:boto:path=/pklxkd-10/ 2023-05-31T19:56:31.125 DEBUG:boto:auth_path=/pklxkd-10/ 2023-05-31T19:56:31.125 DEBUG:boto:path=/pklxkd-10/?versions&max-keys=999 2023-05-31T19:56:31.126 DEBUG:boto:auth_path=/pklxkd-10/?versions&max-keys=999 2023-05-31T19:56:31.126 DEBUG:boto:Method: GET 2023-05-31T19:56:31.126 DEBUG:boto:Path: /pklxkd-10/?versions&max-keys=999 2023-05-31T19:56:31.127 DEBUG:boto:Data: 2023-05-31T19:56:31.127 DEBUG:boto:Headers: {} 2023-05-31T19:56:31.127 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T19:56:31.128 DEBUG:boto:Port: 8000 2023-05-31T19:56:31.128 DEBUG:boto:Params: {} 2023-05-31T19:56:31.128 DEBUG:boto:Token: None 2023-05-31T19:56:31.129 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 19:56:31 GMT /pklxkd-10/?versions 2023-05-31T19:56:31.129 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:hhrTj7ftpwxVqr8QWYnwY53Nb0Y= 2023-05-31T19:56:31.130 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 19:56:31 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:hhrTj7ftpwxVqr8QWYnwY53Nb0Y=', 'Content-Length': '0'} 2023-05-31T19:56:31.132 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000009d23e5bdcaf30fa6-006477a66f-110e-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 19:56:31 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T19:56:31.132 DEBUG:boto:b'pklxkd-10999falsemyobjnulltrue2023-05-31T19:56:10.369Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T19:56:31.133 DEBUG:boto:path=/pklxkd-10/ 2023-05-31T19:56:31.133 DEBUG:boto:auth_path=/pklxkd-10/ 2023-05-31T19:56:31.134 DEBUG:boto:path=/pklxkd-10/?versions&max-keys=999 2023-05-31T19:56:31.134 DEBUG:boto:auth_path=/pklxkd-10/?versions&max-keys=999 2023-05-31T19:56:31.134 DEBUG:boto:Method: GET 2023-05-31T19:56:31.135 DEBUG:boto:Path: /pklxkd-10/?versions&max-keys=999 2023-05-31T19:56:31.135 DEBUG:boto:Data: 2023-05-31T19:56:31.135 DEBUG:boto:Headers: {} 2023-05-31T19:56:31.136 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T19:56:31.136 DEBUG:boto:Port: 8001 2023-05-31T19:56:31.136 DEBUG:boto:Params: {} 2023-05-31T19:56:31.137 DEBUG:boto:Token: None 2023-05-31T19:56:31.137 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 19:56:31 GMT /pklxkd-10/?versions 2023-05-31T19:56:31.137 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:hhrTj7ftpwxVqr8QWYnwY53Nb0Y= 2023-05-31T19:56:31.138 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 19:56:31 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:hhrTj7ftpwxVqr8QWYnwY53Nb0Y=', 'Content-Length': '0'} 2023-05-31T19:56:31.140 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000000faedb8302dfce9a-006477a66f-1112-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 19:56:31 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T19:56:31.140 DEBUG:boto:b'pklxkd-10999falsemyobjnulltrue2023-05-31T19:56:10.369Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T19:56:31.141 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2023-05-31T19:56:31.141 DEBUG:boto:path=/pklxkd-10/myobj 2023-05-31T19:56:31.142 DEBUG:boto:auth_path=/pklxkd-10/myobj 2023-05-31T19:56:31.142 DEBUG:boto:path=/pklxkd-10/myobj?versionId=null 2023-05-31T19:56:31.142 DEBUG:boto:auth_path=/pklxkd-10/myobj?versionId=null 2023-05-31T19:56:31.143 DEBUG:boto:Method: GET 2023-05-31T19:56:31.143 DEBUG:boto:Path: /pklxkd-10/myobj?versionId=null 2023-05-31T19:56:31.143 DEBUG:boto:Data: 2023-05-31T19:56:31.144 DEBUG:boto:Headers: {} 2023-05-31T19:56:31.144 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T19:56:31.144 DEBUG:boto:Port: 8000 2023-05-31T19:56:31.145 DEBUG:boto:Params: {} 2023-05-31T19:56:31.145 DEBUG:boto:Token: None 2023-05-31T19:56:31.146 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 19:56:31 GMT /pklxkd-10/myobj?versionId=null 2023-05-31T19:56:31.146 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:8U/TYsI5PY1g/v6sg8VBaUz7jJg= 2023-05-31T19:56:31.146 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 19:56:31 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:8U/TYsI5PY1g/v6sg8VBaUz7jJg=', 'Content-Length': '0'} 2023-05-31T19:56:31.148 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 19:56:10 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000cb807a061197b8f6-006477a66f-110e-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 19:56:31 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T19:56:31.149 DEBUG:boto:path=/pklxkd-10/myobj 2023-05-31T19:56:31.149 DEBUG:boto:auth_path=/pklxkd-10/myobj 2023-05-31T19:56:31.149 DEBUG:boto:path=/pklxkd-10/myobj?versionId=null 2023-05-31T19:56:31.150 DEBUG:boto:auth_path=/pklxkd-10/myobj?versionId=null 2023-05-31T19:56:31.150 DEBUG:boto:Method: GET 2023-05-31T19:56:31.150 DEBUG:boto:Path: /pklxkd-10/myobj?versionId=null 2023-05-31T19:56:31.151 DEBUG:boto:Data: 2023-05-31T19:56:31.151 DEBUG:boto:Headers: {} 2023-05-31T19:56:31.152 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T19:56:31.152 DEBUG:boto:Port: 8001 2023-05-31T19:56:31.152 DEBUG:boto:Params: {} 2023-05-31T19:56:31.153 DEBUG:boto:Token: None 2023-05-31T19:56:31.153 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 19:56:31 GMT /pklxkd-10/myobj?versionId=null 2023-05-31T19:56:31.153 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:8U/TYsI5PY1g/v6sg8VBaUz7jJg= 2023-05-31T19:56:31.154 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 19:56:31 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:8U/TYsI5PY1g/v6sg8VBaUz7jJg=', 'Content-Length': '0'} 2023-05-31T19:56:31.155 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 19:56:10 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000032c688c44560cdae-006477a66f-1112-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 19:56:31 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T19:56:31.156 DEBUG:boto:path=/pklxkd-10/myobj 2023-05-31T19:56:31.156 DEBUG:boto:auth_path=/pklxkd-10/myobj 2023-05-31T19:56:31.157 DEBUG:boto:path=/pklxkd-10/myobj?versionId=null 2023-05-31T19:56:31.157 DEBUG:boto:auth_path=/pklxkd-10/myobj?versionId=null 2023-05-31T19:56:31.157 DEBUG:boto:Method: HEAD 2023-05-31T19:56:31.158 DEBUG:boto:Path: /pklxkd-10/myobj?versionId=null 2023-05-31T19:56:31.158 DEBUG:boto:Data: 2023-05-31T19:56:31.158 DEBUG:boto:Headers: {} 2023-05-31T19:56:31.159 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T19:56:31.159 DEBUG:boto:Port: 8000 2023-05-31T19:56:31.159 DEBUG:boto:Params: {} 2023-05-31T19:56:31.160 DEBUG:boto:Token: None 2023-05-31T19:56:31.160 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 19:56:31 GMT /pklxkd-10/myobj?versionId=null 2023-05-31T19:56:31.160 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:iPkHk0ZyBY/uZyFZEJ9Ts3/wT2M= 2023-05-31T19:56:31.161 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 19:56:31 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:iPkHk0ZyBY/uZyFZEJ9Ts3/wT2M=', 'Content-Length': '0'} 2023-05-31T19:56:31.162 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 19:56:10 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000ae54439382f2c324-006477a66f-110e-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 19:56:31 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T19:56:31.163 DEBUG:boto:path=/pklxkd-10/myobj 2023-05-31T19:56:31.163 DEBUG:boto:auth_path=/pklxkd-10/myobj 2023-05-31T19:56:31.164 DEBUG:boto:path=/pklxkd-10/myobj?versionId=null 2023-05-31T19:56:31.164 DEBUG:boto:auth_path=/pklxkd-10/myobj?versionId=null 2023-05-31T19:56:31.164 DEBUG:boto:Method: HEAD 2023-05-31T19:56:31.165 DEBUG:boto:Path: /pklxkd-10/myobj?versionId=null 2023-05-31T19:56:31.165 DEBUG:boto:Data: 2023-05-31T19:56:31.165 DEBUG:boto:Headers: {} 2023-05-31T19:56:31.166 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T19:56:31.166 DEBUG:boto:Port: 8001 2023-05-31T19:56:31.166 DEBUG:boto:Params: {} 2023-05-31T19:56:31.167 DEBUG:boto:Token: None 2023-05-31T19:56:31.167 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 19:56:31 GMT /pklxkd-10/myobj?versionId=null 2023-05-31T19:56:31.167 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:iPkHk0ZyBY/uZyFZEJ9Ts3/wT2M= 2023-05-31T19:56:31.168 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 19:56:31 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:iPkHk0ZyBY/uZyFZEJ9Ts3/wT2M=', 'Content-Length': '0'} 2023-05-31T19:56:31.169 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 19:56:10 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000eed5f5870680d524-006477a66f-1112-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 19:56:31 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T19:56:31.170 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2023-05-31T19:56:31.170 DEBUG:boto:path=/pklxkd-10/myobj 2023-05-31T19:56:31.171 DEBUG:boto:auth_path=/pklxkd-10/myobj 2023-05-31T19:56:31.171 DEBUG:boto:path=/pklxkd-10/myobj?versionId=null 2023-05-31T19:56:31.171 DEBUG:boto:auth_path=/pklxkd-10/myobj?versionId=null 2023-05-31T19:56:31.172 DEBUG:boto:Method: GET 2023-05-31T19:56:31.172 DEBUG:boto:Path: /pklxkd-10/myobj?versionId=null 2023-05-31T19:56:31.172 DEBUG:boto:Data: 2023-05-31T19:56:31.173 DEBUG:boto:Headers: {} 2023-05-31T19:56:31.173 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T19:56:31.173 DEBUG:boto:Port: 8000 2023-05-31T19:56:31.174 DEBUG:boto:Params: {} 2023-05-31T19:56:31.174 DEBUG:boto:Token: None 2023-05-31T19:56:31.175 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 19:56:31 GMT /pklxkd-10/myobj?versionId=null 2023-05-31T19:56:31.175 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:8U/TYsI5PY1g/v6sg8VBaUz7jJg= 2023-05-31T19:56:31.175 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 19:56:31 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:8U/TYsI5PY1g/v6sg8VBaUz7jJg=', 'Content-Length': '0'} 2023-05-31T19:56:31.177 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 19:56:10 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000002a0ebf53b42bb712-006477a66f-110e-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 19:56:31 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T19:56:31.177 DEBUG:boto:path=/pklxkd-10/myobj 2023-05-31T19:56:31.178 DEBUG:boto:auth_path=/pklxkd-10/myobj 2023-05-31T19:56:31.178 DEBUG:boto:path=/pklxkd-10/myobj?versionId=null 2023-05-31T19:56:31.178 DEBUG:boto:auth_path=/pklxkd-10/myobj?versionId=null 2023-05-31T19:56:31.179 DEBUG:boto:Method: GET 2023-05-31T19:56:31.179 DEBUG:boto:Path: /pklxkd-10/myobj?versionId=null 2023-05-31T19:56:31.180 DEBUG:boto:Data: 2023-05-31T19:56:31.180 DEBUG:boto:Headers: {} 2023-05-31T19:56:31.180 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T19:56:31.181 DEBUG:boto:Port: 8001 2023-05-31T19:56:31.181 DEBUG:boto:Params: {} 2023-05-31T19:56:31.181 DEBUG:boto:Token: None 2023-05-31T19:56:31.182 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 19:56:31 GMT /pklxkd-10/myobj?versionId=null 2023-05-31T19:56:31.182 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:8U/TYsI5PY1g/v6sg8VBaUz7jJg= 2023-05-31T19:56:31.183 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 19:56:31 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:8U/TYsI5PY1g/v6sg8VBaUz7jJg=', 'Content-Length': '0'} 2023-05-31T19:56:31.184 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 19:56:10 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000962ff1c7f297a168-006477a66f-1112-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 19:56:31 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T19:56:31.185 DEBUG:boto:path=/pklxkd-10/ 2023-05-31T19:56:31.185 DEBUG:boto:auth_path=/pklxkd-10/ 2023-05-31T19:56:31.185 DEBUG:boto:path=/pklxkd-10/?prefix=myobj 2023-05-31T19:56:31.186 DEBUG:boto:auth_path=/pklxkd-10/?prefix=myobj 2023-05-31T19:56:31.186 DEBUG:boto:Method: GET 2023-05-31T19:56:31.187 DEBUG:boto:Path: /pklxkd-10/?prefix=myobj 2023-05-31T19:56:31.187 DEBUG:boto:Data: 2023-05-31T19:56:31.187 DEBUG:boto:Headers: {} 2023-05-31T19:56:31.188 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T19:56:31.188 DEBUG:boto:Port: 8000 2023-05-31T19:56:31.188 DEBUG:boto:Params: {} 2023-05-31T19:56:31.189 DEBUG:boto:Token: None 2023-05-31T19:56:31.189 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 19:56:31 GMT /pklxkd-10/ 2023-05-31T19:56:31.189 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:y3eWZUMnO75EAG4v1xRYNyudYBg= 2023-05-31T19:56:31.190 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 19:56:31 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:y3eWZUMnO75EAG4v1xRYNyudYBg=', 'Content-Length': '0'} 2023-05-31T19:56:31.192 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000008e71c78665ffde08-006477a66f-110e-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 19:56:31 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T19:56:31.232 DEBUG:boto:b'pklxkd-10myobj1000falsemyobj2023-05-31T19:56:10.369Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T19:56:31.233 DEBUG:boto:path=/pklxkd-10/ 2023-05-31T19:56:31.233 DEBUG:boto:auth_path=/pklxkd-10/ 2023-05-31T19:56:31.234 DEBUG:boto:path=/pklxkd-10/?prefix=myobj 2023-05-31T19:56:31.234 DEBUG:boto:auth_path=/pklxkd-10/?prefix=myobj 2023-05-31T19:56:31.234 DEBUG:boto:Method: GET 2023-05-31T19:56:31.235 DEBUG:boto:Path: /pklxkd-10/?prefix=myobj 2023-05-31T19:56:31.235 DEBUG:boto:Data: 2023-05-31T19:56:31.235 DEBUG:boto:Headers: {} 2023-05-31T19:56:31.236 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T19:56:31.236 DEBUG:boto:Port: 8001 2023-05-31T19:56:31.236 DEBUG:boto:Params: {} 2023-05-31T19:56:31.237 DEBUG:boto:Token: None 2023-05-31T19:56:31.237 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 19:56:31 GMT /pklxkd-10/ 2023-05-31T19:56:31.238 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:y3eWZUMnO75EAG4v1xRYNyudYBg= 2023-05-31T19:56:31.238 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 19:56:31 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:y3eWZUMnO75EAG4v1xRYNyudYBg=', 'Content-Length': '0'} 2023-05-31T19:56:31.240 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a99cd425c7e73e81-006477a66f-1112-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 19:56:31 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T19:56:31.241 DEBUG:boto:b'pklxkd-10myobj1000falsemyobj2023-05-31T19:56:10.369Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T19:56:31.241 DEBUG:boto:path=/pklxkd-10/myobj 2023-05-31T19:56:31.242 DEBUG:boto:auth_path=/pklxkd-10/myobj 2023-05-31T19:56:31.242 DEBUG:boto:Method: HEAD 2023-05-31T19:56:31.243 DEBUG:boto:Path: /pklxkd-10/myobj 2023-05-31T19:56:31.243 DEBUG:boto:Data: 2023-05-31T19:56:31.243 DEBUG:boto:Headers: {} 2023-05-31T19:56:31.243 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T19:56:31.244 DEBUG:boto:Port: 8000 2023-05-31T19:56:31.244 DEBUG:boto:Params: {} 2023-05-31T19:56:31.245 DEBUG:boto:Token: None 2023-05-31T19:56:31.245 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 19:56:31 GMT /pklxkd-10/myobj 2023-05-31T19:56:31.245 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:xOrf+I0iTahB4Q4p2mJ6AXHyj8I= 2023-05-31T19:56:31.246 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 19:56:31 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:xOrf+I0iTahB4Q4p2mJ6AXHyj8I=', 'Content-Length': '0'} 2023-05-31T19:56:31.247 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 19:56:10 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000ce706636b0ced37e-006477a66f-110e-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 19:56:31 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T19:56:31.248 DEBUG:boto:path=/pklxkd-10/myobj 2023-05-31T19:56:31.248 DEBUG:boto:auth_path=/pklxkd-10/myobj 2023-05-31T19:56:31.249 DEBUG:boto:Method: HEAD 2023-05-31T19:56:31.249 DEBUG:boto:Path: /pklxkd-10/myobj 2023-05-31T19:56:31.249 DEBUG:boto:Data: 2023-05-31T19:56:31.250 DEBUG:boto:Headers: {} 2023-05-31T19:56:31.250 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T19:56:31.250 DEBUG:boto:Port: 8001 2023-05-31T19:56:31.251 DEBUG:boto:Params: {} 2023-05-31T19:56:31.251 DEBUG:boto:Token: None 2023-05-31T19:56:31.251 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 19:56:31 GMT /pklxkd-10/myobj 2023-05-31T19:56:31.252 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:xOrf+I0iTahB4Q4p2mJ6AXHyj8I= 2023-05-31T19:56:31.252 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 19:56:31 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:xOrf+I0iTahB4Q4p2mJ6AXHyj8I=', 'Content-Length': '0'} 2023-05-31T19:56:31.254 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 19:56:10 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000fa7040d7d5a51975-006477a66f-1112-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 19:56:31 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T19:56:31.254 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2023-05-31T19:56:31.255 DEBUG:boto:path=/pklxkd-10/myobj 2023-05-31T19:56:31.255 DEBUG:boto:auth_path=/pklxkd-10/myobj 2023-05-31T19:56:31.255 DEBUG:boto:Method: GET 2023-05-31T19:56:31.256 DEBUG:boto:Path: /pklxkd-10/myobj 2023-05-31T19:56:31.256 DEBUG:boto:Data: 2023-05-31T19:56:31.256 DEBUG:boto:Headers: {} 2023-05-31T19:56:31.257 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T19:56:31.257 DEBUG:boto:Port: 8000 2023-05-31T19:56:31.257 DEBUG:boto:Params: {} 2023-05-31T19:56:31.258 DEBUG:boto:Token: None 2023-05-31T19:56:31.258 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 19:56:31 GMT /pklxkd-10/myobj 2023-05-31T19:56:31.259 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:5K9aPCCkKXzQO0t98Jdz0VGWOro= 2023-05-31T19:56:31.259 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 19:56:31 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:5K9aPCCkKXzQO0t98Jdz0VGWOro=', 'Content-Length': '0'} 2023-05-31T19:56:31.261 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 19:56:10 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000001e9388e7d5b4ac13-006477a66f-110e-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 19:56:31 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T19:56:31.261 DEBUG:boto:path=/pklxkd-10/myobj 2023-05-31T19:56:31.262 DEBUG:boto:auth_path=/pklxkd-10/myobj 2023-05-31T19:56:31.262 DEBUG:boto:Method: GET 2023-05-31T19:56:31.262 DEBUG:boto:Path: /pklxkd-10/myobj 2023-05-31T19:56:31.263 DEBUG:boto:Data: 2023-05-31T19:56:31.263 DEBUG:boto:Headers: {} 2023-05-31T19:56:31.263 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T19:56:31.264 DEBUG:boto:Port: 8001 2023-05-31T19:56:31.264 DEBUG:boto:Params: {} 2023-05-31T19:56:31.265 DEBUG:boto:Token: None 2023-05-31T19:56:31.265 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 19:56:31 GMT /pklxkd-10/myobj 2023-05-31T19:56:31.266 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:5K9aPCCkKXzQO0t98Jdz0VGWOro= 2023-05-31T19:56:31.266 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 19:56:31 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:5K9aPCCkKXzQO0t98Jdz0VGWOro=', 'Content-Length': '0'} 2023-05-31T19:56:31.268 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 19:56:10 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000cafc98a96f263e32-006477a66f-1112-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 19:56:31 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T19:56:31.312 DEBUG:boto:path=/pklxkd-10/ 2023-05-31T19:56:31.313 DEBUG:boto:auth_path=/pklxkd-10/ 2023-05-31T19:56:31.313 DEBUG:boto:path=/pklxkd-10/?prefix=myobj 2023-05-31T19:56:31.314 DEBUG:boto:auth_path=/pklxkd-10/?prefix=myobj 2023-05-31T19:56:31.314 DEBUG:boto:Method: GET 2023-05-31T19:56:31.314 DEBUG:boto:Path: /pklxkd-10/?prefix=myobj 2023-05-31T19:56:31.315 DEBUG:boto:Data: 2023-05-31T19:56:31.315 DEBUG:boto:Headers: {} 2023-05-31T19:56:31.316 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T19:56:31.316 DEBUG:boto:Port: 8000 2023-05-31T19:56:31.316 DEBUG:boto:Params: {} 2023-05-31T19:56:31.317 DEBUG:boto:Token: None 2023-05-31T19:56:31.317 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 19:56:31 GMT /pklxkd-10/ 2023-05-31T19:56:31.318 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:y3eWZUMnO75EAG4v1xRYNyudYBg= 2023-05-31T19:56:31.318 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 19:56:31 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:y3eWZUMnO75EAG4v1xRYNyudYBg=', 'Content-Length': '0'} 2023-05-31T19:56:31.320 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d2740d433858c85f-006477a66f-110e-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 19:56:31 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T19:56:31.321 DEBUG:boto:b'pklxkd-10myobj1000falsemyobj2023-05-31T19:56:10.369Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T19:56:31.321 DEBUG:boto:path=/pklxkd-10/ 2023-05-31T19:56:31.322 DEBUG:boto:auth_path=/pklxkd-10/ 2023-05-31T19:56:31.322 DEBUG:boto:path=/pklxkd-10/?prefix=myobj 2023-05-31T19:56:31.323 DEBUG:boto:auth_path=/pklxkd-10/?prefix=myobj 2023-05-31T19:56:31.323 DEBUG:boto:Method: GET 2023-05-31T19:56:31.323 DEBUG:boto:Path: /pklxkd-10/?prefix=myobj 2023-05-31T19:56:31.324 DEBUG:boto:Data: 2023-05-31T19:56:31.324 DEBUG:boto:Headers: {} 2023-05-31T19:56:31.324 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T19:56:31.325 DEBUG:boto:Port: 8001 2023-05-31T19:56:31.325 DEBUG:boto:Params: {} 2023-05-31T19:56:31.325 DEBUG:boto:Token: None 2023-05-31T19:56:31.326 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 19:56:31 GMT /pklxkd-10/ 2023-05-31T19:56:31.326 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:y3eWZUMnO75EAG4v1xRYNyudYBg= 2023-05-31T19:56:31.327 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 19:56:31 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:y3eWZUMnO75EAG4v1xRYNyudYBg=', 'Content-Length': '0'} 2023-05-31T19:56:31.329 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000003c06b189c6c9b0c2-006477a66f-1112-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 19:56:31 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T19:56:31.329 DEBUG:boto:b'pklxkd-10myobj1000falsemyobj2023-05-31T19:56:10.369Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T19:56:31.330 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=pklxkd-10 zones={a1, a2} 2023-05-31T19:56:31.330 DEBUG:boto:path=/pklxkd-9/ 2023-05-31T19:56:31.330 DEBUG:boto:auth_path=/pklxkd-9/ 2023-05-31T19:56:31.331 DEBUG:boto:Method: HEAD 2023-05-31T19:56:31.331 DEBUG:boto:Path: /pklxkd-9/ 2023-05-31T19:56:31.331 DEBUG:boto:Data: 2023-05-31T19:56:31.332 DEBUG:boto:Headers: {} 2023-05-31T19:56:31.332 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T19:56:31.332 DEBUG:boto:Port: 8000 2023-05-31T19:56:31.333 DEBUG:boto:Params: {} 2023-05-31T19:56:31.333 DEBUG:boto:Token: None 2023-05-31T19:56:31.334 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 19:56:31 GMT /pklxkd-9/ 2023-05-31T19:56:31.334 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:U3M5JnCj3YC393LwJwA6lDYK3X0= 2023-05-31T19:56:31.334 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 19:56:31 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:U3M5JnCj3YC393LwJwA6lDYK3X0=', 'Content-Length': '0'} 2023-05-31T19:56:31.336 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '6'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000003a9c2cf4dc271d3c-006477a66f-110e-a1'), ('Content-Length', '0'), ('Date', 'Wed, 31 May 2023 19:56:31 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T19:56:31.336 DEBUG:boto:path=/pklxkd-9/myobj 2023-05-31T19:56:31.337 DEBUG:boto:auth_path=/pklxkd-9/myobj 2023-05-31T19:56:31.337 DEBUG:boto:Method: HEAD 2023-05-31T19:56:31.338 DEBUG:boto:Path: /pklxkd-9/myobj 2023-05-31T19:56:31.338 DEBUG:boto:Data: 2023-05-31T19:56:31.338 DEBUG:boto:Headers: {} 2023-05-31T19:56:31.339 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T19:56:31.339 DEBUG:boto:Port: 8000 2023-05-31T19:56:31.339 DEBUG:boto:Params: {} 2023-05-31T19:56:31.339 DEBUG:boto:Token: None 2023-05-31T19:56:31.340 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 19:56:31 GMT /pklxkd-9/myobj 2023-05-31T19:56:31.340 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:UC7S5ZZcLrt/VZY5/EL5CRgcugg= 2023-05-31T19:56:31.341 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 19:56:31 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:UC7S5ZZcLrt/VZY5/EL5CRgcugg=', 'Content-Length': '0'} 2023-05-31T19:56:31.342 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 19:56:10 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000fd977cd952b139a0-006477a66f-110e-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 19:56:31 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T19:56:31.343 DEBUG:boto:path=/pklxkd-9/myobj 2023-05-31T19:56:31.343 DEBUG:boto:auth_path=/pklxkd-9/myobj 2023-05-31T19:56:31.344 DEBUG:boto:Method: DELETE 2023-05-31T19:56:31.344 DEBUG:boto:Path: /pklxkd-9/myobj 2023-05-31T19:56:31.344 DEBUG:boto:Data: 2023-05-31T19:56:31.345 DEBUG:boto:Headers: {} 2023-05-31T19:56:31.345 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T19:56:31.345 DEBUG:boto:Port: 8000 2023-05-31T19:56:31.346 DEBUG:boto:Params: {} 2023-05-31T19:56:31.346 DEBUG:boto:Token: None 2023-05-31T19:56:31.346 DEBUG:boto:StringToSign: DELETE Wed, 31 May 2023 19:56:31 GMT /pklxkd-9/myobj 2023-05-31T19:56:31.347 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:ZS2Hpy5H9z5lkZyga/SDGUuO8ZI= 2023-05-31T19:56:31.347 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 19:56:31 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:ZS2Hpy5H9z5lkZyga/SDGUuO8ZI=', 'Content-Length': '0'} 2023-05-31T19:56:31.352 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000e38df55beb9a7a7e-006477a66f-110e-a1'), ('Date', 'Wed, 31 May 2023 19:56:31 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T19:56:31.353 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'pklxkd-9', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-05-31T19:56:31.353 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'pklxkd-9', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-05-31T19:56:31.354 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket pklxkd-9 --source-zone a1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2023-05-31T19:56:31.410 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2023-05-31T19:56:31.411 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2023-05-31T19:56:31.467 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T19:56:31.461+0000 7f506a21c780 1 waiting for incremental sync to catch up: 2023-05-31T19:56:31.467 INFO:teuthology.orchestra.run.smithi099.stderr: local status: [, 00000000001.8.6, , , , , , , , , ] 2023-05-31T19:56:31.467 INFO:teuthology.orchestra.run.smithi099.stderr: remote markers: [, 00000000002.10.6, , , , , , , , , ] 2023-05-31T19:56:36.468 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T19:56:36.462+0000 7f506a21c780 1 waiting for incremental sync to catch up: 2023-05-31T19:56:36.468 INFO:teuthology.orchestra.run.smithi099.stderr: local status: [, 00000000001.8.6, , , , , , , , , ] 2023-05-31T19:56:36.469 INFO:teuthology.orchestra.run.smithi099.stderr: remote markers: [, 00000000002.10.6, , , , , , , , , ] 2023-05-31T19:56:41.469 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T19:56:41.464+0000 7f506a21c780 1 waiting for incremental sync to catch up: 2023-05-31T19:56:41.470 INFO:teuthology.orchestra.run.smithi099.stderr: local status: [, 00000000001.8.6, , , , , , , , , ] 2023-05-31T19:56:41.470 INFO:teuthology.orchestra.run.smithi099.stderr: remote markers: [, 00000000002.10.6, , , , , , , , , ] 2023-05-31T19:56:46.471 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T19:56:46.465+0000 7f506a21c780 1 bucket sync caught up with source: 2023-05-31T19:56:46.471 INFO:teuthology.orchestra.run.smithi099.stderr: local status: [, 00000000002.10.6, , , , , , , , , ] 2023-05-31T19:56:46.471 INFO:teuthology.orchestra.run.smithi099.stderr: remote markers: [, 00000000002.10.6, , , , , , , , , ] 2023-05-31T19:56:46.472 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T19:56:46.465+0000 7f506a21c780 0 bucket checkpoint complete 2023-05-31T19:56:46.493 INFO:tasks.rgw_multi.zone_rados:comparing bucket=pklxkd-9 zones={a2, a1} 2023-05-31T19:56:46.493 DEBUG:boto:path=/pklxkd-9/ 2023-05-31T19:56:46.494 DEBUG:boto:auth_path=/pklxkd-9/ 2023-05-31T19:56:46.495 DEBUG:boto:Method: HEAD 2023-05-31T19:56:46.495 DEBUG:boto:Path: /pklxkd-9/ 2023-05-31T19:56:46.495 DEBUG:boto:Data: 2023-05-31T19:56:46.496 DEBUG:boto:Headers: {} 2023-05-31T19:56:46.496 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T19:56:46.496 DEBUG:boto:Port: 8001 2023-05-31T19:56:46.497 DEBUG:boto:Params: {} 2023-05-31T19:56:46.497 DEBUG:boto:Token: None 2023-05-31T19:56:46.498 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 19:56:46 GMT /pklxkd-9/ 2023-05-31T19:56:46.498 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:LwggfTPtQXzEG1dQP+Lg1cxem+Q= 2023-05-31T19:56:46.499 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 19:56:46 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:LwggfTPtQXzEG1dQP+Lg1cxem+Q=', 'Content-Length': '0'} 2023-05-31T19:56:46.501 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000ef88641c08ab0bed-006477a67e-1112-a2'), ('Content-Length', '0'), ('Date', 'Wed, 31 May 2023 19:56:46 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T19:56:46.501 DEBUG:boto:path=/pklxkd-9/ 2023-05-31T19:56:46.502 DEBUG:boto:auth_path=/pklxkd-9/ 2023-05-31T19:56:46.503 DEBUG:boto:Method: HEAD 2023-05-31T19:56:46.504 DEBUG:boto:Path: /pklxkd-9/ 2023-05-31T19:56:46.504 DEBUG:boto:Data: 2023-05-31T19:56:46.505 DEBUG:boto:Headers: {} 2023-05-31T19:56:46.505 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T19:56:46.506 DEBUG:boto:Port: 8000 2023-05-31T19:56:46.506 DEBUG:boto:Params: {} 2023-05-31T19:56:46.506 DEBUG:boto:Token: None 2023-05-31T19:56:46.507 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 19:56:46 GMT /pklxkd-9/ 2023-05-31T19:56:46.507 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:LwggfTPtQXzEG1dQP+Lg1cxem+Q= 2023-05-31T19:56:46.508 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 19:56:46 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:LwggfTPtQXzEG1dQP+Lg1cxem+Q=', 'Content-Length': '0'} 2023-05-31T19:56:46.510 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000009ac0893741e17ad2-006477a67e-110e-a1'), ('Content-Length', '0'), ('Date', 'Wed, 31 May 2023 19:56:46 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T19:56:46.510 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2023-05-31T19:56:46.511 DEBUG:boto:path=/pklxkd-9/ 2023-05-31T19:56:46.511 DEBUG:boto:auth_path=/pklxkd-9/ 2023-05-31T19:56:46.511 DEBUG:boto:path=/pklxkd-9/?versions&max-keys=999 2023-05-31T19:56:46.512 DEBUG:boto:auth_path=/pklxkd-9/?versions&max-keys=999 2023-05-31T19:56:46.512 DEBUG:boto:Method: GET 2023-05-31T19:56:46.513 DEBUG:boto:Path: /pklxkd-9/?versions&max-keys=999 2023-05-31T19:56:46.513 DEBUG:boto:Data: 2023-05-31T19:56:46.513 DEBUG:boto:Headers: {} 2023-05-31T19:56:46.514 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T19:56:46.514 DEBUG:boto:Port: 8001 2023-05-31T19:56:46.514 DEBUG:boto:Params: {} 2023-05-31T19:56:46.515 DEBUG:boto:Token: None 2023-05-31T19:56:46.515 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 19:56:46 GMT /pklxkd-9/?versions 2023-05-31T19:56:46.515 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:puEZnPbH7DgcNk5eXFWDcGXrX/o= 2023-05-31T19:56:46.516 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 19:56:46 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:puEZnPbH7DgcNk5eXFWDcGXrX/o=', 'Content-Length': '0'} 2023-05-31T19:56:46.518 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000051072dad9874565c-006477a67e-1112-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 19:56:46 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T19:56:46.518 DEBUG:boto:b'pklxkd-9999false' 2023-05-31T19:56:46.519 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2023-05-31T19:56:46.519 DEBUG:boto:path=/pklxkd-9/ 2023-05-31T19:56:46.520 DEBUG:boto:auth_path=/pklxkd-9/ 2023-05-31T19:56:46.520 DEBUG:boto:path=/pklxkd-9/?versions&max-keys=999 2023-05-31T19:56:46.520 DEBUG:boto:auth_path=/pklxkd-9/?versions&max-keys=999 2023-05-31T19:56:46.521 DEBUG:boto:Method: GET 2023-05-31T19:56:46.521 DEBUG:boto:Path: /pklxkd-9/?versions&max-keys=999 2023-05-31T19:56:46.521 DEBUG:boto:Data: 2023-05-31T19:56:46.522 DEBUG:boto:Headers: {} 2023-05-31T19:56:46.522 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T19:56:46.522 DEBUG:boto:Port: 8000 2023-05-31T19:56:46.523 DEBUG:boto:Params: {} 2023-05-31T19:56:46.523 DEBUG:boto:Token: None 2023-05-31T19:56:46.524 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 19:56:46 GMT /pklxkd-9/?versions 2023-05-31T19:56:46.524 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:puEZnPbH7DgcNk5eXFWDcGXrX/o= 2023-05-31T19:56:46.525 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 19:56:46 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:puEZnPbH7DgcNk5eXFWDcGXrX/o=', 'Content-Length': '0'} 2023-05-31T19:56:46.527 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000747a209d29ff8419-006477a67e-110e-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 19:56:46 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T19:56:46.528 DEBUG:boto:b'pklxkd-9999false' 2023-05-31T19:56:46.528 DEBUG:boto:path=/pklxkd-9/ 2023-05-31T19:56:46.529 DEBUG:boto:auth_path=/pklxkd-9/ 2023-05-31T19:56:46.529 DEBUG:boto:path=/pklxkd-9/?versions&max-keys=999 2023-05-31T19:56:46.529 DEBUG:boto:auth_path=/pklxkd-9/?versions&max-keys=999 2023-05-31T19:56:46.530 DEBUG:boto:Method: GET 2023-05-31T19:56:46.530 DEBUG:boto:Path: /pklxkd-9/?versions&max-keys=999 2023-05-31T19:56:46.531 DEBUG:boto:Data: 2023-05-31T19:56:46.531 DEBUG:boto:Headers: {} 2023-05-31T19:56:46.531 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T19:56:46.532 DEBUG:boto:Port: 8001 2023-05-31T19:56:46.532 DEBUG:boto:Params: {} 2023-05-31T19:56:46.532 DEBUG:boto:Token: None 2023-05-31T19:56:46.533 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 19:56:46 GMT /pklxkd-9/?versions 2023-05-31T19:56:46.533 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:puEZnPbH7DgcNk5eXFWDcGXrX/o= 2023-05-31T19:56:46.534 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 19:56:46 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:puEZnPbH7DgcNk5eXFWDcGXrX/o=', 'Content-Length': '0'} 2023-05-31T19:56:46.536 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000005a198b039d869e2c-006477a67e-1112-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 19:56:46 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T19:56:46.536 DEBUG:boto:b'pklxkd-9999false' 2023-05-31T19:56:46.537 DEBUG:boto:path=/pklxkd-9/ 2023-05-31T19:56:46.538 DEBUG:boto:auth_path=/pklxkd-9/ 2023-05-31T19:56:46.538 DEBUG:boto:path=/pklxkd-9/?versions&max-keys=999 2023-05-31T19:56:46.538 DEBUG:boto:auth_path=/pklxkd-9/?versions&max-keys=999 2023-05-31T19:56:46.539 DEBUG:boto:Method: GET 2023-05-31T19:56:46.539 DEBUG:boto:Path: /pklxkd-9/?versions&max-keys=999 2023-05-31T19:56:46.539 DEBUG:boto:Data: 2023-05-31T19:56:46.540 DEBUG:boto:Headers: {} 2023-05-31T19:56:46.540 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T19:56:46.541 DEBUG:boto:Port: 8000 2023-05-31T19:56:46.541 DEBUG:boto:Params: {} 2023-05-31T19:56:46.542 DEBUG:boto:Token: None 2023-05-31T19:56:46.542 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 19:56:46 GMT /pklxkd-9/?versions 2023-05-31T19:56:46.543 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:puEZnPbH7DgcNk5eXFWDcGXrX/o= 2023-05-31T19:56:46.543 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 19:56:46 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:puEZnPbH7DgcNk5eXFWDcGXrX/o=', 'Content-Length': '0'} 2023-05-31T19:56:46.545 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000072bcc4fac35f1f7d-006477a67e-110e-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 19:56:46 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T19:56:46.546 DEBUG:boto:b'pklxkd-9999false' 2023-05-31T19:56:46.546 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=pklxkd-9 zones={a2, a1} 2023-05-31T19:56:46.547 DEBUG:boto:path=/pklxkd-10/ 2023-05-31T19:56:46.547 DEBUG:boto:auth_path=/pklxkd-10/ 2023-05-31T19:56:46.547 DEBUG:boto:Method: HEAD 2023-05-31T19:56:46.548 DEBUG:boto:Path: /pklxkd-10/ 2023-05-31T19:56:46.548 DEBUG:boto:Data: 2023-05-31T19:56:46.548 DEBUG:boto:Headers: {} 2023-05-31T19:56:46.549 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T19:56:46.549 DEBUG:boto:Port: 8001 2023-05-31T19:56:46.549 DEBUG:boto:Params: {} 2023-05-31T19:56:46.550 DEBUG:boto:Token: None 2023-05-31T19:56:46.550 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 19:56:46 GMT /pklxkd-10/ 2023-05-31T19:56:46.551 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:LBRKE54eyR+mjGWadIYL+apgfFg= 2023-05-31T19:56:46.551 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 19:56:46 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:LBRKE54eyR+mjGWadIYL+apgfFg=', 'Content-Length': '0'} 2023-05-31T19:56:46.553 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '6'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000ae96ad2f92301e80-006477a67e-1112-a2'), ('Content-Length', '0'), ('Date', 'Wed, 31 May 2023 19:56:46 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T19:56:46.553 DEBUG:boto:path=/pklxkd-10/myobj 2023-05-31T19:56:46.554 DEBUG:boto:auth_path=/pklxkd-10/myobj 2023-05-31T19:56:46.554 DEBUG:boto:Method: HEAD 2023-05-31T19:56:46.554 DEBUG:boto:Path: /pklxkd-10/myobj 2023-05-31T19:56:46.555 DEBUG:boto:Data: 2023-05-31T19:56:46.555 DEBUG:boto:Headers: {} 2023-05-31T19:56:46.555 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T19:56:46.556 DEBUG:boto:Port: 8001 2023-05-31T19:56:46.556 DEBUG:boto:Params: {} 2023-05-31T19:56:46.557 DEBUG:boto:Token: None 2023-05-31T19:56:46.557 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 19:56:46 GMT /pklxkd-10/myobj 2023-05-31T19:56:46.557 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:s36Btd3BBSL+Uztw8yCaHBsONGM= 2023-05-31T19:56:46.558 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 19:56:46 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:s36Btd3BBSL+Uztw8yCaHBsONGM=', 'Content-Length': '0'} 2023-05-31T19:56:46.559 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 19:56:10 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000023c6d3df15e50374-006477a67e-1112-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 19:56:46 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T19:56:46.560 DEBUG:boto:path=/pklxkd-10/myobj 2023-05-31T19:56:46.560 DEBUG:boto:auth_path=/pklxkd-10/myobj 2023-05-31T19:56:46.561 DEBUG:boto:Method: DELETE 2023-05-31T19:56:46.561 DEBUG:boto:Path: /pklxkd-10/myobj 2023-05-31T19:56:46.561 DEBUG:boto:Data: 2023-05-31T19:56:46.562 DEBUG:boto:Headers: {} 2023-05-31T19:56:46.562 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T19:56:46.562 DEBUG:boto:Port: 8001 2023-05-31T19:56:46.563 DEBUG:boto:Params: {} 2023-05-31T19:56:46.563 DEBUG:boto:Token: None 2023-05-31T19:56:46.563 DEBUG:boto:StringToSign: DELETE Wed, 31 May 2023 19:56:46 GMT /pklxkd-10/myobj 2023-05-31T19:56:46.564 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:lBH+0kBvFdOMKYsAcL9C8jZXx/Q= 2023-05-31T19:56:46.564 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 19:56:46 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:lBH+0kBvFdOMKYsAcL9C8jZXx/Q=', 'Content-Length': '0'} 2023-05-31T19:56:46.570 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000a93bd57e898e2cd8-006477a67e-1112-a2'), ('Date', 'Wed, 31 May 2023 19:56:46 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T19:56:46.570 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'pklxkd-10', '--source-zone', 'a2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-05-31T19:56:46.571 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'pklxkd-10', '--source-zone', 'a2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-05-31T19:56:46.571 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket pklxkd-10 --source-zone a2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2023-05-31T19:56:46.633 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2023-05-31T19:56:46.633 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2023-05-31T19:56:46.692 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T19:56:46.686+0000 7f66ef1e6780 1 waiting for incremental sync to catch up: 2023-05-31T19:56:46.692 INFO:teuthology.orchestra.run.smithi099.stderr: local status: [, 00000000001.3.6, , , , , , , , , ] 2023-05-31T19:56:46.692 INFO:teuthology.orchestra.run.smithi099.stderr: remote markers: [, 00000000002.7.6, , , , , , , , , ] 2023-05-31T19:56:51.693 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T19:56:51.687+0000 7f66ef1e6780 1 waiting for incremental sync to catch up: 2023-05-31T19:56:51.693 INFO:teuthology.orchestra.run.smithi099.stderr: local status: [, 00000000001.3.6, , , , , , , , , ] 2023-05-31T19:56:51.694 INFO:teuthology.orchestra.run.smithi099.stderr: remote markers: [, 00000000002.7.6, , , , , , , , , ] 2023-05-31T19:56:56.694 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T19:56:56.689+0000 7f66ef1e6780 1 waiting for incremental sync to catch up: 2023-05-31T19:56:56.694 INFO:teuthology.orchestra.run.smithi099.stderr: local status: [, 00000000001.3.6, , , , , , , , , ] 2023-05-31T19:56:56.695 INFO:teuthology.orchestra.run.smithi099.stderr: remote markers: [, 00000000002.7.6, , , , , , , , , ] 2023-05-31T19:57:01.696 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T19:57:01.691+0000 7f66ef1e6780 1 waiting for incremental sync to catch up: 2023-05-31T19:57:01.696 INFO:teuthology.orchestra.run.smithi099.stderr: local status: [, 00000000001.3.6, , , , , , , , , ] 2023-05-31T19:57:01.696 INFO:teuthology.orchestra.run.smithi099.stderr: remote markers: [, 00000000002.7.6, , , , , , , , , ] 2023-05-31T19:57:06.697 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T19:57:06.692+0000 7f66ef1e6780 1 bucket sync caught up with source: 2023-05-31T19:57:06.697 INFO:teuthology.orchestra.run.smithi099.stderr: local status: [, 00000000002.7.6, , , , , , , , , ] 2023-05-31T19:57:06.698 INFO:teuthology.orchestra.run.smithi099.stderr: remote markers: [, 00000000002.7.6, , , , , , , , , ] 2023-05-31T19:57:06.698 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T19:57:06.692+0000 7f66ef1e6780 0 bucket checkpoint complete 2023-05-31T19:57:06.715 INFO:tasks.rgw_multi.zone_rados:comparing bucket=pklxkd-10 zones={a1, a2} 2023-05-31T19:57:06.716 DEBUG:boto:path=/pklxkd-10/ 2023-05-31T19:57:06.716 DEBUG:boto:auth_path=/pklxkd-10/ 2023-05-31T19:57:06.716 DEBUG:boto:Method: HEAD 2023-05-31T19:57:06.717 DEBUG:boto:Path: /pklxkd-10/ 2023-05-31T19:57:06.717 DEBUG:boto:Data: 2023-05-31T19:57:06.717 DEBUG:boto:Headers: {} 2023-05-31T19:57:06.718 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T19:57:06.718 DEBUG:boto:Port: 8000 2023-05-31T19:57:06.718 DEBUG:boto:Params: {} 2023-05-31T19:57:06.719 DEBUG:boto:Token: None 2023-05-31T19:57:06.719 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 19:57:06 GMT /pklxkd-10/ 2023-05-31T19:57:06.720 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:4fsL77Zhc8chMr4JAfum/SPAtcI= 2023-05-31T19:57:06.720 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 19:57:06 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:4fsL77Zhc8chMr4JAfum/SPAtcI=', 'Content-Length': '0'} 2023-05-31T19:57:06.723 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000007f837fe42676b097-006477a692-110e-a1'), ('Content-Length', '0'), ('Date', 'Wed, 31 May 2023 19:57:06 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T19:57:06.723 DEBUG:boto:path=/pklxkd-10/ 2023-05-31T19:57:06.723 DEBUG:boto:auth_path=/pklxkd-10/ 2023-05-31T19:57:06.724 DEBUG:boto:Method: HEAD 2023-05-31T19:57:06.724 DEBUG:boto:Path: /pklxkd-10/ 2023-05-31T19:57:06.725 DEBUG:boto:Data: 2023-05-31T19:57:06.725 DEBUG:boto:Headers: {} 2023-05-31T19:57:06.725 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T19:57:06.726 DEBUG:boto:Port: 8001 2023-05-31T19:57:06.726 DEBUG:boto:Params: {} 2023-05-31T19:57:06.726 DEBUG:boto:Token: None 2023-05-31T19:57:06.727 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 19:57:06 GMT /pklxkd-10/ 2023-05-31T19:57:06.727 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:4fsL77Zhc8chMr4JAfum/SPAtcI= 2023-05-31T19:57:06.728 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 19:57:06 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:4fsL77Zhc8chMr4JAfum/SPAtcI=', 'Content-Length': '0'} 2023-05-31T19:57:06.730 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000005f648b788670a46c-006477a692-1112-a2'), ('Content-Length', '0'), ('Date', 'Wed, 31 May 2023 19:57:06 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T19:57:06.730 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2023-05-31T19:57:06.731 DEBUG:boto:path=/pklxkd-10/ 2023-05-31T19:57:06.731 DEBUG:boto:auth_path=/pklxkd-10/ 2023-05-31T19:57:06.731 DEBUG:boto:path=/pklxkd-10/?versions&max-keys=999 2023-05-31T19:57:06.732 DEBUG:boto:auth_path=/pklxkd-10/?versions&max-keys=999 2023-05-31T19:57:06.732 DEBUG:boto:Method: GET 2023-05-31T19:57:06.733 DEBUG:boto:Path: /pklxkd-10/?versions&max-keys=999 2023-05-31T19:57:06.733 DEBUG:boto:Data: 2023-05-31T19:57:06.733 DEBUG:boto:Headers: {} 2023-05-31T19:57:06.734 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T19:57:06.734 DEBUG:boto:Port: 8000 2023-05-31T19:57:06.734 DEBUG:boto:Params: {} 2023-05-31T19:57:06.735 DEBUG:boto:Token: None 2023-05-31T19:57:06.735 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 19:57:06 GMT /pklxkd-10/?versions 2023-05-31T19:57:06.736 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:0vgmpFpX8VxdM7OEhrXKmLg8IRE= 2023-05-31T19:57:06.736 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 19:57:06 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:0vgmpFpX8VxdM7OEhrXKmLg8IRE=', 'Content-Length': '0'} 2023-05-31T19:57:06.738 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000047fc55746eaca19c-006477a692-110e-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 19:57:06 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T19:57:06.739 DEBUG:boto:b'pklxkd-10999false' 2023-05-31T19:57:06.739 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2023-05-31T19:57:06.740 DEBUG:boto:path=/pklxkd-10/ 2023-05-31T19:57:06.740 DEBUG:boto:auth_path=/pklxkd-10/ 2023-05-31T19:57:06.741 DEBUG:boto:path=/pklxkd-10/?versions&max-keys=999 2023-05-31T19:57:06.741 DEBUG:boto:auth_path=/pklxkd-10/?versions&max-keys=999 2023-05-31T19:57:06.741 DEBUG:boto:Method: GET 2023-05-31T19:57:06.742 DEBUG:boto:Path: /pklxkd-10/?versions&max-keys=999 2023-05-31T19:57:06.742 DEBUG:boto:Data: 2023-05-31T19:57:06.742 DEBUG:boto:Headers: {} 2023-05-31T19:57:06.743 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T19:57:06.743 DEBUG:boto:Port: 8001 2023-05-31T19:57:06.743 DEBUG:boto:Params: {} 2023-05-31T19:57:06.744 DEBUG:boto:Token: None 2023-05-31T19:57:06.744 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 19:57:06 GMT /pklxkd-10/?versions 2023-05-31T19:57:06.745 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:0vgmpFpX8VxdM7OEhrXKmLg8IRE= 2023-05-31T19:57:06.745 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 19:57:06 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:0vgmpFpX8VxdM7OEhrXKmLg8IRE=', 'Content-Length': '0'} 2023-05-31T19:57:06.747 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000006b7336d4045e0bbd-006477a692-1112-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 19:57:06 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T19:57:06.748 DEBUG:boto:b'pklxkd-10999false' 2023-05-31T19:57:06.748 DEBUG:boto:path=/pklxkd-10/ 2023-05-31T19:57:06.749 DEBUG:boto:auth_path=/pklxkd-10/ 2023-05-31T19:57:06.749 DEBUG:boto:path=/pklxkd-10/?versions&max-keys=999 2023-05-31T19:57:06.749 DEBUG:boto:auth_path=/pklxkd-10/?versions&max-keys=999 2023-05-31T19:57:06.750 DEBUG:boto:Method: GET 2023-05-31T19:57:06.750 DEBUG:boto:Path: /pklxkd-10/?versions&max-keys=999 2023-05-31T19:57:06.750 DEBUG:boto:Data: 2023-05-31T19:57:06.751 DEBUG:boto:Headers: {} 2023-05-31T19:57:06.751 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T19:57:06.752 DEBUG:boto:Port: 8000 2023-05-31T19:57:06.752 DEBUG:boto:Params: {} 2023-05-31T19:57:06.752 DEBUG:boto:Token: None 2023-05-31T19:57:06.753 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 19:57:06 GMT /pklxkd-10/?versions 2023-05-31T19:57:06.753 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:0vgmpFpX8VxdM7OEhrXKmLg8IRE= 2023-05-31T19:57:06.754 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 19:57:06 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:0vgmpFpX8VxdM7OEhrXKmLg8IRE=', 'Content-Length': '0'} 2023-05-31T19:57:06.756 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000004c7546751cf591d7-006477a692-110e-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 19:57:06 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T19:57:06.756 DEBUG:boto:b'pklxkd-10999false' 2023-05-31T19:57:06.757 DEBUG:boto:path=/pklxkd-10/ 2023-05-31T19:57:06.757 DEBUG:boto:auth_path=/pklxkd-10/ 2023-05-31T19:57:06.758 DEBUG:boto:path=/pklxkd-10/?versions&max-keys=999 2023-05-31T19:57:06.758 DEBUG:boto:auth_path=/pklxkd-10/?versions&max-keys=999 2023-05-31T19:57:06.758 DEBUG:boto:Method: GET 2023-05-31T19:57:06.759 DEBUG:boto:Path: /pklxkd-10/?versions&max-keys=999 2023-05-31T19:57:06.759 DEBUG:boto:Data: 2023-05-31T19:57:06.759 DEBUG:boto:Headers: {} 2023-05-31T19:57:06.760 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T19:57:06.760 DEBUG:boto:Port: 8001 2023-05-31T19:57:06.761 DEBUG:boto:Params: {} 2023-05-31T19:57:06.761 DEBUG:boto:Token: None 2023-05-31T19:57:06.762 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 19:57:06 GMT /pklxkd-10/?versions 2023-05-31T19:57:06.762 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:0vgmpFpX8VxdM7OEhrXKmLg8IRE= 2023-05-31T19:57:06.762 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 19:57:06 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:0vgmpFpX8VxdM7OEhrXKmLg8IRE=', 'Content-Length': '0'} 2023-05-31T19:57:06.764 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e0b8532847d93a33-006477a692-1112-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 19:57:06 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T19:57:06.765 DEBUG:boto:b'pklxkd-10999false' 2023-05-31T19:57:06.765 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=pklxkd-10 zones={a1, a2} 2023-05-31T19:57:06.766 INFO:tasks.rgw_multisite_tests:rgw_multi.tests.test_object_delete ... ok 2023-05-31T19:57:06.767 INFO:rgw_multi.tests:create bucket zone=a1 name=pklxkd-11 2023-05-31T19:57:06.767 DEBUG:boto:path=/pklxkd-11/ 2023-05-31T19:57:06.768 DEBUG:boto:auth_path=/pklxkd-11/ 2023-05-31T19:57:06.768 DEBUG:boto:Method: PUT 2023-05-31T19:57:06.768 DEBUG:boto:Path: /pklxkd-11/ 2023-05-31T19:57:06.769 DEBUG:boto:Data: 2023-05-31T19:57:06.769 DEBUG:boto:Headers: {} 2023-05-31T19:57:06.769 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T19:57:06.770 DEBUG:boto:Port: 8000 2023-05-31T19:57:06.770 DEBUG:boto:Params: {} 2023-05-31T19:57:06.770 DEBUG:boto:Token: None 2023-05-31T19:57:06.771 DEBUG:boto:StringToSign: PUT Wed, 31 May 2023 19:57:06 GMT /pklxkd-11/ 2023-05-31T19:57:06.771 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:U+gPdwm+pahXFW6oOru6P1uJ/LY= 2023-05-31T19:57:06.772 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 19:57:06 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:U+gPdwm+pahXFW6oOru6P1uJ/LY=', 'Content-Length': '0'} 2023-05-31T19:57:06.786 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx0000035b2dfcf0e3c5936-006477a692-110e-a1'), ('Content-Length', '0'), ('Date', 'Wed, 31 May 2023 19:57:06 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T19:57:06.786 INFO:rgw_multi.tests:create bucket zone=a2 name=pklxkd-12 2023-05-31T19:57:06.787 DEBUG:boto:path=/pklxkd-12/ 2023-05-31T19:57:06.787 DEBUG:boto:auth_path=/pklxkd-12/ 2023-05-31T19:57:06.787 DEBUG:boto:Method: PUT 2023-05-31T19:57:06.788 DEBUG:boto:Path: /pklxkd-12/ 2023-05-31T19:57:06.788 DEBUG:boto:Data: 2023-05-31T19:57:06.789 DEBUG:boto:Headers: {} 2023-05-31T19:57:06.789 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T19:57:06.789 DEBUG:boto:Port: 8001 2023-05-31T19:57:06.790 DEBUG:boto:Params: {} 2023-05-31T19:57:06.790 DEBUG:boto:Token: None 2023-05-31T19:57:06.790 DEBUG:boto:StringToSign: PUT Wed, 31 May 2023 19:57:06 GMT /pklxkd-12/ 2023-05-31T19:57:06.791 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:73hkroyuVyfiEe7P3ND3cRuOYPU= 2023-05-31T19:57:06.791 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 19:57:06 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:73hkroyuVyfiEe7P3ND3cRuOYPU=', 'Content-Length': '0'} 2023-05-31T19:57:06.815 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000783a8675b3bb03ad-006477a692-1112-a2'), ('Content-Length', '0'), ('Date', 'Wed, 31 May 2023 19:57:06 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T19:57:06.816 DEBUG:boto:path=/pklxkd-11/ 2023-05-31T19:57:06.816 DEBUG:boto:auth_path=/pklxkd-11/ 2023-05-31T19:57:06.816 DEBUG:boto:path=/pklxkd-11/?versioning 2023-05-31T19:57:06.817 DEBUG:boto:auth_path=/pklxkd-11/?versioning 2023-05-31T19:57:06.817 DEBUG:boto:Method: PUT 2023-05-31T19:57:06.817 DEBUG:boto:Path: /pklxkd-11/?versioning 2023-05-31T19:57:06.818 DEBUG:boto:Data: Enabled Disabled 2023-05-31T19:57:06.818 DEBUG:boto:Headers: {} 2023-05-31T19:57:06.818 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T19:57:06.819 DEBUG:boto:Port: 8000 2023-05-31T19:57:06.819 DEBUG:boto:Params: {} 2023-05-31T19:57:06.819 DEBUG:boto:Token: None 2023-05-31T19:57:06.820 DEBUG:boto:StringToSign: PUT Wed, 31 May 2023 19:57:06 GMT /pklxkd-11/?versioning 2023-05-31T19:57:06.820 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:Px3EanbsiDmkgXTmnZFBfJmxna4= 2023-05-31T19:57:06.821 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 19:57:06 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:Px3EanbsiDmkgXTmnZFBfJmxna4=', 'Content-Length': '228'} 2023-05-31T19:57:06.826 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx000007b57b08d86b33829-006477a692-110e-a1'), ('Content-Type', 'application/xml'), ('Content-Length', '0'), ('Date', 'Wed, 31 May 2023 19:57:06 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T19:57:06.826 DEBUG:boto:path=/pklxkd-12/ 2023-05-31T19:57:06.826 DEBUG:boto:auth_path=/pklxkd-12/ 2023-05-31T19:57:06.827 DEBUG:boto:path=/pklxkd-12/?versioning 2023-05-31T19:57:06.827 DEBUG:boto:auth_path=/pklxkd-12/?versioning 2023-05-31T19:57:06.827 DEBUG:boto:Method: PUT 2023-05-31T19:57:06.828 DEBUG:boto:Path: /pklxkd-12/?versioning 2023-05-31T19:57:06.828 DEBUG:boto:Data: Enabled Disabled 2023-05-31T19:57:06.828 DEBUG:boto:Headers: {} 2023-05-31T19:57:06.829 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T19:57:06.829 DEBUG:boto:Port: 8001 2023-05-31T19:57:06.829 DEBUG:boto:Params: {} 2023-05-31T19:57:06.830 DEBUG:boto:Token: None 2023-05-31T19:57:06.830 DEBUG:boto:StringToSign: PUT Wed, 31 May 2023 19:57:06 GMT /pklxkd-12/?versioning 2023-05-31T19:57:06.831 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:i5cjpWEQL0v+u3E8HVKKhdn+WDo= 2023-05-31T19:57:06.831 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 19:57:06 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:i5cjpWEQL0v+u3E8HVKKhdn+WDo=', 'Content-Length': '228'} 2023-05-31T19:57:06.840 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000c93a00bf5929531a-006477a692-1112-a2'), ('Content-Type', 'application/xml'), ('Content-Length', '0'), ('Date', 'Wed, 31 May 2023 19:57:06 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T19:57:06.841 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-31T19:57:06.841 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-31T19:57:06.842 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-05-31T19:57:06.892 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2023-05-31T19:57:06.893 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2023-05-31T19:57:06.948 INFO:teuthology.orchestra.run.smithi099.stderr:No --period given, using current period=2e6123d2-d695-4350-9a01-61a455bc782b 2023-05-31T19:57:06.980 INFO:teuthology.orchestra.run.smithi099.stdout:[ 2023-05-31T19:57:06.980 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T19:57:06.981 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1685562631.516089_59.1", 2023-05-31T19:57:06.981 INFO:teuthology.orchestra.run.smithi099.stdout: "last_update": "2023-05-31T19:50:31.516089Z" 2023-05-31T19:57:06.981 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T19:57:06.981 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T19:57:06.981 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "", 2023-05-31T19:57:06.981 INFO:teuthology.orchestra.run.smithi099.stdout: "last_update": "0.000000" 2023-05-31T19:57:06.982 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T19:57:06.982 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T19:57:06.982 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1685562970.274572_93.1", 2023-05-31T19:57:06.982 INFO:teuthology.orchestra.run.smithi099.stdout: "last_update": "2023-05-31T19:56:10.274572Z" 2023-05-31T19:57:06.982 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T19:57:06.982 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T19:57:06.982 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1685563026.832102_54.1", 2023-05-31T19:57:06.983 INFO:teuthology.orchestra.run.smithi099.stdout: "last_update": "2023-05-31T19:57:06.832102Z" 2023-05-31T19:57:06.983 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T19:57:06.983 INFO:teuthology.orchestra.run.smithi099.stdout:] 2023-05-31T19:57:06.984 DEBUG:rgw_multi.tests:master meta markers={0: '1_1685562631.516089_59.1', 1: '', 2: '1_1685562970.274572_93.1', 3: '1_1685563026.832102_54.1'} 2023-05-31T19:57:06.984 INFO:rgw_multi.tests:starting meta checkpoint for zone=a2 2023-05-31T19:57:06.985 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-31T19:57:06.985 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-31T19:57:06.986 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-05-31T19:57:07.034 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2023-05-31T19:57:07.034 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2023-05-31T19:57:07.080 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T19:57:07.076+0000 7f835f480780 20 RGW-SYNC:meta: read sync status 2023-05-31T19:57:07.081 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T19:57:07.077+0000 7f835f480780 20 RGW-SYNC:meta: read sync status 2023-05-31T19:57:07.093 INFO:teuthology.orchestra.run.smithi099.stdout:{ 2023-05-31T19:57:07.093 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_status": { 2023-05-31T19:57:07.093 INFO:teuthology.orchestra.run.smithi099.stdout: "info": { 2023-05-31T19:57:07.093 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "sync", 2023-05-31T19:57:07.093 INFO:teuthology.orchestra.run.smithi099.stdout: "num_shards": 4, 2023-05-31T19:57:07.093 INFO:teuthology.orchestra.run.smithi099.stdout: "period": "2e6123d2-d695-4350-9a01-61a455bc782b", 2023-05-31T19:57:07.094 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2023-05-31T19:57:07.094 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T19:57:07.094 INFO:teuthology.orchestra.run.smithi099.stdout: "markers": [ 2023-05-31T19:57:07.094 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T19:57:07.094 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 0, 2023-05-31T19:57:07.094 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T19:57:07.095 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2023-05-31T19:57:07.095 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1685562631.516089_59.1", 2023-05-31T19:57:07.095 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T19:57:07.095 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 1, 2023-05-31T19:57:07.095 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T19:57:07.095 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2023-05-31T19:50:31.516089Z", 2023-05-31T19:57:07.095 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2023-05-31T19:57:07.096 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T19:57:07.096 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T19:57:07.096 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T19:57:07.096 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 1, 2023-05-31T19:57:07.096 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T19:57:07.096 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2023-05-31T19:57:07.096 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "", 2023-05-31T19:57:07.097 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T19:57:07.097 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2023-05-31T19:57:07.097 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T19:57:07.097 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "0.000000", 2023-05-31T19:57:07.097 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2023-05-31T19:57:07.097 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T19:57:07.097 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T19:57:07.098 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T19:57:07.098 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 2, 2023-05-31T19:57:07.098 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T19:57:07.098 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2023-05-31T19:57:07.098 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1685562970.274572_93.1", 2023-05-31T19:57:07.098 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T19:57:07.098 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2023-05-31T19:57:07.099 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T19:57:07.099 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2023-05-31T19:56:10.274572Z", 2023-05-31T19:57:07.099 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2023-05-31T19:57:07.099 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T19:57:07.099 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T19:57:07.099 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T19:57:07.100 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 3, 2023-05-31T19:57:07.100 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T19:57:07.100 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2023-05-31T19:57:07.100 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1685562970.292149_34.1", 2023-05-31T19:57:07.100 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T19:57:07.100 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2023-05-31T19:57:07.100 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T19:57:07.101 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2023-05-31T19:56:10.292149Z", 2023-05-31T19:57:07.101 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2023-05-31T19:57:07.101 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T19:57:07.101 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T19:57:07.101 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2023-05-31T19:57:07.101 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T19:57:07.101 INFO:teuthology.orchestra.run.smithi099.stdout: "full_sync": { 2023-05-31T19:57:07.102 INFO:teuthology.orchestra.run.smithi099.stdout: "total": 1, 2023-05-31T19:57:07.102 INFO:teuthology.orchestra.run.smithi099.stdout: "complete": 1 2023-05-31T19:57:07.102 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T19:57:07.102 INFO:teuthology.orchestra.run.smithi099.stdout: "current_time": "2023-05-31T19:57:07Z" 2023-05-31T19:57:07.102 INFO:teuthology.orchestra.run.smithi099.stdout:} 2023-05-31T19:57:07.103 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "2e6123d2-d695-4350-9a01-61a455bc782b", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1685562631.516089_59.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-05-31T19:50:31.516089Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1685562970.274572_93.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-05-31T19:56:10.274572Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1685562970.292149_34.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-05-31T19:56:10.292149Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-05-31T19:57:07Z" } 2023-05-31T19:57:07.103 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1685562631.516089_59.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-05-31T19:50:31.516089Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1685562970.274572_93.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-05-31T19:56:10.274572Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1685562970.292149_34.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-05-31T19:56:10.292149Z', 'realm_epoch': 2}}] 2023-05-31T19:57:07.104 DEBUG:rgw_multi.tests:log_status={0: '1_1685562631.516089_59.1', 1: '', 2: '1_1685562970.274572_93.1', 3: '1_1685563026.832102_54.1'} 2023-05-31T19:57:07.104 DEBUG:rgw_multi.tests:sync_status={0: '1_1685562631.516089_59.1', 1: '', 2: '1_1685562970.274572_93.1', 3: '1_1685562970.292149_34.1'} 2023-05-31T19:57:07.105 WARNING:rgw_multi.tests:zone a2 behind master: shard=3 master=1_1685563026.832102_54.1 target=1_1685562970.292149_34.1 2023-05-31T19:57:12.105 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-31T19:57:12.107 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-31T19:57:12.108 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-05-31T19:57:12.166 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2023-05-31T19:57:12.166 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2023-05-31T19:57:12.214 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T19:57:12.208+0000 7f1511ddf780 20 RGW-SYNC:meta: read sync status 2023-05-31T19:57:12.214 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T19:57:12.209+0000 7f1511ddf780 20 RGW-SYNC:meta: read sync status 2023-05-31T19:57:12.238 INFO:teuthology.orchestra.run.smithi099.stdout:{ 2023-05-31T19:57:12.238 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_status": { 2023-05-31T19:57:12.238 INFO:teuthology.orchestra.run.smithi099.stdout: "info": { 2023-05-31T19:57:12.238 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "sync", 2023-05-31T19:57:12.238 INFO:teuthology.orchestra.run.smithi099.stdout: "num_shards": 4, 2023-05-31T19:57:12.239 INFO:teuthology.orchestra.run.smithi099.stdout: "period": "2e6123d2-d695-4350-9a01-61a455bc782b", 2023-05-31T19:57:12.239 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2023-05-31T19:57:12.239 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T19:57:12.239 INFO:teuthology.orchestra.run.smithi099.stdout: "markers": [ 2023-05-31T19:57:12.239 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T19:57:12.239 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 0, 2023-05-31T19:57:12.240 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T19:57:12.240 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2023-05-31T19:57:12.240 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1685562631.516089_59.1", 2023-05-31T19:57:12.240 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T19:57:12.240 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 1, 2023-05-31T19:57:12.240 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T19:57:12.241 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2023-05-31T19:50:31.516089Z", 2023-05-31T19:57:12.241 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2023-05-31T19:57:12.241 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T19:57:12.241 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T19:57:12.241 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T19:57:12.241 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 1, 2023-05-31T19:57:12.242 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T19:57:12.242 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2023-05-31T19:57:12.242 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "", 2023-05-31T19:57:12.242 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T19:57:12.242 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2023-05-31T19:57:12.243 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T19:57:12.243 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "0.000000", 2023-05-31T19:57:12.243 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2023-05-31T19:57:12.244 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T19:57:12.244 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T19:57:12.245 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T19:57:12.245 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 2, 2023-05-31T19:57:12.245 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T19:57:12.246 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2023-05-31T19:57:12.246 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1685562970.274572_93.1", 2023-05-31T19:57:12.246 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T19:57:12.247 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2023-05-31T19:57:12.247 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T19:57:12.247 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2023-05-31T19:56:10.274572Z", 2023-05-31T19:57:12.247 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2023-05-31T19:57:12.248 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T19:57:12.248 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T19:57:12.248 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T19:57:12.249 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 3, 2023-05-31T19:57:12.249 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T19:57:12.249 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2023-05-31T19:57:12.250 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1685563026.832102_54.1", 2023-05-31T19:57:12.250 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T19:57:12.250 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2023-05-31T19:57:12.250 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T19:57:12.250 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2023-05-31T19:57:06.832102Z", 2023-05-31T19:57:12.250 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2023-05-31T19:57:12.251 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T19:57:12.251 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T19:57:12.251 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2023-05-31T19:57:12.251 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T19:57:12.251 INFO:teuthology.orchestra.run.smithi099.stdout: "full_sync": { 2023-05-31T19:57:12.252 INFO:teuthology.orchestra.run.smithi099.stdout: "total": 1, 2023-05-31T19:57:12.252 INFO:teuthology.orchestra.run.smithi099.stdout: "complete": 1 2023-05-31T19:57:12.252 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T19:57:12.252 INFO:teuthology.orchestra.run.smithi099.stdout: "current_time": "2023-05-31T19:57:12Z" 2023-05-31T19:57:12.252 INFO:teuthology.orchestra.run.smithi099.stdout:} 2023-05-31T19:57:12.254 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "2e6123d2-d695-4350-9a01-61a455bc782b", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1685562631.516089_59.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-05-31T19:50:31.516089Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1685562970.274572_93.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-05-31T19:56:10.274572Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1685563026.832102_54.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-05-31T19:57:06.832102Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-05-31T19:57:12Z" } 2023-05-31T19:57:12.254 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1685562631.516089_59.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-05-31T19:50:31.516089Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1685562970.274572_93.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-05-31T19:56:10.274572Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1685563026.832102_54.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-05-31T19:57:06.832102Z', 'realm_epoch': 2}}] 2023-05-31T19:57:12.255 DEBUG:rgw_multi.tests:log_status={0: '1_1685562631.516089_59.1', 1: '', 2: '1_1685562970.274572_93.1', 3: '1_1685563026.832102_54.1'} 2023-05-31T19:57:12.255 DEBUG:rgw_multi.tests:sync_status={0: '1_1685562631.516089_59.1', 1: '', 2: '1_1685562970.274572_93.1', 3: '1_1685563026.832102_54.1'} 2023-05-31T19:57:12.255 INFO:rgw_multi.tests:finish meta checkpoint for zone=a2 2023-05-31T19:57:12.256 DEBUG:boto:path=/pklxkd-11/ 2023-05-31T19:57:12.256 DEBUG:boto:auth_path=/pklxkd-11/ 2023-05-31T19:57:12.256 DEBUG:boto:Method: HEAD 2023-05-31T19:57:12.257 DEBUG:boto:Path: /pklxkd-11/ 2023-05-31T19:57:12.257 DEBUG:boto:Data: 2023-05-31T19:57:12.257 DEBUG:boto:Headers: {} 2023-05-31T19:57:12.258 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T19:57:12.258 DEBUG:boto:Port: 8000 2023-05-31T19:57:12.258 DEBUG:boto:Params: {} 2023-05-31T19:57:12.259 DEBUG:boto:Token: None 2023-05-31T19:57:12.259 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 19:57:12 GMT /pklxkd-11/ 2023-05-31T19:57:12.260 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:bGCps8K68cebWRKi9FRicUkU4eM= 2023-05-31T19:57:12.260 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 19:57:12 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:bGCps8K68cebWRKi9FRicUkU4eM=', 'Content-Length': '0'} 2023-05-31T19:57:12.262 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000537ebf4a7e87c746-006477a698-110e-a1'), ('Content-Length', '0'), ('Date', 'Wed, 31 May 2023 19:57:12 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T19:57:12.263 DEBUG:boto:path=/pklxkd-11/dummy 2023-05-31T19:57:12.263 DEBUG:boto:auth_path=/pklxkd-11/dummy 2023-05-31T19:57:12.264 DEBUG:boto:Method: PUT 2023-05-31T19:57:12.264 DEBUG:boto:Path: /pklxkd-11/dummy 2023-05-31T19:57:12.265 DEBUG:boto:Data: 2023-05-31T19:57:12.265 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': '1B2M2Y8AsgTpgAmY7PhCfg==', 'Content-Length': '0', 'Expect': '100-Continue'} 2023-05-31T19:57:12.265 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T19:57:12.266 DEBUG:boto:Port: 8000 2023-05-31T19:57:12.266 DEBUG:boto:Params: {} 2023-05-31T19:57:12.267 DEBUG:boto:Token: None 2023-05-31T19:57:12.267 DEBUG:boto:StringToSign: PUT 1B2M2Y8AsgTpgAmY7PhCfg== application/octet-stream Wed, 31 May 2023 19:57:12 GMT /pklxkd-11/dummy 2023-05-31T19:57:12.267 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:S0cWMc8dSWKrN+tEqmoDAFX5HmY= 2023-05-31T19:57:12.268 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': '1B2M2Y8AsgTpgAmY7PhCfg==', 'Content-Length': '0', 'Expect': '100-Continue', 'Date': 'Wed, 31 May 2023 19:57:12 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:S0cWMc8dSWKrN+tEqmoDAFX5HmY='} 2023-05-31T19:57:12.279 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('Accept-Ranges', 'bytes'), ('x-amz-version-id', 'puen5pSFPkhRYSzopBIdJIXBQzttKnu'), ('x-amz-request-id', 'tx000001de2f6d1c7e92f02-006477a698-110e-a1'), ('Date', 'Wed, 31 May 2023 19:57:12 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T19:57:12.280 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'pklxkd-11', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-05-31T19:57:12.280 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'pklxkd-11', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-05-31T19:57:12.281 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket pklxkd-11 --source-zone a1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2023-05-31T19:57:12.337 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2023-05-31T19:57:12.337 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2023-05-31T19:57:12.373 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T19:57:12.367+0000 7f6bb14eb780 1 waiting to reach incremental sync.. 2023-05-31T19:57:17.374 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T19:57:17.368+0000 7f6bb14eb780 1 waiting to reach incremental sync.. 2023-05-31T19:57:22.374 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T19:57:22.368+0000 7f6bb14eb780 1 waiting to reach incremental sync.. 2023-05-31T19:57:27.376 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T19:57:27.370+0000 7f6bb14eb780 1 bucket sync caught up with source: 2023-05-31T19:57:27.377 INFO:teuthology.orchestra.run.smithi099.stderr: local status: [, , , , , 00000000002.9.11, , , , , ] 2023-05-31T19:57:27.377 INFO:teuthology.orchestra.run.smithi099.stderr: remote markers: [, , , , , 00000000002.9.11, , , , , ] 2023-05-31T19:57:27.377 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T19:57:27.370+0000 7f6bb14eb780 0 bucket checkpoint complete 2023-05-31T19:57:27.392 DEBUG:boto:path=/pklxkd-12/ 2023-05-31T19:57:27.393 DEBUG:boto:auth_path=/pklxkd-12/ 2023-05-31T19:57:27.393 DEBUG:boto:Method: HEAD 2023-05-31T19:57:27.394 DEBUG:boto:Path: /pklxkd-12/ 2023-05-31T19:57:27.394 DEBUG:boto:Data: 2023-05-31T19:57:27.394 DEBUG:boto:Headers: {} 2023-05-31T19:57:27.395 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T19:57:27.395 DEBUG:boto:Port: 8001 2023-05-31T19:57:27.395 DEBUG:boto:Params: {} 2023-05-31T19:57:27.396 DEBUG:boto:Token: None 2023-05-31T19:57:27.396 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 19:57:27 GMT /pklxkd-12/ 2023-05-31T19:57:27.397 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:EZmAcxN5vw+UAgaWE40B4aqd8SY= 2023-05-31T19:57:27.397 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 19:57:27 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:EZmAcxN5vw+UAgaWE40B4aqd8SY=', 'Content-Length': '0'} 2023-05-31T19:57:27.400 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000002cea8495fda3c6ba-006477a6a7-1112-a2'), ('Content-Length', '0'), ('Date', 'Wed, 31 May 2023 19:57:27 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T19:57:27.400 DEBUG:boto:path=/pklxkd-12/dummy 2023-05-31T19:57:27.401 DEBUG:boto:auth_path=/pklxkd-12/dummy 2023-05-31T19:57:27.401 DEBUG:boto:Method: PUT 2023-05-31T19:57:27.401 DEBUG:boto:Path: /pklxkd-12/dummy 2023-05-31T19:57:27.403 DEBUG:boto:Data: 2023-05-31T19:57:27.404 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': '1B2M2Y8AsgTpgAmY7PhCfg==', 'Content-Length': '0', 'Expect': '100-Continue'} 2023-05-31T19:57:27.404 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T19:57:27.404 DEBUG:boto:Port: 8001 2023-05-31T19:57:27.405 DEBUG:boto:Params: {} 2023-05-31T19:57:27.405 DEBUG:boto:Token: None 2023-05-31T19:57:27.406 DEBUG:boto:StringToSign: PUT 1B2M2Y8AsgTpgAmY7PhCfg== application/octet-stream Wed, 31 May 2023 19:57:27 GMT /pklxkd-12/dummy 2023-05-31T19:57:27.406 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:zcaahEQuBz48rP2S88d16fG4V4I= 2023-05-31T19:57:27.407 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': '1B2M2Y8AsgTpgAmY7PhCfg==', 'Content-Length': '0', 'Expect': '100-Continue', 'Date': 'Wed, 31 May 2023 19:57:27 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:zcaahEQuBz48rP2S88d16fG4V4I='} 2023-05-31T19:57:27.419 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('Accept-Ranges', 'bytes'), ('x-amz-version-id', 'HuvfE2dxsypTRiytlVrY6cgwTbY4DCS'), ('x-amz-request-id', 'tx0000060a155517d4cc7c5-006477a6a7-1112-a2'), ('Date', 'Wed, 31 May 2023 19:57:27 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T19:57:27.419 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'pklxkd-12', '--source-zone', 'a2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-05-31T19:57:27.420 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'pklxkd-12', '--source-zone', 'a2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-05-31T19:57:27.420 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket pklxkd-12 --source-zone a2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2023-05-31T19:57:27.481 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2023-05-31T19:57:27.481 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2023-05-31T19:57:27.524 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T19:57:27.518+0000 7f1ec9825780 1 waiting to reach incremental sync.. 2023-05-31T19:57:32.524 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T19:57:32.519+0000 7f1ec9825780 1 waiting to reach incremental sync.. 2023-05-31T19:57:37.525 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T19:57:37.520+0000 7f1ec9825780 1 waiting to reach incremental sync.. 2023-05-31T19:57:42.526 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T19:57:42.521+0000 7f1ec9825780 1 waiting to reach incremental sync.. 2023-05-31T19:57:47.528 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T19:57:47.523+0000 7f1ec9825780 1 bucket sync caught up with source: 2023-05-31T19:57:47.528 INFO:teuthology.orchestra.run.smithi099.stderr: local status: [, , , , , 00000000002.8.11, , , , , ] 2023-05-31T19:57:47.528 INFO:teuthology.orchestra.run.smithi099.stderr: remote markers: [, , , , , 00000000002.8.11, , , , , ] 2023-05-31T19:57:47.528 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T19:57:47.523+0000 7f1ec9825780 0 bucket checkpoint complete 2023-05-31T19:57:47.546 DEBUG:boto:path=/pklxkd-11/ 2023-05-31T19:57:47.546 DEBUG:boto:auth_path=/pklxkd-11/ 2023-05-31T19:57:47.547 DEBUG:boto:Method: HEAD 2023-05-31T19:57:47.547 DEBUG:boto:Path: /pklxkd-11/ 2023-05-31T19:57:47.547 DEBUG:boto:Data: 2023-05-31T19:57:47.548 DEBUG:boto:Headers: {} 2023-05-31T19:57:47.548 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T19:57:47.549 DEBUG:boto:Port: 8000 2023-05-31T19:57:47.549 DEBUG:boto:Params: {} 2023-05-31T19:57:47.549 DEBUG:boto:Token: None 2023-05-31T19:57:47.550 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 19:57:47 GMT /pklxkd-11/ 2023-05-31T19:57:47.550 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:CS3J+K3vTVaJOR27Itg9aO5nTYQ= 2023-05-31T19:57:47.550 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 19:57:47 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:CS3J+K3vTVaJOR27Itg9aO5nTYQ=', 'Content-Length': '0'} 2023-05-31T19:57:47.553 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000008f8c98f0f0e9e773-006477a6bb-110e-a1'), ('Content-Length', '0'), ('Date', 'Wed, 31 May 2023 19:57:47 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T19:57:47.554 DEBUG:boto:path=/pklxkd-11/obj-a1 2023-05-31T19:57:47.554 DEBUG:boto:auth_path=/pklxkd-11/obj-a1 2023-05-31T19:57:47.554 DEBUG:boto:Method: PUT 2023-05-31T19:57:47.555 DEBUG:boto:Path: /pklxkd-11/obj-a1 2023-05-31T19:57:47.555 DEBUG:boto:Data: 2023-05-31T19:57:47.555 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'lmY06/L8E1cH1nU2kr9LHg==', 'Content-Length': '8', 'Expect': '100-Continue'} 2023-05-31T19:57:47.556 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T19:57:47.556 DEBUG:boto:Port: 8000 2023-05-31T19:57:47.556 DEBUG:boto:Params: {} 2023-05-31T19:57:47.557 DEBUG:boto:Token: None 2023-05-31T19:57:47.557 DEBUG:boto:StringToSign: PUT lmY06/L8E1cH1nU2kr9LHg== application/octet-stream Wed, 31 May 2023 19:57:47 GMT /pklxkd-11/obj-a1 2023-05-31T19:57:47.557 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:mkTQAz8LKCR0kypNhiApUF8aDjY= 2023-05-31T19:57:47.558 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'lmY06/L8E1cH1nU2kr9LHg==', 'Content-Length': '8', 'Expect': '100-Continue', 'Date': 'Wed, 31 May 2023 19:57:47 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:mkTQAz8LKCR0kypNhiApUF8aDjY='} 2023-05-31T19:57:47.570 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('Accept-Ranges', 'bytes'), ('x-amz-version-id', 'L7ViqSXBtkdfIYltLrLN1BhDNjOq8xz'), ('x-amz-request-id', 'tx00000a03325497adfd39c-006477a6bb-110e-a1'), ('Date', 'Wed, 31 May 2023 19:57:47 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T19:57:47.571 DEBUG:rgw_multi.tests:version1 id=L7ViqSXBtkdfIYltLrLN1BhDNjOq8xz 2023-05-31T19:57:47.571 DEBUG:boto:path=/pklxkd-11/obj-a1 2023-05-31T19:57:47.571 DEBUG:boto:auth_path=/pklxkd-11/obj-a1 2023-05-31T19:57:47.572 DEBUG:boto:Method: PUT 2023-05-31T19:57:47.572 DEBUG:boto:Path: /pklxkd-11/obj-a1 2023-05-31T19:57:47.572 DEBUG:boto:Data: 2023-05-31T19:57:47.573 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'Lg6VKF8IoH3qF+fuERshyA==', 'Content-Length': '8', 'Expect': '100-Continue'} 2023-05-31T19:57:47.573 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T19:57:47.573 DEBUG:boto:Port: 8000 2023-05-31T19:57:47.574 DEBUG:boto:Params: {} 2023-05-31T19:57:47.574 DEBUG:boto:Token: None 2023-05-31T19:57:47.575 DEBUG:boto:StringToSign: PUT Lg6VKF8IoH3qF+fuERshyA== application/octet-stream Wed, 31 May 2023 19:57:47 GMT /pklxkd-11/obj-a1 2023-05-31T19:57:47.575 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:xmbB2Znpcku5+TkgC7Jgq1wi3zM= 2023-05-31T19:57:47.575 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'Lg6VKF8IoH3qF+fuERshyA==', 'Content-Length': '8', 'Expect': '100-Continue', 'Date': 'Wed, 31 May 2023 19:57:47 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:xmbB2Znpcku5+TkgC7Jgq1wi3zM='} 2023-05-31T19:57:47.586 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"2e0e95285f08a07dea17e7ee111b21c8"'), ('Accept-Ranges', 'bytes'), ('x-amz-version-id', '707mFT8d8KooRcvvR0xAJIa9FedswIk'), ('x-amz-request-id', 'tx0000047953b848e9951d2-006477a6bb-110e-a1'), ('Date', 'Wed, 31 May 2023 19:57:47 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T19:57:47.587 DEBUG:rgw_multi.tests:version2 id=707mFT8d8KooRcvvR0xAJIa9FedswIk 2023-05-31T19:57:47.587 DEBUG:boto:path=/pklxkd-11/obj-a1 2023-05-31T19:57:47.587 DEBUG:boto:auth_path=/pklxkd-11/obj-a1 2023-05-31T19:57:47.588 DEBUG:boto:path=/pklxkd-11/obj-a1?versionId=707mFT8d8KooRcvvR0xAJIa9FedswIk 2023-05-31T19:57:47.588 DEBUG:boto:auth_path=/pklxkd-11/obj-a1?versionId=707mFT8d8KooRcvvR0xAJIa9FedswIk 2023-05-31T19:57:47.588 DEBUG:boto:Method: DELETE 2023-05-31T19:57:47.589 DEBUG:boto:Path: /pklxkd-11/obj-a1?versionId=707mFT8d8KooRcvvR0xAJIa9FedswIk 2023-05-31T19:57:47.589 DEBUG:boto:Data: 2023-05-31T19:57:47.589 DEBUG:boto:Headers: {} 2023-05-31T19:57:47.590 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T19:57:47.590 DEBUG:boto:Port: 8000 2023-05-31T19:57:47.590 DEBUG:boto:Params: {} 2023-05-31T19:57:47.591 DEBUG:boto:Token: None 2023-05-31T19:57:47.591 DEBUG:boto:StringToSign: DELETE Wed, 31 May 2023 19:57:47 GMT /pklxkd-11/obj-a1?versionId=707mFT8d8KooRcvvR0xAJIa9FedswIk 2023-05-31T19:57:47.591 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:k17UVIRjPdhwLhlqG8iksD7cUgQ= 2023-05-31T19:57:47.592 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 19:57:47 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:k17UVIRjPdhwLhlqG8iksD7cUgQ=', 'Content-Length': '0'} 2023-05-31T19:57:47.603 DEBUG:boto:Response headers: [('x-amz-version-id', '707mFT8d8KooRcvvR0xAJIa9FedswIk'), ('x-amz-request-id', 'tx00000ae53218aee34932e-006477a6bb-110e-a1'), ('Date', 'Wed, 31 May 2023 19:57:47 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T19:57:47.603 DEBUG:boto:path=/pklxkd-11/obj-a1 2023-05-31T19:57:47.604 DEBUG:boto:auth_path=/pklxkd-11/obj-a1 2023-05-31T19:57:47.604 DEBUG:boto:Method: PUT 2023-05-31T19:57:47.605 DEBUG:boto:Path: /pklxkd-11/obj-a1 2023-05-31T19:57:47.605 DEBUG:boto:Data: 2023-05-31T19:57:47.605 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'HiNA5/HU7CwjRIhUZJ8d7Q==', 'Content-Length': '8', 'Expect': '100-Continue'} 2023-05-31T19:57:47.606 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T19:57:47.606 DEBUG:boto:Port: 8000 2023-05-31T19:57:47.606 DEBUG:boto:Params: {} 2023-05-31T19:57:47.607 DEBUG:boto:Token: None 2023-05-31T19:57:47.607 DEBUG:boto:StringToSign: PUT HiNA5/HU7CwjRIhUZJ8d7Q== application/octet-stream Wed, 31 May 2023 19:57:47 GMT /pklxkd-11/obj-a1 2023-05-31T19:57:47.607 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:aRQ71mwanfQYNbD/cev6Tb0KQS0= 2023-05-31T19:57:47.608 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'HiNA5/HU7CwjRIhUZJ8d7Q==', 'Content-Length': '8', 'Expect': '100-Continue', 'Date': 'Wed, 31 May 2023 19:57:47 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:aRQ71mwanfQYNbD/cev6Tb0KQS0='} 2023-05-31T19:57:47.652 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"1e2340e7f1d4ec2c23448854649f1ded"'), ('Accept-Ranges', 'bytes'), ('x-amz-version-id', '7UoFp1HA2D63.sycMrfCUGytz-QO8WJ'), ('x-amz-request-id', 'tx00000364625021537738b-006477a6bb-110e-a1'), ('Date', 'Wed, 31 May 2023 19:57:47 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T19:57:47.653 DEBUG:rgw_multi.tests:version3 id=7UoFp1HA2D63.sycMrfCUGytz-QO8WJ 2023-05-31T19:57:47.653 DEBUG:boto:path=/pklxkd-11/obj-a1 2023-05-31T19:57:47.654 DEBUG:boto:auth_path=/pklxkd-11/obj-a1 2023-05-31T19:57:47.654 DEBUG:boto:path=/pklxkd-11/obj-a1?versionId=7UoFp1HA2D63.sycMrfCUGytz-QO8WJ 2023-05-31T19:57:47.654 DEBUG:boto:auth_path=/pklxkd-11/obj-a1?versionId=7UoFp1HA2D63.sycMrfCUGytz-QO8WJ 2023-05-31T19:57:47.655 DEBUG:boto:Method: DELETE 2023-05-31T19:57:47.655 DEBUG:boto:Path: /pklxkd-11/obj-a1?versionId=7UoFp1HA2D63.sycMrfCUGytz-QO8WJ 2023-05-31T19:57:47.655 DEBUG:boto:Data: 2023-05-31T19:57:47.656 DEBUG:boto:Headers: {} 2023-05-31T19:57:47.656 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T19:57:47.656 DEBUG:boto:Port: 8000 2023-05-31T19:57:47.656 DEBUG:boto:Params: {} 2023-05-31T19:57:47.657 DEBUG:boto:Token: None 2023-05-31T19:57:47.657 DEBUG:boto:StringToSign: DELETE Wed, 31 May 2023 19:57:47 GMT /pklxkd-11/obj-a1?versionId=7UoFp1HA2D63.sycMrfCUGytz-QO8WJ 2023-05-31T19:57:47.658 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:BARqVZrqOmI1gtYodhda8sEytSg= 2023-05-31T19:57:47.658 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 19:57:47 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:BARqVZrqOmI1gtYodhda8sEytSg=', 'Content-Length': '0'} 2023-05-31T19:57:47.673 DEBUG:boto:Response headers: [('x-amz-version-id', '7UoFp1HA2D63.sycMrfCUGytz-QO8WJ'), ('x-amz-request-id', 'tx00000e28868f2c9262818-006477a6bb-110e-a1'), ('Date', 'Wed, 31 May 2023 19:57:47 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T19:57:47.675 DEBUG:boto:path=/pklxkd-11/ 2023-05-31T19:57:47.675 DEBUG:boto:auth_path=/pklxkd-11/ 2023-05-31T19:57:47.675 DEBUG:boto:Method: HEAD 2023-05-31T19:57:47.676 DEBUG:boto:Path: /pklxkd-11/ 2023-05-31T19:57:47.676 DEBUG:boto:Data: 2023-05-31T19:57:47.676 DEBUG:boto:Headers: {} 2023-05-31T19:57:47.677 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T19:57:47.677 DEBUG:boto:Port: 8001 2023-05-31T19:57:47.677 DEBUG:boto:Params: {} 2023-05-31T19:57:47.678 DEBUG:boto:Token: None 2023-05-31T19:57:47.678 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 19:57:47 GMT /pklxkd-11/ 2023-05-31T19:57:47.679 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:CS3J+K3vTVaJOR27Itg9aO5nTYQ= 2023-05-31T19:57:47.679 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 19:57:47 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:CS3J+K3vTVaJOR27Itg9aO5nTYQ=', 'Content-Length': '0'} 2023-05-31T19:57:47.681 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000020e85733e7c9346f-006477a6bb-1112-a2'), ('Content-Length', '0'), ('Date', 'Wed, 31 May 2023 19:57:47 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T19:57:47.682 DEBUG:boto:path=/pklxkd-11/obj-a2 2023-05-31T19:57:47.682 DEBUG:boto:auth_path=/pklxkd-11/obj-a2 2023-05-31T19:57:47.683 DEBUG:boto:Method: PUT 2023-05-31T19:57:47.683 DEBUG:boto:Path: /pklxkd-11/obj-a2 2023-05-31T19:57:47.683 DEBUG:boto:Data: 2023-05-31T19:57:47.684 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'lmY06/L8E1cH1nU2kr9LHg==', 'Content-Length': '8', 'Expect': '100-Continue'} 2023-05-31T19:57:47.684 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T19:57:47.684 DEBUG:boto:Port: 8001 2023-05-31T19:57:47.685 DEBUG:boto:Params: {} 2023-05-31T19:57:47.685 DEBUG:boto:Token: None 2023-05-31T19:57:47.686 DEBUG:boto:StringToSign: PUT lmY06/L8E1cH1nU2kr9LHg== application/octet-stream Wed, 31 May 2023 19:57:47 GMT /pklxkd-11/obj-a2 2023-05-31T19:57:47.686 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:yuLyp4qlgb92k4UnEtOunFE6TBk= 2023-05-31T19:57:47.686 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'lmY06/L8E1cH1nU2kr9LHg==', 'Content-Length': '8', 'Expect': '100-Continue', 'Date': 'Wed, 31 May 2023 19:57:47 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:yuLyp4qlgb92k4UnEtOunFE6TBk='} 2023-05-31T19:57:47.728 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('Accept-Ranges', 'bytes'), ('x-amz-version-id', 'd.2JtI5ycFHjEYrzZkvtbFVUQjr5kaL'), ('x-amz-request-id', 'tx000005431d700d76f063f-006477a6bb-1112-a2'), ('Date', 'Wed, 31 May 2023 19:57:47 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T19:57:47.729 DEBUG:rgw_multi.tests:version1 id=d.2JtI5ycFHjEYrzZkvtbFVUQjr5kaL 2023-05-31T19:57:47.730 DEBUG:boto:path=/pklxkd-11/obj-a2 2023-05-31T19:57:47.730 DEBUG:boto:auth_path=/pklxkd-11/obj-a2 2023-05-31T19:57:47.730 DEBUG:boto:Method: PUT 2023-05-31T19:57:47.731 DEBUG:boto:Path: /pklxkd-11/obj-a2 2023-05-31T19:57:47.731 DEBUG:boto:Data: 2023-05-31T19:57:47.731 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'Lg6VKF8IoH3qF+fuERshyA==', 'Content-Length': '8', 'Expect': '100-Continue'} 2023-05-31T19:57:47.732 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T19:57:47.732 DEBUG:boto:Port: 8001 2023-05-31T19:57:47.732 DEBUG:boto:Params: {} 2023-05-31T19:57:47.733 DEBUG:boto:Token: None 2023-05-31T19:57:47.733 DEBUG:boto:StringToSign: PUT Lg6VKF8IoH3qF+fuERshyA== application/octet-stream Wed, 31 May 2023 19:57:47 GMT /pklxkd-11/obj-a2 2023-05-31T19:57:47.734 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:46gu/XepqiGYEnHTqz5zEQDtruA= 2023-05-31T19:57:47.734 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'Lg6VKF8IoH3qF+fuERshyA==', 'Content-Length': '8', 'Expect': '100-Continue', 'Date': 'Wed, 31 May 2023 19:57:47 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:46gu/XepqiGYEnHTqz5zEQDtruA='} 2023-05-31T19:57:47.743 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"2e0e95285f08a07dea17e7ee111b21c8"'), ('Accept-Ranges', 'bytes'), ('x-amz-version-id', 'q0QDUMqvAu5p6edrZLzDudSejvJ2Qr2'), ('x-amz-request-id', 'tx000001935557a14287ec5-006477a6bb-1112-a2'), ('Date', 'Wed, 31 May 2023 19:57:47 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T19:57:47.743 DEBUG:rgw_multi.tests:version2 id=q0QDUMqvAu5p6edrZLzDudSejvJ2Qr2 2023-05-31T19:57:47.744 DEBUG:boto:path=/pklxkd-11/obj-a2 2023-05-31T19:57:47.744 DEBUG:boto:auth_path=/pklxkd-11/obj-a2 2023-05-31T19:57:47.744 DEBUG:boto:path=/pklxkd-11/obj-a2?versionId=q0QDUMqvAu5p6edrZLzDudSejvJ2Qr2 2023-05-31T19:57:47.745 DEBUG:boto:auth_path=/pklxkd-11/obj-a2?versionId=q0QDUMqvAu5p6edrZLzDudSejvJ2Qr2 2023-05-31T19:57:47.745 DEBUG:boto:Method: DELETE 2023-05-31T19:57:47.745 DEBUG:boto:Path: /pklxkd-11/obj-a2?versionId=q0QDUMqvAu5p6edrZLzDudSejvJ2Qr2 2023-05-31T19:57:47.746 DEBUG:boto:Data: 2023-05-31T19:57:47.746 DEBUG:boto:Headers: {} 2023-05-31T19:57:47.746 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T19:57:47.747 DEBUG:boto:Port: 8001 2023-05-31T19:57:47.747 DEBUG:boto:Params: {} 2023-05-31T19:57:47.747 DEBUG:boto:Token: None 2023-05-31T19:57:47.748 DEBUG:boto:StringToSign: DELETE Wed, 31 May 2023 19:57:47 GMT /pklxkd-11/obj-a2?versionId=q0QDUMqvAu5p6edrZLzDudSejvJ2Qr2 2023-05-31T19:57:47.748 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:SQX7TnWo3J6usbovH021HZxyxX8= 2023-05-31T19:57:47.749 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 19:57:47 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:SQX7TnWo3J6usbovH021HZxyxX8=', 'Content-Length': '0'} 2023-05-31T19:57:47.757 DEBUG:boto:Response headers: [('x-amz-version-id', 'q0QDUMqvAu5p6edrZLzDudSejvJ2Qr2'), ('x-amz-request-id', 'tx0000054385f61311bd3ab-006477a6bb-1112-a2'), ('Date', 'Wed, 31 May 2023 19:57:47 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T19:57:47.758 DEBUG:boto:path=/pklxkd-11/obj-a2 2023-05-31T19:57:47.758 DEBUG:boto:auth_path=/pklxkd-11/obj-a2 2023-05-31T19:57:47.759 DEBUG:boto:Method: PUT 2023-05-31T19:57:47.759 DEBUG:boto:Path: /pklxkd-11/obj-a2 2023-05-31T19:57:47.760 DEBUG:boto:Data: 2023-05-31T19:57:47.760 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'HiNA5/HU7CwjRIhUZJ8d7Q==', 'Content-Length': '8', 'Expect': '100-Continue'} 2023-05-31T19:57:47.760 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T19:57:47.761 DEBUG:boto:Port: 8001 2023-05-31T19:57:47.761 DEBUG:boto:Params: {} 2023-05-31T19:57:47.761 DEBUG:boto:Token: None 2023-05-31T19:57:47.762 DEBUG:boto:StringToSign: PUT HiNA5/HU7CwjRIhUZJ8d7Q== application/octet-stream Wed, 31 May 2023 19:57:47 GMT /pklxkd-11/obj-a2 2023-05-31T19:57:47.762 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:ZxjQAjVzHVzAzCVchndrfcZl/wQ= 2023-05-31T19:57:47.763 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'HiNA5/HU7CwjRIhUZJ8d7Q==', 'Content-Length': '8', 'Expect': '100-Continue', 'Date': 'Wed, 31 May 2023 19:57:47 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:ZxjQAjVzHVzAzCVchndrfcZl/wQ='} 2023-05-31T19:57:47.812 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"1e2340e7f1d4ec2c23448854649f1ded"'), ('Accept-Ranges', 'bytes'), ('x-amz-version-id', 'hAId7SlJRib1U67UsQMiJWlfkDuAnQn'), ('x-amz-request-id', 'tx00000a3bc17399443d842-006477a6bb-1112-a2'), ('Date', 'Wed, 31 May 2023 19:57:47 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T19:57:47.813 DEBUG:rgw_multi.tests:version3 id=hAId7SlJRib1U67UsQMiJWlfkDuAnQn 2023-05-31T19:57:47.813 DEBUG:boto:path=/pklxkd-11/obj-a2 2023-05-31T19:57:47.814 DEBUG:boto:auth_path=/pklxkd-11/obj-a2 2023-05-31T19:57:47.814 DEBUG:boto:path=/pklxkd-11/obj-a2?versionId=hAId7SlJRib1U67UsQMiJWlfkDuAnQn 2023-05-31T19:57:47.814 DEBUG:boto:auth_path=/pklxkd-11/obj-a2?versionId=hAId7SlJRib1U67UsQMiJWlfkDuAnQn 2023-05-31T19:57:47.815 DEBUG:boto:Method: DELETE 2023-05-31T19:57:47.815 DEBUG:boto:Path: /pklxkd-11/obj-a2?versionId=hAId7SlJRib1U67UsQMiJWlfkDuAnQn 2023-05-31T19:57:47.815 DEBUG:boto:Data: 2023-05-31T19:57:47.816 DEBUG:boto:Headers: {} 2023-05-31T19:57:47.816 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T19:57:47.816 DEBUG:boto:Port: 8001 2023-05-31T19:57:47.817 DEBUG:boto:Params: {} 2023-05-31T19:57:47.817 DEBUG:boto:Token: None 2023-05-31T19:57:47.817 DEBUG:boto:StringToSign: DELETE Wed, 31 May 2023 19:57:47 GMT /pklxkd-11/obj-a2?versionId=hAId7SlJRib1U67UsQMiJWlfkDuAnQn 2023-05-31T19:57:47.818 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:9F+k/E/8tsI3AUlAG0s+uzYHnp8= 2023-05-31T19:57:47.818 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 19:57:47 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:9F+k/E/8tsI3AUlAG0s+uzYHnp8=', 'Content-Length': '0'} 2023-05-31T19:57:47.827 DEBUG:boto:Response headers: [('x-amz-version-id', 'hAId7SlJRib1U67UsQMiJWlfkDuAnQn'), ('x-amz-request-id', 'tx00000e7bd20f9c48ef475-006477a6bb-1112-a2'), ('Date', 'Wed, 31 May 2023 19:57:47 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T19:57:47.828 DEBUG:boto:path=/pklxkd-12/ 2023-05-31T19:57:47.828 DEBUG:boto:auth_path=/pklxkd-12/ 2023-05-31T19:57:47.829 DEBUG:boto:Method: HEAD 2023-05-31T19:57:47.829 DEBUG:boto:Path: /pklxkd-12/ 2023-05-31T19:57:47.829 DEBUG:boto:Data: 2023-05-31T19:57:47.829 DEBUG:boto:Headers: {} 2023-05-31T19:57:47.830 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T19:57:47.830 DEBUG:boto:Port: 8000 2023-05-31T19:57:47.830 DEBUG:boto:Params: {} 2023-05-31T19:57:47.831 DEBUG:boto:Token: None 2023-05-31T19:57:47.831 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 19:57:47 GMT /pklxkd-12/ 2023-05-31T19:57:47.832 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:gK38zJnkbsN63VKyx2iMSCthLTQ= 2023-05-31T19:57:47.832 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 19:57:47 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:gK38zJnkbsN63VKyx2iMSCthLTQ=', 'Content-Length': '0'} 2023-05-31T19:57:47.834 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000056c7ca325239a9b5-006477a6bb-110e-a1'), ('Content-Length', '0'), ('Date', 'Wed, 31 May 2023 19:57:47 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T19:57:47.835 DEBUG:boto:path=/pklxkd-12/obj-a1 2023-05-31T19:57:47.835 DEBUG:boto:auth_path=/pklxkd-12/obj-a1 2023-05-31T19:57:47.836 DEBUG:boto:Method: PUT 2023-05-31T19:57:47.836 DEBUG:boto:Path: /pklxkd-12/obj-a1 2023-05-31T19:57:47.836 DEBUG:boto:Data: 2023-05-31T19:57:47.837 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'lmY06/L8E1cH1nU2kr9LHg==', 'Content-Length': '8', 'Expect': '100-Continue'} 2023-05-31T19:57:47.837 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T19:57:47.837 DEBUG:boto:Port: 8000 2023-05-31T19:57:47.838 DEBUG:boto:Params: {} 2023-05-31T19:57:47.838 DEBUG:boto:Token: None 2023-05-31T19:57:47.839 DEBUG:boto:StringToSign: PUT lmY06/L8E1cH1nU2kr9LHg== application/octet-stream Wed, 31 May 2023 19:57:47 GMT /pklxkd-12/obj-a1 2023-05-31T19:57:47.839 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:bZQe0i0rovDa592AgwL1M8Gb9wU= 2023-05-31T19:57:47.839 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'lmY06/L8E1cH1nU2kr9LHg==', 'Content-Length': '8', 'Expect': '100-Continue', 'Date': 'Wed, 31 May 2023 19:57:47 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:bZQe0i0rovDa592AgwL1M8Gb9wU='} 2023-05-31T19:57:47.849 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('Accept-Ranges', 'bytes'), ('x-amz-version-id', 'G1y4LrpL0rJndINHI52T89ADrfTweD3'), ('x-amz-request-id', 'tx0000089f35803e663d633-006477a6bb-110e-a1'), ('Date', 'Wed, 31 May 2023 19:57:47 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T19:57:47.850 DEBUG:rgw_multi.tests:version1 id=G1y4LrpL0rJndINHI52T89ADrfTweD3 2023-05-31T19:57:47.850 DEBUG:boto:path=/pklxkd-12/obj-a1 2023-05-31T19:57:47.850 DEBUG:boto:auth_path=/pklxkd-12/obj-a1 2023-05-31T19:57:47.851 DEBUG:boto:Method: PUT 2023-05-31T19:57:47.851 DEBUG:boto:Path: /pklxkd-12/obj-a1 2023-05-31T19:57:47.851 DEBUG:boto:Data: 2023-05-31T19:57:47.852 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'Lg6VKF8IoH3qF+fuERshyA==', 'Content-Length': '8', 'Expect': '100-Continue'} 2023-05-31T19:57:47.852 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T19:57:47.852 DEBUG:boto:Port: 8000 2023-05-31T19:57:47.853 DEBUG:boto:Params: {} 2023-05-31T19:57:47.853 DEBUG:boto:Token: None 2023-05-31T19:57:47.854 DEBUG:boto:StringToSign: PUT Lg6VKF8IoH3qF+fuERshyA== application/octet-stream Wed, 31 May 2023 19:57:47 GMT /pklxkd-12/obj-a1 2023-05-31T19:57:47.854 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:/46ZyiOwOMYFWkai4ARZFvWJjKs= 2023-05-31T19:57:47.854 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'Lg6VKF8IoH3qF+fuERshyA==', 'Content-Length': '8', 'Expect': '100-Continue', 'Date': 'Wed, 31 May 2023 19:57:47 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:/46ZyiOwOMYFWkai4ARZFvWJjKs='} 2023-05-31T19:57:47.863 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"2e0e95285f08a07dea17e7ee111b21c8"'), ('Accept-Ranges', 'bytes'), ('x-amz-version-id', 'YSE3DeWwY1mfsgzZAI-LLpd.5o3WUrs'), ('x-amz-request-id', 'tx000000efb188ed226975a-006477a6bb-110e-a1'), ('Date', 'Wed, 31 May 2023 19:57:47 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T19:57:47.863 DEBUG:rgw_multi.tests:version2 id=YSE3DeWwY1mfsgzZAI-LLpd.5o3WUrs 2023-05-31T19:57:47.864 DEBUG:boto:path=/pklxkd-12/obj-a1 2023-05-31T19:57:47.864 DEBUG:boto:auth_path=/pklxkd-12/obj-a1 2023-05-31T19:57:47.864 DEBUG:boto:path=/pklxkd-12/obj-a1?versionId=YSE3DeWwY1mfsgzZAI-LLpd.5o3WUrs 2023-05-31T19:57:47.865 DEBUG:boto:auth_path=/pklxkd-12/obj-a1?versionId=YSE3DeWwY1mfsgzZAI-LLpd.5o3WUrs 2023-05-31T19:57:47.865 DEBUG:boto:Method: DELETE 2023-05-31T19:57:47.865 DEBUG:boto:Path: /pklxkd-12/obj-a1?versionId=YSE3DeWwY1mfsgzZAI-LLpd.5o3WUrs 2023-05-31T19:57:47.866 DEBUG:boto:Data: 2023-05-31T19:57:47.866 DEBUG:boto:Headers: {} 2023-05-31T19:57:47.866 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T19:57:47.867 DEBUG:boto:Port: 8000 2023-05-31T19:57:47.867 DEBUG:boto:Params: {} 2023-05-31T19:57:47.867 DEBUG:boto:Token: None 2023-05-31T19:57:47.868 DEBUG:boto:StringToSign: DELETE Wed, 31 May 2023 19:57:47 GMT /pklxkd-12/obj-a1?versionId=YSE3DeWwY1mfsgzZAI-LLpd.5o3WUrs 2023-05-31T19:57:47.868 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:spXW8m63j7jZtLDrBCwzPNRsQ/E= 2023-05-31T19:57:47.868 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 19:57:47 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:spXW8m63j7jZtLDrBCwzPNRsQ/E=', 'Content-Length': '0'} 2023-05-31T19:57:47.877 DEBUG:boto:Response headers: [('x-amz-version-id', 'YSE3DeWwY1mfsgzZAI-LLpd.5o3WUrs'), ('x-amz-request-id', 'tx00000f2f2580c3d1fa473-006477a6bb-110e-a1'), ('Date', 'Wed, 31 May 2023 19:57:47 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T19:57:47.878 DEBUG:boto:path=/pklxkd-12/obj-a1 2023-05-31T19:57:47.878 DEBUG:boto:auth_path=/pklxkd-12/obj-a1 2023-05-31T19:57:47.879 DEBUG:boto:Method: PUT 2023-05-31T19:57:47.879 DEBUG:boto:Path: /pklxkd-12/obj-a1 2023-05-31T19:57:47.879 DEBUG:boto:Data: 2023-05-31T19:57:47.880 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'HiNA5/HU7CwjRIhUZJ8d7Q==', 'Content-Length': '8', 'Expect': '100-Continue'} 2023-05-31T19:57:47.880 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T19:57:47.880 DEBUG:boto:Port: 8000 2023-05-31T19:57:47.881 DEBUG:boto:Params: {} 2023-05-31T19:57:47.881 DEBUG:boto:Token: None 2023-05-31T19:57:47.881 DEBUG:boto:StringToSign: PUT HiNA5/HU7CwjRIhUZJ8d7Q== application/octet-stream Wed, 31 May 2023 19:57:47 GMT /pklxkd-12/obj-a1 2023-05-31T19:57:47.882 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:QFtD+xCNxXYJoCnKfgzoCSQ2/cc= 2023-05-31T19:57:47.882 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'HiNA5/HU7CwjRIhUZJ8d7Q==', 'Content-Length': '8', 'Expect': '100-Continue', 'Date': 'Wed, 31 May 2023 19:57:47 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:QFtD+xCNxXYJoCnKfgzoCSQ2/cc='} 2023-05-31T19:57:47.928 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"1e2340e7f1d4ec2c23448854649f1ded"'), ('Accept-Ranges', 'bytes'), ('x-amz-version-id', 'tkizpCWCNOHWInOFjOQ6Mw-tyovGThU'), ('x-amz-request-id', 'tx00000c801da41c33bf913-006477a6bb-110e-a1'), ('Date', 'Wed, 31 May 2023 19:57:47 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T19:57:47.929 DEBUG:rgw_multi.tests:version3 id=tkizpCWCNOHWInOFjOQ6Mw-tyovGThU 2023-05-31T19:57:47.929 DEBUG:boto:path=/pklxkd-12/obj-a1 2023-05-31T19:57:47.930 DEBUG:boto:auth_path=/pklxkd-12/obj-a1 2023-05-31T19:57:47.930 DEBUG:boto:path=/pklxkd-12/obj-a1?versionId=tkizpCWCNOHWInOFjOQ6Mw-tyovGThU 2023-05-31T19:57:47.930 DEBUG:boto:auth_path=/pklxkd-12/obj-a1?versionId=tkizpCWCNOHWInOFjOQ6Mw-tyovGThU 2023-05-31T19:57:47.931 DEBUG:boto:Method: DELETE 2023-05-31T19:57:47.931 DEBUG:boto:Path: /pklxkd-12/obj-a1?versionId=tkizpCWCNOHWInOFjOQ6Mw-tyovGThU 2023-05-31T19:57:47.931 DEBUG:boto:Data: 2023-05-31T19:57:47.932 DEBUG:boto:Headers: {} 2023-05-31T19:57:47.932 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T19:57:47.932 DEBUG:boto:Port: 8000 2023-05-31T19:57:47.933 DEBUG:boto:Params: {} 2023-05-31T19:57:47.933 DEBUG:boto:Token: None 2023-05-31T19:57:47.933 DEBUG:boto:StringToSign: DELETE Wed, 31 May 2023 19:57:47 GMT /pklxkd-12/obj-a1?versionId=tkizpCWCNOHWInOFjOQ6Mw-tyovGThU 2023-05-31T19:57:47.934 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:+xFOVzeVQl0GHZTl9LLoTBK0jIM= 2023-05-31T19:57:47.934 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 19:57:47 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:+xFOVzeVQl0GHZTl9LLoTBK0jIM=', 'Content-Length': '0'} 2023-05-31T19:57:47.952 DEBUG:boto:Response headers: [('x-amz-version-id', 'tkizpCWCNOHWInOFjOQ6Mw-tyovGThU'), ('x-amz-request-id', 'tx00000a66ba731b1af60b3-006477a6bb-110e-a1'), ('Date', 'Wed, 31 May 2023 19:57:47 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T19:57:47.953 DEBUG:boto:path=/pklxkd-12/ 2023-05-31T19:57:47.954 DEBUG:boto:auth_path=/pklxkd-12/ 2023-05-31T19:57:47.954 DEBUG:boto:Method: HEAD 2023-05-31T19:57:47.954 DEBUG:boto:Path: /pklxkd-12/ 2023-05-31T19:57:47.955 DEBUG:boto:Data: 2023-05-31T19:57:47.955 DEBUG:boto:Headers: {} 2023-05-31T19:57:47.955 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T19:57:47.956 DEBUG:boto:Port: 8001 2023-05-31T19:57:47.956 DEBUG:boto:Params: {} 2023-05-31T19:57:47.956 DEBUG:boto:Token: None 2023-05-31T19:57:47.957 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 19:57:47 GMT /pklxkd-12/ 2023-05-31T19:57:47.957 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:gK38zJnkbsN63VKyx2iMSCthLTQ= 2023-05-31T19:57:47.958 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 19:57:47 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:gK38zJnkbsN63VKyx2iMSCthLTQ=', 'Content-Length': '0'} 2023-05-31T19:57:47.960 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000f2fbf631abd44a2e-006477a6bb-1112-a2'), ('Content-Length', '0'), ('Date', 'Wed, 31 May 2023 19:57:47 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T19:57:47.960 DEBUG:boto:path=/pklxkd-12/obj-a2 2023-05-31T19:57:47.961 DEBUG:boto:auth_path=/pklxkd-12/obj-a2 2023-05-31T19:57:47.961 DEBUG:boto:Method: PUT 2023-05-31T19:57:47.961 DEBUG:boto:Path: /pklxkd-12/obj-a2 2023-05-31T19:57:47.962 DEBUG:boto:Data: 2023-05-31T19:57:47.962 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'lmY06/L8E1cH1nU2kr9LHg==', 'Content-Length': '8', 'Expect': '100-Continue'} 2023-05-31T19:57:47.962 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T19:57:47.963 DEBUG:boto:Port: 8001 2023-05-31T19:57:47.963 DEBUG:boto:Params: {} 2023-05-31T19:57:47.964 DEBUG:boto:Token: None 2023-05-31T19:57:47.964 DEBUG:boto:StringToSign: PUT lmY06/L8E1cH1nU2kr9LHg== application/octet-stream Wed, 31 May 2023 19:57:47 GMT /pklxkd-12/obj-a2 2023-05-31T19:57:47.964 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:rZr/Fy/Tba05gY7rJykeRWwbyWY= 2023-05-31T19:57:47.965 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'lmY06/L8E1cH1nU2kr9LHg==', 'Content-Length': '8', 'Expect': '100-Continue', 'Date': 'Wed, 31 May 2023 19:57:47 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:rZr/Fy/Tba05gY7rJykeRWwbyWY='} 2023-05-31T19:57:47.980 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('Accept-Ranges', 'bytes'), ('x-amz-version-id', 'qiyfRuud8o74hAGpczwf4-FuN5p5887'), ('x-amz-request-id', 'tx0000098760a19794c5fcc-006477a6bb-1112-a2'), ('Date', 'Wed, 31 May 2023 19:57:47 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T19:57:47.981 DEBUG:rgw_multi.tests:version1 id=qiyfRuud8o74hAGpczwf4-FuN5p5887 2023-05-31T19:57:47.981 DEBUG:boto:path=/pklxkd-12/obj-a2 2023-05-31T19:57:47.981 DEBUG:boto:auth_path=/pklxkd-12/obj-a2 2023-05-31T19:57:47.982 DEBUG:boto:Method: PUT 2023-05-31T19:57:47.982 DEBUG:boto:Path: /pklxkd-12/obj-a2 2023-05-31T19:57:47.982 DEBUG:boto:Data: 2023-05-31T19:57:47.983 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'Lg6VKF8IoH3qF+fuERshyA==', 'Content-Length': '8', 'Expect': '100-Continue'} 2023-05-31T19:57:47.983 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T19:57:47.983 DEBUG:boto:Port: 8001 2023-05-31T19:57:47.984 DEBUG:boto:Params: {} 2023-05-31T19:57:47.984 DEBUG:boto:Token: None 2023-05-31T19:57:47.985 DEBUG:boto:StringToSign: PUT Lg6VKF8IoH3qF+fuERshyA== application/octet-stream Wed, 31 May 2023 19:57:47 GMT /pklxkd-12/obj-a2 2023-05-31T19:57:47.985 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:dKXZq9AKeYgMzTwssqNZoEIjWts= 2023-05-31T19:57:47.985 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'Lg6VKF8IoH3qF+fuERshyA==', 'Content-Length': '8', 'Expect': '100-Continue', 'Date': 'Wed, 31 May 2023 19:57:47 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:dKXZq9AKeYgMzTwssqNZoEIjWts='} 2023-05-31T19:57:47.994 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"2e0e95285f08a07dea17e7ee111b21c8"'), ('Accept-Ranges', 'bytes'), ('x-amz-version-id', 'KEcToQiI6N2oDTCmCnOAtCYelwO9Umz'), ('x-amz-request-id', 'tx00000681302aa4fdd7622-006477a6bb-1112-a2'), ('Date', 'Wed, 31 May 2023 19:57:47 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T19:57:47.995 DEBUG:rgw_multi.tests:version2 id=KEcToQiI6N2oDTCmCnOAtCYelwO9Umz 2023-05-31T19:57:47.995 DEBUG:boto:path=/pklxkd-12/obj-a2 2023-05-31T19:57:47.995 DEBUG:boto:auth_path=/pklxkd-12/obj-a2 2023-05-31T19:57:47.996 DEBUG:boto:path=/pklxkd-12/obj-a2?versionId=KEcToQiI6N2oDTCmCnOAtCYelwO9Umz 2023-05-31T19:57:47.996 DEBUG:boto:auth_path=/pklxkd-12/obj-a2?versionId=KEcToQiI6N2oDTCmCnOAtCYelwO9Umz 2023-05-31T19:57:47.996 DEBUG:boto:Method: DELETE 2023-05-31T19:57:47.997 DEBUG:boto:Path: /pklxkd-12/obj-a2?versionId=KEcToQiI6N2oDTCmCnOAtCYelwO9Umz 2023-05-31T19:57:47.997 DEBUG:boto:Data: 2023-05-31T19:57:47.997 DEBUG:boto:Headers: {} 2023-05-31T19:57:47.998 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T19:57:47.998 DEBUG:boto:Port: 8001 2023-05-31T19:57:47.998 DEBUG:boto:Params: {} 2023-05-31T19:57:47.999 DEBUG:boto:Token: None 2023-05-31T19:57:47.999 DEBUG:boto:StringToSign: DELETE Wed, 31 May 2023 19:57:47 GMT /pklxkd-12/obj-a2?versionId=KEcToQiI6N2oDTCmCnOAtCYelwO9Umz 2023-05-31T19:57:47.999 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:FVky9LsX1uXeF7/ji8GaWxUce1c= 2023-05-31T19:57:48.000 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 19:57:47 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:FVky9LsX1uXeF7/ji8GaWxUce1c=', 'Content-Length': '0'} 2023-05-31T19:57:48.008 DEBUG:boto:Response headers: [('x-amz-version-id', 'KEcToQiI6N2oDTCmCnOAtCYelwO9Umz'), ('x-amz-request-id', 'tx00000888cd8598b6eadfa-006477a6bb-1112-a2'), ('Date', 'Wed, 31 May 2023 19:57:48 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T19:57:48.009 DEBUG:boto:path=/pklxkd-12/obj-a2 2023-05-31T19:57:48.009 DEBUG:boto:auth_path=/pklxkd-12/obj-a2 2023-05-31T19:57:48.010 DEBUG:boto:Method: PUT 2023-05-31T19:57:48.010 DEBUG:boto:Path: /pklxkd-12/obj-a2 2023-05-31T19:57:48.010 DEBUG:boto:Data: 2023-05-31T19:57:48.011 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'HiNA5/HU7CwjRIhUZJ8d7Q==', 'Content-Length': '8', 'Expect': '100-Continue'} 2023-05-31T19:57:48.011 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T19:57:48.011 DEBUG:boto:Port: 8001 2023-05-31T19:57:48.012 DEBUG:boto:Params: {} 2023-05-31T19:57:48.012 DEBUG:boto:Token: None 2023-05-31T19:57:48.013 DEBUG:boto:StringToSign: PUT HiNA5/HU7CwjRIhUZJ8d7Q== application/octet-stream Wed, 31 May 2023 19:57:48 GMT /pklxkd-12/obj-a2 2023-05-31T19:57:48.013 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:d4K+GCwepvxH21UUCAw9xDxbNro= 2023-05-31T19:57:48.013 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'HiNA5/HU7CwjRIhUZJ8d7Q==', 'Content-Length': '8', 'Expect': '100-Continue', 'Date': 'Wed, 31 May 2023 19:57:48 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:d4K+GCwepvxH21UUCAw9xDxbNro='} 2023-05-31T19:57:48.060 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"1e2340e7f1d4ec2c23448854649f1ded"'), ('Accept-Ranges', 'bytes'), ('x-amz-version-id', 'ju.oSHxFyd0VNt5VssjXTRF2fH1wPIE'), ('x-amz-request-id', 'tx00000e1a210d1277788fd-006477a6bc-1112-a2'), ('Date', 'Wed, 31 May 2023 19:57:48 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T19:57:48.061 DEBUG:rgw_multi.tests:version3 id=ju.oSHxFyd0VNt5VssjXTRF2fH1wPIE 2023-05-31T19:57:48.061 DEBUG:boto:path=/pklxkd-12/obj-a2 2023-05-31T19:57:48.062 DEBUG:boto:auth_path=/pklxkd-12/obj-a2 2023-05-31T19:57:48.062 DEBUG:boto:path=/pklxkd-12/obj-a2?versionId=ju.oSHxFyd0VNt5VssjXTRF2fH1wPIE 2023-05-31T19:57:48.062 DEBUG:boto:auth_path=/pklxkd-12/obj-a2?versionId=ju.oSHxFyd0VNt5VssjXTRF2fH1wPIE 2023-05-31T19:57:48.063 DEBUG:boto:Method: DELETE 2023-05-31T19:57:48.063 DEBUG:boto:Path: /pklxkd-12/obj-a2?versionId=ju.oSHxFyd0VNt5VssjXTRF2fH1wPIE 2023-05-31T19:57:48.063 DEBUG:boto:Data: 2023-05-31T19:57:48.064 DEBUG:boto:Headers: {} 2023-05-31T19:57:48.064 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T19:57:48.064 DEBUG:boto:Port: 8001 2023-05-31T19:57:48.065 DEBUG:boto:Params: {} 2023-05-31T19:57:48.065 DEBUG:boto:Token: None 2023-05-31T19:57:48.065 DEBUG:boto:StringToSign: DELETE Wed, 31 May 2023 19:57:48 GMT /pklxkd-12/obj-a2?versionId=ju.oSHxFyd0VNt5VssjXTRF2fH1wPIE 2023-05-31T19:57:48.066 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:iXUXYAfazgnQzvdPS8tYQVcA1Cc= 2023-05-31T19:57:48.066 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 19:57:48 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:iXUXYAfazgnQzvdPS8tYQVcA1Cc=', 'Content-Length': '0'} 2023-05-31T19:57:48.074 DEBUG:boto:Response headers: [('x-amz-version-id', 'ju.oSHxFyd0VNt5VssjXTRF2fH1wPIE'), ('x-amz-request-id', 'tx00000f26d55db868deec0-006477a6bc-1112-a2'), ('Date', 'Wed, 31 May 2023 19:57:48 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T19:57:48.075 DEBUG:rgw_multi.tests:bucket checkpoint: source=a1 target=a2 bucket=pklxkd-11 2023-05-31T19:57:48.075 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'pklxkd-11', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-05-31T19:57:48.076 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'pklxkd-11', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-05-31T19:57:48.076 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket pklxkd-11 --source-zone a1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2023-05-31T19:57:48.133 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2023-05-31T19:57:48.133 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2023-05-31T19:57:48.169 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T19:57:48.164+0000 7f83525b4780 1 waiting for incremental sync to catch up: 2023-05-31T19:57:48.169 INFO:teuthology.orchestra.run.smithi099.stderr: local status: [, , , , , 00000000002.9.11, , , , , ] 2023-05-31T19:57:48.170 INFO:teuthology.orchestra.run.smithi099.stderr: remote markers: [, , , , , 00000000003.11.10, , , , 00000000010.26.6, ] 2023-05-31T19:57:53.171 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T19:57:53.166+0000 7f83525b4780 1 waiting for incremental sync to catch up: 2023-05-31T19:57:53.171 INFO:teuthology.orchestra.run.smithi099.stderr: local status: [, , , , , 00000000002.9.11, , , , , ] 2023-05-31T19:57:53.172 INFO:teuthology.orchestra.run.smithi099.stderr: remote markers: [, , , , , 00000000003.11.10, , , , 00000000010.26.6, ] 2023-05-31T19:57:58.172 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T19:57:58.166+0000 7f83525b4780 1 waiting for incremental sync to catch up: 2023-05-31T19:57:58.173 INFO:teuthology.orchestra.run.smithi099.stderr: local status: [, , , , , 00000000002.9.11, , , , , ] 2023-05-31T19:57:58.173 INFO:teuthology.orchestra.run.smithi099.stderr: remote markers: [, , , , , 00000000003.11.10, , , , 00000000010.26.6, ] 2023-05-31T19:58:03.173 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T19:58:03.168+0000 7f83525b4780 1 waiting for incremental sync to catch up: 2023-05-31T19:58:03.174 INFO:teuthology.orchestra.run.smithi099.stderr: local status: [, , , , , 00000000002.9.11, , , , , ] 2023-05-31T19:58:03.174 INFO:teuthology.orchestra.run.smithi099.stderr: remote markers: [, , , , , 00000000003.11.10, , , , 00000000010.26.6, ] 2023-05-31T19:58:08.175 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T19:58:08.169+0000 7f83525b4780 1 waiting for incremental sync to catch up: 2023-05-31T19:58:08.175 INFO:teuthology.orchestra.run.smithi099.stderr: local status: [, , , , , 00000000002.9.11, , , 00000000002.4.11, 00000000010.26.6, ] 2023-05-31T19:58:08.176 INFO:teuthology.orchestra.run.smithi099.stderr: remote markers: [, , , , , 00000000003.11.10, , , , 00000000010.26.6, ] 2023-05-31T19:58:13.176 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T19:58:13.171+0000 7f83525b4780 1 waiting for incremental sync to catch up: 2023-05-31T19:58:13.177 INFO:teuthology.orchestra.run.smithi099.stderr: local status: [, , , , , 00000000002.9.11, , , 00000000002.4.11, 00000000010.26.6, ] 2023-05-31T19:58:13.177 INFO:teuthology.orchestra.run.smithi099.stderr: remote markers: [, , , , , 00000000003.11.10, , , , 00000000010.26.6, ] 2023-05-31T19:58:18.178 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T19:58:18.173+0000 7f83525b4780 1 waiting for incremental sync to catch up: 2023-05-31T19:58:18.178 INFO:teuthology.orchestra.run.smithi099.stderr: local status: [, , , , , 00000000002.9.11, , , 00000000002.4.11, 00000000010.26.6, ] 2023-05-31T19:58:18.178 INFO:teuthology.orchestra.run.smithi099.stderr: remote markers: [, , , , , 00000000003.11.10, , , , 00000000010.26.6, ] 2023-05-31T19:58:23.179 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T19:58:23.174+0000 7f83525b4780 1 waiting for incremental sync to catch up: 2023-05-31T19:58:23.179 INFO:teuthology.orchestra.run.smithi099.stderr: local status: [, , , , , 00000000002.9.11, , , 00000000002.4.11, 00000000010.26.6, ] 2023-05-31T19:58:23.180 INFO:teuthology.orchestra.run.smithi099.stderr: remote markers: [, , , , , 00000000003.11.10, , , , 00000000010.26.6, ] 2023-05-31T19:58:28.181 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T19:58:28.176+0000 7f83525b4780 1 bucket sync caught up with source: 2023-05-31T19:58:28.181 INFO:teuthology.orchestra.run.smithi099.stderr: local status: [, , , , , 00000000003.11.10, , , 00000000002.4.11, 00000000010.26.6, ] 2023-05-31T19:58:28.181 INFO:teuthology.orchestra.run.smithi099.stderr: remote markers: [, , , , , 00000000003.11.10, , , , 00000000010.26.6, ] 2023-05-31T19:58:28.181 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T19:58:28.176+0000 7f83525b4780 0 bucket checkpoint complete 2023-05-31T19:58:28.198 DEBUG:rgw_multi.tests:bucket checkpoint: source=a2 target=a1 bucket=pklxkd-11 2023-05-31T19:58:28.199 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'pklxkd-11', '--source-zone', 'a2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-05-31T19:58:28.199 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'pklxkd-11', '--source-zone', 'a2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-05-31T19:58:28.200 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket pklxkd-11 --source-zone a2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2023-05-31T19:58:28.260 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2023-05-31T19:58:28.261 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2023-05-31T19:58:28.304 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T19:58:28.300+0000 7f5e0baff780 1 bucket sync caught up with source: 2023-05-31T19:58:28.305 INFO:teuthology.orchestra.run.smithi099.stderr: local status: [, , , , , 00000000002.27.11, , , 00000000010.27.6, 00000000002.26.11, ] 2023-05-31T19:58:28.305 INFO:teuthology.orchestra.run.smithi099.stderr: remote markers: [, , , , , 00000000002.27.11, , , 00000000010.27.6, 00000000002.26.11, ] 2023-05-31T19:58:28.305 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T19:58:28.300+0000 7f5e0baff780 0 bucket checkpoint complete 2023-05-31T19:58:28.318 INFO:tasks.rgw_multi.zone_rados:comparing bucket=pklxkd-11 zones={a2, a1} 2023-05-31T19:58:28.318 DEBUG:boto:path=/pklxkd-11/ 2023-05-31T19:58:28.318 DEBUG:boto:auth_path=/pklxkd-11/ 2023-05-31T19:58:28.319 DEBUG:boto:Method: HEAD 2023-05-31T19:58:28.319 DEBUG:boto:Path: /pklxkd-11/ 2023-05-31T19:58:28.320 DEBUG:boto:Data: 2023-05-31T19:58:28.320 DEBUG:boto:Headers: {} 2023-05-31T19:58:28.320 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T19:58:28.321 DEBUG:boto:Port: 8001 2023-05-31T19:58:28.321 DEBUG:boto:Params: {} 2023-05-31T19:58:28.321 DEBUG:boto:Token: None 2023-05-31T19:58:28.322 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 19:58:28 GMT /pklxkd-11/ 2023-05-31T19:58:28.322 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:1VC3m3+Y1qs07v8AUwMpo+PSsAI= 2023-05-31T19:58:28.323 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 19:58:28 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:1VC3m3+Y1qs07v8AUwMpo+PSsAI=', 'Content-Length': '0'} 2023-05-31T19:58:28.325 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '3'), ('X-RGW-Bytes-Used', '16'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000004b68e0acf74accf4-006477a6e4-1112-a2'), ('Content-Length', '0'), ('Date', 'Wed, 31 May 2023 19:58:28 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T19:58:28.325 DEBUG:boto:path=/pklxkd-11/ 2023-05-31T19:58:28.326 DEBUG:boto:auth_path=/pklxkd-11/ 2023-05-31T19:58:28.326 DEBUG:boto:Method: HEAD 2023-05-31T19:58:28.326 DEBUG:boto:Path: /pklxkd-11/ 2023-05-31T19:58:28.327 DEBUG:boto:Data: 2023-05-31T19:58:28.327 DEBUG:boto:Headers: {} 2023-05-31T19:58:28.327 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T19:58:28.328 DEBUG:boto:Port: 8000 2023-05-31T19:58:28.328 DEBUG:boto:Params: {} 2023-05-31T19:58:28.328 DEBUG:boto:Token: None 2023-05-31T19:58:28.329 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 19:58:28 GMT /pklxkd-11/ 2023-05-31T19:58:28.329 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:1VC3m3+Y1qs07v8AUwMpo+PSsAI= 2023-05-31T19:58:28.330 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 19:58:28 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:1VC3m3+Y1qs07v8AUwMpo+PSsAI=', 'Content-Length': '0'} 2023-05-31T19:58:28.332 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '3'), ('X-RGW-Bytes-Used', '16'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000008c182c655db72115-006477a6e4-110e-a1'), ('Content-Length', '0'), ('Date', 'Wed, 31 May 2023 19:58:28 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T19:58:28.332 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2023-05-31T19:58:28.333 DEBUG:boto:path=/pklxkd-11/ 2023-05-31T19:58:28.333 DEBUG:boto:auth_path=/pklxkd-11/ 2023-05-31T19:58:28.333 DEBUG:boto:path=/pklxkd-11/?versions&max-keys=999 2023-05-31T19:58:28.334 DEBUG:boto:auth_path=/pklxkd-11/?versions&max-keys=999 2023-05-31T19:58:28.334 DEBUG:boto:Method: GET 2023-05-31T19:58:28.334 DEBUG:boto:Path: /pklxkd-11/?versions&max-keys=999 2023-05-31T19:58:28.335 DEBUG:boto:Data: 2023-05-31T19:58:28.335 DEBUG:boto:Headers: {} 2023-05-31T19:58:28.335 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T19:58:28.336 DEBUG:boto:Port: 8001 2023-05-31T19:58:28.336 DEBUG:boto:Params: {} 2023-05-31T19:58:28.336 DEBUG:boto:Token: None 2023-05-31T19:58:28.337 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 19:58:28 GMT /pklxkd-11/?versions 2023-05-31T19:58:28.337 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:eroZmkpWsM1LIgOrTUoOaZje1I0= 2023-05-31T19:58:28.338 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 19:58:28 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:eroZmkpWsM1LIgOrTUoOaZje1I0=', 'Content-Length': '0'} 2023-05-31T19:58:28.340 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000006076f278c079fc-006477a6e4-1112-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 19:58:28 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T19:58:28.340 DEBUG:boto:b'pklxkd-11999falsedummypuen5pSFPkhRYSzopBIdJIXBQzttKnutrue2023-05-31T19:57:12.265Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-a1L7ViqSXBtkdfIYltLrLN1BhDNjOq8xztrue2023-05-31T19:57:47.555Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-a2d.2JtI5ycFHjEYrzZkvtbFVUQjr5kaLtrue2023-05-31T19:57:47.683Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T19:58:28.341 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2023-05-31T19:58:28.341 DEBUG:tasks.rgw_multi.zone_rados:o=obj-a1 2023-05-31T19:58:28.342 DEBUG:tasks.rgw_multi.zone_rados:o=obj-a2 2023-05-31T19:58:28.342 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2023-05-31T19:58:28.342 DEBUG:boto:path=/pklxkd-11/ 2023-05-31T19:58:28.343 DEBUG:boto:auth_path=/pklxkd-11/ 2023-05-31T19:58:28.343 DEBUG:boto:path=/pklxkd-11/?versions&max-keys=999 2023-05-31T19:58:28.343 DEBUG:boto:auth_path=/pklxkd-11/?versions&max-keys=999 2023-05-31T19:58:28.344 DEBUG:boto:Method: GET 2023-05-31T19:58:28.344 DEBUG:boto:Path: /pklxkd-11/?versions&max-keys=999 2023-05-31T19:58:28.344 DEBUG:boto:Data: 2023-05-31T19:58:28.345 DEBUG:boto:Headers: {} 2023-05-31T19:58:28.345 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T19:58:28.345 DEBUG:boto:Port: 8000 2023-05-31T19:58:28.346 DEBUG:boto:Params: {} 2023-05-31T19:58:28.346 DEBUG:boto:Token: None 2023-05-31T19:58:28.347 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 19:58:28 GMT /pklxkd-11/?versions 2023-05-31T19:58:28.347 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:eroZmkpWsM1LIgOrTUoOaZje1I0= 2023-05-31T19:58:28.347 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 19:58:28 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:eroZmkpWsM1LIgOrTUoOaZje1I0=', 'Content-Length': '0'} 2023-05-31T19:58:28.350 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000005dbded44c9524bb2-006477a6e4-110e-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 19:58:28 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T19:58:28.350 DEBUG:boto:b'pklxkd-11999falsedummypuen5pSFPkhRYSzopBIdJIXBQzttKnutrue2023-05-31T19:57:12.265Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-a1L7ViqSXBtkdfIYltLrLN1BhDNjOq8xztrue2023-05-31T19:57:47.555Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-a2d.2JtI5ycFHjEYrzZkvtbFVUQjr5kaLtrue2023-05-31T19:57:47.683Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T19:58:28.351 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2023-05-31T19:58:28.351 DEBUG:tasks.rgw_multi.zone_rados:o=obj-a1 2023-05-31T19:58:28.351 DEBUG:tasks.rgw_multi.zone_rados:o=obj-a2 2023-05-31T19:58:28.352 DEBUG:boto:path=/pklxkd-11/ 2023-05-31T19:58:28.352 DEBUG:boto:auth_path=/pklxkd-11/ 2023-05-31T19:58:28.352 DEBUG:boto:path=/pklxkd-11/?versions&max-keys=999 2023-05-31T19:58:28.353 DEBUG:boto:auth_path=/pklxkd-11/?versions&max-keys=999 2023-05-31T19:58:28.353 DEBUG:boto:Method: GET 2023-05-31T19:58:28.354 DEBUG:boto:Path: /pklxkd-11/?versions&max-keys=999 2023-05-31T19:58:28.354 DEBUG:boto:Data: 2023-05-31T19:58:28.354 DEBUG:boto:Headers: {} 2023-05-31T19:58:28.355 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T19:58:28.355 DEBUG:boto:Port: 8001 2023-05-31T19:58:28.355 DEBUG:boto:Params: {} 2023-05-31T19:58:28.356 DEBUG:boto:Token: None 2023-05-31T19:58:28.356 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 19:58:28 GMT /pklxkd-11/?versions 2023-05-31T19:58:28.357 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:eroZmkpWsM1LIgOrTUoOaZje1I0= 2023-05-31T19:58:28.357 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 19:58:28 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:eroZmkpWsM1LIgOrTUoOaZje1I0=', 'Content-Length': '0'} 2023-05-31T19:58:28.359 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000004e5b8052f91ee7f6-006477a6e4-1112-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 19:58:28 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T19:58:28.360 DEBUG:boto:b'pklxkd-11999falsedummypuen5pSFPkhRYSzopBIdJIXBQzttKnutrue2023-05-31T19:57:12.265Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-a1L7ViqSXBtkdfIYltLrLN1BhDNjOq8xztrue2023-05-31T19:57:47.555Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-a2d.2JtI5ycFHjEYrzZkvtbFVUQjr5kaLtrue2023-05-31T19:57:47.683Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T19:58:28.361 DEBUG:boto:path=/pklxkd-11/ 2023-05-31T19:58:28.361 DEBUG:boto:auth_path=/pklxkd-11/ 2023-05-31T19:58:28.361 DEBUG:boto:path=/pklxkd-11/?versions&max-keys=999 2023-05-31T19:58:28.362 DEBUG:boto:auth_path=/pklxkd-11/?versions&max-keys=999 2023-05-31T19:58:28.362 DEBUG:boto:Method: GET 2023-05-31T19:58:28.362 DEBUG:boto:Path: /pklxkd-11/?versions&max-keys=999 2023-05-31T19:58:28.363 DEBUG:boto:Data: 2023-05-31T19:58:28.363 DEBUG:boto:Headers: {} 2023-05-31T19:58:28.363 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T19:58:28.364 DEBUG:boto:Port: 8000 2023-05-31T19:58:28.364 DEBUG:boto:Params: {} 2023-05-31T19:58:28.364 DEBUG:boto:Token: None 2023-05-31T19:58:28.365 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 19:58:28 GMT /pklxkd-11/?versions 2023-05-31T19:58:28.365 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:eroZmkpWsM1LIgOrTUoOaZje1I0= 2023-05-31T19:58:28.366 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 19:58:28 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:eroZmkpWsM1LIgOrTUoOaZje1I0=', 'Content-Length': '0'} 2023-05-31T19:58:28.368 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000002f407fea2a1a6b67-006477a6e4-110e-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 19:58:28 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T19:58:28.368 DEBUG:boto:b'pklxkd-11999falsedummypuen5pSFPkhRYSzopBIdJIXBQzttKnutrue2023-05-31T19:57:12.265Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-a1L7ViqSXBtkdfIYltLrLN1BhDNjOq8xztrue2023-05-31T19:57:47.555Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-a2d.2JtI5ycFHjEYrzZkvtbFVUQjr5kaLtrue2023-05-31T19:57:47.683Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T19:58:28.369 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2023-05-31T19:58:28.370 DEBUG:boto:path=/pklxkd-11/dummy 2023-05-31T19:58:28.370 DEBUG:boto:auth_path=/pklxkd-11/dummy 2023-05-31T19:58:28.370 DEBUG:boto:path=/pklxkd-11/dummy?versionId=puen5pSFPkhRYSzopBIdJIXBQzttKnu 2023-05-31T19:58:28.371 DEBUG:boto:auth_path=/pklxkd-11/dummy?versionId=puen5pSFPkhRYSzopBIdJIXBQzttKnu 2023-05-31T19:58:28.371 DEBUG:boto:Method: GET 2023-05-31T19:58:28.371 DEBUG:boto:Path: /pklxkd-11/dummy?versionId=puen5pSFPkhRYSzopBIdJIXBQzttKnu 2023-05-31T19:58:28.372 DEBUG:boto:Data: 2023-05-31T19:58:28.372 DEBUG:boto:Headers: {} 2023-05-31T19:58:28.372 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T19:58:28.373 DEBUG:boto:Port: 8001 2023-05-31T19:58:28.373 DEBUG:boto:Params: {} 2023-05-31T19:58:28.374 DEBUG:boto:Token: None 2023-05-31T19:58:28.374 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 19:58:28 GMT /pklxkd-11/dummy?versionId=puen5pSFPkhRYSzopBIdJIXBQzttKnu 2023-05-31T19:58:28.374 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:GjiIasHg85BfSWgcFj2bgH1l+fk= 2023-05-31T19:58:28.375 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 19:58:28 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:GjiIasHg85BfSWgcFj2bgH1l+fk=', 'Content-Length': '0'} 2023-05-31T19:58:28.376 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 19:57:12 GMT'), ('x-amz-version-id', 'puen5pSFPkhRYSzopBIdJIXBQzttKnu'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx0000079fe24d850533575-006477a6e4-1112-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 19:58:28 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T19:58:28.377 DEBUG:boto:path=/pklxkd-11/dummy 2023-05-31T19:58:28.377 DEBUG:boto:auth_path=/pklxkd-11/dummy 2023-05-31T19:58:28.378 DEBUG:boto:path=/pklxkd-11/dummy?versionId=puen5pSFPkhRYSzopBIdJIXBQzttKnu 2023-05-31T19:58:28.378 DEBUG:boto:auth_path=/pklxkd-11/dummy?versionId=puen5pSFPkhRYSzopBIdJIXBQzttKnu 2023-05-31T19:58:28.379 DEBUG:boto:Method: GET 2023-05-31T19:58:28.379 DEBUG:boto:Path: /pklxkd-11/dummy?versionId=puen5pSFPkhRYSzopBIdJIXBQzttKnu 2023-05-31T19:58:28.379 DEBUG:boto:Data: 2023-05-31T19:58:28.380 DEBUG:boto:Headers: {} 2023-05-31T19:58:28.380 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T19:58:28.380 DEBUG:boto:Port: 8000 2023-05-31T19:58:28.381 DEBUG:boto:Params: {} 2023-05-31T19:58:28.381 DEBUG:boto:Token: None 2023-05-31T19:58:28.382 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 19:58:28 GMT /pklxkd-11/dummy?versionId=puen5pSFPkhRYSzopBIdJIXBQzttKnu 2023-05-31T19:58:28.382 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:GjiIasHg85BfSWgcFj2bgH1l+fk= 2023-05-31T19:58:28.382 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 19:58:28 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:GjiIasHg85BfSWgcFj2bgH1l+fk=', 'Content-Length': '0'} 2023-05-31T19:58:28.384 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 19:57:12 GMT'), ('x-amz-version-id', 'puen5pSFPkhRYSzopBIdJIXBQzttKnu'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000f106c6dca3bc1a9a-006477a6e4-110e-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 19:58:28 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T19:58:28.385 DEBUG:boto:path=/pklxkd-11/dummy 2023-05-31T19:58:28.385 DEBUG:boto:auth_path=/pklxkd-11/dummy 2023-05-31T19:58:28.385 DEBUG:boto:path=/pklxkd-11/dummy?versionId=puen5pSFPkhRYSzopBIdJIXBQzttKnu 2023-05-31T19:58:28.386 DEBUG:boto:auth_path=/pklxkd-11/dummy?versionId=puen5pSFPkhRYSzopBIdJIXBQzttKnu 2023-05-31T19:58:28.386 DEBUG:boto:Method: HEAD 2023-05-31T19:58:28.386 DEBUG:boto:Path: /pklxkd-11/dummy?versionId=puen5pSFPkhRYSzopBIdJIXBQzttKnu 2023-05-31T19:58:28.387 DEBUG:boto:Data: 2023-05-31T19:58:28.387 DEBUG:boto:Headers: {} 2023-05-31T19:58:28.387 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T19:58:28.388 DEBUG:boto:Port: 8001 2023-05-31T19:58:28.388 DEBUG:boto:Params: {} 2023-05-31T19:58:28.388 DEBUG:boto:Token: None 2023-05-31T19:58:28.389 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 19:58:28 GMT /pklxkd-11/dummy?versionId=puen5pSFPkhRYSzopBIdJIXBQzttKnu 2023-05-31T19:58:28.389 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:gJR/m6Q+tA0s+IvDsg74NnZMwj4= 2023-05-31T19:58:28.390 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 19:58:28 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:gJR/m6Q+tA0s+IvDsg74NnZMwj4=', 'Content-Length': '0'} 2023-05-31T19:58:28.391 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 19:57:12 GMT'), ('x-amz-version-id', 'puen5pSFPkhRYSzopBIdJIXBQzttKnu'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000fec608c2fbbe28d5-006477a6e4-1112-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 19:58:28 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T19:58:28.392 DEBUG:boto:path=/pklxkd-11/dummy 2023-05-31T19:58:28.392 DEBUG:boto:auth_path=/pklxkd-11/dummy 2023-05-31T19:58:28.393 DEBUG:boto:path=/pklxkd-11/dummy?versionId=puen5pSFPkhRYSzopBIdJIXBQzttKnu 2023-05-31T19:58:28.393 DEBUG:boto:auth_path=/pklxkd-11/dummy?versionId=puen5pSFPkhRYSzopBIdJIXBQzttKnu 2023-05-31T19:58:28.394 DEBUG:boto:Method: HEAD 2023-05-31T19:58:28.394 DEBUG:boto:Path: /pklxkd-11/dummy?versionId=puen5pSFPkhRYSzopBIdJIXBQzttKnu 2023-05-31T19:58:28.394 DEBUG:boto:Data: 2023-05-31T19:58:28.395 DEBUG:boto:Headers: {} 2023-05-31T19:58:28.395 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T19:58:28.395 DEBUG:boto:Port: 8000 2023-05-31T19:58:28.396 DEBUG:boto:Params: {} 2023-05-31T19:58:28.396 DEBUG:boto:Token: None 2023-05-31T19:58:28.397 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 19:58:28 GMT /pklxkd-11/dummy?versionId=puen5pSFPkhRYSzopBIdJIXBQzttKnu 2023-05-31T19:58:28.397 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:gJR/m6Q+tA0s+IvDsg74NnZMwj4= 2023-05-31T19:58:28.397 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 19:58:28 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:gJR/m6Q+tA0s+IvDsg74NnZMwj4=', 'Content-Length': '0'} 2023-05-31T19:58:28.399 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 19:57:12 GMT'), ('x-amz-version-id', 'puen5pSFPkhRYSzopBIdJIXBQzttKnu'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000e359876c0831248e-006477a6e4-110e-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 19:58:28 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T19:58:28.399 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2023-05-31T19:58:28.400 DEBUG:boto:path=/pklxkd-11/dummy 2023-05-31T19:58:28.400 DEBUG:boto:auth_path=/pklxkd-11/dummy 2023-05-31T19:58:28.401 DEBUG:boto:path=/pklxkd-11/dummy?versionId=puen5pSFPkhRYSzopBIdJIXBQzttKnu 2023-05-31T19:58:28.401 DEBUG:boto:auth_path=/pklxkd-11/dummy?versionId=puen5pSFPkhRYSzopBIdJIXBQzttKnu 2023-05-31T19:58:28.401 DEBUG:boto:Method: GET 2023-05-31T19:58:28.402 DEBUG:boto:Path: /pklxkd-11/dummy?versionId=puen5pSFPkhRYSzopBIdJIXBQzttKnu 2023-05-31T19:58:28.402 DEBUG:boto:Data: 2023-05-31T19:58:28.402 DEBUG:boto:Headers: {} 2023-05-31T19:58:28.403 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T19:58:28.403 DEBUG:boto:Port: 8001 2023-05-31T19:58:28.403 DEBUG:boto:Params: {} 2023-05-31T19:58:28.404 DEBUG:boto:Token: None 2023-05-31T19:58:28.404 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 19:58:28 GMT /pklxkd-11/dummy?versionId=puen5pSFPkhRYSzopBIdJIXBQzttKnu 2023-05-31T19:58:28.405 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:GjiIasHg85BfSWgcFj2bgH1l+fk= 2023-05-31T19:58:28.405 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 19:58:28 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:GjiIasHg85BfSWgcFj2bgH1l+fk=', 'Content-Length': '0'} 2023-05-31T19:58:28.407 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 19:57:12 GMT'), ('x-amz-version-id', 'puen5pSFPkhRYSzopBIdJIXBQzttKnu'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000001ac7bfd874e99b33-006477a6e4-1112-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 19:58:28 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T19:58:28.407 DEBUG:boto:path=/pklxkd-11/dummy 2023-05-31T19:58:28.408 DEBUG:boto:auth_path=/pklxkd-11/dummy 2023-05-31T19:58:28.408 DEBUG:boto:path=/pklxkd-11/dummy?versionId=puen5pSFPkhRYSzopBIdJIXBQzttKnu 2023-05-31T19:58:28.409 DEBUG:boto:auth_path=/pklxkd-11/dummy?versionId=puen5pSFPkhRYSzopBIdJIXBQzttKnu 2023-05-31T19:58:28.409 DEBUG:boto:Method: GET 2023-05-31T19:58:28.409 DEBUG:boto:Path: /pklxkd-11/dummy?versionId=puen5pSFPkhRYSzopBIdJIXBQzttKnu 2023-05-31T19:58:28.410 DEBUG:boto:Data: 2023-05-31T19:58:28.410 DEBUG:boto:Headers: {} 2023-05-31T19:58:28.410 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T19:58:28.411 DEBUG:boto:Port: 8000 2023-05-31T19:58:28.411 DEBUG:boto:Params: {} 2023-05-31T19:58:28.411 DEBUG:boto:Token: None 2023-05-31T19:58:28.412 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 19:58:28 GMT /pklxkd-11/dummy?versionId=puen5pSFPkhRYSzopBIdJIXBQzttKnu 2023-05-31T19:58:28.412 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:GjiIasHg85BfSWgcFj2bgH1l+fk= 2023-05-31T19:58:28.413 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 19:58:28 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:GjiIasHg85BfSWgcFj2bgH1l+fk=', 'Content-Length': '0'} 2023-05-31T19:58:28.414 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 19:57:12 GMT'), ('x-amz-version-id', 'puen5pSFPkhRYSzopBIdJIXBQzttKnu'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx0000032c80e680d341972-006477a6e4-110e-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 19:58:28 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T19:58:28.415 DEBUG:boto:path=/pklxkd-11/ 2023-05-31T19:58:28.415 DEBUG:boto:auth_path=/pklxkd-11/ 2023-05-31T19:58:28.416 DEBUG:boto:path=/pklxkd-11/?prefix=dummy 2023-05-31T19:58:28.416 DEBUG:boto:auth_path=/pklxkd-11/?prefix=dummy 2023-05-31T19:58:28.416 DEBUG:boto:Method: GET 2023-05-31T19:58:28.417 DEBUG:boto:Path: /pklxkd-11/?prefix=dummy 2023-05-31T19:58:28.417 DEBUG:boto:Data: 2023-05-31T19:58:28.417 DEBUG:boto:Headers: {} 2023-05-31T19:58:28.418 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T19:58:28.418 DEBUG:boto:Port: 8001 2023-05-31T19:58:28.419 DEBUG:boto:Params: {} 2023-05-31T19:58:28.419 DEBUG:boto:Token: None 2023-05-31T19:58:28.419 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 19:58:28 GMT /pklxkd-11/ 2023-05-31T19:58:28.420 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:cd0GuYT/8SKWQPD06IPg+xJIWY4= 2023-05-31T19:58:28.420 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 19:58:28 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:cd0GuYT/8SKWQPD06IPg+xJIWY4=', 'Content-Length': '0'} 2023-05-31T19:58:28.422 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d692eccab5a7c044-006477a6e4-1112-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 19:58:28 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T19:58:28.468 DEBUG:boto:b'pklxkd-11dummy1000falsedummy2023-05-31T19:57:12.265Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T19:58:28.469 DEBUG:boto:path=/pklxkd-11/ 2023-05-31T19:58:28.469 DEBUG:boto:auth_path=/pklxkd-11/ 2023-05-31T19:58:28.470 DEBUG:boto:path=/pklxkd-11/?prefix=dummy 2023-05-31T19:58:28.470 DEBUG:boto:auth_path=/pklxkd-11/?prefix=dummy 2023-05-31T19:58:28.471 DEBUG:boto:Method: GET 2023-05-31T19:58:28.471 DEBUG:boto:Path: /pklxkd-11/?prefix=dummy 2023-05-31T19:58:28.471 DEBUG:boto:Data: 2023-05-31T19:58:28.472 DEBUG:boto:Headers: {} 2023-05-31T19:58:28.472 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T19:58:28.472 DEBUG:boto:Port: 8000 2023-05-31T19:58:28.473 DEBUG:boto:Params: {} 2023-05-31T19:58:28.473 DEBUG:boto:Token: None 2023-05-31T19:58:28.474 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 19:58:28 GMT /pklxkd-11/ 2023-05-31T19:58:28.474 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:cd0GuYT/8SKWQPD06IPg+xJIWY4= 2023-05-31T19:58:28.474 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 19:58:28 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:cd0GuYT/8SKWQPD06IPg+xJIWY4=', 'Content-Length': '0'} 2023-05-31T19:58:28.476 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000018e1c50ab91a403-006477a6e4-110e-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 19:58:28 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T19:58:28.477 DEBUG:boto:b'pklxkd-11dummy1000falsedummy2023-05-31T19:57:12.265Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T19:58:28.478 DEBUG:boto:path=/pklxkd-11/dummy 2023-05-31T19:58:28.478 DEBUG:boto:auth_path=/pklxkd-11/dummy 2023-05-31T19:58:28.478 DEBUG:boto:Method: HEAD 2023-05-31T19:58:28.479 DEBUG:boto:Path: /pklxkd-11/dummy 2023-05-31T19:58:28.479 DEBUG:boto:Data: 2023-05-31T19:58:28.479 DEBUG:boto:Headers: {} 2023-05-31T19:58:28.480 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T19:58:28.480 DEBUG:boto:Port: 8001 2023-05-31T19:58:28.480 DEBUG:boto:Params: {} 2023-05-31T19:58:28.481 DEBUG:boto:Token: None 2023-05-31T19:58:28.481 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 19:58:28 GMT /pklxkd-11/dummy 2023-05-31T19:58:28.482 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:LlQscpkF1VmtM4IG3V4dP0hE6UA= 2023-05-31T19:58:28.482 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 19:58:28 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:LlQscpkF1VmtM4IG3V4dP0hE6UA=', 'Content-Length': '0'} 2023-05-31T19:58:28.484 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 19:57:12 GMT'), ('x-amz-version-id', 'puen5pSFPkhRYSzopBIdJIXBQzttKnu'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000db713aff5fef3def-006477a6e4-1112-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 19:58:28 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T19:58:28.485 DEBUG:boto:path=/pklxkd-11/dummy 2023-05-31T19:58:28.485 DEBUG:boto:auth_path=/pklxkd-11/dummy 2023-05-31T19:58:28.485 DEBUG:boto:Method: HEAD 2023-05-31T19:58:28.486 DEBUG:boto:Path: /pklxkd-11/dummy 2023-05-31T19:58:28.486 DEBUG:boto:Data: 2023-05-31T19:58:28.486 DEBUG:boto:Headers: {} 2023-05-31T19:58:28.487 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T19:58:28.487 DEBUG:boto:Port: 8000 2023-05-31T19:58:28.487 DEBUG:boto:Params: {} 2023-05-31T19:58:28.488 DEBUG:boto:Token: None 2023-05-31T19:58:28.488 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 19:58:28 GMT /pklxkd-11/dummy 2023-05-31T19:58:28.489 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:LlQscpkF1VmtM4IG3V4dP0hE6UA= 2023-05-31T19:58:28.489 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 19:58:28 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:LlQscpkF1VmtM4IG3V4dP0hE6UA=', 'Content-Length': '0'} 2023-05-31T19:58:28.491 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 19:57:12 GMT'), ('x-amz-version-id', 'puen5pSFPkhRYSzopBIdJIXBQzttKnu'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000004dd7762c41c20328-006477a6e4-110e-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 19:58:28 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T19:58:28.491 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2023-05-31T19:58:28.492 DEBUG:boto:path=/pklxkd-11/dummy 2023-05-31T19:58:28.492 DEBUG:boto:auth_path=/pklxkd-11/dummy 2023-05-31T19:58:28.492 DEBUG:boto:path=/pklxkd-11/dummy?versionId=puen5pSFPkhRYSzopBIdJIXBQzttKnu 2023-05-31T19:58:28.493 DEBUG:boto:auth_path=/pklxkd-11/dummy?versionId=puen5pSFPkhRYSzopBIdJIXBQzttKnu 2023-05-31T19:58:28.493 DEBUG:boto:Method: GET 2023-05-31T19:58:28.494 DEBUG:boto:Path: /pklxkd-11/dummy?versionId=puen5pSFPkhRYSzopBIdJIXBQzttKnu 2023-05-31T19:58:28.494 DEBUG:boto:Data: 2023-05-31T19:58:28.494 DEBUG:boto:Headers: {} 2023-05-31T19:58:28.495 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T19:58:28.495 DEBUG:boto:Port: 8001 2023-05-31T19:58:28.495 DEBUG:boto:Params: {} 2023-05-31T19:58:28.496 DEBUG:boto:Token: None 2023-05-31T19:58:28.496 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 19:58:28 GMT /pklxkd-11/dummy?versionId=puen5pSFPkhRYSzopBIdJIXBQzttKnu 2023-05-31T19:58:28.497 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:GjiIasHg85BfSWgcFj2bgH1l+fk= 2023-05-31T19:58:28.497 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 19:58:28 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:GjiIasHg85BfSWgcFj2bgH1l+fk=', 'Content-Length': '0'} 2023-05-31T19:58:28.499 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 19:57:12 GMT'), ('x-amz-version-id', 'puen5pSFPkhRYSzopBIdJIXBQzttKnu'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000a809c1a790ed6a78-006477a6e4-1112-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 19:58:28 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T19:58:28.499 DEBUG:boto:path=/pklxkd-11/dummy 2023-05-31T19:58:28.500 DEBUG:boto:auth_path=/pklxkd-11/dummy 2023-05-31T19:58:28.500 DEBUG:boto:path=/pklxkd-11/dummy?versionId=puen5pSFPkhRYSzopBIdJIXBQzttKnu 2023-05-31T19:58:28.500 DEBUG:boto:auth_path=/pklxkd-11/dummy?versionId=puen5pSFPkhRYSzopBIdJIXBQzttKnu 2023-05-31T19:58:28.501 DEBUG:boto:Method: GET 2023-05-31T19:58:28.501 DEBUG:boto:Path: /pklxkd-11/dummy?versionId=puen5pSFPkhRYSzopBIdJIXBQzttKnu 2023-05-31T19:58:28.502 DEBUG:boto:Data: 2023-05-31T19:58:28.502 DEBUG:boto:Headers: {} 2023-05-31T19:58:28.502 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T19:58:28.503 DEBUG:boto:Port: 8000 2023-05-31T19:58:28.503 DEBUG:boto:Params: {} 2023-05-31T19:58:28.503 DEBUG:boto:Token: None 2023-05-31T19:58:28.504 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 19:58:28 GMT /pklxkd-11/dummy?versionId=puen5pSFPkhRYSzopBIdJIXBQzttKnu 2023-05-31T19:58:28.504 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:GjiIasHg85BfSWgcFj2bgH1l+fk= 2023-05-31T19:58:28.505 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 19:58:28 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:GjiIasHg85BfSWgcFj2bgH1l+fk=', 'Content-Length': '0'} 2023-05-31T19:58:28.506 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 19:57:12 GMT'), ('x-amz-version-id', 'puen5pSFPkhRYSzopBIdJIXBQzttKnu'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx0000058302769a21803b2-006477a6e4-110e-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 19:58:28 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T19:58:28.507 DEBUG:boto:path=/pklxkd-11/ 2023-05-31T19:58:28.507 DEBUG:boto:auth_path=/pklxkd-11/ 2023-05-31T19:58:28.508 DEBUG:boto:path=/pklxkd-11/?prefix=dummy 2023-05-31T19:58:28.508 DEBUG:boto:auth_path=/pklxkd-11/?prefix=dummy 2023-05-31T19:58:28.508 DEBUG:boto:Method: GET 2023-05-31T19:58:28.509 DEBUG:boto:Path: /pklxkd-11/?prefix=dummy 2023-05-31T19:58:28.509 DEBUG:boto:Data: 2023-05-31T19:58:28.509 DEBUG:boto:Headers: {} 2023-05-31T19:58:28.510 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T19:58:28.510 DEBUG:boto:Port: 8001 2023-05-31T19:58:28.510 DEBUG:boto:Params: {} 2023-05-31T19:58:28.511 DEBUG:boto:Token: None 2023-05-31T19:58:28.511 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 19:58:28 GMT /pklxkd-11/ 2023-05-31T19:58:28.512 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:cd0GuYT/8SKWQPD06IPg+xJIWY4= 2023-05-31T19:58:28.512 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 19:58:28 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:cd0GuYT/8SKWQPD06IPg+xJIWY4=', 'Content-Length': '0'} 2023-05-31T19:58:28.514 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000fea1f8affd17ca32-006477a6e4-1112-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 19:58:28 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T19:58:28.514 DEBUG:boto:b'pklxkd-11dummy1000falsedummy2023-05-31T19:57:12.265Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T19:58:28.515 DEBUG:boto:path=/pklxkd-11/ 2023-05-31T19:58:28.516 DEBUG:boto:auth_path=/pklxkd-11/ 2023-05-31T19:58:28.516 DEBUG:boto:path=/pklxkd-11/?prefix=dummy 2023-05-31T19:58:28.516 DEBUG:boto:auth_path=/pklxkd-11/?prefix=dummy 2023-05-31T19:58:28.517 DEBUG:boto:Method: GET 2023-05-31T19:58:28.517 DEBUG:boto:Path: /pklxkd-11/?prefix=dummy 2023-05-31T19:58:28.517 DEBUG:boto:Data: 2023-05-31T19:58:28.518 DEBUG:boto:Headers: {} 2023-05-31T19:58:28.518 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T19:58:28.518 DEBUG:boto:Port: 8000 2023-05-31T19:58:28.519 DEBUG:boto:Params: {} 2023-05-31T19:58:28.519 DEBUG:boto:Token: None 2023-05-31T19:58:28.520 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 19:58:28 GMT /pklxkd-11/ 2023-05-31T19:58:28.520 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:cd0GuYT/8SKWQPD06IPg+xJIWY4= 2023-05-31T19:58:28.520 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 19:58:28 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:cd0GuYT/8SKWQPD06IPg+xJIWY4=', 'Content-Length': '0'} 2023-05-31T19:58:28.522 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000022dd9abffa3fa28f-006477a6e4-110e-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 19:58:28 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T19:58:28.523 DEBUG:boto:b'pklxkd-11dummy1000falsedummy2023-05-31T19:57:12.265Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T19:58:28.523 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-a1 2023-05-31T19:58:28.524 DEBUG:boto:path=/pklxkd-11/obj-a1 2023-05-31T19:58:28.524 DEBUG:boto:auth_path=/pklxkd-11/obj-a1 2023-05-31T19:58:28.524 DEBUG:boto:path=/pklxkd-11/obj-a1?versionId=L7ViqSXBtkdfIYltLrLN1BhDNjOq8xz 2023-05-31T19:58:28.525 DEBUG:boto:auth_path=/pklxkd-11/obj-a1?versionId=L7ViqSXBtkdfIYltLrLN1BhDNjOq8xz 2023-05-31T19:58:28.525 DEBUG:boto:Method: GET 2023-05-31T19:58:28.526 DEBUG:boto:Path: /pklxkd-11/obj-a1?versionId=L7ViqSXBtkdfIYltLrLN1BhDNjOq8xz 2023-05-31T19:58:28.526 DEBUG:boto:Data: 2023-05-31T19:58:28.526 DEBUG:boto:Headers: {} 2023-05-31T19:58:28.527 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T19:58:28.527 DEBUG:boto:Port: 8001 2023-05-31T19:58:28.527 DEBUG:boto:Params: {} 2023-05-31T19:58:28.528 DEBUG:boto:Token: None 2023-05-31T19:58:28.528 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 19:58:28 GMT /pklxkd-11/obj-a1?versionId=L7ViqSXBtkdfIYltLrLN1BhDNjOq8xz 2023-05-31T19:58:28.529 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:zNvaEyW+m/BqA75GapOwL+3+JI0= 2023-05-31T19:58:28.529 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 19:58:28 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:zNvaEyW+m/BqA75GapOwL+3+JI0=', 'Content-Length': '0'} 2023-05-31T19:58:28.531 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 19:57:47 GMT'), ('x-amz-version-id', 'L7ViqSXBtkdfIYltLrLN1BhDNjOq8xz'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000004d356144a2358117-006477a6e4-1112-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 19:58:28 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T19:58:28.532 DEBUG:boto:path=/pklxkd-11/obj-a1 2023-05-31T19:58:28.532 DEBUG:boto:auth_path=/pklxkd-11/obj-a1 2023-05-31T19:58:28.532 DEBUG:boto:path=/pklxkd-11/obj-a1?versionId=L7ViqSXBtkdfIYltLrLN1BhDNjOq8xz 2023-05-31T19:58:28.533 DEBUG:boto:auth_path=/pklxkd-11/obj-a1?versionId=L7ViqSXBtkdfIYltLrLN1BhDNjOq8xz 2023-05-31T19:58:28.533 DEBUG:boto:Method: GET 2023-05-31T19:58:28.533 DEBUG:boto:Path: /pklxkd-11/obj-a1?versionId=L7ViqSXBtkdfIYltLrLN1BhDNjOq8xz 2023-05-31T19:58:28.534 DEBUG:boto:Data: 2023-05-31T19:58:28.534 DEBUG:boto:Headers: {} 2023-05-31T19:58:28.534 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T19:58:28.535 DEBUG:boto:Port: 8000 2023-05-31T19:58:28.535 DEBUG:boto:Params: {} 2023-05-31T19:58:28.535 DEBUG:boto:Token: None 2023-05-31T19:58:28.536 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 19:58:28 GMT /pklxkd-11/obj-a1?versionId=L7ViqSXBtkdfIYltLrLN1BhDNjOq8xz 2023-05-31T19:58:28.536 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:zNvaEyW+m/BqA75GapOwL+3+JI0= 2023-05-31T19:58:28.537 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 19:58:28 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:zNvaEyW+m/BqA75GapOwL+3+JI0=', 'Content-Length': '0'} 2023-05-31T19:58:28.538 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 19:57:47 GMT'), ('x-amz-version-id', 'L7ViqSXBtkdfIYltLrLN1BhDNjOq8xz'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000b8b28294bc6ec8b6-006477a6e4-110e-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 19:58:28 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T19:58:28.539 DEBUG:boto:path=/pklxkd-11/obj-a1 2023-05-31T19:58:28.539 DEBUG:boto:auth_path=/pklxkd-11/obj-a1 2023-05-31T19:58:28.540 DEBUG:boto:path=/pklxkd-11/obj-a1?versionId=L7ViqSXBtkdfIYltLrLN1BhDNjOq8xz 2023-05-31T19:58:28.540 DEBUG:boto:auth_path=/pklxkd-11/obj-a1?versionId=L7ViqSXBtkdfIYltLrLN1BhDNjOq8xz 2023-05-31T19:58:28.540 DEBUG:boto:Method: HEAD 2023-05-31T19:58:28.541 DEBUG:boto:Path: /pklxkd-11/obj-a1?versionId=L7ViqSXBtkdfIYltLrLN1BhDNjOq8xz 2023-05-31T19:58:28.541 DEBUG:boto:Data: 2023-05-31T19:58:28.541 DEBUG:boto:Headers: {} 2023-05-31T19:58:28.542 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T19:58:28.542 DEBUG:boto:Port: 8001 2023-05-31T19:58:28.542 DEBUG:boto:Params: {} 2023-05-31T19:58:28.543 DEBUG:boto:Token: None 2023-05-31T19:58:28.543 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 19:58:28 GMT /pklxkd-11/obj-a1?versionId=L7ViqSXBtkdfIYltLrLN1BhDNjOq8xz 2023-05-31T19:58:28.543 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:kjH5ujj2c2AP/P4x2ZJnqhzes6o= 2023-05-31T19:58:28.544 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 19:58:28 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:kjH5ujj2c2AP/P4x2ZJnqhzes6o=', 'Content-Length': '0'} 2023-05-31T19:58:28.546 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 19:57:47 GMT'), ('x-amz-version-id', 'L7ViqSXBtkdfIYltLrLN1BhDNjOq8xz'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000009f2b890549b5dae6-006477a6e4-1112-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 19:58:28 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T19:58:28.546 DEBUG:boto:path=/pklxkd-11/obj-a1 2023-05-31T19:58:28.546 DEBUG:boto:auth_path=/pklxkd-11/obj-a1 2023-05-31T19:58:28.547 DEBUG:boto:path=/pklxkd-11/obj-a1?versionId=L7ViqSXBtkdfIYltLrLN1BhDNjOq8xz 2023-05-31T19:58:28.547 DEBUG:boto:auth_path=/pklxkd-11/obj-a1?versionId=L7ViqSXBtkdfIYltLrLN1BhDNjOq8xz 2023-05-31T19:58:28.547 DEBUG:boto:Method: HEAD 2023-05-31T19:58:28.548 DEBUG:boto:Path: /pklxkd-11/obj-a1?versionId=L7ViqSXBtkdfIYltLrLN1BhDNjOq8xz 2023-05-31T19:58:28.548 DEBUG:boto:Data: 2023-05-31T19:58:28.548 DEBUG:boto:Headers: {} 2023-05-31T19:58:28.549 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T19:58:28.549 DEBUG:boto:Port: 8000 2023-05-31T19:58:28.549 DEBUG:boto:Params: {} 2023-05-31T19:58:28.550 DEBUG:boto:Token: None 2023-05-31T19:58:28.550 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 19:58:28 GMT /pklxkd-11/obj-a1?versionId=L7ViqSXBtkdfIYltLrLN1BhDNjOq8xz 2023-05-31T19:58:28.551 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:kjH5ujj2c2AP/P4x2ZJnqhzes6o= 2023-05-31T19:58:28.551 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 19:58:28 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:kjH5ujj2c2AP/P4x2ZJnqhzes6o=', 'Content-Length': '0'} 2023-05-31T19:58:28.553 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 19:57:47 GMT'), ('x-amz-version-id', 'L7ViqSXBtkdfIYltLrLN1BhDNjOq8xz'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000ce392140d88b43cf-006477a6e4-110e-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 19:58:28 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T19:58:28.553 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-a1 2023-05-31T19:58:28.554 DEBUG:boto:path=/pklxkd-11/obj-a1 2023-05-31T19:58:28.554 DEBUG:boto:auth_path=/pklxkd-11/obj-a1 2023-05-31T19:58:28.554 DEBUG:boto:path=/pklxkd-11/obj-a1?versionId=L7ViqSXBtkdfIYltLrLN1BhDNjOq8xz 2023-05-31T19:58:28.555 DEBUG:boto:auth_path=/pklxkd-11/obj-a1?versionId=L7ViqSXBtkdfIYltLrLN1BhDNjOq8xz 2023-05-31T19:58:28.555 DEBUG:boto:Method: GET 2023-05-31T19:58:28.556 DEBUG:boto:Path: /pklxkd-11/obj-a1?versionId=L7ViqSXBtkdfIYltLrLN1BhDNjOq8xz 2023-05-31T19:58:28.556 DEBUG:boto:Data: 2023-05-31T19:58:28.556 DEBUG:boto:Headers: {} 2023-05-31T19:58:28.557 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T19:58:28.557 DEBUG:boto:Port: 8001 2023-05-31T19:58:28.557 DEBUG:boto:Params: {} 2023-05-31T19:58:28.558 DEBUG:boto:Token: None 2023-05-31T19:58:28.558 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 19:58:28 GMT /pklxkd-11/obj-a1?versionId=L7ViqSXBtkdfIYltLrLN1BhDNjOq8xz 2023-05-31T19:58:28.559 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:zNvaEyW+m/BqA75GapOwL+3+JI0= 2023-05-31T19:58:28.559 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 19:58:28 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:zNvaEyW+m/BqA75GapOwL+3+JI0=', 'Content-Length': '0'} 2023-05-31T19:58:28.560 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 19:57:47 GMT'), ('x-amz-version-id', 'L7ViqSXBtkdfIYltLrLN1BhDNjOq8xz'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000b0d50a78a1f372e7-006477a6e4-1112-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 19:58:28 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T19:58:28.561 DEBUG:boto:path=/pklxkd-11/obj-a1 2023-05-31T19:58:28.561 DEBUG:boto:auth_path=/pklxkd-11/obj-a1 2023-05-31T19:58:28.562 DEBUG:boto:path=/pklxkd-11/obj-a1?versionId=L7ViqSXBtkdfIYltLrLN1BhDNjOq8xz 2023-05-31T19:58:28.562 DEBUG:boto:auth_path=/pklxkd-11/obj-a1?versionId=L7ViqSXBtkdfIYltLrLN1BhDNjOq8xz 2023-05-31T19:58:28.563 DEBUG:boto:Method: GET 2023-05-31T19:58:28.563 DEBUG:boto:Path: /pklxkd-11/obj-a1?versionId=L7ViqSXBtkdfIYltLrLN1BhDNjOq8xz 2023-05-31T19:58:28.563 DEBUG:boto:Data: 2023-05-31T19:58:28.564 DEBUG:boto:Headers: {} 2023-05-31T19:58:28.564 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T19:58:28.564 DEBUG:boto:Port: 8000 2023-05-31T19:58:28.565 DEBUG:boto:Params: {} 2023-05-31T19:58:28.565 DEBUG:boto:Token: None 2023-05-31T19:58:28.565 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 19:58:28 GMT /pklxkd-11/obj-a1?versionId=L7ViqSXBtkdfIYltLrLN1BhDNjOq8xz 2023-05-31T19:58:28.566 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:zNvaEyW+m/BqA75GapOwL+3+JI0= 2023-05-31T19:58:28.566 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 19:58:28 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:zNvaEyW+m/BqA75GapOwL+3+JI0=', 'Content-Length': '0'} 2023-05-31T19:58:28.568 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 19:57:47 GMT'), ('x-amz-version-id', 'L7ViqSXBtkdfIYltLrLN1BhDNjOq8xz'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000bedc1f41cfa2a00a-006477a6e4-110e-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 19:58:28 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T19:58:28.608 DEBUG:boto:path=/pklxkd-11/ 2023-05-31T19:58:28.609 DEBUG:boto:auth_path=/pklxkd-11/ 2023-05-31T19:58:28.609 DEBUG:boto:path=/pklxkd-11/?prefix=obj-a1 2023-05-31T19:58:28.609 DEBUG:boto:auth_path=/pklxkd-11/?prefix=obj-a1 2023-05-31T19:58:28.610 DEBUG:boto:Method: GET 2023-05-31T19:58:28.610 DEBUG:boto:Path: /pklxkd-11/?prefix=obj-a1 2023-05-31T19:58:28.610 DEBUG:boto:Data: 2023-05-31T19:58:28.611 DEBUG:boto:Headers: {} 2023-05-31T19:58:28.611 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T19:58:28.612 DEBUG:boto:Port: 8001 2023-05-31T19:58:28.612 DEBUG:boto:Params: {} 2023-05-31T19:58:28.612 DEBUG:boto:Token: None 2023-05-31T19:58:28.613 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 19:58:28 GMT /pklxkd-11/ 2023-05-31T19:58:28.613 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:cd0GuYT/8SKWQPD06IPg+xJIWY4= 2023-05-31T19:58:28.614 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 19:58:28 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:cd0GuYT/8SKWQPD06IPg+xJIWY4=', 'Content-Length': '0'} 2023-05-31T19:58:28.616 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000006e3e00e61863adce-006477a6e4-1112-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 19:58:28 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T19:58:28.616 DEBUG:boto:b'pklxkd-11obj-a11000falseobj-a12023-05-31T19:57:47.555Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T19:58:28.617 DEBUG:boto:path=/pklxkd-11/ 2023-05-31T19:58:28.617 DEBUG:boto:auth_path=/pklxkd-11/ 2023-05-31T19:58:28.618 DEBUG:boto:path=/pklxkd-11/?prefix=obj-a1 2023-05-31T19:58:28.618 DEBUG:boto:auth_path=/pklxkd-11/?prefix=obj-a1 2023-05-31T19:58:28.619 DEBUG:boto:Method: GET 2023-05-31T19:58:28.619 DEBUG:boto:Path: /pklxkd-11/?prefix=obj-a1 2023-05-31T19:58:28.619 DEBUG:boto:Data: 2023-05-31T19:58:28.620 DEBUG:boto:Headers: {} 2023-05-31T19:58:28.620 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T19:58:28.620 DEBUG:boto:Port: 8000 2023-05-31T19:58:28.621 DEBUG:boto:Params: {} 2023-05-31T19:58:28.621 DEBUG:boto:Token: None 2023-05-31T19:58:28.622 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 19:58:28 GMT /pklxkd-11/ 2023-05-31T19:58:28.622 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:cd0GuYT/8SKWQPD06IPg+xJIWY4= 2023-05-31T19:58:28.622 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 19:58:28 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:cd0GuYT/8SKWQPD06IPg+xJIWY4=', 'Content-Length': '0'} 2023-05-31T19:58:28.624 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ac27a5890d2db260-006477a6e4-110e-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 19:58:28 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T19:58:28.625 DEBUG:boto:b'pklxkd-11obj-a11000falseobj-a12023-05-31T19:57:47.555Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T19:58:28.626 DEBUG:boto:path=/pklxkd-11/obj-a1 2023-05-31T19:58:28.626 DEBUG:boto:auth_path=/pklxkd-11/obj-a1 2023-05-31T19:58:28.626 DEBUG:boto:Method: HEAD 2023-05-31T19:58:28.627 DEBUG:boto:Path: /pklxkd-11/obj-a1 2023-05-31T19:58:28.627 DEBUG:boto:Data: 2023-05-31T19:58:28.627 DEBUG:boto:Headers: {} 2023-05-31T19:58:28.628 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T19:58:28.628 DEBUG:boto:Port: 8001 2023-05-31T19:58:28.628 DEBUG:boto:Params: {} 2023-05-31T19:58:28.629 DEBUG:boto:Token: None 2023-05-31T19:58:28.629 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 19:58:28 GMT /pklxkd-11/obj-a1 2023-05-31T19:58:28.630 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:1IoT+1vokhTtCmB3PiP6/eegTnQ= 2023-05-31T19:58:28.630 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 19:58:28 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:1IoT+1vokhTtCmB3PiP6/eegTnQ=', 'Content-Length': '0'} 2023-05-31T19:58:28.632 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 19:57:47 GMT'), ('x-amz-version-id', 'L7ViqSXBtkdfIYltLrLN1BhDNjOq8xz'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000094bf32b0d6379e48-006477a6e4-1112-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 19:58:28 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T19:58:28.633 DEBUG:boto:path=/pklxkd-11/obj-a1 2023-05-31T19:58:28.633 DEBUG:boto:auth_path=/pklxkd-11/obj-a1 2023-05-31T19:58:28.633 DEBUG:boto:Method: HEAD 2023-05-31T19:58:28.634 DEBUG:boto:Path: /pklxkd-11/obj-a1 2023-05-31T19:58:28.634 DEBUG:boto:Data: 2023-05-31T19:58:28.634 DEBUG:boto:Headers: {} 2023-05-31T19:58:28.635 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T19:58:28.635 DEBUG:boto:Port: 8000 2023-05-31T19:58:28.635 DEBUG:boto:Params: {} 2023-05-31T19:58:28.636 DEBUG:boto:Token: None 2023-05-31T19:58:28.636 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 19:58:28 GMT /pklxkd-11/obj-a1 2023-05-31T19:58:28.637 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:1IoT+1vokhTtCmB3PiP6/eegTnQ= 2023-05-31T19:58:28.637 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 19:58:28 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:1IoT+1vokhTtCmB3PiP6/eegTnQ=', 'Content-Length': '0'} 2023-05-31T19:58:28.639 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 19:57:47 GMT'), ('x-amz-version-id', 'L7ViqSXBtkdfIYltLrLN1BhDNjOq8xz'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000000d8e2a9b2ae5f15b-006477a6e4-110e-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 19:58:28 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T19:58:28.639 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-a1 2023-05-31T19:58:28.640 DEBUG:boto:path=/pklxkd-11/obj-a1 2023-05-31T19:58:28.640 DEBUG:boto:auth_path=/pklxkd-11/obj-a1 2023-05-31T19:58:28.641 DEBUG:boto:path=/pklxkd-11/obj-a1?versionId=L7ViqSXBtkdfIYltLrLN1BhDNjOq8xz 2023-05-31T19:58:28.641 DEBUG:boto:auth_path=/pklxkd-11/obj-a1?versionId=L7ViqSXBtkdfIYltLrLN1BhDNjOq8xz 2023-05-31T19:58:28.641 DEBUG:boto:Method: GET 2023-05-31T19:58:28.642 DEBUG:boto:Path: /pklxkd-11/obj-a1?versionId=L7ViqSXBtkdfIYltLrLN1BhDNjOq8xz 2023-05-31T19:58:28.642 DEBUG:boto:Data: 2023-05-31T19:58:28.642 DEBUG:boto:Headers: {} 2023-05-31T19:58:28.643 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T19:58:28.643 DEBUG:boto:Port: 8001 2023-05-31T19:58:28.643 DEBUG:boto:Params: {} 2023-05-31T19:58:28.644 DEBUG:boto:Token: None 2023-05-31T19:58:28.644 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 19:58:28 GMT /pklxkd-11/obj-a1?versionId=L7ViqSXBtkdfIYltLrLN1BhDNjOq8xz 2023-05-31T19:58:28.645 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:zNvaEyW+m/BqA75GapOwL+3+JI0= 2023-05-31T19:58:28.645 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 19:58:28 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:zNvaEyW+m/BqA75GapOwL+3+JI0=', 'Content-Length': '0'} 2023-05-31T19:58:28.647 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 19:57:47 GMT'), ('x-amz-version-id', 'L7ViqSXBtkdfIYltLrLN1BhDNjOq8xz'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000020743b75dc06f50c-006477a6e4-1112-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 19:58:28 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T19:58:28.647 DEBUG:boto:path=/pklxkd-11/obj-a1 2023-05-31T19:58:28.648 DEBUG:boto:auth_path=/pklxkd-11/obj-a1 2023-05-31T19:58:28.648 DEBUG:boto:path=/pklxkd-11/obj-a1?versionId=L7ViqSXBtkdfIYltLrLN1BhDNjOq8xz 2023-05-31T19:58:28.648 DEBUG:boto:auth_path=/pklxkd-11/obj-a1?versionId=L7ViqSXBtkdfIYltLrLN1BhDNjOq8xz 2023-05-31T19:58:28.649 DEBUG:boto:Method: GET 2023-05-31T19:58:28.649 DEBUG:boto:Path: /pklxkd-11/obj-a1?versionId=L7ViqSXBtkdfIYltLrLN1BhDNjOq8xz 2023-05-31T19:58:28.649 DEBUG:boto:Data: 2023-05-31T19:58:28.650 DEBUG:boto:Headers: {} 2023-05-31T19:58:28.650 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T19:58:28.651 DEBUG:boto:Port: 8000 2023-05-31T19:58:28.651 DEBUG:boto:Params: {} 2023-05-31T19:58:28.651 DEBUG:boto:Token: None 2023-05-31T19:58:28.652 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 19:58:28 GMT /pklxkd-11/obj-a1?versionId=L7ViqSXBtkdfIYltLrLN1BhDNjOq8xz 2023-05-31T19:58:28.652 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:zNvaEyW+m/BqA75GapOwL+3+JI0= 2023-05-31T19:58:28.653 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 19:58:28 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:zNvaEyW+m/BqA75GapOwL+3+JI0=', 'Content-Length': '0'} 2023-05-31T19:58:28.654 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 19:57:47 GMT'), ('x-amz-version-id', 'L7ViqSXBtkdfIYltLrLN1BhDNjOq8xz'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000008153898ea18bfee3-006477a6e4-110e-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 19:58:28 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T19:58:28.655 DEBUG:boto:path=/pklxkd-11/ 2023-05-31T19:58:28.655 DEBUG:boto:auth_path=/pklxkd-11/ 2023-05-31T19:58:28.656 DEBUG:boto:path=/pklxkd-11/?prefix=obj-a1 2023-05-31T19:58:28.656 DEBUG:boto:auth_path=/pklxkd-11/?prefix=obj-a1 2023-05-31T19:58:28.656 DEBUG:boto:Method: GET 2023-05-31T19:58:28.657 DEBUG:boto:Path: /pklxkd-11/?prefix=obj-a1 2023-05-31T19:58:28.657 DEBUG:boto:Data: 2023-05-31T19:58:28.657 DEBUG:boto:Headers: {} 2023-05-31T19:58:28.658 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T19:58:28.658 DEBUG:boto:Port: 8001 2023-05-31T19:58:28.658 DEBUG:boto:Params: {} 2023-05-31T19:58:28.659 DEBUG:boto:Token: None 2023-05-31T19:58:28.659 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 19:58:28 GMT /pklxkd-11/ 2023-05-31T19:58:28.660 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:cd0GuYT/8SKWQPD06IPg+xJIWY4= 2023-05-31T19:58:28.660 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 19:58:28 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:cd0GuYT/8SKWQPD06IPg+xJIWY4=', 'Content-Length': '0'} 2023-05-31T19:58:28.662 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000053f8ac438609b2bd-006477a6e4-1112-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 19:58:28 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T19:58:28.663 DEBUG:boto:b'pklxkd-11obj-a11000falseobj-a12023-05-31T19:57:47.555Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T19:58:28.664 DEBUG:boto:path=/pklxkd-11/ 2023-05-31T19:58:28.664 DEBUG:boto:auth_path=/pklxkd-11/ 2023-05-31T19:58:28.664 DEBUG:boto:path=/pklxkd-11/?prefix=obj-a1 2023-05-31T19:58:28.665 DEBUG:boto:auth_path=/pklxkd-11/?prefix=obj-a1 2023-05-31T19:58:28.665 DEBUG:boto:Method: GET 2023-05-31T19:58:28.665 DEBUG:boto:Path: /pklxkd-11/?prefix=obj-a1 2023-05-31T19:58:28.666 DEBUG:boto:Data: 2023-05-31T19:58:28.666 DEBUG:boto:Headers: {} 2023-05-31T19:58:28.666 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T19:58:28.667 DEBUG:boto:Port: 8000 2023-05-31T19:58:28.667 DEBUG:boto:Params: {} 2023-05-31T19:58:28.668 DEBUG:boto:Token: None 2023-05-31T19:58:28.668 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 19:58:28 GMT /pklxkd-11/ 2023-05-31T19:58:28.668 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:cd0GuYT/8SKWQPD06IPg+xJIWY4= 2023-05-31T19:58:28.669 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 19:58:28 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:cd0GuYT/8SKWQPD06IPg+xJIWY4=', 'Content-Length': '0'} 2023-05-31T19:58:28.671 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000fe126f4a604b21c8-006477a6e4-110e-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 19:58:28 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T19:58:28.671 DEBUG:boto:b'pklxkd-11obj-a11000falseobj-a12023-05-31T19:57:47.555Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T19:58:28.672 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-a2 2023-05-31T19:58:28.672 DEBUG:boto:path=/pklxkd-11/obj-a2 2023-05-31T19:58:28.673 DEBUG:boto:auth_path=/pklxkd-11/obj-a2 2023-05-31T19:58:28.673 DEBUG:boto:path=/pklxkd-11/obj-a2?versionId=d.2JtI5ycFHjEYrzZkvtbFVUQjr5kaL 2023-05-31T19:58:28.673 DEBUG:boto:auth_path=/pklxkd-11/obj-a2?versionId=d.2JtI5ycFHjEYrzZkvtbFVUQjr5kaL 2023-05-31T19:58:28.674 DEBUG:boto:Method: GET 2023-05-31T19:58:28.674 DEBUG:boto:Path: /pklxkd-11/obj-a2?versionId=d.2JtI5ycFHjEYrzZkvtbFVUQjr5kaL 2023-05-31T19:58:28.674 DEBUG:boto:Data: 2023-05-31T19:58:28.675 DEBUG:boto:Headers: {} 2023-05-31T19:58:28.675 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T19:58:28.675 DEBUG:boto:Port: 8001 2023-05-31T19:58:28.676 DEBUG:boto:Params: {} 2023-05-31T19:58:28.676 DEBUG:boto:Token: None 2023-05-31T19:58:28.677 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 19:58:28 GMT /pklxkd-11/obj-a2?versionId=d.2JtI5ycFHjEYrzZkvtbFVUQjr5kaL 2023-05-31T19:58:28.677 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:5OsejQBYkHWNASiLK/EJd5kUuVk= 2023-05-31T19:58:28.678 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 19:58:28 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:5OsejQBYkHWNASiLK/EJd5kUuVk=', 'Content-Length': '0'} 2023-05-31T19:58:28.679 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 19:57:47 GMT'), ('x-amz-version-id', 'd.2JtI5ycFHjEYrzZkvtbFVUQjr5kaL'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000c58cfafabf321708-006477a6e4-1112-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 19:58:28 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T19:58:28.680 DEBUG:boto:path=/pklxkd-11/obj-a2 2023-05-31T19:58:28.680 DEBUG:boto:auth_path=/pklxkd-11/obj-a2 2023-05-31T19:58:28.680 DEBUG:boto:path=/pklxkd-11/obj-a2?versionId=d.2JtI5ycFHjEYrzZkvtbFVUQjr5kaL 2023-05-31T19:58:28.681 DEBUG:boto:auth_path=/pklxkd-11/obj-a2?versionId=d.2JtI5ycFHjEYrzZkvtbFVUQjr5kaL 2023-05-31T19:58:28.681 DEBUG:boto:Method: GET 2023-05-31T19:58:28.681 DEBUG:boto:Path: /pklxkd-11/obj-a2?versionId=d.2JtI5ycFHjEYrzZkvtbFVUQjr5kaL 2023-05-31T19:58:28.682 DEBUG:boto:Data: 2023-05-31T19:58:28.682 DEBUG:boto:Headers: {} 2023-05-31T19:58:28.683 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T19:58:28.683 DEBUG:boto:Port: 8000 2023-05-31T19:58:28.683 DEBUG:boto:Params: {} 2023-05-31T19:58:28.684 DEBUG:boto:Token: None 2023-05-31T19:58:28.684 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 19:58:28 GMT /pklxkd-11/obj-a2?versionId=d.2JtI5ycFHjEYrzZkvtbFVUQjr5kaL 2023-05-31T19:58:28.685 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:5OsejQBYkHWNASiLK/EJd5kUuVk= 2023-05-31T19:58:28.685 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 19:58:28 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:5OsejQBYkHWNASiLK/EJd5kUuVk=', 'Content-Length': '0'} 2023-05-31T19:58:28.687 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 19:57:47 GMT'), ('x-amz-version-id', 'd.2JtI5ycFHjEYrzZkvtbFVUQjr5kaL'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000033f82a388357d66f-006477a6e4-110e-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 19:58:28 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T19:58:28.688 DEBUG:boto:path=/pklxkd-11/obj-a2 2023-05-31T19:58:28.688 DEBUG:boto:auth_path=/pklxkd-11/obj-a2 2023-05-31T19:58:28.688 DEBUG:boto:path=/pklxkd-11/obj-a2?versionId=d.2JtI5ycFHjEYrzZkvtbFVUQjr5kaL 2023-05-31T19:58:28.689 DEBUG:boto:auth_path=/pklxkd-11/obj-a2?versionId=d.2JtI5ycFHjEYrzZkvtbFVUQjr5kaL 2023-05-31T19:58:28.689 DEBUG:boto:Method: HEAD 2023-05-31T19:58:28.689 DEBUG:boto:Path: /pklxkd-11/obj-a2?versionId=d.2JtI5ycFHjEYrzZkvtbFVUQjr5kaL 2023-05-31T19:58:28.690 DEBUG:boto:Data: 2023-05-31T19:58:28.690 DEBUG:boto:Headers: {} 2023-05-31T19:58:28.690 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T19:58:28.691 DEBUG:boto:Port: 8001 2023-05-31T19:58:28.691 DEBUG:boto:Params: {} 2023-05-31T19:58:28.691 DEBUG:boto:Token: None 2023-05-31T19:58:28.692 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 19:58:28 GMT /pklxkd-11/obj-a2?versionId=d.2JtI5ycFHjEYrzZkvtbFVUQjr5kaL 2023-05-31T19:58:28.692 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:FJoyzaZD5J26PD+ViJzqOOnrkzc= 2023-05-31T19:58:28.693 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 19:58:28 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:FJoyzaZD5J26PD+ViJzqOOnrkzc=', 'Content-Length': '0'} 2023-05-31T19:58:28.694 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 19:57:47 GMT'), ('x-amz-version-id', 'd.2JtI5ycFHjEYrzZkvtbFVUQjr5kaL'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000081fcddc67896ffc1-006477a6e4-1112-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 19:58:28 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T19:58:28.695 DEBUG:boto:path=/pklxkd-11/obj-a2 2023-05-31T19:58:28.695 DEBUG:boto:auth_path=/pklxkd-11/obj-a2 2023-05-31T19:58:28.695 DEBUG:boto:path=/pklxkd-11/obj-a2?versionId=d.2JtI5ycFHjEYrzZkvtbFVUQjr5kaL 2023-05-31T19:58:28.696 DEBUG:boto:auth_path=/pklxkd-11/obj-a2?versionId=d.2JtI5ycFHjEYrzZkvtbFVUQjr5kaL 2023-05-31T19:58:28.696 DEBUG:boto:Method: HEAD 2023-05-31T19:58:28.696 DEBUG:boto:Path: /pklxkd-11/obj-a2?versionId=d.2JtI5ycFHjEYrzZkvtbFVUQjr5kaL 2023-05-31T19:58:28.697 DEBUG:boto:Data: 2023-05-31T19:58:28.697 DEBUG:boto:Headers: {} 2023-05-31T19:58:28.697 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T19:58:28.698 DEBUG:boto:Port: 8000 2023-05-31T19:58:28.698 DEBUG:boto:Params: {} 2023-05-31T19:58:28.698 DEBUG:boto:Token: None 2023-05-31T19:58:28.699 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 19:58:28 GMT /pklxkd-11/obj-a2?versionId=d.2JtI5ycFHjEYrzZkvtbFVUQjr5kaL 2023-05-31T19:58:28.699 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:FJoyzaZD5J26PD+ViJzqOOnrkzc= 2023-05-31T19:58:28.699 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 19:58:28 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:FJoyzaZD5J26PD+ViJzqOOnrkzc=', 'Content-Length': '0'} 2023-05-31T19:58:28.701 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 19:57:47 GMT'), ('x-amz-version-id', 'd.2JtI5ycFHjEYrzZkvtbFVUQjr5kaL'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000091d56fb75fdf92ac-006477a6e4-110e-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 19:58:28 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T19:58:28.702 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-a2 2023-05-31T19:58:28.702 DEBUG:boto:path=/pklxkd-11/obj-a2 2023-05-31T19:58:28.702 DEBUG:boto:auth_path=/pklxkd-11/obj-a2 2023-05-31T19:58:28.703 DEBUG:boto:path=/pklxkd-11/obj-a2?versionId=d.2JtI5ycFHjEYrzZkvtbFVUQjr5kaL 2023-05-31T19:58:28.703 DEBUG:boto:auth_path=/pklxkd-11/obj-a2?versionId=d.2JtI5ycFHjEYrzZkvtbFVUQjr5kaL 2023-05-31T19:58:28.704 DEBUG:boto:Method: GET 2023-05-31T19:58:28.704 DEBUG:boto:Path: /pklxkd-11/obj-a2?versionId=d.2JtI5ycFHjEYrzZkvtbFVUQjr5kaL 2023-05-31T19:58:28.704 DEBUG:boto:Data: 2023-05-31T19:58:28.705 DEBUG:boto:Headers: {} 2023-05-31T19:58:28.705 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T19:58:28.705 DEBUG:boto:Port: 8001 2023-05-31T19:58:28.706 DEBUG:boto:Params: {} 2023-05-31T19:58:28.706 DEBUG:boto:Token: None 2023-05-31T19:58:28.707 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 19:58:28 GMT /pklxkd-11/obj-a2?versionId=d.2JtI5ycFHjEYrzZkvtbFVUQjr5kaL 2023-05-31T19:58:28.707 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:5OsejQBYkHWNASiLK/EJd5kUuVk= 2023-05-31T19:58:28.707 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 19:58:28 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:5OsejQBYkHWNASiLK/EJd5kUuVk=', 'Content-Length': '0'} 2023-05-31T19:58:28.709 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 19:57:47 GMT'), ('x-amz-version-id', 'd.2JtI5ycFHjEYrzZkvtbFVUQjr5kaL'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000000ae4506a7b260b1b-006477a6e4-1112-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 19:58:28 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T19:58:28.710 DEBUG:boto:path=/pklxkd-11/obj-a2 2023-05-31T19:58:28.710 DEBUG:boto:auth_path=/pklxkd-11/obj-a2 2023-05-31T19:58:28.710 DEBUG:boto:path=/pklxkd-11/obj-a2?versionId=d.2JtI5ycFHjEYrzZkvtbFVUQjr5kaL 2023-05-31T19:58:28.711 DEBUG:boto:auth_path=/pklxkd-11/obj-a2?versionId=d.2JtI5ycFHjEYrzZkvtbFVUQjr5kaL 2023-05-31T19:58:28.711 DEBUG:boto:Method: GET 2023-05-31T19:58:28.711 DEBUG:boto:Path: /pklxkd-11/obj-a2?versionId=d.2JtI5ycFHjEYrzZkvtbFVUQjr5kaL 2023-05-31T19:58:28.712 DEBUG:boto:Data: 2023-05-31T19:58:28.712 DEBUG:boto:Headers: {} 2023-05-31T19:58:28.713 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T19:58:28.713 DEBUG:boto:Port: 8000 2023-05-31T19:58:28.713 DEBUG:boto:Params: {} 2023-05-31T19:58:28.714 DEBUG:boto:Token: None 2023-05-31T19:58:28.714 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 19:58:28 GMT /pklxkd-11/obj-a2?versionId=d.2JtI5ycFHjEYrzZkvtbFVUQjr5kaL 2023-05-31T19:58:28.715 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:5OsejQBYkHWNASiLK/EJd5kUuVk= 2023-05-31T19:58:28.715 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 19:58:28 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:5OsejQBYkHWNASiLK/EJd5kUuVk=', 'Content-Length': '0'} 2023-05-31T19:58:28.716 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 19:57:47 GMT'), ('x-amz-version-id', 'd.2JtI5ycFHjEYrzZkvtbFVUQjr5kaL'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000005a073fa93bad8670-006477a6e4-110e-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 19:58:28 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T19:58:28.717 DEBUG:boto:path=/pklxkd-11/ 2023-05-31T19:58:28.717 DEBUG:boto:auth_path=/pklxkd-11/ 2023-05-31T19:58:28.718 DEBUG:boto:path=/pklxkd-11/?prefix=obj-a2 2023-05-31T19:58:28.718 DEBUG:boto:auth_path=/pklxkd-11/?prefix=obj-a2 2023-05-31T19:58:28.719 DEBUG:boto:Method: GET 2023-05-31T19:58:28.719 DEBUG:boto:Path: /pklxkd-11/?prefix=obj-a2 2023-05-31T19:58:28.719 DEBUG:boto:Data: 2023-05-31T19:58:28.720 DEBUG:boto:Headers: {} 2023-05-31T19:58:28.720 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T19:58:28.720 DEBUG:boto:Port: 8001 2023-05-31T19:58:28.721 DEBUG:boto:Params: {} 2023-05-31T19:58:28.721 DEBUG:boto:Token: None 2023-05-31T19:58:28.722 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 19:58:28 GMT /pklxkd-11/ 2023-05-31T19:58:28.722 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:cd0GuYT/8SKWQPD06IPg+xJIWY4= 2023-05-31T19:58:28.722 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 19:58:28 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:cd0GuYT/8SKWQPD06IPg+xJIWY4=', 'Content-Length': '0'} 2023-05-31T19:58:28.724 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000005da31557051116ca-006477a6e4-1112-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 19:58:28 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T19:58:28.725 DEBUG:boto:b'pklxkd-11obj-a21000falseobj-a22023-05-31T19:57:47.683Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T19:58:28.726 DEBUG:boto:path=/pklxkd-11/ 2023-05-31T19:58:28.726 DEBUG:boto:auth_path=/pklxkd-11/ 2023-05-31T19:58:28.726 DEBUG:boto:path=/pklxkd-11/?prefix=obj-a2 2023-05-31T19:58:28.727 DEBUG:boto:auth_path=/pklxkd-11/?prefix=obj-a2 2023-05-31T19:58:28.727 DEBUG:boto:Method: GET 2023-05-31T19:58:28.727 DEBUG:boto:Path: /pklxkd-11/?prefix=obj-a2 2023-05-31T19:58:28.728 DEBUG:boto:Data: 2023-05-31T19:58:28.728 DEBUG:boto:Headers: {} 2023-05-31T19:58:28.729 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T19:58:28.729 DEBUG:boto:Port: 8000 2023-05-31T19:58:28.729 DEBUG:boto:Params: {} 2023-05-31T19:58:28.730 DEBUG:boto:Token: None 2023-05-31T19:58:28.730 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 19:58:28 GMT /pklxkd-11/ 2023-05-31T19:58:28.731 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:cd0GuYT/8SKWQPD06IPg+xJIWY4= 2023-05-31T19:58:28.731 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 19:58:28 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:cd0GuYT/8SKWQPD06IPg+xJIWY4=', 'Content-Length': '0'} 2023-05-31T19:58:28.733 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ffb03bf197a2025c-006477a6e4-110e-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 19:58:28 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T19:58:28.776 DEBUG:boto:b'pklxkd-11obj-a21000falseobj-a22023-05-31T19:57:47.683Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T19:58:28.777 DEBUG:boto:path=/pklxkd-11/obj-a2 2023-05-31T19:58:28.777 DEBUG:boto:auth_path=/pklxkd-11/obj-a2 2023-05-31T19:58:28.778 DEBUG:boto:Method: HEAD 2023-05-31T19:58:28.778 DEBUG:boto:Path: /pklxkd-11/obj-a2 2023-05-31T19:58:28.778 DEBUG:boto:Data: 2023-05-31T19:58:28.779 DEBUG:boto:Headers: {} 2023-05-31T19:58:28.779 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T19:58:28.779 DEBUG:boto:Port: 8001 2023-05-31T19:58:28.780 DEBUG:boto:Params: {} 2023-05-31T19:58:28.780 DEBUG:boto:Token: None 2023-05-31T19:58:28.781 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 19:58:28 GMT /pklxkd-11/obj-a2 2023-05-31T19:58:28.781 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:TJKjKJLeunf1kMiE7o10WXXTavo= 2023-05-31T19:58:28.781 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 19:58:28 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:TJKjKJLeunf1kMiE7o10WXXTavo=', 'Content-Length': '0'} 2023-05-31T19:58:28.783 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 19:57:47 GMT'), ('x-amz-version-id', 'd.2JtI5ycFHjEYrzZkvtbFVUQjr5kaL'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000046d3e5f1e7ec9d33-006477a6e4-1112-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 19:58:28 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T19:58:28.784 DEBUG:boto:path=/pklxkd-11/obj-a2 2023-05-31T19:58:28.784 DEBUG:boto:auth_path=/pklxkd-11/obj-a2 2023-05-31T19:58:28.785 DEBUG:boto:Method: HEAD 2023-05-31T19:58:28.785 DEBUG:boto:Path: /pklxkd-11/obj-a2 2023-05-31T19:58:28.785 DEBUG:boto:Data: 2023-05-31T19:58:28.786 DEBUG:boto:Headers: {} 2023-05-31T19:58:28.786 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T19:58:28.786 DEBUG:boto:Port: 8000 2023-05-31T19:58:28.787 DEBUG:boto:Params: {} 2023-05-31T19:58:28.788 DEBUG:boto:Token: None 2023-05-31T19:58:28.788 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 19:58:28 GMT /pklxkd-11/obj-a2 2023-05-31T19:58:28.789 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:TJKjKJLeunf1kMiE7o10WXXTavo= 2023-05-31T19:58:28.789 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 19:58:28 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:TJKjKJLeunf1kMiE7o10WXXTavo=', 'Content-Length': '0'} 2023-05-31T19:58:28.791 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 19:57:47 GMT'), ('x-amz-version-id', 'd.2JtI5ycFHjEYrzZkvtbFVUQjr5kaL'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000005eb4df90eeb13aa7-006477a6e4-110e-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 19:58:28 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T19:58:28.792 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-a2 2023-05-31T19:58:28.792 DEBUG:boto:path=/pklxkd-11/obj-a2 2023-05-31T19:58:28.792 DEBUG:boto:auth_path=/pklxkd-11/obj-a2 2023-05-31T19:58:28.793 DEBUG:boto:path=/pklxkd-11/obj-a2?versionId=d.2JtI5ycFHjEYrzZkvtbFVUQjr5kaL 2023-05-31T19:58:28.793 DEBUG:boto:auth_path=/pklxkd-11/obj-a2?versionId=d.2JtI5ycFHjEYrzZkvtbFVUQjr5kaL 2023-05-31T19:58:28.793 DEBUG:boto:Method: GET 2023-05-31T19:58:28.794 DEBUG:boto:Path: /pklxkd-11/obj-a2?versionId=d.2JtI5ycFHjEYrzZkvtbFVUQjr5kaL 2023-05-31T19:58:28.794 DEBUG:boto:Data: 2023-05-31T19:58:28.794 DEBUG:boto:Headers: {} 2023-05-31T19:58:28.795 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T19:58:28.795 DEBUG:boto:Port: 8001 2023-05-31T19:58:28.795 DEBUG:boto:Params: {} 2023-05-31T19:58:28.796 DEBUG:boto:Token: None 2023-05-31T19:58:28.796 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 19:58:28 GMT /pklxkd-11/obj-a2?versionId=d.2JtI5ycFHjEYrzZkvtbFVUQjr5kaL 2023-05-31T19:58:28.797 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:5OsejQBYkHWNASiLK/EJd5kUuVk= 2023-05-31T19:58:28.797 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 19:58:28 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:5OsejQBYkHWNASiLK/EJd5kUuVk=', 'Content-Length': '0'} 2023-05-31T19:58:28.799 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 19:57:47 GMT'), ('x-amz-version-id', 'd.2JtI5ycFHjEYrzZkvtbFVUQjr5kaL'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000250511af4b8452da-006477a6e4-1112-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 19:58:28 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T19:58:28.800 DEBUG:boto:path=/pklxkd-11/obj-a2 2023-05-31T19:58:28.800 DEBUG:boto:auth_path=/pklxkd-11/obj-a2 2023-05-31T19:58:28.800 DEBUG:boto:path=/pklxkd-11/obj-a2?versionId=d.2JtI5ycFHjEYrzZkvtbFVUQjr5kaL 2023-05-31T19:58:28.801 DEBUG:boto:auth_path=/pklxkd-11/obj-a2?versionId=d.2JtI5ycFHjEYrzZkvtbFVUQjr5kaL 2023-05-31T19:58:28.801 DEBUG:boto:Method: GET 2023-05-31T19:58:28.801 DEBUG:boto:Path: /pklxkd-11/obj-a2?versionId=d.2JtI5ycFHjEYrzZkvtbFVUQjr5kaL 2023-05-31T19:58:28.802 DEBUG:boto:Data: 2023-05-31T19:58:28.802 DEBUG:boto:Headers: {} 2023-05-31T19:58:28.803 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T19:58:28.803 DEBUG:boto:Port: 8000 2023-05-31T19:58:28.803 DEBUG:boto:Params: {} 2023-05-31T19:58:28.804 DEBUG:boto:Token: None 2023-05-31T19:58:28.804 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 19:58:28 GMT /pklxkd-11/obj-a2?versionId=d.2JtI5ycFHjEYrzZkvtbFVUQjr5kaL 2023-05-31T19:58:28.804 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:5OsejQBYkHWNASiLK/EJd5kUuVk= 2023-05-31T19:58:28.805 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 19:58:28 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:5OsejQBYkHWNASiLK/EJd5kUuVk=', 'Content-Length': '0'} 2023-05-31T19:58:28.807 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 19:57:47 GMT'), ('x-amz-version-id', 'd.2JtI5ycFHjEYrzZkvtbFVUQjr5kaL'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000003c4f175967d04d87-006477a6e4-110e-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 19:58:28 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T19:58:28.807 DEBUG:boto:path=/pklxkd-11/ 2023-05-31T19:58:28.808 DEBUG:boto:auth_path=/pklxkd-11/ 2023-05-31T19:58:28.808 DEBUG:boto:path=/pklxkd-11/?prefix=obj-a2 2023-05-31T19:58:28.808 DEBUG:boto:auth_path=/pklxkd-11/?prefix=obj-a2 2023-05-31T19:58:28.809 DEBUG:boto:Method: GET 2023-05-31T19:58:28.809 DEBUG:boto:Path: /pklxkd-11/?prefix=obj-a2 2023-05-31T19:58:28.809 DEBUG:boto:Data: 2023-05-31T19:58:28.810 DEBUG:boto:Headers: {} 2023-05-31T19:58:28.810 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T19:58:28.810 DEBUG:boto:Port: 8001 2023-05-31T19:58:28.811 DEBUG:boto:Params: {} 2023-05-31T19:58:28.811 DEBUG:boto:Token: None 2023-05-31T19:58:28.812 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 19:58:28 GMT /pklxkd-11/ 2023-05-31T19:58:28.812 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:cd0GuYT/8SKWQPD06IPg+xJIWY4= 2023-05-31T19:58:28.813 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 19:58:28 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:cd0GuYT/8SKWQPD06IPg+xJIWY4=', 'Content-Length': '0'} 2023-05-31T19:58:28.815 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000368a02e863628bf2-006477a6e4-1112-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 19:58:28 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T19:58:28.815 DEBUG:boto:b'pklxkd-11obj-a21000falseobj-a22023-05-31T19:57:47.683Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T19:58:28.816 DEBUG:boto:path=/pklxkd-11/ 2023-05-31T19:58:28.816 DEBUG:boto:auth_path=/pklxkd-11/ 2023-05-31T19:58:28.817 DEBUG:boto:path=/pklxkd-11/?prefix=obj-a2 2023-05-31T19:58:28.817 DEBUG:boto:auth_path=/pklxkd-11/?prefix=obj-a2 2023-05-31T19:58:28.817 DEBUG:boto:Method: GET 2023-05-31T19:58:28.818 DEBUG:boto:Path: /pklxkd-11/?prefix=obj-a2 2023-05-31T19:58:28.818 DEBUG:boto:Data: 2023-05-31T19:58:28.818 DEBUG:boto:Headers: {} 2023-05-31T19:58:28.819 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T19:58:28.819 DEBUG:boto:Port: 8000 2023-05-31T19:58:28.820 DEBUG:boto:Params: {} 2023-05-31T19:58:28.820 DEBUG:boto:Token: None 2023-05-31T19:58:28.820 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 19:58:28 GMT /pklxkd-11/ 2023-05-31T19:58:28.821 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:cd0GuYT/8SKWQPD06IPg+xJIWY4= 2023-05-31T19:58:28.821 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 19:58:28 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:cd0GuYT/8SKWQPD06IPg+xJIWY4=', 'Content-Length': '0'} 2023-05-31T19:58:28.823 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000009e57e580bfdbc840-006477a6e4-110e-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 19:58:28 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T19:58:28.824 DEBUG:boto:b'pklxkd-11obj-a21000falseobj-a22023-05-31T19:57:47.683Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T19:58:28.824 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=pklxkd-11 zones={a2, a1} 2023-05-31T19:58:28.825 DEBUG:rgw_multi.tests:bucket checkpoint: source=a1 target=a2 bucket=pklxkd-12 2023-05-31T19:58:28.825 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'pklxkd-12', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-05-31T19:58:28.825 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'pklxkd-12', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-05-31T19:58:28.826 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket pklxkd-12 --source-zone a1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2023-05-31T19:58:28.882 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2023-05-31T19:58:28.883 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2023-05-31T19:58:28.918 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T19:58:28.913+0000 7ff95f44b780 1 bucket sync caught up with source: 2023-05-31T19:58:28.918 INFO:teuthology.orchestra.run.smithi099.stderr: local status: [, , , , , 00000000002.3.11, , , 00000000002.7.11, 00000000010.25.6, ] 2023-05-31T19:58:28.918 INFO:teuthology.orchestra.run.smithi099.stderr: remote markers: [, , , , , 00000000002.3.11, , , 00000000002.7.11, 00000000010.25.6, ] 2023-05-31T19:58:28.918 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T19:58:28.913+0000 7ff95f44b780 0 bucket checkpoint complete 2023-05-31T19:58:28.929 DEBUG:rgw_multi.tests:bucket checkpoint: source=a2 target=a1 bucket=pklxkd-12 2023-05-31T19:58:28.930 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'pklxkd-12', '--source-zone', 'a2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-05-31T19:58:28.930 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'pklxkd-12', '--source-zone', 'a2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-05-31T19:58:28.931 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket pklxkd-12 --source-zone a2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2023-05-31T19:58:28.986 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2023-05-31T19:58:28.987 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2023-05-31T19:58:29.022 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T19:58:29.018+0000 7fdd1ae1e780 1 bucket sync caught up with source: 2023-05-31T19:58:29.023 INFO:teuthology.orchestra.run.smithi099.stderr: local status: [, , , , , 00000000003.10.10, , , 00000000010.37.6, 00000000003.35.10, ] 2023-05-31T19:58:29.023 INFO:teuthology.orchestra.run.smithi099.stderr: remote markers: [, , , , , 00000000003.10.10, , , 00000000010.37.6, 00000000003.35.10, ] 2023-05-31T19:58:29.023 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T19:58:29.018+0000 7fdd1ae1e780 0 bucket checkpoint complete 2023-05-31T19:58:29.035 INFO:tasks.rgw_multi.zone_rados:comparing bucket=pklxkd-12 zones={a2, a1} 2023-05-31T19:58:29.035 DEBUG:boto:path=/pklxkd-12/ 2023-05-31T19:58:29.036 DEBUG:boto:auth_path=/pklxkd-12/ 2023-05-31T19:58:29.036 DEBUG:boto:Method: HEAD 2023-05-31T19:58:29.036 DEBUG:boto:Path: /pklxkd-12/ 2023-05-31T19:58:29.037 DEBUG:boto:Data: 2023-05-31T19:58:29.037 DEBUG:boto:Headers: {} 2023-05-31T19:58:29.037 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T19:58:29.038 DEBUG:boto:Port: 8001 2023-05-31T19:58:29.038 DEBUG:boto:Params: {} 2023-05-31T19:58:29.039 DEBUG:boto:Token: None 2023-05-31T19:58:29.039 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 19:58:29 GMT /pklxkd-12/ 2023-05-31T19:58:29.039 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:caPUtJ6nzFp2mIPlm10+0Of6oQE= 2023-05-31T19:58:29.040 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 19:58:29 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:caPUtJ6nzFp2mIPlm10+0Of6oQE=', 'Content-Length': '0'} 2023-05-31T19:58:29.042 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '3'), ('X-RGW-Bytes-Used', '16'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000ba208718f28ba455-006477a6e5-1112-a2'), ('Content-Length', '0'), ('Date', 'Wed, 31 May 2023 19:58:29 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T19:58:29.043 DEBUG:boto:path=/pklxkd-12/ 2023-05-31T19:58:29.044 DEBUG:boto:auth_path=/pklxkd-12/ 2023-05-31T19:58:29.045 DEBUG:boto:Method: HEAD 2023-05-31T19:58:29.045 DEBUG:boto:Path: /pklxkd-12/ 2023-05-31T19:58:29.045 DEBUG:boto:Data: 2023-05-31T19:58:29.046 DEBUG:boto:Headers: {} 2023-05-31T19:58:29.046 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T19:58:29.046 DEBUG:boto:Port: 8000 2023-05-31T19:58:29.047 DEBUG:boto:Params: {} 2023-05-31T19:58:29.047 DEBUG:boto:Token: None 2023-05-31T19:58:29.048 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 19:58:29 GMT /pklxkd-12/ 2023-05-31T19:58:29.048 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:caPUtJ6nzFp2mIPlm10+0Of6oQE= 2023-05-31T19:58:29.049 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 19:58:29 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:caPUtJ6nzFp2mIPlm10+0Of6oQE=', 'Content-Length': '0'} 2023-05-31T19:58:29.051 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '3'), ('X-RGW-Bytes-Used', '16'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000f75057aa9a196699-006477a6e5-110e-a1'), ('Content-Length', '0'), ('Date', 'Wed, 31 May 2023 19:58:29 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T19:58:29.051 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2023-05-31T19:58:29.051 DEBUG:boto:path=/pklxkd-12/ 2023-05-31T19:58:29.052 DEBUG:boto:auth_path=/pklxkd-12/ 2023-05-31T19:58:29.052 DEBUG:boto:path=/pklxkd-12/?versions&max-keys=999 2023-05-31T19:58:29.053 DEBUG:boto:auth_path=/pklxkd-12/?versions&max-keys=999 2023-05-31T19:58:29.053 DEBUG:boto:Method: GET 2023-05-31T19:58:29.053 DEBUG:boto:Path: /pklxkd-12/?versions&max-keys=999 2023-05-31T19:58:29.054 DEBUG:boto:Data: 2023-05-31T19:58:29.054 DEBUG:boto:Headers: {} 2023-05-31T19:58:29.055 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T19:58:29.055 DEBUG:boto:Port: 8001 2023-05-31T19:58:29.055 DEBUG:boto:Params: {} 2023-05-31T19:58:29.056 DEBUG:boto:Token: None 2023-05-31T19:58:29.056 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 19:58:29 GMT /pklxkd-12/?versions 2023-05-31T19:58:29.057 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:CVVAz/PYvezLubxbq2d8CWPZWbA= 2023-05-31T19:58:29.057 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 19:58:29 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:CVVAz/PYvezLubxbq2d8CWPZWbA=', 'Content-Length': '0'} 2023-05-31T19:58:29.060 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c722cef65ffb11db-006477a6e5-1112-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 19:58:29 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T19:58:29.060 DEBUG:boto:b'pklxkd-12999falsedummyHuvfE2dxsypTRiytlVrY6cgwTbY4DCStrue2023-05-31T19:57:27.404Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-a1G1y4LrpL0rJndINHI52T89ADrfTweD3true2023-05-31T19:57:47.837Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-a2qiyfRuud8o74hAGpczwf4-FuN5p5887true2023-05-31T19:57:47.962Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T19:58:29.061 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2023-05-31T19:58:29.061 DEBUG:tasks.rgw_multi.zone_rados:o=obj-a1 2023-05-31T19:58:29.062 DEBUG:tasks.rgw_multi.zone_rados:o=obj-a2 2023-05-31T19:58:29.062 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2023-05-31T19:58:29.062 DEBUG:boto:path=/pklxkd-12/ 2023-05-31T19:58:29.063 DEBUG:boto:auth_path=/pklxkd-12/ 2023-05-31T19:58:29.063 DEBUG:boto:path=/pklxkd-12/?versions&max-keys=999 2023-05-31T19:58:29.064 DEBUG:boto:auth_path=/pklxkd-12/?versions&max-keys=999 2023-05-31T19:58:29.064 DEBUG:boto:Method: GET 2023-05-31T19:58:29.065 DEBUG:boto:Path: /pklxkd-12/?versions&max-keys=999 2023-05-31T19:58:29.065 DEBUG:boto:Data: 2023-05-31T19:58:29.066 DEBUG:boto:Headers: {} 2023-05-31T19:58:29.066 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T19:58:29.066 DEBUG:boto:Port: 8000 2023-05-31T19:58:29.067 DEBUG:boto:Params: {} 2023-05-31T19:58:29.067 DEBUG:boto:Token: None 2023-05-31T19:58:29.067 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 19:58:29 GMT /pklxkd-12/?versions 2023-05-31T19:58:29.068 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:CVVAz/PYvezLubxbq2d8CWPZWbA= 2023-05-31T19:58:29.068 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 19:58:29 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:CVVAz/PYvezLubxbq2d8CWPZWbA=', 'Content-Length': '0'} 2023-05-31T19:58:29.070 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000005c7ae93720e03600-006477a6e5-110e-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 19:58:29 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T19:58:29.071 DEBUG:boto:b'pklxkd-12999falsedummyHuvfE2dxsypTRiytlVrY6cgwTbY4DCStrue2023-05-31T19:57:27.404Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-a1G1y4LrpL0rJndINHI52T89ADrfTweD3true2023-05-31T19:57:47.837Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-a2qiyfRuud8o74hAGpczwf4-FuN5p5887true2023-05-31T19:57:47.962Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T19:58:29.071 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2023-05-31T19:58:29.072 DEBUG:tasks.rgw_multi.zone_rados:o=obj-a1 2023-05-31T19:58:29.072 DEBUG:tasks.rgw_multi.zone_rados:o=obj-a2 2023-05-31T19:58:29.073 DEBUG:boto:path=/pklxkd-12/ 2023-05-31T19:58:29.073 DEBUG:boto:auth_path=/pklxkd-12/ 2023-05-31T19:58:29.073 DEBUG:boto:path=/pklxkd-12/?versions&max-keys=999 2023-05-31T19:58:29.074 DEBUG:boto:auth_path=/pklxkd-12/?versions&max-keys=999 2023-05-31T19:58:29.074 DEBUG:boto:Method: GET 2023-05-31T19:58:29.074 DEBUG:boto:Path: /pklxkd-12/?versions&max-keys=999 2023-05-31T19:58:29.075 DEBUG:boto:Data: 2023-05-31T19:58:29.075 DEBUG:boto:Headers: {} 2023-05-31T19:58:29.075 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T19:58:29.076 DEBUG:boto:Port: 8001 2023-05-31T19:58:29.076 DEBUG:boto:Params: {} 2023-05-31T19:58:29.077 DEBUG:boto:Token: None 2023-05-31T19:58:29.077 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 19:58:29 GMT /pklxkd-12/?versions 2023-05-31T19:58:29.077 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:CVVAz/PYvezLubxbq2d8CWPZWbA= 2023-05-31T19:58:29.078 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 19:58:29 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:CVVAz/PYvezLubxbq2d8CWPZWbA=', 'Content-Length': '0'} 2023-05-31T19:58:29.080 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000006dd3a7508dce47aa-006477a6e5-1112-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 19:58:29 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T19:58:29.081 DEBUG:boto:b'pklxkd-12999falsedummyHuvfE2dxsypTRiytlVrY6cgwTbY4DCStrue2023-05-31T19:57:27.404Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-a1G1y4LrpL0rJndINHI52T89ADrfTweD3true2023-05-31T19:57:47.837Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-a2qiyfRuud8o74hAGpczwf4-FuN5p5887true2023-05-31T19:57:47.962Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T19:58:29.082 DEBUG:boto:path=/pklxkd-12/ 2023-05-31T19:58:29.082 DEBUG:boto:auth_path=/pklxkd-12/ 2023-05-31T19:58:29.082 DEBUG:boto:path=/pklxkd-12/?versions&max-keys=999 2023-05-31T19:58:29.083 DEBUG:boto:auth_path=/pklxkd-12/?versions&max-keys=999 2023-05-31T19:58:29.083 DEBUG:boto:Method: GET 2023-05-31T19:58:29.083 DEBUG:boto:Path: /pklxkd-12/?versions&max-keys=999 2023-05-31T19:58:29.084 DEBUG:boto:Data: 2023-05-31T19:58:29.084 DEBUG:boto:Headers: {} 2023-05-31T19:58:29.084 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T19:58:29.085 DEBUG:boto:Port: 8000 2023-05-31T19:58:29.085 DEBUG:boto:Params: {} 2023-05-31T19:58:29.086 DEBUG:boto:Token: None 2023-05-31T19:58:29.086 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 19:58:29 GMT /pklxkd-12/?versions 2023-05-31T19:58:29.086 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:CVVAz/PYvezLubxbq2d8CWPZWbA= 2023-05-31T19:58:29.087 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 19:58:29 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:CVVAz/PYvezLubxbq2d8CWPZWbA=', 'Content-Length': '0'} 2023-05-31T19:58:29.089 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000de90ea677dda2d33-006477a6e5-110e-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 19:58:29 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T19:58:29.090 DEBUG:boto:b'pklxkd-12999falsedummyHuvfE2dxsypTRiytlVrY6cgwTbY4DCStrue2023-05-31T19:57:27.404Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-a1G1y4LrpL0rJndINHI52T89ADrfTweD3true2023-05-31T19:57:47.837Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-a2qiyfRuud8o74hAGpczwf4-FuN5p5887true2023-05-31T19:57:47.962Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T19:58:29.091 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2023-05-31T19:58:29.091 DEBUG:boto:path=/pklxkd-12/dummy 2023-05-31T19:58:29.092 DEBUG:boto:auth_path=/pklxkd-12/dummy 2023-05-31T19:58:29.092 DEBUG:boto:path=/pklxkd-12/dummy?versionId=HuvfE2dxsypTRiytlVrY6cgwTbY4DCS 2023-05-31T19:58:29.093 DEBUG:boto:auth_path=/pklxkd-12/dummy?versionId=HuvfE2dxsypTRiytlVrY6cgwTbY4DCS 2023-05-31T19:58:29.093 DEBUG:boto:Method: GET 2023-05-31T19:58:29.094 DEBUG:boto:Path: /pklxkd-12/dummy?versionId=HuvfE2dxsypTRiytlVrY6cgwTbY4DCS 2023-05-31T19:58:29.094 DEBUG:boto:Data: 2023-05-31T19:58:29.094 DEBUG:boto:Headers: {} 2023-05-31T19:58:29.095 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T19:58:29.095 DEBUG:boto:Port: 8001 2023-05-31T19:58:29.095 DEBUG:boto:Params: {} 2023-05-31T19:58:29.096 DEBUG:boto:Token: None 2023-05-31T19:58:29.096 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 19:58:29 GMT /pklxkd-12/dummy?versionId=HuvfE2dxsypTRiytlVrY6cgwTbY4DCS 2023-05-31T19:58:29.097 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:jhDUtzrq8IvkXnjIc55XaxC8I1M= 2023-05-31T19:58:29.097 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 19:58:29 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:jhDUtzrq8IvkXnjIc55XaxC8I1M=', 'Content-Length': '0'} 2023-05-31T19:58:29.099 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 19:57:27 GMT'), ('x-amz-version-id', 'HuvfE2dxsypTRiytlVrY6cgwTbY4DCS'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000006869895a2fcc755b-006477a6e5-1112-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 19:58:29 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T19:58:29.099 DEBUG:boto:path=/pklxkd-12/dummy 2023-05-31T19:58:29.100 DEBUG:boto:auth_path=/pklxkd-12/dummy 2023-05-31T19:58:29.100 DEBUG:boto:path=/pklxkd-12/dummy?versionId=HuvfE2dxsypTRiytlVrY6cgwTbY4DCS 2023-05-31T19:58:29.101 DEBUG:boto:auth_path=/pklxkd-12/dummy?versionId=HuvfE2dxsypTRiytlVrY6cgwTbY4DCS 2023-05-31T19:58:29.101 DEBUG:boto:Method: GET 2023-05-31T19:58:29.101 DEBUG:boto:Path: /pklxkd-12/dummy?versionId=HuvfE2dxsypTRiytlVrY6cgwTbY4DCS 2023-05-31T19:58:29.102 DEBUG:boto:Data: 2023-05-31T19:58:29.102 DEBUG:boto:Headers: {} 2023-05-31T19:58:29.102 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T19:58:29.103 DEBUG:boto:Port: 8000 2023-05-31T19:58:29.103 DEBUG:boto:Params: {} 2023-05-31T19:58:29.104 DEBUG:boto:Token: None 2023-05-31T19:58:29.104 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 19:58:29 GMT /pklxkd-12/dummy?versionId=HuvfE2dxsypTRiytlVrY6cgwTbY4DCS 2023-05-31T19:58:29.105 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:jhDUtzrq8IvkXnjIc55XaxC8I1M= 2023-05-31T19:58:29.105 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 19:58:29 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:jhDUtzrq8IvkXnjIc55XaxC8I1M=', 'Content-Length': '0'} 2023-05-31T19:58:29.107 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 19:57:27 GMT'), ('x-amz-version-id', 'HuvfE2dxsypTRiytlVrY6cgwTbY4DCS'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000f8417c09456a8807-006477a6e5-110e-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 19:58:29 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T19:58:29.107 DEBUG:boto:path=/pklxkd-12/dummy 2023-05-31T19:58:29.108 DEBUG:boto:auth_path=/pklxkd-12/dummy 2023-05-31T19:58:29.108 DEBUG:boto:path=/pklxkd-12/dummy?versionId=HuvfE2dxsypTRiytlVrY6cgwTbY4DCS 2023-05-31T19:58:29.109 DEBUG:boto:auth_path=/pklxkd-12/dummy?versionId=HuvfE2dxsypTRiytlVrY6cgwTbY4DCS 2023-05-31T19:58:29.109 DEBUG:boto:Method: HEAD 2023-05-31T19:58:29.109 DEBUG:boto:Path: /pklxkd-12/dummy?versionId=HuvfE2dxsypTRiytlVrY6cgwTbY4DCS 2023-05-31T19:58:29.110 DEBUG:boto:Data: 2023-05-31T19:58:29.110 DEBUG:boto:Headers: {} 2023-05-31T19:58:29.110 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T19:58:29.111 DEBUG:boto:Port: 8001 2023-05-31T19:58:29.111 DEBUG:boto:Params: {} 2023-05-31T19:58:29.112 DEBUG:boto:Token: None 2023-05-31T19:58:29.112 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 19:58:29 GMT /pklxkd-12/dummy?versionId=HuvfE2dxsypTRiytlVrY6cgwTbY4DCS 2023-05-31T19:58:29.112 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:0w/WC5Vgkw9p9Md3wLZAGfSn/Yw= 2023-05-31T19:58:29.113 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 19:58:29 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:0w/WC5Vgkw9p9Md3wLZAGfSn/Yw=', 'Content-Length': '0'} 2023-05-31T19:58:29.114 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 19:57:27 GMT'), ('x-amz-version-id', 'HuvfE2dxsypTRiytlVrY6cgwTbY4DCS'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx0000079126c046c9b4b3b-006477a6e5-1112-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 19:58:29 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T19:58:29.115 DEBUG:boto:path=/pklxkd-12/dummy 2023-05-31T19:58:29.115 DEBUG:boto:auth_path=/pklxkd-12/dummy 2023-05-31T19:58:29.116 DEBUG:boto:path=/pklxkd-12/dummy?versionId=HuvfE2dxsypTRiytlVrY6cgwTbY4DCS 2023-05-31T19:58:29.116 DEBUG:boto:auth_path=/pklxkd-12/dummy?versionId=HuvfE2dxsypTRiytlVrY6cgwTbY4DCS 2023-05-31T19:58:29.117 DEBUG:boto:Method: HEAD 2023-05-31T19:58:29.117 DEBUG:boto:Path: /pklxkd-12/dummy?versionId=HuvfE2dxsypTRiytlVrY6cgwTbY4DCS 2023-05-31T19:58:29.117 DEBUG:boto:Data: 2023-05-31T19:58:29.118 DEBUG:boto:Headers: {} 2023-05-31T19:58:29.118 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T19:58:29.118 DEBUG:boto:Port: 8000 2023-05-31T19:58:29.119 DEBUG:boto:Params: {} 2023-05-31T19:58:29.119 DEBUG:boto:Token: None 2023-05-31T19:58:29.120 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 19:58:29 GMT /pklxkd-12/dummy?versionId=HuvfE2dxsypTRiytlVrY6cgwTbY4DCS 2023-05-31T19:58:29.120 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:0w/WC5Vgkw9p9Md3wLZAGfSn/Yw= 2023-05-31T19:58:29.120 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 19:58:29 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:0w/WC5Vgkw9p9Md3wLZAGfSn/Yw=', 'Content-Length': '0'} 2023-05-31T19:58:29.122 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 19:57:27 GMT'), ('x-amz-version-id', 'HuvfE2dxsypTRiytlVrY6cgwTbY4DCS'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000f893639ec698fa12-006477a6e5-110e-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 19:58:29 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T19:58:29.123 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2023-05-31T19:58:29.123 DEBUG:boto:path=/pklxkd-12/dummy 2023-05-31T19:58:29.123 DEBUG:boto:auth_path=/pklxkd-12/dummy 2023-05-31T19:58:29.124 DEBUG:boto:path=/pklxkd-12/dummy?versionId=HuvfE2dxsypTRiytlVrY6cgwTbY4DCS 2023-05-31T19:58:29.124 DEBUG:boto:auth_path=/pklxkd-12/dummy?versionId=HuvfE2dxsypTRiytlVrY6cgwTbY4DCS 2023-05-31T19:58:29.124 DEBUG:boto:Method: GET 2023-05-31T19:58:29.125 DEBUG:boto:Path: /pklxkd-12/dummy?versionId=HuvfE2dxsypTRiytlVrY6cgwTbY4DCS 2023-05-31T19:58:29.125 DEBUG:boto:Data: 2023-05-31T19:58:29.126 DEBUG:boto:Headers: {} 2023-05-31T19:58:29.126 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T19:58:29.126 DEBUG:boto:Port: 8001 2023-05-31T19:58:29.127 DEBUG:boto:Params: {} 2023-05-31T19:58:29.127 DEBUG:boto:Token: None 2023-05-31T19:58:29.128 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 19:58:29 GMT /pklxkd-12/dummy?versionId=HuvfE2dxsypTRiytlVrY6cgwTbY4DCS 2023-05-31T19:58:29.129 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:jhDUtzrq8IvkXnjIc55XaxC8I1M= 2023-05-31T19:58:29.130 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 19:58:29 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:jhDUtzrq8IvkXnjIc55XaxC8I1M=', 'Content-Length': '0'} 2023-05-31T19:58:29.133 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 19:57:27 GMT'), ('x-amz-version-id', 'HuvfE2dxsypTRiytlVrY6cgwTbY4DCS'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000f243a671a5bdfbf1-006477a6e5-1112-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 19:58:29 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T19:58:29.134 DEBUG:boto:path=/pklxkd-12/dummy 2023-05-31T19:58:29.134 DEBUG:boto:auth_path=/pklxkd-12/dummy 2023-05-31T19:58:29.135 DEBUG:boto:path=/pklxkd-12/dummy?versionId=HuvfE2dxsypTRiytlVrY6cgwTbY4DCS 2023-05-31T19:58:29.136 DEBUG:boto:auth_path=/pklxkd-12/dummy?versionId=HuvfE2dxsypTRiytlVrY6cgwTbY4DCS 2023-05-31T19:58:29.136 DEBUG:boto:Method: GET 2023-05-31T19:58:29.137 DEBUG:boto:Path: /pklxkd-12/dummy?versionId=HuvfE2dxsypTRiytlVrY6cgwTbY4DCS 2023-05-31T19:58:29.138 DEBUG:boto:Data: 2023-05-31T19:58:29.138 DEBUG:boto:Headers: {} 2023-05-31T19:58:29.139 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T19:58:29.139 DEBUG:boto:Port: 8000 2023-05-31T19:58:29.139 DEBUG:boto:Params: {} 2023-05-31T19:58:29.140 DEBUG:boto:Token: None 2023-05-31T19:58:29.140 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 19:58:29 GMT /pklxkd-12/dummy?versionId=HuvfE2dxsypTRiytlVrY6cgwTbY4DCS 2023-05-31T19:58:29.141 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:jhDUtzrq8IvkXnjIc55XaxC8I1M= 2023-05-31T19:58:29.142 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 19:58:29 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:jhDUtzrq8IvkXnjIc55XaxC8I1M=', 'Content-Length': '0'} 2023-05-31T19:58:29.144 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 19:57:27 GMT'), ('x-amz-version-id', 'HuvfE2dxsypTRiytlVrY6cgwTbY4DCS'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000141548f0b27abeea-006477a6e5-110e-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 19:58:29 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T19:58:29.144 DEBUG:boto:path=/pklxkd-12/ 2023-05-31T19:58:29.145 DEBUG:boto:auth_path=/pklxkd-12/ 2023-05-31T19:58:29.145 DEBUG:boto:path=/pklxkd-12/?prefix=dummy 2023-05-31T19:58:29.145 DEBUG:boto:auth_path=/pklxkd-12/?prefix=dummy 2023-05-31T19:58:29.146 DEBUG:boto:Method: GET 2023-05-31T19:58:29.146 DEBUG:boto:Path: /pklxkd-12/?prefix=dummy 2023-05-31T19:58:29.147 DEBUG:boto:Data: 2023-05-31T19:58:29.147 DEBUG:boto:Headers: {} 2023-05-31T19:58:29.148 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T19:58:29.148 DEBUG:boto:Port: 8001 2023-05-31T19:58:29.148 DEBUG:boto:Params: {} 2023-05-31T19:58:29.149 DEBUG:boto:Token: None 2023-05-31T19:58:29.149 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 19:58:29 GMT /pklxkd-12/ 2023-05-31T19:58:29.150 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:V8XeFrflDLZUWfbnuWB7iCX+rEs= 2023-05-31T19:58:29.150 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 19:58:29 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:V8XeFrflDLZUWfbnuWB7iCX+rEs=', 'Content-Length': '0'} 2023-05-31T19:58:29.152 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000009407c870c5d4270c-006477a6e5-1112-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 19:58:29 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T19:58:29.196 DEBUG:boto:b'pklxkd-12dummy1000falsedummy2023-05-31T19:57:27.404Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T19:58:29.197 DEBUG:boto:path=/pklxkd-12/ 2023-05-31T19:58:29.198 DEBUG:boto:auth_path=/pklxkd-12/ 2023-05-31T19:58:29.198 DEBUG:boto:path=/pklxkd-12/?prefix=dummy 2023-05-31T19:58:29.198 DEBUG:boto:auth_path=/pklxkd-12/?prefix=dummy 2023-05-31T19:58:29.199 DEBUG:boto:Method: GET 2023-05-31T19:58:29.199 DEBUG:boto:Path: /pklxkd-12/?prefix=dummy 2023-05-31T19:58:29.200 DEBUG:boto:Data: 2023-05-31T19:58:29.200 DEBUG:boto:Headers: {} 2023-05-31T19:58:29.200 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T19:58:29.201 DEBUG:boto:Port: 8000 2023-05-31T19:58:29.202 DEBUG:boto:Params: {} 2023-05-31T19:58:29.203 DEBUG:boto:Token: None 2023-05-31T19:58:29.204 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 19:58:29 GMT /pklxkd-12/ 2023-05-31T19:58:29.205 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:V8XeFrflDLZUWfbnuWB7iCX+rEs= 2023-05-31T19:58:29.206 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 19:58:29 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:V8XeFrflDLZUWfbnuWB7iCX+rEs=', 'Content-Length': '0'} 2023-05-31T19:58:29.209 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000fd78f6902b270d67-006477a6e5-110e-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 19:58:29 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T19:58:29.210 DEBUG:boto:b'pklxkd-12dummy1000falsedummy2023-05-31T19:57:27.404Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T19:58:29.211 DEBUG:boto:path=/pklxkd-12/dummy 2023-05-31T19:58:29.212 DEBUG:boto:auth_path=/pklxkd-12/dummy 2023-05-31T19:58:29.212 DEBUG:boto:Method: HEAD 2023-05-31T19:58:29.213 DEBUG:boto:Path: /pklxkd-12/dummy 2023-05-31T19:58:29.213 DEBUG:boto:Data: 2023-05-31T19:58:29.213 DEBUG:boto:Headers: {} 2023-05-31T19:58:29.214 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T19:58:29.214 DEBUG:boto:Port: 8001 2023-05-31T19:58:29.214 DEBUG:boto:Params: {} 2023-05-31T19:58:29.215 DEBUG:boto:Token: None 2023-05-31T19:58:29.215 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 19:58:29 GMT /pklxkd-12/dummy 2023-05-31T19:58:29.216 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:xGBl6UMP9G08meXulI9/sNR1dDs= 2023-05-31T19:58:29.216 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 19:58:29 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:xGBl6UMP9G08meXulI9/sNR1dDs=', 'Content-Length': '0'} 2023-05-31T19:58:29.219 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 19:57:27 GMT'), ('x-amz-version-id', 'HuvfE2dxsypTRiytlVrY6cgwTbY4DCS'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000005f853f07cbc6b8e3-006477a6e5-1112-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 19:58:29 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T19:58:29.220 DEBUG:boto:path=/pklxkd-12/dummy 2023-05-31T19:58:29.221 DEBUG:boto:auth_path=/pklxkd-12/dummy 2023-05-31T19:58:29.221 DEBUG:boto:Method: HEAD 2023-05-31T19:58:29.222 DEBUG:boto:Path: /pklxkd-12/dummy 2023-05-31T19:58:29.223 DEBUG:boto:Data: 2023-05-31T19:58:29.223 DEBUG:boto:Headers: {} 2023-05-31T19:58:29.223 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T19:58:29.224 DEBUG:boto:Port: 8000 2023-05-31T19:58:29.224 DEBUG:boto:Params: {} 2023-05-31T19:58:29.224 DEBUG:boto:Token: None 2023-05-31T19:58:29.225 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 19:58:29 GMT /pklxkd-12/dummy 2023-05-31T19:58:29.225 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:xGBl6UMP9G08meXulI9/sNR1dDs= 2023-05-31T19:58:29.226 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 19:58:29 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:xGBl6UMP9G08meXulI9/sNR1dDs=', 'Content-Length': '0'} 2023-05-31T19:58:29.228 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 19:57:27 GMT'), ('x-amz-version-id', 'HuvfE2dxsypTRiytlVrY6cgwTbY4DCS'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx0000002bc19ef8ba5522f-006477a6e5-110e-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 19:58:29 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T19:58:29.228 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2023-05-31T19:58:29.229 DEBUG:boto:path=/pklxkd-12/dummy 2023-05-31T19:58:29.229 DEBUG:boto:auth_path=/pklxkd-12/dummy 2023-05-31T19:58:29.230 DEBUG:boto:path=/pklxkd-12/dummy?versionId=HuvfE2dxsypTRiytlVrY6cgwTbY4DCS 2023-05-31T19:58:29.230 DEBUG:boto:auth_path=/pklxkd-12/dummy?versionId=HuvfE2dxsypTRiytlVrY6cgwTbY4DCS 2023-05-31T19:58:29.230 DEBUG:boto:Method: GET 2023-05-31T19:58:29.231 DEBUG:boto:Path: /pklxkd-12/dummy?versionId=HuvfE2dxsypTRiytlVrY6cgwTbY4DCS 2023-05-31T19:58:29.231 DEBUG:boto:Data: 2023-05-31T19:58:29.231 DEBUG:boto:Headers: {} 2023-05-31T19:58:29.232 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T19:58:29.232 DEBUG:boto:Port: 8001 2023-05-31T19:58:29.232 DEBUG:boto:Params: {} 2023-05-31T19:58:29.233 DEBUG:boto:Token: None 2023-05-31T19:58:29.233 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 19:58:29 GMT /pklxkd-12/dummy?versionId=HuvfE2dxsypTRiytlVrY6cgwTbY4DCS 2023-05-31T19:58:29.234 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:jhDUtzrq8IvkXnjIc55XaxC8I1M= 2023-05-31T19:58:29.234 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 19:58:29 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:jhDUtzrq8IvkXnjIc55XaxC8I1M=', 'Content-Length': '0'} 2023-05-31T19:58:29.236 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 19:57:27 GMT'), ('x-amz-version-id', 'HuvfE2dxsypTRiytlVrY6cgwTbY4DCS'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000e29cd646c2ffbe75-006477a6e5-1112-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 19:58:29 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T19:58:29.237 DEBUG:boto:path=/pklxkd-12/dummy 2023-05-31T19:58:29.237 DEBUG:boto:auth_path=/pklxkd-12/dummy 2023-05-31T19:58:29.237 DEBUG:boto:path=/pklxkd-12/dummy?versionId=HuvfE2dxsypTRiytlVrY6cgwTbY4DCS 2023-05-31T19:58:29.238 DEBUG:boto:auth_path=/pklxkd-12/dummy?versionId=HuvfE2dxsypTRiytlVrY6cgwTbY4DCS 2023-05-31T19:58:29.238 DEBUG:boto:Method: GET 2023-05-31T19:58:29.238 DEBUG:boto:Path: /pklxkd-12/dummy?versionId=HuvfE2dxsypTRiytlVrY6cgwTbY4DCS 2023-05-31T19:58:29.239 DEBUG:boto:Data: 2023-05-31T19:58:29.239 DEBUG:boto:Headers: {} 2023-05-31T19:58:29.239 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T19:58:29.240 DEBUG:boto:Port: 8000 2023-05-31T19:58:29.240 DEBUG:boto:Params: {} 2023-05-31T19:58:29.241 DEBUG:boto:Token: None 2023-05-31T19:58:29.241 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 19:58:29 GMT /pklxkd-12/dummy?versionId=HuvfE2dxsypTRiytlVrY6cgwTbY4DCS 2023-05-31T19:58:29.242 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:jhDUtzrq8IvkXnjIc55XaxC8I1M= 2023-05-31T19:58:29.242 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 19:58:29 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:jhDUtzrq8IvkXnjIc55XaxC8I1M=', 'Content-Length': '0'} 2023-05-31T19:58:29.244 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 19:57:27 GMT'), ('x-amz-version-id', 'HuvfE2dxsypTRiytlVrY6cgwTbY4DCS'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx0000021ffadbda9d0d772-006477a6e5-110e-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 19:58:29 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T19:58:29.244 DEBUG:boto:path=/pklxkd-12/ 2023-05-31T19:58:29.245 DEBUG:boto:auth_path=/pklxkd-12/ 2023-05-31T19:58:29.245 DEBUG:boto:path=/pklxkd-12/?prefix=dummy 2023-05-31T19:58:29.246 DEBUG:boto:auth_path=/pklxkd-12/?prefix=dummy 2023-05-31T19:58:29.246 DEBUG:boto:Method: GET 2023-05-31T19:58:29.246 DEBUG:boto:Path: /pklxkd-12/?prefix=dummy 2023-05-31T19:58:29.247 DEBUG:boto:Data: 2023-05-31T19:58:29.247 DEBUG:boto:Headers: {} 2023-05-31T19:58:29.247 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T19:58:29.248 DEBUG:boto:Port: 8001 2023-05-31T19:58:29.248 DEBUG:boto:Params: {} 2023-05-31T19:58:29.249 DEBUG:boto:Token: None 2023-05-31T19:58:29.249 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 19:58:29 GMT /pklxkd-12/ 2023-05-31T19:58:29.249 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:V8XeFrflDLZUWfbnuWB7iCX+rEs= 2023-05-31T19:58:29.250 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 19:58:29 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:V8XeFrflDLZUWfbnuWB7iCX+rEs=', 'Content-Length': '0'} 2023-05-31T19:58:29.252 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000087b50580dc28bc9-006477a6e5-1112-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 19:58:29 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T19:58:29.252 DEBUG:boto:b'pklxkd-12dummy1000falsedummy2023-05-31T19:57:27.404Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T19:58:29.253 DEBUG:boto:path=/pklxkd-12/ 2023-05-31T19:58:29.254 DEBUG:boto:auth_path=/pklxkd-12/ 2023-05-31T19:58:29.254 DEBUG:boto:path=/pklxkd-12/?prefix=dummy 2023-05-31T19:58:29.255 DEBUG:boto:auth_path=/pklxkd-12/?prefix=dummy 2023-05-31T19:58:29.255 DEBUG:boto:Method: GET 2023-05-31T19:58:29.256 DEBUG:boto:Path: /pklxkd-12/?prefix=dummy 2023-05-31T19:58:29.256 DEBUG:boto:Data: 2023-05-31T19:58:29.257 DEBUG:boto:Headers: {} 2023-05-31T19:58:29.257 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T19:58:29.257 DEBUG:boto:Port: 8000 2023-05-31T19:58:29.258 DEBUG:boto:Params: {} 2023-05-31T19:58:29.258 DEBUG:boto:Token: None 2023-05-31T19:58:29.258 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 19:58:29 GMT /pklxkd-12/ 2023-05-31T19:58:29.259 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:V8XeFrflDLZUWfbnuWB7iCX+rEs= 2023-05-31T19:58:29.259 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 19:58:29 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:V8XeFrflDLZUWfbnuWB7iCX+rEs=', 'Content-Length': '0'} 2023-05-31T19:58:29.261 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ed8f23e76ac39d3a-006477a6e5-110e-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 19:58:29 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T19:58:29.304 DEBUG:boto:b'pklxkd-12dummy1000falsedummy2023-05-31T19:57:27.404Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T19:58:29.305 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-a1 2023-05-31T19:58:29.305 DEBUG:boto:path=/pklxkd-12/obj-a1 2023-05-31T19:58:29.306 DEBUG:boto:auth_path=/pklxkd-12/obj-a1 2023-05-31T19:58:29.306 DEBUG:boto:path=/pklxkd-12/obj-a1?versionId=G1y4LrpL0rJndINHI52T89ADrfTweD3 2023-05-31T19:58:29.307 DEBUG:boto:auth_path=/pklxkd-12/obj-a1?versionId=G1y4LrpL0rJndINHI52T89ADrfTweD3 2023-05-31T19:58:29.307 DEBUG:boto:Method: GET 2023-05-31T19:58:29.307 DEBUG:boto:Path: /pklxkd-12/obj-a1?versionId=G1y4LrpL0rJndINHI52T89ADrfTweD3 2023-05-31T19:58:29.308 DEBUG:boto:Data: 2023-05-31T19:58:29.308 DEBUG:boto:Headers: {} 2023-05-31T19:58:29.308 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T19:58:29.309 DEBUG:boto:Port: 8001 2023-05-31T19:58:29.309 DEBUG:boto:Params: {} 2023-05-31T19:58:29.310 DEBUG:boto:Token: None 2023-05-31T19:58:29.310 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 19:58:29 GMT /pklxkd-12/obj-a1?versionId=G1y4LrpL0rJndINHI52T89ADrfTweD3 2023-05-31T19:58:29.310 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:kIsTk7vbCVU6tm3vmHK7dYyWl2Q= 2023-05-31T19:58:29.311 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 19:58:29 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:kIsTk7vbCVU6tm3vmHK7dYyWl2Q=', 'Content-Length': '0'} 2023-05-31T19:58:29.313 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 19:57:47 GMT'), ('x-amz-version-id', 'G1y4LrpL0rJndINHI52T89ADrfTweD3'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000001cbb57baac0c6c2d-006477a6e5-1112-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 19:58:29 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T19:58:29.313 DEBUG:boto:path=/pklxkd-12/obj-a1 2023-05-31T19:58:29.314 DEBUG:boto:auth_path=/pklxkd-12/obj-a1 2023-05-31T19:58:29.314 DEBUG:boto:path=/pklxkd-12/obj-a1?versionId=G1y4LrpL0rJndINHI52T89ADrfTweD3 2023-05-31T19:58:29.314 DEBUG:boto:auth_path=/pklxkd-12/obj-a1?versionId=G1y4LrpL0rJndINHI52T89ADrfTweD3 2023-05-31T19:58:29.315 DEBUG:boto:Method: GET 2023-05-31T19:58:29.315 DEBUG:boto:Path: /pklxkd-12/obj-a1?versionId=G1y4LrpL0rJndINHI52T89ADrfTweD3 2023-05-31T19:58:29.315 DEBUG:boto:Data: 2023-05-31T19:58:29.316 DEBUG:boto:Headers: {} 2023-05-31T19:58:29.316 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T19:58:29.317 DEBUG:boto:Port: 8000 2023-05-31T19:58:29.317 DEBUG:boto:Params: {} 2023-05-31T19:58:29.317 DEBUG:boto:Token: None 2023-05-31T19:58:29.318 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 19:58:29 GMT /pklxkd-12/obj-a1?versionId=G1y4LrpL0rJndINHI52T89ADrfTweD3 2023-05-31T19:58:29.318 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:kIsTk7vbCVU6tm3vmHK7dYyWl2Q= 2023-05-31T19:58:29.319 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 19:58:29 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:kIsTk7vbCVU6tm3vmHK7dYyWl2Q=', 'Content-Length': '0'} 2023-05-31T19:58:29.320 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 19:57:47 GMT'), ('x-amz-version-id', 'G1y4LrpL0rJndINHI52T89ADrfTweD3'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000067531ac864bc09af-006477a6e5-110e-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 19:58:29 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T19:58:29.321 DEBUG:boto:path=/pklxkd-12/obj-a1 2023-05-31T19:58:29.321 DEBUG:boto:auth_path=/pklxkd-12/obj-a1 2023-05-31T19:58:29.321 DEBUG:boto:path=/pklxkd-12/obj-a1?versionId=G1y4LrpL0rJndINHI52T89ADrfTweD3 2023-05-31T19:58:29.322 DEBUG:boto:auth_path=/pklxkd-12/obj-a1?versionId=G1y4LrpL0rJndINHI52T89ADrfTweD3 2023-05-31T19:58:29.322 DEBUG:boto:Method: HEAD 2023-05-31T19:58:29.322 DEBUG:boto:Path: /pklxkd-12/obj-a1?versionId=G1y4LrpL0rJndINHI52T89ADrfTweD3 2023-05-31T19:58:29.323 DEBUG:boto:Data: 2023-05-31T19:58:29.323 DEBUG:boto:Headers: {} 2023-05-31T19:58:29.323 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T19:58:29.324 DEBUG:boto:Port: 8001 2023-05-31T19:58:29.324 DEBUG:boto:Params: {} 2023-05-31T19:58:29.325 DEBUG:boto:Token: None 2023-05-31T19:58:29.325 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 19:58:29 GMT /pklxkd-12/obj-a1?versionId=G1y4LrpL0rJndINHI52T89ADrfTweD3 2023-05-31T19:58:29.325 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:rtYpDR6f9oZ5c3DWJxr/0jOiuXk= 2023-05-31T19:58:29.326 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 19:58:29 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:rtYpDR6f9oZ5c3DWJxr/0jOiuXk=', 'Content-Length': '0'} 2023-05-31T19:58:29.327 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 19:57:47 GMT'), ('x-amz-version-id', 'G1y4LrpL0rJndINHI52T89ADrfTweD3'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000a17f74f6e7b4a598-006477a6e5-1112-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 19:58:29 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T19:58:29.328 DEBUG:boto:path=/pklxkd-12/obj-a1 2023-05-31T19:58:29.328 DEBUG:boto:auth_path=/pklxkd-12/obj-a1 2023-05-31T19:58:29.329 DEBUG:boto:path=/pklxkd-12/obj-a1?versionId=G1y4LrpL0rJndINHI52T89ADrfTweD3 2023-05-31T19:58:29.329 DEBUG:boto:auth_path=/pklxkd-12/obj-a1?versionId=G1y4LrpL0rJndINHI52T89ADrfTweD3 2023-05-31T19:58:29.329 DEBUG:boto:Method: HEAD 2023-05-31T19:58:29.330 DEBUG:boto:Path: /pklxkd-12/obj-a1?versionId=G1y4LrpL0rJndINHI52T89ADrfTweD3 2023-05-31T19:58:29.330 DEBUG:boto:Data: 2023-05-31T19:58:29.330 DEBUG:boto:Headers: {} 2023-05-31T19:58:29.331 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T19:58:29.331 DEBUG:boto:Port: 8000 2023-05-31T19:58:29.331 DEBUG:boto:Params: {} 2023-05-31T19:58:29.332 DEBUG:boto:Token: None 2023-05-31T19:58:29.332 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 19:58:29 GMT /pklxkd-12/obj-a1?versionId=G1y4LrpL0rJndINHI52T89ADrfTweD3 2023-05-31T19:58:29.333 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:rtYpDR6f9oZ5c3DWJxr/0jOiuXk= 2023-05-31T19:58:29.333 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 19:58:29 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:rtYpDR6f9oZ5c3DWJxr/0jOiuXk=', 'Content-Length': '0'} 2023-05-31T19:58:29.335 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 19:57:47 GMT'), ('x-amz-version-id', 'G1y4LrpL0rJndINHI52T89ADrfTweD3'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000050f350f2e8c318df-006477a6e5-110e-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 19:58:29 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T19:58:29.335 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-a1 2023-05-31T19:58:29.336 DEBUG:boto:path=/pklxkd-12/obj-a1 2023-05-31T19:58:29.336 DEBUG:boto:auth_path=/pklxkd-12/obj-a1 2023-05-31T19:58:29.336 DEBUG:boto:path=/pklxkd-12/obj-a1?versionId=G1y4LrpL0rJndINHI52T89ADrfTweD3 2023-05-31T19:58:29.337 DEBUG:boto:auth_path=/pklxkd-12/obj-a1?versionId=G1y4LrpL0rJndINHI52T89ADrfTweD3 2023-05-31T19:58:29.337 DEBUG:boto:Method: GET 2023-05-31T19:58:29.337 DEBUG:boto:Path: /pklxkd-12/obj-a1?versionId=G1y4LrpL0rJndINHI52T89ADrfTweD3 2023-05-31T19:58:29.338 DEBUG:boto:Data: 2023-05-31T19:58:29.338 DEBUG:boto:Headers: {} 2023-05-31T19:58:29.338 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T19:58:29.339 DEBUG:boto:Port: 8001 2023-05-31T19:58:29.339 DEBUG:boto:Params: {} 2023-05-31T19:58:29.340 DEBUG:boto:Token: None 2023-05-31T19:58:29.340 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 19:58:29 GMT /pklxkd-12/obj-a1?versionId=G1y4LrpL0rJndINHI52T89ADrfTweD3 2023-05-31T19:58:29.340 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:kIsTk7vbCVU6tm3vmHK7dYyWl2Q= 2023-05-31T19:58:29.341 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 19:58:29 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:kIsTk7vbCVU6tm3vmHK7dYyWl2Q=', 'Content-Length': '0'} 2023-05-31T19:58:29.342 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 19:57:47 GMT'), ('x-amz-version-id', 'G1y4LrpL0rJndINHI52T89ADrfTweD3'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000047f19189694ebea3-006477a6e5-1112-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 19:58:29 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T19:58:29.343 DEBUG:boto:path=/pklxkd-12/obj-a1 2023-05-31T19:58:29.343 DEBUG:boto:auth_path=/pklxkd-12/obj-a1 2023-05-31T19:58:29.344 DEBUG:boto:path=/pklxkd-12/obj-a1?versionId=G1y4LrpL0rJndINHI52T89ADrfTweD3 2023-05-31T19:58:29.344 DEBUG:boto:auth_path=/pklxkd-12/obj-a1?versionId=G1y4LrpL0rJndINHI52T89ADrfTweD3 2023-05-31T19:58:29.344 DEBUG:boto:Method: GET 2023-05-31T19:58:29.345 DEBUG:boto:Path: /pklxkd-12/obj-a1?versionId=G1y4LrpL0rJndINHI52T89ADrfTweD3 2023-05-31T19:58:29.345 DEBUG:boto:Data: 2023-05-31T19:58:29.346 DEBUG:boto:Headers: {} 2023-05-31T19:58:29.346 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T19:58:29.346 DEBUG:boto:Port: 8000 2023-05-31T19:58:29.347 DEBUG:boto:Params: {} 2023-05-31T19:58:29.347 DEBUG:boto:Token: None 2023-05-31T19:58:29.347 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 19:58:29 GMT /pklxkd-12/obj-a1?versionId=G1y4LrpL0rJndINHI52T89ADrfTweD3 2023-05-31T19:58:29.348 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:kIsTk7vbCVU6tm3vmHK7dYyWl2Q= 2023-05-31T19:58:29.348 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 19:58:29 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:kIsTk7vbCVU6tm3vmHK7dYyWl2Q=', 'Content-Length': '0'} 2023-05-31T19:58:29.350 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 19:57:47 GMT'), ('x-amz-version-id', 'G1y4LrpL0rJndINHI52T89ADrfTweD3'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000ed8103312bf0c4ea-006477a6e5-110e-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 19:58:29 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T19:58:29.351 DEBUG:boto:path=/pklxkd-12/ 2023-05-31T19:58:29.351 DEBUG:boto:auth_path=/pklxkd-12/ 2023-05-31T19:58:29.351 DEBUG:boto:path=/pklxkd-12/?prefix=obj-a1 2023-05-31T19:58:29.352 DEBUG:boto:auth_path=/pklxkd-12/?prefix=obj-a1 2023-05-31T19:58:29.352 DEBUG:boto:Method: GET 2023-05-31T19:58:29.353 DEBUG:boto:Path: /pklxkd-12/?prefix=obj-a1 2023-05-31T19:58:29.353 DEBUG:boto:Data: 2023-05-31T19:58:29.353 DEBUG:boto:Headers: {} 2023-05-31T19:58:29.354 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T19:58:29.354 DEBUG:boto:Port: 8001 2023-05-31T19:58:29.354 DEBUG:boto:Params: {} 2023-05-31T19:58:29.355 DEBUG:boto:Token: None 2023-05-31T19:58:29.355 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 19:58:29 GMT /pklxkd-12/ 2023-05-31T19:58:29.355 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:V8XeFrflDLZUWfbnuWB7iCX+rEs= 2023-05-31T19:58:29.356 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 19:58:29 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:V8XeFrflDLZUWfbnuWB7iCX+rEs=', 'Content-Length': '0'} 2023-05-31T19:58:29.358 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c3e9286271136d6a-006477a6e5-1112-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 19:58:29 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T19:58:29.359 DEBUG:boto:b'pklxkd-12obj-a11000falseobj-a12023-05-31T19:57:47.837Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T19:58:29.359 DEBUG:boto:path=/pklxkd-12/ 2023-05-31T19:58:29.360 DEBUG:boto:auth_path=/pklxkd-12/ 2023-05-31T19:58:29.360 DEBUG:boto:path=/pklxkd-12/?prefix=obj-a1 2023-05-31T19:58:29.361 DEBUG:boto:auth_path=/pklxkd-12/?prefix=obj-a1 2023-05-31T19:58:29.361 DEBUG:boto:Method: GET 2023-05-31T19:58:29.361 DEBUG:boto:Path: /pklxkd-12/?prefix=obj-a1 2023-05-31T19:58:29.362 DEBUG:boto:Data: 2023-05-31T19:58:29.362 DEBUG:boto:Headers: {} 2023-05-31T19:58:29.362 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T19:58:29.363 DEBUG:boto:Port: 8000 2023-05-31T19:58:29.363 DEBUG:boto:Params: {} 2023-05-31T19:58:29.363 DEBUG:boto:Token: None 2023-05-31T19:58:29.364 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 19:58:29 GMT /pklxkd-12/ 2023-05-31T19:58:29.364 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:V8XeFrflDLZUWfbnuWB7iCX+rEs= 2023-05-31T19:58:29.365 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 19:58:29 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:V8XeFrflDLZUWfbnuWB7iCX+rEs=', 'Content-Length': '0'} 2023-05-31T19:58:29.367 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000846b38dc251956f6-006477a6e5-110e-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 19:58:29 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T19:58:29.367 DEBUG:boto:b'pklxkd-12obj-a11000falseobj-a12023-05-31T19:57:47.837Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T19:58:29.368 DEBUG:boto:path=/pklxkd-12/obj-a1 2023-05-31T19:58:29.368 DEBUG:boto:auth_path=/pklxkd-12/obj-a1 2023-05-31T19:58:29.369 DEBUG:boto:Method: HEAD 2023-05-31T19:58:29.369 DEBUG:boto:Path: /pklxkd-12/obj-a1 2023-05-31T19:58:29.369 DEBUG:boto:Data: 2023-05-31T19:58:29.370 DEBUG:boto:Headers: {} 2023-05-31T19:58:29.370 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T19:58:29.370 DEBUG:boto:Port: 8001 2023-05-31T19:58:29.370 DEBUG:boto:Params: {} 2023-05-31T19:58:29.371 DEBUG:boto:Token: None 2023-05-31T19:58:29.371 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 19:58:29 GMT /pklxkd-12/obj-a1 2023-05-31T19:58:29.372 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:7N6nBR6qtRPQK8alNTPPsfBtZgg= 2023-05-31T19:58:29.372 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 19:58:29 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:7N6nBR6qtRPQK8alNTPPsfBtZgg=', 'Content-Length': '0'} 2023-05-31T19:58:29.374 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 19:57:47 GMT'), ('x-amz-version-id', 'G1y4LrpL0rJndINHI52T89ADrfTweD3'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000002a51f86f2dd34bca-006477a6e5-1112-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 19:58:29 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T19:58:29.374 DEBUG:boto:path=/pklxkd-12/obj-a1 2023-05-31T19:58:29.375 DEBUG:boto:auth_path=/pklxkd-12/obj-a1 2023-05-31T19:58:29.375 DEBUG:boto:Method: HEAD 2023-05-31T19:58:29.376 DEBUG:boto:Path: /pklxkd-12/obj-a1 2023-05-31T19:58:29.376 DEBUG:boto:Data: 2023-05-31T19:58:29.376 DEBUG:boto:Headers: {} 2023-05-31T19:58:29.377 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T19:58:29.377 DEBUG:boto:Port: 8000 2023-05-31T19:58:29.377 DEBUG:boto:Params: {} 2023-05-31T19:58:29.378 DEBUG:boto:Token: None 2023-05-31T19:58:29.378 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 19:58:29 GMT /pklxkd-12/obj-a1 2023-05-31T19:58:29.378 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:7N6nBR6qtRPQK8alNTPPsfBtZgg= 2023-05-31T19:58:29.379 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 19:58:29 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:7N6nBR6qtRPQK8alNTPPsfBtZgg=', 'Content-Length': '0'} 2023-05-31T19:58:29.381 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 19:57:47 GMT'), ('x-amz-version-id', 'G1y4LrpL0rJndINHI52T89ADrfTweD3'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000fb032b45ae1327d8-006477a6e5-110e-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 19:58:29 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T19:58:29.381 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-a1 2023-05-31T19:58:29.382 DEBUG:boto:path=/pklxkd-12/obj-a1 2023-05-31T19:58:29.382 DEBUG:boto:auth_path=/pklxkd-12/obj-a1 2023-05-31T19:58:29.382 DEBUG:boto:path=/pklxkd-12/obj-a1?versionId=G1y4LrpL0rJndINHI52T89ADrfTweD3 2023-05-31T19:58:29.383 DEBUG:boto:auth_path=/pklxkd-12/obj-a1?versionId=G1y4LrpL0rJndINHI52T89ADrfTweD3 2023-05-31T19:58:29.383 DEBUG:boto:Method: GET 2023-05-31T19:58:29.383 DEBUG:boto:Path: /pklxkd-12/obj-a1?versionId=G1y4LrpL0rJndINHI52T89ADrfTweD3 2023-05-31T19:58:29.384 DEBUG:boto:Data: 2023-05-31T19:58:29.384 DEBUG:boto:Headers: {} 2023-05-31T19:58:29.385 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T19:58:29.385 DEBUG:boto:Port: 8001 2023-05-31T19:58:29.385 DEBUG:boto:Params: {} 2023-05-31T19:58:29.386 DEBUG:boto:Token: None 2023-05-31T19:58:29.386 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 19:58:29 GMT /pklxkd-12/obj-a1?versionId=G1y4LrpL0rJndINHI52T89ADrfTweD3 2023-05-31T19:58:29.387 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:kIsTk7vbCVU6tm3vmHK7dYyWl2Q= 2023-05-31T19:58:29.387 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 19:58:29 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:kIsTk7vbCVU6tm3vmHK7dYyWl2Q=', 'Content-Length': '0'} 2023-05-31T19:58:29.389 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 19:57:47 GMT'), ('x-amz-version-id', 'G1y4LrpL0rJndINHI52T89ADrfTweD3'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000a4d8cf0c18a1b3bf-006477a6e5-1112-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 19:58:29 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T19:58:29.389 DEBUG:boto:path=/pklxkd-12/obj-a1 2023-05-31T19:58:29.390 DEBUG:boto:auth_path=/pklxkd-12/obj-a1 2023-05-31T19:58:29.390 DEBUG:boto:path=/pklxkd-12/obj-a1?versionId=G1y4LrpL0rJndINHI52T89ADrfTweD3 2023-05-31T19:58:29.390 DEBUG:boto:auth_path=/pklxkd-12/obj-a1?versionId=G1y4LrpL0rJndINHI52T89ADrfTweD3 2023-05-31T19:58:29.391 DEBUG:boto:Method: GET 2023-05-31T19:58:29.391 DEBUG:boto:Path: /pklxkd-12/obj-a1?versionId=G1y4LrpL0rJndINHI52T89ADrfTweD3 2023-05-31T19:58:29.391 DEBUG:boto:Data: 2023-05-31T19:58:29.392 DEBUG:boto:Headers: {} 2023-05-31T19:58:29.392 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T19:58:29.393 DEBUG:boto:Port: 8000 2023-05-31T19:58:29.393 DEBUG:boto:Params: {} 2023-05-31T19:58:29.393 DEBUG:boto:Token: None 2023-05-31T19:58:29.394 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 19:58:29 GMT /pklxkd-12/obj-a1?versionId=G1y4LrpL0rJndINHI52T89ADrfTweD3 2023-05-31T19:58:29.394 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:kIsTk7vbCVU6tm3vmHK7dYyWl2Q= 2023-05-31T19:58:29.395 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 19:58:29 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:kIsTk7vbCVU6tm3vmHK7dYyWl2Q=', 'Content-Length': '0'} 2023-05-31T19:58:29.396 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 19:57:47 GMT'), ('x-amz-version-id', 'G1y4LrpL0rJndINHI52T89ADrfTweD3'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000009273126737b0aa2e-006477a6e5-110e-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 19:58:29 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T19:58:29.444 DEBUG:boto:path=/pklxkd-12/ 2023-05-31T19:58:29.445 DEBUG:boto:auth_path=/pklxkd-12/ 2023-05-31T19:58:29.445 DEBUG:boto:path=/pklxkd-12/?prefix=obj-a1 2023-05-31T19:58:29.446 DEBUG:boto:auth_path=/pklxkd-12/?prefix=obj-a1 2023-05-31T19:58:29.446 DEBUG:boto:Method: GET 2023-05-31T19:58:29.446 DEBUG:boto:Path: /pklxkd-12/?prefix=obj-a1 2023-05-31T19:58:29.447 DEBUG:boto:Data: 2023-05-31T19:58:29.447 DEBUG:boto:Headers: {} 2023-05-31T19:58:29.447 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T19:58:29.448 DEBUG:boto:Port: 8001 2023-05-31T19:58:29.448 DEBUG:boto:Params: {} 2023-05-31T19:58:29.449 DEBUG:boto:Token: None 2023-05-31T19:58:29.449 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 19:58:29 GMT /pklxkd-12/ 2023-05-31T19:58:29.449 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:V8XeFrflDLZUWfbnuWB7iCX+rEs= 2023-05-31T19:58:29.450 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 19:58:29 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:V8XeFrflDLZUWfbnuWB7iCX+rEs=', 'Content-Length': '0'} 2023-05-31T19:58:29.452 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000dcc7350ac1bd6921-006477a6e5-1112-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 19:58:29 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T19:58:29.452 DEBUG:boto:b'pklxkd-12obj-a11000falseobj-a12023-05-31T19:57:47.837Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T19:58:29.453 DEBUG:boto:path=/pklxkd-12/ 2023-05-31T19:58:29.454 DEBUG:boto:auth_path=/pklxkd-12/ 2023-05-31T19:58:29.454 DEBUG:boto:path=/pklxkd-12/?prefix=obj-a1 2023-05-31T19:58:29.454 DEBUG:boto:auth_path=/pklxkd-12/?prefix=obj-a1 2023-05-31T19:58:29.455 DEBUG:boto:Method: GET 2023-05-31T19:58:29.455 DEBUG:boto:Path: /pklxkd-12/?prefix=obj-a1 2023-05-31T19:58:29.455 DEBUG:boto:Data: 2023-05-31T19:58:29.456 DEBUG:boto:Headers: {} 2023-05-31T19:58:29.456 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T19:58:29.456 DEBUG:boto:Port: 8000 2023-05-31T19:58:29.457 DEBUG:boto:Params: {} 2023-05-31T19:58:29.457 DEBUG:boto:Token: None 2023-05-31T19:58:29.458 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 19:58:29 GMT /pklxkd-12/ 2023-05-31T19:58:29.458 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:V8XeFrflDLZUWfbnuWB7iCX+rEs= 2023-05-31T19:58:29.458 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 19:58:29 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:V8XeFrflDLZUWfbnuWB7iCX+rEs=', 'Content-Length': '0'} 2023-05-31T19:58:29.460 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000041ce601620d45274-006477a6e5-110e-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 19:58:29 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T19:58:29.461 DEBUG:boto:b'pklxkd-12obj-a11000falseobj-a12023-05-31T19:57:47.837Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T19:58:29.462 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-a2 2023-05-31T19:58:29.462 DEBUG:boto:path=/pklxkd-12/obj-a2 2023-05-31T19:58:29.462 DEBUG:boto:auth_path=/pklxkd-12/obj-a2 2023-05-31T19:58:29.463 DEBUG:boto:path=/pklxkd-12/obj-a2?versionId=qiyfRuud8o74hAGpczwf4-FuN5p5887 2023-05-31T19:58:29.463 DEBUG:boto:auth_path=/pklxkd-12/obj-a2?versionId=qiyfRuud8o74hAGpczwf4-FuN5p5887 2023-05-31T19:58:29.463 DEBUG:boto:Method: GET 2023-05-31T19:58:29.464 DEBUG:boto:Path: /pklxkd-12/obj-a2?versionId=qiyfRuud8o74hAGpczwf4-FuN5p5887 2023-05-31T19:58:29.464 DEBUG:boto:Data: 2023-05-31T19:58:29.464 DEBUG:boto:Headers: {} 2023-05-31T19:58:29.465 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T19:58:29.465 DEBUG:boto:Port: 8001 2023-05-31T19:58:29.466 DEBUG:boto:Params: {} 2023-05-31T19:58:29.466 DEBUG:boto:Token: None 2023-05-31T19:58:29.466 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 19:58:29 GMT /pklxkd-12/obj-a2?versionId=qiyfRuud8o74hAGpczwf4-FuN5p5887 2023-05-31T19:58:29.467 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:wQSffbIDPKVoZvzPeZLs1AJ8QTs= 2023-05-31T19:58:29.467 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 19:58:29 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:wQSffbIDPKVoZvzPeZLs1AJ8QTs=', 'Content-Length': '0'} 2023-05-31T19:58:29.469 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 19:57:47 GMT'), ('x-amz-version-id', 'qiyfRuud8o74hAGpczwf4-FuN5p5887'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000002dffde183a9d2d6a-006477a6e5-1112-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 19:58:29 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T19:58:29.470 DEBUG:boto:path=/pklxkd-12/obj-a2 2023-05-31T19:58:29.470 DEBUG:boto:auth_path=/pklxkd-12/obj-a2 2023-05-31T19:58:29.470 DEBUG:boto:path=/pklxkd-12/obj-a2?versionId=qiyfRuud8o74hAGpczwf4-FuN5p5887 2023-05-31T19:58:29.471 DEBUG:boto:auth_path=/pklxkd-12/obj-a2?versionId=qiyfRuud8o74hAGpczwf4-FuN5p5887 2023-05-31T19:58:29.471 DEBUG:boto:Method: GET 2023-05-31T19:58:29.471 DEBUG:boto:Path: /pklxkd-12/obj-a2?versionId=qiyfRuud8o74hAGpczwf4-FuN5p5887 2023-05-31T19:58:29.472 DEBUG:boto:Data: 2023-05-31T19:58:29.472 DEBUG:boto:Headers: {} 2023-05-31T19:58:29.472 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T19:58:29.473 DEBUG:boto:Port: 8000 2023-05-31T19:58:29.473 DEBUG:boto:Params: {} 2023-05-31T19:58:29.474 DEBUG:boto:Token: None 2023-05-31T19:58:29.474 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 19:58:29 GMT /pklxkd-12/obj-a2?versionId=qiyfRuud8o74hAGpczwf4-FuN5p5887 2023-05-31T19:58:29.474 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:wQSffbIDPKVoZvzPeZLs1AJ8QTs= 2023-05-31T19:58:29.475 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 19:58:29 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:wQSffbIDPKVoZvzPeZLs1AJ8QTs=', 'Content-Length': '0'} 2023-05-31T19:58:29.476 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 19:57:47 GMT'), ('x-amz-version-id', 'qiyfRuud8o74hAGpczwf4-FuN5p5887'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000b5f92a869acbe639-006477a6e5-110e-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 19:58:29 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T19:58:29.477 DEBUG:boto:path=/pklxkd-12/obj-a2 2023-05-31T19:58:29.477 DEBUG:boto:auth_path=/pklxkd-12/obj-a2 2023-05-31T19:58:29.478 DEBUG:boto:path=/pklxkd-12/obj-a2?versionId=qiyfRuud8o74hAGpczwf4-FuN5p5887 2023-05-31T19:58:29.478 DEBUG:boto:auth_path=/pklxkd-12/obj-a2?versionId=qiyfRuud8o74hAGpczwf4-FuN5p5887 2023-05-31T19:58:29.478 DEBUG:boto:Method: HEAD 2023-05-31T19:58:29.479 DEBUG:boto:Path: /pklxkd-12/obj-a2?versionId=qiyfRuud8o74hAGpczwf4-FuN5p5887 2023-05-31T19:58:29.479 DEBUG:boto:Data: 2023-05-31T19:58:29.479 DEBUG:boto:Headers: {} 2023-05-31T19:58:29.480 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T19:58:29.480 DEBUG:boto:Port: 8001 2023-05-31T19:58:29.480 DEBUG:boto:Params: {} 2023-05-31T19:58:29.481 DEBUG:boto:Token: None 2023-05-31T19:58:29.481 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 19:58:29 GMT /pklxkd-12/obj-a2?versionId=qiyfRuud8o74hAGpczwf4-FuN5p5887 2023-05-31T19:58:29.481 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:e23NL6xskkRRbeuNKl3BvQS+5DU= 2023-05-31T19:58:29.482 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 19:58:29 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:e23NL6xskkRRbeuNKl3BvQS+5DU=', 'Content-Length': '0'} 2023-05-31T19:58:29.483 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 19:57:47 GMT'), ('x-amz-version-id', 'qiyfRuud8o74hAGpczwf4-FuN5p5887'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000008776d6fdf86075ea-006477a6e5-1112-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 19:58:29 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T19:58:29.484 DEBUG:boto:path=/pklxkd-12/obj-a2 2023-05-31T19:58:29.484 DEBUG:boto:auth_path=/pklxkd-12/obj-a2 2023-05-31T19:58:29.485 DEBUG:boto:path=/pklxkd-12/obj-a2?versionId=qiyfRuud8o74hAGpczwf4-FuN5p5887 2023-05-31T19:58:29.485 DEBUG:boto:auth_path=/pklxkd-12/obj-a2?versionId=qiyfRuud8o74hAGpczwf4-FuN5p5887 2023-05-31T19:58:29.485 DEBUG:boto:Method: HEAD 2023-05-31T19:58:29.486 DEBUG:boto:Path: /pklxkd-12/obj-a2?versionId=qiyfRuud8o74hAGpczwf4-FuN5p5887 2023-05-31T19:58:29.486 DEBUG:boto:Data: 2023-05-31T19:58:29.486 DEBUG:boto:Headers: {} 2023-05-31T19:58:29.487 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T19:58:29.487 DEBUG:boto:Port: 8000 2023-05-31T19:58:29.487 DEBUG:boto:Params: {} 2023-05-31T19:58:29.488 DEBUG:boto:Token: None 2023-05-31T19:58:29.488 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 19:58:29 GMT /pklxkd-12/obj-a2?versionId=qiyfRuud8o74hAGpczwf4-FuN5p5887 2023-05-31T19:58:29.488 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:e23NL6xskkRRbeuNKl3BvQS+5DU= 2023-05-31T19:58:29.489 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 19:58:29 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:e23NL6xskkRRbeuNKl3BvQS+5DU=', 'Content-Length': '0'} 2023-05-31T19:58:29.490 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 19:57:47 GMT'), ('x-amz-version-id', 'qiyfRuud8o74hAGpczwf4-FuN5p5887'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000001dc9bf5b593d8327-006477a6e5-110e-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 19:58:29 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T19:58:29.491 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-a2 2023-05-31T19:58:29.491 DEBUG:boto:path=/pklxkd-12/obj-a2 2023-05-31T19:58:29.492 DEBUG:boto:auth_path=/pklxkd-12/obj-a2 2023-05-31T19:58:29.492 DEBUG:boto:path=/pklxkd-12/obj-a2?versionId=qiyfRuud8o74hAGpczwf4-FuN5p5887 2023-05-31T19:58:29.492 DEBUG:boto:auth_path=/pklxkd-12/obj-a2?versionId=qiyfRuud8o74hAGpczwf4-FuN5p5887 2023-05-31T19:58:29.493 DEBUG:boto:Method: GET 2023-05-31T19:58:29.493 DEBUG:boto:Path: /pklxkd-12/obj-a2?versionId=qiyfRuud8o74hAGpczwf4-FuN5p5887 2023-05-31T19:58:29.493 DEBUG:boto:Data: 2023-05-31T19:58:29.494 DEBUG:boto:Headers: {} 2023-05-31T19:58:29.494 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T19:58:29.494 DEBUG:boto:Port: 8001 2023-05-31T19:58:29.495 DEBUG:boto:Params: {} 2023-05-31T19:58:29.495 DEBUG:boto:Token: None 2023-05-31T19:58:29.496 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 19:58:29 GMT /pklxkd-12/obj-a2?versionId=qiyfRuud8o74hAGpczwf4-FuN5p5887 2023-05-31T19:58:29.496 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:wQSffbIDPKVoZvzPeZLs1AJ8QTs= 2023-05-31T19:58:29.496 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 19:58:29 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:wQSffbIDPKVoZvzPeZLs1AJ8QTs=', 'Content-Length': '0'} 2023-05-31T19:58:29.498 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 19:57:47 GMT'), ('x-amz-version-id', 'qiyfRuud8o74hAGpczwf4-FuN5p5887'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000009397ee0a68bc9178-006477a6e5-1112-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 19:58:29 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T19:58:29.540 DEBUG:boto:path=/pklxkd-12/obj-a2 2023-05-31T19:58:29.541 DEBUG:boto:auth_path=/pklxkd-12/obj-a2 2023-05-31T19:58:29.541 DEBUG:boto:path=/pklxkd-12/obj-a2?versionId=qiyfRuud8o74hAGpczwf4-FuN5p5887 2023-05-31T19:58:29.541 DEBUG:boto:auth_path=/pklxkd-12/obj-a2?versionId=qiyfRuud8o74hAGpczwf4-FuN5p5887 2023-05-31T19:58:29.542 DEBUG:boto:Method: GET 2023-05-31T19:58:29.542 DEBUG:boto:Path: /pklxkd-12/obj-a2?versionId=qiyfRuud8o74hAGpczwf4-FuN5p5887 2023-05-31T19:58:29.542 DEBUG:boto:Data: 2023-05-31T19:58:29.543 DEBUG:boto:Headers: {} 2023-05-31T19:58:29.543 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T19:58:29.544 DEBUG:boto:Port: 8000 2023-05-31T19:58:29.544 DEBUG:boto:Params: {} 2023-05-31T19:58:29.544 DEBUG:boto:Token: None 2023-05-31T19:58:29.545 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 19:58:29 GMT /pklxkd-12/obj-a2?versionId=qiyfRuud8o74hAGpczwf4-FuN5p5887 2023-05-31T19:58:29.545 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:wQSffbIDPKVoZvzPeZLs1AJ8QTs= 2023-05-31T19:58:29.546 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 19:58:29 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:wQSffbIDPKVoZvzPeZLs1AJ8QTs=', 'Content-Length': '0'} 2023-05-31T19:58:29.547 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 19:57:47 GMT'), ('x-amz-version-id', 'qiyfRuud8o74hAGpczwf4-FuN5p5887'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000b091fbd83e155863-006477a6e5-110e-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 19:58:29 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T19:58:29.548 DEBUG:boto:path=/pklxkd-12/ 2023-05-31T19:58:29.548 DEBUG:boto:auth_path=/pklxkd-12/ 2023-05-31T19:58:29.549 DEBUG:boto:path=/pklxkd-12/?prefix=obj-a2 2023-05-31T19:58:29.549 DEBUG:boto:auth_path=/pklxkd-12/?prefix=obj-a2 2023-05-31T19:58:29.549 DEBUG:boto:Method: GET 2023-05-31T19:58:29.550 DEBUG:boto:Path: /pklxkd-12/?prefix=obj-a2 2023-05-31T19:58:29.550 DEBUG:boto:Data: 2023-05-31T19:58:29.550 DEBUG:boto:Headers: {} 2023-05-31T19:58:29.551 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T19:58:29.551 DEBUG:boto:Port: 8001 2023-05-31T19:58:29.552 DEBUG:boto:Params: {} 2023-05-31T19:58:29.552 DEBUG:boto:Token: None 2023-05-31T19:58:29.552 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 19:58:29 GMT /pklxkd-12/ 2023-05-31T19:58:29.553 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:V8XeFrflDLZUWfbnuWB7iCX+rEs= 2023-05-31T19:58:29.553 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 19:58:29 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:V8XeFrflDLZUWfbnuWB7iCX+rEs=', 'Content-Length': '0'} 2023-05-31T19:58:29.555 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000092151e8c88bf1b7f-006477a6e5-1112-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 19:58:29 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T19:58:29.556 DEBUG:boto:b'pklxkd-12obj-a21000falseobj-a22023-05-31T19:57:47.962Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T19:58:29.556 DEBUG:boto:path=/pklxkd-12/ 2023-05-31T19:58:29.557 DEBUG:boto:auth_path=/pklxkd-12/ 2023-05-31T19:58:29.557 DEBUG:boto:path=/pklxkd-12/?prefix=obj-a2 2023-05-31T19:58:29.557 DEBUG:boto:auth_path=/pklxkd-12/?prefix=obj-a2 2023-05-31T19:58:29.558 DEBUG:boto:Method: GET 2023-05-31T19:58:29.558 DEBUG:boto:Path: /pklxkd-12/?prefix=obj-a2 2023-05-31T19:58:29.559 DEBUG:boto:Data: 2023-05-31T19:58:29.559 DEBUG:boto:Headers: {} 2023-05-31T19:58:29.559 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T19:58:29.560 DEBUG:boto:Port: 8000 2023-05-31T19:58:29.560 DEBUG:boto:Params: {} 2023-05-31T19:58:29.560 DEBUG:boto:Token: None 2023-05-31T19:58:29.561 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 19:58:29 GMT /pklxkd-12/ 2023-05-31T19:58:29.561 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:V8XeFrflDLZUWfbnuWB7iCX+rEs= 2023-05-31T19:58:29.562 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 19:58:29 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:V8XeFrflDLZUWfbnuWB7iCX+rEs=', 'Content-Length': '0'} 2023-05-31T19:58:29.564 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000fb156a6789a78c19-006477a6e5-110e-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 19:58:29 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T19:58:29.564 DEBUG:boto:b'pklxkd-12obj-a21000falseobj-a22023-05-31T19:57:47.962Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T19:58:29.565 DEBUG:boto:path=/pklxkd-12/obj-a2 2023-05-31T19:58:29.565 DEBUG:boto:auth_path=/pklxkd-12/obj-a2 2023-05-31T19:58:29.565 DEBUG:boto:Method: HEAD 2023-05-31T19:58:29.566 DEBUG:boto:Path: /pklxkd-12/obj-a2 2023-05-31T19:58:29.566 DEBUG:boto:Data: 2023-05-31T19:58:29.566 DEBUG:boto:Headers: {} 2023-05-31T19:58:29.567 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T19:58:29.567 DEBUG:boto:Port: 8001 2023-05-31T19:58:29.567 DEBUG:boto:Params: {} 2023-05-31T19:58:29.568 DEBUG:boto:Token: None 2023-05-31T19:58:29.568 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 19:58:29 GMT /pklxkd-12/obj-a2 2023-05-31T19:58:29.569 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:nahcfu2yM6Jc+tt5Gf/XBgPkV3A= 2023-05-31T19:58:29.569 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 19:58:29 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:nahcfu2yM6Jc+tt5Gf/XBgPkV3A=', 'Content-Length': '0'} 2023-05-31T19:58:29.571 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 19:57:47 GMT'), ('x-amz-version-id', 'qiyfRuud8o74hAGpczwf4-FuN5p5887'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000566546c44c7e6184-006477a6e5-1112-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 19:58:29 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T19:58:29.572 DEBUG:boto:path=/pklxkd-12/obj-a2 2023-05-31T19:58:29.572 DEBUG:boto:auth_path=/pklxkd-12/obj-a2 2023-05-31T19:58:29.573 DEBUG:boto:Method: HEAD 2023-05-31T19:58:29.573 DEBUG:boto:Path: /pklxkd-12/obj-a2 2023-05-31T19:58:29.573 DEBUG:boto:Data: 2023-05-31T19:58:29.574 DEBUG:boto:Headers: {} 2023-05-31T19:58:29.574 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T19:58:29.574 DEBUG:boto:Port: 8000 2023-05-31T19:58:29.575 DEBUG:boto:Params: {} 2023-05-31T19:58:29.575 DEBUG:boto:Token: None 2023-05-31T19:58:29.575 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 19:58:29 GMT /pklxkd-12/obj-a2 2023-05-31T19:58:29.576 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:nahcfu2yM6Jc+tt5Gf/XBgPkV3A= 2023-05-31T19:58:29.576 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 19:58:29 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:nahcfu2yM6Jc+tt5Gf/XBgPkV3A=', 'Content-Length': '0'} 2023-05-31T19:58:29.578 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 19:57:47 GMT'), ('x-amz-version-id', 'qiyfRuud8o74hAGpczwf4-FuN5p5887'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000009f4a166a74e56937-006477a6e5-110e-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 19:58:29 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T19:58:29.579 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-a2 2023-05-31T19:58:29.579 DEBUG:boto:path=/pklxkd-12/obj-a2 2023-05-31T19:58:29.579 DEBUG:boto:auth_path=/pklxkd-12/obj-a2 2023-05-31T19:58:29.580 DEBUG:boto:path=/pklxkd-12/obj-a2?versionId=qiyfRuud8o74hAGpczwf4-FuN5p5887 2023-05-31T19:58:29.580 DEBUG:boto:auth_path=/pklxkd-12/obj-a2?versionId=qiyfRuud8o74hAGpczwf4-FuN5p5887 2023-05-31T19:58:29.581 DEBUG:boto:Method: GET 2023-05-31T19:58:29.581 DEBUG:boto:Path: /pklxkd-12/obj-a2?versionId=qiyfRuud8o74hAGpczwf4-FuN5p5887 2023-05-31T19:58:29.581 DEBUG:boto:Data: 2023-05-31T19:58:29.582 DEBUG:boto:Headers: {} 2023-05-31T19:58:29.582 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T19:58:29.582 DEBUG:boto:Port: 8001 2023-05-31T19:58:29.583 DEBUG:boto:Params: {} 2023-05-31T19:58:29.583 DEBUG:boto:Token: None 2023-05-31T19:58:29.583 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 19:58:29 GMT /pklxkd-12/obj-a2?versionId=qiyfRuud8o74hAGpczwf4-FuN5p5887 2023-05-31T19:58:29.584 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:wQSffbIDPKVoZvzPeZLs1AJ8QTs= 2023-05-31T19:58:29.584 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 19:58:29 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:wQSffbIDPKVoZvzPeZLs1AJ8QTs=', 'Content-Length': '0'} 2023-05-31T19:58:29.586 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 19:57:47 GMT'), ('x-amz-version-id', 'qiyfRuud8o74hAGpczwf4-FuN5p5887'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000f1f11173cba131d6-006477a6e5-1112-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 19:58:29 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T19:58:29.586 DEBUG:boto:path=/pklxkd-12/obj-a2 2023-05-31T19:58:29.587 DEBUG:boto:auth_path=/pklxkd-12/obj-a2 2023-05-31T19:58:29.587 DEBUG:boto:path=/pklxkd-12/obj-a2?versionId=qiyfRuud8o74hAGpczwf4-FuN5p5887 2023-05-31T19:58:29.587 DEBUG:boto:auth_path=/pklxkd-12/obj-a2?versionId=qiyfRuud8o74hAGpczwf4-FuN5p5887 2023-05-31T19:58:29.588 DEBUG:boto:Method: GET 2023-05-31T19:58:29.588 DEBUG:boto:Path: /pklxkd-12/obj-a2?versionId=qiyfRuud8o74hAGpczwf4-FuN5p5887 2023-05-31T19:58:29.589 DEBUG:boto:Data: 2023-05-31T19:58:29.589 DEBUG:boto:Headers: {} 2023-05-31T19:58:29.589 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T19:58:29.590 DEBUG:boto:Port: 8000 2023-05-31T19:58:29.590 DEBUG:boto:Params: {} 2023-05-31T19:58:29.590 DEBUG:boto:Token: None 2023-05-31T19:58:29.591 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 19:58:29 GMT /pklxkd-12/obj-a2?versionId=qiyfRuud8o74hAGpczwf4-FuN5p5887 2023-05-31T19:58:29.591 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:wQSffbIDPKVoZvzPeZLs1AJ8QTs= 2023-05-31T19:58:29.592 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 19:58:29 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:wQSffbIDPKVoZvzPeZLs1AJ8QTs=', 'Content-Length': '0'} 2023-05-31T19:58:29.593 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 19:57:47 GMT'), ('x-amz-version-id', 'qiyfRuud8o74hAGpczwf4-FuN5p5887'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000052bfcf3d127f7a1b-006477a6e5-110e-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 19:58:29 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T19:58:29.594 DEBUG:boto:path=/pklxkd-12/ 2023-05-31T19:58:29.594 DEBUG:boto:auth_path=/pklxkd-12/ 2023-05-31T19:58:29.594 DEBUG:boto:path=/pklxkd-12/?prefix=obj-a2 2023-05-31T19:58:29.595 DEBUG:boto:auth_path=/pklxkd-12/?prefix=obj-a2 2023-05-31T19:58:29.595 DEBUG:boto:Method: GET 2023-05-31T19:58:29.596 DEBUG:boto:Path: /pklxkd-12/?prefix=obj-a2 2023-05-31T19:58:29.596 DEBUG:boto:Data: 2023-05-31T19:58:29.596 DEBUG:boto:Headers: {} 2023-05-31T19:58:29.597 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T19:58:29.597 DEBUG:boto:Port: 8001 2023-05-31T19:58:29.597 DEBUG:boto:Params: {} 2023-05-31T19:58:29.598 DEBUG:boto:Token: None 2023-05-31T19:58:29.598 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 19:58:29 GMT /pklxkd-12/ 2023-05-31T19:58:29.599 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:V8XeFrflDLZUWfbnuWB7iCX+rEs= 2023-05-31T19:58:29.599 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 19:58:29 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:V8XeFrflDLZUWfbnuWB7iCX+rEs=', 'Content-Length': '0'} 2023-05-31T19:58:29.601 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c15214516222a932-006477a6e5-1112-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 19:58:29 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T19:58:29.602 DEBUG:boto:b'pklxkd-12obj-a21000falseobj-a22023-05-31T19:57:47.962Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T19:58:29.602 DEBUG:boto:path=/pklxkd-12/ 2023-05-31T19:58:29.603 DEBUG:boto:auth_path=/pklxkd-12/ 2023-05-31T19:58:29.603 DEBUG:boto:path=/pklxkd-12/?prefix=obj-a2 2023-05-31T19:58:29.603 DEBUG:boto:auth_path=/pklxkd-12/?prefix=obj-a2 2023-05-31T19:58:29.604 DEBUG:boto:Method: GET 2023-05-31T19:58:29.604 DEBUG:boto:Path: /pklxkd-12/?prefix=obj-a2 2023-05-31T19:58:29.605 DEBUG:boto:Data: 2023-05-31T19:58:29.605 DEBUG:boto:Headers: {} 2023-05-31T19:58:29.605 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T19:58:29.606 DEBUG:boto:Port: 8000 2023-05-31T19:58:29.606 DEBUG:boto:Params: {} 2023-05-31T19:58:29.606 DEBUG:boto:Token: None 2023-05-31T19:58:29.607 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 19:58:29 GMT /pklxkd-12/ 2023-05-31T19:58:29.607 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:V8XeFrflDLZUWfbnuWB7iCX+rEs= 2023-05-31T19:58:29.608 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 19:58:29 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:V8XeFrflDLZUWfbnuWB7iCX+rEs=', 'Content-Length': '0'} 2023-05-31T19:58:29.610 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000cc5769bacc62617f-006477a6e5-110e-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 19:58:29 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T19:58:29.610 DEBUG:boto:b'pklxkd-12obj-a21000falseobj-a22023-05-31T19:57:47.962Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T19:58:29.611 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=pklxkd-12 zones={a2, a1} 2023-05-31T19:58:29.611 DEBUG:boto:path=/pklxkd-11/ 2023-05-31T19:58:29.612 DEBUG:boto:auth_path=/pklxkd-11/ 2023-05-31T19:58:29.612 DEBUG:boto:Method: HEAD 2023-05-31T19:58:29.612 DEBUG:boto:Path: /pklxkd-11/ 2023-05-31T19:58:29.613 DEBUG:boto:Data: 2023-05-31T19:58:29.613 DEBUG:boto:Headers: {} 2023-05-31T19:58:29.613 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T19:58:29.614 DEBUG:boto:Port: 8000 2023-05-31T19:58:29.614 DEBUG:boto:Params: {} 2023-05-31T19:58:29.614 DEBUG:boto:Token: None 2023-05-31T19:58:29.615 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 19:58:29 GMT /pklxkd-11/ 2023-05-31T19:58:29.615 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:DAEpCm1sosNgn1c6d5H8Upz6P2A= 2023-05-31T19:58:29.616 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 19:58:29 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:DAEpCm1sosNgn1c6d5H8Upz6P2A=', 'Content-Length': '0'} 2023-05-31T19:58:29.617 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '3'), ('X-RGW-Bytes-Used', '16'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000ab903d2666ff1dcc-006477a6e5-110e-a1'), ('Content-Length', '0'), ('Date', 'Wed, 31 May 2023 19:58:29 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T19:58:29.618 DEBUG:boto:path=/pklxkd-11/ 2023-05-31T19:58:29.618 DEBUG:boto:auth_path=/pklxkd-11/ 2023-05-31T19:58:29.619 DEBUG:boto:path=/pklxkd-11/?versions&max-keys=999&prefix=obj-a1 2023-05-31T19:58:29.619 DEBUG:boto:auth_path=/pklxkd-11/?versions&max-keys=999&prefix=obj-a1 2023-05-31T19:58:29.619 DEBUG:boto:Method: GET 2023-05-31T19:58:29.620 DEBUG:boto:Path: /pklxkd-11/?versions&max-keys=999&prefix=obj-a1 2023-05-31T19:58:29.620 DEBUG:boto:Data: 2023-05-31T19:58:29.620 DEBUG:boto:Headers: {} 2023-05-31T19:58:29.621 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T19:58:29.621 DEBUG:boto:Port: 8000 2023-05-31T19:58:29.621 DEBUG:boto:Params: {} 2023-05-31T19:58:29.622 DEBUG:boto:Token: None 2023-05-31T19:58:29.622 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 19:58:29 GMT /pklxkd-11/?versions 2023-05-31T19:58:29.623 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:1QX8kvHsplO/Fqm0MCpX+sImj5U= 2023-05-31T19:58:29.623 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 19:58:29 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:1QX8kvHsplO/Fqm0MCpX+sImj5U=', 'Content-Length': '0'} 2023-05-31T19:58:29.625 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000007945dbfbcdc54f7b-006477a6e5-110e-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 19:58:29 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T19:58:29.625 DEBUG:boto:b'pklxkd-11obj-a1999falseobj-a1L7ViqSXBtkdfIYltLrLN1BhDNjOq8xztrue2023-05-31T19:57:47.555Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T19:58:29.626 DEBUG:boto:path=/pklxkd-11/obj-a1 2023-05-31T19:58:29.626 DEBUG:boto:auth_path=/pklxkd-11/obj-a1 2023-05-31T19:58:29.627 DEBUG:boto:path=/pklxkd-11/obj-a1?acl 2023-05-31T19:58:29.627 DEBUG:boto:auth_path=/pklxkd-11/obj-a1?acl 2023-05-31T19:58:29.627 DEBUG:boto:Method: PUT 2023-05-31T19:58:29.628 DEBUG:boto:Path: /pklxkd-11/obj-a1?acl 2023-05-31T19:58:29.628 DEBUG:boto:Data: 2023-05-31T19:58:29.628 DEBUG:boto:Headers: {'x-amz-acl': 'public-read'} 2023-05-31T19:58:29.629 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T19:58:29.629 DEBUG:boto:Port: 8000 2023-05-31T19:58:29.629 DEBUG:boto:Params: {} 2023-05-31T19:58:29.630 DEBUG:boto:Token: None 2023-05-31T19:58:29.630 DEBUG:boto:StringToSign: PUT Wed, 31 May 2023 19:58:29 GMT x-amz-acl:public-read /pklxkd-11/obj-a1?acl 2023-05-31T19:58:29.630 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:1hOQFNmfMcQhzvKK2NYEWSlxJI4= 2023-05-31T19:58:29.631 DEBUG:boto:Final headers: {'x-amz-acl': 'public-read', 'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 19:58:29 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:1hOQFNmfMcQhzvKK2NYEWSlxJI4=', 'Content-Length': '0'} 2023-05-31T19:58:29.635 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx000007dc6321bc6f09959-006477a6e5-110e-a1'), ('Content-Type', 'application/xml'), ('Content-Length', '0'), ('Date', 'Wed, 31 May 2023 19:58:29 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T19:58:29.636 DEBUG:boto:path=/pklxkd-11/ 2023-05-31T19:58:29.636 DEBUG:boto:auth_path=/pklxkd-11/ 2023-05-31T19:58:29.636 DEBUG:boto:Method: HEAD 2023-05-31T19:58:29.637 DEBUG:boto:Path: /pklxkd-11/ 2023-05-31T19:58:29.637 DEBUG:boto:Data: 2023-05-31T19:58:29.637 DEBUG:boto:Headers: {} 2023-05-31T19:58:29.638 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T19:58:29.638 DEBUG:boto:Port: 8001 2023-05-31T19:58:29.638 DEBUG:boto:Params: {} 2023-05-31T19:58:29.639 DEBUG:boto:Token: None 2023-05-31T19:58:29.639 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 19:58:29 GMT /pklxkd-11/ 2023-05-31T19:58:29.639 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:DAEpCm1sosNgn1c6d5H8Upz6P2A= 2023-05-31T19:58:29.640 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 19:58:29 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:DAEpCm1sosNgn1c6d5H8Upz6P2A=', 'Content-Length': '0'} 2023-05-31T19:58:29.642 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '3'), ('X-RGW-Bytes-Used', '16'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000327ac23be8e1906b-006477a6e5-1112-a2'), ('Content-Length', '0'), ('Date', 'Wed, 31 May 2023 19:58:29 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T19:58:29.642 DEBUG:boto:path=/pklxkd-11/ 2023-05-31T19:58:29.643 DEBUG:boto:auth_path=/pklxkd-11/ 2023-05-31T19:58:29.643 DEBUG:boto:path=/pklxkd-11/?versions&max-keys=999&prefix=obj-a2 2023-05-31T19:58:29.643 DEBUG:boto:auth_path=/pklxkd-11/?versions&max-keys=999&prefix=obj-a2 2023-05-31T19:58:29.644 DEBUG:boto:Method: GET 2023-05-31T19:58:29.644 DEBUG:boto:Path: /pklxkd-11/?versions&max-keys=999&prefix=obj-a2 2023-05-31T19:58:29.644 DEBUG:boto:Data: 2023-05-31T19:58:29.645 DEBUG:boto:Headers: {} 2023-05-31T19:58:29.645 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T19:58:29.645 DEBUG:boto:Port: 8001 2023-05-31T19:58:29.646 DEBUG:boto:Params: {} 2023-05-31T19:58:29.646 DEBUG:boto:Token: None 2023-05-31T19:58:29.647 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 19:58:29 GMT /pklxkd-11/?versions 2023-05-31T19:58:29.647 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:1QX8kvHsplO/Fqm0MCpX+sImj5U= 2023-05-31T19:58:29.647 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 19:58:29 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:1QX8kvHsplO/Fqm0MCpX+sImj5U=', 'Content-Length': '0'} 2023-05-31T19:58:29.649 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000008d8e9240e00fe688-006477a6e5-1112-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 19:58:29 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T19:58:29.650 DEBUG:boto:b'pklxkd-11obj-a2999falseobj-a2d.2JtI5ycFHjEYrzZkvtbFVUQjr5kaLtrue2023-05-31T19:57:47.683Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T19:58:29.651 DEBUG:boto:path=/pklxkd-11/obj-a2 2023-05-31T19:58:29.651 DEBUG:boto:auth_path=/pklxkd-11/obj-a2 2023-05-31T19:58:29.651 DEBUG:boto:path=/pklxkd-11/obj-a2?acl 2023-05-31T19:58:29.652 DEBUG:boto:auth_path=/pklxkd-11/obj-a2?acl 2023-05-31T19:58:29.652 DEBUG:boto:Method: PUT 2023-05-31T19:58:29.652 DEBUG:boto:Path: /pklxkd-11/obj-a2?acl 2023-05-31T19:58:29.653 DEBUG:boto:Data: 2023-05-31T19:58:29.653 DEBUG:boto:Headers: {'x-amz-acl': 'public-read'} 2023-05-31T19:58:29.653 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T19:58:29.654 DEBUG:boto:Port: 8001 2023-05-31T19:58:29.654 DEBUG:boto:Params: {} 2023-05-31T19:58:29.654 DEBUG:boto:Token: None 2023-05-31T19:58:29.655 DEBUG:boto:StringToSign: PUT Wed, 31 May 2023 19:58:29 GMT x-amz-acl:public-read /pklxkd-11/obj-a2?acl 2023-05-31T19:58:29.655 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:7xsfkBaHgHUWnDxVMih6sO/P6UQ= 2023-05-31T19:58:29.655 DEBUG:boto:Final headers: {'x-amz-acl': 'public-read', 'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 19:58:29 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:7xsfkBaHgHUWnDxVMih6sO/P6UQ=', 'Content-Length': '0'} 2023-05-31T19:58:29.659 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx000006cc8836a8d37341c-006477a6e5-1112-a2'), ('Content-Type', 'application/xml'), ('Content-Length', '0'), ('Date', 'Wed, 31 May 2023 19:58:29 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T19:58:29.660 DEBUG:boto:path=/pklxkd-12/ 2023-05-31T19:58:29.660 DEBUG:boto:auth_path=/pklxkd-12/ 2023-05-31T19:58:29.660 DEBUG:boto:Method: HEAD 2023-05-31T19:58:29.661 DEBUG:boto:Path: /pklxkd-12/ 2023-05-31T19:58:29.661 DEBUG:boto:Data: 2023-05-31T19:58:29.661 DEBUG:boto:Headers: {} 2023-05-31T19:58:29.662 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T19:58:29.662 DEBUG:boto:Port: 8000 2023-05-31T19:58:29.662 DEBUG:boto:Params: {} 2023-05-31T19:58:29.663 DEBUG:boto:Token: None 2023-05-31T19:58:29.663 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 19:58:29 GMT /pklxkd-12/ 2023-05-31T19:58:29.664 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:caPUtJ6nzFp2mIPlm10+0Of6oQE= 2023-05-31T19:58:29.664 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 19:58:29 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:caPUtJ6nzFp2mIPlm10+0Of6oQE=', 'Content-Length': '0'} 2023-05-31T19:58:29.666 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '3'), ('X-RGW-Bytes-Used', '16'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000738f0dab1b5a3334-006477a6e5-110e-a1'), ('Content-Length', '0'), ('Date', 'Wed, 31 May 2023 19:58:29 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T19:58:29.666 DEBUG:boto:path=/pklxkd-12/ 2023-05-31T19:58:29.667 DEBUG:boto:auth_path=/pklxkd-12/ 2023-05-31T19:58:29.667 DEBUG:boto:path=/pklxkd-12/?versions&max-keys=999&prefix=obj-a1 2023-05-31T19:58:29.667 DEBUG:boto:auth_path=/pklxkd-12/?versions&max-keys=999&prefix=obj-a1 2023-05-31T19:58:29.668 DEBUG:boto:Method: GET 2023-05-31T19:58:29.668 DEBUG:boto:Path: /pklxkd-12/?versions&max-keys=999&prefix=obj-a1 2023-05-31T19:58:29.668 DEBUG:boto:Data: 2023-05-31T19:58:29.669 DEBUG:boto:Headers: {} 2023-05-31T19:58:29.669 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T19:58:29.669 DEBUG:boto:Port: 8000 2023-05-31T19:58:29.670 DEBUG:boto:Params: {} 2023-05-31T19:58:29.670 DEBUG:boto:Token: None 2023-05-31T19:58:29.670 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 19:58:29 GMT /pklxkd-12/?versions 2023-05-31T19:58:29.671 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:CVVAz/PYvezLubxbq2d8CWPZWbA= 2023-05-31T19:58:29.671 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 19:58:29 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:CVVAz/PYvezLubxbq2d8CWPZWbA=', 'Content-Length': '0'} 2023-05-31T19:58:29.673 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000053acd2079f6fe917-006477a6e5-110e-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 19:58:29 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T19:58:29.674 DEBUG:boto:b'pklxkd-12obj-a1999falseobj-a1G1y4LrpL0rJndINHI52T89ADrfTweD3true2023-05-31T19:57:47.837Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T19:58:29.675 DEBUG:boto:path=/pklxkd-12/obj-a1 2023-05-31T19:58:29.675 DEBUG:boto:auth_path=/pklxkd-12/obj-a1 2023-05-31T19:58:29.675 DEBUG:boto:path=/pklxkd-12/obj-a1?acl 2023-05-31T19:58:29.676 DEBUG:boto:auth_path=/pklxkd-12/obj-a1?acl 2023-05-31T19:58:29.676 DEBUG:boto:Method: PUT 2023-05-31T19:58:29.676 DEBUG:boto:Path: /pklxkd-12/obj-a1?acl 2023-05-31T19:58:29.677 DEBUG:boto:Data: 2023-05-31T19:58:29.677 DEBUG:boto:Headers: {'x-amz-acl': 'public-read'} 2023-05-31T19:58:29.677 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T19:58:29.678 DEBUG:boto:Port: 8000 2023-05-31T19:58:29.678 DEBUG:boto:Params: {} 2023-05-31T19:58:29.678 DEBUG:boto:Token: None 2023-05-31T19:58:29.679 DEBUG:boto:StringToSign: PUT Wed, 31 May 2023 19:58:29 GMT x-amz-acl:public-read /pklxkd-12/obj-a1?acl 2023-05-31T19:58:29.679 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:/zLEGFz+zlBSsHDAyVQzeFDDKnQ= 2023-05-31T19:58:29.679 DEBUG:boto:Final headers: {'x-amz-acl': 'public-read', 'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 19:58:29 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:/zLEGFz+zlBSsHDAyVQzeFDDKnQ=', 'Content-Length': '0'} 2023-05-31T19:58:29.683 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000052a910cc6879a0f-006477a6e5-110e-a1'), ('Content-Type', 'application/xml'), ('Content-Length', '0'), ('Date', 'Wed, 31 May 2023 19:58:29 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T19:58:29.684 DEBUG:boto:path=/pklxkd-12/ 2023-05-31T19:58:29.684 DEBUG:boto:auth_path=/pklxkd-12/ 2023-05-31T19:58:29.685 DEBUG:boto:Method: HEAD 2023-05-31T19:58:29.685 DEBUG:boto:Path: /pklxkd-12/ 2023-05-31T19:58:29.685 DEBUG:boto:Data: 2023-05-31T19:58:29.686 DEBUG:boto:Headers: {} 2023-05-31T19:58:29.686 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T19:58:29.686 DEBUG:boto:Port: 8001 2023-05-31T19:58:29.687 DEBUG:boto:Params: {} 2023-05-31T19:58:29.687 DEBUG:boto:Token: None 2023-05-31T19:58:29.687 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 19:58:29 GMT /pklxkd-12/ 2023-05-31T19:58:29.688 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:caPUtJ6nzFp2mIPlm10+0Of6oQE= 2023-05-31T19:58:29.688 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 19:58:29 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:caPUtJ6nzFp2mIPlm10+0Of6oQE=', 'Content-Length': '0'} 2023-05-31T19:58:29.690 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '3'), ('X-RGW-Bytes-Used', '16'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000005a144452e8d39bfc-006477a6e5-1112-a2'), ('Content-Length', '0'), ('Date', 'Wed, 31 May 2023 19:58:29 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T19:58:29.691 DEBUG:boto:path=/pklxkd-12/ 2023-05-31T19:58:29.691 DEBUG:boto:auth_path=/pklxkd-12/ 2023-05-31T19:58:29.692 DEBUG:boto:path=/pklxkd-12/?versions&max-keys=999&prefix=obj-a2 2023-05-31T19:58:29.692 DEBUG:boto:auth_path=/pklxkd-12/?versions&max-keys=999&prefix=obj-a2 2023-05-31T19:58:29.692 DEBUG:boto:Method: GET 2023-05-31T19:58:29.693 DEBUG:boto:Path: /pklxkd-12/?versions&max-keys=999&prefix=obj-a2 2023-05-31T19:58:29.693 DEBUG:boto:Data: 2023-05-31T19:58:29.693 DEBUG:boto:Headers: {} 2023-05-31T19:58:29.694 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T19:58:29.694 DEBUG:boto:Port: 8001 2023-05-31T19:58:29.694 DEBUG:boto:Params: {} 2023-05-31T19:58:29.695 DEBUG:boto:Token: None 2023-05-31T19:58:29.695 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 19:58:29 GMT /pklxkd-12/?versions 2023-05-31T19:58:29.695 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:CVVAz/PYvezLubxbq2d8CWPZWbA= 2023-05-31T19:58:29.696 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 19:58:29 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:CVVAz/PYvezLubxbq2d8CWPZWbA=', 'Content-Length': '0'} 2023-05-31T19:58:29.698 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d37613110be69f4e-006477a6e5-1112-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 19:58:29 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T19:58:29.698 DEBUG:boto:b'pklxkd-12obj-a2999falseobj-a2qiyfRuud8o74hAGpczwf4-FuN5p5887true2023-05-31T19:57:47.962Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T19:58:29.699 DEBUG:boto:path=/pklxkd-12/obj-a2 2023-05-31T19:58:29.699 DEBUG:boto:auth_path=/pklxkd-12/obj-a2 2023-05-31T19:58:29.700 DEBUG:boto:path=/pklxkd-12/obj-a2?acl 2023-05-31T19:58:29.700 DEBUG:boto:auth_path=/pklxkd-12/obj-a2?acl 2023-05-31T19:58:29.700 DEBUG:boto:Method: PUT 2023-05-31T19:58:29.701 DEBUG:boto:Path: /pklxkd-12/obj-a2?acl 2023-05-31T19:58:29.701 DEBUG:boto:Data: 2023-05-31T19:58:29.701 DEBUG:boto:Headers: {'x-amz-acl': 'public-read'} 2023-05-31T19:58:29.702 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T19:58:29.702 DEBUG:boto:Port: 8001 2023-05-31T19:58:29.702 DEBUG:boto:Params: {} 2023-05-31T19:58:29.703 DEBUG:boto:Token: None 2023-05-31T19:58:29.703 DEBUG:boto:StringToSign: PUT Wed, 31 May 2023 19:58:29 GMT x-amz-acl:public-read /pklxkd-12/obj-a2?acl 2023-05-31T19:58:29.703 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:UdKPywgmx6hLrJipoSg6w+cSkL8= 2023-05-31T19:58:29.704 DEBUG:boto:Final headers: {'x-amz-acl': 'public-read', 'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 19:58:29 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:UdKPywgmx6hLrJipoSg6w+cSkL8=', 'Content-Length': '0'} 2023-05-31T19:58:29.709 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000248621e0b21c5527-006477a6e5-1112-a2'), ('Content-Type', 'application/xml'), ('Content-Length', '0'), ('Date', 'Wed, 31 May 2023 19:58:29 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T19:58:29.709 DEBUG:rgw_multi.tests:bucket checkpoint: source=a1 target=a2 bucket=pklxkd-11 2023-05-31T19:58:29.710 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'pklxkd-11', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-05-31T19:58:29.710 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'pklxkd-11', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-05-31T19:58:29.710 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket pklxkd-11 --source-zone a1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2023-05-31T19:58:29.767 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2023-05-31T19:58:29.767 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2023-05-31T19:58:29.803 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T19:58:29.798+0000 7fb0a3963780 1 waiting for incremental sync to catch up: 2023-05-31T19:58:29.803 INFO:teuthology.orchestra.run.smithi099.stderr: local status: [, , , , , 00000000003.11.10, , , 00000000002.4.11, 00000000010.26.6, ] 2023-05-31T19:58:29.803 INFO:teuthology.orchestra.run.smithi099.stderr: remote markers: [, , , , , 00000000003.11.10, , , 00000000002.4.11, 00000000011.29.6, ] 2023-05-31T19:58:34.804 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T19:58:34.799+0000 7fb0a3963780 1 waiting for incremental sync to catch up: 2023-05-31T19:58:34.805 INFO:teuthology.orchestra.run.smithi099.stderr: local status: [, , , , , 00000000003.11.10, , , 00000000002.4.11, 00000000010.26.6, ] 2023-05-31T19:58:34.805 INFO:teuthology.orchestra.run.smithi099.stderr: remote markers: [, , , , , 00000000003.11.10, , , 00000000002.4.11, 00000000011.29.6, ] 2023-05-31T19:58:39.805 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T19:58:39.799+0000 7fb0a3963780 1 waiting for incremental sync to catch up: 2023-05-31T19:58:39.806 INFO:teuthology.orchestra.run.smithi099.stderr: local status: [, , , , , 00000000003.11.10, , , 00000000002.4.11, 00000000010.26.6, ] 2023-05-31T19:58:39.806 INFO:teuthology.orchestra.run.smithi099.stderr: remote markers: [, , , , , 00000000003.11.10, , , 00000000002.4.11, 00000000011.29.6, ] 2023-05-31T19:58:44.807 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T19:58:44.801+0000 7fb0a3963780 1 waiting for incremental sync to catch up: 2023-05-31T19:58:44.808 INFO:teuthology.orchestra.run.smithi099.stderr: local status: [, , , , , 00000000003.11.10, , , 00000000002.4.11, 00000000010.26.6, ] 2023-05-31T19:58:44.809 INFO:teuthology.orchestra.run.smithi099.stderr: remote markers: [, , , , , 00000000003.11.10, , , 00000000002.4.11, 00000000011.29.6, ] 2023-05-31T19:58:49.808 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T19:58:49.802+0000 7fb0a3963780 1 waiting for incremental sync to catch up: 2023-05-31T19:58:49.809 INFO:teuthology.orchestra.run.smithi099.stderr: local status: [, , , , , 00000000003.11.10, , , 00000000002.4.11, 00000000010.26.6, ] 2023-05-31T19:58:49.809 INFO:teuthology.orchestra.run.smithi099.stderr: remote markers: [, , , , , 00000000003.11.10, , , 00000000002.4.11, 00000000011.29.6, ] 2023-05-31T19:58:54.810 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T19:58:54.804+0000 7fb0a3963780 1 waiting for incremental sync to catch up: 2023-05-31T19:58:54.810 INFO:teuthology.orchestra.run.smithi099.stderr: local status: [, , , , , 00000000003.11.10, , , 00000000002.4.11, 00000000010.26.6, ] 2023-05-31T19:58:54.810 INFO:teuthology.orchestra.run.smithi099.stderr: remote markers: [, , , , , 00000000003.11.10, , , 00000000002.4.11, 00000000011.29.6, ] 2023-05-31T19:58:59.811 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T19:58:59.806+0000 7fb0a3963780 1 waiting for incremental sync to catch up: 2023-05-31T19:58:59.811 INFO:teuthology.orchestra.run.smithi099.stderr: local status: [, , , , , 00000000003.11.10, , , 00000000002.4.11, 00000000010.26.6, ] 2023-05-31T19:58:59.812 INFO:teuthology.orchestra.run.smithi099.stderr: remote markers: [, , , , , 00000000003.11.10, , , 00000000002.4.11, 00000000011.29.6, ] 2023-05-31T19:59:04.812 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T19:59:04.807+0000 7fb0a3963780 1 waiting for incremental sync to catch up: 2023-05-31T19:59:04.813 INFO:teuthology.orchestra.run.smithi099.stderr: local status: [, , , , , 00000000003.11.10, , , 00000000002.4.11, 00000000010.26.6, ] 2023-05-31T19:59:04.813 INFO:teuthology.orchestra.run.smithi099.stderr: remote markers: [, , , , , 00000000003.11.10, , , 00000000002.4.11, 00000000011.29.6, ] 2023-05-31T19:59:09.814 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T19:59:09.809+0000 7fb0a3963780 1 bucket sync caught up with source: 2023-05-31T19:59:09.814 INFO:teuthology.orchestra.run.smithi099.stderr: local status: [, , , , , 00000000003.11.10, , , 00000000003.7.6, 00000000011.29.6, ] 2023-05-31T19:59:09.814 INFO:teuthology.orchestra.run.smithi099.stderr: remote markers: [, , , , , 00000000003.11.10, , , 00000000002.4.11, 00000000011.29.6, ] 2023-05-31T19:59:09.814 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T19:59:09.809+0000 7fb0a3963780 0 bucket checkpoint complete 2023-05-31T19:59:09.831 DEBUG:rgw_multi.tests:bucket checkpoint: source=a2 target=a1 bucket=pklxkd-11 2023-05-31T19:59:09.832 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'pklxkd-11', '--source-zone', 'a2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-05-31T19:59:09.833 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'pklxkd-11', '--source-zone', 'a2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-05-31T19:59:09.833 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket pklxkd-11 --source-zone a2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2023-05-31T19:59:09.897 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2023-05-31T19:59:09.898 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2023-05-31T19:59:09.943 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T19:59:09.938+0000 7f6f7de2d780 1 waiting for incremental sync to catch up: 2023-05-31T19:59:09.943 INFO:teuthology.orchestra.run.smithi099.stderr: local status: [, , , , , 00000000002.27.11, , , 00000000011.30.6, 00000000002.26.11, ] 2023-05-31T19:59:09.943 INFO:teuthology.orchestra.run.smithi099.stderr: remote markers: [, , , , , 00000000002.27.11, , , 00000000011.30.6, 00000000003.29.6, ] 2023-05-31T19:59:14.944 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T19:59:14.939+0000 7f6f7de2d780 1 waiting for incremental sync to catch up: 2023-05-31T19:59:14.945 INFO:teuthology.orchestra.run.smithi099.stderr: local status: [, , , , , 00000000002.27.11, , , 00000000011.30.6, 00000000002.26.11, ] 2023-05-31T19:59:14.945 INFO:teuthology.orchestra.run.smithi099.stderr: remote markers: [, , , , , 00000000002.27.11, , , 00000000011.30.6, 00000000003.29.6, ] 2023-05-31T19:59:19.945 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T19:59:19.940+0000 7f6f7de2d780 1 waiting for incremental sync to catch up: 2023-05-31T19:59:19.946 INFO:teuthology.orchestra.run.smithi099.stderr: local status: [, , , , , 00000000002.27.11, , , 00000000011.30.6, 00000000002.26.11, ] 2023-05-31T19:59:19.946 INFO:teuthology.orchestra.run.smithi099.stderr: remote markers: [, , , , , 00000000002.27.11, , , 00000000011.30.6, 00000000003.29.6, ] 2023-05-31T19:59:24.947 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T19:59:24.941+0000 7f6f7de2d780 1 waiting for incremental sync to catch up: 2023-05-31T19:59:24.947 INFO:teuthology.orchestra.run.smithi099.stderr: local status: [, , , , , 00000000002.27.11, , , 00000000011.30.6, 00000000002.26.11, ] 2023-05-31T19:59:24.947 INFO:teuthology.orchestra.run.smithi099.stderr: remote markers: [, , , , , 00000000002.27.11, , , 00000000011.30.6, 00000000003.29.6, ] 2023-05-31T19:59:29.948 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T19:59:29.942+0000 7f6f7de2d780 1 bucket sync caught up with source: 2023-05-31T19:59:29.949 INFO:teuthology.orchestra.run.smithi099.stderr: local status: [, , , , , 00000000002.27.11, , , 00000000011.30.6, 00000000003.29.6, ] 2023-05-31T19:59:29.949 INFO:teuthology.orchestra.run.smithi099.stderr: remote markers: [, , , , , 00000000002.27.11, , , 00000000011.30.6, 00000000003.29.6, ] 2023-05-31T19:59:29.949 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T19:59:29.942+0000 7f6f7de2d780 0 bucket checkpoint complete 2023-05-31T19:59:29.967 INFO:tasks.rgw_multi.zone_rados:comparing bucket=pklxkd-11 zones={a2, a1} 2023-05-31T19:59:29.968 DEBUG:boto:path=/pklxkd-11/ 2023-05-31T19:59:29.969 DEBUG:boto:auth_path=/pklxkd-11/ 2023-05-31T19:59:29.970 DEBUG:boto:Method: HEAD 2023-05-31T19:59:29.970 DEBUG:boto:Path: /pklxkd-11/ 2023-05-31T19:59:29.971 DEBUG:boto:Data: 2023-05-31T19:59:29.971 DEBUG:boto:Headers: {} 2023-05-31T19:59:29.971 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T19:59:29.972 DEBUG:boto:Port: 8001 2023-05-31T19:59:29.972 DEBUG:boto:Params: {} 2023-05-31T19:59:29.973 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8001} 2023-05-31T19:59:29.973 DEBUG:boto:Token: None 2023-05-31T19:59:29.973 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 19:59:29 GMT /pklxkd-11/ 2023-05-31T19:59:29.974 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:k5Kcw7KDEugr3rQfCCsvJJxhKng= 2023-05-31T19:59:29.975 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 19:59:29 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:k5Kcw7KDEugr3rQfCCsvJJxhKng=', 'Content-Length': '0'} 2023-05-31T19:59:29.978 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '3'), ('X-RGW-Bytes-Used', '16'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000b89ddb0fa4bd4d2f-006477a721-1112-a2'), ('Content-Length', '0'), ('Date', 'Wed, 31 May 2023 19:59:29 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T19:59:29.979 DEBUG:boto:path=/pklxkd-11/ 2023-05-31T19:59:29.979 DEBUG:boto:auth_path=/pklxkd-11/ 2023-05-31T19:59:29.980 DEBUG:boto:Method: HEAD 2023-05-31T19:59:29.980 DEBUG:boto:Path: /pklxkd-11/ 2023-05-31T19:59:29.980 DEBUG:boto:Data: 2023-05-31T19:59:29.981 DEBUG:boto:Headers: {} 2023-05-31T19:59:29.981 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T19:59:29.982 DEBUG:boto:Port: 8000 2023-05-31T19:59:29.982 DEBUG:boto:Params: {} 2023-05-31T19:59:29.983 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2023-05-31T19:59:29.983 DEBUG:boto:Token: None 2023-05-31T19:59:29.984 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 19:59:29 GMT /pklxkd-11/ 2023-05-31T19:59:29.984 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:k5Kcw7KDEugr3rQfCCsvJJxhKng= 2023-05-31T19:59:29.985 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 19:59:29 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:k5Kcw7KDEugr3rQfCCsvJJxhKng=', 'Content-Length': '0'} 2023-05-31T19:59:29.988 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '3'), ('X-RGW-Bytes-Used', '16'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000d528228a274a0505-006477a721-110e-a1'), ('Content-Length', '0'), ('Date', 'Wed, 31 May 2023 19:59:29 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T19:59:29.988 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2023-05-31T19:59:29.989 DEBUG:boto:path=/pklxkd-11/ 2023-05-31T19:59:29.989 DEBUG:boto:auth_path=/pklxkd-11/ 2023-05-31T19:59:29.990 DEBUG:boto:path=/pklxkd-11/?versions&max-keys=999 2023-05-31T19:59:29.990 DEBUG:boto:auth_path=/pklxkd-11/?versions&max-keys=999 2023-05-31T19:59:29.991 DEBUG:boto:Method: GET 2023-05-31T19:59:29.991 DEBUG:boto:Path: /pklxkd-11/?versions&max-keys=999 2023-05-31T19:59:29.992 DEBUG:boto:Data: 2023-05-31T19:59:29.993 DEBUG:boto:Headers: {} 2023-05-31T19:59:29.993 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T19:59:29.994 DEBUG:boto:Port: 8001 2023-05-31T19:59:29.995 DEBUG:boto:Params: {} 2023-05-31T19:59:29.995 DEBUG:boto:Token: None 2023-05-31T19:59:29.996 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 19:59:29 GMT /pklxkd-11/?versions 2023-05-31T19:59:29.996 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:ZxFNb3Jp3eA5uXLmCGGOW6gqLdQ= 2023-05-31T19:59:29.997 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 19:59:29 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:ZxFNb3Jp3eA5uXLmCGGOW6gqLdQ=', 'Content-Length': '0'} 2023-05-31T19:59:30.000 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000000bcd38ad6d537868-006477a721-1112-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 19:59:29 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T19:59:30.001 DEBUG:boto:b'pklxkd-11999falsedummypuen5pSFPkhRYSzopBIdJIXBQzttKnutrue2023-05-31T19:57:12.265Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-a1L7ViqSXBtkdfIYltLrLN1BhDNjOq8xztrue2023-05-31T19:57:47.555Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-a2d.2JtI5ycFHjEYrzZkvtbFVUQjr5kaLtrue2023-05-31T19:57:47.683Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T19:59:30.002 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2023-05-31T19:59:30.003 DEBUG:tasks.rgw_multi.zone_rados:o=obj-a1 2023-05-31T19:59:30.004 DEBUG:tasks.rgw_multi.zone_rados:o=obj-a2 2023-05-31T19:59:30.004 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2023-05-31T19:59:30.005 DEBUG:boto:path=/pklxkd-11/ 2023-05-31T19:59:30.006 DEBUG:boto:auth_path=/pklxkd-11/ 2023-05-31T19:59:30.006 DEBUG:boto:path=/pklxkd-11/?versions&max-keys=999 2023-05-31T19:59:30.007 DEBUG:boto:auth_path=/pklxkd-11/?versions&max-keys=999 2023-05-31T19:59:30.008 DEBUG:boto:Method: GET 2023-05-31T19:59:30.009 DEBUG:boto:Path: /pklxkd-11/?versions&max-keys=999 2023-05-31T19:59:30.010 DEBUG:boto:Data: 2023-05-31T19:59:30.011 DEBUG:boto:Headers: {} 2023-05-31T19:59:30.011 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T19:59:30.012 DEBUG:boto:Port: 8000 2023-05-31T19:59:30.012 DEBUG:boto:Params: {} 2023-05-31T19:59:30.013 DEBUG:boto:Token: None 2023-05-31T19:59:30.013 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 19:59:30 GMT /pklxkd-11/?versions 2023-05-31T19:59:30.014 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:H8QoCOBf4MDgrpdBbG4TQrBgAvg= 2023-05-31T19:59:30.014 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 19:59:30 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:H8QoCOBf4MDgrpdBbG4TQrBgAvg=', 'Content-Length': '0'} 2023-05-31T19:59:30.017 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b386262f8a541d90-006477a722-110e-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 19:59:30 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T19:59:30.018 DEBUG:boto:b'pklxkd-11999falsedummypuen5pSFPkhRYSzopBIdJIXBQzttKnutrue2023-05-31T19:57:12.265Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-a1L7ViqSXBtkdfIYltLrLN1BhDNjOq8xztrue2023-05-31T19:57:47.555Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-a2d.2JtI5ycFHjEYrzZkvtbFVUQjr5kaLtrue2023-05-31T19:57:47.683Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T19:59:30.018 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2023-05-31T19:59:30.019 DEBUG:tasks.rgw_multi.zone_rados:o=obj-a1 2023-05-31T19:59:30.019 DEBUG:tasks.rgw_multi.zone_rados:o=obj-a2 2023-05-31T19:59:30.020 DEBUG:boto:path=/pklxkd-11/ 2023-05-31T19:59:30.020 DEBUG:boto:auth_path=/pklxkd-11/ 2023-05-31T19:59:30.021 DEBUG:boto:path=/pklxkd-11/?versions&max-keys=999 2023-05-31T19:59:30.021 DEBUG:boto:auth_path=/pklxkd-11/?versions&max-keys=999 2023-05-31T19:59:30.021 DEBUG:boto:Method: GET 2023-05-31T19:59:30.022 DEBUG:boto:Path: /pklxkd-11/?versions&max-keys=999 2023-05-31T19:59:30.022 DEBUG:boto:Data: 2023-05-31T19:59:30.023 DEBUG:boto:Headers: {} 2023-05-31T19:59:30.023 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T19:59:30.023 DEBUG:boto:Port: 8001 2023-05-31T19:59:30.024 DEBUG:boto:Params: {} 2023-05-31T19:59:30.024 DEBUG:boto:Token: None 2023-05-31T19:59:30.025 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 19:59:30 GMT /pklxkd-11/?versions 2023-05-31T19:59:30.025 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:H8QoCOBf4MDgrpdBbG4TQrBgAvg= 2023-05-31T19:59:30.026 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 19:59:30 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:H8QoCOBf4MDgrpdBbG4TQrBgAvg=', 'Content-Length': '0'} 2023-05-31T19:59:30.028 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000567de643ed3abaf8-006477a722-1112-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 19:59:30 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T19:59:30.029 DEBUG:boto:b'pklxkd-11999falsedummypuen5pSFPkhRYSzopBIdJIXBQzttKnutrue2023-05-31T19:57:12.265Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-a1L7ViqSXBtkdfIYltLrLN1BhDNjOq8xztrue2023-05-31T19:57:47.555Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-a2d.2JtI5ycFHjEYrzZkvtbFVUQjr5kaLtrue2023-05-31T19:57:47.683Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T19:59:30.029 DEBUG:boto:path=/pklxkd-11/ 2023-05-31T19:59:30.030 DEBUG:boto:auth_path=/pklxkd-11/ 2023-05-31T19:59:30.030 DEBUG:boto:path=/pklxkd-11/?versions&max-keys=999 2023-05-31T19:59:30.031 DEBUG:boto:auth_path=/pklxkd-11/?versions&max-keys=999 2023-05-31T19:59:30.031 DEBUG:boto:Method: GET 2023-05-31T19:59:30.032 DEBUG:boto:Path: /pklxkd-11/?versions&max-keys=999 2023-05-31T19:59:30.033 DEBUG:boto:Data: 2023-05-31T19:59:30.033 DEBUG:boto:Headers: {} 2023-05-31T19:59:30.034 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T19:59:30.035 DEBUG:boto:Port: 8000 2023-05-31T19:59:30.035 DEBUG:boto:Params: {} 2023-05-31T19:59:30.037 DEBUG:boto:Token: None 2023-05-31T19:59:30.037 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 19:59:30 GMT /pklxkd-11/?versions 2023-05-31T19:59:30.038 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:H8QoCOBf4MDgrpdBbG4TQrBgAvg= 2023-05-31T19:59:30.039 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 19:59:30 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:H8QoCOBf4MDgrpdBbG4TQrBgAvg=', 'Content-Length': '0'} 2023-05-31T19:59:30.042 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000001471e110af171889-006477a722-110e-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 19:59:30 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T19:59:30.042 DEBUG:boto:b'pklxkd-11999falsedummypuen5pSFPkhRYSzopBIdJIXBQzttKnutrue2023-05-31T19:57:12.265Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-a1L7ViqSXBtkdfIYltLrLN1BhDNjOq8xztrue2023-05-31T19:57:47.555Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-a2d.2JtI5ycFHjEYrzZkvtbFVUQjr5kaLtrue2023-05-31T19:57:47.683Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T19:59:30.043 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2023-05-31T19:59:30.043 DEBUG:boto:path=/pklxkd-11/dummy 2023-05-31T19:59:30.044 DEBUG:boto:auth_path=/pklxkd-11/dummy 2023-05-31T19:59:30.044 DEBUG:boto:path=/pklxkd-11/dummy?versionId=puen5pSFPkhRYSzopBIdJIXBQzttKnu 2023-05-31T19:59:30.045 DEBUG:boto:auth_path=/pklxkd-11/dummy?versionId=puen5pSFPkhRYSzopBIdJIXBQzttKnu 2023-05-31T19:59:30.045 DEBUG:boto:Method: GET 2023-05-31T19:59:30.046 DEBUG:boto:Path: /pklxkd-11/dummy?versionId=puen5pSFPkhRYSzopBIdJIXBQzttKnu 2023-05-31T19:59:30.046 DEBUG:boto:Data: 2023-05-31T19:59:30.046 DEBUG:boto:Headers: {} 2023-05-31T19:59:30.047 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T19:59:30.047 DEBUG:boto:Port: 8001 2023-05-31T19:59:30.048 DEBUG:boto:Params: {} 2023-05-31T19:59:30.048 DEBUG:boto:Token: None 2023-05-31T19:59:30.049 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 19:59:30 GMT /pklxkd-11/dummy?versionId=puen5pSFPkhRYSzopBIdJIXBQzttKnu 2023-05-31T19:59:30.049 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:jOILtfqDYdd7Tw5JmHNnMHV06TM= 2023-05-31T19:59:30.049 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 19:59:30 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:jOILtfqDYdd7Tw5JmHNnMHV06TM=', 'Content-Length': '0'} 2023-05-31T19:59:30.051 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 19:57:12 GMT'), ('x-amz-version-id', 'puen5pSFPkhRYSzopBIdJIXBQzttKnu'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx0000082626b808d7825ed-006477a722-1112-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 19:59:30 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T19:59:30.052 DEBUG:boto:path=/pklxkd-11/dummy 2023-05-31T19:59:30.052 DEBUG:boto:auth_path=/pklxkd-11/dummy 2023-05-31T19:59:30.053 DEBUG:boto:path=/pklxkd-11/dummy?versionId=puen5pSFPkhRYSzopBIdJIXBQzttKnu 2023-05-31T19:59:30.053 DEBUG:boto:auth_path=/pklxkd-11/dummy?versionId=puen5pSFPkhRYSzopBIdJIXBQzttKnu 2023-05-31T19:59:30.054 DEBUG:boto:Method: GET 2023-05-31T19:59:30.054 DEBUG:boto:Path: /pklxkd-11/dummy?versionId=puen5pSFPkhRYSzopBIdJIXBQzttKnu 2023-05-31T19:59:30.054 DEBUG:boto:Data: 2023-05-31T19:59:30.055 DEBUG:boto:Headers: {} 2023-05-31T19:59:30.055 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T19:59:30.056 DEBUG:boto:Port: 8000 2023-05-31T19:59:30.056 DEBUG:boto:Params: {} 2023-05-31T19:59:30.056 DEBUG:boto:Token: None 2023-05-31T19:59:30.057 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 19:59:30 GMT /pklxkd-11/dummy?versionId=puen5pSFPkhRYSzopBIdJIXBQzttKnu 2023-05-31T19:59:30.057 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:jOILtfqDYdd7Tw5JmHNnMHV06TM= 2023-05-31T19:59:30.058 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 19:59:30 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:jOILtfqDYdd7Tw5JmHNnMHV06TM=', 'Content-Length': '0'} 2023-05-31T19:59:30.060 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 19:57:12 GMT'), ('x-amz-version-id', 'puen5pSFPkhRYSzopBIdJIXBQzttKnu'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx0000058937fa594f5f7ca-006477a722-110e-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 19:59:30 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T19:59:30.060 DEBUG:boto:path=/pklxkd-11/dummy 2023-05-31T19:59:30.061 DEBUG:boto:auth_path=/pklxkd-11/dummy 2023-05-31T19:59:30.061 DEBUG:boto:path=/pklxkd-11/dummy?versionId=puen5pSFPkhRYSzopBIdJIXBQzttKnu 2023-05-31T19:59:30.061 DEBUG:boto:auth_path=/pklxkd-11/dummy?versionId=puen5pSFPkhRYSzopBIdJIXBQzttKnu 2023-05-31T19:59:30.062 DEBUG:boto:Method: HEAD 2023-05-31T19:59:30.062 DEBUG:boto:Path: /pklxkd-11/dummy?versionId=puen5pSFPkhRYSzopBIdJIXBQzttKnu 2023-05-31T19:59:30.063 DEBUG:boto:Data: 2023-05-31T19:59:30.063 DEBUG:boto:Headers: {} 2023-05-31T19:59:30.063 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T19:59:30.064 DEBUG:boto:Port: 8001 2023-05-31T19:59:30.064 DEBUG:boto:Params: {} 2023-05-31T19:59:30.064 DEBUG:boto:Token: None 2023-05-31T19:59:30.065 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 19:59:30 GMT /pklxkd-11/dummy?versionId=puen5pSFPkhRYSzopBIdJIXBQzttKnu 2023-05-31T19:59:30.065 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:D3dxEn/FWUOcRXzPaPDGZ8GLPaI= 2023-05-31T19:59:30.066 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 19:59:30 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:D3dxEn/FWUOcRXzPaPDGZ8GLPaI=', 'Content-Length': '0'} 2023-05-31T19:59:30.067 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 19:57:12 GMT'), ('x-amz-version-id', 'puen5pSFPkhRYSzopBIdJIXBQzttKnu'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000affcb1ac2df9216c-006477a722-1112-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 19:59:30 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T19:59:30.069 DEBUG:boto:path=/pklxkd-11/dummy 2023-05-31T19:59:30.070 DEBUG:boto:auth_path=/pklxkd-11/dummy 2023-05-31T19:59:30.071 DEBUG:boto:path=/pklxkd-11/dummy?versionId=puen5pSFPkhRYSzopBIdJIXBQzttKnu 2023-05-31T19:59:30.071 DEBUG:boto:auth_path=/pklxkd-11/dummy?versionId=puen5pSFPkhRYSzopBIdJIXBQzttKnu 2023-05-31T19:59:30.072 DEBUG:boto:Method: HEAD 2023-05-31T19:59:30.073 DEBUG:boto:Path: /pklxkd-11/dummy?versionId=puen5pSFPkhRYSzopBIdJIXBQzttKnu 2023-05-31T19:59:30.074 DEBUG:boto:Data: 2023-05-31T19:59:30.075 DEBUG:boto:Headers: {} 2023-05-31T19:59:30.075 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T19:59:30.076 DEBUG:boto:Port: 8000 2023-05-31T19:59:30.077 DEBUG:boto:Params: {} 2023-05-31T19:59:30.077 DEBUG:boto:Token: None 2023-05-31T19:59:30.078 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 19:59:30 GMT /pklxkd-11/dummy?versionId=puen5pSFPkhRYSzopBIdJIXBQzttKnu 2023-05-31T19:59:30.078 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:D3dxEn/FWUOcRXzPaPDGZ8GLPaI= 2023-05-31T19:59:30.079 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 19:59:30 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:D3dxEn/FWUOcRXzPaPDGZ8GLPaI=', 'Content-Length': '0'} 2023-05-31T19:59:30.080 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 19:57:12 GMT'), ('x-amz-version-id', 'puen5pSFPkhRYSzopBIdJIXBQzttKnu'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000873e5a09801b3983-006477a722-110e-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 19:59:30 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T19:59:30.081 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2023-05-31T19:59:30.082 DEBUG:boto:path=/pklxkd-11/dummy 2023-05-31T19:59:30.082 DEBUG:boto:auth_path=/pklxkd-11/dummy 2023-05-31T19:59:30.083 DEBUG:boto:path=/pklxkd-11/dummy?versionId=puen5pSFPkhRYSzopBIdJIXBQzttKnu 2023-05-31T19:59:30.083 DEBUG:boto:auth_path=/pklxkd-11/dummy?versionId=puen5pSFPkhRYSzopBIdJIXBQzttKnu 2023-05-31T19:59:30.084 DEBUG:boto:Method: GET 2023-05-31T19:59:30.085 DEBUG:boto:Path: /pklxkd-11/dummy?versionId=puen5pSFPkhRYSzopBIdJIXBQzttKnu 2023-05-31T19:59:30.085 DEBUG:boto:Data: 2023-05-31T19:59:30.086 DEBUG:boto:Headers: {} 2023-05-31T19:59:30.087 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T19:59:30.088 DEBUG:boto:Port: 8001 2023-05-31T19:59:30.089 DEBUG:boto:Params: {} 2023-05-31T19:59:30.089 DEBUG:boto:Token: None 2023-05-31T19:59:30.089 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 19:59:30 GMT /pklxkd-11/dummy?versionId=puen5pSFPkhRYSzopBIdJIXBQzttKnu 2023-05-31T19:59:30.090 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:jOILtfqDYdd7Tw5JmHNnMHV06TM= 2023-05-31T19:59:30.090 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 19:59:30 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:jOILtfqDYdd7Tw5JmHNnMHV06TM=', 'Content-Length': '0'} 2023-05-31T19:59:30.092 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 19:57:12 GMT'), ('x-amz-version-id', 'puen5pSFPkhRYSzopBIdJIXBQzttKnu'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000008458d252354f14ff-006477a722-1112-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 19:59:30 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T19:59:30.093 DEBUG:boto:path=/pklxkd-11/dummy 2023-05-31T19:59:30.093 DEBUG:boto:auth_path=/pklxkd-11/dummy 2023-05-31T19:59:30.094 DEBUG:boto:path=/pklxkd-11/dummy?versionId=puen5pSFPkhRYSzopBIdJIXBQzttKnu 2023-05-31T19:59:30.094 DEBUG:boto:auth_path=/pklxkd-11/dummy?versionId=puen5pSFPkhRYSzopBIdJIXBQzttKnu 2023-05-31T19:59:30.094 DEBUG:boto:Method: GET 2023-05-31T19:59:30.095 DEBUG:boto:Path: /pklxkd-11/dummy?versionId=puen5pSFPkhRYSzopBIdJIXBQzttKnu 2023-05-31T19:59:30.095 DEBUG:boto:Data: 2023-05-31T19:59:30.095 DEBUG:boto:Headers: {} 2023-05-31T19:59:30.096 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T19:59:30.096 DEBUG:boto:Port: 8000 2023-05-31T19:59:30.096 DEBUG:boto:Params: {} 2023-05-31T19:59:30.097 DEBUG:boto:Token: None 2023-05-31T19:59:30.097 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 19:59:30 GMT /pklxkd-11/dummy?versionId=puen5pSFPkhRYSzopBIdJIXBQzttKnu 2023-05-31T19:59:30.098 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:jOILtfqDYdd7Tw5JmHNnMHV06TM= 2023-05-31T19:59:30.098 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 19:59:30 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:jOILtfqDYdd7Tw5JmHNnMHV06TM=', 'Content-Length': '0'} 2023-05-31T19:59:30.100 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 19:57:12 GMT'), ('x-amz-version-id', 'puen5pSFPkhRYSzopBIdJIXBQzttKnu'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000b4fa552569f1942a-006477a722-110e-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 19:59:30 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T19:59:30.100 DEBUG:boto:path=/pklxkd-11/ 2023-05-31T19:59:30.101 DEBUG:boto:auth_path=/pklxkd-11/ 2023-05-31T19:59:30.101 DEBUG:boto:path=/pklxkd-11/?prefix=dummy 2023-05-31T19:59:30.102 DEBUG:boto:auth_path=/pklxkd-11/?prefix=dummy 2023-05-31T19:59:30.102 DEBUG:boto:Method: GET 2023-05-31T19:59:30.102 DEBUG:boto:Path: /pklxkd-11/?prefix=dummy 2023-05-31T19:59:30.103 DEBUG:boto:Data: 2023-05-31T19:59:30.103 DEBUG:boto:Headers: {} 2023-05-31T19:59:30.103 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T19:59:30.104 DEBUG:boto:Port: 8001 2023-05-31T19:59:30.104 DEBUG:boto:Params: {} 2023-05-31T19:59:30.105 DEBUG:boto:Token: None 2023-05-31T19:59:30.105 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 19:59:30 GMT /pklxkd-11/ 2023-05-31T19:59:30.105 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:ObxMJzN1CLgWV5APWOP4fGp5x04= 2023-05-31T19:59:30.106 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 19:59:30 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:ObxMJzN1CLgWV5APWOP4fGp5x04=', 'Content-Length': '0'} 2023-05-31T19:59:30.108 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f305d76dbcf15b36-006477a722-1112-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 19:59:30 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T19:59:30.109 DEBUG:boto:b'pklxkd-11dummy1000falsedummy2023-05-31T19:57:12.265Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T19:59:30.110 DEBUG:boto:path=/pklxkd-11/ 2023-05-31T19:59:30.110 DEBUG:boto:auth_path=/pklxkd-11/ 2023-05-31T19:59:30.110 DEBUG:boto:path=/pklxkd-11/?prefix=dummy 2023-05-31T19:59:30.111 DEBUG:boto:auth_path=/pklxkd-11/?prefix=dummy 2023-05-31T19:59:30.111 DEBUG:boto:Method: GET 2023-05-31T19:59:30.111 DEBUG:boto:Path: /pklxkd-11/?prefix=dummy 2023-05-31T19:59:30.112 DEBUG:boto:Data: 2023-05-31T19:59:30.112 DEBUG:boto:Headers: {} 2023-05-31T19:59:30.112 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T19:59:30.113 DEBUG:boto:Port: 8000 2023-05-31T19:59:30.113 DEBUG:boto:Params: {} 2023-05-31T19:59:30.114 DEBUG:boto:Token: None 2023-05-31T19:59:30.114 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 19:59:30 GMT /pklxkd-11/ 2023-05-31T19:59:30.114 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:ObxMJzN1CLgWV5APWOP4fGp5x04= 2023-05-31T19:59:30.115 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 19:59:30 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:ObxMJzN1CLgWV5APWOP4fGp5x04=', 'Content-Length': '0'} 2023-05-31T19:59:30.117 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b59945caed07d524-006477a722-110e-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 19:59:30 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T19:59:30.118 DEBUG:boto:b'pklxkd-11dummy1000falsedummy2023-05-31T19:57:12.265Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T19:59:30.118 DEBUG:boto:path=/pklxkd-11/dummy 2023-05-31T19:59:30.119 DEBUG:boto:auth_path=/pklxkd-11/dummy 2023-05-31T19:59:30.119 DEBUG:boto:Method: HEAD 2023-05-31T19:59:30.120 DEBUG:boto:Path: /pklxkd-11/dummy 2023-05-31T19:59:30.120 DEBUG:boto:Data: 2023-05-31T19:59:30.120 DEBUG:boto:Headers: {} 2023-05-31T19:59:30.121 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T19:59:30.121 DEBUG:boto:Port: 8001 2023-05-31T19:59:30.121 DEBUG:boto:Params: {} 2023-05-31T19:59:30.122 DEBUG:boto:Token: None 2023-05-31T19:59:30.122 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 19:59:30 GMT /pklxkd-11/dummy 2023-05-31T19:59:30.123 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:k2ddKMdczDOv6NIKHJ48zd3Ctbo= 2023-05-31T19:59:30.123 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 19:59:30 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:k2ddKMdczDOv6NIKHJ48zd3Ctbo=', 'Content-Length': '0'} 2023-05-31T19:59:30.126 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 19:57:12 GMT'), ('x-amz-version-id', 'puen5pSFPkhRYSzopBIdJIXBQzttKnu'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000004492cb77708e43fa-006477a722-1112-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 19:59:30 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T19:59:30.126 DEBUG:boto:path=/pklxkd-11/dummy 2023-05-31T19:59:30.127 DEBUG:boto:auth_path=/pklxkd-11/dummy 2023-05-31T19:59:30.127 DEBUG:boto:Method: HEAD 2023-05-31T19:59:30.127 DEBUG:boto:Path: /pklxkd-11/dummy 2023-05-31T19:59:30.128 DEBUG:boto:Data: 2023-05-31T19:59:30.128 DEBUG:boto:Headers: {} 2023-05-31T19:59:30.128 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T19:59:30.129 DEBUG:boto:Port: 8000 2023-05-31T19:59:30.129 DEBUG:boto:Params: {} 2023-05-31T19:59:30.130 DEBUG:boto:Token: None 2023-05-31T19:59:30.130 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 19:59:30 GMT /pklxkd-11/dummy 2023-05-31T19:59:30.130 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:k2ddKMdczDOv6NIKHJ48zd3Ctbo= 2023-05-31T19:59:30.131 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 19:59:30 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:k2ddKMdczDOv6NIKHJ48zd3Ctbo=', 'Content-Length': '0'} 2023-05-31T19:59:30.133 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 19:57:12 GMT'), ('x-amz-version-id', 'puen5pSFPkhRYSzopBIdJIXBQzttKnu'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000fa6fee8cab70eec7-006477a722-110e-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 19:59:30 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T19:59:30.133 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2023-05-31T19:59:30.134 DEBUG:boto:path=/pklxkd-11/dummy 2023-05-31T19:59:30.134 DEBUG:boto:auth_path=/pklxkd-11/dummy 2023-05-31T19:59:30.135 DEBUG:boto:path=/pklxkd-11/dummy?versionId=puen5pSFPkhRYSzopBIdJIXBQzttKnu 2023-05-31T19:59:30.135 DEBUG:boto:auth_path=/pklxkd-11/dummy?versionId=puen5pSFPkhRYSzopBIdJIXBQzttKnu 2023-05-31T19:59:30.135 DEBUG:boto:Method: GET 2023-05-31T19:59:30.136 DEBUG:boto:Path: /pklxkd-11/dummy?versionId=puen5pSFPkhRYSzopBIdJIXBQzttKnu 2023-05-31T19:59:30.136 DEBUG:boto:Data: 2023-05-31T19:59:30.137 DEBUG:boto:Headers: {} 2023-05-31T19:59:30.137 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T19:59:30.137 DEBUG:boto:Port: 8001 2023-05-31T19:59:30.138 DEBUG:boto:Params: {} 2023-05-31T19:59:30.138 DEBUG:boto:Token: None 2023-05-31T19:59:30.139 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 19:59:30 GMT /pklxkd-11/dummy?versionId=puen5pSFPkhRYSzopBIdJIXBQzttKnu 2023-05-31T19:59:30.139 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:jOILtfqDYdd7Tw5JmHNnMHV06TM= 2023-05-31T19:59:30.139 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 19:59:30 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:jOILtfqDYdd7Tw5JmHNnMHV06TM=', 'Content-Length': '0'} 2023-05-31T19:59:30.141 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 19:57:12 GMT'), ('x-amz-version-id', 'puen5pSFPkhRYSzopBIdJIXBQzttKnu'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000009d733cf8fb857805-006477a722-1112-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 19:59:30 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T19:59:30.142 DEBUG:boto:path=/pklxkd-11/dummy 2023-05-31T19:59:30.142 DEBUG:boto:auth_path=/pklxkd-11/dummy 2023-05-31T19:59:30.143 DEBUG:boto:path=/pklxkd-11/dummy?versionId=puen5pSFPkhRYSzopBIdJIXBQzttKnu 2023-05-31T19:59:30.143 DEBUG:boto:auth_path=/pklxkd-11/dummy?versionId=puen5pSFPkhRYSzopBIdJIXBQzttKnu 2023-05-31T19:59:30.144 DEBUG:boto:Method: GET 2023-05-31T19:59:30.144 DEBUG:boto:Path: /pklxkd-11/dummy?versionId=puen5pSFPkhRYSzopBIdJIXBQzttKnu 2023-05-31T19:59:30.144 DEBUG:boto:Data: 2023-05-31T19:59:30.145 DEBUG:boto:Headers: {} 2023-05-31T19:59:30.145 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T19:59:30.145 DEBUG:boto:Port: 8000 2023-05-31T19:59:30.146 DEBUG:boto:Params: {} 2023-05-31T19:59:30.146 DEBUG:boto:Token: None 2023-05-31T19:59:30.147 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 19:59:30 GMT /pklxkd-11/dummy?versionId=puen5pSFPkhRYSzopBIdJIXBQzttKnu 2023-05-31T19:59:30.147 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:jOILtfqDYdd7Tw5JmHNnMHV06TM= 2023-05-31T19:59:30.148 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 19:59:30 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:jOILtfqDYdd7Tw5JmHNnMHV06TM=', 'Content-Length': '0'} 2023-05-31T19:59:30.149 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 19:57:12 GMT'), ('x-amz-version-id', 'puen5pSFPkhRYSzopBIdJIXBQzttKnu'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000bf656f48bb47579d-006477a722-110e-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 19:59:30 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T19:59:30.150 DEBUG:boto:path=/pklxkd-11/ 2023-05-31T19:59:30.150 DEBUG:boto:auth_path=/pklxkd-11/ 2023-05-31T19:59:30.151 DEBUG:boto:path=/pklxkd-11/?prefix=dummy 2023-05-31T19:59:30.151 DEBUG:boto:auth_path=/pklxkd-11/?prefix=dummy 2023-05-31T19:59:30.152 DEBUG:boto:Method: GET 2023-05-31T19:59:30.152 DEBUG:boto:Path: /pklxkd-11/?prefix=dummy 2023-05-31T19:59:30.153 DEBUG:boto:Data: 2023-05-31T19:59:30.153 DEBUG:boto:Headers: {} 2023-05-31T19:59:30.154 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T19:59:30.154 DEBUG:boto:Port: 8001 2023-05-31T19:59:30.154 DEBUG:boto:Params: {} 2023-05-31T19:59:30.155 DEBUG:boto:Token: None 2023-05-31T19:59:30.156 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 19:59:30 GMT /pklxkd-11/ 2023-05-31T19:59:30.157 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:ObxMJzN1CLgWV5APWOP4fGp5x04= 2023-05-31T19:59:30.158 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 19:59:30 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:ObxMJzN1CLgWV5APWOP4fGp5x04=', 'Content-Length': '0'} 2023-05-31T19:59:30.160 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000447ab7d7178d75fc-006477a722-1112-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 19:59:30 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T19:59:30.204 DEBUG:boto:b'pklxkd-11dummy1000falsedummy2023-05-31T19:57:12.265Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T19:59:30.205 DEBUG:boto:path=/pklxkd-11/ 2023-05-31T19:59:30.206 DEBUG:boto:auth_path=/pklxkd-11/ 2023-05-31T19:59:30.206 DEBUG:boto:path=/pklxkd-11/?prefix=dummy 2023-05-31T19:59:30.206 DEBUG:boto:auth_path=/pklxkd-11/?prefix=dummy 2023-05-31T19:59:30.207 DEBUG:boto:Method: GET 2023-05-31T19:59:30.207 DEBUG:boto:Path: /pklxkd-11/?prefix=dummy 2023-05-31T19:59:30.208 DEBUG:boto:Data: 2023-05-31T19:59:30.208 DEBUG:boto:Headers: {} 2023-05-31T19:59:30.208 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T19:59:30.209 DEBUG:boto:Port: 8000 2023-05-31T19:59:30.209 DEBUG:boto:Params: {} 2023-05-31T19:59:30.210 DEBUG:boto:Token: None 2023-05-31T19:59:30.210 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 19:59:30 GMT /pklxkd-11/ 2023-05-31T19:59:30.210 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:ObxMJzN1CLgWV5APWOP4fGp5x04= 2023-05-31T19:59:30.211 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 19:59:30 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:ObxMJzN1CLgWV5APWOP4fGp5x04=', 'Content-Length': '0'} 2023-05-31T19:59:30.213 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000df2b8b6a4af141b0-006477a722-110e-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 19:59:30 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T19:59:30.214 DEBUG:boto:b'pklxkd-11dummy1000falsedummy2023-05-31T19:57:12.265Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T19:59:30.214 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-a1 2023-05-31T19:59:30.215 DEBUG:boto:path=/pklxkd-11/obj-a1 2023-05-31T19:59:30.215 DEBUG:boto:auth_path=/pklxkd-11/obj-a1 2023-05-31T19:59:30.216 DEBUG:boto:path=/pklxkd-11/obj-a1?versionId=L7ViqSXBtkdfIYltLrLN1BhDNjOq8xz 2023-05-31T19:59:30.216 DEBUG:boto:auth_path=/pklxkd-11/obj-a1?versionId=L7ViqSXBtkdfIYltLrLN1BhDNjOq8xz 2023-05-31T19:59:30.216 DEBUG:boto:Method: GET 2023-05-31T19:59:30.217 DEBUG:boto:Path: /pklxkd-11/obj-a1?versionId=L7ViqSXBtkdfIYltLrLN1BhDNjOq8xz 2023-05-31T19:59:30.217 DEBUG:boto:Data: 2023-05-31T19:59:30.217 DEBUG:boto:Headers: {} 2023-05-31T19:59:30.218 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T19:59:30.218 DEBUG:boto:Port: 8001 2023-05-31T19:59:30.218 DEBUG:boto:Params: {} 2023-05-31T19:59:30.219 DEBUG:boto:Token: None 2023-05-31T19:59:30.219 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 19:59:30 GMT /pklxkd-11/obj-a1?versionId=L7ViqSXBtkdfIYltLrLN1BhDNjOq8xz 2023-05-31T19:59:30.220 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:+aeDdN+jdclMW9Iq6k2B1vKvQJM= 2023-05-31T19:59:30.220 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 19:59:30 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:+aeDdN+jdclMW9Iq6k2B1vKvQJM=', 'Content-Length': '0'} 2023-05-31T19:59:30.222 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 19:58:29 GMT'), ('x-amz-version-id', 'L7ViqSXBtkdfIYltLrLN1BhDNjOq8xz'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000009d31980dd12ae04-006477a722-1112-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 19:59:30 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T19:59:30.223 DEBUG:boto:path=/pklxkd-11/obj-a1 2023-05-31T19:59:30.223 DEBUG:boto:auth_path=/pklxkd-11/obj-a1 2023-05-31T19:59:30.224 DEBUG:boto:path=/pklxkd-11/obj-a1?versionId=L7ViqSXBtkdfIYltLrLN1BhDNjOq8xz 2023-05-31T19:59:30.224 DEBUG:boto:auth_path=/pklxkd-11/obj-a1?versionId=L7ViqSXBtkdfIYltLrLN1BhDNjOq8xz 2023-05-31T19:59:30.225 DEBUG:boto:Method: GET 2023-05-31T19:59:30.225 DEBUG:boto:Path: /pklxkd-11/obj-a1?versionId=L7ViqSXBtkdfIYltLrLN1BhDNjOq8xz 2023-05-31T19:59:30.225 DEBUG:boto:Data: 2023-05-31T19:59:30.226 DEBUG:boto:Headers: {} 2023-05-31T19:59:30.226 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T19:59:30.227 DEBUG:boto:Port: 8000 2023-05-31T19:59:30.227 DEBUG:boto:Params: {} 2023-05-31T19:59:30.227 DEBUG:boto:Token: None 2023-05-31T19:59:30.228 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 19:59:30 GMT /pklxkd-11/obj-a1?versionId=L7ViqSXBtkdfIYltLrLN1BhDNjOq8xz 2023-05-31T19:59:30.228 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:+aeDdN+jdclMW9Iq6k2B1vKvQJM= 2023-05-31T19:59:30.229 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 19:59:30 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:+aeDdN+jdclMW9Iq6k2B1vKvQJM=', 'Content-Length': '0'} 2023-05-31T19:59:30.231 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 19:58:29 GMT'), ('x-amz-version-id', 'L7ViqSXBtkdfIYltLrLN1BhDNjOq8xz'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000e78821fc96c1c81d-006477a722-110e-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 19:59:30 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T19:59:30.232 DEBUG:boto:path=/pklxkd-11/obj-a1 2023-05-31T19:59:30.233 DEBUG:boto:auth_path=/pklxkd-11/obj-a1 2023-05-31T19:59:30.234 DEBUG:boto:path=/pklxkd-11/obj-a1?versionId=L7ViqSXBtkdfIYltLrLN1BhDNjOq8xz 2023-05-31T19:59:30.235 DEBUG:boto:auth_path=/pklxkd-11/obj-a1?versionId=L7ViqSXBtkdfIYltLrLN1BhDNjOq8xz 2023-05-31T19:59:30.235 DEBUG:boto:Method: HEAD 2023-05-31T19:59:30.236 DEBUG:boto:Path: /pklxkd-11/obj-a1?versionId=L7ViqSXBtkdfIYltLrLN1BhDNjOq8xz 2023-05-31T19:59:30.237 DEBUG:boto:Data: 2023-05-31T19:59:30.237 DEBUG:boto:Headers: {} 2023-05-31T19:59:30.238 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T19:59:30.238 DEBUG:boto:Port: 8001 2023-05-31T19:59:30.239 DEBUG:boto:Params: {} 2023-05-31T19:59:30.240 DEBUG:boto:Token: None 2023-05-31T19:59:30.240 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 19:59:30 GMT /pklxkd-11/obj-a1?versionId=L7ViqSXBtkdfIYltLrLN1BhDNjOq8xz 2023-05-31T19:59:30.241 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:W3nzKm5sW5Sjzcawr0H1qhjRkso= 2023-05-31T19:59:30.241 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 19:59:30 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:W3nzKm5sW5Sjzcawr0H1qhjRkso=', 'Content-Length': '0'} 2023-05-31T19:59:30.243 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 19:58:29 GMT'), ('x-amz-version-id', 'L7ViqSXBtkdfIYltLrLN1BhDNjOq8xz'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000008e915ba017e9ba07-006477a722-1112-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 19:59:30 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T19:59:30.244 DEBUG:boto:path=/pklxkd-11/obj-a1 2023-05-31T19:59:30.244 DEBUG:boto:auth_path=/pklxkd-11/obj-a1 2023-05-31T19:59:30.245 DEBUG:boto:path=/pklxkd-11/obj-a1?versionId=L7ViqSXBtkdfIYltLrLN1BhDNjOq8xz 2023-05-31T19:59:30.245 DEBUG:boto:auth_path=/pklxkd-11/obj-a1?versionId=L7ViqSXBtkdfIYltLrLN1BhDNjOq8xz 2023-05-31T19:59:30.246 DEBUG:boto:Method: HEAD 2023-05-31T19:59:30.246 DEBUG:boto:Path: /pklxkd-11/obj-a1?versionId=L7ViqSXBtkdfIYltLrLN1BhDNjOq8xz 2023-05-31T19:59:30.247 DEBUG:boto:Data: 2023-05-31T19:59:30.247 DEBUG:boto:Headers: {} 2023-05-31T19:59:30.248 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T19:59:30.249 DEBUG:boto:Port: 8000 2023-05-31T19:59:30.250 DEBUG:boto:Params: {} 2023-05-31T19:59:30.251 DEBUG:boto:Token: None 2023-05-31T19:59:30.252 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 19:59:30 GMT /pklxkd-11/obj-a1?versionId=L7ViqSXBtkdfIYltLrLN1BhDNjOq8xz 2023-05-31T19:59:30.253 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:W3nzKm5sW5Sjzcawr0H1qhjRkso= 2023-05-31T19:59:30.253 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 19:59:30 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:W3nzKm5sW5Sjzcawr0H1qhjRkso=', 'Content-Length': '0'} 2023-05-31T19:59:30.255 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 19:58:29 GMT'), ('x-amz-version-id', 'L7ViqSXBtkdfIYltLrLN1BhDNjOq8xz'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000d30fa89685cf27fc-006477a722-110e-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 19:59:30 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T19:59:30.256 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-a1 2023-05-31T19:59:30.256 DEBUG:boto:path=/pklxkd-11/obj-a1 2023-05-31T19:59:30.257 DEBUG:boto:auth_path=/pklxkd-11/obj-a1 2023-05-31T19:59:30.257 DEBUG:boto:path=/pklxkd-11/obj-a1?versionId=L7ViqSXBtkdfIYltLrLN1BhDNjOq8xz 2023-05-31T19:59:30.258 DEBUG:boto:auth_path=/pklxkd-11/obj-a1?versionId=L7ViqSXBtkdfIYltLrLN1BhDNjOq8xz 2023-05-31T19:59:30.258 DEBUG:boto:Method: GET 2023-05-31T19:59:30.258 DEBUG:boto:Path: /pklxkd-11/obj-a1?versionId=L7ViqSXBtkdfIYltLrLN1BhDNjOq8xz 2023-05-31T19:59:30.259 DEBUG:boto:Data: 2023-05-31T19:59:30.259 DEBUG:boto:Headers: {} 2023-05-31T19:59:30.259 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T19:59:30.260 DEBUG:boto:Port: 8001 2023-05-31T19:59:30.260 DEBUG:boto:Params: {} 2023-05-31T19:59:30.260 DEBUG:boto:Token: None 2023-05-31T19:59:30.261 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 19:59:30 GMT /pklxkd-11/obj-a1?versionId=L7ViqSXBtkdfIYltLrLN1BhDNjOq8xz 2023-05-31T19:59:30.261 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:+aeDdN+jdclMW9Iq6k2B1vKvQJM= 2023-05-31T19:59:30.262 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 19:59:30 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:+aeDdN+jdclMW9Iq6k2B1vKvQJM=', 'Content-Length': '0'} 2023-05-31T19:59:30.264 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 19:58:29 GMT'), ('x-amz-version-id', 'L7ViqSXBtkdfIYltLrLN1BhDNjOq8xz'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000426f51aaaaf92586-006477a722-1112-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 19:59:30 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T19:59:30.264 DEBUG:boto:path=/pklxkd-11/obj-a1 2023-05-31T19:59:30.265 DEBUG:boto:auth_path=/pklxkd-11/obj-a1 2023-05-31T19:59:30.265 DEBUG:boto:path=/pklxkd-11/obj-a1?versionId=L7ViqSXBtkdfIYltLrLN1BhDNjOq8xz 2023-05-31T19:59:30.266 DEBUG:boto:auth_path=/pklxkd-11/obj-a1?versionId=L7ViqSXBtkdfIYltLrLN1BhDNjOq8xz 2023-05-31T19:59:30.266 DEBUG:boto:Method: GET 2023-05-31T19:59:30.266 DEBUG:boto:Path: /pklxkd-11/obj-a1?versionId=L7ViqSXBtkdfIYltLrLN1BhDNjOq8xz 2023-05-31T19:59:30.267 DEBUG:boto:Data: 2023-05-31T19:59:30.267 DEBUG:boto:Headers: {} 2023-05-31T19:59:30.267 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T19:59:30.268 DEBUG:boto:Port: 8000 2023-05-31T19:59:30.268 DEBUG:boto:Params: {} 2023-05-31T19:59:30.269 DEBUG:boto:Token: None 2023-05-31T19:59:30.269 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 19:59:30 GMT /pklxkd-11/obj-a1?versionId=L7ViqSXBtkdfIYltLrLN1BhDNjOq8xz 2023-05-31T19:59:30.269 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:+aeDdN+jdclMW9Iq6k2B1vKvQJM= 2023-05-31T19:59:30.270 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 19:59:30 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:+aeDdN+jdclMW9Iq6k2B1vKvQJM=', 'Content-Length': '0'} 2023-05-31T19:59:30.272 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 19:58:29 GMT'), ('x-amz-version-id', 'L7ViqSXBtkdfIYltLrLN1BhDNjOq8xz'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000b6ec86ce695d6c03-006477a722-110e-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 19:59:30 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T19:59:30.272 DEBUG:boto:path=/pklxkd-11/ 2023-05-31T19:59:30.273 DEBUG:boto:auth_path=/pklxkd-11/ 2023-05-31T19:59:30.273 DEBUG:boto:path=/pklxkd-11/?prefix=obj-a1 2023-05-31T19:59:30.274 DEBUG:boto:auth_path=/pklxkd-11/?prefix=obj-a1 2023-05-31T19:59:30.274 DEBUG:boto:Method: GET 2023-05-31T19:59:30.274 DEBUG:boto:Path: /pklxkd-11/?prefix=obj-a1 2023-05-31T19:59:30.275 DEBUG:boto:Data: 2023-05-31T19:59:30.275 DEBUG:boto:Headers: {} 2023-05-31T19:59:30.275 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T19:59:30.276 DEBUG:boto:Port: 8001 2023-05-31T19:59:30.276 DEBUG:boto:Params: {} 2023-05-31T19:59:30.277 DEBUG:boto:Token: None 2023-05-31T19:59:30.277 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 19:59:30 GMT /pklxkd-11/ 2023-05-31T19:59:30.277 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:ObxMJzN1CLgWV5APWOP4fGp5x04= 2023-05-31T19:59:30.278 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 19:59:30 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:ObxMJzN1CLgWV5APWOP4fGp5x04=', 'Content-Length': '0'} 2023-05-31T19:59:30.280 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d65d4b9534a8c616-006477a722-1112-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 19:59:30 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T19:59:30.281 DEBUG:boto:b'pklxkd-11obj-a11000falseobj-a12023-05-31T19:57:47.555Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T19:59:30.282 DEBUG:boto:path=/pklxkd-11/ 2023-05-31T19:59:30.282 DEBUG:boto:auth_path=/pklxkd-11/ 2023-05-31T19:59:30.283 DEBUG:boto:path=/pklxkd-11/?prefix=obj-a1 2023-05-31T19:59:30.283 DEBUG:boto:auth_path=/pklxkd-11/?prefix=obj-a1 2023-05-31T19:59:30.283 DEBUG:boto:Method: GET 2023-05-31T19:59:30.284 DEBUG:boto:Path: /pklxkd-11/?prefix=obj-a1 2023-05-31T19:59:30.284 DEBUG:boto:Data: 2023-05-31T19:59:30.284 DEBUG:boto:Headers: {} 2023-05-31T19:59:30.285 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T19:59:30.285 DEBUG:boto:Port: 8000 2023-05-31T19:59:30.285 DEBUG:boto:Params: {} 2023-05-31T19:59:30.286 DEBUG:boto:Token: None 2023-05-31T19:59:30.286 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 19:59:30 GMT /pklxkd-11/ 2023-05-31T19:59:30.287 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:ObxMJzN1CLgWV5APWOP4fGp5x04= 2023-05-31T19:59:30.287 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 19:59:30 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:ObxMJzN1CLgWV5APWOP4fGp5x04=', 'Content-Length': '0'} 2023-05-31T19:59:30.289 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f6b30d6befa3f467-006477a722-110e-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 19:59:30 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T19:59:30.290 DEBUG:boto:b'pklxkd-11obj-a11000falseobj-a12023-05-31T19:57:47.555Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T19:59:30.291 DEBUG:boto:path=/pklxkd-11/obj-a1 2023-05-31T19:59:30.291 DEBUG:boto:auth_path=/pklxkd-11/obj-a1 2023-05-31T19:59:30.291 DEBUG:boto:Method: HEAD 2023-05-31T19:59:30.292 DEBUG:boto:Path: /pklxkd-11/obj-a1 2023-05-31T19:59:30.292 DEBUG:boto:Data: 2023-05-31T19:59:30.292 DEBUG:boto:Headers: {} 2023-05-31T19:59:30.293 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T19:59:30.293 DEBUG:boto:Port: 8001 2023-05-31T19:59:30.293 DEBUG:boto:Params: {} 2023-05-31T19:59:30.294 DEBUG:boto:Token: None 2023-05-31T19:59:30.294 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 19:59:30 GMT /pklxkd-11/obj-a1 2023-05-31T19:59:30.295 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:WbyVLD5P82L/KzJTjNWsl47CVA0= 2023-05-31T19:59:30.295 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 19:59:30 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:WbyVLD5P82L/KzJTjNWsl47CVA0=', 'Content-Length': '0'} 2023-05-31T19:59:30.297 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 19:58:29 GMT'), ('x-amz-version-id', 'L7ViqSXBtkdfIYltLrLN1BhDNjOq8xz'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000aa3cb86ac7ee8a17-006477a722-1112-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 19:59:30 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T19:59:30.298 DEBUG:boto:path=/pklxkd-11/obj-a1 2023-05-31T19:59:30.298 DEBUG:boto:auth_path=/pklxkd-11/obj-a1 2023-05-31T19:59:30.299 DEBUG:boto:Method: HEAD 2023-05-31T19:59:30.299 DEBUG:boto:Path: /pklxkd-11/obj-a1 2023-05-31T19:59:30.300 DEBUG:boto:Data: 2023-05-31T19:59:30.300 DEBUG:boto:Headers: {} 2023-05-31T19:59:30.300 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T19:59:30.301 DEBUG:boto:Port: 8000 2023-05-31T19:59:30.301 DEBUG:boto:Params: {} 2023-05-31T19:59:30.301 DEBUG:boto:Token: None 2023-05-31T19:59:30.302 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 19:59:30 GMT /pklxkd-11/obj-a1 2023-05-31T19:59:30.302 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:WbyVLD5P82L/KzJTjNWsl47CVA0= 2023-05-31T19:59:30.303 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 19:59:30 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:WbyVLD5P82L/KzJTjNWsl47CVA0=', 'Content-Length': '0'} 2023-05-31T19:59:30.305 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 19:58:29 GMT'), ('x-amz-version-id', 'L7ViqSXBtkdfIYltLrLN1BhDNjOq8xz'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000ec4de52eaacecf76-006477a722-110e-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 19:59:30 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T19:59:30.305 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-a1 2023-05-31T19:59:30.306 DEBUG:boto:path=/pklxkd-11/obj-a1 2023-05-31T19:59:30.306 DEBUG:boto:auth_path=/pklxkd-11/obj-a1 2023-05-31T19:59:30.306 DEBUG:boto:path=/pklxkd-11/obj-a1?versionId=L7ViqSXBtkdfIYltLrLN1BhDNjOq8xz 2023-05-31T19:59:30.307 DEBUG:boto:auth_path=/pklxkd-11/obj-a1?versionId=L7ViqSXBtkdfIYltLrLN1BhDNjOq8xz 2023-05-31T19:59:30.307 DEBUG:boto:Method: GET 2023-05-31T19:59:30.308 DEBUG:boto:Path: /pklxkd-11/obj-a1?versionId=L7ViqSXBtkdfIYltLrLN1BhDNjOq8xz 2023-05-31T19:59:30.308 DEBUG:boto:Data: 2023-05-31T19:59:30.308 DEBUG:boto:Headers: {} 2023-05-31T19:59:30.309 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T19:59:30.309 DEBUG:boto:Port: 8001 2023-05-31T19:59:30.309 DEBUG:boto:Params: {} 2023-05-31T19:59:30.310 DEBUG:boto:Token: None 2023-05-31T19:59:30.310 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 19:59:30 GMT /pklxkd-11/obj-a1?versionId=L7ViqSXBtkdfIYltLrLN1BhDNjOq8xz 2023-05-31T19:59:30.311 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:+aeDdN+jdclMW9Iq6k2B1vKvQJM= 2023-05-31T19:59:30.311 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 19:59:30 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:+aeDdN+jdclMW9Iq6k2B1vKvQJM=', 'Content-Length': '0'} 2023-05-31T19:59:30.313 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 19:58:29 GMT'), ('x-amz-version-id', 'L7ViqSXBtkdfIYltLrLN1BhDNjOq8xz'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000445e2d7b7480a4ec-006477a722-1112-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 19:59:30 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T19:59:30.356 DEBUG:boto:path=/pklxkd-11/obj-a1 2023-05-31T19:59:30.357 DEBUG:boto:auth_path=/pklxkd-11/obj-a1 2023-05-31T19:59:30.357 DEBUG:boto:path=/pklxkd-11/obj-a1?versionId=L7ViqSXBtkdfIYltLrLN1BhDNjOq8xz 2023-05-31T19:59:30.358 DEBUG:boto:auth_path=/pklxkd-11/obj-a1?versionId=L7ViqSXBtkdfIYltLrLN1BhDNjOq8xz 2023-05-31T19:59:30.358 DEBUG:boto:Method: GET 2023-05-31T19:59:30.359 DEBUG:boto:Path: /pklxkd-11/obj-a1?versionId=L7ViqSXBtkdfIYltLrLN1BhDNjOq8xz 2023-05-31T19:59:30.360 DEBUG:boto:Data: 2023-05-31T19:59:30.360 DEBUG:boto:Headers: {} 2023-05-31T19:59:30.360 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T19:59:30.361 DEBUG:boto:Port: 8000 2023-05-31T19:59:30.362 DEBUG:boto:Params: {} 2023-05-31T19:59:30.362 DEBUG:boto:Token: None 2023-05-31T19:59:30.362 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 19:59:30 GMT /pklxkd-11/obj-a1?versionId=L7ViqSXBtkdfIYltLrLN1BhDNjOq8xz 2023-05-31T19:59:30.363 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:+aeDdN+jdclMW9Iq6k2B1vKvQJM= 2023-05-31T19:59:30.363 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 19:59:30 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:+aeDdN+jdclMW9Iq6k2B1vKvQJM=', 'Content-Length': '0'} 2023-05-31T19:59:30.365 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 19:58:29 GMT'), ('x-amz-version-id', 'L7ViqSXBtkdfIYltLrLN1BhDNjOq8xz'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000001c35fab546e5b78d-006477a722-110e-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 19:59:30 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T19:59:30.366 DEBUG:boto:path=/pklxkd-11/ 2023-05-31T19:59:30.366 DEBUG:boto:auth_path=/pklxkd-11/ 2023-05-31T19:59:30.367 DEBUG:boto:path=/pklxkd-11/?prefix=obj-a1 2023-05-31T19:59:30.367 DEBUG:boto:auth_path=/pklxkd-11/?prefix=obj-a1 2023-05-31T19:59:30.368 DEBUG:boto:Method: GET 2023-05-31T19:59:30.368 DEBUG:boto:Path: /pklxkd-11/?prefix=obj-a1 2023-05-31T19:59:30.369 DEBUG:boto:Data: 2023-05-31T19:59:30.369 DEBUG:boto:Headers: {} 2023-05-31T19:59:30.370 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T19:59:30.370 DEBUG:boto:Port: 8001 2023-05-31T19:59:30.371 DEBUG:boto:Params: {} 2023-05-31T19:59:30.371 DEBUG:boto:Token: None 2023-05-31T19:59:30.371 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 19:59:30 GMT /pklxkd-11/ 2023-05-31T19:59:30.372 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:ObxMJzN1CLgWV5APWOP4fGp5x04= 2023-05-31T19:59:30.372 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 19:59:30 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:ObxMJzN1CLgWV5APWOP4fGp5x04=', 'Content-Length': '0'} 2023-05-31T19:59:30.375 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e18f73249b66af8d-006477a722-1112-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 19:59:30 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T19:59:30.375 DEBUG:boto:b'pklxkd-11obj-a11000falseobj-a12023-05-31T19:57:47.555Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T19:59:30.376 DEBUG:boto:path=/pklxkd-11/ 2023-05-31T19:59:30.377 DEBUG:boto:auth_path=/pklxkd-11/ 2023-05-31T19:59:30.377 DEBUG:boto:path=/pklxkd-11/?prefix=obj-a1 2023-05-31T19:59:30.377 DEBUG:boto:auth_path=/pklxkd-11/?prefix=obj-a1 2023-05-31T19:59:30.378 DEBUG:boto:Method: GET 2023-05-31T19:59:30.378 DEBUG:boto:Path: /pklxkd-11/?prefix=obj-a1 2023-05-31T19:59:30.378 DEBUG:boto:Data: 2023-05-31T19:59:30.379 DEBUG:boto:Headers: {} 2023-05-31T19:59:30.380 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T19:59:30.380 DEBUG:boto:Port: 8000 2023-05-31T19:59:30.381 DEBUG:boto:Params: {} 2023-05-31T19:59:30.381 DEBUG:boto:Token: None 2023-05-31T19:59:30.381 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 19:59:30 GMT /pklxkd-11/ 2023-05-31T19:59:30.382 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:ObxMJzN1CLgWV5APWOP4fGp5x04= 2023-05-31T19:59:30.383 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 19:59:30 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:ObxMJzN1CLgWV5APWOP4fGp5x04=', 'Content-Length': '0'} 2023-05-31T19:59:30.385 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000004d121af68b094467-006477a722-110e-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 19:59:30 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T19:59:30.386 DEBUG:boto:b'pklxkd-11obj-a11000falseobj-a12023-05-31T19:57:47.555Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T19:59:30.386 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-a2 2023-05-31T19:59:30.387 DEBUG:boto:path=/pklxkd-11/obj-a2 2023-05-31T19:59:30.387 DEBUG:boto:auth_path=/pklxkd-11/obj-a2 2023-05-31T19:59:30.388 DEBUG:boto:path=/pklxkd-11/obj-a2?versionId=d.2JtI5ycFHjEYrzZkvtbFVUQjr5kaL 2023-05-31T19:59:30.388 DEBUG:boto:auth_path=/pklxkd-11/obj-a2?versionId=d.2JtI5ycFHjEYrzZkvtbFVUQjr5kaL 2023-05-31T19:59:30.389 DEBUG:boto:Method: GET 2023-05-31T19:59:30.389 DEBUG:boto:Path: /pklxkd-11/obj-a2?versionId=d.2JtI5ycFHjEYrzZkvtbFVUQjr5kaL 2023-05-31T19:59:30.390 DEBUG:boto:Data: 2023-05-31T19:59:30.391 DEBUG:boto:Headers: {} 2023-05-31T19:59:30.392 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T19:59:30.392 DEBUG:boto:Port: 8001 2023-05-31T19:59:30.393 DEBUG:boto:Params: {} 2023-05-31T19:59:30.394 DEBUG:boto:Token: None 2023-05-31T19:59:30.395 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 19:59:30 GMT /pklxkd-11/obj-a2?versionId=d.2JtI5ycFHjEYrzZkvtbFVUQjr5kaL 2023-05-31T19:59:30.395 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:eG9PH/FXJI2Fqoh7uIs1qHtYRwM= 2023-05-31T19:59:30.396 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 19:59:30 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:eG9PH/FXJI2Fqoh7uIs1qHtYRwM=', 'Content-Length': '0'} 2023-05-31T19:59:30.398 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 19:58:29 GMT'), ('x-amz-version-id', 'd.2JtI5ycFHjEYrzZkvtbFVUQjr5kaL'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000002fff4f80857c712c-006477a722-1112-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 19:59:30 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T19:59:30.399 DEBUG:boto:path=/pklxkd-11/obj-a2 2023-05-31T19:59:30.400 DEBUG:boto:auth_path=/pklxkd-11/obj-a2 2023-05-31T19:59:30.401 DEBUG:boto:path=/pklxkd-11/obj-a2?versionId=d.2JtI5ycFHjEYrzZkvtbFVUQjr5kaL 2023-05-31T19:59:30.401 DEBUG:boto:auth_path=/pklxkd-11/obj-a2?versionId=d.2JtI5ycFHjEYrzZkvtbFVUQjr5kaL 2023-05-31T19:59:30.402 DEBUG:boto:Method: GET 2023-05-31T19:59:30.403 DEBUG:boto:Path: /pklxkd-11/obj-a2?versionId=d.2JtI5ycFHjEYrzZkvtbFVUQjr5kaL 2023-05-31T19:59:30.404 DEBUG:boto:Data: 2023-05-31T19:59:30.404 DEBUG:boto:Headers: {} 2023-05-31T19:59:30.405 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T19:59:30.406 DEBUG:boto:Port: 8000 2023-05-31T19:59:30.406 DEBUG:boto:Params: {} 2023-05-31T19:59:30.407 DEBUG:boto:Token: None 2023-05-31T19:59:30.408 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 19:59:30 GMT /pklxkd-11/obj-a2?versionId=d.2JtI5ycFHjEYrzZkvtbFVUQjr5kaL 2023-05-31T19:59:30.409 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:eG9PH/FXJI2Fqoh7uIs1qHtYRwM= 2023-05-31T19:59:30.409 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 19:59:30 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:eG9PH/FXJI2Fqoh7uIs1qHtYRwM=', 'Content-Length': '0'} 2023-05-31T19:59:30.412 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 19:58:29 GMT'), ('x-amz-version-id', 'd.2JtI5ycFHjEYrzZkvtbFVUQjr5kaL'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000004a16f310a90506dc-006477a722-110e-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 19:59:30 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T19:59:30.452 DEBUG:boto:path=/pklxkd-11/obj-a2 2023-05-31T19:59:30.453 DEBUG:boto:auth_path=/pklxkd-11/obj-a2 2023-05-31T19:59:30.454 DEBUG:boto:path=/pklxkd-11/obj-a2?versionId=d.2JtI5ycFHjEYrzZkvtbFVUQjr5kaL 2023-05-31T19:59:30.454 DEBUG:boto:auth_path=/pklxkd-11/obj-a2?versionId=d.2JtI5ycFHjEYrzZkvtbFVUQjr5kaL 2023-05-31T19:59:30.455 DEBUG:boto:Method: HEAD 2023-05-31T19:59:30.456 DEBUG:boto:Path: /pklxkd-11/obj-a2?versionId=d.2JtI5ycFHjEYrzZkvtbFVUQjr5kaL 2023-05-31T19:59:30.456 DEBUG:boto:Data: 2023-05-31T19:59:30.457 DEBUG:boto:Headers: {} 2023-05-31T19:59:30.458 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T19:59:30.459 DEBUG:boto:Port: 8001 2023-05-31T19:59:30.460 DEBUG:boto:Params: {} 2023-05-31T19:59:30.461 DEBUG:boto:Token: None 2023-05-31T19:59:30.461 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 19:59:30 GMT /pklxkd-11/obj-a2?versionId=d.2JtI5ycFHjEYrzZkvtbFVUQjr5kaL 2023-05-31T19:59:30.462 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:LoVOhANIJaTv/xET+pjm0+8j6v8= 2023-05-31T19:59:30.462 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 19:59:30 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:LoVOhANIJaTv/xET+pjm0+8j6v8=', 'Content-Length': '0'} 2023-05-31T19:59:30.464 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 19:58:29 GMT'), ('x-amz-version-id', 'd.2JtI5ycFHjEYrzZkvtbFVUQjr5kaL'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000003c331f448fc819fe-006477a722-1112-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 19:59:30 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T19:59:30.465 DEBUG:boto:path=/pklxkd-11/obj-a2 2023-05-31T19:59:30.465 DEBUG:boto:auth_path=/pklxkd-11/obj-a2 2023-05-31T19:59:30.465 DEBUG:boto:path=/pklxkd-11/obj-a2?versionId=d.2JtI5ycFHjEYrzZkvtbFVUQjr5kaL 2023-05-31T19:59:30.466 DEBUG:boto:auth_path=/pklxkd-11/obj-a2?versionId=d.2JtI5ycFHjEYrzZkvtbFVUQjr5kaL 2023-05-31T19:59:30.466 DEBUG:boto:Method: HEAD 2023-05-31T19:59:30.466 DEBUG:boto:Path: /pklxkd-11/obj-a2?versionId=d.2JtI5ycFHjEYrzZkvtbFVUQjr5kaL 2023-05-31T19:59:30.467 DEBUG:boto:Data: 2023-05-31T19:59:30.467 DEBUG:boto:Headers: {} 2023-05-31T19:59:30.467 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T19:59:30.468 DEBUG:boto:Port: 8000 2023-05-31T19:59:30.468 DEBUG:boto:Params: {} 2023-05-31T19:59:30.469 DEBUG:boto:Token: None 2023-05-31T19:59:30.469 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 19:59:30 GMT /pklxkd-11/obj-a2?versionId=d.2JtI5ycFHjEYrzZkvtbFVUQjr5kaL 2023-05-31T19:59:30.470 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:LoVOhANIJaTv/xET+pjm0+8j6v8= 2023-05-31T19:59:30.471 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 19:59:30 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:LoVOhANIJaTv/xET+pjm0+8j6v8=', 'Content-Length': '0'} 2023-05-31T19:59:30.473 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 19:58:29 GMT'), ('x-amz-version-id', 'd.2JtI5ycFHjEYrzZkvtbFVUQjr5kaL'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000e1229c25c9ec0c67-006477a722-110e-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 19:59:30 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T19:59:30.473 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-a2 2023-05-31T19:59:30.474 DEBUG:boto:path=/pklxkd-11/obj-a2 2023-05-31T19:59:30.474 DEBUG:boto:auth_path=/pklxkd-11/obj-a2 2023-05-31T19:59:30.475 DEBUG:boto:path=/pklxkd-11/obj-a2?versionId=d.2JtI5ycFHjEYrzZkvtbFVUQjr5kaL 2023-05-31T19:59:30.475 DEBUG:boto:auth_path=/pklxkd-11/obj-a2?versionId=d.2JtI5ycFHjEYrzZkvtbFVUQjr5kaL 2023-05-31T19:59:30.475 DEBUG:boto:Method: GET 2023-05-31T19:59:30.476 DEBUG:boto:Path: /pklxkd-11/obj-a2?versionId=d.2JtI5ycFHjEYrzZkvtbFVUQjr5kaL 2023-05-31T19:59:30.476 DEBUG:boto:Data: 2023-05-31T19:59:30.477 DEBUG:boto:Headers: {} 2023-05-31T19:59:30.477 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T19:59:30.477 DEBUG:boto:Port: 8001 2023-05-31T19:59:30.478 DEBUG:boto:Params: {} 2023-05-31T19:59:30.478 DEBUG:boto:Token: None 2023-05-31T19:59:30.479 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 19:59:30 GMT /pklxkd-11/obj-a2?versionId=d.2JtI5ycFHjEYrzZkvtbFVUQjr5kaL 2023-05-31T19:59:30.479 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:eG9PH/FXJI2Fqoh7uIs1qHtYRwM= 2023-05-31T19:59:30.480 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 19:59:30 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:eG9PH/FXJI2Fqoh7uIs1qHtYRwM=', 'Content-Length': '0'} 2023-05-31T19:59:30.481 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 19:58:29 GMT'), ('x-amz-version-id', 'd.2JtI5ycFHjEYrzZkvtbFVUQjr5kaL'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000c82ba55f3240c845-006477a722-1112-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 19:59:30 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T19:59:30.482 DEBUG:boto:path=/pklxkd-11/obj-a2 2023-05-31T19:59:30.483 DEBUG:boto:auth_path=/pklxkd-11/obj-a2 2023-05-31T19:59:30.483 DEBUG:boto:path=/pklxkd-11/obj-a2?versionId=d.2JtI5ycFHjEYrzZkvtbFVUQjr5kaL 2023-05-31T19:59:30.484 DEBUG:boto:auth_path=/pklxkd-11/obj-a2?versionId=d.2JtI5ycFHjEYrzZkvtbFVUQjr5kaL 2023-05-31T19:59:30.484 DEBUG:boto:Method: GET 2023-05-31T19:59:30.484 DEBUG:boto:Path: /pklxkd-11/obj-a2?versionId=d.2JtI5ycFHjEYrzZkvtbFVUQjr5kaL 2023-05-31T19:59:30.485 DEBUG:boto:Data: 2023-05-31T19:59:30.485 DEBUG:boto:Headers: {} 2023-05-31T19:59:30.486 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T19:59:30.486 DEBUG:boto:Port: 8000 2023-05-31T19:59:30.486 DEBUG:boto:Params: {} 2023-05-31T19:59:30.487 DEBUG:boto:Token: None 2023-05-31T19:59:30.487 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 19:59:30 GMT /pklxkd-11/obj-a2?versionId=d.2JtI5ycFHjEYrzZkvtbFVUQjr5kaL 2023-05-31T19:59:30.488 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:eG9PH/FXJI2Fqoh7uIs1qHtYRwM= 2023-05-31T19:59:30.488 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 19:59:30 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:eG9PH/FXJI2Fqoh7uIs1qHtYRwM=', 'Content-Length': '0'} 2023-05-31T19:59:30.490 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 19:58:29 GMT'), ('x-amz-version-id', 'd.2JtI5ycFHjEYrzZkvtbFVUQjr5kaL'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000c732c01c0cfd38e8-006477a722-110e-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 19:59:30 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T19:59:30.491 DEBUG:boto:path=/pklxkd-11/ 2023-05-31T19:59:30.491 DEBUG:boto:auth_path=/pklxkd-11/ 2023-05-31T19:59:30.491 DEBUG:boto:path=/pklxkd-11/?prefix=obj-a2 2023-05-31T19:59:30.492 DEBUG:boto:auth_path=/pklxkd-11/?prefix=obj-a2 2023-05-31T19:59:30.492 DEBUG:boto:Method: GET 2023-05-31T19:59:30.493 DEBUG:boto:Path: /pklxkd-11/?prefix=obj-a2 2023-05-31T19:59:30.493 DEBUG:boto:Data: 2023-05-31T19:59:30.493 DEBUG:boto:Headers: {} 2023-05-31T19:59:30.494 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T19:59:30.494 DEBUG:boto:Port: 8001 2023-05-31T19:59:30.494 DEBUG:boto:Params: {} 2023-05-31T19:59:30.495 DEBUG:boto:Token: None 2023-05-31T19:59:30.495 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 19:59:30 GMT /pklxkd-11/ 2023-05-31T19:59:30.496 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:ObxMJzN1CLgWV5APWOP4fGp5x04= 2023-05-31T19:59:30.496 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 19:59:30 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:ObxMJzN1CLgWV5APWOP4fGp5x04=', 'Content-Length': '0'} 2023-05-31T19:59:30.499 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000005fc9fc8b8ca00d52-006477a722-1112-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 19:59:30 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T19:59:30.499 DEBUG:boto:b'pklxkd-11obj-a21000falseobj-a22023-05-31T19:57:47.683Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T19:59:30.500 DEBUG:boto:path=/pklxkd-11/ 2023-05-31T19:59:30.500 DEBUG:boto:auth_path=/pklxkd-11/ 2023-05-31T19:59:30.501 DEBUG:boto:path=/pklxkd-11/?prefix=obj-a2 2023-05-31T19:59:30.501 DEBUG:boto:auth_path=/pklxkd-11/?prefix=obj-a2 2023-05-31T19:59:30.502 DEBUG:boto:Method: GET 2023-05-31T19:59:30.502 DEBUG:boto:Path: /pklxkd-11/?prefix=obj-a2 2023-05-31T19:59:30.502 DEBUG:boto:Data: 2023-05-31T19:59:30.503 DEBUG:boto:Headers: {} 2023-05-31T19:59:30.503 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T19:59:30.504 DEBUG:boto:Port: 8000 2023-05-31T19:59:30.504 DEBUG:boto:Params: {} 2023-05-31T19:59:30.504 DEBUG:boto:Token: None 2023-05-31T19:59:30.505 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 19:59:30 GMT /pklxkd-11/ 2023-05-31T19:59:30.505 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:ObxMJzN1CLgWV5APWOP4fGp5x04= 2023-05-31T19:59:30.506 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 19:59:30 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:ObxMJzN1CLgWV5APWOP4fGp5x04=', 'Content-Length': '0'} 2023-05-31T19:59:30.508 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e26cb63014aaf443-006477a722-110e-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 19:59:30 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T19:59:30.510 DEBUG:boto:b'pklxkd-11obj-a21000falseobj-a22023-05-31T19:57:47.683Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T19:59:30.511 DEBUG:boto:path=/pklxkd-11/obj-a2 2023-05-31T19:59:30.512 DEBUG:boto:auth_path=/pklxkd-11/obj-a2 2023-05-31T19:59:30.513 DEBUG:boto:Method: HEAD 2023-05-31T19:59:30.513 DEBUG:boto:Path: /pklxkd-11/obj-a2 2023-05-31T19:59:30.514 DEBUG:boto:Data: 2023-05-31T19:59:30.514 DEBUG:boto:Headers: {} 2023-05-31T19:59:30.515 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T19:59:30.516 DEBUG:boto:Port: 8001 2023-05-31T19:59:30.517 DEBUG:boto:Params: {} 2023-05-31T19:59:30.517 DEBUG:boto:Token: None 2023-05-31T19:59:30.517 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 19:59:30 GMT /pklxkd-11/obj-a2 2023-05-31T19:59:30.518 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:EkTDPOcIL2pZUCJMfPuA/2wSEEY= 2023-05-31T19:59:30.518 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 19:59:30 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:EkTDPOcIL2pZUCJMfPuA/2wSEEY=', 'Content-Length': '0'} 2023-05-31T19:59:30.521 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 19:58:29 GMT'), ('x-amz-version-id', 'd.2JtI5ycFHjEYrzZkvtbFVUQjr5kaL'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000001def03d6f60e8180-006477a722-1112-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 19:59:30 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T19:59:30.521 DEBUG:boto:path=/pklxkd-11/obj-a2 2023-05-31T19:59:30.522 DEBUG:boto:auth_path=/pklxkd-11/obj-a2 2023-05-31T19:59:30.522 DEBUG:boto:Method: HEAD 2023-05-31T19:59:30.522 DEBUG:boto:Path: /pklxkd-11/obj-a2 2023-05-31T19:59:30.523 DEBUG:boto:Data: 2023-05-31T19:59:30.523 DEBUG:boto:Headers: {} 2023-05-31T19:59:30.523 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T19:59:30.524 DEBUG:boto:Port: 8000 2023-05-31T19:59:30.524 DEBUG:boto:Params: {} 2023-05-31T19:59:30.525 DEBUG:boto:Token: None 2023-05-31T19:59:30.525 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 19:59:30 GMT /pklxkd-11/obj-a2 2023-05-31T19:59:30.525 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:EkTDPOcIL2pZUCJMfPuA/2wSEEY= 2023-05-31T19:59:30.526 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 19:59:30 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:EkTDPOcIL2pZUCJMfPuA/2wSEEY=', 'Content-Length': '0'} 2023-05-31T19:59:30.528 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 19:58:29 GMT'), ('x-amz-version-id', 'd.2JtI5ycFHjEYrzZkvtbFVUQjr5kaL'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000069419bbc7faa2e32-006477a722-110e-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 19:59:30 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T19:59:30.529 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-a2 2023-05-31T19:59:30.529 DEBUG:boto:path=/pklxkd-11/obj-a2 2023-05-31T19:59:30.529 DEBUG:boto:auth_path=/pklxkd-11/obj-a2 2023-05-31T19:59:30.530 DEBUG:boto:path=/pklxkd-11/obj-a2?versionId=d.2JtI5ycFHjEYrzZkvtbFVUQjr5kaL 2023-05-31T19:59:30.530 DEBUG:boto:auth_path=/pklxkd-11/obj-a2?versionId=d.2JtI5ycFHjEYrzZkvtbFVUQjr5kaL 2023-05-31T19:59:30.530 DEBUG:boto:Method: GET 2023-05-31T19:59:30.531 DEBUG:boto:Path: /pklxkd-11/obj-a2?versionId=d.2JtI5ycFHjEYrzZkvtbFVUQjr5kaL 2023-05-31T19:59:30.531 DEBUG:boto:Data: 2023-05-31T19:59:30.531 DEBUG:boto:Headers: {} 2023-05-31T19:59:30.532 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T19:59:30.532 DEBUG:boto:Port: 8001 2023-05-31T19:59:30.533 DEBUG:boto:Params: {} 2023-05-31T19:59:30.533 DEBUG:boto:Token: None 2023-05-31T19:59:30.533 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 19:59:30 GMT /pklxkd-11/obj-a2?versionId=d.2JtI5ycFHjEYrzZkvtbFVUQjr5kaL 2023-05-31T19:59:30.534 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:eG9PH/FXJI2Fqoh7uIs1qHtYRwM= 2023-05-31T19:59:30.534 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 19:59:30 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:eG9PH/FXJI2Fqoh7uIs1qHtYRwM=', 'Content-Length': '0'} 2023-05-31T19:59:30.536 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 19:58:29 GMT'), ('x-amz-version-id', 'd.2JtI5ycFHjEYrzZkvtbFVUQjr5kaL'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000004d7b934d221a5154-006477a722-1112-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 19:59:30 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T19:59:30.537 DEBUG:boto:path=/pklxkd-11/obj-a2 2023-05-31T19:59:30.538 DEBUG:boto:auth_path=/pklxkd-11/obj-a2 2023-05-31T19:59:30.539 DEBUG:boto:path=/pklxkd-11/obj-a2?versionId=d.2JtI5ycFHjEYrzZkvtbFVUQjr5kaL 2023-05-31T19:59:30.540 DEBUG:boto:auth_path=/pklxkd-11/obj-a2?versionId=d.2JtI5ycFHjEYrzZkvtbFVUQjr5kaL 2023-05-31T19:59:30.541 DEBUG:boto:Method: GET 2023-05-31T19:59:30.541 DEBUG:boto:Path: /pklxkd-11/obj-a2?versionId=d.2JtI5ycFHjEYrzZkvtbFVUQjr5kaL 2023-05-31T19:59:30.542 DEBUG:boto:Data: 2023-05-31T19:59:30.543 DEBUG:boto:Headers: {} 2023-05-31T19:59:30.544 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T19:59:30.544 DEBUG:boto:Port: 8000 2023-05-31T19:59:30.545 DEBUG:boto:Params: {} 2023-05-31T19:59:30.545 DEBUG:boto:Token: None 2023-05-31T19:59:30.545 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 19:59:30 GMT /pklxkd-11/obj-a2?versionId=d.2JtI5ycFHjEYrzZkvtbFVUQjr5kaL 2023-05-31T19:59:30.546 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:eG9PH/FXJI2Fqoh7uIs1qHtYRwM= 2023-05-31T19:59:30.546 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 19:59:30 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:eG9PH/FXJI2Fqoh7uIs1qHtYRwM=', 'Content-Length': '0'} 2023-05-31T19:59:30.548 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 19:58:29 GMT'), ('x-amz-version-id', 'd.2JtI5ycFHjEYrzZkvtbFVUQjr5kaL'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000701974acee11fc3c-006477a722-110e-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 19:59:30 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T19:59:30.592 DEBUG:boto:path=/pklxkd-11/ 2023-05-31T19:59:30.593 DEBUG:boto:auth_path=/pklxkd-11/ 2023-05-31T19:59:30.593 DEBUG:boto:path=/pklxkd-11/?prefix=obj-a2 2023-05-31T19:59:30.594 DEBUG:boto:auth_path=/pklxkd-11/?prefix=obj-a2 2023-05-31T19:59:30.594 DEBUG:boto:Method: GET 2023-05-31T19:59:30.594 DEBUG:boto:Path: /pklxkd-11/?prefix=obj-a2 2023-05-31T19:59:30.595 DEBUG:boto:Data: 2023-05-31T19:59:30.595 DEBUG:boto:Headers: {} 2023-05-31T19:59:30.595 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T19:59:30.596 DEBUG:boto:Port: 8001 2023-05-31T19:59:30.596 DEBUG:boto:Params: {} 2023-05-31T19:59:30.597 DEBUG:boto:Token: None 2023-05-31T19:59:30.597 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 19:59:30 GMT /pklxkd-11/ 2023-05-31T19:59:30.598 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:ObxMJzN1CLgWV5APWOP4fGp5x04= 2023-05-31T19:59:30.598 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 19:59:30 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:ObxMJzN1CLgWV5APWOP4fGp5x04=', 'Content-Length': '0'} 2023-05-31T19:59:30.601 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000004c510f9a8a6be709-006477a722-1112-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 19:59:30 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T19:59:30.601 DEBUG:boto:b'pklxkd-11obj-a21000falseobj-a22023-05-31T19:57:47.683Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T19:59:30.602 DEBUG:boto:path=/pklxkd-11/ 2023-05-31T19:59:30.602 DEBUG:boto:auth_path=/pklxkd-11/ 2023-05-31T19:59:30.603 DEBUG:boto:path=/pklxkd-11/?prefix=obj-a2 2023-05-31T19:59:30.603 DEBUG:boto:auth_path=/pklxkd-11/?prefix=obj-a2 2023-05-31T19:59:30.604 DEBUG:boto:Method: GET 2023-05-31T19:59:30.604 DEBUG:boto:Path: /pklxkd-11/?prefix=obj-a2 2023-05-31T19:59:30.604 DEBUG:boto:Data: 2023-05-31T19:59:30.605 DEBUG:boto:Headers: {} 2023-05-31T19:59:30.605 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T19:59:30.605 DEBUG:boto:Port: 8000 2023-05-31T19:59:30.606 DEBUG:boto:Params: {} 2023-05-31T19:59:30.606 DEBUG:boto:Token: None 2023-05-31T19:59:30.607 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 19:59:30 GMT /pklxkd-11/ 2023-05-31T19:59:30.607 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:ObxMJzN1CLgWV5APWOP4fGp5x04= 2023-05-31T19:59:30.608 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 19:59:30 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:ObxMJzN1CLgWV5APWOP4fGp5x04=', 'Content-Length': '0'} 2023-05-31T19:59:30.610 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a07bd2f040be7b2d-006477a722-110e-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 19:59:30 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T19:59:30.610 DEBUG:boto:b'pklxkd-11obj-a21000falseobj-a22023-05-31T19:57:47.683Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T19:59:30.611 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=pklxkd-11 zones={a2, a1} 2023-05-31T19:59:30.611 DEBUG:rgw_multi.tests:bucket checkpoint: source=a1 target=a2 bucket=pklxkd-12 2023-05-31T19:59:30.612 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'pklxkd-12', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-05-31T19:59:30.612 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'pklxkd-12', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-05-31T19:59:30.613 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket pklxkd-12 --source-zone a1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2023-05-31T19:59:30.676 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2023-05-31T19:59:30.677 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2023-05-31T19:59:30.724 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T19:59:30.718+0000 7fb89fd3c780 1 bucket sync caught up with source: 2023-05-31T19:59:30.724 INFO:teuthology.orchestra.run.smithi099.stderr: local status: [, , , , , 00000000002.3.11, , , 00000000003.10.6, 00000000011.28.6, ] 2023-05-31T19:59:30.725 INFO:teuthology.orchestra.run.smithi099.stderr: remote markers: [, , , , , 00000000002.3.11, , , 00000000003.10.6, 00000000011.28.6, ] 2023-05-31T19:59:30.725 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T19:59:30.718+0000 7fb89fd3c780 0 bucket checkpoint complete 2023-05-31T19:59:30.738 DEBUG:rgw_multi.tests:bucket checkpoint: source=a2 target=a1 bucket=pklxkd-12 2023-05-31T19:59:30.738 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'pklxkd-12', '--source-zone', 'a2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-05-31T19:59:30.739 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'pklxkd-12', '--source-zone', 'a2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-05-31T19:59:30.739 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket pklxkd-12 --source-zone a2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2023-05-31T19:59:30.796 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2023-05-31T19:59:30.796 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2023-05-31T19:59:30.838 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T19:59:30.832+0000 7fcf41095780 1 bucket sync caught up with source: 2023-05-31T19:59:30.838 INFO:teuthology.orchestra.run.smithi099.stderr: local status: [, , , , , 00000000003.10.10, , , 00000000011.42.6, 00000000004.38.6, ] 2023-05-31T19:59:30.838 INFO:teuthology.orchestra.run.smithi099.stderr: remote markers: [, , , , , 00000000003.10.10, , , 00000000011.42.6, 00000000004.38.6, ] 2023-05-31T19:59:30.838 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T19:59:30.832+0000 7fcf41095780 0 bucket checkpoint complete 2023-05-31T19:59:30.852 INFO:tasks.rgw_multi.zone_rados:comparing bucket=pklxkd-12 zones={a2, a1} 2023-05-31T19:59:30.852 DEBUG:boto:path=/pklxkd-12/ 2023-05-31T19:59:30.853 DEBUG:boto:auth_path=/pklxkd-12/ 2023-05-31T19:59:30.853 DEBUG:boto:Method: HEAD 2023-05-31T19:59:30.854 DEBUG:boto:Path: /pklxkd-12/ 2023-05-31T19:59:30.854 DEBUG:boto:Data: 2023-05-31T19:59:30.854 DEBUG:boto:Headers: {} 2023-05-31T19:59:30.855 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T19:59:30.855 DEBUG:boto:Port: 8001 2023-05-31T19:59:30.855 DEBUG:boto:Params: {} 2023-05-31T19:59:30.856 DEBUG:boto:Token: None 2023-05-31T19:59:30.856 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 19:59:30 GMT /pklxkd-12/ 2023-05-31T19:59:30.857 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:MZzdJVaeOKACKCAHo2Afrce7qbU= 2023-05-31T19:59:30.857 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 19:59:30 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:MZzdJVaeOKACKCAHo2Afrce7qbU=', 'Content-Length': '0'} 2023-05-31T19:59:30.859 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '3'), ('X-RGW-Bytes-Used', '16'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000909c7f6be2737918-006477a722-1112-a2'), ('Content-Length', '0'), ('Date', 'Wed, 31 May 2023 19:59:30 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T19:59:30.860 DEBUG:boto:path=/pklxkd-12/ 2023-05-31T19:59:30.861 DEBUG:boto:auth_path=/pklxkd-12/ 2023-05-31T19:59:30.861 DEBUG:boto:Method: HEAD 2023-05-31T19:59:30.862 DEBUG:boto:Path: /pklxkd-12/ 2023-05-31T19:59:30.862 DEBUG:boto:Data: 2023-05-31T19:59:30.863 DEBUG:boto:Headers: {} 2023-05-31T19:59:30.863 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T19:59:30.863 DEBUG:boto:Port: 8000 2023-05-31T19:59:30.864 DEBUG:boto:Params: {} 2023-05-31T19:59:30.864 DEBUG:boto:Token: None 2023-05-31T19:59:30.865 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 19:59:30 GMT /pklxkd-12/ 2023-05-31T19:59:30.866 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:MZzdJVaeOKACKCAHo2Afrce7qbU= 2023-05-31T19:59:30.867 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 19:59:30 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:MZzdJVaeOKACKCAHo2Afrce7qbU=', 'Content-Length': '0'} 2023-05-31T19:59:30.869 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '3'), ('X-RGW-Bytes-Used', '16'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000b468b8273c843829-006477a722-110e-a1'), ('Content-Length', '0'), ('Date', 'Wed, 31 May 2023 19:59:30 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T19:59:30.870 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2023-05-31T19:59:30.870 DEBUG:boto:path=/pklxkd-12/ 2023-05-31T19:59:30.870 DEBUG:boto:auth_path=/pklxkd-12/ 2023-05-31T19:59:30.871 DEBUG:boto:path=/pklxkd-12/?versions&max-keys=999 2023-05-31T19:59:30.871 DEBUG:boto:auth_path=/pklxkd-12/?versions&max-keys=999 2023-05-31T19:59:30.872 DEBUG:boto:Method: GET 2023-05-31T19:59:30.872 DEBUG:boto:Path: /pklxkd-12/?versions&max-keys=999 2023-05-31T19:59:30.873 DEBUG:boto:Data: 2023-05-31T19:59:30.873 DEBUG:boto:Headers: {} 2023-05-31T19:59:30.874 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T19:59:30.874 DEBUG:boto:Port: 8001 2023-05-31T19:59:30.874 DEBUG:boto:Params: {} 2023-05-31T19:59:30.875 DEBUG:boto:Token: None 2023-05-31T19:59:30.875 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 19:59:30 GMT /pklxkd-12/?versions 2023-05-31T19:59:30.876 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:r0xXnNmCpq2rJkb/a6im0wrK1Ws= 2023-05-31T19:59:30.876 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 19:59:30 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:r0xXnNmCpq2rJkb/a6im0wrK1Ws=', 'Content-Length': '0'} 2023-05-31T19:59:30.879 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a78b57da3c3eac0f-006477a722-1112-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 19:59:30 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T19:59:30.879 DEBUG:boto:b'pklxkd-12999falsedummyHuvfE2dxsypTRiytlVrY6cgwTbY4DCStrue2023-05-31T19:57:27.404Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-a1G1y4LrpL0rJndINHI52T89ADrfTweD3true2023-05-31T19:57:47.837Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-a2qiyfRuud8o74hAGpczwf4-FuN5p5887true2023-05-31T19:57:47.962Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T19:59:30.880 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2023-05-31T19:59:30.880 DEBUG:tasks.rgw_multi.zone_rados:o=obj-a1 2023-05-31T19:59:30.881 DEBUG:tasks.rgw_multi.zone_rados:o=obj-a2 2023-05-31T19:59:30.881 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2023-05-31T19:59:30.881 DEBUG:boto:path=/pklxkd-12/ 2023-05-31T19:59:30.882 DEBUG:boto:auth_path=/pklxkd-12/ 2023-05-31T19:59:30.882 DEBUG:boto:path=/pklxkd-12/?versions&max-keys=999 2023-05-31T19:59:30.882 DEBUG:boto:auth_path=/pklxkd-12/?versions&max-keys=999 2023-05-31T19:59:30.883 DEBUG:boto:Method: GET 2023-05-31T19:59:30.883 DEBUG:boto:Path: /pklxkd-12/?versions&max-keys=999 2023-05-31T19:59:30.883 DEBUG:boto:Data: 2023-05-31T19:59:30.884 DEBUG:boto:Headers: {} 2023-05-31T19:59:30.885 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T19:59:30.885 DEBUG:boto:Port: 8000 2023-05-31T19:59:30.886 DEBUG:boto:Params: {} 2023-05-31T19:59:30.886 DEBUG:boto:Token: None 2023-05-31T19:59:30.886 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 19:59:30 GMT /pklxkd-12/?versions 2023-05-31T19:59:30.887 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:r0xXnNmCpq2rJkb/a6im0wrK1Ws= 2023-05-31T19:59:30.887 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 19:59:30 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:r0xXnNmCpq2rJkb/a6im0wrK1Ws=', 'Content-Length': '0'} 2023-05-31T19:59:30.890 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000093b8ef825607664a-006477a722-110e-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 19:59:30 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T19:59:30.890 DEBUG:boto:b'pklxkd-12999falsedummyHuvfE2dxsypTRiytlVrY6cgwTbY4DCStrue2023-05-31T19:57:27.404Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-a1G1y4LrpL0rJndINHI52T89ADrfTweD3true2023-05-31T19:57:47.837Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-a2qiyfRuud8o74hAGpczwf4-FuN5p5887true2023-05-31T19:57:47.962Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T19:59:30.891 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2023-05-31T19:59:30.891 DEBUG:tasks.rgw_multi.zone_rados:o=obj-a1 2023-05-31T19:59:30.892 DEBUG:tasks.rgw_multi.zone_rados:o=obj-a2 2023-05-31T19:59:30.892 DEBUG:boto:path=/pklxkd-12/ 2023-05-31T19:59:30.893 DEBUG:boto:auth_path=/pklxkd-12/ 2023-05-31T19:59:30.893 DEBUG:boto:path=/pklxkd-12/?versions&max-keys=999 2023-05-31T19:59:30.893 DEBUG:boto:auth_path=/pklxkd-12/?versions&max-keys=999 2023-05-31T19:59:30.894 DEBUG:boto:Method: GET 2023-05-31T19:59:30.894 DEBUG:boto:Path: /pklxkd-12/?versions&max-keys=999 2023-05-31T19:59:30.894 DEBUG:boto:Data: 2023-05-31T19:59:30.895 DEBUG:boto:Headers: {} 2023-05-31T19:59:30.895 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T19:59:30.895 DEBUG:boto:Port: 8001 2023-05-31T19:59:30.896 DEBUG:boto:Params: {} 2023-05-31T19:59:30.896 DEBUG:boto:Token: None 2023-05-31T19:59:30.897 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 19:59:30 GMT /pklxkd-12/?versions 2023-05-31T19:59:30.897 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:r0xXnNmCpq2rJkb/a6im0wrK1Ws= 2023-05-31T19:59:30.898 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 19:59:30 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:r0xXnNmCpq2rJkb/a6im0wrK1Ws=', 'Content-Length': '0'} 2023-05-31T19:59:30.900 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b07570991a93996a-006477a722-1112-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 19:59:30 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T19:59:30.900 DEBUG:boto:b'pklxkd-12999falsedummyHuvfE2dxsypTRiytlVrY6cgwTbY4DCStrue2023-05-31T19:57:27.404Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-a1G1y4LrpL0rJndINHI52T89ADrfTweD3true2023-05-31T19:57:47.837Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-a2qiyfRuud8o74hAGpczwf4-FuN5p5887true2023-05-31T19:57:47.962Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T19:59:30.901 DEBUG:boto:path=/pklxkd-12/ 2023-05-31T19:59:30.901 DEBUG:boto:auth_path=/pklxkd-12/ 2023-05-31T19:59:30.902 DEBUG:boto:path=/pklxkd-12/?versions&max-keys=999 2023-05-31T19:59:30.902 DEBUG:boto:auth_path=/pklxkd-12/?versions&max-keys=999 2023-05-31T19:59:30.903 DEBUG:boto:Method: GET 2023-05-31T19:59:30.903 DEBUG:boto:Path: /pklxkd-12/?versions&max-keys=999 2023-05-31T19:59:30.904 DEBUG:boto:Data: 2023-05-31T19:59:30.904 DEBUG:boto:Headers: {} 2023-05-31T19:59:30.905 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T19:59:30.905 DEBUG:boto:Port: 8000 2023-05-31T19:59:30.905 DEBUG:boto:Params: {} 2023-05-31T19:59:30.906 DEBUG:boto:Token: None 2023-05-31T19:59:30.906 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 19:59:30 GMT /pklxkd-12/?versions 2023-05-31T19:59:30.907 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:r0xXnNmCpq2rJkb/a6im0wrK1Ws= 2023-05-31T19:59:30.907 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 19:59:30 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:r0xXnNmCpq2rJkb/a6im0wrK1Ws=', 'Content-Length': '0'} 2023-05-31T19:59:30.909 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000055daacd9b7aae3ad-006477a722-110e-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 19:59:30 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T19:59:30.910 DEBUG:boto:b'pklxkd-12999falsedummyHuvfE2dxsypTRiytlVrY6cgwTbY4DCStrue2023-05-31T19:57:27.404Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-a1G1y4LrpL0rJndINHI52T89ADrfTweD3true2023-05-31T19:57:47.837Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-a2qiyfRuud8o74hAGpczwf4-FuN5p5887true2023-05-31T19:57:47.962Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T19:59:30.911 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2023-05-31T19:59:30.911 DEBUG:boto:path=/pklxkd-12/dummy 2023-05-31T19:59:30.911 DEBUG:boto:auth_path=/pklxkd-12/dummy 2023-05-31T19:59:30.912 DEBUG:boto:path=/pklxkd-12/dummy?versionId=HuvfE2dxsypTRiytlVrY6cgwTbY4DCS 2023-05-31T19:59:30.912 DEBUG:boto:auth_path=/pklxkd-12/dummy?versionId=HuvfE2dxsypTRiytlVrY6cgwTbY4DCS 2023-05-31T19:59:30.913 DEBUG:boto:Method: GET 2023-05-31T19:59:30.913 DEBUG:boto:Path: /pklxkd-12/dummy?versionId=HuvfE2dxsypTRiytlVrY6cgwTbY4DCS 2023-05-31T19:59:30.913 DEBUG:boto:Data: 2023-05-31T19:59:30.914 DEBUG:boto:Headers: {} 2023-05-31T19:59:30.914 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T19:59:30.915 DEBUG:boto:Port: 8001 2023-05-31T19:59:30.915 DEBUG:boto:Params: {} 2023-05-31T19:59:30.915 DEBUG:boto:Token: None 2023-05-31T19:59:30.916 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 19:59:30 GMT /pklxkd-12/dummy?versionId=HuvfE2dxsypTRiytlVrY6cgwTbY4DCS 2023-05-31T19:59:30.916 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:+Z5mMzGS0kbwoLopmEHXBEXlNX8= 2023-05-31T19:59:30.917 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 19:59:30 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:+Z5mMzGS0kbwoLopmEHXBEXlNX8=', 'Content-Length': '0'} 2023-05-31T19:59:30.919 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 19:57:27 GMT'), ('x-amz-version-id', 'HuvfE2dxsypTRiytlVrY6cgwTbY4DCS'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000be2320ed7003dd5f-006477a722-1112-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 19:59:30 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T19:59:30.919 DEBUG:boto:path=/pklxkd-12/dummy 2023-05-31T19:59:30.920 DEBUG:boto:auth_path=/pklxkd-12/dummy 2023-05-31T19:59:30.920 DEBUG:boto:path=/pklxkd-12/dummy?versionId=HuvfE2dxsypTRiytlVrY6cgwTbY4DCS 2023-05-31T19:59:30.921 DEBUG:boto:auth_path=/pklxkd-12/dummy?versionId=HuvfE2dxsypTRiytlVrY6cgwTbY4DCS 2023-05-31T19:59:30.921 DEBUG:boto:Method: GET 2023-05-31T19:59:30.921 DEBUG:boto:Path: /pklxkd-12/dummy?versionId=HuvfE2dxsypTRiytlVrY6cgwTbY4DCS 2023-05-31T19:59:30.922 DEBUG:boto:Data: 2023-05-31T19:59:30.922 DEBUG:boto:Headers: {} 2023-05-31T19:59:30.923 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T19:59:30.923 DEBUG:boto:Port: 8000 2023-05-31T19:59:30.923 DEBUG:boto:Params: {} 2023-05-31T19:59:30.924 DEBUG:boto:Token: None 2023-05-31T19:59:30.924 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 19:59:30 GMT /pklxkd-12/dummy?versionId=HuvfE2dxsypTRiytlVrY6cgwTbY4DCS 2023-05-31T19:59:30.925 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:+Z5mMzGS0kbwoLopmEHXBEXlNX8= 2023-05-31T19:59:30.925 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 19:59:30 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:+Z5mMzGS0kbwoLopmEHXBEXlNX8=', 'Content-Length': '0'} 2023-05-31T19:59:30.927 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 19:57:27 GMT'), ('x-amz-version-id', 'HuvfE2dxsypTRiytlVrY6cgwTbY4DCS'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000009560e02240b4d63d-006477a722-110e-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 19:59:30 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T19:59:30.928 DEBUG:boto:path=/pklxkd-12/dummy 2023-05-31T19:59:30.928 DEBUG:boto:auth_path=/pklxkd-12/dummy 2023-05-31T19:59:30.928 DEBUG:boto:path=/pklxkd-12/dummy?versionId=HuvfE2dxsypTRiytlVrY6cgwTbY4DCS 2023-05-31T19:59:30.929 DEBUG:boto:auth_path=/pklxkd-12/dummy?versionId=HuvfE2dxsypTRiytlVrY6cgwTbY4DCS 2023-05-31T19:59:30.929 DEBUG:boto:Method: HEAD 2023-05-31T19:59:30.929 DEBUG:boto:Path: /pklxkd-12/dummy?versionId=HuvfE2dxsypTRiytlVrY6cgwTbY4DCS 2023-05-31T19:59:30.930 DEBUG:boto:Data: 2023-05-31T19:59:30.930 DEBUG:boto:Headers: {} 2023-05-31T19:59:30.930 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T19:59:30.931 DEBUG:boto:Port: 8001 2023-05-31T19:59:30.931 DEBUG:boto:Params: {} 2023-05-31T19:59:30.931 DEBUG:boto:Token: None 2023-05-31T19:59:30.932 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 19:59:30 GMT /pklxkd-12/dummy?versionId=HuvfE2dxsypTRiytlVrY6cgwTbY4DCS 2023-05-31T19:59:30.932 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:KrttpEKcCEmrbRNF5NUaItGIUI4= 2023-05-31T19:59:30.933 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 19:59:30 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:KrttpEKcCEmrbRNF5NUaItGIUI4=', 'Content-Length': '0'} 2023-05-31T19:59:30.937 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 19:57:27 GMT'), ('x-amz-version-id', 'HuvfE2dxsypTRiytlVrY6cgwTbY4DCS'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx0000023f9dc4ec1f68263-006477a722-1112-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 19:59:30 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T19:59:30.938 DEBUG:boto:path=/pklxkd-12/dummy 2023-05-31T19:59:30.938 DEBUG:boto:auth_path=/pklxkd-12/dummy 2023-05-31T19:59:30.938 DEBUG:boto:path=/pklxkd-12/dummy?versionId=HuvfE2dxsypTRiytlVrY6cgwTbY4DCS 2023-05-31T19:59:30.939 DEBUG:boto:auth_path=/pklxkd-12/dummy?versionId=HuvfE2dxsypTRiytlVrY6cgwTbY4DCS 2023-05-31T19:59:30.939 DEBUG:boto:Method: HEAD 2023-05-31T19:59:30.940 DEBUG:boto:Path: /pklxkd-12/dummy?versionId=HuvfE2dxsypTRiytlVrY6cgwTbY4DCS 2023-05-31T19:59:30.940 DEBUG:boto:Data: 2023-05-31T19:59:30.941 DEBUG:boto:Headers: {} 2023-05-31T19:59:30.941 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T19:59:30.942 DEBUG:boto:Port: 8000 2023-05-31T19:59:30.942 DEBUG:boto:Params: {} 2023-05-31T19:59:30.943 DEBUG:boto:Token: None 2023-05-31T19:59:30.943 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 19:59:30 GMT /pklxkd-12/dummy?versionId=HuvfE2dxsypTRiytlVrY6cgwTbY4DCS 2023-05-31T19:59:30.943 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:KrttpEKcCEmrbRNF5NUaItGIUI4= 2023-05-31T19:59:30.944 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 19:59:30 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:KrttpEKcCEmrbRNF5NUaItGIUI4=', 'Content-Length': '0'} 2023-05-31T19:59:30.945 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 19:57:27 GMT'), ('x-amz-version-id', 'HuvfE2dxsypTRiytlVrY6cgwTbY4DCS'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000b983503b52066f2d-006477a722-110e-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 19:59:30 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T19:59:30.946 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2023-05-31T19:59:30.946 DEBUG:boto:path=/pklxkd-12/dummy 2023-05-31T19:59:30.947 DEBUG:boto:auth_path=/pklxkd-12/dummy 2023-05-31T19:59:30.947 DEBUG:boto:path=/pklxkd-12/dummy?versionId=HuvfE2dxsypTRiytlVrY6cgwTbY4DCS 2023-05-31T19:59:30.947 DEBUG:boto:auth_path=/pklxkd-12/dummy?versionId=HuvfE2dxsypTRiytlVrY6cgwTbY4DCS 2023-05-31T19:59:30.948 DEBUG:boto:Method: GET 2023-05-31T19:59:30.948 DEBUG:boto:Path: /pklxkd-12/dummy?versionId=HuvfE2dxsypTRiytlVrY6cgwTbY4DCS 2023-05-31T19:59:30.949 DEBUG:boto:Data: 2023-05-31T19:59:30.949 DEBUG:boto:Headers: {} 2023-05-31T19:59:30.949 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T19:59:30.950 DEBUG:boto:Port: 8001 2023-05-31T19:59:30.950 DEBUG:boto:Params: {} 2023-05-31T19:59:30.950 DEBUG:boto:Token: None 2023-05-31T19:59:30.951 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 19:59:30 GMT /pklxkd-12/dummy?versionId=HuvfE2dxsypTRiytlVrY6cgwTbY4DCS 2023-05-31T19:59:30.951 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:+Z5mMzGS0kbwoLopmEHXBEXlNX8= 2023-05-31T19:59:30.952 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 19:59:30 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:+Z5mMzGS0kbwoLopmEHXBEXlNX8=', 'Content-Length': '0'} 2023-05-31T19:59:30.953 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 19:57:27 GMT'), ('x-amz-version-id', 'HuvfE2dxsypTRiytlVrY6cgwTbY4DCS'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx0000053888dcb86097b9a-006477a722-1112-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 19:59:30 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T19:59:30.954 DEBUG:boto:path=/pklxkd-12/dummy 2023-05-31T19:59:30.954 DEBUG:boto:auth_path=/pklxkd-12/dummy 2023-05-31T19:59:30.955 DEBUG:boto:path=/pklxkd-12/dummy?versionId=HuvfE2dxsypTRiytlVrY6cgwTbY4DCS 2023-05-31T19:59:30.955 DEBUG:boto:auth_path=/pklxkd-12/dummy?versionId=HuvfE2dxsypTRiytlVrY6cgwTbY4DCS 2023-05-31T19:59:30.956 DEBUG:boto:Method: GET 2023-05-31T19:59:30.956 DEBUG:boto:Path: /pklxkd-12/dummy?versionId=HuvfE2dxsypTRiytlVrY6cgwTbY4DCS 2023-05-31T19:59:30.957 DEBUG:boto:Data: 2023-05-31T19:59:30.957 DEBUG:boto:Headers: {} 2023-05-31T19:59:30.957 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T19:59:30.958 DEBUG:boto:Port: 8000 2023-05-31T19:59:30.958 DEBUG:boto:Params: {} 2023-05-31T19:59:30.958 DEBUG:boto:Token: None 2023-05-31T19:59:30.959 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 19:59:30 GMT /pklxkd-12/dummy?versionId=HuvfE2dxsypTRiytlVrY6cgwTbY4DCS 2023-05-31T19:59:30.959 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:+Z5mMzGS0kbwoLopmEHXBEXlNX8= 2023-05-31T19:59:30.960 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 19:59:30 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:+Z5mMzGS0kbwoLopmEHXBEXlNX8=', 'Content-Length': '0'} 2023-05-31T19:59:30.961 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 19:57:27 GMT'), ('x-amz-version-id', 'HuvfE2dxsypTRiytlVrY6cgwTbY4DCS'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000000733272cc1d56165-006477a722-110e-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 19:59:30 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T19:59:30.962 DEBUG:boto:path=/pklxkd-12/ 2023-05-31T19:59:30.962 DEBUG:boto:auth_path=/pklxkd-12/ 2023-05-31T19:59:30.963 DEBUG:boto:path=/pklxkd-12/?prefix=dummy 2023-05-31T19:59:30.963 DEBUG:boto:auth_path=/pklxkd-12/?prefix=dummy 2023-05-31T19:59:30.963 DEBUG:boto:Method: GET 2023-05-31T19:59:30.964 DEBUG:boto:Path: /pklxkd-12/?prefix=dummy 2023-05-31T19:59:30.964 DEBUG:boto:Data: 2023-05-31T19:59:30.965 DEBUG:boto:Headers: {} 2023-05-31T19:59:30.965 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T19:59:30.965 DEBUG:boto:Port: 8001 2023-05-31T19:59:30.966 DEBUG:boto:Params: {} 2023-05-31T19:59:30.966 DEBUG:boto:Token: None 2023-05-31T19:59:30.967 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 19:59:30 GMT /pklxkd-12/ 2023-05-31T19:59:30.967 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:lewkj48/tJtXyL5Qm7CMfZewstU= 2023-05-31T19:59:30.968 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 19:59:30 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:lewkj48/tJtXyL5Qm7CMfZewstU=', 'Content-Length': '0'} 2023-05-31T19:59:30.970 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000892641c7b8ba7263-006477a722-1112-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 19:59:30 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T19:59:30.970 DEBUG:boto:b'pklxkd-12dummy1000falsedummy2023-05-31T19:57:27.404Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T19:59:30.971 DEBUG:boto:path=/pklxkd-12/ 2023-05-31T19:59:30.971 DEBUG:boto:auth_path=/pklxkd-12/ 2023-05-31T19:59:30.972 DEBUG:boto:path=/pklxkd-12/?prefix=dummy 2023-05-31T19:59:30.972 DEBUG:boto:auth_path=/pklxkd-12/?prefix=dummy 2023-05-31T19:59:30.973 DEBUG:boto:Method: GET 2023-05-31T19:59:30.974 DEBUG:boto:Path: /pklxkd-12/?prefix=dummy 2023-05-31T19:59:30.975 DEBUG:boto:Data: 2023-05-31T19:59:30.975 DEBUG:boto:Headers: {} 2023-05-31T19:59:30.975 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T19:59:30.976 DEBUG:boto:Port: 8000 2023-05-31T19:59:30.976 DEBUG:boto:Params: {} 2023-05-31T19:59:30.977 DEBUG:boto:Token: None 2023-05-31T19:59:30.978 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 19:59:30 GMT /pklxkd-12/ 2023-05-31T19:59:30.978 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:lewkj48/tJtXyL5Qm7CMfZewstU= 2023-05-31T19:59:30.979 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 19:59:30 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:lewkj48/tJtXyL5Qm7CMfZewstU=', 'Content-Length': '0'} 2023-05-31T19:59:30.981 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000007301817283aec0ce-006477a722-110e-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 19:59:30 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T19:59:31.028 DEBUG:boto:b'pklxkd-12dummy1000falsedummy2023-05-31T19:57:27.404Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T19:59:31.029 DEBUG:boto:path=/pklxkd-12/dummy 2023-05-31T19:59:31.029 DEBUG:boto:auth_path=/pklxkd-12/dummy 2023-05-31T19:59:31.030 DEBUG:boto:Method: HEAD 2023-05-31T19:59:31.030 DEBUG:boto:Path: /pklxkd-12/dummy 2023-05-31T19:59:31.031 DEBUG:boto:Data: 2023-05-31T19:59:31.031 DEBUG:boto:Headers: {} 2023-05-31T19:59:31.031 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T19:59:31.032 DEBUG:boto:Port: 8001 2023-05-31T19:59:31.032 DEBUG:boto:Params: {} 2023-05-31T19:59:31.032 DEBUG:boto:Token: None 2023-05-31T19:59:31.033 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 19:59:31 GMT /pklxkd-12/dummy 2023-05-31T19:59:31.033 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:KSpiXAsATIUWtKD0t330aI0Uhq8= 2023-05-31T19:59:31.034 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 19:59:31 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:KSpiXAsATIUWtKD0t330aI0Uhq8=', 'Content-Length': '0'} 2023-05-31T19:59:31.036 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 19:57:27 GMT'), ('x-amz-version-id', 'HuvfE2dxsypTRiytlVrY6cgwTbY4DCS'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000bd6f6d3f471fe25b-006477a723-1112-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 19:59:31 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T19:59:31.036 DEBUG:boto:path=/pklxkd-12/dummy 2023-05-31T19:59:31.037 DEBUG:boto:auth_path=/pklxkd-12/dummy 2023-05-31T19:59:31.037 DEBUG:boto:Method: HEAD 2023-05-31T19:59:31.038 DEBUG:boto:Path: /pklxkd-12/dummy 2023-05-31T19:59:31.038 DEBUG:boto:Data: 2023-05-31T19:59:31.038 DEBUG:boto:Headers: {} 2023-05-31T19:59:31.039 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T19:59:31.039 DEBUG:boto:Port: 8000 2023-05-31T19:59:31.039 DEBUG:boto:Params: {} 2023-05-31T19:59:31.040 DEBUG:boto:Token: None 2023-05-31T19:59:31.040 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 19:59:31 GMT /pklxkd-12/dummy 2023-05-31T19:59:31.041 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:KSpiXAsATIUWtKD0t330aI0Uhq8= 2023-05-31T19:59:31.041 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 19:59:31 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:KSpiXAsATIUWtKD0t330aI0Uhq8=', 'Content-Length': '0'} 2023-05-31T19:59:31.043 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 19:57:27 GMT'), ('x-amz-version-id', 'HuvfE2dxsypTRiytlVrY6cgwTbY4DCS'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000a189def2bef33c7b-006477a723-110e-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 19:59:31 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T19:59:31.044 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2023-05-31T19:59:31.044 DEBUG:boto:path=/pklxkd-12/dummy 2023-05-31T19:59:31.045 DEBUG:boto:auth_path=/pklxkd-12/dummy 2023-05-31T19:59:31.045 DEBUG:boto:path=/pklxkd-12/dummy?versionId=HuvfE2dxsypTRiytlVrY6cgwTbY4DCS 2023-05-31T19:59:31.045 DEBUG:boto:auth_path=/pklxkd-12/dummy?versionId=HuvfE2dxsypTRiytlVrY6cgwTbY4DCS 2023-05-31T19:59:31.046 DEBUG:boto:Method: GET 2023-05-31T19:59:31.046 DEBUG:boto:Path: /pklxkd-12/dummy?versionId=HuvfE2dxsypTRiytlVrY6cgwTbY4DCS 2023-05-31T19:59:31.046 DEBUG:boto:Data: 2023-05-31T19:59:31.047 DEBUG:boto:Headers: {} 2023-05-31T19:59:31.047 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T19:59:31.048 DEBUG:boto:Port: 8001 2023-05-31T19:59:31.048 DEBUG:boto:Params: {} 2023-05-31T19:59:31.048 DEBUG:boto:Token: None 2023-05-31T19:59:31.049 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 19:59:31 GMT /pklxkd-12/dummy?versionId=HuvfE2dxsypTRiytlVrY6cgwTbY4DCS 2023-05-31T19:59:31.049 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:KQDdFnyUsJcjmruEdnQhSm1llJo= 2023-05-31T19:59:31.050 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 19:59:31 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:KQDdFnyUsJcjmruEdnQhSm1llJo=', 'Content-Length': '0'} 2023-05-31T19:59:31.051 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 19:57:27 GMT'), ('x-amz-version-id', 'HuvfE2dxsypTRiytlVrY6cgwTbY4DCS'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000230caea51f0991f1-006477a723-1112-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 19:59:31 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T19:59:31.052 DEBUG:boto:path=/pklxkd-12/dummy 2023-05-31T19:59:31.052 DEBUG:boto:auth_path=/pklxkd-12/dummy 2023-05-31T19:59:31.053 DEBUG:boto:path=/pklxkd-12/dummy?versionId=HuvfE2dxsypTRiytlVrY6cgwTbY4DCS 2023-05-31T19:59:31.053 DEBUG:boto:auth_path=/pklxkd-12/dummy?versionId=HuvfE2dxsypTRiytlVrY6cgwTbY4DCS 2023-05-31T19:59:31.054 DEBUG:boto:Method: GET 2023-05-31T19:59:31.054 DEBUG:boto:Path: /pklxkd-12/dummy?versionId=HuvfE2dxsypTRiytlVrY6cgwTbY4DCS 2023-05-31T19:59:31.054 DEBUG:boto:Data: 2023-05-31T19:59:31.055 DEBUG:boto:Headers: {} 2023-05-31T19:59:31.055 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T19:59:31.056 DEBUG:boto:Port: 8000 2023-05-31T19:59:31.056 DEBUG:boto:Params: {} 2023-05-31T19:59:31.057 DEBUG:boto:Token: None 2023-05-31T19:59:31.057 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 19:59:31 GMT /pklxkd-12/dummy?versionId=HuvfE2dxsypTRiytlVrY6cgwTbY4DCS 2023-05-31T19:59:31.058 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:KQDdFnyUsJcjmruEdnQhSm1llJo= 2023-05-31T19:59:31.058 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 19:59:31 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:KQDdFnyUsJcjmruEdnQhSm1llJo=', 'Content-Length': '0'} 2023-05-31T19:59:31.060 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 19:57:27 GMT'), ('x-amz-version-id', 'HuvfE2dxsypTRiytlVrY6cgwTbY4DCS'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx0000030f47dda4313e62c-006477a723-110e-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 19:59:31 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T19:59:31.061 DEBUG:boto:path=/pklxkd-12/ 2023-05-31T19:59:31.061 DEBUG:boto:auth_path=/pklxkd-12/ 2023-05-31T19:59:31.061 DEBUG:boto:path=/pklxkd-12/?prefix=dummy 2023-05-31T19:59:31.062 DEBUG:boto:auth_path=/pklxkd-12/?prefix=dummy 2023-05-31T19:59:31.062 DEBUG:boto:Method: GET 2023-05-31T19:59:31.063 DEBUG:boto:Path: /pklxkd-12/?prefix=dummy 2023-05-31T19:59:31.063 DEBUG:boto:Data: 2023-05-31T19:59:31.064 DEBUG:boto:Headers: {} 2023-05-31T19:59:31.065 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T19:59:31.065 DEBUG:boto:Port: 8001 2023-05-31T19:59:31.066 DEBUG:boto:Params: {} 2023-05-31T19:59:31.066 DEBUG:boto:Token: None 2023-05-31T19:59:31.067 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 19:59:31 GMT /pklxkd-12/ 2023-05-31T19:59:31.067 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:F24m3Kl7PG1u2KvM3vxRKbORtqQ= 2023-05-31T19:59:31.068 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 19:59:31 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:F24m3Kl7PG1u2KvM3vxRKbORtqQ=', 'Content-Length': '0'} 2023-05-31T19:59:31.070 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000008647ec1f1ea0733d-006477a723-1112-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 19:59:31 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T19:59:31.070 DEBUG:boto:b'pklxkd-12dummy1000falsedummy2023-05-31T19:57:27.404Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T19:59:31.071 DEBUG:boto:path=/pklxkd-12/ 2023-05-31T19:59:31.072 DEBUG:boto:auth_path=/pklxkd-12/ 2023-05-31T19:59:31.072 DEBUG:boto:path=/pklxkd-12/?prefix=dummy 2023-05-31T19:59:31.073 DEBUG:boto:auth_path=/pklxkd-12/?prefix=dummy 2023-05-31T19:59:31.073 DEBUG:boto:Method: GET 2023-05-31T19:59:31.074 DEBUG:boto:Path: /pklxkd-12/?prefix=dummy 2023-05-31T19:59:31.074 DEBUG:boto:Data: 2023-05-31T19:59:31.074 DEBUG:boto:Headers: {} 2023-05-31T19:59:31.075 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T19:59:31.075 DEBUG:boto:Port: 8000 2023-05-31T19:59:31.076 DEBUG:boto:Params: {} 2023-05-31T19:59:31.076 DEBUG:boto:Token: None 2023-05-31T19:59:31.077 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 19:59:31 GMT /pklxkd-12/ 2023-05-31T19:59:31.077 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:F24m3Kl7PG1u2KvM3vxRKbORtqQ= 2023-05-31T19:59:31.078 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 19:59:31 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:F24m3Kl7PG1u2KvM3vxRKbORtqQ=', 'Content-Length': '0'} 2023-05-31T19:59:31.080 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b114f52d6987ea7b-006477a723-110e-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 19:59:31 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T19:59:31.081 DEBUG:boto:b'pklxkd-12dummy1000falsedummy2023-05-31T19:57:27.404Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T19:59:31.081 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-a1 2023-05-31T19:59:31.082 DEBUG:boto:path=/pklxkd-12/obj-a1 2023-05-31T19:59:31.082 DEBUG:boto:auth_path=/pklxkd-12/obj-a1 2023-05-31T19:59:31.083 DEBUG:boto:path=/pklxkd-12/obj-a1?versionId=G1y4LrpL0rJndINHI52T89ADrfTweD3 2023-05-31T19:59:31.083 DEBUG:boto:auth_path=/pklxkd-12/obj-a1?versionId=G1y4LrpL0rJndINHI52T89ADrfTweD3 2023-05-31T19:59:31.084 DEBUG:boto:Method: GET 2023-05-31T19:59:31.084 DEBUG:boto:Path: /pklxkd-12/obj-a1?versionId=G1y4LrpL0rJndINHI52T89ADrfTweD3 2023-05-31T19:59:31.085 DEBUG:boto:Data: 2023-05-31T19:59:31.085 DEBUG:boto:Headers: {} 2023-05-31T19:59:31.086 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T19:59:31.086 DEBUG:boto:Port: 8001 2023-05-31T19:59:31.087 DEBUG:boto:Params: {} 2023-05-31T19:59:31.087 DEBUG:boto:Token: None 2023-05-31T19:59:31.087 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 19:59:31 GMT /pklxkd-12/obj-a1?versionId=G1y4LrpL0rJndINHI52T89ADrfTweD3 2023-05-31T19:59:31.088 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:jTs4fcG5IS1i4eUZxy/lvdoQCso= 2023-05-31T19:59:31.088 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 19:59:31 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:jTs4fcG5IS1i4eUZxy/lvdoQCso=', 'Content-Length': '0'} 2023-05-31T19:59:31.090 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 19:58:29 GMT'), ('x-amz-version-id', 'G1y4LrpL0rJndINHI52T89ADrfTweD3'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000dd2daf39cfe2c820-006477a723-1112-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 19:59:31 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T19:59:31.091 DEBUG:boto:path=/pklxkd-12/obj-a1 2023-05-31T19:59:31.091 DEBUG:boto:auth_path=/pklxkd-12/obj-a1 2023-05-31T19:59:31.092 DEBUG:boto:path=/pklxkd-12/obj-a1?versionId=G1y4LrpL0rJndINHI52T89ADrfTweD3 2023-05-31T19:59:31.092 DEBUG:boto:auth_path=/pklxkd-12/obj-a1?versionId=G1y4LrpL0rJndINHI52T89ADrfTweD3 2023-05-31T19:59:31.093 DEBUG:boto:Method: GET 2023-05-31T19:59:31.094 DEBUG:boto:Path: /pklxkd-12/obj-a1?versionId=G1y4LrpL0rJndINHI52T89ADrfTweD3 2023-05-31T19:59:31.094 DEBUG:boto:Data: 2023-05-31T19:59:31.095 DEBUG:boto:Headers: {} 2023-05-31T19:59:31.096 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T19:59:31.097 DEBUG:boto:Port: 8000 2023-05-31T19:59:31.098 DEBUG:boto:Params: {} 2023-05-31T19:59:31.099 DEBUG:boto:Token: None 2023-05-31T19:59:31.100 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 19:59:31 GMT /pklxkd-12/obj-a1?versionId=G1y4LrpL0rJndINHI52T89ADrfTweD3 2023-05-31T19:59:31.100 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:jTs4fcG5IS1i4eUZxy/lvdoQCso= 2023-05-31T19:59:31.101 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 19:59:31 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:jTs4fcG5IS1i4eUZxy/lvdoQCso=', 'Content-Length': '0'} 2023-05-31T19:59:31.102 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 19:58:29 GMT'), ('x-amz-version-id', 'G1y4LrpL0rJndINHI52T89ADrfTweD3'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000001da630f29d16df4e-006477a723-110e-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 19:59:31 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T19:59:31.103 DEBUG:boto:path=/pklxkd-12/obj-a1 2023-05-31T19:59:31.104 DEBUG:boto:auth_path=/pklxkd-12/obj-a1 2023-05-31T19:59:31.104 DEBUG:boto:path=/pklxkd-12/obj-a1?versionId=G1y4LrpL0rJndINHI52T89ADrfTweD3 2023-05-31T19:59:31.105 DEBUG:boto:auth_path=/pklxkd-12/obj-a1?versionId=G1y4LrpL0rJndINHI52T89ADrfTweD3 2023-05-31T19:59:31.105 DEBUG:boto:Method: HEAD 2023-05-31T19:59:31.105 DEBUG:boto:Path: /pklxkd-12/obj-a1?versionId=G1y4LrpL0rJndINHI52T89ADrfTweD3 2023-05-31T19:59:31.106 DEBUG:boto:Data: 2023-05-31T19:59:31.106 DEBUG:boto:Headers: {} 2023-05-31T19:59:31.106 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T19:59:31.107 DEBUG:boto:Port: 8001 2023-05-31T19:59:31.107 DEBUG:boto:Params: {} 2023-05-31T19:59:31.107 DEBUG:boto:Token: None 2023-05-31T19:59:31.108 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 19:59:31 GMT /pklxkd-12/obj-a1?versionId=G1y4LrpL0rJndINHI52T89ADrfTweD3 2023-05-31T19:59:31.108 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:/YG+kL2/CdIozzWcyzA4m3xHMyU= 2023-05-31T19:59:31.109 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 19:59:31 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:/YG+kL2/CdIozzWcyzA4m3xHMyU=', 'Content-Length': '0'} 2023-05-31T19:59:31.111 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 19:58:29 GMT'), ('x-amz-version-id', 'G1y4LrpL0rJndINHI52T89ADrfTweD3'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000009776a8c680acc3c8-006477a723-1112-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 19:59:31 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T19:59:31.111 DEBUG:boto:path=/pklxkd-12/obj-a1 2023-05-31T19:59:31.112 DEBUG:boto:auth_path=/pklxkd-12/obj-a1 2023-05-31T19:59:31.112 DEBUG:boto:path=/pklxkd-12/obj-a1?versionId=G1y4LrpL0rJndINHI52T89ADrfTweD3 2023-05-31T19:59:31.112 DEBUG:boto:auth_path=/pklxkd-12/obj-a1?versionId=G1y4LrpL0rJndINHI52T89ADrfTweD3 2023-05-31T19:59:31.113 DEBUG:boto:Method: HEAD 2023-05-31T19:59:31.113 DEBUG:boto:Path: /pklxkd-12/obj-a1?versionId=G1y4LrpL0rJndINHI52T89ADrfTweD3 2023-05-31T19:59:31.113 DEBUG:boto:Data: 2023-05-31T19:59:31.114 DEBUG:boto:Headers: {} 2023-05-31T19:59:31.114 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T19:59:31.114 DEBUG:boto:Port: 8000 2023-05-31T19:59:31.115 DEBUG:boto:Params: {} 2023-05-31T19:59:31.116 DEBUG:boto:Token: None 2023-05-31T19:59:31.117 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 19:59:31 GMT /pklxkd-12/obj-a1?versionId=G1y4LrpL0rJndINHI52T89ADrfTweD3 2023-05-31T19:59:31.118 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:/YG+kL2/CdIozzWcyzA4m3xHMyU= 2023-05-31T19:59:31.119 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 19:59:31 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:/YG+kL2/CdIozzWcyzA4m3xHMyU=', 'Content-Length': '0'} 2023-05-31T19:59:31.121 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 19:58:29 GMT'), ('x-amz-version-id', 'G1y4LrpL0rJndINHI52T89ADrfTweD3'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000d16de5b0c6515f79-006477a723-110e-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 19:59:31 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T19:59:31.122 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-a1 2023-05-31T19:59:31.123 DEBUG:boto:path=/pklxkd-12/obj-a1 2023-05-31T19:59:31.124 DEBUG:boto:auth_path=/pklxkd-12/obj-a1 2023-05-31T19:59:31.124 DEBUG:boto:path=/pklxkd-12/obj-a1?versionId=G1y4LrpL0rJndINHI52T89ADrfTweD3 2023-05-31T19:59:31.125 DEBUG:boto:auth_path=/pklxkd-12/obj-a1?versionId=G1y4LrpL0rJndINHI52T89ADrfTweD3 2023-05-31T19:59:31.126 DEBUG:boto:Method: GET 2023-05-31T19:59:31.127 DEBUG:boto:Path: /pklxkd-12/obj-a1?versionId=G1y4LrpL0rJndINHI52T89ADrfTweD3 2023-05-31T19:59:31.127 DEBUG:boto:Data: 2023-05-31T19:59:31.127 DEBUG:boto:Headers: {} 2023-05-31T19:59:31.128 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T19:59:31.128 DEBUG:boto:Port: 8001 2023-05-31T19:59:31.129 DEBUG:boto:Params: {} 2023-05-31T19:59:31.129 DEBUG:boto:Token: None 2023-05-31T19:59:31.130 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 19:59:31 GMT /pklxkd-12/obj-a1?versionId=G1y4LrpL0rJndINHI52T89ADrfTweD3 2023-05-31T19:59:31.130 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:jTs4fcG5IS1i4eUZxy/lvdoQCso= 2023-05-31T19:59:31.131 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 19:59:31 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:jTs4fcG5IS1i4eUZxy/lvdoQCso=', 'Content-Length': '0'} 2023-05-31T19:59:31.132 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 19:58:29 GMT'), ('x-amz-version-id', 'G1y4LrpL0rJndINHI52T89ADrfTweD3'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000278073d9a66a752f-006477a723-1112-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 19:59:31 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T19:59:31.134 DEBUG:boto:path=/pklxkd-12/obj-a1 2023-05-31T19:59:31.135 DEBUG:boto:auth_path=/pklxkd-12/obj-a1 2023-05-31T19:59:31.136 DEBUG:boto:path=/pklxkd-12/obj-a1?versionId=G1y4LrpL0rJndINHI52T89ADrfTweD3 2023-05-31T19:59:31.137 DEBUG:boto:auth_path=/pklxkd-12/obj-a1?versionId=G1y4LrpL0rJndINHI52T89ADrfTweD3 2023-05-31T19:59:31.137 DEBUG:boto:Method: GET 2023-05-31T19:59:31.137 DEBUG:boto:Path: /pklxkd-12/obj-a1?versionId=G1y4LrpL0rJndINHI52T89ADrfTweD3 2023-05-31T19:59:31.138 DEBUG:boto:Data: 2023-05-31T19:59:31.138 DEBUG:boto:Headers: {} 2023-05-31T19:59:31.138 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T19:59:31.139 DEBUG:boto:Port: 8000 2023-05-31T19:59:31.139 DEBUG:boto:Params: {} 2023-05-31T19:59:31.140 DEBUG:boto:Token: None 2023-05-31T19:59:31.140 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 19:59:31 GMT /pklxkd-12/obj-a1?versionId=G1y4LrpL0rJndINHI52T89ADrfTweD3 2023-05-31T19:59:31.141 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:jTs4fcG5IS1i4eUZxy/lvdoQCso= 2023-05-31T19:59:31.141 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 19:59:31 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:jTs4fcG5IS1i4eUZxy/lvdoQCso=', 'Content-Length': '0'} 2023-05-31T19:59:31.143 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 19:58:29 GMT'), ('x-amz-version-id', 'G1y4LrpL0rJndINHI52T89ADrfTweD3'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000764d06f59cd5650d-006477a723-110e-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 19:59:31 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T19:59:31.143 DEBUG:boto:path=/pklxkd-12/ 2023-05-31T19:59:31.144 DEBUG:boto:auth_path=/pklxkd-12/ 2023-05-31T19:59:31.144 DEBUG:boto:path=/pklxkd-12/?prefix=obj-a1 2023-05-31T19:59:31.144 DEBUG:boto:auth_path=/pklxkd-12/?prefix=obj-a1 2023-05-31T19:59:31.145 DEBUG:boto:Method: GET 2023-05-31T19:59:31.145 DEBUG:boto:Path: /pklxkd-12/?prefix=obj-a1 2023-05-31T19:59:31.146 DEBUG:boto:Data: 2023-05-31T19:59:31.146 DEBUG:boto:Headers: {} 2023-05-31T19:59:31.146 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T19:59:31.147 DEBUG:boto:Port: 8001 2023-05-31T19:59:31.147 DEBUG:boto:Params: {} 2023-05-31T19:59:31.147 DEBUG:boto:Token: None 2023-05-31T19:59:31.148 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 19:59:31 GMT /pklxkd-12/ 2023-05-31T19:59:31.148 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:F24m3Kl7PG1u2KvM3vxRKbORtqQ= 2023-05-31T19:59:31.149 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 19:59:31 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:F24m3Kl7PG1u2KvM3vxRKbORtqQ=', 'Content-Length': '0'} 2023-05-31T19:59:31.151 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000646c8941119d9f0a-006477a723-1112-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 19:59:31 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T19:59:31.192 DEBUG:boto:b'pklxkd-12obj-a11000falseobj-a12023-05-31T19:57:47.837Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T19:59:31.193 DEBUG:boto:path=/pklxkd-12/ 2023-05-31T19:59:31.194 DEBUG:boto:auth_path=/pklxkd-12/ 2023-05-31T19:59:31.194 DEBUG:boto:path=/pklxkd-12/?prefix=obj-a1 2023-05-31T19:59:31.194 DEBUG:boto:auth_path=/pklxkd-12/?prefix=obj-a1 2023-05-31T19:59:31.195 DEBUG:boto:Method: GET 2023-05-31T19:59:31.195 DEBUG:boto:Path: /pklxkd-12/?prefix=obj-a1 2023-05-31T19:59:31.195 DEBUG:boto:Data: 2023-05-31T19:59:31.196 DEBUG:boto:Headers: {} 2023-05-31T19:59:31.196 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T19:59:31.197 DEBUG:boto:Port: 8000 2023-05-31T19:59:31.197 DEBUG:boto:Params: {} 2023-05-31T19:59:31.197 DEBUG:boto:Token: None 2023-05-31T19:59:31.198 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 19:59:31 GMT /pklxkd-12/ 2023-05-31T19:59:31.198 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:F24m3Kl7PG1u2KvM3vxRKbORtqQ= 2023-05-31T19:59:31.199 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 19:59:31 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:F24m3Kl7PG1u2KvM3vxRKbORtqQ=', 'Content-Length': '0'} 2023-05-31T19:59:31.201 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d54c5de4bc22be79-006477a723-110e-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 19:59:31 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T19:59:31.201 DEBUG:boto:b'pklxkd-12obj-a11000falseobj-a12023-05-31T19:57:47.837Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T19:59:31.202 DEBUG:boto:path=/pklxkd-12/obj-a1 2023-05-31T19:59:31.203 DEBUG:boto:auth_path=/pklxkd-12/obj-a1 2023-05-31T19:59:31.203 DEBUG:boto:Method: HEAD 2023-05-31T19:59:31.203 DEBUG:boto:Path: /pklxkd-12/obj-a1 2023-05-31T19:59:31.204 DEBUG:boto:Data: 2023-05-31T19:59:31.204 DEBUG:boto:Headers: {} 2023-05-31T19:59:31.204 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T19:59:31.205 DEBUG:boto:Port: 8001 2023-05-31T19:59:31.205 DEBUG:boto:Params: {} 2023-05-31T19:59:31.205 DEBUG:boto:Token: None 2023-05-31T19:59:31.206 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 19:59:31 GMT /pklxkd-12/obj-a1 2023-05-31T19:59:31.206 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:FD04eDP7d5LGkzaPZy/a3SD1cgQ= 2023-05-31T19:59:31.207 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 19:59:31 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:FD04eDP7d5LGkzaPZy/a3SD1cgQ=', 'Content-Length': '0'} 2023-05-31T19:59:31.208 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 19:58:29 GMT'), ('x-amz-version-id', 'G1y4LrpL0rJndINHI52T89ADrfTweD3'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000049f72aae5c4af468-006477a723-1112-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 19:59:31 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T19:59:31.209 DEBUG:boto:path=/pklxkd-12/obj-a1 2023-05-31T19:59:31.210 DEBUG:boto:auth_path=/pklxkd-12/obj-a1 2023-05-31T19:59:31.210 DEBUG:boto:Method: HEAD 2023-05-31T19:59:31.210 DEBUG:boto:Path: /pklxkd-12/obj-a1 2023-05-31T19:59:31.211 DEBUG:boto:Data: 2023-05-31T19:59:31.211 DEBUG:boto:Headers: {} 2023-05-31T19:59:31.211 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T19:59:31.212 DEBUG:boto:Port: 8000 2023-05-31T19:59:31.212 DEBUG:boto:Params: {} 2023-05-31T19:59:31.212 DEBUG:boto:Token: None 2023-05-31T19:59:31.213 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 19:59:31 GMT /pklxkd-12/obj-a1 2023-05-31T19:59:31.213 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:FD04eDP7d5LGkzaPZy/a3SD1cgQ= 2023-05-31T19:59:31.214 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 19:59:31 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:FD04eDP7d5LGkzaPZy/a3SD1cgQ=', 'Content-Length': '0'} 2023-05-31T19:59:31.216 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 19:58:29 GMT'), ('x-amz-version-id', 'G1y4LrpL0rJndINHI52T89ADrfTweD3'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000024ec4e369855286c-006477a723-110e-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 19:59:31 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T19:59:31.216 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-a1 2023-05-31T19:59:31.217 DEBUG:boto:path=/pklxkd-12/obj-a1 2023-05-31T19:59:31.217 DEBUG:boto:auth_path=/pklxkd-12/obj-a1 2023-05-31T19:59:31.217 DEBUG:boto:path=/pklxkd-12/obj-a1?versionId=G1y4LrpL0rJndINHI52T89ADrfTweD3 2023-05-31T19:59:31.218 DEBUG:boto:auth_path=/pklxkd-12/obj-a1?versionId=G1y4LrpL0rJndINHI52T89ADrfTweD3 2023-05-31T19:59:31.218 DEBUG:boto:Method: GET 2023-05-31T19:59:31.219 DEBUG:boto:Path: /pklxkd-12/obj-a1?versionId=G1y4LrpL0rJndINHI52T89ADrfTweD3 2023-05-31T19:59:31.219 DEBUG:boto:Data: 2023-05-31T19:59:31.219 DEBUG:boto:Headers: {} 2023-05-31T19:59:31.220 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T19:59:31.220 DEBUG:boto:Port: 8001 2023-05-31T19:59:31.220 DEBUG:boto:Params: {} 2023-05-31T19:59:31.221 DEBUG:boto:Token: None 2023-05-31T19:59:31.221 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 19:59:31 GMT /pklxkd-12/obj-a1?versionId=G1y4LrpL0rJndINHI52T89ADrfTweD3 2023-05-31T19:59:31.222 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:jTs4fcG5IS1i4eUZxy/lvdoQCso= 2023-05-31T19:59:31.222 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 19:59:31 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:jTs4fcG5IS1i4eUZxy/lvdoQCso=', 'Content-Length': '0'} 2023-05-31T19:59:31.224 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 19:58:29 GMT'), ('x-amz-version-id', 'G1y4LrpL0rJndINHI52T89ADrfTweD3'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000110db0f76f84212f-006477a723-1112-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 19:59:31 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T19:59:31.224 DEBUG:boto:path=/pklxkd-12/obj-a1 2023-05-31T19:59:31.225 DEBUG:boto:auth_path=/pklxkd-12/obj-a1 2023-05-31T19:59:31.225 DEBUG:boto:path=/pklxkd-12/obj-a1?versionId=G1y4LrpL0rJndINHI52T89ADrfTweD3 2023-05-31T19:59:31.226 DEBUG:boto:auth_path=/pklxkd-12/obj-a1?versionId=G1y4LrpL0rJndINHI52T89ADrfTweD3 2023-05-31T19:59:31.226 DEBUG:boto:Method: GET 2023-05-31T19:59:31.226 DEBUG:boto:Path: /pklxkd-12/obj-a1?versionId=G1y4LrpL0rJndINHI52T89ADrfTweD3 2023-05-31T19:59:31.227 DEBUG:boto:Data: 2023-05-31T19:59:31.227 DEBUG:boto:Headers: {} 2023-05-31T19:59:31.227 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T19:59:31.228 DEBUG:boto:Port: 8000 2023-05-31T19:59:31.228 DEBUG:boto:Params: {} 2023-05-31T19:59:31.229 DEBUG:boto:Token: None 2023-05-31T19:59:31.229 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 19:59:31 GMT /pklxkd-12/obj-a1?versionId=G1y4LrpL0rJndINHI52T89ADrfTweD3 2023-05-31T19:59:31.230 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:jTs4fcG5IS1i4eUZxy/lvdoQCso= 2023-05-31T19:59:31.230 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 19:59:31 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:jTs4fcG5IS1i4eUZxy/lvdoQCso=', 'Content-Length': '0'} 2023-05-31T19:59:31.231 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 19:58:29 GMT'), ('x-amz-version-id', 'G1y4LrpL0rJndINHI52T89ADrfTweD3'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000059ffa70dd663d6f5-006477a723-110e-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 19:59:31 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T19:59:31.272 DEBUG:boto:path=/pklxkd-12/ 2023-05-31T19:59:31.273 DEBUG:boto:auth_path=/pklxkd-12/ 2023-05-31T19:59:31.273 DEBUG:boto:path=/pklxkd-12/?prefix=obj-a1 2023-05-31T19:59:31.274 DEBUG:boto:auth_path=/pklxkd-12/?prefix=obj-a1 2023-05-31T19:59:31.274 DEBUG:boto:Method: GET 2023-05-31T19:59:31.274 DEBUG:boto:Path: /pklxkd-12/?prefix=obj-a1 2023-05-31T19:59:31.275 DEBUG:boto:Data: 2023-05-31T19:59:31.275 DEBUG:boto:Headers: {} 2023-05-31T19:59:31.275 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T19:59:31.276 DEBUG:boto:Port: 8001 2023-05-31T19:59:31.276 DEBUG:boto:Params: {} 2023-05-31T19:59:31.277 DEBUG:boto:Token: None 2023-05-31T19:59:31.277 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 19:59:31 GMT /pklxkd-12/ 2023-05-31T19:59:31.277 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:F24m3Kl7PG1u2KvM3vxRKbORtqQ= 2023-05-31T19:59:31.278 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 19:59:31 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:F24m3Kl7PG1u2KvM3vxRKbORtqQ=', 'Content-Length': '0'} 2023-05-31T19:59:31.280 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000061b519b6ea837eb7-006477a723-1112-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 19:59:31 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T19:59:31.280 DEBUG:boto:b'pklxkd-12obj-a11000falseobj-a12023-05-31T19:57:47.837Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T19:59:31.281 DEBUG:boto:path=/pklxkd-12/ 2023-05-31T19:59:31.282 DEBUG:boto:auth_path=/pklxkd-12/ 2023-05-31T19:59:31.282 DEBUG:boto:path=/pklxkd-12/?prefix=obj-a1 2023-05-31T19:59:31.282 DEBUG:boto:auth_path=/pklxkd-12/?prefix=obj-a1 2023-05-31T19:59:31.283 DEBUG:boto:Method: GET 2023-05-31T19:59:31.283 DEBUG:boto:Path: /pklxkd-12/?prefix=obj-a1 2023-05-31T19:59:31.283 DEBUG:boto:Data: 2023-05-31T19:59:31.284 DEBUG:boto:Headers: {} 2023-05-31T19:59:31.284 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T19:59:31.284 DEBUG:boto:Port: 8000 2023-05-31T19:59:31.285 DEBUG:boto:Params: {} 2023-05-31T19:59:31.288 DEBUG:boto:Token: None 2023-05-31T19:59:31.288 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 19:59:31 GMT /pklxkd-12/ 2023-05-31T19:59:31.289 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:F24m3Kl7PG1u2KvM3vxRKbORtqQ= 2023-05-31T19:59:31.289 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 19:59:31 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:F24m3Kl7PG1u2KvM3vxRKbORtqQ=', 'Content-Length': '0'} 2023-05-31T19:59:31.292 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000072fd18a6c0c85377-006477a723-110e-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 19:59:31 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T19:59:31.293 DEBUG:boto:b'pklxkd-12obj-a11000falseobj-a12023-05-31T19:57:47.837Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T19:59:31.293 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-a2 2023-05-31T19:59:31.294 DEBUG:boto:path=/pklxkd-12/obj-a2 2023-05-31T19:59:31.294 DEBUG:boto:auth_path=/pklxkd-12/obj-a2 2023-05-31T19:59:31.294 DEBUG:boto:path=/pklxkd-12/obj-a2?versionId=qiyfRuud8o74hAGpczwf4-FuN5p5887 2023-05-31T19:59:31.295 DEBUG:boto:auth_path=/pklxkd-12/obj-a2?versionId=qiyfRuud8o74hAGpczwf4-FuN5p5887 2023-05-31T19:59:31.295 DEBUG:boto:Method: GET 2023-05-31T19:59:31.296 DEBUG:boto:Path: /pklxkd-12/obj-a2?versionId=qiyfRuud8o74hAGpczwf4-FuN5p5887 2023-05-31T19:59:31.296 DEBUG:boto:Data: 2023-05-31T19:59:31.296 DEBUG:boto:Headers: {} 2023-05-31T19:59:31.297 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T19:59:31.297 DEBUG:boto:Port: 8001 2023-05-31T19:59:31.297 DEBUG:boto:Params: {} 2023-05-31T19:59:31.298 DEBUG:boto:Token: None 2023-05-31T19:59:31.298 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 19:59:31 GMT /pklxkd-12/obj-a2?versionId=qiyfRuud8o74hAGpczwf4-FuN5p5887 2023-05-31T19:59:31.299 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:kxLc/j8Hj5DYJDmEK66sjDiKbhw= 2023-05-31T19:59:31.299 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 19:59:31 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:kxLc/j8Hj5DYJDmEK66sjDiKbhw=', 'Content-Length': '0'} 2023-05-31T19:59:31.301 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 19:58:29 GMT'), ('x-amz-version-id', 'qiyfRuud8o74hAGpczwf4-FuN5p5887'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000787d2fe2e3b2c473-006477a723-1112-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 19:59:31 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T19:59:31.301 DEBUG:boto:path=/pklxkd-12/obj-a2 2023-05-31T19:59:31.302 DEBUG:boto:auth_path=/pklxkd-12/obj-a2 2023-05-31T19:59:31.302 DEBUG:boto:path=/pklxkd-12/obj-a2?versionId=qiyfRuud8o74hAGpczwf4-FuN5p5887 2023-05-31T19:59:31.303 DEBUG:boto:auth_path=/pklxkd-12/obj-a2?versionId=qiyfRuud8o74hAGpczwf4-FuN5p5887 2023-05-31T19:59:31.303 DEBUG:boto:Method: GET 2023-05-31T19:59:31.303 DEBUG:boto:Path: /pklxkd-12/obj-a2?versionId=qiyfRuud8o74hAGpczwf4-FuN5p5887 2023-05-31T19:59:31.304 DEBUG:boto:Data: 2023-05-31T19:59:31.304 DEBUG:boto:Headers: {} 2023-05-31T19:59:31.304 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T19:59:31.305 DEBUG:boto:Port: 8000 2023-05-31T19:59:31.305 DEBUG:boto:Params: {} 2023-05-31T19:59:31.306 DEBUG:boto:Token: None 2023-05-31T19:59:31.306 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 19:59:31 GMT /pklxkd-12/obj-a2?versionId=qiyfRuud8o74hAGpczwf4-FuN5p5887 2023-05-31T19:59:31.306 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:kxLc/j8Hj5DYJDmEK66sjDiKbhw= 2023-05-31T19:59:31.307 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 19:59:31 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:kxLc/j8Hj5DYJDmEK66sjDiKbhw=', 'Content-Length': '0'} 2023-05-31T19:59:31.309 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 19:58:29 GMT'), ('x-amz-version-id', 'qiyfRuud8o74hAGpczwf4-FuN5p5887'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000f2db683bf9c6a096-006477a723-110e-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 19:59:31 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T19:59:31.309 DEBUG:boto:path=/pklxkd-12/obj-a2 2023-05-31T19:59:31.310 DEBUG:boto:auth_path=/pklxkd-12/obj-a2 2023-05-31T19:59:31.310 DEBUG:boto:path=/pklxkd-12/obj-a2?versionId=qiyfRuud8o74hAGpczwf4-FuN5p5887 2023-05-31T19:59:31.310 DEBUG:boto:auth_path=/pklxkd-12/obj-a2?versionId=qiyfRuud8o74hAGpczwf4-FuN5p5887 2023-05-31T19:59:31.311 DEBUG:boto:Method: HEAD 2023-05-31T19:59:31.311 DEBUG:boto:Path: /pklxkd-12/obj-a2?versionId=qiyfRuud8o74hAGpczwf4-FuN5p5887 2023-05-31T19:59:31.311 DEBUG:boto:Data: 2023-05-31T19:59:31.312 DEBUG:boto:Headers: {} 2023-05-31T19:59:31.312 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T19:59:31.313 DEBUG:boto:Port: 8001 2023-05-31T19:59:31.313 DEBUG:boto:Params: {} 2023-05-31T19:59:31.313 DEBUG:boto:Token: None 2023-05-31T19:59:31.314 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 19:59:31 GMT /pklxkd-12/obj-a2?versionId=qiyfRuud8o74hAGpczwf4-FuN5p5887 2023-05-31T19:59:31.314 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:zI9T6RTTDiugGGjVrxswUAR/BrU= 2023-05-31T19:59:31.314 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 19:59:31 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:zI9T6RTTDiugGGjVrxswUAR/BrU=', 'Content-Length': '0'} 2023-05-31T19:59:31.316 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 19:58:29 GMT'), ('x-amz-version-id', 'qiyfRuud8o74hAGpczwf4-FuN5p5887'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000083e938f2f9912217-006477a723-1112-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 19:59:31 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T19:59:31.316 DEBUG:boto:path=/pklxkd-12/obj-a2 2023-05-31T19:59:31.317 DEBUG:boto:auth_path=/pklxkd-12/obj-a2 2023-05-31T19:59:31.317 DEBUG:boto:path=/pklxkd-12/obj-a2?versionId=qiyfRuud8o74hAGpczwf4-FuN5p5887 2023-05-31T19:59:31.318 DEBUG:boto:auth_path=/pklxkd-12/obj-a2?versionId=qiyfRuud8o74hAGpczwf4-FuN5p5887 2023-05-31T19:59:31.318 DEBUG:boto:Method: HEAD 2023-05-31T19:59:31.318 DEBUG:boto:Path: /pklxkd-12/obj-a2?versionId=qiyfRuud8o74hAGpczwf4-FuN5p5887 2023-05-31T19:59:31.319 DEBUG:boto:Data: 2023-05-31T19:59:31.319 DEBUG:boto:Headers: {} 2023-05-31T19:59:31.319 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T19:59:31.319 DEBUG:boto:Port: 8000 2023-05-31T19:59:31.320 DEBUG:boto:Params: {} 2023-05-31T19:59:31.320 DEBUG:boto:Token: None 2023-05-31T19:59:31.321 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 19:59:31 GMT /pklxkd-12/obj-a2?versionId=qiyfRuud8o74hAGpczwf4-FuN5p5887 2023-05-31T19:59:31.321 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:zI9T6RTTDiugGGjVrxswUAR/BrU= 2023-05-31T19:59:31.321 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 19:59:31 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:zI9T6RTTDiugGGjVrxswUAR/BrU=', 'Content-Length': '0'} 2023-05-31T19:59:31.323 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 19:58:29 GMT'), ('x-amz-version-id', 'qiyfRuud8o74hAGpczwf4-FuN5p5887'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000a063322a1f5aafd8-006477a723-110e-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 19:59:31 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T19:59:31.323 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-a2 2023-05-31T19:59:31.324 DEBUG:boto:path=/pklxkd-12/obj-a2 2023-05-31T19:59:31.324 DEBUG:boto:auth_path=/pklxkd-12/obj-a2 2023-05-31T19:59:31.325 DEBUG:boto:path=/pklxkd-12/obj-a2?versionId=qiyfRuud8o74hAGpczwf4-FuN5p5887 2023-05-31T19:59:31.325 DEBUG:boto:auth_path=/pklxkd-12/obj-a2?versionId=qiyfRuud8o74hAGpczwf4-FuN5p5887 2023-05-31T19:59:31.325 DEBUG:boto:Method: GET 2023-05-31T19:59:31.326 DEBUG:boto:Path: /pklxkd-12/obj-a2?versionId=qiyfRuud8o74hAGpczwf4-FuN5p5887 2023-05-31T19:59:31.326 DEBUG:boto:Data: 2023-05-31T19:59:31.326 DEBUG:boto:Headers: {} 2023-05-31T19:59:31.327 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T19:59:31.327 DEBUG:boto:Port: 8001 2023-05-31T19:59:31.327 DEBUG:boto:Params: {} 2023-05-31T19:59:31.328 DEBUG:boto:Token: None 2023-05-31T19:59:31.328 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 19:59:31 GMT /pklxkd-12/obj-a2?versionId=qiyfRuud8o74hAGpczwf4-FuN5p5887 2023-05-31T19:59:31.329 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:kxLc/j8Hj5DYJDmEK66sjDiKbhw= 2023-05-31T19:59:31.329 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 19:59:31 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:kxLc/j8Hj5DYJDmEK66sjDiKbhw=', 'Content-Length': '0'} 2023-05-31T19:59:31.331 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 19:58:29 GMT'), ('x-amz-version-id', 'qiyfRuud8o74hAGpczwf4-FuN5p5887'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000007116678a27774821-006477a723-1112-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 19:59:31 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T19:59:31.372 DEBUG:boto:path=/pklxkd-12/obj-a2 2023-05-31T19:59:31.373 DEBUG:boto:auth_path=/pklxkd-12/obj-a2 2023-05-31T19:59:31.374 DEBUG:boto:path=/pklxkd-12/obj-a2?versionId=qiyfRuud8o74hAGpczwf4-FuN5p5887 2023-05-31T19:59:31.374 DEBUG:boto:auth_path=/pklxkd-12/obj-a2?versionId=qiyfRuud8o74hAGpczwf4-FuN5p5887 2023-05-31T19:59:31.375 DEBUG:boto:Method: GET 2023-05-31T19:59:31.375 DEBUG:boto:Path: /pklxkd-12/obj-a2?versionId=qiyfRuud8o74hAGpczwf4-FuN5p5887 2023-05-31T19:59:31.376 DEBUG:boto:Data: 2023-05-31T19:59:31.376 DEBUG:boto:Headers: {} 2023-05-31T19:59:31.377 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T19:59:31.377 DEBUG:boto:Port: 8000 2023-05-31T19:59:31.378 DEBUG:boto:Params: {} 2023-05-31T19:59:31.379 DEBUG:boto:Token: None 2023-05-31T19:59:31.379 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 19:59:31 GMT /pklxkd-12/obj-a2?versionId=qiyfRuud8o74hAGpczwf4-FuN5p5887 2023-05-31T19:59:31.380 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:kxLc/j8Hj5DYJDmEK66sjDiKbhw= 2023-05-31T19:59:31.381 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 19:59:31 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:kxLc/j8Hj5DYJDmEK66sjDiKbhw=', 'Content-Length': '0'} 2023-05-31T19:59:31.383 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 19:58:29 GMT'), ('x-amz-version-id', 'qiyfRuud8o74hAGpczwf4-FuN5p5887'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000004b62a031427f459-006477a723-110e-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 19:59:31 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T19:59:31.384 DEBUG:boto:path=/pklxkd-12/ 2023-05-31T19:59:31.384 DEBUG:boto:auth_path=/pklxkd-12/ 2023-05-31T19:59:31.385 DEBUG:boto:path=/pklxkd-12/?prefix=obj-a2 2023-05-31T19:59:31.385 DEBUG:boto:auth_path=/pklxkd-12/?prefix=obj-a2 2023-05-31T19:59:31.386 DEBUG:boto:Method: GET 2023-05-31T19:59:31.387 DEBUG:boto:Path: /pklxkd-12/?prefix=obj-a2 2023-05-31T19:59:31.387 DEBUG:boto:Data: 2023-05-31T19:59:31.388 DEBUG:boto:Headers: {} 2023-05-31T19:59:31.388 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T19:59:31.389 DEBUG:boto:Port: 8001 2023-05-31T19:59:31.390 DEBUG:boto:Params: {} 2023-05-31T19:59:31.390 DEBUG:boto:Token: None 2023-05-31T19:59:31.391 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 19:59:31 GMT /pklxkd-12/ 2023-05-31T19:59:31.392 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:F24m3Kl7PG1u2KvM3vxRKbORtqQ= 2023-05-31T19:59:31.392 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 19:59:31 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:F24m3Kl7PG1u2KvM3vxRKbORtqQ=', 'Content-Length': '0'} 2023-05-31T19:59:31.395 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000003821247f2f42883a-006477a723-1112-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 19:59:31 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T19:59:31.396 DEBUG:boto:b'pklxkd-12obj-a21000falseobj-a22023-05-31T19:57:47.962Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T19:59:31.397 DEBUG:boto:path=/pklxkd-12/ 2023-05-31T19:59:31.397 DEBUG:boto:auth_path=/pklxkd-12/ 2023-05-31T19:59:31.398 DEBUG:boto:path=/pklxkd-12/?prefix=obj-a2 2023-05-31T19:59:31.398 DEBUG:boto:auth_path=/pklxkd-12/?prefix=obj-a2 2023-05-31T19:59:31.399 DEBUG:boto:Method: GET 2023-05-31T19:59:31.400 DEBUG:boto:Path: /pklxkd-12/?prefix=obj-a2 2023-05-31T19:59:31.400 DEBUG:boto:Data: 2023-05-31T19:59:31.401 DEBUG:boto:Headers: {} 2023-05-31T19:59:31.401 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T19:59:31.402 DEBUG:boto:Port: 8000 2023-05-31T19:59:31.403 DEBUG:boto:Params: {} 2023-05-31T19:59:31.403 DEBUG:boto:Token: None 2023-05-31T19:59:31.404 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 19:59:31 GMT /pklxkd-12/ 2023-05-31T19:59:31.405 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:F24m3Kl7PG1u2KvM3vxRKbORtqQ= 2023-05-31T19:59:31.405 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 19:59:31 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:F24m3Kl7PG1u2KvM3vxRKbORtqQ=', 'Content-Length': '0'} 2023-05-31T19:59:31.408 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a4c5ab0647aa37ef-006477a723-110e-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 19:59:31 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T19:59:31.409 DEBUG:boto:b'pklxkd-12obj-a21000falseobj-a22023-05-31T19:57:47.962Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T19:59:31.410 DEBUG:boto:path=/pklxkd-12/obj-a2 2023-05-31T19:59:31.410 DEBUG:boto:auth_path=/pklxkd-12/obj-a2 2023-05-31T19:59:31.411 DEBUG:boto:Method: HEAD 2023-05-31T19:59:31.411 DEBUG:boto:Path: /pklxkd-12/obj-a2 2023-05-31T19:59:31.412 DEBUG:boto:Data: 2023-05-31T19:59:31.412 DEBUG:boto:Headers: {} 2023-05-31T19:59:31.413 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T19:59:31.414 DEBUG:boto:Port: 8001 2023-05-31T19:59:31.414 DEBUG:boto:Params: {} 2023-05-31T19:59:31.414 DEBUG:boto:Token: None 2023-05-31T19:59:31.415 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 19:59:31 GMT /pklxkd-12/obj-a2 2023-05-31T19:59:31.415 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:7RtugV1BAcOvZkUiQSs06Zp+A5Y= 2023-05-31T19:59:31.416 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 19:59:31 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:7RtugV1BAcOvZkUiQSs06Zp+A5Y=', 'Content-Length': '0'} 2023-05-31T19:59:31.418 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 19:58:29 GMT'), ('x-amz-version-id', 'qiyfRuud8o74hAGpczwf4-FuN5p5887'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000f01e34ed5dd49d96-006477a723-1112-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 19:59:31 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T19:59:31.418 DEBUG:boto:path=/pklxkd-12/obj-a2 2023-05-31T19:59:31.419 DEBUG:boto:auth_path=/pklxkd-12/obj-a2 2023-05-31T19:59:31.419 DEBUG:boto:Method: HEAD 2023-05-31T19:59:31.420 DEBUG:boto:Path: /pklxkd-12/obj-a2 2023-05-31T19:59:31.420 DEBUG:boto:Data: 2023-05-31T19:59:31.420 DEBUG:boto:Headers: {} 2023-05-31T19:59:31.421 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T19:59:31.421 DEBUG:boto:Port: 8000 2023-05-31T19:59:31.421 DEBUG:boto:Params: {} 2023-05-31T19:59:31.422 DEBUG:boto:Token: None 2023-05-31T19:59:31.422 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 19:59:31 GMT /pklxkd-12/obj-a2 2023-05-31T19:59:31.422 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:7RtugV1BAcOvZkUiQSs06Zp+A5Y= 2023-05-31T19:59:31.423 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 19:59:31 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:7RtugV1BAcOvZkUiQSs06Zp+A5Y=', 'Content-Length': '0'} 2023-05-31T19:59:31.425 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 19:58:29 GMT'), ('x-amz-version-id', 'qiyfRuud8o74hAGpczwf4-FuN5p5887'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000001b5990b93cf31fd3-006477a723-110e-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 19:59:31 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T19:59:31.425 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-a2 2023-05-31T19:59:31.426 DEBUG:boto:path=/pklxkd-12/obj-a2 2023-05-31T19:59:31.426 DEBUG:boto:auth_path=/pklxkd-12/obj-a2 2023-05-31T19:59:31.426 DEBUG:boto:path=/pklxkd-12/obj-a2?versionId=qiyfRuud8o74hAGpczwf4-FuN5p5887 2023-05-31T19:59:31.427 DEBUG:boto:auth_path=/pklxkd-12/obj-a2?versionId=qiyfRuud8o74hAGpczwf4-FuN5p5887 2023-05-31T19:59:31.427 DEBUG:boto:Method: GET 2023-05-31T19:59:31.427 DEBUG:boto:Path: /pklxkd-12/obj-a2?versionId=qiyfRuud8o74hAGpczwf4-FuN5p5887 2023-05-31T19:59:31.428 DEBUG:boto:Data: 2023-05-31T19:59:31.428 DEBUG:boto:Headers: {} 2023-05-31T19:59:31.429 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T19:59:31.429 DEBUG:boto:Port: 8001 2023-05-31T19:59:31.429 DEBUG:boto:Params: {} 2023-05-31T19:59:31.430 DEBUG:boto:Token: None 2023-05-31T19:59:31.430 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 19:59:31 GMT /pklxkd-12/obj-a2?versionId=qiyfRuud8o74hAGpczwf4-FuN5p5887 2023-05-31T19:59:31.431 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:kxLc/j8Hj5DYJDmEK66sjDiKbhw= 2023-05-31T19:59:31.431 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 19:59:31 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:kxLc/j8Hj5DYJDmEK66sjDiKbhw=', 'Content-Length': '0'} 2023-05-31T19:59:31.433 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 19:58:29 GMT'), ('x-amz-version-id', 'qiyfRuud8o74hAGpczwf4-FuN5p5887'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000086ddf5fa910ba66-006477a723-1112-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 19:59:31 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T19:59:31.433 DEBUG:boto:path=/pklxkd-12/obj-a2 2023-05-31T19:59:31.434 DEBUG:boto:auth_path=/pklxkd-12/obj-a2 2023-05-31T19:59:31.434 DEBUG:boto:path=/pklxkd-12/obj-a2?versionId=qiyfRuud8o74hAGpczwf4-FuN5p5887 2023-05-31T19:59:31.435 DEBUG:boto:auth_path=/pklxkd-12/obj-a2?versionId=qiyfRuud8o74hAGpczwf4-FuN5p5887 2023-05-31T19:59:31.435 DEBUG:boto:Method: GET 2023-05-31T19:59:31.435 DEBUG:boto:Path: /pklxkd-12/obj-a2?versionId=qiyfRuud8o74hAGpczwf4-FuN5p5887 2023-05-31T19:59:31.436 DEBUG:boto:Data: 2023-05-31T19:59:31.436 DEBUG:boto:Headers: {} 2023-05-31T19:59:31.436 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T19:59:31.437 DEBUG:boto:Port: 8000 2023-05-31T19:59:31.437 DEBUG:boto:Params: {} 2023-05-31T19:59:31.437 DEBUG:boto:Token: None 2023-05-31T19:59:31.438 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 19:59:31 GMT /pklxkd-12/obj-a2?versionId=qiyfRuud8o74hAGpczwf4-FuN5p5887 2023-05-31T19:59:31.438 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:kxLc/j8Hj5DYJDmEK66sjDiKbhw= 2023-05-31T19:59:31.439 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 19:59:31 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:kxLc/j8Hj5DYJDmEK66sjDiKbhw=', 'Content-Length': '0'} 2023-05-31T19:59:31.441 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 19:58:29 GMT'), ('x-amz-version-id', 'qiyfRuud8o74hAGpczwf4-FuN5p5887'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000001a9cb7c5e357ce2e-006477a723-110e-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 19:59:31 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T19:59:31.488 DEBUG:boto:path=/pklxkd-12/ 2023-05-31T19:59:31.489 DEBUG:boto:auth_path=/pklxkd-12/ 2023-05-31T19:59:31.489 DEBUG:boto:path=/pklxkd-12/?prefix=obj-a2 2023-05-31T19:59:31.490 DEBUG:boto:auth_path=/pklxkd-12/?prefix=obj-a2 2023-05-31T19:59:31.490 DEBUG:boto:Method: GET 2023-05-31T19:59:31.490 DEBUG:boto:Path: /pklxkd-12/?prefix=obj-a2 2023-05-31T19:59:31.491 DEBUG:boto:Data: 2023-05-31T19:59:31.491 DEBUG:boto:Headers: {} 2023-05-31T19:59:31.491 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T19:59:31.492 DEBUG:boto:Port: 8001 2023-05-31T19:59:31.492 DEBUG:boto:Params: {} 2023-05-31T19:59:31.493 DEBUG:boto:Token: None 2023-05-31T19:59:31.493 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 19:59:31 GMT /pklxkd-12/ 2023-05-31T19:59:31.493 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:F24m3Kl7PG1u2KvM3vxRKbORtqQ= 2023-05-31T19:59:31.494 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 19:59:31 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:F24m3Kl7PG1u2KvM3vxRKbORtqQ=', 'Content-Length': '0'} 2023-05-31T19:59:31.496 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000006cbc4cd449c0690e-006477a723-1112-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 19:59:31 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T19:59:31.497 DEBUG:boto:b'pklxkd-12obj-a21000falseobj-a22023-05-31T19:57:47.962Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T19:59:31.497 DEBUG:boto:path=/pklxkd-12/ 2023-05-31T19:59:31.498 DEBUG:boto:auth_path=/pklxkd-12/ 2023-05-31T19:59:31.498 DEBUG:boto:path=/pklxkd-12/?prefix=obj-a2 2023-05-31T19:59:31.499 DEBUG:boto:auth_path=/pklxkd-12/?prefix=obj-a2 2023-05-31T19:59:31.499 DEBUG:boto:Method: GET 2023-05-31T19:59:31.499 DEBUG:boto:Path: /pklxkd-12/?prefix=obj-a2 2023-05-31T19:59:31.500 DEBUG:boto:Data: 2023-05-31T19:59:31.500 DEBUG:boto:Headers: {} 2023-05-31T19:59:31.500 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T19:59:31.501 DEBUG:boto:Port: 8000 2023-05-31T19:59:31.501 DEBUG:boto:Params: {} 2023-05-31T19:59:31.501 DEBUG:boto:Token: None 2023-05-31T19:59:31.502 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 19:59:31 GMT /pklxkd-12/ 2023-05-31T19:59:31.502 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:F24m3Kl7PG1u2KvM3vxRKbORtqQ= 2023-05-31T19:59:31.503 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 19:59:31 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:F24m3Kl7PG1u2KvM3vxRKbORtqQ=', 'Content-Length': '0'} 2023-05-31T19:59:31.505 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000dcfb810013b9fece-006477a723-110e-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 19:59:31 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T19:59:31.505 DEBUG:boto:b'pklxkd-12obj-a21000falseobj-a22023-05-31T19:57:47.962Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T19:59:31.506 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=pklxkd-12 zones={a2, a1} 2023-05-31T19:59:31.507 INFO:tasks.rgw_multisite_tests:rgw_multi.tests.test_versioned_object_incremental_sync ... ok 2023-05-31T19:59:31.508 DEBUG:boto:path=/pklxkd-13/ 2023-05-31T19:59:31.508 DEBUG:boto:auth_path=/pklxkd-13/ 2023-05-31T19:59:31.508 DEBUG:boto:Method: PUT 2023-05-31T19:59:31.509 DEBUG:boto:Path: /pklxkd-13/ 2023-05-31T19:59:31.509 DEBUG:boto:Data: 2023-05-31T19:59:31.510 DEBUG:boto:Headers: {} 2023-05-31T19:59:31.510 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T19:59:31.511 DEBUG:boto:Port: 8000 2023-05-31T19:59:31.511 DEBUG:boto:Params: {} 2023-05-31T19:59:31.511 DEBUG:boto:Token: None 2023-05-31T19:59:31.512 DEBUG:boto:StringToSign: PUT Wed, 31 May 2023 19:59:31 GMT /pklxkd-13/ 2023-05-31T19:59:31.512 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:PhYL5BBkaw4YpR3QcMID3jFUt+w= 2023-05-31T19:59:31.513 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 19:59:31 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:PhYL5BBkaw4YpR3QcMID3jFUt+w=', 'Content-Length': '0'} 2023-05-31T19:59:31.526 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000cc24a405c2899bf9-006477a723-110e-a1'), ('Content-Length', '0'), ('Date', 'Wed, 31 May 2023 19:59:31 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T19:59:31.527 DEBUG:rgw_multi.tests:created bucket=pklxkd-13 2023-05-31T19:59:31.527 DEBUG:boto:path=/pklxkd-13/ 2023-05-31T19:59:31.528 DEBUG:boto:auth_path=/pklxkd-13/ 2023-05-31T19:59:31.528 DEBUG:boto:path=/pklxkd-13/?versioning 2023-05-31T19:59:31.528 DEBUG:boto:auth_path=/pklxkd-13/?versioning 2023-05-31T19:59:31.529 DEBUG:boto:Method: PUT 2023-05-31T19:59:31.529 DEBUG:boto:Path: /pklxkd-13/?versioning 2023-05-31T19:59:31.529 DEBUG:boto:Data: Enabled Disabled 2023-05-31T19:59:31.530 DEBUG:boto:Headers: {} 2023-05-31T19:59:31.530 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T19:59:31.530 DEBUG:boto:Port: 8000 2023-05-31T19:59:31.531 DEBUG:boto:Params: {} 2023-05-31T19:59:31.531 DEBUG:boto:Token: None 2023-05-31T19:59:31.532 DEBUG:boto:StringToSign: PUT Wed, 31 May 2023 19:59:31 GMT /pklxkd-13/?versioning 2023-05-31T19:59:31.532 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:UsmOok89HuEZs7Qt0RTJ06/J1zo= 2023-05-31T19:59:31.532 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 19:59:31 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:UsmOok89HuEZs7Qt0RTJ06/J1zo=', 'Content-Length': '228'} 2023-05-31T19:59:31.537 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx0000078d7fb53f40c667a-006477a723-110e-a1'), ('Content-Type', 'application/xml'), ('Content-Length', '0'), ('Date', 'Wed, 31 May 2023 19:59:31 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T19:59:31.537 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-31T19:59:31.538 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-31T19:59:31.538 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-05-31T19:59:31.590 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2023-05-31T19:59:31.590 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2023-05-31T19:59:31.620 INFO:teuthology.orchestra.run.smithi099.stderr:No --period given, using current period=2e6123d2-d695-4350-9a01-61a455bc782b 2023-05-31T19:59:31.642 INFO:teuthology.orchestra.run.smithi099.stdout:[ 2023-05-31T19:59:31.642 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T19:59:31.642 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1685562631.516089_59.1", 2023-05-31T19:59:31.643 INFO:teuthology.orchestra.run.smithi099.stdout: "last_update": "2023-05-31T19:50:31.516089Z" 2023-05-31T19:59:31.643 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T19:59:31.643 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T19:59:31.643 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "", 2023-05-31T19:59:31.644 INFO:teuthology.orchestra.run.smithi099.stdout: "last_update": "0.000000" 2023-05-31T19:59:31.644 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T19:59:31.644 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T19:59:31.645 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1685562970.274572_93.1", 2023-05-31T19:59:31.645 INFO:teuthology.orchestra.run.smithi099.stdout: "last_update": "2023-05-31T19:56:10.274572Z" 2023-05-31T19:59:31.645 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T19:59:31.645 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T19:59:31.646 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1685563171.532330_62.1", 2023-05-31T19:59:31.646 INFO:teuthology.orchestra.run.smithi099.stdout: "last_update": "2023-05-31T19:59:31.532330Z" 2023-05-31T19:59:31.646 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T19:59:31.646 INFO:teuthology.orchestra.run.smithi099.stdout:] 2023-05-31T19:59:31.647 DEBUG:rgw_multi.tests:master meta markers={0: '1_1685562631.516089_59.1', 1: '', 2: '1_1685562970.274572_93.1', 3: '1_1685563171.532330_62.1'} 2023-05-31T19:59:31.648 INFO:rgw_multi.tests:starting meta checkpoint for zone=a2 2023-05-31T19:59:31.648 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-31T19:59:31.648 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-31T19:59:31.649 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-05-31T19:59:31.698 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2023-05-31T19:59:31.699 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2023-05-31T19:59:31.739 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T19:59:31.733+0000 7f3d36aab780 20 RGW-SYNC:meta: read sync status 2023-05-31T19:59:31.740 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T19:59:31.734+0000 7f3d36aab780 20 RGW-SYNC:meta: read sync status 2023-05-31T19:59:31.756 INFO:teuthology.orchestra.run.smithi099.stdout:{ 2023-05-31T19:59:31.757 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_status": { 2023-05-31T19:59:31.757 INFO:teuthology.orchestra.run.smithi099.stdout: "info": { 2023-05-31T19:59:31.757 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "sync", 2023-05-31T19:59:31.757 INFO:teuthology.orchestra.run.smithi099.stdout: "num_shards": 4, 2023-05-31T19:59:31.757 INFO:teuthology.orchestra.run.smithi099.stdout: "period": "2e6123d2-d695-4350-9a01-61a455bc782b", 2023-05-31T19:59:31.758 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2023-05-31T19:59:31.758 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T19:59:31.758 INFO:teuthology.orchestra.run.smithi099.stdout: "markers": [ 2023-05-31T19:59:31.758 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T19:59:31.758 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 0, 2023-05-31T19:59:31.758 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T19:59:31.759 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2023-05-31T19:59:31.759 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1685562631.516089_59.1", 2023-05-31T19:59:31.759 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T19:59:31.759 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 1, 2023-05-31T19:59:31.759 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T19:59:31.759 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2023-05-31T19:50:31.516089Z", 2023-05-31T19:59:31.760 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2023-05-31T19:59:31.760 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T19:59:31.760 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T19:59:31.760 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T19:59:31.760 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 1, 2023-05-31T19:59:31.760 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T19:59:31.760 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2023-05-31T19:59:31.761 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "", 2023-05-31T19:59:31.761 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T19:59:31.761 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2023-05-31T19:59:31.761 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T19:59:31.761 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "0.000000", 2023-05-31T19:59:31.761 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2023-05-31T19:59:31.762 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T19:59:31.762 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T19:59:31.762 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T19:59:31.762 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 2, 2023-05-31T19:59:31.762 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T19:59:31.762 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2023-05-31T19:59:31.763 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1685562970.274572_93.1", 2023-05-31T19:59:31.763 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T19:59:31.763 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2023-05-31T19:59:31.763 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T19:59:31.763 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2023-05-31T19:56:10.274572Z", 2023-05-31T19:59:31.763 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2023-05-31T19:59:31.763 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T19:59:31.764 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T19:59:31.764 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T19:59:31.764 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 3, 2023-05-31T19:59:31.764 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T19:59:31.764 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2023-05-31T19:59:31.764 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1685563026.832102_54.1", 2023-05-31T19:59:31.765 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T19:59:31.765 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2023-05-31T19:59:31.765 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T19:59:31.765 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2023-05-31T19:57:06.832102Z", 2023-05-31T19:59:31.765 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2023-05-31T19:59:31.765 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T19:59:31.766 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T19:59:31.766 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2023-05-31T19:59:31.766 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T19:59:31.766 INFO:teuthology.orchestra.run.smithi099.stdout: "full_sync": { 2023-05-31T19:59:31.766 INFO:teuthology.orchestra.run.smithi099.stdout: "total": 1, 2023-05-31T19:59:31.766 INFO:teuthology.orchestra.run.smithi099.stdout: "complete": 1 2023-05-31T19:59:31.766 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T19:59:31.767 INFO:teuthology.orchestra.run.smithi099.stdout: "current_time": "2023-05-31T19:59:31Z" 2023-05-31T19:59:31.767 INFO:teuthology.orchestra.run.smithi099.stdout:} 2023-05-31T19:59:31.768 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "2e6123d2-d695-4350-9a01-61a455bc782b", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1685562631.516089_59.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-05-31T19:50:31.516089Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1685562970.274572_93.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-05-31T19:56:10.274572Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1685563026.832102_54.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-05-31T19:57:06.832102Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-05-31T19:59:31Z" } 2023-05-31T19:59:31.768 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1685562631.516089_59.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-05-31T19:50:31.516089Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1685562970.274572_93.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-05-31T19:56:10.274572Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1685563026.832102_54.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-05-31T19:57:06.832102Z', 'realm_epoch': 2}}] 2023-05-31T19:59:31.769 DEBUG:rgw_multi.tests:log_status={0: '1_1685562631.516089_59.1', 1: '', 2: '1_1685562970.274572_93.1', 3: '1_1685563171.532330_62.1'} 2023-05-31T19:59:31.769 DEBUG:rgw_multi.tests:sync_status={0: '1_1685562631.516089_59.1', 1: '', 2: '1_1685562970.274572_93.1', 3: '1_1685563026.832102_54.1'} 2023-05-31T19:59:31.769 WARNING:rgw_multi.tests:zone a2 behind master: shard=3 master=1_1685563171.532330_62.1 target=1_1685563026.832102_54.1 2023-05-31T19:59:36.771 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-31T19:59:36.771 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-31T19:59:36.774 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-05-31T19:59:36.831 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2023-05-31T19:59:36.832 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2023-05-31T19:59:36.873 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T19:59:36.867+0000 7f5a8770e780 20 RGW-SYNC:meta: read sync status 2023-05-31T19:59:36.874 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T19:59:36.868+0000 7f5a8770e780 20 RGW-SYNC:meta: read sync status 2023-05-31T19:59:36.892 INFO:teuthology.orchestra.run.smithi099.stdout:{ 2023-05-31T19:59:36.892 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_status": { 2023-05-31T19:59:36.893 INFO:teuthology.orchestra.run.smithi099.stdout: "info": { 2023-05-31T19:59:36.893 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "sync", 2023-05-31T19:59:36.893 INFO:teuthology.orchestra.run.smithi099.stdout: "num_shards": 4, 2023-05-31T19:59:36.893 INFO:teuthology.orchestra.run.smithi099.stdout: "period": "2e6123d2-d695-4350-9a01-61a455bc782b", 2023-05-31T19:59:36.893 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2023-05-31T19:59:36.893 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T19:59:36.894 INFO:teuthology.orchestra.run.smithi099.stdout: "markers": [ 2023-05-31T19:59:36.894 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T19:59:36.894 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 0, 2023-05-31T19:59:36.894 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T19:59:36.894 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2023-05-31T19:59:36.894 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1685562631.516089_59.1", 2023-05-31T19:59:36.895 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T19:59:36.895 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 1, 2023-05-31T19:59:36.895 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T19:59:36.895 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2023-05-31T19:50:31.516089Z", 2023-05-31T19:59:36.895 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2023-05-31T19:59:36.895 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T19:59:36.895 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T19:59:36.896 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T19:59:36.896 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 1, 2023-05-31T19:59:36.896 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T19:59:36.896 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2023-05-31T19:59:36.897 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "", 2023-05-31T19:59:36.897 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T19:59:36.897 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2023-05-31T19:59:36.897 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T19:59:36.897 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "0.000000", 2023-05-31T19:59:36.897 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2023-05-31T19:59:36.897 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T19:59:36.898 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T19:59:36.898 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T19:59:36.898 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 2, 2023-05-31T19:59:36.898 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T19:59:36.898 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2023-05-31T19:59:36.898 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1685562970.274572_93.1", 2023-05-31T19:59:36.899 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T19:59:36.899 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2023-05-31T19:59:36.899 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T19:59:36.899 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2023-05-31T19:56:10.274572Z", 2023-05-31T19:59:36.899 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2023-05-31T19:59:36.899 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T19:59:36.900 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T19:59:36.900 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T19:59:36.900 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 3, 2023-05-31T19:59:36.900 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T19:59:36.900 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2023-05-31T19:59:36.900 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1685563171.532330_62.1", 2023-05-31T19:59:36.900 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T19:59:36.901 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2023-05-31T19:59:36.901 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T19:59:36.901 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2023-05-31T19:59:31.532330Z", 2023-05-31T19:59:36.901 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2023-05-31T19:59:36.901 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T19:59:36.901 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T19:59:36.901 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2023-05-31T19:59:36.902 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T19:59:36.902 INFO:teuthology.orchestra.run.smithi099.stdout: "full_sync": { 2023-05-31T19:59:36.902 INFO:teuthology.orchestra.run.smithi099.stdout: "total": 1, 2023-05-31T19:59:36.902 INFO:teuthology.orchestra.run.smithi099.stdout: "complete": 1 2023-05-31T19:59:36.902 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T19:59:36.902 INFO:teuthology.orchestra.run.smithi099.stdout: "current_time": "2023-05-31T19:59:36Z" 2023-05-31T19:59:36.902 INFO:teuthology.orchestra.run.smithi099.stdout:} 2023-05-31T19:59:36.903 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "2e6123d2-d695-4350-9a01-61a455bc782b", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1685562631.516089_59.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-05-31T19:50:31.516089Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1685562970.274572_93.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-05-31T19:56:10.274572Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1685563171.532330_62.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-05-31T19:59:31.532330Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-05-31T19:59:36Z" } 2023-05-31T19:59:36.904 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1685562631.516089_59.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-05-31T19:50:31.516089Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1685562970.274572_93.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-05-31T19:56:10.274572Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1685563171.532330_62.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-05-31T19:59:31.532330Z', 'realm_epoch': 2}}] 2023-05-31T19:59:36.904 DEBUG:rgw_multi.tests:log_status={0: '1_1685562631.516089_59.1', 1: '', 2: '1_1685562970.274572_93.1', 3: '1_1685563171.532330_62.1'} 2023-05-31T19:59:36.905 DEBUG:rgw_multi.tests:sync_status={0: '1_1685562631.516089_59.1', 1: '', 2: '1_1685562970.274572_93.1', 3: '1_1685563171.532330_62.1'} 2023-05-31T19:59:36.905 INFO:rgw_multi.tests:finish meta checkpoint for zone=a2 2023-05-31T19:59:36.906 DEBUG:boto:path=/pklxkd-13/ 2023-05-31T19:59:36.906 DEBUG:boto:auth_path=/pklxkd-13/ 2023-05-31T19:59:36.906 DEBUG:boto:Method: HEAD 2023-05-31T19:59:36.907 DEBUG:boto:Path: /pklxkd-13/ 2023-05-31T19:59:36.907 DEBUG:boto:Data: 2023-05-31T19:59:36.908 DEBUG:boto:Headers: {} 2023-05-31T19:59:36.908 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T19:59:36.908 DEBUG:boto:Port: 8000 2023-05-31T19:59:36.909 DEBUG:boto:Params: {} 2023-05-31T19:59:36.909 DEBUG:boto:Token: None 2023-05-31T19:59:36.909 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 19:59:36 GMT /pklxkd-13/ 2023-05-31T19:59:36.910 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:6uMQhWm14YoXl2bVUvAQ1QXINwA= 2023-05-31T19:59:36.910 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 19:59:36 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:6uMQhWm14YoXl2bVUvAQ1QXINwA=', 'Content-Length': '0'} 2023-05-31T19:59:36.912 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000030940ba9ccc6a2c-006477a728-110e-a1'), ('Content-Length', '0'), ('Date', 'Wed, 31 May 2023 19:59:36 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T19:59:36.913 DEBUG:boto:path=/pklxkd-13/dummy 2023-05-31T19:59:36.914 DEBUG:boto:auth_path=/pklxkd-13/dummy 2023-05-31T19:59:36.914 DEBUG:boto:Method: PUT 2023-05-31T19:59:36.914 DEBUG:boto:Path: /pklxkd-13/dummy 2023-05-31T19:59:36.915 DEBUG:boto:Data: 2023-05-31T19:59:36.915 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': '1B2M2Y8AsgTpgAmY7PhCfg==', 'Content-Length': '0', 'Expect': '100-Continue'} 2023-05-31T19:59:36.915 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T19:59:36.916 DEBUG:boto:Port: 8000 2023-05-31T19:59:36.916 DEBUG:boto:Params: {} 2023-05-31T19:59:36.916 DEBUG:boto:Token: None 2023-05-31T19:59:36.917 DEBUG:boto:StringToSign: PUT 1B2M2Y8AsgTpgAmY7PhCfg== application/octet-stream Wed, 31 May 2023 19:59:36 GMT /pklxkd-13/dummy 2023-05-31T19:59:36.917 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:gGi4HDS2DqkkQIeCd0+UR0S+cgg= 2023-05-31T19:59:36.918 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': '1B2M2Y8AsgTpgAmY7PhCfg==', 'Content-Length': '0', 'Expect': '100-Continue', 'Date': 'Wed, 31 May 2023 19:59:36 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:gGi4HDS2DqkkQIeCd0+UR0S+cgg='} 2023-05-31T19:59:36.937 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('Accept-Ranges', 'bytes'), ('x-amz-version-id', '1KZ1.YSVQS7.XBkKCHg.G4C.eFHwKQn'), ('x-amz-request-id', 'tx000001a73900bc81e2440-006477a728-110e-a1'), ('Date', 'Wed, 31 May 2023 19:59:36 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T19:59:36.937 DEBUG:rgw_multi.tests:bucket checkpoint: source=a1 target=a2 bucket=pklxkd-13 2023-05-31T19:59:36.938 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'pklxkd-13', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-05-31T19:59:36.938 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'pklxkd-13', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-05-31T19:59:36.939 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket pklxkd-13 --source-zone a1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2023-05-31T19:59:36.995 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2023-05-31T19:59:36.996 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2023-05-31T19:59:37.034 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T19:59:37.028+0000 7feb1f882780 1 waiting to reach incremental sync.. 2023-05-31T19:59:42.034 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T19:59:42.029+0000 7feb1f882780 1 waiting to reach incremental sync.. 2023-05-31T19:59:47.036 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T19:59:47.031+0000 7feb1f882780 1 bucket sync caught up with source: 2023-05-31T19:59:47.037 INFO:teuthology.orchestra.run.smithi099.stderr: local status: [, , , , , 00000000002.33.11, , , , , ] 2023-05-31T19:59:47.037 INFO:teuthology.orchestra.run.smithi099.stderr: remote markers: [, , , , , 00000000002.33.11, , , , , ] 2023-05-31T19:59:47.037 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T19:59:47.031+0000 7feb1f882780 0 bucket checkpoint complete 2023-05-31T19:59:47.059 DEBUG:rgw_multi.tests:bucket checkpoint: source=a2 target=a1 bucket=pklxkd-13 2023-05-31T19:59:47.060 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'pklxkd-13', '--source-zone', 'a2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-05-31T19:59:47.060 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'pklxkd-13', '--source-zone', 'a2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-05-31T19:59:47.061 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket pklxkd-13 --source-zone a2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2023-05-31T19:59:47.123 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2023-05-31T19:59:47.123 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2023-05-31T19:59:47.169 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T19:59:47.164+0000 7f2a2733e780 1 waiting to reach incremental sync.. 2023-05-31T19:59:52.170 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T19:59:52.165+0000 7f2a2733e780 1 waiting to reach incremental sync.. 2023-05-31T19:59:57.170 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T19:59:57.165+0000 7f2a2733e780 1 waiting to reach incremental sync.. 2023-05-31T20:00:02.170 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:00:02.164+0000 7f2a2733e780 1 waiting to reach incremental sync.. 2023-05-31T20:00:07.172 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:00:07.166+0000 7f2a2733e780 1 bucket sync caught up with source: 2023-05-31T20:00:07.172 INFO:teuthology.orchestra.run.smithi099.stderr: local status: [, , , , , 00000000002.36.11, , , , , ] 2023-05-31T20:00:07.172 INFO:teuthology.orchestra.run.smithi099.stderr: remote markers: [, , , , , 00000000002.36.11, , , , , ] 2023-05-31T20:00:07.173 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:00:07.166+0000 7f2a2733e780 0 bucket checkpoint complete 2023-05-31T20:00:07.192 INFO:tasks.rgw_multi.zone_rados:comparing bucket=pklxkd-13 zones={a2, a1} 2023-05-31T20:00:07.192 DEBUG:boto:path=/pklxkd-13/ 2023-05-31T20:00:07.193 DEBUG:boto:auth_path=/pklxkd-13/ 2023-05-31T20:00:07.193 DEBUG:boto:Method: HEAD 2023-05-31T20:00:07.193 DEBUG:boto:Path: /pklxkd-13/ 2023-05-31T20:00:07.194 DEBUG:boto:Data: 2023-05-31T20:00:07.194 DEBUG:boto:Headers: {} 2023-05-31T20:00:07.194 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:00:07.195 DEBUG:boto:Port: 8001 2023-05-31T20:00:07.195 DEBUG:boto:Params: {} 2023-05-31T20:00:07.196 DEBUG:boto:Token: None 2023-05-31T20:00:07.196 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:00:07 GMT /pklxkd-13/ 2023-05-31T20:00:07.197 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:ljgRCrbX9BOLkRhZ67n9Rl1KCqM= 2023-05-31T20:00:07.197 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:00:07 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:ljgRCrbX9BOLkRhZ67n9Rl1KCqM=', 'Content-Length': '0'} 2023-05-31T20:00:07.200 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000545558cf0d1ef5ef-006477a747-1112-a2'), ('Content-Length', '0'), ('Date', 'Wed, 31 May 2023 20:00:07 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:00:07.200 DEBUG:boto:path=/pklxkd-13/ 2023-05-31T20:00:07.201 DEBUG:boto:auth_path=/pklxkd-13/ 2023-05-31T20:00:07.201 DEBUG:boto:Method: HEAD 2023-05-31T20:00:07.201 DEBUG:boto:Path: /pklxkd-13/ 2023-05-31T20:00:07.202 DEBUG:boto:Data: 2023-05-31T20:00:07.202 DEBUG:boto:Headers: {} 2023-05-31T20:00:07.202 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:00:07.203 DEBUG:boto:Port: 8000 2023-05-31T20:00:07.203 DEBUG:boto:Params: {} 2023-05-31T20:00:07.203 DEBUG:boto:Token: None 2023-05-31T20:00:07.204 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:00:07 GMT /pklxkd-13/ 2023-05-31T20:00:07.204 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:ljgRCrbX9BOLkRhZ67n9Rl1KCqM= 2023-05-31T20:00:07.205 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:00:07 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:ljgRCrbX9BOLkRhZ67n9Rl1KCqM=', 'Content-Length': '0'} 2023-05-31T20:00:07.207 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000001bafb9056c79aa23-006477a747-110e-a1'), ('Content-Length', '0'), ('Date', 'Wed, 31 May 2023 20:00:07 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:00:07.207 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2023-05-31T20:00:07.208 DEBUG:boto:path=/pklxkd-13/ 2023-05-31T20:00:07.208 DEBUG:boto:auth_path=/pklxkd-13/ 2023-05-31T20:00:07.208 DEBUG:boto:path=/pklxkd-13/?versions&max-keys=999 2023-05-31T20:00:07.209 DEBUG:boto:auth_path=/pklxkd-13/?versions&max-keys=999 2023-05-31T20:00:07.209 DEBUG:boto:Method: GET 2023-05-31T20:00:07.209 DEBUG:boto:Path: /pklxkd-13/?versions&max-keys=999 2023-05-31T20:00:07.210 DEBUG:boto:Data: 2023-05-31T20:00:07.210 DEBUG:boto:Headers: {} 2023-05-31T20:00:07.210 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:00:07.211 DEBUG:boto:Port: 8001 2023-05-31T20:00:07.211 DEBUG:boto:Params: {} 2023-05-31T20:00:07.212 DEBUG:boto:Token: None 2023-05-31T20:00:07.212 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:00:07 GMT /pklxkd-13/?versions 2023-05-31T20:00:07.212 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:K7NwJcNyQbypZNkWw891Ox3pR8A= 2023-05-31T20:00:07.213 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:00:07 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:K7NwJcNyQbypZNkWw891Ox3pR8A=', 'Content-Length': '0'} 2023-05-31T20:00:07.215 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000001ed57b9927f5a826-006477a747-1112-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:00:07 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:00:07.216 DEBUG:boto:b'pklxkd-13999falsedummy1KZ1.YSVQS7.XBkKCHg.G4C.eFHwKQntrue2023-05-31T19:59:36.915Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:00:07.216 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2023-05-31T20:00:07.217 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2023-05-31T20:00:07.217 DEBUG:boto:path=/pklxkd-13/ 2023-05-31T20:00:07.217 DEBUG:boto:auth_path=/pklxkd-13/ 2023-05-31T20:00:07.218 DEBUG:boto:path=/pklxkd-13/?versions&max-keys=999 2023-05-31T20:00:07.218 DEBUG:boto:auth_path=/pklxkd-13/?versions&max-keys=999 2023-05-31T20:00:07.218 DEBUG:boto:Method: GET 2023-05-31T20:00:07.219 DEBUG:boto:Path: /pklxkd-13/?versions&max-keys=999 2023-05-31T20:00:07.219 DEBUG:boto:Data: 2023-05-31T20:00:07.219 DEBUG:boto:Headers: {} 2023-05-31T20:00:07.220 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:00:07.220 DEBUG:boto:Port: 8000 2023-05-31T20:00:07.220 DEBUG:boto:Params: {} 2023-05-31T20:00:07.221 DEBUG:boto:Token: None 2023-05-31T20:00:07.221 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:00:07 GMT /pklxkd-13/?versions 2023-05-31T20:00:07.222 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:K7NwJcNyQbypZNkWw891Ox3pR8A= 2023-05-31T20:00:07.222 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:00:07 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:K7NwJcNyQbypZNkWw891Ox3pR8A=', 'Content-Length': '0'} 2023-05-31T20:00:07.224 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000009062babcc21b1147-006477a747-110e-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:00:07 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:00:07.225 DEBUG:boto:b'pklxkd-13999falsedummy1KZ1.YSVQS7.XBkKCHg.G4C.eFHwKQntrue2023-05-31T19:59:36.915Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:00:07.226 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2023-05-31T20:00:07.226 DEBUG:boto:path=/pklxkd-13/ 2023-05-31T20:00:07.226 DEBUG:boto:auth_path=/pklxkd-13/ 2023-05-31T20:00:07.227 DEBUG:boto:path=/pklxkd-13/?versions&max-keys=999 2023-05-31T20:00:07.227 DEBUG:boto:auth_path=/pklxkd-13/?versions&max-keys=999 2023-05-31T20:00:07.227 DEBUG:boto:Method: GET 2023-05-31T20:00:07.228 DEBUG:boto:Path: /pklxkd-13/?versions&max-keys=999 2023-05-31T20:00:07.228 DEBUG:boto:Data: 2023-05-31T20:00:07.228 DEBUG:boto:Headers: {} 2023-05-31T20:00:07.229 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:00:07.229 DEBUG:boto:Port: 8001 2023-05-31T20:00:07.229 DEBUG:boto:Params: {} 2023-05-31T20:00:07.230 DEBUG:boto:Token: None 2023-05-31T20:00:07.230 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:00:07 GMT /pklxkd-13/?versions 2023-05-31T20:00:07.231 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:K7NwJcNyQbypZNkWw891Ox3pR8A= 2023-05-31T20:00:07.231 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:00:07 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:K7NwJcNyQbypZNkWw891Ox3pR8A=', 'Content-Length': '0'} 2023-05-31T20:00:07.234 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000340357e6bef3566c-006477a747-1112-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:00:07 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:00:07.234 DEBUG:boto:b'pklxkd-13999falsedummy1KZ1.YSVQS7.XBkKCHg.G4C.eFHwKQntrue2023-05-31T19:59:36.915Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:00:07.235 DEBUG:boto:path=/pklxkd-13/ 2023-05-31T20:00:07.235 DEBUG:boto:auth_path=/pklxkd-13/ 2023-05-31T20:00:07.235 DEBUG:boto:path=/pklxkd-13/?versions&max-keys=999 2023-05-31T20:00:07.236 DEBUG:boto:auth_path=/pklxkd-13/?versions&max-keys=999 2023-05-31T20:00:07.236 DEBUG:boto:Method: GET 2023-05-31T20:00:07.237 DEBUG:boto:Path: /pklxkd-13/?versions&max-keys=999 2023-05-31T20:00:07.237 DEBUG:boto:Data: 2023-05-31T20:00:07.237 DEBUG:boto:Headers: {} 2023-05-31T20:00:07.238 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:00:07.238 DEBUG:boto:Port: 8000 2023-05-31T20:00:07.238 DEBUG:boto:Params: {} 2023-05-31T20:00:07.239 DEBUG:boto:Token: None 2023-05-31T20:00:07.239 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:00:07 GMT /pklxkd-13/?versions 2023-05-31T20:00:07.239 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:K7NwJcNyQbypZNkWw891Ox3pR8A= 2023-05-31T20:00:07.240 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:00:07 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:K7NwJcNyQbypZNkWw891Ox3pR8A=', 'Content-Length': '0'} 2023-05-31T20:00:07.242 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000154c68acc1a72081-006477a747-110e-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:00:07 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:00:07.243 DEBUG:boto:b'pklxkd-13999falsedummy1KZ1.YSVQS7.XBkKCHg.G4C.eFHwKQntrue2023-05-31T19:59:36.915Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:00:07.243 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2023-05-31T20:00:07.244 DEBUG:boto:path=/pklxkd-13/dummy 2023-05-31T20:00:07.244 DEBUG:boto:auth_path=/pklxkd-13/dummy 2023-05-31T20:00:07.244 DEBUG:boto:path=/pklxkd-13/dummy?versionId=1KZ1.YSVQS7.XBkKCHg.G4C.eFHwKQn 2023-05-31T20:00:07.245 DEBUG:boto:auth_path=/pklxkd-13/dummy?versionId=1KZ1.YSVQS7.XBkKCHg.G4C.eFHwKQn 2023-05-31T20:00:07.245 DEBUG:boto:Method: GET 2023-05-31T20:00:07.245 DEBUG:boto:Path: /pklxkd-13/dummy?versionId=1KZ1.YSVQS7.XBkKCHg.G4C.eFHwKQn 2023-05-31T20:00:07.246 DEBUG:boto:Data: 2023-05-31T20:00:07.246 DEBUG:boto:Headers: {} 2023-05-31T20:00:07.246 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:00:07.247 DEBUG:boto:Port: 8001 2023-05-31T20:00:07.247 DEBUG:boto:Params: {} 2023-05-31T20:00:07.248 DEBUG:boto:Token: None 2023-05-31T20:00:07.248 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:00:07 GMT /pklxkd-13/dummy?versionId=1KZ1.YSVQS7.XBkKCHg.G4C.eFHwKQn 2023-05-31T20:00:07.248 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:YiALWyK9+v8DPh8QV9PA5PGz1O4= 2023-05-31T20:00:07.249 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:00:07 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:YiALWyK9+v8DPh8QV9PA5PGz1O4=', 'Content-Length': '0'} 2023-05-31T20:00:07.251 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 19:59:36 GMT'), ('x-amz-version-id', '1KZ1.YSVQS7.XBkKCHg.G4C.eFHwKQn'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000002efc9b56d8831fa4-006477a747-1112-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:00:07 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:00:07.251 DEBUG:boto:path=/pklxkd-13/dummy 2023-05-31T20:00:07.252 DEBUG:boto:auth_path=/pklxkd-13/dummy 2023-05-31T20:00:07.252 DEBUG:boto:path=/pklxkd-13/dummy?versionId=1KZ1.YSVQS7.XBkKCHg.G4C.eFHwKQn 2023-05-31T20:00:07.252 DEBUG:boto:auth_path=/pklxkd-13/dummy?versionId=1KZ1.YSVQS7.XBkKCHg.G4C.eFHwKQn 2023-05-31T20:00:07.253 DEBUG:boto:Method: GET 2023-05-31T20:00:07.253 DEBUG:boto:Path: /pklxkd-13/dummy?versionId=1KZ1.YSVQS7.XBkKCHg.G4C.eFHwKQn 2023-05-31T20:00:07.254 DEBUG:boto:Data: 2023-05-31T20:00:07.254 DEBUG:boto:Headers: {} 2023-05-31T20:00:07.254 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:00:07.255 DEBUG:boto:Port: 8000 2023-05-31T20:00:07.255 DEBUG:boto:Params: {} 2023-05-31T20:00:07.255 DEBUG:boto:Token: None 2023-05-31T20:00:07.256 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:00:07 GMT /pklxkd-13/dummy?versionId=1KZ1.YSVQS7.XBkKCHg.G4C.eFHwKQn 2023-05-31T20:00:07.256 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:YiALWyK9+v8DPh8QV9PA5PGz1O4= 2023-05-31T20:00:07.257 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:00:07 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:YiALWyK9+v8DPh8QV9PA5PGz1O4=', 'Content-Length': '0'} 2023-05-31T20:00:07.259 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 19:59:36 GMT'), ('x-amz-version-id', '1KZ1.YSVQS7.XBkKCHg.G4C.eFHwKQn'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000b7893e55c894d04f-006477a747-110e-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:00:07 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:00:07.259 DEBUG:boto:path=/pklxkd-13/dummy 2023-05-31T20:00:07.260 DEBUG:boto:auth_path=/pklxkd-13/dummy 2023-05-31T20:00:07.260 DEBUG:boto:path=/pklxkd-13/dummy?versionId=1KZ1.YSVQS7.XBkKCHg.G4C.eFHwKQn 2023-05-31T20:00:07.260 DEBUG:boto:auth_path=/pklxkd-13/dummy?versionId=1KZ1.YSVQS7.XBkKCHg.G4C.eFHwKQn 2023-05-31T20:00:07.261 DEBUG:boto:Method: HEAD 2023-05-31T20:00:07.261 DEBUG:boto:Path: /pklxkd-13/dummy?versionId=1KZ1.YSVQS7.XBkKCHg.G4C.eFHwKQn 2023-05-31T20:00:07.261 DEBUG:boto:Data: 2023-05-31T20:00:07.262 DEBUG:boto:Headers: {} 2023-05-31T20:00:07.262 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:00:07.262 DEBUG:boto:Port: 8001 2023-05-31T20:00:07.263 DEBUG:boto:Params: {} 2023-05-31T20:00:07.263 DEBUG:boto:Token: None 2023-05-31T20:00:07.264 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:00:07 GMT /pklxkd-13/dummy?versionId=1KZ1.YSVQS7.XBkKCHg.G4C.eFHwKQn 2023-05-31T20:00:07.264 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:wyVQCl2t7BJSZz4g79RyxAAZO7U= 2023-05-31T20:00:07.264 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:00:07 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:wyVQCl2t7BJSZz4g79RyxAAZO7U=', 'Content-Length': '0'} 2023-05-31T20:00:07.266 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 19:59:36 GMT'), ('x-amz-version-id', '1KZ1.YSVQS7.XBkKCHg.G4C.eFHwKQn'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000004a5244ec150e030c-006477a747-1112-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:00:07 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:00:07.267 DEBUG:boto:path=/pklxkd-13/dummy 2023-05-31T20:00:07.267 DEBUG:boto:auth_path=/pklxkd-13/dummy 2023-05-31T20:00:07.268 DEBUG:boto:path=/pklxkd-13/dummy?versionId=1KZ1.YSVQS7.XBkKCHg.G4C.eFHwKQn 2023-05-31T20:00:07.268 DEBUG:boto:auth_path=/pklxkd-13/dummy?versionId=1KZ1.YSVQS7.XBkKCHg.G4C.eFHwKQn 2023-05-31T20:00:07.268 DEBUG:boto:Method: HEAD 2023-05-31T20:00:07.269 DEBUG:boto:Path: /pklxkd-13/dummy?versionId=1KZ1.YSVQS7.XBkKCHg.G4C.eFHwKQn 2023-05-31T20:00:07.269 DEBUG:boto:Data: 2023-05-31T20:00:07.269 DEBUG:boto:Headers: {} 2023-05-31T20:00:07.270 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:00:07.270 DEBUG:boto:Port: 8000 2023-05-31T20:00:07.270 DEBUG:boto:Params: {} 2023-05-31T20:00:07.271 DEBUG:boto:Token: None 2023-05-31T20:00:07.271 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:00:07 GMT /pklxkd-13/dummy?versionId=1KZ1.YSVQS7.XBkKCHg.G4C.eFHwKQn 2023-05-31T20:00:07.272 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:wyVQCl2t7BJSZz4g79RyxAAZO7U= 2023-05-31T20:00:07.272 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:00:07 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:wyVQCl2t7BJSZz4g79RyxAAZO7U=', 'Content-Length': '0'} 2023-05-31T20:00:07.274 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 19:59:36 GMT'), ('x-amz-version-id', '1KZ1.YSVQS7.XBkKCHg.G4C.eFHwKQn'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000a8abb5d49fb10720-006477a747-110e-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:00:07 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:00:07.274 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2023-05-31T20:00:07.275 DEBUG:boto:path=/pklxkd-13/dummy 2023-05-31T20:00:07.275 DEBUG:boto:auth_path=/pklxkd-13/dummy 2023-05-31T20:00:07.276 DEBUG:boto:path=/pklxkd-13/dummy?versionId=1KZ1.YSVQS7.XBkKCHg.G4C.eFHwKQn 2023-05-31T20:00:07.276 DEBUG:boto:auth_path=/pklxkd-13/dummy?versionId=1KZ1.YSVQS7.XBkKCHg.G4C.eFHwKQn 2023-05-31T20:00:07.276 DEBUG:boto:Method: GET 2023-05-31T20:00:07.277 DEBUG:boto:Path: /pklxkd-13/dummy?versionId=1KZ1.YSVQS7.XBkKCHg.G4C.eFHwKQn 2023-05-31T20:00:07.277 DEBUG:boto:Data: 2023-05-31T20:00:07.277 DEBUG:boto:Headers: {} 2023-05-31T20:00:07.278 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:00:07.278 DEBUG:boto:Port: 8001 2023-05-31T20:00:07.278 DEBUG:boto:Params: {} 2023-05-31T20:00:07.279 DEBUG:boto:Token: None 2023-05-31T20:00:07.279 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:00:07 GMT /pklxkd-13/dummy?versionId=1KZ1.YSVQS7.XBkKCHg.G4C.eFHwKQn 2023-05-31T20:00:07.280 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:YiALWyK9+v8DPh8QV9PA5PGz1O4= 2023-05-31T20:00:07.280 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:00:07 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:YiALWyK9+v8DPh8QV9PA5PGz1O4=', 'Content-Length': '0'} 2023-05-31T20:00:07.282 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 19:59:36 GMT'), ('x-amz-version-id', '1KZ1.YSVQS7.XBkKCHg.G4C.eFHwKQn'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx0000030afa21b72b9a3af-006477a747-1112-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:00:07 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:00:07.283 DEBUG:boto:path=/pklxkd-13/dummy 2023-05-31T20:00:07.284 DEBUG:boto:auth_path=/pklxkd-13/dummy 2023-05-31T20:00:07.284 DEBUG:boto:path=/pklxkd-13/dummy?versionId=1KZ1.YSVQS7.XBkKCHg.G4C.eFHwKQn 2023-05-31T20:00:07.285 DEBUG:boto:auth_path=/pklxkd-13/dummy?versionId=1KZ1.YSVQS7.XBkKCHg.G4C.eFHwKQn 2023-05-31T20:00:07.285 DEBUG:boto:Method: GET 2023-05-31T20:00:07.285 DEBUG:boto:Path: /pklxkd-13/dummy?versionId=1KZ1.YSVQS7.XBkKCHg.G4C.eFHwKQn 2023-05-31T20:00:07.286 DEBUG:boto:Data: 2023-05-31T20:00:07.286 DEBUG:boto:Headers: {} 2023-05-31T20:00:07.286 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:00:07.287 DEBUG:boto:Port: 8000 2023-05-31T20:00:07.287 DEBUG:boto:Params: {} 2023-05-31T20:00:07.287 DEBUG:boto:Token: None 2023-05-31T20:00:07.288 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:00:07 GMT /pklxkd-13/dummy?versionId=1KZ1.YSVQS7.XBkKCHg.G4C.eFHwKQn 2023-05-31T20:00:07.288 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:YiALWyK9+v8DPh8QV9PA5PGz1O4= 2023-05-31T20:00:07.289 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:00:07 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:YiALWyK9+v8DPh8QV9PA5PGz1O4=', 'Content-Length': '0'} 2023-05-31T20:00:07.290 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 19:59:36 GMT'), ('x-amz-version-id', '1KZ1.YSVQS7.XBkKCHg.G4C.eFHwKQn'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx0000053df8c9a2392cf24-006477a747-110e-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:00:07 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:00:07.291 DEBUG:boto:path=/pklxkd-13/ 2023-05-31T20:00:07.292 DEBUG:boto:auth_path=/pklxkd-13/ 2023-05-31T20:00:07.292 DEBUG:boto:path=/pklxkd-13/?prefix=dummy 2023-05-31T20:00:07.292 DEBUG:boto:auth_path=/pklxkd-13/?prefix=dummy 2023-05-31T20:00:07.293 DEBUG:boto:Method: GET 2023-05-31T20:00:07.293 DEBUG:boto:Path: /pklxkd-13/?prefix=dummy 2023-05-31T20:00:07.293 DEBUG:boto:Data: 2023-05-31T20:00:07.294 DEBUG:boto:Headers: {} 2023-05-31T20:00:07.294 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:00:07.294 DEBUG:boto:Port: 8001 2023-05-31T20:00:07.295 DEBUG:boto:Params: {} 2023-05-31T20:00:07.295 DEBUG:boto:Token: None 2023-05-31T20:00:07.296 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:00:07 GMT /pklxkd-13/ 2023-05-31T20:00:07.296 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:zuU924DxrbUQWVesaPSrKzJh+Nc= 2023-05-31T20:00:07.296 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:00:07 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:zuU924DxrbUQWVesaPSrKzJh+Nc=', 'Content-Length': '0'} 2023-05-31T20:00:07.299 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000186af45349f1b555-006477a747-1112-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:00:07 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:00:07.299 DEBUG:boto:b'pklxkd-13dummy1000falsedummy2023-05-31T19:59:36.915Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:00:07.300 DEBUG:boto:path=/pklxkd-13/ 2023-05-31T20:00:07.301 DEBUG:boto:auth_path=/pklxkd-13/ 2023-05-31T20:00:07.301 DEBUG:boto:path=/pklxkd-13/?prefix=dummy 2023-05-31T20:00:07.301 DEBUG:boto:auth_path=/pklxkd-13/?prefix=dummy 2023-05-31T20:00:07.302 DEBUG:boto:Method: GET 2023-05-31T20:00:07.302 DEBUG:boto:Path: /pklxkd-13/?prefix=dummy 2023-05-31T20:00:07.302 DEBUG:boto:Data: 2023-05-31T20:00:07.303 DEBUG:boto:Headers: {} 2023-05-31T20:00:07.303 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:00:07.303 DEBUG:boto:Port: 8000 2023-05-31T20:00:07.304 DEBUG:boto:Params: {} 2023-05-31T20:00:07.304 DEBUG:boto:Token: None 2023-05-31T20:00:07.305 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:00:07 GMT /pklxkd-13/ 2023-05-31T20:00:07.305 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:zuU924DxrbUQWVesaPSrKzJh+Nc= 2023-05-31T20:00:07.305 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:00:07 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:zuU924DxrbUQWVesaPSrKzJh+Nc=', 'Content-Length': '0'} 2023-05-31T20:00:07.308 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000004f4b949a18e51a72-006477a747-110e-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:00:07 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:00:07.309 DEBUG:boto:b'pklxkd-13dummy1000falsedummy2023-05-31T19:59:36.915Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:00:07.309 DEBUG:boto:path=/pklxkd-13/dummy 2023-05-31T20:00:07.310 DEBUG:boto:auth_path=/pklxkd-13/dummy 2023-05-31T20:00:07.310 DEBUG:boto:Method: HEAD 2023-05-31T20:00:07.310 DEBUG:boto:Path: /pklxkd-13/dummy 2023-05-31T20:00:07.311 DEBUG:boto:Data: 2023-05-31T20:00:07.311 DEBUG:boto:Headers: {} 2023-05-31T20:00:07.311 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:00:07.312 DEBUG:boto:Port: 8001 2023-05-31T20:00:07.312 DEBUG:boto:Params: {} 2023-05-31T20:00:07.312 DEBUG:boto:Token: None 2023-05-31T20:00:07.313 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:00:07 GMT /pklxkd-13/dummy 2023-05-31T20:00:07.313 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:+nGebq1OGhapcaSI2T0JE1s6MdM= 2023-05-31T20:00:07.314 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:00:07 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:+nGebq1OGhapcaSI2T0JE1s6MdM=', 'Content-Length': '0'} 2023-05-31T20:00:07.316 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 19:59:36 GMT'), ('x-amz-version-id', '1KZ1.YSVQS7.XBkKCHg.G4C.eFHwKQn'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000005d51df2b5edf2b64-006477a747-1112-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:00:07 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:00:07.316 DEBUG:boto:path=/pklxkd-13/dummy 2023-05-31T20:00:07.317 DEBUG:boto:auth_path=/pklxkd-13/dummy 2023-05-31T20:00:07.317 DEBUG:boto:Method: HEAD 2023-05-31T20:00:07.317 DEBUG:boto:Path: /pklxkd-13/dummy 2023-05-31T20:00:07.318 DEBUG:boto:Data: 2023-05-31T20:00:07.318 DEBUG:boto:Headers: {} 2023-05-31T20:00:07.318 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:00:07.319 DEBUG:boto:Port: 8000 2023-05-31T20:00:07.319 DEBUG:boto:Params: {} 2023-05-31T20:00:07.319 DEBUG:boto:Token: None 2023-05-31T20:00:07.320 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:00:07 GMT /pklxkd-13/dummy 2023-05-31T20:00:07.320 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:+nGebq1OGhapcaSI2T0JE1s6MdM= 2023-05-31T20:00:07.321 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:00:07 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:+nGebq1OGhapcaSI2T0JE1s6MdM=', 'Content-Length': '0'} 2023-05-31T20:00:07.323 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 19:59:36 GMT'), ('x-amz-version-id', '1KZ1.YSVQS7.XBkKCHg.G4C.eFHwKQn'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000004c92b31e341175cf-006477a747-110e-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:00:07 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:00:07.324 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2023-05-31T20:00:07.324 DEBUG:boto:path=/pklxkd-13/dummy 2023-05-31T20:00:07.324 DEBUG:boto:auth_path=/pklxkd-13/dummy 2023-05-31T20:00:07.325 DEBUG:boto:path=/pklxkd-13/dummy?versionId=1KZ1.YSVQS7.XBkKCHg.G4C.eFHwKQn 2023-05-31T20:00:07.325 DEBUG:boto:auth_path=/pklxkd-13/dummy?versionId=1KZ1.YSVQS7.XBkKCHg.G4C.eFHwKQn 2023-05-31T20:00:07.325 DEBUG:boto:Method: GET 2023-05-31T20:00:07.326 DEBUG:boto:Path: /pklxkd-13/dummy?versionId=1KZ1.YSVQS7.XBkKCHg.G4C.eFHwKQn 2023-05-31T20:00:07.326 DEBUG:boto:Data: 2023-05-31T20:00:07.326 DEBUG:boto:Headers: {} 2023-05-31T20:00:07.327 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:00:07.327 DEBUG:boto:Port: 8001 2023-05-31T20:00:07.328 DEBUG:boto:Params: {} 2023-05-31T20:00:07.328 DEBUG:boto:Token: None 2023-05-31T20:00:07.328 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:00:07 GMT /pklxkd-13/dummy?versionId=1KZ1.YSVQS7.XBkKCHg.G4C.eFHwKQn 2023-05-31T20:00:07.329 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:YiALWyK9+v8DPh8QV9PA5PGz1O4= 2023-05-31T20:00:07.329 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:00:07 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:YiALWyK9+v8DPh8QV9PA5PGz1O4=', 'Content-Length': '0'} 2023-05-31T20:00:07.331 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 19:59:36 GMT'), ('x-amz-version-id', '1KZ1.YSVQS7.XBkKCHg.G4C.eFHwKQn'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx0000063f44063e18f3607-006477a747-1112-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:00:07 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:00:07.332 DEBUG:boto:path=/pklxkd-13/dummy 2023-05-31T20:00:07.332 DEBUG:boto:auth_path=/pklxkd-13/dummy 2023-05-31T20:00:07.332 DEBUG:boto:path=/pklxkd-13/dummy?versionId=1KZ1.YSVQS7.XBkKCHg.G4C.eFHwKQn 2023-05-31T20:00:07.333 DEBUG:boto:auth_path=/pklxkd-13/dummy?versionId=1KZ1.YSVQS7.XBkKCHg.G4C.eFHwKQn 2023-05-31T20:00:07.333 DEBUG:boto:Method: GET 2023-05-31T20:00:07.334 DEBUG:boto:Path: /pklxkd-13/dummy?versionId=1KZ1.YSVQS7.XBkKCHg.G4C.eFHwKQn 2023-05-31T20:00:07.334 DEBUG:boto:Data: 2023-05-31T20:00:07.334 DEBUG:boto:Headers: {} 2023-05-31T20:00:07.335 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:00:07.335 DEBUG:boto:Port: 8000 2023-05-31T20:00:07.335 DEBUG:boto:Params: {} 2023-05-31T20:00:07.336 DEBUG:boto:Token: None 2023-05-31T20:00:07.336 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:00:07 GMT /pklxkd-13/dummy?versionId=1KZ1.YSVQS7.XBkKCHg.G4C.eFHwKQn 2023-05-31T20:00:07.337 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:YiALWyK9+v8DPh8QV9PA5PGz1O4= 2023-05-31T20:00:07.337 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:00:07 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:YiALWyK9+v8DPh8QV9PA5PGz1O4=', 'Content-Length': '0'} 2023-05-31T20:00:07.339 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 19:59:36 GMT'), ('x-amz-version-id', '1KZ1.YSVQS7.XBkKCHg.G4C.eFHwKQn'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000d64cf2eed291f264-006477a747-110e-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:00:07 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:00:07.339 DEBUG:boto:path=/pklxkd-13/ 2023-05-31T20:00:07.340 DEBUG:boto:auth_path=/pklxkd-13/ 2023-05-31T20:00:07.340 DEBUG:boto:path=/pklxkd-13/?prefix=dummy 2023-05-31T20:00:07.340 DEBUG:boto:auth_path=/pklxkd-13/?prefix=dummy 2023-05-31T20:00:07.341 DEBUG:boto:Method: GET 2023-05-31T20:00:07.341 DEBUG:boto:Path: /pklxkd-13/?prefix=dummy 2023-05-31T20:00:07.342 DEBUG:boto:Data: 2023-05-31T20:00:07.342 DEBUG:boto:Headers: {} 2023-05-31T20:00:07.342 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:00:07.343 DEBUG:boto:Port: 8001 2023-05-31T20:00:07.343 DEBUG:boto:Params: {} 2023-05-31T20:00:07.343 DEBUG:boto:Token: None 2023-05-31T20:00:07.344 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:00:07 GMT /pklxkd-13/ 2023-05-31T20:00:07.344 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:zuU924DxrbUQWVesaPSrKzJh+Nc= 2023-05-31T20:00:07.345 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:00:07 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:zuU924DxrbUQWVesaPSrKzJh+Nc=', 'Content-Length': '0'} 2023-05-31T20:00:07.347 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000120710305353d3a9-006477a747-1112-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:00:07 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:00:07.347 DEBUG:boto:b'pklxkd-13dummy1000falsedummy2023-05-31T19:59:36.915Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:00:07.348 DEBUG:boto:path=/pklxkd-13/ 2023-05-31T20:00:07.349 DEBUG:boto:auth_path=/pklxkd-13/ 2023-05-31T20:00:07.349 DEBUG:boto:path=/pklxkd-13/?prefix=dummy 2023-05-31T20:00:07.349 DEBUG:boto:auth_path=/pklxkd-13/?prefix=dummy 2023-05-31T20:00:07.350 DEBUG:boto:Method: GET 2023-05-31T20:00:07.350 DEBUG:boto:Path: /pklxkd-13/?prefix=dummy 2023-05-31T20:00:07.350 DEBUG:boto:Data: 2023-05-31T20:00:07.351 DEBUG:boto:Headers: {} 2023-05-31T20:00:07.351 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:00:07.351 DEBUG:boto:Port: 8000 2023-05-31T20:00:07.352 DEBUG:boto:Params: {} 2023-05-31T20:00:07.352 DEBUG:boto:Token: None 2023-05-31T20:00:07.353 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:00:07 GMT /pklxkd-13/ 2023-05-31T20:00:07.353 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:zuU924DxrbUQWVesaPSrKzJh+Nc= 2023-05-31T20:00:07.353 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:00:07 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:zuU924DxrbUQWVesaPSrKzJh+Nc=', 'Content-Length': '0'} 2023-05-31T20:00:07.356 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000008bb16bee2b8657a0-006477a747-110e-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:00:07 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:00:07.396 DEBUG:boto:b'pklxkd-13dummy1000falsedummy2023-05-31T19:59:36.915Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:00:07.397 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=pklxkd-13 zones={a2, a1} 2023-05-31T20:00:07.397 DEBUG:boto:path=/pklxkd-13/ 2023-05-31T20:00:07.398 DEBUG:boto:auth_path=/pklxkd-13/ 2023-05-31T20:00:07.398 DEBUG:boto:Method: HEAD 2023-05-31T20:00:07.398 DEBUG:boto:Path: /pklxkd-13/ 2023-05-31T20:00:07.399 DEBUG:boto:Data: 2023-05-31T20:00:07.399 DEBUG:boto:Headers: {} 2023-05-31T20:00:07.399 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:00:07.400 DEBUG:boto:Port: 8000 2023-05-31T20:00:07.400 DEBUG:boto:Params: {} 2023-05-31T20:00:07.400 DEBUG:boto:Token: None 2023-05-31T20:00:07.401 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:00:07 GMT /pklxkd-13/ 2023-05-31T20:00:07.401 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:ljgRCrbX9BOLkRhZ67n9Rl1KCqM= 2023-05-31T20:00:07.402 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:00:07 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:ljgRCrbX9BOLkRhZ67n9Rl1KCqM=', 'Content-Length': '0'} 2023-05-31T20:00:07.404 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000b6431f09084b5bbe-006477a747-110e-a1'), ('Content-Length', '0'), ('Date', 'Wed, 31 May 2023 20:00:07 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:00:07.405 DEBUG:boto:path=/pklxkd-13/obj 2023-05-31T20:00:07.405 DEBUG:boto:auth_path=/pklxkd-13/obj 2023-05-31T20:00:07.405 DEBUG:boto:Method: PUT 2023-05-31T20:00:07.406 DEBUG:boto:Path: /pklxkd-13/obj 2023-05-31T20:00:07.406 DEBUG:boto:Data: 2023-05-31T20:00:07.406 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'lmY06/L8E1cH1nU2kr9LHg==', 'Content-Length': '8', 'Expect': '100-Continue'} 2023-05-31T20:00:07.407 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:00:07.407 DEBUG:boto:Port: 8000 2023-05-31T20:00:07.407 DEBUG:boto:Params: {} 2023-05-31T20:00:07.408 DEBUG:boto:Token: None 2023-05-31T20:00:07.408 DEBUG:boto:StringToSign: PUT lmY06/L8E1cH1nU2kr9LHg== application/octet-stream Wed, 31 May 2023 20:00:07 GMT /pklxkd-13/obj 2023-05-31T20:00:07.408 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:oQTZSVNviz1HSJVc+j82uk5C1P4= 2023-05-31T20:00:07.409 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'lmY06/L8E1cH1nU2kr9LHg==', 'Content-Length': '8', 'Expect': '100-Continue', 'Date': 'Wed, 31 May 2023 20:00:07 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:oQTZSVNviz1HSJVc+j82uk5C1P4='} 2023-05-31T20:00:07.426 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('Accept-Ranges', 'bytes'), ('x-amz-version-id', '0DuL1bo3E1FXXi7mJ3oxDT6slXlrtRA'), ('x-amz-request-id', 'tx000009908c534b97588d6-006477a747-110e-a1'), ('Date', 'Wed, 31 May 2023 20:00:07 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:00:07.426 DEBUG:rgw_multi.tests:zone=a1 version=0DuL1bo3E1FXXi7mJ3oxDT6slXlrtRA 2023-05-31T20:00:07.427 DEBUG:boto:path=/pklxkd-13/ 2023-05-31T20:00:07.427 DEBUG:boto:auth_path=/pklxkd-13/ 2023-05-31T20:00:07.427 DEBUG:boto:Method: HEAD 2023-05-31T20:00:07.428 DEBUG:boto:Path: /pklxkd-13/ 2023-05-31T20:00:07.428 DEBUG:boto:Data: 2023-05-31T20:00:07.428 DEBUG:boto:Headers: {} 2023-05-31T20:00:07.429 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:00:07.429 DEBUG:boto:Port: 8001 2023-05-31T20:00:07.429 DEBUG:boto:Params: {} 2023-05-31T20:00:07.430 DEBUG:boto:Token: None 2023-05-31T20:00:07.430 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:00:07 GMT /pklxkd-13/ 2023-05-31T20:00:07.430 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:ljgRCrbX9BOLkRhZ67n9Rl1KCqM= 2023-05-31T20:00:07.431 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:00:07 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:ljgRCrbX9BOLkRhZ67n9Rl1KCqM=', 'Content-Length': '0'} 2023-05-31T20:00:07.433 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000543cc3e0f1ce1f0a-006477a747-1112-a2'), ('Content-Length', '0'), ('Date', 'Wed, 31 May 2023 20:00:07 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:00:07.434 DEBUG:boto:path=/pklxkd-13/obj 2023-05-31T20:00:07.434 DEBUG:boto:auth_path=/pklxkd-13/obj 2023-05-31T20:00:07.434 DEBUG:boto:Method: PUT 2023-05-31T20:00:07.435 DEBUG:boto:Path: /pklxkd-13/obj 2023-05-31T20:00:07.435 DEBUG:boto:Data: 2023-05-31T20:00:07.435 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'lmY06/L8E1cH1nU2kr9LHg==', 'Content-Length': '8', 'Expect': '100-Continue'} 2023-05-31T20:00:07.436 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:00:07.436 DEBUG:boto:Port: 8001 2023-05-31T20:00:07.436 DEBUG:boto:Params: {} 2023-05-31T20:00:07.437 DEBUG:boto:Token: None 2023-05-31T20:00:07.437 DEBUG:boto:StringToSign: PUT lmY06/L8E1cH1nU2kr9LHg== application/octet-stream Wed, 31 May 2023 20:00:07 GMT /pklxkd-13/obj 2023-05-31T20:00:07.438 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:oQTZSVNviz1HSJVc+j82uk5C1P4= 2023-05-31T20:00:07.438 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'lmY06/L8E1cH1nU2kr9LHg==', 'Content-Length': '8', 'Expect': '100-Continue', 'Date': 'Wed, 31 May 2023 20:00:07 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:oQTZSVNviz1HSJVc+j82uk5C1P4='} 2023-05-31T20:00:07.480 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('Accept-Ranges', 'bytes'), ('x-amz-version-id', 'pm7-Sh2bsjDbtvOCwujAMZA4JCZRKlh'), ('x-amz-request-id', 'tx00000f856127c545ade5a-006477a747-1112-a2'), ('Date', 'Wed, 31 May 2023 20:00:07 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:00:07.481 DEBUG:rgw_multi.tests:zone=a2 version=pm7-Sh2bsjDbtvOCwujAMZA4JCZRKlh 2023-05-31T20:00:07.481 DEBUG:boto:path=/pklxkd-13/ 2023-05-31T20:00:07.482 DEBUG:boto:auth_path=/pklxkd-13/ 2023-05-31T20:00:07.482 DEBUG:boto:Method: HEAD 2023-05-31T20:00:07.482 DEBUG:boto:Path: /pklxkd-13/ 2023-05-31T20:00:07.483 DEBUG:boto:Data: 2023-05-31T20:00:07.483 DEBUG:boto:Headers: {} 2023-05-31T20:00:07.483 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:00:07.484 DEBUG:boto:Port: 8000 2023-05-31T20:00:07.484 DEBUG:boto:Params: {} 2023-05-31T20:00:07.484 DEBUG:boto:Token: None 2023-05-31T20:00:07.485 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:00:07 GMT /pklxkd-13/ 2023-05-31T20:00:07.485 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:ljgRCrbX9BOLkRhZ67n9Rl1KCqM= 2023-05-31T20:00:07.486 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:00:07 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:ljgRCrbX9BOLkRhZ67n9Rl1KCqM=', 'Content-Length': '0'} 2023-05-31T20:00:07.488 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '2'), ('X-RGW-Bytes-Used', '8'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000006e1ac48d0652b0f0-006477a747-110e-a1'), ('Content-Length', '0'), ('Date', 'Wed, 31 May 2023 20:00:07 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:00:07.489 DEBUG:boto:path=/pklxkd-13/obj 2023-05-31T20:00:07.489 DEBUG:boto:auth_path=/pklxkd-13/obj 2023-05-31T20:00:07.489 DEBUG:boto:Method: PUT 2023-05-31T20:00:07.490 DEBUG:boto:Path: /pklxkd-13/obj 2023-05-31T20:00:07.490 DEBUG:boto:Data: 2023-05-31T20:00:07.490 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'lmY06/L8E1cH1nU2kr9LHg==', 'Content-Length': '8', 'Expect': '100-Continue'} 2023-05-31T20:00:07.491 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:00:07.491 DEBUG:boto:Port: 8000 2023-05-31T20:00:07.491 DEBUG:boto:Params: {} 2023-05-31T20:00:07.492 DEBUG:boto:Token: None 2023-05-31T20:00:07.492 DEBUG:boto:StringToSign: PUT lmY06/L8E1cH1nU2kr9LHg== application/octet-stream Wed, 31 May 2023 20:00:07 GMT /pklxkd-13/obj 2023-05-31T20:00:07.492 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:oQTZSVNviz1HSJVc+j82uk5C1P4= 2023-05-31T20:00:07.493 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'lmY06/L8E1cH1nU2kr9LHg==', 'Content-Length': '8', 'Expect': '100-Continue', 'Date': 'Wed, 31 May 2023 20:00:07 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:oQTZSVNviz1HSJVc+j82uk5C1P4='} 2023-05-31T20:00:07.503 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('Accept-Ranges', 'bytes'), ('x-amz-version-id', 'K5b8GHNb3Mn1saWR47Y7KNPxakeIWEc'), ('x-amz-request-id', 'tx000007d407e18352cc54d-006477a747-110e-a1'), ('Date', 'Wed, 31 May 2023 20:00:07 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:00:07.503 DEBUG:rgw_multi.tests:zone=a1 version=K5b8GHNb3Mn1saWR47Y7KNPxakeIWEc 2023-05-31T20:00:07.504 DEBUG:boto:path=/pklxkd-13/ 2023-05-31T20:00:07.504 DEBUG:boto:auth_path=/pklxkd-13/ 2023-05-31T20:00:07.504 DEBUG:boto:Method: HEAD 2023-05-31T20:00:07.505 DEBUG:boto:Path: /pklxkd-13/ 2023-05-31T20:00:07.505 DEBUG:boto:Data: 2023-05-31T20:00:07.505 DEBUG:boto:Headers: {} 2023-05-31T20:00:07.506 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:00:07.506 DEBUG:boto:Port: 8001 2023-05-31T20:00:07.506 DEBUG:boto:Params: {} 2023-05-31T20:00:07.507 DEBUG:boto:Token: None 2023-05-31T20:00:07.507 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:00:07 GMT /pklxkd-13/ 2023-05-31T20:00:07.507 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:ljgRCrbX9BOLkRhZ67n9Rl1KCqM= 2023-05-31T20:00:07.508 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:00:07 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:ljgRCrbX9BOLkRhZ67n9Rl1KCqM=', 'Content-Length': '0'} 2023-05-31T20:00:07.510 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '2'), ('X-RGW-Bytes-Used', '8'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000008c4cecf6eb813fda-006477a747-1112-a2'), ('Content-Length', '0'), ('Date', 'Wed, 31 May 2023 20:00:07 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:00:07.511 DEBUG:boto:path=/pklxkd-13/obj 2023-05-31T20:00:07.511 DEBUG:boto:auth_path=/pklxkd-13/obj 2023-05-31T20:00:07.511 DEBUG:boto:Method: PUT 2023-05-31T20:00:07.512 DEBUG:boto:Path: /pklxkd-13/obj 2023-05-31T20:00:07.512 DEBUG:boto:Data: 2023-05-31T20:00:07.513 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'lmY06/L8E1cH1nU2kr9LHg==', 'Content-Length': '8', 'Expect': '100-Continue'} 2023-05-31T20:00:07.513 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:00:07.513 DEBUG:boto:Port: 8001 2023-05-31T20:00:07.513 DEBUG:boto:Params: {} 2023-05-31T20:00:07.514 DEBUG:boto:Token: None 2023-05-31T20:00:07.514 DEBUG:boto:StringToSign: PUT lmY06/L8E1cH1nU2kr9LHg== application/octet-stream Wed, 31 May 2023 20:00:07 GMT /pklxkd-13/obj 2023-05-31T20:00:07.515 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:oQTZSVNviz1HSJVc+j82uk5C1P4= 2023-05-31T20:00:07.515 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'lmY06/L8E1cH1nU2kr9LHg==', 'Content-Length': '8', 'Expect': '100-Continue', 'Date': 'Wed, 31 May 2023 20:00:07 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:oQTZSVNviz1HSJVc+j82uk5C1P4='} 2023-05-31T20:00:07.526 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('Accept-Ranges', 'bytes'), ('x-amz-version-id', 'p5Ybj.5dD2GwjElOd3T2Sa50bcy.UgE'), ('x-amz-request-id', 'tx0000043a3573322d799b7-006477a747-1112-a2'), ('Date', 'Wed, 31 May 2023 20:00:07 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:00:07.526 DEBUG:rgw_multi.tests:zone=a2 version=p5Ybj.5dD2GwjElOd3T2Sa50bcy.UgE 2023-05-31T20:00:07.526 DEBUG:boto:path=/pklxkd-13/ 2023-05-31T20:00:07.527 DEBUG:boto:auth_path=/pklxkd-13/ 2023-05-31T20:00:07.527 DEBUG:boto:Method: HEAD 2023-05-31T20:00:07.527 DEBUG:boto:Path: /pklxkd-13/ 2023-05-31T20:00:07.528 DEBUG:boto:Data: 2023-05-31T20:00:07.528 DEBUG:boto:Headers: {} 2023-05-31T20:00:07.528 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:00:07.529 DEBUG:boto:Port: 8000 2023-05-31T20:00:07.529 DEBUG:boto:Params: {} 2023-05-31T20:00:07.529 DEBUG:boto:Token: None 2023-05-31T20:00:07.530 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:00:07 GMT /pklxkd-13/ 2023-05-31T20:00:07.530 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:ljgRCrbX9BOLkRhZ67n9Rl1KCqM= 2023-05-31T20:00:07.531 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:00:07 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:ljgRCrbX9BOLkRhZ67n9Rl1KCqM=', 'Content-Length': '0'} 2023-05-31T20:00:07.533 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '3'), ('X-RGW-Bytes-Used', '16'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000009224a357fe2fa0e2-006477a747-110e-a1'), ('Content-Length', '0'), ('Date', 'Wed, 31 May 2023 20:00:07 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:00:07.533 DEBUG:boto:path=/pklxkd-13/obj 2023-05-31T20:00:07.534 DEBUG:boto:auth_path=/pklxkd-13/obj 2023-05-31T20:00:07.534 DEBUG:boto:Method: PUT 2023-05-31T20:00:07.534 DEBUG:boto:Path: /pklxkd-13/obj 2023-05-31T20:00:07.535 DEBUG:boto:Data: 2023-05-31T20:00:07.535 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'lmY06/L8E1cH1nU2kr9LHg==', 'Content-Length': '8', 'Expect': '100-Continue'} 2023-05-31T20:00:07.535 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:00:07.536 DEBUG:boto:Port: 8000 2023-05-31T20:00:07.536 DEBUG:boto:Params: {} 2023-05-31T20:00:07.537 DEBUG:boto:Token: None 2023-05-31T20:00:07.537 DEBUG:boto:StringToSign: PUT lmY06/L8E1cH1nU2kr9LHg== application/octet-stream Wed, 31 May 2023 20:00:07 GMT /pklxkd-13/obj 2023-05-31T20:00:07.537 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:oQTZSVNviz1HSJVc+j82uk5C1P4= 2023-05-31T20:00:07.538 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'lmY06/L8E1cH1nU2kr9LHg==', 'Content-Length': '8', 'Expect': '100-Continue', 'Date': 'Wed, 31 May 2023 20:00:07 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:oQTZSVNviz1HSJVc+j82uk5C1P4='} 2023-05-31T20:00:07.580 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('Accept-Ranges', 'bytes'), ('x-amz-version-id', 'O9dH9NhVFdlcrFlXwres-hSyGvDhDaQ'), ('x-amz-request-id', 'tx0000070331877144bb2ba-006477a747-110e-a1'), ('Date', 'Wed, 31 May 2023 20:00:07 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:00:07.581 DEBUG:rgw_multi.tests:zone=a1 version=O9dH9NhVFdlcrFlXwres-hSyGvDhDaQ 2023-05-31T20:00:07.581 DEBUG:boto:path=/pklxkd-13/ 2023-05-31T20:00:07.582 DEBUG:boto:auth_path=/pklxkd-13/ 2023-05-31T20:00:07.582 DEBUG:boto:Method: HEAD 2023-05-31T20:00:07.582 DEBUG:boto:Path: /pklxkd-13/ 2023-05-31T20:00:07.583 DEBUG:boto:Data: 2023-05-31T20:00:07.583 DEBUG:boto:Headers: {} 2023-05-31T20:00:07.583 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:00:07.584 DEBUG:boto:Port: 8001 2023-05-31T20:00:07.584 DEBUG:boto:Params: {} 2023-05-31T20:00:07.584 DEBUG:boto:Token: None 2023-05-31T20:00:07.585 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:00:07 GMT /pklxkd-13/ 2023-05-31T20:00:07.585 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:ljgRCrbX9BOLkRhZ67n9Rl1KCqM= 2023-05-31T20:00:07.586 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:00:07 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:ljgRCrbX9BOLkRhZ67n9Rl1KCqM=', 'Content-Length': '0'} 2023-05-31T20:00:07.588 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '3'), ('X-RGW-Bytes-Used', '16'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000f262c8af1d435a58-006477a747-1112-a2'), ('Content-Length', '0'), ('Date', 'Wed, 31 May 2023 20:00:07 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:00:07.589 DEBUG:boto:path=/pklxkd-13/obj 2023-05-31T20:00:07.589 DEBUG:boto:auth_path=/pklxkd-13/obj 2023-05-31T20:00:07.590 DEBUG:boto:Method: PUT 2023-05-31T20:00:07.590 DEBUG:boto:Path: /pklxkd-13/obj 2023-05-31T20:00:07.590 DEBUG:boto:Data: 2023-05-31T20:00:07.591 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'lmY06/L8E1cH1nU2kr9LHg==', 'Content-Length': '8', 'Expect': '100-Continue'} 2023-05-31T20:00:07.591 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:00:07.591 DEBUG:boto:Port: 8001 2023-05-31T20:00:07.592 DEBUG:boto:Params: {} 2023-05-31T20:00:07.592 DEBUG:boto:Token: None 2023-05-31T20:00:07.593 DEBUG:boto:StringToSign: PUT lmY06/L8E1cH1nU2kr9LHg== application/octet-stream Wed, 31 May 2023 20:00:07 GMT /pklxkd-13/obj 2023-05-31T20:00:07.593 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:oQTZSVNviz1HSJVc+j82uk5C1P4= 2023-05-31T20:00:07.593 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'lmY06/L8E1cH1nU2kr9LHg==', 'Content-Length': '8', 'Expect': '100-Continue', 'Date': 'Wed, 31 May 2023 20:00:07 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:oQTZSVNviz1HSJVc+j82uk5C1P4='} 2023-05-31T20:00:07.604 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('Accept-Ranges', 'bytes'), ('x-amz-version-id', 'aS0yzteNN24E5q0Twvp.6Z7sdzsi2hw'), ('x-amz-request-id', 'tx000006954252a0158f06c-006477a747-1112-a2'), ('Date', 'Wed, 31 May 2023 20:00:07 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:00:07.604 DEBUG:rgw_multi.tests:zone=a2 version=aS0yzteNN24E5q0Twvp.6Z7sdzsi2hw 2023-05-31T20:00:07.605 DEBUG:boto:path=/pklxkd-13/ 2023-05-31T20:00:07.605 DEBUG:boto:auth_path=/pklxkd-13/ 2023-05-31T20:00:07.605 DEBUG:boto:Method: HEAD 2023-05-31T20:00:07.606 DEBUG:boto:Path: /pklxkd-13/ 2023-05-31T20:00:07.606 DEBUG:boto:Data: 2023-05-31T20:00:07.606 DEBUG:boto:Headers: {} 2023-05-31T20:00:07.607 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:00:07.607 DEBUG:boto:Port: 8000 2023-05-31T20:00:07.607 DEBUG:boto:Params: {} 2023-05-31T20:00:07.608 DEBUG:boto:Token: None 2023-05-31T20:00:07.608 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:00:07 GMT /pklxkd-13/ 2023-05-31T20:00:07.609 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:ljgRCrbX9BOLkRhZ67n9Rl1KCqM= 2023-05-31T20:00:07.609 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:00:07 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:ljgRCrbX9BOLkRhZ67n9Rl1KCqM=', 'Content-Length': '0'} 2023-05-31T20:00:07.611 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '4'), ('X-RGW-Bytes-Used', '24'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000006ff58c0403a2403d-006477a747-110e-a1'), ('Content-Length', '0'), ('Date', 'Wed, 31 May 2023 20:00:07 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:00:07.612 DEBUG:boto:path=/pklxkd-13/obj 2023-05-31T20:00:07.612 DEBUG:boto:auth_path=/pklxkd-13/obj 2023-05-31T20:00:07.613 DEBUG:boto:Method: PUT 2023-05-31T20:00:07.613 DEBUG:boto:Path: /pklxkd-13/obj 2023-05-31T20:00:07.613 DEBUG:boto:Data: 2023-05-31T20:00:07.614 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'lmY06/L8E1cH1nU2kr9LHg==', 'Content-Length': '8', 'Expect': '100-Continue'} 2023-05-31T20:00:07.614 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:00:07.614 DEBUG:boto:Port: 8000 2023-05-31T20:00:07.615 DEBUG:boto:Params: {} 2023-05-31T20:00:07.615 DEBUG:boto:Token: None 2023-05-31T20:00:07.615 DEBUG:boto:StringToSign: PUT lmY06/L8E1cH1nU2kr9LHg== application/octet-stream Wed, 31 May 2023 20:00:07 GMT /pklxkd-13/obj 2023-05-31T20:00:07.616 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:oQTZSVNviz1HSJVc+j82uk5C1P4= 2023-05-31T20:00:07.616 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'lmY06/L8E1cH1nU2kr9LHg==', 'Content-Length': '8', 'Expect': '100-Continue', 'Date': 'Wed, 31 May 2023 20:00:07 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:oQTZSVNviz1HSJVc+j82uk5C1P4='} 2023-05-31T20:00:07.634 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('Accept-Ranges', 'bytes'), ('x-amz-version-id', 'JR8T3JjcBmfQYfHtj3icgZUj81rYFms'), ('x-amz-request-id', 'tx00000ea73284632e938da-006477a747-110e-a1'), ('Date', 'Wed, 31 May 2023 20:00:07 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:00:07.635 DEBUG:rgw_multi.tests:zone=a1 version=JR8T3JjcBmfQYfHtj3icgZUj81rYFms 2023-05-31T20:00:07.637 DEBUG:boto:path=/pklxkd-13/ 2023-05-31T20:00:07.638 DEBUG:boto:auth_path=/pklxkd-13/ 2023-05-31T20:00:07.639 DEBUG:boto:Method: HEAD 2023-05-31T20:00:07.640 DEBUG:boto:Path: /pklxkd-13/ 2023-05-31T20:00:07.640 DEBUG:boto:Data: 2023-05-31T20:00:07.641 DEBUG:boto:Headers: {} 2023-05-31T20:00:07.642 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:00:07.643 DEBUG:boto:Port: 8001 2023-05-31T20:00:07.644 DEBUG:boto:Params: {} 2023-05-31T20:00:07.644 DEBUG:boto:Token: None 2023-05-31T20:00:07.645 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:00:07 GMT /pklxkd-13/ 2023-05-31T20:00:07.646 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:ljgRCrbX9BOLkRhZ67n9Rl1KCqM= 2023-05-31T20:00:07.646 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:00:07 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:ljgRCrbX9BOLkRhZ67n9Rl1KCqM=', 'Content-Length': '0'} 2023-05-31T20:00:07.649 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '4'), ('X-RGW-Bytes-Used', '24'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000f0be464d74fdc3fa-006477a747-1112-a2'), ('Content-Length', '0'), ('Date', 'Wed, 31 May 2023 20:00:07 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:00:07.650 DEBUG:boto:path=/pklxkd-13/obj 2023-05-31T20:00:07.651 DEBUG:boto:auth_path=/pklxkd-13/obj 2023-05-31T20:00:07.651 DEBUG:boto:Method: PUT 2023-05-31T20:00:07.651 DEBUG:boto:Path: /pklxkd-13/obj 2023-05-31T20:00:07.652 DEBUG:boto:Data: 2023-05-31T20:00:07.652 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'lmY06/L8E1cH1nU2kr9LHg==', 'Content-Length': '8', 'Expect': '100-Continue'} 2023-05-31T20:00:07.653 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:00:07.653 DEBUG:boto:Port: 8001 2023-05-31T20:00:07.653 DEBUG:boto:Params: {} 2023-05-31T20:00:07.654 DEBUG:boto:Token: None 2023-05-31T20:00:07.654 DEBUG:boto:StringToSign: PUT lmY06/L8E1cH1nU2kr9LHg== application/octet-stream Wed, 31 May 2023 20:00:07 GMT /pklxkd-13/obj 2023-05-31T20:00:07.655 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:oQTZSVNviz1HSJVc+j82uk5C1P4= 2023-05-31T20:00:07.655 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'lmY06/L8E1cH1nU2kr9LHg==', 'Content-Length': '8', 'Expect': '100-Continue', 'Date': 'Wed, 31 May 2023 20:00:07 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:oQTZSVNviz1HSJVc+j82uk5C1P4='} 2023-05-31T20:00:07.700 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('Accept-Ranges', 'bytes'), ('x-amz-version-id', 'ooEeuYF3vuTJzW20FXOdWWO9-xNIuan'), ('x-amz-request-id', 'tx0000024f0bc425decda43-006477a747-1112-a2'), ('Date', 'Wed, 31 May 2023 20:00:07 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:00:07.701 DEBUG:rgw_multi.tests:zone=a2 version=ooEeuYF3vuTJzW20FXOdWWO9-xNIuan 2023-05-31T20:00:07.701 DEBUG:boto:path=/pklxkd-13/ 2023-05-31T20:00:07.702 DEBUG:boto:auth_path=/pklxkd-13/ 2023-05-31T20:00:07.702 DEBUG:boto:Method: HEAD 2023-05-31T20:00:07.703 DEBUG:boto:Path: /pklxkd-13/ 2023-05-31T20:00:07.703 DEBUG:boto:Data: 2023-05-31T20:00:07.703 DEBUG:boto:Headers: {} 2023-05-31T20:00:07.704 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:00:07.704 DEBUG:boto:Port: 8000 2023-05-31T20:00:07.704 DEBUG:boto:Params: {} 2023-05-31T20:00:07.705 DEBUG:boto:Token: None 2023-05-31T20:00:07.705 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:00:07 GMT /pklxkd-13/ 2023-05-31T20:00:07.706 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:ljgRCrbX9BOLkRhZ67n9Rl1KCqM= 2023-05-31T20:00:07.706 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:00:07 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:ljgRCrbX9BOLkRhZ67n9Rl1KCqM=', 'Content-Length': '0'} 2023-05-31T20:00:07.709 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '5'), ('X-RGW-Bytes-Used', '32'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000333ecc07f28fca2c-006477a747-110e-a1'), ('Content-Length', '0'), ('Date', 'Wed, 31 May 2023 20:00:07 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:00:07.709 DEBUG:boto:path=/pklxkd-13/obj 2023-05-31T20:00:07.710 DEBUG:boto:auth_path=/pklxkd-13/obj 2023-05-31T20:00:07.710 DEBUG:boto:Method: PUT 2023-05-31T20:00:07.711 DEBUG:boto:Path: /pklxkd-13/obj 2023-05-31T20:00:07.711 DEBUG:boto:Data: 2023-05-31T20:00:07.711 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'lmY06/L8E1cH1nU2kr9LHg==', 'Content-Length': '8', 'Expect': '100-Continue'} 2023-05-31T20:00:07.712 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:00:07.712 DEBUG:boto:Port: 8000 2023-05-31T20:00:07.712 DEBUG:boto:Params: {} 2023-05-31T20:00:07.713 DEBUG:boto:Token: None 2023-05-31T20:00:07.713 DEBUG:boto:StringToSign: PUT lmY06/L8E1cH1nU2kr9LHg== application/octet-stream Wed, 31 May 2023 20:00:07 GMT /pklxkd-13/obj 2023-05-31T20:00:07.714 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:oQTZSVNviz1HSJVc+j82uk5C1P4= 2023-05-31T20:00:07.714 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'lmY06/L8E1cH1nU2kr9LHg==', 'Content-Length': '8', 'Expect': '100-Continue', 'Date': 'Wed, 31 May 2023 20:00:07 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:oQTZSVNviz1HSJVc+j82uk5C1P4='} 2023-05-31T20:00:07.730 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('Accept-Ranges', 'bytes'), ('x-amz-version-id', 'RsWecIE0wyYuGR3yXB85CvZ9iY5NhBA'), ('x-amz-request-id', 'tx0000058578c1555fedfed-006477a747-110e-a1'), ('Date', 'Wed, 31 May 2023 20:00:07 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:00:07.731 DEBUG:rgw_multi.tests:zone=a1 version=RsWecIE0wyYuGR3yXB85CvZ9iY5NhBA 2023-05-31T20:00:07.731 DEBUG:boto:path=/pklxkd-13/ 2023-05-31T20:00:07.731 DEBUG:boto:auth_path=/pklxkd-13/ 2023-05-31T20:00:07.732 DEBUG:boto:Method: HEAD 2023-05-31T20:00:07.732 DEBUG:boto:Path: /pklxkd-13/ 2023-05-31T20:00:07.733 DEBUG:boto:Data: 2023-05-31T20:00:07.734 DEBUG:boto:Headers: {} 2023-05-31T20:00:07.734 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:00:07.734 DEBUG:boto:Port: 8001 2023-05-31T20:00:07.735 DEBUG:boto:Params: {} 2023-05-31T20:00:07.735 DEBUG:boto:Token: None 2023-05-31T20:00:07.736 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:00:07 GMT /pklxkd-13/ 2023-05-31T20:00:07.736 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:ljgRCrbX9BOLkRhZ67n9Rl1KCqM= 2023-05-31T20:00:07.736 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:00:07 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:ljgRCrbX9BOLkRhZ67n9Rl1KCqM=', 'Content-Length': '0'} 2023-05-31T20:00:07.739 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '5'), ('X-RGW-Bytes-Used', '32'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000006b4be902dc2f9a2b-006477a747-1112-a2'), ('Content-Length', '0'), ('Date', 'Wed, 31 May 2023 20:00:07 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:00:07.739 DEBUG:boto:path=/pklxkd-13/obj 2023-05-31T20:00:07.740 DEBUG:boto:auth_path=/pklxkd-13/obj 2023-05-31T20:00:07.740 DEBUG:boto:Method: PUT 2023-05-31T20:00:07.741 DEBUG:boto:Path: /pklxkd-13/obj 2023-05-31T20:00:07.741 DEBUG:boto:Data: 2023-05-31T20:00:07.741 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'lmY06/L8E1cH1nU2kr9LHg==', 'Content-Length': '8', 'Expect': '100-Continue'} 2023-05-31T20:00:07.742 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:00:07.742 DEBUG:boto:Port: 8001 2023-05-31T20:00:07.742 DEBUG:boto:Params: {} 2023-05-31T20:00:07.743 DEBUG:boto:Token: None 2023-05-31T20:00:07.743 DEBUG:boto:StringToSign: PUT lmY06/L8E1cH1nU2kr9LHg== application/octet-stream Wed, 31 May 2023 20:00:07 GMT /pklxkd-13/obj 2023-05-31T20:00:07.743 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:oQTZSVNviz1HSJVc+j82uk5C1P4= 2023-05-31T20:00:07.744 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'lmY06/L8E1cH1nU2kr9LHg==', 'Content-Length': '8', 'Expect': '100-Continue', 'Date': 'Wed, 31 May 2023 20:00:07 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:oQTZSVNviz1HSJVc+j82uk5C1P4='} 2023-05-31T20:00:07.754 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('Accept-Ranges', 'bytes'), ('x-amz-version-id', 'BCM6vBdxKthZLIE0R8UhG0.omDRBvP7'), ('x-amz-request-id', 'tx00000c4f1c5aba60d4093-006477a747-1112-a2'), ('Date', 'Wed, 31 May 2023 20:00:07 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:00:07.755 DEBUG:rgw_multi.tests:zone=a2 version=BCM6vBdxKthZLIE0R8UhG0.omDRBvP7 2023-05-31T20:00:07.755 DEBUG:boto:path=/pklxkd-13/ 2023-05-31T20:00:07.756 DEBUG:boto:auth_path=/pklxkd-13/ 2023-05-31T20:00:07.756 DEBUG:boto:Method: HEAD 2023-05-31T20:00:07.756 DEBUG:boto:Path: /pklxkd-13/ 2023-05-31T20:00:07.757 DEBUG:boto:Data: 2023-05-31T20:00:07.757 DEBUG:boto:Headers: {} 2023-05-31T20:00:07.757 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:00:07.758 DEBUG:boto:Port: 8000 2023-05-31T20:00:07.758 DEBUG:boto:Params: {} 2023-05-31T20:00:07.758 DEBUG:boto:Token: None 2023-05-31T20:00:07.759 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:00:07 GMT /pklxkd-13/ 2023-05-31T20:00:07.759 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:ljgRCrbX9BOLkRhZ67n9Rl1KCqM= 2023-05-31T20:00:07.760 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:00:07 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:ljgRCrbX9BOLkRhZ67n9Rl1KCqM=', 'Content-Length': '0'} 2023-05-31T20:00:07.762 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '6'), ('X-RGW-Bytes-Used', '40'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000027c604af53a393ad-006477a747-110e-a1'), ('Content-Length', '0'), ('Date', 'Wed, 31 May 2023 20:00:07 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:00:07.762 DEBUG:boto:path=/pklxkd-13/obj 2023-05-31T20:00:07.763 DEBUG:boto:auth_path=/pklxkd-13/obj 2023-05-31T20:00:07.763 DEBUG:boto:Method: PUT 2023-05-31T20:00:07.763 DEBUG:boto:Path: /pklxkd-13/obj 2023-05-31T20:00:07.764 DEBUG:boto:Data: 2023-05-31T20:00:07.764 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'lmY06/L8E1cH1nU2kr9LHg==', 'Content-Length': '8', 'Expect': '100-Continue'} 2023-05-31T20:00:07.764 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:00:07.765 DEBUG:boto:Port: 8000 2023-05-31T20:00:07.765 DEBUG:boto:Params: {} 2023-05-31T20:00:07.765 DEBUG:boto:Token: None 2023-05-31T20:00:07.766 DEBUG:boto:StringToSign: PUT lmY06/L8E1cH1nU2kr9LHg== application/octet-stream Wed, 31 May 2023 20:00:07 GMT /pklxkd-13/obj 2023-05-31T20:00:07.766 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:oQTZSVNviz1HSJVc+j82uk5C1P4= 2023-05-31T20:00:07.767 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'lmY06/L8E1cH1nU2kr9LHg==', 'Content-Length': '8', 'Expect': '100-Continue', 'Date': 'Wed, 31 May 2023 20:00:07 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:oQTZSVNviz1HSJVc+j82uk5C1P4='} 2023-05-31T20:00:07.812 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('Accept-Ranges', 'bytes'), ('x-amz-version-id', 'nD.VY2SV.sxc3KR89HB3qbvIEu7ie0D'), ('x-amz-request-id', 'tx00000992284d4edf5cd30-006477a747-110e-a1'), ('Date', 'Wed, 31 May 2023 20:00:07 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:00:07.813 DEBUG:rgw_multi.tests:zone=a1 version=nD.VY2SV.sxc3KR89HB3qbvIEu7ie0D 2023-05-31T20:00:07.813 DEBUG:boto:path=/pklxkd-13/ 2023-05-31T20:00:07.814 DEBUG:boto:auth_path=/pklxkd-13/ 2023-05-31T20:00:07.814 DEBUG:boto:Method: HEAD 2023-05-31T20:00:07.814 DEBUG:boto:Path: /pklxkd-13/ 2023-05-31T20:00:07.815 DEBUG:boto:Data: 2023-05-31T20:00:07.815 DEBUG:boto:Headers: {} 2023-05-31T20:00:07.815 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:00:07.816 DEBUG:boto:Port: 8001 2023-05-31T20:00:07.816 DEBUG:boto:Params: {} 2023-05-31T20:00:07.816 DEBUG:boto:Token: None 2023-05-31T20:00:07.817 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:00:07 GMT /pklxkd-13/ 2023-05-31T20:00:07.817 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:ljgRCrbX9BOLkRhZ67n9Rl1KCqM= 2023-05-31T20:00:07.817 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:00:07 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:ljgRCrbX9BOLkRhZ67n9Rl1KCqM=', 'Content-Length': '0'} 2023-05-31T20:00:07.820 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '6'), ('X-RGW-Bytes-Used', '40'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000002f34109449897c24-006477a747-1112-a2'), ('Content-Length', '0'), ('Date', 'Wed, 31 May 2023 20:00:07 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:00:07.820 DEBUG:boto:path=/pklxkd-13/obj 2023-05-31T20:00:07.821 DEBUG:boto:auth_path=/pklxkd-13/obj 2023-05-31T20:00:07.821 DEBUG:boto:Method: PUT 2023-05-31T20:00:07.821 DEBUG:boto:Path: /pklxkd-13/obj 2023-05-31T20:00:07.822 DEBUG:boto:Data: 2023-05-31T20:00:07.822 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'lmY06/L8E1cH1nU2kr9LHg==', 'Content-Length': '8', 'Expect': '100-Continue'} 2023-05-31T20:00:07.822 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:00:07.823 DEBUG:boto:Port: 8001 2023-05-31T20:00:07.823 DEBUG:boto:Params: {} 2023-05-31T20:00:07.823 DEBUG:boto:Token: None 2023-05-31T20:00:07.824 DEBUG:boto:StringToSign: PUT lmY06/L8E1cH1nU2kr9LHg== application/octet-stream Wed, 31 May 2023 20:00:07 GMT /pklxkd-13/obj 2023-05-31T20:00:07.824 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:oQTZSVNviz1HSJVc+j82uk5C1P4= 2023-05-31T20:00:07.825 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'lmY06/L8E1cH1nU2kr9LHg==', 'Content-Length': '8', 'Expect': '100-Continue', 'Date': 'Wed, 31 May 2023 20:00:07 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:oQTZSVNviz1HSJVc+j82uk5C1P4='} 2023-05-31T20:00:07.834 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('Accept-Ranges', 'bytes'), ('x-amz-version-id', 'bBAtwbgprRto0Kkb2LYN6Gd-LmGQA5.'), ('x-amz-request-id', 'tx00000da3f156c5e362737-006477a747-1112-a2'), ('Date', 'Wed, 31 May 2023 20:00:07 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:00:07.834 DEBUG:rgw_multi.tests:zone=a2 version=bBAtwbgprRto0Kkb2LYN6Gd-LmGQA5. 2023-05-31T20:00:07.835 DEBUG:boto:path=/pklxkd-13/ 2023-05-31T20:00:07.835 DEBUG:boto:auth_path=/pklxkd-13/ 2023-05-31T20:00:07.835 DEBUG:boto:Method: HEAD 2023-05-31T20:00:07.836 DEBUG:boto:Path: /pklxkd-13/ 2023-05-31T20:00:07.836 DEBUG:boto:Data: 2023-05-31T20:00:07.836 DEBUG:boto:Headers: {} 2023-05-31T20:00:07.837 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:00:07.837 DEBUG:boto:Port: 8000 2023-05-31T20:00:07.837 DEBUG:boto:Params: {} 2023-05-31T20:00:07.838 DEBUG:boto:Token: None 2023-05-31T20:00:07.838 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:00:07 GMT /pklxkd-13/ 2023-05-31T20:00:07.838 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:ljgRCrbX9BOLkRhZ67n9Rl1KCqM= 2023-05-31T20:00:07.839 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:00:07 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:ljgRCrbX9BOLkRhZ67n9Rl1KCqM=', 'Content-Length': '0'} 2023-05-31T20:00:07.841 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '7'), ('X-RGW-Bytes-Used', '48'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000a408a940d91fa396-006477a747-110e-a1'), ('Content-Length', '0'), ('Date', 'Wed, 31 May 2023 20:00:07 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:00:07.842 DEBUG:boto:path=/pklxkd-13/obj 2023-05-31T20:00:07.842 DEBUG:boto:auth_path=/pklxkd-13/obj 2023-05-31T20:00:07.842 DEBUG:boto:Method: PUT 2023-05-31T20:00:07.843 DEBUG:boto:Path: /pklxkd-13/obj 2023-05-31T20:00:07.843 DEBUG:boto:Data: 2023-05-31T20:00:07.843 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'lmY06/L8E1cH1nU2kr9LHg==', 'Content-Length': '8', 'Expect': '100-Continue'} 2023-05-31T20:00:07.844 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:00:07.844 DEBUG:boto:Port: 8000 2023-05-31T20:00:07.844 DEBUG:boto:Params: {} 2023-05-31T20:00:07.845 DEBUG:boto:Token: None 2023-05-31T20:00:07.845 DEBUG:boto:StringToSign: PUT lmY06/L8E1cH1nU2kr9LHg== application/octet-stream Wed, 31 May 2023 20:00:07 GMT /pklxkd-13/obj 2023-05-31T20:00:07.846 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:oQTZSVNviz1HSJVc+j82uk5C1P4= 2023-05-31T20:00:07.846 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'lmY06/L8E1cH1nU2kr9LHg==', 'Content-Length': '8', 'Expect': '100-Continue', 'Date': 'Wed, 31 May 2023 20:00:07 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:oQTZSVNviz1HSJVc+j82uk5C1P4='} 2023-05-31T20:00:07.856 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('Accept-Ranges', 'bytes'), ('x-amz-version-id', 'mWkpyutJAg628MwnVVNvQnYfQFw1fqJ'), ('x-amz-request-id', 'tx000007dc84a6659f9be64-006477a747-110e-a1'), ('Date', 'Wed, 31 May 2023 20:00:07 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:00:07.857 DEBUG:rgw_multi.tests:zone=a1 version=mWkpyutJAg628MwnVVNvQnYfQFw1fqJ 2023-05-31T20:00:07.857 DEBUG:boto:path=/pklxkd-13/ 2023-05-31T20:00:07.857 DEBUG:boto:auth_path=/pklxkd-13/ 2023-05-31T20:00:07.858 DEBUG:boto:Method: HEAD 2023-05-31T20:00:07.858 DEBUG:boto:Path: /pklxkd-13/ 2023-05-31T20:00:07.858 DEBUG:boto:Data: 2023-05-31T20:00:07.859 DEBUG:boto:Headers: {} 2023-05-31T20:00:07.859 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:00:07.859 DEBUG:boto:Port: 8001 2023-05-31T20:00:07.860 DEBUG:boto:Params: {} 2023-05-31T20:00:07.860 DEBUG:boto:Token: None 2023-05-31T20:00:07.860 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:00:07 GMT /pklxkd-13/ 2023-05-31T20:00:07.861 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:ljgRCrbX9BOLkRhZ67n9Rl1KCqM= 2023-05-31T20:00:07.861 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:00:07 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:ljgRCrbX9BOLkRhZ67n9Rl1KCqM=', 'Content-Length': '0'} 2023-05-31T20:00:07.863 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '7'), ('X-RGW-Bytes-Used', '48'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000088aedb88fce6b3e-006477a747-1112-a2'), ('Content-Length', '0'), ('Date', 'Wed, 31 May 2023 20:00:07 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:00:07.864 DEBUG:boto:path=/pklxkd-13/obj 2023-05-31T20:00:07.865 DEBUG:boto:auth_path=/pklxkd-13/obj 2023-05-31T20:00:07.865 DEBUG:boto:Method: PUT 2023-05-31T20:00:07.865 DEBUG:boto:Path: /pklxkd-13/obj 2023-05-31T20:00:07.866 DEBUG:boto:Data: 2023-05-31T20:00:07.866 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'lmY06/L8E1cH1nU2kr9LHg==', 'Content-Length': '8', 'Expect': '100-Continue'} 2023-05-31T20:00:07.866 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:00:07.867 DEBUG:boto:Port: 8001 2023-05-31T20:00:07.867 DEBUG:boto:Params: {} 2023-05-31T20:00:07.867 DEBUG:boto:Token: None 2023-05-31T20:00:07.868 DEBUG:boto:StringToSign: PUT lmY06/L8E1cH1nU2kr9LHg== application/octet-stream Wed, 31 May 2023 20:00:07 GMT /pklxkd-13/obj 2023-05-31T20:00:07.868 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:oQTZSVNviz1HSJVc+j82uk5C1P4= 2023-05-31T20:00:07.869 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'lmY06/L8E1cH1nU2kr9LHg==', 'Content-Length': '8', 'Expect': '100-Continue', 'Date': 'Wed, 31 May 2023 20:00:07 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:oQTZSVNviz1HSJVc+j82uk5C1P4='} 2023-05-31T20:00:07.877 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('Accept-Ranges', 'bytes'), ('x-amz-version-id', 'Wr6wnEZSP3WRSGvZ8nB1.KinMc4wxvZ'), ('x-amz-request-id', 'tx00000c3119f0ce77eb7b0-006477a747-1112-a2'), ('Date', 'Wed, 31 May 2023 20:00:07 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:00:07.878 DEBUG:rgw_multi.tests:zone=a2 version=Wr6wnEZSP3WRSGvZ8nB1.KinMc4wxvZ 2023-05-31T20:00:07.878 DEBUG:boto:path=/pklxkd-13/ 2023-05-31T20:00:07.879 DEBUG:boto:auth_path=/pklxkd-13/ 2023-05-31T20:00:07.879 DEBUG:boto:Method: HEAD 2023-05-31T20:00:07.879 DEBUG:boto:Path: /pklxkd-13/ 2023-05-31T20:00:07.880 DEBUG:boto:Data: 2023-05-31T20:00:07.880 DEBUG:boto:Headers: {} 2023-05-31T20:00:07.880 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:00:07.881 DEBUG:boto:Port: 8000 2023-05-31T20:00:07.881 DEBUG:boto:Params: {} 2023-05-31T20:00:07.881 DEBUG:boto:Token: None 2023-05-31T20:00:07.882 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:00:07 GMT /pklxkd-13/ 2023-05-31T20:00:07.882 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:ljgRCrbX9BOLkRhZ67n9Rl1KCqM= 2023-05-31T20:00:07.883 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:00:07 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:ljgRCrbX9BOLkRhZ67n9Rl1KCqM=', 'Content-Length': '0'} 2023-05-31T20:00:07.885 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '8'), ('X-RGW-Bytes-Used', '56'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000005658540b288b8cfd-006477a747-110e-a1'), ('Content-Length', '0'), ('Date', 'Wed, 31 May 2023 20:00:07 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:00:07.886 DEBUG:boto:path=/pklxkd-13/obj 2023-05-31T20:00:07.886 DEBUG:boto:auth_path=/pklxkd-13/obj 2023-05-31T20:00:07.886 DEBUG:boto:Method: PUT 2023-05-31T20:00:07.887 DEBUG:boto:Path: /pklxkd-13/obj 2023-05-31T20:00:07.887 DEBUG:boto:Data: 2023-05-31T20:00:07.887 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'lmY06/L8E1cH1nU2kr9LHg==', 'Content-Length': '8', 'Expect': '100-Continue'} 2023-05-31T20:00:07.888 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:00:07.888 DEBUG:boto:Port: 8000 2023-05-31T20:00:07.888 DEBUG:boto:Params: {} 2023-05-31T20:00:07.889 DEBUG:boto:Token: None 2023-05-31T20:00:07.889 DEBUG:boto:StringToSign: PUT lmY06/L8E1cH1nU2kr9LHg== application/octet-stream Wed, 31 May 2023 20:00:07 GMT /pklxkd-13/obj 2023-05-31T20:00:07.890 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:oQTZSVNviz1HSJVc+j82uk5C1P4= 2023-05-31T20:00:07.890 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'lmY06/L8E1cH1nU2kr9LHg==', 'Content-Length': '8', 'Expect': '100-Continue', 'Date': 'Wed, 31 May 2023 20:00:07 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:oQTZSVNviz1HSJVc+j82uk5C1P4='} 2023-05-31T20:00:07.932 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('Accept-Ranges', 'bytes'), ('x-amz-version-id', 'oJ31LeYY.9AEY1WQGQs-wBnfC1MDSr6'), ('x-amz-request-id', 'tx0000031d4dd5354ca6b0c-006477a747-110e-a1'), ('Date', 'Wed, 31 May 2023 20:00:07 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:00:07.933 DEBUG:rgw_multi.tests:zone=a1 version=oJ31LeYY.9AEY1WQGQs-wBnfC1MDSr6 2023-05-31T20:00:07.933 DEBUG:boto:path=/pklxkd-13/ 2023-05-31T20:00:07.934 DEBUG:boto:auth_path=/pklxkd-13/ 2023-05-31T20:00:07.934 DEBUG:boto:Method: HEAD 2023-05-31T20:00:07.934 DEBUG:boto:Path: /pklxkd-13/ 2023-05-31T20:00:07.935 DEBUG:boto:Data: 2023-05-31T20:00:07.935 DEBUG:boto:Headers: {} 2023-05-31T20:00:07.935 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:00:07.936 DEBUG:boto:Port: 8001 2023-05-31T20:00:07.936 DEBUG:boto:Params: {} 2023-05-31T20:00:07.936 DEBUG:boto:Token: None 2023-05-31T20:00:07.937 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:00:07 GMT /pklxkd-13/ 2023-05-31T20:00:07.937 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:ljgRCrbX9BOLkRhZ67n9Rl1KCqM= 2023-05-31T20:00:07.938 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:00:07 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:ljgRCrbX9BOLkRhZ67n9Rl1KCqM=', 'Content-Length': '0'} 2023-05-31T20:00:07.940 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '8'), ('X-RGW-Bytes-Used', '56'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000021283191a6fc6ddd-006477a747-1112-a2'), ('Content-Length', '0'), ('Date', 'Wed, 31 May 2023 20:00:07 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:00:07.940 DEBUG:boto:path=/pklxkd-13/obj 2023-05-31T20:00:07.941 DEBUG:boto:auth_path=/pklxkd-13/obj 2023-05-31T20:00:07.941 DEBUG:boto:Method: PUT 2023-05-31T20:00:07.942 DEBUG:boto:Path: /pklxkd-13/obj 2023-05-31T20:00:07.942 DEBUG:boto:Data: 2023-05-31T20:00:07.942 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'lmY06/L8E1cH1nU2kr9LHg==', 'Content-Length': '8', 'Expect': '100-Continue'} 2023-05-31T20:00:07.943 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:00:07.943 DEBUG:boto:Port: 8001 2023-05-31T20:00:07.943 DEBUG:boto:Params: {} 2023-05-31T20:00:07.944 DEBUG:boto:Token: None 2023-05-31T20:00:07.944 DEBUG:boto:StringToSign: PUT lmY06/L8E1cH1nU2kr9LHg== application/octet-stream Wed, 31 May 2023 20:00:07 GMT /pklxkd-13/obj 2023-05-31T20:00:07.944 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:oQTZSVNviz1HSJVc+j82uk5C1P4= 2023-05-31T20:00:07.945 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'lmY06/L8E1cH1nU2kr9LHg==', 'Content-Length': '8', 'Expect': '100-Continue', 'Date': 'Wed, 31 May 2023 20:00:07 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:oQTZSVNviz1HSJVc+j82uk5C1P4='} 2023-05-31T20:00:07.961 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('Accept-Ranges', 'bytes'), ('x-amz-version-id', 'LGC78se33tZFR9UpgQStirsP0mP9GEu'), ('x-amz-request-id', 'tx00000051db5ac2e8dce9d-006477a747-1112-a2'), ('Date', 'Wed, 31 May 2023 20:00:07 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:00:07.962 DEBUG:rgw_multi.tests:zone=a2 version=LGC78se33tZFR9UpgQStirsP0mP9GEu 2023-05-31T20:00:07.962 DEBUG:boto:path=/pklxkd-13/ 2023-05-31T20:00:07.963 DEBUG:boto:auth_path=/pklxkd-13/ 2023-05-31T20:00:07.963 DEBUG:boto:Method: HEAD 2023-05-31T20:00:07.963 DEBUG:boto:Path: /pklxkd-13/ 2023-05-31T20:00:07.964 DEBUG:boto:Data: 2023-05-31T20:00:07.964 DEBUG:boto:Headers: {} 2023-05-31T20:00:07.964 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:00:07.965 DEBUG:boto:Port: 8000 2023-05-31T20:00:07.965 DEBUG:boto:Params: {} 2023-05-31T20:00:07.965 DEBUG:boto:Token: None 2023-05-31T20:00:07.966 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:00:07 GMT /pklxkd-13/ 2023-05-31T20:00:07.966 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:ljgRCrbX9BOLkRhZ67n9Rl1KCqM= 2023-05-31T20:00:07.966 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:00:07 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:ljgRCrbX9BOLkRhZ67n9Rl1KCqM=', 'Content-Length': '0'} 2023-05-31T20:00:07.969 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '9'), ('X-RGW-Bytes-Used', '64'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000029d34b7836b2e6e2-006477a747-110e-a1'), ('Content-Length', '0'), ('Date', 'Wed, 31 May 2023 20:00:07 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:00:07.970 DEBUG:boto:path=/pklxkd-13/obj 2023-05-31T20:00:07.970 DEBUG:boto:auth_path=/pklxkd-13/obj 2023-05-31T20:00:07.970 DEBUG:boto:Method: PUT 2023-05-31T20:00:07.971 DEBUG:boto:Path: /pklxkd-13/obj 2023-05-31T20:00:07.971 DEBUG:boto:Data: 2023-05-31T20:00:07.971 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'lmY06/L8E1cH1nU2kr9LHg==', 'Content-Length': '8', 'Expect': '100-Continue'} 2023-05-31T20:00:07.972 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:00:07.972 DEBUG:boto:Port: 8000 2023-05-31T20:00:07.973 DEBUG:boto:Params: {} 2023-05-31T20:00:07.973 DEBUG:boto:Token: None 2023-05-31T20:00:07.973 DEBUG:boto:StringToSign: PUT lmY06/L8E1cH1nU2kr9LHg== application/octet-stream Wed, 31 May 2023 20:00:07 GMT /pklxkd-13/obj 2023-05-31T20:00:07.974 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:oQTZSVNviz1HSJVc+j82uk5C1P4= 2023-05-31T20:00:07.974 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'lmY06/L8E1cH1nU2kr9LHg==', 'Content-Length': '8', 'Expect': '100-Continue', 'Date': 'Wed, 31 May 2023 20:00:07 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:oQTZSVNviz1HSJVc+j82uk5C1P4='} 2023-05-31T20:00:07.992 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('Accept-Ranges', 'bytes'), ('x-amz-version-id', 'nE3onm9L7VUWceoER2mn2jkX5yHK5GL'), ('x-amz-request-id', 'tx0000043dc754aedc0533c-006477a747-110e-a1'), ('Date', 'Wed, 31 May 2023 20:00:07 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:00:07.992 DEBUG:rgw_multi.tests:zone=a1 version=nE3onm9L7VUWceoER2mn2jkX5yHK5GL 2023-05-31T20:00:07.993 DEBUG:boto:path=/pklxkd-13/ 2023-05-31T20:00:07.993 DEBUG:boto:auth_path=/pklxkd-13/ 2023-05-31T20:00:07.994 DEBUG:boto:Method: HEAD 2023-05-31T20:00:07.994 DEBUG:boto:Path: /pklxkd-13/ 2023-05-31T20:00:07.994 DEBUG:boto:Data: 2023-05-31T20:00:07.995 DEBUG:boto:Headers: {} 2023-05-31T20:00:07.995 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:00:07.995 DEBUG:boto:Port: 8001 2023-05-31T20:00:07.996 DEBUG:boto:Params: {} 2023-05-31T20:00:07.996 DEBUG:boto:Token: None 2023-05-31T20:00:07.996 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:00:07 GMT /pklxkd-13/ 2023-05-31T20:00:07.997 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:ljgRCrbX9BOLkRhZ67n9Rl1KCqM= 2023-05-31T20:00:07.997 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:00:07 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:ljgRCrbX9BOLkRhZ67n9Rl1KCqM=', 'Content-Length': '0'} 2023-05-31T20:00:07.999 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '9'), ('X-RGW-Bytes-Used', '64'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000ea51fb40af4864a1-006477a747-1112-a2'), ('Content-Length', '0'), ('Date', 'Wed, 31 May 2023 20:00:07 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:00:08.000 DEBUG:boto:path=/pklxkd-13/obj 2023-05-31T20:00:08.000 DEBUG:boto:auth_path=/pklxkd-13/obj 2023-05-31T20:00:08.001 DEBUG:boto:Method: PUT 2023-05-31T20:00:08.001 DEBUG:boto:Path: /pklxkd-13/obj 2023-05-31T20:00:08.001 DEBUG:boto:Data: 2023-05-31T20:00:08.002 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'lmY06/L8E1cH1nU2kr9LHg==', 'Content-Length': '8', 'Expect': '100-Continue'} 2023-05-31T20:00:08.002 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:00:08.002 DEBUG:boto:Port: 8001 2023-05-31T20:00:08.003 DEBUG:boto:Params: {} 2023-05-31T20:00:08.003 DEBUG:boto:Token: None 2023-05-31T20:00:08.003 DEBUG:boto:StringToSign: PUT lmY06/L8E1cH1nU2kr9LHg== application/octet-stream Wed, 31 May 2023 20:00:08 GMT /pklxkd-13/obj 2023-05-31T20:00:08.004 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:tssy2ETnwGm93hohATZguVCR3b8= 2023-05-31T20:00:08.004 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'lmY06/L8E1cH1nU2kr9LHg==', 'Content-Length': '8', 'Expect': '100-Continue', 'Date': 'Wed, 31 May 2023 20:00:08 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:tssy2ETnwGm93hohATZguVCR3b8='} 2023-05-31T20:00:08.048 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('Accept-Ranges', 'bytes'), ('x-amz-version-id', '.q2JxPcqi3BYUXNGXuFQE0hVEIAq1Yk'), ('x-amz-request-id', 'tx00000ad2463022b31635d-006477a748-1112-a2'), ('Date', 'Wed, 31 May 2023 20:00:08 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:00:08.049 DEBUG:rgw_multi.tests:zone=a2 version=.q2JxPcqi3BYUXNGXuFQE0hVEIAq1Yk 2023-05-31T20:00:08.049 DEBUG:boto:path=/pklxkd-13/ 2023-05-31T20:00:08.050 DEBUG:boto:auth_path=/pklxkd-13/ 2023-05-31T20:00:08.050 DEBUG:boto:Method: HEAD 2023-05-31T20:00:08.050 DEBUG:boto:Path: /pklxkd-13/ 2023-05-31T20:00:08.051 DEBUG:boto:Data: 2023-05-31T20:00:08.051 DEBUG:boto:Headers: {} 2023-05-31T20:00:08.051 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:00:08.052 DEBUG:boto:Port: 8000 2023-05-31T20:00:08.052 DEBUG:boto:Params: {} 2023-05-31T20:00:08.053 DEBUG:boto:Token: None 2023-05-31T20:00:08.053 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:00:08 GMT /pklxkd-13/ 2023-05-31T20:00:08.053 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:a0/t9dxApZt1px2xYFybJ2AD7sM= 2023-05-31T20:00:08.054 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:00:08 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:a0/t9dxApZt1px2xYFybJ2AD7sM=', 'Content-Length': '0'} 2023-05-31T20:00:08.056 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '10'), ('X-RGW-Bytes-Used', '72'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000f6f4d27f2a7fe6ce-006477a748-110e-a1'), ('Content-Length', '0'), ('Date', 'Wed, 31 May 2023 20:00:08 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:00:08.057 DEBUG:boto:path=/pklxkd-13/obj 2023-05-31T20:00:08.057 DEBUG:boto:auth_path=/pklxkd-13/obj 2023-05-31T20:00:08.057 DEBUG:boto:Method: PUT 2023-05-31T20:00:08.058 DEBUG:boto:Path: /pklxkd-13/obj 2023-05-31T20:00:08.058 DEBUG:boto:Data: 2023-05-31T20:00:08.058 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'lmY06/L8E1cH1nU2kr9LHg==', 'Content-Length': '8', 'Expect': '100-Continue'} 2023-05-31T20:00:08.059 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:00:08.059 DEBUG:boto:Port: 8000 2023-05-31T20:00:08.059 DEBUG:boto:Params: {} 2023-05-31T20:00:08.060 DEBUG:boto:Token: None 2023-05-31T20:00:08.060 DEBUG:boto:StringToSign: PUT lmY06/L8E1cH1nU2kr9LHg== application/octet-stream Wed, 31 May 2023 20:00:08 GMT /pklxkd-13/obj 2023-05-31T20:00:08.061 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:tssy2ETnwGm93hohATZguVCR3b8= 2023-05-31T20:00:08.061 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'lmY06/L8E1cH1nU2kr9LHg==', 'Content-Length': '8', 'Expect': '100-Continue', 'Date': 'Wed, 31 May 2023 20:00:08 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:tssy2ETnwGm93hohATZguVCR3b8='} 2023-05-31T20:00:08.076 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('Accept-Ranges', 'bytes'), ('x-amz-version-id', '.K0qHk3QiH4acREzLteeQLJmFzn4x-1'), ('x-amz-request-id', 'tx0000053139c5b0caf1f1d-006477a748-110e-a1'), ('Date', 'Wed, 31 May 2023 20:00:08 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:00:08.077 DEBUG:rgw_multi.tests:zone=a1 version=.K0qHk3QiH4acREzLteeQLJmFzn4x-1 2023-05-31T20:00:08.077 DEBUG:boto:path=/pklxkd-13/ 2023-05-31T20:00:08.077 DEBUG:boto:auth_path=/pklxkd-13/ 2023-05-31T20:00:08.078 DEBUG:boto:Method: HEAD 2023-05-31T20:00:08.078 DEBUG:boto:Path: /pklxkd-13/ 2023-05-31T20:00:08.078 DEBUG:boto:Data: 2023-05-31T20:00:08.079 DEBUG:boto:Headers: {} 2023-05-31T20:00:08.079 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:00:08.079 DEBUG:boto:Port: 8001 2023-05-31T20:00:08.080 DEBUG:boto:Params: {} 2023-05-31T20:00:08.080 DEBUG:boto:Token: None 2023-05-31T20:00:08.080 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:00:08 GMT /pklxkd-13/ 2023-05-31T20:00:08.081 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:a0/t9dxApZt1px2xYFybJ2AD7sM= 2023-05-31T20:00:08.081 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:00:08 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:a0/t9dxApZt1px2xYFybJ2AD7sM=', 'Content-Length': '0'} 2023-05-31T20:00:08.083 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '10'), ('X-RGW-Bytes-Used', '72'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000992e9dbea234bee1-006477a748-1112-a2'), ('Content-Length', '0'), ('Date', 'Wed, 31 May 2023 20:00:08 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:00:08.084 DEBUG:boto:path=/pklxkd-13/obj 2023-05-31T20:00:08.084 DEBUG:boto:auth_path=/pklxkd-13/obj 2023-05-31T20:00:08.085 DEBUG:boto:Method: PUT 2023-05-31T20:00:08.085 DEBUG:boto:Path: /pklxkd-13/obj 2023-05-31T20:00:08.086 DEBUG:boto:Data: 2023-05-31T20:00:08.086 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'lmY06/L8E1cH1nU2kr9LHg==', 'Content-Length': '8', 'Expect': '100-Continue'} 2023-05-31T20:00:08.086 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:00:08.087 DEBUG:boto:Port: 8001 2023-05-31T20:00:08.087 DEBUG:boto:Params: {} 2023-05-31T20:00:08.087 DEBUG:boto:Token: None 2023-05-31T20:00:08.088 DEBUG:boto:StringToSign: PUT lmY06/L8E1cH1nU2kr9LHg== application/octet-stream Wed, 31 May 2023 20:00:08 GMT /pklxkd-13/obj 2023-05-31T20:00:08.088 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:tssy2ETnwGm93hohATZguVCR3b8= 2023-05-31T20:00:08.089 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'lmY06/L8E1cH1nU2kr9LHg==', 'Content-Length': '8', 'Expect': '100-Continue', 'Date': 'Wed, 31 May 2023 20:00:08 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:tssy2ETnwGm93hohATZguVCR3b8='} 2023-05-31T20:00:08.097 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('Accept-Ranges', 'bytes'), ('x-amz-version-id', 'gr0pDclEGPIgGhrGevSqfnt7WtJ89yD'), ('x-amz-request-id', 'tx000006e92860470f0c476-006477a748-1112-a2'), ('Date', 'Wed, 31 May 2023 20:00:08 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:00:08.097 DEBUG:rgw_multi.tests:zone=a2 version=gr0pDclEGPIgGhrGevSqfnt7WtJ89yD 2023-05-31T20:00:08.098 DEBUG:rgw_multi.tests:bucket checkpoint: source=a1 target=a2 bucket=pklxkd-13 2023-05-31T20:00:08.098 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'pklxkd-13', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-05-31T20:00:08.098 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'pklxkd-13', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-05-31T20:00:08.099 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket pklxkd-13 --source-zone a1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2023-05-31T20:00:08.155 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2023-05-31T20:00:08.156 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2023-05-31T20:00:08.193 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:00:08.187+0000 7fa30b7d0780 1 waiting for incremental sync to catch up: 2023-05-31T20:00:08.193 INFO:teuthology.orchestra.run.smithi099.stderr: local status: [, , , , , 00000000003.35.10, , , , , ] 2023-05-31T20:00:08.194 INFO:teuthology.orchestra.run.smithi099.stderr: remote markers: [, , , , , 00000000003.35.10, , , 00000000020.47.12, , ] 2023-05-31T20:00:13.194 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:00:13.188+0000 7fa30b7d0780 1 waiting for incremental sync to catch up: 2023-05-31T20:00:13.195 INFO:teuthology.orchestra.run.smithi099.stderr: local status: [, , , , , 00000000003.35.10, , , , , ] 2023-05-31T20:00:13.195 INFO:teuthology.orchestra.run.smithi099.stderr: remote markers: [, , , , , 00000000003.35.10, , , 00000000020.47.12, , ] 2023-05-31T20:00:18.196 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:00:18.190+0000 7fa30b7d0780 1 waiting for incremental sync to catch up: 2023-05-31T20:00:18.196 INFO:teuthology.orchestra.run.smithi099.stderr: local status: [, , , , , 00000000003.35.10, , , , , ] 2023-05-31T20:00:18.196 INFO:teuthology.orchestra.run.smithi099.stderr: remote markers: [, , , , , 00000000003.35.10, , , 00000000020.47.12, , ] 2023-05-31T20:00:23.197 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:00:23.191+0000 7fa30b7d0780 1 waiting for incremental sync to catch up: 2023-05-31T20:00:23.198 INFO:teuthology.orchestra.run.smithi099.stderr: local status: [, , , , , 00000000003.35.10, , , , , ] 2023-05-31T20:00:23.198 INFO:teuthology.orchestra.run.smithi099.stderr: remote markers: [, , , , , 00000000003.35.10, , , 00000000020.47.12, , ] 2023-05-31T20:00:28.198 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:00:28.194+0000 7fa30b7d0780 1 bucket sync caught up with source: 2023-05-31T20:00:28.199 INFO:teuthology.orchestra.run.smithi099.stderr: local status: [, , , , , 00000000003.35.10, , , 00000000031.78.9, , ] 2023-05-31T20:00:28.199 INFO:teuthology.orchestra.run.smithi099.stderr: remote markers: [, , , , , 00000000003.35.10, , , 00000000020.47.12, , ] 2023-05-31T20:00:28.199 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:00:28.194+0000 7fa30b7d0780 0 bucket checkpoint complete 2023-05-31T20:00:28.220 DEBUG:rgw_multi.tests:bucket checkpoint: source=a2 target=a1 bucket=pklxkd-13 2023-05-31T20:00:28.221 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'pklxkd-13', '--source-zone', 'a2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-05-31T20:00:28.221 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'pklxkd-13', '--source-zone', 'a2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-05-31T20:00:28.222 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket pklxkd-13 --source-zone a2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2023-05-31T20:00:28.284 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2023-05-31T20:00:28.284 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2023-05-31T20:00:28.330 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:00:28.325+0000 7f9a9f258780 1 waiting for incremental sync to catch up: 2023-05-31T20:00:28.330 INFO:teuthology.orchestra.run.smithi099.stderr: local status: [, , , , , 00000000002.36.11, , , 00000000024.79.6, , ] 2023-05-31T20:00:28.331 INFO:teuthology.orchestra.run.smithi099.stderr: remote markers: [, , , , , 00000000002.36.11, , , 00000000031.98.12, , ] 2023-05-31T20:00:33.331 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:00:33.326+0000 7f9a9f258780 1 waiting for incremental sync to catch up: 2023-05-31T20:00:33.332 INFO:teuthology.orchestra.run.smithi099.stderr: local status: [, , , , , 00000000002.36.11, , , 00000000024.79.6, , ] 2023-05-31T20:00:33.332 INFO:teuthology.orchestra.run.smithi099.stderr: remote markers: [, , , , , 00000000002.36.11, , , 00000000031.98.12, , ] 2023-05-31T20:00:38.333 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:00:38.328+0000 7f9a9f258780 1 waiting for incremental sync to catch up: 2023-05-31T20:00:38.333 INFO:teuthology.orchestra.run.smithi099.stderr: local status: [, , , , , 00000000002.36.11, , , 00000000024.79.6, , ] 2023-05-31T20:00:38.333 INFO:teuthology.orchestra.run.smithi099.stderr: remote markers: [, , , , , 00000000002.36.11, , , 00000000031.98.12, , ] 2023-05-31T20:00:43.334 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:00:43.329+0000 7f9a9f258780 1 waiting for incremental sync to catch up: 2023-05-31T20:00:43.335 INFO:teuthology.orchestra.run.smithi099.stderr: local status: [, , , , , 00000000002.36.11, , , 00000000024.79.6, , ] 2023-05-31T20:00:43.335 INFO:teuthology.orchestra.run.smithi099.stderr: remote markers: [, , , , , 00000000002.36.11, , , 00000000031.98.12, , ] 2023-05-31T20:00:48.336 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:00:48.330+0000 7f9a9f258780 1 bucket sync caught up with source: 2023-05-31T20:00:48.336 INFO:teuthology.orchestra.run.smithi099.stderr: local status: [, , , , , 00000000002.36.11, , , 00000000031.98.12, , ] 2023-05-31T20:00:48.336 INFO:teuthology.orchestra.run.smithi099.stderr: remote markers: [, , , , , 00000000002.36.11, , , 00000000031.98.12, , ] 2023-05-31T20:00:48.336 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:00:48.330+0000 7f9a9f258780 0 bucket checkpoint complete 2023-05-31T20:00:48.355 INFO:tasks.rgw_multi.zone_rados:comparing bucket=pklxkd-13 zones={a2, a1} 2023-05-31T20:00:48.355 DEBUG:boto:path=/pklxkd-13/ 2023-05-31T20:00:48.356 DEBUG:boto:auth_path=/pklxkd-13/ 2023-05-31T20:00:48.356 DEBUG:boto:Method: HEAD 2023-05-31T20:00:48.356 DEBUG:boto:Path: /pklxkd-13/ 2023-05-31T20:00:48.357 DEBUG:boto:Data: 2023-05-31T20:00:48.357 DEBUG:boto:Headers: {} 2023-05-31T20:00:48.358 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:00:48.358 DEBUG:boto:Port: 8001 2023-05-31T20:00:48.358 DEBUG:boto:Params: {} 2023-05-31T20:00:48.359 DEBUG:boto:Token: None 2023-05-31T20:00:48.359 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:00:48 GMT /pklxkd-13/ 2023-05-31T20:00:48.360 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:1/M/MsHBHoAcZTGqOC1dALopUsQ= 2023-05-31T20:00:48.360 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:00:48 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:1/M/MsHBHoAcZTGqOC1dALopUsQ=', 'Content-Length': '0'} 2023-05-31T20:00:48.363 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '21'), ('X-RGW-Bytes-Used', '160'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000f980fb88a003a2aa-006477a770-1112-a2'), ('Content-Length', '0'), ('Date', 'Wed, 31 May 2023 20:00:48 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:00:48.365 DEBUG:boto:path=/pklxkd-13/ 2023-05-31T20:00:48.366 DEBUG:boto:auth_path=/pklxkd-13/ 2023-05-31T20:00:48.366 DEBUG:boto:Method: HEAD 2023-05-31T20:00:48.367 DEBUG:boto:Path: /pklxkd-13/ 2023-05-31T20:00:48.368 DEBUG:boto:Data: 2023-05-31T20:00:48.369 DEBUG:boto:Headers: {} 2023-05-31T20:00:48.370 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:00:48.370 DEBUG:boto:Port: 8000 2023-05-31T20:00:48.371 DEBUG:boto:Params: {} 2023-05-31T20:00:48.372 DEBUG:boto:Token: None 2023-05-31T20:00:48.372 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:00:48 GMT /pklxkd-13/ 2023-05-31T20:00:48.373 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:1/M/MsHBHoAcZTGqOC1dALopUsQ= 2023-05-31T20:00:48.374 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:00:48 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:1/M/MsHBHoAcZTGqOC1dALopUsQ=', 'Content-Length': '0'} 2023-05-31T20:00:48.376 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '21'), ('X-RGW-Bytes-Used', '160'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000e64c6e88aeab8b51-006477a770-110e-a1'), ('Content-Length', '0'), ('Date', 'Wed, 31 May 2023 20:00:48 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:00:48.377 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2023-05-31T20:00:48.377 DEBUG:boto:path=/pklxkd-13/ 2023-05-31T20:00:48.377 DEBUG:boto:auth_path=/pklxkd-13/ 2023-05-31T20:00:48.378 DEBUG:boto:path=/pklxkd-13/?versions&max-keys=999 2023-05-31T20:00:48.378 DEBUG:boto:auth_path=/pklxkd-13/?versions&max-keys=999 2023-05-31T20:00:48.378 DEBUG:boto:Method: GET 2023-05-31T20:00:48.379 DEBUG:boto:Path: /pklxkd-13/?versions&max-keys=999 2023-05-31T20:00:48.379 DEBUG:boto:Data: 2023-05-31T20:00:48.379 DEBUG:boto:Headers: {} 2023-05-31T20:00:48.380 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:00:48.380 DEBUG:boto:Port: 8001 2023-05-31T20:00:48.381 DEBUG:boto:Params: {} 2023-05-31T20:00:48.381 DEBUG:boto:Token: None 2023-05-31T20:00:48.381 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:00:48 GMT /pklxkd-13/?versions 2023-05-31T20:00:48.382 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:akROOoU7L7W4tp353UhyN0iHl84= 2023-05-31T20:00:48.382 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:00:48 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:akROOoU7L7W4tp353UhyN0iHl84=', 'Content-Length': '0'} 2023-05-31T20:00:48.385 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d04052171c9ce5bf-006477a770-1112-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:00:48 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:00:48.386 DEBUG:boto:b'pklxkd-13999falsedummy1KZ1.YSVQS7.XBkKCHg.G4C.eFHwKQntrue2023-05-31T19:59:36.915Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormalobj.K0qHk3QiH4acREzLteeQLJmFzn4x-1true2023-05-31T20:00:08.058Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjgr0pDclEGPIgGhrGevSqfnt7WtJ89yDfalse2023-05-31T20:00:08.086Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj.q2JxPcqi3BYUXNGXuFQE0hVEIAq1Ykfalse2023-05-31T20:00:08.002Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjnE3onm9L7VUWceoER2mn2jkX5yHK5GLfalse2023-05-31T20:00:07.971Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjLGC78se33tZFR9UpgQStirsP0mP9GEufalse2023-05-31T20:00:07.942Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjoJ31LeYY.9AEY1WQGQs-wBnfC1MDSr6false2023-05-31T20:00:07.887Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjWr6wnEZSP3WRSGvZ8nB1.KinMc4wxvZfalse2023-05-31T20:00:07.866Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjmWkpyutJAg628MwnVVNvQnYfQFw1fqJfalse2023-05-31T20:00:07.843Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjbBAtwbgprRto0Kkb2LYN6Gd-LmGQA5.false2023-05-31T20:00:07.822Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjnD.VY2SV.sxc3KR89HB3qbvIEu7ie0Dfalse2023-05-31T20:00:07.764Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjBCM6vBdxKthZLIE0R8UhG0.omDRBvP7false2023-05-31T20:00:07.741Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjRsWecIE0wyYuGR3yXB85CvZ9iY5NhBAfalse2023-05-31T20:00:07.712Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjJR8T3JjcBmfQYfHtj3icgZUj81rYFmsfalse2023-05-31T20:00:07.613Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjooEeuYF3vuTJzW20FXOdWWO9-xNIuanfalse2023-05-31T20:00:07.653Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjO9dH9NhVFdlcrFlXwres-hSyGvDhDaQfalse2023-05-31T20:00:07.535Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjaS0yzteNN24E5q0Twvp.6Z7sdzsi2hwfalse2023-05-31T20:00:07.591Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjK5b8GHNb3Mn1saWR47Y7KNPxakeIWEcfalse2023-05-31T20:00:07.490Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjp5Ybj.5dD2GwjElOd3T2Sa50bcy.UgEfalse2023-05-31T20:00:07.512Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj0DuL1bo3E1FXXi7mJ3oxDT6slXlrtRAfalse2023-05-31T20:00:07.406Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjpm7-Sh2bsjDbtvOCwujAMZA4JCZRKlhfalse2023-05-31T20:00:07.435Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:00:48.388 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2023-05-31T20:00:48.388 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2023-05-31T20:00:48.388 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2023-05-31T20:00:48.389 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2023-05-31T20:00:48.389 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2023-05-31T20:00:48.389 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2023-05-31T20:00:48.390 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2023-05-31T20:00:48.390 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2023-05-31T20:00:48.390 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2023-05-31T20:00:48.391 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2023-05-31T20:00:48.391 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2023-05-31T20:00:48.391 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2023-05-31T20:00:48.391 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2023-05-31T20:00:48.392 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2023-05-31T20:00:48.392 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2023-05-31T20:00:48.393 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2023-05-31T20:00:48.393 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2023-05-31T20:00:48.393 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2023-05-31T20:00:48.394 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2023-05-31T20:00:48.394 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2023-05-31T20:00:48.395 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2023-05-31T20:00:48.395 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2023-05-31T20:00:48.395 DEBUG:boto:path=/pklxkd-13/ 2023-05-31T20:00:48.396 DEBUG:boto:auth_path=/pklxkd-13/ 2023-05-31T20:00:48.396 DEBUG:boto:path=/pklxkd-13/?versions&max-keys=999 2023-05-31T20:00:48.397 DEBUG:boto:auth_path=/pklxkd-13/?versions&max-keys=999 2023-05-31T20:00:48.397 DEBUG:boto:Method: GET 2023-05-31T20:00:48.397 DEBUG:boto:Path: /pklxkd-13/?versions&max-keys=999 2023-05-31T20:00:48.398 DEBUG:boto:Data: 2023-05-31T20:00:48.398 DEBUG:boto:Headers: {} 2023-05-31T20:00:48.398 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:00:48.399 DEBUG:boto:Port: 8000 2023-05-31T20:00:48.399 DEBUG:boto:Params: {} 2023-05-31T20:00:48.399 DEBUG:boto:Token: None 2023-05-31T20:00:48.400 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:00:48 GMT /pklxkd-13/?versions 2023-05-31T20:00:48.400 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:akROOoU7L7W4tp353UhyN0iHl84= 2023-05-31T20:00:48.401 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:00:48 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:akROOoU7L7W4tp353UhyN0iHl84=', 'Content-Length': '0'} 2023-05-31T20:00:48.404 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000000c63ee4837d2072b-006477a770-110e-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:00:48 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:00:48.405 DEBUG:boto:b'pklxkd-13999falsedummy1KZ1.YSVQS7.XBkKCHg.G4C.eFHwKQntrue2023-05-31T19:59:36.915Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormalobj.K0qHk3QiH4acREzLteeQLJmFzn4x-1true2023-05-31T20:00:08.058Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjgr0pDclEGPIgGhrGevSqfnt7WtJ89yDfalse2023-05-31T20:00:08.086Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj.q2JxPcqi3BYUXNGXuFQE0hVEIAq1Ykfalse2023-05-31T20:00:08.002Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjnE3onm9L7VUWceoER2mn2jkX5yHK5GLfalse2023-05-31T20:00:07.971Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjLGC78se33tZFR9UpgQStirsP0mP9GEufalse2023-05-31T20:00:07.942Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjoJ31LeYY.9AEY1WQGQs-wBnfC1MDSr6false2023-05-31T20:00:07.887Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjWr6wnEZSP3WRSGvZ8nB1.KinMc4wxvZfalse2023-05-31T20:00:07.866Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjmWkpyutJAg628MwnVVNvQnYfQFw1fqJfalse2023-05-31T20:00:07.843Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjbBAtwbgprRto0Kkb2LYN6Gd-LmGQA5.false2023-05-31T20:00:07.822Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjnD.VY2SV.sxc3KR89HB3qbvIEu7ie0Dfalse2023-05-31T20:00:07.764Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjBCM6vBdxKthZLIE0R8UhG0.omDRBvP7false2023-05-31T20:00:07.741Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjRsWecIE0wyYuGR3yXB85CvZ9iY5NhBAfalse2023-05-31T20:00:07.712Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjJR8T3JjcBmfQYfHtj3icgZUj81rYFmsfalse2023-05-31T20:00:07.613Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjooEeuYF3vuTJzW20FXOdWWO9-xNIuanfalse2023-05-31T20:00:07.653Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjO9dH9NhVFdlcrFlXwres-hSyGvDhDaQfalse2023-05-31T20:00:07.535Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjaS0yzteNN24E5q0Twvp.6Z7sdzsi2hwfalse2023-05-31T20:00:07.591Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjK5b8GHNb3Mn1saWR47Y7KNPxakeIWEcfalse2023-05-31T20:00:07.490Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjp5Ybj.5dD2GwjElOd3T2Sa50bcy.UgEfalse2023-05-31T20:00:07.512Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj0DuL1bo3E1FXXi7mJ3oxDT6slXlrtRAfalse2023-05-31T20:00:07.406Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjpm7-Sh2bsjDbtvOCwujAMZA4JCZRKlhfalse2023-05-31T20:00:07.435Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:00:48.407 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2023-05-31T20:00:48.407 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2023-05-31T20:00:48.407 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2023-05-31T20:00:48.408 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2023-05-31T20:00:48.408 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2023-05-31T20:00:48.408 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2023-05-31T20:00:48.409 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2023-05-31T20:00:48.409 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2023-05-31T20:00:48.409 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2023-05-31T20:00:48.410 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2023-05-31T20:00:48.410 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2023-05-31T20:00:48.411 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2023-05-31T20:00:48.411 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2023-05-31T20:00:48.412 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2023-05-31T20:00:48.412 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2023-05-31T20:00:48.412 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2023-05-31T20:00:48.413 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2023-05-31T20:00:48.413 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2023-05-31T20:00:48.413 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2023-05-31T20:00:48.414 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2023-05-31T20:00:48.414 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2023-05-31T20:00:48.414 DEBUG:boto:path=/pklxkd-13/ 2023-05-31T20:00:48.415 DEBUG:boto:auth_path=/pklxkd-13/ 2023-05-31T20:00:48.415 DEBUG:boto:path=/pklxkd-13/?versions&max-keys=999 2023-05-31T20:00:48.415 DEBUG:boto:auth_path=/pklxkd-13/?versions&max-keys=999 2023-05-31T20:00:48.416 DEBUG:boto:Method: GET 2023-05-31T20:00:48.416 DEBUG:boto:Path: /pklxkd-13/?versions&max-keys=999 2023-05-31T20:00:48.417 DEBUG:boto:Data: 2023-05-31T20:00:48.417 DEBUG:boto:Headers: {} 2023-05-31T20:00:48.417 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:00:48.418 DEBUG:boto:Port: 8001 2023-05-31T20:00:48.418 DEBUG:boto:Params: {} 2023-05-31T20:00:48.418 DEBUG:boto:Token: None 2023-05-31T20:00:48.419 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:00:48 GMT /pklxkd-13/?versions 2023-05-31T20:00:48.419 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:akROOoU7L7W4tp353UhyN0iHl84= 2023-05-31T20:00:48.420 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:00:48 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:akROOoU7L7W4tp353UhyN0iHl84=', 'Content-Length': '0'} 2023-05-31T20:00:48.423 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000dbcb87472f12231c-006477a770-1112-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:00:48 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:00:48.423 DEBUG:boto:b'pklxkd-13999falsedummy1KZ1.YSVQS7.XBkKCHg.G4C.eFHwKQntrue2023-05-31T19:59:36.915Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormalobj.K0qHk3QiH4acREzLteeQLJmFzn4x-1true2023-05-31T20:00:08.058Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjgr0pDclEGPIgGhrGevSqfnt7WtJ89yDfalse2023-05-31T20:00:08.086Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj.q2JxPcqi3BYUXNGXuFQE0hVEIAq1Ykfalse2023-05-31T20:00:08.002Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjnE3onm9L7VUWceoER2mn2jkX5yHK5GLfalse2023-05-31T20:00:07.971Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjLGC78se33tZFR9UpgQStirsP0mP9GEufalse2023-05-31T20:00:07.942Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjoJ31LeYY.9AEY1WQGQs-wBnfC1MDSr6false2023-05-31T20:00:07.887Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjWr6wnEZSP3WRSGvZ8nB1.KinMc4wxvZfalse2023-05-31T20:00:07.866Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjmWkpyutJAg628MwnVVNvQnYfQFw1fqJfalse2023-05-31T20:00:07.843Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjbBAtwbgprRto0Kkb2LYN6Gd-LmGQA5.false2023-05-31T20:00:07.822Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjnD.VY2SV.sxc3KR89HB3qbvIEu7ie0Dfalse2023-05-31T20:00:07.764Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjBCM6vBdxKthZLIE0R8UhG0.omDRBvP7false2023-05-31T20:00:07.741Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjRsWecIE0wyYuGR3yXB85CvZ9iY5NhBAfalse2023-05-31T20:00:07.712Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjJR8T3JjcBmfQYfHtj3icgZUj81rYFmsfalse2023-05-31T20:00:07.613Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjooEeuYF3vuTJzW20FXOdWWO9-xNIuanfalse2023-05-31T20:00:07.653Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjO9dH9NhVFdlcrFlXwres-hSyGvDhDaQfalse2023-05-31T20:00:07.535Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjaS0yzteNN24E5q0Twvp.6Z7sdzsi2hwfalse2023-05-31T20:00:07.591Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjK5b8GHNb3Mn1saWR47Y7KNPxakeIWEcfalse2023-05-31T20:00:07.490Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjp5Ybj.5dD2GwjElOd3T2Sa50bcy.UgEfalse2023-05-31T20:00:07.512Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj0DuL1bo3E1FXXi7mJ3oxDT6slXlrtRAfalse2023-05-31T20:00:07.406Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjpm7-Sh2bsjDbtvOCwujAMZA4JCZRKlhfalse2023-05-31T20:00:07.435Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:00:48.425 DEBUG:boto:path=/pklxkd-13/ 2023-05-31T20:00:48.426 DEBUG:boto:auth_path=/pklxkd-13/ 2023-05-31T20:00:48.426 DEBUG:boto:path=/pklxkd-13/?versions&max-keys=999 2023-05-31T20:00:48.426 DEBUG:boto:auth_path=/pklxkd-13/?versions&max-keys=999 2023-05-31T20:00:48.427 DEBUG:boto:Method: GET 2023-05-31T20:00:48.427 DEBUG:boto:Path: /pklxkd-13/?versions&max-keys=999 2023-05-31T20:00:48.427 DEBUG:boto:Data: 2023-05-31T20:00:48.428 DEBUG:boto:Headers: {} 2023-05-31T20:00:48.428 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:00:48.429 DEBUG:boto:Port: 8000 2023-05-31T20:00:48.429 DEBUG:boto:Params: {} 2023-05-31T20:00:48.429 DEBUG:boto:Token: None 2023-05-31T20:00:48.430 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:00:48 GMT /pklxkd-13/?versions 2023-05-31T20:00:48.431 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:akROOoU7L7W4tp353UhyN0iHl84= 2023-05-31T20:00:48.431 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:00:48 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:akROOoU7L7W4tp353UhyN0iHl84=', 'Content-Length': '0'} 2023-05-31T20:00:48.434 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a06ed96f195158d6-006477a770-110e-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:00:48 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:00:48.434 DEBUG:boto:b'pklxkd-13999falsedummy1KZ1.YSVQS7.XBkKCHg.G4C.eFHwKQntrue2023-05-31T19:59:36.915Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormalobj.K0qHk3QiH4acREzLteeQLJmFzn4x-1true2023-05-31T20:00:08.058Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjgr0pDclEGPIgGhrGevSqfnt7WtJ89yDfalse2023-05-31T20:00:08.086Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj.q2JxPcqi3BYUXNGXuFQE0hVEIAq1Ykfalse2023-05-31T20:00:08.002Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjnE3onm9L7VUWceoER2mn2jkX5yHK5GLfalse2023-05-31T20:00:07.971Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjLGC78se33tZFR9UpgQStirsP0mP9GEufalse2023-05-31T20:00:07.942Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjoJ31LeYY.9AEY1WQGQs-wBnfC1MDSr6false2023-05-31T20:00:07.887Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjWr6wnEZSP3WRSGvZ8nB1.KinMc4wxvZfalse2023-05-31T20:00:07.866Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjmWkpyutJAg628MwnVVNvQnYfQFw1fqJfalse2023-05-31T20:00:07.843Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjbBAtwbgprRto0Kkb2LYN6Gd-LmGQA5.false2023-05-31T20:00:07.822Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjnD.VY2SV.sxc3KR89HB3qbvIEu7ie0Dfalse2023-05-31T20:00:07.764Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjBCM6vBdxKthZLIE0R8UhG0.omDRBvP7false2023-05-31T20:00:07.741Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjRsWecIE0wyYuGR3yXB85CvZ9iY5NhBAfalse2023-05-31T20:00:07.712Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjJR8T3JjcBmfQYfHtj3icgZUj81rYFmsfalse2023-05-31T20:00:07.613Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjooEeuYF3vuTJzW20FXOdWWO9-xNIuanfalse2023-05-31T20:00:07.653Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjO9dH9NhVFdlcrFlXwres-hSyGvDhDaQfalse2023-05-31T20:00:07.535Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjaS0yzteNN24E5q0Twvp.6Z7sdzsi2hwfalse2023-05-31T20:00:07.591Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjK5b8GHNb3Mn1saWR47Y7KNPxakeIWEcfalse2023-05-31T20:00:07.490Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjp5Ybj.5dD2GwjElOd3T2Sa50bcy.UgEfalse2023-05-31T20:00:07.512Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj0DuL1bo3E1FXXi7mJ3oxDT6slXlrtRAfalse2023-05-31T20:00:07.406Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjpm7-Sh2bsjDbtvOCwujAMZA4JCZRKlhfalse2023-05-31T20:00:07.435Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:00:48.436 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2023-05-31T20:00:48.437 DEBUG:boto:path=/pklxkd-13/dummy 2023-05-31T20:00:48.437 DEBUG:boto:auth_path=/pklxkd-13/dummy 2023-05-31T20:00:48.437 DEBUG:boto:path=/pklxkd-13/dummy?versionId=1KZ1.YSVQS7.XBkKCHg.G4C.eFHwKQn 2023-05-31T20:00:48.438 DEBUG:boto:auth_path=/pklxkd-13/dummy?versionId=1KZ1.YSVQS7.XBkKCHg.G4C.eFHwKQn 2023-05-31T20:00:48.438 DEBUG:boto:Method: GET 2023-05-31T20:00:48.438 DEBUG:boto:Path: /pklxkd-13/dummy?versionId=1KZ1.YSVQS7.XBkKCHg.G4C.eFHwKQn 2023-05-31T20:00:48.439 DEBUG:boto:Data: 2023-05-31T20:00:48.439 DEBUG:boto:Headers: {} 2023-05-31T20:00:48.439 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:00:48.440 DEBUG:boto:Port: 8001 2023-05-31T20:00:48.440 DEBUG:boto:Params: {} 2023-05-31T20:00:48.441 DEBUG:boto:Token: None 2023-05-31T20:00:48.441 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:00:48 GMT /pklxkd-13/dummy?versionId=1KZ1.YSVQS7.XBkKCHg.G4C.eFHwKQn 2023-05-31T20:00:48.441 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:VayVOJhEBIpmFByZ5XD2FesJfVk= 2023-05-31T20:00:48.442 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:00:48 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:VayVOJhEBIpmFByZ5XD2FesJfVk=', 'Content-Length': '0'} 2023-05-31T20:00:48.444 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 19:59:36 GMT'), ('x-amz-version-id', '1KZ1.YSVQS7.XBkKCHg.G4C.eFHwKQn'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000819f837018609ee0-006477a770-1112-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:00:48 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:00:48.445 DEBUG:boto:path=/pklxkd-13/dummy 2023-05-31T20:00:48.445 DEBUG:boto:auth_path=/pklxkd-13/dummy 2023-05-31T20:00:48.446 DEBUG:boto:path=/pklxkd-13/dummy?versionId=1KZ1.YSVQS7.XBkKCHg.G4C.eFHwKQn 2023-05-31T20:00:48.446 DEBUG:boto:auth_path=/pklxkd-13/dummy?versionId=1KZ1.YSVQS7.XBkKCHg.G4C.eFHwKQn 2023-05-31T20:00:48.446 DEBUG:boto:Method: GET 2023-05-31T20:00:48.447 DEBUG:boto:Path: /pklxkd-13/dummy?versionId=1KZ1.YSVQS7.XBkKCHg.G4C.eFHwKQn 2023-05-31T20:00:48.447 DEBUG:boto:Data: 2023-05-31T20:00:48.448 DEBUG:boto:Headers: {} 2023-05-31T20:00:48.448 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:00:48.448 DEBUG:boto:Port: 8000 2023-05-31T20:00:48.449 DEBUG:boto:Params: {} 2023-05-31T20:00:48.449 DEBUG:boto:Token: None 2023-05-31T20:00:48.450 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:00:48 GMT /pklxkd-13/dummy?versionId=1KZ1.YSVQS7.XBkKCHg.G4C.eFHwKQn 2023-05-31T20:00:48.450 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:VayVOJhEBIpmFByZ5XD2FesJfVk= 2023-05-31T20:00:48.450 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:00:48 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:VayVOJhEBIpmFByZ5XD2FesJfVk=', 'Content-Length': '0'} 2023-05-31T20:00:48.452 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 19:59:36 GMT'), ('x-amz-version-id', '1KZ1.YSVQS7.XBkKCHg.G4C.eFHwKQn'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000009f55315326e06b9d-006477a770-110e-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:00:48 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:00:48.453 DEBUG:boto:path=/pklxkd-13/dummy 2023-05-31T20:00:48.454 DEBUG:boto:auth_path=/pklxkd-13/dummy 2023-05-31T20:00:48.454 DEBUG:boto:path=/pklxkd-13/dummy?versionId=1KZ1.YSVQS7.XBkKCHg.G4C.eFHwKQn 2023-05-31T20:00:48.455 DEBUG:boto:auth_path=/pklxkd-13/dummy?versionId=1KZ1.YSVQS7.XBkKCHg.G4C.eFHwKQn 2023-05-31T20:00:48.455 DEBUG:boto:Method: HEAD 2023-05-31T20:00:48.456 DEBUG:boto:Path: /pklxkd-13/dummy?versionId=1KZ1.YSVQS7.XBkKCHg.G4C.eFHwKQn 2023-05-31T20:00:48.456 DEBUG:boto:Data: 2023-05-31T20:00:48.457 DEBUG:boto:Headers: {} 2023-05-31T20:00:48.457 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:00:48.458 DEBUG:boto:Port: 8001 2023-05-31T20:00:48.458 DEBUG:boto:Params: {} 2023-05-31T20:00:48.459 DEBUG:boto:Token: None 2023-05-31T20:00:48.460 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:00:48 GMT /pklxkd-13/dummy?versionId=1KZ1.YSVQS7.XBkKCHg.G4C.eFHwKQn 2023-05-31T20:00:48.460 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:tn8BTWt6fp8FStu7Z/M4ldUu/4Q= 2023-05-31T20:00:48.461 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:00:48 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:tn8BTWt6fp8FStu7Z/M4ldUu/4Q=', 'Content-Length': '0'} 2023-05-31T20:00:48.463 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 19:59:36 GMT'), ('x-amz-version-id', '1KZ1.YSVQS7.XBkKCHg.G4C.eFHwKQn'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000005f72d47ed9efd236-006477a770-1112-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:00:48 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:00:48.464 DEBUG:boto:path=/pklxkd-13/dummy 2023-05-31T20:00:48.465 DEBUG:boto:auth_path=/pklxkd-13/dummy 2023-05-31T20:00:48.465 DEBUG:boto:path=/pklxkd-13/dummy?versionId=1KZ1.YSVQS7.XBkKCHg.G4C.eFHwKQn 2023-05-31T20:00:48.466 DEBUG:boto:auth_path=/pklxkd-13/dummy?versionId=1KZ1.YSVQS7.XBkKCHg.G4C.eFHwKQn 2023-05-31T20:00:48.466 DEBUG:boto:Method: HEAD 2023-05-31T20:00:48.467 DEBUG:boto:Path: /pklxkd-13/dummy?versionId=1KZ1.YSVQS7.XBkKCHg.G4C.eFHwKQn 2023-05-31T20:00:48.468 DEBUG:boto:Data: 2023-05-31T20:00:48.468 DEBUG:boto:Headers: {} 2023-05-31T20:00:48.469 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:00:48.470 DEBUG:boto:Port: 8000 2023-05-31T20:00:48.470 DEBUG:boto:Params: {} 2023-05-31T20:00:48.470 DEBUG:boto:Token: None 2023-05-31T20:00:48.471 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:00:48 GMT /pklxkd-13/dummy?versionId=1KZ1.YSVQS7.XBkKCHg.G4C.eFHwKQn 2023-05-31T20:00:48.471 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:tn8BTWt6fp8FStu7Z/M4ldUu/4Q= 2023-05-31T20:00:48.472 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:00:48 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:tn8BTWt6fp8FStu7Z/M4ldUu/4Q=', 'Content-Length': '0'} 2023-05-31T20:00:48.473 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 19:59:36 GMT'), ('x-amz-version-id', '1KZ1.YSVQS7.XBkKCHg.G4C.eFHwKQn'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx0000062902a8643708a13-006477a770-110e-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:00:48 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:00:48.474 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2023-05-31T20:00:48.474 DEBUG:boto:path=/pklxkd-13/dummy 2023-05-31T20:00:48.475 DEBUG:boto:auth_path=/pklxkd-13/dummy 2023-05-31T20:00:48.475 DEBUG:boto:path=/pklxkd-13/dummy?versionId=1KZ1.YSVQS7.XBkKCHg.G4C.eFHwKQn 2023-05-31T20:00:48.475 DEBUG:boto:auth_path=/pklxkd-13/dummy?versionId=1KZ1.YSVQS7.XBkKCHg.G4C.eFHwKQn 2023-05-31T20:00:48.476 DEBUG:boto:Method: GET 2023-05-31T20:00:48.476 DEBUG:boto:Path: /pklxkd-13/dummy?versionId=1KZ1.YSVQS7.XBkKCHg.G4C.eFHwKQn 2023-05-31T20:00:48.477 DEBUG:boto:Data: 2023-05-31T20:00:48.477 DEBUG:boto:Headers: {} 2023-05-31T20:00:48.477 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:00:48.478 DEBUG:boto:Port: 8001 2023-05-31T20:00:48.478 DEBUG:boto:Params: {} 2023-05-31T20:00:48.479 DEBUG:boto:Token: None 2023-05-31T20:00:48.479 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:00:48 GMT /pklxkd-13/dummy?versionId=1KZ1.YSVQS7.XBkKCHg.G4C.eFHwKQn 2023-05-31T20:00:48.479 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:VayVOJhEBIpmFByZ5XD2FesJfVk= 2023-05-31T20:00:48.480 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:00:48 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:VayVOJhEBIpmFByZ5XD2FesJfVk=', 'Content-Length': '0'} 2023-05-31T20:00:48.482 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 19:59:36 GMT'), ('x-amz-version-id', '1KZ1.YSVQS7.XBkKCHg.G4C.eFHwKQn'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000008e4ab6295cd241fe-006477a770-1112-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:00:48 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:00:48.482 DEBUG:boto:path=/pklxkd-13/dummy 2023-05-31T20:00:48.483 DEBUG:boto:auth_path=/pklxkd-13/dummy 2023-05-31T20:00:48.483 DEBUG:boto:path=/pklxkd-13/dummy?versionId=1KZ1.YSVQS7.XBkKCHg.G4C.eFHwKQn 2023-05-31T20:00:48.483 DEBUG:boto:auth_path=/pklxkd-13/dummy?versionId=1KZ1.YSVQS7.XBkKCHg.G4C.eFHwKQn 2023-05-31T20:00:48.484 DEBUG:boto:Method: GET 2023-05-31T20:00:48.484 DEBUG:boto:Path: /pklxkd-13/dummy?versionId=1KZ1.YSVQS7.XBkKCHg.G4C.eFHwKQn 2023-05-31T20:00:48.485 DEBUG:boto:Data: 2023-05-31T20:00:48.485 DEBUG:boto:Headers: {} 2023-05-31T20:00:48.485 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:00:48.486 DEBUG:boto:Port: 8000 2023-05-31T20:00:48.486 DEBUG:boto:Params: {} 2023-05-31T20:00:48.486 DEBUG:boto:Token: None 2023-05-31T20:00:48.487 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:00:48 GMT /pklxkd-13/dummy?versionId=1KZ1.YSVQS7.XBkKCHg.G4C.eFHwKQn 2023-05-31T20:00:48.488 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:VayVOJhEBIpmFByZ5XD2FesJfVk= 2023-05-31T20:00:48.488 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:00:48 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:VayVOJhEBIpmFByZ5XD2FesJfVk=', 'Content-Length': '0'} 2023-05-31T20:00:48.490 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 19:59:36 GMT'), ('x-amz-version-id', '1KZ1.YSVQS7.XBkKCHg.G4C.eFHwKQn'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000007b89898797242046-006477a770-110e-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:00:48 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:00:48.491 DEBUG:boto:path=/pklxkd-13/ 2023-05-31T20:00:48.491 DEBUG:boto:auth_path=/pklxkd-13/ 2023-05-31T20:00:48.491 DEBUG:boto:path=/pklxkd-13/?prefix=dummy 2023-05-31T20:00:48.492 DEBUG:boto:auth_path=/pklxkd-13/?prefix=dummy 2023-05-31T20:00:48.492 DEBUG:boto:Method: GET 2023-05-31T20:00:48.492 DEBUG:boto:Path: /pklxkd-13/?prefix=dummy 2023-05-31T20:00:48.493 DEBUG:boto:Data: 2023-05-31T20:00:48.493 DEBUG:boto:Headers: {} 2023-05-31T20:00:48.493 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:00:48.494 DEBUG:boto:Port: 8001 2023-05-31T20:00:48.494 DEBUG:boto:Params: {} 2023-05-31T20:00:48.495 DEBUG:boto:Token: None 2023-05-31T20:00:48.495 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:00:48 GMT /pklxkd-13/ 2023-05-31T20:00:48.496 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:41Yoo9/eFVv+2e+S6Z9kRaR3sIg= 2023-05-31T20:00:48.496 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:00:48 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:41Yoo9/eFVv+2e+S6Z9kRaR3sIg=', 'Content-Length': '0'} 2023-05-31T20:00:48.499 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000040e4b8dc4cb74cc0-006477a770-1112-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:00:48 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:00:48.499 DEBUG:boto:b'pklxkd-13dummy1000falsedummy2023-05-31T19:59:36.915Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:00:48.500 DEBUG:boto:path=/pklxkd-13/ 2023-05-31T20:00:48.500 DEBUG:boto:auth_path=/pklxkd-13/ 2023-05-31T20:00:48.501 DEBUG:boto:path=/pklxkd-13/?prefix=dummy 2023-05-31T20:00:48.501 DEBUG:boto:auth_path=/pklxkd-13/?prefix=dummy 2023-05-31T20:00:48.502 DEBUG:boto:Method: GET 2023-05-31T20:00:48.502 DEBUG:boto:Path: /pklxkd-13/?prefix=dummy 2023-05-31T20:00:48.502 DEBUG:boto:Data: 2023-05-31T20:00:48.503 DEBUG:boto:Headers: {} 2023-05-31T20:00:48.503 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:00:48.504 DEBUG:boto:Port: 8000 2023-05-31T20:00:48.504 DEBUG:boto:Params: {} 2023-05-31T20:00:48.504 DEBUG:boto:Token: None 2023-05-31T20:00:48.505 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:00:48 GMT /pklxkd-13/ 2023-05-31T20:00:48.505 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:41Yoo9/eFVv+2e+S6Z9kRaR3sIg= 2023-05-31T20:00:48.506 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:00:48 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:41Yoo9/eFVv+2e+S6Z9kRaR3sIg=', 'Content-Length': '0'} 2023-05-31T20:00:48.509 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000097ffcb0b54d9fb47-006477a770-110e-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:00:48 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:00:48.552 DEBUG:boto:b'pklxkd-13dummy1000falsedummy2023-05-31T19:59:36.915Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:00:48.553 DEBUG:boto:path=/pklxkd-13/dummy 2023-05-31T20:00:48.553 DEBUG:boto:auth_path=/pklxkd-13/dummy 2023-05-31T20:00:48.554 DEBUG:boto:Method: HEAD 2023-05-31T20:00:48.554 DEBUG:boto:Path: /pklxkd-13/dummy 2023-05-31T20:00:48.555 DEBUG:boto:Data: 2023-05-31T20:00:48.555 DEBUG:boto:Headers: {} 2023-05-31T20:00:48.555 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:00:48.556 DEBUG:boto:Port: 8001 2023-05-31T20:00:48.556 DEBUG:boto:Params: {} 2023-05-31T20:00:48.556 DEBUG:boto:Token: None 2023-05-31T20:00:48.557 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:00:48 GMT /pklxkd-13/dummy 2023-05-31T20:00:48.557 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:k1maYEF2Nxefo4y5S+WaSGeoslU= 2023-05-31T20:00:48.558 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:00:48 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:k1maYEF2Nxefo4y5S+WaSGeoslU=', 'Content-Length': '0'} 2023-05-31T20:00:48.560 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 19:59:36 GMT'), ('x-amz-version-id', '1KZ1.YSVQS7.XBkKCHg.G4C.eFHwKQn'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000dd70a3ae26550f92-006477a770-1112-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:00:48 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:00:48.560 DEBUG:boto:path=/pklxkd-13/dummy 2023-05-31T20:00:48.561 DEBUG:boto:auth_path=/pklxkd-13/dummy 2023-05-31T20:00:48.561 DEBUG:boto:Method: HEAD 2023-05-31T20:00:48.561 DEBUG:boto:Path: /pklxkd-13/dummy 2023-05-31T20:00:48.562 DEBUG:boto:Data: 2023-05-31T20:00:48.562 DEBUG:boto:Headers: {} 2023-05-31T20:00:48.562 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:00:48.563 DEBUG:boto:Port: 8000 2023-05-31T20:00:48.563 DEBUG:boto:Params: {} 2023-05-31T20:00:48.563 DEBUG:boto:Token: None 2023-05-31T20:00:48.564 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:00:48 GMT /pklxkd-13/dummy 2023-05-31T20:00:48.564 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:k1maYEF2Nxefo4y5S+WaSGeoslU= 2023-05-31T20:00:48.565 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:00:48 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:k1maYEF2Nxefo4y5S+WaSGeoslU=', 'Content-Length': '0'} 2023-05-31T20:00:48.567 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 19:59:36 GMT'), ('x-amz-version-id', '1KZ1.YSVQS7.XBkKCHg.G4C.eFHwKQn'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx0000009f32e98d123cd6f-006477a770-110e-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:00:48 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:00:48.568 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2023-05-31T20:00:48.568 DEBUG:boto:path=/pklxkd-13/dummy 2023-05-31T20:00:48.568 DEBUG:boto:auth_path=/pklxkd-13/dummy 2023-05-31T20:00:48.569 DEBUG:boto:path=/pklxkd-13/dummy?versionId=1KZ1.YSVQS7.XBkKCHg.G4C.eFHwKQn 2023-05-31T20:00:48.569 DEBUG:boto:auth_path=/pklxkd-13/dummy?versionId=1KZ1.YSVQS7.XBkKCHg.G4C.eFHwKQn 2023-05-31T20:00:48.570 DEBUG:boto:Method: GET 2023-05-31T20:00:48.570 DEBUG:boto:Path: /pklxkd-13/dummy?versionId=1KZ1.YSVQS7.XBkKCHg.G4C.eFHwKQn 2023-05-31T20:00:48.570 DEBUG:boto:Data: 2023-05-31T20:00:48.571 DEBUG:boto:Headers: {} 2023-05-31T20:00:48.571 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:00:48.571 DEBUG:boto:Port: 8001 2023-05-31T20:00:48.572 DEBUG:boto:Params: {} 2023-05-31T20:00:48.572 DEBUG:boto:Token: None 2023-05-31T20:00:48.573 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:00:48 GMT /pklxkd-13/dummy?versionId=1KZ1.YSVQS7.XBkKCHg.G4C.eFHwKQn 2023-05-31T20:00:48.573 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:VayVOJhEBIpmFByZ5XD2FesJfVk= 2023-05-31T20:00:48.573 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:00:48 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:VayVOJhEBIpmFByZ5XD2FesJfVk=', 'Content-Length': '0'} 2023-05-31T20:00:48.575 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 19:59:36 GMT'), ('x-amz-version-id', '1KZ1.YSVQS7.XBkKCHg.G4C.eFHwKQn'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx0000036aa390768640e09-006477a770-1112-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:00:48 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:00:48.576 DEBUG:boto:path=/pklxkd-13/dummy 2023-05-31T20:00:48.576 DEBUG:boto:auth_path=/pklxkd-13/dummy 2023-05-31T20:00:48.576 DEBUG:boto:path=/pklxkd-13/dummy?versionId=1KZ1.YSVQS7.XBkKCHg.G4C.eFHwKQn 2023-05-31T20:00:48.577 DEBUG:boto:auth_path=/pklxkd-13/dummy?versionId=1KZ1.YSVQS7.XBkKCHg.G4C.eFHwKQn 2023-05-31T20:00:48.577 DEBUG:boto:Method: GET 2023-05-31T20:00:48.578 DEBUG:boto:Path: /pklxkd-13/dummy?versionId=1KZ1.YSVQS7.XBkKCHg.G4C.eFHwKQn 2023-05-31T20:00:48.578 DEBUG:boto:Data: 2023-05-31T20:00:48.578 DEBUG:boto:Headers: {} 2023-05-31T20:00:48.579 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:00:48.579 DEBUG:boto:Port: 8000 2023-05-31T20:00:48.579 DEBUG:boto:Params: {} 2023-05-31T20:00:48.580 DEBUG:boto:Token: None 2023-05-31T20:00:48.580 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:00:48 GMT /pklxkd-13/dummy?versionId=1KZ1.YSVQS7.XBkKCHg.G4C.eFHwKQn 2023-05-31T20:00:48.581 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:VayVOJhEBIpmFByZ5XD2FesJfVk= 2023-05-31T20:00:48.581 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:00:48 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:VayVOJhEBIpmFByZ5XD2FesJfVk=', 'Content-Length': '0'} 2023-05-31T20:00:48.583 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 19:59:36 GMT'), ('x-amz-version-id', '1KZ1.YSVQS7.XBkKCHg.G4C.eFHwKQn'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000b17b43496112d2fc-006477a770-110e-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:00:48 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:00:48.583 DEBUG:boto:path=/pklxkd-13/ 2023-05-31T20:00:48.584 DEBUG:boto:auth_path=/pklxkd-13/ 2023-05-31T20:00:48.584 DEBUG:boto:path=/pklxkd-13/?prefix=dummy 2023-05-31T20:00:48.584 DEBUG:boto:auth_path=/pklxkd-13/?prefix=dummy 2023-05-31T20:00:48.585 DEBUG:boto:Method: GET 2023-05-31T20:00:48.585 DEBUG:boto:Path: /pklxkd-13/?prefix=dummy 2023-05-31T20:00:48.585 DEBUG:boto:Data: 2023-05-31T20:00:48.586 DEBUG:boto:Headers: {} 2023-05-31T20:00:48.586 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:00:48.586 DEBUG:boto:Port: 8001 2023-05-31T20:00:48.587 DEBUG:boto:Params: {} 2023-05-31T20:00:48.587 DEBUG:boto:Token: None 2023-05-31T20:00:48.588 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:00:48 GMT /pklxkd-13/ 2023-05-31T20:00:48.588 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:41Yoo9/eFVv+2e+S6Z9kRaR3sIg= 2023-05-31T20:00:48.589 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:00:48 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:41Yoo9/eFVv+2e+S6Z9kRaR3sIg=', 'Content-Length': '0'} 2023-05-31T20:00:48.591 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000002e3d6127c675fea4-006477a770-1112-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:00:48 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:00:48.592 DEBUG:boto:b'pklxkd-13dummy1000falsedummy2023-05-31T19:59:36.915Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:00:48.592 DEBUG:boto:path=/pklxkd-13/ 2023-05-31T20:00:48.593 DEBUG:boto:auth_path=/pklxkd-13/ 2023-05-31T20:00:48.593 DEBUG:boto:path=/pklxkd-13/?prefix=dummy 2023-05-31T20:00:48.594 DEBUG:boto:auth_path=/pklxkd-13/?prefix=dummy 2023-05-31T20:00:48.594 DEBUG:boto:Method: GET 2023-05-31T20:00:48.594 DEBUG:boto:Path: /pklxkd-13/?prefix=dummy 2023-05-31T20:00:48.595 DEBUG:boto:Data: 2023-05-31T20:00:48.595 DEBUG:boto:Headers: {} 2023-05-31T20:00:48.595 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:00:48.596 DEBUG:boto:Port: 8000 2023-05-31T20:00:48.596 DEBUG:boto:Params: {} 2023-05-31T20:00:48.596 DEBUG:boto:Token: None 2023-05-31T20:00:48.597 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:00:48 GMT /pklxkd-13/ 2023-05-31T20:00:48.597 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:41Yoo9/eFVv+2e+S6Z9kRaR3sIg= 2023-05-31T20:00:48.598 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:00:48 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:41Yoo9/eFVv+2e+S6Z9kRaR3sIg=', 'Content-Length': '0'} 2023-05-31T20:00:48.600 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c7aa8ea0fe4cfad5-006477a770-110e-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:00:48 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:00:48.600 DEBUG:boto:b'pklxkd-13dummy1000falsedummy2023-05-31T19:59:36.915Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:00:48.601 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2023-05-31T20:00:48.601 DEBUG:boto:path=/pklxkd-13/obj 2023-05-31T20:00:48.602 DEBUG:boto:auth_path=/pklxkd-13/obj 2023-05-31T20:00:48.602 DEBUG:boto:path=/pklxkd-13/obj?versionId=.K0qHk3QiH4acREzLteeQLJmFzn4x-1 2023-05-31T20:00:48.603 DEBUG:boto:auth_path=/pklxkd-13/obj?versionId=.K0qHk3QiH4acREzLteeQLJmFzn4x-1 2023-05-31T20:00:48.603 DEBUG:boto:Method: GET 2023-05-31T20:00:48.603 DEBUG:boto:Path: /pklxkd-13/obj?versionId=.K0qHk3QiH4acREzLteeQLJmFzn4x-1 2023-05-31T20:00:48.604 DEBUG:boto:Data: 2023-05-31T20:00:48.604 DEBUG:boto:Headers: {} 2023-05-31T20:00:48.604 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:00:48.605 DEBUG:boto:Port: 8001 2023-05-31T20:00:48.605 DEBUG:boto:Params: {} 2023-05-31T20:00:48.605 DEBUG:boto:Token: None 2023-05-31T20:00:48.606 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:00:48 GMT /pklxkd-13/obj?versionId=.K0qHk3QiH4acREzLteeQLJmFzn4x-1 2023-05-31T20:00:48.606 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:Hoe1CkHUCUKxYAKGgcfwNXW876g= 2023-05-31T20:00:48.607 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:00:48 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:Hoe1CkHUCUKxYAKGgcfwNXW876g=', 'Content-Length': '0'} 2023-05-31T20:00:48.609 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:00:08 GMT'), ('x-amz-version-id', '.K0qHk3QiH4acREzLteeQLJmFzn4x-1'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000001ba952bd2449414d-006477a770-1112-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:00:48 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:00:48.610 DEBUG:boto:path=/pklxkd-13/obj 2023-05-31T20:00:48.610 DEBUG:boto:auth_path=/pklxkd-13/obj 2023-05-31T20:00:48.610 DEBUG:boto:path=/pklxkd-13/obj?versionId=.K0qHk3QiH4acREzLteeQLJmFzn4x-1 2023-05-31T20:00:48.611 DEBUG:boto:auth_path=/pklxkd-13/obj?versionId=.K0qHk3QiH4acREzLteeQLJmFzn4x-1 2023-05-31T20:00:48.611 DEBUG:boto:Method: GET 2023-05-31T20:00:48.612 DEBUG:boto:Path: /pklxkd-13/obj?versionId=.K0qHk3QiH4acREzLteeQLJmFzn4x-1 2023-05-31T20:00:48.612 DEBUG:boto:Data: 2023-05-31T20:00:48.612 DEBUG:boto:Headers: {} 2023-05-31T20:00:48.613 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:00:48.613 DEBUG:boto:Port: 8000 2023-05-31T20:00:48.613 DEBUG:boto:Params: {} 2023-05-31T20:00:48.614 DEBUG:boto:Token: None 2023-05-31T20:00:48.614 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:00:48 GMT /pklxkd-13/obj?versionId=.K0qHk3QiH4acREzLteeQLJmFzn4x-1 2023-05-31T20:00:48.615 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:Hoe1CkHUCUKxYAKGgcfwNXW876g= 2023-05-31T20:00:48.615 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:00:48 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:Hoe1CkHUCUKxYAKGgcfwNXW876g=', 'Content-Length': '0'} 2023-05-31T20:00:48.617 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:00:08 GMT'), ('x-amz-version-id', '.K0qHk3QiH4acREzLteeQLJmFzn4x-1'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000002e3796e6ce4a867a-006477a770-110e-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:00:48 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:00:48.617 DEBUG:boto:path=/pklxkd-13/obj 2023-05-31T20:00:48.618 DEBUG:boto:auth_path=/pklxkd-13/obj 2023-05-31T20:00:48.618 DEBUG:boto:path=/pklxkd-13/obj?versionId=.K0qHk3QiH4acREzLteeQLJmFzn4x-1 2023-05-31T20:00:48.618 DEBUG:boto:auth_path=/pklxkd-13/obj?versionId=.K0qHk3QiH4acREzLteeQLJmFzn4x-1 2023-05-31T20:00:48.619 DEBUG:boto:Method: HEAD 2023-05-31T20:00:48.619 DEBUG:boto:Path: /pklxkd-13/obj?versionId=.K0qHk3QiH4acREzLteeQLJmFzn4x-1 2023-05-31T20:00:48.619 DEBUG:boto:Data: 2023-05-31T20:00:48.620 DEBUG:boto:Headers: {} 2023-05-31T20:00:48.620 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:00:48.620 DEBUG:boto:Port: 8001 2023-05-31T20:00:48.621 DEBUG:boto:Params: {} 2023-05-31T20:00:48.621 DEBUG:boto:Token: None 2023-05-31T20:00:48.621 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:00:48 GMT /pklxkd-13/obj?versionId=.K0qHk3QiH4acREzLteeQLJmFzn4x-1 2023-05-31T20:00:48.622 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:m/WAwnJy6JbY2BCYGF1kF3VPaXs= 2023-05-31T20:00:48.622 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:00:48 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:m/WAwnJy6JbY2BCYGF1kF3VPaXs=', 'Content-Length': '0'} 2023-05-31T20:00:48.624 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:00:08 GMT'), ('x-amz-version-id', '.K0qHk3QiH4acREzLteeQLJmFzn4x-1'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000031c79a8887c73b50-006477a770-1112-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:00:48 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:00:48.625 DEBUG:boto:path=/pklxkd-13/obj 2023-05-31T20:00:48.625 DEBUG:boto:auth_path=/pklxkd-13/obj 2023-05-31T20:00:48.625 DEBUG:boto:path=/pklxkd-13/obj?versionId=.K0qHk3QiH4acREzLteeQLJmFzn4x-1 2023-05-31T20:00:48.626 DEBUG:boto:auth_path=/pklxkd-13/obj?versionId=.K0qHk3QiH4acREzLteeQLJmFzn4x-1 2023-05-31T20:00:48.626 DEBUG:boto:Method: HEAD 2023-05-31T20:00:48.626 DEBUG:boto:Path: /pklxkd-13/obj?versionId=.K0qHk3QiH4acREzLteeQLJmFzn4x-1 2023-05-31T20:00:48.627 DEBUG:boto:Data: 2023-05-31T20:00:48.627 DEBUG:boto:Headers: {} 2023-05-31T20:00:48.627 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:00:48.628 DEBUG:boto:Port: 8000 2023-05-31T20:00:48.628 DEBUG:boto:Params: {} 2023-05-31T20:00:48.628 DEBUG:boto:Token: None 2023-05-31T20:00:48.629 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:00:48 GMT /pklxkd-13/obj?versionId=.K0qHk3QiH4acREzLteeQLJmFzn4x-1 2023-05-31T20:00:48.629 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:m/WAwnJy6JbY2BCYGF1kF3VPaXs= 2023-05-31T20:00:48.630 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:00:48 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:m/WAwnJy6JbY2BCYGF1kF3VPaXs=', 'Content-Length': '0'} 2023-05-31T20:00:48.632 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:00:08 GMT'), ('x-amz-version-id', '.K0qHk3QiH4acREzLteeQLJmFzn4x-1'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000009b56cfd9d3770de5-006477a770-110e-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:00:48 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:00:48.632 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2023-05-31T20:00:48.633 DEBUG:boto:path=/pklxkd-13/obj 2023-05-31T20:00:48.633 DEBUG:boto:auth_path=/pklxkd-13/obj 2023-05-31T20:00:48.634 DEBUG:boto:path=/pklxkd-13/obj?versionId=.K0qHk3QiH4acREzLteeQLJmFzn4x-1 2023-05-31T20:00:48.634 DEBUG:boto:auth_path=/pklxkd-13/obj?versionId=.K0qHk3QiH4acREzLteeQLJmFzn4x-1 2023-05-31T20:00:48.635 DEBUG:boto:Method: GET 2023-05-31T20:00:48.635 DEBUG:boto:Path: /pklxkd-13/obj?versionId=.K0qHk3QiH4acREzLteeQLJmFzn4x-1 2023-05-31T20:00:48.635 DEBUG:boto:Data: 2023-05-31T20:00:48.636 DEBUG:boto:Headers: {} 2023-05-31T20:00:48.636 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:00:48.636 DEBUG:boto:Port: 8001 2023-05-31T20:00:48.637 DEBUG:boto:Params: {} 2023-05-31T20:00:48.637 DEBUG:boto:Token: None 2023-05-31T20:00:48.638 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:00:48 GMT /pklxkd-13/obj?versionId=.K0qHk3QiH4acREzLteeQLJmFzn4x-1 2023-05-31T20:00:48.638 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:Hoe1CkHUCUKxYAKGgcfwNXW876g= 2023-05-31T20:00:48.638 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:00:48 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:Hoe1CkHUCUKxYAKGgcfwNXW876g=', 'Content-Length': '0'} 2023-05-31T20:00:48.640 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:00:08 GMT'), ('x-amz-version-id', '.K0qHk3QiH4acREzLteeQLJmFzn4x-1'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000005af4920bf16fbf5c-006477a770-1112-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:00:48 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:00:48.688 DEBUG:boto:path=/pklxkd-13/obj 2023-05-31T20:00:48.689 DEBUG:boto:auth_path=/pklxkd-13/obj 2023-05-31T20:00:48.689 DEBUG:boto:path=/pklxkd-13/obj?versionId=.K0qHk3QiH4acREzLteeQLJmFzn4x-1 2023-05-31T20:00:48.690 DEBUG:boto:auth_path=/pklxkd-13/obj?versionId=.K0qHk3QiH4acREzLteeQLJmFzn4x-1 2023-05-31T20:00:48.690 DEBUG:boto:Method: GET 2023-05-31T20:00:48.690 DEBUG:boto:Path: /pklxkd-13/obj?versionId=.K0qHk3QiH4acREzLteeQLJmFzn4x-1 2023-05-31T20:00:48.691 DEBUG:boto:Data: 2023-05-31T20:00:48.691 DEBUG:boto:Headers: {} 2023-05-31T20:00:48.691 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:00:48.692 DEBUG:boto:Port: 8000 2023-05-31T20:00:48.692 DEBUG:boto:Params: {} 2023-05-31T20:00:48.692 DEBUG:boto:Token: None 2023-05-31T20:00:48.693 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:00:48 GMT /pklxkd-13/obj?versionId=.K0qHk3QiH4acREzLteeQLJmFzn4x-1 2023-05-31T20:00:48.693 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:Hoe1CkHUCUKxYAKGgcfwNXW876g= 2023-05-31T20:00:48.694 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:00:48 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:Hoe1CkHUCUKxYAKGgcfwNXW876g=', 'Content-Length': '0'} 2023-05-31T20:00:48.696 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:00:08 GMT'), ('x-amz-version-id', '.K0qHk3QiH4acREzLteeQLJmFzn4x-1'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000006eecbeb4b60737b2-006477a770-110e-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:00:48 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:00:48.696 DEBUG:boto:path=/pklxkd-13/ 2023-05-31T20:00:48.697 DEBUG:boto:auth_path=/pklxkd-13/ 2023-05-31T20:00:48.697 DEBUG:boto:path=/pklxkd-13/?prefix=obj 2023-05-31T20:00:48.697 DEBUG:boto:auth_path=/pklxkd-13/?prefix=obj 2023-05-31T20:00:48.698 DEBUG:boto:Method: GET 2023-05-31T20:00:48.698 DEBUG:boto:Path: /pklxkd-13/?prefix=obj 2023-05-31T20:00:48.698 DEBUG:boto:Data: 2023-05-31T20:00:48.699 DEBUG:boto:Headers: {} 2023-05-31T20:00:48.699 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:00:48.699 DEBUG:boto:Port: 8001 2023-05-31T20:00:48.700 DEBUG:boto:Params: {} 2023-05-31T20:00:48.700 DEBUG:boto:Token: None 2023-05-31T20:00:48.701 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:00:48 GMT /pklxkd-13/ 2023-05-31T20:00:48.701 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:41Yoo9/eFVv+2e+S6Z9kRaR3sIg= 2023-05-31T20:00:48.702 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:00:48 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:41Yoo9/eFVv+2e+S6Z9kRaR3sIg=', 'Content-Length': '0'} 2023-05-31T20:00:48.704 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000003922b24a454466f0-006477a770-1112-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:00:48 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:00:48.704 DEBUG:boto:b'pklxkd-13obj1000falseobj2023-05-31T20:00:08.058Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:00:48.705 DEBUG:boto:path=/pklxkd-13/ 2023-05-31T20:00:48.706 DEBUG:boto:auth_path=/pklxkd-13/ 2023-05-31T20:00:48.706 DEBUG:boto:path=/pklxkd-13/?prefix=obj 2023-05-31T20:00:48.706 DEBUG:boto:auth_path=/pklxkd-13/?prefix=obj 2023-05-31T20:00:48.707 DEBUG:boto:Method: GET 2023-05-31T20:00:48.707 DEBUG:boto:Path: /pklxkd-13/?prefix=obj 2023-05-31T20:00:48.707 DEBUG:boto:Data: 2023-05-31T20:00:48.708 DEBUG:boto:Headers: {} 2023-05-31T20:00:48.708 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:00:48.708 DEBUG:boto:Port: 8000 2023-05-31T20:00:48.709 DEBUG:boto:Params: {} 2023-05-31T20:00:48.709 DEBUG:boto:Token: None 2023-05-31T20:00:48.710 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:00:48 GMT /pklxkd-13/ 2023-05-31T20:00:48.710 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:41Yoo9/eFVv+2e+S6Z9kRaR3sIg= 2023-05-31T20:00:48.710 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:00:48 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:41Yoo9/eFVv+2e+S6Z9kRaR3sIg=', 'Content-Length': '0'} 2023-05-31T20:00:48.713 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000042d3cfbf348a155a-006477a770-110e-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:00:48 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:00:48.713 DEBUG:boto:b'pklxkd-13obj1000falseobj2023-05-31T20:00:08.058Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:00:48.714 DEBUG:boto:path=/pklxkd-13/obj 2023-05-31T20:00:48.714 DEBUG:boto:auth_path=/pklxkd-13/obj 2023-05-31T20:00:48.715 DEBUG:boto:Method: HEAD 2023-05-31T20:00:48.715 DEBUG:boto:Path: /pklxkd-13/obj 2023-05-31T20:00:48.715 DEBUG:boto:Data: 2023-05-31T20:00:48.716 DEBUG:boto:Headers: {} 2023-05-31T20:00:48.716 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:00:48.716 DEBUG:boto:Port: 8001 2023-05-31T20:00:48.717 DEBUG:boto:Params: {} 2023-05-31T20:00:48.717 DEBUG:boto:Token: None 2023-05-31T20:00:48.717 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:00:48 GMT /pklxkd-13/obj 2023-05-31T20:00:48.718 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:imJgsiaqaL4feC/VoikfJgZf+6Y= 2023-05-31T20:00:48.718 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:00:48 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:imJgsiaqaL4feC/VoikfJgZf+6Y=', 'Content-Length': '0'} 2023-05-31T20:00:48.721 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:00:08 GMT'), ('x-amz-version-id', '.K0qHk3QiH4acREzLteeQLJmFzn4x-1'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000000f06959a36755d93-006477a770-1112-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:00:48 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:00:48.722 DEBUG:boto:path=/pklxkd-13/obj 2023-05-31T20:00:48.722 DEBUG:boto:auth_path=/pklxkd-13/obj 2023-05-31T20:00:48.723 DEBUG:boto:Method: HEAD 2023-05-31T20:00:48.723 DEBUG:boto:Path: /pklxkd-13/obj 2023-05-31T20:00:48.723 DEBUG:boto:Data: 2023-05-31T20:00:48.724 DEBUG:boto:Headers: {} 2023-05-31T20:00:48.724 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:00:48.724 DEBUG:boto:Port: 8000 2023-05-31T20:00:48.725 DEBUG:boto:Params: {} 2023-05-31T20:00:48.725 DEBUG:boto:Token: None 2023-05-31T20:00:48.725 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:00:48 GMT /pklxkd-13/obj 2023-05-31T20:00:48.726 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:imJgsiaqaL4feC/VoikfJgZf+6Y= 2023-05-31T20:00:48.726 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:00:48 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:imJgsiaqaL4feC/VoikfJgZf+6Y=', 'Content-Length': '0'} 2023-05-31T20:00:48.729 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:00:08 GMT'), ('x-amz-version-id', '.K0qHk3QiH4acREzLteeQLJmFzn4x-1'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000b9885208e3651d19-006477a770-110e-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:00:48 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:00:48.729 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2023-05-31T20:00:48.730 DEBUG:boto:path=/pklxkd-13/obj 2023-05-31T20:00:48.730 DEBUG:boto:auth_path=/pklxkd-13/obj 2023-05-31T20:00:48.731 DEBUG:boto:path=/pklxkd-13/obj?versionId=.K0qHk3QiH4acREzLteeQLJmFzn4x-1 2023-05-31T20:00:48.731 DEBUG:boto:auth_path=/pklxkd-13/obj?versionId=.K0qHk3QiH4acREzLteeQLJmFzn4x-1 2023-05-31T20:00:48.731 DEBUG:boto:Method: GET 2023-05-31T20:00:48.732 DEBUG:boto:Path: /pklxkd-13/obj?versionId=.K0qHk3QiH4acREzLteeQLJmFzn4x-1 2023-05-31T20:00:48.732 DEBUG:boto:Data: 2023-05-31T20:00:48.732 DEBUG:boto:Headers: {} 2023-05-31T20:00:48.733 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:00:48.733 DEBUG:boto:Port: 8001 2023-05-31T20:00:48.733 DEBUG:boto:Params: {} 2023-05-31T20:00:48.734 DEBUG:boto:Token: None 2023-05-31T20:00:48.734 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:00:48 GMT /pklxkd-13/obj?versionId=.K0qHk3QiH4acREzLteeQLJmFzn4x-1 2023-05-31T20:00:48.735 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:Hoe1CkHUCUKxYAKGgcfwNXW876g= 2023-05-31T20:00:48.735 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:00:48 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:Hoe1CkHUCUKxYAKGgcfwNXW876g=', 'Content-Length': '0'} 2023-05-31T20:00:48.737 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:00:08 GMT'), ('x-amz-version-id', '.K0qHk3QiH4acREzLteeQLJmFzn4x-1'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000001f81f4031b6eb57b-006477a770-1112-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:00:48 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:00:48.737 DEBUG:boto:path=/pklxkd-13/obj 2023-05-31T20:00:48.738 DEBUG:boto:auth_path=/pklxkd-13/obj 2023-05-31T20:00:48.738 DEBUG:boto:path=/pklxkd-13/obj?versionId=.K0qHk3QiH4acREzLteeQLJmFzn4x-1 2023-05-31T20:00:48.738 DEBUG:boto:auth_path=/pklxkd-13/obj?versionId=.K0qHk3QiH4acREzLteeQLJmFzn4x-1 2023-05-31T20:00:48.739 DEBUG:boto:Method: GET 2023-05-31T20:00:48.739 DEBUG:boto:Path: /pklxkd-13/obj?versionId=.K0qHk3QiH4acREzLteeQLJmFzn4x-1 2023-05-31T20:00:48.739 DEBUG:boto:Data: 2023-05-31T20:00:48.740 DEBUG:boto:Headers: {} 2023-05-31T20:00:48.740 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:00:48.741 DEBUG:boto:Port: 8000 2023-05-31T20:00:48.741 DEBUG:boto:Params: {} 2023-05-31T20:00:48.741 DEBUG:boto:Token: None 2023-05-31T20:00:48.742 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:00:48 GMT /pklxkd-13/obj?versionId=.K0qHk3QiH4acREzLteeQLJmFzn4x-1 2023-05-31T20:00:48.742 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:Hoe1CkHUCUKxYAKGgcfwNXW876g= 2023-05-31T20:00:48.743 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:00:48 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:Hoe1CkHUCUKxYAKGgcfwNXW876g=', 'Content-Length': '0'} 2023-05-31T20:00:48.744 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:00:08 GMT'), ('x-amz-version-id', '.K0qHk3QiH4acREzLteeQLJmFzn4x-1'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000007520795079790327-006477a770-110e-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:00:48 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:00:48.792 DEBUG:boto:path=/pklxkd-13/ 2023-05-31T20:00:48.793 DEBUG:boto:auth_path=/pklxkd-13/ 2023-05-31T20:00:48.793 DEBUG:boto:path=/pklxkd-13/?prefix=obj 2023-05-31T20:00:48.794 DEBUG:boto:auth_path=/pklxkd-13/?prefix=obj 2023-05-31T20:00:48.794 DEBUG:boto:Method: GET 2023-05-31T20:00:48.794 DEBUG:boto:Path: /pklxkd-13/?prefix=obj 2023-05-31T20:00:48.795 DEBUG:boto:Data: 2023-05-31T20:00:48.795 DEBUG:boto:Headers: {} 2023-05-31T20:00:48.795 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:00:48.796 DEBUG:boto:Port: 8001 2023-05-31T20:00:48.796 DEBUG:boto:Params: {} 2023-05-31T20:00:48.797 DEBUG:boto:Token: None 2023-05-31T20:00:48.797 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:00:48 GMT /pklxkd-13/ 2023-05-31T20:00:48.797 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:41Yoo9/eFVv+2e+S6Z9kRaR3sIg= 2023-05-31T20:00:48.798 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:00:48 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:41Yoo9/eFVv+2e+S6Z9kRaR3sIg=', 'Content-Length': '0'} 2023-05-31T20:00:48.800 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000008affbd598bd3e27d-006477a770-1112-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:00:48 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:00:48.801 DEBUG:boto:b'pklxkd-13obj1000falseobj2023-05-31T20:00:08.058Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:00:48.801 DEBUG:boto:path=/pklxkd-13/ 2023-05-31T20:00:48.802 DEBUG:boto:auth_path=/pklxkd-13/ 2023-05-31T20:00:48.802 DEBUG:boto:path=/pklxkd-13/?prefix=obj 2023-05-31T20:00:48.803 DEBUG:boto:auth_path=/pklxkd-13/?prefix=obj 2023-05-31T20:00:48.803 DEBUG:boto:Method: GET 2023-05-31T20:00:48.803 DEBUG:boto:Path: /pklxkd-13/?prefix=obj 2023-05-31T20:00:48.804 DEBUG:boto:Data: 2023-05-31T20:00:48.804 DEBUG:boto:Headers: {} 2023-05-31T20:00:48.804 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:00:48.805 DEBUG:boto:Port: 8000 2023-05-31T20:00:48.805 DEBUG:boto:Params: {} 2023-05-31T20:00:48.805 DEBUG:boto:Token: None 2023-05-31T20:00:48.806 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:00:48 GMT /pklxkd-13/ 2023-05-31T20:00:48.806 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:41Yoo9/eFVv+2e+S6Z9kRaR3sIg= 2023-05-31T20:00:48.807 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:00:48 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:41Yoo9/eFVv+2e+S6Z9kRaR3sIg=', 'Content-Length': '0'} 2023-05-31T20:00:48.809 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000007cefa32b3dc63578-006477a770-110e-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:00:48 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:00:48.810 DEBUG:boto:b'pklxkd-13obj1000falseobj2023-05-31T20:00:08.058Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:00:48.810 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2023-05-31T20:00:48.811 DEBUG:boto:path=/pklxkd-13/obj 2023-05-31T20:00:48.811 DEBUG:boto:auth_path=/pklxkd-13/obj 2023-05-31T20:00:48.812 DEBUG:boto:path=/pklxkd-13/obj?versionId=gr0pDclEGPIgGhrGevSqfnt7WtJ89yD 2023-05-31T20:00:48.812 DEBUG:boto:auth_path=/pklxkd-13/obj?versionId=gr0pDclEGPIgGhrGevSqfnt7WtJ89yD 2023-05-31T20:00:48.812 DEBUG:boto:Method: GET 2023-05-31T20:00:48.813 DEBUG:boto:Path: /pklxkd-13/obj?versionId=gr0pDclEGPIgGhrGevSqfnt7WtJ89yD 2023-05-31T20:00:48.813 DEBUG:boto:Data: 2023-05-31T20:00:48.813 DEBUG:boto:Headers: {} 2023-05-31T20:00:48.814 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:00:48.814 DEBUG:boto:Port: 8001 2023-05-31T20:00:48.814 DEBUG:boto:Params: {} 2023-05-31T20:00:48.815 DEBUG:boto:Token: None 2023-05-31T20:00:48.815 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:00:48 GMT /pklxkd-13/obj?versionId=gr0pDclEGPIgGhrGevSqfnt7WtJ89yD 2023-05-31T20:00:48.816 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:yimfUihepUPnhoSf3JQCE5IVfqM= 2023-05-31T20:00:48.816 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:00:48 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:yimfUihepUPnhoSf3JQCE5IVfqM=', 'Content-Length': '0'} 2023-05-31T20:00:48.818 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:00:08 GMT'), ('x-amz-version-id', 'gr0pDclEGPIgGhrGevSqfnt7WtJ89yD'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000078263e444cecd0c9-006477a770-1112-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:00:48 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:00:48.860 DEBUG:boto:path=/pklxkd-13/obj 2023-05-31T20:00:48.861 DEBUG:boto:auth_path=/pklxkd-13/obj 2023-05-31T20:00:48.861 DEBUG:boto:path=/pklxkd-13/obj?versionId=gr0pDclEGPIgGhrGevSqfnt7WtJ89yD 2023-05-31T20:00:48.862 DEBUG:boto:auth_path=/pklxkd-13/obj?versionId=gr0pDclEGPIgGhrGevSqfnt7WtJ89yD 2023-05-31T20:00:48.862 DEBUG:boto:Method: GET 2023-05-31T20:00:48.863 DEBUG:boto:Path: /pklxkd-13/obj?versionId=gr0pDclEGPIgGhrGevSqfnt7WtJ89yD 2023-05-31T20:00:48.863 DEBUG:boto:Data: 2023-05-31T20:00:48.863 DEBUG:boto:Headers: {} 2023-05-31T20:00:48.864 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:00:48.864 DEBUG:boto:Port: 8000 2023-05-31T20:00:48.865 DEBUG:boto:Params: {} 2023-05-31T20:00:48.865 DEBUG:boto:Token: None 2023-05-31T20:00:48.865 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:00:48 GMT /pklxkd-13/obj?versionId=gr0pDclEGPIgGhrGevSqfnt7WtJ89yD 2023-05-31T20:00:48.866 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:yimfUihepUPnhoSf3JQCE5IVfqM= 2023-05-31T20:00:48.866 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:00:48 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:yimfUihepUPnhoSf3JQCE5IVfqM=', 'Content-Length': '0'} 2023-05-31T20:00:48.869 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:00:08 GMT'), ('x-amz-version-id', 'gr0pDclEGPIgGhrGevSqfnt7WtJ89yD'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000009f1825ae58459eac-006477a770-110e-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:00:48 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:00:48.869 DEBUG:boto:path=/pklxkd-13/obj 2023-05-31T20:00:48.870 DEBUG:boto:auth_path=/pklxkd-13/obj 2023-05-31T20:00:48.870 DEBUG:boto:path=/pklxkd-13/obj?versionId=gr0pDclEGPIgGhrGevSqfnt7WtJ89yD 2023-05-31T20:00:48.870 DEBUG:boto:auth_path=/pklxkd-13/obj?versionId=gr0pDclEGPIgGhrGevSqfnt7WtJ89yD 2023-05-31T20:00:48.871 DEBUG:boto:Method: HEAD 2023-05-31T20:00:48.871 DEBUG:boto:Path: /pklxkd-13/obj?versionId=gr0pDclEGPIgGhrGevSqfnt7WtJ89yD 2023-05-31T20:00:48.871 DEBUG:boto:Data: 2023-05-31T20:00:48.872 DEBUG:boto:Headers: {} 2023-05-31T20:00:48.872 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:00:48.873 DEBUG:boto:Port: 8001 2023-05-31T20:00:48.873 DEBUG:boto:Params: {} 2023-05-31T20:00:48.873 DEBUG:boto:Token: None 2023-05-31T20:00:48.874 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:00:48 GMT /pklxkd-13/obj?versionId=gr0pDclEGPIgGhrGevSqfnt7WtJ89yD 2023-05-31T20:00:48.874 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:67b/DFcLbcoYnq6/9iclsd/G0SE= 2023-05-31T20:00:48.875 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:00:48 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:67b/DFcLbcoYnq6/9iclsd/G0SE=', 'Content-Length': '0'} 2023-05-31T20:00:48.876 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:00:08 GMT'), ('x-amz-version-id', 'gr0pDclEGPIgGhrGevSqfnt7WtJ89yD'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000004128dd32bf021187-006477a770-1112-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:00:48 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:00:48.877 DEBUG:boto:path=/pklxkd-13/obj 2023-05-31T20:00:48.877 DEBUG:boto:auth_path=/pklxkd-13/obj 2023-05-31T20:00:48.878 DEBUG:boto:path=/pklxkd-13/obj?versionId=gr0pDclEGPIgGhrGevSqfnt7WtJ89yD 2023-05-31T20:00:48.878 DEBUG:boto:auth_path=/pklxkd-13/obj?versionId=gr0pDclEGPIgGhrGevSqfnt7WtJ89yD 2023-05-31T20:00:48.879 DEBUG:boto:Method: HEAD 2023-05-31T20:00:48.879 DEBUG:boto:Path: /pklxkd-13/obj?versionId=gr0pDclEGPIgGhrGevSqfnt7WtJ89yD 2023-05-31T20:00:48.880 DEBUG:boto:Data: 2023-05-31T20:00:48.881 DEBUG:boto:Headers: {} 2023-05-31T20:00:48.881 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:00:48.882 DEBUG:boto:Port: 8000 2023-05-31T20:00:48.883 DEBUG:boto:Params: {} 2023-05-31T20:00:48.884 DEBUG:boto:Token: None 2023-05-31T20:00:48.885 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:00:48 GMT /pklxkd-13/obj?versionId=gr0pDclEGPIgGhrGevSqfnt7WtJ89yD 2023-05-31T20:00:48.885 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:67b/DFcLbcoYnq6/9iclsd/G0SE= 2023-05-31T20:00:48.886 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:00:48 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:67b/DFcLbcoYnq6/9iclsd/G0SE=', 'Content-Length': '0'} 2023-05-31T20:00:48.888 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:00:08 GMT'), ('x-amz-version-id', 'gr0pDclEGPIgGhrGevSqfnt7WtJ89yD'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000007b5d8df788b276d3-006477a770-110e-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:00:48 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:00:48.889 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2023-05-31T20:00:48.890 DEBUG:boto:path=/pklxkd-13/obj 2023-05-31T20:00:48.890 DEBUG:boto:auth_path=/pklxkd-13/obj 2023-05-31T20:00:48.891 DEBUG:boto:path=/pklxkd-13/obj?versionId=gr0pDclEGPIgGhrGevSqfnt7WtJ89yD 2023-05-31T20:00:48.892 DEBUG:boto:auth_path=/pklxkd-13/obj?versionId=gr0pDclEGPIgGhrGevSqfnt7WtJ89yD 2023-05-31T20:00:48.892 DEBUG:boto:Method: GET 2023-05-31T20:00:48.893 DEBUG:boto:Path: /pklxkd-13/obj?versionId=gr0pDclEGPIgGhrGevSqfnt7WtJ89yD 2023-05-31T20:00:48.894 DEBUG:boto:Data: 2023-05-31T20:00:48.894 DEBUG:boto:Headers: {} 2023-05-31T20:00:48.895 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:00:48.896 DEBUG:boto:Port: 8001 2023-05-31T20:00:48.896 DEBUG:boto:Params: {} 2023-05-31T20:00:48.897 DEBUG:boto:Token: None 2023-05-31T20:00:48.898 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:00:48 GMT /pklxkd-13/obj?versionId=gr0pDclEGPIgGhrGevSqfnt7WtJ89yD 2023-05-31T20:00:48.899 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:yimfUihepUPnhoSf3JQCE5IVfqM= 2023-05-31T20:00:48.899 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:00:48 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:yimfUihepUPnhoSf3JQCE5IVfqM=', 'Content-Length': '0'} 2023-05-31T20:00:48.902 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:00:08 GMT'), ('x-amz-version-id', 'gr0pDclEGPIgGhrGevSqfnt7WtJ89yD'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000f472c078e1ad5ada-006477a770-1112-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:00:48 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:00:48.903 DEBUG:boto:path=/pklxkd-13/obj 2023-05-31T20:00:48.903 DEBUG:boto:auth_path=/pklxkd-13/obj 2023-05-31T20:00:48.904 DEBUG:boto:path=/pklxkd-13/obj?versionId=gr0pDclEGPIgGhrGevSqfnt7WtJ89yD 2023-05-31T20:00:48.905 DEBUG:boto:auth_path=/pklxkd-13/obj?versionId=gr0pDclEGPIgGhrGevSqfnt7WtJ89yD 2023-05-31T20:00:48.906 DEBUG:boto:Method: GET 2023-05-31T20:00:48.906 DEBUG:boto:Path: /pklxkd-13/obj?versionId=gr0pDclEGPIgGhrGevSqfnt7WtJ89yD 2023-05-31T20:00:48.907 DEBUG:boto:Data: 2023-05-31T20:00:48.908 DEBUG:boto:Headers: {} 2023-05-31T20:00:48.909 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:00:48.910 DEBUG:boto:Port: 8000 2023-05-31T20:00:48.911 DEBUG:boto:Params: {} 2023-05-31T20:00:48.911 DEBUG:boto:Token: None 2023-05-31T20:00:48.912 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:00:48 GMT /pklxkd-13/obj?versionId=gr0pDclEGPIgGhrGevSqfnt7WtJ89yD 2023-05-31T20:00:48.912 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:yimfUihepUPnhoSf3JQCE5IVfqM= 2023-05-31T20:00:48.913 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:00:48 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:yimfUihepUPnhoSf3JQCE5IVfqM=', 'Content-Length': '0'} 2023-05-31T20:00:48.915 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:00:08 GMT'), ('x-amz-version-id', 'gr0pDclEGPIgGhrGevSqfnt7WtJ89yD'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000b70805f803626a9e-006477a770-110e-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:00:48 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:00:48.916 DEBUG:boto:path=/pklxkd-13/ 2023-05-31T20:00:48.916 DEBUG:boto:auth_path=/pklxkd-13/ 2023-05-31T20:00:48.917 DEBUG:boto:path=/pklxkd-13/?prefix=obj 2023-05-31T20:00:48.917 DEBUG:boto:auth_path=/pklxkd-13/?prefix=obj 2023-05-31T20:00:48.918 DEBUG:boto:Method: GET 2023-05-31T20:00:48.918 DEBUG:boto:Path: /pklxkd-13/?prefix=obj 2023-05-31T20:00:48.918 DEBUG:boto:Data: 2023-05-31T20:00:48.919 DEBUG:boto:Headers: {} 2023-05-31T20:00:48.919 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:00:48.919 DEBUG:boto:Port: 8001 2023-05-31T20:00:48.920 DEBUG:boto:Params: {} 2023-05-31T20:00:48.920 DEBUG:boto:Token: None 2023-05-31T20:00:48.921 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:00:48 GMT /pklxkd-13/ 2023-05-31T20:00:48.921 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:41Yoo9/eFVv+2e+S6Z9kRaR3sIg= 2023-05-31T20:00:48.922 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:00:48 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:41Yoo9/eFVv+2e+S6Z9kRaR3sIg=', 'Content-Length': '0'} 2023-05-31T20:00:48.924 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000bb20b81dd5af5fce-006477a770-1112-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:00:48 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:00:48.925 DEBUG:boto:b'pklxkd-13obj1000falseobj2023-05-31T20:00:08.058Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:00:48.926 DEBUG:boto:path=/pklxkd-13/ 2023-05-31T20:00:48.927 DEBUG:boto:auth_path=/pklxkd-13/ 2023-05-31T20:00:48.927 DEBUG:boto:path=/pklxkd-13/?prefix=obj 2023-05-31T20:00:48.927 DEBUG:boto:auth_path=/pklxkd-13/?prefix=obj 2023-05-31T20:00:48.928 DEBUG:boto:Method: GET 2023-05-31T20:00:48.928 DEBUG:boto:Path: /pklxkd-13/?prefix=obj 2023-05-31T20:00:48.929 DEBUG:boto:Data: 2023-05-31T20:00:48.929 DEBUG:boto:Headers: {} 2023-05-31T20:00:48.930 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:00:48.930 DEBUG:boto:Port: 8000 2023-05-31T20:00:48.930 DEBUG:boto:Params: {} 2023-05-31T20:00:48.931 DEBUG:boto:Token: None 2023-05-31T20:00:48.931 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:00:48 GMT /pklxkd-13/ 2023-05-31T20:00:48.932 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:41Yoo9/eFVv+2e+S6Z9kRaR3sIg= 2023-05-31T20:00:48.932 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:00:48 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:41Yoo9/eFVv+2e+S6Z9kRaR3sIg=', 'Content-Length': '0'} 2023-05-31T20:00:48.935 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f03cf7562434496a-006477a770-110e-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:00:48 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:00:48.936 DEBUG:boto:b'pklxkd-13obj1000falseobj2023-05-31T20:00:08.058Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:00:48.936 DEBUG:boto:path=/pklxkd-13/obj 2023-05-31T20:00:48.937 DEBUG:boto:auth_path=/pklxkd-13/obj 2023-05-31T20:00:48.937 DEBUG:boto:Method: HEAD 2023-05-31T20:00:48.938 DEBUG:boto:Path: /pklxkd-13/obj 2023-05-31T20:00:48.938 DEBUG:boto:Data: 2023-05-31T20:00:48.938 DEBUG:boto:Headers: {} 2023-05-31T20:00:48.939 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:00:48.939 DEBUG:boto:Port: 8001 2023-05-31T20:00:48.940 DEBUG:boto:Params: {} 2023-05-31T20:00:48.940 DEBUG:boto:Token: None 2023-05-31T20:00:48.941 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:00:48 GMT /pklxkd-13/obj 2023-05-31T20:00:48.941 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:imJgsiaqaL4feC/VoikfJgZf+6Y= 2023-05-31T20:00:48.942 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:00:48 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:imJgsiaqaL4feC/VoikfJgZf+6Y=', 'Content-Length': '0'} 2023-05-31T20:00:48.945 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:00:08 GMT'), ('x-amz-version-id', '.K0qHk3QiH4acREzLteeQLJmFzn4x-1'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000b056cd51e42b11c5-006477a770-1112-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:00:48 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:00:48.946 DEBUG:boto:path=/pklxkd-13/obj 2023-05-31T20:00:48.946 DEBUG:boto:auth_path=/pklxkd-13/obj 2023-05-31T20:00:48.946 DEBUG:boto:Method: HEAD 2023-05-31T20:00:48.947 DEBUG:boto:Path: /pklxkd-13/obj 2023-05-31T20:00:48.947 DEBUG:boto:Data: 2023-05-31T20:00:48.947 DEBUG:boto:Headers: {} 2023-05-31T20:00:48.948 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:00:48.948 DEBUG:boto:Port: 8000 2023-05-31T20:00:48.949 DEBUG:boto:Params: {} 2023-05-31T20:00:48.949 DEBUG:boto:Token: None 2023-05-31T20:00:48.949 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:00:48 GMT /pklxkd-13/obj 2023-05-31T20:00:48.950 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:imJgsiaqaL4feC/VoikfJgZf+6Y= 2023-05-31T20:00:48.950 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:00:48 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:imJgsiaqaL4feC/VoikfJgZf+6Y=', 'Content-Length': '0'} 2023-05-31T20:00:48.953 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:00:08 GMT'), ('x-amz-version-id', '.K0qHk3QiH4acREzLteeQLJmFzn4x-1'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000ec0366837674b0bc-006477a770-110e-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:00:48 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:00:48.954 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2023-05-31T20:00:48.954 DEBUG:boto:path=/pklxkd-13/obj 2023-05-31T20:00:48.955 DEBUG:boto:auth_path=/pklxkd-13/obj 2023-05-31T20:00:48.955 DEBUG:boto:path=/pklxkd-13/obj?versionId=.K0qHk3QiH4acREzLteeQLJmFzn4x-1 2023-05-31T20:00:48.956 DEBUG:boto:auth_path=/pklxkd-13/obj?versionId=.K0qHk3QiH4acREzLteeQLJmFzn4x-1 2023-05-31T20:00:48.956 DEBUG:boto:Method: GET 2023-05-31T20:00:48.957 DEBUG:boto:Path: /pklxkd-13/obj?versionId=.K0qHk3QiH4acREzLteeQLJmFzn4x-1 2023-05-31T20:00:48.957 DEBUG:boto:Data: 2023-05-31T20:00:48.958 DEBUG:boto:Headers: {} 2023-05-31T20:00:48.959 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:00:48.960 DEBUG:boto:Port: 8001 2023-05-31T20:00:48.960 DEBUG:boto:Params: {} 2023-05-31T20:00:48.961 DEBUG:boto:Token: None 2023-05-31T20:00:48.962 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:00:48 GMT /pklxkd-13/obj?versionId=.K0qHk3QiH4acREzLteeQLJmFzn4x-1 2023-05-31T20:00:48.963 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:Hoe1CkHUCUKxYAKGgcfwNXW876g= 2023-05-31T20:00:48.964 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:00:48 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:Hoe1CkHUCUKxYAKGgcfwNXW876g=', 'Content-Length': '0'} 2023-05-31T20:00:48.966 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:00:08 GMT'), ('x-amz-version-id', '.K0qHk3QiH4acREzLteeQLJmFzn4x-1'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000fc6a22154622afc3-006477a770-1112-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:00:48 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:00:49.012 DEBUG:boto:path=/pklxkd-13/obj 2023-05-31T20:00:49.013 DEBUG:boto:auth_path=/pklxkd-13/obj 2023-05-31T20:00:49.014 DEBUG:boto:path=/pklxkd-13/obj?versionId=.K0qHk3QiH4acREzLteeQLJmFzn4x-1 2023-05-31T20:00:49.015 DEBUG:boto:auth_path=/pklxkd-13/obj?versionId=.K0qHk3QiH4acREzLteeQLJmFzn4x-1 2023-05-31T20:00:49.015 DEBUG:boto:Method: GET 2023-05-31T20:00:49.016 DEBUG:boto:Path: /pklxkd-13/obj?versionId=.K0qHk3QiH4acREzLteeQLJmFzn4x-1 2023-05-31T20:00:49.016 DEBUG:boto:Data: 2023-05-31T20:00:49.017 DEBUG:boto:Headers: {} 2023-05-31T20:00:49.017 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:00:49.017 DEBUG:boto:Port: 8000 2023-05-31T20:00:49.018 DEBUG:boto:Params: {} 2023-05-31T20:00:49.018 DEBUG:boto:Token: None 2023-05-31T20:00:49.018 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:00:49 GMT /pklxkd-13/obj?versionId=.K0qHk3QiH4acREzLteeQLJmFzn4x-1 2023-05-31T20:00:49.019 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:erctR9bIPl4pq01jjrIApCDKd2Y= 2023-05-31T20:00:49.019 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:00:49 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:erctR9bIPl4pq01jjrIApCDKd2Y=', 'Content-Length': '0'} 2023-05-31T20:00:49.021 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:00:08 GMT'), ('x-amz-version-id', '.K0qHk3QiH4acREzLteeQLJmFzn4x-1'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000dde9c2b5c438deb9-006477a771-110e-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:00:49 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:00:49.022 DEBUG:boto:path=/pklxkd-13/ 2023-05-31T20:00:49.023 DEBUG:boto:auth_path=/pklxkd-13/ 2023-05-31T20:00:49.023 DEBUG:boto:path=/pklxkd-13/?prefix=obj 2023-05-31T20:00:49.023 DEBUG:boto:auth_path=/pklxkd-13/?prefix=obj 2023-05-31T20:00:49.024 DEBUG:boto:Method: GET 2023-05-31T20:00:49.024 DEBUG:boto:Path: /pklxkd-13/?prefix=obj 2023-05-31T20:00:49.024 DEBUG:boto:Data: 2023-05-31T20:00:49.025 DEBUG:boto:Headers: {} 2023-05-31T20:00:49.025 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:00:49.026 DEBUG:boto:Port: 8001 2023-05-31T20:00:49.026 DEBUG:boto:Params: {} 2023-05-31T20:00:49.026 DEBUG:boto:Token: None 2023-05-31T20:00:49.027 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:00:49 GMT /pklxkd-13/ 2023-05-31T20:00:49.027 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:2Rhl0KA1WOxWz2tqNLYakr1DRVE= 2023-05-31T20:00:49.028 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:00:49 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:2Rhl0KA1WOxWz2tqNLYakr1DRVE=', 'Content-Length': '0'} 2023-05-31T20:00:49.030 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000000ded1d98dcb02929-006477a771-1112-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:00:49 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:00:49.031 DEBUG:boto:b'pklxkd-13obj1000falseobj2023-05-31T20:00:08.058Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:00:49.032 DEBUG:boto:path=/pklxkd-13/ 2023-05-31T20:00:49.032 DEBUG:boto:auth_path=/pklxkd-13/ 2023-05-31T20:00:49.032 DEBUG:boto:path=/pklxkd-13/?prefix=obj 2023-05-31T20:00:49.033 DEBUG:boto:auth_path=/pklxkd-13/?prefix=obj 2023-05-31T20:00:49.033 DEBUG:boto:Method: GET 2023-05-31T20:00:49.033 DEBUG:boto:Path: /pklxkd-13/?prefix=obj 2023-05-31T20:00:49.034 DEBUG:boto:Data: 2023-05-31T20:00:49.034 DEBUG:boto:Headers: {} 2023-05-31T20:00:49.035 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:00:49.035 DEBUG:boto:Port: 8000 2023-05-31T20:00:49.035 DEBUG:boto:Params: {} 2023-05-31T20:00:49.036 DEBUG:boto:Token: None 2023-05-31T20:00:49.036 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:00:49 GMT /pklxkd-13/ 2023-05-31T20:00:49.037 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:2Rhl0KA1WOxWz2tqNLYakr1DRVE= 2023-05-31T20:00:49.037 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:00:49 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:2Rhl0KA1WOxWz2tqNLYakr1DRVE=', 'Content-Length': '0'} 2023-05-31T20:00:49.039 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000043a81f8c16bf0e76-006477a771-110e-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:00:49 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:00:49.040 DEBUG:boto:b'pklxkd-13obj1000falseobj2023-05-31T20:00:08.058Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:00:49.041 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2023-05-31T20:00:49.041 DEBUG:boto:path=/pklxkd-13/obj 2023-05-31T20:00:49.041 DEBUG:boto:auth_path=/pklxkd-13/obj 2023-05-31T20:00:49.042 DEBUG:boto:path=/pklxkd-13/obj?versionId=.q2JxPcqi3BYUXNGXuFQE0hVEIAq1Yk 2023-05-31T20:00:49.042 DEBUG:boto:auth_path=/pklxkd-13/obj?versionId=.q2JxPcqi3BYUXNGXuFQE0hVEIAq1Yk 2023-05-31T20:00:49.042 DEBUG:boto:Method: GET 2023-05-31T20:00:49.043 DEBUG:boto:Path: /pklxkd-13/obj?versionId=.q2JxPcqi3BYUXNGXuFQE0hVEIAq1Yk 2023-05-31T20:00:49.043 DEBUG:boto:Data: 2023-05-31T20:00:49.044 DEBUG:boto:Headers: {} 2023-05-31T20:00:49.044 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:00:49.044 DEBUG:boto:Port: 8001 2023-05-31T20:00:49.045 DEBUG:boto:Params: {} 2023-05-31T20:00:49.045 DEBUG:boto:Token: None 2023-05-31T20:00:49.046 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:00:49 GMT /pklxkd-13/obj?versionId=.q2JxPcqi3BYUXNGXuFQE0hVEIAq1Yk 2023-05-31T20:00:49.046 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:9Og6XkBexWrcEP2f+sAbojYLrQk= 2023-05-31T20:00:49.046 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:00:49 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:9Og6XkBexWrcEP2f+sAbojYLrQk=', 'Content-Length': '0'} 2023-05-31T20:00:49.048 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:00:08 GMT'), ('x-amz-version-id', '.q2JxPcqi3BYUXNGXuFQE0hVEIAq1Yk'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000eb85d1a50ef9cc6a-006477a771-1112-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:00:49 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:00:49.049 DEBUG:boto:path=/pklxkd-13/obj 2023-05-31T20:00:49.049 DEBUG:boto:auth_path=/pklxkd-13/obj 2023-05-31T20:00:49.050 DEBUG:boto:path=/pklxkd-13/obj?versionId=.q2JxPcqi3BYUXNGXuFQE0hVEIAq1Yk 2023-05-31T20:00:49.050 DEBUG:boto:auth_path=/pklxkd-13/obj?versionId=.q2JxPcqi3BYUXNGXuFQE0hVEIAq1Yk 2023-05-31T20:00:49.050 DEBUG:boto:Method: GET 2023-05-31T20:00:49.051 DEBUG:boto:Path: /pklxkd-13/obj?versionId=.q2JxPcqi3BYUXNGXuFQE0hVEIAq1Yk 2023-05-31T20:00:49.051 DEBUG:boto:Data: 2023-05-31T20:00:49.052 DEBUG:boto:Headers: {} 2023-05-31T20:00:49.052 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:00:49.052 DEBUG:boto:Port: 8000 2023-05-31T20:00:49.053 DEBUG:boto:Params: {} 2023-05-31T20:00:49.053 DEBUG:boto:Token: None 2023-05-31T20:00:49.054 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:00:49 GMT /pklxkd-13/obj?versionId=.q2JxPcqi3BYUXNGXuFQE0hVEIAq1Yk 2023-05-31T20:00:49.054 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:9Og6XkBexWrcEP2f+sAbojYLrQk= 2023-05-31T20:00:49.054 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:00:49 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:9Og6XkBexWrcEP2f+sAbojYLrQk=', 'Content-Length': '0'} 2023-05-31T20:00:49.056 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:00:08 GMT'), ('x-amz-version-id', '.q2JxPcqi3BYUXNGXuFQE0hVEIAq1Yk'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000031be3e95bb058fae-006477a771-110e-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:00:49 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:00:49.057 DEBUG:boto:path=/pklxkd-13/obj 2023-05-31T20:00:49.057 DEBUG:boto:auth_path=/pklxkd-13/obj 2023-05-31T20:00:49.058 DEBUG:boto:path=/pklxkd-13/obj?versionId=.q2JxPcqi3BYUXNGXuFQE0hVEIAq1Yk 2023-05-31T20:00:49.058 DEBUG:boto:auth_path=/pklxkd-13/obj?versionId=.q2JxPcqi3BYUXNGXuFQE0hVEIAq1Yk 2023-05-31T20:00:49.058 DEBUG:boto:Method: HEAD 2023-05-31T20:00:49.059 DEBUG:boto:Path: /pklxkd-13/obj?versionId=.q2JxPcqi3BYUXNGXuFQE0hVEIAq1Yk 2023-05-31T20:00:49.059 DEBUG:boto:Data: 2023-05-31T20:00:49.059 DEBUG:boto:Headers: {} 2023-05-31T20:00:49.060 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:00:49.060 DEBUG:boto:Port: 8001 2023-05-31T20:00:49.060 DEBUG:boto:Params: {} 2023-05-31T20:00:49.061 DEBUG:boto:Token: None 2023-05-31T20:00:49.061 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:00:49 GMT /pklxkd-13/obj?versionId=.q2JxPcqi3BYUXNGXuFQE0hVEIAq1Yk 2023-05-31T20:00:49.062 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:Ru9hSN9d3ZQtxCnksb4Si3n0BcE= 2023-05-31T20:00:49.062 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:00:49 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:Ru9hSN9d3ZQtxCnksb4Si3n0BcE=', 'Content-Length': '0'} 2023-05-31T20:00:49.064 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:00:08 GMT'), ('x-amz-version-id', '.q2JxPcqi3BYUXNGXuFQE0hVEIAq1Yk'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000da74851645cc6c92-006477a771-1112-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:00:49 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:00:49.064 DEBUG:boto:path=/pklxkd-13/obj 2023-05-31T20:00:49.065 DEBUG:boto:auth_path=/pklxkd-13/obj 2023-05-31T20:00:49.065 DEBUG:boto:path=/pklxkd-13/obj?versionId=.q2JxPcqi3BYUXNGXuFQE0hVEIAq1Yk 2023-05-31T20:00:49.065 DEBUG:boto:auth_path=/pklxkd-13/obj?versionId=.q2JxPcqi3BYUXNGXuFQE0hVEIAq1Yk 2023-05-31T20:00:49.066 DEBUG:boto:Method: HEAD 2023-05-31T20:00:49.066 DEBUG:boto:Path: /pklxkd-13/obj?versionId=.q2JxPcqi3BYUXNGXuFQE0hVEIAq1Yk 2023-05-31T20:00:49.066 DEBUG:boto:Data: 2023-05-31T20:00:49.066 DEBUG:boto:Headers: {} 2023-05-31T20:00:49.067 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:00:49.067 DEBUG:boto:Port: 8000 2023-05-31T20:00:49.068 DEBUG:boto:Params: {} 2023-05-31T20:00:49.068 DEBUG:boto:Token: None 2023-05-31T20:00:49.068 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:00:49 GMT /pklxkd-13/obj?versionId=.q2JxPcqi3BYUXNGXuFQE0hVEIAq1Yk 2023-05-31T20:00:49.069 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:Ru9hSN9d3ZQtxCnksb4Si3n0BcE= 2023-05-31T20:00:49.069 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:00:49 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:Ru9hSN9d3ZQtxCnksb4Si3n0BcE=', 'Content-Length': '0'} 2023-05-31T20:00:49.071 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:00:08 GMT'), ('x-amz-version-id', '.q2JxPcqi3BYUXNGXuFQE0hVEIAq1Yk'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000501c2bc5128bceb6-006477a771-110e-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:00:49 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:00:49.072 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2023-05-31T20:00:49.073 DEBUG:boto:path=/pklxkd-13/obj 2023-05-31T20:00:49.073 DEBUG:boto:auth_path=/pklxkd-13/obj 2023-05-31T20:00:49.073 DEBUG:boto:path=/pklxkd-13/obj?versionId=.q2JxPcqi3BYUXNGXuFQE0hVEIAq1Yk 2023-05-31T20:00:49.074 DEBUG:boto:auth_path=/pklxkd-13/obj?versionId=.q2JxPcqi3BYUXNGXuFQE0hVEIAq1Yk 2023-05-31T20:00:49.074 DEBUG:boto:Method: GET 2023-05-31T20:00:49.075 DEBUG:boto:Path: /pklxkd-13/obj?versionId=.q2JxPcqi3BYUXNGXuFQE0hVEIAq1Yk 2023-05-31T20:00:49.075 DEBUG:boto:Data: 2023-05-31T20:00:49.075 DEBUG:boto:Headers: {} 2023-05-31T20:00:49.076 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:00:49.076 DEBUG:boto:Port: 8001 2023-05-31T20:00:49.076 DEBUG:boto:Params: {} 2023-05-31T20:00:49.077 DEBUG:boto:Token: None 2023-05-31T20:00:49.077 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:00:49 GMT /pklxkd-13/obj?versionId=.q2JxPcqi3BYUXNGXuFQE0hVEIAq1Yk 2023-05-31T20:00:49.078 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:9Og6XkBexWrcEP2f+sAbojYLrQk= 2023-05-31T20:00:49.078 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:00:49 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:9Og6XkBexWrcEP2f+sAbojYLrQk=', 'Content-Length': '0'} 2023-05-31T20:00:49.080 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:00:08 GMT'), ('x-amz-version-id', '.q2JxPcqi3BYUXNGXuFQE0hVEIAq1Yk'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000087961bcc81a0053b-006477a771-1112-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:00:49 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:00:49.081 DEBUG:boto:path=/pklxkd-13/obj 2023-05-31T20:00:49.081 DEBUG:boto:auth_path=/pklxkd-13/obj 2023-05-31T20:00:49.081 DEBUG:boto:path=/pklxkd-13/obj?versionId=.q2JxPcqi3BYUXNGXuFQE0hVEIAq1Yk 2023-05-31T20:00:49.082 DEBUG:boto:auth_path=/pklxkd-13/obj?versionId=.q2JxPcqi3BYUXNGXuFQE0hVEIAq1Yk 2023-05-31T20:00:49.082 DEBUG:boto:Method: GET 2023-05-31T20:00:49.083 DEBUG:boto:Path: /pklxkd-13/obj?versionId=.q2JxPcqi3BYUXNGXuFQE0hVEIAq1Yk 2023-05-31T20:00:49.083 DEBUG:boto:Data: 2023-05-31T20:00:49.084 DEBUG:boto:Headers: {} 2023-05-31T20:00:49.084 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:00:49.084 DEBUG:boto:Port: 8000 2023-05-31T20:00:49.085 DEBUG:boto:Params: {} 2023-05-31T20:00:49.085 DEBUG:boto:Token: None 2023-05-31T20:00:49.086 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:00:49 GMT /pklxkd-13/obj?versionId=.q2JxPcqi3BYUXNGXuFQE0hVEIAq1Yk 2023-05-31T20:00:49.086 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:9Og6XkBexWrcEP2f+sAbojYLrQk= 2023-05-31T20:00:49.086 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:00:49 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:9Og6XkBexWrcEP2f+sAbojYLrQk=', 'Content-Length': '0'} 2023-05-31T20:00:49.088 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:00:08 GMT'), ('x-amz-version-id', '.q2JxPcqi3BYUXNGXuFQE0hVEIAq1Yk'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000c379aaffc3af8559-006477a771-110e-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:00:49 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:00:49.132 DEBUG:boto:path=/pklxkd-13/ 2023-05-31T20:00:49.133 DEBUG:boto:auth_path=/pklxkd-13/ 2023-05-31T20:00:49.133 DEBUG:boto:path=/pklxkd-13/?prefix=obj 2023-05-31T20:00:49.134 DEBUG:boto:auth_path=/pklxkd-13/?prefix=obj 2023-05-31T20:00:49.134 DEBUG:boto:Method: GET 2023-05-31T20:00:49.134 DEBUG:boto:Path: /pklxkd-13/?prefix=obj 2023-05-31T20:00:49.135 DEBUG:boto:Data: 2023-05-31T20:00:49.135 DEBUG:boto:Headers: {} 2023-05-31T20:00:49.135 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:00:49.136 DEBUG:boto:Port: 8001 2023-05-31T20:00:49.136 DEBUG:boto:Params: {} 2023-05-31T20:00:49.137 DEBUG:boto:Token: None 2023-05-31T20:00:49.137 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:00:49 GMT /pklxkd-13/ 2023-05-31T20:00:49.137 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:2Rhl0KA1WOxWz2tqNLYakr1DRVE= 2023-05-31T20:00:49.138 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:00:49 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:2Rhl0KA1WOxWz2tqNLYakr1DRVE=', 'Content-Length': '0'} 2023-05-31T20:00:49.140 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000969b243a6c64ce6f-006477a771-1112-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:00:49 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:00:49.141 DEBUG:boto:b'pklxkd-13obj1000falseobj2023-05-31T20:00:08.058Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:00:49.141 DEBUG:boto:path=/pklxkd-13/ 2023-05-31T20:00:49.142 DEBUG:boto:auth_path=/pklxkd-13/ 2023-05-31T20:00:49.142 DEBUG:boto:path=/pklxkd-13/?prefix=obj 2023-05-31T20:00:49.143 DEBUG:boto:auth_path=/pklxkd-13/?prefix=obj 2023-05-31T20:00:49.143 DEBUG:boto:Method: GET 2023-05-31T20:00:49.143 DEBUG:boto:Path: /pklxkd-13/?prefix=obj 2023-05-31T20:00:49.144 DEBUG:boto:Data: 2023-05-31T20:00:49.144 DEBUG:boto:Headers: {} 2023-05-31T20:00:49.144 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:00:49.145 DEBUG:boto:Port: 8000 2023-05-31T20:00:49.145 DEBUG:boto:Params: {} 2023-05-31T20:00:49.145 DEBUG:boto:Token: None 2023-05-31T20:00:49.146 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:00:49 GMT /pklxkd-13/ 2023-05-31T20:00:49.146 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:2Rhl0KA1WOxWz2tqNLYakr1DRVE= 2023-05-31T20:00:49.147 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:00:49 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:2Rhl0KA1WOxWz2tqNLYakr1DRVE=', 'Content-Length': '0'} 2023-05-31T20:00:49.149 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000841d490facdc6009-006477a771-110e-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:00:49 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:00:49.150 DEBUG:boto:b'pklxkd-13obj1000falseobj2023-05-31T20:00:08.058Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:00:49.151 DEBUG:boto:path=/pklxkd-13/obj 2023-05-31T20:00:49.151 DEBUG:boto:auth_path=/pklxkd-13/obj 2023-05-31T20:00:49.152 DEBUG:boto:Method: HEAD 2023-05-31T20:00:49.152 DEBUG:boto:Path: /pklxkd-13/obj 2023-05-31T20:00:49.152 DEBUG:boto:Data: 2023-05-31T20:00:49.153 DEBUG:boto:Headers: {} 2023-05-31T20:00:49.153 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:00:49.153 DEBUG:boto:Port: 8001 2023-05-31T20:00:49.154 DEBUG:boto:Params: {} 2023-05-31T20:00:49.154 DEBUG:boto:Token: None 2023-05-31T20:00:49.154 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:00:49 GMT /pklxkd-13/obj 2023-05-31T20:00:49.155 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:Mv9iPJUU0LzM4+d+e0C61wmvE7Y= 2023-05-31T20:00:49.155 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:00:49 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:Mv9iPJUU0LzM4+d+e0C61wmvE7Y=', 'Content-Length': '0'} 2023-05-31T20:00:49.158 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:00:08 GMT'), ('x-amz-version-id', '.K0qHk3QiH4acREzLteeQLJmFzn4x-1'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000006865302874e65888-006477a771-1112-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:00:49 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:00:49.159 DEBUG:boto:path=/pklxkd-13/obj 2023-05-31T20:00:49.159 DEBUG:boto:auth_path=/pklxkd-13/obj 2023-05-31T20:00:49.160 DEBUG:boto:Method: HEAD 2023-05-31T20:00:49.160 DEBUG:boto:Path: /pklxkd-13/obj 2023-05-31T20:00:49.160 DEBUG:boto:Data: 2023-05-31T20:00:49.161 DEBUG:boto:Headers: {} 2023-05-31T20:00:49.161 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:00:49.161 DEBUG:boto:Port: 8000 2023-05-31T20:00:49.162 DEBUG:boto:Params: {} 2023-05-31T20:00:49.162 DEBUG:boto:Token: None 2023-05-31T20:00:49.162 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:00:49 GMT /pklxkd-13/obj 2023-05-31T20:00:49.163 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:Mv9iPJUU0LzM4+d+e0C61wmvE7Y= 2023-05-31T20:00:49.163 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:00:49 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:Mv9iPJUU0LzM4+d+e0C61wmvE7Y=', 'Content-Length': '0'} 2023-05-31T20:00:49.166 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:00:08 GMT'), ('x-amz-version-id', '.K0qHk3QiH4acREzLteeQLJmFzn4x-1'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000005caec9ce3bcc66db-006477a771-110e-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:00:49 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:00:49.167 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2023-05-31T20:00:49.167 DEBUG:boto:path=/pklxkd-13/obj 2023-05-31T20:00:49.168 DEBUG:boto:auth_path=/pklxkd-13/obj 2023-05-31T20:00:49.168 DEBUG:boto:path=/pklxkd-13/obj?versionId=.K0qHk3QiH4acREzLteeQLJmFzn4x-1 2023-05-31T20:00:49.168 DEBUG:boto:auth_path=/pklxkd-13/obj?versionId=.K0qHk3QiH4acREzLteeQLJmFzn4x-1 2023-05-31T20:00:49.169 DEBUG:boto:Method: GET 2023-05-31T20:00:49.169 DEBUG:boto:Path: /pklxkd-13/obj?versionId=.K0qHk3QiH4acREzLteeQLJmFzn4x-1 2023-05-31T20:00:49.170 DEBUG:boto:Data: 2023-05-31T20:00:49.170 DEBUG:boto:Headers: {} 2023-05-31T20:00:49.170 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:00:49.171 DEBUG:boto:Port: 8001 2023-05-31T20:00:49.171 DEBUG:boto:Params: {} 2023-05-31T20:00:49.172 DEBUG:boto:Token: None 2023-05-31T20:00:49.172 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:00:49 GMT /pklxkd-13/obj?versionId=.K0qHk3QiH4acREzLteeQLJmFzn4x-1 2023-05-31T20:00:49.173 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:erctR9bIPl4pq01jjrIApCDKd2Y= 2023-05-31T20:00:49.173 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:00:49 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:erctR9bIPl4pq01jjrIApCDKd2Y=', 'Content-Length': '0'} 2023-05-31T20:00:49.175 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:00:08 GMT'), ('x-amz-version-id', '.K0qHk3QiH4acREzLteeQLJmFzn4x-1'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000072857326e481fbfc-006477a771-1112-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:00:49 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:00:49.216 DEBUG:boto:path=/pklxkd-13/obj 2023-05-31T20:00:49.217 DEBUG:boto:auth_path=/pklxkd-13/obj 2023-05-31T20:00:49.217 DEBUG:boto:path=/pklxkd-13/obj?versionId=.K0qHk3QiH4acREzLteeQLJmFzn4x-1 2023-05-31T20:00:49.218 DEBUG:boto:auth_path=/pklxkd-13/obj?versionId=.K0qHk3QiH4acREzLteeQLJmFzn4x-1 2023-05-31T20:00:49.218 DEBUG:boto:Method: GET 2023-05-31T20:00:49.218 DEBUG:boto:Path: /pklxkd-13/obj?versionId=.K0qHk3QiH4acREzLteeQLJmFzn4x-1 2023-05-31T20:00:49.219 DEBUG:boto:Data: 2023-05-31T20:00:49.219 DEBUG:boto:Headers: {} 2023-05-31T20:00:49.220 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:00:49.220 DEBUG:boto:Port: 8000 2023-05-31T20:00:49.220 DEBUG:boto:Params: {} 2023-05-31T20:00:49.221 DEBUG:boto:Token: None 2023-05-31T20:00:49.221 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:00:49 GMT /pklxkd-13/obj?versionId=.K0qHk3QiH4acREzLteeQLJmFzn4x-1 2023-05-31T20:00:49.222 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:erctR9bIPl4pq01jjrIApCDKd2Y= 2023-05-31T20:00:49.222 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:00:49 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:erctR9bIPl4pq01jjrIApCDKd2Y=', 'Content-Length': '0'} 2023-05-31T20:00:49.225 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:00:08 GMT'), ('x-amz-version-id', '.K0qHk3QiH4acREzLteeQLJmFzn4x-1'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000d2c8c22a296706d6-006477a771-110e-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:00:49 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:00:49.225 DEBUG:boto:path=/pklxkd-13/ 2023-05-31T20:00:49.226 DEBUG:boto:auth_path=/pklxkd-13/ 2023-05-31T20:00:49.226 DEBUG:boto:path=/pklxkd-13/?prefix=obj 2023-05-31T20:00:49.227 DEBUG:boto:auth_path=/pklxkd-13/?prefix=obj 2023-05-31T20:00:49.227 DEBUG:boto:Method: GET 2023-05-31T20:00:49.227 DEBUG:boto:Path: /pklxkd-13/?prefix=obj 2023-05-31T20:00:49.228 DEBUG:boto:Data: 2023-05-31T20:00:49.228 DEBUG:boto:Headers: {} 2023-05-31T20:00:49.229 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:00:49.229 DEBUG:boto:Port: 8001 2023-05-31T20:00:49.229 DEBUG:boto:Params: {} 2023-05-31T20:00:49.230 DEBUG:boto:Token: None 2023-05-31T20:00:49.230 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:00:49 GMT /pklxkd-13/ 2023-05-31T20:00:49.231 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:2Rhl0KA1WOxWz2tqNLYakr1DRVE= 2023-05-31T20:00:49.231 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:00:49 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:2Rhl0KA1WOxWz2tqNLYakr1DRVE=', 'Content-Length': '0'} 2023-05-31T20:00:49.234 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000007729062cf006927a-006477a771-1112-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:00:49 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:00:49.234 DEBUG:boto:b'pklxkd-13obj1000falseobj2023-05-31T20:00:08.058Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:00:49.235 DEBUG:boto:path=/pklxkd-13/ 2023-05-31T20:00:49.236 DEBUG:boto:auth_path=/pklxkd-13/ 2023-05-31T20:00:49.236 DEBUG:boto:path=/pklxkd-13/?prefix=obj 2023-05-31T20:00:49.236 DEBUG:boto:auth_path=/pklxkd-13/?prefix=obj 2023-05-31T20:00:49.237 DEBUG:boto:Method: GET 2023-05-31T20:00:49.237 DEBUG:boto:Path: /pklxkd-13/?prefix=obj 2023-05-31T20:00:49.238 DEBUG:boto:Data: 2023-05-31T20:00:49.238 DEBUG:boto:Headers: {} 2023-05-31T20:00:49.238 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:00:49.239 DEBUG:boto:Port: 8000 2023-05-31T20:00:49.239 DEBUG:boto:Params: {} 2023-05-31T20:00:49.240 DEBUG:boto:Token: None 2023-05-31T20:00:49.240 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:00:49 GMT /pklxkd-13/ 2023-05-31T20:00:49.241 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:2Rhl0KA1WOxWz2tqNLYakr1DRVE= 2023-05-31T20:00:49.241 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:00:49 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:2Rhl0KA1WOxWz2tqNLYakr1DRVE=', 'Content-Length': '0'} 2023-05-31T20:00:49.244 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000004a938f8f911f4112-006477a771-110e-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:00:49 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:00:49.284 DEBUG:boto:b'pklxkd-13obj1000falseobj2023-05-31T20:00:08.058Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:00:49.285 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2023-05-31T20:00:49.285 DEBUG:boto:path=/pklxkd-13/obj 2023-05-31T20:00:49.286 DEBUG:boto:auth_path=/pklxkd-13/obj 2023-05-31T20:00:49.286 DEBUG:boto:path=/pklxkd-13/obj?versionId=nE3onm9L7VUWceoER2mn2jkX5yHK5GL 2023-05-31T20:00:49.287 DEBUG:boto:auth_path=/pklxkd-13/obj?versionId=nE3onm9L7VUWceoER2mn2jkX5yHK5GL 2023-05-31T20:00:49.287 DEBUG:boto:Method: GET 2023-05-31T20:00:49.287 DEBUG:boto:Path: /pklxkd-13/obj?versionId=nE3onm9L7VUWceoER2mn2jkX5yHK5GL 2023-05-31T20:00:49.288 DEBUG:boto:Data: 2023-05-31T20:00:49.288 DEBUG:boto:Headers: {} 2023-05-31T20:00:49.289 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:00:49.289 DEBUG:boto:Port: 8001 2023-05-31T20:00:49.290 DEBUG:boto:Params: {} 2023-05-31T20:00:49.290 DEBUG:boto:Token: None 2023-05-31T20:00:49.290 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:00:49 GMT /pklxkd-13/obj?versionId=nE3onm9L7VUWceoER2mn2jkX5yHK5GL 2023-05-31T20:00:49.291 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:oDJvS7rC1Z62llwhVDXXvqgLA2c= 2023-05-31T20:00:49.291 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:00:49 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:oDJvS7rC1Z62llwhVDXXvqgLA2c=', 'Content-Length': '0'} 2023-05-31T20:00:49.294 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:00:07 GMT'), ('x-amz-version-id', 'nE3onm9L7VUWceoER2mn2jkX5yHK5GL'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000000e114741e875af28-006477a771-1112-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:00:49 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:00:49.294 DEBUG:boto:path=/pklxkd-13/obj 2023-05-31T20:00:49.295 DEBUG:boto:auth_path=/pklxkd-13/obj 2023-05-31T20:00:49.295 DEBUG:boto:path=/pklxkd-13/obj?versionId=nE3onm9L7VUWceoER2mn2jkX5yHK5GL 2023-05-31T20:00:49.295 DEBUG:boto:auth_path=/pklxkd-13/obj?versionId=nE3onm9L7VUWceoER2mn2jkX5yHK5GL 2023-05-31T20:00:49.296 DEBUG:boto:Method: GET 2023-05-31T20:00:49.296 DEBUG:boto:Path: /pklxkd-13/obj?versionId=nE3onm9L7VUWceoER2mn2jkX5yHK5GL 2023-05-31T20:00:49.297 DEBUG:boto:Data: 2023-05-31T20:00:49.297 DEBUG:boto:Headers: {} 2023-05-31T20:00:49.297 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:00:49.298 DEBUG:boto:Port: 8000 2023-05-31T20:00:49.298 DEBUG:boto:Params: {} 2023-05-31T20:00:49.298 DEBUG:boto:Token: None 2023-05-31T20:00:49.299 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:00:49 GMT /pklxkd-13/obj?versionId=nE3onm9L7VUWceoER2mn2jkX5yHK5GL 2023-05-31T20:00:49.299 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:oDJvS7rC1Z62llwhVDXXvqgLA2c= 2023-05-31T20:00:49.300 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:00:49 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:oDJvS7rC1Z62llwhVDXXvqgLA2c=', 'Content-Length': '0'} 2023-05-31T20:00:49.302 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:00:07 GMT'), ('x-amz-version-id', 'nE3onm9L7VUWceoER2mn2jkX5yHK5GL'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000094388eebc97e77cc-006477a771-110e-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:00:49 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:00:49.302 DEBUG:boto:path=/pklxkd-13/obj 2023-05-31T20:00:49.303 DEBUG:boto:auth_path=/pklxkd-13/obj 2023-05-31T20:00:49.303 DEBUG:boto:path=/pklxkd-13/obj?versionId=nE3onm9L7VUWceoER2mn2jkX5yHK5GL 2023-05-31T20:00:49.303 DEBUG:boto:auth_path=/pklxkd-13/obj?versionId=nE3onm9L7VUWceoER2mn2jkX5yHK5GL 2023-05-31T20:00:49.304 DEBUG:boto:Method: HEAD 2023-05-31T20:00:49.304 DEBUG:boto:Path: /pklxkd-13/obj?versionId=nE3onm9L7VUWceoER2mn2jkX5yHK5GL 2023-05-31T20:00:49.304 DEBUG:boto:Data: 2023-05-31T20:00:49.305 DEBUG:boto:Headers: {} 2023-05-31T20:00:49.305 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:00:49.305 DEBUG:boto:Port: 8001 2023-05-31T20:00:49.306 DEBUG:boto:Params: {} 2023-05-31T20:00:49.306 DEBUG:boto:Token: None 2023-05-31T20:00:49.306 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:00:49 GMT /pklxkd-13/obj?versionId=nE3onm9L7VUWceoER2mn2jkX5yHK5GL 2023-05-31T20:00:49.307 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:lK2+4rBQTTJ7y5IzVuu0CY3Sf+M= 2023-05-31T20:00:49.307 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:00:49 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:lK2+4rBQTTJ7y5IzVuu0CY3Sf+M=', 'Content-Length': '0'} 2023-05-31T20:00:49.309 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:00:07 GMT'), ('x-amz-version-id', 'nE3onm9L7VUWceoER2mn2jkX5yHK5GL'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000e101e6a90398e0c9-006477a771-1112-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:00:49 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:00:49.310 DEBUG:boto:path=/pklxkd-13/obj 2023-05-31T20:00:49.310 DEBUG:boto:auth_path=/pklxkd-13/obj 2023-05-31T20:00:49.310 DEBUG:boto:path=/pklxkd-13/obj?versionId=nE3onm9L7VUWceoER2mn2jkX5yHK5GL 2023-05-31T20:00:49.311 DEBUG:boto:auth_path=/pklxkd-13/obj?versionId=nE3onm9L7VUWceoER2mn2jkX5yHK5GL 2023-05-31T20:00:49.311 DEBUG:boto:Method: HEAD 2023-05-31T20:00:49.311 DEBUG:boto:Path: /pklxkd-13/obj?versionId=nE3onm9L7VUWceoER2mn2jkX5yHK5GL 2023-05-31T20:00:49.312 DEBUG:boto:Data: 2023-05-31T20:00:49.312 DEBUG:boto:Headers: {} 2023-05-31T20:00:49.312 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:00:49.313 DEBUG:boto:Port: 8000 2023-05-31T20:00:49.313 DEBUG:boto:Params: {} 2023-05-31T20:00:49.313 DEBUG:boto:Token: None 2023-05-31T20:00:49.314 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:00:49 GMT /pklxkd-13/obj?versionId=nE3onm9L7VUWceoER2mn2jkX5yHK5GL 2023-05-31T20:00:49.314 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:lK2+4rBQTTJ7y5IzVuu0CY3Sf+M= 2023-05-31T20:00:49.315 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:00:49 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:lK2+4rBQTTJ7y5IzVuu0CY3Sf+M=', 'Content-Length': '0'} 2023-05-31T20:00:49.316 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:00:07 GMT'), ('x-amz-version-id', 'nE3onm9L7VUWceoER2mn2jkX5yHK5GL'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000a7ffa81762ed56ce-006477a771-110e-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:00:49 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:00:49.317 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2023-05-31T20:00:49.317 DEBUG:boto:path=/pklxkd-13/obj 2023-05-31T20:00:49.318 DEBUG:boto:auth_path=/pklxkd-13/obj 2023-05-31T20:00:49.318 DEBUG:boto:path=/pklxkd-13/obj?versionId=nE3onm9L7VUWceoER2mn2jkX5yHK5GL 2023-05-31T20:00:49.318 DEBUG:boto:auth_path=/pklxkd-13/obj?versionId=nE3onm9L7VUWceoER2mn2jkX5yHK5GL 2023-05-31T20:00:49.319 DEBUG:boto:Method: GET 2023-05-31T20:00:49.319 DEBUG:boto:Path: /pklxkd-13/obj?versionId=nE3onm9L7VUWceoER2mn2jkX5yHK5GL 2023-05-31T20:00:49.319 DEBUG:boto:Data: 2023-05-31T20:00:49.320 DEBUG:boto:Headers: {} 2023-05-31T20:00:49.320 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:00:49.320 DEBUG:boto:Port: 8001 2023-05-31T20:00:49.321 DEBUG:boto:Params: {} 2023-05-31T20:00:49.321 DEBUG:boto:Token: None 2023-05-31T20:00:49.322 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:00:49 GMT /pklxkd-13/obj?versionId=nE3onm9L7VUWceoER2mn2jkX5yHK5GL 2023-05-31T20:00:49.322 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:oDJvS7rC1Z62llwhVDXXvqgLA2c= 2023-05-31T20:00:49.322 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:00:49 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:oDJvS7rC1Z62llwhVDXXvqgLA2c=', 'Content-Length': '0'} 2023-05-31T20:00:49.324 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:00:07 GMT'), ('x-amz-version-id', 'nE3onm9L7VUWceoER2mn2jkX5yHK5GL'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000475a52c75879124e-006477a771-1112-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:00:49 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:00:49.325 DEBUG:boto:path=/pklxkd-13/obj 2023-05-31T20:00:49.326 DEBUG:boto:auth_path=/pklxkd-13/obj 2023-05-31T20:00:49.326 DEBUG:boto:path=/pklxkd-13/obj?versionId=nE3onm9L7VUWceoER2mn2jkX5yHK5GL 2023-05-31T20:00:49.326 DEBUG:boto:auth_path=/pklxkd-13/obj?versionId=nE3onm9L7VUWceoER2mn2jkX5yHK5GL 2023-05-31T20:00:49.327 DEBUG:boto:Method: GET 2023-05-31T20:00:49.327 DEBUG:boto:Path: /pklxkd-13/obj?versionId=nE3onm9L7VUWceoER2mn2jkX5yHK5GL 2023-05-31T20:00:49.327 DEBUG:boto:Data: 2023-05-31T20:00:49.328 DEBUG:boto:Headers: {} 2023-05-31T20:00:49.328 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:00:49.328 DEBUG:boto:Port: 8000 2023-05-31T20:00:49.329 DEBUG:boto:Params: {} 2023-05-31T20:00:49.329 DEBUG:boto:Token: None 2023-05-31T20:00:49.329 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:00:49 GMT /pklxkd-13/obj?versionId=nE3onm9L7VUWceoER2mn2jkX5yHK5GL 2023-05-31T20:00:49.330 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:oDJvS7rC1Z62llwhVDXXvqgLA2c= 2023-05-31T20:00:49.330 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:00:49 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:oDJvS7rC1Z62llwhVDXXvqgLA2c=', 'Content-Length': '0'} 2023-05-31T20:00:49.332 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:00:07 GMT'), ('x-amz-version-id', 'nE3onm9L7VUWceoER2mn2jkX5yHK5GL'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000c7e4631ea6c09fd7-006477a771-110e-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:00:49 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:00:49.333 DEBUG:boto:path=/pklxkd-13/ 2023-05-31T20:00:49.333 DEBUG:boto:auth_path=/pklxkd-13/ 2023-05-31T20:00:49.333 DEBUG:boto:path=/pklxkd-13/?prefix=obj 2023-05-31T20:00:49.334 DEBUG:boto:auth_path=/pklxkd-13/?prefix=obj 2023-05-31T20:00:49.334 DEBUG:boto:Method: GET 2023-05-31T20:00:49.334 DEBUG:boto:Path: /pklxkd-13/?prefix=obj 2023-05-31T20:00:49.335 DEBUG:boto:Data: 2023-05-31T20:00:49.335 DEBUG:boto:Headers: {} 2023-05-31T20:00:49.336 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:00:49.336 DEBUG:boto:Port: 8001 2023-05-31T20:00:49.336 DEBUG:boto:Params: {} 2023-05-31T20:00:49.337 DEBUG:boto:Token: None 2023-05-31T20:00:49.337 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:00:49 GMT /pklxkd-13/ 2023-05-31T20:00:49.338 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:2Rhl0KA1WOxWz2tqNLYakr1DRVE= 2023-05-31T20:00:49.338 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:00:49 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:2Rhl0KA1WOxWz2tqNLYakr1DRVE=', 'Content-Length': '0'} 2023-05-31T20:00:49.340 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000088b44e45c82e885a-006477a771-1112-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:00:49 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:00:49.388 DEBUG:boto:b'pklxkd-13obj1000falseobj2023-05-31T20:00:08.058Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:00:49.389 DEBUG:boto:path=/pklxkd-13/ 2023-05-31T20:00:49.390 DEBUG:boto:auth_path=/pklxkd-13/ 2023-05-31T20:00:49.390 DEBUG:boto:path=/pklxkd-13/?prefix=obj 2023-05-31T20:00:49.390 DEBUG:boto:auth_path=/pklxkd-13/?prefix=obj 2023-05-31T20:00:49.391 DEBUG:boto:Method: GET 2023-05-31T20:00:49.391 DEBUG:boto:Path: /pklxkd-13/?prefix=obj 2023-05-31T20:00:49.391 DEBUG:boto:Data: 2023-05-31T20:00:49.392 DEBUG:boto:Headers: {} 2023-05-31T20:00:49.392 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:00:49.392 DEBUG:boto:Port: 8000 2023-05-31T20:00:49.393 DEBUG:boto:Params: {} 2023-05-31T20:00:49.393 DEBUG:boto:Token: None 2023-05-31T20:00:49.394 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:00:49 GMT /pklxkd-13/ 2023-05-31T20:00:49.394 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:2Rhl0KA1WOxWz2tqNLYakr1DRVE= 2023-05-31T20:00:49.395 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:00:49 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:2Rhl0KA1WOxWz2tqNLYakr1DRVE=', 'Content-Length': '0'} 2023-05-31T20:00:49.397 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000034cdd0421510b289-006477a771-110e-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:00:49 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:00:49.398 DEBUG:boto:b'pklxkd-13obj1000falseobj2023-05-31T20:00:08.058Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:00:49.398 DEBUG:boto:path=/pklxkd-13/obj 2023-05-31T20:00:49.399 DEBUG:boto:auth_path=/pklxkd-13/obj 2023-05-31T20:00:49.399 DEBUG:boto:Method: HEAD 2023-05-31T20:00:49.399 DEBUG:boto:Path: /pklxkd-13/obj 2023-05-31T20:00:49.400 DEBUG:boto:Data: 2023-05-31T20:00:49.400 DEBUG:boto:Headers: {} 2023-05-31T20:00:49.400 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:00:49.401 DEBUG:boto:Port: 8001 2023-05-31T20:00:49.401 DEBUG:boto:Params: {} 2023-05-31T20:00:49.401 DEBUG:boto:Token: None 2023-05-31T20:00:49.402 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:00:49 GMT /pklxkd-13/obj 2023-05-31T20:00:49.402 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:Mv9iPJUU0LzM4+d+e0C61wmvE7Y= 2023-05-31T20:00:49.403 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:00:49 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:Mv9iPJUU0LzM4+d+e0C61wmvE7Y=', 'Content-Length': '0'} 2023-05-31T20:00:49.406 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:00:08 GMT'), ('x-amz-version-id', '.K0qHk3QiH4acREzLteeQLJmFzn4x-1'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000074daa21728265fc0-006477a771-1112-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:00:49 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:00:49.406 DEBUG:boto:path=/pklxkd-13/obj 2023-05-31T20:00:49.406 DEBUG:boto:auth_path=/pklxkd-13/obj 2023-05-31T20:00:49.407 DEBUG:boto:Method: HEAD 2023-05-31T20:00:49.407 DEBUG:boto:Path: /pklxkd-13/obj 2023-05-31T20:00:49.408 DEBUG:boto:Data: 2023-05-31T20:00:49.408 DEBUG:boto:Headers: {} 2023-05-31T20:00:49.408 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:00:49.409 DEBUG:boto:Port: 8000 2023-05-31T20:00:49.409 DEBUG:boto:Params: {} 2023-05-31T20:00:49.409 DEBUG:boto:Token: None 2023-05-31T20:00:49.410 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:00:49 GMT /pklxkd-13/obj 2023-05-31T20:00:49.410 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:Mv9iPJUU0LzM4+d+e0C61wmvE7Y= 2023-05-31T20:00:49.411 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:00:49 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:Mv9iPJUU0LzM4+d+e0C61wmvE7Y=', 'Content-Length': '0'} 2023-05-31T20:00:49.414 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:00:08 GMT'), ('x-amz-version-id', '.K0qHk3QiH4acREzLteeQLJmFzn4x-1'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000005241ef4c8484ac95-006477a771-110e-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:00:49 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:00:49.414 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2023-05-31T20:00:49.415 DEBUG:boto:path=/pklxkd-13/obj 2023-05-31T20:00:49.415 DEBUG:boto:auth_path=/pklxkd-13/obj 2023-05-31T20:00:49.415 DEBUG:boto:path=/pklxkd-13/obj?versionId=.K0qHk3QiH4acREzLteeQLJmFzn4x-1 2023-05-31T20:00:49.416 DEBUG:boto:auth_path=/pklxkd-13/obj?versionId=.K0qHk3QiH4acREzLteeQLJmFzn4x-1 2023-05-31T20:00:49.416 DEBUG:boto:Method: GET 2023-05-31T20:00:49.416 DEBUG:boto:Path: /pklxkd-13/obj?versionId=.K0qHk3QiH4acREzLteeQLJmFzn4x-1 2023-05-31T20:00:49.417 DEBUG:boto:Data: 2023-05-31T20:00:49.417 DEBUG:boto:Headers: {} 2023-05-31T20:00:49.417 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:00:49.418 DEBUG:boto:Port: 8001 2023-05-31T20:00:49.418 DEBUG:boto:Params: {} 2023-05-31T20:00:49.419 DEBUG:boto:Token: None 2023-05-31T20:00:49.419 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:00:49 GMT /pklxkd-13/obj?versionId=.K0qHk3QiH4acREzLteeQLJmFzn4x-1 2023-05-31T20:00:49.419 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:erctR9bIPl4pq01jjrIApCDKd2Y= 2023-05-31T20:00:49.420 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:00:49 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:erctR9bIPl4pq01jjrIApCDKd2Y=', 'Content-Length': '0'} 2023-05-31T20:00:49.422 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:00:08 GMT'), ('x-amz-version-id', '.K0qHk3QiH4acREzLteeQLJmFzn4x-1'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000d5f7d34daac76a83-006477a771-1112-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:00:49 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:00:49.422 DEBUG:boto:path=/pklxkd-13/obj 2023-05-31T20:00:49.423 DEBUG:boto:auth_path=/pklxkd-13/obj 2023-05-31T20:00:49.423 DEBUG:boto:path=/pklxkd-13/obj?versionId=.K0qHk3QiH4acREzLteeQLJmFzn4x-1 2023-05-31T20:00:49.423 DEBUG:boto:auth_path=/pklxkd-13/obj?versionId=.K0qHk3QiH4acREzLteeQLJmFzn4x-1 2023-05-31T20:00:49.424 DEBUG:boto:Method: GET 2023-05-31T20:00:49.424 DEBUG:boto:Path: /pklxkd-13/obj?versionId=.K0qHk3QiH4acREzLteeQLJmFzn4x-1 2023-05-31T20:00:49.425 DEBUG:boto:Data: 2023-05-31T20:00:49.425 DEBUG:boto:Headers: {} 2023-05-31T20:00:49.425 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:00:49.426 DEBUG:boto:Port: 8000 2023-05-31T20:00:49.426 DEBUG:boto:Params: {} 2023-05-31T20:00:49.426 DEBUG:boto:Token: None 2023-05-31T20:00:49.427 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:00:49 GMT /pklxkd-13/obj?versionId=.K0qHk3QiH4acREzLteeQLJmFzn4x-1 2023-05-31T20:00:49.427 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:erctR9bIPl4pq01jjrIApCDKd2Y= 2023-05-31T20:00:49.428 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:00:49 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:erctR9bIPl4pq01jjrIApCDKd2Y=', 'Content-Length': '0'} 2023-05-31T20:00:49.430 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:00:08 GMT'), ('x-amz-version-id', '.K0qHk3QiH4acREzLteeQLJmFzn4x-1'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000006ddb743f0df0e59-006477a771-110e-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:00:49 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:00:49.472 DEBUG:boto:path=/pklxkd-13/ 2023-05-31T20:00:49.473 DEBUG:boto:auth_path=/pklxkd-13/ 2023-05-31T20:00:49.473 DEBUG:boto:path=/pklxkd-13/?prefix=obj 2023-05-31T20:00:49.474 DEBUG:boto:auth_path=/pklxkd-13/?prefix=obj 2023-05-31T20:00:49.474 DEBUG:boto:Method: GET 2023-05-31T20:00:49.474 DEBUG:boto:Path: /pklxkd-13/?prefix=obj 2023-05-31T20:00:49.475 DEBUG:boto:Data: 2023-05-31T20:00:49.475 DEBUG:boto:Headers: {} 2023-05-31T20:00:49.475 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:00:49.476 DEBUG:boto:Port: 8001 2023-05-31T20:00:49.476 DEBUG:boto:Params: {} 2023-05-31T20:00:49.477 DEBUG:boto:Token: None 2023-05-31T20:00:49.477 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:00:49 GMT /pklxkd-13/ 2023-05-31T20:00:49.478 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:2Rhl0KA1WOxWz2tqNLYakr1DRVE= 2023-05-31T20:00:49.478 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:00:49 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:2Rhl0KA1WOxWz2tqNLYakr1DRVE=', 'Content-Length': '0'} 2023-05-31T20:00:49.481 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000fec436f38cda95fe-006477a771-1112-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:00:49 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:00:49.481 DEBUG:boto:b'pklxkd-13obj1000falseobj2023-05-31T20:00:08.058Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:00:49.482 DEBUG:boto:path=/pklxkd-13/ 2023-05-31T20:00:49.482 DEBUG:boto:auth_path=/pklxkd-13/ 2023-05-31T20:00:49.483 DEBUG:boto:path=/pklxkd-13/?prefix=obj 2023-05-31T20:00:49.483 DEBUG:boto:auth_path=/pklxkd-13/?prefix=obj 2023-05-31T20:00:49.484 DEBUG:boto:Method: GET 2023-05-31T20:00:49.484 DEBUG:boto:Path: /pklxkd-13/?prefix=obj 2023-05-31T20:00:49.484 DEBUG:boto:Data: 2023-05-31T20:00:49.485 DEBUG:boto:Headers: {} 2023-05-31T20:00:49.485 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:00:49.485 DEBUG:boto:Port: 8000 2023-05-31T20:00:49.486 DEBUG:boto:Params: {} 2023-05-31T20:00:49.486 DEBUG:boto:Token: None 2023-05-31T20:00:49.487 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:00:49 GMT /pklxkd-13/ 2023-05-31T20:00:49.487 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:2Rhl0KA1WOxWz2tqNLYakr1DRVE= 2023-05-31T20:00:49.487 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:00:49 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:2Rhl0KA1WOxWz2tqNLYakr1DRVE=', 'Content-Length': '0'} 2023-05-31T20:00:49.490 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000015f749f164524c7d-006477a771-110e-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:00:49 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:00:49.491 DEBUG:boto:b'pklxkd-13obj1000falseobj2023-05-31T20:00:08.058Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:00:49.491 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2023-05-31T20:00:49.492 DEBUG:boto:path=/pklxkd-13/obj 2023-05-31T20:00:49.492 DEBUG:boto:auth_path=/pklxkd-13/obj 2023-05-31T20:00:49.492 DEBUG:boto:path=/pklxkd-13/obj?versionId=LGC78se33tZFR9UpgQStirsP0mP9GEu 2023-05-31T20:00:49.493 DEBUG:boto:auth_path=/pklxkd-13/obj?versionId=LGC78se33tZFR9UpgQStirsP0mP9GEu 2023-05-31T20:00:49.493 DEBUG:boto:Method: GET 2023-05-31T20:00:49.493 DEBUG:boto:Path: /pklxkd-13/obj?versionId=LGC78se33tZFR9UpgQStirsP0mP9GEu 2023-05-31T20:00:49.494 DEBUG:boto:Data: 2023-05-31T20:00:49.494 DEBUG:boto:Headers: {} 2023-05-31T20:00:49.494 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:00:49.495 DEBUG:boto:Port: 8001 2023-05-31T20:00:49.495 DEBUG:boto:Params: {} 2023-05-31T20:00:49.496 DEBUG:boto:Token: None 2023-05-31T20:00:49.496 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:00:49 GMT /pklxkd-13/obj?versionId=LGC78se33tZFR9UpgQStirsP0mP9GEu 2023-05-31T20:00:49.496 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:l6EHuYovFzR1sIfcFXD3xQVDRfA= 2023-05-31T20:00:49.497 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:00:49 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:l6EHuYovFzR1sIfcFXD3xQVDRfA=', 'Content-Length': '0'} 2023-05-31T20:00:49.499 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:00:07 GMT'), ('x-amz-version-id', 'LGC78se33tZFR9UpgQStirsP0mP9GEu'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000002fb705e2f2eb90db-006477a771-1112-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:00:49 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:00:49.540 DEBUG:boto:path=/pklxkd-13/obj 2023-05-31T20:00:49.541 DEBUG:boto:auth_path=/pklxkd-13/obj 2023-05-31T20:00:49.541 DEBUG:boto:path=/pklxkd-13/obj?versionId=LGC78se33tZFR9UpgQStirsP0mP9GEu 2023-05-31T20:00:49.542 DEBUG:boto:auth_path=/pklxkd-13/obj?versionId=LGC78se33tZFR9UpgQStirsP0mP9GEu 2023-05-31T20:00:49.542 DEBUG:boto:Method: GET 2023-05-31T20:00:49.542 DEBUG:boto:Path: /pklxkd-13/obj?versionId=LGC78se33tZFR9UpgQStirsP0mP9GEu 2023-05-31T20:00:49.543 DEBUG:boto:Data: 2023-05-31T20:00:49.543 DEBUG:boto:Headers: {} 2023-05-31T20:00:49.544 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:00:49.544 DEBUG:boto:Port: 8000 2023-05-31T20:00:49.544 DEBUG:boto:Params: {} 2023-05-31T20:00:49.545 DEBUG:boto:Token: None 2023-05-31T20:00:49.545 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:00:49 GMT /pklxkd-13/obj?versionId=LGC78se33tZFR9UpgQStirsP0mP9GEu 2023-05-31T20:00:49.546 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:l6EHuYovFzR1sIfcFXD3xQVDRfA= 2023-05-31T20:00:49.546 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:00:49 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:l6EHuYovFzR1sIfcFXD3xQVDRfA=', 'Content-Length': '0'} 2023-05-31T20:00:49.548 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:00:07 GMT'), ('x-amz-version-id', 'LGC78se33tZFR9UpgQStirsP0mP9GEu'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000098081ce599243253-006477a771-110e-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:00:49 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:00:49.549 DEBUG:boto:path=/pklxkd-13/obj 2023-05-31T20:00:49.549 DEBUG:boto:auth_path=/pklxkd-13/obj 2023-05-31T20:00:49.550 DEBUG:boto:path=/pklxkd-13/obj?versionId=LGC78se33tZFR9UpgQStirsP0mP9GEu 2023-05-31T20:00:49.550 DEBUG:boto:auth_path=/pklxkd-13/obj?versionId=LGC78se33tZFR9UpgQStirsP0mP9GEu 2023-05-31T20:00:49.550 DEBUG:boto:Method: HEAD 2023-05-31T20:00:49.551 DEBUG:boto:Path: /pklxkd-13/obj?versionId=LGC78se33tZFR9UpgQStirsP0mP9GEu 2023-05-31T20:00:49.551 DEBUG:boto:Data: 2023-05-31T20:00:49.551 DEBUG:boto:Headers: {} 2023-05-31T20:00:49.552 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:00:49.552 DEBUG:boto:Port: 8001 2023-05-31T20:00:49.552 DEBUG:boto:Params: {} 2023-05-31T20:00:49.553 DEBUG:boto:Token: None 2023-05-31T20:00:49.553 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:00:49 GMT /pklxkd-13/obj?versionId=LGC78se33tZFR9UpgQStirsP0mP9GEu 2023-05-31T20:00:49.553 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:I1RzS9o3nF3bIVEed2G+jMzCNyQ= 2023-05-31T20:00:49.554 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:00:49 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:I1RzS9o3nF3bIVEed2G+jMzCNyQ=', 'Content-Length': '0'} 2023-05-31T20:00:49.555 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:00:07 GMT'), ('x-amz-version-id', 'LGC78se33tZFR9UpgQStirsP0mP9GEu'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000f20d90b9e8fece2f-006477a771-1112-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:00:49 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:00:49.556 DEBUG:boto:path=/pklxkd-13/obj 2023-05-31T20:00:49.556 DEBUG:boto:auth_path=/pklxkd-13/obj 2023-05-31T20:00:49.557 DEBUG:boto:path=/pklxkd-13/obj?versionId=LGC78se33tZFR9UpgQStirsP0mP9GEu 2023-05-31T20:00:49.557 DEBUG:boto:auth_path=/pklxkd-13/obj?versionId=LGC78se33tZFR9UpgQStirsP0mP9GEu 2023-05-31T20:00:49.557 DEBUG:boto:Method: HEAD 2023-05-31T20:00:49.558 DEBUG:boto:Path: /pklxkd-13/obj?versionId=LGC78se33tZFR9UpgQStirsP0mP9GEu 2023-05-31T20:00:49.558 DEBUG:boto:Data: 2023-05-31T20:00:49.558 DEBUG:boto:Headers: {} 2023-05-31T20:00:49.559 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:00:49.559 DEBUG:boto:Port: 8000 2023-05-31T20:00:49.559 DEBUG:boto:Params: {} 2023-05-31T20:00:49.560 DEBUG:boto:Token: None 2023-05-31T20:00:49.560 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:00:49 GMT /pklxkd-13/obj?versionId=LGC78se33tZFR9UpgQStirsP0mP9GEu 2023-05-31T20:00:49.561 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:I1RzS9o3nF3bIVEed2G+jMzCNyQ= 2023-05-31T20:00:49.561 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:00:49 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:I1RzS9o3nF3bIVEed2G+jMzCNyQ=', 'Content-Length': '0'} 2023-05-31T20:00:49.563 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:00:07 GMT'), ('x-amz-version-id', 'LGC78se33tZFR9UpgQStirsP0mP9GEu'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000071636a892642a326-006477a771-110e-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:00:49 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:00:49.563 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2023-05-31T20:00:49.564 DEBUG:boto:path=/pklxkd-13/obj 2023-05-31T20:00:49.564 DEBUG:boto:auth_path=/pklxkd-13/obj 2023-05-31T20:00:49.565 DEBUG:boto:path=/pklxkd-13/obj?versionId=LGC78se33tZFR9UpgQStirsP0mP9GEu 2023-05-31T20:00:49.565 DEBUG:boto:auth_path=/pklxkd-13/obj?versionId=LGC78se33tZFR9UpgQStirsP0mP9GEu 2023-05-31T20:00:49.565 DEBUG:boto:Method: GET 2023-05-31T20:00:49.566 DEBUG:boto:Path: /pklxkd-13/obj?versionId=LGC78se33tZFR9UpgQStirsP0mP9GEu 2023-05-31T20:00:49.566 DEBUG:boto:Data: 2023-05-31T20:00:49.566 DEBUG:boto:Headers: {} 2023-05-31T20:00:49.567 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:00:49.567 DEBUG:boto:Port: 8001 2023-05-31T20:00:49.567 DEBUG:boto:Params: {} 2023-05-31T20:00:49.568 DEBUG:boto:Token: None 2023-05-31T20:00:49.568 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:00:49 GMT /pklxkd-13/obj?versionId=LGC78se33tZFR9UpgQStirsP0mP9GEu 2023-05-31T20:00:49.569 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:l6EHuYovFzR1sIfcFXD3xQVDRfA= 2023-05-31T20:00:49.569 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:00:49 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:l6EHuYovFzR1sIfcFXD3xQVDRfA=', 'Content-Length': '0'} 2023-05-31T20:00:49.571 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:00:07 GMT'), ('x-amz-version-id', 'LGC78se33tZFR9UpgQStirsP0mP9GEu'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000008a436f527977beb6-006477a771-1112-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:00:49 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:00:49.571 DEBUG:boto:path=/pklxkd-13/obj 2023-05-31T20:00:49.572 DEBUG:boto:auth_path=/pklxkd-13/obj 2023-05-31T20:00:49.572 DEBUG:boto:path=/pklxkd-13/obj?versionId=LGC78se33tZFR9UpgQStirsP0mP9GEu 2023-05-31T20:00:49.573 DEBUG:boto:auth_path=/pklxkd-13/obj?versionId=LGC78se33tZFR9UpgQStirsP0mP9GEu 2023-05-31T20:00:49.573 DEBUG:boto:Method: GET 2023-05-31T20:00:49.573 DEBUG:boto:Path: /pklxkd-13/obj?versionId=LGC78se33tZFR9UpgQStirsP0mP9GEu 2023-05-31T20:00:49.574 DEBUG:boto:Data: 2023-05-31T20:00:49.574 DEBUG:boto:Headers: {} 2023-05-31T20:00:49.574 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:00:49.575 DEBUG:boto:Port: 8000 2023-05-31T20:00:49.575 DEBUG:boto:Params: {} 2023-05-31T20:00:49.575 DEBUG:boto:Token: None 2023-05-31T20:00:49.576 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:00:49 GMT /pklxkd-13/obj?versionId=LGC78se33tZFR9UpgQStirsP0mP9GEu 2023-05-31T20:00:49.576 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:l6EHuYovFzR1sIfcFXD3xQVDRfA= 2023-05-31T20:00:49.577 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:00:49 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:l6EHuYovFzR1sIfcFXD3xQVDRfA=', 'Content-Length': '0'} 2023-05-31T20:00:49.579 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:00:07 GMT'), ('x-amz-version-id', 'LGC78se33tZFR9UpgQStirsP0mP9GEu'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000e2454bd7dddaed05-006477a771-110e-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:00:49 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:00:49.579 DEBUG:boto:path=/pklxkd-13/ 2023-05-31T20:00:49.580 DEBUG:boto:auth_path=/pklxkd-13/ 2023-05-31T20:00:49.580 DEBUG:boto:path=/pklxkd-13/?prefix=obj 2023-05-31T20:00:49.580 DEBUG:boto:auth_path=/pklxkd-13/?prefix=obj 2023-05-31T20:00:49.581 DEBUG:boto:Method: GET 2023-05-31T20:00:49.581 DEBUG:boto:Path: /pklxkd-13/?prefix=obj 2023-05-31T20:00:49.581 DEBUG:boto:Data: 2023-05-31T20:00:49.582 DEBUG:boto:Headers: {} 2023-05-31T20:00:49.582 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:00:49.582 DEBUG:boto:Port: 8001 2023-05-31T20:00:49.583 DEBUG:boto:Params: {} 2023-05-31T20:00:49.583 DEBUG:boto:Token: None 2023-05-31T20:00:49.584 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:00:49 GMT /pklxkd-13/ 2023-05-31T20:00:49.584 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:2Rhl0KA1WOxWz2tqNLYakr1DRVE= 2023-05-31T20:00:49.584 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:00:49 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:2Rhl0KA1WOxWz2tqNLYakr1DRVE=', 'Content-Length': '0'} 2023-05-31T20:00:49.587 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000003718220346625b52-006477a771-1112-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:00:49 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:00:49.587 DEBUG:boto:b'pklxkd-13obj1000falseobj2023-05-31T20:00:08.058Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:00:49.588 DEBUG:boto:path=/pklxkd-13/ 2023-05-31T20:00:49.589 DEBUG:boto:auth_path=/pklxkd-13/ 2023-05-31T20:00:49.589 DEBUG:boto:path=/pklxkd-13/?prefix=obj 2023-05-31T20:00:49.589 DEBUG:boto:auth_path=/pklxkd-13/?prefix=obj 2023-05-31T20:00:49.590 DEBUG:boto:Method: GET 2023-05-31T20:00:49.590 DEBUG:boto:Path: /pklxkd-13/?prefix=obj 2023-05-31T20:00:49.590 DEBUG:boto:Data: 2023-05-31T20:00:49.591 DEBUG:boto:Headers: {} 2023-05-31T20:00:49.591 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:00:49.591 DEBUG:boto:Port: 8000 2023-05-31T20:00:49.592 DEBUG:boto:Params: {} 2023-05-31T20:00:49.592 DEBUG:boto:Token: None 2023-05-31T20:00:49.593 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:00:49 GMT /pklxkd-13/ 2023-05-31T20:00:49.593 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:2Rhl0KA1WOxWz2tqNLYakr1DRVE= 2023-05-31T20:00:49.593 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:00:49 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:2Rhl0KA1WOxWz2tqNLYakr1DRVE=', 'Content-Length': '0'} 2023-05-31T20:00:49.596 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000120cc25b3f25551d-006477a771-110e-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:00:49 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:00:49.597 DEBUG:boto:b'pklxkd-13obj1000falseobj2023-05-31T20:00:08.058Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:00:49.597 DEBUG:boto:path=/pklxkd-13/obj 2023-05-31T20:00:49.598 DEBUG:boto:auth_path=/pklxkd-13/obj 2023-05-31T20:00:49.598 DEBUG:boto:Method: HEAD 2023-05-31T20:00:49.598 DEBUG:boto:Path: /pklxkd-13/obj 2023-05-31T20:00:49.599 DEBUG:boto:Data: 2023-05-31T20:00:49.599 DEBUG:boto:Headers: {} 2023-05-31T20:00:49.599 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:00:49.600 DEBUG:boto:Port: 8001 2023-05-31T20:00:49.600 DEBUG:boto:Params: {} 2023-05-31T20:00:49.600 DEBUG:boto:Token: None 2023-05-31T20:00:49.601 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:00:49 GMT /pklxkd-13/obj 2023-05-31T20:00:49.601 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:Mv9iPJUU0LzM4+d+e0C61wmvE7Y= 2023-05-31T20:00:49.601 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:00:49 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:Mv9iPJUU0LzM4+d+e0C61wmvE7Y=', 'Content-Length': '0'} 2023-05-31T20:00:49.605 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:00:08 GMT'), ('x-amz-version-id', '.K0qHk3QiH4acREzLteeQLJmFzn4x-1'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000a78478eb362dedf2-006477a771-1112-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:00:49 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:00:49.605 DEBUG:boto:path=/pklxkd-13/obj 2023-05-31T20:00:49.606 DEBUG:boto:auth_path=/pklxkd-13/obj 2023-05-31T20:00:49.606 DEBUG:boto:Method: HEAD 2023-05-31T20:00:49.606 DEBUG:boto:Path: /pklxkd-13/obj 2023-05-31T20:00:49.607 DEBUG:boto:Data: 2023-05-31T20:00:49.607 DEBUG:boto:Headers: {} 2023-05-31T20:00:49.607 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:00:49.608 DEBUG:boto:Port: 8000 2023-05-31T20:00:49.608 DEBUG:boto:Params: {} 2023-05-31T20:00:49.608 DEBUG:boto:Token: None 2023-05-31T20:00:49.609 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:00:49 GMT /pklxkd-13/obj 2023-05-31T20:00:49.609 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:Mv9iPJUU0LzM4+d+e0C61wmvE7Y= 2023-05-31T20:00:49.609 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:00:49 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:Mv9iPJUU0LzM4+d+e0C61wmvE7Y=', 'Content-Length': '0'} 2023-05-31T20:00:49.612 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:00:08 GMT'), ('x-amz-version-id', '.K0qHk3QiH4acREzLteeQLJmFzn4x-1'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000080487b142bf11ee8-006477a771-110e-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:00:49 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:00:49.613 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2023-05-31T20:00:49.613 DEBUG:boto:path=/pklxkd-13/obj 2023-05-31T20:00:49.614 DEBUG:boto:auth_path=/pklxkd-13/obj 2023-05-31T20:00:49.614 DEBUG:boto:path=/pklxkd-13/obj?versionId=.K0qHk3QiH4acREzLteeQLJmFzn4x-1 2023-05-31T20:00:49.614 DEBUG:boto:auth_path=/pklxkd-13/obj?versionId=.K0qHk3QiH4acREzLteeQLJmFzn4x-1 2023-05-31T20:00:49.615 DEBUG:boto:Method: GET 2023-05-31T20:00:49.615 DEBUG:boto:Path: /pklxkd-13/obj?versionId=.K0qHk3QiH4acREzLteeQLJmFzn4x-1 2023-05-31T20:00:49.615 DEBUG:boto:Data: 2023-05-31T20:00:49.616 DEBUG:boto:Headers: {} 2023-05-31T20:00:49.616 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:00:49.616 DEBUG:boto:Port: 8001 2023-05-31T20:00:49.617 DEBUG:boto:Params: {} 2023-05-31T20:00:49.617 DEBUG:boto:Token: None 2023-05-31T20:00:49.618 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:00:49 GMT /pklxkd-13/obj?versionId=.K0qHk3QiH4acREzLteeQLJmFzn4x-1 2023-05-31T20:00:49.618 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:erctR9bIPl4pq01jjrIApCDKd2Y= 2023-05-31T20:00:49.618 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:00:49 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:erctR9bIPl4pq01jjrIApCDKd2Y=', 'Content-Length': '0'} 2023-05-31T20:00:49.620 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:00:08 GMT'), ('x-amz-version-id', '.K0qHk3QiH4acREzLteeQLJmFzn4x-1'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000021646fa20c28d060-006477a771-1112-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:00:49 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:00:49.621 DEBUG:boto:path=/pklxkd-13/obj 2023-05-31T20:00:49.621 DEBUG:boto:auth_path=/pklxkd-13/obj 2023-05-31T20:00:49.622 DEBUG:boto:path=/pklxkd-13/obj?versionId=.K0qHk3QiH4acREzLteeQLJmFzn4x-1 2023-05-31T20:00:49.622 DEBUG:boto:auth_path=/pklxkd-13/obj?versionId=.K0qHk3QiH4acREzLteeQLJmFzn4x-1 2023-05-31T20:00:49.622 DEBUG:boto:Method: GET 2023-05-31T20:00:49.623 DEBUG:boto:Path: /pklxkd-13/obj?versionId=.K0qHk3QiH4acREzLteeQLJmFzn4x-1 2023-05-31T20:00:49.623 DEBUG:boto:Data: 2023-05-31T20:00:49.623 DEBUG:boto:Headers: {} 2023-05-31T20:00:49.624 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:00:49.624 DEBUG:boto:Port: 8000 2023-05-31T20:00:49.624 DEBUG:boto:Params: {} 2023-05-31T20:00:49.625 DEBUG:boto:Token: None 2023-05-31T20:00:49.625 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:00:49 GMT /pklxkd-13/obj?versionId=.K0qHk3QiH4acREzLteeQLJmFzn4x-1 2023-05-31T20:00:49.626 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:erctR9bIPl4pq01jjrIApCDKd2Y= 2023-05-31T20:00:49.626 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:00:49 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:erctR9bIPl4pq01jjrIApCDKd2Y=', 'Content-Length': '0'} 2023-05-31T20:00:49.628 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:00:08 GMT'), ('x-amz-version-id', '.K0qHk3QiH4acREzLteeQLJmFzn4x-1'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000d80492cb4873ff64-006477a771-110e-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:00:49 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:00:49.668 DEBUG:boto:path=/pklxkd-13/ 2023-05-31T20:00:49.669 DEBUG:boto:auth_path=/pklxkd-13/ 2023-05-31T20:00:49.669 DEBUG:boto:path=/pklxkd-13/?prefix=obj 2023-05-31T20:00:49.670 DEBUG:boto:auth_path=/pklxkd-13/?prefix=obj 2023-05-31T20:00:49.670 DEBUG:boto:Method: GET 2023-05-31T20:00:49.670 DEBUG:boto:Path: /pklxkd-13/?prefix=obj 2023-05-31T20:00:49.671 DEBUG:boto:Data: 2023-05-31T20:00:49.671 DEBUG:boto:Headers: {} 2023-05-31T20:00:49.671 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:00:49.672 DEBUG:boto:Port: 8001 2023-05-31T20:00:49.672 DEBUG:boto:Params: {} 2023-05-31T20:00:49.672 DEBUG:boto:Token: None 2023-05-31T20:00:49.673 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:00:49 GMT /pklxkd-13/ 2023-05-31T20:00:49.673 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:2Rhl0KA1WOxWz2tqNLYakr1DRVE= 2023-05-31T20:00:49.674 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:00:49 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:2Rhl0KA1WOxWz2tqNLYakr1DRVE=', 'Content-Length': '0'} 2023-05-31T20:00:49.677 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000000167665c5144edae-006477a771-1112-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:00:49 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:00:49.677 DEBUG:boto:b'pklxkd-13obj1000falseobj2023-05-31T20:00:08.058Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:00:49.678 DEBUG:boto:path=/pklxkd-13/ 2023-05-31T20:00:49.678 DEBUG:boto:auth_path=/pklxkd-13/ 2023-05-31T20:00:49.679 DEBUG:boto:path=/pklxkd-13/?prefix=obj 2023-05-31T20:00:49.679 DEBUG:boto:auth_path=/pklxkd-13/?prefix=obj 2023-05-31T20:00:49.679 DEBUG:boto:Method: GET 2023-05-31T20:00:49.680 DEBUG:boto:Path: /pklxkd-13/?prefix=obj 2023-05-31T20:00:49.680 DEBUG:boto:Data: 2023-05-31T20:00:49.680 DEBUG:boto:Headers: {} 2023-05-31T20:00:49.681 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:00:49.681 DEBUG:boto:Port: 8000 2023-05-31T20:00:49.681 DEBUG:boto:Params: {} 2023-05-31T20:00:49.682 DEBUG:boto:Token: None 2023-05-31T20:00:49.682 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:00:49 GMT /pklxkd-13/ 2023-05-31T20:00:49.683 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:2Rhl0KA1WOxWz2tqNLYakr1DRVE= 2023-05-31T20:00:49.683 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:00:49 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:2Rhl0KA1WOxWz2tqNLYakr1DRVE=', 'Content-Length': '0'} 2023-05-31T20:00:49.686 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000fa0d6fc3a8176d48-006477a771-110e-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:00:49 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:00:49.686 DEBUG:boto:b'pklxkd-13obj1000falseobj2023-05-31T20:00:08.058Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:00:49.687 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2023-05-31T20:00:49.687 DEBUG:boto:path=/pklxkd-13/obj 2023-05-31T20:00:49.688 DEBUG:boto:auth_path=/pklxkd-13/obj 2023-05-31T20:00:49.688 DEBUG:boto:path=/pklxkd-13/obj?versionId=oJ31LeYY.9AEY1WQGQs-wBnfC1MDSr6 2023-05-31T20:00:49.688 DEBUG:boto:auth_path=/pklxkd-13/obj?versionId=oJ31LeYY.9AEY1WQGQs-wBnfC1MDSr6 2023-05-31T20:00:49.689 DEBUG:boto:Method: GET 2023-05-31T20:00:49.689 DEBUG:boto:Path: /pklxkd-13/obj?versionId=oJ31LeYY.9AEY1WQGQs-wBnfC1MDSr6 2023-05-31T20:00:49.689 DEBUG:boto:Data: 2023-05-31T20:00:49.690 DEBUG:boto:Headers: {} 2023-05-31T20:00:49.690 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:00:49.691 DEBUG:boto:Port: 8001 2023-05-31T20:00:49.691 DEBUG:boto:Params: {} 2023-05-31T20:00:49.691 DEBUG:boto:Token: None 2023-05-31T20:00:49.692 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:00:49 GMT /pklxkd-13/obj?versionId=oJ31LeYY.9AEY1WQGQs-wBnfC1MDSr6 2023-05-31T20:00:49.692 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:RkQJHzTftXNZ/9jLuKebVKhSWhg= 2023-05-31T20:00:49.693 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:00:49 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:RkQJHzTftXNZ/9jLuKebVKhSWhg=', 'Content-Length': '0'} 2023-05-31T20:00:49.695 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:00:07 GMT'), ('x-amz-version-id', 'oJ31LeYY.9AEY1WQGQs-wBnfC1MDSr6'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000e6bd3dc38952a3d3-006477a771-1112-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:00:49 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:00:49.736 DEBUG:boto:path=/pklxkd-13/obj 2023-05-31T20:00:49.737 DEBUG:boto:auth_path=/pklxkd-13/obj 2023-05-31T20:00:49.737 DEBUG:boto:path=/pklxkd-13/obj?versionId=oJ31LeYY.9AEY1WQGQs-wBnfC1MDSr6 2023-05-31T20:00:49.738 DEBUG:boto:auth_path=/pklxkd-13/obj?versionId=oJ31LeYY.9AEY1WQGQs-wBnfC1MDSr6 2023-05-31T20:00:49.738 DEBUG:boto:Method: GET 2023-05-31T20:00:49.738 DEBUG:boto:Path: /pklxkd-13/obj?versionId=oJ31LeYY.9AEY1WQGQs-wBnfC1MDSr6 2023-05-31T20:00:49.739 DEBUG:boto:Data: 2023-05-31T20:00:49.739 DEBUG:boto:Headers: {} 2023-05-31T20:00:49.740 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:00:49.740 DEBUG:boto:Port: 8000 2023-05-31T20:00:49.740 DEBUG:boto:Params: {} 2023-05-31T20:00:49.741 DEBUG:boto:Token: None 2023-05-31T20:00:49.741 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:00:49 GMT /pklxkd-13/obj?versionId=oJ31LeYY.9AEY1WQGQs-wBnfC1MDSr6 2023-05-31T20:00:49.742 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:RkQJHzTftXNZ/9jLuKebVKhSWhg= 2023-05-31T20:00:49.742 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:00:49 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:RkQJHzTftXNZ/9jLuKebVKhSWhg=', 'Content-Length': '0'} 2023-05-31T20:00:49.744 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:00:07 GMT'), ('x-amz-version-id', 'oJ31LeYY.9AEY1WQGQs-wBnfC1MDSr6'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000931ef92fe079fa53-006477a771-110e-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:00:49 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:00:49.745 DEBUG:boto:path=/pklxkd-13/obj 2023-05-31T20:00:49.745 DEBUG:boto:auth_path=/pklxkd-13/obj 2023-05-31T20:00:49.745 DEBUG:boto:path=/pklxkd-13/obj?versionId=oJ31LeYY.9AEY1WQGQs-wBnfC1MDSr6 2023-05-31T20:00:49.746 DEBUG:boto:auth_path=/pklxkd-13/obj?versionId=oJ31LeYY.9AEY1WQGQs-wBnfC1MDSr6 2023-05-31T20:00:49.746 DEBUG:boto:Method: HEAD 2023-05-31T20:00:49.746 DEBUG:boto:Path: /pklxkd-13/obj?versionId=oJ31LeYY.9AEY1WQGQs-wBnfC1MDSr6 2023-05-31T20:00:49.747 DEBUG:boto:Data: 2023-05-31T20:00:49.747 DEBUG:boto:Headers: {} 2023-05-31T20:00:49.748 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:00:49.748 DEBUG:boto:Port: 8001 2023-05-31T20:00:49.748 DEBUG:boto:Params: {} 2023-05-31T20:00:49.749 DEBUG:boto:Token: None 2023-05-31T20:00:49.749 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:00:49 GMT /pklxkd-13/obj?versionId=oJ31LeYY.9AEY1WQGQs-wBnfC1MDSr6 2023-05-31T20:00:49.749 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:hZhyT34JZstHzEF+JjZVDlt5+20= 2023-05-31T20:00:49.750 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:00:49 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:hZhyT34JZstHzEF+JjZVDlt5+20=', 'Content-Length': '0'} 2023-05-31T20:00:49.752 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:00:07 GMT'), ('x-amz-version-id', 'oJ31LeYY.9AEY1WQGQs-wBnfC1MDSr6'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000220a02f57668e007-006477a771-1112-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:00:49 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:00:49.752 DEBUG:boto:path=/pklxkd-13/obj 2023-05-31T20:00:49.753 DEBUG:boto:auth_path=/pklxkd-13/obj 2023-05-31T20:00:49.753 DEBUG:boto:path=/pklxkd-13/obj?versionId=oJ31LeYY.9AEY1WQGQs-wBnfC1MDSr6 2023-05-31T20:00:49.753 DEBUG:boto:auth_path=/pklxkd-13/obj?versionId=oJ31LeYY.9AEY1WQGQs-wBnfC1MDSr6 2023-05-31T20:00:49.754 DEBUG:boto:Method: HEAD 2023-05-31T20:00:49.754 DEBUG:boto:Path: /pklxkd-13/obj?versionId=oJ31LeYY.9AEY1WQGQs-wBnfC1MDSr6 2023-05-31T20:00:49.754 DEBUG:boto:Data: 2023-05-31T20:00:49.755 DEBUG:boto:Headers: {} 2023-05-31T20:00:49.755 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:00:49.755 DEBUG:boto:Port: 8000 2023-05-31T20:00:49.756 DEBUG:boto:Params: {} 2023-05-31T20:00:49.756 DEBUG:boto:Token: None 2023-05-31T20:00:49.757 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:00:49 GMT /pklxkd-13/obj?versionId=oJ31LeYY.9AEY1WQGQs-wBnfC1MDSr6 2023-05-31T20:00:49.757 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:hZhyT34JZstHzEF+JjZVDlt5+20= 2023-05-31T20:00:49.757 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:00:49 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:hZhyT34JZstHzEF+JjZVDlt5+20=', 'Content-Length': '0'} 2023-05-31T20:00:49.759 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:00:07 GMT'), ('x-amz-version-id', 'oJ31LeYY.9AEY1WQGQs-wBnfC1MDSr6'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000009ab3334b6b0ee08c-006477a771-110e-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:00:49 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:00:49.760 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2023-05-31T20:00:49.760 DEBUG:boto:path=/pklxkd-13/obj 2023-05-31T20:00:49.760 DEBUG:boto:auth_path=/pklxkd-13/obj 2023-05-31T20:00:49.761 DEBUG:boto:path=/pklxkd-13/obj?versionId=oJ31LeYY.9AEY1WQGQs-wBnfC1MDSr6 2023-05-31T20:00:49.761 DEBUG:boto:auth_path=/pklxkd-13/obj?versionId=oJ31LeYY.9AEY1WQGQs-wBnfC1MDSr6 2023-05-31T20:00:49.762 DEBUG:boto:Method: GET 2023-05-31T20:00:49.762 DEBUG:boto:Path: /pklxkd-13/obj?versionId=oJ31LeYY.9AEY1WQGQs-wBnfC1MDSr6 2023-05-31T20:00:49.762 DEBUG:boto:Data: 2023-05-31T20:00:49.763 DEBUG:boto:Headers: {} 2023-05-31T20:00:49.763 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:00:49.763 DEBUG:boto:Port: 8001 2023-05-31T20:00:49.764 DEBUG:boto:Params: {} 2023-05-31T20:00:49.764 DEBUG:boto:Token: None 2023-05-31T20:00:49.765 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:00:49 GMT /pklxkd-13/obj?versionId=oJ31LeYY.9AEY1WQGQs-wBnfC1MDSr6 2023-05-31T20:00:49.765 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:RkQJHzTftXNZ/9jLuKebVKhSWhg= 2023-05-31T20:00:49.766 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:00:49 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:RkQJHzTftXNZ/9jLuKebVKhSWhg=', 'Content-Length': '0'} 2023-05-31T20:00:49.767 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:00:07 GMT'), ('x-amz-version-id', 'oJ31LeYY.9AEY1WQGQs-wBnfC1MDSr6'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000008e3e1f79e812e4b6-006477a771-1112-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:00:49 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:00:49.768 DEBUG:boto:path=/pklxkd-13/obj 2023-05-31T20:00:49.768 DEBUG:boto:auth_path=/pklxkd-13/obj 2023-05-31T20:00:49.769 DEBUG:boto:path=/pklxkd-13/obj?versionId=oJ31LeYY.9AEY1WQGQs-wBnfC1MDSr6 2023-05-31T20:00:49.769 DEBUG:boto:auth_path=/pklxkd-13/obj?versionId=oJ31LeYY.9AEY1WQGQs-wBnfC1MDSr6 2023-05-31T20:00:49.770 DEBUG:boto:Method: GET 2023-05-31T20:00:49.770 DEBUG:boto:Path: /pklxkd-13/obj?versionId=oJ31LeYY.9AEY1WQGQs-wBnfC1MDSr6 2023-05-31T20:00:49.770 DEBUG:boto:Data: 2023-05-31T20:00:49.771 DEBUG:boto:Headers: {} 2023-05-31T20:00:49.771 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:00:49.771 DEBUG:boto:Port: 8000 2023-05-31T20:00:49.772 DEBUG:boto:Params: {} 2023-05-31T20:00:49.772 DEBUG:boto:Token: None 2023-05-31T20:00:49.773 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:00:49 GMT /pklxkd-13/obj?versionId=oJ31LeYY.9AEY1WQGQs-wBnfC1MDSr6 2023-05-31T20:00:49.773 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:RkQJHzTftXNZ/9jLuKebVKhSWhg= 2023-05-31T20:00:49.774 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:00:49 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:RkQJHzTftXNZ/9jLuKebVKhSWhg=', 'Content-Length': '0'} 2023-05-31T20:00:49.775 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:00:07 GMT'), ('x-amz-version-id', 'oJ31LeYY.9AEY1WQGQs-wBnfC1MDSr6'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000a13cf7374e234d79-006477a771-110e-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:00:49 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:00:49.776 DEBUG:boto:path=/pklxkd-13/ 2023-05-31T20:00:49.776 DEBUG:boto:auth_path=/pklxkd-13/ 2023-05-31T20:00:49.777 DEBUG:boto:path=/pklxkd-13/?prefix=obj 2023-05-31T20:00:49.777 DEBUG:boto:auth_path=/pklxkd-13/?prefix=obj 2023-05-31T20:00:49.778 DEBUG:boto:Method: GET 2023-05-31T20:00:49.778 DEBUG:boto:Path: /pklxkd-13/?prefix=obj 2023-05-31T20:00:49.778 DEBUG:boto:Data: 2023-05-31T20:00:49.779 DEBUG:boto:Headers: {} 2023-05-31T20:00:49.779 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:00:49.779 DEBUG:boto:Port: 8001 2023-05-31T20:00:49.780 DEBUG:boto:Params: {} 2023-05-31T20:00:49.780 DEBUG:boto:Token: None 2023-05-31T20:00:49.781 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:00:49 GMT /pklxkd-13/ 2023-05-31T20:00:49.781 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:2Rhl0KA1WOxWz2tqNLYakr1DRVE= 2023-05-31T20:00:49.782 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:00:49 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:2Rhl0KA1WOxWz2tqNLYakr1DRVE=', 'Content-Length': '0'} 2023-05-31T20:00:49.784 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000760a2a9352bb8a75-006477a771-1112-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:00:49 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:00:49.785 DEBUG:boto:b'pklxkd-13obj1000falseobj2023-05-31T20:00:08.058Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:00:49.785 DEBUG:boto:path=/pklxkd-13/ 2023-05-31T20:00:49.786 DEBUG:boto:auth_path=/pklxkd-13/ 2023-05-31T20:00:49.786 DEBUG:boto:path=/pklxkd-13/?prefix=obj 2023-05-31T20:00:49.787 DEBUG:boto:auth_path=/pklxkd-13/?prefix=obj 2023-05-31T20:00:49.787 DEBUG:boto:Method: GET 2023-05-31T20:00:49.787 DEBUG:boto:Path: /pklxkd-13/?prefix=obj 2023-05-31T20:00:49.788 DEBUG:boto:Data: 2023-05-31T20:00:49.788 DEBUG:boto:Headers: {} 2023-05-31T20:00:49.788 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:00:49.789 DEBUG:boto:Port: 8000 2023-05-31T20:00:49.789 DEBUG:boto:Params: {} 2023-05-31T20:00:49.790 DEBUG:boto:Token: None 2023-05-31T20:00:49.790 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:00:49 GMT /pklxkd-13/ 2023-05-31T20:00:49.791 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:2Rhl0KA1WOxWz2tqNLYakr1DRVE= 2023-05-31T20:00:49.791 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:00:49 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:2Rhl0KA1WOxWz2tqNLYakr1DRVE=', 'Content-Length': '0'} 2023-05-31T20:00:49.794 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c7d16bb0eca8119d-006477a771-110e-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:00:49 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:00:49.794 DEBUG:boto:b'pklxkd-13obj1000falseobj2023-05-31T20:00:08.058Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:00:49.795 DEBUG:boto:path=/pklxkd-13/obj 2023-05-31T20:00:49.795 DEBUG:boto:auth_path=/pklxkd-13/obj 2023-05-31T20:00:49.796 DEBUG:boto:Method: HEAD 2023-05-31T20:00:49.796 DEBUG:boto:Path: /pklxkd-13/obj 2023-05-31T20:00:49.796 DEBUG:boto:Data: 2023-05-31T20:00:49.797 DEBUG:boto:Headers: {} 2023-05-31T20:00:49.797 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:00:49.797 DEBUG:boto:Port: 8001 2023-05-31T20:00:49.798 DEBUG:boto:Params: {} 2023-05-31T20:00:49.798 DEBUG:boto:Token: None 2023-05-31T20:00:49.799 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:00:49 GMT /pklxkd-13/obj 2023-05-31T20:00:49.799 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:Mv9iPJUU0LzM4+d+e0C61wmvE7Y= 2023-05-31T20:00:49.799 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:00:49 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:Mv9iPJUU0LzM4+d+e0C61wmvE7Y=', 'Content-Length': '0'} 2023-05-31T20:00:49.803 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:00:08 GMT'), ('x-amz-version-id', '.K0qHk3QiH4acREzLteeQLJmFzn4x-1'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000372dab31d1cece59-006477a771-1112-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:00:49 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:00:49.803 DEBUG:boto:path=/pklxkd-13/obj 2023-05-31T20:00:49.804 DEBUG:boto:auth_path=/pklxkd-13/obj 2023-05-31T20:00:49.804 DEBUG:boto:Method: HEAD 2023-05-31T20:00:49.804 DEBUG:boto:Path: /pklxkd-13/obj 2023-05-31T20:00:49.805 DEBUG:boto:Data: 2023-05-31T20:00:49.805 DEBUG:boto:Headers: {} 2023-05-31T20:00:49.805 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:00:49.806 DEBUG:boto:Port: 8000 2023-05-31T20:00:49.806 DEBUG:boto:Params: {} 2023-05-31T20:00:49.806 DEBUG:boto:Token: None 2023-05-31T20:00:49.807 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:00:49 GMT /pklxkd-13/obj 2023-05-31T20:00:49.807 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:Mv9iPJUU0LzM4+d+e0C61wmvE7Y= 2023-05-31T20:00:49.808 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:00:49 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:Mv9iPJUU0LzM4+d+e0C61wmvE7Y=', 'Content-Length': '0'} 2023-05-31T20:00:49.811 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:00:08 GMT'), ('x-amz-version-id', '.K0qHk3QiH4acREzLteeQLJmFzn4x-1'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000003494664c233f2f32-006477a771-110e-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:00:49 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:00:49.811 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2023-05-31T20:00:49.812 DEBUG:boto:path=/pklxkd-13/obj 2023-05-31T20:00:49.812 DEBUG:boto:auth_path=/pklxkd-13/obj 2023-05-31T20:00:49.813 DEBUG:boto:path=/pklxkd-13/obj?versionId=.K0qHk3QiH4acREzLteeQLJmFzn4x-1 2023-05-31T20:00:49.813 DEBUG:boto:auth_path=/pklxkd-13/obj?versionId=.K0qHk3QiH4acREzLteeQLJmFzn4x-1 2023-05-31T20:00:49.814 DEBUG:boto:Method: GET 2023-05-31T20:00:49.814 DEBUG:boto:Path: /pklxkd-13/obj?versionId=.K0qHk3QiH4acREzLteeQLJmFzn4x-1 2023-05-31T20:00:49.814 DEBUG:boto:Data: 2023-05-31T20:00:49.815 DEBUG:boto:Headers: {} 2023-05-31T20:00:49.815 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:00:49.815 DEBUG:boto:Port: 8001 2023-05-31T20:00:49.816 DEBUG:boto:Params: {} 2023-05-31T20:00:49.816 DEBUG:boto:Token: None 2023-05-31T20:00:49.817 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:00:49 GMT /pklxkd-13/obj?versionId=.K0qHk3QiH4acREzLteeQLJmFzn4x-1 2023-05-31T20:00:49.817 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:erctR9bIPl4pq01jjrIApCDKd2Y= 2023-05-31T20:00:49.818 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:00:49 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:erctR9bIPl4pq01jjrIApCDKd2Y=', 'Content-Length': '0'} 2023-05-31T20:00:49.819 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:00:08 GMT'), ('x-amz-version-id', '.K0qHk3QiH4acREzLteeQLJmFzn4x-1'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000ee0de4306bcc8dd6-006477a771-1112-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:00:49 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:00:49.820 DEBUG:boto:path=/pklxkd-13/obj 2023-05-31T20:00:49.821 DEBUG:boto:auth_path=/pklxkd-13/obj 2023-05-31T20:00:49.821 DEBUG:boto:path=/pklxkd-13/obj?versionId=.K0qHk3QiH4acREzLteeQLJmFzn4x-1 2023-05-31T20:00:49.821 DEBUG:boto:auth_path=/pklxkd-13/obj?versionId=.K0qHk3QiH4acREzLteeQLJmFzn4x-1 2023-05-31T20:00:49.822 DEBUG:boto:Method: GET 2023-05-31T20:00:49.822 DEBUG:boto:Path: /pklxkd-13/obj?versionId=.K0qHk3QiH4acREzLteeQLJmFzn4x-1 2023-05-31T20:00:49.822 DEBUG:boto:Data: 2023-05-31T20:00:49.823 DEBUG:boto:Headers: {} 2023-05-31T20:00:49.823 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:00:49.824 DEBUG:boto:Port: 8000 2023-05-31T20:00:49.824 DEBUG:boto:Params: {} 2023-05-31T20:00:49.824 DEBUG:boto:Token: None 2023-05-31T20:00:49.825 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:00:49 GMT /pklxkd-13/obj?versionId=.K0qHk3QiH4acREzLteeQLJmFzn4x-1 2023-05-31T20:00:49.825 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:erctR9bIPl4pq01jjrIApCDKd2Y= 2023-05-31T20:00:49.826 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:00:49 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:erctR9bIPl4pq01jjrIApCDKd2Y=', 'Content-Length': '0'} 2023-05-31T20:00:49.827 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:00:08 GMT'), ('x-amz-version-id', '.K0qHk3QiH4acREzLteeQLJmFzn4x-1'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000006d0ff4c830e0d1ec-006477a771-110e-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:00:49 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:00:49.828 DEBUG:boto:path=/pklxkd-13/ 2023-05-31T20:00:49.829 DEBUG:boto:auth_path=/pklxkd-13/ 2023-05-31T20:00:49.829 DEBUG:boto:path=/pklxkd-13/?prefix=obj 2023-05-31T20:00:49.829 DEBUG:boto:auth_path=/pklxkd-13/?prefix=obj 2023-05-31T20:00:49.830 DEBUG:boto:Method: GET 2023-05-31T20:00:49.830 DEBUG:boto:Path: /pklxkd-13/?prefix=obj 2023-05-31T20:00:49.830 DEBUG:boto:Data: 2023-05-31T20:00:49.831 DEBUG:boto:Headers: {} 2023-05-31T20:00:49.831 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:00:49.832 DEBUG:boto:Port: 8001 2023-05-31T20:00:49.832 DEBUG:boto:Params: {} 2023-05-31T20:00:49.832 DEBUG:boto:Token: None 2023-05-31T20:00:49.833 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:00:49 GMT /pklxkd-13/ 2023-05-31T20:00:49.833 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:2Rhl0KA1WOxWz2tqNLYakr1DRVE= 2023-05-31T20:00:49.834 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:00:49 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:2Rhl0KA1WOxWz2tqNLYakr1DRVE=', 'Content-Length': '0'} 2023-05-31T20:00:49.836 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000024bbc73fb53dbfd4-006477a771-1112-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:00:49 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:00:49.837 DEBUG:boto:b'pklxkd-13obj1000falseobj2023-05-31T20:00:08.058Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:00:49.838 DEBUG:boto:path=/pklxkd-13/ 2023-05-31T20:00:49.838 DEBUG:boto:auth_path=/pklxkd-13/ 2023-05-31T20:00:49.838 DEBUG:boto:path=/pklxkd-13/?prefix=obj 2023-05-31T20:00:49.839 DEBUG:boto:auth_path=/pklxkd-13/?prefix=obj 2023-05-31T20:00:49.839 DEBUG:boto:Method: GET 2023-05-31T20:00:49.839 DEBUG:boto:Path: /pklxkd-13/?prefix=obj 2023-05-31T20:00:49.840 DEBUG:boto:Data: 2023-05-31T20:00:49.840 DEBUG:boto:Headers: {} 2023-05-31T20:00:49.841 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:00:49.841 DEBUG:boto:Port: 8000 2023-05-31T20:00:49.841 DEBUG:boto:Params: {} 2023-05-31T20:00:49.842 DEBUG:boto:Token: None 2023-05-31T20:00:49.842 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:00:49 GMT /pklxkd-13/ 2023-05-31T20:00:49.843 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:2Rhl0KA1WOxWz2tqNLYakr1DRVE= 2023-05-31T20:00:49.843 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:00:49 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:2Rhl0KA1WOxWz2tqNLYakr1DRVE=', 'Content-Length': '0'} 2023-05-31T20:00:49.846 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000006ed428f59a4b3163-006477a771-110e-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:00:49 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:00:49.846 DEBUG:boto:b'pklxkd-13obj1000falseobj2023-05-31T20:00:08.058Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:00:49.847 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2023-05-31T20:00:49.848 DEBUG:boto:path=/pklxkd-13/obj 2023-05-31T20:00:49.848 DEBUG:boto:auth_path=/pklxkd-13/obj 2023-05-31T20:00:49.848 DEBUG:boto:path=/pklxkd-13/obj?versionId=Wr6wnEZSP3WRSGvZ8nB1.KinMc4wxvZ 2023-05-31T20:00:49.849 DEBUG:boto:auth_path=/pklxkd-13/obj?versionId=Wr6wnEZSP3WRSGvZ8nB1.KinMc4wxvZ 2023-05-31T20:00:49.849 DEBUG:boto:Method: GET 2023-05-31T20:00:49.849 DEBUG:boto:Path: /pklxkd-13/obj?versionId=Wr6wnEZSP3WRSGvZ8nB1.KinMc4wxvZ 2023-05-31T20:00:49.850 DEBUG:boto:Data: 2023-05-31T20:00:49.850 DEBUG:boto:Headers: {} 2023-05-31T20:00:49.851 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:00:49.851 DEBUG:boto:Port: 8001 2023-05-31T20:00:49.851 DEBUG:boto:Params: {} 2023-05-31T20:00:49.852 DEBUG:boto:Token: None 2023-05-31T20:00:49.852 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:00:49 GMT /pklxkd-13/obj?versionId=Wr6wnEZSP3WRSGvZ8nB1.KinMc4wxvZ 2023-05-31T20:00:49.853 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:WnETOPfSjulxCwDuMBzA1WBP14M= 2023-05-31T20:00:49.853 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:00:49 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:WnETOPfSjulxCwDuMBzA1WBP14M=', 'Content-Length': '0'} 2023-05-31T20:00:49.855 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:00:07 GMT'), ('x-amz-version-id', 'Wr6wnEZSP3WRSGvZ8nB1.KinMc4wxvZ'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000010d8400a0747f3d1-006477a771-1112-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:00:49 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:00:49.896 DEBUG:boto:path=/pklxkd-13/obj 2023-05-31T20:00:49.897 DEBUG:boto:auth_path=/pklxkd-13/obj 2023-05-31T20:00:49.897 DEBUG:boto:path=/pklxkd-13/obj?versionId=Wr6wnEZSP3WRSGvZ8nB1.KinMc4wxvZ 2023-05-31T20:00:49.898 DEBUG:boto:auth_path=/pklxkd-13/obj?versionId=Wr6wnEZSP3WRSGvZ8nB1.KinMc4wxvZ 2023-05-31T20:00:49.898 DEBUG:boto:Method: GET 2023-05-31T20:00:49.898 DEBUG:boto:Path: /pklxkd-13/obj?versionId=Wr6wnEZSP3WRSGvZ8nB1.KinMc4wxvZ 2023-05-31T20:00:49.899 DEBUG:boto:Data: 2023-05-31T20:00:49.899 DEBUG:boto:Headers: {} 2023-05-31T20:00:49.900 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:00:49.900 DEBUG:boto:Port: 8000 2023-05-31T20:00:49.900 DEBUG:boto:Params: {} 2023-05-31T20:00:49.901 DEBUG:boto:Token: None 2023-05-31T20:00:49.901 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:00:49 GMT /pklxkd-13/obj?versionId=Wr6wnEZSP3WRSGvZ8nB1.KinMc4wxvZ 2023-05-31T20:00:49.902 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:WnETOPfSjulxCwDuMBzA1WBP14M= 2023-05-31T20:00:49.902 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:00:49 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:WnETOPfSjulxCwDuMBzA1WBP14M=', 'Content-Length': '0'} 2023-05-31T20:00:49.904 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:00:07 GMT'), ('x-amz-version-id', 'Wr6wnEZSP3WRSGvZ8nB1.KinMc4wxvZ'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000e2b6e7fea19a05c0-006477a771-110e-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:00:49 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:00:49.908 DEBUG:boto:path=/pklxkd-13/obj 2023-05-31T20:00:49.909 DEBUG:boto:auth_path=/pklxkd-13/obj 2023-05-31T20:00:49.909 DEBUG:boto:path=/pklxkd-13/obj?versionId=Wr6wnEZSP3WRSGvZ8nB1.KinMc4wxvZ 2023-05-31T20:00:49.910 DEBUG:boto:auth_path=/pklxkd-13/obj?versionId=Wr6wnEZSP3WRSGvZ8nB1.KinMc4wxvZ 2023-05-31T20:00:49.910 DEBUG:boto:Method: HEAD 2023-05-31T20:00:49.910 DEBUG:boto:Path: /pklxkd-13/obj?versionId=Wr6wnEZSP3WRSGvZ8nB1.KinMc4wxvZ 2023-05-31T20:00:49.911 DEBUG:boto:Data: 2023-05-31T20:00:49.911 DEBUG:boto:Headers: {} 2023-05-31T20:00:49.911 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:00:49.912 DEBUG:boto:Port: 8001 2023-05-31T20:00:49.912 DEBUG:boto:Params: {} 2023-05-31T20:00:49.912 DEBUG:boto:Token: None 2023-05-31T20:00:49.913 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:00:49 GMT /pklxkd-13/obj?versionId=Wr6wnEZSP3WRSGvZ8nB1.KinMc4wxvZ 2023-05-31T20:00:49.913 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:73aRBUO70WzcyBuxWh1CZ1E6vLs= 2023-05-31T20:00:49.914 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:00:49 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:73aRBUO70WzcyBuxWh1CZ1E6vLs=', 'Content-Length': '0'} 2023-05-31T20:00:49.916 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:00:07 GMT'), ('x-amz-version-id', 'Wr6wnEZSP3WRSGvZ8nB1.KinMc4wxvZ'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000000ee5ac75d5a48d0a-006477a771-1112-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:00:49 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:00:49.916 DEBUG:boto:path=/pklxkd-13/obj 2023-05-31T20:00:49.917 DEBUG:boto:auth_path=/pklxkd-13/obj 2023-05-31T20:00:49.917 DEBUG:boto:path=/pklxkd-13/obj?versionId=Wr6wnEZSP3WRSGvZ8nB1.KinMc4wxvZ 2023-05-31T20:00:49.917 DEBUG:boto:auth_path=/pklxkd-13/obj?versionId=Wr6wnEZSP3WRSGvZ8nB1.KinMc4wxvZ 2023-05-31T20:00:49.918 DEBUG:boto:Method: HEAD 2023-05-31T20:00:49.918 DEBUG:boto:Path: /pklxkd-13/obj?versionId=Wr6wnEZSP3WRSGvZ8nB1.KinMc4wxvZ 2023-05-31T20:00:49.919 DEBUG:boto:Data: 2023-05-31T20:00:49.919 DEBUG:boto:Headers: {} 2023-05-31T20:00:49.919 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:00:49.920 DEBUG:boto:Port: 8000 2023-05-31T20:00:49.920 DEBUG:boto:Params: {} 2023-05-31T20:00:49.920 DEBUG:boto:Token: None 2023-05-31T20:00:49.921 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:00:49 GMT /pklxkd-13/obj?versionId=Wr6wnEZSP3WRSGvZ8nB1.KinMc4wxvZ 2023-05-31T20:00:49.921 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:73aRBUO70WzcyBuxWh1CZ1E6vLs= 2023-05-31T20:00:49.922 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:00:49 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:73aRBUO70WzcyBuxWh1CZ1E6vLs=', 'Content-Length': '0'} 2023-05-31T20:00:49.923 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:00:07 GMT'), ('x-amz-version-id', 'Wr6wnEZSP3WRSGvZ8nB1.KinMc4wxvZ'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000012c8d6fcb411af67-006477a771-110e-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:00:49 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:00:49.924 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2023-05-31T20:00:49.924 DEBUG:boto:path=/pklxkd-13/obj 2023-05-31T20:00:49.925 DEBUG:boto:auth_path=/pklxkd-13/obj 2023-05-31T20:00:49.925 DEBUG:boto:path=/pklxkd-13/obj?versionId=Wr6wnEZSP3WRSGvZ8nB1.KinMc4wxvZ 2023-05-31T20:00:49.926 DEBUG:boto:auth_path=/pklxkd-13/obj?versionId=Wr6wnEZSP3WRSGvZ8nB1.KinMc4wxvZ 2023-05-31T20:00:49.926 DEBUG:boto:Method: GET 2023-05-31T20:00:49.926 DEBUG:boto:Path: /pklxkd-13/obj?versionId=Wr6wnEZSP3WRSGvZ8nB1.KinMc4wxvZ 2023-05-31T20:00:49.927 DEBUG:boto:Data: 2023-05-31T20:00:49.927 DEBUG:boto:Headers: {} 2023-05-31T20:00:49.927 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:00:49.928 DEBUG:boto:Port: 8001 2023-05-31T20:00:49.928 DEBUG:boto:Params: {} 2023-05-31T20:00:49.929 DEBUG:boto:Token: None 2023-05-31T20:00:49.929 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:00:49 GMT /pklxkd-13/obj?versionId=Wr6wnEZSP3WRSGvZ8nB1.KinMc4wxvZ 2023-05-31T20:00:49.929 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:WnETOPfSjulxCwDuMBzA1WBP14M= 2023-05-31T20:00:49.930 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:00:49 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:WnETOPfSjulxCwDuMBzA1WBP14M=', 'Content-Length': '0'} 2023-05-31T20:00:49.932 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:00:07 GMT'), ('x-amz-version-id', 'Wr6wnEZSP3WRSGvZ8nB1.KinMc4wxvZ'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000bce194ba0852fab2-006477a771-1112-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:00:49 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:00:49.933 DEBUG:boto:path=/pklxkd-13/obj 2023-05-31T20:00:49.933 DEBUG:boto:auth_path=/pklxkd-13/obj 2023-05-31T20:00:49.933 DEBUG:boto:path=/pklxkd-13/obj?versionId=Wr6wnEZSP3WRSGvZ8nB1.KinMc4wxvZ 2023-05-31T20:00:49.934 DEBUG:boto:auth_path=/pklxkd-13/obj?versionId=Wr6wnEZSP3WRSGvZ8nB1.KinMc4wxvZ 2023-05-31T20:00:49.934 DEBUG:boto:Method: GET 2023-05-31T20:00:49.934 DEBUG:boto:Path: /pklxkd-13/obj?versionId=Wr6wnEZSP3WRSGvZ8nB1.KinMc4wxvZ 2023-05-31T20:00:49.935 DEBUG:boto:Data: 2023-05-31T20:00:49.935 DEBUG:boto:Headers: {} 2023-05-31T20:00:49.936 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:00:49.936 DEBUG:boto:Port: 8000 2023-05-31T20:00:49.936 DEBUG:boto:Params: {} 2023-05-31T20:00:49.937 DEBUG:boto:Token: None 2023-05-31T20:00:49.937 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:00:49 GMT /pklxkd-13/obj?versionId=Wr6wnEZSP3WRSGvZ8nB1.KinMc4wxvZ 2023-05-31T20:00:49.938 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:WnETOPfSjulxCwDuMBzA1WBP14M= 2023-05-31T20:00:49.938 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:00:49 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:WnETOPfSjulxCwDuMBzA1WBP14M=', 'Content-Length': '0'} 2023-05-31T20:00:49.940 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:00:07 GMT'), ('x-amz-version-id', 'Wr6wnEZSP3WRSGvZ8nB1.KinMc4wxvZ'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000038b8b04f679c62f5-006477a771-110e-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:00:49 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:00:49.980 DEBUG:boto:path=/pklxkd-13/ 2023-05-31T20:00:49.981 DEBUG:boto:auth_path=/pklxkd-13/ 2023-05-31T20:00:49.981 DEBUG:boto:path=/pklxkd-13/?prefix=obj 2023-05-31T20:00:49.982 DEBUG:boto:auth_path=/pklxkd-13/?prefix=obj 2023-05-31T20:00:49.982 DEBUG:boto:Method: GET 2023-05-31T20:00:49.982 DEBUG:boto:Path: /pklxkd-13/?prefix=obj 2023-05-31T20:00:49.983 DEBUG:boto:Data: 2023-05-31T20:00:49.983 DEBUG:boto:Headers: {} 2023-05-31T20:00:49.983 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:00:49.984 DEBUG:boto:Port: 8001 2023-05-31T20:00:49.984 DEBUG:boto:Params: {} 2023-05-31T20:00:49.985 DEBUG:boto:Token: None 2023-05-31T20:00:49.985 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:00:49 GMT /pklxkd-13/ 2023-05-31T20:00:49.985 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:2Rhl0KA1WOxWz2tqNLYakr1DRVE= 2023-05-31T20:00:49.986 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:00:49 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:2Rhl0KA1WOxWz2tqNLYakr1DRVE=', 'Content-Length': '0'} 2023-05-31T20:00:49.989 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000857cddfbfbb2734f-006477a771-1112-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:00:49 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:00:49.989 DEBUG:boto:b'pklxkd-13obj1000falseobj2023-05-31T20:00:08.058Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:00:49.990 DEBUG:boto:path=/pklxkd-13/ 2023-05-31T20:00:49.990 DEBUG:boto:auth_path=/pklxkd-13/ 2023-05-31T20:00:49.991 DEBUG:boto:path=/pklxkd-13/?prefix=obj 2023-05-31T20:00:49.991 DEBUG:boto:auth_path=/pklxkd-13/?prefix=obj 2023-05-31T20:00:49.992 DEBUG:boto:Method: GET 2023-05-31T20:00:49.992 DEBUG:boto:Path: /pklxkd-13/?prefix=obj 2023-05-31T20:00:49.992 DEBUG:boto:Data: 2023-05-31T20:00:49.993 DEBUG:boto:Headers: {} 2023-05-31T20:00:49.993 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:00:49.994 DEBUG:boto:Port: 8000 2023-05-31T20:00:49.994 DEBUG:boto:Params: {} 2023-05-31T20:00:49.994 DEBUG:boto:Token: None 2023-05-31T20:00:49.995 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:00:49 GMT /pklxkd-13/ 2023-05-31T20:00:49.995 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:2Rhl0KA1WOxWz2tqNLYakr1DRVE= 2023-05-31T20:00:49.996 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:00:49 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:2Rhl0KA1WOxWz2tqNLYakr1DRVE=', 'Content-Length': '0'} 2023-05-31T20:00:49.998 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000905926a3d88d7703-006477a771-110e-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:00:49 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:00:49.999 DEBUG:boto:b'pklxkd-13obj1000falseobj2023-05-31T20:00:08.058Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:00:50.000 DEBUG:boto:path=/pklxkd-13/obj 2023-05-31T20:00:50.000 DEBUG:boto:auth_path=/pklxkd-13/obj 2023-05-31T20:00:50.001 DEBUG:boto:Method: HEAD 2023-05-31T20:00:50.001 DEBUG:boto:Path: /pklxkd-13/obj 2023-05-31T20:00:50.001 DEBUG:boto:Data: 2023-05-31T20:00:50.002 DEBUG:boto:Headers: {} 2023-05-31T20:00:50.002 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:00:50.002 DEBUG:boto:Port: 8001 2023-05-31T20:00:50.003 DEBUG:boto:Params: {} 2023-05-31T20:00:50.003 DEBUG:boto:Token: None 2023-05-31T20:00:50.003 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:00:50 GMT /pklxkd-13/obj 2023-05-31T20:00:50.004 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:ZtBcbTTttD967FiFrEBCJTED0dE= 2023-05-31T20:00:50.004 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:00:50 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:ZtBcbTTttD967FiFrEBCJTED0dE=', 'Content-Length': '0'} 2023-05-31T20:00:50.007 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:00:08 GMT'), ('x-amz-version-id', '.K0qHk3QiH4acREzLteeQLJmFzn4x-1'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000056ad15b6a02609c1-006477a772-1112-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:00:50 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:00:50.008 DEBUG:boto:path=/pklxkd-13/obj 2023-05-31T20:00:50.008 DEBUG:boto:auth_path=/pklxkd-13/obj 2023-05-31T20:00:50.009 DEBUG:boto:Method: HEAD 2023-05-31T20:00:50.009 DEBUG:boto:Path: /pklxkd-13/obj 2023-05-31T20:00:50.009 DEBUG:boto:Data: 2023-05-31T20:00:50.010 DEBUG:boto:Headers: {} 2023-05-31T20:00:50.010 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:00:50.010 DEBUG:boto:Port: 8000 2023-05-31T20:00:50.011 DEBUG:boto:Params: {} 2023-05-31T20:00:50.011 DEBUG:boto:Token: None 2023-05-31T20:00:50.012 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:00:50 GMT /pklxkd-13/obj 2023-05-31T20:00:50.012 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:ZtBcbTTttD967FiFrEBCJTED0dE= 2023-05-31T20:00:50.012 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:00:50 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:ZtBcbTTttD967FiFrEBCJTED0dE=', 'Content-Length': '0'} 2023-05-31T20:00:50.016 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:00:08 GMT'), ('x-amz-version-id', '.K0qHk3QiH4acREzLteeQLJmFzn4x-1'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000d1b259726156da2e-006477a772-110e-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:00:50 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:00:50.016 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2023-05-31T20:00:50.017 DEBUG:boto:path=/pklxkd-13/obj 2023-05-31T20:00:50.017 DEBUG:boto:auth_path=/pklxkd-13/obj 2023-05-31T20:00:50.017 DEBUG:boto:path=/pklxkd-13/obj?versionId=.K0qHk3QiH4acREzLteeQLJmFzn4x-1 2023-05-31T20:00:50.018 DEBUG:boto:auth_path=/pklxkd-13/obj?versionId=.K0qHk3QiH4acREzLteeQLJmFzn4x-1 2023-05-31T20:00:50.018 DEBUG:boto:Method: GET 2023-05-31T20:00:50.018 DEBUG:boto:Path: /pklxkd-13/obj?versionId=.K0qHk3QiH4acREzLteeQLJmFzn4x-1 2023-05-31T20:00:50.019 DEBUG:boto:Data: 2023-05-31T20:00:50.019 DEBUG:boto:Headers: {} 2023-05-31T20:00:50.020 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:00:50.020 DEBUG:boto:Port: 8001 2023-05-31T20:00:50.020 DEBUG:boto:Params: {} 2023-05-31T20:00:50.021 DEBUG:boto:Token: None 2023-05-31T20:00:50.021 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:00:50 GMT /pklxkd-13/obj?versionId=.K0qHk3QiH4acREzLteeQLJmFzn4x-1 2023-05-31T20:00:50.022 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:x2wvHW1glkO3VmBqamrmQxj1RSo= 2023-05-31T20:00:50.022 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:00:50 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:x2wvHW1glkO3VmBqamrmQxj1RSo=', 'Content-Length': '0'} 2023-05-31T20:00:50.024 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:00:08 GMT'), ('x-amz-version-id', '.K0qHk3QiH4acREzLteeQLJmFzn4x-1'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000af40c8dc4f85a631-006477a772-1112-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:00:50 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:00:50.025 DEBUG:boto:path=/pklxkd-13/obj 2023-05-31T20:00:50.025 DEBUG:boto:auth_path=/pklxkd-13/obj 2023-05-31T20:00:50.025 DEBUG:boto:path=/pklxkd-13/obj?versionId=.K0qHk3QiH4acREzLteeQLJmFzn4x-1 2023-05-31T20:00:50.026 DEBUG:boto:auth_path=/pklxkd-13/obj?versionId=.K0qHk3QiH4acREzLteeQLJmFzn4x-1 2023-05-31T20:00:50.026 DEBUG:boto:Method: GET 2023-05-31T20:00:50.026 DEBUG:boto:Path: /pklxkd-13/obj?versionId=.K0qHk3QiH4acREzLteeQLJmFzn4x-1 2023-05-31T20:00:50.027 DEBUG:boto:Data: 2023-05-31T20:00:50.027 DEBUG:boto:Headers: {} 2023-05-31T20:00:50.028 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:00:50.028 DEBUG:boto:Port: 8000 2023-05-31T20:00:50.028 DEBUG:boto:Params: {} 2023-05-31T20:00:50.029 DEBUG:boto:Token: None 2023-05-31T20:00:50.029 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:00:50 GMT /pklxkd-13/obj?versionId=.K0qHk3QiH4acREzLteeQLJmFzn4x-1 2023-05-31T20:00:50.030 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:x2wvHW1glkO3VmBqamrmQxj1RSo= 2023-05-31T20:00:50.030 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:00:50 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:x2wvHW1glkO3VmBqamrmQxj1RSo=', 'Content-Length': '0'} 2023-05-31T20:00:50.032 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:00:08 GMT'), ('x-amz-version-id', '.K0qHk3QiH4acREzLteeQLJmFzn4x-1'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000dbffb3a6f0dbd56c-006477a772-110e-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:00:50 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:00:50.033 DEBUG:boto:path=/pklxkd-13/ 2023-05-31T20:00:50.033 DEBUG:boto:auth_path=/pklxkd-13/ 2023-05-31T20:00:50.033 DEBUG:boto:path=/pklxkd-13/?prefix=obj 2023-05-31T20:00:50.034 DEBUG:boto:auth_path=/pklxkd-13/?prefix=obj 2023-05-31T20:00:50.034 DEBUG:boto:Method: GET 2023-05-31T20:00:50.034 DEBUG:boto:Path: /pklxkd-13/?prefix=obj 2023-05-31T20:00:50.035 DEBUG:boto:Data: 2023-05-31T20:00:50.035 DEBUG:boto:Headers: {} 2023-05-31T20:00:50.036 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:00:50.036 DEBUG:boto:Port: 8001 2023-05-31T20:00:50.036 DEBUG:boto:Params: {} 2023-05-31T20:00:50.037 DEBUG:boto:Token: None 2023-05-31T20:00:50.037 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:00:50 GMT /pklxkd-13/ 2023-05-31T20:00:50.038 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:25BcELGsmXWmRFB0zykFzQaOY+s= 2023-05-31T20:00:50.038 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:00:50 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:25BcELGsmXWmRFB0zykFzQaOY+s=', 'Content-Length': '0'} 2023-05-31T20:00:50.041 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000075ac5f88b8b8798c-006477a772-1112-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:00:50 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:00:50.041 DEBUG:boto:b'pklxkd-13obj1000falseobj2023-05-31T20:00:08.058Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:00:50.042 DEBUG:boto:path=/pklxkd-13/ 2023-05-31T20:00:50.042 DEBUG:boto:auth_path=/pklxkd-13/ 2023-05-31T20:00:50.043 DEBUG:boto:path=/pklxkd-13/?prefix=obj 2023-05-31T20:00:50.043 DEBUG:boto:auth_path=/pklxkd-13/?prefix=obj 2023-05-31T20:00:50.043 DEBUG:boto:Method: GET 2023-05-31T20:00:50.044 DEBUG:boto:Path: /pklxkd-13/?prefix=obj 2023-05-31T20:00:50.044 DEBUG:boto:Data: 2023-05-31T20:00:50.045 DEBUG:boto:Headers: {} 2023-05-31T20:00:50.045 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:00:50.045 DEBUG:boto:Port: 8000 2023-05-31T20:00:50.046 DEBUG:boto:Params: {} 2023-05-31T20:00:50.046 DEBUG:boto:Token: None 2023-05-31T20:00:50.046 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:00:50 GMT /pklxkd-13/ 2023-05-31T20:00:50.047 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:25BcELGsmXWmRFB0zykFzQaOY+s= 2023-05-31T20:00:50.047 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:00:50 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:25BcELGsmXWmRFB0zykFzQaOY+s=', 'Content-Length': '0'} 2023-05-31T20:00:50.050 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000520c90fda5d4fc88-006477a772-110e-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:00:50 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:00:50.050 DEBUG:boto:b'pklxkd-13obj1000falseobj2023-05-31T20:00:08.058Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:00:50.051 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2023-05-31T20:00:50.051 DEBUG:boto:path=/pklxkd-13/obj 2023-05-31T20:00:50.052 DEBUG:boto:auth_path=/pklxkd-13/obj 2023-05-31T20:00:50.052 DEBUG:boto:path=/pklxkd-13/obj?versionId=mWkpyutJAg628MwnVVNvQnYfQFw1fqJ 2023-05-31T20:00:50.052 DEBUG:boto:auth_path=/pklxkd-13/obj?versionId=mWkpyutJAg628MwnVVNvQnYfQFw1fqJ 2023-05-31T20:00:50.053 DEBUG:boto:Method: GET 2023-05-31T20:00:50.053 DEBUG:boto:Path: /pklxkd-13/obj?versionId=mWkpyutJAg628MwnVVNvQnYfQFw1fqJ 2023-05-31T20:00:50.053 DEBUG:boto:Data: 2023-05-31T20:00:50.054 DEBUG:boto:Headers: {} 2023-05-31T20:00:50.054 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:00:50.055 DEBUG:boto:Port: 8001 2023-05-31T20:00:50.055 DEBUG:boto:Params: {} 2023-05-31T20:00:50.055 DEBUG:boto:Token: None 2023-05-31T20:00:50.056 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:00:50 GMT /pklxkd-13/obj?versionId=mWkpyutJAg628MwnVVNvQnYfQFw1fqJ 2023-05-31T20:00:50.056 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:uEnm8DYZmdyKmTu0dEu4C4tST+c= 2023-05-31T20:00:50.057 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:00:50 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:uEnm8DYZmdyKmTu0dEu4C4tST+c=', 'Content-Length': '0'} 2023-05-31T20:00:50.059 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:00:07 GMT'), ('x-amz-version-id', 'mWkpyutJAg628MwnVVNvQnYfQFw1fqJ'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000ab57164e1799197c-006477a772-1112-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:00:50 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:00:50.104 DEBUG:boto:path=/pklxkd-13/obj 2023-05-31T20:00:50.105 DEBUG:boto:auth_path=/pklxkd-13/obj 2023-05-31T20:00:50.105 DEBUG:boto:path=/pklxkd-13/obj?versionId=mWkpyutJAg628MwnVVNvQnYfQFw1fqJ 2023-05-31T20:00:50.106 DEBUG:boto:auth_path=/pklxkd-13/obj?versionId=mWkpyutJAg628MwnVVNvQnYfQFw1fqJ 2023-05-31T20:00:50.106 DEBUG:boto:Method: GET 2023-05-31T20:00:50.106 DEBUG:boto:Path: /pklxkd-13/obj?versionId=mWkpyutJAg628MwnVVNvQnYfQFw1fqJ 2023-05-31T20:00:50.107 DEBUG:boto:Data: 2023-05-31T20:00:50.107 DEBUG:boto:Headers: {} 2023-05-31T20:00:50.107 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:00:50.108 DEBUG:boto:Port: 8000 2023-05-31T20:00:50.108 DEBUG:boto:Params: {} 2023-05-31T20:00:50.109 DEBUG:boto:Token: None 2023-05-31T20:00:50.109 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:00:50 GMT /pklxkd-13/obj?versionId=mWkpyutJAg628MwnVVNvQnYfQFw1fqJ 2023-05-31T20:00:50.110 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:uEnm8DYZmdyKmTu0dEu4C4tST+c= 2023-05-31T20:00:50.110 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:00:50 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:uEnm8DYZmdyKmTu0dEu4C4tST+c=', 'Content-Length': '0'} 2023-05-31T20:00:50.112 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:00:07 GMT'), ('x-amz-version-id', 'mWkpyutJAg628MwnVVNvQnYfQFw1fqJ'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000d85c44afc1d5594b-006477a772-110e-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:00:50 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:00:50.113 DEBUG:boto:path=/pklxkd-13/obj 2023-05-31T20:00:50.113 DEBUG:boto:auth_path=/pklxkd-13/obj 2023-05-31T20:00:50.114 DEBUG:boto:path=/pklxkd-13/obj?versionId=mWkpyutJAg628MwnVVNvQnYfQFw1fqJ 2023-05-31T20:00:50.114 DEBUG:boto:auth_path=/pklxkd-13/obj?versionId=mWkpyutJAg628MwnVVNvQnYfQFw1fqJ 2023-05-31T20:00:50.114 DEBUG:boto:Method: HEAD 2023-05-31T20:00:50.115 DEBUG:boto:Path: /pklxkd-13/obj?versionId=mWkpyutJAg628MwnVVNvQnYfQFw1fqJ 2023-05-31T20:00:50.115 DEBUG:boto:Data: 2023-05-31T20:00:50.115 DEBUG:boto:Headers: {} 2023-05-31T20:00:50.116 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:00:50.116 DEBUG:boto:Port: 8001 2023-05-31T20:00:50.116 DEBUG:boto:Params: {} 2023-05-31T20:00:50.117 DEBUG:boto:Token: None 2023-05-31T20:00:50.117 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:00:50 GMT /pklxkd-13/obj?versionId=mWkpyutJAg628MwnVVNvQnYfQFw1fqJ 2023-05-31T20:00:50.118 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:a403yj4brfL74REoUe+ACAW5bdc= 2023-05-31T20:00:50.118 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:00:50 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:a403yj4brfL74REoUe+ACAW5bdc=', 'Content-Length': '0'} 2023-05-31T20:00:50.120 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:00:07 GMT'), ('x-amz-version-id', 'mWkpyutJAg628MwnVVNvQnYfQFw1fqJ'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000b4987f03bd168997-006477a772-1112-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:00:50 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:00:50.120 DEBUG:boto:path=/pklxkd-13/obj 2023-05-31T20:00:50.121 DEBUG:boto:auth_path=/pklxkd-13/obj 2023-05-31T20:00:50.121 DEBUG:boto:path=/pklxkd-13/obj?versionId=mWkpyutJAg628MwnVVNvQnYfQFw1fqJ 2023-05-31T20:00:50.121 DEBUG:boto:auth_path=/pklxkd-13/obj?versionId=mWkpyutJAg628MwnVVNvQnYfQFw1fqJ 2023-05-31T20:00:50.122 DEBUG:boto:Method: HEAD 2023-05-31T20:00:50.122 DEBUG:boto:Path: /pklxkd-13/obj?versionId=mWkpyutJAg628MwnVVNvQnYfQFw1fqJ 2023-05-31T20:00:50.122 DEBUG:boto:Data: 2023-05-31T20:00:50.123 DEBUG:boto:Headers: {} 2023-05-31T20:00:50.123 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:00:50.123 DEBUG:boto:Port: 8000 2023-05-31T20:00:50.124 DEBUG:boto:Params: {} 2023-05-31T20:00:50.124 DEBUG:boto:Token: None 2023-05-31T20:00:50.125 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:00:50 GMT /pklxkd-13/obj?versionId=mWkpyutJAg628MwnVVNvQnYfQFw1fqJ 2023-05-31T20:00:50.125 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:a403yj4brfL74REoUe+ACAW5bdc= 2023-05-31T20:00:50.125 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:00:50 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:a403yj4brfL74REoUe+ACAW5bdc=', 'Content-Length': '0'} 2023-05-31T20:00:50.127 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:00:07 GMT'), ('x-amz-version-id', 'mWkpyutJAg628MwnVVNvQnYfQFw1fqJ'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000416044c9290806c1-006477a772-110e-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:00:50 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:00:50.128 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2023-05-31T20:00:50.128 DEBUG:boto:path=/pklxkd-13/obj 2023-05-31T20:00:50.129 DEBUG:boto:auth_path=/pklxkd-13/obj 2023-05-31T20:00:50.129 DEBUG:boto:path=/pklxkd-13/obj?versionId=mWkpyutJAg628MwnVVNvQnYfQFw1fqJ 2023-05-31T20:00:50.129 DEBUG:boto:auth_path=/pklxkd-13/obj?versionId=mWkpyutJAg628MwnVVNvQnYfQFw1fqJ 2023-05-31T20:00:50.130 DEBUG:boto:Method: GET 2023-05-31T20:00:50.130 DEBUG:boto:Path: /pklxkd-13/obj?versionId=mWkpyutJAg628MwnVVNvQnYfQFw1fqJ 2023-05-31T20:00:50.130 DEBUG:boto:Data: 2023-05-31T20:00:50.131 DEBUG:boto:Headers: {} 2023-05-31T20:00:50.131 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:00:50.132 DEBUG:boto:Port: 8001 2023-05-31T20:00:50.132 DEBUG:boto:Params: {} 2023-05-31T20:00:50.132 DEBUG:boto:Token: None 2023-05-31T20:00:50.133 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:00:50 GMT /pklxkd-13/obj?versionId=mWkpyutJAg628MwnVVNvQnYfQFw1fqJ 2023-05-31T20:00:50.133 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:uEnm8DYZmdyKmTu0dEu4C4tST+c= 2023-05-31T20:00:50.134 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:00:50 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:uEnm8DYZmdyKmTu0dEu4C4tST+c=', 'Content-Length': '0'} 2023-05-31T20:00:50.135 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:00:07 GMT'), ('x-amz-version-id', 'mWkpyutJAg628MwnVVNvQnYfQFw1fqJ'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000853efb183eb43127-006477a772-1112-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:00:50 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:00:50.136 DEBUG:boto:path=/pklxkd-13/obj 2023-05-31T20:00:50.137 DEBUG:boto:auth_path=/pklxkd-13/obj 2023-05-31T20:00:50.137 DEBUG:boto:path=/pklxkd-13/obj?versionId=mWkpyutJAg628MwnVVNvQnYfQFw1fqJ 2023-05-31T20:00:50.137 DEBUG:boto:auth_path=/pklxkd-13/obj?versionId=mWkpyutJAg628MwnVVNvQnYfQFw1fqJ 2023-05-31T20:00:50.138 DEBUG:boto:Method: GET 2023-05-31T20:00:50.138 DEBUG:boto:Path: /pklxkd-13/obj?versionId=mWkpyutJAg628MwnVVNvQnYfQFw1fqJ 2023-05-31T20:00:50.138 DEBUG:boto:Data: 2023-05-31T20:00:50.139 DEBUG:boto:Headers: {} 2023-05-31T20:00:50.139 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:00:50.140 DEBUG:boto:Port: 8000 2023-05-31T20:00:50.140 DEBUG:boto:Params: {} 2023-05-31T20:00:50.140 DEBUG:boto:Token: None 2023-05-31T20:00:50.141 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:00:50 GMT /pklxkd-13/obj?versionId=mWkpyutJAg628MwnVVNvQnYfQFw1fqJ 2023-05-31T20:00:50.141 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:uEnm8DYZmdyKmTu0dEu4C4tST+c= 2023-05-31T20:00:50.142 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:00:50 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:uEnm8DYZmdyKmTu0dEu4C4tST+c=', 'Content-Length': '0'} 2023-05-31T20:00:50.144 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:00:07 GMT'), ('x-amz-version-id', 'mWkpyutJAg628MwnVVNvQnYfQFw1fqJ'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000292cf321676919b7-006477a772-110e-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:00:50 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:00:50.144 DEBUG:boto:path=/pklxkd-13/ 2023-05-31T20:00:50.145 DEBUG:boto:auth_path=/pklxkd-13/ 2023-05-31T20:00:50.145 DEBUG:boto:path=/pklxkd-13/?prefix=obj 2023-05-31T20:00:50.145 DEBUG:boto:auth_path=/pklxkd-13/?prefix=obj 2023-05-31T20:00:50.146 DEBUG:boto:Method: GET 2023-05-31T20:00:50.146 DEBUG:boto:Path: /pklxkd-13/?prefix=obj 2023-05-31T20:00:50.147 DEBUG:boto:Data: 2023-05-31T20:00:50.147 DEBUG:boto:Headers: {} 2023-05-31T20:00:50.147 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:00:50.148 DEBUG:boto:Port: 8001 2023-05-31T20:00:50.148 DEBUG:boto:Params: {} 2023-05-31T20:00:50.148 DEBUG:boto:Token: None 2023-05-31T20:00:50.149 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:00:50 GMT /pklxkd-13/ 2023-05-31T20:00:50.149 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:25BcELGsmXWmRFB0zykFzQaOY+s= 2023-05-31T20:00:50.150 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:00:50 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:25BcELGsmXWmRFB0zykFzQaOY+s=', 'Content-Length': '0'} 2023-05-31T20:00:50.152 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a54f6f91ef10acf5-006477a772-1112-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:00:50 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:00:50.153 DEBUG:boto:b'pklxkd-13obj1000falseobj2023-05-31T20:00:08.058Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:00:50.153 DEBUG:boto:path=/pklxkd-13/ 2023-05-31T20:00:50.154 DEBUG:boto:auth_path=/pklxkd-13/ 2023-05-31T20:00:50.154 DEBUG:boto:path=/pklxkd-13/?prefix=obj 2023-05-31T20:00:50.155 DEBUG:boto:auth_path=/pklxkd-13/?prefix=obj 2023-05-31T20:00:50.155 DEBUG:boto:Method: GET 2023-05-31T20:00:50.155 DEBUG:boto:Path: /pklxkd-13/?prefix=obj 2023-05-31T20:00:50.156 DEBUG:boto:Data: 2023-05-31T20:00:50.156 DEBUG:boto:Headers: {} 2023-05-31T20:00:50.156 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:00:50.157 DEBUG:boto:Port: 8000 2023-05-31T20:00:50.157 DEBUG:boto:Params: {} 2023-05-31T20:00:50.158 DEBUG:boto:Token: None 2023-05-31T20:00:50.158 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:00:50 GMT /pklxkd-13/ 2023-05-31T20:00:50.158 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:25BcELGsmXWmRFB0zykFzQaOY+s= 2023-05-31T20:00:50.159 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:00:50 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:25BcELGsmXWmRFB0zykFzQaOY+s=', 'Content-Length': '0'} 2023-05-31T20:00:50.161 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000560f965e5a7531b0-006477a772-110e-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:00:50 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:00:50.162 DEBUG:boto:b'pklxkd-13obj1000falseobj2023-05-31T20:00:08.058Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:00:50.163 DEBUG:boto:path=/pklxkd-13/obj 2023-05-31T20:00:50.163 DEBUG:boto:auth_path=/pklxkd-13/obj 2023-05-31T20:00:50.163 DEBUG:boto:Method: HEAD 2023-05-31T20:00:50.164 DEBUG:boto:Path: /pklxkd-13/obj 2023-05-31T20:00:50.164 DEBUG:boto:Data: 2023-05-31T20:00:50.165 DEBUG:boto:Headers: {} 2023-05-31T20:00:50.165 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:00:50.165 DEBUG:boto:Port: 8001 2023-05-31T20:00:50.166 DEBUG:boto:Params: {} 2023-05-31T20:00:50.166 DEBUG:boto:Token: None 2023-05-31T20:00:50.166 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:00:50 GMT /pklxkd-13/obj 2023-05-31T20:00:50.167 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:ZtBcbTTttD967FiFrEBCJTED0dE= 2023-05-31T20:00:50.167 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:00:50 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:ZtBcbTTttD967FiFrEBCJTED0dE=', 'Content-Length': '0'} 2023-05-31T20:00:50.170 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:00:08 GMT'), ('x-amz-version-id', '.K0qHk3QiH4acREzLteeQLJmFzn4x-1'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000192b0cc3cdfa581f-006477a772-1112-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:00:50 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:00:50.171 DEBUG:boto:path=/pklxkd-13/obj 2023-05-31T20:00:50.171 DEBUG:boto:auth_path=/pklxkd-13/obj 2023-05-31T20:00:50.171 DEBUG:boto:Method: HEAD 2023-05-31T20:00:50.172 DEBUG:boto:Path: /pklxkd-13/obj 2023-05-31T20:00:50.172 DEBUG:boto:Data: 2023-05-31T20:00:50.173 DEBUG:boto:Headers: {} 2023-05-31T20:00:50.173 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:00:50.173 DEBUG:boto:Port: 8000 2023-05-31T20:00:50.174 DEBUG:boto:Params: {} 2023-05-31T20:00:50.174 DEBUG:boto:Token: None 2023-05-31T20:00:50.174 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:00:50 GMT /pklxkd-13/obj 2023-05-31T20:00:50.175 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:ZtBcbTTttD967FiFrEBCJTED0dE= 2023-05-31T20:00:50.175 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:00:50 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:ZtBcbTTttD967FiFrEBCJTED0dE=', 'Content-Length': '0'} 2023-05-31T20:00:50.178 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:00:08 GMT'), ('x-amz-version-id', '.K0qHk3QiH4acREzLteeQLJmFzn4x-1'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000ee5e4291a3537932-006477a772-110e-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:00:50 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:00:50.179 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2023-05-31T20:00:50.179 DEBUG:boto:path=/pklxkd-13/obj 2023-05-31T20:00:50.180 DEBUG:boto:auth_path=/pklxkd-13/obj 2023-05-31T20:00:50.180 DEBUG:boto:path=/pklxkd-13/obj?versionId=.K0qHk3QiH4acREzLteeQLJmFzn4x-1 2023-05-31T20:00:50.181 DEBUG:boto:auth_path=/pklxkd-13/obj?versionId=.K0qHk3QiH4acREzLteeQLJmFzn4x-1 2023-05-31T20:00:50.181 DEBUG:boto:Method: GET 2023-05-31T20:00:50.181 DEBUG:boto:Path: /pklxkd-13/obj?versionId=.K0qHk3QiH4acREzLteeQLJmFzn4x-1 2023-05-31T20:00:50.182 DEBUG:boto:Data: 2023-05-31T20:00:50.182 DEBUG:boto:Headers: {} 2023-05-31T20:00:50.182 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:00:50.183 DEBUG:boto:Port: 8001 2023-05-31T20:00:50.183 DEBUG:boto:Params: {} 2023-05-31T20:00:50.183 DEBUG:boto:Token: None 2023-05-31T20:00:50.184 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:00:50 GMT /pklxkd-13/obj?versionId=.K0qHk3QiH4acREzLteeQLJmFzn4x-1 2023-05-31T20:00:50.184 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:x2wvHW1glkO3VmBqamrmQxj1RSo= 2023-05-31T20:00:50.185 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:00:50 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:x2wvHW1glkO3VmBqamrmQxj1RSo=', 'Content-Length': '0'} 2023-05-31T20:00:50.187 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:00:08 GMT'), ('x-amz-version-id', '.K0qHk3QiH4acREzLteeQLJmFzn4x-1'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000052e81324b8e2d355-006477a772-1112-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:00:50 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:00:50.232 DEBUG:boto:path=/pklxkd-13/obj 2023-05-31T20:00:50.233 DEBUG:boto:auth_path=/pklxkd-13/obj 2023-05-31T20:00:50.233 DEBUG:boto:path=/pklxkd-13/obj?versionId=.K0qHk3QiH4acREzLteeQLJmFzn4x-1 2023-05-31T20:00:50.234 DEBUG:boto:auth_path=/pklxkd-13/obj?versionId=.K0qHk3QiH4acREzLteeQLJmFzn4x-1 2023-05-31T20:00:50.234 DEBUG:boto:Method: GET 2023-05-31T20:00:50.234 DEBUG:boto:Path: /pklxkd-13/obj?versionId=.K0qHk3QiH4acREzLteeQLJmFzn4x-1 2023-05-31T20:00:50.235 DEBUG:boto:Data: 2023-05-31T20:00:50.235 DEBUG:boto:Headers: {} 2023-05-31T20:00:50.235 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:00:50.236 DEBUG:boto:Port: 8000 2023-05-31T20:00:50.236 DEBUG:boto:Params: {} 2023-05-31T20:00:50.237 DEBUG:boto:Token: None 2023-05-31T20:00:50.237 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:00:50 GMT /pklxkd-13/obj?versionId=.K0qHk3QiH4acREzLteeQLJmFzn4x-1 2023-05-31T20:00:50.237 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:x2wvHW1glkO3VmBqamrmQxj1RSo= 2023-05-31T20:00:50.238 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:00:50 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:x2wvHW1glkO3VmBqamrmQxj1RSo=', 'Content-Length': '0'} 2023-05-31T20:00:50.240 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:00:08 GMT'), ('x-amz-version-id', '.K0qHk3QiH4acREzLteeQLJmFzn4x-1'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000010415537a17e5add-006477a772-110e-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:00:50 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:00:50.240 DEBUG:boto:path=/pklxkd-13/ 2023-05-31T20:00:50.241 DEBUG:boto:auth_path=/pklxkd-13/ 2023-05-31T20:00:50.241 DEBUG:boto:path=/pklxkd-13/?prefix=obj 2023-05-31T20:00:50.242 DEBUG:boto:auth_path=/pklxkd-13/?prefix=obj 2023-05-31T20:00:50.242 DEBUG:boto:Method: GET 2023-05-31T20:00:50.242 DEBUG:boto:Path: /pklxkd-13/?prefix=obj 2023-05-31T20:00:50.243 DEBUG:boto:Data: 2023-05-31T20:00:50.243 DEBUG:boto:Headers: {} 2023-05-31T20:00:50.243 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:00:50.244 DEBUG:boto:Port: 8001 2023-05-31T20:00:50.244 DEBUG:boto:Params: {} 2023-05-31T20:00:50.244 DEBUG:boto:Token: None 2023-05-31T20:00:50.245 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:00:50 GMT /pklxkd-13/ 2023-05-31T20:00:50.245 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:25BcELGsmXWmRFB0zykFzQaOY+s= 2023-05-31T20:00:50.246 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:00:50 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:25BcELGsmXWmRFB0zykFzQaOY+s=', 'Content-Length': '0'} 2023-05-31T20:00:50.248 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000001941fa20101108ce-006477a772-1112-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:00:50 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:00:50.249 DEBUG:boto:b'pklxkd-13obj1000falseobj2023-05-31T20:00:08.058Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:00:50.250 DEBUG:boto:path=/pklxkd-13/ 2023-05-31T20:00:50.250 DEBUG:boto:auth_path=/pklxkd-13/ 2023-05-31T20:00:50.250 DEBUG:boto:path=/pklxkd-13/?prefix=obj 2023-05-31T20:00:50.251 DEBUG:boto:auth_path=/pklxkd-13/?prefix=obj 2023-05-31T20:00:50.251 DEBUG:boto:Method: GET 2023-05-31T20:00:50.251 DEBUG:boto:Path: /pklxkd-13/?prefix=obj 2023-05-31T20:00:50.252 DEBUG:boto:Data: 2023-05-31T20:00:50.252 DEBUG:boto:Headers: {} 2023-05-31T20:00:50.253 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:00:50.253 DEBUG:boto:Port: 8000 2023-05-31T20:00:50.253 DEBUG:boto:Params: {} 2023-05-31T20:00:50.254 DEBUG:boto:Token: None 2023-05-31T20:00:50.254 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:00:50 GMT /pklxkd-13/ 2023-05-31T20:00:50.254 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:25BcELGsmXWmRFB0zykFzQaOY+s= 2023-05-31T20:00:50.255 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:00:50 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:25BcELGsmXWmRFB0zykFzQaOY+s=', 'Content-Length': '0'} 2023-05-31T20:00:50.257 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000cfe7bb80adf25a63-006477a772-110e-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:00:50 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:00:50.258 DEBUG:boto:b'pklxkd-13obj1000falseobj2023-05-31T20:00:08.058Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:00:50.258 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2023-05-31T20:00:50.259 DEBUG:boto:path=/pklxkd-13/obj 2023-05-31T20:00:50.259 DEBUG:boto:auth_path=/pklxkd-13/obj 2023-05-31T20:00:50.259 DEBUG:boto:path=/pklxkd-13/obj?versionId=bBAtwbgprRto0Kkb2LYN6Gd-LmGQA5. 2023-05-31T20:00:50.260 DEBUG:boto:auth_path=/pklxkd-13/obj?versionId=bBAtwbgprRto0Kkb2LYN6Gd-LmGQA5. 2023-05-31T20:00:50.260 DEBUG:boto:Method: GET 2023-05-31T20:00:50.261 DEBUG:boto:Path: /pklxkd-13/obj?versionId=bBAtwbgprRto0Kkb2LYN6Gd-LmGQA5. 2023-05-31T20:00:50.261 DEBUG:boto:Data: 2023-05-31T20:00:50.261 DEBUG:boto:Headers: {} 2023-05-31T20:00:50.262 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:00:50.262 DEBUG:boto:Port: 8001 2023-05-31T20:00:50.262 DEBUG:boto:Params: {} 2023-05-31T20:00:50.263 DEBUG:boto:Token: None 2023-05-31T20:00:50.263 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:00:50 GMT /pklxkd-13/obj?versionId=bBAtwbgprRto0Kkb2LYN6Gd-LmGQA5. 2023-05-31T20:00:50.264 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:vjHb+cdtcfF+SbKnVH/AIzS3k2c= 2023-05-31T20:00:50.264 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:00:50 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:vjHb+cdtcfF+SbKnVH/AIzS3k2c=', 'Content-Length': '0'} 2023-05-31T20:00:50.266 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:00:07 GMT'), ('x-amz-version-id', 'bBAtwbgprRto0Kkb2LYN6Gd-LmGQA5.'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000062b7e0aa211f9ca4-006477a772-1112-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:00:50 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:00:50.267 DEBUG:boto:path=/pklxkd-13/obj 2023-05-31T20:00:50.267 DEBUG:boto:auth_path=/pklxkd-13/obj 2023-05-31T20:00:50.267 DEBUG:boto:path=/pklxkd-13/obj?versionId=bBAtwbgprRto0Kkb2LYN6Gd-LmGQA5. 2023-05-31T20:00:50.268 DEBUG:boto:auth_path=/pklxkd-13/obj?versionId=bBAtwbgprRto0Kkb2LYN6Gd-LmGQA5. 2023-05-31T20:00:50.268 DEBUG:boto:Method: GET 2023-05-31T20:00:50.268 DEBUG:boto:Path: /pklxkd-13/obj?versionId=bBAtwbgprRto0Kkb2LYN6Gd-LmGQA5. 2023-05-31T20:00:50.269 DEBUG:boto:Data: 2023-05-31T20:00:50.269 DEBUG:boto:Headers: {} 2023-05-31T20:00:50.270 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:00:50.270 DEBUG:boto:Port: 8000 2023-05-31T20:00:50.270 DEBUG:boto:Params: {} 2023-05-31T20:00:50.271 DEBUG:boto:Token: None 2023-05-31T20:00:50.271 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:00:50 GMT /pklxkd-13/obj?versionId=bBAtwbgprRto0Kkb2LYN6Gd-LmGQA5. 2023-05-31T20:00:50.271 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:vjHb+cdtcfF+SbKnVH/AIzS3k2c= 2023-05-31T20:00:50.272 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:00:50 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:vjHb+cdtcfF+SbKnVH/AIzS3k2c=', 'Content-Length': '0'} 2023-05-31T20:00:50.274 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:00:07 GMT'), ('x-amz-version-id', 'bBAtwbgprRto0Kkb2LYN6Gd-LmGQA5.'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000cece42b9d3c7dc62-006477a772-110e-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:00:50 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:00:50.275 DEBUG:boto:path=/pklxkd-13/obj 2023-05-31T20:00:50.275 DEBUG:boto:auth_path=/pklxkd-13/obj 2023-05-31T20:00:50.275 DEBUG:boto:path=/pklxkd-13/obj?versionId=bBAtwbgprRto0Kkb2LYN6Gd-LmGQA5. 2023-05-31T20:00:50.276 DEBUG:boto:auth_path=/pklxkd-13/obj?versionId=bBAtwbgprRto0Kkb2LYN6Gd-LmGQA5. 2023-05-31T20:00:50.276 DEBUG:boto:Method: HEAD 2023-05-31T20:00:50.276 DEBUG:boto:Path: /pklxkd-13/obj?versionId=bBAtwbgprRto0Kkb2LYN6Gd-LmGQA5. 2023-05-31T20:00:50.277 DEBUG:boto:Data: 2023-05-31T20:00:50.277 DEBUG:boto:Headers: {} 2023-05-31T20:00:50.277 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:00:50.278 DEBUG:boto:Port: 8001 2023-05-31T20:00:50.278 DEBUG:boto:Params: {} 2023-05-31T20:00:50.278 DEBUG:boto:Token: None 2023-05-31T20:00:50.279 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:00:50 GMT /pklxkd-13/obj?versionId=bBAtwbgprRto0Kkb2LYN6Gd-LmGQA5. 2023-05-31T20:00:50.279 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:a0bugL1GJ6kca/9AHAm/lQZj0Us= 2023-05-31T20:00:50.280 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:00:50 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:a0bugL1GJ6kca/9AHAm/lQZj0Us=', 'Content-Length': '0'} 2023-05-31T20:00:50.281 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:00:07 GMT'), ('x-amz-version-id', 'bBAtwbgprRto0Kkb2LYN6Gd-LmGQA5.'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000567d945d433a1ad7-006477a772-1112-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:00:50 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:00:50.282 DEBUG:boto:path=/pklxkd-13/obj 2023-05-31T20:00:50.282 DEBUG:boto:auth_path=/pklxkd-13/obj 2023-05-31T20:00:50.283 DEBUG:boto:path=/pklxkd-13/obj?versionId=bBAtwbgprRto0Kkb2LYN6Gd-LmGQA5. 2023-05-31T20:00:50.283 DEBUG:boto:auth_path=/pklxkd-13/obj?versionId=bBAtwbgprRto0Kkb2LYN6Gd-LmGQA5. 2023-05-31T20:00:50.283 DEBUG:boto:Method: HEAD 2023-05-31T20:00:50.284 DEBUG:boto:Path: /pklxkd-13/obj?versionId=bBAtwbgprRto0Kkb2LYN6Gd-LmGQA5. 2023-05-31T20:00:50.284 DEBUG:boto:Data: 2023-05-31T20:00:50.284 DEBUG:boto:Headers: {} 2023-05-31T20:00:50.285 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:00:50.285 DEBUG:boto:Port: 8000 2023-05-31T20:00:50.285 DEBUG:boto:Params: {} 2023-05-31T20:00:50.286 DEBUG:boto:Token: None 2023-05-31T20:00:50.286 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:00:50 GMT /pklxkd-13/obj?versionId=bBAtwbgprRto0Kkb2LYN6Gd-LmGQA5. 2023-05-31T20:00:50.287 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:a0bugL1GJ6kca/9AHAm/lQZj0Us= 2023-05-31T20:00:50.287 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:00:50 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:a0bugL1GJ6kca/9AHAm/lQZj0Us=', 'Content-Length': '0'} 2023-05-31T20:00:50.289 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:00:07 GMT'), ('x-amz-version-id', 'bBAtwbgprRto0Kkb2LYN6Gd-LmGQA5.'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000030577ed33f319885-006477a772-110e-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:00:50 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:00:50.289 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2023-05-31T20:00:50.290 DEBUG:boto:path=/pklxkd-13/obj 2023-05-31T20:00:50.290 DEBUG:boto:auth_path=/pklxkd-13/obj 2023-05-31T20:00:50.291 DEBUG:boto:path=/pklxkd-13/obj?versionId=bBAtwbgprRto0Kkb2LYN6Gd-LmGQA5. 2023-05-31T20:00:50.291 DEBUG:boto:auth_path=/pklxkd-13/obj?versionId=bBAtwbgprRto0Kkb2LYN6Gd-LmGQA5. 2023-05-31T20:00:50.291 DEBUG:boto:Method: GET 2023-05-31T20:00:50.292 DEBUG:boto:Path: /pklxkd-13/obj?versionId=bBAtwbgprRto0Kkb2LYN6Gd-LmGQA5. 2023-05-31T20:00:50.292 DEBUG:boto:Data: 2023-05-31T20:00:50.292 DEBUG:boto:Headers: {} 2023-05-31T20:00:50.293 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:00:50.293 DEBUG:boto:Port: 8001 2023-05-31T20:00:50.293 DEBUG:boto:Params: {} 2023-05-31T20:00:50.294 DEBUG:boto:Token: None 2023-05-31T20:00:50.294 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:00:50 GMT /pklxkd-13/obj?versionId=bBAtwbgprRto0Kkb2LYN6Gd-LmGQA5. 2023-05-31T20:00:50.295 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:vjHb+cdtcfF+SbKnVH/AIzS3k2c= 2023-05-31T20:00:50.295 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:00:50 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:vjHb+cdtcfF+SbKnVH/AIzS3k2c=', 'Content-Length': '0'} 2023-05-31T20:00:50.297 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:00:07 GMT'), ('x-amz-version-id', 'bBAtwbgprRto0Kkb2LYN6Gd-LmGQA5.'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000ade9fec80306bd64-006477a772-1112-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:00:50 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:00:50.298 DEBUG:boto:path=/pklxkd-13/obj 2023-05-31T20:00:50.298 DEBUG:boto:auth_path=/pklxkd-13/obj 2023-05-31T20:00:50.298 DEBUG:boto:path=/pklxkd-13/obj?versionId=bBAtwbgprRto0Kkb2LYN6Gd-LmGQA5. 2023-05-31T20:00:50.299 DEBUG:boto:auth_path=/pklxkd-13/obj?versionId=bBAtwbgprRto0Kkb2LYN6Gd-LmGQA5. 2023-05-31T20:00:50.299 DEBUG:boto:Method: GET 2023-05-31T20:00:50.299 DEBUG:boto:Path: /pklxkd-13/obj?versionId=bBAtwbgprRto0Kkb2LYN6Gd-LmGQA5. 2023-05-31T20:00:50.300 DEBUG:boto:Data: 2023-05-31T20:00:50.300 DEBUG:boto:Headers: {} 2023-05-31T20:00:50.300 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:00:50.301 DEBUG:boto:Port: 8000 2023-05-31T20:00:50.301 DEBUG:boto:Params: {} 2023-05-31T20:00:50.302 DEBUG:boto:Token: None 2023-05-31T20:00:50.302 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:00:50 GMT /pklxkd-13/obj?versionId=bBAtwbgprRto0Kkb2LYN6Gd-LmGQA5. 2023-05-31T20:00:50.302 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:vjHb+cdtcfF+SbKnVH/AIzS3k2c= 2023-05-31T20:00:50.303 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:00:50 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:vjHb+cdtcfF+SbKnVH/AIzS3k2c=', 'Content-Length': '0'} 2023-05-31T20:00:50.304 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:00:07 GMT'), ('x-amz-version-id', 'bBAtwbgprRto0Kkb2LYN6Gd-LmGQA5.'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000006489b29f8636eff9-006477a772-110e-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:00:50 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:00:50.305 DEBUG:boto:path=/pklxkd-13/ 2023-05-31T20:00:50.306 DEBUG:boto:auth_path=/pklxkd-13/ 2023-05-31T20:00:50.306 DEBUG:boto:path=/pklxkd-13/?prefix=obj 2023-05-31T20:00:50.306 DEBUG:boto:auth_path=/pklxkd-13/?prefix=obj 2023-05-31T20:00:50.307 DEBUG:boto:Method: GET 2023-05-31T20:00:50.307 DEBUG:boto:Path: /pklxkd-13/?prefix=obj 2023-05-31T20:00:50.307 DEBUG:boto:Data: 2023-05-31T20:00:50.308 DEBUG:boto:Headers: {} 2023-05-31T20:00:50.308 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:00:50.308 DEBUG:boto:Port: 8001 2023-05-31T20:00:50.309 DEBUG:boto:Params: {} 2023-05-31T20:00:50.309 DEBUG:boto:Token: None 2023-05-31T20:00:50.310 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:00:50 GMT /pklxkd-13/ 2023-05-31T20:00:50.310 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:25BcELGsmXWmRFB0zykFzQaOY+s= 2023-05-31T20:00:50.310 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:00:50 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:25BcELGsmXWmRFB0zykFzQaOY+s=', 'Content-Length': '0'} 2023-05-31T20:00:50.313 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000033f6d22252e11e12-006477a772-1112-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:00:50 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:00:50.360 DEBUG:boto:b'pklxkd-13obj1000falseobj2023-05-31T20:00:08.058Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:00:50.361 DEBUG:boto:path=/pklxkd-13/ 2023-05-31T20:00:50.362 DEBUG:boto:auth_path=/pklxkd-13/ 2023-05-31T20:00:50.362 DEBUG:boto:path=/pklxkd-13/?prefix=obj 2023-05-31T20:00:50.362 DEBUG:boto:auth_path=/pklxkd-13/?prefix=obj 2023-05-31T20:00:50.363 DEBUG:boto:Method: GET 2023-05-31T20:00:50.363 DEBUG:boto:Path: /pklxkd-13/?prefix=obj 2023-05-31T20:00:50.363 DEBUG:boto:Data: 2023-05-31T20:00:50.364 DEBUG:boto:Headers: {} 2023-05-31T20:00:50.364 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:00:50.364 DEBUG:boto:Port: 8000 2023-05-31T20:00:50.365 DEBUG:boto:Params: {} 2023-05-31T20:00:50.365 DEBUG:boto:Token: None 2023-05-31T20:00:50.366 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:00:50 GMT /pklxkd-13/ 2023-05-31T20:00:50.366 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:25BcELGsmXWmRFB0zykFzQaOY+s= 2023-05-31T20:00:50.367 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:00:50 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:25BcELGsmXWmRFB0zykFzQaOY+s=', 'Content-Length': '0'} 2023-05-31T20:00:50.369 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000004b2b9fb50d57a05-006477a772-110e-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:00:50 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:00:50.370 DEBUG:boto:b'pklxkd-13obj1000falseobj2023-05-31T20:00:08.058Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:00:50.370 DEBUG:boto:path=/pklxkd-13/obj 2023-05-31T20:00:50.371 DEBUG:boto:auth_path=/pklxkd-13/obj 2023-05-31T20:00:50.371 DEBUG:boto:Method: HEAD 2023-05-31T20:00:50.372 DEBUG:boto:Path: /pklxkd-13/obj 2023-05-31T20:00:50.372 DEBUG:boto:Data: 2023-05-31T20:00:50.372 DEBUG:boto:Headers: {} 2023-05-31T20:00:50.373 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:00:50.373 DEBUG:boto:Port: 8001 2023-05-31T20:00:50.373 DEBUG:boto:Params: {} 2023-05-31T20:00:50.374 DEBUG:boto:Token: None 2023-05-31T20:00:50.374 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:00:50 GMT /pklxkd-13/obj 2023-05-31T20:00:50.374 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:ZtBcbTTttD967FiFrEBCJTED0dE= 2023-05-31T20:00:50.375 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:00:50 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:ZtBcbTTttD967FiFrEBCJTED0dE=', 'Content-Length': '0'} 2023-05-31T20:00:50.378 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:00:08 GMT'), ('x-amz-version-id', '.K0qHk3QiH4acREzLteeQLJmFzn4x-1'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000a51b157615d24ca7-006477a772-1112-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:00:50 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:00:50.378 DEBUG:boto:path=/pklxkd-13/obj 2023-05-31T20:00:50.379 DEBUG:boto:auth_path=/pklxkd-13/obj 2023-05-31T20:00:50.379 DEBUG:boto:Method: HEAD 2023-05-31T20:00:50.379 DEBUG:boto:Path: /pklxkd-13/obj 2023-05-31T20:00:50.380 DEBUG:boto:Data: 2023-05-31T20:00:50.380 DEBUG:boto:Headers: {} 2023-05-31T20:00:50.380 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:00:50.381 DEBUG:boto:Port: 8000 2023-05-31T20:00:50.381 DEBUG:boto:Params: {} 2023-05-31T20:00:50.382 DEBUG:boto:Token: None 2023-05-31T20:00:50.382 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:00:50 GMT /pklxkd-13/obj 2023-05-31T20:00:50.382 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:ZtBcbTTttD967FiFrEBCJTED0dE= 2023-05-31T20:00:50.383 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:00:50 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:ZtBcbTTttD967FiFrEBCJTED0dE=', 'Content-Length': '0'} 2023-05-31T20:00:50.386 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:00:08 GMT'), ('x-amz-version-id', '.K0qHk3QiH4acREzLteeQLJmFzn4x-1'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000007c465dd6ee5decdd-006477a772-110e-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:00:50 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:00:50.386 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2023-05-31T20:00:50.387 DEBUG:boto:path=/pklxkd-13/obj 2023-05-31T20:00:50.387 DEBUG:boto:auth_path=/pklxkd-13/obj 2023-05-31T20:00:50.388 DEBUG:boto:path=/pklxkd-13/obj?versionId=.K0qHk3QiH4acREzLteeQLJmFzn4x-1 2023-05-31T20:00:50.388 DEBUG:boto:auth_path=/pklxkd-13/obj?versionId=.K0qHk3QiH4acREzLteeQLJmFzn4x-1 2023-05-31T20:00:50.388 DEBUG:boto:Method: GET 2023-05-31T20:00:50.389 DEBUG:boto:Path: /pklxkd-13/obj?versionId=.K0qHk3QiH4acREzLteeQLJmFzn4x-1 2023-05-31T20:00:50.389 DEBUG:boto:Data: 2023-05-31T20:00:50.389 DEBUG:boto:Headers: {} 2023-05-31T20:00:50.390 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:00:50.390 DEBUG:boto:Port: 8001 2023-05-31T20:00:50.390 DEBUG:boto:Params: {} 2023-05-31T20:00:50.391 DEBUG:boto:Token: None 2023-05-31T20:00:50.391 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:00:50 GMT /pklxkd-13/obj?versionId=.K0qHk3QiH4acREzLteeQLJmFzn4x-1 2023-05-31T20:00:50.392 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:x2wvHW1glkO3VmBqamrmQxj1RSo= 2023-05-31T20:00:50.392 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:00:50 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:x2wvHW1glkO3VmBqamrmQxj1RSo=', 'Content-Length': '0'} 2023-05-31T20:00:50.394 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:00:08 GMT'), ('x-amz-version-id', '.K0qHk3QiH4acREzLteeQLJmFzn4x-1'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000be2f6490ba2c152f-006477a772-1112-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:00:50 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:00:50.395 DEBUG:boto:path=/pklxkd-13/obj 2023-05-31T20:00:50.395 DEBUG:boto:auth_path=/pklxkd-13/obj 2023-05-31T20:00:50.395 DEBUG:boto:path=/pklxkd-13/obj?versionId=.K0qHk3QiH4acREzLteeQLJmFzn4x-1 2023-05-31T20:00:50.396 DEBUG:boto:auth_path=/pklxkd-13/obj?versionId=.K0qHk3QiH4acREzLteeQLJmFzn4x-1 2023-05-31T20:00:50.396 DEBUG:boto:Method: GET 2023-05-31T20:00:50.396 DEBUG:boto:Path: /pklxkd-13/obj?versionId=.K0qHk3QiH4acREzLteeQLJmFzn4x-1 2023-05-31T20:00:50.397 DEBUG:boto:Data: 2023-05-31T20:00:50.397 DEBUG:boto:Headers: {} 2023-05-31T20:00:50.397 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:00:50.398 DEBUG:boto:Port: 8000 2023-05-31T20:00:50.398 DEBUG:boto:Params: {} 2023-05-31T20:00:50.399 DEBUG:boto:Token: None 2023-05-31T20:00:50.399 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:00:50 GMT /pklxkd-13/obj?versionId=.K0qHk3QiH4acREzLteeQLJmFzn4x-1 2023-05-31T20:00:50.399 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:x2wvHW1glkO3VmBqamrmQxj1RSo= 2023-05-31T20:00:50.400 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:00:50 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:x2wvHW1glkO3VmBqamrmQxj1RSo=', 'Content-Length': '0'} 2023-05-31T20:00:50.402 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:00:08 GMT'), ('x-amz-version-id', '.K0qHk3QiH4acREzLteeQLJmFzn4x-1'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000078475842c6cb8ffc-006477a772-110e-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:00:50 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:00:50.444 DEBUG:boto:path=/pklxkd-13/ 2023-05-31T20:00:50.445 DEBUG:boto:auth_path=/pklxkd-13/ 2023-05-31T20:00:50.445 DEBUG:boto:path=/pklxkd-13/?prefix=obj 2023-05-31T20:00:50.446 DEBUG:boto:auth_path=/pklxkd-13/?prefix=obj 2023-05-31T20:00:50.446 DEBUG:boto:Method: GET 2023-05-31T20:00:50.446 DEBUG:boto:Path: /pklxkd-13/?prefix=obj 2023-05-31T20:00:50.447 DEBUG:boto:Data: 2023-05-31T20:00:50.447 DEBUG:boto:Headers: {} 2023-05-31T20:00:50.447 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:00:50.448 DEBUG:boto:Port: 8001 2023-05-31T20:00:50.448 DEBUG:boto:Params: {} 2023-05-31T20:00:50.449 DEBUG:boto:Token: None 2023-05-31T20:00:50.449 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:00:50 GMT /pklxkd-13/ 2023-05-31T20:00:50.449 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:25BcELGsmXWmRFB0zykFzQaOY+s= 2023-05-31T20:00:50.450 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:00:50 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:25BcELGsmXWmRFB0zykFzQaOY+s=', 'Content-Length': '0'} 2023-05-31T20:00:50.453 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000128fc3a1eeaac784-006477a772-1112-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:00:50 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:00:50.453 DEBUG:boto:b'pklxkd-13obj1000falseobj2023-05-31T20:00:08.058Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:00:50.454 DEBUG:boto:path=/pklxkd-13/ 2023-05-31T20:00:50.454 DEBUG:boto:auth_path=/pklxkd-13/ 2023-05-31T20:00:50.455 DEBUG:boto:path=/pklxkd-13/?prefix=obj 2023-05-31T20:00:50.455 DEBUG:boto:auth_path=/pklxkd-13/?prefix=obj 2023-05-31T20:00:50.455 DEBUG:boto:Method: GET 2023-05-31T20:00:50.456 DEBUG:boto:Path: /pklxkd-13/?prefix=obj 2023-05-31T20:00:50.456 DEBUG:boto:Data: 2023-05-31T20:00:50.456 DEBUG:boto:Headers: {} 2023-05-31T20:00:50.457 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:00:50.457 DEBUG:boto:Port: 8000 2023-05-31T20:00:50.458 DEBUG:boto:Params: {} 2023-05-31T20:00:50.458 DEBUG:boto:Token: None 2023-05-31T20:00:50.458 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:00:50 GMT /pklxkd-13/ 2023-05-31T20:00:50.459 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:25BcELGsmXWmRFB0zykFzQaOY+s= 2023-05-31T20:00:50.459 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:00:50 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:25BcELGsmXWmRFB0zykFzQaOY+s=', 'Content-Length': '0'} 2023-05-31T20:00:50.462 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ac476e365a0e7d28-006477a772-110e-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:00:50 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:00:50.462 DEBUG:boto:b'pklxkd-13obj1000falseobj2023-05-31T20:00:08.058Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:00:50.463 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2023-05-31T20:00:50.463 DEBUG:boto:path=/pklxkd-13/obj 2023-05-31T20:00:50.464 DEBUG:boto:auth_path=/pklxkd-13/obj 2023-05-31T20:00:50.464 DEBUG:boto:path=/pklxkd-13/obj?versionId=nD.VY2SV.sxc3KR89HB3qbvIEu7ie0D 2023-05-31T20:00:50.465 DEBUG:boto:auth_path=/pklxkd-13/obj?versionId=nD.VY2SV.sxc3KR89HB3qbvIEu7ie0D 2023-05-31T20:00:50.465 DEBUG:boto:Method: GET 2023-05-31T20:00:50.465 DEBUG:boto:Path: /pklxkd-13/obj?versionId=nD.VY2SV.sxc3KR89HB3qbvIEu7ie0D 2023-05-31T20:00:50.466 DEBUG:boto:Data: 2023-05-31T20:00:50.466 DEBUG:boto:Headers: {} 2023-05-31T20:00:50.466 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:00:50.467 DEBUG:boto:Port: 8001 2023-05-31T20:00:50.467 DEBUG:boto:Params: {} 2023-05-31T20:00:50.467 DEBUG:boto:Token: None 2023-05-31T20:00:50.468 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:00:50 GMT /pklxkd-13/obj?versionId=nD.VY2SV.sxc3KR89HB3qbvIEu7ie0D 2023-05-31T20:00:50.468 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:8tMKm1mlc79JrU4r6gGWdIirCOI= 2023-05-31T20:00:50.469 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:00:50 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:8tMKm1mlc79JrU4r6gGWdIirCOI=', 'Content-Length': '0'} 2023-05-31T20:00:50.471 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:00:07 GMT'), ('x-amz-version-id', 'nD.VY2SV.sxc3KR89HB3qbvIEu7ie0D'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000009fbe73522212fcbf-006477a772-1112-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:00:50 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:00:50.516 DEBUG:boto:path=/pklxkd-13/obj 2023-05-31T20:00:50.517 DEBUG:boto:auth_path=/pklxkd-13/obj 2023-05-31T20:00:50.517 DEBUG:boto:path=/pklxkd-13/obj?versionId=nD.VY2SV.sxc3KR89HB3qbvIEu7ie0D 2023-05-31T20:00:50.517 DEBUG:boto:auth_path=/pklxkd-13/obj?versionId=nD.VY2SV.sxc3KR89HB3qbvIEu7ie0D 2023-05-31T20:00:50.518 DEBUG:boto:Method: GET 2023-05-31T20:00:50.518 DEBUG:boto:Path: /pklxkd-13/obj?versionId=nD.VY2SV.sxc3KR89HB3qbvIEu7ie0D 2023-05-31T20:00:50.519 DEBUG:boto:Data: 2023-05-31T20:00:50.519 DEBUG:boto:Headers: {} 2023-05-31T20:00:50.519 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:00:50.520 DEBUG:boto:Port: 8000 2023-05-31T20:00:50.520 DEBUG:boto:Params: {} 2023-05-31T20:00:50.521 DEBUG:boto:Token: None 2023-05-31T20:00:50.521 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:00:50 GMT /pklxkd-13/obj?versionId=nD.VY2SV.sxc3KR89HB3qbvIEu7ie0D 2023-05-31T20:00:50.521 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:8tMKm1mlc79JrU4r6gGWdIirCOI= 2023-05-31T20:00:50.522 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:00:50 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:8tMKm1mlc79JrU4r6gGWdIirCOI=', 'Content-Length': '0'} 2023-05-31T20:00:50.524 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:00:07 GMT'), ('x-amz-version-id', 'nD.VY2SV.sxc3KR89HB3qbvIEu7ie0D'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000bbe53cea9e72aada-006477a772-110e-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:00:50 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:00:50.525 DEBUG:boto:path=/pklxkd-13/obj 2023-05-31T20:00:50.525 DEBUG:boto:auth_path=/pklxkd-13/obj 2023-05-31T20:00:50.525 DEBUG:boto:path=/pklxkd-13/obj?versionId=nD.VY2SV.sxc3KR89HB3qbvIEu7ie0D 2023-05-31T20:00:50.526 DEBUG:boto:auth_path=/pklxkd-13/obj?versionId=nD.VY2SV.sxc3KR89HB3qbvIEu7ie0D 2023-05-31T20:00:50.526 DEBUG:boto:Method: HEAD 2023-05-31T20:00:50.526 DEBUG:boto:Path: /pklxkd-13/obj?versionId=nD.VY2SV.sxc3KR89HB3qbvIEu7ie0D 2023-05-31T20:00:50.527 DEBUG:boto:Data: 2023-05-31T20:00:50.527 DEBUG:boto:Headers: {} 2023-05-31T20:00:50.527 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:00:50.528 DEBUG:boto:Port: 8001 2023-05-31T20:00:50.528 DEBUG:boto:Params: {} 2023-05-31T20:00:50.528 DEBUG:boto:Token: None 2023-05-31T20:00:50.529 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:00:50 GMT /pklxkd-13/obj?versionId=nD.VY2SV.sxc3KR89HB3qbvIEu7ie0D 2023-05-31T20:00:50.529 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:tgZ/HHd6saPQOLdtAn6SnCtBSSM= 2023-05-31T20:00:50.530 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:00:50 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:tgZ/HHd6saPQOLdtAn6SnCtBSSM=', 'Content-Length': '0'} 2023-05-31T20:00:50.532 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:00:07 GMT'), ('x-amz-version-id', 'nD.VY2SV.sxc3KR89HB3qbvIEu7ie0D'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000052c34b80dc1831c7-006477a772-1112-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:00:50 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:00:50.532 DEBUG:boto:path=/pklxkd-13/obj 2023-05-31T20:00:50.533 DEBUG:boto:auth_path=/pklxkd-13/obj 2023-05-31T20:00:50.533 DEBUG:boto:path=/pklxkd-13/obj?versionId=nD.VY2SV.sxc3KR89HB3qbvIEu7ie0D 2023-05-31T20:00:50.533 DEBUG:boto:auth_path=/pklxkd-13/obj?versionId=nD.VY2SV.sxc3KR89HB3qbvIEu7ie0D 2023-05-31T20:00:50.534 DEBUG:boto:Method: HEAD 2023-05-31T20:00:50.534 DEBUG:boto:Path: /pklxkd-13/obj?versionId=nD.VY2SV.sxc3KR89HB3qbvIEu7ie0D 2023-05-31T20:00:50.534 DEBUG:boto:Data: 2023-05-31T20:00:50.535 DEBUG:boto:Headers: {} 2023-05-31T20:00:50.535 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:00:50.535 DEBUG:boto:Port: 8000 2023-05-31T20:00:50.536 DEBUG:boto:Params: {} 2023-05-31T20:00:50.537 DEBUG:boto:Token: None 2023-05-31T20:00:50.537 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:00:50 GMT /pklxkd-13/obj?versionId=nD.VY2SV.sxc3KR89HB3qbvIEu7ie0D 2023-05-31T20:00:50.537 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:tgZ/HHd6saPQOLdtAn6SnCtBSSM= 2023-05-31T20:00:50.538 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:00:50 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:tgZ/HHd6saPQOLdtAn6SnCtBSSM=', 'Content-Length': '0'} 2023-05-31T20:00:50.540 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:00:07 GMT'), ('x-amz-version-id', 'nD.VY2SV.sxc3KR89HB3qbvIEu7ie0D'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000138b47246baa93bf-006477a772-110e-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:00:50 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:00:50.540 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2023-05-31T20:00:50.541 DEBUG:boto:path=/pklxkd-13/obj 2023-05-31T20:00:50.541 DEBUG:boto:auth_path=/pklxkd-13/obj 2023-05-31T20:00:50.541 DEBUG:boto:path=/pklxkd-13/obj?versionId=nD.VY2SV.sxc3KR89HB3qbvIEu7ie0D 2023-05-31T20:00:50.542 DEBUG:boto:auth_path=/pklxkd-13/obj?versionId=nD.VY2SV.sxc3KR89HB3qbvIEu7ie0D 2023-05-31T20:00:50.542 DEBUG:boto:Method: GET 2023-05-31T20:00:50.542 DEBUG:boto:Path: /pklxkd-13/obj?versionId=nD.VY2SV.sxc3KR89HB3qbvIEu7ie0D 2023-05-31T20:00:50.543 DEBUG:boto:Data: 2023-05-31T20:00:50.543 DEBUG:boto:Headers: {} 2023-05-31T20:00:50.544 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:00:50.544 DEBUG:boto:Port: 8001 2023-05-31T20:00:50.544 DEBUG:boto:Params: {} 2023-05-31T20:00:50.545 DEBUG:boto:Token: None 2023-05-31T20:00:50.545 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:00:50 GMT /pklxkd-13/obj?versionId=nD.VY2SV.sxc3KR89HB3qbvIEu7ie0D 2023-05-31T20:00:50.546 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:8tMKm1mlc79JrU4r6gGWdIirCOI= 2023-05-31T20:00:50.546 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:00:50 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:8tMKm1mlc79JrU4r6gGWdIirCOI=', 'Content-Length': '0'} 2023-05-31T20:00:50.548 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:00:07 GMT'), ('x-amz-version-id', 'nD.VY2SV.sxc3KR89HB3qbvIEu7ie0D'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000002dbcf7afeb5f5b16-006477a772-1112-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:00:50 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:00:50.548 DEBUG:boto:path=/pklxkd-13/obj 2023-05-31T20:00:50.549 DEBUG:boto:auth_path=/pklxkd-13/obj 2023-05-31T20:00:50.549 DEBUG:boto:path=/pklxkd-13/obj?versionId=nD.VY2SV.sxc3KR89HB3qbvIEu7ie0D 2023-05-31T20:00:50.550 DEBUG:boto:auth_path=/pklxkd-13/obj?versionId=nD.VY2SV.sxc3KR89HB3qbvIEu7ie0D 2023-05-31T20:00:50.550 DEBUG:boto:Method: GET 2023-05-31T20:00:50.550 DEBUG:boto:Path: /pklxkd-13/obj?versionId=nD.VY2SV.sxc3KR89HB3qbvIEu7ie0D 2023-05-31T20:00:50.551 DEBUG:boto:Data: 2023-05-31T20:00:50.551 DEBUG:boto:Headers: {} 2023-05-31T20:00:50.551 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:00:50.552 DEBUG:boto:Port: 8000 2023-05-31T20:00:50.553 DEBUG:boto:Params: {} 2023-05-31T20:00:50.553 DEBUG:boto:Token: None 2023-05-31T20:00:50.553 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:00:50 GMT /pklxkd-13/obj?versionId=nD.VY2SV.sxc3KR89HB3qbvIEu7ie0D 2023-05-31T20:00:50.554 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:8tMKm1mlc79JrU4r6gGWdIirCOI= 2023-05-31T20:00:50.554 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:00:50 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:8tMKm1mlc79JrU4r6gGWdIirCOI=', 'Content-Length': '0'} 2023-05-31T20:00:50.556 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:00:07 GMT'), ('x-amz-version-id', 'nD.VY2SV.sxc3KR89HB3qbvIEu7ie0D'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000000da9a191c0b67838-006477a772-110e-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:00:50 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:00:50.557 DEBUG:boto:path=/pklxkd-13/ 2023-05-31T20:00:50.557 DEBUG:boto:auth_path=/pklxkd-13/ 2023-05-31T20:00:50.558 DEBUG:boto:path=/pklxkd-13/?prefix=obj 2023-05-31T20:00:50.558 DEBUG:boto:auth_path=/pklxkd-13/?prefix=obj 2023-05-31T20:00:50.558 DEBUG:boto:Method: GET 2023-05-31T20:00:50.559 DEBUG:boto:Path: /pklxkd-13/?prefix=obj 2023-05-31T20:00:50.559 DEBUG:boto:Data: 2023-05-31T20:00:50.559 DEBUG:boto:Headers: {} 2023-05-31T20:00:50.560 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:00:50.560 DEBUG:boto:Port: 8001 2023-05-31T20:00:50.560 DEBUG:boto:Params: {} 2023-05-31T20:00:50.561 DEBUG:boto:Token: None 2023-05-31T20:00:50.561 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:00:50 GMT /pklxkd-13/ 2023-05-31T20:00:50.562 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:25BcELGsmXWmRFB0zykFzQaOY+s= 2023-05-31T20:00:50.562 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:00:50 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:25BcELGsmXWmRFB0zykFzQaOY+s=', 'Content-Length': '0'} 2023-05-31T20:00:50.565 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000007667dd96e06c1458-006477a772-1112-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:00:50 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:00:50.565 DEBUG:boto:b'pklxkd-13obj1000falseobj2023-05-31T20:00:08.058Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:00:50.566 DEBUG:boto:path=/pklxkd-13/ 2023-05-31T20:00:50.566 DEBUG:boto:auth_path=/pklxkd-13/ 2023-05-31T20:00:50.567 DEBUG:boto:path=/pklxkd-13/?prefix=obj 2023-05-31T20:00:50.567 DEBUG:boto:auth_path=/pklxkd-13/?prefix=obj 2023-05-31T20:00:50.568 DEBUG:boto:Method: GET 2023-05-31T20:00:50.568 DEBUG:boto:Path: /pklxkd-13/?prefix=obj 2023-05-31T20:00:50.568 DEBUG:boto:Data: 2023-05-31T20:00:50.569 DEBUG:boto:Headers: {} 2023-05-31T20:00:50.569 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:00:50.569 DEBUG:boto:Port: 8000 2023-05-31T20:00:50.570 DEBUG:boto:Params: {} 2023-05-31T20:00:50.570 DEBUG:boto:Token: None 2023-05-31T20:00:50.570 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:00:50 GMT /pklxkd-13/ 2023-05-31T20:00:50.571 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:25BcELGsmXWmRFB0zykFzQaOY+s= 2023-05-31T20:00:50.571 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:00:50 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:25BcELGsmXWmRFB0zykFzQaOY+s=', 'Content-Length': '0'} 2023-05-31T20:00:50.574 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000004d3de39cff5efb8-006477a772-110e-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:00:50 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:00:50.575 DEBUG:boto:b'pklxkd-13obj1000falseobj2023-05-31T20:00:08.058Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:00:50.575 DEBUG:boto:path=/pklxkd-13/obj 2023-05-31T20:00:50.576 DEBUG:boto:auth_path=/pklxkd-13/obj 2023-05-31T20:00:50.576 DEBUG:boto:Method: HEAD 2023-05-31T20:00:50.576 DEBUG:boto:Path: /pklxkd-13/obj 2023-05-31T20:00:50.577 DEBUG:boto:Data: 2023-05-31T20:00:50.577 DEBUG:boto:Headers: {} 2023-05-31T20:00:50.577 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:00:50.578 DEBUG:boto:Port: 8001 2023-05-31T20:00:50.578 DEBUG:boto:Params: {} 2023-05-31T20:00:50.578 DEBUG:boto:Token: None 2023-05-31T20:00:50.579 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:00:50 GMT /pklxkd-13/obj 2023-05-31T20:00:50.579 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:ZtBcbTTttD967FiFrEBCJTED0dE= 2023-05-31T20:00:50.580 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:00:50 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:ZtBcbTTttD967FiFrEBCJTED0dE=', 'Content-Length': '0'} 2023-05-31T20:00:50.583 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:00:08 GMT'), ('x-amz-version-id', '.K0qHk3QiH4acREzLteeQLJmFzn4x-1'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000916e18a3e0716c1f-006477a772-1112-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:00:50 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:00:50.583 DEBUG:boto:path=/pklxkd-13/obj 2023-05-31T20:00:50.584 DEBUG:boto:auth_path=/pklxkd-13/obj 2023-05-31T20:00:50.584 DEBUG:boto:Method: HEAD 2023-05-31T20:00:50.584 DEBUG:boto:Path: /pklxkd-13/obj 2023-05-31T20:00:50.585 DEBUG:boto:Data: 2023-05-31T20:00:50.585 DEBUG:boto:Headers: {} 2023-05-31T20:00:50.585 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:00:50.586 DEBUG:boto:Port: 8000 2023-05-31T20:00:50.586 DEBUG:boto:Params: {} 2023-05-31T20:00:50.586 DEBUG:boto:Token: None 2023-05-31T20:00:50.587 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:00:50 GMT /pklxkd-13/obj 2023-05-31T20:00:50.587 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:ZtBcbTTttD967FiFrEBCJTED0dE= 2023-05-31T20:00:50.588 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:00:50 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:ZtBcbTTttD967FiFrEBCJTED0dE=', 'Content-Length': '0'} 2023-05-31T20:00:50.591 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:00:08 GMT'), ('x-amz-version-id', '.K0qHk3QiH4acREzLteeQLJmFzn4x-1'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000034d4868c064941f4-006477a772-110e-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:00:50 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:00:50.591 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2023-05-31T20:00:50.592 DEBUG:boto:path=/pklxkd-13/obj 2023-05-31T20:00:50.592 DEBUG:boto:auth_path=/pklxkd-13/obj 2023-05-31T20:00:50.593 DEBUG:boto:path=/pklxkd-13/obj?versionId=.K0qHk3QiH4acREzLteeQLJmFzn4x-1 2023-05-31T20:00:50.593 DEBUG:boto:auth_path=/pklxkd-13/obj?versionId=.K0qHk3QiH4acREzLteeQLJmFzn4x-1 2023-05-31T20:00:50.593 DEBUG:boto:Method: GET 2023-05-31T20:00:50.594 DEBUG:boto:Path: /pklxkd-13/obj?versionId=.K0qHk3QiH4acREzLteeQLJmFzn4x-1 2023-05-31T20:00:50.594 DEBUG:boto:Data: 2023-05-31T20:00:50.594 DEBUG:boto:Headers: {} 2023-05-31T20:00:50.595 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:00:50.595 DEBUG:boto:Port: 8001 2023-05-31T20:00:50.595 DEBUG:boto:Params: {} 2023-05-31T20:00:50.596 DEBUG:boto:Token: None 2023-05-31T20:00:50.596 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:00:50 GMT /pklxkd-13/obj?versionId=.K0qHk3QiH4acREzLteeQLJmFzn4x-1 2023-05-31T20:00:50.597 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:x2wvHW1glkO3VmBqamrmQxj1RSo= 2023-05-31T20:00:50.597 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:00:50 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:x2wvHW1glkO3VmBqamrmQxj1RSo=', 'Content-Length': '0'} 2023-05-31T20:00:50.599 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:00:08 GMT'), ('x-amz-version-id', '.K0qHk3QiH4acREzLteeQLJmFzn4x-1'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000cb5154c1e3e5271e-006477a772-1112-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:00:50 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:00:50.600 DEBUG:boto:path=/pklxkd-13/obj 2023-05-31T20:00:50.600 DEBUG:boto:auth_path=/pklxkd-13/obj 2023-05-31T20:00:50.601 DEBUG:boto:path=/pklxkd-13/obj?versionId=.K0qHk3QiH4acREzLteeQLJmFzn4x-1 2023-05-31T20:00:50.601 DEBUG:boto:auth_path=/pklxkd-13/obj?versionId=.K0qHk3QiH4acREzLteeQLJmFzn4x-1 2023-05-31T20:00:50.601 DEBUG:boto:Method: GET 2023-05-31T20:00:50.602 DEBUG:boto:Path: /pklxkd-13/obj?versionId=.K0qHk3QiH4acREzLteeQLJmFzn4x-1 2023-05-31T20:00:50.602 DEBUG:boto:Data: 2023-05-31T20:00:50.602 DEBUG:boto:Headers: {} 2023-05-31T20:00:50.603 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:00:50.603 DEBUG:boto:Port: 8000 2023-05-31T20:00:50.604 DEBUG:boto:Params: {} 2023-05-31T20:00:50.604 DEBUG:boto:Token: None 2023-05-31T20:00:50.604 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:00:50 GMT /pklxkd-13/obj?versionId=.K0qHk3QiH4acREzLteeQLJmFzn4x-1 2023-05-31T20:00:50.605 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:x2wvHW1glkO3VmBqamrmQxj1RSo= 2023-05-31T20:00:50.605 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:00:50 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:x2wvHW1glkO3VmBqamrmQxj1RSo=', 'Content-Length': '0'} 2023-05-31T20:00:50.607 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:00:08 GMT'), ('x-amz-version-id', '.K0qHk3QiH4acREzLteeQLJmFzn4x-1'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000007d6bf69080630029-006477a772-110e-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:00:50 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:00:50.649 DEBUG:boto:path=/pklxkd-13/ 2023-05-31T20:00:50.650 DEBUG:boto:auth_path=/pklxkd-13/ 2023-05-31T20:00:50.650 DEBUG:boto:path=/pklxkd-13/?prefix=obj 2023-05-31T20:00:50.651 DEBUG:boto:auth_path=/pklxkd-13/?prefix=obj 2023-05-31T20:00:50.651 DEBUG:boto:Method: GET 2023-05-31T20:00:50.651 DEBUG:boto:Path: /pklxkd-13/?prefix=obj 2023-05-31T20:00:50.652 DEBUG:boto:Data: 2023-05-31T20:00:50.652 DEBUG:boto:Headers: {} 2023-05-31T20:00:50.652 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:00:50.653 DEBUG:boto:Port: 8001 2023-05-31T20:00:50.653 DEBUG:boto:Params: {} 2023-05-31T20:00:50.654 DEBUG:boto:Token: None 2023-05-31T20:00:50.654 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:00:50 GMT /pklxkd-13/ 2023-05-31T20:00:50.654 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:25BcELGsmXWmRFB0zykFzQaOY+s= 2023-05-31T20:00:50.655 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:00:50 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:25BcELGsmXWmRFB0zykFzQaOY+s=', 'Content-Length': '0'} 2023-05-31T20:00:50.658 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000bdc0f4e8971cd62e-006477a772-1112-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:00:50 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:00:50.658 DEBUG:boto:b'pklxkd-13obj1000falseobj2023-05-31T20:00:08.058Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:00:50.659 DEBUG:boto:path=/pklxkd-13/ 2023-05-31T20:00:50.659 DEBUG:boto:auth_path=/pklxkd-13/ 2023-05-31T20:00:50.660 DEBUG:boto:path=/pklxkd-13/?prefix=obj 2023-05-31T20:00:50.660 DEBUG:boto:auth_path=/pklxkd-13/?prefix=obj 2023-05-31T20:00:50.661 DEBUG:boto:Method: GET 2023-05-31T20:00:50.661 DEBUG:boto:Path: /pklxkd-13/?prefix=obj 2023-05-31T20:00:50.661 DEBUG:boto:Data: 2023-05-31T20:00:50.662 DEBUG:boto:Headers: {} 2023-05-31T20:00:50.662 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:00:50.662 DEBUG:boto:Port: 8000 2023-05-31T20:00:50.663 DEBUG:boto:Params: {} 2023-05-31T20:00:50.663 DEBUG:boto:Token: None 2023-05-31T20:00:50.664 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:00:50 GMT /pklxkd-13/ 2023-05-31T20:00:50.664 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:25BcELGsmXWmRFB0zykFzQaOY+s= 2023-05-31T20:00:50.665 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:00:50 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:25BcELGsmXWmRFB0zykFzQaOY+s=', 'Content-Length': '0'} 2023-05-31T20:00:50.667 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b70d55a9bcb38a18-006477a772-110e-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:00:50 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:00:50.668 DEBUG:boto:b'pklxkd-13obj1000falseobj2023-05-31T20:00:08.058Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:00:50.668 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2023-05-31T20:00:50.669 DEBUG:boto:path=/pklxkd-13/obj 2023-05-31T20:00:50.669 DEBUG:boto:auth_path=/pklxkd-13/obj 2023-05-31T20:00:50.670 DEBUG:boto:path=/pklxkd-13/obj?versionId=BCM6vBdxKthZLIE0R8UhG0.omDRBvP7 2023-05-31T20:00:50.670 DEBUG:boto:auth_path=/pklxkd-13/obj?versionId=BCM6vBdxKthZLIE0R8UhG0.omDRBvP7 2023-05-31T20:00:50.670 DEBUG:boto:Method: GET 2023-05-31T20:00:50.671 DEBUG:boto:Path: /pklxkd-13/obj?versionId=BCM6vBdxKthZLIE0R8UhG0.omDRBvP7 2023-05-31T20:00:50.671 DEBUG:boto:Data: 2023-05-31T20:00:50.671 DEBUG:boto:Headers: {} 2023-05-31T20:00:50.672 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:00:50.673 DEBUG:boto:Port: 8001 2023-05-31T20:00:50.674 DEBUG:boto:Params: {} 2023-05-31T20:00:50.675 DEBUG:boto:Token: None 2023-05-31T20:00:50.676 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:00:50 GMT /pklxkd-13/obj?versionId=BCM6vBdxKthZLIE0R8UhG0.omDRBvP7 2023-05-31T20:00:50.676 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:FHPLpibn2wuBbj+vexUb/O4ddEQ= 2023-05-31T20:00:50.677 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:00:50 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:FHPLpibn2wuBbj+vexUb/O4ddEQ=', 'Content-Length': '0'} 2023-05-31T20:00:50.679 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:00:07 GMT'), ('x-amz-version-id', 'BCM6vBdxKthZLIE0R8UhG0.omDRBvP7'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000002d8ee4104a03104c-006477a772-1112-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:00:50 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:00:50.720 DEBUG:boto:path=/pklxkd-13/obj 2023-05-31T20:00:50.721 DEBUG:boto:auth_path=/pklxkd-13/obj 2023-05-31T20:00:50.721 DEBUG:boto:path=/pklxkd-13/obj?versionId=BCM6vBdxKthZLIE0R8UhG0.omDRBvP7 2023-05-31T20:00:50.722 DEBUG:boto:auth_path=/pklxkd-13/obj?versionId=BCM6vBdxKthZLIE0R8UhG0.omDRBvP7 2023-05-31T20:00:50.722 DEBUG:boto:Method: GET 2023-05-31T20:00:50.722 DEBUG:boto:Path: /pklxkd-13/obj?versionId=BCM6vBdxKthZLIE0R8UhG0.omDRBvP7 2023-05-31T20:00:50.723 DEBUG:boto:Data: 2023-05-31T20:00:50.723 DEBUG:boto:Headers: {} 2023-05-31T20:00:50.723 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:00:50.724 DEBUG:boto:Port: 8000 2023-05-31T20:00:50.724 DEBUG:boto:Params: {} 2023-05-31T20:00:50.725 DEBUG:boto:Token: None 2023-05-31T20:00:50.725 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:00:50 GMT /pklxkd-13/obj?versionId=BCM6vBdxKthZLIE0R8UhG0.omDRBvP7 2023-05-31T20:00:50.726 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:FHPLpibn2wuBbj+vexUb/O4ddEQ= 2023-05-31T20:00:50.726 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:00:50 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:FHPLpibn2wuBbj+vexUb/O4ddEQ=', 'Content-Length': '0'} 2023-05-31T20:00:50.728 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:00:07 GMT'), ('x-amz-version-id', 'BCM6vBdxKthZLIE0R8UhG0.omDRBvP7'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000007e50d4936026b6f2-006477a772-110e-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:00:50 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:00:50.729 DEBUG:boto:path=/pklxkd-13/obj 2023-05-31T20:00:50.729 DEBUG:boto:auth_path=/pklxkd-13/obj 2023-05-31T20:00:50.730 DEBUG:boto:path=/pklxkd-13/obj?versionId=BCM6vBdxKthZLIE0R8UhG0.omDRBvP7 2023-05-31T20:00:50.730 DEBUG:boto:auth_path=/pklxkd-13/obj?versionId=BCM6vBdxKthZLIE0R8UhG0.omDRBvP7 2023-05-31T20:00:50.730 DEBUG:boto:Method: HEAD 2023-05-31T20:00:50.731 DEBUG:boto:Path: /pklxkd-13/obj?versionId=BCM6vBdxKthZLIE0R8UhG0.omDRBvP7 2023-05-31T20:00:50.731 DEBUG:boto:Data: 2023-05-31T20:00:50.731 DEBUG:boto:Headers: {} 2023-05-31T20:00:50.732 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:00:50.732 DEBUG:boto:Port: 8001 2023-05-31T20:00:50.732 DEBUG:boto:Params: {} 2023-05-31T20:00:50.733 DEBUG:boto:Token: None 2023-05-31T20:00:50.733 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:00:50 GMT /pklxkd-13/obj?versionId=BCM6vBdxKthZLIE0R8UhG0.omDRBvP7 2023-05-31T20:00:50.734 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:ig/4O13WIosAsRpftmLDJ+zifHc= 2023-05-31T20:00:50.734 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:00:50 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:ig/4O13WIosAsRpftmLDJ+zifHc=', 'Content-Length': '0'} 2023-05-31T20:00:50.737 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:00:07 GMT'), ('x-amz-version-id', 'BCM6vBdxKthZLIE0R8UhG0.omDRBvP7'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000fa30d1cdcd7ce250-006477a772-1112-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:00:50 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:00:50.738 DEBUG:boto:path=/pklxkd-13/obj 2023-05-31T20:00:50.739 DEBUG:boto:auth_path=/pklxkd-13/obj 2023-05-31T20:00:50.740 DEBUG:boto:path=/pklxkd-13/obj?versionId=BCM6vBdxKthZLIE0R8UhG0.omDRBvP7 2023-05-31T20:00:50.740 DEBUG:boto:auth_path=/pklxkd-13/obj?versionId=BCM6vBdxKthZLIE0R8UhG0.omDRBvP7 2023-05-31T20:00:50.741 DEBUG:boto:Method: HEAD 2023-05-31T20:00:50.742 DEBUG:boto:Path: /pklxkd-13/obj?versionId=BCM6vBdxKthZLIE0R8UhG0.omDRBvP7 2023-05-31T20:00:50.742 DEBUG:boto:Data: 2023-05-31T20:00:50.743 DEBUG:boto:Headers: {} 2023-05-31T20:00:50.743 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:00:50.744 DEBUG:boto:Port: 8000 2023-05-31T20:00:50.745 DEBUG:boto:Params: {} 2023-05-31T20:00:50.745 DEBUG:boto:Token: None 2023-05-31T20:00:50.746 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:00:50 GMT /pklxkd-13/obj?versionId=BCM6vBdxKthZLIE0R8UhG0.omDRBvP7 2023-05-31T20:00:50.746 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:ig/4O13WIosAsRpftmLDJ+zifHc= 2023-05-31T20:00:50.746 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:00:50 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:ig/4O13WIosAsRpftmLDJ+zifHc=', 'Content-Length': '0'} 2023-05-31T20:00:50.748 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:00:07 GMT'), ('x-amz-version-id', 'BCM6vBdxKthZLIE0R8UhG0.omDRBvP7'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000e135ff413b3dc4ee-006477a772-110e-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:00:50 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:00:50.749 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2023-05-31T20:00:50.749 DEBUG:boto:path=/pklxkd-13/obj 2023-05-31T20:00:50.749 DEBUG:boto:auth_path=/pklxkd-13/obj 2023-05-31T20:00:50.750 DEBUG:boto:path=/pklxkd-13/obj?versionId=BCM6vBdxKthZLIE0R8UhG0.omDRBvP7 2023-05-31T20:00:50.750 DEBUG:boto:auth_path=/pklxkd-13/obj?versionId=BCM6vBdxKthZLIE0R8UhG0.omDRBvP7 2023-05-31T20:00:50.750 DEBUG:boto:Method: GET 2023-05-31T20:00:50.751 DEBUG:boto:Path: /pklxkd-13/obj?versionId=BCM6vBdxKthZLIE0R8UhG0.omDRBvP7 2023-05-31T20:00:50.751 DEBUG:boto:Data: 2023-05-31T20:00:50.752 DEBUG:boto:Headers: {} 2023-05-31T20:00:50.752 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:00:50.752 DEBUG:boto:Port: 8001 2023-05-31T20:00:50.753 DEBUG:boto:Params: {} 2023-05-31T20:00:50.753 DEBUG:boto:Token: None 2023-05-31T20:00:50.753 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:00:50 GMT /pklxkd-13/obj?versionId=BCM6vBdxKthZLIE0R8UhG0.omDRBvP7 2023-05-31T20:00:50.754 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:FHPLpibn2wuBbj+vexUb/O4ddEQ= 2023-05-31T20:00:50.754 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:00:50 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:FHPLpibn2wuBbj+vexUb/O4ddEQ=', 'Content-Length': '0'} 2023-05-31T20:00:50.756 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:00:07 GMT'), ('x-amz-version-id', 'BCM6vBdxKthZLIE0R8UhG0.omDRBvP7'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000c6f8bd204cb167d4-006477a772-1112-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:00:50 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:00:50.757 DEBUG:boto:path=/pklxkd-13/obj 2023-05-31T20:00:50.757 DEBUG:boto:auth_path=/pklxkd-13/obj 2023-05-31T20:00:50.758 DEBUG:boto:path=/pklxkd-13/obj?versionId=BCM6vBdxKthZLIE0R8UhG0.omDRBvP7 2023-05-31T20:00:50.758 DEBUG:boto:auth_path=/pklxkd-13/obj?versionId=BCM6vBdxKthZLIE0R8UhG0.omDRBvP7 2023-05-31T20:00:50.758 DEBUG:boto:Method: GET 2023-05-31T20:00:50.759 DEBUG:boto:Path: /pklxkd-13/obj?versionId=BCM6vBdxKthZLIE0R8UhG0.omDRBvP7 2023-05-31T20:00:50.759 DEBUG:boto:Data: 2023-05-31T20:00:50.759 DEBUG:boto:Headers: {} 2023-05-31T20:00:50.760 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:00:50.760 DEBUG:boto:Port: 8000 2023-05-31T20:00:50.761 DEBUG:boto:Params: {} 2023-05-31T20:00:50.761 DEBUG:boto:Token: None 2023-05-31T20:00:50.761 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:00:50 GMT /pklxkd-13/obj?versionId=BCM6vBdxKthZLIE0R8UhG0.omDRBvP7 2023-05-31T20:00:50.762 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:FHPLpibn2wuBbj+vexUb/O4ddEQ= 2023-05-31T20:00:50.763 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:00:50 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:FHPLpibn2wuBbj+vexUb/O4ddEQ=', 'Content-Length': '0'} 2023-05-31T20:00:50.764 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:00:07 GMT'), ('x-amz-version-id', 'BCM6vBdxKthZLIE0R8UhG0.omDRBvP7'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000001971db491e205b7f-006477a772-110e-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:00:50 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:00:50.765 DEBUG:boto:path=/pklxkd-13/ 2023-05-31T20:00:50.765 DEBUG:boto:auth_path=/pklxkd-13/ 2023-05-31T20:00:50.766 DEBUG:boto:path=/pklxkd-13/?prefix=obj 2023-05-31T20:00:50.766 DEBUG:boto:auth_path=/pklxkd-13/?prefix=obj 2023-05-31T20:00:50.767 DEBUG:boto:Method: GET 2023-05-31T20:00:50.767 DEBUG:boto:Path: /pklxkd-13/?prefix=obj 2023-05-31T20:00:50.767 DEBUG:boto:Data: 2023-05-31T20:00:50.768 DEBUG:boto:Headers: {} 2023-05-31T20:00:50.768 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:00:50.769 DEBUG:boto:Port: 8001 2023-05-31T20:00:50.769 DEBUG:boto:Params: {} 2023-05-31T20:00:50.769 DEBUG:boto:Token: None 2023-05-31T20:00:50.770 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:00:50 GMT /pklxkd-13/ 2023-05-31T20:00:50.770 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:25BcELGsmXWmRFB0zykFzQaOY+s= 2023-05-31T20:00:50.771 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:00:50 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:25BcELGsmXWmRFB0zykFzQaOY+s=', 'Content-Length': '0'} 2023-05-31T20:00:50.773 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a204866f261c5c76-006477a772-1112-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:00:50 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:00:50.816 DEBUG:boto:b'pklxkd-13obj1000falseobj2023-05-31T20:00:08.058Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:00:50.817 DEBUG:boto:path=/pklxkd-13/ 2023-05-31T20:00:50.817 DEBUG:boto:auth_path=/pklxkd-13/ 2023-05-31T20:00:50.818 DEBUG:boto:path=/pklxkd-13/?prefix=obj 2023-05-31T20:00:50.818 DEBUG:boto:auth_path=/pklxkd-13/?prefix=obj 2023-05-31T20:00:50.819 DEBUG:boto:Method: GET 2023-05-31T20:00:50.819 DEBUG:boto:Path: /pklxkd-13/?prefix=obj 2023-05-31T20:00:50.820 DEBUG:boto:Data: 2023-05-31T20:00:50.820 DEBUG:boto:Headers: {} 2023-05-31T20:00:50.820 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:00:50.821 DEBUG:boto:Port: 8000 2023-05-31T20:00:50.821 DEBUG:boto:Params: {} 2023-05-31T20:00:50.822 DEBUG:boto:Token: None 2023-05-31T20:00:50.822 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:00:50 GMT /pklxkd-13/ 2023-05-31T20:00:50.822 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:25BcELGsmXWmRFB0zykFzQaOY+s= 2023-05-31T20:00:50.823 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:00:50 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:25BcELGsmXWmRFB0zykFzQaOY+s=', 'Content-Length': '0'} 2023-05-31T20:00:50.826 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000009ea3c8c7ab18d014-006477a772-110e-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:00:50 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:00:50.827 DEBUG:boto:b'pklxkd-13obj1000falseobj2023-05-31T20:00:08.058Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:00:50.828 DEBUG:boto:path=/pklxkd-13/obj 2023-05-31T20:00:50.828 DEBUG:boto:auth_path=/pklxkd-13/obj 2023-05-31T20:00:50.829 DEBUG:boto:Method: HEAD 2023-05-31T20:00:50.829 DEBUG:boto:Path: /pklxkd-13/obj 2023-05-31T20:00:50.830 DEBUG:boto:Data: 2023-05-31T20:00:50.830 DEBUG:boto:Headers: {} 2023-05-31T20:00:50.830 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:00:50.831 DEBUG:boto:Port: 8001 2023-05-31T20:00:50.831 DEBUG:boto:Params: {} 2023-05-31T20:00:50.832 DEBUG:boto:Token: None 2023-05-31T20:00:50.832 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:00:50 GMT /pklxkd-13/obj 2023-05-31T20:00:50.832 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:ZtBcbTTttD967FiFrEBCJTED0dE= 2023-05-31T20:00:50.833 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:00:50 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:ZtBcbTTttD967FiFrEBCJTED0dE=', 'Content-Length': '0'} 2023-05-31T20:00:50.836 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:00:08 GMT'), ('x-amz-version-id', '.K0qHk3QiH4acREzLteeQLJmFzn4x-1'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000086c4b2d81c629037-006477a772-1112-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:00:50 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:00:50.837 DEBUG:boto:path=/pklxkd-13/obj 2023-05-31T20:00:50.837 DEBUG:boto:auth_path=/pklxkd-13/obj 2023-05-31T20:00:50.837 DEBUG:boto:Method: HEAD 2023-05-31T20:00:50.838 DEBUG:boto:Path: /pklxkd-13/obj 2023-05-31T20:00:50.838 DEBUG:boto:Data: 2023-05-31T20:00:50.838 DEBUG:boto:Headers: {} 2023-05-31T20:00:50.839 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:00:50.839 DEBUG:boto:Port: 8000 2023-05-31T20:00:50.839 DEBUG:boto:Params: {} 2023-05-31T20:00:50.840 DEBUG:boto:Token: None 2023-05-31T20:00:50.840 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:00:50 GMT /pklxkd-13/obj 2023-05-31T20:00:50.841 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:ZtBcbTTttD967FiFrEBCJTED0dE= 2023-05-31T20:00:50.841 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:00:50 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:ZtBcbTTttD967FiFrEBCJTED0dE=', 'Content-Length': '0'} 2023-05-31T20:00:50.844 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:00:08 GMT'), ('x-amz-version-id', '.K0qHk3QiH4acREzLteeQLJmFzn4x-1'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000009edf121392bde449-006477a772-110e-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:00:50 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:00:50.845 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2023-05-31T20:00:50.845 DEBUG:boto:path=/pklxkd-13/obj 2023-05-31T20:00:50.845 DEBUG:boto:auth_path=/pklxkd-13/obj 2023-05-31T20:00:50.846 DEBUG:boto:path=/pklxkd-13/obj?versionId=.K0qHk3QiH4acREzLteeQLJmFzn4x-1 2023-05-31T20:00:50.846 DEBUG:boto:auth_path=/pklxkd-13/obj?versionId=.K0qHk3QiH4acREzLteeQLJmFzn4x-1 2023-05-31T20:00:50.846 DEBUG:boto:Method: GET 2023-05-31T20:00:50.847 DEBUG:boto:Path: /pklxkd-13/obj?versionId=.K0qHk3QiH4acREzLteeQLJmFzn4x-1 2023-05-31T20:00:50.847 DEBUG:boto:Data: 2023-05-31T20:00:50.848 DEBUG:boto:Headers: {} 2023-05-31T20:00:50.848 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:00:50.848 DEBUG:boto:Port: 8001 2023-05-31T20:00:50.849 DEBUG:boto:Params: {} 2023-05-31T20:00:50.849 DEBUG:boto:Token: None 2023-05-31T20:00:50.850 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:00:50 GMT /pklxkd-13/obj?versionId=.K0qHk3QiH4acREzLteeQLJmFzn4x-1 2023-05-31T20:00:50.850 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:x2wvHW1glkO3VmBqamrmQxj1RSo= 2023-05-31T20:00:50.851 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:00:50 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:x2wvHW1glkO3VmBqamrmQxj1RSo=', 'Content-Length': '0'} 2023-05-31T20:00:50.853 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:00:08 GMT'), ('x-amz-version-id', '.K0qHk3QiH4acREzLteeQLJmFzn4x-1'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000008f26d320bf450745-006477a772-1112-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:00:50 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:00:50.855 DEBUG:boto:path=/pklxkd-13/obj 2023-05-31T20:00:50.855 DEBUG:boto:auth_path=/pklxkd-13/obj 2023-05-31T20:00:50.856 DEBUG:boto:path=/pklxkd-13/obj?versionId=.K0qHk3QiH4acREzLteeQLJmFzn4x-1 2023-05-31T20:00:50.857 DEBUG:boto:auth_path=/pklxkd-13/obj?versionId=.K0qHk3QiH4acREzLteeQLJmFzn4x-1 2023-05-31T20:00:50.858 DEBUG:boto:Method: GET 2023-05-31T20:00:50.858 DEBUG:boto:Path: /pklxkd-13/obj?versionId=.K0qHk3QiH4acREzLteeQLJmFzn4x-1 2023-05-31T20:00:50.859 DEBUG:boto:Data: 2023-05-31T20:00:50.860 DEBUG:boto:Headers: {} 2023-05-31T20:00:50.860 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:00:50.861 DEBUG:boto:Port: 8000 2023-05-31T20:00:50.861 DEBUG:boto:Params: {} 2023-05-31T20:00:50.862 DEBUG:boto:Token: None 2023-05-31T20:00:50.862 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:00:50 GMT /pklxkd-13/obj?versionId=.K0qHk3QiH4acREzLteeQLJmFzn4x-1 2023-05-31T20:00:50.863 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:x2wvHW1glkO3VmBqamrmQxj1RSo= 2023-05-31T20:00:50.863 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:00:50 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:x2wvHW1glkO3VmBqamrmQxj1RSo=', 'Content-Length': '0'} 2023-05-31T20:00:50.865 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:00:08 GMT'), ('x-amz-version-id', '.K0qHk3QiH4acREzLteeQLJmFzn4x-1'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000057d9de2361a7c178-006477a772-110e-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:00:50 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:00:50.908 DEBUG:boto:path=/pklxkd-13/ 2023-05-31T20:00:50.909 DEBUG:boto:auth_path=/pklxkd-13/ 2023-05-31T20:00:50.909 DEBUG:boto:path=/pklxkd-13/?prefix=obj 2023-05-31T20:00:50.910 DEBUG:boto:auth_path=/pklxkd-13/?prefix=obj 2023-05-31T20:00:50.910 DEBUG:boto:Method: GET 2023-05-31T20:00:50.911 DEBUG:boto:Path: /pklxkd-13/?prefix=obj 2023-05-31T20:00:50.911 DEBUG:boto:Data: 2023-05-31T20:00:50.911 DEBUG:boto:Headers: {} 2023-05-31T20:00:50.912 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:00:50.912 DEBUG:boto:Port: 8001 2023-05-31T20:00:50.913 DEBUG:boto:Params: {} 2023-05-31T20:00:50.913 DEBUG:boto:Token: None 2023-05-31T20:00:50.914 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:00:50 GMT /pklxkd-13/ 2023-05-31T20:00:50.914 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:25BcELGsmXWmRFB0zykFzQaOY+s= 2023-05-31T20:00:50.915 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:00:50 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:25BcELGsmXWmRFB0zykFzQaOY+s=', 'Content-Length': '0'} 2023-05-31T20:00:50.918 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a3f458b3ee0833bf-006477a772-1112-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:00:50 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:00:50.918 DEBUG:boto:b'pklxkd-13obj1000falseobj2023-05-31T20:00:08.058Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:00:50.919 DEBUG:boto:path=/pklxkd-13/ 2023-05-31T20:00:50.920 DEBUG:boto:auth_path=/pklxkd-13/ 2023-05-31T20:00:50.920 DEBUG:boto:path=/pklxkd-13/?prefix=obj 2023-05-31T20:00:50.920 DEBUG:boto:auth_path=/pklxkd-13/?prefix=obj 2023-05-31T20:00:50.921 DEBUG:boto:Method: GET 2023-05-31T20:00:50.921 DEBUG:boto:Path: /pklxkd-13/?prefix=obj 2023-05-31T20:00:50.922 DEBUG:boto:Data: 2023-05-31T20:00:50.922 DEBUG:boto:Headers: {} 2023-05-31T20:00:50.922 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:00:50.923 DEBUG:boto:Port: 8000 2023-05-31T20:00:50.923 DEBUG:boto:Params: {} 2023-05-31T20:00:50.924 DEBUG:boto:Token: None 2023-05-31T20:00:50.924 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:00:50 GMT /pklxkd-13/ 2023-05-31T20:00:50.925 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:25BcELGsmXWmRFB0zykFzQaOY+s= 2023-05-31T20:00:50.925 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:00:50 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:25BcELGsmXWmRFB0zykFzQaOY+s=', 'Content-Length': '0'} 2023-05-31T20:00:50.928 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000008fb09960e78d238-006477a772-110e-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:00:50 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:00:50.929 DEBUG:boto:b'pklxkd-13obj1000falseobj2023-05-31T20:00:08.058Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:00:50.929 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2023-05-31T20:00:50.930 DEBUG:boto:path=/pklxkd-13/obj 2023-05-31T20:00:50.930 DEBUG:boto:auth_path=/pklxkd-13/obj 2023-05-31T20:00:50.930 DEBUG:boto:path=/pklxkd-13/obj?versionId=RsWecIE0wyYuGR3yXB85CvZ9iY5NhBA 2023-05-31T20:00:50.931 DEBUG:boto:auth_path=/pklxkd-13/obj?versionId=RsWecIE0wyYuGR3yXB85CvZ9iY5NhBA 2023-05-31T20:00:50.931 DEBUG:boto:Method: GET 2023-05-31T20:00:50.932 DEBUG:boto:Path: /pklxkd-13/obj?versionId=RsWecIE0wyYuGR3yXB85CvZ9iY5NhBA 2023-05-31T20:00:50.932 DEBUG:boto:Data: 2023-05-31T20:00:50.932 DEBUG:boto:Headers: {} 2023-05-31T20:00:50.933 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:00:50.933 DEBUG:boto:Port: 8001 2023-05-31T20:00:50.934 DEBUG:boto:Params: {} 2023-05-31T20:00:50.935 DEBUG:boto:Token: None 2023-05-31T20:00:50.935 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:00:50 GMT /pklxkd-13/obj?versionId=RsWecIE0wyYuGR3yXB85CvZ9iY5NhBA 2023-05-31T20:00:50.936 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:R9eIB1AJsio4Q+UyDOA6EX75RT0= 2023-05-31T20:00:50.937 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:00:50 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:R9eIB1AJsio4Q+UyDOA6EX75RT0=', 'Content-Length': '0'} 2023-05-31T20:00:50.939 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:00:07 GMT'), ('x-amz-version-id', 'RsWecIE0wyYuGR3yXB85CvZ9iY5NhBA'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000016560954fba58a21-006477a772-1112-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:00:50 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:00:50.940 DEBUG:boto:path=/pklxkd-13/obj 2023-05-31T20:00:50.940 DEBUG:boto:auth_path=/pklxkd-13/obj 2023-05-31T20:00:50.941 DEBUG:boto:path=/pklxkd-13/obj?versionId=RsWecIE0wyYuGR3yXB85CvZ9iY5NhBA 2023-05-31T20:00:50.941 DEBUG:boto:auth_path=/pklxkd-13/obj?versionId=RsWecIE0wyYuGR3yXB85CvZ9iY5NhBA 2023-05-31T20:00:50.941 DEBUG:boto:Method: GET 2023-05-31T20:00:50.942 DEBUG:boto:Path: /pklxkd-13/obj?versionId=RsWecIE0wyYuGR3yXB85CvZ9iY5NhBA 2023-05-31T20:00:50.942 DEBUG:boto:Data: 2023-05-31T20:00:50.942 DEBUG:boto:Headers: {} 2023-05-31T20:00:50.943 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:00:50.943 DEBUG:boto:Port: 8000 2023-05-31T20:00:50.944 DEBUG:boto:Params: {} 2023-05-31T20:00:50.944 DEBUG:boto:Token: None 2023-05-31T20:00:50.944 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:00:50 GMT /pklxkd-13/obj?versionId=RsWecIE0wyYuGR3yXB85CvZ9iY5NhBA 2023-05-31T20:00:50.945 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:R9eIB1AJsio4Q+UyDOA6EX75RT0= 2023-05-31T20:00:50.945 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:00:50 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:R9eIB1AJsio4Q+UyDOA6EX75RT0=', 'Content-Length': '0'} 2023-05-31T20:00:50.947 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:00:07 GMT'), ('x-amz-version-id', 'RsWecIE0wyYuGR3yXB85CvZ9iY5NhBA'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000369a59b144847d1d-006477a772-110e-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:00:50 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:00:50.948 DEBUG:boto:path=/pklxkd-13/obj 2023-05-31T20:00:50.949 DEBUG:boto:auth_path=/pklxkd-13/obj 2023-05-31T20:00:50.950 DEBUG:boto:path=/pklxkd-13/obj?versionId=RsWecIE0wyYuGR3yXB85CvZ9iY5NhBA 2023-05-31T20:00:50.950 DEBUG:boto:auth_path=/pklxkd-13/obj?versionId=RsWecIE0wyYuGR3yXB85CvZ9iY5NhBA 2023-05-31T20:00:50.951 DEBUG:boto:Method: HEAD 2023-05-31T20:00:50.952 DEBUG:boto:Path: /pklxkd-13/obj?versionId=RsWecIE0wyYuGR3yXB85CvZ9iY5NhBA 2023-05-31T20:00:50.952 DEBUG:boto:Data: 2023-05-31T20:00:50.953 DEBUG:boto:Headers: {} 2023-05-31T20:00:50.953 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:00:50.954 DEBUG:boto:Port: 8001 2023-05-31T20:00:50.955 DEBUG:boto:Params: {} 2023-05-31T20:00:50.955 DEBUG:boto:Token: None 2023-05-31T20:00:50.956 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:00:50 GMT /pklxkd-13/obj?versionId=RsWecIE0wyYuGR3yXB85CvZ9iY5NhBA 2023-05-31T20:00:50.957 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:1XcOcCTLGCB4wGt1xuOmDNyL86M= 2023-05-31T20:00:50.958 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:00:50 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:1XcOcCTLGCB4wGt1xuOmDNyL86M=', 'Content-Length': '0'} 2023-05-31T20:00:50.960 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:00:07 GMT'), ('x-amz-version-id', 'RsWecIE0wyYuGR3yXB85CvZ9iY5NhBA'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000006174e0bcfacb59d7-006477a772-1112-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:00:50 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:00:50.961 DEBUG:boto:path=/pklxkd-13/obj 2023-05-31T20:00:50.962 DEBUG:boto:auth_path=/pklxkd-13/obj 2023-05-31T20:00:50.962 DEBUG:boto:path=/pklxkd-13/obj?versionId=RsWecIE0wyYuGR3yXB85CvZ9iY5NhBA 2023-05-31T20:00:50.963 DEBUG:boto:auth_path=/pklxkd-13/obj?versionId=RsWecIE0wyYuGR3yXB85CvZ9iY5NhBA 2023-05-31T20:00:50.964 DEBUG:boto:Method: HEAD 2023-05-31T20:00:50.965 DEBUG:boto:Path: /pklxkd-13/obj?versionId=RsWecIE0wyYuGR3yXB85CvZ9iY5NhBA 2023-05-31T20:00:50.965 DEBUG:boto:Data: 2023-05-31T20:00:50.966 DEBUG:boto:Headers: {} 2023-05-31T20:00:50.967 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:00:50.968 DEBUG:boto:Port: 8000 2023-05-31T20:00:50.969 DEBUG:boto:Params: {} 2023-05-31T20:00:50.969 DEBUG:boto:Token: None 2023-05-31T20:00:50.969 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:00:50 GMT /pklxkd-13/obj?versionId=RsWecIE0wyYuGR3yXB85CvZ9iY5NhBA 2023-05-31T20:00:50.970 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:1XcOcCTLGCB4wGt1xuOmDNyL86M= 2023-05-31T20:00:50.970 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:00:50 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:1XcOcCTLGCB4wGt1xuOmDNyL86M=', 'Content-Length': '0'} 2023-05-31T20:00:50.972 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:00:07 GMT'), ('x-amz-version-id', 'RsWecIE0wyYuGR3yXB85CvZ9iY5NhBA'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000dd7d38801a1f8a53-006477a772-110e-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:00:50 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:00:50.973 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2023-05-31T20:00:50.973 DEBUG:boto:path=/pklxkd-13/obj 2023-05-31T20:00:50.973 DEBUG:boto:auth_path=/pklxkd-13/obj 2023-05-31T20:00:50.974 DEBUG:boto:path=/pklxkd-13/obj?versionId=RsWecIE0wyYuGR3yXB85CvZ9iY5NhBA 2023-05-31T20:00:50.974 DEBUG:boto:auth_path=/pklxkd-13/obj?versionId=RsWecIE0wyYuGR3yXB85CvZ9iY5NhBA 2023-05-31T20:00:50.974 DEBUG:boto:Method: GET 2023-05-31T20:00:50.975 DEBUG:boto:Path: /pklxkd-13/obj?versionId=RsWecIE0wyYuGR3yXB85CvZ9iY5NhBA 2023-05-31T20:00:50.975 DEBUG:boto:Data: 2023-05-31T20:00:50.976 DEBUG:boto:Headers: {} 2023-05-31T20:00:50.976 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:00:50.976 DEBUG:boto:Port: 8001 2023-05-31T20:00:50.977 DEBUG:boto:Params: {} 2023-05-31T20:00:50.977 DEBUG:boto:Token: None 2023-05-31T20:00:50.978 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:00:50 GMT /pklxkd-13/obj?versionId=RsWecIE0wyYuGR3yXB85CvZ9iY5NhBA 2023-05-31T20:00:50.978 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:R9eIB1AJsio4Q+UyDOA6EX75RT0= 2023-05-31T20:00:50.979 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:00:50 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:R9eIB1AJsio4Q+UyDOA6EX75RT0=', 'Content-Length': '0'} 2023-05-31T20:00:50.981 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:00:07 GMT'), ('x-amz-version-id', 'RsWecIE0wyYuGR3yXB85CvZ9iY5NhBA'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000c0031a3797534ffc-006477a772-1112-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:00:50 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:00:50.981 DEBUG:boto:path=/pklxkd-13/obj 2023-05-31T20:00:50.982 DEBUG:boto:auth_path=/pklxkd-13/obj 2023-05-31T20:00:50.982 DEBUG:boto:path=/pklxkd-13/obj?versionId=RsWecIE0wyYuGR3yXB85CvZ9iY5NhBA 2023-05-31T20:00:50.982 DEBUG:boto:auth_path=/pklxkd-13/obj?versionId=RsWecIE0wyYuGR3yXB85CvZ9iY5NhBA 2023-05-31T20:00:50.983 DEBUG:boto:Method: GET 2023-05-31T20:00:50.983 DEBUG:boto:Path: /pklxkd-13/obj?versionId=RsWecIE0wyYuGR3yXB85CvZ9iY5NhBA 2023-05-31T20:00:50.984 DEBUG:boto:Data: 2023-05-31T20:00:50.984 DEBUG:boto:Headers: {} 2023-05-31T20:00:50.984 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:00:50.985 DEBUG:boto:Port: 8000 2023-05-31T20:00:50.985 DEBUG:boto:Params: {} 2023-05-31T20:00:50.986 DEBUG:boto:Token: None 2023-05-31T20:00:50.986 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:00:50 GMT /pklxkd-13/obj?versionId=RsWecIE0wyYuGR3yXB85CvZ9iY5NhBA 2023-05-31T20:00:50.986 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:R9eIB1AJsio4Q+UyDOA6EX75RT0= 2023-05-31T20:00:50.987 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:00:50 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:R9eIB1AJsio4Q+UyDOA6EX75RT0=', 'Content-Length': '0'} 2023-05-31T20:00:50.989 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:00:07 GMT'), ('x-amz-version-id', 'RsWecIE0wyYuGR3yXB85CvZ9iY5NhBA'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000709491d69194a739-006477a772-110e-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:00:50 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:00:50.989 DEBUG:boto:path=/pklxkd-13/ 2023-05-31T20:00:50.990 DEBUG:boto:auth_path=/pklxkd-13/ 2023-05-31T20:00:50.990 DEBUG:boto:path=/pklxkd-13/?prefix=obj 2023-05-31T20:00:50.990 DEBUG:boto:auth_path=/pklxkd-13/?prefix=obj 2023-05-31T20:00:50.991 DEBUG:boto:Method: GET 2023-05-31T20:00:50.991 DEBUG:boto:Path: /pklxkd-13/?prefix=obj 2023-05-31T20:00:50.992 DEBUG:boto:Data: 2023-05-31T20:00:50.992 DEBUG:boto:Headers: {} 2023-05-31T20:00:50.992 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:00:50.993 DEBUG:boto:Port: 8001 2023-05-31T20:00:50.993 DEBUG:boto:Params: {} 2023-05-31T20:00:50.993 DEBUG:boto:Token: None 2023-05-31T20:00:50.994 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:00:50 GMT /pklxkd-13/ 2023-05-31T20:00:50.995 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:25BcELGsmXWmRFB0zykFzQaOY+s= 2023-05-31T20:00:50.996 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:00:50 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:25BcELGsmXWmRFB0zykFzQaOY+s=', 'Content-Length': '0'} 2023-05-31T20:00:50.999 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000646878617bce5feb-006477a772-1112-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:00:50 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:00:51.040 DEBUG:boto:b'pklxkd-13obj1000falseobj2023-05-31T20:00:08.058Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:00:51.041 DEBUG:boto:path=/pklxkd-13/ 2023-05-31T20:00:51.042 DEBUG:boto:auth_path=/pklxkd-13/ 2023-05-31T20:00:51.042 DEBUG:boto:path=/pklxkd-13/?prefix=obj 2023-05-31T20:00:51.043 DEBUG:boto:auth_path=/pklxkd-13/?prefix=obj 2023-05-31T20:00:51.043 DEBUG:boto:Method: GET 2023-05-31T20:00:51.043 DEBUG:boto:Path: /pklxkd-13/?prefix=obj 2023-05-31T20:00:51.044 DEBUG:boto:Data: 2023-05-31T20:00:51.044 DEBUG:boto:Headers: {} 2023-05-31T20:00:51.045 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:00:51.045 DEBUG:boto:Port: 8000 2023-05-31T20:00:51.045 DEBUG:boto:Params: {} 2023-05-31T20:00:51.046 DEBUG:boto:Token: None 2023-05-31T20:00:51.046 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:00:51 GMT /pklxkd-13/ 2023-05-31T20:00:51.047 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:8nY7ypnD0gQg08MwmrHH+KcmiAo= 2023-05-31T20:00:51.047 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:00:51 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:8nY7ypnD0gQg08MwmrHH+KcmiAo=', 'Content-Length': '0'} 2023-05-31T20:00:51.050 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c20c2c4b52328635-006477a773-110e-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:00:51 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:00:51.050 DEBUG:boto:b'pklxkd-13obj1000falseobj2023-05-31T20:00:08.058Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:00:51.051 DEBUG:boto:path=/pklxkd-13/obj 2023-05-31T20:00:51.051 DEBUG:boto:auth_path=/pklxkd-13/obj 2023-05-31T20:00:51.052 DEBUG:boto:Method: HEAD 2023-05-31T20:00:51.052 DEBUG:boto:Path: /pklxkd-13/obj 2023-05-31T20:00:51.053 DEBUG:boto:Data: 2023-05-31T20:00:51.053 DEBUG:boto:Headers: {} 2023-05-31T20:00:51.053 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:00:51.054 DEBUG:boto:Port: 8001 2023-05-31T20:00:51.054 DEBUG:boto:Params: {} 2023-05-31T20:00:51.054 DEBUG:boto:Token: None 2023-05-31T20:00:51.055 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:00:51 GMT /pklxkd-13/obj 2023-05-31T20:00:51.055 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:sLcfXAyGTtS5A8hAUE3sGSDaL70= 2023-05-31T20:00:51.056 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:00:51 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:sLcfXAyGTtS5A8hAUE3sGSDaL70=', 'Content-Length': '0'} 2023-05-31T20:00:51.059 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:00:08 GMT'), ('x-amz-version-id', '.K0qHk3QiH4acREzLteeQLJmFzn4x-1'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000000332260b15f6e51b-006477a773-1112-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:00:51 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:00:51.060 DEBUG:boto:path=/pklxkd-13/obj 2023-05-31T20:00:51.061 DEBUG:boto:auth_path=/pklxkd-13/obj 2023-05-31T20:00:51.062 DEBUG:boto:Method: HEAD 2023-05-31T20:00:51.063 DEBUG:boto:Path: /pklxkd-13/obj 2023-05-31T20:00:51.064 DEBUG:boto:Data: 2023-05-31T20:00:51.064 DEBUG:boto:Headers: {} 2023-05-31T20:00:51.064 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:00:51.065 DEBUG:boto:Port: 8000 2023-05-31T20:00:51.065 DEBUG:boto:Params: {} 2023-05-31T20:00:51.065 DEBUG:boto:Token: None 2023-05-31T20:00:51.066 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:00:51 GMT /pklxkd-13/obj 2023-05-31T20:00:51.066 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:sLcfXAyGTtS5A8hAUE3sGSDaL70= 2023-05-31T20:00:51.067 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:00:51 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:sLcfXAyGTtS5A8hAUE3sGSDaL70=', 'Content-Length': '0'} 2023-05-31T20:00:51.070 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:00:08 GMT'), ('x-amz-version-id', '.K0qHk3QiH4acREzLteeQLJmFzn4x-1'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000f185a074b1bfc90c-006477a773-110e-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:00:51 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:00:51.070 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2023-05-31T20:00:51.071 DEBUG:boto:path=/pklxkd-13/obj 2023-05-31T20:00:51.071 DEBUG:boto:auth_path=/pklxkd-13/obj 2023-05-31T20:00:51.071 DEBUG:boto:path=/pklxkd-13/obj?versionId=.K0qHk3QiH4acREzLteeQLJmFzn4x-1 2023-05-31T20:00:51.072 DEBUG:boto:auth_path=/pklxkd-13/obj?versionId=.K0qHk3QiH4acREzLteeQLJmFzn4x-1 2023-05-31T20:00:51.072 DEBUG:boto:Method: GET 2023-05-31T20:00:51.073 DEBUG:boto:Path: /pklxkd-13/obj?versionId=.K0qHk3QiH4acREzLteeQLJmFzn4x-1 2023-05-31T20:00:51.073 DEBUG:boto:Data: 2023-05-31T20:00:51.073 DEBUG:boto:Headers: {} 2023-05-31T20:00:51.074 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:00:51.074 DEBUG:boto:Port: 8001 2023-05-31T20:00:51.074 DEBUG:boto:Params: {} 2023-05-31T20:00:51.075 DEBUG:boto:Token: None 2023-05-31T20:00:51.075 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:00:51 GMT /pklxkd-13/obj?versionId=.K0qHk3QiH4acREzLteeQLJmFzn4x-1 2023-05-31T20:00:51.076 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:MWWK358ZEpKN0sicgroa3DvTmS4= 2023-05-31T20:00:51.076 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:00:51 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:MWWK358ZEpKN0sicgroa3DvTmS4=', 'Content-Length': '0'} 2023-05-31T20:00:51.078 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:00:08 GMT'), ('x-amz-version-id', '.K0qHk3QiH4acREzLteeQLJmFzn4x-1'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000feef9483ca4451ca-006477a773-1112-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:00:51 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:00:51.079 DEBUG:boto:path=/pklxkd-13/obj 2023-05-31T20:00:51.079 DEBUG:boto:auth_path=/pklxkd-13/obj 2023-05-31T20:00:51.079 DEBUG:boto:path=/pklxkd-13/obj?versionId=.K0qHk3QiH4acREzLteeQLJmFzn4x-1 2023-05-31T20:00:51.080 DEBUG:boto:auth_path=/pklxkd-13/obj?versionId=.K0qHk3QiH4acREzLteeQLJmFzn4x-1 2023-05-31T20:00:51.080 DEBUG:boto:Method: GET 2023-05-31T20:00:51.081 DEBUG:boto:Path: /pklxkd-13/obj?versionId=.K0qHk3QiH4acREzLteeQLJmFzn4x-1 2023-05-31T20:00:51.081 DEBUG:boto:Data: 2023-05-31T20:00:51.081 DEBUG:boto:Headers: {} 2023-05-31T20:00:51.082 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:00:51.082 DEBUG:boto:Port: 8000 2023-05-31T20:00:51.082 DEBUG:boto:Params: {} 2023-05-31T20:00:51.083 DEBUG:boto:Token: None 2023-05-31T20:00:51.083 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:00:51 GMT /pklxkd-13/obj?versionId=.K0qHk3QiH4acREzLteeQLJmFzn4x-1 2023-05-31T20:00:51.084 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:MWWK358ZEpKN0sicgroa3DvTmS4= 2023-05-31T20:00:51.084 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:00:51 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:MWWK358ZEpKN0sicgroa3DvTmS4=', 'Content-Length': '0'} 2023-05-31T20:00:51.086 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:00:08 GMT'), ('x-amz-version-id', '.K0qHk3QiH4acREzLteeQLJmFzn4x-1'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000005360ca522183cdd2-006477a773-110e-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:00:51 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:00:51.128 DEBUG:boto:path=/pklxkd-13/ 2023-05-31T20:00:51.129 DEBUG:boto:auth_path=/pklxkd-13/ 2023-05-31T20:00:51.129 DEBUG:boto:path=/pklxkd-13/?prefix=obj 2023-05-31T20:00:51.130 DEBUG:boto:auth_path=/pklxkd-13/?prefix=obj 2023-05-31T20:00:51.130 DEBUG:boto:Method: GET 2023-05-31T20:00:51.130 DEBUG:boto:Path: /pklxkd-13/?prefix=obj 2023-05-31T20:00:51.131 DEBUG:boto:Data: 2023-05-31T20:00:51.131 DEBUG:boto:Headers: {} 2023-05-31T20:00:51.132 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:00:51.132 DEBUG:boto:Port: 8001 2023-05-31T20:00:51.133 DEBUG:boto:Params: {} 2023-05-31T20:00:51.135 DEBUG:boto:Token: None 2023-05-31T20:00:51.135 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:00:51 GMT /pklxkd-13/ 2023-05-31T20:00:51.136 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:8nY7ypnD0gQg08MwmrHH+KcmiAo= 2023-05-31T20:00:51.137 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:00:51 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:8nY7ypnD0gQg08MwmrHH+KcmiAo=', 'Content-Length': '0'} 2023-05-31T20:00:51.140 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000002b421e5ed6a81421-006477a773-1112-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:00:51 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:00:51.142 DEBUG:boto:b'pklxkd-13obj1000falseobj2023-05-31T20:00:08.058Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:00:51.143 DEBUG:boto:path=/pklxkd-13/ 2023-05-31T20:00:51.144 DEBUG:boto:auth_path=/pklxkd-13/ 2023-05-31T20:00:51.145 DEBUG:boto:path=/pklxkd-13/?prefix=obj 2023-05-31T20:00:51.145 DEBUG:boto:auth_path=/pklxkd-13/?prefix=obj 2023-05-31T20:00:51.146 DEBUG:boto:Method: GET 2023-05-31T20:00:51.146 DEBUG:boto:Path: /pklxkd-13/?prefix=obj 2023-05-31T20:00:51.146 DEBUG:boto:Data: 2023-05-31T20:00:51.147 DEBUG:boto:Headers: {} 2023-05-31T20:00:51.147 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:00:51.147 DEBUG:boto:Port: 8000 2023-05-31T20:00:51.148 DEBUG:boto:Params: {} 2023-05-31T20:00:51.148 DEBUG:boto:Token: None 2023-05-31T20:00:51.149 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:00:51 GMT /pklxkd-13/ 2023-05-31T20:00:51.149 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:8nY7ypnD0gQg08MwmrHH+KcmiAo= 2023-05-31T20:00:51.150 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:00:51 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:8nY7ypnD0gQg08MwmrHH+KcmiAo=', 'Content-Length': '0'} 2023-05-31T20:00:51.152 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000066f2e1280b6e3f67-006477a773-110e-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:00:51 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:00:51.153 DEBUG:boto:b'pklxkd-13obj1000falseobj2023-05-31T20:00:08.058Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:00:51.154 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2023-05-31T20:00:51.154 DEBUG:boto:path=/pklxkd-13/obj 2023-05-31T20:00:51.155 DEBUG:boto:auth_path=/pklxkd-13/obj 2023-05-31T20:00:51.155 DEBUG:boto:path=/pklxkd-13/obj?versionId=JR8T3JjcBmfQYfHtj3icgZUj81rYFms 2023-05-31T20:00:51.155 DEBUG:boto:auth_path=/pklxkd-13/obj?versionId=JR8T3JjcBmfQYfHtj3icgZUj81rYFms 2023-05-31T20:00:51.156 DEBUG:boto:Method: GET 2023-05-31T20:00:51.156 DEBUG:boto:Path: /pklxkd-13/obj?versionId=JR8T3JjcBmfQYfHtj3icgZUj81rYFms 2023-05-31T20:00:51.157 DEBUG:boto:Data: 2023-05-31T20:00:51.157 DEBUG:boto:Headers: {} 2023-05-31T20:00:51.157 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:00:51.158 DEBUG:boto:Port: 8001 2023-05-31T20:00:51.158 DEBUG:boto:Params: {} 2023-05-31T20:00:51.159 DEBUG:boto:Token: None 2023-05-31T20:00:51.159 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:00:51 GMT /pklxkd-13/obj?versionId=JR8T3JjcBmfQYfHtj3icgZUj81rYFms 2023-05-31T20:00:51.160 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:kRg4fH0Cu434Rap7287SP6X8UCY= 2023-05-31T20:00:51.160 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:00:51 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:kRg4fH0Cu434Rap7287SP6X8UCY=', 'Content-Length': '0'} 2023-05-31T20:00:51.163 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:00:07 GMT'), ('x-amz-version-id', 'JR8T3JjcBmfQYfHtj3icgZUj81rYFms'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000bab4e9b40be576bb-006477a773-1112-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:00:51 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:00:51.164 DEBUG:boto:path=/pklxkd-13/obj 2023-05-31T20:00:51.164 DEBUG:boto:auth_path=/pklxkd-13/obj 2023-05-31T20:00:51.165 DEBUG:boto:path=/pklxkd-13/obj?versionId=JR8T3JjcBmfQYfHtj3icgZUj81rYFms 2023-05-31T20:00:51.165 DEBUG:boto:auth_path=/pklxkd-13/obj?versionId=JR8T3JjcBmfQYfHtj3icgZUj81rYFms 2023-05-31T20:00:51.165 DEBUG:boto:Method: GET 2023-05-31T20:00:51.166 DEBUG:boto:Path: /pklxkd-13/obj?versionId=JR8T3JjcBmfQYfHtj3icgZUj81rYFms 2023-05-31T20:00:51.166 DEBUG:boto:Data: 2023-05-31T20:00:51.167 DEBUG:boto:Headers: {} 2023-05-31T20:00:51.167 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:00:51.167 DEBUG:boto:Port: 8000 2023-05-31T20:00:51.168 DEBUG:boto:Params: {} 2023-05-31T20:00:51.168 DEBUG:boto:Token: None 2023-05-31T20:00:51.169 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:00:51 GMT /pklxkd-13/obj?versionId=JR8T3JjcBmfQYfHtj3icgZUj81rYFms 2023-05-31T20:00:51.170 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:kRg4fH0Cu434Rap7287SP6X8UCY= 2023-05-31T20:00:51.170 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:00:51 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:kRg4fH0Cu434Rap7287SP6X8UCY=', 'Content-Length': '0'} 2023-05-31T20:00:51.172 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:00:07 GMT'), ('x-amz-version-id', 'JR8T3JjcBmfQYfHtj3icgZUj81rYFms'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000e84ab3135f3e7e8b-006477a773-110e-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:00:51 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:00:51.173 DEBUG:boto:path=/pklxkd-13/obj 2023-05-31T20:00:51.173 DEBUG:boto:auth_path=/pklxkd-13/obj 2023-05-31T20:00:51.174 DEBUG:boto:path=/pklxkd-13/obj?versionId=JR8T3JjcBmfQYfHtj3icgZUj81rYFms 2023-05-31T20:00:51.174 DEBUG:boto:auth_path=/pklxkd-13/obj?versionId=JR8T3JjcBmfQYfHtj3icgZUj81rYFms 2023-05-31T20:00:51.174 DEBUG:boto:Method: HEAD 2023-05-31T20:00:51.175 DEBUG:boto:Path: /pklxkd-13/obj?versionId=JR8T3JjcBmfQYfHtj3icgZUj81rYFms 2023-05-31T20:00:51.175 DEBUG:boto:Data: 2023-05-31T20:00:51.175 DEBUG:boto:Headers: {} 2023-05-31T20:00:51.176 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:00:51.176 DEBUG:boto:Port: 8001 2023-05-31T20:00:51.176 DEBUG:boto:Params: {} 2023-05-31T20:00:51.177 DEBUG:boto:Token: None 2023-05-31T20:00:51.177 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:00:51 GMT /pklxkd-13/obj?versionId=JR8T3JjcBmfQYfHtj3icgZUj81rYFms 2023-05-31T20:00:51.177 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:QQXbtJ3cS6XYWNOD9B522bZT5hw= 2023-05-31T20:00:51.178 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:00:51 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:QQXbtJ3cS6XYWNOD9B522bZT5hw=', 'Content-Length': '0'} 2023-05-31T20:00:51.180 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:00:07 GMT'), ('x-amz-version-id', 'JR8T3JjcBmfQYfHtj3icgZUj81rYFms'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000ce207ecab649a9f4-006477a773-1112-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:00:51 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:00:51.180 DEBUG:boto:path=/pklxkd-13/obj 2023-05-31T20:00:51.181 DEBUG:boto:auth_path=/pklxkd-13/obj 2023-05-31T20:00:51.181 DEBUG:boto:path=/pklxkd-13/obj?versionId=JR8T3JjcBmfQYfHtj3icgZUj81rYFms 2023-05-31T20:00:51.181 DEBUG:boto:auth_path=/pklxkd-13/obj?versionId=JR8T3JjcBmfQYfHtj3icgZUj81rYFms 2023-05-31T20:00:51.182 DEBUG:boto:Method: HEAD 2023-05-31T20:00:51.182 DEBUG:boto:Path: /pklxkd-13/obj?versionId=JR8T3JjcBmfQYfHtj3icgZUj81rYFms 2023-05-31T20:00:51.182 DEBUG:boto:Data: 2023-05-31T20:00:51.183 DEBUG:boto:Headers: {} 2023-05-31T20:00:51.183 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:00:51.183 DEBUG:boto:Port: 8000 2023-05-31T20:00:51.184 DEBUG:boto:Params: {} 2023-05-31T20:00:51.184 DEBUG:boto:Token: None 2023-05-31T20:00:51.184 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:00:51 GMT /pklxkd-13/obj?versionId=JR8T3JjcBmfQYfHtj3icgZUj81rYFms 2023-05-31T20:00:51.185 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:QQXbtJ3cS6XYWNOD9B522bZT5hw= 2023-05-31T20:00:51.185 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:00:51 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:QQXbtJ3cS6XYWNOD9B522bZT5hw=', 'Content-Length': '0'} 2023-05-31T20:00:51.187 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:00:07 GMT'), ('x-amz-version-id', 'JR8T3JjcBmfQYfHtj3icgZUj81rYFms'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000086de274caa55d77e-006477a773-110e-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:00:51 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:00:51.188 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2023-05-31T20:00:51.188 DEBUG:boto:path=/pklxkd-13/obj 2023-05-31T20:00:51.189 DEBUG:boto:auth_path=/pklxkd-13/obj 2023-05-31T20:00:51.189 DEBUG:boto:path=/pklxkd-13/obj?versionId=JR8T3JjcBmfQYfHtj3icgZUj81rYFms 2023-05-31T20:00:51.189 DEBUG:boto:auth_path=/pklxkd-13/obj?versionId=JR8T3JjcBmfQYfHtj3icgZUj81rYFms 2023-05-31T20:00:51.190 DEBUG:boto:Method: GET 2023-05-31T20:00:51.190 DEBUG:boto:Path: /pklxkd-13/obj?versionId=JR8T3JjcBmfQYfHtj3icgZUj81rYFms 2023-05-31T20:00:51.190 DEBUG:boto:Data: 2023-05-31T20:00:51.191 DEBUG:boto:Headers: {} 2023-05-31T20:00:51.191 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:00:51.191 DEBUG:boto:Port: 8001 2023-05-31T20:00:51.192 DEBUG:boto:Params: {} 2023-05-31T20:00:51.192 DEBUG:boto:Token: None 2023-05-31T20:00:51.193 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:00:51 GMT /pklxkd-13/obj?versionId=JR8T3JjcBmfQYfHtj3icgZUj81rYFms 2023-05-31T20:00:51.193 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:kRg4fH0Cu434Rap7287SP6X8UCY= 2023-05-31T20:00:51.193 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:00:51 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:kRg4fH0Cu434Rap7287SP6X8UCY=', 'Content-Length': '0'} 2023-05-31T20:00:51.195 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:00:07 GMT'), ('x-amz-version-id', 'JR8T3JjcBmfQYfHtj3icgZUj81rYFms'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000ac0ec8d04d0b39cc-006477a773-1112-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:00:51 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:00:51.196 DEBUG:boto:path=/pklxkd-13/obj 2023-05-31T20:00:51.197 DEBUG:boto:auth_path=/pklxkd-13/obj 2023-05-31T20:00:51.197 DEBUG:boto:path=/pklxkd-13/obj?versionId=JR8T3JjcBmfQYfHtj3icgZUj81rYFms 2023-05-31T20:00:51.197 DEBUG:boto:auth_path=/pklxkd-13/obj?versionId=JR8T3JjcBmfQYfHtj3icgZUj81rYFms 2023-05-31T20:00:51.198 DEBUG:boto:Method: GET 2023-05-31T20:00:51.198 DEBUG:boto:Path: /pklxkd-13/obj?versionId=JR8T3JjcBmfQYfHtj3icgZUj81rYFms 2023-05-31T20:00:51.198 DEBUG:boto:Data: 2023-05-31T20:00:51.199 DEBUG:boto:Headers: {} 2023-05-31T20:00:51.199 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:00:51.199 DEBUG:boto:Port: 8000 2023-05-31T20:00:51.200 DEBUG:boto:Params: {} 2023-05-31T20:00:51.200 DEBUG:boto:Token: None 2023-05-31T20:00:51.201 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:00:51 GMT /pklxkd-13/obj?versionId=JR8T3JjcBmfQYfHtj3icgZUj81rYFms 2023-05-31T20:00:51.201 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:kRg4fH0Cu434Rap7287SP6X8UCY= 2023-05-31T20:00:51.201 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:00:51 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:kRg4fH0Cu434Rap7287SP6X8UCY=', 'Content-Length': '0'} 2023-05-31T20:00:51.204 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:00:07 GMT'), ('x-amz-version-id', 'JR8T3JjcBmfQYfHtj3icgZUj81rYFms'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000007b1f64da27b0bd32-006477a773-110e-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:00:51 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:00:51.244 DEBUG:boto:path=/pklxkd-13/ 2023-05-31T20:00:51.245 DEBUG:boto:auth_path=/pklxkd-13/ 2023-05-31T20:00:51.245 DEBUG:boto:path=/pklxkd-13/?prefix=obj 2023-05-31T20:00:51.246 DEBUG:boto:auth_path=/pklxkd-13/?prefix=obj 2023-05-31T20:00:51.246 DEBUG:boto:Method: GET 2023-05-31T20:00:51.246 DEBUG:boto:Path: /pklxkd-13/?prefix=obj 2023-05-31T20:00:51.247 DEBUG:boto:Data: 2023-05-31T20:00:51.247 DEBUG:boto:Headers: {} 2023-05-31T20:00:51.247 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:00:51.248 DEBUG:boto:Port: 8001 2023-05-31T20:00:51.248 DEBUG:boto:Params: {} 2023-05-31T20:00:51.249 DEBUG:boto:Token: None 2023-05-31T20:00:51.249 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:00:51 GMT /pklxkd-13/ 2023-05-31T20:00:51.249 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:8nY7ypnD0gQg08MwmrHH+KcmiAo= 2023-05-31T20:00:51.250 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:00:51 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:8nY7ypnD0gQg08MwmrHH+KcmiAo=', 'Content-Length': '0'} 2023-05-31T20:00:51.252 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000405e0ced3bce4c20-006477a773-1112-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:00:51 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:00:51.253 DEBUG:boto:b'pklxkd-13obj1000falseobj2023-05-31T20:00:08.058Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:00:51.254 DEBUG:boto:path=/pklxkd-13/ 2023-05-31T20:00:51.254 DEBUG:boto:auth_path=/pklxkd-13/ 2023-05-31T20:00:51.254 DEBUG:boto:path=/pklxkd-13/?prefix=obj 2023-05-31T20:00:51.255 DEBUG:boto:auth_path=/pklxkd-13/?prefix=obj 2023-05-31T20:00:51.255 DEBUG:boto:Method: GET 2023-05-31T20:00:51.255 DEBUG:boto:Path: /pklxkd-13/?prefix=obj 2023-05-31T20:00:51.256 DEBUG:boto:Data: 2023-05-31T20:00:51.256 DEBUG:boto:Headers: {} 2023-05-31T20:00:51.256 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:00:51.257 DEBUG:boto:Port: 8000 2023-05-31T20:00:51.257 DEBUG:boto:Params: {} 2023-05-31T20:00:51.258 DEBUG:boto:Token: None 2023-05-31T20:00:51.258 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:00:51 GMT /pklxkd-13/ 2023-05-31T20:00:51.258 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:8nY7ypnD0gQg08MwmrHH+KcmiAo= 2023-05-31T20:00:51.259 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:00:51 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:8nY7ypnD0gQg08MwmrHH+KcmiAo=', 'Content-Length': '0'} 2023-05-31T20:00:51.261 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000008278a19ef19524af-006477a773-110e-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:00:51 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:00:51.262 DEBUG:boto:b'pklxkd-13obj1000falseobj2023-05-31T20:00:08.058Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:00:51.263 DEBUG:boto:path=/pklxkd-13/obj 2023-05-31T20:00:51.263 DEBUG:boto:auth_path=/pklxkd-13/obj 2023-05-31T20:00:51.263 DEBUG:boto:Method: HEAD 2023-05-31T20:00:51.264 DEBUG:boto:Path: /pklxkd-13/obj 2023-05-31T20:00:51.264 DEBUG:boto:Data: 2023-05-31T20:00:51.264 DEBUG:boto:Headers: {} 2023-05-31T20:00:51.265 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:00:51.265 DEBUG:boto:Port: 8001 2023-05-31T20:00:51.265 DEBUG:boto:Params: {} 2023-05-31T20:00:51.266 DEBUG:boto:Token: None 2023-05-31T20:00:51.266 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:00:51 GMT /pklxkd-13/obj 2023-05-31T20:00:51.267 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:sLcfXAyGTtS5A8hAUE3sGSDaL70= 2023-05-31T20:00:51.267 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:00:51 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:sLcfXAyGTtS5A8hAUE3sGSDaL70=', 'Content-Length': '0'} 2023-05-31T20:00:51.270 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:00:08 GMT'), ('x-amz-version-id', '.K0qHk3QiH4acREzLteeQLJmFzn4x-1'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000e31b59b4a4d306c3-006477a773-1112-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:00:51 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:00:51.270 DEBUG:boto:path=/pklxkd-13/obj 2023-05-31T20:00:51.271 DEBUG:boto:auth_path=/pklxkd-13/obj 2023-05-31T20:00:51.271 DEBUG:boto:Method: HEAD 2023-05-31T20:00:51.272 DEBUG:boto:Path: /pklxkd-13/obj 2023-05-31T20:00:51.272 DEBUG:boto:Data: 2023-05-31T20:00:51.272 DEBUG:boto:Headers: {} 2023-05-31T20:00:51.273 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:00:51.273 DEBUG:boto:Port: 8000 2023-05-31T20:00:51.273 DEBUG:boto:Params: {} 2023-05-31T20:00:51.274 DEBUG:boto:Token: None 2023-05-31T20:00:51.274 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:00:51 GMT /pklxkd-13/obj 2023-05-31T20:00:51.274 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:sLcfXAyGTtS5A8hAUE3sGSDaL70= 2023-05-31T20:00:51.275 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:00:51 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:sLcfXAyGTtS5A8hAUE3sGSDaL70=', 'Content-Length': '0'} 2023-05-31T20:00:51.278 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:00:08 GMT'), ('x-amz-version-id', '.K0qHk3QiH4acREzLteeQLJmFzn4x-1'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000005ee80872ab66b230-006477a773-110e-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:00:51 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:00:51.278 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2023-05-31T20:00:51.279 DEBUG:boto:path=/pklxkd-13/obj 2023-05-31T20:00:51.279 DEBUG:boto:auth_path=/pklxkd-13/obj 2023-05-31T20:00:51.279 DEBUG:boto:path=/pklxkd-13/obj?versionId=.K0qHk3QiH4acREzLteeQLJmFzn4x-1 2023-05-31T20:00:51.280 DEBUG:boto:auth_path=/pklxkd-13/obj?versionId=.K0qHk3QiH4acREzLteeQLJmFzn4x-1 2023-05-31T20:00:51.280 DEBUG:boto:Method: GET 2023-05-31T20:00:51.281 DEBUG:boto:Path: /pklxkd-13/obj?versionId=.K0qHk3QiH4acREzLteeQLJmFzn4x-1 2023-05-31T20:00:51.281 DEBUG:boto:Data: 2023-05-31T20:00:51.281 DEBUG:boto:Headers: {} 2023-05-31T20:00:51.282 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:00:51.282 DEBUG:boto:Port: 8001 2023-05-31T20:00:51.282 DEBUG:boto:Params: {} 2023-05-31T20:00:51.283 DEBUG:boto:Token: None 2023-05-31T20:00:51.284 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:00:51 GMT /pklxkd-13/obj?versionId=.K0qHk3QiH4acREzLteeQLJmFzn4x-1 2023-05-31T20:00:51.284 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:MWWK358ZEpKN0sicgroa3DvTmS4= 2023-05-31T20:00:51.284 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:00:51 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:MWWK358ZEpKN0sicgroa3DvTmS4=', 'Content-Length': '0'} 2023-05-31T20:00:51.286 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:00:08 GMT'), ('x-amz-version-id', '.K0qHk3QiH4acREzLteeQLJmFzn4x-1'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000bbee03069e5c42fb-006477a773-1112-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:00:51 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:00:51.328 DEBUG:boto:path=/pklxkd-13/obj 2023-05-31T20:00:51.329 DEBUG:boto:auth_path=/pklxkd-13/obj 2023-05-31T20:00:51.329 DEBUG:boto:path=/pklxkd-13/obj?versionId=.K0qHk3QiH4acREzLteeQLJmFzn4x-1 2023-05-31T20:00:51.330 DEBUG:boto:auth_path=/pklxkd-13/obj?versionId=.K0qHk3QiH4acREzLteeQLJmFzn4x-1 2023-05-31T20:00:51.330 DEBUG:boto:Method: GET 2023-05-31T20:00:51.330 DEBUG:boto:Path: /pklxkd-13/obj?versionId=.K0qHk3QiH4acREzLteeQLJmFzn4x-1 2023-05-31T20:00:51.331 DEBUG:boto:Data: 2023-05-31T20:00:51.331 DEBUG:boto:Headers: {} 2023-05-31T20:00:51.331 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:00:51.332 DEBUG:boto:Port: 8000 2023-05-31T20:00:51.332 DEBUG:boto:Params: {} 2023-05-31T20:00:51.333 DEBUG:boto:Token: None 2023-05-31T20:00:51.333 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:00:51 GMT /pklxkd-13/obj?versionId=.K0qHk3QiH4acREzLteeQLJmFzn4x-1 2023-05-31T20:00:51.333 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:MWWK358ZEpKN0sicgroa3DvTmS4= 2023-05-31T20:00:51.334 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:00:51 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:MWWK358ZEpKN0sicgroa3DvTmS4=', 'Content-Length': '0'} 2023-05-31T20:00:51.336 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:00:08 GMT'), ('x-amz-version-id', '.K0qHk3QiH4acREzLteeQLJmFzn4x-1'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000023844b9ef6f0e0b-006477a773-110e-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:00:51 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:00:51.337 DEBUG:boto:path=/pklxkd-13/ 2023-05-31T20:00:51.337 DEBUG:boto:auth_path=/pklxkd-13/ 2023-05-31T20:00:51.338 DEBUG:boto:path=/pklxkd-13/?prefix=obj 2023-05-31T20:00:51.338 DEBUG:boto:auth_path=/pklxkd-13/?prefix=obj 2023-05-31T20:00:51.339 DEBUG:boto:Method: GET 2023-05-31T20:00:51.339 DEBUG:boto:Path: /pklxkd-13/?prefix=obj 2023-05-31T20:00:51.339 DEBUG:boto:Data: 2023-05-31T20:00:51.340 DEBUG:boto:Headers: {} 2023-05-31T20:00:51.340 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:00:51.340 DEBUG:boto:Port: 8001 2023-05-31T20:00:51.341 DEBUG:boto:Params: {} 2023-05-31T20:00:51.341 DEBUG:boto:Token: None 2023-05-31T20:00:51.342 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:00:51 GMT /pklxkd-13/ 2023-05-31T20:00:51.342 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:8nY7ypnD0gQg08MwmrHH+KcmiAo= 2023-05-31T20:00:51.343 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:00:51 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:8nY7ypnD0gQg08MwmrHH+KcmiAo=', 'Content-Length': '0'} 2023-05-31T20:00:51.345 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000aa2e3045540ffeb1-006477a773-1112-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:00:51 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:00:51.346 DEBUG:boto:b'pklxkd-13obj1000falseobj2023-05-31T20:00:08.058Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:00:51.347 DEBUG:boto:path=/pklxkd-13/ 2023-05-31T20:00:51.347 DEBUG:boto:auth_path=/pklxkd-13/ 2023-05-31T20:00:51.348 DEBUG:boto:path=/pklxkd-13/?prefix=obj 2023-05-31T20:00:51.349 DEBUG:boto:auth_path=/pklxkd-13/?prefix=obj 2023-05-31T20:00:51.350 DEBUG:boto:Method: GET 2023-05-31T20:00:51.351 DEBUG:boto:Path: /pklxkd-13/?prefix=obj 2023-05-31T20:00:51.351 DEBUG:boto:Data: 2023-05-31T20:00:51.352 DEBUG:boto:Headers: {} 2023-05-31T20:00:51.352 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:00:51.353 DEBUG:boto:Port: 8000 2023-05-31T20:00:51.354 DEBUG:boto:Params: {} 2023-05-31T20:00:51.354 DEBUG:boto:Token: None 2023-05-31T20:00:51.355 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:00:51 GMT /pklxkd-13/ 2023-05-31T20:00:51.357 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:8nY7ypnD0gQg08MwmrHH+KcmiAo= 2023-05-31T20:00:51.358 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:00:51 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:8nY7ypnD0gQg08MwmrHH+KcmiAo=', 'Content-Length': '0'} 2023-05-31T20:00:51.361 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000002dfbd00ca6413b93-006477a773-110e-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:00:51 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:00:51.361 DEBUG:boto:b'pklxkd-13obj1000falseobj2023-05-31T20:00:08.058Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:00:51.363 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2023-05-31T20:00:51.364 DEBUG:boto:path=/pklxkd-13/obj 2023-05-31T20:00:51.364 DEBUG:boto:auth_path=/pklxkd-13/obj 2023-05-31T20:00:51.365 DEBUG:boto:path=/pklxkd-13/obj?versionId=ooEeuYF3vuTJzW20FXOdWWO9-xNIuan 2023-05-31T20:00:51.366 DEBUG:boto:auth_path=/pklxkd-13/obj?versionId=ooEeuYF3vuTJzW20FXOdWWO9-xNIuan 2023-05-31T20:00:51.366 DEBUG:boto:Method: GET 2023-05-31T20:00:51.367 DEBUG:boto:Path: /pklxkd-13/obj?versionId=ooEeuYF3vuTJzW20FXOdWWO9-xNIuan 2023-05-31T20:00:51.368 DEBUG:boto:Data: 2023-05-31T20:00:51.369 DEBUG:boto:Headers: {} 2023-05-31T20:00:51.370 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:00:51.370 DEBUG:boto:Port: 8001 2023-05-31T20:00:51.371 DEBUG:boto:Params: {} 2023-05-31T20:00:51.372 DEBUG:boto:Token: None 2023-05-31T20:00:51.373 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:00:51 GMT /pklxkd-13/obj?versionId=ooEeuYF3vuTJzW20FXOdWWO9-xNIuan 2023-05-31T20:00:51.374 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:O2J7cCT8632TJu6MpNy8WTKboB4= 2023-05-31T20:00:51.375 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:00:51 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:O2J7cCT8632TJu6MpNy8WTKboB4=', 'Content-Length': '0'} 2023-05-31T20:00:51.377 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:00:07 GMT'), ('x-amz-version-id', 'ooEeuYF3vuTJzW20FXOdWWO9-xNIuan'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000004ab6db81ab300b2a-006477a773-1112-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:00:51 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:00:51.378 DEBUG:boto:path=/pklxkd-13/obj 2023-05-31T20:00:51.379 DEBUG:boto:auth_path=/pklxkd-13/obj 2023-05-31T20:00:51.379 DEBUG:boto:path=/pklxkd-13/obj?versionId=ooEeuYF3vuTJzW20FXOdWWO9-xNIuan 2023-05-31T20:00:51.379 DEBUG:boto:auth_path=/pklxkd-13/obj?versionId=ooEeuYF3vuTJzW20FXOdWWO9-xNIuan 2023-05-31T20:00:51.380 DEBUG:boto:Method: GET 2023-05-31T20:00:51.380 DEBUG:boto:Path: /pklxkd-13/obj?versionId=ooEeuYF3vuTJzW20FXOdWWO9-xNIuan 2023-05-31T20:00:51.381 DEBUG:boto:Data: 2023-05-31T20:00:51.381 DEBUG:boto:Headers: {} 2023-05-31T20:00:51.382 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:00:51.382 DEBUG:boto:Port: 8000 2023-05-31T20:00:51.383 DEBUG:boto:Params: {} 2023-05-31T20:00:51.384 DEBUG:boto:Token: None 2023-05-31T20:00:51.385 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:00:51 GMT /pklxkd-13/obj?versionId=ooEeuYF3vuTJzW20FXOdWWO9-xNIuan 2023-05-31T20:00:51.386 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:O2J7cCT8632TJu6MpNy8WTKboB4= 2023-05-31T20:00:51.387 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:00:51 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:O2J7cCT8632TJu6MpNy8WTKboB4=', 'Content-Length': '0'} 2023-05-31T20:00:51.389 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:00:07 GMT'), ('x-amz-version-id', 'ooEeuYF3vuTJzW20FXOdWWO9-xNIuan'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000b73b9270f32af0cd-006477a773-110e-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:00:51 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:00:51.432 DEBUG:boto:path=/pklxkd-13/obj 2023-05-31T20:00:51.433 DEBUG:boto:auth_path=/pklxkd-13/obj 2023-05-31T20:00:51.433 DEBUG:boto:path=/pklxkd-13/obj?versionId=ooEeuYF3vuTJzW20FXOdWWO9-xNIuan 2023-05-31T20:00:51.434 DEBUG:boto:auth_path=/pklxkd-13/obj?versionId=ooEeuYF3vuTJzW20FXOdWWO9-xNIuan 2023-05-31T20:00:51.434 DEBUG:boto:Method: HEAD 2023-05-31T20:00:51.435 DEBUG:boto:Path: /pklxkd-13/obj?versionId=ooEeuYF3vuTJzW20FXOdWWO9-xNIuan 2023-05-31T20:00:51.435 DEBUG:boto:Data: 2023-05-31T20:00:51.435 DEBUG:boto:Headers: {} 2023-05-31T20:00:51.436 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:00:51.436 DEBUG:boto:Port: 8001 2023-05-31T20:00:51.437 DEBUG:boto:Params: {} 2023-05-31T20:00:51.437 DEBUG:boto:Token: None 2023-05-31T20:00:51.438 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:00:51 GMT /pklxkd-13/obj?versionId=ooEeuYF3vuTJzW20FXOdWWO9-xNIuan 2023-05-31T20:00:51.438 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:9/Pd56dh6BAARCampef1SpPtJyY= 2023-05-31T20:00:51.439 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:00:51 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:9/Pd56dh6BAARCampef1SpPtJyY=', 'Content-Length': '0'} 2023-05-31T20:00:51.441 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:00:07 GMT'), ('x-amz-version-id', 'ooEeuYF3vuTJzW20FXOdWWO9-xNIuan'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000047a3d21209db3f48-006477a773-1112-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:00:51 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:00:51.442 DEBUG:boto:path=/pklxkd-13/obj 2023-05-31T20:00:51.442 DEBUG:boto:auth_path=/pklxkd-13/obj 2023-05-31T20:00:51.442 DEBUG:boto:path=/pklxkd-13/obj?versionId=ooEeuYF3vuTJzW20FXOdWWO9-xNIuan 2023-05-31T20:00:51.443 DEBUG:boto:auth_path=/pklxkd-13/obj?versionId=ooEeuYF3vuTJzW20FXOdWWO9-xNIuan 2023-05-31T20:00:51.443 DEBUG:boto:Method: HEAD 2023-05-31T20:00:51.444 DEBUG:boto:Path: /pklxkd-13/obj?versionId=ooEeuYF3vuTJzW20FXOdWWO9-xNIuan 2023-05-31T20:00:51.444 DEBUG:boto:Data: 2023-05-31T20:00:51.444 DEBUG:boto:Headers: {} 2023-05-31T20:00:51.445 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:00:51.445 DEBUG:boto:Port: 8000 2023-05-31T20:00:51.445 DEBUG:boto:Params: {} 2023-05-31T20:00:51.446 DEBUG:boto:Token: None 2023-05-31T20:00:51.446 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:00:51 GMT /pklxkd-13/obj?versionId=ooEeuYF3vuTJzW20FXOdWWO9-xNIuan 2023-05-31T20:00:51.447 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:9/Pd56dh6BAARCampef1SpPtJyY= 2023-05-31T20:00:51.447 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:00:51 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:9/Pd56dh6BAARCampef1SpPtJyY=', 'Content-Length': '0'} 2023-05-31T20:00:51.449 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:00:07 GMT'), ('x-amz-version-id', 'ooEeuYF3vuTJzW20FXOdWWO9-xNIuan'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000097c6d5d49ee7f426-006477a773-110e-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:00:51 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:00:51.449 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2023-05-31T20:00:51.450 DEBUG:boto:path=/pklxkd-13/obj 2023-05-31T20:00:51.450 DEBUG:boto:auth_path=/pklxkd-13/obj 2023-05-31T20:00:51.451 DEBUG:boto:path=/pklxkd-13/obj?versionId=ooEeuYF3vuTJzW20FXOdWWO9-xNIuan 2023-05-31T20:00:51.451 DEBUG:boto:auth_path=/pklxkd-13/obj?versionId=ooEeuYF3vuTJzW20FXOdWWO9-xNIuan 2023-05-31T20:00:51.451 DEBUG:boto:Method: GET 2023-05-31T20:00:51.452 DEBUG:boto:Path: /pklxkd-13/obj?versionId=ooEeuYF3vuTJzW20FXOdWWO9-xNIuan 2023-05-31T20:00:51.452 DEBUG:boto:Data: 2023-05-31T20:00:51.452 DEBUG:boto:Headers: {} 2023-05-31T20:00:51.453 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:00:51.453 DEBUG:boto:Port: 8001 2023-05-31T20:00:51.453 DEBUG:boto:Params: {} 2023-05-31T20:00:51.454 DEBUG:boto:Token: None 2023-05-31T20:00:51.454 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:00:51 GMT /pklxkd-13/obj?versionId=ooEeuYF3vuTJzW20FXOdWWO9-xNIuan 2023-05-31T20:00:51.455 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:O2J7cCT8632TJu6MpNy8WTKboB4= 2023-05-31T20:00:51.455 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:00:51 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:O2J7cCT8632TJu6MpNy8WTKboB4=', 'Content-Length': '0'} 2023-05-31T20:00:51.457 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:00:07 GMT'), ('x-amz-version-id', 'ooEeuYF3vuTJzW20FXOdWWO9-xNIuan'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000061ca92f0edc13791-006477a773-1112-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:00:51 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:00:51.500 DEBUG:boto:path=/pklxkd-13/obj 2023-05-31T20:00:51.501 DEBUG:boto:auth_path=/pklxkd-13/obj 2023-05-31T20:00:51.501 DEBUG:boto:path=/pklxkd-13/obj?versionId=ooEeuYF3vuTJzW20FXOdWWO9-xNIuan 2023-05-31T20:00:51.502 DEBUG:boto:auth_path=/pklxkd-13/obj?versionId=ooEeuYF3vuTJzW20FXOdWWO9-xNIuan 2023-05-31T20:00:51.502 DEBUG:boto:Method: GET 2023-05-31T20:00:51.502 DEBUG:boto:Path: /pklxkd-13/obj?versionId=ooEeuYF3vuTJzW20FXOdWWO9-xNIuan 2023-05-31T20:00:51.503 DEBUG:boto:Data: 2023-05-31T20:00:51.503 DEBUG:boto:Headers: {} 2023-05-31T20:00:51.503 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:00:51.504 DEBUG:boto:Port: 8000 2023-05-31T20:00:51.504 DEBUG:boto:Params: {} 2023-05-31T20:00:51.505 DEBUG:boto:Token: None 2023-05-31T20:00:51.505 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:00:51 GMT /pklxkd-13/obj?versionId=ooEeuYF3vuTJzW20FXOdWWO9-xNIuan 2023-05-31T20:00:51.506 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:O2J7cCT8632TJu6MpNy8WTKboB4= 2023-05-31T20:00:51.506 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:00:51 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:O2J7cCT8632TJu6MpNy8WTKboB4=', 'Content-Length': '0'} 2023-05-31T20:00:51.508 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:00:07 GMT'), ('x-amz-version-id', 'ooEeuYF3vuTJzW20FXOdWWO9-xNIuan'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000ccc8906bd3e465c3-006477a773-110e-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:00:51 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:00:51.509 DEBUG:boto:path=/pklxkd-13/ 2023-05-31T20:00:51.509 DEBUG:boto:auth_path=/pklxkd-13/ 2023-05-31T20:00:51.509 DEBUG:boto:path=/pklxkd-13/?prefix=obj 2023-05-31T20:00:51.510 DEBUG:boto:auth_path=/pklxkd-13/?prefix=obj 2023-05-31T20:00:51.510 DEBUG:boto:Method: GET 2023-05-31T20:00:51.511 DEBUG:boto:Path: /pklxkd-13/?prefix=obj 2023-05-31T20:00:51.511 DEBUG:boto:Data: 2023-05-31T20:00:51.511 DEBUG:boto:Headers: {} 2023-05-31T20:00:51.512 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:00:51.512 DEBUG:boto:Port: 8001 2023-05-31T20:00:51.512 DEBUG:boto:Params: {} 2023-05-31T20:00:51.513 DEBUG:boto:Token: None 2023-05-31T20:00:51.513 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:00:51 GMT /pklxkd-13/ 2023-05-31T20:00:51.514 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:8nY7ypnD0gQg08MwmrHH+KcmiAo= 2023-05-31T20:00:51.514 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:00:51 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:8nY7ypnD0gQg08MwmrHH+KcmiAo=', 'Content-Length': '0'} 2023-05-31T20:00:51.517 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000041468b42f7a89245-006477a773-1112-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:00:51 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:00:51.517 DEBUG:boto:b'pklxkd-13obj1000falseobj2023-05-31T20:00:08.058Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:00:51.518 DEBUG:boto:path=/pklxkd-13/ 2023-05-31T20:00:51.518 DEBUG:boto:auth_path=/pklxkd-13/ 2023-05-31T20:00:51.519 DEBUG:boto:path=/pklxkd-13/?prefix=obj 2023-05-31T20:00:51.519 DEBUG:boto:auth_path=/pklxkd-13/?prefix=obj 2023-05-31T20:00:51.520 DEBUG:boto:Method: GET 2023-05-31T20:00:51.520 DEBUG:boto:Path: /pklxkd-13/?prefix=obj 2023-05-31T20:00:51.520 DEBUG:boto:Data: 2023-05-31T20:00:51.521 DEBUG:boto:Headers: {} 2023-05-31T20:00:51.521 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:00:51.521 DEBUG:boto:Port: 8000 2023-05-31T20:00:51.522 DEBUG:boto:Params: {} 2023-05-31T20:00:51.522 DEBUG:boto:Token: None 2023-05-31T20:00:51.522 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:00:51 GMT /pklxkd-13/ 2023-05-31T20:00:51.523 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:8nY7ypnD0gQg08MwmrHH+KcmiAo= 2023-05-31T20:00:51.523 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:00:51 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:8nY7ypnD0gQg08MwmrHH+KcmiAo=', 'Content-Length': '0'} 2023-05-31T20:00:51.526 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a3f32fb855b9a88b-006477a773-110e-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:00:51 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:00:51.526 DEBUG:boto:b'pklxkd-13obj1000falseobj2023-05-31T20:00:08.058Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:00:51.527 DEBUG:boto:path=/pklxkd-13/obj 2023-05-31T20:00:51.527 DEBUG:boto:auth_path=/pklxkd-13/obj 2023-05-31T20:00:51.528 DEBUG:boto:Method: HEAD 2023-05-31T20:00:51.528 DEBUG:boto:Path: /pklxkd-13/obj 2023-05-31T20:00:51.528 DEBUG:boto:Data: 2023-05-31T20:00:51.529 DEBUG:boto:Headers: {} 2023-05-31T20:00:51.529 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:00:51.529 DEBUG:boto:Port: 8001 2023-05-31T20:00:51.530 DEBUG:boto:Params: {} 2023-05-31T20:00:51.530 DEBUG:boto:Token: None 2023-05-31T20:00:51.530 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:00:51 GMT /pklxkd-13/obj 2023-05-31T20:00:51.531 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:sLcfXAyGTtS5A8hAUE3sGSDaL70= 2023-05-31T20:00:51.531 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:00:51 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:sLcfXAyGTtS5A8hAUE3sGSDaL70=', 'Content-Length': '0'} 2023-05-31T20:00:51.534 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:00:08 GMT'), ('x-amz-version-id', '.K0qHk3QiH4acREzLteeQLJmFzn4x-1'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000088280ecf5b2a8c09-006477a773-1112-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:00:51 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:00:51.535 DEBUG:boto:path=/pklxkd-13/obj 2023-05-31T20:00:51.535 DEBUG:boto:auth_path=/pklxkd-13/obj 2023-05-31T20:00:51.536 DEBUG:boto:Method: HEAD 2023-05-31T20:00:51.536 DEBUG:boto:Path: /pklxkd-13/obj 2023-05-31T20:00:51.536 DEBUG:boto:Data: 2023-05-31T20:00:51.537 DEBUG:boto:Headers: {} 2023-05-31T20:00:51.537 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:00:51.537 DEBUG:boto:Port: 8000 2023-05-31T20:00:51.538 DEBUG:boto:Params: {} 2023-05-31T20:00:51.538 DEBUG:boto:Token: None 2023-05-31T20:00:51.538 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:00:51 GMT /pklxkd-13/obj 2023-05-31T20:00:51.539 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:sLcfXAyGTtS5A8hAUE3sGSDaL70= 2023-05-31T20:00:51.540 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:00:51 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:sLcfXAyGTtS5A8hAUE3sGSDaL70=', 'Content-Length': '0'} 2023-05-31T20:00:51.543 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:00:08 GMT'), ('x-amz-version-id', '.K0qHk3QiH4acREzLteeQLJmFzn4x-1'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000aaa4cc1da818d540-006477a773-110e-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:00:51 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:00:51.544 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2023-05-31T20:00:51.545 DEBUG:boto:path=/pklxkd-13/obj 2023-05-31T20:00:51.545 DEBUG:boto:auth_path=/pklxkd-13/obj 2023-05-31T20:00:51.546 DEBUG:boto:path=/pklxkd-13/obj?versionId=.K0qHk3QiH4acREzLteeQLJmFzn4x-1 2023-05-31T20:00:51.546 DEBUG:boto:auth_path=/pklxkd-13/obj?versionId=.K0qHk3QiH4acREzLteeQLJmFzn4x-1 2023-05-31T20:00:51.546 DEBUG:boto:Method: GET 2023-05-31T20:00:51.547 DEBUG:boto:Path: /pklxkd-13/obj?versionId=.K0qHk3QiH4acREzLteeQLJmFzn4x-1 2023-05-31T20:00:51.547 DEBUG:boto:Data: 2023-05-31T20:00:51.547 DEBUG:boto:Headers: {} 2023-05-31T20:00:51.548 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:00:51.548 DEBUG:boto:Port: 8001 2023-05-31T20:00:51.549 DEBUG:boto:Params: {} 2023-05-31T20:00:51.549 DEBUG:boto:Token: None 2023-05-31T20:00:51.549 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:00:51 GMT /pklxkd-13/obj?versionId=.K0qHk3QiH4acREzLteeQLJmFzn4x-1 2023-05-31T20:00:51.550 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:MWWK358ZEpKN0sicgroa3DvTmS4= 2023-05-31T20:00:51.550 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:00:51 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:MWWK358ZEpKN0sicgroa3DvTmS4=', 'Content-Length': '0'} 2023-05-31T20:00:51.552 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:00:08 GMT'), ('x-amz-version-id', '.K0qHk3QiH4acREzLteeQLJmFzn4x-1'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000002068f21d923bb991-006477a773-1112-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:00:51 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:00:51.592 DEBUG:boto:path=/pklxkd-13/obj 2023-05-31T20:00:51.593 DEBUG:boto:auth_path=/pklxkd-13/obj 2023-05-31T20:00:51.593 DEBUG:boto:path=/pklxkd-13/obj?versionId=.K0qHk3QiH4acREzLteeQLJmFzn4x-1 2023-05-31T20:00:51.594 DEBUG:boto:auth_path=/pklxkd-13/obj?versionId=.K0qHk3QiH4acREzLteeQLJmFzn4x-1 2023-05-31T20:00:51.594 DEBUG:boto:Method: GET 2023-05-31T20:00:51.595 DEBUG:boto:Path: /pklxkd-13/obj?versionId=.K0qHk3QiH4acREzLteeQLJmFzn4x-1 2023-05-31T20:00:51.596 DEBUG:boto:Data: 2023-05-31T20:00:51.597 DEBUG:boto:Headers: {} 2023-05-31T20:00:51.598 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:00:51.599 DEBUG:boto:Port: 8000 2023-05-31T20:00:51.599 DEBUG:boto:Params: {} 2023-05-31T20:00:51.600 DEBUG:boto:Token: None 2023-05-31T20:00:51.601 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:00:51 GMT /pklxkd-13/obj?versionId=.K0qHk3QiH4acREzLteeQLJmFzn4x-1 2023-05-31T20:00:51.602 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:MWWK358ZEpKN0sicgroa3DvTmS4= 2023-05-31T20:00:51.603 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:00:51 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:MWWK358ZEpKN0sicgroa3DvTmS4=', 'Content-Length': '0'} 2023-05-31T20:00:51.605 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:00:08 GMT'), ('x-amz-version-id', '.K0qHk3QiH4acREzLteeQLJmFzn4x-1'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000ddf05961253a2ed3-006477a773-110e-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:00:51 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:00:51.606 DEBUG:boto:path=/pklxkd-13/ 2023-05-31T20:00:51.606 DEBUG:boto:auth_path=/pklxkd-13/ 2023-05-31T20:00:51.607 DEBUG:boto:path=/pklxkd-13/?prefix=obj 2023-05-31T20:00:51.607 DEBUG:boto:auth_path=/pklxkd-13/?prefix=obj 2023-05-31T20:00:51.607 DEBUG:boto:Method: GET 2023-05-31T20:00:51.608 DEBUG:boto:Path: /pklxkd-13/?prefix=obj 2023-05-31T20:00:51.608 DEBUG:boto:Data: 2023-05-31T20:00:51.608 DEBUG:boto:Headers: {} 2023-05-31T20:00:51.609 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:00:51.609 DEBUG:boto:Port: 8001 2023-05-31T20:00:51.610 DEBUG:boto:Params: {} 2023-05-31T20:00:51.610 DEBUG:boto:Token: None 2023-05-31T20:00:51.610 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:00:51 GMT /pklxkd-13/ 2023-05-31T20:00:51.611 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:8nY7ypnD0gQg08MwmrHH+KcmiAo= 2023-05-31T20:00:51.611 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:00:51 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:8nY7ypnD0gQg08MwmrHH+KcmiAo=', 'Content-Length': '0'} 2023-05-31T20:00:51.614 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000927b855f0cf68c41-006477a773-1112-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:00:51 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:00:51.615 DEBUG:boto:b'pklxkd-13obj1000falseobj2023-05-31T20:00:08.058Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:00:51.615 DEBUG:boto:path=/pklxkd-13/ 2023-05-31T20:00:51.616 DEBUG:boto:auth_path=/pklxkd-13/ 2023-05-31T20:00:51.616 DEBUG:boto:path=/pklxkd-13/?prefix=obj 2023-05-31T20:00:51.617 DEBUG:boto:auth_path=/pklxkd-13/?prefix=obj 2023-05-31T20:00:51.617 DEBUG:boto:Method: GET 2023-05-31T20:00:51.618 DEBUG:boto:Path: /pklxkd-13/?prefix=obj 2023-05-31T20:00:51.618 DEBUG:boto:Data: 2023-05-31T20:00:51.619 DEBUG:boto:Headers: {} 2023-05-31T20:00:51.619 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:00:51.619 DEBUG:boto:Port: 8000 2023-05-31T20:00:51.620 DEBUG:boto:Params: {} 2023-05-31T20:00:51.620 DEBUG:boto:Token: None 2023-05-31T20:00:51.621 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:00:51 GMT /pklxkd-13/ 2023-05-31T20:00:51.622 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:8nY7ypnD0gQg08MwmrHH+KcmiAo= 2023-05-31T20:00:51.623 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:00:51 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:8nY7ypnD0gQg08MwmrHH+KcmiAo=', 'Content-Length': '0'} 2023-05-31T20:00:51.626 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000081ee2de8f0598f85-006477a773-110e-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:00:51 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:00:51.627 DEBUG:boto:b'pklxkd-13obj1000falseobj2023-05-31T20:00:08.058Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:00:51.628 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2023-05-31T20:00:51.628 DEBUG:boto:path=/pklxkd-13/obj 2023-05-31T20:00:51.628 DEBUG:boto:auth_path=/pklxkd-13/obj 2023-05-31T20:00:51.629 DEBUG:boto:path=/pklxkd-13/obj?versionId=O9dH9NhVFdlcrFlXwres-hSyGvDhDaQ 2023-05-31T20:00:51.629 DEBUG:boto:auth_path=/pklxkd-13/obj?versionId=O9dH9NhVFdlcrFlXwres-hSyGvDhDaQ 2023-05-31T20:00:51.630 DEBUG:boto:Method: GET 2023-05-31T20:00:51.630 DEBUG:boto:Path: /pklxkd-13/obj?versionId=O9dH9NhVFdlcrFlXwres-hSyGvDhDaQ 2023-05-31T20:00:51.630 DEBUG:boto:Data: 2023-05-31T20:00:51.631 DEBUG:boto:Headers: {} 2023-05-31T20:00:51.631 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:00:51.631 DEBUG:boto:Port: 8001 2023-05-31T20:00:51.632 DEBUG:boto:Params: {} 2023-05-31T20:00:51.632 DEBUG:boto:Token: None 2023-05-31T20:00:51.633 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:00:51 GMT /pklxkd-13/obj?versionId=O9dH9NhVFdlcrFlXwres-hSyGvDhDaQ 2023-05-31T20:00:51.633 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:liRvmQr2OgnPgLWlEwTAhz4dZ+c= 2023-05-31T20:00:51.634 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:00:51 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:liRvmQr2OgnPgLWlEwTAhz4dZ+c=', 'Content-Length': '0'} 2023-05-31T20:00:51.636 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:00:07 GMT'), ('x-amz-version-id', 'O9dH9NhVFdlcrFlXwres-hSyGvDhDaQ'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000006d2529d411274b7a-006477a773-1112-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:00:51 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:00:51.637 DEBUG:boto:path=/pklxkd-13/obj 2023-05-31T20:00:51.637 DEBUG:boto:auth_path=/pklxkd-13/obj 2023-05-31T20:00:51.637 DEBUG:boto:path=/pklxkd-13/obj?versionId=O9dH9NhVFdlcrFlXwres-hSyGvDhDaQ 2023-05-31T20:00:51.638 DEBUG:boto:auth_path=/pklxkd-13/obj?versionId=O9dH9NhVFdlcrFlXwres-hSyGvDhDaQ 2023-05-31T20:00:51.638 DEBUG:boto:Method: GET 2023-05-31T20:00:51.639 DEBUG:boto:Path: /pklxkd-13/obj?versionId=O9dH9NhVFdlcrFlXwres-hSyGvDhDaQ 2023-05-31T20:00:51.639 DEBUG:boto:Data: 2023-05-31T20:00:51.639 DEBUG:boto:Headers: {} 2023-05-31T20:00:51.640 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:00:51.640 DEBUG:boto:Port: 8000 2023-05-31T20:00:51.641 DEBUG:boto:Params: {} 2023-05-31T20:00:51.641 DEBUG:boto:Token: None 2023-05-31T20:00:51.642 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:00:51 GMT /pklxkd-13/obj?versionId=O9dH9NhVFdlcrFlXwres-hSyGvDhDaQ 2023-05-31T20:00:51.642 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:liRvmQr2OgnPgLWlEwTAhz4dZ+c= 2023-05-31T20:00:51.642 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:00:51 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:liRvmQr2OgnPgLWlEwTAhz4dZ+c=', 'Content-Length': '0'} 2023-05-31T20:00:51.644 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:00:07 GMT'), ('x-amz-version-id', 'O9dH9NhVFdlcrFlXwres-hSyGvDhDaQ'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000089f2a82d8c363eb1-006477a773-110e-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:00:51 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:00:51.645 DEBUG:boto:path=/pklxkd-13/obj 2023-05-31T20:00:51.646 DEBUG:boto:auth_path=/pklxkd-13/obj 2023-05-31T20:00:51.646 DEBUG:boto:path=/pklxkd-13/obj?versionId=O9dH9NhVFdlcrFlXwres-hSyGvDhDaQ 2023-05-31T20:00:51.646 DEBUG:boto:auth_path=/pklxkd-13/obj?versionId=O9dH9NhVFdlcrFlXwres-hSyGvDhDaQ 2023-05-31T20:00:51.647 DEBUG:boto:Method: HEAD 2023-05-31T20:00:51.647 DEBUG:boto:Path: /pklxkd-13/obj?versionId=O9dH9NhVFdlcrFlXwres-hSyGvDhDaQ 2023-05-31T20:00:51.648 DEBUG:boto:Data: 2023-05-31T20:00:51.648 DEBUG:boto:Headers: {} 2023-05-31T20:00:51.648 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:00:51.649 DEBUG:boto:Port: 8001 2023-05-31T20:00:51.649 DEBUG:boto:Params: {} 2023-05-31T20:00:51.649 DEBUG:boto:Token: None 2023-05-31T20:00:51.650 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:00:51 GMT /pklxkd-13/obj?versionId=O9dH9NhVFdlcrFlXwres-hSyGvDhDaQ 2023-05-31T20:00:51.650 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:he5Y9CmiiKACgn6lop614NJN3Kc= 2023-05-31T20:00:51.651 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:00:51 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:he5Y9CmiiKACgn6lop614NJN3Kc=', 'Content-Length': '0'} 2023-05-31T20:00:51.653 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:00:07 GMT'), ('x-amz-version-id', 'O9dH9NhVFdlcrFlXwres-hSyGvDhDaQ'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000551ec962dbf53d86-006477a773-1112-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:00:51 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:00:51.654 DEBUG:boto:path=/pklxkd-13/obj 2023-05-31T20:00:51.654 DEBUG:boto:auth_path=/pklxkd-13/obj 2023-05-31T20:00:51.655 DEBUG:boto:path=/pklxkd-13/obj?versionId=O9dH9NhVFdlcrFlXwres-hSyGvDhDaQ 2023-05-31T20:00:51.655 DEBUG:boto:auth_path=/pklxkd-13/obj?versionId=O9dH9NhVFdlcrFlXwres-hSyGvDhDaQ 2023-05-31T20:00:51.655 DEBUG:boto:Method: HEAD 2023-05-31T20:00:51.656 DEBUG:boto:Path: /pklxkd-13/obj?versionId=O9dH9NhVFdlcrFlXwres-hSyGvDhDaQ 2023-05-31T20:00:51.656 DEBUG:boto:Data: 2023-05-31T20:00:51.656 DEBUG:boto:Headers: {} 2023-05-31T20:00:51.657 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:00:51.657 DEBUG:boto:Port: 8000 2023-05-31T20:00:51.657 DEBUG:boto:Params: {} 2023-05-31T20:00:51.658 DEBUG:boto:Token: None 2023-05-31T20:00:51.658 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:00:51 GMT /pklxkd-13/obj?versionId=O9dH9NhVFdlcrFlXwres-hSyGvDhDaQ 2023-05-31T20:00:51.659 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:he5Y9CmiiKACgn6lop614NJN3Kc= 2023-05-31T20:00:51.659 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:00:51 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:he5Y9CmiiKACgn6lop614NJN3Kc=', 'Content-Length': '0'} 2023-05-31T20:00:51.661 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:00:07 GMT'), ('x-amz-version-id', 'O9dH9NhVFdlcrFlXwres-hSyGvDhDaQ'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000005cbd53aa308bd482-006477a773-110e-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:00:51 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:00:51.662 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2023-05-31T20:00:51.663 DEBUG:boto:path=/pklxkd-13/obj 2023-05-31T20:00:51.664 DEBUG:boto:auth_path=/pklxkd-13/obj 2023-05-31T20:00:51.665 DEBUG:boto:path=/pklxkd-13/obj?versionId=O9dH9NhVFdlcrFlXwres-hSyGvDhDaQ 2023-05-31T20:00:51.666 DEBUG:boto:auth_path=/pklxkd-13/obj?versionId=O9dH9NhVFdlcrFlXwres-hSyGvDhDaQ 2023-05-31T20:00:51.667 DEBUG:boto:Method: GET 2023-05-31T20:00:51.667 DEBUG:boto:Path: /pklxkd-13/obj?versionId=O9dH9NhVFdlcrFlXwres-hSyGvDhDaQ 2023-05-31T20:00:51.667 DEBUG:boto:Data: 2023-05-31T20:00:51.668 DEBUG:boto:Headers: {} 2023-05-31T20:00:51.668 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:00:51.668 DEBUG:boto:Port: 8001 2023-05-31T20:00:51.669 DEBUG:boto:Params: {} 2023-05-31T20:00:51.669 DEBUG:boto:Token: None 2023-05-31T20:00:51.670 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:00:51 GMT /pklxkd-13/obj?versionId=O9dH9NhVFdlcrFlXwres-hSyGvDhDaQ 2023-05-31T20:00:51.670 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:liRvmQr2OgnPgLWlEwTAhz4dZ+c= 2023-05-31T20:00:51.671 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:00:51 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:liRvmQr2OgnPgLWlEwTAhz4dZ+c=', 'Content-Length': '0'} 2023-05-31T20:00:51.672 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:00:07 GMT'), ('x-amz-version-id', 'O9dH9NhVFdlcrFlXwres-hSyGvDhDaQ'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000007105bf741ca36fb4-006477a773-1112-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:00:51 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:00:51.673 DEBUG:boto:path=/pklxkd-13/obj 2023-05-31T20:00:51.674 DEBUG:boto:auth_path=/pklxkd-13/obj 2023-05-31T20:00:51.674 DEBUG:boto:path=/pklxkd-13/obj?versionId=O9dH9NhVFdlcrFlXwres-hSyGvDhDaQ 2023-05-31T20:00:51.674 DEBUG:boto:auth_path=/pklxkd-13/obj?versionId=O9dH9NhVFdlcrFlXwres-hSyGvDhDaQ 2023-05-31T20:00:51.675 DEBUG:boto:Method: GET 2023-05-31T20:00:51.675 DEBUG:boto:Path: /pklxkd-13/obj?versionId=O9dH9NhVFdlcrFlXwres-hSyGvDhDaQ 2023-05-31T20:00:51.675 DEBUG:boto:Data: 2023-05-31T20:00:51.676 DEBUG:boto:Headers: {} 2023-05-31T20:00:51.676 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:00:51.677 DEBUG:boto:Port: 8000 2023-05-31T20:00:51.677 DEBUG:boto:Params: {} 2023-05-31T20:00:51.677 DEBUG:boto:Token: None 2023-05-31T20:00:51.678 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:00:51 GMT /pklxkd-13/obj?versionId=O9dH9NhVFdlcrFlXwres-hSyGvDhDaQ 2023-05-31T20:00:51.678 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:liRvmQr2OgnPgLWlEwTAhz4dZ+c= 2023-05-31T20:00:51.679 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:00:51 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:liRvmQr2OgnPgLWlEwTAhz4dZ+c=', 'Content-Length': '0'} 2023-05-31T20:00:51.681 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:00:07 GMT'), ('x-amz-version-id', 'O9dH9NhVFdlcrFlXwres-hSyGvDhDaQ'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000071dbb1605547646d-006477a773-110e-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:00:51 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:00:51.724 DEBUG:boto:path=/pklxkd-13/ 2023-05-31T20:00:51.725 DEBUG:boto:auth_path=/pklxkd-13/ 2023-05-31T20:00:51.726 DEBUG:boto:path=/pklxkd-13/?prefix=obj 2023-05-31T20:00:51.727 DEBUG:boto:auth_path=/pklxkd-13/?prefix=obj 2023-05-31T20:00:51.727 DEBUG:boto:Method: GET 2023-05-31T20:00:51.728 DEBUG:boto:Path: /pklxkd-13/?prefix=obj 2023-05-31T20:00:51.729 DEBUG:boto:Data: 2023-05-31T20:00:51.730 DEBUG:boto:Headers: {} 2023-05-31T20:00:51.730 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:00:51.731 DEBUG:boto:Port: 8001 2023-05-31T20:00:51.732 DEBUG:boto:Params: {} 2023-05-31T20:00:51.732 DEBUG:boto:Token: None 2023-05-31T20:00:51.733 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:00:51 GMT /pklxkd-13/ 2023-05-31T20:00:51.734 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:8nY7ypnD0gQg08MwmrHH+KcmiAo= 2023-05-31T20:00:51.735 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:00:51 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:8nY7ypnD0gQg08MwmrHH+KcmiAo=', 'Content-Length': '0'} 2023-05-31T20:00:51.738 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d1ecabb3ca4a25d9-006477a773-1112-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:00:51 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:00:51.738 DEBUG:boto:b'pklxkd-13obj1000falseobj2023-05-31T20:00:08.058Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:00:51.739 DEBUG:boto:path=/pklxkd-13/ 2023-05-31T20:00:51.740 DEBUG:boto:auth_path=/pklxkd-13/ 2023-05-31T20:00:51.741 DEBUG:boto:path=/pklxkd-13/?prefix=obj 2023-05-31T20:00:51.742 DEBUG:boto:auth_path=/pklxkd-13/?prefix=obj 2023-05-31T20:00:51.743 DEBUG:boto:Method: GET 2023-05-31T20:00:51.743 DEBUG:boto:Path: /pklxkd-13/?prefix=obj 2023-05-31T20:00:51.744 DEBUG:boto:Data: 2023-05-31T20:00:51.745 DEBUG:boto:Headers: {} 2023-05-31T20:00:51.746 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:00:51.746 DEBUG:boto:Port: 8000 2023-05-31T20:00:51.746 DEBUG:boto:Params: {} 2023-05-31T20:00:51.747 DEBUG:boto:Token: None 2023-05-31T20:00:51.747 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:00:51 GMT /pklxkd-13/ 2023-05-31T20:00:51.748 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:8nY7ypnD0gQg08MwmrHH+KcmiAo= 2023-05-31T20:00:51.748 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:00:51 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:8nY7ypnD0gQg08MwmrHH+KcmiAo=', 'Content-Length': '0'} 2023-05-31T20:00:51.751 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000bb4605d8773e475d-006477a773-110e-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:00:51 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:00:51.752 DEBUG:boto:b'pklxkd-13obj1000falseobj2023-05-31T20:00:08.058Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:00:51.752 DEBUG:boto:path=/pklxkd-13/obj 2023-05-31T20:00:51.753 DEBUG:boto:auth_path=/pklxkd-13/obj 2023-05-31T20:00:51.753 DEBUG:boto:Method: HEAD 2023-05-31T20:00:51.754 DEBUG:boto:Path: /pklxkd-13/obj 2023-05-31T20:00:51.754 DEBUG:boto:Data: 2023-05-31T20:00:51.755 DEBUG:boto:Headers: {} 2023-05-31T20:00:51.755 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:00:51.755 DEBUG:boto:Port: 8001 2023-05-31T20:00:51.756 DEBUG:boto:Params: {} 2023-05-31T20:00:51.756 DEBUG:boto:Token: None 2023-05-31T20:00:51.757 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:00:51 GMT /pklxkd-13/obj 2023-05-31T20:00:51.757 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:sLcfXAyGTtS5A8hAUE3sGSDaL70= 2023-05-31T20:00:51.757 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:00:51 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:sLcfXAyGTtS5A8hAUE3sGSDaL70=', 'Content-Length': '0'} 2023-05-31T20:00:51.760 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:00:08 GMT'), ('x-amz-version-id', '.K0qHk3QiH4acREzLteeQLJmFzn4x-1'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000bdcd47e9e9265784-006477a773-1112-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:00:51 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:00:51.761 DEBUG:boto:path=/pklxkd-13/obj 2023-05-31T20:00:51.761 DEBUG:boto:auth_path=/pklxkd-13/obj 2023-05-31T20:00:51.762 DEBUG:boto:Method: HEAD 2023-05-31T20:00:51.762 DEBUG:boto:Path: /pklxkd-13/obj 2023-05-31T20:00:51.762 DEBUG:boto:Data: 2023-05-31T20:00:51.763 DEBUG:boto:Headers: {} 2023-05-31T20:00:51.763 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:00:51.763 DEBUG:boto:Port: 8000 2023-05-31T20:00:51.764 DEBUG:boto:Params: {} 2023-05-31T20:00:51.764 DEBUG:boto:Token: None 2023-05-31T20:00:51.764 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:00:51 GMT /pklxkd-13/obj 2023-05-31T20:00:51.765 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:sLcfXAyGTtS5A8hAUE3sGSDaL70= 2023-05-31T20:00:51.765 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:00:51 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:sLcfXAyGTtS5A8hAUE3sGSDaL70=', 'Content-Length': '0'} 2023-05-31T20:00:51.768 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:00:08 GMT'), ('x-amz-version-id', '.K0qHk3QiH4acREzLteeQLJmFzn4x-1'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000dc477fb04305acf5-006477a773-110e-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:00:51 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:00:51.769 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2023-05-31T20:00:51.769 DEBUG:boto:path=/pklxkd-13/obj 2023-05-31T20:00:51.769 DEBUG:boto:auth_path=/pklxkd-13/obj 2023-05-31T20:00:51.770 DEBUG:boto:path=/pklxkd-13/obj?versionId=.K0qHk3QiH4acREzLteeQLJmFzn4x-1 2023-05-31T20:00:51.770 DEBUG:boto:auth_path=/pklxkd-13/obj?versionId=.K0qHk3QiH4acREzLteeQLJmFzn4x-1 2023-05-31T20:00:51.770 DEBUG:boto:Method: GET 2023-05-31T20:00:51.771 DEBUG:boto:Path: /pklxkd-13/obj?versionId=.K0qHk3QiH4acREzLteeQLJmFzn4x-1 2023-05-31T20:00:51.771 DEBUG:boto:Data: 2023-05-31T20:00:51.772 DEBUG:boto:Headers: {} 2023-05-31T20:00:51.772 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:00:51.773 DEBUG:boto:Port: 8001 2023-05-31T20:00:51.773 DEBUG:boto:Params: {} 2023-05-31T20:00:51.773 DEBUG:boto:Token: None 2023-05-31T20:00:51.774 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:00:51 GMT /pklxkd-13/obj?versionId=.K0qHk3QiH4acREzLteeQLJmFzn4x-1 2023-05-31T20:00:51.774 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:MWWK358ZEpKN0sicgroa3DvTmS4= 2023-05-31T20:00:51.775 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:00:51 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:MWWK358ZEpKN0sicgroa3DvTmS4=', 'Content-Length': '0'} 2023-05-31T20:00:51.776 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:00:08 GMT'), ('x-amz-version-id', '.K0qHk3QiH4acREzLteeQLJmFzn4x-1'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000447b088ec0f2b187-006477a773-1112-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:00:51 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:00:51.820 DEBUG:boto:path=/pklxkd-13/obj 2023-05-31T20:00:51.821 DEBUG:boto:auth_path=/pklxkd-13/obj 2023-05-31T20:00:51.822 DEBUG:boto:path=/pklxkd-13/obj?versionId=.K0qHk3QiH4acREzLteeQLJmFzn4x-1 2023-05-31T20:00:51.823 DEBUG:boto:auth_path=/pklxkd-13/obj?versionId=.K0qHk3QiH4acREzLteeQLJmFzn4x-1 2023-05-31T20:00:51.824 DEBUG:boto:Method: GET 2023-05-31T20:00:51.824 DEBUG:boto:Path: /pklxkd-13/obj?versionId=.K0qHk3QiH4acREzLteeQLJmFzn4x-1 2023-05-31T20:00:51.825 DEBUG:boto:Data: 2023-05-31T20:00:51.826 DEBUG:boto:Headers: {} 2023-05-31T20:00:51.826 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:00:51.827 DEBUG:boto:Port: 8000 2023-05-31T20:00:51.827 DEBUG:boto:Params: {} 2023-05-31T20:00:51.828 DEBUG:boto:Token: None 2023-05-31T20:00:51.828 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:00:51 GMT /pklxkd-13/obj?versionId=.K0qHk3QiH4acREzLteeQLJmFzn4x-1 2023-05-31T20:00:51.829 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:MWWK358ZEpKN0sicgroa3DvTmS4= 2023-05-31T20:00:51.829 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:00:51 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:MWWK358ZEpKN0sicgroa3DvTmS4=', 'Content-Length': '0'} 2023-05-31T20:00:51.831 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:00:08 GMT'), ('x-amz-version-id', '.K0qHk3QiH4acREzLteeQLJmFzn4x-1'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000013774d972d310843-006477a773-110e-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:00:51 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:00:51.832 DEBUG:boto:path=/pklxkd-13/ 2023-05-31T20:00:51.832 DEBUG:boto:auth_path=/pklxkd-13/ 2023-05-31T20:00:51.833 DEBUG:boto:path=/pklxkd-13/?prefix=obj 2023-05-31T20:00:51.833 DEBUG:boto:auth_path=/pklxkd-13/?prefix=obj 2023-05-31T20:00:51.833 DEBUG:boto:Method: GET 2023-05-31T20:00:51.834 DEBUG:boto:Path: /pklxkd-13/?prefix=obj 2023-05-31T20:00:51.834 DEBUG:boto:Data: 2023-05-31T20:00:51.835 DEBUG:boto:Headers: {} 2023-05-31T20:00:51.835 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:00:51.835 DEBUG:boto:Port: 8001 2023-05-31T20:00:51.836 DEBUG:boto:Params: {} 2023-05-31T20:00:51.836 DEBUG:boto:Token: None 2023-05-31T20:00:51.837 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:00:51 GMT /pklxkd-13/ 2023-05-31T20:00:51.837 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:8nY7ypnD0gQg08MwmrHH+KcmiAo= 2023-05-31T20:00:51.837 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:00:51 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:8nY7ypnD0gQg08MwmrHH+KcmiAo=', 'Content-Length': '0'} 2023-05-31T20:00:51.840 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000039917eaa97e09b21-006477a773-1112-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:00:51 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:00:51.841 DEBUG:boto:b'pklxkd-13obj1000falseobj2023-05-31T20:00:08.058Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:00:51.841 DEBUG:boto:path=/pklxkd-13/ 2023-05-31T20:00:51.842 DEBUG:boto:auth_path=/pklxkd-13/ 2023-05-31T20:00:51.842 DEBUG:boto:path=/pklxkd-13/?prefix=obj 2023-05-31T20:00:51.842 DEBUG:boto:auth_path=/pklxkd-13/?prefix=obj 2023-05-31T20:00:51.843 DEBUG:boto:Method: GET 2023-05-31T20:00:51.843 DEBUG:boto:Path: /pklxkd-13/?prefix=obj 2023-05-31T20:00:51.844 DEBUG:boto:Data: 2023-05-31T20:00:51.844 DEBUG:boto:Headers: {} 2023-05-31T20:00:51.844 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:00:51.845 DEBUG:boto:Port: 8000 2023-05-31T20:00:51.845 DEBUG:boto:Params: {} 2023-05-31T20:00:51.845 DEBUG:boto:Token: None 2023-05-31T20:00:51.846 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:00:51 GMT /pklxkd-13/ 2023-05-31T20:00:51.846 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:8nY7ypnD0gQg08MwmrHH+KcmiAo= 2023-05-31T20:00:51.847 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:00:51 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:8nY7ypnD0gQg08MwmrHH+KcmiAo=', 'Content-Length': '0'} 2023-05-31T20:00:51.850 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000819eedc9cba1293e-006477a773-110e-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:00:51 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:00:51.850 DEBUG:boto:b'pklxkd-13obj1000falseobj2023-05-31T20:00:08.058Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:00:51.851 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2023-05-31T20:00:51.852 DEBUG:boto:path=/pklxkd-13/obj 2023-05-31T20:00:51.852 DEBUG:boto:auth_path=/pklxkd-13/obj 2023-05-31T20:00:51.852 DEBUG:boto:path=/pklxkd-13/obj?versionId=aS0yzteNN24E5q0Twvp.6Z7sdzsi2hw 2023-05-31T20:00:51.853 DEBUG:boto:auth_path=/pklxkd-13/obj?versionId=aS0yzteNN24E5q0Twvp.6Z7sdzsi2hw 2023-05-31T20:00:51.853 DEBUG:boto:Method: GET 2023-05-31T20:00:51.853 DEBUG:boto:Path: /pklxkd-13/obj?versionId=aS0yzteNN24E5q0Twvp.6Z7sdzsi2hw 2023-05-31T20:00:51.854 DEBUG:boto:Data: 2023-05-31T20:00:51.854 DEBUG:boto:Headers: {} 2023-05-31T20:00:51.855 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:00:51.855 DEBUG:boto:Port: 8001 2023-05-31T20:00:51.855 DEBUG:boto:Params: {} 2023-05-31T20:00:51.856 DEBUG:boto:Token: None 2023-05-31T20:00:51.856 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:00:51 GMT /pklxkd-13/obj?versionId=aS0yzteNN24E5q0Twvp.6Z7sdzsi2hw 2023-05-31T20:00:51.857 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:Hn2BkMesftYSzPTpa++gFVvfzXo= 2023-05-31T20:00:51.857 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:00:51 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:Hn2BkMesftYSzPTpa++gFVvfzXo=', 'Content-Length': '0'} 2023-05-31T20:00:51.859 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:00:07 GMT'), ('x-amz-version-id', 'aS0yzteNN24E5q0Twvp.6Z7sdzsi2hw'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000ef8498c19682f440-006477a773-1112-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:00:51 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:00:51.860 DEBUG:boto:path=/pklxkd-13/obj 2023-05-31T20:00:51.860 DEBUG:boto:auth_path=/pklxkd-13/obj 2023-05-31T20:00:51.860 DEBUG:boto:path=/pklxkd-13/obj?versionId=aS0yzteNN24E5q0Twvp.6Z7sdzsi2hw 2023-05-31T20:00:51.861 DEBUG:boto:auth_path=/pklxkd-13/obj?versionId=aS0yzteNN24E5q0Twvp.6Z7sdzsi2hw 2023-05-31T20:00:51.861 DEBUG:boto:Method: GET 2023-05-31T20:00:51.862 DEBUG:boto:Path: /pklxkd-13/obj?versionId=aS0yzteNN24E5q0Twvp.6Z7sdzsi2hw 2023-05-31T20:00:51.862 DEBUG:boto:Data: 2023-05-31T20:00:51.862 DEBUG:boto:Headers: {} 2023-05-31T20:00:51.863 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:00:51.863 DEBUG:boto:Port: 8000 2023-05-31T20:00:51.863 DEBUG:boto:Params: {} 2023-05-31T20:00:51.864 DEBUG:boto:Token: None 2023-05-31T20:00:51.864 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:00:51 GMT /pklxkd-13/obj?versionId=aS0yzteNN24E5q0Twvp.6Z7sdzsi2hw 2023-05-31T20:00:51.865 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:Hn2BkMesftYSzPTpa++gFVvfzXo= 2023-05-31T20:00:51.865 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:00:51 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:Hn2BkMesftYSzPTpa++gFVvfzXo=', 'Content-Length': '0'} 2023-05-31T20:00:51.867 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:00:07 GMT'), ('x-amz-version-id', 'aS0yzteNN24E5q0Twvp.6Z7sdzsi2hw'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000000d950067ac4763ff-006477a773-110e-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:00:51 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:00:51.868 DEBUG:boto:path=/pklxkd-13/obj 2023-05-31T20:00:51.868 DEBUG:boto:auth_path=/pklxkd-13/obj 2023-05-31T20:00:51.868 DEBUG:boto:path=/pklxkd-13/obj?versionId=aS0yzteNN24E5q0Twvp.6Z7sdzsi2hw 2023-05-31T20:00:51.869 DEBUG:boto:auth_path=/pklxkd-13/obj?versionId=aS0yzteNN24E5q0Twvp.6Z7sdzsi2hw 2023-05-31T20:00:51.869 DEBUG:boto:Method: HEAD 2023-05-31T20:00:51.869 DEBUG:boto:Path: /pklxkd-13/obj?versionId=aS0yzteNN24E5q0Twvp.6Z7sdzsi2hw 2023-05-31T20:00:51.870 DEBUG:boto:Data: 2023-05-31T20:00:51.870 DEBUG:boto:Headers: {} 2023-05-31T20:00:51.870 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:00:51.871 DEBUG:boto:Port: 8001 2023-05-31T20:00:51.871 DEBUG:boto:Params: {} 2023-05-31T20:00:51.871 DEBUG:boto:Token: None 2023-05-31T20:00:51.872 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:00:51 GMT /pklxkd-13/obj?versionId=aS0yzteNN24E5q0Twvp.6Z7sdzsi2hw 2023-05-31T20:00:51.872 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:vxZyhTMlzqKgf/eS75tHK/SVQXI= 2023-05-31T20:00:51.873 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:00:51 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:vxZyhTMlzqKgf/eS75tHK/SVQXI=', 'Content-Length': '0'} 2023-05-31T20:00:51.875 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:00:07 GMT'), ('x-amz-version-id', 'aS0yzteNN24E5q0Twvp.6Z7sdzsi2hw'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000b9ea25ac0ef8b39b-006477a773-1112-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:00:51 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:00:51.875 DEBUG:boto:path=/pklxkd-13/obj 2023-05-31T20:00:51.875 DEBUG:boto:auth_path=/pklxkd-13/obj 2023-05-31T20:00:51.876 DEBUG:boto:path=/pklxkd-13/obj?versionId=aS0yzteNN24E5q0Twvp.6Z7sdzsi2hw 2023-05-31T20:00:51.876 DEBUG:boto:auth_path=/pklxkd-13/obj?versionId=aS0yzteNN24E5q0Twvp.6Z7sdzsi2hw 2023-05-31T20:00:51.877 DEBUG:boto:Method: HEAD 2023-05-31T20:00:51.877 DEBUG:boto:Path: /pklxkd-13/obj?versionId=aS0yzteNN24E5q0Twvp.6Z7sdzsi2hw 2023-05-31T20:00:51.877 DEBUG:boto:Data: 2023-05-31T20:00:51.877 DEBUG:boto:Headers: {} 2023-05-31T20:00:51.878 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:00:51.878 DEBUG:boto:Port: 8000 2023-05-31T20:00:51.878 DEBUG:boto:Params: {} 2023-05-31T20:00:51.879 DEBUG:boto:Token: None 2023-05-31T20:00:51.879 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:00:51 GMT /pklxkd-13/obj?versionId=aS0yzteNN24E5q0Twvp.6Z7sdzsi2hw 2023-05-31T20:00:51.880 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:vxZyhTMlzqKgf/eS75tHK/SVQXI= 2023-05-31T20:00:51.880 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:00:51 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:vxZyhTMlzqKgf/eS75tHK/SVQXI=', 'Content-Length': '0'} 2023-05-31T20:00:51.882 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:00:07 GMT'), ('x-amz-version-id', 'aS0yzteNN24E5q0Twvp.6Z7sdzsi2hw'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000d0022b3bbcc3578c-006477a773-110e-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:00:51 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:00:51.882 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2023-05-31T20:00:51.883 DEBUG:boto:path=/pklxkd-13/obj 2023-05-31T20:00:51.883 DEBUG:boto:auth_path=/pklxkd-13/obj 2023-05-31T20:00:51.883 DEBUG:boto:path=/pklxkd-13/obj?versionId=aS0yzteNN24E5q0Twvp.6Z7sdzsi2hw 2023-05-31T20:00:51.884 DEBUG:boto:auth_path=/pklxkd-13/obj?versionId=aS0yzteNN24E5q0Twvp.6Z7sdzsi2hw 2023-05-31T20:00:51.884 DEBUG:boto:Method: GET 2023-05-31T20:00:51.885 DEBUG:boto:Path: /pklxkd-13/obj?versionId=aS0yzteNN24E5q0Twvp.6Z7sdzsi2hw 2023-05-31T20:00:51.885 DEBUG:boto:Data: 2023-05-31T20:00:51.885 DEBUG:boto:Headers: {} 2023-05-31T20:00:51.886 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:00:51.886 DEBUG:boto:Port: 8001 2023-05-31T20:00:51.886 DEBUG:boto:Params: {} 2023-05-31T20:00:51.887 DEBUG:boto:Token: None 2023-05-31T20:00:51.887 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:00:51 GMT /pklxkd-13/obj?versionId=aS0yzteNN24E5q0Twvp.6Z7sdzsi2hw 2023-05-31T20:00:51.888 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:Hn2BkMesftYSzPTpa++gFVvfzXo= 2023-05-31T20:00:51.888 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:00:51 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:Hn2BkMesftYSzPTpa++gFVvfzXo=', 'Content-Length': '0'} 2023-05-31T20:00:51.890 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:00:07 GMT'), ('x-amz-version-id', 'aS0yzteNN24E5q0Twvp.6Z7sdzsi2hw'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000001daff3591e638479-006477a773-1112-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:00:51 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:00:51.891 DEBUG:boto:path=/pklxkd-13/obj 2023-05-31T20:00:51.891 DEBUG:boto:auth_path=/pklxkd-13/obj 2023-05-31T20:00:51.892 DEBUG:boto:path=/pklxkd-13/obj?versionId=aS0yzteNN24E5q0Twvp.6Z7sdzsi2hw 2023-05-31T20:00:51.892 DEBUG:boto:auth_path=/pklxkd-13/obj?versionId=aS0yzteNN24E5q0Twvp.6Z7sdzsi2hw 2023-05-31T20:00:51.892 DEBUG:boto:Method: GET 2023-05-31T20:00:51.893 DEBUG:boto:Path: /pklxkd-13/obj?versionId=aS0yzteNN24E5q0Twvp.6Z7sdzsi2hw 2023-05-31T20:00:51.893 DEBUG:boto:Data: 2023-05-31T20:00:51.894 DEBUG:boto:Headers: {} 2023-05-31T20:00:51.894 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:00:51.894 DEBUG:boto:Port: 8000 2023-05-31T20:00:51.895 DEBUG:boto:Params: {} 2023-05-31T20:00:51.895 DEBUG:boto:Token: None 2023-05-31T20:00:51.895 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:00:51 GMT /pklxkd-13/obj?versionId=aS0yzteNN24E5q0Twvp.6Z7sdzsi2hw 2023-05-31T20:00:51.896 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:Hn2BkMesftYSzPTpa++gFVvfzXo= 2023-05-31T20:00:51.896 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:00:51 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:Hn2BkMesftYSzPTpa++gFVvfzXo=', 'Content-Length': '0'} 2023-05-31T20:00:51.898 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:00:07 GMT'), ('x-amz-version-id', 'aS0yzteNN24E5q0Twvp.6Z7sdzsi2hw'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000c80f563e2d94059e-006477a773-110e-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:00:51 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:00:51.899 DEBUG:boto:path=/pklxkd-13/ 2023-05-31T20:00:51.899 DEBUG:boto:auth_path=/pklxkd-13/ 2023-05-31T20:00:51.900 DEBUG:boto:path=/pklxkd-13/?prefix=obj 2023-05-31T20:00:51.900 DEBUG:boto:auth_path=/pklxkd-13/?prefix=obj 2023-05-31T20:00:51.900 DEBUG:boto:Method: GET 2023-05-31T20:00:51.901 DEBUG:boto:Path: /pklxkd-13/?prefix=obj 2023-05-31T20:00:51.901 DEBUG:boto:Data: 2023-05-31T20:00:51.901 DEBUG:boto:Headers: {} 2023-05-31T20:00:51.902 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:00:51.902 DEBUG:boto:Port: 8001 2023-05-31T20:00:51.903 DEBUG:boto:Params: {} 2023-05-31T20:00:51.903 DEBUG:boto:Token: None 2023-05-31T20:00:51.903 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:00:51 GMT /pklxkd-13/ 2023-05-31T20:00:51.904 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:8nY7ypnD0gQg08MwmrHH+KcmiAo= 2023-05-31T20:00:51.904 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:00:51 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:8nY7ypnD0gQg08MwmrHH+KcmiAo=', 'Content-Length': '0'} 2023-05-31T20:00:51.907 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000377541a5087d24e0-006477a773-1112-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:00:51 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:00:51.907 DEBUG:boto:b'pklxkd-13obj1000falseobj2023-05-31T20:00:08.058Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:00:51.908 DEBUG:boto:path=/pklxkd-13/ 2023-05-31T20:00:51.909 DEBUG:boto:auth_path=/pklxkd-13/ 2023-05-31T20:00:51.909 DEBUG:boto:path=/pklxkd-13/?prefix=obj 2023-05-31T20:00:51.909 DEBUG:boto:auth_path=/pklxkd-13/?prefix=obj 2023-05-31T20:00:51.910 DEBUG:boto:Method: GET 2023-05-31T20:00:51.910 DEBUG:boto:Path: /pklxkd-13/?prefix=obj 2023-05-31T20:00:51.910 DEBUG:boto:Data: 2023-05-31T20:00:51.911 DEBUG:boto:Headers: {} 2023-05-31T20:00:51.911 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:00:51.911 DEBUG:boto:Port: 8000 2023-05-31T20:00:51.912 DEBUG:boto:Params: {} 2023-05-31T20:00:51.912 DEBUG:boto:Token: None 2023-05-31T20:00:51.913 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:00:51 GMT /pklxkd-13/ 2023-05-31T20:00:51.913 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:8nY7ypnD0gQg08MwmrHH+KcmiAo= 2023-05-31T20:00:51.913 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:00:51 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:8nY7ypnD0gQg08MwmrHH+KcmiAo=', 'Content-Length': '0'} 2023-05-31T20:00:51.916 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a119be17bdb9abba-006477a773-110e-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:00:51 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:00:51.917 DEBUG:boto:b'pklxkd-13obj1000falseobj2023-05-31T20:00:08.058Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:00:51.917 DEBUG:boto:path=/pklxkd-13/obj 2023-05-31T20:00:51.918 DEBUG:boto:auth_path=/pklxkd-13/obj 2023-05-31T20:00:51.918 DEBUG:boto:Method: HEAD 2023-05-31T20:00:51.918 DEBUG:boto:Path: /pklxkd-13/obj 2023-05-31T20:00:51.919 DEBUG:boto:Data: 2023-05-31T20:00:51.919 DEBUG:boto:Headers: {} 2023-05-31T20:00:51.919 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:00:51.920 DEBUG:boto:Port: 8001 2023-05-31T20:00:51.920 DEBUG:boto:Params: {} 2023-05-31T20:00:51.921 DEBUG:boto:Token: None 2023-05-31T20:00:51.924 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:00:51 GMT /pklxkd-13/obj 2023-05-31T20:00:51.925 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:sLcfXAyGTtS5A8hAUE3sGSDaL70= 2023-05-31T20:00:51.925 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:00:51 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:sLcfXAyGTtS5A8hAUE3sGSDaL70=', 'Content-Length': '0'} 2023-05-31T20:00:51.928 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:00:08 GMT'), ('x-amz-version-id', '.K0qHk3QiH4acREzLteeQLJmFzn4x-1'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000cbc11e6c386770b4-006477a773-1112-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:00:51 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:00:51.929 DEBUG:boto:path=/pklxkd-13/obj 2023-05-31T20:00:51.929 DEBUG:boto:auth_path=/pklxkd-13/obj 2023-05-31T20:00:51.930 DEBUG:boto:Method: HEAD 2023-05-31T20:00:51.930 DEBUG:boto:Path: /pklxkd-13/obj 2023-05-31T20:00:51.930 DEBUG:boto:Data: 2023-05-31T20:00:51.931 DEBUG:boto:Headers: {} 2023-05-31T20:00:51.931 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:00:51.931 DEBUG:boto:Port: 8000 2023-05-31T20:00:51.932 DEBUG:boto:Params: {} 2023-05-31T20:00:51.932 DEBUG:boto:Token: None 2023-05-31T20:00:51.933 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:00:51 GMT /pklxkd-13/obj 2023-05-31T20:00:51.933 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:sLcfXAyGTtS5A8hAUE3sGSDaL70= 2023-05-31T20:00:51.933 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:00:51 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:sLcfXAyGTtS5A8hAUE3sGSDaL70=', 'Content-Length': '0'} 2023-05-31T20:00:51.937 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:00:08 GMT'), ('x-amz-version-id', '.K0qHk3QiH4acREzLteeQLJmFzn4x-1'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000000688badb5b245ca7-006477a773-110e-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:00:51 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:00:51.937 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2023-05-31T20:00:51.938 DEBUG:boto:path=/pklxkd-13/obj 2023-05-31T20:00:51.938 DEBUG:boto:auth_path=/pklxkd-13/obj 2023-05-31T20:00:51.938 DEBUG:boto:path=/pklxkd-13/obj?versionId=.K0qHk3QiH4acREzLteeQLJmFzn4x-1 2023-05-31T20:00:51.939 DEBUG:boto:auth_path=/pklxkd-13/obj?versionId=.K0qHk3QiH4acREzLteeQLJmFzn4x-1 2023-05-31T20:00:51.939 DEBUG:boto:Method: GET 2023-05-31T20:00:51.939 DEBUG:boto:Path: /pklxkd-13/obj?versionId=.K0qHk3QiH4acREzLteeQLJmFzn4x-1 2023-05-31T20:00:51.940 DEBUG:boto:Data: 2023-05-31T20:00:51.940 DEBUG:boto:Headers: {} 2023-05-31T20:00:51.941 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:00:51.941 DEBUG:boto:Port: 8001 2023-05-31T20:00:51.941 DEBUG:boto:Params: {} 2023-05-31T20:00:51.942 DEBUG:boto:Token: None 2023-05-31T20:00:51.942 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:00:51 GMT /pklxkd-13/obj?versionId=.K0qHk3QiH4acREzLteeQLJmFzn4x-1 2023-05-31T20:00:51.943 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:MWWK358ZEpKN0sicgroa3DvTmS4= 2023-05-31T20:00:51.943 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:00:51 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:MWWK358ZEpKN0sicgroa3DvTmS4=', 'Content-Length': '0'} 2023-05-31T20:00:51.945 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:00:08 GMT'), ('x-amz-version-id', '.K0qHk3QiH4acREzLteeQLJmFzn4x-1'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000009093cac8eb7fd8f1-006477a773-1112-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:00:51 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:00:51.988 DEBUG:boto:path=/pklxkd-13/obj 2023-05-31T20:00:51.989 DEBUG:boto:auth_path=/pklxkd-13/obj 2023-05-31T20:00:51.990 DEBUG:boto:path=/pklxkd-13/obj?versionId=.K0qHk3QiH4acREzLteeQLJmFzn4x-1 2023-05-31T20:00:51.990 DEBUG:boto:auth_path=/pklxkd-13/obj?versionId=.K0qHk3QiH4acREzLteeQLJmFzn4x-1 2023-05-31T20:00:51.991 DEBUG:boto:Method: GET 2023-05-31T20:00:51.992 DEBUG:boto:Path: /pklxkd-13/obj?versionId=.K0qHk3QiH4acREzLteeQLJmFzn4x-1 2023-05-31T20:00:51.993 DEBUG:boto:Data: 2023-05-31T20:00:51.993 DEBUG:boto:Headers: {} 2023-05-31T20:00:51.993 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:00:51.994 DEBUG:boto:Port: 8000 2023-05-31T20:00:51.994 DEBUG:boto:Params: {} 2023-05-31T20:00:51.995 DEBUG:boto:Token: None 2023-05-31T20:00:51.995 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:00:51 GMT /pklxkd-13/obj?versionId=.K0qHk3QiH4acREzLteeQLJmFzn4x-1 2023-05-31T20:00:51.996 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:MWWK358ZEpKN0sicgroa3DvTmS4= 2023-05-31T20:00:51.996 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:00:51 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:MWWK358ZEpKN0sicgroa3DvTmS4=', 'Content-Length': '0'} 2023-05-31T20:00:51.998 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:00:08 GMT'), ('x-amz-version-id', '.K0qHk3QiH4acREzLteeQLJmFzn4x-1'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000c1bac67b9391e967-006477a773-110e-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:00:51 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:00:51.999 DEBUG:boto:path=/pklxkd-13/ 2023-05-31T20:00:51.999 DEBUG:boto:auth_path=/pklxkd-13/ 2023-05-31T20:00:52.000 DEBUG:boto:path=/pklxkd-13/?prefix=obj 2023-05-31T20:00:52.000 DEBUG:boto:auth_path=/pklxkd-13/?prefix=obj 2023-05-31T20:00:52.000 DEBUG:boto:Method: GET 2023-05-31T20:00:52.001 DEBUG:boto:Path: /pklxkd-13/?prefix=obj 2023-05-31T20:00:52.001 DEBUG:boto:Data: 2023-05-31T20:00:52.002 DEBUG:boto:Headers: {} 2023-05-31T20:00:52.003 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:00:52.004 DEBUG:boto:Port: 8001 2023-05-31T20:00:52.005 DEBUG:boto:Params: {} 2023-05-31T20:00:52.005 DEBUG:boto:Token: None 2023-05-31T20:00:52.006 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:00:52 GMT /pklxkd-13/ 2023-05-31T20:00:52.007 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:FGXZKT+GD0K7Ay6Sb8IIHLqRZf8= 2023-05-31T20:00:52.008 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:00:52 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:FGXZKT+GD0K7Ay6Sb8IIHLqRZf8=', 'Content-Length': '0'} 2023-05-31T20:00:52.010 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000005123e12050940eb8-006477a774-1112-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:00:52 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:00:52.011 DEBUG:boto:b'pklxkd-13obj1000falseobj2023-05-31T20:00:08.058Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:00:52.012 DEBUG:boto:path=/pklxkd-13/ 2023-05-31T20:00:52.012 DEBUG:boto:auth_path=/pklxkd-13/ 2023-05-31T20:00:52.012 DEBUG:boto:path=/pklxkd-13/?prefix=obj 2023-05-31T20:00:52.013 DEBUG:boto:auth_path=/pklxkd-13/?prefix=obj 2023-05-31T20:00:52.013 DEBUG:boto:Method: GET 2023-05-31T20:00:52.014 DEBUG:boto:Path: /pklxkd-13/?prefix=obj 2023-05-31T20:00:52.014 DEBUG:boto:Data: 2023-05-31T20:00:52.014 DEBUG:boto:Headers: {} 2023-05-31T20:00:52.015 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:00:52.015 DEBUG:boto:Port: 8000 2023-05-31T20:00:52.015 DEBUG:boto:Params: {} 2023-05-31T20:00:52.016 DEBUG:boto:Token: None 2023-05-31T20:00:52.016 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:00:52 GMT /pklxkd-13/ 2023-05-31T20:00:52.017 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:FGXZKT+GD0K7Ay6Sb8IIHLqRZf8= 2023-05-31T20:00:52.017 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:00:52 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:FGXZKT+GD0K7Ay6Sb8IIHLqRZf8=', 'Content-Length': '0'} 2023-05-31T20:00:52.020 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000006679e2cdfe4c694-006477a774-110e-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:00:52 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:00:52.020 DEBUG:boto:b'pklxkd-13obj1000falseobj2023-05-31T20:00:08.058Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:00:52.021 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2023-05-31T20:00:52.021 DEBUG:boto:path=/pklxkd-13/obj 2023-05-31T20:00:52.022 DEBUG:boto:auth_path=/pklxkd-13/obj 2023-05-31T20:00:52.022 DEBUG:boto:path=/pklxkd-13/obj?versionId=K5b8GHNb3Mn1saWR47Y7KNPxakeIWEc 2023-05-31T20:00:52.023 DEBUG:boto:auth_path=/pklxkd-13/obj?versionId=K5b8GHNb3Mn1saWR47Y7KNPxakeIWEc 2023-05-31T20:00:52.023 DEBUG:boto:Method: GET 2023-05-31T20:00:52.023 DEBUG:boto:Path: /pklxkd-13/obj?versionId=K5b8GHNb3Mn1saWR47Y7KNPxakeIWEc 2023-05-31T20:00:52.024 DEBUG:boto:Data: 2023-05-31T20:00:52.024 DEBUG:boto:Headers: {} 2023-05-31T20:00:52.024 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:00:52.025 DEBUG:boto:Port: 8001 2023-05-31T20:00:52.025 DEBUG:boto:Params: {} 2023-05-31T20:00:52.026 DEBUG:boto:Token: None 2023-05-31T20:00:52.026 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:00:52 GMT /pklxkd-13/obj?versionId=K5b8GHNb3Mn1saWR47Y7KNPxakeIWEc 2023-05-31T20:00:52.027 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:96pvbjUrRkcceZOwMfGIvNyuwHo= 2023-05-31T20:00:52.027 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:00:52 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:96pvbjUrRkcceZOwMfGIvNyuwHo=', 'Content-Length': '0'} 2023-05-31T20:00:52.029 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:00:07 GMT'), ('x-amz-version-id', 'K5b8GHNb3Mn1saWR47Y7KNPxakeIWEc'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000001d01c007c7c5f9b7-006477a774-1112-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:00:52 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:00:52.030 DEBUG:boto:path=/pklxkd-13/obj 2023-05-31T20:00:52.030 DEBUG:boto:auth_path=/pklxkd-13/obj 2023-05-31T20:00:52.031 DEBUG:boto:path=/pklxkd-13/obj?versionId=K5b8GHNb3Mn1saWR47Y7KNPxakeIWEc 2023-05-31T20:00:52.031 DEBUG:boto:auth_path=/pklxkd-13/obj?versionId=K5b8GHNb3Mn1saWR47Y7KNPxakeIWEc 2023-05-31T20:00:52.032 DEBUG:boto:Method: GET 2023-05-31T20:00:52.032 DEBUG:boto:Path: /pklxkd-13/obj?versionId=K5b8GHNb3Mn1saWR47Y7KNPxakeIWEc 2023-05-31T20:00:52.032 DEBUG:boto:Data: 2023-05-31T20:00:52.033 DEBUG:boto:Headers: {} 2023-05-31T20:00:52.033 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:00:52.034 DEBUG:boto:Port: 8000 2023-05-31T20:00:52.034 DEBUG:boto:Params: {} 2023-05-31T20:00:52.035 DEBUG:boto:Token: None 2023-05-31T20:00:52.035 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:00:52 GMT /pklxkd-13/obj?versionId=K5b8GHNb3Mn1saWR47Y7KNPxakeIWEc 2023-05-31T20:00:52.036 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:96pvbjUrRkcceZOwMfGIvNyuwHo= 2023-05-31T20:00:52.036 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:00:52 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:96pvbjUrRkcceZOwMfGIvNyuwHo=', 'Content-Length': '0'} 2023-05-31T20:00:52.038 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:00:07 GMT'), ('x-amz-version-id', 'K5b8GHNb3Mn1saWR47Y7KNPxakeIWEc'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000dbce74322f26e00d-006477a774-110e-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:00:52 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:00:52.039 DEBUG:boto:path=/pklxkd-13/obj 2023-05-31T20:00:52.039 DEBUG:boto:auth_path=/pklxkd-13/obj 2023-05-31T20:00:52.040 DEBUG:boto:path=/pklxkd-13/obj?versionId=K5b8GHNb3Mn1saWR47Y7KNPxakeIWEc 2023-05-31T20:00:52.040 DEBUG:boto:auth_path=/pklxkd-13/obj?versionId=K5b8GHNb3Mn1saWR47Y7KNPxakeIWEc 2023-05-31T20:00:52.040 DEBUG:boto:Method: HEAD 2023-05-31T20:00:52.041 DEBUG:boto:Path: /pklxkd-13/obj?versionId=K5b8GHNb3Mn1saWR47Y7KNPxakeIWEc 2023-05-31T20:00:52.042 DEBUG:boto:Data: 2023-05-31T20:00:52.042 DEBUG:boto:Headers: {} 2023-05-31T20:00:52.043 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:00:52.044 DEBUG:boto:Port: 8001 2023-05-31T20:00:52.045 DEBUG:boto:Params: {} 2023-05-31T20:00:52.046 DEBUG:boto:Token: None 2023-05-31T20:00:52.047 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:00:52 GMT /pklxkd-13/obj?versionId=K5b8GHNb3Mn1saWR47Y7KNPxakeIWEc 2023-05-31T20:00:52.047 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:410J2I5m/ashIX6cPupbr60rDrI= 2023-05-31T20:00:52.048 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:00:52 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:410J2I5m/ashIX6cPupbr60rDrI=', 'Content-Length': '0'} 2023-05-31T20:00:52.051 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:00:07 GMT'), ('x-amz-version-id', 'K5b8GHNb3Mn1saWR47Y7KNPxakeIWEc'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000004422eac6103c7bb8-006477a774-1112-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:00:52 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:00:52.052 DEBUG:boto:path=/pklxkd-13/obj 2023-05-31T20:00:52.052 DEBUG:boto:auth_path=/pklxkd-13/obj 2023-05-31T20:00:52.053 DEBUG:boto:path=/pklxkd-13/obj?versionId=K5b8GHNb3Mn1saWR47Y7KNPxakeIWEc 2023-05-31T20:00:52.053 DEBUG:boto:auth_path=/pklxkd-13/obj?versionId=K5b8GHNb3Mn1saWR47Y7KNPxakeIWEc 2023-05-31T20:00:52.053 DEBUG:boto:Method: HEAD 2023-05-31T20:00:52.054 DEBUG:boto:Path: /pklxkd-13/obj?versionId=K5b8GHNb3Mn1saWR47Y7KNPxakeIWEc 2023-05-31T20:00:52.054 DEBUG:boto:Data: 2023-05-31T20:00:52.055 DEBUG:boto:Headers: {} 2023-05-31T20:00:52.055 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:00:52.055 DEBUG:boto:Port: 8000 2023-05-31T20:00:52.056 DEBUG:boto:Params: {} 2023-05-31T20:00:52.056 DEBUG:boto:Token: None 2023-05-31T20:00:52.057 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:00:52 GMT /pklxkd-13/obj?versionId=K5b8GHNb3Mn1saWR47Y7KNPxakeIWEc 2023-05-31T20:00:52.057 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:410J2I5m/ashIX6cPupbr60rDrI= 2023-05-31T20:00:52.057 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:00:52 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:410J2I5m/ashIX6cPupbr60rDrI=', 'Content-Length': '0'} 2023-05-31T20:00:52.059 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:00:07 GMT'), ('x-amz-version-id', 'K5b8GHNb3Mn1saWR47Y7KNPxakeIWEc'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000c0ca80b35a6822fb-006477a774-110e-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:00:52 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:00:52.060 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2023-05-31T20:00:52.060 DEBUG:boto:path=/pklxkd-13/obj 2023-05-31T20:00:52.061 DEBUG:boto:auth_path=/pklxkd-13/obj 2023-05-31T20:00:52.061 DEBUG:boto:path=/pklxkd-13/obj?versionId=K5b8GHNb3Mn1saWR47Y7KNPxakeIWEc 2023-05-31T20:00:52.061 DEBUG:boto:auth_path=/pklxkd-13/obj?versionId=K5b8GHNb3Mn1saWR47Y7KNPxakeIWEc 2023-05-31T20:00:52.062 DEBUG:boto:Method: GET 2023-05-31T20:00:52.062 DEBUG:boto:Path: /pklxkd-13/obj?versionId=K5b8GHNb3Mn1saWR47Y7KNPxakeIWEc 2023-05-31T20:00:52.062 DEBUG:boto:Data: 2023-05-31T20:00:52.063 DEBUG:boto:Headers: {} 2023-05-31T20:00:52.063 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:00:52.063 DEBUG:boto:Port: 8001 2023-05-31T20:00:52.064 DEBUG:boto:Params: {} 2023-05-31T20:00:52.064 DEBUG:boto:Token: None 2023-05-31T20:00:52.065 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:00:52 GMT /pklxkd-13/obj?versionId=K5b8GHNb3Mn1saWR47Y7KNPxakeIWEc 2023-05-31T20:00:52.065 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:96pvbjUrRkcceZOwMfGIvNyuwHo= 2023-05-31T20:00:52.065 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:00:52 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:96pvbjUrRkcceZOwMfGIvNyuwHo=', 'Content-Length': '0'} 2023-05-31T20:00:52.067 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:00:07 GMT'), ('x-amz-version-id', 'K5b8GHNb3Mn1saWR47Y7KNPxakeIWEc'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000008303dbf758af4c53-006477a774-1112-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:00:52 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:00:52.068 DEBUG:boto:path=/pklxkd-13/obj 2023-05-31T20:00:52.068 DEBUG:boto:auth_path=/pklxkd-13/obj 2023-05-31T20:00:52.069 DEBUG:boto:path=/pklxkd-13/obj?versionId=K5b8GHNb3Mn1saWR47Y7KNPxakeIWEc 2023-05-31T20:00:52.069 DEBUG:boto:auth_path=/pklxkd-13/obj?versionId=K5b8GHNb3Mn1saWR47Y7KNPxakeIWEc 2023-05-31T20:00:52.069 DEBUG:boto:Method: GET 2023-05-31T20:00:52.070 DEBUG:boto:Path: /pklxkd-13/obj?versionId=K5b8GHNb3Mn1saWR47Y7KNPxakeIWEc 2023-05-31T20:00:52.070 DEBUG:boto:Data: 2023-05-31T20:00:52.071 DEBUG:boto:Headers: {} 2023-05-31T20:00:52.071 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:00:52.072 DEBUG:boto:Port: 8000 2023-05-31T20:00:52.072 DEBUG:boto:Params: {} 2023-05-31T20:00:52.072 DEBUG:boto:Token: None 2023-05-31T20:00:52.073 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:00:52 GMT /pklxkd-13/obj?versionId=K5b8GHNb3Mn1saWR47Y7KNPxakeIWEc 2023-05-31T20:00:52.073 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:96pvbjUrRkcceZOwMfGIvNyuwHo= 2023-05-31T20:00:52.074 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:00:52 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:96pvbjUrRkcceZOwMfGIvNyuwHo=', 'Content-Length': '0'} 2023-05-31T20:00:52.077 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:00:07 GMT'), ('x-amz-version-id', 'K5b8GHNb3Mn1saWR47Y7KNPxakeIWEc'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000061b093941891113a-006477a774-110e-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:00:52 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:00:52.078 DEBUG:boto:path=/pklxkd-13/ 2023-05-31T20:00:52.078 DEBUG:boto:auth_path=/pklxkd-13/ 2023-05-31T20:00:52.079 DEBUG:boto:path=/pklxkd-13/?prefix=obj 2023-05-31T20:00:52.079 DEBUG:boto:auth_path=/pklxkd-13/?prefix=obj 2023-05-31T20:00:52.079 DEBUG:boto:Method: GET 2023-05-31T20:00:52.080 DEBUG:boto:Path: /pklxkd-13/?prefix=obj 2023-05-31T20:00:52.080 DEBUG:boto:Data: 2023-05-31T20:00:52.080 DEBUG:boto:Headers: {} 2023-05-31T20:00:52.081 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:00:52.081 DEBUG:boto:Port: 8001 2023-05-31T20:00:52.081 DEBUG:boto:Params: {} 2023-05-31T20:00:52.082 DEBUG:boto:Token: None 2023-05-31T20:00:52.082 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:00:52 GMT /pklxkd-13/ 2023-05-31T20:00:52.083 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:FGXZKT+GD0K7Ay6Sb8IIHLqRZf8= 2023-05-31T20:00:52.083 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:00:52 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:FGXZKT+GD0K7Ay6Sb8IIHLqRZf8=', 'Content-Length': '0'} 2023-05-31T20:00:52.086 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000079bd304d80696404-006477a774-1112-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:00:52 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:00:52.086 DEBUG:boto:b'pklxkd-13obj1000falseobj2023-05-31T20:00:08.058Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:00:52.087 DEBUG:boto:path=/pklxkd-13/ 2023-05-31T20:00:52.087 DEBUG:boto:auth_path=/pklxkd-13/ 2023-05-31T20:00:52.087 DEBUG:boto:path=/pklxkd-13/?prefix=obj 2023-05-31T20:00:52.088 DEBUG:boto:auth_path=/pklxkd-13/?prefix=obj 2023-05-31T20:00:52.088 DEBUG:boto:Method: GET 2023-05-31T20:00:52.089 DEBUG:boto:Path: /pklxkd-13/?prefix=obj 2023-05-31T20:00:52.089 DEBUG:boto:Data: 2023-05-31T20:00:52.089 DEBUG:boto:Headers: {} 2023-05-31T20:00:52.090 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:00:52.090 DEBUG:boto:Port: 8000 2023-05-31T20:00:52.090 DEBUG:boto:Params: {} 2023-05-31T20:00:52.091 DEBUG:boto:Token: None 2023-05-31T20:00:52.091 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:00:52 GMT /pklxkd-13/ 2023-05-31T20:00:52.092 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:FGXZKT+GD0K7Ay6Sb8IIHLqRZf8= 2023-05-31T20:00:52.092 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:00:52 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:FGXZKT+GD0K7Ay6Sb8IIHLqRZf8=', 'Content-Length': '0'} 2023-05-31T20:00:52.095 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000003967d3c7b9634dfe-006477a774-110e-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:00:52 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:00:52.095 DEBUG:boto:b'pklxkd-13obj1000falseobj2023-05-31T20:00:08.058Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:00:52.096 DEBUG:boto:path=/pklxkd-13/obj 2023-05-31T20:00:52.096 DEBUG:boto:auth_path=/pklxkd-13/obj 2023-05-31T20:00:52.097 DEBUG:boto:Method: HEAD 2023-05-31T20:00:52.097 DEBUG:boto:Path: /pklxkd-13/obj 2023-05-31T20:00:52.097 DEBUG:boto:Data: 2023-05-31T20:00:52.098 DEBUG:boto:Headers: {} 2023-05-31T20:00:52.098 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:00:52.098 DEBUG:boto:Port: 8001 2023-05-31T20:00:52.099 DEBUG:boto:Params: {} 2023-05-31T20:00:52.099 DEBUG:boto:Token: None 2023-05-31T20:00:52.099 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:00:52 GMT /pklxkd-13/obj 2023-05-31T20:00:52.100 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:47OSVk8UFhfayV7BvQVLHanT+oY= 2023-05-31T20:00:52.100 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:00:52 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:47OSVk8UFhfayV7BvQVLHanT+oY=', 'Content-Length': '0'} 2023-05-31T20:00:52.103 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:00:08 GMT'), ('x-amz-version-id', '.K0qHk3QiH4acREzLteeQLJmFzn4x-1'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000a61422db4faadbd8-006477a774-1112-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:00:52 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:00:52.104 DEBUG:boto:path=/pklxkd-13/obj 2023-05-31T20:00:52.104 DEBUG:boto:auth_path=/pklxkd-13/obj 2023-05-31T20:00:52.105 DEBUG:boto:Method: HEAD 2023-05-31T20:00:52.105 DEBUG:boto:Path: /pklxkd-13/obj 2023-05-31T20:00:52.105 DEBUG:boto:Data: 2023-05-31T20:00:52.106 DEBUG:boto:Headers: {} 2023-05-31T20:00:52.106 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:00:52.106 DEBUG:boto:Port: 8000 2023-05-31T20:00:52.107 DEBUG:boto:Params: {} 2023-05-31T20:00:52.107 DEBUG:boto:Token: None 2023-05-31T20:00:52.107 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:00:52 GMT /pklxkd-13/obj 2023-05-31T20:00:52.108 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:47OSVk8UFhfayV7BvQVLHanT+oY= 2023-05-31T20:00:52.108 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:00:52 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:47OSVk8UFhfayV7BvQVLHanT+oY=', 'Content-Length': '0'} 2023-05-31T20:00:52.111 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:00:08 GMT'), ('x-amz-version-id', '.K0qHk3QiH4acREzLteeQLJmFzn4x-1'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000400b7b05634666bb-006477a774-110e-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:00:52 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:00:52.112 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2023-05-31T20:00:52.112 DEBUG:boto:path=/pklxkd-13/obj 2023-05-31T20:00:52.112 DEBUG:boto:auth_path=/pklxkd-13/obj 2023-05-31T20:00:52.113 DEBUG:boto:path=/pklxkd-13/obj?versionId=.K0qHk3QiH4acREzLteeQLJmFzn4x-1 2023-05-31T20:00:52.113 DEBUG:boto:auth_path=/pklxkd-13/obj?versionId=.K0qHk3QiH4acREzLteeQLJmFzn4x-1 2023-05-31T20:00:52.113 DEBUG:boto:Method: GET 2023-05-31T20:00:52.114 DEBUG:boto:Path: /pklxkd-13/obj?versionId=.K0qHk3QiH4acREzLteeQLJmFzn4x-1 2023-05-31T20:00:52.114 DEBUG:boto:Data: 2023-05-31T20:00:52.114 DEBUG:boto:Headers: {} 2023-05-31T20:00:52.115 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:00:52.115 DEBUG:boto:Port: 8001 2023-05-31T20:00:52.116 DEBUG:boto:Params: {} 2023-05-31T20:00:52.116 DEBUG:boto:Token: None 2023-05-31T20:00:52.116 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:00:52 GMT /pklxkd-13/obj?versionId=.K0qHk3QiH4acREzLteeQLJmFzn4x-1 2023-05-31T20:00:52.117 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:qzlVetv8hVapoLo7pVuvQqLVOm4= 2023-05-31T20:00:52.117 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:00:52 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:qzlVetv8hVapoLo7pVuvQqLVOm4=', 'Content-Length': '0'} 2023-05-31T20:00:52.119 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:00:08 GMT'), ('x-amz-version-id', '.K0qHk3QiH4acREzLteeQLJmFzn4x-1'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000c74e033a0836f9b0-006477a774-1112-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:00:52 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:00:52.160 DEBUG:boto:path=/pklxkd-13/obj 2023-05-31T20:00:52.161 DEBUG:boto:auth_path=/pklxkd-13/obj 2023-05-31T20:00:52.161 DEBUG:boto:path=/pklxkd-13/obj?versionId=.K0qHk3QiH4acREzLteeQLJmFzn4x-1 2023-05-31T20:00:52.162 DEBUG:boto:auth_path=/pklxkd-13/obj?versionId=.K0qHk3QiH4acREzLteeQLJmFzn4x-1 2023-05-31T20:00:52.162 DEBUG:boto:Method: GET 2023-05-31T20:00:52.162 DEBUG:boto:Path: /pklxkd-13/obj?versionId=.K0qHk3QiH4acREzLteeQLJmFzn4x-1 2023-05-31T20:00:52.163 DEBUG:boto:Data: 2023-05-31T20:00:52.163 DEBUG:boto:Headers: {} 2023-05-31T20:00:52.163 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:00:52.164 DEBUG:boto:Port: 8000 2023-05-31T20:00:52.164 DEBUG:boto:Params: {} 2023-05-31T20:00:52.164 DEBUG:boto:Token: None 2023-05-31T20:00:52.165 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:00:52 GMT /pklxkd-13/obj?versionId=.K0qHk3QiH4acREzLteeQLJmFzn4x-1 2023-05-31T20:00:52.165 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:qzlVetv8hVapoLo7pVuvQqLVOm4= 2023-05-31T20:00:52.166 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:00:52 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:qzlVetv8hVapoLo7pVuvQqLVOm4=', 'Content-Length': '0'} 2023-05-31T20:00:52.168 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:00:08 GMT'), ('x-amz-version-id', '.K0qHk3QiH4acREzLteeQLJmFzn4x-1'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000c3ebff5065e11693-006477a774-110e-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:00:52 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:00:52.168 DEBUG:boto:path=/pklxkd-13/ 2023-05-31T20:00:52.169 DEBUG:boto:auth_path=/pklxkd-13/ 2023-05-31T20:00:52.169 DEBUG:boto:path=/pklxkd-13/?prefix=obj 2023-05-31T20:00:52.170 DEBUG:boto:auth_path=/pklxkd-13/?prefix=obj 2023-05-31T20:00:52.170 DEBUG:boto:Method: GET 2023-05-31T20:00:52.170 DEBUG:boto:Path: /pklxkd-13/?prefix=obj 2023-05-31T20:00:52.171 DEBUG:boto:Data: 2023-05-31T20:00:52.171 DEBUG:boto:Headers: {} 2023-05-31T20:00:52.171 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:00:52.172 DEBUG:boto:Port: 8001 2023-05-31T20:00:52.172 DEBUG:boto:Params: {} 2023-05-31T20:00:52.172 DEBUG:boto:Token: None 2023-05-31T20:00:52.173 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:00:52 GMT /pklxkd-13/ 2023-05-31T20:00:52.173 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:FGXZKT+GD0K7Ay6Sb8IIHLqRZf8= 2023-05-31T20:00:52.174 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:00:52 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:FGXZKT+GD0K7Ay6Sb8IIHLqRZf8=', 'Content-Length': '0'} 2023-05-31T20:00:52.176 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e0bcd0cf1572017e-006477a774-1112-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:00:52 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:00:52.177 DEBUG:boto:b'pklxkd-13obj1000falseobj2023-05-31T20:00:08.058Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:00:52.177 DEBUG:boto:path=/pklxkd-13/ 2023-05-31T20:00:52.178 DEBUG:boto:auth_path=/pklxkd-13/ 2023-05-31T20:00:52.178 DEBUG:boto:path=/pklxkd-13/?prefix=obj 2023-05-31T20:00:52.178 DEBUG:boto:auth_path=/pklxkd-13/?prefix=obj 2023-05-31T20:00:52.179 DEBUG:boto:Method: GET 2023-05-31T20:00:52.179 DEBUG:boto:Path: /pklxkd-13/?prefix=obj 2023-05-31T20:00:52.180 DEBUG:boto:Data: 2023-05-31T20:00:52.180 DEBUG:boto:Headers: {} 2023-05-31T20:00:52.180 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:00:52.181 DEBUG:boto:Port: 8000 2023-05-31T20:00:52.181 DEBUG:boto:Params: {} 2023-05-31T20:00:52.181 DEBUG:boto:Token: None 2023-05-31T20:00:52.182 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:00:52 GMT /pklxkd-13/ 2023-05-31T20:00:52.182 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:FGXZKT+GD0K7Ay6Sb8IIHLqRZf8= 2023-05-31T20:00:52.183 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:00:52 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:FGXZKT+GD0K7Ay6Sb8IIHLqRZf8=', 'Content-Length': '0'} 2023-05-31T20:00:52.185 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000dab53f2d77bbfc7a-006477a774-110e-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:00:52 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:00:52.186 DEBUG:boto:b'pklxkd-13obj1000falseobj2023-05-31T20:00:08.058Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:00:52.186 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2023-05-31T20:00:52.187 DEBUG:boto:path=/pklxkd-13/obj 2023-05-31T20:00:52.187 DEBUG:boto:auth_path=/pklxkd-13/obj 2023-05-31T20:00:52.187 DEBUG:boto:path=/pklxkd-13/obj?versionId=p5Ybj.5dD2GwjElOd3T2Sa50bcy.UgE 2023-05-31T20:00:52.188 DEBUG:boto:auth_path=/pklxkd-13/obj?versionId=p5Ybj.5dD2GwjElOd3T2Sa50bcy.UgE 2023-05-31T20:00:52.188 DEBUG:boto:Method: GET 2023-05-31T20:00:52.188 DEBUG:boto:Path: /pklxkd-13/obj?versionId=p5Ybj.5dD2GwjElOd3T2Sa50bcy.UgE 2023-05-31T20:00:52.189 DEBUG:boto:Data: 2023-05-31T20:00:52.189 DEBUG:boto:Headers: {} 2023-05-31T20:00:52.189 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:00:52.190 DEBUG:boto:Port: 8001 2023-05-31T20:00:52.190 DEBUG:boto:Params: {} 2023-05-31T20:00:52.191 DEBUG:boto:Token: None 2023-05-31T20:00:52.191 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:00:52 GMT /pklxkd-13/obj?versionId=p5Ybj.5dD2GwjElOd3T2Sa50bcy.UgE 2023-05-31T20:00:52.191 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:dp4CMA4XiDwzd7q47sra4NpN2Dw= 2023-05-31T20:00:52.192 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:00:52 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:dp4CMA4XiDwzd7q47sra4NpN2Dw=', 'Content-Length': '0'} 2023-05-31T20:00:52.194 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:00:07 GMT'), ('x-amz-version-id', 'p5Ybj.5dD2GwjElOd3T2Sa50bcy.UgE'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000bc3e90c06e2b276c-006477a774-1112-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:00:52 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:00:52.194 DEBUG:boto:path=/pklxkd-13/obj 2023-05-31T20:00:52.195 DEBUG:boto:auth_path=/pklxkd-13/obj 2023-05-31T20:00:52.195 DEBUG:boto:path=/pklxkd-13/obj?versionId=p5Ybj.5dD2GwjElOd3T2Sa50bcy.UgE 2023-05-31T20:00:52.195 DEBUG:boto:auth_path=/pklxkd-13/obj?versionId=p5Ybj.5dD2GwjElOd3T2Sa50bcy.UgE 2023-05-31T20:00:52.196 DEBUG:boto:Method: GET 2023-05-31T20:00:52.196 DEBUG:boto:Path: /pklxkd-13/obj?versionId=p5Ybj.5dD2GwjElOd3T2Sa50bcy.UgE 2023-05-31T20:00:52.196 DEBUG:boto:Data: 2023-05-31T20:00:52.197 DEBUG:boto:Headers: {} 2023-05-31T20:00:52.197 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:00:52.197 DEBUG:boto:Port: 8000 2023-05-31T20:00:52.198 DEBUG:boto:Params: {} 2023-05-31T20:00:52.198 DEBUG:boto:Token: None 2023-05-31T20:00:52.199 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:00:52 GMT /pklxkd-13/obj?versionId=p5Ybj.5dD2GwjElOd3T2Sa50bcy.UgE 2023-05-31T20:00:52.199 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:dp4CMA4XiDwzd7q47sra4NpN2Dw= 2023-05-31T20:00:52.199 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:00:52 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:dp4CMA4XiDwzd7q47sra4NpN2Dw=', 'Content-Length': '0'} 2023-05-31T20:00:52.201 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:00:07 GMT'), ('x-amz-version-id', 'p5Ybj.5dD2GwjElOd3T2Sa50bcy.UgE'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000004a6b11eccd2086ca-006477a774-110e-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:00:52 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:00:52.202 DEBUG:boto:path=/pklxkd-13/obj 2023-05-31T20:00:52.202 DEBUG:boto:auth_path=/pklxkd-13/obj 2023-05-31T20:00:52.203 DEBUG:boto:path=/pklxkd-13/obj?versionId=p5Ybj.5dD2GwjElOd3T2Sa50bcy.UgE 2023-05-31T20:00:52.203 DEBUG:boto:auth_path=/pklxkd-13/obj?versionId=p5Ybj.5dD2GwjElOd3T2Sa50bcy.UgE 2023-05-31T20:00:52.203 DEBUG:boto:Method: HEAD 2023-05-31T20:00:52.204 DEBUG:boto:Path: /pklxkd-13/obj?versionId=p5Ybj.5dD2GwjElOd3T2Sa50bcy.UgE 2023-05-31T20:00:52.204 DEBUG:boto:Data: 2023-05-31T20:00:52.204 DEBUG:boto:Headers: {} 2023-05-31T20:00:52.205 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:00:52.205 DEBUG:boto:Port: 8001 2023-05-31T20:00:52.205 DEBUG:boto:Params: {} 2023-05-31T20:00:52.206 DEBUG:boto:Token: None 2023-05-31T20:00:52.206 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:00:52 GMT /pklxkd-13/obj?versionId=p5Ybj.5dD2GwjElOd3T2Sa50bcy.UgE 2023-05-31T20:00:52.207 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:cWfzarSepO0U2AG5+d5UlWJ3oPI= 2023-05-31T20:00:52.207 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:00:52 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:cWfzarSepO0U2AG5+d5UlWJ3oPI=', 'Content-Length': '0'} 2023-05-31T20:00:52.209 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:00:07 GMT'), ('x-amz-version-id', 'p5Ybj.5dD2GwjElOd3T2Sa50bcy.UgE'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000002cae90a0be78a105-006477a774-1112-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:00:52 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:00:52.209 DEBUG:boto:path=/pklxkd-13/obj 2023-05-31T20:00:52.210 DEBUG:boto:auth_path=/pklxkd-13/obj 2023-05-31T20:00:52.210 DEBUG:boto:path=/pklxkd-13/obj?versionId=p5Ybj.5dD2GwjElOd3T2Sa50bcy.UgE 2023-05-31T20:00:52.211 DEBUG:boto:auth_path=/pklxkd-13/obj?versionId=p5Ybj.5dD2GwjElOd3T2Sa50bcy.UgE 2023-05-31T20:00:52.211 DEBUG:boto:Method: HEAD 2023-05-31T20:00:52.211 DEBUG:boto:Path: /pklxkd-13/obj?versionId=p5Ybj.5dD2GwjElOd3T2Sa50bcy.UgE 2023-05-31T20:00:52.212 DEBUG:boto:Data: 2023-05-31T20:00:52.212 DEBUG:boto:Headers: {} 2023-05-31T20:00:52.212 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:00:52.213 DEBUG:boto:Port: 8000 2023-05-31T20:00:52.213 DEBUG:boto:Params: {} 2023-05-31T20:00:52.213 DEBUG:boto:Token: None 2023-05-31T20:00:52.214 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:00:52 GMT /pklxkd-13/obj?versionId=p5Ybj.5dD2GwjElOd3T2Sa50bcy.UgE 2023-05-31T20:00:52.214 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:cWfzarSepO0U2AG5+d5UlWJ3oPI= 2023-05-31T20:00:52.214 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:00:52 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:cWfzarSepO0U2AG5+d5UlWJ3oPI=', 'Content-Length': '0'} 2023-05-31T20:00:52.216 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:00:07 GMT'), ('x-amz-version-id', 'p5Ybj.5dD2GwjElOd3T2Sa50bcy.UgE'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000009febf110f691d17e-006477a774-110e-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:00:52 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:00:52.217 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2023-05-31T20:00:52.217 DEBUG:boto:path=/pklxkd-13/obj 2023-05-31T20:00:52.217 DEBUG:boto:auth_path=/pklxkd-13/obj 2023-05-31T20:00:52.218 DEBUG:boto:path=/pklxkd-13/obj?versionId=p5Ybj.5dD2GwjElOd3T2Sa50bcy.UgE 2023-05-31T20:00:52.218 DEBUG:boto:auth_path=/pklxkd-13/obj?versionId=p5Ybj.5dD2GwjElOd3T2Sa50bcy.UgE 2023-05-31T20:00:52.218 DEBUG:boto:Method: GET 2023-05-31T20:00:52.219 DEBUG:boto:Path: /pklxkd-13/obj?versionId=p5Ybj.5dD2GwjElOd3T2Sa50bcy.UgE 2023-05-31T20:00:52.219 DEBUG:boto:Data: 2023-05-31T20:00:52.220 DEBUG:boto:Headers: {} 2023-05-31T20:00:52.220 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:00:52.220 DEBUG:boto:Port: 8001 2023-05-31T20:00:52.221 DEBUG:boto:Params: {} 2023-05-31T20:00:52.221 DEBUG:boto:Token: None 2023-05-31T20:00:52.221 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:00:52 GMT /pklxkd-13/obj?versionId=p5Ybj.5dD2GwjElOd3T2Sa50bcy.UgE 2023-05-31T20:00:52.222 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:dp4CMA4XiDwzd7q47sra4NpN2Dw= 2023-05-31T20:00:52.222 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:00:52 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:dp4CMA4XiDwzd7q47sra4NpN2Dw=', 'Content-Length': '0'} 2023-05-31T20:00:52.224 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:00:07 GMT'), ('x-amz-version-id', 'p5Ybj.5dD2GwjElOd3T2Sa50bcy.UgE'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000ca326f9e41b57d8f-006477a774-1112-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:00:52 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:00:52.264 DEBUG:boto:path=/pklxkd-13/obj 2023-05-31T20:00:52.265 DEBUG:boto:auth_path=/pklxkd-13/obj 2023-05-31T20:00:52.265 DEBUG:boto:path=/pklxkd-13/obj?versionId=p5Ybj.5dD2GwjElOd3T2Sa50bcy.UgE 2023-05-31T20:00:52.265 DEBUG:boto:auth_path=/pklxkd-13/obj?versionId=p5Ybj.5dD2GwjElOd3T2Sa50bcy.UgE 2023-05-31T20:00:52.266 DEBUG:boto:Method: GET 2023-05-31T20:00:52.266 DEBUG:boto:Path: /pklxkd-13/obj?versionId=p5Ybj.5dD2GwjElOd3T2Sa50bcy.UgE 2023-05-31T20:00:52.267 DEBUG:boto:Data: 2023-05-31T20:00:52.267 DEBUG:boto:Headers: {} 2023-05-31T20:00:52.267 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:00:52.268 DEBUG:boto:Port: 8000 2023-05-31T20:00:52.268 DEBUG:boto:Params: {} 2023-05-31T20:00:52.268 DEBUG:boto:Token: None 2023-05-31T20:00:52.269 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:00:52 GMT /pklxkd-13/obj?versionId=p5Ybj.5dD2GwjElOd3T2Sa50bcy.UgE 2023-05-31T20:00:52.269 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:dp4CMA4XiDwzd7q47sra4NpN2Dw= 2023-05-31T20:00:52.270 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:00:52 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:dp4CMA4XiDwzd7q47sra4NpN2Dw=', 'Content-Length': '0'} 2023-05-31T20:00:52.271 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:00:07 GMT'), ('x-amz-version-id', 'p5Ybj.5dD2GwjElOd3T2Sa50bcy.UgE'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000007e643ee1d5299d37-006477a774-110e-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:00:52 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:00:52.272 DEBUG:boto:path=/pklxkd-13/ 2023-05-31T20:00:52.273 DEBUG:boto:auth_path=/pklxkd-13/ 2023-05-31T20:00:52.273 DEBUG:boto:path=/pklxkd-13/?prefix=obj 2023-05-31T20:00:52.273 DEBUG:boto:auth_path=/pklxkd-13/?prefix=obj 2023-05-31T20:00:52.274 DEBUG:boto:Method: GET 2023-05-31T20:00:52.274 DEBUG:boto:Path: /pklxkd-13/?prefix=obj 2023-05-31T20:00:52.274 DEBUG:boto:Data: 2023-05-31T20:00:52.275 DEBUG:boto:Headers: {} 2023-05-31T20:00:52.275 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:00:52.275 DEBUG:boto:Port: 8001 2023-05-31T20:00:52.276 DEBUG:boto:Params: {} 2023-05-31T20:00:52.276 DEBUG:boto:Token: None 2023-05-31T20:00:52.277 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:00:52 GMT /pklxkd-13/ 2023-05-31T20:00:52.277 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:FGXZKT+GD0K7Ay6Sb8IIHLqRZf8= 2023-05-31T20:00:52.277 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:00:52 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:FGXZKT+GD0K7Ay6Sb8IIHLqRZf8=', 'Content-Length': '0'} 2023-05-31T20:00:52.280 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000cc0c24ffb274d224-006477a774-1112-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:00:52 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:00:52.281 DEBUG:boto:b'pklxkd-13obj1000falseobj2023-05-31T20:00:08.058Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:00:52.281 DEBUG:boto:path=/pklxkd-13/ 2023-05-31T20:00:52.282 DEBUG:boto:auth_path=/pklxkd-13/ 2023-05-31T20:00:52.282 DEBUG:boto:path=/pklxkd-13/?prefix=obj 2023-05-31T20:00:52.282 DEBUG:boto:auth_path=/pklxkd-13/?prefix=obj 2023-05-31T20:00:52.283 DEBUG:boto:Method: GET 2023-05-31T20:00:52.283 DEBUG:boto:Path: /pklxkd-13/?prefix=obj 2023-05-31T20:00:52.283 DEBUG:boto:Data: 2023-05-31T20:00:52.284 DEBUG:boto:Headers: {} 2023-05-31T20:00:52.284 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:00:52.284 DEBUG:boto:Port: 8000 2023-05-31T20:00:52.285 DEBUG:boto:Params: {} 2023-05-31T20:00:52.285 DEBUG:boto:Token: None 2023-05-31T20:00:52.286 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:00:52 GMT /pklxkd-13/ 2023-05-31T20:00:52.286 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:FGXZKT+GD0K7Ay6Sb8IIHLqRZf8= 2023-05-31T20:00:52.286 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:00:52 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:FGXZKT+GD0K7Ay6Sb8IIHLqRZf8=', 'Content-Length': '0'} 2023-05-31T20:00:52.289 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000003d4d53f3ca50e2b5-006477a774-110e-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:00:52 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:00:52.290 DEBUG:boto:b'pklxkd-13obj1000falseobj2023-05-31T20:00:08.058Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:00:52.291 DEBUG:boto:path=/pklxkd-13/obj 2023-05-31T20:00:52.291 DEBUG:boto:auth_path=/pklxkd-13/obj 2023-05-31T20:00:52.291 DEBUG:boto:Method: HEAD 2023-05-31T20:00:52.292 DEBUG:boto:Path: /pklxkd-13/obj 2023-05-31T20:00:52.292 DEBUG:boto:Data: 2023-05-31T20:00:52.292 DEBUG:boto:Headers: {} 2023-05-31T20:00:52.293 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:00:52.293 DEBUG:boto:Port: 8001 2023-05-31T20:00:52.293 DEBUG:boto:Params: {} 2023-05-31T20:00:52.294 DEBUG:boto:Token: None 2023-05-31T20:00:52.294 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:00:52 GMT /pklxkd-13/obj 2023-05-31T20:00:52.295 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:47OSVk8UFhfayV7BvQVLHanT+oY= 2023-05-31T20:00:52.295 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:00:52 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:47OSVk8UFhfayV7BvQVLHanT+oY=', 'Content-Length': '0'} 2023-05-31T20:00:52.298 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:00:08 GMT'), ('x-amz-version-id', '.K0qHk3QiH4acREzLteeQLJmFzn4x-1'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000001a973eaf94cc4a27-006477a774-1112-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:00:52 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:00:52.298 DEBUG:boto:path=/pklxkd-13/obj 2023-05-31T20:00:52.299 DEBUG:boto:auth_path=/pklxkd-13/obj 2023-05-31T20:00:52.299 DEBUG:boto:Method: HEAD 2023-05-31T20:00:52.300 DEBUG:boto:Path: /pklxkd-13/obj 2023-05-31T20:00:52.300 DEBUG:boto:Data: 2023-05-31T20:00:52.300 DEBUG:boto:Headers: {} 2023-05-31T20:00:52.301 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:00:52.301 DEBUG:boto:Port: 8000 2023-05-31T20:00:52.301 DEBUG:boto:Params: {} 2023-05-31T20:00:52.302 DEBUG:boto:Token: None 2023-05-31T20:00:52.302 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:00:52 GMT /pklxkd-13/obj 2023-05-31T20:00:52.302 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:47OSVk8UFhfayV7BvQVLHanT+oY= 2023-05-31T20:00:52.303 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:00:52 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:47OSVk8UFhfayV7BvQVLHanT+oY=', 'Content-Length': '0'} 2023-05-31T20:00:52.305 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:00:08 GMT'), ('x-amz-version-id', '.K0qHk3QiH4acREzLteeQLJmFzn4x-1'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000008d4ac059980bc2d7-006477a774-110e-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:00:52 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:00:52.306 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2023-05-31T20:00:52.306 DEBUG:boto:path=/pklxkd-13/obj 2023-05-31T20:00:52.307 DEBUG:boto:auth_path=/pklxkd-13/obj 2023-05-31T20:00:52.307 DEBUG:boto:path=/pklxkd-13/obj?versionId=.K0qHk3QiH4acREzLteeQLJmFzn4x-1 2023-05-31T20:00:52.308 DEBUG:boto:auth_path=/pklxkd-13/obj?versionId=.K0qHk3QiH4acREzLteeQLJmFzn4x-1 2023-05-31T20:00:52.308 DEBUG:boto:Method: GET 2023-05-31T20:00:52.309 DEBUG:boto:Path: /pklxkd-13/obj?versionId=.K0qHk3QiH4acREzLteeQLJmFzn4x-1 2023-05-31T20:00:52.309 DEBUG:boto:Data: 2023-05-31T20:00:52.309 DEBUG:boto:Headers: {} 2023-05-31T20:00:52.310 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:00:52.310 DEBUG:boto:Port: 8001 2023-05-31T20:00:52.310 DEBUG:boto:Params: {} 2023-05-31T20:00:52.311 DEBUG:boto:Token: None 2023-05-31T20:00:52.311 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:00:52 GMT /pklxkd-13/obj?versionId=.K0qHk3QiH4acREzLteeQLJmFzn4x-1 2023-05-31T20:00:52.312 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:qzlVetv8hVapoLo7pVuvQqLVOm4= 2023-05-31T20:00:52.312 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:00:52 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:qzlVetv8hVapoLo7pVuvQqLVOm4=', 'Content-Length': '0'} 2023-05-31T20:00:52.314 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:00:08 GMT'), ('x-amz-version-id', '.K0qHk3QiH4acREzLteeQLJmFzn4x-1'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000a8cbabfd03e4e876-006477a774-1112-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:00:52 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:00:52.356 DEBUG:boto:path=/pklxkd-13/obj 2023-05-31T20:00:52.357 DEBUG:boto:auth_path=/pklxkd-13/obj 2023-05-31T20:00:52.357 DEBUG:boto:path=/pklxkd-13/obj?versionId=.K0qHk3QiH4acREzLteeQLJmFzn4x-1 2023-05-31T20:00:52.357 DEBUG:boto:auth_path=/pklxkd-13/obj?versionId=.K0qHk3QiH4acREzLteeQLJmFzn4x-1 2023-05-31T20:00:52.358 DEBUG:boto:Method: GET 2023-05-31T20:00:52.358 DEBUG:boto:Path: /pklxkd-13/obj?versionId=.K0qHk3QiH4acREzLteeQLJmFzn4x-1 2023-05-31T20:00:52.358 DEBUG:boto:Data: 2023-05-31T20:00:52.359 DEBUG:boto:Headers: {} 2023-05-31T20:00:52.359 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:00:52.359 DEBUG:boto:Port: 8000 2023-05-31T20:00:52.360 DEBUG:boto:Params: {} 2023-05-31T20:00:52.360 DEBUG:boto:Token: None 2023-05-31T20:00:52.361 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:00:52 GMT /pklxkd-13/obj?versionId=.K0qHk3QiH4acREzLteeQLJmFzn4x-1 2023-05-31T20:00:52.361 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:qzlVetv8hVapoLo7pVuvQqLVOm4= 2023-05-31T20:00:52.362 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:00:52 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:qzlVetv8hVapoLo7pVuvQqLVOm4=', 'Content-Length': '0'} 2023-05-31T20:00:52.363 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:00:08 GMT'), ('x-amz-version-id', '.K0qHk3QiH4acREzLteeQLJmFzn4x-1'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000a654ce247a02fbda-006477a774-110e-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:00:52 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:00:52.364 DEBUG:boto:path=/pklxkd-13/ 2023-05-31T20:00:52.364 DEBUG:boto:auth_path=/pklxkd-13/ 2023-05-31T20:00:52.365 DEBUG:boto:path=/pklxkd-13/?prefix=obj 2023-05-31T20:00:52.365 DEBUG:boto:auth_path=/pklxkd-13/?prefix=obj 2023-05-31T20:00:52.365 DEBUG:boto:Method: GET 2023-05-31T20:00:52.366 DEBUG:boto:Path: /pklxkd-13/?prefix=obj 2023-05-31T20:00:52.366 DEBUG:boto:Data: 2023-05-31T20:00:52.366 DEBUG:boto:Headers: {} 2023-05-31T20:00:52.367 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:00:52.367 DEBUG:boto:Port: 8001 2023-05-31T20:00:52.367 DEBUG:boto:Params: {} 2023-05-31T20:00:52.368 DEBUG:boto:Token: None 2023-05-31T20:00:52.368 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:00:52 GMT /pklxkd-13/ 2023-05-31T20:00:52.369 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:FGXZKT+GD0K7Ay6Sb8IIHLqRZf8= 2023-05-31T20:00:52.369 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:00:52 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:FGXZKT+GD0K7Ay6Sb8IIHLqRZf8=', 'Content-Length': '0'} 2023-05-31T20:00:52.371 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000007403cd847b45f50b-006477a774-1112-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:00:52 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:00:52.372 DEBUG:boto:b'pklxkd-13obj1000falseobj2023-05-31T20:00:08.058Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:00:52.373 DEBUG:boto:path=/pklxkd-13/ 2023-05-31T20:00:52.373 DEBUG:boto:auth_path=/pklxkd-13/ 2023-05-31T20:00:52.373 DEBUG:boto:path=/pklxkd-13/?prefix=obj 2023-05-31T20:00:52.374 DEBUG:boto:auth_path=/pklxkd-13/?prefix=obj 2023-05-31T20:00:52.374 DEBUG:boto:Method: GET 2023-05-31T20:00:52.375 DEBUG:boto:Path: /pklxkd-13/?prefix=obj 2023-05-31T20:00:52.375 DEBUG:boto:Data: 2023-05-31T20:00:52.375 DEBUG:boto:Headers: {} 2023-05-31T20:00:52.376 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:00:52.376 DEBUG:boto:Port: 8000 2023-05-31T20:00:52.376 DEBUG:boto:Params: {} 2023-05-31T20:00:52.377 DEBUG:boto:Token: None 2023-05-31T20:00:52.377 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:00:52 GMT /pklxkd-13/ 2023-05-31T20:00:52.378 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:FGXZKT+GD0K7Ay6Sb8IIHLqRZf8= 2023-05-31T20:00:52.378 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:00:52 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:FGXZKT+GD0K7Ay6Sb8IIHLqRZf8=', 'Content-Length': '0'} 2023-05-31T20:00:52.380 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e058cd54cac1502e-006477a774-110e-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:00:52 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:00:52.381 DEBUG:boto:b'pklxkd-13obj1000falseobj2023-05-31T20:00:08.058Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:00:52.381 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2023-05-31T20:00:52.382 DEBUG:boto:path=/pklxkd-13/obj 2023-05-31T20:00:52.382 DEBUG:boto:auth_path=/pklxkd-13/obj 2023-05-31T20:00:52.383 DEBUG:boto:path=/pklxkd-13/obj?versionId=0DuL1bo3E1FXXi7mJ3oxDT6slXlrtRA 2023-05-31T20:00:52.383 DEBUG:boto:auth_path=/pklxkd-13/obj?versionId=0DuL1bo3E1FXXi7mJ3oxDT6slXlrtRA 2023-05-31T20:00:52.383 DEBUG:boto:Method: GET 2023-05-31T20:00:52.384 DEBUG:boto:Path: /pklxkd-13/obj?versionId=0DuL1bo3E1FXXi7mJ3oxDT6slXlrtRA 2023-05-31T20:00:52.384 DEBUG:boto:Data: 2023-05-31T20:00:52.384 DEBUG:boto:Headers: {} 2023-05-31T20:00:52.385 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:00:52.385 DEBUG:boto:Port: 8001 2023-05-31T20:00:52.385 DEBUG:boto:Params: {} 2023-05-31T20:00:52.386 DEBUG:boto:Token: None 2023-05-31T20:00:52.386 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:00:52 GMT /pklxkd-13/obj?versionId=0DuL1bo3E1FXXi7mJ3oxDT6slXlrtRA 2023-05-31T20:00:52.387 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:OM3hjFXqGtzxmfSGJ0z7BnCoXro= 2023-05-31T20:00:52.387 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:00:52 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:OM3hjFXqGtzxmfSGJ0z7BnCoXro=', 'Content-Length': '0'} 2023-05-31T20:00:52.389 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:00:07 GMT'), ('x-amz-version-id', '0DuL1bo3E1FXXi7mJ3oxDT6slXlrtRA'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000ecde2db3a99fd57c-006477a774-1112-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:00:52 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:00:52.389 DEBUG:boto:path=/pklxkd-13/obj 2023-05-31T20:00:52.390 DEBUG:boto:auth_path=/pklxkd-13/obj 2023-05-31T20:00:52.390 DEBUG:boto:path=/pklxkd-13/obj?versionId=0DuL1bo3E1FXXi7mJ3oxDT6slXlrtRA 2023-05-31T20:00:52.391 DEBUG:boto:auth_path=/pklxkd-13/obj?versionId=0DuL1bo3E1FXXi7mJ3oxDT6slXlrtRA 2023-05-31T20:00:52.391 DEBUG:boto:Method: GET 2023-05-31T20:00:52.391 DEBUG:boto:Path: /pklxkd-13/obj?versionId=0DuL1bo3E1FXXi7mJ3oxDT6slXlrtRA 2023-05-31T20:00:52.392 DEBUG:boto:Data: 2023-05-31T20:00:52.392 DEBUG:boto:Headers: {} 2023-05-31T20:00:52.392 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:00:52.393 DEBUG:boto:Port: 8000 2023-05-31T20:00:52.393 DEBUG:boto:Params: {} 2023-05-31T20:00:52.394 DEBUG:boto:Token: None 2023-05-31T20:00:52.394 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:00:52 GMT /pklxkd-13/obj?versionId=0DuL1bo3E1FXXi7mJ3oxDT6slXlrtRA 2023-05-31T20:00:52.394 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:OM3hjFXqGtzxmfSGJ0z7BnCoXro= 2023-05-31T20:00:52.395 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:00:52 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:OM3hjFXqGtzxmfSGJ0z7BnCoXro=', 'Content-Length': '0'} 2023-05-31T20:00:52.396 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:00:07 GMT'), ('x-amz-version-id', '0DuL1bo3E1FXXi7mJ3oxDT6slXlrtRA'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000006e2f5112902ba01c-006477a774-110e-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:00:52 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:00:52.397 DEBUG:boto:path=/pklxkd-13/obj 2023-05-31T20:00:52.397 DEBUG:boto:auth_path=/pklxkd-13/obj 2023-05-31T20:00:52.398 DEBUG:boto:path=/pklxkd-13/obj?versionId=0DuL1bo3E1FXXi7mJ3oxDT6slXlrtRA 2023-05-31T20:00:52.398 DEBUG:boto:auth_path=/pklxkd-13/obj?versionId=0DuL1bo3E1FXXi7mJ3oxDT6slXlrtRA 2023-05-31T20:00:52.399 DEBUG:boto:Method: HEAD 2023-05-31T20:00:52.399 DEBUG:boto:Path: /pklxkd-13/obj?versionId=0DuL1bo3E1FXXi7mJ3oxDT6slXlrtRA 2023-05-31T20:00:52.399 DEBUG:boto:Data: 2023-05-31T20:00:52.400 DEBUG:boto:Headers: {} 2023-05-31T20:00:52.400 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:00:52.400 DEBUG:boto:Port: 8001 2023-05-31T20:00:52.401 DEBUG:boto:Params: {} 2023-05-31T20:00:52.401 DEBUG:boto:Token: None 2023-05-31T20:00:52.402 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:00:52 GMT /pklxkd-13/obj?versionId=0DuL1bo3E1FXXi7mJ3oxDT6slXlrtRA 2023-05-31T20:00:52.402 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:5shDwRG6fHsWiVXx1ASfWsWQjvs= 2023-05-31T20:00:52.402 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:00:52 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:5shDwRG6fHsWiVXx1ASfWsWQjvs=', 'Content-Length': '0'} 2023-05-31T20:00:52.404 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:00:07 GMT'), ('x-amz-version-id', '0DuL1bo3E1FXXi7mJ3oxDT6slXlrtRA'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000aada4549948f602f-006477a774-1112-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:00:52 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:00:52.405 DEBUG:boto:path=/pklxkd-13/obj 2023-05-31T20:00:52.405 DEBUG:boto:auth_path=/pklxkd-13/obj 2023-05-31T20:00:52.405 DEBUG:boto:path=/pklxkd-13/obj?versionId=0DuL1bo3E1FXXi7mJ3oxDT6slXlrtRA 2023-05-31T20:00:52.406 DEBUG:boto:auth_path=/pklxkd-13/obj?versionId=0DuL1bo3E1FXXi7mJ3oxDT6slXlrtRA 2023-05-31T20:00:52.406 DEBUG:boto:Method: HEAD 2023-05-31T20:00:52.406 DEBUG:boto:Path: /pklxkd-13/obj?versionId=0DuL1bo3E1FXXi7mJ3oxDT6slXlrtRA 2023-05-31T20:00:52.407 DEBUG:boto:Data: 2023-05-31T20:00:52.407 DEBUG:boto:Headers: {} 2023-05-31T20:00:52.407 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:00:52.408 DEBUG:boto:Port: 8000 2023-05-31T20:00:52.408 DEBUG:boto:Params: {} 2023-05-31T20:00:52.409 DEBUG:boto:Token: None 2023-05-31T20:00:52.409 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:00:52 GMT /pklxkd-13/obj?versionId=0DuL1bo3E1FXXi7mJ3oxDT6slXlrtRA 2023-05-31T20:00:52.409 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:5shDwRG6fHsWiVXx1ASfWsWQjvs= 2023-05-31T20:00:52.410 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:00:52 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:5shDwRG6fHsWiVXx1ASfWsWQjvs=', 'Content-Length': '0'} 2023-05-31T20:00:52.411 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:00:07 GMT'), ('x-amz-version-id', '0DuL1bo3E1FXXi7mJ3oxDT6slXlrtRA'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000a68b28d45a21014c-006477a774-110e-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:00:52 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:00:52.412 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2023-05-31T20:00:52.412 DEBUG:boto:path=/pklxkd-13/obj 2023-05-31T20:00:52.413 DEBUG:boto:auth_path=/pklxkd-13/obj 2023-05-31T20:00:52.413 DEBUG:boto:path=/pklxkd-13/obj?versionId=0DuL1bo3E1FXXi7mJ3oxDT6slXlrtRA 2023-05-31T20:00:52.413 DEBUG:boto:auth_path=/pklxkd-13/obj?versionId=0DuL1bo3E1FXXi7mJ3oxDT6slXlrtRA 2023-05-31T20:00:52.414 DEBUG:boto:Method: GET 2023-05-31T20:00:52.414 DEBUG:boto:Path: /pklxkd-13/obj?versionId=0DuL1bo3E1FXXi7mJ3oxDT6slXlrtRA 2023-05-31T20:00:52.414 DEBUG:boto:Data: 2023-05-31T20:00:52.415 DEBUG:boto:Headers: {} 2023-05-31T20:00:52.415 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:00:52.415 DEBUG:boto:Port: 8001 2023-05-31T20:00:52.416 DEBUG:boto:Params: {} 2023-05-31T20:00:52.416 DEBUG:boto:Token: None 2023-05-31T20:00:52.417 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:00:52 GMT /pklxkd-13/obj?versionId=0DuL1bo3E1FXXi7mJ3oxDT6slXlrtRA 2023-05-31T20:00:52.417 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:OM3hjFXqGtzxmfSGJ0z7BnCoXro= 2023-05-31T20:00:52.417 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:00:52 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:OM3hjFXqGtzxmfSGJ0z7BnCoXro=', 'Content-Length': '0'} 2023-05-31T20:00:52.419 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:00:07 GMT'), ('x-amz-version-id', '0DuL1bo3E1FXXi7mJ3oxDT6slXlrtRA'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000a7e99dd4154dd4b4-006477a774-1112-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:00:52 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:00:52.420 DEBUG:boto:path=/pklxkd-13/obj 2023-05-31T20:00:52.420 DEBUG:boto:auth_path=/pklxkd-13/obj 2023-05-31T20:00:52.420 DEBUG:boto:path=/pklxkd-13/obj?versionId=0DuL1bo3E1FXXi7mJ3oxDT6slXlrtRA 2023-05-31T20:00:52.421 DEBUG:boto:auth_path=/pklxkd-13/obj?versionId=0DuL1bo3E1FXXi7mJ3oxDT6slXlrtRA 2023-05-31T20:00:52.421 DEBUG:boto:Method: GET 2023-05-31T20:00:52.421 DEBUG:boto:Path: /pklxkd-13/obj?versionId=0DuL1bo3E1FXXi7mJ3oxDT6slXlrtRA 2023-05-31T20:00:52.422 DEBUG:boto:Data: 2023-05-31T20:00:52.422 DEBUG:boto:Headers: {} 2023-05-31T20:00:52.422 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:00:52.423 DEBUG:boto:Port: 8000 2023-05-31T20:00:52.423 DEBUG:boto:Params: {} 2023-05-31T20:00:52.424 DEBUG:boto:Token: None 2023-05-31T20:00:52.424 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:00:52 GMT /pklxkd-13/obj?versionId=0DuL1bo3E1FXXi7mJ3oxDT6slXlrtRA 2023-05-31T20:00:52.424 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:OM3hjFXqGtzxmfSGJ0z7BnCoXro= 2023-05-31T20:00:52.425 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:00:52 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:OM3hjFXqGtzxmfSGJ0z7BnCoXro=', 'Content-Length': '0'} 2023-05-31T20:00:52.427 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:00:07 GMT'), ('x-amz-version-id', '0DuL1bo3E1FXXi7mJ3oxDT6slXlrtRA'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000a46ccd34b477e485-006477a774-110e-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:00:52 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:00:52.427 DEBUG:boto:path=/pklxkd-13/ 2023-05-31T20:00:52.428 DEBUG:boto:auth_path=/pklxkd-13/ 2023-05-31T20:00:52.428 DEBUG:boto:path=/pklxkd-13/?prefix=obj 2023-05-31T20:00:52.428 DEBUG:boto:auth_path=/pklxkd-13/?prefix=obj 2023-05-31T20:00:52.429 DEBUG:boto:Method: GET 2023-05-31T20:00:52.429 DEBUG:boto:Path: /pklxkd-13/?prefix=obj 2023-05-31T20:00:52.429 DEBUG:boto:Data: 2023-05-31T20:00:52.430 DEBUG:boto:Headers: {} 2023-05-31T20:00:52.430 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:00:52.431 DEBUG:boto:Port: 8001 2023-05-31T20:00:52.431 DEBUG:boto:Params: {} 2023-05-31T20:00:52.431 DEBUG:boto:Token: None 2023-05-31T20:00:52.432 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:00:52 GMT /pklxkd-13/ 2023-05-31T20:00:52.432 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:FGXZKT+GD0K7Ay6Sb8IIHLqRZf8= 2023-05-31T20:00:52.433 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:00:52 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:FGXZKT+GD0K7Ay6Sb8IIHLqRZf8=', 'Content-Length': '0'} 2023-05-31T20:00:52.435 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000005dd373ba53594a3f-006477a774-1112-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:00:52 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:00:52.435 DEBUG:boto:b'pklxkd-13obj1000falseobj2023-05-31T20:00:08.058Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:00:52.436 DEBUG:boto:path=/pklxkd-13/ 2023-05-31T20:00:52.436 DEBUG:boto:auth_path=/pklxkd-13/ 2023-05-31T20:00:52.437 DEBUG:boto:path=/pklxkd-13/?prefix=obj 2023-05-31T20:00:52.437 DEBUG:boto:auth_path=/pklxkd-13/?prefix=obj 2023-05-31T20:00:52.437 DEBUG:boto:Method: GET 2023-05-31T20:00:52.438 DEBUG:boto:Path: /pklxkd-13/?prefix=obj 2023-05-31T20:00:52.438 DEBUG:boto:Data: 2023-05-31T20:00:52.438 DEBUG:boto:Headers: {} 2023-05-31T20:00:52.439 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:00:52.439 DEBUG:boto:Port: 8000 2023-05-31T20:00:52.439 DEBUG:boto:Params: {} 2023-05-31T20:00:52.440 DEBUG:boto:Token: None 2023-05-31T20:00:52.440 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:00:52 GMT /pklxkd-13/ 2023-05-31T20:00:52.441 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:FGXZKT+GD0K7Ay6Sb8IIHLqRZf8= 2023-05-31T20:00:52.441 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:00:52 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:FGXZKT+GD0K7Ay6Sb8IIHLqRZf8=', 'Content-Length': '0'} 2023-05-31T20:00:52.444 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b345812f63356b4e-006477a774-110e-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:00:52 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:00:52.444 DEBUG:boto:b'pklxkd-13obj1000falseobj2023-05-31T20:00:08.058Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:00:52.445 DEBUG:boto:path=/pklxkd-13/obj 2023-05-31T20:00:52.445 DEBUG:boto:auth_path=/pklxkd-13/obj 2023-05-31T20:00:52.445 DEBUG:boto:Method: HEAD 2023-05-31T20:00:52.446 DEBUG:boto:Path: /pklxkd-13/obj 2023-05-31T20:00:52.446 DEBUG:boto:Data: 2023-05-31T20:00:52.447 DEBUG:boto:Headers: {} 2023-05-31T20:00:52.447 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:00:52.447 DEBUG:boto:Port: 8001 2023-05-31T20:00:52.447 DEBUG:boto:Params: {} 2023-05-31T20:00:52.448 DEBUG:boto:Token: None 2023-05-31T20:00:52.448 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:00:52 GMT /pklxkd-13/obj 2023-05-31T20:00:52.449 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:47OSVk8UFhfayV7BvQVLHanT+oY= 2023-05-31T20:00:52.449 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:00:52 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:47OSVk8UFhfayV7BvQVLHanT+oY=', 'Content-Length': '0'} 2023-05-31T20:00:52.452 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:00:08 GMT'), ('x-amz-version-id', '.K0qHk3QiH4acREzLteeQLJmFzn4x-1'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000d37d19e2a66bcc64-006477a774-1112-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:00:52 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:00:52.452 DEBUG:boto:path=/pklxkd-13/obj 2023-05-31T20:00:52.453 DEBUG:boto:auth_path=/pklxkd-13/obj 2023-05-31T20:00:52.453 DEBUG:boto:Method: HEAD 2023-05-31T20:00:52.454 DEBUG:boto:Path: /pklxkd-13/obj 2023-05-31T20:00:52.454 DEBUG:boto:Data: 2023-05-31T20:00:52.454 DEBUG:boto:Headers: {} 2023-05-31T20:00:52.454 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:00:52.455 DEBUG:boto:Port: 8000 2023-05-31T20:00:52.455 DEBUG:boto:Params: {} 2023-05-31T20:00:52.456 DEBUG:boto:Token: None 2023-05-31T20:00:52.456 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:00:52 GMT /pklxkd-13/obj 2023-05-31T20:00:52.456 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:47OSVk8UFhfayV7BvQVLHanT+oY= 2023-05-31T20:00:52.457 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:00:52 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:47OSVk8UFhfayV7BvQVLHanT+oY=', 'Content-Length': '0'} 2023-05-31T20:00:52.459 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:00:08 GMT'), ('x-amz-version-id', '.K0qHk3QiH4acREzLteeQLJmFzn4x-1'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000006519889d4a4b8453-006477a774-110e-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:00:52 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:00:52.460 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2023-05-31T20:00:52.460 DEBUG:boto:path=/pklxkd-13/obj 2023-05-31T20:00:52.461 DEBUG:boto:auth_path=/pklxkd-13/obj 2023-05-31T20:00:52.461 DEBUG:boto:path=/pklxkd-13/obj?versionId=.K0qHk3QiH4acREzLteeQLJmFzn4x-1 2023-05-31T20:00:52.461 DEBUG:boto:auth_path=/pklxkd-13/obj?versionId=.K0qHk3QiH4acREzLteeQLJmFzn4x-1 2023-05-31T20:00:52.462 DEBUG:boto:Method: GET 2023-05-31T20:00:52.462 DEBUG:boto:Path: /pklxkd-13/obj?versionId=.K0qHk3QiH4acREzLteeQLJmFzn4x-1 2023-05-31T20:00:52.462 DEBUG:boto:Data: 2023-05-31T20:00:52.463 DEBUG:boto:Headers: {} 2023-05-31T20:00:52.463 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:00:52.463 DEBUG:boto:Port: 8001 2023-05-31T20:00:52.464 DEBUG:boto:Params: {} 2023-05-31T20:00:52.464 DEBUG:boto:Token: None 2023-05-31T20:00:52.465 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:00:52 GMT /pklxkd-13/obj?versionId=.K0qHk3QiH4acREzLteeQLJmFzn4x-1 2023-05-31T20:00:52.465 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:qzlVetv8hVapoLo7pVuvQqLVOm4= 2023-05-31T20:00:52.465 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:00:52 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:qzlVetv8hVapoLo7pVuvQqLVOm4=', 'Content-Length': '0'} 2023-05-31T20:00:52.467 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:00:08 GMT'), ('x-amz-version-id', '.K0qHk3QiH4acREzLteeQLJmFzn4x-1'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000cfe571a6017e31b8-006477a774-1112-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:00:52 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:00:52.468 DEBUG:boto:path=/pklxkd-13/obj 2023-05-31T20:00:52.468 DEBUG:boto:auth_path=/pklxkd-13/obj 2023-05-31T20:00:52.469 DEBUG:boto:path=/pklxkd-13/obj?versionId=.K0qHk3QiH4acREzLteeQLJmFzn4x-1 2023-05-31T20:00:52.469 DEBUG:boto:auth_path=/pklxkd-13/obj?versionId=.K0qHk3QiH4acREzLteeQLJmFzn4x-1 2023-05-31T20:00:52.469 DEBUG:boto:Method: GET 2023-05-31T20:00:52.470 DEBUG:boto:Path: /pklxkd-13/obj?versionId=.K0qHk3QiH4acREzLteeQLJmFzn4x-1 2023-05-31T20:00:52.470 DEBUG:boto:Data: 2023-05-31T20:00:52.470 DEBUG:boto:Headers: {} 2023-05-31T20:00:52.471 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:00:52.471 DEBUG:boto:Port: 8000 2023-05-31T20:00:52.471 DEBUG:boto:Params: {} 2023-05-31T20:00:52.472 DEBUG:boto:Token: None 2023-05-31T20:00:52.472 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:00:52 GMT /pklxkd-13/obj?versionId=.K0qHk3QiH4acREzLteeQLJmFzn4x-1 2023-05-31T20:00:52.473 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:qzlVetv8hVapoLo7pVuvQqLVOm4= 2023-05-31T20:00:52.473 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:00:52 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:qzlVetv8hVapoLo7pVuvQqLVOm4=', 'Content-Length': '0'} 2023-05-31T20:00:52.475 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:00:08 GMT'), ('x-amz-version-id', '.K0qHk3QiH4acREzLteeQLJmFzn4x-1'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000008c541aff9a22e7d6-006477a774-110e-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:00:52 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:00:52.475 DEBUG:boto:path=/pklxkd-13/ 2023-05-31T20:00:52.476 DEBUG:boto:auth_path=/pklxkd-13/ 2023-05-31T20:00:52.476 DEBUG:boto:path=/pklxkd-13/?prefix=obj 2023-05-31T20:00:52.477 DEBUG:boto:auth_path=/pklxkd-13/?prefix=obj 2023-05-31T20:00:52.477 DEBUG:boto:Method: GET 2023-05-31T20:00:52.477 DEBUG:boto:Path: /pklxkd-13/?prefix=obj 2023-05-31T20:00:52.478 DEBUG:boto:Data: 2023-05-31T20:00:52.478 DEBUG:boto:Headers: {} 2023-05-31T20:00:52.478 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:00:52.479 DEBUG:boto:Port: 8001 2023-05-31T20:00:52.479 DEBUG:boto:Params: {} 2023-05-31T20:00:52.480 DEBUG:boto:Token: None 2023-05-31T20:00:52.480 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:00:52 GMT /pklxkd-13/ 2023-05-31T20:00:52.480 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:FGXZKT+GD0K7Ay6Sb8IIHLqRZf8= 2023-05-31T20:00:52.481 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:00:52 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:FGXZKT+GD0K7Ay6Sb8IIHLqRZf8=', 'Content-Length': '0'} 2023-05-31T20:00:52.483 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000072818ad7e3806822-006477a774-1112-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:00:52 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:00:52.484 DEBUG:boto:b'pklxkd-13obj1000falseobj2023-05-31T20:00:08.058Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:00:52.485 DEBUG:boto:path=/pklxkd-13/ 2023-05-31T20:00:52.485 DEBUG:boto:auth_path=/pklxkd-13/ 2023-05-31T20:00:52.485 DEBUG:boto:path=/pklxkd-13/?prefix=obj 2023-05-31T20:00:52.486 DEBUG:boto:auth_path=/pklxkd-13/?prefix=obj 2023-05-31T20:00:52.486 DEBUG:boto:Method: GET 2023-05-31T20:00:52.487 DEBUG:boto:Path: /pklxkd-13/?prefix=obj 2023-05-31T20:00:52.487 DEBUG:boto:Data: 2023-05-31T20:00:52.487 DEBUG:boto:Headers: {} 2023-05-31T20:00:52.488 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:00:52.488 DEBUG:boto:Port: 8000 2023-05-31T20:00:52.488 DEBUG:boto:Params: {} 2023-05-31T20:00:52.489 DEBUG:boto:Token: None 2023-05-31T20:00:52.489 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:00:52 GMT /pklxkd-13/ 2023-05-31T20:00:52.490 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:FGXZKT+GD0K7Ay6Sb8IIHLqRZf8= 2023-05-31T20:00:52.490 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:00:52 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:FGXZKT+GD0K7Ay6Sb8IIHLqRZf8=', 'Content-Length': '0'} 2023-05-31T20:00:52.495 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000000aa1b63852fda0bd-006477a774-110e-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:00:52 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:00:52.495 DEBUG:boto:b'pklxkd-13obj1000falseobj2023-05-31T20:00:08.058Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:00:52.496 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2023-05-31T20:00:52.497 DEBUG:boto:path=/pklxkd-13/obj 2023-05-31T20:00:52.497 DEBUG:boto:auth_path=/pklxkd-13/obj 2023-05-31T20:00:52.497 DEBUG:boto:path=/pklxkd-13/obj?versionId=pm7-Sh2bsjDbtvOCwujAMZA4JCZRKlh 2023-05-31T20:00:52.498 DEBUG:boto:auth_path=/pklxkd-13/obj?versionId=pm7-Sh2bsjDbtvOCwujAMZA4JCZRKlh 2023-05-31T20:00:52.498 DEBUG:boto:Method: GET 2023-05-31T20:00:52.498 DEBUG:boto:Path: /pklxkd-13/obj?versionId=pm7-Sh2bsjDbtvOCwujAMZA4JCZRKlh 2023-05-31T20:00:52.499 DEBUG:boto:Data: 2023-05-31T20:00:52.499 DEBUG:boto:Headers: {} 2023-05-31T20:00:52.499 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:00:52.500 DEBUG:boto:Port: 8001 2023-05-31T20:00:52.500 DEBUG:boto:Params: {} 2023-05-31T20:00:52.501 DEBUG:boto:Token: None 2023-05-31T20:00:52.501 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:00:52 GMT /pklxkd-13/obj?versionId=pm7-Sh2bsjDbtvOCwujAMZA4JCZRKlh 2023-05-31T20:00:52.501 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:b0+13Szt8ZMZo83hB/LsRe4fUFc= 2023-05-31T20:00:52.502 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:00:52 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:b0+13Szt8ZMZo83hB/LsRe4fUFc=', 'Content-Length': '0'} 2023-05-31T20:00:52.503 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:00:07 GMT'), ('x-amz-version-id', 'pm7-Sh2bsjDbtvOCwujAMZA4JCZRKlh'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000bbe88d18d16f6229-006477a774-1112-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:00:52 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:00:52.504 DEBUG:boto:path=/pklxkd-13/obj 2023-05-31T20:00:52.504 DEBUG:boto:auth_path=/pklxkd-13/obj 2023-05-31T20:00:52.505 DEBUG:boto:path=/pklxkd-13/obj?versionId=pm7-Sh2bsjDbtvOCwujAMZA4JCZRKlh 2023-05-31T20:00:52.505 DEBUG:boto:auth_path=/pklxkd-13/obj?versionId=pm7-Sh2bsjDbtvOCwujAMZA4JCZRKlh 2023-05-31T20:00:52.506 DEBUG:boto:Method: GET 2023-05-31T20:00:52.506 DEBUG:boto:Path: /pklxkd-13/obj?versionId=pm7-Sh2bsjDbtvOCwujAMZA4JCZRKlh 2023-05-31T20:00:52.506 DEBUG:boto:Data: 2023-05-31T20:00:52.507 DEBUG:boto:Headers: {} 2023-05-31T20:00:52.507 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:00:52.507 DEBUG:boto:Port: 8000 2023-05-31T20:00:52.508 DEBUG:boto:Params: {} 2023-05-31T20:00:52.508 DEBUG:boto:Token: None 2023-05-31T20:00:52.509 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:00:52 GMT /pklxkd-13/obj?versionId=pm7-Sh2bsjDbtvOCwujAMZA4JCZRKlh 2023-05-31T20:00:52.509 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:b0+13Szt8ZMZo83hB/LsRe4fUFc= 2023-05-31T20:00:52.509 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:00:52 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:b0+13Szt8ZMZo83hB/LsRe4fUFc=', 'Content-Length': '0'} 2023-05-31T20:00:52.511 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:00:07 GMT'), ('x-amz-version-id', 'pm7-Sh2bsjDbtvOCwujAMZA4JCZRKlh'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000048401625addce80b-006477a774-110e-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:00:52 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:00:52.552 DEBUG:boto:path=/pklxkd-13/obj 2023-05-31T20:00:52.553 DEBUG:boto:auth_path=/pklxkd-13/obj 2023-05-31T20:00:52.553 DEBUG:boto:path=/pklxkd-13/obj?versionId=pm7-Sh2bsjDbtvOCwujAMZA4JCZRKlh 2023-05-31T20:00:52.553 DEBUG:boto:auth_path=/pklxkd-13/obj?versionId=pm7-Sh2bsjDbtvOCwujAMZA4JCZRKlh 2023-05-31T20:00:52.554 DEBUG:boto:Method: HEAD 2023-05-31T20:00:52.554 DEBUG:boto:Path: /pklxkd-13/obj?versionId=pm7-Sh2bsjDbtvOCwujAMZA4JCZRKlh 2023-05-31T20:00:52.554 DEBUG:boto:Data: 2023-05-31T20:00:52.555 DEBUG:boto:Headers: {} 2023-05-31T20:00:52.555 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:00:52.555 DEBUG:boto:Port: 8001 2023-05-31T20:00:52.556 DEBUG:boto:Params: {} 2023-05-31T20:00:52.556 DEBUG:boto:Token: None 2023-05-31T20:00:52.556 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:00:52 GMT /pklxkd-13/obj?versionId=pm7-Sh2bsjDbtvOCwujAMZA4JCZRKlh 2023-05-31T20:00:52.557 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:sLaovFPETaaoa65b9OMA8fTnKX4= 2023-05-31T20:00:52.557 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:00:52 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:sLaovFPETaaoa65b9OMA8fTnKX4=', 'Content-Length': '0'} 2023-05-31T20:00:52.559 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:00:07 GMT'), ('x-amz-version-id', 'pm7-Sh2bsjDbtvOCwujAMZA4JCZRKlh'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000ee5dd1e8e98caefd-006477a774-1112-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:00:52 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:00:52.560 DEBUG:boto:path=/pklxkd-13/obj 2023-05-31T20:00:52.560 DEBUG:boto:auth_path=/pklxkd-13/obj 2023-05-31T20:00:52.560 DEBUG:boto:path=/pklxkd-13/obj?versionId=pm7-Sh2bsjDbtvOCwujAMZA4JCZRKlh 2023-05-31T20:00:52.561 DEBUG:boto:auth_path=/pklxkd-13/obj?versionId=pm7-Sh2bsjDbtvOCwujAMZA4JCZRKlh 2023-05-31T20:00:52.561 DEBUG:boto:Method: HEAD 2023-05-31T20:00:52.562 DEBUG:boto:Path: /pklxkd-13/obj?versionId=pm7-Sh2bsjDbtvOCwujAMZA4JCZRKlh 2023-05-31T20:00:52.562 DEBUG:boto:Data: 2023-05-31T20:00:52.562 DEBUG:boto:Headers: {} 2023-05-31T20:00:52.563 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:00:52.563 DEBUG:boto:Port: 8000 2023-05-31T20:00:52.563 DEBUG:boto:Params: {} 2023-05-31T20:00:52.564 DEBUG:boto:Token: None 2023-05-31T20:00:52.564 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:00:52 GMT /pklxkd-13/obj?versionId=pm7-Sh2bsjDbtvOCwujAMZA4JCZRKlh 2023-05-31T20:00:52.565 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:sLaovFPETaaoa65b9OMA8fTnKX4= 2023-05-31T20:00:52.565 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:00:52 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:sLaovFPETaaoa65b9OMA8fTnKX4=', 'Content-Length': '0'} 2023-05-31T20:00:52.567 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:00:07 GMT'), ('x-amz-version-id', 'pm7-Sh2bsjDbtvOCwujAMZA4JCZRKlh'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000e31ee9f2f8aab458-006477a774-110e-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:00:52 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:00:52.568 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2023-05-31T20:00:52.568 DEBUG:boto:path=/pklxkd-13/obj 2023-05-31T20:00:52.569 DEBUG:boto:auth_path=/pklxkd-13/obj 2023-05-31T20:00:52.569 DEBUG:boto:path=/pklxkd-13/obj?versionId=pm7-Sh2bsjDbtvOCwujAMZA4JCZRKlh 2023-05-31T20:00:52.569 DEBUG:boto:auth_path=/pklxkd-13/obj?versionId=pm7-Sh2bsjDbtvOCwujAMZA4JCZRKlh 2023-05-31T20:00:52.570 DEBUG:boto:Method: GET 2023-05-31T20:00:52.570 DEBUG:boto:Path: /pklxkd-13/obj?versionId=pm7-Sh2bsjDbtvOCwujAMZA4JCZRKlh 2023-05-31T20:00:52.570 DEBUG:boto:Data: 2023-05-31T20:00:52.572 DEBUG:boto:Headers: {} 2023-05-31T20:00:52.573 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:00:52.573 DEBUG:boto:Port: 8001 2023-05-31T20:00:52.574 DEBUG:boto:Params: {} 2023-05-31T20:00:52.575 DEBUG:boto:Token: None 2023-05-31T20:00:52.575 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:00:52 GMT /pklxkd-13/obj?versionId=pm7-Sh2bsjDbtvOCwujAMZA4JCZRKlh 2023-05-31T20:00:52.576 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:b0+13Szt8ZMZo83hB/LsRe4fUFc= 2023-05-31T20:00:52.576 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:00:52 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:b0+13Szt8ZMZo83hB/LsRe4fUFc=', 'Content-Length': '0'} 2023-05-31T20:00:52.579 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:00:07 GMT'), ('x-amz-version-id', 'pm7-Sh2bsjDbtvOCwujAMZA4JCZRKlh'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000009be3d3b42532fb1f-006477a774-1112-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:00:52 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:00:52.579 DEBUG:boto:path=/pklxkd-13/obj 2023-05-31T20:00:52.580 DEBUG:boto:auth_path=/pklxkd-13/obj 2023-05-31T20:00:52.580 DEBUG:boto:path=/pklxkd-13/obj?versionId=pm7-Sh2bsjDbtvOCwujAMZA4JCZRKlh 2023-05-31T20:00:52.581 DEBUG:boto:auth_path=/pklxkd-13/obj?versionId=pm7-Sh2bsjDbtvOCwujAMZA4JCZRKlh 2023-05-31T20:00:52.581 DEBUG:boto:Method: GET 2023-05-31T20:00:52.582 DEBUG:boto:Path: /pklxkd-13/obj?versionId=pm7-Sh2bsjDbtvOCwujAMZA4JCZRKlh 2023-05-31T20:00:52.582 DEBUG:boto:Data: 2023-05-31T20:00:52.582 DEBUG:boto:Headers: {} 2023-05-31T20:00:52.583 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:00:52.583 DEBUG:boto:Port: 8000 2023-05-31T20:00:52.584 DEBUG:boto:Params: {} 2023-05-31T20:00:52.585 DEBUG:boto:Token: None 2023-05-31T20:00:52.585 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:00:52 GMT /pklxkd-13/obj?versionId=pm7-Sh2bsjDbtvOCwujAMZA4JCZRKlh 2023-05-31T20:00:52.585 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:b0+13Szt8ZMZo83hB/LsRe4fUFc= 2023-05-31T20:00:52.586 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:00:52 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:b0+13Szt8ZMZo83hB/LsRe4fUFc=', 'Content-Length': '0'} 2023-05-31T20:00:52.588 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:00:07 GMT'), ('x-amz-version-id', 'pm7-Sh2bsjDbtvOCwujAMZA4JCZRKlh'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000eee1bcc310dc2088-006477a774-110e-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:00:52 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:00:52.589 DEBUG:boto:path=/pklxkd-13/ 2023-05-31T20:00:52.589 DEBUG:boto:auth_path=/pklxkd-13/ 2023-05-31T20:00:52.589 DEBUG:boto:path=/pklxkd-13/?prefix=obj 2023-05-31T20:00:52.590 DEBUG:boto:auth_path=/pklxkd-13/?prefix=obj 2023-05-31T20:00:52.590 DEBUG:boto:Method: GET 2023-05-31T20:00:52.590 DEBUG:boto:Path: /pklxkd-13/?prefix=obj 2023-05-31T20:00:52.591 DEBUG:boto:Data: 2023-05-31T20:00:52.591 DEBUG:boto:Headers: {} 2023-05-31T20:00:52.592 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:00:52.592 DEBUG:boto:Port: 8001 2023-05-31T20:00:52.592 DEBUG:boto:Params: {} 2023-05-31T20:00:52.593 DEBUG:boto:Token: None 2023-05-31T20:00:52.593 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:00:52 GMT /pklxkd-13/ 2023-05-31T20:00:52.594 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:FGXZKT+GD0K7Ay6Sb8IIHLqRZf8= 2023-05-31T20:00:52.594 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:00:52 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:FGXZKT+GD0K7Ay6Sb8IIHLqRZf8=', 'Content-Length': '0'} 2023-05-31T20:00:52.597 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000049cc95ce85cd4b7c-006477a774-1112-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:00:52 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:00:52.597 DEBUG:boto:b'pklxkd-13obj1000falseobj2023-05-31T20:00:08.058Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:00:52.598 DEBUG:boto:path=/pklxkd-13/ 2023-05-31T20:00:52.599 DEBUG:boto:auth_path=/pklxkd-13/ 2023-05-31T20:00:52.599 DEBUG:boto:path=/pklxkd-13/?prefix=obj 2023-05-31T20:00:52.599 DEBUG:boto:auth_path=/pklxkd-13/?prefix=obj 2023-05-31T20:00:52.600 DEBUG:boto:Method: GET 2023-05-31T20:00:52.600 DEBUG:boto:Path: /pklxkd-13/?prefix=obj 2023-05-31T20:00:52.600 DEBUG:boto:Data: 2023-05-31T20:00:52.601 DEBUG:boto:Headers: {} 2023-05-31T20:00:52.601 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:00:52.602 DEBUG:boto:Port: 8000 2023-05-31T20:00:52.602 DEBUG:boto:Params: {} 2023-05-31T20:00:52.602 DEBUG:boto:Token: None 2023-05-31T20:00:52.603 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:00:52 GMT /pklxkd-13/ 2023-05-31T20:00:52.603 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:FGXZKT+GD0K7Ay6Sb8IIHLqRZf8= 2023-05-31T20:00:52.604 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:00:52 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:FGXZKT+GD0K7Ay6Sb8IIHLqRZf8=', 'Content-Length': '0'} 2023-05-31T20:00:52.608 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000008de94b2ee6eb25b1-006477a774-110e-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:00:52 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:00:52.609 DEBUG:boto:b'pklxkd-13obj1000falseobj2023-05-31T20:00:08.058Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:00:52.610 DEBUG:boto:path=/pklxkd-13/obj 2023-05-31T20:00:52.611 DEBUG:boto:auth_path=/pklxkd-13/obj 2023-05-31T20:00:52.611 DEBUG:boto:Method: HEAD 2023-05-31T20:00:52.612 DEBUG:boto:Path: /pklxkd-13/obj 2023-05-31T20:00:52.613 DEBUG:boto:Data: 2023-05-31T20:00:52.613 DEBUG:boto:Headers: {} 2023-05-31T20:00:52.614 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:00:52.614 DEBUG:boto:Port: 8001 2023-05-31T20:00:52.614 DEBUG:boto:Params: {} 2023-05-31T20:00:52.615 DEBUG:boto:Token: None 2023-05-31T20:00:52.615 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:00:52 GMT /pklxkd-13/obj 2023-05-31T20:00:52.615 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:47OSVk8UFhfayV7BvQVLHanT+oY= 2023-05-31T20:00:52.616 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:00:52 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:47OSVk8UFhfayV7BvQVLHanT+oY=', 'Content-Length': '0'} 2023-05-31T20:00:52.619 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:00:08 GMT'), ('x-amz-version-id', '.K0qHk3QiH4acREzLteeQLJmFzn4x-1'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000048f2c2c54fd86d5d-006477a774-1112-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:00:52 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:00:52.621 DEBUG:boto:path=/pklxkd-13/obj 2023-05-31T20:00:52.621 DEBUG:boto:auth_path=/pklxkd-13/obj 2023-05-31T20:00:52.622 DEBUG:boto:Method: HEAD 2023-05-31T20:00:52.623 DEBUG:boto:Path: /pklxkd-13/obj 2023-05-31T20:00:52.623 DEBUG:boto:Data: 2023-05-31T20:00:52.623 DEBUG:boto:Headers: {} 2023-05-31T20:00:52.624 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:00:52.624 DEBUG:boto:Port: 8000 2023-05-31T20:00:52.625 DEBUG:boto:Params: {} 2023-05-31T20:00:52.625 DEBUG:boto:Token: None 2023-05-31T20:00:52.626 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:00:52 GMT /pklxkd-13/obj 2023-05-31T20:00:52.626 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:47OSVk8UFhfayV7BvQVLHanT+oY= 2023-05-31T20:00:52.626 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:00:52 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:47OSVk8UFhfayV7BvQVLHanT+oY=', 'Content-Length': '0'} 2023-05-31T20:00:52.629 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:00:08 GMT'), ('x-amz-version-id', '.K0qHk3QiH4acREzLteeQLJmFzn4x-1'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000c1cfc598d82f7ede-006477a774-110e-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:00:52 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:00:52.630 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2023-05-31T20:00:52.631 DEBUG:boto:path=/pklxkd-13/obj 2023-05-31T20:00:52.631 DEBUG:boto:auth_path=/pklxkd-13/obj 2023-05-31T20:00:52.631 DEBUG:boto:path=/pklxkd-13/obj?versionId=.K0qHk3QiH4acREzLteeQLJmFzn4x-1 2023-05-31T20:00:52.632 DEBUG:boto:auth_path=/pklxkd-13/obj?versionId=.K0qHk3QiH4acREzLteeQLJmFzn4x-1 2023-05-31T20:00:52.632 DEBUG:boto:Method: GET 2023-05-31T20:00:52.633 DEBUG:boto:Path: /pklxkd-13/obj?versionId=.K0qHk3QiH4acREzLteeQLJmFzn4x-1 2023-05-31T20:00:52.633 DEBUG:boto:Data: 2023-05-31T20:00:52.633 DEBUG:boto:Headers: {} 2023-05-31T20:00:52.634 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:00:52.634 DEBUG:boto:Port: 8001 2023-05-31T20:00:52.635 DEBUG:boto:Params: {} 2023-05-31T20:00:52.635 DEBUG:boto:Token: None 2023-05-31T20:00:52.636 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:00:52 GMT /pklxkd-13/obj?versionId=.K0qHk3QiH4acREzLteeQLJmFzn4x-1 2023-05-31T20:00:52.637 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:qzlVetv8hVapoLo7pVuvQqLVOm4= 2023-05-31T20:00:52.638 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:00:52 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:qzlVetv8hVapoLo7pVuvQqLVOm4=', 'Content-Length': '0'} 2023-05-31T20:00:52.640 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:00:08 GMT'), ('x-amz-version-id', '.K0qHk3QiH4acREzLteeQLJmFzn4x-1'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000897b500e579ea69b-006477a774-1112-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:00:52 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:00:52.680 DEBUG:boto:path=/pklxkd-13/obj 2023-05-31T20:00:52.681 DEBUG:boto:auth_path=/pklxkd-13/obj 2023-05-31T20:00:52.681 DEBUG:boto:path=/pklxkd-13/obj?versionId=.K0qHk3QiH4acREzLteeQLJmFzn4x-1 2023-05-31T20:00:52.682 DEBUG:boto:auth_path=/pklxkd-13/obj?versionId=.K0qHk3QiH4acREzLteeQLJmFzn4x-1 2023-05-31T20:00:52.682 DEBUG:boto:Method: GET 2023-05-31T20:00:52.683 DEBUG:boto:Path: /pklxkd-13/obj?versionId=.K0qHk3QiH4acREzLteeQLJmFzn4x-1 2023-05-31T20:00:52.683 DEBUG:boto:Data: 2023-05-31T20:00:52.684 DEBUG:boto:Headers: {} 2023-05-31T20:00:52.684 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:00:52.685 DEBUG:boto:Port: 8000 2023-05-31T20:00:52.685 DEBUG:boto:Params: {} 2023-05-31T20:00:52.685 DEBUG:boto:Token: None 2023-05-31T20:00:52.686 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:00:52 GMT /pklxkd-13/obj?versionId=.K0qHk3QiH4acREzLteeQLJmFzn4x-1 2023-05-31T20:00:52.686 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:qzlVetv8hVapoLo7pVuvQqLVOm4= 2023-05-31T20:00:52.687 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:00:52 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:qzlVetv8hVapoLo7pVuvQqLVOm4=', 'Content-Length': '0'} 2023-05-31T20:00:52.689 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:00:08 GMT'), ('x-amz-version-id', '.K0qHk3QiH4acREzLteeQLJmFzn4x-1'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000000135ef9f7335e3d9-006477a774-110e-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:00:52 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:00:52.689 DEBUG:boto:path=/pklxkd-13/ 2023-05-31T20:00:52.690 DEBUG:boto:auth_path=/pklxkd-13/ 2023-05-31T20:00:52.690 DEBUG:boto:path=/pklxkd-13/?prefix=obj 2023-05-31T20:00:52.690 DEBUG:boto:auth_path=/pklxkd-13/?prefix=obj 2023-05-31T20:00:52.691 DEBUG:boto:Method: GET 2023-05-31T20:00:52.691 DEBUG:boto:Path: /pklxkd-13/?prefix=obj 2023-05-31T20:00:52.691 DEBUG:boto:Data: 2023-05-31T20:00:52.692 DEBUG:boto:Headers: {} 2023-05-31T20:00:52.692 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:00:52.693 DEBUG:boto:Port: 8001 2023-05-31T20:00:52.693 DEBUG:boto:Params: {} 2023-05-31T20:00:52.693 DEBUG:boto:Token: None 2023-05-31T20:00:52.694 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:00:52 GMT /pklxkd-13/ 2023-05-31T20:00:52.694 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:FGXZKT+GD0K7Ay6Sb8IIHLqRZf8= 2023-05-31T20:00:52.695 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:00:52 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:FGXZKT+GD0K7Ay6Sb8IIHLqRZf8=', 'Content-Length': '0'} 2023-05-31T20:00:52.697 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000003735e16b87ede780-006477a774-1112-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:00:52 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:00:52.698 DEBUG:boto:b'pklxkd-13obj1000falseobj2023-05-31T20:00:08.058Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:00:52.699 DEBUG:boto:path=/pklxkd-13/ 2023-05-31T20:00:52.699 DEBUG:boto:auth_path=/pklxkd-13/ 2023-05-31T20:00:52.699 DEBUG:boto:path=/pklxkd-13/?prefix=obj 2023-05-31T20:00:52.700 DEBUG:boto:auth_path=/pklxkd-13/?prefix=obj 2023-05-31T20:00:52.700 DEBUG:boto:Method: GET 2023-05-31T20:00:52.701 DEBUG:boto:Path: /pklxkd-13/?prefix=obj 2023-05-31T20:00:52.701 DEBUG:boto:Data: 2023-05-31T20:00:52.702 DEBUG:boto:Headers: {} 2023-05-31T20:00:52.702 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:00:52.702 DEBUG:boto:Port: 8000 2023-05-31T20:00:52.703 DEBUG:boto:Params: {} 2023-05-31T20:00:52.703 DEBUG:boto:Token: None 2023-05-31T20:00:52.703 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:00:52 GMT /pklxkd-13/ 2023-05-31T20:00:52.704 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:FGXZKT+GD0K7Ay6Sb8IIHLqRZf8= 2023-05-31T20:00:52.704 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:00:52 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:FGXZKT+GD0K7Ay6Sb8IIHLqRZf8=', 'Content-Length': '0'} 2023-05-31T20:00:52.707 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000006699251f86458b16-006477a774-110e-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:00:52 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:00:52.708 DEBUG:boto:b'pklxkd-13obj1000falseobj2023-05-31T20:00:08.058Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:00:52.708 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=pklxkd-13 zones={a2, a1} 2023-05-31T20:00:52.709 DEBUG:rgw_multi.tests:data checkpoint: source=a1 target=a2 2023-05-31T20:00:52.709 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['datalog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-31T20:00:52.710 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'datalog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-31T20:00:52.711 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 datalog status --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-05-31T20:00:52.768 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2023-05-31T20:00:52.768 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2023-05-31T20:00:52.857 INFO:teuthology.orchestra.run.smithi099.stdout:[ 2023-05-31T20:00:52.857 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T20:00:52.858 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "00000000000000000000:00000000000000000929", 2023-05-31T20:00:52.858 INFO:teuthology.orchestra.run.smithi099.stdout: "last_update": "2023-05-31T19:56:38.214723Z" 2023-05-31T20:00:52.858 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:00:52.858 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T20:00:52.858 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "", 2023-05-31T20:00:52.858 INFO:teuthology.orchestra.run.smithi099.stdout: "last_update": "0.000000" 2023-05-31T20:00:52.859 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:00:52.859 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T20:00:52.859 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "00000000000000000000:00000000000000002196", 2023-05-31T20:00:52.859 INFO:teuthology.orchestra.run.smithi099.stdout: "last_update": "2023-05-31T20:00:40.228882Z" 2023-05-31T20:00:52.860 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:00:52.860 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T20:00:52.860 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "00000000000000000000:00000000000000002905", 2023-05-31T20:00:52.860 INFO:teuthology.orchestra.run.smithi099.stdout: "last_update": "2023-05-31T20:00:18.227192Z" 2023-05-31T20:00:52.860 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T20:00:52.860 INFO:teuthology.orchestra.run.smithi099.stdout:] 2023-05-31T20:00:52.861 DEBUG:rgw_multi.tests:data markers for zone=a1 markers={0: '00000000000000000000:00000000000000000929', 1: '', 2: '00000000000000000000:00000000000000002196', 3: '00000000000000000000:00000000000000002905'} 2023-05-31T20:00:52.862 INFO:rgw_multi.tests:starting data checkpoint for target_zone=a2 source_zone=a1 2023-05-31T20:00:52.862 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-31T20:00:52.863 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-31T20:00:52.863 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-05-31T20:00:52.914 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2023-05-31T20:00:52.915 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2023-05-31T20:00:52.967 INFO:teuthology.orchestra.run.smithi099.stdout:{ 2023-05-31T20:00:52.967 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_status": { 2023-05-31T20:00:52.968 INFO:teuthology.orchestra.run.smithi099.stdout: "info": { 2023-05-31T20:00:52.968 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "sync", 2023-05-31T20:00:52.968 INFO:teuthology.orchestra.run.smithi099.stdout: "num_shards": 4, 2023-05-31T20:00:52.968 INFO:teuthology.orchestra.run.smithi099.stdout: "instance_id": 949931097149414600 2023-05-31T20:00:52.968 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:00:52.968 INFO:teuthology.orchestra.run.smithi099.stdout: "markers": [ 2023-05-31T20:00:52.969 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T20:00:52.969 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 0, 2023-05-31T20:00:52.969 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T20:00:52.969 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "incremental-sync", 2023-05-31T20:00:52.969 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "00000000000000000000:00000000000000000929", 2023-05-31T20:00:52.969 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T20:00:52.969 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2023-05-31T20:00:52.970 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T20:00:52.970 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2023-05-31T19:56:38.214723Z" 2023-05-31T20:00:52.970 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T20:00:52.970 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:00:52.970 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T20:00:52.970 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 1, 2023-05-31T20:00:52.971 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T20:00:52.971 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "incremental-sync", 2023-05-31T20:00:52.971 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "", 2023-05-31T20:00:52.971 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T20:00:52.971 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2023-05-31T20:00:52.971 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T20:00:52.972 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "0.000000" 2023-05-31T20:00:52.972 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T20:00:52.972 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:00:52.972 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T20:00:52.972 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 2, 2023-05-31T20:00:52.973 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T20:00:52.973 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "incremental-sync", 2023-05-31T20:00:52.973 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "00000000000000000000:00000000000000002196", 2023-05-31T20:00:52.973 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T20:00:52.974 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2023-05-31T20:00:52.974 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T20:00:52.975 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2023-05-31T20:00:40.228882Z" 2023-05-31T20:00:52.975 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T20:00:52.975 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:00:52.976 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T20:00:52.976 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 3, 2023-05-31T20:00:52.976 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T20:00:52.977 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "incremental-sync", 2023-05-31T20:00:52.977 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "00000000000000000000:00000000000000002905", 2023-05-31T20:00:52.977 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T20:00:52.978 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2023-05-31T20:00:52.978 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T20:00:52.978 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2023-05-31T20:00:18.227192Z" 2023-05-31T20:00:52.979 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T20:00:52.979 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T20:00:52.979 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2023-05-31T20:00:52.980 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:00:52.980 INFO:teuthology.orchestra.run.smithi099.stdout: "full_sync": { 2023-05-31T20:00:52.980 INFO:teuthology.orchestra.run.smithi099.stdout: "total": 0, 2023-05-31T20:00:52.981 INFO:teuthology.orchestra.run.smithi099.stdout: "complete": 0 2023-05-31T20:00:52.981 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:00:52.981 INFO:teuthology.orchestra.run.smithi099.stdout: "current_time": "2023-05-31T20:00:52Z" 2023-05-31T20:00:52.981 INFO:teuthology.orchestra.run.smithi099.stdout:} 2023-05-31T20:00:52.983 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 949931097149414600 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000000929", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-05-31T19:56:38.214723Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000002196", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-05-31T20:00:40.228882Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000002905", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-05-31T20:00:18.227192Z" } } ] }, "full_sync": { "total": 0, "complete": 0 }, "current_time": "2023-05-31T20:00:52Z" } 2023-05-31T20:00:52.984 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000000929', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-05-31T19:56:38.214723Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000002196', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-05-31T20:00:40.228882Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000002905', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-05-31T20:00:18.227192Z'}}] 2023-05-31T20:00:52.985 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000000929', 1: '', 2: '00000000000000000000:00000000000000002196', 3: '00000000000000000000:00000000000000002905'} 2023-05-31T20:00:52.986 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000000929', 1: '', 2: '00000000000000000000:00000000000000002196', 3: '00000000000000000000:00000000000000002905'} 2023-05-31T20:00:52.986 INFO:rgw_multi.tests:finished data checkpoint for target_zone=a2 source_zone=a1 2023-05-31T20:00:52.986 DEBUG:rgw_multi.tests:data checkpoint: source=a2 target=a1 2023-05-31T20:00:52.987 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['datalog', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-31T20:00:52.987 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'datalog', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-31T20:00:52.987 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 datalog status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-05-31T20:00:53.035 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2023-05-31T20:00:53.036 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2023-05-31T20:00:53.106 INFO:teuthology.orchestra.run.smithi099.stdout:[ 2023-05-31T20:00:53.106 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T20:00:53.106 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "00000000000000000000:00000000000000000929", 2023-05-31T20:00:53.107 INFO:teuthology.orchestra.run.smithi099.stdout: "last_update": "2023-05-31T19:57:00.436067Z" 2023-05-31T20:00:53.107 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:00:53.107 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T20:00:53.107 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "", 2023-05-31T20:00:53.108 INFO:teuthology.orchestra.run.smithi099.stdout: "last_update": "0.000000" 2023-05-31T20:00:53.108 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:00:53.108 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T20:00:53.108 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "00000000000000000000:00000000000000002196", 2023-05-31T20:00:53.109 INFO:teuthology.orchestra.run.smithi099.stdout: "last_update": "2023-05-31T20:00:40.448120Z" 2023-05-31T20:00:53.109 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:00:53.109 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T20:00:53.109 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "00000000000000000000:00000000000000002764", 2023-05-31T20:00:53.109 INFO:teuthology.orchestra.run.smithi099.stdout: "last_update": "2023-05-31T19:59:56.444860Z" 2023-05-31T20:00:53.109 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T20:00:53.110 INFO:teuthology.orchestra.run.smithi099.stdout:] 2023-05-31T20:00:53.110 DEBUG:rgw_multi.tests:data markers for zone=a2 markers={0: '00000000000000000000:00000000000000000929', 1: '', 2: '00000000000000000000:00000000000000002196', 3: '00000000000000000000:00000000000000002764'} 2023-05-31T20:00:53.111 INFO:rgw_multi.tests:starting data checkpoint for target_zone=a1 source_zone=a2 2023-05-31T20:00:53.112 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-31T20:00:53.112 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-31T20:00:53.113 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --source-zone a2 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-05-31T20:00:53.163 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2023-05-31T20:00:53.164 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2023-05-31T20:00:53.236 INFO:teuthology.orchestra.run.smithi099.stdout:{ 2023-05-31T20:00:53.236 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_status": { 2023-05-31T20:00:53.237 INFO:teuthology.orchestra.run.smithi099.stdout: "info": { 2023-05-31T20:00:53.237 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "sync", 2023-05-31T20:00:53.237 INFO:teuthology.orchestra.run.smithi099.stdout: "num_shards": 4, 2023-05-31T20:00:53.238 INFO:teuthology.orchestra.run.smithi099.stdout: "instance_id": 17941554979570305020 2023-05-31T20:00:53.238 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:00:53.239 INFO:teuthology.orchestra.run.smithi099.stdout: "markers": [ 2023-05-31T20:00:53.239 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T20:00:53.239 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 0, 2023-05-31T20:00:53.239 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T20:00:53.240 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "incremental-sync", 2023-05-31T20:00:53.240 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "00000000000000000000:00000000000000000929", 2023-05-31T20:00:53.240 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T20:00:53.241 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2023-05-31T20:00:53.241 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T20:00:53.241 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2023-05-31T19:57:00.436067Z" 2023-05-31T20:00:53.241 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T20:00:53.241 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:00:53.241 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T20:00:53.241 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 1, 2023-05-31T20:00:53.242 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T20:00:53.242 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "incremental-sync", 2023-05-31T20:00:53.242 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "", 2023-05-31T20:00:53.242 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T20:00:53.242 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2023-05-31T20:00:53.242 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T20:00:53.242 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "0.000000" 2023-05-31T20:00:53.243 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T20:00:53.243 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:00:53.243 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T20:00:53.243 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 2, 2023-05-31T20:00:53.243 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T20:00:53.244 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "incremental-sync", 2023-05-31T20:00:53.244 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "00000000000000000000:00000000000000002196", 2023-05-31T20:00:53.244 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T20:00:53.244 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2023-05-31T20:00:53.244 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T20:00:53.244 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2023-05-31T20:00:40.448120Z" 2023-05-31T20:00:53.244 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T20:00:53.245 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:00:53.245 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T20:00:53.245 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 3, 2023-05-31T20:00:53.245 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T20:00:53.245 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "incremental-sync", 2023-05-31T20:00:53.246 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "00000000000000000000:00000000000000002764", 2023-05-31T20:00:53.246 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T20:00:53.246 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2023-05-31T20:00:53.246 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T20:00:53.246 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2023-05-31T19:59:56.444860Z" 2023-05-31T20:00:53.246 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T20:00:53.247 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T20:00:53.247 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2023-05-31T20:00:53.247 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:00:53.247 INFO:teuthology.orchestra.run.smithi099.stdout: "full_sync": { 2023-05-31T20:00:53.247 INFO:teuthology.orchestra.run.smithi099.stdout: "total": 0, 2023-05-31T20:00:53.247 INFO:teuthology.orchestra.run.smithi099.stdout: "complete": 0 2023-05-31T20:00:53.247 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:00:53.248 INFO:teuthology.orchestra.run.smithi099.stdout: "current_time": "2023-05-31T20:00:53Z" 2023-05-31T20:00:53.248 INFO:teuthology.orchestra.run.smithi099.stdout:} 2023-05-31T20:00:53.249 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 17941554979570305020 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000000929", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-05-31T19:57:00.436067Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000002196", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-05-31T20:00:40.448120Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000002764", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-05-31T19:59:56.444860Z" } } ] }, "full_sync": { "total": 0, "complete": 0 }, "current_time": "2023-05-31T20:00:53Z" } 2023-05-31T20:00:53.249 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000000929', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-05-31T19:57:00.436067Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000002196', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-05-31T20:00:40.448120Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000002764', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-05-31T19:59:56.444860Z'}}] 2023-05-31T20:00:53.250 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000000929', 1: '', 2: '00000000000000000000:00000000000000002196', 3: '00000000000000000000:00000000000000002764'} 2023-05-31T20:00:53.250 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000000929', 1: '', 2: '00000000000000000000:00000000000000002196', 3: '00000000000000000000:00000000000000002764'} 2023-05-31T20:00:53.250 INFO:rgw_multi.tests:finished data checkpoint for target_zone=a1 source_zone=a2 2023-05-31T20:00:53.251 INFO:tasks.rgw_multisite_tests:rgw_multi.tests.test_concurrent_versioned_object_incremental_sync ... ok 2023-05-31T20:00:53.252 DEBUG:boto:path=/pklxkd-14/ 2023-05-31T20:00:53.253 DEBUG:boto:auth_path=/pklxkd-14/ 2023-05-31T20:00:53.253 DEBUG:boto:Method: PUT 2023-05-31T20:00:53.253 DEBUG:boto:Path: /pklxkd-14/ 2023-05-31T20:00:53.254 DEBUG:boto:Data: 2023-05-31T20:00:53.254 DEBUG:boto:Headers: {} 2023-05-31T20:00:53.254 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:00:53.255 DEBUG:boto:Port: 8000 2023-05-31T20:00:53.255 DEBUG:boto:Params: {} 2023-05-31T20:00:53.255 DEBUG:boto:Token: None 2023-05-31T20:00:53.256 DEBUG:boto:StringToSign: PUT Wed, 31 May 2023 20:00:53 GMT /pklxkd-14/ 2023-05-31T20:00:53.256 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:A7ZqAr25h56nMuySac1pe9gyDdc= 2023-05-31T20:00:53.257 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:00:53 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:A7ZqAr25h56nMuySac1pe9gyDdc=', 'Content-Length': '0'} 2023-05-31T20:00:53.278 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx000006627268c900cbb23-006477a775-110e-a1'), ('Content-Length', '0'), ('Date', 'Wed, 31 May 2023 20:00:53 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:00:53.279 DEBUG:rgw_multi.tests:created bucket=pklxkd-14 2023-05-31T20:00:53.279 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-31T20:00:53.280 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-31T20:00:53.280 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-05-31T20:00:53.329 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2023-05-31T20:00:53.329 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2023-05-31T20:00:53.373 INFO:teuthology.orchestra.run.smithi099.stderr:No --period given, using current period=2e6123d2-d695-4350-9a01-61a455bc782b 2023-05-31T20:00:53.390 INFO:teuthology.orchestra.run.smithi099.stdout:[ 2023-05-31T20:00:53.390 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T20:00:53.390 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1685562631.516089_59.1", 2023-05-31T20:00:53.391 INFO:teuthology.orchestra.run.smithi099.stdout: "last_update": "2023-05-31T19:50:31.516089Z" 2023-05-31T20:00:53.391 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:00:53.391 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T20:00:53.391 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "", 2023-05-31T20:00:53.391 INFO:teuthology.orchestra.run.smithi099.stdout: "last_update": "0.000000" 2023-05-31T20:00:53.391 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:00:53.392 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T20:00:53.392 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1685562970.274572_93.1", 2023-05-31T20:00:53.392 INFO:teuthology.orchestra.run.smithi099.stdout: "last_update": "2023-05-31T19:56:10.274572Z" 2023-05-31T20:00:53.392 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:00:53.392 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T20:00:53.392 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1685563253.271463_72.1", 2023-05-31T20:00:53.392 INFO:teuthology.orchestra.run.smithi099.stdout: "last_update": "2023-05-31T20:00:53.271463Z" 2023-05-31T20:00:53.393 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T20:00:53.393 INFO:teuthology.orchestra.run.smithi099.stdout:] 2023-05-31T20:00:53.394 DEBUG:rgw_multi.tests:master meta markers={0: '1_1685562631.516089_59.1', 1: '', 2: '1_1685562970.274572_93.1', 3: '1_1685563253.271463_72.1'} 2023-05-31T20:00:53.394 INFO:rgw_multi.tests:starting meta checkpoint for zone=a2 2023-05-31T20:00:53.394 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-31T20:00:53.395 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-31T20:00:53.395 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-05-31T20:00:53.446 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2023-05-31T20:00:53.446 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2023-05-31T20:00:53.494 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:00:53.488+0000 7f2167170780 20 RGW-SYNC:meta: read sync status 2023-05-31T20:00:53.495 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:00:53.489+0000 7f2167170780 20 RGW-SYNC:meta: read sync status 2023-05-31T20:00:53.506 INFO:teuthology.orchestra.run.smithi099.stdout:{ 2023-05-31T20:00:53.506 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_status": { 2023-05-31T20:00:53.506 INFO:teuthology.orchestra.run.smithi099.stdout: "info": { 2023-05-31T20:00:53.507 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "sync", 2023-05-31T20:00:53.507 INFO:teuthology.orchestra.run.smithi099.stdout: "num_shards": 4, 2023-05-31T20:00:53.507 INFO:teuthology.orchestra.run.smithi099.stdout: "period": "2e6123d2-d695-4350-9a01-61a455bc782b", 2023-05-31T20:00:53.507 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2023-05-31T20:00:53.507 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:00:53.507 INFO:teuthology.orchestra.run.smithi099.stdout: "markers": [ 2023-05-31T20:00:53.508 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T20:00:53.508 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 0, 2023-05-31T20:00:53.508 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T20:00:53.508 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2023-05-31T20:00:53.508 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1685562631.516089_59.1", 2023-05-31T20:00:53.508 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T20:00:53.509 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 1, 2023-05-31T20:00:53.509 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T20:00:53.509 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2023-05-31T19:50:31.516089Z", 2023-05-31T20:00:53.509 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2023-05-31T20:00:53.509 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T20:00:53.509 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:00:53.509 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T20:00:53.510 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 1, 2023-05-31T20:00:53.510 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T20:00:53.510 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2023-05-31T20:00:53.510 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "", 2023-05-31T20:00:53.510 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T20:00:53.510 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2023-05-31T20:00:53.511 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T20:00:53.511 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "0.000000", 2023-05-31T20:00:53.511 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2023-05-31T20:00:53.511 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T20:00:53.511 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:00:53.511 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T20:00:53.511 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 2, 2023-05-31T20:00:53.512 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T20:00:53.512 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2023-05-31T20:00:53.512 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1685562970.274572_93.1", 2023-05-31T20:00:53.512 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T20:00:53.512 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2023-05-31T20:00:53.512 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T20:00:53.512 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2023-05-31T19:56:10.274572Z", 2023-05-31T20:00:53.513 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2023-05-31T20:00:53.513 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T20:00:53.513 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:00:53.513 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T20:00:53.513 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 3, 2023-05-31T20:00:53.513 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T20:00:53.513 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2023-05-31T20:00:53.514 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1685563253.271463_72.1", 2023-05-31T20:00:53.514 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T20:00:53.514 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2023-05-31T20:00:53.514 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T20:00:53.514 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2023-05-31T20:00:53.271463Z", 2023-05-31T20:00:53.514 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2023-05-31T20:00:53.514 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T20:00:53.515 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T20:00:53.515 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2023-05-31T20:00:53.515 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:00:53.515 INFO:teuthology.orchestra.run.smithi099.stdout: "full_sync": { 2023-05-31T20:00:53.515 INFO:teuthology.orchestra.run.smithi099.stdout: "total": 1, 2023-05-31T20:00:53.515 INFO:teuthology.orchestra.run.smithi099.stdout: "complete": 1 2023-05-31T20:00:53.515 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:00:53.516 INFO:teuthology.orchestra.run.smithi099.stdout: "current_time": "2023-05-31T20:00:53Z" 2023-05-31T20:00:53.516 INFO:teuthology.orchestra.run.smithi099.stdout:} 2023-05-31T20:00:53.517 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "2e6123d2-d695-4350-9a01-61a455bc782b", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1685562631.516089_59.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-05-31T19:50:31.516089Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1685562970.274572_93.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-05-31T19:56:10.274572Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1685563253.271463_72.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-05-31T20:00:53.271463Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-05-31T20:00:53Z" } 2023-05-31T20:00:53.517 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1685562631.516089_59.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-05-31T19:50:31.516089Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1685562970.274572_93.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-05-31T19:56:10.274572Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1685563253.271463_72.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-05-31T20:00:53.271463Z', 'realm_epoch': 2}}] 2023-05-31T20:00:53.518 DEBUG:rgw_multi.tests:log_status={0: '1_1685562631.516089_59.1', 1: '', 2: '1_1685562970.274572_93.1', 3: '1_1685563253.271463_72.1'} 2023-05-31T20:00:53.518 DEBUG:rgw_multi.tests:sync_status={0: '1_1685562631.516089_59.1', 1: '', 2: '1_1685562970.274572_93.1', 3: '1_1685563253.271463_72.1'} 2023-05-31T20:00:53.518 INFO:rgw_multi.tests:finish meta checkpoint for zone=a2 2023-05-31T20:00:53.519 DEBUG:boto:path=/pklxkd-14/ 2023-05-31T20:00:53.519 DEBUG:boto:auth_path=/pklxkd-14/ 2023-05-31T20:00:53.519 DEBUG:boto:Method: HEAD 2023-05-31T20:00:53.520 DEBUG:boto:Path: /pklxkd-14/ 2023-05-31T20:00:53.520 DEBUG:boto:Data: 2023-05-31T20:00:53.520 DEBUG:boto:Headers: {} 2023-05-31T20:00:53.521 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:00:53.521 DEBUG:boto:Port: 8000 2023-05-31T20:00:53.521 DEBUG:boto:Params: {} 2023-05-31T20:00:53.522 DEBUG:boto:Token: None 2023-05-31T20:00:53.522 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:00:53 GMT /pklxkd-14/ 2023-05-31T20:00:53.522 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:Wo5RwaoLIIc3QauCnodd1uiVr/U= 2023-05-31T20:00:53.523 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:00:53 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:Wo5RwaoLIIc3QauCnodd1uiVr/U=', 'Content-Length': '0'} 2023-05-31T20:00:53.525 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000002edcd67e8d996f6e-006477a775-110e-a1'), ('Content-Length', '0'), ('Date', 'Wed, 31 May 2023 20:00:53 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:00:53.526 DEBUG:boto:path=/pklxkd-14/obj 2023-05-31T20:00:53.526 DEBUG:boto:auth_path=/pklxkd-14/obj 2023-05-31T20:00:53.527 DEBUG:boto:Method: PUT 2023-05-31T20:00:53.527 DEBUG:boto:Path: /pklxkd-14/obj 2023-05-31T20:00:53.527 DEBUG:boto:Data: 2023-05-31T20:00:53.528 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': '1B2M2Y8AsgTpgAmY7PhCfg==', 'Content-Length': '0', 'Expect': '100-Continue'} 2023-05-31T20:00:53.528 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:00:53.528 DEBUG:boto:Port: 8000 2023-05-31T20:00:53.529 DEBUG:boto:Params: {} 2023-05-31T20:00:53.529 DEBUG:boto:Token: None 2023-05-31T20:00:53.529 DEBUG:boto:StringToSign: PUT 1B2M2Y8AsgTpgAmY7PhCfg== application/octet-stream Wed, 31 May 2023 20:00:53 GMT /pklxkd-14/obj 2023-05-31T20:00:53.530 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:lQnLoqEIlPSDy15EWvPQp/FDqWY= 2023-05-31T20:00:53.530 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': '1B2M2Y8AsgTpgAmY7PhCfg==', 'Content-Length': '0', 'Expect': '100-Continue', 'Date': 'Wed, 31 May 2023 20:00:53 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:lQnLoqEIlPSDy15EWvPQp/FDqWY='} 2023-05-31T20:00:53.572 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000021ce2d44c6c3c945-006477a775-110e-a1'), ('Date', 'Wed, 31 May 2023 20:00:53 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:00:53.573 DEBUG:rgw_multi.tests:created initial version id=None 2023-05-31T20:00:53.574 DEBUG:rgw_multi.tests:bucket checkpoint: source=a1 target=a2 bucket=pklxkd-14 2023-05-31T20:00:53.574 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'pklxkd-14', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-05-31T20:00:53.575 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'pklxkd-14', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-05-31T20:00:53.575 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket pklxkd-14 --source-zone a1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2023-05-31T20:00:53.631 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2023-05-31T20:00:53.632 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2023-05-31T20:00:53.688 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:00:53.682+0000 7f30327f1780 1 waiting to reach incremental sync.. 2023-05-31T20:00:58.689 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:00:58.683+0000 7f30327f1780 1 waiting to reach incremental sync.. 2023-05-31T20:01:03.689 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:01:03.684+0000 7f30327f1780 1 waiting to reach incremental sync.. 2023-05-31T20:01:08.691 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:01:08.686+0000 7f30327f1780 1 bucket sync caught up with source: 2023-05-31T20:01:08.692 INFO:teuthology.orchestra.run.smithi099.stderr: local status: [, , , , , , , , 00000000001.5.6, , ] 2023-05-31T20:01:08.692 INFO:teuthology.orchestra.run.smithi099.stderr: remote markers: [, , , , , , , , 00000000001.5.6, , ] 2023-05-31T20:01:08.692 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:01:08.686+0000 7f30327f1780 0 bucket checkpoint complete 2023-05-31T20:01:08.713 DEBUG:rgw_multi.tests:bucket checkpoint: source=a2 target=a1 bucket=pklxkd-14 2023-05-31T20:01:08.715 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'pklxkd-14', '--source-zone', 'a2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-05-31T20:01:08.716 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'pklxkd-14', '--source-zone', 'a2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-05-31T20:01:08.717 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket pklxkd-14 --source-zone a2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2023-05-31T20:01:08.781 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2023-05-31T20:01:08.781 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2023-05-31T20:01:08.833 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:01:08.828+0000 7fa5ee216780 1 waiting to reach incremental sync.. 2023-05-31T20:01:13.834 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:01:13.829+0000 7fa5ee216780 1 waiting to reach incremental sync.. 2023-05-31T20:01:18.834 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:01:18.829+0000 7fa5ee216780 1 waiting to reach incremental sync.. 2023-05-31T20:01:23.835 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:01:23.830+0000 7fa5ee216780 1 waiting to reach incremental sync.. 2023-05-31T20:01:28.837 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:01:28.831+0000 7fa5ee216780 1 bucket sync caught up with source: 2023-05-31T20:01:28.837 INFO:teuthology.orchestra.run.smithi099.stderr: local status: [, , , , , , , , 00000000001.7.6, , ] 2023-05-31T20:01:28.837 INFO:teuthology.orchestra.run.smithi099.stderr: remote markers: [, , , , , , , , 00000000001.7.6, , ] 2023-05-31T20:01:28.837 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:01:28.831+0000 7fa5ee216780 0 bucket checkpoint complete 2023-05-31T20:01:28.861 INFO:tasks.rgw_multi.zone_rados:comparing bucket=pklxkd-14 zones={a2, a1} 2023-05-31T20:01:28.861 DEBUG:boto:path=/pklxkd-14/ 2023-05-31T20:01:28.862 DEBUG:boto:auth_path=/pklxkd-14/ 2023-05-31T20:01:28.862 DEBUG:boto:Method: HEAD 2023-05-31T20:01:28.863 DEBUG:boto:Path: /pklxkd-14/ 2023-05-31T20:01:28.863 DEBUG:boto:Data: 2023-05-31T20:01:28.863 DEBUG:boto:Headers: {} 2023-05-31T20:01:28.864 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:01:28.864 DEBUG:boto:Port: 8001 2023-05-31T20:01:28.865 DEBUG:boto:Params: {} 2023-05-31T20:01:28.865 DEBUG:boto:Token: None 2023-05-31T20:01:28.865 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:01:28 GMT /pklxkd-14/ 2023-05-31T20:01:28.866 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:/5DJ3Vt+7vw2BXYs0/bkxHSDlco= 2023-05-31T20:01:28.866 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:01:28 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:/5DJ3Vt+7vw2BXYs0/bkxHSDlco=', 'Content-Length': '0'} 2023-05-31T20:01:28.869 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000004f10449f2dd28aa7-006477a798-1112-a2'), ('Content-Length', '0'), ('Date', 'Wed, 31 May 2023 20:01:28 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:01:28.869 DEBUG:boto:path=/pklxkd-14/ 2023-05-31T20:01:28.870 DEBUG:boto:auth_path=/pklxkd-14/ 2023-05-31T20:01:28.870 DEBUG:boto:Method: HEAD 2023-05-31T20:01:28.870 DEBUG:boto:Path: /pklxkd-14/ 2023-05-31T20:01:28.871 DEBUG:boto:Data: 2023-05-31T20:01:28.871 DEBUG:boto:Headers: {} 2023-05-31T20:01:28.871 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:01:28.872 DEBUG:boto:Port: 8000 2023-05-31T20:01:28.872 DEBUG:boto:Params: {} 2023-05-31T20:01:28.872 DEBUG:boto:Token: None 2023-05-31T20:01:28.873 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:01:28 GMT /pklxkd-14/ 2023-05-31T20:01:28.873 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:/5DJ3Vt+7vw2BXYs0/bkxHSDlco= 2023-05-31T20:01:28.874 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:01:28 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:/5DJ3Vt+7vw2BXYs0/bkxHSDlco=', 'Content-Length': '0'} 2023-05-31T20:01:28.876 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000db9f02595a2dec22-006477a798-110e-a1'), ('Content-Length', '0'), ('Date', 'Wed, 31 May 2023 20:01:28 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:01:28.876 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2023-05-31T20:01:28.877 DEBUG:boto:path=/pklxkd-14/ 2023-05-31T20:01:28.877 DEBUG:boto:auth_path=/pklxkd-14/ 2023-05-31T20:01:28.878 DEBUG:boto:path=/pklxkd-14/?versions&max-keys=999 2023-05-31T20:01:28.878 DEBUG:boto:auth_path=/pklxkd-14/?versions&max-keys=999 2023-05-31T20:01:28.878 DEBUG:boto:Method: GET 2023-05-31T20:01:28.879 DEBUG:boto:Path: /pklxkd-14/?versions&max-keys=999 2023-05-31T20:01:28.879 DEBUG:boto:Data: 2023-05-31T20:01:28.879 DEBUG:boto:Headers: {} 2023-05-31T20:01:28.880 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:01:28.880 DEBUG:boto:Port: 8001 2023-05-31T20:01:28.880 DEBUG:boto:Params: {} 2023-05-31T20:01:28.881 DEBUG:boto:Token: None 2023-05-31T20:01:28.881 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:01:28 GMT /pklxkd-14/?versions 2023-05-31T20:01:28.882 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:OkK+mx40BkwNGcJtcrMbJgMmdUk= 2023-05-31T20:01:28.882 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:01:28 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:OkK+mx40BkwNGcJtcrMbJgMmdUk=', 'Content-Length': '0'} 2023-05-31T20:01:28.885 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000caaf99ff7293a2bf-006477a798-1112-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:01:28 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:01:28.885 DEBUG:boto:b'pklxkd-14999falseobjnulltrue2023-05-31T20:00:53.528Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:01:28.886 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2023-05-31T20:01:28.887 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2023-05-31T20:01:28.887 DEBUG:boto:path=/pklxkd-14/ 2023-05-31T20:01:28.887 DEBUG:boto:auth_path=/pklxkd-14/ 2023-05-31T20:01:28.888 DEBUG:boto:path=/pklxkd-14/?versions&max-keys=999 2023-05-31T20:01:28.888 DEBUG:boto:auth_path=/pklxkd-14/?versions&max-keys=999 2023-05-31T20:01:28.888 DEBUG:boto:Method: GET 2023-05-31T20:01:28.889 DEBUG:boto:Path: /pklxkd-14/?versions&max-keys=999 2023-05-31T20:01:28.889 DEBUG:boto:Data: 2023-05-31T20:01:28.889 DEBUG:boto:Headers: {} 2023-05-31T20:01:28.890 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:01:28.890 DEBUG:boto:Port: 8000 2023-05-31T20:01:28.890 DEBUG:boto:Params: {} 2023-05-31T20:01:28.891 DEBUG:boto:Token: None 2023-05-31T20:01:28.891 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:01:28 GMT /pklxkd-14/?versions 2023-05-31T20:01:28.892 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:OkK+mx40BkwNGcJtcrMbJgMmdUk= 2023-05-31T20:01:28.892 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:01:28 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:OkK+mx40BkwNGcJtcrMbJgMmdUk=', 'Content-Length': '0'} 2023-05-31T20:01:28.895 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000061a973d87144b0da-006477a798-110e-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:01:28 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:01:28.896 DEBUG:boto:b'pklxkd-14999falseobjnulltrue2023-05-31T20:00:53.528Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:01:28.897 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2023-05-31T20:01:28.898 DEBUG:boto:path=/pklxkd-14/ 2023-05-31T20:01:28.899 DEBUG:boto:auth_path=/pklxkd-14/ 2023-05-31T20:01:28.900 DEBUG:boto:path=/pklxkd-14/?versions&max-keys=999 2023-05-31T20:01:28.900 DEBUG:boto:auth_path=/pklxkd-14/?versions&max-keys=999 2023-05-31T20:01:28.901 DEBUG:boto:Method: GET 2023-05-31T20:01:28.902 DEBUG:boto:Path: /pklxkd-14/?versions&max-keys=999 2023-05-31T20:01:28.902 DEBUG:boto:Data: 2023-05-31T20:01:28.903 DEBUG:boto:Headers: {} 2023-05-31T20:01:28.904 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:01:28.905 DEBUG:boto:Port: 8001 2023-05-31T20:01:28.905 DEBUG:boto:Params: {} 2023-05-31T20:01:28.905 DEBUG:boto:Token: None 2023-05-31T20:01:28.906 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:01:28 GMT /pklxkd-14/?versions 2023-05-31T20:01:28.906 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:OkK+mx40BkwNGcJtcrMbJgMmdUk= 2023-05-31T20:01:28.907 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:01:28 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:OkK+mx40BkwNGcJtcrMbJgMmdUk=', 'Content-Length': '0'} 2023-05-31T20:01:28.909 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000068df4429106c62aa-006477a798-1112-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:01:28 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:01:28.956 DEBUG:boto:b'pklxkd-14999falseobjnulltrue2023-05-31T20:00:53.528Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:01:28.957 DEBUG:boto:path=/pklxkd-14/ 2023-05-31T20:01:28.958 DEBUG:boto:auth_path=/pklxkd-14/ 2023-05-31T20:01:28.958 DEBUG:boto:path=/pklxkd-14/?versions&max-keys=999 2023-05-31T20:01:28.959 DEBUG:boto:auth_path=/pklxkd-14/?versions&max-keys=999 2023-05-31T20:01:28.959 DEBUG:boto:Method: GET 2023-05-31T20:01:28.960 DEBUG:boto:Path: /pklxkd-14/?versions&max-keys=999 2023-05-31T20:01:28.960 DEBUG:boto:Data: 2023-05-31T20:01:28.961 DEBUG:boto:Headers: {} 2023-05-31T20:01:28.961 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:01:28.962 DEBUG:boto:Port: 8000 2023-05-31T20:01:28.962 DEBUG:boto:Params: {} 2023-05-31T20:01:28.963 DEBUG:boto:Token: None 2023-05-31T20:01:28.963 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:01:28 GMT /pklxkd-14/?versions 2023-05-31T20:01:28.964 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:OkK+mx40BkwNGcJtcrMbJgMmdUk= 2023-05-31T20:01:28.964 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:01:28 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:OkK+mx40BkwNGcJtcrMbJgMmdUk=', 'Content-Length': '0'} 2023-05-31T20:01:28.967 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000044f0ec28367e08df-006477a798-110e-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:01:28 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:01:28.967 DEBUG:boto:b'pklxkd-14999falseobjnulltrue2023-05-31T20:00:53.528Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:01:28.968 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2023-05-31T20:01:28.969 DEBUG:boto:path=/pklxkd-14/obj 2023-05-31T20:01:28.969 DEBUG:boto:auth_path=/pklxkd-14/obj 2023-05-31T20:01:28.970 DEBUG:boto:path=/pklxkd-14/obj?versionId=null 2023-05-31T20:01:28.970 DEBUG:boto:auth_path=/pklxkd-14/obj?versionId=null 2023-05-31T20:01:28.970 DEBUG:boto:Method: GET 2023-05-31T20:01:28.971 DEBUG:boto:Path: /pklxkd-14/obj?versionId=null 2023-05-31T20:01:28.971 DEBUG:boto:Data: 2023-05-31T20:01:28.971 DEBUG:boto:Headers: {} 2023-05-31T20:01:28.972 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:01:28.972 DEBUG:boto:Port: 8001 2023-05-31T20:01:28.973 DEBUG:boto:Params: {} 2023-05-31T20:01:28.973 DEBUG:boto:Token: None 2023-05-31T20:01:28.974 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:01:28 GMT /pklxkd-14/obj?versionId=null 2023-05-31T20:01:28.975 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:pET94+S+1U0GZ3ITmoGmOWxw1Ec= 2023-05-31T20:01:28.976 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:01:28 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:pET94+S+1U0GZ3ITmoGmOWxw1Ec=', 'Content-Length': '0'} 2023-05-31T20:01:28.979 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:00:53 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000006fafb1e7b0d49e38-006477a798-1112-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:01:28 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:01:28.980 DEBUG:boto:path=/pklxkd-14/obj 2023-05-31T20:01:28.981 DEBUG:boto:auth_path=/pklxkd-14/obj 2023-05-31T20:01:28.981 DEBUG:boto:path=/pklxkd-14/obj?versionId=null 2023-05-31T20:01:28.982 DEBUG:boto:auth_path=/pklxkd-14/obj?versionId=null 2023-05-31T20:01:28.983 DEBUG:boto:Method: GET 2023-05-31T20:01:28.983 DEBUG:boto:Path: /pklxkd-14/obj?versionId=null 2023-05-31T20:01:28.984 DEBUG:boto:Data: 2023-05-31T20:01:28.984 DEBUG:boto:Headers: {} 2023-05-31T20:01:28.985 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:01:28.985 DEBUG:boto:Port: 8000 2023-05-31T20:01:28.985 DEBUG:boto:Params: {} 2023-05-31T20:01:28.986 DEBUG:boto:Token: None 2023-05-31T20:01:28.986 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:01:28 GMT /pklxkd-14/obj?versionId=null 2023-05-31T20:01:28.987 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:pET94+S+1U0GZ3ITmoGmOWxw1Ec= 2023-05-31T20:01:28.987 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:01:28 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:pET94+S+1U0GZ3ITmoGmOWxw1Ec=', 'Content-Length': '0'} 2023-05-31T20:01:28.989 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:00:53 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000005754503a636e8095-006477a798-110e-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:01:28 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:01:28.990 DEBUG:boto:path=/pklxkd-14/obj 2023-05-31T20:01:28.990 DEBUG:boto:auth_path=/pklxkd-14/obj 2023-05-31T20:01:28.991 DEBUG:boto:path=/pklxkd-14/obj?versionId=null 2023-05-31T20:01:28.991 DEBUG:boto:auth_path=/pklxkd-14/obj?versionId=null 2023-05-31T20:01:28.992 DEBUG:boto:Method: HEAD 2023-05-31T20:01:28.993 DEBUG:boto:Path: /pklxkd-14/obj?versionId=null 2023-05-31T20:01:28.994 DEBUG:boto:Data: 2023-05-31T20:01:28.994 DEBUG:boto:Headers: {} 2023-05-31T20:01:28.995 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:01:28.996 DEBUG:boto:Port: 8001 2023-05-31T20:01:28.996 DEBUG:boto:Params: {} 2023-05-31T20:01:28.996 DEBUG:boto:Token: None 2023-05-31T20:01:28.997 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:01:28 GMT /pklxkd-14/obj?versionId=null 2023-05-31T20:01:28.997 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:ZlYkjDpYo4kf749qeOadKxmDpRo= 2023-05-31T20:01:28.998 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:01:28 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:ZlYkjDpYo4kf749qeOadKxmDpRo=', 'Content-Length': '0'} 2023-05-31T20:01:29.000 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:00:53 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000574d8527e78d55b5-006477a798-1112-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:01:28 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:01:29.000 DEBUG:boto:path=/pklxkd-14/obj 2023-05-31T20:01:29.001 DEBUG:boto:auth_path=/pklxkd-14/obj 2023-05-31T20:01:29.001 DEBUG:boto:path=/pklxkd-14/obj?versionId=null 2023-05-31T20:01:29.001 DEBUG:boto:auth_path=/pklxkd-14/obj?versionId=null 2023-05-31T20:01:29.002 DEBUG:boto:Method: HEAD 2023-05-31T20:01:29.002 DEBUG:boto:Path: /pklxkd-14/obj?versionId=null 2023-05-31T20:01:29.002 DEBUG:boto:Data: 2023-05-31T20:01:29.003 DEBUG:boto:Headers: {} 2023-05-31T20:01:29.003 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:01:29.003 DEBUG:boto:Port: 8000 2023-05-31T20:01:29.004 DEBUG:boto:Params: {} 2023-05-31T20:01:29.004 DEBUG:boto:Token: None 2023-05-31T20:01:29.004 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:01:29 GMT /pklxkd-14/obj?versionId=null 2023-05-31T20:01:29.005 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:uIPhmawBZk94zMy6jUli0bpecyQ= 2023-05-31T20:01:29.005 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:01:29 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:uIPhmawBZk94zMy6jUli0bpecyQ=', 'Content-Length': '0'} 2023-05-31T20:01:29.007 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:00:53 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000003c5b6c403ad30d5a-006477a799-110e-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:01:29 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:01:29.008 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2023-05-31T20:01:29.008 DEBUG:boto:path=/pklxkd-14/obj 2023-05-31T20:01:29.008 DEBUG:boto:auth_path=/pklxkd-14/obj 2023-05-31T20:01:29.009 DEBUG:boto:path=/pklxkd-14/obj?versionId=null 2023-05-31T20:01:29.009 DEBUG:boto:auth_path=/pklxkd-14/obj?versionId=null 2023-05-31T20:01:29.010 DEBUG:boto:Method: GET 2023-05-31T20:01:29.010 DEBUG:boto:Path: /pklxkd-14/obj?versionId=null 2023-05-31T20:01:29.010 DEBUG:boto:Data: 2023-05-31T20:01:29.011 DEBUG:boto:Headers: {} 2023-05-31T20:01:29.011 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:01:29.011 DEBUG:boto:Port: 8001 2023-05-31T20:01:29.012 DEBUG:boto:Params: {} 2023-05-31T20:01:29.012 DEBUG:boto:Token: None 2023-05-31T20:01:29.013 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:01:29 GMT /pklxkd-14/obj?versionId=null 2023-05-31T20:01:29.013 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:58WeqBPN8aukGAJcFuOtaeOHY48= 2023-05-31T20:01:29.014 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:01:29 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:58WeqBPN8aukGAJcFuOtaeOHY48=', 'Content-Length': '0'} 2023-05-31T20:01:29.016 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:00:53 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000001a4a1fb187a55a6c-006477a799-1112-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:01:29 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:01:29.016 DEBUG:boto:path=/pklxkd-14/obj 2023-05-31T20:01:29.017 DEBUG:boto:auth_path=/pklxkd-14/obj 2023-05-31T20:01:29.017 DEBUG:boto:path=/pklxkd-14/obj?versionId=null 2023-05-31T20:01:29.017 DEBUG:boto:auth_path=/pklxkd-14/obj?versionId=null 2023-05-31T20:01:29.018 DEBUG:boto:Method: GET 2023-05-31T20:01:29.018 DEBUG:boto:Path: /pklxkd-14/obj?versionId=null 2023-05-31T20:01:29.018 DEBUG:boto:Data: 2023-05-31T20:01:29.019 DEBUG:boto:Headers: {} 2023-05-31T20:01:29.019 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:01:29.020 DEBUG:boto:Port: 8000 2023-05-31T20:01:29.020 DEBUG:boto:Params: {} 2023-05-31T20:01:29.020 DEBUG:boto:Token: None 2023-05-31T20:01:29.021 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:01:29 GMT /pklxkd-14/obj?versionId=null 2023-05-31T20:01:29.021 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:58WeqBPN8aukGAJcFuOtaeOHY48= 2023-05-31T20:01:29.022 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:01:29 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:58WeqBPN8aukGAJcFuOtaeOHY48=', 'Content-Length': '0'} 2023-05-31T20:01:29.023 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:00:53 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000db635cdac34059d4-006477a799-110e-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:01:29 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:01:29.024 DEBUG:boto:path=/pklxkd-14/ 2023-05-31T20:01:29.025 DEBUG:boto:auth_path=/pklxkd-14/ 2023-05-31T20:01:29.025 DEBUG:boto:path=/pklxkd-14/?prefix=obj 2023-05-31T20:01:29.025 DEBUG:boto:auth_path=/pklxkd-14/?prefix=obj 2023-05-31T20:01:29.026 DEBUG:boto:Method: GET 2023-05-31T20:01:29.026 DEBUG:boto:Path: /pklxkd-14/?prefix=obj 2023-05-31T20:01:29.026 DEBUG:boto:Data: 2023-05-31T20:01:29.027 DEBUG:boto:Headers: {} 2023-05-31T20:01:29.027 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:01:29.028 DEBUG:boto:Port: 8001 2023-05-31T20:01:29.028 DEBUG:boto:Params: {} 2023-05-31T20:01:29.028 DEBUG:boto:Token: None 2023-05-31T20:01:29.029 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:01:29 GMT /pklxkd-14/ 2023-05-31T20:01:29.029 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:TQmgubpeUC+u+FxEuLFm/hGarDA= 2023-05-31T20:01:29.030 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:01:29 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:TQmgubpeUC+u+FxEuLFm/hGarDA=', 'Content-Length': '0'} 2023-05-31T20:01:29.032 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000053621f8650804c81-006477a799-1112-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:01:29 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:01:29.033 DEBUG:boto:b'pklxkd-14obj1000falseobj2023-05-31T20:00:53.528Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:01:29.033 DEBUG:boto:path=/pklxkd-14/ 2023-05-31T20:01:29.034 DEBUG:boto:auth_path=/pklxkd-14/ 2023-05-31T20:01:29.034 DEBUG:boto:path=/pklxkd-14/?prefix=obj 2023-05-31T20:01:29.035 DEBUG:boto:auth_path=/pklxkd-14/?prefix=obj 2023-05-31T20:01:29.035 DEBUG:boto:Method: GET 2023-05-31T20:01:29.036 DEBUG:boto:Path: /pklxkd-14/?prefix=obj 2023-05-31T20:01:29.036 DEBUG:boto:Data: 2023-05-31T20:01:29.037 DEBUG:boto:Headers: {} 2023-05-31T20:01:29.037 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:01:29.037 DEBUG:boto:Port: 8000 2023-05-31T20:01:29.038 DEBUG:boto:Params: {} 2023-05-31T20:01:29.038 DEBUG:boto:Token: None 2023-05-31T20:01:29.039 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:01:29 GMT /pklxkd-14/ 2023-05-31T20:01:29.039 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:TQmgubpeUC+u+FxEuLFm/hGarDA= 2023-05-31T20:01:29.040 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:01:29 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:TQmgubpeUC+u+FxEuLFm/hGarDA=', 'Content-Length': '0'} 2023-05-31T20:01:29.042 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000439b5a110a0007db-006477a799-110e-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:01:29 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:01:29.043 DEBUG:boto:b'pklxkd-14obj1000falseobj2023-05-31T20:00:53.528Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:01:29.043 DEBUG:boto:path=/pklxkd-14/obj 2023-05-31T20:01:29.044 DEBUG:boto:auth_path=/pklxkd-14/obj 2023-05-31T20:01:29.044 DEBUG:boto:Method: HEAD 2023-05-31T20:01:29.045 DEBUG:boto:Path: /pklxkd-14/obj 2023-05-31T20:01:29.045 DEBUG:boto:Data: 2023-05-31T20:01:29.045 DEBUG:boto:Headers: {} 2023-05-31T20:01:29.046 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:01:29.046 DEBUG:boto:Port: 8001 2023-05-31T20:01:29.046 DEBUG:boto:Params: {} 2023-05-31T20:01:29.047 DEBUG:boto:Token: None 2023-05-31T20:01:29.047 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:01:29 GMT /pklxkd-14/obj 2023-05-31T20:01:29.048 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:ceDCfQH122PlPGi8FTTfRMRauFY= 2023-05-31T20:01:29.048 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:01:29 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:ceDCfQH122PlPGi8FTTfRMRauFY=', 'Content-Length': '0'} 2023-05-31T20:01:29.050 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:00:53 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000001bf48adab7be92a1-006477a799-1112-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:01:29 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:01:29.051 DEBUG:boto:path=/pklxkd-14/obj 2023-05-31T20:01:29.051 DEBUG:boto:auth_path=/pklxkd-14/obj 2023-05-31T20:01:29.052 DEBUG:boto:Method: HEAD 2023-05-31T20:01:29.053 DEBUG:boto:Path: /pklxkd-14/obj 2023-05-31T20:01:29.053 DEBUG:boto:Data: 2023-05-31T20:01:29.055 DEBUG:boto:Headers: {} 2023-05-31T20:01:29.055 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:01:29.056 DEBUG:boto:Port: 8000 2023-05-31T20:01:29.057 DEBUG:boto:Params: {} 2023-05-31T20:01:29.057 DEBUG:boto:Token: None 2023-05-31T20:01:29.057 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:01:29 GMT /pklxkd-14/obj 2023-05-31T20:01:29.058 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:ceDCfQH122PlPGi8FTTfRMRauFY= 2023-05-31T20:01:29.058 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:01:29 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:ceDCfQH122PlPGi8FTTfRMRauFY=', 'Content-Length': '0'} 2023-05-31T20:01:29.060 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:00:53 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000008b8113af6064d9de-006477a799-110e-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:01:29 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:01:29.060 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2023-05-31T20:01:29.061 DEBUG:boto:path=/pklxkd-14/obj 2023-05-31T20:01:29.061 DEBUG:boto:auth_path=/pklxkd-14/obj 2023-05-31T20:01:29.062 DEBUG:boto:Method: GET 2023-05-31T20:01:29.062 DEBUG:boto:Path: /pklxkd-14/obj 2023-05-31T20:01:29.062 DEBUG:boto:Data: 2023-05-31T20:01:29.063 DEBUG:boto:Headers: {} 2023-05-31T20:01:29.063 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:01:29.063 DEBUG:boto:Port: 8001 2023-05-31T20:01:29.064 DEBUG:boto:Params: {} 2023-05-31T20:01:29.064 DEBUG:boto:Token: None 2023-05-31T20:01:29.065 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:01:29 GMT /pklxkd-14/obj 2023-05-31T20:01:29.065 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:q4r5mTpyUPbPmYvbhrwTgxmE8yQ= 2023-05-31T20:01:29.065 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:01:29 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:q4r5mTpyUPbPmYvbhrwTgxmE8yQ=', 'Content-Length': '0'} 2023-05-31T20:01:29.067 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:00:53 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000cf543baa08e4a24b-006477a799-1112-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:01:29 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:01:29.068 DEBUG:boto:path=/pklxkd-14/obj 2023-05-31T20:01:29.068 DEBUG:boto:auth_path=/pklxkd-14/obj 2023-05-31T20:01:29.068 DEBUG:boto:Method: GET 2023-05-31T20:01:29.069 DEBUG:boto:Path: /pklxkd-14/obj 2023-05-31T20:01:29.069 DEBUG:boto:Data: 2023-05-31T20:01:29.070 DEBUG:boto:Headers: {} 2023-05-31T20:01:29.070 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:01:29.070 DEBUG:boto:Port: 8000 2023-05-31T20:01:29.071 DEBUG:boto:Params: {} 2023-05-31T20:01:29.071 DEBUG:boto:Token: None 2023-05-31T20:01:29.071 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:01:29 GMT /pklxkd-14/obj 2023-05-31T20:01:29.072 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:q4r5mTpyUPbPmYvbhrwTgxmE8yQ= 2023-05-31T20:01:29.072 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:01:29 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:q4r5mTpyUPbPmYvbhrwTgxmE8yQ=', 'Content-Length': '0'} 2023-05-31T20:01:29.074 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:00:53 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000189ae8cae1e78691-006477a799-110e-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:01:29 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:01:29.075 DEBUG:boto:path=/pklxkd-14/ 2023-05-31T20:01:29.075 DEBUG:boto:auth_path=/pklxkd-14/ 2023-05-31T20:01:29.075 DEBUG:boto:path=/pklxkd-14/?prefix=obj 2023-05-31T20:01:29.076 DEBUG:boto:auth_path=/pklxkd-14/?prefix=obj 2023-05-31T20:01:29.076 DEBUG:boto:Method: GET 2023-05-31T20:01:29.076 DEBUG:boto:Path: /pklxkd-14/?prefix=obj 2023-05-31T20:01:29.077 DEBUG:boto:Data: 2023-05-31T20:01:29.077 DEBUG:boto:Headers: {} 2023-05-31T20:01:29.077 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:01:29.078 DEBUG:boto:Port: 8001 2023-05-31T20:01:29.078 DEBUG:boto:Params: {} 2023-05-31T20:01:29.079 DEBUG:boto:Token: None 2023-05-31T20:01:29.079 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:01:29 GMT /pklxkd-14/ 2023-05-31T20:01:29.079 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:TQmgubpeUC+u+FxEuLFm/hGarDA= 2023-05-31T20:01:29.080 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:01:29 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:TQmgubpeUC+u+FxEuLFm/hGarDA=', 'Content-Length': '0'} 2023-05-31T20:01:29.082 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000efea90f2e1fe0778-006477a799-1112-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:01:29 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:01:29.082 DEBUG:boto:b'pklxkd-14obj1000falseobj2023-05-31T20:00:53.528Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:01:29.083 DEBUG:boto:path=/pklxkd-14/ 2023-05-31T20:01:29.083 DEBUG:boto:auth_path=/pklxkd-14/ 2023-05-31T20:01:29.084 DEBUG:boto:path=/pklxkd-14/?prefix=obj 2023-05-31T20:01:29.084 DEBUG:boto:auth_path=/pklxkd-14/?prefix=obj 2023-05-31T20:01:29.085 DEBUG:boto:Method: GET 2023-05-31T20:01:29.085 DEBUG:boto:Path: /pklxkd-14/?prefix=obj 2023-05-31T20:01:29.085 DEBUG:boto:Data: 2023-05-31T20:01:29.086 DEBUG:boto:Headers: {} 2023-05-31T20:01:29.086 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:01:29.086 DEBUG:boto:Port: 8000 2023-05-31T20:01:29.087 DEBUG:boto:Params: {} 2023-05-31T20:01:29.087 DEBUG:boto:Token: None 2023-05-31T20:01:29.088 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:01:29 GMT /pklxkd-14/ 2023-05-31T20:01:29.088 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:TQmgubpeUC+u+FxEuLFm/hGarDA= 2023-05-31T20:01:29.088 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:01:29 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:TQmgubpeUC+u+FxEuLFm/hGarDA=', 'Content-Length': '0'} 2023-05-31T20:01:29.091 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ff5890fc21cef68b-006477a799-110e-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:01:29 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:01:29.091 DEBUG:boto:b'pklxkd-14obj1000falseobj2023-05-31T20:00:53.528Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:01:29.092 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=pklxkd-14 zones={a2, a1} 2023-05-31T20:01:29.092 DEBUG:boto:path=/pklxkd-14/ 2023-05-31T20:01:29.093 DEBUG:boto:auth_path=/pklxkd-14/ 2023-05-31T20:01:29.093 DEBUG:boto:path=/pklxkd-14/?versioning 2023-05-31T20:01:29.093 DEBUG:boto:auth_path=/pklxkd-14/?versioning 2023-05-31T20:01:29.094 DEBUG:boto:Method: PUT 2023-05-31T20:01:29.094 DEBUG:boto:Path: /pklxkd-14/?versioning 2023-05-31T20:01:29.094 DEBUG:boto:Data: Enabled Disabled 2023-05-31T20:01:29.094 DEBUG:boto:Headers: {} 2023-05-31T20:01:29.095 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:01:29.095 DEBUG:boto:Port: 8000 2023-05-31T20:01:29.095 DEBUG:boto:Params: {} 2023-05-31T20:01:29.096 DEBUG:boto:Token: None 2023-05-31T20:01:29.096 DEBUG:boto:StringToSign: PUT Wed, 31 May 2023 20:01:29 GMT /pklxkd-14/?versioning 2023-05-31T20:01:29.097 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:IXHBg1GsEDTODCjcptC/fj1KPao= 2023-05-31T20:01:29.097 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:01:29 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:IXHBg1GsEDTODCjcptC/fj1KPao=', 'Content-Length': '228'} 2023-05-31T20:01:29.102 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000e66e4d620d7f42d3-006477a799-110e-a1'), ('Content-Type', 'application/xml'), ('Content-Length', '0'), ('Date', 'Wed, 31 May 2023 20:01:29 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:01:29.103 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-31T20:01:29.103 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-31T20:01:29.103 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-05-31T20:01:29.156 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2023-05-31T20:01:29.156 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2023-05-31T20:01:29.190 INFO:teuthology.orchestra.run.smithi099.stderr:No --period given, using current period=2e6123d2-d695-4350-9a01-61a455bc782b 2023-05-31T20:01:29.203 INFO:teuthology.orchestra.run.smithi099.stdout:[ 2023-05-31T20:01:29.203 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T20:01:29.203 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1685562631.516089_59.1", 2023-05-31T20:01:29.203 INFO:teuthology.orchestra.run.smithi099.stdout: "last_update": "2023-05-31T19:50:31.516089Z" 2023-05-31T20:01:29.203 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:01:29.204 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T20:01:29.204 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "", 2023-05-31T20:01:29.204 INFO:teuthology.orchestra.run.smithi099.stdout: "last_update": "0.000000" 2023-05-31T20:01:29.204 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:01:29.204 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T20:01:29.204 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1685562970.274572_93.1", 2023-05-31T20:01:29.204 INFO:teuthology.orchestra.run.smithi099.stdout: "last_update": "2023-05-31T19:56:10.274572Z" 2023-05-31T20:01:29.205 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:01:29.205 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T20:01:29.205 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1685563289.097487_74.1", 2023-05-31T20:01:29.205 INFO:teuthology.orchestra.run.smithi099.stdout: "last_update": "2023-05-31T20:01:29.097487Z" 2023-05-31T20:01:29.205 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T20:01:29.205 INFO:teuthology.orchestra.run.smithi099.stdout:] 2023-05-31T20:01:29.206 DEBUG:rgw_multi.tests:master meta markers={0: '1_1685562631.516089_59.1', 1: '', 2: '1_1685562970.274572_93.1', 3: '1_1685563289.097487_74.1'} 2023-05-31T20:01:29.207 INFO:rgw_multi.tests:starting meta checkpoint for zone=a2 2023-05-31T20:01:29.207 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-31T20:01:29.207 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-31T20:01:29.208 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-05-31T20:01:29.258 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2023-05-31T20:01:29.258 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2023-05-31T20:01:29.308 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:01:29.302+0000 7fc91425a780 20 RGW-SYNC:meta: read sync status 2023-05-31T20:01:29.309 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:01:29.303+0000 7fc91425a780 20 RGW-SYNC:meta: read sync status 2023-05-31T20:01:29.325 INFO:teuthology.orchestra.run.smithi099.stdout:{ 2023-05-31T20:01:29.326 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_status": { 2023-05-31T20:01:29.326 INFO:teuthology.orchestra.run.smithi099.stdout: "info": { 2023-05-31T20:01:29.326 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "sync", 2023-05-31T20:01:29.326 INFO:teuthology.orchestra.run.smithi099.stdout: "num_shards": 4, 2023-05-31T20:01:29.326 INFO:teuthology.orchestra.run.smithi099.stdout: "period": "2e6123d2-d695-4350-9a01-61a455bc782b", 2023-05-31T20:01:29.327 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2023-05-31T20:01:29.327 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:01:29.327 INFO:teuthology.orchestra.run.smithi099.stdout: "markers": [ 2023-05-31T20:01:29.327 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T20:01:29.327 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 0, 2023-05-31T20:01:29.327 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T20:01:29.327 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2023-05-31T20:01:29.328 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1685562631.516089_59.1", 2023-05-31T20:01:29.328 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T20:01:29.328 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 1, 2023-05-31T20:01:29.328 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T20:01:29.328 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2023-05-31T19:50:31.516089Z", 2023-05-31T20:01:29.328 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2023-05-31T20:01:29.329 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T20:01:29.329 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:01:29.329 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T20:01:29.329 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 1, 2023-05-31T20:01:29.329 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T20:01:29.329 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2023-05-31T20:01:29.329 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "", 2023-05-31T20:01:29.330 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T20:01:29.330 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2023-05-31T20:01:29.330 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T20:01:29.330 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "0.000000", 2023-05-31T20:01:29.330 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2023-05-31T20:01:29.330 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T20:01:29.330 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:01:29.331 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T20:01:29.331 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 2, 2023-05-31T20:01:29.331 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T20:01:29.331 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2023-05-31T20:01:29.331 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1685562970.274572_93.1", 2023-05-31T20:01:29.331 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T20:01:29.332 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2023-05-31T20:01:29.332 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T20:01:29.332 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2023-05-31T19:56:10.274572Z", 2023-05-31T20:01:29.332 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2023-05-31T20:01:29.332 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T20:01:29.332 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:01:29.332 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T20:01:29.333 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 3, 2023-05-31T20:01:29.333 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T20:01:29.333 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2023-05-31T20:01:29.333 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1685563289.097487_74.1", 2023-05-31T20:01:29.333 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T20:01:29.333 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2023-05-31T20:01:29.333 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T20:01:29.334 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2023-05-31T20:01:29.097487Z", 2023-05-31T20:01:29.334 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2023-05-31T20:01:29.334 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T20:01:29.334 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T20:01:29.334 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2023-05-31T20:01:29.334 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:01:29.335 INFO:teuthology.orchestra.run.smithi099.stdout: "full_sync": { 2023-05-31T20:01:29.335 INFO:teuthology.orchestra.run.smithi099.stdout: "total": 1, 2023-05-31T20:01:29.335 INFO:teuthology.orchestra.run.smithi099.stdout: "complete": 1 2023-05-31T20:01:29.335 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:01:29.335 INFO:teuthology.orchestra.run.smithi099.stdout: "current_time": "2023-05-31T20:01:29Z" 2023-05-31T20:01:29.335 INFO:teuthology.orchestra.run.smithi099.stdout:} 2023-05-31T20:01:29.336 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "2e6123d2-d695-4350-9a01-61a455bc782b", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1685562631.516089_59.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-05-31T19:50:31.516089Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1685562970.274572_93.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-05-31T19:56:10.274572Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1685563289.097487_74.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-05-31T20:01:29.097487Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-05-31T20:01:29Z" } 2023-05-31T20:01:29.337 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1685562631.516089_59.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-05-31T19:50:31.516089Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1685562970.274572_93.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-05-31T19:56:10.274572Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1685563289.097487_74.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-05-31T20:01:29.097487Z', 'realm_epoch': 2}}] 2023-05-31T20:01:29.337 DEBUG:rgw_multi.tests:log_status={0: '1_1685562631.516089_59.1', 1: '', 2: '1_1685562970.274572_93.1', 3: '1_1685563289.097487_74.1'} 2023-05-31T20:01:29.338 DEBUG:rgw_multi.tests:sync_status={0: '1_1685562631.516089_59.1', 1: '', 2: '1_1685562970.274572_93.1', 3: '1_1685563289.097487_74.1'} 2023-05-31T20:01:29.338 INFO:rgw_multi.tests:finish meta checkpoint for zone=a2 2023-05-31T20:01:29.338 DEBUG:boto:path=/pklxkd-14/ 2023-05-31T20:01:29.339 DEBUG:boto:auth_path=/pklxkd-14/ 2023-05-31T20:01:29.339 DEBUG:boto:Method: HEAD 2023-05-31T20:01:29.339 DEBUG:boto:Path: /pklxkd-14/ 2023-05-31T20:01:29.340 DEBUG:boto:Data: 2023-05-31T20:01:29.340 DEBUG:boto:Headers: {} 2023-05-31T20:01:29.340 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:01:29.341 DEBUG:boto:Port: 8000 2023-05-31T20:01:29.341 DEBUG:boto:Params: {} 2023-05-31T20:01:29.341 DEBUG:boto:Token: None 2023-05-31T20:01:29.342 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:01:29 GMT /pklxkd-14/ 2023-05-31T20:01:29.342 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:bLaHXrJdL8IovMd4qwF3z1FbjK4= 2023-05-31T20:01:29.343 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:01:29 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:bLaHXrJdL8IovMd4qwF3z1FbjK4=', 'Content-Length': '0'} 2023-05-31T20:01:29.344 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000021702fb512d80eea-006477a799-110e-a1'), ('Content-Length', '0'), ('Date', 'Wed, 31 May 2023 20:01:29 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:01:29.345 DEBUG:boto:path=/pklxkd-14/obj 2023-05-31T20:01:29.346 DEBUG:boto:auth_path=/pklxkd-14/obj 2023-05-31T20:01:29.346 DEBUG:boto:Method: PUT 2023-05-31T20:01:29.346 DEBUG:boto:Path: /pklxkd-14/obj 2023-05-31T20:01:29.347 DEBUG:boto:Data: 2023-05-31T20:01:29.347 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': '1B2M2Y8AsgTpgAmY7PhCfg==', 'Content-Length': '0', 'Expect': '100-Continue'} 2023-05-31T20:01:29.347 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:01:29.348 DEBUG:boto:Port: 8000 2023-05-31T20:01:29.348 DEBUG:boto:Params: {} 2023-05-31T20:01:29.348 DEBUG:boto:Token: None 2023-05-31T20:01:29.349 DEBUG:boto:StringToSign: PUT 1B2M2Y8AsgTpgAmY7PhCfg== application/octet-stream Wed, 31 May 2023 20:01:29 GMT /pklxkd-14/obj 2023-05-31T20:01:29.349 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:dpJaRmcPqE1j80j3ySLqtjwx0AI= 2023-05-31T20:01:29.349 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': '1B2M2Y8AsgTpgAmY7PhCfg==', 'Content-Length': '0', 'Expect': '100-Continue', 'Date': 'Wed, 31 May 2023 20:01:29 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:dpJaRmcPqE1j80j3ySLqtjwx0AI='} 2023-05-31T20:01:29.359 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('Accept-Ranges', 'bytes'), ('x-amz-version-id', '1c5VIADR1kuOHtAIi0CwrS4UaPxAfUI'), ('x-amz-request-id', 'tx000000275f93614a0efc8-006477a799-110e-a1'), ('Date', 'Wed, 31 May 2023 20:01:29 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:01:29.360 DEBUG:rgw_multi.tests:created new version id=1c5VIADR1kuOHtAIi0CwrS4UaPxAfUI 2023-05-31T20:01:29.360 DEBUG:rgw_multi.tests:bucket checkpoint: source=a1 target=a2 bucket=pklxkd-14 2023-05-31T20:01:29.361 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'pklxkd-14', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-05-31T20:01:29.361 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'pklxkd-14', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-05-31T20:01:29.361 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket pklxkd-14 --source-zone a1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2023-05-31T20:01:29.417 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2023-05-31T20:01:29.418 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2023-05-31T20:01:29.456 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:01:29.450+0000 7f4fc1845780 1 waiting for incremental sync to catch up: 2023-05-31T20:01:29.456 INFO:teuthology.orchestra.run.smithi099.stderr: local status: [, , , , , , , , 00000000001.5.6, , ] 2023-05-31T20:01:29.457 INFO:teuthology.orchestra.run.smithi099.stderr: remote markers: [, , , , , , , , 00000000003.8.13, , ] 2023-05-31T20:01:34.457 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:01:34.451+0000 7f4fc1845780 1 waiting for incremental sync to catch up: 2023-05-31T20:01:34.457 INFO:teuthology.orchestra.run.smithi099.stderr: local status: [, , , , , , , , 00000000001.5.6, , ] 2023-05-31T20:01:34.458 INFO:teuthology.orchestra.run.smithi099.stderr: remote markers: [, , , , , , , , 00000000003.8.13, , ] 2023-05-31T20:01:39.459 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:01:39.452+0000 7f4fc1845780 1 waiting for incremental sync to catch up: 2023-05-31T20:01:39.459 INFO:teuthology.orchestra.run.smithi099.stderr: local status: [, , , , , , , , 00000000001.5.6, , ] 2023-05-31T20:01:39.459 INFO:teuthology.orchestra.run.smithi099.stderr: remote markers: [, , , , , , , , 00000000003.8.13, , ] 2023-05-31T20:01:44.460 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:01:44.454+0000 7f4fc1845780 1 waiting for incremental sync to catch up: 2023-05-31T20:01:44.460 INFO:teuthology.orchestra.run.smithi099.stderr: local status: [, , , , , , , , 00000000001.5.6, , ] 2023-05-31T20:01:44.460 INFO:teuthology.orchestra.run.smithi099.stderr: remote markers: [, , , , , , , , 00000000003.8.13, , ] 2023-05-31T20:01:49.461 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:01:49.456+0000 7f4fc1845780 1 bucket sync caught up with source: 2023-05-31T20:01:49.461 INFO:teuthology.orchestra.run.smithi099.stderr: local status: [, , , , , , , , 00000000003.8.13, , ] 2023-05-31T20:01:49.461 INFO:teuthology.orchestra.run.smithi099.stderr: remote markers: [, , , , , , , , 00000000003.8.13, , ] 2023-05-31T20:01:49.462 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:01:49.456+0000 7f4fc1845780 0 bucket checkpoint complete 2023-05-31T20:01:49.479 DEBUG:rgw_multi.tests:bucket checkpoint: source=a2 target=a1 bucket=pklxkd-14 2023-05-31T20:01:49.480 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'pklxkd-14', '--source-zone', 'a2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-05-31T20:01:49.481 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'pklxkd-14', '--source-zone', 'a2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-05-31T20:01:49.482 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket pklxkd-14 --source-zone a2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2023-05-31T20:01:49.546 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2023-05-31T20:01:49.546 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2023-05-31T20:01:49.595 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:01:49.590+0000 7f4005903780 1 waiting for incremental sync to catch up: 2023-05-31T20:01:49.595 INFO:teuthology.orchestra.run.smithi099.stderr: local status: [, , , , , , , , 00000000001.7.6, , ] 2023-05-31T20:01:49.596 INFO:teuthology.orchestra.run.smithi099.stderr: remote markers: [, , , , , , , , 00000000003.11.13, , ] 2023-05-31T20:01:54.596 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:01:54.591+0000 7f4005903780 1 waiting for incremental sync to catch up: 2023-05-31T20:01:54.597 INFO:teuthology.orchestra.run.smithi099.stderr: local status: [, , , , , , , , 00000000001.7.6, , ] 2023-05-31T20:01:54.597 INFO:teuthology.orchestra.run.smithi099.stderr: remote markers: [, , , , , , , , 00000000003.11.13, , ] 2023-05-31T20:01:59.598 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:01:59.593+0000 7f4005903780 1 waiting for incremental sync to catch up: 2023-05-31T20:01:59.598 INFO:teuthology.orchestra.run.smithi099.stderr: local status: [, , , , , , , , 00000000001.7.6, , ] 2023-05-31T20:01:59.598 INFO:teuthology.orchestra.run.smithi099.stderr: remote markers: [, , , , , , , , 00000000003.11.13, , ] 2023-05-31T20:02:04.599 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:02:04.594+0000 7f4005903780 1 waiting for incremental sync to catch up: 2023-05-31T20:02:04.599 INFO:teuthology.orchestra.run.smithi099.stderr: local status: [, , , , , , , , 00000000001.7.6, , ] 2023-05-31T20:02:04.600 INFO:teuthology.orchestra.run.smithi099.stderr: remote markers: [, , , , , , , , 00000000003.11.13, , ] 2023-05-31T20:02:09.600 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:02:09.594+0000 7f4005903780 1 bucket sync caught up with source: 2023-05-31T20:02:09.601 INFO:teuthology.orchestra.run.smithi099.stderr: local status: [, , , , , , , , 00000000003.11.13, , ] 2023-05-31T20:02:09.601 INFO:teuthology.orchestra.run.smithi099.stderr: remote markers: [, , , , , , , , 00000000003.11.13, , ] 2023-05-31T20:02:09.601 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:02:09.594+0000 7f4005903780 0 bucket checkpoint complete 2023-05-31T20:02:09.620 INFO:tasks.rgw_multi.zone_rados:comparing bucket=pklxkd-14 zones={a2, a1} 2023-05-31T20:02:09.621 DEBUG:boto:path=/pklxkd-14/ 2023-05-31T20:02:09.621 DEBUG:boto:auth_path=/pklxkd-14/ 2023-05-31T20:02:09.622 DEBUG:boto:Method: HEAD 2023-05-31T20:02:09.622 DEBUG:boto:Path: /pklxkd-14/ 2023-05-31T20:02:09.623 DEBUG:boto:Data: 2023-05-31T20:02:09.623 DEBUG:boto:Headers: {} 2023-05-31T20:02:09.623 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:02:09.624 DEBUG:boto:Port: 8001 2023-05-31T20:02:09.624 DEBUG:boto:Params: {} 2023-05-31T20:02:09.625 DEBUG:boto:Token: None 2023-05-31T20:02:09.625 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:02:09 GMT /pklxkd-14/ 2023-05-31T20:02:09.626 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:Cb5JvQCPDBUtgVM63kmdFpMTz90= 2023-05-31T20:02:09.626 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:02:09 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:Cb5JvQCPDBUtgVM63kmdFpMTz90=', 'Content-Length': '0'} 2023-05-31T20:02:09.629 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '2'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000971e4ab32efdd8f0-006477a7c1-1112-a2'), ('Content-Length', '0'), ('Date', 'Wed, 31 May 2023 20:02:09 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:02:09.629 DEBUG:boto:path=/pklxkd-14/ 2023-05-31T20:02:09.630 DEBUG:boto:auth_path=/pklxkd-14/ 2023-05-31T20:02:09.630 DEBUG:boto:Method: HEAD 2023-05-31T20:02:09.630 DEBUG:boto:Path: /pklxkd-14/ 2023-05-31T20:02:09.631 DEBUG:boto:Data: 2023-05-31T20:02:09.631 DEBUG:boto:Headers: {} 2023-05-31T20:02:09.631 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:02:09.632 DEBUG:boto:Port: 8000 2023-05-31T20:02:09.632 DEBUG:boto:Params: {} 2023-05-31T20:02:09.633 DEBUG:boto:Token: None 2023-05-31T20:02:09.633 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:02:09 GMT /pklxkd-14/ 2023-05-31T20:02:09.634 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:Cb5JvQCPDBUtgVM63kmdFpMTz90= 2023-05-31T20:02:09.634 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:02:09 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:Cb5JvQCPDBUtgVM63kmdFpMTz90=', 'Content-Length': '0'} 2023-05-31T20:02:09.636 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '2'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000db5482c02559be9c-006477a7c1-110e-a1'), ('Content-Length', '0'), ('Date', 'Wed, 31 May 2023 20:02:09 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:02:09.637 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2023-05-31T20:02:09.637 DEBUG:boto:path=/pklxkd-14/ 2023-05-31T20:02:09.638 DEBUG:boto:auth_path=/pklxkd-14/ 2023-05-31T20:02:09.638 DEBUG:boto:path=/pklxkd-14/?versions&max-keys=999 2023-05-31T20:02:09.638 DEBUG:boto:auth_path=/pklxkd-14/?versions&max-keys=999 2023-05-31T20:02:09.639 DEBUG:boto:Method: GET 2023-05-31T20:02:09.639 DEBUG:boto:Path: /pklxkd-14/?versions&max-keys=999 2023-05-31T20:02:09.640 DEBUG:boto:Data: 2023-05-31T20:02:09.640 DEBUG:boto:Headers: {} 2023-05-31T20:02:09.640 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:02:09.641 DEBUG:boto:Port: 8001 2023-05-31T20:02:09.641 DEBUG:boto:Params: {} 2023-05-31T20:02:09.642 DEBUG:boto:Token: None 2023-05-31T20:02:09.642 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:02:09 GMT /pklxkd-14/?versions 2023-05-31T20:02:09.643 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:TL/hJUl7tPtStN40R0I4QupYSKg= 2023-05-31T20:02:09.643 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:02:09 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:TL/hJUl7tPtStN40R0I4QupYSKg=', 'Content-Length': '0'} 2023-05-31T20:02:09.646 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000005b9e602586fc5fe9-006477a7c1-1112-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:02:09 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:02:09.646 DEBUG:boto:b'pklxkd-14999falseobj1c5VIADR1kuOHtAIi0CwrS4UaPxAfUItrue2023-05-31T20:01:29.347Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormalobjnullfalse2023-05-31T20:00:53.528Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:02:09.647 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2023-05-31T20:02:09.647 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2023-05-31T20:02:09.648 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2023-05-31T20:02:09.648 DEBUG:boto:path=/pklxkd-14/ 2023-05-31T20:02:09.649 DEBUG:boto:auth_path=/pklxkd-14/ 2023-05-31T20:02:09.650 DEBUG:boto:path=/pklxkd-14/?versions&max-keys=999 2023-05-31T20:02:09.651 DEBUG:boto:auth_path=/pklxkd-14/?versions&max-keys=999 2023-05-31T20:02:09.652 DEBUG:boto:Method: GET 2023-05-31T20:02:09.653 DEBUG:boto:Path: /pklxkd-14/?versions&max-keys=999 2023-05-31T20:02:09.654 DEBUG:boto:Data: 2023-05-31T20:02:09.655 DEBUG:boto:Headers: {} 2023-05-31T20:02:09.655 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:02:09.656 DEBUG:boto:Port: 8000 2023-05-31T20:02:09.657 DEBUG:boto:Params: {} 2023-05-31T20:02:09.658 DEBUG:boto:Token: None 2023-05-31T20:02:09.658 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:02:09 GMT /pklxkd-14/?versions 2023-05-31T20:02:09.659 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:TL/hJUl7tPtStN40R0I4QupYSKg= 2023-05-31T20:02:09.660 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:02:09 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:TL/hJUl7tPtStN40R0I4QupYSKg=', 'Content-Length': '0'} 2023-05-31T20:02:09.662 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b60a2e94d4ca7dfc-006477a7c1-110e-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:02:09 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:02:09.704 DEBUG:boto:b'pklxkd-14999falseobj1c5VIADR1kuOHtAIi0CwrS4UaPxAfUItrue2023-05-31T20:01:29.347Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormalobjnullfalse2023-05-31T20:00:53.528Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:02:09.705 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2023-05-31T20:02:09.706 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2023-05-31T20:02:09.706 DEBUG:boto:path=/pklxkd-14/ 2023-05-31T20:02:09.707 DEBUG:boto:auth_path=/pklxkd-14/ 2023-05-31T20:02:09.707 DEBUG:boto:path=/pklxkd-14/?versions&max-keys=999 2023-05-31T20:02:09.707 DEBUG:boto:auth_path=/pklxkd-14/?versions&max-keys=999 2023-05-31T20:02:09.708 DEBUG:boto:Method: GET 2023-05-31T20:02:09.708 DEBUG:boto:Path: /pklxkd-14/?versions&max-keys=999 2023-05-31T20:02:09.708 DEBUG:boto:Data: 2023-05-31T20:02:09.709 DEBUG:boto:Headers: {} 2023-05-31T20:02:09.709 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:02:09.710 DEBUG:boto:Port: 8001 2023-05-31T20:02:09.710 DEBUG:boto:Params: {} 2023-05-31T20:02:09.710 DEBUG:boto:Token: None 2023-05-31T20:02:09.711 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:02:09 GMT /pklxkd-14/?versions 2023-05-31T20:02:09.711 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:TL/hJUl7tPtStN40R0I4QupYSKg= 2023-05-31T20:02:09.712 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:02:09 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:TL/hJUl7tPtStN40R0I4QupYSKg=', 'Content-Length': '0'} 2023-05-31T20:02:09.714 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000875bb5ee8e9df8d3-006477a7c1-1112-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:02:09 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:02:09.715 DEBUG:boto:b'pklxkd-14999falseobj1c5VIADR1kuOHtAIi0CwrS4UaPxAfUItrue2023-05-31T20:01:29.347Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormalobjnullfalse2023-05-31T20:00:53.528Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:02:09.715 DEBUG:boto:path=/pklxkd-14/ 2023-05-31T20:02:09.716 DEBUG:boto:auth_path=/pklxkd-14/ 2023-05-31T20:02:09.716 DEBUG:boto:path=/pklxkd-14/?versions&max-keys=999 2023-05-31T20:02:09.717 DEBUG:boto:auth_path=/pklxkd-14/?versions&max-keys=999 2023-05-31T20:02:09.717 DEBUG:boto:Method: GET 2023-05-31T20:02:09.717 DEBUG:boto:Path: /pklxkd-14/?versions&max-keys=999 2023-05-31T20:02:09.718 DEBUG:boto:Data: 2023-05-31T20:02:09.718 DEBUG:boto:Headers: {} 2023-05-31T20:02:09.718 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:02:09.719 DEBUG:boto:Port: 8000 2023-05-31T20:02:09.719 DEBUG:boto:Params: {} 2023-05-31T20:02:09.720 DEBUG:boto:Token: None 2023-05-31T20:02:09.720 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:02:09 GMT /pklxkd-14/?versions 2023-05-31T20:02:09.721 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:TL/hJUl7tPtStN40R0I4QupYSKg= 2023-05-31T20:02:09.721 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:02:09 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:TL/hJUl7tPtStN40R0I4QupYSKg=', 'Content-Length': '0'} 2023-05-31T20:02:09.723 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ae466c1569acfeaa-006477a7c1-110e-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:02:09 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:02:09.724 DEBUG:boto:b'pklxkd-14999falseobj1c5VIADR1kuOHtAIi0CwrS4UaPxAfUItrue2023-05-31T20:01:29.347Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormalobjnullfalse2023-05-31T20:00:53.528Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:02:09.725 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2023-05-31T20:02:09.725 DEBUG:boto:path=/pklxkd-14/obj 2023-05-31T20:02:09.726 DEBUG:boto:auth_path=/pklxkd-14/obj 2023-05-31T20:02:09.726 DEBUG:boto:path=/pklxkd-14/obj?versionId=1c5VIADR1kuOHtAIi0CwrS4UaPxAfUI 2023-05-31T20:02:09.727 DEBUG:boto:auth_path=/pklxkd-14/obj?versionId=1c5VIADR1kuOHtAIi0CwrS4UaPxAfUI 2023-05-31T20:02:09.727 DEBUG:boto:Method: GET 2023-05-31T20:02:09.727 DEBUG:boto:Path: /pklxkd-14/obj?versionId=1c5VIADR1kuOHtAIi0CwrS4UaPxAfUI 2023-05-31T20:02:09.728 DEBUG:boto:Data: 2023-05-31T20:02:09.728 DEBUG:boto:Headers: {} 2023-05-31T20:02:09.729 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:02:09.729 DEBUG:boto:Port: 8001 2023-05-31T20:02:09.730 DEBUG:boto:Params: {} 2023-05-31T20:02:09.730 DEBUG:boto:Token: None 2023-05-31T20:02:09.731 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:02:09 GMT /pklxkd-14/obj?versionId=1c5VIADR1kuOHtAIi0CwrS4UaPxAfUI 2023-05-31T20:02:09.732 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:QvOC71pNUl1kw9X4fVYFLIjYt7g= 2023-05-31T20:02:09.733 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:02:09 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:QvOC71pNUl1kw9X4fVYFLIjYt7g=', 'Content-Length': '0'} 2023-05-31T20:02:09.736 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:01:29 GMT'), ('x-amz-version-id', '1c5VIADR1kuOHtAIi0CwrS4UaPxAfUI'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000acefca6ab1e7c0dc-006477a7c1-1112-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:02:09 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:02:09.737 DEBUG:boto:path=/pklxkd-14/obj 2023-05-31T20:02:09.738 DEBUG:boto:auth_path=/pklxkd-14/obj 2023-05-31T20:02:09.738 DEBUG:boto:path=/pklxkd-14/obj?versionId=1c5VIADR1kuOHtAIi0CwrS4UaPxAfUI 2023-05-31T20:02:09.738 DEBUG:boto:auth_path=/pklxkd-14/obj?versionId=1c5VIADR1kuOHtAIi0CwrS4UaPxAfUI 2023-05-31T20:02:09.739 DEBUG:boto:Method: GET 2023-05-31T20:02:09.739 DEBUG:boto:Path: /pklxkd-14/obj?versionId=1c5VIADR1kuOHtAIi0CwrS4UaPxAfUI 2023-05-31T20:02:09.739 DEBUG:boto:Data: 2023-05-31T20:02:09.740 DEBUG:boto:Headers: {} 2023-05-31T20:02:09.740 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:02:09.741 DEBUG:boto:Port: 8000 2023-05-31T20:02:09.741 DEBUG:boto:Params: {} 2023-05-31T20:02:09.742 DEBUG:boto:Token: None 2023-05-31T20:02:09.742 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:02:09 GMT /pklxkd-14/obj?versionId=1c5VIADR1kuOHtAIi0CwrS4UaPxAfUI 2023-05-31T20:02:09.743 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:QvOC71pNUl1kw9X4fVYFLIjYt7g= 2023-05-31T20:02:09.743 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:02:09 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:QvOC71pNUl1kw9X4fVYFLIjYt7g=', 'Content-Length': '0'} 2023-05-31T20:02:09.745 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:01:29 GMT'), ('x-amz-version-id', '1c5VIADR1kuOHtAIi0CwrS4UaPxAfUI'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000c2b3422202abdc07-006477a7c1-110e-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:02:09 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:02:09.746 DEBUG:boto:path=/pklxkd-14/obj 2023-05-31T20:02:09.747 DEBUG:boto:auth_path=/pklxkd-14/obj 2023-05-31T20:02:09.747 DEBUG:boto:path=/pklxkd-14/obj?versionId=1c5VIADR1kuOHtAIi0CwrS4UaPxAfUI 2023-05-31T20:02:09.748 DEBUG:boto:auth_path=/pklxkd-14/obj?versionId=1c5VIADR1kuOHtAIi0CwrS4UaPxAfUI 2023-05-31T20:02:09.748 DEBUG:boto:Method: HEAD 2023-05-31T20:02:09.748 DEBUG:boto:Path: /pklxkd-14/obj?versionId=1c5VIADR1kuOHtAIi0CwrS4UaPxAfUI 2023-05-31T20:02:09.749 DEBUG:boto:Data: 2023-05-31T20:02:09.749 DEBUG:boto:Headers: {} 2023-05-31T20:02:09.750 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:02:09.750 DEBUG:boto:Port: 8001 2023-05-31T20:02:09.750 DEBUG:boto:Params: {} 2023-05-31T20:02:09.751 DEBUG:boto:Token: None 2023-05-31T20:02:09.751 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:02:09 GMT /pklxkd-14/obj?versionId=1c5VIADR1kuOHtAIi0CwrS4UaPxAfUI 2023-05-31T20:02:09.752 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:9han8mm6Z10H/b3/hnV05PyoV5c= 2023-05-31T20:02:09.752 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:02:09 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:9han8mm6Z10H/b3/hnV05PyoV5c=', 'Content-Length': '0'} 2023-05-31T20:02:09.754 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:01:29 GMT'), ('x-amz-version-id', '1c5VIADR1kuOHtAIi0CwrS4UaPxAfUI'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000f3282fcada167d3b-006477a7c1-1112-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:02:09 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:02:09.755 DEBUG:boto:path=/pklxkd-14/obj 2023-05-31T20:02:09.756 DEBUG:boto:auth_path=/pklxkd-14/obj 2023-05-31T20:02:09.756 DEBUG:boto:path=/pklxkd-14/obj?versionId=1c5VIADR1kuOHtAIi0CwrS4UaPxAfUI 2023-05-31T20:02:09.757 DEBUG:boto:auth_path=/pklxkd-14/obj?versionId=1c5VIADR1kuOHtAIi0CwrS4UaPxAfUI 2023-05-31T20:02:09.757 DEBUG:boto:Method: HEAD 2023-05-31T20:02:09.758 DEBUG:boto:Path: /pklxkd-14/obj?versionId=1c5VIADR1kuOHtAIi0CwrS4UaPxAfUI 2023-05-31T20:02:09.759 DEBUG:boto:Data: 2023-05-31T20:02:09.759 DEBUG:boto:Headers: {} 2023-05-31T20:02:09.760 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:02:09.760 DEBUG:boto:Port: 8000 2023-05-31T20:02:09.761 DEBUG:boto:Params: {} 2023-05-31T20:02:09.762 DEBUG:boto:Token: None 2023-05-31T20:02:09.763 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:02:09 GMT /pklxkd-14/obj?versionId=1c5VIADR1kuOHtAIi0CwrS4UaPxAfUI 2023-05-31T20:02:09.764 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:9han8mm6Z10H/b3/hnV05PyoV5c= 2023-05-31T20:02:09.764 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:02:09 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:9han8mm6Z10H/b3/hnV05PyoV5c=', 'Content-Length': '0'} 2023-05-31T20:02:09.767 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:01:29 GMT'), ('x-amz-version-id', '1c5VIADR1kuOHtAIi0CwrS4UaPxAfUI'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx0000051892b5042d21100-006477a7c1-110e-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:02:09 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:02:09.768 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2023-05-31T20:02:09.769 DEBUG:boto:path=/pklxkd-14/obj 2023-05-31T20:02:09.769 DEBUG:boto:auth_path=/pklxkd-14/obj 2023-05-31T20:02:09.770 DEBUG:boto:path=/pklxkd-14/obj?versionId=1c5VIADR1kuOHtAIi0CwrS4UaPxAfUI 2023-05-31T20:02:09.770 DEBUG:boto:auth_path=/pklxkd-14/obj?versionId=1c5VIADR1kuOHtAIi0CwrS4UaPxAfUI 2023-05-31T20:02:09.771 DEBUG:boto:Method: GET 2023-05-31T20:02:09.771 DEBUG:boto:Path: /pklxkd-14/obj?versionId=1c5VIADR1kuOHtAIi0CwrS4UaPxAfUI 2023-05-31T20:02:09.772 DEBUG:boto:Data: 2023-05-31T20:02:09.772 DEBUG:boto:Headers: {} 2023-05-31T20:02:09.772 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:02:09.773 DEBUG:boto:Port: 8001 2023-05-31T20:02:09.773 DEBUG:boto:Params: {} 2023-05-31T20:02:09.773 DEBUG:boto:Token: None 2023-05-31T20:02:09.774 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:02:09 GMT /pklxkd-14/obj?versionId=1c5VIADR1kuOHtAIi0CwrS4UaPxAfUI 2023-05-31T20:02:09.774 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:QvOC71pNUl1kw9X4fVYFLIjYt7g= 2023-05-31T20:02:09.775 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:02:09 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:QvOC71pNUl1kw9X4fVYFLIjYt7g=', 'Content-Length': '0'} 2023-05-31T20:02:09.776 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:01:29 GMT'), ('x-amz-version-id', '1c5VIADR1kuOHtAIi0CwrS4UaPxAfUI'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000db76f2a5ce908796-006477a7c1-1112-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:02:09 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:02:09.777 DEBUG:boto:path=/pklxkd-14/obj 2023-05-31T20:02:09.777 DEBUG:boto:auth_path=/pklxkd-14/obj 2023-05-31T20:02:09.778 DEBUG:boto:path=/pklxkd-14/obj?versionId=1c5VIADR1kuOHtAIi0CwrS4UaPxAfUI 2023-05-31T20:02:09.778 DEBUG:boto:auth_path=/pklxkd-14/obj?versionId=1c5VIADR1kuOHtAIi0CwrS4UaPxAfUI 2023-05-31T20:02:09.778 DEBUG:boto:Method: GET 2023-05-31T20:02:09.779 DEBUG:boto:Path: /pklxkd-14/obj?versionId=1c5VIADR1kuOHtAIi0CwrS4UaPxAfUI 2023-05-31T20:02:09.779 DEBUG:boto:Data: 2023-05-31T20:02:09.780 DEBUG:boto:Headers: {} 2023-05-31T20:02:09.780 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:02:09.780 DEBUG:boto:Port: 8000 2023-05-31T20:02:09.781 DEBUG:boto:Params: {} 2023-05-31T20:02:09.781 DEBUG:boto:Token: None 2023-05-31T20:02:09.782 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:02:09 GMT /pklxkd-14/obj?versionId=1c5VIADR1kuOHtAIi0CwrS4UaPxAfUI 2023-05-31T20:02:09.782 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:QvOC71pNUl1kw9X4fVYFLIjYt7g= 2023-05-31T20:02:09.782 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:02:09 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:QvOC71pNUl1kw9X4fVYFLIjYt7g=', 'Content-Length': '0'} 2023-05-31T20:02:09.784 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:01:29 GMT'), ('x-amz-version-id', '1c5VIADR1kuOHtAIi0CwrS4UaPxAfUI'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000008bf60ac45ca40568-006477a7c1-110e-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:02:09 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:02:09.785 DEBUG:boto:path=/pklxkd-14/ 2023-05-31T20:02:09.785 DEBUG:boto:auth_path=/pklxkd-14/ 2023-05-31T20:02:09.786 DEBUG:boto:path=/pklxkd-14/?prefix=obj 2023-05-31T20:02:09.786 DEBUG:boto:auth_path=/pklxkd-14/?prefix=obj 2023-05-31T20:02:09.786 DEBUG:boto:Method: GET 2023-05-31T20:02:09.787 DEBUG:boto:Path: /pklxkd-14/?prefix=obj 2023-05-31T20:02:09.787 DEBUG:boto:Data: 2023-05-31T20:02:09.787 DEBUG:boto:Headers: {} 2023-05-31T20:02:09.788 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:02:09.788 DEBUG:boto:Port: 8001 2023-05-31T20:02:09.788 DEBUG:boto:Params: {} 2023-05-31T20:02:09.789 DEBUG:boto:Token: None 2023-05-31T20:02:09.789 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:02:09 GMT /pklxkd-14/ 2023-05-31T20:02:09.790 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:z5snMU9Z9WC1VLCc92tV09vADOQ= 2023-05-31T20:02:09.790 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:02:09 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:z5snMU9Z9WC1VLCc92tV09vADOQ=', 'Content-Length': '0'} 2023-05-31T20:02:09.792 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c3a0eb1764d412cc-006477a7c1-1112-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:02:09 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:02:09.836 DEBUG:boto:b'pklxkd-14obj1000falseobj2023-05-31T20:01:29.347Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:02:09.837 DEBUG:boto:path=/pklxkd-14/ 2023-05-31T20:02:09.837 DEBUG:boto:auth_path=/pklxkd-14/ 2023-05-31T20:02:09.838 DEBUG:boto:path=/pklxkd-14/?prefix=obj 2023-05-31T20:02:09.838 DEBUG:boto:auth_path=/pklxkd-14/?prefix=obj 2023-05-31T20:02:09.839 DEBUG:boto:Method: GET 2023-05-31T20:02:09.839 DEBUG:boto:Path: /pklxkd-14/?prefix=obj 2023-05-31T20:02:09.839 DEBUG:boto:Data: 2023-05-31T20:02:09.840 DEBUG:boto:Headers: {} 2023-05-31T20:02:09.840 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:02:09.840 DEBUG:boto:Port: 8000 2023-05-31T20:02:09.841 DEBUG:boto:Params: {} 2023-05-31T20:02:09.841 DEBUG:boto:Token: None 2023-05-31T20:02:09.842 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:02:09 GMT /pklxkd-14/ 2023-05-31T20:02:09.842 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:z5snMU9Z9WC1VLCc92tV09vADOQ= 2023-05-31T20:02:09.842 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:02:09 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:z5snMU9Z9WC1VLCc92tV09vADOQ=', 'Content-Length': '0'} 2023-05-31T20:02:09.845 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000da5bf955af308554-006477a7c1-110e-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:02:09 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:02:09.846 DEBUG:boto:b'pklxkd-14obj1000falseobj2023-05-31T20:01:29.347Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:02:09.846 DEBUG:boto:path=/pklxkd-14/obj 2023-05-31T20:02:09.847 DEBUG:boto:auth_path=/pklxkd-14/obj 2023-05-31T20:02:09.847 DEBUG:boto:Method: HEAD 2023-05-31T20:02:09.847 DEBUG:boto:Path: /pklxkd-14/obj 2023-05-31T20:02:09.848 DEBUG:boto:Data: 2023-05-31T20:02:09.848 DEBUG:boto:Headers: {} 2023-05-31T20:02:09.849 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:02:09.849 DEBUG:boto:Port: 8001 2023-05-31T20:02:09.849 DEBUG:boto:Params: {} 2023-05-31T20:02:09.850 DEBUG:boto:Token: None 2023-05-31T20:02:09.850 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:02:09 GMT /pklxkd-14/obj 2023-05-31T20:02:09.851 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:HX1mFp2vgVVzuxBqtmXzzHZBUzM= 2023-05-31T20:02:09.851 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:02:09 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:HX1mFp2vgVVzuxBqtmXzzHZBUzM=', 'Content-Length': '0'} 2023-05-31T20:02:09.854 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:01:29 GMT'), ('x-amz-version-id', '1c5VIADR1kuOHtAIi0CwrS4UaPxAfUI'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx0000023f1476bdb4b4fbc-006477a7c1-1112-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:02:09 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:02:09.854 DEBUG:boto:path=/pklxkd-14/obj 2023-05-31T20:02:09.855 DEBUG:boto:auth_path=/pklxkd-14/obj 2023-05-31T20:02:09.856 DEBUG:boto:Method: HEAD 2023-05-31T20:02:09.856 DEBUG:boto:Path: /pklxkd-14/obj 2023-05-31T20:02:09.857 DEBUG:boto:Data: 2023-05-31T20:02:09.857 DEBUG:boto:Headers: {} 2023-05-31T20:02:09.857 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:02:09.858 DEBUG:boto:Port: 8000 2023-05-31T20:02:09.858 DEBUG:boto:Params: {} 2023-05-31T20:02:09.858 DEBUG:boto:Token: None 2023-05-31T20:02:09.859 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:02:09 GMT /pklxkd-14/obj 2023-05-31T20:02:09.859 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:HX1mFp2vgVVzuxBqtmXzzHZBUzM= 2023-05-31T20:02:09.860 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:02:09 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:HX1mFp2vgVVzuxBqtmXzzHZBUzM=', 'Content-Length': '0'} 2023-05-31T20:02:09.862 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:01:29 GMT'), ('x-amz-version-id', '1c5VIADR1kuOHtAIi0CwrS4UaPxAfUI'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000007af46edf1f663178-006477a7c1-110e-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:02:09 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:02:09.863 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2023-05-31T20:02:09.863 DEBUG:boto:path=/pklxkd-14/obj 2023-05-31T20:02:09.864 DEBUG:boto:auth_path=/pklxkd-14/obj 2023-05-31T20:02:09.864 DEBUG:boto:path=/pklxkd-14/obj?versionId=1c5VIADR1kuOHtAIi0CwrS4UaPxAfUI 2023-05-31T20:02:09.864 DEBUG:boto:auth_path=/pklxkd-14/obj?versionId=1c5VIADR1kuOHtAIi0CwrS4UaPxAfUI 2023-05-31T20:02:09.865 DEBUG:boto:Method: GET 2023-05-31T20:02:09.865 DEBUG:boto:Path: /pklxkd-14/obj?versionId=1c5VIADR1kuOHtAIi0CwrS4UaPxAfUI 2023-05-31T20:02:09.866 DEBUG:boto:Data: 2023-05-31T20:02:09.866 DEBUG:boto:Headers: {} 2023-05-31T20:02:09.867 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:02:09.867 DEBUG:boto:Port: 8001 2023-05-31T20:02:09.867 DEBUG:boto:Params: {} 2023-05-31T20:02:09.868 DEBUG:boto:Token: None 2023-05-31T20:02:09.868 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:02:09 GMT /pklxkd-14/obj?versionId=1c5VIADR1kuOHtAIi0CwrS4UaPxAfUI 2023-05-31T20:02:09.869 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:QvOC71pNUl1kw9X4fVYFLIjYt7g= 2023-05-31T20:02:09.870 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:02:09 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:QvOC71pNUl1kw9X4fVYFLIjYt7g=', 'Content-Length': '0'} 2023-05-31T20:02:09.872 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:01:29 GMT'), ('x-amz-version-id', '1c5VIADR1kuOHtAIi0CwrS4UaPxAfUI'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx0000072066dce400e2665-006477a7c1-1112-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:02:09 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:02:09.873 DEBUG:boto:path=/pklxkd-14/obj 2023-05-31T20:02:09.873 DEBUG:boto:auth_path=/pklxkd-14/obj 2023-05-31T20:02:09.873 DEBUG:boto:path=/pklxkd-14/obj?versionId=1c5VIADR1kuOHtAIi0CwrS4UaPxAfUI 2023-05-31T20:02:09.874 DEBUG:boto:auth_path=/pklxkd-14/obj?versionId=1c5VIADR1kuOHtAIi0CwrS4UaPxAfUI 2023-05-31T20:02:09.874 DEBUG:boto:Method: GET 2023-05-31T20:02:09.875 DEBUG:boto:Path: /pklxkd-14/obj?versionId=1c5VIADR1kuOHtAIi0CwrS4UaPxAfUI 2023-05-31T20:02:09.875 DEBUG:boto:Data: 2023-05-31T20:02:09.876 DEBUG:boto:Headers: {} 2023-05-31T20:02:09.876 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:02:09.876 DEBUG:boto:Port: 8000 2023-05-31T20:02:09.877 DEBUG:boto:Params: {} 2023-05-31T20:02:09.877 DEBUG:boto:Token: None 2023-05-31T20:02:09.878 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:02:09 GMT /pklxkd-14/obj?versionId=1c5VIADR1kuOHtAIi0CwrS4UaPxAfUI 2023-05-31T20:02:09.878 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:QvOC71pNUl1kw9X4fVYFLIjYt7g= 2023-05-31T20:02:09.879 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:02:09 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:QvOC71pNUl1kw9X4fVYFLIjYt7g=', 'Content-Length': '0'} 2023-05-31T20:02:09.880 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:01:29 GMT'), ('x-amz-version-id', '1c5VIADR1kuOHtAIi0CwrS4UaPxAfUI'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx0000036e8367b3301add1-006477a7c1-110e-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:02:09 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:02:09.881 DEBUG:boto:path=/pklxkd-14/ 2023-05-31T20:02:09.881 DEBUG:boto:auth_path=/pklxkd-14/ 2023-05-31T20:02:09.882 DEBUG:boto:path=/pklxkd-14/?prefix=obj 2023-05-31T20:02:09.882 DEBUG:boto:auth_path=/pklxkd-14/?prefix=obj 2023-05-31T20:02:09.883 DEBUG:boto:Method: GET 2023-05-31T20:02:09.883 DEBUG:boto:Path: /pklxkd-14/?prefix=obj 2023-05-31T20:02:09.884 DEBUG:boto:Data: 2023-05-31T20:02:09.884 DEBUG:boto:Headers: {} 2023-05-31T20:02:09.885 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:02:09.885 DEBUG:boto:Port: 8001 2023-05-31T20:02:09.886 DEBUG:boto:Params: {} 2023-05-31T20:02:09.886 DEBUG:boto:Token: None 2023-05-31T20:02:09.886 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:02:09 GMT /pklxkd-14/ 2023-05-31T20:02:09.887 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:z5snMU9Z9WC1VLCc92tV09vADOQ= 2023-05-31T20:02:09.887 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:02:09 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:z5snMU9Z9WC1VLCc92tV09vADOQ=', 'Content-Length': '0'} 2023-05-31T20:02:09.890 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000fa2612644a6e9146-006477a7c1-1112-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:02:09 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:02:09.890 DEBUG:boto:b'pklxkd-14obj1000falseobj2023-05-31T20:01:29.347Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:02:09.891 DEBUG:boto:path=/pklxkd-14/ 2023-05-31T20:02:09.892 DEBUG:boto:auth_path=/pklxkd-14/ 2023-05-31T20:02:09.892 DEBUG:boto:path=/pklxkd-14/?prefix=obj 2023-05-31T20:02:09.893 DEBUG:boto:auth_path=/pklxkd-14/?prefix=obj 2023-05-31T20:02:09.893 DEBUG:boto:Method: GET 2023-05-31T20:02:09.894 DEBUG:boto:Path: /pklxkd-14/?prefix=obj 2023-05-31T20:02:09.895 DEBUG:boto:Data: 2023-05-31T20:02:09.895 DEBUG:boto:Headers: {} 2023-05-31T20:02:09.896 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:02:09.896 DEBUG:boto:Port: 8000 2023-05-31T20:02:09.897 DEBUG:boto:Params: {} 2023-05-31T20:02:09.897 DEBUG:boto:Token: None 2023-05-31T20:02:09.898 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:02:09 GMT /pklxkd-14/ 2023-05-31T20:02:09.898 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:z5snMU9Z9WC1VLCc92tV09vADOQ= 2023-05-31T20:02:09.899 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:02:09 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:z5snMU9Z9WC1VLCc92tV09vADOQ=', 'Content-Length': '0'} 2023-05-31T20:02:09.901 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000fb7f94a5fbf5be17-006477a7c1-110e-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:02:09 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:02:09.944 DEBUG:boto:b'pklxkd-14obj1000falseobj2023-05-31T20:01:29.347Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:02:09.945 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2023-05-31T20:02:09.946 DEBUG:boto:path=/pklxkd-14/obj 2023-05-31T20:02:09.946 DEBUG:boto:auth_path=/pklxkd-14/obj 2023-05-31T20:02:09.946 DEBUG:boto:path=/pklxkd-14/obj?versionId=null 2023-05-31T20:02:09.947 DEBUG:boto:auth_path=/pklxkd-14/obj?versionId=null 2023-05-31T20:02:09.947 DEBUG:boto:Method: GET 2023-05-31T20:02:09.947 DEBUG:boto:Path: /pklxkd-14/obj?versionId=null 2023-05-31T20:02:09.948 DEBUG:boto:Data: 2023-05-31T20:02:09.948 DEBUG:boto:Headers: {} 2023-05-31T20:02:09.949 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:02:09.949 DEBUG:boto:Port: 8001 2023-05-31T20:02:09.949 DEBUG:boto:Params: {} 2023-05-31T20:02:09.950 DEBUG:boto:Token: None 2023-05-31T20:02:09.950 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:02:09 GMT /pklxkd-14/obj?versionId=null 2023-05-31T20:02:09.951 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:Gtf8/60ec3G1FzEwPyLdQ73TMW0= 2023-05-31T20:02:09.951 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:02:09 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:Gtf8/60ec3G1FzEwPyLdQ73TMW0=', 'Content-Length': '0'} 2023-05-31T20:02:09.953 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:00:53 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx0000033cb733f14c945a2-006477a7c1-1112-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:02:09 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:02:09.953 DEBUG:boto:path=/pklxkd-14/obj 2023-05-31T20:02:09.954 DEBUG:boto:auth_path=/pklxkd-14/obj 2023-05-31T20:02:09.954 DEBUG:boto:path=/pklxkd-14/obj?versionId=null 2023-05-31T20:02:09.955 DEBUG:boto:auth_path=/pklxkd-14/obj?versionId=null 2023-05-31T20:02:09.955 DEBUG:boto:Method: GET 2023-05-31T20:02:09.955 DEBUG:boto:Path: /pklxkd-14/obj?versionId=null 2023-05-31T20:02:09.956 DEBUG:boto:Data: 2023-05-31T20:02:09.957 DEBUG:boto:Headers: {} 2023-05-31T20:02:09.957 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:02:09.957 DEBUG:boto:Port: 8000 2023-05-31T20:02:09.958 DEBUG:boto:Params: {} 2023-05-31T20:02:09.958 DEBUG:boto:Token: None 2023-05-31T20:02:09.958 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:02:09 GMT /pklxkd-14/obj?versionId=null 2023-05-31T20:02:09.959 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:Gtf8/60ec3G1FzEwPyLdQ73TMW0= 2023-05-31T20:02:09.959 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:02:09 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:Gtf8/60ec3G1FzEwPyLdQ73TMW0=', 'Content-Length': '0'} 2023-05-31T20:02:09.961 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:00:53 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000bc2be156590134b5-006477a7c1-110e-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:02:09 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:02:09.962 DEBUG:boto:path=/pklxkd-14/obj 2023-05-31T20:02:09.962 DEBUG:boto:auth_path=/pklxkd-14/obj 2023-05-31T20:02:09.962 DEBUG:boto:path=/pklxkd-14/obj?versionId=null 2023-05-31T20:02:09.963 DEBUG:boto:auth_path=/pklxkd-14/obj?versionId=null 2023-05-31T20:02:09.964 DEBUG:boto:Method: HEAD 2023-05-31T20:02:09.964 DEBUG:boto:Path: /pklxkd-14/obj?versionId=null 2023-05-31T20:02:09.964 DEBUG:boto:Data: 2023-05-31T20:02:09.965 DEBUG:boto:Headers: {} 2023-05-31T20:02:09.965 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:02:09.966 DEBUG:boto:Port: 8001 2023-05-31T20:02:09.966 DEBUG:boto:Params: {} 2023-05-31T20:02:09.967 DEBUG:boto:Token: None 2023-05-31T20:02:09.967 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:02:09 GMT /pklxkd-14/obj?versionId=null 2023-05-31T20:02:09.968 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:XuS7hdnAhyhQq7BeyEIQtoWOFZc= 2023-05-31T20:02:09.969 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:02:09 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:XuS7hdnAhyhQq7BeyEIQtoWOFZc=', 'Content-Length': '0'} 2023-05-31T20:02:09.970 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:00:53 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000008344eaaab60366f9-006477a7c1-1112-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:02:09 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:02:09.971 DEBUG:boto:path=/pklxkd-14/obj 2023-05-31T20:02:09.971 DEBUG:boto:auth_path=/pklxkd-14/obj 2023-05-31T20:02:09.972 DEBUG:boto:path=/pklxkd-14/obj?versionId=null 2023-05-31T20:02:09.972 DEBUG:boto:auth_path=/pklxkd-14/obj?versionId=null 2023-05-31T20:02:09.972 DEBUG:boto:Method: HEAD 2023-05-31T20:02:09.973 DEBUG:boto:Path: /pklxkd-14/obj?versionId=null 2023-05-31T20:02:09.973 DEBUG:boto:Data: 2023-05-31T20:02:09.973 DEBUG:boto:Headers: {} 2023-05-31T20:02:09.974 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:02:09.974 DEBUG:boto:Port: 8000 2023-05-31T20:02:09.974 DEBUG:boto:Params: {} 2023-05-31T20:02:09.975 DEBUG:boto:Token: None 2023-05-31T20:02:09.975 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:02:09 GMT /pklxkd-14/obj?versionId=null 2023-05-31T20:02:09.976 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:XuS7hdnAhyhQq7BeyEIQtoWOFZc= 2023-05-31T20:02:09.976 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:02:09 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:XuS7hdnAhyhQq7BeyEIQtoWOFZc=', 'Content-Length': '0'} 2023-05-31T20:02:09.978 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:00:53 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000c2d01d794f40c84a-006477a7c1-110e-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:02:09 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:02:09.979 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2023-05-31T20:02:09.980 DEBUG:boto:path=/pklxkd-14/obj 2023-05-31T20:02:09.980 DEBUG:boto:auth_path=/pklxkd-14/obj 2023-05-31T20:02:09.981 DEBUG:boto:path=/pklxkd-14/obj?versionId=null 2023-05-31T20:02:09.982 DEBUG:boto:auth_path=/pklxkd-14/obj?versionId=null 2023-05-31T20:02:09.983 DEBUG:boto:Method: GET 2023-05-31T20:02:09.983 DEBUG:boto:Path: /pklxkd-14/obj?versionId=null 2023-05-31T20:02:09.984 DEBUG:boto:Data: 2023-05-31T20:02:09.985 DEBUG:boto:Headers: {} 2023-05-31T20:02:09.985 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:02:09.986 DEBUG:boto:Port: 8001 2023-05-31T20:02:09.987 DEBUG:boto:Params: {} 2023-05-31T20:02:09.988 DEBUG:boto:Token: None 2023-05-31T20:02:09.988 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:02:09 GMT /pklxkd-14/obj?versionId=null 2023-05-31T20:02:09.989 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:Gtf8/60ec3G1FzEwPyLdQ73TMW0= 2023-05-31T20:02:09.990 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:02:09 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:Gtf8/60ec3G1FzEwPyLdQ73TMW0=', 'Content-Length': '0'} 2023-05-31T20:02:09.992 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:00:53 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000055cfe5d230b6508-006477a7c1-1112-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:02:09 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:02:09.993 DEBUG:boto:path=/pklxkd-14/obj 2023-05-31T20:02:09.994 DEBUG:boto:auth_path=/pklxkd-14/obj 2023-05-31T20:02:09.995 DEBUG:boto:path=/pklxkd-14/obj?versionId=null 2023-05-31T20:02:09.995 DEBUG:boto:auth_path=/pklxkd-14/obj?versionId=null 2023-05-31T20:02:09.996 DEBUG:boto:Method: GET 2023-05-31T20:02:09.997 DEBUG:boto:Path: /pklxkd-14/obj?versionId=null 2023-05-31T20:02:09.998 DEBUG:boto:Data: 2023-05-31T20:02:09.999 DEBUG:boto:Headers: {} 2023-05-31T20:02:10.000 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:02:10.001 DEBUG:boto:Port: 8000 2023-05-31T20:02:10.002 DEBUG:boto:Params: {} 2023-05-31T20:02:10.002 DEBUG:boto:Token: None 2023-05-31T20:02:10.003 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:02:10 GMT /pklxkd-14/obj?versionId=null 2023-05-31T20:02:10.003 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:UiX6Szi4exIgrOF4ScnSzMKwdw8= 2023-05-31T20:02:10.004 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:02:10 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:UiX6Szi4exIgrOF4ScnSzMKwdw8=', 'Content-Length': '0'} 2023-05-31T20:02:10.006 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:00:53 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx0000075205dcfeb5e4e82-006477a7c1-110e-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:02:10 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:02:10.006 DEBUG:boto:path=/pklxkd-14/ 2023-05-31T20:02:10.007 DEBUG:boto:auth_path=/pklxkd-14/ 2023-05-31T20:02:10.007 DEBUG:boto:path=/pklxkd-14/?prefix=obj 2023-05-31T20:02:10.008 DEBUG:boto:auth_path=/pklxkd-14/?prefix=obj 2023-05-31T20:02:10.008 DEBUG:boto:Method: GET 2023-05-31T20:02:10.008 DEBUG:boto:Path: /pklxkd-14/?prefix=obj 2023-05-31T20:02:10.009 DEBUG:boto:Data: 2023-05-31T20:02:10.009 DEBUG:boto:Headers: {} 2023-05-31T20:02:10.009 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:02:10.010 DEBUG:boto:Port: 8001 2023-05-31T20:02:10.010 DEBUG:boto:Params: {} 2023-05-31T20:02:10.011 DEBUG:boto:Token: None 2023-05-31T20:02:10.011 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:02:10 GMT /pklxkd-14/ 2023-05-31T20:02:10.012 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:3FyUD1McPgSqxAlr56rV9uN9FDo= 2023-05-31T20:02:10.012 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:02:10 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:3FyUD1McPgSqxAlr56rV9uN9FDo=', 'Content-Length': '0'} 2023-05-31T20:02:10.015 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000411162ab34854fc4-006477a7c2-1112-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:02:10 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:02:10.056 DEBUG:boto:b'pklxkd-14obj1000falseobj2023-05-31T20:01:29.347Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:02:10.057 DEBUG:boto:path=/pklxkd-14/ 2023-05-31T20:02:10.058 DEBUG:boto:auth_path=/pklxkd-14/ 2023-05-31T20:02:10.058 DEBUG:boto:path=/pklxkd-14/?prefix=obj 2023-05-31T20:02:10.058 DEBUG:boto:auth_path=/pklxkd-14/?prefix=obj 2023-05-31T20:02:10.059 DEBUG:boto:Method: GET 2023-05-31T20:02:10.059 DEBUG:boto:Path: /pklxkd-14/?prefix=obj 2023-05-31T20:02:10.060 DEBUG:boto:Data: 2023-05-31T20:02:10.061 DEBUG:boto:Headers: {} 2023-05-31T20:02:10.061 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:02:10.061 DEBUG:boto:Port: 8000 2023-05-31T20:02:10.062 DEBUG:boto:Params: {} 2023-05-31T20:02:10.062 DEBUG:boto:Token: None 2023-05-31T20:02:10.063 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:02:10 GMT /pklxkd-14/ 2023-05-31T20:02:10.064 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:3FyUD1McPgSqxAlr56rV9uN9FDo= 2023-05-31T20:02:10.064 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:02:10 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:3FyUD1McPgSqxAlr56rV9uN9FDo=', 'Content-Length': '0'} 2023-05-31T20:02:10.067 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000618382e87c027cf9-006477a7c2-110e-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:02:10 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:02:10.068 DEBUG:boto:b'pklxkd-14obj1000falseobj2023-05-31T20:01:29.347Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:02:10.069 DEBUG:boto:path=/pklxkd-14/obj 2023-05-31T20:02:10.070 DEBUG:boto:auth_path=/pklxkd-14/obj 2023-05-31T20:02:10.070 DEBUG:boto:Method: HEAD 2023-05-31T20:02:10.071 DEBUG:boto:Path: /pklxkd-14/obj 2023-05-31T20:02:10.072 DEBUG:boto:Data: 2023-05-31T20:02:10.072 DEBUG:boto:Headers: {} 2023-05-31T20:02:10.073 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:02:10.074 DEBUG:boto:Port: 8001 2023-05-31T20:02:10.074 DEBUG:boto:Params: {} 2023-05-31T20:02:10.075 DEBUG:boto:Token: None 2023-05-31T20:02:10.076 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:02:10 GMT /pklxkd-14/obj 2023-05-31T20:02:10.077 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:N3tUJt1XZNpjoqNoWc6wPk5A5k4= 2023-05-31T20:02:10.077 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:02:10 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:N3tUJt1XZNpjoqNoWc6wPk5A5k4=', 'Content-Length': '0'} 2023-05-31T20:02:10.080 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:01:29 GMT'), ('x-amz-version-id', '1c5VIADR1kuOHtAIi0CwrS4UaPxAfUI'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000008d4c158fc7090674-006477a7c2-1112-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:02:10 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:02:10.081 DEBUG:boto:path=/pklxkd-14/obj 2023-05-31T20:02:10.082 DEBUG:boto:auth_path=/pklxkd-14/obj 2023-05-31T20:02:10.082 DEBUG:boto:Method: HEAD 2023-05-31T20:02:10.083 DEBUG:boto:Path: /pklxkd-14/obj 2023-05-31T20:02:10.083 DEBUG:boto:Data: 2023-05-31T20:02:10.084 DEBUG:boto:Headers: {} 2023-05-31T20:02:10.085 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:02:10.085 DEBUG:boto:Port: 8000 2023-05-31T20:02:10.086 DEBUG:boto:Params: {} 2023-05-31T20:02:10.087 DEBUG:boto:Token: None 2023-05-31T20:02:10.087 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:02:10 GMT /pklxkd-14/obj 2023-05-31T20:02:10.087 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:N3tUJt1XZNpjoqNoWc6wPk5A5k4= 2023-05-31T20:02:10.088 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:02:10 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:N3tUJt1XZNpjoqNoWc6wPk5A5k4=', 'Content-Length': '0'} 2023-05-31T20:02:10.090 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:01:29 GMT'), ('x-amz-version-id', '1c5VIADR1kuOHtAIi0CwrS4UaPxAfUI'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000c1cbcf6a501089e9-006477a7c2-110e-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:02:10 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:02:10.091 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2023-05-31T20:02:10.091 DEBUG:boto:path=/pklxkd-14/obj 2023-05-31T20:02:10.091 DEBUG:boto:auth_path=/pklxkd-14/obj 2023-05-31T20:02:10.092 DEBUG:boto:path=/pklxkd-14/obj?versionId=1c5VIADR1kuOHtAIi0CwrS4UaPxAfUI 2023-05-31T20:02:10.092 DEBUG:boto:auth_path=/pklxkd-14/obj?versionId=1c5VIADR1kuOHtAIi0CwrS4UaPxAfUI 2023-05-31T20:02:10.093 DEBUG:boto:Method: GET 2023-05-31T20:02:10.093 DEBUG:boto:Path: /pklxkd-14/obj?versionId=1c5VIADR1kuOHtAIi0CwrS4UaPxAfUI 2023-05-31T20:02:10.093 DEBUG:boto:Data: 2023-05-31T20:02:10.094 DEBUG:boto:Headers: {} 2023-05-31T20:02:10.094 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:02:10.094 DEBUG:boto:Port: 8001 2023-05-31T20:02:10.095 DEBUG:boto:Params: {} 2023-05-31T20:02:10.095 DEBUG:boto:Token: None 2023-05-31T20:02:10.096 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:02:10 GMT /pklxkd-14/obj?versionId=1c5VIADR1kuOHtAIi0CwrS4UaPxAfUI 2023-05-31T20:02:10.096 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:P4IfDB7M4HdjLlPYekH98MU71EY= 2023-05-31T20:02:10.096 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:02:10 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:P4IfDB7M4HdjLlPYekH98MU71EY=', 'Content-Length': '0'} 2023-05-31T20:02:10.098 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:01:29 GMT'), ('x-amz-version-id', '1c5VIADR1kuOHtAIi0CwrS4UaPxAfUI'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000b4b08ff914b8cf81-006477a7c2-1112-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:02:10 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:02:10.099 DEBUG:boto:path=/pklxkd-14/obj 2023-05-31T20:02:10.099 DEBUG:boto:auth_path=/pklxkd-14/obj 2023-05-31T20:02:10.100 DEBUG:boto:path=/pklxkd-14/obj?versionId=1c5VIADR1kuOHtAIi0CwrS4UaPxAfUI 2023-05-31T20:02:10.100 DEBUG:boto:auth_path=/pklxkd-14/obj?versionId=1c5VIADR1kuOHtAIi0CwrS4UaPxAfUI 2023-05-31T20:02:10.100 DEBUG:boto:Method: GET 2023-05-31T20:02:10.101 DEBUG:boto:Path: /pklxkd-14/obj?versionId=1c5VIADR1kuOHtAIi0CwrS4UaPxAfUI 2023-05-31T20:02:10.101 DEBUG:boto:Data: 2023-05-31T20:02:10.101 DEBUG:boto:Headers: {} 2023-05-31T20:02:10.102 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:02:10.102 DEBUG:boto:Port: 8000 2023-05-31T20:02:10.102 DEBUG:boto:Params: {} 2023-05-31T20:02:10.103 DEBUG:boto:Token: None 2023-05-31T20:02:10.103 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:02:10 GMT /pklxkd-14/obj?versionId=1c5VIADR1kuOHtAIi0CwrS4UaPxAfUI 2023-05-31T20:02:10.104 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:P4IfDB7M4HdjLlPYekH98MU71EY= 2023-05-31T20:02:10.104 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:02:10 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:P4IfDB7M4HdjLlPYekH98MU71EY=', 'Content-Length': '0'} 2023-05-31T20:02:10.106 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:01:29 GMT'), ('x-amz-version-id', '1c5VIADR1kuOHtAIi0CwrS4UaPxAfUI'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000173c44e08eab23e0-006477a7c2-110e-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:02:10 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:02:10.107 DEBUG:boto:path=/pklxkd-14/ 2023-05-31T20:02:10.107 DEBUG:boto:auth_path=/pklxkd-14/ 2023-05-31T20:02:10.107 DEBUG:boto:path=/pklxkd-14/?prefix=obj 2023-05-31T20:02:10.108 DEBUG:boto:auth_path=/pklxkd-14/?prefix=obj 2023-05-31T20:02:10.108 DEBUG:boto:Method: GET 2023-05-31T20:02:10.109 DEBUG:boto:Path: /pklxkd-14/?prefix=obj 2023-05-31T20:02:10.109 DEBUG:boto:Data: 2023-05-31T20:02:10.110 DEBUG:boto:Headers: {} 2023-05-31T20:02:10.110 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:02:10.110 DEBUG:boto:Port: 8001 2023-05-31T20:02:10.111 DEBUG:boto:Params: {} 2023-05-31T20:02:10.111 DEBUG:boto:Token: None 2023-05-31T20:02:10.111 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:02:10 GMT /pklxkd-14/ 2023-05-31T20:02:10.112 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:3FyUD1McPgSqxAlr56rV9uN9FDo= 2023-05-31T20:02:10.112 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:02:10 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:3FyUD1McPgSqxAlr56rV9uN9FDo=', 'Content-Length': '0'} 2023-05-31T20:02:10.114 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d0bb4c5b2e3a09e5-006477a7c2-1112-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:02:10 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:02:10.115 DEBUG:boto:b'pklxkd-14obj1000falseobj2023-05-31T20:01:29.347Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:02:10.116 DEBUG:boto:path=/pklxkd-14/ 2023-05-31T20:02:10.116 DEBUG:boto:auth_path=/pklxkd-14/ 2023-05-31T20:02:10.117 DEBUG:boto:path=/pklxkd-14/?prefix=obj 2023-05-31T20:02:10.117 DEBUG:boto:auth_path=/pklxkd-14/?prefix=obj 2023-05-31T20:02:10.117 DEBUG:boto:Method: GET 2023-05-31T20:02:10.118 DEBUG:boto:Path: /pklxkd-14/?prefix=obj 2023-05-31T20:02:10.118 DEBUG:boto:Data: 2023-05-31T20:02:10.118 DEBUG:boto:Headers: {} 2023-05-31T20:02:10.119 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:02:10.119 DEBUG:boto:Port: 8000 2023-05-31T20:02:10.119 DEBUG:boto:Params: {} 2023-05-31T20:02:10.120 DEBUG:boto:Token: None 2023-05-31T20:02:10.120 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:02:10 GMT /pklxkd-14/ 2023-05-31T20:02:10.121 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:3FyUD1McPgSqxAlr56rV9uN9FDo= 2023-05-31T20:02:10.121 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:02:10 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:3FyUD1McPgSqxAlr56rV9uN9FDo=', 'Content-Length': '0'} 2023-05-31T20:02:10.123 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000000e1211e0d05f9e1f-006477a7c2-110e-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:02:10 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:02:10.124 DEBUG:boto:b'pklxkd-14obj1000falseobj2023-05-31T20:01:29.347Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:02:10.125 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=pklxkd-14 zones={a2, a1} 2023-05-31T20:02:10.125 DEBUG:boto:path=/pklxkd-14/ 2023-05-31T20:02:10.125 DEBUG:boto:auth_path=/pklxkd-14/ 2023-05-31T20:02:10.126 DEBUG:boto:path=/pklxkd-14/?versioning 2023-05-31T20:02:10.126 DEBUG:boto:auth_path=/pklxkd-14/?versioning 2023-05-31T20:02:10.126 DEBUG:boto:Method: PUT 2023-05-31T20:02:10.127 DEBUG:boto:Path: /pklxkd-14/?versioning 2023-05-31T20:02:10.127 DEBUG:boto:Data: Suspended Disabled 2023-05-31T20:02:10.127 DEBUG:boto:Headers: {} 2023-05-31T20:02:10.128 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:02:10.128 DEBUG:boto:Port: 8000 2023-05-31T20:02:10.128 DEBUG:boto:Params: {} 2023-05-31T20:02:10.129 DEBUG:boto:Token: None 2023-05-31T20:02:10.129 DEBUG:boto:StringToSign: PUT Wed, 31 May 2023 20:02:10 GMT /pklxkd-14/?versioning 2023-05-31T20:02:10.130 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:2VpCMsSibgCKbN3CyTEsr7EiUa0= 2023-05-31T20:02:10.130 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:02:10 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:2VpCMsSibgCKbN3CyTEsr7EiUa0=', 'Content-Length': '230'} 2023-05-31T20:02:10.136 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000813722f390d504f8-006477a7c2-110e-a1'), ('Content-Type', 'application/xml'), ('Content-Length', '0'), ('Date', 'Wed, 31 May 2023 20:02:10 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:02:10.136 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-31T20:02:10.137 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-31T20:02:10.137 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-05-31T20:02:10.193 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2023-05-31T20:02:10.193 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2023-05-31T20:02:10.231 INFO:teuthology.orchestra.run.smithi099.stderr:No --period given, using current period=2e6123d2-d695-4350-9a01-61a455bc782b 2023-05-31T20:02:10.249 INFO:teuthology.orchestra.run.smithi099.stdout:[ 2023-05-31T20:02:10.250 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T20:02:10.250 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1685562631.516089_59.1", 2023-05-31T20:02:10.250 INFO:teuthology.orchestra.run.smithi099.stdout: "last_update": "2023-05-31T19:50:31.516089Z" 2023-05-31T20:02:10.250 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:02:10.250 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T20:02:10.250 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "", 2023-05-31T20:02:10.251 INFO:teuthology.orchestra.run.smithi099.stdout: "last_update": "0.000000" 2023-05-31T20:02:10.251 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:02:10.251 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T20:02:10.251 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1685562970.274572_93.1", 2023-05-31T20:02:10.251 INFO:teuthology.orchestra.run.smithi099.stdout: "last_update": "2023-05-31T19:56:10.274572Z" 2023-05-31T20:02:10.251 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:02:10.251 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T20:02:10.252 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1685563330.130935_76.1", 2023-05-31T20:02:10.252 INFO:teuthology.orchestra.run.smithi099.stdout: "last_update": "2023-05-31T20:02:10.130935Z" 2023-05-31T20:02:10.252 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T20:02:10.252 INFO:teuthology.orchestra.run.smithi099.stdout:] 2023-05-31T20:02:10.253 DEBUG:rgw_multi.tests:master meta markers={0: '1_1685562631.516089_59.1', 1: '', 2: '1_1685562970.274572_93.1', 3: '1_1685563330.130935_76.1'} 2023-05-31T20:02:10.253 INFO:rgw_multi.tests:starting meta checkpoint for zone=a2 2023-05-31T20:02:10.254 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-31T20:02:10.254 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-31T20:02:10.254 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-05-31T20:02:10.305 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2023-05-31T20:02:10.306 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2023-05-31T20:02:10.357 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:02:10.351+0000 7f31061f9780 20 RGW-SYNC:meta: read sync status 2023-05-31T20:02:10.358 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:02:10.352+0000 7f31061f9780 20 RGW-SYNC:meta: read sync status 2023-05-31T20:02:10.373 INFO:teuthology.orchestra.run.smithi099.stdout:{ 2023-05-31T20:02:10.373 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_status": { 2023-05-31T20:02:10.373 INFO:teuthology.orchestra.run.smithi099.stdout: "info": { 2023-05-31T20:02:10.374 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "sync", 2023-05-31T20:02:10.374 INFO:teuthology.orchestra.run.smithi099.stdout: "num_shards": 4, 2023-05-31T20:02:10.374 INFO:teuthology.orchestra.run.smithi099.stdout: "period": "2e6123d2-d695-4350-9a01-61a455bc782b", 2023-05-31T20:02:10.374 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2023-05-31T20:02:10.374 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:02:10.374 INFO:teuthology.orchestra.run.smithi099.stdout: "markers": [ 2023-05-31T20:02:10.374 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T20:02:10.375 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 0, 2023-05-31T20:02:10.375 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T20:02:10.375 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2023-05-31T20:02:10.375 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1685562631.516089_59.1", 2023-05-31T20:02:10.375 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T20:02:10.375 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 1, 2023-05-31T20:02:10.375 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T20:02:10.376 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2023-05-31T19:50:31.516089Z", 2023-05-31T20:02:10.376 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2023-05-31T20:02:10.376 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T20:02:10.376 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:02:10.376 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T20:02:10.376 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 1, 2023-05-31T20:02:10.377 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T20:02:10.377 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2023-05-31T20:02:10.377 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "", 2023-05-31T20:02:10.377 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T20:02:10.377 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2023-05-31T20:02:10.377 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T20:02:10.377 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "0.000000", 2023-05-31T20:02:10.378 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2023-05-31T20:02:10.378 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T20:02:10.378 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:02:10.378 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T20:02:10.378 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 2, 2023-05-31T20:02:10.378 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T20:02:10.378 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2023-05-31T20:02:10.379 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1685562970.274572_93.1", 2023-05-31T20:02:10.379 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T20:02:10.379 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2023-05-31T20:02:10.379 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T20:02:10.379 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2023-05-31T19:56:10.274572Z", 2023-05-31T20:02:10.379 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2023-05-31T20:02:10.379 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T20:02:10.380 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:02:10.380 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T20:02:10.380 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 3, 2023-05-31T20:02:10.380 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T20:02:10.380 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2023-05-31T20:02:10.380 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1685563330.130935_76.1", 2023-05-31T20:02:10.381 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T20:02:10.381 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2023-05-31T20:02:10.381 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T20:02:10.381 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2023-05-31T20:02:10.130935Z", 2023-05-31T20:02:10.381 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2023-05-31T20:02:10.381 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T20:02:10.381 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T20:02:10.382 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2023-05-31T20:02:10.382 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:02:10.382 INFO:teuthology.orchestra.run.smithi099.stdout: "full_sync": { 2023-05-31T20:02:10.382 INFO:teuthology.orchestra.run.smithi099.stdout: "total": 1, 2023-05-31T20:02:10.382 INFO:teuthology.orchestra.run.smithi099.stdout: "complete": 1 2023-05-31T20:02:10.382 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:02:10.382 INFO:teuthology.orchestra.run.smithi099.stdout: "current_time": "2023-05-31T20:02:10Z" 2023-05-31T20:02:10.383 INFO:teuthology.orchestra.run.smithi099.stdout:} 2023-05-31T20:02:10.383 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "2e6123d2-d695-4350-9a01-61a455bc782b", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1685562631.516089_59.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-05-31T19:50:31.516089Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1685562970.274572_93.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-05-31T19:56:10.274572Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1685563330.130935_76.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-05-31T20:02:10.130935Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-05-31T20:02:10Z" } 2023-05-31T20:02:10.384 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1685562631.516089_59.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-05-31T19:50:31.516089Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1685562970.274572_93.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-05-31T19:56:10.274572Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1685563330.130935_76.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-05-31T20:02:10.130935Z', 'realm_epoch': 2}}] 2023-05-31T20:02:10.384 DEBUG:rgw_multi.tests:log_status={0: '1_1685562631.516089_59.1', 1: '', 2: '1_1685562970.274572_93.1', 3: '1_1685563330.130935_76.1'} 2023-05-31T20:02:10.385 DEBUG:rgw_multi.tests:sync_status={0: '1_1685562631.516089_59.1', 1: '', 2: '1_1685562970.274572_93.1', 3: '1_1685563330.130935_76.1'} 2023-05-31T20:02:10.385 INFO:rgw_multi.tests:finish meta checkpoint for zone=a2 2023-05-31T20:02:10.385 DEBUG:boto:path=/pklxkd-14/ 2023-05-31T20:02:10.386 DEBUG:boto:auth_path=/pklxkd-14/ 2023-05-31T20:02:10.386 DEBUG:boto:Method: HEAD 2023-05-31T20:02:10.386 DEBUG:boto:Path: /pklxkd-14/ 2023-05-31T20:02:10.387 DEBUG:boto:Data: 2023-05-31T20:02:10.387 DEBUG:boto:Headers: {} 2023-05-31T20:02:10.387 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:02:10.388 DEBUG:boto:Port: 8000 2023-05-31T20:02:10.388 DEBUG:boto:Params: {} 2023-05-31T20:02:10.388 DEBUG:boto:Token: None 2023-05-31T20:02:10.389 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:02:10 GMT /pklxkd-14/ 2023-05-31T20:02:10.389 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:0TtzD30lsRF95RO0RDzQrJBxhYE= 2023-05-31T20:02:10.390 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:02:10 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:0TtzD30lsRF95RO0RDzQrJBxhYE=', 'Content-Length': '0'} 2023-05-31T20:02:10.392 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '2'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000eaaa2f766b7f1e98-006477a7c2-110e-a1'), ('Content-Length', '0'), ('Date', 'Wed, 31 May 2023 20:02:10 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:02:10.392 DEBUG:boto:path=/pklxkd-14/obj 2023-05-31T20:02:10.393 DEBUG:boto:auth_path=/pklxkd-14/obj 2023-05-31T20:02:10.393 DEBUG:boto:Method: PUT 2023-05-31T20:02:10.393 DEBUG:boto:Path: /pklxkd-14/obj 2023-05-31T20:02:10.394 DEBUG:boto:Data: 2023-05-31T20:02:10.394 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': '1B2M2Y8AsgTpgAmY7PhCfg==', 'Content-Length': '0', 'Expect': '100-Continue'} 2023-05-31T20:02:10.394 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:02:10.395 DEBUG:boto:Port: 8000 2023-05-31T20:02:10.395 DEBUG:boto:Params: {} 2023-05-31T20:02:10.395 DEBUG:boto:Token: None 2023-05-31T20:02:10.396 DEBUG:boto:StringToSign: PUT 1B2M2Y8AsgTpgAmY7PhCfg== application/octet-stream Wed, 31 May 2023 20:02:10 GMT /pklxkd-14/obj 2023-05-31T20:02:10.396 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:AQVZVpLtRMtzhUuUOnnuOyw77+s= 2023-05-31T20:02:10.397 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': '1B2M2Y8AsgTpgAmY7PhCfg==', 'Content-Length': '0', 'Expect': '100-Continue', 'Date': 'Wed, 31 May 2023 20:02:10 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:AQVZVpLtRMtzhUuUOnnuOyw77+s='} 2023-05-31T20:02:10.412 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000b804dcb42e5af39f-006477a7c2-110e-a1'), ('Date', 'Wed, 31 May 2023 20:02:10 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:02:10.412 DEBUG:rgw_multi.tests:created null version id=None 2023-05-31T20:02:10.413 DEBUG:rgw_multi.tests:bucket checkpoint: source=a1 target=a2 bucket=pklxkd-14 2023-05-31T20:02:10.413 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'pklxkd-14', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-05-31T20:02:10.413 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'pklxkd-14', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-05-31T20:02:10.414 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket pklxkd-14 --source-zone a1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2023-05-31T20:02:10.470 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2023-05-31T20:02:10.470 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2023-05-31T20:02:10.521 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:02:10.515+0000 7fe27c0db780 1 waiting for incremental sync to catch up: 2023-05-31T20:02:10.521 INFO:teuthology.orchestra.run.smithi099.stderr: local status: [, , , , , , , , 00000000003.8.13, , ] 2023-05-31T20:02:10.521 INFO:teuthology.orchestra.run.smithi099.stderr: remote markers: [, , , , , , , , 00000000005.12.18, , ] 2023-05-31T20:02:15.522 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:02:15.516+0000 7fe27c0db780 1 waiting for incremental sync to catch up: 2023-05-31T20:02:15.522 INFO:teuthology.orchestra.run.smithi099.stderr: local status: [, , , , , , , , 00000000003.8.13, , ] 2023-05-31T20:02:15.522 INFO:teuthology.orchestra.run.smithi099.stderr: remote markers: [, , , , , , , , 00000000005.12.18, , ] 2023-05-31T20:02:20.523 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:02:20.517+0000 7fe27c0db780 1 waiting for incremental sync to catch up: 2023-05-31T20:02:20.524 INFO:teuthology.orchestra.run.smithi099.stderr: local status: [, , , , , , , , 00000000003.8.13, , ] 2023-05-31T20:02:20.524 INFO:teuthology.orchestra.run.smithi099.stderr: remote markers: [, , , , , , , , 00000000005.12.18, , ] 2023-05-31T20:02:25.524 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:02:25.519+0000 7fe27c0db780 1 waiting for incremental sync to catch up: 2023-05-31T20:02:25.525 INFO:teuthology.orchestra.run.smithi099.stderr: local status: [, , , , , , , , 00000000003.8.13, , ] 2023-05-31T20:02:25.525 INFO:teuthology.orchestra.run.smithi099.stderr: remote markers: [, , , , , , , , 00000000005.12.18, , ] 2023-05-31T20:02:30.526 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:02:30.520+0000 7fe27c0db780 1 waiting for incremental sync to catch up: 2023-05-31T20:02:30.526 INFO:teuthology.orchestra.run.smithi099.stderr: local status: [, , , , , , , , 00000000003.8.13, , ] 2023-05-31T20:02:30.526 INFO:teuthology.orchestra.run.smithi099.stderr: remote markers: [, , , , , , , , 00000000005.12.18, , ] 2023-05-31T20:02:35.527 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:02:35.522+0000 7fe27c0db780 1 waiting for incremental sync to catch up: 2023-05-31T20:02:35.528 INFO:teuthology.orchestra.run.smithi099.stderr: local status: [, , , , , , , , 00000000003.8.13, , ] 2023-05-31T20:02:35.528 INFO:teuthology.orchestra.run.smithi099.stderr: remote markers: [, , , , , , , , 00000000005.12.18, , ] 2023-05-31T20:02:40.528 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:02:40.523+0000 7fe27c0db780 1 waiting for incremental sync to catch up: 2023-05-31T20:02:40.529 INFO:teuthology.orchestra.run.smithi099.stderr: local status: [, , , , , , , , 00000000003.8.13, , ] 2023-05-31T20:02:40.529 INFO:teuthology.orchestra.run.smithi099.stderr: remote markers: [, , , , , , , , 00000000005.12.18, , ] 2023-05-31T20:02:45.530 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:02:45.525+0000 7fe27c0db780 1 waiting for incremental sync to catch up: 2023-05-31T20:02:45.530 INFO:teuthology.orchestra.run.smithi099.stderr: local status: [, , , , , , , , 00000000003.8.13, , ] 2023-05-31T20:02:45.530 INFO:teuthology.orchestra.run.smithi099.stderr: remote markers: [, , , , , , , , 00000000005.12.18, , ] 2023-05-31T20:02:50.531 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:02:50.526+0000 7fe27c0db780 1 bucket sync caught up with source: 2023-05-31T20:02:50.531 INFO:teuthology.orchestra.run.smithi099.stderr: local status: [, , , , , , , , 00000000005.12.18, , ] 2023-05-31T20:02:50.531 INFO:teuthology.orchestra.run.smithi099.stderr: remote markers: [, , , , , , , , 00000000005.12.18, , ] 2023-05-31T20:02:50.531 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:02:50.526+0000 7fe27c0db780 0 bucket checkpoint complete 2023-05-31T20:02:50.548 DEBUG:rgw_multi.tests:bucket checkpoint: source=a2 target=a1 bucket=pklxkd-14 2023-05-31T20:02:50.549 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'pklxkd-14', '--source-zone', 'a2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-05-31T20:02:50.549 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'pklxkd-14', '--source-zone', 'a2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-05-31T20:02:50.550 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket pklxkd-14 --source-zone a2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2023-05-31T20:02:50.611 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2023-05-31T20:02:50.612 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2023-05-31T20:02:50.662 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:02:50.657+0000 7fdba118d780 1 waiting for incremental sync to catch up: 2023-05-31T20:02:50.662 INFO:teuthology.orchestra.run.smithi099.stderr: local status: [, , , , , , , , 00000000003.11.13, , ] 2023-05-31T20:02:50.662 INFO:teuthology.orchestra.run.smithi099.stderr: remote markers: [, , , , , , , , 00000000005.16.18, , ] 2023-05-31T20:02:55.663 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:02:55.657+0000 7fdba118d780 1 waiting for incremental sync to catch up: 2023-05-31T20:02:55.663 INFO:teuthology.orchestra.run.smithi099.stderr: local status: [, , , , , , , , 00000000003.11.13, , ] 2023-05-31T20:02:55.664 INFO:teuthology.orchestra.run.smithi099.stderr: remote markers: [, , , , , , , , 00000000005.16.18, , ] 2023-05-31T20:03:00.664 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:03:00.658+0000 7fdba118d780 1 waiting for incremental sync to catch up: 2023-05-31T20:03:00.665 INFO:teuthology.orchestra.run.smithi099.stderr: local status: [, , , , , , , , 00000000003.11.13, , ] 2023-05-31T20:03:00.665 INFO:teuthology.orchestra.run.smithi099.stderr: remote markers: [, , , , , , , , 00000000005.16.18, , ] 2023-05-31T20:03:05.666 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:03:05.660+0000 7fdba118d780 1 waiting for incremental sync to catch up: 2023-05-31T20:03:05.666 INFO:teuthology.orchestra.run.smithi099.stderr: local status: [, , , , , , , , 00000000003.11.13, , ] 2023-05-31T20:03:05.666 INFO:teuthology.orchestra.run.smithi099.stderr: remote markers: [, , , , , , , , 00000000005.16.18, , ] 2023-05-31T20:03:10.667 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:03:10.661+0000 7fdba118d780 1 bucket sync caught up with source: 2023-05-31T20:03:10.668 INFO:teuthology.orchestra.run.smithi099.stderr: local status: [, , , , , , , , 00000000005.16.18, , ] 2023-05-31T20:03:10.668 INFO:teuthology.orchestra.run.smithi099.stderr: remote markers: [, , , , , , , , 00000000005.16.18, , ] 2023-05-31T20:03:10.668 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:03:10.661+0000 7fdba118d780 0 bucket checkpoint complete 2023-05-31T20:03:10.687 INFO:tasks.rgw_multi.zone_rados:comparing bucket=pklxkd-14 zones={a2, a1} 2023-05-31T20:03:10.687 DEBUG:boto:path=/pklxkd-14/ 2023-05-31T20:03:10.688 DEBUG:boto:auth_path=/pklxkd-14/ 2023-05-31T20:03:10.688 DEBUG:boto:Method: HEAD 2023-05-31T20:03:10.689 DEBUG:boto:Path: /pklxkd-14/ 2023-05-31T20:03:10.689 DEBUG:boto:Data: 2023-05-31T20:03:10.689 DEBUG:boto:Headers: {} 2023-05-31T20:03:10.690 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:03:10.690 DEBUG:boto:Port: 8001 2023-05-31T20:03:10.690 DEBUG:boto:Params: {} 2023-05-31T20:03:10.691 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8001} 2023-05-31T20:03:10.691 DEBUG:boto:Token: None 2023-05-31T20:03:10.692 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:03:10 GMT /pklxkd-14/ 2023-05-31T20:03:10.692 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:R3edKW6M6dC9SOBTh1KFm3yvcOg= 2023-05-31T20:03:10.693 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:03:10 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:R3edKW6M6dC9SOBTh1KFm3yvcOg=', 'Content-Length': '0'} 2023-05-31T20:03:10.696 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '2'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000005965ce3aeff73059-006477a7fe-1112-a2'), ('Content-Length', '0'), ('Date', 'Wed, 31 May 2023 20:03:10 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:03:10.697 DEBUG:boto:path=/pklxkd-14/ 2023-05-31T20:03:10.697 DEBUG:boto:auth_path=/pklxkd-14/ 2023-05-31T20:03:10.698 DEBUG:boto:Method: HEAD 2023-05-31T20:03:10.698 DEBUG:boto:Path: /pklxkd-14/ 2023-05-31T20:03:10.698 DEBUG:boto:Data: 2023-05-31T20:03:10.699 DEBUG:boto:Headers: {} 2023-05-31T20:03:10.699 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:03:10.699 DEBUG:boto:Port: 8000 2023-05-31T20:03:10.700 DEBUG:boto:Params: {} 2023-05-31T20:03:10.700 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2023-05-31T20:03:10.700 DEBUG:boto:Token: None 2023-05-31T20:03:10.701 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:03:10 GMT /pklxkd-14/ 2023-05-31T20:03:10.701 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:R3edKW6M6dC9SOBTh1KFm3yvcOg= 2023-05-31T20:03:10.702 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:03:10 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:R3edKW6M6dC9SOBTh1KFm3yvcOg=', 'Content-Length': '0'} 2023-05-31T20:03:10.705 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '2'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000003f7f3ae875ecc522-006477a7fe-110e-a1'), ('Content-Length', '0'), ('Date', 'Wed, 31 May 2023 20:03:10 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:03:10.705 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2023-05-31T20:03:10.706 DEBUG:boto:path=/pklxkd-14/ 2023-05-31T20:03:10.707 DEBUG:boto:auth_path=/pklxkd-14/ 2023-05-31T20:03:10.707 DEBUG:boto:path=/pklxkd-14/?versions&max-keys=999 2023-05-31T20:03:10.707 DEBUG:boto:auth_path=/pklxkd-14/?versions&max-keys=999 2023-05-31T20:03:10.708 DEBUG:boto:Method: GET 2023-05-31T20:03:10.708 DEBUG:boto:Path: /pklxkd-14/?versions&max-keys=999 2023-05-31T20:03:10.708 DEBUG:boto:Data: 2023-05-31T20:03:10.709 DEBUG:boto:Headers: {} 2023-05-31T20:03:10.709 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:03:10.709 DEBUG:boto:Port: 8001 2023-05-31T20:03:10.710 DEBUG:boto:Params: {} 2023-05-31T20:03:10.710 DEBUG:boto:Token: None 2023-05-31T20:03:10.711 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:03:10 GMT /pklxkd-14/?versions 2023-05-31T20:03:10.711 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:uRAp/aCbxRHW/61ow6JL3nsGpPI= 2023-05-31T20:03:10.712 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:03:10 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:uRAp/aCbxRHW/61ow6JL3nsGpPI=', 'Content-Length': '0'} 2023-05-31T20:03:10.714 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000009e99bc0803aae089-006477a7fe-1112-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:03:10 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:03:10.715 DEBUG:boto:b'pklxkd-14999falseobjnulltrue2023-05-31T20:02:10.394Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormalobj1c5VIADR1kuOHtAIi0CwrS4UaPxAfUIfalse2023-05-31T20:01:29.347Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:03:10.715 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2023-05-31T20:03:10.716 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2023-05-31T20:03:10.716 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2023-05-31T20:03:10.716 DEBUG:boto:path=/pklxkd-14/ 2023-05-31T20:03:10.717 DEBUG:boto:auth_path=/pklxkd-14/ 2023-05-31T20:03:10.717 DEBUG:boto:path=/pklxkd-14/?versions&max-keys=999 2023-05-31T20:03:10.718 DEBUG:boto:auth_path=/pklxkd-14/?versions&max-keys=999 2023-05-31T20:03:10.718 DEBUG:boto:Method: GET 2023-05-31T20:03:10.719 DEBUG:boto:Path: /pklxkd-14/?versions&max-keys=999 2023-05-31T20:03:10.719 DEBUG:boto:Data: 2023-05-31T20:03:10.719 DEBUG:boto:Headers: {} 2023-05-31T20:03:10.720 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:03:10.720 DEBUG:boto:Port: 8000 2023-05-31T20:03:10.720 DEBUG:boto:Params: {} 2023-05-31T20:03:10.721 DEBUG:boto:Token: None 2023-05-31T20:03:10.721 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:03:10 GMT /pklxkd-14/?versions 2023-05-31T20:03:10.721 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:uRAp/aCbxRHW/61ow6JL3nsGpPI= 2023-05-31T20:03:10.722 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:03:10 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:uRAp/aCbxRHW/61ow6JL3nsGpPI=', 'Content-Length': '0'} 2023-05-31T20:03:10.724 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000bb6850400d3abfeb-006477a7fe-110e-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:03:10 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:03:10.725 DEBUG:boto:b'pklxkd-14999falseobjnulltrue2023-05-31T20:02:10.394Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormalobj1c5VIADR1kuOHtAIi0CwrS4UaPxAfUIfalse2023-05-31T20:01:29.347Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:03:10.725 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2023-05-31T20:03:10.726 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2023-05-31T20:03:10.726 DEBUG:boto:path=/pklxkd-14/ 2023-05-31T20:03:10.727 DEBUG:boto:auth_path=/pklxkd-14/ 2023-05-31T20:03:10.727 DEBUG:boto:path=/pklxkd-14/?versions&max-keys=999 2023-05-31T20:03:10.727 DEBUG:boto:auth_path=/pklxkd-14/?versions&max-keys=999 2023-05-31T20:03:10.728 DEBUG:boto:Method: GET 2023-05-31T20:03:10.728 DEBUG:boto:Path: /pklxkd-14/?versions&max-keys=999 2023-05-31T20:03:10.728 DEBUG:boto:Data: 2023-05-31T20:03:10.729 DEBUG:boto:Headers: {} 2023-05-31T20:03:10.729 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:03:10.729 DEBUG:boto:Port: 8001 2023-05-31T20:03:10.730 DEBUG:boto:Params: {} 2023-05-31T20:03:10.730 DEBUG:boto:Token: None 2023-05-31T20:03:10.730 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:03:10 GMT /pklxkd-14/?versions 2023-05-31T20:03:10.731 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:uRAp/aCbxRHW/61ow6JL3nsGpPI= 2023-05-31T20:03:10.731 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:03:10 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:uRAp/aCbxRHW/61ow6JL3nsGpPI=', 'Content-Length': '0'} 2023-05-31T20:03:10.733 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000cdc7bf8832c358e0-006477a7fe-1112-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:03:10 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:03:10.734 DEBUG:boto:b'pklxkd-14999falseobjnulltrue2023-05-31T20:02:10.394Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormalobj1c5VIADR1kuOHtAIi0CwrS4UaPxAfUIfalse2023-05-31T20:01:29.347Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:03:10.735 DEBUG:boto:path=/pklxkd-14/ 2023-05-31T20:03:10.735 DEBUG:boto:auth_path=/pklxkd-14/ 2023-05-31T20:03:10.735 DEBUG:boto:path=/pklxkd-14/?versions&max-keys=999 2023-05-31T20:03:10.736 DEBUG:boto:auth_path=/pklxkd-14/?versions&max-keys=999 2023-05-31T20:03:10.736 DEBUG:boto:Method: GET 2023-05-31T20:03:10.737 DEBUG:boto:Path: /pklxkd-14/?versions&max-keys=999 2023-05-31T20:03:10.737 DEBUG:boto:Data: 2023-05-31T20:03:10.737 DEBUG:boto:Headers: {} 2023-05-31T20:03:10.738 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:03:10.738 DEBUG:boto:Port: 8000 2023-05-31T20:03:10.738 DEBUG:boto:Params: {} 2023-05-31T20:03:10.739 DEBUG:boto:Token: None 2023-05-31T20:03:10.739 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:03:10 GMT /pklxkd-14/?versions 2023-05-31T20:03:10.740 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:uRAp/aCbxRHW/61ow6JL3nsGpPI= 2023-05-31T20:03:10.740 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:03:10 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:uRAp/aCbxRHW/61ow6JL3nsGpPI=', 'Content-Length': '0'} 2023-05-31T20:03:10.742 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000fe44786f896ff753-006477a7fe-110e-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:03:10 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:03:10.743 DEBUG:boto:b'pklxkd-14999falseobjnulltrue2023-05-31T20:02:10.394Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormalobj1c5VIADR1kuOHtAIi0CwrS4UaPxAfUIfalse2023-05-31T20:01:29.347Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:03:10.743 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2023-05-31T20:03:10.744 DEBUG:boto:path=/pklxkd-14/obj 2023-05-31T20:03:10.744 DEBUG:boto:auth_path=/pklxkd-14/obj 2023-05-31T20:03:10.745 DEBUG:boto:path=/pklxkd-14/obj?versionId=null 2023-05-31T20:03:10.745 DEBUG:boto:auth_path=/pklxkd-14/obj?versionId=null 2023-05-31T20:03:10.746 DEBUG:boto:Method: GET 2023-05-31T20:03:10.746 DEBUG:boto:Path: /pklxkd-14/obj?versionId=null 2023-05-31T20:03:10.746 DEBUG:boto:Data: 2023-05-31T20:03:10.747 DEBUG:boto:Headers: {} 2023-05-31T20:03:10.747 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:03:10.748 DEBUG:boto:Port: 8001 2023-05-31T20:03:10.748 DEBUG:boto:Params: {} 2023-05-31T20:03:10.748 DEBUG:boto:Token: None 2023-05-31T20:03:10.749 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:03:10 GMT /pklxkd-14/obj?versionId=null 2023-05-31T20:03:10.749 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:dbHSRGUo4woas/kaPTCzWUVR120= 2023-05-31T20:03:10.750 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:03:10 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:dbHSRGUo4woas/kaPTCzWUVR120=', 'Content-Length': '0'} 2023-05-31T20:03:10.752 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:02:10 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx0000048a0564c8a8368b4-006477a7fe-1112-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:03:10 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:03:10.752 DEBUG:boto:path=/pklxkd-14/obj 2023-05-31T20:03:10.753 DEBUG:boto:auth_path=/pklxkd-14/obj 2023-05-31T20:03:10.753 DEBUG:boto:path=/pklxkd-14/obj?versionId=null 2023-05-31T20:03:10.754 DEBUG:boto:auth_path=/pklxkd-14/obj?versionId=null 2023-05-31T20:03:10.754 DEBUG:boto:Method: GET 2023-05-31T20:03:10.754 DEBUG:boto:Path: /pklxkd-14/obj?versionId=null 2023-05-31T20:03:10.755 DEBUG:boto:Data: 2023-05-31T20:03:10.755 DEBUG:boto:Headers: {} 2023-05-31T20:03:10.756 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:03:10.756 DEBUG:boto:Port: 8000 2023-05-31T20:03:10.756 DEBUG:boto:Params: {} 2023-05-31T20:03:10.757 DEBUG:boto:Token: None 2023-05-31T20:03:10.757 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:03:10 GMT /pklxkd-14/obj?versionId=null 2023-05-31T20:03:10.758 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:dbHSRGUo4woas/kaPTCzWUVR120= 2023-05-31T20:03:10.758 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:03:10 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:dbHSRGUo4woas/kaPTCzWUVR120=', 'Content-Length': '0'} 2023-05-31T20:03:10.760 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:02:10 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx0000048dbf2491c90b56e-006477a7fe-110e-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:03:10 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:03:10.760 DEBUG:boto:path=/pklxkd-14/obj 2023-05-31T20:03:10.761 DEBUG:boto:auth_path=/pklxkd-14/obj 2023-05-31T20:03:10.761 DEBUG:boto:path=/pklxkd-14/obj?versionId=null 2023-05-31T20:03:10.762 DEBUG:boto:auth_path=/pklxkd-14/obj?versionId=null 2023-05-31T20:03:10.762 DEBUG:boto:Method: HEAD 2023-05-31T20:03:10.762 DEBUG:boto:Path: /pklxkd-14/obj?versionId=null 2023-05-31T20:03:10.763 DEBUG:boto:Data: 2023-05-31T20:03:10.763 DEBUG:boto:Headers: {} 2023-05-31T20:03:10.764 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:03:10.764 DEBUG:boto:Port: 8001 2023-05-31T20:03:10.764 DEBUG:boto:Params: {} 2023-05-31T20:03:10.765 DEBUG:boto:Token: None 2023-05-31T20:03:10.765 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:03:10 GMT /pklxkd-14/obj?versionId=null 2023-05-31T20:03:10.766 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:e34qLzl/Lon2bE7OaAngo4PNR/I= 2023-05-31T20:03:10.766 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:03:10 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:e34qLzl/Lon2bE7OaAngo4PNR/I=', 'Content-Length': '0'} 2023-05-31T20:03:10.768 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:02:10 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000bccb89613a256987-006477a7fe-1112-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:03:10 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:03:10.769 DEBUG:boto:path=/pklxkd-14/obj 2023-05-31T20:03:10.769 DEBUG:boto:auth_path=/pklxkd-14/obj 2023-05-31T20:03:10.770 DEBUG:boto:path=/pklxkd-14/obj?versionId=null 2023-05-31T20:03:10.770 DEBUG:boto:auth_path=/pklxkd-14/obj?versionId=null 2023-05-31T20:03:10.770 DEBUG:boto:Method: HEAD 2023-05-31T20:03:10.771 DEBUG:boto:Path: /pklxkd-14/obj?versionId=null 2023-05-31T20:03:10.771 DEBUG:boto:Data: 2023-05-31T20:03:10.771 DEBUG:boto:Headers: {} 2023-05-31T20:03:10.772 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:03:10.772 DEBUG:boto:Port: 8000 2023-05-31T20:03:10.772 DEBUG:boto:Params: {} 2023-05-31T20:03:10.773 DEBUG:boto:Token: None 2023-05-31T20:03:10.773 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:03:10 GMT /pklxkd-14/obj?versionId=null 2023-05-31T20:03:10.774 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:e34qLzl/Lon2bE7OaAngo4PNR/I= 2023-05-31T20:03:10.774 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:03:10 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:e34qLzl/Lon2bE7OaAngo4PNR/I=', 'Content-Length': '0'} 2023-05-31T20:03:10.776 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:02:10 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000e451baa552261f06-006477a7fe-110e-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:03:10 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:03:10.777 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2023-05-31T20:03:10.777 DEBUG:boto:path=/pklxkd-14/obj 2023-05-31T20:03:10.778 DEBUG:boto:auth_path=/pklxkd-14/obj 2023-05-31T20:03:10.778 DEBUG:boto:path=/pklxkd-14/obj?versionId=null 2023-05-31T20:03:10.779 DEBUG:boto:auth_path=/pklxkd-14/obj?versionId=null 2023-05-31T20:03:10.780 DEBUG:boto:Method: GET 2023-05-31T20:03:10.781 DEBUG:boto:Path: /pklxkd-14/obj?versionId=null 2023-05-31T20:03:10.782 DEBUG:boto:Data: 2023-05-31T20:03:10.783 DEBUG:boto:Headers: {} 2023-05-31T20:03:10.784 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:03:10.785 DEBUG:boto:Port: 8001 2023-05-31T20:03:10.785 DEBUG:boto:Params: {} 2023-05-31T20:03:10.786 DEBUG:boto:Token: None 2023-05-31T20:03:10.787 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:03:10 GMT /pklxkd-14/obj?versionId=null 2023-05-31T20:03:10.788 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:dbHSRGUo4woas/kaPTCzWUVR120= 2023-05-31T20:03:10.789 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:03:10 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:dbHSRGUo4woas/kaPTCzWUVR120=', 'Content-Length': '0'} 2023-05-31T20:03:10.792 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:02:10 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000000cb36d59fcd2f8b1-006477a7fe-1112-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:03:10 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:03:10.793 DEBUG:boto:path=/pklxkd-14/obj 2023-05-31T20:03:10.794 DEBUG:boto:auth_path=/pklxkd-14/obj 2023-05-31T20:03:10.794 DEBUG:boto:path=/pklxkd-14/obj?versionId=null 2023-05-31T20:03:10.795 DEBUG:boto:auth_path=/pklxkd-14/obj?versionId=null 2023-05-31T20:03:10.796 DEBUG:boto:Method: GET 2023-05-31T20:03:10.797 DEBUG:boto:Path: /pklxkd-14/obj?versionId=null 2023-05-31T20:03:10.797 DEBUG:boto:Data: 2023-05-31T20:03:10.798 DEBUG:boto:Headers: {} 2023-05-31T20:03:10.799 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:03:10.799 DEBUG:boto:Port: 8000 2023-05-31T20:03:10.800 DEBUG:boto:Params: {} 2023-05-31T20:03:10.801 DEBUG:boto:Token: None 2023-05-31T20:03:10.802 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:03:10 GMT /pklxkd-14/obj?versionId=null 2023-05-31T20:03:10.803 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:dbHSRGUo4woas/kaPTCzWUVR120= 2023-05-31T20:03:10.804 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:03:10 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:dbHSRGUo4woas/kaPTCzWUVR120=', 'Content-Length': '0'} 2023-05-31T20:03:10.806 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:02:10 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx0000018698d2611bd99ac-006477a7fe-110e-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:03:10 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:03:10.807 DEBUG:boto:path=/pklxkd-14/ 2023-05-31T20:03:10.807 DEBUG:boto:auth_path=/pklxkd-14/ 2023-05-31T20:03:10.808 DEBUG:boto:path=/pklxkd-14/?prefix=obj 2023-05-31T20:03:10.808 DEBUG:boto:auth_path=/pklxkd-14/?prefix=obj 2023-05-31T20:03:10.809 DEBUG:boto:Method: GET 2023-05-31T20:03:10.809 DEBUG:boto:Path: /pklxkd-14/?prefix=obj 2023-05-31T20:03:10.810 DEBUG:boto:Data: 2023-05-31T20:03:10.810 DEBUG:boto:Headers: {} 2023-05-31T20:03:10.810 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:03:10.811 DEBUG:boto:Port: 8001 2023-05-31T20:03:10.811 DEBUG:boto:Params: {} 2023-05-31T20:03:10.811 DEBUG:boto:Token: None 2023-05-31T20:03:10.812 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:03:10 GMT /pklxkd-14/ 2023-05-31T20:03:10.812 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:KTv0ugxJeD2EOtBBRnu5inyZKDs= 2023-05-31T20:03:10.813 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:03:10 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:KTv0ugxJeD2EOtBBRnu5inyZKDs=', 'Content-Length': '0'} 2023-05-31T20:03:10.816 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000007bbe2c041580cdf9-006477a7fe-1112-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:03:10 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:03:10.817 DEBUG:boto:b'pklxkd-14obj1000falseobj2023-05-31T20:02:10.394Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:03:10.818 DEBUG:boto:path=/pklxkd-14/ 2023-05-31T20:03:10.818 DEBUG:boto:auth_path=/pklxkd-14/ 2023-05-31T20:03:10.819 DEBUG:boto:path=/pklxkd-14/?prefix=obj 2023-05-31T20:03:10.820 DEBUG:boto:auth_path=/pklxkd-14/?prefix=obj 2023-05-31T20:03:10.820 DEBUG:boto:Method: GET 2023-05-31T20:03:10.821 DEBUG:boto:Path: /pklxkd-14/?prefix=obj 2023-05-31T20:03:10.822 DEBUG:boto:Data: 2023-05-31T20:03:10.822 DEBUG:boto:Headers: {} 2023-05-31T20:03:10.823 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:03:10.824 DEBUG:boto:Port: 8000 2023-05-31T20:03:10.824 DEBUG:boto:Params: {} 2023-05-31T20:03:10.825 DEBUG:boto:Token: None 2023-05-31T20:03:10.826 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:03:10 GMT /pklxkd-14/ 2023-05-31T20:03:10.827 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:KTv0ugxJeD2EOtBBRnu5inyZKDs= 2023-05-31T20:03:10.827 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:03:10 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:KTv0ugxJeD2EOtBBRnu5inyZKDs=', 'Content-Length': '0'} 2023-05-31T20:03:10.830 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ce1e7837ac568a74-006477a7fe-110e-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:03:10 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:03:10.831 DEBUG:boto:b'pklxkd-14obj1000falseobj2023-05-31T20:02:10.394Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:03:10.833 DEBUG:boto:path=/pklxkd-14/obj 2023-05-31T20:03:10.834 DEBUG:boto:auth_path=/pklxkd-14/obj 2023-05-31T20:03:10.834 DEBUG:boto:Method: HEAD 2023-05-31T20:03:10.835 DEBUG:boto:Path: /pklxkd-14/obj 2023-05-31T20:03:10.835 DEBUG:boto:Data: 2023-05-31T20:03:10.835 DEBUG:boto:Headers: {} 2023-05-31T20:03:10.836 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:03:10.836 DEBUG:boto:Port: 8001 2023-05-31T20:03:10.836 DEBUG:boto:Params: {} 2023-05-31T20:03:10.837 DEBUG:boto:Token: None 2023-05-31T20:03:10.837 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:03:10 GMT /pklxkd-14/obj 2023-05-31T20:03:10.838 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:/GeWiZ+AKRL2mn0hZ50z6oZaM7g= 2023-05-31T20:03:10.838 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:03:10 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:/GeWiZ+AKRL2mn0hZ50z6oZaM7g=', 'Content-Length': '0'} 2023-05-31T20:03:10.840 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:02:10 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000005569dfbade18bb43-006477a7fe-1112-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:03:10 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:03:10.840 DEBUG:boto:path=/pklxkd-14/obj 2023-05-31T20:03:10.841 DEBUG:boto:auth_path=/pklxkd-14/obj 2023-05-31T20:03:10.841 DEBUG:boto:Method: HEAD 2023-05-31T20:03:10.841 DEBUG:boto:Path: /pklxkd-14/obj 2023-05-31T20:03:10.842 DEBUG:boto:Data: 2023-05-31T20:03:10.842 DEBUG:boto:Headers: {} 2023-05-31T20:03:10.842 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:03:10.843 DEBUG:boto:Port: 8000 2023-05-31T20:03:10.843 DEBUG:boto:Params: {} 2023-05-31T20:03:10.844 DEBUG:boto:Token: None 2023-05-31T20:03:10.844 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:03:10 GMT /pklxkd-14/obj 2023-05-31T20:03:10.845 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:/GeWiZ+AKRL2mn0hZ50z6oZaM7g= 2023-05-31T20:03:10.845 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:03:10 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:/GeWiZ+AKRL2mn0hZ50z6oZaM7g=', 'Content-Length': '0'} 2023-05-31T20:03:10.847 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:02:10 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000c9197481a7fd50b2-006477a7fe-110e-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:03:10 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:03:10.848 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2023-05-31T20:03:10.848 DEBUG:boto:path=/pklxkd-14/obj 2023-05-31T20:03:10.848 DEBUG:boto:auth_path=/pklxkd-14/obj 2023-05-31T20:03:10.849 DEBUG:boto:Method: GET 2023-05-31T20:03:10.849 DEBUG:boto:Path: /pklxkd-14/obj 2023-05-31T20:03:10.850 DEBUG:boto:Data: 2023-05-31T20:03:10.850 DEBUG:boto:Headers: {} 2023-05-31T20:03:10.850 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:03:10.851 DEBUG:boto:Port: 8001 2023-05-31T20:03:10.851 DEBUG:boto:Params: {} 2023-05-31T20:03:10.852 DEBUG:boto:Token: None 2023-05-31T20:03:10.852 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:03:10 GMT /pklxkd-14/obj 2023-05-31T20:03:10.853 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:x2y2dZ7Jgf+UXkh2cPkRrTD3F6I= 2023-05-31T20:03:10.853 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:03:10 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:x2y2dZ7Jgf+UXkh2cPkRrTD3F6I=', 'Content-Length': '0'} 2023-05-31T20:03:10.855 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:02:10 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000ecbc5878cfbb42f1-006477a7fe-1112-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:03:10 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:03:10.855 DEBUG:boto:path=/pklxkd-14/obj 2023-05-31T20:03:10.856 DEBUG:boto:auth_path=/pklxkd-14/obj 2023-05-31T20:03:10.856 DEBUG:boto:Method: GET 2023-05-31T20:03:10.856 DEBUG:boto:Path: /pklxkd-14/obj 2023-05-31T20:03:10.857 DEBUG:boto:Data: 2023-05-31T20:03:10.857 DEBUG:boto:Headers: {} 2023-05-31T20:03:10.857 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:03:10.858 DEBUG:boto:Port: 8000 2023-05-31T20:03:10.858 DEBUG:boto:Params: {} 2023-05-31T20:03:10.859 DEBUG:boto:Token: None 2023-05-31T20:03:10.859 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:03:10 GMT /pklxkd-14/obj 2023-05-31T20:03:10.859 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:x2y2dZ7Jgf+UXkh2cPkRrTD3F6I= 2023-05-31T20:03:10.860 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:03:10 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:x2y2dZ7Jgf+UXkh2cPkRrTD3F6I=', 'Content-Length': '0'} 2023-05-31T20:03:10.862 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:02:10 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000a33a2632f05d96b6-006477a7fe-110e-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:03:10 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:03:10.862 DEBUG:boto:path=/pklxkd-14/ 2023-05-31T20:03:10.863 DEBUG:boto:auth_path=/pklxkd-14/ 2023-05-31T20:03:10.863 DEBUG:boto:path=/pklxkd-14/?prefix=obj 2023-05-31T20:03:10.863 DEBUG:boto:auth_path=/pklxkd-14/?prefix=obj 2023-05-31T20:03:10.864 DEBUG:boto:Method: GET 2023-05-31T20:03:10.864 DEBUG:boto:Path: /pklxkd-14/?prefix=obj 2023-05-31T20:03:10.865 DEBUG:boto:Data: 2023-05-31T20:03:10.865 DEBUG:boto:Headers: {} 2023-05-31T20:03:10.865 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:03:10.866 DEBUG:boto:Port: 8001 2023-05-31T20:03:10.866 DEBUG:boto:Params: {} 2023-05-31T20:03:10.866 DEBUG:boto:Token: None 2023-05-31T20:03:10.867 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:03:10 GMT /pklxkd-14/ 2023-05-31T20:03:10.867 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:KTv0ugxJeD2EOtBBRnu5inyZKDs= 2023-05-31T20:03:10.868 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:03:10 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:KTv0ugxJeD2EOtBBRnu5inyZKDs=', 'Content-Length': '0'} 2023-05-31T20:03:10.870 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000001556f5c3aae6ae32-006477a7fe-1112-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:03:10 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:03:10.871 DEBUG:boto:b'pklxkd-14obj1000falseobj2023-05-31T20:02:10.394Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:03:10.871 DEBUG:boto:path=/pklxkd-14/ 2023-05-31T20:03:10.872 DEBUG:boto:auth_path=/pklxkd-14/ 2023-05-31T20:03:10.872 DEBUG:boto:path=/pklxkd-14/?prefix=obj 2023-05-31T20:03:10.873 DEBUG:boto:auth_path=/pklxkd-14/?prefix=obj 2023-05-31T20:03:10.873 DEBUG:boto:Method: GET 2023-05-31T20:03:10.874 DEBUG:boto:Path: /pklxkd-14/?prefix=obj 2023-05-31T20:03:10.874 DEBUG:boto:Data: 2023-05-31T20:03:10.874 DEBUG:boto:Headers: {} 2023-05-31T20:03:10.875 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:03:10.875 DEBUG:boto:Port: 8000 2023-05-31T20:03:10.875 DEBUG:boto:Params: {} 2023-05-31T20:03:10.876 DEBUG:boto:Token: None 2023-05-31T20:03:10.876 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:03:10 GMT /pklxkd-14/ 2023-05-31T20:03:10.877 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:KTv0ugxJeD2EOtBBRnu5inyZKDs= 2023-05-31T20:03:10.877 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:03:10 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:KTv0ugxJeD2EOtBBRnu5inyZKDs=', 'Content-Length': '0'} 2023-05-31T20:03:10.879 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000048289b21a3229ba-006477a7fe-110e-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:03:10 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:03:10.880 DEBUG:boto:b'pklxkd-14obj1000falseobj2023-05-31T20:02:10.394Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:03:10.880 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2023-05-31T20:03:10.881 DEBUG:boto:path=/pklxkd-14/obj 2023-05-31T20:03:10.881 DEBUG:boto:auth_path=/pklxkd-14/obj 2023-05-31T20:03:10.882 DEBUG:boto:path=/pklxkd-14/obj?versionId=1c5VIADR1kuOHtAIi0CwrS4UaPxAfUI 2023-05-31T20:03:10.882 DEBUG:boto:auth_path=/pklxkd-14/obj?versionId=1c5VIADR1kuOHtAIi0CwrS4UaPxAfUI 2023-05-31T20:03:10.882 DEBUG:boto:Method: GET 2023-05-31T20:03:10.883 DEBUG:boto:Path: /pklxkd-14/obj?versionId=1c5VIADR1kuOHtAIi0CwrS4UaPxAfUI 2023-05-31T20:03:10.883 DEBUG:boto:Data: 2023-05-31T20:03:10.883 DEBUG:boto:Headers: {} 2023-05-31T20:03:10.884 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:03:10.884 DEBUG:boto:Port: 8001 2023-05-31T20:03:10.884 DEBUG:boto:Params: {} 2023-05-31T20:03:10.885 DEBUG:boto:Token: None 2023-05-31T20:03:10.885 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:03:10 GMT /pklxkd-14/obj?versionId=1c5VIADR1kuOHtAIi0CwrS4UaPxAfUI 2023-05-31T20:03:10.886 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:TeI9UaPg5iYNVzTWMkFvbQYtRk8= 2023-05-31T20:03:10.886 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:03:10 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:TeI9UaPg5iYNVzTWMkFvbQYtRk8=', 'Content-Length': '0'} 2023-05-31T20:03:10.888 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:01:29 GMT'), ('x-amz-version-id', '1c5VIADR1kuOHtAIi0CwrS4UaPxAfUI'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000001c49ac97e09e0a9a-006477a7fe-1112-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:03:10 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:03:10.889 DEBUG:boto:path=/pklxkd-14/obj 2023-05-31T20:03:10.890 DEBUG:boto:auth_path=/pklxkd-14/obj 2023-05-31T20:03:10.890 DEBUG:boto:path=/pklxkd-14/obj?versionId=1c5VIADR1kuOHtAIi0CwrS4UaPxAfUI 2023-05-31T20:03:10.890 DEBUG:boto:auth_path=/pklxkd-14/obj?versionId=1c5VIADR1kuOHtAIi0CwrS4UaPxAfUI 2023-05-31T20:03:10.891 DEBUG:boto:Method: GET 2023-05-31T20:03:10.891 DEBUG:boto:Path: /pklxkd-14/obj?versionId=1c5VIADR1kuOHtAIi0CwrS4UaPxAfUI 2023-05-31T20:03:10.892 DEBUG:boto:Data: 2023-05-31T20:03:10.892 DEBUG:boto:Headers: {} 2023-05-31T20:03:10.892 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:03:10.893 DEBUG:boto:Port: 8000 2023-05-31T20:03:10.893 DEBUG:boto:Params: {} 2023-05-31T20:03:10.894 DEBUG:boto:Token: None 2023-05-31T20:03:10.894 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:03:10 GMT /pklxkd-14/obj?versionId=1c5VIADR1kuOHtAIi0CwrS4UaPxAfUI 2023-05-31T20:03:10.895 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:TeI9UaPg5iYNVzTWMkFvbQYtRk8= 2023-05-31T20:03:10.895 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:03:10 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:TeI9UaPg5iYNVzTWMkFvbQYtRk8=', 'Content-Length': '0'} 2023-05-31T20:03:10.897 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:01:29 GMT'), ('x-amz-version-id', '1c5VIADR1kuOHtAIi0CwrS4UaPxAfUI'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000006b133d72cc0310ab-006477a7fe-110e-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:03:10 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:03:10.898 DEBUG:boto:path=/pklxkd-14/obj 2023-05-31T20:03:10.898 DEBUG:boto:auth_path=/pklxkd-14/obj 2023-05-31T20:03:10.898 DEBUG:boto:path=/pklxkd-14/obj?versionId=1c5VIADR1kuOHtAIi0CwrS4UaPxAfUI 2023-05-31T20:03:10.899 DEBUG:boto:auth_path=/pklxkd-14/obj?versionId=1c5VIADR1kuOHtAIi0CwrS4UaPxAfUI 2023-05-31T20:03:10.899 DEBUG:boto:Method: HEAD 2023-05-31T20:03:10.899 DEBUG:boto:Path: /pklxkd-14/obj?versionId=1c5VIADR1kuOHtAIi0CwrS4UaPxAfUI 2023-05-31T20:03:10.900 DEBUG:boto:Data: 2023-05-31T20:03:10.900 DEBUG:boto:Headers: {} 2023-05-31T20:03:10.900 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:03:10.901 DEBUG:boto:Port: 8001 2023-05-31T20:03:10.901 DEBUG:boto:Params: {} 2023-05-31T20:03:10.902 DEBUG:boto:Token: None 2023-05-31T20:03:10.902 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:03:10 GMT /pklxkd-14/obj?versionId=1c5VIADR1kuOHtAIi0CwrS4UaPxAfUI 2023-05-31T20:03:10.902 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:EQ3mVNNNjbEux/6tPGpBu95Mf7E= 2023-05-31T20:03:10.903 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:03:10 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:EQ3mVNNNjbEux/6tPGpBu95Mf7E=', 'Content-Length': '0'} 2023-05-31T20:03:10.905 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:01:29 GMT'), ('x-amz-version-id', '1c5VIADR1kuOHtAIi0CwrS4UaPxAfUI'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000000be399cee77fb0ae-006477a7fe-1112-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:03:10 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:03:10.905 DEBUG:boto:path=/pklxkd-14/obj 2023-05-31T20:03:10.906 DEBUG:boto:auth_path=/pklxkd-14/obj 2023-05-31T20:03:10.906 DEBUG:boto:path=/pklxkd-14/obj?versionId=1c5VIADR1kuOHtAIi0CwrS4UaPxAfUI 2023-05-31T20:03:10.906 DEBUG:boto:auth_path=/pklxkd-14/obj?versionId=1c5VIADR1kuOHtAIi0CwrS4UaPxAfUI 2023-05-31T20:03:10.907 DEBUG:boto:Method: HEAD 2023-05-31T20:03:10.907 DEBUG:boto:Path: /pklxkd-14/obj?versionId=1c5VIADR1kuOHtAIi0CwrS4UaPxAfUI 2023-05-31T20:03:10.908 DEBUG:boto:Data: 2023-05-31T20:03:10.908 DEBUG:boto:Headers: {} 2023-05-31T20:03:10.909 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:03:10.909 DEBUG:boto:Port: 8000 2023-05-31T20:03:10.909 DEBUG:boto:Params: {} 2023-05-31T20:03:10.910 DEBUG:boto:Token: None 2023-05-31T20:03:10.910 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:03:10 GMT /pklxkd-14/obj?versionId=1c5VIADR1kuOHtAIi0CwrS4UaPxAfUI 2023-05-31T20:03:10.911 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:EQ3mVNNNjbEux/6tPGpBu95Mf7E= 2023-05-31T20:03:10.911 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:03:10 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:EQ3mVNNNjbEux/6tPGpBu95Mf7E=', 'Content-Length': '0'} 2023-05-31T20:03:10.913 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:01:29 GMT'), ('x-amz-version-id', '1c5VIADR1kuOHtAIi0CwrS4UaPxAfUI'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000d1d88e2bf95d0474-006477a7fe-110e-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:03:10 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:03:10.913 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2023-05-31T20:03:10.915 DEBUG:boto:path=/pklxkd-14/obj 2023-05-31T20:03:10.915 DEBUG:boto:auth_path=/pklxkd-14/obj 2023-05-31T20:03:10.916 DEBUG:boto:path=/pklxkd-14/obj?versionId=1c5VIADR1kuOHtAIi0CwrS4UaPxAfUI 2023-05-31T20:03:10.916 DEBUG:boto:auth_path=/pklxkd-14/obj?versionId=1c5VIADR1kuOHtAIi0CwrS4UaPxAfUI 2023-05-31T20:03:10.917 DEBUG:boto:Method: GET 2023-05-31T20:03:10.917 DEBUG:boto:Path: /pklxkd-14/obj?versionId=1c5VIADR1kuOHtAIi0CwrS4UaPxAfUI 2023-05-31T20:03:10.918 DEBUG:boto:Data: 2023-05-31T20:03:10.919 DEBUG:boto:Headers: {} 2023-05-31T20:03:10.920 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:03:10.922 DEBUG:boto:Port: 8001 2023-05-31T20:03:10.923 DEBUG:boto:Params: {} 2023-05-31T20:03:10.923 DEBUG:boto:Token: None 2023-05-31T20:03:10.924 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:03:10 GMT /pklxkd-14/obj?versionId=1c5VIADR1kuOHtAIi0CwrS4UaPxAfUI 2023-05-31T20:03:10.925 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:TeI9UaPg5iYNVzTWMkFvbQYtRk8= 2023-05-31T20:03:10.925 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:03:10 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:TeI9UaPg5iYNVzTWMkFvbQYtRk8=', 'Content-Length': '0'} 2023-05-31T20:03:10.927 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:01:29 GMT'), ('x-amz-version-id', '1c5VIADR1kuOHtAIi0CwrS4UaPxAfUI'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000d5142f6bec1d6445-006477a7fe-1112-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:03:10 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:03:10.927 DEBUG:boto:path=/pklxkd-14/obj 2023-05-31T20:03:10.928 DEBUG:boto:auth_path=/pklxkd-14/obj 2023-05-31T20:03:10.929 DEBUG:boto:path=/pklxkd-14/obj?versionId=1c5VIADR1kuOHtAIi0CwrS4UaPxAfUI 2023-05-31T20:03:10.929 DEBUG:boto:auth_path=/pklxkd-14/obj?versionId=1c5VIADR1kuOHtAIi0CwrS4UaPxAfUI 2023-05-31T20:03:10.929 DEBUG:boto:Method: GET 2023-05-31T20:03:10.930 DEBUG:boto:Path: /pklxkd-14/obj?versionId=1c5VIADR1kuOHtAIi0CwrS4UaPxAfUI 2023-05-31T20:03:10.930 DEBUG:boto:Data: 2023-05-31T20:03:10.931 DEBUG:boto:Headers: {} 2023-05-31T20:03:10.932 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:03:10.932 DEBUG:boto:Port: 8000 2023-05-31T20:03:10.933 DEBUG:boto:Params: {} 2023-05-31T20:03:10.933 DEBUG:boto:Token: None 2023-05-31T20:03:10.933 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:03:10 GMT /pklxkd-14/obj?versionId=1c5VIADR1kuOHtAIi0CwrS4UaPxAfUI 2023-05-31T20:03:10.934 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:TeI9UaPg5iYNVzTWMkFvbQYtRk8= 2023-05-31T20:03:10.934 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:03:10 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:TeI9UaPg5iYNVzTWMkFvbQYtRk8=', 'Content-Length': '0'} 2023-05-31T20:03:10.937 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:01:29 GMT'), ('x-amz-version-id', '1c5VIADR1kuOHtAIi0CwrS4UaPxAfUI'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000bf803c7cfa0844c0-006477a7fe-110e-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:03:10 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:03:10.939 DEBUG:boto:path=/pklxkd-14/ 2023-05-31T20:03:10.939 DEBUG:boto:auth_path=/pklxkd-14/ 2023-05-31T20:03:10.940 DEBUG:boto:path=/pklxkd-14/?prefix=obj 2023-05-31T20:03:10.940 DEBUG:boto:auth_path=/pklxkd-14/?prefix=obj 2023-05-31T20:03:10.940 DEBUG:boto:Method: GET 2023-05-31T20:03:10.941 DEBUG:boto:Path: /pklxkd-14/?prefix=obj 2023-05-31T20:03:10.941 DEBUG:boto:Data: 2023-05-31T20:03:10.941 DEBUG:boto:Headers: {} 2023-05-31T20:03:10.942 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:03:10.942 DEBUG:boto:Port: 8001 2023-05-31T20:03:10.943 DEBUG:boto:Params: {} 2023-05-31T20:03:10.943 DEBUG:boto:Token: None 2023-05-31T20:03:10.943 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:03:10 GMT /pklxkd-14/ 2023-05-31T20:03:10.944 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:KTv0ugxJeD2EOtBBRnu5inyZKDs= 2023-05-31T20:03:10.944 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:03:10 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:KTv0ugxJeD2EOtBBRnu5inyZKDs=', 'Content-Length': '0'} 2023-05-31T20:03:10.946 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000993e42a9a81eabdf-006477a7fe-1112-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:03:10 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:03:10.988 DEBUG:boto:b'pklxkd-14obj1000falseobj2023-05-31T20:02:10.394Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:03:10.989 DEBUG:boto:path=/pklxkd-14/ 2023-05-31T20:03:10.990 DEBUG:boto:auth_path=/pklxkd-14/ 2023-05-31T20:03:10.990 DEBUG:boto:path=/pklxkd-14/?prefix=obj 2023-05-31T20:03:10.991 DEBUG:boto:auth_path=/pklxkd-14/?prefix=obj 2023-05-31T20:03:10.991 DEBUG:boto:Method: GET 2023-05-31T20:03:10.991 DEBUG:boto:Path: /pklxkd-14/?prefix=obj 2023-05-31T20:03:10.992 DEBUG:boto:Data: 2023-05-31T20:03:10.992 DEBUG:boto:Headers: {} 2023-05-31T20:03:10.992 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:03:10.993 DEBUG:boto:Port: 8000 2023-05-31T20:03:10.993 DEBUG:boto:Params: {} 2023-05-31T20:03:10.994 DEBUG:boto:Token: None 2023-05-31T20:03:10.994 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:03:10 GMT /pklxkd-14/ 2023-05-31T20:03:10.995 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:KTv0ugxJeD2EOtBBRnu5inyZKDs= 2023-05-31T20:03:10.995 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:03:10 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:KTv0ugxJeD2EOtBBRnu5inyZKDs=', 'Content-Length': '0'} 2023-05-31T20:03:10.998 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ee49c0a91eebf268-006477a7fe-110e-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:03:10 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:03:10.998 DEBUG:boto:b'pklxkd-14obj1000falseobj2023-05-31T20:02:10.394Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:03:10.999 DEBUG:boto:path=/pklxkd-14/obj 2023-05-31T20:03:10.999 DEBUG:boto:auth_path=/pklxkd-14/obj 2023-05-31T20:03:11.000 DEBUG:boto:Method: HEAD 2023-05-31T20:03:11.000 DEBUG:boto:Path: /pklxkd-14/obj 2023-05-31T20:03:11.000 DEBUG:boto:Data: 2023-05-31T20:03:11.001 DEBUG:boto:Headers: {} 2023-05-31T20:03:11.001 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:03:11.002 DEBUG:boto:Port: 8001 2023-05-31T20:03:11.002 DEBUG:boto:Params: {} 2023-05-31T20:03:11.003 DEBUG:boto:Token: None 2023-05-31T20:03:11.003 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:03:11 GMT /pklxkd-14/obj 2023-05-31T20:03:11.004 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:mzgv/Kq+r/G7GR6n9PRUlVj2WTk= 2023-05-31T20:03:11.004 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:03:11 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:mzgv/Kq+r/G7GR6n9PRUlVj2WTk=', 'Content-Length': '0'} 2023-05-31T20:03:11.006 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:02:10 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000ff0e6adb1add0db5-006477a7fe-1112-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:03:11 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:03:11.007 DEBUG:boto:path=/pklxkd-14/obj 2023-05-31T20:03:11.007 DEBUG:boto:auth_path=/pklxkd-14/obj 2023-05-31T20:03:11.007 DEBUG:boto:Method: HEAD 2023-05-31T20:03:11.008 DEBUG:boto:Path: /pklxkd-14/obj 2023-05-31T20:03:11.008 DEBUG:boto:Data: 2023-05-31T20:03:11.008 DEBUG:boto:Headers: {} 2023-05-31T20:03:11.009 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:03:11.009 DEBUG:boto:Port: 8000 2023-05-31T20:03:11.009 DEBUG:boto:Params: {} 2023-05-31T20:03:11.010 DEBUG:boto:Token: None 2023-05-31T20:03:11.010 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:03:11 GMT /pklxkd-14/obj 2023-05-31T20:03:11.011 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:mzgv/Kq+r/G7GR6n9PRUlVj2WTk= 2023-05-31T20:03:11.011 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:03:11 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:mzgv/Kq+r/G7GR6n9PRUlVj2WTk=', 'Content-Length': '0'} 2023-05-31T20:03:11.013 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:02:10 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000ba47344dc85bd4cf-006477a7ff-110e-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:03:11 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:03:11.013 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2023-05-31T20:03:11.014 DEBUG:boto:path=/pklxkd-14/obj 2023-05-31T20:03:11.014 DEBUG:boto:auth_path=/pklxkd-14/obj 2023-05-31T20:03:11.015 DEBUG:boto:Method: GET 2023-05-31T20:03:11.015 DEBUG:boto:Path: /pklxkd-14/obj 2023-05-31T20:03:11.015 DEBUG:boto:Data: 2023-05-31T20:03:11.016 DEBUG:boto:Headers: {} 2023-05-31T20:03:11.016 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:03:11.017 DEBUG:boto:Port: 8001 2023-05-31T20:03:11.017 DEBUG:boto:Params: {} 2023-05-31T20:03:11.018 DEBUG:boto:Token: None 2023-05-31T20:03:11.018 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:03:11 GMT /pklxkd-14/obj 2023-05-31T20:03:11.019 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:PI2mOx1Nk4wZ0nL8mRZ+K8XA6cE= 2023-05-31T20:03:11.019 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:03:11 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:PI2mOx1Nk4wZ0nL8mRZ+K8XA6cE=', 'Content-Length': '0'} 2023-05-31T20:03:11.021 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:02:10 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000514360d8780a2208-006477a7ff-1112-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:03:11 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:03:11.021 DEBUG:boto:path=/pklxkd-14/obj 2023-05-31T20:03:11.022 DEBUG:boto:auth_path=/pklxkd-14/obj 2023-05-31T20:03:11.022 DEBUG:boto:Method: GET 2023-05-31T20:03:11.023 DEBUG:boto:Path: /pklxkd-14/obj 2023-05-31T20:03:11.023 DEBUG:boto:Data: 2023-05-31T20:03:11.023 DEBUG:boto:Headers: {} 2023-05-31T20:03:11.024 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:03:11.024 DEBUG:boto:Port: 8000 2023-05-31T20:03:11.024 DEBUG:boto:Params: {} 2023-05-31T20:03:11.025 DEBUG:boto:Token: None 2023-05-31T20:03:11.025 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:03:11 GMT /pklxkd-14/obj 2023-05-31T20:03:11.026 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:PI2mOx1Nk4wZ0nL8mRZ+K8XA6cE= 2023-05-31T20:03:11.026 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:03:11 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:PI2mOx1Nk4wZ0nL8mRZ+K8XA6cE=', 'Content-Length': '0'} 2023-05-31T20:03:11.028 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:02:10 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000008b528a7c02bbd64b-006477a7ff-110e-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:03:11 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:03:11.029 DEBUG:boto:path=/pklxkd-14/ 2023-05-31T20:03:11.031 DEBUG:boto:auth_path=/pklxkd-14/ 2023-05-31T20:03:11.032 DEBUG:boto:path=/pklxkd-14/?prefix=obj 2023-05-31T20:03:11.032 DEBUG:boto:auth_path=/pklxkd-14/?prefix=obj 2023-05-31T20:03:11.033 DEBUG:boto:Method: GET 2023-05-31T20:03:11.033 DEBUG:boto:Path: /pklxkd-14/?prefix=obj 2023-05-31T20:03:11.034 DEBUG:boto:Data: 2023-05-31T20:03:11.034 DEBUG:boto:Headers: {} 2023-05-31T20:03:11.034 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:03:11.035 DEBUG:boto:Port: 8001 2023-05-31T20:03:11.035 DEBUG:boto:Params: {} 2023-05-31T20:03:11.036 DEBUG:boto:Token: None 2023-05-31T20:03:11.036 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:03:11 GMT /pklxkd-14/ 2023-05-31T20:03:11.036 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:VrC6rAbqLw5gaJDEGINpVwZ9oFM= 2023-05-31T20:03:11.037 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:03:11 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:VrC6rAbqLw5gaJDEGINpVwZ9oFM=', 'Content-Length': '0'} 2023-05-31T20:03:11.039 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000000b2ba1e28e740014-006477a7ff-1112-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:03:11 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:03:11.040 DEBUG:boto:b'pklxkd-14obj1000falseobj2023-05-31T20:02:10.394Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:03:11.041 DEBUG:boto:path=/pklxkd-14/ 2023-05-31T20:03:11.041 DEBUG:boto:auth_path=/pklxkd-14/ 2023-05-31T20:03:11.041 DEBUG:boto:path=/pklxkd-14/?prefix=obj 2023-05-31T20:03:11.042 DEBUG:boto:auth_path=/pklxkd-14/?prefix=obj 2023-05-31T20:03:11.042 DEBUG:boto:Method: GET 2023-05-31T20:03:11.042 DEBUG:boto:Path: /pklxkd-14/?prefix=obj 2023-05-31T20:03:11.043 DEBUG:boto:Data: 2023-05-31T20:03:11.043 DEBUG:boto:Headers: {} 2023-05-31T20:03:11.044 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:03:11.044 DEBUG:boto:Port: 8000 2023-05-31T20:03:11.044 DEBUG:boto:Params: {} 2023-05-31T20:03:11.045 DEBUG:boto:Token: None 2023-05-31T20:03:11.045 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:03:11 GMT /pklxkd-14/ 2023-05-31T20:03:11.046 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:VrC6rAbqLw5gaJDEGINpVwZ9oFM= 2023-05-31T20:03:11.046 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:03:11 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:VrC6rAbqLw5gaJDEGINpVwZ9oFM=', 'Content-Length': '0'} 2023-05-31T20:03:11.048 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000006906f5286b784b2e-006477a7ff-110e-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:03:11 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:03:11.092 DEBUG:boto:b'pklxkd-14obj1000falseobj2023-05-31T20:02:10.394Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:03:11.094 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=pklxkd-14 zones={a2, a1} 2023-05-31T20:03:11.094 INFO:tasks.rgw_multisite_tests:rgw_multi.tests.test_version_suspended_incremental_sync ... ok 2023-05-31T20:03:11.095 INFO:rgw_multi.tests:create bucket zone=a1 name=pklxkd-15 2023-05-31T20:03:11.096 DEBUG:boto:path=/pklxkd-15/ 2023-05-31T20:03:11.096 DEBUG:boto:auth_path=/pklxkd-15/ 2023-05-31T20:03:11.097 DEBUG:boto:Method: PUT 2023-05-31T20:03:11.097 DEBUG:boto:Path: /pklxkd-15/ 2023-05-31T20:03:11.097 DEBUG:boto:Data: 2023-05-31T20:03:11.098 DEBUG:boto:Headers: {} 2023-05-31T20:03:11.098 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:03:11.098 DEBUG:boto:Port: 8000 2023-05-31T20:03:11.099 DEBUG:boto:Params: {} 2023-05-31T20:03:11.099 DEBUG:boto:Token: None 2023-05-31T20:03:11.099 DEBUG:boto:StringToSign: PUT Wed, 31 May 2023 20:03:11 GMT /pklxkd-15/ 2023-05-31T20:03:11.100 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:NKBYePWjlZsZQpcq+Yh0GvEHguQ= 2023-05-31T20:03:11.101 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:03:11 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:NKBYePWjlZsZQpcq+Yh0GvEHguQ=', 'Content-Length': '0'} 2023-05-31T20:03:11.115 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000d57fcd111ce46565-006477a7ff-110e-a1'), ('Content-Length', '0'), ('Date', 'Wed, 31 May 2023 20:03:11 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:03:11.116 INFO:rgw_multi.tests:create bucket zone=a2 name=pklxkd-16 2023-05-31T20:03:11.116 DEBUG:boto:path=/pklxkd-16/ 2023-05-31T20:03:11.117 DEBUG:boto:auth_path=/pklxkd-16/ 2023-05-31T20:03:11.117 DEBUG:boto:Method: PUT 2023-05-31T20:03:11.117 DEBUG:boto:Path: /pklxkd-16/ 2023-05-31T20:03:11.118 DEBUG:boto:Data: 2023-05-31T20:03:11.118 DEBUG:boto:Headers: {} 2023-05-31T20:03:11.119 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:03:11.119 DEBUG:boto:Port: 8001 2023-05-31T20:03:11.119 DEBUG:boto:Params: {} 2023-05-31T20:03:11.120 DEBUG:boto:Token: None 2023-05-31T20:03:11.120 DEBUG:boto:StringToSign: PUT Wed, 31 May 2023 20:03:11 GMT /pklxkd-16/ 2023-05-31T20:03:11.121 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:bYQ8UoxywBvPQxaibzeRYnEJ2/o= 2023-05-31T20:03:11.121 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:03:11 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:bYQ8UoxywBvPQxaibzeRYnEJ2/o=', 'Content-Length': '0'} 2023-05-31T20:03:11.146 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx0000085f0db84f7f6d860-006477a7ff-1112-a2'), ('Content-Length', '0'), ('Date', 'Wed, 31 May 2023 20:03:11 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:03:11.147 DEBUG:boto:path=/pklxkd-15/ 2023-05-31T20:03:11.147 DEBUG:boto:auth_path=/pklxkd-15/ 2023-05-31T20:03:11.148 DEBUG:boto:path=/pklxkd-15/?versioning 2023-05-31T20:03:11.148 DEBUG:boto:auth_path=/pklxkd-15/?versioning 2023-05-31T20:03:11.148 DEBUG:boto:Method: PUT 2023-05-31T20:03:11.149 DEBUG:boto:Path: /pklxkd-15/?versioning 2023-05-31T20:03:11.149 DEBUG:boto:Data: Enabled Disabled 2023-05-31T20:03:11.149 DEBUG:boto:Headers: {} 2023-05-31T20:03:11.150 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:03:11.150 DEBUG:boto:Port: 8000 2023-05-31T20:03:11.150 DEBUG:boto:Params: {} 2023-05-31T20:03:11.151 DEBUG:boto:Token: None 2023-05-31T20:03:11.151 DEBUG:boto:StringToSign: PUT Wed, 31 May 2023 20:03:11 GMT /pklxkd-15/?versioning 2023-05-31T20:03:11.152 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:lx6qfKUXUXr4zfLPSLcJq83Gc3M= 2023-05-31T20:03:11.152 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:03:11 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:lx6qfKUXUXr4zfLPSLcJq83Gc3M=', 'Content-Length': '228'} 2023-05-31T20:03:11.157 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000ca43ef6bbb2a9788-006477a7ff-110e-a1'), ('Content-Type', 'application/xml'), ('Content-Length', '0'), ('Date', 'Wed, 31 May 2023 20:03:11 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:03:11.158 DEBUG:boto:path=/pklxkd-16/ 2023-05-31T20:03:11.158 DEBUG:boto:auth_path=/pklxkd-16/ 2023-05-31T20:03:11.158 DEBUG:boto:path=/pklxkd-16/?versioning 2023-05-31T20:03:11.159 DEBUG:boto:auth_path=/pklxkd-16/?versioning 2023-05-31T20:03:11.159 DEBUG:boto:Method: PUT 2023-05-31T20:03:11.160 DEBUG:boto:Path: /pklxkd-16/?versioning 2023-05-31T20:03:11.160 DEBUG:boto:Data: Enabled Disabled 2023-05-31T20:03:11.160 DEBUG:boto:Headers: {} 2023-05-31T20:03:11.161 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:03:11.161 DEBUG:boto:Port: 8001 2023-05-31T20:03:11.161 DEBUG:boto:Params: {} 2023-05-31T20:03:11.162 DEBUG:boto:Token: None 2023-05-31T20:03:11.162 DEBUG:boto:StringToSign: PUT Wed, 31 May 2023 20:03:11 GMT /pklxkd-16/?versioning 2023-05-31T20:03:11.163 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:HrJ/lAEcxncFk0/9hGus5J6y5NM= 2023-05-31T20:03:11.163 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:03:11 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:HrJ/lAEcxncFk0/9hGus5J6y5NM=', 'Content-Length': '228'} 2023-05-31T20:03:11.172 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx0000070585039aa965899-006477a7ff-1112-a2'), ('Content-Type', 'application/xml'), ('Content-Length', '0'), ('Date', 'Wed, 31 May 2023 20:03:11 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:03:11.173 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-31T20:03:11.174 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-31T20:03:11.174 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-05-31T20:03:11.225 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2023-05-31T20:03:11.225 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2023-05-31T20:03:11.272 INFO:teuthology.orchestra.run.smithi099.stderr:No --period given, using current period=2e6123d2-d695-4350-9a01-61a455bc782b 2023-05-31T20:03:11.310 INFO:teuthology.orchestra.run.smithi099.stdout:[ 2023-05-31T20:03:11.310 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T20:03:11.310 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1685562631.516089_59.1", 2023-05-31T20:03:11.311 INFO:teuthology.orchestra.run.smithi099.stdout: "last_update": "2023-05-31T19:50:31.516089Z" 2023-05-31T20:03:11.311 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:03:11.311 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T20:03:11.311 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "", 2023-05-31T20:03:11.311 INFO:teuthology.orchestra.run.smithi099.stdout: "last_update": "0.000000" 2023-05-31T20:03:11.311 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:03:11.312 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T20:03:11.312 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1685562970.274572_93.1", 2023-05-31T20:03:11.312 INFO:teuthology.orchestra.run.smithi099.stdout: "last_update": "2023-05-31T19:56:10.274572Z" 2023-05-31T20:03:11.312 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:03:11.312 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T20:03:11.312 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1685563391.165849_88.1", 2023-05-31T20:03:11.312 INFO:teuthology.orchestra.run.smithi099.stdout: "last_update": "2023-05-31T20:03:11.165849Z" 2023-05-31T20:03:11.313 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T20:03:11.313 INFO:teuthology.orchestra.run.smithi099.stdout:] 2023-05-31T20:03:11.314 DEBUG:rgw_multi.tests:master meta markers={0: '1_1685562631.516089_59.1', 1: '', 2: '1_1685562970.274572_93.1', 3: '1_1685563391.165849_88.1'} 2023-05-31T20:03:11.314 INFO:rgw_multi.tests:starting meta checkpoint for zone=a2 2023-05-31T20:03:11.315 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-31T20:03:11.315 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-31T20:03:11.316 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-05-31T20:03:11.367 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2023-05-31T20:03:11.368 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2023-05-31T20:03:11.397 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:03:11.391+0000 7ff0b0da5780 20 RGW-SYNC:meta: read sync status 2023-05-31T20:03:11.397 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:03:11.392+0000 7ff0b0da5780 20 RGW-SYNC:meta: read sync status 2023-05-31T20:03:11.409 INFO:teuthology.orchestra.run.smithi099.stdout:{ 2023-05-31T20:03:11.410 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_status": { 2023-05-31T20:03:11.410 INFO:teuthology.orchestra.run.smithi099.stdout: "info": { 2023-05-31T20:03:11.410 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "sync", 2023-05-31T20:03:11.410 INFO:teuthology.orchestra.run.smithi099.stdout: "num_shards": 4, 2023-05-31T20:03:11.410 INFO:teuthology.orchestra.run.smithi099.stdout: "period": "2e6123d2-d695-4350-9a01-61a455bc782b", 2023-05-31T20:03:11.410 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2023-05-31T20:03:11.411 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:03:11.411 INFO:teuthology.orchestra.run.smithi099.stdout: "markers": [ 2023-05-31T20:03:11.411 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T20:03:11.411 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 0, 2023-05-31T20:03:11.411 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T20:03:11.411 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2023-05-31T20:03:11.412 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1685562631.516089_59.1", 2023-05-31T20:03:11.412 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T20:03:11.412 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 1, 2023-05-31T20:03:11.412 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T20:03:11.412 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2023-05-31T19:50:31.516089Z", 2023-05-31T20:03:11.412 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2023-05-31T20:03:11.413 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T20:03:11.413 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:03:11.413 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T20:03:11.413 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 1, 2023-05-31T20:03:11.413 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T20:03:11.413 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2023-05-31T20:03:11.414 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "", 2023-05-31T20:03:11.414 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T20:03:11.414 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2023-05-31T20:03:11.414 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T20:03:11.414 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "0.000000", 2023-05-31T20:03:11.414 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2023-05-31T20:03:11.414 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T20:03:11.415 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:03:11.415 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T20:03:11.415 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 2, 2023-05-31T20:03:11.415 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T20:03:11.415 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2023-05-31T20:03:11.415 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1685562970.274572_93.1", 2023-05-31T20:03:11.416 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T20:03:11.416 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2023-05-31T20:03:11.416 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T20:03:11.416 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2023-05-31T19:56:10.274572Z", 2023-05-31T20:03:11.416 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2023-05-31T20:03:11.416 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T20:03:11.416 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:03:11.417 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T20:03:11.417 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 3, 2023-05-31T20:03:11.417 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T20:03:11.417 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2023-05-31T20:03:11.417 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1685563391.103136_77.1", 2023-05-31T20:03:11.417 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T20:03:11.417 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2023-05-31T20:03:11.418 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T20:03:11.418 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2023-05-31T20:03:11.103136Z", 2023-05-31T20:03:11.418 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2023-05-31T20:03:11.418 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T20:03:11.418 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T20:03:11.418 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2023-05-31T20:03:11.418 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:03:11.419 INFO:teuthology.orchestra.run.smithi099.stdout: "full_sync": { 2023-05-31T20:03:11.419 INFO:teuthology.orchestra.run.smithi099.stdout: "total": 1, 2023-05-31T20:03:11.419 INFO:teuthology.orchestra.run.smithi099.stdout: "complete": 1 2023-05-31T20:03:11.419 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:03:11.420 INFO:teuthology.orchestra.run.smithi099.stdout: "current_time": "2023-05-31T20:03:11Z" 2023-05-31T20:03:11.420 INFO:teuthology.orchestra.run.smithi099.stdout:} 2023-05-31T20:03:11.421 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "2e6123d2-d695-4350-9a01-61a455bc782b", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1685562631.516089_59.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-05-31T19:50:31.516089Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1685562970.274572_93.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-05-31T19:56:10.274572Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1685563391.103136_77.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-05-31T20:03:11.103136Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-05-31T20:03:11Z" } 2023-05-31T20:03:11.421 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1685562631.516089_59.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-05-31T19:50:31.516089Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1685562970.274572_93.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-05-31T19:56:10.274572Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1685563391.103136_77.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-05-31T20:03:11.103136Z', 'realm_epoch': 2}}] 2023-05-31T20:03:11.422 DEBUG:rgw_multi.tests:log_status={0: '1_1685562631.516089_59.1', 1: '', 2: '1_1685562970.274572_93.1', 3: '1_1685563391.165849_88.1'} 2023-05-31T20:03:11.422 DEBUG:rgw_multi.tests:sync_status={0: '1_1685562631.516089_59.1', 1: '', 2: '1_1685562970.274572_93.1', 3: '1_1685563391.103136_77.1'} 2023-05-31T20:03:11.422 WARNING:rgw_multi.tests:zone a2 behind master: shard=3 master=1_1685563391.165849_88.1 target=1_1685563391.103136_77.1 2023-05-31T20:03:16.424 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-31T20:03:16.425 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-31T20:03:16.426 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-05-31T20:03:16.483 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2023-05-31T20:03:16.484 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2023-05-31T20:03:16.528 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:03:16.523+0000 7f906b63e780 20 RGW-SYNC:meta: read sync status 2023-05-31T20:03:16.529 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:03:16.525+0000 7f906b63e780 20 RGW-SYNC:meta: read sync status 2023-05-31T20:03:16.550 INFO:teuthology.orchestra.run.smithi099.stdout:{ 2023-05-31T20:03:16.551 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_status": { 2023-05-31T20:03:16.551 INFO:teuthology.orchestra.run.smithi099.stdout: "info": { 2023-05-31T20:03:16.551 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "sync", 2023-05-31T20:03:16.551 INFO:teuthology.orchestra.run.smithi099.stdout: "num_shards": 4, 2023-05-31T20:03:16.551 INFO:teuthology.orchestra.run.smithi099.stdout: "period": "2e6123d2-d695-4350-9a01-61a455bc782b", 2023-05-31T20:03:16.552 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2023-05-31T20:03:16.552 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:03:16.552 INFO:teuthology.orchestra.run.smithi099.stdout: "markers": [ 2023-05-31T20:03:16.552 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T20:03:16.552 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 0, 2023-05-31T20:03:16.553 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T20:03:16.553 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2023-05-31T20:03:16.553 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1685562631.516089_59.1", 2023-05-31T20:03:16.553 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T20:03:16.553 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 1, 2023-05-31T20:03:16.553 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T20:03:16.553 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2023-05-31T19:50:31.516089Z", 2023-05-31T20:03:16.554 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2023-05-31T20:03:16.554 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T20:03:16.554 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:03:16.554 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T20:03:16.554 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 1, 2023-05-31T20:03:16.554 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T20:03:16.555 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2023-05-31T20:03:16.555 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "", 2023-05-31T20:03:16.555 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T20:03:16.555 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2023-05-31T20:03:16.555 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T20:03:16.555 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "0.000000", 2023-05-31T20:03:16.555 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2023-05-31T20:03:16.556 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T20:03:16.556 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:03:16.556 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T20:03:16.556 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 2, 2023-05-31T20:03:16.556 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T20:03:16.556 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2023-05-31T20:03:16.557 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1685562970.274572_93.1", 2023-05-31T20:03:16.557 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T20:03:16.557 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2023-05-31T20:03:16.557 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T20:03:16.557 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2023-05-31T19:56:10.274572Z", 2023-05-31T20:03:16.557 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2023-05-31T20:03:16.558 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T20:03:16.558 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:03:16.558 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T20:03:16.558 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 3, 2023-05-31T20:03:16.558 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T20:03:16.558 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2023-05-31T20:03:16.558 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1685563391.165849_88.1", 2023-05-31T20:03:16.559 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T20:03:16.559 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2023-05-31T20:03:16.559 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T20:03:16.559 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2023-05-31T20:03:11.165849Z", 2023-05-31T20:03:16.559 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2023-05-31T20:03:16.560 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T20:03:16.560 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T20:03:16.560 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2023-05-31T20:03:16.560 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:03:16.560 INFO:teuthology.orchestra.run.smithi099.stdout: "full_sync": { 2023-05-31T20:03:16.560 INFO:teuthology.orchestra.run.smithi099.stdout: "total": 1, 2023-05-31T20:03:16.561 INFO:teuthology.orchestra.run.smithi099.stdout: "complete": 1 2023-05-31T20:03:16.561 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:03:16.561 INFO:teuthology.orchestra.run.smithi099.stdout: "current_time": "2023-05-31T20:03:16Z" 2023-05-31T20:03:16.561 INFO:teuthology.orchestra.run.smithi099.stdout:} 2023-05-31T20:03:16.562 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "2e6123d2-d695-4350-9a01-61a455bc782b", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1685562631.516089_59.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-05-31T19:50:31.516089Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1685562970.274572_93.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-05-31T19:56:10.274572Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1685563391.165849_88.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-05-31T20:03:11.165849Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-05-31T20:03:16Z" } 2023-05-31T20:03:16.563 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1685562631.516089_59.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-05-31T19:50:31.516089Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1685562970.274572_93.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-05-31T19:56:10.274572Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1685563391.165849_88.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-05-31T20:03:11.165849Z', 'realm_epoch': 2}}] 2023-05-31T20:03:16.563 DEBUG:rgw_multi.tests:log_status={0: '1_1685562631.516089_59.1', 1: '', 2: '1_1685562970.274572_93.1', 3: '1_1685563391.165849_88.1'} 2023-05-31T20:03:16.563 DEBUG:rgw_multi.tests:sync_status={0: '1_1685562631.516089_59.1', 1: '', 2: '1_1685562970.274572_93.1', 3: '1_1685563391.165849_88.1'} 2023-05-31T20:03:16.564 INFO:rgw_multi.tests:finish meta checkpoint for zone=a2 2023-05-31T20:03:16.564 DEBUG:boto:path=/pklxkd-15/ 2023-05-31T20:03:16.565 DEBUG:boto:auth_path=/pklxkd-15/ 2023-05-31T20:03:16.565 DEBUG:boto:Method: HEAD 2023-05-31T20:03:16.566 DEBUG:boto:Path: /pklxkd-15/ 2023-05-31T20:03:16.566 DEBUG:boto:Data: 2023-05-31T20:03:16.566 DEBUG:boto:Headers: {} 2023-05-31T20:03:16.567 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:03:16.567 DEBUG:boto:Port: 8000 2023-05-31T20:03:16.567 DEBUG:boto:Params: {} 2023-05-31T20:03:16.568 DEBUG:boto:Token: None 2023-05-31T20:03:16.568 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:03:16 GMT /pklxkd-15/ 2023-05-31T20:03:16.569 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:xR1JdmNj0F21b4bxQD36HKltqgs= 2023-05-31T20:03:16.569 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:03:16 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:xR1JdmNj0F21b4bxQD36HKltqgs=', 'Content-Length': '0'} 2023-05-31T20:03:16.571 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000f87b6d357c93c2fa-006477a804-110e-a1'), ('Content-Length', '0'), ('Date', 'Wed, 31 May 2023 20:03:16 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:03:16.572 DEBUG:boto:path=/pklxkd-15/obj 2023-05-31T20:03:16.572 DEBUG:boto:auth_path=/pklxkd-15/obj 2023-05-31T20:03:16.573 DEBUG:boto:Method: PUT 2023-05-31T20:03:16.573 DEBUG:boto:Path: /pklxkd-15/obj 2023-05-31T20:03:16.574 DEBUG:boto:Data: 2023-05-31T20:03:16.574 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': '1B2M2Y8AsgTpgAmY7PhCfg==', 'Content-Length': '0', 'Expect': '100-Continue'} 2023-05-31T20:03:16.575 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:03:16.576 DEBUG:boto:Port: 8000 2023-05-31T20:03:16.577 DEBUG:boto:Params: {} 2023-05-31T20:03:16.578 DEBUG:boto:Token: None 2023-05-31T20:03:16.578 DEBUG:boto:StringToSign: PUT 1B2M2Y8AsgTpgAmY7PhCfg== application/octet-stream Wed, 31 May 2023 20:03:16 GMT /pklxkd-15/obj 2023-05-31T20:03:16.579 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:tuckVi0xsEiHOcLKbgjq/8MgFz0= 2023-05-31T20:03:16.580 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': '1B2M2Y8AsgTpgAmY7PhCfg==', 'Content-Length': '0', 'Expect': '100-Continue', 'Date': 'Wed, 31 May 2023 20:03:16 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:tuckVi0xsEiHOcLKbgjq/8MgFz0='} 2023-05-31T20:03:16.592 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('Accept-Ranges', 'bytes'), ('x-amz-version-id', 'f4Oe4Q1FAhoCXvv5PYPL-KsctMNpUQb'), ('x-amz-request-id', 'tx000005bba818774332c5a-006477a804-110e-a1'), ('Date', 'Wed, 31 May 2023 20:03:16 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:03:16.592 DEBUG:boto:path=/pklxkd-15/ 2023-05-31T20:03:16.593 DEBUG:boto:auth_path=/pklxkd-15/ 2023-05-31T20:03:16.593 DEBUG:boto:Method: HEAD 2023-05-31T20:03:16.594 DEBUG:boto:Path: /pklxkd-15/ 2023-05-31T20:03:16.594 DEBUG:boto:Data: 2023-05-31T20:03:16.594 DEBUG:boto:Headers: {} 2023-05-31T20:03:16.595 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:03:16.595 DEBUG:boto:Port: 8000 2023-05-31T20:03:16.596 DEBUG:boto:Params: {} 2023-05-31T20:03:16.596 DEBUG:boto:Token: None 2023-05-31T20:03:16.597 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:03:16 GMT /pklxkd-15/ 2023-05-31T20:03:16.597 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:xR1JdmNj0F21b4bxQD36HKltqgs= 2023-05-31T20:03:16.597 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:03:16 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:xR1JdmNj0F21b4bxQD36HKltqgs=', 'Content-Length': '0'} 2023-05-31T20:03:16.599 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000147b5c1e0bcc5154-006477a804-110e-a1'), ('Content-Length', '0'), ('Date', 'Wed, 31 May 2023 20:03:16 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:03:16.600 DEBUG:boto:path=/pklxkd-15/obj 2023-05-31T20:03:16.600 DEBUG:boto:auth_path=/pklxkd-15/obj 2023-05-31T20:03:16.601 DEBUG:boto:Method: DELETE 2023-05-31T20:03:16.601 DEBUG:boto:Path: /pklxkd-15/obj 2023-05-31T20:03:16.601 DEBUG:boto:Data: 2023-05-31T20:03:16.602 DEBUG:boto:Headers: {} 2023-05-31T20:03:16.602 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:03:16.603 DEBUG:boto:Port: 8000 2023-05-31T20:03:16.603 DEBUG:boto:Params: {} 2023-05-31T20:03:16.603 DEBUG:boto:Token: None 2023-05-31T20:03:16.604 DEBUG:boto:StringToSign: DELETE Wed, 31 May 2023 20:03:16 GMT /pklxkd-15/obj 2023-05-31T20:03:16.604 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:ZklO0QYHOS6LUITP8NZE+ecZf2E= 2023-05-31T20:03:16.605 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:03:16 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:ZklO0QYHOS6LUITP8NZE+ecZf2E=', 'Content-Length': '0'} 2023-05-31T20:03:16.611 DEBUG:boto:Response headers: [('x-amz-version-id', 'DbIMGNowgdjqHv-6ZEpJLD71gcijEAq'), ('x-amz-delete-marker', 'true'), ('x-amz-request-id', 'tx000005f980da4f0f0c958-006477a804-110e-a1'), ('Date', 'Wed, 31 May 2023 20:03:16 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:03:16.612 DEBUG:boto:path=/pklxkd-16/ 2023-05-31T20:03:16.612 DEBUG:boto:auth_path=/pklxkd-16/ 2023-05-31T20:03:16.613 DEBUG:boto:Method: HEAD 2023-05-31T20:03:16.613 DEBUG:boto:Path: /pklxkd-16/ 2023-05-31T20:03:16.613 DEBUG:boto:Data: 2023-05-31T20:03:16.614 DEBUG:boto:Headers: {} 2023-05-31T20:03:16.614 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:03:16.614 DEBUG:boto:Port: 8001 2023-05-31T20:03:16.615 DEBUG:boto:Params: {} 2023-05-31T20:03:16.615 DEBUG:boto:Token: None 2023-05-31T20:03:16.615 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:03:16 GMT /pklxkd-16/ 2023-05-31T20:03:16.616 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:c4FrO3Vq2XIcwbZEoCe+rxPFcYM= 2023-05-31T20:03:16.616 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:03:16 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:c4FrO3Vq2XIcwbZEoCe+rxPFcYM=', 'Content-Length': '0'} 2023-05-31T20:03:16.618 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000a85bd52928a252bb-006477a804-1112-a2'), ('Content-Length', '0'), ('Date', 'Wed, 31 May 2023 20:03:16 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:03:16.619 DEBUG:boto:path=/pklxkd-16/obj 2023-05-31T20:03:16.619 DEBUG:boto:auth_path=/pklxkd-16/obj 2023-05-31T20:03:16.620 DEBUG:boto:Method: PUT 2023-05-31T20:03:16.620 DEBUG:boto:Path: /pklxkd-16/obj 2023-05-31T20:03:16.620 DEBUG:boto:Data: 2023-05-31T20:03:16.621 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': '1B2M2Y8AsgTpgAmY7PhCfg==', 'Content-Length': '0', 'Expect': '100-Continue'} 2023-05-31T20:03:16.621 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:03:16.621 DEBUG:boto:Port: 8001 2023-05-31T20:03:16.622 DEBUG:boto:Params: {} 2023-05-31T20:03:16.622 DEBUG:boto:Token: None 2023-05-31T20:03:16.623 DEBUG:boto:StringToSign: PUT 1B2M2Y8AsgTpgAmY7PhCfg== application/octet-stream Wed, 31 May 2023 20:03:16 GMT /pklxkd-16/obj 2023-05-31T20:03:16.623 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:ZfU0nImhwhkanjylamw9KDdCAqo= 2023-05-31T20:03:16.624 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': '1B2M2Y8AsgTpgAmY7PhCfg==', 'Content-Length': '0', 'Expect': '100-Continue', 'Date': 'Wed, 31 May 2023 20:03:16 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:ZfU0nImhwhkanjylamw9KDdCAqo='} 2023-05-31T20:03:16.633 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('Accept-Ranges', 'bytes'), ('x-amz-version-id', 'Ywu847uXzkc4Jr7tsfrFKB5zTC.q-ji'), ('x-amz-request-id', 'tx00000cae32d7465f121c6-006477a804-1112-a2'), ('Date', 'Wed, 31 May 2023 20:03:16 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:03:16.634 DEBUG:boto:path=/pklxkd-16/ 2023-05-31T20:03:16.634 DEBUG:boto:auth_path=/pklxkd-16/ 2023-05-31T20:03:16.635 DEBUG:boto:Method: HEAD 2023-05-31T20:03:16.635 DEBUG:boto:Path: /pklxkd-16/ 2023-05-31T20:03:16.635 DEBUG:boto:Data: 2023-05-31T20:03:16.636 DEBUG:boto:Headers: {} 2023-05-31T20:03:16.636 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:03:16.636 DEBUG:boto:Port: 8001 2023-05-31T20:03:16.637 DEBUG:boto:Params: {} 2023-05-31T20:03:16.637 DEBUG:boto:Token: None 2023-05-31T20:03:16.637 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:03:16 GMT /pklxkd-16/ 2023-05-31T20:03:16.638 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:c4FrO3Vq2XIcwbZEoCe+rxPFcYM= 2023-05-31T20:03:16.638 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:03:16 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:c4FrO3Vq2XIcwbZEoCe+rxPFcYM=', 'Content-Length': '0'} 2023-05-31T20:03:16.640 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000df4fa347cd96805c-006477a804-1112-a2'), ('Content-Length', '0'), ('Date', 'Wed, 31 May 2023 20:03:16 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:03:16.640 DEBUG:boto:path=/pklxkd-16/obj 2023-05-31T20:03:16.641 DEBUG:boto:auth_path=/pklxkd-16/obj 2023-05-31T20:03:16.641 DEBUG:boto:Method: DELETE 2023-05-31T20:03:16.641 DEBUG:boto:Path: /pklxkd-16/obj 2023-05-31T20:03:16.642 DEBUG:boto:Data: 2023-05-31T20:03:16.642 DEBUG:boto:Headers: {} 2023-05-31T20:03:16.642 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:03:16.643 DEBUG:boto:Port: 8001 2023-05-31T20:03:16.643 DEBUG:boto:Params: {} 2023-05-31T20:03:16.644 DEBUG:boto:Token: None 2023-05-31T20:03:16.644 DEBUG:boto:StringToSign: DELETE Wed, 31 May 2023 20:03:16 GMT /pklxkd-16/obj 2023-05-31T20:03:16.645 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:OZF9bgXrn+3JKOTnxPe7j9/ucwc= 2023-05-31T20:03:16.646 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:03:16 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:OZF9bgXrn+3JKOTnxPe7j9/ucwc=', 'Content-Length': '0'} 2023-05-31T20:03:16.652 DEBUG:boto:Response headers: [('x-amz-version-id', 'CDaD4VG7xbuJHlHwO15k4ncmEwzgWUp'), ('x-amz-delete-marker', 'true'), ('x-amz-request-id', 'tx000009c1f8f4f4777872a-006477a804-1112-a2'), ('Date', 'Wed, 31 May 2023 20:03:16 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:03:16.652 DEBUG:rgw_multi.tests:bucket checkpoint: source=a1 target=a2 bucket=pklxkd-15 2023-05-31T20:03:16.653 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'pklxkd-15', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-05-31T20:03:16.654 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'pklxkd-15', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-05-31T20:03:16.655 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket pklxkd-15 --source-zone a1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2023-05-31T20:03:16.711 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2023-05-31T20:03:16.712 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2023-05-31T20:03:16.752 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:03:16.747+0000 7ff471d3d780 1 waiting to reach incremental sync.. 2023-05-31T20:03:21.752 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:03:21.748+0000 7ff471d3d780 1 waiting to reach incremental sync.. 2023-05-31T20:03:26.753 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:03:26.748+0000 7ff471d3d780 1 waiting to reach incremental sync.. 2023-05-31T20:03:31.755 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:03:31.750+0000 7ff471d3d780 1 bucket sync caught up with source: 2023-05-31T20:03:31.755 INFO:teuthology.orchestra.run.smithi099.stderr: local status: [, , , , , , , , 00000000003.8.12, , ] 2023-05-31T20:03:31.756 INFO:teuthology.orchestra.run.smithi099.stderr: remote markers: [, , , , , , , , 00000000003.8.12, , ] 2023-05-31T20:03:31.756 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:03:31.750+0000 7ff471d3d780 0 bucket checkpoint complete 2023-05-31T20:03:31.771 DEBUG:rgw_multi.tests:bucket checkpoint: source=a2 target=a1 bucket=pklxkd-15 2023-05-31T20:03:31.772 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'pklxkd-15', '--source-zone', 'a2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-05-31T20:03:31.772 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'pklxkd-15', '--source-zone', 'a2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-05-31T20:03:31.773 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket pklxkd-15 --source-zone a2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2023-05-31T20:03:31.834 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2023-05-31T20:03:31.835 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2023-05-31T20:03:31.873 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:03:31.868+0000 7f0f8005b780 1 waiting to reach incremental sync.. 2023-05-31T20:03:36.874 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:03:36.868+0000 7f0f8005b780 1 waiting to reach incremental sync.. 2023-05-31T20:03:41.874 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:03:41.868+0000 7f0f8005b780 1 waiting to reach incremental sync.. 2023-05-31T20:03:46.876 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:03:46.870+0000 7f0f8005b780 1 bucket sync caught up with source: 2023-05-31T20:03:46.876 INFO:teuthology.orchestra.run.smithi099.stderr: local status: [, , , , , , , , 00000000002.9.6, , ] 2023-05-31T20:03:46.876 INFO:teuthology.orchestra.run.smithi099.stderr: remote markers: [, , , , , , , , 00000000002.9.6, , ] 2023-05-31T20:03:46.876 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:03:46.870+0000 7f0f8005b780 0 bucket checkpoint complete 2023-05-31T20:03:46.890 INFO:tasks.rgw_multi.zone_rados:comparing bucket=pklxkd-15 zones={a2, a1} 2023-05-31T20:03:46.891 DEBUG:boto:path=/pklxkd-15/ 2023-05-31T20:03:46.891 DEBUG:boto:auth_path=/pklxkd-15/ 2023-05-31T20:03:46.892 DEBUG:boto:Method: HEAD 2023-05-31T20:03:46.892 DEBUG:boto:Path: /pklxkd-15/ 2023-05-31T20:03:46.893 DEBUG:boto:Data: 2023-05-31T20:03:46.893 DEBUG:boto:Headers: {} 2023-05-31T20:03:46.893 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:03:46.894 DEBUG:boto:Port: 8001 2023-05-31T20:03:46.894 DEBUG:boto:Params: {} 2023-05-31T20:03:46.895 DEBUG:boto:Token: None 2023-05-31T20:03:46.895 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:03:46 GMT /pklxkd-15/ 2023-05-31T20:03:46.895 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:2KqfWNhCePwMDfmGsR6PrjBIbMM= 2023-05-31T20:03:46.896 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:03:46 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:2KqfWNhCePwMDfmGsR6PrjBIbMM=', 'Content-Length': '0'} 2023-05-31T20:03:46.898 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000000d1df1699d311419-006477a822-1112-a2'), ('Content-Length', '0'), ('Date', 'Wed, 31 May 2023 20:03:46 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:03:46.899 DEBUG:boto:path=/pklxkd-15/ 2023-05-31T20:03:46.899 DEBUG:boto:auth_path=/pklxkd-15/ 2023-05-31T20:03:46.900 DEBUG:boto:Method: HEAD 2023-05-31T20:03:46.900 DEBUG:boto:Path: /pklxkd-15/ 2023-05-31T20:03:46.900 DEBUG:boto:Data: 2023-05-31T20:03:46.901 DEBUG:boto:Headers: {} 2023-05-31T20:03:46.901 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:03:46.901 DEBUG:boto:Port: 8000 2023-05-31T20:03:46.902 DEBUG:boto:Params: {} 2023-05-31T20:03:46.902 DEBUG:boto:Token: None 2023-05-31T20:03:46.902 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:03:46 GMT /pklxkd-15/ 2023-05-31T20:03:46.903 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:2KqfWNhCePwMDfmGsR6PrjBIbMM= 2023-05-31T20:03:46.903 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:03:46 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:2KqfWNhCePwMDfmGsR6PrjBIbMM=', 'Content-Length': '0'} 2023-05-31T20:03:46.906 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000bd672c2b8b701830-006477a822-110e-a1'), ('Content-Length', '0'), ('Date', 'Wed, 31 May 2023 20:03:46 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:03:46.906 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2023-05-31T20:03:46.907 DEBUG:boto:path=/pklxkd-15/ 2023-05-31T20:03:46.907 DEBUG:boto:auth_path=/pklxkd-15/ 2023-05-31T20:03:46.908 DEBUG:boto:path=/pklxkd-15/?versions&max-keys=999 2023-05-31T20:03:46.908 DEBUG:boto:auth_path=/pklxkd-15/?versions&max-keys=999 2023-05-31T20:03:46.909 DEBUG:boto:Method: GET 2023-05-31T20:03:46.909 DEBUG:boto:Path: /pklxkd-15/?versions&max-keys=999 2023-05-31T20:03:46.909 DEBUG:boto:Data: 2023-05-31T20:03:46.910 DEBUG:boto:Headers: {} 2023-05-31T20:03:46.910 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:03:46.910 DEBUG:boto:Port: 8001 2023-05-31T20:03:46.911 DEBUG:boto:Params: {} 2023-05-31T20:03:46.911 DEBUG:boto:Token: None 2023-05-31T20:03:46.912 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:03:46 GMT /pklxkd-15/?versions 2023-05-31T20:03:46.912 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:klkBN5r1uaioeQo5gF5ArqPYPWA= 2023-05-31T20:03:46.912 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:03:46 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:klkBN5r1uaioeQo5gF5ArqPYPWA=', 'Content-Length': '0'} 2023-05-31T20:03:46.915 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e8f473066d895477-006477a822-1112-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:03:46 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:03:46.915 DEBUG:boto:b'pklxkd-15999falseobjDbIMGNowgdjqHv-6ZEpJLD71gcijEAqtrue2023-05-31T20:03:16.603Zrgw-multisite-test-userMultisite Test UserNormalobjf4Oe4Q1FAhoCXvv5PYPL-KsctMNpUQbfalse2023-05-31T20:03:16.578Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:03:46.916 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2023-05-31T20:03:46.916 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2023-05-31T20:03:46.917 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2023-05-31T20:03:46.917 DEBUG:boto:path=/pklxkd-15/ 2023-05-31T20:03:46.917 DEBUG:boto:auth_path=/pklxkd-15/ 2023-05-31T20:03:46.918 DEBUG:boto:path=/pklxkd-15/?versions&max-keys=999 2023-05-31T20:03:46.918 DEBUG:boto:auth_path=/pklxkd-15/?versions&max-keys=999 2023-05-31T20:03:46.918 DEBUG:boto:Method: GET 2023-05-31T20:03:46.919 DEBUG:boto:Path: /pklxkd-15/?versions&max-keys=999 2023-05-31T20:03:46.919 DEBUG:boto:Data: 2023-05-31T20:03:46.919 DEBUG:boto:Headers: {} 2023-05-31T20:03:46.920 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:03:46.920 DEBUG:boto:Port: 8000 2023-05-31T20:03:46.920 DEBUG:boto:Params: {} 2023-05-31T20:03:46.921 DEBUG:boto:Token: None 2023-05-31T20:03:46.921 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:03:46 GMT /pklxkd-15/?versions 2023-05-31T20:03:46.922 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:klkBN5r1uaioeQo5gF5ArqPYPWA= 2023-05-31T20:03:46.922 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:03:46 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:klkBN5r1uaioeQo5gF5ArqPYPWA=', 'Content-Length': '0'} 2023-05-31T20:03:46.924 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000bdec1e65b9d1cb7c-006477a822-110e-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:03:46 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:03:46.968 DEBUG:boto:b'pklxkd-15999falseobjDbIMGNowgdjqHv-6ZEpJLD71gcijEAqtrue2023-05-31T20:03:16.603Zrgw-multisite-test-userMultisite Test UserNormalobjf4Oe4Q1FAhoCXvv5PYPL-KsctMNpUQbfalse2023-05-31T20:03:16.578Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:03:46.969 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2023-05-31T20:03:46.970 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2023-05-31T20:03:46.970 DEBUG:boto:path=/pklxkd-15/ 2023-05-31T20:03:46.970 DEBUG:boto:auth_path=/pklxkd-15/ 2023-05-31T20:03:46.971 DEBUG:boto:path=/pklxkd-15/?versions&max-keys=999 2023-05-31T20:03:46.971 DEBUG:boto:auth_path=/pklxkd-15/?versions&max-keys=999 2023-05-31T20:03:46.971 DEBUG:boto:Method: GET 2023-05-31T20:03:46.972 DEBUG:boto:Path: /pklxkd-15/?versions&max-keys=999 2023-05-31T20:03:46.972 DEBUG:boto:Data: 2023-05-31T20:03:46.972 DEBUG:boto:Headers: {} 2023-05-31T20:03:46.973 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:03:46.973 DEBUG:boto:Port: 8001 2023-05-31T20:03:46.974 DEBUG:boto:Params: {} 2023-05-31T20:03:46.974 DEBUG:boto:Token: None 2023-05-31T20:03:46.974 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:03:46 GMT /pklxkd-15/?versions 2023-05-31T20:03:46.975 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:klkBN5r1uaioeQo5gF5ArqPYPWA= 2023-05-31T20:03:46.975 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:03:46 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:klkBN5r1uaioeQo5gF5ArqPYPWA=', 'Content-Length': '0'} 2023-05-31T20:03:46.978 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d1da5a7de5700a2a-006477a822-1112-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:03:46 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:03:46.978 DEBUG:boto:b'pklxkd-15999falseobjDbIMGNowgdjqHv-6ZEpJLD71gcijEAqtrue2023-05-31T20:03:16.603Zrgw-multisite-test-userMultisite Test UserNormalobjf4Oe4Q1FAhoCXvv5PYPL-KsctMNpUQbfalse2023-05-31T20:03:16.578Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:03:46.979 DEBUG:boto:path=/pklxkd-15/ 2023-05-31T20:03:46.979 DEBUG:boto:auth_path=/pklxkd-15/ 2023-05-31T20:03:46.980 DEBUG:boto:path=/pklxkd-15/?versions&max-keys=999 2023-05-31T20:03:46.980 DEBUG:boto:auth_path=/pklxkd-15/?versions&max-keys=999 2023-05-31T20:03:46.980 DEBUG:boto:Method: GET 2023-05-31T20:03:46.981 DEBUG:boto:Path: /pklxkd-15/?versions&max-keys=999 2023-05-31T20:03:46.981 DEBUG:boto:Data: 2023-05-31T20:03:46.981 DEBUG:boto:Headers: {} 2023-05-31T20:03:46.982 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:03:46.982 DEBUG:boto:Port: 8000 2023-05-31T20:03:46.983 DEBUG:boto:Params: {} 2023-05-31T20:03:46.983 DEBUG:boto:Token: None 2023-05-31T20:03:46.983 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:03:46 GMT /pklxkd-15/?versions 2023-05-31T20:03:46.984 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:klkBN5r1uaioeQo5gF5ArqPYPWA= 2023-05-31T20:03:46.984 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:03:46 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:klkBN5r1uaioeQo5gF5ArqPYPWA=', 'Content-Length': '0'} 2023-05-31T20:03:46.987 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000496e66c93883e304-006477a822-110e-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:03:46 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:03:46.987 DEBUG:boto:b'pklxkd-15999falseobjDbIMGNowgdjqHv-6ZEpJLD71gcijEAqtrue2023-05-31T20:03:16.603Zrgw-multisite-test-userMultisite Test UserNormalobjf4Oe4Q1FAhoCXvv5PYPL-KsctMNpUQbfalse2023-05-31T20:03:16.578Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:03:46.988 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2023-05-31T20:03:46.988 DEBUG:boto:path=/pklxkd-15/obj 2023-05-31T20:03:46.989 DEBUG:boto:auth_path=/pklxkd-15/obj 2023-05-31T20:03:46.989 DEBUG:boto:Method: HEAD 2023-05-31T20:03:46.989 DEBUG:boto:Path: /pklxkd-15/obj 2023-05-31T20:03:46.990 DEBUG:boto:Data: 2023-05-31T20:03:46.990 DEBUG:boto:Headers: {} 2023-05-31T20:03:46.990 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:03:46.991 DEBUG:boto:Port: 8001 2023-05-31T20:03:46.991 DEBUG:boto:Params: {} 2023-05-31T20:03:46.992 DEBUG:boto:Token: None 2023-05-31T20:03:46.992 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:03:46 GMT /pklxkd-15/obj 2023-05-31T20:03:46.992 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:htyEGLpkvnNtL3UW8D+A0m51M2w= 2023-05-31T20:03:46.993 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:03:46 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:htyEGLpkvnNtL3UW8D+A0m51M2w=', 'Content-Length': '0'} 2023-05-31T20:03:46.995 DEBUG:boto:Response headers: [('Content-Length', '219'), ('x-amz-request-id', 'tx000007010870b59e3a704-006477a822-1112-a2'), ('Accept-Ranges', 'bytes'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:03:46 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:03:46.996 DEBUG:boto:path=/pklxkd-15/obj 2023-05-31T20:03:46.996 DEBUG:boto:auth_path=/pklxkd-15/obj 2023-05-31T20:03:46.996 DEBUG:boto:Method: HEAD 2023-05-31T20:03:46.997 DEBUG:boto:Path: /pklxkd-15/obj 2023-05-31T20:03:46.997 DEBUG:boto:Data: 2023-05-31T20:03:46.997 DEBUG:boto:Headers: {} 2023-05-31T20:03:46.998 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:03:46.998 DEBUG:boto:Port: 8000 2023-05-31T20:03:46.998 DEBUG:boto:Params: {} 2023-05-31T20:03:46.999 DEBUG:boto:Token: None 2023-05-31T20:03:46.999 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:03:46 GMT /pklxkd-15/obj 2023-05-31T20:03:47.000 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:htyEGLpkvnNtL3UW8D+A0m51M2w= 2023-05-31T20:03:47.000 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:03:46 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:htyEGLpkvnNtL3UW8D+A0m51M2w=', 'Content-Length': '0'} 2023-05-31T20:03:47.002 DEBUG:boto:Response headers: [('Content-Length', '219'), ('x-amz-request-id', 'tx00000e74363d7d595e224-006477a822-110e-a1'), ('Accept-Ranges', 'bytes'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:03:46 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:03:47.003 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2023-05-31T20:03:47.003 DEBUG:boto:path=/pklxkd-15/obj 2023-05-31T20:03:47.004 DEBUG:boto:auth_path=/pklxkd-15/obj 2023-05-31T20:03:47.004 DEBUG:boto:path=/pklxkd-15/obj?versionId=f4Oe4Q1FAhoCXvv5PYPL-KsctMNpUQb 2023-05-31T20:03:47.004 DEBUG:boto:auth_path=/pklxkd-15/obj?versionId=f4Oe4Q1FAhoCXvv5PYPL-KsctMNpUQb 2023-05-31T20:03:47.005 DEBUG:boto:Method: GET 2023-05-31T20:03:47.005 DEBUG:boto:Path: /pklxkd-15/obj?versionId=f4Oe4Q1FAhoCXvv5PYPL-KsctMNpUQb 2023-05-31T20:03:47.005 DEBUG:boto:Data: 2023-05-31T20:03:47.006 DEBUG:boto:Headers: {} 2023-05-31T20:03:47.006 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:03:47.007 DEBUG:boto:Port: 8001 2023-05-31T20:03:47.007 DEBUG:boto:Params: {} 2023-05-31T20:03:47.007 DEBUG:boto:Token: None 2023-05-31T20:03:47.008 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:03:47 GMT /pklxkd-15/obj?versionId=f4Oe4Q1FAhoCXvv5PYPL-KsctMNpUQb 2023-05-31T20:03:47.008 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:2bq7LVh6w6IDAdNarrir4cAsOuI= 2023-05-31T20:03:47.009 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:03:47 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:2bq7LVh6w6IDAdNarrir4cAsOuI=', 'Content-Length': '0'} 2023-05-31T20:03:47.011 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:03:16 GMT'), ('x-amz-version-id', 'f4Oe4Q1FAhoCXvv5PYPL-KsctMNpUQb'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000fc6c97e48921aa9b-006477a823-1112-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:03:47 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:03:47.011 DEBUG:boto:path=/pklxkd-15/obj 2023-05-31T20:03:47.012 DEBUG:boto:auth_path=/pklxkd-15/obj 2023-05-31T20:03:47.012 DEBUG:boto:path=/pklxkd-15/obj?versionId=f4Oe4Q1FAhoCXvv5PYPL-KsctMNpUQb 2023-05-31T20:03:47.013 DEBUG:boto:auth_path=/pklxkd-15/obj?versionId=f4Oe4Q1FAhoCXvv5PYPL-KsctMNpUQb 2023-05-31T20:03:47.013 DEBUG:boto:Method: GET 2023-05-31T20:03:47.013 DEBUG:boto:Path: /pklxkd-15/obj?versionId=f4Oe4Q1FAhoCXvv5PYPL-KsctMNpUQb 2023-05-31T20:03:47.014 DEBUG:boto:Data: 2023-05-31T20:03:47.014 DEBUG:boto:Headers: {} 2023-05-31T20:03:47.014 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:03:47.015 DEBUG:boto:Port: 8000 2023-05-31T20:03:47.015 DEBUG:boto:Params: {} 2023-05-31T20:03:47.016 DEBUG:boto:Token: None 2023-05-31T20:03:47.016 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:03:47 GMT /pklxkd-15/obj?versionId=f4Oe4Q1FAhoCXvv5PYPL-KsctMNpUQb 2023-05-31T20:03:47.017 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:2bq7LVh6w6IDAdNarrir4cAsOuI= 2023-05-31T20:03:47.017 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:03:47 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:2bq7LVh6w6IDAdNarrir4cAsOuI=', 'Content-Length': '0'} 2023-05-31T20:03:47.019 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:03:16 GMT'), ('x-amz-version-id', 'f4Oe4Q1FAhoCXvv5PYPL-KsctMNpUQb'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000003ef15cf03454b2e1-006477a823-110e-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:03:47 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:03:47.019 DEBUG:boto:path=/pklxkd-15/obj 2023-05-31T20:03:47.020 DEBUG:boto:auth_path=/pklxkd-15/obj 2023-05-31T20:03:47.020 DEBUG:boto:path=/pklxkd-15/obj?versionId=f4Oe4Q1FAhoCXvv5PYPL-KsctMNpUQb 2023-05-31T20:03:47.020 DEBUG:boto:auth_path=/pklxkd-15/obj?versionId=f4Oe4Q1FAhoCXvv5PYPL-KsctMNpUQb 2023-05-31T20:03:47.021 DEBUG:boto:Method: HEAD 2023-05-31T20:03:47.021 DEBUG:boto:Path: /pklxkd-15/obj?versionId=f4Oe4Q1FAhoCXvv5PYPL-KsctMNpUQb 2023-05-31T20:03:47.021 DEBUG:boto:Data: 2023-05-31T20:03:47.022 DEBUG:boto:Headers: {} 2023-05-31T20:03:47.022 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:03:47.022 DEBUG:boto:Port: 8001 2023-05-31T20:03:47.023 DEBUG:boto:Params: {} 2023-05-31T20:03:47.023 DEBUG:boto:Token: None 2023-05-31T20:03:47.024 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:03:47 GMT /pklxkd-15/obj?versionId=f4Oe4Q1FAhoCXvv5PYPL-KsctMNpUQb 2023-05-31T20:03:47.024 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:cfPu1M9gAlEEQihqGIjdcRrDNX8= 2023-05-31T20:03:47.024 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:03:47 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:cfPu1M9gAlEEQihqGIjdcRrDNX8=', 'Content-Length': '0'} 2023-05-31T20:03:47.026 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:03:16 GMT'), ('x-amz-version-id', 'f4Oe4Q1FAhoCXvv5PYPL-KsctMNpUQb'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000005dfe5212cf84d980-006477a823-1112-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:03:47 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:03:47.027 DEBUG:boto:path=/pklxkd-15/obj 2023-05-31T20:03:47.027 DEBUG:boto:auth_path=/pklxkd-15/obj 2023-05-31T20:03:47.027 DEBUG:boto:path=/pklxkd-15/obj?versionId=f4Oe4Q1FAhoCXvv5PYPL-KsctMNpUQb 2023-05-31T20:03:47.028 DEBUG:boto:auth_path=/pklxkd-15/obj?versionId=f4Oe4Q1FAhoCXvv5PYPL-KsctMNpUQb 2023-05-31T20:03:47.028 DEBUG:boto:Method: HEAD 2023-05-31T20:03:47.028 DEBUG:boto:Path: /pklxkd-15/obj?versionId=f4Oe4Q1FAhoCXvv5PYPL-KsctMNpUQb 2023-05-31T20:03:47.029 DEBUG:boto:Data: 2023-05-31T20:03:47.029 DEBUG:boto:Headers: {} 2023-05-31T20:03:47.029 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:03:47.030 DEBUG:boto:Port: 8000 2023-05-31T20:03:47.030 DEBUG:boto:Params: {} 2023-05-31T20:03:47.031 DEBUG:boto:Token: None 2023-05-31T20:03:47.031 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:03:47 GMT /pklxkd-15/obj?versionId=f4Oe4Q1FAhoCXvv5PYPL-KsctMNpUQb 2023-05-31T20:03:47.031 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:cfPu1M9gAlEEQihqGIjdcRrDNX8= 2023-05-31T20:03:47.032 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:03:47 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:cfPu1M9gAlEEQihqGIjdcRrDNX8=', 'Content-Length': '0'} 2023-05-31T20:03:47.033 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:03:16 GMT'), ('x-amz-version-id', 'f4Oe4Q1FAhoCXvv5PYPL-KsctMNpUQb'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000001cb5d381d7cb08ad-006477a823-110e-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:03:47 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:03:47.034 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2023-05-31T20:03:47.034 DEBUG:boto:path=/pklxkd-15/obj 2023-05-31T20:03:47.035 DEBUG:boto:auth_path=/pklxkd-15/obj 2023-05-31T20:03:47.035 DEBUG:boto:path=/pklxkd-15/obj?versionId=f4Oe4Q1FAhoCXvv5PYPL-KsctMNpUQb 2023-05-31T20:03:47.036 DEBUG:boto:auth_path=/pklxkd-15/obj?versionId=f4Oe4Q1FAhoCXvv5PYPL-KsctMNpUQb 2023-05-31T20:03:47.036 DEBUG:boto:Method: GET 2023-05-31T20:03:47.036 DEBUG:boto:Path: /pklxkd-15/obj?versionId=f4Oe4Q1FAhoCXvv5PYPL-KsctMNpUQb 2023-05-31T20:03:47.037 DEBUG:boto:Data: 2023-05-31T20:03:47.037 DEBUG:boto:Headers: {} 2023-05-31T20:03:47.037 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:03:47.038 DEBUG:boto:Port: 8001 2023-05-31T20:03:47.038 DEBUG:boto:Params: {} 2023-05-31T20:03:47.039 DEBUG:boto:Token: None 2023-05-31T20:03:47.039 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:03:47 GMT /pklxkd-15/obj?versionId=f4Oe4Q1FAhoCXvv5PYPL-KsctMNpUQb 2023-05-31T20:03:47.040 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:2bq7LVh6w6IDAdNarrir4cAsOuI= 2023-05-31T20:03:47.040 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:03:47 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:2bq7LVh6w6IDAdNarrir4cAsOuI=', 'Content-Length': '0'} 2023-05-31T20:03:47.042 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:03:16 GMT'), ('x-amz-version-id', 'f4Oe4Q1FAhoCXvv5PYPL-KsctMNpUQb'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000048b21cd82843436-006477a823-1112-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:03:47 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:03:47.042 DEBUG:boto:path=/pklxkd-15/obj 2023-05-31T20:03:47.043 DEBUG:boto:auth_path=/pklxkd-15/obj 2023-05-31T20:03:47.043 DEBUG:boto:path=/pklxkd-15/obj?versionId=f4Oe4Q1FAhoCXvv5PYPL-KsctMNpUQb 2023-05-31T20:03:47.043 DEBUG:boto:auth_path=/pklxkd-15/obj?versionId=f4Oe4Q1FAhoCXvv5PYPL-KsctMNpUQb 2023-05-31T20:03:47.044 DEBUG:boto:Method: GET 2023-05-31T20:03:47.044 DEBUG:boto:Path: /pklxkd-15/obj?versionId=f4Oe4Q1FAhoCXvv5PYPL-KsctMNpUQb 2023-05-31T20:03:47.045 DEBUG:boto:Data: 2023-05-31T20:03:47.045 DEBUG:boto:Headers: {} 2023-05-31T20:03:47.045 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:03:47.046 DEBUG:boto:Port: 8000 2023-05-31T20:03:47.046 DEBUG:boto:Params: {} 2023-05-31T20:03:47.047 DEBUG:boto:Token: None 2023-05-31T20:03:47.047 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:03:47 GMT /pklxkd-15/obj?versionId=f4Oe4Q1FAhoCXvv5PYPL-KsctMNpUQb 2023-05-31T20:03:47.047 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:2bq7LVh6w6IDAdNarrir4cAsOuI= 2023-05-31T20:03:47.048 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:03:47 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:2bq7LVh6w6IDAdNarrir4cAsOuI=', 'Content-Length': '0'} 2023-05-31T20:03:47.049 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:03:16 GMT'), ('x-amz-version-id', 'f4Oe4Q1FAhoCXvv5PYPL-KsctMNpUQb'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000009cc4e23e90b20f8f-006477a823-110e-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:03:47 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:03:47.050 DEBUG:boto:path=/pklxkd-15/ 2023-05-31T20:03:47.050 DEBUG:boto:auth_path=/pklxkd-15/ 2023-05-31T20:03:47.051 DEBUG:boto:path=/pklxkd-15/?prefix=obj 2023-05-31T20:03:47.051 DEBUG:boto:auth_path=/pklxkd-15/?prefix=obj 2023-05-31T20:03:47.052 DEBUG:boto:Method: GET 2023-05-31T20:03:47.052 DEBUG:boto:Path: /pklxkd-15/?prefix=obj 2023-05-31T20:03:47.052 DEBUG:boto:Data: 2023-05-31T20:03:47.053 DEBUG:boto:Headers: {} 2023-05-31T20:03:47.053 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:03:47.053 DEBUG:boto:Port: 8001 2023-05-31T20:03:47.054 DEBUG:boto:Params: {} 2023-05-31T20:03:47.054 DEBUG:boto:Token: None 2023-05-31T20:03:47.055 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:03:47 GMT /pklxkd-15/ 2023-05-31T20:03:47.055 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:8XMNKLvD0Ty+QaFOvrRXDaSIcZE= 2023-05-31T20:03:47.056 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:03:47 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:8XMNKLvD0Ty+QaFOvrRXDaSIcZE=', 'Content-Length': '0'} 2023-05-31T20:03:47.058 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000de04a3600cf50f9e-006477a823-1112-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:03:47 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:03:47.058 DEBUG:boto:b'pklxkd-15obj1000false' 2023-05-31T20:03:47.059 DEBUG:boto:path=/pklxkd-15/ 2023-05-31T20:03:47.059 DEBUG:boto:auth_path=/pklxkd-15/ 2023-05-31T20:03:47.060 DEBUG:boto:path=/pklxkd-15/?prefix=obj 2023-05-31T20:03:47.060 DEBUG:boto:auth_path=/pklxkd-15/?prefix=obj 2023-05-31T20:03:47.060 DEBUG:boto:Method: GET 2023-05-31T20:03:47.061 DEBUG:boto:Path: /pklxkd-15/?prefix=obj 2023-05-31T20:03:47.061 DEBUG:boto:Data: 2023-05-31T20:03:47.061 DEBUG:boto:Headers: {} 2023-05-31T20:03:47.062 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:03:47.062 DEBUG:boto:Port: 8000 2023-05-31T20:03:47.062 DEBUG:boto:Params: {} 2023-05-31T20:03:47.063 DEBUG:boto:Token: None 2023-05-31T20:03:47.063 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:03:47 GMT /pklxkd-15/ 2023-05-31T20:03:47.064 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:8XMNKLvD0Ty+QaFOvrRXDaSIcZE= 2023-05-31T20:03:47.064 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:03:47 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:8XMNKLvD0Ty+QaFOvrRXDaSIcZE=', 'Content-Length': '0'} 2023-05-31T20:03:47.066 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000003a27011d44d929f9-006477a823-110e-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:03:47 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:03:47.067 DEBUG:boto:b'pklxkd-15obj1000false' 2023-05-31T20:03:47.067 DEBUG:boto:path=/pklxkd-15/obj 2023-05-31T20:03:47.068 DEBUG:boto:auth_path=/pklxkd-15/obj 2023-05-31T20:03:47.068 DEBUG:boto:Method: HEAD 2023-05-31T20:03:47.069 DEBUG:boto:Path: /pklxkd-15/obj 2023-05-31T20:03:47.069 DEBUG:boto:Data: 2023-05-31T20:03:47.069 DEBUG:boto:Headers: {} 2023-05-31T20:03:47.070 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:03:47.070 DEBUG:boto:Port: 8001 2023-05-31T20:03:47.070 DEBUG:boto:Params: {} 2023-05-31T20:03:47.071 DEBUG:boto:Token: None 2023-05-31T20:03:47.071 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:03:47 GMT /pklxkd-15/obj 2023-05-31T20:03:47.072 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:05064YZMB585O+neNo1dZ2Pac6A= 2023-05-31T20:03:47.072 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:03:47 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:05064YZMB585O+neNo1dZ2Pac6A=', 'Content-Length': '0'} 2023-05-31T20:03:47.074 DEBUG:boto:Response headers: [('Content-Length', '219'), ('x-amz-request-id', 'tx00000bdd241aff666dcc5-006477a823-1112-a2'), ('Accept-Ranges', 'bytes'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:03:47 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:03:47.074 DEBUG:boto:path=/pklxkd-15/obj 2023-05-31T20:03:47.075 DEBUG:boto:auth_path=/pklxkd-15/obj 2023-05-31T20:03:47.075 DEBUG:boto:Method: HEAD 2023-05-31T20:03:47.076 DEBUG:boto:Path: /pklxkd-15/obj 2023-05-31T20:03:47.076 DEBUG:boto:Data: 2023-05-31T20:03:47.076 DEBUG:boto:Headers: {} 2023-05-31T20:03:47.077 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:03:47.077 DEBUG:boto:Port: 8000 2023-05-31T20:03:47.077 DEBUG:boto:Params: {} 2023-05-31T20:03:47.078 DEBUG:boto:Token: None 2023-05-31T20:03:47.078 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:03:47 GMT /pklxkd-15/obj 2023-05-31T20:03:47.079 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:05064YZMB585O+neNo1dZ2Pac6A= 2023-05-31T20:03:47.079 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:03:47 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:05064YZMB585O+neNo1dZ2Pac6A=', 'Content-Length': '0'} 2023-05-31T20:03:47.081 DEBUG:boto:Response headers: [('Content-Length', '219'), ('x-amz-request-id', 'tx00000a4679787087e97d9-006477a823-110e-a1'), ('Accept-Ranges', 'bytes'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:03:47 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:03:47.082 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=pklxkd-15 zones={a2, a1} 2023-05-31T20:03:47.083 DEBUG:rgw_multi.tests:bucket checkpoint: source=a1 target=a2 bucket=pklxkd-16 2023-05-31T20:03:47.083 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'pklxkd-16', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-05-31T20:03:47.083 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'pklxkd-16', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-05-31T20:03:47.084 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket pklxkd-16 --source-zone a1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2023-05-31T20:03:47.143 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2023-05-31T20:03:47.143 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2023-05-31T20:03:47.181 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:03:47.175+0000 7f8f12849780 1 bucket sync caught up with source: 2023-05-31T20:03:47.181 INFO:teuthology.orchestra.run.smithi099.stderr: local status: [, , , , , , , , 00000000002.10.6, , ] 2023-05-31T20:03:47.181 INFO:teuthology.orchestra.run.smithi099.stderr: remote markers: [, , , , , , , , 00000000002.10.6, , ] 2023-05-31T20:03:47.182 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:03:47.175+0000 7f8f12849780 0 bucket checkpoint complete 2023-05-31T20:03:47.193 DEBUG:rgw_multi.tests:bucket checkpoint: source=a2 target=a1 bucket=pklxkd-16 2023-05-31T20:03:47.194 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'pklxkd-16', '--source-zone', 'a2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-05-31T20:03:47.194 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'pklxkd-16', '--source-zone', 'a2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-05-31T20:03:47.194 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket pklxkd-16 --source-zone a2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2023-05-31T20:03:47.251 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2023-05-31T20:03:47.251 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2023-05-31T20:03:47.296 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:03:47.290+0000 7f37c3518780 1 bucket sync caught up with source: 2023-05-31T20:03:47.297 INFO:teuthology.orchestra.run.smithi099.stderr: local status: [, , , , , , , , 00000000003.22.12, , ] 2023-05-31T20:03:47.297 INFO:teuthology.orchestra.run.smithi099.stderr: remote markers: [, , , , , , , , 00000000003.22.12, , ] 2023-05-31T20:03:47.297 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:03:47.290+0000 7f37c3518780 0 bucket checkpoint complete 2023-05-31T20:03:47.313 INFO:tasks.rgw_multi.zone_rados:comparing bucket=pklxkd-16 zones={a2, a1} 2023-05-31T20:03:47.314 DEBUG:boto:path=/pklxkd-16/ 2023-05-31T20:03:47.315 DEBUG:boto:auth_path=/pklxkd-16/ 2023-05-31T20:03:47.316 DEBUG:boto:Method: HEAD 2023-05-31T20:03:47.316 DEBUG:boto:Path: /pklxkd-16/ 2023-05-31T20:03:47.317 DEBUG:boto:Data: 2023-05-31T20:03:47.317 DEBUG:boto:Headers: {} 2023-05-31T20:03:47.317 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:03:47.318 DEBUG:boto:Port: 8001 2023-05-31T20:03:47.318 DEBUG:boto:Params: {} 2023-05-31T20:03:47.318 DEBUG:boto:Token: None 2023-05-31T20:03:47.319 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:03:47 GMT /pklxkd-16/ 2023-05-31T20:03:47.319 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:RSssfd85gGCX2VT0gSVanHT5Jek= 2023-05-31T20:03:47.320 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:03:47 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:RSssfd85gGCX2VT0gSVanHT5Jek=', 'Content-Length': '0'} 2023-05-31T20:03:47.322 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000f236de5f17aaaf0b-006477a823-1112-a2'), ('Content-Length', '0'), ('Date', 'Wed, 31 May 2023 20:03:47 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:03:47.323 DEBUG:boto:path=/pklxkd-16/ 2023-05-31T20:03:47.323 DEBUG:boto:auth_path=/pklxkd-16/ 2023-05-31T20:03:47.323 DEBUG:boto:Method: HEAD 2023-05-31T20:03:47.324 DEBUG:boto:Path: /pklxkd-16/ 2023-05-31T20:03:47.324 DEBUG:boto:Data: 2023-05-31T20:03:47.324 DEBUG:boto:Headers: {} 2023-05-31T20:03:47.325 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:03:47.325 DEBUG:boto:Port: 8000 2023-05-31T20:03:47.325 DEBUG:boto:Params: {} 2023-05-31T20:03:47.326 DEBUG:boto:Token: None 2023-05-31T20:03:47.326 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:03:47 GMT /pklxkd-16/ 2023-05-31T20:03:47.327 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:RSssfd85gGCX2VT0gSVanHT5Jek= 2023-05-31T20:03:47.327 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:03:47 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:RSssfd85gGCX2VT0gSVanHT5Jek=', 'Content-Length': '0'} 2023-05-31T20:03:47.329 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000025516e567500d214-006477a823-110e-a1'), ('Content-Length', '0'), ('Date', 'Wed, 31 May 2023 20:03:47 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:03:47.329 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2023-05-31T20:03:47.330 DEBUG:boto:path=/pklxkd-16/ 2023-05-31T20:03:47.330 DEBUG:boto:auth_path=/pklxkd-16/ 2023-05-31T20:03:47.330 DEBUG:boto:path=/pklxkd-16/?versions&max-keys=999 2023-05-31T20:03:47.331 DEBUG:boto:auth_path=/pklxkd-16/?versions&max-keys=999 2023-05-31T20:03:47.331 DEBUG:boto:Method: GET 2023-05-31T20:03:47.331 DEBUG:boto:Path: /pklxkd-16/?versions&max-keys=999 2023-05-31T20:03:47.332 DEBUG:boto:Data: 2023-05-31T20:03:47.332 DEBUG:boto:Headers: {} 2023-05-31T20:03:47.333 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:03:47.333 DEBUG:boto:Port: 8001 2023-05-31T20:03:47.333 DEBUG:boto:Params: {} 2023-05-31T20:03:47.334 DEBUG:boto:Token: None 2023-05-31T20:03:47.334 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:03:47 GMT /pklxkd-16/?versions 2023-05-31T20:03:47.334 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:HuzgJR1ntWKgnRd0tA6XNvx10K4= 2023-05-31T20:03:47.335 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:03:47 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:HuzgJR1ntWKgnRd0tA6XNvx10K4=', 'Content-Length': '0'} 2023-05-31T20:03:47.337 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000086d4ae68cebc398c-006477a823-1112-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:03:47 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:03:47.337 DEBUG:boto:b'pklxkd-16999falseobjCDaD4VG7xbuJHlHwO15k4ncmEwzgWUptrue2023-05-31T20:03:16.644Zrgw-multisite-test-userMultisite Test UserNormalobjYwu847uXzkc4Jr7tsfrFKB5zTC.q-jifalse2023-05-31T20:03:16.621Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:03:47.338 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2023-05-31T20:03:47.338 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2023-05-31T20:03:47.339 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2023-05-31T20:03:47.339 DEBUG:boto:path=/pklxkd-16/ 2023-05-31T20:03:47.339 DEBUG:boto:auth_path=/pklxkd-16/ 2023-05-31T20:03:47.340 DEBUG:boto:path=/pklxkd-16/?versions&max-keys=999 2023-05-31T20:03:47.340 DEBUG:boto:auth_path=/pklxkd-16/?versions&max-keys=999 2023-05-31T20:03:47.341 DEBUG:boto:Method: GET 2023-05-31T20:03:47.341 DEBUG:boto:Path: /pklxkd-16/?versions&max-keys=999 2023-05-31T20:03:47.341 DEBUG:boto:Data: 2023-05-31T20:03:47.342 DEBUG:boto:Headers: {} 2023-05-31T20:03:47.342 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:03:47.342 DEBUG:boto:Port: 8000 2023-05-31T20:03:47.343 DEBUG:boto:Params: {} 2023-05-31T20:03:47.343 DEBUG:boto:Token: None 2023-05-31T20:03:47.343 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:03:47 GMT /pklxkd-16/?versions 2023-05-31T20:03:47.344 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:HuzgJR1ntWKgnRd0tA6XNvx10K4= 2023-05-31T20:03:47.344 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:03:47 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:HuzgJR1ntWKgnRd0tA6XNvx10K4=', 'Content-Length': '0'} 2023-05-31T20:03:47.346 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000608e1b91a13117d7-006477a823-110e-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:03:47 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:03:47.347 DEBUG:boto:b'pklxkd-16999falseobjCDaD4VG7xbuJHlHwO15k4ncmEwzgWUptrue2023-05-31T20:03:16.644Zrgw-multisite-test-userMultisite Test UserNormalobjYwu847uXzkc4Jr7tsfrFKB5zTC.q-jifalse2023-05-31T20:03:16.621Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:03:47.347 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2023-05-31T20:03:47.348 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2023-05-31T20:03:47.348 DEBUG:boto:path=/pklxkd-16/ 2023-05-31T20:03:47.348 DEBUG:boto:auth_path=/pklxkd-16/ 2023-05-31T20:03:47.349 DEBUG:boto:path=/pklxkd-16/?versions&max-keys=999 2023-05-31T20:03:47.349 DEBUG:boto:auth_path=/pklxkd-16/?versions&max-keys=999 2023-05-31T20:03:47.349 DEBUG:boto:Method: GET 2023-05-31T20:03:47.350 DEBUG:boto:Path: /pklxkd-16/?versions&max-keys=999 2023-05-31T20:03:47.350 DEBUG:boto:Data: 2023-05-31T20:03:47.351 DEBUG:boto:Headers: {} 2023-05-31T20:03:47.351 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:03:47.351 DEBUG:boto:Port: 8001 2023-05-31T20:03:47.352 DEBUG:boto:Params: {} 2023-05-31T20:03:47.352 DEBUG:boto:Token: None 2023-05-31T20:03:47.352 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:03:47 GMT /pklxkd-16/?versions 2023-05-31T20:03:47.353 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:HuzgJR1ntWKgnRd0tA6XNvx10K4= 2023-05-31T20:03:47.353 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:03:47 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:HuzgJR1ntWKgnRd0tA6XNvx10K4=', 'Content-Length': '0'} 2023-05-31T20:03:47.355 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000005d81b380665a0b27-006477a823-1112-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:03:47 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:03:47.396 DEBUG:boto:b'pklxkd-16999falseobjCDaD4VG7xbuJHlHwO15k4ncmEwzgWUptrue2023-05-31T20:03:16.644Zrgw-multisite-test-userMultisite Test UserNormalobjYwu847uXzkc4Jr7tsfrFKB5zTC.q-jifalse2023-05-31T20:03:16.621Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:03:47.397 DEBUG:boto:path=/pklxkd-16/ 2023-05-31T20:03:47.398 DEBUG:boto:auth_path=/pklxkd-16/ 2023-05-31T20:03:47.398 DEBUG:boto:path=/pklxkd-16/?versions&max-keys=999 2023-05-31T20:03:47.399 DEBUG:boto:auth_path=/pklxkd-16/?versions&max-keys=999 2023-05-31T20:03:47.399 DEBUG:boto:Method: GET 2023-05-31T20:03:47.399 DEBUG:boto:Path: /pklxkd-16/?versions&max-keys=999 2023-05-31T20:03:47.400 DEBUG:boto:Data: 2023-05-31T20:03:47.400 DEBUG:boto:Headers: {} 2023-05-31T20:03:47.400 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:03:47.401 DEBUG:boto:Port: 8000 2023-05-31T20:03:47.401 DEBUG:boto:Params: {} 2023-05-31T20:03:47.402 DEBUG:boto:Token: None 2023-05-31T20:03:47.402 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:03:47 GMT /pklxkd-16/?versions 2023-05-31T20:03:47.402 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:HuzgJR1ntWKgnRd0tA6XNvx10K4= 2023-05-31T20:03:47.403 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:03:47 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:HuzgJR1ntWKgnRd0tA6XNvx10K4=', 'Content-Length': '0'} 2023-05-31T20:03:47.405 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000001f90e20bc38ddd7f-006477a823-110e-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:03:47 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:03:47.406 DEBUG:boto:b'pklxkd-16999falseobjCDaD4VG7xbuJHlHwO15k4ncmEwzgWUptrue2023-05-31T20:03:16.644Zrgw-multisite-test-userMultisite Test UserNormalobjYwu847uXzkc4Jr7tsfrFKB5zTC.q-jifalse2023-05-31T20:03:16.621Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:03:47.406 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2023-05-31T20:03:47.407 DEBUG:boto:path=/pklxkd-16/obj 2023-05-31T20:03:47.407 DEBUG:boto:auth_path=/pklxkd-16/obj 2023-05-31T20:03:47.407 DEBUG:boto:Method: HEAD 2023-05-31T20:03:47.408 DEBUG:boto:Path: /pklxkd-16/obj 2023-05-31T20:03:47.408 DEBUG:boto:Data: 2023-05-31T20:03:47.409 DEBUG:boto:Headers: {} 2023-05-31T20:03:47.409 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:03:47.410 DEBUG:boto:Port: 8001 2023-05-31T20:03:47.410 DEBUG:boto:Params: {} 2023-05-31T20:03:47.410 DEBUG:boto:Token: None 2023-05-31T20:03:47.411 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:03:47 GMT /pklxkd-16/obj 2023-05-31T20:03:47.411 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:dr2vFeSphdhy9uJvcGPui8pX7NM= 2023-05-31T20:03:47.412 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:03:47 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:dr2vFeSphdhy9uJvcGPui8pX7NM=', 'Content-Length': '0'} 2023-05-31T20:03:47.414 DEBUG:boto:Response headers: [('Content-Length', '219'), ('x-amz-request-id', 'tx000001828df866632cacf-006477a823-1112-a2'), ('Accept-Ranges', 'bytes'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:03:47 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:03:47.414 DEBUG:boto:path=/pklxkd-16/obj 2023-05-31T20:03:47.415 DEBUG:boto:auth_path=/pklxkd-16/obj 2023-05-31T20:03:47.415 DEBUG:boto:Method: HEAD 2023-05-31T20:03:47.415 DEBUG:boto:Path: /pklxkd-16/obj 2023-05-31T20:03:47.416 DEBUG:boto:Data: 2023-05-31T20:03:47.416 DEBUG:boto:Headers: {} 2023-05-31T20:03:47.416 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:03:47.417 DEBUG:boto:Port: 8000 2023-05-31T20:03:47.417 DEBUG:boto:Params: {} 2023-05-31T20:03:47.417 DEBUG:boto:Token: None 2023-05-31T20:03:47.418 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:03:47 GMT /pklxkd-16/obj 2023-05-31T20:03:47.418 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:dr2vFeSphdhy9uJvcGPui8pX7NM= 2023-05-31T20:03:47.419 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:03:47 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:dr2vFeSphdhy9uJvcGPui8pX7NM=', 'Content-Length': '0'} 2023-05-31T20:03:47.420 DEBUG:boto:Response headers: [('Content-Length', '219'), ('x-amz-request-id', 'tx000003151e41ba966b855-006477a823-110e-a1'), ('Accept-Ranges', 'bytes'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:03:47 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:03:47.421 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2023-05-31T20:03:47.421 DEBUG:boto:path=/pklxkd-16/obj 2023-05-31T20:03:47.422 DEBUG:boto:auth_path=/pklxkd-16/obj 2023-05-31T20:03:47.422 DEBUG:boto:path=/pklxkd-16/obj?versionId=Ywu847uXzkc4Jr7tsfrFKB5zTC.q-ji 2023-05-31T20:03:47.422 DEBUG:boto:auth_path=/pklxkd-16/obj?versionId=Ywu847uXzkc4Jr7tsfrFKB5zTC.q-ji 2023-05-31T20:03:47.423 DEBUG:boto:Method: GET 2023-05-31T20:03:47.423 DEBUG:boto:Path: /pklxkd-16/obj?versionId=Ywu847uXzkc4Jr7tsfrFKB5zTC.q-ji 2023-05-31T20:03:47.424 DEBUG:boto:Data: 2023-05-31T20:03:47.424 DEBUG:boto:Headers: {} 2023-05-31T20:03:47.424 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:03:47.425 DEBUG:boto:Port: 8001 2023-05-31T20:03:47.425 DEBUG:boto:Params: {} 2023-05-31T20:03:47.425 DEBUG:boto:Token: None 2023-05-31T20:03:47.426 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:03:47 GMT /pklxkd-16/obj?versionId=Ywu847uXzkc4Jr7tsfrFKB5zTC.q-ji 2023-05-31T20:03:47.426 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:bzWlvhfOKmkB+F16CzwZ2JYYZUU= 2023-05-31T20:03:47.427 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:03:47 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:bzWlvhfOKmkB+F16CzwZ2JYYZUU=', 'Content-Length': '0'} 2023-05-31T20:03:47.428 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:03:16 GMT'), ('x-amz-version-id', 'Ywu847uXzkc4Jr7tsfrFKB5zTC.q-ji'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx0000051771054f9c791ba-006477a823-1112-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:03:47 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:03:47.429 DEBUG:boto:path=/pklxkd-16/obj 2023-05-31T20:03:47.429 DEBUG:boto:auth_path=/pklxkd-16/obj 2023-05-31T20:03:47.430 DEBUG:boto:path=/pklxkd-16/obj?versionId=Ywu847uXzkc4Jr7tsfrFKB5zTC.q-ji 2023-05-31T20:03:47.430 DEBUG:boto:auth_path=/pklxkd-16/obj?versionId=Ywu847uXzkc4Jr7tsfrFKB5zTC.q-ji 2023-05-31T20:03:47.431 DEBUG:boto:Method: GET 2023-05-31T20:03:47.431 DEBUG:boto:Path: /pklxkd-16/obj?versionId=Ywu847uXzkc4Jr7tsfrFKB5zTC.q-ji 2023-05-31T20:03:47.431 DEBUG:boto:Data: 2023-05-31T20:03:47.432 DEBUG:boto:Headers: {} 2023-05-31T20:03:47.432 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:03:47.432 DEBUG:boto:Port: 8000 2023-05-31T20:03:47.433 DEBUG:boto:Params: {} 2023-05-31T20:03:47.433 DEBUG:boto:Token: None 2023-05-31T20:03:47.434 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:03:47 GMT /pklxkd-16/obj?versionId=Ywu847uXzkc4Jr7tsfrFKB5zTC.q-ji 2023-05-31T20:03:47.434 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:bzWlvhfOKmkB+F16CzwZ2JYYZUU= 2023-05-31T20:03:47.435 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:03:47 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:bzWlvhfOKmkB+F16CzwZ2JYYZUU=', 'Content-Length': '0'} 2023-05-31T20:03:47.436 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:03:16 GMT'), ('x-amz-version-id', 'Ywu847uXzkc4Jr7tsfrFKB5zTC.q-ji'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000001995547c4f667d61-006477a823-110e-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:03:47 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:03:47.437 DEBUG:boto:path=/pklxkd-16/obj 2023-05-31T20:03:47.437 DEBUG:boto:auth_path=/pklxkd-16/obj 2023-05-31T20:03:47.438 DEBUG:boto:path=/pklxkd-16/obj?versionId=Ywu847uXzkc4Jr7tsfrFKB5zTC.q-ji 2023-05-31T20:03:47.438 DEBUG:boto:auth_path=/pklxkd-16/obj?versionId=Ywu847uXzkc4Jr7tsfrFKB5zTC.q-ji 2023-05-31T20:03:47.438 DEBUG:boto:Method: HEAD 2023-05-31T20:03:47.439 DEBUG:boto:Path: /pklxkd-16/obj?versionId=Ywu847uXzkc4Jr7tsfrFKB5zTC.q-ji 2023-05-31T20:03:47.439 DEBUG:boto:Data: 2023-05-31T20:03:47.439 DEBUG:boto:Headers: {} 2023-05-31T20:03:47.440 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:03:47.440 DEBUG:boto:Port: 8001 2023-05-31T20:03:47.441 DEBUG:boto:Params: {} 2023-05-31T20:03:47.441 DEBUG:boto:Token: None 2023-05-31T20:03:47.442 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:03:47 GMT /pklxkd-16/obj?versionId=Ywu847uXzkc4Jr7tsfrFKB5zTC.q-ji 2023-05-31T20:03:47.442 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:KgVax0nI7N1/bRPut5f6N3gePJU= 2023-05-31T20:03:47.443 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:03:47 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:KgVax0nI7N1/bRPut5f6N3gePJU=', 'Content-Length': '0'} 2023-05-31T20:03:47.444 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:03:16 GMT'), ('x-amz-version-id', 'Ywu847uXzkc4Jr7tsfrFKB5zTC.q-ji'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000b497e5f66e98632f-006477a823-1112-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:03:47 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:03:47.445 DEBUG:boto:path=/pklxkd-16/obj 2023-05-31T20:03:47.446 DEBUG:boto:auth_path=/pklxkd-16/obj 2023-05-31T20:03:47.446 DEBUG:boto:path=/pklxkd-16/obj?versionId=Ywu847uXzkc4Jr7tsfrFKB5zTC.q-ji 2023-05-31T20:03:47.447 DEBUG:boto:auth_path=/pklxkd-16/obj?versionId=Ywu847uXzkc4Jr7tsfrFKB5zTC.q-ji 2023-05-31T20:03:47.447 DEBUG:boto:Method: HEAD 2023-05-31T20:03:47.448 DEBUG:boto:Path: /pklxkd-16/obj?versionId=Ywu847uXzkc4Jr7tsfrFKB5zTC.q-ji 2023-05-31T20:03:47.448 DEBUG:boto:Data: 2023-05-31T20:03:47.448 DEBUG:boto:Headers: {} 2023-05-31T20:03:47.449 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:03:47.449 DEBUG:boto:Port: 8000 2023-05-31T20:03:47.449 DEBUG:boto:Params: {} 2023-05-31T20:03:47.450 DEBUG:boto:Token: None 2023-05-31T20:03:47.450 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:03:47 GMT /pklxkd-16/obj?versionId=Ywu847uXzkc4Jr7tsfrFKB5zTC.q-ji 2023-05-31T20:03:47.451 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:KgVax0nI7N1/bRPut5f6N3gePJU= 2023-05-31T20:03:47.451 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:03:47 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:KgVax0nI7N1/bRPut5f6N3gePJU=', 'Content-Length': '0'} 2023-05-31T20:03:47.453 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:03:16 GMT'), ('x-amz-version-id', 'Ywu847uXzkc4Jr7tsfrFKB5zTC.q-ji'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000eb92c6ffb4460820-006477a823-110e-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:03:47 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:03:47.454 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2023-05-31T20:03:47.454 DEBUG:boto:path=/pklxkd-16/obj 2023-05-31T20:03:47.455 DEBUG:boto:auth_path=/pklxkd-16/obj 2023-05-31T20:03:47.455 DEBUG:boto:path=/pklxkd-16/obj?versionId=Ywu847uXzkc4Jr7tsfrFKB5zTC.q-ji 2023-05-31T20:03:47.456 DEBUG:boto:auth_path=/pklxkd-16/obj?versionId=Ywu847uXzkc4Jr7tsfrFKB5zTC.q-ji 2023-05-31T20:03:47.456 DEBUG:boto:Method: GET 2023-05-31T20:03:47.457 DEBUG:boto:Path: /pklxkd-16/obj?versionId=Ywu847uXzkc4Jr7tsfrFKB5zTC.q-ji 2023-05-31T20:03:47.457 DEBUG:boto:Data: 2023-05-31T20:03:47.457 DEBUG:boto:Headers: {} 2023-05-31T20:03:47.458 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:03:47.458 DEBUG:boto:Port: 8001 2023-05-31T20:03:47.459 DEBUG:boto:Params: {} 2023-05-31T20:03:47.459 DEBUG:boto:Token: None 2023-05-31T20:03:47.460 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:03:47 GMT /pklxkd-16/obj?versionId=Ywu847uXzkc4Jr7tsfrFKB5zTC.q-ji 2023-05-31T20:03:47.460 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:bzWlvhfOKmkB+F16CzwZ2JYYZUU= 2023-05-31T20:03:47.461 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:03:47 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:bzWlvhfOKmkB+F16CzwZ2JYYZUU=', 'Content-Length': '0'} 2023-05-31T20:03:47.462 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:03:16 GMT'), ('x-amz-version-id', 'Ywu847uXzkc4Jr7tsfrFKB5zTC.q-ji'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000003594be411a5afc0b-006477a823-1112-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:03:47 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:03:47.463 DEBUG:boto:path=/pklxkd-16/obj 2023-05-31T20:03:47.464 DEBUG:boto:auth_path=/pklxkd-16/obj 2023-05-31T20:03:47.464 DEBUG:boto:path=/pklxkd-16/obj?versionId=Ywu847uXzkc4Jr7tsfrFKB5zTC.q-ji 2023-05-31T20:03:47.464 DEBUG:boto:auth_path=/pklxkd-16/obj?versionId=Ywu847uXzkc4Jr7tsfrFKB5zTC.q-ji 2023-05-31T20:03:47.465 DEBUG:boto:Method: GET 2023-05-31T20:03:47.465 DEBUG:boto:Path: /pklxkd-16/obj?versionId=Ywu847uXzkc4Jr7tsfrFKB5zTC.q-ji 2023-05-31T20:03:47.465 DEBUG:boto:Data: 2023-05-31T20:03:47.466 DEBUG:boto:Headers: {} 2023-05-31T20:03:47.466 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:03:47.467 DEBUG:boto:Port: 8000 2023-05-31T20:03:47.467 DEBUG:boto:Params: {} 2023-05-31T20:03:47.467 DEBUG:boto:Token: None 2023-05-31T20:03:47.468 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:03:47 GMT /pklxkd-16/obj?versionId=Ywu847uXzkc4Jr7tsfrFKB5zTC.q-ji 2023-05-31T20:03:47.468 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:bzWlvhfOKmkB+F16CzwZ2JYYZUU= 2023-05-31T20:03:47.469 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:03:47 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:bzWlvhfOKmkB+F16CzwZ2JYYZUU=', 'Content-Length': '0'} 2023-05-31T20:03:47.470 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:03:16 GMT'), ('x-amz-version-id', 'Ywu847uXzkc4Jr7tsfrFKB5zTC.q-ji'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000002cad8df2b9ced0b8-006477a823-110e-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:03:47 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:03:47.471 DEBUG:boto:path=/pklxkd-16/ 2023-05-31T20:03:47.471 DEBUG:boto:auth_path=/pklxkd-16/ 2023-05-31T20:03:47.472 DEBUG:boto:path=/pklxkd-16/?prefix=obj 2023-05-31T20:03:47.472 DEBUG:boto:auth_path=/pklxkd-16/?prefix=obj 2023-05-31T20:03:47.473 DEBUG:boto:Method: GET 2023-05-31T20:03:47.473 DEBUG:boto:Path: /pklxkd-16/?prefix=obj 2023-05-31T20:03:47.474 DEBUG:boto:Data: 2023-05-31T20:03:47.474 DEBUG:boto:Headers: {} 2023-05-31T20:03:47.475 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:03:47.475 DEBUG:boto:Port: 8001 2023-05-31T20:03:47.476 DEBUG:boto:Params: {} 2023-05-31T20:03:47.476 DEBUG:boto:Token: None 2023-05-31T20:03:47.477 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:03:47 GMT /pklxkd-16/ 2023-05-31T20:03:47.477 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:BOnEhkjJDQ2U0sEY70RA0TwDKec= 2023-05-31T20:03:47.478 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:03:47 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:BOnEhkjJDQ2U0sEY70RA0TwDKec=', 'Content-Length': '0'} 2023-05-31T20:03:47.480 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000bf638d65c424e2fd-006477a823-1112-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:03:47 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:03:47.524 DEBUG:boto:b'pklxkd-16obj1000false' 2023-05-31T20:03:47.526 DEBUG:boto:path=/pklxkd-16/ 2023-05-31T20:03:47.527 DEBUG:boto:auth_path=/pklxkd-16/ 2023-05-31T20:03:47.527 DEBUG:boto:path=/pklxkd-16/?prefix=obj 2023-05-31T20:03:47.528 DEBUG:boto:auth_path=/pklxkd-16/?prefix=obj 2023-05-31T20:03:47.529 DEBUG:boto:Method: GET 2023-05-31T20:03:47.530 DEBUG:boto:Path: /pklxkd-16/?prefix=obj 2023-05-31T20:03:47.531 DEBUG:boto:Data: 2023-05-31T20:03:47.532 DEBUG:boto:Headers: {} 2023-05-31T20:03:47.533 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:03:47.533 DEBUG:boto:Port: 8000 2023-05-31T20:03:47.534 DEBUG:boto:Params: {} 2023-05-31T20:03:47.535 DEBUG:boto:Token: None 2023-05-31T20:03:47.536 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:03:47 GMT /pklxkd-16/ 2023-05-31T20:03:47.536 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:BOnEhkjJDQ2U0sEY70RA0TwDKec= 2023-05-31T20:03:47.537 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:03:47 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:BOnEhkjJDQ2U0sEY70RA0TwDKec=', 'Content-Length': '0'} 2023-05-31T20:03:47.539 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d3f422a83251a81b-006477a823-110e-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:03:47 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:03:47.540 DEBUG:boto:b'pklxkd-16obj1000false' 2023-05-31T20:03:47.541 DEBUG:boto:path=/pklxkd-16/obj 2023-05-31T20:03:47.542 DEBUG:boto:auth_path=/pklxkd-16/obj 2023-05-31T20:03:47.542 DEBUG:boto:Method: HEAD 2023-05-31T20:03:47.543 DEBUG:boto:Path: /pklxkd-16/obj 2023-05-31T20:03:47.543 DEBUG:boto:Data: 2023-05-31T20:03:47.544 DEBUG:boto:Headers: {} 2023-05-31T20:03:47.545 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:03:47.545 DEBUG:boto:Port: 8001 2023-05-31T20:03:47.546 DEBUG:boto:Params: {} 2023-05-31T20:03:47.546 DEBUG:boto:Token: None 2023-05-31T20:03:47.547 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:03:47 GMT /pklxkd-16/obj 2023-05-31T20:03:47.548 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:dr2vFeSphdhy9uJvcGPui8pX7NM= 2023-05-31T20:03:47.549 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:03:47 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:dr2vFeSphdhy9uJvcGPui8pX7NM=', 'Content-Length': '0'} 2023-05-31T20:03:47.551 DEBUG:boto:Response headers: [('Content-Length', '219'), ('x-amz-request-id', 'tx000002cbd36d9c1c3d88e-006477a823-1112-a2'), ('Accept-Ranges', 'bytes'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:03:47 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:03:47.552 DEBUG:boto:path=/pklxkd-16/obj 2023-05-31T20:03:47.553 DEBUG:boto:auth_path=/pklxkd-16/obj 2023-05-31T20:03:47.553 DEBUG:boto:Method: HEAD 2023-05-31T20:03:47.554 DEBUG:boto:Path: /pklxkd-16/obj 2023-05-31T20:03:47.554 DEBUG:boto:Data: 2023-05-31T20:03:47.555 DEBUG:boto:Headers: {} 2023-05-31T20:03:47.555 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:03:47.556 DEBUG:boto:Port: 8000 2023-05-31T20:03:47.556 DEBUG:boto:Params: {} 2023-05-31T20:03:47.556 DEBUG:boto:Token: None 2023-05-31T20:03:47.557 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:03:47 GMT /pklxkd-16/obj 2023-05-31T20:03:47.557 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:dr2vFeSphdhy9uJvcGPui8pX7NM= 2023-05-31T20:03:47.558 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:03:47 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:dr2vFeSphdhy9uJvcGPui8pX7NM=', 'Content-Length': '0'} 2023-05-31T20:03:47.561 DEBUG:boto:Response headers: [('Content-Length', '219'), ('x-amz-request-id', 'tx00000327fbb9875ebfaad-006477a823-110e-a1'), ('Accept-Ranges', 'bytes'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:03:47 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:03:47.562 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=pklxkd-16 zones={a2, a1} 2023-05-31T20:03:47.563 INFO:tasks.rgw_multisite_tests:rgw_multi.tests.test_delete_marker_full_sync ... ok 2023-05-31T20:03:47.564 INFO:rgw_multi.tests:create bucket zone=a1 name=pklxkd-17 2023-05-31T20:03:47.565 DEBUG:boto:path=/pklxkd-17/ 2023-05-31T20:03:47.565 DEBUG:boto:auth_path=/pklxkd-17/ 2023-05-31T20:03:47.566 DEBUG:boto:Method: PUT 2023-05-31T20:03:47.567 DEBUG:boto:Path: /pklxkd-17/ 2023-05-31T20:03:47.567 DEBUG:boto:Data: 2023-05-31T20:03:47.568 DEBUG:boto:Headers: {} 2023-05-31T20:03:47.569 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:03:47.569 DEBUG:boto:Port: 8000 2023-05-31T20:03:47.569 DEBUG:boto:Params: {} 2023-05-31T20:03:47.570 DEBUG:boto:Token: None 2023-05-31T20:03:47.570 DEBUG:boto:StringToSign: PUT Wed, 31 May 2023 20:03:47 GMT /pklxkd-17/ 2023-05-31T20:03:47.571 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:RiJ3+auG5bKv8NHr+YeFg/3egsg= 2023-05-31T20:03:47.571 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:03:47 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:RiJ3+auG5bKv8NHr+YeFg/3egsg=', 'Content-Length': '0'} 2023-05-31T20:03:47.590 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx000007a67ae7af043265a-006477a823-110e-a1'), ('Content-Length', '0'), ('Date', 'Wed, 31 May 2023 20:03:47 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:03:47.592 INFO:rgw_multi.tests:create bucket zone=a2 name=pklxkd-18 2023-05-31T20:03:47.592 DEBUG:boto:path=/pklxkd-18/ 2023-05-31T20:03:47.593 DEBUG:boto:auth_path=/pklxkd-18/ 2023-05-31T20:03:47.594 DEBUG:boto:Method: PUT 2023-05-31T20:03:47.594 DEBUG:boto:Path: /pklxkd-18/ 2023-05-31T20:03:47.595 DEBUG:boto:Data: 2023-05-31T20:03:47.596 DEBUG:boto:Headers: {} 2023-05-31T20:03:47.597 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:03:47.598 DEBUG:boto:Port: 8001 2023-05-31T20:03:47.598 DEBUG:boto:Params: {} 2023-05-31T20:03:47.599 DEBUG:boto:Token: None 2023-05-31T20:03:47.600 DEBUG:boto:StringToSign: PUT Wed, 31 May 2023 20:03:47 GMT /pklxkd-18/ 2023-05-31T20:03:47.600 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:x4LNfYeQ8ETg76XI4B1kSVJNiHw= 2023-05-31T20:03:47.601 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:03:47 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:x4LNfYeQ8ETg76XI4B1kSVJNiHw=', 'Content-Length': '0'} 2023-05-31T20:03:47.623 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000ce11881d09bb933e-006477a823-1112-a2'), ('Content-Length', '0'), ('Date', 'Wed, 31 May 2023 20:03:47 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:03:47.624 DEBUG:boto:path=/pklxkd-17/ 2023-05-31T20:03:47.624 DEBUG:boto:auth_path=/pklxkd-17/ 2023-05-31T20:03:47.625 DEBUG:boto:path=/pklxkd-17/?versioning 2023-05-31T20:03:47.625 DEBUG:boto:auth_path=/pklxkd-17/?versioning 2023-05-31T20:03:47.625 DEBUG:boto:Method: PUT 2023-05-31T20:03:47.626 DEBUG:boto:Path: /pklxkd-17/?versioning 2023-05-31T20:03:47.626 DEBUG:boto:Data: Enabled Disabled 2023-05-31T20:03:47.626 DEBUG:boto:Headers: {} 2023-05-31T20:03:47.627 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:03:47.627 DEBUG:boto:Port: 8000 2023-05-31T20:03:47.628 DEBUG:boto:Params: {} 2023-05-31T20:03:47.628 DEBUG:boto:Token: None 2023-05-31T20:03:47.628 DEBUG:boto:StringToSign: PUT Wed, 31 May 2023 20:03:47 GMT /pklxkd-17/?versioning 2023-05-31T20:03:47.629 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:IkL23/lv7ArhkH25NrSZ0Vjtg3U= 2023-05-31T20:03:47.629 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:03:47 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:IkL23/lv7ArhkH25NrSZ0Vjtg3U=', 'Content-Length': '228'} 2023-05-31T20:03:47.634 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx0000003858fd95d37e4fc-006477a823-110e-a1'), ('Content-Type', 'application/xml'), ('Content-Length', '0'), ('Date', 'Wed, 31 May 2023 20:03:47 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:03:47.635 DEBUG:boto:path=/pklxkd-17/ 2023-05-31T20:03:47.635 DEBUG:boto:auth_path=/pklxkd-17/ 2023-05-31T20:03:47.635 DEBUG:boto:path=/pklxkd-17/?versioning 2023-05-31T20:03:47.636 DEBUG:boto:auth_path=/pklxkd-17/?versioning 2023-05-31T20:03:47.636 DEBUG:boto:Method: PUT 2023-05-31T20:03:47.636 DEBUG:boto:Path: /pklxkd-17/?versioning 2023-05-31T20:03:47.637 DEBUG:boto:Data: Suspended Disabled 2023-05-31T20:03:47.637 DEBUG:boto:Headers: {} 2023-05-31T20:03:47.638 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:03:47.638 DEBUG:boto:Port: 8000 2023-05-31T20:03:47.638 DEBUG:boto:Params: {} 2023-05-31T20:03:47.639 DEBUG:boto:Token: None 2023-05-31T20:03:47.639 DEBUG:boto:StringToSign: PUT Wed, 31 May 2023 20:03:47 GMT /pklxkd-17/?versioning 2023-05-31T20:03:47.640 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:IkL23/lv7ArhkH25NrSZ0Vjtg3U= 2023-05-31T20:03:47.640 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:03:47 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:IkL23/lv7ArhkH25NrSZ0Vjtg3U=', 'Content-Length': '230'} 2023-05-31T20:03:47.645 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000ebb706b9cb5ca209-006477a823-110e-a1'), ('Content-Type', 'application/xml'), ('Content-Length', '0'), ('Date', 'Wed, 31 May 2023 20:03:47 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:03:47.645 DEBUG:boto:path=/pklxkd-18/ 2023-05-31T20:03:47.645 DEBUG:boto:auth_path=/pklxkd-18/ 2023-05-31T20:03:47.646 DEBUG:boto:path=/pklxkd-18/?versioning 2023-05-31T20:03:47.646 DEBUG:boto:auth_path=/pklxkd-18/?versioning 2023-05-31T20:03:47.646 DEBUG:boto:Method: PUT 2023-05-31T20:03:47.647 DEBUG:boto:Path: /pklxkd-18/?versioning 2023-05-31T20:03:47.647 DEBUG:boto:Data: Enabled Disabled 2023-05-31T20:03:47.647 DEBUG:boto:Headers: {} 2023-05-31T20:03:47.648 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:03:47.648 DEBUG:boto:Port: 8001 2023-05-31T20:03:47.648 DEBUG:boto:Params: {} 2023-05-31T20:03:47.649 DEBUG:boto:Token: None 2023-05-31T20:03:47.649 DEBUG:boto:StringToSign: PUT Wed, 31 May 2023 20:03:47 GMT /pklxkd-18/?versioning 2023-05-31T20:03:47.650 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:cMjqLz+47IfPVUFyFH+PgfDfiMU= 2023-05-31T20:03:47.650 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:03:47 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:cMjqLz+47IfPVUFyFH+PgfDfiMU=', 'Content-Length': '228'} 2023-05-31T20:03:47.658 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx000006de0da8552d3340a-006477a823-1112-a2'), ('Content-Type', 'application/xml'), ('Content-Length', '0'), ('Date', 'Wed, 31 May 2023 20:03:47 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:03:47.658 DEBUG:boto:path=/pklxkd-18/ 2023-05-31T20:03:47.659 DEBUG:boto:auth_path=/pklxkd-18/ 2023-05-31T20:03:47.659 DEBUG:boto:path=/pklxkd-18/?versioning 2023-05-31T20:03:47.659 DEBUG:boto:auth_path=/pklxkd-18/?versioning 2023-05-31T20:03:47.660 DEBUG:boto:Method: PUT 2023-05-31T20:03:47.660 DEBUG:boto:Path: /pklxkd-18/?versioning 2023-05-31T20:03:47.660 DEBUG:boto:Data: Suspended Disabled 2023-05-31T20:03:47.661 DEBUG:boto:Headers: {} 2023-05-31T20:03:47.661 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:03:47.662 DEBUG:boto:Port: 8001 2023-05-31T20:03:47.662 DEBUG:boto:Params: {} 2023-05-31T20:03:47.662 DEBUG:boto:Token: None 2023-05-31T20:03:47.663 DEBUG:boto:StringToSign: PUT Wed, 31 May 2023 20:03:47 GMT /pklxkd-18/?versioning 2023-05-31T20:03:47.663 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:cMjqLz+47IfPVUFyFH+PgfDfiMU= 2023-05-31T20:03:47.664 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:03:47 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:cMjqLz+47IfPVUFyFH+PgfDfiMU=', 'Content-Length': '230'} 2023-05-31T20:03:47.671 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx000002350cbbe0d9dc671-006477a823-1112-a2'), ('Content-Type', 'application/xml'), ('Content-Length', '0'), ('Date', 'Wed, 31 May 2023 20:03:47 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:03:47.671 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-31T20:03:47.672 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-31T20:03:47.672 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-05-31T20:03:47.723 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2023-05-31T20:03:47.723 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2023-05-31T20:03:47.782 INFO:teuthology.orchestra.run.smithi099.stderr:No --period given, using current period=2e6123d2-d695-4350-9a01-61a455bc782b 2023-05-31T20:03:47.799 INFO:teuthology.orchestra.run.smithi099.stdout:[ 2023-05-31T20:03:47.799 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T20:03:47.799 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1685562631.516089_59.1", 2023-05-31T20:03:47.799 INFO:teuthology.orchestra.run.smithi099.stdout: "last_update": "2023-05-31T19:50:31.516089Z" 2023-05-31T20:03:47.800 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:03:47.800 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T20:03:47.800 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "", 2023-05-31T20:03:47.800 INFO:teuthology.orchestra.run.smithi099.stdout: "last_update": "0.000000" 2023-05-31T20:03:47.800 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:03:47.801 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T20:03:47.801 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1685562970.274572_93.1", 2023-05-31T20:03:47.801 INFO:teuthology.orchestra.run.smithi099.stdout: "last_update": "2023-05-31T19:56:10.274572Z" 2023-05-31T20:03:47.801 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:03:47.801 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T20:03:47.802 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1685563427.664924_105.1", 2023-05-31T20:03:47.802 INFO:teuthology.orchestra.run.smithi099.stdout: "last_update": "2023-05-31T20:03:47.664924Z" 2023-05-31T20:03:47.802 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T20:03:47.802 INFO:teuthology.orchestra.run.smithi099.stdout:] 2023-05-31T20:03:47.803 DEBUG:rgw_multi.tests:master meta markers={0: '1_1685562631.516089_59.1', 1: '', 2: '1_1685562970.274572_93.1', 3: '1_1685563427.664924_105.1'} 2023-05-31T20:03:47.804 INFO:rgw_multi.tests:starting meta checkpoint for zone=a2 2023-05-31T20:03:47.804 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-31T20:03:47.805 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-31T20:03:47.805 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-05-31T20:03:47.855 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2023-05-31T20:03:47.856 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2023-05-31T20:03:47.895 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:03:47.889+0000 7fd078f72780 20 RGW-SYNC:meta: read sync status 2023-05-31T20:03:47.896 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:03:47.890+0000 7fd078f72780 20 RGW-SYNC:meta: read sync status 2023-05-31T20:03:47.907 INFO:teuthology.orchestra.run.smithi099.stdout:{ 2023-05-31T20:03:47.908 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_status": { 2023-05-31T20:03:47.908 INFO:teuthology.orchestra.run.smithi099.stdout: "info": { 2023-05-31T20:03:47.908 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "sync", 2023-05-31T20:03:47.908 INFO:teuthology.orchestra.run.smithi099.stdout: "num_shards": 4, 2023-05-31T20:03:47.908 INFO:teuthology.orchestra.run.smithi099.stdout: "period": "2e6123d2-d695-4350-9a01-61a455bc782b", 2023-05-31T20:03:47.908 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2023-05-31T20:03:47.909 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:03:47.909 INFO:teuthology.orchestra.run.smithi099.stdout: "markers": [ 2023-05-31T20:03:47.909 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T20:03:47.909 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 0, 2023-05-31T20:03:47.909 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T20:03:47.909 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2023-05-31T20:03:47.910 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1685562631.516089_59.1", 2023-05-31T20:03:47.910 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T20:03:47.910 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 1, 2023-05-31T20:03:47.910 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T20:03:47.911 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2023-05-31T19:50:31.516089Z", 2023-05-31T20:03:47.911 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2023-05-31T20:03:47.911 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T20:03:47.911 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:03:47.911 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T20:03:47.911 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 1, 2023-05-31T20:03:47.912 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T20:03:47.912 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2023-05-31T20:03:47.912 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "", 2023-05-31T20:03:47.912 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T20:03:47.912 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2023-05-31T20:03:47.912 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T20:03:47.912 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "0.000000", 2023-05-31T20:03:47.913 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2023-05-31T20:03:47.913 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T20:03:47.913 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:03:47.913 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T20:03:47.913 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 2, 2023-05-31T20:03:47.913 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T20:03:47.913 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2023-05-31T20:03:47.914 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1685562970.274572_93.1", 2023-05-31T20:03:47.914 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T20:03:47.914 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2023-05-31T20:03:47.914 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T20:03:47.914 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2023-05-31T19:56:10.274572Z", 2023-05-31T20:03:47.915 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2023-05-31T20:03:47.915 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T20:03:47.915 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:03:47.915 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T20:03:47.915 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 3, 2023-05-31T20:03:47.915 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T20:03:47.916 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2023-05-31T20:03:47.916 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1685563427.574465_90.1", 2023-05-31T20:03:47.916 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T20:03:47.916 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2023-05-31T20:03:47.916 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T20:03:47.916 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2023-05-31T20:03:47.574465Z", 2023-05-31T20:03:47.917 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2023-05-31T20:03:47.917 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T20:03:47.917 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T20:03:47.917 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2023-05-31T20:03:47.917 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:03:47.917 INFO:teuthology.orchestra.run.smithi099.stdout: "full_sync": { 2023-05-31T20:03:47.917 INFO:teuthology.orchestra.run.smithi099.stdout: "total": 1, 2023-05-31T20:03:47.918 INFO:teuthology.orchestra.run.smithi099.stdout: "complete": 1 2023-05-31T20:03:47.918 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:03:47.918 INFO:teuthology.orchestra.run.smithi099.stdout: "current_time": "2023-05-31T20:03:47Z" 2023-05-31T20:03:47.918 INFO:teuthology.orchestra.run.smithi099.stdout:} 2023-05-31T20:03:47.919 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "2e6123d2-d695-4350-9a01-61a455bc782b", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1685562631.516089_59.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-05-31T19:50:31.516089Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1685562970.274572_93.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-05-31T19:56:10.274572Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1685563427.574465_90.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-05-31T20:03:47.574465Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-05-31T20:03:47Z" } 2023-05-31T20:03:47.919 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1685562631.516089_59.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-05-31T19:50:31.516089Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1685562970.274572_93.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-05-31T19:56:10.274572Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1685563427.574465_90.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-05-31T20:03:47.574465Z', 'realm_epoch': 2}}] 2023-05-31T20:03:47.920 DEBUG:rgw_multi.tests:log_status={0: '1_1685562631.516089_59.1', 1: '', 2: '1_1685562970.274572_93.1', 3: '1_1685563427.664924_105.1'} 2023-05-31T20:03:47.921 DEBUG:rgw_multi.tests:sync_status={0: '1_1685562631.516089_59.1', 1: '', 2: '1_1685562970.274572_93.1', 3: '1_1685563427.574465_90.1'} 2023-05-31T20:03:47.921 WARNING:rgw_multi.tests:zone a2 behind master: shard=3 master=1_1685563427.664924_105.1 target=1_1685563427.574465_90.1 2023-05-31T20:03:52.922 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-31T20:03:52.923 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-31T20:03:52.923 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-05-31T20:03:52.981 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2023-05-31T20:03:52.981 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2023-05-31T20:03:53.015 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:03:53.009+0000 7f3d5fe19780 20 RGW-SYNC:meta: read sync status 2023-05-31T20:03:53.016 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:03:53.010+0000 7f3d5fe19780 20 RGW-SYNC:meta: read sync status 2023-05-31T20:03:53.038 INFO:teuthology.orchestra.run.smithi099.stdout:{ 2023-05-31T20:03:53.038 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_status": { 2023-05-31T20:03:53.038 INFO:teuthology.orchestra.run.smithi099.stdout: "info": { 2023-05-31T20:03:53.038 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "sync", 2023-05-31T20:03:53.038 INFO:teuthology.orchestra.run.smithi099.stdout: "num_shards": 4, 2023-05-31T20:03:53.039 INFO:teuthology.orchestra.run.smithi099.stdout: "period": "2e6123d2-d695-4350-9a01-61a455bc782b", 2023-05-31T20:03:53.039 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2023-05-31T20:03:53.039 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:03:53.040 INFO:teuthology.orchestra.run.smithi099.stdout: "markers": [ 2023-05-31T20:03:53.040 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T20:03:53.040 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 0, 2023-05-31T20:03:53.040 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T20:03:53.041 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2023-05-31T20:03:53.041 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1685562631.516089_59.1", 2023-05-31T20:03:53.041 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T20:03:53.041 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 1, 2023-05-31T20:03:53.041 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T20:03:53.041 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2023-05-31T19:50:31.516089Z", 2023-05-31T20:03:53.041 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2023-05-31T20:03:53.042 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T20:03:53.042 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:03:53.043 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T20:03:53.043 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 1, 2023-05-31T20:03:53.043 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T20:03:53.043 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2023-05-31T20:03:53.043 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "", 2023-05-31T20:03:53.043 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T20:03:53.044 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2023-05-31T20:03:53.044 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T20:03:53.044 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "0.000000", 2023-05-31T20:03:53.044 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2023-05-31T20:03:53.044 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T20:03:53.045 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:03:53.045 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T20:03:53.045 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 2, 2023-05-31T20:03:53.045 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T20:03:53.046 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2023-05-31T20:03:53.046 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1685562970.274572_93.1", 2023-05-31T20:03:53.046 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T20:03:53.046 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2023-05-31T20:03:53.046 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T20:03:53.046 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2023-05-31T19:56:10.274572Z", 2023-05-31T20:03:53.046 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2023-05-31T20:03:53.047 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T20:03:53.047 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:03:53.047 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T20:03:53.047 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 3, 2023-05-31T20:03:53.047 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T20:03:53.047 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2023-05-31T20:03:53.047 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1685563427.664924_105.1", 2023-05-31T20:03:53.048 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T20:03:53.048 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2023-05-31T20:03:53.048 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T20:03:53.048 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2023-05-31T20:03:47.664924Z", 2023-05-31T20:03:53.048 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2023-05-31T20:03:53.048 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T20:03:53.049 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T20:03:53.049 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2023-05-31T20:03:53.049 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:03:53.049 INFO:teuthology.orchestra.run.smithi099.stdout: "full_sync": { 2023-05-31T20:03:53.049 INFO:teuthology.orchestra.run.smithi099.stdout: "total": 1, 2023-05-31T20:03:53.049 INFO:teuthology.orchestra.run.smithi099.stdout: "complete": 1 2023-05-31T20:03:53.050 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:03:53.050 INFO:teuthology.orchestra.run.smithi099.stdout: "current_time": "2023-05-31T20:03:53Z" 2023-05-31T20:03:53.050 INFO:teuthology.orchestra.run.smithi099.stdout:} 2023-05-31T20:03:53.051 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "2e6123d2-d695-4350-9a01-61a455bc782b", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1685562631.516089_59.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-05-31T19:50:31.516089Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1685562970.274572_93.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-05-31T19:56:10.274572Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1685563427.664924_105.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-05-31T20:03:47.664924Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-05-31T20:03:53Z" } 2023-05-31T20:03:53.051 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1685562631.516089_59.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-05-31T19:50:31.516089Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1685562970.274572_93.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-05-31T19:56:10.274572Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1685563427.664924_105.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-05-31T20:03:47.664924Z', 'realm_epoch': 2}}] 2023-05-31T20:03:53.052 DEBUG:rgw_multi.tests:log_status={0: '1_1685562631.516089_59.1', 1: '', 2: '1_1685562970.274572_93.1', 3: '1_1685563427.664924_105.1'} 2023-05-31T20:03:53.052 DEBUG:rgw_multi.tests:sync_status={0: '1_1685562631.516089_59.1', 1: '', 2: '1_1685562970.274572_93.1', 3: '1_1685563427.664924_105.1'} 2023-05-31T20:03:53.053 INFO:rgw_multi.tests:finish meta checkpoint for zone=a2 2023-05-31T20:03:53.053 DEBUG:boto:path=/pklxkd-17/ 2023-05-31T20:03:53.054 DEBUG:boto:auth_path=/pklxkd-17/ 2023-05-31T20:03:53.054 DEBUG:boto:Method: HEAD 2023-05-31T20:03:53.055 DEBUG:boto:Path: /pklxkd-17/ 2023-05-31T20:03:53.055 DEBUG:boto:Data: 2023-05-31T20:03:53.056 DEBUG:boto:Headers: {} 2023-05-31T20:03:53.057 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:03:53.057 DEBUG:boto:Port: 8000 2023-05-31T20:03:53.058 DEBUG:boto:Params: {} 2023-05-31T20:03:53.059 DEBUG:boto:Token: None 2023-05-31T20:03:53.060 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:03:53 GMT /pklxkd-17/ 2023-05-31T20:03:53.061 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:LQPCwG7XWKpjHs/6spHg5NsR1Yg= 2023-05-31T20:03:53.062 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:03:53 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:LQPCwG7XWKpjHs/6spHg5NsR1Yg=', 'Content-Length': '0'} 2023-05-31T20:03:53.064 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000c7a383bbe1b0a59c-006477a829-110e-a1'), ('Content-Length', '0'), ('Date', 'Wed, 31 May 2023 20:03:53 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:03:53.066 DEBUG:boto:path=/pklxkd-17/obj 2023-05-31T20:03:53.066 DEBUG:boto:auth_path=/pklxkd-17/obj 2023-05-31T20:03:53.067 DEBUG:boto:Method: PUT 2023-05-31T20:03:53.068 DEBUG:boto:Path: /pklxkd-17/obj 2023-05-31T20:03:53.069 DEBUG:boto:Data: 2023-05-31T20:03:53.069 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': '1B2M2Y8AsgTpgAmY7PhCfg==', 'Content-Length': '0', 'Expect': '100-Continue'} 2023-05-31T20:03:53.070 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:03:53.071 DEBUG:boto:Port: 8000 2023-05-31T20:03:53.072 DEBUG:boto:Params: {} 2023-05-31T20:03:53.072 DEBUG:boto:Token: None 2023-05-31T20:03:53.073 DEBUG:boto:StringToSign: PUT 1B2M2Y8AsgTpgAmY7PhCfg== application/octet-stream Wed, 31 May 2023 20:03:53 GMT /pklxkd-17/obj 2023-05-31T20:03:53.074 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:bCbAsZb2jeNGNGdV1gAAF/D2FD4= 2023-05-31T20:03:53.075 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': '1B2M2Y8AsgTpgAmY7PhCfg==', 'Content-Length': '0', 'Expect': '100-Continue', 'Date': 'Wed, 31 May 2023 20:03:53 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:bCbAsZb2jeNGNGdV1gAAF/D2FD4='} 2023-05-31T20:03:53.084 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000559d311eaf946e60-006477a829-110e-a1'), ('Date', 'Wed, 31 May 2023 20:03:53 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:03:53.085 DEBUG:boto:path=/pklxkd-17/ 2023-05-31T20:03:53.085 DEBUG:boto:auth_path=/pklxkd-17/ 2023-05-31T20:03:53.086 DEBUG:boto:Method: HEAD 2023-05-31T20:03:53.087 DEBUG:boto:Path: /pklxkd-17/ 2023-05-31T20:03:53.088 DEBUG:boto:Data: 2023-05-31T20:03:53.089 DEBUG:boto:Headers: {} 2023-05-31T20:03:53.089 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:03:53.090 DEBUG:boto:Port: 8000 2023-05-31T20:03:53.090 DEBUG:boto:Params: {} 2023-05-31T20:03:53.091 DEBUG:boto:Token: None 2023-05-31T20:03:53.091 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:03:53 GMT /pklxkd-17/ 2023-05-31T20:03:53.092 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:LQPCwG7XWKpjHs/6spHg5NsR1Yg= 2023-05-31T20:03:53.092 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:03:53 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:LQPCwG7XWKpjHs/6spHg5NsR1Yg=', 'Content-Length': '0'} 2023-05-31T20:03:53.094 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000018f9c26706ad420a-006477a829-110e-a1'), ('Content-Length', '0'), ('Date', 'Wed, 31 May 2023 20:03:53 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:03:53.095 DEBUG:boto:path=/pklxkd-17/obj 2023-05-31T20:03:53.095 DEBUG:boto:auth_path=/pklxkd-17/obj 2023-05-31T20:03:53.096 DEBUG:boto:Method: DELETE 2023-05-31T20:03:53.096 DEBUG:boto:Path: /pklxkd-17/obj 2023-05-31T20:03:53.096 DEBUG:boto:Data: 2023-05-31T20:03:53.097 DEBUG:boto:Headers: {} 2023-05-31T20:03:53.097 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:03:53.097 DEBUG:boto:Port: 8000 2023-05-31T20:03:53.098 DEBUG:boto:Params: {} 2023-05-31T20:03:53.098 DEBUG:boto:Token: None 2023-05-31T20:03:53.098 DEBUG:boto:StringToSign: DELETE Wed, 31 May 2023 20:03:53 GMT /pklxkd-17/obj 2023-05-31T20:03:53.099 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:NPpwHYuA+tSx3SnGDqLQMGeJW18= 2023-05-31T20:03:53.099 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:03:53 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:NPpwHYuA+tSx3SnGDqLQMGeJW18=', 'Content-Length': '0'} 2023-05-31T20:03:53.111 DEBUG:boto:Response headers: [('x-amz-version-id', 'null'), ('x-amz-delete-marker', 'true'), ('x-amz-request-id', 'tx00000d2bdf2ab1e190f15-006477a829-110e-a1'), ('Date', 'Wed, 31 May 2023 20:03:53 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:03:53.112 DEBUG:boto:path=/pklxkd-18/ 2023-05-31T20:03:53.113 DEBUG:boto:auth_path=/pklxkd-18/ 2023-05-31T20:03:53.113 DEBUG:boto:Method: HEAD 2023-05-31T20:03:53.113 DEBUG:boto:Path: /pklxkd-18/ 2023-05-31T20:03:53.114 DEBUG:boto:Data: 2023-05-31T20:03:53.114 DEBUG:boto:Headers: {} 2023-05-31T20:03:53.114 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:03:53.115 DEBUG:boto:Port: 8001 2023-05-31T20:03:53.115 DEBUG:boto:Params: {} 2023-05-31T20:03:53.115 DEBUG:boto:Token: None 2023-05-31T20:03:53.116 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:03:53 GMT /pklxkd-18/ 2023-05-31T20:03:53.116 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:zv9lz7xGBNyBHRKdv0nZmfsOrfs= 2023-05-31T20:03:53.117 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:03:53 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:zv9lz7xGBNyBHRKdv0nZmfsOrfs=', 'Content-Length': '0'} 2023-05-31T20:03:53.119 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000d527ac92eaca869f-006477a829-1112-a2'), ('Content-Length', '0'), ('Date', 'Wed, 31 May 2023 20:03:53 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:03:53.119 DEBUG:boto:path=/pklxkd-18/obj 2023-05-31T20:03:53.120 DEBUG:boto:auth_path=/pklxkd-18/obj 2023-05-31T20:03:53.120 DEBUG:boto:Method: PUT 2023-05-31T20:03:53.120 DEBUG:boto:Path: /pklxkd-18/obj 2023-05-31T20:03:53.121 DEBUG:boto:Data: 2023-05-31T20:03:53.121 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': '1B2M2Y8AsgTpgAmY7PhCfg==', 'Content-Length': '0', 'Expect': '100-Continue'} 2023-05-31T20:03:53.121 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:03:53.122 DEBUG:boto:Port: 8001 2023-05-31T20:03:53.122 DEBUG:boto:Params: {} 2023-05-31T20:03:53.122 DEBUG:boto:Token: None 2023-05-31T20:03:53.123 DEBUG:boto:StringToSign: PUT 1B2M2Y8AsgTpgAmY7PhCfg== application/octet-stream Wed, 31 May 2023 20:03:53 GMT /pklxkd-18/obj 2023-05-31T20:03:53.123 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:oH0GfKePR4At3h8Flv3bOfm0iaQ= 2023-05-31T20:03:53.124 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': '1B2M2Y8AsgTpgAmY7PhCfg==', 'Content-Length': '0', 'Expect': '100-Continue', 'Date': 'Wed, 31 May 2023 20:03:53 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:oH0GfKePR4At3h8Flv3bOfm0iaQ='} 2023-05-31T20:03:53.130 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000f298310b811adacc-006477a829-1112-a2'), ('Date', 'Wed, 31 May 2023 20:03:53 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:03:53.131 DEBUG:boto:path=/pklxkd-18/ 2023-05-31T20:03:53.132 DEBUG:boto:auth_path=/pklxkd-18/ 2023-05-31T20:03:53.132 DEBUG:boto:Method: HEAD 2023-05-31T20:03:53.132 DEBUG:boto:Path: /pklxkd-18/ 2023-05-31T20:03:53.133 DEBUG:boto:Data: 2023-05-31T20:03:53.133 DEBUG:boto:Headers: {} 2023-05-31T20:03:53.133 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:03:53.134 DEBUG:boto:Port: 8001 2023-05-31T20:03:53.134 DEBUG:boto:Params: {} 2023-05-31T20:03:53.135 DEBUG:boto:Token: None 2023-05-31T20:03:53.135 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:03:53 GMT /pklxkd-18/ 2023-05-31T20:03:53.135 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:zv9lz7xGBNyBHRKdv0nZmfsOrfs= 2023-05-31T20:03:53.136 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:03:53 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:zv9lz7xGBNyBHRKdv0nZmfsOrfs=', 'Content-Length': '0'} 2023-05-31T20:03:53.138 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000c2d1f95cc34fc28f-006477a829-1112-a2'), ('Content-Length', '0'), ('Date', 'Wed, 31 May 2023 20:03:53 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:03:53.138 DEBUG:boto:path=/pklxkd-18/obj 2023-05-31T20:03:53.139 DEBUG:boto:auth_path=/pklxkd-18/obj 2023-05-31T20:03:53.139 DEBUG:boto:Method: DELETE 2023-05-31T20:03:53.139 DEBUG:boto:Path: /pklxkd-18/obj 2023-05-31T20:03:53.140 DEBUG:boto:Data: 2023-05-31T20:03:53.140 DEBUG:boto:Headers: {} 2023-05-31T20:03:53.140 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:03:53.141 DEBUG:boto:Port: 8001 2023-05-31T20:03:53.141 DEBUG:boto:Params: {} 2023-05-31T20:03:53.141 DEBUG:boto:Token: None 2023-05-31T20:03:53.142 DEBUG:boto:StringToSign: DELETE Wed, 31 May 2023 20:03:53 GMT /pklxkd-18/obj 2023-05-31T20:03:53.142 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:/jAtzwXdzH+kqFfL5ikmD6aAv3o= 2023-05-31T20:03:53.143 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:03:53 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:/jAtzwXdzH+kqFfL5ikmD6aAv3o=', 'Content-Length': '0'} 2023-05-31T20:03:53.156 DEBUG:boto:Response headers: [('x-amz-version-id', 'null'), ('x-amz-delete-marker', 'true'), ('x-amz-request-id', 'tx00000841a20618e2193ba-006477a829-1112-a2'), ('Date', 'Wed, 31 May 2023 20:03:53 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:03:53.157 DEBUG:rgw_multi.tests:bucket checkpoint: source=a1 target=a2 bucket=pklxkd-17 2023-05-31T20:03:53.157 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'pklxkd-17', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-05-31T20:03:53.158 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'pklxkd-17', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-05-31T20:03:53.158 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket pklxkd-17 --source-zone a1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2023-05-31T20:03:53.214 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2023-05-31T20:03:53.215 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2023-05-31T20:03:53.272 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:03:53.266+0000 7f16128b6780 1 waiting to reach incremental sync.. 2023-05-31T20:03:58.272 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:03:58.268+0000 7f16128b6780 1 waiting to reach incremental sync.. 2023-05-31T20:04:03.273 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:04:03.268+0000 7f16128b6780 1 waiting to reach incremental sync.. 2023-05-31T20:04:08.275 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:04:08.270+0000 7f16128b6780 1 bucket sync caught up with source: 2023-05-31T20:04:08.275 INFO:teuthology.orchestra.run.smithi099.stderr: local status: [, , , , , , , , 00000000003.17.7, , ] 2023-05-31T20:04:08.275 INFO:teuthology.orchestra.run.smithi099.stderr: remote markers: [, , , , , , , , 00000000003.17.7, , ] 2023-05-31T20:04:08.275 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:04:08.270+0000 7f16128b6780 0 bucket checkpoint complete 2023-05-31T20:04:08.297 DEBUG:rgw_multi.tests:bucket checkpoint: source=a2 target=a1 bucket=pklxkd-17 2023-05-31T20:04:08.297 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'pklxkd-17', '--source-zone', 'a2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-05-31T20:04:08.298 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'pklxkd-17', '--source-zone', 'a2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-05-31T20:04:08.298 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket pklxkd-17 --source-zone a2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2023-05-31T20:04:08.361 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2023-05-31T20:04:08.361 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2023-05-31T20:04:08.423 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:04:08.418+0000 7fdf44349780 1 waiting to reach incremental sync.. 2023-05-31T20:04:13.423 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:04:13.419+0000 7fdf44349780 1 waiting to reach incremental sync.. 2023-05-31T20:04:18.424 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:04:18.418+0000 7fdf44349780 1 waiting to reach incremental sync.. 2023-05-31T20:04:23.424 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:04:23.419+0000 7fdf44349780 1 waiting to reach incremental sync.. 2023-05-31T20:04:28.426 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:04:28.420+0000 7fdf44349780 1 bucket sync caught up with source: 2023-05-31T20:04:28.427 INFO:teuthology.orchestra.run.smithi099.stderr: local status: [, , , , , , , , 00000000001.30.13, , ] 2023-05-31T20:04:28.427 INFO:teuthology.orchestra.run.smithi099.stderr: remote markers: [, , , , , , , , 00000000001.30.13, , ] 2023-05-31T20:04:28.428 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:04:28.420+0000 7fdf44349780 0 bucket checkpoint complete 2023-05-31T20:04:28.442 INFO:tasks.rgw_multi.zone_rados:comparing bucket=pklxkd-17 zones={a2, a1} 2023-05-31T20:04:28.443 DEBUG:boto:path=/pklxkd-17/ 2023-05-31T20:04:28.443 DEBUG:boto:auth_path=/pklxkd-17/ 2023-05-31T20:04:28.444 DEBUG:boto:Method: HEAD 2023-05-31T20:04:28.444 DEBUG:boto:Path: /pklxkd-17/ 2023-05-31T20:04:28.444 DEBUG:boto:Data: 2023-05-31T20:04:28.445 DEBUG:boto:Headers: {} 2023-05-31T20:04:28.445 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:04:28.445 DEBUG:boto:Port: 8001 2023-05-31T20:04:28.446 DEBUG:boto:Params: {} 2023-05-31T20:04:28.446 DEBUG:boto:Token: None 2023-05-31T20:04:28.447 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:04:28 GMT /pklxkd-17/ 2023-05-31T20:04:28.447 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:1SlHFVNxdda1OGEkL5DRzMjSpaI= 2023-05-31T20:04:28.448 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:04:28 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:1SlHFVNxdda1OGEkL5DRzMjSpaI=', 'Content-Length': '0'} 2023-05-31T20:04:28.450 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000336f8e43f40ca3b6-006477a84c-1112-a2'), ('Content-Length', '0'), ('Date', 'Wed, 31 May 2023 20:04:28 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:04:28.451 DEBUG:boto:path=/pklxkd-17/ 2023-05-31T20:04:28.451 DEBUG:boto:auth_path=/pklxkd-17/ 2023-05-31T20:04:28.452 DEBUG:boto:Method: HEAD 2023-05-31T20:04:28.452 DEBUG:boto:Path: /pklxkd-17/ 2023-05-31T20:04:28.452 DEBUG:boto:Data: 2023-05-31T20:04:28.453 DEBUG:boto:Headers: {} 2023-05-31T20:04:28.453 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:04:28.453 DEBUG:boto:Port: 8000 2023-05-31T20:04:28.454 DEBUG:boto:Params: {} 2023-05-31T20:04:28.454 DEBUG:boto:Token: None 2023-05-31T20:04:28.454 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:04:28 GMT /pklxkd-17/ 2023-05-31T20:04:28.455 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:1SlHFVNxdda1OGEkL5DRzMjSpaI= 2023-05-31T20:04:28.455 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:04:28 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:1SlHFVNxdda1OGEkL5DRzMjSpaI=', 'Content-Length': '0'} 2023-05-31T20:04:28.457 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000008651f9d04689fb71-006477a84c-110e-a1'), ('Content-Length', '0'), ('Date', 'Wed, 31 May 2023 20:04:28 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:04:28.458 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2023-05-31T20:04:28.458 DEBUG:boto:path=/pklxkd-17/ 2023-05-31T20:04:28.459 DEBUG:boto:auth_path=/pklxkd-17/ 2023-05-31T20:04:28.459 DEBUG:boto:path=/pklxkd-17/?versions&max-keys=999 2023-05-31T20:04:28.459 DEBUG:boto:auth_path=/pklxkd-17/?versions&max-keys=999 2023-05-31T20:04:28.460 DEBUG:boto:Method: GET 2023-05-31T20:04:28.460 DEBUG:boto:Path: /pklxkd-17/?versions&max-keys=999 2023-05-31T20:04:28.460 DEBUG:boto:Data: 2023-05-31T20:04:28.461 DEBUG:boto:Headers: {} 2023-05-31T20:04:28.461 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:04:28.461 DEBUG:boto:Port: 8001 2023-05-31T20:04:28.462 DEBUG:boto:Params: {} 2023-05-31T20:04:28.462 DEBUG:boto:Token: None 2023-05-31T20:04:28.463 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:04:28 GMT /pklxkd-17/?versions 2023-05-31T20:04:28.463 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:i5F9+6BJ9oHo3YaxOshvKaV1iE0= 2023-05-31T20:04:28.463 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:04:28 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:i5F9+6BJ9oHo3YaxOshvKaV1iE0=', 'Content-Length': '0'} 2023-05-31T20:04:28.466 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f2baf8e9d76f7cc1-006477a84c-1112-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:04:28 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:04:28.466 DEBUG:boto:b'pklxkd-17999falseobjnulltrue2023-05-31T20:03:53.097Zrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:04:28.467 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2023-05-31T20:04:28.467 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2023-05-31T20:04:28.468 DEBUG:boto:path=/pklxkd-17/ 2023-05-31T20:04:28.468 DEBUG:boto:auth_path=/pklxkd-17/ 2023-05-31T20:04:28.468 DEBUG:boto:path=/pklxkd-17/?versions&max-keys=999 2023-05-31T20:04:28.469 DEBUG:boto:auth_path=/pklxkd-17/?versions&max-keys=999 2023-05-31T20:04:28.469 DEBUG:boto:Method: GET 2023-05-31T20:04:28.469 DEBUG:boto:Path: /pklxkd-17/?versions&max-keys=999 2023-05-31T20:04:28.470 DEBUG:boto:Data: 2023-05-31T20:04:28.470 DEBUG:boto:Headers: {} 2023-05-31T20:04:28.470 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:04:28.471 DEBUG:boto:Port: 8000 2023-05-31T20:04:28.471 DEBUG:boto:Params: {} 2023-05-31T20:04:28.471 DEBUG:boto:Token: None 2023-05-31T20:04:28.472 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:04:28 GMT /pklxkd-17/?versions 2023-05-31T20:04:28.472 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:i5F9+6BJ9oHo3YaxOshvKaV1iE0= 2023-05-31T20:04:28.473 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:04:28 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:i5F9+6BJ9oHo3YaxOshvKaV1iE0=', 'Content-Length': '0'} 2023-05-31T20:04:28.475 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000beaffad3ac3bce11-006477a84c-110e-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:04:28 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:04:28.516 DEBUG:boto:b'pklxkd-17999falseobjnulltrue2023-05-31T20:03:53.097Zrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:04:28.518 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2023-05-31T20:04:28.519 DEBUG:boto:path=/pklxkd-17/ 2023-05-31T20:04:28.520 DEBUG:boto:auth_path=/pklxkd-17/ 2023-05-31T20:04:28.521 DEBUG:boto:path=/pklxkd-17/?versions&max-keys=999 2023-05-31T20:04:28.522 DEBUG:boto:auth_path=/pklxkd-17/?versions&max-keys=999 2023-05-31T20:04:28.523 DEBUG:boto:Method: GET 2023-05-31T20:04:28.523 DEBUG:boto:Path: /pklxkd-17/?versions&max-keys=999 2023-05-31T20:04:28.523 DEBUG:boto:Data: 2023-05-31T20:04:28.524 DEBUG:boto:Headers: {} 2023-05-31T20:04:28.524 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:04:28.525 DEBUG:boto:Port: 8001 2023-05-31T20:04:28.526 DEBUG:boto:Params: {} 2023-05-31T20:04:28.527 DEBUG:boto:Token: None 2023-05-31T20:04:28.527 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:04:28 GMT /pklxkd-17/?versions 2023-05-31T20:04:28.528 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:i5F9+6BJ9oHo3YaxOshvKaV1iE0= 2023-05-31T20:04:28.528 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:04:28 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:i5F9+6BJ9oHo3YaxOshvKaV1iE0=', 'Content-Length': '0'} 2023-05-31T20:04:28.531 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e7ec6a7084d2160c-006477a84c-1112-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:04:28 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:04:28.532 DEBUG:boto:b'pklxkd-17999falseobjnulltrue2023-05-31T20:03:53.097Zrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:04:28.533 DEBUG:boto:path=/pklxkd-17/ 2023-05-31T20:04:28.533 DEBUG:boto:auth_path=/pklxkd-17/ 2023-05-31T20:04:28.534 DEBUG:boto:path=/pklxkd-17/?versions&max-keys=999 2023-05-31T20:04:28.535 DEBUG:boto:auth_path=/pklxkd-17/?versions&max-keys=999 2023-05-31T20:04:28.535 DEBUG:boto:Method: GET 2023-05-31T20:04:28.536 DEBUG:boto:Path: /pklxkd-17/?versions&max-keys=999 2023-05-31T20:04:28.536 DEBUG:boto:Data: 2023-05-31T20:04:28.537 DEBUG:boto:Headers: {} 2023-05-31T20:04:28.537 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:04:28.537 DEBUG:boto:Port: 8000 2023-05-31T20:04:28.538 DEBUG:boto:Params: {} 2023-05-31T20:04:28.538 DEBUG:boto:Token: None 2023-05-31T20:04:28.539 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:04:28 GMT /pklxkd-17/?versions 2023-05-31T20:04:28.539 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:i5F9+6BJ9oHo3YaxOshvKaV1iE0= 2023-05-31T20:04:28.539 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:04:28 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:i5F9+6BJ9oHo3YaxOshvKaV1iE0=', 'Content-Length': '0'} 2023-05-31T20:04:28.542 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000094c5350aeb285a52-006477a84c-110e-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:04:28 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:04:28.543 DEBUG:boto:b'pklxkd-17999falseobjnulltrue2023-05-31T20:03:53.097Zrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:04:28.543 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2023-05-31T20:04:28.544 DEBUG:boto:path=/pklxkd-17/obj 2023-05-31T20:04:28.544 DEBUG:boto:auth_path=/pklxkd-17/obj 2023-05-31T20:04:28.544 DEBUG:boto:Method: HEAD 2023-05-31T20:04:28.545 DEBUG:boto:Path: /pklxkd-17/obj 2023-05-31T20:04:28.546 DEBUG:boto:Data: 2023-05-31T20:04:28.546 DEBUG:boto:Headers: {} 2023-05-31T20:04:28.546 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:04:28.547 DEBUG:boto:Port: 8001 2023-05-31T20:04:28.547 DEBUG:boto:Params: {} 2023-05-31T20:04:28.548 DEBUG:boto:Token: None 2023-05-31T20:04:28.548 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:04:28 GMT /pklxkd-17/obj 2023-05-31T20:04:28.549 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:NiqbM//FsmP+sJCz2g/7ZoU1X8o= 2023-05-31T20:04:28.550 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:04:28 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:NiqbM//FsmP+sJCz2g/7ZoU1X8o=', 'Content-Length': '0'} 2023-05-31T20:04:28.552 DEBUG:boto:Response headers: [('Content-Length', '219'), ('x-amz-request-id', 'tx000008b1cebb502fc1b8e-006477a84c-1112-a2'), ('Accept-Ranges', 'bytes'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:04:28 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:04:28.553 DEBUG:boto:path=/pklxkd-17/obj 2023-05-31T20:04:28.554 DEBUG:boto:auth_path=/pklxkd-17/obj 2023-05-31T20:04:28.554 DEBUG:boto:Method: HEAD 2023-05-31T20:04:28.555 DEBUG:boto:Path: /pklxkd-17/obj 2023-05-31T20:04:28.555 DEBUG:boto:Data: 2023-05-31T20:04:28.556 DEBUG:boto:Headers: {} 2023-05-31T20:04:28.556 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:04:28.557 DEBUG:boto:Port: 8000 2023-05-31T20:04:28.557 DEBUG:boto:Params: {} 2023-05-31T20:04:28.557 DEBUG:boto:Token: None 2023-05-31T20:04:28.558 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:04:28 GMT /pklxkd-17/obj 2023-05-31T20:04:28.559 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:NiqbM//FsmP+sJCz2g/7ZoU1X8o= 2023-05-31T20:04:28.560 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:04:28 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:NiqbM//FsmP+sJCz2g/7ZoU1X8o=', 'Content-Length': '0'} 2023-05-31T20:04:28.562 DEBUG:boto:Response headers: [('Content-Length', '219'), ('x-amz-request-id', 'tx00000a63c6caf7917f0ef-006477a84c-110e-a1'), ('Accept-Ranges', 'bytes'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:04:28 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:04:28.562 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=pklxkd-17 zones={a2, a1} 2023-05-31T20:04:28.563 DEBUG:rgw_multi.tests:bucket checkpoint: source=a1 target=a2 bucket=pklxkd-18 2023-05-31T20:04:28.563 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'pklxkd-18', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-05-31T20:04:28.563 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'pklxkd-18', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-05-31T20:04:28.564 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket pklxkd-18 --source-zone a1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2023-05-31T20:04:28.628 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2023-05-31T20:04:28.628 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2023-05-31T20:04:28.673 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:04:28.667+0000 7f6c78030780 1 bucket sync caught up with source: 2023-05-31T20:04:28.674 INFO:teuthology.orchestra.run.smithi099.stderr: local status: [, , , , , , , , 00000000001.17.13, , ] 2023-05-31T20:04:28.674 INFO:teuthology.orchestra.run.smithi099.stderr: remote markers: [, , , , , , , , 00000000001.17.13, , ] 2023-05-31T20:04:28.674 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:04:28.667+0000 7f6c78030780 0 bucket checkpoint complete 2023-05-31T20:04:28.690 DEBUG:rgw_multi.tests:bucket checkpoint: source=a2 target=a1 bucket=pklxkd-18 2023-05-31T20:04:28.691 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'pklxkd-18', '--source-zone', 'a2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-05-31T20:04:28.691 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'pklxkd-18', '--source-zone', 'a2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-05-31T20:04:28.692 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket pklxkd-18 --source-zone a2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2023-05-31T20:04:28.748 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2023-05-31T20:04:28.749 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2023-05-31T20:04:28.794 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:04:28.788+0000 7f46dbb4d780 1 bucket sync caught up with source: 2023-05-31T20:04:28.794 INFO:teuthology.orchestra.run.smithi099.stderr: local status: [, , , , , , , , 00000000003.34.7, , ] 2023-05-31T20:04:28.795 INFO:teuthology.orchestra.run.smithi099.stderr: remote markers: [, , , , , , , , 00000000003.34.7, , ] 2023-05-31T20:04:28.795 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:04:28.788+0000 7f46dbb4d780 0 bucket checkpoint complete 2023-05-31T20:04:28.811 INFO:tasks.rgw_multi.zone_rados:comparing bucket=pklxkd-18 zones={a2, a1} 2023-05-31T20:04:28.812 DEBUG:boto:path=/pklxkd-18/ 2023-05-31T20:04:28.813 DEBUG:boto:auth_path=/pklxkd-18/ 2023-05-31T20:04:28.813 DEBUG:boto:Method: HEAD 2023-05-31T20:04:28.814 DEBUG:boto:Path: /pklxkd-18/ 2023-05-31T20:04:28.815 DEBUG:boto:Data: 2023-05-31T20:04:28.816 DEBUG:boto:Headers: {} 2023-05-31T20:04:28.816 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:04:28.817 DEBUG:boto:Port: 8001 2023-05-31T20:04:28.817 DEBUG:boto:Params: {} 2023-05-31T20:04:28.818 DEBUG:boto:Token: None 2023-05-31T20:04:28.818 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:04:28 GMT /pklxkd-18/ 2023-05-31T20:04:28.819 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:GRWbChdHF5R/HemOOGT27uYYIiE= 2023-05-31T20:04:28.819 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:04:28 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:GRWbChdHF5R/HemOOGT27uYYIiE=', 'Content-Length': '0'} 2023-05-31T20:04:28.821 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000082e9ccf5fb7972ef-006477a84c-1112-a2'), ('Content-Length', '0'), ('Date', 'Wed, 31 May 2023 20:04:28 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:04:28.823 DEBUG:boto:path=/pklxkd-18/ 2023-05-31T20:04:28.823 DEBUG:boto:auth_path=/pklxkd-18/ 2023-05-31T20:04:28.824 DEBUG:boto:Method: HEAD 2023-05-31T20:04:28.825 DEBUG:boto:Path: /pklxkd-18/ 2023-05-31T20:04:28.826 DEBUG:boto:Data: 2023-05-31T20:04:28.828 DEBUG:boto:Headers: {} 2023-05-31T20:04:28.829 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:04:28.830 DEBUG:boto:Port: 8000 2023-05-31T20:04:28.831 DEBUG:boto:Params: {} 2023-05-31T20:04:28.831 DEBUG:boto:Token: None 2023-05-31T20:04:28.832 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:04:28 GMT /pklxkd-18/ 2023-05-31T20:04:28.833 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:GRWbChdHF5R/HemOOGT27uYYIiE= 2023-05-31T20:04:28.834 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:04:28 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:GRWbChdHF5R/HemOOGT27uYYIiE=', 'Content-Length': '0'} 2023-05-31T20:04:28.838 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000069bca0421689cdc7-006477a84c-110e-a1'), ('Content-Length', '0'), ('Date', 'Wed, 31 May 2023 20:04:28 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:04:28.839 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2023-05-31T20:04:28.839 DEBUG:boto:path=/pklxkd-18/ 2023-05-31T20:04:28.839 DEBUG:boto:auth_path=/pklxkd-18/ 2023-05-31T20:04:28.840 DEBUG:boto:path=/pklxkd-18/?versions&max-keys=999 2023-05-31T20:04:28.840 DEBUG:boto:auth_path=/pklxkd-18/?versions&max-keys=999 2023-05-31T20:04:28.840 DEBUG:boto:Method: GET 2023-05-31T20:04:28.841 DEBUG:boto:Path: /pklxkd-18/?versions&max-keys=999 2023-05-31T20:04:28.841 DEBUG:boto:Data: 2023-05-31T20:04:28.841 DEBUG:boto:Headers: {} 2023-05-31T20:04:28.842 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:04:28.842 DEBUG:boto:Port: 8001 2023-05-31T20:04:28.842 DEBUG:boto:Params: {} 2023-05-31T20:04:28.843 DEBUG:boto:Token: None 2023-05-31T20:04:28.843 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:04:28 GMT /pklxkd-18/?versions 2023-05-31T20:04:28.844 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:OROp2OtuHwtOWuVVs65F01y3reM= 2023-05-31T20:04:28.844 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:04:28 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:OROp2OtuHwtOWuVVs65F01y3reM=', 'Content-Length': '0'} 2023-05-31T20:04:28.847 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000001c833242e0a52f43-006477a84c-1112-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:04:28 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:04:28.847 DEBUG:boto:b'pklxkd-18999falseobjnulltrue2023-05-31T20:03:53.140Zrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:04:28.848 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2023-05-31T20:04:28.849 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2023-05-31T20:04:28.849 DEBUG:boto:path=/pklxkd-18/ 2023-05-31T20:04:28.849 DEBUG:boto:auth_path=/pklxkd-18/ 2023-05-31T20:04:28.850 DEBUG:boto:path=/pklxkd-18/?versions&max-keys=999 2023-05-31T20:04:28.850 DEBUG:boto:auth_path=/pklxkd-18/?versions&max-keys=999 2023-05-31T20:04:28.850 DEBUG:boto:Method: GET 2023-05-31T20:04:28.851 DEBUG:boto:Path: /pklxkd-18/?versions&max-keys=999 2023-05-31T20:04:28.851 DEBUG:boto:Data: 2023-05-31T20:04:28.851 DEBUG:boto:Headers: {} 2023-05-31T20:04:28.852 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:04:28.852 DEBUG:boto:Port: 8000 2023-05-31T20:04:28.853 DEBUG:boto:Params: {} 2023-05-31T20:04:28.853 DEBUG:boto:Token: None 2023-05-31T20:04:28.853 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:04:28 GMT /pklxkd-18/?versions 2023-05-31T20:04:28.854 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:OROp2OtuHwtOWuVVs65F01y3reM= 2023-05-31T20:04:28.854 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:04:28 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:OROp2OtuHwtOWuVVs65F01y3reM=', 'Content-Length': '0'} 2023-05-31T20:04:28.856 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000003e066d0ab7a8e919-006477a84c-110e-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:04:28 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:04:28.857 DEBUG:boto:b'pklxkd-18999falseobjnulltrue2023-05-31T20:03:53.140Zrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:04:28.857 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2023-05-31T20:04:28.858 DEBUG:boto:path=/pklxkd-18/ 2023-05-31T20:04:28.858 DEBUG:boto:auth_path=/pklxkd-18/ 2023-05-31T20:04:28.858 DEBUG:boto:path=/pklxkd-18/?versions&max-keys=999 2023-05-31T20:04:28.859 DEBUG:boto:auth_path=/pklxkd-18/?versions&max-keys=999 2023-05-31T20:04:28.859 DEBUG:boto:Method: GET 2023-05-31T20:04:28.860 DEBUG:boto:Path: /pklxkd-18/?versions&max-keys=999 2023-05-31T20:04:28.860 DEBUG:boto:Data: 2023-05-31T20:04:28.860 DEBUG:boto:Headers: {} 2023-05-31T20:04:28.861 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:04:28.861 DEBUG:boto:Port: 8001 2023-05-31T20:04:28.861 DEBUG:boto:Params: {} 2023-05-31T20:04:28.862 DEBUG:boto:Token: None 2023-05-31T20:04:28.862 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:04:28 GMT /pklxkd-18/?versions 2023-05-31T20:04:28.862 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:OROp2OtuHwtOWuVVs65F01y3reM= 2023-05-31T20:04:28.863 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:04:28 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:OROp2OtuHwtOWuVVs65F01y3reM=', 'Content-Length': '0'} 2023-05-31T20:04:28.865 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000006b590100133f005a-006477a84c-1112-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:04:28 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:04:28.866 DEBUG:boto:b'pklxkd-18999falseobjnulltrue2023-05-31T20:03:53.140Zrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:04:28.867 DEBUG:boto:path=/pklxkd-18/ 2023-05-31T20:04:28.867 DEBUG:boto:auth_path=/pklxkd-18/ 2023-05-31T20:04:28.867 DEBUG:boto:path=/pklxkd-18/?versions&max-keys=999 2023-05-31T20:04:28.868 DEBUG:boto:auth_path=/pklxkd-18/?versions&max-keys=999 2023-05-31T20:04:28.868 DEBUG:boto:Method: GET 2023-05-31T20:04:28.869 DEBUG:boto:Path: /pklxkd-18/?versions&max-keys=999 2023-05-31T20:04:28.870 DEBUG:boto:Data: 2023-05-31T20:04:28.870 DEBUG:boto:Headers: {} 2023-05-31T20:04:28.871 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:04:28.871 DEBUG:boto:Port: 8000 2023-05-31T20:04:28.872 DEBUG:boto:Params: {} 2023-05-31T20:04:28.873 DEBUG:boto:Token: None 2023-05-31T20:04:28.873 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:04:28 GMT /pklxkd-18/?versions 2023-05-31T20:04:28.874 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:OROp2OtuHwtOWuVVs65F01y3reM= 2023-05-31T20:04:28.875 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:04:28 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:OROp2OtuHwtOWuVVs65F01y3reM=', 'Content-Length': '0'} 2023-05-31T20:04:28.877 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000aae790c266061c37-006477a84c-110e-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:04:28 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:04:28.924 DEBUG:boto:b'pklxkd-18999falseobjnulltrue2023-05-31T20:03:53.140Zrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:04:28.925 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2023-05-31T20:04:28.926 DEBUG:boto:path=/pklxkd-18/obj 2023-05-31T20:04:28.926 DEBUG:boto:auth_path=/pklxkd-18/obj 2023-05-31T20:04:28.927 DEBUG:boto:Method: HEAD 2023-05-31T20:04:28.927 DEBUG:boto:Path: /pklxkd-18/obj 2023-05-31T20:04:28.928 DEBUG:boto:Data: 2023-05-31T20:04:28.928 DEBUG:boto:Headers: {} 2023-05-31T20:04:28.928 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:04:28.929 DEBUG:boto:Port: 8001 2023-05-31T20:04:28.929 DEBUG:boto:Params: {} 2023-05-31T20:04:28.929 DEBUG:boto:Token: None 2023-05-31T20:04:28.930 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:04:28 GMT /pklxkd-18/obj 2023-05-31T20:04:28.931 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:x3nB2InmgXIw4eb7wzT7EA2zxpg= 2023-05-31T20:04:28.931 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:04:28 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:x3nB2InmgXIw4eb7wzT7EA2zxpg=', 'Content-Length': '0'} 2023-05-31T20:04:28.933 DEBUG:boto:Response headers: [('Content-Length', '219'), ('x-amz-request-id', 'tx000002ba0ed36f5e866f2-006477a84c-1112-a2'), ('Accept-Ranges', 'bytes'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:04:28 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:04:28.933 DEBUG:boto:path=/pklxkd-18/obj 2023-05-31T20:04:28.934 DEBUG:boto:auth_path=/pklxkd-18/obj 2023-05-31T20:04:28.934 DEBUG:boto:Method: HEAD 2023-05-31T20:04:28.935 DEBUG:boto:Path: /pklxkd-18/obj 2023-05-31T20:04:28.935 DEBUG:boto:Data: 2023-05-31T20:04:28.936 DEBUG:boto:Headers: {} 2023-05-31T20:04:28.936 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:04:28.937 DEBUG:boto:Port: 8000 2023-05-31T20:04:28.937 DEBUG:boto:Params: {} 2023-05-31T20:04:28.938 DEBUG:boto:Token: None 2023-05-31T20:04:28.938 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:04:28 GMT /pklxkd-18/obj 2023-05-31T20:04:28.938 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:x3nB2InmgXIw4eb7wzT7EA2zxpg= 2023-05-31T20:04:28.939 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:04:28 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:x3nB2InmgXIw4eb7wzT7EA2zxpg=', 'Content-Length': '0'} 2023-05-31T20:04:28.940 DEBUG:boto:Response headers: [('Content-Length', '219'), ('x-amz-request-id', 'tx00000271e3c7a47b094af-006477a84c-110e-a1'), ('Accept-Ranges', 'bytes'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:04:28 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:04:28.941 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=pklxkd-18 zones={a2, a1} 2023-05-31T20:04:28.941 INFO:tasks.rgw_multisite_tests:rgw_multi.tests.test_suspended_delete_marker_full_sync ... ok 2023-05-31T20:04:28.942 INFO:rgw_multi.tests:create bucket zone=a1 name=pklxkd-19 2023-05-31T20:04:28.943 DEBUG:boto:path=/pklxkd-19/ 2023-05-31T20:04:28.943 DEBUG:boto:auth_path=/pklxkd-19/ 2023-05-31T20:04:28.943 DEBUG:boto:Method: PUT 2023-05-31T20:04:28.944 DEBUG:boto:Path: /pklxkd-19/ 2023-05-31T20:04:28.944 DEBUG:boto:Data: 2023-05-31T20:04:28.945 DEBUG:boto:Headers: {} 2023-05-31T20:04:28.945 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:04:28.945 DEBUG:boto:Port: 8000 2023-05-31T20:04:28.946 DEBUG:boto:Params: {} 2023-05-31T20:04:28.946 DEBUG:boto:Token: None 2023-05-31T20:04:28.947 DEBUG:boto:StringToSign: PUT Wed, 31 May 2023 20:04:28 GMT /pklxkd-19/ 2023-05-31T20:04:28.948 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:Jnb3O7dcrh+1Qcg12brUZp42E20= 2023-05-31T20:04:28.949 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:04:28 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:Jnb3O7dcrh+1Qcg12brUZp42E20=', 'Content-Length': '0'} 2023-05-31T20:04:28.962 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx000009db33682cab5382b-006477a84c-110e-a1'), ('Content-Length', '0'), ('Date', 'Wed, 31 May 2023 20:04:28 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:04:28.963 INFO:rgw_multi.tests:create bucket zone=a2 name=pklxkd-20 2023-05-31T20:04:28.964 DEBUG:boto:path=/pklxkd-20/ 2023-05-31T20:04:28.964 DEBUG:boto:auth_path=/pklxkd-20/ 2023-05-31T20:04:28.965 DEBUG:boto:Method: PUT 2023-05-31T20:04:28.965 DEBUG:boto:Path: /pklxkd-20/ 2023-05-31T20:04:28.966 DEBUG:boto:Data: 2023-05-31T20:04:28.966 DEBUG:boto:Headers: {} 2023-05-31T20:04:28.966 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:04:28.967 DEBUG:boto:Port: 8001 2023-05-31T20:04:28.967 DEBUG:boto:Params: {} 2023-05-31T20:04:28.968 DEBUG:boto:Token: None 2023-05-31T20:04:28.968 DEBUG:boto:StringToSign: PUT Wed, 31 May 2023 20:04:28 GMT /pklxkd-20/ 2023-05-31T20:04:28.968 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:C1MoybyzSEodIYdsywjbtMY4DD8= 2023-05-31T20:04:28.969 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:04:28 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:C1MoybyzSEodIYdsywjbtMY4DD8=', 'Content-Length': '0'} 2023-05-31T20:04:29.005 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx000000edf617d9b75b7ae-006477a84c-1112-a2'), ('Content-Length', '0'), ('Date', 'Wed, 31 May 2023 20:04:28 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:04:29.007 DEBUG:boto:Using access key provided by client. 2023-05-31T20:04:29.008 DEBUG:boto:Using secret key provided by client. 2023-05-31T20:04:29.010 DEBUG:boto:Using access key provided by client. 2023-05-31T20:04:29.011 DEBUG:boto:Using secret key provided by client. 2023-05-31T20:04:29.012 DEBUG:boto:Using access key provided by client. 2023-05-31T20:04:29.013 DEBUG:boto:Using secret key provided by client. 2023-05-31T20:04:29.014 DEBUG:boto:Using access key provided by client. 2023-05-31T20:04:29.015 DEBUG:boto:Using secret key provided by client. 2023-05-31T20:04:29.016 INFO:rgw_multi.tests:create bucket zone=b1 name=pklxkd-21 2023-05-31T20:04:29.017 DEBUG:boto:path=/pklxkd-21/ 2023-05-31T20:04:29.018 DEBUG:boto:auth_path=/pklxkd-21/ 2023-05-31T20:04:29.018 DEBUG:boto:Method: PUT 2023-05-31T20:04:29.019 DEBUG:boto:Path: /pklxkd-21/ 2023-05-31T20:04:29.019 DEBUG:boto:Data: 2023-05-31T20:04:29.020 DEBUG:boto:Headers: {} 2023-05-31T20:04:29.020 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-05-31T20:04:29.020 DEBUG:boto:Port: 8000 2023-05-31T20:04:29.021 DEBUG:boto:Params: {} 2023-05-31T20:04:29.021 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2023-05-31T20:04:29.022 DEBUG:boto:Token: None 2023-05-31T20:04:29.023 DEBUG:boto:StringToSign: PUT Wed, 31 May 2023 20:04:29 GMT /pklxkd-21/ 2023-05-31T20:04:29.023 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:F1jLVEtoM8mewj0z05tL96TOWO4= 2023-05-31T20:04:29.024 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:04:29 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:F1jLVEtoM8mewj0z05tL96TOWO4=', 'Content-Length': '0'} 2023-05-31T20:04:29.054 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000f08714b4a6a82287-006477a84d-10fb-b1'), ('Content-Length', '0'), ('Date', 'Wed, 31 May 2023 20:04:29 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:04:29.054 INFO:rgw_multi.tests:create bucket zone=b2 name=pklxkd-22 2023-05-31T20:04:29.055 DEBUG:boto:path=/pklxkd-22/ 2023-05-31T20:04:29.055 DEBUG:boto:auth_path=/pklxkd-22/ 2023-05-31T20:04:29.056 DEBUG:boto:Method: PUT 2023-05-31T20:04:29.056 DEBUG:boto:Path: /pklxkd-22/ 2023-05-31T20:04:29.056 DEBUG:boto:Data: 2023-05-31T20:04:29.057 DEBUG:boto:Headers: {} 2023-05-31T20:04:29.057 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-05-31T20:04:29.057 DEBUG:boto:Port: 8001 2023-05-31T20:04:29.058 DEBUG:boto:Params: {} 2023-05-31T20:04:29.058 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8001} 2023-05-31T20:04:29.058 DEBUG:boto:Token: None 2023-05-31T20:04:29.059 DEBUG:boto:StringToSign: PUT Wed, 31 May 2023 20:04:29 GMT /pklxkd-22/ 2023-05-31T20:04:29.059 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:+/pftDIPzOpc9HW1A/0PYSL8cAo= 2023-05-31T20:04:29.060 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:04:29 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:+/pftDIPzOpc9HW1A/0PYSL8cAo=', 'Content-Length': '0'} 2023-05-31T20:04:29.100 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx000004a563ee89f207b8d-006477a84d-1101-b2'), ('Content-Length', '0'), ('Date', 'Wed, 31 May 2023 20:04:29 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:04:29.101 DEBUG:boto:path=/pklxkd-19/ 2023-05-31T20:04:29.101 DEBUG:boto:auth_path=/pklxkd-19/ 2023-05-31T20:04:29.102 DEBUG:boto:path=/pklxkd-19/?versioning 2023-05-31T20:04:29.102 DEBUG:boto:auth_path=/pklxkd-19/?versioning 2023-05-31T20:04:29.102 DEBUG:boto:Method: PUT 2023-05-31T20:04:29.103 DEBUG:boto:Path: /pklxkd-19/?versioning 2023-05-31T20:04:29.103 DEBUG:boto:Data: Enabled Disabled 2023-05-31T20:04:29.103 DEBUG:boto:Headers: {} 2023-05-31T20:04:29.104 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:04:29.104 DEBUG:boto:Port: 8000 2023-05-31T20:04:29.105 DEBUG:boto:Params: {} 2023-05-31T20:04:29.105 DEBUG:boto:Token: None 2023-05-31T20:04:29.105 DEBUG:boto:StringToSign: PUT Wed, 31 May 2023 20:04:29 GMT /pklxkd-19/?versioning 2023-05-31T20:04:29.106 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:ctSG/HLrfpZUwvoA02siR2GIWQ8= 2023-05-31T20:04:29.106 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:04:29 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:ctSG/HLrfpZUwvoA02siR2GIWQ8=', 'Content-Length': '228'} 2023-05-31T20:04:29.111 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000e79112712c62e753-006477a84d-110e-a1'), ('Content-Type', 'application/xml'), ('Content-Length', '0'), ('Date', 'Wed, 31 May 2023 20:04:29 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:04:29.111 DEBUG:boto:path=/pklxkd-19/ 2023-05-31T20:04:29.112 DEBUG:boto:auth_path=/pklxkd-19/ 2023-05-31T20:04:29.112 DEBUG:boto:path=/pklxkd-19/?versioning 2023-05-31T20:04:29.113 DEBUG:boto:auth_path=/pklxkd-19/?versioning 2023-05-31T20:04:29.113 DEBUG:boto:Method: GET 2023-05-31T20:04:29.113 DEBUG:boto:Path: /pklxkd-19/?versioning 2023-05-31T20:04:29.114 DEBUG:boto:Data: 2023-05-31T20:04:29.114 DEBUG:boto:Headers: {} 2023-05-31T20:04:29.114 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:04:29.115 DEBUG:boto:Port: 8000 2023-05-31T20:04:29.115 DEBUG:boto:Params: {} 2023-05-31T20:04:29.115 DEBUG:boto:Token: None 2023-05-31T20:04:29.116 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:04:29 GMT /pklxkd-19/?versioning 2023-05-31T20:04:29.116 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:KIH+9Yd6lFnuMh+xonKVrLb1Crg= 2023-05-31T20:04:29.117 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:04:29 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:KIH+9Yd6lFnuMh+xonKVrLb1Crg=', 'Content-Length': '0'} 2023-05-31T20:04:29.118 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx000001bd98874ddb5af09-006477a84d-110e-a1'), ('Content-Type', 'application/xml'), ('Content-Length', '192'), ('Date', 'Wed, 31 May 2023 20:04:29 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:04:29.160 DEBUG:boto:EnabledDisabled 2023-05-31T20:04:29.161 DEBUG:boto:path=/pklxkd-20/ 2023-05-31T20:04:29.162 DEBUG:boto:auth_path=/pklxkd-20/ 2023-05-31T20:04:29.162 DEBUG:boto:path=/pklxkd-20/?versioning 2023-05-31T20:04:29.162 DEBUG:boto:auth_path=/pklxkd-20/?versioning 2023-05-31T20:04:29.163 DEBUG:boto:Method: PUT 2023-05-31T20:04:29.163 DEBUG:boto:Path: /pklxkd-20/?versioning 2023-05-31T20:04:29.164 DEBUG:boto:Data: Enabled Disabled 2023-05-31T20:04:29.164 DEBUG:boto:Headers: {} 2023-05-31T20:04:29.164 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:04:29.165 DEBUG:boto:Port: 8001 2023-05-31T20:04:29.165 DEBUG:boto:Params: {} 2023-05-31T20:04:29.166 DEBUG:boto:Token: None 2023-05-31T20:04:29.166 DEBUG:boto:StringToSign: PUT Wed, 31 May 2023 20:04:29 GMT /pklxkd-20/?versioning 2023-05-31T20:04:29.167 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:iVFtILPCvvDb6yoj6X0lz4cX0sw= 2023-05-31T20:04:29.169 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:04:29 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:iVFtILPCvvDb6yoj6X0lz4cX0sw=', 'Content-Length': '228'} 2023-05-31T20:04:29.186 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx000008456af679cd2fdb4-006477a84d-1112-a2'), ('Content-Type', 'application/xml'), ('Content-Length', '0'), ('Date', 'Wed, 31 May 2023 20:04:29 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:04:29.187 DEBUG:boto:path=/pklxkd-20/ 2023-05-31T20:04:29.187 DEBUG:boto:auth_path=/pklxkd-20/ 2023-05-31T20:04:29.188 DEBUG:boto:path=/pklxkd-20/?versioning 2023-05-31T20:04:29.188 DEBUG:boto:auth_path=/pklxkd-20/?versioning 2023-05-31T20:04:29.189 DEBUG:boto:Method: GET 2023-05-31T20:04:29.190 DEBUG:boto:Path: /pklxkd-20/?versioning 2023-05-31T20:04:29.190 DEBUG:boto:Data: 2023-05-31T20:04:29.191 DEBUG:boto:Headers: {} 2023-05-31T20:04:29.191 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:04:29.192 DEBUG:boto:Port: 8001 2023-05-31T20:04:29.193 DEBUG:boto:Params: {} 2023-05-31T20:04:29.193 DEBUG:boto:Token: None 2023-05-31T20:04:29.194 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:04:29 GMT /pklxkd-20/?versioning 2023-05-31T20:04:29.194 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:k/szMdBKRWHYe1hrHj0930ufyyc= 2023-05-31T20:04:29.195 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:04:29 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:k/szMdBKRWHYe1hrHj0930ufyyc=', 'Content-Length': '0'} 2023-05-31T20:04:29.197 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx000007b584be9566034c2-006477a84d-1112-a2'), ('Content-Type', 'application/xml'), ('Content-Length', '192'), ('Date', 'Wed, 31 May 2023 20:04:29 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:04:29.197 DEBUG:boto:EnabledDisabled 2023-05-31T20:04:29.198 DEBUG:boto:path=/pklxkd-21/ 2023-05-31T20:04:29.199 DEBUG:boto:auth_path=/pklxkd-21/ 2023-05-31T20:04:29.200 DEBUG:boto:path=/pklxkd-21/?versioning 2023-05-31T20:04:29.201 DEBUG:boto:auth_path=/pklxkd-21/?versioning 2023-05-31T20:04:29.201 DEBUG:boto:Method: PUT 2023-05-31T20:04:29.202 DEBUG:boto:Path: /pklxkd-21/?versioning 2023-05-31T20:04:29.203 DEBUG:boto:Data: Enabled Disabled 2023-05-31T20:04:29.204 DEBUG:boto:Headers: {} 2023-05-31T20:04:29.204 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-05-31T20:04:29.205 DEBUG:boto:Port: 8000 2023-05-31T20:04:29.205 DEBUG:boto:Params: {} 2023-05-31T20:04:29.206 DEBUG:boto:Token: None 2023-05-31T20:04:29.207 DEBUG:boto:StringToSign: PUT Wed, 31 May 2023 20:04:29 GMT /pklxkd-21/?versioning 2023-05-31T20:04:29.207 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:EYhAnyiDJxTbWO/HHAgBCaQg6LI= 2023-05-31T20:04:29.208 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:04:29 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:EYhAnyiDJxTbWO/HHAgBCaQg6LI=', 'Content-Length': '228'} 2023-05-31T20:04:29.216 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000eacf73d0883eeec2-006477a84d-10fb-b1'), ('Content-Type', 'application/xml'), ('Content-Length', '0'), ('Date', 'Wed, 31 May 2023 20:04:29 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:04:29.217 DEBUG:boto:path=/pklxkd-21/ 2023-05-31T20:04:29.217 DEBUG:boto:auth_path=/pklxkd-21/ 2023-05-31T20:04:29.217 DEBUG:boto:path=/pklxkd-21/?versioning 2023-05-31T20:04:29.218 DEBUG:boto:auth_path=/pklxkd-21/?versioning 2023-05-31T20:04:29.218 DEBUG:boto:Method: GET 2023-05-31T20:04:29.218 DEBUG:boto:Path: /pklxkd-21/?versioning 2023-05-31T20:04:29.219 DEBUG:boto:Data: 2023-05-31T20:04:29.219 DEBUG:boto:Headers: {} 2023-05-31T20:04:29.219 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-05-31T20:04:29.220 DEBUG:boto:Port: 8000 2023-05-31T20:04:29.220 DEBUG:boto:Params: {} 2023-05-31T20:04:29.220 DEBUG:boto:Token: None 2023-05-31T20:04:29.221 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:04:29 GMT /pklxkd-21/?versioning 2023-05-31T20:04:29.221 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:cqlDyfpkvcJMfD2GeTzWCk9Lyhs= 2023-05-31T20:04:29.222 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:04:29 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:cqlDyfpkvcJMfD2GeTzWCk9Lyhs=', 'Content-Length': '0'} 2023-05-31T20:04:29.223 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000c66d26b6b65ff046-006477a84d-10fb-b1'), ('Content-Type', 'application/xml'), ('Content-Length', '192'), ('Date', 'Wed, 31 May 2023 20:04:29 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:04:29.223 DEBUG:boto:EnabledDisabled 2023-05-31T20:04:29.224 DEBUG:boto:path=/pklxkd-22/ 2023-05-31T20:04:29.224 DEBUG:boto:auth_path=/pklxkd-22/ 2023-05-31T20:04:29.225 DEBUG:boto:path=/pklxkd-22/?versioning 2023-05-31T20:04:29.225 DEBUG:boto:auth_path=/pklxkd-22/?versioning 2023-05-31T20:04:29.225 DEBUG:boto:Method: PUT 2023-05-31T20:04:29.226 DEBUG:boto:Path: /pklxkd-22/?versioning 2023-05-31T20:04:29.226 DEBUG:boto:Data: Enabled Disabled 2023-05-31T20:04:29.227 DEBUG:boto:Headers: {} 2023-05-31T20:04:29.227 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-05-31T20:04:29.227 DEBUG:boto:Port: 8001 2023-05-31T20:04:29.228 DEBUG:boto:Params: {} 2023-05-31T20:04:29.228 DEBUG:boto:Token: None 2023-05-31T20:04:29.229 DEBUG:boto:StringToSign: PUT Wed, 31 May 2023 20:04:29 GMT /pklxkd-22/?versioning 2023-05-31T20:04:29.229 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:AYVlJ2a7PrlDd8oqpkyiOV9os3A= 2023-05-31T20:04:29.229 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:04:29 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:AYVlJ2a7PrlDd8oqpkyiOV9os3A=', 'Content-Length': '228'} 2023-05-31T20:04:29.238 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx000002d90e125cf183905-006477a84d-1101-b2'), ('Content-Type', 'application/xml'), ('Content-Length', '0'), ('Date', 'Wed, 31 May 2023 20:04:29 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:04:29.239 DEBUG:boto:path=/pklxkd-22/ 2023-05-31T20:04:29.239 DEBUG:boto:auth_path=/pklxkd-22/ 2023-05-31T20:04:29.240 DEBUG:boto:path=/pklxkd-22/?versioning 2023-05-31T20:04:29.240 DEBUG:boto:auth_path=/pklxkd-22/?versioning 2023-05-31T20:04:29.240 DEBUG:boto:Method: GET 2023-05-31T20:04:29.241 DEBUG:boto:Path: /pklxkd-22/?versioning 2023-05-31T20:04:29.241 DEBUG:boto:Data: 2023-05-31T20:04:29.241 DEBUG:boto:Headers: {} 2023-05-31T20:04:29.242 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-05-31T20:04:29.242 DEBUG:boto:Port: 8001 2023-05-31T20:04:29.242 DEBUG:boto:Params: {} 2023-05-31T20:04:29.243 DEBUG:boto:Token: None 2023-05-31T20:04:29.243 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:04:29 GMT /pklxkd-22/?versioning 2023-05-31T20:04:29.244 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:VLsW7j/wporIqDddGzSnZNJUJSM= 2023-05-31T20:04:29.244 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:04:29 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:VLsW7j/wporIqDddGzSnZNJUJSM=', 'Content-Length': '0'} 2023-05-31T20:04:29.245 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000d0b1951a32a4ce11-006477a84d-1101-b2'), ('Content-Type', 'application/xml'), ('Content-Length', '192'), ('Date', 'Wed, 31 May 2023 20:04:29 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:04:29.246 DEBUG:boto:EnabledDisabled 2023-05-31T20:04:29.246 INFO:tasks.rgw_multisite_tests:rgw_multi.tests.test_bucket_versioning ... ok 2023-05-31T20:04:29.247 INFO:rgw_multi.tests:create bucket zone=a1 name=pklxkd-23 2023-05-31T20:04:29.247 DEBUG:boto:path=/pklxkd-23/ 2023-05-31T20:04:29.248 DEBUG:boto:auth_path=/pklxkd-23/ 2023-05-31T20:04:29.248 DEBUG:boto:Method: PUT 2023-05-31T20:04:29.249 DEBUG:boto:Path: /pklxkd-23/ 2023-05-31T20:04:29.249 DEBUG:boto:Data: 2023-05-31T20:04:29.249 DEBUG:boto:Headers: {} 2023-05-31T20:04:29.250 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:04:29.250 DEBUG:boto:Port: 8000 2023-05-31T20:04:29.250 DEBUG:boto:Params: {} 2023-05-31T20:04:29.251 DEBUG:boto:Token: None 2023-05-31T20:04:29.251 DEBUG:boto:StringToSign: PUT Wed, 31 May 2023 20:04:29 GMT /pklxkd-23/ 2023-05-31T20:04:29.251 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:1yiEAwq1+tCoT2puT5bzg7SP8GY= 2023-05-31T20:04:29.252 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:04:29 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:1yiEAwq1+tCoT2puT5bzg7SP8GY=', 'Content-Length': '0'} 2023-05-31T20:04:29.288 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000b013cf9888f7cd61-006477a84d-110e-a1'), ('Content-Length', '0'), ('Date', 'Wed, 31 May 2023 20:04:29 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:04:29.289 INFO:rgw_multi.tests:create bucket zone=a2 name=pklxkd-24 2023-05-31T20:04:29.289 DEBUG:boto:path=/pklxkd-24/ 2023-05-31T20:04:29.290 DEBUG:boto:auth_path=/pklxkd-24/ 2023-05-31T20:04:29.290 DEBUG:boto:Method: PUT 2023-05-31T20:04:29.290 DEBUG:boto:Path: /pklxkd-24/ 2023-05-31T20:04:29.291 DEBUG:boto:Data: 2023-05-31T20:04:29.291 DEBUG:boto:Headers: {} 2023-05-31T20:04:29.291 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:04:29.292 DEBUG:boto:Port: 8001 2023-05-31T20:04:29.292 DEBUG:boto:Params: {} 2023-05-31T20:04:29.293 DEBUG:boto:Token: None 2023-05-31T20:04:29.293 DEBUG:boto:StringToSign: PUT Wed, 31 May 2023 20:04:29 GMT /pklxkd-24/ 2023-05-31T20:04:29.293 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:R6qIagnpWWjU5FXCKxBqK2iGpp4= 2023-05-31T20:04:29.294 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:04:29 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:R6qIagnpWWjU5FXCKxBqK2iGpp4=', 'Content-Length': '0'} 2023-05-31T20:04:29.358 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000e3578ce8046d2e64-006477a84d-1112-a2'), ('Content-Length', '0'), ('Date', 'Wed, 31 May 2023 20:04:29 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:04:29.359 INFO:rgw_multi.tests:create bucket zone=b1 name=pklxkd-25 2023-05-31T20:04:29.359 DEBUG:boto:path=/pklxkd-25/ 2023-05-31T20:04:29.360 DEBUG:boto:auth_path=/pklxkd-25/ 2023-05-31T20:04:29.360 DEBUG:boto:Method: PUT 2023-05-31T20:04:29.360 DEBUG:boto:Path: /pklxkd-25/ 2023-05-31T20:04:29.361 DEBUG:boto:Data: 2023-05-31T20:04:29.361 DEBUG:boto:Headers: {} 2023-05-31T20:04:29.362 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-05-31T20:04:29.362 DEBUG:boto:Port: 8000 2023-05-31T20:04:29.363 DEBUG:boto:Params: {} 2023-05-31T20:04:29.363 DEBUG:boto:Token: None 2023-05-31T20:04:29.364 DEBUG:boto:StringToSign: PUT Wed, 31 May 2023 20:04:29 GMT /pklxkd-25/ 2023-05-31T20:04:29.365 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:USHBWBTciEonWt/Wzu2H7KlUw1g= 2023-05-31T20:04:29.366 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:04:29 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:USHBWBTciEonWt/Wzu2H7KlUw1g=', 'Content-Length': '0'} 2023-05-31T20:04:29.397 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx000003d14979cb189ec55-006477a84d-10fb-b1'), ('Content-Length', '0'), ('Date', 'Wed, 31 May 2023 20:04:29 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:04:29.397 INFO:rgw_multi.tests:create bucket zone=b2 name=pklxkd-26 2023-05-31T20:04:29.398 DEBUG:boto:path=/pklxkd-26/ 2023-05-31T20:04:29.398 DEBUG:boto:auth_path=/pklxkd-26/ 2023-05-31T20:04:29.398 DEBUG:boto:Method: PUT 2023-05-31T20:04:29.399 DEBUG:boto:Path: /pklxkd-26/ 2023-05-31T20:04:29.399 DEBUG:boto:Data: 2023-05-31T20:04:29.399 DEBUG:boto:Headers: {} 2023-05-31T20:04:29.400 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-05-31T20:04:29.400 DEBUG:boto:Port: 8001 2023-05-31T20:04:29.400 DEBUG:boto:Params: {} 2023-05-31T20:04:29.401 DEBUG:boto:Token: None 2023-05-31T20:04:29.401 DEBUG:boto:StringToSign: PUT Wed, 31 May 2023 20:04:29 GMT /pklxkd-26/ 2023-05-31T20:04:29.402 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:Bkg0fuY2k4vCxinC3EEGU5m2xxo= 2023-05-31T20:04:29.402 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:04:29 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:Bkg0fuY2k4vCxinC3EEGU5m2xxo=', 'Content-Length': '0'} 2023-05-31T20:04:29.430 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx000007ac78af619cc14f0-006477a84d-1101-b2'), ('Content-Length', '0'), ('Date', 'Wed, 31 May 2023 20:04:29 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:04:29.430 DEBUG:boto:path=/pklxkd-23/ 2023-05-31T20:04:29.431 DEBUG:boto:auth_path=/pklxkd-23/ 2023-05-31T20:04:29.431 DEBUG:boto:path=/pklxkd-23/?acl 2023-05-31T20:04:29.431 DEBUG:boto:auth_path=/pklxkd-23/?acl 2023-05-31T20:04:29.432 DEBUG:boto:Method: GET 2023-05-31T20:04:29.432 DEBUG:boto:Path: /pklxkd-23/?acl 2023-05-31T20:04:29.432 DEBUG:boto:Data: 2023-05-31T20:04:29.433 DEBUG:boto:Headers: {} 2023-05-31T20:04:29.433 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:04:29.433 DEBUG:boto:Port: 8000 2023-05-31T20:04:29.434 DEBUG:boto:Params: {} 2023-05-31T20:04:29.434 DEBUG:boto:Token: None 2023-05-31T20:04:29.435 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:04:29 GMT /pklxkd-23/?acl 2023-05-31T20:04:29.435 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:SqMpQUgtvc/cT+qsU2dCBCAbEP4= 2023-05-31T20:04:29.436 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:04:29 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:SqMpQUgtvc/cT+qsU2dCBCAbEP4=', 'Content-Length': '0'} 2023-05-31T20:04:29.437 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx0000005500f1d80e0be20-006477a84d-110e-a1'), ('Content-Type', 'application/xml'), ('Content-Length', '489'), ('Date', 'Wed, 31 May 2023 20:04:29 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:04:29.438 DEBUG:boto:path=/pklxkd-23/ 2023-05-31T20:04:29.438 DEBUG:boto:auth_path=/pklxkd-23/ 2023-05-31T20:04:29.439 DEBUG:boto:path=/pklxkd-23/?acl 2023-05-31T20:04:29.439 DEBUG:boto:auth_path=/pklxkd-23/?acl 2023-05-31T20:04:29.439 DEBUG:boto:Method: PUT 2023-05-31T20:04:29.440 DEBUG:boto:Path: /pklxkd-23/?acl 2023-05-31T20:04:29.440 DEBUG:boto:Data: 2023-05-31T20:04:29.441 DEBUG:boto:Headers: {'x-amz-acl': 'public-read'} 2023-05-31T20:04:29.441 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:04:29.441 DEBUG:boto:Port: 8000 2023-05-31T20:04:29.442 DEBUG:boto:Params: {} 2023-05-31T20:04:29.442 DEBUG:boto:Token: None 2023-05-31T20:04:29.442 DEBUG:boto:StringToSign: PUT Wed, 31 May 2023 20:04:29 GMT x-amz-acl:public-read /pklxkd-23/?acl 2023-05-31T20:04:29.443 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:TWxZbpk9HHhikfkAziWTj++optc= 2023-05-31T20:04:29.443 DEBUG:boto:Final headers: {'x-amz-acl': 'public-read', 'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:04:29 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:TWxZbpk9HHhikfkAziWTj++optc=', 'Content-Length': '0'} 2023-05-31T20:04:29.448 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx000000f54eab662a78da1-006477a84d-110e-a1'), ('Content-Type', 'application/xml'), ('Content-Length', '0'), ('Date', 'Wed, 31 May 2023 20:04:29 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:04:29.449 DEBUG:boto:path=/pklxkd-23/ 2023-05-31T20:04:29.449 DEBUG:boto:auth_path=/pklxkd-23/ 2023-05-31T20:04:29.449 DEBUG:boto:path=/pklxkd-23/?acl 2023-05-31T20:04:29.450 DEBUG:boto:auth_path=/pklxkd-23/?acl 2023-05-31T20:04:29.450 DEBUG:boto:Method: GET 2023-05-31T20:04:29.451 DEBUG:boto:Path: /pklxkd-23/?acl 2023-05-31T20:04:29.451 DEBUG:boto:Data: 2023-05-31T20:04:29.452 DEBUG:boto:Headers: {} 2023-05-31T20:04:29.453 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:04:29.453 DEBUG:boto:Port: 8000 2023-05-31T20:04:29.454 DEBUG:boto:Params: {} 2023-05-31T20:04:29.454 DEBUG:boto:Token: None 2023-05-31T20:04:29.455 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:04:29 GMT /pklxkd-23/?acl 2023-05-31T20:04:29.456 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:SqMpQUgtvc/cT+qsU2dCBCAbEP4= 2023-05-31T20:04:29.457 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:04:29 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:SqMpQUgtvc/cT+qsU2dCBCAbEP4=', 'Content-Length': '0'} 2023-05-31T20:04:29.458 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000689699ea4c4cef29-006477a84d-110e-a1'), ('Content-Type', 'application/xml'), ('Content-Length', '681'), ('Date', 'Wed, 31 May 2023 20:04:29 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:04:29.459 DEBUG:boto:path=/pklxkd-24/ 2023-05-31T20:04:29.460 DEBUG:boto:auth_path=/pklxkd-24/ 2023-05-31T20:04:29.461 DEBUG:boto:path=/pklxkd-24/?acl 2023-05-31T20:04:29.461 DEBUG:boto:auth_path=/pklxkd-24/?acl 2023-05-31T20:04:29.462 DEBUG:boto:Method: GET 2023-05-31T20:04:29.462 DEBUG:boto:Path: /pklxkd-24/?acl 2023-05-31T20:04:29.462 DEBUG:boto:Data: 2023-05-31T20:04:29.463 DEBUG:boto:Headers: {} 2023-05-31T20:04:29.463 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:04:29.463 DEBUG:boto:Port: 8001 2023-05-31T20:04:29.464 DEBUG:boto:Params: {} 2023-05-31T20:04:29.464 DEBUG:boto:Token: None 2023-05-31T20:04:29.464 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:04:29 GMT /pklxkd-24/?acl 2023-05-31T20:04:29.465 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:YC7dHAFq/8ceMI5jMMvwwmPbJ/g= 2023-05-31T20:04:29.465 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:04:29 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:YC7dHAFq/8ceMI5jMMvwwmPbJ/g=', 'Content-Length': '0'} 2023-05-31T20:04:29.466 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx000001d8e137dbf0b7c63-006477a84d-1112-a2'), ('Content-Type', 'application/xml'), ('Content-Length', '489'), ('Date', 'Wed, 31 May 2023 20:04:29 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:04:29.467 DEBUG:boto:path=/pklxkd-24/ 2023-05-31T20:04:29.468 DEBUG:boto:auth_path=/pklxkd-24/ 2023-05-31T20:04:29.468 DEBUG:boto:path=/pklxkd-24/?acl 2023-05-31T20:04:29.468 DEBUG:boto:auth_path=/pklxkd-24/?acl 2023-05-31T20:04:29.469 DEBUG:boto:Method: PUT 2023-05-31T20:04:29.469 DEBUG:boto:Path: /pklxkd-24/?acl 2023-05-31T20:04:29.469 DEBUG:boto:Data: 2023-05-31T20:04:29.469 DEBUG:boto:Headers: {'x-amz-acl': 'public-read'} 2023-05-31T20:04:29.470 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:04:29.470 DEBUG:boto:Port: 8001 2023-05-31T20:04:29.470 DEBUG:boto:Params: {} 2023-05-31T20:04:29.471 DEBUG:boto:Token: None 2023-05-31T20:04:29.471 DEBUG:boto:StringToSign: PUT Wed, 31 May 2023 20:04:29 GMT x-amz-acl:public-read /pklxkd-24/?acl 2023-05-31T20:04:29.472 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:ln7fwgxjsJo/iTiQGif1elIHEuc= 2023-05-31T20:04:29.472 DEBUG:boto:Final headers: {'x-amz-acl': 'public-read', 'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:04:29 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:ln7fwgxjsJo/iTiQGif1elIHEuc=', 'Content-Length': '0'} 2023-05-31T20:04:29.479 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx0000091bad58cbb4b42c8-006477a84d-1112-a2'), ('Content-Type', 'application/xml'), ('Content-Length', '0'), ('Date', 'Wed, 31 May 2023 20:04:29 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:04:29.479 DEBUG:boto:path=/pklxkd-24/ 2023-05-31T20:04:29.480 DEBUG:boto:auth_path=/pklxkd-24/ 2023-05-31T20:04:29.480 DEBUG:boto:path=/pklxkd-24/?acl 2023-05-31T20:04:29.480 DEBUG:boto:auth_path=/pklxkd-24/?acl 2023-05-31T20:04:29.481 DEBUG:boto:Method: GET 2023-05-31T20:04:29.481 DEBUG:boto:Path: /pklxkd-24/?acl 2023-05-31T20:04:29.481 DEBUG:boto:Data: 2023-05-31T20:04:29.482 DEBUG:boto:Headers: {} 2023-05-31T20:04:29.482 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:04:29.482 DEBUG:boto:Port: 8001 2023-05-31T20:04:29.483 DEBUG:boto:Params: {} 2023-05-31T20:04:29.483 DEBUG:boto:Token: None 2023-05-31T20:04:29.483 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:04:29 GMT /pklxkd-24/?acl 2023-05-31T20:04:29.484 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:YC7dHAFq/8ceMI5jMMvwwmPbJ/g= 2023-05-31T20:04:29.484 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:04:29 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:YC7dHAFq/8ceMI5jMMvwwmPbJ/g=', 'Content-Length': '0'} 2023-05-31T20:04:29.485 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx000006b8f993d12ee3934-006477a84d-1112-a2'), ('Content-Type', 'application/xml'), ('Content-Length', '681'), ('Date', 'Wed, 31 May 2023 20:04:29 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:04:29.486 DEBUG:boto:path=/pklxkd-25/ 2023-05-31T20:04:29.486 DEBUG:boto:auth_path=/pklxkd-25/ 2023-05-31T20:04:29.486 DEBUG:boto:path=/pklxkd-25/?acl 2023-05-31T20:04:29.487 DEBUG:boto:auth_path=/pklxkd-25/?acl 2023-05-31T20:04:29.487 DEBUG:boto:Method: GET 2023-05-31T20:04:29.487 DEBUG:boto:Path: /pklxkd-25/?acl 2023-05-31T20:04:29.488 DEBUG:boto:Data: 2023-05-31T20:04:29.488 DEBUG:boto:Headers: {} 2023-05-31T20:04:29.488 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-05-31T20:04:29.489 DEBUG:boto:Port: 8000 2023-05-31T20:04:29.489 DEBUG:boto:Params: {} 2023-05-31T20:04:29.489 DEBUG:boto:Token: None 2023-05-31T20:04:29.490 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:04:29 GMT /pklxkd-25/?acl 2023-05-31T20:04:29.490 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:VP5UUKAYiOUT+uhuGsPo2hLbfeU= 2023-05-31T20:04:29.490 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:04:29 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:VP5UUKAYiOUT+uhuGsPo2hLbfeU=', 'Content-Length': '0'} 2023-05-31T20:04:29.492 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000b485cf7ba7dcf28e-006477a84d-10fb-b1'), ('Content-Type', 'application/xml'), ('Content-Length', '489'), ('Date', 'Wed, 31 May 2023 20:04:29 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:04:29.492 DEBUG:boto:path=/pklxkd-25/ 2023-05-31T20:04:29.493 DEBUG:boto:auth_path=/pklxkd-25/ 2023-05-31T20:04:29.493 DEBUG:boto:path=/pklxkd-25/?acl 2023-05-31T20:04:29.493 DEBUG:boto:auth_path=/pklxkd-25/?acl 2023-05-31T20:04:29.494 DEBUG:boto:Method: PUT 2023-05-31T20:04:29.494 DEBUG:boto:Path: /pklxkd-25/?acl 2023-05-31T20:04:29.494 DEBUG:boto:Data: 2023-05-31T20:04:29.495 DEBUG:boto:Headers: {'x-amz-acl': 'public-read'} 2023-05-31T20:04:29.495 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-05-31T20:04:29.495 DEBUG:boto:Port: 8000 2023-05-31T20:04:29.496 DEBUG:boto:Params: {} 2023-05-31T20:04:29.496 DEBUG:boto:Token: None 2023-05-31T20:04:29.496 DEBUG:boto:StringToSign: PUT Wed, 31 May 2023 20:04:29 GMT x-amz-acl:public-read /pklxkd-25/?acl 2023-05-31T20:04:29.497 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:jcPPbZyVPvd8z77XHdRQdV3LRJk= 2023-05-31T20:04:29.497 DEBUG:boto:Final headers: {'x-amz-acl': 'public-read', 'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:04:29 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:jcPPbZyVPvd8z77XHdRQdV3LRJk=', 'Content-Length': '0'} 2023-05-31T20:04:29.505 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000777a5cbba6d4c524-006477a84d-10fb-b1'), ('Content-Type', 'application/xml'), ('Content-Length', '0'), ('Date', 'Wed, 31 May 2023 20:04:29 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:04:29.506 DEBUG:boto:path=/pklxkd-25/ 2023-05-31T20:04:29.506 DEBUG:boto:auth_path=/pklxkd-25/ 2023-05-31T20:04:29.507 DEBUG:boto:path=/pklxkd-25/?acl 2023-05-31T20:04:29.507 DEBUG:boto:auth_path=/pklxkd-25/?acl 2023-05-31T20:04:29.507 DEBUG:boto:Method: GET 2023-05-31T20:04:29.507 DEBUG:boto:Path: /pklxkd-25/?acl 2023-05-31T20:04:29.508 DEBUG:boto:Data: 2023-05-31T20:04:29.508 DEBUG:boto:Headers: {} 2023-05-31T20:04:29.508 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-05-31T20:04:29.509 DEBUG:boto:Port: 8000 2023-05-31T20:04:29.509 DEBUG:boto:Params: {} 2023-05-31T20:04:29.509 DEBUG:boto:Token: None 2023-05-31T20:04:29.510 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:04:29 GMT /pklxkd-25/?acl 2023-05-31T20:04:29.510 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:VP5UUKAYiOUT+uhuGsPo2hLbfeU= 2023-05-31T20:04:29.511 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:04:29 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:VP5UUKAYiOUT+uhuGsPo2hLbfeU=', 'Content-Length': '0'} 2023-05-31T20:04:29.512 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000dcbfdf02fe9872c9-006477a84d-10fb-b1'), ('Content-Type', 'application/xml'), ('Content-Length', '681'), ('Date', 'Wed, 31 May 2023 20:04:29 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:04:29.512 DEBUG:boto:path=/pklxkd-26/ 2023-05-31T20:04:29.513 DEBUG:boto:auth_path=/pklxkd-26/ 2023-05-31T20:04:29.513 DEBUG:boto:path=/pklxkd-26/?acl 2023-05-31T20:04:29.513 DEBUG:boto:auth_path=/pklxkd-26/?acl 2023-05-31T20:04:29.514 DEBUG:boto:Method: GET 2023-05-31T20:04:29.514 DEBUG:boto:Path: /pklxkd-26/?acl 2023-05-31T20:04:29.514 DEBUG:boto:Data: 2023-05-31T20:04:29.515 DEBUG:boto:Headers: {} 2023-05-31T20:04:29.515 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-05-31T20:04:29.515 DEBUG:boto:Port: 8001 2023-05-31T20:04:29.516 DEBUG:boto:Params: {} 2023-05-31T20:04:29.516 DEBUG:boto:Token: None 2023-05-31T20:04:29.516 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:04:29 GMT /pklxkd-26/?acl 2023-05-31T20:04:29.517 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:Us1IsvHGgAYXvIe3QcQV0fLlYGo= 2023-05-31T20:04:29.517 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:04:29 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:Us1IsvHGgAYXvIe3QcQV0fLlYGo=', 'Content-Length': '0'} 2023-05-31T20:04:29.518 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx0000095067c11e55f779b-006477a84d-1101-b2'), ('Content-Type', 'application/xml'), ('Content-Length', '489'), ('Date', 'Wed, 31 May 2023 20:04:29 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:04:29.519 DEBUG:boto:path=/pklxkd-26/ 2023-05-31T20:04:29.519 DEBUG:boto:auth_path=/pklxkd-26/ 2023-05-31T20:04:29.520 DEBUG:boto:path=/pklxkd-26/?acl 2023-05-31T20:04:29.520 DEBUG:boto:auth_path=/pklxkd-26/?acl 2023-05-31T20:04:29.520 DEBUG:boto:Method: PUT 2023-05-31T20:04:29.521 DEBUG:boto:Path: /pklxkd-26/?acl 2023-05-31T20:04:29.521 DEBUG:boto:Data: 2023-05-31T20:04:29.521 DEBUG:boto:Headers: {'x-amz-acl': 'public-read'} 2023-05-31T20:04:29.521 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-05-31T20:04:29.522 DEBUG:boto:Port: 8001 2023-05-31T20:04:29.522 DEBUG:boto:Params: {} 2023-05-31T20:04:29.522 DEBUG:boto:Token: None 2023-05-31T20:04:29.523 DEBUG:boto:StringToSign: PUT Wed, 31 May 2023 20:04:29 GMT x-amz-acl:public-read /pklxkd-26/?acl 2023-05-31T20:04:29.523 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:mbiObvX6NNzPqWU1/pvLIOu6Xo0= 2023-05-31T20:04:29.524 DEBUG:boto:Final headers: {'x-amz-acl': 'public-read', 'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:04:29 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:mbiObvX6NNzPqWU1/pvLIOu6Xo0=', 'Content-Length': '0'} 2023-05-31T20:04:29.543 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000a70a42d9892f57e6-006477a84d-1101-b2'), ('Content-Type', 'application/xml'), ('Content-Length', '0'), ('Date', 'Wed, 31 May 2023 20:04:29 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:04:29.544 DEBUG:boto:path=/pklxkd-26/ 2023-05-31T20:04:29.544 DEBUG:boto:auth_path=/pklxkd-26/ 2023-05-31T20:04:29.545 DEBUG:boto:path=/pklxkd-26/?acl 2023-05-31T20:04:29.545 DEBUG:boto:auth_path=/pklxkd-26/?acl 2023-05-31T20:04:29.545 DEBUG:boto:Method: GET 2023-05-31T20:04:29.546 DEBUG:boto:Path: /pklxkd-26/?acl 2023-05-31T20:04:29.546 DEBUG:boto:Data: 2023-05-31T20:04:29.546 DEBUG:boto:Headers: {} 2023-05-31T20:04:29.546 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-05-31T20:04:29.547 DEBUG:boto:Port: 8001 2023-05-31T20:04:29.547 DEBUG:boto:Params: {} 2023-05-31T20:04:29.547 DEBUG:boto:Token: None 2023-05-31T20:04:29.548 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:04:29 GMT /pklxkd-26/?acl 2023-05-31T20:04:29.548 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:Us1IsvHGgAYXvIe3QcQV0fLlYGo= 2023-05-31T20:04:29.550 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:04:29 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:Us1IsvHGgAYXvIe3QcQV0fLlYGo=', 'Content-Length': '0'} 2023-05-31T20:04:29.551 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx000004e4c87e64bab930a-006477a84d-1101-b2'), ('Content-Type', 'application/xml'), ('Content-Length', '681'), ('Date', 'Wed, 31 May 2023 20:04:29 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:04:29.552 INFO:tasks.rgw_multisite_tests:rgw_multi.tests.test_bucket_acl ... ok 2023-05-31T20:04:29.553 INFO:rgw_multi.tests:create bucket zone=a1 name=pklxkd-27 2023-05-31T20:04:29.553 DEBUG:boto:path=/pklxkd-27/ 2023-05-31T20:04:29.553 DEBUG:boto:auth_path=/pklxkd-27/ 2023-05-31T20:04:29.554 DEBUG:boto:Method: PUT 2023-05-31T20:04:29.554 DEBUG:boto:Path: /pklxkd-27/ 2023-05-31T20:04:29.554 DEBUG:boto:Data: 2023-05-31T20:04:29.555 DEBUG:boto:Headers: {} 2023-05-31T20:04:29.555 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:04:29.555 DEBUG:boto:Port: 8000 2023-05-31T20:04:29.556 DEBUG:boto:Params: {} 2023-05-31T20:04:29.556 DEBUG:boto:Token: None 2023-05-31T20:04:29.557 DEBUG:boto:StringToSign: PUT Wed, 31 May 2023 20:04:29 GMT /pklxkd-27/ 2023-05-31T20:04:29.557 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:K1Be6f9GJA9lyOht93b38xleVa0= 2023-05-31T20:04:29.557 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:04:29 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:K1Be6f9GJA9lyOht93b38xleVa0=', 'Content-Length': '0'} 2023-05-31T20:04:29.585 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx0000096c9e98568df374d-006477a84d-110e-a1'), ('Content-Length', '0'), ('Date', 'Wed, 31 May 2023 20:04:29 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:04:29.585 INFO:rgw_multi.tests:create bucket zone=a2 name=pklxkd-28 2023-05-31T20:04:29.586 DEBUG:boto:path=/pklxkd-28/ 2023-05-31T20:04:29.586 DEBUG:boto:auth_path=/pklxkd-28/ 2023-05-31T20:04:29.586 DEBUG:boto:Method: PUT 2023-05-31T20:04:29.587 DEBUG:boto:Path: /pklxkd-28/ 2023-05-31T20:04:29.587 DEBUG:boto:Data: 2023-05-31T20:04:29.587 DEBUG:boto:Headers: {} 2023-05-31T20:04:29.588 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:04:29.588 DEBUG:boto:Port: 8001 2023-05-31T20:04:29.588 DEBUG:boto:Params: {} 2023-05-31T20:04:29.589 DEBUG:boto:Token: None 2023-05-31T20:04:29.589 DEBUG:boto:StringToSign: PUT Wed, 31 May 2023 20:04:29 GMT /pklxkd-28/ 2023-05-31T20:04:29.590 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:zmsoT7pXpFdUEnn2FkAp8c9CNrc= 2023-05-31T20:04:29.590 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:04:29 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:zmsoT7pXpFdUEnn2FkAp8c9CNrc=', 'Content-Length': '0'} 2023-05-31T20:04:29.632 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx000000eca4182c5fd49b0-006477a84d-1112-a2'), ('Content-Length', '0'), ('Date', 'Wed, 31 May 2023 20:04:29 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:04:29.633 INFO:rgw_multi.tests:create bucket zone=b1 name=pklxkd-29 2023-05-31T20:04:29.634 DEBUG:boto:path=/pklxkd-29/ 2023-05-31T20:04:29.635 DEBUG:boto:auth_path=/pklxkd-29/ 2023-05-31T20:04:29.635 DEBUG:boto:Method: PUT 2023-05-31T20:04:29.635 DEBUG:boto:Path: /pklxkd-29/ 2023-05-31T20:04:29.636 DEBUG:boto:Data: 2023-05-31T20:04:29.636 DEBUG:boto:Headers: {} 2023-05-31T20:04:29.637 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-05-31T20:04:29.638 DEBUG:boto:Port: 8000 2023-05-31T20:04:29.639 DEBUG:boto:Params: {} 2023-05-31T20:04:29.640 DEBUG:boto:Token: None 2023-05-31T20:04:29.641 DEBUG:boto:StringToSign: PUT Wed, 31 May 2023 20:04:29 GMT /pklxkd-29/ 2023-05-31T20:04:29.642 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:ZMTYg4ZLH19p6URkTGEQdqRjMv8= 2023-05-31T20:04:29.643 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:04:29 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:ZMTYg4ZLH19p6URkTGEQdqRjMv8=', 'Content-Length': '0'} 2023-05-31T20:04:29.685 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx000006672835a3652b757-006477a84d-10fb-b1'), ('Content-Length', '0'), ('Date', 'Wed, 31 May 2023 20:04:29 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:04:29.685 INFO:rgw_multi.tests:create bucket zone=b2 name=pklxkd-30 2023-05-31T20:04:29.686 DEBUG:boto:path=/pklxkd-30/ 2023-05-31T20:04:29.686 DEBUG:boto:auth_path=/pklxkd-30/ 2023-05-31T20:04:29.686 DEBUG:boto:Method: PUT 2023-05-31T20:04:29.687 DEBUG:boto:Path: /pklxkd-30/ 2023-05-31T20:04:29.687 DEBUG:boto:Data: 2023-05-31T20:04:29.687 DEBUG:boto:Headers: {} 2023-05-31T20:04:29.688 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-05-31T20:04:29.688 DEBUG:boto:Port: 8001 2023-05-31T20:04:29.688 DEBUG:boto:Params: {} 2023-05-31T20:04:29.690 DEBUG:boto:Token: None 2023-05-31T20:04:29.691 DEBUG:boto:StringToSign: PUT Wed, 31 May 2023 20:04:29 GMT /pklxkd-30/ 2023-05-31T20:04:29.691 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:qWFdqrhz8nc/GaiuqsPKX0pV2ug= 2023-05-31T20:04:29.691 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:04:29 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:qWFdqrhz8nc/GaiuqsPKX0pV2ug=', 'Content-Length': '0'} 2023-05-31T20:04:29.732 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx0000001871853928005da-006477a84d-1101-b2'), ('Content-Length', '0'), ('Date', 'Wed, 31 May 2023 20:04:29 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:04:29.732 DEBUG:boto:path=/pklxkd-27/ 2023-05-31T20:04:29.733 DEBUG:boto:auth_path=/pklxkd-27/ 2023-05-31T20:04:29.733 DEBUG:boto:path=/pklxkd-27/?cors 2023-05-31T20:04:29.733 DEBUG:boto:auth_path=/pklxkd-27/?cors 2023-05-31T20:04:29.734 DEBUG:boto:Method: PUT 2023-05-31T20:04:29.734 DEBUG:boto:Path: /pklxkd-27/?cors 2023-05-31T20:04:29.734 DEBUG:boto:Data: DELETEhttps://www.example.com*3000 2023-05-31T20:04:29.735 DEBUG:boto:Headers: {'Content-MD5': 'c5gEzAIzM7zcw8WVE+AbrQ==', 'Content-Type': 'text/xml'} 2023-05-31T20:04:29.735 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:04:29.735 DEBUG:boto:Port: 8000 2023-05-31T20:04:29.736 DEBUG:boto:Params: {} 2023-05-31T20:04:29.736 DEBUG:boto:Token: None 2023-05-31T20:04:29.736 DEBUG:boto:StringToSign: PUT c5gEzAIzM7zcw8WVE+AbrQ== text/xml Wed, 31 May 2023 20:04:29 GMT /pklxkd-27/?cors 2023-05-31T20:04:29.737 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:N8610Dv3e3uQHBKgGEan18J/PHk= 2023-05-31T20:04:29.737 DEBUG:boto:Final headers: {'Content-MD5': 'c5gEzAIzM7zcw8WVE+AbrQ==', 'Content-Type': 'text/xml', 'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:04:29 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:N8610Dv3e3uQHBKgGEan18J/PHk=', 'Content-Length': '218'} 2023-05-31T20:04:29.744 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000b06960616aca82d3-006477a84d-110e-a1'), ('Content-Type', 'application/xml'), ('Content-Length', '0'), ('Date', 'Wed, 31 May 2023 20:04:29 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:04:29.745 DEBUG:boto:path=/pklxkd-27/ 2023-05-31T20:04:29.745 DEBUG:boto:auth_path=/pklxkd-27/ 2023-05-31T20:04:29.745 DEBUG:boto:path=/pklxkd-27/?cors 2023-05-31T20:04:29.746 DEBUG:boto:auth_path=/pklxkd-27/?cors 2023-05-31T20:04:29.746 DEBUG:boto:Method: GET 2023-05-31T20:04:29.746 DEBUG:boto:Path: /pklxkd-27/?cors 2023-05-31T20:04:29.747 DEBUG:boto:Data: 2023-05-31T20:04:29.747 DEBUG:boto:Headers: {} 2023-05-31T20:04:29.747 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:04:29.747 DEBUG:boto:Port: 8000 2023-05-31T20:04:29.748 DEBUG:boto:Params: {} 2023-05-31T20:04:29.748 DEBUG:boto:Token: None 2023-05-31T20:04:29.749 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:04:29 GMT /pklxkd-27/?cors 2023-05-31T20:04:29.749 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:gHSYJ4EXskaraaN2kKhkEaYrVOU= 2023-05-31T20:04:29.749 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:04:29 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:gHSYJ4EXskaraaN2kKhkEaYrVOU=', 'Content-Length': '0'} 2023-05-31T20:04:29.751 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000def14f656d655902-006477a84d-110e-a1'), ('Content-Type', 'application/xml'), ('Content-Length', '266'), ('Date', 'Wed, 31 May 2023 20:04:29 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:04:29.752 DEBUG:boto:b'DELETEhttps://www.example.com*3000' 2023-05-31T20:04:29.753 DEBUG:boto:path=/pklxkd-28/ 2023-05-31T20:04:29.753 DEBUG:boto:auth_path=/pklxkd-28/ 2023-05-31T20:04:29.754 DEBUG:boto:path=/pklxkd-28/?cors 2023-05-31T20:04:29.754 DEBUG:boto:auth_path=/pklxkd-28/?cors 2023-05-31T20:04:29.754 DEBUG:boto:Method: PUT 2023-05-31T20:04:29.754 DEBUG:boto:Path: /pklxkd-28/?cors 2023-05-31T20:04:29.755 DEBUG:boto:Data: DELETEhttps://www.example.com*3000 2023-05-31T20:04:29.755 DEBUG:boto:Headers: {'Content-MD5': 'c5gEzAIzM7zcw8WVE+AbrQ==', 'Content-Type': 'text/xml'} 2023-05-31T20:04:29.755 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:04:29.756 DEBUG:boto:Port: 8001 2023-05-31T20:04:29.756 DEBUG:boto:Params: {} 2023-05-31T20:04:29.756 DEBUG:boto:Token: None 2023-05-31T20:04:29.757 DEBUG:boto:StringToSign: PUT c5gEzAIzM7zcw8WVE+AbrQ== text/xml Wed, 31 May 2023 20:04:29 GMT /pklxkd-28/?cors 2023-05-31T20:04:29.758 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:7IE04iIZlwyTDrMN12uAHze+8U0= 2023-05-31T20:04:29.758 DEBUG:boto:Final headers: {'Content-MD5': 'c5gEzAIzM7zcw8WVE+AbrQ==', 'Content-Type': 'text/xml', 'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:04:29 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:7IE04iIZlwyTDrMN12uAHze+8U0=', 'Content-Length': '218'} 2023-05-31T20:04:30.767 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000a2d036a19c272360-006477a84d-1112-a2'), ('Content-Type', 'application/xml'), ('Content-Length', '0'), ('Date', 'Wed, 31 May 2023 20:04:30 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:04:30.768 DEBUG:boto:path=/pklxkd-28/ 2023-05-31T20:04:30.768 DEBUG:boto:auth_path=/pklxkd-28/ 2023-05-31T20:04:30.769 DEBUG:boto:path=/pklxkd-28/?cors 2023-05-31T20:04:30.769 DEBUG:boto:auth_path=/pklxkd-28/?cors 2023-05-31T20:04:30.769 DEBUG:boto:Method: GET 2023-05-31T20:04:30.770 DEBUG:boto:Path: /pklxkd-28/?cors 2023-05-31T20:04:30.770 DEBUG:boto:Data: 2023-05-31T20:04:30.770 DEBUG:boto:Headers: {} 2023-05-31T20:04:30.771 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:04:30.771 DEBUG:boto:Port: 8001 2023-05-31T20:04:30.771 DEBUG:boto:Params: {} 2023-05-31T20:04:30.772 DEBUG:boto:Token: None 2023-05-31T20:04:30.772 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:04:30 GMT /pklxkd-28/?cors 2023-05-31T20:04:30.773 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:FzL7TAmJqYZ89o3d/iqqsnGYgIc= 2023-05-31T20:04:30.773 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:04:30 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:FzL7TAmJqYZ89o3d/iqqsnGYgIc=', 'Content-Length': '0'} 2023-05-31T20:04:30.774 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx000009fe2d06a48392dd4-006477a84e-1112-a2'), ('Content-Type', 'application/xml'), ('Content-Length', '266'), ('Date', 'Wed, 31 May 2023 20:04:30 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:04:30.816 DEBUG:boto:b'DELETEhttps://www.example.com*3000' 2023-05-31T20:04:30.817 DEBUG:boto:path=/pklxkd-29/ 2023-05-31T20:04:30.817 DEBUG:boto:auth_path=/pklxkd-29/ 2023-05-31T20:04:30.818 DEBUG:boto:path=/pklxkd-29/?cors 2023-05-31T20:04:30.818 DEBUG:boto:auth_path=/pklxkd-29/?cors 2023-05-31T20:04:30.819 DEBUG:boto:Method: PUT 2023-05-31T20:04:30.819 DEBUG:boto:Path: /pklxkd-29/?cors 2023-05-31T20:04:30.819 DEBUG:boto:Data: DELETEhttps://www.example.com*3000 2023-05-31T20:04:30.820 DEBUG:boto:Headers: {'Content-MD5': 'c5gEzAIzM7zcw8WVE+AbrQ==', 'Content-Type': 'text/xml'} 2023-05-31T20:04:30.820 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-05-31T20:04:30.820 DEBUG:boto:Port: 8000 2023-05-31T20:04:30.821 DEBUG:boto:Params: {} 2023-05-31T20:04:30.821 DEBUG:boto:Token: None 2023-05-31T20:04:30.822 DEBUG:boto:StringToSign: PUT c5gEzAIzM7zcw8WVE+AbrQ== text/xml Wed, 31 May 2023 20:04:30 GMT /pklxkd-29/?cors 2023-05-31T20:04:30.822 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:FQTtN2kIVXsRFlw0iMUR1N+pL9c= 2023-05-31T20:04:30.823 DEBUG:boto:Final headers: {'Content-MD5': 'c5gEzAIzM7zcw8WVE+AbrQ==', 'Content-Type': 'text/xml', 'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:04:30 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:FQTtN2kIVXsRFlw0iMUR1N+pL9c=', 'Content-Length': '218'} 2023-05-31T20:04:31.834 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx0000029e65d43fe516fee-006477a84e-10fb-b1'), ('Content-Type', 'application/xml'), ('Content-Length', '0'), ('Date', 'Wed, 31 May 2023 20:04:31 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:04:31.834 DEBUG:boto:path=/pklxkd-29/ 2023-05-31T20:04:31.835 DEBUG:boto:auth_path=/pklxkd-29/ 2023-05-31T20:04:31.835 DEBUG:boto:path=/pklxkd-29/?cors 2023-05-31T20:04:31.835 DEBUG:boto:auth_path=/pklxkd-29/?cors 2023-05-31T20:04:31.836 DEBUG:boto:Method: GET 2023-05-31T20:04:31.836 DEBUG:boto:Path: /pklxkd-29/?cors 2023-05-31T20:04:31.836 DEBUG:boto:Data: 2023-05-31T20:04:31.837 DEBUG:boto:Headers: {} 2023-05-31T20:04:31.837 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-05-31T20:04:31.837 DEBUG:boto:Port: 8000 2023-05-31T20:04:31.838 DEBUG:boto:Params: {} 2023-05-31T20:04:31.838 DEBUG:boto:Token: None 2023-05-31T20:04:31.838 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:04:31 GMT /pklxkd-29/?cors 2023-05-31T20:04:31.839 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:MKjs/LvF4OpagxOtllYssyy6GRg= 2023-05-31T20:04:31.839 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:04:31 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:MKjs/LvF4OpagxOtllYssyy6GRg=', 'Content-Length': '0'} 2023-05-31T20:04:31.840 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx0000087dc1ce8e86ded40-006477a84f-10fb-b1'), ('Content-Type', 'application/xml'), ('Content-Length', '266'), ('Date', 'Wed, 31 May 2023 20:04:31 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:04:31.884 DEBUG:boto:b'DELETEhttps://www.example.com*3000' 2023-05-31T20:04:31.885 DEBUG:boto:path=/pklxkd-30/ 2023-05-31T20:04:31.885 DEBUG:boto:auth_path=/pklxkd-30/ 2023-05-31T20:04:31.886 DEBUG:boto:path=/pklxkd-30/?cors 2023-05-31T20:04:31.886 DEBUG:boto:auth_path=/pklxkd-30/?cors 2023-05-31T20:04:31.886 DEBUG:boto:Method: PUT 2023-05-31T20:04:31.887 DEBUG:boto:Path: /pklxkd-30/?cors 2023-05-31T20:04:31.887 DEBUG:boto:Data: DELETEhttps://www.example.com*3000 2023-05-31T20:04:31.887 DEBUG:boto:Headers: {'Content-MD5': 'c5gEzAIzM7zcw8WVE+AbrQ==', 'Content-Type': 'text/xml'} 2023-05-31T20:04:31.888 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-05-31T20:04:31.888 DEBUG:boto:Port: 8001 2023-05-31T20:04:31.888 DEBUG:boto:Params: {} 2023-05-31T20:04:31.889 DEBUG:boto:Token: None 2023-05-31T20:04:31.889 DEBUG:boto:StringToSign: PUT c5gEzAIzM7zcw8WVE+AbrQ== text/xml Wed, 31 May 2023 20:04:31 GMT /pklxkd-30/?cors 2023-05-31T20:04:31.889 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:cQ2YV4S7WGoiuzguKKZ2VOG9fqM= 2023-05-31T20:04:31.890 DEBUG:boto:Final headers: {'Content-MD5': 'c5gEzAIzM7zcw8WVE+AbrQ==', 'Content-Type': 'text/xml', 'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:04:31 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:cQ2YV4S7WGoiuzguKKZ2VOG9fqM=', 'Content-Length': '218'} 2023-05-31T20:04:32.901 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx000003910709ee862a8ca-006477a84f-1101-b2'), ('Content-Type', 'application/xml'), ('Content-Length', '0'), ('Date', 'Wed, 31 May 2023 20:04:32 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:04:32.902 DEBUG:boto:path=/pklxkd-30/ 2023-05-31T20:04:32.902 DEBUG:boto:auth_path=/pklxkd-30/ 2023-05-31T20:04:32.902 DEBUG:boto:path=/pklxkd-30/?cors 2023-05-31T20:04:32.903 DEBUG:boto:auth_path=/pklxkd-30/?cors 2023-05-31T20:04:32.903 DEBUG:boto:Method: GET 2023-05-31T20:04:32.903 DEBUG:boto:Path: /pklxkd-30/?cors 2023-05-31T20:04:32.904 DEBUG:boto:Data: 2023-05-31T20:04:32.904 DEBUG:boto:Headers: {} 2023-05-31T20:04:32.904 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-05-31T20:04:32.905 DEBUG:boto:Port: 8001 2023-05-31T20:04:32.905 DEBUG:boto:Params: {} 2023-05-31T20:04:32.906 DEBUG:boto:Token: None 2023-05-31T20:04:32.906 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:04:32 GMT /pklxkd-30/?cors 2023-05-31T20:04:32.907 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:RrLvLR9OU4JHCn4jnbylIpKAqMw= 2023-05-31T20:04:32.907 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:04:32 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:RrLvLR9OU4JHCn4jnbylIpKAqMw=', 'Content-Length': '0'} 2023-05-31T20:04:32.908 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx0000070f6e69cff62bc7f-006477a850-1101-b2'), ('Content-Type', 'application/xml'), ('Content-Length', '266'), ('Date', 'Wed, 31 May 2023 20:04:32 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:04:32.952 DEBUG:boto:b'DELETEhttps://www.example.com*3000' 2023-05-31T20:04:32.953 INFO:tasks.rgw_multisite_tests:rgw_multi.tests.test_bucket_cors ... ok 2023-05-31T20:04:32.954 INFO:rgw_multi.tests:create bucket zone=a1 name=pklxkd-31 2023-05-31T20:04:32.955 DEBUG:boto:path=/pklxkd-31/ 2023-05-31T20:04:32.955 DEBUG:boto:auth_path=/pklxkd-31/ 2023-05-31T20:04:32.955 DEBUG:boto:Method: PUT 2023-05-31T20:04:32.956 DEBUG:boto:Path: /pklxkd-31/ 2023-05-31T20:04:32.956 DEBUG:boto:Data: 2023-05-31T20:04:32.956 DEBUG:boto:Headers: {} 2023-05-31T20:04:32.957 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:04:32.957 DEBUG:boto:Port: 8000 2023-05-31T20:04:32.957 DEBUG:boto:Params: {} 2023-05-31T20:04:32.958 DEBUG:boto:Token: None 2023-05-31T20:04:32.958 DEBUG:boto:StringToSign: PUT Wed, 31 May 2023 20:04:32 GMT /pklxkd-31/ 2023-05-31T20:04:32.958 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:V+NJw+ffiH4U6zQ5EoHGtN4OEtk= 2023-05-31T20:04:32.959 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:04:32 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:V+NJw+ffiH4U6zQ5EoHGtN4OEtk=', 'Content-Length': '0'} 2023-05-31T20:04:32.999 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000585f38b2c58b5dfc-006477a850-110e-a1'), ('Content-Length', '0'), ('Date', 'Wed, 31 May 2023 20:04:32 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:04:32.999 INFO:rgw_multi.tests:create bucket zone=a2 name=pklxkd-32 2023-05-31T20:04:33.000 DEBUG:boto:path=/pklxkd-32/ 2023-05-31T20:04:33.000 DEBUG:boto:auth_path=/pklxkd-32/ 2023-05-31T20:04:33.000 DEBUG:boto:Method: PUT 2023-05-31T20:04:33.001 DEBUG:boto:Path: /pklxkd-32/ 2023-05-31T20:04:33.001 DEBUG:boto:Data: 2023-05-31T20:04:33.001 DEBUG:boto:Headers: {} 2023-05-31T20:04:33.002 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:04:33.002 DEBUG:boto:Port: 8001 2023-05-31T20:04:33.002 DEBUG:boto:Params: {} 2023-05-31T20:04:33.003 DEBUG:boto:Token: None 2023-05-31T20:04:33.003 DEBUG:boto:StringToSign: PUT Wed, 31 May 2023 20:04:33 GMT /pklxkd-32/ 2023-05-31T20:04:33.004 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:1EcbrTp/gSmZ4DbzQBIoXxb1c7s= 2023-05-31T20:04:33.004 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:04:33 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:1EcbrTp/gSmZ4DbzQBIoXxb1c7s=', 'Content-Length': '0'} 2023-05-31T20:04:33.050 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000a2a1d883549652df-006477a851-1112-a2'), ('Content-Length', '0'), ('Date', 'Wed, 31 May 2023 20:04:33 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:04:33.051 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-31T20:04:33.052 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-31T20:04:33.052 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-05-31T20:04:33.103 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2023-05-31T20:04:33.104 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2023-05-31T20:04:33.165 INFO:teuthology.orchestra.run.smithi099.stderr:No --period given, using current period=2e6123d2-d695-4350-9a01-61a455bc782b 2023-05-31T20:04:33.187 INFO:teuthology.orchestra.run.smithi099.stdout:[ 2023-05-31T20:04:33.187 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T20:04:33.187 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1685562631.516089_59.1", 2023-05-31T20:04:33.188 INFO:teuthology.orchestra.run.smithi099.stdout: "last_update": "2023-05-31T19:50:31.516089Z" 2023-05-31T20:04:33.188 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:04:33.188 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T20:04:33.188 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "", 2023-05-31T20:04:33.189 INFO:teuthology.orchestra.run.smithi099.stdout: "last_update": "0.000000" 2023-05-31T20:04:33.189 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:04:33.189 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T20:04:33.189 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1685562970.274572_93.1", 2023-05-31T20:04:33.190 INFO:teuthology.orchestra.run.smithi099.stdout: "last_update": "2023-05-31T19:56:10.274572Z" 2023-05-31T20:04:33.190 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:04:33.190 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T20:04:33.190 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1685563473.026869_185.1", 2023-05-31T20:04:33.191 INFO:teuthology.orchestra.run.smithi099.stdout: "last_update": "2023-05-31T20:04:33.026869Z" 2023-05-31T20:04:33.191 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T20:04:33.191 INFO:teuthology.orchestra.run.smithi099.stdout:] 2023-05-31T20:04:33.192 DEBUG:rgw_multi.tests:master meta markers={0: '1_1685562631.516089_59.1', 1: '', 2: '1_1685562970.274572_93.1', 3: '1_1685563473.026869_185.1'} 2023-05-31T20:04:33.193 INFO:rgw_multi.tests:starting meta checkpoint for zone=a2 2023-05-31T20:04:33.194 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-31T20:04:33.194 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-31T20:04:33.195 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-05-31T20:04:33.245 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2023-05-31T20:04:33.246 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2023-05-31T20:04:33.282 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:04:33.276+0000 7f2b63505780 20 RGW-SYNC:meta: read sync status 2023-05-31T20:04:33.282 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:04:33.277+0000 7f2b63505780 20 RGW-SYNC:meta: read sync status 2023-05-31T20:04:33.295 INFO:teuthology.orchestra.run.smithi099.stdout:{ 2023-05-31T20:04:33.295 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_status": { 2023-05-31T20:04:33.296 INFO:teuthology.orchestra.run.smithi099.stdout: "info": { 2023-05-31T20:04:33.296 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "sync", 2023-05-31T20:04:33.296 INFO:teuthology.orchestra.run.smithi099.stdout: "num_shards": 4, 2023-05-31T20:04:33.296 INFO:teuthology.orchestra.run.smithi099.stdout: "period": "2e6123d2-d695-4350-9a01-61a455bc782b", 2023-05-31T20:04:33.297 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2023-05-31T20:04:33.297 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:04:33.297 INFO:teuthology.orchestra.run.smithi099.stdout: "markers": [ 2023-05-31T20:04:33.298 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T20:04:33.298 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 0, 2023-05-31T20:04:33.298 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T20:04:33.298 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2023-05-31T20:04:33.299 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1685562631.516089_59.1", 2023-05-31T20:04:33.299 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T20:04:33.299 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 1, 2023-05-31T20:04:33.299 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T20:04:33.300 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2023-05-31T19:50:31.516089Z", 2023-05-31T20:04:33.300 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2023-05-31T20:04:33.300 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T20:04:33.300 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:04:33.301 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T20:04:33.301 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 1, 2023-05-31T20:04:33.301 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T20:04:33.301 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2023-05-31T20:04:33.302 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "", 2023-05-31T20:04:33.302 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T20:04:33.302 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2023-05-31T20:04:33.302 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T20:04:33.303 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "0.000000", 2023-05-31T20:04:33.303 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2023-05-31T20:04:33.303 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T20:04:33.303 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:04:33.304 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T20:04:33.304 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 2, 2023-05-31T20:04:33.304 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T20:04:33.304 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2023-05-31T20:04:33.305 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1685562970.274572_93.1", 2023-05-31T20:04:33.305 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T20:04:33.305 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2023-05-31T20:04:33.305 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T20:04:33.306 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2023-05-31T19:56:10.274572Z", 2023-05-31T20:04:33.306 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2023-05-31T20:04:33.306 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T20:04:33.306 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:04:33.306 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T20:04:33.307 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 3, 2023-05-31T20:04:33.307 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T20:04:33.307 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2023-05-31T20:04:33.307 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1685563472.965359_178.1", 2023-05-31T20:04:33.307 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T20:04:33.308 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2023-05-31T20:04:33.308 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T20:04:33.308 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2023-05-31T20:04:32.965359Z", 2023-05-31T20:04:33.308 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2023-05-31T20:04:33.308 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T20:04:33.309 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T20:04:33.309 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2023-05-31T20:04:33.309 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:04:33.309 INFO:teuthology.orchestra.run.smithi099.stdout: "full_sync": { 2023-05-31T20:04:33.309 INFO:teuthology.orchestra.run.smithi099.stdout: "total": 1, 2023-05-31T20:04:33.309 INFO:teuthology.orchestra.run.smithi099.stdout: "complete": 1 2023-05-31T20:04:33.309 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:04:33.310 INFO:teuthology.orchestra.run.smithi099.stdout: "current_time": "2023-05-31T20:04:33Z" 2023-05-31T20:04:33.310 INFO:teuthology.orchestra.run.smithi099.stdout:} 2023-05-31T20:04:33.311 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "2e6123d2-d695-4350-9a01-61a455bc782b", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1685562631.516089_59.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-05-31T19:50:31.516089Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1685562970.274572_93.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-05-31T19:56:10.274572Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1685563472.965359_178.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-05-31T20:04:32.965359Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-05-31T20:04:33Z" } 2023-05-31T20:04:33.311 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1685562631.516089_59.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-05-31T19:50:31.516089Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1685562970.274572_93.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-05-31T19:56:10.274572Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1685563472.965359_178.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-05-31T20:04:32.965359Z', 'realm_epoch': 2}}] 2023-05-31T20:04:33.312 DEBUG:rgw_multi.tests:log_status={0: '1_1685562631.516089_59.1', 1: '', 2: '1_1685562970.274572_93.1', 3: '1_1685563473.026869_185.1'} 2023-05-31T20:04:33.312 DEBUG:rgw_multi.tests:sync_status={0: '1_1685562631.516089_59.1', 1: '', 2: '1_1685562970.274572_93.1', 3: '1_1685563472.965359_178.1'} 2023-05-31T20:04:33.313 WARNING:rgw_multi.tests:zone a2 behind master: shard=3 master=1_1685563473.026869_185.1 target=1_1685563472.965359_178.1 2023-05-31T20:04:38.313 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-31T20:04:38.314 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-31T20:04:38.314 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-05-31T20:04:38.372 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2023-05-31T20:04:38.372 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2023-05-31T20:04:38.416 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:04:38.410+0000 7f7376cf8780 20 RGW-SYNC:meta: read sync status 2023-05-31T20:04:38.417 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:04:38.411+0000 7f7376cf8780 20 RGW-SYNC:meta: read sync status 2023-05-31T20:04:38.433 INFO:teuthology.orchestra.run.smithi099.stdout:{ 2023-05-31T20:04:38.433 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_status": { 2023-05-31T20:04:38.433 INFO:teuthology.orchestra.run.smithi099.stdout: "info": { 2023-05-31T20:04:38.433 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "sync", 2023-05-31T20:04:38.434 INFO:teuthology.orchestra.run.smithi099.stdout: "num_shards": 4, 2023-05-31T20:04:38.434 INFO:teuthology.orchestra.run.smithi099.stdout: "period": "2e6123d2-d695-4350-9a01-61a455bc782b", 2023-05-31T20:04:38.434 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2023-05-31T20:04:38.434 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:04:38.434 INFO:teuthology.orchestra.run.smithi099.stdout: "markers": [ 2023-05-31T20:04:38.434 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T20:04:38.434 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 0, 2023-05-31T20:04:38.435 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T20:04:38.435 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2023-05-31T20:04:38.435 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1685562631.516089_59.1", 2023-05-31T20:04:38.435 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T20:04:38.435 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 1, 2023-05-31T20:04:38.435 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T20:04:38.436 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2023-05-31T19:50:31.516089Z", 2023-05-31T20:04:38.436 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2023-05-31T20:04:38.436 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T20:04:38.436 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:04:38.436 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T20:04:38.436 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 1, 2023-05-31T20:04:38.436 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T20:04:38.437 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2023-05-31T20:04:38.437 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "", 2023-05-31T20:04:38.437 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T20:04:38.437 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2023-05-31T20:04:38.437 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T20:04:38.437 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "0.000000", 2023-05-31T20:04:38.437 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2023-05-31T20:04:38.438 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T20:04:38.438 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:04:38.438 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T20:04:38.438 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 2, 2023-05-31T20:04:38.438 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T20:04:38.438 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2023-05-31T20:04:38.439 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1685562970.274572_93.1", 2023-05-31T20:04:38.439 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T20:04:38.439 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2023-05-31T20:04:38.439 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T20:04:38.439 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2023-05-31T19:56:10.274572Z", 2023-05-31T20:04:38.439 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2023-05-31T20:04:38.439 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T20:04:38.440 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:04:38.440 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T20:04:38.440 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 3, 2023-05-31T20:04:38.440 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T20:04:38.440 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2023-05-31T20:04:38.440 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1685563473.026869_185.1", 2023-05-31T20:04:38.440 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T20:04:38.441 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2023-05-31T20:04:38.441 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T20:04:38.441 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2023-05-31T20:04:33.026869Z", 2023-05-31T20:04:38.441 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2023-05-31T20:04:38.441 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T20:04:38.441 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T20:04:38.441 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2023-05-31T20:04:38.442 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:04:38.442 INFO:teuthology.orchestra.run.smithi099.stdout: "full_sync": { 2023-05-31T20:04:38.442 INFO:teuthology.orchestra.run.smithi099.stdout: "total": 1, 2023-05-31T20:04:38.442 INFO:teuthology.orchestra.run.smithi099.stdout: "complete": 1 2023-05-31T20:04:38.442 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:04:38.442 INFO:teuthology.orchestra.run.smithi099.stdout: "current_time": "2023-05-31T20:04:38Z" 2023-05-31T20:04:38.443 INFO:teuthology.orchestra.run.smithi099.stdout:} 2023-05-31T20:04:38.443 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "2e6123d2-d695-4350-9a01-61a455bc782b", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1685562631.516089_59.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-05-31T19:50:31.516089Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1685562970.274572_93.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-05-31T19:56:10.274572Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1685563473.026869_185.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-05-31T20:04:33.026869Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-05-31T20:04:38Z" } 2023-05-31T20:04:38.444 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1685562631.516089_59.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-05-31T19:50:31.516089Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1685562970.274572_93.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-05-31T19:56:10.274572Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1685563473.026869_185.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-05-31T20:04:33.026869Z', 'realm_epoch': 2}}] 2023-05-31T20:04:38.444 DEBUG:rgw_multi.tests:log_status={0: '1_1685562631.516089_59.1', 1: '', 2: '1_1685562970.274572_93.1', 3: '1_1685563473.026869_185.1'} 2023-05-31T20:04:38.445 DEBUG:rgw_multi.tests:sync_status={0: '1_1685562631.516089_59.1', 1: '', 2: '1_1685562970.274572_93.1', 3: '1_1685563473.026869_185.1'} 2023-05-31T20:04:38.445 INFO:rgw_multi.tests:finish meta checkpoint for zone=a2 2023-05-31T20:04:38.445 DEBUG:boto:path=/pklxkd-31/ 2023-05-31T20:04:38.446 DEBUG:boto:auth_path=/pklxkd-31/ 2023-05-31T20:04:38.446 DEBUG:boto:Method: HEAD 2023-05-31T20:04:38.446 DEBUG:boto:Path: /pklxkd-31/ 2023-05-31T20:04:38.447 DEBUG:boto:Data: 2023-05-31T20:04:38.447 DEBUG:boto:Headers: {} 2023-05-31T20:04:38.447 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:04:38.448 DEBUG:boto:Port: 8000 2023-05-31T20:04:38.448 DEBUG:boto:Params: {} 2023-05-31T20:04:38.448 DEBUG:boto:Token: None 2023-05-31T20:04:38.449 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:04:38 GMT /pklxkd-31/ 2023-05-31T20:04:38.449 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:1T2ETbyGoUipIsBYtTjWHsS6bUg= 2023-05-31T20:04:38.450 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:04:38 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:1T2ETbyGoUipIsBYtTjWHsS6bUg=', 'Content-Length': '0'} 2023-05-31T20:04:38.452 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000452e4c0b304ce9bc-006477a856-110e-a1'), ('Content-Length', '0'), ('Date', 'Wed, 31 May 2023 20:04:38 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:04:38.452 DEBUG:boto:path=/pklxkd-31/foo 2023-05-31T20:04:38.453 DEBUG:boto:auth_path=/pklxkd-31/foo 2023-05-31T20:04:38.453 DEBUG:boto:Method: PUT 2023-05-31T20:04:38.453 DEBUG:boto:Path: /pklxkd-31/foo 2023-05-31T20:04:38.454 DEBUG:boto:Data: 2023-05-31T20:04:38.454 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'N7UdGUp1E+RbVvZSTy1R8g==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-05-31T20:04:38.455 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:04:38.455 DEBUG:boto:Port: 8000 2023-05-31T20:04:38.455 DEBUG:boto:Params: {} 2023-05-31T20:04:38.456 DEBUG:boto:Token: None 2023-05-31T20:04:38.456 DEBUG:boto:StringToSign: PUT N7UdGUp1E+RbVvZSTy1R8g== application/octet-stream Wed, 31 May 2023 20:04:38 GMT /pklxkd-31/foo 2023-05-31T20:04:38.457 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:RbD/82N2N639MPOXs44TIea3br8= 2023-05-31T20:04:38.457 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'N7UdGUp1E+RbVvZSTy1R8g==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Wed, 31 May 2023 20:04:38 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:RbD/82N2N639MPOXs44TIea3br8='} 2023-05-31T20:04:38.466 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"37b51d194a7513e45b56f6524f2d51f2"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000b4bc6a31a8d0cd62-006477a856-110e-a1'), ('Date', 'Wed, 31 May 2023 20:04:38 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:04:38.466 DEBUG:boto:path=/pklxkd-31/ 2023-05-31T20:04:38.466 DEBUG:boto:auth_path=/pklxkd-31/ 2023-05-31T20:04:38.467 DEBUG:boto:Method: DELETE 2023-05-31T20:04:38.467 DEBUG:boto:Path: /pklxkd-31/ 2023-05-31T20:04:38.467 DEBUG:boto:Data: 2023-05-31T20:04:38.468 DEBUG:boto:Headers: {} 2023-05-31T20:04:38.468 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:04:38.468 DEBUG:boto:Port: 8000 2023-05-31T20:04:38.469 DEBUG:boto:Params: {} 2023-05-31T20:04:38.469 DEBUG:boto:Token: None 2023-05-31T20:04:38.469 DEBUG:boto:StringToSign: DELETE Wed, 31 May 2023 20:04:38 GMT /pklxkd-31/ 2023-05-31T20:04:38.470 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:N0Eu/C9p8x5wOJe9UDFBUfk1OG0= 2023-05-31T20:04:38.470 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:04:38 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:N0Eu/C9p8x5wOJe9UDFBUfk1OG0=', 'Content-Length': '0'} 2023-05-31T20:04:38.476 DEBUG:boto:Response headers: [('Content-Length', '224'), ('x-amz-request-id', 'tx000000ebfb00fda816567-006477a856-110e-a1'), ('Accept-Ranges', 'bytes'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:04:38 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:04:38.521 DEBUG:boto:path=/pklxkd-32/ 2023-05-31T20:04:38.522 DEBUG:boto:auth_path=/pklxkd-32/ 2023-05-31T20:04:38.523 DEBUG:boto:Method: HEAD 2023-05-31T20:04:38.523 DEBUG:boto:Path: /pklxkd-32/ 2023-05-31T20:04:38.524 DEBUG:boto:Data: 2023-05-31T20:04:38.525 DEBUG:boto:Headers: {} 2023-05-31T20:04:38.525 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:04:38.526 DEBUG:boto:Port: 8001 2023-05-31T20:04:38.526 DEBUG:boto:Params: {} 2023-05-31T20:04:38.527 DEBUG:boto:Token: None 2023-05-31T20:04:38.527 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:04:38 GMT /pklxkd-32/ 2023-05-31T20:04:38.527 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:P0vWsNTv6Qq7lgwaDSur1jtw4r8= 2023-05-31T20:04:38.528 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:04:38 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:P0vWsNTv6Qq7lgwaDSur1jtw4r8=', 'Content-Length': '0'} 2023-05-31T20:04:38.530 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000582d27e218778e5e-006477a856-1112-a2'), ('Content-Length', '0'), ('Date', 'Wed, 31 May 2023 20:04:38 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:04:38.531 DEBUG:boto:path=/pklxkd-32/foo 2023-05-31T20:04:38.531 DEBUG:boto:auth_path=/pklxkd-32/foo 2023-05-31T20:04:38.532 DEBUG:boto:Method: PUT 2023-05-31T20:04:38.532 DEBUG:boto:Path: /pklxkd-32/foo 2023-05-31T20:04:38.532 DEBUG:boto:Data: 2023-05-31T20:04:38.533 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'N7UdGUp1E+RbVvZSTy1R8g==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-05-31T20:04:38.533 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:04:38.533 DEBUG:boto:Port: 8001 2023-05-31T20:04:38.534 DEBUG:boto:Params: {} 2023-05-31T20:04:38.535 DEBUG:boto:Token: None 2023-05-31T20:04:38.536 DEBUG:boto:StringToSign: PUT N7UdGUp1E+RbVvZSTy1R8g== application/octet-stream Wed, 31 May 2023 20:04:38 GMT /pklxkd-32/foo 2023-05-31T20:04:38.537 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:3eQW4vSMTy0qKtwIVgY6q2QvxMI= 2023-05-31T20:04:38.538 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'N7UdGUp1E+RbVvZSTy1R8g==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Wed, 31 May 2023 20:04:38 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:3eQW4vSMTy0qKtwIVgY6q2QvxMI='} 2023-05-31T20:04:38.546 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"37b51d194a7513e45b56f6524f2d51f2"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000092e25aa0c9d0f7bf-006477a856-1112-a2'), ('Date', 'Wed, 31 May 2023 20:04:38 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:04:38.547 DEBUG:boto:path=/pklxkd-32/ 2023-05-31T20:04:38.547 DEBUG:boto:auth_path=/pklxkd-32/ 2023-05-31T20:04:38.547 DEBUG:boto:Method: DELETE 2023-05-31T20:04:38.548 DEBUG:boto:Path: /pklxkd-32/ 2023-05-31T20:04:38.548 DEBUG:boto:Data: 2023-05-31T20:04:38.548 DEBUG:boto:Headers: {} 2023-05-31T20:04:38.549 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:04:38.549 DEBUG:boto:Port: 8001 2023-05-31T20:04:38.549 DEBUG:boto:Params: {} 2023-05-31T20:04:38.550 DEBUG:boto:Token: None 2023-05-31T20:04:38.550 DEBUG:boto:StringToSign: DELETE Wed, 31 May 2023 20:04:38 GMT /pklxkd-32/ 2023-05-31T20:04:38.550 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:Qie3jn9jrba/yr+5IWWZU2wUpHI= 2023-05-31T20:04:38.551 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:04:38 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:Qie3jn9jrba/yr+5IWWZU2wUpHI=', 'Content-Length': '0'} 2023-05-31T20:04:38.557 DEBUG:boto:Response headers: [('Content-Length', '224'), ('x-amz-request-id', 'tx0000080aaed0328b35b63-006477a856-1112-a2'), ('Accept-Ranges', 'bytes'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:04:38 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:04:38.558 DEBUG:boto:path=/pklxkd-31/ 2023-05-31T20:04:38.558 DEBUG:boto:auth_path=/pklxkd-31/ 2023-05-31T20:04:38.558 DEBUG:boto:Method: HEAD 2023-05-31T20:04:38.559 DEBUG:boto:Path: /pklxkd-31/ 2023-05-31T20:04:38.559 DEBUG:boto:Data: 2023-05-31T20:04:38.559 DEBUG:boto:Headers: {} 2023-05-31T20:04:38.560 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:04:38.560 DEBUG:boto:Port: 8000 2023-05-31T20:04:38.560 DEBUG:boto:Params: {} 2023-05-31T20:04:38.561 DEBUG:boto:Token: None 2023-05-31T20:04:38.561 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:04:38 GMT /pklxkd-31/ 2023-05-31T20:04:38.561 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:1T2ETbyGoUipIsBYtTjWHsS6bUg= 2023-05-31T20:04:38.562 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:04:38 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:1T2ETbyGoUipIsBYtTjWHsS6bUg=', 'Content-Length': '0'} 2023-05-31T20:04:38.564 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '3'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000003af0b356b55a1be7-006477a856-110e-a1'), ('Content-Length', '0'), ('Date', 'Wed, 31 May 2023 20:04:38 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:04:38.564 DEBUG:boto:path=/pklxkd-32/ 2023-05-31T20:04:38.565 DEBUG:boto:auth_path=/pklxkd-32/ 2023-05-31T20:04:38.565 DEBUG:boto:Method: HEAD 2023-05-31T20:04:38.565 DEBUG:boto:Path: /pklxkd-32/ 2023-05-31T20:04:38.566 DEBUG:boto:Data: 2023-05-31T20:04:38.566 DEBUG:boto:Headers: {} 2023-05-31T20:04:38.566 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:04:38.567 DEBUG:boto:Port: 8000 2023-05-31T20:04:38.568 DEBUG:boto:Params: {} 2023-05-31T20:04:38.569 DEBUG:boto:Token: None 2023-05-31T20:04:38.570 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:04:38 GMT /pklxkd-32/ 2023-05-31T20:04:38.570 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:P0vWsNTv6Qq7lgwaDSur1jtw4r8= 2023-05-31T20:04:38.571 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:04:38 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:P0vWsNTv6Qq7lgwaDSur1jtw4r8=', 'Content-Length': '0'} 2023-05-31T20:04:38.573 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000817745c975325770-006477a856-110e-a1'), ('Content-Length', '0'), ('Date', 'Wed, 31 May 2023 20:04:38 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:04:38.574 INFO:tasks.rgw_multisite_tests:rgw_multi.tests.test_bucket_delete_notempty ... ok 2023-05-31T20:04:38.576 INFO:tasks.rgw_multisite_tests:rgw_multi.tests.test_multi_period_incremental_sync ... SKIP: test_multi_period_incremental_sync skipped. Requires 3 or more zones in master zonegroup. 2023-05-31T20:04:38.577 INFO:rgw_multi.tests:create bucket zone=a1 name=pklxkd-33 2023-05-31T20:04:38.578 DEBUG:boto:path=/pklxkd-33/ 2023-05-31T20:04:38.579 DEBUG:boto:auth_path=/pklxkd-33/ 2023-05-31T20:04:38.579 DEBUG:boto:Method: PUT 2023-05-31T20:04:38.580 DEBUG:boto:Path: /pklxkd-33/ 2023-05-31T20:04:38.580 DEBUG:boto:Data: 2023-05-31T20:04:38.581 DEBUG:boto:Headers: {} 2023-05-31T20:04:38.582 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:04:38.582 DEBUG:boto:Port: 8000 2023-05-31T20:04:38.582 DEBUG:boto:Params: {} 2023-05-31T20:04:38.583 DEBUG:boto:Token: None 2023-05-31T20:04:38.584 DEBUG:boto:StringToSign: PUT Wed, 31 May 2023 20:04:38 GMT /pklxkd-33/ 2023-05-31T20:04:38.584 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:eRxDkMcPgTjvhrkIicZ2LexxdIc= 2023-05-31T20:04:38.585 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:04:38 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:eRxDkMcPgTjvhrkIicZ2LexxdIc=', 'Content-Length': '0'} 2023-05-31T20:04:38.603 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000948747d128dd194e-006477a856-110e-a1'), ('Content-Length', '0'), ('Date', 'Wed, 31 May 2023 20:04:38 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:04:38.604 INFO:rgw_multi.tests:create bucket zone=a2 name=pklxkd-34 2023-05-31T20:04:38.605 DEBUG:boto:path=/pklxkd-34/ 2023-05-31T20:04:38.605 DEBUG:boto:auth_path=/pklxkd-34/ 2023-05-31T20:04:38.606 DEBUG:boto:Method: PUT 2023-05-31T20:04:38.607 DEBUG:boto:Path: /pklxkd-34/ 2023-05-31T20:04:38.608 DEBUG:boto:Data: 2023-05-31T20:04:38.608 DEBUG:boto:Headers: {} 2023-05-31T20:04:38.609 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:04:38.610 DEBUG:boto:Port: 8001 2023-05-31T20:04:38.611 DEBUG:boto:Params: {} 2023-05-31T20:04:38.612 DEBUG:boto:Token: None 2023-05-31T20:04:38.613 DEBUG:boto:StringToSign: PUT Wed, 31 May 2023 20:04:38 GMT /pklxkd-34/ 2023-05-31T20:04:38.614 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:jDdS8VXEZnFfmM66GWqGgMNny/o= 2023-05-31T20:04:38.615 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:04:38 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:jDdS8VXEZnFfmM66GWqGgMNny/o=', 'Content-Length': '0'} 2023-05-31T20:04:38.646 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx000002ea0442e7d383410-006477a856-1112-a2'), ('Content-Length', '0'), ('Date', 'Wed, 31 May 2023 20:04:38 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:04:38.647 DEBUG:boto:path=/pklxkd-33/ 2023-05-31T20:04:38.647 DEBUG:boto:auth_path=/pklxkd-33/ 2023-05-31T20:04:38.647 DEBUG:boto:Method: HEAD 2023-05-31T20:04:38.648 DEBUG:boto:Path: /pklxkd-33/ 2023-05-31T20:04:38.648 DEBUG:boto:Data: 2023-05-31T20:04:38.649 DEBUG:boto:Headers: {} 2023-05-31T20:04:38.649 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:04:38.649 DEBUG:boto:Port: 8000 2023-05-31T20:04:38.649 DEBUG:boto:Params: {} 2023-05-31T20:04:38.650 DEBUG:boto:Token: None 2023-05-31T20:04:38.650 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:04:38 GMT /pklxkd-33/ 2023-05-31T20:04:38.651 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:gwQqQ1XzXPYwIksWCoQP5N2pQrU= 2023-05-31T20:04:38.651 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:04:38 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:gwQqQ1XzXPYwIksWCoQP5N2pQrU=', 'Content-Length': '0'} 2023-05-31T20:04:38.653 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000017cada2f2e49ba49-006477a856-110e-a1'), ('Content-Length', '0'), ('Date', 'Wed, 31 May 2023 20:04:38 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:04:38.654 DEBUG:boto:path=/pklxkd-33/key 2023-05-31T20:04:38.654 DEBUG:boto:auth_path=/pklxkd-33/key 2023-05-31T20:04:38.655 DEBUG:boto:Method: PUT 2023-05-31T20:04:38.655 DEBUG:boto:Path: /pklxkd-33/key 2023-05-31T20:04:38.655 DEBUG:boto:Data: 2023-05-31T20:04:38.656 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'hBotaJrYa9FhFEdFPCLG/A==', 'Content-Length': '4', 'Expect': '100-Continue'} 2023-05-31T20:04:38.656 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:04:38.656 DEBUG:boto:Port: 8000 2023-05-31T20:04:38.657 DEBUG:boto:Params: {} 2023-05-31T20:04:38.657 DEBUG:boto:Token: None 2023-05-31T20:04:38.658 DEBUG:boto:StringToSign: PUT hBotaJrYa9FhFEdFPCLG/A== application/octet-stream Wed, 31 May 2023 20:04:38 GMT /pklxkd-33/key 2023-05-31T20:04:38.658 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:4EdZtBohoh7lUpWJQs6RsubgXaw= 2023-05-31T20:04:38.658 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'hBotaJrYa9FhFEdFPCLG/A==', 'Content-Length': '4', 'Expect': '100-Continue', 'Date': 'Wed, 31 May 2023 20:04:38 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:4EdZtBohoh7lUpWJQs6RsubgXaw='} 2023-05-31T20:04:38.667 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"841a2d689ad86bd1611447453c22c6fc"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000049252a0aa942d828-006477a856-110e-a1'), ('Date', 'Wed, 31 May 2023 20:04:38 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:04:38.668 DEBUG:boto:path=/pklxkd-34/ 2023-05-31T20:04:38.668 DEBUG:boto:auth_path=/pklxkd-34/ 2023-05-31T20:04:38.668 DEBUG:boto:Method: HEAD 2023-05-31T20:04:38.669 DEBUG:boto:Path: /pklxkd-34/ 2023-05-31T20:04:38.669 DEBUG:boto:Data: 2023-05-31T20:04:38.670 DEBUG:boto:Headers: {} 2023-05-31T20:04:38.670 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:04:38.670 DEBUG:boto:Port: 8001 2023-05-31T20:04:38.670 DEBUG:boto:Params: {} 2023-05-31T20:04:38.671 DEBUG:boto:Token: None 2023-05-31T20:04:38.672 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:04:38 GMT /pklxkd-34/ 2023-05-31T20:04:38.672 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:C34WvL8kwHT3Au1xKFRy+zqapF4= 2023-05-31T20:04:38.673 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:04:38 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:C34WvL8kwHT3Au1xKFRy+zqapF4=', 'Content-Length': '0'} 2023-05-31T20:04:38.674 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000f5e6321a6031d986-006477a856-1112-a2'), ('Content-Length', '0'), ('Date', 'Wed, 31 May 2023 20:04:38 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:04:38.675 DEBUG:boto:path=/pklxkd-34/key 2023-05-31T20:04:38.676 DEBUG:boto:auth_path=/pklxkd-34/key 2023-05-31T20:04:38.676 DEBUG:boto:Method: PUT 2023-05-31T20:04:38.676 DEBUG:boto:Path: /pklxkd-34/key 2023-05-31T20:04:38.677 DEBUG:boto:Data: 2023-05-31T20:04:38.677 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'hBotaJrYa9FhFEdFPCLG/A==', 'Content-Length': '4', 'Expect': '100-Continue'} 2023-05-31T20:04:38.677 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:04:38.678 DEBUG:boto:Port: 8001 2023-05-31T20:04:38.678 DEBUG:boto:Params: {} 2023-05-31T20:04:38.678 DEBUG:boto:Token: None 2023-05-31T20:04:38.679 DEBUG:boto:StringToSign: PUT hBotaJrYa9FhFEdFPCLG/A== application/octet-stream Wed, 31 May 2023 20:04:38 GMT /pklxkd-34/key 2023-05-31T20:04:38.679 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:l5PnhffjL/XSML55/owhQ892bOs= 2023-05-31T20:04:38.680 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'hBotaJrYa9FhFEdFPCLG/A==', 'Content-Length': '4', 'Expect': '100-Continue', 'Date': 'Wed, 31 May 2023 20:04:38 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:l5PnhffjL/XSML55/owhQ892bOs='} 2023-05-31T20:04:38.688 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"841a2d689ad86bd1611447453c22c6fc"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000ee4a98638d8ebdd2-006477a856-1112-a2'), ('Date', 'Wed, 31 May 2023 20:04:38 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:04:38.689 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-31T20:04:38.689 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-31T20:04:38.690 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-05-31T20:04:38.741 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2023-05-31T20:04:38.741 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2023-05-31T20:04:38.806 INFO:teuthology.orchestra.run.smithi099.stderr:No --period given, using current period=2e6123d2-d695-4350-9a01-61a455bc782b 2023-05-31T20:04:38.829 INFO:teuthology.orchestra.run.smithi099.stdout:[ 2023-05-31T20:04:38.829 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T20:04:38.829 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1685562631.516089_59.1", 2023-05-31T20:04:38.830 INFO:teuthology.orchestra.run.smithi099.stdout: "last_update": "2023-05-31T19:50:31.516089Z" 2023-05-31T20:04:38.830 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:04:38.830 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T20:04:38.830 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "", 2023-05-31T20:04:38.830 INFO:teuthology.orchestra.run.smithi099.stdout: "last_update": "0.000000" 2023-05-31T20:04:38.830 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:04:38.831 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T20:04:38.831 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1685562970.274572_93.1", 2023-05-31T20:04:38.831 INFO:teuthology.orchestra.run.smithi099.stdout: "last_update": "2023-05-31T19:56:10.274572Z" 2023-05-31T20:04:38.831 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:04:38.831 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T20:04:38.831 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1685563478.630332_193.1", 2023-05-31T20:04:38.832 INFO:teuthology.orchestra.run.smithi099.stdout: "last_update": "2023-05-31T20:04:38.630332Z" 2023-05-31T20:04:38.832 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T20:04:38.832 INFO:teuthology.orchestra.run.smithi099.stdout:] 2023-05-31T20:04:38.833 DEBUG:rgw_multi.tests:master meta markers={0: '1_1685562631.516089_59.1', 1: '', 2: '1_1685562970.274572_93.1', 3: '1_1685563478.630332_193.1'} 2023-05-31T20:04:38.833 INFO:rgw_multi.tests:starting meta checkpoint for zone=a2 2023-05-31T20:04:38.834 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-31T20:04:38.834 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-31T20:04:38.835 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-05-31T20:04:38.884 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2023-05-31T20:04:38.885 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2023-05-31T20:04:38.924 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:04:38.919+0000 7f9a8e29b780 20 RGW-SYNC:meta: read sync status 2023-05-31T20:04:38.925 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:04:38.919+0000 7f9a8e29b780 20 RGW-SYNC:meta: read sync status 2023-05-31T20:04:38.938 INFO:teuthology.orchestra.run.smithi099.stdout:{ 2023-05-31T20:04:38.938 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_status": { 2023-05-31T20:04:38.938 INFO:teuthology.orchestra.run.smithi099.stdout: "info": { 2023-05-31T20:04:38.938 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "sync", 2023-05-31T20:04:38.939 INFO:teuthology.orchestra.run.smithi099.stdout: "num_shards": 4, 2023-05-31T20:04:38.939 INFO:teuthology.orchestra.run.smithi099.stdout: "period": "2e6123d2-d695-4350-9a01-61a455bc782b", 2023-05-31T20:04:38.939 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2023-05-31T20:04:38.939 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:04:38.939 INFO:teuthology.orchestra.run.smithi099.stdout: "markers": [ 2023-05-31T20:04:38.939 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T20:04:38.940 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 0, 2023-05-31T20:04:38.940 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T20:04:38.940 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2023-05-31T20:04:38.940 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1685562631.516089_59.1", 2023-05-31T20:04:38.940 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T20:04:38.940 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 1, 2023-05-31T20:04:38.940 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T20:04:38.941 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2023-05-31T19:50:31.516089Z", 2023-05-31T20:04:38.941 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2023-05-31T20:04:38.941 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T20:04:38.941 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:04:38.941 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T20:04:38.941 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 1, 2023-05-31T20:04:38.941 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T20:04:38.942 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2023-05-31T20:04:38.942 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "", 2023-05-31T20:04:38.942 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T20:04:38.942 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2023-05-31T20:04:38.942 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T20:04:38.942 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "0.000000", 2023-05-31T20:04:38.943 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2023-05-31T20:04:38.943 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T20:04:38.943 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:04:38.943 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T20:04:38.943 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 2, 2023-05-31T20:04:38.943 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T20:04:38.943 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2023-05-31T20:04:38.944 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1685562970.274572_93.1", 2023-05-31T20:04:38.944 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T20:04:38.944 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2023-05-31T20:04:38.944 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T20:04:38.944 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2023-05-31T19:56:10.274572Z", 2023-05-31T20:04:38.944 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2023-05-31T20:04:38.945 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T20:04:38.945 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:04:38.945 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T20:04:38.945 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 3, 2023-05-31T20:04:38.945 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T20:04:38.945 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2023-05-31T20:04:38.945 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1685563473.026869_185.1", 2023-05-31T20:04:38.946 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T20:04:38.946 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2023-05-31T20:04:38.946 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T20:04:38.946 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2023-05-31T20:04:33.026869Z", 2023-05-31T20:04:38.946 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2023-05-31T20:04:38.946 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T20:04:38.946 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T20:04:38.947 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2023-05-31T20:04:38.947 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:04:38.947 INFO:teuthology.orchestra.run.smithi099.stdout: "full_sync": { 2023-05-31T20:04:38.947 INFO:teuthology.orchestra.run.smithi099.stdout: "total": 1, 2023-05-31T20:04:38.947 INFO:teuthology.orchestra.run.smithi099.stdout: "complete": 1 2023-05-31T20:04:38.947 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:04:38.948 INFO:teuthology.orchestra.run.smithi099.stdout: "current_time": "2023-05-31T20:04:38Z" 2023-05-31T20:04:38.948 INFO:teuthology.orchestra.run.smithi099.stdout:} 2023-05-31T20:04:38.949 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "2e6123d2-d695-4350-9a01-61a455bc782b", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1685562631.516089_59.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-05-31T19:50:31.516089Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1685562970.274572_93.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-05-31T19:56:10.274572Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1685563473.026869_185.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-05-31T20:04:33.026869Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-05-31T20:04:38Z" } 2023-05-31T20:04:38.950 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1685562631.516089_59.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-05-31T19:50:31.516089Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1685562970.274572_93.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-05-31T19:56:10.274572Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1685563473.026869_185.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-05-31T20:04:33.026869Z', 'realm_epoch': 2}}] 2023-05-31T20:04:38.950 DEBUG:rgw_multi.tests:log_status={0: '1_1685562631.516089_59.1', 1: '', 2: '1_1685562970.274572_93.1', 3: '1_1685563478.630332_193.1'} 2023-05-31T20:04:38.950 DEBUG:rgw_multi.tests:sync_status={0: '1_1685562631.516089_59.1', 1: '', 2: '1_1685562970.274572_93.1', 3: '1_1685563473.026869_185.1'} 2023-05-31T20:04:38.951 WARNING:rgw_multi.tests:zone a2 behind master: shard=3 master=1_1685563478.630332_193.1 target=1_1685563473.026869_185.1 2023-05-31T20:04:43.952 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-31T20:04:43.952 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-31T20:04:43.953 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-05-31T20:04:44.011 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2023-05-31T20:04:44.011 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2023-05-31T20:04:44.065 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:04:44.061+0000 7f622d8c1780 20 RGW-SYNC:meta: read sync status 2023-05-31T20:04:44.066 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:04:44.062+0000 7f622d8c1780 20 RGW-SYNC:meta: read sync status 2023-05-31T20:04:44.081 INFO:teuthology.orchestra.run.smithi099.stdout:{ 2023-05-31T20:04:44.082 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_status": { 2023-05-31T20:04:44.082 INFO:teuthology.orchestra.run.smithi099.stdout: "info": { 2023-05-31T20:04:44.082 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "sync", 2023-05-31T20:04:44.082 INFO:teuthology.orchestra.run.smithi099.stdout: "num_shards": 4, 2023-05-31T20:04:44.082 INFO:teuthology.orchestra.run.smithi099.stdout: "period": "2e6123d2-d695-4350-9a01-61a455bc782b", 2023-05-31T20:04:44.082 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2023-05-31T20:04:44.083 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:04:44.083 INFO:teuthology.orchestra.run.smithi099.stdout: "markers": [ 2023-05-31T20:04:44.083 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T20:04:44.083 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 0, 2023-05-31T20:04:44.083 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T20:04:44.084 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2023-05-31T20:04:44.084 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1685562631.516089_59.1", 2023-05-31T20:04:44.084 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T20:04:44.085 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 1, 2023-05-31T20:04:44.085 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T20:04:44.085 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2023-05-31T19:50:31.516089Z", 2023-05-31T20:04:44.085 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2023-05-31T20:04:44.085 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T20:04:44.085 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:04:44.085 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T20:04:44.086 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 1, 2023-05-31T20:04:44.086 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T20:04:44.086 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2023-05-31T20:04:44.086 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "", 2023-05-31T20:04:44.087 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T20:04:44.087 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2023-05-31T20:04:44.087 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T20:04:44.088 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "0.000000", 2023-05-31T20:04:44.088 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2023-05-31T20:04:44.089 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T20:04:44.089 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:04:44.089 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T20:04:44.090 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 2, 2023-05-31T20:04:44.090 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T20:04:44.090 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2023-05-31T20:04:44.090 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1685562970.274572_93.1", 2023-05-31T20:04:44.091 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T20:04:44.091 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2023-05-31T20:04:44.092 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T20:04:44.092 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2023-05-31T19:56:10.274572Z", 2023-05-31T20:04:44.092 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2023-05-31T20:04:44.093 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T20:04:44.093 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:04:44.094 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T20:04:44.094 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 3, 2023-05-31T20:04:44.095 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T20:04:44.095 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2023-05-31T20:04:44.095 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1685563478.630332_193.1", 2023-05-31T20:04:44.095 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T20:04:44.096 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2023-05-31T20:04:44.096 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T20:04:44.096 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2023-05-31T20:04:38.630332Z", 2023-05-31T20:04:44.096 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2023-05-31T20:04:44.096 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T20:04:44.096 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T20:04:44.096 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2023-05-31T20:04:44.097 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:04:44.097 INFO:teuthology.orchestra.run.smithi099.stdout: "full_sync": { 2023-05-31T20:04:44.097 INFO:teuthology.orchestra.run.smithi099.stdout: "total": 1, 2023-05-31T20:04:44.097 INFO:teuthology.orchestra.run.smithi099.stdout: "complete": 1 2023-05-31T20:04:44.098 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:04:44.098 INFO:teuthology.orchestra.run.smithi099.stdout: "current_time": "2023-05-31T20:04:44Z" 2023-05-31T20:04:44.098 INFO:teuthology.orchestra.run.smithi099.stdout:} 2023-05-31T20:04:44.099 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "2e6123d2-d695-4350-9a01-61a455bc782b", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1685562631.516089_59.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-05-31T19:50:31.516089Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1685562970.274572_93.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-05-31T19:56:10.274572Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1685563478.630332_193.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-05-31T20:04:38.630332Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-05-31T20:04:44Z" } 2023-05-31T20:04:44.100 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1685562631.516089_59.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-05-31T19:50:31.516089Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1685562970.274572_93.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-05-31T19:56:10.274572Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1685563478.630332_193.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-05-31T20:04:38.630332Z', 'realm_epoch': 2}}] 2023-05-31T20:04:44.100 DEBUG:rgw_multi.tests:log_status={0: '1_1685562631.516089_59.1', 1: '', 2: '1_1685562970.274572_93.1', 3: '1_1685563478.630332_193.1'} 2023-05-31T20:04:44.101 DEBUG:rgw_multi.tests:sync_status={0: '1_1685562631.516089_59.1', 1: '', 2: '1_1685562970.274572_93.1', 3: '1_1685563478.630332_193.1'} 2023-05-31T20:04:44.101 INFO:rgw_multi.tests:finish meta checkpoint for zone=a2 2023-05-31T20:04:44.101 DEBUG:rgw_multi.tests:data checkpoint: source=a1 target=a2 2023-05-31T20:04:44.102 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['datalog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-31T20:04:44.102 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'datalog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-31T20:04:44.103 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 datalog status --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-05-31T20:04:44.152 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2023-05-31T20:04:44.152 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2023-05-31T20:04:44.215 INFO:teuthology.orchestra.run.smithi099.stdout:[ 2023-05-31T20:04:44.215 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T20:04:44.215 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "00000000000000000000:00000000000000001068", 2023-05-31T20:04:44.215 INFO:teuthology.orchestra.run.smithi099.stdout: "last_update": "2023-05-31T20:04:38.459334Z" 2023-05-31T20:04:44.216 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:04:44.216 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T20:04:44.216 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "", 2023-05-31T20:04:44.216 INFO:teuthology.orchestra.run.smithi099.stdout: "last_update": "0.000000" 2023-05-31T20:04:44.216 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:04:44.216 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T20:04:44.217 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "00000000000000000000:00000000000000004170", 2023-05-31T20:04:44.217 INFO:teuthology.orchestra.run.smithi099.stdout: "last_update": "2023-05-31T20:04:38.660140Z" 2023-05-31T20:04:44.217 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:04:44.217 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T20:04:44.217 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "00000000000000000000:00000000000000002905", 2023-05-31T20:04:44.217 INFO:teuthology.orchestra.run.smithi099.stdout: "last_update": "2023-05-31T20:00:18.227192Z" 2023-05-31T20:04:44.217 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T20:04:44.218 INFO:teuthology.orchestra.run.smithi099.stdout:] 2023-05-31T20:04:44.219 DEBUG:rgw_multi.tests:data markers for zone=a1 markers={0: '00000000000000000000:00000000000000001068', 1: '', 2: '00000000000000000000:00000000000000004170', 3: '00000000000000000000:00000000000000002905'} 2023-05-31T20:04:44.219 INFO:rgw_multi.tests:starting data checkpoint for target_zone=a2 source_zone=a1 2023-05-31T20:04:44.219 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-31T20:04:44.220 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-31T20:04:44.220 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-05-31T20:04:44.271 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2023-05-31T20:04:44.272 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2023-05-31T20:04:44.336 INFO:teuthology.orchestra.run.smithi099.stdout:{ 2023-05-31T20:04:44.337 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_status": { 2023-05-31T20:04:44.337 INFO:teuthology.orchestra.run.smithi099.stdout: "info": { 2023-05-31T20:04:44.337 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "sync", 2023-05-31T20:04:44.338 INFO:teuthology.orchestra.run.smithi099.stdout: "num_shards": 4, 2023-05-31T20:04:44.338 INFO:teuthology.orchestra.run.smithi099.stdout: "instance_id": 949931097149414600 2023-05-31T20:04:44.338 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:04:44.338 INFO:teuthology.orchestra.run.smithi099.stdout: "markers": [ 2023-05-31T20:04:44.338 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T20:04:44.338 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 0, 2023-05-31T20:04:44.339 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T20:04:44.339 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "incremental-sync", 2023-05-31T20:04:44.339 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "00000000000000000000:00000000000000000929", 2023-05-31T20:04:44.339 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T20:04:44.340 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2023-05-31T20:04:44.340 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T20:04:44.340 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2023-05-31T19:56:38.214723Z" 2023-05-31T20:04:44.340 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T20:04:44.341 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:04:44.341 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T20:04:44.341 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 1, 2023-05-31T20:04:44.341 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T20:04:44.341 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "incremental-sync", 2023-05-31T20:04:44.341 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "", 2023-05-31T20:04:44.342 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T20:04:44.342 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2023-05-31T20:04:44.342 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T20:04:44.343 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "0.000000" 2023-05-31T20:04:44.343 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T20:04:44.343 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:04:44.343 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T20:04:44.343 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 2, 2023-05-31T20:04:44.343 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T20:04:44.344 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "incremental-sync", 2023-05-31T20:04:44.344 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "00000000000000000000:00000000000000004029", 2023-05-31T20:04:44.344 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T20:04:44.344 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2023-05-31T20:04:44.344 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T20:04:44.345 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2023-05-31T20:04:20.238614Z" 2023-05-31T20:04:44.345 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T20:04:44.345 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:04:44.345 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T20:04:44.345 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 3, 2023-05-31T20:04:44.346 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T20:04:44.346 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "incremental-sync", 2023-05-31T20:04:44.346 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "00000000000000000000:00000000000000002905", 2023-05-31T20:04:44.346 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T20:04:44.346 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2023-05-31T20:04:44.346 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T20:04:44.347 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2023-05-31T20:00:18.227192Z" 2023-05-31T20:04:44.347 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T20:04:44.347 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T20:04:44.347 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2023-05-31T20:04:44.347 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:04:44.347 INFO:teuthology.orchestra.run.smithi099.stdout: "full_sync": { 2023-05-31T20:04:44.348 INFO:teuthology.orchestra.run.smithi099.stdout: "total": 0, 2023-05-31T20:04:44.348 INFO:teuthology.orchestra.run.smithi099.stdout: "complete": 0 2023-05-31T20:04:44.348 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:04:44.348 INFO:teuthology.orchestra.run.smithi099.stdout: "current_time": "2023-05-31T20:04:44Z" 2023-05-31T20:04:44.348 INFO:teuthology.orchestra.run.smithi099.stdout:} 2023-05-31T20:04:44.349 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 949931097149414600 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000000929", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-05-31T19:56:38.214723Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000004029", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-05-31T20:04:20.238614Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000002905", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-05-31T20:00:18.227192Z" } } ] }, "full_sync": { "total": 0, "complete": 0 }, "current_time": "2023-05-31T20:04:44Z" } 2023-05-31T20:04:44.350 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000000929', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-05-31T19:56:38.214723Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000004029', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-05-31T20:04:20.238614Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000002905', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-05-31T20:00:18.227192Z'}}] 2023-05-31T20:04:44.350 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000001068', 1: '', 2: '00000000000000000000:00000000000000004170', 3: '00000000000000000000:00000000000000002905'} 2023-05-31T20:04:44.350 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000000929', 1: '', 2: '00000000000000000000:00000000000000004029', 3: '00000000000000000000:00000000000000002905'} 2023-05-31T20:04:44.351 WARNING:rgw_multi.tests:data of zone a2 behind zone a1: shard=0 master=00000000000000000000:00000000000000001068 target=00000000000000000000:00000000000000000929, shard=2 master=00000000000000000000:00000000000000004170 target=00000000000000000000:00000000000000004029 2023-05-31T20:04:49.353 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-31T20:04:49.353 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-31T20:04:49.354 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-05-31T20:04:49.413 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2023-05-31T20:04:49.413 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2023-05-31T20:04:49.485 INFO:teuthology.orchestra.run.smithi099.stdout:{ 2023-05-31T20:04:49.486 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_status": { 2023-05-31T20:04:49.486 INFO:teuthology.orchestra.run.smithi099.stdout: "info": { 2023-05-31T20:04:49.487 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "sync", 2023-05-31T20:04:49.487 INFO:teuthology.orchestra.run.smithi099.stdout: "num_shards": 4, 2023-05-31T20:04:49.487 INFO:teuthology.orchestra.run.smithi099.stdout: "instance_id": 949931097149414600 2023-05-31T20:04:49.487 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:04:49.488 INFO:teuthology.orchestra.run.smithi099.stdout: "markers": [ 2023-05-31T20:04:49.488 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T20:04:49.488 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 0, 2023-05-31T20:04:49.488 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T20:04:49.489 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "incremental-sync", 2023-05-31T20:04:49.489 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "00000000000000000000:00000000000000001068", 2023-05-31T20:04:49.489 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T20:04:49.490 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2023-05-31T20:04:49.491 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T20:04:49.491 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2023-05-31T20:04:38.459334Z" 2023-05-31T20:04:49.491 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T20:04:49.491 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:04:49.492 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T20:04:49.492 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 1, 2023-05-31T20:04:49.492 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T20:04:49.493 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "incremental-sync", 2023-05-31T20:04:49.493 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "", 2023-05-31T20:04:49.493 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T20:04:49.493 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2023-05-31T20:04:49.493 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T20:04:49.494 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "0.000000" 2023-05-31T20:04:49.494 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T20:04:49.494 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:04:49.494 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T20:04:49.494 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 2, 2023-05-31T20:04:49.494 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T20:04:49.495 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "incremental-sync", 2023-05-31T20:04:49.495 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "00000000000000000000:00000000000000004311", 2023-05-31T20:04:49.495 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T20:04:49.496 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2023-05-31T20:04:49.496 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T20:04:49.496 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2023-05-31T20:04:46.796980Z" 2023-05-31T20:04:49.496 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T20:04:49.496 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:04:49.496 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T20:04:49.497 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 3, 2023-05-31T20:04:49.497 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T20:04:49.497 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "incremental-sync", 2023-05-31T20:04:49.497 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "00000000000000000000:00000000000000002905", 2023-05-31T20:04:49.497 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T20:04:49.498 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2023-05-31T20:04:49.498 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T20:04:49.498 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2023-05-31T20:00:18.227192Z" 2023-05-31T20:04:49.498 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T20:04:49.498 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T20:04:49.498 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2023-05-31T20:04:49.498 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:04:49.499 INFO:teuthology.orchestra.run.smithi099.stdout: "full_sync": { 2023-05-31T20:04:49.499 INFO:teuthology.orchestra.run.smithi099.stdout: "total": 0, 2023-05-31T20:04:49.499 INFO:teuthology.orchestra.run.smithi099.stdout: "complete": 0 2023-05-31T20:04:49.499 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:04:49.499 INFO:teuthology.orchestra.run.smithi099.stdout: "current_time": "2023-05-31T20:04:49Z" 2023-05-31T20:04:49.499 INFO:teuthology.orchestra.run.smithi099.stdout:} 2023-05-31T20:04:49.500 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 949931097149414600 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000001068", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-05-31T20:04:38.459334Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000004311", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-05-31T20:04:46.796980Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000002905", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-05-31T20:00:18.227192Z" } } ] }, "full_sync": { "total": 0, "complete": 0 }, "current_time": "2023-05-31T20:04:49Z" } 2023-05-31T20:04:49.501 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000001068', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-05-31T20:04:38.459334Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000004311', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-05-31T20:04:46.796980Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000002905', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-05-31T20:00:18.227192Z'}}] 2023-05-31T20:04:49.501 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000001068', 1: '', 2: '00000000000000000000:00000000000000004170', 3: '00000000000000000000:00000000000000002905'} 2023-05-31T20:04:49.502 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000001068', 1: '', 2: '00000000000000000000:00000000000000004311', 3: '00000000000000000000:00000000000000002905'} 2023-05-31T20:04:49.502 INFO:rgw_multi.tests:finished data checkpoint for target_zone=a2 source_zone=a1 2023-05-31T20:04:49.502 DEBUG:rgw_multi.tests:data checkpoint: source=a2 target=a1 2023-05-31T20:04:49.503 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['datalog', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-31T20:04:49.503 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'datalog', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-31T20:04:49.504 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 datalog status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-05-31T20:04:49.554 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2023-05-31T20:04:49.554 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2023-05-31T20:04:49.620 INFO:teuthology.orchestra.run.smithi099.stdout:[ 2023-05-31T20:04:49.620 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T20:04:49.621 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "00000000000000000000:00000000000000001210", 2023-05-31T20:04:49.621 INFO:teuthology.orchestra.run.smithi099.stdout: "last_update": "2023-05-31T20:04:45.842306Z" 2023-05-31T20:04:49.622 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:04:49.622 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T20:04:49.622 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "", 2023-05-31T20:04:49.622 INFO:teuthology.orchestra.run.smithi099.stdout: "last_update": "0.000000" 2023-05-31T20:04:49.623 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:04:49.623 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T20:04:49.623 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "00000000000000000000:00000000000000004311", 2023-05-31T20:04:49.624 INFO:teuthology.orchestra.run.smithi099.stdout: "last_update": "2023-05-31T20:04:47.151916Z" 2023-05-31T20:04:49.624 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:04:49.624 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T20:04:49.625 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "00000000000000000000:00000000000000002764", 2023-05-31T20:04:49.625 INFO:teuthology.orchestra.run.smithi099.stdout: "last_update": "2023-05-31T19:59:56.444860Z" 2023-05-31T20:04:49.625 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T20:04:49.626 INFO:teuthology.orchestra.run.smithi099.stdout:] 2023-05-31T20:04:49.627 DEBUG:rgw_multi.tests:data markers for zone=a2 markers={0: '00000000000000000000:00000000000000001210', 1: '', 2: '00000000000000000000:00000000000000004311', 3: '00000000000000000000:00000000000000002764'} 2023-05-31T20:04:49.628 INFO:rgw_multi.tests:starting data checkpoint for target_zone=a1 source_zone=a2 2023-05-31T20:04:49.629 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-31T20:04:49.630 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-31T20:04:49.631 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --source-zone a2 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-05-31T20:04:49.682 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2023-05-31T20:04:49.683 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2023-05-31T20:04:49.740 INFO:teuthology.orchestra.run.smithi099.stdout:{ 2023-05-31T20:04:49.741 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_status": { 2023-05-31T20:04:49.741 INFO:teuthology.orchestra.run.smithi099.stdout: "info": { 2023-05-31T20:04:49.741 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "sync", 2023-05-31T20:04:49.741 INFO:teuthology.orchestra.run.smithi099.stdout: "num_shards": 4, 2023-05-31T20:04:49.741 INFO:teuthology.orchestra.run.smithi099.stdout: "instance_id": 17941554979570305020 2023-05-31T20:04:49.741 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:04:49.742 INFO:teuthology.orchestra.run.smithi099.stdout: "markers": [ 2023-05-31T20:04:49.742 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T20:04:49.742 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 0, 2023-05-31T20:04:49.742 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T20:04:49.742 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "incremental-sync", 2023-05-31T20:04:49.742 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "00000000000000000000:00000000000000001068", 2023-05-31T20:04:49.743 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T20:04:49.743 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2023-05-31T20:04:49.743 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T20:04:49.743 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2023-05-31T20:04:38.541430Z" 2023-05-31T20:04:49.743 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T20:04:49.743 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:04:49.743 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T20:04:49.744 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 1, 2023-05-31T20:04:49.744 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T20:04:49.744 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "incremental-sync", 2023-05-31T20:04:49.744 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "", 2023-05-31T20:04:49.744 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T20:04:49.744 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2023-05-31T20:04:49.744 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T20:04:49.745 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "0.000000" 2023-05-31T20:04:49.745 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T20:04:49.745 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:04:49.745 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T20:04:49.745 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 2, 2023-05-31T20:04:49.745 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T20:04:49.745 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "incremental-sync", 2023-05-31T20:04:49.748 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "00000000000000000000:00000000000000004170", 2023-05-31T20:04:49.748 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T20:04:49.748 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2023-05-31T20:04:49.748 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T20:04:49.748 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2023-05-31T20:04:38.682491Z" 2023-05-31T20:04:49.749 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T20:04:49.749 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:04:49.749 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T20:04:49.749 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 3, 2023-05-31T20:04:49.749 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T20:04:49.749 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "incremental-sync", 2023-05-31T20:04:49.750 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "00000000000000000000:00000000000000002764", 2023-05-31T20:04:49.750 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T20:04:49.750 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2023-05-31T20:04:49.750 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T20:04:49.750 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2023-05-31T19:59:56.444860Z" 2023-05-31T20:04:49.750 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T20:04:49.751 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T20:04:49.751 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2023-05-31T20:04:49.751 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:04:49.751 INFO:teuthology.orchestra.run.smithi099.stdout: "full_sync": { 2023-05-31T20:04:49.752 INFO:teuthology.orchestra.run.smithi099.stdout: "total": 0, 2023-05-31T20:04:49.752 INFO:teuthology.orchestra.run.smithi099.stdout: "complete": 0 2023-05-31T20:04:49.752 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:04:49.752 INFO:teuthology.orchestra.run.smithi099.stdout: "current_time": "2023-05-31T20:04:49Z" 2023-05-31T20:04:49.752 INFO:teuthology.orchestra.run.smithi099.stdout:} 2023-05-31T20:04:49.753 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 17941554979570305020 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000001068", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-05-31T20:04:38.541430Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000004170", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-05-31T20:04:38.682491Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000002764", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-05-31T19:59:56.444860Z" } } ] }, "full_sync": { "total": 0, "complete": 0 }, "current_time": "2023-05-31T20:04:49Z" } 2023-05-31T20:04:49.754 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000001068', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-05-31T20:04:38.541430Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000004170', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-05-31T20:04:38.682491Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000002764', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-05-31T19:59:56.444860Z'}}] 2023-05-31T20:04:49.754 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000001210', 1: '', 2: '00000000000000000000:00000000000000004311', 3: '00000000000000000000:00000000000000002764'} 2023-05-31T20:04:49.755 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000001068', 1: '', 2: '00000000000000000000:00000000000000004170', 3: '00000000000000000000:00000000000000002764'} 2023-05-31T20:04:49.755 WARNING:rgw_multi.tests:data of zone a1 behind zone a2: shard=0 master=00000000000000000000:00000000000000001210 target=00000000000000000000:00000000000000001068, shard=2 master=00000000000000000000:00000000000000004311 target=00000000000000000000:00000000000000004170 2023-05-31T20:04:54.756 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-31T20:04:54.757 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-31T20:04:54.758 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --source-zone a2 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-05-31T20:04:54.817 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2023-05-31T20:04:54.817 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2023-05-31T20:04:54.886 INFO:teuthology.orchestra.run.smithi099.stdout:{ 2023-05-31T20:04:54.887 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_status": { 2023-05-31T20:04:54.887 INFO:teuthology.orchestra.run.smithi099.stdout: "info": { 2023-05-31T20:04:54.887 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "sync", 2023-05-31T20:04:54.887 INFO:teuthology.orchestra.run.smithi099.stdout: "num_shards": 4, 2023-05-31T20:04:54.887 INFO:teuthology.orchestra.run.smithi099.stdout: "instance_id": 17941554979570305020 2023-05-31T20:04:54.888 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:04:54.888 INFO:teuthology.orchestra.run.smithi099.stdout: "markers": [ 2023-05-31T20:04:54.888 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T20:04:54.888 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 0, 2023-05-31T20:04:54.888 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T20:04:54.888 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "incremental-sync", 2023-05-31T20:04:54.889 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "00000000000000000000:00000000000000001068", 2023-05-31T20:04:54.889 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T20:04:54.889 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2023-05-31T20:04:54.889 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T20:04:54.889 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2023-05-31T20:04:38.541430Z" 2023-05-31T20:04:54.890 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T20:04:54.890 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:04:54.890 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T20:04:54.890 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 1, 2023-05-31T20:04:54.890 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T20:04:54.890 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "incremental-sync", 2023-05-31T20:04:54.891 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "", 2023-05-31T20:04:54.891 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T20:04:54.891 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2023-05-31T20:04:54.891 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T20:04:54.891 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "0.000000" 2023-05-31T20:04:54.891 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T20:04:54.891 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:04:54.892 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T20:04:54.892 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 2, 2023-05-31T20:04:54.892 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T20:04:54.892 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "incremental-sync", 2023-05-31T20:04:54.892 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "00000000000000000000:00000000000000004170", 2023-05-31T20:04:54.892 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T20:04:54.893 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2023-05-31T20:04:54.893 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T20:04:54.893 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2023-05-31T20:04:38.682491Z" 2023-05-31T20:04:54.893 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T20:04:54.893 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:04:54.893 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T20:04:54.894 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 3, 2023-05-31T20:04:54.894 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T20:04:54.894 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "incremental-sync", 2023-05-31T20:04:54.894 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "00000000000000000000:00000000000000002764", 2023-05-31T20:04:54.894 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T20:04:54.894 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2023-05-31T20:04:54.895 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T20:04:54.895 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2023-05-31T19:59:56.444860Z" 2023-05-31T20:04:54.895 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T20:04:54.896 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T20:04:54.896 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2023-05-31T20:04:54.896 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:04:54.897 INFO:teuthology.orchestra.run.smithi099.stdout: "full_sync": { 2023-05-31T20:04:54.897 INFO:teuthology.orchestra.run.smithi099.stdout: "total": 0, 2023-05-31T20:04:54.897 INFO:teuthology.orchestra.run.smithi099.stdout: "complete": 0 2023-05-31T20:04:54.897 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:04:54.897 INFO:teuthology.orchestra.run.smithi099.stdout: "current_time": "2023-05-31T20:04:54Z" 2023-05-31T20:04:54.897 INFO:teuthology.orchestra.run.smithi099.stdout:} 2023-05-31T20:04:54.898 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 17941554979570305020 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000001068", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-05-31T20:04:38.541430Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000004170", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-05-31T20:04:38.682491Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000002764", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-05-31T19:59:56.444860Z" } } ] }, "full_sync": { "total": 0, "complete": 0 }, "current_time": "2023-05-31T20:04:54Z" } 2023-05-31T20:04:54.899 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000001068', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-05-31T20:04:38.541430Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000004170', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-05-31T20:04:38.682491Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000002764', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-05-31T19:59:56.444860Z'}}] 2023-05-31T20:04:54.899 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000001210', 1: '', 2: '00000000000000000000:00000000000000004311', 3: '00000000000000000000:00000000000000002764'} 2023-05-31T20:04:54.900 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000001068', 1: '', 2: '00000000000000000000:00000000000000004170', 3: '00000000000000000000:00000000000000002764'} 2023-05-31T20:04:54.900 WARNING:rgw_multi.tests:data of zone a1 behind zone a2: shard=0 master=00000000000000000000:00000000000000001210 target=00000000000000000000:00000000000000001068, shard=2 master=00000000000000000000:00000000000000004311 target=00000000000000000000:00000000000000004170 2023-05-31T20:04:59.901 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-31T20:04:59.902 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-31T20:04:59.902 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --source-zone a2 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-05-31T20:04:59.963 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2023-05-31T20:04:59.963 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2023-05-31T20:05:00.014 INFO:teuthology.orchestra.run.smithi099.stdout:{ 2023-05-31T20:05:00.015 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_status": { 2023-05-31T20:05:00.015 INFO:teuthology.orchestra.run.smithi099.stdout: "info": { 2023-05-31T20:05:00.016 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "sync", 2023-05-31T20:05:00.016 INFO:teuthology.orchestra.run.smithi099.stdout: "num_shards": 4, 2023-05-31T20:05:00.016 INFO:teuthology.orchestra.run.smithi099.stdout: "instance_id": 17941554979570305020 2023-05-31T20:05:00.016 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:05:00.017 INFO:teuthology.orchestra.run.smithi099.stdout: "markers": [ 2023-05-31T20:05:00.017 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T20:05:00.017 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 0, 2023-05-31T20:05:00.017 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T20:05:00.018 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "incremental-sync", 2023-05-31T20:05:00.018 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "00000000000000000000:00000000000000001068", 2023-05-31T20:05:00.018 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T20:05:00.018 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2023-05-31T20:05:00.018 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T20:05:00.019 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2023-05-31T20:04:38.541430Z" 2023-05-31T20:05:00.019 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T20:05:00.019 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:05:00.019 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T20:05:00.019 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 1, 2023-05-31T20:05:00.019 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T20:05:00.019 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "incremental-sync", 2023-05-31T20:05:00.020 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "", 2023-05-31T20:05:00.020 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T20:05:00.020 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2023-05-31T20:05:00.020 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T20:05:00.020 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "0.000000" 2023-05-31T20:05:00.020 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T20:05:00.021 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:05:00.021 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T20:05:00.021 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 2, 2023-05-31T20:05:00.021 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T20:05:00.021 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "incremental-sync", 2023-05-31T20:05:00.021 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "00000000000000000000:00000000000000004170", 2023-05-31T20:05:00.022 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T20:05:00.022 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2023-05-31T20:05:00.022 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T20:05:00.022 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2023-05-31T20:04:38.682491Z" 2023-05-31T20:05:00.022 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T20:05:00.022 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:05:00.023 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T20:05:00.023 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 3, 2023-05-31T20:05:00.023 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T20:05:00.023 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "incremental-sync", 2023-05-31T20:05:00.023 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "00000000000000000000:00000000000000002764", 2023-05-31T20:05:00.023 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T20:05:00.023 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2023-05-31T20:05:00.024 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T20:05:00.024 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2023-05-31T19:59:56.444860Z" 2023-05-31T20:05:00.024 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T20:05:00.024 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T20:05:00.025 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2023-05-31T20:05:00.025 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:05:00.025 INFO:teuthology.orchestra.run.smithi099.stdout: "full_sync": { 2023-05-31T20:05:00.025 INFO:teuthology.orchestra.run.smithi099.stdout: "total": 0, 2023-05-31T20:05:00.025 INFO:teuthology.orchestra.run.smithi099.stdout: "complete": 0 2023-05-31T20:05:00.026 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:05:00.026 INFO:teuthology.orchestra.run.smithi099.stdout: "current_time": "2023-05-31T20:04:59Z" 2023-05-31T20:05:00.026 INFO:teuthology.orchestra.run.smithi099.stdout:} 2023-05-31T20:05:00.029 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 17941554979570305020 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000001068", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-05-31T20:04:38.541430Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000004170", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-05-31T20:04:38.682491Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000002764", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-05-31T19:59:56.444860Z" } } ] }, "full_sync": { "total": 0, "complete": 0 }, "current_time": "2023-05-31T20:04:59Z" } 2023-05-31T20:05:00.030 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000001068', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-05-31T20:04:38.541430Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000004170', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-05-31T20:04:38.682491Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000002764', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-05-31T19:59:56.444860Z'}}] 2023-05-31T20:05:00.031 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000001210', 1: '', 2: '00000000000000000000:00000000000000004311', 3: '00000000000000000000:00000000000000002764'} 2023-05-31T20:05:00.031 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000001068', 1: '', 2: '00000000000000000000:00000000000000004170', 3: '00000000000000000000:00000000000000002764'} 2023-05-31T20:05:00.033 WARNING:rgw_multi.tests:data of zone a1 behind zone a2: shard=0 master=00000000000000000000:00000000000000001210 target=00000000000000000000:00000000000000001068, shard=2 master=00000000000000000000:00000000000000004311 target=00000000000000000000:00000000000000004170 2023-05-31T20:05:05.034 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-31T20:05:05.034 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-31T20:05:05.035 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --source-zone a2 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-05-31T20:05:05.095 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2023-05-31T20:05:05.095 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2023-05-31T20:05:05.159 INFO:teuthology.orchestra.run.smithi099.stdout:{ 2023-05-31T20:05:05.160 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_status": { 2023-05-31T20:05:05.160 INFO:teuthology.orchestra.run.smithi099.stdout: "info": { 2023-05-31T20:05:05.160 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "sync", 2023-05-31T20:05:05.161 INFO:teuthology.orchestra.run.smithi099.stdout: "num_shards": 4, 2023-05-31T20:05:05.161 INFO:teuthology.orchestra.run.smithi099.stdout: "instance_id": 17941554979570305020 2023-05-31T20:05:05.161 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:05:05.162 INFO:teuthology.orchestra.run.smithi099.stdout: "markers": [ 2023-05-31T20:05:05.162 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T20:05:05.162 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 0, 2023-05-31T20:05:05.163 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T20:05:05.163 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "incremental-sync", 2023-05-31T20:05:05.163 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "00000000000000000000:00000000000000001068", 2023-05-31T20:05:05.163 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T20:05:05.163 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2023-05-31T20:05:05.163 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T20:05:05.164 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2023-05-31T20:04:38.541430Z" 2023-05-31T20:05:05.164 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T20:05:05.164 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:05:05.164 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T20:05:05.165 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 1, 2023-05-31T20:05:05.165 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T20:05:05.166 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "incremental-sync", 2023-05-31T20:05:05.166 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "", 2023-05-31T20:05:05.167 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T20:05:05.167 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2023-05-31T20:05:05.167 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T20:05:05.167 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "0.000000" 2023-05-31T20:05:05.167 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T20:05:05.167 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:05:05.167 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T20:05:05.168 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 2, 2023-05-31T20:05:05.168 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T20:05:05.168 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "incremental-sync", 2023-05-31T20:05:05.168 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "00000000000000000000:00000000000000004170", 2023-05-31T20:05:05.168 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T20:05:05.169 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2023-05-31T20:05:05.169 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T20:05:05.169 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2023-05-31T20:04:38.682491Z" 2023-05-31T20:05:05.170 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T20:05:05.170 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:05:05.170 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T20:05:05.170 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 3, 2023-05-31T20:05:05.171 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T20:05:05.171 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "incremental-sync", 2023-05-31T20:05:05.171 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "00000000000000000000:00000000000000002764", 2023-05-31T20:05:05.172 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T20:05:05.172 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2023-05-31T20:05:05.172 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T20:05:05.173 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2023-05-31T19:59:56.444860Z" 2023-05-31T20:05:05.173 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T20:05:05.173 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T20:05:05.174 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2023-05-31T20:05:05.174 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:05:05.174 INFO:teuthology.orchestra.run.smithi099.stdout: "full_sync": { 2023-05-31T20:05:05.175 INFO:teuthology.orchestra.run.smithi099.stdout: "total": 0, 2023-05-31T20:05:05.175 INFO:teuthology.orchestra.run.smithi099.stdout: "complete": 0 2023-05-31T20:05:05.175 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:05:05.176 INFO:teuthology.orchestra.run.smithi099.stdout: "current_time": "2023-05-31T20:05:05Z" 2023-05-31T20:05:05.176 INFO:teuthology.orchestra.run.smithi099.stdout:} 2023-05-31T20:05:05.178 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 17941554979570305020 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000001068", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-05-31T20:04:38.541430Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000004170", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-05-31T20:04:38.682491Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000002764", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-05-31T19:59:56.444860Z" } } ] }, "full_sync": { "total": 0, "complete": 0 }, "current_time": "2023-05-31T20:05:05Z" } 2023-05-31T20:05:05.178 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000001068', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-05-31T20:04:38.541430Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000004170', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-05-31T20:04:38.682491Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000002764', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-05-31T19:59:56.444860Z'}}] 2023-05-31T20:05:05.179 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000001210', 1: '', 2: '00000000000000000000:00000000000000004311', 3: '00000000000000000000:00000000000000002764'} 2023-05-31T20:05:05.180 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000001068', 1: '', 2: '00000000000000000000:00000000000000004170', 3: '00000000000000000000:00000000000000002764'} 2023-05-31T20:05:05.180 WARNING:rgw_multi.tests:data of zone a1 behind zone a2: shard=0 master=00000000000000000000:00000000000000001210 target=00000000000000000000:00000000000000001068, shard=2 master=00000000000000000000:00000000000000004311 target=00000000000000000000:00000000000000004170 2023-05-31T20:05:10.181 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-31T20:05:10.182 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-31T20:05:10.182 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --source-zone a2 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-05-31T20:05:10.239 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2023-05-31T20:05:10.239 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2023-05-31T20:05:10.298 INFO:teuthology.orchestra.run.smithi099.stdout:{ 2023-05-31T20:05:10.299 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_status": { 2023-05-31T20:05:10.299 INFO:teuthology.orchestra.run.smithi099.stdout: "info": { 2023-05-31T20:05:10.299 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "sync", 2023-05-31T20:05:10.299 INFO:teuthology.orchestra.run.smithi099.stdout: "num_shards": 4, 2023-05-31T20:05:10.299 INFO:teuthology.orchestra.run.smithi099.stdout: "instance_id": 17941554979570305020 2023-05-31T20:05:10.299 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:05:10.300 INFO:teuthology.orchestra.run.smithi099.stdout: "markers": [ 2023-05-31T20:05:10.300 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T20:05:10.300 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 0, 2023-05-31T20:05:10.300 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T20:05:10.300 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "incremental-sync", 2023-05-31T20:05:10.300 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "00000000000000000000:00000000000000001210", 2023-05-31T20:05:10.301 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T20:05:10.301 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2023-05-31T20:05:10.301 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T20:05:10.301 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2023-05-31T20:04:45.842306Z" 2023-05-31T20:05:10.301 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T20:05:10.301 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:05:10.301 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T20:05:10.302 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 1, 2023-05-31T20:05:10.302 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T20:05:10.302 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "incremental-sync", 2023-05-31T20:05:10.302 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "", 2023-05-31T20:05:10.302 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T20:05:10.302 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2023-05-31T20:05:10.303 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T20:05:10.303 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "0.000000" 2023-05-31T20:05:10.303 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T20:05:10.303 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:05:10.303 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T20:05:10.303 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 2, 2023-05-31T20:05:10.303 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T20:05:10.304 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "incremental-sync", 2023-05-31T20:05:10.304 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "00000000000000000000:00000000000000004311", 2023-05-31T20:05:10.304 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T20:05:10.304 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2023-05-31T20:05:10.304 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T20:05:10.304 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2023-05-31T20:04:47.151916Z" 2023-05-31T20:05:10.305 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T20:05:10.305 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:05:10.305 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T20:05:10.305 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 3, 2023-05-31T20:05:10.305 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T20:05:10.305 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "incremental-sync", 2023-05-31T20:05:10.305 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "00000000000000000000:00000000000000002764", 2023-05-31T20:05:10.306 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T20:05:10.306 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2023-05-31T20:05:10.306 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T20:05:10.306 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2023-05-31T19:59:56.444860Z" 2023-05-31T20:05:10.306 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T20:05:10.306 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T20:05:10.306 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2023-05-31T20:05:10.307 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:05:10.307 INFO:teuthology.orchestra.run.smithi099.stdout: "full_sync": { 2023-05-31T20:05:10.307 INFO:teuthology.orchestra.run.smithi099.stdout: "total": 0, 2023-05-31T20:05:10.307 INFO:teuthology.orchestra.run.smithi099.stdout: "complete": 0 2023-05-31T20:05:10.307 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:05:10.307 INFO:teuthology.orchestra.run.smithi099.stdout: "current_time": "2023-05-31T20:05:10Z" 2023-05-31T20:05:10.307 INFO:teuthology.orchestra.run.smithi099.stdout:} 2023-05-31T20:05:10.308 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 17941554979570305020 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000001210", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-05-31T20:04:45.842306Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000004311", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-05-31T20:04:47.151916Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000002764", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-05-31T19:59:56.444860Z" } } ] }, "full_sync": { "total": 0, "complete": 0 }, "current_time": "2023-05-31T20:05:10Z" } 2023-05-31T20:05:10.309 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000001210', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-05-31T20:04:45.842306Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000004311', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-05-31T20:04:47.151916Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000002764', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-05-31T19:59:56.444860Z'}}] 2023-05-31T20:05:10.309 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000001210', 1: '', 2: '00000000000000000000:00000000000000004311', 3: '00000000000000000000:00000000000000002764'} 2023-05-31T20:05:10.310 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000001210', 1: '', 2: '00000000000000000000:00000000000000004311', 3: '00000000000000000000:00000000000000002764'} 2023-05-31T20:05:10.310 INFO:rgw_multi.tests:finished data checkpoint for target_zone=a1 source_zone=a2 2023-05-31T20:05:10.310 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['datalog', 'status', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-31T20:05:10.311 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'datalog', 'status', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-31T20:05:10.311 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 datalog status --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-05-31T20:05:10.356 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2023-05-31T20:05:10.356 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2023-05-31T20:05:10.408 INFO:teuthology.orchestra.run.smithi099.stdout:[ 2023-05-31T20:05:10.408 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T20:05:10.408 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "00000000000000000000:00000000000000001210", 2023-05-31T20:05:10.409 INFO:teuthology.orchestra.run.smithi099.stdout: "last_update": "2023-05-31T20:04:45.921417Z" 2023-05-31T20:05:10.409 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:05:10.409 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T20:05:10.409 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "", 2023-05-31T20:05:10.409 INFO:teuthology.orchestra.run.smithi099.stdout: "last_update": "0.000000" 2023-05-31T20:05:10.409 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:05:10.409 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T20:05:10.410 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "00000000000000000000:00000000000000004311", 2023-05-31T20:05:10.410 INFO:teuthology.orchestra.run.smithi099.stdout: "last_update": "2023-05-31T20:04:46.796980Z" 2023-05-31T20:05:10.410 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:05:10.410 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T20:05:10.410 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "00000000000000000000:00000000000000002905", 2023-05-31T20:05:10.410 INFO:teuthology.orchestra.run.smithi099.stdout: "last_update": "2023-05-31T20:00:18.227192Z" 2023-05-31T20:05:10.410 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T20:05:10.411 INFO:teuthology.orchestra.run.smithi099.stdout:] 2023-05-31T20:05:10.412 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['datalog', 'autotrim', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-05-31T20:05:10.412 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'datalog', 'autotrim', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-05-31T20:05:10.413 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 datalog autotrim --cluster c1 --debug-rgw 0 --debug-ms 0 2023-05-31T20:05:10.456 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2023-05-31T20:05:10.457 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2023-05-31T20:05:10.532 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['datalog', 'list', '--shard-id', '0', '--max-entries', '1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-31T20:05:10.532 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'datalog', 'list', '--shard-id', '0', '--max-entries', '1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-31T20:05:10.533 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 datalog list --shard-id 0 --max-entries 1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-05-31T20:05:10.587 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2023-05-31T20:05:10.588 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2023-05-31T20:05:10.632 INFO:teuthology.orchestra.run.smithi099.stdout:[] 2023-05-31T20:05:10.634 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['datalog', 'list', '--shard-id', '2', '--max-entries', '1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-31T20:05:10.635 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'datalog', 'list', '--shard-id', '2', '--max-entries', '1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-31T20:05:10.635 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 datalog list --shard-id 2 --max-entries 1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-05-31T20:05:10.685 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2023-05-31T20:05:10.685 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2023-05-31T20:05:10.727 INFO:teuthology.orchestra.run.smithi099.stdout:[] 2023-05-31T20:05:10.728 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['datalog', 'list', '--shard-id', '3', '--max-entries', '1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-31T20:05:10.729 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'datalog', 'list', '--shard-id', '3', '--max-entries', '1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-31T20:05:10.730 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 datalog list --shard-id 3 --max-entries 1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-05-31T20:05:10.779 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2023-05-31T20:05:10.780 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2023-05-31T20:05:10.851 INFO:teuthology.orchestra.run.smithi099.stdout:[] 2023-05-31T20:05:10.853 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['datalog', 'status', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-31T20:05:10.854 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'datalog', 'status', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-31T20:05:10.855 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 datalog status --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-05-31T20:05:10.901 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2023-05-31T20:05:10.902 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2023-05-31T20:05:10.965 INFO:teuthology.orchestra.run.smithi099.stdout:[ 2023-05-31T20:05:10.965 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T20:05:10.966 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "00000000000000000000:00000000000000001210", 2023-05-31T20:05:10.966 INFO:teuthology.orchestra.run.smithi099.stdout: "last_update": "2023-05-31T20:04:45.921417Z" 2023-05-31T20:05:10.966 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:05:10.966 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T20:05:10.966 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "", 2023-05-31T20:05:10.966 INFO:teuthology.orchestra.run.smithi099.stdout: "last_update": "0.000000" 2023-05-31T20:05:10.966 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:05:10.967 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T20:05:10.967 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "00000000000000000000:00000000000000004311", 2023-05-31T20:05:10.967 INFO:teuthology.orchestra.run.smithi099.stdout: "last_update": "2023-05-31T20:04:46.796980Z" 2023-05-31T20:05:10.967 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:05:10.967 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T20:05:10.967 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "00000000000000000000:00000000000000002905", 2023-05-31T20:05:10.967 INFO:teuthology.orchestra.run.smithi099.stdout: "last_update": "2023-05-31T20:00:18.227192Z" 2023-05-31T20:05:10.968 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T20:05:10.968 INFO:teuthology.orchestra.run.smithi099.stdout:] 2023-05-31T20:05:10.969 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['datalog', 'autotrim', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-05-31T20:05:10.969 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'datalog', 'autotrim', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-05-31T20:05:10.970 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 datalog autotrim --cluster c1 --debug-rgw 0 --debug-ms 0 2023-05-31T20:05:11.014 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2023-05-31T20:05:11.014 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2023-05-31T20:05:11.077 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['datalog', 'list', '--shard-id', '0', '--max-entries', '1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-31T20:05:11.078 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'datalog', 'list', '--shard-id', '0', '--max-entries', '1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-31T20:05:11.078 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 datalog list --shard-id 0 --max-entries 1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-05-31T20:05:11.128 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2023-05-31T20:05:11.128 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2023-05-31T20:05:11.205 INFO:teuthology.orchestra.run.smithi099.stdout:[] 2023-05-31T20:05:11.206 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['datalog', 'list', '--shard-id', '2', '--max-entries', '1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-31T20:05:11.207 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'datalog', 'list', '--shard-id', '2', '--max-entries', '1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-31T20:05:11.207 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 datalog list --shard-id 2 --max-entries 1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-05-31T20:05:11.257 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2023-05-31T20:05:11.257 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2023-05-31T20:05:11.316 INFO:teuthology.orchestra.run.smithi099.stdout:[] 2023-05-31T20:05:11.319 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['datalog', 'list', '--shard-id', '3', '--max-entries', '1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-31T20:05:11.320 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'datalog', 'list', '--shard-id', '3', '--max-entries', '1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-31T20:05:11.321 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 datalog list --shard-id 3 --max-entries 1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-05-31T20:05:11.371 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2023-05-31T20:05:11.372 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2023-05-31T20:05:11.412 INFO:teuthology.orchestra.run.smithi099.stdout:[] 2023-05-31T20:05:11.413 INFO:tasks.rgw_multisite_tests:rgw_multi.tests.test_datalog_autotrim ... ok 2023-05-31T20:05:11.414 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['zone', 'modify', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--sync-from-all=false', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-05-31T20:05:11.414 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'zone', 'modify', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--sync-from-all=false', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-05-31T20:05:11.415 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 zone modify --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --sync-from-all=false --cluster c1 --debug-rgw 0 --debug-ms 0 2023-05-31T20:05:11.464 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2023-05-31T20:05:11.464 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2023-05-31T20:05:11.483 INFO:teuthology.orchestra.run.smithi099.stdout:{ 2023-05-31T20:05:11.484 INFO:teuthology.orchestra.run.smithi099.stdout: "id": "5d8acec7-7e57-4e56-8b32-e3d6b5036001", 2023-05-31T20:05:11.484 INFO:teuthology.orchestra.run.smithi099.stdout: "name": "a2", 2023-05-31T20:05:11.484 INFO:teuthology.orchestra.run.smithi099.stdout: "domain_root": "a2.rgw.meta:root", 2023-05-31T20:05:11.484 INFO:teuthology.orchestra.run.smithi099.stdout: "control_pool": "a2.rgw.control", 2023-05-31T20:05:11.485 INFO:teuthology.orchestra.run.smithi099.stdout: "gc_pool": "a2.rgw.log:gc", 2023-05-31T20:05:11.485 INFO:teuthology.orchestra.run.smithi099.stdout: "lc_pool": "a2.rgw.log:lc", 2023-05-31T20:05:11.485 INFO:teuthology.orchestra.run.smithi099.stdout: "log_pool": "a2.rgw.log", 2023-05-31T20:05:11.485 INFO:teuthology.orchestra.run.smithi099.stdout: "intent_log_pool": "a2.rgw.log:intent", 2023-05-31T20:05:11.486 INFO:teuthology.orchestra.run.smithi099.stdout: "usage_log_pool": "a2.rgw.log:usage", 2023-05-31T20:05:11.486 INFO:teuthology.orchestra.run.smithi099.stdout: "roles_pool": "a2.rgw.meta:roles", 2023-05-31T20:05:11.486 INFO:teuthology.orchestra.run.smithi099.stdout: "reshard_pool": "a2.rgw.log:reshard", 2023-05-31T20:05:11.486 INFO:teuthology.orchestra.run.smithi099.stdout: "user_keys_pool": "a2.rgw.meta:users.keys", 2023-05-31T20:05:11.487 INFO:teuthology.orchestra.run.smithi099.stdout: "user_email_pool": "a2.rgw.meta:users.email", 2023-05-31T20:05:11.487 INFO:teuthology.orchestra.run.smithi099.stdout: "user_swift_pool": "a2.rgw.meta:users.swift", 2023-05-31T20:05:11.487 INFO:teuthology.orchestra.run.smithi099.stdout: "user_uid_pool": "a2.rgw.meta:users.uid", 2023-05-31T20:05:11.487 INFO:teuthology.orchestra.run.smithi099.stdout: "otp_pool": "a2.rgw.otp", 2023-05-31T20:05:11.487 INFO:teuthology.orchestra.run.smithi099.stdout: "system_key": { 2023-05-31T20:05:11.487 INFO:teuthology.orchestra.run.smithi099.stdout: "access_key": "RVN19ED3W7FH7E08", 2023-05-31T20:05:11.487 INFO:teuthology.orchestra.run.smithi099.stdout: "secret_key": "XzQjGe60gbIUStGfs96TKhuPw3IQi3hK" 2023-05-31T20:05:11.488 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:05:11.488 INFO:teuthology.orchestra.run.smithi099.stdout: "placement_pools": [ 2023-05-31T20:05:11.488 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T20:05:11.489 INFO:teuthology.orchestra.run.smithi099.stdout: "key": "default-placement", 2023-05-31T20:05:11.489 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T20:05:11.489 INFO:teuthology.orchestra.run.smithi099.stdout: "index_pool": "a2.rgw.buckets.index", 2023-05-31T20:05:11.489 INFO:teuthology.orchestra.run.smithi099.stdout: "storage_classes": { 2023-05-31T20:05:11.490 INFO:teuthology.orchestra.run.smithi099.stdout: "STANDARD": { 2023-05-31T20:05:11.490 INFO:teuthology.orchestra.run.smithi099.stdout: "data_pool": "a2.rgw.buckets.data", 2023-05-31T20:05:11.490 INFO:teuthology.orchestra.run.smithi099.stdout: "compression_type": "random" 2023-05-31T20:05:11.491 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T20:05:11.491 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:05:11.491 INFO:teuthology.orchestra.run.smithi099.stdout: "data_extra_pool": "a2.rgw.buckets.non-ec", 2023-05-31T20:05:11.492 INFO:teuthology.orchestra.run.smithi099.stdout: "index_type": 0, 2023-05-31T20:05:11.492 INFO:teuthology.orchestra.run.smithi099.stdout: "inline_data": true 2023-05-31T20:05:11.492 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T20:05:11.493 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T20:05:11.493 INFO:teuthology.orchestra.run.smithi099.stdout: ], 2023-05-31T20:05:11.493 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_id": "aaa33de6-515b-4cac-8f1b-3a1a30452117", 2023-05-31T20:05:11.494 INFO:teuthology.orchestra.run.smithi099.stdout: "notif_pool": "a2.rgw.log:notif" 2023-05-31T20:05:11.494 INFO:teuthology.orchestra.run.smithi099.stdout:} 2023-05-31T20:05:11.495 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['period', 'update', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-05-31T20:05:11.496 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'period', 'update', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-05-31T20:05:11.497 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 period update --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --commit --cluster c1 --debug-rgw 0 --debug-ms 0 2023-05-31T20:05:11.546 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2023-05-31T20:05:11.547 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2023-05-31T20:05:11.589 INFO:teuthology.orchestra.run.smithi099.stderr:Sending period to new master zone 3d163b1e-09c7-47e2-ae87-add4aace3f38 2023-05-31T20:05:11.614 INFO:teuthology.orchestra.run.smithi099.stdout:{ 2023-05-31T20:05:11.615 INFO:teuthology.orchestra.run.smithi099.stdout: "id": "2e6123d2-d695-4350-9a01-61a455bc782b", 2023-05-31T20:05:11.615 INFO:teuthology.orchestra.run.smithi099.stdout: "epoch": 5, 2023-05-31T20:05:11.615 INFO:teuthology.orchestra.run.smithi099.stdout: "predecessor_uuid": "629b9ced-80af-41ed-8a6c-f2cfc334cbb4", 2023-05-31T20:05:11.615 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_status": [], 2023-05-31T20:05:11.615 INFO:teuthology.orchestra.run.smithi099.stdout: "period_map": { 2023-05-31T20:05:11.615 INFO:teuthology.orchestra.run.smithi099.stdout: "id": "2e6123d2-d695-4350-9a01-61a455bc782b", 2023-05-31T20:05:11.616 INFO:teuthology.orchestra.run.smithi099.stdout: "zonegroups": [ 2023-05-31T20:05:11.616 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T20:05:11.616 INFO:teuthology.orchestra.run.smithi099.stdout: "id": "a", 2023-05-31T20:05:11.616 INFO:teuthology.orchestra.run.smithi099.stdout: "name": "a", 2023-05-31T20:05:11.616 INFO:teuthology.orchestra.run.smithi099.stdout: "api_name": "", 2023-05-31T20:05:11.616 INFO:teuthology.orchestra.run.smithi099.stdout: "is_master": true, 2023-05-31T20:05:11.616 INFO:teuthology.orchestra.run.smithi099.stdout: "endpoints": [ 2023-05-31T20:05:11.617 INFO:teuthology.orchestra.run.smithi099.stdout: "http://smithi099.front.sepia.ceph.com:8000" 2023-05-31T20:05:11.617 INFO:teuthology.orchestra.run.smithi099.stdout: ], 2023-05-31T20:05:11.617 INFO:teuthology.orchestra.run.smithi099.stdout: "hostnames": [], 2023-05-31T20:05:11.617 INFO:teuthology.orchestra.run.smithi099.stdout: "hostnames_s3website": [], 2023-05-31T20:05:11.617 INFO:teuthology.orchestra.run.smithi099.stdout: "master_zone": "3d163b1e-09c7-47e2-ae87-add4aace3f38", 2023-05-31T20:05:11.617 INFO:teuthology.orchestra.run.smithi099.stdout: "zones": [ 2023-05-31T20:05:11.617 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T20:05:11.618 INFO:teuthology.orchestra.run.smithi099.stdout: "id": "3d163b1e-09c7-47e2-ae87-add4aace3f38", 2023-05-31T20:05:11.618 INFO:teuthology.orchestra.run.smithi099.stdout: "name": "a1", 2023-05-31T20:05:11.618 INFO:teuthology.orchestra.run.smithi099.stdout: "endpoints": [ 2023-05-31T20:05:11.618 INFO:teuthology.orchestra.run.smithi099.stdout: "http://smithi099.front.sepia.ceph.com:8000" 2023-05-31T20:05:11.618 INFO:teuthology.orchestra.run.smithi099.stdout: ], 2023-05-31T20:05:11.618 INFO:teuthology.orchestra.run.smithi099.stdout: "log_meta": false, 2023-05-31T20:05:11.619 INFO:teuthology.orchestra.run.smithi099.stdout: "log_data": true, 2023-05-31T20:05:11.619 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket_index_max_shards": 11, 2023-05-31T20:05:11.619 INFO:teuthology.orchestra.run.smithi099.stdout: "read_only": false, 2023-05-31T20:05:11.619 INFO:teuthology.orchestra.run.smithi099.stdout: "tier_type": "", 2023-05-31T20:05:11.619 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_from_all": true, 2023-05-31T20:05:11.619 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_from": [], 2023-05-31T20:05:11.619 INFO:teuthology.orchestra.run.smithi099.stdout: "redirect_zone": "", 2023-05-31T20:05:11.620 INFO:teuthology.orchestra.run.smithi099.stdout: "supported_features": [ 2023-05-31T20:05:11.620 INFO:teuthology.orchestra.run.smithi099.stdout: "resharding" 2023-05-31T20:05:11.620 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2023-05-31T20:05:11.620 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:05:11.620 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T20:05:11.620 INFO:teuthology.orchestra.run.smithi099.stdout: "id": "5d8acec7-7e57-4e56-8b32-e3d6b5036001", 2023-05-31T20:05:11.620 INFO:teuthology.orchestra.run.smithi099.stdout: "name": "a2", 2023-05-31T20:05:11.621 INFO:teuthology.orchestra.run.smithi099.stdout: "endpoints": [ 2023-05-31T20:05:11.621 INFO:teuthology.orchestra.run.smithi099.stdout: "http://smithi099.front.sepia.ceph.com:8001" 2023-05-31T20:05:11.621 INFO:teuthology.orchestra.run.smithi099.stdout: ], 2023-05-31T20:05:11.621 INFO:teuthology.orchestra.run.smithi099.stdout: "log_meta": false, 2023-05-31T20:05:11.621 INFO:teuthology.orchestra.run.smithi099.stdout: "log_data": true, 2023-05-31T20:05:11.621 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket_index_max_shards": 11, 2023-05-31T20:05:11.621 INFO:teuthology.orchestra.run.smithi099.stdout: "read_only": false, 2023-05-31T20:05:11.622 INFO:teuthology.orchestra.run.smithi099.stdout: "tier_type": "", 2023-05-31T20:05:11.622 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_from_all": false, 2023-05-31T20:05:11.622 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_from": [], 2023-05-31T20:05:11.622 INFO:teuthology.orchestra.run.smithi099.stdout: "redirect_zone": "", 2023-05-31T20:05:11.622 INFO:teuthology.orchestra.run.smithi099.stdout: "supported_features": [ 2023-05-31T20:05:11.622 INFO:teuthology.orchestra.run.smithi099.stdout: "resharding" 2023-05-31T20:05:11.622 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2023-05-31T20:05:11.623 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T20:05:11.623 INFO:teuthology.orchestra.run.smithi099.stdout: ], 2023-05-31T20:05:11.623 INFO:teuthology.orchestra.run.smithi099.stdout: "placement_targets": [ 2023-05-31T20:05:11.623 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T20:05:11.623 INFO:teuthology.orchestra.run.smithi099.stdout: "name": "default-placement", 2023-05-31T20:05:11.623 INFO:teuthology.orchestra.run.smithi099.stdout: "tags": [], 2023-05-31T20:05:11.623 INFO:teuthology.orchestra.run.smithi099.stdout: "storage_classes": [ 2023-05-31T20:05:11.624 INFO:teuthology.orchestra.run.smithi099.stdout: "STANDARD" 2023-05-31T20:05:11.624 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2023-05-31T20:05:11.625 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T20:05:11.625 INFO:teuthology.orchestra.run.smithi099.stdout: ], 2023-05-31T20:05:11.625 INFO:teuthology.orchestra.run.smithi099.stdout: "default_placement": "default-placement", 2023-05-31T20:05:11.625 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_id": "aaa33de6-515b-4cac-8f1b-3a1a30452117", 2023-05-31T20:05:11.626 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_policy": { 2023-05-31T20:05:11.626 INFO:teuthology.orchestra.run.smithi099.stdout: "groups": [] 2023-05-31T20:05:11.626 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:05:11.626 INFO:teuthology.orchestra.run.smithi099.stdout: "enabled_features": [ 2023-05-31T20:05:11.626 INFO:teuthology.orchestra.run.smithi099.stdout: "resharding" 2023-05-31T20:05:11.627 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2023-05-31T20:05:11.627 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:05:11.627 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T20:05:11.627 INFO:teuthology.orchestra.run.smithi099.stdout: "id": "b", 2023-05-31T20:05:11.627 INFO:teuthology.orchestra.run.smithi099.stdout: "name": "b", 2023-05-31T20:05:11.627 INFO:teuthology.orchestra.run.smithi099.stdout: "api_name": "", 2023-05-31T20:05:11.628 INFO:teuthology.orchestra.run.smithi099.stdout: "is_master": false, 2023-05-31T20:05:11.628 INFO:teuthology.orchestra.run.smithi099.stdout: "endpoints": [ 2023-05-31T20:05:11.628 INFO:teuthology.orchestra.run.smithi099.stdout: "http://smithi112.front.sepia.ceph.com:8000" 2023-05-31T20:05:11.628 INFO:teuthology.orchestra.run.smithi099.stdout: ], 2023-05-31T20:05:11.628 INFO:teuthology.orchestra.run.smithi099.stdout: "hostnames": [], 2023-05-31T20:05:11.628 INFO:teuthology.orchestra.run.smithi099.stdout: "hostnames_s3website": [], 2023-05-31T20:05:11.629 INFO:teuthology.orchestra.run.smithi099.stdout: "master_zone": "f85d7236-2f27-4988-af26-9c66ab280b1b", 2023-05-31T20:05:11.629 INFO:teuthology.orchestra.run.smithi099.stdout: "zones": [ 2023-05-31T20:05:11.629 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T20:05:11.629 INFO:teuthology.orchestra.run.smithi099.stdout: "id": "a6a3aab5-bc52-44a4-8543-5c5ea6056acd", 2023-05-31T20:05:11.629 INFO:teuthology.orchestra.run.smithi099.stdout: "name": "b2", 2023-05-31T20:05:11.629 INFO:teuthology.orchestra.run.smithi099.stdout: "endpoints": [ 2023-05-31T20:05:11.630 INFO:teuthology.orchestra.run.smithi099.stdout: "http://smithi112.front.sepia.ceph.com:8001" 2023-05-31T20:05:11.630 INFO:teuthology.orchestra.run.smithi099.stdout: ], 2023-05-31T20:05:11.630 INFO:teuthology.orchestra.run.smithi099.stdout: "log_meta": false, 2023-05-31T20:05:11.630 INFO:teuthology.orchestra.run.smithi099.stdout: "log_data": true, 2023-05-31T20:05:11.630 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket_index_max_shards": 11, 2023-05-31T20:05:11.630 INFO:teuthology.orchestra.run.smithi099.stdout: "read_only": false, 2023-05-31T20:05:11.631 INFO:teuthology.orchestra.run.smithi099.stdout: "tier_type": "", 2023-05-31T20:05:11.631 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_from_all": true, 2023-05-31T20:05:11.631 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_from": [], 2023-05-31T20:05:11.631 INFO:teuthology.orchestra.run.smithi099.stdout: "redirect_zone": "", 2023-05-31T20:05:11.632 INFO:teuthology.orchestra.run.smithi099.stdout: "supported_features": [ 2023-05-31T20:05:11.632 INFO:teuthology.orchestra.run.smithi099.stdout: "resharding" 2023-05-31T20:05:11.632 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2023-05-31T20:05:11.632 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:05:11.632 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T20:05:11.649 INFO:teuthology.orchestra.run.smithi099.stdout: "id": "f85d7236-2f27-4988-af26-9c66ab280b1b", 2023-05-31T20:05:11.650 INFO:teuthology.orchestra.run.smithi099.stdout: "name": "b1", 2023-05-31T20:05:11.650 INFO:teuthology.orchestra.run.smithi099.stdout: "endpoints": [ 2023-05-31T20:05:11.650 INFO:teuthology.orchestra.run.smithi099.stdout: "http://smithi112.front.sepia.ceph.com:8000" 2023-05-31T20:05:11.650 INFO:teuthology.orchestra.run.smithi099.stdout: ], 2023-05-31T20:05:11.650 INFO:teuthology.orchestra.run.smithi099.stdout: "log_meta": false, 2023-05-31T20:05:11.650 INFO:teuthology.orchestra.run.smithi099.stdout: "log_data": true, 2023-05-31T20:05:11.651 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket_index_max_shards": 11, 2023-05-31T20:05:11.651 INFO:teuthology.orchestra.run.smithi099.stdout: "read_only": false, 2023-05-31T20:05:11.651 INFO:teuthology.orchestra.run.smithi099.stdout: "tier_type": "", 2023-05-31T20:05:11.651 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_from_all": true, 2023-05-31T20:05:11.651 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_from": [], 2023-05-31T20:05:11.651 INFO:teuthology.orchestra.run.smithi099.stdout: "redirect_zone": "", 2023-05-31T20:05:11.651 INFO:teuthology.orchestra.run.smithi099.stdout: "supported_features": [ 2023-05-31T20:05:11.652 INFO:teuthology.orchestra.run.smithi099.stdout: "resharding" 2023-05-31T20:05:11.652 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2023-05-31T20:05:11.652 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T20:05:11.652 INFO:teuthology.orchestra.run.smithi099.stdout: ], 2023-05-31T20:05:11.652 INFO:teuthology.orchestra.run.smithi099.stdout: "placement_targets": [ 2023-05-31T20:05:11.652 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T20:05:11.653 INFO:teuthology.orchestra.run.smithi099.stdout: "name": "default-placement", 2023-05-31T20:05:11.653 INFO:teuthology.orchestra.run.smithi099.stdout: "tags": [], 2023-05-31T20:05:11.653 INFO:teuthology.orchestra.run.smithi099.stdout: "storage_classes": [ 2023-05-31T20:05:11.653 INFO:teuthology.orchestra.run.smithi099.stdout: "STANDARD" 2023-05-31T20:05:11.653 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2023-05-31T20:05:11.653 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T20:05:11.653 INFO:teuthology.orchestra.run.smithi099.stdout: ], 2023-05-31T20:05:11.654 INFO:teuthology.orchestra.run.smithi099.stdout: "default_placement": "default-placement", 2023-05-31T20:05:11.654 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_id": "aaa33de6-515b-4cac-8f1b-3a1a30452117", 2023-05-31T20:05:11.654 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_policy": { 2023-05-31T20:05:11.654 INFO:teuthology.orchestra.run.smithi099.stdout: "groups": [] 2023-05-31T20:05:11.654 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:05:11.654 INFO:teuthology.orchestra.run.smithi099.stdout: "enabled_features": [ 2023-05-31T20:05:11.654 INFO:teuthology.orchestra.run.smithi099.stdout: "resharding" 2023-05-31T20:05:11.655 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2023-05-31T20:05:11.655 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T20:05:11.655 INFO:teuthology.orchestra.run.smithi099.stdout: ], 2023-05-31T20:05:11.655 INFO:teuthology.orchestra.run.smithi099.stdout: "short_zone_ids": [ 2023-05-31T20:05:11.655 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T20:05:11.655 INFO:teuthology.orchestra.run.smithi099.stdout: "key": "3d163b1e-09c7-47e2-ae87-add4aace3f38", 2023-05-31T20:05:11.655 INFO:teuthology.orchestra.run.smithi099.stdout: "val": 3101833771 2023-05-31T20:05:11.656 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:05:11.656 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T20:05:11.656 INFO:teuthology.orchestra.run.smithi099.stdout: "key": "5d8acec7-7e57-4e56-8b32-e3d6b5036001", 2023-05-31T20:05:11.656 INFO:teuthology.orchestra.run.smithi099.stdout: "val": 4255530623 2023-05-31T20:05:11.656 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:05:11.656 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T20:05:11.657 INFO:teuthology.orchestra.run.smithi099.stdout: "key": "a6a3aab5-bc52-44a4-8543-5c5ea6056acd", 2023-05-31T20:05:11.657 INFO:teuthology.orchestra.run.smithi099.stdout: "val": 3708073710 2023-05-31T20:05:11.657 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:05:11.657 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T20:05:11.657 INFO:teuthology.orchestra.run.smithi099.stdout: "key": "f85d7236-2f27-4988-af26-9c66ab280b1b", 2023-05-31T20:05:11.657 INFO:teuthology.orchestra.run.smithi099.stdout: "val": 233395200 2023-05-31T20:05:11.657 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T20:05:11.658 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2023-05-31T20:05:11.658 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:05:11.658 INFO:teuthology.orchestra.run.smithi099.stdout: "master_zonegroup": "a", 2023-05-31T20:05:11.658 INFO:teuthology.orchestra.run.smithi099.stdout: "master_zone": "3d163b1e-09c7-47e2-ae87-add4aace3f38", 2023-05-31T20:05:11.658 INFO:teuthology.orchestra.run.smithi099.stdout: "period_config": { 2023-05-31T20:05:11.658 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket_quota": { 2023-05-31T20:05:11.658 INFO:teuthology.orchestra.run.smithi099.stdout: "enabled": false, 2023-05-31T20:05:11.659 INFO:teuthology.orchestra.run.smithi099.stdout: "check_on_raw": false, 2023-05-31T20:05:11.659 INFO:teuthology.orchestra.run.smithi099.stdout: "max_size": -1, 2023-05-31T20:05:11.659 INFO:teuthology.orchestra.run.smithi099.stdout: "max_size_kb": 0, 2023-05-31T20:05:11.659 INFO:teuthology.orchestra.run.smithi099.stdout: "max_objects": -1 2023-05-31T20:05:11.659 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:05:11.659 INFO:teuthology.orchestra.run.smithi099.stdout: "user_quota": { 2023-05-31T20:05:11.660 INFO:teuthology.orchestra.run.smithi099.stdout: "enabled": false, 2023-05-31T20:05:11.660 INFO:teuthology.orchestra.run.smithi099.stdout: "check_on_raw": false, 2023-05-31T20:05:11.660 INFO:teuthology.orchestra.run.smithi099.stdout: "max_size": -1, 2023-05-31T20:05:11.660 INFO:teuthology.orchestra.run.smithi099.stdout: "max_size_kb": 0, 2023-05-31T20:05:11.660 INFO:teuthology.orchestra.run.smithi099.stdout: "max_objects": -1 2023-05-31T20:05:11.660 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:05:11.660 INFO:teuthology.orchestra.run.smithi099.stdout: "user_ratelimit": { 2023-05-31T20:05:11.661 INFO:teuthology.orchestra.run.smithi099.stdout: "max_read_ops": 0, 2023-05-31T20:05:11.661 INFO:teuthology.orchestra.run.smithi099.stdout: "max_write_ops": 0, 2023-05-31T20:05:11.661 INFO:teuthology.orchestra.run.smithi099.stdout: "max_read_bytes": 0, 2023-05-31T20:05:11.661 INFO:teuthology.orchestra.run.smithi099.stdout: "max_write_bytes": 0, 2023-05-31T20:05:11.661 INFO:teuthology.orchestra.run.smithi099.stdout: "enabled": false 2023-05-31T20:05:11.661 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:05:11.661 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket_ratelimit": { 2023-05-31T20:05:11.662 INFO:teuthology.orchestra.run.smithi099.stdout: "max_read_ops": 0, 2023-05-31T20:05:11.662 INFO:teuthology.orchestra.run.smithi099.stdout: "max_write_ops": 0, 2023-05-31T20:05:11.662 INFO:teuthology.orchestra.run.smithi099.stdout: "max_read_bytes": 0, 2023-05-31T20:05:11.662 INFO:teuthology.orchestra.run.smithi099.stdout: "max_write_bytes": 0, 2023-05-31T20:05:11.662 INFO:teuthology.orchestra.run.smithi099.stdout: "enabled": false 2023-05-31T20:05:11.662 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:05:11.662 INFO:teuthology.orchestra.run.smithi099.stdout: "anonymous_ratelimit": { 2023-05-31T20:05:11.663 INFO:teuthology.orchestra.run.smithi099.stdout: "max_read_ops": 0, 2023-05-31T20:05:11.663 INFO:teuthology.orchestra.run.smithi099.stdout: "max_write_ops": 0, 2023-05-31T20:05:11.663 INFO:teuthology.orchestra.run.smithi099.stdout: "max_read_bytes": 0, 2023-05-31T20:05:11.663 INFO:teuthology.orchestra.run.smithi099.stdout: "max_write_bytes": 0, 2023-05-31T20:05:11.663 INFO:teuthology.orchestra.run.smithi099.stdout: "enabled": false 2023-05-31T20:05:11.663 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T20:05:11.663 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:05:11.664 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_id": "aaa33de6-515b-4cac-8f1b-3a1a30452117", 2023-05-31T20:05:11.664 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_name": "test-realm", 2023-05-31T20:05:11.664 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2023-05-31T20:05:11.664 INFO:teuthology.orchestra.run.smithi099.stdout:} 2023-05-31T20:05:11.665 INFO:rgw_multi.tests:Set sync_from_all flag on zone a2 to false 2023-05-31T20:06:11.666 INFO:rgw_multi.tests:create bucket zone=a1 name=pklxkd-35 2023-05-31T20:06:11.667 DEBUG:boto:path=/pklxkd-35/ 2023-05-31T20:06:11.668 DEBUG:boto:auth_path=/pklxkd-35/ 2023-05-31T20:06:11.669 DEBUG:boto:Method: PUT 2023-05-31T20:06:11.669 DEBUG:boto:Path: /pklxkd-35/ 2023-05-31T20:06:11.670 DEBUG:boto:Data: 2023-05-31T20:06:11.670 DEBUG:boto:Headers: {} 2023-05-31T20:06:11.671 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:06:11.672 DEBUG:boto:Port: 8000 2023-05-31T20:06:11.673 DEBUG:boto:Params: {} 2023-05-31T20:06:11.674 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2023-05-31T20:06:11.675 DEBUG:boto:Token: None 2023-05-31T20:06:11.675 DEBUG:boto:StringToSign: PUT Wed, 31 May 2023 20:06:11 GMT /pklxkd-35/ 2023-05-31T20:06:11.676 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:tu9/OxURiZBMoHyRP8L0lpYAoUo= 2023-05-31T20:06:11.677 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:06:11 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:tu9/OxURiZBMoHyRP8L0lpYAoUo=', 'Content-Length': '0'} 2023-05-31T20:06:11.708 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000603e961958e7d930-006477a8b3-1512-a1'), ('Content-Length', '0'), ('Date', 'Wed, 31 May 2023 20:06:11 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:06:11.709 DEBUG:boto:path=/pklxkd-35/testredirect 2023-05-31T20:06:11.709 DEBUG:boto:auth_path=/pklxkd-35/testredirect 2023-05-31T20:06:11.710 DEBUG:boto:Method: PUT 2023-05-31T20:06:11.710 DEBUG:boto:Path: /pklxkd-35/testredirect 2023-05-31T20:06:11.711 DEBUG:boto:Data: 2023-05-31T20:06:11.711 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': '3FCGuEcom6i4veFJuDiBdQ==', 'Content-Length': '512', 'Expect': '100-Continue'} 2023-05-31T20:06:11.711 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:06:11.712 DEBUG:boto:Port: 8000 2023-05-31T20:06:11.712 DEBUG:boto:Params: {} 2023-05-31T20:06:11.713 DEBUG:boto:Token: None 2023-05-31T20:06:11.713 DEBUG:boto:StringToSign: PUT 3FCGuEcom6i4veFJuDiBdQ== application/octet-stream Wed, 31 May 2023 20:06:11 GMT /pklxkd-35/testredirect 2023-05-31T20:06:11.713 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:kGEbpd1N/A33qhlYPdYaHqo3zN4= 2023-05-31T20:06:11.714 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': '3FCGuEcom6i4veFJuDiBdQ==', 'Content-Length': '512', 'Expect': '100-Continue', 'Date': 'Wed, 31 May 2023 20:06:11 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:kGEbpd1N/A33qhlYPdYaHqo3zN4='} 2023-05-31T20:06:11.726 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"dc5086b847289ba8b8bde149b8388175"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000004c2e8594d3cf30cf-006477a8b3-1512-a1'), ('Date', 'Wed, 31 May 2023 20:06:11 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:06:11.726 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-31T20:06:11.727 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-31T20:06:11.727 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-05-31T20:06:11.783 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2023-05-31T20:06:11.784 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2023-05-31T20:06:11.823 INFO:teuthology.orchestra.run.smithi099.stderr:No --period given, using current period=2e6123d2-d695-4350-9a01-61a455bc782b 2023-05-31T20:06:11.835 INFO:teuthology.orchestra.run.smithi099.stdout:[ 2023-05-31T20:06:11.836 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T20:06:11.836 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1685562631.516089_59.1", 2023-05-31T20:06:11.836 INFO:teuthology.orchestra.run.smithi099.stdout: "last_update": "2023-05-31T19:50:31.516089Z" 2023-05-31T20:06:11.836 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:06:11.836 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T20:06:11.836 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "", 2023-05-31T20:06:11.837 INFO:teuthology.orchestra.run.smithi099.stdout: "last_update": "0.000000" 2023-05-31T20:06:11.837 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:06:11.837 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T20:06:11.837 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1685562970.274572_93.1", 2023-05-31T20:06:11.837 INFO:teuthology.orchestra.run.smithi099.stdout: "last_update": "2023-05-31T19:56:10.274572Z" 2023-05-31T20:06:11.837 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:06:11.837 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T20:06:11.838 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1685563571.700188_212.1", 2023-05-31T20:06:11.838 INFO:teuthology.orchestra.run.smithi099.stdout: "last_update": "2023-05-31T20:06:11.700188Z" 2023-05-31T20:06:11.838 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T20:06:11.838 INFO:teuthology.orchestra.run.smithi099.stdout:] 2023-05-31T20:06:11.839 DEBUG:rgw_multi.tests:master meta markers={0: '1_1685562631.516089_59.1', 1: '', 2: '1_1685562970.274572_93.1', 3: '1_1685563571.700188_212.1'} 2023-05-31T20:06:11.839 INFO:rgw_multi.tests:starting meta checkpoint for zone=a2 2023-05-31T20:06:11.840 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-31T20:06:11.840 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-31T20:06:11.841 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-05-31T20:06:11.892 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2023-05-31T20:06:11.892 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2023-05-31T20:06:11.923 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:06:11.918+0000 7f5c251d6780 20 RGW-SYNC:meta: read sync status 2023-05-31T20:06:11.924 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:06:11.919+0000 7f5c251d6780 20 RGW-SYNC:meta: read sync status 2023-05-31T20:06:11.936 INFO:teuthology.orchestra.run.smithi099.stdout:{ 2023-05-31T20:06:11.937 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_status": { 2023-05-31T20:06:11.937 INFO:teuthology.orchestra.run.smithi099.stdout: "info": { 2023-05-31T20:06:11.937 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "sync", 2023-05-31T20:06:11.938 INFO:teuthology.orchestra.run.smithi099.stdout: "num_shards": 4, 2023-05-31T20:06:11.938 INFO:teuthology.orchestra.run.smithi099.stdout: "period": "2e6123d2-d695-4350-9a01-61a455bc782b", 2023-05-31T20:06:11.938 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2023-05-31T20:06:11.939 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:06:11.939 INFO:teuthology.orchestra.run.smithi099.stdout: "markers": [ 2023-05-31T20:06:11.939 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T20:06:11.940 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 0, 2023-05-31T20:06:11.940 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T20:06:11.941 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2023-05-31T20:06:11.941 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1685562631.516089_59.1", 2023-05-31T20:06:11.941 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T20:06:11.942 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 1, 2023-05-31T20:06:11.942 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T20:06:11.942 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2023-05-31T19:50:31.516089Z", 2023-05-31T20:06:11.943 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2023-05-31T20:06:11.943 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T20:06:11.943 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:06:11.943 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T20:06:11.944 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 1, 2023-05-31T20:06:11.944 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T20:06:11.944 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2023-05-31T20:06:11.944 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "", 2023-05-31T20:06:11.945 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T20:06:11.945 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2023-05-31T20:06:11.945 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T20:06:11.946 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "0.000000", 2023-05-31T20:06:11.946 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2023-05-31T20:06:11.946 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T20:06:11.947 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:06:11.947 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T20:06:11.947 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 2, 2023-05-31T20:06:11.947 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T20:06:11.947 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2023-05-31T20:06:11.947 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1685562970.274572_93.1", 2023-05-31T20:06:11.948 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T20:06:11.948 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2023-05-31T20:06:11.948 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T20:06:11.948 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2023-05-31T19:56:10.274572Z", 2023-05-31T20:06:11.948 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2023-05-31T20:06:11.948 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T20:06:11.949 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:06:11.949 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T20:06:11.949 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 3, 2023-05-31T20:06:11.949 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T20:06:11.949 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2023-05-31T20:06:11.949 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1685563478.630332_193.1", 2023-05-31T20:06:11.950 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T20:06:11.950 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2023-05-31T20:06:11.950 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T20:06:11.950 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2023-05-31T20:04:38.630332Z", 2023-05-31T20:06:11.950 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2023-05-31T20:06:11.950 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T20:06:11.950 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T20:06:11.951 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2023-05-31T20:06:11.951 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:06:11.951 INFO:teuthology.orchestra.run.smithi099.stdout: "full_sync": { 2023-05-31T20:06:11.951 INFO:teuthology.orchestra.run.smithi099.stdout: "total": 1, 2023-05-31T20:06:11.952 INFO:teuthology.orchestra.run.smithi099.stdout: "complete": 1 2023-05-31T20:06:11.952 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:06:11.952 INFO:teuthology.orchestra.run.smithi099.stdout: "current_time": "2023-05-31T20:06:11Z" 2023-05-31T20:06:11.952 INFO:teuthology.orchestra.run.smithi099.stdout:} 2023-05-31T20:06:11.953 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "2e6123d2-d695-4350-9a01-61a455bc782b", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1685562631.516089_59.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-05-31T19:50:31.516089Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1685562970.274572_93.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-05-31T19:56:10.274572Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1685563478.630332_193.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-05-31T20:04:38.630332Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-05-31T20:06:11Z" } 2023-05-31T20:06:11.954 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1685562631.516089_59.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-05-31T19:50:31.516089Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1685562970.274572_93.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-05-31T19:56:10.274572Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1685563478.630332_193.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-05-31T20:04:38.630332Z', 'realm_epoch': 2}}] 2023-05-31T20:06:11.954 DEBUG:rgw_multi.tests:log_status={0: '1_1685562631.516089_59.1', 1: '', 2: '1_1685562970.274572_93.1', 3: '1_1685563571.700188_212.1'} 2023-05-31T20:06:11.955 DEBUG:rgw_multi.tests:sync_status={0: '1_1685562631.516089_59.1', 1: '', 2: '1_1685562970.274572_93.1', 3: '1_1685563478.630332_193.1'} 2023-05-31T20:06:11.955 WARNING:rgw_multi.tests:zone a2 behind master: shard=3 master=1_1685563571.700188_212.1 target=1_1685563478.630332_193.1 2023-05-31T20:06:16.956 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-31T20:06:16.957 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-31T20:06:16.957 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-05-31T20:06:17.013 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2023-05-31T20:06:17.013 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2023-05-31T20:06:17.054 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:06:17.049+0000 7f5622bb9780 20 RGW-SYNC:meta: read sync status 2023-05-31T20:06:17.055 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:06:17.050+0000 7f5622bb9780 20 RGW-SYNC:meta: read sync status 2023-05-31T20:06:17.072 INFO:teuthology.orchestra.run.smithi099.stdout:{ 2023-05-31T20:06:17.072 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_status": { 2023-05-31T20:06:17.073 INFO:teuthology.orchestra.run.smithi099.stdout: "info": { 2023-05-31T20:06:17.073 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "sync", 2023-05-31T20:06:17.073 INFO:teuthology.orchestra.run.smithi099.stdout: "num_shards": 4, 2023-05-31T20:06:17.074 INFO:teuthology.orchestra.run.smithi099.stdout: "period": "2e6123d2-d695-4350-9a01-61a455bc782b", 2023-05-31T20:06:17.074 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2023-05-31T20:06:17.074 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:06:17.074 INFO:teuthology.orchestra.run.smithi099.stdout: "markers": [ 2023-05-31T20:06:17.074 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T20:06:17.074 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 0, 2023-05-31T20:06:17.075 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T20:06:17.075 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2023-05-31T20:06:17.075 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1685562631.516089_59.1", 2023-05-31T20:06:17.075 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T20:06:17.078 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 1, 2023-05-31T20:06:17.078 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T20:06:17.078 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2023-05-31T19:50:31.516089Z", 2023-05-31T20:06:17.079 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2023-05-31T20:06:17.079 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T20:06:17.079 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:06:17.080 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T20:06:17.080 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 1, 2023-05-31T20:06:17.080 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T20:06:17.081 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2023-05-31T20:06:17.081 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "", 2023-05-31T20:06:17.081 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T20:06:17.082 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2023-05-31T20:06:17.082 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T20:06:17.082 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "0.000000", 2023-05-31T20:06:17.083 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2023-05-31T20:06:17.083 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T20:06:17.083 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:06:17.084 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T20:06:17.084 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 2, 2023-05-31T20:06:17.084 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T20:06:17.085 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2023-05-31T20:06:17.085 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1685562970.274572_93.1", 2023-05-31T20:06:17.085 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T20:06:17.086 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2023-05-31T20:06:17.086 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T20:06:17.086 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2023-05-31T19:56:10.274572Z", 2023-05-31T20:06:17.087 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2023-05-31T20:06:17.087 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T20:06:17.087 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:06:17.088 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T20:06:17.088 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 3, 2023-05-31T20:06:17.088 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T20:06:17.089 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2023-05-31T20:06:17.089 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1685563478.630332_193.1", 2023-05-31T20:06:17.089 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T20:06:17.089 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2023-05-31T20:06:17.089 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T20:06:17.090 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2023-05-31T20:04:38.630332Z", 2023-05-31T20:06:17.090 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2023-05-31T20:06:17.090 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T20:06:17.090 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T20:06:17.090 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2023-05-31T20:06:17.091 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:06:17.091 INFO:teuthology.orchestra.run.smithi099.stdout: "full_sync": { 2023-05-31T20:06:17.091 INFO:teuthology.orchestra.run.smithi099.stdout: "total": 1, 2023-05-31T20:06:17.091 INFO:teuthology.orchestra.run.smithi099.stdout: "complete": 1 2023-05-31T20:06:17.091 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:06:17.092 INFO:teuthology.orchestra.run.smithi099.stdout: "current_time": "2023-05-31T20:06:17Z" 2023-05-31T20:06:17.092 INFO:teuthology.orchestra.run.smithi099.stdout:} 2023-05-31T20:06:17.094 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "2e6123d2-d695-4350-9a01-61a455bc782b", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1685562631.516089_59.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-05-31T19:50:31.516089Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1685562970.274572_93.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-05-31T19:56:10.274572Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1685563478.630332_193.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-05-31T20:04:38.630332Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-05-31T20:06:17Z" } 2023-05-31T20:06:17.094 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1685562631.516089_59.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-05-31T19:50:31.516089Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1685562970.274572_93.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-05-31T19:56:10.274572Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1685563478.630332_193.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-05-31T20:04:38.630332Z', 'realm_epoch': 2}}] 2023-05-31T20:06:17.095 DEBUG:rgw_multi.tests:log_status={0: '1_1685562631.516089_59.1', 1: '', 2: '1_1685562970.274572_93.1', 3: '1_1685563571.700188_212.1'} 2023-05-31T20:06:17.095 DEBUG:rgw_multi.tests:sync_status={0: '1_1685562631.516089_59.1', 1: '', 2: '1_1685562970.274572_93.1', 3: '1_1685563478.630332_193.1'} 2023-05-31T20:06:17.096 WARNING:rgw_multi.tests:zone a2 behind master: shard=3 master=1_1685563571.700188_212.1 target=1_1685563478.630332_193.1 2023-05-31T20:06:22.097 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-31T20:06:22.097 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-31T20:06:22.098 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-05-31T20:06:22.156 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2023-05-31T20:06:22.156 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2023-05-31T20:06:22.208 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:06:22.200+0000 7f6fe133e780 20 RGW-SYNC:meta: read sync status 2023-05-31T20:06:22.208 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:06:22.201+0000 7f6fe133e780 20 RGW-SYNC:meta: read sync status 2023-05-31T20:06:22.227 INFO:teuthology.orchestra.run.smithi099.stdout:{ 2023-05-31T20:06:22.227 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_status": { 2023-05-31T20:06:22.228 INFO:teuthology.orchestra.run.smithi099.stdout: "info": { 2023-05-31T20:06:22.228 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "sync", 2023-05-31T20:06:22.228 INFO:teuthology.orchestra.run.smithi099.stdout: "num_shards": 4, 2023-05-31T20:06:22.228 INFO:teuthology.orchestra.run.smithi099.stdout: "period": "2e6123d2-d695-4350-9a01-61a455bc782b", 2023-05-31T20:06:22.228 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2023-05-31T20:06:22.228 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:06:22.229 INFO:teuthology.orchestra.run.smithi099.stdout: "markers": [ 2023-05-31T20:06:22.229 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T20:06:22.229 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 0, 2023-05-31T20:06:22.229 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T20:06:22.229 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2023-05-31T20:06:22.229 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1685562631.516089_59.1", 2023-05-31T20:06:22.230 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T20:06:22.230 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 1, 2023-05-31T20:06:22.230 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T20:06:22.230 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2023-05-31T19:50:31.516089Z", 2023-05-31T20:06:22.230 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2023-05-31T20:06:22.230 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T20:06:22.231 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:06:22.231 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T20:06:22.231 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 1, 2023-05-31T20:06:22.231 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T20:06:22.231 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2023-05-31T20:06:22.231 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "", 2023-05-31T20:06:22.232 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T20:06:22.232 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2023-05-31T20:06:22.232 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T20:06:22.232 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "0.000000", 2023-05-31T20:06:22.232 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2023-05-31T20:06:22.233 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T20:06:22.233 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:06:22.233 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T20:06:22.233 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 2, 2023-05-31T20:06:22.233 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T20:06:22.233 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2023-05-31T20:06:22.233 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1685562970.274572_93.1", 2023-05-31T20:06:22.234 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T20:06:22.234 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2023-05-31T20:06:22.234 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T20:06:22.234 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2023-05-31T19:56:10.274572Z", 2023-05-31T20:06:22.234 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2023-05-31T20:06:22.234 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T20:06:22.235 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:06:22.235 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T20:06:22.235 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 3, 2023-05-31T20:06:22.235 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T20:06:22.235 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2023-05-31T20:06:22.235 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1685563478.630332_193.1", 2023-05-31T20:06:22.236 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T20:06:22.236 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2023-05-31T20:06:22.236 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T20:06:22.236 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2023-05-31T20:04:38.630332Z", 2023-05-31T20:06:22.236 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2023-05-31T20:06:22.237 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T20:06:22.237 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T20:06:22.237 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2023-05-31T20:06:22.237 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:06:22.237 INFO:teuthology.orchestra.run.smithi099.stdout: "full_sync": { 2023-05-31T20:06:22.237 INFO:teuthology.orchestra.run.smithi099.stdout: "total": 1, 2023-05-31T20:06:22.238 INFO:teuthology.orchestra.run.smithi099.stdout: "complete": 1 2023-05-31T20:06:22.238 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:06:22.238 INFO:teuthology.orchestra.run.smithi099.stdout: "current_time": "2023-05-31T20:06:22Z" 2023-05-31T20:06:22.238 INFO:teuthology.orchestra.run.smithi099.stdout:} 2023-05-31T20:06:22.239 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "2e6123d2-d695-4350-9a01-61a455bc782b", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1685562631.516089_59.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-05-31T19:50:31.516089Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1685562970.274572_93.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-05-31T19:56:10.274572Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1685563478.630332_193.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-05-31T20:04:38.630332Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-05-31T20:06:22Z" } 2023-05-31T20:06:22.240 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1685562631.516089_59.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-05-31T19:50:31.516089Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1685562970.274572_93.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-05-31T19:56:10.274572Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1685563478.630332_193.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-05-31T20:04:38.630332Z', 'realm_epoch': 2}}] 2023-05-31T20:06:22.241 DEBUG:rgw_multi.tests:log_status={0: '1_1685562631.516089_59.1', 1: '', 2: '1_1685562970.274572_93.1', 3: '1_1685563571.700188_212.1'} 2023-05-31T20:06:22.241 DEBUG:rgw_multi.tests:sync_status={0: '1_1685562631.516089_59.1', 1: '', 2: '1_1685562970.274572_93.1', 3: '1_1685563478.630332_193.1'} 2023-05-31T20:06:22.242 WARNING:rgw_multi.tests:zone a2 behind master: shard=3 master=1_1685563571.700188_212.1 target=1_1685563478.630332_193.1 2023-05-31T20:06:27.243 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-31T20:06:27.243 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-31T20:06:27.244 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-05-31T20:06:27.300 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2023-05-31T20:06:27.300 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2023-05-31T20:06:27.332 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:06:27.327+0000 7f72d703c780 20 RGW-SYNC:meta: read sync status 2023-05-31T20:06:27.333 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:06:27.328+0000 7f72d703c780 20 RGW-SYNC:meta: read sync status 2023-05-31T20:06:27.347 INFO:teuthology.orchestra.run.smithi099.stdout:{ 2023-05-31T20:06:27.348 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_status": { 2023-05-31T20:06:27.348 INFO:teuthology.orchestra.run.smithi099.stdout: "info": { 2023-05-31T20:06:27.348 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "sync", 2023-05-31T20:06:27.348 INFO:teuthology.orchestra.run.smithi099.stdout: "num_shards": 4, 2023-05-31T20:06:27.349 INFO:teuthology.orchestra.run.smithi099.stdout: "period": "2e6123d2-d695-4350-9a01-61a455bc782b", 2023-05-31T20:06:27.349 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2023-05-31T20:06:27.349 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:06:27.349 INFO:teuthology.orchestra.run.smithi099.stdout: "markers": [ 2023-05-31T20:06:27.349 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T20:06:27.349 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 0, 2023-05-31T20:06:27.349 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T20:06:27.350 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2023-05-31T20:06:27.350 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1685562631.516089_59.1", 2023-05-31T20:06:27.350 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T20:06:27.350 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 1, 2023-05-31T20:06:27.350 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T20:06:27.351 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2023-05-31T19:50:31.516089Z", 2023-05-31T20:06:27.351 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2023-05-31T20:06:27.351 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T20:06:27.351 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:06:27.351 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T20:06:27.351 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 1, 2023-05-31T20:06:27.351 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T20:06:27.352 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2023-05-31T20:06:27.353 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "", 2023-05-31T20:06:27.353 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T20:06:27.353 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2023-05-31T20:06:27.353 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T20:06:27.353 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "0.000000", 2023-05-31T20:06:27.353 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2023-05-31T20:06:27.354 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T20:06:27.354 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:06:27.354 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T20:06:27.354 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 2, 2023-05-31T20:06:27.354 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T20:06:27.354 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2023-05-31T20:06:27.355 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1685562970.274572_93.1", 2023-05-31T20:06:27.355 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T20:06:27.355 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2023-05-31T20:06:27.355 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T20:06:27.355 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2023-05-31T19:56:10.274572Z", 2023-05-31T20:06:27.355 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2023-05-31T20:06:27.356 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T20:06:27.356 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:06:27.356 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T20:06:27.356 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 3, 2023-05-31T20:06:27.356 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T20:06:27.357 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2023-05-31T20:06:27.357 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1685563478.630332_193.1", 2023-05-31T20:06:27.357 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T20:06:27.357 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2023-05-31T20:06:27.357 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T20:06:27.357 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2023-05-31T20:04:38.630332Z", 2023-05-31T20:06:27.357 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2023-05-31T20:06:27.358 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T20:06:27.358 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T20:06:27.358 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2023-05-31T20:06:27.358 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:06:27.358 INFO:teuthology.orchestra.run.smithi099.stdout: "full_sync": { 2023-05-31T20:06:27.358 INFO:teuthology.orchestra.run.smithi099.stdout: "total": 1, 2023-05-31T20:06:27.359 INFO:teuthology.orchestra.run.smithi099.stdout: "complete": 1 2023-05-31T20:06:27.359 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:06:27.359 INFO:teuthology.orchestra.run.smithi099.stdout: "current_time": "2023-05-31T20:06:27Z" 2023-05-31T20:06:27.359 INFO:teuthology.orchestra.run.smithi099.stdout:} 2023-05-31T20:06:27.359 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "2e6123d2-d695-4350-9a01-61a455bc782b", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1685562631.516089_59.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-05-31T19:50:31.516089Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1685562970.274572_93.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-05-31T19:56:10.274572Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1685563478.630332_193.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-05-31T20:04:38.630332Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-05-31T20:06:27Z" } 2023-05-31T20:06:27.360 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1685562631.516089_59.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-05-31T19:50:31.516089Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1685562970.274572_93.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-05-31T19:56:10.274572Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1685563478.630332_193.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-05-31T20:04:38.630332Z', 'realm_epoch': 2}}] 2023-05-31T20:06:27.360 DEBUG:rgw_multi.tests:log_status={0: '1_1685562631.516089_59.1', 1: '', 2: '1_1685562970.274572_93.1', 3: '1_1685563571.700188_212.1'} 2023-05-31T20:06:27.361 DEBUG:rgw_multi.tests:sync_status={0: '1_1685562631.516089_59.1', 1: '', 2: '1_1685562970.274572_93.1', 3: '1_1685563478.630332_193.1'} 2023-05-31T20:06:27.361 WARNING:rgw_multi.tests:zone a2 behind master: shard=3 master=1_1685563571.700188_212.1 target=1_1685563478.630332_193.1 2023-05-31T20:06:32.362 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-31T20:06:32.363 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-31T20:06:32.364 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-05-31T20:06:32.422 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2023-05-31T20:06:32.422 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2023-05-31T20:06:32.465 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:06:32.459+0000 7fc4a1567780 20 RGW-SYNC:meta: read sync status 2023-05-31T20:06:32.466 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:06:32.460+0000 7fc4a1567780 20 RGW-SYNC:meta: read sync status 2023-05-31T20:06:32.483 INFO:teuthology.orchestra.run.smithi099.stdout:{ 2023-05-31T20:06:32.483 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_status": { 2023-05-31T20:06:32.483 INFO:teuthology.orchestra.run.smithi099.stdout: "info": { 2023-05-31T20:06:32.484 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "sync", 2023-05-31T20:06:32.484 INFO:teuthology.orchestra.run.smithi099.stdout: "num_shards": 4, 2023-05-31T20:06:32.484 INFO:teuthology.orchestra.run.smithi099.stdout: "period": "2e6123d2-d695-4350-9a01-61a455bc782b", 2023-05-31T20:06:32.484 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2023-05-31T20:06:32.484 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:06:32.484 INFO:teuthology.orchestra.run.smithi099.stdout: "markers": [ 2023-05-31T20:06:32.485 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T20:06:32.485 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 0, 2023-05-31T20:06:32.485 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T20:06:32.485 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2023-05-31T20:06:32.485 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1685562631.516089_59.1", 2023-05-31T20:06:32.485 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T20:06:32.486 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 1, 2023-05-31T20:06:32.486 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T20:06:32.486 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2023-05-31T19:50:31.516089Z", 2023-05-31T20:06:32.486 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2023-05-31T20:06:32.486 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T20:06:32.486 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:06:32.486 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T20:06:32.487 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 1, 2023-05-31T20:06:32.487 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T20:06:32.487 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2023-05-31T20:06:32.487 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "", 2023-05-31T20:06:32.487 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T20:06:32.487 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2023-05-31T20:06:32.487 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T20:06:32.488 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "0.000000", 2023-05-31T20:06:32.488 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2023-05-31T20:06:32.488 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T20:06:32.488 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:06:32.488 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T20:06:32.488 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 2, 2023-05-31T20:06:32.489 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T20:06:32.489 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2023-05-31T20:06:32.489 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1685562970.274572_93.1", 2023-05-31T20:06:32.489 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T20:06:32.489 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2023-05-31T20:06:32.489 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T20:06:32.489 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2023-05-31T19:56:10.274572Z", 2023-05-31T20:06:32.490 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2023-05-31T20:06:32.490 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T20:06:32.490 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:06:32.490 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T20:06:32.490 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 3, 2023-05-31T20:06:32.490 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T20:06:32.490 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2023-05-31T20:06:32.491 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1685563478.630332_193.1", 2023-05-31T20:06:32.491 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T20:06:32.491 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2023-05-31T20:06:32.491 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T20:06:32.491 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2023-05-31T20:04:38.630332Z", 2023-05-31T20:06:32.491 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2023-05-31T20:06:32.491 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T20:06:32.492 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T20:06:32.492 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2023-05-31T20:06:32.492 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:06:32.492 INFO:teuthology.orchestra.run.smithi099.stdout: "full_sync": { 2023-05-31T20:06:32.492 INFO:teuthology.orchestra.run.smithi099.stdout: "total": 1, 2023-05-31T20:06:32.492 INFO:teuthology.orchestra.run.smithi099.stdout: "complete": 1 2023-05-31T20:06:32.492 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:06:32.493 INFO:teuthology.orchestra.run.smithi099.stdout: "current_time": "2023-05-31T20:06:32Z" 2023-05-31T20:06:32.493 INFO:teuthology.orchestra.run.smithi099.stdout:} 2023-05-31T20:06:32.494 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "2e6123d2-d695-4350-9a01-61a455bc782b", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1685562631.516089_59.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-05-31T19:50:31.516089Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1685562970.274572_93.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-05-31T19:56:10.274572Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1685563478.630332_193.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-05-31T20:04:38.630332Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-05-31T20:06:32Z" } 2023-05-31T20:06:32.494 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1685562631.516089_59.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-05-31T19:50:31.516089Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1685562970.274572_93.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-05-31T19:56:10.274572Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1685563478.630332_193.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-05-31T20:04:38.630332Z', 'realm_epoch': 2}}] 2023-05-31T20:06:32.495 DEBUG:rgw_multi.tests:log_status={0: '1_1685562631.516089_59.1', 1: '', 2: '1_1685562970.274572_93.1', 3: '1_1685563571.700188_212.1'} 2023-05-31T20:06:32.495 DEBUG:rgw_multi.tests:sync_status={0: '1_1685562631.516089_59.1', 1: '', 2: '1_1685562970.274572_93.1', 3: '1_1685563478.630332_193.1'} 2023-05-31T20:06:32.495 WARNING:rgw_multi.tests:zone a2 behind master: shard=3 master=1_1685563571.700188_212.1 target=1_1685563478.630332_193.1 2023-05-31T20:06:37.496 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-31T20:06:37.497 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-31T20:06:37.497 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-05-31T20:06:37.555 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2023-05-31T20:06:37.555 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2023-05-31T20:06:37.597 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:06:37.592+0000 7f08918de780 20 RGW-SYNC:meta: read sync status 2023-05-31T20:06:37.598 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:06:37.593+0000 7f08918de780 20 RGW-SYNC:meta: read sync status 2023-05-31T20:06:37.612 INFO:teuthology.orchestra.run.smithi099.stdout:{ 2023-05-31T20:06:37.612 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_status": { 2023-05-31T20:06:37.612 INFO:teuthology.orchestra.run.smithi099.stdout: "info": { 2023-05-31T20:06:37.612 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "sync", 2023-05-31T20:06:37.613 INFO:teuthology.orchestra.run.smithi099.stdout: "num_shards": 4, 2023-05-31T20:06:37.613 INFO:teuthology.orchestra.run.smithi099.stdout: "period": "2e6123d2-d695-4350-9a01-61a455bc782b", 2023-05-31T20:06:37.613 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2023-05-31T20:06:37.613 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:06:37.613 INFO:teuthology.orchestra.run.smithi099.stdout: "markers": [ 2023-05-31T20:06:37.613 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T20:06:37.614 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 0, 2023-05-31T20:06:37.614 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T20:06:37.614 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2023-05-31T20:06:37.614 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1685562631.516089_59.1", 2023-05-31T20:06:37.614 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T20:06:37.614 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 1, 2023-05-31T20:06:37.614 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T20:06:37.615 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2023-05-31T19:50:31.516089Z", 2023-05-31T20:06:37.615 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2023-05-31T20:06:37.615 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T20:06:37.615 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:06:37.615 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T20:06:37.615 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 1, 2023-05-31T20:06:37.615 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T20:06:37.616 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2023-05-31T20:06:37.616 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "", 2023-05-31T20:06:37.616 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T20:06:37.616 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2023-05-31T20:06:37.616 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T20:06:37.617 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "0.000000", 2023-05-31T20:06:37.617 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2023-05-31T20:06:37.617 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T20:06:37.617 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:06:37.617 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T20:06:37.617 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 2, 2023-05-31T20:06:37.617 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T20:06:37.618 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2023-05-31T20:06:37.618 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1685562970.274572_93.1", 2023-05-31T20:06:37.618 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T20:06:37.618 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2023-05-31T20:06:37.618 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T20:06:37.618 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2023-05-31T19:56:10.274572Z", 2023-05-31T20:06:37.618 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2023-05-31T20:06:37.619 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T20:06:37.619 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:06:37.619 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T20:06:37.619 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 3, 2023-05-31T20:06:37.619 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T20:06:37.619 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2023-05-31T20:06:37.619 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1685563478.630332_193.1", 2023-05-31T20:06:37.620 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T20:06:37.620 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2023-05-31T20:06:37.620 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T20:06:37.620 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2023-05-31T20:04:38.630332Z", 2023-05-31T20:06:37.620 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2023-05-31T20:06:37.621 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T20:06:37.621 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T20:06:37.621 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2023-05-31T20:06:37.621 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:06:37.621 INFO:teuthology.orchestra.run.smithi099.stdout: "full_sync": { 2023-05-31T20:06:37.621 INFO:teuthology.orchestra.run.smithi099.stdout: "total": 1, 2023-05-31T20:06:37.621 INFO:teuthology.orchestra.run.smithi099.stdout: "complete": 1 2023-05-31T20:06:37.622 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:06:37.622 INFO:teuthology.orchestra.run.smithi099.stdout: "current_time": "2023-05-31T20:06:37Z" 2023-05-31T20:06:37.622 INFO:teuthology.orchestra.run.smithi099.stdout:} 2023-05-31T20:06:37.623 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "2e6123d2-d695-4350-9a01-61a455bc782b", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1685562631.516089_59.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-05-31T19:50:31.516089Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1685562970.274572_93.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-05-31T19:56:10.274572Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1685563478.630332_193.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-05-31T20:04:38.630332Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-05-31T20:06:37Z" } 2023-05-31T20:06:37.623 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1685562631.516089_59.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-05-31T19:50:31.516089Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1685562970.274572_93.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-05-31T19:56:10.274572Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1685563478.630332_193.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-05-31T20:04:38.630332Z', 'realm_epoch': 2}}] 2023-05-31T20:06:37.624 DEBUG:rgw_multi.tests:log_status={0: '1_1685562631.516089_59.1', 1: '', 2: '1_1685562970.274572_93.1', 3: '1_1685563571.700188_212.1'} 2023-05-31T20:06:37.624 DEBUG:rgw_multi.tests:sync_status={0: '1_1685562631.516089_59.1', 1: '', 2: '1_1685562970.274572_93.1', 3: '1_1685563478.630332_193.1'} 2023-05-31T20:06:37.625 WARNING:rgw_multi.tests:zone a2 behind master: shard=3 master=1_1685563571.700188_212.1 target=1_1685563478.630332_193.1 2023-05-31T20:06:42.626 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-31T20:06:42.627 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-31T20:06:42.627 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-05-31T20:06:42.688 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2023-05-31T20:06:42.688 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2023-05-31T20:06:42.738 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:06:42.732+0000 7f7babc86780 20 RGW-SYNC:meta: read sync status 2023-05-31T20:06:42.739 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:06:42.733+0000 7f7babc86780 20 RGW-SYNC:meta: read sync status 2023-05-31T20:06:42.761 INFO:teuthology.orchestra.run.smithi099.stdout:{ 2023-05-31T20:06:42.762 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_status": { 2023-05-31T20:06:42.762 INFO:teuthology.orchestra.run.smithi099.stdout: "info": { 2023-05-31T20:06:42.763 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "sync", 2023-05-31T20:06:42.763 INFO:teuthology.orchestra.run.smithi099.stdout: "num_shards": 4, 2023-05-31T20:06:42.763 INFO:teuthology.orchestra.run.smithi099.stdout: "period": "2e6123d2-d695-4350-9a01-61a455bc782b", 2023-05-31T20:06:42.764 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2023-05-31T20:06:42.764 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:06:42.764 INFO:teuthology.orchestra.run.smithi099.stdout: "markers": [ 2023-05-31T20:06:42.765 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T20:06:42.765 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 0, 2023-05-31T20:06:42.765 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T20:06:42.766 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2023-05-31T20:06:42.766 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1685562631.516089_59.1", 2023-05-31T20:06:42.766 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T20:06:42.767 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 1, 2023-05-31T20:06:42.767 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T20:06:42.767 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2023-05-31T19:50:31.516089Z", 2023-05-31T20:06:42.768 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2023-05-31T20:06:42.768 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T20:06:42.768 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:06:42.769 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T20:06:42.769 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 1, 2023-05-31T20:06:42.769 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T20:06:42.770 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2023-05-31T20:06:42.770 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "", 2023-05-31T20:06:42.770 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T20:06:42.770 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2023-05-31T20:06:42.770 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T20:06:42.770 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "0.000000", 2023-05-31T20:06:42.770 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2023-05-31T20:06:42.771 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T20:06:42.771 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:06:42.771 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T20:06:42.771 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 2, 2023-05-31T20:06:42.771 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T20:06:42.772 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2023-05-31T20:06:42.772 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1685562970.274572_93.1", 2023-05-31T20:06:42.772 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T20:06:42.773 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2023-05-31T20:06:42.773 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T20:06:42.773 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2023-05-31T19:56:10.274572Z", 2023-05-31T20:06:42.773 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2023-05-31T20:06:42.774 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T20:06:42.774 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:06:42.774 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T20:06:42.775 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 3, 2023-05-31T20:06:42.775 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T20:06:42.775 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2023-05-31T20:06:42.776 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1685563478.630332_193.1", 2023-05-31T20:06:42.776 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T20:06:42.776 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2023-05-31T20:06:42.776 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T20:06:42.777 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2023-05-31T20:04:38.630332Z", 2023-05-31T20:06:42.777 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2023-05-31T20:06:42.777 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T20:06:42.777 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T20:06:42.777 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2023-05-31T20:06:42.778 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:06:42.778 INFO:teuthology.orchestra.run.smithi099.stdout: "full_sync": { 2023-05-31T20:06:42.778 INFO:teuthology.orchestra.run.smithi099.stdout: "total": 1, 2023-05-31T20:06:42.778 INFO:teuthology.orchestra.run.smithi099.stdout: "complete": 1 2023-05-31T20:06:42.778 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:06:42.779 INFO:teuthology.orchestra.run.smithi099.stdout: "current_time": "2023-05-31T20:06:42Z" 2023-05-31T20:06:42.779 INFO:teuthology.orchestra.run.smithi099.stdout:} 2023-05-31T20:06:42.780 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "2e6123d2-d695-4350-9a01-61a455bc782b", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1685562631.516089_59.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-05-31T19:50:31.516089Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1685562970.274572_93.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-05-31T19:56:10.274572Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1685563478.630332_193.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-05-31T20:04:38.630332Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-05-31T20:06:42Z" } 2023-05-31T20:06:42.780 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1685562631.516089_59.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-05-31T19:50:31.516089Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1685562970.274572_93.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-05-31T19:56:10.274572Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1685563478.630332_193.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-05-31T20:04:38.630332Z', 'realm_epoch': 2}}] 2023-05-31T20:06:42.781 DEBUG:rgw_multi.tests:log_status={0: '1_1685562631.516089_59.1', 1: '', 2: '1_1685562970.274572_93.1', 3: '1_1685563571.700188_212.1'} 2023-05-31T20:06:42.781 DEBUG:rgw_multi.tests:sync_status={0: '1_1685562631.516089_59.1', 1: '', 2: '1_1685562970.274572_93.1', 3: '1_1685563478.630332_193.1'} 2023-05-31T20:06:42.782 WARNING:rgw_multi.tests:zone a2 behind master: shard=3 master=1_1685563571.700188_212.1 target=1_1685563478.630332_193.1 2023-05-31T20:06:47.783 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-31T20:06:47.784 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-31T20:06:47.784 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-05-31T20:06:47.842 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2023-05-31T20:06:47.843 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2023-05-31T20:06:47.878 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:06:47.873+0000 7f232bb95780 20 RGW-SYNC:meta: read sync status 2023-05-31T20:06:47.879 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:06:47.874+0000 7f232bb95780 20 RGW-SYNC:meta: read sync status 2023-05-31T20:06:47.892 INFO:teuthology.orchestra.run.smithi099.stdout:{ 2023-05-31T20:06:47.892 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_status": { 2023-05-31T20:06:47.892 INFO:teuthology.orchestra.run.smithi099.stdout: "info": { 2023-05-31T20:06:47.892 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "sync", 2023-05-31T20:06:47.893 INFO:teuthology.orchestra.run.smithi099.stdout: "num_shards": 4, 2023-05-31T20:06:47.893 INFO:teuthology.orchestra.run.smithi099.stdout: "period": "2e6123d2-d695-4350-9a01-61a455bc782b", 2023-05-31T20:06:47.893 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2023-05-31T20:06:47.893 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:06:47.894 INFO:teuthology.orchestra.run.smithi099.stdout: "markers": [ 2023-05-31T20:06:47.894 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T20:06:47.894 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 0, 2023-05-31T20:06:47.895 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T20:06:47.895 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2023-05-31T20:06:47.895 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1685562631.516089_59.1", 2023-05-31T20:06:47.896 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T20:06:47.896 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 1, 2023-05-31T20:06:47.896 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T20:06:47.897 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2023-05-31T19:50:31.516089Z", 2023-05-31T20:06:47.897 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2023-05-31T20:06:47.897 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T20:06:47.898 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:06:47.898 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T20:06:47.898 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 1, 2023-05-31T20:06:47.898 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T20:06:47.899 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2023-05-31T20:06:47.899 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "", 2023-05-31T20:06:47.899 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T20:06:47.900 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2023-05-31T20:06:47.900 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T20:06:47.900 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "0.000000", 2023-05-31T20:06:47.900 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2023-05-31T20:06:47.900 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T20:06:47.901 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:06:47.901 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T20:06:47.902 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 2, 2023-05-31T20:06:47.902 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T20:06:47.902 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2023-05-31T20:06:47.902 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1685562970.274572_93.1", 2023-05-31T20:06:47.902 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T20:06:47.903 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2023-05-31T20:06:47.903 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T20:06:47.903 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2023-05-31T19:56:10.274572Z", 2023-05-31T20:06:47.903 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2023-05-31T20:06:47.903 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T20:06:47.903 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:06:47.903 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T20:06:47.904 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 3, 2023-05-31T20:06:47.904 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T20:06:47.904 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2023-05-31T20:06:47.904 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1685563571.700188_212.1", 2023-05-31T20:06:47.905 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T20:06:47.905 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2023-05-31T20:06:47.905 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T20:06:47.905 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2023-05-31T20:06:11.700188Z", 2023-05-31T20:06:47.905 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2023-05-31T20:06:47.905 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T20:06:47.905 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T20:06:47.906 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2023-05-31T20:06:47.906 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:06:47.906 INFO:teuthology.orchestra.run.smithi099.stdout: "full_sync": { 2023-05-31T20:06:47.906 INFO:teuthology.orchestra.run.smithi099.stdout: "total": 1, 2023-05-31T20:06:47.906 INFO:teuthology.orchestra.run.smithi099.stdout: "complete": 1 2023-05-31T20:06:47.907 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:06:47.907 INFO:teuthology.orchestra.run.smithi099.stdout: "current_time": "2023-05-31T20:06:47Z" 2023-05-31T20:06:47.907 INFO:teuthology.orchestra.run.smithi099.stdout:} 2023-05-31T20:06:47.908 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "2e6123d2-d695-4350-9a01-61a455bc782b", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1685562631.516089_59.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-05-31T19:50:31.516089Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1685562970.274572_93.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-05-31T19:56:10.274572Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1685563571.700188_212.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-05-31T20:06:11.700188Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-05-31T20:06:47Z" } 2023-05-31T20:06:47.909 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1685562631.516089_59.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-05-31T19:50:31.516089Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1685562970.274572_93.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-05-31T19:56:10.274572Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1685563571.700188_212.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-05-31T20:06:11.700188Z', 'realm_epoch': 2}}] 2023-05-31T20:06:47.909 DEBUG:rgw_multi.tests:log_status={0: '1_1685562631.516089_59.1', 1: '', 2: '1_1685562970.274572_93.1', 3: '1_1685563571.700188_212.1'} 2023-05-31T20:06:47.910 DEBUG:rgw_multi.tests:sync_status={0: '1_1685562631.516089_59.1', 1: '', 2: '1_1685562970.274572_93.1', 3: '1_1685563571.700188_212.1'} 2023-05-31T20:06:47.910 INFO:rgw_multi.tests:finish meta checkpoint for zone=a2 2023-05-31T20:06:47.911 DEBUG:boto:path=/pklxkd-35/ 2023-05-31T20:06:47.911 DEBUG:boto:auth_path=/pklxkd-35/ 2023-05-31T20:06:47.912 DEBUG:boto:Method: HEAD 2023-05-31T20:06:47.912 DEBUG:boto:Path: /pklxkd-35/ 2023-05-31T20:06:47.912 DEBUG:boto:Data: 2023-05-31T20:06:47.913 DEBUG:boto:Headers: {} 2023-05-31T20:06:47.914 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:06:47.915 DEBUG:boto:Port: 8001 2023-05-31T20:06:47.915 DEBUG:boto:Params: {} 2023-05-31T20:06:47.916 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8001} 2023-05-31T20:06:47.916 DEBUG:boto:Token: None 2023-05-31T20:06:47.917 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:06:47 GMT /pklxkd-35/ 2023-05-31T20:06:47.917 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:6CP9R6ZyCH4qaAZqCRdzr+ioQdc= 2023-05-31T20:06:47.918 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:06:47 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:6CP9R6ZyCH4qaAZqCRdzr+ioQdc=', 'Content-Length': '0'} 2023-05-31T20:06:47.923 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000852c74e2beac051d-006477a8d7-150e-a2'), ('Content-Length', '0'), ('Date', 'Wed, 31 May 2023 20:06:47 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:06:47.923 DEBUG:boto:path=/pklxkd-35/testredirect 2023-05-31T20:06:47.924 DEBUG:boto:auth_path=/pklxkd-35/testredirect 2023-05-31T20:06:47.924 DEBUG:boto:Method: HEAD 2023-05-31T20:06:47.925 DEBUG:boto:Path: /pklxkd-35/testredirect 2023-05-31T20:06:47.925 DEBUG:boto:Data: 2023-05-31T20:06:47.926 DEBUG:boto:Headers: {} 2023-05-31T20:06:47.926 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:06:47.926 DEBUG:boto:Port: 8001 2023-05-31T20:06:47.927 DEBUG:boto:Params: {} 2023-05-31T20:06:47.927 DEBUG:boto:Token: None 2023-05-31T20:06:47.927 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:06:47 GMT /pklxkd-35/testredirect 2023-05-31T20:06:47.928 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:b75e0UlgNemNAyCPfDKFVeOscMM= 2023-05-31T20:06:47.928 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:06:47 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:b75e0UlgNemNAyCPfDKFVeOscMM=', 'Content-Length': '0'} 2023-05-31T20:06:47.930 DEBUG:boto:Response headers: [('Content-Length', '219'), ('x-amz-request-id', 'tx00000593f48ccae8d5466-006477a8d7-150e-a2'), ('Accept-Ranges', 'bytes'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:06:47 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:06:47.931 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['zone', 'modify', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--redirect-zone=3d163b1e-09c7-47e2-ae87-add4aace3f38', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-05-31T20:06:47.932 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'zone', 'modify', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--redirect-zone=3d163b1e-09c7-47e2-ae87-add4aace3f38', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-05-31T20:06:47.933 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 zone modify --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --redirect-zone=3d163b1e-09c7-47e2-ae87-add4aace3f38 --cluster c1 --debug-rgw 0 --debug-ms 0 2023-05-31T20:06:47.985 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2023-05-31T20:06:47.986 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2023-05-31T20:06:48.006 INFO:teuthology.orchestra.run.smithi099.stdout:{ 2023-05-31T20:06:48.006 INFO:teuthology.orchestra.run.smithi099.stdout: "id": "5d8acec7-7e57-4e56-8b32-e3d6b5036001", 2023-05-31T20:06:48.007 INFO:teuthology.orchestra.run.smithi099.stdout: "name": "a2", 2023-05-31T20:06:48.007 INFO:teuthology.orchestra.run.smithi099.stdout: "domain_root": "a2.rgw.meta:root", 2023-05-31T20:06:48.007 INFO:teuthology.orchestra.run.smithi099.stdout: "control_pool": "a2.rgw.control", 2023-05-31T20:06:48.007 INFO:teuthology.orchestra.run.smithi099.stdout: "gc_pool": "a2.rgw.log:gc", 2023-05-31T20:06:48.007 INFO:teuthology.orchestra.run.smithi099.stdout: "lc_pool": "a2.rgw.log:lc", 2023-05-31T20:06:48.007 INFO:teuthology.orchestra.run.smithi099.stdout: "log_pool": "a2.rgw.log", 2023-05-31T20:06:48.008 INFO:teuthology.orchestra.run.smithi099.stdout: "intent_log_pool": "a2.rgw.log:intent", 2023-05-31T20:06:48.008 INFO:teuthology.orchestra.run.smithi099.stdout: "usage_log_pool": "a2.rgw.log:usage", 2023-05-31T20:06:48.008 INFO:teuthology.orchestra.run.smithi099.stdout: "roles_pool": "a2.rgw.meta:roles", 2023-05-31T20:06:48.008 INFO:teuthology.orchestra.run.smithi099.stdout: "reshard_pool": "a2.rgw.log:reshard", 2023-05-31T20:06:48.009 INFO:teuthology.orchestra.run.smithi099.stdout: "user_keys_pool": "a2.rgw.meta:users.keys", 2023-05-31T20:06:48.009 INFO:teuthology.orchestra.run.smithi099.stdout: "user_email_pool": "a2.rgw.meta:users.email", 2023-05-31T20:06:48.009 INFO:teuthology.orchestra.run.smithi099.stdout: "user_swift_pool": "a2.rgw.meta:users.swift", 2023-05-31T20:06:48.009 INFO:teuthology.orchestra.run.smithi099.stdout: "user_uid_pool": "a2.rgw.meta:users.uid", 2023-05-31T20:06:48.009 INFO:teuthology.orchestra.run.smithi099.stdout: "otp_pool": "a2.rgw.otp", 2023-05-31T20:06:48.010 INFO:teuthology.orchestra.run.smithi099.stdout: "system_key": { 2023-05-31T20:06:48.010 INFO:teuthology.orchestra.run.smithi099.stdout: "access_key": "RVN19ED3W7FH7E08", 2023-05-31T20:06:48.011 INFO:teuthology.orchestra.run.smithi099.stdout: "secret_key": "XzQjGe60gbIUStGfs96TKhuPw3IQi3hK" 2023-05-31T20:06:48.011 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:06:48.011 INFO:teuthology.orchestra.run.smithi099.stdout: "placement_pools": [ 2023-05-31T20:06:48.011 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T20:06:48.011 INFO:teuthology.orchestra.run.smithi099.stdout: "key": "default-placement", 2023-05-31T20:06:48.012 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T20:06:48.012 INFO:teuthology.orchestra.run.smithi099.stdout: "index_pool": "a2.rgw.buckets.index", 2023-05-31T20:06:48.012 INFO:teuthology.orchestra.run.smithi099.stdout: "storage_classes": { 2023-05-31T20:06:48.012 INFO:teuthology.orchestra.run.smithi099.stdout: "STANDARD": { 2023-05-31T20:06:48.012 INFO:teuthology.orchestra.run.smithi099.stdout: "data_pool": "a2.rgw.buckets.data", 2023-05-31T20:06:48.013 INFO:teuthology.orchestra.run.smithi099.stdout: "compression_type": "random" 2023-05-31T20:06:48.013 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T20:06:48.013 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:06:48.014 INFO:teuthology.orchestra.run.smithi099.stdout: "data_extra_pool": "a2.rgw.buckets.non-ec", 2023-05-31T20:06:48.014 INFO:teuthology.orchestra.run.smithi099.stdout: "index_type": 0, 2023-05-31T20:06:48.014 INFO:teuthology.orchestra.run.smithi099.stdout: "inline_data": true 2023-05-31T20:06:48.014 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T20:06:48.014 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T20:06:48.014 INFO:teuthology.orchestra.run.smithi099.stdout: ], 2023-05-31T20:06:48.015 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_id": "aaa33de6-515b-4cac-8f1b-3a1a30452117", 2023-05-31T20:06:48.015 INFO:teuthology.orchestra.run.smithi099.stdout: "notif_pool": "a2.rgw.log:notif" 2023-05-31T20:06:48.015 INFO:teuthology.orchestra.run.smithi099.stdout:} 2023-05-31T20:06:48.016 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['period', 'update', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-05-31T20:06:48.016 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'period', 'update', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-05-31T20:06:48.017 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 period update --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --commit --cluster c1 --debug-rgw 0 --debug-ms 0 2023-05-31T20:06:48.066 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2023-05-31T20:06:48.066 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2023-05-31T20:06:48.117 INFO:teuthology.orchestra.run.smithi099.stderr:Sending period to new master zone 3d163b1e-09c7-47e2-ae87-add4aace3f38 2023-05-31T20:06:48.163 INFO:teuthology.orchestra.run.smithi099.stdout:{ 2023-05-31T20:06:48.163 INFO:teuthology.orchestra.run.smithi099.stdout: "id": "2e6123d2-d695-4350-9a01-61a455bc782b", 2023-05-31T20:06:48.164 INFO:teuthology.orchestra.run.smithi099.stdout: "epoch": 6, 2023-05-31T20:06:48.164 INFO:teuthology.orchestra.run.smithi099.stdout: "predecessor_uuid": "629b9ced-80af-41ed-8a6c-f2cfc334cbb4", 2023-05-31T20:06:48.165 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_status": [], 2023-05-31T20:06:48.165 INFO:teuthology.orchestra.run.smithi099.stdout: "period_map": { 2023-05-31T20:06:48.165 INFO:teuthology.orchestra.run.smithi099.stdout: "id": "2e6123d2-d695-4350-9a01-61a455bc782b", 2023-05-31T20:06:48.166 INFO:teuthology.orchestra.run.smithi099.stdout: "zonegroups": [ 2023-05-31T20:06:48.166 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T20:06:48.166 INFO:teuthology.orchestra.run.smithi099.stdout: "id": "a", 2023-05-31T20:06:48.167 INFO:teuthology.orchestra.run.smithi099.stdout: "name": "a", 2023-05-31T20:06:48.167 INFO:teuthology.orchestra.run.smithi099.stdout: "api_name": "", 2023-05-31T20:06:48.167 INFO:teuthology.orchestra.run.smithi099.stdout: "is_master": true, 2023-05-31T20:06:48.168 INFO:teuthology.orchestra.run.smithi099.stdout: "endpoints": [ 2023-05-31T20:06:48.168 INFO:teuthology.orchestra.run.smithi099.stdout: "http://smithi099.front.sepia.ceph.com:8000" 2023-05-31T20:06:48.168 INFO:teuthology.orchestra.run.smithi099.stdout: ], 2023-05-31T20:06:48.169 INFO:teuthology.orchestra.run.smithi099.stdout: "hostnames": [], 2023-05-31T20:06:48.169 INFO:teuthology.orchestra.run.smithi099.stdout: "hostnames_s3website": [], 2023-05-31T20:06:48.169 INFO:teuthology.orchestra.run.smithi099.stdout: "master_zone": "3d163b1e-09c7-47e2-ae87-add4aace3f38", 2023-05-31T20:06:48.170 INFO:teuthology.orchestra.run.smithi099.stdout: "zones": [ 2023-05-31T20:06:48.170 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T20:06:48.171 INFO:teuthology.orchestra.run.smithi099.stdout: "id": "3d163b1e-09c7-47e2-ae87-add4aace3f38", 2023-05-31T20:06:48.171 INFO:teuthology.orchestra.run.smithi099.stdout: "name": "a1", 2023-05-31T20:06:48.171 INFO:teuthology.orchestra.run.smithi099.stdout: "endpoints": [ 2023-05-31T20:06:48.172 INFO:teuthology.orchestra.run.smithi099.stdout: "http://smithi099.front.sepia.ceph.com:8000" 2023-05-31T20:06:48.172 INFO:teuthology.orchestra.run.smithi099.stdout: ], 2023-05-31T20:06:48.173 INFO:teuthology.orchestra.run.smithi099.stdout: "log_meta": false, 2023-05-31T20:06:48.173 INFO:teuthology.orchestra.run.smithi099.stdout: "log_data": true, 2023-05-31T20:06:48.173 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket_index_max_shards": 11, 2023-05-31T20:06:48.173 INFO:teuthology.orchestra.run.smithi099.stdout: "read_only": false, 2023-05-31T20:06:48.174 INFO:teuthology.orchestra.run.smithi099.stdout: "tier_type": "", 2023-05-31T20:06:48.174 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_from_all": true, 2023-05-31T20:06:48.174 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_from": [], 2023-05-31T20:06:48.174 INFO:teuthology.orchestra.run.smithi099.stdout: "redirect_zone": "", 2023-05-31T20:06:48.176 INFO:teuthology.orchestra.run.smithi099.stdout: "supported_features": [ 2023-05-31T20:06:48.176 INFO:teuthology.orchestra.run.smithi099.stdout: "resharding" 2023-05-31T20:06:48.176 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2023-05-31T20:06:48.177 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:06:48.177 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T20:06:48.177 INFO:teuthology.orchestra.run.smithi099.stdout: "id": "5d8acec7-7e57-4e56-8b32-e3d6b5036001", 2023-05-31T20:06:48.177 INFO:teuthology.orchestra.run.smithi099.stdout: "name": "a2", 2023-05-31T20:06:48.178 INFO:teuthology.orchestra.run.smithi099.stdout: "endpoints": [ 2023-05-31T20:06:48.178 INFO:teuthology.orchestra.run.smithi099.stdout: "http://smithi099.front.sepia.ceph.com:8001" 2023-05-31T20:06:48.178 INFO:teuthology.orchestra.run.smithi099.stdout: ], 2023-05-31T20:06:48.178 INFO:teuthology.orchestra.run.smithi099.stdout: "log_meta": false, 2023-05-31T20:06:48.178 INFO:teuthology.orchestra.run.smithi099.stdout: "log_data": true, 2023-05-31T20:06:48.178 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket_index_max_shards": 11, 2023-05-31T20:06:48.178 INFO:teuthology.orchestra.run.smithi099.stdout: "read_only": false, 2023-05-31T20:06:48.179 INFO:teuthology.orchestra.run.smithi099.stdout: "tier_type": "", 2023-05-31T20:06:48.179 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_from_all": false, 2023-05-31T20:06:48.179 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_from": [], 2023-05-31T20:06:48.179 INFO:teuthology.orchestra.run.smithi099.stdout: "redirect_zone": "3d163b1e-09c7-47e2-ae87-add4aace3f38", 2023-05-31T20:06:48.179 INFO:teuthology.orchestra.run.smithi099.stdout: "supported_features": [ 2023-05-31T20:06:48.180 INFO:teuthology.orchestra.run.smithi099.stdout: "resharding" 2023-05-31T20:06:48.180 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2023-05-31T20:06:48.180 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T20:06:48.180 INFO:teuthology.orchestra.run.smithi099.stdout: ], 2023-05-31T20:06:48.180 INFO:teuthology.orchestra.run.smithi099.stdout: "placement_targets": [ 2023-05-31T20:06:48.180 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T20:06:48.181 INFO:teuthology.orchestra.run.smithi099.stdout: "name": "default-placement", 2023-05-31T20:06:48.181 INFO:teuthology.orchestra.run.smithi099.stdout: "tags": [], 2023-05-31T20:06:48.181 INFO:teuthology.orchestra.run.smithi099.stdout: "storage_classes": [ 2023-05-31T20:06:48.181 INFO:teuthology.orchestra.run.smithi099.stdout: "STANDARD" 2023-05-31T20:06:48.181 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2023-05-31T20:06:48.181 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T20:06:48.182 INFO:teuthology.orchestra.run.smithi099.stdout: ], 2023-05-31T20:06:48.182 INFO:teuthology.orchestra.run.smithi099.stdout: "default_placement": "default-placement", 2023-05-31T20:06:48.182 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_id": "aaa33de6-515b-4cac-8f1b-3a1a30452117", 2023-05-31T20:06:48.182 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_policy": { 2023-05-31T20:06:48.182 INFO:teuthology.orchestra.run.smithi099.stdout: "groups": [] 2023-05-31T20:06:48.182 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:06:48.182 INFO:teuthology.orchestra.run.smithi099.stdout: "enabled_features": [ 2023-05-31T20:06:48.183 INFO:teuthology.orchestra.run.smithi099.stdout: "resharding" 2023-05-31T20:06:48.183 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2023-05-31T20:06:48.183 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:06:48.183 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T20:06:48.183 INFO:teuthology.orchestra.run.smithi099.stdout: "id": "b", 2023-05-31T20:06:48.183 INFO:teuthology.orchestra.run.smithi099.stdout: "name": "b", 2023-05-31T20:06:48.183 INFO:teuthology.orchestra.run.smithi099.stdout: "api_name": "", 2023-05-31T20:06:48.184 INFO:teuthology.orchestra.run.smithi099.stdout: "is_master": false, 2023-05-31T20:06:48.184 INFO:teuthology.orchestra.run.smithi099.stdout: "endpoints": [ 2023-05-31T20:06:48.184 INFO:teuthology.orchestra.run.smithi099.stdout: "http://smithi112.front.sepia.ceph.com:8000" 2023-05-31T20:06:48.184 INFO:teuthology.orchestra.run.smithi099.stdout: ], 2023-05-31T20:06:48.184 INFO:teuthology.orchestra.run.smithi099.stdout: "hostnames": [], 2023-05-31T20:06:48.184 INFO:teuthology.orchestra.run.smithi099.stdout: "hostnames_s3website": [], 2023-05-31T20:06:48.185 INFO:teuthology.orchestra.run.smithi099.stdout: "master_zone": "f85d7236-2f27-4988-af26-9c66ab280b1b", 2023-05-31T20:06:48.185 INFO:teuthology.orchestra.run.smithi099.stdout: "zones": [ 2023-05-31T20:06:48.185 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T20:06:48.185 INFO:teuthology.orchestra.run.smithi099.stdout: "id": "a6a3aab5-bc52-44a4-8543-5c5ea6056acd", 2023-05-31T20:06:48.185 INFO:teuthology.orchestra.run.smithi099.stdout: "name": "b2", 2023-05-31T20:06:48.185 INFO:teuthology.orchestra.run.smithi099.stdout: "endpoints": [ 2023-05-31T20:06:48.185 INFO:teuthology.orchestra.run.smithi099.stdout: "http://smithi112.front.sepia.ceph.com:8001" 2023-05-31T20:06:48.186 INFO:teuthology.orchestra.run.smithi099.stdout: ], 2023-05-31T20:06:48.186 INFO:teuthology.orchestra.run.smithi099.stdout: "log_meta": false, 2023-05-31T20:06:48.186 INFO:teuthology.orchestra.run.smithi099.stdout: "log_data": true, 2023-05-31T20:06:48.186 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket_index_max_shards": 11, 2023-05-31T20:06:48.186 INFO:teuthology.orchestra.run.smithi099.stdout: "read_only": false, 2023-05-31T20:06:48.186 INFO:teuthology.orchestra.run.smithi099.stdout: "tier_type": "", 2023-05-31T20:06:48.186 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_from_all": true, 2023-05-31T20:06:48.187 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_from": [], 2023-05-31T20:06:48.187 INFO:teuthology.orchestra.run.smithi099.stdout: "redirect_zone": "", 2023-05-31T20:06:48.187 INFO:teuthology.orchestra.run.smithi099.stdout: "supported_features": [ 2023-05-31T20:06:48.187 INFO:teuthology.orchestra.run.smithi099.stdout: "resharding" 2023-05-31T20:06:48.187 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2023-05-31T20:06:48.187 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:06:48.187 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T20:06:48.188 INFO:teuthology.orchestra.run.smithi099.stdout: "id": "f85d7236-2f27-4988-af26-9c66ab280b1b", 2023-05-31T20:06:48.189 INFO:teuthology.orchestra.run.smithi099.stdout: "name": "b1", 2023-05-31T20:06:48.189 INFO:teuthology.orchestra.run.smithi099.stdout: "endpoints": [ 2023-05-31T20:06:48.189 INFO:teuthology.orchestra.run.smithi099.stdout: "http://smithi112.front.sepia.ceph.com:8000" 2023-05-31T20:06:48.189 INFO:teuthology.orchestra.run.smithi099.stdout: ], 2023-05-31T20:06:48.189 INFO:teuthology.orchestra.run.smithi099.stdout: "log_meta": false, 2023-05-31T20:06:48.189 INFO:teuthology.orchestra.run.smithi099.stdout: "log_data": true, 2023-05-31T20:06:48.190 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket_index_max_shards": 11, 2023-05-31T20:06:48.190 INFO:teuthology.orchestra.run.smithi099.stdout: "read_only": false, 2023-05-31T20:06:48.190 INFO:teuthology.orchestra.run.smithi099.stdout: "tier_type": "", 2023-05-31T20:06:48.190 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_from_all": true, 2023-05-31T20:06:48.190 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_from": [], 2023-05-31T20:06:48.190 INFO:teuthology.orchestra.run.smithi099.stdout: "redirect_zone": "", 2023-05-31T20:06:48.190 INFO:teuthology.orchestra.run.smithi099.stdout: "supported_features": [ 2023-05-31T20:06:48.191 INFO:teuthology.orchestra.run.smithi099.stdout: "resharding" 2023-05-31T20:06:48.191 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2023-05-31T20:06:48.191 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T20:06:48.191 INFO:teuthology.orchestra.run.smithi099.stdout: ], 2023-05-31T20:06:48.191 INFO:teuthology.orchestra.run.smithi099.stdout: "placement_targets": [ 2023-05-31T20:06:48.191 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T20:06:48.192 INFO:teuthology.orchestra.run.smithi099.stdout: "name": "default-placement", 2023-05-31T20:06:48.192 INFO:teuthology.orchestra.run.smithi099.stdout: "tags": [], 2023-05-31T20:06:48.192 INFO:teuthology.orchestra.run.smithi099.stdout: "storage_classes": [ 2023-05-31T20:06:48.192 INFO:teuthology.orchestra.run.smithi099.stdout: "STANDARD" 2023-05-31T20:06:48.192 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2023-05-31T20:06:48.192 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T20:06:48.192 INFO:teuthology.orchestra.run.smithi099.stdout: ], 2023-05-31T20:06:48.193 INFO:teuthology.orchestra.run.smithi099.stdout: "default_placement": "default-placement", 2023-05-31T20:06:48.193 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_id": "aaa33de6-515b-4cac-8f1b-3a1a30452117", 2023-05-31T20:06:48.193 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_policy": { 2023-05-31T20:06:48.193 INFO:teuthology.orchestra.run.smithi099.stdout: "groups": [] 2023-05-31T20:06:48.193 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:06:48.193 INFO:teuthology.orchestra.run.smithi099.stdout: "enabled_features": [ 2023-05-31T20:06:48.193 INFO:teuthology.orchestra.run.smithi099.stdout: "resharding" 2023-05-31T20:06:48.194 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2023-05-31T20:06:48.194 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T20:06:48.194 INFO:teuthology.orchestra.run.smithi099.stdout: ], 2023-05-31T20:06:48.194 INFO:teuthology.orchestra.run.smithi099.stdout: "short_zone_ids": [ 2023-05-31T20:06:48.194 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T20:06:48.194 INFO:teuthology.orchestra.run.smithi099.stdout: "key": "3d163b1e-09c7-47e2-ae87-add4aace3f38", 2023-05-31T20:06:48.194 INFO:teuthology.orchestra.run.smithi099.stdout: "val": 3101833771 2023-05-31T20:06:48.195 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:06:48.195 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T20:06:48.195 INFO:teuthology.orchestra.run.smithi099.stdout: "key": "5d8acec7-7e57-4e56-8b32-e3d6b5036001", 2023-05-31T20:06:48.195 INFO:teuthology.orchestra.run.smithi099.stdout: "val": 4255530623 2023-05-31T20:06:48.195 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:06:48.195 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T20:06:48.195 INFO:teuthology.orchestra.run.smithi099.stdout: "key": "a6a3aab5-bc52-44a4-8543-5c5ea6056acd", 2023-05-31T20:06:48.196 INFO:teuthology.orchestra.run.smithi099.stdout: "val": 3708073710 2023-05-31T20:06:48.196 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:06:48.196 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T20:06:48.196 INFO:teuthology.orchestra.run.smithi099.stdout: "key": "f85d7236-2f27-4988-af26-9c66ab280b1b", 2023-05-31T20:06:48.196 INFO:teuthology.orchestra.run.smithi099.stdout: "val": 233395200 2023-05-31T20:06:48.196 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T20:06:48.197 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2023-05-31T20:06:48.197 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:06:48.197 INFO:teuthology.orchestra.run.smithi099.stdout: "master_zonegroup": "a", 2023-05-31T20:06:48.197 INFO:teuthology.orchestra.run.smithi099.stdout: "master_zone": "3d163b1e-09c7-47e2-ae87-add4aace3f38", 2023-05-31T20:06:48.197 INFO:teuthology.orchestra.run.smithi099.stdout: "period_config": { 2023-05-31T20:06:48.197 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket_quota": { 2023-05-31T20:06:48.197 INFO:teuthology.orchestra.run.smithi099.stdout: "enabled": false, 2023-05-31T20:06:48.198 INFO:teuthology.orchestra.run.smithi099.stdout: "check_on_raw": false, 2023-05-31T20:06:48.198 INFO:teuthology.orchestra.run.smithi099.stdout: "max_size": -1, 2023-05-31T20:06:48.198 INFO:teuthology.orchestra.run.smithi099.stdout: "max_size_kb": 0, 2023-05-31T20:06:48.198 INFO:teuthology.orchestra.run.smithi099.stdout: "max_objects": -1 2023-05-31T20:06:48.198 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:06:48.198 INFO:teuthology.orchestra.run.smithi099.stdout: "user_quota": { 2023-05-31T20:06:48.198 INFO:teuthology.orchestra.run.smithi099.stdout: "enabled": false, 2023-05-31T20:06:48.199 INFO:teuthology.orchestra.run.smithi099.stdout: "check_on_raw": false, 2023-05-31T20:06:48.199 INFO:teuthology.orchestra.run.smithi099.stdout: "max_size": -1, 2023-05-31T20:06:48.199 INFO:teuthology.orchestra.run.smithi099.stdout: "max_size_kb": 0, 2023-05-31T20:06:48.199 INFO:teuthology.orchestra.run.smithi099.stdout: "max_objects": -1 2023-05-31T20:06:48.199 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:06:48.199 INFO:teuthology.orchestra.run.smithi099.stdout: "user_ratelimit": { 2023-05-31T20:06:48.199 INFO:teuthology.orchestra.run.smithi099.stdout: "max_read_ops": 0, 2023-05-31T20:06:48.200 INFO:teuthology.orchestra.run.smithi099.stdout: "max_write_ops": 0, 2023-05-31T20:06:48.200 INFO:teuthology.orchestra.run.smithi099.stdout: "max_read_bytes": 0, 2023-05-31T20:06:48.200 INFO:teuthology.orchestra.run.smithi099.stdout: "max_write_bytes": 0, 2023-05-31T20:06:48.200 INFO:teuthology.orchestra.run.smithi099.stdout: "enabled": false 2023-05-31T20:06:48.200 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:06:48.200 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket_ratelimit": { 2023-05-31T20:06:48.200 INFO:teuthology.orchestra.run.smithi099.stdout: "max_read_ops": 0, 2023-05-31T20:06:48.201 INFO:teuthology.orchestra.run.smithi099.stdout: "max_write_ops": 0, 2023-05-31T20:06:48.201 INFO:teuthology.orchestra.run.smithi099.stdout: "max_read_bytes": 0, 2023-05-31T20:06:48.201 INFO:teuthology.orchestra.run.smithi099.stdout: "max_write_bytes": 0, 2023-05-31T20:06:48.201 INFO:teuthology.orchestra.run.smithi099.stdout: "enabled": false 2023-05-31T20:06:48.201 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:06:48.201 INFO:teuthology.orchestra.run.smithi099.stdout: "anonymous_ratelimit": { 2023-05-31T20:06:48.202 INFO:teuthology.orchestra.run.smithi099.stdout: "max_read_ops": 0, 2023-05-31T20:06:48.202 INFO:teuthology.orchestra.run.smithi099.stdout: "max_write_ops": 0, 2023-05-31T20:06:48.202 INFO:teuthology.orchestra.run.smithi099.stdout: "max_read_bytes": 0, 2023-05-31T20:06:48.202 INFO:teuthology.orchestra.run.smithi099.stdout: "max_write_bytes": 0, 2023-05-31T20:06:48.202 INFO:teuthology.orchestra.run.smithi099.stdout: "enabled": false 2023-05-31T20:06:48.202 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T20:06:48.202 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:06:48.203 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_id": "aaa33de6-515b-4cac-8f1b-3a1a30452117", 2023-05-31T20:06:48.203 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_name": "test-realm", 2023-05-31T20:06:48.203 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2023-05-31T20:06:48.203 INFO:teuthology.orchestra.run.smithi099.stdout:} 2023-05-31T20:06:48.203 INFO:rgw_multi.tests:Set redirect_zone zone a2 to "3d163b1e-09c7-47e2-ae87-add4aace3f38" 2023-05-31T20:07:48.204 DEBUG:boto:path=/pklxkd-35/testredirect 2023-05-31T20:07:48.205 DEBUG:boto:auth_path=/pklxkd-35/testredirect 2023-05-31T20:07:48.206 DEBUG:boto:Method: HEAD 2023-05-31T20:07:48.206 DEBUG:boto:Path: /pklxkd-35/testredirect 2023-05-31T20:07:48.206 DEBUG:boto:Data: 2023-05-31T20:07:48.207 DEBUG:boto:Headers: {} 2023-05-31T20:07:48.207 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:07:48.207 DEBUG:boto:Port: 8001 2023-05-31T20:07:48.208 DEBUG:boto:Params: {} 2023-05-31T20:07:48.208 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8001} 2023-05-31T20:07:48.209 DEBUG:boto:Token: None 2023-05-31T20:07:48.209 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:07:48 GMT /pklxkd-35/testredirect 2023-05-31T20:07:48.210 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:8WF1fHBMar0Mtllhrlf+LxxDH/c= 2023-05-31T20:07:48.210 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:07:48 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:8WF1fHBMar0Mtllhrlf+LxxDH/c=', 'Content-Length': '0'} 2023-05-31T20:07:48.215 DEBUG:boto:Response headers: [('Location', 'http://smithi099.front.sepia.ceph.com:8000/pklxkd-35/testredirect?'), ('x-amz-request-id', 'tx0000060e64ea55cd2d654-006477a914-156e-a2'), ('Content-Length', '0'), ('Date', 'Wed, 31 May 2023 20:07:48 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:07:48.215 DEBUG:boto:Redirecting: http://smithi099.front.sepia.ceph.com/pklxkd-35/testredirect 2023-05-31T20:07:48.216 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': '8000'} 2023-05-31T20:07:48.216 DEBUG:boto:Token: None 2023-05-31T20:07:48.216 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:07:48 GMT /pklxkd-35/testredirect 2023-05-31T20:07:48.217 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:8WF1fHBMar0Mtllhrlf+LxxDH/c= 2023-05-31T20:07:48.217 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:07:48 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:8WF1fHBMar0Mtllhrlf+LxxDH/c=', 'Content-Length': '0'} 2023-05-31T20:07:48.222 DEBUG:boto:Response headers: [('Content-Length', '512'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:06:11 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"dc5086b847289ba8b8bde149b8388175"'), ('x-amz-request-id', 'tx00000e4f1d32c6cfa8d56-006477a914-1572-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:07:48 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:07:48.223 DEBUG:boto:path=/pklxkd-35/testredirect 2023-05-31T20:07:48.223 DEBUG:boto:auth_path=/pklxkd-35/testredirect 2023-05-31T20:07:48.224 DEBUG:boto:Method: GET 2023-05-31T20:07:48.224 DEBUG:boto:Path: /pklxkd-35/testredirect 2023-05-31T20:07:48.225 DEBUG:boto:Data: 2023-05-31T20:07:48.225 DEBUG:boto:Headers: {} 2023-05-31T20:07:48.225 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:07:48.226 DEBUG:boto:Port: 8001 2023-05-31T20:07:48.226 DEBUG:boto:Params: {} 2023-05-31T20:07:48.227 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8001} 2023-05-31T20:07:48.227 DEBUG:boto:Token: None 2023-05-31T20:07:48.227 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:07:48 GMT /pklxkd-35/testredirect 2023-05-31T20:07:48.228 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:emPPRtML3tQydRGM1ja0Z+DQ2yA= 2023-05-31T20:07:48.228 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:07:48 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:emPPRtML3tQydRGM1ja0Z+DQ2yA=', 'Content-Length': '0'} 2023-05-31T20:07:48.231 DEBUG:boto:Response headers: [('Location', 'http://smithi099.front.sepia.ceph.com:8000/pklxkd-35/testredirect?'), ('x-amz-request-id', 'tx00000e6687f0813398ec9-006477a914-156e-a2'), ('Content-Length', '0'), ('Date', 'Wed, 31 May 2023 20:07:48 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:07:48.231 DEBUG:boto:Redirecting: http://smithi099.front.sepia.ceph.com/pklxkd-35/testredirect 2023-05-31T20:07:48.232 DEBUG:boto:Token: None 2023-05-31T20:07:48.232 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:07:48 GMT /pklxkd-35/testredirect 2023-05-31T20:07:48.233 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:emPPRtML3tQydRGM1ja0Z+DQ2yA= 2023-05-31T20:07:48.233 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:07:48 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:emPPRtML3tQydRGM1ja0Z+DQ2yA=', 'Content-Length': '0'} 2023-05-31T20:07:48.235 DEBUG:boto:Response headers: [('Content-Length', '512'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:06:11 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"dc5086b847289ba8b8bde149b8388175"'), ('x-amz-request-id', 'tx00000d3247cd6b4718437-006477a914-1572-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:07:48 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:07:48.236 DEBUG:boto:path=/pklxkd-35/testredirect 2023-05-31T20:07:48.236 DEBUG:boto:auth_path=/pklxkd-35/testredirect 2023-05-31T20:07:48.236 DEBUG:boto:Method: PUT 2023-05-31T20:07:48.237 DEBUG:boto:Path: /pklxkd-35/testredirect 2023-05-31T20:07:48.237 DEBUG:boto:Data: 2023-05-31T20:07:48.237 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'VpBzljOcorCZvRIkX5Nt3A==', 'Content-Length': '512', 'Expect': '100-Continue'} 2023-05-31T20:07:48.238 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:07:48.238 DEBUG:boto:Port: 8000 2023-05-31T20:07:48.238 DEBUG:boto:Params: {} 2023-05-31T20:07:48.240 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2023-05-31T20:07:48.240 DEBUG:boto:Token: None 2023-05-31T20:07:48.241 DEBUG:boto:StringToSign: PUT VpBzljOcorCZvRIkX5Nt3A== application/octet-stream Wed, 31 May 2023 20:07:48 GMT /pklxkd-35/testredirect 2023-05-31T20:07:48.241 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:VFT93P5mU+Rtv0BWmue6/an+uL8= 2023-05-31T20:07:48.241 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'VpBzljOcorCZvRIkX5Nt3A==', 'Content-Length': '512', 'Expect': '100-Continue', 'Date': 'Wed, 31 May 2023 20:07:48 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:VFT93P5mU+Rtv0BWmue6/an+uL8='} 2023-05-31T20:07:48.253 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"56907396339ca2b099bd12245f936ddc"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000007f51bf8322a3715a-006477a914-1572-a1'), ('Date', 'Wed, 31 May 2023 20:07:48 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:07:48.254 DEBUG:boto:path=/pklxkd-35/testredirect 2023-05-31T20:07:48.255 DEBUG:boto:auth_path=/pklxkd-35/testredirect 2023-05-31T20:07:48.255 DEBUG:boto:Method: GET 2023-05-31T20:07:48.255 DEBUG:boto:Path: /pklxkd-35/testredirect 2023-05-31T20:07:48.256 DEBUG:boto:Data: 2023-05-31T20:07:48.256 DEBUG:boto:Headers: {} 2023-05-31T20:07:48.257 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:07:48.257 DEBUG:boto:Port: 8001 2023-05-31T20:07:48.258 DEBUG:boto:Params: {} 2023-05-31T20:07:48.258 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8001} 2023-05-31T20:07:48.258 DEBUG:boto:Token: None 2023-05-31T20:07:48.259 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:07:48 GMT /pklxkd-35/testredirect 2023-05-31T20:07:48.259 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:emPPRtML3tQydRGM1ja0Z+DQ2yA= 2023-05-31T20:07:48.260 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:07:48 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:emPPRtML3tQydRGM1ja0Z+DQ2yA=', 'Content-Length': '0'} 2023-05-31T20:07:48.263 DEBUG:boto:Response headers: [('Location', 'http://smithi099.front.sepia.ceph.com:8000/pklxkd-35/testredirect?'), ('x-amz-request-id', 'tx00000e0466a605ccd97ee-006477a914-156e-a2'), ('Content-Length', '0'), ('Date', 'Wed, 31 May 2023 20:07:48 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:07:48.263 DEBUG:boto:Redirecting: http://smithi099.front.sepia.ceph.com/pklxkd-35/testredirect 2023-05-31T20:07:48.264 DEBUG:boto:Token: None 2023-05-31T20:07:48.264 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:07:48 GMT /pklxkd-35/testredirect 2023-05-31T20:07:48.265 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:emPPRtML3tQydRGM1ja0Z+DQ2yA= 2023-05-31T20:07:48.265 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:07:48 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:emPPRtML3tQydRGM1ja0Z+DQ2yA=', 'Content-Length': '0'} 2023-05-31T20:07:48.268 DEBUG:boto:Response headers: [('Content-Length', '512'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:07:48 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"56907396339ca2b099bd12245f936ddc"'), ('x-amz-request-id', 'tx00000cd0cd6ddd7ad18fd-006477a914-1572-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:07:48 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:07:48.269 DEBUG:boto:path=/pklxkd-35/testredirect 2023-05-31T20:07:48.269 DEBUG:boto:auth_path=/pklxkd-35/testredirect 2023-05-31T20:07:48.270 DEBUG:boto:Method: PUT 2023-05-31T20:07:48.270 DEBUG:boto:Path: /pklxkd-35/testredirect 2023-05-31T20:07:48.270 DEBUG:boto:Data: 2023-05-31T20:07:48.271 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'uk9S5NXZfBvPq4jGr+LM5g==', 'Content-Length': '512', 'Expect': '100-Continue'} 2023-05-31T20:07:48.271 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:07:48.272 DEBUG:boto:Port: 8000 2023-05-31T20:07:48.272 DEBUG:boto:Params: {} 2023-05-31T20:07:48.273 DEBUG:boto:Token: None 2023-05-31T20:07:48.274 DEBUG:boto:StringToSign: PUT uk9S5NXZfBvPq4jGr+LM5g== application/octet-stream Wed, 31 May 2023 20:07:48 GMT /pklxkd-35/testredirect 2023-05-31T20:07:48.274 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:QsB6Gzki44m7wvvPlrEwwu62bW8= 2023-05-31T20:07:48.275 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'uk9S5NXZfBvPq4jGr+LM5g==', 'Content-Length': '512', 'Expect': '100-Continue', 'Date': 'Wed, 31 May 2023 20:07:48 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:QsB6Gzki44m7wvvPlrEwwu62bW8='} 2023-05-31T20:07:48.284 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"ba4f52e4d5d97c1bcfab88c6afe2cce6"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000d012fb09dbf614cf-006477a914-1572-a1'), ('Date', 'Wed, 31 May 2023 20:07:48 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:07:48.284 DEBUG:boto:path=/pklxkd-35/testredirect 2023-05-31T20:07:48.285 DEBUG:boto:auth_path=/pklxkd-35/testredirect 2023-05-31T20:07:48.285 DEBUG:boto:Method: GET 2023-05-31T20:07:48.286 DEBUG:boto:Path: /pklxkd-35/testredirect 2023-05-31T20:07:48.286 DEBUG:boto:Data: 2023-05-31T20:07:48.286 DEBUG:boto:Headers: {} 2023-05-31T20:07:48.287 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:07:48.287 DEBUG:boto:Port: 8001 2023-05-31T20:07:48.289 DEBUG:boto:Params: {} 2023-05-31T20:07:48.289 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8001} 2023-05-31T20:07:48.290 DEBUG:boto:Token: None 2023-05-31T20:07:48.290 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:07:48 GMT /pklxkd-35/testredirect 2023-05-31T20:07:48.291 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:emPPRtML3tQydRGM1ja0Z+DQ2yA= 2023-05-31T20:07:48.291 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:07:48 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:emPPRtML3tQydRGM1ja0Z+DQ2yA=', 'Content-Length': '0'} 2023-05-31T20:07:48.294 DEBUG:boto:Response headers: [('Location', 'http://smithi099.front.sepia.ceph.com:8000/pklxkd-35/testredirect?'), ('x-amz-request-id', 'tx00000930d954b75552cb4-006477a914-156e-a2'), ('Content-Length', '0'), ('Date', 'Wed, 31 May 2023 20:07:48 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:07:48.295 DEBUG:boto:Redirecting: http://smithi099.front.sepia.ceph.com/pklxkd-35/testredirect 2023-05-31T20:07:48.295 DEBUG:boto:Token: None 2023-05-31T20:07:48.296 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:07:48 GMT /pklxkd-35/testredirect 2023-05-31T20:07:48.296 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:emPPRtML3tQydRGM1ja0Z+DQ2yA= 2023-05-31T20:07:48.297 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:07:48 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:emPPRtML3tQydRGM1ja0Z+DQ2yA=', 'Content-Length': '0'} 2023-05-31T20:07:48.299 DEBUG:boto:Response headers: [('Content-Length', '512'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:07:48 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"ba4f52e4d5d97c1bcfab88c6afe2cce6"'), ('x-amz-request-id', 'tx00000b0eea103c16259d5-006477a914-1572-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:07:48 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:07:48.300 DEBUG:boto:path=/pklxkd-35/testredirect 2023-05-31T20:07:48.301 DEBUG:boto:auth_path=/pklxkd-35/testredirect 2023-05-31T20:07:48.301 DEBUG:boto:Method: PUT 2023-05-31T20:07:48.302 DEBUG:boto:Path: /pklxkd-35/testredirect 2023-05-31T20:07:48.302 DEBUG:boto:Data: 2023-05-31T20:07:48.303 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': '6aKmvaJMgaK7Z2rfDZvVeg==', 'Content-Length': '512', 'Expect': '100-Continue'} 2023-05-31T20:07:48.303 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:07:48.304 DEBUG:boto:Port: 8000 2023-05-31T20:07:48.305 DEBUG:boto:Params: {} 2023-05-31T20:07:48.305 DEBUG:boto:Token: None 2023-05-31T20:07:48.305 DEBUG:boto:StringToSign: PUT 6aKmvaJMgaK7Z2rfDZvVeg== application/octet-stream Wed, 31 May 2023 20:07:48 GMT /pklxkd-35/testredirect 2023-05-31T20:07:48.306 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:twsaz2EySu3GuR4fzFy4aq5W0ck= 2023-05-31T20:07:48.307 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': '6aKmvaJMgaK7Z2rfDZvVeg==', 'Content-Length': '512', 'Expect': '100-Continue', 'Date': 'Wed, 31 May 2023 20:07:48 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:twsaz2EySu3GuR4fzFy4aq5W0ck='} 2023-05-31T20:07:48.315 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"e9a2a6bda24c81a2bb676adf0d9bd57a"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000b8a8cc9a4180fec3-006477a914-1572-a1'), ('Date', 'Wed, 31 May 2023 20:07:48 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:07:48.315 DEBUG:boto:path=/pklxkd-35/testredirect 2023-05-31T20:07:48.316 DEBUG:boto:auth_path=/pklxkd-35/testredirect 2023-05-31T20:07:48.316 DEBUG:boto:Method: GET 2023-05-31T20:07:48.317 DEBUG:boto:Path: /pklxkd-35/testredirect 2023-05-31T20:07:48.317 DEBUG:boto:Data: 2023-05-31T20:07:48.317 DEBUG:boto:Headers: {} 2023-05-31T20:07:48.318 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:07:48.318 DEBUG:boto:Port: 8001 2023-05-31T20:07:48.319 DEBUG:boto:Params: {} 2023-05-31T20:07:48.319 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8001} 2023-05-31T20:07:48.319 DEBUG:boto:Token: None 2023-05-31T20:07:48.320 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:07:48 GMT /pklxkd-35/testredirect 2023-05-31T20:07:48.320 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:emPPRtML3tQydRGM1ja0Z+DQ2yA= 2023-05-31T20:07:48.321 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:07:48 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:emPPRtML3tQydRGM1ja0Z+DQ2yA=', 'Content-Length': '0'} 2023-05-31T20:07:48.324 DEBUG:boto:Response headers: [('Location', 'http://smithi099.front.sepia.ceph.com:8000/pklxkd-35/testredirect?'), ('x-amz-request-id', 'tx00000e320351ea281b16c-006477a914-156e-a2'), ('Content-Length', '0'), ('Date', 'Wed, 31 May 2023 20:07:48 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:07:48.324 DEBUG:boto:Redirecting: http://smithi099.front.sepia.ceph.com/pklxkd-35/testredirect 2023-05-31T20:07:48.325 DEBUG:boto:Token: None 2023-05-31T20:07:48.325 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:07:48 GMT /pklxkd-35/testredirect 2023-05-31T20:07:48.326 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:emPPRtML3tQydRGM1ja0Z+DQ2yA= 2023-05-31T20:07:48.326 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:07:48 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:emPPRtML3tQydRGM1ja0Z+DQ2yA=', 'Content-Length': '0'} 2023-05-31T20:07:48.328 DEBUG:boto:Response headers: [('Content-Length', '512'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:07:48 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"e9a2a6bda24c81a2bb676adf0d9bd57a"'), ('x-amz-request-id', 'tx00000b27a88804b47fae8-006477a914-1572-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:07:48 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:07:48.329 DEBUG:boto:path=/pklxkd-35/testredirect 2023-05-31T20:07:48.330 DEBUG:boto:auth_path=/pklxkd-35/testredirect 2023-05-31T20:07:48.330 DEBUG:boto:Method: PUT 2023-05-31T20:07:48.330 DEBUG:boto:Path: /pklxkd-35/testredirect 2023-05-31T20:07:48.331 DEBUG:boto:Data: 2023-05-31T20:07:48.331 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'GbNgNQ5MAukCtgY+rsT5Xw==', 'Content-Length': '512', 'Expect': '100-Continue'} 2023-05-31T20:07:48.331 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:07:48.332 DEBUG:boto:Port: 8000 2023-05-31T20:07:48.332 DEBUG:boto:Params: {} 2023-05-31T20:07:48.333 DEBUG:boto:Token: None 2023-05-31T20:07:48.333 DEBUG:boto:StringToSign: PUT GbNgNQ5MAukCtgY+rsT5Xw== application/octet-stream Wed, 31 May 2023 20:07:48 GMT /pklxkd-35/testredirect 2023-05-31T20:07:48.334 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:8Ga+g/BmjKR7l1BP91OZnR8q0wE= 2023-05-31T20:07:48.334 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'GbNgNQ5MAukCtgY+rsT5Xw==', 'Content-Length': '512', 'Expect': '100-Continue', 'Date': 'Wed, 31 May 2023 20:07:48 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:8Ga+g/BmjKR7l1BP91OZnR8q0wE='} 2023-05-31T20:07:48.376 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"19b360350e4c02e902b6063eaec4f95f"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000005181a5a5e73f2e2f-006477a914-1572-a1'), ('Date', 'Wed, 31 May 2023 20:07:48 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:07:48.377 DEBUG:boto:path=/pklxkd-35/testredirect 2023-05-31T20:07:48.377 DEBUG:boto:auth_path=/pklxkd-35/testredirect 2023-05-31T20:07:48.378 DEBUG:boto:Method: GET 2023-05-31T20:07:48.378 DEBUG:boto:Path: /pklxkd-35/testredirect 2023-05-31T20:07:48.379 DEBUG:boto:Data: 2023-05-31T20:07:48.379 DEBUG:boto:Headers: {} 2023-05-31T20:07:48.379 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:07:48.380 DEBUG:boto:Port: 8001 2023-05-31T20:07:48.380 DEBUG:boto:Params: {} 2023-05-31T20:07:48.380 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8001} 2023-05-31T20:07:48.381 DEBUG:boto:Token: None 2023-05-31T20:07:48.381 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:07:48 GMT /pklxkd-35/testredirect 2023-05-31T20:07:48.382 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:emPPRtML3tQydRGM1ja0Z+DQ2yA= 2023-05-31T20:07:48.382 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:07:48 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:emPPRtML3tQydRGM1ja0Z+DQ2yA=', 'Content-Length': '0'} 2023-05-31T20:07:48.384 DEBUG:boto:Response headers: [('Location', 'http://smithi099.front.sepia.ceph.com:8000/pklxkd-35/testredirect?'), ('x-amz-request-id', 'tx00000225291a0ac63e065-006477a914-156e-a2'), ('Content-Length', '0'), ('Date', 'Wed, 31 May 2023 20:07:48 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:07:48.385 DEBUG:boto:Redirecting: http://smithi099.front.sepia.ceph.com/pklxkd-35/testredirect 2023-05-31T20:07:48.385 DEBUG:boto:Token: None 2023-05-31T20:07:48.386 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:07:48 GMT /pklxkd-35/testredirect 2023-05-31T20:07:48.386 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:emPPRtML3tQydRGM1ja0Z+DQ2yA= 2023-05-31T20:07:48.387 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:07:48 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:emPPRtML3tQydRGM1ja0Z+DQ2yA=', 'Content-Length': '0'} 2023-05-31T20:07:48.389 DEBUG:boto:Response headers: [('Content-Length', '512'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:07:48 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"19b360350e4c02e902b6063eaec4f95f"'), ('x-amz-request-id', 'tx0000034f593fb79b732d1-006477a914-1572-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:07:48 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:07:48.389 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['zone', 'modify', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--sync-from-all=true', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-05-31T20:07:48.390 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'zone', 'modify', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--sync-from-all=true', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-05-31T20:07:48.391 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 zone modify --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --sync-from-all=true --cluster c1 --debug-rgw 0 --debug-ms 0 2023-05-31T20:07:48.444 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2023-05-31T20:07:48.444 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2023-05-31T20:07:48.465 INFO:teuthology.orchestra.run.smithi099.stdout:{ 2023-05-31T20:07:48.465 INFO:teuthology.orchestra.run.smithi099.stdout: "id": "5d8acec7-7e57-4e56-8b32-e3d6b5036001", 2023-05-31T20:07:48.465 INFO:teuthology.orchestra.run.smithi099.stdout: "name": "a2", 2023-05-31T20:07:48.465 INFO:teuthology.orchestra.run.smithi099.stdout: "domain_root": "a2.rgw.meta:root", 2023-05-31T20:07:48.466 INFO:teuthology.orchestra.run.smithi099.stdout: "control_pool": "a2.rgw.control", 2023-05-31T20:07:48.466 INFO:teuthology.orchestra.run.smithi099.stdout: "gc_pool": "a2.rgw.log:gc", 2023-05-31T20:07:48.466 INFO:teuthology.orchestra.run.smithi099.stdout: "lc_pool": "a2.rgw.log:lc", 2023-05-31T20:07:48.466 INFO:teuthology.orchestra.run.smithi099.stdout: "log_pool": "a2.rgw.log", 2023-05-31T20:07:48.466 INFO:teuthology.orchestra.run.smithi099.stdout: "intent_log_pool": "a2.rgw.log:intent", 2023-05-31T20:07:48.466 INFO:teuthology.orchestra.run.smithi099.stdout: "usage_log_pool": "a2.rgw.log:usage", 2023-05-31T20:07:48.466 INFO:teuthology.orchestra.run.smithi099.stdout: "roles_pool": "a2.rgw.meta:roles", 2023-05-31T20:07:48.467 INFO:teuthology.orchestra.run.smithi099.stdout: "reshard_pool": "a2.rgw.log:reshard", 2023-05-31T20:07:48.467 INFO:teuthology.orchestra.run.smithi099.stdout: "user_keys_pool": "a2.rgw.meta:users.keys", 2023-05-31T20:07:48.467 INFO:teuthology.orchestra.run.smithi099.stdout: "user_email_pool": "a2.rgw.meta:users.email", 2023-05-31T20:07:48.467 INFO:teuthology.orchestra.run.smithi099.stdout: "user_swift_pool": "a2.rgw.meta:users.swift", 2023-05-31T20:07:48.467 INFO:teuthology.orchestra.run.smithi099.stdout: "user_uid_pool": "a2.rgw.meta:users.uid", 2023-05-31T20:07:48.467 INFO:teuthology.orchestra.run.smithi099.stdout: "otp_pool": "a2.rgw.otp", 2023-05-31T20:07:48.468 INFO:teuthology.orchestra.run.smithi099.stdout: "system_key": { 2023-05-31T20:07:48.468 INFO:teuthology.orchestra.run.smithi099.stdout: "access_key": "RVN19ED3W7FH7E08", 2023-05-31T20:07:48.468 INFO:teuthology.orchestra.run.smithi099.stdout: "secret_key": "XzQjGe60gbIUStGfs96TKhuPw3IQi3hK" 2023-05-31T20:07:48.468 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:07:48.468 INFO:teuthology.orchestra.run.smithi099.stdout: "placement_pools": [ 2023-05-31T20:07:48.468 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T20:07:48.468 INFO:teuthology.orchestra.run.smithi099.stdout: "key": "default-placement", 2023-05-31T20:07:48.469 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T20:07:48.469 INFO:teuthology.orchestra.run.smithi099.stdout: "index_pool": "a2.rgw.buckets.index", 2023-05-31T20:07:48.469 INFO:teuthology.orchestra.run.smithi099.stdout: "storage_classes": { 2023-05-31T20:07:48.469 INFO:teuthology.orchestra.run.smithi099.stdout: "STANDARD": { 2023-05-31T20:07:48.469 INFO:teuthology.orchestra.run.smithi099.stdout: "data_pool": "a2.rgw.buckets.data", 2023-05-31T20:07:48.469 INFO:teuthology.orchestra.run.smithi099.stdout: "compression_type": "random" 2023-05-31T20:07:48.469 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T20:07:48.470 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:07:48.470 INFO:teuthology.orchestra.run.smithi099.stdout: "data_extra_pool": "a2.rgw.buckets.non-ec", 2023-05-31T20:07:48.470 INFO:teuthology.orchestra.run.smithi099.stdout: "index_type": 0, 2023-05-31T20:07:48.470 INFO:teuthology.orchestra.run.smithi099.stdout: "inline_data": true 2023-05-31T20:07:48.470 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T20:07:48.470 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T20:07:48.470 INFO:teuthology.orchestra.run.smithi099.stdout: ], 2023-05-31T20:07:48.471 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_id": "aaa33de6-515b-4cac-8f1b-3a1a30452117", 2023-05-31T20:07:48.471 INFO:teuthology.orchestra.run.smithi099.stdout: "notif_pool": "a2.rgw.log:notif" 2023-05-31T20:07:48.471 INFO:teuthology.orchestra.run.smithi099.stdout:} 2023-05-31T20:07:48.472 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['period', 'update', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-05-31T20:07:48.474 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'period', 'update', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-05-31T20:07:48.474 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 period update --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --commit --cluster c1 --debug-rgw 0 --debug-ms 0 2023-05-31T20:07:48.526 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2023-05-31T20:07:48.526 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2023-05-31T20:07:48.579 INFO:teuthology.orchestra.run.smithi099.stderr:Sending period to new master zone 3d163b1e-09c7-47e2-ae87-add4aace3f38 2023-05-31T20:07:48.618 INFO:teuthology.orchestra.run.smithi099.stdout:{ 2023-05-31T20:07:48.619 INFO:teuthology.orchestra.run.smithi099.stdout: "id": "2e6123d2-d695-4350-9a01-61a455bc782b", 2023-05-31T20:07:48.619 INFO:teuthology.orchestra.run.smithi099.stdout: "epoch": 7, 2023-05-31T20:07:48.619 INFO:teuthology.orchestra.run.smithi099.stdout: "predecessor_uuid": "629b9ced-80af-41ed-8a6c-f2cfc334cbb4", 2023-05-31T20:07:48.619 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_status": [], 2023-05-31T20:07:48.619 INFO:teuthology.orchestra.run.smithi099.stdout: "period_map": { 2023-05-31T20:07:48.619 INFO:teuthology.orchestra.run.smithi099.stdout: "id": "2e6123d2-d695-4350-9a01-61a455bc782b", 2023-05-31T20:07:48.620 INFO:teuthology.orchestra.run.smithi099.stdout: "zonegroups": [ 2023-05-31T20:07:48.620 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T20:07:48.620 INFO:teuthology.orchestra.run.smithi099.stdout: "id": "a", 2023-05-31T20:07:48.620 INFO:teuthology.orchestra.run.smithi099.stdout: "name": "a", 2023-05-31T20:07:48.620 INFO:teuthology.orchestra.run.smithi099.stdout: "api_name": "", 2023-05-31T20:07:48.620 INFO:teuthology.orchestra.run.smithi099.stdout: "is_master": true, 2023-05-31T20:07:48.621 INFO:teuthology.orchestra.run.smithi099.stdout: "endpoints": [ 2023-05-31T20:07:48.621 INFO:teuthology.orchestra.run.smithi099.stdout: "http://smithi099.front.sepia.ceph.com:8000" 2023-05-31T20:07:48.621 INFO:teuthology.orchestra.run.smithi099.stdout: ], 2023-05-31T20:07:48.621 INFO:teuthology.orchestra.run.smithi099.stdout: "hostnames": [], 2023-05-31T20:07:48.621 INFO:teuthology.orchestra.run.smithi099.stdout: "hostnames_s3website": [], 2023-05-31T20:07:48.621 INFO:teuthology.orchestra.run.smithi099.stdout: "master_zone": "3d163b1e-09c7-47e2-ae87-add4aace3f38", 2023-05-31T20:07:48.621 INFO:teuthology.orchestra.run.smithi099.stdout: "zones": [ 2023-05-31T20:07:48.622 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T20:07:48.622 INFO:teuthology.orchestra.run.smithi099.stdout: "id": "3d163b1e-09c7-47e2-ae87-add4aace3f38", 2023-05-31T20:07:48.622 INFO:teuthology.orchestra.run.smithi099.stdout: "name": "a1", 2023-05-31T20:07:48.622 INFO:teuthology.orchestra.run.smithi099.stdout: "endpoints": [ 2023-05-31T20:07:48.622 INFO:teuthology.orchestra.run.smithi099.stdout: "http://smithi099.front.sepia.ceph.com:8000" 2023-05-31T20:07:48.622 INFO:teuthology.orchestra.run.smithi099.stdout: ], 2023-05-31T20:07:48.622 INFO:teuthology.orchestra.run.smithi099.stdout: "log_meta": false, 2023-05-31T20:07:48.623 INFO:teuthology.orchestra.run.smithi099.stdout: "log_data": true, 2023-05-31T20:07:48.623 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket_index_max_shards": 11, 2023-05-31T20:07:48.623 INFO:teuthology.orchestra.run.smithi099.stdout: "read_only": false, 2023-05-31T20:07:48.623 INFO:teuthology.orchestra.run.smithi099.stdout: "tier_type": "", 2023-05-31T20:07:48.623 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_from_all": true, 2023-05-31T20:07:48.623 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_from": [], 2023-05-31T20:07:48.624 INFO:teuthology.orchestra.run.smithi099.stdout: "redirect_zone": "", 2023-05-31T20:07:48.624 INFO:teuthology.orchestra.run.smithi099.stdout: "supported_features": [ 2023-05-31T20:07:48.624 INFO:teuthology.orchestra.run.smithi099.stdout: "resharding" 2023-05-31T20:07:48.624 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2023-05-31T20:07:48.624 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:07:48.624 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T20:07:48.624 INFO:teuthology.orchestra.run.smithi099.stdout: "id": "5d8acec7-7e57-4e56-8b32-e3d6b5036001", 2023-05-31T20:07:48.625 INFO:teuthology.orchestra.run.smithi099.stdout: "name": "a2", 2023-05-31T20:07:48.625 INFO:teuthology.orchestra.run.smithi099.stdout: "endpoints": [ 2023-05-31T20:07:48.625 INFO:teuthology.orchestra.run.smithi099.stdout: "http://smithi099.front.sepia.ceph.com:8001" 2023-05-31T20:07:48.625 INFO:teuthology.orchestra.run.smithi099.stdout: ], 2023-05-31T20:07:48.625 INFO:teuthology.orchestra.run.smithi099.stdout: "log_meta": false, 2023-05-31T20:07:48.625 INFO:teuthology.orchestra.run.smithi099.stdout: "log_data": true, 2023-05-31T20:07:48.626 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket_index_max_shards": 11, 2023-05-31T20:07:48.626 INFO:teuthology.orchestra.run.smithi099.stdout: "read_only": false, 2023-05-31T20:07:48.626 INFO:teuthology.orchestra.run.smithi099.stdout: "tier_type": "", 2023-05-31T20:07:48.627 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_from_all": true, 2023-05-31T20:07:48.627 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_from": [], 2023-05-31T20:07:48.627 INFO:teuthology.orchestra.run.smithi099.stdout: "redirect_zone": "3d163b1e-09c7-47e2-ae87-add4aace3f38", 2023-05-31T20:07:48.627 INFO:teuthology.orchestra.run.smithi099.stdout: "supported_features": [ 2023-05-31T20:07:48.627 INFO:teuthology.orchestra.run.smithi099.stdout: "resharding" 2023-05-31T20:07:48.627 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2023-05-31T20:07:48.628 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T20:07:48.628 INFO:teuthology.orchestra.run.smithi099.stdout: ], 2023-05-31T20:07:48.628 INFO:teuthology.orchestra.run.smithi099.stdout: "placement_targets": [ 2023-05-31T20:07:48.628 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T20:07:48.628 INFO:teuthology.orchestra.run.smithi099.stdout: "name": "default-placement", 2023-05-31T20:07:48.628 INFO:teuthology.orchestra.run.smithi099.stdout: "tags": [], 2023-05-31T20:07:48.628 INFO:teuthology.orchestra.run.smithi099.stdout: "storage_classes": [ 2023-05-31T20:07:48.629 INFO:teuthology.orchestra.run.smithi099.stdout: "STANDARD" 2023-05-31T20:07:48.629 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2023-05-31T20:07:48.629 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T20:07:48.629 INFO:teuthology.orchestra.run.smithi099.stdout: ], 2023-05-31T20:07:48.629 INFO:teuthology.orchestra.run.smithi099.stdout: "default_placement": "default-placement", 2023-05-31T20:07:48.629 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_id": "aaa33de6-515b-4cac-8f1b-3a1a30452117", 2023-05-31T20:07:48.629 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_policy": { 2023-05-31T20:07:48.630 INFO:teuthology.orchestra.run.smithi099.stdout: "groups": [] 2023-05-31T20:07:48.630 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:07:48.630 INFO:teuthology.orchestra.run.smithi099.stdout: "enabled_features": [ 2023-05-31T20:07:48.630 INFO:teuthology.orchestra.run.smithi099.stdout: "resharding" 2023-05-31T20:07:48.630 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2023-05-31T20:07:48.630 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:07:48.630 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T20:07:48.631 INFO:teuthology.orchestra.run.smithi099.stdout: "id": "b", 2023-05-31T20:07:48.631 INFO:teuthology.orchestra.run.smithi099.stdout: "name": "b", 2023-05-31T20:07:48.631 INFO:teuthology.orchestra.run.smithi099.stdout: "api_name": "", 2023-05-31T20:07:48.631 INFO:teuthology.orchestra.run.smithi099.stdout: "is_master": false, 2023-05-31T20:07:48.631 INFO:teuthology.orchestra.run.smithi099.stdout: "endpoints": [ 2023-05-31T20:07:48.631 INFO:teuthology.orchestra.run.smithi099.stdout: "http://smithi112.front.sepia.ceph.com:8000" 2023-05-31T20:07:48.631 INFO:teuthology.orchestra.run.smithi099.stdout: ], 2023-05-31T20:07:48.632 INFO:teuthology.orchestra.run.smithi099.stdout: "hostnames": [], 2023-05-31T20:07:48.632 INFO:teuthology.orchestra.run.smithi099.stdout: "hostnames_s3website": [], 2023-05-31T20:07:48.632 INFO:teuthology.orchestra.run.smithi099.stdout: "master_zone": "f85d7236-2f27-4988-af26-9c66ab280b1b", 2023-05-31T20:07:48.632 INFO:teuthology.orchestra.run.smithi099.stdout: "zones": [ 2023-05-31T20:07:48.632 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T20:07:48.632 INFO:teuthology.orchestra.run.smithi099.stdout: "id": "a6a3aab5-bc52-44a4-8543-5c5ea6056acd", 2023-05-31T20:07:48.633 INFO:teuthology.orchestra.run.smithi099.stdout: "name": "b2", 2023-05-31T20:07:48.633 INFO:teuthology.orchestra.run.smithi099.stdout: "endpoints": [ 2023-05-31T20:07:48.633 INFO:teuthology.orchestra.run.smithi099.stdout: "http://smithi112.front.sepia.ceph.com:8001" 2023-05-31T20:07:48.633 INFO:teuthology.orchestra.run.smithi099.stdout: ], 2023-05-31T20:07:48.633 INFO:teuthology.orchestra.run.smithi099.stdout: "log_meta": false, 2023-05-31T20:07:48.633 INFO:teuthology.orchestra.run.smithi099.stdout: "log_data": true, 2023-05-31T20:07:48.633 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket_index_max_shards": 11, 2023-05-31T20:07:48.634 INFO:teuthology.orchestra.run.smithi099.stdout: "read_only": false, 2023-05-31T20:07:48.634 INFO:teuthology.orchestra.run.smithi099.stdout: "tier_type": "", 2023-05-31T20:07:48.634 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_from_all": true, 2023-05-31T20:07:48.634 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_from": [], 2023-05-31T20:07:48.634 INFO:teuthology.orchestra.run.smithi099.stdout: "redirect_zone": "", 2023-05-31T20:07:48.636 INFO:teuthology.orchestra.run.smithi099.stdout: "supported_features": [ 2023-05-31T20:07:48.636 INFO:teuthology.orchestra.run.smithi099.stdout: "resharding" 2023-05-31T20:07:48.637 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2023-05-31T20:07:48.637 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:07:48.637 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T20:07:48.638 INFO:teuthology.orchestra.run.smithi099.stdout: "id": "f85d7236-2f27-4988-af26-9c66ab280b1b", 2023-05-31T20:07:48.638 INFO:teuthology.orchestra.run.smithi099.stdout: "name": "b1", 2023-05-31T20:07:48.638 INFO:teuthology.orchestra.run.smithi099.stdout: "endpoints": [ 2023-05-31T20:07:48.638 INFO:teuthology.orchestra.run.smithi099.stdout: "http://smithi112.front.sepia.ceph.com:8000" 2023-05-31T20:07:48.639 INFO:teuthology.orchestra.run.smithi099.stdout: ], 2023-05-31T20:07:48.639 INFO:teuthology.orchestra.run.smithi099.stdout: "log_meta": false, 2023-05-31T20:07:48.639 INFO:teuthology.orchestra.run.smithi099.stdout: "log_data": true, 2023-05-31T20:07:48.639 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket_index_max_shards": 11, 2023-05-31T20:07:48.639 INFO:teuthology.orchestra.run.smithi099.stdout: "read_only": false, 2023-05-31T20:07:48.639 INFO:teuthology.orchestra.run.smithi099.stdout: "tier_type": "", 2023-05-31T20:07:48.640 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_from_all": true, 2023-05-31T20:07:48.640 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_from": [], 2023-05-31T20:07:48.640 INFO:teuthology.orchestra.run.smithi099.stdout: "redirect_zone": "", 2023-05-31T20:07:48.640 INFO:teuthology.orchestra.run.smithi099.stdout: "supported_features": [ 2023-05-31T20:07:48.641 INFO:teuthology.orchestra.run.smithi099.stdout: "resharding" 2023-05-31T20:07:48.641 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2023-05-31T20:07:48.641 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T20:07:48.641 INFO:teuthology.orchestra.run.smithi099.stdout: ], 2023-05-31T20:07:48.641 INFO:teuthology.orchestra.run.smithi099.stdout: "placement_targets": [ 2023-05-31T20:07:48.641 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T20:07:48.642 INFO:teuthology.orchestra.run.smithi099.stdout: "name": "default-placement", 2023-05-31T20:07:48.642 INFO:teuthology.orchestra.run.smithi099.stdout: "tags": [], 2023-05-31T20:07:48.642 INFO:teuthology.orchestra.run.smithi099.stdout: "storage_classes": [ 2023-05-31T20:07:48.642 INFO:teuthology.orchestra.run.smithi099.stdout: "STANDARD" 2023-05-31T20:07:48.642 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2023-05-31T20:07:48.642 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T20:07:48.642 INFO:teuthology.orchestra.run.smithi099.stdout: ], 2023-05-31T20:07:48.643 INFO:teuthology.orchestra.run.smithi099.stdout: "default_placement": "default-placement", 2023-05-31T20:07:48.643 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_id": "aaa33de6-515b-4cac-8f1b-3a1a30452117", 2023-05-31T20:07:48.643 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_policy": { 2023-05-31T20:07:48.643 INFO:teuthology.orchestra.run.smithi099.stdout: "groups": [] 2023-05-31T20:07:48.643 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:07:48.643 INFO:teuthology.orchestra.run.smithi099.stdout: "enabled_features": [ 2023-05-31T20:07:48.643 INFO:teuthology.orchestra.run.smithi099.stdout: "resharding" 2023-05-31T20:07:48.644 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2023-05-31T20:07:48.644 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T20:07:48.644 INFO:teuthology.orchestra.run.smithi099.stdout: ], 2023-05-31T20:07:48.644 INFO:teuthology.orchestra.run.smithi099.stdout: "short_zone_ids": [ 2023-05-31T20:07:48.645 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T20:07:48.645 INFO:teuthology.orchestra.run.smithi099.stdout: "key": "3d163b1e-09c7-47e2-ae87-add4aace3f38", 2023-05-31T20:07:48.645 INFO:teuthology.orchestra.run.smithi099.stdout: "val": 3101833771 2023-05-31T20:07:48.645 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:07:48.645 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T20:07:48.646 INFO:teuthology.orchestra.run.smithi099.stdout: "key": "5d8acec7-7e57-4e56-8b32-e3d6b5036001", 2023-05-31T20:07:48.646 INFO:teuthology.orchestra.run.smithi099.stdout: "val": 4255530623 2023-05-31T20:07:48.647 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:07:48.647 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T20:07:48.647 INFO:teuthology.orchestra.run.smithi099.stdout: "key": "a6a3aab5-bc52-44a4-8543-5c5ea6056acd", 2023-05-31T20:07:48.647 INFO:teuthology.orchestra.run.smithi099.stdout: "val": 3708073710 2023-05-31T20:07:48.648 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:07:48.648 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T20:07:48.649 INFO:teuthology.orchestra.run.smithi099.stdout: "key": "f85d7236-2f27-4988-af26-9c66ab280b1b", 2023-05-31T20:07:48.649 INFO:teuthology.orchestra.run.smithi099.stdout: "val": 233395200 2023-05-31T20:07:48.649 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T20:07:48.649 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2023-05-31T20:07:48.650 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:07:48.650 INFO:teuthology.orchestra.run.smithi099.stdout: "master_zonegroup": "a", 2023-05-31T20:07:48.650 INFO:teuthology.orchestra.run.smithi099.stdout: "master_zone": "3d163b1e-09c7-47e2-ae87-add4aace3f38", 2023-05-31T20:07:48.650 INFO:teuthology.orchestra.run.smithi099.stdout: "period_config": { 2023-05-31T20:07:48.651 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket_quota": { 2023-05-31T20:07:48.651 INFO:teuthology.orchestra.run.smithi099.stdout: "enabled": false, 2023-05-31T20:07:48.651 INFO:teuthology.orchestra.run.smithi099.stdout: "check_on_raw": false, 2023-05-31T20:07:48.651 INFO:teuthology.orchestra.run.smithi099.stdout: "max_size": -1, 2023-05-31T20:07:48.652 INFO:teuthology.orchestra.run.smithi099.stdout: "max_size_kb": 0, 2023-05-31T20:07:48.652 INFO:teuthology.orchestra.run.smithi099.stdout: "max_objects": -1 2023-05-31T20:07:48.652 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:07:48.653 INFO:teuthology.orchestra.run.smithi099.stdout: "user_quota": { 2023-05-31T20:07:48.653 INFO:teuthology.orchestra.run.smithi099.stdout: "enabled": false, 2023-05-31T20:07:48.653 INFO:teuthology.orchestra.run.smithi099.stdout: "check_on_raw": false, 2023-05-31T20:07:48.653 INFO:teuthology.orchestra.run.smithi099.stdout: "max_size": -1, 2023-05-31T20:07:48.653 INFO:teuthology.orchestra.run.smithi099.stdout: "max_size_kb": 0, 2023-05-31T20:07:48.654 INFO:teuthology.orchestra.run.smithi099.stdout: "max_objects": -1 2023-05-31T20:07:48.654 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:07:48.654 INFO:teuthology.orchestra.run.smithi099.stdout: "user_ratelimit": { 2023-05-31T20:07:48.654 INFO:teuthology.orchestra.run.smithi099.stdout: "max_read_ops": 0, 2023-05-31T20:07:48.654 INFO:teuthology.orchestra.run.smithi099.stdout: "max_write_ops": 0, 2023-05-31T20:07:48.654 INFO:teuthology.orchestra.run.smithi099.stdout: "max_read_bytes": 0, 2023-05-31T20:07:48.655 INFO:teuthology.orchestra.run.smithi099.stdout: "max_write_bytes": 0, 2023-05-31T20:07:48.655 INFO:teuthology.orchestra.run.smithi099.stdout: "enabled": false 2023-05-31T20:07:48.655 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:07:48.655 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket_ratelimit": { 2023-05-31T20:07:48.656 INFO:teuthology.orchestra.run.smithi099.stdout: "max_read_ops": 0, 2023-05-31T20:07:48.656 INFO:teuthology.orchestra.run.smithi099.stdout: "max_write_ops": 0, 2023-05-31T20:07:48.656 INFO:teuthology.orchestra.run.smithi099.stdout: "max_read_bytes": 0, 2023-05-31T20:07:48.657 INFO:teuthology.orchestra.run.smithi099.stdout: "max_write_bytes": 0, 2023-05-31T20:07:48.657 INFO:teuthology.orchestra.run.smithi099.stdout: "enabled": false 2023-05-31T20:07:48.657 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:07:48.657 INFO:teuthology.orchestra.run.smithi099.stdout: "anonymous_ratelimit": { 2023-05-31T20:07:48.658 INFO:teuthology.orchestra.run.smithi099.stdout: "max_read_ops": 0, 2023-05-31T20:07:48.658 INFO:teuthology.orchestra.run.smithi099.stdout: "max_write_ops": 0, 2023-05-31T20:07:48.658 INFO:teuthology.orchestra.run.smithi099.stdout: "max_read_bytes": 0, 2023-05-31T20:07:48.659 INFO:teuthology.orchestra.run.smithi099.stdout: "max_write_bytes": 0, 2023-05-31T20:07:48.659 INFO:teuthology.orchestra.run.smithi099.stdout: "enabled": false 2023-05-31T20:07:48.659 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T20:07:48.660 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:07:48.660 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_id": "aaa33de6-515b-4cac-8f1b-3a1a30452117", 2023-05-31T20:07:48.660 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_name": "test-realm", 2023-05-31T20:07:48.661 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2023-05-31T20:07:48.661 INFO:teuthology.orchestra.run.smithi099.stdout:} 2023-05-31T20:07:48.662 INFO:rgw_multi.tests:Set sync_from_all flag on zone a2 to true 2023-05-31T20:08:48.663 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['zone', 'modify', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--redirect-zone=', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-05-31T20:08:48.664 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'zone', 'modify', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--redirect-zone=', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-05-31T20:08:48.665 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 zone modify --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --redirect-zone= --cluster c1 --debug-rgw 0 --debug-ms 0 2023-05-31T20:08:48.722 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2023-05-31T20:08:48.722 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2023-05-31T20:08:48.744 INFO:teuthology.orchestra.run.smithi099.stdout:{ 2023-05-31T20:08:48.744 INFO:teuthology.orchestra.run.smithi099.stdout: "id": "5d8acec7-7e57-4e56-8b32-e3d6b5036001", 2023-05-31T20:08:48.745 INFO:teuthology.orchestra.run.smithi099.stdout: "name": "a2", 2023-05-31T20:08:48.745 INFO:teuthology.orchestra.run.smithi099.stdout: "domain_root": "a2.rgw.meta:root", 2023-05-31T20:08:48.745 INFO:teuthology.orchestra.run.smithi099.stdout: "control_pool": "a2.rgw.control", 2023-05-31T20:08:48.745 INFO:teuthology.orchestra.run.smithi099.stdout: "gc_pool": "a2.rgw.log:gc", 2023-05-31T20:08:48.745 INFO:teuthology.orchestra.run.smithi099.stdout: "lc_pool": "a2.rgw.log:lc", 2023-05-31T20:08:48.745 INFO:teuthology.orchestra.run.smithi099.stdout: "log_pool": "a2.rgw.log", 2023-05-31T20:08:48.746 INFO:teuthology.orchestra.run.smithi099.stdout: "intent_log_pool": "a2.rgw.log:intent", 2023-05-31T20:08:48.746 INFO:teuthology.orchestra.run.smithi099.stdout: "usage_log_pool": "a2.rgw.log:usage", 2023-05-31T20:08:48.746 INFO:teuthology.orchestra.run.smithi099.stdout: "roles_pool": "a2.rgw.meta:roles", 2023-05-31T20:08:48.746 INFO:teuthology.orchestra.run.smithi099.stdout: "reshard_pool": "a2.rgw.log:reshard", 2023-05-31T20:08:48.746 INFO:teuthology.orchestra.run.smithi099.stdout: "user_keys_pool": "a2.rgw.meta:users.keys", 2023-05-31T20:08:48.746 INFO:teuthology.orchestra.run.smithi099.stdout: "user_email_pool": "a2.rgw.meta:users.email", 2023-05-31T20:08:48.746 INFO:teuthology.orchestra.run.smithi099.stdout: "user_swift_pool": "a2.rgw.meta:users.swift", 2023-05-31T20:08:48.747 INFO:teuthology.orchestra.run.smithi099.stdout: "user_uid_pool": "a2.rgw.meta:users.uid", 2023-05-31T20:08:48.747 INFO:teuthology.orchestra.run.smithi099.stdout: "otp_pool": "a2.rgw.otp", 2023-05-31T20:08:48.747 INFO:teuthology.orchestra.run.smithi099.stdout: "system_key": { 2023-05-31T20:08:48.747 INFO:teuthology.orchestra.run.smithi099.stdout: "access_key": "RVN19ED3W7FH7E08", 2023-05-31T20:08:48.747 INFO:teuthology.orchestra.run.smithi099.stdout: "secret_key": "XzQjGe60gbIUStGfs96TKhuPw3IQi3hK" 2023-05-31T20:08:48.747 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:08:48.747 INFO:teuthology.orchestra.run.smithi099.stdout: "placement_pools": [ 2023-05-31T20:08:48.748 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T20:08:48.748 INFO:teuthology.orchestra.run.smithi099.stdout: "key": "default-placement", 2023-05-31T20:08:48.748 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T20:08:48.748 INFO:teuthology.orchestra.run.smithi099.stdout: "index_pool": "a2.rgw.buckets.index", 2023-05-31T20:08:48.748 INFO:teuthology.orchestra.run.smithi099.stdout: "storage_classes": { 2023-05-31T20:08:48.748 INFO:teuthology.orchestra.run.smithi099.stdout: "STANDARD": { 2023-05-31T20:08:48.749 INFO:teuthology.orchestra.run.smithi099.stdout: "data_pool": "a2.rgw.buckets.data", 2023-05-31T20:08:48.749 INFO:teuthology.orchestra.run.smithi099.stdout: "compression_type": "random" 2023-05-31T20:08:48.749 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T20:08:48.749 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:08:48.749 INFO:teuthology.orchestra.run.smithi099.stdout: "data_extra_pool": "a2.rgw.buckets.non-ec", 2023-05-31T20:08:48.749 INFO:teuthology.orchestra.run.smithi099.stdout: "index_type": 0, 2023-05-31T20:08:48.750 INFO:teuthology.orchestra.run.smithi099.stdout: "inline_data": true 2023-05-31T20:08:48.750 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T20:08:48.750 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T20:08:48.750 INFO:teuthology.orchestra.run.smithi099.stdout: ], 2023-05-31T20:08:48.750 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_id": "aaa33de6-515b-4cac-8f1b-3a1a30452117", 2023-05-31T20:08:48.750 INFO:teuthology.orchestra.run.smithi099.stdout: "notif_pool": "a2.rgw.log:notif" 2023-05-31T20:08:48.750 INFO:teuthology.orchestra.run.smithi099.stdout:} 2023-05-31T20:08:48.751 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['period', 'update', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-05-31T20:08:48.752 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'period', 'update', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-05-31T20:08:48.753 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 period update --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --commit --cluster c1 --debug-rgw 0 --debug-ms 0 2023-05-31T20:08:48.804 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2023-05-31T20:08:48.805 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2023-05-31T20:08:48.856 INFO:teuthology.orchestra.run.smithi099.stderr:Sending period to new master zone 3d163b1e-09c7-47e2-ae87-add4aace3f38 2023-05-31T20:08:48.885 INFO:teuthology.orchestra.run.smithi099.stdout:{ 2023-05-31T20:08:48.886 INFO:teuthology.orchestra.run.smithi099.stdout: "id": "2e6123d2-d695-4350-9a01-61a455bc782b", 2023-05-31T20:08:48.886 INFO:teuthology.orchestra.run.smithi099.stdout: "epoch": 8, 2023-05-31T20:08:48.886 INFO:teuthology.orchestra.run.smithi099.stdout: "predecessor_uuid": "629b9ced-80af-41ed-8a6c-f2cfc334cbb4", 2023-05-31T20:08:48.886 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_status": [], 2023-05-31T20:08:48.886 INFO:teuthology.orchestra.run.smithi099.stdout: "period_map": { 2023-05-31T20:08:48.887 INFO:teuthology.orchestra.run.smithi099.stdout: "id": "2e6123d2-d695-4350-9a01-61a455bc782b", 2023-05-31T20:08:48.887 INFO:teuthology.orchestra.run.smithi099.stdout: "zonegroups": [ 2023-05-31T20:08:48.888 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T20:08:48.888 INFO:teuthology.orchestra.run.smithi099.stdout: "id": "a", 2023-05-31T20:08:48.888 INFO:teuthology.orchestra.run.smithi099.stdout: "name": "a", 2023-05-31T20:08:48.888 INFO:teuthology.orchestra.run.smithi099.stdout: "api_name": "", 2023-05-31T20:08:48.889 INFO:teuthology.orchestra.run.smithi099.stdout: "is_master": true, 2023-05-31T20:08:48.889 INFO:teuthology.orchestra.run.smithi099.stdout: "endpoints": [ 2023-05-31T20:08:48.889 INFO:teuthology.orchestra.run.smithi099.stdout: "http://smithi099.front.sepia.ceph.com:8000" 2023-05-31T20:08:48.889 INFO:teuthology.orchestra.run.smithi099.stdout: ], 2023-05-31T20:08:48.889 INFO:teuthology.orchestra.run.smithi099.stdout: "hostnames": [], 2023-05-31T20:08:48.890 INFO:teuthology.orchestra.run.smithi099.stdout: "hostnames_s3website": [], 2023-05-31T20:08:48.890 INFO:teuthology.orchestra.run.smithi099.stdout: "master_zone": "3d163b1e-09c7-47e2-ae87-add4aace3f38", 2023-05-31T20:08:48.890 INFO:teuthology.orchestra.run.smithi099.stdout: "zones": [ 2023-05-31T20:08:48.890 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T20:08:48.890 INFO:teuthology.orchestra.run.smithi099.stdout: "id": "3d163b1e-09c7-47e2-ae87-add4aace3f38", 2023-05-31T20:08:48.891 INFO:teuthology.orchestra.run.smithi099.stdout: "name": "a1", 2023-05-31T20:08:48.891 INFO:teuthology.orchestra.run.smithi099.stdout: "endpoints": [ 2023-05-31T20:08:48.891 INFO:teuthology.orchestra.run.smithi099.stdout: "http://smithi099.front.sepia.ceph.com:8000" 2023-05-31T20:08:48.891 INFO:teuthology.orchestra.run.smithi099.stdout: ], 2023-05-31T20:08:48.891 INFO:teuthology.orchestra.run.smithi099.stdout: "log_meta": false, 2023-05-31T20:08:48.891 INFO:teuthology.orchestra.run.smithi099.stdout: "log_data": true, 2023-05-31T20:08:48.892 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket_index_max_shards": 11, 2023-05-31T20:08:48.892 INFO:teuthology.orchestra.run.smithi099.stdout: "read_only": false, 2023-05-31T20:08:48.892 INFO:teuthology.orchestra.run.smithi099.stdout: "tier_type": "", 2023-05-31T20:08:48.892 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_from_all": true, 2023-05-31T20:08:48.892 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_from": [], 2023-05-31T20:08:48.892 INFO:teuthology.orchestra.run.smithi099.stdout: "redirect_zone": "", 2023-05-31T20:08:48.892 INFO:teuthology.orchestra.run.smithi099.stdout: "supported_features": [ 2023-05-31T20:08:48.893 INFO:teuthology.orchestra.run.smithi099.stdout: "resharding" 2023-05-31T20:08:48.893 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2023-05-31T20:08:48.893 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:08:48.893 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T20:08:48.893 INFO:teuthology.orchestra.run.smithi099.stdout: "id": "5d8acec7-7e57-4e56-8b32-e3d6b5036001", 2023-05-31T20:08:48.893 INFO:teuthology.orchestra.run.smithi099.stdout: "name": "a2", 2023-05-31T20:08:48.893 INFO:teuthology.orchestra.run.smithi099.stdout: "endpoints": [ 2023-05-31T20:08:48.894 INFO:teuthology.orchestra.run.smithi099.stdout: "http://smithi099.front.sepia.ceph.com:8001" 2023-05-31T20:08:48.894 INFO:teuthology.orchestra.run.smithi099.stdout: ], 2023-05-31T20:08:48.894 INFO:teuthology.orchestra.run.smithi099.stdout: "log_meta": false, 2023-05-31T20:08:48.894 INFO:teuthology.orchestra.run.smithi099.stdout: "log_data": true, 2023-05-31T20:08:48.894 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket_index_max_shards": 11, 2023-05-31T20:08:48.894 INFO:teuthology.orchestra.run.smithi099.stdout: "read_only": false, 2023-05-31T20:08:48.894 INFO:teuthology.orchestra.run.smithi099.stdout: "tier_type": "", 2023-05-31T20:08:48.895 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_from_all": true, 2023-05-31T20:08:48.895 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_from": [], 2023-05-31T20:08:48.895 INFO:teuthology.orchestra.run.smithi099.stdout: "redirect_zone": "", 2023-05-31T20:08:48.895 INFO:teuthology.orchestra.run.smithi099.stdout: "supported_features": [ 2023-05-31T20:08:48.895 INFO:teuthology.orchestra.run.smithi099.stdout: "resharding" 2023-05-31T20:08:48.895 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2023-05-31T20:08:48.895 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T20:08:48.896 INFO:teuthology.orchestra.run.smithi099.stdout: ], 2023-05-31T20:08:48.896 INFO:teuthology.orchestra.run.smithi099.stdout: "placement_targets": [ 2023-05-31T20:08:48.896 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T20:08:48.896 INFO:teuthology.orchestra.run.smithi099.stdout: "name": "default-placement", 2023-05-31T20:08:48.896 INFO:teuthology.orchestra.run.smithi099.stdout: "tags": [], 2023-05-31T20:08:48.896 INFO:teuthology.orchestra.run.smithi099.stdout: "storage_classes": [ 2023-05-31T20:08:48.896 INFO:teuthology.orchestra.run.smithi099.stdout: "STANDARD" 2023-05-31T20:08:48.897 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2023-05-31T20:08:48.897 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T20:08:48.897 INFO:teuthology.orchestra.run.smithi099.stdout: ], 2023-05-31T20:08:48.897 INFO:teuthology.orchestra.run.smithi099.stdout: "default_placement": "default-placement", 2023-05-31T20:08:48.897 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_id": "aaa33de6-515b-4cac-8f1b-3a1a30452117", 2023-05-31T20:08:48.897 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_policy": { 2023-05-31T20:08:48.897 INFO:teuthology.orchestra.run.smithi099.stdout: "groups": [] 2023-05-31T20:08:48.898 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:08:48.898 INFO:teuthology.orchestra.run.smithi099.stdout: "enabled_features": [ 2023-05-31T20:08:48.898 INFO:teuthology.orchestra.run.smithi099.stdout: "resharding" 2023-05-31T20:08:48.898 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2023-05-31T20:08:48.898 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:08:48.898 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T20:08:48.899 INFO:teuthology.orchestra.run.smithi099.stdout: "id": "b", 2023-05-31T20:08:48.899 INFO:teuthology.orchestra.run.smithi099.stdout: "name": "b", 2023-05-31T20:08:48.899 INFO:teuthology.orchestra.run.smithi099.stdout: "api_name": "", 2023-05-31T20:08:48.899 INFO:teuthology.orchestra.run.smithi099.stdout: "is_master": false, 2023-05-31T20:08:48.899 INFO:teuthology.orchestra.run.smithi099.stdout: "endpoints": [ 2023-05-31T20:08:48.899 INFO:teuthology.orchestra.run.smithi099.stdout: "http://smithi112.front.sepia.ceph.com:8000" 2023-05-31T20:08:48.899 INFO:teuthology.orchestra.run.smithi099.stdout: ], 2023-05-31T20:08:48.900 INFO:teuthology.orchestra.run.smithi099.stdout: "hostnames": [], 2023-05-31T20:08:48.900 INFO:teuthology.orchestra.run.smithi099.stdout: "hostnames_s3website": [], 2023-05-31T20:08:48.900 INFO:teuthology.orchestra.run.smithi099.stdout: "master_zone": "f85d7236-2f27-4988-af26-9c66ab280b1b", 2023-05-31T20:08:48.900 INFO:teuthology.orchestra.run.smithi099.stdout: "zones": [ 2023-05-31T20:08:48.900 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T20:08:48.900 INFO:teuthology.orchestra.run.smithi099.stdout: "id": "a6a3aab5-bc52-44a4-8543-5c5ea6056acd", 2023-05-31T20:08:48.900 INFO:teuthology.orchestra.run.smithi099.stdout: "name": "b2", 2023-05-31T20:08:48.901 INFO:teuthology.orchestra.run.smithi099.stdout: "endpoints": [ 2023-05-31T20:08:48.901 INFO:teuthology.orchestra.run.smithi099.stdout: "http://smithi112.front.sepia.ceph.com:8001" 2023-05-31T20:08:48.901 INFO:teuthology.orchestra.run.smithi099.stdout: ], 2023-05-31T20:08:48.901 INFO:teuthology.orchestra.run.smithi099.stdout: "log_meta": false, 2023-05-31T20:08:48.901 INFO:teuthology.orchestra.run.smithi099.stdout: "log_data": true, 2023-05-31T20:08:48.901 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket_index_max_shards": 11, 2023-05-31T20:08:48.901 INFO:teuthology.orchestra.run.smithi099.stdout: "read_only": false, 2023-05-31T20:08:48.902 INFO:teuthology.orchestra.run.smithi099.stdout: "tier_type": "", 2023-05-31T20:08:48.902 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_from_all": true, 2023-05-31T20:08:48.902 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_from": [], 2023-05-31T20:08:48.902 INFO:teuthology.orchestra.run.smithi099.stdout: "redirect_zone": "", 2023-05-31T20:08:48.902 INFO:teuthology.orchestra.run.smithi099.stdout: "supported_features": [ 2023-05-31T20:08:48.902 INFO:teuthology.orchestra.run.smithi099.stdout: "resharding" 2023-05-31T20:08:48.902 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2023-05-31T20:08:48.903 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:08:48.903 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T20:08:48.904 INFO:teuthology.orchestra.run.smithi099.stdout: "id": "f85d7236-2f27-4988-af26-9c66ab280b1b", 2023-05-31T20:08:48.904 INFO:teuthology.orchestra.run.smithi099.stdout: "name": "b1", 2023-05-31T20:08:48.904 INFO:teuthology.orchestra.run.smithi099.stdout: "endpoints": [ 2023-05-31T20:08:48.904 INFO:teuthology.orchestra.run.smithi099.stdout: "http://smithi112.front.sepia.ceph.com:8000" 2023-05-31T20:08:48.905 INFO:teuthology.orchestra.run.smithi099.stdout: ], 2023-05-31T20:08:48.905 INFO:teuthology.orchestra.run.smithi099.stdout: "log_meta": false, 2023-05-31T20:08:48.905 INFO:teuthology.orchestra.run.smithi099.stdout: "log_data": true, 2023-05-31T20:08:48.905 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket_index_max_shards": 11, 2023-05-31T20:08:48.905 INFO:teuthology.orchestra.run.smithi099.stdout: "read_only": false, 2023-05-31T20:08:48.905 INFO:teuthology.orchestra.run.smithi099.stdout: "tier_type": "", 2023-05-31T20:08:48.905 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_from_all": true, 2023-05-31T20:08:48.906 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_from": [], 2023-05-31T20:08:48.906 INFO:teuthology.orchestra.run.smithi099.stdout: "redirect_zone": "", 2023-05-31T20:08:48.906 INFO:teuthology.orchestra.run.smithi099.stdout: "supported_features": [ 2023-05-31T20:08:48.906 INFO:teuthology.orchestra.run.smithi099.stdout: "resharding" 2023-05-31T20:08:48.906 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2023-05-31T20:08:48.906 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T20:08:48.907 INFO:teuthology.orchestra.run.smithi099.stdout: ], 2023-05-31T20:08:48.907 INFO:teuthology.orchestra.run.smithi099.stdout: "placement_targets": [ 2023-05-31T20:08:48.907 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T20:08:48.907 INFO:teuthology.orchestra.run.smithi099.stdout: "name": "default-placement", 2023-05-31T20:08:48.907 INFO:teuthology.orchestra.run.smithi099.stdout: "tags": [], 2023-05-31T20:08:48.908 INFO:teuthology.orchestra.run.smithi099.stdout: "storage_classes": [ 2023-05-31T20:08:48.908 INFO:teuthology.orchestra.run.smithi099.stdout: "STANDARD" 2023-05-31T20:08:48.908 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2023-05-31T20:08:48.908 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T20:08:48.908 INFO:teuthology.orchestra.run.smithi099.stdout: ], 2023-05-31T20:08:48.908 INFO:teuthology.orchestra.run.smithi099.stdout: "default_placement": "default-placement", 2023-05-31T20:08:48.908 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_id": "aaa33de6-515b-4cac-8f1b-3a1a30452117", 2023-05-31T20:08:48.909 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_policy": { 2023-05-31T20:08:48.909 INFO:teuthology.orchestra.run.smithi099.stdout: "groups": [] 2023-05-31T20:08:48.909 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:08:48.909 INFO:teuthology.orchestra.run.smithi099.stdout: "enabled_features": [ 2023-05-31T20:08:48.909 INFO:teuthology.orchestra.run.smithi099.stdout: "resharding" 2023-05-31T20:08:48.909 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2023-05-31T20:08:48.909 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T20:08:48.910 INFO:teuthology.orchestra.run.smithi099.stdout: ], 2023-05-31T20:08:48.910 INFO:teuthology.orchestra.run.smithi099.stdout: "short_zone_ids": [ 2023-05-31T20:08:48.910 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T20:08:48.910 INFO:teuthology.orchestra.run.smithi099.stdout: "key": "3d163b1e-09c7-47e2-ae87-add4aace3f38", 2023-05-31T20:08:48.910 INFO:teuthology.orchestra.run.smithi099.stdout: "val": 3101833771 2023-05-31T20:08:48.910 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:08:48.910 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T20:08:48.911 INFO:teuthology.orchestra.run.smithi099.stdout: "key": "5d8acec7-7e57-4e56-8b32-e3d6b5036001", 2023-05-31T20:08:48.911 INFO:teuthology.orchestra.run.smithi099.stdout: "val": 4255530623 2023-05-31T20:08:48.911 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:08:48.911 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T20:08:48.911 INFO:teuthology.orchestra.run.smithi099.stdout: "key": "a6a3aab5-bc52-44a4-8543-5c5ea6056acd", 2023-05-31T20:08:48.911 INFO:teuthology.orchestra.run.smithi099.stdout: "val": 3708073710 2023-05-31T20:08:48.912 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:08:48.912 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T20:08:48.912 INFO:teuthology.orchestra.run.smithi099.stdout: "key": "f85d7236-2f27-4988-af26-9c66ab280b1b", 2023-05-31T20:08:48.912 INFO:teuthology.orchestra.run.smithi099.stdout: "val": 233395200 2023-05-31T20:08:48.912 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T20:08:48.912 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2023-05-31T20:08:48.912 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:08:48.913 INFO:teuthology.orchestra.run.smithi099.stdout: "master_zonegroup": "a", 2023-05-31T20:08:48.913 INFO:teuthology.orchestra.run.smithi099.stdout: "master_zone": "3d163b1e-09c7-47e2-ae87-add4aace3f38", 2023-05-31T20:08:48.913 INFO:teuthology.orchestra.run.smithi099.stdout: "period_config": { 2023-05-31T20:08:48.913 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket_quota": { 2023-05-31T20:08:48.913 INFO:teuthology.orchestra.run.smithi099.stdout: "enabled": false, 2023-05-31T20:08:48.913 INFO:teuthology.orchestra.run.smithi099.stdout: "check_on_raw": false, 2023-05-31T20:08:48.913 INFO:teuthology.orchestra.run.smithi099.stdout: "max_size": -1, 2023-05-31T20:08:48.914 INFO:teuthology.orchestra.run.smithi099.stdout: "max_size_kb": 0, 2023-05-31T20:08:48.914 INFO:teuthology.orchestra.run.smithi099.stdout: "max_objects": -1 2023-05-31T20:08:48.914 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:08:48.914 INFO:teuthology.orchestra.run.smithi099.stdout: "user_quota": { 2023-05-31T20:08:48.914 INFO:teuthology.orchestra.run.smithi099.stdout: "enabled": false, 2023-05-31T20:08:48.914 INFO:teuthology.orchestra.run.smithi099.stdout: "check_on_raw": false, 2023-05-31T20:08:48.914 INFO:teuthology.orchestra.run.smithi099.stdout: "max_size": -1, 2023-05-31T20:08:48.915 INFO:teuthology.orchestra.run.smithi099.stdout: "max_size_kb": 0, 2023-05-31T20:08:48.915 INFO:teuthology.orchestra.run.smithi099.stdout: "max_objects": -1 2023-05-31T20:08:48.915 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:08:48.915 INFO:teuthology.orchestra.run.smithi099.stdout: "user_ratelimit": { 2023-05-31T20:08:48.915 INFO:teuthology.orchestra.run.smithi099.stdout: "max_read_ops": 0, 2023-05-31T20:08:48.915 INFO:teuthology.orchestra.run.smithi099.stdout: "max_write_ops": 0, 2023-05-31T20:08:48.915 INFO:teuthology.orchestra.run.smithi099.stdout: "max_read_bytes": 0, 2023-05-31T20:08:48.916 INFO:teuthology.orchestra.run.smithi099.stdout: "max_write_bytes": 0, 2023-05-31T20:08:48.916 INFO:teuthology.orchestra.run.smithi099.stdout: "enabled": false 2023-05-31T20:08:48.916 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:08:48.916 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket_ratelimit": { 2023-05-31T20:08:48.916 INFO:teuthology.orchestra.run.smithi099.stdout: "max_read_ops": 0, 2023-05-31T20:08:48.916 INFO:teuthology.orchestra.run.smithi099.stdout: "max_write_ops": 0, 2023-05-31T20:08:48.916 INFO:teuthology.orchestra.run.smithi099.stdout: "max_read_bytes": 0, 2023-05-31T20:08:48.917 INFO:teuthology.orchestra.run.smithi099.stdout: "max_write_bytes": 0, 2023-05-31T20:08:48.917 INFO:teuthology.orchestra.run.smithi099.stdout: "enabled": false 2023-05-31T20:08:48.917 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:08:48.917 INFO:teuthology.orchestra.run.smithi099.stdout: "anonymous_ratelimit": { 2023-05-31T20:08:48.917 INFO:teuthology.orchestra.run.smithi099.stdout: "max_read_ops": 0, 2023-05-31T20:08:48.917 INFO:teuthology.orchestra.run.smithi099.stdout: "max_write_ops": 0, 2023-05-31T20:08:48.917 INFO:teuthology.orchestra.run.smithi099.stdout: "max_read_bytes": 0, 2023-05-31T20:08:48.918 INFO:teuthology.orchestra.run.smithi099.stdout: "max_write_bytes": 0, 2023-05-31T20:08:48.918 INFO:teuthology.orchestra.run.smithi099.stdout: "enabled": false 2023-05-31T20:08:48.918 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T20:08:48.918 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:08:48.918 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_id": "aaa33de6-515b-4cac-8f1b-3a1a30452117", 2023-05-31T20:08:48.918 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_name": "test-realm", 2023-05-31T20:08:48.918 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2023-05-31T20:08:48.919 INFO:teuthology.orchestra.run.smithi099.stdout:} 2023-05-31T20:08:48.919 INFO:rgw_multi.tests:Set redirect_zone zone a2 to "" 2023-05-31T20:09:48.921 INFO:tasks.rgw_multisite_tests:rgw_multi.tests.test_multi_zone_redirect ... ok 2023-05-31T20:09:48.923 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-31T20:09:48.924 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-31T20:09:48.925 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-05-31T20:09:48.984 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2023-05-31T20:09:48.985 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2023-05-31T20:09:49.020 INFO:teuthology.orchestra.run.smithi099.stderr:No --period given, using current period=2e6123d2-d695-4350-9a01-61a455bc782b 2023-05-31T20:09:49.042 INFO:teuthology.orchestra.run.smithi099.stdout:[ 2023-05-31T20:09:49.042 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T20:09:49.042 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1685562631.516089_59.1", 2023-05-31T20:09:49.043 INFO:teuthology.orchestra.run.smithi099.stdout: "last_update": "2023-05-31T19:50:31.516089Z" 2023-05-31T20:09:49.043 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:09:49.043 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T20:09:49.043 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "", 2023-05-31T20:09:49.043 INFO:teuthology.orchestra.run.smithi099.stdout: "last_update": "0.000000" 2023-05-31T20:09:49.043 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:09:49.044 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T20:09:49.044 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1685562970.274572_93.1", 2023-05-31T20:09:49.044 INFO:teuthology.orchestra.run.smithi099.stdout: "last_update": "2023-05-31T19:56:10.274572Z" 2023-05-31T20:09:49.044 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:09:49.044 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T20:09:49.045 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1685563571.700188_212.1", 2023-05-31T20:09:49.045 INFO:teuthology.orchestra.run.smithi099.stdout: "last_update": "2023-05-31T20:06:11.700188Z" 2023-05-31T20:09:49.045 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T20:09:49.046 INFO:teuthology.orchestra.run.smithi099.stdout:] 2023-05-31T20:09:49.047 DEBUG:rgw_multi.tests:master meta markers={0: '1_1685562631.516089_59.1', 1: '', 2: '1_1685562970.274572_93.1', 3: '1_1685563571.700188_212.1'} 2023-05-31T20:09:49.047 INFO:rgw_multi.tests:starting meta checkpoint for zone=a2 2023-05-31T20:09:49.048 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-31T20:09:49.048 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-31T20:09:49.048 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-05-31T20:09:49.101 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2023-05-31T20:09:49.103 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2023-05-31T20:09:49.152 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:09:49.147+0000 7f6cb2c02780 20 RGW-SYNC:meta: read sync status 2023-05-31T20:09:49.152 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:09:49.148+0000 7f6cb2c02780 20 RGW-SYNC:meta: read sync status 2023-05-31T20:09:49.176 INFO:teuthology.orchestra.run.smithi099.stdout:{ 2023-05-31T20:09:49.176 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_status": { 2023-05-31T20:09:49.177 INFO:teuthology.orchestra.run.smithi099.stdout: "info": { 2023-05-31T20:09:49.177 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "sync", 2023-05-31T20:09:49.177 INFO:teuthology.orchestra.run.smithi099.stdout: "num_shards": 4, 2023-05-31T20:09:49.178 INFO:teuthology.orchestra.run.smithi099.stdout: "period": "2e6123d2-d695-4350-9a01-61a455bc782b", 2023-05-31T20:09:49.178 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2023-05-31T20:09:49.178 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:09:49.179 INFO:teuthology.orchestra.run.smithi099.stdout: "markers": [ 2023-05-31T20:09:49.179 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T20:09:49.179 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 0, 2023-05-31T20:09:49.179 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T20:09:49.179 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2023-05-31T20:09:49.180 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1685562631.516089_59.1", 2023-05-31T20:09:49.180 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T20:09:49.180 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 1, 2023-05-31T20:09:49.180 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T20:09:49.180 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2023-05-31T19:50:31.516089Z", 2023-05-31T20:09:49.180 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2023-05-31T20:09:49.181 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T20:09:49.181 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:09:49.181 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T20:09:49.181 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 1, 2023-05-31T20:09:49.182 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T20:09:49.182 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2023-05-31T20:09:49.182 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "", 2023-05-31T20:09:49.182 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T20:09:49.182 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2023-05-31T20:09:49.182 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T20:09:49.183 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "0.000000", 2023-05-31T20:09:49.183 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2023-05-31T20:09:49.183 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T20:09:49.184 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:09:49.184 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T20:09:49.184 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 2, 2023-05-31T20:09:49.184 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T20:09:49.185 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2023-05-31T20:09:49.185 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1685562970.274572_93.1", 2023-05-31T20:09:49.185 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T20:09:49.186 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2023-05-31T20:09:49.186 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T20:09:49.186 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2023-05-31T19:56:10.274572Z", 2023-05-31T20:09:49.186 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2023-05-31T20:09:49.186 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T20:09:49.187 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:09:49.187 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T20:09:49.187 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 3, 2023-05-31T20:09:49.187 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T20:09:49.188 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2023-05-31T20:09:49.188 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1685563571.700188_212.1", 2023-05-31T20:09:49.188 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T20:09:49.189 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2023-05-31T20:09:49.189 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T20:09:49.189 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2023-05-31T20:06:11.700188Z", 2023-05-31T20:09:49.189 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2023-05-31T20:09:49.190 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T20:09:49.190 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T20:09:49.190 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2023-05-31T20:09:49.191 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:09:49.191 INFO:teuthology.orchestra.run.smithi099.stdout: "full_sync": { 2023-05-31T20:09:49.191 INFO:teuthology.orchestra.run.smithi099.stdout: "total": 1, 2023-05-31T20:09:49.192 INFO:teuthology.orchestra.run.smithi099.stdout: "complete": 1 2023-05-31T20:09:49.192 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:09:49.192 INFO:teuthology.orchestra.run.smithi099.stdout: "current_time": "2023-05-31T20:09:49Z" 2023-05-31T20:09:49.193 INFO:teuthology.orchestra.run.smithi099.stdout:} 2023-05-31T20:09:49.194 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "2e6123d2-d695-4350-9a01-61a455bc782b", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1685562631.516089_59.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-05-31T19:50:31.516089Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1685562970.274572_93.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-05-31T19:56:10.274572Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1685563571.700188_212.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-05-31T20:06:11.700188Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-05-31T20:09:49Z" } 2023-05-31T20:09:49.195 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1685562631.516089_59.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-05-31T19:50:31.516089Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1685562970.274572_93.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-05-31T19:56:10.274572Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1685563571.700188_212.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-05-31T20:06:11.700188Z', 'realm_epoch': 2}}] 2023-05-31T20:09:49.196 DEBUG:rgw_multi.tests:log_status={0: '1_1685562631.516089_59.1', 1: '', 2: '1_1685562970.274572_93.1', 3: '1_1685563571.700188_212.1'} 2023-05-31T20:09:49.197 DEBUG:rgw_multi.tests:sync_status={0: '1_1685562631.516089_59.1', 1: '', 2: '1_1685562970.274572_93.1', 3: '1_1685563571.700188_212.1'} 2023-05-31T20:09:49.197 INFO:rgw_multi.tests:finish meta checkpoint for zone=a2 2023-05-31T20:09:49.198 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['zone', 'create', '--rgw-zone', 'remove', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--access-key', 'RVN19ED3W7FH7E08', '--secret', 'XzQjGe60gbIUStGfs96TKhuPw3IQi3hK', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-05-31T20:09:49.199 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'zone', 'create', '--rgw-zone', 'remove', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--access-key', 'RVN19ED3W7FH7E08', '--secret', 'XzQjGe60gbIUStGfs96TKhuPw3IQi3hK', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-05-31T20:09:49.200 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 zone create --rgw-zone remove --rgw-zonegroup a --rgw-realm test-realm --access-key RVN19ED3W7FH7E08 --secret XzQjGe60gbIUStGfs96TKhuPw3IQi3hK --cluster c1 --debug-rgw 0 --debug-ms 0 2023-05-31T20:09:49.253 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2023-05-31T20:09:49.254 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2023-05-31T20:09:49.268 INFO:teuthology.orchestra.run.smithi099.stderr:NOTICE: set zone's realm_id=aaa33de6-515b-4cac-8f1b-3a1a30452117 2023-05-31T20:09:49.288 INFO:teuthology.orchestra.run.smithi099.stdout:{ 2023-05-31T20:09:49.288 INFO:teuthology.orchestra.run.smithi099.stdout: "id": "47ab7fdb-dc77-43f2-bdeb-2eeb8ff72e9d", 2023-05-31T20:09:49.288 INFO:teuthology.orchestra.run.smithi099.stdout: "name": "remove", 2023-05-31T20:09:49.288 INFO:teuthology.orchestra.run.smithi099.stdout: "domain_root": "remove.rgw.meta:root", 2023-05-31T20:09:49.289 INFO:teuthology.orchestra.run.smithi099.stdout: "control_pool": "remove.rgw.control", 2023-05-31T20:09:49.289 INFO:teuthology.orchestra.run.smithi099.stdout: "gc_pool": "remove.rgw.log:gc", 2023-05-31T20:09:49.289 INFO:teuthology.orchestra.run.smithi099.stdout: "lc_pool": "remove.rgw.log:lc", 2023-05-31T20:09:49.289 INFO:teuthology.orchestra.run.smithi099.stdout: "log_pool": "remove.rgw.log", 2023-05-31T20:09:49.289 INFO:teuthology.orchestra.run.smithi099.stdout: "intent_log_pool": "remove.rgw.log:intent", 2023-05-31T20:09:49.289 INFO:teuthology.orchestra.run.smithi099.stdout: "usage_log_pool": "remove.rgw.log:usage", 2023-05-31T20:09:49.289 INFO:teuthology.orchestra.run.smithi099.stdout: "roles_pool": "remove.rgw.meta:roles", 2023-05-31T20:09:49.290 INFO:teuthology.orchestra.run.smithi099.stdout: "reshard_pool": "remove.rgw.log:reshard", 2023-05-31T20:09:49.290 INFO:teuthology.orchestra.run.smithi099.stdout: "user_keys_pool": "remove.rgw.meta:users.keys", 2023-05-31T20:09:49.290 INFO:teuthology.orchestra.run.smithi099.stdout: "user_email_pool": "remove.rgw.meta:users.email", 2023-05-31T20:09:49.290 INFO:teuthology.orchestra.run.smithi099.stdout: "user_swift_pool": "remove.rgw.meta:users.swift", 2023-05-31T20:09:49.290 INFO:teuthology.orchestra.run.smithi099.stdout: "user_uid_pool": "remove.rgw.meta:users.uid", 2023-05-31T20:09:49.290 INFO:teuthology.orchestra.run.smithi099.stdout: "otp_pool": "remove.rgw.otp", 2023-05-31T20:09:49.290 INFO:teuthology.orchestra.run.smithi099.stdout: "system_key": { 2023-05-31T20:09:49.291 INFO:teuthology.orchestra.run.smithi099.stdout: "access_key": "RVN19ED3W7FH7E08", 2023-05-31T20:09:49.291 INFO:teuthology.orchestra.run.smithi099.stdout: "secret_key": "XzQjGe60gbIUStGfs96TKhuPw3IQi3hK" 2023-05-31T20:09:49.291 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:09:49.292 INFO:teuthology.orchestra.run.smithi099.stdout: "placement_pools": [ 2023-05-31T20:09:49.292 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T20:09:49.292 INFO:teuthology.orchestra.run.smithi099.stdout: "key": "default-placement", 2023-05-31T20:09:49.292 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T20:09:49.292 INFO:teuthology.orchestra.run.smithi099.stdout: "index_pool": "remove.rgw.buckets.index", 2023-05-31T20:09:49.292 INFO:teuthology.orchestra.run.smithi099.stdout: "storage_classes": { 2023-05-31T20:09:49.293 INFO:teuthology.orchestra.run.smithi099.stdout: "STANDARD": { 2023-05-31T20:09:49.293 INFO:teuthology.orchestra.run.smithi099.stdout: "data_pool": "remove.rgw.buckets.data" 2023-05-31T20:09:49.293 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T20:09:49.293 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:09:49.293 INFO:teuthology.orchestra.run.smithi099.stdout: "data_extra_pool": "remove.rgw.buckets.non-ec", 2023-05-31T20:09:49.293 INFO:teuthology.orchestra.run.smithi099.stdout: "index_type": 0, 2023-05-31T20:09:49.293 INFO:teuthology.orchestra.run.smithi099.stdout: "inline_data": true 2023-05-31T20:09:49.294 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T20:09:49.294 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T20:09:49.294 INFO:teuthology.orchestra.run.smithi099.stdout: ], 2023-05-31T20:09:49.294 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_id": "aaa33de6-515b-4cac-8f1b-3a1a30452117", 2023-05-31T20:09:49.294 INFO:teuthology.orchestra.run.smithi099.stdout: "notif_pool": "remove.rgw.log:notif" 2023-05-31T20:09:49.294 INFO:teuthology.orchestra.run.smithi099.stdout:} 2023-05-31T20:09:49.296 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['period', 'update', '--rgw-zone', 'remove', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-05-31T20:09:49.296 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'period', 'update', '--rgw-zone', 'remove', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-05-31T20:09:49.298 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 period update --rgw-zone remove --rgw-zonegroup a --rgw-realm test-realm --commit --cluster c1 --debug-rgw 0 --debug-ms 0 2023-05-31T20:09:49.347 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2023-05-31T20:09:49.347 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2023-05-31T20:09:49.359 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:09:49.354+0000 7f08b6981780 0 period (2e6123d2-d695-4350-9a01-61a455bc782b does not have zone 47ab7fdb-dc77-43f2-bdeb-2eeb8ff72e9d configured 2023-05-31T20:09:56.229 INFO:teuthology.orchestra.run.smithi099.stderr:Sending period to new master zone 3d163b1e-09c7-47e2-ae87-add4aace3f38 2023-05-31T20:09:56.262 INFO:teuthology.orchestra.run.smithi099.stdout:{ 2023-05-31T20:09:56.263 INFO:teuthology.orchestra.run.smithi099.stdout: "id": "2e6123d2-d695-4350-9a01-61a455bc782b", 2023-05-31T20:09:56.263 INFO:teuthology.orchestra.run.smithi099.stdout: "epoch": 9, 2023-05-31T20:09:56.263 INFO:teuthology.orchestra.run.smithi099.stdout: "predecessor_uuid": "629b9ced-80af-41ed-8a6c-f2cfc334cbb4", 2023-05-31T20:09:56.263 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_status": [], 2023-05-31T20:09:56.264 INFO:teuthology.orchestra.run.smithi099.stdout: "period_map": { 2023-05-31T20:09:56.264 INFO:teuthology.orchestra.run.smithi099.stdout: "id": "2e6123d2-d695-4350-9a01-61a455bc782b", 2023-05-31T20:09:56.264 INFO:teuthology.orchestra.run.smithi099.stdout: "zonegroups": [ 2023-05-31T20:09:56.264 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T20:09:56.264 INFO:teuthology.orchestra.run.smithi099.stdout: "id": "a", 2023-05-31T20:09:56.264 INFO:teuthology.orchestra.run.smithi099.stdout: "name": "a", 2023-05-31T20:09:56.265 INFO:teuthology.orchestra.run.smithi099.stdout: "api_name": "", 2023-05-31T20:09:56.265 INFO:teuthology.orchestra.run.smithi099.stdout: "is_master": true, 2023-05-31T20:09:56.265 INFO:teuthology.orchestra.run.smithi099.stdout: "endpoints": [ 2023-05-31T20:09:56.265 INFO:teuthology.orchestra.run.smithi099.stdout: "http://smithi099.front.sepia.ceph.com:8000" 2023-05-31T20:09:56.265 INFO:teuthology.orchestra.run.smithi099.stdout: ], 2023-05-31T20:09:56.265 INFO:teuthology.orchestra.run.smithi099.stdout: "hostnames": [], 2023-05-31T20:09:56.266 INFO:teuthology.orchestra.run.smithi099.stdout: "hostnames_s3website": [], 2023-05-31T20:09:56.266 INFO:teuthology.orchestra.run.smithi099.stdout: "master_zone": "3d163b1e-09c7-47e2-ae87-add4aace3f38", 2023-05-31T20:09:56.266 INFO:teuthology.orchestra.run.smithi099.stdout: "zones": [ 2023-05-31T20:09:56.266 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T20:09:56.266 INFO:teuthology.orchestra.run.smithi099.stdout: "id": "3d163b1e-09c7-47e2-ae87-add4aace3f38", 2023-05-31T20:09:56.266 INFO:teuthology.orchestra.run.smithi099.stdout: "name": "a1", 2023-05-31T20:09:56.267 INFO:teuthology.orchestra.run.smithi099.stdout: "endpoints": [ 2023-05-31T20:09:56.267 INFO:teuthology.orchestra.run.smithi099.stdout: "http://smithi099.front.sepia.ceph.com:8000" 2023-05-31T20:09:56.267 INFO:teuthology.orchestra.run.smithi099.stdout: ], 2023-05-31T20:09:56.267 INFO:teuthology.orchestra.run.smithi099.stdout: "log_meta": false, 2023-05-31T20:09:56.267 INFO:teuthology.orchestra.run.smithi099.stdout: "log_data": true, 2023-05-31T20:09:56.267 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket_index_max_shards": 11, 2023-05-31T20:09:56.267 INFO:teuthology.orchestra.run.smithi099.stdout: "read_only": false, 2023-05-31T20:09:56.268 INFO:teuthology.orchestra.run.smithi099.stdout: "tier_type": "", 2023-05-31T20:09:56.268 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_from_all": true, 2023-05-31T20:09:56.269 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_from": [], 2023-05-31T20:09:56.270 INFO:teuthology.orchestra.run.smithi099.stdout: "redirect_zone": "", 2023-05-31T20:09:56.270 INFO:teuthology.orchestra.run.smithi099.stdout: "supported_features": [ 2023-05-31T20:09:56.270 INFO:teuthology.orchestra.run.smithi099.stdout: "resharding" 2023-05-31T20:09:56.270 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2023-05-31T20:09:56.270 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:09:56.270 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T20:09:56.271 INFO:teuthology.orchestra.run.smithi099.stdout: "id": "47ab7fdb-dc77-43f2-bdeb-2eeb8ff72e9d", 2023-05-31T20:09:56.271 INFO:teuthology.orchestra.run.smithi099.stdout: "name": "remove", 2023-05-31T20:09:56.271 INFO:teuthology.orchestra.run.smithi099.stdout: "endpoints": [], 2023-05-31T20:09:56.271 INFO:teuthology.orchestra.run.smithi099.stdout: "log_meta": false, 2023-05-31T20:09:56.271 INFO:teuthology.orchestra.run.smithi099.stdout: "log_data": true, 2023-05-31T20:09:56.271 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket_index_max_shards": 11, 2023-05-31T20:09:56.271 INFO:teuthology.orchestra.run.smithi099.stdout: "read_only": false, 2023-05-31T20:09:56.272 INFO:teuthology.orchestra.run.smithi099.stdout: "tier_type": "", 2023-05-31T20:09:56.272 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_from_all": true, 2023-05-31T20:09:56.272 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_from": [], 2023-05-31T20:09:56.272 INFO:teuthology.orchestra.run.smithi099.stdout: "redirect_zone": "", 2023-05-31T20:09:56.272 INFO:teuthology.orchestra.run.smithi099.stdout: "supported_features": [ 2023-05-31T20:09:56.272 INFO:teuthology.orchestra.run.smithi099.stdout: "resharding" 2023-05-31T20:09:56.273 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2023-05-31T20:09:56.273 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:09:56.273 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T20:09:56.273 INFO:teuthology.orchestra.run.smithi099.stdout: "id": "5d8acec7-7e57-4e56-8b32-e3d6b5036001", 2023-05-31T20:09:56.273 INFO:teuthology.orchestra.run.smithi099.stdout: "name": "a2", 2023-05-31T20:09:56.273 INFO:teuthology.orchestra.run.smithi099.stdout: "endpoints": [ 2023-05-31T20:09:56.273 INFO:teuthology.orchestra.run.smithi099.stdout: "http://smithi099.front.sepia.ceph.com:8001" 2023-05-31T20:09:56.274 INFO:teuthology.orchestra.run.smithi099.stdout: ], 2023-05-31T20:09:56.274 INFO:teuthology.orchestra.run.smithi099.stdout: "log_meta": false, 2023-05-31T20:09:56.274 INFO:teuthology.orchestra.run.smithi099.stdout: "log_data": true, 2023-05-31T20:09:56.274 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket_index_max_shards": 11, 2023-05-31T20:09:56.274 INFO:teuthology.orchestra.run.smithi099.stdout: "read_only": false, 2023-05-31T20:09:56.274 INFO:teuthology.orchestra.run.smithi099.stdout: "tier_type": "", 2023-05-31T20:09:56.274 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_from_all": true, 2023-05-31T20:09:56.279 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_from": [], 2023-05-31T20:09:56.279 INFO:teuthology.orchestra.run.smithi099.stdout: "redirect_zone": "", 2023-05-31T20:09:56.279 INFO:teuthology.orchestra.run.smithi099.stdout: "supported_features": [ 2023-05-31T20:09:56.280 INFO:teuthology.orchestra.run.smithi099.stdout: "resharding" 2023-05-31T20:09:56.280 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2023-05-31T20:09:56.280 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T20:09:56.280 INFO:teuthology.orchestra.run.smithi099.stdout: ], 2023-05-31T20:09:56.280 INFO:teuthology.orchestra.run.smithi099.stdout: "placement_targets": [ 2023-05-31T20:09:56.280 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T20:09:56.281 INFO:teuthology.orchestra.run.smithi099.stdout: "name": "default-placement", 2023-05-31T20:09:56.281 INFO:teuthology.orchestra.run.smithi099.stdout: "tags": [], 2023-05-31T20:09:56.281 INFO:teuthology.orchestra.run.smithi099.stdout: "storage_classes": [ 2023-05-31T20:09:56.281 INFO:teuthology.orchestra.run.smithi099.stdout: "STANDARD" 2023-05-31T20:09:56.281 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2023-05-31T20:09:56.281 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T20:09:56.281 INFO:teuthology.orchestra.run.smithi099.stdout: ], 2023-05-31T20:09:56.282 INFO:teuthology.orchestra.run.smithi099.stdout: "default_placement": "default-placement", 2023-05-31T20:09:56.282 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_id": "aaa33de6-515b-4cac-8f1b-3a1a30452117", 2023-05-31T20:09:56.282 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_policy": { 2023-05-31T20:09:56.282 INFO:teuthology.orchestra.run.smithi099.stdout: "groups": [] 2023-05-31T20:09:56.283 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:09:56.283 INFO:teuthology.orchestra.run.smithi099.stdout: "enabled_features": [ 2023-05-31T20:09:56.283 INFO:teuthology.orchestra.run.smithi099.stdout: "resharding" 2023-05-31T20:09:56.283 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2023-05-31T20:09:56.283 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:09:56.284 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T20:09:56.284 INFO:teuthology.orchestra.run.smithi099.stdout: "id": "b", 2023-05-31T20:09:56.284 INFO:teuthology.orchestra.run.smithi099.stdout: "name": "b", 2023-05-31T20:09:56.284 INFO:teuthology.orchestra.run.smithi099.stdout: "api_name": "", 2023-05-31T20:09:56.284 INFO:teuthology.orchestra.run.smithi099.stdout: "is_master": false, 2023-05-31T20:09:56.284 INFO:teuthology.orchestra.run.smithi099.stdout: "endpoints": [ 2023-05-31T20:09:56.284 INFO:teuthology.orchestra.run.smithi099.stdout: "http://smithi112.front.sepia.ceph.com:8000" 2023-05-31T20:09:56.285 INFO:teuthology.orchestra.run.smithi099.stdout: ], 2023-05-31T20:09:56.285 INFO:teuthology.orchestra.run.smithi099.stdout: "hostnames": [], 2023-05-31T20:09:56.285 INFO:teuthology.orchestra.run.smithi099.stdout: "hostnames_s3website": [], 2023-05-31T20:09:56.285 INFO:teuthology.orchestra.run.smithi099.stdout: "master_zone": "f85d7236-2f27-4988-af26-9c66ab280b1b", 2023-05-31T20:09:56.285 INFO:teuthology.orchestra.run.smithi099.stdout: "zones": [ 2023-05-31T20:09:56.285 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T20:09:56.286 INFO:teuthology.orchestra.run.smithi099.stdout: "id": "a6a3aab5-bc52-44a4-8543-5c5ea6056acd", 2023-05-31T20:09:56.286 INFO:teuthology.orchestra.run.smithi099.stdout: "name": "b2", 2023-05-31T20:09:56.287 INFO:teuthology.orchestra.run.smithi099.stdout: "endpoints": [ 2023-05-31T20:09:56.287 INFO:teuthology.orchestra.run.smithi099.stdout: "http://smithi112.front.sepia.ceph.com:8001" 2023-05-31T20:09:56.288 INFO:teuthology.orchestra.run.smithi099.stdout: ], 2023-05-31T20:09:56.288 INFO:teuthology.orchestra.run.smithi099.stdout: "log_meta": false, 2023-05-31T20:09:56.288 INFO:teuthology.orchestra.run.smithi099.stdout: "log_data": true, 2023-05-31T20:09:56.288 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket_index_max_shards": 11, 2023-05-31T20:09:56.289 INFO:teuthology.orchestra.run.smithi099.stdout: "read_only": false, 2023-05-31T20:09:56.289 INFO:teuthology.orchestra.run.smithi099.stdout: "tier_type": "", 2023-05-31T20:09:56.289 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_from_all": true, 2023-05-31T20:09:56.289 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_from": [], 2023-05-31T20:09:56.289 INFO:teuthology.orchestra.run.smithi099.stdout: "redirect_zone": "", 2023-05-31T20:09:56.289 INFO:teuthology.orchestra.run.smithi099.stdout: "supported_features": [ 2023-05-31T20:09:56.289 INFO:teuthology.orchestra.run.smithi099.stdout: "resharding" 2023-05-31T20:09:56.290 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2023-05-31T20:09:56.290 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:09:56.290 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T20:09:56.290 INFO:teuthology.orchestra.run.smithi099.stdout: "id": "f85d7236-2f27-4988-af26-9c66ab280b1b", 2023-05-31T20:09:56.290 INFO:teuthology.orchestra.run.smithi099.stdout: "name": "b1", 2023-05-31T20:09:56.290 INFO:teuthology.orchestra.run.smithi099.stdout: "endpoints": [ 2023-05-31T20:09:56.290 INFO:teuthology.orchestra.run.smithi099.stdout: "http://smithi112.front.sepia.ceph.com:8000" 2023-05-31T20:09:56.291 INFO:teuthology.orchestra.run.smithi099.stdout: ], 2023-05-31T20:09:56.291 INFO:teuthology.orchestra.run.smithi099.stdout: "log_meta": false, 2023-05-31T20:09:56.291 INFO:teuthology.orchestra.run.smithi099.stdout: "log_data": true, 2023-05-31T20:09:56.291 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket_index_max_shards": 11, 2023-05-31T20:09:56.291 INFO:teuthology.orchestra.run.smithi099.stdout: "read_only": false, 2023-05-31T20:09:56.291 INFO:teuthology.orchestra.run.smithi099.stdout: "tier_type": "", 2023-05-31T20:09:56.292 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_from_all": true, 2023-05-31T20:09:56.292 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_from": [], 2023-05-31T20:09:56.292 INFO:teuthology.orchestra.run.smithi099.stdout: "redirect_zone": "", 2023-05-31T20:09:56.292 INFO:teuthology.orchestra.run.smithi099.stdout: "supported_features": [ 2023-05-31T20:09:56.292 INFO:teuthology.orchestra.run.smithi099.stdout: "resharding" 2023-05-31T20:09:56.292 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2023-05-31T20:09:56.292 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T20:09:56.293 INFO:teuthology.orchestra.run.smithi099.stdout: ], 2023-05-31T20:09:56.293 INFO:teuthology.orchestra.run.smithi099.stdout: "placement_targets": [ 2023-05-31T20:09:56.293 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T20:09:56.293 INFO:teuthology.orchestra.run.smithi099.stdout: "name": "default-placement", 2023-05-31T20:09:56.293 INFO:teuthology.orchestra.run.smithi099.stdout: "tags": [], 2023-05-31T20:09:56.293 INFO:teuthology.orchestra.run.smithi099.stdout: "storage_classes": [ 2023-05-31T20:09:56.293 INFO:teuthology.orchestra.run.smithi099.stdout: "STANDARD" 2023-05-31T20:09:56.294 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2023-05-31T20:09:56.294 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T20:09:56.294 INFO:teuthology.orchestra.run.smithi099.stdout: ], 2023-05-31T20:09:56.294 INFO:teuthology.orchestra.run.smithi099.stdout: "default_placement": "default-placement", 2023-05-31T20:09:56.294 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_id": "aaa33de6-515b-4cac-8f1b-3a1a30452117", 2023-05-31T20:09:56.294 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_policy": { 2023-05-31T20:09:56.294 INFO:teuthology.orchestra.run.smithi099.stdout: "groups": [] 2023-05-31T20:09:56.295 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:09:56.295 INFO:teuthology.orchestra.run.smithi099.stdout: "enabled_features": [ 2023-05-31T20:09:56.295 INFO:teuthology.orchestra.run.smithi099.stdout: "resharding" 2023-05-31T20:09:56.295 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2023-05-31T20:09:56.295 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T20:09:56.295 INFO:teuthology.orchestra.run.smithi099.stdout: ], 2023-05-31T20:09:56.295 INFO:teuthology.orchestra.run.smithi099.stdout: "short_zone_ids": [ 2023-05-31T20:09:56.296 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T20:09:56.296 INFO:teuthology.orchestra.run.smithi099.stdout: "key": "3d163b1e-09c7-47e2-ae87-add4aace3f38", 2023-05-31T20:09:56.296 INFO:teuthology.orchestra.run.smithi099.stdout: "val": 3101833771 2023-05-31T20:09:56.296 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:09:56.296 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T20:09:56.296 INFO:teuthology.orchestra.run.smithi099.stdout: "key": "47ab7fdb-dc77-43f2-bdeb-2eeb8ff72e9d", 2023-05-31T20:09:56.296 INFO:teuthology.orchestra.run.smithi099.stdout: "val": 3429578837 2023-05-31T20:09:56.297 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:09:56.297 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T20:09:56.297 INFO:teuthology.orchestra.run.smithi099.stdout: "key": "5d8acec7-7e57-4e56-8b32-e3d6b5036001", 2023-05-31T20:09:56.297 INFO:teuthology.orchestra.run.smithi099.stdout: "val": 4255530623 2023-05-31T20:09:56.297 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:09:56.297 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T20:09:56.297 INFO:teuthology.orchestra.run.smithi099.stdout: "key": "a6a3aab5-bc52-44a4-8543-5c5ea6056acd", 2023-05-31T20:09:56.298 INFO:teuthology.orchestra.run.smithi099.stdout: "val": 3708073710 2023-05-31T20:09:56.298 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:09:56.298 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T20:09:56.298 INFO:teuthology.orchestra.run.smithi099.stdout: "key": "f85d7236-2f27-4988-af26-9c66ab280b1b", 2023-05-31T20:09:56.298 INFO:teuthology.orchestra.run.smithi099.stdout: "val": 233395200 2023-05-31T20:09:56.298 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T20:09:56.298 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2023-05-31T20:09:56.299 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:09:56.299 INFO:teuthology.orchestra.run.smithi099.stdout: "master_zonegroup": "a", 2023-05-31T20:09:56.299 INFO:teuthology.orchestra.run.smithi099.stdout: "master_zone": "3d163b1e-09c7-47e2-ae87-add4aace3f38", 2023-05-31T20:09:56.299 INFO:teuthology.orchestra.run.smithi099.stdout: "period_config": { 2023-05-31T20:09:56.299 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket_quota": { 2023-05-31T20:09:56.299 INFO:teuthology.orchestra.run.smithi099.stdout: "enabled": false, 2023-05-31T20:09:56.299 INFO:teuthology.orchestra.run.smithi099.stdout: "check_on_raw": false, 2023-05-31T20:09:56.300 INFO:teuthology.orchestra.run.smithi099.stdout: "max_size": -1, 2023-05-31T20:09:56.300 INFO:teuthology.orchestra.run.smithi099.stdout: "max_size_kb": 0, 2023-05-31T20:09:56.300 INFO:teuthology.orchestra.run.smithi099.stdout: "max_objects": -1 2023-05-31T20:09:56.300 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:09:56.300 INFO:teuthology.orchestra.run.smithi099.stdout: "user_quota": { 2023-05-31T20:09:56.300 INFO:teuthology.orchestra.run.smithi099.stdout: "enabled": false, 2023-05-31T20:09:56.301 INFO:teuthology.orchestra.run.smithi099.stdout: "check_on_raw": false, 2023-05-31T20:09:56.301 INFO:teuthology.orchestra.run.smithi099.stdout: "max_size": -1, 2023-05-31T20:09:56.301 INFO:teuthology.orchestra.run.smithi099.stdout: "max_size_kb": 0, 2023-05-31T20:09:56.301 INFO:teuthology.orchestra.run.smithi099.stdout: "max_objects": -1 2023-05-31T20:09:56.301 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:09:56.301 INFO:teuthology.orchestra.run.smithi099.stdout: "user_ratelimit": { 2023-05-31T20:09:56.301 INFO:teuthology.orchestra.run.smithi099.stdout: "max_read_ops": 0, 2023-05-31T20:09:56.302 INFO:teuthology.orchestra.run.smithi099.stdout: "max_write_ops": 0, 2023-05-31T20:09:56.302 INFO:teuthology.orchestra.run.smithi099.stdout: "max_read_bytes": 0, 2023-05-31T20:09:56.302 INFO:teuthology.orchestra.run.smithi099.stdout: "max_write_bytes": 0, 2023-05-31T20:09:56.302 INFO:teuthology.orchestra.run.smithi099.stdout: "enabled": false 2023-05-31T20:09:56.302 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:09:56.302 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket_ratelimit": { 2023-05-31T20:09:56.302 INFO:teuthology.orchestra.run.smithi099.stdout: "max_read_ops": 0, 2023-05-31T20:09:56.303 INFO:teuthology.orchestra.run.smithi099.stdout: "max_write_ops": 0, 2023-05-31T20:09:56.303 INFO:teuthology.orchestra.run.smithi099.stdout: "max_read_bytes": 0, 2023-05-31T20:09:56.303 INFO:teuthology.orchestra.run.smithi099.stdout: "max_write_bytes": 0, 2023-05-31T20:09:56.303 INFO:teuthology.orchestra.run.smithi099.stdout: "enabled": false 2023-05-31T20:09:56.303 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:09:56.303 INFO:teuthology.orchestra.run.smithi099.stdout: "anonymous_ratelimit": { 2023-05-31T20:09:56.303 INFO:teuthology.orchestra.run.smithi099.stdout: "max_read_ops": 0, 2023-05-31T20:09:56.304 INFO:teuthology.orchestra.run.smithi099.stdout: "max_write_ops": 0, 2023-05-31T20:09:56.304 INFO:teuthology.orchestra.run.smithi099.stdout: "max_read_bytes": 0, 2023-05-31T20:09:56.304 INFO:teuthology.orchestra.run.smithi099.stdout: "max_write_bytes": 0, 2023-05-31T20:09:56.304 INFO:teuthology.orchestra.run.smithi099.stdout: "enabled": false 2023-05-31T20:09:56.304 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T20:09:56.304 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:09:56.304 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_id": "aaa33de6-515b-4cac-8f1b-3a1a30452117", 2023-05-31T20:09:56.305 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_name": "test-realm", 2023-05-31T20:09:56.305 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2023-05-31T20:09:56.305 INFO:teuthology.orchestra.run.smithi099.stdout:} 2023-05-31T20:09:56.305 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['zonegroup', 'remove', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--rgw-zone', 'remove', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-05-31T20:09:56.306 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'zonegroup', 'remove', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--rgw-zone', 'remove', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-05-31T20:09:56.306 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 zonegroup remove --rgw-zonegroup a --rgw-realm test-realm --rgw-zone remove --cluster c1 --debug-rgw 0 --debug-ms 0 2023-05-31T20:09:56.359 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2023-05-31T20:09:56.359 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2023-05-31T20:09:56.372 INFO:teuthology.orchestra.run.smithi099.stdout:{ 2023-05-31T20:09:56.373 INFO:teuthology.orchestra.run.smithi099.stdout: "id": "a", 2023-05-31T20:09:56.373 INFO:teuthology.orchestra.run.smithi099.stdout: "name": "a", 2023-05-31T20:09:56.373 INFO:teuthology.orchestra.run.smithi099.stdout: "api_name": "", 2023-05-31T20:09:56.373 INFO:teuthology.orchestra.run.smithi099.stdout: "is_master": true, 2023-05-31T20:09:56.373 INFO:teuthology.orchestra.run.smithi099.stdout: "endpoints": [ 2023-05-31T20:09:56.373 INFO:teuthology.orchestra.run.smithi099.stdout: "http://smithi099.front.sepia.ceph.com:8000" 2023-05-31T20:09:56.374 INFO:teuthology.orchestra.run.smithi099.stdout: ], 2023-05-31T20:09:56.376 INFO:teuthology.orchestra.run.smithi099.stdout: "hostnames": [], 2023-05-31T20:09:56.376 INFO:teuthology.orchestra.run.smithi099.stdout: "hostnames_s3website": [], 2023-05-31T20:09:56.376 INFO:teuthology.orchestra.run.smithi099.stdout: "master_zone": "3d163b1e-09c7-47e2-ae87-add4aace3f38", 2023-05-31T20:09:56.376 INFO:teuthology.orchestra.run.smithi099.stdout: "zones": [ 2023-05-31T20:09:56.377 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T20:09:56.377 INFO:teuthology.orchestra.run.smithi099.stdout: "id": "3d163b1e-09c7-47e2-ae87-add4aace3f38", 2023-05-31T20:09:56.377 INFO:teuthology.orchestra.run.smithi099.stdout: "name": "a1", 2023-05-31T20:09:56.377 INFO:teuthology.orchestra.run.smithi099.stdout: "endpoints": [ 2023-05-31T20:09:56.377 INFO:teuthology.orchestra.run.smithi099.stdout: "http://smithi099.front.sepia.ceph.com:8000" 2023-05-31T20:09:56.377 INFO:teuthology.orchestra.run.smithi099.stdout: ], 2023-05-31T20:09:56.378 INFO:teuthology.orchestra.run.smithi099.stdout: "log_meta": false, 2023-05-31T20:09:56.378 INFO:teuthology.orchestra.run.smithi099.stdout: "log_data": true, 2023-05-31T20:09:56.378 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket_index_max_shards": 11, 2023-05-31T20:09:56.378 INFO:teuthology.orchestra.run.smithi099.stdout: "read_only": false, 2023-05-31T20:09:56.378 INFO:teuthology.orchestra.run.smithi099.stdout: "tier_type": "", 2023-05-31T20:09:56.378 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_from_all": true, 2023-05-31T20:09:56.378 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_from": [], 2023-05-31T20:09:56.379 INFO:teuthology.orchestra.run.smithi099.stdout: "redirect_zone": "", 2023-05-31T20:09:56.379 INFO:teuthology.orchestra.run.smithi099.stdout: "supported_features": [ 2023-05-31T20:09:56.379 INFO:teuthology.orchestra.run.smithi099.stdout: "resharding" 2023-05-31T20:09:56.379 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2023-05-31T20:09:56.379 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:09:56.379 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T20:09:56.380 INFO:teuthology.orchestra.run.smithi099.stdout: "id": "5d8acec7-7e57-4e56-8b32-e3d6b5036001", 2023-05-31T20:09:56.380 INFO:teuthology.orchestra.run.smithi099.stdout: "name": "a2", 2023-05-31T20:09:56.380 INFO:teuthology.orchestra.run.smithi099.stdout: "endpoints": [ 2023-05-31T20:09:56.380 INFO:teuthology.orchestra.run.smithi099.stdout: "http://smithi099.front.sepia.ceph.com:8001" 2023-05-31T20:09:56.380 INFO:teuthology.orchestra.run.smithi099.stdout: ], 2023-05-31T20:09:56.380 INFO:teuthology.orchestra.run.smithi099.stdout: "log_meta": false, 2023-05-31T20:09:56.380 INFO:teuthology.orchestra.run.smithi099.stdout: "log_data": true, 2023-05-31T20:09:56.381 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket_index_max_shards": 11, 2023-05-31T20:09:56.381 INFO:teuthology.orchestra.run.smithi099.stdout: "read_only": false, 2023-05-31T20:09:56.381 INFO:teuthology.orchestra.run.smithi099.stdout: "tier_type": "", 2023-05-31T20:09:56.381 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_from_all": true, 2023-05-31T20:09:56.381 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_from": [], 2023-05-31T20:09:56.381 INFO:teuthology.orchestra.run.smithi099.stdout: "redirect_zone": "", 2023-05-31T20:09:56.382 INFO:teuthology.orchestra.run.smithi099.stdout: "supported_features": [ 2023-05-31T20:09:56.382 INFO:teuthology.orchestra.run.smithi099.stdout: "resharding" 2023-05-31T20:09:56.382 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2023-05-31T20:09:56.382 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T20:09:56.382 INFO:teuthology.orchestra.run.smithi099.stdout: ], 2023-05-31T20:09:56.382 INFO:teuthology.orchestra.run.smithi099.stdout: "placement_targets": [ 2023-05-31T20:09:56.382 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T20:09:56.383 INFO:teuthology.orchestra.run.smithi099.stdout: "name": "default-placement", 2023-05-31T20:09:56.383 INFO:teuthology.orchestra.run.smithi099.stdout: "tags": [], 2023-05-31T20:09:56.383 INFO:teuthology.orchestra.run.smithi099.stdout: "storage_classes": [ 2023-05-31T20:09:56.383 INFO:teuthology.orchestra.run.smithi099.stdout: "STANDARD" 2023-05-31T20:09:56.383 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2023-05-31T20:09:56.383 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T20:09:56.384 INFO:teuthology.orchestra.run.smithi099.stdout: ], 2023-05-31T20:09:56.384 INFO:teuthology.orchestra.run.smithi099.stdout: "default_placement": "default-placement", 2023-05-31T20:09:56.384 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_id": "aaa33de6-515b-4cac-8f1b-3a1a30452117", 2023-05-31T20:09:56.384 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_policy": { 2023-05-31T20:09:56.384 INFO:teuthology.orchestra.run.smithi099.stdout: "groups": [] 2023-05-31T20:09:56.384 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:09:56.384 INFO:teuthology.orchestra.run.smithi099.stdout: "enabled_features": [ 2023-05-31T20:09:56.385 INFO:teuthology.orchestra.run.smithi099.stdout: "resharding" 2023-05-31T20:09:56.385 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2023-05-31T20:09:56.385 INFO:teuthology.orchestra.run.smithi099.stdout:} 2023-05-31T20:09:56.386 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['zonegroup', 'remove', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--rgw-zone', 'remove', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-05-31T20:09:56.388 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'zonegroup', 'remove', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--rgw-zone', 'remove', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-05-31T20:09:56.388 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 zonegroup remove --rgw-zonegroup a --rgw-realm test-realm --rgw-zone remove --cluster c1 --debug-rgw 0 --debug-ms 0 2023-05-31T20:09:56.446 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2023-05-31T20:09:56.446 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2023-05-31T20:09:56.476 INFO:teuthology.orchestra.run.smithi099.stderr:zone name remove not found in zonegroup a 2023-05-31T20:09:56.482 DEBUG:teuthology.orchestra.run:got remote process result: 2 2023-05-31T20:09:56.483 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['zone', 'delete', '--rgw-zone', 'remove', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-05-31T20:09:56.484 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'zone', 'delete', '--rgw-zone', 'remove', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-05-31T20:09:56.484 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 zone delete --rgw-zone remove --cluster c1 --debug-rgw 0 --debug-ms 0 2023-05-31T20:09:56.535 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2023-05-31T20:09:56.535 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2023-05-31T20:09:56.603 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['period', 'update', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-05-31T20:09:56.604 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'period', 'update', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-05-31T20:09:56.604 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 period update --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --commit --cluster c1 --debug-rgw 0 --debug-ms 0 2023-05-31T20:09:56.661 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2023-05-31T20:09:56.662 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2023-05-31T20:09:56.678 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:09:56.672+0000 7f9a5b6ba780 0 WARNING: can't generate connection for zone 47ab7fdb-dc77-43f2-bdeb-2eeb8ff72e9d id remove: no endpoints defined 2023-05-31T20:09:56.788 INFO:teuthology.orchestra.run.smithi099.stdout:{ 2023-05-31T20:09:56.789 INFO:teuthology.orchestra.run.smithi099.stdout: "id": "2e6123d2-d695-4350-9a01-61a455bc782b", 2023-05-31T20:09:56.789 INFO:teuthology.orchestra.run.smithi099.stdout: "epoch": 10, 2023-05-31T20:09:56.789 INFO:teuthology.orchestra.run.smithi099.stdout: "predecessor_uuid": "629b9ced-80af-41ed-8a6c-f2cfc334cbb4", 2023-05-31T20:09:56.789 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_status": [], 2023-05-31T20:09:56.790 INFO:teuthology.orchestra.run.smithi099.stdout: "period_map": { 2023-05-31T20:09:56.790 INFO:teuthology.orchestra.run.smithi099.stdout: "id": "2e6123d2-d695-4350-9a01-61a455bc782b", 2023-05-31T20:09:56.790 INFO:teuthology.orchestra.run.smithi099.stdout: "zonegroups": [ 2023-05-31T20:09:56.790 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T20:09:56.790 INFO:teuthology.orchestra.run.smithi099.stdout: "id": "a", 2023-05-31T20:09:56.791 INFO:teuthology.orchestra.run.smithi099.stdout: "name": "a", 2023-05-31T20:09:56.791 INFO:teuthology.orchestra.run.smithi099.stdout: "api_name": "", 2023-05-31T20:09:56.791 INFO:teuthology.orchestra.run.smithi099.stdout: "is_master": true, 2023-05-31T20:09:56.791 INFO:teuthology.orchestra.run.smithi099.stdout: "endpoints": [ 2023-05-31T20:09:56.791 INFO:teuthology.orchestra.run.smithi099.stdout: "http://smithi099.front.sepia.ceph.com:8000" 2023-05-31T20:09:56.791 INFO:teuthology.orchestra.run.smithi099.stdout: ], 2023-05-31T20:09:56.792 INFO:teuthology.orchestra.run.smithi099.stdout: "hostnames": [], 2023-05-31T20:09:56.792 INFO:teuthology.orchestra.run.smithi099.stdout: "hostnames_s3website": [], 2023-05-31T20:09:56.792 INFO:teuthology.orchestra.run.smithi099.stdout: "master_zone": "3d163b1e-09c7-47e2-ae87-add4aace3f38", 2023-05-31T20:09:56.792 INFO:teuthology.orchestra.run.smithi099.stdout: "zones": [ 2023-05-31T20:09:56.792 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T20:09:56.792 INFO:teuthology.orchestra.run.smithi099.stdout: "id": "3d163b1e-09c7-47e2-ae87-add4aace3f38", 2023-05-31T20:09:56.793 INFO:teuthology.orchestra.run.smithi099.stdout: "name": "a1", 2023-05-31T20:09:56.793 INFO:teuthology.orchestra.run.smithi099.stdout: "endpoints": [ 2023-05-31T20:09:56.793 INFO:teuthology.orchestra.run.smithi099.stdout: "http://smithi099.front.sepia.ceph.com:8000" 2023-05-31T20:09:56.794 INFO:teuthology.orchestra.run.smithi099.stdout: ], 2023-05-31T20:09:56.794 INFO:teuthology.orchestra.run.smithi099.stdout: "log_meta": false, 2023-05-31T20:09:56.794 INFO:teuthology.orchestra.run.smithi099.stdout: "log_data": true, 2023-05-31T20:09:56.794 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket_index_max_shards": 11, 2023-05-31T20:09:56.794 INFO:teuthology.orchestra.run.smithi099.stdout: "read_only": false, 2023-05-31T20:09:56.795 INFO:teuthology.orchestra.run.smithi099.stdout: "tier_type": "", 2023-05-31T20:09:56.795 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_from_all": true, 2023-05-31T20:09:56.795 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_from": [], 2023-05-31T20:09:56.795 INFO:teuthology.orchestra.run.smithi099.stdout: "redirect_zone": "", 2023-05-31T20:09:56.795 INFO:teuthology.orchestra.run.smithi099.stdout: "supported_features": [ 2023-05-31T20:09:56.795 INFO:teuthology.orchestra.run.smithi099.stdout: "resharding" 2023-05-31T20:09:56.795 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2023-05-31T20:09:56.796 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:09:56.796 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T20:09:56.796 INFO:teuthology.orchestra.run.smithi099.stdout: "id": "5d8acec7-7e57-4e56-8b32-e3d6b5036001", 2023-05-31T20:09:56.796 INFO:teuthology.orchestra.run.smithi099.stdout: "name": "a2", 2023-05-31T20:09:56.796 INFO:teuthology.orchestra.run.smithi099.stdout: "endpoints": [ 2023-05-31T20:09:56.796 INFO:teuthology.orchestra.run.smithi099.stdout: "http://smithi099.front.sepia.ceph.com:8001" 2023-05-31T20:09:56.797 INFO:teuthology.orchestra.run.smithi099.stdout: ], 2023-05-31T20:09:56.797 INFO:teuthology.orchestra.run.smithi099.stdout: "log_meta": false, 2023-05-31T20:09:56.797 INFO:teuthology.orchestra.run.smithi099.stdout: "log_data": true, 2023-05-31T20:09:56.797 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket_index_max_shards": 11, 2023-05-31T20:09:56.797 INFO:teuthology.orchestra.run.smithi099.stdout: "read_only": false, 2023-05-31T20:09:56.797 INFO:teuthology.orchestra.run.smithi099.stdout: "tier_type": "", 2023-05-31T20:09:56.797 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_from_all": true, 2023-05-31T20:09:56.798 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_from": [], 2023-05-31T20:09:56.798 INFO:teuthology.orchestra.run.smithi099.stdout: "redirect_zone": "", 2023-05-31T20:09:56.798 INFO:teuthology.orchestra.run.smithi099.stdout: "supported_features": [ 2023-05-31T20:09:56.798 INFO:teuthology.orchestra.run.smithi099.stdout: "resharding" 2023-05-31T20:09:56.798 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2023-05-31T20:09:56.798 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T20:09:56.798 INFO:teuthology.orchestra.run.smithi099.stdout: ], 2023-05-31T20:09:56.799 INFO:teuthology.orchestra.run.smithi099.stdout: "placement_targets": [ 2023-05-31T20:09:56.799 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T20:09:56.799 INFO:teuthology.orchestra.run.smithi099.stdout: "name": "default-placement", 2023-05-31T20:09:56.799 INFO:teuthology.orchestra.run.smithi099.stdout: "tags": [], 2023-05-31T20:09:56.799 INFO:teuthology.orchestra.run.smithi099.stdout: "storage_classes": [ 2023-05-31T20:09:56.799 INFO:teuthology.orchestra.run.smithi099.stdout: "STANDARD" 2023-05-31T20:09:56.800 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2023-05-31T20:09:56.800 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T20:09:56.800 INFO:teuthology.orchestra.run.smithi099.stdout: ], 2023-05-31T20:09:56.800 INFO:teuthology.orchestra.run.smithi099.stdout: "default_placement": "default-placement", 2023-05-31T20:09:56.800 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_id": "aaa33de6-515b-4cac-8f1b-3a1a30452117", 2023-05-31T20:09:56.800 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_policy": { 2023-05-31T20:09:56.800 INFO:teuthology.orchestra.run.smithi099.stdout: "groups": [] 2023-05-31T20:09:56.801 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:09:56.801 INFO:teuthology.orchestra.run.smithi099.stdout: "enabled_features": [ 2023-05-31T20:09:56.801 INFO:teuthology.orchestra.run.smithi099.stdout: "resharding" 2023-05-31T20:09:56.801 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2023-05-31T20:09:56.801 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:09:56.801 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T20:09:56.801 INFO:teuthology.orchestra.run.smithi099.stdout: "id": "b", 2023-05-31T20:09:56.802 INFO:teuthology.orchestra.run.smithi099.stdout: "name": "b", 2023-05-31T20:09:56.802 INFO:teuthology.orchestra.run.smithi099.stdout: "api_name": "", 2023-05-31T20:09:56.802 INFO:teuthology.orchestra.run.smithi099.stdout: "is_master": false, 2023-05-31T20:09:56.802 INFO:teuthology.orchestra.run.smithi099.stdout: "endpoints": [ 2023-05-31T20:09:56.802 INFO:teuthology.orchestra.run.smithi099.stdout: "http://smithi112.front.sepia.ceph.com:8000" 2023-05-31T20:09:56.802 INFO:teuthology.orchestra.run.smithi099.stdout: ], 2023-05-31T20:09:56.802 INFO:teuthology.orchestra.run.smithi099.stdout: "hostnames": [], 2023-05-31T20:09:56.803 INFO:teuthology.orchestra.run.smithi099.stdout: "hostnames_s3website": [], 2023-05-31T20:09:56.804 INFO:teuthology.orchestra.run.smithi099.stdout: "master_zone": "f85d7236-2f27-4988-af26-9c66ab280b1b", 2023-05-31T20:09:56.805 INFO:teuthology.orchestra.run.smithi099.stdout: "zones": [ 2023-05-31T20:09:56.805 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T20:09:56.805 INFO:teuthology.orchestra.run.smithi099.stdout: "id": "a6a3aab5-bc52-44a4-8543-5c5ea6056acd", 2023-05-31T20:09:56.805 INFO:teuthology.orchestra.run.smithi099.stdout: "name": "b2", 2023-05-31T20:09:56.805 INFO:teuthology.orchestra.run.smithi099.stdout: "endpoints": [ 2023-05-31T20:09:56.805 INFO:teuthology.orchestra.run.smithi099.stdout: "http://smithi112.front.sepia.ceph.com:8001" 2023-05-31T20:09:56.805 INFO:teuthology.orchestra.run.smithi099.stdout: ], 2023-05-31T20:09:56.806 INFO:teuthology.orchestra.run.smithi099.stdout: "log_meta": false, 2023-05-31T20:09:56.806 INFO:teuthology.orchestra.run.smithi099.stdout: "log_data": true, 2023-05-31T20:09:56.806 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket_index_max_shards": 11, 2023-05-31T20:09:56.806 INFO:teuthology.orchestra.run.smithi099.stdout: "read_only": false, 2023-05-31T20:09:56.806 INFO:teuthology.orchestra.run.smithi099.stdout: "tier_type": "", 2023-05-31T20:09:56.806 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_from_all": true, 2023-05-31T20:09:56.807 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_from": [], 2023-05-31T20:09:56.807 INFO:teuthology.orchestra.run.smithi099.stdout: "redirect_zone": "", 2023-05-31T20:09:56.807 INFO:teuthology.orchestra.run.smithi099.stdout: "supported_features": [ 2023-05-31T20:09:56.807 INFO:teuthology.orchestra.run.smithi099.stdout: "resharding" 2023-05-31T20:09:56.807 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2023-05-31T20:09:56.807 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:09:56.807 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T20:09:56.813 INFO:teuthology.orchestra.run.smithi099.stdout: "id": "f85d7236-2f27-4988-af26-9c66ab280b1b", 2023-05-31T20:09:56.813 INFO:teuthology.orchestra.run.smithi099.stdout: "name": "b1", 2023-05-31T20:09:56.814 INFO:teuthology.orchestra.run.smithi099.stdout: "endpoints": [ 2023-05-31T20:09:56.814 INFO:teuthology.orchestra.run.smithi099.stdout: "http://smithi112.front.sepia.ceph.com:8000" 2023-05-31T20:09:56.814 INFO:teuthology.orchestra.run.smithi099.stdout: ], 2023-05-31T20:09:56.814 INFO:teuthology.orchestra.run.smithi099.stdout: "log_meta": false, 2023-05-31T20:09:56.814 INFO:teuthology.orchestra.run.smithi099.stdout: "log_data": true, 2023-05-31T20:09:56.814 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket_index_max_shards": 11, 2023-05-31T20:09:56.815 INFO:teuthology.orchestra.run.smithi099.stdout: "read_only": false, 2023-05-31T20:09:56.815 INFO:teuthology.orchestra.run.smithi099.stdout: "tier_type": "", 2023-05-31T20:09:56.815 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_from_all": true, 2023-05-31T20:09:56.815 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_from": [], 2023-05-31T20:09:56.815 INFO:teuthology.orchestra.run.smithi099.stdout: "redirect_zone": "", 2023-05-31T20:09:56.815 INFO:teuthology.orchestra.run.smithi099.stdout: "supported_features": [ 2023-05-31T20:09:56.815 INFO:teuthology.orchestra.run.smithi099.stdout: "resharding" 2023-05-31T20:09:56.816 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2023-05-31T20:09:56.816 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T20:09:56.816 INFO:teuthology.orchestra.run.smithi099.stdout: ], 2023-05-31T20:09:56.816 INFO:teuthology.orchestra.run.smithi099.stdout: "placement_targets": [ 2023-05-31T20:09:56.816 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T20:09:56.816 INFO:teuthology.orchestra.run.smithi099.stdout: "name": "default-placement", 2023-05-31T20:09:56.817 INFO:teuthology.orchestra.run.smithi099.stdout: "tags": [], 2023-05-31T20:09:56.817 INFO:teuthology.orchestra.run.smithi099.stdout: "storage_classes": [ 2023-05-31T20:09:56.817 INFO:teuthology.orchestra.run.smithi099.stdout: "STANDARD" 2023-05-31T20:09:56.817 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2023-05-31T20:09:56.817 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T20:09:56.817 INFO:teuthology.orchestra.run.smithi099.stdout: ], 2023-05-31T20:09:56.817 INFO:teuthology.orchestra.run.smithi099.stdout: "default_placement": "default-placement", 2023-05-31T20:09:56.818 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_id": "aaa33de6-515b-4cac-8f1b-3a1a30452117", 2023-05-31T20:09:56.818 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_policy": { 2023-05-31T20:09:56.818 INFO:teuthology.orchestra.run.smithi099.stdout: "groups": [] 2023-05-31T20:09:56.818 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:09:56.818 INFO:teuthology.orchestra.run.smithi099.stdout: "enabled_features": [ 2023-05-31T20:09:56.818 INFO:teuthology.orchestra.run.smithi099.stdout: "resharding" 2023-05-31T20:09:56.818 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2023-05-31T20:09:56.819 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T20:09:56.819 INFO:teuthology.orchestra.run.smithi099.stdout: ], 2023-05-31T20:09:56.819 INFO:teuthology.orchestra.run.smithi099.stdout: "short_zone_ids": [ 2023-05-31T20:09:56.819 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T20:09:56.819 INFO:teuthology.orchestra.run.smithi099.stdout: "key": "3d163b1e-09c7-47e2-ae87-add4aace3f38", 2023-05-31T20:09:56.819 INFO:teuthology.orchestra.run.smithi099.stdout: "val": 3101833771 2023-05-31T20:09:56.820 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:09:56.820 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T20:09:56.820 INFO:teuthology.orchestra.run.smithi099.stdout: "key": "5d8acec7-7e57-4e56-8b32-e3d6b5036001", 2023-05-31T20:09:56.820 INFO:teuthology.orchestra.run.smithi099.stdout: "val": 4255530623 2023-05-31T20:09:56.820 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:09:56.820 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T20:09:56.820 INFO:teuthology.orchestra.run.smithi099.stdout: "key": "a6a3aab5-bc52-44a4-8543-5c5ea6056acd", 2023-05-31T20:09:56.821 INFO:teuthology.orchestra.run.smithi099.stdout: "val": 3708073710 2023-05-31T20:09:56.821 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:09:56.821 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T20:09:56.821 INFO:teuthology.orchestra.run.smithi099.stdout: "key": "f85d7236-2f27-4988-af26-9c66ab280b1b", 2023-05-31T20:09:56.821 INFO:teuthology.orchestra.run.smithi099.stdout: "val": 233395200 2023-05-31T20:09:56.821 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T20:09:56.821 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2023-05-31T20:09:56.822 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:09:56.822 INFO:teuthology.orchestra.run.smithi099.stdout: "master_zonegroup": "a", 2023-05-31T20:09:56.822 INFO:teuthology.orchestra.run.smithi099.stdout: "master_zone": "3d163b1e-09c7-47e2-ae87-add4aace3f38", 2023-05-31T20:09:56.822 INFO:teuthology.orchestra.run.smithi099.stdout: "period_config": { 2023-05-31T20:09:56.822 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket_quota": { 2023-05-31T20:09:56.822 INFO:teuthology.orchestra.run.smithi099.stdout: "enabled": false, 2023-05-31T20:09:56.823 INFO:teuthology.orchestra.run.smithi099.stdout: "check_on_raw": false, 2023-05-31T20:09:56.823 INFO:teuthology.orchestra.run.smithi099.stdout: "max_size": -1, 2023-05-31T20:09:56.823 INFO:teuthology.orchestra.run.smithi099.stdout: "max_size_kb": 0, 2023-05-31T20:09:56.823 INFO:teuthology.orchestra.run.smithi099.stdout: "max_objects": -1 2023-05-31T20:09:56.823 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:09:56.824 INFO:teuthology.orchestra.run.smithi099.stdout: "user_quota": { 2023-05-31T20:09:56.824 INFO:teuthology.orchestra.run.smithi099.stdout: "enabled": false, 2023-05-31T20:09:56.824 INFO:teuthology.orchestra.run.smithi099.stdout: "check_on_raw": false, 2023-05-31T20:09:56.824 INFO:teuthology.orchestra.run.smithi099.stdout: "max_size": -1, 2023-05-31T20:09:56.824 INFO:teuthology.orchestra.run.smithi099.stdout: "max_size_kb": 0, 2023-05-31T20:09:56.824 INFO:teuthology.orchestra.run.smithi099.stdout: "max_objects": -1 2023-05-31T20:09:56.824 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:09:56.825 INFO:teuthology.orchestra.run.smithi099.stdout: "user_ratelimit": { 2023-05-31T20:09:56.825 INFO:teuthology.orchestra.run.smithi099.stdout: "max_read_ops": 0, 2023-05-31T20:09:56.825 INFO:teuthology.orchestra.run.smithi099.stdout: "max_write_ops": 0, 2023-05-31T20:09:56.825 INFO:teuthology.orchestra.run.smithi099.stdout: "max_read_bytes": 0, 2023-05-31T20:09:56.825 INFO:teuthology.orchestra.run.smithi099.stdout: "max_write_bytes": 0, 2023-05-31T20:09:56.825 INFO:teuthology.orchestra.run.smithi099.stdout: "enabled": false 2023-05-31T20:09:56.825 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:09:56.826 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket_ratelimit": { 2023-05-31T20:09:56.826 INFO:teuthology.orchestra.run.smithi099.stdout: "max_read_ops": 0, 2023-05-31T20:09:56.826 INFO:teuthology.orchestra.run.smithi099.stdout: "max_write_ops": 0, 2023-05-31T20:09:56.826 INFO:teuthology.orchestra.run.smithi099.stdout: "max_read_bytes": 0, 2023-05-31T20:09:56.826 INFO:teuthology.orchestra.run.smithi099.stdout: "max_write_bytes": 0, 2023-05-31T20:09:56.826 INFO:teuthology.orchestra.run.smithi099.stdout: "enabled": false 2023-05-31T20:09:56.827 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:09:56.827 INFO:teuthology.orchestra.run.smithi099.stdout: "anonymous_ratelimit": { 2023-05-31T20:09:56.827 INFO:teuthology.orchestra.run.smithi099.stdout: "max_read_ops": 0, 2023-05-31T20:09:56.827 INFO:teuthology.orchestra.run.smithi099.stdout: "max_write_ops": 0, 2023-05-31T20:09:56.827 INFO:teuthology.orchestra.run.smithi099.stdout: "max_read_bytes": 0, 2023-05-31T20:09:56.827 INFO:teuthology.orchestra.run.smithi099.stdout: "max_write_bytes": 0, 2023-05-31T20:09:56.827 INFO:teuthology.orchestra.run.smithi099.stdout: "enabled": false 2023-05-31T20:09:56.828 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T20:09:56.828 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:09:56.828 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_id": "aaa33de6-515b-4cac-8f1b-3a1a30452117", 2023-05-31T20:09:56.828 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_name": "test-realm", 2023-05-31T20:09:56.828 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2023-05-31T20:09:56.829 INFO:teuthology.orchestra.run.smithi099.stdout:} 2023-05-31T20:09:56.830 INFO:tasks.rgw_multisite_tests:rgw_multi.tests.test_zonegroup_remove ... ok 2023-05-31T20:09:56.830 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['zonegroup', 'create', '--rgw-zonegroup', 'remove_zg', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-05-31T20:09:56.831 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'zonegroup', 'create', '--rgw-zonegroup', 'remove_zg', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-05-31T20:09:56.831 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 zonegroup create --rgw-zonegroup remove_zg --cluster c1 --debug-rgw 0 --debug-ms 0 2023-05-31T20:09:56.882 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2023-05-31T20:09:56.883 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2023-05-31T20:09:56.915 INFO:teuthology.orchestra.run.smithi099.stdout:{ 2023-05-31T20:09:56.915 INFO:teuthology.orchestra.run.smithi099.stdout: "id": "5dd70538-9f66-4a64-b2e5-7c92d6d3ba08", 2023-05-31T20:09:56.915 INFO:teuthology.orchestra.run.smithi099.stdout: "name": "remove_zg", 2023-05-31T20:09:56.916 INFO:teuthology.orchestra.run.smithi099.stdout: "api_name": "remove_zg", 2023-05-31T20:09:56.916 INFO:teuthology.orchestra.run.smithi099.stdout: "is_master": false, 2023-05-31T20:09:56.916 INFO:teuthology.orchestra.run.smithi099.stdout: "endpoints": [], 2023-05-31T20:09:56.916 INFO:teuthology.orchestra.run.smithi099.stdout: "hostnames": [], 2023-05-31T20:09:56.916 INFO:teuthology.orchestra.run.smithi099.stdout: "hostnames_s3website": [], 2023-05-31T20:09:56.916 INFO:teuthology.orchestra.run.smithi099.stdout: "master_zone": "", 2023-05-31T20:09:56.916 INFO:teuthology.orchestra.run.smithi099.stdout: "zones": [], 2023-05-31T20:09:56.917 INFO:teuthology.orchestra.run.smithi099.stdout: "placement_targets": [ 2023-05-31T20:09:56.917 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T20:09:56.917 INFO:teuthology.orchestra.run.smithi099.stdout: "name": "default-placement", 2023-05-31T20:09:56.917 INFO:teuthology.orchestra.run.smithi099.stdout: "tags": [], 2023-05-31T20:09:56.917 INFO:teuthology.orchestra.run.smithi099.stdout: "storage_classes": [] 2023-05-31T20:09:56.917 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T20:09:56.918 INFO:teuthology.orchestra.run.smithi099.stdout: ], 2023-05-31T20:09:56.918 INFO:teuthology.orchestra.run.smithi099.stdout: "default_placement": "default-placement", 2023-05-31T20:09:56.918 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_id": "aaa33de6-515b-4cac-8f1b-3a1a30452117", 2023-05-31T20:09:56.918 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_policy": { 2023-05-31T20:09:56.918 INFO:teuthology.orchestra.run.smithi099.stdout: "groups": [] 2023-05-31T20:09:56.918 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:09:56.918 INFO:teuthology.orchestra.run.smithi099.stdout: "enabled_features": [ 2023-05-31T20:09:56.919 INFO:teuthology.orchestra.run.smithi099.stdout: "resharding" 2023-05-31T20:09:56.919 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2023-05-31T20:09:56.919 INFO:teuthology.orchestra.run.smithi099.stdout:} 2023-05-31T20:09:56.920 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['zone', 'create', '--rgw-zone', 'remove', '--rgw-zonegroup', 'remove_zg', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-05-31T20:09:56.921 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'zone', 'create', '--rgw-zone', 'remove', '--rgw-zonegroup', 'remove_zg', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-05-31T20:09:56.921 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 zone create --rgw-zone remove --rgw-zonegroup remove_zg --cluster c1 --debug-rgw 0 --debug-ms 0 2023-05-31T20:09:56.975 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2023-05-31T20:09:56.976 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2023-05-31T20:09:57.014 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:09:57.008+0000 7f936103a780 0 NOTICE: promoted remove as new master_zone of zonegroup remove_zg 2023-05-31T20:09:57.023 INFO:teuthology.orchestra.run.smithi099.stdout:{ 2023-05-31T20:09:57.023 INFO:teuthology.orchestra.run.smithi099.stdout: "id": "27861e7d-8881-42fd-8069-97f0a22b91b7", 2023-05-31T20:09:57.024 INFO:teuthology.orchestra.run.smithi099.stdout: "name": "remove", 2023-05-31T20:09:57.024 INFO:teuthology.orchestra.run.smithi099.stdout: "domain_root": "remove.rgw.meta:root", 2023-05-31T20:09:57.024 INFO:teuthology.orchestra.run.smithi099.stdout: "control_pool": "remove.rgw.control", 2023-05-31T20:09:57.024 INFO:teuthology.orchestra.run.smithi099.stdout: "gc_pool": "remove.rgw.log:gc", 2023-05-31T20:09:57.024 INFO:teuthology.orchestra.run.smithi099.stdout: "lc_pool": "remove.rgw.log:lc", 2023-05-31T20:09:57.024 INFO:teuthology.orchestra.run.smithi099.stdout: "log_pool": "remove.rgw.log", 2023-05-31T20:09:57.024 INFO:teuthology.orchestra.run.smithi099.stdout: "intent_log_pool": "remove.rgw.log:intent", 2023-05-31T20:09:57.025 INFO:teuthology.orchestra.run.smithi099.stdout: "usage_log_pool": "remove.rgw.log:usage", 2023-05-31T20:09:57.025 INFO:teuthology.orchestra.run.smithi099.stdout: "roles_pool": "remove.rgw.meta:roles", 2023-05-31T20:09:57.025 INFO:teuthology.orchestra.run.smithi099.stdout: "reshard_pool": "remove.rgw.log:reshard", 2023-05-31T20:09:57.025 INFO:teuthology.orchestra.run.smithi099.stdout: "user_keys_pool": "remove.rgw.meta:users.keys", 2023-05-31T20:09:57.025 INFO:teuthology.orchestra.run.smithi099.stdout: "user_email_pool": "remove.rgw.meta:users.email", 2023-05-31T20:09:57.025 INFO:teuthology.orchestra.run.smithi099.stdout: "user_swift_pool": "remove.rgw.meta:users.swift", 2023-05-31T20:09:57.025 INFO:teuthology.orchestra.run.smithi099.stdout: "user_uid_pool": "remove.rgw.meta:users.uid", 2023-05-31T20:09:57.026 INFO:teuthology.orchestra.run.smithi099.stdout: "otp_pool": "remove.rgw.otp", 2023-05-31T20:09:57.026 INFO:teuthology.orchestra.run.smithi099.stdout: "system_key": { 2023-05-31T20:09:57.026 INFO:teuthology.orchestra.run.smithi099.stdout: "access_key": "", 2023-05-31T20:09:57.026 INFO:teuthology.orchestra.run.smithi099.stdout: "secret_key": "" 2023-05-31T20:09:57.026 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:09:57.026 INFO:teuthology.orchestra.run.smithi099.stdout: "placement_pools": [ 2023-05-31T20:09:57.027 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T20:09:57.027 INFO:teuthology.orchestra.run.smithi099.stdout: "key": "default-placement", 2023-05-31T20:09:57.027 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T20:09:57.027 INFO:teuthology.orchestra.run.smithi099.stdout: "index_pool": "remove.rgw.buckets.index", 2023-05-31T20:09:57.027 INFO:teuthology.orchestra.run.smithi099.stdout: "storage_classes": { 2023-05-31T20:09:57.027 INFO:teuthology.orchestra.run.smithi099.stdout: "STANDARD": { 2023-05-31T20:09:57.027 INFO:teuthology.orchestra.run.smithi099.stdout: "data_pool": "remove.rgw.buckets.data" 2023-05-31T20:09:57.028 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T20:09:57.028 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:09:57.028 INFO:teuthology.orchestra.run.smithi099.stdout: "data_extra_pool": "remove.rgw.buckets.non-ec", 2023-05-31T20:09:57.028 INFO:teuthology.orchestra.run.smithi099.stdout: "index_type": 0, 2023-05-31T20:09:57.028 INFO:teuthology.orchestra.run.smithi099.stdout: "inline_data": true 2023-05-31T20:09:57.028 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T20:09:57.028 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T20:09:57.029 INFO:teuthology.orchestra.run.smithi099.stdout: ], 2023-05-31T20:09:57.029 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_id": "aaa33de6-515b-4cac-8f1b-3a1a30452117", 2023-05-31T20:09:57.029 INFO:teuthology.orchestra.run.smithi099.stdout: "notif_pool": "remove.rgw.log:notif" 2023-05-31T20:09:57.029 INFO:teuthology.orchestra.run.smithi099.stdout:} 2023-05-31T20:09:57.030 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['period', 'update', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-05-31T20:09:57.031 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'period', 'update', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-05-31T20:09:57.031 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 period update --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --commit --cluster c1 --debug-rgw 0 --debug-ms 0 2023-05-31T20:09:57.084 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2023-05-31T20:09:57.084 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2023-05-31T20:09:57.205 INFO:teuthology.orchestra.run.smithi099.stdout:{ 2023-05-31T20:09:57.206 INFO:teuthology.orchestra.run.smithi099.stdout: "id": "2e6123d2-d695-4350-9a01-61a455bc782b", 2023-05-31T20:09:57.206 INFO:teuthology.orchestra.run.smithi099.stdout: "epoch": 11, 2023-05-31T20:09:57.206 INFO:teuthology.orchestra.run.smithi099.stdout: "predecessor_uuid": "629b9ced-80af-41ed-8a6c-f2cfc334cbb4", 2023-05-31T20:09:57.207 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_status": [], 2023-05-31T20:09:57.207 INFO:teuthology.orchestra.run.smithi099.stdout: "period_map": { 2023-05-31T20:09:57.207 INFO:teuthology.orchestra.run.smithi099.stdout: "id": "2e6123d2-d695-4350-9a01-61a455bc782b", 2023-05-31T20:09:57.208 INFO:teuthology.orchestra.run.smithi099.stdout: "zonegroups": [ 2023-05-31T20:09:57.208 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T20:09:57.209 INFO:teuthology.orchestra.run.smithi099.stdout: "id": "5dd70538-9f66-4a64-b2e5-7c92d6d3ba08", 2023-05-31T20:09:57.209 INFO:teuthology.orchestra.run.smithi099.stdout: "name": "remove_zg", 2023-05-31T20:09:57.209 INFO:teuthology.orchestra.run.smithi099.stdout: "api_name": "remove_zg", 2023-05-31T20:09:57.209 INFO:teuthology.orchestra.run.smithi099.stdout: "is_master": false, 2023-05-31T20:09:57.210 INFO:teuthology.orchestra.run.smithi099.stdout: "endpoints": [], 2023-05-31T20:09:57.210 INFO:teuthology.orchestra.run.smithi099.stdout: "hostnames": [], 2023-05-31T20:09:57.210 INFO:teuthology.orchestra.run.smithi099.stdout: "hostnames_s3website": [], 2023-05-31T20:09:57.210 INFO:teuthology.orchestra.run.smithi099.stdout: "master_zone": "27861e7d-8881-42fd-8069-97f0a22b91b7", 2023-05-31T20:09:57.210 INFO:teuthology.orchestra.run.smithi099.stdout: "zones": [ 2023-05-31T20:09:57.211 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T20:09:57.211 INFO:teuthology.orchestra.run.smithi099.stdout: "id": "27861e7d-8881-42fd-8069-97f0a22b91b7", 2023-05-31T20:09:57.211 INFO:teuthology.orchestra.run.smithi099.stdout: "name": "remove", 2023-05-31T20:09:57.212 INFO:teuthology.orchestra.run.smithi099.stdout: "endpoints": [], 2023-05-31T20:09:57.212 INFO:teuthology.orchestra.run.smithi099.stdout: "log_meta": false, 2023-05-31T20:09:57.213 INFO:teuthology.orchestra.run.smithi099.stdout: "log_data": false, 2023-05-31T20:09:57.213 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket_index_max_shards": 11, 2023-05-31T20:09:57.213 INFO:teuthology.orchestra.run.smithi099.stdout: "read_only": false, 2023-05-31T20:09:57.213 INFO:teuthology.orchestra.run.smithi099.stdout: "tier_type": "", 2023-05-31T20:09:57.213 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_from_all": true, 2023-05-31T20:09:57.213 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_from": [], 2023-05-31T20:09:57.214 INFO:teuthology.orchestra.run.smithi099.stdout: "redirect_zone": "", 2023-05-31T20:09:57.214 INFO:teuthology.orchestra.run.smithi099.stdout: "supported_features": [ 2023-05-31T20:09:57.214 INFO:teuthology.orchestra.run.smithi099.stdout: "resharding" 2023-05-31T20:09:57.214 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2023-05-31T20:09:57.214 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T20:09:57.215 INFO:teuthology.orchestra.run.smithi099.stdout: ], 2023-05-31T20:09:57.215 INFO:teuthology.orchestra.run.smithi099.stdout: "placement_targets": [ 2023-05-31T20:09:57.215 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T20:09:57.215 INFO:teuthology.orchestra.run.smithi099.stdout: "name": "default-placement", 2023-05-31T20:09:57.215 INFO:teuthology.orchestra.run.smithi099.stdout: "tags": [], 2023-05-31T20:09:57.215 INFO:teuthology.orchestra.run.smithi099.stdout: "storage_classes": [ 2023-05-31T20:09:57.216 INFO:teuthology.orchestra.run.smithi099.stdout: "STANDARD" 2023-05-31T20:09:57.216 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2023-05-31T20:09:57.216 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T20:09:57.216 INFO:teuthology.orchestra.run.smithi099.stdout: ], 2023-05-31T20:09:57.216 INFO:teuthology.orchestra.run.smithi099.stdout: "default_placement": "default-placement", 2023-05-31T20:09:57.217 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_id": "aaa33de6-515b-4cac-8f1b-3a1a30452117", 2023-05-31T20:09:57.217 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_policy": { 2023-05-31T20:09:57.217 INFO:teuthology.orchestra.run.smithi099.stdout: "groups": [] 2023-05-31T20:09:57.217 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:09:57.218 INFO:teuthology.orchestra.run.smithi099.stdout: "enabled_features": [ 2023-05-31T20:09:57.218 INFO:teuthology.orchestra.run.smithi099.stdout: "resharding" 2023-05-31T20:09:57.218 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2023-05-31T20:09:57.218 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:09:57.218 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T20:09:57.219 INFO:teuthology.orchestra.run.smithi099.stdout: "id": "a", 2023-05-31T20:09:57.219 INFO:teuthology.orchestra.run.smithi099.stdout: "name": "a", 2023-05-31T20:09:57.219 INFO:teuthology.orchestra.run.smithi099.stdout: "api_name": "", 2023-05-31T20:09:57.219 INFO:teuthology.orchestra.run.smithi099.stdout: "is_master": true, 2023-05-31T20:09:57.219 INFO:teuthology.orchestra.run.smithi099.stdout: "endpoints": [ 2023-05-31T20:09:57.219 INFO:teuthology.orchestra.run.smithi099.stdout: "http://smithi099.front.sepia.ceph.com:8000" 2023-05-31T20:09:57.219 INFO:teuthology.orchestra.run.smithi099.stdout: ], 2023-05-31T20:09:57.220 INFO:teuthology.orchestra.run.smithi099.stdout: "hostnames": [], 2023-05-31T20:09:57.220 INFO:teuthology.orchestra.run.smithi099.stdout: "hostnames_s3website": [], 2023-05-31T20:09:57.220 INFO:teuthology.orchestra.run.smithi099.stdout: "master_zone": "3d163b1e-09c7-47e2-ae87-add4aace3f38", 2023-05-31T20:09:57.220 INFO:teuthology.orchestra.run.smithi099.stdout: "zones": [ 2023-05-31T20:09:57.220 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T20:09:57.221 INFO:teuthology.orchestra.run.smithi099.stdout: "id": "3d163b1e-09c7-47e2-ae87-add4aace3f38", 2023-05-31T20:09:57.221 INFO:teuthology.orchestra.run.smithi099.stdout: "name": "a1", 2023-05-31T20:09:57.221 INFO:teuthology.orchestra.run.smithi099.stdout: "endpoints": [ 2023-05-31T20:09:57.221 INFO:teuthology.orchestra.run.smithi099.stdout: "http://smithi099.front.sepia.ceph.com:8000" 2023-05-31T20:09:57.221 INFO:teuthology.orchestra.run.smithi099.stdout: ], 2023-05-31T20:09:57.221 INFO:teuthology.orchestra.run.smithi099.stdout: "log_meta": false, 2023-05-31T20:09:57.221 INFO:teuthology.orchestra.run.smithi099.stdout: "log_data": true, 2023-05-31T20:09:57.222 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket_index_max_shards": 11, 2023-05-31T20:09:57.222 INFO:teuthology.orchestra.run.smithi099.stdout: "read_only": false, 2023-05-31T20:09:57.222 INFO:teuthology.orchestra.run.smithi099.stdout: "tier_type": "", 2023-05-31T20:09:57.222 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_from_all": true, 2023-05-31T20:09:57.222 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_from": [], 2023-05-31T20:09:57.222 INFO:teuthology.orchestra.run.smithi099.stdout: "redirect_zone": "", 2023-05-31T20:09:57.223 INFO:teuthology.orchestra.run.smithi099.stdout: "supported_features": [ 2023-05-31T20:09:57.223 INFO:teuthology.orchestra.run.smithi099.stdout: "resharding" 2023-05-31T20:09:57.223 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2023-05-31T20:09:57.223 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:09:57.223 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T20:09:57.223 INFO:teuthology.orchestra.run.smithi099.stdout: "id": "5d8acec7-7e57-4e56-8b32-e3d6b5036001", 2023-05-31T20:09:57.223 INFO:teuthology.orchestra.run.smithi099.stdout: "name": "a2", 2023-05-31T20:09:57.224 INFO:teuthology.orchestra.run.smithi099.stdout: "endpoints": [ 2023-05-31T20:09:57.224 INFO:teuthology.orchestra.run.smithi099.stdout: "http://smithi099.front.sepia.ceph.com:8001" 2023-05-31T20:09:57.224 INFO:teuthology.orchestra.run.smithi099.stdout: ], 2023-05-31T20:09:57.224 INFO:teuthology.orchestra.run.smithi099.stdout: "log_meta": false, 2023-05-31T20:09:57.224 INFO:teuthology.orchestra.run.smithi099.stdout: "log_data": true, 2023-05-31T20:09:57.224 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket_index_max_shards": 11, 2023-05-31T20:09:57.225 INFO:teuthology.orchestra.run.smithi099.stdout: "read_only": false, 2023-05-31T20:09:57.225 INFO:teuthology.orchestra.run.smithi099.stdout: "tier_type": "", 2023-05-31T20:09:57.225 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_from_all": true, 2023-05-31T20:09:57.225 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_from": [], 2023-05-31T20:09:57.225 INFO:teuthology.orchestra.run.smithi099.stdout: "redirect_zone": "", 2023-05-31T20:09:57.226 INFO:teuthology.orchestra.run.smithi099.stdout: "supported_features": [ 2023-05-31T20:09:57.226 INFO:teuthology.orchestra.run.smithi099.stdout: "resharding" 2023-05-31T20:09:57.226 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2023-05-31T20:09:57.226 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T20:09:57.226 INFO:teuthology.orchestra.run.smithi099.stdout: ], 2023-05-31T20:09:57.226 INFO:teuthology.orchestra.run.smithi099.stdout: "placement_targets": [ 2023-05-31T20:09:57.227 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T20:09:57.227 INFO:teuthology.orchestra.run.smithi099.stdout: "name": "default-placement", 2023-05-31T20:09:57.227 INFO:teuthology.orchestra.run.smithi099.stdout: "tags": [], 2023-05-31T20:09:57.227 INFO:teuthology.orchestra.run.smithi099.stdout: "storage_classes": [ 2023-05-31T20:09:57.228 INFO:teuthology.orchestra.run.smithi099.stdout: "STANDARD" 2023-05-31T20:09:57.228 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2023-05-31T20:09:57.228 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T20:09:57.229 INFO:teuthology.orchestra.run.smithi099.stdout: ], 2023-05-31T20:09:57.229 INFO:teuthology.orchestra.run.smithi099.stdout: "default_placement": "default-placement", 2023-05-31T20:09:57.229 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_id": "aaa33de6-515b-4cac-8f1b-3a1a30452117", 2023-05-31T20:09:57.230 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_policy": { 2023-05-31T20:09:57.230 INFO:teuthology.orchestra.run.smithi099.stdout: "groups": [] 2023-05-31T20:09:57.230 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:09:57.231 INFO:teuthology.orchestra.run.smithi099.stdout: "enabled_features": [ 2023-05-31T20:09:57.231 INFO:teuthology.orchestra.run.smithi099.stdout: "resharding" 2023-05-31T20:09:57.231 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2023-05-31T20:09:57.232 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:09:57.232 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T20:09:57.232 INFO:teuthology.orchestra.run.smithi099.stdout: "id": "b", 2023-05-31T20:09:57.233 INFO:teuthology.orchestra.run.smithi099.stdout: "name": "b", 2023-05-31T20:09:57.233 INFO:teuthology.orchestra.run.smithi099.stdout: "api_name": "", 2023-05-31T20:09:57.234 INFO:teuthology.orchestra.run.smithi099.stdout: "is_master": false, 2023-05-31T20:09:57.234 INFO:teuthology.orchestra.run.smithi099.stdout: "endpoints": [ 2023-05-31T20:09:57.234 INFO:teuthology.orchestra.run.smithi099.stdout: "http://smithi112.front.sepia.ceph.com:8000" 2023-05-31T20:09:57.234 INFO:teuthology.orchestra.run.smithi099.stdout: ], 2023-05-31T20:09:57.234 INFO:teuthology.orchestra.run.smithi099.stdout: "hostnames": [], 2023-05-31T20:09:57.235 INFO:teuthology.orchestra.run.smithi099.stdout: "hostnames_s3website": [], 2023-05-31T20:09:57.235 INFO:teuthology.orchestra.run.smithi099.stdout: "master_zone": "f85d7236-2f27-4988-af26-9c66ab280b1b", 2023-05-31T20:09:57.235 INFO:teuthology.orchestra.run.smithi099.stdout: "zones": [ 2023-05-31T20:09:57.235 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T20:09:57.235 INFO:teuthology.orchestra.run.smithi099.stdout: "id": "a6a3aab5-bc52-44a4-8543-5c5ea6056acd", 2023-05-31T20:09:57.235 INFO:teuthology.orchestra.run.smithi099.stdout: "name": "b2", 2023-05-31T20:09:57.236 INFO:teuthology.orchestra.run.smithi099.stdout: "endpoints": [ 2023-05-31T20:09:57.236 INFO:teuthology.orchestra.run.smithi099.stdout: "http://smithi112.front.sepia.ceph.com:8001" 2023-05-31T20:09:57.236 INFO:teuthology.orchestra.run.smithi099.stdout: ], 2023-05-31T20:09:57.236 INFO:teuthology.orchestra.run.smithi099.stdout: "log_meta": false, 2023-05-31T20:09:57.236 INFO:teuthology.orchestra.run.smithi099.stdout: "log_data": true, 2023-05-31T20:09:57.236 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket_index_max_shards": 11, 2023-05-31T20:09:57.237 INFO:teuthology.orchestra.run.smithi099.stdout: "read_only": false, 2023-05-31T20:09:57.237 INFO:teuthology.orchestra.run.smithi099.stdout: "tier_type": "", 2023-05-31T20:09:57.237 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_from_all": true, 2023-05-31T20:09:57.237 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_from": [], 2023-05-31T20:09:57.237 INFO:teuthology.orchestra.run.smithi099.stdout: "redirect_zone": "", 2023-05-31T20:09:57.237 INFO:teuthology.orchestra.run.smithi099.stdout: "supported_features": [ 2023-05-31T20:09:57.238 INFO:teuthology.orchestra.run.smithi099.stdout: "resharding" 2023-05-31T20:09:57.238 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2023-05-31T20:09:57.238 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:09:57.238 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T20:09:57.238 INFO:teuthology.orchestra.run.smithi099.stdout: "id": "f85d7236-2f27-4988-af26-9c66ab280b1b", 2023-05-31T20:09:57.239 INFO:teuthology.orchestra.run.smithi099.stdout: "name": "b1", 2023-05-31T20:09:57.239 INFO:teuthology.orchestra.run.smithi099.stdout: "endpoints": [ 2023-05-31T20:09:57.240 INFO:teuthology.orchestra.run.smithi099.stdout: "http://smithi112.front.sepia.ceph.com:8000" 2023-05-31T20:09:57.240 INFO:teuthology.orchestra.run.smithi099.stdout: ], 2023-05-31T20:09:57.240 INFO:teuthology.orchestra.run.smithi099.stdout: "log_meta": false, 2023-05-31T20:09:57.240 INFO:teuthology.orchestra.run.smithi099.stdout: "log_data": true, 2023-05-31T20:09:57.240 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket_index_max_shards": 11, 2023-05-31T20:09:57.240 INFO:teuthology.orchestra.run.smithi099.stdout: "read_only": false, 2023-05-31T20:09:57.241 INFO:teuthology.orchestra.run.smithi099.stdout: "tier_type": "", 2023-05-31T20:09:57.241 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_from_all": true, 2023-05-31T20:09:57.241 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_from": [], 2023-05-31T20:09:57.241 INFO:teuthology.orchestra.run.smithi099.stdout: "redirect_zone": "", 2023-05-31T20:09:57.241 INFO:teuthology.orchestra.run.smithi099.stdout: "supported_features": [ 2023-05-31T20:09:57.241 INFO:teuthology.orchestra.run.smithi099.stdout: "resharding" 2023-05-31T20:09:57.241 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2023-05-31T20:09:57.242 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T20:09:57.242 INFO:teuthology.orchestra.run.smithi099.stdout: ], 2023-05-31T20:09:57.242 INFO:teuthology.orchestra.run.smithi099.stdout: "placement_targets": [ 2023-05-31T20:09:57.242 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T20:09:57.242 INFO:teuthology.orchestra.run.smithi099.stdout: "name": "default-placement", 2023-05-31T20:09:57.242 INFO:teuthology.orchestra.run.smithi099.stdout: "tags": [], 2023-05-31T20:09:57.243 INFO:teuthology.orchestra.run.smithi099.stdout: "storage_classes": [ 2023-05-31T20:09:57.243 INFO:teuthology.orchestra.run.smithi099.stdout: "STANDARD" 2023-05-31T20:09:57.243 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2023-05-31T20:09:57.243 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T20:09:57.243 INFO:teuthology.orchestra.run.smithi099.stdout: ], 2023-05-31T20:09:57.243 INFO:teuthology.orchestra.run.smithi099.stdout: "default_placement": "default-placement", 2023-05-31T20:09:57.243 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_id": "aaa33de6-515b-4cac-8f1b-3a1a30452117", 2023-05-31T20:09:57.244 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_policy": { 2023-05-31T20:09:57.244 INFO:teuthology.orchestra.run.smithi099.stdout: "groups": [] 2023-05-31T20:09:57.244 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:09:57.244 INFO:teuthology.orchestra.run.smithi099.stdout: "enabled_features": [ 2023-05-31T20:09:57.244 INFO:teuthology.orchestra.run.smithi099.stdout: "resharding" 2023-05-31T20:09:57.244 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2023-05-31T20:09:57.245 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T20:09:57.245 INFO:teuthology.orchestra.run.smithi099.stdout: ], 2023-05-31T20:09:57.245 INFO:teuthology.orchestra.run.smithi099.stdout: "short_zone_ids": [ 2023-05-31T20:09:57.245 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T20:09:57.245 INFO:teuthology.orchestra.run.smithi099.stdout: "key": "27861e7d-8881-42fd-8069-97f0a22b91b7", 2023-05-31T20:09:57.245 INFO:teuthology.orchestra.run.smithi099.stdout: "val": 3824387031 2023-05-31T20:09:57.245 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:09:57.246 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T20:09:57.246 INFO:teuthology.orchestra.run.smithi099.stdout: "key": "3d163b1e-09c7-47e2-ae87-add4aace3f38", 2023-05-31T20:09:57.246 INFO:teuthology.orchestra.run.smithi099.stdout: "val": 3101833771 2023-05-31T20:09:57.246 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:09:57.246 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T20:09:57.246 INFO:teuthology.orchestra.run.smithi099.stdout: "key": "5d8acec7-7e57-4e56-8b32-e3d6b5036001", 2023-05-31T20:09:57.247 INFO:teuthology.orchestra.run.smithi099.stdout: "val": 4255530623 2023-05-31T20:09:57.247 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:09:57.247 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T20:09:57.247 INFO:teuthology.orchestra.run.smithi099.stdout: "key": "a6a3aab5-bc52-44a4-8543-5c5ea6056acd", 2023-05-31T20:09:57.247 INFO:teuthology.orchestra.run.smithi099.stdout: "val": 3708073710 2023-05-31T20:09:57.247 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:09:57.247 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T20:09:57.248 INFO:teuthology.orchestra.run.smithi099.stdout: "key": "f85d7236-2f27-4988-af26-9c66ab280b1b", 2023-05-31T20:09:57.248 INFO:teuthology.orchestra.run.smithi099.stdout: "val": 233395200 2023-05-31T20:09:57.248 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T20:09:57.248 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2023-05-31T20:09:57.248 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:09:57.248 INFO:teuthology.orchestra.run.smithi099.stdout: "master_zonegroup": "a", 2023-05-31T20:09:57.248 INFO:teuthology.orchestra.run.smithi099.stdout: "master_zone": "3d163b1e-09c7-47e2-ae87-add4aace3f38", 2023-05-31T20:09:57.249 INFO:teuthology.orchestra.run.smithi099.stdout: "period_config": { 2023-05-31T20:09:57.249 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket_quota": { 2023-05-31T20:09:57.249 INFO:teuthology.orchestra.run.smithi099.stdout: "enabled": false, 2023-05-31T20:09:57.249 INFO:teuthology.orchestra.run.smithi099.stdout: "check_on_raw": false, 2023-05-31T20:09:57.249 INFO:teuthology.orchestra.run.smithi099.stdout: "max_size": -1, 2023-05-31T20:09:57.249 INFO:teuthology.orchestra.run.smithi099.stdout: "max_size_kb": 0, 2023-05-31T20:09:57.249 INFO:teuthology.orchestra.run.smithi099.stdout: "max_objects": -1 2023-05-31T20:09:57.250 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:09:57.250 INFO:teuthology.orchestra.run.smithi099.stdout: "user_quota": { 2023-05-31T20:09:57.250 INFO:teuthology.orchestra.run.smithi099.stdout: "enabled": false, 2023-05-31T20:09:57.251 INFO:teuthology.orchestra.run.smithi099.stdout: "check_on_raw": false, 2023-05-31T20:09:57.251 INFO:teuthology.orchestra.run.smithi099.stdout: "max_size": -1, 2023-05-31T20:09:57.251 INFO:teuthology.orchestra.run.smithi099.stdout: "max_size_kb": 0, 2023-05-31T20:09:57.251 INFO:teuthology.orchestra.run.smithi099.stdout: "max_objects": -1 2023-05-31T20:09:57.252 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:09:57.252 INFO:teuthology.orchestra.run.smithi099.stdout: "user_ratelimit": { 2023-05-31T20:09:57.252 INFO:teuthology.orchestra.run.smithi099.stdout: "max_read_ops": 0, 2023-05-31T20:09:57.252 INFO:teuthology.orchestra.run.smithi099.stdout: "max_write_ops": 0, 2023-05-31T20:09:57.252 INFO:teuthology.orchestra.run.smithi099.stdout: "max_read_bytes": 0, 2023-05-31T20:09:57.252 INFO:teuthology.orchestra.run.smithi099.stdout: "max_write_bytes": 0, 2023-05-31T20:09:57.253 INFO:teuthology.orchestra.run.smithi099.stdout: "enabled": false 2023-05-31T20:09:57.253 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:09:57.253 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket_ratelimit": { 2023-05-31T20:09:57.253 INFO:teuthology.orchestra.run.smithi099.stdout: "max_read_ops": 0, 2023-05-31T20:09:57.253 INFO:teuthology.orchestra.run.smithi099.stdout: "max_write_ops": 0, 2023-05-31T20:09:57.253 INFO:teuthology.orchestra.run.smithi099.stdout: "max_read_bytes": 0, 2023-05-31T20:09:57.254 INFO:teuthology.orchestra.run.smithi099.stdout: "max_write_bytes": 0, 2023-05-31T20:09:57.254 INFO:teuthology.orchestra.run.smithi099.stdout: "enabled": false 2023-05-31T20:09:57.254 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:09:57.254 INFO:teuthology.orchestra.run.smithi099.stdout: "anonymous_ratelimit": { 2023-05-31T20:09:57.254 INFO:teuthology.orchestra.run.smithi099.stdout: "max_read_ops": 0, 2023-05-31T20:09:57.254 INFO:teuthology.orchestra.run.smithi099.stdout: "max_write_ops": 0, 2023-05-31T20:09:57.254 INFO:teuthology.orchestra.run.smithi099.stdout: "max_read_bytes": 0, 2023-05-31T20:09:57.255 INFO:teuthology.orchestra.run.smithi099.stdout: "max_write_bytes": 0, 2023-05-31T20:09:57.255 INFO:teuthology.orchestra.run.smithi099.stdout: "enabled": false 2023-05-31T20:09:57.255 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T20:09:57.255 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:09:57.255 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_id": "aaa33de6-515b-4cac-8f1b-3a1a30452117", 2023-05-31T20:09:57.255 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_name": "test-realm", 2023-05-31T20:09:57.255 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2023-05-31T20:09:57.256 INFO:teuthology.orchestra.run.smithi099.stdout:} 2023-05-31T20:09:57.256 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['zone', 'delete', '--rgw-zone', 'remove', '--rgw-zonegroup', 'remove_zg', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-05-31T20:09:57.256 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'zone', 'delete', '--rgw-zone', 'remove', '--rgw-zonegroup', 'remove_zg', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-05-31T20:09:57.257 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 zone delete --rgw-zone remove --rgw-zonegroup remove_zg --cluster c1 --debug-rgw 0 --debug-ms 0 2023-05-31T20:09:57.307 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2023-05-31T20:09:57.307 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2023-05-31T20:09:57.333 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:09:57.327+0000 7fcb11a4b780 0 NOTICE: removed master_zone of zonegroup remove_zg 2023-05-31T20:09:57.334 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:09:57.328+0000 7fcb11a4b780 0 Removed zone from zonegroup remove_zg 2023-05-31T20:09:57.347 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['period', 'update', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-05-31T20:09:57.347 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'period', 'update', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-05-31T20:09:57.348 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 period update --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 2023-05-31T20:09:57.401 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2023-05-31T20:09:57.402 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2023-05-31T20:09:57.429 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:09:57.423+0000 7f49092dd780 0 ERROR: zonegroup remove_zg has a non existent master zone 2023-05-31T20:09:57.434 DEBUG:teuthology.orchestra.run:got remote process result: 22 2023-05-31T20:09:57.435 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['zonegroup', 'delete', '--rgw-zonegroup', 'remove_zg', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-05-31T20:09:57.435 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'zonegroup', 'delete', '--rgw-zonegroup', 'remove_zg', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-05-31T20:09:57.436 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 zonegroup delete --rgw-zonegroup remove_zg --cluster c1 --debug-rgw 0 --debug-ms 0 2023-05-31T20:09:57.488 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2023-05-31T20:09:57.489 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2023-05-31T20:09:57.521 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['period', 'update', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-05-31T20:09:57.521 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'period', 'update', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-05-31T20:09:57.522 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 period update --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --commit --cluster c1 --debug-rgw 0 --debug-ms 0 2023-05-31T20:09:57.576 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2023-05-31T20:09:57.576 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2023-05-31T20:09:57.749 INFO:teuthology.orchestra.run.smithi099.stdout:{ 2023-05-31T20:09:57.749 INFO:teuthology.orchestra.run.smithi099.stdout: "id": "2e6123d2-d695-4350-9a01-61a455bc782b", 2023-05-31T20:09:57.750 INFO:teuthology.orchestra.run.smithi099.stdout: "epoch": 12, 2023-05-31T20:09:57.750 INFO:teuthology.orchestra.run.smithi099.stdout: "predecessor_uuid": "629b9ced-80af-41ed-8a6c-f2cfc334cbb4", 2023-05-31T20:09:57.750 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_status": [], 2023-05-31T20:09:57.751 INFO:teuthology.orchestra.run.smithi099.stdout: "period_map": { 2023-05-31T20:09:57.751 INFO:teuthology.orchestra.run.smithi099.stdout: "id": "2e6123d2-d695-4350-9a01-61a455bc782b", 2023-05-31T20:09:57.751 INFO:teuthology.orchestra.run.smithi099.stdout: "zonegroups": [ 2023-05-31T20:09:57.752 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T20:09:57.752 INFO:teuthology.orchestra.run.smithi099.stdout: "id": "a", 2023-05-31T20:09:57.752 INFO:teuthology.orchestra.run.smithi099.stdout: "name": "a", 2023-05-31T20:09:57.753 INFO:teuthology.orchestra.run.smithi099.stdout: "api_name": "", 2023-05-31T20:09:57.753 INFO:teuthology.orchestra.run.smithi099.stdout: "is_master": true, 2023-05-31T20:09:57.753 INFO:teuthology.orchestra.run.smithi099.stdout: "endpoints": [ 2023-05-31T20:09:57.753 INFO:teuthology.orchestra.run.smithi099.stdout: "http://smithi099.front.sepia.ceph.com:8000" 2023-05-31T20:09:57.754 INFO:teuthology.orchestra.run.smithi099.stdout: ], 2023-05-31T20:09:57.754 INFO:teuthology.orchestra.run.smithi099.stdout: "hostnames": [], 2023-05-31T20:09:57.754 INFO:teuthology.orchestra.run.smithi099.stdout: "hostnames_s3website": [], 2023-05-31T20:09:57.755 INFO:teuthology.orchestra.run.smithi099.stdout: "master_zone": "3d163b1e-09c7-47e2-ae87-add4aace3f38", 2023-05-31T20:09:57.755 INFO:teuthology.orchestra.run.smithi099.stdout: "zones": [ 2023-05-31T20:09:57.755 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T20:09:57.756 INFO:teuthology.orchestra.run.smithi099.stdout: "id": "3d163b1e-09c7-47e2-ae87-add4aace3f38", 2023-05-31T20:09:57.757 INFO:teuthology.orchestra.run.smithi099.stdout: "name": "a1", 2023-05-31T20:09:57.757 INFO:teuthology.orchestra.run.smithi099.stdout: "endpoints": [ 2023-05-31T20:09:57.757 INFO:teuthology.orchestra.run.smithi099.stdout: "http://smithi099.front.sepia.ceph.com:8000" 2023-05-31T20:09:57.757 INFO:teuthology.orchestra.run.smithi099.stdout: ], 2023-05-31T20:09:57.757 INFO:teuthology.orchestra.run.smithi099.stdout: "log_meta": false, 2023-05-31T20:09:57.758 INFO:teuthology.orchestra.run.smithi099.stdout: "log_data": true, 2023-05-31T20:09:57.758 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket_index_max_shards": 11, 2023-05-31T20:09:57.759 INFO:teuthology.orchestra.run.smithi099.stdout: "read_only": false, 2023-05-31T20:09:57.759 INFO:teuthology.orchestra.run.smithi099.stdout: "tier_type": "", 2023-05-31T20:09:57.759 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_from_all": true, 2023-05-31T20:09:57.759 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_from": [], 2023-05-31T20:09:57.759 INFO:teuthology.orchestra.run.smithi099.stdout: "redirect_zone": "", 2023-05-31T20:09:57.759 INFO:teuthology.orchestra.run.smithi099.stdout: "supported_features": [ 2023-05-31T20:09:57.760 INFO:teuthology.orchestra.run.smithi099.stdout: "resharding" 2023-05-31T20:09:57.760 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2023-05-31T20:09:57.760 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:09:57.760 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T20:09:57.761 INFO:teuthology.orchestra.run.smithi099.stdout: "id": "5d8acec7-7e57-4e56-8b32-e3d6b5036001", 2023-05-31T20:09:57.761 INFO:teuthology.orchestra.run.smithi099.stdout: "name": "a2", 2023-05-31T20:09:57.761 INFO:teuthology.orchestra.run.smithi099.stdout: "endpoints": [ 2023-05-31T20:09:57.761 INFO:teuthology.orchestra.run.smithi099.stdout: "http://smithi099.front.sepia.ceph.com:8001" 2023-05-31T20:09:57.761 INFO:teuthology.orchestra.run.smithi099.stdout: ], 2023-05-31T20:09:57.762 INFO:teuthology.orchestra.run.smithi099.stdout: "log_meta": false, 2023-05-31T20:09:57.762 INFO:teuthology.orchestra.run.smithi099.stdout: "log_data": true, 2023-05-31T20:09:57.762 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket_index_max_shards": 11, 2023-05-31T20:09:57.762 INFO:teuthology.orchestra.run.smithi099.stdout: "read_only": false, 2023-05-31T20:09:57.762 INFO:teuthology.orchestra.run.smithi099.stdout: "tier_type": "", 2023-05-31T20:09:57.762 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_from_all": true, 2023-05-31T20:09:57.762 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_from": [], 2023-05-31T20:09:57.763 INFO:teuthology.orchestra.run.smithi099.stdout: "redirect_zone": "", 2023-05-31T20:09:57.763 INFO:teuthology.orchestra.run.smithi099.stdout: "supported_features": [ 2023-05-31T20:09:57.763 INFO:teuthology.orchestra.run.smithi099.stdout: "resharding" 2023-05-31T20:09:57.763 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2023-05-31T20:09:57.764 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T20:09:57.764 INFO:teuthology.orchestra.run.smithi099.stdout: ], 2023-05-31T20:09:57.764 INFO:teuthology.orchestra.run.smithi099.stdout: "placement_targets": [ 2023-05-31T20:09:57.765 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T20:09:57.765 INFO:teuthology.orchestra.run.smithi099.stdout: "name": "default-placement", 2023-05-31T20:09:57.765 INFO:teuthology.orchestra.run.smithi099.stdout: "tags": [], 2023-05-31T20:09:57.765 INFO:teuthology.orchestra.run.smithi099.stdout: "storage_classes": [ 2023-05-31T20:09:57.766 INFO:teuthology.orchestra.run.smithi099.stdout: "STANDARD" 2023-05-31T20:09:57.766 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2023-05-31T20:09:57.767 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T20:09:57.767 INFO:teuthology.orchestra.run.smithi099.stdout: ], 2023-05-31T20:09:57.767 INFO:teuthology.orchestra.run.smithi099.stdout: "default_placement": "default-placement", 2023-05-31T20:09:57.768 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_id": "aaa33de6-515b-4cac-8f1b-3a1a30452117", 2023-05-31T20:09:57.768 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_policy": { 2023-05-31T20:09:57.768 INFO:teuthology.orchestra.run.smithi099.stdout: "groups": [] 2023-05-31T20:09:57.769 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:09:57.769 INFO:teuthology.orchestra.run.smithi099.stdout: "enabled_features": [ 2023-05-31T20:09:57.769 INFO:teuthology.orchestra.run.smithi099.stdout: "resharding" 2023-05-31T20:09:57.770 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2023-05-31T20:09:57.770 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:09:57.770 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T20:09:57.771 INFO:teuthology.orchestra.run.smithi099.stdout: "id": "b", 2023-05-31T20:09:57.771 INFO:teuthology.orchestra.run.smithi099.stdout: "name": "b", 2023-05-31T20:09:57.771 INFO:teuthology.orchestra.run.smithi099.stdout: "api_name": "", 2023-05-31T20:09:57.772 INFO:teuthology.orchestra.run.smithi099.stdout: "is_master": false, 2023-05-31T20:09:57.772 INFO:teuthology.orchestra.run.smithi099.stdout: "endpoints": [ 2023-05-31T20:09:57.772 INFO:teuthology.orchestra.run.smithi099.stdout: "http://smithi112.front.sepia.ceph.com:8000" 2023-05-31T20:09:57.772 INFO:teuthology.orchestra.run.smithi099.stdout: ], 2023-05-31T20:09:57.773 INFO:teuthology.orchestra.run.smithi099.stdout: "hostnames": [], 2023-05-31T20:09:57.773 INFO:teuthology.orchestra.run.smithi099.stdout: "hostnames_s3website": [], 2023-05-31T20:09:57.773 INFO:teuthology.orchestra.run.smithi099.stdout: "master_zone": "f85d7236-2f27-4988-af26-9c66ab280b1b", 2023-05-31T20:09:57.774 INFO:teuthology.orchestra.run.smithi099.stdout: "zones": [ 2023-05-31T20:09:57.774 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T20:09:57.774 INFO:teuthology.orchestra.run.smithi099.stdout: "id": "a6a3aab5-bc52-44a4-8543-5c5ea6056acd", 2023-05-31T20:09:57.775 INFO:teuthology.orchestra.run.smithi099.stdout: "name": "b2", 2023-05-31T20:09:57.775 INFO:teuthology.orchestra.run.smithi099.stdout: "endpoints": [ 2023-05-31T20:09:57.775 INFO:teuthology.orchestra.run.smithi099.stdout: "http://smithi112.front.sepia.ceph.com:8001" 2023-05-31T20:09:57.775 INFO:teuthology.orchestra.run.smithi099.stdout: ], 2023-05-31T20:09:57.776 INFO:teuthology.orchestra.run.smithi099.stdout: "log_meta": false, 2023-05-31T20:09:57.776 INFO:teuthology.orchestra.run.smithi099.stdout: "log_data": true, 2023-05-31T20:09:57.776 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket_index_max_shards": 11, 2023-05-31T20:09:57.777 INFO:teuthology.orchestra.run.smithi099.stdout: "read_only": false, 2023-05-31T20:09:57.778 INFO:teuthology.orchestra.run.smithi099.stdout: "tier_type": "", 2023-05-31T20:09:57.778 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_from_all": true, 2023-05-31T20:09:57.778 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_from": [], 2023-05-31T20:09:57.779 INFO:teuthology.orchestra.run.smithi099.stdout: "redirect_zone": "", 2023-05-31T20:09:57.779 INFO:teuthology.orchestra.run.smithi099.stdout: "supported_features": [ 2023-05-31T20:09:57.779 INFO:teuthology.orchestra.run.smithi099.stdout: "resharding" 2023-05-31T20:09:57.779 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2023-05-31T20:09:57.780 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:09:57.780 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T20:09:57.789 INFO:teuthology.orchestra.run.smithi099.stdout: "id": "f85d7236-2f27-4988-af26-9c66ab280b1b", 2023-05-31T20:09:57.789 INFO:teuthology.orchestra.run.smithi099.stdout: "name": "b1", 2023-05-31T20:09:57.790 INFO:teuthology.orchestra.run.smithi099.stdout: "endpoints": [ 2023-05-31T20:09:57.790 INFO:teuthology.orchestra.run.smithi099.stdout: "http://smithi112.front.sepia.ceph.com:8000" 2023-05-31T20:09:57.790 INFO:teuthology.orchestra.run.smithi099.stdout: ], 2023-05-31T20:09:57.790 INFO:teuthology.orchestra.run.smithi099.stdout: "log_meta": false, 2023-05-31T20:09:57.790 INFO:teuthology.orchestra.run.smithi099.stdout: "log_data": true, 2023-05-31T20:09:57.790 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket_index_max_shards": 11, 2023-05-31T20:09:57.791 INFO:teuthology.orchestra.run.smithi099.stdout: "read_only": false, 2023-05-31T20:09:57.791 INFO:teuthology.orchestra.run.smithi099.stdout: "tier_type": "", 2023-05-31T20:09:57.791 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_from_all": true, 2023-05-31T20:09:57.791 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_from": [], 2023-05-31T20:09:57.791 INFO:teuthology.orchestra.run.smithi099.stdout: "redirect_zone": "", 2023-05-31T20:09:57.791 INFO:teuthology.orchestra.run.smithi099.stdout: "supported_features": [ 2023-05-31T20:09:57.791 INFO:teuthology.orchestra.run.smithi099.stdout: "resharding" 2023-05-31T20:09:57.792 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2023-05-31T20:09:57.792 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T20:09:57.792 INFO:teuthology.orchestra.run.smithi099.stdout: ], 2023-05-31T20:09:57.792 INFO:teuthology.orchestra.run.smithi099.stdout: "placement_targets": [ 2023-05-31T20:09:57.792 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T20:09:57.792 INFO:teuthology.orchestra.run.smithi099.stdout: "name": "default-placement", 2023-05-31T20:09:57.792 INFO:teuthology.orchestra.run.smithi099.stdout: "tags": [], 2023-05-31T20:09:57.793 INFO:teuthology.orchestra.run.smithi099.stdout: "storage_classes": [ 2023-05-31T20:09:57.793 INFO:teuthology.orchestra.run.smithi099.stdout: "STANDARD" 2023-05-31T20:09:57.793 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2023-05-31T20:09:57.793 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T20:09:57.793 INFO:teuthology.orchestra.run.smithi099.stdout: ], 2023-05-31T20:09:57.794 INFO:teuthology.orchestra.run.smithi099.stdout: "default_placement": "default-placement", 2023-05-31T20:09:57.794 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_id": "aaa33de6-515b-4cac-8f1b-3a1a30452117", 2023-05-31T20:09:57.794 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_policy": { 2023-05-31T20:09:57.794 INFO:teuthology.orchestra.run.smithi099.stdout: "groups": [] 2023-05-31T20:09:57.794 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:09:57.795 INFO:teuthology.orchestra.run.smithi099.stdout: "enabled_features": [ 2023-05-31T20:09:57.795 INFO:teuthology.orchestra.run.smithi099.stdout: "resharding" 2023-05-31T20:09:57.795 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2023-05-31T20:09:57.795 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T20:09:57.795 INFO:teuthology.orchestra.run.smithi099.stdout: ], 2023-05-31T20:09:57.795 INFO:teuthology.orchestra.run.smithi099.stdout: "short_zone_ids": [ 2023-05-31T20:09:57.796 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T20:09:57.796 INFO:teuthology.orchestra.run.smithi099.stdout: "key": "3d163b1e-09c7-47e2-ae87-add4aace3f38", 2023-05-31T20:09:57.796 INFO:teuthology.orchestra.run.smithi099.stdout: "val": 3101833771 2023-05-31T20:09:57.796 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:09:57.796 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T20:09:57.796 INFO:teuthology.orchestra.run.smithi099.stdout: "key": "5d8acec7-7e57-4e56-8b32-e3d6b5036001", 2023-05-31T20:09:57.796 INFO:teuthology.orchestra.run.smithi099.stdout: "val": 4255530623 2023-05-31T20:09:57.797 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:09:57.797 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T20:09:57.797 INFO:teuthology.orchestra.run.smithi099.stdout: "key": "a6a3aab5-bc52-44a4-8543-5c5ea6056acd", 2023-05-31T20:09:57.797 INFO:teuthology.orchestra.run.smithi099.stdout: "val": 3708073710 2023-05-31T20:09:57.797 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:09:57.797 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T20:09:57.798 INFO:teuthology.orchestra.run.smithi099.stdout: "key": "f85d7236-2f27-4988-af26-9c66ab280b1b", 2023-05-31T20:09:57.798 INFO:teuthology.orchestra.run.smithi099.stdout: "val": 233395200 2023-05-31T20:09:57.799 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T20:09:57.799 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2023-05-31T20:09:57.799 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:09:57.799 INFO:teuthology.orchestra.run.smithi099.stdout: "master_zonegroup": "a", 2023-05-31T20:09:57.799 INFO:teuthology.orchestra.run.smithi099.stdout: "master_zone": "3d163b1e-09c7-47e2-ae87-add4aace3f38", 2023-05-31T20:09:57.799 INFO:teuthology.orchestra.run.smithi099.stdout: "period_config": { 2023-05-31T20:09:57.799 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket_quota": { 2023-05-31T20:09:57.800 INFO:teuthology.orchestra.run.smithi099.stdout: "enabled": false, 2023-05-31T20:09:57.800 INFO:teuthology.orchestra.run.smithi099.stdout: "check_on_raw": false, 2023-05-31T20:09:57.800 INFO:teuthology.orchestra.run.smithi099.stdout: "max_size": -1, 2023-05-31T20:09:57.800 INFO:teuthology.orchestra.run.smithi099.stdout: "max_size_kb": 0, 2023-05-31T20:09:57.800 INFO:teuthology.orchestra.run.smithi099.stdout: "max_objects": -1 2023-05-31T20:09:57.800 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:09:57.801 INFO:teuthology.orchestra.run.smithi099.stdout: "user_quota": { 2023-05-31T20:09:57.801 INFO:teuthology.orchestra.run.smithi099.stdout: "enabled": false, 2023-05-31T20:09:57.801 INFO:teuthology.orchestra.run.smithi099.stdout: "check_on_raw": false, 2023-05-31T20:09:57.801 INFO:teuthology.orchestra.run.smithi099.stdout: "max_size": -1, 2023-05-31T20:09:57.801 INFO:teuthology.orchestra.run.smithi099.stdout: "max_size_kb": 0, 2023-05-31T20:09:57.801 INFO:teuthology.orchestra.run.smithi099.stdout: "max_objects": -1 2023-05-31T20:09:57.801 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:09:57.802 INFO:teuthology.orchestra.run.smithi099.stdout: "user_ratelimit": { 2023-05-31T20:09:57.802 INFO:teuthology.orchestra.run.smithi099.stdout: "max_read_ops": 0, 2023-05-31T20:09:57.802 INFO:teuthology.orchestra.run.smithi099.stdout: "max_write_ops": 0, 2023-05-31T20:09:57.802 INFO:teuthology.orchestra.run.smithi099.stdout: "max_read_bytes": 0, 2023-05-31T20:09:57.802 INFO:teuthology.orchestra.run.smithi099.stdout: "max_write_bytes": 0, 2023-05-31T20:09:57.802 INFO:teuthology.orchestra.run.smithi099.stdout: "enabled": false 2023-05-31T20:09:57.802 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:09:57.803 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket_ratelimit": { 2023-05-31T20:09:57.803 INFO:teuthology.orchestra.run.smithi099.stdout: "max_read_ops": 0, 2023-05-31T20:09:57.803 INFO:teuthology.orchestra.run.smithi099.stdout: "max_write_ops": 0, 2023-05-31T20:09:57.803 INFO:teuthology.orchestra.run.smithi099.stdout: "max_read_bytes": 0, 2023-05-31T20:09:57.803 INFO:teuthology.orchestra.run.smithi099.stdout: "max_write_bytes": 0, 2023-05-31T20:09:57.803 INFO:teuthology.orchestra.run.smithi099.stdout: "enabled": false 2023-05-31T20:09:57.803 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:09:57.804 INFO:teuthology.orchestra.run.smithi099.stdout: "anonymous_ratelimit": { 2023-05-31T20:09:57.804 INFO:teuthology.orchestra.run.smithi099.stdout: "max_read_ops": 0, 2023-05-31T20:09:57.804 INFO:teuthology.orchestra.run.smithi099.stdout: "max_write_ops": 0, 2023-05-31T20:09:57.804 INFO:teuthology.orchestra.run.smithi099.stdout: "max_read_bytes": 0, 2023-05-31T20:09:57.804 INFO:teuthology.orchestra.run.smithi099.stdout: "max_write_bytes": 0, 2023-05-31T20:09:57.804 INFO:teuthology.orchestra.run.smithi099.stdout: "enabled": false 2023-05-31T20:09:57.805 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T20:09:57.805 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:09:57.805 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_id": "aaa33de6-515b-4cac-8f1b-3a1a30452117", 2023-05-31T20:09:57.805 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_name": "test-realm", 2023-05-31T20:09:57.805 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2023-05-31T20:09:57.805 INFO:teuthology.orchestra.run.smithi099.stdout:} 2023-05-31T20:09:57.806 INFO:tasks.rgw_multisite_tests:rgw_multi.tests.test_zg_master_zone_delete ... ok 2023-05-31T20:09:57.807 INFO:rgw_multi.tests:create bucket zone=a1 name=pklxkd-36 2023-05-31T20:09:57.808 DEBUG:boto:path=/pklxkd-36/ 2023-05-31T20:09:57.808 DEBUG:boto:auth_path=/pklxkd-36/ 2023-05-31T20:09:57.808 DEBUG:boto:Method: PUT 2023-05-31T20:09:57.809 DEBUG:boto:Path: /pklxkd-36/ 2023-05-31T20:09:57.809 DEBUG:boto:Data: 2023-05-31T20:09:57.809 DEBUG:boto:Headers: {} 2023-05-31T20:09:57.810 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:09:57.810 DEBUG:boto:Port: 8000 2023-05-31T20:09:57.810 DEBUG:boto:Params: {} 2023-05-31T20:09:57.811 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2023-05-31T20:09:57.811 DEBUG:boto:Token: None 2023-05-31T20:09:57.812 DEBUG:boto:StringToSign: PUT Wed, 31 May 2023 20:09:57 GMT /pklxkd-36/ 2023-05-31T20:09:57.812 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:Ura+AgF4z544T9JM72zpKvHy/CQ= 2023-05-31T20:09:57.813 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:09:57 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:Ura+AgF4z544T9JM72zpKvHy/CQ=', 'Content-Length': '0'} 2023-05-31T20:09:58.790 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx0000011ada90b79b4139b-006477a996-1630-a1'), ('Content-Length', '0'), ('Date', 'Wed, 31 May 2023 20:09:58 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:09:58.791 INFO:rgw_multi.tests:create bucket zone=a2 name=pklxkd-37 2023-05-31T20:09:58.791 DEBUG:boto:path=/pklxkd-37/ 2023-05-31T20:09:58.792 DEBUG:boto:auth_path=/pklxkd-37/ 2023-05-31T20:09:58.792 DEBUG:boto:Method: PUT 2023-05-31T20:09:58.792 DEBUG:boto:Path: /pklxkd-37/ 2023-05-31T20:09:58.793 DEBUG:boto:Data: 2023-05-31T20:09:58.793 DEBUG:boto:Headers: {} 2023-05-31T20:09:58.793 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:09:58.794 DEBUG:boto:Port: 8001 2023-05-31T20:09:58.794 DEBUG:boto:Params: {} 2023-05-31T20:09:58.795 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8001} 2023-05-31T20:09:58.795 DEBUG:boto:Token: None 2023-05-31T20:09:58.795 DEBUG:boto:StringToSign: PUT Wed, 31 May 2023 20:09:58 GMT /pklxkd-37/ 2023-05-31T20:09:58.796 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:eNi54zGACKudLi3zX6uh5M5zs3Y= 2023-05-31T20:09:58.796 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:09:58 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:eNi54zGACKudLi3zX6uh5M5zs3Y=', 'Content-Length': '0'} 2023-05-31T20:09:58.865 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx000000f8609e493cf1b1f-006477a996-162e-a2'), ('Content-Length', '0'), ('Date', 'Wed, 31 May 2023 20:09:58 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:09:58.866 INFO:rgw_multi.tests:create bucket zone=b1 name=pklxkd-38 2023-05-31T20:09:58.867 DEBUG:boto:path=/pklxkd-38/ 2023-05-31T20:09:58.867 DEBUG:boto:auth_path=/pklxkd-38/ 2023-05-31T20:09:58.867 DEBUG:boto:Method: PUT 2023-05-31T20:09:58.868 DEBUG:boto:Path: /pklxkd-38/ 2023-05-31T20:09:58.868 DEBUG:boto:Data: 2023-05-31T20:09:58.869 DEBUG:boto:Headers: {} 2023-05-31T20:09:58.869 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-05-31T20:09:58.869 DEBUG:boto:Port: 8000 2023-05-31T20:09:58.870 DEBUG:boto:Params: {} 2023-05-31T20:09:58.870 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2023-05-31T20:09:58.871 DEBUG:boto:Token: None 2023-05-31T20:09:58.871 DEBUG:boto:StringToSign: PUT Wed, 31 May 2023 20:09:58 GMT /pklxkd-38/ 2023-05-31T20:09:58.871 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:NZ+8XwdGHE1N2jEf1lUTPlkxXTo= 2023-05-31T20:09:58.872 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:09:58 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:NZ+8XwdGHE1N2jEf1lUTPlkxXTo=', 'Content-Length': '0'} 2023-05-31T20:09:58.951 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx0000028146f258ac8fdbf-006477a996-115b-b1'), ('Content-Length', '0'), ('Date', 'Wed, 31 May 2023 20:09:58 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:09:58.952 INFO:rgw_multi.tests:create bucket zone=b2 name=pklxkd-39 2023-05-31T20:09:58.952 DEBUG:boto:path=/pklxkd-39/ 2023-05-31T20:09:58.952 DEBUG:boto:auth_path=/pklxkd-39/ 2023-05-31T20:09:58.953 DEBUG:boto:Method: PUT 2023-05-31T20:09:58.953 DEBUG:boto:Path: /pklxkd-39/ 2023-05-31T20:09:58.953 DEBUG:boto:Data: 2023-05-31T20:09:58.954 DEBUG:boto:Headers: {} 2023-05-31T20:09:58.954 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-05-31T20:09:58.954 DEBUG:boto:Port: 8001 2023-05-31T20:09:58.955 DEBUG:boto:Params: {} 2023-05-31T20:09:58.955 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8001} 2023-05-31T20:09:58.956 DEBUG:boto:Token: None 2023-05-31T20:09:58.956 DEBUG:boto:StringToSign: PUT Wed, 31 May 2023 20:09:58 GMT /pklxkd-39/ 2023-05-31T20:09:58.957 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:V2lQjvGzD9V+P2wWx2kKHzhpJDQ= 2023-05-31T20:09:58.957 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:09:58 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:V2lQjvGzD9V+P2wWx2kKHzhpJDQ=', 'Content-Length': '0'} 2023-05-31T20:09:59.031 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000414e82cd8da2417c-006477a996-1157-b2'), ('Content-Length', '0'), ('Date', 'Wed, 31 May 2023 20:09:59 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:09:59.032 DEBUG:boto:path=/pklxkd-36/ 2023-05-31T20:09:59.032 DEBUG:boto:auth_path=/pklxkd-36/ 2023-05-31T20:09:59.032 DEBUG:boto:path=/pklxkd-36/?website 2023-05-31T20:09:59.033 DEBUG:boto:auth_path=/pklxkd-36/?website 2023-05-31T20:09:59.033 DEBUG:boto:Method: PUT 2023-05-31T20:09:59.033 DEBUG:boto:Path: /pklxkd-36/?website 2023-05-31T20:09:59.034 DEBUG:boto:Data: index.htmlerror.html 2023-05-31T20:09:59.034 DEBUG:boto:Headers: {} 2023-05-31T20:09:59.034 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:09:59.035 DEBUG:boto:Port: 8000 2023-05-31T20:09:59.035 DEBUG:boto:Params: {} 2023-05-31T20:09:59.036 DEBUG:boto:Token: None 2023-05-31T20:09:59.036 DEBUG:boto:StringToSign: PUT Wed, 31 May 2023 20:09:59 GMT /pklxkd-36/?website 2023-05-31T20:09:59.036 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:DDMHkJF3oVkKUd+pIWMEoyKj/qE= 2023-05-31T20:09:59.037 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:09:59 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:DDMHkJF3oVkKUd+pIWMEoyKj/qE=', 'Content-Length': '241'} 2023-05-31T20:09:59.038 DEBUG:boto:Response headers: [('Content-Length', '192'), ('x-amz-request-id', 'tx0000014af5e5ab7fa2e2c-006477a997-1630-a1'), ('Accept-Ranges', 'bytes'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:09:59 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:09:59.040 INFO:tasks.rgw_multisite_tests:rgw_multi.tests.test_set_bucket_website ... SKIP: test_set_bucket_website skipped. Requires rgw_enable_static_website = 1. 2023-05-31T20:09:59.042 INFO:rgw_multi.tests:create bucket zone=a1 name=pklxkd-40 2023-05-31T20:09:59.043 DEBUG:boto:path=/pklxkd-40/ 2023-05-31T20:09:59.043 DEBUG:boto:auth_path=/pklxkd-40/ 2023-05-31T20:09:59.044 DEBUG:boto:Method: PUT 2023-05-31T20:09:59.044 DEBUG:boto:Path: /pklxkd-40/ 2023-05-31T20:09:59.044 DEBUG:boto:Data: 2023-05-31T20:09:59.045 DEBUG:boto:Headers: {} 2023-05-31T20:09:59.045 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:09:59.045 DEBUG:boto:Port: 8000 2023-05-31T20:09:59.046 DEBUG:boto:Params: {} 2023-05-31T20:09:59.046 DEBUG:boto:Token: None 2023-05-31T20:09:59.047 DEBUG:boto:StringToSign: PUT Wed, 31 May 2023 20:09:59 GMT /pklxkd-40/ 2023-05-31T20:09:59.047 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:u0NN+GxYzMy1r72/i6lXzqK5Rhg= 2023-05-31T20:09:59.047 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:09:59 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:u0NN+GxYzMy1r72/i6lXzqK5Rhg=', 'Content-Length': '0'} 2023-05-31T20:09:59.086 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx000006f954d40a7b210d7-006477a997-1630-a1'), ('Content-Length', '0'), ('Date', 'Wed, 31 May 2023 20:09:59 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:09:59.086 INFO:rgw_multi.tests:create bucket zone=a2 name=pklxkd-41 2023-05-31T20:09:59.087 DEBUG:boto:path=/pklxkd-41/ 2023-05-31T20:09:59.087 DEBUG:boto:auth_path=/pklxkd-41/ 2023-05-31T20:09:59.087 DEBUG:boto:Method: PUT 2023-05-31T20:09:59.088 DEBUG:boto:Path: /pklxkd-41/ 2023-05-31T20:09:59.088 DEBUG:boto:Data: 2023-05-31T20:09:59.088 DEBUG:boto:Headers: {} 2023-05-31T20:09:59.089 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:09:59.089 DEBUG:boto:Port: 8001 2023-05-31T20:09:59.089 DEBUG:boto:Params: {} 2023-05-31T20:09:59.090 DEBUG:boto:Token: None 2023-05-31T20:09:59.090 DEBUG:boto:StringToSign: PUT Wed, 31 May 2023 20:09:59 GMT /pklxkd-41/ 2023-05-31T20:09:59.091 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:wVuApNbl6REEqQpY68fAbDPV/54= 2023-05-31T20:09:59.091 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:09:59 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:wVuApNbl6REEqQpY68fAbDPV/54=', 'Content-Length': '0'} 2023-05-31T20:09:59.142 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx000005a0e506f5b4db1b3-006477a997-162e-a2'), ('Content-Length', '0'), ('Date', 'Wed, 31 May 2023 20:09:59 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:09:59.143 INFO:rgw_multi.tests:create bucket zone=b1 name=pklxkd-42 2023-05-31T20:09:59.144 DEBUG:boto:path=/pklxkd-42/ 2023-05-31T20:09:59.144 DEBUG:boto:auth_path=/pklxkd-42/ 2023-05-31T20:09:59.144 DEBUG:boto:Method: PUT 2023-05-31T20:09:59.145 DEBUG:boto:Path: /pklxkd-42/ 2023-05-31T20:09:59.145 DEBUG:boto:Data: 2023-05-31T20:09:59.145 DEBUG:boto:Headers: {} 2023-05-31T20:09:59.146 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-05-31T20:09:59.146 DEBUG:boto:Port: 8000 2023-05-31T20:09:59.146 DEBUG:boto:Params: {} 2023-05-31T20:09:59.147 DEBUG:boto:Token: None 2023-05-31T20:09:59.147 DEBUG:boto:StringToSign: PUT Wed, 31 May 2023 20:09:59 GMT /pklxkd-42/ 2023-05-31T20:09:59.148 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:Zp1dJDZ+MK9QHUqM8ejqWe8oC8g= 2023-05-31T20:09:59.148 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:09:59 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:Zp1dJDZ+MK9QHUqM8ejqWe8oC8g=', 'Content-Length': '0'} 2023-05-31T20:09:59.232 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000e3a4e8d430d384eb-006477a997-115b-b1'), ('Content-Length', '0'), ('Date', 'Wed, 31 May 2023 20:09:59 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:09:59.233 INFO:rgw_multi.tests:create bucket zone=b2 name=pklxkd-43 2023-05-31T20:09:59.233 DEBUG:boto:path=/pklxkd-43/ 2023-05-31T20:09:59.234 DEBUG:boto:auth_path=/pklxkd-43/ 2023-05-31T20:09:59.234 DEBUG:boto:Method: PUT 2023-05-31T20:09:59.234 DEBUG:boto:Path: /pklxkd-43/ 2023-05-31T20:09:59.235 DEBUG:boto:Data: 2023-05-31T20:09:59.235 DEBUG:boto:Headers: {} 2023-05-31T20:09:59.235 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-05-31T20:09:59.236 DEBUG:boto:Port: 8001 2023-05-31T20:09:59.236 DEBUG:boto:Params: {} 2023-05-31T20:09:59.237 DEBUG:boto:Token: None 2023-05-31T20:09:59.237 DEBUG:boto:StringToSign: PUT Wed, 31 May 2023 20:09:59 GMT /pklxkd-43/ 2023-05-31T20:09:59.238 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:Qrpa4u+kc35bnomUduk0VNmKbJw= 2023-05-31T20:09:59.238 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:09:59 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:Qrpa4u+kc35bnomUduk0VNmKbJw=', 'Content-Length': '0'} 2023-05-31T20:09:59.290 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx0000039c1a0db4749907d-006477a997-1157-b2'), ('Content-Length', '0'), ('Date', 'Wed, 31 May 2023 20:09:59 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:09:59.291 DEBUG:boto:path=/pklxkd-40/ 2023-05-31T20:09:59.291 DEBUG:boto:auth_path=/pklxkd-40/ 2023-05-31T20:09:59.291 DEBUG:boto:path=/pklxkd-40/?policy 2023-05-31T20:09:59.292 DEBUG:boto:auth_path=/pklxkd-40/?policy 2023-05-31T20:09:59.292 DEBUG:boto:Method: PUT 2023-05-31T20:09:59.292 DEBUG:boto:Path: /pklxkd-40/?policy 2023-05-31T20:09:59.293 DEBUG:boto:Data: { "Version": "2012-10-17", "Statement": [{ "Effect": "Allow", "Principal": "*" }] } 2023-05-31T20:09:59.293 DEBUG:boto:Headers: {} 2023-05-31T20:09:59.294 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:09:59.294 DEBUG:boto:Port: 8000 2023-05-31T20:09:59.295 DEBUG:boto:Params: {} 2023-05-31T20:09:59.295 DEBUG:boto:Token: None 2023-05-31T20:09:59.296 DEBUG:boto:StringToSign: PUT Wed, 31 May 2023 20:09:59 GMT /pklxkd-40/?policy 2023-05-31T20:09:59.297 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:U7kzjb/bvaq1XSbrsygA0c5z9/E= 2023-05-31T20:09:59.297 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:09:59 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:U7kzjb/bvaq1XSbrsygA0c5z9/E=', 'Content-Length': '97'} 2023-05-31T20:09:59.308 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000e3b888a54147aa69-006477a997-1630-a1'), ('Date', 'Wed, 31 May 2023 20:09:59 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:09:59.308 DEBUG:boto:path=/pklxkd-40/ 2023-05-31T20:09:59.309 DEBUG:boto:auth_path=/pklxkd-40/ 2023-05-31T20:09:59.309 DEBUG:boto:path=/pklxkd-40/?policy 2023-05-31T20:09:59.310 DEBUG:boto:auth_path=/pklxkd-40/?policy 2023-05-31T20:09:59.310 DEBUG:boto:Method: GET 2023-05-31T20:09:59.311 DEBUG:boto:Path: /pklxkd-40/?policy 2023-05-31T20:09:59.311 DEBUG:boto:Data: 2023-05-31T20:09:59.312 DEBUG:boto:Headers: {} 2023-05-31T20:09:59.312 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:09:59.313 DEBUG:boto:Port: 8000 2023-05-31T20:09:59.313 DEBUG:boto:Params: {} 2023-05-31T20:09:59.314 DEBUG:boto:Token: None 2023-05-31T20:09:59.314 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:09:59 GMT /pklxkd-40/?policy 2023-05-31T20:09:59.315 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:irBpAD0tNXW6DaoG78IwS+DD/Sc= 2023-05-31T20:09:59.316 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:09:59 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:irBpAD0tNXW6DaoG78IwS+DD/Sc=', 'Content-Length': '0'} 2023-05-31T20:09:59.320 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx000008db31d6c9dd763c0-006477a997-1630-a1'), ('Content-Type', 'application/json'), ('Content-Length', '97'), ('Date', 'Wed, 31 May 2023 20:09:59 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:09:59.321 DEBUG:boto:path=/pklxkd-41/ 2023-05-31T20:09:59.321 DEBUG:boto:auth_path=/pklxkd-41/ 2023-05-31T20:09:59.322 DEBUG:boto:path=/pklxkd-41/?policy 2023-05-31T20:09:59.322 DEBUG:boto:auth_path=/pklxkd-41/?policy 2023-05-31T20:09:59.322 DEBUG:boto:Method: PUT 2023-05-31T20:09:59.322 DEBUG:boto:Path: /pklxkd-41/?policy 2023-05-31T20:09:59.323 DEBUG:boto:Data: { "Version": "2012-10-17", "Statement": [{ "Effect": "Allow", "Principal": "*" }] } 2023-05-31T20:09:59.323 DEBUG:boto:Headers: {} 2023-05-31T20:09:59.323 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:09:59.324 DEBUG:boto:Port: 8001 2023-05-31T20:09:59.324 DEBUG:boto:Params: {} 2023-05-31T20:09:59.325 DEBUG:boto:Token: None 2023-05-31T20:09:59.325 DEBUG:boto:StringToSign: PUT Wed, 31 May 2023 20:09:59 GMT /pklxkd-41/?policy 2023-05-31T20:09:59.325 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:HQCjSK4wrwU/Qh1AjxFJ6ynREbY= 2023-05-31T20:09:59.326 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:09:59 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:HQCjSK4wrwU/Qh1AjxFJ6ynREbY=', 'Content-Length': '97'} 2023-05-31T20:10:00.335 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000db0e2ef314239e12-006477a997-162e-a2'), ('Date', 'Wed, 31 May 2023 20:10:00 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:10:00.336 DEBUG:boto:path=/pklxkd-41/ 2023-05-31T20:10:00.337 DEBUG:boto:auth_path=/pklxkd-41/ 2023-05-31T20:10:00.338 DEBUG:boto:path=/pklxkd-41/?policy 2023-05-31T20:10:00.338 DEBUG:boto:auth_path=/pklxkd-41/?policy 2023-05-31T20:10:00.339 DEBUG:boto:Method: GET 2023-05-31T20:10:00.340 DEBUG:boto:Path: /pklxkd-41/?policy 2023-05-31T20:10:00.342 DEBUG:boto:Data: 2023-05-31T20:10:00.343 DEBUG:boto:Headers: {} 2023-05-31T20:10:00.344 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:10:00.344 DEBUG:boto:Port: 8001 2023-05-31T20:10:00.344 DEBUG:boto:Params: {} 2023-05-31T20:10:00.345 DEBUG:boto:Token: None 2023-05-31T20:10:00.345 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:10:00 GMT /pklxkd-41/?policy 2023-05-31T20:10:00.346 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:VQ84PE9FskPFglh+a6CM6jZ6dbk= 2023-05-31T20:10:00.346 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:10:00 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:VQ84PE9FskPFglh+a6CM6jZ6dbk=', 'Content-Length': '0'} 2023-05-31T20:10:00.348 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx0000042c1efce89c74843-006477a998-162e-a2'), ('Content-Type', 'application/json'), ('Content-Length', '97'), ('Date', 'Wed, 31 May 2023 20:10:00 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:10:00.349 DEBUG:boto:path=/pklxkd-42/ 2023-05-31T20:10:00.349 DEBUG:boto:auth_path=/pklxkd-42/ 2023-05-31T20:10:00.350 DEBUG:boto:path=/pklxkd-42/?policy 2023-05-31T20:10:00.350 DEBUG:boto:auth_path=/pklxkd-42/?policy 2023-05-31T20:10:00.350 DEBUG:boto:Method: PUT 2023-05-31T20:10:00.351 DEBUG:boto:Path: /pklxkd-42/?policy 2023-05-31T20:10:00.351 DEBUG:boto:Data: { "Version": "2012-10-17", "Statement": [{ "Effect": "Allow", "Principal": "*" }] } 2023-05-31T20:10:00.352 DEBUG:boto:Headers: {} 2023-05-31T20:10:00.352 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-05-31T20:10:00.352 DEBUG:boto:Port: 8000 2023-05-31T20:10:00.353 DEBUG:boto:Params: {} 2023-05-31T20:10:00.353 DEBUG:boto:Token: None 2023-05-31T20:10:00.354 DEBUG:boto:StringToSign: PUT Wed, 31 May 2023 20:10:00 GMT /pklxkd-42/?policy 2023-05-31T20:10:00.354 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:ozIGS/mFzISCU71w4R/fKBNiLbU= 2023-05-31T20:10:00.354 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:10:00 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:ozIGS/mFzISCU71w4R/fKBNiLbU=', 'Content-Length': '97'} 2023-05-31T20:10:01.368 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx0000055e7e3c87e98d02b-006477a998-115b-b1'), ('Date', 'Wed, 31 May 2023 20:10:01 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:10:01.369 DEBUG:boto:path=/pklxkd-42/ 2023-05-31T20:10:01.369 DEBUG:boto:auth_path=/pklxkd-42/ 2023-05-31T20:10:01.370 DEBUG:boto:path=/pklxkd-42/?policy 2023-05-31T20:10:01.370 DEBUG:boto:auth_path=/pklxkd-42/?policy 2023-05-31T20:10:01.370 DEBUG:boto:Method: GET 2023-05-31T20:10:01.371 DEBUG:boto:Path: /pklxkd-42/?policy 2023-05-31T20:10:01.371 DEBUG:boto:Data: 2023-05-31T20:10:01.371 DEBUG:boto:Headers: {} 2023-05-31T20:10:01.371 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-05-31T20:10:01.372 DEBUG:boto:Port: 8000 2023-05-31T20:10:01.372 DEBUG:boto:Params: {} 2023-05-31T20:10:01.373 DEBUG:boto:Token: None 2023-05-31T20:10:01.373 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:10:01 GMT /pklxkd-42/?policy 2023-05-31T20:10:01.373 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:BqUpDi6YYxppC9gNqQsjVCigMcc= 2023-05-31T20:10:01.374 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:10:01 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:BqUpDi6YYxppC9gNqQsjVCigMcc=', 'Content-Length': '0'} 2023-05-31T20:10:01.377 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx0000031799a3707116ef1-006477a999-115b-b1'), ('Content-Type', 'application/json'), ('Content-Length', '97'), ('Date', 'Wed, 31 May 2023 20:10:01 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:10:01.378 DEBUG:boto:path=/pklxkd-43/ 2023-05-31T20:10:01.379 DEBUG:boto:auth_path=/pklxkd-43/ 2023-05-31T20:10:01.380 DEBUG:boto:path=/pklxkd-43/?policy 2023-05-31T20:10:01.380 DEBUG:boto:auth_path=/pklxkd-43/?policy 2023-05-31T20:10:01.381 DEBUG:boto:Method: PUT 2023-05-31T20:10:01.382 DEBUG:boto:Path: /pklxkd-43/?policy 2023-05-31T20:10:01.382 DEBUG:boto:Data: { "Version": "2012-10-17", "Statement": [{ "Effect": "Allow", "Principal": "*" }] } 2023-05-31T20:10:01.383 DEBUG:boto:Headers: {} 2023-05-31T20:10:01.384 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-05-31T20:10:01.385 DEBUG:boto:Port: 8001 2023-05-31T20:10:01.385 DEBUG:boto:Params: {} 2023-05-31T20:10:01.386 DEBUG:boto:Token: None 2023-05-31T20:10:01.387 DEBUG:boto:StringToSign: PUT Wed, 31 May 2023 20:10:01 GMT /pklxkd-43/?policy 2023-05-31T20:10:01.388 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:ppll0ZVtWnjBA4idB+Uwe9Li9cw= 2023-05-31T20:10:01.389 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:10:01 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:ppll0ZVtWnjBA4idB+Uwe9Li9cw=', 'Content-Length': '97'} 2023-05-31T20:10:02.401 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000e37a9cce7fb2baa9-006477a999-1157-b2'), ('Date', 'Wed, 31 May 2023 20:10:02 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:10:02.402 DEBUG:boto:path=/pklxkd-43/ 2023-05-31T20:10:02.403 DEBUG:boto:auth_path=/pklxkd-43/ 2023-05-31T20:10:02.403 DEBUG:boto:path=/pklxkd-43/?policy 2023-05-31T20:10:02.403 DEBUG:boto:auth_path=/pklxkd-43/?policy 2023-05-31T20:10:02.404 DEBUG:boto:Method: GET 2023-05-31T20:10:02.404 DEBUG:boto:Path: /pklxkd-43/?policy 2023-05-31T20:10:02.404 DEBUG:boto:Data: 2023-05-31T20:10:02.405 DEBUG:boto:Headers: {} 2023-05-31T20:10:02.405 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-05-31T20:10:02.405 DEBUG:boto:Port: 8001 2023-05-31T20:10:02.406 DEBUG:boto:Params: {} 2023-05-31T20:10:02.406 DEBUG:boto:Token: None 2023-05-31T20:10:02.407 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:10:02 GMT /pklxkd-43/?policy 2023-05-31T20:10:02.407 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:0OcLMuMQ/PQPBoTEZarkGyejmO4= 2023-05-31T20:10:02.407 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:10:02 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:0OcLMuMQ/PQPBoTEZarkGyejmO4=', 'Content-Length': '0'} 2023-05-31T20:10:02.409 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000688dbf49c1dd9b3b-006477a99a-1157-b2'), ('Content-Type', 'application/json'), ('Content-Length', '97'), ('Date', 'Wed, 31 May 2023 20:10:02 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:10:02.410 INFO:tasks.rgw_multisite_tests:rgw_multi.tests.test_set_bucket_policy ... ok 2023-05-31T20:10:02.410 INFO:rgw_multi.tests:create bucket zone=a1 name=pklxkd-44 2023-05-31T20:10:02.411 DEBUG:boto:path=/pklxkd-44/ 2023-05-31T20:10:02.411 DEBUG:boto:auth_path=/pklxkd-44/ 2023-05-31T20:10:02.412 DEBUG:boto:Method: PUT 2023-05-31T20:10:02.412 DEBUG:boto:Path: /pklxkd-44/ 2023-05-31T20:10:02.412 DEBUG:boto:Data: 2023-05-31T20:10:02.413 DEBUG:boto:Headers: {} 2023-05-31T20:10:02.413 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:10:02.413 DEBUG:boto:Port: 8000 2023-05-31T20:10:02.414 DEBUG:boto:Params: {} 2023-05-31T20:10:02.414 DEBUG:boto:Token: None 2023-05-31T20:10:02.414 DEBUG:boto:StringToSign: PUT Wed, 31 May 2023 20:10:02 GMT /pklxkd-44/ 2023-05-31T20:10:02.415 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:tplv2S6oje6AFjLCtWLkuSyBmvg= 2023-05-31T20:10:02.415 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:10:02 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:tplv2S6oje6AFjLCtWLkuSyBmvg=', 'Content-Length': '0'} 2023-05-31T20:10:02.443 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx000002280b60cd65dabcc-006477a99a-1630-a1'), ('Content-Length', '0'), ('Date', 'Wed, 31 May 2023 20:10:02 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:10:02.444 INFO:rgw_multi.tests:create bucket zone=a2 name=pklxkd-45 2023-05-31T20:10:02.444 DEBUG:boto:path=/pklxkd-45/ 2023-05-31T20:10:02.445 DEBUG:boto:auth_path=/pklxkd-45/ 2023-05-31T20:10:02.445 DEBUG:boto:Method: PUT 2023-05-31T20:10:02.445 DEBUG:boto:Path: /pklxkd-45/ 2023-05-31T20:10:02.446 DEBUG:boto:Data: 2023-05-31T20:10:02.446 DEBUG:boto:Headers: {} 2023-05-31T20:10:02.447 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:10:02.447 DEBUG:boto:Port: 8001 2023-05-31T20:10:02.447 DEBUG:boto:Params: {} 2023-05-31T20:10:02.448 DEBUG:boto:Token: None 2023-05-31T20:10:02.449 DEBUG:boto:StringToSign: PUT Wed, 31 May 2023 20:10:02 GMT /pklxkd-45/ 2023-05-31T20:10:02.449 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:PByA4jdQn+Mx1Qjcwt4TX9Y2iEQ= 2023-05-31T20:10:02.450 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:10:02 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:PByA4jdQn+Mx1Qjcwt4TX9Y2iEQ=', 'Content-Length': '0'} 2023-05-31T20:10:02.488 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx000004328383d7e6905c2-006477a99a-162e-a2'), ('Content-Length', '0'), ('Date', 'Wed, 31 May 2023 20:10:02 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:10:02.489 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-31T20:10:02.489 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-31T20:10:02.490 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-05-31T20:10:02.541 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2023-05-31T20:10:02.541 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2023-05-31T20:10:02.584 INFO:teuthology.orchestra.run.smithi099.stderr:No --period given, using current period=2e6123d2-d695-4350-9a01-61a455bc782b 2023-05-31T20:10:02.595 INFO:teuthology.orchestra.run.smithi099.stdout:[ 2023-05-31T20:10:02.595 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T20:10:02.595 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1685562631.516089_59.1", 2023-05-31T20:10:02.595 INFO:teuthology.orchestra.run.smithi099.stdout: "last_update": "2023-05-31T19:50:31.516089Z" 2023-05-31T20:10:02.596 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:10:02.596 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T20:10:02.596 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "", 2023-05-31T20:10:02.596 INFO:teuthology.orchestra.run.smithi099.stdout: "last_update": "0.000000" 2023-05-31T20:10:02.596 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:10:02.596 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T20:10:02.596 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1685562970.274572_93.1", 2023-05-31T20:10:02.597 INFO:teuthology.orchestra.run.smithi099.stdout: "last_update": "2023-05-31T19:56:10.274572Z" 2023-05-31T20:10:02.597 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:10:02.597 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T20:10:02.597 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1685563802.468126_296.1", 2023-05-31T20:10:02.597 INFO:teuthology.orchestra.run.smithi099.stdout: "last_update": "2023-05-31T20:10:02.468126Z" 2023-05-31T20:10:02.597 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T20:10:02.598 INFO:teuthology.orchestra.run.smithi099.stdout:] 2023-05-31T20:10:02.598 DEBUG:rgw_multi.tests:master meta markers={0: '1_1685562631.516089_59.1', 1: '', 2: '1_1685562970.274572_93.1', 3: '1_1685563802.468126_296.1'} 2023-05-31T20:10:02.599 INFO:rgw_multi.tests:starting meta checkpoint for zone=a2 2023-05-31T20:10:02.599 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-31T20:10:02.600 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-31T20:10:02.600 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-05-31T20:10:02.649 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2023-05-31T20:10:02.650 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2023-05-31T20:10:02.680 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:10:02.674+0000 7fe79bb04780 20 RGW-SYNC:meta: read sync status 2023-05-31T20:10:02.681 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:10:02.675+0000 7fe79bb04780 20 RGW-SYNC:meta: read sync status 2023-05-31T20:10:02.693 INFO:teuthology.orchestra.run.smithi099.stdout:{ 2023-05-31T20:10:02.693 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_status": { 2023-05-31T20:10:02.693 INFO:teuthology.orchestra.run.smithi099.stdout: "info": { 2023-05-31T20:10:02.694 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "sync", 2023-05-31T20:10:02.694 INFO:teuthology.orchestra.run.smithi099.stdout: "num_shards": 4, 2023-05-31T20:10:02.694 INFO:teuthology.orchestra.run.smithi099.stdout: "period": "2e6123d2-d695-4350-9a01-61a455bc782b", 2023-05-31T20:10:02.695 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2023-05-31T20:10:02.695 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:10:02.695 INFO:teuthology.orchestra.run.smithi099.stdout: "markers": [ 2023-05-31T20:10:02.695 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T20:10:02.695 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 0, 2023-05-31T20:10:02.696 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T20:10:02.696 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2023-05-31T20:10:02.696 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1685562631.516089_59.1", 2023-05-31T20:10:02.696 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T20:10:02.696 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 1, 2023-05-31T20:10:02.697 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T20:10:02.697 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2023-05-31T19:50:31.516089Z", 2023-05-31T20:10:02.697 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2023-05-31T20:10:02.697 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T20:10:02.698 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:10:02.698 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T20:10:02.698 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 1, 2023-05-31T20:10:02.699 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T20:10:02.699 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2023-05-31T20:10:02.699 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "", 2023-05-31T20:10:02.699 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T20:10:02.700 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2023-05-31T20:10:02.700 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T20:10:02.703 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "0.000000", 2023-05-31T20:10:02.707 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2023-05-31T20:10:02.707 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T20:10:02.707 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:10:02.707 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T20:10:02.707 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 2, 2023-05-31T20:10:02.708 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T20:10:02.708 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2023-05-31T20:10:02.708 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1685562970.274572_93.1", 2023-05-31T20:10:02.708 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T20:10:02.708 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2023-05-31T20:10:02.708 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T20:10:02.709 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2023-05-31T19:56:10.274572Z", 2023-05-31T20:10:02.709 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2023-05-31T20:10:02.709 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T20:10:02.709 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:10:02.709 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T20:10:02.710 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 3, 2023-05-31T20:10:02.710 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T20:10:02.710 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2023-05-31T20:10:02.710 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1685563571.700188_212.1", 2023-05-31T20:10:02.710 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T20:10:02.710 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2023-05-31T20:10:02.711 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T20:10:02.712 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2023-05-31T20:06:11.700188Z", 2023-05-31T20:10:02.712 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2023-05-31T20:10:02.712 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T20:10:02.712 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T20:10:02.712 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2023-05-31T20:10:02.712 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:10:02.713 INFO:teuthology.orchestra.run.smithi099.stdout: "full_sync": { 2023-05-31T20:10:02.713 INFO:teuthology.orchestra.run.smithi099.stdout: "total": 1, 2023-05-31T20:10:02.713 INFO:teuthology.orchestra.run.smithi099.stdout: "complete": 1 2023-05-31T20:10:02.713 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:10:02.714 INFO:teuthology.orchestra.run.smithi099.stdout: "current_time": "2023-05-31T20:10:02Z" 2023-05-31T20:10:02.714 INFO:teuthology.orchestra.run.smithi099.stdout:} 2023-05-31T20:10:02.715 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "2e6123d2-d695-4350-9a01-61a455bc782b", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1685562631.516089_59.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-05-31T19:50:31.516089Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1685562970.274572_93.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-05-31T19:56:10.274572Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1685563571.700188_212.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-05-31T20:06:11.700188Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-05-31T20:10:02Z" } 2023-05-31T20:10:02.716 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1685562631.516089_59.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-05-31T19:50:31.516089Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1685562970.274572_93.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-05-31T19:56:10.274572Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1685563571.700188_212.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-05-31T20:06:11.700188Z', 'realm_epoch': 2}}] 2023-05-31T20:10:02.716 DEBUG:rgw_multi.tests:log_status={0: '1_1685562631.516089_59.1', 1: '', 2: '1_1685562970.274572_93.1', 3: '1_1685563802.468126_296.1'} 2023-05-31T20:10:02.716 DEBUG:rgw_multi.tests:sync_status={0: '1_1685562631.516089_59.1', 1: '', 2: '1_1685562970.274572_93.1', 3: '1_1685563571.700188_212.1'} 2023-05-31T20:10:02.717 WARNING:rgw_multi.tests:zone a2 behind master: shard=3 master=1_1685563802.468126_296.1 target=1_1685563571.700188_212.1 2023-05-31T20:10:07.718 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-31T20:10:07.719 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-31T20:10:07.719 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-05-31T20:10:07.779 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2023-05-31T20:10:07.780 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2023-05-31T20:10:07.832 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:10:07.826+0000 7fd59bd4d780 20 RGW-SYNC:meta: read sync status 2023-05-31T20:10:07.836 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:10:07.830+0000 7fd59bd4d780 20 RGW-SYNC:meta: read sync status 2023-05-31T20:10:07.858 INFO:teuthology.orchestra.run.smithi099.stdout:{ 2023-05-31T20:10:07.858 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_status": { 2023-05-31T20:10:07.858 INFO:teuthology.orchestra.run.smithi099.stdout: "info": { 2023-05-31T20:10:07.858 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "sync", 2023-05-31T20:10:07.859 INFO:teuthology.orchestra.run.smithi099.stdout: "num_shards": 4, 2023-05-31T20:10:07.859 INFO:teuthology.orchestra.run.smithi099.stdout: "period": "2e6123d2-d695-4350-9a01-61a455bc782b", 2023-05-31T20:10:07.859 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2023-05-31T20:10:07.859 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:10:07.860 INFO:teuthology.orchestra.run.smithi099.stdout: "markers": [ 2023-05-31T20:10:07.861 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T20:10:07.861 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 0, 2023-05-31T20:10:07.861 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T20:10:07.862 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2023-05-31T20:10:07.862 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1685562631.516089_59.1", 2023-05-31T20:10:07.862 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T20:10:07.863 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 1, 2023-05-31T20:10:07.863 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T20:10:07.863 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2023-05-31T19:50:31.516089Z", 2023-05-31T20:10:07.863 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2023-05-31T20:10:07.864 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T20:10:07.864 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:10:07.864 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T20:10:07.864 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 1, 2023-05-31T20:10:07.865 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T20:10:07.865 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2023-05-31T20:10:07.866 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "", 2023-05-31T20:10:07.866 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T20:10:07.866 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2023-05-31T20:10:07.867 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T20:10:07.867 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "0.000000", 2023-05-31T20:10:07.868 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2023-05-31T20:10:07.868 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T20:10:07.868 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:10:07.869 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T20:10:07.869 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 2, 2023-05-31T20:10:07.869 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T20:10:07.870 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2023-05-31T20:10:07.870 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1685562970.274572_93.1", 2023-05-31T20:10:07.870 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T20:10:07.870 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2023-05-31T20:10:07.871 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T20:10:07.871 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2023-05-31T19:56:10.274572Z", 2023-05-31T20:10:07.871 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2023-05-31T20:10:07.872 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T20:10:07.872 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:10:07.872 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T20:10:07.873 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 3, 2023-05-31T20:10:07.873 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T20:10:07.873 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2023-05-31T20:10:07.873 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1685563571.700188_212.1", 2023-05-31T20:10:07.873 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T20:10:07.873 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2023-05-31T20:10:07.874 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T20:10:07.874 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2023-05-31T20:06:11.700188Z", 2023-05-31T20:10:07.874 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2023-05-31T20:10:07.875 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T20:10:07.875 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T20:10:07.875 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2023-05-31T20:10:07.876 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:10:07.876 INFO:teuthology.orchestra.run.smithi099.stdout: "full_sync": { 2023-05-31T20:10:07.876 INFO:teuthology.orchestra.run.smithi099.stdout: "total": 1, 2023-05-31T20:10:07.877 INFO:teuthology.orchestra.run.smithi099.stdout: "complete": 1 2023-05-31T20:10:07.877 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:10:07.877 INFO:teuthology.orchestra.run.smithi099.stdout: "current_time": "2023-05-31T20:10:07Z" 2023-05-31T20:10:07.878 INFO:teuthology.orchestra.run.smithi099.stdout:} 2023-05-31T20:10:07.880 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "2e6123d2-d695-4350-9a01-61a455bc782b", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1685562631.516089_59.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-05-31T19:50:31.516089Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1685562970.274572_93.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-05-31T19:56:10.274572Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1685563571.700188_212.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-05-31T20:06:11.700188Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-05-31T20:10:07Z" } 2023-05-31T20:10:07.881 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1685562631.516089_59.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-05-31T19:50:31.516089Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1685562970.274572_93.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-05-31T19:56:10.274572Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1685563571.700188_212.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-05-31T20:06:11.700188Z', 'realm_epoch': 2}}] 2023-05-31T20:10:07.882 DEBUG:rgw_multi.tests:log_status={0: '1_1685562631.516089_59.1', 1: '', 2: '1_1685562970.274572_93.1', 3: '1_1685563802.468126_296.1'} 2023-05-31T20:10:07.884 DEBUG:rgw_multi.tests:sync_status={0: '1_1685562631.516089_59.1', 1: '', 2: '1_1685562970.274572_93.1', 3: '1_1685563571.700188_212.1'} 2023-05-31T20:10:07.884 WARNING:rgw_multi.tests:zone a2 behind master: shard=3 master=1_1685563802.468126_296.1 target=1_1685563571.700188_212.1 2023-05-31T20:10:12.885 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-31T20:10:12.886 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-31T20:10:12.887 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-05-31T20:10:12.944 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2023-05-31T20:10:12.945 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2023-05-31T20:10:12.989 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:10:12.983+0000 7f53462a4780 20 RGW-SYNC:meta: read sync status 2023-05-31T20:10:12.990 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:10:12.984+0000 7f53462a4780 20 RGW-SYNC:meta: read sync status 2023-05-31T20:10:13.003 INFO:teuthology.orchestra.run.smithi099.stdout:{ 2023-05-31T20:10:13.003 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_status": { 2023-05-31T20:10:13.004 INFO:teuthology.orchestra.run.smithi099.stdout: "info": { 2023-05-31T20:10:13.004 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "sync", 2023-05-31T20:10:13.004 INFO:teuthology.orchestra.run.smithi099.stdout: "num_shards": 4, 2023-05-31T20:10:13.004 INFO:teuthology.orchestra.run.smithi099.stdout: "period": "2e6123d2-d695-4350-9a01-61a455bc782b", 2023-05-31T20:10:13.005 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2023-05-31T20:10:13.005 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:10:13.005 INFO:teuthology.orchestra.run.smithi099.stdout: "markers": [ 2023-05-31T20:10:13.005 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T20:10:13.005 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 0, 2023-05-31T20:10:13.005 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T20:10:13.006 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2023-05-31T20:10:13.006 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1685562631.516089_59.1", 2023-05-31T20:10:13.006 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T20:10:13.006 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 1, 2023-05-31T20:10:13.006 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T20:10:13.006 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2023-05-31T19:50:31.516089Z", 2023-05-31T20:10:13.006 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2023-05-31T20:10:13.007 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T20:10:13.007 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:10:13.007 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T20:10:13.007 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 1, 2023-05-31T20:10:13.007 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T20:10:13.007 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2023-05-31T20:10:13.007 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "", 2023-05-31T20:10:13.008 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T20:10:13.008 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2023-05-31T20:10:13.008 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T20:10:13.008 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "0.000000", 2023-05-31T20:10:13.008 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2023-05-31T20:10:13.008 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T20:10:13.009 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:10:13.009 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T20:10:13.009 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 2, 2023-05-31T20:10:13.009 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T20:10:13.009 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2023-05-31T20:10:13.009 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1685562970.274572_93.1", 2023-05-31T20:10:13.009 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T20:10:13.010 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2023-05-31T20:10:13.010 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T20:10:13.010 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2023-05-31T19:56:10.274572Z", 2023-05-31T20:10:13.010 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2023-05-31T20:10:13.010 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T20:10:13.010 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:10:13.010 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T20:10:13.011 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 3, 2023-05-31T20:10:13.011 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T20:10:13.011 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2023-05-31T20:10:13.011 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1685563571.700188_212.1", 2023-05-31T20:10:13.011 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T20:10:13.011 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2023-05-31T20:10:13.011 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T20:10:13.012 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2023-05-31T20:06:11.700188Z", 2023-05-31T20:10:13.012 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2023-05-31T20:10:13.012 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T20:10:13.012 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T20:10:13.012 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2023-05-31T20:10:13.013 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:10:13.013 INFO:teuthology.orchestra.run.smithi099.stdout: "full_sync": { 2023-05-31T20:10:13.014 INFO:teuthology.orchestra.run.smithi099.stdout: "total": 1, 2023-05-31T20:10:13.015 INFO:teuthology.orchestra.run.smithi099.stdout: "complete": 1 2023-05-31T20:10:13.015 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:10:13.015 INFO:teuthology.orchestra.run.smithi099.stdout: "current_time": "2023-05-31T20:10:12Z" 2023-05-31T20:10:13.015 INFO:teuthology.orchestra.run.smithi099.stdout:} 2023-05-31T20:10:13.016 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "2e6123d2-d695-4350-9a01-61a455bc782b", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1685562631.516089_59.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-05-31T19:50:31.516089Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1685562970.274572_93.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-05-31T19:56:10.274572Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1685563571.700188_212.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-05-31T20:06:11.700188Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-05-31T20:10:12Z" } 2023-05-31T20:10:13.017 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1685562631.516089_59.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-05-31T19:50:31.516089Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1685562970.274572_93.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-05-31T19:56:10.274572Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1685563571.700188_212.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-05-31T20:06:11.700188Z', 'realm_epoch': 2}}] 2023-05-31T20:10:13.017 DEBUG:rgw_multi.tests:log_status={0: '1_1685562631.516089_59.1', 1: '', 2: '1_1685562970.274572_93.1', 3: '1_1685563802.468126_296.1'} 2023-05-31T20:10:13.017 DEBUG:rgw_multi.tests:sync_status={0: '1_1685562631.516089_59.1', 1: '', 2: '1_1685562970.274572_93.1', 3: '1_1685563571.700188_212.1'} 2023-05-31T20:10:13.018 WARNING:rgw_multi.tests:zone a2 behind master: shard=3 master=1_1685563802.468126_296.1 target=1_1685563571.700188_212.1 2023-05-31T20:10:18.019 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-31T20:10:18.019 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-31T20:10:18.020 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-05-31T20:10:18.077 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2023-05-31T20:10:18.077 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2023-05-31T20:10:18.131 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:10:18.125+0000 7fcee7df8780 20 RGW-SYNC:meta: read sync status 2023-05-31T20:10:18.132 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:10:18.126+0000 7fcee7df8780 20 RGW-SYNC:meta: read sync status 2023-05-31T20:10:18.155 INFO:teuthology.orchestra.run.smithi099.stdout:{ 2023-05-31T20:10:18.156 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_status": { 2023-05-31T20:10:18.156 INFO:teuthology.orchestra.run.smithi099.stdout: "info": { 2023-05-31T20:10:18.157 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "sync", 2023-05-31T20:10:18.157 INFO:teuthology.orchestra.run.smithi099.stdout: "num_shards": 4, 2023-05-31T20:10:18.157 INFO:teuthology.orchestra.run.smithi099.stdout: "period": "2e6123d2-d695-4350-9a01-61a455bc782b", 2023-05-31T20:10:18.157 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2023-05-31T20:10:18.158 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:10:18.158 INFO:teuthology.orchestra.run.smithi099.stdout: "markers": [ 2023-05-31T20:10:18.158 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T20:10:18.158 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 0, 2023-05-31T20:10:18.159 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T20:10:18.159 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2023-05-31T20:10:18.160 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1685562631.516089_59.1", 2023-05-31T20:10:18.160 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T20:10:18.161 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 1, 2023-05-31T20:10:18.161 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T20:10:18.161 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2023-05-31T19:50:31.516089Z", 2023-05-31T20:10:18.161 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2023-05-31T20:10:18.161 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T20:10:18.161 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:10:18.162 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T20:10:18.162 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 1, 2023-05-31T20:10:18.162 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T20:10:18.162 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2023-05-31T20:10:18.163 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "", 2023-05-31T20:10:18.163 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T20:10:18.163 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2023-05-31T20:10:18.164 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T20:10:18.164 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "0.000000", 2023-05-31T20:10:18.164 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2023-05-31T20:10:18.164 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T20:10:18.165 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:10:18.165 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T20:10:18.165 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 2, 2023-05-31T20:10:18.166 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T20:10:18.166 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2023-05-31T20:10:18.166 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1685562970.274572_93.1", 2023-05-31T20:10:18.166 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T20:10:18.167 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2023-05-31T20:10:18.167 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T20:10:18.168 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2023-05-31T19:56:10.274572Z", 2023-05-31T20:10:18.168 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2023-05-31T20:10:18.169 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T20:10:18.169 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:10:18.169 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T20:10:18.169 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 3, 2023-05-31T20:10:18.170 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T20:10:18.170 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2023-05-31T20:10:18.170 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1685563571.700188_212.1", 2023-05-31T20:10:18.170 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T20:10:18.170 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2023-05-31T20:10:18.170 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T20:10:18.171 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2023-05-31T20:06:11.700188Z", 2023-05-31T20:10:18.171 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2023-05-31T20:10:18.171 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T20:10:18.171 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T20:10:18.171 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2023-05-31T20:10:18.172 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:10:18.172 INFO:teuthology.orchestra.run.smithi099.stdout: "full_sync": { 2023-05-31T20:10:18.172 INFO:teuthology.orchestra.run.smithi099.stdout: "total": 1, 2023-05-31T20:10:18.172 INFO:teuthology.orchestra.run.smithi099.stdout: "complete": 1 2023-05-31T20:10:18.172 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:10:18.173 INFO:teuthology.orchestra.run.smithi099.stdout: "current_time": "2023-05-31T20:10:18Z" 2023-05-31T20:10:18.173 INFO:teuthology.orchestra.run.smithi099.stdout:} 2023-05-31T20:10:18.174 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "2e6123d2-d695-4350-9a01-61a455bc782b", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1685562631.516089_59.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-05-31T19:50:31.516089Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1685562970.274572_93.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-05-31T19:56:10.274572Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1685563571.700188_212.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-05-31T20:06:11.700188Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-05-31T20:10:18Z" } 2023-05-31T20:10:18.176 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1685562631.516089_59.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-05-31T19:50:31.516089Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1685562970.274572_93.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-05-31T19:56:10.274572Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1685563571.700188_212.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-05-31T20:06:11.700188Z', 'realm_epoch': 2}}] 2023-05-31T20:10:18.177 DEBUG:rgw_multi.tests:log_status={0: '1_1685562631.516089_59.1', 1: '', 2: '1_1685562970.274572_93.1', 3: '1_1685563802.468126_296.1'} 2023-05-31T20:10:18.177 DEBUG:rgw_multi.tests:sync_status={0: '1_1685562631.516089_59.1', 1: '', 2: '1_1685562970.274572_93.1', 3: '1_1685563571.700188_212.1'} 2023-05-31T20:10:18.178 WARNING:rgw_multi.tests:zone a2 behind master: shard=3 master=1_1685563802.468126_296.1 target=1_1685563571.700188_212.1 2023-05-31T20:10:23.180 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-31T20:10:23.181 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-31T20:10:23.181 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-05-31T20:10:23.238 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2023-05-31T20:10:23.239 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2023-05-31T20:10:23.290 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:10:23.285+0000 7f341a52d780 20 RGW-SYNC:meta: read sync status 2023-05-31T20:10:23.291 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:10:23.286+0000 7f341a52d780 20 RGW-SYNC:meta: read sync status 2023-05-31T20:10:23.310 INFO:teuthology.orchestra.run.smithi099.stdout:{ 2023-05-31T20:10:23.311 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_status": { 2023-05-31T20:10:23.311 INFO:teuthology.orchestra.run.smithi099.stdout: "info": { 2023-05-31T20:10:23.311 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "sync", 2023-05-31T20:10:23.311 INFO:teuthology.orchestra.run.smithi099.stdout: "num_shards": 4, 2023-05-31T20:10:23.311 INFO:teuthology.orchestra.run.smithi099.stdout: "period": "2e6123d2-d695-4350-9a01-61a455bc782b", 2023-05-31T20:10:23.311 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2023-05-31T20:10:23.312 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:10:23.312 INFO:teuthology.orchestra.run.smithi099.stdout: "markers": [ 2023-05-31T20:10:23.312 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T20:10:23.312 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 0, 2023-05-31T20:10:23.312 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T20:10:23.312 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2023-05-31T20:10:23.313 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1685562631.516089_59.1", 2023-05-31T20:10:23.313 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T20:10:23.313 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 1, 2023-05-31T20:10:23.313 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T20:10:23.313 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2023-05-31T19:50:31.516089Z", 2023-05-31T20:10:23.313 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2023-05-31T20:10:23.313 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T20:10:23.314 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:10:23.314 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T20:10:23.314 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 1, 2023-05-31T20:10:23.314 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T20:10:23.314 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2023-05-31T20:10:23.314 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "", 2023-05-31T20:10:23.315 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T20:10:23.315 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2023-05-31T20:10:23.315 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T20:10:23.315 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "0.000000", 2023-05-31T20:10:23.315 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2023-05-31T20:10:23.315 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T20:10:23.315 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:10:23.316 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T20:10:23.316 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 2, 2023-05-31T20:10:23.316 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T20:10:23.316 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2023-05-31T20:10:23.317 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1685562970.274572_93.1", 2023-05-31T20:10:23.317 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T20:10:23.317 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2023-05-31T20:10:23.318 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T20:10:23.318 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2023-05-31T19:56:10.274572Z", 2023-05-31T20:10:23.318 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2023-05-31T20:10:23.318 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T20:10:23.318 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:10:23.318 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T20:10:23.318 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 3, 2023-05-31T20:10:23.319 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T20:10:23.319 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2023-05-31T20:10:23.319 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1685563571.700188_212.1", 2023-05-31T20:10:23.319 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T20:10:23.319 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2023-05-31T20:10:23.319 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T20:10:23.319 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2023-05-31T20:06:11.700188Z", 2023-05-31T20:10:23.320 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2023-05-31T20:10:23.320 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T20:10:23.320 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T20:10:23.320 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2023-05-31T20:10:23.320 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:10:23.320 INFO:teuthology.orchestra.run.smithi099.stdout: "full_sync": { 2023-05-31T20:10:23.321 INFO:teuthology.orchestra.run.smithi099.stdout: "total": 1, 2023-05-31T20:10:23.321 INFO:teuthology.orchestra.run.smithi099.stdout: "complete": 1 2023-05-31T20:10:23.321 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:10:23.321 INFO:teuthology.orchestra.run.smithi099.stdout: "current_time": "2023-05-31T20:10:23Z" 2023-05-31T20:10:23.321 INFO:teuthology.orchestra.run.smithi099.stdout:} 2023-05-31T20:10:23.322 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "2e6123d2-d695-4350-9a01-61a455bc782b", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1685562631.516089_59.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-05-31T19:50:31.516089Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1685562970.274572_93.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-05-31T19:56:10.274572Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1685563571.700188_212.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-05-31T20:06:11.700188Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-05-31T20:10:23Z" } 2023-05-31T20:10:23.323 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1685562631.516089_59.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-05-31T19:50:31.516089Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1685562970.274572_93.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-05-31T19:56:10.274572Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1685563571.700188_212.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-05-31T20:06:11.700188Z', 'realm_epoch': 2}}] 2023-05-31T20:10:23.323 DEBUG:rgw_multi.tests:log_status={0: '1_1685562631.516089_59.1', 1: '', 2: '1_1685562970.274572_93.1', 3: '1_1685563802.468126_296.1'} 2023-05-31T20:10:23.323 DEBUG:rgw_multi.tests:sync_status={0: '1_1685562631.516089_59.1', 1: '', 2: '1_1685562970.274572_93.1', 3: '1_1685563571.700188_212.1'} 2023-05-31T20:10:23.324 WARNING:rgw_multi.tests:zone a2 behind master: shard=3 master=1_1685563802.468126_296.1 target=1_1685563571.700188_212.1 2023-05-31T20:10:28.325 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-31T20:10:28.325 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-31T20:10:28.326 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-05-31T20:10:28.383 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2023-05-31T20:10:28.384 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2023-05-31T20:10:28.423 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:10:28.418+0000 7f124688b780 20 RGW-SYNC:meta: read sync status 2023-05-31T20:10:28.423 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:10:28.419+0000 7f124688b780 20 RGW-SYNC:meta: read sync status 2023-05-31T20:10:28.437 INFO:teuthology.orchestra.run.smithi099.stdout:{ 2023-05-31T20:10:28.437 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_status": { 2023-05-31T20:10:28.438 INFO:teuthology.orchestra.run.smithi099.stdout: "info": { 2023-05-31T20:10:28.438 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "sync", 2023-05-31T20:10:28.438 INFO:teuthology.orchestra.run.smithi099.stdout: "num_shards": 4, 2023-05-31T20:10:28.438 INFO:teuthology.orchestra.run.smithi099.stdout: "period": "2e6123d2-d695-4350-9a01-61a455bc782b", 2023-05-31T20:10:28.438 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2023-05-31T20:10:28.438 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:10:28.439 INFO:teuthology.orchestra.run.smithi099.stdout: "markers": [ 2023-05-31T20:10:28.439 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T20:10:28.439 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 0, 2023-05-31T20:10:28.439 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T20:10:28.440 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2023-05-31T20:10:28.440 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1685562631.516089_59.1", 2023-05-31T20:10:28.440 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T20:10:28.440 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 1, 2023-05-31T20:10:28.440 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T20:10:28.440 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2023-05-31T19:50:31.516089Z", 2023-05-31T20:10:28.441 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2023-05-31T20:10:28.441 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T20:10:28.441 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:10:28.441 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T20:10:28.441 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 1, 2023-05-31T20:10:28.441 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T20:10:28.441 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2023-05-31T20:10:28.442 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "", 2023-05-31T20:10:28.442 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T20:10:28.442 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2023-05-31T20:10:28.442 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T20:10:28.442 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "0.000000", 2023-05-31T20:10:28.442 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2023-05-31T20:10:28.442 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T20:10:28.443 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:10:28.443 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T20:10:28.443 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 2, 2023-05-31T20:10:28.443 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T20:10:28.443 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2023-05-31T20:10:28.443 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1685562970.274572_93.1", 2023-05-31T20:10:28.443 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T20:10:28.444 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2023-05-31T20:10:28.444 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T20:10:28.444 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2023-05-31T19:56:10.274572Z", 2023-05-31T20:10:28.444 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2023-05-31T20:10:28.444 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T20:10:28.444 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:10:28.444 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T20:10:28.445 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 3, 2023-05-31T20:10:28.445 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T20:10:28.445 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2023-05-31T20:10:28.445 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1685563571.700188_212.1", 2023-05-31T20:10:28.445 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T20:10:28.445 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2023-05-31T20:10:28.445 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T20:10:28.446 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2023-05-31T20:06:11.700188Z", 2023-05-31T20:10:28.446 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2023-05-31T20:10:28.446 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T20:10:28.446 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T20:10:28.446 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2023-05-31T20:10:28.446 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:10:28.447 INFO:teuthology.orchestra.run.smithi099.stdout: "full_sync": { 2023-05-31T20:10:28.447 INFO:teuthology.orchestra.run.smithi099.stdout: "total": 1, 2023-05-31T20:10:28.447 INFO:teuthology.orchestra.run.smithi099.stdout: "complete": 1 2023-05-31T20:10:28.447 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:10:28.447 INFO:teuthology.orchestra.run.smithi099.stdout: "current_time": "2023-05-31T20:10:28Z" 2023-05-31T20:10:28.448 INFO:teuthology.orchestra.run.smithi099.stdout:} 2023-05-31T20:10:28.449 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "2e6123d2-d695-4350-9a01-61a455bc782b", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1685562631.516089_59.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-05-31T19:50:31.516089Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1685562970.274572_93.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-05-31T19:56:10.274572Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1685563571.700188_212.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-05-31T20:06:11.700188Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-05-31T20:10:28Z" } 2023-05-31T20:10:28.449 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1685562631.516089_59.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-05-31T19:50:31.516089Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1685562970.274572_93.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-05-31T19:56:10.274572Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1685563571.700188_212.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-05-31T20:06:11.700188Z', 'realm_epoch': 2}}] 2023-05-31T20:10:28.450 DEBUG:rgw_multi.tests:log_status={0: '1_1685562631.516089_59.1', 1: '', 2: '1_1685562970.274572_93.1', 3: '1_1685563802.468126_296.1'} 2023-05-31T20:10:28.450 DEBUG:rgw_multi.tests:sync_status={0: '1_1685562631.516089_59.1', 1: '', 2: '1_1685562970.274572_93.1', 3: '1_1685563571.700188_212.1'} 2023-05-31T20:10:28.450 WARNING:rgw_multi.tests:zone a2 behind master: shard=3 master=1_1685563802.468126_296.1 target=1_1685563571.700188_212.1 2023-05-31T20:10:33.451 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-31T20:10:33.452 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-31T20:10:33.452 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-05-31T20:10:33.511 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2023-05-31T20:10:33.511 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2023-05-31T20:10:33.562 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:10:33.557+0000 7f31b188c780 20 RGW-SYNC:meta: read sync status 2023-05-31T20:10:33.563 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:10:33.558+0000 7f31b188c780 20 RGW-SYNC:meta: read sync status 2023-05-31T20:10:33.584 INFO:teuthology.orchestra.run.smithi099.stdout:{ 2023-05-31T20:10:33.584 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_status": { 2023-05-31T20:10:33.584 INFO:teuthology.orchestra.run.smithi099.stdout: "info": { 2023-05-31T20:10:33.584 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "sync", 2023-05-31T20:10:33.585 INFO:teuthology.orchestra.run.smithi099.stdout: "num_shards": 4, 2023-05-31T20:10:33.585 INFO:teuthology.orchestra.run.smithi099.stdout: "period": "2e6123d2-d695-4350-9a01-61a455bc782b", 2023-05-31T20:10:33.585 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2023-05-31T20:10:33.585 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:10:33.585 INFO:teuthology.orchestra.run.smithi099.stdout: "markers": [ 2023-05-31T20:10:33.585 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T20:10:33.586 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 0, 2023-05-31T20:10:33.586 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T20:10:33.586 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2023-05-31T20:10:33.586 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1685562631.516089_59.1", 2023-05-31T20:10:33.586 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T20:10:33.586 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 1, 2023-05-31T20:10:33.587 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T20:10:33.587 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2023-05-31T19:50:31.516089Z", 2023-05-31T20:10:33.587 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2023-05-31T20:10:33.587 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T20:10:33.587 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:10:33.587 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T20:10:33.588 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 1, 2023-05-31T20:10:33.588 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T20:10:33.588 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2023-05-31T20:10:33.588 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "", 2023-05-31T20:10:33.589 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T20:10:33.589 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2023-05-31T20:10:33.589 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T20:10:33.589 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "0.000000", 2023-05-31T20:10:33.590 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2023-05-31T20:10:33.590 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T20:10:33.590 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:10:33.590 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T20:10:33.590 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 2, 2023-05-31T20:10:33.590 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T20:10:33.591 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2023-05-31T20:10:33.591 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1685562970.274572_93.1", 2023-05-31T20:10:33.591 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T20:10:33.591 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2023-05-31T20:10:33.591 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T20:10:33.591 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2023-05-31T19:56:10.274572Z", 2023-05-31T20:10:33.591 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2023-05-31T20:10:33.592 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T20:10:33.592 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:10:33.592 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T20:10:33.592 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 3, 2023-05-31T20:10:33.593 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T20:10:33.593 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2023-05-31T20:10:33.593 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1685563571.700188_212.1", 2023-05-31T20:10:33.593 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T20:10:33.594 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2023-05-31T20:10:33.594 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T20:10:33.594 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2023-05-31T20:06:11.700188Z", 2023-05-31T20:10:33.594 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2023-05-31T20:10:33.594 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T20:10:33.594 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T20:10:33.594 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2023-05-31T20:10:33.595 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:10:33.595 INFO:teuthology.orchestra.run.smithi099.stdout: "full_sync": { 2023-05-31T20:10:33.595 INFO:teuthology.orchestra.run.smithi099.stdout: "total": 1, 2023-05-31T20:10:33.595 INFO:teuthology.orchestra.run.smithi099.stdout: "complete": 1 2023-05-31T20:10:33.595 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:10:33.595 INFO:teuthology.orchestra.run.smithi099.stdout: "current_time": "2023-05-31T20:10:33Z" 2023-05-31T20:10:33.595 INFO:teuthology.orchestra.run.smithi099.stdout:} 2023-05-31T20:10:33.597 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "2e6123d2-d695-4350-9a01-61a455bc782b", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1685562631.516089_59.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-05-31T19:50:31.516089Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1685562970.274572_93.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-05-31T19:56:10.274572Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1685563571.700188_212.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-05-31T20:06:11.700188Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-05-31T20:10:33Z" } 2023-05-31T20:10:33.597 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1685562631.516089_59.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-05-31T19:50:31.516089Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1685562970.274572_93.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-05-31T19:56:10.274572Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1685563571.700188_212.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-05-31T20:06:11.700188Z', 'realm_epoch': 2}}] 2023-05-31T20:10:33.598 DEBUG:rgw_multi.tests:log_status={0: '1_1685562631.516089_59.1', 1: '', 2: '1_1685562970.274572_93.1', 3: '1_1685563802.468126_296.1'} 2023-05-31T20:10:33.598 DEBUG:rgw_multi.tests:sync_status={0: '1_1685562631.516089_59.1', 1: '', 2: '1_1685562970.274572_93.1', 3: '1_1685563571.700188_212.1'} 2023-05-31T20:10:33.599 WARNING:rgw_multi.tests:zone a2 behind master: shard=3 master=1_1685563802.468126_296.1 target=1_1685563571.700188_212.1 2023-05-31T20:10:38.599 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-31T20:10:38.600 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-31T20:10:38.601 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-05-31T20:10:38.658 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2023-05-31T20:10:38.659 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2023-05-31T20:10:38.694 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:10:38.688+0000 7f54dbcab780 20 RGW-SYNC:meta: read sync status 2023-05-31T20:10:38.694 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:10:38.689+0000 7f54dbcab780 20 RGW-SYNC:meta: read sync status 2023-05-31T20:10:38.708 INFO:teuthology.orchestra.run.smithi099.stdout:{ 2023-05-31T20:10:38.708 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_status": { 2023-05-31T20:10:38.708 INFO:teuthology.orchestra.run.smithi099.stdout: "info": { 2023-05-31T20:10:38.708 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "sync", 2023-05-31T20:10:38.709 INFO:teuthology.orchestra.run.smithi099.stdout: "num_shards": 4, 2023-05-31T20:10:38.709 INFO:teuthology.orchestra.run.smithi099.stdout: "period": "2e6123d2-d695-4350-9a01-61a455bc782b", 2023-05-31T20:10:38.710 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2023-05-31T20:10:38.710 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:10:38.710 INFO:teuthology.orchestra.run.smithi099.stdout: "markers": [ 2023-05-31T20:10:38.710 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T20:10:38.710 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 0, 2023-05-31T20:10:38.711 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T20:10:38.711 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2023-05-31T20:10:38.711 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1685562631.516089_59.1", 2023-05-31T20:10:38.711 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T20:10:38.711 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 1, 2023-05-31T20:10:38.711 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T20:10:38.712 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2023-05-31T19:50:31.516089Z", 2023-05-31T20:10:38.712 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2023-05-31T20:10:38.712 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T20:10:38.712 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:10:38.712 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T20:10:38.712 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 1, 2023-05-31T20:10:38.713 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T20:10:38.714 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2023-05-31T20:10:38.715 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "", 2023-05-31T20:10:38.715 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T20:10:38.715 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2023-05-31T20:10:38.715 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T20:10:38.715 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "0.000000", 2023-05-31T20:10:38.715 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2023-05-31T20:10:38.716 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T20:10:38.716 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:10:38.716 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T20:10:38.716 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 2, 2023-05-31T20:10:38.716 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T20:10:38.716 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2023-05-31T20:10:38.716 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1685562970.274572_93.1", 2023-05-31T20:10:38.717 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T20:10:38.717 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2023-05-31T20:10:38.717 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T20:10:38.717 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2023-05-31T19:56:10.274572Z", 2023-05-31T20:10:38.717 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2023-05-31T20:10:38.717 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T20:10:38.718 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:10:38.718 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T20:10:38.718 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 3, 2023-05-31T20:10:38.718 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T20:10:38.718 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2023-05-31T20:10:38.718 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1685563571.700188_212.1", 2023-05-31T20:10:38.718 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T20:10:38.719 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2023-05-31T20:10:38.719 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T20:10:38.719 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2023-05-31T20:06:11.700188Z", 2023-05-31T20:10:38.719 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2023-05-31T20:10:38.719 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T20:10:38.719 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T20:10:38.719 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2023-05-31T20:10:38.720 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:10:38.720 INFO:teuthology.orchestra.run.smithi099.stdout: "full_sync": { 2023-05-31T20:10:38.720 INFO:teuthology.orchestra.run.smithi099.stdout: "total": 1, 2023-05-31T20:10:38.720 INFO:teuthology.orchestra.run.smithi099.stdout: "complete": 1 2023-05-31T20:10:38.720 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:10:38.720 INFO:teuthology.orchestra.run.smithi099.stdout: "current_time": "2023-05-31T20:10:38Z" 2023-05-31T20:10:38.720 INFO:teuthology.orchestra.run.smithi099.stdout:} 2023-05-31T20:10:38.722 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "2e6123d2-d695-4350-9a01-61a455bc782b", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1685562631.516089_59.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-05-31T19:50:31.516089Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1685562970.274572_93.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-05-31T19:56:10.274572Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1685563571.700188_212.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-05-31T20:06:11.700188Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-05-31T20:10:38Z" } 2023-05-31T20:10:38.722 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1685562631.516089_59.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-05-31T19:50:31.516089Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1685562970.274572_93.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-05-31T19:56:10.274572Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1685563571.700188_212.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-05-31T20:06:11.700188Z', 'realm_epoch': 2}}] 2023-05-31T20:10:38.723 DEBUG:rgw_multi.tests:log_status={0: '1_1685562631.516089_59.1', 1: '', 2: '1_1685562970.274572_93.1', 3: '1_1685563802.468126_296.1'} 2023-05-31T20:10:38.723 DEBUG:rgw_multi.tests:sync_status={0: '1_1685562631.516089_59.1', 1: '', 2: '1_1685562970.274572_93.1', 3: '1_1685563571.700188_212.1'} 2023-05-31T20:10:38.723 WARNING:rgw_multi.tests:zone a2 behind master: shard=3 master=1_1685563802.468126_296.1 target=1_1685563571.700188_212.1 2023-05-31T20:10:43.725 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-31T20:10:43.725 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-31T20:10:43.726 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-05-31T20:10:43.783 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2023-05-31T20:10:43.784 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2023-05-31T20:10:43.835 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:10:43.829+0000 7f02164ea780 20 RGW-SYNC:meta: read sync status 2023-05-31T20:10:43.836 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:10:43.830+0000 7f02164ea780 20 RGW-SYNC:meta: read sync status 2023-05-31T20:10:43.859 INFO:teuthology.orchestra.run.smithi099.stdout:{ 2023-05-31T20:10:43.859 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_status": { 2023-05-31T20:10:43.860 INFO:teuthology.orchestra.run.smithi099.stdout: "info": { 2023-05-31T20:10:43.860 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "sync", 2023-05-31T20:10:43.860 INFO:teuthology.orchestra.run.smithi099.stdout: "num_shards": 4, 2023-05-31T20:10:43.861 INFO:teuthology.orchestra.run.smithi099.stdout: "period": "2e6123d2-d695-4350-9a01-61a455bc782b", 2023-05-31T20:10:43.861 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2023-05-31T20:10:43.862 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:10:43.862 INFO:teuthology.orchestra.run.smithi099.stdout: "markers": [ 2023-05-31T20:10:43.868 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T20:10:43.868 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 0, 2023-05-31T20:10:43.868 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T20:10:43.868 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2023-05-31T20:10:43.868 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1685562631.516089_59.1", 2023-05-31T20:10:43.868 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T20:10:43.869 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 1, 2023-05-31T20:10:43.869 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T20:10:43.869 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2023-05-31T19:50:31.516089Z", 2023-05-31T20:10:43.869 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2023-05-31T20:10:43.869 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T20:10:43.869 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:10:43.870 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T20:10:43.870 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 1, 2023-05-31T20:10:43.870 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T20:10:43.870 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2023-05-31T20:10:43.870 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "", 2023-05-31T20:10:43.870 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T20:10:43.870 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2023-05-31T20:10:43.871 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T20:10:43.871 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "0.000000", 2023-05-31T20:10:43.871 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2023-05-31T20:10:43.871 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T20:10:43.871 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:10:43.872 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T20:10:43.872 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 2, 2023-05-31T20:10:43.872 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T20:10:43.872 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2023-05-31T20:10:43.872 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1685562970.274572_93.1", 2023-05-31T20:10:43.872 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T20:10:43.873 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2023-05-31T20:10:43.873 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T20:10:43.873 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2023-05-31T19:56:10.274572Z", 2023-05-31T20:10:43.873 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2023-05-31T20:10:43.873 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T20:10:43.873 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:10:43.873 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T20:10:43.874 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 3, 2023-05-31T20:10:43.874 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T20:10:43.874 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2023-05-31T20:10:43.874 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1685563571.700188_212.1", 2023-05-31T20:10:43.874 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T20:10:43.874 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2023-05-31T20:10:43.874 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T20:10:43.875 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2023-05-31T20:06:11.700188Z", 2023-05-31T20:10:43.875 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2023-05-31T20:10:43.875 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T20:10:43.875 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T20:10:43.875 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2023-05-31T20:10:43.875 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:10:43.875 INFO:teuthology.orchestra.run.smithi099.stdout: "full_sync": { 2023-05-31T20:10:43.876 INFO:teuthology.orchestra.run.smithi099.stdout: "total": 1, 2023-05-31T20:10:43.876 INFO:teuthology.orchestra.run.smithi099.stdout: "complete": 1 2023-05-31T20:10:43.876 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:10:43.876 INFO:teuthology.orchestra.run.smithi099.stdout: "current_time": "2023-05-31T20:10:43Z" 2023-05-31T20:10:43.876 INFO:teuthology.orchestra.run.smithi099.stdout:} 2023-05-31T20:10:43.877 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "2e6123d2-d695-4350-9a01-61a455bc782b", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1685562631.516089_59.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-05-31T19:50:31.516089Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1685562970.274572_93.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-05-31T19:56:10.274572Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1685563571.700188_212.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-05-31T20:06:11.700188Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-05-31T20:10:43Z" } 2023-05-31T20:10:43.878 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1685562631.516089_59.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-05-31T19:50:31.516089Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1685562970.274572_93.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-05-31T19:56:10.274572Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1685563571.700188_212.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-05-31T20:06:11.700188Z', 'realm_epoch': 2}}] 2023-05-31T20:10:43.878 DEBUG:rgw_multi.tests:log_status={0: '1_1685562631.516089_59.1', 1: '', 2: '1_1685562970.274572_93.1', 3: '1_1685563802.468126_296.1'} 2023-05-31T20:10:43.879 DEBUG:rgw_multi.tests:sync_status={0: '1_1685562631.516089_59.1', 1: '', 2: '1_1685562970.274572_93.1', 3: '1_1685563571.700188_212.1'} 2023-05-31T20:10:43.879 WARNING:rgw_multi.tests:zone a2 behind master: shard=3 master=1_1685563802.468126_296.1 target=1_1685563571.700188_212.1 2023-05-31T20:10:48.880 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-31T20:10:48.881 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-31T20:10:48.881 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-05-31T20:10:48.939 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2023-05-31T20:10:48.940 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2023-05-31T20:10:48.983 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:10:48.977+0000 7f7b91c2b780 20 RGW-SYNC:meta: read sync status 2023-05-31T20:10:48.983 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:10:48.978+0000 7f7b91c2b780 20 RGW-SYNC:meta: read sync status 2023-05-31T20:10:48.997 INFO:teuthology.orchestra.run.smithi099.stdout:{ 2023-05-31T20:10:48.998 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_status": { 2023-05-31T20:10:48.999 INFO:teuthology.orchestra.run.smithi099.stdout: "info": { 2023-05-31T20:10:48.999 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "sync", 2023-05-31T20:10:49.000 INFO:teuthology.orchestra.run.smithi099.stdout: "num_shards": 4, 2023-05-31T20:10:49.000 INFO:teuthology.orchestra.run.smithi099.stdout: "period": "2e6123d2-d695-4350-9a01-61a455bc782b", 2023-05-31T20:10:49.001 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2023-05-31T20:10:49.001 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:10:49.002 INFO:teuthology.orchestra.run.smithi099.stdout: "markers": [ 2023-05-31T20:10:49.002 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T20:10:49.003 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 0, 2023-05-31T20:10:49.003 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T20:10:49.004 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2023-05-31T20:10:49.004 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1685562631.516089_59.1", 2023-05-31T20:10:49.005 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T20:10:49.005 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 1, 2023-05-31T20:10:49.006 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T20:10:49.006 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2023-05-31T19:50:31.516089Z", 2023-05-31T20:10:49.007 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2023-05-31T20:10:49.007 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T20:10:49.008 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:10:49.008 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T20:10:49.008 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 1, 2023-05-31T20:10:49.009 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T20:10:49.009 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2023-05-31T20:10:49.010 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "", 2023-05-31T20:10:49.010 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T20:10:49.010 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2023-05-31T20:10:49.011 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T20:10:49.011 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "0.000000", 2023-05-31T20:10:49.011 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2023-05-31T20:10:49.012 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T20:10:49.012 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:10:49.012 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T20:10:49.013 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 2, 2023-05-31T20:10:49.013 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T20:10:49.013 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2023-05-31T20:10:49.014 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1685562970.274572_93.1", 2023-05-31T20:10:49.014 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T20:10:49.014 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2023-05-31T20:10:49.014 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T20:10:49.015 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2023-05-31T19:56:10.274572Z", 2023-05-31T20:10:49.015 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2023-05-31T20:10:49.015 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T20:10:49.016 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:10:49.016 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T20:10:49.016 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 3, 2023-05-31T20:10:49.017 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T20:10:49.017 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2023-05-31T20:10:49.017 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1685563571.700188_212.1", 2023-05-31T20:10:49.018 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T20:10:49.018 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2023-05-31T20:10:49.018 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T20:10:49.019 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2023-05-31T20:06:11.700188Z", 2023-05-31T20:10:49.019 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2023-05-31T20:10:49.019 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T20:10:49.019 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T20:10:49.020 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2023-05-31T20:10:49.020 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:10:49.020 INFO:teuthology.orchestra.run.smithi099.stdout: "full_sync": { 2023-05-31T20:10:49.021 INFO:teuthology.orchestra.run.smithi099.stdout: "total": 1, 2023-05-31T20:10:49.021 INFO:teuthology.orchestra.run.smithi099.stdout: "complete": 1 2023-05-31T20:10:49.021 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:10:49.021 INFO:teuthology.orchestra.run.smithi099.stdout: "current_time": "2023-05-31T20:10:48Z" 2023-05-31T20:10:49.022 INFO:teuthology.orchestra.run.smithi099.stdout:} 2023-05-31T20:10:49.023 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "2e6123d2-d695-4350-9a01-61a455bc782b", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1685562631.516089_59.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-05-31T19:50:31.516089Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1685562970.274572_93.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-05-31T19:56:10.274572Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1685563571.700188_212.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-05-31T20:06:11.700188Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-05-31T20:10:48Z" } 2023-05-31T20:10:49.024 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1685562631.516089_59.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-05-31T19:50:31.516089Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1685562970.274572_93.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-05-31T19:56:10.274572Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1685563571.700188_212.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-05-31T20:06:11.700188Z', 'realm_epoch': 2}}] 2023-05-31T20:10:49.025 DEBUG:rgw_multi.tests:log_status={0: '1_1685562631.516089_59.1', 1: '', 2: '1_1685562970.274572_93.1', 3: '1_1685563802.468126_296.1'} 2023-05-31T20:10:49.026 DEBUG:rgw_multi.tests:sync_status={0: '1_1685562631.516089_59.1', 1: '', 2: '1_1685562970.274572_93.1', 3: '1_1685563571.700188_212.1'} 2023-05-31T20:10:49.026 WARNING:rgw_multi.tests:zone a2 behind master: shard=3 master=1_1685563802.468126_296.1 target=1_1685563571.700188_212.1 2023-05-31T20:10:54.027 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-31T20:10:54.028 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-31T20:10:54.028 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-05-31T20:10:54.086 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2023-05-31T20:10:54.086 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2023-05-31T20:10:54.122 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:10:54.114+0000 7f01ff776780 20 RGW-SYNC:meta: read sync status 2023-05-31T20:10:54.123 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:10:54.115+0000 7f01ff776780 20 RGW-SYNC:meta: read sync status 2023-05-31T20:10:54.141 INFO:teuthology.orchestra.run.smithi099.stdout:{ 2023-05-31T20:10:54.142 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_status": { 2023-05-31T20:10:54.142 INFO:teuthology.orchestra.run.smithi099.stdout: "info": { 2023-05-31T20:10:54.142 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "sync", 2023-05-31T20:10:54.142 INFO:teuthology.orchestra.run.smithi099.stdout: "num_shards": 4, 2023-05-31T20:10:54.143 INFO:teuthology.orchestra.run.smithi099.stdout: "period": "2e6123d2-d695-4350-9a01-61a455bc782b", 2023-05-31T20:10:54.143 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2023-05-31T20:10:54.143 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:10:54.143 INFO:teuthology.orchestra.run.smithi099.stdout: "markers": [ 2023-05-31T20:10:54.143 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T20:10:54.143 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 0, 2023-05-31T20:10:54.144 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T20:10:54.144 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2023-05-31T20:10:54.144 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1685562631.516089_59.1", 2023-05-31T20:10:54.144 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T20:10:54.144 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 1, 2023-05-31T20:10:54.144 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T20:10:54.144 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2023-05-31T19:50:31.516089Z", 2023-05-31T20:10:54.145 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2023-05-31T20:10:54.145 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T20:10:54.145 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:10:54.145 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T20:10:54.145 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 1, 2023-05-31T20:10:54.145 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T20:10:54.146 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2023-05-31T20:10:54.146 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "", 2023-05-31T20:10:54.146 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T20:10:54.146 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2023-05-31T20:10:54.146 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T20:10:54.146 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "0.000000", 2023-05-31T20:10:54.146 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2023-05-31T20:10:54.147 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T20:10:54.147 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:10:54.147 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T20:10:54.147 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 2, 2023-05-31T20:10:54.148 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T20:10:54.148 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2023-05-31T20:10:54.148 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1685562970.274572_93.1", 2023-05-31T20:10:54.148 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T20:10:54.149 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2023-05-31T20:10:54.149 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T20:10:54.149 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2023-05-31T19:56:10.274572Z", 2023-05-31T20:10:54.150 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2023-05-31T20:10:54.150 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T20:10:54.150 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:10:54.150 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T20:10:54.150 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 3, 2023-05-31T20:10:54.150 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T20:10:54.151 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2023-05-31T20:10:54.151 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1685563571.700188_212.1", 2023-05-31T20:10:54.151 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T20:10:54.151 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2023-05-31T20:10:54.151 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T20:10:54.151 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2023-05-31T20:06:11.700188Z", 2023-05-31T20:10:54.151 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2023-05-31T20:10:54.152 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T20:10:54.152 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T20:10:54.152 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2023-05-31T20:10:54.152 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:10:54.153 INFO:teuthology.orchestra.run.smithi099.stdout: "full_sync": { 2023-05-31T20:10:54.153 INFO:teuthology.orchestra.run.smithi099.stdout: "total": 1, 2023-05-31T20:10:54.153 INFO:teuthology.orchestra.run.smithi099.stdout: "complete": 1 2023-05-31T20:10:54.153 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:10:54.153 INFO:teuthology.orchestra.run.smithi099.stdout: "current_time": "2023-05-31T20:10:54Z" 2023-05-31T20:10:54.153 INFO:teuthology.orchestra.run.smithi099.stdout:} 2023-05-31T20:10:54.155 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "2e6123d2-d695-4350-9a01-61a455bc782b", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1685562631.516089_59.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-05-31T19:50:31.516089Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1685562970.274572_93.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-05-31T19:56:10.274572Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1685563571.700188_212.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-05-31T20:06:11.700188Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-05-31T20:10:54Z" } 2023-05-31T20:10:54.155 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1685562631.516089_59.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-05-31T19:50:31.516089Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1685562970.274572_93.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-05-31T19:56:10.274572Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1685563571.700188_212.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-05-31T20:06:11.700188Z', 'realm_epoch': 2}}] 2023-05-31T20:10:54.156 DEBUG:rgw_multi.tests:log_status={0: '1_1685562631.516089_59.1', 1: '', 2: '1_1685562970.274572_93.1', 3: '1_1685563802.468126_296.1'} 2023-05-31T20:10:54.156 DEBUG:rgw_multi.tests:sync_status={0: '1_1685562631.516089_59.1', 1: '', 2: '1_1685562970.274572_93.1', 3: '1_1685563571.700188_212.1'} 2023-05-31T20:10:54.157 WARNING:rgw_multi.tests:zone a2 behind master: shard=3 master=1_1685563802.468126_296.1 target=1_1685563571.700188_212.1 2023-05-31T20:10:59.158 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-31T20:10:59.160 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-31T20:10:59.161 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-05-31T20:10:59.219 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2023-05-31T20:10:59.220 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2023-05-31T20:10:59.272 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:10:59.266+0000 7f98ef4a8780 20 RGW-SYNC:meta: read sync status 2023-05-31T20:10:59.273 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:10:59.267+0000 7f98ef4a8780 20 RGW-SYNC:meta: read sync status 2023-05-31T20:10:59.288 INFO:teuthology.orchestra.run.smithi099.stdout:{ 2023-05-31T20:10:59.288 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_status": { 2023-05-31T20:10:59.288 INFO:teuthology.orchestra.run.smithi099.stdout: "info": { 2023-05-31T20:10:59.288 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "sync", 2023-05-31T20:10:59.289 INFO:teuthology.orchestra.run.smithi099.stdout: "num_shards": 4, 2023-05-31T20:10:59.289 INFO:teuthology.orchestra.run.smithi099.stdout: "period": "2e6123d2-d695-4350-9a01-61a455bc782b", 2023-05-31T20:10:59.289 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2023-05-31T20:10:59.289 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:10:59.289 INFO:teuthology.orchestra.run.smithi099.stdout: "markers": [ 2023-05-31T20:10:59.289 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T20:10:59.289 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 0, 2023-05-31T20:10:59.290 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T20:10:59.290 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2023-05-31T20:10:59.290 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1685562631.516089_59.1", 2023-05-31T20:10:59.290 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T20:10:59.290 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 1, 2023-05-31T20:10:59.290 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T20:10:59.290 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2023-05-31T19:50:31.516089Z", 2023-05-31T20:10:59.291 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2023-05-31T20:10:59.291 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T20:10:59.291 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:10:59.291 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T20:10:59.291 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 1, 2023-05-31T20:10:59.291 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T20:10:59.291 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2023-05-31T20:10:59.292 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "", 2023-05-31T20:10:59.292 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T20:10:59.292 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2023-05-31T20:10:59.292 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T20:10:59.292 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "0.000000", 2023-05-31T20:10:59.292 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2023-05-31T20:10:59.293 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T20:10:59.293 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:10:59.293 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T20:10:59.293 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 2, 2023-05-31T20:10:59.293 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T20:10:59.293 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2023-05-31T20:10:59.293 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1685562970.274572_93.1", 2023-05-31T20:10:59.294 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T20:10:59.294 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2023-05-31T20:10:59.294 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T20:10:59.294 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2023-05-31T19:56:10.274572Z", 2023-05-31T20:10:59.294 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2023-05-31T20:10:59.294 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T20:10:59.294 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:10:59.295 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T20:10:59.295 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 3, 2023-05-31T20:10:59.295 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T20:10:59.295 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2023-05-31T20:10:59.295 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1685563571.700188_212.1", 2023-05-31T20:10:59.295 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T20:10:59.296 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2023-05-31T20:10:59.296 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T20:10:59.296 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2023-05-31T20:06:11.700188Z", 2023-05-31T20:10:59.296 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2023-05-31T20:10:59.296 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T20:10:59.296 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T20:10:59.297 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2023-05-31T20:10:59.297 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:10:59.297 INFO:teuthology.orchestra.run.smithi099.stdout: "full_sync": { 2023-05-31T20:10:59.297 INFO:teuthology.orchestra.run.smithi099.stdout: "total": 1, 2023-05-31T20:10:59.297 INFO:teuthology.orchestra.run.smithi099.stdout: "complete": 1 2023-05-31T20:10:59.298 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:10:59.298 INFO:teuthology.orchestra.run.smithi099.stdout: "current_time": "2023-05-31T20:10:59Z" 2023-05-31T20:10:59.298 INFO:teuthology.orchestra.run.smithi099.stdout:} 2023-05-31T20:10:59.299 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "2e6123d2-d695-4350-9a01-61a455bc782b", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1685562631.516089_59.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-05-31T19:50:31.516089Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1685562970.274572_93.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-05-31T19:56:10.274572Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1685563571.700188_212.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-05-31T20:06:11.700188Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-05-31T20:10:59Z" } 2023-05-31T20:10:59.299 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1685562631.516089_59.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-05-31T19:50:31.516089Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1685562970.274572_93.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-05-31T19:56:10.274572Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1685563571.700188_212.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-05-31T20:06:11.700188Z', 'realm_epoch': 2}}] 2023-05-31T20:10:59.300 DEBUG:rgw_multi.tests:log_status={0: '1_1685562631.516089_59.1', 1: '', 2: '1_1685562970.274572_93.1', 3: '1_1685563802.468126_296.1'} 2023-05-31T20:10:59.300 DEBUG:rgw_multi.tests:sync_status={0: '1_1685562631.516089_59.1', 1: '', 2: '1_1685562970.274572_93.1', 3: '1_1685563571.700188_212.1'} 2023-05-31T20:10:59.300 WARNING:rgw_multi.tests:zone a2 behind master: shard=3 master=1_1685563802.468126_296.1 target=1_1685563571.700188_212.1 2023-05-31T20:11:04.301 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-31T20:11:04.302 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-31T20:11:04.302 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-05-31T20:11:04.359 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2023-05-31T20:11:04.360 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2023-05-31T20:11:04.393 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:11:04.389+0000 7f92045d6780 20 RGW-SYNC:meta: read sync status 2023-05-31T20:11:04.394 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:11:04.390+0000 7f92045d6780 20 RGW-SYNC:meta: read sync status 2023-05-31T20:11:04.407 INFO:teuthology.orchestra.run.smithi099.stdout:{ 2023-05-31T20:11:04.408 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_status": { 2023-05-31T20:11:04.408 INFO:teuthology.orchestra.run.smithi099.stdout: "info": { 2023-05-31T20:11:04.408 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "sync", 2023-05-31T20:11:04.408 INFO:teuthology.orchestra.run.smithi099.stdout: "num_shards": 4, 2023-05-31T20:11:04.409 INFO:teuthology.orchestra.run.smithi099.stdout: "period": "2e6123d2-d695-4350-9a01-61a455bc782b", 2023-05-31T20:11:04.409 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2023-05-31T20:11:04.409 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:11:04.409 INFO:teuthology.orchestra.run.smithi099.stdout: "markers": [ 2023-05-31T20:11:04.409 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T20:11:04.409 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 0, 2023-05-31T20:11:04.410 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T20:11:04.410 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2023-05-31T20:11:04.410 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1685562631.516089_59.1", 2023-05-31T20:11:04.410 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T20:11:04.410 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 1, 2023-05-31T20:11:04.410 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T20:11:04.410 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2023-05-31T19:50:31.516089Z", 2023-05-31T20:11:04.411 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2023-05-31T20:11:04.411 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T20:11:04.411 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:11:04.411 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T20:11:04.411 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 1, 2023-05-31T20:11:04.411 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T20:11:04.411 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2023-05-31T20:11:04.412 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "", 2023-05-31T20:11:04.412 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T20:11:04.412 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2023-05-31T20:11:04.413 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T20:11:04.413 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "0.000000", 2023-05-31T20:11:04.413 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2023-05-31T20:11:04.413 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T20:11:04.413 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:11:04.413 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T20:11:04.414 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 2, 2023-05-31T20:11:04.414 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T20:11:04.414 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2023-05-31T20:11:04.414 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1685562970.274572_93.1", 2023-05-31T20:11:04.414 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T20:11:04.414 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2023-05-31T20:11:04.415 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T20:11:04.415 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2023-05-31T19:56:10.274572Z", 2023-05-31T20:11:04.415 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2023-05-31T20:11:04.415 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T20:11:04.415 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:11:04.415 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T20:11:04.416 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 3, 2023-05-31T20:11:04.416 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T20:11:04.416 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2023-05-31T20:11:04.416 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1685563571.700188_212.1", 2023-05-31T20:11:04.416 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T20:11:04.416 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2023-05-31T20:11:04.417 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T20:11:04.417 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2023-05-31T20:06:11.700188Z", 2023-05-31T20:11:04.417 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2023-05-31T20:11:04.417 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T20:11:04.417 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T20:11:04.417 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2023-05-31T20:11:04.417 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:11:04.418 INFO:teuthology.orchestra.run.smithi099.stdout: "full_sync": { 2023-05-31T20:11:04.418 INFO:teuthology.orchestra.run.smithi099.stdout: "total": 1, 2023-05-31T20:11:04.418 INFO:teuthology.orchestra.run.smithi099.stdout: "complete": 1 2023-05-31T20:11:04.418 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:11:04.418 INFO:teuthology.orchestra.run.smithi099.stdout: "current_time": "2023-05-31T20:11:04Z" 2023-05-31T20:11:04.418 INFO:teuthology.orchestra.run.smithi099.stdout:} 2023-05-31T20:11:04.419 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "2e6123d2-d695-4350-9a01-61a455bc782b", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1685562631.516089_59.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-05-31T19:50:31.516089Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1685562970.274572_93.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-05-31T19:56:10.274572Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1685563571.700188_212.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-05-31T20:06:11.700188Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-05-31T20:11:04Z" } 2023-05-31T20:11:04.420 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1685562631.516089_59.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-05-31T19:50:31.516089Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1685562970.274572_93.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-05-31T19:56:10.274572Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1685563571.700188_212.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-05-31T20:06:11.700188Z', 'realm_epoch': 2}}] 2023-05-31T20:11:04.420 DEBUG:rgw_multi.tests:log_status={0: '1_1685562631.516089_59.1', 1: '', 2: '1_1685562970.274572_93.1', 3: '1_1685563802.468126_296.1'} 2023-05-31T20:11:04.421 DEBUG:rgw_multi.tests:sync_status={0: '1_1685562631.516089_59.1', 1: '', 2: '1_1685562970.274572_93.1', 3: '1_1685563571.700188_212.1'} 2023-05-31T20:11:04.421 WARNING:rgw_multi.tests:zone a2 behind master: shard=3 master=1_1685563802.468126_296.1 target=1_1685563571.700188_212.1 2023-05-31T20:11:09.422 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-31T20:11:09.423 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-31T20:11:09.423 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-05-31T20:11:09.480 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2023-05-31T20:11:09.480 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2023-05-31T20:11:09.530 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:11:09.526+0000 7f7e4818f780 20 RGW-SYNC:meta: read sync status 2023-05-31T20:11:09.531 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:11:09.527+0000 7f7e4818f780 20 RGW-SYNC:meta: read sync status 2023-05-31T20:11:09.553 INFO:teuthology.orchestra.run.smithi099.stdout:{ 2023-05-31T20:11:09.554 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_status": { 2023-05-31T20:11:09.554 INFO:teuthology.orchestra.run.smithi099.stdout: "info": { 2023-05-31T20:11:09.555 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "sync", 2023-05-31T20:11:09.555 INFO:teuthology.orchestra.run.smithi099.stdout: "num_shards": 4, 2023-05-31T20:11:09.555 INFO:teuthology.orchestra.run.smithi099.stdout: "period": "2e6123d2-d695-4350-9a01-61a455bc782b", 2023-05-31T20:11:09.556 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2023-05-31T20:11:09.556 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:11:09.556 INFO:teuthology.orchestra.run.smithi099.stdout: "markers": [ 2023-05-31T20:11:09.557 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T20:11:09.559 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 0, 2023-05-31T20:11:09.559 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T20:11:09.560 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2023-05-31T20:11:09.560 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1685562631.516089_59.1", 2023-05-31T20:11:09.560 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T20:11:09.560 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 1, 2023-05-31T20:11:09.560 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T20:11:09.560 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2023-05-31T19:50:31.516089Z", 2023-05-31T20:11:09.561 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2023-05-31T20:11:09.561 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T20:11:09.561 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:11:09.561 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T20:11:09.561 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 1, 2023-05-31T20:11:09.561 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T20:11:09.561 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2023-05-31T20:11:09.562 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "", 2023-05-31T20:11:09.562 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T20:11:09.562 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2023-05-31T20:11:09.562 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T20:11:09.562 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "0.000000", 2023-05-31T20:11:09.562 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2023-05-31T20:11:09.563 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T20:11:09.563 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:11:09.563 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T20:11:09.563 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 2, 2023-05-31T20:11:09.563 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T20:11:09.563 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2023-05-31T20:11:09.563 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1685562970.274572_93.1", 2023-05-31T20:11:09.564 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T20:11:09.564 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2023-05-31T20:11:09.564 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T20:11:09.564 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2023-05-31T19:56:10.274572Z", 2023-05-31T20:11:09.564 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2023-05-31T20:11:09.564 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T20:11:09.564 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:11:09.565 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T20:11:09.565 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 3, 2023-05-31T20:11:09.565 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T20:11:09.565 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2023-05-31T20:11:09.565 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1685563571.700188_212.1", 2023-05-31T20:11:09.565 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T20:11:09.565 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2023-05-31T20:11:09.566 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T20:11:09.566 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2023-05-31T20:06:11.700188Z", 2023-05-31T20:11:09.566 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2023-05-31T20:11:09.566 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T20:11:09.566 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T20:11:09.566 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2023-05-31T20:11:09.566 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:11:09.567 INFO:teuthology.orchestra.run.smithi099.stdout: "full_sync": { 2023-05-31T20:11:09.567 INFO:teuthology.orchestra.run.smithi099.stdout: "total": 1, 2023-05-31T20:11:09.567 INFO:teuthology.orchestra.run.smithi099.stdout: "complete": 1 2023-05-31T20:11:09.567 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:11:09.567 INFO:teuthology.orchestra.run.smithi099.stdout: "current_time": "2023-05-31T20:11:09Z" 2023-05-31T20:11:09.567 INFO:teuthology.orchestra.run.smithi099.stdout:} 2023-05-31T20:11:09.568 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "2e6123d2-d695-4350-9a01-61a455bc782b", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1685562631.516089_59.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-05-31T19:50:31.516089Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1685562970.274572_93.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-05-31T19:56:10.274572Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1685563571.700188_212.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-05-31T20:06:11.700188Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-05-31T20:11:09Z" } 2023-05-31T20:11:09.572 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1685562631.516089_59.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-05-31T19:50:31.516089Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1685562970.274572_93.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-05-31T19:56:10.274572Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1685563571.700188_212.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-05-31T20:06:11.700188Z', 'realm_epoch': 2}}] 2023-05-31T20:11:09.573 DEBUG:rgw_multi.tests:log_status={0: '1_1685562631.516089_59.1', 1: '', 2: '1_1685562970.274572_93.1', 3: '1_1685563802.468126_296.1'} 2023-05-31T20:11:09.573 DEBUG:rgw_multi.tests:sync_status={0: '1_1685562631.516089_59.1', 1: '', 2: '1_1685562970.274572_93.1', 3: '1_1685563571.700188_212.1'} 2023-05-31T20:11:09.574 WARNING:rgw_multi.tests:zone a2 behind master: shard=3 master=1_1685563802.468126_296.1 target=1_1685563571.700188_212.1 2023-05-31T20:11:14.575 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-31T20:11:14.576 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-31T20:11:14.577 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-05-31T20:11:14.634 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2023-05-31T20:11:14.635 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2023-05-31T20:11:14.670 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:11:14.666+0000 7f91656c0780 20 RGW-SYNC:meta: read sync status 2023-05-31T20:11:14.671 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:11:14.667+0000 7f91656c0780 20 RGW-SYNC:meta: read sync status 2023-05-31T20:11:14.684 INFO:teuthology.orchestra.run.smithi099.stdout:{ 2023-05-31T20:11:14.684 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_status": { 2023-05-31T20:11:14.685 INFO:teuthology.orchestra.run.smithi099.stdout: "info": { 2023-05-31T20:11:14.685 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "sync", 2023-05-31T20:11:14.685 INFO:teuthology.orchestra.run.smithi099.stdout: "num_shards": 4, 2023-05-31T20:11:14.685 INFO:teuthology.orchestra.run.smithi099.stdout: "period": "2e6123d2-d695-4350-9a01-61a455bc782b", 2023-05-31T20:11:14.685 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2023-05-31T20:11:14.685 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:11:14.685 INFO:teuthology.orchestra.run.smithi099.stdout: "markers": [ 2023-05-31T20:11:14.686 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T20:11:14.686 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 0, 2023-05-31T20:11:14.686 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T20:11:14.686 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2023-05-31T20:11:14.686 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1685562631.516089_59.1", 2023-05-31T20:11:14.686 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T20:11:14.687 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 1, 2023-05-31T20:11:14.687 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T20:11:14.687 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2023-05-31T19:50:31.516089Z", 2023-05-31T20:11:14.687 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2023-05-31T20:11:14.687 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T20:11:14.687 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:11:14.687 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T20:11:14.688 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 1, 2023-05-31T20:11:14.688 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T20:11:14.688 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2023-05-31T20:11:14.688 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "", 2023-05-31T20:11:14.688 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T20:11:14.688 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2023-05-31T20:11:14.688 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T20:11:14.689 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "0.000000", 2023-05-31T20:11:14.689 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2023-05-31T20:11:14.689 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T20:11:14.689 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:11:14.689 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T20:11:14.689 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 2, 2023-05-31T20:11:14.689 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T20:11:14.690 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2023-05-31T20:11:14.690 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1685562970.274572_93.1", 2023-05-31T20:11:14.690 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T20:11:14.690 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2023-05-31T20:11:14.690 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T20:11:14.690 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2023-05-31T19:56:10.274572Z", 2023-05-31T20:11:14.690 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2023-05-31T20:11:14.691 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T20:11:14.691 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:11:14.691 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T20:11:14.691 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 3, 2023-05-31T20:11:14.691 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T20:11:14.691 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2023-05-31T20:11:14.691 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1685563571.700188_212.1", 2023-05-31T20:11:14.692 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T20:11:14.692 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2023-05-31T20:11:14.692 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T20:11:14.692 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2023-05-31T20:06:11.700188Z", 2023-05-31T20:11:14.692 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2023-05-31T20:11:14.692 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T20:11:14.693 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T20:11:14.693 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2023-05-31T20:11:14.693 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:11:14.693 INFO:teuthology.orchestra.run.smithi099.stdout: "full_sync": { 2023-05-31T20:11:14.693 INFO:teuthology.orchestra.run.smithi099.stdout: "total": 1, 2023-05-31T20:11:14.693 INFO:teuthology.orchestra.run.smithi099.stdout: "complete": 1 2023-05-31T20:11:14.694 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:11:14.694 INFO:teuthology.orchestra.run.smithi099.stdout: "current_time": "2023-05-31T20:11:14Z" 2023-05-31T20:11:14.694 INFO:teuthology.orchestra.run.smithi099.stdout:} 2023-05-31T20:11:14.697 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "2e6123d2-d695-4350-9a01-61a455bc782b", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1685562631.516089_59.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-05-31T19:50:31.516089Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1685562970.274572_93.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-05-31T19:56:10.274572Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1685563571.700188_212.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-05-31T20:06:11.700188Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-05-31T20:11:14Z" } 2023-05-31T20:11:14.697 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1685562631.516089_59.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-05-31T19:50:31.516089Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1685562970.274572_93.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-05-31T19:56:10.274572Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1685563571.700188_212.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-05-31T20:06:11.700188Z', 'realm_epoch': 2}}] 2023-05-31T20:11:14.698 DEBUG:rgw_multi.tests:log_status={0: '1_1685562631.516089_59.1', 1: '', 2: '1_1685562970.274572_93.1', 3: '1_1685563802.468126_296.1'} 2023-05-31T20:11:14.698 DEBUG:rgw_multi.tests:sync_status={0: '1_1685562631.516089_59.1', 1: '', 2: '1_1685562970.274572_93.1', 3: '1_1685563571.700188_212.1'} 2023-05-31T20:11:14.698 WARNING:rgw_multi.tests:zone a2 behind master: shard=3 master=1_1685563802.468126_296.1 target=1_1685563571.700188_212.1 2023-05-31T20:11:19.699 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-31T20:11:19.700 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-31T20:11:19.700 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-05-31T20:11:19.758 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2023-05-31T20:11:19.758 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2023-05-31T20:11:19.812 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:11:19.807+0000 7f078c578780 20 RGW-SYNC:meta: read sync status 2023-05-31T20:11:19.813 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:11:19.808+0000 7f078c578780 20 RGW-SYNC:meta: read sync status 2023-05-31T20:11:19.837 INFO:teuthology.orchestra.run.smithi099.stdout:{ 2023-05-31T20:11:19.838 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_status": { 2023-05-31T20:11:19.839 INFO:teuthology.orchestra.run.smithi099.stdout: "info": { 2023-05-31T20:11:19.839 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "sync", 2023-05-31T20:11:19.839 INFO:teuthology.orchestra.run.smithi099.stdout: "num_shards": 4, 2023-05-31T20:11:19.840 INFO:teuthology.orchestra.run.smithi099.stdout: "period": "2e6123d2-d695-4350-9a01-61a455bc782b", 2023-05-31T20:11:19.840 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2023-05-31T20:11:19.840 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:11:19.841 INFO:teuthology.orchestra.run.smithi099.stdout: "markers": [ 2023-05-31T20:11:19.841 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T20:11:19.841 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 0, 2023-05-31T20:11:19.842 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T20:11:19.842 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2023-05-31T20:11:19.842 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1685562631.516089_59.1", 2023-05-31T20:11:19.842 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T20:11:19.843 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 1, 2023-05-31T20:11:19.843 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T20:11:19.843 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2023-05-31T19:50:31.516089Z", 2023-05-31T20:11:19.844 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2023-05-31T20:11:19.845 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T20:11:19.845 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:11:19.845 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T20:11:19.846 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 1, 2023-05-31T20:11:19.846 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T20:11:19.846 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2023-05-31T20:11:19.846 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "", 2023-05-31T20:11:19.846 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T20:11:19.847 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2023-05-31T20:11:19.847 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T20:11:19.847 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "0.000000", 2023-05-31T20:11:19.847 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2023-05-31T20:11:19.847 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T20:11:19.847 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:11:19.848 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T20:11:19.848 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 2, 2023-05-31T20:11:19.848 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T20:11:19.848 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2023-05-31T20:11:19.848 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1685562970.274572_93.1", 2023-05-31T20:11:19.848 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T20:11:19.849 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2023-05-31T20:11:19.849 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T20:11:19.849 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2023-05-31T19:56:10.274572Z", 2023-05-31T20:11:19.849 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2023-05-31T20:11:19.849 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T20:11:19.849 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:11:19.849 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T20:11:19.850 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 3, 2023-05-31T20:11:19.850 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T20:11:19.850 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2023-05-31T20:11:19.850 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1685563571.700188_212.1", 2023-05-31T20:11:19.850 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T20:11:19.850 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2023-05-31T20:11:19.850 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T20:11:19.851 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2023-05-31T20:06:11.700188Z", 2023-05-31T20:11:19.851 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2023-05-31T20:11:19.851 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T20:11:19.851 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T20:11:19.851 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2023-05-31T20:11:19.852 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:11:19.852 INFO:teuthology.orchestra.run.smithi099.stdout: "full_sync": { 2023-05-31T20:11:19.852 INFO:teuthology.orchestra.run.smithi099.stdout: "total": 1, 2023-05-31T20:11:19.852 INFO:teuthology.orchestra.run.smithi099.stdout: "complete": 1 2023-05-31T20:11:19.852 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:11:19.853 INFO:teuthology.orchestra.run.smithi099.stdout: "current_time": "2023-05-31T20:11:19Z" 2023-05-31T20:11:19.853 INFO:teuthology.orchestra.run.smithi099.stdout:} 2023-05-31T20:11:19.855 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "2e6123d2-d695-4350-9a01-61a455bc782b", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1685562631.516089_59.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-05-31T19:50:31.516089Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1685562970.274572_93.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-05-31T19:56:10.274572Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1685563571.700188_212.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-05-31T20:06:11.700188Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-05-31T20:11:19Z" } 2023-05-31T20:11:19.856 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1685562631.516089_59.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-05-31T19:50:31.516089Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1685562970.274572_93.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-05-31T19:56:10.274572Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1685563571.700188_212.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-05-31T20:06:11.700188Z', 'realm_epoch': 2}}] 2023-05-31T20:11:19.857 DEBUG:rgw_multi.tests:log_status={0: '1_1685562631.516089_59.1', 1: '', 2: '1_1685562970.274572_93.1', 3: '1_1685563802.468126_296.1'} 2023-05-31T20:11:19.858 DEBUG:rgw_multi.tests:sync_status={0: '1_1685562631.516089_59.1', 1: '', 2: '1_1685562970.274572_93.1', 3: '1_1685563571.700188_212.1'} 2023-05-31T20:11:19.859 WARNING:rgw_multi.tests:zone a2 behind master: shard=3 master=1_1685563802.468126_296.1 target=1_1685563571.700188_212.1 2023-05-31T20:11:24.861 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-31T20:11:24.861 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-31T20:11:24.862 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-05-31T20:11:24.920 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2023-05-31T20:11:24.921 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2023-05-31T20:11:24.970 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:11:24.964+0000 7f78077af780 20 RGW-SYNC:meta: read sync status 2023-05-31T20:11:24.971 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:11:24.965+0000 7f78077af780 20 RGW-SYNC:meta: read sync status 2023-05-31T20:11:24.984 INFO:teuthology.orchestra.run.smithi099.stdout:{ 2023-05-31T20:11:24.984 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_status": { 2023-05-31T20:11:24.984 INFO:teuthology.orchestra.run.smithi099.stdout: "info": { 2023-05-31T20:11:24.985 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "sync", 2023-05-31T20:11:24.985 INFO:teuthology.orchestra.run.smithi099.stdout: "num_shards": 4, 2023-05-31T20:11:24.985 INFO:teuthology.orchestra.run.smithi099.stdout: "period": "2e6123d2-d695-4350-9a01-61a455bc782b", 2023-05-31T20:11:24.985 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2023-05-31T20:11:24.985 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:11:24.986 INFO:teuthology.orchestra.run.smithi099.stdout: "markers": [ 2023-05-31T20:11:24.986 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T20:11:24.986 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 0, 2023-05-31T20:11:24.986 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T20:11:24.986 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2023-05-31T20:11:24.986 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1685562631.516089_59.1", 2023-05-31T20:11:24.986 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T20:11:24.987 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 1, 2023-05-31T20:11:24.987 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T20:11:24.987 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2023-05-31T19:50:31.516089Z", 2023-05-31T20:11:24.987 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2023-05-31T20:11:24.987 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T20:11:24.988 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:11:24.988 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T20:11:24.988 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 1, 2023-05-31T20:11:24.988 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T20:11:24.988 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2023-05-31T20:11:24.988 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "", 2023-05-31T20:11:24.988 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T20:11:24.989 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2023-05-31T20:11:24.989 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T20:11:24.989 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "0.000000", 2023-05-31T20:11:24.989 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2023-05-31T20:11:24.990 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T20:11:24.990 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:11:24.991 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T20:11:24.991 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 2, 2023-05-31T20:11:24.991 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T20:11:24.991 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2023-05-31T20:11:24.991 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1685562970.274572_93.1", 2023-05-31T20:11:24.992 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T20:11:24.992 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2023-05-31T20:11:24.992 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T20:11:24.992 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2023-05-31T19:56:10.274572Z", 2023-05-31T20:11:24.992 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2023-05-31T20:11:24.992 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T20:11:24.993 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:11:24.993 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T20:11:24.993 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 3, 2023-05-31T20:11:24.993 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T20:11:24.993 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2023-05-31T20:11:24.993 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1685563571.700188_212.1", 2023-05-31T20:11:24.993 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T20:11:24.994 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2023-05-31T20:11:24.994 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T20:11:24.994 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2023-05-31T20:06:11.700188Z", 2023-05-31T20:11:24.994 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2023-05-31T20:11:24.994 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T20:11:24.994 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T20:11:24.994 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2023-05-31T20:11:24.995 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:11:24.995 INFO:teuthology.orchestra.run.smithi099.stdout: "full_sync": { 2023-05-31T20:11:24.995 INFO:teuthology.orchestra.run.smithi099.stdout: "total": 1, 2023-05-31T20:11:24.995 INFO:teuthology.orchestra.run.smithi099.stdout: "complete": 1 2023-05-31T20:11:24.995 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:11:24.995 INFO:teuthology.orchestra.run.smithi099.stdout: "current_time": "2023-05-31T20:11:24Z" 2023-05-31T20:11:24.995 INFO:teuthology.orchestra.run.smithi099.stdout:} 2023-05-31T20:11:24.997 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "2e6123d2-d695-4350-9a01-61a455bc782b", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1685562631.516089_59.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-05-31T19:50:31.516089Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1685562970.274572_93.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-05-31T19:56:10.274572Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1685563571.700188_212.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-05-31T20:06:11.700188Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-05-31T20:11:24Z" } 2023-05-31T20:11:24.997 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1685562631.516089_59.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-05-31T19:50:31.516089Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1685562970.274572_93.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-05-31T19:56:10.274572Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1685563571.700188_212.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-05-31T20:06:11.700188Z', 'realm_epoch': 2}}] 2023-05-31T20:11:24.998 DEBUG:rgw_multi.tests:log_status={0: '1_1685562631.516089_59.1', 1: '', 2: '1_1685562970.274572_93.1', 3: '1_1685563802.468126_296.1'} 2023-05-31T20:11:24.998 DEBUG:rgw_multi.tests:sync_status={0: '1_1685562631.516089_59.1', 1: '', 2: '1_1685562970.274572_93.1', 3: '1_1685563571.700188_212.1'} 2023-05-31T20:11:24.998 WARNING:rgw_multi.tests:zone a2 behind master: shard=3 master=1_1685563802.468126_296.1 target=1_1685563571.700188_212.1 2023-05-31T20:11:29.999 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-31T20:11:30.000 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-31T20:11:30.001 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-05-31T20:11:30.057 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2023-05-31T20:11:30.058 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2023-05-31T20:11:30.097 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:11:30.091+0000 7f983a1f7780 20 RGW-SYNC:meta: read sync status 2023-05-31T20:11:30.098 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:11:30.092+0000 7f983a1f7780 20 RGW-SYNC:meta: read sync status 2023-05-31T20:11:30.117 INFO:teuthology.orchestra.run.smithi099.stdout:{ 2023-05-31T20:11:30.117 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_status": { 2023-05-31T20:11:30.117 INFO:teuthology.orchestra.run.smithi099.stdout: "info": { 2023-05-31T20:11:30.117 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "sync", 2023-05-31T20:11:30.118 INFO:teuthology.orchestra.run.smithi099.stdout: "num_shards": 4, 2023-05-31T20:11:30.118 INFO:teuthology.orchestra.run.smithi099.stdout: "period": "2e6123d2-d695-4350-9a01-61a455bc782b", 2023-05-31T20:11:30.118 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2023-05-31T20:11:30.118 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:11:30.118 INFO:teuthology.orchestra.run.smithi099.stdout: "markers": [ 2023-05-31T20:11:30.118 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T20:11:30.119 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 0, 2023-05-31T20:11:30.119 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T20:11:30.119 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2023-05-31T20:11:30.119 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1685562631.516089_59.1", 2023-05-31T20:11:30.119 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T20:11:30.119 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 1, 2023-05-31T20:11:30.120 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T20:11:30.120 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2023-05-31T19:50:31.516089Z", 2023-05-31T20:11:30.120 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2023-05-31T20:11:30.120 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T20:11:30.120 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:11:30.120 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T20:11:30.120 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 1, 2023-05-31T20:11:30.121 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T20:11:30.121 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2023-05-31T20:11:30.121 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "", 2023-05-31T20:11:30.121 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T20:11:30.121 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2023-05-31T20:11:30.121 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T20:11:30.121 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "0.000000", 2023-05-31T20:11:30.122 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2023-05-31T20:11:30.122 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T20:11:30.122 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:11:30.122 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T20:11:30.122 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 2, 2023-05-31T20:11:30.122 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T20:11:30.122 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2023-05-31T20:11:30.123 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1685562970.274572_93.1", 2023-05-31T20:11:30.123 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T20:11:30.123 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2023-05-31T20:11:30.123 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T20:11:30.123 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2023-05-31T19:56:10.274572Z", 2023-05-31T20:11:30.123 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2023-05-31T20:11:30.123 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T20:11:30.124 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:11:30.124 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T20:11:30.124 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 3, 2023-05-31T20:11:30.124 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T20:11:30.124 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2023-05-31T20:11:30.124 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1685563571.700188_212.1", 2023-05-31T20:11:30.124 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T20:11:30.125 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2023-05-31T20:11:30.125 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T20:11:30.125 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2023-05-31T20:06:11.700188Z", 2023-05-31T20:11:30.125 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2023-05-31T20:11:30.125 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T20:11:30.125 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T20:11:30.125 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2023-05-31T20:11:30.126 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:11:30.126 INFO:teuthology.orchestra.run.smithi099.stdout: "full_sync": { 2023-05-31T20:11:30.126 INFO:teuthology.orchestra.run.smithi099.stdout: "total": 1, 2023-05-31T20:11:30.126 INFO:teuthology.orchestra.run.smithi099.stdout: "complete": 1 2023-05-31T20:11:30.126 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:11:30.126 INFO:teuthology.orchestra.run.smithi099.stdout: "current_time": "2023-05-31T20:11:30Z" 2023-05-31T20:11:30.126 INFO:teuthology.orchestra.run.smithi099.stdout:} 2023-05-31T20:11:30.127 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "2e6123d2-d695-4350-9a01-61a455bc782b", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1685562631.516089_59.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-05-31T19:50:31.516089Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1685562970.274572_93.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-05-31T19:56:10.274572Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1685563571.700188_212.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-05-31T20:06:11.700188Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-05-31T20:11:30Z" } 2023-05-31T20:11:30.128 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1685562631.516089_59.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-05-31T19:50:31.516089Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1685562970.274572_93.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-05-31T19:56:10.274572Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1685563571.700188_212.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-05-31T20:06:11.700188Z', 'realm_epoch': 2}}] 2023-05-31T20:11:30.128 DEBUG:rgw_multi.tests:log_status={0: '1_1685562631.516089_59.1', 1: '', 2: '1_1685562970.274572_93.1', 3: '1_1685563802.468126_296.1'} 2023-05-31T20:11:30.129 DEBUG:rgw_multi.tests:sync_status={0: '1_1685562631.516089_59.1', 1: '', 2: '1_1685562970.274572_93.1', 3: '1_1685563571.700188_212.1'} 2023-05-31T20:11:30.129 WARNING:rgw_multi.tests:zone a2 behind master: shard=3 master=1_1685563802.468126_296.1 target=1_1685563571.700188_212.1 2023-05-31T20:11:35.130 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-31T20:11:35.132 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-31T20:11:35.133 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-05-31T20:11:35.192 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2023-05-31T20:11:35.192 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2023-05-31T20:11:35.245 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:11:35.240+0000 7f58836f9780 20 RGW-SYNC:meta: read sync status 2023-05-31T20:11:35.246 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:11:35.241+0000 7f58836f9780 20 RGW-SYNC:meta: read sync status 2023-05-31T20:11:35.261 INFO:teuthology.orchestra.run.smithi099.stdout:{ 2023-05-31T20:11:35.261 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_status": { 2023-05-31T20:11:35.262 INFO:teuthology.orchestra.run.smithi099.stdout: "info": { 2023-05-31T20:11:35.262 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "sync", 2023-05-31T20:11:35.262 INFO:teuthology.orchestra.run.smithi099.stdout: "num_shards": 4, 2023-05-31T20:11:35.262 INFO:teuthology.orchestra.run.smithi099.stdout: "period": "2e6123d2-d695-4350-9a01-61a455bc782b", 2023-05-31T20:11:35.262 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2023-05-31T20:11:35.262 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:11:35.263 INFO:teuthology.orchestra.run.smithi099.stdout: "markers": [ 2023-05-31T20:11:35.263 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T20:11:35.263 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 0, 2023-05-31T20:11:35.263 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T20:11:35.263 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2023-05-31T20:11:35.263 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1685562631.516089_59.1", 2023-05-31T20:11:35.264 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T20:11:35.264 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 1, 2023-05-31T20:11:35.264 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T20:11:35.265 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2023-05-31T19:50:31.516089Z", 2023-05-31T20:11:35.265 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2023-05-31T20:11:35.265 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T20:11:35.265 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:11:35.266 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T20:11:35.266 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 1, 2023-05-31T20:11:35.266 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T20:11:35.266 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2023-05-31T20:11:35.266 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "", 2023-05-31T20:11:35.266 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T20:11:35.266 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2023-05-31T20:11:35.267 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T20:11:35.268 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "0.000000", 2023-05-31T20:11:35.268 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2023-05-31T20:11:35.268 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T20:11:35.268 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:11:35.268 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T20:11:35.268 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 2, 2023-05-31T20:11:35.269 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T20:11:35.269 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2023-05-31T20:11:35.269 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1685562970.274572_93.1", 2023-05-31T20:11:35.269 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T20:11:35.270 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2023-05-31T20:11:35.270 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T20:11:35.270 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2023-05-31T19:56:10.274572Z", 2023-05-31T20:11:35.270 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2023-05-31T20:11:35.270 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T20:11:35.270 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:11:35.270 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T20:11:35.271 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 3, 2023-05-31T20:11:35.271 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T20:11:35.271 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2023-05-31T20:11:35.271 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1685563571.700188_212.1", 2023-05-31T20:11:35.271 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T20:11:35.272 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2023-05-31T20:11:35.272 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T20:11:35.272 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2023-05-31T20:06:11.700188Z", 2023-05-31T20:11:35.272 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2023-05-31T20:11:35.273 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T20:11:35.273 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T20:11:35.273 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2023-05-31T20:11:35.273 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:11:35.274 INFO:teuthology.orchestra.run.smithi099.stdout: "full_sync": { 2023-05-31T20:11:35.274 INFO:teuthology.orchestra.run.smithi099.stdout: "total": 1, 2023-05-31T20:11:35.274 INFO:teuthology.orchestra.run.smithi099.stdout: "complete": 1 2023-05-31T20:11:35.274 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:11:35.274 INFO:teuthology.orchestra.run.smithi099.stdout: "current_time": "2023-05-31T20:11:35Z" 2023-05-31T20:11:35.275 INFO:teuthology.orchestra.run.smithi099.stdout:} 2023-05-31T20:11:35.276 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "2e6123d2-d695-4350-9a01-61a455bc782b", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1685562631.516089_59.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-05-31T19:50:31.516089Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1685562970.274572_93.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-05-31T19:56:10.274572Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1685563571.700188_212.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-05-31T20:06:11.700188Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-05-31T20:11:35Z" } 2023-05-31T20:11:35.277 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1685562631.516089_59.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-05-31T19:50:31.516089Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1685562970.274572_93.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-05-31T19:56:10.274572Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1685563571.700188_212.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-05-31T20:06:11.700188Z', 'realm_epoch': 2}}] 2023-05-31T20:11:35.277 DEBUG:rgw_multi.tests:log_status={0: '1_1685562631.516089_59.1', 1: '', 2: '1_1685562970.274572_93.1', 3: '1_1685563802.468126_296.1'} 2023-05-31T20:11:35.278 DEBUG:rgw_multi.tests:sync_status={0: '1_1685562631.516089_59.1', 1: '', 2: '1_1685562970.274572_93.1', 3: '1_1685563571.700188_212.1'} 2023-05-31T20:11:35.278 WARNING:rgw_multi.tests:zone a2 behind master: shard=3 master=1_1685563802.468126_296.1 target=1_1685563571.700188_212.1 2023-05-31T20:11:40.279 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-31T20:11:40.280 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-31T20:11:40.280 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-05-31T20:11:40.337 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2023-05-31T20:11:40.337 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2023-05-31T20:11:40.371 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:11:40.366+0000 7f1cd83d2780 20 RGW-SYNC:meta: read sync status 2023-05-31T20:11:40.372 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:11:40.367+0000 7f1cd83d2780 20 RGW-SYNC:meta: read sync status 2023-05-31T20:11:40.385 INFO:teuthology.orchestra.run.smithi099.stdout:{ 2023-05-31T20:11:40.386 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_status": { 2023-05-31T20:11:40.386 INFO:teuthology.orchestra.run.smithi099.stdout: "info": { 2023-05-31T20:11:40.386 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "sync", 2023-05-31T20:11:40.386 INFO:teuthology.orchestra.run.smithi099.stdout: "num_shards": 4, 2023-05-31T20:11:40.386 INFO:teuthology.orchestra.run.smithi099.stdout: "period": "2e6123d2-d695-4350-9a01-61a455bc782b", 2023-05-31T20:11:40.386 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2023-05-31T20:11:40.387 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:11:40.387 INFO:teuthology.orchestra.run.smithi099.stdout: "markers": [ 2023-05-31T20:11:40.387 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T20:11:40.387 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 0, 2023-05-31T20:11:40.387 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T20:11:40.387 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2023-05-31T20:11:40.388 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1685562631.516089_59.1", 2023-05-31T20:11:40.388 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T20:11:40.388 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 1, 2023-05-31T20:11:40.388 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T20:11:40.388 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2023-05-31T19:50:31.516089Z", 2023-05-31T20:11:40.388 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2023-05-31T20:11:40.389 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T20:11:40.389 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:11:40.389 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T20:11:40.389 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 1, 2023-05-31T20:11:40.389 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T20:11:40.389 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2023-05-31T20:11:40.389 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "", 2023-05-31T20:11:40.390 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T20:11:40.390 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2023-05-31T20:11:40.390 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T20:11:40.390 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "0.000000", 2023-05-31T20:11:40.390 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2023-05-31T20:11:40.390 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T20:11:40.390 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:11:40.391 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T20:11:40.391 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 2, 2023-05-31T20:11:40.391 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T20:11:40.391 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2023-05-31T20:11:40.391 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1685562970.274572_93.1", 2023-05-31T20:11:40.391 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T20:11:40.391 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2023-05-31T20:11:40.392 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T20:11:40.392 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2023-05-31T19:56:10.274572Z", 2023-05-31T20:11:40.392 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2023-05-31T20:11:40.392 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T20:11:40.392 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:11:40.392 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T20:11:40.392 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 3, 2023-05-31T20:11:40.393 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T20:11:40.393 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2023-05-31T20:11:40.393 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1685563571.700188_212.1", 2023-05-31T20:11:40.393 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T20:11:40.393 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2023-05-31T20:11:40.393 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T20:11:40.393 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2023-05-31T20:06:11.700188Z", 2023-05-31T20:11:40.394 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2023-05-31T20:11:40.394 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T20:11:40.394 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T20:11:40.394 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2023-05-31T20:11:40.394 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:11:40.394 INFO:teuthology.orchestra.run.smithi099.stdout: "full_sync": { 2023-05-31T20:11:40.395 INFO:teuthology.orchestra.run.smithi099.stdout: "total": 1, 2023-05-31T20:11:40.395 INFO:teuthology.orchestra.run.smithi099.stdout: "complete": 1 2023-05-31T20:11:40.395 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:11:40.395 INFO:teuthology.orchestra.run.smithi099.stdout: "current_time": "2023-05-31T20:11:40Z" 2023-05-31T20:11:40.395 INFO:teuthology.orchestra.run.smithi099.stdout:} 2023-05-31T20:11:40.396 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "2e6123d2-d695-4350-9a01-61a455bc782b", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1685562631.516089_59.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-05-31T19:50:31.516089Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1685562970.274572_93.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-05-31T19:56:10.274572Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1685563571.700188_212.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-05-31T20:06:11.700188Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-05-31T20:11:40Z" } 2023-05-31T20:11:40.397 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1685562631.516089_59.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-05-31T19:50:31.516089Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1685562970.274572_93.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-05-31T19:56:10.274572Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1685563571.700188_212.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-05-31T20:06:11.700188Z', 'realm_epoch': 2}}] 2023-05-31T20:11:40.397 DEBUG:rgw_multi.tests:log_status={0: '1_1685562631.516089_59.1', 1: '', 2: '1_1685562970.274572_93.1', 3: '1_1685563802.468126_296.1'} 2023-05-31T20:11:40.397 DEBUG:rgw_multi.tests:sync_status={0: '1_1685562631.516089_59.1', 1: '', 2: '1_1685562970.274572_93.1', 3: '1_1685563571.700188_212.1'} 2023-05-31T20:11:40.398 WARNING:rgw_multi.tests:zone a2 behind master: shard=3 master=1_1685563802.468126_296.1 target=1_1685563571.700188_212.1 2023-05-31T20:11:45.399 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-31T20:11:45.400 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-31T20:11:45.400 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-05-31T20:11:45.458 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2023-05-31T20:11:45.459 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2023-05-31T20:11:45.509 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:11:45.502+0000 7f923f3a9780 20 RGW-SYNC:meta: read sync status 2023-05-31T20:11:45.510 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:11:45.503+0000 7f923f3a9780 20 RGW-SYNC:meta: read sync status 2023-05-31T20:11:45.526 INFO:teuthology.orchestra.run.smithi099.stdout:{ 2023-05-31T20:11:45.527 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_status": { 2023-05-31T20:11:45.527 INFO:teuthology.orchestra.run.smithi099.stdout: "info": { 2023-05-31T20:11:45.528 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "sync", 2023-05-31T20:11:45.528 INFO:teuthology.orchestra.run.smithi099.stdout: "num_shards": 4, 2023-05-31T20:11:45.528 INFO:teuthology.orchestra.run.smithi099.stdout: "period": "2e6123d2-d695-4350-9a01-61a455bc782b", 2023-05-31T20:11:45.529 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2023-05-31T20:11:45.529 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:11:45.529 INFO:teuthology.orchestra.run.smithi099.stdout: "markers": [ 2023-05-31T20:11:45.530 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T20:11:45.531 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 0, 2023-05-31T20:11:45.531 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T20:11:45.532 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2023-05-31T20:11:45.532 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1685562631.516089_59.1", 2023-05-31T20:11:45.532 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T20:11:45.532 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 1, 2023-05-31T20:11:45.532 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T20:11:45.532 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2023-05-31T19:50:31.516089Z", 2023-05-31T20:11:45.533 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2023-05-31T20:11:45.533 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T20:11:45.533 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:11:45.533 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T20:11:45.533 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 1, 2023-05-31T20:11:45.533 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T20:11:45.534 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2023-05-31T20:11:45.534 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "", 2023-05-31T20:11:45.534 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T20:11:45.534 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2023-05-31T20:11:45.534 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T20:11:45.534 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "0.000000", 2023-05-31T20:11:45.535 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2023-05-31T20:11:45.535 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T20:11:45.535 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:11:45.535 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T20:11:45.535 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 2, 2023-05-31T20:11:45.535 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T20:11:45.536 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2023-05-31T20:11:45.536 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1685562970.274572_93.1", 2023-05-31T20:11:45.536 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T20:11:45.536 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2023-05-31T20:11:45.536 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T20:11:45.536 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2023-05-31T19:56:10.274572Z", 2023-05-31T20:11:45.537 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2023-05-31T20:11:45.537 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T20:11:45.537 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:11:45.537 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T20:11:45.537 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 3, 2023-05-31T20:11:45.537 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T20:11:45.538 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2023-05-31T20:11:45.538 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1685563571.700188_212.1", 2023-05-31T20:11:45.538 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T20:11:45.538 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2023-05-31T20:11:45.538 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T20:11:45.538 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2023-05-31T20:06:11.700188Z", 2023-05-31T20:11:45.538 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2023-05-31T20:11:45.539 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T20:11:45.539 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T20:11:45.539 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2023-05-31T20:11:45.539 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:11:45.539 INFO:teuthology.orchestra.run.smithi099.stdout: "full_sync": { 2023-05-31T20:11:45.539 INFO:teuthology.orchestra.run.smithi099.stdout: "total": 1, 2023-05-31T20:11:45.540 INFO:teuthology.orchestra.run.smithi099.stdout: "complete": 1 2023-05-31T20:11:45.540 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:11:45.540 INFO:teuthology.orchestra.run.smithi099.stdout: "current_time": "2023-05-31T20:11:45Z" 2023-05-31T20:11:45.540 INFO:teuthology.orchestra.run.smithi099.stdout:} 2023-05-31T20:11:45.542 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "2e6123d2-d695-4350-9a01-61a455bc782b", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1685562631.516089_59.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-05-31T19:50:31.516089Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1685562970.274572_93.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-05-31T19:56:10.274572Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1685563571.700188_212.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-05-31T20:06:11.700188Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-05-31T20:11:45Z" } 2023-05-31T20:11:45.543 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1685562631.516089_59.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-05-31T19:50:31.516089Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1685562970.274572_93.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-05-31T19:56:10.274572Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1685563571.700188_212.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-05-31T20:06:11.700188Z', 'realm_epoch': 2}}] 2023-05-31T20:11:45.544 DEBUG:rgw_multi.tests:log_status={0: '1_1685562631.516089_59.1', 1: '', 2: '1_1685562970.274572_93.1', 3: '1_1685563802.468126_296.1'} 2023-05-31T20:11:45.545 DEBUG:rgw_multi.tests:sync_status={0: '1_1685562631.516089_59.1', 1: '', 2: '1_1685562970.274572_93.1', 3: '1_1685563571.700188_212.1'} 2023-05-31T20:11:45.545 WARNING:rgw_multi.tests:zone a2 behind master: shard=3 master=1_1685563802.468126_296.1 target=1_1685563571.700188_212.1 2023-05-31T20:11:50.547 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-31T20:11:50.547 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-31T20:11:50.548 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-05-31T20:11:50.605 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2023-05-31T20:11:50.605 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2023-05-31T20:11:50.637 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:11:50.632+0000 7fdc7493c780 20 RGW-SYNC:meta: read sync status 2023-05-31T20:11:50.638 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:11:50.633+0000 7fdc7493c780 20 RGW-SYNC:meta: read sync status 2023-05-31T20:11:50.651 INFO:teuthology.orchestra.run.smithi099.stdout:{ 2023-05-31T20:11:50.651 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_status": { 2023-05-31T20:11:50.651 INFO:teuthology.orchestra.run.smithi099.stdout: "info": { 2023-05-31T20:11:50.652 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "sync", 2023-05-31T20:11:50.652 INFO:teuthology.orchestra.run.smithi099.stdout: "num_shards": 4, 2023-05-31T20:11:50.652 INFO:teuthology.orchestra.run.smithi099.stdout: "period": "2e6123d2-d695-4350-9a01-61a455bc782b", 2023-05-31T20:11:50.652 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2023-05-31T20:11:50.653 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:11:50.653 INFO:teuthology.orchestra.run.smithi099.stdout: "markers": [ 2023-05-31T20:11:50.653 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T20:11:50.653 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 0, 2023-05-31T20:11:50.653 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T20:11:50.653 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2023-05-31T20:11:50.653 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1685562631.516089_59.1", 2023-05-31T20:11:50.654 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T20:11:50.654 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 1, 2023-05-31T20:11:50.654 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T20:11:50.654 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2023-05-31T19:50:31.516089Z", 2023-05-31T20:11:50.654 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2023-05-31T20:11:50.654 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T20:11:50.654 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:11:50.655 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T20:11:50.655 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 1, 2023-05-31T20:11:50.655 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T20:11:50.655 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2023-05-31T20:11:50.655 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "", 2023-05-31T20:11:50.656 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T20:11:50.656 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2023-05-31T20:11:50.656 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T20:11:50.656 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "0.000000", 2023-05-31T20:11:50.656 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2023-05-31T20:11:50.656 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T20:11:50.657 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:11:50.657 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T20:11:50.657 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 2, 2023-05-31T20:11:50.657 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T20:11:50.657 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2023-05-31T20:11:50.658 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1685562970.274572_93.1", 2023-05-31T20:11:50.658 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T20:11:50.658 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2023-05-31T20:11:50.658 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T20:11:50.658 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2023-05-31T19:56:10.274572Z", 2023-05-31T20:11:50.658 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2023-05-31T20:11:50.659 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T20:11:50.659 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:11:50.659 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T20:11:50.659 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 3, 2023-05-31T20:11:50.659 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T20:11:50.659 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2023-05-31T20:11:50.660 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1685563571.700188_212.1", 2023-05-31T20:11:50.660 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T20:11:50.660 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2023-05-31T20:11:50.660 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T20:11:50.661 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2023-05-31T20:06:11.700188Z", 2023-05-31T20:11:50.661 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2023-05-31T20:11:50.661 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T20:11:50.661 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T20:11:50.661 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2023-05-31T20:11:50.661 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:11:50.662 INFO:teuthology.orchestra.run.smithi099.stdout: "full_sync": { 2023-05-31T20:11:50.662 INFO:teuthology.orchestra.run.smithi099.stdout: "total": 1, 2023-05-31T20:11:50.662 INFO:teuthology.orchestra.run.smithi099.stdout: "complete": 1 2023-05-31T20:11:50.662 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:11:50.662 INFO:teuthology.orchestra.run.smithi099.stdout: "current_time": "2023-05-31T20:11:50Z" 2023-05-31T20:11:50.663 INFO:teuthology.orchestra.run.smithi099.stdout:} 2023-05-31T20:11:50.663 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "2e6123d2-d695-4350-9a01-61a455bc782b", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1685562631.516089_59.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-05-31T19:50:31.516089Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1685562970.274572_93.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-05-31T19:56:10.274572Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1685563571.700188_212.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-05-31T20:06:11.700188Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-05-31T20:11:50Z" } 2023-05-31T20:11:50.664 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1685562631.516089_59.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-05-31T19:50:31.516089Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1685562970.274572_93.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-05-31T19:56:10.274572Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1685563571.700188_212.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-05-31T20:06:11.700188Z', 'realm_epoch': 2}}] 2023-05-31T20:11:50.665 DEBUG:rgw_multi.tests:log_status={0: '1_1685562631.516089_59.1', 1: '', 2: '1_1685562970.274572_93.1', 3: '1_1685563802.468126_296.1'} 2023-05-31T20:11:50.665 DEBUG:rgw_multi.tests:sync_status={0: '1_1685562631.516089_59.1', 1: '', 2: '1_1685562970.274572_93.1', 3: '1_1685563571.700188_212.1'} 2023-05-31T20:11:50.666 WARNING:rgw_multi.tests:zone a2 behind master: shard=3 master=1_1685563802.468126_296.1 target=1_1685563571.700188_212.1 2023-05-31T20:11:55.667 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-31T20:11:55.668 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-31T20:11:55.668 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-05-31T20:11:55.724 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2023-05-31T20:11:55.725 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2023-05-31T20:11:55.772 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:11:55.767+0000 7f20eeed3780 20 RGW-SYNC:meta: read sync status 2023-05-31T20:11:55.773 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:11:55.768+0000 7f20eeed3780 20 RGW-SYNC:meta: read sync status 2023-05-31T20:11:55.797 INFO:teuthology.orchestra.run.smithi099.stdout:{ 2023-05-31T20:11:55.797 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_status": { 2023-05-31T20:11:55.798 INFO:teuthology.orchestra.run.smithi099.stdout: "info": { 2023-05-31T20:11:55.798 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "sync", 2023-05-31T20:11:55.798 INFO:teuthology.orchestra.run.smithi099.stdout: "num_shards": 4, 2023-05-31T20:11:55.798 INFO:teuthology.orchestra.run.smithi099.stdout: "period": "2e6123d2-d695-4350-9a01-61a455bc782b", 2023-05-31T20:11:55.799 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2023-05-31T20:11:55.799 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:11:55.799 INFO:teuthology.orchestra.run.smithi099.stdout: "markers": [ 2023-05-31T20:11:55.799 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T20:11:55.800 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 0, 2023-05-31T20:11:55.800 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T20:11:55.801 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2023-05-31T20:11:55.801 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1685562631.516089_59.1", 2023-05-31T20:11:55.802 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T20:11:55.802 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 1, 2023-05-31T20:11:55.802 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T20:11:55.803 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2023-05-31T19:50:31.516089Z", 2023-05-31T20:11:55.803 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2023-05-31T20:11:55.803 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T20:11:55.803 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:11:55.804 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T20:11:55.804 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 1, 2023-05-31T20:11:55.805 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T20:11:55.805 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2023-05-31T20:11:55.805 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "", 2023-05-31T20:11:55.806 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T20:11:55.806 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2023-05-31T20:11:55.806 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T20:11:55.806 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "0.000000", 2023-05-31T20:11:55.806 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2023-05-31T20:11:55.806 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T20:11:55.806 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:11:55.807 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T20:11:55.807 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 2, 2023-05-31T20:11:55.807 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T20:11:55.807 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2023-05-31T20:11:55.807 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1685562970.274572_93.1", 2023-05-31T20:11:55.807 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T20:11:55.808 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2023-05-31T20:11:55.808 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T20:11:55.808 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2023-05-31T19:56:10.274572Z", 2023-05-31T20:11:55.808 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2023-05-31T20:11:55.808 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T20:11:55.808 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:11:55.809 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T20:11:55.809 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 3, 2023-05-31T20:11:55.809 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T20:11:55.809 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2023-05-31T20:11:55.809 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1685563571.700188_212.1", 2023-05-31T20:11:55.809 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T20:11:55.809 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2023-05-31T20:11:55.810 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T20:11:55.810 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2023-05-31T20:06:11.700188Z", 2023-05-31T20:11:55.810 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2023-05-31T20:11:55.810 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T20:11:55.811 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T20:11:55.811 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2023-05-31T20:11:55.811 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:11:55.811 INFO:teuthology.orchestra.run.smithi099.stdout: "full_sync": { 2023-05-31T20:11:55.811 INFO:teuthology.orchestra.run.smithi099.stdout: "total": 1, 2023-05-31T20:11:55.812 INFO:teuthology.orchestra.run.smithi099.stdout: "complete": 1 2023-05-31T20:11:55.812 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:11:55.812 INFO:teuthology.orchestra.run.smithi099.stdout: "current_time": "2023-05-31T20:11:55Z" 2023-05-31T20:11:55.812 INFO:teuthology.orchestra.run.smithi099.stdout:} 2023-05-31T20:11:55.813 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "2e6123d2-d695-4350-9a01-61a455bc782b", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1685562631.516089_59.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-05-31T19:50:31.516089Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1685562970.274572_93.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-05-31T19:56:10.274572Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1685563571.700188_212.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-05-31T20:06:11.700188Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-05-31T20:11:55Z" } 2023-05-31T20:11:55.814 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1685562631.516089_59.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-05-31T19:50:31.516089Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1685562970.274572_93.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-05-31T19:56:10.274572Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1685563571.700188_212.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-05-31T20:06:11.700188Z', 'realm_epoch': 2}}] 2023-05-31T20:11:55.815 DEBUG:rgw_multi.tests:log_status={0: '1_1685562631.516089_59.1', 1: '', 2: '1_1685562970.274572_93.1', 3: '1_1685563802.468126_296.1'} 2023-05-31T20:11:55.815 DEBUG:rgw_multi.tests:sync_status={0: '1_1685562631.516089_59.1', 1: '', 2: '1_1685562970.274572_93.1', 3: '1_1685563571.700188_212.1'} 2023-05-31T20:11:55.816 WARNING:rgw_multi.tests:zone a2 behind master: shard=3 master=1_1685563802.468126_296.1 target=1_1685563571.700188_212.1 2023-05-31T20:12:00.817 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-31T20:12:00.817 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-31T20:12:00.818 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-05-31T20:12:00.875 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2023-05-31T20:12:00.876 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2023-05-31T20:12:00.912 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:12:00.905+0000 7feb68131780 20 RGW-SYNC:meta: read sync status 2023-05-31T20:12:00.912 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:12:00.906+0000 7feb68131780 20 RGW-SYNC:meta: read sync status 2023-05-31T20:12:00.924 INFO:teuthology.orchestra.run.smithi099.stdout:{ 2023-05-31T20:12:00.924 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_status": { 2023-05-31T20:12:00.925 INFO:teuthology.orchestra.run.smithi099.stdout: "info": { 2023-05-31T20:12:00.925 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "sync", 2023-05-31T20:12:00.925 INFO:teuthology.orchestra.run.smithi099.stdout: "num_shards": 4, 2023-05-31T20:12:00.925 INFO:teuthology.orchestra.run.smithi099.stdout: "period": "2e6123d2-d695-4350-9a01-61a455bc782b", 2023-05-31T20:12:00.925 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2023-05-31T20:12:00.925 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:12:00.926 INFO:teuthology.orchestra.run.smithi099.stdout: "markers": [ 2023-05-31T20:12:00.926 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T20:12:00.926 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 0, 2023-05-31T20:12:00.926 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T20:12:00.926 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2023-05-31T20:12:00.926 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1685562631.516089_59.1", 2023-05-31T20:12:00.926 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T20:12:00.927 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 1, 2023-05-31T20:12:00.927 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T20:12:00.927 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2023-05-31T19:50:31.516089Z", 2023-05-31T20:12:00.928 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2023-05-31T20:12:00.928 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T20:12:00.928 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:12:00.928 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T20:12:00.928 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 1, 2023-05-31T20:12:00.929 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T20:12:00.929 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2023-05-31T20:12:00.930 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "", 2023-05-31T20:12:00.930 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T20:12:00.931 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2023-05-31T20:12:00.931 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T20:12:00.931 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "0.000000", 2023-05-31T20:12:00.931 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2023-05-31T20:12:00.931 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T20:12:00.932 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:12:00.932 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T20:12:00.932 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 2, 2023-05-31T20:12:00.932 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T20:12:00.933 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2023-05-31T20:12:00.933 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1685562970.274572_93.1", 2023-05-31T20:12:00.933 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T20:12:00.934 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2023-05-31T20:12:00.934 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T20:12:00.934 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2023-05-31T19:56:10.274572Z", 2023-05-31T20:12:00.934 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2023-05-31T20:12:00.935 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T20:12:00.935 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:12:00.935 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T20:12:00.936 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 3, 2023-05-31T20:12:00.936 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T20:12:00.936 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2023-05-31T20:12:00.937 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1685563798.744769_245.1", 2023-05-31T20:12:00.937 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T20:12:00.937 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2023-05-31T20:12:00.938 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T20:12:00.939 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2023-05-31T20:09:58.744769Z", 2023-05-31T20:12:00.939 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2023-05-31T20:12:00.939 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T20:12:00.940 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T20:12:00.940 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2023-05-31T20:12:00.940 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:12:00.941 INFO:teuthology.orchestra.run.smithi099.stdout: "full_sync": { 2023-05-31T20:12:00.941 INFO:teuthology.orchestra.run.smithi099.stdout: "total": 1, 2023-05-31T20:12:00.942 INFO:teuthology.orchestra.run.smithi099.stdout: "complete": 1 2023-05-31T20:12:00.942 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:12:00.942 INFO:teuthology.orchestra.run.smithi099.stdout: "current_time": "2023-05-31T20:12:00Z" 2023-05-31T20:12:00.943 INFO:teuthology.orchestra.run.smithi099.stdout:} 2023-05-31T20:12:00.944 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "2e6123d2-d695-4350-9a01-61a455bc782b", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1685562631.516089_59.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-05-31T19:50:31.516089Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1685562970.274572_93.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-05-31T19:56:10.274572Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1685563798.744769_245.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-05-31T20:09:58.744769Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-05-31T20:12:00Z" } 2023-05-31T20:12:00.945 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1685562631.516089_59.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-05-31T19:50:31.516089Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1685562970.274572_93.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-05-31T19:56:10.274572Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1685563798.744769_245.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-05-31T20:09:58.744769Z', 'realm_epoch': 2}}] 2023-05-31T20:12:00.945 DEBUG:rgw_multi.tests:log_status={0: '1_1685562631.516089_59.1', 1: '', 2: '1_1685562970.274572_93.1', 3: '1_1685563802.468126_296.1'} 2023-05-31T20:12:00.945 DEBUG:rgw_multi.tests:sync_status={0: '1_1685562631.516089_59.1', 1: '', 2: '1_1685562970.274572_93.1', 3: '1_1685563798.744769_245.1'} 2023-05-31T20:12:00.946 WARNING:rgw_multi.tests:zone a2 behind master: shard=3 master=1_1685563802.468126_296.1 target=1_1685563798.744769_245.1 2023-05-31T20:12:05.946 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-31T20:12:05.947 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-31T20:12:05.948 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-05-31T20:12:06.000 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2023-05-31T20:12:06.000 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2023-05-31T20:12:06.038 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:12:06.033+0000 7f7aa0265780 20 RGW-SYNC:meta: read sync status 2023-05-31T20:12:06.040 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:12:06.034+0000 7f7aa0265780 20 RGW-SYNC:meta: read sync status 2023-05-31T20:12:06.056 INFO:teuthology.orchestra.run.smithi099.stdout:{ 2023-05-31T20:12:06.056 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_status": { 2023-05-31T20:12:06.057 INFO:teuthology.orchestra.run.smithi099.stdout: "info": { 2023-05-31T20:12:06.057 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "sync", 2023-05-31T20:12:06.057 INFO:teuthology.orchestra.run.smithi099.stdout: "num_shards": 4, 2023-05-31T20:12:06.057 INFO:teuthology.orchestra.run.smithi099.stdout: "period": "2e6123d2-d695-4350-9a01-61a455bc782b", 2023-05-31T20:12:06.057 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2023-05-31T20:12:06.057 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:12:06.058 INFO:teuthology.orchestra.run.smithi099.stdout: "markers": [ 2023-05-31T20:12:06.058 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T20:12:06.058 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 0, 2023-05-31T20:12:06.058 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T20:12:06.058 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2023-05-31T20:12:06.058 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1685562631.516089_59.1", 2023-05-31T20:12:06.059 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T20:12:06.059 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 1, 2023-05-31T20:12:06.059 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T20:12:06.059 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2023-05-31T19:50:31.516089Z", 2023-05-31T20:12:06.059 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2023-05-31T20:12:06.059 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T20:12:06.059 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:12:06.060 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T20:12:06.060 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 1, 2023-05-31T20:12:06.060 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T20:12:06.060 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2023-05-31T20:12:06.060 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "", 2023-05-31T20:12:06.060 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T20:12:06.060 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2023-05-31T20:12:06.061 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T20:12:06.061 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "0.000000", 2023-05-31T20:12:06.061 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2023-05-31T20:12:06.061 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T20:12:06.061 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:12:06.062 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T20:12:06.062 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 2, 2023-05-31T20:12:06.063 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T20:12:06.063 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2023-05-31T20:12:06.063 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1685562970.274572_93.1", 2023-05-31T20:12:06.063 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T20:12:06.064 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2023-05-31T20:12:06.064 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T20:12:06.064 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2023-05-31T19:56:10.274572Z", 2023-05-31T20:12:06.064 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2023-05-31T20:12:06.064 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T20:12:06.065 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:12:06.065 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T20:12:06.065 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 3, 2023-05-31T20:12:06.065 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T20:12:06.066 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2023-05-31T20:12:06.066 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1685563798.970303_257.1", 2023-05-31T20:12:06.066 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T20:12:06.067 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2023-05-31T20:12:06.067 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T20:12:06.067 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2023-05-31T20:09:58.970303Z", 2023-05-31T20:12:06.067 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2023-05-31T20:12:06.067 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T20:12:06.067 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T20:12:06.067 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2023-05-31T20:12:06.068 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:12:06.068 INFO:teuthology.orchestra.run.smithi099.stdout: "full_sync": { 2023-05-31T20:12:06.068 INFO:teuthology.orchestra.run.smithi099.stdout: "total": 1, 2023-05-31T20:12:06.068 INFO:teuthology.orchestra.run.smithi099.stdout: "complete": 1 2023-05-31T20:12:06.068 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:12:06.069 INFO:teuthology.orchestra.run.smithi099.stdout: "current_time": "2023-05-31T20:12:06Z" 2023-05-31T20:12:06.069 INFO:teuthology.orchestra.run.smithi099.stdout:} 2023-05-31T20:12:06.070 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "2e6123d2-d695-4350-9a01-61a455bc782b", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1685562631.516089_59.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-05-31T19:50:31.516089Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1685562970.274572_93.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-05-31T19:56:10.274572Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1685563798.970303_257.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-05-31T20:09:58.970303Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-05-31T20:12:06Z" } 2023-05-31T20:12:06.070 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1685562631.516089_59.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-05-31T19:50:31.516089Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1685562970.274572_93.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-05-31T19:56:10.274572Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1685563798.970303_257.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-05-31T20:09:58.970303Z', 'realm_epoch': 2}}] 2023-05-31T20:12:06.071 DEBUG:rgw_multi.tests:log_status={0: '1_1685562631.516089_59.1', 1: '', 2: '1_1685562970.274572_93.1', 3: '1_1685563802.468126_296.1'} 2023-05-31T20:12:06.071 DEBUG:rgw_multi.tests:sync_status={0: '1_1685562631.516089_59.1', 1: '', 2: '1_1685562970.274572_93.1', 3: '1_1685563798.970303_257.1'} 2023-05-31T20:12:06.071 WARNING:rgw_multi.tests:zone a2 behind master: shard=3 master=1_1685563802.468126_296.1 target=1_1685563798.970303_257.1 2023-05-31T20:12:11.072 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-31T20:12:11.073 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-31T20:12:11.073 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-05-31T20:12:11.125 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2023-05-31T20:12:11.125 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2023-05-31T20:12:11.164 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:12:11.158+0000 7fd83d63a780 20 RGW-SYNC:meta: read sync status 2023-05-31T20:12:11.165 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:12:11.159+0000 7fd83d63a780 20 RGW-SYNC:meta: read sync status 2023-05-31T20:12:11.177 INFO:teuthology.orchestra.run.smithi099.stdout:{ 2023-05-31T20:12:11.177 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_status": { 2023-05-31T20:12:11.177 INFO:teuthology.orchestra.run.smithi099.stdout: "info": { 2023-05-31T20:12:11.177 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "sync", 2023-05-31T20:12:11.177 INFO:teuthology.orchestra.run.smithi099.stdout: "num_shards": 4, 2023-05-31T20:12:11.178 INFO:teuthology.orchestra.run.smithi099.stdout: "period": "2e6123d2-d695-4350-9a01-61a455bc782b", 2023-05-31T20:12:11.178 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2023-05-31T20:12:11.178 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:12:11.178 INFO:teuthology.orchestra.run.smithi099.stdout: "markers": [ 2023-05-31T20:12:11.179 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T20:12:11.179 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 0, 2023-05-31T20:12:11.179 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T20:12:11.179 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2023-05-31T20:12:11.179 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1685562631.516089_59.1", 2023-05-31T20:12:11.179 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T20:12:11.180 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 1, 2023-05-31T20:12:11.180 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T20:12:11.180 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2023-05-31T19:50:31.516089Z", 2023-05-31T20:12:11.180 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2023-05-31T20:12:11.180 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T20:12:11.180 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:12:11.180 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T20:12:11.181 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 1, 2023-05-31T20:12:11.181 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T20:12:11.181 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2023-05-31T20:12:11.181 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "", 2023-05-31T20:12:11.181 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T20:12:11.181 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2023-05-31T20:12:11.182 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T20:12:11.182 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "0.000000", 2023-05-31T20:12:11.182 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2023-05-31T20:12:11.182 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T20:12:11.182 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:12:11.182 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T20:12:11.182 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 2, 2023-05-31T20:12:11.183 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T20:12:11.183 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2023-05-31T20:12:11.183 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1685562970.274572_93.1", 2023-05-31T20:12:11.183 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T20:12:11.183 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2023-05-31T20:12:11.183 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T20:12:11.184 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2023-05-31T19:56:10.274572Z", 2023-05-31T20:12:11.184 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2023-05-31T20:12:11.184 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T20:12:11.184 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:12:11.184 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T20:12:11.184 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 3, 2023-05-31T20:12:11.185 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T20:12:11.185 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2023-05-31T20:12:11.185 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1685563798.885419_253.1", 2023-05-31T20:12:11.185 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T20:12:11.185 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2023-05-31T20:12:11.185 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T20:12:11.185 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2023-05-31T20:09:58.885419Z", 2023-05-31T20:12:11.186 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2023-05-31T20:12:11.186 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T20:12:11.186 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T20:12:11.186 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2023-05-31T20:12:11.186 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:12:11.186 INFO:teuthology.orchestra.run.smithi099.stdout: "full_sync": { 2023-05-31T20:12:11.186 INFO:teuthology.orchestra.run.smithi099.stdout: "total": 1, 2023-05-31T20:12:11.187 INFO:teuthology.orchestra.run.smithi099.stdout: "complete": 1 2023-05-31T20:12:11.187 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:12:11.187 INFO:teuthology.orchestra.run.smithi099.stdout: "current_time": "2023-05-31T20:12:11Z" 2023-05-31T20:12:11.187 INFO:teuthology.orchestra.run.smithi099.stdout:} 2023-05-31T20:12:11.188 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "2e6123d2-d695-4350-9a01-61a455bc782b", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1685562631.516089_59.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-05-31T19:50:31.516089Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1685562970.274572_93.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-05-31T19:56:10.274572Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1685563798.885419_253.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-05-31T20:09:58.885419Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-05-31T20:12:11Z" } 2023-05-31T20:12:11.189 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1685562631.516089_59.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-05-31T19:50:31.516089Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1685562970.274572_93.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-05-31T19:56:10.274572Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1685563798.885419_253.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-05-31T20:09:58.885419Z', 'realm_epoch': 2}}] 2023-05-31T20:12:11.189 DEBUG:rgw_multi.tests:log_status={0: '1_1685562631.516089_59.1', 1: '', 2: '1_1685562970.274572_93.1', 3: '1_1685563802.468126_296.1'} 2023-05-31T20:12:11.192 DEBUG:rgw_multi.tests:sync_status={0: '1_1685562631.516089_59.1', 1: '', 2: '1_1685562970.274572_93.1', 3: '1_1685563798.885419_253.1'} 2023-05-31T20:12:11.192 WARNING:rgw_multi.tests:zone a2 behind master: shard=3 master=1_1685563802.468126_296.1 target=1_1685563798.885419_253.1 2023-05-31T20:12:16.193 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-31T20:12:16.194 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-31T20:12:16.196 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-05-31T20:12:16.257 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2023-05-31T20:12:16.258 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2023-05-31T20:12:16.317 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:12:16.312+0000 7f46d13db780 20 RGW-SYNC:meta: read sync status 2023-05-31T20:12:16.318 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:12:16.312+0000 7f46d13db780 20 RGW-SYNC:meta: read sync status 2023-05-31T20:12:16.350 INFO:teuthology.orchestra.run.smithi099.stdout:{ 2023-05-31T20:12:16.351 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_status": { 2023-05-31T20:12:16.351 INFO:teuthology.orchestra.run.smithi099.stdout: "info": { 2023-05-31T20:12:16.351 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "sync", 2023-05-31T20:12:16.351 INFO:teuthology.orchestra.run.smithi099.stdout: "num_shards": 4, 2023-05-31T20:12:16.352 INFO:teuthology.orchestra.run.smithi099.stdout: "period": "2e6123d2-d695-4350-9a01-61a455bc782b", 2023-05-31T20:12:16.352 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2023-05-31T20:12:16.352 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:12:16.352 INFO:teuthology.orchestra.run.smithi099.stdout: "markers": [ 2023-05-31T20:12:16.352 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T20:12:16.352 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 0, 2023-05-31T20:12:16.353 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T20:12:16.353 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2023-05-31T20:12:16.353 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1685562631.516089_59.1", 2023-05-31T20:12:16.353 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T20:12:16.353 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 1, 2023-05-31T20:12:16.353 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T20:12:16.353 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2023-05-31T19:50:31.516089Z", 2023-05-31T20:12:16.354 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2023-05-31T20:12:16.354 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T20:12:16.354 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:12:16.354 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T20:12:16.354 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 1, 2023-05-31T20:12:16.354 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T20:12:16.355 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2023-05-31T20:12:16.355 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "", 2023-05-31T20:12:16.356 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T20:12:16.356 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2023-05-31T20:12:16.356 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T20:12:16.356 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "0.000000", 2023-05-31T20:12:16.357 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2023-05-31T20:12:16.357 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T20:12:16.357 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:12:16.357 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T20:12:16.357 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 2, 2023-05-31T20:12:16.357 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T20:12:16.358 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2023-05-31T20:12:16.358 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1685562970.274572_93.1", 2023-05-31T20:12:16.358 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T20:12:16.358 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2023-05-31T20:12:16.358 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T20:12:16.358 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2023-05-31T19:56:10.274572Z", 2023-05-31T20:12:16.359 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2023-05-31T20:12:16.359 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T20:12:16.359 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:12:16.359 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T20:12:16.359 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 3, 2023-05-31T20:12:16.359 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T20:12:16.360 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2023-05-31T20:12:16.360 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1685563798.885419_253.1", 2023-05-31T20:12:16.360 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T20:12:16.361 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2023-05-31T20:12:16.361 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T20:12:16.361 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2023-05-31T20:09:58.885419Z", 2023-05-31T20:12:16.362 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2023-05-31T20:12:16.362 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T20:12:16.362 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T20:12:16.363 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2023-05-31T20:12:16.363 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:12:16.364 INFO:teuthology.orchestra.run.smithi099.stdout: "full_sync": { 2023-05-31T20:12:16.364 INFO:teuthology.orchestra.run.smithi099.stdout: "total": 1, 2023-05-31T20:12:16.364 INFO:teuthology.orchestra.run.smithi099.stdout: "complete": 1 2023-05-31T20:12:16.366 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:12:16.367 INFO:teuthology.orchestra.run.smithi099.stdout: "current_time": "2023-05-31T20:12:16Z" 2023-05-31T20:12:16.367 INFO:teuthology.orchestra.run.smithi099.stdout:} 2023-05-31T20:12:16.368 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "2e6123d2-d695-4350-9a01-61a455bc782b", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1685562631.516089_59.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-05-31T19:50:31.516089Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1685562970.274572_93.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-05-31T19:56:10.274572Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1685563798.885419_253.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-05-31T20:09:58.885419Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-05-31T20:12:16Z" } 2023-05-31T20:12:16.369 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1685562631.516089_59.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-05-31T19:50:31.516089Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1685562970.274572_93.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-05-31T19:56:10.274572Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1685563798.885419_253.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-05-31T20:09:58.885419Z', 'realm_epoch': 2}}] 2023-05-31T20:12:16.369 DEBUG:rgw_multi.tests:log_status={0: '1_1685562631.516089_59.1', 1: '', 2: '1_1685562970.274572_93.1', 3: '1_1685563802.468126_296.1'} 2023-05-31T20:12:16.370 DEBUG:rgw_multi.tests:sync_status={0: '1_1685562631.516089_59.1', 1: '', 2: '1_1685562970.274572_93.1', 3: '1_1685563798.885419_253.1'} 2023-05-31T20:12:16.370 WARNING:rgw_multi.tests:zone a2 behind master: shard=3 master=1_1685563802.468126_296.1 target=1_1685563798.885419_253.1 2023-05-31T20:12:21.371 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-31T20:12:21.372 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-31T20:12:21.372 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-05-31T20:12:21.424 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2023-05-31T20:12:21.425 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2023-05-31T20:12:21.467 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:12:21.461+0000 7fe19c197780 20 RGW-SYNC:meta: read sync status 2023-05-31T20:12:21.467 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:12:21.462+0000 7fe19c197780 20 RGW-SYNC:meta: read sync status 2023-05-31T20:12:21.484 INFO:teuthology.orchestra.run.smithi099.stdout:{ 2023-05-31T20:12:21.485 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_status": { 2023-05-31T20:12:21.485 INFO:teuthology.orchestra.run.smithi099.stdout: "info": { 2023-05-31T20:12:21.485 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "sync", 2023-05-31T20:12:21.485 INFO:teuthology.orchestra.run.smithi099.stdout: "num_shards": 4, 2023-05-31T20:12:21.485 INFO:teuthology.orchestra.run.smithi099.stdout: "period": "2e6123d2-d695-4350-9a01-61a455bc782b", 2023-05-31T20:12:21.485 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2023-05-31T20:12:21.485 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:12:21.486 INFO:teuthology.orchestra.run.smithi099.stdout: "markers": [ 2023-05-31T20:12:21.486 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T20:12:21.486 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 0, 2023-05-31T20:12:21.486 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T20:12:21.486 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2023-05-31T20:12:21.486 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1685562631.516089_59.1", 2023-05-31T20:12:21.486 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T20:12:21.487 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 1, 2023-05-31T20:12:21.487 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T20:12:21.487 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2023-05-31T19:50:31.516089Z", 2023-05-31T20:12:21.487 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2023-05-31T20:12:21.487 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T20:12:21.487 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:12:21.488 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T20:12:21.488 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 1, 2023-05-31T20:12:21.488 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T20:12:21.488 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2023-05-31T20:12:21.488 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "", 2023-05-31T20:12:21.488 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T20:12:21.488 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2023-05-31T20:12:21.489 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T20:12:21.489 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "0.000000", 2023-05-31T20:12:21.489 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2023-05-31T20:12:21.489 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T20:12:21.489 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:12:21.489 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T20:12:21.489 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 2, 2023-05-31T20:12:21.490 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T20:12:21.490 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2023-05-31T20:12:21.490 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1685562970.274572_93.1", 2023-05-31T20:12:21.490 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T20:12:21.490 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2023-05-31T20:12:21.490 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T20:12:21.490 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2023-05-31T19:56:10.274572Z", 2023-05-31T20:12:21.491 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2023-05-31T20:12:21.491 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T20:12:21.491 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:12:21.491 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T20:12:21.491 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 3, 2023-05-31T20:12:21.491 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T20:12:21.491 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2023-05-31T20:12:21.492 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1685563798.815084_250.1", 2023-05-31T20:12:21.492 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T20:12:21.493 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2023-05-31T20:12:21.493 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T20:12:21.493 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2023-05-31T20:09:58.815084Z", 2023-05-31T20:12:21.493 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2023-05-31T20:12:21.493 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T20:12:21.494 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T20:12:21.494 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2023-05-31T20:12:21.494 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:12:21.494 INFO:teuthology.orchestra.run.smithi099.stdout: "full_sync": { 2023-05-31T20:12:21.494 INFO:teuthology.orchestra.run.smithi099.stdout: "total": 1, 2023-05-31T20:12:21.494 INFO:teuthology.orchestra.run.smithi099.stdout: "complete": 1 2023-05-31T20:12:21.495 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:12:21.495 INFO:teuthology.orchestra.run.smithi099.stdout: "current_time": "2023-05-31T20:12:21Z" 2023-05-31T20:12:21.495 INFO:teuthology.orchestra.run.smithi099.stdout:} 2023-05-31T20:12:21.496 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "2e6123d2-d695-4350-9a01-61a455bc782b", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1685562631.516089_59.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-05-31T19:50:31.516089Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1685562970.274572_93.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-05-31T19:56:10.274572Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1685563798.815084_250.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-05-31T20:09:58.815084Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-05-31T20:12:21Z" } 2023-05-31T20:12:21.497 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1685562631.516089_59.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-05-31T19:50:31.516089Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1685562970.274572_93.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-05-31T19:56:10.274572Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1685563798.815084_250.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-05-31T20:09:58.815084Z', 'realm_epoch': 2}}] 2023-05-31T20:12:21.497 DEBUG:rgw_multi.tests:log_status={0: '1_1685562631.516089_59.1', 1: '', 2: '1_1685562970.274572_93.1', 3: '1_1685563802.468126_296.1'} 2023-05-31T20:12:21.497 DEBUG:rgw_multi.tests:sync_status={0: '1_1685562631.516089_59.1', 1: '', 2: '1_1685562970.274572_93.1', 3: '1_1685563798.815084_250.1'} 2023-05-31T20:12:21.498 WARNING:rgw_multi.tests:zone a2 behind master: shard=3 master=1_1685563802.468126_296.1 target=1_1685563798.815084_250.1 2023-05-31T20:12:26.499 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-31T20:12:26.500 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-31T20:12:26.501 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-05-31T20:12:26.549 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2023-05-31T20:12:26.549 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2023-05-31T20:12:26.599 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:12:26.594+0000 7fbad1a63780 20 RGW-SYNC:meta: read sync status 2023-05-31T20:12:26.600 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:12:26.596+0000 7fbad1a63780 20 RGW-SYNC:meta: read sync status 2023-05-31T20:12:26.615 INFO:teuthology.orchestra.run.smithi099.stdout:{ 2023-05-31T20:12:26.615 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_status": { 2023-05-31T20:12:26.615 INFO:teuthology.orchestra.run.smithi099.stdout: "info": { 2023-05-31T20:12:26.616 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "sync", 2023-05-31T20:12:26.616 INFO:teuthology.orchestra.run.smithi099.stdout: "num_shards": 4, 2023-05-31T20:12:26.616 INFO:teuthology.orchestra.run.smithi099.stdout: "period": "2e6123d2-d695-4350-9a01-61a455bc782b", 2023-05-31T20:12:26.616 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2023-05-31T20:12:26.616 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:12:26.616 INFO:teuthology.orchestra.run.smithi099.stdout: "markers": [ 2023-05-31T20:12:26.617 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T20:12:26.617 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 0, 2023-05-31T20:12:26.617 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T20:12:26.617 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2023-05-31T20:12:26.617 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1685562631.516089_59.1", 2023-05-31T20:12:26.617 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T20:12:26.618 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 1, 2023-05-31T20:12:26.618 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T20:12:26.618 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2023-05-31T19:50:31.516089Z", 2023-05-31T20:12:26.618 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2023-05-31T20:12:26.618 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T20:12:26.618 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:12:26.618 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T20:12:26.619 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 1, 2023-05-31T20:12:26.619 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T20:12:26.619 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2023-05-31T20:12:26.619 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "", 2023-05-31T20:12:26.619 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T20:12:26.619 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2023-05-31T20:12:26.619 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T20:12:26.620 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "0.000000", 2023-05-31T20:12:26.620 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2023-05-31T20:12:26.620 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T20:12:26.620 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:12:26.620 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T20:12:26.620 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 2, 2023-05-31T20:12:26.620 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T20:12:26.621 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2023-05-31T20:12:26.621 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1685562970.274572_93.1", 2023-05-31T20:12:26.621 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T20:12:26.621 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2023-05-31T20:12:26.621 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T20:12:26.621 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2023-05-31T19:56:10.274572Z", 2023-05-31T20:12:26.622 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2023-05-31T20:12:26.622 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T20:12:26.622 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:12:26.622 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T20:12:26.622 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 3, 2023-05-31T20:12:26.622 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T20:12:26.622 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2023-05-31T20:12:26.623 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1685563799.098535_267.1", 2023-05-31T20:12:26.623 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T20:12:26.623 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2023-05-31T20:12:26.623 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T20:12:26.623 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2023-05-31T20:09:59.098535Z", 2023-05-31T20:12:26.624 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2023-05-31T20:12:26.624 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T20:12:26.624 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T20:12:26.624 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2023-05-31T20:12:26.624 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:12:26.625 INFO:teuthology.orchestra.run.smithi099.stdout: "full_sync": { 2023-05-31T20:12:26.625 INFO:teuthology.orchestra.run.smithi099.stdout: "total": 1, 2023-05-31T20:12:26.625 INFO:teuthology.orchestra.run.smithi099.stdout: "complete": 1 2023-05-31T20:12:26.625 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:12:26.625 INFO:teuthology.orchestra.run.smithi099.stdout: "current_time": "2023-05-31T20:12:26Z" 2023-05-31T20:12:26.625 INFO:teuthology.orchestra.run.smithi099.stdout:} 2023-05-31T20:12:26.626 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "2e6123d2-d695-4350-9a01-61a455bc782b", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1685562631.516089_59.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-05-31T19:50:31.516089Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1685562970.274572_93.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-05-31T19:56:10.274572Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1685563799.098535_267.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-05-31T20:09:59.098535Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-05-31T20:12:26Z" } 2023-05-31T20:12:26.627 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1685562631.516089_59.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-05-31T19:50:31.516089Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1685562970.274572_93.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-05-31T19:56:10.274572Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1685563799.098535_267.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-05-31T20:09:59.098535Z', 'realm_epoch': 2}}] 2023-05-31T20:12:26.627 DEBUG:rgw_multi.tests:log_status={0: '1_1685562631.516089_59.1', 1: '', 2: '1_1685562970.274572_93.1', 3: '1_1685563802.468126_296.1'} 2023-05-31T20:12:26.628 DEBUG:rgw_multi.tests:sync_status={0: '1_1685562631.516089_59.1', 1: '', 2: '1_1685562970.274572_93.1', 3: '1_1685563799.098535_267.1'} 2023-05-31T20:12:26.628 WARNING:rgw_multi.tests:zone a2 behind master: shard=3 master=1_1685563802.468126_296.1 target=1_1685563799.098535_267.1 2023-05-31T20:12:31.629 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-31T20:12:31.629 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-31T20:12:31.630 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-05-31T20:12:31.681 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2023-05-31T20:12:31.682 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2023-05-31T20:12:31.718 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:12:31.713+0000 7f1795161780 20 RGW-SYNC:meta: read sync status 2023-05-31T20:12:31.720 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:12:31.714+0000 7f1795161780 20 RGW-SYNC:meta: read sync status 2023-05-31T20:12:31.734 INFO:teuthology.orchestra.run.smithi099.stdout:{ 2023-05-31T20:12:31.735 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_status": { 2023-05-31T20:12:31.735 INFO:teuthology.orchestra.run.smithi099.stdout: "info": { 2023-05-31T20:12:31.735 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "sync", 2023-05-31T20:12:31.735 INFO:teuthology.orchestra.run.smithi099.stdout: "num_shards": 4, 2023-05-31T20:12:31.736 INFO:teuthology.orchestra.run.smithi099.stdout: "period": "2e6123d2-d695-4350-9a01-61a455bc782b", 2023-05-31T20:12:31.736 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2023-05-31T20:12:31.736 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:12:31.736 INFO:teuthology.orchestra.run.smithi099.stdout: "markers": [ 2023-05-31T20:12:31.737 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T20:12:31.737 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 0, 2023-05-31T20:12:31.737 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T20:12:31.738 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2023-05-31T20:12:31.738 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1685562631.516089_59.1", 2023-05-31T20:12:31.738 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T20:12:31.738 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 1, 2023-05-31T20:12:31.739 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T20:12:31.739 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2023-05-31T19:50:31.516089Z", 2023-05-31T20:12:31.739 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2023-05-31T20:12:31.740 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T20:12:31.741 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:12:31.741 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T20:12:31.741 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 1, 2023-05-31T20:12:31.742 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T20:12:31.742 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2023-05-31T20:12:31.742 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "", 2023-05-31T20:12:31.742 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T20:12:31.743 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2023-05-31T20:12:31.743 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T20:12:31.743 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "0.000000", 2023-05-31T20:12:31.744 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2023-05-31T20:12:31.745 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T20:12:31.745 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:12:31.745 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T20:12:31.745 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 2, 2023-05-31T20:12:31.745 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T20:12:31.745 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2023-05-31T20:12:31.745 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1685562970.274572_93.1", 2023-05-31T20:12:31.746 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T20:12:31.746 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2023-05-31T20:12:31.746 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T20:12:31.746 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2023-05-31T19:56:10.274572Z", 2023-05-31T20:12:31.746 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2023-05-31T20:12:31.746 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T20:12:31.747 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:12:31.747 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T20:12:31.747 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 3, 2023-05-31T20:12:31.747 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T20:12:31.747 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2023-05-31T20:12:31.747 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1685563798.970303_257.1", 2023-05-31T20:12:31.747 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T20:12:31.748 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2023-05-31T20:12:31.748 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T20:12:31.748 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2023-05-31T20:09:58.970303Z", 2023-05-31T20:12:31.748 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2023-05-31T20:12:31.748 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T20:12:31.748 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T20:12:31.749 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2023-05-31T20:12:31.749 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:12:31.749 INFO:teuthology.orchestra.run.smithi099.stdout: "full_sync": { 2023-05-31T20:12:31.749 INFO:teuthology.orchestra.run.smithi099.stdout: "total": 1, 2023-05-31T20:12:31.750 INFO:teuthology.orchestra.run.smithi099.stdout: "complete": 1 2023-05-31T20:12:31.750 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:12:31.750 INFO:teuthology.orchestra.run.smithi099.stdout: "current_time": "2023-05-31T20:12:31Z" 2023-05-31T20:12:31.750 INFO:teuthology.orchestra.run.smithi099.stdout:} 2023-05-31T20:12:31.751 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "2e6123d2-d695-4350-9a01-61a455bc782b", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1685562631.516089_59.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-05-31T19:50:31.516089Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1685562970.274572_93.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-05-31T19:56:10.274572Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1685563798.970303_257.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-05-31T20:09:58.970303Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-05-31T20:12:31Z" } 2023-05-31T20:12:31.753 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1685562631.516089_59.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-05-31T19:50:31.516089Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1685562970.274572_93.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-05-31T19:56:10.274572Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1685563798.970303_257.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-05-31T20:09:58.970303Z', 'realm_epoch': 2}}] 2023-05-31T20:12:31.753 DEBUG:rgw_multi.tests:log_status={0: '1_1685562631.516089_59.1', 1: '', 2: '1_1685562970.274572_93.1', 3: '1_1685563802.468126_296.1'} 2023-05-31T20:12:31.754 DEBUG:rgw_multi.tests:sync_status={0: '1_1685562631.516089_59.1', 1: '', 2: '1_1685562970.274572_93.1', 3: '1_1685563798.970303_257.1'} 2023-05-31T20:12:31.754 WARNING:rgw_multi.tests:zone a2 behind master: shard=3 master=1_1685563802.468126_296.1 target=1_1685563798.970303_257.1 2023-05-31T20:12:36.755 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-31T20:12:36.756 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-31T20:12:36.758 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-05-31T20:12:36.811 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2023-05-31T20:12:36.811 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2023-05-31T20:12:36.863 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:12:36.858+0000 7f8a1ea16780 20 RGW-SYNC:meta: read sync status 2023-05-31T20:12:36.864 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:12:36.859+0000 7f8a1ea16780 20 RGW-SYNC:meta: read sync status 2023-05-31T20:12:36.879 INFO:teuthology.orchestra.run.smithi099.stdout:{ 2023-05-31T20:12:36.879 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_status": { 2023-05-31T20:12:36.880 INFO:teuthology.orchestra.run.smithi099.stdout: "info": { 2023-05-31T20:12:36.880 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "sync", 2023-05-31T20:12:36.880 INFO:teuthology.orchestra.run.smithi099.stdout: "num_shards": 4, 2023-05-31T20:12:36.880 INFO:teuthology.orchestra.run.smithi099.stdout: "period": "2e6123d2-d695-4350-9a01-61a455bc782b", 2023-05-31T20:12:36.880 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2023-05-31T20:12:36.880 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:12:36.881 INFO:teuthology.orchestra.run.smithi099.stdout: "markers": [ 2023-05-31T20:12:36.881 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T20:12:36.881 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 0, 2023-05-31T20:12:36.881 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T20:12:36.881 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2023-05-31T20:12:36.881 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1685562631.516089_59.1", 2023-05-31T20:12:36.881 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T20:12:36.882 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 1, 2023-05-31T20:12:36.882 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T20:12:36.882 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2023-05-31T19:50:31.516089Z", 2023-05-31T20:12:36.883 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2023-05-31T20:12:36.883 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T20:12:36.883 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:12:36.883 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T20:12:36.883 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 1, 2023-05-31T20:12:36.884 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T20:12:36.884 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2023-05-31T20:12:36.884 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "", 2023-05-31T20:12:36.884 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T20:12:36.884 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2023-05-31T20:12:36.884 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T20:12:36.885 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "0.000000", 2023-05-31T20:12:36.885 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2023-05-31T20:12:36.885 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T20:12:36.885 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:12:36.885 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T20:12:36.885 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 2, 2023-05-31T20:12:36.885 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T20:12:36.886 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2023-05-31T20:12:36.886 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1685562970.274572_93.1", 2023-05-31T20:12:36.886 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T20:12:36.886 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2023-05-31T20:12:36.886 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T20:12:36.886 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2023-05-31T19:56:10.274572Z", 2023-05-31T20:12:36.886 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2023-05-31T20:12:36.887 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T20:12:36.887 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:12:36.889 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T20:12:36.889 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 3, 2023-05-31T20:12:36.890 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T20:12:36.890 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2023-05-31T20:12:36.890 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1685563798.885419_253.1", 2023-05-31T20:12:36.890 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T20:12:36.891 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2023-05-31T20:12:36.891 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T20:12:36.891 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2023-05-31T20:09:58.885419Z", 2023-05-31T20:12:36.891 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2023-05-31T20:12:36.892 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T20:12:36.892 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T20:12:36.892 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2023-05-31T20:12:36.893 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:12:36.893 INFO:teuthology.orchestra.run.smithi099.stdout: "full_sync": { 2023-05-31T20:12:36.893 INFO:teuthology.orchestra.run.smithi099.stdout: "total": 1, 2023-05-31T20:12:36.894 INFO:teuthology.orchestra.run.smithi099.stdout: "complete": 1 2023-05-31T20:12:36.894 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:12:36.894 INFO:teuthology.orchestra.run.smithi099.stdout: "current_time": "2023-05-31T20:12:36Z" 2023-05-31T20:12:36.895 INFO:teuthology.orchestra.run.smithi099.stdout:} 2023-05-31T20:12:36.896 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "2e6123d2-d695-4350-9a01-61a455bc782b", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1685562631.516089_59.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-05-31T19:50:31.516089Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1685562970.274572_93.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-05-31T19:56:10.274572Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1685563798.885419_253.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-05-31T20:09:58.885419Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-05-31T20:12:36Z" } 2023-05-31T20:12:36.897 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1685562631.516089_59.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-05-31T19:50:31.516089Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1685562970.274572_93.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-05-31T19:56:10.274572Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1685563798.885419_253.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-05-31T20:09:58.885419Z', 'realm_epoch': 2}}] 2023-05-31T20:12:36.898 DEBUG:rgw_multi.tests:log_status={0: '1_1685562631.516089_59.1', 1: '', 2: '1_1685562970.274572_93.1', 3: '1_1685563802.468126_296.1'} 2023-05-31T20:12:36.899 DEBUG:rgw_multi.tests:sync_status={0: '1_1685562631.516089_59.1', 1: '', 2: '1_1685562970.274572_93.1', 3: '1_1685563798.885419_253.1'} 2023-05-31T20:12:36.899 WARNING:rgw_multi.tests:zone a2 behind master: shard=3 master=1_1685563802.468126_296.1 target=1_1685563798.885419_253.1 2023-05-31T20:12:41.901 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-31T20:12:41.902 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-31T20:12:41.903 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-05-31T20:12:41.958 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2023-05-31T20:12:41.959 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2023-05-31T20:12:42.001 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:12:41.996+0000 7f0995824780 20 RGW-SYNC:meta: read sync status 2023-05-31T20:12:42.003 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:12:41.998+0000 7f0995824780 20 RGW-SYNC:meta: read sync status 2023-05-31T20:12:42.021 INFO:teuthology.orchestra.run.smithi099.stdout:{ 2023-05-31T20:12:42.022 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_status": { 2023-05-31T20:12:42.022 INFO:teuthology.orchestra.run.smithi099.stdout: "info": { 2023-05-31T20:12:42.022 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "sync", 2023-05-31T20:12:42.022 INFO:teuthology.orchestra.run.smithi099.stdout: "num_shards": 4, 2023-05-31T20:12:42.023 INFO:teuthology.orchestra.run.smithi099.stdout: "period": "2e6123d2-d695-4350-9a01-61a455bc782b", 2023-05-31T20:12:42.023 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2023-05-31T20:12:42.023 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:12:42.023 INFO:teuthology.orchestra.run.smithi099.stdout: "markers": [ 2023-05-31T20:12:42.023 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T20:12:42.024 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 0, 2023-05-31T20:12:42.024 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T20:12:42.024 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2023-05-31T20:12:42.024 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1685562631.516089_59.1", 2023-05-31T20:12:42.024 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T20:12:42.025 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 1, 2023-05-31T20:12:42.025 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T20:12:42.025 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2023-05-31T19:50:31.516089Z", 2023-05-31T20:12:42.025 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2023-05-31T20:12:42.025 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T20:12:42.026 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:12:42.026 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T20:12:42.026 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 1, 2023-05-31T20:12:42.026 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T20:12:42.026 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2023-05-31T20:12:42.026 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "", 2023-05-31T20:12:42.027 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T20:12:42.027 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2023-05-31T20:12:42.027 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T20:12:42.027 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "0.000000", 2023-05-31T20:12:42.027 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2023-05-31T20:12:42.027 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T20:12:42.027 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:12:42.028 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T20:12:42.029 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 2, 2023-05-31T20:12:42.029 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T20:12:42.029 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2023-05-31T20:12:42.030 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1685562970.274572_93.1", 2023-05-31T20:12:42.030 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T20:12:42.030 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2023-05-31T20:12:42.031 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T20:12:42.031 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2023-05-31T19:56:10.274572Z", 2023-05-31T20:12:42.031 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2023-05-31T20:12:42.032 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T20:12:42.032 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:12:42.032 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T20:12:42.033 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 3, 2023-05-31T20:12:42.033 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T20:12:42.033 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2023-05-31T20:12:42.033 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1685563798.807987_249.1", 2023-05-31T20:12:42.033 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T20:12:42.033 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2023-05-31T20:12:42.034 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T20:12:42.034 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2023-05-31T20:09:58.807987Z", 2023-05-31T20:12:42.034 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2023-05-31T20:12:42.034 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T20:12:42.034 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T20:12:42.034 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2023-05-31T20:12:42.034 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:12:42.035 INFO:teuthology.orchestra.run.smithi099.stdout: "full_sync": { 2023-05-31T20:12:42.035 INFO:teuthology.orchestra.run.smithi099.stdout: "total": 1, 2023-05-31T20:12:42.035 INFO:teuthology.orchestra.run.smithi099.stdout: "complete": 1 2023-05-31T20:12:42.035 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:12:42.036 INFO:teuthology.orchestra.run.smithi099.stdout: "current_time": "2023-05-31T20:12:41Z" 2023-05-31T20:12:42.036 INFO:teuthology.orchestra.run.smithi099.stdout:} 2023-05-31T20:12:42.037 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "2e6123d2-d695-4350-9a01-61a455bc782b", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1685562631.516089_59.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-05-31T19:50:31.516089Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1685562970.274572_93.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-05-31T19:56:10.274572Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1685563798.807987_249.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-05-31T20:09:58.807987Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-05-31T20:12:41Z" } 2023-05-31T20:12:42.037 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1685562631.516089_59.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-05-31T19:50:31.516089Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1685562970.274572_93.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-05-31T19:56:10.274572Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1685563798.807987_249.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-05-31T20:09:58.807987Z', 'realm_epoch': 2}}] 2023-05-31T20:12:42.038 DEBUG:rgw_multi.tests:log_status={0: '1_1685562631.516089_59.1', 1: '', 2: '1_1685562970.274572_93.1', 3: '1_1685563802.468126_296.1'} 2023-05-31T20:12:42.038 DEBUG:rgw_multi.tests:sync_status={0: '1_1685562631.516089_59.1', 1: '', 2: '1_1685562970.274572_93.1', 3: '1_1685563798.807987_249.1'} 2023-05-31T20:12:42.039 WARNING:rgw_multi.tests:zone a2 behind master: shard=3 master=1_1685563802.468126_296.1 target=1_1685563798.807987_249.1 2023-05-31T20:12:47.040 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-31T20:12:47.040 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-31T20:12:47.041 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-05-31T20:12:47.092 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2023-05-31T20:12:47.092 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2023-05-31T20:12:47.133 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:12:47.127+0000 7f20dd552780 20 RGW-SYNC:meta: read sync status 2023-05-31T20:12:47.134 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:12:47.128+0000 7f20dd552780 20 RGW-SYNC:meta: read sync status 2023-05-31T20:12:47.151 INFO:teuthology.orchestra.run.smithi099.stdout:{ 2023-05-31T20:12:47.151 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_status": { 2023-05-31T20:12:47.151 INFO:teuthology.orchestra.run.smithi099.stdout: "info": { 2023-05-31T20:12:47.152 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "sync", 2023-05-31T20:12:47.152 INFO:teuthology.orchestra.run.smithi099.stdout: "num_shards": 4, 2023-05-31T20:12:47.152 INFO:teuthology.orchestra.run.smithi099.stdout: "period": "2e6123d2-d695-4350-9a01-61a455bc782b", 2023-05-31T20:12:47.152 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2023-05-31T20:12:47.152 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:12:47.152 INFO:teuthology.orchestra.run.smithi099.stdout: "markers": [ 2023-05-31T20:12:47.152 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T20:12:47.153 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 0, 2023-05-31T20:12:47.153 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T20:12:47.153 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2023-05-31T20:12:47.153 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1685562631.516089_59.1", 2023-05-31T20:12:47.153 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T20:12:47.153 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 1, 2023-05-31T20:12:47.154 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T20:12:47.154 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2023-05-31T19:50:31.516089Z", 2023-05-31T20:12:47.154 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2023-05-31T20:12:47.154 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T20:12:47.154 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:12:47.154 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T20:12:47.154 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 1, 2023-05-31T20:12:47.155 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T20:12:47.155 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2023-05-31T20:12:47.155 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "", 2023-05-31T20:12:47.155 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T20:12:47.156 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2023-05-31T20:12:47.156 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T20:12:47.156 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "0.000000", 2023-05-31T20:12:47.156 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2023-05-31T20:12:47.156 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T20:12:47.156 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:12:47.157 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T20:12:47.157 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 2, 2023-05-31T20:12:47.157 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T20:12:47.157 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2023-05-31T20:12:47.157 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1685562970.274572_93.1", 2023-05-31T20:12:47.158 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T20:12:47.158 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2023-05-31T20:12:47.158 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T20:12:47.158 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2023-05-31T19:56:10.274572Z", 2023-05-31T20:12:47.158 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2023-05-31T20:12:47.159 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T20:12:47.159 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:12:47.159 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T20:12:47.159 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 3, 2023-05-31T20:12:47.159 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T20:12:47.161 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2023-05-31T20:12:47.162 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1685563798.772373_248.1", 2023-05-31T20:12:47.162 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T20:12:47.163 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2023-05-31T20:12:47.163 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T20:12:47.164 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2023-05-31T20:09:58.772373Z", 2023-05-31T20:12:47.165 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2023-05-31T20:12:47.165 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T20:12:47.165 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T20:12:47.165 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2023-05-31T20:12:47.165 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:12:47.165 INFO:teuthology.orchestra.run.smithi099.stdout: "full_sync": { 2023-05-31T20:12:47.166 INFO:teuthology.orchestra.run.smithi099.stdout: "total": 1, 2023-05-31T20:12:47.166 INFO:teuthology.orchestra.run.smithi099.stdout: "complete": 1 2023-05-31T20:12:47.166 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:12:47.166 INFO:teuthology.orchestra.run.smithi099.stdout: "current_time": "2023-05-31T20:12:47Z" 2023-05-31T20:12:47.166 INFO:teuthology.orchestra.run.smithi099.stdout:} 2023-05-31T20:12:47.167 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "2e6123d2-d695-4350-9a01-61a455bc782b", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1685562631.516089_59.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-05-31T19:50:31.516089Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1685562970.274572_93.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-05-31T19:56:10.274572Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1685563798.772373_248.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-05-31T20:09:58.772373Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-05-31T20:12:47Z" } 2023-05-31T20:12:47.168 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1685562631.516089_59.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-05-31T19:50:31.516089Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1685562970.274572_93.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-05-31T19:56:10.274572Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1685563798.772373_248.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-05-31T20:09:58.772373Z', 'realm_epoch': 2}}] 2023-05-31T20:12:47.168 DEBUG:rgw_multi.tests:log_status={0: '1_1685562631.516089_59.1', 1: '', 2: '1_1685562970.274572_93.1', 3: '1_1685563802.468126_296.1'} 2023-05-31T20:12:47.169 DEBUG:rgw_multi.tests:sync_status={0: '1_1685562631.516089_59.1', 1: '', 2: '1_1685562970.274572_93.1', 3: '1_1685563798.772373_248.1'} 2023-05-31T20:12:47.169 WARNING:rgw_multi.tests:zone a2 behind master: shard=3 master=1_1685563802.468126_296.1 target=1_1685563798.772373_248.1 2023-05-31T20:12:52.170 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-31T20:12:52.171 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-31T20:12:52.171 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-05-31T20:12:52.223 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2023-05-31T20:12:52.223 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2023-05-31T20:12:52.287 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:12:52.281+0000 7f860d200780 20 RGW-SYNC:meta: read sync status 2023-05-31T20:12:52.287 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:12:52.282+0000 7f860d200780 20 RGW-SYNC:meta: read sync status 2023-05-31T20:12:52.309 INFO:teuthology.orchestra.run.smithi099.stdout:{ 2023-05-31T20:12:52.310 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_status": { 2023-05-31T20:12:52.310 INFO:teuthology.orchestra.run.smithi099.stdout: "info": { 2023-05-31T20:12:52.310 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "sync", 2023-05-31T20:12:52.310 INFO:teuthology.orchestra.run.smithi099.stdout: "num_shards": 4, 2023-05-31T20:12:52.310 INFO:teuthology.orchestra.run.smithi099.stdout: "period": "2e6123d2-d695-4350-9a01-61a455bc782b", 2023-05-31T20:12:52.310 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2023-05-31T20:12:52.310 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:12:52.311 INFO:teuthology.orchestra.run.smithi099.stdout: "markers": [ 2023-05-31T20:12:52.311 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T20:12:52.311 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 0, 2023-05-31T20:12:52.311 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T20:12:52.311 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2023-05-31T20:12:52.312 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1685562631.516089_59.1", 2023-05-31T20:12:52.312 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T20:12:52.312 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 1, 2023-05-31T20:12:52.312 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T20:12:52.312 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2023-05-31T19:50:31.516089Z", 2023-05-31T20:12:52.313 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2023-05-31T20:12:52.313 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T20:12:52.313 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:12:52.313 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T20:12:52.313 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 1, 2023-05-31T20:12:52.313 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T20:12:52.313 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2023-05-31T20:12:52.314 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "", 2023-05-31T20:12:52.314 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T20:12:52.314 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2023-05-31T20:12:52.314 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T20:12:52.314 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "0.000000", 2023-05-31T20:12:52.314 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2023-05-31T20:12:52.314 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T20:12:52.315 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:12:52.319 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T20:12:52.319 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 2, 2023-05-31T20:12:52.320 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T20:12:52.320 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2023-05-31T20:12:52.320 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1685562970.274572_93.1", 2023-05-31T20:12:52.320 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T20:12:52.320 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2023-05-31T20:12:52.320 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T20:12:52.321 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2023-05-31T19:56:10.274572Z", 2023-05-31T20:12:52.321 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2023-05-31T20:12:52.321 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T20:12:52.321 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:12:52.321 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T20:12:52.321 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 3, 2023-05-31T20:12:52.321 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T20:12:52.322 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2023-05-31T20:12:52.322 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1685563798.885419_253.1", 2023-05-31T20:12:52.322 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T20:12:52.322 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2023-05-31T20:12:52.322 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T20:12:52.322 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2023-05-31T20:09:58.885419Z", 2023-05-31T20:12:52.322 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2023-05-31T20:12:52.323 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T20:12:52.323 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T20:12:52.323 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2023-05-31T20:12:52.323 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:12:52.323 INFO:teuthology.orchestra.run.smithi099.stdout: "full_sync": { 2023-05-31T20:12:52.323 INFO:teuthology.orchestra.run.smithi099.stdout: "total": 1, 2023-05-31T20:12:52.323 INFO:teuthology.orchestra.run.smithi099.stdout: "complete": 1 2023-05-31T20:12:52.324 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:12:52.324 INFO:teuthology.orchestra.run.smithi099.stdout: "current_time": "2023-05-31T20:12:52Z" 2023-05-31T20:12:52.324 INFO:teuthology.orchestra.run.smithi099.stdout:} 2023-05-31T20:12:52.325 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "2e6123d2-d695-4350-9a01-61a455bc782b", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1685562631.516089_59.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-05-31T19:50:31.516089Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1685562970.274572_93.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-05-31T19:56:10.274572Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1685563798.885419_253.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-05-31T20:09:58.885419Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-05-31T20:12:52Z" } 2023-05-31T20:12:52.325 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1685562631.516089_59.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-05-31T19:50:31.516089Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1685562970.274572_93.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-05-31T19:56:10.274572Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1685563798.885419_253.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-05-31T20:09:58.885419Z', 'realm_epoch': 2}}] 2023-05-31T20:12:52.326 DEBUG:rgw_multi.tests:log_status={0: '1_1685562631.516089_59.1', 1: '', 2: '1_1685562970.274572_93.1', 3: '1_1685563802.468126_296.1'} 2023-05-31T20:12:52.326 DEBUG:rgw_multi.tests:sync_status={0: '1_1685562631.516089_59.1', 1: '', 2: '1_1685562970.274572_93.1', 3: '1_1685563798.885419_253.1'} 2023-05-31T20:12:52.326 WARNING:rgw_multi.tests:zone a2 behind master: shard=3 master=1_1685563802.468126_296.1 target=1_1685563798.885419_253.1 2023-05-31T20:12:57.327 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-31T20:12:57.328 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-31T20:12:57.328 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-05-31T20:12:57.380 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2023-05-31T20:12:57.380 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2023-05-31T20:12:57.412 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:12:57.406+0000 7f7753179780 20 RGW-SYNC:meta: read sync status 2023-05-31T20:12:57.424 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:12:57.407+0000 7f7753179780 20 RGW-SYNC:meta: read sync status 2023-05-31T20:12:57.425 INFO:teuthology.orchestra.run.smithi099.stdout:{ 2023-05-31T20:12:57.426 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_status": { 2023-05-31T20:12:57.426 INFO:teuthology.orchestra.run.smithi099.stdout: "info": { 2023-05-31T20:12:57.427 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "sync", 2023-05-31T20:12:57.427 INFO:teuthology.orchestra.run.smithi099.stdout: "num_shards": 4, 2023-05-31T20:12:57.427 INFO:teuthology.orchestra.run.smithi099.stdout: "period": "2e6123d2-d695-4350-9a01-61a455bc782b", 2023-05-31T20:12:57.427 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2023-05-31T20:12:57.430 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:12:57.430 INFO:teuthology.orchestra.run.smithi099.stdout: "markers": [ 2023-05-31T20:12:57.430 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T20:12:57.431 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 0, 2023-05-31T20:12:57.431 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T20:12:57.431 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2023-05-31T20:12:57.431 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1685562631.516089_59.1", 2023-05-31T20:12:57.432 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T20:12:57.432 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 1, 2023-05-31T20:12:57.432 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T20:12:57.434 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2023-05-31T19:50:31.516089Z", 2023-05-31T20:12:57.435 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2023-05-31T20:12:57.435 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T20:12:57.435 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:12:57.435 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T20:12:57.435 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 1, 2023-05-31T20:12:57.435 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T20:12:57.435 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2023-05-31T20:12:57.436 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "", 2023-05-31T20:12:57.436 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T20:12:57.436 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2023-05-31T20:12:57.436 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T20:12:57.436 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "0.000000", 2023-05-31T20:12:57.436 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2023-05-31T20:12:57.437 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T20:12:57.437 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:12:57.437 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T20:12:57.437 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 2, 2023-05-31T20:12:57.437 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T20:12:57.438 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2023-05-31T20:12:57.438 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1685562970.274572_93.1", 2023-05-31T20:12:57.438 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T20:12:57.438 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2023-05-31T20:12:57.438 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T20:12:57.439 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2023-05-31T19:56:10.274572Z", 2023-05-31T20:12:57.439 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2023-05-31T20:12:57.439 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T20:12:57.440 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:12:57.440 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T20:12:57.440 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 3, 2023-05-31T20:12:57.440 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T20:12:57.440 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2023-05-31T20:12:57.440 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1685563799.051608_261.1", 2023-05-31T20:12:57.441 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T20:12:57.441 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2023-05-31T20:12:57.441 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T20:12:57.441 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2023-05-31T20:09:59.051608Z", 2023-05-31T20:12:57.441 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2023-05-31T20:12:57.441 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T20:12:57.442 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T20:12:57.442 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2023-05-31T20:12:57.442 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:12:57.442 INFO:teuthology.orchestra.run.smithi099.stdout: "full_sync": { 2023-05-31T20:12:57.442 INFO:teuthology.orchestra.run.smithi099.stdout: "total": 1, 2023-05-31T20:12:57.442 INFO:teuthology.orchestra.run.smithi099.stdout: "complete": 1 2023-05-31T20:12:57.442 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:12:57.443 INFO:teuthology.orchestra.run.smithi099.stdout: "current_time": "2023-05-31T20:12:57Z" 2023-05-31T20:12:57.443 INFO:teuthology.orchestra.run.smithi099.stdout:} 2023-05-31T20:12:57.444 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "2e6123d2-d695-4350-9a01-61a455bc782b", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1685562631.516089_59.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-05-31T19:50:31.516089Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1685562970.274572_93.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-05-31T19:56:10.274572Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1685563799.051608_261.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-05-31T20:09:59.051608Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-05-31T20:12:57Z" } 2023-05-31T20:12:57.444 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1685562631.516089_59.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-05-31T19:50:31.516089Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1685562970.274572_93.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-05-31T19:56:10.274572Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1685563799.051608_261.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-05-31T20:09:59.051608Z', 'realm_epoch': 2}}] 2023-05-31T20:12:57.445 DEBUG:rgw_multi.tests:log_status={0: '1_1685562631.516089_59.1', 1: '', 2: '1_1685562970.274572_93.1', 3: '1_1685563802.468126_296.1'} 2023-05-31T20:12:57.445 DEBUG:rgw_multi.tests:sync_status={0: '1_1685562631.516089_59.1', 1: '', 2: '1_1685562970.274572_93.1', 3: '1_1685563799.051608_261.1'} 2023-05-31T20:12:57.445 WARNING:rgw_multi.tests:zone a2 behind master: shard=3 master=1_1685563802.468126_296.1 target=1_1685563799.051608_261.1 2023-05-31T20:13:02.447 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-31T20:13:02.448 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-31T20:13:02.448 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-05-31T20:13:02.528 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2023-05-31T20:13:02.529 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2023-05-31T20:13:02.583 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:13:02.578+0000 7f6618cde780 20 RGW-SYNC:meta: read sync status 2023-05-31T20:13:02.585 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:13:02.579+0000 7f6618cde780 20 RGW-SYNC:meta: read sync status 2023-05-31T20:13:02.598 INFO:teuthology.orchestra.run.smithi099.stdout:{ 2023-05-31T20:13:02.598 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_status": { 2023-05-31T20:13:02.598 INFO:teuthology.orchestra.run.smithi099.stdout: "info": { 2023-05-31T20:13:02.598 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "sync", 2023-05-31T20:13:02.599 INFO:teuthology.orchestra.run.smithi099.stdout: "num_shards": 4, 2023-05-31T20:13:02.599 INFO:teuthology.orchestra.run.smithi099.stdout: "period": "2e6123d2-d695-4350-9a01-61a455bc782b", 2023-05-31T20:13:02.599 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2023-05-31T20:13:02.599 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:13:02.599 INFO:teuthology.orchestra.run.smithi099.stdout: "markers": [ 2023-05-31T20:13:02.600 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T20:13:02.600 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 0, 2023-05-31T20:13:02.600 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T20:13:02.600 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2023-05-31T20:13:02.600 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1685562631.516089_59.1", 2023-05-31T20:13:02.600 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T20:13:02.601 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 1, 2023-05-31T20:13:02.601 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T20:13:02.601 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2023-05-31T19:50:31.516089Z", 2023-05-31T20:13:02.601 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2023-05-31T20:13:02.601 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T20:13:02.601 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:13:02.602 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T20:13:02.602 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 1, 2023-05-31T20:13:02.602 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T20:13:02.602 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2023-05-31T20:13:02.602 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "", 2023-05-31T20:13:02.602 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T20:13:02.602 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2023-05-31T20:13:02.603 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T20:13:02.603 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "0.000000", 2023-05-31T20:13:02.603 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2023-05-31T20:13:02.603 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T20:13:02.603 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:13:02.603 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T20:13:02.603 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 2, 2023-05-31T20:13:02.604 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T20:13:02.604 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2023-05-31T20:13:02.604 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1685562970.274572_93.1", 2023-05-31T20:13:02.604 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T20:13:02.604 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2023-05-31T20:13:02.604 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T20:13:02.605 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2023-05-31T19:56:10.274572Z", 2023-05-31T20:13:02.605 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2023-05-31T20:13:02.605 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T20:13:02.605 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:13:02.605 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T20:13:02.605 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 3, 2023-05-31T20:13:02.605 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T20:13:02.606 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2023-05-31T20:13:02.606 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1685563798.756239_246.1", 2023-05-31T20:13:02.606 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T20:13:02.606 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2023-05-31T20:13:02.606 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T20:13:02.606 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2023-05-31T20:09:58.756239Z", 2023-05-31T20:13:02.606 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2023-05-31T20:13:02.607 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T20:13:02.607 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T20:13:02.607 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2023-05-31T20:13:02.607 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:13:02.607 INFO:teuthology.orchestra.run.smithi099.stdout: "full_sync": { 2023-05-31T20:13:02.607 INFO:teuthology.orchestra.run.smithi099.stdout: "total": 1, 2023-05-31T20:13:02.607 INFO:teuthology.orchestra.run.smithi099.stdout: "complete": 1 2023-05-31T20:13:02.608 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:13:02.608 INFO:teuthology.orchestra.run.smithi099.stdout: "current_time": "2023-05-31T20:13:02Z" 2023-05-31T20:13:02.608 INFO:teuthology.orchestra.run.smithi099.stdout:} 2023-05-31T20:13:02.609 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "2e6123d2-d695-4350-9a01-61a455bc782b", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1685562631.516089_59.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-05-31T19:50:31.516089Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1685562970.274572_93.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-05-31T19:56:10.274572Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1685563798.756239_246.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-05-31T20:09:58.756239Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-05-31T20:13:02Z" } 2023-05-31T20:13:02.610 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1685562631.516089_59.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-05-31T19:50:31.516089Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1685562970.274572_93.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-05-31T19:56:10.274572Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1685563798.756239_246.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-05-31T20:09:58.756239Z', 'realm_epoch': 2}}] 2023-05-31T20:13:02.610 DEBUG:rgw_multi.tests:log_status={0: '1_1685562631.516089_59.1', 1: '', 2: '1_1685562970.274572_93.1', 3: '1_1685563802.468126_296.1'} 2023-05-31T20:13:02.610 DEBUG:rgw_multi.tests:sync_status={0: '1_1685562631.516089_59.1', 1: '', 2: '1_1685562970.274572_93.1', 3: '1_1685563798.756239_246.1'} 2023-05-31T20:13:02.611 WARNING:rgw_multi.tests:zone a2 behind master: shard=3 master=1_1685563802.468126_296.1 target=1_1685563798.756239_246.1 2023-05-31T20:13:07.611 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-31T20:13:07.612 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-31T20:13:07.613 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-05-31T20:13:07.689 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2023-05-31T20:13:07.690 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2023-05-31T20:13:07.744 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:13:07.739+0000 7f4fd7d70780 20 RGW-SYNC:meta: read sync status 2023-05-31T20:13:07.745 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:13:07.740+0000 7f4fd7d70780 20 RGW-SYNC:meta: read sync status 2023-05-31T20:13:07.763 INFO:teuthology.orchestra.run.smithi099.stdout:{ 2023-05-31T20:13:07.763 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_status": { 2023-05-31T20:13:07.764 INFO:teuthology.orchestra.run.smithi099.stdout: "info": { 2023-05-31T20:13:07.764 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "sync", 2023-05-31T20:13:07.769 INFO:teuthology.orchestra.run.smithi099.stdout: "num_shards": 4, 2023-05-31T20:13:07.769 INFO:teuthology.orchestra.run.smithi099.stdout: "period": "2e6123d2-d695-4350-9a01-61a455bc782b", 2023-05-31T20:13:07.769 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2023-05-31T20:13:07.769 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:13:07.770 INFO:teuthology.orchestra.run.smithi099.stdout: "markers": [ 2023-05-31T20:13:07.770 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T20:13:07.770 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 0, 2023-05-31T20:13:07.770 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T20:13:07.770 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2023-05-31T20:13:07.770 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1685562631.516089_59.1", 2023-05-31T20:13:07.770 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T20:13:07.771 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 1, 2023-05-31T20:13:07.771 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T20:13:07.771 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2023-05-31T19:50:31.516089Z", 2023-05-31T20:13:07.771 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2023-05-31T20:13:07.771 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T20:13:07.771 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:13:07.772 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T20:13:07.772 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 1, 2023-05-31T20:13:07.772 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T20:13:07.775 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2023-05-31T20:13:07.775 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "", 2023-05-31T20:13:07.775 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T20:13:07.775 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2023-05-31T20:13:07.776 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T20:13:07.776 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "0.000000", 2023-05-31T20:13:07.776 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2023-05-31T20:13:07.776 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T20:13:07.776 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:13:07.777 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T20:13:07.777 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 2, 2023-05-31T20:13:07.777 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T20:13:07.777 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2023-05-31T20:13:07.777 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1685562970.274572_93.1", 2023-05-31T20:13:07.777 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T20:13:07.777 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2023-05-31T20:13:07.778 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T20:13:07.778 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2023-05-31T19:56:10.274572Z", 2023-05-31T20:13:07.779 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2023-05-31T20:13:07.779 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T20:13:07.780 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:13:07.780 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T20:13:07.780 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 3, 2023-05-31T20:13:07.780 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T20:13:07.780 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2023-05-31T20:13:07.780 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1685563798.901482_255.1", 2023-05-31T20:13:07.781 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T20:13:07.781 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2023-05-31T20:13:07.781 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T20:13:07.781 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2023-05-31T20:09:58.901482Z", 2023-05-31T20:13:07.781 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2023-05-31T20:13:07.781 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T20:13:07.782 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T20:13:07.782 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2023-05-31T20:13:07.782 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:13:07.782 INFO:teuthology.orchestra.run.smithi099.stdout: "full_sync": { 2023-05-31T20:13:07.782 INFO:teuthology.orchestra.run.smithi099.stdout: "total": 1, 2023-05-31T20:13:07.782 INFO:teuthology.orchestra.run.smithi099.stdout: "complete": 1 2023-05-31T20:13:07.783 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:13:07.783 INFO:teuthology.orchestra.run.smithi099.stdout: "current_time": "2023-05-31T20:13:07Z" 2023-05-31T20:13:07.783 INFO:teuthology.orchestra.run.smithi099.stdout:} 2023-05-31T20:13:07.784 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "2e6123d2-d695-4350-9a01-61a455bc782b", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1685562631.516089_59.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-05-31T19:50:31.516089Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1685562970.274572_93.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-05-31T19:56:10.274572Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1685563798.901482_255.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-05-31T20:09:58.901482Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-05-31T20:13:07Z" } 2023-05-31T20:13:07.784 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1685562631.516089_59.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-05-31T19:50:31.516089Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1685562970.274572_93.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-05-31T19:56:10.274572Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1685563798.901482_255.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-05-31T20:09:58.901482Z', 'realm_epoch': 2}}] 2023-05-31T20:13:07.785 DEBUG:rgw_multi.tests:log_status={0: '1_1685562631.516089_59.1', 1: '', 2: '1_1685562970.274572_93.1', 3: '1_1685563802.468126_296.1'} 2023-05-31T20:13:07.785 DEBUG:rgw_multi.tests:sync_status={0: '1_1685562631.516089_59.1', 1: '', 2: '1_1685562970.274572_93.1', 3: '1_1685563798.901482_255.1'} 2023-05-31T20:13:07.785 WARNING:rgw_multi.tests:zone a2 behind master: shard=3 master=1_1685563802.468126_296.1 target=1_1685563798.901482_255.1 2023-05-31T20:13:12.786 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-31T20:13:12.787 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-31T20:13:12.788 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-05-31T20:13:12.841 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2023-05-31T20:13:12.841 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2023-05-31T20:13:12.885 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:13:12.880+0000 7f610134f780 20 RGW-SYNC:meta: read sync status 2023-05-31T20:13:12.886 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:13:12.881+0000 7f610134f780 20 RGW-SYNC:meta: read sync status 2023-05-31T20:13:12.902 INFO:teuthology.orchestra.run.smithi099.stdout:{ 2023-05-31T20:13:12.902 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_status": { 2023-05-31T20:13:12.902 INFO:teuthology.orchestra.run.smithi099.stdout: "info": { 2023-05-31T20:13:12.903 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "sync", 2023-05-31T20:13:12.903 INFO:teuthology.orchestra.run.smithi099.stdout: "num_shards": 4, 2023-05-31T20:13:12.903 INFO:teuthology.orchestra.run.smithi099.stdout: "period": "2e6123d2-d695-4350-9a01-61a455bc782b", 2023-05-31T20:13:12.903 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2023-05-31T20:13:12.903 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:13:12.903 INFO:teuthology.orchestra.run.smithi099.stdout: "markers": [ 2023-05-31T20:13:12.903 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T20:13:12.904 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 0, 2023-05-31T20:13:12.904 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T20:13:12.904 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2023-05-31T20:13:12.904 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1685562631.516089_59.1", 2023-05-31T20:13:12.904 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T20:13:12.904 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 1, 2023-05-31T20:13:12.905 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T20:13:12.905 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2023-05-31T19:50:31.516089Z", 2023-05-31T20:13:12.905 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2023-05-31T20:13:12.905 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T20:13:12.905 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:13:12.905 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T20:13:12.905 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 1, 2023-05-31T20:13:12.906 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T20:13:12.906 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2023-05-31T20:13:12.906 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "", 2023-05-31T20:13:12.906 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T20:13:12.906 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2023-05-31T20:13:12.906 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T20:13:12.906 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "0.000000", 2023-05-31T20:13:12.907 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2023-05-31T20:13:12.907 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T20:13:12.907 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:13:12.907 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T20:13:12.907 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 2, 2023-05-31T20:13:12.907 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T20:13:12.907 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2023-05-31T20:13:12.908 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1685562970.274572_93.1", 2023-05-31T20:13:12.908 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T20:13:12.908 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2023-05-31T20:13:12.908 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T20:13:12.908 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2023-05-31T19:56:10.274572Z", 2023-05-31T20:13:12.908 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2023-05-31T20:13:12.909 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T20:13:12.909 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:13:12.909 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T20:13:12.909 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 3, 2023-05-31T20:13:12.909 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T20:13:12.909 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2023-05-31T20:13:12.909 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1685563798.772373_248.1", 2023-05-31T20:13:12.910 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T20:13:12.910 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2023-05-31T20:13:12.910 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T20:13:12.910 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2023-05-31T20:09:58.772373Z", 2023-05-31T20:13:12.910 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2023-05-31T20:13:12.910 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T20:13:12.910 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T20:13:12.911 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2023-05-31T20:13:12.911 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:13:12.911 INFO:teuthology.orchestra.run.smithi099.stdout: "full_sync": { 2023-05-31T20:13:12.911 INFO:teuthology.orchestra.run.smithi099.stdout: "total": 1, 2023-05-31T20:13:12.911 INFO:teuthology.orchestra.run.smithi099.stdout: "complete": 1 2023-05-31T20:13:12.911 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:13:12.911 INFO:teuthology.orchestra.run.smithi099.stdout: "current_time": "2023-05-31T20:13:12Z" 2023-05-31T20:13:12.912 INFO:teuthology.orchestra.run.smithi099.stdout:} 2023-05-31T20:13:12.913 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "2e6123d2-d695-4350-9a01-61a455bc782b", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1685562631.516089_59.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-05-31T19:50:31.516089Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1685562970.274572_93.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-05-31T19:56:10.274572Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1685563798.772373_248.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-05-31T20:09:58.772373Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-05-31T20:13:12Z" } 2023-05-31T20:13:12.913 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1685562631.516089_59.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-05-31T19:50:31.516089Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1685562970.274572_93.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-05-31T19:56:10.274572Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1685563798.772373_248.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-05-31T20:09:58.772373Z', 'realm_epoch': 2}}] 2023-05-31T20:13:12.913 DEBUG:rgw_multi.tests:log_status={0: '1_1685562631.516089_59.1', 1: '', 2: '1_1685562970.274572_93.1', 3: '1_1685563802.468126_296.1'} 2023-05-31T20:13:12.914 DEBUG:rgw_multi.tests:sync_status={0: '1_1685562631.516089_59.1', 1: '', 2: '1_1685562970.274572_93.1', 3: '1_1685563798.772373_248.1'} 2023-05-31T20:13:12.914 WARNING:rgw_multi.tests:zone a2 behind master: shard=3 master=1_1685563802.468126_296.1 target=1_1685563798.772373_248.1 2023-05-31T20:13:17.916 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-31T20:13:17.916 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-31T20:13:17.917 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-05-31T20:13:17.971 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2023-05-31T20:13:17.971 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2023-05-31T20:13:18.024 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:13:18.019+0000 7f41f1ed0780 20 RGW-SYNC:meta: read sync status 2023-05-31T20:13:18.025 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:13:18.021+0000 7f41f1ed0780 20 RGW-SYNC:meta: read sync status 2023-05-31T20:13:18.043 INFO:teuthology.orchestra.run.smithi099.stdout:{ 2023-05-31T20:13:18.044 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_status": { 2023-05-31T20:13:18.044 INFO:teuthology.orchestra.run.smithi099.stdout: "info": { 2023-05-31T20:13:18.044 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "sync", 2023-05-31T20:13:18.044 INFO:teuthology.orchestra.run.smithi099.stdout: "num_shards": 4, 2023-05-31T20:13:18.045 INFO:teuthology.orchestra.run.smithi099.stdout: "period": "2e6123d2-d695-4350-9a01-61a455bc782b", 2023-05-31T20:13:18.045 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2023-05-31T20:13:18.045 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:13:18.046 INFO:teuthology.orchestra.run.smithi099.stdout: "markers": [ 2023-05-31T20:13:18.046 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T20:13:18.046 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 0, 2023-05-31T20:13:18.046 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T20:13:18.046 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2023-05-31T20:13:18.046 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1685562631.516089_59.1", 2023-05-31T20:13:18.047 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T20:13:18.047 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 1, 2023-05-31T20:13:18.047 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T20:13:18.047 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2023-05-31T19:50:31.516089Z", 2023-05-31T20:13:18.047 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2023-05-31T20:13:18.047 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T20:13:18.047 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:13:18.048 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T20:13:18.048 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 1, 2023-05-31T20:13:18.048 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T20:13:18.048 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2023-05-31T20:13:18.048 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "", 2023-05-31T20:13:18.048 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T20:13:18.049 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2023-05-31T20:13:18.049 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T20:13:18.049 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "0.000000", 2023-05-31T20:13:18.049 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2023-05-31T20:13:18.049 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T20:13:18.049 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:13:18.049 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T20:13:18.050 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 2, 2023-05-31T20:13:18.050 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T20:13:18.050 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2023-05-31T20:13:18.050 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1685562970.274572_93.1", 2023-05-31T20:13:18.050 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T20:13:18.052 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2023-05-31T20:13:18.052 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T20:13:18.052 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2023-05-31T19:56:10.274572Z", 2023-05-31T20:13:18.053 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2023-05-31T20:13:18.053 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T20:13:18.053 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:13:18.053 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T20:13:18.053 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 3, 2023-05-31T20:13:18.053 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T20:13:18.054 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2023-05-31T20:13:18.054 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1685563799.058314_262.1", 2023-05-31T20:13:18.054 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T20:13:18.054 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2023-05-31T20:13:18.054 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T20:13:18.054 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2023-05-31T20:09:59.058314Z", 2023-05-31T20:13:18.055 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2023-05-31T20:13:18.055 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T20:13:18.055 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T20:13:18.055 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2023-05-31T20:13:18.055 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:13:18.056 INFO:teuthology.orchestra.run.smithi099.stdout: "full_sync": { 2023-05-31T20:13:18.056 INFO:teuthology.orchestra.run.smithi099.stdout: "total": 1, 2023-05-31T20:13:18.056 INFO:teuthology.orchestra.run.smithi099.stdout: "complete": 1 2023-05-31T20:13:18.056 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:13:18.056 INFO:teuthology.orchestra.run.smithi099.stdout: "current_time": "2023-05-31T20:13:18Z" 2023-05-31T20:13:18.056 INFO:teuthology.orchestra.run.smithi099.stdout:} 2023-05-31T20:13:18.057 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "2e6123d2-d695-4350-9a01-61a455bc782b", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1685562631.516089_59.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-05-31T19:50:31.516089Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1685562970.274572_93.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-05-31T19:56:10.274572Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1685563799.058314_262.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-05-31T20:09:59.058314Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-05-31T20:13:18Z" } 2023-05-31T20:13:18.058 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1685562631.516089_59.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-05-31T19:50:31.516089Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1685562970.274572_93.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-05-31T19:56:10.274572Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1685563799.058314_262.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-05-31T20:09:59.058314Z', 'realm_epoch': 2}}] 2023-05-31T20:13:18.059 DEBUG:rgw_multi.tests:log_status={0: '1_1685562631.516089_59.1', 1: '', 2: '1_1685562970.274572_93.1', 3: '1_1685563802.468126_296.1'} 2023-05-31T20:13:18.060 DEBUG:rgw_multi.tests:sync_status={0: '1_1685562631.516089_59.1', 1: '', 2: '1_1685562970.274572_93.1', 3: '1_1685563799.058314_262.1'} 2023-05-31T20:13:18.060 WARNING:rgw_multi.tests:zone a2 behind master: shard=3 master=1_1685563802.468126_296.1 target=1_1685563799.058314_262.1 2023-05-31T20:13:23.061 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-31T20:13:23.062 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-31T20:13:23.063 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-05-31T20:13:23.112 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2023-05-31T20:13:23.112 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2023-05-31T20:13:23.145 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:13:23.140+0000 7f93f83a3780 20 RGW-SYNC:meta: read sync status 2023-05-31T20:13:23.146 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:13:23.141+0000 7f93f83a3780 20 RGW-SYNC:meta: read sync status 2023-05-31T20:13:23.158 INFO:teuthology.orchestra.run.smithi099.stdout:{ 2023-05-31T20:13:23.158 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_status": { 2023-05-31T20:13:23.158 INFO:teuthology.orchestra.run.smithi099.stdout: "info": { 2023-05-31T20:13:23.159 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "sync", 2023-05-31T20:13:23.159 INFO:teuthology.orchestra.run.smithi099.stdout: "num_shards": 4, 2023-05-31T20:13:23.159 INFO:teuthology.orchestra.run.smithi099.stdout: "period": "2e6123d2-d695-4350-9a01-61a455bc782b", 2023-05-31T20:13:23.159 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2023-05-31T20:13:23.159 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:13:23.159 INFO:teuthology.orchestra.run.smithi099.stdout: "markers": [ 2023-05-31T20:13:23.160 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T20:13:23.160 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 0, 2023-05-31T20:13:23.160 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T20:13:23.160 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2023-05-31T20:13:23.160 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1685562631.516089_59.1", 2023-05-31T20:13:23.160 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T20:13:23.160 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 1, 2023-05-31T20:13:23.161 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T20:13:23.161 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2023-05-31T19:50:31.516089Z", 2023-05-31T20:13:23.161 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2023-05-31T20:13:23.161 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T20:13:23.161 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:13:23.161 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T20:13:23.161 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 1, 2023-05-31T20:13:23.162 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T20:13:23.162 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2023-05-31T20:13:23.162 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "", 2023-05-31T20:13:23.162 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T20:13:23.162 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2023-05-31T20:13:23.162 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T20:13:23.162 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "0.000000", 2023-05-31T20:13:23.163 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2023-05-31T20:13:23.163 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T20:13:23.163 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:13:23.163 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T20:13:23.163 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 2, 2023-05-31T20:13:23.163 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T20:13:23.163 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2023-05-31T20:13:23.164 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1685562970.274572_93.1", 2023-05-31T20:13:23.164 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T20:13:23.164 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2023-05-31T20:13:23.164 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T20:13:23.164 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2023-05-31T19:56:10.274572Z", 2023-05-31T20:13:23.164 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2023-05-31T20:13:23.165 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T20:13:23.166 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:13:23.166 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T20:13:23.166 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 3, 2023-05-31T20:13:23.167 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T20:13:23.167 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2023-05-31T20:13:23.167 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1685563798.772373_248.1", 2023-05-31T20:13:23.168 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T20:13:23.168 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2023-05-31T20:13:23.168 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T20:13:23.168 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2023-05-31T20:09:58.772373Z", 2023-05-31T20:13:23.168 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2023-05-31T20:13:23.168 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T20:13:23.169 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T20:13:23.169 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2023-05-31T20:13:23.169 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:13:23.169 INFO:teuthology.orchestra.run.smithi099.stdout: "full_sync": { 2023-05-31T20:13:23.169 INFO:teuthology.orchestra.run.smithi099.stdout: "total": 1, 2023-05-31T20:13:23.169 INFO:teuthology.orchestra.run.smithi099.stdout: "complete": 1 2023-05-31T20:13:23.169 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:13:23.170 INFO:teuthology.orchestra.run.smithi099.stdout: "current_time": "2023-05-31T20:13:23Z" 2023-05-31T20:13:23.170 INFO:teuthology.orchestra.run.smithi099.stdout:} 2023-05-31T20:13:23.171 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "2e6123d2-d695-4350-9a01-61a455bc782b", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1685562631.516089_59.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-05-31T19:50:31.516089Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1685562970.274572_93.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-05-31T19:56:10.274572Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1685563798.772373_248.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-05-31T20:09:58.772373Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-05-31T20:13:23Z" } 2023-05-31T20:13:23.172 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1685562631.516089_59.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-05-31T19:50:31.516089Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1685562970.274572_93.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-05-31T19:56:10.274572Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1685563798.772373_248.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-05-31T20:09:58.772373Z', 'realm_epoch': 2}}] 2023-05-31T20:13:23.172 DEBUG:rgw_multi.tests:log_status={0: '1_1685562631.516089_59.1', 1: '', 2: '1_1685562970.274572_93.1', 3: '1_1685563802.468126_296.1'} 2023-05-31T20:13:23.172 DEBUG:rgw_multi.tests:sync_status={0: '1_1685562631.516089_59.1', 1: '', 2: '1_1685562970.274572_93.1', 3: '1_1685563798.772373_248.1'} 2023-05-31T20:13:23.173 WARNING:rgw_multi.tests:zone a2 behind master: shard=3 master=1_1685563802.468126_296.1 target=1_1685563798.772373_248.1 2023-05-31T20:13:28.173 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-31T20:13:28.175 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-31T20:13:28.175 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-05-31T20:13:28.228 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2023-05-31T20:13:28.229 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2023-05-31T20:13:28.258 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:13:28.252+0000 7fe5bf30e780 20 RGW-SYNC:meta: read sync status 2023-05-31T20:13:28.259 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:13:28.253+0000 7fe5bf30e780 20 RGW-SYNC:meta: read sync status 2023-05-31T20:13:28.271 INFO:teuthology.orchestra.run.smithi099.stdout:{ 2023-05-31T20:13:28.271 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_status": { 2023-05-31T20:13:28.272 INFO:teuthology.orchestra.run.smithi099.stdout: "info": { 2023-05-31T20:13:28.272 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "sync", 2023-05-31T20:13:28.272 INFO:teuthology.orchestra.run.smithi099.stdout: "num_shards": 4, 2023-05-31T20:13:28.272 INFO:teuthology.orchestra.run.smithi099.stdout: "period": "2e6123d2-d695-4350-9a01-61a455bc782b", 2023-05-31T20:13:28.272 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2023-05-31T20:13:28.272 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:13:28.273 INFO:teuthology.orchestra.run.smithi099.stdout: "markers": [ 2023-05-31T20:13:28.273 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T20:13:28.273 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 0, 2023-05-31T20:13:28.273 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T20:13:28.273 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2023-05-31T20:13:28.273 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1685562631.516089_59.1", 2023-05-31T20:13:28.274 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T20:13:28.274 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 1, 2023-05-31T20:13:28.274 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T20:13:28.274 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2023-05-31T19:50:31.516089Z", 2023-05-31T20:13:28.274 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2023-05-31T20:13:28.274 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T20:13:28.274 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:13:28.275 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T20:13:28.275 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 1, 2023-05-31T20:13:28.275 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T20:13:28.275 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2023-05-31T20:13:28.275 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "", 2023-05-31T20:13:28.275 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T20:13:28.275 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2023-05-31T20:13:28.276 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T20:13:28.276 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "0.000000", 2023-05-31T20:13:28.276 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2023-05-31T20:13:28.276 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T20:13:28.276 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:13:28.276 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T20:13:28.277 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 2, 2023-05-31T20:13:28.277 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T20:13:28.277 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2023-05-31T20:13:28.277 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1685562970.274572_93.1", 2023-05-31T20:13:28.277 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T20:13:28.277 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2023-05-31T20:13:28.278 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T20:13:28.278 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2023-05-31T19:56:10.274572Z", 2023-05-31T20:13:28.278 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2023-05-31T20:13:28.278 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T20:13:28.278 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:13:28.278 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T20:13:28.278 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 3, 2023-05-31T20:13:28.279 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T20:13:28.279 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2023-05-31T20:13:28.279 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1685563798.817516_251.1", 2023-05-31T20:13:28.279 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T20:13:28.279 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2023-05-31T20:13:28.279 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T20:13:28.279 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2023-05-31T20:09:58.817516Z", 2023-05-31T20:13:28.280 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2023-05-31T20:13:28.280 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T20:13:28.280 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T20:13:28.280 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2023-05-31T20:13:28.280 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:13:28.280 INFO:teuthology.orchestra.run.smithi099.stdout: "full_sync": { 2023-05-31T20:13:28.280 INFO:teuthology.orchestra.run.smithi099.stdout: "total": 1, 2023-05-31T20:13:28.281 INFO:teuthology.orchestra.run.smithi099.stdout: "complete": 1 2023-05-31T20:13:28.281 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:13:28.281 INFO:teuthology.orchestra.run.smithi099.stdout: "current_time": "2023-05-31T20:13:28Z" 2023-05-31T20:13:28.281 INFO:teuthology.orchestra.run.smithi099.stdout:} 2023-05-31T20:13:28.282 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "2e6123d2-d695-4350-9a01-61a455bc782b", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1685562631.516089_59.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-05-31T19:50:31.516089Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1685562970.274572_93.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-05-31T19:56:10.274572Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1685563798.817516_251.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-05-31T20:09:58.817516Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-05-31T20:13:28Z" } 2023-05-31T20:13:28.282 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1685562631.516089_59.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-05-31T19:50:31.516089Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1685562970.274572_93.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-05-31T19:56:10.274572Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1685563798.817516_251.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-05-31T20:09:58.817516Z', 'realm_epoch': 2}}] 2023-05-31T20:13:28.283 DEBUG:rgw_multi.tests:log_status={0: '1_1685562631.516089_59.1', 1: '', 2: '1_1685562970.274572_93.1', 3: '1_1685563802.468126_296.1'} 2023-05-31T20:13:28.283 DEBUG:rgw_multi.tests:sync_status={0: '1_1685562631.516089_59.1', 1: '', 2: '1_1685562970.274572_93.1', 3: '1_1685563798.817516_251.1'} 2023-05-31T20:13:28.283 WARNING:rgw_multi.tests:zone a2 behind master: shard=3 master=1_1685563802.468126_296.1 target=1_1685563798.817516_251.1 2023-05-31T20:13:33.285 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-31T20:13:33.285 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-31T20:13:33.286 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-05-31T20:13:33.338 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2023-05-31T20:13:33.338 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2023-05-31T20:13:33.369 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:13:33.363+0000 7f06758be780 20 RGW-SYNC:meta: read sync status 2023-05-31T20:13:33.369 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:13:33.364+0000 7f06758be780 20 RGW-SYNC:meta: read sync status 2023-05-31T20:13:33.396 INFO:teuthology.orchestra.run.smithi099.stdout:{ 2023-05-31T20:13:33.397 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_status": { 2023-05-31T20:13:33.397 INFO:teuthology.orchestra.run.smithi099.stdout: "info": { 2023-05-31T20:13:33.397 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "sync", 2023-05-31T20:13:33.397 INFO:teuthology.orchestra.run.smithi099.stdout: "num_shards": 4, 2023-05-31T20:13:33.397 INFO:teuthology.orchestra.run.smithi099.stdout: "period": "2e6123d2-d695-4350-9a01-61a455bc782b", 2023-05-31T20:13:33.397 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2023-05-31T20:13:33.398 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:13:33.398 INFO:teuthology.orchestra.run.smithi099.stdout: "markers": [ 2023-05-31T20:13:33.398 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T20:13:33.398 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 0, 2023-05-31T20:13:33.398 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T20:13:33.398 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2023-05-31T20:13:33.398 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1685562631.516089_59.1", 2023-05-31T20:13:33.399 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T20:13:33.399 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 1, 2023-05-31T20:13:33.399 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T20:13:33.400 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2023-05-31T19:50:31.516089Z", 2023-05-31T20:13:33.400 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2023-05-31T20:13:33.400 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T20:13:33.400 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:13:33.400 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T20:13:33.400 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 1, 2023-05-31T20:13:33.400 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T20:13:33.401 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2023-05-31T20:13:33.401 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "", 2023-05-31T20:13:33.401 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T20:13:33.401 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2023-05-31T20:13:33.402 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T20:13:33.402 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "0.000000", 2023-05-31T20:13:33.402 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2023-05-31T20:13:33.402 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T20:13:33.402 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:13:33.402 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T20:13:33.403 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 2, 2023-05-31T20:13:33.404 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T20:13:33.404 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2023-05-31T20:13:33.404 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1685562970.274572_93.1", 2023-05-31T20:13:33.404 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T20:13:33.404 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2023-05-31T20:13:33.405 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T20:13:33.405 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2023-05-31T19:56:10.274572Z", 2023-05-31T20:13:33.405 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2023-05-31T20:13:33.405 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T20:13:33.405 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:13:33.405 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T20:13:33.405 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 3, 2023-05-31T20:13:33.406 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T20:13:33.406 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2023-05-31T20:13:33.406 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1685563798.970303_257.1", 2023-05-31T20:13:33.406 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T20:13:33.406 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2023-05-31T20:13:33.406 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T20:13:33.407 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2023-05-31T20:09:58.970303Z", 2023-05-31T20:13:33.407 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2023-05-31T20:13:33.407 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T20:13:33.407 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T20:13:33.407 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2023-05-31T20:13:33.407 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:13:33.407 INFO:teuthology.orchestra.run.smithi099.stdout: "full_sync": { 2023-05-31T20:13:33.408 INFO:teuthology.orchestra.run.smithi099.stdout: "total": 1, 2023-05-31T20:13:33.408 INFO:teuthology.orchestra.run.smithi099.stdout: "complete": 1 2023-05-31T20:13:33.408 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:13:33.408 INFO:teuthology.orchestra.run.smithi099.stdout: "current_time": "2023-05-31T20:13:33Z" 2023-05-31T20:13:33.408 INFO:teuthology.orchestra.run.smithi099.stdout:} 2023-05-31T20:13:33.409 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "2e6123d2-d695-4350-9a01-61a455bc782b", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1685562631.516089_59.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-05-31T19:50:31.516089Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1685562970.274572_93.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-05-31T19:56:10.274572Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1685563798.970303_257.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-05-31T20:09:58.970303Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-05-31T20:13:33Z" } 2023-05-31T20:13:33.410 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1685562631.516089_59.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-05-31T19:50:31.516089Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1685562970.274572_93.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-05-31T19:56:10.274572Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1685563798.970303_257.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-05-31T20:09:58.970303Z', 'realm_epoch': 2}}] 2023-05-31T20:13:33.410 DEBUG:rgw_multi.tests:log_status={0: '1_1685562631.516089_59.1', 1: '', 2: '1_1685562970.274572_93.1', 3: '1_1685563802.468126_296.1'} 2023-05-31T20:13:33.411 DEBUG:rgw_multi.tests:sync_status={0: '1_1685562631.516089_59.1', 1: '', 2: '1_1685562970.274572_93.1', 3: '1_1685563798.970303_257.1'} 2023-05-31T20:13:33.411 WARNING:rgw_multi.tests:zone a2 behind master: shard=3 master=1_1685563802.468126_296.1 target=1_1685563798.970303_257.1 2023-05-31T20:13:38.413 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-31T20:13:38.413 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-31T20:13:38.414 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-05-31T20:13:38.483 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2023-05-31T20:13:38.484 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2023-05-31T20:13:38.538 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:13:38.532+0000 7fb7258f9780 20 RGW-SYNC:meta: read sync status 2023-05-31T20:13:38.538 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:13:38.532+0000 7fb7258f9780 20 RGW-SYNC:meta: read sync status 2023-05-31T20:13:38.552 INFO:teuthology.orchestra.run.smithi099.stdout:{ 2023-05-31T20:13:38.553 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_status": { 2023-05-31T20:13:38.553 INFO:teuthology.orchestra.run.smithi099.stdout: "info": { 2023-05-31T20:13:38.553 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "sync", 2023-05-31T20:13:38.554 INFO:teuthology.orchestra.run.smithi099.stdout: "num_shards": 4, 2023-05-31T20:13:38.554 INFO:teuthology.orchestra.run.smithi099.stdout: "period": "2e6123d2-d695-4350-9a01-61a455bc782b", 2023-05-31T20:13:38.554 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2023-05-31T20:13:38.554 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:13:38.554 INFO:teuthology.orchestra.run.smithi099.stdout: "markers": [ 2023-05-31T20:13:38.555 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T20:13:38.555 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 0, 2023-05-31T20:13:38.555 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T20:13:38.556 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2023-05-31T20:13:38.556 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1685562631.516089_59.1", 2023-05-31T20:13:38.556 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T20:13:38.557 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 1, 2023-05-31T20:13:38.557 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T20:13:38.558 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2023-05-31T19:50:31.516089Z", 2023-05-31T20:13:38.558 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2023-05-31T20:13:38.559 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T20:13:38.563 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:13:38.563 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T20:13:38.563 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 1, 2023-05-31T20:13:38.564 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T20:13:38.564 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2023-05-31T20:13:38.564 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "", 2023-05-31T20:13:38.564 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T20:13:38.564 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2023-05-31T20:13:38.564 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T20:13:38.565 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "0.000000", 2023-05-31T20:13:38.565 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2023-05-31T20:13:38.565 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T20:13:38.565 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:13:38.566 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T20:13:38.566 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 2, 2023-05-31T20:13:38.566 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T20:13:38.566 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2023-05-31T20:13:38.566 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1685562970.274572_93.1", 2023-05-31T20:13:38.567 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T20:13:38.567 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2023-05-31T20:13:38.567 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T20:13:38.567 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2023-05-31T19:56:10.274572Z", 2023-05-31T20:13:38.567 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2023-05-31T20:13:38.567 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T20:13:38.568 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:13:38.568 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T20:13:38.568 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 3, 2023-05-31T20:13:38.568 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T20:13:38.568 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2023-05-31T20:13:38.568 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1685563798.885419_253.1", 2023-05-31T20:13:38.569 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T20:13:38.569 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2023-05-31T20:13:38.569 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T20:13:38.569 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2023-05-31T20:09:58.885419Z", 2023-05-31T20:13:38.569 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2023-05-31T20:13:38.569 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T20:13:38.570 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T20:13:38.570 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2023-05-31T20:13:38.570 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:13:38.570 INFO:teuthology.orchestra.run.smithi099.stdout: "full_sync": { 2023-05-31T20:13:38.570 INFO:teuthology.orchestra.run.smithi099.stdout: "total": 1, 2023-05-31T20:13:38.570 INFO:teuthology.orchestra.run.smithi099.stdout: "complete": 1 2023-05-31T20:13:38.570 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:13:38.571 INFO:teuthology.orchestra.run.smithi099.stdout: "current_time": "2023-05-31T20:13:38Z" 2023-05-31T20:13:38.571 INFO:teuthology.orchestra.run.smithi099.stdout:} 2023-05-31T20:13:38.572 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "2e6123d2-d695-4350-9a01-61a455bc782b", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1685562631.516089_59.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-05-31T19:50:31.516089Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1685562970.274572_93.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-05-31T19:56:10.274572Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1685563798.885419_253.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-05-31T20:09:58.885419Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-05-31T20:13:38Z" } 2023-05-31T20:13:38.573 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1685562631.516089_59.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-05-31T19:50:31.516089Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1685562970.274572_93.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-05-31T19:56:10.274572Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1685563798.885419_253.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-05-31T20:09:58.885419Z', 'realm_epoch': 2}}] 2023-05-31T20:13:38.574 DEBUG:rgw_multi.tests:log_status={0: '1_1685562631.516089_59.1', 1: '', 2: '1_1685562970.274572_93.1', 3: '1_1685563802.468126_296.1'} 2023-05-31T20:13:38.574 DEBUG:rgw_multi.tests:sync_status={0: '1_1685562631.516089_59.1', 1: '', 2: '1_1685562970.274572_93.1', 3: '1_1685563798.885419_253.1'} 2023-05-31T20:13:38.574 WARNING:rgw_multi.tests:zone a2 behind master: shard=3 master=1_1685563802.468126_296.1 target=1_1685563798.885419_253.1 2023-05-31T20:13:43.576 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-31T20:13:43.576 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-31T20:13:43.577 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-05-31T20:13:43.628 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2023-05-31T20:13:43.629 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2023-05-31T20:13:43.681 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:13:43.676+0000 7fd7735c1780 20 RGW-SYNC:meta: read sync status 2023-05-31T20:13:43.682 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:13:43.676+0000 7fd7735c1780 20 RGW-SYNC:meta: read sync status 2023-05-31T20:13:43.695 INFO:teuthology.orchestra.run.smithi099.stdout:{ 2023-05-31T20:13:43.695 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_status": { 2023-05-31T20:13:43.695 INFO:teuthology.orchestra.run.smithi099.stdout: "info": { 2023-05-31T20:13:43.695 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "sync", 2023-05-31T20:13:43.696 INFO:teuthology.orchestra.run.smithi099.stdout: "num_shards": 4, 2023-05-31T20:13:43.696 INFO:teuthology.orchestra.run.smithi099.stdout: "period": "2e6123d2-d695-4350-9a01-61a455bc782b", 2023-05-31T20:13:43.696 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2023-05-31T20:13:43.696 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:13:43.696 INFO:teuthology.orchestra.run.smithi099.stdout: "markers": [ 2023-05-31T20:13:43.697 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T20:13:43.697 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 0, 2023-05-31T20:13:43.697 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T20:13:43.697 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2023-05-31T20:13:43.697 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1685562631.516089_59.1", 2023-05-31T20:13:43.697 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T20:13:43.698 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 1, 2023-05-31T20:13:43.698 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T20:13:43.698 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2023-05-31T19:50:31.516089Z", 2023-05-31T20:13:43.698 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2023-05-31T20:13:43.698 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T20:13:43.698 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:13:43.698 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T20:13:43.699 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 1, 2023-05-31T20:13:43.699 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T20:13:43.699 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2023-05-31T20:13:43.699 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "", 2023-05-31T20:13:43.699 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T20:13:43.699 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2023-05-31T20:13:43.699 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T20:13:43.700 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "0.000000", 2023-05-31T20:13:43.700 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2023-05-31T20:13:43.700 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T20:13:43.700 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:13:43.700 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T20:13:43.700 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 2, 2023-05-31T20:13:43.701 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T20:13:43.701 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2023-05-31T20:13:43.701 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1685562970.274572_93.1", 2023-05-31T20:13:43.701 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T20:13:43.701 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2023-05-31T20:13:43.702 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T20:13:43.702 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2023-05-31T19:56:10.274572Z", 2023-05-31T20:13:43.702 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2023-05-31T20:13:43.702 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T20:13:43.702 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:13:43.703 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T20:13:43.703 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 3, 2023-05-31T20:13:43.703 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T20:13:43.703 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2023-05-31T20:13:43.703 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1685563798.807987_249.1", 2023-05-31T20:13:43.703 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T20:13:43.704 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2023-05-31T20:13:43.704 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T20:13:43.705 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2023-05-31T20:09:58.807987Z", 2023-05-31T20:13:43.705 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2023-05-31T20:13:43.706 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T20:13:43.706 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T20:13:43.706 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2023-05-31T20:13:43.707 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:13:43.707 INFO:teuthology.orchestra.run.smithi099.stdout: "full_sync": { 2023-05-31T20:13:43.708 INFO:teuthology.orchestra.run.smithi099.stdout: "total": 1, 2023-05-31T20:13:43.708 INFO:teuthology.orchestra.run.smithi099.stdout: "complete": 1 2023-05-31T20:13:43.708 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:13:43.708 INFO:teuthology.orchestra.run.smithi099.stdout: "current_time": "2023-05-31T20:13:43Z" 2023-05-31T20:13:43.708 INFO:teuthology.orchestra.run.smithi099.stdout:} 2023-05-31T20:13:43.710 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "2e6123d2-d695-4350-9a01-61a455bc782b", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1685562631.516089_59.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-05-31T19:50:31.516089Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1685562970.274572_93.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-05-31T19:56:10.274572Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1685563798.807987_249.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-05-31T20:09:58.807987Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-05-31T20:13:43Z" } 2023-05-31T20:13:43.711 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1685562631.516089_59.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-05-31T19:50:31.516089Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1685562970.274572_93.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-05-31T19:56:10.274572Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1685563798.807987_249.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-05-31T20:09:58.807987Z', 'realm_epoch': 2}}] 2023-05-31T20:13:43.711 DEBUG:rgw_multi.tests:log_status={0: '1_1685562631.516089_59.1', 1: '', 2: '1_1685562970.274572_93.1', 3: '1_1685563802.468126_296.1'} 2023-05-31T20:13:43.711 DEBUG:rgw_multi.tests:sync_status={0: '1_1685562631.516089_59.1', 1: '', 2: '1_1685562970.274572_93.1', 3: '1_1685563798.807987_249.1'} 2023-05-31T20:13:43.712 WARNING:rgw_multi.tests:zone a2 behind master: shard=3 master=1_1685563802.468126_296.1 target=1_1685563798.807987_249.1 2023-05-31T20:13:48.716 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-31T20:13:48.716 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-31T20:13:48.717 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-05-31T20:13:48.796 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2023-05-31T20:13:48.796 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2023-05-31T20:13:48.857 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:13:48.851+0000 7f0917f83780 20 RGW-SYNC:meta: read sync status 2023-05-31T20:13:48.858 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:13:48.853+0000 7f0917f83780 20 RGW-SYNC:meta: read sync status 2023-05-31T20:13:48.879 INFO:teuthology.orchestra.run.smithi099.stdout:{ 2023-05-31T20:13:48.879 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_status": { 2023-05-31T20:13:48.880 INFO:teuthology.orchestra.run.smithi099.stdout: "info": { 2023-05-31T20:13:48.880 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "sync", 2023-05-31T20:13:48.880 INFO:teuthology.orchestra.run.smithi099.stdout: "num_shards": 4, 2023-05-31T20:13:48.880 INFO:teuthology.orchestra.run.smithi099.stdout: "period": "2e6123d2-d695-4350-9a01-61a455bc782b", 2023-05-31T20:13:48.880 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2023-05-31T20:13:48.881 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:13:48.881 INFO:teuthology.orchestra.run.smithi099.stdout: "markers": [ 2023-05-31T20:13:48.881 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T20:13:48.881 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 0, 2023-05-31T20:13:48.881 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T20:13:48.881 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2023-05-31T20:13:48.881 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1685562631.516089_59.1", 2023-05-31T20:13:48.882 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T20:13:48.882 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 1, 2023-05-31T20:13:48.882 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T20:13:48.882 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2023-05-31T19:50:31.516089Z", 2023-05-31T20:13:48.882 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2023-05-31T20:13:48.882 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T20:13:48.882 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:13:48.883 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T20:13:48.883 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 1, 2023-05-31T20:13:48.883 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T20:13:48.883 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2023-05-31T20:13:48.883 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "", 2023-05-31T20:13:48.883 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T20:13:48.883 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2023-05-31T20:13:48.884 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T20:13:48.884 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "0.000000", 2023-05-31T20:13:48.884 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2023-05-31T20:13:48.884 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T20:13:48.884 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:13:48.884 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T20:13:48.884 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 2, 2023-05-31T20:13:48.885 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T20:13:48.885 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2023-05-31T20:13:48.885 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1685562970.274572_93.1", 2023-05-31T20:13:48.885 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T20:13:48.885 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2023-05-31T20:13:48.885 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T20:13:48.885 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2023-05-31T19:56:10.274572Z", 2023-05-31T20:13:48.886 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2023-05-31T20:13:48.886 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T20:13:48.886 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:13:48.886 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T20:13:48.886 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 3, 2023-05-31T20:13:48.886 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T20:13:48.886 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2023-05-31T20:13:48.887 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1685563798.807987_249.1", 2023-05-31T20:13:48.887 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T20:13:48.887 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2023-05-31T20:13:48.887 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T20:13:48.887 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2023-05-31T20:09:58.807987Z", 2023-05-31T20:13:48.887 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2023-05-31T20:13:48.888 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T20:13:48.888 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T20:13:48.888 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2023-05-31T20:13:48.888 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:13:48.889 INFO:teuthology.orchestra.run.smithi099.stdout: "full_sync": { 2023-05-31T20:13:48.889 INFO:teuthology.orchestra.run.smithi099.stdout: "total": 1, 2023-05-31T20:13:48.889 INFO:teuthology.orchestra.run.smithi099.stdout: "complete": 1 2023-05-31T20:13:48.889 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:13:48.889 INFO:teuthology.orchestra.run.smithi099.stdout: "current_time": "2023-05-31T20:13:48Z" 2023-05-31T20:13:48.889 INFO:teuthology.orchestra.run.smithi099.stdout:} 2023-05-31T20:13:48.890 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "2e6123d2-d695-4350-9a01-61a455bc782b", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1685562631.516089_59.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-05-31T19:50:31.516089Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1685562970.274572_93.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-05-31T19:56:10.274572Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1685563798.807987_249.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-05-31T20:09:58.807987Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-05-31T20:13:48Z" } 2023-05-31T20:13:48.891 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1685562631.516089_59.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-05-31T19:50:31.516089Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1685562970.274572_93.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-05-31T19:56:10.274572Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1685563798.807987_249.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-05-31T20:09:58.807987Z', 'realm_epoch': 2}}] 2023-05-31T20:13:48.891 DEBUG:rgw_multi.tests:log_status={0: '1_1685562631.516089_59.1', 1: '', 2: '1_1685562970.274572_93.1', 3: '1_1685563802.468126_296.1'} 2023-05-31T20:13:48.892 DEBUG:rgw_multi.tests:sync_status={0: '1_1685562631.516089_59.1', 1: '', 2: '1_1685562970.274572_93.1', 3: '1_1685563798.807987_249.1'} 2023-05-31T20:13:48.892 WARNING:rgw_multi.tests:zone a2 behind master: shard=3 master=1_1685563802.468126_296.1 target=1_1685563798.807987_249.1 2023-05-31T20:13:53.893 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-31T20:13:53.893 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-31T20:13:53.894 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-05-31T20:13:53.970 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2023-05-31T20:13:53.971 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2023-05-31T20:13:54.020 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:13:54.016+0000 7fa090498780 20 RGW-SYNC:meta: read sync status 2023-05-31T20:13:54.022 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:13:54.017+0000 7fa090498780 20 RGW-SYNC:meta: read sync status 2023-05-31T20:13:54.048 INFO:teuthology.orchestra.run.smithi099.stdout:{ 2023-05-31T20:13:54.049 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_status": { 2023-05-31T20:13:54.049 INFO:teuthology.orchestra.run.smithi099.stdout: "info": { 2023-05-31T20:13:54.049 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "sync", 2023-05-31T20:13:54.050 INFO:teuthology.orchestra.run.smithi099.stdout: "num_shards": 4, 2023-05-31T20:13:54.050 INFO:teuthology.orchestra.run.smithi099.stdout: "period": "2e6123d2-d695-4350-9a01-61a455bc782b", 2023-05-31T20:13:54.050 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2023-05-31T20:13:54.051 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:13:54.051 INFO:teuthology.orchestra.run.smithi099.stdout: "markers": [ 2023-05-31T20:13:54.051 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T20:13:54.051 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 0, 2023-05-31T20:13:54.051 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T20:13:54.051 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2023-05-31T20:13:54.052 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1685562631.516089_59.1", 2023-05-31T20:13:54.052 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T20:13:54.052 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 1, 2023-05-31T20:13:54.052 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T20:13:54.052 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2023-05-31T19:50:31.516089Z", 2023-05-31T20:13:54.053 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2023-05-31T20:13:54.053 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T20:13:54.053 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:13:54.053 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T20:13:54.053 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 1, 2023-05-31T20:13:54.053 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T20:13:54.053 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2023-05-31T20:13:54.054 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "", 2023-05-31T20:13:54.054 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T20:13:54.054 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2023-05-31T20:13:54.054 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T20:13:54.055 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "0.000000", 2023-05-31T20:13:54.055 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2023-05-31T20:13:54.055 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T20:13:54.055 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:13:54.055 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T20:13:54.055 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 2, 2023-05-31T20:13:54.055 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T20:13:54.056 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2023-05-31T20:13:54.056 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1685562970.274572_93.1", 2023-05-31T20:13:54.056 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T20:13:54.056 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2023-05-31T20:13:54.056 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T20:13:54.056 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2023-05-31T19:56:10.274572Z", 2023-05-31T20:13:54.057 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2023-05-31T20:13:54.057 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T20:13:54.057 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:13:54.057 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T20:13:54.057 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 3, 2023-05-31T20:13:54.057 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T20:13:54.057 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2023-05-31T20:13:54.058 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1685563798.970303_257.1", 2023-05-31T20:13:54.058 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T20:13:54.058 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2023-05-31T20:13:54.058 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T20:13:54.058 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2023-05-31T20:09:58.970303Z", 2023-05-31T20:13:54.058 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2023-05-31T20:13:54.061 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T20:13:54.061 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T20:13:54.061 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2023-05-31T20:13:54.062 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:13:54.062 INFO:teuthology.orchestra.run.smithi099.stdout: "full_sync": { 2023-05-31T20:13:54.062 INFO:teuthology.orchestra.run.smithi099.stdout: "total": 1, 2023-05-31T20:13:54.062 INFO:teuthology.orchestra.run.smithi099.stdout: "complete": 1 2023-05-31T20:13:54.062 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:13:54.062 INFO:teuthology.orchestra.run.smithi099.stdout: "current_time": "2023-05-31T20:13:54Z" 2023-05-31T20:13:54.062 INFO:teuthology.orchestra.run.smithi099.stdout:} 2023-05-31T20:13:54.063 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "2e6123d2-d695-4350-9a01-61a455bc782b", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1685562631.516089_59.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-05-31T19:50:31.516089Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1685562970.274572_93.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-05-31T19:56:10.274572Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1685563798.970303_257.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-05-31T20:09:58.970303Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-05-31T20:13:54Z" } 2023-05-31T20:13:54.064 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1685562631.516089_59.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-05-31T19:50:31.516089Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1685562970.274572_93.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-05-31T19:56:10.274572Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1685563798.970303_257.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-05-31T20:09:58.970303Z', 'realm_epoch': 2}}] 2023-05-31T20:13:54.065 DEBUG:rgw_multi.tests:log_status={0: '1_1685562631.516089_59.1', 1: '', 2: '1_1685562970.274572_93.1', 3: '1_1685563802.468126_296.1'} 2023-05-31T20:13:54.065 DEBUG:rgw_multi.tests:sync_status={0: '1_1685562631.516089_59.1', 1: '', 2: '1_1685562970.274572_93.1', 3: '1_1685563798.970303_257.1'} 2023-05-31T20:13:54.065 WARNING:rgw_multi.tests:zone a2 behind master: shard=3 master=1_1685563802.468126_296.1 target=1_1685563798.970303_257.1 2023-05-31T20:13:59.066 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-31T20:13:59.067 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-31T20:13:59.067 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-05-31T20:13:59.118 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2023-05-31T20:13:59.119 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2023-05-31T20:13:59.165 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:13:59.155+0000 7f22fff34780 20 RGW-SYNC:meta: read sync status 2023-05-31T20:13:59.166 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:13:59.156+0000 7f22fff34780 20 RGW-SYNC:meta: read sync status 2023-05-31T20:13:59.180 INFO:teuthology.orchestra.run.smithi099.stdout:{ 2023-05-31T20:13:59.180 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_status": { 2023-05-31T20:13:59.180 INFO:teuthology.orchestra.run.smithi099.stdout: "info": { 2023-05-31T20:13:59.181 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "sync", 2023-05-31T20:13:59.181 INFO:teuthology.orchestra.run.smithi099.stdout: "num_shards": 4, 2023-05-31T20:13:59.181 INFO:teuthology.orchestra.run.smithi099.stdout: "period": "2e6123d2-d695-4350-9a01-61a455bc782b", 2023-05-31T20:13:59.181 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2023-05-31T20:13:59.181 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:13:59.181 INFO:teuthology.orchestra.run.smithi099.stdout: "markers": [ 2023-05-31T20:13:59.182 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T20:13:59.182 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 0, 2023-05-31T20:13:59.182 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T20:13:59.182 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2023-05-31T20:13:59.182 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1685562631.516089_59.1", 2023-05-31T20:13:59.182 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T20:13:59.182 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 1, 2023-05-31T20:13:59.183 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T20:13:59.183 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2023-05-31T19:50:31.516089Z", 2023-05-31T20:13:59.183 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2023-05-31T20:13:59.183 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T20:13:59.183 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:13:59.183 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T20:13:59.183 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 1, 2023-05-31T20:13:59.184 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T20:13:59.184 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2023-05-31T20:13:59.184 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "", 2023-05-31T20:13:59.184 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T20:13:59.184 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2023-05-31T20:13:59.184 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T20:13:59.191 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "0.000000", 2023-05-31T20:13:59.192 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2023-05-31T20:13:59.192 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T20:13:59.192 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:13:59.193 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T20:13:59.193 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 2, 2023-05-31T20:13:59.193 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T20:13:59.193 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2023-05-31T20:13:59.193 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1685562970.274572_93.1", 2023-05-31T20:13:59.193 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T20:13:59.193 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2023-05-31T20:13:59.194 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T20:13:59.194 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2023-05-31T19:56:10.274572Z", 2023-05-31T20:13:59.194 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2023-05-31T20:13:59.194 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T20:13:59.194 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:13:59.194 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T20:13:59.195 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 3, 2023-05-31T20:13:59.195 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T20:13:59.195 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2023-05-31T20:13:59.195 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1685563798.760350_247.1", 2023-05-31T20:13:59.195 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T20:13:59.195 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2023-05-31T20:13:59.195 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T20:13:59.196 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2023-05-31T20:09:58.760350Z", 2023-05-31T20:13:59.196 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2023-05-31T20:13:59.196 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T20:13:59.196 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T20:13:59.196 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2023-05-31T20:13:59.196 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:13:59.196 INFO:teuthology.orchestra.run.smithi099.stdout: "full_sync": { 2023-05-31T20:13:59.197 INFO:teuthology.orchestra.run.smithi099.stdout: "total": 1, 2023-05-31T20:13:59.197 INFO:teuthology.orchestra.run.smithi099.stdout: "complete": 1 2023-05-31T20:13:59.197 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:13:59.197 INFO:teuthology.orchestra.run.smithi099.stdout: "current_time": "2023-05-31T20:13:59Z" 2023-05-31T20:13:59.197 INFO:teuthology.orchestra.run.smithi099.stdout:} 2023-05-31T20:13:59.197 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "2e6123d2-d695-4350-9a01-61a455bc782b", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1685562631.516089_59.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-05-31T19:50:31.516089Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1685562970.274572_93.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-05-31T19:56:10.274572Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1685563798.760350_247.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-05-31T20:09:58.760350Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-05-31T20:13:59Z" } 2023-05-31T20:13:59.198 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1685562631.516089_59.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-05-31T19:50:31.516089Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1685562970.274572_93.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-05-31T19:56:10.274572Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1685563798.760350_247.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-05-31T20:09:58.760350Z', 'realm_epoch': 2}}] 2023-05-31T20:13:59.198 DEBUG:rgw_multi.tests:log_status={0: '1_1685562631.516089_59.1', 1: '', 2: '1_1685562970.274572_93.1', 3: '1_1685563802.468126_296.1'} 2023-05-31T20:13:59.199 DEBUG:rgw_multi.tests:sync_status={0: '1_1685562631.516089_59.1', 1: '', 2: '1_1685562970.274572_93.1', 3: '1_1685563798.760350_247.1'} 2023-05-31T20:13:59.199 WARNING:rgw_multi.tests:zone a2 behind master: shard=3 master=1_1685563802.468126_296.1 target=1_1685563798.760350_247.1 2023-05-31T20:14:04.200 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-31T20:14:04.201 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-31T20:14:04.202 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-05-31T20:14:04.258 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2023-05-31T20:14:04.258 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2023-05-31T20:14:04.304 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:14:04.299+0000 7f652309b780 20 RGW-SYNC:meta: read sync status 2023-05-31T20:14:04.305 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:14:04.300+0000 7f652309b780 20 RGW-SYNC:meta: read sync status 2023-05-31T20:14:04.324 INFO:teuthology.orchestra.run.smithi099.stdout:{ 2023-05-31T20:14:04.325 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_status": { 2023-05-31T20:14:04.325 INFO:teuthology.orchestra.run.smithi099.stdout: "info": { 2023-05-31T20:14:04.325 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "sync", 2023-05-31T20:14:04.325 INFO:teuthology.orchestra.run.smithi099.stdout: "num_shards": 4, 2023-05-31T20:14:04.325 INFO:teuthology.orchestra.run.smithi099.stdout: "period": "2e6123d2-d695-4350-9a01-61a455bc782b", 2023-05-31T20:14:04.325 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2023-05-31T20:14:04.326 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:14:04.326 INFO:teuthology.orchestra.run.smithi099.stdout: "markers": [ 2023-05-31T20:14:04.326 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T20:14:04.326 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 0, 2023-05-31T20:14:04.326 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T20:14:04.327 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2023-05-31T20:14:04.327 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1685562631.516089_59.1", 2023-05-31T20:14:04.327 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T20:14:04.328 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 1, 2023-05-31T20:14:04.328 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T20:14:04.328 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2023-05-31T19:50:31.516089Z", 2023-05-31T20:14:04.328 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2023-05-31T20:14:04.328 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T20:14:04.328 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:14:04.329 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T20:14:04.329 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 1, 2023-05-31T20:14:04.329 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T20:14:04.329 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2023-05-31T20:14:04.329 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "", 2023-05-31T20:14:04.330 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T20:14:04.330 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2023-05-31T20:14:04.330 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T20:14:04.330 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "0.000000", 2023-05-31T20:14:04.330 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2023-05-31T20:14:04.331 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T20:14:04.331 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:14:04.331 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T20:14:04.333 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 2, 2023-05-31T20:14:04.333 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T20:14:04.333 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2023-05-31T20:14:04.333 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1685562970.274572_93.1", 2023-05-31T20:14:04.334 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T20:14:04.334 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2023-05-31T20:14:04.334 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T20:14:04.334 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2023-05-31T19:56:10.274572Z", 2023-05-31T20:14:04.334 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2023-05-31T20:14:04.334 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T20:14:04.335 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:14:04.335 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T20:14:04.335 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 3, 2023-05-31T20:14:04.335 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T20:14:04.335 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2023-05-31T20:14:04.335 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1685563798.772373_248.1", 2023-05-31T20:14:04.335 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T20:14:04.336 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2023-05-31T20:14:04.336 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T20:14:04.336 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2023-05-31T20:09:58.772373Z", 2023-05-31T20:14:04.336 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2023-05-31T20:14:04.336 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T20:14:04.336 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T20:14:04.337 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2023-05-31T20:14:04.337 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:14:04.337 INFO:teuthology.orchestra.run.smithi099.stdout: "full_sync": { 2023-05-31T20:14:04.337 INFO:teuthology.orchestra.run.smithi099.stdout: "total": 1, 2023-05-31T20:14:04.337 INFO:teuthology.orchestra.run.smithi099.stdout: "complete": 1 2023-05-31T20:14:04.337 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:14:04.337 INFO:teuthology.orchestra.run.smithi099.stdout: "current_time": "2023-05-31T20:14:04Z" 2023-05-31T20:14:04.338 INFO:teuthology.orchestra.run.smithi099.stdout:} 2023-05-31T20:14:04.339 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "2e6123d2-d695-4350-9a01-61a455bc782b", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1685562631.516089_59.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-05-31T19:50:31.516089Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1685562970.274572_93.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-05-31T19:56:10.274572Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1685563798.772373_248.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-05-31T20:09:58.772373Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-05-31T20:14:04Z" } 2023-05-31T20:14:04.339 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1685562631.516089_59.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-05-31T19:50:31.516089Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1685562970.274572_93.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-05-31T19:56:10.274572Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1685563798.772373_248.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-05-31T20:09:58.772373Z', 'realm_epoch': 2}}] 2023-05-31T20:14:04.340 DEBUG:rgw_multi.tests:log_status={0: '1_1685562631.516089_59.1', 1: '', 2: '1_1685562970.274572_93.1', 3: '1_1685563802.468126_296.1'} 2023-05-31T20:14:04.340 DEBUG:rgw_multi.tests:sync_status={0: '1_1685562631.516089_59.1', 1: '', 2: '1_1685562970.274572_93.1', 3: '1_1685563798.772373_248.1'} 2023-05-31T20:14:04.340 WARNING:rgw_multi.tests:zone a2 behind master: shard=3 master=1_1685563802.468126_296.1 target=1_1685563798.772373_248.1 2023-05-31T20:14:09.342 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-31T20:14:09.342 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-31T20:14:09.343 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-05-31T20:14:09.400 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2023-05-31T20:14:09.401 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2023-05-31T20:14:09.439 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:14:09.434+0000 7fbc96cf3780 20 RGW-SYNC:meta: read sync status 2023-05-31T20:14:09.440 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:14:09.435+0000 7fbc96cf3780 20 RGW-SYNC:meta: read sync status 2023-05-31T20:14:09.462 INFO:teuthology.orchestra.run.smithi099.stdout:{ 2023-05-31T20:14:09.463 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_status": { 2023-05-31T20:14:09.463 INFO:teuthology.orchestra.run.smithi099.stdout: "info": { 2023-05-31T20:14:09.463 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "sync", 2023-05-31T20:14:09.464 INFO:teuthology.orchestra.run.smithi099.stdout: "num_shards": 4, 2023-05-31T20:14:09.464 INFO:teuthology.orchestra.run.smithi099.stdout: "period": "2e6123d2-d695-4350-9a01-61a455bc782b", 2023-05-31T20:14:09.464 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2023-05-31T20:14:09.464 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:14:09.465 INFO:teuthology.orchestra.run.smithi099.stdout: "markers": [ 2023-05-31T20:14:09.465 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T20:14:09.465 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 0, 2023-05-31T20:14:09.465 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T20:14:09.465 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2023-05-31T20:14:09.465 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1685562631.516089_59.1", 2023-05-31T20:14:09.466 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T20:14:09.466 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 1, 2023-05-31T20:14:09.466 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T20:14:09.466 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2023-05-31T19:50:31.516089Z", 2023-05-31T20:14:09.466 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2023-05-31T20:14:09.466 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T20:14:09.467 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:14:09.467 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T20:14:09.468 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 1, 2023-05-31T20:14:09.468 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T20:14:09.468 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2023-05-31T20:14:09.468 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "", 2023-05-31T20:14:09.468 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T20:14:09.469 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2023-05-31T20:14:09.469 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T20:14:09.469 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "0.000000", 2023-05-31T20:14:09.469 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2023-05-31T20:14:09.469 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T20:14:09.469 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:14:09.470 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T20:14:09.470 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 2, 2023-05-31T20:14:09.470 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T20:14:09.470 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2023-05-31T20:14:09.470 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1685562970.274572_93.1", 2023-05-31T20:14:09.470 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T20:14:09.471 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2023-05-31T20:14:09.471 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T20:14:09.471 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2023-05-31T19:56:10.274572Z", 2023-05-31T20:14:09.471 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2023-05-31T20:14:09.471 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T20:14:09.471 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:14:09.471 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T20:14:09.472 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 3, 2023-05-31T20:14:09.472 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T20:14:09.472 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2023-05-31T20:14:09.472 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1685563798.756239_246.1", 2023-05-31T20:14:09.473 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T20:14:09.473 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2023-05-31T20:14:09.474 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T20:14:09.474 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2023-05-31T20:09:58.756239Z", 2023-05-31T20:14:09.474 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2023-05-31T20:14:09.474 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T20:14:09.474 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T20:14:09.474 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2023-05-31T20:14:09.475 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:14:09.475 INFO:teuthology.orchestra.run.smithi099.stdout: "full_sync": { 2023-05-31T20:14:09.475 INFO:teuthology.orchestra.run.smithi099.stdout: "total": 1, 2023-05-31T20:14:09.475 INFO:teuthology.orchestra.run.smithi099.stdout: "complete": 1 2023-05-31T20:14:09.475 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:14:09.475 INFO:teuthology.orchestra.run.smithi099.stdout: "current_time": "2023-05-31T20:14:09Z" 2023-05-31T20:14:09.475 INFO:teuthology.orchestra.run.smithi099.stdout:} 2023-05-31T20:14:09.477 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "2e6123d2-d695-4350-9a01-61a455bc782b", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1685562631.516089_59.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-05-31T19:50:31.516089Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1685562970.274572_93.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-05-31T19:56:10.274572Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1685563798.756239_246.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-05-31T20:09:58.756239Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-05-31T20:14:09Z" } 2023-05-31T20:14:09.477 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1685562631.516089_59.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-05-31T19:50:31.516089Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1685562970.274572_93.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-05-31T19:56:10.274572Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1685563798.756239_246.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-05-31T20:09:58.756239Z', 'realm_epoch': 2}}] 2023-05-31T20:14:09.478 DEBUG:rgw_multi.tests:log_status={0: '1_1685562631.516089_59.1', 1: '', 2: '1_1685562970.274572_93.1', 3: '1_1685563802.468126_296.1'} 2023-05-31T20:14:09.478 DEBUG:rgw_multi.tests:sync_status={0: '1_1685562631.516089_59.1', 1: '', 2: '1_1685562970.274572_93.1', 3: '1_1685563798.756239_246.1'} 2023-05-31T20:14:09.478 WARNING:rgw_multi.tests:zone a2 behind master: shard=3 master=1_1685563802.468126_296.1 target=1_1685563798.756239_246.1 2023-05-31T20:14:14.480 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-31T20:14:14.481 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-31T20:14:14.482 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-05-31T20:14:14.539 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2023-05-31T20:14:14.539 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2023-05-31T20:14:14.583 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:14:14.578+0000 7f6839109780 20 RGW-SYNC:meta: read sync status 2023-05-31T20:14:14.584 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:14:14.579+0000 7f6839109780 20 RGW-SYNC:meta: read sync status 2023-05-31T20:14:14.601 INFO:teuthology.orchestra.run.smithi099.stdout:{ 2023-05-31T20:14:14.602 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_status": { 2023-05-31T20:14:14.602 INFO:teuthology.orchestra.run.smithi099.stdout: "info": { 2023-05-31T20:14:14.602 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "sync", 2023-05-31T20:14:14.602 INFO:teuthology.orchestra.run.smithi099.stdout: "num_shards": 4, 2023-05-31T20:14:14.602 INFO:teuthology.orchestra.run.smithi099.stdout: "period": "2e6123d2-d695-4350-9a01-61a455bc782b", 2023-05-31T20:14:14.602 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2023-05-31T20:14:14.603 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:14:14.603 INFO:teuthology.orchestra.run.smithi099.stdout: "markers": [ 2023-05-31T20:14:14.603 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T20:14:14.603 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 0, 2023-05-31T20:14:14.604 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T20:14:14.604 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2023-05-31T20:14:14.604 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1685562631.516089_59.1", 2023-05-31T20:14:14.604 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T20:14:14.604 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 1, 2023-05-31T20:14:14.604 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T20:14:14.605 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2023-05-31T19:50:31.516089Z", 2023-05-31T20:14:14.605 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2023-05-31T20:14:14.605 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T20:14:14.605 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:14:14.605 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T20:14:14.605 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 1, 2023-05-31T20:14:14.605 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T20:14:14.606 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2023-05-31T20:14:14.606 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "", 2023-05-31T20:14:14.606 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T20:14:14.606 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2023-05-31T20:14:14.606 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T20:14:14.606 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "0.000000", 2023-05-31T20:14:14.607 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2023-05-31T20:14:14.607 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T20:14:14.607 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:14:14.607 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T20:14:14.607 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 2, 2023-05-31T20:14:14.607 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T20:14:14.608 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2023-05-31T20:14:14.608 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1685562970.274572_93.1", 2023-05-31T20:14:14.608 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T20:14:14.608 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2023-05-31T20:14:14.608 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T20:14:14.608 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2023-05-31T19:56:10.274572Z", 2023-05-31T20:14:14.608 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2023-05-31T20:14:14.609 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T20:14:14.609 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:14:14.609 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T20:14:14.609 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 3, 2023-05-31T20:14:14.609 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T20:14:14.609 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2023-05-31T20:14:14.609 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1685563798.885419_253.1", 2023-05-31T20:14:14.610 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T20:14:14.610 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2023-05-31T20:14:14.610 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T20:14:14.610 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2023-05-31T20:09:58.885419Z", 2023-05-31T20:14:14.610 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2023-05-31T20:14:14.610 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T20:14:14.611 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T20:14:14.611 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2023-05-31T20:14:14.611 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:14:14.611 INFO:teuthology.orchestra.run.smithi099.stdout: "full_sync": { 2023-05-31T20:14:14.611 INFO:teuthology.orchestra.run.smithi099.stdout: "total": 1, 2023-05-31T20:14:14.611 INFO:teuthology.orchestra.run.smithi099.stdout: "complete": 1 2023-05-31T20:14:14.611 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:14:14.612 INFO:teuthology.orchestra.run.smithi099.stdout: "current_time": "2023-05-31T20:14:14Z" 2023-05-31T20:14:14.612 INFO:teuthology.orchestra.run.smithi099.stdout:} 2023-05-31T20:14:14.613 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "2e6123d2-d695-4350-9a01-61a455bc782b", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1685562631.516089_59.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-05-31T19:50:31.516089Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1685562970.274572_93.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-05-31T19:56:10.274572Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1685563798.885419_253.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-05-31T20:09:58.885419Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-05-31T20:14:14Z" } 2023-05-31T20:14:14.614 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1685562631.516089_59.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-05-31T19:50:31.516089Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1685562970.274572_93.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-05-31T19:56:10.274572Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1685563798.885419_253.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-05-31T20:09:58.885419Z', 'realm_epoch': 2}}] 2023-05-31T20:14:14.615 DEBUG:rgw_multi.tests:log_status={0: '1_1685562631.516089_59.1', 1: '', 2: '1_1685562970.274572_93.1', 3: '1_1685563802.468126_296.1'} 2023-05-31T20:14:14.615 DEBUG:rgw_multi.tests:sync_status={0: '1_1685562631.516089_59.1', 1: '', 2: '1_1685562970.274572_93.1', 3: '1_1685563798.885419_253.1'} 2023-05-31T20:14:14.615 WARNING:rgw_multi.tests:zone a2 behind master: shard=3 master=1_1685563802.468126_296.1 target=1_1685563798.885419_253.1 2023-05-31T20:14:19.616 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-31T20:14:19.617 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-31T20:14:19.618 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-05-31T20:14:19.678 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2023-05-31T20:14:19.678 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2023-05-31T20:14:19.733 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:14:19.727+0000 7f60548f0780 20 RGW-SYNC:meta: read sync status 2023-05-31T20:14:19.734 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:14:19.728+0000 7f60548f0780 20 RGW-SYNC:meta: read sync status 2023-05-31T20:14:19.747 INFO:teuthology.orchestra.run.smithi099.stdout:{ 2023-05-31T20:14:19.747 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_status": { 2023-05-31T20:14:19.747 INFO:teuthology.orchestra.run.smithi099.stdout: "info": { 2023-05-31T20:14:19.748 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "sync", 2023-05-31T20:14:19.748 INFO:teuthology.orchestra.run.smithi099.stdout: "num_shards": 4, 2023-05-31T20:14:19.748 INFO:teuthology.orchestra.run.smithi099.stdout: "period": "2e6123d2-d695-4350-9a01-61a455bc782b", 2023-05-31T20:14:19.748 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2023-05-31T20:14:19.748 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:14:19.748 INFO:teuthology.orchestra.run.smithi099.stdout: "markers": [ 2023-05-31T20:14:19.749 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T20:14:19.749 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 0, 2023-05-31T20:14:19.749 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T20:14:19.749 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2023-05-31T20:14:19.749 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1685562631.516089_59.1", 2023-05-31T20:14:19.749 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T20:14:19.749 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 1, 2023-05-31T20:14:19.750 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T20:14:19.750 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2023-05-31T19:50:31.516089Z", 2023-05-31T20:14:19.750 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2023-05-31T20:14:19.750 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T20:14:19.750 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:14:19.750 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T20:14:19.750 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 1, 2023-05-31T20:14:19.751 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T20:14:19.751 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2023-05-31T20:14:19.751 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "", 2023-05-31T20:14:19.751 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T20:14:19.751 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2023-05-31T20:14:19.751 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T20:14:19.752 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "0.000000", 2023-05-31T20:14:19.752 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2023-05-31T20:14:19.752 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T20:14:19.752 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:14:19.752 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T20:14:19.752 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 2, 2023-05-31T20:14:19.752 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T20:14:19.753 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2023-05-31T20:14:19.753 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1685562970.274572_93.1", 2023-05-31T20:14:19.753 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T20:14:19.753 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2023-05-31T20:14:19.753 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T20:14:19.753 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2023-05-31T19:56:10.274572Z", 2023-05-31T20:14:19.753 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2023-05-31T20:14:19.754 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T20:14:19.754 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:14:19.754 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T20:14:19.754 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 3, 2023-05-31T20:14:19.754 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T20:14:19.754 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2023-05-31T20:14:19.754 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1685563798.896727_254.1", 2023-05-31T20:14:19.755 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T20:14:19.755 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2023-05-31T20:14:19.755 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T20:14:19.755 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2023-05-31T20:09:58.896727Z", 2023-05-31T20:14:19.755 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2023-05-31T20:14:19.755 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T20:14:19.756 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T20:14:19.756 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2023-05-31T20:14:19.756 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:14:19.756 INFO:teuthology.orchestra.run.smithi099.stdout: "full_sync": { 2023-05-31T20:14:19.756 INFO:teuthology.orchestra.run.smithi099.stdout: "total": 1, 2023-05-31T20:14:19.756 INFO:teuthology.orchestra.run.smithi099.stdout: "complete": 1 2023-05-31T20:14:19.756 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:14:19.757 INFO:teuthology.orchestra.run.smithi099.stdout: "current_time": "2023-05-31T20:14:19Z" 2023-05-31T20:14:19.757 INFO:teuthology.orchestra.run.smithi099.stdout:} 2023-05-31T20:14:19.758 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "2e6123d2-d695-4350-9a01-61a455bc782b", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1685562631.516089_59.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-05-31T19:50:31.516089Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1685562970.274572_93.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-05-31T19:56:10.274572Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1685563798.896727_254.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-05-31T20:09:58.896727Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-05-31T20:14:19Z" } 2023-05-31T20:14:19.758 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1685562631.516089_59.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-05-31T19:50:31.516089Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1685562970.274572_93.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-05-31T19:56:10.274572Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1685563798.896727_254.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-05-31T20:09:58.896727Z', 'realm_epoch': 2}}] 2023-05-31T20:14:19.759 DEBUG:rgw_multi.tests:log_status={0: '1_1685562631.516089_59.1', 1: '', 2: '1_1685562970.274572_93.1', 3: '1_1685563802.468126_296.1'} 2023-05-31T20:14:19.759 DEBUG:rgw_multi.tests:sync_status={0: '1_1685562631.516089_59.1', 1: '', 2: '1_1685562970.274572_93.1', 3: '1_1685563798.896727_254.1'} 2023-05-31T20:14:19.760 WARNING:rgw_multi.tests:zone a2 behind master: shard=3 master=1_1685563802.468126_296.1 target=1_1685563798.896727_254.1 2023-05-31T20:14:24.760 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-31T20:14:24.761 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-31T20:14:24.762 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-05-31T20:14:24.820 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2023-05-31T20:14:24.820 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2023-05-31T20:14:24.854 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:14:24.846+0000 7fc5f1475780 20 RGW-SYNC:meta: read sync status 2023-05-31T20:14:24.854 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:14:24.847+0000 7fc5f1475780 20 RGW-SYNC:meta: read sync status 2023-05-31T20:14:24.868 INFO:teuthology.orchestra.run.smithi099.stdout:{ 2023-05-31T20:14:24.868 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_status": { 2023-05-31T20:14:24.868 INFO:teuthology.orchestra.run.smithi099.stdout: "info": { 2023-05-31T20:14:24.868 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "sync", 2023-05-31T20:14:24.869 INFO:teuthology.orchestra.run.smithi099.stdout: "num_shards": 4, 2023-05-31T20:14:24.869 INFO:teuthology.orchestra.run.smithi099.stdout: "period": "2e6123d2-d695-4350-9a01-61a455bc782b", 2023-05-31T20:14:24.869 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2023-05-31T20:14:24.869 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:14:24.869 INFO:teuthology.orchestra.run.smithi099.stdout: "markers": [ 2023-05-31T20:14:24.869 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T20:14:24.870 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 0, 2023-05-31T20:14:24.870 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T20:14:24.870 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2023-05-31T20:14:24.870 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1685562631.516089_59.1", 2023-05-31T20:14:24.870 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T20:14:24.870 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 1, 2023-05-31T20:14:24.871 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T20:14:24.871 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2023-05-31T19:50:31.516089Z", 2023-05-31T20:14:24.871 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2023-05-31T20:14:24.871 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T20:14:24.871 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:14:24.871 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T20:14:24.871 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 1, 2023-05-31T20:14:24.872 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T20:14:24.872 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2023-05-31T20:14:24.873 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "", 2023-05-31T20:14:24.873 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T20:14:24.873 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2023-05-31T20:14:24.873 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T20:14:24.874 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "0.000000", 2023-05-31T20:14:24.874 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2023-05-31T20:14:24.874 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T20:14:24.874 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:14:24.874 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T20:14:24.874 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 2, 2023-05-31T20:14:24.874 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T20:14:24.875 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2023-05-31T20:14:24.875 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1685562970.274572_93.1", 2023-05-31T20:14:24.875 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T20:14:24.875 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2023-05-31T20:14:24.875 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T20:14:24.875 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2023-05-31T19:56:10.274572Z", 2023-05-31T20:14:24.876 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2023-05-31T20:14:24.876 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T20:14:24.876 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:14:24.876 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T20:14:24.876 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 3, 2023-05-31T20:14:24.878 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T20:14:24.878 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2023-05-31T20:14:24.878 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1685563798.756239_246.1", 2023-05-31T20:14:24.879 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T20:14:24.879 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2023-05-31T20:14:24.879 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T20:14:24.879 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2023-05-31T20:09:58.756239Z", 2023-05-31T20:14:24.879 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2023-05-31T20:14:24.879 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T20:14:24.879 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T20:14:24.880 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2023-05-31T20:14:24.880 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:14:24.880 INFO:teuthology.orchestra.run.smithi099.stdout: "full_sync": { 2023-05-31T20:14:24.880 INFO:teuthology.orchestra.run.smithi099.stdout: "total": 1, 2023-05-31T20:14:24.880 INFO:teuthology.orchestra.run.smithi099.stdout: "complete": 1 2023-05-31T20:14:24.880 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:14:24.881 INFO:teuthology.orchestra.run.smithi099.stdout: "current_time": "2023-05-31T20:14:24Z" 2023-05-31T20:14:24.881 INFO:teuthology.orchestra.run.smithi099.stdout:} 2023-05-31T20:14:24.882 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "2e6123d2-d695-4350-9a01-61a455bc782b", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1685562631.516089_59.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-05-31T19:50:31.516089Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1685562970.274572_93.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-05-31T19:56:10.274572Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1685563798.756239_246.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-05-31T20:09:58.756239Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-05-31T20:14:24Z" } 2023-05-31T20:14:24.885 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1685562631.516089_59.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-05-31T19:50:31.516089Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1685562970.274572_93.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-05-31T19:56:10.274572Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1685563798.756239_246.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-05-31T20:09:58.756239Z', 'realm_epoch': 2}}] 2023-05-31T20:14:24.885 DEBUG:rgw_multi.tests:log_status={0: '1_1685562631.516089_59.1', 1: '', 2: '1_1685562970.274572_93.1', 3: '1_1685563802.468126_296.1'} 2023-05-31T20:14:24.886 DEBUG:rgw_multi.tests:sync_status={0: '1_1685562631.516089_59.1', 1: '', 2: '1_1685562970.274572_93.1', 3: '1_1685563798.756239_246.1'} 2023-05-31T20:14:24.886 WARNING:rgw_multi.tests:zone a2 behind master: shard=3 master=1_1685563802.468126_296.1 target=1_1685563798.756239_246.1 2023-05-31T20:14:29.887 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-31T20:14:29.888 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-31T20:14:29.889 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-05-31T20:14:29.942 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2023-05-31T20:14:29.943 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2023-05-31T20:14:29.974 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:14:29.968+0000 7fc62da3a780 20 RGW-SYNC:meta: read sync status 2023-05-31T20:14:29.974 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:14:29.969+0000 7fc62da3a780 20 RGW-SYNC:meta: read sync status 2023-05-31T20:14:29.991 INFO:teuthology.orchestra.run.smithi099.stdout:{ 2023-05-31T20:14:29.993 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_status": { 2023-05-31T20:14:29.993 INFO:teuthology.orchestra.run.smithi099.stdout: "info": { 2023-05-31T20:14:29.994 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "sync", 2023-05-31T20:14:29.994 INFO:teuthology.orchestra.run.smithi099.stdout: "num_shards": 4, 2023-05-31T20:14:29.994 INFO:teuthology.orchestra.run.smithi099.stdout: "period": "2e6123d2-d695-4350-9a01-61a455bc782b", 2023-05-31T20:14:29.995 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2023-05-31T20:14:29.995 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:14:29.996 INFO:teuthology.orchestra.run.smithi099.stdout: "markers": [ 2023-05-31T20:14:29.996 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T20:14:29.996 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 0, 2023-05-31T20:14:29.996 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T20:14:29.997 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2023-05-31T20:14:29.999 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1685562631.516089_59.1", 2023-05-31T20:14:29.999 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T20:14:29.999 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 1, 2023-05-31T20:14:29.999 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T20:14:30.000 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2023-05-31T19:50:31.516089Z", 2023-05-31T20:14:30.000 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2023-05-31T20:14:30.000 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T20:14:30.000 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:14:30.000 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T20:14:30.000 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 1, 2023-05-31T20:14:30.000 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T20:14:30.001 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2023-05-31T20:14:30.001 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "", 2023-05-31T20:14:30.001 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T20:14:30.001 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2023-05-31T20:14:30.001 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T20:14:30.001 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "0.000000", 2023-05-31T20:14:30.002 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2023-05-31T20:14:30.002 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T20:14:30.002 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:14:30.002 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T20:14:30.002 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 2, 2023-05-31T20:14:30.002 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T20:14:30.002 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2023-05-31T20:14:30.003 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1685562970.274572_93.1", 2023-05-31T20:14:30.003 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T20:14:30.003 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2023-05-31T20:14:30.003 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T20:14:30.003 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2023-05-31T19:56:10.274572Z", 2023-05-31T20:14:30.003 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2023-05-31T20:14:30.003 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T20:14:30.004 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:14:30.004 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T20:14:30.004 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 3, 2023-05-31T20:14:30.004 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T20:14:30.004 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2023-05-31T20:14:30.004 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1685563798.807987_249.1", 2023-05-31T20:14:30.005 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T20:14:30.005 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2023-05-31T20:14:30.005 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T20:14:30.005 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2023-05-31T20:09:58.807987Z", 2023-05-31T20:14:30.005 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2023-05-31T20:14:30.005 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T20:14:30.005 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T20:14:30.006 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2023-05-31T20:14:30.006 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:14:30.006 INFO:teuthology.orchestra.run.smithi099.stdout: "full_sync": { 2023-05-31T20:14:30.006 INFO:teuthology.orchestra.run.smithi099.stdout: "total": 1, 2023-05-31T20:14:30.006 INFO:teuthology.orchestra.run.smithi099.stdout: "complete": 1 2023-05-31T20:14:30.006 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:14:30.006 INFO:teuthology.orchestra.run.smithi099.stdout: "current_time": "2023-05-31T20:14:29Z" 2023-05-31T20:14:30.007 INFO:teuthology.orchestra.run.smithi099.stdout:} 2023-05-31T20:14:30.008 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "2e6123d2-d695-4350-9a01-61a455bc782b", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1685562631.516089_59.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-05-31T19:50:31.516089Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1685562970.274572_93.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-05-31T19:56:10.274572Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1685563798.807987_249.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-05-31T20:09:58.807987Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-05-31T20:14:29Z" } 2023-05-31T20:14:30.008 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1685562631.516089_59.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-05-31T19:50:31.516089Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1685562970.274572_93.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-05-31T19:56:10.274572Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1685563798.807987_249.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-05-31T20:09:58.807987Z', 'realm_epoch': 2}}] 2023-05-31T20:14:30.009 DEBUG:rgw_multi.tests:log_status={0: '1_1685562631.516089_59.1', 1: '', 2: '1_1685562970.274572_93.1', 3: '1_1685563802.468126_296.1'} 2023-05-31T20:14:30.009 DEBUG:rgw_multi.tests:sync_status={0: '1_1685562631.516089_59.1', 1: '', 2: '1_1685562970.274572_93.1', 3: '1_1685563798.807987_249.1'} 2023-05-31T20:14:30.009 WARNING:rgw_multi.tests:zone a2 behind master: shard=3 master=1_1685563802.468126_296.1 target=1_1685563798.807987_249.1 2023-05-31T20:14:35.010 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-31T20:14:35.011 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-31T20:14:35.012 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-05-31T20:14:35.062 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2023-05-31T20:14:35.063 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2023-05-31T20:14:35.102 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:14:35.097+0000 7f04fa476780 20 RGW-SYNC:meta: read sync status 2023-05-31T20:14:35.103 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:14:35.098+0000 7f04fa476780 20 RGW-SYNC:meta: read sync status 2023-05-31T20:14:35.122 INFO:teuthology.orchestra.run.smithi099.stdout:{ 2023-05-31T20:14:35.123 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_status": { 2023-05-31T20:14:35.123 INFO:teuthology.orchestra.run.smithi099.stdout: "info": { 2023-05-31T20:14:35.124 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "sync", 2023-05-31T20:14:35.124 INFO:teuthology.orchestra.run.smithi099.stdout: "num_shards": 4, 2023-05-31T20:14:35.124 INFO:teuthology.orchestra.run.smithi099.stdout: "period": "2e6123d2-d695-4350-9a01-61a455bc782b", 2023-05-31T20:14:35.125 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2023-05-31T20:14:35.125 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:14:35.125 INFO:teuthology.orchestra.run.smithi099.stdout: "markers": [ 2023-05-31T20:14:35.125 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T20:14:35.126 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 0, 2023-05-31T20:14:35.126 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T20:14:35.126 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2023-05-31T20:14:35.126 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1685562631.516089_59.1", 2023-05-31T20:14:35.126 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T20:14:35.126 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 1, 2023-05-31T20:14:35.126 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T20:14:35.127 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2023-05-31T19:50:31.516089Z", 2023-05-31T20:14:35.127 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2023-05-31T20:14:35.127 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T20:14:35.127 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:14:35.127 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T20:14:35.127 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 1, 2023-05-31T20:14:35.127 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T20:14:35.128 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2023-05-31T20:14:35.128 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "", 2023-05-31T20:14:35.128 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T20:14:35.128 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2023-05-31T20:14:35.128 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T20:14:35.128 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "0.000000", 2023-05-31T20:14:35.129 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2023-05-31T20:14:35.129 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T20:14:35.129 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:14:35.129 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T20:14:35.129 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 2, 2023-05-31T20:14:35.129 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T20:14:35.130 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2023-05-31T20:14:35.130 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1685562970.274572_93.1", 2023-05-31T20:14:35.130 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T20:14:35.130 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2023-05-31T20:14:35.130 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T20:14:35.130 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2023-05-31T19:56:10.274572Z", 2023-05-31T20:14:35.130 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2023-05-31T20:14:35.131 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T20:14:35.131 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:14:35.131 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T20:14:35.131 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 3, 2023-05-31T20:14:35.131 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T20:14:35.131 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2023-05-31T20:14:35.131 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1685563798.756239_246.1", 2023-05-31T20:14:35.132 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T20:14:35.132 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2023-05-31T20:14:35.132 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T20:14:35.132 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2023-05-31T20:09:58.756239Z", 2023-05-31T20:14:35.132 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2023-05-31T20:14:35.132 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T20:14:35.132 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T20:14:35.133 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2023-05-31T20:14:35.133 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:14:35.133 INFO:teuthology.orchestra.run.smithi099.stdout: "full_sync": { 2023-05-31T20:14:35.133 INFO:teuthology.orchestra.run.smithi099.stdout: "total": 1, 2023-05-31T20:14:35.133 INFO:teuthology.orchestra.run.smithi099.stdout: "complete": 1 2023-05-31T20:14:35.133 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:14:35.133 INFO:teuthology.orchestra.run.smithi099.stdout: "current_time": "2023-05-31T20:14:35Z" 2023-05-31T20:14:35.134 INFO:teuthology.orchestra.run.smithi099.stdout:} 2023-05-31T20:14:35.135 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "2e6123d2-d695-4350-9a01-61a455bc782b", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1685562631.516089_59.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-05-31T19:50:31.516089Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1685562970.274572_93.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-05-31T19:56:10.274572Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1685563798.756239_246.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-05-31T20:09:58.756239Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-05-31T20:14:35Z" } 2023-05-31T20:14:35.135 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1685562631.516089_59.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-05-31T19:50:31.516089Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1685562970.274572_93.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-05-31T19:56:10.274572Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1685563798.756239_246.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-05-31T20:09:58.756239Z', 'realm_epoch': 2}}] 2023-05-31T20:14:35.136 DEBUG:rgw_multi.tests:log_status={0: '1_1685562631.516089_59.1', 1: '', 2: '1_1685562970.274572_93.1', 3: '1_1685563802.468126_296.1'} 2023-05-31T20:14:35.136 DEBUG:rgw_multi.tests:sync_status={0: '1_1685562631.516089_59.1', 1: '', 2: '1_1685562970.274572_93.1', 3: '1_1685563798.756239_246.1'} 2023-05-31T20:14:35.136 WARNING:rgw_multi.tests:zone a2 behind master: shard=3 master=1_1685563802.468126_296.1 target=1_1685563798.756239_246.1 2023-05-31T20:14:40.137 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-31T20:14:40.138 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-31T20:14:40.140 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-05-31T20:14:40.191 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2023-05-31T20:14:40.191 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2023-05-31T20:14:40.239 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:14:40.234+0000 7f4b0c479780 20 RGW-SYNC:meta: read sync status 2023-05-31T20:14:40.242 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:14:40.235+0000 7f4b0c479780 20 RGW-SYNC:meta: read sync status 2023-05-31T20:14:40.252 INFO:teuthology.orchestra.run.smithi099.stdout:{ 2023-05-31T20:14:40.253 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_status": { 2023-05-31T20:14:40.253 INFO:teuthology.orchestra.run.smithi099.stdout: "info": { 2023-05-31T20:14:40.254 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "sync", 2023-05-31T20:14:40.254 INFO:teuthology.orchestra.run.smithi099.stdout: "num_shards": 4, 2023-05-31T20:14:40.254 INFO:teuthology.orchestra.run.smithi099.stdout: "period": "2e6123d2-d695-4350-9a01-61a455bc782b", 2023-05-31T20:14:40.255 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2023-05-31T20:14:40.255 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:14:40.256 INFO:teuthology.orchestra.run.smithi099.stdout: "markers": [ 2023-05-31T20:14:40.256 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T20:14:40.256 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 0, 2023-05-31T20:14:40.256 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T20:14:40.257 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2023-05-31T20:14:40.257 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1685562631.516089_59.1", 2023-05-31T20:14:40.257 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T20:14:40.257 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 1, 2023-05-31T20:14:40.259 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T20:14:40.260 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2023-05-31T19:50:31.516089Z", 2023-05-31T20:14:40.260 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2023-05-31T20:14:40.261 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T20:14:40.261 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:14:40.261 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T20:14:40.261 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 1, 2023-05-31T20:14:40.262 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T20:14:40.262 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2023-05-31T20:14:40.263 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "", 2023-05-31T20:14:40.263 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T20:14:40.264 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2023-05-31T20:14:40.264 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T20:14:40.264 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "0.000000", 2023-05-31T20:14:40.264 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2023-05-31T20:14:40.264 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T20:14:40.264 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:14:40.265 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T20:14:40.266 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 2, 2023-05-31T20:14:40.266 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T20:14:40.266 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2023-05-31T20:14:40.267 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1685562970.274572_93.1", 2023-05-31T20:14:40.267 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T20:14:40.268 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2023-05-31T20:14:40.268 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T20:14:40.269 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2023-05-31T19:56:10.274572Z", 2023-05-31T20:14:40.269 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2023-05-31T20:14:40.269 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T20:14:40.270 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:14:40.270 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T20:14:40.270 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 3, 2023-05-31T20:14:40.270 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T20:14:40.271 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2023-05-31T20:14:40.271 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1685563798.885419_253.1", 2023-05-31T20:14:40.272 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T20:14:40.273 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2023-05-31T20:14:40.273 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T20:14:40.273 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2023-05-31T20:09:58.885419Z", 2023-05-31T20:14:40.273 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2023-05-31T20:14:40.273 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T20:14:40.274 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T20:14:40.274 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2023-05-31T20:14:40.274 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:14:40.274 INFO:teuthology.orchestra.run.smithi099.stdout: "full_sync": { 2023-05-31T20:14:40.274 INFO:teuthology.orchestra.run.smithi099.stdout: "total": 1, 2023-05-31T20:14:40.275 INFO:teuthology.orchestra.run.smithi099.stdout: "complete": 1 2023-05-31T20:14:40.275 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:14:40.275 INFO:teuthology.orchestra.run.smithi099.stdout: "current_time": "2023-05-31T20:14:40Z" 2023-05-31T20:14:40.275 INFO:teuthology.orchestra.run.smithi099.stdout:} 2023-05-31T20:14:40.276 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "2e6123d2-d695-4350-9a01-61a455bc782b", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1685562631.516089_59.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-05-31T19:50:31.516089Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1685562970.274572_93.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-05-31T19:56:10.274572Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1685563798.885419_253.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-05-31T20:09:58.885419Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-05-31T20:14:40Z" } 2023-05-31T20:14:40.277 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1685562631.516089_59.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-05-31T19:50:31.516089Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1685562970.274572_93.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-05-31T19:56:10.274572Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1685563798.885419_253.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-05-31T20:09:58.885419Z', 'realm_epoch': 2}}] 2023-05-31T20:14:40.278 DEBUG:rgw_multi.tests:log_status={0: '1_1685562631.516089_59.1', 1: '', 2: '1_1685562970.274572_93.1', 3: '1_1685563802.468126_296.1'} 2023-05-31T20:14:40.278 DEBUG:rgw_multi.tests:sync_status={0: '1_1685562631.516089_59.1', 1: '', 2: '1_1685562970.274572_93.1', 3: '1_1685563798.885419_253.1'} 2023-05-31T20:14:40.279 WARNING:rgw_multi.tests:zone a2 behind master: shard=3 master=1_1685563802.468126_296.1 target=1_1685563798.885419_253.1 2023-05-31T20:14:45.280 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-31T20:14:45.280 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-31T20:14:45.281 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-05-31T20:14:45.362 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2023-05-31T20:14:45.362 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2023-05-31T20:14:45.427 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:14:45.418+0000 7f7f52405780 20 RGW-SYNC:meta: read sync status 2023-05-31T20:14:45.428 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:14:45.419+0000 7f7f52405780 20 RGW-SYNC:meta: read sync status 2023-05-31T20:14:45.443 INFO:teuthology.orchestra.run.smithi099.stdout:{ 2023-05-31T20:14:45.444 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_status": { 2023-05-31T20:14:45.444 INFO:teuthology.orchestra.run.smithi099.stdout: "info": { 2023-05-31T20:14:45.445 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "sync", 2023-05-31T20:14:45.445 INFO:teuthology.orchestra.run.smithi099.stdout: "num_shards": 4, 2023-05-31T20:14:45.446 INFO:teuthology.orchestra.run.smithi099.stdout: "period": "2e6123d2-d695-4350-9a01-61a455bc782b", 2023-05-31T20:14:45.446 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2023-05-31T20:14:45.446 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:14:45.447 INFO:teuthology.orchestra.run.smithi099.stdout: "markers": [ 2023-05-31T20:14:45.448 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T20:14:45.448 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 0, 2023-05-31T20:14:45.448 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T20:14:45.449 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2023-05-31T20:14:45.450 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1685562631.516089_59.1", 2023-05-31T20:14:45.450 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T20:14:45.450 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 1, 2023-05-31T20:14:45.451 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T20:14:45.451 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2023-05-31T19:50:31.516089Z", 2023-05-31T20:14:45.451 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2023-05-31T20:14:45.452 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T20:14:45.452 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:14:45.452 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T20:14:45.453 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 1, 2023-05-31T20:14:45.453 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T20:14:45.453 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2023-05-31T20:14:45.453 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "", 2023-05-31T20:14:45.454 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T20:14:45.454 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2023-05-31T20:14:45.455 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T20:14:45.455 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "0.000000", 2023-05-31T20:14:45.456 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2023-05-31T20:14:45.456 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T20:14:45.456 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:14:45.457 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T20:14:45.457 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 2, 2023-05-31T20:14:45.457 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T20:14:45.458 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2023-05-31T20:14:45.458 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1685562970.274572_93.1", 2023-05-31T20:14:45.458 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T20:14:45.459 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2023-05-31T20:14:45.459 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T20:14:45.459 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2023-05-31T19:56:10.274572Z", 2023-05-31T20:14:45.460 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2023-05-31T20:14:45.460 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T20:14:45.461 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:14:45.465 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T20:14:45.465 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 3, 2023-05-31T20:14:45.465 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T20:14:45.466 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2023-05-31T20:14:45.466 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1685563798.827280_252.1", 2023-05-31T20:14:45.466 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T20:14:45.466 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2023-05-31T20:14:45.467 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T20:14:45.467 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2023-05-31T20:09:58.827280Z", 2023-05-31T20:14:45.467 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2023-05-31T20:14:45.468 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T20:14:45.468 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T20:14:45.468 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2023-05-31T20:14:45.469 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:14:45.469 INFO:teuthology.orchestra.run.smithi099.stdout: "full_sync": { 2023-05-31T20:14:45.469 INFO:teuthology.orchestra.run.smithi099.stdout: "total": 1, 2023-05-31T20:14:45.469 INFO:teuthology.orchestra.run.smithi099.stdout: "complete": 1 2023-05-31T20:14:45.470 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:14:45.470 INFO:teuthology.orchestra.run.smithi099.stdout: "current_time": "2023-05-31T20:14:45Z" 2023-05-31T20:14:45.470 INFO:teuthology.orchestra.run.smithi099.stdout:} 2023-05-31T20:14:45.471 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "2e6123d2-d695-4350-9a01-61a455bc782b", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1685562631.516089_59.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-05-31T19:50:31.516089Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1685562970.274572_93.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-05-31T19:56:10.274572Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1685563798.827280_252.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-05-31T20:09:58.827280Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-05-31T20:14:45Z" } 2023-05-31T20:14:45.472 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1685562631.516089_59.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-05-31T19:50:31.516089Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1685562970.274572_93.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-05-31T19:56:10.274572Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1685563798.827280_252.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-05-31T20:09:58.827280Z', 'realm_epoch': 2}}] 2023-05-31T20:14:45.474 DEBUG:rgw_multi.tests:log_status={0: '1_1685562631.516089_59.1', 1: '', 2: '1_1685562970.274572_93.1', 3: '1_1685563802.468126_296.1'} 2023-05-31T20:14:45.475 DEBUG:rgw_multi.tests:sync_status={0: '1_1685562631.516089_59.1', 1: '', 2: '1_1685562970.274572_93.1', 3: '1_1685563798.827280_252.1'} 2023-05-31T20:14:45.475 WARNING:rgw_multi.tests:zone a2 behind master: shard=3 master=1_1685563802.468126_296.1 target=1_1685563798.827280_252.1 2023-05-31T20:14:50.476 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-31T20:14:50.477 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-31T20:14:50.479 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-05-31T20:14:50.540 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2023-05-31T20:14:50.540 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2023-05-31T20:14:50.586 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:14:50.579+0000 7fa08b133780 20 RGW-SYNC:meta: read sync status 2023-05-31T20:14:50.586 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:14:50.580+0000 7fa08b133780 20 RGW-SYNC:meta: read sync status 2023-05-31T20:14:50.603 INFO:teuthology.orchestra.run.smithi099.stdout:{ 2023-05-31T20:14:50.604 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_status": { 2023-05-31T20:14:50.604 INFO:teuthology.orchestra.run.smithi099.stdout: "info": { 2023-05-31T20:14:50.604 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "sync", 2023-05-31T20:14:50.604 INFO:teuthology.orchestra.run.smithi099.stdout: "num_shards": 4, 2023-05-31T20:14:50.605 INFO:teuthology.orchestra.run.smithi099.stdout: "period": "2e6123d2-d695-4350-9a01-61a455bc782b", 2023-05-31T20:14:50.605 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2023-05-31T20:14:50.605 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:14:50.605 INFO:teuthology.orchestra.run.smithi099.stdout: "markers": [ 2023-05-31T20:14:50.605 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T20:14:50.605 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 0, 2023-05-31T20:14:50.606 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T20:14:50.606 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2023-05-31T20:14:50.606 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1685562631.516089_59.1", 2023-05-31T20:14:50.606 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T20:14:50.607 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 1, 2023-05-31T20:14:50.607 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T20:14:50.607 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2023-05-31T19:50:31.516089Z", 2023-05-31T20:14:50.607 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2023-05-31T20:14:50.607 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T20:14:50.607 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:14:50.608 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T20:14:50.608 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 1, 2023-05-31T20:14:50.608 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T20:14:50.608 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2023-05-31T20:14:50.608 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "", 2023-05-31T20:14:50.608 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T20:14:50.608 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2023-05-31T20:14:50.609 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T20:14:50.609 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "0.000000", 2023-05-31T20:14:50.609 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2023-05-31T20:14:50.609 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T20:14:50.609 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:14:50.609 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T20:14:50.609 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 2, 2023-05-31T20:14:50.610 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T20:14:50.610 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2023-05-31T20:14:50.610 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1685562970.274572_93.1", 2023-05-31T20:14:50.610 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T20:14:50.610 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2023-05-31T20:14:50.610 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T20:14:50.611 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2023-05-31T19:56:10.274572Z", 2023-05-31T20:14:50.611 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2023-05-31T20:14:50.611 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T20:14:50.611 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:14:50.612 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T20:14:50.612 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 3, 2023-05-31T20:14:50.612 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T20:14:50.612 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2023-05-31T20:14:50.612 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1685563799.098535_267.1", 2023-05-31T20:14:50.612 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T20:14:50.613 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2023-05-31T20:14:50.613 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T20:14:50.613 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2023-05-31T20:09:59.098535Z", 2023-05-31T20:14:50.613 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2023-05-31T20:14:50.613 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T20:14:50.613 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T20:14:50.613 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2023-05-31T20:14:50.614 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:14:50.614 INFO:teuthology.orchestra.run.smithi099.stdout: "full_sync": { 2023-05-31T20:14:50.614 INFO:teuthology.orchestra.run.smithi099.stdout: "total": 1, 2023-05-31T20:14:50.614 INFO:teuthology.orchestra.run.smithi099.stdout: "complete": 1 2023-05-31T20:14:50.614 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:14:50.614 INFO:teuthology.orchestra.run.smithi099.stdout: "current_time": "2023-05-31T20:14:50Z" 2023-05-31T20:14:50.614 INFO:teuthology.orchestra.run.smithi099.stdout:} 2023-05-31T20:14:50.621 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "2e6123d2-d695-4350-9a01-61a455bc782b", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1685562631.516089_59.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-05-31T19:50:31.516089Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1685562970.274572_93.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-05-31T19:56:10.274572Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1685563799.098535_267.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-05-31T20:09:59.098535Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-05-31T20:14:50Z" } 2023-05-31T20:14:50.622 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1685562631.516089_59.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-05-31T19:50:31.516089Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1685562970.274572_93.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-05-31T19:56:10.274572Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1685563799.098535_267.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-05-31T20:09:59.098535Z', 'realm_epoch': 2}}] 2023-05-31T20:14:50.622 DEBUG:rgw_multi.tests:log_status={0: '1_1685562631.516089_59.1', 1: '', 2: '1_1685562970.274572_93.1', 3: '1_1685563802.468126_296.1'} 2023-05-31T20:14:50.623 DEBUG:rgw_multi.tests:sync_status={0: '1_1685562631.516089_59.1', 1: '', 2: '1_1685562970.274572_93.1', 3: '1_1685563799.098535_267.1'} 2023-05-31T20:14:50.623 WARNING:rgw_multi.tests:zone a2 behind master: shard=3 master=1_1685563802.468126_296.1 target=1_1685563799.098535_267.1 2023-05-31T20:14:55.625 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-31T20:14:55.625 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-31T20:14:55.626 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-05-31T20:14:55.686 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2023-05-31T20:14:55.687 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2023-05-31T20:14:55.732 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:14:55.726+0000 7fd5e28f5780 20 RGW-SYNC:meta: read sync status 2023-05-31T20:14:55.733 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:14:55.727+0000 7fd5e28f5780 20 RGW-SYNC:meta: read sync status 2023-05-31T20:14:55.752 INFO:teuthology.orchestra.run.smithi099.stdout:{ 2023-05-31T20:14:55.753 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_status": { 2023-05-31T20:14:55.753 INFO:teuthology.orchestra.run.smithi099.stdout: "info": { 2023-05-31T20:14:55.753 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "sync", 2023-05-31T20:14:55.754 INFO:teuthology.orchestra.run.smithi099.stdout: "num_shards": 4, 2023-05-31T20:14:55.754 INFO:teuthology.orchestra.run.smithi099.stdout: "period": "2e6123d2-d695-4350-9a01-61a455bc782b", 2023-05-31T20:14:55.754 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2023-05-31T20:14:55.755 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:14:55.755 INFO:teuthology.orchestra.run.smithi099.stdout: "markers": [ 2023-05-31T20:14:55.755 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T20:14:55.755 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 0, 2023-05-31T20:14:55.756 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T20:14:55.756 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2023-05-31T20:14:55.756 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1685562631.516089_59.1", 2023-05-31T20:14:55.757 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T20:14:55.757 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 1, 2023-05-31T20:14:55.757 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T20:14:55.757 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2023-05-31T19:50:31.516089Z", 2023-05-31T20:14:55.758 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2023-05-31T20:14:55.758 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T20:14:55.758 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:14:55.759 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T20:14:55.759 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 1, 2023-05-31T20:14:55.759 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T20:14:55.759 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2023-05-31T20:14:55.760 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "", 2023-05-31T20:14:55.760 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T20:14:55.760 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2023-05-31T20:14:55.761 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T20:14:55.761 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "0.000000", 2023-05-31T20:14:55.761 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2023-05-31T20:14:55.761 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T20:14:55.762 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:14:55.762 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T20:14:55.763 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 2, 2023-05-31T20:14:55.763 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T20:14:55.763 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2023-05-31T20:14:55.763 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1685562970.274572_93.1", 2023-05-31T20:14:55.764 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T20:14:55.764 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2023-05-31T20:14:55.764 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T20:14:55.765 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2023-05-31T19:56:10.274572Z", 2023-05-31T20:14:55.765 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2023-05-31T20:14:55.765 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T20:14:55.766 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:14:55.766 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T20:14:55.766 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 3, 2023-05-31T20:14:55.766 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T20:14:55.766 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2023-05-31T20:14:55.766 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1685563798.815084_250.1", 2023-05-31T20:14:55.767 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T20:14:55.767 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2023-05-31T20:14:55.767 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T20:14:55.768 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2023-05-31T20:09:58.815084Z", 2023-05-31T20:14:55.768 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2023-05-31T20:14:55.768 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T20:14:55.769 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T20:14:55.769 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2023-05-31T20:14:55.769 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:14:55.770 INFO:teuthology.orchestra.run.smithi099.stdout: "full_sync": { 2023-05-31T20:14:55.771 INFO:teuthology.orchestra.run.smithi099.stdout: "total": 1, 2023-05-31T20:14:55.771 INFO:teuthology.orchestra.run.smithi099.stdout: "complete": 1 2023-05-31T20:14:55.772 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:14:55.772 INFO:teuthology.orchestra.run.smithi099.stdout: "current_time": "2023-05-31T20:14:55Z" 2023-05-31T20:14:55.772 INFO:teuthology.orchestra.run.smithi099.stdout:} 2023-05-31T20:14:55.774 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "2e6123d2-d695-4350-9a01-61a455bc782b", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1685562631.516089_59.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-05-31T19:50:31.516089Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1685562970.274572_93.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-05-31T19:56:10.274572Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1685563798.815084_250.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-05-31T20:09:58.815084Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-05-31T20:14:55Z" } 2023-05-31T20:14:55.776 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1685562631.516089_59.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-05-31T19:50:31.516089Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1685562970.274572_93.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-05-31T19:56:10.274572Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1685563798.815084_250.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-05-31T20:09:58.815084Z', 'realm_epoch': 2}}] 2023-05-31T20:14:55.776 DEBUG:rgw_multi.tests:log_status={0: '1_1685562631.516089_59.1', 1: '', 2: '1_1685562970.274572_93.1', 3: '1_1685563802.468126_296.1'} 2023-05-31T20:14:55.777 DEBUG:rgw_multi.tests:sync_status={0: '1_1685562631.516089_59.1', 1: '', 2: '1_1685562970.274572_93.1', 3: '1_1685563798.815084_250.1'} 2023-05-31T20:14:55.777 WARNING:rgw_multi.tests:zone a2 behind master: shard=3 master=1_1685563802.468126_296.1 target=1_1685563798.815084_250.1 2023-05-31T20:15:00.779 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-31T20:15:00.779 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-31T20:15:00.780 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-05-31T20:15:00.832 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2023-05-31T20:15:00.832 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2023-05-31T20:15:00.880 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:15:00.874+0000 7fb53b89f780 20 RGW-SYNC:meta: read sync status 2023-05-31T20:15:00.881 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:15:00.875+0000 7fb53b89f780 20 RGW-SYNC:meta: read sync status 2023-05-31T20:15:00.893 INFO:teuthology.orchestra.run.smithi099.stdout:{ 2023-05-31T20:15:00.893 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_status": { 2023-05-31T20:15:00.894 INFO:teuthology.orchestra.run.smithi099.stdout: "info": { 2023-05-31T20:15:00.894 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "sync", 2023-05-31T20:15:00.894 INFO:teuthology.orchestra.run.smithi099.stdout: "num_shards": 4, 2023-05-31T20:15:00.894 INFO:teuthology.orchestra.run.smithi099.stdout: "period": "2e6123d2-d695-4350-9a01-61a455bc782b", 2023-05-31T20:15:00.894 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2023-05-31T20:15:00.894 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:15:00.894 INFO:teuthology.orchestra.run.smithi099.stdout: "markers": [ 2023-05-31T20:15:00.895 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T20:15:00.895 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 0, 2023-05-31T20:15:00.895 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T20:15:00.895 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2023-05-31T20:15:00.895 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1685562631.516089_59.1", 2023-05-31T20:15:00.895 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T20:15:00.895 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 1, 2023-05-31T20:15:00.896 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T20:15:00.896 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2023-05-31T19:50:31.516089Z", 2023-05-31T20:15:00.896 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2023-05-31T20:15:00.896 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T20:15:00.896 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:15:00.897 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T20:15:00.897 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 1, 2023-05-31T20:15:00.897 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T20:15:00.898 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2023-05-31T20:15:00.898 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "", 2023-05-31T20:15:00.898 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T20:15:00.898 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2023-05-31T20:15:00.898 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T20:15:00.898 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "0.000000", 2023-05-31T20:15:00.899 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2023-05-31T20:15:00.899 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T20:15:00.899 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:15:00.899 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T20:15:00.899 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 2, 2023-05-31T20:15:00.899 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T20:15:00.899 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2023-05-31T20:15:00.900 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1685562970.274572_93.1", 2023-05-31T20:15:00.900 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T20:15:00.900 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2023-05-31T20:15:00.900 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T20:15:00.900 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2023-05-31T19:56:10.274572Z", 2023-05-31T20:15:00.900 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2023-05-31T20:15:00.901 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T20:15:00.901 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:15:00.901 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T20:15:00.901 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 3, 2023-05-31T20:15:00.901 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T20:15:00.901 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2023-05-31T20:15:00.901 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1685563798.827280_252.1", 2023-05-31T20:15:00.902 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T20:15:00.902 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2023-05-31T20:15:00.902 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T20:15:00.902 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2023-05-31T20:09:58.827280Z", 2023-05-31T20:15:00.902 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2023-05-31T20:15:00.902 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T20:15:00.902 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T20:15:00.903 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2023-05-31T20:15:00.903 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:15:00.903 INFO:teuthology.orchestra.run.smithi099.stdout: "full_sync": { 2023-05-31T20:15:00.903 INFO:teuthology.orchestra.run.smithi099.stdout: "total": 1, 2023-05-31T20:15:00.903 INFO:teuthology.orchestra.run.smithi099.stdout: "complete": 1 2023-05-31T20:15:00.903 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:15:00.904 INFO:teuthology.orchestra.run.smithi099.stdout: "current_time": "2023-05-31T20:15:00Z" 2023-05-31T20:15:00.904 INFO:teuthology.orchestra.run.smithi099.stdout:} 2023-05-31T20:15:00.905 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "2e6123d2-d695-4350-9a01-61a455bc782b", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1685562631.516089_59.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-05-31T19:50:31.516089Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1685562970.274572_93.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-05-31T19:56:10.274572Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1685563798.827280_252.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-05-31T20:09:58.827280Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-05-31T20:15:00Z" } 2023-05-31T20:15:00.905 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1685562631.516089_59.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-05-31T19:50:31.516089Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1685562970.274572_93.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-05-31T19:56:10.274572Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1685563798.827280_252.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-05-31T20:09:58.827280Z', 'realm_epoch': 2}}] 2023-05-31T20:15:00.906 DEBUG:rgw_multi.tests:log_status={0: '1_1685562631.516089_59.1', 1: '', 2: '1_1685562970.274572_93.1', 3: '1_1685563802.468126_296.1'} 2023-05-31T20:15:00.906 DEBUG:rgw_multi.tests:sync_status={0: '1_1685562631.516089_59.1', 1: '', 2: '1_1685562970.274572_93.1', 3: '1_1685563798.827280_252.1'} 2023-05-31T20:15:00.906 WARNING:rgw_multi.tests:zone a2 behind master: shard=3 master=1_1685563802.468126_296.1 target=1_1685563798.827280_252.1 2023-05-31T20:15:05.913 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-31T20:15:05.914 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-31T20:15:05.914 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-05-31T20:15:05.972 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2023-05-31T20:15:05.973 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2023-05-31T20:15:06.012 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:15:06.006+0000 7fd05fd93780 20 RGW-SYNC:meta: read sync status 2023-05-31T20:15:06.013 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:15:06.007+0000 7fd05fd93780 20 RGW-SYNC:meta: read sync status 2023-05-31T20:15:06.031 INFO:teuthology.orchestra.run.smithi099.stdout:{ 2023-05-31T20:15:06.031 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_status": { 2023-05-31T20:15:06.032 INFO:teuthology.orchestra.run.smithi099.stdout: "info": { 2023-05-31T20:15:06.032 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "sync", 2023-05-31T20:15:06.032 INFO:teuthology.orchestra.run.smithi099.stdout: "num_shards": 4, 2023-05-31T20:15:06.032 INFO:teuthology.orchestra.run.smithi099.stdout: "period": "2e6123d2-d695-4350-9a01-61a455bc782b", 2023-05-31T20:15:06.032 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2023-05-31T20:15:06.032 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:15:06.033 INFO:teuthology.orchestra.run.smithi099.stdout: "markers": [ 2023-05-31T20:15:06.033 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T20:15:06.033 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 0, 2023-05-31T20:15:06.033 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T20:15:06.033 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2023-05-31T20:15:06.033 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1685562631.516089_59.1", 2023-05-31T20:15:06.034 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T20:15:06.034 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 1, 2023-05-31T20:15:06.034 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T20:15:06.034 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2023-05-31T19:50:31.516089Z", 2023-05-31T20:15:06.034 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2023-05-31T20:15:06.034 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T20:15:06.034 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:15:06.035 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T20:15:06.035 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 1, 2023-05-31T20:15:06.035 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T20:15:06.035 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2023-05-31T20:15:06.035 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "", 2023-05-31T20:15:06.035 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T20:15:06.035 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2023-05-31T20:15:06.036 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T20:15:06.036 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "0.000000", 2023-05-31T20:15:06.036 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2023-05-31T20:15:06.036 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T20:15:06.036 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:15:06.036 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T20:15:06.037 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 2, 2023-05-31T20:15:06.037 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T20:15:06.037 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2023-05-31T20:15:06.037 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1685562970.274572_93.1", 2023-05-31T20:15:06.037 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T20:15:06.037 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2023-05-31T20:15:06.037 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T20:15:06.038 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2023-05-31T19:56:10.274572Z", 2023-05-31T20:15:06.038 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2023-05-31T20:15:06.038 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T20:15:06.038 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:15:06.038 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T20:15:06.042 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 3, 2023-05-31T20:15:06.042 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T20:15:06.042 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2023-05-31T20:15:06.042 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1685563798.760350_247.1", 2023-05-31T20:15:06.042 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T20:15:06.043 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2023-05-31T20:15:06.043 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T20:15:06.043 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2023-05-31T20:09:58.760350Z", 2023-05-31T20:15:06.043 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2023-05-31T20:15:06.043 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T20:15:06.043 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T20:15:06.043 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2023-05-31T20:15:06.044 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:15:06.044 INFO:teuthology.orchestra.run.smithi099.stdout: "full_sync": { 2023-05-31T20:15:06.044 INFO:teuthology.orchestra.run.smithi099.stdout: "total": 1, 2023-05-31T20:15:06.044 INFO:teuthology.orchestra.run.smithi099.stdout: "complete": 1 2023-05-31T20:15:06.044 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:15:06.044 INFO:teuthology.orchestra.run.smithi099.stdout: "current_time": "2023-05-31T20:15:06Z" 2023-05-31T20:15:06.045 INFO:teuthology.orchestra.run.smithi099.stdout:} 2023-05-31T20:15:06.045 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "2e6123d2-d695-4350-9a01-61a455bc782b", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1685562631.516089_59.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-05-31T19:50:31.516089Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1685562970.274572_93.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-05-31T19:56:10.274572Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1685563798.760350_247.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-05-31T20:09:58.760350Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-05-31T20:15:06Z" } 2023-05-31T20:15:06.045 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1685562631.516089_59.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-05-31T19:50:31.516089Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1685562970.274572_93.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-05-31T19:56:10.274572Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1685563798.760350_247.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-05-31T20:09:58.760350Z', 'realm_epoch': 2}}] 2023-05-31T20:15:06.046 DEBUG:rgw_multi.tests:log_status={0: '1_1685562631.516089_59.1', 1: '', 2: '1_1685562970.274572_93.1', 3: '1_1685563802.468126_296.1'} 2023-05-31T20:15:06.046 DEBUG:rgw_multi.tests:sync_status={0: '1_1685562631.516089_59.1', 1: '', 2: '1_1685562970.274572_93.1', 3: '1_1685563798.760350_247.1'} 2023-05-31T20:15:06.046 WARNING:rgw_multi.tests:zone a2 behind master: shard=3 master=1_1685563802.468126_296.1 target=1_1685563798.760350_247.1 2023-05-31T20:15:11.052 INFO:tasks.rgw_multisite_tests:rgw_multi.tests.test_bucket_sync_disable ... FAIL 2023-05-31T20:15:11.054 INFO:rgw_multi.tests:create bucket zone=a1 name=pklxkd-46 2023-05-31T20:15:11.054 DEBUG:boto:path=/pklxkd-46/ 2023-05-31T20:15:11.055 DEBUG:boto:auth_path=/pklxkd-46/ 2023-05-31T20:15:11.056 DEBUG:boto:Method: PUT 2023-05-31T20:15:11.056 DEBUG:boto:Path: /pklxkd-46/ 2023-05-31T20:15:11.057 DEBUG:boto:Data: 2023-05-31T20:15:11.057 DEBUG:boto:Headers: {} 2023-05-31T20:15:11.058 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:15:11.058 DEBUG:boto:Port: 8000 2023-05-31T20:15:11.058 DEBUG:boto:Params: {} 2023-05-31T20:15:11.059 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2023-05-31T20:15:11.060 DEBUG:boto:Token: None 2023-05-31T20:15:11.060 DEBUG:boto:StringToSign: PUT Wed, 31 May 2023 20:15:11 GMT /pklxkd-46/ 2023-05-31T20:15:11.061 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:j0lNbP01hCOmq5XaHB3MuipbZvA= 2023-05-31T20:15:11.061 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:15:11 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:j0lNbP01hCOmq5XaHB3MuipbZvA=', 'Content-Length': '0'} 2023-05-31T20:15:11.088 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000b965852e8a82164c-006477aacf-1630-a1'), ('Content-Length', '0'), ('Date', 'Wed, 31 May 2023 20:15:11 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:15:11.089 INFO:rgw_multi.tests:create bucket zone=a2 name=pklxkd-47 2023-05-31T20:15:11.089 DEBUG:boto:path=/pklxkd-47/ 2023-05-31T20:15:11.090 DEBUG:boto:auth_path=/pklxkd-47/ 2023-05-31T20:15:11.090 DEBUG:boto:Method: PUT 2023-05-31T20:15:11.090 DEBUG:boto:Path: /pklxkd-47/ 2023-05-31T20:15:11.091 DEBUG:boto:Data: 2023-05-31T20:15:11.091 DEBUG:boto:Headers: {} 2023-05-31T20:15:11.091 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:15:11.092 DEBUG:boto:Port: 8001 2023-05-31T20:15:11.092 DEBUG:boto:Params: {} 2023-05-31T20:15:11.093 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8001} 2023-05-31T20:15:11.093 DEBUG:boto:Token: None 2023-05-31T20:15:11.094 DEBUG:boto:StringToSign: PUT Wed, 31 May 2023 20:15:11 GMT /pklxkd-47/ 2023-05-31T20:15:11.094 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:Y+cka98NJhuu2X4aer71rV8kIdY= 2023-05-31T20:15:11.094 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:15:11 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:Y+cka98NJhuu2X4aer71rV8kIdY=', 'Content-Length': '0'} 2023-05-31T20:15:11.188 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx0000050cb146546252f4f-006477aacf-162e-a2'), ('Content-Length', '0'), ('Date', 'Wed, 31 May 2023 20:15:11 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:15:11.189 DEBUG:boto:path=/pklxkd-46/ 2023-05-31T20:15:11.190 DEBUG:boto:auth_path=/pklxkd-46/ 2023-05-31T20:15:11.190 DEBUG:boto:Method: HEAD 2023-05-31T20:15:11.190 DEBUG:boto:Path: /pklxkd-46/ 2023-05-31T20:15:11.191 DEBUG:boto:Data: 2023-05-31T20:15:11.191 DEBUG:boto:Headers: {} 2023-05-31T20:15:11.191 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:15:11.192 DEBUG:boto:Port: 8000 2023-05-31T20:15:11.192 DEBUG:boto:Params: {} 2023-05-31T20:15:11.193 DEBUG:boto:Token: None 2023-05-31T20:15:11.193 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:15:11 GMT /pklxkd-46/ 2023-05-31T20:15:11.194 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:Fs8PHcO+d3Yo/ehTuzGVJkoOlVM= 2023-05-31T20:15:11.194 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:15:11 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:Fs8PHcO+d3Yo/ehTuzGVJkoOlVM=', 'Content-Length': '0'} 2023-05-31T20:15:11.197 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000008232f595b668cece-006477aacf-1630-a1'), ('Content-Length', '0'), ('Date', 'Wed, 31 May 2023 20:15:11 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:15:11.198 DEBUG:boto:path=/pklxkd-46/obj1 2023-05-31T20:15:11.198 DEBUG:boto:auth_path=/pklxkd-46/obj1 2023-05-31T20:15:11.199 DEBUG:boto:Method: PUT 2023-05-31T20:15:11.199 DEBUG:boto:Path: /pklxkd-46/obj1 2023-05-31T20:15:11.199 DEBUG:boto:Data: 2023-05-31T20:15:11.200 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2023-05-31T20:15:11.200 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:15:11.201 DEBUG:boto:Port: 8000 2023-05-31T20:15:11.201 DEBUG:boto:Params: {} 2023-05-31T20:15:11.201 DEBUG:boto:Token: None 2023-05-31T20:15:11.202 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Wed, 31 May 2023 20:15:11 GMT /pklxkd-46/obj1 2023-05-31T20:15:11.203 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:Hx9VyWXTk063/vpxlXt1+FSpCsY= 2023-05-31T20:15:11.203 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Wed, 31 May 2023 20:15:11 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:Hx9VyWXTk063/vpxlXt1+FSpCsY='} 2023-05-31T20:15:11.214 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000ac820fc16bab5092-006477aacf-1630-a1'), ('Date', 'Wed, 31 May 2023 20:15:11 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:15:11.215 DEBUG:boto:path=/pklxkd-46/ 2023-05-31T20:15:11.215 DEBUG:boto:auth_path=/pklxkd-46/ 2023-05-31T20:15:11.216 DEBUG:boto:Method: HEAD 2023-05-31T20:15:11.216 DEBUG:boto:Path: /pklxkd-46/ 2023-05-31T20:15:11.216 DEBUG:boto:Data: 2023-05-31T20:15:11.217 DEBUG:boto:Headers: {} 2023-05-31T20:15:11.217 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:15:11.217 DEBUG:boto:Port: 8000 2023-05-31T20:15:11.218 DEBUG:boto:Params: {} 2023-05-31T20:15:11.218 DEBUG:boto:Token: None 2023-05-31T20:15:11.219 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:15:11 GMT /pklxkd-46/ 2023-05-31T20:15:11.219 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:Fs8PHcO+d3Yo/ehTuzGVJkoOlVM= 2023-05-31T20:15:11.220 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:15:11 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:Fs8PHcO+d3Yo/ehTuzGVJkoOlVM=', 'Content-Length': '0'} 2023-05-31T20:15:11.222 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '6'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000baa42fa4f7da0154-006477aacf-1630-a1'), ('Content-Length', '0'), ('Date', 'Wed, 31 May 2023 20:15:11 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:15:11.223 DEBUG:boto:path=/pklxkd-46/obj2 2023-05-31T20:15:11.224 DEBUG:boto:auth_path=/pklxkd-46/obj2 2023-05-31T20:15:11.224 DEBUG:boto:Method: PUT 2023-05-31T20:15:11.225 DEBUG:boto:Path: /pklxkd-46/obj2 2023-05-31T20:15:11.225 DEBUG:boto:Data: 2023-05-31T20:15:11.225 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2023-05-31T20:15:11.226 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:15:11.226 DEBUG:boto:Port: 8000 2023-05-31T20:15:11.226 DEBUG:boto:Params: {} 2023-05-31T20:15:11.227 DEBUG:boto:Token: None 2023-05-31T20:15:11.227 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Wed, 31 May 2023 20:15:11 GMT /pklxkd-46/obj2 2023-05-31T20:15:11.228 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:gmSPCjFOwdcCAv91DuW0WquddYI= 2023-05-31T20:15:11.228 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Wed, 31 May 2023 20:15:11 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:gmSPCjFOwdcCAv91DuW0WquddYI='} 2023-05-31T20:15:11.239 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000005f110692b14a879a-006477aacf-1630-a1'), ('Date', 'Wed, 31 May 2023 20:15:11 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:15:11.240 DEBUG:boto:path=/pklxkd-46/ 2023-05-31T20:15:11.241 DEBUG:boto:auth_path=/pklxkd-46/ 2023-05-31T20:15:11.241 DEBUG:boto:Method: HEAD 2023-05-31T20:15:11.242 DEBUG:boto:Path: /pklxkd-46/ 2023-05-31T20:15:11.243 DEBUG:boto:Data: 2023-05-31T20:15:11.244 DEBUG:boto:Headers: {} 2023-05-31T20:15:11.245 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:15:11.246 DEBUG:boto:Port: 8000 2023-05-31T20:15:11.247 DEBUG:boto:Params: {} 2023-05-31T20:15:11.248 DEBUG:boto:Token: None 2023-05-31T20:15:11.249 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:15:11 GMT /pklxkd-46/ 2023-05-31T20:15:11.250 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:Fs8PHcO+d3Yo/ehTuzGVJkoOlVM= 2023-05-31T20:15:11.250 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:15:11 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:Fs8PHcO+d3Yo/ehTuzGVJkoOlVM=', 'Content-Length': '0'} 2023-05-31T20:15:11.253 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '2'), ('X-RGW-Bytes-Used', '12'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000f14216bd8ec154e1-006477aacf-1630-a1'), ('Content-Length', '0'), ('Date', 'Wed, 31 May 2023 20:15:11 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:15:11.254 DEBUG:boto:path=/pklxkd-46/obj3 2023-05-31T20:15:11.254 DEBUG:boto:auth_path=/pklxkd-46/obj3 2023-05-31T20:15:11.255 DEBUG:boto:Method: PUT 2023-05-31T20:15:11.255 DEBUG:boto:Path: /pklxkd-46/obj3 2023-05-31T20:15:11.255 DEBUG:boto:Data: 2023-05-31T20:15:11.256 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2023-05-31T20:15:11.256 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:15:11.256 DEBUG:boto:Port: 8000 2023-05-31T20:15:11.257 DEBUG:boto:Params: {} 2023-05-31T20:15:11.257 DEBUG:boto:Token: None 2023-05-31T20:15:11.257 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Wed, 31 May 2023 20:15:11 GMT /pklxkd-46/obj3 2023-05-31T20:15:11.258 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:+AEKyzF0yf9fDw9WWmh99b+ttaI= 2023-05-31T20:15:11.258 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Wed, 31 May 2023 20:15:11 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:+AEKyzF0yf9fDw9WWmh99b+ttaI='} 2023-05-31T20:15:11.301 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000da907dc71e24e2d9-006477aacf-1630-a1'), ('Date', 'Wed, 31 May 2023 20:15:11 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:15:11.302 DEBUG:boto:path=/pklxkd-46/ 2023-05-31T20:15:11.303 DEBUG:boto:auth_path=/pklxkd-46/ 2023-05-31T20:15:11.303 DEBUG:boto:Method: HEAD 2023-05-31T20:15:11.305 DEBUG:boto:Path: /pklxkd-46/ 2023-05-31T20:15:11.305 DEBUG:boto:Data: 2023-05-31T20:15:11.305 DEBUG:boto:Headers: {} 2023-05-31T20:15:11.306 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:15:11.306 DEBUG:boto:Port: 8000 2023-05-31T20:15:11.307 DEBUG:boto:Params: {} 2023-05-31T20:15:11.308 DEBUG:boto:Token: None 2023-05-31T20:15:11.308 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:15:11 GMT /pklxkd-46/ 2023-05-31T20:15:11.309 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:Fs8PHcO+d3Yo/ehTuzGVJkoOlVM= 2023-05-31T20:15:11.310 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:15:11 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:Fs8PHcO+d3Yo/ehTuzGVJkoOlVM=', 'Content-Length': '0'} 2023-05-31T20:15:11.312 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '3'), ('X-RGW-Bytes-Used', '18'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000005459720d2738fbdf-006477aacf-1630-a1'), ('Content-Length', '0'), ('Date', 'Wed, 31 May 2023 20:15:11 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:15:11.313 DEBUG:boto:path=/pklxkd-46/obj4 2023-05-31T20:15:11.314 DEBUG:boto:auth_path=/pklxkd-46/obj4 2023-05-31T20:15:11.314 DEBUG:boto:Method: PUT 2023-05-31T20:15:11.314 DEBUG:boto:Path: /pklxkd-46/obj4 2023-05-31T20:15:11.316 DEBUG:boto:Data: 2023-05-31T20:15:11.317 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2023-05-31T20:15:11.317 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:15:11.317 DEBUG:boto:Port: 8000 2023-05-31T20:15:11.318 DEBUG:boto:Params: {} 2023-05-31T20:15:11.318 DEBUG:boto:Token: None 2023-05-31T20:15:11.319 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Wed, 31 May 2023 20:15:11 GMT /pklxkd-46/obj4 2023-05-31T20:15:11.319 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:/Qggjq1EN/aOewqJ3Qy6+R0ynkQ= 2023-05-31T20:15:11.319 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Wed, 31 May 2023 20:15:11 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:/Qggjq1EN/aOewqJ3Qy6+R0ynkQ='} 2023-05-31T20:15:11.328 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000005720e95f06f4098-006477aacf-1630-a1'), ('Date', 'Wed, 31 May 2023 20:15:11 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:15:11.330 DEBUG:boto:path=/pklxkd-47/ 2023-05-31T20:15:11.330 DEBUG:boto:auth_path=/pklxkd-47/ 2023-05-31T20:15:11.330 DEBUG:boto:Method: HEAD 2023-05-31T20:15:11.331 DEBUG:boto:Path: /pklxkd-47/ 2023-05-31T20:15:11.331 DEBUG:boto:Data: 2023-05-31T20:15:11.331 DEBUG:boto:Headers: {} 2023-05-31T20:15:11.333 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:15:11.333 DEBUG:boto:Port: 8001 2023-05-31T20:15:11.334 DEBUG:boto:Params: {} 2023-05-31T20:15:11.335 DEBUG:boto:Token: None 2023-05-31T20:15:11.335 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:15:11 GMT /pklxkd-47/ 2023-05-31T20:15:11.336 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:PuHda4Ncgw7MHKHf5tEStHii+oY= 2023-05-31T20:15:11.338 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:15:11 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:PuHda4Ncgw7MHKHf5tEStHii+oY=', 'Content-Length': '0'} 2023-05-31T20:15:11.340 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000000b9819e82f39829a-006477aacf-162e-a2'), ('Content-Length', '0'), ('Date', 'Wed, 31 May 2023 20:15:11 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:15:11.341 DEBUG:boto:path=/pklxkd-47/obj1 2023-05-31T20:15:11.342 DEBUG:boto:auth_path=/pklxkd-47/obj1 2023-05-31T20:15:11.342 DEBUG:boto:Method: PUT 2023-05-31T20:15:11.344 DEBUG:boto:Path: /pklxkd-47/obj1 2023-05-31T20:15:11.344 DEBUG:boto:Data: 2023-05-31T20:15:11.345 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2023-05-31T20:15:11.345 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:15:11.345 DEBUG:boto:Port: 8001 2023-05-31T20:15:11.346 DEBUG:boto:Params: {} 2023-05-31T20:15:11.346 DEBUG:boto:Token: None 2023-05-31T20:15:11.347 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Wed, 31 May 2023 20:15:11 GMT /pklxkd-47/obj1 2023-05-31T20:15:11.347 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:yLT61qZZmGDQGRitHaqAZGTnai0= 2023-05-31T20:15:11.348 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Wed, 31 May 2023 20:15:11 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:yLT61qZZmGDQGRitHaqAZGTnai0='} 2023-05-31T20:15:11.355 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000002f0869dd669ba611-006477aacf-162e-a2'), ('Date', 'Wed, 31 May 2023 20:15:11 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:15:11.355 DEBUG:boto:path=/pklxkd-47/ 2023-05-31T20:15:11.356 DEBUG:boto:auth_path=/pklxkd-47/ 2023-05-31T20:15:11.356 DEBUG:boto:Method: HEAD 2023-05-31T20:15:11.357 DEBUG:boto:Path: /pklxkd-47/ 2023-05-31T20:15:11.357 DEBUG:boto:Data: 2023-05-31T20:15:11.357 DEBUG:boto:Headers: {} 2023-05-31T20:15:11.358 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:15:11.358 DEBUG:boto:Port: 8001 2023-05-31T20:15:11.359 DEBUG:boto:Params: {} 2023-05-31T20:15:11.359 DEBUG:boto:Token: None 2023-05-31T20:15:11.360 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:15:11 GMT /pklxkd-47/ 2023-05-31T20:15:11.360 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:PuHda4Ncgw7MHKHf5tEStHii+oY= 2023-05-31T20:15:11.361 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:15:11 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:PuHda4Ncgw7MHKHf5tEStHii+oY=', 'Content-Length': '0'} 2023-05-31T20:15:11.363 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '6'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000abc224197e9a393d-006477aacf-162e-a2'), ('Content-Length', '0'), ('Date', 'Wed, 31 May 2023 20:15:11 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:15:11.364 DEBUG:boto:path=/pklxkd-47/obj2 2023-05-31T20:15:11.364 DEBUG:boto:auth_path=/pklxkd-47/obj2 2023-05-31T20:15:11.365 DEBUG:boto:Method: PUT 2023-05-31T20:15:11.365 DEBUG:boto:Path: /pklxkd-47/obj2 2023-05-31T20:15:11.365 DEBUG:boto:Data: 2023-05-31T20:15:11.366 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2023-05-31T20:15:11.366 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:15:11.366 DEBUG:boto:Port: 8001 2023-05-31T20:15:11.367 DEBUG:boto:Params: {} 2023-05-31T20:15:11.368 DEBUG:boto:Token: None 2023-05-31T20:15:11.369 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Wed, 31 May 2023 20:15:11 GMT /pklxkd-47/obj2 2023-05-31T20:15:11.370 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:lQ9CFb0fN6Ny+VBePH6cmPCX8U4= 2023-05-31T20:15:11.370 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Wed, 31 May 2023 20:15:11 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:lQ9CFb0fN6Ny+VBePH6cmPCX8U4='} 2023-05-31T20:15:11.384 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000aa710b961d9dd760-006477aacf-162e-a2'), ('Date', 'Wed, 31 May 2023 20:15:11 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:15:11.385 DEBUG:boto:path=/pklxkd-47/ 2023-05-31T20:15:11.385 DEBUG:boto:auth_path=/pklxkd-47/ 2023-05-31T20:15:11.386 DEBUG:boto:Method: HEAD 2023-05-31T20:15:11.386 DEBUG:boto:Path: /pklxkd-47/ 2023-05-31T20:15:11.387 DEBUG:boto:Data: 2023-05-31T20:15:11.387 DEBUG:boto:Headers: {} 2023-05-31T20:15:11.387 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:15:11.388 DEBUG:boto:Port: 8001 2023-05-31T20:15:11.388 DEBUG:boto:Params: {} 2023-05-31T20:15:11.388 DEBUG:boto:Token: None 2023-05-31T20:15:11.389 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:15:11 GMT /pklxkd-47/ 2023-05-31T20:15:11.389 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:PuHda4Ncgw7MHKHf5tEStHii+oY= 2023-05-31T20:15:11.390 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:15:11 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:PuHda4Ncgw7MHKHf5tEStHii+oY=', 'Content-Length': '0'} 2023-05-31T20:15:11.394 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '2'), ('X-RGW-Bytes-Used', '12'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000c542302763c93b18-006477aacf-162e-a2'), ('Content-Length', '0'), ('Date', 'Wed, 31 May 2023 20:15:11 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:15:11.395 DEBUG:boto:path=/pklxkd-47/obj3 2023-05-31T20:15:11.396 DEBUG:boto:auth_path=/pklxkd-47/obj3 2023-05-31T20:15:11.397 DEBUG:boto:Method: PUT 2023-05-31T20:15:11.398 DEBUG:boto:Path: /pklxkd-47/obj3 2023-05-31T20:15:11.398 DEBUG:boto:Data: 2023-05-31T20:15:11.399 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2023-05-31T20:15:11.400 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:15:11.400 DEBUG:boto:Port: 8001 2023-05-31T20:15:11.401 DEBUG:boto:Params: {} 2023-05-31T20:15:11.402 DEBUG:boto:Token: None 2023-05-31T20:15:11.403 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Wed, 31 May 2023 20:15:11 GMT /pklxkd-47/obj3 2023-05-31T20:15:11.403 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:VmBXW42ukB90hEu38wgqSx/4CPg= 2023-05-31T20:15:11.404 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Wed, 31 May 2023 20:15:11 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:VmBXW42ukB90hEu38wgqSx/4CPg='} 2023-05-31T20:15:11.448 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000d1c4f1466a683989-006477aacf-162e-a2'), ('Date', 'Wed, 31 May 2023 20:15:11 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:15:11.450 DEBUG:boto:path=/pklxkd-47/ 2023-05-31T20:15:11.451 DEBUG:boto:auth_path=/pklxkd-47/ 2023-05-31T20:15:11.452 DEBUG:boto:Method: HEAD 2023-05-31T20:15:11.453 DEBUG:boto:Path: /pklxkd-47/ 2023-05-31T20:15:11.454 DEBUG:boto:Data: 2023-05-31T20:15:11.455 DEBUG:boto:Headers: {} 2023-05-31T20:15:11.456 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:15:11.456 DEBUG:boto:Port: 8001 2023-05-31T20:15:11.457 DEBUG:boto:Params: {} 2023-05-31T20:15:11.458 DEBUG:boto:Token: None 2023-05-31T20:15:11.459 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:15:11 GMT /pklxkd-47/ 2023-05-31T20:15:11.459 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:PuHda4Ncgw7MHKHf5tEStHii+oY= 2023-05-31T20:15:11.460 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:15:11 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:PuHda4Ncgw7MHKHf5tEStHii+oY=', 'Content-Length': '0'} 2023-05-31T20:15:11.462 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '3'), ('X-RGW-Bytes-Used', '18'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000009dd3f9b91814137-006477aacf-162e-a2'), ('Content-Length', '0'), ('Date', 'Wed, 31 May 2023 20:15:11 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:15:11.463 DEBUG:boto:path=/pklxkd-47/obj4 2023-05-31T20:15:11.463 DEBUG:boto:auth_path=/pklxkd-47/obj4 2023-05-31T20:15:11.463 DEBUG:boto:Method: PUT 2023-05-31T20:15:11.464 DEBUG:boto:Path: /pklxkd-47/obj4 2023-05-31T20:15:11.465 DEBUG:boto:Data: 2023-05-31T20:15:11.466 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2023-05-31T20:15:11.466 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:15:11.467 DEBUG:boto:Port: 8001 2023-05-31T20:15:11.468 DEBUG:boto:Params: {} 2023-05-31T20:15:11.469 DEBUG:boto:Token: None 2023-05-31T20:15:11.470 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Wed, 31 May 2023 20:15:11 GMT /pklxkd-47/obj4 2023-05-31T20:15:11.471 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:hrkOXgUjYTheqcylNx+5I35h7Pc= 2023-05-31T20:15:11.472 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Wed, 31 May 2023 20:15:11 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:hrkOXgUjYTheqcylNx+5I35h7Pc='} 2023-05-31T20:15:11.493 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000c1d003ad7805ee4b-006477aacf-162e-a2'), ('Date', 'Wed, 31 May 2023 20:15:11 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:15:11.495 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-31T20:15:11.495 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-31T20:15:11.496 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-05-31T20:15:11.544 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2023-05-31T20:15:11.544 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2023-05-31T20:15:11.573 INFO:teuthology.orchestra.run.smithi099.stderr:No --period given, using current period=2e6123d2-d695-4350-9a01-61a455bc782b 2023-05-31T20:15:11.585 INFO:teuthology.orchestra.run.smithi099.stdout:[ 2023-05-31T20:15:11.586 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T20:15:11.586 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1685562631.516089_59.1", 2023-05-31T20:15:11.586 INFO:teuthology.orchestra.run.smithi099.stdout: "last_update": "2023-05-31T19:50:31.516089Z" 2023-05-31T20:15:11.586 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:15:11.587 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T20:15:11.587 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "", 2023-05-31T20:15:11.587 INFO:teuthology.orchestra.run.smithi099.stdout: "last_update": "0.000000" 2023-05-31T20:15:11.587 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:15:11.587 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T20:15:11.587 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1685562970.274572_93.1", 2023-05-31T20:15:11.587 INFO:teuthology.orchestra.run.smithi099.stdout: "last_update": "2023-05-31T19:56:10.274572Z" 2023-05-31T20:15:11.588 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:15:11.588 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T20:15:11.588 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1685564111.129168_304.1", 2023-05-31T20:15:11.588 INFO:teuthology.orchestra.run.smithi099.stdout: "last_update": "2023-05-31T20:15:11.129168Z" 2023-05-31T20:15:11.588 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T20:15:11.588 INFO:teuthology.orchestra.run.smithi099.stdout:] 2023-05-31T20:15:11.589 DEBUG:rgw_multi.tests:master meta markers={0: '1_1685562631.516089_59.1', 1: '', 2: '1_1685562970.274572_93.1', 3: '1_1685564111.129168_304.1'} 2023-05-31T20:15:11.590 INFO:rgw_multi.tests:starting meta checkpoint for zone=a2 2023-05-31T20:15:11.590 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-31T20:15:11.591 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-31T20:15:11.592 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-05-31T20:15:11.640 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2023-05-31T20:15:11.641 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2023-05-31T20:15:11.671 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:15:11.665+0000 7f8e20ec9780 20 RGW-SYNC:meta: read sync status 2023-05-31T20:15:11.672 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:15:11.666+0000 7f8e20ec9780 20 RGW-SYNC:meta: read sync status 2023-05-31T20:15:11.684 INFO:teuthology.orchestra.run.smithi099.stdout:{ 2023-05-31T20:15:11.684 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_status": { 2023-05-31T20:15:11.685 INFO:teuthology.orchestra.run.smithi099.stdout: "info": { 2023-05-31T20:15:11.685 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "sync", 2023-05-31T20:15:11.685 INFO:teuthology.orchestra.run.smithi099.stdout: "num_shards": 4, 2023-05-31T20:15:11.685 INFO:teuthology.orchestra.run.smithi099.stdout: "period": "2e6123d2-d695-4350-9a01-61a455bc782b", 2023-05-31T20:15:11.685 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2023-05-31T20:15:11.685 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:15:11.686 INFO:teuthology.orchestra.run.smithi099.stdout: "markers": [ 2023-05-31T20:15:11.686 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T20:15:11.686 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 0, 2023-05-31T20:15:11.686 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T20:15:11.686 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2023-05-31T20:15:11.686 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1685562631.516089_59.1", 2023-05-31T20:15:11.686 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T20:15:11.687 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 1, 2023-05-31T20:15:11.687 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T20:15:11.687 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2023-05-31T19:50:31.516089Z", 2023-05-31T20:15:11.687 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2023-05-31T20:15:11.687 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T20:15:11.687 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:15:11.687 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T20:15:11.688 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 1, 2023-05-31T20:15:11.688 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T20:15:11.688 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2023-05-31T20:15:11.688 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "", 2023-05-31T20:15:11.688 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T20:15:11.688 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2023-05-31T20:15:11.688 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T20:15:11.689 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "0.000000", 2023-05-31T20:15:11.689 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2023-05-31T20:15:11.689 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T20:15:11.689 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:15:11.689 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T20:15:11.689 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 2, 2023-05-31T20:15:11.689 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T20:15:11.690 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2023-05-31T20:15:11.690 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1685562970.274572_93.1", 2023-05-31T20:15:11.690 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T20:15:11.690 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2023-05-31T20:15:11.690 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T20:15:11.690 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2023-05-31T19:56:10.274572Z", 2023-05-31T20:15:11.690 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2023-05-31T20:15:11.691 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T20:15:11.691 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:15:11.691 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T20:15:11.691 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 3, 2023-05-31T20:15:11.691 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T20:15:11.691 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2023-05-31T20:15:11.692 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1685563798.885419_253.1", 2023-05-31T20:15:11.692 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T20:15:11.692 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2023-05-31T20:15:11.692 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T20:15:11.692 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2023-05-31T20:09:58.885419Z", 2023-05-31T20:15:11.692 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2023-05-31T20:15:11.693 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T20:15:11.693 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T20:15:11.693 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2023-05-31T20:15:11.693 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:15:11.693 INFO:teuthology.orchestra.run.smithi099.stdout: "full_sync": { 2023-05-31T20:15:11.693 INFO:teuthology.orchestra.run.smithi099.stdout: "total": 1, 2023-05-31T20:15:11.693 INFO:teuthology.orchestra.run.smithi099.stdout: "complete": 1 2023-05-31T20:15:11.694 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:15:11.694 INFO:teuthology.orchestra.run.smithi099.stdout: "current_time": "2023-05-31T20:15:11Z" 2023-05-31T20:15:11.694 INFO:teuthology.orchestra.run.smithi099.stdout:} 2023-05-31T20:15:11.695 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "2e6123d2-d695-4350-9a01-61a455bc782b", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1685562631.516089_59.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-05-31T19:50:31.516089Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1685562970.274572_93.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-05-31T19:56:10.274572Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1685563798.885419_253.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-05-31T20:09:58.885419Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-05-31T20:15:11Z" } 2023-05-31T20:15:11.695 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1685562631.516089_59.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-05-31T19:50:31.516089Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1685562970.274572_93.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-05-31T19:56:10.274572Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1685563798.885419_253.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-05-31T20:09:58.885419Z', 'realm_epoch': 2}}] 2023-05-31T20:15:11.696 DEBUG:rgw_multi.tests:log_status={0: '1_1685562631.516089_59.1', 1: '', 2: '1_1685562970.274572_93.1', 3: '1_1685564111.129168_304.1'} 2023-05-31T20:15:11.696 DEBUG:rgw_multi.tests:sync_status={0: '1_1685562631.516089_59.1', 1: '', 2: '1_1685562970.274572_93.1', 3: '1_1685563798.885419_253.1'} 2023-05-31T20:15:11.697 WARNING:rgw_multi.tests:zone a2 behind master: shard=3 master=1_1685564111.129168_304.1 target=1_1685563798.885419_253.1 2023-05-31T20:15:16.698 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-31T20:15:16.699 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-31T20:15:16.699 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-05-31T20:15:16.750 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2023-05-31T20:15:16.751 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2023-05-31T20:15:16.796 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:15:16.791+0000 7fb4bda9e780 20 RGW-SYNC:meta: read sync status 2023-05-31T20:15:16.796 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:15:16.792+0000 7fb4bda9e780 20 RGW-SYNC:meta: read sync status 2023-05-31T20:15:16.810 INFO:teuthology.orchestra.run.smithi099.stdout:{ 2023-05-31T20:15:16.810 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_status": { 2023-05-31T20:15:16.810 INFO:teuthology.orchestra.run.smithi099.stdout: "info": { 2023-05-31T20:15:16.810 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "sync", 2023-05-31T20:15:16.811 INFO:teuthology.orchestra.run.smithi099.stdout: "num_shards": 4, 2023-05-31T20:15:16.811 INFO:teuthology.orchestra.run.smithi099.stdout: "period": "2e6123d2-d695-4350-9a01-61a455bc782b", 2023-05-31T20:15:16.811 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2023-05-31T20:15:16.811 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:15:16.811 INFO:teuthology.orchestra.run.smithi099.stdout: "markers": [ 2023-05-31T20:15:16.811 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T20:15:16.811 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 0, 2023-05-31T20:15:16.812 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T20:15:16.812 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2023-05-31T20:15:16.812 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1685562631.516089_59.1", 2023-05-31T20:15:16.812 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T20:15:16.812 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 1, 2023-05-31T20:15:16.812 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T20:15:16.812 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2023-05-31T19:50:31.516089Z", 2023-05-31T20:15:16.813 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2023-05-31T20:15:16.813 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T20:15:16.813 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:15:16.813 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T20:15:16.813 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 1, 2023-05-31T20:15:16.813 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T20:15:16.814 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2023-05-31T20:15:16.814 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "", 2023-05-31T20:15:16.814 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T20:15:16.814 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2023-05-31T20:15:16.814 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T20:15:16.814 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "0.000000", 2023-05-31T20:15:16.814 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2023-05-31T20:15:16.815 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T20:15:16.815 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:15:16.815 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T20:15:16.815 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 2, 2023-05-31T20:15:16.815 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T20:15:16.815 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2023-05-31T20:15:16.815 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1685562970.274572_93.1", 2023-05-31T20:15:16.816 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T20:15:16.816 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2023-05-31T20:15:16.816 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T20:15:16.816 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2023-05-31T19:56:10.274572Z", 2023-05-31T20:15:16.816 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2023-05-31T20:15:16.816 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T20:15:16.816 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:15:16.817 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T20:15:16.817 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 3, 2023-05-31T20:15:16.817 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T20:15:16.817 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2023-05-31T20:15:16.817 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1685563798.915243_256.1", 2023-05-31T20:15:16.817 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T20:15:16.822 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2023-05-31T20:15:16.822 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T20:15:16.823 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2023-05-31T20:09:58.915243Z", 2023-05-31T20:15:16.823 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2023-05-31T20:15:16.823 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T20:15:16.823 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T20:15:16.823 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2023-05-31T20:15:16.824 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:15:16.824 INFO:teuthology.orchestra.run.smithi099.stdout: "full_sync": { 2023-05-31T20:15:16.824 INFO:teuthology.orchestra.run.smithi099.stdout: "total": 1, 2023-05-31T20:15:16.824 INFO:teuthology.orchestra.run.smithi099.stdout: "complete": 1 2023-05-31T20:15:16.824 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:15:16.824 INFO:teuthology.orchestra.run.smithi099.stdout: "current_time": "2023-05-31T20:15:16Z" 2023-05-31T20:15:16.825 INFO:teuthology.orchestra.run.smithi099.stdout:} 2023-05-31T20:15:16.825 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "2e6123d2-d695-4350-9a01-61a455bc782b", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1685562631.516089_59.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-05-31T19:50:31.516089Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1685562970.274572_93.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-05-31T19:56:10.274572Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1685563798.915243_256.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-05-31T20:09:58.915243Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-05-31T20:15:16Z" } 2023-05-31T20:15:16.826 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1685562631.516089_59.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-05-31T19:50:31.516089Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1685562970.274572_93.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-05-31T19:56:10.274572Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1685563798.915243_256.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-05-31T20:09:58.915243Z', 'realm_epoch': 2}}] 2023-05-31T20:15:16.827 DEBUG:rgw_multi.tests:log_status={0: '1_1685562631.516089_59.1', 1: '', 2: '1_1685562970.274572_93.1', 3: '1_1685564111.129168_304.1'} 2023-05-31T20:15:16.827 DEBUG:rgw_multi.tests:sync_status={0: '1_1685562631.516089_59.1', 1: '', 2: '1_1685562970.274572_93.1', 3: '1_1685563798.915243_256.1'} 2023-05-31T20:15:16.827 WARNING:rgw_multi.tests:zone a2 behind master: shard=3 master=1_1685564111.129168_304.1 target=1_1685563798.915243_256.1 2023-05-31T20:15:21.828 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-31T20:15:21.829 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-31T20:15:21.830 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-05-31T20:15:21.913 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2023-05-31T20:15:21.913 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2023-05-31T20:15:21.966 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:15:21.962+0000 7f12314a6780 20 RGW-SYNC:meta: read sync status 2023-05-31T20:15:21.967 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:15:21.962+0000 7f12314a6780 20 RGW-SYNC:meta: read sync status 2023-05-31T20:15:21.982 INFO:teuthology.orchestra.run.smithi099.stdout:{ 2023-05-31T20:15:21.982 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_status": { 2023-05-31T20:15:21.982 INFO:teuthology.orchestra.run.smithi099.stdout: "info": { 2023-05-31T20:15:21.982 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "sync", 2023-05-31T20:15:21.983 INFO:teuthology.orchestra.run.smithi099.stdout: "num_shards": 4, 2023-05-31T20:15:21.983 INFO:teuthology.orchestra.run.smithi099.stdout: "period": "2e6123d2-d695-4350-9a01-61a455bc782b", 2023-05-31T20:15:21.983 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2023-05-31T20:15:21.983 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:15:21.983 INFO:teuthology.orchestra.run.smithi099.stdout: "markers": [ 2023-05-31T20:15:21.983 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T20:15:21.984 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 0, 2023-05-31T20:15:21.984 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T20:15:21.984 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2023-05-31T20:15:21.984 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1685562631.516089_59.1", 2023-05-31T20:15:21.984 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T20:15:21.984 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 1, 2023-05-31T20:15:21.984 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T20:15:21.985 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2023-05-31T19:50:31.516089Z", 2023-05-31T20:15:21.985 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2023-05-31T20:15:21.985 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T20:15:21.985 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:15:21.985 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T20:15:21.985 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 1, 2023-05-31T20:15:21.986 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T20:15:21.986 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2023-05-31T20:15:21.986 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "", 2023-05-31T20:15:21.986 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T20:15:21.986 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2023-05-31T20:15:21.987 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T20:15:21.987 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "0.000000", 2023-05-31T20:15:21.987 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2023-05-31T20:15:21.987 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T20:15:21.987 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:15:21.987 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T20:15:21.987 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 2, 2023-05-31T20:15:21.988 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T20:15:21.988 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2023-05-31T20:15:21.988 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1685562970.274572_93.1", 2023-05-31T20:15:21.988 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T20:15:21.988 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2023-05-31T20:15:21.988 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T20:15:21.989 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2023-05-31T19:56:10.274572Z", 2023-05-31T20:15:21.989 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2023-05-31T20:15:21.989 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T20:15:21.989 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:15:21.989 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T20:15:21.989 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 3, 2023-05-31T20:15:21.989 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T20:15:21.990 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2023-05-31T20:15:21.990 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1685563798.827280_252.1", 2023-05-31T20:15:21.990 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T20:15:21.990 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2023-05-31T20:15:21.990 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T20:15:21.990 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2023-05-31T20:09:58.827280Z", 2023-05-31T20:15:21.991 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2023-05-31T20:15:21.991 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T20:15:21.991 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T20:15:21.991 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2023-05-31T20:15:21.991 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:15:21.991 INFO:teuthology.orchestra.run.smithi099.stdout: "full_sync": { 2023-05-31T20:15:21.991 INFO:teuthology.orchestra.run.smithi099.stdout: "total": 1, 2023-05-31T20:15:21.992 INFO:teuthology.orchestra.run.smithi099.stdout: "complete": 1 2023-05-31T20:15:21.993 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:15:21.993 INFO:teuthology.orchestra.run.smithi099.stdout: "current_time": "2023-05-31T20:15:21Z" 2023-05-31T20:15:21.993 INFO:teuthology.orchestra.run.smithi099.stdout:} 2023-05-31T20:15:21.995 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "2e6123d2-d695-4350-9a01-61a455bc782b", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1685562631.516089_59.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-05-31T19:50:31.516089Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1685562970.274572_93.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-05-31T19:56:10.274572Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1685563798.827280_252.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-05-31T20:09:58.827280Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-05-31T20:15:21Z" } 2023-05-31T20:15:21.996 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1685562631.516089_59.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-05-31T19:50:31.516089Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1685562970.274572_93.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-05-31T19:56:10.274572Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1685563798.827280_252.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-05-31T20:09:58.827280Z', 'realm_epoch': 2}}] 2023-05-31T20:15:21.997 DEBUG:rgw_multi.tests:log_status={0: '1_1685562631.516089_59.1', 1: '', 2: '1_1685562970.274572_93.1', 3: '1_1685564111.129168_304.1'} 2023-05-31T20:15:21.997 DEBUG:rgw_multi.tests:sync_status={0: '1_1685562631.516089_59.1', 1: '', 2: '1_1685562970.274572_93.1', 3: '1_1685563798.827280_252.1'} 2023-05-31T20:15:21.998 WARNING:rgw_multi.tests:zone a2 behind master: shard=3 master=1_1685564111.129168_304.1 target=1_1685563798.827280_252.1 2023-05-31T20:15:26.999 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-31T20:15:27.000 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-31T20:15:27.001 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-05-31T20:15:27.083 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2023-05-31T20:15:27.084 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2023-05-31T20:15:27.123 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:15:27.118+0000 7fcf03b9b780 20 RGW-SYNC:meta: read sync status 2023-05-31T20:15:27.123 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:15:27.119+0000 7fcf03b9b780 20 RGW-SYNC:meta: read sync status 2023-05-31T20:15:27.136 INFO:teuthology.orchestra.run.smithi099.stdout:{ 2023-05-31T20:15:27.136 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_status": { 2023-05-31T20:15:27.137 INFO:teuthology.orchestra.run.smithi099.stdout: "info": { 2023-05-31T20:15:27.137 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "sync", 2023-05-31T20:15:27.137 INFO:teuthology.orchestra.run.smithi099.stdout: "num_shards": 4, 2023-05-31T20:15:27.137 INFO:teuthology.orchestra.run.smithi099.stdout: "period": "2e6123d2-d695-4350-9a01-61a455bc782b", 2023-05-31T20:15:27.137 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2023-05-31T20:15:27.137 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:15:27.138 INFO:teuthology.orchestra.run.smithi099.stdout: "markers": [ 2023-05-31T20:15:27.138 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T20:15:27.138 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 0, 2023-05-31T20:15:27.138 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T20:15:27.138 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2023-05-31T20:15:27.138 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1685562631.516089_59.1", 2023-05-31T20:15:27.138 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T20:15:27.139 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 1, 2023-05-31T20:15:27.139 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T20:15:27.139 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2023-05-31T19:50:31.516089Z", 2023-05-31T20:15:27.139 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2023-05-31T20:15:27.139 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T20:15:27.139 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:15:27.139 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T20:15:27.140 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 1, 2023-05-31T20:15:27.140 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T20:15:27.141 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2023-05-31T20:15:27.141 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "", 2023-05-31T20:15:27.141 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T20:15:27.141 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2023-05-31T20:15:27.141 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T20:15:27.141 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "0.000000", 2023-05-31T20:15:27.142 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2023-05-31T20:15:27.142 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T20:15:27.142 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:15:27.142 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T20:15:27.142 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 2, 2023-05-31T20:15:27.142 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T20:15:27.142 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2023-05-31T20:15:27.143 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1685562970.274572_93.1", 2023-05-31T20:15:27.143 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T20:15:27.143 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2023-05-31T20:15:27.143 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T20:15:27.143 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2023-05-31T19:56:10.274572Z", 2023-05-31T20:15:27.143 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2023-05-31T20:15:27.143 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T20:15:27.144 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:15:27.144 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T20:15:27.144 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 3, 2023-05-31T20:15:27.144 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T20:15:27.144 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2023-05-31T20:15:27.144 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1685563798.970303_257.1", 2023-05-31T20:15:27.145 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T20:15:27.145 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2023-05-31T20:15:27.145 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T20:15:27.145 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2023-05-31T20:09:58.970303Z", 2023-05-31T20:15:27.145 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2023-05-31T20:15:27.145 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T20:15:27.145 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T20:15:27.146 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2023-05-31T20:15:27.146 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:15:27.146 INFO:teuthology.orchestra.run.smithi099.stdout: "full_sync": { 2023-05-31T20:15:27.146 INFO:teuthology.orchestra.run.smithi099.stdout: "total": 1, 2023-05-31T20:15:27.146 INFO:teuthology.orchestra.run.smithi099.stdout: "complete": 1 2023-05-31T20:15:27.146 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:15:27.146 INFO:teuthology.orchestra.run.smithi099.stdout: "current_time": "2023-05-31T20:15:27Z" 2023-05-31T20:15:27.147 INFO:teuthology.orchestra.run.smithi099.stdout:} 2023-05-31T20:15:27.148 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "2e6123d2-d695-4350-9a01-61a455bc782b", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1685562631.516089_59.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-05-31T19:50:31.516089Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1685562970.274572_93.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-05-31T19:56:10.274572Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1685563798.970303_257.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-05-31T20:09:58.970303Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-05-31T20:15:27Z" } 2023-05-31T20:15:27.148 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1685562631.516089_59.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-05-31T19:50:31.516089Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1685562970.274572_93.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-05-31T19:56:10.274572Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1685563798.970303_257.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-05-31T20:09:58.970303Z', 'realm_epoch': 2}}] 2023-05-31T20:15:27.149 DEBUG:rgw_multi.tests:log_status={0: '1_1685562631.516089_59.1', 1: '', 2: '1_1685562970.274572_93.1', 3: '1_1685564111.129168_304.1'} 2023-05-31T20:15:27.149 DEBUG:rgw_multi.tests:sync_status={0: '1_1685562631.516089_59.1', 1: '', 2: '1_1685562970.274572_93.1', 3: '1_1685563798.970303_257.1'} 2023-05-31T20:15:27.150 WARNING:rgw_multi.tests:zone a2 behind master: shard=3 master=1_1685564111.129168_304.1 target=1_1685563798.970303_257.1 2023-05-31T20:15:32.152 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-31T20:15:32.152 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-31T20:15:32.153 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-05-31T20:15:32.210 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2023-05-31T20:15:32.211 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2023-05-31T20:15:32.258 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:15:32.253+0000 7f8aaee69780 20 RGW-SYNC:meta: read sync status 2023-05-31T20:15:32.259 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:15:32.254+0000 7f8aaee69780 20 RGW-SYNC:meta: read sync status 2023-05-31T20:15:32.279 INFO:teuthology.orchestra.run.smithi099.stdout:{ 2023-05-31T20:15:32.279 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_status": { 2023-05-31T20:15:32.279 INFO:teuthology.orchestra.run.smithi099.stdout: "info": { 2023-05-31T20:15:32.280 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "sync", 2023-05-31T20:15:32.280 INFO:teuthology.orchestra.run.smithi099.stdout: "num_shards": 4, 2023-05-31T20:15:32.280 INFO:teuthology.orchestra.run.smithi099.stdout: "period": "2e6123d2-d695-4350-9a01-61a455bc782b", 2023-05-31T20:15:32.280 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2023-05-31T20:15:32.280 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:15:32.280 INFO:teuthology.orchestra.run.smithi099.stdout: "markers": [ 2023-05-31T20:15:32.281 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T20:15:32.281 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 0, 2023-05-31T20:15:32.281 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T20:15:32.281 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2023-05-31T20:15:32.281 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1685562631.516089_59.1", 2023-05-31T20:15:32.281 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T20:15:32.282 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 1, 2023-05-31T20:15:32.282 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T20:15:32.282 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2023-05-31T19:50:31.516089Z", 2023-05-31T20:15:32.282 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2023-05-31T20:15:32.282 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T20:15:32.282 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:15:32.282 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T20:15:32.283 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 1, 2023-05-31T20:15:32.283 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T20:15:32.283 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2023-05-31T20:15:32.283 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "", 2023-05-31T20:15:32.283 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T20:15:32.283 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2023-05-31T20:15:32.283 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T20:15:32.284 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "0.000000", 2023-05-31T20:15:32.284 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2023-05-31T20:15:32.284 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T20:15:32.284 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:15:32.284 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T20:15:32.284 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 2, 2023-05-31T20:15:32.284 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T20:15:32.285 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2023-05-31T20:15:32.285 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1685562970.274572_93.1", 2023-05-31T20:15:32.285 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T20:15:32.285 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2023-05-31T20:15:32.285 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T20:15:32.286 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2023-05-31T19:56:10.274572Z", 2023-05-31T20:15:32.286 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2023-05-31T20:15:32.286 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T20:15:32.286 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:15:32.286 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T20:15:32.286 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 3, 2023-05-31T20:15:32.286 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T20:15:32.287 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2023-05-31T20:15:32.287 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1685563798.756239_246.1", 2023-05-31T20:15:32.287 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T20:15:32.287 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2023-05-31T20:15:32.288 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T20:15:32.288 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2023-05-31T20:09:58.756239Z", 2023-05-31T20:15:32.288 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2023-05-31T20:15:32.288 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T20:15:32.288 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T20:15:32.289 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2023-05-31T20:15:32.289 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:15:32.289 INFO:teuthology.orchestra.run.smithi099.stdout: "full_sync": { 2023-05-31T20:15:32.289 INFO:teuthology.orchestra.run.smithi099.stdout: "total": 1, 2023-05-31T20:15:32.289 INFO:teuthology.orchestra.run.smithi099.stdout: "complete": 1 2023-05-31T20:15:32.290 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:15:32.290 INFO:teuthology.orchestra.run.smithi099.stdout: "current_time": "2023-05-31T20:15:32Z" 2023-05-31T20:15:32.290 INFO:teuthology.orchestra.run.smithi099.stdout:} 2023-05-31T20:15:32.291 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "2e6123d2-d695-4350-9a01-61a455bc782b", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1685562631.516089_59.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-05-31T19:50:31.516089Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1685562970.274572_93.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-05-31T19:56:10.274572Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1685563798.756239_246.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-05-31T20:09:58.756239Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-05-31T20:15:32Z" } 2023-05-31T20:15:32.291 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1685562631.516089_59.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-05-31T19:50:31.516089Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1685562970.274572_93.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-05-31T19:56:10.274572Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1685563798.756239_246.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-05-31T20:09:58.756239Z', 'realm_epoch': 2}}] 2023-05-31T20:15:32.292 DEBUG:rgw_multi.tests:log_status={0: '1_1685562631.516089_59.1', 1: '', 2: '1_1685562970.274572_93.1', 3: '1_1685564111.129168_304.1'} 2023-05-31T20:15:32.292 DEBUG:rgw_multi.tests:sync_status={0: '1_1685562631.516089_59.1', 1: '', 2: '1_1685562970.274572_93.1', 3: '1_1685563798.756239_246.1'} 2023-05-31T20:15:32.293 WARNING:rgw_multi.tests:zone a2 behind master: shard=3 master=1_1685564111.129168_304.1 target=1_1685563798.756239_246.1 2023-05-31T20:15:37.293 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-31T20:15:37.294 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-31T20:15:37.295 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-05-31T20:15:37.353 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2023-05-31T20:15:37.353 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2023-05-31T20:15:37.405 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:15:37.394+0000 7f50d40ab780 20 RGW-SYNC:meta: read sync status 2023-05-31T20:15:37.405 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:15:37.395+0000 7f50d40ab780 20 RGW-SYNC:meta: read sync status 2023-05-31T20:15:37.416 INFO:teuthology.orchestra.run.smithi099.stdout:{ 2023-05-31T20:15:37.417 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_status": { 2023-05-31T20:15:37.417 INFO:teuthology.orchestra.run.smithi099.stdout: "info": { 2023-05-31T20:15:37.417 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "sync", 2023-05-31T20:15:37.418 INFO:teuthology.orchestra.run.smithi099.stdout: "num_shards": 4, 2023-05-31T20:15:37.418 INFO:teuthology.orchestra.run.smithi099.stdout: "period": "2e6123d2-d695-4350-9a01-61a455bc782b", 2023-05-31T20:15:37.418 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2023-05-31T20:15:37.419 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:15:37.419 INFO:teuthology.orchestra.run.smithi099.stdout: "markers": [ 2023-05-31T20:15:37.420 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T20:15:37.420 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 0, 2023-05-31T20:15:37.421 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T20:15:37.421 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2023-05-31T20:15:37.421 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1685562631.516089_59.1", 2023-05-31T20:15:37.422 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T20:15:37.422 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 1, 2023-05-31T20:15:37.422 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T20:15:37.422 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2023-05-31T19:50:31.516089Z", 2023-05-31T20:15:37.423 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2023-05-31T20:15:37.423 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T20:15:37.423 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:15:37.424 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T20:15:37.424 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 1, 2023-05-31T20:15:37.424 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T20:15:37.424 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2023-05-31T20:15:37.425 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "", 2023-05-31T20:15:37.425 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T20:15:37.425 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2023-05-31T20:15:37.426 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T20:15:37.427 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "0.000000", 2023-05-31T20:15:37.427 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2023-05-31T20:15:37.428 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T20:15:37.430 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:15:37.430 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T20:15:37.430 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 2, 2023-05-31T20:15:37.430 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T20:15:37.430 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2023-05-31T20:15:37.430 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1685562970.274572_93.1", 2023-05-31T20:15:37.431 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T20:15:37.431 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2023-05-31T20:15:37.431 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T20:15:37.432 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2023-05-31T19:56:10.274572Z", 2023-05-31T20:15:37.432 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2023-05-31T20:15:37.432 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T20:15:37.432 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:15:37.432 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T20:15:37.433 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 3, 2023-05-31T20:15:37.433 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T20:15:37.433 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2023-05-31T20:15:37.433 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1685563798.756239_246.1", 2023-05-31T20:15:37.433 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T20:15:37.433 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2023-05-31T20:15:37.434 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T20:15:37.434 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2023-05-31T20:09:58.756239Z", 2023-05-31T20:15:37.434 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2023-05-31T20:15:37.434 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T20:15:37.434 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T20:15:37.434 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2023-05-31T20:15:37.434 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:15:37.435 INFO:teuthology.orchestra.run.smithi099.stdout: "full_sync": { 2023-05-31T20:15:37.435 INFO:teuthology.orchestra.run.smithi099.stdout: "total": 1, 2023-05-31T20:15:37.435 INFO:teuthology.orchestra.run.smithi099.stdout: "complete": 1 2023-05-31T20:15:37.435 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:15:37.435 INFO:teuthology.orchestra.run.smithi099.stdout: "current_time": "2023-05-31T20:15:37Z" 2023-05-31T20:15:37.435 INFO:teuthology.orchestra.run.smithi099.stdout:} 2023-05-31T20:15:37.436 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "2e6123d2-d695-4350-9a01-61a455bc782b", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1685562631.516089_59.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-05-31T19:50:31.516089Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1685562970.274572_93.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-05-31T19:56:10.274572Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1685563798.756239_246.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-05-31T20:09:58.756239Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-05-31T20:15:37Z" } 2023-05-31T20:15:37.437 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1685562631.516089_59.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-05-31T19:50:31.516089Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1685562970.274572_93.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-05-31T19:56:10.274572Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1685563798.756239_246.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-05-31T20:09:58.756239Z', 'realm_epoch': 2}}] 2023-05-31T20:15:37.437 DEBUG:rgw_multi.tests:log_status={0: '1_1685562631.516089_59.1', 1: '', 2: '1_1685562970.274572_93.1', 3: '1_1685564111.129168_304.1'} 2023-05-31T20:15:37.438 DEBUG:rgw_multi.tests:sync_status={0: '1_1685562631.516089_59.1', 1: '', 2: '1_1685562970.274572_93.1', 3: '1_1685563798.756239_246.1'} 2023-05-31T20:15:37.438 WARNING:rgw_multi.tests:zone a2 behind master: shard=3 master=1_1685564111.129168_304.1 target=1_1685563798.756239_246.1 2023-05-31T20:15:42.439 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-31T20:15:42.440 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-31T20:15:42.441 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-05-31T20:15:42.491 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2023-05-31T20:15:42.492 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2023-05-31T20:15:42.540 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:15:42.534+0000 7fdc06ba0780 20 RGW-SYNC:meta: read sync status 2023-05-31T20:15:42.540 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:15:42.535+0000 7fdc06ba0780 20 RGW-SYNC:meta: read sync status 2023-05-31T20:15:42.556 INFO:teuthology.orchestra.run.smithi099.stdout:{ 2023-05-31T20:15:42.556 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_status": { 2023-05-31T20:15:42.556 INFO:teuthology.orchestra.run.smithi099.stdout: "info": { 2023-05-31T20:15:42.557 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "sync", 2023-05-31T20:15:42.557 INFO:teuthology.orchestra.run.smithi099.stdout: "num_shards": 4, 2023-05-31T20:15:42.557 INFO:teuthology.orchestra.run.smithi099.stdout: "period": "2e6123d2-d695-4350-9a01-61a455bc782b", 2023-05-31T20:15:42.557 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2023-05-31T20:15:42.557 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:15:42.557 INFO:teuthology.orchestra.run.smithi099.stdout: "markers": [ 2023-05-31T20:15:42.558 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T20:15:42.558 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 0, 2023-05-31T20:15:42.558 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T20:15:42.558 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2023-05-31T20:15:42.558 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1685562631.516089_59.1", 2023-05-31T20:15:42.558 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T20:15:42.558 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 1, 2023-05-31T20:15:42.559 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T20:15:42.559 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2023-05-31T19:50:31.516089Z", 2023-05-31T20:15:42.559 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2023-05-31T20:15:42.559 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T20:15:42.559 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:15:42.559 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T20:15:42.559 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 1, 2023-05-31T20:15:42.560 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T20:15:42.560 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2023-05-31T20:15:42.560 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "", 2023-05-31T20:15:42.560 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T20:15:42.560 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2023-05-31T20:15:42.560 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T20:15:42.561 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "0.000000", 2023-05-31T20:15:42.561 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2023-05-31T20:15:42.561 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T20:15:42.561 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:15:42.561 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T20:15:42.561 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 2, 2023-05-31T20:15:42.561 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T20:15:42.562 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2023-05-31T20:15:42.562 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1685562970.274572_93.1", 2023-05-31T20:15:42.562 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T20:15:42.562 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2023-05-31T20:15:42.562 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T20:15:42.562 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2023-05-31T19:56:10.274572Z", 2023-05-31T20:15:42.562 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2023-05-31T20:15:42.563 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T20:15:42.563 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:15:42.563 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T20:15:42.563 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 3, 2023-05-31T20:15:42.564 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T20:15:42.564 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2023-05-31T20:15:42.564 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1685563798.817516_251.1", 2023-05-31T20:15:42.564 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T20:15:42.564 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2023-05-31T20:15:42.564 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T20:15:42.565 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2023-05-31T20:09:58.817516Z", 2023-05-31T20:15:42.565 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2023-05-31T20:15:42.565 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T20:15:42.565 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T20:15:42.565 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2023-05-31T20:15:42.565 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:15:42.565 INFO:teuthology.orchestra.run.smithi099.stdout: "full_sync": { 2023-05-31T20:15:42.566 INFO:teuthology.orchestra.run.smithi099.stdout: "total": 1, 2023-05-31T20:15:42.566 INFO:teuthology.orchestra.run.smithi099.stdout: "complete": 1 2023-05-31T20:15:42.566 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:15:42.566 INFO:teuthology.orchestra.run.smithi099.stdout: "current_time": "2023-05-31T20:15:42Z" 2023-05-31T20:15:42.566 INFO:teuthology.orchestra.run.smithi099.stdout:} 2023-05-31T20:15:42.567 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "2e6123d2-d695-4350-9a01-61a455bc782b", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1685562631.516089_59.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-05-31T19:50:31.516089Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1685562970.274572_93.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-05-31T19:56:10.274572Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1685563798.817516_251.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-05-31T20:09:58.817516Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-05-31T20:15:42Z" } 2023-05-31T20:15:42.568 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1685562631.516089_59.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-05-31T19:50:31.516089Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1685562970.274572_93.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-05-31T19:56:10.274572Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1685563798.817516_251.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-05-31T20:09:58.817516Z', 'realm_epoch': 2}}] 2023-05-31T20:15:42.568 DEBUG:rgw_multi.tests:log_status={0: '1_1685562631.516089_59.1', 1: '', 2: '1_1685562970.274572_93.1', 3: '1_1685564111.129168_304.1'} 2023-05-31T20:15:42.569 DEBUG:rgw_multi.tests:sync_status={0: '1_1685562631.516089_59.1', 1: '', 2: '1_1685562970.274572_93.1', 3: '1_1685563798.817516_251.1'} 2023-05-31T20:15:42.570 WARNING:rgw_multi.tests:zone a2 behind master: shard=3 master=1_1685564111.129168_304.1 target=1_1685563798.817516_251.1 2023-05-31T20:15:47.571 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-31T20:15:47.572 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-31T20:15:47.572 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-05-31T20:15:47.624 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2023-05-31T20:15:47.625 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2023-05-31T20:15:47.672 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:15:47.666+0000 7f1b50379780 20 RGW-SYNC:meta: read sync status 2023-05-31T20:15:47.673 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:15:47.667+0000 7f1b50379780 20 RGW-SYNC:meta: read sync status 2023-05-31T20:15:47.692 INFO:teuthology.orchestra.run.smithi099.stdout:{ 2023-05-31T20:15:47.693 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_status": { 2023-05-31T20:15:47.693 INFO:teuthology.orchestra.run.smithi099.stdout: "info": { 2023-05-31T20:15:47.693 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "sync", 2023-05-31T20:15:47.693 INFO:teuthology.orchestra.run.smithi099.stdout: "num_shards": 4, 2023-05-31T20:15:47.693 INFO:teuthology.orchestra.run.smithi099.stdout: "period": "2e6123d2-d695-4350-9a01-61a455bc782b", 2023-05-31T20:15:47.694 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2023-05-31T20:15:47.694 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:15:47.694 INFO:teuthology.orchestra.run.smithi099.stdout: "markers": [ 2023-05-31T20:15:47.694 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T20:15:47.694 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 0, 2023-05-31T20:15:47.694 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T20:15:47.694 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2023-05-31T20:15:47.695 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1685562631.516089_59.1", 2023-05-31T20:15:47.695 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T20:15:47.695 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 1, 2023-05-31T20:15:47.695 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T20:15:47.695 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2023-05-31T19:50:31.516089Z", 2023-05-31T20:15:47.695 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2023-05-31T20:15:47.696 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T20:15:47.696 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:15:47.696 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T20:15:47.696 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 1, 2023-05-31T20:15:47.696 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T20:15:47.696 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2023-05-31T20:15:47.696 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "", 2023-05-31T20:15:47.697 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T20:15:47.697 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2023-05-31T20:15:47.697 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T20:15:47.697 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "0.000000", 2023-05-31T20:15:47.697 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2023-05-31T20:15:47.698 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T20:15:47.698 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:15:47.698 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T20:15:47.698 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 2, 2023-05-31T20:15:47.698 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T20:15:47.699 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2023-05-31T20:15:47.699 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1685562970.274572_93.1", 2023-05-31T20:15:47.699 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T20:15:47.699 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2023-05-31T20:15:47.700 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T20:15:47.700 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2023-05-31T19:56:10.274572Z", 2023-05-31T20:15:47.700 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2023-05-31T20:15:47.700 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T20:15:47.700 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:15:47.700 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T20:15:47.700 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 3, 2023-05-31T20:15:47.701 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T20:15:47.701 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2023-05-31T20:15:47.701 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1685563798.756239_246.1", 2023-05-31T20:15:47.701 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T20:15:47.701 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2023-05-31T20:15:47.701 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T20:15:47.702 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2023-05-31T20:09:58.756239Z", 2023-05-31T20:15:47.702 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2023-05-31T20:15:47.702 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T20:15:47.703 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T20:15:47.703 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2023-05-31T20:15:47.703 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:15:47.703 INFO:teuthology.orchestra.run.smithi099.stdout: "full_sync": { 2023-05-31T20:15:47.703 INFO:teuthology.orchestra.run.smithi099.stdout: "total": 1, 2023-05-31T20:15:47.704 INFO:teuthology.orchestra.run.smithi099.stdout: "complete": 1 2023-05-31T20:15:47.704 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:15:47.704 INFO:teuthology.orchestra.run.smithi099.stdout: "current_time": "2023-05-31T20:15:47Z" 2023-05-31T20:15:47.704 INFO:teuthology.orchestra.run.smithi099.stdout:} 2023-05-31T20:15:47.705 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "2e6123d2-d695-4350-9a01-61a455bc782b", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1685562631.516089_59.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-05-31T19:50:31.516089Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1685562970.274572_93.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-05-31T19:56:10.274572Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1685563798.756239_246.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-05-31T20:09:58.756239Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-05-31T20:15:47Z" } 2023-05-31T20:15:47.706 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1685562631.516089_59.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-05-31T19:50:31.516089Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1685562970.274572_93.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-05-31T19:56:10.274572Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1685563798.756239_246.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-05-31T20:09:58.756239Z', 'realm_epoch': 2}}] 2023-05-31T20:15:47.706 DEBUG:rgw_multi.tests:log_status={0: '1_1685562631.516089_59.1', 1: '', 2: '1_1685562970.274572_93.1', 3: '1_1685564111.129168_304.1'} 2023-05-31T20:15:47.706 DEBUG:rgw_multi.tests:sync_status={0: '1_1685562631.516089_59.1', 1: '', 2: '1_1685562970.274572_93.1', 3: '1_1685563798.756239_246.1'} 2023-05-31T20:15:47.707 WARNING:rgw_multi.tests:zone a2 behind master: shard=3 master=1_1685564111.129168_304.1 target=1_1685563798.756239_246.1 2023-05-31T20:15:52.708 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-31T20:15:52.709 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-31T20:15:52.709 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-05-31T20:15:52.767 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2023-05-31T20:15:52.767 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2023-05-31T20:15:52.804 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:15:52.795+0000 7f6006d04780 20 RGW-SYNC:meta: read sync status 2023-05-31T20:15:52.805 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:15:52.796+0000 7f6006d04780 20 RGW-SYNC:meta: read sync status 2023-05-31T20:15:52.819 INFO:teuthology.orchestra.run.smithi099.stdout:{ 2023-05-31T20:15:52.819 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_status": { 2023-05-31T20:15:52.819 INFO:teuthology.orchestra.run.smithi099.stdout: "info": { 2023-05-31T20:15:52.820 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "sync", 2023-05-31T20:15:52.820 INFO:teuthology.orchestra.run.smithi099.stdout: "num_shards": 4, 2023-05-31T20:15:52.820 INFO:teuthology.orchestra.run.smithi099.stdout: "period": "2e6123d2-d695-4350-9a01-61a455bc782b", 2023-05-31T20:15:52.820 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2023-05-31T20:15:52.820 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:15:52.820 INFO:teuthology.orchestra.run.smithi099.stdout: "markers": [ 2023-05-31T20:15:52.821 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T20:15:52.821 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 0, 2023-05-31T20:15:52.821 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T20:15:52.821 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2023-05-31T20:15:52.821 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1685562631.516089_59.1", 2023-05-31T20:15:52.821 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T20:15:52.821 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 1, 2023-05-31T20:15:52.822 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T20:15:52.822 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2023-05-31T19:50:31.516089Z", 2023-05-31T20:15:52.822 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2023-05-31T20:15:52.822 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T20:15:52.822 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:15:52.822 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T20:15:52.822 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 1, 2023-05-31T20:15:52.823 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T20:15:52.823 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2023-05-31T20:15:52.823 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "", 2023-05-31T20:15:52.823 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T20:15:52.823 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2023-05-31T20:15:52.823 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T20:15:52.824 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "0.000000", 2023-05-31T20:15:52.825 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2023-05-31T20:15:52.825 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T20:15:52.825 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:15:52.825 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T20:15:52.825 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 2, 2023-05-31T20:15:52.826 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T20:15:52.826 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2023-05-31T20:15:52.826 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1685562970.274572_93.1", 2023-05-31T20:15:52.826 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T20:15:52.827 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2023-05-31T20:15:52.827 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T20:15:52.827 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2023-05-31T19:56:10.274572Z", 2023-05-31T20:15:52.827 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2023-05-31T20:15:52.827 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T20:15:52.828 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:15:52.828 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T20:15:52.828 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 3, 2023-05-31T20:15:52.828 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T20:15:52.828 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2023-05-31T20:15:52.828 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1685564111.129168_304.1", 2023-05-31T20:15:52.829 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T20:15:52.829 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2023-05-31T20:15:52.829 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T20:15:52.829 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2023-05-31T20:15:11.129168Z", 2023-05-31T20:15:52.829 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2023-05-31T20:15:52.829 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T20:15:52.829 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T20:15:52.830 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2023-05-31T20:15:52.830 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:15:52.830 INFO:teuthology.orchestra.run.smithi099.stdout: "full_sync": { 2023-05-31T20:15:52.830 INFO:teuthology.orchestra.run.smithi099.stdout: "total": 1, 2023-05-31T20:15:52.830 INFO:teuthology.orchestra.run.smithi099.stdout: "complete": 1 2023-05-31T20:15:52.830 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:15:52.830 INFO:teuthology.orchestra.run.smithi099.stdout: "current_time": "2023-05-31T20:15:52Z" 2023-05-31T20:15:52.831 INFO:teuthology.orchestra.run.smithi099.stdout:} 2023-05-31T20:15:52.832 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "2e6123d2-d695-4350-9a01-61a455bc782b", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1685562631.516089_59.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-05-31T19:50:31.516089Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1685562970.274572_93.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-05-31T19:56:10.274572Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1685564111.129168_304.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-05-31T20:15:11.129168Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-05-31T20:15:52Z" } 2023-05-31T20:15:52.832 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1685562631.516089_59.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-05-31T19:50:31.516089Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1685562970.274572_93.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-05-31T19:56:10.274572Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1685564111.129168_304.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-05-31T20:15:11.129168Z', 'realm_epoch': 2}}] 2023-05-31T20:15:52.833 DEBUG:rgw_multi.tests:log_status={0: '1_1685562631.516089_59.1', 1: '', 2: '1_1685562970.274572_93.1', 3: '1_1685564111.129168_304.1'} 2023-05-31T20:15:52.833 DEBUG:rgw_multi.tests:sync_status={0: '1_1685562631.516089_59.1', 1: '', 2: '1_1685562970.274572_93.1', 3: '1_1685564111.129168_304.1'} 2023-05-31T20:15:52.834 INFO:rgw_multi.tests:finish meta checkpoint for zone=a2 2023-05-31T20:15:52.834 DEBUG:rgw_multi.tests:bucket checkpoint: source=a1 target=a2 bucket=pklxkd-46 2023-05-31T20:15:52.834 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'pklxkd-46', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-05-31T20:15:52.835 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'pklxkd-46', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-05-31T20:15:52.835 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket pklxkd-46 --source-zone a1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2023-05-31T20:15:52.892 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2023-05-31T20:15:52.893 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2023-05-31T20:15:52.952 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:15:52.947+0000 7ff45ae3c780 1 bucket sync caught up with source: 2023-05-31T20:15:52.953 INFO:teuthology.orchestra.run.smithi099.stderr: local status: [00000000001.18.6, , 00000000001.22.6, , 00000000001.14.6, , , 00000000001.23.6, , , ] 2023-05-31T20:15:52.953 INFO:teuthology.orchestra.run.smithi099.stderr: remote markers: [00000000001.18.6, , 00000000001.22.6, , 00000000001.14.6, , , 00000000001.23.6, , , ] 2023-05-31T20:15:52.953 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:15:52.947+0000 7ff45ae3c780 0 bucket checkpoint complete 2023-05-31T20:15:52.975 DEBUG:rgw_multi.tests:bucket checkpoint: source=a2 target=a1 bucket=pklxkd-46 2023-05-31T20:15:52.976 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'pklxkd-46', '--source-zone', 'a2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-05-31T20:15:52.976 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'pklxkd-46', '--source-zone', 'a2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-05-31T20:15:52.977 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket pklxkd-46 --source-zone a2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2023-05-31T20:15:53.033 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2023-05-31T20:15:53.034 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2023-05-31T20:15:53.098 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:15:53.086+0000 7fc9b8621780 1 waiting to reach incremental sync.. 2023-05-31T20:15:58.093 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:15:58.088+0000 7fc9b8621780 1 waiting to reach incremental sync.. 2023-05-31T20:16:03.093 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:16:03.088+0000 7fc9b8621780 1 waiting to reach incremental sync.. 2023-05-31T20:16:08.094 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:16:08.089+0000 7fc9b8621780 1 waiting to reach incremental sync.. 2023-05-31T20:16:13.095 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:16:13.090+0000 7fc9b8621780 1 waiting to reach incremental sync.. 2023-05-31T20:16:18.095 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:16:18.089+0000 7fc9b8621780 1 waiting to reach incremental sync.. 2023-05-31T20:16:23.097 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:16:23.090+0000 7fc9b8621780 1 waiting to reach incremental sync.. 2023-05-31T20:16:28.096 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:16:28.091+0000 7fc9b8621780 1 waiting to reach incremental sync.. 2023-05-31T20:16:33.097 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:16:33.092+0000 7fc9b8621780 1 waiting to reach incremental sync.. 2023-05-31T20:16:38.097 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:16:38.092+0000 7fc9b8621780 1 waiting to reach incremental sync.. 2023-05-31T20:16:43.098 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:16:43.093+0000 7fc9b8621780 1 waiting to reach incremental sync.. 2023-05-31T20:16:48.099 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:16:48.094+0000 7fc9b8621780 1 waiting to reach incremental sync.. 2023-05-31T20:16:53.100 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:16:53.095+0000 7fc9b8621780 1 waiting to reach incremental sync.. 2023-05-31T20:16:58.100 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:16:58.095+0000 7fc9b8621780 1 waiting to reach incremental sync.. 2023-05-31T20:17:03.101 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:17:03.095+0000 7fc9b8621780 1 waiting to reach incremental sync.. 2023-05-31T20:17:08.102 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:17:08.096+0000 7fc9b8621780 1 waiting to reach incremental sync.. 2023-05-31T20:17:13.102 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:17:13.096+0000 7fc9b8621780 1 waiting to reach incremental sync.. 2023-05-31T20:17:18.103 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:17:18.097+0000 7fc9b8621780 1 waiting to reach incremental sync.. 2023-05-31T20:17:23.104 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:17:23.100+0000 7fc9b8621780 1 bucket sync caught up with source: 2023-05-31T20:17:23.105 INFO:teuthology.orchestra.run.smithi099.stderr: local status: [00000000001.1473.6, , 00000000001.1182.6, , 00000000001.2018.6, , , 00000000001.664.6, , , ] 2023-05-31T20:17:23.105 INFO:teuthology.orchestra.run.smithi099.stderr: remote markers: [00000000001.1473.6, , 00000000001.1182.6, , 00000000001.2018.6, , , 00000000001.664.6, , , ] 2023-05-31T20:17:23.105 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:17:23.100+0000 7fc9b8621780 0 bucket checkpoint complete 2023-05-31T20:17:23.124 INFO:tasks.rgw_multi.zone_rados:comparing bucket=pklxkd-46 zones={a2, a1} 2023-05-31T20:17:23.125 DEBUG:boto:path=/pklxkd-46/ 2023-05-31T20:17:23.125 DEBUG:boto:auth_path=/pklxkd-46/ 2023-05-31T20:17:23.126 DEBUG:boto:Method: HEAD 2023-05-31T20:17:23.126 DEBUG:boto:Path: /pklxkd-46/ 2023-05-31T20:17:23.127 DEBUG:boto:Data: 2023-05-31T20:17:23.128 DEBUG:boto:Headers: {} 2023-05-31T20:17:23.128 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:17:23.129 DEBUG:boto:Port: 8001 2023-05-31T20:17:23.129 DEBUG:boto:Params: {} 2023-05-31T20:17:23.130 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8001} 2023-05-31T20:17:23.130 DEBUG:boto:Token: None 2023-05-31T20:17:23.131 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:17:23 GMT /pklxkd-46/ 2023-05-31T20:17:23.131 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:ZDWsOkF6VNgdbT32tYkRc5FU21I= 2023-05-31T20:17:23.131 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:17:23 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:ZDWsOkF6VNgdbT32tYkRc5FU21I=', 'Content-Length': '0'} 2023-05-31T20:17:23.135 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '4'), ('X-RGW-Bytes-Used', '24'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000a04937ec4cca9fd1-006477ab53-162e-a2'), ('Content-Length', '0'), ('Date', 'Wed, 31 May 2023 20:17:23 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:17:23.135 DEBUG:boto:path=/pklxkd-46/ 2023-05-31T20:17:23.136 DEBUG:boto:auth_path=/pklxkd-46/ 2023-05-31T20:17:23.136 DEBUG:boto:Method: HEAD 2023-05-31T20:17:23.137 DEBUG:boto:Path: /pklxkd-46/ 2023-05-31T20:17:23.137 DEBUG:boto:Data: 2023-05-31T20:17:23.138 DEBUG:boto:Headers: {} 2023-05-31T20:17:23.138 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:17:23.138 DEBUG:boto:Port: 8000 2023-05-31T20:17:23.139 DEBUG:boto:Params: {} 2023-05-31T20:17:23.139 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2023-05-31T20:17:23.140 DEBUG:boto:Token: None 2023-05-31T20:17:23.140 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:17:23 GMT /pklxkd-46/ 2023-05-31T20:17:23.140 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:ZDWsOkF6VNgdbT32tYkRc5FU21I= 2023-05-31T20:17:23.141 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:17:23 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:ZDWsOkF6VNgdbT32tYkRc5FU21I=', 'Content-Length': '0'} 2023-05-31T20:17:23.144 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '4'), ('X-RGW-Bytes-Used', '24'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000000c8f6f0e874ad439-006477ab53-1630-a1'), ('Content-Length', '0'), ('Date', 'Wed, 31 May 2023 20:17:23 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:17:23.145 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2023-05-31T20:17:23.145 DEBUG:boto:path=/pklxkd-46/ 2023-05-31T20:17:23.145 DEBUG:boto:auth_path=/pklxkd-46/ 2023-05-31T20:17:23.146 DEBUG:boto:path=/pklxkd-46/?versions&max-keys=999 2023-05-31T20:17:23.146 DEBUG:boto:auth_path=/pklxkd-46/?versions&max-keys=999 2023-05-31T20:17:23.146 DEBUG:boto:Method: GET 2023-05-31T20:17:23.147 DEBUG:boto:Path: /pklxkd-46/?versions&max-keys=999 2023-05-31T20:17:23.147 DEBUG:boto:Data: 2023-05-31T20:17:23.147 DEBUG:boto:Headers: {} 2023-05-31T20:17:23.148 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:17:23.148 DEBUG:boto:Port: 8001 2023-05-31T20:17:23.148 DEBUG:boto:Params: {} 2023-05-31T20:17:23.149 DEBUG:boto:Token: None 2023-05-31T20:17:23.149 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:17:23 GMT /pklxkd-46/?versions 2023-05-31T20:17:23.150 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:pV3Pp4jeWd93mE9JGgoBoxR8DjQ= 2023-05-31T20:17:23.150 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:17:23 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:pV3Pp4jeWd93mE9JGgoBoxR8DjQ=', 'Content-Length': '0'} 2023-05-31T20:17:23.153 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000005b09284c0045edf1-006477ab53-162e-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:17:23 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:17:23.153 DEBUG:boto:b'pklxkd-46999falseobj1nulltrue2023-05-31T20:15:11.203Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj2nulltrue2023-05-31T20:15:11.225Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj3nulltrue2023-05-31T20:15:11.255Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj4nulltrue2023-05-31T20:15:11.317Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:17:23.154 DEBUG:tasks.rgw_multi.zone_rados:o=obj1 2023-05-31T20:17:23.155 DEBUG:tasks.rgw_multi.zone_rados:o=obj2 2023-05-31T20:17:23.155 DEBUG:tasks.rgw_multi.zone_rados:o=obj3 2023-05-31T20:17:23.155 DEBUG:tasks.rgw_multi.zone_rados:o=obj4 2023-05-31T20:17:23.155 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2023-05-31T20:17:23.156 DEBUG:boto:path=/pklxkd-46/ 2023-05-31T20:17:23.156 DEBUG:boto:auth_path=/pklxkd-46/ 2023-05-31T20:17:23.157 DEBUG:boto:path=/pklxkd-46/?versions&max-keys=999 2023-05-31T20:17:23.157 DEBUG:boto:auth_path=/pklxkd-46/?versions&max-keys=999 2023-05-31T20:17:23.157 DEBUG:boto:Method: GET 2023-05-31T20:17:23.158 DEBUG:boto:Path: /pklxkd-46/?versions&max-keys=999 2023-05-31T20:17:23.158 DEBUG:boto:Data: 2023-05-31T20:17:23.158 DEBUG:boto:Headers: {} 2023-05-31T20:17:23.159 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:17:23.159 DEBUG:boto:Port: 8000 2023-05-31T20:17:23.159 DEBUG:boto:Params: {} 2023-05-31T20:17:23.160 DEBUG:boto:Token: None 2023-05-31T20:17:23.160 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:17:23 GMT /pklxkd-46/?versions 2023-05-31T20:17:23.161 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:pV3Pp4jeWd93mE9JGgoBoxR8DjQ= 2023-05-31T20:17:23.161 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:17:23 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:pV3Pp4jeWd93mE9JGgoBoxR8DjQ=', 'Content-Length': '0'} 2023-05-31T20:17:23.163 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000002892472494b0eaf6-006477ab53-1630-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:17:23 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:17:23.164 DEBUG:boto:b'pklxkd-46999falseobj1nulltrue2023-05-31T20:15:11.203Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj2nulltrue2023-05-31T20:15:11.225Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj3nulltrue2023-05-31T20:15:11.255Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj4nulltrue2023-05-31T20:15:11.317Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:17:23.165 DEBUG:tasks.rgw_multi.zone_rados:o=obj1 2023-05-31T20:17:23.165 DEBUG:tasks.rgw_multi.zone_rados:o=obj2 2023-05-31T20:17:23.165 DEBUG:tasks.rgw_multi.zone_rados:o=obj3 2023-05-31T20:17:23.166 DEBUG:tasks.rgw_multi.zone_rados:o=obj4 2023-05-31T20:17:23.166 DEBUG:boto:path=/pklxkd-46/ 2023-05-31T20:17:23.166 DEBUG:boto:auth_path=/pklxkd-46/ 2023-05-31T20:17:23.167 DEBUG:boto:path=/pklxkd-46/?versions&max-keys=999 2023-05-31T20:17:23.167 DEBUG:boto:auth_path=/pklxkd-46/?versions&max-keys=999 2023-05-31T20:17:23.167 DEBUG:boto:Method: GET 2023-05-31T20:17:23.168 DEBUG:boto:Path: /pklxkd-46/?versions&max-keys=999 2023-05-31T20:17:23.168 DEBUG:boto:Data: 2023-05-31T20:17:23.168 DEBUG:boto:Headers: {} 2023-05-31T20:17:23.169 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:17:23.169 DEBUG:boto:Port: 8001 2023-05-31T20:17:23.169 DEBUG:boto:Params: {} 2023-05-31T20:17:23.170 DEBUG:boto:Token: None 2023-05-31T20:17:23.170 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:17:23 GMT /pklxkd-46/?versions 2023-05-31T20:17:23.171 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:pV3Pp4jeWd93mE9JGgoBoxR8DjQ= 2023-05-31T20:17:23.171 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:17:23 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:pV3Pp4jeWd93mE9JGgoBoxR8DjQ=', 'Content-Length': '0'} 2023-05-31T20:17:23.174 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000017dab338ceee5398-006477ab53-162e-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:17:23 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:17:23.174 DEBUG:boto:b'pklxkd-46999falseobj1nulltrue2023-05-31T20:15:11.203Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj2nulltrue2023-05-31T20:15:11.225Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj3nulltrue2023-05-31T20:15:11.255Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj4nulltrue2023-05-31T20:15:11.317Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:17:23.175 DEBUG:boto:path=/pklxkd-46/ 2023-05-31T20:17:23.175 DEBUG:boto:auth_path=/pklxkd-46/ 2023-05-31T20:17:23.176 DEBUG:boto:path=/pklxkd-46/?versions&max-keys=999 2023-05-31T20:17:23.176 DEBUG:boto:auth_path=/pklxkd-46/?versions&max-keys=999 2023-05-31T20:17:23.177 DEBUG:boto:Method: GET 2023-05-31T20:17:23.177 DEBUG:boto:Path: /pklxkd-46/?versions&max-keys=999 2023-05-31T20:17:23.177 DEBUG:boto:Data: 2023-05-31T20:17:23.178 DEBUG:boto:Headers: {} 2023-05-31T20:17:23.178 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:17:23.178 DEBUG:boto:Port: 8000 2023-05-31T20:17:23.179 DEBUG:boto:Params: {} 2023-05-31T20:17:23.179 DEBUG:boto:Token: None 2023-05-31T20:17:23.180 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:17:23 GMT /pklxkd-46/?versions 2023-05-31T20:17:23.180 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:pV3Pp4jeWd93mE9JGgoBoxR8DjQ= 2023-05-31T20:17:23.181 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:17:23 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:pV3Pp4jeWd93mE9JGgoBoxR8DjQ=', 'Content-Length': '0'} 2023-05-31T20:17:23.183 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000006bcc2b532331b5ce-006477ab53-1630-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:17:23 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:17:23.184 DEBUG:boto:b'pklxkd-46999falseobj1nulltrue2023-05-31T20:15:11.203Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj2nulltrue2023-05-31T20:15:11.225Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj3nulltrue2023-05-31T20:15:11.255Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj4nulltrue2023-05-31T20:15:11.317Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:17:23.185 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2023-05-31T20:17:23.185 DEBUG:boto:path=/pklxkd-46/obj1 2023-05-31T20:17:23.186 DEBUG:boto:auth_path=/pklxkd-46/obj1 2023-05-31T20:17:23.186 DEBUG:boto:path=/pklxkd-46/obj1?versionId=null 2023-05-31T20:17:23.186 DEBUG:boto:auth_path=/pklxkd-46/obj1?versionId=null 2023-05-31T20:17:23.187 DEBUG:boto:Method: GET 2023-05-31T20:17:23.187 DEBUG:boto:Path: /pklxkd-46/obj1?versionId=null 2023-05-31T20:17:23.187 DEBUG:boto:Data: 2023-05-31T20:17:23.188 DEBUG:boto:Headers: {} 2023-05-31T20:17:23.188 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:17:23.188 DEBUG:boto:Port: 8001 2023-05-31T20:17:23.189 DEBUG:boto:Params: {} 2023-05-31T20:17:23.189 DEBUG:boto:Token: None 2023-05-31T20:17:23.190 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:17:23 GMT /pklxkd-46/obj1?versionId=null 2023-05-31T20:17:23.190 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:tdMRahQztx5xKssIvS+Nn47/UoQ= 2023-05-31T20:17:23.191 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:17:23 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:tdMRahQztx5xKssIvS+Nn47/UoQ=', 'Content-Length': '0'} 2023-05-31T20:17:23.192 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:15:11 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000766ca21dd25df863-006477ab53-162e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:17:23 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:17:23.193 DEBUG:boto:path=/pklxkd-46/obj1 2023-05-31T20:17:23.193 DEBUG:boto:auth_path=/pklxkd-46/obj1 2023-05-31T20:17:23.194 DEBUG:boto:path=/pklxkd-46/obj1?versionId=null 2023-05-31T20:17:23.194 DEBUG:boto:auth_path=/pklxkd-46/obj1?versionId=null 2023-05-31T20:17:23.195 DEBUG:boto:Method: GET 2023-05-31T20:17:23.195 DEBUG:boto:Path: /pklxkd-46/obj1?versionId=null 2023-05-31T20:17:23.195 DEBUG:boto:Data: 2023-05-31T20:17:23.196 DEBUG:boto:Headers: {} 2023-05-31T20:17:23.196 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:17:23.196 DEBUG:boto:Port: 8000 2023-05-31T20:17:23.197 DEBUG:boto:Params: {} 2023-05-31T20:17:23.197 DEBUG:boto:Token: None 2023-05-31T20:17:23.198 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:17:23 GMT /pklxkd-46/obj1?versionId=null 2023-05-31T20:17:23.198 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:tdMRahQztx5xKssIvS+Nn47/UoQ= 2023-05-31T20:17:23.198 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:17:23 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:tdMRahQztx5xKssIvS+Nn47/UoQ=', 'Content-Length': '0'} 2023-05-31T20:17:23.200 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:15:11 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000b401f368082159e4-006477ab53-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:17:23 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:17:23.201 DEBUG:boto:path=/pklxkd-46/obj1 2023-05-31T20:17:23.201 DEBUG:boto:auth_path=/pklxkd-46/obj1 2023-05-31T20:17:23.201 DEBUG:boto:path=/pklxkd-46/obj1?versionId=null 2023-05-31T20:17:23.202 DEBUG:boto:auth_path=/pklxkd-46/obj1?versionId=null 2023-05-31T20:17:23.202 DEBUG:boto:Method: HEAD 2023-05-31T20:17:23.202 DEBUG:boto:Path: /pklxkd-46/obj1?versionId=null 2023-05-31T20:17:23.203 DEBUG:boto:Data: 2023-05-31T20:17:23.203 DEBUG:boto:Headers: {} 2023-05-31T20:17:23.203 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:17:23.204 DEBUG:boto:Port: 8001 2023-05-31T20:17:23.204 DEBUG:boto:Params: {} 2023-05-31T20:17:23.205 DEBUG:boto:Token: None 2023-05-31T20:17:23.205 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:17:23 GMT /pklxkd-46/obj1?versionId=null 2023-05-31T20:17:23.205 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:7VMSRV18y0esFjBZOHzSYWg/7iU= 2023-05-31T20:17:23.206 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:17:23 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:7VMSRV18y0esFjBZOHzSYWg/7iU=', 'Content-Length': '0'} 2023-05-31T20:17:23.207 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:15:11 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000007f48b1e97e0fb4cc-006477ab53-162e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:17:23 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:17:23.208 DEBUG:boto:path=/pklxkd-46/obj1 2023-05-31T20:17:23.208 DEBUG:boto:auth_path=/pklxkd-46/obj1 2023-05-31T20:17:23.209 DEBUG:boto:path=/pklxkd-46/obj1?versionId=null 2023-05-31T20:17:23.209 DEBUG:boto:auth_path=/pklxkd-46/obj1?versionId=null 2023-05-31T20:17:23.209 DEBUG:boto:Method: HEAD 2023-05-31T20:17:23.210 DEBUG:boto:Path: /pklxkd-46/obj1?versionId=null 2023-05-31T20:17:23.210 DEBUG:boto:Data: 2023-05-31T20:17:23.210 DEBUG:boto:Headers: {} 2023-05-31T20:17:23.211 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:17:23.211 DEBUG:boto:Port: 8000 2023-05-31T20:17:23.211 DEBUG:boto:Params: {} 2023-05-31T20:17:23.212 DEBUG:boto:Token: None 2023-05-31T20:17:23.212 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:17:23 GMT /pklxkd-46/obj1?versionId=null 2023-05-31T20:17:23.213 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:7VMSRV18y0esFjBZOHzSYWg/7iU= 2023-05-31T20:17:23.213 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:17:23 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:7VMSRV18y0esFjBZOHzSYWg/7iU=', 'Content-Length': '0'} 2023-05-31T20:17:23.215 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:15:11 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000f01f6fe1ff5ac5c4-006477ab53-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:17:23 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:17:23.215 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2023-05-31T20:17:23.216 DEBUG:boto:path=/pklxkd-46/obj1 2023-05-31T20:17:23.216 DEBUG:boto:auth_path=/pklxkd-46/obj1 2023-05-31T20:17:23.217 DEBUG:boto:path=/pklxkd-46/obj1?versionId=null 2023-05-31T20:17:23.217 DEBUG:boto:auth_path=/pklxkd-46/obj1?versionId=null 2023-05-31T20:17:23.217 DEBUG:boto:Method: GET 2023-05-31T20:17:23.218 DEBUG:boto:Path: /pklxkd-46/obj1?versionId=null 2023-05-31T20:17:23.218 DEBUG:boto:Data: 2023-05-31T20:17:23.218 DEBUG:boto:Headers: {} 2023-05-31T20:17:23.219 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:17:23.219 DEBUG:boto:Port: 8001 2023-05-31T20:17:23.220 DEBUG:boto:Params: {} 2023-05-31T20:17:23.220 DEBUG:boto:Token: None 2023-05-31T20:17:23.220 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:17:23 GMT /pklxkd-46/obj1?versionId=null 2023-05-31T20:17:23.221 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:tdMRahQztx5xKssIvS+Nn47/UoQ= 2023-05-31T20:17:23.221 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:17:23 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:tdMRahQztx5xKssIvS+Nn47/UoQ=', 'Content-Length': '0'} 2023-05-31T20:17:23.223 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:15:11 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000733c88e7498fc7dd-006477ab53-162e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:17:23 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:17:23.264 DEBUG:boto:path=/pklxkd-46/obj1 2023-05-31T20:17:23.265 DEBUG:boto:auth_path=/pklxkd-46/obj1 2023-05-31T20:17:23.265 DEBUG:boto:path=/pklxkd-46/obj1?versionId=null 2023-05-31T20:17:23.266 DEBUG:boto:auth_path=/pklxkd-46/obj1?versionId=null 2023-05-31T20:17:23.266 DEBUG:boto:Method: GET 2023-05-31T20:17:23.266 DEBUG:boto:Path: /pklxkd-46/obj1?versionId=null 2023-05-31T20:17:23.267 DEBUG:boto:Data: 2023-05-31T20:17:23.267 DEBUG:boto:Headers: {} 2023-05-31T20:17:23.267 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:17:23.268 DEBUG:boto:Port: 8000 2023-05-31T20:17:23.268 DEBUG:boto:Params: {} 2023-05-31T20:17:23.269 DEBUG:boto:Token: None 2023-05-31T20:17:23.269 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:17:23 GMT /pklxkd-46/obj1?versionId=null 2023-05-31T20:17:23.270 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:tdMRahQztx5xKssIvS+Nn47/UoQ= 2023-05-31T20:17:23.270 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:17:23 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:tdMRahQztx5xKssIvS+Nn47/UoQ=', 'Content-Length': '0'} 2023-05-31T20:17:23.272 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:15:11 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000442203ab99b71e8e-006477ab53-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:17:23 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:17:23.273 DEBUG:boto:path=/pklxkd-46/ 2023-05-31T20:17:23.273 DEBUG:boto:auth_path=/pklxkd-46/ 2023-05-31T20:17:23.273 DEBUG:boto:path=/pklxkd-46/?prefix=obj1 2023-05-31T20:17:23.274 DEBUG:boto:auth_path=/pklxkd-46/?prefix=obj1 2023-05-31T20:17:23.274 DEBUG:boto:Method: GET 2023-05-31T20:17:23.274 DEBUG:boto:Path: /pklxkd-46/?prefix=obj1 2023-05-31T20:17:23.275 DEBUG:boto:Data: 2023-05-31T20:17:23.275 DEBUG:boto:Headers: {} 2023-05-31T20:17:23.275 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:17:23.276 DEBUG:boto:Port: 8001 2023-05-31T20:17:23.276 DEBUG:boto:Params: {} 2023-05-31T20:17:23.277 DEBUG:boto:Token: None 2023-05-31T20:17:23.277 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:17:23 GMT /pklxkd-46/ 2023-05-31T20:17:23.277 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:5JjXDyL51Vtp5kJiXsm2+F/F3Xg= 2023-05-31T20:17:23.278 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:17:23 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:5JjXDyL51Vtp5kJiXsm2+F/F3Xg=', 'Content-Length': '0'} 2023-05-31T20:17:23.280 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000037d521fe3401cba2-006477ab53-162e-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:17:23 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:17:23.281 DEBUG:boto:b'pklxkd-46obj11000falseobj12023-05-31T20:15:11.203Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:17:23.281 DEBUG:boto:path=/pklxkd-46/ 2023-05-31T20:17:23.282 DEBUG:boto:auth_path=/pklxkd-46/ 2023-05-31T20:17:23.282 DEBUG:boto:path=/pklxkd-46/?prefix=obj1 2023-05-31T20:17:23.282 DEBUG:boto:auth_path=/pklxkd-46/?prefix=obj1 2023-05-31T20:17:23.283 DEBUG:boto:Method: GET 2023-05-31T20:17:23.283 DEBUG:boto:Path: /pklxkd-46/?prefix=obj1 2023-05-31T20:17:23.284 DEBUG:boto:Data: 2023-05-31T20:17:23.284 DEBUG:boto:Headers: {} 2023-05-31T20:17:23.284 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:17:23.285 DEBUG:boto:Port: 8000 2023-05-31T20:17:23.285 DEBUG:boto:Params: {} 2023-05-31T20:17:23.285 DEBUG:boto:Token: None 2023-05-31T20:17:23.286 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:17:23 GMT /pklxkd-46/ 2023-05-31T20:17:23.286 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:5JjXDyL51Vtp5kJiXsm2+F/F3Xg= 2023-05-31T20:17:23.287 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:17:23 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:5JjXDyL51Vtp5kJiXsm2+F/F3Xg=', 'Content-Length': '0'} 2023-05-31T20:17:23.289 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d11f7ceb1009c3ec-006477ab53-1630-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:17:23 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:17:23.290 DEBUG:boto:b'pklxkd-46obj11000falseobj12023-05-31T20:15:11.203Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:17:23.290 DEBUG:boto:path=/pklxkd-46/obj1 2023-05-31T20:17:23.291 DEBUG:boto:auth_path=/pklxkd-46/obj1 2023-05-31T20:17:23.291 DEBUG:boto:Method: HEAD 2023-05-31T20:17:23.291 DEBUG:boto:Path: /pklxkd-46/obj1 2023-05-31T20:17:23.292 DEBUG:boto:Data: 2023-05-31T20:17:23.292 DEBUG:boto:Headers: {} 2023-05-31T20:17:23.292 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:17:23.293 DEBUG:boto:Port: 8001 2023-05-31T20:17:23.293 DEBUG:boto:Params: {} 2023-05-31T20:17:23.293 DEBUG:boto:Token: None 2023-05-31T20:17:23.294 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:17:23 GMT /pklxkd-46/obj1 2023-05-31T20:17:23.294 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:HSyn3AzC/pH7wlEqVrdt1GD9s1Y= 2023-05-31T20:17:23.295 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:17:23 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:HSyn3AzC/pH7wlEqVrdt1GD9s1Y=', 'Content-Length': '0'} 2023-05-31T20:17:23.297 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:15:11 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000004ca47a965247c837-006477ab53-162e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:17:23 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:17:23.297 DEBUG:boto:path=/pklxkd-46/obj1 2023-05-31T20:17:23.298 DEBUG:boto:auth_path=/pklxkd-46/obj1 2023-05-31T20:17:23.298 DEBUG:boto:Method: HEAD 2023-05-31T20:17:23.298 DEBUG:boto:Path: /pklxkd-46/obj1 2023-05-31T20:17:23.299 DEBUG:boto:Data: 2023-05-31T20:17:23.299 DEBUG:boto:Headers: {} 2023-05-31T20:17:23.299 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:17:23.300 DEBUG:boto:Port: 8000 2023-05-31T20:17:23.300 DEBUG:boto:Params: {} 2023-05-31T20:17:23.301 DEBUG:boto:Token: None 2023-05-31T20:17:23.301 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:17:23 GMT /pklxkd-46/obj1 2023-05-31T20:17:23.301 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:HSyn3AzC/pH7wlEqVrdt1GD9s1Y= 2023-05-31T20:17:23.302 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:17:23 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:HSyn3AzC/pH7wlEqVrdt1GD9s1Y=', 'Content-Length': '0'} 2023-05-31T20:17:23.304 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:15:11 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000b7f8ea438dd70be8-006477ab53-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:17:23 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:17:23.304 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2023-05-31T20:17:23.305 DEBUG:boto:path=/pklxkd-46/obj1 2023-05-31T20:17:23.305 DEBUG:boto:auth_path=/pklxkd-46/obj1 2023-05-31T20:17:23.305 DEBUG:boto:Method: GET 2023-05-31T20:17:23.306 DEBUG:boto:Path: /pklxkd-46/obj1 2023-05-31T20:17:23.306 DEBUG:boto:Data: 2023-05-31T20:17:23.306 DEBUG:boto:Headers: {} 2023-05-31T20:17:23.307 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:17:23.307 DEBUG:boto:Port: 8001 2023-05-31T20:17:23.307 DEBUG:boto:Params: {} 2023-05-31T20:17:23.308 DEBUG:boto:Token: None 2023-05-31T20:17:23.308 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:17:23 GMT /pklxkd-46/obj1 2023-05-31T20:17:23.309 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:Tvgz2Ku4/HWbuqbSQgD6wN3lcIg= 2023-05-31T20:17:23.309 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:17:23 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:Tvgz2Ku4/HWbuqbSQgD6wN3lcIg=', 'Content-Length': '0'} 2023-05-31T20:17:23.311 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:15:11 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000b644100283e0b05e-006477ab53-162e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:17:23 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:17:23.356 DEBUG:boto:path=/pklxkd-46/obj1 2023-05-31T20:17:23.357 DEBUG:boto:auth_path=/pklxkd-46/obj1 2023-05-31T20:17:23.357 DEBUG:boto:Method: GET 2023-05-31T20:17:23.358 DEBUG:boto:Path: /pklxkd-46/obj1 2023-05-31T20:17:23.358 DEBUG:boto:Data: 2023-05-31T20:17:23.358 DEBUG:boto:Headers: {} 2023-05-31T20:17:23.359 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:17:23.359 DEBUG:boto:Port: 8000 2023-05-31T20:17:23.359 DEBUG:boto:Params: {} 2023-05-31T20:17:23.360 DEBUG:boto:Token: None 2023-05-31T20:17:23.360 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:17:23 GMT /pklxkd-46/obj1 2023-05-31T20:17:23.361 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:Tvgz2Ku4/HWbuqbSQgD6wN3lcIg= 2023-05-31T20:17:23.361 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:17:23 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:Tvgz2Ku4/HWbuqbSQgD6wN3lcIg=', 'Content-Length': '0'} 2023-05-31T20:17:23.363 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:15:11 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000043d779b9df94684d-006477ab53-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:17:23 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:17:23.364 DEBUG:boto:path=/pklxkd-46/ 2023-05-31T20:17:23.364 DEBUG:boto:auth_path=/pklxkd-46/ 2023-05-31T20:17:23.365 DEBUG:boto:path=/pklxkd-46/?prefix=obj1 2023-05-31T20:17:23.365 DEBUG:boto:auth_path=/pklxkd-46/?prefix=obj1 2023-05-31T20:17:23.365 DEBUG:boto:Method: GET 2023-05-31T20:17:23.366 DEBUG:boto:Path: /pklxkd-46/?prefix=obj1 2023-05-31T20:17:23.366 DEBUG:boto:Data: 2023-05-31T20:17:23.366 DEBUG:boto:Headers: {} 2023-05-31T20:17:23.367 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:17:23.367 DEBUG:boto:Port: 8001 2023-05-31T20:17:23.368 DEBUG:boto:Params: {} 2023-05-31T20:17:23.368 DEBUG:boto:Token: None 2023-05-31T20:17:23.368 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:17:23 GMT /pklxkd-46/ 2023-05-31T20:17:23.369 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:5JjXDyL51Vtp5kJiXsm2+F/F3Xg= 2023-05-31T20:17:23.369 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:17:23 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:5JjXDyL51Vtp5kJiXsm2+F/F3Xg=', 'Content-Length': '0'} 2023-05-31T20:17:23.372 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e3074a97db4ad14c-006477ab53-162e-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:17:23 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:17:23.372 DEBUG:boto:b'pklxkd-46obj11000falseobj12023-05-31T20:15:11.203Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:17:23.373 DEBUG:boto:path=/pklxkd-46/ 2023-05-31T20:17:23.373 DEBUG:boto:auth_path=/pklxkd-46/ 2023-05-31T20:17:23.374 DEBUG:boto:path=/pklxkd-46/?prefix=obj1 2023-05-31T20:17:23.374 DEBUG:boto:auth_path=/pklxkd-46/?prefix=obj1 2023-05-31T20:17:23.375 DEBUG:boto:Method: GET 2023-05-31T20:17:23.375 DEBUG:boto:Path: /pklxkd-46/?prefix=obj1 2023-05-31T20:17:23.375 DEBUG:boto:Data: 2023-05-31T20:17:23.376 DEBUG:boto:Headers: {} 2023-05-31T20:17:23.376 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:17:23.376 DEBUG:boto:Port: 8000 2023-05-31T20:17:23.377 DEBUG:boto:Params: {} 2023-05-31T20:17:23.377 DEBUG:boto:Token: None 2023-05-31T20:17:23.378 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:17:23 GMT /pklxkd-46/ 2023-05-31T20:17:23.378 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:5JjXDyL51Vtp5kJiXsm2+F/F3Xg= 2023-05-31T20:17:23.379 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:17:23 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:5JjXDyL51Vtp5kJiXsm2+F/F3Xg=', 'Content-Length': '0'} 2023-05-31T20:17:23.381 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000006be499bcaac19ce6-006477ab53-1630-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:17:23 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:17:23.382 DEBUG:boto:b'pklxkd-46obj11000falseobj12023-05-31T20:15:11.203Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:17:23.382 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2023-05-31T20:17:23.383 DEBUG:boto:path=/pklxkd-46/obj2 2023-05-31T20:17:23.383 DEBUG:boto:auth_path=/pklxkd-46/obj2 2023-05-31T20:17:23.383 DEBUG:boto:path=/pklxkd-46/obj2?versionId=null 2023-05-31T20:17:23.384 DEBUG:boto:auth_path=/pklxkd-46/obj2?versionId=null 2023-05-31T20:17:23.384 DEBUG:boto:Method: GET 2023-05-31T20:17:23.384 DEBUG:boto:Path: /pklxkd-46/obj2?versionId=null 2023-05-31T20:17:23.385 DEBUG:boto:Data: 2023-05-31T20:17:23.385 DEBUG:boto:Headers: {} 2023-05-31T20:17:23.386 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:17:23.386 DEBUG:boto:Port: 8001 2023-05-31T20:17:23.386 DEBUG:boto:Params: {} 2023-05-31T20:17:23.387 DEBUG:boto:Token: None 2023-05-31T20:17:23.387 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:17:23 GMT /pklxkd-46/obj2?versionId=null 2023-05-31T20:17:23.388 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:dWFd56rFRafj3kVf9FIhGAgKK5k= 2023-05-31T20:17:23.388 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:17:23 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:dWFd56rFRafj3kVf9FIhGAgKK5k=', 'Content-Length': '0'} 2023-05-31T20:17:23.390 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:15:11 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000001904a7cd9d3746dc-006477ab53-162e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:17:23 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:17:23.391 DEBUG:boto:path=/pklxkd-46/obj2 2023-05-31T20:17:23.391 DEBUG:boto:auth_path=/pklxkd-46/obj2 2023-05-31T20:17:23.391 DEBUG:boto:path=/pklxkd-46/obj2?versionId=null 2023-05-31T20:17:23.392 DEBUG:boto:auth_path=/pklxkd-46/obj2?versionId=null 2023-05-31T20:17:23.392 DEBUG:boto:Method: GET 2023-05-31T20:17:23.393 DEBUG:boto:Path: /pklxkd-46/obj2?versionId=null 2023-05-31T20:17:23.393 DEBUG:boto:Data: 2023-05-31T20:17:23.393 DEBUG:boto:Headers: {} 2023-05-31T20:17:23.394 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:17:23.394 DEBUG:boto:Port: 8000 2023-05-31T20:17:23.394 DEBUG:boto:Params: {} 2023-05-31T20:17:23.395 DEBUG:boto:Token: None 2023-05-31T20:17:23.395 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:17:23 GMT /pklxkd-46/obj2?versionId=null 2023-05-31T20:17:23.396 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:dWFd56rFRafj3kVf9FIhGAgKK5k= 2023-05-31T20:17:23.396 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:17:23 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:dWFd56rFRafj3kVf9FIhGAgKK5k=', 'Content-Length': '0'} 2023-05-31T20:17:23.398 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:15:11 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000e2337c46bb7dba09-006477ab53-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:17:23 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:17:23.440 DEBUG:boto:path=/pklxkd-46/obj2 2023-05-31T20:17:23.441 DEBUG:boto:auth_path=/pklxkd-46/obj2 2023-05-31T20:17:23.441 DEBUG:boto:path=/pklxkd-46/obj2?versionId=null 2023-05-31T20:17:23.442 DEBUG:boto:auth_path=/pklxkd-46/obj2?versionId=null 2023-05-31T20:17:23.442 DEBUG:boto:Method: HEAD 2023-05-31T20:17:23.442 DEBUG:boto:Path: /pklxkd-46/obj2?versionId=null 2023-05-31T20:17:23.443 DEBUG:boto:Data: 2023-05-31T20:17:23.443 DEBUG:boto:Headers: {} 2023-05-31T20:17:23.443 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:17:23.444 DEBUG:boto:Port: 8001 2023-05-31T20:17:23.444 DEBUG:boto:Params: {} 2023-05-31T20:17:23.444 DEBUG:boto:Token: None 2023-05-31T20:17:23.445 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:17:23 GMT /pklxkd-46/obj2?versionId=null 2023-05-31T20:17:23.445 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:5+Wh3NfPwhQqOCKdksLYXc4YEKI= 2023-05-31T20:17:23.446 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:17:23 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:5+Wh3NfPwhQqOCKdksLYXc4YEKI=', 'Content-Length': '0'} 2023-05-31T20:17:23.448 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:15:11 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000808bc4fd2e36b787-006477ab53-162e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:17:23 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:17:23.448 DEBUG:boto:path=/pklxkd-46/obj2 2023-05-31T20:17:23.449 DEBUG:boto:auth_path=/pklxkd-46/obj2 2023-05-31T20:17:23.449 DEBUG:boto:path=/pklxkd-46/obj2?versionId=null 2023-05-31T20:17:23.449 DEBUG:boto:auth_path=/pklxkd-46/obj2?versionId=null 2023-05-31T20:17:23.450 DEBUG:boto:Method: HEAD 2023-05-31T20:17:23.450 DEBUG:boto:Path: /pklxkd-46/obj2?versionId=null 2023-05-31T20:17:23.450 DEBUG:boto:Data: 2023-05-31T20:17:23.451 DEBUG:boto:Headers: {} 2023-05-31T20:17:23.451 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:17:23.451 DEBUG:boto:Port: 8000 2023-05-31T20:17:23.452 DEBUG:boto:Params: {} 2023-05-31T20:17:23.452 DEBUG:boto:Token: None 2023-05-31T20:17:23.453 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:17:23 GMT /pklxkd-46/obj2?versionId=null 2023-05-31T20:17:23.453 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:5+Wh3NfPwhQqOCKdksLYXc4YEKI= 2023-05-31T20:17:23.453 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:17:23 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:5+Wh3NfPwhQqOCKdksLYXc4YEKI=', 'Content-Length': '0'} 2023-05-31T20:17:23.455 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:15:11 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000066caa2eb67f34c41-006477ab53-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:17:23 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:17:23.456 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2023-05-31T20:17:23.456 DEBUG:boto:path=/pklxkd-46/obj2 2023-05-31T20:17:23.457 DEBUG:boto:auth_path=/pklxkd-46/obj2 2023-05-31T20:17:23.457 DEBUG:boto:path=/pklxkd-46/obj2?versionId=null 2023-05-31T20:17:23.457 DEBUG:boto:auth_path=/pklxkd-46/obj2?versionId=null 2023-05-31T20:17:23.458 DEBUG:boto:Method: GET 2023-05-31T20:17:23.458 DEBUG:boto:Path: /pklxkd-46/obj2?versionId=null 2023-05-31T20:17:23.458 DEBUG:boto:Data: 2023-05-31T20:17:23.459 DEBUG:boto:Headers: {} 2023-05-31T20:17:23.459 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:17:23.459 DEBUG:boto:Port: 8001 2023-05-31T20:17:23.460 DEBUG:boto:Params: {} 2023-05-31T20:17:23.460 DEBUG:boto:Token: None 2023-05-31T20:17:23.461 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:17:23 GMT /pklxkd-46/obj2?versionId=null 2023-05-31T20:17:23.461 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:dWFd56rFRafj3kVf9FIhGAgKK5k= 2023-05-31T20:17:23.462 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:17:23 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:dWFd56rFRafj3kVf9FIhGAgKK5k=', 'Content-Length': '0'} 2023-05-31T20:17:23.463 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:15:11 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000020c96e918dcf0f7c-006477ab53-162e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:17:23 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:17:23.464 DEBUG:boto:path=/pklxkd-46/obj2 2023-05-31T20:17:23.464 DEBUG:boto:auth_path=/pklxkd-46/obj2 2023-05-31T20:17:23.465 DEBUG:boto:path=/pklxkd-46/obj2?versionId=null 2023-05-31T20:17:23.465 DEBUG:boto:auth_path=/pklxkd-46/obj2?versionId=null 2023-05-31T20:17:23.466 DEBUG:boto:Method: GET 2023-05-31T20:17:23.466 DEBUG:boto:Path: /pklxkd-46/obj2?versionId=null 2023-05-31T20:17:23.466 DEBUG:boto:Data: 2023-05-31T20:17:23.467 DEBUG:boto:Headers: {} 2023-05-31T20:17:23.467 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:17:23.467 DEBUG:boto:Port: 8000 2023-05-31T20:17:23.468 DEBUG:boto:Params: {} 2023-05-31T20:17:23.468 DEBUG:boto:Token: None 2023-05-31T20:17:23.469 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:17:23 GMT /pklxkd-46/obj2?versionId=null 2023-05-31T20:17:23.469 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:dWFd56rFRafj3kVf9FIhGAgKK5k= 2023-05-31T20:17:23.469 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:17:23 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:dWFd56rFRafj3kVf9FIhGAgKK5k=', 'Content-Length': '0'} 2023-05-31T20:17:23.471 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:15:11 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000039408b2593dd275c-006477ab53-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:17:23 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:17:23.472 DEBUG:boto:path=/pklxkd-46/ 2023-05-31T20:17:23.472 DEBUG:boto:auth_path=/pklxkd-46/ 2023-05-31T20:17:23.473 DEBUG:boto:path=/pklxkd-46/?prefix=obj2 2023-05-31T20:17:23.473 DEBUG:boto:auth_path=/pklxkd-46/?prefix=obj2 2023-05-31T20:17:23.473 DEBUG:boto:Method: GET 2023-05-31T20:17:23.474 DEBUG:boto:Path: /pklxkd-46/?prefix=obj2 2023-05-31T20:17:23.474 DEBUG:boto:Data: 2023-05-31T20:17:23.474 DEBUG:boto:Headers: {} 2023-05-31T20:17:23.475 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:17:23.475 DEBUG:boto:Port: 8001 2023-05-31T20:17:23.475 DEBUG:boto:Params: {} 2023-05-31T20:17:23.476 DEBUG:boto:Token: None 2023-05-31T20:17:23.476 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:17:23 GMT /pklxkd-46/ 2023-05-31T20:17:23.477 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:5JjXDyL51Vtp5kJiXsm2+F/F3Xg= 2023-05-31T20:17:23.477 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:17:23 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:5JjXDyL51Vtp5kJiXsm2+F/F3Xg=', 'Content-Length': '0'} 2023-05-31T20:17:23.480 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000003dea878be9289546-006477ab53-162e-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:17:23 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:17:23.480 DEBUG:boto:b'pklxkd-46obj21000falseobj22023-05-31T20:15:11.225Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:17:23.481 DEBUG:boto:path=/pklxkd-46/ 2023-05-31T20:17:23.481 DEBUG:boto:auth_path=/pklxkd-46/ 2023-05-31T20:17:23.482 DEBUG:boto:path=/pklxkd-46/?prefix=obj2 2023-05-31T20:17:23.482 DEBUG:boto:auth_path=/pklxkd-46/?prefix=obj2 2023-05-31T20:17:23.482 DEBUG:boto:Method: GET 2023-05-31T20:17:23.483 DEBUG:boto:Path: /pklxkd-46/?prefix=obj2 2023-05-31T20:17:23.483 DEBUG:boto:Data: 2023-05-31T20:17:23.483 DEBUG:boto:Headers: {} 2023-05-31T20:17:23.484 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:17:23.484 DEBUG:boto:Port: 8000 2023-05-31T20:17:23.485 DEBUG:boto:Params: {} 2023-05-31T20:17:23.485 DEBUG:boto:Token: None 2023-05-31T20:17:23.485 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:17:23 GMT /pklxkd-46/ 2023-05-31T20:17:23.486 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:5JjXDyL51Vtp5kJiXsm2+F/F3Xg= 2023-05-31T20:17:23.486 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:17:23 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:5JjXDyL51Vtp5kJiXsm2+F/F3Xg=', 'Content-Length': '0'} 2023-05-31T20:17:23.489 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000849154b5305d6ed7-006477ab53-1630-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:17:23 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:17:23.489 DEBUG:boto:b'pklxkd-46obj21000falseobj22023-05-31T20:15:11.225Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:17:23.490 DEBUG:boto:path=/pklxkd-46/obj2 2023-05-31T20:17:23.490 DEBUG:boto:auth_path=/pklxkd-46/obj2 2023-05-31T20:17:23.491 DEBUG:boto:Method: HEAD 2023-05-31T20:17:23.491 DEBUG:boto:Path: /pklxkd-46/obj2 2023-05-31T20:17:23.491 DEBUG:boto:Data: 2023-05-31T20:17:23.492 DEBUG:boto:Headers: {} 2023-05-31T20:17:23.492 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:17:23.493 DEBUG:boto:Port: 8001 2023-05-31T20:17:23.493 DEBUG:boto:Params: {} 2023-05-31T20:17:23.493 DEBUG:boto:Token: None 2023-05-31T20:17:23.494 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:17:23 GMT /pklxkd-46/obj2 2023-05-31T20:17:23.494 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:hfzOsgRLMpU2GSlNO3uOsG3fri8= 2023-05-31T20:17:23.494 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:17:23 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:hfzOsgRLMpU2GSlNO3uOsG3fri8=', 'Content-Length': '0'} 2023-05-31T20:17:23.496 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:15:11 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000001dbbe6213f140a55-006477ab53-162e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:17:23 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:17:23.497 DEBUG:boto:path=/pklxkd-46/obj2 2023-05-31T20:17:23.497 DEBUG:boto:auth_path=/pklxkd-46/obj2 2023-05-31T20:17:23.497 DEBUG:boto:Method: HEAD 2023-05-31T20:17:23.498 DEBUG:boto:Path: /pklxkd-46/obj2 2023-05-31T20:17:23.498 DEBUG:boto:Data: 2023-05-31T20:17:23.498 DEBUG:boto:Headers: {} 2023-05-31T20:17:23.499 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:17:23.499 DEBUG:boto:Port: 8000 2023-05-31T20:17:23.499 DEBUG:boto:Params: {} 2023-05-31T20:17:23.500 DEBUG:boto:Token: None 2023-05-31T20:17:23.500 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:17:23 GMT /pklxkd-46/obj2 2023-05-31T20:17:23.501 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:hfzOsgRLMpU2GSlNO3uOsG3fri8= 2023-05-31T20:17:23.501 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:17:23 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:hfzOsgRLMpU2GSlNO3uOsG3fri8=', 'Content-Length': '0'} 2023-05-31T20:17:23.503 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:15:11 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000e21b7fac6e4ba5c7-006477ab53-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:17:23 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:17:23.504 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2023-05-31T20:17:23.504 DEBUG:boto:path=/pklxkd-46/obj2 2023-05-31T20:17:23.504 DEBUG:boto:auth_path=/pklxkd-46/obj2 2023-05-31T20:17:23.505 DEBUG:boto:Method: GET 2023-05-31T20:17:23.505 DEBUG:boto:Path: /pklxkd-46/obj2 2023-05-31T20:17:23.506 DEBUG:boto:Data: 2023-05-31T20:17:23.506 DEBUG:boto:Headers: {} 2023-05-31T20:17:23.506 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:17:23.507 DEBUG:boto:Port: 8001 2023-05-31T20:17:23.507 DEBUG:boto:Params: {} 2023-05-31T20:17:23.507 DEBUG:boto:Token: None 2023-05-31T20:17:23.508 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:17:23 GMT /pklxkd-46/obj2 2023-05-31T20:17:23.508 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:3wwSwaRgIGDQrj+IZ61dqGB+ue0= 2023-05-31T20:17:23.509 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:17:23 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:3wwSwaRgIGDQrj+IZ61dqGB+ue0=', 'Content-Length': '0'} 2023-05-31T20:17:23.511 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:15:11 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000002da31388a4233110-006477ab53-162e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:17:23 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:17:23.511 DEBUG:boto:path=/pklxkd-46/obj2 2023-05-31T20:17:23.512 DEBUG:boto:auth_path=/pklxkd-46/obj2 2023-05-31T20:17:23.512 DEBUG:boto:Method: GET 2023-05-31T20:17:23.512 DEBUG:boto:Path: /pklxkd-46/obj2 2023-05-31T20:17:23.513 DEBUG:boto:Data: 2023-05-31T20:17:23.513 DEBUG:boto:Headers: {} 2023-05-31T20:17:23.514 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:17:23.514 DEBUG:boto:Port: 8000 2023-05-31T20:17:23.514 DEBUG:boto:Params: {} 2023-05-31T20:17:23.515 DEBUG:boto:Token: None 2023-05-31T20:17:23.515 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:17:23 GMT /pklxkd-46/obj2 2023-05-31T20:17:23.516 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:3wwSwaRgIGDQrj+IZ61dqGB+ue0= 2023-05-31T20:17:23.516 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:17:23 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:3wwSwaRgIGDQrj+IZ61dqGB+ue0=', 'Content-Length': '0'} 2023-05-31T20:17:23.518 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:15:11 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000da639ebd1b0a4bc7-006477ab53-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:17:23 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:17:23.519 DEBUG:boto:path=/pklxkd-46/ 2023-05-31T20:17:23.519 DEBUG:boto:auth_path=/pklxkd-46/ 2023-05-31T20:17:23.519 DEBUG:boto:path=/pklxkd-46/?prefix=obj2 2023-05-31T20:17:23.520 DEBUG:boto:auth_path=/pklxkd-46/?prefix=obj2 2023-05-31T20:17:23.520 DEBUG:boto:Method: GET 2023-05-31T20:17:23.521 DEBUG:boto:Path: /pklxkd-46/?prefix=obj2 2023-05-31T20:17:23.521 DEBUG:boto:Data: 2023-05-31T20:17:23.521 DEBUG:boto:Headers: {} 2023-05-31T20:17:23.522 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:17:23.522 DEBUG:boto:Port: 8001 2023-05-31T20:17:23.522 DEBUG:boto:Params: {} 2023-05-31T20:17:23.523 DEBUG:boto:Token: None 2023-05-31T20:17:23.523 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:17:23 GMT /pklxkd-46/ 2023-05-31T20:17:23.524 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:5JjXDyL51Vtp5kJiXsm2+F/F3Xg= 2023-05-31T20:17:23.524 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:17:23 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:5JjXDyL51Vtp5kJiXsm2+F/F3Xg=', 'Content-Length': '0'} 2023-05-31T20:17:23.526 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c95ce897bfe42153-006477ab53-162e-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:17:23 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:17:23.527 DEBUG:boto:b'pklxkd-46obj21000falseobj22023-05-31T20:15:11.225Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:17:23.528 DEBUG:boto:path=/pklxkd-46/ 2023-05-31T20:17:23.528 DEBUG:boto:auth_path=/pklxkd-46/ 2023-05-31T20:17:23.528 DEBUG:boto:path=/pklxkd-46/?prefix=obj2 2023-05-31T20:17:23.529 DEBUG:boto:auth_path=/pklxkd-46/?prefix=obj2 2023-05-31T20:17:23.529 DEBUG:boto:Method: GET 2023-05-31T20:17:23.529 DEBUG:boto:Path: /pklxkd-46/?prefix=obj2 2023-05-31T20:17:23.530 DEBUG:boto:Data: 2023-05-31T20:17:23.530 DEBUG:boto:Headers: {} 2023-05-31T20:17:23.530 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:17:23.531 DEBUG:boto:Port: 8000 2023-05-31T20:17:23.531 DEBUG:boto:Params: {} 2023-05-31T20:17:23.532 DEBUG:boto:Token: None 2023-05-31T20:17:23.532 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:17:23 GMT /pklxkd-46/ 2023-05-31T20:17:23.533 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:5JjXDyL51Vtp5kJiXsm2+F/F3Xg= 2023-05-31T20:17:23.533 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:17:23 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:5JjXDyL51Vtp5kJiXsm2+F/F3Xg=', 'Content-Length': '0'} 2023-05-31T20:17:23.536 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b086db6b0945f030-006477ab53-1630-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:17:23 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:17:23.536 DEBUG:boto:b'pklxkd-46obj21000falseobj22023-05-31T20:15:11.225Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:17:23.537 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2023-05-31T20:17:23.537 DEBUG:boto:path=/pklxkd-46/obj3 2023-05-31T20:17:23.538 DEBUG:boto:auth_path=/pklxkd-46/obj3 2023-05-31T20:17:23.538 DEBUG:boto:path=/pklxkd-46/obj3?versionId=null 2023-05-31T20:17:23.539 DEBUG:boto:auth_path=/pklxkd-46/obj3?versionId=null 2023-05-31T20:17:23.539 DEBUG:boto:Method: GET 2023-05-31T20:17:23.539 DEBUG:boto:Path: /pklxkd-46/obj3?versionId=null 2023-05-31T20:17:23.540 DEBUG:boto:Data: 2023-05-31T20:17:23.540 DEBUG:boto:Headers: {} 2023-05-31T20:17:23.540 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:17:23.541 DEBUG:boto:Port: 8001 2023-05-31T20:17:23.541 DEBUG:boto:Params: {} 2023-05-31T20:17:23.541 DEBUG:boto:Token: None 2023-05-31T20:17:23.542 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:17:23 GMT /pklxkd-46/obj3?versionId=null 2023-05-31T20:17:23.542 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:L4Pd/Yc9yI0dDfKFeDCM5lfTSVw= 2023-05-31T20:17:23.543 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:17:23 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:L4Pd/Yc9yI0dDfKFeDCM5lfTSVw=', 'Content-Length': '0'} 2023-05-31T20:17:23.545 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:15:11 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000003624750b4e6f72a8-006477ab53-162e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:17:23 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:17:23.546 DEBUG:boto:path=/pklxkd-46/obj3 2023-05-31T20:17:23.546 DEBUG:boto:auth_path=/pklxkd-46/obj3 2023-05-31T20:17:23.546 DEBUG:boto:path=/pklxkd-46/obj3?versionId=null 2023-05-31T20:17:23.547 DEBUG:boto:auth_path=/pklxkd-46/obj3?versionId=null 2023-05-31T20:17:23.547 DEBUG:boto:Method: GET 2023-05-31T20:17:23.547 DEBUG:boto:Path: /pklxkd-46/obj3?versionId=null 2023-05-31T20:17:23.548 DEBUG:boto:Data: 2023-05-31T20:17:23.548 DEBUG:boto:Headers: {} 2023-05-31T20:17:23.549 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:17:23.549 DEBUG:boto:Port: 8000 2023-05-31T20:17:23.549 DEBUG:boto:Params: {} 2023-05-31T20:17:23.550 DEBUG:boto:Token: None 2023-05-31T20:17:23.550 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:17:23 GMT /pklxkd-46/obj3?versionId=null 2023-05-31T20:17:23.551 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:L4Pd/Yc9yI0dDfKFeDCM5lfTSVw= 2023-05-31T20:17:23.551 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:17:23 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:L4Pd/Yc9yI0dDfKFeDCM5lfTSVw=', 'Content-Length': '0'} 2023-05-31T20:17:23.553 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:15:11 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000002a53eab1ad5b5a85-006477ab53-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:17:23 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:17:23.553 DEBUG:boto:path=/pklxkd-46/obj3 2023-05-31T20:17:23.554 DEBUG:boto:auth_path=/pklxkd-46/obj3 2023-05-31T20:17:23.554 DEBUG:boto:path=/pklxkd-46/obj3?versionId=null 2023-05-31T20:17:23.554 DEBUG:boto:auth_path=/pklxkd-46/obj3?versionId=null 2023-05-31T20:17:23.555 DEBUG:boto:Method: HEAD 2023-05-31T20:17:23.555 DEBUG:boto:Path: /pklxkd-46/obj3?versionId=null 2023-05-31T20:17:23.555 DEBUG:boto:Data: 2023-05-31T20:17:23.556 DEBUG:boto:Headers: {} 2023-05-31T20:17:23.556 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:17:23.556 DEBUG:boto:Port: 8001 2023-05-31T20:17:23.557 DEBUG:boto:Params: {} 2023-05-31T20:17:23.557 DEBUG:boto:Token: None 2023-05-31T20:17:23.558 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:17:23 GMT /pklxkd-46/obj3?versionId=null 2023-05-31T20:17:23.558 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:ITxfGjC6+chLc2mEA/TECQx5wbQ= 2023-05-31T20:17:23.558 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:17:23 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:ITxfGjC6+chLc2mEA/TECQx5wbQ=', 'Content-Length': '0'} 2023-05-31T20:17:23.560 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:15:11 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000b6b03ff2ffae9dd2-006477ab53-162e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:17:23 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:17:23.561 DEBUG:boto:path=/pklxkd-46/obj3 2023-05-31T20:17:23.561 DEBUG:boto:auth_path=/pklxkd-46/obj3 2023-05-31T20:17:23.562 DEBUG:boto:path=/pklxkd-46/obj3?versionId=null 2023-05-31T20:17:23.562 DEBUG:boto:auth_path=/pklxkd-46/obj3?versionId=null 2023-05-31T20:17:23.562 DEBUG:boto:Method: HEAD 2023-05-31T20:17:23.563 DEBUG:boto:Path: /pklxkd-46/obj3?versionId=null 2023-05-31T20:17:23.563 DEBUG:boto:Data: 2023-05-31T20:17:23.564 DEBUG:boto:Headers: {} 2023-05-31T20:17:23.564 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:17:23.564 DEBUG:boto:Port: 8000 2023-05-31T20:17:23.565 DEBUG:boto:Params: {} 2023-05-31T20:17:23.565 DEBUG:boto:Token: None 2023-05-31T20:17:23.566 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:17:23 GMT /pklxkd-46/obj3?versionId=null 2023-05-31T20:17:23.566 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:ITxfGjC6+chLc2mEA/TECQx5wbQ= 2023-05-31T20:17:23.566 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:17:23 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:ITxfGjC6+chLc2mEA/TECQx5wbQ=', 'Content-Length': '0'} 2023-05-31T20:17:23.568 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:15:11 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000c678faafc743a14c-006477ab53-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:17:23 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:17:23.569 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2023-05-31T20:17:23.569 DEBUG:boto:path=/pklxkd-46/obj3 2023-05-31T20:17:23.571 DEBUG:boto:auth_path=/pklxkd-46/obj3 2023-05-31T20:17:23.572 DEBUG:boto:path=/pklxkd-46/obj3?versionId=null 2023-05-31T20:17:23.572 DEBUG:boto:auth_path=/pklxkd-46/obj3?versionId=null 2023-05-31T20:17:23.573 DEBUG:boto:Method: GET 2023-05-31T20:17:23.573 DEBUG:boto:Path: /pklxkd-46/obj3?versionId=null 2023-05-31T20:17:23.573 DEBUG:boto:Data: 2023-05-31T20:17:23.574 DEBUG:boto:Headers: {} 2023-05-31T20:17:23.574 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:17:23.574 DEBUG:boto:Port: 8001 2023-05-31T20:17:23.575 DEBUG:boto:Params: {} 2023-05-31T20:17:23.575 DEBUG:boto:Token: None 2023-05-31T20:17:23.576 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:17:23 GMT /pklxkd-46/obj3?versionId=null 2023-05-31T20:17:23.576 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:L4Pd/Yc9yI0dDfKFeDCM5lfTSVw= 2023-05-31T20:17:23.577 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:17:23 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:L4Pd/Yc9yI0dDfKFeDCM5lfTSVw=', 'Content-Length': '0'} 2023-05-31T20:17:23.579 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:15:11 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000a543759ccab1cb71-006477ab53-162e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:17:23 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:17:23.620 DEBUG:boto:path=/pklxkd-46/obj3 2023-05-31T20:17:23.621 DEBUG:boto:auth_path=/pklxkd-46/obj3 2023-05-31T20:17:23.621 DEBUG:boto:path=/pklxkd-46/obj3?versionId=null 2023-05-31T20:17:23.622 DEBUG:boto:auth_path=/pklxkd-46/obj3?versionId=null 2023-05-31T20:17:23.622 DEBUG:boto:Method: GET 2023-05-31T20:17:23.623 DEBUG:boto:Path: /pklxkd-46/obj3?versionId=null 2023-05-31T20:17:23.623 DEBUG:boto:Data: 2023-05-31T20:17:23.624 DEBUG:boto:Headers: {} 2023-05-31T20:17:23.624 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:17:23.625 DEBUG:boto:Port: 8000 2023-05-31T20:17:23.625 DEBUG:boto:Params: {} 2023-05-31T20:17:23.626 DEBUG:boto:Token: None 2023-05-31T20:17:23.627 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:17:23 GMT /pklxkd-46/obj3?versionId=null 2023-05-31T20:17:23.627 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:L4Pd/Yc9yI0dDfKFeDCM5lfTSVw= 2023-05-31T20:17:23.628 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:17:23 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:L4Pd/Yc9yI0dDfKFeDCM5lfTSVw=', 'Content-Length': '0'} 2023-05-31T20:17:23.630 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:15:11 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000b83defe4f3ac515e-006477ab53-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:17:23 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:17:23.631 DEBUG:boto:path=/pklxkd-46/ 2023-05-31T20:17:23.631 DEBUG:boto:auth_path=/pklxkd-46/ 2023-05-31T20:17:23.632 DEBUG:boto:path=/pklxkd-46/?prefix=obj3 2023-05-31T20:17:23.632 DEBUG:boto:auth_path=/pklxkd-46/?prefix=obj3 2023-05-31T20:17:23.632 DEBUG:boto:Method: GET 2023-05-31T20:17:23.633 DEBUG:boto:Path: /pklxkd-46/?prefix=obj3 2023-05-31T20:17:23.633 DEBUG:boto:Data: 2023-05-31T20:17:23.634 DEBUG:boto:Headers: {} 2023-05-31T20:17:23.634 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:17:23.634 DEBUG:boto:Port: 8001 2023-05-31T20:17:23.635 DEBUG:boto:Params: {} 2023-05-31T20:17:23.636 DEBUG:boto:Token: None 2023-05-31T20:17:23.636 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:17:23 GMT /pklxkd-46/ 2023-05-31T20:17:23.638 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:5JjXDyL51Vtp5kJiXsm2+F/F3Xg= 2023-05-31T20:17:23.639 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:17:23 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:5JjXDyL51Vtp5kJiXsm2+F/F3Xg=', 'Content-Length': '0'} 2023-05-31T20:17:23.641 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000504432a499a229fa-006477ab53-162e-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:17:23 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:17:23.642 DEBUG:boto:b'pklxkd-46obj31000falseobj32023-05-31T20:15:11.255Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:17:23.643 DEBUG:boto:path=/pklxkd-46/ 2023-05-31T20:17:23.643 DEBUG:boto:auth_path=/pklxkd-46/ 2023-05-31T20:17:23.644 DEBUG:boto:path=/pklxkd-46/?prefix=obj3 2023-05-31T20:17:23.644 DEBUG:boto:auth_path=/pklxkd-46/?prefix=obj3 2023-05-31T20:17:23.644 DEBUG:boto:Method: GET 2023-05-31T20:17:23.645 DEBUG:boto:Path: /pklxkd-46/?prefix=obj3 2023-05-31T20:17:23.645 DEBUG:boto:Data: 2023-05-31T20:17:23.645 DEBUG:boto:Headers: {} 2023-05-31T20:17:23.646 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:17:23.647 DEBUG:boto:Port: 8000 2023-05-31T20:17:23.648 DEBUG:boto:Params: {} 2023-05-31T20:17:23.649 DEBUG:boto:Token: None 2023-05-31T20:17:23.650 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:17:23 GMT /pklxkd-46/ 2023-05-31T20:17:23.651 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:5JjXDyL51Vtp5kJiXsm2+F/F3Xg= 2023-05-31T20:17:23.652 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:17:23 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:5JjXDyL51Vtp5kJiXsm2+F/F3Xg=', 'Content-Length': '0'} 2023-05-31T20:17:23.654 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e59abc5a8d207f4f-006477ab53-1630-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:17:23 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:17:23.655 DEBUG:boto:b'pklxkd-46obj31000falseobj32023-05-31T20:15:11.255Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:17:23.655 DEBUG:boto:path=/pklxkd-46/obj3 2023-05-31T20:17:23.656 DEBUG:boto:auth_path=/pklxkd-46/obj3 2023-05-31T20:17:23.657 DEBUG:boto:Method: HEAD 2023-05-31T20:17:23.657 DEBUG:boto:Path: /pklxkd-46/obj3 2023-05-31T20:17:23.657 DEBUG:boto:Data: 2023-05-31T20:17:23.658 DEBUG:boto:Headers: {} 2023-05-31T20:17:23.658 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:17:23.658 DEBUG:boto:Port: 8001 2023-05-31T20:17:23.659 DEBUG:boto:Params: {} 2023-05-31T20:17:23.659 DEBUG:boto:Token: None 2023-05-31T20:17:23.660 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:17:23 GMT /pklxkd-46/obj3 2023-05-31T20:17:23.660 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:OZSkl4LSJG3qqRbnQvdCXXqbDSw= 2023-05-31T20:17:23.660 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:17:23 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:OZSkl4LSJG3qqRbnQvdCXXqbDSw=', 'Content-Length': '0'} 2023-05-31T20:17:23.663 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:15:11 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000025b0321c0e937a0c-006477ab53-162e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:17:23 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:17:23.664 DEBUG:boto:path=/pklxkd-46/obj3 2023-05-31T20:17:23.665 DEBUG:boto:auth_path=/pklxkd-46/obj3 2023-05-31T20:17:23.666 DEBUG:boto:Method: HEAD 2023-05-31T20:17:23.666 DEBUG:boto:Path: /pklxkd-46/obj3 2023-05-31T20:17:23.666 DEBUG:boto:Data: 2023-05-31T20:17:23.667 DEBUG:boto:Headers: {} 2023-05-31T20:17:23.667 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:17:23.668 DEBUG:boto:Port: 8000 2023-05-31T20:17:23.668 DEBUG:boto:Params: {} 2023-05-31T20:17:23.669 DEBUG:boto:Token: None 2023-05-31T20:17:23.669 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:17:23 GMT /pklxkd-46/obj3 2023-05-31T20:17:23.670 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:OZSkl4LSJG3qqRbnQvdCXXqbDSw= 2023-05-31T20:17:23.670 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:17:23 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:OZSkl4LSJG3qqRbnQvdCXXqbDSw=', 'Content-Length': '0'} 2023-05-31T20:17:23.673 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:15:11 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000002c29faf33f8d93d6-006477ab53-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:17:23 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:17:23.674 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2023-05-31T20:17:23.675 DEBUG:boto:path=/pklxkd-46/obj3 2023-05-31T20:17:23.675 DEBUG:boto:auth_path=/pklxkd-46/obj3 2023-05-31T20:17:23.676 DEBUG:boto:Method: GET 2023-05-31T20:17:23.677 DEBUG:boto:Path: /pklxkd-46/obj3 2023-05-31T20:17:23.677 DEBUG:boto:Data: 2023-05-31T20:17:23.678 DEBUG:boto:Headers: {} 2023-05-31T20:17:23.679 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:17:23.679 DEBUG:boto:Port: 8001 2023-05-31T20:17:23.680 DEBUG:boto:Params: {} 2023-05-31T20:17:23.681 DEBUG:boto:Token: None 2023-05-31T20:17:23.682 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:17:23 GMT /pklxkd-46/obj3 2023-05-31T20:17:23.682 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:Iw30sbTzMt7TB8h/NlmHhpedT5Q= 2023-05-31T20:17:23.683 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:17:23 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:Iw30sbTzMt7TB8h/NlmHhpedT5Q=', 'Content-Length': '0'} 2023-05-31T20:17:23.685 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:15:11 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000289167872202aeee-006477ab53-162e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:17:23 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:17:23.686 DEBUG:boto:path=/pklxkd-46/obj3 2023-05-31T20:17:23.687 DEBUG:boto:auth_path=/pklxkd-46/obj3 2023-05-31T20:17:23.688 DEBUG:boto:Method: GET 2023-05-31T20:17:23.688 DEBUG:boto:Path: /pklxkd-46/obj3 2023-05-31T20:17:23.689 DEBUG:boto:Data: 2023-05-31T20:17:23.690 DEBUG:boto:Headers: {} 2023-05-31T20:17:23.690 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:17:23.690 DEBUG:boto:Port: 8000 2023-05-31T20:17:23.691 DEBUG:boto:Params: {} 2023-05-31T20:17:23.691 DEBUG:boto:Token: None 2023-05-31T20:17:23.691 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:17:23 GMT /pklxkd-46/obj3 2023-05-31T20:17:23.692 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:Iw30sbTzMt7TB8h/NlmHhpedT5Q= 2023-05-31T20:17:23.692 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:17:23 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:Iw30sbTzMt7TB8h/NlmHhpedT5Q=', 'Content-Length': '0'} 2023-05-31T20:17:23.694 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:15:11 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000b92ab5ffe17d671f-006477ab53-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:17:23 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:17:23.695 DEBUG:boto:path=/pklxkd-46/ 2023-05-31T20:17:23.695 DEBUG:boto:auth_path=/pklxkd-46/ 2023-05-31T20:17:23.696 DEBUG:boto:path=/pklxkd-46/?prefix=obj3 2023-05-31T20:17:23.696 DEBUG:boto:auth_path=/pklxkd-46/?prefix=obj3 2023-05-31T20:17:23.696 DEBUG:boto:Method: GET 2023-05-31T20:17:23.697 DEBUG:boto:Path: /pklxkd-46/?prefix=obj3 2023-05-31T20:17:23.697 DEBUG:boto:Data: 2023-05-31T20:17:23.698 DEBUG:boto:Headers: {} 2023-05-31T20:17:23.698 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:17:23.699 DEBUG:boto:Port: 8001 2023-05-31T20:17:23.699 DEBUG:boto:Params: {} 2023-05-31T20:17:23.699 DEBUG:boto:Token: None 2023-05-31T20:17:23.700 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:17:23 GMT /pklxkd-46/ 2023-05-31T20:17:23.700 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:5JjXDyL51Vtp5kJiXsm2+F/F3Xg= 2023-05-31T20:17:23.701 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:17:23 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:5JjXDyL51Vtp5kJiXsm2+F/F3Xg=', 'Content-Length': '0'} 2023-05-31T20:17:23.703 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f5ef11ec6d4b0243-006477ab53-162e-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:17:23 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:17:23.748 DEBUG:boto:b'pklxkd-46obj31000falseobj32023-05-31T20:15:11.255Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:17:23.749 DEBUG:boto:path=/pklxkd-46/ 2023-05-31T20:17:23.750 DEBUG:boto:auth_path=/pklxkd-46/ 2023-05-31T20:17:23.750 DEBUG:boto:path=/pklxkd-46/?prefix=obj3 2023-05-31T20:17:23.750 DEBUG:boto:auth_path=/pklxkd-46/?prefix=obj3 2023-05-31T20:17:23.751 DEBUG:boto:Method: GET 2023-05-31T20:17:23.751 DEBUG:boto:Path: /pklxkd-46/?prefix=obj3 2023-05-31T20:17:23.751 DEBUG:boto:Data: 2023-05-31T20:17:23.752 DEBUG:boto:Headers: {} 2023-05-31T20:17:23.752 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:17:23.752 DEBUG:boto:Port: 8000 2023-05-31T20:17:23.753 DEBUG:boto:Params: {} 2023-05-31T20:17:23.753 DEBUG:boto:Token: None 2023-05-31T20:17:23.754 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:17:23 GMT /pklxkd-46/ 2023-05-31T20:17:23.754 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:5JjXDyL51Vtp5kJiXsm2+F/F3Xg= 2023-05-31T20:17:23.755 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:17:23 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:5JjXDyL51Vtp5kJiXsm2+F/F3Xg=', 'Content-Length': '0'} 2023-05-31T20:17:23.757 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000002912666149b5e876-006477ab53-1630-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:17:23 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:17:23.758 DEBUG:boto:b'pklxkd-46obj31000falseobj32023-05-31T20:15:11.255Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:17:23.758 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2023-05-31T20:17:23.759 DEBUG:boto:path=/pklxkd-46/obj4 2023-05-31T20:17:23.759 DEBUG:boto:auth_path=/pklxkd-46/obj4 2023-05-31T20:17:23.759 DEBUG:boto:path=/pklxkd-46/obj4?versionId=null 2023-05-31T20:17:23.760 DEBUG:boto:auth_path=/pklxkd-46/obj4?versionId=null 2023-05-31T20:17:23.760 DEBUG:boto:Method: GET 2023-05-31T20:17:23.761 DEBUG:boto:Path: /pklxkd-46/obj4?versionId=null 2023-05-31T20:17:23.761 DEBUG:boto:Data: 2023-05-31T20:17:23.761 DEBUG:boto:Headers: {} 2023-05-31T20:17:23.762 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:17:23.762 DEBUG:boto:Port: 8001 2023-05-31T20:17:23.762 DEBUG:boto:Params: {} 2023-05-31T20:17:23.763 DEBUG:boto:Token: None 2023-05-31T20:17:23.763 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:17:23 GMT /pklxkd-46/obj4?versionId=null 2023-05-31T20:17:23.764 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:3RddTX+v3lUChF2Qp2GsQ3PbVoE= 2023-05-31T20:17:23.764 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:17:23 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:3RddTX+v3lUChF2Qp2GsQ3PbVoE=', 'Content-Length': '0'} 2023-05-31T20:17:23.766 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:15:11 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000c15b4fc71e3b3b58-006477ab53-162e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:17:23 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:17:23.767 DEBUG:boto:path=/pklxkd-46/obj4 2023-05-31T20:17:23.767 DEBUG:boto:auth_path=/pklxkd-46/obj4 2023-05-31T20:17:23.767 DEBUG:boto:path=/pklxkd-46/obj4?versionId=null 2023-05-31T20:17:23.768 DEBUG:boto:auth_path=/pklxkd-46/obj4?versionId=null 2023-05-31T20:17:23.768 DEBUG:boto:Method: GET 2023-05-31T20:17:23.769 DEBUG:boto:Path: /pklxkd-46/obj4?versionId=null 2023-05-31T20:17:23.769 DEBUG:boto:Data: 2023-05-31T20:17:23.769 DEBUG:boto:Headers: {} 2023-05-31T20:17:23.770 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:17:23.770 DEBUG:boto:Port: 8000 2023-05-31T20:17:23.770 DEBUG:boto:Params: {} 2023-05-31T20:17:23.771 DEBUG:boto:Token: None 2023-05-31T20:17:23.771 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:17:23 GMT /pklxkd-46/obj4?versionId=null 2023-05-31T20:17:23.772 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:3RddTX+v3lUChF2Qp2GsQ3PbVoE= 2023-05-31T20:17:23.772 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:17:23 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:3RddTX+v3lUChF2Qp2GsQ3PbVoE=', 'Content-Length': '0'} 2023-05-31T20:17:23.774 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:15:11 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000a35fe5dacc59f56d-006477ab53-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:17:23 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:17:23.775 DEBUG:boto:path=/pklxkd-46/obj4 2023-05-31T20:17:23.775 DEBUG:boto:auth_path=/pklxkd-46/obj4 2023-05-31T20:17:23.776 DEBUG:boto:path=/pklxkd-46/obj4?versionId=null 2023-05-31T20:17:23.776 DEBUG:boto:auth_path=/pklxkd-46/obj4?versionId=null 2023-05-31T20:17:23.776 DEBUG:boto:Method: HEAD 2023-05-31T20:17:23.777 DEBUG:boto:Path: /pklxkd-46/obj4?versionId=null 2023-05-31T20:17:23.777 DEBUG:boto:Data: 2023-05-31T20:17:23.777 DEBUG:boto:Headers: {} 2023-05-31T20:17:23.778 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:17:23.778 DEBUG:boto:Port: 8001 2023-05-31T20:17:23.778 DEBUG:boto:Params: {} 2023-05-31T20:17:23.779 DEBUG:boto:Token: None 2023-05-31T20:17:23.779 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:17:23 GMT /pklxkd-46/obj4?versionId=null 2023-05-31T20:17:23.779 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:S9FYLJS4BSdlcOCCcgnNJ7BhNng= 2023-05-31T20:17:23.780 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:17:23 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:S9FYLJS4BSdlcOCCcgnNJ7BhNng=', 'Content-Length': '0'} 2023-05-31T20:17:23.782 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:15:11 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000071869f9f349a3da6-006477ab53-162e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:17:23 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:17:23.782 DEBUG:boto:path=/pklxkd-46/obj4 2023-05-31T20:17:23.783 DEBUG:boto:auth_path=/pklxkd-46/obj4 2023-05-31T20:17:23.783 DEBUG:boto:path=/pklxkd-46/obj4?versionId=null 2023-05-31T20:17:23.783 DEBUG:boto:auth_path=/pklxkd-46/obj4?versionId=null 2023-05-31T20:17:23.784 DEBUG:boto:Method: HEAD 2023-05-31T20:17:23.784 DEBUG:boto:Path: /pklxkd-46/obj4?versionId=null 2023-05-31T20:17:23.785 DEBUG:boto:Data: 2023-05-31T20:17:23.785 DEBUG:boto:Headers: {} 2023-05-31T20:17:23.785 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:17:23.785 DEBUG:boto:Port: 8000 2023-05-31T20:17:23.786 DEBUG:boto:Params: {} 2023-05-31T20:17:23.786 DEBUG:boto:Token: None 2023-05-31T20:17:23.787 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:17:23 GMT /pklxkd-46/obj4?versionId=null 2023-05-31T20:17:23.787 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:S9FYLJS4BSdlcOCCcgnNJ7BhNng= 2023-05-31T20:17:23.787 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:17:23 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:S9FYLJS4BSdlcOCCcgnNJ7BhNng=', 'Content-Length': '0'} 2023-05-31T20:17:23.790 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:15:11 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000a59bc9a6973b34b0-006477ab53-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:17:23 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:17:23.790 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2023-05-31T20:17:23.790 DEBUG:boto:path=/pklxkd-46/obj4 2023-05-31T20:17:23.791 DEBUG:boto:auth_path=/pklxkd-46/obj4 2023-05-31T20:17:23.791 DEBUG:boto:path=/pklxkd-46/obj4?versionId=null 2023-05-31T20:17:23.792 DEBUG:boto:auth_path=/pklxkd-46/obj4?versionId=null 2023-05-31T20:17:23.792 DEBUG:boto:Method: GET 2023-05-31T20:17:23.792 DEBUG:boto:Path: /pklxkd-46/obj4?versionId=null 2023-05-31T20:17:23.793 DEBUG:boto:Data: 2023-05-31T20:17:23.793 DEBUG:boto:Headers: {} 2023-05-31T20:17:23.793 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:17:23.794 DEBUG:boto:Port: 8001 2023-05-31T20:17:23.794 DEBUG:boto:Params: {} 2023-05-31T20:17:23.794 DEBUG:boto:Token: None 2023-05-31T20:17:23.795 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:17:23 GMT /pklxkd-46/obj4?versionId=null 2023-05-31T20:17:23.795 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:3RddTX+v3lUChF2Qp2GsQ3PbVoE= 2023-05-31T20:17:23.796 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:17:23 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:3RddTX+v3lUChF2Qp2GsQ3PbVoE=', 'Content-Length': '0'} 2023-05-31T20:17:23.798 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:15:11 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000e1383de661f24fa2-006477ab53-162e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:17:23 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:17:23.798 DEBUG:boto:path=/pklxkd-46/obj4 2023-05-31T20:17:23.799 DEBUG:boto:auth_path=/pklxkd-46/obj4 2023-05-31T20:17:23.799 DEBUG:boto:path=/pklxkd-46/obj4?versionId=null 2023-05-31T20:17:23.799 DEBUG:boto:auth_path=/pklxkd-46/obj4?versionId=null 2023-05-31T20:17:23.800 DEBUG:boto:Method: GET 2023-05-31T20:17:23.800 DEBUG:boto:Path: /pklxkd-46/obj4?versionId=null 2023-05-31T20:17:23.800 DEBUG:boto:Data: 2023-05-31T20:17:23.801 DEBUG:boto:Headers: {} 2023-05-31T20:17:23.801 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:17:23.802 DEBUG:boto:Port: 8000 2023-05-31T20:17:23.802 DEBUG:boto:Params: {} 2023-05-31T20:17:23.802 DEBUG:boto:Token: None 2023-05-31T20:17:23.803 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:17:23 GMT /pklxkd-46/obj4?versionId=null 2023-05-31T20:17:23.803 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:3RddTX+v3lUChF2Qp2GsQ3PbVoE= 2023-05-31T20:17:23.804 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:17:23 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:3RddTX+v3lUChF2Qp2GsQ3PbVoE=', 'Content-Length': '0'} 2023-05-31T20:17:23.806 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:15:11 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000023ec8c926e500058-006477ab53-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:17:23 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:17:23.806 DEBUG:boto:path=/pklxkd-46/ 2023-05-31T20:17:23.807 DEBUG:boto:auth_path=/pklxkd-46/ 2023-05-31T20:17:23.807 DEBUG:boto:path=/pklxkd-46/?prefix=obj4 2023-05-31T20:17:23.808 DEBUG:boto:auth_path=/pklxkd-46/?prefix=obj4 2023-05-31T20:17:23.808 DEBUG:boto:Method: GET 2023-05-31T20:17:23.808 DEBUG:boto:Path: /pklxkd-46/?prefix=obj4 2023-05-31T20:17:23.809 DEBUG:boto:Data: 2023-05-31T20:17:23.809 DEBUG:boto:Headers: {} 2023-05-31T20:17:23.809 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:17:23.810 DEBUG:boto:Port: 8001 2023-05-31T20:17:23.810 DEBUG:boto:Params: {} 2023-05-31T20:17:23.811 DEBUG:boto:Token: None 2023-05-31T20:17:23.811 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:17:23 GMT /pklxkd-46/ 2023-05-31T20:17:23.811 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:5JjXDyL51Vtp5kJiXsm2+F/F3Xg= 2023-05-31T20:17:23.812 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:17:23 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:5JjXDyL51Vtp5kJiXsm2+F/F3Xg=', 'Content-Length': '0'} 2023-05-31T20:17:23.814 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000001952252fdac6ab7f-006477ab53-162e-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:17:23 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:17:23.815 DEBUG:boto:b'pklxkd-46obj41000falseobj42023-05-31T20:15:11.317Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:17:23.815 DEBUG:boto:path=/pklxkd-46/ 2023-05-31T20:17:23.816 DEBUG:boto:auth_path=/pklxkd-46/ 2023-05-31T20:17:23.816 DEBUG:boto:path=/pklxkd-46/?prefix=obj4 2023-05-31T20:17:23.817 DEBUG:boto:auth_path=/pklxkd-46/?prefix=obj4 2023-05-31T20:17:23.817 DEBUG:boto:Method: GET 2023-05-31T20:17:23.817 DEBUG:boto:Path: /pklxkd-46/?prefix=obj4 2023-05-31T20:17:23.818 DEBUG:boto:Data: 2023-05-31T20:17:23.818 DEBUG:boto:Headers: {} 2023-05-31T20:17:23.818 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:17:23.819 DEBUG:boto:Port: 8000 2023-05-31T20:17:23.819 DEBUG:boto:Params: {} 2023-05-31T20:17:23.819 DEBUG:boto:Token: None 2023-05-31T20:17:23.820 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:17:23 GMT /pklxkd-46/ 2023-05-31T20:17:23.820 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:5JjXDyL51Vtp5kJiXsm2+F/F3Xg= 2023-05-31T20:17:23.821 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:17:23 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:5JjXDyL51Vtp5kJiXsm2+F/F3Xg=', 'Content-Length': '0'} 2023-05-31T20:17:23.823 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000058dfe956c52259e8-006477ab53-1630-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:17:23 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:17:23.824 DEBUG:boto:b'pklxkd-46obj41000falseobj42023-05-31T20:15:11.317Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:17:23.824 DEBUG:boto:path=/pklxkd-46/obj4 2023-05-31T20:17:23.825 DEBUG:boto:auth_path=/pklxkd-46/obj4 2023-05-31T20:17:23.825 DEBUG:boto:Method: HEAD 2023-05-31T20:17:23.826 DEBUG:boto:Path: /pklxkd-46/obj4 2023-05-31T20:17:23.826 DEBUG:boto:Data: 2023-05-31T20:17:23.826 DEBUG:boto:Headers: {} 2023-05-31T20:17:23.827 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:17:23.827 DEBUG:boto:Port: 8001 2023-05-31T20:17:23.827 DEBUG:boto:Params: {} 2023-05-31T20:17:23.828 DEBUG:boto:Token: None 2023-05-31T20:17:23.828 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:17:23 GMT /pklxkd-46/obj4 2023-05-31T20:17:23.828 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:ApGDFfQv9CR2a1iuL+0j4IXD4is= 2023-05-31T20:17:23.829 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:17:23 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:ApGDFfQv9CR2a1iuL+0j4IXD4is=', 'Content-Length': '0'} 2023-05-31T20:17:23.831 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:15:11 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000c29b4779b6792db9-006477ab53-162e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:17:23 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:17:23.832 DEBUG:boto:path=/pklxkd-46/obj4 2023-05-31T20:17:23.832 DEBUG:boto:auth_path=/pklxkd-46/obj4 2023-05-31T20:17:23.833 DEBUG:boto:Method: HEAD 2023-05-31T20:17:23.834 DEBUG:boto:Path: /pklxkd-46/obj4 2023-05-31T20:17:23.834 DEBUG:boto:Data: 2023-05-31T20:17:23.834 DEBUG:boto:Headers: {} 2023-05-31T20:17:23.835 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:17:23.835 DEBUG:boto:Port: 8000 2023-05-31T20:17:23.835 DEBUG:boto:Params: {} 2023-05-31T20:17:23.836 DEBUG:boto:Token: None 2023-05-31T20:17:23.836 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:17:23 GMT /pklxkd-46/obj4 2023-05-31T20:17:23.838 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:ApGDFfQv9CR2a1iuL+0j4IXD4is= 2023-05-31T20:17:23.838 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:17:23 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:ApGDFfQv9CR2a1iuL+0j4IXD4is=', 'Content-Length': '0'} 2023-05-31T20:17:23.840 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:15:11 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000001d64dca61c115b6e-006477ab53-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:17:23 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:17:23.841 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2023-05-31T20:17:23.842 DEBUG:boto:path=/pklxkd-46/obj4 2023-05-31T20:17:23.843 DEBUG:boto:auth_path=/pklxkd-46/obj4 2023-05-31T20:17:23.844 DEBUG:boto:Method: GET 2023-05-31T20:17:23.845 DEBUG:boto:Path: /pklxkd-46/obj4 2023-05-31T20:17:23.846 DEBUG:boto:Data: 2023-05-31T20:17:23.846 DEBUG:boto:Headers: {} 2023-05-31T20:17:23.847 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:17:23.848 DEBUG:boto:Port: 8001 2023-05-31T20:17:23.849 DEBUG:boto:Params: {} 2023-05-31T20:17:23.850 DEBUG:boto:Token: None 2023-05-31T20:17:23.851 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:17:23 GMT /pklxkd-46/obj4 2023-05-31T20:17:23.852 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:pviIcpBX4q8301tz1oI2YNsrBgo= 2023-05-31T20:17:23.853 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:17:23 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:pviIcpBX4q8301tz1oI2YNsrBgo=', 'Content-Length': '0'} 2023-05-31T20:17:23.856 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:15:11 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000008727ecb8ffaf6ea7-006477ab53-162e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:17:23 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:17:23.856 DEBUG:boto:path=/pklxkd-46/obj4 2023-05-31T20:17:23.857 DEBUG:boto:auth_path=/pklxkd-46/obj4 2023-05-31T20:17:23.857 DEBUG:boto:Method: GET 2023-05-31T20:17:23.857 DEBUG:boto:Path: /pklxkd-46/obj4 2023-05-31T20:17:23.858 DEBUG:boto:Data: 2023-05-31T20:17:23.858 DEBUG:boto:Headers: {} 2023-05-31T20:17:23.859 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:17:23.859 DEBUG:boto:Port: 8000 2023-05-31T20:17:23.859 DEBUG:boto:Params: {} 2023-05-31T20:17:23.860 DEBUG:boto:Token: None 2023-05-31T20:17:23.860 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:17:23 GMT /pklxkd-46/obj4 2023-05-31T20:17:23.861 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:pviIcpBX4q8301tz1oI2YNsrBgo= 2023-05-31T20:17:23.861 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:17:23 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:pviIcpBX4q8301tz1oI2YNsrBgo=', 'Content-Length': '0'} 2023-05-31T20:17:23.863 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:15:11 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000b23d0c6b43035d97-006477ab53-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:17:23 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:17:23.904 DEBUG:boto:path=/pklxkd-46/ 2023-05-31T20:17:23.905 DEBUG:boto:auth_path=/pklxkd-46/ 2023-05-31T20:17:23.905 DEBUG:boto:path=/pklxkd-46/?prefix=obj4 2023-05-31T20:17:23.906 DEBUG:boto:auth_path=/pklxkd-46/?prefix=obj4 2023-05-31T20:17:23.906 DEBUG:boto:Method: GET 2023-05-31T20:17:23.907 DEBUG:boto:Path: /pklxkd-46/?prefix=obj4 2023-05-31T20:17:23.907 DEBUG:boto:Data: 2023-05-31T20:17:23.908 DEBUG:boto:Headers: {} 2023-05-31T20:17:23.908 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:17:23.909 DEBUG:boto:Port: 8001 2023-05-31T20:17:23.909 DEBUG:boto:Params: {} 2023-05-31T20:17:23.909 DEBUG:boto:Token: None 2023-05-31T20:17:23.910 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:17:23 GMT /pklxkd-46/ 2023-05-31T20:17:23.910 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:5JjXDyL51Vtp5kJiXsm2+F/F3Xg= 2023-05-31T20:17:23.911 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:17:23 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:5JjXDyL51Vtp5kJiXsm2+F/F3Xg=', 'Content-Length': '0'} 2023-05-31T20:17:23.913 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000103c52ba8fc5abb6-006477ab53-162e-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:17:23 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:17:23.914 DEBUG:boto:b'pklxkd-46obj41000falseobj42023-05-31T20:15:11.317Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:17:23.914 DEBUG:boto:path=/pklxkd-46/ 2023-05-31T20:17:23.915 DEBUG:boto:auth_path=/pklxkd-46/ 2023-05-31T20:17:23.915 DEBUG:boto:path=/pklxkd-46/?prefix=obj4 2023-05-31T20:17:23.916 DEBUG:boto:auth_path=/pklxkd-46/?prefix=obj4 2023-05-31T20:17:23.916 DEBUG:boto:Method: GET 2023-05-31T20:17:23.916 DEBUG:boto:Path: /pklxkd-46/?prefix=obj4 2023-05-31T20:17:23.917 DEBUG:boto:Data: 2023-05-31T20:17:23.917 DEBUG:boto:Headers: {} 2023-05-31T20:17:23.918 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:17:23.918 DEBUG:boto:Port: 8000 2023-05-31T20:17:23.918 DEBUG:boto:Params: {} 2023-05-31T20:17:23.919 DEBUG:boto:Token: None 2023-05-31T20:17:23.919 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:17:23 GMT /pklxkd-46/ 2023-05-31T20:17:23.920 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:5JjXDyL51Vtp5kJiXsm2+F/F3Xg= 2023-05-31T20:17:23.920 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:17:23 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:5JjXDyL51Vtp5kJiXsm2+F/F3Xg=', 'Content-Length': '0'} 2023-05-31T20:17:23.922 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000007f0dae663a34a7b2-006477ab53-1630-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:17:23 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:17:23.923 DEBUG:boto:b'pklxkd-46obj41000falseobj42023-05-31T20:15:11.317Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:17:23.924 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=pklxkd-46 zones={a2, a1} 2023-05-31T20:17:23.924 DEBUG:rgw_multi.tests:bucket checkpoint: source=a1 target=a2 bucket=pklxkd-47 2023-05-31T20:17:23.924 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'pklxkd-47', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-05-31T20:17:23.925 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'pklxkd-47', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-05-31T20:17:23.926 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket pklxkd-47 --source-zone a1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2023-05-31T20:17:23.990 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2023-05-31T20:17:23.991 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2023-05-31T20:17:24.032 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:17:24.028+0000 7f74c56f4780 1 bucket sync caught up with source: 2023-05-31T20:17:24.033 INFO:teuthology.orchestra.run.smithi099.stderr: local status: [00000000001.21.6, , 00000000001.24.6, , 00000000001.26.6, , , 00000000001.95.6, , , ] 2023-05-31T20:17:24.033 INFO:teuthology.orchestra.run.smithi099.stderr: remote markers: [00000000001.21.6, , 00000000001.24.6, , 00000000001.26.6, , , 00000000001.95.6, , , ] 2023-05-31T20:17:24.033 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:17:24.028+0000 7f74c56f4780 0 bucket checkpoint complete 2023-05-31T20:17:24.048 DEBUG:rgw_multi.tests:bucket checkpoint: source=a2 target=a1 bucket=pklxkd-47 2023-05-31T20:17:24.049 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'pklxkd-47', '--source-zone', 'a2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-05-31T20:17:24.049 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'pklxkd-47', '--source-zone', 'a2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-05-31T20:17:24.050 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket pklxkd-47 --source-zone a2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2023-05-31T20:17:24.108 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2023-05-31T20:17:24.108 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2023-05-31T20:17:24.163 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:17:24.158+0000 7fa0ccefb780 1 bucket sync caught up with source: 2023-05-31T20:17:24.163 INFO:teuthology.orchestra.run.smithi099.stderr: local status: [00000000001.605.6, , 00000000001.1669.6, , 00000000001.3029.6, , , 00000000001.2023.6, , , ] 2023-05-31T20:17:24.163 INFO:teuthology.orchestra.run.smithi099.stderr: remote markers: [00000000001.605.6, , 00000000001.1669.6, , 00000000001.3029.6, , , 00000000001.2023.6, , , ] 2023-05-31T20:17:24.163 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:17:24.158+0000 7fa0ccefb780 0 bucket checkpoint complete 2023-05-31T20:17:24.189 INFO:tasks.rgw_multi.zone_rados:comparing bucket=pklxkd-47 zones={a2, a1} 2023-05-31T20:17:24.190 DEBUG:boto:path=/pklxkd-47/ 2023-05-31T20:17:24.190 DEBUG:boto:auth_path=/pklxkd-47/ 2023-05-31T20:17:24.190 DEBUG:boto:Method: HEAD 2023-05-31T20:17:24.191 DEBUG:boto:Path: /pklxkd-47/ 2023-05-31T20:17:24.191 DEBUG:boto:Data: 2023-05-31T20:17:24.192 DEBUG:boto:Headers: {} 2023-05-31T20:17:24.192 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:17:24.192 DEBUG:boto:Port: 8001 2023-05-31T20:17:24.193 DEBUG:boto:Params: {} 2023-05-31T20:17:24.193 DEBUG:boto:Token: None 2023-05-31T20:17:24.193 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:17:24 GMT /pklxkd-47/ 2023-05-31T20:17:24.194 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:jXUIoEyxy1oXYeB4CfH6GQOHq2Q= 2023-05-31T20:17:24.194 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:17:24 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:jXUIoEyxy1oXYeB4CfH6GQOHq2Q=', 'Content-Length': '0'} 2023-05-31T20:17:24.197 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '4'), ('X-RGW-Bytes-Used', '24'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000002add313b14221b51-006477ab54-162e-a2'), ('Content-Length', '0'), ('Date', 'Wed, 31 May 2023 20:17:24 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:17:24.197 DEBUG:boto:path=/pklxkd-47/ 2023-05-31T20:17:24.198 DEBUG:boto:auth_path=/pklxkd-47/ 2023-05-31T20:17:24.198 DEBUG:boto:Method: HEAD 2023-05-31T20:17:24.199 DEBUG:boto:Path: /pklxkd-47/ 2023-05-31T20:17:24.199 DEBUG:boto:Data: 2023-05-31T20:17:24.199 DEBUG:boto:Headers: {} 2023-05-31T20:17:24.200 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:17:24.200 DEBUG:boto:Port: 8000 2023-05-31T20:17:24.200 DEBUG:boto:Params: {} 2023-05-31T20:17:24.201 DEBUG:boto:Token: None 2023-05-31T20:17:24.201 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:17:24 GMT /pklxkd-47/ 2023-05-31T20:17:24.202 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:jXUIoEyxy1oXYeB4CfH6GQOHq2Q= 2023-05-31T20:17:24.202 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:17:24 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:jXUIoEyxy1oXYeB4CfH6GQOHq2Q=', 'Content-Length': '0'} 2023-05-31T20:17:24.204 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '4'), ('X-RGW-Bytes-Used', '24'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000005d9443d8eb950e4f-006477ab54-1630-a1'), ('Content-Length', '0'), ('Date', 'Wed, 31 May 2023 20:17:24 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:17:24.204 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2023-05-31T20:17:24.206 DEBUG:boto:path=/pklxkd-47/ 2023-05-31T20:17:24.206 DEBUG:boto:auth_path=/pklxkd-47/ 2023-05-31T20:17:24.206 DEBUG:boto:path=/pklxkd-47/?versions&max-keys=999 2023-05-31T20:17:24.207 DEBUG:boto:auth_path=/pklxkd-47/?versions&max-keys=999 2023-05-31T20:17:24.207 DEBUG:boto:Method: GET 2023-05-31T20:17:24.207 DEBUG:boto:Path: /pklxkd-47/?versions&max-keys=999 2023-05-31T20:17:24.208 DEBUG:boto:Data: 2023-05-31T20:17:24.208 DEBUG:boto:Headers: {} 2023-05-31T20:17:24.209 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:17:24.209 DEBUG:boto:Port: 8001 2023-05-31T20:17:24.209 DEBUG:boto:Params: {} 2023-05-31T20:17:24.210 DEBUG:boto:Token: None 2023-05-31T20:17:24.210 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:17:24 GMT /pklxkd-47/?versions 2023-05-31T20:17:24.211 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:PlJwpEvdgvxckG5pulP2o2V7ahs= 2023-05-31T20:17:24.211 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:17:24 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:PlJwpEvdgvxckG5pulP2o2V7ahs=', 'Content-Length': '0'} 2023-05-31T20:17:24.213 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ffa5dbeec580b873-006477ab54-162e-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:17:24 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:17:24.214 DEBUG:boto:b'pklxkd-47999falseobj1nulltrue2023-05-31T20:15:11.346Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj2nulltrue2023-05-31T20:15:11.368Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj3nulltrue2023-05-31T20:15:11.402Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj4nulltrue2023-05-31T20:15:11.471Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:17:24.215 DEBUG:tasks.rgw_multi.zone_rados:o=obj1 2023-05-31T20:17:24.215 DEBUG:tasks.rgw_multi.zone_rados:o=obj2 2023-05-31T20:17:24.215 DEBUG:tasks.rgw_multi.zone_rados:o=obj3 2023-05-31T20:17:24.216 DEBUG:tasks.rgw_multi.zone_rados:o=obj4 2023-05-31T20:17:24.216 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2023-05-31T20:17:24.216 DEBUG:boto:path=/pklxkd-47/ 2023-05-31T20:17:24.217 DEBUG:boto:auth_path=/pklxkd-47/ 2023-05-31T20:17:24.217 DEBUG:boto:path=/pklxkd-47/?versions&max-keys=999 2023-05-31T20:17:24.217 DEBUG:boto:auth_path=/pklxkd-47/?versions&max-keys=999 2023-05-31T20:17:24.218 DEBUG:boto:Method: GET 2023-05-31T20:17:24.218 DEBUG:boto:Path: /pklxkd-47/?versions&max-keys=999 2023-05-31T20:17:24.219 DEBUG:boto:Data: 2023-05-31T20:17:24.219 DEBUG:boto:Headers: {} 2023-05-31T20:17:24.219 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:17:24.220 DEBUG:boto:Port: 8000 2023-05-31T20:17:24.220 DEBUG:boto:Params: {} 2023-05-31T20:17:24.221 DEBUG:boto:Token: None 2023-05-31T20:17:24.221 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:17:24 GMT /pklxkd-47/?versions 2023-05-31T20:17:24.221 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:PlJwpEvdgvxckG5pulP2o2V7ahs= 2023-05-31T20:17:24.222 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:17:24 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:PlJwpEvdgvxckG5pulP2o2V7ahs=', 'Content-Length': '0'} 2023-05-31T20:17:24.224 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000322d787fc5cf0bc4-006477ab54-1630-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:17:24 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:17:24.224 DEBUG:boto:b'pklxkd-47999falseobj1nulltrue2023-05-31T20:15:11.346Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj2nulltrue2023-05-31T20:15:11.368Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj3nulltrue2023-05-31T20:15:11.402Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj4nulltrue2023-05-31T20:15:11.471Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:17:24.225 DEBUG:tasks.rgw_multi.zone_rados:o=obj1 2023-05-31T20:17:24.226 DEBUG:tasks.rgw_multi.zone_rados:o=obj2 2023-05-31T20:17:24.226 DEBUG:tasks.rgw_multi.zone_rados:o=obj3 2023-05-31T20:17:24.226 DEBUG:tasks.rgw_multi.zone_rados:o=obj4 2023-05-31T20:17:24.227 DEBUG:boto:path=/pklxkd-47/ 2023-05-31T20:17:24.227 DEBUG:boto:auth_path=/pklxkd-47/ 2023-05-31T20:17:24.228 DEBUG:boto:path=/pklxkd-47/?versions&max-keys=999 2023-05-31T20:17:24.228 DEBUG:boto:auth_path=/pklxkd-47/?versions&max-keys=999 2023-05-31T20:17:24.228 DEBUG:boto:Method: GET 2023-05-31T20:17:24.229 DEBUG:boto:Path: /pklxkd-47/?versions&max-keys=999 2023-05-31T20:17:24.229 DEBUG:boto:Data: 2023-05-31T20:17:24.230 DEBUG:boto:Headers: {} 2023-05-31T20:17:24.230 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:17:24.230 DEBUG:boto:Port: 8001 2023-05-31T20:17:24.231 DEBUG:boto:Params: {} 2023-05-31T20:17:24.231 DEBUG:boto:Token: None 2023-05-31T20:17:24.231 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:17:24 GMT /pklxkd-47/?versions 2023-05-31T20:17:24.232 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:PlJwpEvdgvxckG5pulP2o2V7ahs= 2023-05-31T20:17:24.232 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:17:24 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:PlJwpEvdgvxckG5pulP2o2V7ahs=', 'Content-Length': '0'} 2023-05-31T20:17:24.234 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000006fd7c6c13f15f1c0-006477ab54-162e-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:17:24 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:17:24.235 DEBUG:boto:b'pklxkd-47999falseobj1nulltrue2023-05-31T20:15:11.346Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj2nulltrue2023-05-31T20:15:11.368Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj3nulltrue2023-05-31T20:15:11.402Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj4nulltrue2023-05-31T20:15:11.471Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:17:24.236 DEBUG:boto:path=/pklxkd-47/ 2023-05-31T20:17:24.236 DEBUG:boto:auth_path=/pklxkd-47/ 2023-05-31T20:17:24.236 DEBUG:boto:path=/pklxkd-47/?versions&max-keys=999 2023-05-31T20:17:24.237 DEBUG:boto:auth_path=/pklxkd-47/?versions&max-keys=999 2023-05-31T20:17:24.237 DEBUG:boto:Method: GET 2023-05-31T20:17:24.238 DEBUG:boto:Path: /pklxkd-47/?versions&max-keys=999 2023-05-31T20:17:24.238 DEBUG:boto:Data: 2023-05-31T20:17:24.238 DEBUG:boto:Headers: {} 2023-05-31T20:17:24.239 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:17:24.239 DEBUG:boto:Port: 8000 2023-05-31T20:17:24.240 DEBUG:boto:Params: {} 2023-05-31T20:17:24.240 DEBUG:boto:Token: None 2023-05-31T20:17:24.241 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:17:24 GMT /pklxkd-47/?versions 2023-05-31T20:17:24.241 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:PlJwpEvdgvxckG5pulP2o2V7ahs= 2023-05-31T20:17:24.241 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:17:24 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:PlJwpEvdgvxckG5pulP2o2V7ahs=', 'Content-Length': '0'} 2023-05-31T20:17:24.244 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000001ca997f391c82795-006477ab54-1630-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:17:24 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:17:24.244 DEBUG:boto:b'pklxkd-47999falseobj1nulltrue2023-05-31T20:15:11.346Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj2nulltrue2023-05-31T20:15:11.368Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj3nulltrue2023-05-31T20:15:11.402Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj4nulltrue2023-05-31T20:15:11.471Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:17:24.245 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2023-05-31T20:17:24.245 DEBUG:boto:path=/pklxkd-47/obj1 2023-05-31T20:17:24.246 DEBUG:boto:auth_path=/pklxkd-47/obj1 2023-05-31T20:17:24.247 DEBUG:boto:path=/pklxkd-47/obj1?versionId=null 2023-05-31T20:17:24.247 DEBUG:boto:auth_path=/pklxkd-47/obj1?versionId=null 2023-05-31T20:17:24.248 DEBUG:boto:Method: GET 2023-05-31T20:17:24.248 DEBUG:boto:Path: /pklxkd-47/obj1?versionId=null 2023-05-31T20:17:24.249 DEBUG:boto:Data: 2023-05-31T20:17:24.250 DEBUG:boto:Headers: {} 2023-05-31T20:17:24.250 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:17:24.250 DEBUG:boto:Port: 8001 2023-05-31T20:17:24.251 DEBUG:boto:Params: {} 2023-05-31T20:17:24.251 DEBUG:boto:Token: None 2023-05-31T20:17:24.252 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:17:24 GMT /pklxkd-47/obj1?versionId=null 2023-05-31T20:17:24.252 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:07a8xDUdxo3HyGJbZw+SZYa/MFY= 2023-05-31T20:17:24.253 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:17:24 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:07a8xDUdxo3HyGJbZw+SZYa/MFY=', 'Content-Length': '0'} 2023-05-31T20:17:24.254 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:15:11 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000c075c196e41d4ee7-006477ab54-162e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:17:24 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:17:24.296 DEBUG:boto:path=/pklxkd-47/obj1 2023-05-31T20:17:24.297 DEBUG:boto:auth_path=/pklxkd-47/obj1 2023-05-31T20:17:24.297 DEBUG:boto:path=/pklxkd-47/obj1?versionId=null 2023-05-31T20:17:24.298 DEBUG:boto:auth_path=/pklxkd-47/obj1?versionId=null 2023-05-31T20:17:24.298 DEBUG:boto:Method: GET 2023-05-31T20:17:24.298 DEBUG:boto:Path: /pklxkd-47/obj1?versionId=null 2023-05-31T20:17:24.299 DEBUG:boto:Data: 2023-05-31T20:17:24.299 DEBUG:boto:Headers: {} 2023-05-31T20:17:24.300 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:17:24.300 DEBUG:boto:Port: 8000 2023-05-31T20:17:24.301 DEBUG:boto:Params: {} 2023-05-31T20:17:24.301 DEBUG:boto:Token: None 2023-05-31T20:17:24.302 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:17:24 GMT /pklxkd-47/obj1?versionId=null 2023-05-31T20:17:24.302 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:07a8xDUdxo3HyGJbZw+SZYa/MFY= 2023-05-31T20:17:24.303 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:17:24 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:07a8xDUdxo3HyGJbZw+SZYa/MFY=', 'Content-Length': '0'} 2023-05-31T20:17:24.304 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:15:11 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000002a56acd79780eb84-006477ab54-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:17:24 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:17:24.305 DEBUG:boto:path=/pklxkd-47/obj1 2023-05-31T20:17:24.305 DEBUG:boto:auth_path=/pklxkd-47/obj1 2023-05-31T20:17:24.306 DEBUG:boto:path=/pklxkd-47/obj1?versionId=null 2023-05-31T20:17:24.306 DEBUG:boto:auth_path=/pklxkd-47/obj1?versionId=null 2023-05-31T20:17:24.306 DEBUG:boto:Method: HEAD 2023-05-31T20:17:24.307 DEBUG:boto:Path: /pklxkd-47/obj1?versionId=null 2023-05-31T20:17:24.307 DEBUG:boto:Data: 2023-05-31T20:17:24.307 DEBUG:boto:Headers: {} 2023-05-31T20:17:24.308 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:17:24.308 DEBUG:boto:Port: 8001 2023-05-31T20:17:24.308 DEBUG:boto:Params: {} 2023-05-31T20:17:24.309 DEBUG:boto:Token: None 2023-05-31T20:17:24.309 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:17:24 GMT /pklxkd-47/obj1?versionId=null 2023-05-31T20:17:24.310 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:OGvJJX0+s2nYhoScfgRzWuR4+hY= 2023-05-31T20:17:24.310 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:17:24 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:OGvJJX0+s2nYhoScfgRzWuR4+hY=', 'Content-Length': '0'} 2023-05-31T20:17:24.312 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:15:11 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000a04ddd102ea3c83e-006477ab54-162e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:17:24 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:17:24.313 DEBUG:boto:path=/pklxkd-47/obj1 2023-05-31T20:17:24.313 DEBUG:boto:auth_path=/pklxkd-47/obj1 2023-05-31T20:17:24.313 DEBUG:boto:path=/pklxkd-47/obj1?versionId=null 2023-05-31T20:17:24.314 DEBUG:boto:auth_path=/pklxkd-47/obj1?versionId=null 2023-05-31T20:17:24.314 DEBUG:boto:Method: HEAD 2023-05-31T20:17:24.314 DEBUG:boto:Path: /pklxkd-47/obj1?versionId=null 2023-05-31T20:17:24.315 DEBUG:boto:Data: 2023-05-31T20:17:24.315 DEBUG:boto:Headers: {} 2023-05-31T20:17:24.316 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:17:24.316 DEBUG:boto:Port: 8000 2023-05-31T20:17:24.316 DEBUG:boto:Params: {} 2023-05-31T20:17:24.317 DEBUG:boto:Token: None 2023-05-31T20:17:24.317 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:17:24 GMT /pklxkd-47/obj1?versionId=null 2023-05-31T20:17:24.318 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:OGvJJX0+s2nYhoScfgRzWuR4+hY= 2023-05-31T20:17:24.318 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:17:24 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:OGvJJX0+s2nYhoScfgRzWuR4+hY=', 'Content-Length': '0'} 2023-05-31T20:17:24.320 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:15:11 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000bc71f836caf8b362-006477ab54-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:17:24 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:17:24.321 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2023-05-31T20:17:24.321 DEBUG:boto:path=/pklxkd-47/obj1 2023-05-31T20:17:24.322 DEBUG:boto:auth_path=/pklxkd-47/obj1 2023-05-31T20:17:24.322 DEBUG:boto:path=/pklxkd-47/obj1?versionId=null 2023-05-31T20:17:24.322 DEBUG:boto:auth_path=/pklxkd-47/obj1?versionId=null 2023-05-31T20:17:24.323 DEBUG:boto:Method: GET 2023-05-31T20:17:24.323 DEBUG:boto:Path: /pklxkd-47/obj1?versionId=null 2023-05-31T20:17:24.323 DEBUG:boto:Data: 2023-05-31T20:17:24.324 DEBUG:boto:Headers: {} 2023-05-31T20:17:24.324 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:17:24.325 DEBUG:boto:Port: 8001 2023-05-31T20:17:24.325 DEBUG:boto:Params: {} 2023-05-31T20:17:24.325 DEBUG:boto:Token: None 2023-05-31T20:17:24.326 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:17:24 GMT /pklxkd-47/obj1?versionId=null 2023-05-31T20:17:24.326 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:07a8xDUdxo3HyGJbZw+SZYa/MFY= 2023-05-31T20:17:24.327 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:17:24 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:07a8xDUdxo3HyGJbZw+SZYa/MFY=', 'Content-Length': '0'} 2023-05-31T20:17:24.328 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:15:11 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000ab86c74c8cf47ebe-006477ab54-162e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:17:24 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:17:24.329 DEBUG:boto:path=/pklxkd-47/obj1 2023-05-31T20:17:24.329 DEBUG:boto:auth_path=/pklxkd-47/obj1 2023-05-31T20:17:24.330 DEBUG:boto:path=/pklxkd-47/obj1?versionId=null 2023-05-31T20:17:24.330 DEBUG:boto:auth_path=/pklxkd-47/obj1?versionId=null 2023-05-31T20:17:24.331 DEBUG:boto:Method: GET 2023-05-31T20:17:24.331 DEBUG:boto:Path: /pklxkd-47/obj1?versionId=null 2023-05-31T20:17:24.332 DEBUG:boto:Data: 2023-05-31T20:17:24.332 DEBUG:boto:Headers: {} 2023-05-31T20:17:24.333 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:17:24.333 DEBUG:boto:Port: 8000 2023-05-31T20:17:24.333 DEBUG:boto:Params: {} 2023-05-31T20:17:24.334 DEBUG:boto:Token: None 2023-05-31T20:17:24.334 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:17:24 GMT /pklxkd-47/obj1?versionId=null 2023-05-31T20:17:24.335 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:07a8xDUdxo3HyGJbZw+SZYa/MFY= 2023-05-31T20:17:24.335 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:17:24 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:07a8xDUdxo3HyGJbZw+SZYa/MFY=', 'Content-Length': '0'} 2023-05-31T20:17:24.337 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:15:11 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000a49e274003cfce8b-006477ab54-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:17:24 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:17:24.384 DEBUG:boto:path=/pklxkd-47/ 2023-05-31T20:17:24.385 DEBUG:boto:auth_path=/pklxkd-47/ 2023-05-31T20:17:24.386 DEBUG:boto:path=/pklxkd-47/?prefix=obj1 2023-05-31T20:17:24.386 DEBUG:boto:auth_path=/pklxkd-47/?prefix=obj1 2023-05-31T20:17:24.386 DEBUG:boto:Method: GET 2023-05-31T20:17:24.387 DEBUG:boto:Path: /pklxkd-47/?prefix=obj1 2023-05-31T20:17:24.387 DEBUG:boto:Data: 2023-05-31T20:17:24.388 DEBUG:boto:Headers: {} 2023-05-31T20:17:24.388 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:17:24.388 DEBUG:boto:Port: 8001 2023-05-31T20:17:24.389 DEBUG:boto:Params: {} 2023-05-31T20:17:24.389 DEBUG:boto:Token: None 2023-05-31T20:17:24.390 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:17:24 GMT /pklxkd-47/ 2023-05-31T20:17:24.390 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:6CCqjBAujwkL/2E5qojwaI79Q9I= 2023-05-31T20:17:24.391 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:17:24 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:6CCqjBAujwkL/2E5qojwaI79Q9I=', 'Content-Length': '0'} 2023-05-31T20:17:24.393 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000001962bd405d6311d5-006477ab54-162e-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:17:24 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:17:24.393 DEBUG:boto:b'pklxkd-47obj11000falseobj12023-05-31T20:15:11.346Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:17:24.394 DEBUG:boto:path=/pklxkd-47/ 2023-05-31T20:17:24.394 DEBUG:boto:auth_path=/pklxkd-47/ 2023-05-31T20:17:24.395 DEBUG:boto:path=/pklxkd-47/?prefix=obj1 2023-05-31T20:17:24.395 DEBUG:boto:auth_path=/pklxkd-47/?prefix=obj1 2023-05-31T20:17:24.396 DEBUG:boto:Method: GET 2023-05-31T20:17:24.396 DEBUG:boto:Path: /pklxkd-47/?prefix=obj1 2023-05-31T20:17:24.397 DEBUG:boto:Data: 2023-05-31T20:17:24.397 DEBUG:boto:Headers: {} 2023-05-31T20:17:24.398 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:17:24.398 DEBUG:boto:Port: 8000 2023-05-31T20:17:24.399 DEBUG:boto:Params: {} 2023-05-31T20:17:24.399 DEBUG:boto:Token: None 2023-05-31T20:17:24.400 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:17:24 GMT /pklxkd-47/ 2023-05-31T20:17:24.400 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:6CCqjBAujwkL/2E5qojwaI79Q9I= 2023-05-31T20:17:24.401 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:17:24 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:6CCqjBAujwkL/2E5qojwaI79Q9I=', 'Content-Length': '0'} 2023-05-31T20:17:24.403 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ab7463f11ecbf3b8-006477ab54-1630-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:17:24 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:17:24.404 DEBUG:boto:b'pklxkd-47obj11000falseobj12023-05-31T20:15:11.346Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:17:24.404 DEBUG:boto:path=/pklxkd-47/obj1 2023-05-31T20:17:24.405 DEBUG:boto:auth_path=/pklxkd-47/obj1 2023-05-31T20:17:24.405 DEBUG:boto:Method: HEAD 2023-05-31T20:17:24.405 DEBUG:boto:Path: /pklxkd-47/obj1 2023-05-31T20:17:24.406 DEBUG:boto:Data: 2023-05-31T20:17:24.406 DEBUG:boto:Headers: {} 2023-05-31T20:17:24.407 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:17:24.407 DEBUG:boto:Port: 8001 2023-05-31T20:17:24.407 DEBUG:boto:Params: {} 2023-05-31T20:17:24.408 DEBUG:boto:Token: None 2023-05-31T20:17:24.408 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:17:24 GMT /pklxkd-47/obj1 2023-05-31T20:17:24.409 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:Tyg/uaYO4UEhpCno1JbWPBqVxcw= 2023-05-31T20:17:24.409 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:17:24 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:Tyg/uaYO4UEhpCno1JbWPBqVxcw=', 'Content-Length': '0'} 2023-05-31T20:17:24.411 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:15:11 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000009fb180dbfe4242f0-006477ab54-162e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:17:24 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:17:24.412 DEBUG:boto:path=/pklxkd-47/obj1 2023-05-31T20:17:24.412 DEBUG:boto:auth_path=/pklxkd-47/obj1 2023-05-31T20:17:24.413 DEBUG:boto:Method: HEAD 2023-05-31T20:17:24.413 DEBUG:boto:Path: /pklxkd-47/obj1 2023-05-31T20:17:24.414 DEBUG:boto:Data: 2023-05-31T20:17:24.414 DEBUG:boto:Headers: {} 2023-05-31T20:17:24.414 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:17:24.415 DEBUG:boto:Port: 8000 2023-05-31T20:17:24.415 DEBUG:boto:Params: {} 2023-05-31T20:17:24.416 DEBUG:boto:Token: None 2023-05-31T20:17:24.416 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:17:24 GMT /pklxkd-47/obj1 2023-05-31T20:17:24.417 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:Tyg/uaYO4UEhpCno1JbWPBqVxcw= 2023-05-31T20:17:24.418 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:17:24 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:Tyg/uaYO4UEhpCno1JbWPBqVxcw=', 'Content-Length': '0'} 2023-05-31T20:17:24.420 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:15:11 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000009936ef61e662c107-006477ab54-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:17:24 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:17:24.420 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2023-05-31T20:17:24.421 DEBUG:boto:path=/pklxkd-47/obj1 2023-05-31T20:17:24.421 DEBUG:boto:auth_path=/pklxkd-47/obj1 2023-05-31T20:17:24.422 DEBUG:boto:Method: GET 2023-05-31T20:17:24.422 DEBUG:boto:Path: /pklxkd-47/obj1 2023-05-31T20:17:24.423 DEBUG:boto:Data: 2023-05-31T20:17:24.423 DEBUG:boto:Headers: {} 2023-05-31T20:17:24.423 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:17:24.424 DEBUG:boto:Port: 8001 2023-05-31T20:17:24.425 DEBUG:boto:Params: {} 2023-05-31T20:17:24.425 DEBUG:boto:Token: None 2023-05-31T20:17:24.425 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:17:24 GMT /pklxkd-47/obj1 2023-05-31T20:17:24.426 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:ywaFT1sxqzsma6JDs/tKjUoi/hw= 2023-05-31T20:17:24.426 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:17:24 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:ywaFT1sxqzsma6JDs/tKjUoi/hw=', 'Content-Length': '0'} 2023-05-31T20:17:24.428 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:15:11 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000b30e23ebfe7aeb12-006477ab54-162e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:17:24 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:17:24.476 DEBUG:boto:path=/pklxkd-47/obj1 2023-05-31T20:17:24.477 DEBUG:boto:auth_path=/pklxkd-47/obj1 2023-05-31T20:17:24.477 DEBUG:boto:Method: GET 2023-05-31T20:17:24.478 DEBUG:boto:Path: /pklxkd-47/obj1 2023-05-31T20:17:24.478 DEBUG:boto:Data: 2023-05-31T20:17:24.478 DEBUG:boto:Headers: {} 2023-05-31T20:17:24.479 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:17:24.479 DEBUG:boto:Port: 8000 2023-05-31T20:17:24.480 DEBUG:boto:Params: {} 2023-05-31T20:17:24.480 DEBUG:boto:Token: None 2023-05-31T20:17:24.481 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:17:24 GMT /pklxkd-47/obj1 2023-05-31T20:17:24.481 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:ywaFT1sxqzsma6JDs/tKjUoi/hw= 2023-05-31T20:17:24.481 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:17:24 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:ywaFT1sxqzsma6JDs/tKjUoi/hw=', 'Content-Length': '0'} 2023-05-31T20:17:24.483 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:15:11 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000008ee216067ea20f7b-006477ab54-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:17:24 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:17:24.484 DEBUG:boto:path=/pklxkd-47/ 2023-05-31T20:17:24.484 DEBUG:boto:auth_path=/pklxkd-47/ 2023-05-31T20:17:24.485 DEBUG:boto:path=/pklxkd-47/?prefix=obj1 2023-05-31T20:17:24.485 DEBUG:boto:auth_path=/pklxkd-47/?prefix=obj1 2023-05-31T20:17:24.485 DEBUG:boto:Method: GET 2023-05-31T20:17:24.486 DEBUG:boto:Path: /pklxkd-47/?prefix=obj1 2023-05-31T20:17:24.486 DEBUG:boto:Data: 2023-05-31T20:17:24.487 DEBUG:boto:Headers: {} 2023-05-31T20:17:24.487 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:17:24.487 DEBUG:boto:Port: 8001 2023-05-31T20:17:24.488 DEBUG:boto:Params: {} 2023-05-31T20:17:24.488 DEBUG:boto:Token: None 2023-05-31T20:17:24.489 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:17:24 GMT /pklxkd-47/ 2023-05-31T20:17:24.489 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:6CCqjBAujwkL/2E5qojwaI79Q9I= 2023-05-31T20:17:24.489 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:17:24 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:6CCqjBAujwkL/2E5qojwaI79Q9I=', 'Content-Length': '0'} 2023-05-31T20:17:24.492 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000cee1f4ac6d88640f-006477ab54-162e-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:17:24 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:17:24.492 DEBUG:boto:b'pklxkd-47obj11000falseobj12023-05-31T20:15:11.346Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:17:24.493 DEBUG:boto:path=/pklxkd-47/ 2023-05-31T20:17:24.493 DEBUG:boto:auth_path=/pklxkd-47/ 2023-05-31T20:17:24.494 DEBUG:boto:path=/pklxkd-47/?prefix=obj1 2023-05-31T20:17:24.494 DEBUG:boto:auth_path=/pklxkd-47/?prefix=obj1 2023-05-31T20:17:24.494 DEBUG:boto:Method: GET 2023-05-31T20:17:24.495 DEBUG:boto:Path: /pklxkd-47/?prefix=obj1 2023-05-31T20:17:24.495 DEBUG:boto:Data: 2023-05-31T20:17:24.496 DEBUG:boto:Headers: {} 2023-05-31T20:17:24.496 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:17:24.496 DEBUG:boto:Port: 8000 2023-05-31T20:17:24.497 DEBUG:boto:Params: {} 2023-05-31T20:17:24.497 DEBUG:boto:Token: None 2023-05-31T20:17:24.498 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:17:24 GMT /pklxkd-47/ 2023-05-31T20:17:24.498 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:6CCqjBAujwkL/2E5qojwaI79Q9I= 2023-05-31T20:17:24.499 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:17:24 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:6CCqjBAujwkL/2E5qojwaI79Q9I=', 'Content-Length': '0'} 2023-05-31T20:17:24.501 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000003db7b28f5779ce95-006477ab54-1630-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:17:24 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:17:24.501 DEBUG:boto:b'pklxkd-47obj11000falseobj12023-05-31T20:15:11.346Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:17:24.502 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2023-05-31T20:17:24.503 DEBUG:boto:path=/pklxkd-47/obj2 2023-05-31T20:17:24.503 DEBUG:boto:auth_path=/pklxkd-47/obj2 2023-05-31T20:17:24.504 DEBUG:boto:path=/pklxkd-47/obj2?versionId=null 2023-05-31T20:17:24.505 DEBUG:boto:auth_path=/pklxkd-47/obj2?versionId=null 2023-05-31T20:17:24.506 DEBUG:boto:Method: GET 2023-05-31T20:17:24.507 DEBUG:boto:Path: /pklxkd-47/obj2?versionId=null 2023-05-31T20:17:24.508 DEBUG:boto:Data: 2023-05-31T20:17:24.509 DEBUG:boto:Headers: {} 2023-05-31T20:17:24.510 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:17:24.510 DEBUG:boto:Port: 8001 2023-05-31T20:17:24.511 DEBUG:boto:Params: {} 2023-05-31T20:17:24.512 DEBUG:boto:Token: None 2023-05-31T20:17:24.512 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:17:24 GMT /pklxkd-47/obj2?versionId=null 2023-05-31T20:17:24.513 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:45kY3ia1HLitUMjXsBVAKPXxcPc= 2023-05-31T20:17:24.514 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:17:24 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:45kY3ia1HLitUMjXsBVAKPXxcPc=', 'Content-Length': '0'} 2023-05-31T20:17:24.516 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:15:11 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000008ef6400fd5f912cf-006477ab54-162e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:17:24 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:17:24.516 DEBUG:boto:path=/pklxkd-47/obj2 2023-05-31T20:17:24.517 DEBUG:boto:auth_path=/pklxkd-47/obj2 2023-05-31T20:17:24.517 DEBUG:boto:path=/pklxkd-47/obj2?versionId=null 2023-05-31T20:17:24.518 DEBUG:boto:auth_path=/pklxkd-47/obj2?versionId=null 2023-05-31T20:17:24.518 DEBUG:boto:Method: GET 2023-05-31T20:17:24.518 DEBUG:boto:Path: /pklxkd-47/obj2?versionId=null 2023-05-31T20:17:24.519 DEBUG:boto:Data: 2023-05-31T20:17:24.519 DEBUG:boto:Headers: {} 2023-05-31T20:17:24.519 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:17:24.520 DEBUG:boto:Port: 8000 2023-05-31T20:17:24.520 DEBUG:boto:Params: {} 2023-05-31T20:17:24.521 DEBUG:boto:Token: None 2023-05-31T20:17:24.521 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:17:24 GMT /pklxkd-47/obj2?versionId=null 2023-05-31T20:17:24.521 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:45kY3ia1HLitUMjXsBVAKPXxcPc= 2023-05-31T20:17:24.522 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:17:24 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:45kY3ia1HLitUMjXsBVAKPXxcPc=', 'Content-Length': '0'} 2023-05-31T20:17:24.524 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:15:11 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000004f9a5b3cff4feee1-006477ab54-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:17:24 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:17:24.525 DEBUG:boto:path=/pklxkd-47/obj2 2023-05-31T20:17:24.525 DEBUG:boto:auth_path=/pklxkd-47/obj2 2023-05-31T20:17:24.525 DEBUG:boto:path=/pklxkd-47/obj2?versionId=null 2023-05-31T20:17:24.526 DEBUG:boto:auth_path=/pklxkd-47/obj2?versionId=null 2023-05-31T20:17:24.526 DEBUG:boto:Method: HEAD 2023-05-31T20:17:24.527 DEBUG:boto:Path: /pklxkd-47/obj2?versionId=null 2023-05-31T20:17:24.527 DEBUG:boto:Data: 2023-05-31T20:17:24.527 DEBUG:boto:Headers: {} 2023-05-31T20:17:24.528 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:17:24.528 DEBUG:boto:Port: 8001 2023-05-31T20:17:24.529 DEBUG:boto:Params: {} 2023-05-31T20:17:24.529 DEBUG:boto:Token: None 2023-05-31T20:17:24.529 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:17:24 GMT /pklxkd-47/obj2?versionId=null 2023-05-31T20:17:24.530 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:eXrNr0Kx7Iz7gVCkKvrHw5rqGNg= 2023-05-31T20:17:24.530 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:17:24 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:eXrNr0Kx7Iz7gVCkKvrHw5rqGNg=', 'Content-Length': '0'} 2023-05-31T20:17:24.531 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:15:11 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000032803cda38361dc2-006477ab54-162e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:17:24 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:17:24.532 DEBUG:boto:path=/pklxkd-47/obj2 2023-05-31T20:17:24.532 DEBUG:boto:auth_path=/pklxkd-47/obj2 2023-05-31T20:17:24.533 DEBUG:boto:path=/pklxkd-47/obj2?versionId=null 2023-05-31T20:17:24.533 DEBUG:boto:auth_path=/pklxkd-47/obj2?versionId=null 2023-05-31T20:17:24.533 DEBUG:boto:Method: HEAD 2023-05-31T20:17:24.534 DEBUG:boto:Path: /pklxkd-47/obj2?versionId=null 2023-05-31T20:17:24.534 DEBUG:boto:Data: 2023-05-31T20:17:24.535 DEBUG:boto:Headers: {} 2023-05-31T20:17:24.535 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:17:24.535 DEBUG:boto:Port: 8000 2023-05-31T20:17:24.536 DEBUG:boto:Params: {} 2023-05-31T20:17:24.537 DEBUG:boto:Token: None 2023-05-31T20:17:24.537 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:17:24 GMT /pklxkd-47/obj2?versionId=null 2023-05-31T20:17:24.538 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:eXrNr0Kx7Iz7gVCkKvrHw5rqGNg= 2023-05-31T20:17:24.538 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:17:24 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:eXrNr0Kx7Iz7gVCkKvrHw5rqGNg=', 'Content-Length': '0'} 2023-05-31T20:17:24.540 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:15:11 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000fb25dff0fa8ee269-006477ab54-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:17:24 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:17:24.541 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2023-05-31T20:17:24.541 DEBUG:boto:path=/pklxkd-47/obj2 2023-05-31T20:17:24.542 DEBUG:boto:auth_path=/pklxkd-47/obj2 2023-05-31T20:17:24.542 DEBUG:boto:path=/pklxkd-47/obj2?versionId=null 2023-05-31T20:17:24.542 DEBUG:boto:auth_path=/pklxkd-47/obj2?versionId=null 2023-05-31T20:17:24.543 DEBUG:boto:Method: GET 2023-05-31T20:17:24.543 DEBUG:boto:Path: /pklxkd-47/obj2?versionId=null 2023-05-31T20:17:24.544 DEBUG:boto:Data: 2023-05-31T20:17:24.544 DEBUG:boto:Headers: {} 2023-05-31T20:17:24.544 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:17:24.545 DEBUG:boto:Port: 8001 2023-05-31T20:17:24.545 DEBUG:boto:Params: {} 2023-05-31T20:17:24.546 DEBUG:boto:Token: None 2023-05-31T20:17:24.546 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:17:24 GMT /pklxkd-47/obj2?versionId=null 2023-05-31T20:17:24.546 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:45kY3ia1HLitUMjXsBVAKPXxcPc= 2023-05-31T20:17:24.547 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:17:24 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:45kY3ia1HLitUMjXsBVAKPXxcPc=', 'Content-Length': '0'} 2023-05-31T20:17:24.549 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:15:11 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000062cb3c210e85f21e-006477ab54-162e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:17:24 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:17:24.549 DEBUG:boto:path=/pklxkd-47/obj2 2023-05-31T20:17:24.550 DEBUG:boto:auth_path=/pklxkd-47/obj2 2023-05-31T20:17:24.550 DEBUG:boto:path=/pklxkd-47/obj2?versionId=null 2023-05-31T20:17:24.550 DEBUG:boto:auth_path=/pklxkd-47/obj2?versionId=null 2023-05-31T20:17:24.551 DEBUG:boto:Method: GET 2023-05-31T20:17:24.551 DEBUG:boto:Path: /pklxkd-47/obj2?versionId=null 2023-05-31T20:17:24.552 DEBUG:boto:Data: 2023-05-31T20:17:24.552 DEBUG:boto:Headers: {} 2023-05-31T20:17:24.552 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:17:24.553 DEBUG:boto:Port: 8000 2023-05-31T20:17:24.553 DEBUG:boto:Params: {} 2023-05-31T20:17:24.554 DEBUG:boto:Token: None 2023-05-31T20:17:24.554 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:17:24 GMT /pklxkd-47/obj2?versionId=null 2023-05-31T20:17:24.554 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:45kY3ia1HLitUMjXsBVAKPXxcPc= 2023-05-31T20:17:24.555 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:17:24 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:45kY3ia1HLitUMjXsBVAKPXxcPc=', 'Content-Length': '0'} 2023-05-31T20:17:24.556 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:15:11 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000e4a36ab536db33fa-006477ab54-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:17:24 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:17:24.557 DEBUG:boto:path=/pklxkd-47/ 2023-05-31T20:17:24.558 DEBUG:boto:auth_path=/pklxkd-47/ 2023-05-31T20:17:24.558 DEBUG:boto:path=/pklxkd-47/?prefix=obj2 2023-05-31T20:17:24.558 DEBUG:boto:auth_path=/pklxkd-47/?prefix=obj2 2023-05-31T20:17:24.559 DEBUG:boto:Method: GET 2023-05-31T20:17:24.559 DEBUG:boto:Path: /pklxkd-47/?prefix=obj2 2023-05-31T20:17:24.560 DEBUG:boto:Data: 2023-05-31T20:17:24.560 DEBUG:boto:Headers: {} 2023-05-31T20:17:24.561 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:17:24.561 DEBUG:boto:Port: 8001 2023-05-31T20:17:24.561 DEBUG:boto:Params: {} 2023-05-31T20:17:24.562 DEBUG:boto:Token: None 2023-05-31T20:17:24.562 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:17:24 GMT /pklxkd-47/ 2023-05-31T20:17:24.563 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:6CCqjBAujwkL/2E5qojwaI79Q9I= 2023-05-31T20:17:24.563 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:17:24 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:6CCqjBAujwkL/2E5qojwaI79Q9I=', 'Content-Length': '0'} 2023-05-31T20:17:24.565 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000947796c11cb8ba9e-006477ab54-162e-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:17:24 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:17:24.566 DEBUG:boto:b'pklxkd-47obj21000falseobj22023-05-31T20:15:11.368Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:17:24.567 DEBUG:boto:path=/pklxkd-47/ 2023-05-31T20:17:24.567 DEBUG:boto:auth_path=/pklxkd-47/ 2023-05-31T20:17:24.567 DEBUG:boto:path=/pklxkd-47/?prefix=obj2 2023-05-31T20:17:24.568 DEBUG:boto:auth_path=/pklxkd-47/?prefix=obj2 2023-05-31T20:17:24.568 DEBUG:boto:Method: GET 2023-05-31T20:17:24.569 DEBUG:boto:Path: /pklxkd-47/?prefix=obj2 2023-05-31T20:17:24.570 DEBUG:boto:Data: 2023-05-31T20:17:24.570 DEBUG:boto:Headers: {} 2023-05-31T20:17:24.570 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:17:24.571 DEBUG:boto:Port: 8000 2023-05-31T20:17:24.571 DEBUG:boto:Params: {} 2023-05-31T20:17:24.571 DEBUG:boto:Token: None 2023-05-31T20:17:24.572 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:17:24 GMT /pklxkd-47/ 2023-05-31T20:17:24.572 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:6CCqjBAujwkL/2E5qojwaI79Q9I= 2023-05-31T20:17:24.573 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:17:24 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:6CCqjBAujwkL/2E5qojwaI79Q9I=', 'Content-Length': '0'} 2023-05-31T20:17:24.575 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000091246b25975d89c2-006477ab54-1630-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:17:24 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:17:24.616 DEBUG:boto:b'pklxkd-47obj21000falseobj22023-05-31T20:15:11.368Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:17:24.617 DEBUG:boto:path=/pklxkd-47/obj2 2023-05-31T20:17:24.618 DEBUG:boto:auth_path=/pklxkd-47/obj2 2023-05-31T20:17:24.618 DEBUG:boto:Method: HEAD 2023-05-31T20:17:24.619 DEBUG:boto:Path: /pklxkd-47/obj2 2023-05-31T20:17:24.619 DEBUG:boto:Data: 2023-05-31T20:17:24.619 DEBUG:boto:Headers: {} 2023-05-31T20:17:24.620 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:17:24.620 DEBUG:boto:Port: 8001 2023-05-31T20:17:24.620 DEBUG:boto:Params: {} 2023-05-31T20:17:24.621 DEBUG:boto:Token: None 2023-05-31T20:17:24.621 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:17:24 GMT /pklxkd-47/obj2 2023-05-31T20:17:24.621 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:HeQjNRjY7UNE3FjXi4DthE9qhho= 2023-05-31T20:17:24.622 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:17:24 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:HeQjNRjY7UNE3FjXi4DthE9qhho=', 'Content-Length': '0'} 2023-05-31T20:17:24.623 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:15:11 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000004732557256475893-006477ab54-162e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:17:24 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:17:24.624 DEBUG:boto:path=/pklxkd-47/obj2 2023-05-31T20:17:24.624 DEBUG:boto:auth_path=/pklxkd-47/obj2 2023-05-31T20:17:24.625 DEBUG:boto:Method: HEAD 2023-05-31T20:17:24.625 DEBUG:boto:Path: /pklxkd-47/obj2 2023-05-31T20:17:24.625 DEBUG:boto:Data: 2023-05-31T20:17:24.626 DEBUG:boto:Headers: {} 2023-05-31T20:17:24.626 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:17:24.626 DEBUG:boto:Port: 8000 2023-05-31T20:17:24.627 DEBUG:boto:Params: {} 2023-05-31T20:17:24.627 DEBUG:boto:Token: None 2023-05-31T20:17:24.627 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:17:24 GMT /pklxkd-47/obj2 2023-05-31T20:17:24.628 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:HeQjNRjY7UNE3FjXi4DthE9qhho= 2023-05-31T20:17:24.629 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:17:24 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:HeQjNRjY7UNE3FjXi4DthE9qhho=', 'Content-Length': '0'} 2023-05-31T20:17:24.630 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:15:11 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000dfd5a15605535c97-006477ab54-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:17:24 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:17:24.631 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2023-05-31T20:17:24.631 DEBUG:boto:path=/pklxkd-47/obj2 2023-05-31T20:17:24.632 DEBUG:boto:auth_path=/pklxkd-47/obj2 2023-05-31T20:17:24.632 DEBUG:boto:Method: GET 2023-05-31T20:17:24.633 DEBUG:boto:Path: /pklxkd-47/obj2 2023-05-31T20:17:24.633 DEBUG:boto:Data: 2023-05-31T20:17:24.633 DEBUG:boto:Headers: {} 2023-05-31T20:17:24.634 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:17:24.634 DEBUG:boto:Port: 8001 2023-05-31T20:17:24.634 DEBUG:boto:Params: {} 2023-05-31T20:17:24.635 DEBUG:boto:Token: None 2023-05-31T20:17:24.635 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:17:24 GMT /pklxkd-47/obj2 2023-05-31T20:17:24.636 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:7eyy0vlXyDN3grFZpBx1Pu7QkzQ= 2023-05-31T20:17:24.636 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:17:24 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:7eyy0vlXyDN3grFZpBx1Pu7QkzQ=', 'Content-Length': '0'} 2023-05-31T20:17:24.637 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:15:11 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000044940fa416cebe58-006477ab54-162e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:17:24 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:17:24.638 DEBUG:boto:path=/pklxkd-47/obj2 2023-05-31T20:17:24.639 DEBUG:boto:auth_path=/pklxkd-47/obj2 2023-05-31T20:17:24.639 DEBUG:boto:Method: GET 2023-05-31T20:17:24.640 DEBUG:boto:Path: /pklxkd-47/obj2 2023-05-31T20:17:24.640 DEBUG:boto:Data: 2023-05-31T20:17:24.640 DEBUG:boto:Headers: {} 2023-05-31T20:17:24.641 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:17:24.641 DEBUG:boto:Port: 8000 2023-05-31T20:17:24.641 DEBUG:boto:Params: {} 2023-05-31T20:17:24.642 DEBUG:boto:Token: None 2023-05-31T20:17:24.642 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:17:24 GMT /pklxkd-47/obj2 2023-05-31T20:17:24.643 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:7eyy0vlXyDN3grFZpBx1Pu7QkzQ= 2023-05-31T20:17:24.643 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:17:24 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:7eyy0vlXyDN3grFZpBx1Pu7QkzQ=', 'Content-Length': '0'} 2023-05-31T20:17:24.645 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:15:11 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000031068e0e86973146-006477ab54-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:17:24 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:17:24.645 DEBUG:boto:path=/pklxkd-47/ 2023-05-31T20:17:24.646 DEBUG:boto:auth_path=/pklxkd-47/ 2023-05-31T20:17:24.646 DEBUG:boto:path=/pklxkd-47/?prefix=obj2 2023-05-31T20:17:24.646 DEBUG:boto:auth_path=/pklxkd-47/?prefix=obj2 2023-05-31T20:17:24.647 DEBUG:boto:Method: GET 2023-05-31T20:17:24.647 DEBUG:boto:Path: /pklxkd-47/?prefix=obj2 2023-05-31T20:17:24.647 DEBUG:boto:Data: 2023-05-31T20:17:24.648 DEBUG:boto:Headers: {} 2023-05-31T20:17:24.648 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:17:24.649 DEBUG:boto:Port: 8001 2023-05-31T20:17:24.649 DEBUG:boto:Params: {} 2023-05-31T20:17:24.649 DEBUG:boto:Token: None 2023-05-31T20:17:24.650 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:17:24 GMT /pklxkd-47/ 2023-05-31T20:17:24.650 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:6CCqjBAujwkL/2E5qojwaI79Q9I= 2023-05-31T20:17:24.651 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:17:24 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:6CCqjBAujwkL/2E5qojwaI79Q9I=', 'Content-Length': '0'} 2023-05-31T20:17:24.653 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e08c803fce414951-006477ab54-162e-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:17:24 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:17:24.653 DEBUG:boto:b'pklxkd-47obj21000falseobj22023-05-31T20:15:11.368Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:17:24.654 DEBUG:boto:path=/pklxkd-47/ 2023-05-31T20:17:24.654 DEBUG:boto:auth_path=/pklxkd-47/ 2023-05-31T20:17:24.655 DEBUG:boto:path=/pklxkd-47/?prefix=obj2 2023-05-31T20:17:24.655 DEBUG:boto:auth_path=/pklxkd-47/?prefix=obj2 2023-05-31T20:17:24.655 DEBUG:boto:Method: GET 2023-05-31T20:17:24.656 DEBUG:boto:Path: /pklxkd-47/?prefix=obj2 2023-05-31T20:17:24.656 DEBUG:boto:Data: 2023-05-31T20:17:24.656 DEBUG:boto:Headers: {} 2023-05-31T20:17:24.657 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:17:24.657 DEBUG:boto:Port: 8000 2023-05-31T20:17:24.657 DEBUG:boto:Params: {} 2023-05-31T20:17:24.658 DEBUG:boto:Token: None 2023-05-31T20:17:24.658 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:17:24 GMT /pklxkd-47/ 2023-05-31T20:17:24.659 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:6CCqjBAujwkL/2E5qojwaI79Q9I= 2023-05-31T20:17:24.659 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:17:24 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:6CCqjBAujwkL/2E5qojwaI79Q9I=', 'Content-Length': '0'} 2023-05-31T20:17:24.661 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000cad73964c4986519-006477ab54-1630-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:17:24 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:17:24.662 DEBUG:boto:b'pklxkd-47obj21000falseobj22023-05-31T20:15:11.368Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:17:24.662 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2023-05-31T20:17:24.663 DEBUG:boto:path=/pklxkd-47/obj3 2023-05-31T20:17:24.663 DEBUG:boto:auth_path=/pklxkd-47/obj3 2023-05-31T20:17:24.663 DEBUG:boto:path=/pklxkd-47/obj3?versionId=null 2023-05-31T20:17:24.664 DEBUG:boto:auth_path=/pklxkd-47/obj3?versionId=null 2023-05-31T20:17:24.664 DEBUG:boto:Method: GET 2023-05-31T20:17:24.664 DEBUG:boto:Path: /pklxkd-47/obj3?versionId=null 2023-05-31T20:17:24.665 DEBUG:boto:Data: 2023-05-31T20:17:24.665 DEBUG:boto:Headers: {} 2023-05-31T20:17:24.666 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:17:24.666 DEBUG:boto:Port: 8001 2023-05-31T20:17:24.666 DEBUG:boto:Params: {} 2023-05-31T20:17:24.667 DEBUG:boto:Token: None 2023-05-31T20:17:24.667 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:17:24 GMT /pklxkd-47/obj3?versionId=null 2023-05-31T20:17:24.667 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:ehNMZl9k9DzoBcU/iHm4wOQyNNU= 2023-05-31T20:17:24.668 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:17:24 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:ehNMZl9k9DzoBcU/iHm4wOQyNNU=', 'Content-Length': '0'} 2023-05-31T20:17:24.670 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:15:11 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000004c0a8c4ae0d84346-006477ab54-162e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:17:24 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:17:24.712 DEBUG:boto:path=/pklxkd-47/obj3 2023-05-31T20:17:24.713 DEBUG:boto:auth_path=/pklxkd-47/obj3 2023-05-31T20:17:24.713 DEBUG:boto:path=/pklxkd-47/obj3?versionId=null 2023-05-31T20:17:24.714 DEBUG:boto:auth_path=/pklxkd-47/obj3?versionId=null 2023-05-31T20:17:24.714 DEBUG:boto:Method: GET 2023-05-31T20:17:24.715 DEBUG:boto:Path: /pklxkd-47/obj3?versionId=null 2023-05-31T20:17:24.715 DEBUG:boto:Data: 2023-05-31T20:17:24.715 DEBUG:boto:Headers: {} 2023-05-31T20:17:24.716 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:17:24.716 DEBUG:boto:Port: 8000 2023-05-31T20:17:24.716 DEBUG:boto:Params: {} 2023-05-31T20:17:24.717 DEBUG:boto:Token: None 2023-05-31T20:17:24.717 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:17:24 GMT /pklxkd-47/obj3?versionId=null 2023-05-31T20:17:24.718 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:ehNMZl9k9DzoBcU/iHm4wOQyNNU= 2023-05-31T20:17:24.718 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:17:24 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:ehNMZl9k9DzoBcU/iHm4wOQyNNU=', 'Content-Length': '0'} 2023-05-31T20:17:24.720 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:15:11 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000ac4ab7be09a214d6-006477ab54-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:17:24 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:17:24.720 DEBUG:boto:path=/pklxkd-47/obj3 2023-05-31T20:17:24.721 DEBUG:boto:auth_path=/pklxkd-47/obj3 2023-05-31T20:17:24.721 DEBUG:boto:path=/pklxkd-47/obj3?versionId=null 2023-05-31T20:17:24.722 DEBUG:boto:auth_path=/pklxkd-47/obj3?versionId=null 2023-05-31T20:17:24.722 DEBUG:boto:Method: HEAD 2023-05-31T20:17:24.722 DEBUG:boto:Path: /pklxkd-47/obj3?versionId=null 2023-05-31T20:17:24.723 DEBUG:boto:Data: 2023-05-31T20:17:24.723 DEBUG:boto:Headers: {} 2023-05-31T20:17:24.724 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:17:24.724 DEBUG:boto:Port: 8001 2023-05-31T20:17:24.724 DEBUG:boto:Params: {} 2023-05-31T20:17:24.725 DEBUG:boto:Token: None 2023-05-31T20:17:24.725 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:17:24 GMT /pklxkd-47/obj3?versionId=null 2023-05-31T20:17:24.725 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:XtWnVUyGn0BhIwpUHFvd2BCmk0E= 2023-05-31T20:17:24.726 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:17:24 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:XtWnVUyGn0BhIwpUHFvd2BCmk0E=', 'Content-Length': '0'} 2023-05-31T20:17:24.727 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:15:11 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000c529423c1ad53411-006477ab54-162e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:17:24 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:17:24.728 DEBUG:boto:path=/pklxkd-47/obj3 2023-05-31T20:17:24.728 DEBUG:boto:auth_path=/pklxkd-47/obj3 2023-05-31T20:17:24.729 DEBUG:boto:path=/pklxkd-47/obj3?versionId=null 2023-05-31T20:17:24.729 DEBUG:boto:auth_path=/pklxkd-47/obj3?versionId=null 2023-05-31T20:17:24.729 DEBUG:boto:Method: HEAD 2023-05-31T20:17:24.730 DEBUG:boto:Path: /pklxkd-47/obj3?versionId=null 2023-05-31T20:17:24.730 DEBUG:boto:Data: 2023-05-31T20:17:24.730 DEBUG:boto:Headers: {} 2023-05-31T20:17:24.731 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:17:24.731 DEBUG:boto:Port: 8000 2023-05-31T20:17:24.731 DEBUG:boto:Params: {} 2023-05-31T20:17:24.732 DEBUG:boto:Token: None 2023-05-31T20:17:24.732 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:17:24 GMT /pklxkd-47/obj3?versionId=null 2023-05-31T20:17:24.732 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:XtWnVUyGn0BhIwpUHFvd2BCmk0E= 2023-05-31T20:17:24.733 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:17:24 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:XtWnVUyGn0BhIwpUHFvd2BCmk0E=', 'Content-Length': '0'} 2023-05-31T20:17:24.735 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:15:11 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000a403c823d884ba2f-006477ab54-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:17:24 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:17:24.736 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2023-05-31T20:17:24.736 DEBUG:boto:path=/pklxkd-47/obj3 2023-05-31T20:17:24.736 DEBUG:boto:auth_path=/pklxkd-47/obj3 2023-05-31T20:17:24.737 DEBUG:boto:path=/pklxkd-47/obj3?versionId=null 2023-05-31T20:17:24.737 DEBUG:boto:auth_path=/pklxkd-47/obj3?versionId=null 2023-05-31T20:17:24.738 DEBUG:boto:Method: GET 2023-05-31T20:17:24.738 DEBUG:boto:Path: /pklxkd-47/obj3?versionId=null 2023-05-31T20:17:24.738 DEBUG:boto:Data: 2023-05-31T20:17:24.739 DEBUG:boto:Headers: {} 2023-05-31T20:17:24.739 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:17:24.739 DEBUG:boto:Port: 8001 2023-05-31T20:17:24.740 DEBUG:boto:Params: {} 2023-05-31T20:17:24.740 DEBUG:boto:Token: None 2023-05-31T20:17:24.741 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:17:24 GMT /pklxkd-47/obj3?versionId=null 2023-05-31T20:17:24.741 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:ehNMZl9k9DzoBcU/iHm4wOQyNNU= 2023-05-31T20:17:24.741 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:17:24 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:ehNMZl9k9DzoBcU/iHm4wOQyNNU=', 'Content-Length': '0'} 2023-05-31T20:17:24.743 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:15:11 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000c48c9f67c63b6483-006477ab54-162e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:17:24 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:17:24.744 DEBUG:boto:path=/pklxkd-47/obj3 2023-05-31T20:17:24.745 DEBUG:boto:auth_path=/pklxkd-47/obj3 2023-05-31T20:17:24.745 DEBUG:boto:path=/pklxkd-47/obj3?versionId=null 2023-05-31T20:17:24.745 DEBUG:boto:auth_path=/pklxkd-47/obj3?versionId=null 2023-05-31T20:17:24.746 DEBUG:boto:Method: GET 2023-05-31T20:17:24.746 DEBUG:boto:Path: /pklxkd-47/obj3?versionId=null 2023-05-31T20:17:24.746 DEBUG:boto:Data: 2023-05-31T20:17:24.747 DEBUG:boto:Headers: {} 2023-05-31T20:17:24.747 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:17:24.747 DEBUG:boto:Port: 8000 2023-05-31T20:17:24.748 DEBUG:boto:Params: {} 2023-05-31T20:17:24.748 DEBUG:boto:Token: None 2023-05-31T20:17:24.749 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:17:24 GMT /pklxkd-47/obj3?versionId=null 2023-05-31T20:17:24.749 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:ehNMZl9k9DzoBcU/iHm4wOQyNNU= 2023-05-31T20:17:24.750 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:17:24 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:ehNMZl9k9DzoBcU/iHm4wOQyNNU=', 'Content-Length': '0'} 2023-05-31T20:17:24.751 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:15:11 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000e85be540b0e4069e-006477ab54-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:17:24 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:17:24.752 DEBUG:boto:path=/pklxkd-47/ 2023-05-31T20:17:24.752 DEBUG:boto:auth_path=/pklxkd-47/ 2023-05-31T20:17:24.753 DEBUG:boto:path=/pklxkd-47/?prefix=obj3 2023-05-31T20:17:24.753 DEBUG:boto:auth_path=/pklxkd-47/?prefix=obj3 2023-05-31T20:17:24.753 DEBUG:boto:Method: GET 2023-05-31T20:17:24.754 DEBUG:boto:Path: /pklxkd-47/?prefix=obj3 2023-05-31T20:17:24.754 DEBUG:boto:Data: 2023-05-31T20:17:24.754 DEBUG:boto:Headers: {} 2023-05-31T20:17:24.755 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:17:24.755 DEBUG:boto:Port: 8001 2023-05-31T20:17:24.756 DEBUG:boto:Params: {} 2023-05-31T20:17:24.756 DEBUG:boto:Token: None 2023-05-31T20:17:24.756 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:17:24 GMT /pklxkd-47/ 2023-05-31T20:17:24.757 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:6CCqjBAujwkL/2E5qojwaI79Q9I= 2023-05-31T20:17:24.757 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:17:24 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:6CCqjBAujwkL/2E5qojwaI79Q9I=', 'Content-Length': '0'} 2023-05-31T20:17:24.759 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000002f422d46064c9c01-006477ab54-162e-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:17:24 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:17:24.760 DEBUG:boto:b'pklxkd-47obj31000falseobj32023-05-31T20:15:11.402Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:17:24.761 DEBUG:boto:path=/pklxkd-47/ 2023-05-31T20:17:24.761 DEBUG:boto:auth_path=/pklxkd-47/ 2023-05-31T20:17:24.761 DEBUG:boto:path=/pklxkd-47/?prefix=obj3 2023-05-31T20:17:24.762 DEBUG:boto:auth_path=/pklxkd-47/?prefix=obj3 2023-05-31T20:17:24.762 DEBUG:boto:Method: GET 2023-05-31T20:17:24.763 DEBUG:boto:Path: /pklxkd-47/?prefix=obj3 2023-05-31T20:17:24.763 DEBUG:boto:Data: 2023-05-31T20:17:24.763 DEBUG:boto:Headers: {} 2023-05-31T20:17:24.764 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:17:24.764 DEBUG:boto:Port: 8000 2023-05-31T20:17:24.764 DEBUG:boto:Params: {} 2023-05-31T20:17:24.765 DEBUG:boto:Token: None 2023-05-31T20:17:24.765 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:17:24 GMT /pklxkd-47/ 2023-05-31T20:17:24.766 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:6CCqjBAujwkL/2E5qojwaI79Q9I= 2023-05-31T20:17:24.766 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:17:24 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:6CCqjBAujwkL/2E5qojwaI79Q9I=', 'Content-Length': '0'} 2023-05-31T20:17:24.768 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000007ba2fec4257df95d-006477ab54-1630-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:17:24 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:17:24.768 DEBUG:boto:b'pklxkd-47obj31000falseobj32023-05-31T20:15:11.402Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:17:24.769 DEBUG:boto:path=/pklxkd-47/obj3 2023-05-31T20:17:24.770 DEBUG:boto:auth_path=/pklxkd-47/obj3 2023-05-31T20:17:24.770 DEBUG:boto:Method: HEAD 2023-05-31T20:17:24.770 DEBUG:boto:Path: /pklxkd-47/obj3 2023-05-31T20:17:24.771 DEBUG:boto:Data: 2023-05-31T20:17:24.771 DEBUG:boto:Headers: {} 2023-05-31T20:17:24.771 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:17:24.772 DEBUG:boto:Port: 8001 2023-05-31T20:17:24.772 DEBUG:boto:Params: {} 2023-05-31T20:17:24.772 DEBUG:boto:Token: None 2023-05-31T20:17:24.773 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:17:24 GMT /pklxkd-47/obj3 2023-05-31T20:17:24.773 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:o7Ch9P43+eRfg9f8OIGxgFkyEqw= 2023-05-31T20:17:24.773 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:17:24 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:o7Ch9P43+eRfg9f8OIGxgFkyEqw=', 'Content-Length': '0'} 2023-05-31T20:17:24.775 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:15:11 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000652858b1e30e27cf-006477ab54-162e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:17:24 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:17:24.776 DEBUG:boto:path=/pklxkd-47/obj3 2023-05-31T20:17:24.776 DEBUG:boto:auth_path=/pklxkd-47/obj3 2023-05-31T20:17:24.777 DEBUG:boto:Method: HEAD 2023-05-31T20:17:24.777 DEBUG:boto:Path: /pklxkd-47/obj3 2023-05-31T20:17:24.777 DEBUG:boto:Data: 2023-05-31T20:17:24.778 DEBUG:boto:Headers: {} 2023-05-31T20:17:24.778 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:17:24.778 DEBUG:boto:Port: 8000 2023-05-31T20:17:24.779 DEBUG:boto:Params: {} 2023-05-31T20:17:24.779 DEBUG:boto:Token: None 2023-05-31T20:17:24.780 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:17:24 GMT /pklxkd-47/obj3 2023-05-31T20:17:24.780 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:o7Ch9P43+eRfg9f8OIGxgFkyEqw= 2023-05-31T20:17:24.781 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:17:24 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:o7Ch9P43+eRfg9f8OIGxgFkyEqw=', 'Content-Length': '0'} 2023-05-31T20:17:24.782 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:15:11 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000a8de0c7770937b72-006477ab54-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:17:24 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:17:24.783 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2023-05-31T20:17:24.783 DEBUG:boto:path=/pklxkd-47/obj3 2023-05-31T20:17:24.784 DEBUG:boto:auth_path=/pklxkd-47/obj3 2023-05-31T20:17:24.784 DEBUG:boto:Method: GET 2023-05-31T20:17:24.784 DEBUG:boto:Path: /pklxkd-47/obj3 2023-05-31T20:17:24.785 DEBUG:boto:Data: 2023-05-31T20:17:24.785 DEBUG:boto:Headers: {} 2023-05-31T20:17:24.785 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:17:24.786 DEBUG:boto:Port: 8001 2023-05-31T20:17:24.786 DEBUG:boto:Params: {} 2023-05-31T20:17:24.787 DEBUG:boto:Token: None 2023-05-31T20:17:24.787 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:17:24 GMT /pklxkd-47/obj3 2023-05-31T20:17:24.787 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:Srfzz/pbK6Kgy9Hp4FQvMOJ6WgQ= 2023-05-31T20:17:24.788 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:17:24 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:Srfzz/pbK6Kgy9Hp4FQvMOJ6WgQ=', 'Content-Length': '0'} 2023-05-31T20:17:24.790 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:15:11 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000000d86e6ecf12a5708-006477ab54-162e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:17:24 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:17:24.790 DEBUG:boto:path=/pklxkd-47/obj3 2023-05-31T20:17:24.791 DEBUG:boto:auth_path=/pklxkd-47/obj3 2023-05-31T20:17:24.791 DEBUG:boto:Method: GET 2023-05-31T20:17:24.791 DEBUG:boto:Path: /pklxkd-47/obj3 2023-05-31T20:17:24.792 DEBUG:boto:Data: 2023-05-31T20:17:24.792 DEBUG:boto:Headers: {} 2023-05-31T20:17:24.792 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:17:24.793 DEBUG:boto:Port: 8000 2023-05-31T20:17:24.793 DEBUG:boto:Params: {} 2023-05-31T20:17:24.794 DEBUG:boto:Token: None 2023-05-31T20:17:24.794 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:17:24 GMT /pklxkd-47/obj3 2023-05-31T20:17:24.794 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:Srfzz/pbK6Kgy9Hp4FQvMOJ6WgQ= 2023-05-31T20:17:24.795 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:17:24 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:Srfzz/pbK6Kgy9Hp4FQvMOJ6WgQ=', 'Content-Length': '0'} 2023-05-31T20:17:24.796 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:15:11 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000a933cce5ead5a73b-006477ab54-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:17:24 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:17:24.840 DEBUG:boto:path=/pklxkd-47/ 2023-05-31T20:17:24.841 DEBUG:boto:auth_path=/pklxkd-47/ 2023-05-31T20:17:24.841 DEBUG:boto:path=/pklxkd-47/?prefix=obj3 2023-05-31T20:17:24.842 DEBUG:boto:auth_path=/pklxkd-47/?prefix=obj3 2023-05-31T20:17:24.842 DEBUG:boto:Method: GET 2023-05-31T20:17:24.842 DEBUG:boto:Path: /pklxkd-47/?prefix=obj3 2023-05-31T20:17:24.843 DEBUG:boto:Data: 2023-05-31T20:17:24.843 DEBUG:boto:Headers: {} 2023-05-31T20:17:24.844 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:17:24.844 DEBUG:boto:Port: 8001 2023-05-31T20:17:24.845 DEBUG:boto:Params: {} 2023-05-31T20:17:24.845 DEBUG:boto:Token: None 2023-05-31T20:17:24.846 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:17:24 GMT /pklxkd-47/ 2023-05-31T20:17:24.846 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:6CCqjBAujwkL/2E5qojwaI79Q9I= 2023-05-31T20:17:24.847 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:17:24 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:6CCqjBAujwkL/2E5qojwaI79Q9I=', 'Content-Length': '0'} 2023-05-31T20:17:24.849 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a103b73cf1d99f55-006477ab54-162e-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:17:24 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:17:24.850 DEBUG:boto:b'pklxkd-47obj31000falseobj32023-05-31T20:15:11.402Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:17:24.850 DEBUG:boto:path=/pklxkd-47/ 2023-05-31T20:17:24.851 DEBUG:boto:auth_path=/pklxkd-47/ 2023-05-31T20:17:24.851 DEBUG:boto:path=/pklxkd-47/?prefix=obj3 2023-05-31T20:17:24.851 DEBUG:boto:auth_path=/pklxkd-47/?prefix=obj3 2023-05-31T20:17:24.852 DEBUG:boto:Method: GET 2023-05-31T20:17:24.852 DEBUG:boto:Path: /pklxkd-47/?prefix=obj3 2023-05-31T20:17:24.852 DEBUG:boto:Data: 2023-05-31T20:17:24.853 DEBUG:boto:Headers: {} 2023-05-31T20:17:24.853 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:17:24.854 DEBUG:boto:Port: 8000 2023-05-31T20:17:24.854 DEBUG:boto:Params: {} 2023-05-31T20:17:24.854 DEBUG:boto:Token: None 2023-05-31T20:17:24.855 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:17:24 GMT /pklxkd-47/ 2023-05-31T20:17:24.856 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:6CCqjBAujwkL/2E5qojwaI79Q9I= 2023-05-31T20:17:24.856 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:17:24 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:6CCqjBAujwkL/2E5qojwaI79Q9I=', 'Content-Length': '0'} 2023-05-31T20:17:24.858 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000038dfd13578c19799-006477ab54-1630-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:17:24 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:17:24.859 DEBUG:boto:b'pklxkd-47obj31000falseobj32023-05-31T20:15:11.402Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:17:24.859 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2023-05-31T20:17:24.860 DEBUG:boto:path=/pklxkd-47/obj4 2023-05-31T20:17:24.860 DEBUG:boto:auth_path=/pklxkd-47/obj4 2023-05-31T20:17:24.860 DEBUG:boto:path=/pklxkd-47/obj4?versionId=null 2023-05-31T20:17:24.861 DEBUG:boto:auth_path=/pklxkd-47/obj4?versionId=null 2023-05-31T20:17:24.861 DEBUG:boto:Method: GET 2023-05-31T20:17:24.862 DEBUG:boto:Path: /pklxkd-47/obj4?versionId=null 2023-05-31T20:17:24.862 DEBUG:boto:Data: 2023-05-31T20:17:24.862 DEBUG:boto:Headers: {} 2023-05-31T20:17:24.863 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:17:24.863 DEBUG:boto:Port: 8001 2023-05-31T20:17:24.863 DEBUG:boto:Params: {} 2023-05-31T20:17:24.864 DEBUG:boto:Token: None 2023-05-31T20:17:24.864 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:17:24 GMT /pklxkd-47/obj4?versionId=null 2023-05-31T20:17:24.865 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:8VaUjv0QTquOlkRS6R4PdZmBQGg= 2023-05-31T20:17:24.865 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:17:24 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:8VaUjv0QTquOlkRS6R4PdZmBQGg=', 'Content-Length': '0'} 2023-05-31T20:17:24.867 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:15:11 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000008cde21f36397c6ec-006477ab54-162e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:17:24 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:17:24.868 DEBUG:boto:path=/pklxkd-47/obj4 2023-05-31T20:17:24.868 DEBUG:boto:auth_path=/pklxkd-47/obj4 2023-05-31T20:17:24.869 DEBUG:boto:path=/pklxkd-47/obj4?versionId=null 2023-05-31T20:17:24.869 DEBUG:boto:auth_path=/pklxkd-47/obj4?versionId=null 2023-05-31T20:17:24.869 DEBUG:boto:Method: GET 2023-05-31T20:17:24.870 DEBUG:boto:Path: /pklxkd-47/obj4?versionId=null 2023-05-31T20:17:24.870 DEBUG:boto:Data: 2023-05-31T20:17:24.870 DEBUG:boto:Headers: {} 2023-05-31T20:17:24.871 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:17:24.871 DEBUG:boto:Port: 8000 2023-05-31T20:17:24.871 DEBUG:boto:Params: {} 2023-05-31T20:17:24.872 DEBUG:boto:Token: None 2023-05-31T20:17:24.872 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:17:24 GMT /pklxkd-47/obj4?versionId=null 2023-05-31T20:17:24.873 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:8VaUjv0QTquOlkRS6R4PdZmBQGg= 2023-05-31T20:17:24.873 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:17:24 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:8VaUjv0QTquOlkRS6R4PdZmBQGg=', 'Content-Length': '0'} 2023-05-31T20:17:24.875 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:15:11 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000303fa54a03fb5a2d-006477ab54-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:17:24 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:17:24.876 DEBUG:boto:path=/pklxkd-47/obj4 2023-05-31T20:17:24.876 DEBUG:boto:auth_path=/pklxkd-47/obj4 2023-05-31T20:17:24.876 DEBUG:boto:path=/pklxkd-47/obj4?versionId=null 2023-05-31T20:17:24.877 DEBUG:boto:auth_path=/pklxkd-47/obj4?versionId=null 2023-05-31T20:17:24.877 DEBUG:boto:Method: HEAD 2023-05-31T20:17:24.878 DEBUG:boto:Path: /pklxkd-47/obj4?versionId=null 2023-05-31T20:17:24.878 DEBUG:boto:Data: 2023-05-31T20:17:24.878 DEBUG:boto:Headers: {} 2023-05-31T20:17:24.879 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:17:24.879 DEBUG:boto:Port: 8001 2023-05-31T20:17:24.879 DEBUG:boto:Params: {} 2023-05-31T20:17:24.880 DEBUG:boto:Token: None 2023-05-31T20:17:24.880 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:17:24 GMT /pklxkd-47/obj4?versionId=null 2023-05-31T20:17:24.881 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:0f813ohopeO2CtkiKVaVV0Q+0yk= 2023-05-31T20:17:24.881 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:17:24 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:0f813ohopeO2CtkiKVaVV0Q+0yk=', 'Content-Length': '0'} 2023-05-31T20:17:24.883 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:15:11 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000437361168a09dfc9-006477ab54-162e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:17:24 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:17:24.883 DEBUG:boto:path=/pklxkd-47/obj4 2023-05-31T20:17:24.884 DEBUG:boto:auth_path=/pklxkd-47/obj4 2023-05-31T20:17:24.884 DEBUG:boto:path=/pklxkd-47/obj4?versionId=null 2023-05-31T20:17:24.884 DEBUG:boto:auth_path=/pklxkd-47/obj4?versionId=null 2023-05-31T20:17:24.885 DEBUG:boto:Method: HEAD 2023-05-31T20:17:24.885 DEBUG:boto:Path: /pklxkd-47/obj4?versionId=null 2023-05-31T20:17:24.885 DEBUG:boto:Data: 2023-05-31T20:17:24.886 DEBUG:boto:Headers: {} 2023-05-31T20:17:24.886 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:17:24.886 DEBUG:boto:Port: 8000 2023-05-31T20:17:24.887 DEBUG:boto:Params: {} 2023-05-31T20:17:24.887 DEBUG:boto:Token: None 2023-05-31T20:17:24.888 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:17:24 GMT /pklxkd-47/obj4?versionId=null 2023-05-31T20:17:24.888 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:0f813ohopeO2CtkiKVaVV0Q+0yk= 2023-05-31T20:17:24.888 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:17:24 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:0f813ohopeO2CtkiKVaVV0Q+0yk=', 'Content-Length': '0'} 2023-05-31T20:17:24.890 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:15:11 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000067c139485e2c197a-006477ab54-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:17:24 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:17:24.891 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2023-05-31T20:17:24.891 DEBUG:boto:path=/pklxkd-47/obj4 2023-05-31T20:17:24.891 DEBUG:boto:auth_path=/pklxkd-47/obj4 2023-05-31T20:17:24.892 DEBUG:boto:path=/pklxkd-47/obj4?versionId=null 2023-05-31T20:17:24.892 DEBUG:boto:auth_path=/pklxkd-47/obj4?versionId=null 2023-05-31T20:17:24.893 DEBUG:boto:Method: GET 2023-05-31T20:17:24.893 DEBUG:boto:Path: /pklxkd-47/obj4?versionId=null 2023-05-31T20:17:24.893 DEBUG:boto:Data: 2023-05-31T20:17:24.894 DEBUG:boto:Headers: {} 2023-05-31T20:17:24.894 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:17:24.894 DEBUG:boto:Port: 8001 2023-05-31T20:17:24.895 DEBUG:boto:Params: {} 2023-05-31T20:17:24.895 DEBUG:boto:Token: None 2023-05-31T20:17:24.895 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:17:24 GMT /pklxkd-47/obj4?versionId=null 2023-05-31T20:17:24.896 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:8VaUjv0QTquOlkRS6R4PdZmBQGg= 2023-05-31T20:17:24.896 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:17:24 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:8VaUjv0QTquOlkRS6R4PdZmBQGg=', 'Content-Length': '0'} 2023-05-31T20:17:24.898 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:15:11 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000f9739433c043450a-006477ab54-162e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:17:24 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:17:24.899 DEBUG:boto:path=/pklxkd-47/obj4 2023-05-31T20:17:24.899 DEBUG:boto:auth_path=/pklxkd-47/obj4 2023-05-31T20:17:24.899 DEBUG:boto:path=/pklxkd-47/obj4?versionId=null 2023-05-31T20:17:24.900 DEBUG:boto:auth_path=/pklxkd-47/obj4?versionId=null 2023-05-31T20:17:24.900 DEBUG:boto:Method: GET 2023-05-31T20:17:24.900 DEBUG:boto:Path: /pklxkd-47/obj4?versionId=null 2023-05-31T20:17:24.901 DEBUG:boto:Data: 2023-05-31T20:17:24.901 DEBUG:boto:Headers: {} 2023-05-31T20:17:24.901 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:17:24.902 DEBUG:boto:Port: 8000 2023-05-31T20:17:24.902 DEBUG:boto:Params: {} 2023-05-31T20:17:24.903 DEBUG:boto:Token: None 2023-05-31T20:17:24.903 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:17:24 GMT /pklxkd-47/obj4?versionId=null 2023-05-31T20:17:24.903 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:8VaUjv0QTquOlkRS6R4PdZmBQGg= 2023-05-31T20:17:24.904 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:17:24 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:8VaUjv0QTquOlkRS6R4PdZmBQGg=', 'Content-Length': '0'} 2023-05-31T20:17:24.905 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:15:11 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000975c8bc91c4aa9e6-006477ab54-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:17:24 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:17:24.906 DEBUG:boto:path=/pklxkd-47/ 2023-05-31T20:17:24.907 DEBUG:boto:auth_path=/pklxkd-47/ 2023-05-31T20:17:24.907 DEBUG:boto:path=/pklxkd-47/?prefix=obj4 2023-05-31T20:17:24.907 DEBUG:boto:auth_path=/pklxkd-47/?prefix=obj4 2023-05-31T20:17:24.908 DEBUG:boto:Method: GET 2023-05-31T20:17:24.908 DEBUG:boto:Path: /pklxkd-47/?prefix=obj4 2023-05-31T20:17:24.909 DEBUG:boto:Data: 2023-05-31T20:17:24.909 DEBUG:boto:Headers: {} 2023-05-31T20:17:24.909 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:17:24.910 DEBUG:boto:Port: 8001 2023-05-31T20:17:24.910 DEBUG:boto:Params: {} 2023-05-31T20:17:24.910 DEBUG:boto:Token: None 2023-05-31T20:17:24.911 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:17:24 GMT /pklxkd-47/ 2023-05-31T20:17:24.911 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:6CCqjBAujwkL/2E5qojwaI79Q9I= 2023-05-31T20:17:24.912 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:17:24 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:6CCqjBAujwkL/2E5qojwaI79Q9I=', 'Content-Length': '0'} 2023-05-31T20:17:24.914 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000006b35585c49a454cd-006477ab54-162e-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:17:24 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:17:24.914 DEBUG:boto:b'pklxkd-47obj41000falseobj42023-05-31T20:15:11.471Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:17:24.915 DEBUG:boto:path=/pklxkd-47/ 2023-05-31T20:17:24.916 DEBUG:boto:auth_path=/pklxkd-47/ 2023-05-31T20:17:24.916 DEBUG:boto:path=/pklxkd-47/?prefix=obj4 2023-05-31T20:17:24.916 DEBUG:boto:auth_path=/pklxkd-47/?prefix=obj4 2023-05-31T20:17:24.917 DEBUG:boto:Method: GET 2023-05-31T20:17:24.917 DEBUG:boto:Path: /pklxkd-47/?prefix=obj4 2023-05-31T20:17:24.917 DEBUG:boto:Data: 2023-05-31T20:17:24.918 DEBUG:boto:Headers: {} 2023-05-31T20:17:24.918 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:17:24.918 DEBUG:boto:Port: 8000 2023-05-31T20:17:24.919 DEBUG:boto:Params: {} 2023-05-31T20:17:24.919 DEBUG:boto:Token: None 2023-05-31T20:17:24.920 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:17:24 GMT /pklxkd-47/ 2023-05-31T20:17:24.920 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:6CCqjBAujwkL/2E5qojwaI79Q9I= 2023-05-31T20:17:24.920 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:17:24 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:6CCqjBAujwkL/2E5qojwaI79Q9I=', 'Content-Length': '0'} 2023-05-31T20:17:24.922 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000005bc8f2625c1fdcb7-006477ab54-1630-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:17:24 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:17:24.923 DEBUG:boto:b'pklxkd-47obj41000falseobj42023-05-31T20:15:11.471Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:17:24.924 DEBUG:boto:path=/pklxkd-47/obj4 2023-05-31T20:17:24.924 DEBUG:boto:auth_path=/pklxkd-47/obj4 2023-05-31T20:17:24.924 DEBUG:boto:Method: HEAD 2023-05-31T20:17:24.925 DEBUG:boto:Path: /pklxkd-47/obj4 2023-05-31T20:17:24.925 DEBUG:boto:Data: 2023-05-31T20:17:24.925 DEBUG:boto:Headers: {} 2023-05-31T20:17:24.926 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:17:24.926 DEBUG:boto:Port: 8001 2023-05-31T20:17:24.927 DEBUG:boto:Params: {} 2023-05-31T20:17:24.927 DEBUG:boto:Token: None 2023-05-31T20:17:24.928 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:17:24 GMT /pklxkd-47/obj4 2023-05-31T20:17:24.928 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:9OxGl4r9/c7g1/p8x51RHynF4+0= 2023-05-31T20:17:24.928 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:17:24 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:9OxGl4r9/c7g1/p8x51RHynF4+0=', 'Content-Length': '0'} 2023-05-31T20:17:24.930 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:15:11 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000003993655c917a6097-006477ab54-162e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:17:24 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:17:24.931 DEBUG:boto:path=/pklxkd-47/obj4 2023-05-31T20:17:24.931 DEBUG:boto:auth_path=/pklxkd-47/obj4 2023-05-31T20:17:24.931 DEBUG:boto:Method: HEAD 2023-05-31T20:17:24.932 DEBUG:boto:Path: /pklxkd-47/obj4 2023-05-31T20:17:24.932 DEBUG:boto:Data: 2023-05-31T20:17:24.932 DEBUG:boto:Headers: {} 2023-05-31T20:17:24.933 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:17:24.933 DEBUG:boto:Port: 8000 2023-05-31T20:17:24.933 DEBUG:boto:Params: {} 2023-05-31T20:17:24.934 DEBUG:boto:Token: None 2023-05-31T20:17:24.934 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:17:24 GMT /pklxkd-47/obj4 2023-05-31T20:17:24.935 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:9OxGl4r9/c7g1/p8x51RHynF4+0= 2023-05-31T20:17:24.935 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:17:24 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:9OxGl4r9/c7g1/p8x51RHynF4+0=', 'Content-Length': '0'} 2023-05-31T20:17:24.936 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:15:11 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000006a85e8f550ec6b53-006477ab54-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:17:24 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:17:24.937 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2023-05-31T20:17:24.937 DEBUG:boto:path=/pklxkd-47/obj4 2023-05-31T20:17:24.938 DEBUG:boto:auth_path=/pklxkd-47/obj4 2023-05-31T20:17:24.938 DEBUG:boto:Method: GET 2023-05-31T20:17:24.938 DEBUG:boto:Path: /pklxkd-47/obj4 2023-05-31T20:17:24.939 DEBUG:boto:Data: 2023-05-31T20:17:24.939 DEBUG:boto:Headers: {} 2023-05-31T20:17:24.940 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:17:24.940 DEBUG:boto:Port: 8001 2023-05-31T20:17:24.940 DEBUG:boto:Params: {} 2023-05-31T20:17:24.941 DEBUG:boto:Token: None 2023-05-31T20:17:24.941 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:17:24 GMT /pklxkd-47/obj4 2023-05-31T20:17:24.941 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:EjuseHTpd1NtlegCgpft3xpUWhI= 2023-05-31T20:17:24.942 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:17:24 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:EjuseHTpd1NtlegCgpft3xpUWhI=', 'Content-Length': '0'} 2023-05-31T20:17:24.943 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:15:11 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000aca9c071c68e739a-006477ab54-162e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:17:24 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:17:24.944 DEBUG:boto:path=/pklxkd-47/obj4 2023-05-31T20:17:24.944 DEBUG:boto:auth_path=/pklxkd-47/obj4 2023-05-31T20:17:24.945 DEBUG:boto:Method: GET 2023-05-31T20:17:24.945 DEBUG:boto:Path: /pklxkd-47/obj4 2023-05-31T20:17:24.946 DEBUG:boto:Data: 2023-05-31T20:17:24.946 DEBUG:boto:Headers: {} 2023-05-31T20:17:24.946 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:17:24.947 DEBUG:boto:Port: 8000 2023-05-31T20:17:24.947 DEBUG:boto:Params: {} 2023-05-31T20:17:24.947 DEBUG:boto:Token: None 2023-05-31T20:17:24.948 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:17:24 GMT /pklxkd-47/obj4 2023-05-31T20:17:24.948 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:EjuseHTpd1NtlegCgpft3xpUWhI= 2023-05-31T20:17:24.949 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:17:24 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:EjuseHTpd1NtlegCgpft3xpUWhI=', 'Content-Length': '0'} 2023-05-31T20:17:24.950 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:15:11 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000007bf88837838d8df2-006477ab54-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:17:24 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:17:24.951 DEBUG:boto:path=/pklxkd-47/ 2023-05-31T20:17:24.952 DEBUG:boto:auth_path=/pklxkd-47/ 2023-05-31T20:17:24.952 DEBUG:boto:path=/pklxkd-47/?prefix=obj4 2023-05-31T20:17:24.952 DEBUG:boto:auth_path=/pklxkd-47/?prefix=obj4 2023-05-31T20:17:24.953 DEBUG:boto:Method: GET 2023-05-31T20:17:24.953 DEBUG:boto:Path: /pklxkd-47/?prefix=obj4 2023-05-31T20:17:24.953 DEBUG:boto:Data: 2023-05-31T20:17:24.954 DEBUG:boto:Headers: {} 2023-05-31T20:17:24.954 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:17:24.954 DEBUG:boto:Port: 8001 2023-05-31T20:17:24.955 DEBUG:boto:Params: {} 2023-05-31T20:17:24.955 DEBUG:boto:Token: None 2023-05-31T20:17:24.956 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:17:24 GMT /pklxkd-47/ 2023-05-31T20:17:24.956 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:6CCqjBAujwkL/2E5qojwaI79Q9I= 2023-05-31T20:17:24.957 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:17:24 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:6CCqjBAujwkL/2E5qojwaI79Q9I=', 'Content-Length': '0'} 2023-05-31T20:17:24.959 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e6f0dcff02ba1dbb-006477ab54-162e-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:17:24 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:17:24.959 DEBUG:boto:b'pklxkd-47obj41000falseobj42023-05-31T20:15:11.471Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:17:24.960 DEBUG:boto:path=/pklxkd-47/ 2023-05-31T20:17:24.960 DEBUG:boto:auth_path=/pklxkd-47/ 2023-05-31T20:17:24.961 DEBUG:boto:path=/pklxkd-47/?prefix=obj4 2023-05-31T20:17:24.961 DEBUG:boto:auth_path=/pklxkd-47/?prefix=obj4 2023-05-31T20:17:24.962 DEBUG:boto:Method: GET 2023-05-31T20:17:24.962 DEBUG:boto:Path: /pklxkd-47/?prefix=obj4 2023-05-31T20:17:24.962 DEBUG:boto:Data: 2023-05-31T20:17:24.963 DEBUG:boto:Headers: {} 2023-05-31T20:17:24.963 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:17:24.963 DEBUG:boto:Port: 8000 2023-05-31T20:17:24.964 DEBUG:boto:Params: {} 2023-05-31T20:17:24.964 DEBUG:boto:Token: None 2023-05-31T20:17:24.964 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:17:24 GMT /pklxkd-47/ 2023-05-31T20:17:24.965 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:6CCqjBAujwkL/2E5qojwaI79Q9I= 2023-05-31T20:17:24.965 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:17:24 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:6CCqjBAujwkL/2E5qojwaI79Q9I=', 'Content-Length': '0'} 2023-05-31T20:17:24.967 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ed68938ca3bf51c4-006477ab54-1630-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:17:24 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:17:24.968 DEBUG:boto:b'pklxkd-47obj41000falseobj42023-05-31T20:15:11.471Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:17:24.969 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=pklxkd-47 zones={a2, a1} 2023-05-31T20:17:24.969 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'disable', '--bucket', 'pklxkd-46', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-05-31T20:17:24.969 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'disable', '--bucket', 'pklxkd-46', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-05-31T20:17:24.970 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync disable --bucket pklxkd-46 --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 2023-05-31T20:17:25.020 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2023-05-31T20:17:25.020 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2023-05-31T20:17:25.116 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'enable', '--bucket', 'pklxkd-46', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-05-31T20:17:25.116 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'enable', '--bucket', 'pklxkd-46', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-05-31T20:17:25.117 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync enable --bucket pklxkd-46 --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 2023-05-31T20:17:25.166 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2023-05-31T20:17:25.167 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2023-05-31T20:17:25.280 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'disable', '--bucket', 'pklxkd-47', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-05-31T20:17:25.281 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'disable', '--bucket', 'pklxkd-47', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-05-31T20:17:25.281 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync disable --bucket pklxkd-47 --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 2023-05-31T20:17:25.331 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2023-05-31T20:17:25.331 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2023-05-31T20:17:25.438 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'enable', '--bucket', 'pklxkd-47', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-05-31T20:17:25.439 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'enable', '--bucket', 'pklxkd-47', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-05-31T20:17:25.439 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync enable --bucket pklxkd-47 --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 2023-05-31T20:17:25.489 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2023-05-31T20:17:25.490 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2023-05-31T20:17:25.580 DEBUG:boto:path=/pklxkd-46/ 2023-05-31T20:17:25.580 DEBUG:boto:auth_path=/pklxkd-46/ 2023-05-31T20:17:25.581 DEBUG:boto:Method: HEAD 2023-05-31T20:17:25.581 DEBUG:boto:Path: /pklxkd-46/ 2023-05-31T20:17:25.581 DEBUG:boto:Data: 2023-05-31T20:17:25.582 DEBUG:boto:Headers: {} 2023-05-31T20:17:25.582 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:17:25.582 DEBUG:boto:Port: 8000 2023-05-31T20:17:25.583 DEBUG:boto:Params: {} 2023-05-31T20:17:25.583 DEBUG:boto:Token: None 2023-05-31T20:17:25.584 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:17:25 GMT /pklxkd-46/ 2023-05-31T20:17:25.584 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:wLP2T+Tov9ofW9HyU1Xl6igr2GA= 2023-05-31T20:17:25.584 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:17:25 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:wLP2T+Tov9ofW9HyU1Xl6igr2GA=', 'Content-Length': '0'} 2023-05-31T20:17:25.586 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '4'), ('X-RGW-Bytes-Used', '24'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000008a9af665dce3c533-006477ab55-1630-a1'), ('Content-Length', '0'), ('Date', 'Wed, 31 May 2023 20:17:25 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:17:25.587 DEBUG:boto:path=/pklxkd-46/obj5 2023-05-31T20:17:25.588 DEBUG:boto:auth_path=/pklxkd-46/obj5 2023-05-31T20:17:25.588 DEBUG:boto:Method: PUT 2023-05-31T20:17:25.588 DEBUG:boto:Path: /pklxkd-46/obj5 2023-05-31T20:17:25.589 DEBUG:boto:Data: 2023-05-31T20:17:25.589 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2023-05-31T20:17:25.589 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:17:25.590 DEBUG:boto:Port: 8000 2023-05-31T20:17:25.590 DEBUG:boto:Params: {} 2023-05-31T20:17:25.590 DEBUG:boto:Token: None 2023-05-31T20:17:25.591 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Wed, 31 May 2023 20:17:25 GMT /pklxkd-46/obj5 2023-05-31T20:17:25.591 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:8NJrYBEpcBjejZzCtKN/4C9KXFU= 2023-05-31T20:17:25.592 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Wed, 31 May 2023 20:17:25 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:8NJrYBEpcBjejZzCtKN/4C9KXFU='} 2023-05-31T20:17:25.600 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000a9fc84addae61fd9-006477ab55-1630-a1'), ('Date', 'Wed, 31 May 2023 20:17:25 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:17:25.601 DEBUG:boto:path=/pklxkd-46/ 2023-05-31T20:17:25.601 DEBUG:boto:auth_path=/pklxkd-46/ 2023-05-31T20:17:25.602 DEBUG:boto:Method: HEAD 2023-05-31T20:17:25.602 DEBUG:boto:Path: /pklxkd-46/ 2023-05-31T20:17:25.602 DEBUG:boto:Data: 2023-05-31T20:17:25.603 DEBUG:boto:Headers: {} 2023-05-31T20:17:25.603 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:17:25.603 DEBUG:boto:Port: 8000 2023-05-31T20:17:25.604 DEBUG:boto:Params: {} 2023-05-31T20:17:25.604 DEBUG:boto:Token: None 2023-05-31T20:17:25.605 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:17:25 GMT /pklxkd-46/ 2023-05-31T20:17:25.605 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:wLP2T+Tov9ofW9HyU1Xl6igr2GA= 2023-05-31T20:17:25.605 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:17:25 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:wLP2T+Tov9ofW9HyU1Xl6igr2GA=', 'Content-Length': '0'} 2023-05-31T20:17:25.607 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '5'), ('X-RGW-Bytes-Used', '30'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000d5b891ebba095dee-006477ab55-1630-a1'), ('Content-Length', '0'), ('Date', 'Wed, 31 May 2023 20:17:25 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:17:25.608 DEBUG:boto:path=/pklxkd-46/obj6 2023-05-31T20:17:25.608 DEBUG:boto:auth_path=/pklxkd-46/obj6 2023-05-31T20:17:25.609 DEBUG:boto:Method: PUT 2023-05-31T20:17:25.609 DEBUG:boto:Path: /pklxkd-46/obj6 2023-05-31T20:17:25.609 DEBUG:boto:Data: 2023-05-31T20:17:25.610 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2023-05-31T20:17:25.610 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:17:25.610 DEBUG:boto:Port: 8000 2023-05-31T20:17:25.611 DEBUG:boto:Params: {} 2023-05-31T20:17:25.611 DEBUG:boto:Token: None 2023-05-31T20:17:25.612 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Wed, 31 May 2023 20:17:25 GMT /pklxkd-46/obj6 2023-05-31T20:17:25.612 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:mbKF8no7A6ag2Qz61cEQXahZaro= 2023-05-31T20:17:25.612 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Wed, 31 May 2023 20:17:25 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:mbKF8no7A6ag2Qz61cEQXahZaro='} 2023-05-31T20:17:25.656 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000001690e45faabf7c8c-006477ab55-1630-a1'), ('Date', 'Wed, 31 May 2023 20:17:25 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:17:25.657 DEBUG:boto:path=/pklxkd-46/ 2023-05-31T20:17:25.657 DEBUG:boto:auth_path=/pklxkd-46/ 2023-05-31T20:17:25.658 DEBUG:boto:Method: HEAD 2023-05-31T20:17:25.659 DEBUG:boto:Path: /pklxkd-46/ 2023-05-31T20:17:25.660 DEBUG:boto:Data: 2023-05-31T20:17:25.661 DEBUG:boto:Headers: {} 2023-05-31T20:17:25.662 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:17:25.663 DEBUG:boto:Port: 8000 2023-05-31T20:17:25.663 DEBUG:boto:Params: {} 2023-05-31T20:17:25.664 DEBUG:boto:Token: None 2023-05-31T20:17:25.665 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:17:25 GMT /pklxkd-46/ 2023-05-31T20:17:25.666 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:wLP2T+Tov9ofW9HyU1Xl6igr2GA= 2023-05-31T20:17:25.667 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:17:25 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:wLP2T+Tov9ofW9HyU1Xl6igr2GA=', 'Content-Length': '0'} 2023-05-31T20:17:25.669 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '6'), ('X-RGW-Bytes-Used', '36'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000007c795d9c95fe2e4e-006477ab55-1630-a1'), ('Content-Length', '0'), ('Date', 'Wed, 31 May 2023 20:17:25 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:17:25.670 DEBUG:boto:path=/pklxkd-46/obj7 2023-05-31T20:17:25.670 DEBUG:boto:auth_path=/pklxkd-46/obj7 2023-05-31T20:17:25.671 DEBUG:boto:Method: PUT 2023-05-31T20:17:25.671 DEBUG:boto:Path: /pklxkd-46/obj7 2023-05-31T20:17:25.671 DEBUG:boto:Data: 2023-05-31T20:17:25.672 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2023-05-31T20:17:25.672 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:17:25.672 DEBUG:boto:Port: 8000 2023-05-31T20:17:25.673 DEBUG:boto:Params: {} 2023-05-31T20:17:25.673 DEBUG:boto:Token: None 2023-05-31T20:17:25.674 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Wed, 31 May 2023 20:17:25 GMT /pklxkd-46/obj7 2023-05-31T20:17:25.675 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:vP/oDy1F3q+c6ueRvp7z8vbg48A= 2023-05-31T20:17:25.675 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Wed, 31 May 2023 20:17:25 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:vP/oDy1F3q+c6ueRvp7z8vbg48A='} 2023-05-31T20:17:25.682 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000e83b33b36c958cbb-006477ab55-1630-a1'), ('Date', 'Wed, 31 May 2023 20:17:25 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:17:25.683 DEBUG:boto:path=/pklxkd-46/ 2023-05-31T20:17:25.683 DEBUG:boto:auth_path=/pklxkd-46/ 2023-05-31T20:17:25.684 DEBUG:boto:Method: HEAD 2023-05-31T20:17:25.684 DEBUG:boto:Path: /pklxkd-46/ 2023-05-31T20:17:25.684 DEBUG:boto:Data: 2023-05-31T20:17:25.685 DEBUG:boto:Headers: {} 2023-05-31T20:17:25.685 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:17:25.685 DEBUG:boto:Port: 8000 2023-05-31T20:17:25.686 DEBUG:boto:Params: {} 2023-05-31T20:17:25.686 DEBUG:boto:Token: None 2023-05-31T20:17:25.687 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:17:25 GMT /pklxkd-46/ 2023-05-31T20:17:25.687 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:wLP2T+Tov9ofW9HyU1Xl6igr2GA= 2023-05-31T20:17:25.688 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:17:25 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:wLP2T+Tov9ofW9HyU1Xl6igr2GA=', 'Content-Length': '0'} 2023-05-31T20:17:25.690 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '7'), ('X-RGW-Bytes-Used', '42'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000afeac9bb7f1c0455-006477ab55-1630-a1'), ('Content-Length', '0'), ('Date', 'Wed, 31 May 2023 20:17:25 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:17:25.691 DEBUG:boto:path=/pklxkd-46/obj8 2023-05-31T20:17:25.692 DEBUG:boto:auth_path=/pklxkd-46/obj8 2023-05-31T20:17:25.692 DEBUG:boto:Method: PUT 2023-05-31T20:17:25.693 DEBUG:boto:Path: /pklxkd-46/obj8 2023-05-31T20:17:25.693 DEBUG:boto:Data: 2023-05-31T20:17:25.694 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2023-05-31T20:17:25.694 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:17:25.695 DEBUG:boto:Port: 8000 2023-05-31T20:17:25.695 DEBUG:boto:Params: {} 2023-05-31T20:17:25.696 DEBUG:boto:Token: None 2023-05-31T20:17:25.697 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Wed, 31 May 2023 20:17:25 GMT /pklxkd-46/obj8 2023-05-31T20:17:25.697 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:uILi0zLHgfzOQOPmHQAZXFVV8D4= 2023-05-31T20:17:25.698 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Wed, 31 May 2023 20:17:25 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:uILi0zLHgfzOQOPmHQAZXFVV8D4='} 2023-05-31T20:17:25.740 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000009a2fe2b892de5c87-006477ab55-1630-a1'), ('Date', 'Wed, 31 May 2023 20:17:25 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:17:25.741 DEBUG:boto:path=/pklxkd-47/ 2023-05-31T20:17:25.741 DEBUG:boto:auth_path=/pklxkd-47/ 2023-05-31T20:17:25.742 DEBUG:boto:Method: HEAD 2023-05-31T20:17:25.742 DEBUG:boto:Path: /pklxkd-47/ 2023-05-31T20:17:25.742 DEBUG:boto:Data: 2023-05-31T20:17:25.743 DEBUG:boto:Headers: {} 2023-05-31T20:17:25.743 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:17:25.743 DEBUG:boto:Port: 8001 2023-05-31T20:17:25.744 DEBUG:boto:Params: {} 2023-05-31T20:17:25.744 DEBUG:boto:Token: None 2023-05-31T20:17:25.745 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:17:25 GMT /pklxkd-47/ 2023-05-31T20:17:25.745 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:SxujIFe8dSIUZtnTgCKGQvcX5xs= 2023-05-31T20:17:25.746 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:17:25 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:SxujIFe8dSIUZtnTgCKGQvcX5xs=', 'Content-Length': '0'} 2023-05-31T20:17:25.747 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '4'), ('X-RGW-Bytes-Used', '24'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000007a62dadaf6931a15-006477ab55-162e-a2'), ('Content-Length', '0'), ('Date', 'Wed, 31 May 2023 20:17:25 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:17:25.748 DEBUG:boto:path=/pklxkd-47/obj5 2023-05-31T20:17:25.748 DEBUG:boto:auth_path=/pklxkd-47/obj5 2023-05-31T20:17:25.749 DEBUG:boto:Method: PUT 2023-05-31T20:17:25.749 DEBUG:boto:Path: /pklxkd-47/obj5 2023-05-31T20:17:25.750 DEBUG:boto:Data: 2023-05-31T20:17:25.751 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2023-05-31T20:17:25.752 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:17:25.753 DEBUG:boto:Port: 8001 2023-05-31T20:17:25.753 DEBUG:boto:Params: {} 2023-05-31T20:17:25.753 DEBUG:boto:Token: None 2023-05-31T20:17:25.754 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Wed, 31 May 2023 20:17:25 GMT /pklxkd-47/obj5 2023-05-31T20:17:25.754 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:en4DDXb9QVQqd0anRbeJPGtY0Z0= 2023-05-31T20:17:25.755 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Wed, 31 May 2023 20:17:25 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:en4DDXb9QVQqd0anRbeJPGtY0Z0='} 2023-05-31T20:17:25.760 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000bc6c3c485a2554eb-006477ab55-162e-a2'), ('Date', 'Wed, 31 May 2023 20:17:25 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:17:25.760 DEBUG:boto:path=/pklxkd-47/ 2023-05-31T20:17:25.761 DEBUG:boto:auth_path=/pklxkd-47/ 2023-05-31T20:17:25.761 DEBUG:boto:Method: HEAD 2023-05-31T20:17:25.761 DEBUG:boto:Path: /pklxkd-47/ 2023-05-31T20:17:25.762 DEBUG:boto:Data: 2023-05-31T20:17:25.762 DEBUG:boto:Headers: {} 2023-05-31T20:17:25.762 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:17:25.763 DEBUG:boto:Port: 8001 2023-05-31T20:17:25.763 DEBUG:boto:Params: {} 2023-05-31T20:17:25.764 DEBUG:boto:Token: None 2023-05-31T20:17:25.764 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:17:25 GMT /pklxkd-47/ 2023-05-31T20:17:25.764 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:SxujIFe8dSIUZtnTgCKGQvcX5xs= 2023-05-31T20:17:25.765 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:17:25 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:SxujIFe8dSIUZtnTgCKGQvcX5xs=', 'Content-Length': '0'} 2023-05-31T20:17:25.767 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '5'), ('X-RGW-Bytes-Used', '30'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000000e8ea3e4b3199e21-006477ab55-162e-a2'), ('Content-Length', '0'), ('Date', 'Wed, 31 May 2023 20:17:25 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:17:25.768 DEBUG:boto:path=/pklxkd-47/obj6 2023-05-31T20:17:25.768 DEBUG:boto:auth_path=/pklxkd-47/obj6 2023-05-31T20:17:25.768 DEBUG:boto:Method: PUT 2023-05-31T20:17:25.769 DEBUG:boto:Path: /pklxkd-47/obj6 2023-05-31T20:17:25.769 DEBUG:boto:Data: 2023-05-31T20:17:25.770 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2023-05-31T20:17:25.770 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:17:25.770 DEBUG:boto:Port: 8001 2023-05-31T20:17:25.771 DEBUG:boto:Params: {} 2023-05-31T20:17:25.771 DEBUG:boto:Token: None 2023-05-31T20:17:25.771 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Wed, 31 May 2023 20:17:25 GMT /pklxkd-47/obj6 2023-05-31T20:17:25.772 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:oGqdHJJyCVYNsDLs8z+AeI2woJg= 2023-05-31T20:17:25.772 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Wed, 31 May 2023 20:17:25 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:oGqdHJJyCVYNsDLs8z+AeI2woJg='} 2023-05-31T20:17:25.818 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000008fad14a0999876cf-006477ab55-162e-a2'), ('Date', 'Wed, 31 May 2023 20:17:25 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:17:25.819 DEBUG:boto:path=/pklxkd-47/ 2023-05-31T20:17:25.819 DEBUG:boto:auth_path=/pklxkd-47/ 2023-05-31T20:17:25.819 DEBUG:boto:Method: HEAD 2023-05-31T20:17:25.820 DEBUG:boto:Path: /pklxkd-47/ 2023-05-31T20:17:25.820 DEBUG:boto:Data: 2023-05-31T20:17:25.820 DEBUG:boto:Headers: {} 2023-05-31T20:17:25.821 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:17:25.821 DEBUG:boto:Port: 8001 2023-05-31T20:17:25.821 DEBUG:boto:Params: {} 2023-05-31T20:17:25.822 DEBUG:boto:Token: None 2023-05-31T20:17:25.822 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:17:25 GMT /pklxkd-47/ 2023-05-31T20:17:25.823 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:SxujIFe8dSIUZtnTgCKGQvcX5xs= 2023-05-31T20:17:25.823 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:17:25 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:SxujIFe8dSIUZtnTgCKGQvcX5xs=', 'Content-Length': '0'} 2023-05-31T20:17:25.825 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '6'), ('X-RGW-Bytes-Used', '36'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000006706f3e656d2a2c-006477ab55-162e-a2'), ('Content-Length', '0'), ('Date', 'Wed, 31 May 2023 20:17:25 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:17:25.826 DEBUG:boto:path=/pklxkd-47/obj7 2023-05-31T20:17:25.827 DEBUG:boto:auth_path=/pklxkd-47/obj7 2023-05-31T20:17:25.828 DEBUG:boto:Method: PUT 2023-05-31T20:17:25.828 DEBUG:boto:Path: /pklxkd-47/obj7 2023-05-31T20:17:25.829 DEBUG:boto:Data: 2023-05-31T20:17:25.830 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2023-05-31T20:17:25.830 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:17:25.831 DEBUG:boto:Port: 8001 2023-05-31T20:17:25.832 DEBUG:boto:Params: {} 2023-05-31T20:17:25.832 DEBUG:boto:Token: None 2023-05-31T20:17:25.834 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Wed, 31 May 2023 20:17:25 GMT /pklxkd-47/obj7 2023-05-31T20:17:25.834 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:sAuPcaJ3EFANSWYTjUeoKkPF6wM= 2023-05-31T20:17:25.835 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Wed, 31 May 2023 20:17:25 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:sAuPcaJ3EFANSWYTjUeoKkPF6wM='} 2023-05-31T20:17:25.843 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000d0be840254a2d6d4-006477ab55-162e-a2'), ('Date', 'Wed, 31 May 2023 20:17:25 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:17:25.844 DEBUG:boto:path=/pklxkd-47/ 2023-05-31T20:17:25.845 DEBUG:boto:auth_path=/pklxkd-47/ 2023-05-31T20:17:25.845 DEBUG:boto:Method: HEAD 2023-05-31T20:17:25.845 DEBUG:boto:Path: /pklxkd-47/ 2023-05-31T20:17:25.846 DEBUG:boto:Data: 2023-05-31T20:17:25.846 DEBUG:boto:Headers: {} 2023-05-31T20:17:25.846 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:17:25.847 DEBUG:boto:Port: 8001 2023-05-31T20:17:25.848 DEBUG:boto:Params: {} 2023-05-31T20:17:25.848 DEBUG:boto:Token: None 2023-05-31T20:17:25.848 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:17:25 GMT /pklxkd-47/ 2023-05-31T20:17:25.849 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:SxujIFe8dSIUZtnTgCKGQvcX5xs= 2023-05-31T20:17:25.849 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:17:25 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:SxujIFe8dSIUZtnTgCKGQvcX5xs=', 'Content-Length': '0'} 2023-05-31T20:17:25.851 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '7'), ('X-RGW-Bytes-Used', '42'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000066482ecf00d325fe-006477ab55-162e-a2'), ('Content-Length', '0'), ('Date', 'Wed, 31 May 2023 20:17:25 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:17:25.852 DEBUG:boto:path=/pklxkd-47/obj8 2023-05-31T20:17:25.852 DEBUG:boto:auth_path=/pklxkd-47/obj8 2023-05-31T20:17:25.853 DEBUG:boto:Method: PUT 2023-05-31T20:17:25.853 DEBUG:boto:Path: /pklxkd-47/obj8 2023-05-31T20:17:25.853 DEBUG:boto:Data: 2023-05-31T20:17:25.854 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2023-05-31T20:17:25.854 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:17:25.854 DEBUG:boto:Port: 8001 2023-05-31T20:17:25.855 DEBUG:boto:Params: {} 2023-05-31T20:17:25.855 DEBUG:boto:Token: None 2023-05-31T20:17:25.856 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Wed, 31 May 2023 20:17:25 GMT /pklxkd-47/obj8 2023-05-31T20:17:25.856 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:12YUiKM+gREUQ7nBxfBjpGxsLTg= 2023-05-31T20:17:25.856 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Wed, 31 May 2023 20:17:25 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:12YUiKM+gREUQ7nBxfBjpGxsLTg='} 2023-05-31T20:17:25.905 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000024d1700dd0e1f576-006477ab55-162e-a2'), ('Date', 'Wed, 31 May 2023 20:17:25 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:17:25.907 DEBUG:rgw_multi.tests:bucket checkpoint: source=a1 target=a2 bucket=pklxkd-46 2023-05-31T20:17:25.908 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'pklxkd-46', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-05-31T20:17:25.909 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'pklxkd-46', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-05-31T20:17:25.910 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket pklxkd-46 --source-zone a1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2023-05-31T20:17:25.967 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2023-05-31T20:17:25.967 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2023-05-31T20:17:26.013 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:17:26.008+0000 7fdd91b8b780 1 waiting for incremental sync to catch up: 2023-05-31T20:17:26.013 INFO:teuthology.orchestra.run.smithi099.stderr: local status: [00000000001.18.6, , 00000000001.22.6, , 00000000001.14.6, , , 00000000001.23.6, , , ] 2023-05-31T20:17:26.013 INFO:teuthology.orchestra.run.smithi099.stderr: remote markers: [00000000004.22.6, 00000000002.44.1, 00000000004.26.6, 00000000002.26.1, 00000000004.18.6, 00000000002.17.1, 00000000002.16.1, 00000000004.29.6, 00000000002.24.1, 00000000002.25.1, 00000000002.23.1] 2023-05-31T20:17:31.014 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:17:31.009+0000 7fdd91b8b780 1 waiting for incremental sync to catch up: 2023-05-31T20:17:31.015 INFO:teuthology.orchestra.run.smithi099.stderr: local status: [00000000001.18.6, , 00000000001.22.6, , 00000000001.14.6, , , 00000000001.23.6, , , ] 2023-05-31T20:17:31.015 INFO:teuthology.orchestra.run.smithi099.stderr: remote markers: [00000000004.22.6, 00000000002.44.1, 00000000004.26.6, 00000000002.26.1, 00000000004.18.6, 00000000002.17.1, 00000000002.16.1, 00000000004.29.6, 00000000002.24.1, 00000000002.25.1, 00000000002.23.1] 2023-05-31T20:17:36.016 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:17:36.011+0000 7fdd91b8b780 1 waiting for incremental sync to catch up: 2023-05-31T20:17:36.017 INFO:teuthology.orchestra.run.smithi099.stderr: local status: [00000000001.18.6, , 00000000001.22.6, , 00000000001.14.6, , , 00000000001.23.6, , , ] 2023-05-31T20:17:36.017 INFO:teuthology.orchestra.run.smithi099.stderr: remote markers: [00000000004.22.6, 00000000002.44.1, 00000000004.26.6, 00000000002.26.1, 00000000004.18.6, 00000000002.17.1, 00000000002.16.1, 00000000004.29.6, 00000000002.24.1, 00000000002.25.1, 00000000002.23.1] 2023-05-31T20:17:41.017 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:17:41.012+0000 7fdd91b8b780 1 bucket sync caught up with source: 2023-05-31T20:17:41.018 INFO:teuthology.orchestra.run.smithi099.stderr: local status: [00000000004.22.6, 00000000002.44.1, 00000000004.26.6, 00000000002.26.1, 00000000004.18.6, 00000000002.17.1, 00000000002.16.1, 00000000004.29.6, 00000000002.24.1, 00000000002.25.1, 00000000002.23.1] 2023-05-31T20:17:41.018 INFO:teuthology.orchestra.run.smithi099.stderr: remote markers: [00000000004.22.6, 00000000002.44.1, 00000000004.26.6, 00000000002.26.1, 00000000004.18.6, 00000000002.17.1, 00000000002.16.1, 00000000004.29.6, 00000000002.24.1, 00000000002.25.1, 00000000002.23.1] 2023-05-31T20:17:41.018 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:17:41.012+0000 7fdd91b8b780 0 bucket checkpoint complete 2023-05-31T20:17:41.034 DEBUG:rgw_multi.tests:bucket checkpoint: source=a2 target=a1 bucket=pklxkd-46 2023-05-31T20:17:41.035 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'pklxkd-46', '--source-zone', 'a2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-05-31T20:17:41.035 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'pklxkd-46', '--source-zone', 'a2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-05-31T20:17:41.036 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket pklxkd-46 --source-zone a2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2023-05-31T20:17:41.097 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2023-05-31T20:17:41.098 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2023-05-31T20:17:41.139 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:17:41.134+0000 7f970ff65780 1 waiting for incremental sync to catch up: 2023-05-31T20:17:41.139 INFO:teuthology.orchestra.run.smithi099.stderr: local status: [00000000001.1473.6, , 00000000001.1182.6, , 00000000001.2018.6, , , 00000000001.664.6, , , ] 2023-05-31T20:17:41.139 INFO:teuthology.orchestra.run.smithi099.stderr: remote markers: [00000000002.1648.6, , 00000000002.1333.6, , 00000000002.2248.6, , , 00000000002.789.6, , , ] 2023-05-31T20:17:46.140 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:17:46.134+0000 7f970ff65780 1 waiting for incremental sync to catch up: 2023-05-31T20:17:46.140 INFO:teuthology.orchestra.run.smithi099.stderr: local status: [00000000001.1473.6, , 00000000001.1182.6, , 00000000001.2018.6, , , 00000000001.664.6, , , ] 2023-05-31T20:17:46.141 INFO:teuthology.orchestra.run.smithi099.stderr: remote markers: [00000000002.1648.6, , 00000000002.1333.6, , 00000000002.2248.6, , , 00000000002.789.6, , , ] 2023-05-31T20:17:51.142 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:17:51.136+0000 7f970ff65780 1 waiting for incremental sync to catch up: 2023-05-31T20:17:51.142 INFO:teuthology.orchestra.run.smithi099.stderr: local status: [00000000001.1473.6, , 00000000001.1182.6, , 00000000001.2018.6, , , 00000000001.664.6, , , ] 2023-05-31T20:17:51.142 INFO:teuthology.orchestra.run.smithi099.stderr: remote markers: [00000000002.1648.6, , 00000000002.1333.6, , 00000000002.2248.6, , , 00000000002.789.6, , , ] 2023-05-31T20:17:56.143 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:17:56.137+0000 7f970ff65780 1 waiting for incremental sync to catch up: 2023-05-31T20:17:56.143 INFO:teuthology.orchestra.run.smithi099.stderr: local status: [00000000001.1473.6, , 00000000001.1182.6, , 00000000001.2018.6, , , 00000000001.664.6, , , ] 2023-05-31T20:17:56.143 INFO:teuthology.orchestra.run.smithi099.stderr: remote markers: [00000000002.1648.6, , 00000000002.1333.6, , 00000000002.2248.6, , , 00000000002.789.6, , , ] 2023-05-31T20:18:01.144 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:18:01.138+0000 7f970ff65780 1 bucket sync caught up with source: 2023-05-31T20:18:01.145 INFO:teuthology.orchestra.run.smithi099.stderr: local status: [00000000002.1648.6, , 00000000002.1333.6, , 00000000002.2248.6, , , 00000000002.789.6, , , ] 2023-05-31T20:18:01.145 INFO:teuthology.orchestra.run.smithi099.stderr: remote markers: [00000000002.1648.6, , 00000000002.1333.6, , 00000000002.2248.6, , , 00000000002.789.6, , , ] 2023-05-31T20:18:01.145 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:18:01.138+0000 7f970ff65780 0 bucket checkpoint complete 2023-05-31T20:18:01.162 INFO:tasks.rgw_multi.zone_rados:comparing bucket=pklxkd-46 zones={a2, a1} 2023-05-31T20:18:01.162 DEBUG:boto:path=/pklxkd-46/ 2023-05-31T20:18:01.163 DEBUG:boto:auth_path=/pklxkd-46/ 2023-05-31T20:18:01.163 DEBUG:boto:Method: HEAD 2023-05-31T20:18:01.163 DEBUG:boto:Path: /pklxkd-46/ 2023-05-31T20:18:01.164 DEBUG:boto:Data: 2023-05-31T20:18:01.164 DEBUG:boto:Headers: {} 2023-05-31T20:18:01.164 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:18:01.165 DEBUG:boto:Port: 8001 2023-05-31T20:18:01.165 DEBUG:boto:Params: {} 2023-05-31T20:18:01.165 DEBUG:boto:Token: None 2023-05-31T20:18:01.166 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:18:01 GMT /pklxkd-46/ 2023-05-31T20:18:01.166 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:JaW/CZZcmItppsd8PB6hayhY7LM= 2023-05-31T20:18:01.167 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:18:01 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:JaW/CZZcmItppsd8PB6hayhY7LM=', 'Content-Length': '0'} 2023-05-31T20:18:01.169 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '8'), ('X-RGW-Bytes-Used', '48'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000a37e0417be87ea2e-006477ab79-162e-a2'), ('Content-Length', '0'), ('Date', 'Wed, 31 May 2023 20:18:01 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:18:01.170 DEBUG:boto:path=/pklxkd-46/ 2023-05-31T20:18:01.170 DEBUG:boto:auth_path=/pklxkd-46/ 2023-05-31T20:18:01.172 DEBUG:boto:Method: HEAD 2023-05-31T20:18:01.172 DEBUG:boto:Path: /pklxkd-46/ 2023-05-31T20:18:01.173 DEBUG:boto:Data: 2023-05-31T20:18:01.173 DEBUG:boto:Headers: {} 2023-05-31T20:18:01.173 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:18:01.174 DEBUG:boto:Port: 8000 2023-05-31T20:18:01.174 DEBUG:boto:Params: {} 2023-05-31T20:18:01.175 DEBUG:boto:Token: None 2023-05-31T20:18:01.175 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:18:01 GMT /pklxkd-46/ 2023-05-31T20:18:01.175 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:JaW/CZZcmItppsd8PB6hayhY7LM= 2023-05-31T20:18:01.176 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:18:01 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:JaW/CZZcmItppsd8PB6hayhY7LM=', 'Content-Length': '0'} 2023-05-31T20:18:01.179 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '8'), ('X-RGW-Bytes-Used', '48'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000055b424b5f4dd1d1d-006477ab79-1630-a1'), ('Content-Length', '0'), ('Date', 'Wed, 31 May 2023 20:18:01 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:18:01.179 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2023-05-31T20:18:01.180 DEBUG:boto:path=/pklxkd-46/ 2023-05-31T20:18:01.181 DEBUG:boto:auth_path=/pklxkd-46/ 2023-05-31T20:18:01.183 DEBUG:boto:path=/pklxkd-46/?versions&max-keys=999 2023-05-31T20:18:01.183 DEBUG:boto:auth_path=/pklxkd-46/?versions&max-keys=999 2023-05-31T20:18:01.184 DEBUG:boto:Method: GET 2023-05-31T20:18:01.185 DEBUG:boto:Path: /pklxkd-46/?versions&max-keys=999 2023-05-31T20:18:01.185 DEBUG:boto:Data: 2023-05-31T20:18:01.186 DEBUG:boto:Headers: {} 2023-05-31T20:18:01.187 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:18:01.187 DEBUG:boto:Port: 8001 2023-05-31T20:18:01.188 DEBUG:boto:Params: {} 2023-05-31T20:18:01.189 DEBUG:boto:Token: None 2023-05-31T20:18:01.190 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:18:01 GMT /pklxkd-46/?versions 2023-05-31T20:18:01.190 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:UnYCJzHw9urIuifQrKFrwX9BCTg= 2023-05-31T20:18:01.191 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:18:01 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:UnYCJzHw9urIuifQrKFrwX9BCTg=', 'Content-Length': '0'} 2023-05-31T20:18:01.194 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000007b22adbdd6f03978-006477ab79-162e-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:18:01 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:18:01.194 DEBUG:boto:b'pklxkd-46999falseobj1nulltrue2023-05-31T20:15:11.203Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj2nulltrue2023-05-31T20:15:11.225Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj3nulltrue2023-05-31T20:15:11.255Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj4nulltrue2023-05-31T20:15:11.317Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj5nulltrue2023-05-31T20:17:25.589Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj6nulltrue2023-05-31T20:17:25.610Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj7nulltrue2023-05-31T20:17:25.673Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj8nulltrue2023-05-31T20:17:25.695Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:18:01.195 DEBUG:tasks.rgw_multi.zone_rados:o=obj1 2023-05-31T20:18:01.196 DEBUG:tasks.rgw_multi.zone_rados:o=obj2 2023-05-31T20:18:01.196 DEBUG:tasks.rgw_multi.zone_rados:o=obj3 2023-05-31T20:18:01.196 DEBUG:tasks.rgw_multi.zone_rados:o=obj4 2023-05-31T20:18:01.197 DEBUG:tasks.rgw_multi.zone_rados:o=obj5 2023-05-31T20:18:01.197 DEBUG:tasks.rgw_multi.zone_rados:o=obj6 2023-05-31T20:18:01.197 DEBUG:tasks.rgw_multi.zone_rados:o=obj7 2023-05-31T20:18:01.198 DEBUG:tasks.rgw_multi.zone_rados:o=obj8 2023-05-31T20:18:01.198 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2023-05-31T20:18:01.198 DEBUG:boto:path=/pklxkd-46/ 2023-05-31T20:18:01.199 DEBUG:boto:auth_path=/pklxkd-46/ 2023-05-31T20:18:01.199 DEBUG:boto:path=/pklxkd-46/?versions&max-keys=999 2023-05-31T20:18:01.199 DEBUG:boto:auth_path=/pklxkd-46/?versions&max-keys=999 2023-05-31T20:18:01.200 DEBUG:boto:Method: GET 2023-05-31T20:18:01.200 DEBUG:boto:Path: /pklxkd-46/?versions&max-keys=999 2023-05-31T20:18:01.200 DEBUG:boto:Data: 2023-05-31T20:18:01.201 DEBUG:boto:Headers: {} 2023-05-31T20:18:01.201 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:18:01.201 DEBUG:boto:Port: 8000 2023-05-31T20:18:01.202 DEBUG:boto:Params: {} 2023-05-31T20:18:01.202 DEBUG:boto:Token: None 2023-05-31T20:18:01.203 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:18:01 GMT /pklxkd-46/?versions 2023-05-31T20:18:01.203 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:UnYCJzHw9urIuifQrKFrwX9BCTg= 2023-05-31T20:18:01.204 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:18:01 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:UnYCJzHw9urIuifQrKFrwX9BCTg=', 'Content-Length': '0'} 2023-05-31T20:18:01.206 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000001c1299c3220fdb75-006477ab79-1630-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:18:01 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:18:01.207 DEBUG:boto:b'pklxkd-46999falseobj1nulltrue2023-05-31T20:15:11.203Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj2nulltrue2023-05-31T20:15:11.225Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj3nulltrue2023-05-31T20:15:11.255Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj4nulltrue2023-05-31T20:15:11.317Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj5nulltrue2023-05-31T20:17:25.589Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj6nulltrue2023-05-31T20:17:25.610Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj7nulltrue2023-05-31T20:17:25.673Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj8nulltrue2023-05-31T20:17:25.695Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:18:01.208 DEBUG:tasks.rgw_multi.zone_rados:o=obj1 2023-05-31T20:18:01.208 DEBUG:tasks.rgw_multi.zone_rados:o=obj2 2023-05-31T20:18:01.208 DEBUG:tasks.rgw_multi.zone_rados:o=obj3 2023-05-31T20:18:01.209 DEBUG:tasks.rgw_multi.zone_rados:o=obj4 2023-05-31T20:18:01.209 DEBUG:tasks.rgw_multi.zone_rados:o=obj5 2023-05-31T20:18:01.209 DEBUG:tasks.rgw_multi.zone_rados:o=obj6 2023-05-31T20:18:01.210 DEBUG:tasks.rgw_multi.zone_rados:o=obj7 2023-05-31T20:18:01.210 DEBUG:tasks.rgw_multi.zone_rados:o=obj8 2023-05-31T20:18:01.210 DEBUG:boto:path=/pklxkd-46/ 2023-05-31T20:18:01.211 DEBUG:boto:auth_path=/pklxkd-46/ 2023-05-31T20:18:01.211 DEBUG:boto:path=/pklxkd-46/?versions&max-keys=999 2023-05-31T20:18:01.211 DEBUG:boto:auth_path=/pklxkd-46/?versions&max-keys=999 2023-05-31T20:18:01.212 DEBUG:boto:Method: GET 2023-05-31T20:18:01.212 DEBUG:boto:Path: /pklxkd-46/?versions&max-keys=999 2023-05-31T20:18:01.212 DEBUG:boto:Data: 2023-05-31T20:18:01.213 DEBUG:boto:Headers: {} 2023-05-31T20:18:01.213 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:18:01.213 DEBUG:boto:Port: 8001 2023-05-31T20:18:01.214 DEBUG:boto:Params: {} 2023-05-31T20:18:01.214 DEBUG:boto:Token: None 2023-05-31T20:18:01.215 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:18:01 GMT /pklxkd-46/?versions 2023-05-31T20:18:01.215 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:UnYCJzHw9urIuifQrKFrwX9BCTg= 2023-05-31T20:18:01.215 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:18:01 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:UnYCJzHw9urIuifQrKFrwX9BCTg=', 'Content-Length': '0'} 2023-05-31T20:18:01.218 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000045b5bfc027ad0f6-006477ab79-162e-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:18:01 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:18:01.218 DEBUG:boto:b'pklxkd-46999falseobj1nulltrue2023-05-31T20:15:11.203Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj2nulltrue2023-05-31T20:15:11.225Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj3nulltrue2023-05-31T20:15:11.255Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj4nulltrue2023-05-31T20:15:11.317Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj5nulltrue2023-05-31T20:17:25.589Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj6nulltrue2023-05-31T20:17:25.610Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj7nulltrue2023-05-31T20:17:25.673Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj8nulltrue2023-05-31T20:17:25.695Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:18:01.219 DEBUG:boto:path=/pklxkd-46/ 2023-05-31T20:18:01.220 DEBUG:boto:auth_path=/pklxkd-46/ 2023-05-31T20:18:01.220 DEBUG:boto:path=/pklxkd-46/?versions&max-keys=999 2023-05-31T20:18:01.220 DEBUG:boto:auth_path=/pklxkd-46/?versions&max-keys=999 2023-05-31T20:18:01.221 DEBUG:boto:Method: GET 2023-05-31T20:18:01.221 DEBUG:boto:Path: /pklxkd-46/?versions&max-keys=999 2023-05-31T20:18:01.221 DEBUG:boto:Data: 2023-05-31T20:18:01.222 DEBUG:boto:Headers: {} 2023-05-31T20:18:01.222 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:18:01.222 DEBUG:boto:Port: 8000 2023-05-31T20:18:01.223 DEBUG:boto:Params: {} 2023-05-31T20:18:01.223 DEBUG:boto:Token: None 2023-05-31T20:18:01.224 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:18:01 GMT /pklxkd-46/?versions 2023-05-31T20:18:01.224 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:UnYCJzHw9urIuifQrKFrwX9BCTg= 2023-05-31T20:18:01.224 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:18:01 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:UnYCJzHw9urIuifQrKFrwX9BCTg=', 'Content-Length': '0'} 2023-05-31T20:18:01.227 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000bfec68d873ffea22-006477ab79-1630-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:18:01 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:18:01.228 DEBUG:boto:b'pklxkd-46999falseobj1nulltrue2023-05-31T20:15:11.203Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj2nulltrue2023-05-31T20:15:11.225Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj3nulltrue2023-05-31T20:15:11.255Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj4nulltrue2023-05-31T20:15:11.317Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj5nulltrue2023-05-31T20:17:25.589Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj6nulltrue2023-05-31T20:17:25.610Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj7nulltrue2023-05-31T20:17:25.673Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj8nulltrue2023-05-31T20:17:25.695Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:18:01.229 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2023-05-31T20:18:01.229 DEBUG:boto:path=/pklxkd-46/obj1 2023-05-31T20:18:01.229 DEBUG:boto:auth_path=/pklxkd-46/obj1 2023-05-31T20:18:01.230 DEBUG:boto:path=/pklxkd-46/obj1?versionId=null 2023-05-31T20:18:01.230 DEBUG:boto:auth_path=/pklxkd-46/obj1?versionId=null 2023-05-31T20:18:01.230 DEBUG:boto:Method: GET 2023-05-31T20:18:01.231 DEBUG:boto:Path: /pklxkd-46/obj1?versionId=null 2023-05-31T20:18:01.231 DEBUG:boto:Data: 2023-05-31T20:18:01.232 DEBUG:boto:Headers: {} 2023-05-31T20:18:01.232 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:18:01.232 DEBUG:boto:Port: 8001 2023-05-31T20:18:01.233 DEBUG:boto:Params: {} 2023-05-31T20:18:01.233 DEBUG:boto:Token: None 2023-05-31T20:18:01.233 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:18:01 GMT /pklxkd-46/obj1?versionId=null 2023-05-31T20:18:01.234 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:uZwGB4RMI+nNkRtqVx1upqQP+B8= 2023-05-31T20:18:01.234 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:18:01 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:uZwGB4RMI+nNkRtqVx1upqQP+B8=', 'Content-Length': '0'} 2023-05-31T20:18:01.236 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:15:11 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000ea767e44e8e19194-006477ab79-162e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:18:01 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:18:01.237 DEBUG:boto:path=/pklxkd-46/obj1 2023-05-31T20:18:01.237 DEBUG:boto:auth_path=/pklxkd-46/obj1 2023-05-31T20:18:01.238 DEBUG:boto:path=/pklxkd-46/obj1?versionId=null 2023-05-31T20:18:01.238 DEBUG:boto:auth_path=/pklxkd-46/obj1?versionId=null 2023-05-31T20:18:01.238 DEBUG:boto:Method: GET 2023-05-31T20:18:01.239 DEBUG:boto:Path: /pklxkd-46/obj1?versionId=null 2023-05-31T20:18:01.239 DEBUG:boto:Data: 2023-05-31T20:18:01.239 DEBUG:boto:Headers: {} 2023-05-31T20:18:01.240 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:18:01.240 DEBUG:boto:Port: 8000 2023-05-31T20:18:01.240 DEBUG:boto:Params: {} 2023-05-31T20:18:01.241 DEBUG:boto:Token: None 2023-05-31T20:18:01.241 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:18:01 GMT /pklxkd-46/obj1?versionId=null 2023-05-31T20:18:01.242 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:uZwGB4RMI+nNkRtqVx1upqQP+B8= 2023-05-31T20:18:01.242 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:18:01 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:uZwGB4RMI+nNkRtqVx1upqQP+B8=', 'Content-Length': '0'} 2023-05-31T20:18:01.244 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:15:11 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000005c641ab1ee9c400-006477ab79-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:18:01 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:18:01.244 DEBUG:boto:path=/pklxkd-46/obj1 2023-05-31T20:18:01.245 DEBUG:boto:auth_path=/pklxkd-46/obj1 2023-05-31T20:18:01.245 DEBUG:boto:path=/pklxkd-46/obj1?versionId=null 2023-05-31T20:18:01.246 DEBUG:boto:auth_path=/pklxkd-46/obj1?versionId=null 2023-05-31T20:18:01.246 DEBUG:boto:Method: HEAD 2023-05-31T20:18:01.246 DEBUG:boto:Path: /pklxkd-46/obj1?versionId=null 2023-05-31T20:18:01.247 DEBUG:boto:Data: 2023-05-31T20:18:01.247 DEBUG:boto:Headers: {} 2023-05-31T20:18:01.247 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:18:01.248 DEBUG:boto:Port: 8001 2023-05-31T20:18:01.248 DEBUG:boto:Params: {} 2023-05-31T20:18:01.248 DEBUG:boto:Token: None 2023-05-31T20:18:01.249 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:18:01 GMT /pklxkd-46/obj1?versionId=null 2023-05-31T20:18:01.249 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:oTfOvNc4iy08MVjlbqPUYpFH5F0= 2023-05-31T20:18:01.249 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:18:01 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:oTfOvNc4iy08MVjlbqPUYpFH5F0=', 'Content-Length': '0'} 2023-05-31T20:18:01.251 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:15:11 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000004fe78b619542873d-006477ab79-162e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:18:01 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:18:01.252 DEBUG:boto:path=/pklxkd-46/obj1 2023-05-31T20:18:01.252 DEBUG:boto:auth_path=/pklxkd-46/obj1 2023-05-31T20:18:01.252 DEBUG:boto:path=/pklxkd-46/obj1?versionId=null 2023-05-31T20:18:01.253 DEBUG:boto:auth_path=/pklxkd-46/obj1?versionId=null 2023-05-31T20:18:01.253 DEBUG:boto:Method: HEAD 2023-05-31T20:18:01.254 DEBUG:boto:Path: /pklxkd-46/obj1?versionId=null 2023-05-31T20:18:01.254 DEBUG:boto:Data: 2023-05-31T20:18:01.254 DEBUG:boto:Headers: {} 2023-05-31T20:18:01.254 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:18:01.255 DEBUG:boto:Port: 8000 2023-05-31T20:18:01.255 DEBUG:boto:Params: {} 2023-05-31T20:18:01.256 DEBUG:boto:Token: None 2023-05-31T20:18:01.256 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:18:01 GMT /pklxkd-46/obj1?versionId=null 2023-05-31T20:18:01.256 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:oTfOvNc4iy08MVjlbqPUYpFH5F0= 2023-05-31T20:18:01.257 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:18:01 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:oTfOvNc4iy08MVjlbqPUYpFH5F0=', 'Content-Length': '0'} 2023-05-31T20:18:01.259 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:15:11 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000006e9fc1edba6c50dd-006477ab79-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:18:01 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:18:01.259 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2023-05-31T20:18:01.260 DEBUG:boto:path=/pklxkd-46/obj1 2023-05-31T20:18:01.260 DEBUG:boto:auth_path=/pklxkd-46/obj1 2023-05-31T20:18:01.260 DEBUG:boto:path=/pklxkd-46/obj1?versionId=null 2023-05-31T20:18:01.261 DEBUG:boto:auth_path=/pklxkd-46/obj1?versionId=null 2023-05-31T20:18:01.261 DEBUG:boto:Method: GET 2023-05-31T20:18:01.261 DEBUG:boto:Path: /pklxkd-46/obj1?versionId=null 2023-05-31T20:18:01.262 DEBUG:boto:Data: 2023-05-31T20:18:01.262 DEBUG:boto:Headers: {} 2023-05-31T20:18:01.262 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:18:01.263 DEBUG:boto:Port: 8001 2023-05-31T20:18:01.263 DEBUG:boto:Params: {} 2023-05-31T20:18:01.263 DEBUG:boto:Token: None 2023-05-31T20:18:01.264 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:18:01 GMT /pklxkd-46/obj1?versionId=null 2023-05-31T20:18:01.264 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:uZwGB4RMI+nNkRtqVx1upqQP+B8= 2023-05-31T20:18:01.265 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:18:01 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:uZwGB4RMI+nNkRtqVx1upqQP+B8=', 'Content-Length': '0'} 2023-05-31T20:18:01.267 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:15:11 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000ced83b97f1036698-006477ab79-162e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:18:01 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:18:01.308 DEBUG:boto:path=/pklxkd-46/obj1 2023-05-31T20:18:01.309 DEBUG:boto:auth_path=/pklxkd-46/obj1 2023-05-31T20:18:01.309 DEBUG:boto:path=/pklxkd-46/obj1?versionId=null 2023-05-31T20:18:01.310 DEBUG:boto:auth_path=/pklxkd-46/obj1?versionId=null 2023-05-31T20:18:01.310 DEBUG:boto:Method: GET 2023-05-31T20:18:01.310 DEBUG:boto:Path: /pklxkd-46/obj1?versionId=null 2023-05-31T20:18:01.311 DEBUG:boto:Data: 2023-05-31T20:18:01.311 DEBUG:boto:Headers: {} 2023-05-31T20:18:01.312 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:18:01.313 DEBUG:boto:Port: 8000 2023-05-31T20:18:01.313 DEBUG:boto:Params: {} 2023-05-31T20:18:01.314 DEBUG:boto:Token: None 2023-05-31T20:18:01.314 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:18:01 GMT /pklxkd-46/obj1?versionId=null 2023-05-31T20:18:01.315 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:uZwGB4RMI+nNkRtqVx1upqQP+B8= 2023-05-31T20:18:01.316 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:18:01 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:uZwGB4RMI+nNkRtqVx1upqQP+B8=', 'Content-Length': '0'} 2023-05-31T20:18:01.318 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:15:11 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000d34a819833dde8cd-006477ab79-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:18:01 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:18:01.319 DEBUG:boto:path=/pklxkd-46/ 2023-05-31T20:18:01.319 DEBUG:boto:auth_path=/pklxkd-46/ 2023-05-31T20:18:01.320 DEBUG:boto:path=/pklxkd-46/?prefix=obj1 2023-05-31T20:18:01.320 DEBUG:boto:auth_path=/pklxkd-46/?prefix=obj1 2023-05-31T20:18:01.320 DEBUG:boto:Method: GET 2023-05-31T20:18:01.321 DEBUG:boto:Path: /pklxkd-46/?prefix=obj1 2023-05-31T20:18:01.321 DEBUG:boto:Data: 2023-05-31T20:18:01.321 DEBUG:boto:Headers: {} 2023-05-31T20:18:01.322 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:18:01.322 DEBUG:boto:Port: 8001 2023-05-31T20:18:01.322 DEBUG:boto:Params: {} 2023-05-31T20:18:01.323 DEBUG:boto:Token: None 2023-05-31T20:18:01.323 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:18:01 GMT /pklxkd-46/ 2023-05-31T20:18:01.324 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:atczRiAmTsKEk0V6aTVUEBRPCHs= 2023-05-31T20:18:01.324 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:18:01 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:atczRiAmTsKEk0V6aTVUEBRPCHs=', 'Content-Length': '0'} 2023-05-31T20:18:01.326 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000000c62143544b144ab-006477ab79-162e-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:18:01 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:18:01.327 DEBUG:boto:b'pklxkd-46obj11000falseobj12023-05-31T20:15:11.203Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:18:01.328 DEBUG:boto:path=/pklxkd-46/ 2023-05-31T20:18:01.328 DEBUG:boto:auth_path=/pklxkd-46/ 2023-05-31T20:18:01.329 DEBUG:boto:path=/pklxkd-46/?prefix=obj1 2023-05-31T20:18:01.329 DEBUG:boto:auth_path=/pklxkd-46/?prefix=obj1 2023-05-31T20:18:01.330 DEBUG:boto:Method: GET 2023-05-31T20:18:01.330 DEBUG:boto:Path: /pklxkd-46/?prefix=obj1 2023-05-31T20:18:01.330 DEBUG:boto:Data: 2023-05-31T20:18:01.331 DEBUG:boto:Headers: {} 2023-05-31T20:18:01.331 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:18:01.332 DEBUG:boto:Port: 8000 2023-05-31T20:18:01.333 DEBUG:boto:Params: {} 2023-05-31T20:18:01.333 DEBUG:boto:Token: None 2023-05-31T20:18:01.334 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:18:01 GMT /pklxkd-46/ 2023-05-31T20:18:01.334 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:atczRiAmTsKEk0V6aTVUEBRPCHs= 2023-05-31T20:18:01.334 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:18:01 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:atczRiAmTsKEk0V6aTVUEBRPCHs=', 'Content-Length': '0'} 2023-05-31T20:18:01.337 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000004e8812bdd4edfc12-006477ab79-1630-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:18:01 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:18:01.338 DEBUG:boto:b'pklxkd-46obj11000falseobj12023-05-31T20:15:11.203Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:18:01.339 DEBUG:boto:path=/pklxkd-46/obj1 2023-05-31T20:18:01.339 DEBUG:boto:auth_path=/pklxkd-46/obj1 2023-05-31T20:18:01.339 DEBUG:boto:Method: HEAD 2023-05-31T20:18:01.340 DEBUG:boto:Path: /pklxkd-46/obj1 2023-05-31T20:18:01.340 DEBUG:boto:Data: 2023-05-31T20:18:01.341 DEBUG:boto:Headers: {} 2023-05-31T20:18:01.341 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:18:01.341 DEBUG:boto:Port: 8001 2023-05-31T20:18:01.341 DEBUG:boto:Params: {} 2023-05-31T20:18:01.342 DEBUG:boto:Token: None 2023-05-31T20:18:01.342 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:18:01 GMT /pklxkd-46/obj1 2023-05-31T20:18:01.343 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:yXwaNnKIg7MsjapoMRvSYRLuaII= 2023-05-31T20:18:01.343 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:18:01 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:yXwaNnKIg7MsjapoMRvSYRLuaII=', 'Content-Length': '0'} 2023-05-31T20:18:01.345 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:15:11 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000008be98c8ff472acae-006477ab79-162e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:18:01 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:18:01.345 DEBUG:boto:path=/pklxkd-46/obj1 2023-05-31T20:18:01.346 DEBUG:boto:auth_path=/pklxkd-46/obj1 2023-05-31T20:18:01.346 DEBUG:boto:Method: HEAD 2023-05-31T20:18:01.347 DEBUG:boto:Path: /pklxkd-46/obj1 2023-05-31T20:18:01.347 DEBUG:boto:Data: 2023-05-31T20:18:01.348 DEBUG:boto:Headers: {} 2023-05-31T20:18:01.349 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:18:01.349 DEBUG:boto:Port: 8000 2023-05-31T20:18:01.350 DEBUG:boto:Params: {} 2023-05-31T20:18:01.350 DEBUG:boto:Token: None 2023-05-31T20:18:01.350 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:18:01 GMT /pklxkd-46/obj1 2023-05-31T20:18:01.351 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:yXwaNnKIg7MsjapoMRvSYRLuaII= 2023-05-31T20:18:01.351 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:18:01 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:yXwaNnKIg7MsjapoMRvSYRLuaII=', 'Content-Length': '0'} 2023-05-31T20:18:01.353 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:15:11 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000b7fe9f79fb38512f-006477ab79-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:18:01 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:18:01.354 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2023-05-31T20:18:01.354 DEBUG:boto:path=/pklxkd-46/obj1 2023-05-31T20:18:01.354 DEBUG:boto:auth_path=/pklxkd-46/obj1 2023-05-31T20:18:01.355 DEBUG:boto:Method: GET 2023-05-31T20:18:01.355 DEBUG:boto:Path: /pklxkd-46/obj1 2023-05-31T20:18:01.356 DEBUG:boto:Data: 2023-05-31T20:18:01.356 DEBUG:boto:Headers: {} 2023-05-31T20:18:01.357 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:18:01.357 DEBUG:boto:Port: 8001 2023-05-31T20:18:01.357 DEBUG:boto:Params: {} 2023-05-31T20:18:01.358 DEBUG:boto:Token: None 2023-05-31T20:18:01.358 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:18:01 GMT /pklxkd-46/obj1 2023-05-31T20:18:01.359 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:EyNhgJ+HVbSmgPDyj4hxEx3UVUs= 2023-05-31T20:18:01.360 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:18:01 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:EyNhgJ+HVbSmgPDyj4hxEx3UVUs=', 'Content-Length': '0'} 2023-05-31T20:18:01.362 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:15:11 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000013b70f7e5fd88bf0-006477ab79-162e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:18:01 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:18:01.362 DEBUG:boto:path=/pklxkd-46/obj1 2023-05-31T20:18:01.363 DEBUG:boto:auth_path=/pklxkd-46/obj1 2023-05-31T20:18:01.363 DEBUG:boto:Method: GET 2023-05-31T20:18:01.364 DEBUG:boto:Path: /pklxkd-46/obj1 2023-05-31T20:18:01.364 DEBUG:boto:Data: 2023-05-31T20:18:01.364 DEBUG:boto:Headers: {} 2023-05-31T20:18:01.365 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:18:01.365 DEBUG:boto:Port: 8000 2023-05-31T20:18:01.366 DEBUG:boto:Params: {} 2023-05-31T20:18:01.366 DEBUG:boto:Token: None 2023-05-31T20:18:01.367 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:18:01 GMT /pklxkd-46/obj1 2023-05-31T20:18:01.367 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:EyNhgJ+HVbSmgPDyj4hxEx3UVUs= 2023-05-31T20:18:01.368 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:18:01 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:EyNhgJ+HVbSmgPDyj4hxEx3UVUs=', 'Content-Length': '0'} 2023-05-31T20:18:01.371 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:15:11 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000034b29e8aa2c36ce2-006477ab79-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:18:01 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:18:01.372 DEBUG:boto:path=/pklxkd-46/ 2023-05-31T20:18:01.373 DEBUG:boto:auth_path=/pklxkd-46/ 2023-05-31T20:18:01.373 DEBUG:boto:path=/pklxkd-46/?prefix=obj1 2023-05-31T20:18:01.374 DEBUG:boto:auth_path=/pklxkd-46/?prefix=obj1 2023-05-31T20:18:01.375 DEBUG:boto:Method: GET 2023-05-31T20:18:01.376 DEBUG:boto:Path: /pklxkd-46/?prefix=obj1 2023-05-31T20:18:01.376 DEBUG:boto:Data: 2023-05-31T20:18:01.377 DEBUG:boto:Headers: {} 2023-05-31T20:18:01.377 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:18:01.378 DEBUG:boto:Port: 8001 2023-05-31T20:18:01.378 DEBUG:boto:Params: {} 2023-05-31T20:18:01.379 DEBUG:boto:Token: None 2023-05-31T20:18:01.379 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:18:01 GMT /pklxkd-46/ 2023-05-31T20:18:01.380 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:atczRiAmTsKEk0V6aTVUEBRPCHs= 2023-05-31T20:18:01.381 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:18:01 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:atczRiAmTsKEk0V6aTVUEBRPCHs=', 'Content-Length': '0'} 2023-05-31T20:18:01.383 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000089bddca678cd5475-006477ab79-162e-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:18:01 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:18:01.384 DEBUG:boto:b'pklxkd-46obj11000falseobj12023-05-31T20:15:11.203Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:18:01.386 DEBUG:boto:path=/pklxkd-46/ 2023-05-31T20:18:01.387 DEBUG:boto:auth_path=/pklxkd-46/ 2023-05-31T20:18:01.388 DEBUG:boto:path=/pklxkd-46/?prefix=obj1 2023-05-31T20:18:01.389 DEBUG:boto:auth_path=/pklxkd-46/?prefix=obj1 2023-05-31T20:18:01.390 DEBUG:boto:Method: GET 2023-05-31T20:18:01.390 DEBUG:boto:Path: /pklxkd-46/?prefix=obj1 2023-05-31T20:18:01.391 DEBUG:boto:Data: 2023-05-31T20:18:01.392 DEBUG:boto:Headers: {} 2023-05-31T20:18:01.393 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:18:01.393 DEBUG:boto:Port: 8000 2023-05-31T20:18:01.394 DEBUG:boto:Params: {} 2023-05-31T20:18:01.395 DEBUG:boto:Token: None 2023-05-31T20:18:01.396 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:18:01 GMT /pklxkd-46/ 2023-05-31T20:18:01.397 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:atczRiAmTsKEk0V6aTVUEBRPCHs= 2023-05-31T20:18:01.397 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:18:01 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:atczRiAmTsKEk0V6aTVUEBRPCHs=', 'Content-Length': '0'} 2023-05-31T20:18:01.399 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000080532283a2b393ae-006477ab79-1630-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:18:01 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:18:01.400 DEBUG:boto:b'pklxkd-46obj11000falseobj12023-05-31T20:15:11.203Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:18:01.401 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2023-05-31T20:18:01.401 DEBUG:boto:path=/pklxkd-46/obj2 2023-05-31T20:18:01.402 DEBUG:boto:auth_path=/pklxkd-46/obj2 2023-05-31T20:18:01.402 DEBUG:boto:path=/pklxkd-46/obj2?versionId=null 2023-05-31T20:18:01.403 DEBUG:boto:auth_path=/pklxkd-46/obj2?versionId=null 2023-05-31T20:18:01.403 DEBUG:boto:Method: GET 2023-05-31T20:18:01.404 DEBUG:boto:Path: /pklxkd-46/obj2?versionId=null 2023-05-31T20:18:01.404 DEBUG:boto:Data: 2023-05-31T20:18:01.404 DEBUG:boto:Headers: {} 2023-05-31T20:18:01.405 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:18:01.405 DEBUG:boto:Port: 8001 2023-05-31T20:18:01.406 DEBUG:boto:Params: {} 2023-05-31T20:18:01.406 DEBUG:boto:Token: None 2023-05-31T20:18:01.407 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:18:01 GMT /pklxkd-46/obj2?versionId=null 2023-05-31T20:18:01.407 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:2EepAFDuOdGW+mA8Ms2GKrCL7+E= 2023-05-31T20:18:01.408 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:18:01 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:2EepAFDuOdGW+mA8Ms2GKrCL7+E=', 'Content-Length': '0'} 2023-05-31T20:18:01.410 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:15:11 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000aa223cf6331a3ccf-006477ab79-162e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:18:01 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:18:01.410 DEBUG:boto:path=/pklxkd-46/obj2 2023-05-31T20:18:01.411 DEBUG:boto:auth_path=/pklxkd-46/obj2 2023-05-31T20:18:01.411 DEBUG:boto:path=/pklxkd-46/obj2?versionId=null 2023-05-31T20:18:01.412 DEBUG:boto:auth_path=/pklxkd-46/obj2?versionId=null 2023-05-31T20:18:01.412 DEBUG:boto:Method: GET 2023-05-31T20:18:01.413 DEBUG:boto:Path: /pklxkd-46/obj2?versionId=null 2023-05-31T20:18:01.413 DEBUG:boto:Data: 2023-05-31T20:18:01.414 DEBUG:boto:Headers: {} 2023-05-31T20:18:01.414 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:18:01.415 DEBUG:boto:Port: 8000 2023-05-31T20:18:01.415 DEBUG:boto:Params: {} 2023-05-31T20:18:01.415 DEBUG:boto:Token: None 2023-05-31T20:18:01.416 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:18:01 GMT /pklxkd-46/obj2?versionId=null 2023-05-31T20:18:01.417 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:2EepAFDuOdGW+mA8Ms2GKrCL7+E= 2023-05-31T20:18:01.418 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:18:01 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:2EepAFDuOdGW+mA8Ms2GKrCL7+E=', 'Content-Length': '0'} 2023-05-31T20:18:01.420 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:15:11 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000793325ce35235dc1-006477ab79-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:18:01 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:18:01.421 DEBUG:boto:path=/pklxkd-46/obj2 2023-05-31T20:18:01.422 DEBUG:boto:auth_path=/pklxkd-46/obj2 2023-05-31T20:18:01.423 DEBUG:boto:path=/pklxkd-46/obj2?versionId=null 2023-05-31T20:18:01.423 DEBUG:boto:auth_path=/pklxkd-46/obj2?versionId=null 2023-05-31T20:18:01.424 DEBUG:boto:Method: HEAD 2023-05-31T20:18:01.425 DEBUG:boto:Path: /pklxkd-46/obj2?versionId=null 2023-05-31T20:18:01.425 DEBUG:boto:Data: 2023-05-31T20:18:01.426 DEBUG:boto:Headers: {} 2023-05-31T20:18:01.427 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:18:01.428 DEBUG:boto:Port: 8001 2023-05-31T20:18:01.429 DEBUG:boto:Params: {} 2023-05-31T20:18:01.429 DEBUG:boto:Token: None 2023-05-31T20:18:01.430 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:18:01 GMT /pklxkd-46/obj2?versionId=null 2023-05-31T20:18:01.431 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:ON6+94VkL908Kxf1t48AEOqlefc= 2023-05-31T20:18:01.432 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:18:01 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:ON6+94VkL908Kxf1t48AEOqlefc=', 'Content-Length': '0'} 2023-05-31T20:18:01.434 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:15:11 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000000495b04cec7c61f6-006477ab79-162e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:18:01 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:18:01.435 DEBUG:boto:path=/pklxkd-46/obj2 2023-05-31T20:18:01.436 DEBUG:boto:auth_path=/pklxkd-46/obj2 2023-05-31T20:18:01.437 DEBUG:boto:path=/pklxkd-46/obj2?versionId=null 2023-05-31T20:18:01.437 DEBUG:boto:auth_path=/pklxkd-46/obj2?versionId=null 2023-05-31T20:18:01.438 DEBUG:boto:Method: HEAD 2023-05-31T20:18:01.438 DEBUG:boto:Path: /pklxkd-46/obj2?versionId=null 2023-05-31T20:18:01.438 DEBUG:boto:Data: 2023-05-31T20:18:01.439 DEBUG:boto:Headers: {} 2023-05-31T20:18:01.439 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:18:01.439 DEBUG:boto:Port: 8000 2023-05-31T20:18:01.440 DEBUG:boto:Params: {} 2023-05-31T20:18:01.440 DEBUG:boto:Token: None 2023-05-31T20:18:01.441 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:18:01 GMT /pklxkd-46/obj2?versionId=null 2023-05-31T20:18:01.441 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:ON6+94VkL908Kxf1t48AEOqlefc= 2023-05-31T20:18:01.442 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:18:01 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:ON6+94VkL908Kxf1t48AEOqlefc=', 'Content-Length': '0'} 2023-05-31T20:18:01.443 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:15:11 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000631a9c38c3a3bfba-006477ab79-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:18:01 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:18:01.444 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2023-05-31T20:18:01.444 DEBUG:boto:path=/pklxkd-46/obj2 2023-05-31T20:18:01.445 DEBUG:boto:auth_path=/pklxkd-46/obj2 2023-05-31T20:18:01.445 DEBUG:boto:path=/pklxkd-46/obj2?versionId=null 2023-05-31T20:18:01.445 DEBUG:boto:auth_path=/pklxkd-46/obj2?versionId=null 2023-05-31T20:18:01.446 DEBUG:boto:Method: GET 2023-05-31T20:18:01.446 DEBUG:boto:Path: /pklxkd-46/obj2?versionId=null 2023-05-31T20:18:01.447 DEBUG:boto:Data: 2023-05-31T20:18:01.447 DEBUG:boto:Headers: {} 2023-05-31T20:18:01.447 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:18:01.448 DEBUG:boto:Port: 8001 2023-05-31T20:18:01.448 DEBUG:boto:Params: {} 2023-05-31T20:18:01.449 DEBUG:boto:Token: None 2023-05-31T20:18:01.449 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:18:01 GMT /pklxkd-46/obj2?versionId=null 2023-05-31T20:18:01.449 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:2EepAFDuOdGW+mA8Ms2GKrCL7+E= 2023-05-31T20:18:01.450 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:18:01 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:2EepAFDuOdGW+mA8Ms2GKrCL7+E=', 'Content-Length': '0'} 2023-05-31T20:18:01.452 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:15:11 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000a8642698ecb62d24-006477ab79-162e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:18:01 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:18:01.452 DEBUG:boto:path=/pklxkd-46/obj2 2023-05-31T20:18:01.454 DEBUG:boto:auth_path=/pklxkd-46/obj2 2023-05-31T20:18:01.455 DEBUG:boto:path=/pklxkd-46/obj2?versionId=null 2023-05-31T20:18:01.455 DEBUG:boto:auth_path=/pklxkd-46/obj2?versionId=null 2023-05-31T20:18:01.455 DEBUG:boto:Method: GET 2023-05-31T20:18:01.456 DEBUG:boto:Path: /pklxkd-46/obj2?versionId=null 2023-05-31T20:18:01.456 DEBUG:boto:Data: 2023-05-31T20:18:01.457 DEBUG:boto:Headers: {} 2023-05-31T20:18:01.457 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:18:01.457 DEBUG:boto:Port: 8000 2023-05-31T20:18:01.458 DEBUG:boto:Params: {} 2023-05-31T20:18:01.458 DEBUG:boto:Token: None 2023-05-31T20:18:01.459 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:18:01 GMT /pklxkd-46/obj2?versionId=null 2023-05-31T20:18:01.459 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:2EepAFDuOdGW+mA8Ms2GKrCL7+E= 2023-05-31T20:18:01.459 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:18:01 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:2EepAFDuOdGW+mA8Ms2GKrCL7+E=', 'Content-Length': '0'} 2023-05-31T20:18:01.461 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:15:11 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000b6304cb261384b2a-006477ab79-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:18:01 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:18:01.462 DEBUG:boto:path=/pklxkd-46/ 2023-05-31T20:18:01.463 DEBUG:boto:auth_path=/pklxkd-46/ 2023-05-31T20:18:01.463 DEBUG:boto:path=/pklxkd-46/?prefix=obj2 2023-05-31T20:18:01.463 DEBUG:boto:auth_path=/pklxkd-46/?prefix=obj2 2023-05-31T20:18:01.464 DEBUG:boto:Method: GET 2023-05-31T20:18:01.464 DEBUG:boto:Path: /pklxkd-46/?prefix=obj2 2023-05-31T20:18:01.464 DEBUG:boto:Data: 2023-05-31T20:18:01.465 DEBUG:boto:Headers: {} 2023-05-31T20:18:01.465 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:18:01.465 DEBUG:boto:Port: 8001 2023-05-31T20:18:01.466 DEBUG:boto:Params: {} 2023-05-31T20:18:01.466 DEBUG:boto:Token: None 2023-05-31T20:18:01.467 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:18:01 GMT /pklxkd-46/ 2023-05-31T20:18:01.467 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:atczRiAmTsKEk0V6aTVUEBRPCHs= 2023-05-31T20:18:01.468 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:18:01 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:atczRiAmTsKEk0V6aTVUEBRPCHs=', 'Content-Length': '0'} 2023-05-31T20:18:01.470 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000006bd1c8df847025f1-006477ab79-162e-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:18:01 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:18:01.471 DEBUG:boto:b'pklxkd-46obj21000falseobj22023-05-31T20:15:11.225Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:18:01.471 DEBUG:boto:path=/pklxkd-46/ 2023-05-31T20:18:01.472 DEBUG:boto:auth_path=/pklxkd-46/ 2023-05-31T20:18:01.472 DEBUG:boto:path=/pklxkd-46/?prefix=obj2 2023-05-31T20:18:01.473 DEBUG:boto:auth_path=/pklxkd-46/?prefix=obj2 2023-05-31T20:18:01.473 DEBUG:boto:Method: GET 2023-05-31T20:18:01.473 DEBUG:boto:Path: /pklxkd-46/?prefix=obj2 2023-05-31T20:18:01.474 DEBUG:boto:Data: 2023-05-31T20:18:01.474 DEBUG:boto:Headers: {} 2023-05-31T20:18:01.474 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:18:01.475 DEBUG:boto:Port: 8000 2023-05-31T20:18:01.475 DEBUG:boto:Params: {} 2023-05-31T20:18:01.475 DEBUG:boto:Token: None 2023-05-31T20:18:01.476 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:18:01 GMT /pklxkd-46/ 2023-05-31T20:18:01.476 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:atczRiAmTsKEk0V6aTVUEBRPCHs= 2023-05-31T20:18:01.477 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:18:01 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:atczRiAmTsKEk0V6aTVUEBRPCHs=', 'Content-Length': '0'} 2023-05-31T20:18:01.479 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ae9dc325b47959a3-006477ab79-1630-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:18:01 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:18:01.480 DEBUG:boto:b'pklxkd-46obj21000falseobj22023-05-31T20:15:11.225Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:18:01.480 DEBUG:boto:path=/pklxkd-46/obj2 2023-05-31T20:18:01.481 DEBUG:boto:auth_path=/pklxkd-46/obj2 2023-05-31T20:18:01.481 DEBUG:boto:Method: HEAD 2023-05-31T20:18:01.482 DEBUG:boto:Path: /pklxkd-46/obj2 2023-05-31T20:18:01.482 DEBUG:boto:Data: 2023-05-31T20:18:01.482 DEBUG:boto:Headers: {} 2023-05-31T20:18:01.483 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:18:01.483 DEBUG:boto:Port: 8001 2023-05-31T20:18:01.483 DEBUG:boto:Params: {} 2023-05-31T20:18:01.484 DEBUG:boto:Token: None 2023-05-31T20:18:01.484 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:18:01 GMT /pklxkd-46/obj2 2023-05-31T20:18:01.485 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:oqGWU+y8XQ6KPUbkx1TplLofKsk= 2023-05-31T20:18:01.485 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:18:01 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:oqGWU+y8XQ6KPUbkx1TplLofKsk=', 'Content-Length': '0'} 2023-05-31T20:18:01.487 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:15:11 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000bb21eda6f46df469-006477ab79-162e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:18:01 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:18:01.487 DEBUG:boto:path=/pklxkd-46/obj2 2023-05-31T20:18:01.488 DEBUG:boto:auth_path=/pklxkd-46/obj2 2023-05-31T20:18:01.488 DEBUG:boto:Method: HEAD 2023-05-31T20:18:01.488 DEBUG:boto:Path: /pklxkd-46/obj2 2023-05-31T20:18:01.489 DEBUG:boto:Data: 2023-05-31T20:18:01.489 DEBUG:boto:Headers: {} 2023-05-31T20:18:01.489 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:18:01.490 DEBUG:boto:Port: 8000 2023-05-31T20:18:01.490 DEBUG:boto:Params: {} 2023-05-31T20:18:01.490 DEBUG:boto:Token: None 2023-05-31T20:18:01.491 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:18:01 GMT /pklxkd-46/obj2 2023-05-31T20:18:01.491 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:oqGWU+y8XQ6KPUbkx1TplLofKsk= 2023-05-31T20:18:01.492 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:18:01 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:oqGWU+y8XQ6KPUbkx1TplLofKsk=', 'Content-Length': '0'} 2023-05-31T20:18:01.493 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:15:11 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000050ef02cbb3864aef-006477ab79-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:18:01 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:18:01.494 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2023-05-31T20:18:01.494 DEBUG:boto:path=/pklxkd-46/obj2 2023-05-31T20:18:01.495 DEBUG:boto:auth_path=/pklxkd-46/obj2 2023-05-31T20:18:01.495 DEBUG:boto:Method: GET 2023-05-31T20:18:01.495 DEBUG:boto:Path: /pklxkd-46/obj2 2023-05-31T20:18:01.496 DEBUG:boto:Data: 2023-05-31T20:18:01.496 DEBUG:boto:Headers: {} 2023-05-31T20:18:01.496 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:18:01.497 DEBUG:boto:Port: 8001 2023-05-31T20:18:01.497 DEBUG:boto:Params: {} 2023-05-31T20:18:01.498 DEBUG:boto:Token: None 2023-05-31T20:18:01.498 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:18:01 GMT /pklxkd-46/obj2 2023-05-31T20:18:01.498 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:TlClcA6pfMB1cpxOPCR77ERxGQw= 2023-05-31T20:18:01.499 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:18:01 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:TlClcA6pfMB1cpxOPCR77ERxGQw=', 'Content-Length': '0'} 2023-05-31T20:18:01.501 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:15:11 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000081a257c48bd93cd0-006477ab79-162e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:18:01 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:18:01.502 DEBUG:boto:path=/pklxkd-46/obj2 2023-05-31T20:18:01.503 DEBUG:boto:auth_path=/pklxkd-46/obj2 2023-05-31T20:18:01.503 DEBUG:boto:Method: GET 2023-05-31T20:18:01.504 DEBUG:boto:Path: /pklxkd-46/obj2 2023-05-31T20:18:01.504 DEBUG:boto:Data: 2023-05-31T20:18:01.504 DEBUG:boto:Headers: {} 2023-05-31T20:18:01.505 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:18:01.505 DEBUG:boto:Port: 8000 2023-05-31T20:18:01.505 DEBUG:boto:Params: {} 2023-05-31T20:18:01.506 DEBUG:boto:Token: None 2023-05-31T20:18:01.506 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:18:01 GMT /pklxkd-46/obj2 2023-05-31T20:18:01.507 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:TlClcA6pfMB1cpxOPCR77ERxGQw= 2023-05-31T20:18:01.507 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:18:01 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:TlClcA6pfMB1cpxOPCR77ERxGQw=', 'Content-Length': '0'} 2023-05-31T20:18:01.509 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:15:11 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000086804ac18758f5f0-006477ab79-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:18:01 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:18:01.510 DEBUG:boto:path=/pklxkd-46/ 2023-05-31T20:18:01.510 DEBUG:boto:auth_path=/pklxkd-46/ 2023-05-31T20:18:01.510 DEBUG:boto:path=/pklxkd-46/?prefix=obj2 2023-05-31T20:18:01.511 DEBUG:boto:auth_path=/pklxkd-46/?prefix=obj2 2023-05-31T20:18:01.511 DEBUG:boto:Method: GET 2023-05-31T20:18:01.512 DEBUG:boto:Path: /pklxkd-46/?prefix=obj2 2023-05-31T20:18:01.512 DEBUG:boto:Data: 2023-05-31T20:18:01.512 DEBUG:boto:Headers: {} 2023-05-31T20:18:01.513 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:18:01.513 DEBUG:boto:Port: 8001 2023-05-31T20:18:01.513 DEBUG:boto:Params: {} 2023-05-31T20:18:01.514 DEBUG:boto:Token: None 2023-05-31T20:18:01.514 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:18:01 GMT /pklxkd-46/ 2023-05-31T20:18:01.515 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:atczRiAmTsKEk0V6aTVUEBRPCHs= 2023-05-31T20:18:01.515 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:18:01 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:atczRiAmTsKEk0V6aTVUEBRPCHs=', 'Content-Length': '0'} 2023-05-31T20:18:01.517 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000006ed6975cd97e11d1-006477ab79-162e-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:18:01 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:18:01.562 DEBUG:boto:b'pklxkd-46obj21000falseobj22023-05-31T20:15:11.225Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:18:01.564 DEBUG:boto:path=/pklxkd-46/ 2023-05-31T20:18:01.565 DEBUG:boto:auth_path=/pklxkd-46/ 2023-05-31T20:18:01.566 DEBUG:boto:path=/pklxkd-46/?prefix=obj2 2023-05-31T20:18:01.567 DEBUG:boto:auth_path=/pklxkd-46/?prefix=obj2 2023-05-31T20:18:01.568 DEBUG:boto:Method: GET 2023-05-31T20:18:01.568 DEBUG:boto:Path: /pklxkd-46/?prefix=obj2 2023-05-31T20:18:01.569 DEBUG:boto:Data: 2023-05-31T20:18:01.570 DEBUG:boto:Headers: {} 2023-05-31T20:18:01.570 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:18:01.571 DEBUG:boto:Port: 8000 2023-05-31T20:18:01.571 DEBUG:boto:Params: {} 2023-05-31T20:18:01.573 DEBUG:boto:Token: None 2023-05-31T20:18:01.573 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:18:01 GMT /pklxkd-46/ 2023-05-31T20:18:01.574 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:atczRiAmTsKEk0V6aTVUEBRPCHs= 2023-05-31T20:18:01.575 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:18:01 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:atczRiAmTsKEk0V6aTVUEBRPCHs=', 'Content-Length': '0'} 2023-05-31T20:18:01.577 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000024be2d6ff03db27-006477ab79-1630-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:18:01 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:18:01.578 DEBUG:boto:b'pklxkd-46obj21000falseobj22023-05-31T20:15:11.225Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:18:01.579 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2023-05-31T20:18:01.579 DEBUG:boto:path=/pklxkd-46/obj3 2023-05-31T20:18:01.580 DEBUG:boto:auth_path=/pklxkd-46/obj3 2023-05-31T20:18:01.580 DEBUG:boto:path=/pklxkd-46/obj3?versionId=null 2023-05-31T20:18:01.581 DEBUG:boto:auth_path=/pklxkd-46/obj3?versionId=null 2023-05-31T20:18:01.582 DEBUG:boto:Method: GET 2023-05-31T20:18:01.582 DEBUG:boto:Path: /pklxkd-46/obj3?versionId=null 2023-05-31T20:18:01.583 DEBUG:boto:Data: 2023-05-31T20:18:01.584 DEBUG:boto:Headers: {} 2023-05-31T20:18:01.585 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:18:01.585 DEBUG:boto:Port: 8001 2023-05-31T20:18:01.585 DEBUG:boto:Params: {} 2023-05-31T20:18:01.586 DEBUG:boto:Token: None 2023-05-31T20:18:01.587 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:18:01 GMT /pklxkd-46/obj3?versionId=null 2023-05-31T20:18:01.588 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:0D2egjc2rgQR/RtLbKvJKuOrOlU= 2023-05-31T20:18:01.589 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:18:01 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:0D2egjc2rgQR/RtLbKvJKuOrOlU=', 'Content-Length': '0'} 2023-05-31T20:18:01.591 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:15:11 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000fa975ad2e33e0d2b-006477ab79-162e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:18:01 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:18:01.593 DEBUG:boto:path=/pklxkd-46/obj3 2023-05-31T20:18:01.594 DEBUG:boto:auth_path=/pklxkd-46/obj3 2023-05-31T20:18:01.595 DEBUG:boto:path=/pklxkd-46/obj3?versionId=null 2023-05-31T20:18:01.596 DEBUG:boto:auth_path=/pklxkd-46/obj3?versionId=null 2023-05-31T20:18:01.596 DEBUG:boto:Method: GET 2023-05-31T20:18:01.597 DEBUG:boto:Path: /pklxkd-46/obj3?versionId=null 2023-05-31T20:18:01.598 DEBUG:boto:Data: 2023-05-31T20:18:01.600 DEBUG:boto:Headers: {} 2023-05-31T20:18:01.600 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:18:01.601 DEBUG:boto:Port: 8000 2023-05-31T20:18:01.601 DEBUG:boto:Params: {} 2023-05-31T20:18:01.602 DEBUG:boto:Token: None 2023-05-31T20:18:01.602 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:18:01 GMT /pklxkd-46/obj3?versionId=null 2023-05-31T20:18:01.604 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:0D2egjc2rgQR/RtLbKvJKuOrOlU= 2023-05-31T20:18:01.604 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:18:01 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:0D2egjc2rgQR/RtLbKvJKuOrOlU=', 'Content-Length': '0'} 2023-05-31T20:18:01.606 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:15:11 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000009169d104ae8e197f-006477ab79-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:18:01 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:18:01.652 DEBUG:boto:path=/pklxkd-46/obj3 2023-05-31T20:18:01.653 DEBUG:boto:auth_path=/pklxkd-46/obj3 2023-05-31T20:18:01.653 DEBUG:boto:path=/pklxkd-46/obj3?versionId=null 2023-05-31T20:18:01.654 DEBUG:boto:auth_path=/pklxkd-46/obj3?versionId=null 2023-05-31T20:18:01.654 DEBUG:boto:Method: HEAD 2023-05-31T20:18:01.655 DEBUG:boto:Path: /pklxkd-46/obj3?versionId=null 2023-05-31T20:18:01.655 DEBUG:boto:Data: 2023-05-31T20:18:01.656 DEBUG:boto:Headers: {} 2023-05-31T20:18:01.656 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:18:01.658 DEBUG:boto:Port: 8001 2023-05-31T20:18:01.659 DEBUG:boto:Params: {} 2023-05-31T20:18:01.659 DEBUG:boto:Token: None 2023-05-31T20:18:01.660 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:18:01 GMT /pklxkd-46/obj3?versionId=null 2023-05-31T20:18:01.661 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:/USNfSLiRUZg6k/FVTeHhktQT64= 2023-05-31T20:18:01.662 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:18:01 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:/USNfSLiRUZg6k/FVTeHhktQT64=', 'Content-Length': '0'} 2023-05-31T20:18:01.664 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:15:11 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000004b0fc63dc50219dd-006477ab79-162e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:18:01 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:18:01.665 DEBUG:boto:path=/pklxkd-46/obj3 2023-05-31T20:18:01.666 DEBUG:boto:auth_path=/pklxkd-46/obj3 2023-05-31T20:18:01.667 DEBUG:boto:path=/pklxkd-46/obj3?versionId=null 2023-05-31T20:18:01.667 DEBUG:boto:auth_path=/pklxkd-46/obj3?versionId=null 2023-05-31T20:18:01.668 DEBUG:boto:Method: HEAD 2023-05-31T20:18:01.670 DEBUG:boto:Path: /pklxkd-46/obj3?versionId=null 2023-05-31T20:18:01.670 DEBUG:boto:Data: 2023-05-31T20:18:01.670 DEBUG:boto:Headers: {} 2023-05-31T20:18:01.671 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:18:01.672 DEBUG:boto:Port: 8000 2023-05-31T20:18:01.672 DEBUG:boto:Params: {} 2023-05-31T20:18:01.673 DEBUG:boto:Token: None 2023-05-31T20:18:01.673 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:18:01 GMT /pklxkd-46/obj3?versionId=null 2023-05-31T20:18:01.674 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:/USNfSLiRUZg6k/FVTeHhktQT64= 2023-05-31T20:18:01.674 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:18:01 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:/USNfSLiRUZg6k/FVTeHhktQT64=', 'Content-Length': '0'} 2023-05-31T20:18:01.676 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:15:11 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000048f36632a3a8b385-006477ab79-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:18:01 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:18:01.677 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2023-05-31T20:18:01.677 DEBUG:boto:path=/pklxkd-46/obj3 2023-05-31T20:18:01.677 DEBUG:boto:auth_path=/pklxkd-46/obj3 2023-05-31T20:18:01.678 DEBUG:boto:path=/pklxkd-46/obj3?versionId=null 2023-05-31T20:18:01.678 DEBUG:boto:auth_path=/pklxkd-46/obj3?versionId=null 2023-05-31T20:18:01.679 DEBUG:boto:Method: GET 2023-05-31T20:18:01.679 DEBUG:boto:Path: /pklxkd-46/obj3?versionId=null 2023-05-31T20:18:01.680 DEBUG:boto:Data: 2023-05-31T20:18:01.680 DEBUG:boto:Headers: {} 2023-05-31T20:18:01.681 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:18:01.681 DEBUG:boto:Port: 8001 2023-05-31T20:18:01.681 DEBUG:boto:Params: {} 2023-05-31T20:18:01.682 DEBUG:boto:Token: None 2023-05-31T20:18:01.682 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:18:01 GMT /pklxkd-46/obj3?versionId=null 2023-05-31T20:18:01.683 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:0D2egjc2rgQR/RtLbKvJKuOrOlU= 2023-05-31T20:18:01.683 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:18:01 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:0D2egjc2rgQR/RtLbKvJKuOrOlU=', 'Content-Length': '0'} 2023-05-31T20:18:01.685 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:15:11 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000c14bf7803f9f61c2-006477ab79-162e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:18:01 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:18:01.686 DEBUG:boto:path=/pklxkd-46/obj3 2023-05-31T20:18:01.686 DEBUG:boto:auth_path=/pklxkd-46/obj3 2023-05-31T20:18:01.687 DEBUG:boto:path=/pklxkd-46/obj3?versionId=null 2023-05-31T20:18:01.687 DEBUG:boto:auth_path=/pklxkd-46/obj3?versionId=null 2023-05-31T20:18:01.687 DEBUG:boto:Method: GET 2023-05-31T20:18:01.688 DEBUG:boto:Path: /pklxkd-46/obj3?versionId=null 2023-05-31T20:18:01.688 DEBUG:boto:Data: 2023-05-31T20:18:01.688 DEBUG:boto:Headers: {} 2023-05-31T20:18:01.689 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:18:01.689 DEBUG:boto:Port: 8000 2023-05-31T20:18:01.690 DEBUG:boto:Params: {} 2023-05-31T20:18:01.690 DEBUG:boto:Token: None 2023-05-31T20:18:01.691 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:18:01 GMT /pklxkd-46/obj3?versionId=null 2023-05-31T20:18:01.691 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:0D2egjc2rgQR/RtLbKvJKuOrOlU= 2023-05-31T20:18:01.692 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:18:01 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:0D2egjc2rgQR/RtLbKvJKuOrOlU=', 'Content-Length': '0'} 2023-05-31T20:18:01.694 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:15:11 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000032b02e74ceb4041c-006477ab79-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:18:01 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:18:01.694 DEBUG:boto:path=/pklxkd-46/ 2023-05-31T20:18:01.695 DEBUG:boto:auth_path=/pklxkd-46/ 2023-05-31T20:18:01.695 DEBUG:boto:path=/pklxkd-46/?prefix=obj3 2023-05-31T20:18:01.695 DEBUG:boto:auth_path=/pklxkd-46/?prefix=obj3 2023-05-31T20:18:01.696 DEBUG:boto:Method: GET 2023-05-31T20:18:01.696 DEBUG:boto:Path: /pklxkd-46/?prefix=obj3 2023-05-31T20:18:01.696 DEBUG:boto:Data: 2023-05-31T20:18:01.697 DEBUG:boto:Headers: {} 2023-05-31T20:18:01.697 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:18:01.698 DEBUG:boto:Port: 8001 2023-05-31T20:18:01.698 DEBUG:boto:Params: {} 2023-05-31T20:18:01.699 DEBUG:boto:Token: None 2023-05-31T20:18:01.699 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:18:01 GMT /pklxkd-46/ 2023-05-31T20:18:01.700 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:atczRiAmTsKEk0V6aTVUEBRPCHs= 2023-05-31T20:18:01.700 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:18:01 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:atczRiAmTsKEk0V6aTVUEBRPCHs=', 'Content-Length': '0'} 2023-05-31T20:18:01.703 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000211911d5655fcbd0-006477ab79-162e-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:18:01 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:18:01.703 DEBUG:boto:b'pklxkd-46obj31000falseobj32023-05-31T20:15:11.255Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:18:01.704 DEBUG:boto:path=/pklxkd-46/ 2023-05-31T20:18:01.704 DEBUG:boto:auth_path=/pklxkd-46/ 2023-05-31T20:18:01.705 DEBUG:boto:path=/pklxkd-46/?prefix=obj3 2023-05-31T20:18:01.706 DEBUG:boto:auth_path=/pklxkd-46/?prefix=obj3 2023-05-31T20:18:01.706 DEBUG:boto:Method: GET 2023-05-31T20:18:01.706 DEBUG:boto:Path: /pklxkd-46/?prefix=obj3 2023-05-31T20:18:01.707 DEBUG:boto:Data: 2023-05-31T20:18:01.707 DEBUG:boto:Headers: {} 2023-05-31T20:18:01.707 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:18:01.708 DEBUG:boto:Port: 8000 2023-05-31T20:18:01.708 DEBUG:boto:Params: {} 2023-05-31T20:18:01.709 DEBUG:boto:Token: None 2023-05-31T20:18:01.709 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:18:01 GMT /pklxkd-46/ 2023-05-31T20:18:01.710 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:atczRiAmTsKEk0V6aTVUEBRPCHs= 2023-05-31T20:18:01.710 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:18:01 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:atczRiAmTsKEk0V6aTVUEBRPCHs=', 'Content-Length': '0'} 2023-05-31T20:18:01.713 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000000c3e2e0d5b8a869c-006477ab79-1630-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:18:01 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:18:01.714 DEBUG:boto:b'pklxkd-46obj31000falseobj32023-05-31T20:15:11.255Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:18:01.715 DEBUG:boto:path=/pklxkd-46/obj3 2023-05-31T20:18:01.716 DEBUG:boto:auth_path=/pklxkd-46/obj3 2023-05-31T20:18:01.716 DEBUG:boto:Method: HEAD 2023-05-31T20:18:01.717 DEBUG:boto:Path: /pklxkd-46/obj3 2023-05-31T20:18:01.717 DEBUG:boto:Data: 2023-05-31T20:18:01.718 DEBUG:boto:Headers: {} 2023-05-31T20:18:01.718 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:18:01.718 DEBUG:boto:Port: 8001 2023-05-31T20:18:01.719 DEBUG:boto:Params: {} 2023-05-31T20:18:01.719 DEBUG:boto:Token: None 2023-05-31T20:18:01.719 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:18:01 GMT /pklxkd-46/obj3 2023-05-31T20:18:01.720 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:CnHu56eqSlzOkGiPGwDREHVdAP4= 2023-05-31T20:18:01.720 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:18:01 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:CnHu56eqSlzOkGiPGwDREHVdAP4=', 'Content-Length': '0'} 2023-05-31T20:18:01.722 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:15:11 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000c7afd9fe35655abc-006477ab79-162e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:18:01 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:18:01.723 DEBUG:boto:path=/pklxkd-46/obj3 2023-05-31T20:18:01.723 DEBUG:boto:auth_path=/pklxkd-46/obj3 2023-05-31T20:18:01.724 DEBUG:boto:Method: HEAD 2023-05-31T20:18:01.724 DEBUG:boto:Path: /pklxkd-46/obj3 2023-05-31T20:18:01.724 DEBUG:boto:Data: 2023-05-31T20:18:01.725 DEBUG:boto:Headers: {} 2023-05-31T20:18:01.725 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:18:01.725 DEBUG:boto:Port: 8000 2023-05-31T20:18:01.726 DEBUG:boto:Params: {} 2023-05-31T20:18:01.727 DEBUG:boto:Token: None 2023-05-31T20:18:01.727 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:18:01 GMT /pklxkd-46/obj3 2023-05-31T20:18:01.728 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:CnHu56eqSlzOkGiPGwDREHVdAP4= 2023-05-31T20:18:01.728 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:18:01 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:CnHu56eqSlzOkGiPGwDREHVdAP4=', 'Content-Length': '0'} 2023-05-31T20:18:01.730 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:15:11 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000061d17ce07976a6f9-006477ab79-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:18:01 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:18:01.731 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2023-05-31T20:18:01.731 DEBUG:boto:path=/pklxkd-46/obj3 2023-05-31T20:18:01.731 DEBUG:boto:auth_path=/pklxkd-46/obj3 2023-05-31T20:18:01.732 DEBUG:boto:Method: GET 2023-05-31T20:18:01.732 DEBUG:boto:Path: /pklxkd-46/obj3 2023-05-31T20:18:01.733 DEBUG:boto:Data: 2023-05-31T20:18:01.733 DEBUG:boto:Headers: {} 2023-05-31T20:18:01.734 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:18:01.734 DEBUG:boto:Port: 8001 2023-05-31T20:18:01.735 DEBUG:boto:Params: {} 2023-05-31T20:18:01.735 DEBUG:boto:Token: None 2023-05-31T20:18:01.736 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:18:01 GMT /pklxkd-46/obj3 2023-05-31T20:18:01.736 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:hEFZAGb32BUNHoZ/3WcIsrYx7No= 2023-05-31T20:18:01.737 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:18:01 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:hEFZAGb32BUNHoZ/3WcIsrYx7No=', 'Content-Length': '0'} 2023-05-31T20:18:01.739 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:15:11 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000ac0f0a626eac25a6-006477ab79-162e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:18:01 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:18:01.739 DEBUG:boto:path=/pklxkd-46/obj3 2023-05-31T20:18:01.740 DEBUG:boto:auth_path=/pklxkd-46/obj3 2023-05-31T20:18:01.740 DEBUG:boto:Method: GET 2023-05-31T20:18:01.741 DEBUG:boto:Path: /pklxkd-46/obj3 2023-05-31T20:18:01.741 DEBUG:boto:Data: 2023-05-31T20:18:01.741 DEBUG:boto:Headers: {} 2023-05-31T20:18:01.742 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:18:01.742 DEBUG:boto:Port: 8000 2023-05-31T20:18:01.742 DEBUG:boto:Params: {} 2023-05-31T20:18:01.743 DEBUG:boto:Token: None 2023-05-31T20:18:01.744 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:18:01 GMT /pklxkd-46/obj3 2023-05-31T20:18:01.744 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:hEFZAGb32BUNHoZ/3WcIsrYx7No= 2023-05-31T20:18:01.745 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:18:01 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:hEFZAGb32BUNHoZ/3WcIsrYx7No=', 'Content-Length': '0'} 2023-05-31T20:18:01.747 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:15:11 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000006711cab665e1ae38-006477ab79-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:18:01 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:18:01.793 DEBUG:boto:path=/pklxkd-46/ 2023-05-31T20:18:01.793 DEBUG:boto:auth_path=/pklxkd-46/ 2023-05-31T20:18:01.794 DEBUG:boto:path=/pklxkd-46/?prefix=obj3 2023-05-31T20:18:01.794 DEBUG:boto:auth_path=/pklxkd-46/?prefix=obj3 2023-05-31T20:18:01.794 DEBUG:boto:Method: GET 2023-05-31T20:18:01.795 DEBUG:boto:Path: /pklxkd-46/?prefix=obj3 2023-05-31T20:18:01.795 DEBUG:boto:Data: 2023-05-31T20:18:01.795 DEBUG:boto:Headers: {} 2023-05-31T20:18:01.796 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:18:01.797 DEBUG:boto:Port: 8001 2023-05-31T20:18:01.797 DEBUG:boto:Params: {} 2023-05-31T20:18:01.797 DEBUG:boto:Token: None 2023-05-31T20:18:01.798 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:18:01 GMT /pklxkd-46/ 2023-05-31T20:18:01.799 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:atczRiAmTsKEk0V6aTVUEBRPCHs= 2023-05-31T20:18:01.799 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:18:01 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:atczRiAmTsKEk0V6aTVUEBRPCHs=', 'Content-Length': '0'} 2023-05-31T20:18:01.801 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000007bfd128a57a602f5-006477ab79-162e-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:18:01 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:18:01.802 DEBUG:boto:b'pklxkd-46obj31000falseobj32023-05-31T20:15:11.255Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:18:01.803 DEBUG:boto:path=/pklxkd-46/ 2023-05-31T20:18:01.803 DEBUG:boto:auth_path=/pklxkd-46/ 2023-05-31T20:18:01.804 DEBUG:boto:path=/pklxkd-46/?prefix=obj3 2023-05-31T20:18:01.804 DEBUG:boto:auth_path=/pklxkd-46/?prefix=obj3 2023-05-31T20:18:01.805 DEBUG:boto:Method: GET 2023-05-31T20:18:01.805 DEBUG:boto:Path: /pklxkd-46/?prefix=obj3 2023-05-31T20:18:01.806 DEBUG:boto:Data: 2023-05-31T20:18:01.807 DEBUG:boto:Headers: {} 2023-05-31T20:18:01.808 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:18:01.809 DEBUG:boto:Port: 8000 2023-05-31T20:18:01.809 DEBUG:boto:Params: {} 2023-05-31T20:18:01.811 DEBUG:boto:Token: None 2023-05-31T20:18:01.812 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:18:01 GMT /pklxkd-46/ 2023-05-31T20:18:01.812 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:atczRiAmTsKEk0V6aTVUEBRPCHs= 2023-05-31T20:18:01.813 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:18:01 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:atczRiAmTsKEk0V6aTVUEBRPCHs=', 'Content-Length': '0'} 2023-05-31T20:18:01.815 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000002d702a2307fcc905-006477ab79-1630-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:18:01 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:18:01.816 DEBUG:boto:b'pklxkd-46obj31000falseobj32023-05-31T20:15:11.255Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:18:01.817 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2023-05-31T20:18:01.817 DEBUG:boto:path=/pklxkd-46/obj4 2023-05-31T20:18:01.817 DEBUG:boto:auth_path=/pklxkd-46/obj4 2023-05-31T20:18:01.818 DEBUG:boto:path=/pklxkd-46/obj4?versionId=null 2023-05-31T20:18:01.818 DEBUG:boto:auth_path=/pklxkd-46/obj4?versionId=null 2023-05-31T20:18:01.819 DEBUG:boto:Method: GET 2023-05-31T20:18:01.819 DEBUG:boto:Path: /pklxkd-46/obj4?versionId=null 2023-05-31T20:18:01.819 DEBUG:boto:Data: 2023-05-31T20:18:01.820 DEBUG:boto:Headers: {} 2023-05-31T20:18:01.820 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:18:01.821 DEBUG:boto:Port: 8001 2023-05-31T20:18:01.821 DEBUG:boto:Params: {} 2023-05-31T20:18:01.821 DEBUG:boto:Token: None 2023-05-31T20:18:01.822 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:18:01 GMT /pklxkd-46/obj4?versionId=null 2023-05-31T20:18:01.823 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:4UGlLk5OUo/S5TG/OUrbVr7W16Q= 2023-05-31T20:18:01.823 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:18:01 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:4UGlLk5OUo/S5TG/OUrbVr7W16Q=', 'Content-Length': '0'} 2023-05-31T20:18:01.825 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:15:11 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000b0816b7c3f6c89ba-006477ab79-162e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:18:01 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:18:01.868 DEBUG:boto:path=/pklxkd-46/obj4 2023-05-31T20:18:01.869 DEBUG:boto:auth_path=/pklxkd-46/obj4 2023-05-31T20:18:01.870 DEBUG:boto:path=/pklxkd-46/obj4?versionId=null 2023-05-31T20:18:01.871 DEBUG:boto:auth_path=/pklxkd-46/obj4?versionId=null 2023-05-31T20:18:01.871 DEBUG:boto:Method: GET 2023-05-31T20:18:01.872 DEBUG:boto:Path: /pklxkd-46/obj4?versionId=null 2023-05-31T20:18:01.873 DEBUG:boto:Data: 2023-05-31T20:18:01.873 DEBUG:boto:Headers: {} 2023-05-31T20:18:01.875 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:18:01.875 DEBUG:boto:Port: 8000 2023-05-31T20:18:01.876 DEBUG:boto:Params: {} 2023-05-31T20:18:01.877 DEBUG:boto:Token: None 2023-05-31T20:18:01.878 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:18:01 GMT /pklxkd-46/obj4?versionId=null 2023-05-31T20:18:01.878 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:4UGlLk5OUo/S5TG/OUrbVr7W16Q= 2023-05-31T20:18:01.879 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:18:01 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:4UGlLk5OUo/S5TG/OUrbVr7W16Q=', 'Content-Length': '0'} 2023-05-31T20:18:01.882 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:15:11 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000b9f7d86715640eb7-006477ab79-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:18:01 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:18:01.883 DEBUG:boto:path=/pklxkd-46/obj4 2023-05-31T20:18:01.885 DEBUG:boto:auth_path=/pklxkd-46/obj4 2023-05-31T20:18:01.886 DEBUG:boto:path=/pklxkd-46/obj4?versionId=null 2023-05-31T20:18:01.887 DEBUG:boto:auth_path=/pklxkd-46/obj4?versionId=null 2023-05-31T20:18:01.888 DEBUG:boto:Method: HEAD 2023-05-31T20:18:01.889 DEBUG:boto:Path: /pklxkd-46/obj4?versionId=null 2023-05-31T20:18:01.890 DEBUG:boto:Data: 2023-05-31T20:18:01.891 DEBUG:boto:Headers: {} 2023-05-31T20:18:01.892 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:18:01.893 DEBUG:boto:Port: 8001 2023-05-31T20:18:01.894 DEBUG:boto:Params: {} 2023-05-31T20:18:01.895 DEBUG:boto:Token: None 2023-05-31T20:18:01.895 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:18:01 GMT /pklxkd-46/obj4?versionId=null 2023-05-31T20:18:01.896 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:/xgFV49NaxmQ1MFbgilPZTwXpQ8= 2023-05-31T20:18:01.897 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:18:01 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:/xgFV49NaxmQ1MFbgilPZTwXpQ8=', 'Content-Length': '0'} 2023-05-31T20:18:01.900 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:15:11 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000f5ca249d7d2efc47-006477ab79-162e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:18:01 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:18:01.901 DEBUG:boto:path=/pklxkd-46/obj4 2023-05-31T20:18:01.901 DEBUG:boto:auth_path=/pklxkd-46/obj4 2023-05-31T20:18:01.902 DEBUG:boto:path=/pklxkd-46/obj4?versionId=null 2023-05-31T20:18:01.903 DEBUG:boto:auth_path=/pklxkd-46/obj4?versionId=null 2023-05-31T20:18:01.903 DEBUG:boto:Method: HEAD 2023-05-31T20:18:01.904 DEBUG:boto:Path: /pklxkd-46/obj4?versionId=null 2023-05-31T20:18:01.905 DEBUG:boto:Data: 2023-05-31T20:18:01.906 DEBUG:boto:Headers: {} 2023-05-31T20:18:01.907 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:18:01.907 DEBUG:boto:Port: 8000 2023-05-31T20:18:01.908 DEBUG:boto:Params: {} 2023-05-31T20:18:01.908 DEBUG:boto:Token: None 2023-05-31T20:18:01.909 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:18:01 GMT /pklxkd-46/obj4?versionId=null 2023-05-31T20:18:01.909 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:/xgFV49NaxmQ1MFbgilPZTwXpQ8= 2023-05-31T20:18:01.910 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:18:01 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:/xgFV49NaxmQ1MFbgilPZTwXpQ8=', 'Content-Length': '0'} 2023-05-31T20:18:01.912 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:15:11 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000e4c0fe7ce2070243-006477ab79-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:18:01 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:18:01.912 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2023-05-31T20:18:01.913 DEBUG:boto:path=/pklxkd-46/obj4 2023-05-31T20:18:01.913 DEBUG:boto:auth_path=/pklxkd-46/obj4 2023-05-31T20:18:01.913 DEBUG:boto:path=/pklxkd-46/obj4?versionId=null 2023-05-31T20:18:01.914 DEBUG:boto:auth_path=/pklxkd-46/obj4?versionId=null 2023-05-31T20:18:01.914 DEBUG:boto:Method: GET 2023-05-31T20:18:01.915 DEBUG:boto:Path: /pklxkd-46/obj4?versionId=null 2023-05-31T20:18:01.915 DEBUG:boto:Data: 2023-05-31T20:18:01.916 DEBUG:boto:Headers: {} 2023-05-31T20:18:01.916 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:18:01.916 DEBUG:boto:Port: 8001 2023-05-31T20:18:01.917 DEBUG:boto:Params: {} 2023-05-31T20:18:01.918 DEBUG:boto:Token: None 2023-05-31T20:18:01.918 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:18:01 GMT /pklxkd-46/obj4?versionId=null 2023-05-31T20:18:01.919 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:4UGlLk5OUo/S5TG/OUrbVr7W16Q= 2023-05-31T20:18:01.919 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:18:01 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:4UGlLk5OUo/S5TG/OUrbVr7W16Q=', 'Content-Length': '0'} 2023-05-31T20:18:01.921 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:15:11 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000008e8723455a1f2054-006477ab79-162e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:18:01 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:18:01.922 DEBUG:boto:path=/pklxkd-46/obj4 2023-05-31T20:18:01.922 DEBUG:boto:auth_path=/pklxkd-46/obj4 2023-05-31T20:18:01.923 DEBUG:boto:path=/pklxkd-46/obj4?versionId=null 2023-05-31T20:18:01.923 DEBUG:boto:auth_path=/pklxkd-46/obj4?versionId=null 2023-05-31T20:18:01.924 DEBUG:boto:Method: GET 2023-05-31T20:18:01.925 DEBUG:boto:Path: /pklxkd-46/obj4?versionId=null 2023-05-31T20:18:01.925 DEBUG:boto:Data: 2023-05-31T20:18:01.926 DEBUG:boto:Headers: {} 2023-05-31T20:18:01.927 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:18:01.927 DEBUG:boto:Port: 8000 2023-05-31T20:18:01.927 DEBUG:boto:Params: {} 2023-05-31T20:18:01.928 DEBUG:boto:Token: None 2023-05-31T20:18:01.929 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:18:01 GMT /pklxkd-46/obj4?versionId=null 2023-05-31T20:18:01.929 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:4UGlLk5OUo/S5TG/OUrbVr7W16Q= 2023-05-31T20:18:01.930 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:18:01 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:4UGlLk5OUo/S5TG/OUrbVr7W16Q=', 'Content-Length': '0'} 2023-05-31T20:18:01.932 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:15:11 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000009595a74751de5625-006477ab79-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:18:01 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:18:01.933 DEBUG:boto:path=/pklxkd-46/ 2023-05-31T20:18:01.934 DEBUG:boto:auth_path=/pklxkd-46/ 2023-05-31T20:18:01.934 DEBUG:boto:path=/pklxkd-46/?prefix=obj4 2023-05-31T20:18:01.935 DEBUG:boto:auth_path=/pklxkd-46/?prefix=obj4 2023-05-31T20:18:01.935 DEBUG:boto:Method: GET 2023-05-31T20:18:01.935 DEBUG:boto:Path: /pklxkd-46/?prefix=obj4 2023-05-31T20:18:01.936 DEBUG:boto:Data: 2023-05-31T20:18:01.936 DEBUG:boto:Headers: {} 2023-05-31T20:18:01.936 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:18:01.937 DEBUG:boto:Port: 8001 2023-05-31T20:18:01.937 DEBUG:boto:Params: {} 2023-05-31T20:18:01.938 DEBUG:boto:Token: None 2023-05-31T20:18:01.938 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:18:01 GMT /pklxkd-46/ 2023-05-31T20:18:01.939 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:atczRiAmTsKEk0V6aTVUEBRPCHs= 2023-05-31T20:18:01.939 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:18:01 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:atczRiAmTsKEk0V6aTVUEBRPCHs=', 'Content-Length': '0'} 2023-05-31T20:18:01.942 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000003d25b74a79b2dd0d-006477ab79-162e-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:18:01 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:18:01.943 DEBUG:boto:b'pklxkd-46obj41000falseobj42023-05-31T20:15:11.317Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:18:01.944 DEBUG:boto:path=/pklxkd-46/ 2023-05-31T20:18:01.944 DEBUG:boto:auth_path=/pklxkd-46/ 2023-05-31T20:18:01.945 DEBUG:boto:path=/pklxkd-46/?prefix=obj4 2023-05-31T20:18:01.945 DEBUG:boto:auth_path=/pklxkd-46/?prefix=obj4 2023-05-31T20:18:01.945 DEBUG:boto:Method: GET 2023-05-31T20:18:01.946 DEBUG:boto:Path: /pklxkd-46/?prefix=obj4 2023-05-31T20:18:01.946 DEBUG:boto:Data: 2023-05-31T20:18:01.947 DEBUG:boto:Headers: {} 2023-05-31T20:18:01.947 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:18:01.948 DEBUG:boto:Port: 8000 2023-05-31T20:18:01.948 DEBUG:boto:Params: {} 2023-05-31T20:18:01.949 DEBUG:boto:Token: None 2023-05-31T20:18:01.949 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:18:01 GMT /pklxkd-46/ 2023-05-31T20:18:01.950 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:atczRiAmTsKEk0V6aTVUEBRPCHs= 2023-05-31T20:18:01.951 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:18:01 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:atczRiAmTsKEk0V6aTVUEBRPCHs=', 'Content-Length': '0'} 2023-05-31T20:18:01.953 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d352b754e9bce6d2-006477ab79-1630-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:18:01 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:18:01.954 DEBUG:boto:b'pklxkd-46obj41000falseobj42023-05-31T20:15:11.317Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:18:01.955 DEBUG:boto:path=/pklxkd-46/obj4 2023-05-31T20:18:01.955 DEBUG:boto:auth_path=/pklxkd-46/obj4 2023-05-31T20:18:01.955 DEBUG:boto:Method: HEAD 2023-05-31T20:18:01.956 DEBUG:boto:Path: /pklxkd-46/obj4 2023-05-31T20:18:01.956 DEBUG:boto:Data: 2023-05-31T20:18:01.957 DEBUG:boto:Headers: {} 2023-05-31T20:18:01.957 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:18:01.958 DEBUG:boto:Port: 8001 2023-05-31T20:18:01.958 DEBUG:boto:Params: {} 2023-05-31T20:18:01.958 DEBUG:boto:Token: None 2023-05-31T20:18:01.959 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:18:01 GMT /pklxkd-46/obj4 2023-05-31T20:18:01.959 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:jJjYtsw7kUyc2QtviIkoe6suiU0= 2023-05-31T20:18:01.960 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:18:01 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:jJjYtsw7kUyc2QtviIkoe6suiU0=', 'Content-Length': '0'} 2023-05-31T20:18:01.962 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:15:11 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000006fdedb4ee0a68cd4-006477ab79-162e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:18:01 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:18:01.963 DEBUG:boto:path=/pklxkd-46/obj4 2023-05-31T20:18:01.963 DEBUG:boto:auth_path=/pklxkd-46/obj4 2023-05-31T20:18:01.964 DEBUG:boto:Method: HEAD 2023-05-31T20:18:01.964 DEBUG:boto:Path: /pklxkd-46/obj4 2023-05-31T20:18:01.964 DEBUG:boto:Data: 2023-05-31T20:18:01.965 DEBUG:boto:Headers: {} 2023-05-31T20:18:01.965 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:18:01.966 DEBUG:boto:Port: 8000 2023-05-31T20:18:01.966 DEBUG:boto:Params: {} 2023-05-31T20:18:01.967 DEBUG:boto:Token: None 2023-05-31T20:18:01.968 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:18:01 GMT /pklxkd-46/obj4 2023-05-31T20:18:01.968 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:jJjYtsw7kUyc2QtviIkoe6suiU0= 2023-05-31T20:18:01.969 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:18:01 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:jJjYtsw7kUyc2QtviIkoe6suiU0=', 'Content-Length': '0'} 2023-05-31T20:18:01.970 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:15:11 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000e1fa8ca50868d0a0-006477ab79-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:18:01 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:18:01.971 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2023-05-31T20:18:01.971 DEBUG:boto:path=/pklxkd-46/obj4 2023-05-31T20:18:01.972 DEBUG:boto:auth_path=/pklxkd-46/obj4 2023-05-31T20:18:01.972 DEBUG:boto:Method: GET 2023-05-31T20:18:01.973 DEBUG:boto:Path: /pklxkd-46/obj4 2023-05-31T20:18:01.974 DEBUG:boto:Data: 2023-05-31T20:18:01.974 DEBUG:boto:Headers: {} 2023-05-31T20:18:01.974 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:18:01.975 DEBUG:boto:Port: 8001 2023-05-31T20:18:01.975 DEBUG:boto:Params: {} 2023-05-31T20:18:01.976 DEBUG:boto:Token: None 2023-05-31T20:18:01.976 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:18:01 GMT /pklxkd-46/obj4 2023-05-31T20:18:01.977 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:tqm7D/yKbcjxxuHEEdwxTR5VDxQ= 2023-05-31T20:18:01.978 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:18:01 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:tqm7D/yKbcjxxuHEEdwxTR5VDxQ=', 'Content-Length': '0'} 2023-05-31T20:18:01.980 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:15:11 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000e3f00f11bb1e0b7f-006477ab79-162e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:18:01 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:18:01.981 DEBUG:boto:path=/pklxkd-46/obj4 2023-05-31T20:18:01.982 DEBUG:boto:auth_path=/pklxkd-46/obj4 2023-05-31T20:18:01.982 DEBUG:boto:Method: GET 2023-05-31T20:18:01.982 DEBUG:boto:Path: /pklxkd-46/obj4 2023-05-31T20:18:01.983 DEBUG:boto:Data: 2023-05-31T20:18:01.983 DEBUG:boto:Headers: {} 2023-05-31T20:18:01.984 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:18:01.984 DEBUG:boto:Port: 8000 2023-05-31T20:18:01.984 DEBUG:boto:Params: {} 2023-05-31T20:18:01.985 DEBUG:boto:Token: None 2023-05-31T20:18:01.985 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:18:01 GMT /pklxkd-46/obj4 2023-05-31T20:18:01.986 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:tqm7D/yKbcjxxuHEEdwxTR5VDxQ= 2023-05-31T20:18:01.986 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:18:01 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:tqm7D/yKbcjxxuHEEdwxTR5VDxQ=', 'Content-Length': '0'} 2023-05-31T20:18:01.988 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:15:11 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000002e0453c0841c69a4-006477ab79-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:18:01 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:18:02.029 DEBUG:boto:path=/pklxkd-46/ 2023-05-31T20:18:02.029 DEBUG:boto:auth_path=/pklxkd-46/ 2023-05-31T20:18:02.030 DEBUG:boto:path=/pklxkd-46/?prefix=obj4 2023-05-31T20:18:02.031 DEBUG:boto:auth_path=/pklxkd-46/?prefix=obj4 2023-05-31T20:18:02.031 DEBUG:boto:Method: GET 2023-05-31T20:18:02.032 DEBUG:boto:Path: /pklxkd-46/?prefix=obj4 2023-05-31T20:18:02.032 DEBUG:boto:Data: 2023-05-31T20:18:02.033 DEBUG:boto:Headers: {} 2023-05-31T20:18:02.034 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:18:02.034 DEBUG:boto:Port: 8001 2023-05-31T20:18:02.034 DEBUG:boto:Params: {} 2023-05-31T20:18:02.035 DEBUG:boto:Token: None 2023-05-31T20:18:02.035 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:18:02 GMT /pklxkd-46/ 2023-05-31T20:18:02.036 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:4Td79Wla6G00iJOPln2dNo/Pl/U= 2023-05-31T20:18:02.037 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:18:02 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:4Td79Wla6G00iJOPln2dNo/Pl/U=', 'Content-Length': '0'} 2023-05-31T20:18:02.039 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000097fd6c829dca8caa-006477ab7a-162e-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:18:02 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:18:02.040 DEBUG:boto:b'pklxkd-46obj41000falseobj42023-05-31T20:15:11.317Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:18:02.041 DEBUG:boto:path=/pklxkd-46/ 2023-05-31T20:18:02.041 DEBUG:boto:auth_path=/pklxkd-46/ 2023-05-31T20:18:02.041 DEBUG:boto:path=/pklxkd-46/?prefix=obj4 2023-05-31T20:18:02.042 DEBUG:boto:auth_path=/pklxkd-46/?prefix=obj4 2023-05-31T20:18:02.042 DEBUG:boto:Method: GET 2023-05-31T20:18:02.043 DEBUG:boto:Path: /pklxkd-46/?prefix=obj4 2023-05-31T20:18:02.043 DEBUG:boto:Data: 2023-05-31T20:18:02.044 DEBUG:boto:Headers: {} 2023-05-31T20:18:02.044 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:18:02.044 DEBUG:boto:Port: 8000 2023-05-31T20:18:02.045 DEBUG:boto:Params: {} 2023-05-31T20:18:02.045 DEBUG:boto:Token: None 2023-05-31T20:18:02.046 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:18:02 GMT /pklxkd-46/ 2023-05-31T20:18:02.046 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:4Td79Wla6G00iJOPln2dNo/Pl/U= 2023-05-31T20:18:02.047 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:18:02 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:4Td79Wla6G00iJOPln2dNo/Pl/U=', 'Content-Length': '0'} 2023-05-31T20:18:02.049 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000107ce98b2f9df5bd-006477ab7a-1630-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:18:02 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:18:02.050 DEBUG:boto:b'pklxkd-46obj41000falseobj42023-05-31T20:15:11.317Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:18:02.051 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj5 2023-05-31T20:18:02.051 DEBUG:boto:path=/pklxkd-46/obj5 2023-05-31T20:18:02.051 DEBUG:boto:auth_path=/pklxkd-46/obj5 2023-05-31T20:18:02.052 DEBUG:boto:path=/pklxkd-46/obj5?versionId=null 2023-05-31T20:18:02.052 DEBUG:boto:auth_path=/pklxkd-46/obj5?versionId=null 2023-05-31T20:18:02.053 DEBUG:boto:Method: GET 2023-05-31T20:18:02.053 DEBUG:boto:Path: /pklxkd-46/obj5?versionId=null 2023-05-31T20:18:02.053 DEBUG:boto:Data: 2023-05-31T20:18:02.054 DEBUG:boto:Headers: {} 2023-05-31T20:18:02.054 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:18:02.055 DEBUG:boto:Port: 8001 2023-05-31T20:18:02.055 DEBUG:boto:Params: {} 2023-05-31T20:18:02.055 DEBUG:boto:Token: None 2023-05-31T20:18:02.056 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:18:02 GMT /pklxkd-46/obj5?versionId=null 2023-05-31T20:18:02.056 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:ueh6tFmRkZ3fQ1vC4vv0aQIneto= 2023-05-31T20:18:02.057 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:18:02 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:ueh6tFmRkZ3fQ1vC4vv0aQIneto=', 'Content-Length': '0'} 2023-05-31T20:18:02.059 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:17:25 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000e86d91d9c15235ea-006477ab7a-162e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:18:02 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:18:02.060 DEBUG:boto:path=/pklxkd-46/obj5 2023-05-31T20:18:02.061 DEBUG:boto:auth_path=/pklxkd-46/obj5 2023-05-31T20:18:02.061 DEBUG:boto:path=/pklxkd-46/obj5?versionId=null 2023-05-31T20:18:02.061 DEBUG:boto:auth_path=/pklxkd-46/obj5?versionId=null 2023-05-31T20:18:02.062 DEBUG:boto:Method: GET 2023-05-31T20:18:02.062 DEBUG:boto:Path: /pklxkd-46/obj5?versionId=null 2023-05-31T20:18:02.062 DEBUG:boto:Data: 2023-05-31T20:18:02.063 DEBUG:boto:Headers: {} 2023-05-31T20:18:02.063 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:18:02.064 DEBUG:boto:Port: 8000 2023-05-31T20:18:02.064 DEBUG:boto:Params: {} 2023-05-31T20:18:02.065 DEBUG:boto:Token: None 2023-05-31T20:18:02.065 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:18:02 GMT /pklxkd-46/obj5?versionId=null 2023-05-31T20:18:02.066 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:ueh6tFmRkZ3fQ1vC4vv0aQIneto= 2023-05-31T20:18:02.066 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:18:02 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:ueh6tFmRkZ3fQ1vC4vv0aQIneto=', 'Content-Length': '0'} 2023-05-31T20:18:02.068 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:17:25 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000035b590946f7882d0-006477ab7a-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:18:02 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:18:02.069 DEBUG:boto:path=/pklxkd-46/obj5 2023-05-31T20:18:02.070 DEBUG:boto:auth_path=/pklxkd-46/obj5 2023-05-31T20:18:02.070 DEBUG:boto:path=/pklxkd-46/obj5?versionId=null 2023-05-31T20:18:02.070 DEBUG:boto:auth_path=/pklxkd-46/obj5?versionId=null 2023-05-31T20:18:02.071 DEBUG:boto:Method: HEAD 2023-05-31T20:18:02.071 DEBUG:boto:Path: /pklxkd-46/obj5?versionId=null 2023-05-31T20:18:02.072 DEBUG:boto:Data: 2023-05-31T20:18:02.072 DEBUG:boto:Headers: {} 2023-05-31T20:18:02.072 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:18:02.073 DEBUG:boto:Port: 8001 2023-05-31T20:18:02.073 DEBUG:boto:Params: {} 2023-05-31T20:18:02.073 DEBUG:boto:Token: None 2023-05-31T20:18:02.074 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:18:02 GMT /pklxkd-46/obj5?versionId=null 2023-05-31T20:18:02.074 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:BxPb0ReP5BlVZAqAMdnB2q16MZY= 2023-05-31T20:18:02.075 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:18:02 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:BxPb0ReP5BlVZAqAMdnB2q16MZY=', 'Content-Length': '0'} 2023-05-31T20:18:02.077 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:17:25 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000aeb782b9e8f3b82b-006477ab7a-162e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:18:02 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:18:02.077 DEBUG:boto:path=/pklxkd-46/obj5 2023-05-31T20:18:02.078 DEBUG:boto:auth_path=/pklxkd-46/obj5 2023-05-31T20:18:02.078 DEBUG:boto:path=/pklxkd-46/obj5?versionId=null 2023-05-31T20:18:02.079 DEBUG:boto:auth_path=/pklxkd-46/obj5?versionId=null 2023-05-31T20:18:02.079 DEBUG:boto:Method: HEAD 2023-05-31T20:18:02.079 DEBUG:boto:Path: /pklxkd-46/obj5?versionId=null 2023-05-31T20:18:02.080 DEBUG:boto:Data: 2023-05-31T20:18:02.080 DEBUG:boto:Headers: {} 2023-05-31T20:18:02.081 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:18:02.081 DEBUG:boto:Port: 8000 2023-05-31T20:18:02.081 DEBUG:boto:Params: {} 2023-05-31T20:18:02.082 DEBUG:boto:Token: None 2023-05-31T20:18:02.082 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:18:02 GMT /pklxkd-46/obj5?versionId=null 2023-05-31T20:18:02.082 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:BxPb0ReP5BlVZAqAMdnB2q16MZY= 2023-05-31T20:18:02.083 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:18:02 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:BxPb0ReP5BlVZAqAMdnB2q16MZY=', 'Content-Length': '0'} 2023-05-31T20:18:02.085 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:17:25 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000003d2e369f55502aa8-006477ab7a-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:18:02 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:18:02.085 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj5 2023-05-31T20:18:02.086 DEBUG:boto:path=/pklxkd-46/obj5 2023-05-31T20:18:02.086 DEBUG:boto:auth_path=/pklxkd-46/obj5 2023-05-31T20:18:02.086 DEBUG:boto:path=/pklxkd-46/obj5?versionId=null 2023-05-31T20:18:02.087 DEBUG:boto:auth_path=/pklxkd-46/obj5?versionId=null 2023-05-31T20:18:02.087 DEBUG:boto:Method: GET 2023-05-31T20:18:02.087 DEBUG:boto:Path: /pklxkd-46/obj5?versionId=null 2023-05-31T20:18:02.088 DEBUG:boto:Data: 2023-05-31T20:18:02.088 DEBUG:boto:Headers: {} 2023-05-31T20:18:02.089 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:18:02.089 DEBUG:boto:Port: 8001 2023-05-31T20:18:02.090 DEBUG:boto:Params: {} 2023-05-31T20:18:02.090 DEBUG:boto:Token: None 2023-05-31T20:18:02.090 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:18:02 GMT /pklxkd-46/obj5?versionId=null 2023-05-31T20:18:02.091 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:ueh6tFmRkZ3fQ1vC4vv0aQIneto= 2023-05-31T20:18:02.091 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:18:02 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:ueh6tFmRkZ3fQ1vC4vv0aQIneto=', 'Content-Length': '0'} 2023-05-31T20:18:02.093 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:17:25 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000a704bb7e335264ed-006477ab7a-162e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:18:02 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:18:02.094 DEBUG:boto:path=/pklxkd-46/obj5 2023-05-31T20:18:02.095 DEBUG:boto:auth_path=/pklxkd-46/obj5 2023-05-31T20:18:02.095 DEBUG:boto:path=/pklxkd-46/obj5?versionId=null 2023-05-31T20:18:02.095 DEBUG:boto:auth_path=/pklxkd-46/obj5?versionId=null 2023-05-31T20:18:02.096 DEBUG:boto:Method: GET 2023-05-31T20:18:02.096 DEBUG:boto:Path: /pklxkd-46/obj5?versionId=null 2023-05-31T20:18:02.097 DEBUG:boto:Data: 2023-05-31T20:18:02.097 DEBUG:boto:Headers: {} 2023-05-31T20:18:02.097 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:18:02.098 DEBUG:boto:Port: 8000 2023-05-31T20:18:02.098 DEBUG:boto:Params: {} 2023-05-31T20:18:02.099 DEBUG:boto:Token: None 2023-05-31T20:18:02.099 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:18:02 GMT /pklxkd-46/obj5?versionId=null 2023-05-31T20:18:02.100 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:ueh6tFmRkZ3fQ1vC4vv0aQIneto= 2023-05-31T20:18:02.100 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:18:02 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:ueh6tFmRkZ3fQ1vC4vv0aQIneto=', 'Content-Length': '0'} 2023-05-31T20:18:02.103 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:17:25 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000e348d370f743acef-006477ab7a-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:18:02 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:18:02.148 DEBUG:boto:path=/pklxkd-46/ 2023-05-31T20:18:02.149 DEBUG:boto:auth_path=/pklxkd-46/ 2023-05-31T20:18:02.150 DEBUG:boto:path=/pklxkd-46/?prefix=obj5 2023-05-31T20:18:02.150 DEBUG:boto:auth_path=/pklxkd-46/?prefix=obj5 2023-05-31T20:18:02.150 DEBUG:boto:Method: GET 2023-05-31T20:18:02.151 DEBUG:boto:Path: /pklxkd-46/?prefix=obj5 2023-05-31T20:18:02.151 DEBUG:boto:Data: 2023-05-31T20:18:02.151 DEBUG:boto:Headers: {} 2023-05-31T20:18:02.152 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:18:02.153 DEBUG:boto:Port: 8001 2023-05-31T20:18:02.153 DEBUG:boto:Params: {} 2023-05-31T20:18:02.153 DEBUG:boto:Token: None 2023-05-31T20:18:02.154 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:18:02 GMT /pklxkd-46/ 2023-05-31T20:18:02.155 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:4Td79Wla6G00iJOPln2dNo/Pl/U= 2023-05-31T20:18:02.155 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:18:02 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:4Td79Wla6G00iJOPln2dNo/Pl/U=', 'Content-Length': '0'} 2023-05-31T20:18:02.158 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000001c30a11c088044a8-006477ab7a-162e-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:18:02 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:18:02.159 DEBUG:boto:b'pklxkd-46obj51000falseobj52023-05-31T20:17:25.589Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:18:02.160 DEBUG:boto:path=/pklxkd-46/ 2023-05-31T20:18:02.160 DEBUG:boto:auth_path=/pklxkd-46/ 2023-05-31T20:18:02.160 DEBUG:boto:path=/pklxkd-46/?prefix=obj5 2023-05-31T20:18:02.161 DEBUG:boto:auth_path=/pklxkd-46/?prefix=obj5 2023-05-31T20:18:02.161 DEBUG:boto:Method: GET 2023-05-31T20:18:02.161 DEBUG:boto:Path: /pklxkd-46/?prefix=obj5 2023-05-31T20:18:02.162 DEBUG:boto:Data: 2023-05-31T20:18:02.162 DEBUG:boto:Headers: {} 2023-05-31T20:18:02.162 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:18:02.163 DEBUG:boto:Port: 8000 2023-05-31T20:18:02.163 DEBUG:boto:Params: {} 2023-05-31T20:18:02.164 DEBUG:boto:Token: None 2023-05-31T20:18:02.164 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:18:02 GMT /pklxkd-46/ 2023-05-31T20:18:02.165 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:4Td79Wla6G00iJOPln2dNo/Pl/U= 2023-05-31T20:18:02.165 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:18:02 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:4Td79Wla6G00iJOPln2dNo/Pl/U=', 'Content-Length': '0'} 2023-05-31T20:18:02.168 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000628b991febb70eed-006477ab7a-1630-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:18:02 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:18:02.168 DEBUG:boto:b'pklxkd-46obj51000falseobj52023-05-31T20:17:25.589Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:18:02.169 DEBUG:boto:path=/pklxkd-46/obj5 2023-05-31T20:18:02.170 DEBUG:boto:auth_path=/pklxkd-46/obj5 2023-05-31T20:18:02.170 DEBUG:boto:Method: HEAD 2023-05-31T20:18:02.170 DEBUG:boto:Path: /pklxkd-46/obj5 2023-05-31T20:18:02.171 DEBUG:boto:Data: 2023-05-31T20:18:02.171 DEBUG:boto:Headers: {} 2023-05-31T20:18:02.171 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:18:02.172 DEBUG:boto:Port: 8001 2023-05-31T20:18:02.172 DEBUG:boto:Params: {} 2023-05-31T20:18:02.173 DEBUG:boto:Token: None 2023-05-31T20:18:02.173 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:18:02 GMT /pklxkd-46/obj5 2023-05-31T20:18:02.174 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:GoOiRGhOKw9VOLCP/hxOfnOKTJs= 2023-05-31T20:18:02.174 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:18:02 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:GoOiRGhOKw9VOLCP/hxOfnOKTJs=', 'Content-Length': '0'} 2023-05-31T20:18:02.177 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:17:25 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000f3ea895986184cb3-006477ab7a-162e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:18:02 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:18:02.177 DEBUG:boto:path=/pklxkd-46/obj5 2023-05-31T20:18:02.178 DEBUG:boto:auth_path=/pklxkd-46/obj5 2023-05-31T20:18:02.178 DEBUG:boto:Method: HEAD 2023-05-31T20:18:02.178 DEBUG:boto:Path: /pklxkd-46/obj5 2023-05-31T20:18:02.179 DEBUG:boto:Data: 2023-05-31T20:18:02.179 DEBUG:boto:Headers: {} 2023-05-31T20:18:02.179 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:18:02.180 DEBUG:boto:Port: 8000 2023-05-31T20:18:02.180 DEBUG:boto:Params: {} 2023-05-31T20:18:02.181 DEBUG:boto:Token: None 2023-05-31T20:18:02.181 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:18:02 GMT /pklxkd-46/obj5 2023-05-31T20:18:02.182 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:GoOiRGhOKw9VOLCP/hxOfnOKTJs= 2023-05-31T20:18:02.182 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:18:02 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:GoOiRGhOKw9VOLCP/hxOfnOKTJs=', 'Content-Length': '0'} 2023-05-31T20:18:02.184 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:17:25 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000021cc02192a0c0c30-006477ab7a-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:18:02 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:18:02.185 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj5 2023-05-31T20:18:02.186 DEBUG:boto:path=/pklxkd-46/obj5 2023-05-31T20:18:02.186 DEBUG:boto:auth_path=/pklxkd-46/obj5 2023-05-31T20:18:02.186 DEBUG:boto:Method: GET 2023-05-31T20:18:02.187 DEBUG:boto:Path: /pklxkd-46/obj5 2023-05-31T20:18:02.187 DEBUG:boto:Data: 2023-05-31T20:18:02.187 DEBUG:boto:Headers: {} 2023-05-31T20:18:02.188 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:18:02.188 DEBUG:boto:Port: 8001 2023-05-31T20:18:02.189 DEBUG:boto:Params: {} 2023-05-31T20:18:02.189 DEBUG:boto:Token: None 2023-05-31T20:18:02.190 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:18:02 GMT /pklxkd-46/obj5 2023-05-31T20:18:02.190 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:H5Ep51QkYTio1a7LcEqZOktKfJ0= 2023-05-31T20:18:02.191 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:18:02 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:H5Ep51QkYTio1a7LcEqZOktKfJ0=', 'Content-Length': '0'} 2023-05-31T20:18:02.192 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:17:25 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000b36c2198327584d0-006477ab7a-162e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:18:02 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:18:02.240 DEBUG:boto:path=/pklxkd-46/obj5 2023-05-31T20:18:02.241 DEBUG:boto:auth_path=/pklxkd-46/obj5 2023-05-31T20:18:02.241 DEBUG:boto:Method: GET 2023-05-31T20:18:02.242 DEBUG:boto:Path: /pklxkd-46/obj5 2023-05-31T20:18:02.243 DEBUG:boto:Data: 2023-05-31T20:18:02.243 DEBUG:boto:Headers: {} 2023-05-31T20:18:02.243 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:18:02.244 DEBUG:boto:Port: 8000 2023-05-31T20:18:02.244 DEBUG:boto:Params: {} 2023-05-31T20:18:02.245 DEBUG:boto:Token: None 2023-05-31T20:18:02.246 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:18:02 GMT /pklxkd-46/obj5 2023-05-31T20:18:02.246 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:H5Ep51QkYTio1a7LcEqZOktKfJ0= 2023-05-31T20:18:02.246 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:18:02 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:H5Ep51QkYTio1a7LcEqZOktKfJ0=', 'Content-Length': '0'} 2023-05-31T20:18:02.248 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:17:25 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000c0de4526fda64898-006477ab7a-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:18:02 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:18:02.249 DEBUG:boto:path=/pklxkd-46/ 2023-05-31T20:18:02.249 DEBUG:boto:auth_path=/pklxkd-46/ 2023-05-31T20:18:02.250 DEBUG:boto:path=/pklxkd-46/?prefix=obj5 2023-05-31T20:18:02.250 DEBUG:boto:auth_path=/pklxkd-46/?prefix=obj5 2023-05-31T20:18:02.251 DEBUG:boto:Method: GET 2023-05-31T20:18:02.251 DEBUG:boto:Path: /pklxkd-46/?prefix=obj5 2023-05-31T20:18:02.251 DEBUG:boto:Data: 2023-05-31T20:18:02.252 DEBUG:boto:Headers: {} 2023-05-31T20:18:02.252 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:18:02.252 DEBUG:boto:Port: 8001 2023-05-31T20:18:02.253 DEBUG:boto:Params: {} 2023-05-31T20:18:02.253 DEBUG:boto:Token: None 2023-05-31T20:18:02.254 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:18:02 GMT /pklxkd-46/ 2023-05-31T20:18:02.254 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:4Td79Wla6G00iJOPln2dNo/Pl/U= 2023-05-31T20:18:02.255 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:18:02 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:4Td79Wla6G00iJOPln2dNo/Pl/U=', 'Content-Length': '0'} 2023-05-31T20:18:02.257 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f07dcbece3bb0253-006477ab7a-162e-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:18:02 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:18:02.258 DEBUG:boto:b'pklxkd-46obj51000falseobj52023-05-31T20:17:25.589Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:18:02.258 DEBUG:boto:path=/pklxkd-46/ 2023-05-31T20:18:02.259 DEBUG:boto:auth_path=/pklxkd-46/ 2023-05-31T20:18:02.259 DEBUG:boto:path=/pklxkd-46/?prefix=obj5 2023-05-31T20:18:02.259 DEBUG:boto:auth_path=/pklxkd-46/?prefix=obj5 2023-05-31T20:18:02.260 DEBUG:boto:Method: GET 2023-05-31T20:18:02.260 DEBUG:boto:Path: /pklxkd-46/?prefix=obj5 2023-05-31T20:18:02.261 DEBUG:boto:Data: 2023-05-31T20:18:02.261 DEBUG:boto:Headers: {} 2023-05-31T20:18:02.261 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:18:02.262 DEBUG:boto:Port: 8000 2023-05-31T20:18:02.262 DEBUG:boto:Params: {} 2023-05-31T20:18:02.263 DEBUG:boto:Token: None 2023-05-31T20:18:02.264 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:18:02 GMT /pklxkd-46/ 2023-05-31T20:18:02.264 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:4Td79Wla6G00iJOPln2dNo/Pl/U= 2023-05-31T20:18:02.265 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:18:02 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:4Td79Wla6G00iJOPln2dNo/Pl/U=', 'Content-Length': '0'} 2023-05-31T20:18:02.267 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000002432f05907b7ad27-006477ab7a-1630-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:18:02 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:18:02.268 DEBUG:boto:b'pklxkd-46obj51000falseobj52023-05-31T20:17:25.589Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:18:02.268 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj6 2023-05-31T20:18:02.269 DEBUG:boto:path=/pklxkd-46/obj6 2023-05-31T20:18:02.269 DEBUG:boto:auth_path=/pklxkd-46/obj6 2023-05-31T20:18:02.269 DEBUG:boto:path=/pklxkd-46/obj6?versionId=null 2023-05-31T20:18:02.270 DEBUG:boto:auth_path=/pklxkd-46/obj6?versionId=null 2023-05-31T20:18:02.270 DEBUG:boto:Method: GET 2023-05-31T20:18:02.270 DEBUG:boto:Path: /pklxkd-46/obj6?versionId=null 2023-05-31T20:18:02.271 DEBUG:boto:Data: 2023-05-31T20:18:02.271 DEBUG:boto:Headers: {} 2023-05-31T20:18:02.272 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:18:02.272 DEBUG:boto:Port: 8001 2023-05-31T20:18:02.273 DEBUG:boto:Params: {} 2023-05-31T20:18:02.273 DEBUG:boto:Token: None 2023-05-31T20:18:02.273 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:18:02 GMT /pklxkd-46/obj6?versionId=null 2023-05-31T20:18:02.274 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:wThorjxCRpDRIBtjhOM2Mu6JSHg= 2023-05-31T20:18:02.274 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:18:02 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:wThorjxCRpDRIBtjhOM2Mu6JSHg=', 'Content-Length': '0'} 2023-05-31T20:18:02.276 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:17:25 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000959a58c8ba0eea19-006477ab7a-162e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:18:02 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:18:02.277 DEBUG:boto:path=/pklxkd-46/obj6 2023-05-31T20:18:02.277 DEBUG:boto:auth_path=/pklxkd-46/obj6 2023-05-31T20:18:02.278 DEBUG:boto:path=/pklxkd-46/obj6?versionId=null 2023-05-31T20:18:02.278 DEBUG:boto:auth_path=/pklxkd-46/obj6?versionId=null 2023-05-31T20:18:02.279 DEBUG:boto:Method: GET 2023-05-31T20:18:02.279 DEBUG:boto:Path: /pklxkd-46/obj6?versionId=null 2023-05-31T20:18:02.279 DEBUG:boto:Data: 2023-05-31T20:18:02.280 DEBUG:boto:Headers: {} 2023-05-31T20:18:02.280 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:18:02.280 DEBUG:boto:Port: 8000 2023-05-31T20:18:02.281 DEBUG:boto:Params: {} 2023-05-31T20:18:02.281 DEBUG:boto:Token: None 2023-05-31T20:18:02.282 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:18:02 GMT /pklxkd-46/obj6?versionId=null 2023-05-31T20:18:02.282 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:wThorjxCRpDRIBtjhOM2Mu6JSHg= 2023-05-31T20:18:02.283 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:18:02 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:wThorjxCRpDRIBtjhOM2Mu6JSHg=', 'Content-Length': '0'} 2023-05-31T20:18:02.284 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:17:25 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000ce31ce5616ff0ad9-006477ab7a-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:18:02 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:18:02.285 DEBUG:boto:path=/pklxkd-46/obj6 2023-05-31T20:18:02.286 DEBUG:boto:auth_path=/pklxkd-46/obj6 2023-05-31T20:18:02.286 DEBUG:boto:path=/pklxkd-46/obj6?versionId=null 2023-05-31T20:18:02.286 DEBUG:boto:auth_path=/pklxkd-46/obj6?versionId=null 2023-05-31T20:18:02.287 DEBUG:boto:Method: HEAD 2023-05-31T20:18:02.287 DEBUG:boto:Path: /pklxkd-46/obj6?versionId=null 2023-05-31T20:18:02.288 DEBUG:boto:Data: 2023-05-31T20:18:02.288 DEBUG:boto:Headers: {} 2023-05-31T20:18:02.289 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:18:02.289 DEBUG:boto:Port: 8001 2023-05-31T20:18:02.289 DEBUG:boto:Params: {} 2023-05-31T20:18:02.290 DEBUG:boto:Token: None 2023-05-31T20:18:02.290 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:18:02 GMT /pklxkd-46/obj6?versionId=null 2023-05-31T20:18:02.290 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:juhjQTRuCBHFcT4XlCxRk+hdms4= 2023-05-31T20:18:02.291 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:18:02 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:juhjQTRuCBHFcT4XlCxRk+hdms4=', 'Content-Length': '0'} 2023-05-31T20:18:02.293 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:17:25 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000bbb4d0a1c90a028e-006477ab7a-162e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:18:02 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:18:02.295 DEBUG:boto:path=/pklxkd-46/obj6 2023-05-31T20:18:02.296 DEBUG:boto:auth_path=/pklxkd-46/obj6 2023-05-31T20:18:02.296 DEBUG:boto:path=/pklxkd-46/obj6?versionId=null 2023-05-31T20:18:02.296 DEBUG:boto:auth_path=/pklxkd-46/obj6?versionId=null 2023-05-31T20:18:02.297 DEBUG:boto:Method: HEAD 2023-05-31T20:18:02.297 DEBUG:boto:Path: /pklxkd-46/obj6?versionId=null 2023-05-31T20:18:02.298 DEBUG:boto:Data: 2023-05-31T20:18:02.298 DEBUG:boto:Headers: {} 2023-05-31T20:18:02.299 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:18:02.299 DEBUG:boto:Port: 8000 2023-05-31T20:18:02.299 DEBUG:boto:Params: {} 2023-05-31T20:18:02.300 DEBUG:boto:Token: None 2023-05-31T20:18:02.300 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:18:02 GMT /pklxkd-46/obj6?versionId=null 2023-05-31T20:18:02.300 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:juhjQTRuCBHFcT4XlCxRk+hdms4= 2023-05-31T20:18:02.301 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:18:02 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:juhjQTRuCBHFcT4XlCxRk+hdms4=', 'Content-Length': '0'} 2023-05-31T20:18:02.303 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:17:25 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000004f95e081b3368229-006477ab7a-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:18:02 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:18:02.303 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj6 2023-05-31T20:18:02.304 DEBUG:boto:path=/pklxkd-46/obj6 2023-05-31T20:18:02.304 DEBUG:boto:auth_path=/pklxkd-46/obj6 2023-05-31T20:18:02.305 DEBUG:boto:path=/pklxkd-46/obj6?versionId=null 2023-05-31T20:18:02.305 DEBUG:boto:auth_path=/pklxkd-46/obj6?versionId=null 2023-05-31T20:18:02.306 DEBUG:boto:Method: GET 2023-05-31T20:18:02.306 DEBUG:boto:Path: /pklxkd-46/obj6?versionId=null 2023-05-31T20:18:02.307 DEBUG:boto:Data: 2023-05-31T20:18:02.307 DEBUG:boto:Headers: {} 2023-05-31T20:18:02.307 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:18:02.308 DEBUG:boto:Port: 8001 2023-05-31T20:18:02.309 DEBUG:boto:Params: {} 2023-05-31T20:18:02.309 DEBUG:boto:Token: None 2023-05-31T20:18:02.309 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:18:02 GMT /pklxkd-46/obj6?versionId=null 2023-05-31T20:18:02.310 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:wThorjxCRpDRIBtjhOM2Mu6JSHg= 2023-05-31T20:18:02.310 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:18:02 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:wThorjxCRpDRIBtjhOM2Mu6JSHg=', 'Content-Length': '0'} 2023-05-31T20:18:02.312 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:17:25 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000005380bc00036bb43f-006477ab7a-162e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:18:02 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:18:02.352 DEBUG:boto:path=/pklxkd-46/obj6 2023-05-31T20:18:02.353 DEBUG:boto:auth_path=/pklxkd-46/obj6 2023-05-31T20:18:02.353 DEBUG:boto:path=/pklxkd-46/obj6?versionId=null 2023-05-31T20:18:02.354 DEBUG:boto:auth_path=/pklxkd-46/obj6?versionId=null 2023-05-31T20:18:02.354 DEBUG:boto:Method: GET 2023-05-31T20:18:02.354 DEBUG:boto:Path: /pklxkd-46/obj6?versionId=null 2023-05-31T20:18:02.355 DEBUG:boto:Data: 2023-05-31T20:18:02.355 DEBUG:boto:Headers: {} 2023-05-31T20:18:02.355 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:18:02.356 DEBUG:boto:Port: 8000 2023-05-31T20:18:02.356 DEBUG:boto:Params: {} 2023-05-31T20:18:02.356 DEBUG:boto:Token: None 2023-05-31T20:18:02.357 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:18:02 GMT /pklxkd-46/obj6?versionId=null 2023-05-31T20:18:02.357 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:wThorjxCRpDRIBtjhOM2Mu6JSHg= 2023-05-31T20:18:02.358 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:18:02 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:wThorjxCRpDRIBtjhOM2Mu6JSHg=', 'Content-Length': '0'} 2023-05-31T20:18:02.360 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:17:25 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000007bce269dc75bcae8-006477ab7a-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:18:02 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:18:02.361 DEBUG:boto:path=/pklxkd-46/ 2023-05-31T20:18:02.362 DEBUG:boto:auth_path=/pklxkd-46/ 2023-05-31T20:18:02.362 DEBUG:boto:path=/pklxkd-46/?prefix=obj6 2023-05-31T20:18:02.363 DEBUG:boto:auth_path=/pklxkd-46/?prefix=obj6 2023-05-31T20:18:02.363 DEBUG:boto:Method: GET 2023-05-31T20:18:02.363 DEBUG:boto:Path: /pklxkd-46/?prefix=obj6 2023-05-31T20:18:02.364 DEBUG:boto:Data: 2023-05-31T20:18:02.364 DEBUG:boto:Headers: {} 2023-05-31T20:18:02.364 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:18:02.365 DEBUG:boto:Port: 8001 2023-05-31T20:18:02.365 DEBUG:boto:Params: {} 2023-05-31T20:18:02.366 DEBUG:boto:Token: None 2023-05-31T20:18:02.366 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:18:02 GMT /pklxkd-46/ 2023-05-31T20:18:02.366 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:4Td79Wla6G00iJOPln2dNo/Pl/U= 2023-05-31T20:18:02.367 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:18:02 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:4Td79Wla6G00iJOPln2dNo/Pl/U=', 'Content-Length': '0'} 2023-05-31T20:18:02.369 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e8ed4f04440ba436-006477ab7a-162e-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:18:02 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:18:02.370 DEBUG:boto:b'pklxkd-46obj61000falseobj62023-05-31T20:17:25.610Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:18:02.371 DEBUG:boto:path=/pklxkd-46/ 2023-05-31T20:18:02.371 DEBUG:boto:auth_path=/pklxkd-46/ 2023-05-31T20:18:02.372 DEBUG:boto:path=/pklxkd-46/?prefix=obj6 2023-05-31T20:18:02.372 DEBUG:boto:auth_path=/pklxkd-46/?prefix=obj6 2023-05-31T20:18:02.372 DEBUG:boto:Method: GET 2023-05-31T20:18:02.373 DEBUG:boto:Path: /pklxkd-46/?prefix=obj6 2023-05-31T20:18:02.373 DEBUG:boto:Data: 2023-05-31T20:18:02.374 DEBUG:boto:Headers: {} 2023-05-31T20:18:02.375 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:18:02.375 DEBUG:boto:Port: 8000 2023-05-31T20:18:02.376 DEBUG:boto:Params: {} 2023-05-31T20:18:02.377 DEBUG:boto:Token: None 2023-05-31T20:18:02.378 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:18:02 GMT /pklxkd-46/ 2023-05-31T20:18:02.378 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:4Td79Wla6G00iJOPln2dNo/Pl/U= 2023-05-31T20:18:02.378 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:18:02 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:4Td79Wla6G00iJOPln2dNo/Pl/U=', 'Content-Length': '0'} 2023-05-31T20:18:02.381 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000097c70f413c5b54a9-006477ab7a-1630-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:18:02 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:18:02.382 DEBUG:boto:b'pklxkd-46obj61000falseobj62023-05-31T20:17:25.610Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:18:02.382 DEBUG:boto:path=/pklxkd-46/obj6 2023-05-31T20:18:02.383 DEBUG:boto:auth_path=/pklxkd-46/obj6 2023-05-31T20:18:02.383 DEBUG:boto:Method: HEAD 2023-05-31T20:18:02.383 DEBUG:boto:Path: /pklxkd-46/obj6 2023-05-31T20:18:02.384 DEBUG:boto:Data: 2023-05-31T20:18:02.384 DEBUG:boto:Headers: {} 2023-05-31T20:18:02.385 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:18:02.386 DEBUG:boto:Port: 8001 2023-05-31T20:18:02.386 DEBUG:boto:Params: {} 2023-05-31T20:18:02.387 DEBUG:boto:Token: None 2023-05-31T20:18:02.387 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:18:02 GMT /pklxkd-46/obj6 2023-05-31T20:18:02.388 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:Rbk/qFhl3pVdqUS3NqemEimyFhM= 2023-05-31T20:18:02.388 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:18:02 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:Rbk/qFhl3pVdqUS3NqemEimyFhM=', 'Content-Length': '0'} 2023-05-31T20:18:02.390 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:17:25 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000ecef4d5af1c5b279-006477ab7a-162e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:18:02 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:18:02.390 DEBUG:boto:path=/pklxkd-46/obj6 2023-05-31T20:18:02.391 DEBUG:boto:auth_path=/pklxkd-46/obj6 2023-05-31T20:18:02.391 DEBUG:boto:Method: HEAD 2023-05-31T20:18:02.392 DEBUG:boto:Path: /pklxkd-46/obj6 2023-05-31T20:18:02.392 DEBUG:boto:Data: 2023-05-31T20:18:02.392 DEBUG:boto:Headers: {} 2023-05-31T20:18:02.393 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:18:02.393 DEBUG:boto:Port: 8000 2023-05-31T20:18:02.393 DEBUG:boto:Params: {} 2023-05-31T20:18:02.394 DEBUG:boto:Token: None 2023-05-31T20:18:02.394 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:18:02 GMT /pklxkd-46/obj6 2023-05-31T20:18:02.395 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:Rbk/qFhl3pVdqUS3NqemEimyFhM= 2023-05-31T20:18:02.396 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:18:02 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:Rbk/qFhl3pVdqUS3NqemEimyFhM=', 'Content-Length': '0'} 2023-05-31T20:18:02.398 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:17:25 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000ba318cfead7c0ae3-006477ab7a-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:18:02 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:18:02.398 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj6 2023-05-31T20:18:02.399 DEBUG:boto:path=/pklxkd-46/obj6 2023-05-31T20:18:02.399 DEBUG:boto:auth_path=/pklxkd-46/obj6 2023-05-31T20:18:02.399 DEBUG:boto:Method: GET 2023-05-31T20:18:02.400 DEBUG:boto:Path: /pklxkd-46/obj6 2023-05-31T20:18:02.400 DEBUG:boto:Data: 2023-05-31T20:18:02.401 DEBUG:boto:Headers: {} 2023-05-31T20:18:02.401 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:18:02.401 DEBUG:boto:Port: 8001 2023-05-31T20:18:02.402 DEBUG:boto:Params: {} 2023-05-31T20:18:02.402 DEBUG:boto:Token: None 2023-05-31T20:18:02.403 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:18:02 GMT /pklxkd-46/obj6 2023-05-31T20:18:02.403 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:nXsEE7DoYapn6Ik/bdHwM1pfdYk= 2023-05-31T20:18:02.403 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:18:02 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:nXsEE7DoYapn6Ik/bdHwM1pfdYk=', 'Content-Length': '0'} 2023-05-31T20:18:02.405 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:17:25 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000068c38814d7298720-006477ab7a-162e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:18:02 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:18:02.406 DEBUG:boto:path=/pklxkd-46/obj6 2023-05-31T20:18:02.407 DEBUG:boto:auth_path=/pklxkd-46/obj6 2023-05-31T20:18:02.407 DEBUG:boto:Method: GET 2023-05-31T20:18:02.407 DEBUG:boto:Path: /pklxkd-46/obj6 2023-05-31T20:18:02.408 DEBUG:boto:Data: 2023-05-31T20:18:02.408 DEBUG:boto:Headers: {} 2023-05-31T20:18:02.408 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:18:02.409 DEBUG:boto:Port: 8000 2023-05-31T20:18:02.409 DEBUG:boto:Params: {} 2023-05-31T20:18:02.410 DEBUG:boto:Token: None 2023-05-31T20:18:02.410 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:18:02 GMT /pklxkd-46/obj6 2023-05-31T20:18:02.410 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:nXsEE7DoYapn6Ik/bdHwM1pfdYk= 2023-05-31T20:18:02.411 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:18:02 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:nXsEE7DoYapn6Ik/bdHwM1pfdYk=', 'Content-Length': '0'} 2023-05-31T20:18:02.413 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:17:25 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000007bb7019568b7bcdb-006477ab7a-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:18:02 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:18:02.413 DEBUG:boto:path=/pklxkd-46/ 2023-05-31T20:18:02.414 DEBUG:boto:auth_path=/pklxkd-46/ 2023-05-31T20:18:02.414 DEBUG:boto:path=/pklxkd-46/?prefix=obj6 2023-05-31T20:18:02.414 DEBUG:boto:auth_path=/pklxkd-46/?prefix=obj6 2023-05-31T20:18:02.415 DEBUG:boto:Method: GET 2023-05-31T20:18:02.415 DEBUG:boto:Path: /pklxkd-46/?prefix=obj6 2023-05-31T20:18:02.415 DEBUG:boto:Data: 2023-05-31T20:18:02.416 DEBUG:boto:Headers: {} 2023-05-31T20:18:02.416 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:18:02.417 DEBUG:boto:Port: 8001 2023-05-31T20:18:02.417 DEBUG:boto:Params: {} 2023-05-31T20:18:02.417 DEBUG:boto:Token: None 2023-05-31T20:18:02.418 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:18:02 GMT /pklxkd-46/ 2023-05-31T20:18:02.418 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:4Td79Wla6G00iJOPln2dNo/Pl/U= 2023-05-31T20:18:02.419 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:18:02 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:4Td79Wla6G00iJOPln2dNo/Pl/U=', 'Content-Length': '0'} 2023-05-31T20:18:02.421 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000202fa5f8afb4c7a4-006477ab7a-162e-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:18:02 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:18:02.421 DEBUG:boto:b'pklxkd-46obj61000falseobj62023-05-31T20:17:25.610Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:18:02.422 DEBUG:boto:path=/pklxkd-46/ 2023-05-31T20:18:02.423 DEBUG:boto:auth_path=/pklxkd-46/ 2023-05-31T20:18:02.423 DEBUG:boto:path=/pklxkd-46/?prefix=obj6 2023-05-31T20:18:02.423 DEBUG:boto:auth_path=/pklxkd-46/?prefix=obj6 2023-05-31T20:18:02.424 DEBUG:boto:Method: GET 2023-05-31T20:18:02.424 DEBUG:boto:Path: /pklxkd-46/?prefix=obj6 2023-05-31T20:18:02.424 DEBUG:boto:Data: 2023-05-31T20:18:02.425 DEBUG:boto:Headers: {} 2023-05-31T20:18:02.425 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:18:02.425 DEBUG:boto:Port: 8000 2023-05-31T20:18:02.426 DEBUG:boto:Params: {} 2023-05-31T20:18:02.426 DEBUG:boto:Token: None 2023-05-31T20:18:02.427 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:18:02 GMT /pklxkd-46/ 2023-05-31T20:18:02.427 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:4Td79Wla6G00iJOPln2dNo/Pl/U= 2023-05-31T20:18:02.427 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:18:02 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:4Td79Wla6G00iJOPln2dNo/Pl/U=', 'Content-Length': '0'} 2023-05-31T20:18:02.430 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000005ae4865436cd75c1-006477ab7a-1630-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:18:02 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:18:02.472 DEBUG:boto:b'pklxkd-46obj61000falseobj62023-05-31T20:17:25.610Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:18:02.473 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj7 2023-05-31T20:18:02.473 DEBUG:boto:path=/pklxkd-46/obj7 2023-05-31T20:18:02.474 DEBUG:boto:auth_path=/pklxkd-46/obj7 2023-05-31T20:18:02.474 DEBUG:boto:path=/pklxkd-46/obj7?versionId=null 2023-05-31T20:18:02.475 DEBUG:boto:auth_path=/pklxkd-46/obj7?versionId=null 2023-05-31T20:18:02.475 DEBUG:boto:Method: GET 2023-05-31T20:18:02.475 DEBUG:boto:Path: /pklxkd-46/obj7?versionId=null 2023-05-31T20:18:02.476 DEBUG:boto:Data: 2023-05-31T20:18:02.476 DEBUG:boto:Headers: {} 2023-05-31T20:18:02.476 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:18:02.477 DEBUG:boto:Port: 8001 2023-05-31T20:18:02.477 DEBUG:boto:Params: {} 2023-05-31T20:18:02.477 DEBUG:boto:Token: None 2023-05-31T20:18:02.478 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:18:02 GMT /pklxkd-46/obj7?versionId=null 2023-05-31T20:18:02.478 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:y6rkRw/vAQfTHTDHpOZVAVSCtUs= 2023-05-31T20:18:02.479 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:18:02 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:y6rkRw/vAQfTHTDHpOZVAVSCtUs=', 'Content-Length': '0'} 2023-05-31T20:18:02.481 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:17:25 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000fbc5934de18152ef-006477ab7a-162e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:18:02 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:18:02.481 DEBUG:boto:path=/pklxkd-46/obj7 2023-05-31T20:18:02.482 DEBUG:boto:auth_path=/pklxkd-46/obj7 2023-05-31T20:18:02.482 DEBUG:boto:path=/pklxkd-46/obj7?versionId=null 2023-05-31T20:18:02.483 DEBUG:boto:auth_path=/pklxkd-46/obj7?versionId=null 2023-05-31T20:18:02.483 DEBUG:boto:Method: GET 2023-05-31T20:18:02.483 DEBUG:boto:Path: /pklxkd-46/obj7?versionId=null 2023-05-31T20:18:02.484 DEBUG:boto:Data: 2023-05-31T20:18:02.485 DEBUG:boto:Headers: {} 2023-05-31T20:18:02.485 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:18:02.486 DEBUG:boto:Port: 8000 2023-05-31T20:18:02.486 DEBUG:boto:Params: {} 2023-05-31T20:18:02.487 DEBUG:boto:Token: None 2023-05-31T20:18:02.487 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:18:02 GMT /pklxkd-46/obj7?versionId=null 2023-05-31T20:18:02.488 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:y6rkRw/vAQfTHTDHpOZVAVSCtUs= 2023-05-31T20:18:02.488 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:18:02 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:y6rkRw/vAQfTHTDHpOZVAVSCtUs=', 'Content-Length': '0'} 2023-05-31T20:18:02.490 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:17:25 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000006ff539b2d96c4a1-006477ab7a-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:18:02 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:18:02.491 DEBUG:boto:path=/pklxkd-46/obj7 2023-05-31T20:18:02.491 DEBUG:boto:auth_path=/pklxkd-46/obj7 2023-05-31T20:18:02.492 DEBUG:boto:path=/pklxkd-46/obj7?versionId=null 2023-05-31T20:18:02.492 DEBUG:boto:auth_path=/pklxkd-46/obj7?versionId=null 2023-05-31T20:18:02.492 DEBUG:boto:Method: HEAD 2023-05-31T20:18:02.493 DEBUG:boto:Path: /pklxkd-46/obj7?versionId=null 2023-05-31T20:18:02.493 DEBUG:boto:Data: 2023-05-31T20:18:02.493 DEBUG:boto:Headers: {} 2023-05-31T20:18:02.494 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:18:02.494 DEBUG:boto:Port: 8001 2023-05-31T20:18:02.495 DEBUG:boto:Params: {} 2023-05-31T20:18:02.495 DEBUG:boto:Token: None 2023-05-31T20:18:02.495 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:18:02 GMT /pklxkd-46/obj7?versionId=null 2023-05-31T20:18:02.496 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:i0CpSE3lC1Y9AV4E+PsSDiz7CvA= 2023-05-31T20:18:02.496 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:18:02 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:i0CpSE3lC1Y9AV4E+PsSDiz7CvA=', 'Content-Length': '0'} 2023-05-31T20:18:02.498 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:17:25 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000008bc534b950fe8508-006477ab7a-162e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:18:02 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:18:02.498 DEBUG:boto:path=/pklxkd-46/obj7 2023-05-31T20:18:02.499 DEBUG:boto:auth_path=/pklxkd-46/obj7 2023-05-31T20:18:02.499 DEBUG:boto:path=/pklxkd-46/obj7?versionId=null 2023-05-31T20:18:02.499 DEBUG:boto:auth_path=/pklxkd-46/obj7?versionId=null 2023-05-31T20:18:02.500 DEBUG:boto:Method: HEAD 2023-05-31T20:18:02.500 DEBUG:boto:Path: /pklxkd-46/obj7?versionId=null 2023-05-31T20:18:02.501 DEBUG:boto:Data: 2023-05-31T20:18:02.501 DEBUG:boto:Headers: {} 2023-05-31T20:18:02.501 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:18:02.502 DEBUG:boto:Port: 8000 2023-05-31T20:18:02.502 DEBUG:boto:Params: {} 2023-05-31T20:18:02.502 DEBUG:boto:Token: None 2023-05-31T20:18:02.503 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:18:02 GMT /pklxkd-46/obj7?versionId=null 2023-05-31T20:18:02.503 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:i0CpSE3lC1Y9AV4E+PsSDiz7CvA= 2023-05-31T20:18:02.503 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:18:02 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:i0CpSE3lC1Y9AV4E+PsSDiz7CvA=', 'Content-Length': '0'} 2023-05-31T20:18:02.505 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:17:25 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000df6c82d39c53c2f2-006477ab7a-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:18:02 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:18:02.506 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj7 2023-05-31T20:18:02.507 DEBUG:boto:path=/pklxkd-46/obj7 2023-05-31T20:18:02.507 DEBUG:boto:auth_path=/pklxkd-46/obj7 2023-05-31T20:18:02.507 DEBUG:boto:path=/pklxkd-46/obj7?versionId=null 2023-05-31T20:18:02.508 DEBUG:boto:auth_path=/pklxkd-46/obj7?versionId=null 2023-05-31T20:18:02.508 DEBUG:boto:Method: GET 2023-05-31T20:18:02.509 DEBUG:boto:Path: /pklxkd-46/obj7?versionId=null 2023-05-31T20:18:02.509 DEBUG:boto:Data: 2023-05-31T20:18:02.510 DEBUG:boto:Headers: {} 2023-05-31T20:18:02.510 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:18:02.510 DEBUG:boto:Port: 8001 2023-05-31T20:18:02.511 DEBUG:boto:Params: {} 2023-05-31T20:18:02.511 DEBUG:boto:Token: None 2023-05-31T20:18:02.511 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:18:02 GMT /pklxkd-46/obj7?versionId=null 2023-05-31T20:18:02.512 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:y6rkRw/vAQfTHTDHpOZVAVSCtUs= 2023-05-31T20:18:02.512 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:18:02 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:y6rkRw/vAQfTHTDHpOZVAVSCtUs=', 'Content-Length': '0'} 2023-05-31T20:18:02.514 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:17:25 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000009c7e5d0966252d4f-006477ab7a-162e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:18:02 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:18:02.556 DEBUG:boto:path=/pklxkd-46/obj7 2023-05-31T20:18:02.557 DEBUG:boto:auth_path=/pklxkd-46/obj7 2023-05-31T20:18:02.558 DEBUG:boto:path=/pklxkd-46/obj7?versionId=null 2023-05-31T20:18:02.559 DEBUG:boto:auth_path=/pklxkd-46/obj7?versionId=null 2023-05-31T20:18:02.559 DEBUG:boto:Method: GET 2023-05-31T20:18:02.560 DEBUG:boto:Path: /pklxkd-46/obj7?versionId=null 2023-05-31T20:18:02.561 DEBUG:boto:Data: 2023-05-31T20:18:02.561 DEBUG:boto:Headers: {} 2023-05-31T20:18:02.561 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:18:02.562 DEBUG:boto:Port: 8000 2023-05-31T20:18:02.562 DEBUG:boto:Params: {} 2023-05-31T20:18:02.562 DEBUG:boto:Token: None 2023-05-31T20:18:02.563 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:18:02 GMT /pklxkd-46/obj7?versionId=null 2023-05-31T20:18:02.564 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:y6rkRw/vAQfTHTDHpOZVAVSCtUs= 2023-05-31T20:18:02.565 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:18:02 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:y6rkRw/vAQfTHTDHpOZVAVSCtUs=', 'Content-Length': '0'} 2023-05-31T20:18:02.567 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:17:25 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000008a80cbca7185e1a5-006477ab7a-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:18:02 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:18:02.568 DEBUG:boto:path=/pklxkd-46/ 2023-05-31T20:18:02.569 DEBUG:boto:auth_path=/pklxkd-46/ 2023-05-31T20:18:02.570 DEBUG:boto:path=/pklxkd-46/?prefix=obj7 2023-05-31T20:18:02.571 DEBUG:boto:auth_path=/pklxkd-46/?prefix=obj7 2023-05-31T20:18:02.571 DEBUG:boto:Method: GET 2023-05-31T20:18:02.571 DEBUG:boto:Path: /pklxkd-46/?prefix=obj7 2023-05-31T20:18:02.572 DEBUG:boto:Data: 2023-05-31T20:18:02.572 DEBUG:boto:Headers: {} 2023-05-31T20:18:02.572 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:18:02.573 DEBUG:boto:Port: 8001 2023-05-31T20:18:02.573 DEBUG:boto:Params: {} 2023-05-31T20:18:02.574 DEBUG:boto:Token: None 2023-05-31T20:18:02.574 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:18:02 GMT /pklxkd-46/ 2023-05-31T20:18:02.574 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:4Td79Wla6G00iJOPln2dNo/Pl/U= 2023-05-31T20:18:02.575 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:18:02 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:4Td79Wla6G00iJOPln2dNo/Pl/U=', 'Content-Length': '0'} 2023-05-31T20:18:02.577 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000052c099707a32477f-006477ab7a-162e-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:18:02 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:18:02.578 DEBUG:boto:b'pklxkd-46obj71000falseobj72023-05-31T20:17:25.673Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:18:02.579 DEBUG:boto:path=/pklxkd-46/ 2023-05-31T20:18:02.580 DEBUG:boto:auth_path=/pklxkd-46/ 2023-05-31T20:18:02.580 DEBUG:boto:path=/pklxkd-46/?prefix=obj7 2023-05-31T20:18:02.580 DEBUG:boto:auth_path=/pklxkd-46/?prefix=obj7 2023-05-31T20:18:02.581 DEBUG:boto:Method: GET 2023-05-31T20:18:02.581 DEBUG:boto:Path: /pklxkd-46/?prefix=obj7 2023-05-31T20:18:02.581 DEBUG:boto:Data: 2023-05-31T20:18:02.582 DEBUG:boto:Headers: {} 2023-05-31T20:18:02.582 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:18:02.583 DEBUG:boto:Port: 8000 2023-05-31T20:18:02.583 DEBUG:boto:Params: {} 2023-05-31T20:18:02.583 DEBUG:boto:Token: None 2023-05-31T20:18:02.584 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:18:02 GMT /pklxkd-46/ 2023-05-31T20:18:02.584 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:4Td79Wla6G00iJOPln2dNo/Pl/U= 2023-05-31T20:18:02.585 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:18:02 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:4Td79Wla6G00iJOPln2dNo/Pl/U=', 'Content-Length': '0'} 2023-05-31T20:18:02.587 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000008f5489fb310db21d-006477ab7a-1630-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:18:02 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:18:02.588 DEBUG:boto:b'pklxkd-46obj71000falseobj72023-05-31T20:17:25.673Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:18:02.589 DEBUG:boto:path=/pklxkd-46/obj7 2023-05-31T20:18:02.590 DEBUG:boto:auth_path=/pklxkd-46/obj7 2023-05-31T20:18:02.590 DEBUG:boto:Method: HEAD 2023-05-31T20:18:02.591 DEBUG:boto:Path: /pklxkd-46/obj7 2023-05-31T20:18:02.592 DEBUG:boto:Data: 2023-05-31T20:18:02.592 DEBUG:boto:Headers: {} 2023-05-31T20:18:02.593 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:18:02.593 DEBUG:boto:Port: 8001 2023-05-31T20:18:02.594 DEBUG:boto:Params: {} 2023-05-31T20:18:02.595 DEBUG:boto:Token: None 2023-05-31T20:18:02.596 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:18:02 GMT /pklxkd-46/obj7 2023-05-31T20:18:02.596 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:oYQahwd37MCBy/UgXl0JS990V/A= 2023-05-31T20:18:02.597 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:18:02 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:oYQahwd37MCBy/UgXl0JS990V/A=', 'Content-Length': '0'} 2023-05-31T20:18:02.599 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:17:25 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000062be9fbc16d90ddf-006477ab7a-162e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:18:02 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:18:02.601 DEBUG:boto:path=/pklxkd-46/obj7 2023-05-31T20:18:02.601 DEBUG:boto:auth_path=/pklxkd-46/obj7 2023-05-31T20:18:02.602 DEBUG:boto:Method: HEAD 2023-05-31T20:18:02.602 DEBUG:boto:Path: /pklxkd-46/obj7 2023-05-31T20:18:02.603 DEBUG:boto:Data: 2023-05-31T20:18:02.604 DEBUG:boto:Headers: {} 2023-05-31T20:18:02.605 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:18:02.605 DEBUG:boto:Port: 8000 2023-05-31T20:18:02.606 DEBUG:boto:Params: {} 2023-05-31T20:18:02.607 DEBUG:boto:Token: None 2023-05-31T20:18:02.608 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:18:02 GMT /pklxkd-46/obj7 2023-05-31T20:18:02.609 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:oYQahwd37MCBy/UgXl0JS990V/A= 2023-05-31T20:18:02.610 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:18:02 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:oYQahwd37MCBy/UgXl0JS990V/A=', 'Content-Length': '0'} 2023-05-31T20:18:02.612 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:17:25 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000090d6f6ecde81079e-006477ab7a-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:18:02 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:18:02.613 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj7 2023-05-31T20:18:02.613 DEBUG:boto:path=/pklxkd-46/obj7 2023-05-31T20:18:02.614 DEBUG:boto:auth_path=/pklxkd-46/obj7 2023-05-31T20:18:02.614 DEBUG:boto:Method: GET 2023-05-31T20:18:02.615 DEBUG:boto:Path: /pklxkd-46/obj7 2023-05-31T20:18:02.615 DEBUG:boto:Data: 2023-05-31T20:18:02.616 DEBUG:boto:Headers: {} 2023-05-31T20:18:02.617 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:18:02.617 DEBUG:boto:Port: 8001 2023-05-31T20:18:02.617 DEBUG:boto:Params: {} 2023-05-31T20:18:02.618 DEBUG:boto:Token: None 2023-05-31T20:18:02.618 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:18:02 GMT /pklxkd-46/obj7 2023-05-31T20:18:02.619 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:Yq3b+BgGIU8IWjS3IGXztx85ac8= 2023-05-31T20:18:02.619 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:18:02 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:Yq3b+BgGIU8IWjS3IGXztx85ac8=', 'Content-Length': '0'} 2023-05-31T20:18:02.621 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:17:25 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000075b20d1e222fe5a3-006477ab7a-162e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:18:02 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:18:02.621 DEBUG:boto:path=/pklxkd-46/obj7 2023-05-31T20:18:02.622 DEBUG:boto:auth_path=/pklxkd-46/obj7 2023-05-31T20:18:02.622 DEBUG:boto:Method: GET 2023-05-31T20:18:02.623 DEBUG:boto:Path: /pklxkd-46/obj7 2023-05-31T20:18:02.623 DEBUG:boto:Data: 2023-05-31T20:18:02.623 DEBUG:boto:Headers: {} 2023-05-31T20:18:02.624 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:18:02.624 DEBUG:boto:Port: 8000 2023-05-31T20:18:02.624 DEBUG:boto:Params: {} 2023-05-31T20:18:02.625 DEBUG:boto:Token: None 2023-05-31T20:18:02.625 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:18:02 GMT /pklxkd-46/obj7 2023-05-31T20:18:02.626 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:Yq3b+BgGIU8IWjS3IGXztx85ac8= 2023-05-31T20:18:02.626 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:18:02 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:Yq3b+BgGIU8IWjS3IGXztx85ac8=', 'Content-Length': '0'} 2023-05-31T20:18:02.628 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:17:25 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000055a46cf8654c07c6-006477ab7a-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:18:02 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:18:02.629 DEBUG:boto:path=/pklxkd-46/ 2023-05-31T20:18:02.629 DEBUG:boto:auth_path=/pklxkd-46/ 2023-05-31T20:18:02.629 DEBUG:boto:path=/pklxkd-46/?prefix=obj7 2023-05-31T20:18:02.630 DEBUG:boto:auth_path=/pklxkd-46/?prefix=obj7 2023-05-31T20:18:02.630 DEBUG:boto:Method: GET 2023-05-31T20:18:02.630 DEBUG:boto:Path: /pklxkd-46/?prefix=obj7 2023-05-31T20:18:02.631 DEBUG:boto:Data: 2023-05-31T20:18:02.631 DEBUG:boto:Headers: {} 2023-05-31T20:18:02.632 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:18:02.632 DEBUG:boto:Port: 8001 2023-05-31T20:18:02.632 DEBUG:boto:Params: {} 2023-05-31T20:18:02.633 DEBUG:boto:Token: None 2023-05-31T20:18:02.633 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:18:02 GMT /pklxkd-46/ 2023-05-31T20:18:02.634 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:4Td79Wla6G00iJOPln2dNo/Pl/U= 2023-05-31T20:18:02.634 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:18:02 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:4Td79Wla6G00iJOPln2dNo/Pl/U=', 'Content-Length': '0'} 2023-05-31T20:18:02.636 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000000ba82b6eecd6a8b1-006477ab7a-162e-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:18:02 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:18:02.680 DEBUG:boto:b'pklxkd-46obj71000falseobj72023-05-31T20:17:25.673Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:18:02.682 DEBUG:boto:path=/pklxkd-46/ 2023-05-31T20:18:02.683 DEBUG:boto:auth_path=/pklxkd-46/ 2023-05-31T20:18:02.684 DEBUG:boto:path=/pklxkd-46/?prefix=obj7 2023-05-31T20:18:02.685 DEBUG:boto:auth_path=/pklxkd-46/?prefix=obj7 2023-05-31T20:18:02.685 DEBUG:boto:Method: GET 2023-05-31T20:18:02.686 DEBUG:boto:Path: /pklxkd-46/?prefix=obj7 2023-05-31T20:18:02.687 DEBUG:boto:Data: 2023-05-31T20:18:02.687 DEBUG:boto:Headers: {} 2023-05-31T20:18:02.688 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:18:02.688 DEBUG:boto:Port: 8000 2023-05-31T20:18:02.688 DEBUG:boto:Params: {} 2023-05-31T20:18:02.689 DEBUG:boto:Token: None 2023-05-31T20:18:02.689 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:18:02 GMT /pklxkd-46/ 2023-05-31T20:18:02.690 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:4Td79Wla6G00iJOPln2dNo/Pl/U= 2023-05-31T20:18:02.690 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:18:02 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:4Td79Wla6G00iJOPln2dNo/Pl/U=', 'Content-Length': '0'} 2023-05-31T20:18:02.693 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ea2f5c226dda0bd2-006477ab7a-1630-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:18:02 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:18:02.694 DEBUG:boto:b'pklxkd-46obj71000falseobj72023-05-31T20:17:25.673Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:18:02.695 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj8 2023-05-31T20:18:02.696 DEBUG:boto:path=/pklxkd-46/obj8 2023-05-31T20:18:02.697 DEBUG:boto:auth_path=/pklxkd-46/obj8 2023-05-31T20:18:02.698 DEBUG:boto:path=/pklxkd-46/obj8?versionId=null 2023-05-31T20:18:02.698 DEBUG:boto:auth_path=/pklxkd-46/obj8?versionId=null 2023-05-31T20:18:02.699 DEBUG:boto:Method: GET 2023-05-31T20:18:02.699 DEBUG:boto:Path: /pklxkd-46/obj8?versionId=null 2023-05-31T20:18:02.700 DEBUG:boto:Data: 2023-05-31T20:18:02.700 DEBUG:boto:Headers: {} 2023-05-31T20:18:02.700 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:18:02.701 DEBUG:boto:Port: 8001 2023-05-31T20:18:02.701 DEBUG:boto:Params: {} 2023-05-31T20:18:02.701 DEBUG:boto:Token: None 2023-05-31T20:18:02.702 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:18:02 GMT /pklxkd-46/obj8?versionId=null 2023-05-31T20:18:02.702 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:YDVPK2gh0J8WI3pSJfFDNuJ08YA= 2023-05-31T20:18:02.703 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:18:02 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:YDVPK2gh0J8WI3pSJfFDNuJ08YA=', 'Content-Length': '0'} 2023-05-31T20:18:02.705 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:17:25 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000005465f3f6915105eb-006477ab7a-162e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:18:02 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:18:02.706 DEBUG:boto:path=/pklxkd-46/obj8 2023-05-31T20:18:02.706 DEBUG:boto:auth_path=/pklxkd-46/obj8 2023-05-31T20:18:02.707 DEBUG:boto:path=/pklxkd-46/obj8?versionId=null 2023-05-31T20:18:02.707 DEBUG:boto:auth_path=/pklxkd-46/obj8?versionId=null 2023-05-31T20:18:02.707 DEBUG:boto:Method: GET 2023-05-31T20:18:02.708 DEBUG:boto:Path: /pklxkd-46/obj8?versionId=null 2023-05-31T20:18:02.708 DEBUG:boto:Data: 2023-05-31T20:18:02.708 DEBUG:boto:Headers: {} 2023-05-31T20:18:02.709 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:18:02.709 DEBUG:boto:Port: 8000 2023-05-31T20:18:02.710 DEBUG:boto:Params: {} 2023-05-31T20:18:02.710 DEBUG:boto:Token: None 2023-05-31T20:18:02.710 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:18:02 GMT /pklxkd-46/obj8?versionId=null 2023-05-31T20:18:02.711 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:YDVPK2gh0J8WI3pSJfFDNuJ08YA= 2023-05-31T20:18:02.712 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:18:02 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:YDVPK2gh0J8WI3pSJfFDNuJ08YA=', 'Content-Length': '0'} 2023-05-31T20:18:02.714 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:17:25 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000d040903a3c7ac4dd-006477ab7a-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:18:02 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:18:02.761 DEBUG:boto:path=/pklxkd-46/obj8 2023-05-31T20:18:02.762 DEBUG:boto:auth_path=/pklxkd-46/obj8 2023-05-31T20:18:02.763 DEBUG:boto:path=/pklxkd-46/obj8?versionId=null 2023-05-31T20:18:02.764 DEBUG:boto:auth_path=/pklxkd-46/obj8?versionId=null 2023-05-31T20:18:02.765 DEBUG:boto:Method: HEAD 2023-05-31T20:18:02.765 DEBUG:boto:Path: /pklxkd-46/obj8?versionId=null 2023-05-31T20:18:02.765 DEBUG:boto:Data: 2023-05-31T20:18:02.766 DEBUG:boto:Headers: {} 2023-05-31T20:18:02.767 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:18:02.768 DEBUG:boto:Port: 8001 2023-05-31T20:18:02.769 DEBUG:boto:Params: {} 2023-05-31T20:18:02.769 DEBUG:boto:Token: None 2023-05-31T20:18:02.770 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:18:02 GMT /pklxkd-46/obj8?versionId=null 2023-05-31T20:18:02.771 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:HExlhwxV3sxKi4gGUawkUCLUXLI= 2023-05-31T20:18:02.772 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:18:02 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:HExlhwxV3sxKi4gGUawkUCLUXLI=', 'Content-Length': '0'} 2023-05-31T20:18:02.774 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:17:25 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000045ec0b5400089099-006477ab7a-162e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:18:02 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:18:02.775 DEBUG:boto:path=/pklxkd-46/obj8 2023-05-31T20:18:02.776 DEBUG:boto:auth_path=/pklxkd-46/obj8 2023-05-31T20:18:02.777 DEBUG:boto:path=/pklxkd-46/obj8?versionId=null 2023-05-31T20:18:02.777 DEBUG:boto:auth_path=/pklxkd-46/obj8?versionId=null 2023-05-31T20:18:02.777 DEBUG:boto:Method: HEAD 2023-05-31T20:18:02.778 DEBUG:boto:Path: /pklxkd-46/obj8?versionId=null 2023-05-31T20:18:02.778 DEBUG:boto:Data: 2023-05-31T20:18:02.778 DEBUG:boto:Headers: {} 2023-05-31T20:18:02.779 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:18:02.779 DEBUG:boto:Port: 8000 2023-05-31T20:18:02.779 DEBUG:boto:Params: {} 2023-05-31T20:18:02.780 DEBUG:boto:Token: None 2023-05-31T20:18:02.781 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:18:02 GMT /pklxkd-46/obj8?versionId=null 2023-05-31T20:18:02.781 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:HExlhwxV3sxKi4gGUawkUCLUXLI= 2023-05-31T20:18:02.782 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:18:02 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:HExlhwxV3sxKi4gGUawkUCLUXLI=', 'Content-Length': '0'} 2023-05-31T20:18:02.784 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:17:25 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000ebe594152b27512c-006477ab7a-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:18:02 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:18:02.785 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj8 2023-05-31T20:18:02.786 DEBUG:boto:path=/pklxkd-46/obj8 2023-05-31T20:18:02.787 DEBUG:boto:auth_path=/pklxkd-46/obj8 2023-05-31T20:18:02.788 DEBUG:boto:path=/pklxkd-46/obj8?versionId=null 2023-05-31T20:18:02.788 DEBUG:boto:auth_path=/pklxkd-46/obj8?versionId=null 2023-05-31T20:18:02.789 DEBUG:boto:Method: GET 2023-05-31T20:18:02.790 DEBUG:boto:Path: /pklxkd-46/obj8?versionId=null 2023-05-31T20:18:02.791 DEBUG:boto:Data: 2023-05-31T20:18:02.792 DEBUG:boto:Headers: {} 2023-05-31T20:18:02.792 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:18:02.793 DEBUG:boto:Port: 8001 2023-05-31T20:18:02.794 DEBUG:boto:Params: {} 2023-05-31T20:18:02.795 DEBUG:boto:Token: None 2023-05-31T20:18:02.796 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:18:02 GMT /pklxkd-46/obj8?versionId=null 2023-05-31T20:18:02.797 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:YDVPK2gh0J8WI3pSJfFDNuJ08YA= 2023-05-31T20:18:02.797 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:18:02 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:YDVPK2gh0J8WI3pSJfFDNuJ08YA=', 'Content-Length': '0'} 2023-05-31T20:18:02.800 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:17:25 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000004652f8f449622142-006477ab7a-162e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:18:02 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:18:02.801 DEBUG:boto:path=/pklxkd-46/obj8 2023-05-31T20:18:02.802 DEBUG:boto:auth_path=/pklxkd-46/obj8 2023-05-31T20:18:02.803 DEBUG:boto:path=/pklxkd-46/obj8?versionId=null 2023-05-31T20:18:02.804 DEBUG:boto:auth_path=/pklxkd-46/obj8?versionId=null 2023-05-31T20:18:02.805 DEBUG:boto:Method: GET 2023-05-31T20:18:02.806 DEBUG:boto:Path: /pklxkd-46/obj8?versionId=null 2023-05-31T20:18:02.806 DEBUG:boto:Data: 2023-05-31T20:18:02.807 DEBUG:boto:Headers: {} 2023-05-31T20:18:02.807 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:18:02.808 DEBUG:boto:Port: 8000 2023-05-31T20:18:02.808 DEBUG:boto:Params: {} 2023-05-31T20:18:02.809 DEBUG:boto:Token: None 2023-05-31T20:18:02.810 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:18:02 GMT /pklxkd-46/obj8?versionId=null 2023-05-31T20:18:02.812 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:YDVPK2gh0J8WI3pSJfFDNuJ08YA= 2023-05-31T20:18:02.813 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:18:02 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:YDVPK2gh0J8WI3pSJfFDNuJ08YA=', 'Content-Length': '0'} 2023-05-31T20:18:02.816 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:17:25 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000081ce5b039e3b6a33-006477ab7a-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:18:02 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:18:02.817 DEBUG:boto:path=/pklxkd-46/ 2023-05-31T20:18:02.818 DEBUG:boto:auth_path=/pklxkd-46/ 2023-05-31T20:18:02.818 DEBUG:boto:path=/pklxkd-46/?prefix=obj8 2023-05-31T20:18:02.819 DEBUG:boto:auth_path=/pklxkd-46/?prefix=obj8 2023-05-31T20:18:02.820 DEBUG:boto:Method: GET 2023-05-31T20:18:02.820 DEBUG:boto:Path: /pklxkd-46/?prefix=obj8 2023-05-31T20:18:02.821 DEBUG:boto:Data: 2023-05-31T20:18:02.821 DEBUG:boto:Headers: {} 2023-05-31T20:18:02.821 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:18:02.822 DEBUG:boto:Port: 8001 2023-05-31T20:18:02.822 DEBUG:boto:Params: {} 2023-05-31T20:18:02.823 DEBUG:boto:Token: None 2023-05-31T20:18:02.823 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:18:02 GMT /pklxkd-46/ 2023-05-31T20:18:02.823 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:4Td79Wla6G00iJOPln2dNo/Pl/U= 2023-05-31T20:18:02.824 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:18:02 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:4Td79Wla6G00iJOPln2dNo/Pl/U=', 'Content-Length': '0'} 2023-05-31T20:18:02.826 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000008fa86306b15b5246-006477ab7a-162e-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:18:02 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:18:02.827 DEBUG:boto:b'pklxkd-46obj81000falseobj82023-05-31T20:17:25.695Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:18:02.828 DEBUG:boto:path=/pklxkd-46/ 2023-05-31T20:18:02.828 DEBUG:boto:auth_path=/pklxkd-46/ 2023-05-31T20:18:02.829 DEBUG:boto:path=/pklxkd-46/?prefix=obj8 2023-05-31T20:18:02.829 DEBUG:boto:auth_path=/pklxkd-46/?prefix=obj8 2023-05-31T20:18:02.829 DEBUG:boto:Method: GET 2023-05-31T20:18:02.830 DEBUG:boto:Path: /pklxkd-46/?prefix=obj8 2023-05-31T20:18:02.830 DEBUG:boto:Data: 2023-05-31T20:18:02.830 DEBUG:boto:Headers: {} 2023-05-31T20:18:02.831 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:18:02.831 DEBUG:boto:Port: 8000 2023-05-31T20:18:02.831 DEBUG:boto:Params: {} 2023-05-31T20:18:02.832 DEBUG:boto:Token: None 2023-05-31T20:18:02.832 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:18:02 GMT /pklxkd-46/ 2023-05-31T20:18:02.833 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:4Td79Wla6G00iJOPln2dNo/Pl/U= 2023-05-31T20:18:02.833 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:18:02 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:4Td79Wla6G00iJOPln2dNo/Pl/U=', 'Content-Length': '0'} 2023-05-31T20:18:02.836 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ad0b17103204994f-006477ab7a-1630-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:18:02 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:18:02.836 DEBUG:boto:b'pklxkd-46obj81000falseobj82023-05-31T20:17:25.695Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:18:02.837 DEBUG:boto:path=/pklxkd-46/obj8 2023-05-31T20:18:02.837 DEBUG:boto:auth_path=/pklxkd-46/obj8 2023-05-31T20:18:02.838 DEBUG:boto:Method: HEAD 2023-05-31T20:18:02.838 DEBUG:boto:Path: /pklxkd-46/obj8 2023-05-31T20:18:02.839 DEBUG:boto:Data: 2023-05-31T20:18:02.839 DEBUG:boto:Headers: {} 2023-05-31T20:18:02.839 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:18:02.840 DEBUG:boto:Port: 8001 2023-05-31T20:18:02.840 DEBUG:boto:Params: {} 2023-05-31T20:18:02.841 DEBUG:boto:Token: None 2023-05-31T20:18:02.841 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:18:02 GMT /pklxkd-46/obj8 2023-05-31T20:18:02.842 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:rT7ndPmztRVbpMszfU/mqSkD65A= 2023-05-31T20:18:02.842 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:18:02 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:rT7ndPmztRVbpMszfU/mqSkD65A=', 'Content-Length': '0'} 2023-05-31T20:18:02.844 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:17:25 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000081790b7c5de01630-006477ab7a-162e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:18:02 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:18:02.845 DEBUG:boto:path=/pklxkd-46/obj8 2023-05-31T20:18:02.846 DEBUG:boto:auth_path=/pklxkd-46/obj8 2023-05-31T20:18:02.847 DEBUG:boto:Method: HEAD 2023-05-31T20:18:02.848 DEBUG:boto:Path: /pklxkd-46/obj8 2023-05-31T20:18:02.849 DEBUG:boto:Data: 2023-05-31T20:18:02.850 DEBUG:boto:Headers: {} 2023-05-31T20:18:02.851 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:18:02.851 DEBUG:boto:Port: 8000 2023-05-31T20:18:02.852 DEBUG:boto:Params: {} 2023-05-31T20:18:02.853 DEBUG:boto:Token: None 2023-05-31T20:18:02.854 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:18:02 GMT /pklxkd-46/obj8 2023-05-31T20:18:02.854 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:rT7ndPmztRVbpMszfU/mqSkD65A= 2023-05-31T20:18:02.855 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:18:02 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:rT7ndPmztRVbpMszfU/mqSkD65A=', 'Content-Length': '0'} 2023-05-31T20:18:02.857 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:17:25 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000001884333b3c6e9c7f-006477ab7a-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:18:02 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:18:02.857 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj8 2023-05-31T20:18:02.858 DEBUG:boto:path=/pklxkd-46/obj8 2023-05-31T20:18:02.858 DEBUG:boto:auth_path=/pklxkd-46/obj8 2023-05-31T20:18:02.859 DEBUG:boto:Method: GET 2023-05-31T20:18:02.859 DEBUG:boto:Path: /pklxkd-46/obj8 2023-05-31T20:18:02.859 DEBUG:boto:Data: 2023-05-31T20:18:02.860 DEBUG:boto:Headers: {} 2023-05-31T20:18:02.860 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:18:02.861 DEBUG:boto:Port: 8001 2023-05-31T20:18:02.861 DEBUG:boto:Params: {} 2023-05-31T20:18:02.861 DEBUG:boto:Token: None 2023-05-31T20:18:02.862 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:18:02 GMT /pklxkd-46/obj8 2023-05-31T20:18:02.862 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:CAlRa/lF/CQ672ftJm5Vr5oW/SY= 2023-05-31T20:18:02.863 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:18:02 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:CAlRa/lF/CQ672ftJm5Vr5oW/SY=', 'Content-Length': '0'} 2023-05-31T20:18:02.864 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:17:25 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000f4e38e0ebb7c8aaa-006477ab7a-162e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:18:02 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:18:02.865 DEBUG:boto:path=/pklxkd-46/obj8 2023-05-31T20:18:02.866 DEBUG:boto:auth_path=/pklxkd-46/obj8 2023-05-31T20:18:02.866 DEBUG:boto:Method: GET 2023-05-31T20:18:02.866 DEBUG:boto:Path: /pklxkd-46/obj8 2023-05-31T20:18:02.867 DEBUG:boto:Data: 2023-05-31T20:18:02.867 DEBUG:boto:Headers: {} 2023-05-31T20:18:02.868 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:18:02.868 DEBUG:boto:Port: 8000 2023-05-31T20:18:02.868 DEBUG:boto:Params: {} 2023-05-31T20:18:02.869 DEBUG:boto:Token: None 2023-05-31T20:18:02.869 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:18:02 GMT /pklxkd-46/obj8 2023-05-31T20:18:02.870 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:CAlRa/lF/CQ672ftJm5Vr5oW/SY= 2023-05-31T20:18:02.870 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:18:02 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:CAlRa/lF/CQ672ftJm5Vr5oW/SY=', 'Content-Length': '0'} 2023-05-31T20:18:02.872 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:17:25 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000000f75c5643c0d8c69-006477ab7a-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:18:02 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:18:02.873 DEBUG:boto:path=/pklxkd-46/ 2023-05-31T20:18:02.874 DEBUG:boto:auth_path=/pklxkd-46/ 2023-05-31T20:18:02.874 DEBUG:boto:path=/pklxkd-46/?prefix=obj8 2023-05-31T20:18:02.874 DEBUG:boto:auth_path=/pklxkd-46/?prefix=obj8 2023-05-31T20:18:02.875 DEBUG:boto:Method: GET 2023-05-31T20:18:02.875 DEBUG:boto:Path: /pklxkd-46/?prefix=obj8 2023-05-31T20:18:02.875 DEBUG:boto:Data: 2023-05-31T20:18:02.876 DEBUG:boto:Headers: {} 2023-05-31T20:18:02.876 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:18:02.877 DEBUG:boto:Port: 8001 2023-05-31T20:18:02.877 DEBUG:boto:Params: {} 2023-05-31T20:18:02.878 DEBUG:boto:Token: None 2023-05-31T20:18:02.878 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:18:02 GMT /pklxkd-46/ 2023-05-31T20:18:02.879 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:4Td79Wla6G00iJOPln2dNo/Pl/U= 2023-05-31T20:18:02.879 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:18:02 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:4Td79Wla6G00iJOPln2dNo/Pl/U=', 'Content-Length': '0'} 2023-05-31T20:18:02.881 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000008c410889d22d714c-006477ab7a-162e-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:18:02 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:18:02.882 DEBUG:boto:b'pklxkd-46obj81000falseobj82023-05-31T20:17:25.695Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:18:02.883 DEBUG:boto:path=/pklxkd-46/ 2023-05-31T20:18:02.883 DEBUG:boto:auth_path=/pklxkd-46/ 2023-05-31T20:18:02.884 DEBUG:boto:path=/pklxkd-46/?prefix=obj8 2023-05-31T20:18:02.884 DEBUG:boto:auth_path=/pklxkd-46/?prefix=obj8 2023-05-31T20:18:02.885 DEBUG:boto:Method: GET 2023-05-31T20:18:02.885 DEBUG:boto:Path: /pklxkd-46/?prefix=obj8 2023-05-31T20:18:02.886 DEBUG:boto:Data: 2023-05-31T20:18:02.886 DEBUG:boto:Headers: {} 2023-05-31T20:18:02.886 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:18:02.887 DEBUG:boto:Port: 8000 2023-05-31T20:18:02.887 DEBUG:boto:Params: {} 2023-05-31T20:18:02.888 DEBUG:boto:Token: None 2023-05-31T20:18:02.888 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:18:02 GMT /pklxkd-46/ 2023-05-31T20:18:02.889 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:4Td79Wla6G00iJOPln2dNo/Pl/U= 2023-05-31T20:18:02.889 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:18:02 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:4Td79Wla6G00iJOPln2dNo/Pl/U=', 'Content-Length': '0'} 2023-05-31T20:18:02.891 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000016959083102f27ae-006477ab7a-1630-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:18:02 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:18:02.936 DEBUG:boto:b'pklxkd-46obj81000falseobj82023-05-31T20:17:25.695Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:18:02.937 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=pklxkd-46 zones={a2, a1} 2023-05-31T20:18:02.938 DEBUG:rgw_multi.tests:bucket checkpoint: source=a1 target=a2 bucket=pklxkd-47 2023-05-31T20:18:02.938 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'pklxkd-47', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-05-31T20:18:02.938 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'pklxkd-47', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-05-31T20:18:02.939 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket pklxkd-47 --source-zone a1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2023-05-31T20:18:03.004 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2023-05-31T20:18:03.005 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2023-05-31T20:18:03.061 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:18:03.055+0000 7f9ee2d24780 1 bucket sync caught up with source: 2023-05-31T20:18:03.062 INFO:teuthology.orchestra.run.smithi099.stderr: local status: [00000000004.25.6, 00000000002.16.1, 00000000004.28.6, 00000000002.14.1, 00000000004.30.6, 00000000002.15.1, 00000000002.24.1, 00000000004.101.6, 00000000002.19.1, 00000000002.99.1, 00000000002.27.1] 2023-05-31T20:18:03.062 INFO:teuthology.orchestra.run.smithi099.stderr: remote markers: [00000000004.25.6, 00000000002.16.1, 00000000004.28.6, 00000000002.14.1, 00000000004.30.6, 00000000002.15.1, 00000000002.24.1, 00000000004.101.6, 00000000002.19.1, 00000000002.99.1, 00000000002.27.1] 2023-05-31T20:18:03.062 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:18:03.055+0000 7f9ee2d24780 0 bucket checkpoint complete 2023-05-31T20:18:03.082 DEBUG:rgw_multi.tests:bucket checkpoint: source=a2 target=a1 bucket=pklxkd-47 2023-05-31T20:18:03.083 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'pklxkd-47', '--source-zone', 'a2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-05-31T20:18:03.083 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'pklxkd-47', '--source-zone', 'a2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-05-31T20:18:03.084 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket pklxkd-47 --source-zone a2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2023-05-31T20:18:03.140 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2023-05-31T20:18:03.141 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2023-05-31T20:18:03.188 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:18:03.182+0000 7f3e5359d780 1 bucket sync caught up with source: 2023-05-31T20:18:03.189 INFO:teuthology.orchestra.run.smithi099.stderr: local status: [00000000002.734.6, , 00000000002.1960.6, , 00000000002.3537.6, , , 00000000002.2393.6, , , ] 2023-05-31T20:18:03.189 INFO:teuthology.orchestra.run.smithi099.stderr: remote markers: [00000000002.734.6, , 00000000002.1960.6, , 00000000002.3537.6, , , 00000000002.2393.6, , , ] 2023-05-31T20:18:03.189 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:18:03.182+0000 7f3e5359d780 0 bucket checkpoint complete 2023-05-31T20:18:03.201 INFO:tasks.rgw_multi.zone_rados:comparing bucket=pklxkd-47 zones={a2, a1} 2023-05-31T20:18:03.201 DEBUG:boto:path=/pklxkd-47/ 2023-05-31T20:18:03.202 DEBUG:boto:auth_path=/pklxkd-47/ 2023-05-31T20:18:03.202 DEBUG:boto:Method: HEAD 2023-05-31T20:18:03.203 DEBUG:boto:Path: /pklxkd-47/ 2023-05-31T20:18:03.203 DEBUG:boto:Data: 2023-05-31T20:18:03.204 DEBUG:boto:Headers: {} 2023-05-31T20:18:03.204 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:18:03.205 DEBUG:boto:Port: 8001 2023-05-31T20:18:03.205 DEBUG:boto:Params: {} 2023-05-31T20:18:03.206 DEBUG:boto:Token: None 2023-05-31T20:18:03.206 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:18:03 GMT /pklxkd-47/ 2023-05-31T20:18:03.207 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:sPdkMdwGA7xy+TeH7o361I1XdHE= 2023-05-31T20:18:03.207 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:18:03 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:sPdkMdwGA7xy+TeH7o361I1XdHE=', 'Content-Length': '0'} 2023-05-31T20:18:03.210 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '8'), ('X-RGW-Bytes-Used', '48'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000356becf7ab7621c8-006477ab7b-162e-a2'), ('Content-Length', '0'), ('Date', 'Wed, 31 May 2023 20:18:03 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:18:03.210 DEBUG:boto:path=/pklxkd-47/ 2023-05-31T20:18:03.211 DEBUG:boto:auth_path=/pklxkd-47/ 2023-05-31T20:18:03.211 DEBUG:boto:Method: HEAD 2023-05-31T20:18:03.211 DEBUG:boto:Path: /pklxkd-47/ 2023-05-31T20:18:03.212 DEBUG:boto:Data: 2023-05-31T20:18:03.212 DEBUG:boto:Headers: {} 2023-05-31T20:18:03.212 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:18:03.213 DEBUG:boto:Port: 8000 2023-05-31T20:18:03.213 DEBUG:boto:Params: {} 2023-05-31T20:18:03.214 DEBUG:boto:Token: None 2023-05-31T20:18:03.214 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:18:03 GMT /pklxkd-47/ 2023-05-31T20:18:03.214 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:sPdkMdwGA7xy+TeH7o361I1XdHE= 2023-05-31T20:18:03.215 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:18:03 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:sPdkMdwGA7xy+TeH7o361I1XdHE=', 'Content-Length': '0'} 2023-05-31T20:18:03.217 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '8'), ('X-RGW-Bytes-Used', '48'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000f5bb962d06163e33-006477ab7b-1630-a1'), ('Content-Length', '0'), ('Date', 'Wed, 31 May 2023 20:18:03 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:18:03.218 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2023-05-31T20:18:03.218 DEBUG:boto:path=/pklxkd-47/ 2023-05-31T20:18:03.218 DEBUG:boto:auth_path=/pklxkd-47/ 2023-05-31T20:18:03.219 DEBUG:boto:path=/pklxkd-47/?versions&max-keys=999 2023-05-31T20:18:03.219 DEBUG:boto:auth_path=/pklxkd-47/?versions&max-keys=999 2023-05-31T20:18:03.220 DEBUG:boto:Method: GET 2023-05-31T20:18:03.220 DEBUG:boto:Path: /pklxkd-47/?versions&max-keys=999 2023-05-31T20:18:03.220 DEBUG:boto:Data: 2023-05-31T20:18:03.221 DEBUG:boto:Headers: {} 2023-05-31T20:18:03.221 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:18:03.221 DEBUG:boto:Port: 8001 2023-05-31T20:18:03.222 DEBUG:boto:Params: {} 2023-05-31T20:18:03.222 DEBUG:boto:Token: None 2023-05-31T20:18:03.223 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:18:03 GMT /pklxkd-47/?versions 2023-05-31T20:18:03.223 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:2/3fZ/zOI1UzmVP5TqA7XgRX6L0= 2023-05-31T20:18:03.223 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:18:03 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:2/3fZ/zOI1UzmVP5TqA7XgRX6L0=', 'Content-Length': '0'} 2023-05-31T20:18:03.226 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b4aa80cef935ef57-006477ab7b-162e-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:18:03 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:18:03.227 DEBUG:boto:b'pklxkd-47999falseobj1nulltrue2023-05-31T20:15:11.346Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj2nulltrue2023-05-31T20:15:11.368Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj3nulltrue2023-05-31T20:15:11.402Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj4nulltrue2023-05-31T20:15:11.471Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj5nulltrue2023-05-31T20:17:25.752Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj6nulltrue2023-05-31T20:17:25.769Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj7nulltrue2023-05-31T20:17:25.833Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj8nulltrue2023-05-31T20:17:25.854Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:18:03.227 DEBUG:tasks.rgw_multi.zone_rados:o=obj1 2023-05-31T20:18:03.228 DEBUG:tasks.rgw_multi.zone_rados:o=obj2 2023-05-31T20:18:03.228 DEBUG:tasks.rgw_multi.zone_rados:o=obj3 2023-05-31T20:18:03.229 DEBUG:tasks.rgw_multi.zone_rados:o=obj4 2023-05-31T20:18:03.229 DEBUG:tasks.rgw_multi.zone_rados:o=obj5 2023-05-31T20:18:03.229 DEBUG:tasks.rgw_multi.zone_rados:o=obj6 2023-05-31T20:18:03.230 DEBUG:tasks.rgw_multi.zone_rados:o=obj7 2023-05-31T20:18:03.230 DEBUG:tasks.rgw_multi.zone_rados:o=obj8 2023-05-31T20:18:03.230 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2023-05-31T20:18:03.231 DEBUG:boto:path=/pklxkd-47/ 2023-05-31T20:18:03.231 DEBUG:boto:auth_path=/pklxkd-47/ 2023-05-31T20:18:03.232 DEBUG:boto:path=/pklxkd-47/?versions&max-keys=999 2023-05-31T20:18:03.232 DEBUG:boto:auth_path=/pklxkd-47/?versions&max-keys=999 2023-05-31T20:18:03.232 DEBUG:boto:Method: GET 2023-05-31T20:18:03.233 DEBUG:boto:Path: /pklxkd-47/?versions&max-keys=999 2023-05-31T20:18:03.233 DEBUG:boto:Data: 2023-05-31T20:18:03.233 DEBUG:boto:Headers: {} 2023-05-31T20:18:03.234 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:18:03.235 DEBUG:boto:Port: 8000 2023-05-31T20:18:03.235 DEBUG:boto:Params: {} 2023-05-31T20:18:03.236 DEBUG:boto:Token: None 2023-05-31T20:18:03.236 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:18:03 GMT /pklxkd-47/?versions 2023-05-31T20:18:03.237 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:2/3fZ/zOI1UzmVP5TqA7XgRX6L0= 2023-05-31T20:18:03.237 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:18:03 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:2/3fZ/zOI1UzmVP5TqA7XgRX6L0=', 'Content-Length': '0'} 2023-05-31T20:18:03.240 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000409d485fa98112fa-006477ab7b-1630-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:18:03 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:18:03.240 DEBUG:boto:b'pklxkd-47999falseobj1nulltrue2023-05-31T20:15:11.346Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj2nulltrue2023-05-31T20:15:11.368Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj3nulltrue2023-05-31T20:15:11.402Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj4nulltrue2023-05-31T20:15:11.471Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj5nulltrue2023-05-31T20:17:25.752Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj6nulltrue2023-05-31T20:17:25.769Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj7nulltrue2023-05-31T20:17:25.833Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj8nulltrue2023-05-31T20:17:25.854Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:18:03.241 DEBUG:tasks.rgw_multi.zone_rados:o=obj1 2023-05-31T20:18:03.242 DEBUG:tasks.rgw_multi.zone_rados:o=obj2 2023-05-31T20:18:03.242 DEBUG:tasks.rgw_multi.zone_rados:o=obj3 2023-05-31T20:18:03.243 DEBUG:tasks.rgw_multi.zone_rados:o=obj4 2023-05-31T20:18:03.243 DEBUG:tasks.rgw_multi.zone_rados:o=obj5 2023-05-31T20:18:03.243 DEBUG:tasks.rgw_multi.zone_rados:o=obj6 2023-05-31T20:18:03.244 DEBUG:tasks.rgw_multi.zone_rados:o=obj7 2023-05-31T20:18:03.244 DEBUG:tasks.rgw_multi.zone_rados:o=obj8 2023-05-31T20:18:03.244 DEBUG:boto:path=/pklxkd-47/ 2023-05-31T20:18:03.245 DEBUG:boto:auth_path=/pklxkd-47/ 2023-05-31T20:18:03.245 DEBUG:boto:path=/pklxkd-47/?versions&max-keys=999 2023-05-31T20:18:03.245 DEBUG:boto:auth_path=/pklxkd-47/?versions&max-keys=999 2023-05-31T20:18:03.246 DEBUG:boto:Method: GET 2023-05-31T20:18:03.246 DEBUG:boto:Path: /pklxkd-47/?versions&max-keys=999 2023-05-31T20:18:03.246 DEBUG:boto:Data: 2023-05-31T20:18:03.247 DEBUG:boto:Headers: {} 2023-05-31T20:18:03.247 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:18:03.247 DEBUG:boto:Port: 8001 2023-05-31T20:18:03.248 DEBUG:boto:Params: {} 2023-05-31T20:18:03.248 DEBUG:boto:Token: None 2023-05-31T20:18:03.249 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:18:03 GMT /pklxkd-47/?versions 2023-05-31T20:18:03.249 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:2/3fZ/zOI1UzmVP5TqA7XgRX6L0= 2023-05-31T20:18:03.249 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:18:03 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:2/3fZ/zOI1UzmVP5TqA7XgRX6L0=', 'Content-Length': '0'} 2023-05-31T20:18:03.252 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000007972126cfa68df52-006477ab7b-162e-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:18:03 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:18:03.252 DEBUG:boto:b'pklxkd-47999falseobj1nulltrue2023-05-31T20:15:11.346Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj2nulltrue2023-05-31T20:15:11.368Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj3nulltrue2023-05-31T20:15:11.402Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj4nulltrue2023-05-31T20:15:11.471Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj5nulltrue2023-05-31T20:17:25.752Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj6nulltrue2023-05-31T20:17:25.769Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj7nulltrue2023-05-31T20:17:25.833Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj8nulltrue2023-05-31T20:17:25.854Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:18:03.253 DEBUG:boto:path=/pklxkd-47/ 2023-05-31T20:18:03.253 DEBUG:boto:auth_path=/pklxkd-47/ 2023-05-31T20:18:03.254 DEBUG:boto:path=/pklxkd-47/?versions&max-keys=999 2023-05-31T20:18:03.254 DEBUG:boto:auth_path=/pklxkd-47/?versions&max-keys=999 2023-05-31T20:18:03.255 DEBUG:boto:Method: GET 2023-05-31T20:18:03.255 DEBUG:boto:Path: /pklxkd-47/?versions&max-keys=999 2023-05-31T20:18:03.255 DEBUG:boto:Data: 2023-05-31T20:18:03.256 DEBUG:boto:Headers: {} 2023-05-31T20:18:03.256 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:18:03.256 DEBUG:boto:Port: 8000 2023-05-31T20:18:03.257 DEBUG:boto:Params: {} 2023-05-31T20:18:03.257 DEBUG:boto:Token: None 2023-05-31T20:18:03.257 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:18:03 GMT /pklxkd-47/?versions 2023-05-31T20:18:03.258 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:2/3fZ/zOI1UzmVP5TqA7XgRX6L0= 2023-05-31T20:18:03.258 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:18:03 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:2/3fZ/zOI1UzmVP5TqA7XgRX6L0=', 'Content-Length': '0'} 2023-05-31T20:18:03.261 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e0b5d31e7f9b5af2-006477ab7b-1630-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:18:03 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:18:03.261 DEBUG:boto:b'pklxkd-47999falseobj1nulltrue2023-05-31T20:15:11.346Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj2nulltrue2023-05-31T20:15:11.368Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj3nulltrue2023-05-31T20:15:11.402Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj4nulltrue2023-05-31T20:15:11.471Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj5nulltrue2023-05-31T20:17:25.752Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj6nulltrue2023-05-31T20:17:25.769Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj7nulltrue2023-05-31T20:17:25.833Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj8nulltrue2023-05-31T20:17:25.854Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:18:03.262 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2023-05-31T20:18:03.263 DEBUG:boto:path=/pklxkd-47/obj1 2023-05-31T20:18:03.263 DEBUG:boto:auth_path=/pklxkd-47/obj1 2023-05-31T20:18:03.264 DEBUG:boto:path=/pklxkd-47/obj1?versionId=null 2023-05-31T20:18:03.264 DEBUG:boto:auth_path=/pklxkd-47/obj1?versionId=null 2023-05-31T20:18:03.264 DEBUG:boto:Method: GET 2023-05-31T20:18:03.265 DEBUG:boto:Path: /pklxkd-47/obj1?versionId=null 2023-05-31T20:18:03.265 DEBUG:boto:Data: 2023-05-31T20:18:03.265 DEBUG:boto:Headers: {} 2023-05-31T20:18:03.266 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:18:03.266 DEBUG:boto:Port: 8001 2023-05-31T20:18:03.267 DEBUG:boto:Params: {} 2023-05-31T20:18:03.267 DEBUG:boto:Token: None 2023-05-31T20:18:03.267 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:18:03 GMT /pklxkd-47/obj1?versionId=null 2023-05-31T20:18:03.268 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:rWwewHe9JKxuVKVDTt4aQehUEhI= 2023-05-31T20:18:03.268 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:18:03 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:rWwewHe9JKxuVKVDTt4aQehUEhI=', 'Content-Length': '0'} 2023-05-31T20:18:03.270 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:15:11 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000e73b9d1bd8a43bc4-006477ab7b-162e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:18:03 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:18:03.312 DEBUG:boto:path=/pklxkd-47/obj1 2023-05-31T20:18:03.313 DEBUG:boto:auth_path=/pklxkd-47/obj1 2023-05-31T20:18:03.314 DEBUG:boto:path=/pklxkd-47/obj1?versionId=null 2023-05-31T20:18:03.315 DEBUG:boto:auth_path=/pklxkd-47/obj1?versionId=null 2023-05-31T20:18:03.315 DEBUG:boto:Method: GET 2023-05-31T20:18:03.316 DEBUG:boto:Path: /pklxkd-47/obj1?versionId=null 2023-05-31T20:18:03.317 DEBUG:boto:Data: 2023-05-31T20:18:03.317 DEBUG:boto:Headers: {} 2023-05-31T20:18:03.318 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:18:03.318 DEBUG:boto:Port: 8000 2023-05-31T20:18:03.319 DEBUG:boto:Params: {} 2023-05-31T20:18:03.319 DEBUG:boto:Token: None 2023-05-31T20:18:03.319 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:18:03 GMT /pklxkd-47/obj1?versionId=null 2023-05-31T20:18:03.320 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:rWwewHe9JKxuVKVDTt4aQehUEhI= 2023-05-31T20:18:03.321 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:18:03 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:rWwewHe9JKxuVKVDTt4aQehUEhI=', 'Content-Length': '0'} 2023-05-31T20:18:03.322 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:15:11 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000f641ee096be179fa-006477ab7b-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:18:03 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:18:03.323 DEBUG:boto:path=/pklxkd-47/obj1 2023-05-31T20:18:03.324 DEBUG:boto:auth_path=/pklxkd-47/obj1 2023-05-31T20:18:03.326 DEBUG:boto:path=/pklxkd-47/obj1?versionId=null 2023-05-31T20:18:03.327 DEBUG:boto:auth_path=/pklxkd-47/obj1?versionId=null 2023-05-31T20:18:03.328 DEBUG:boto:Method: HEAD 2023-05-31T20:18:03.328 DEBUG:boto:Path: /pklxkd-47/obj1?versionId=null 2023-05-31T20:18:03.329 DEBUG:boto:Data: 2023-05-31T20:18:03.330 DEBUG:boto:Headers: {} 2023-05-31T20:18:03.331 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:18:03.332 DEBUG:boto:Port: 8001 2023-05-31T20:18:03.333 DEBUG:boto:Params: {} 2023-05-31T20:18:03.334 DEBUG:boto:Token: None 2023-05-31T20:18:03.335 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:18:03 GMT /pklxkd-47/obj1?versionId=null 2023-05-31T20:18:03.335 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:n5gRYbkAs2XxoPxJDgD5xAa7AVA= 2023-05-31T20:18:03.336 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:18:03 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:n5gRYbkAs2XxoPxJDgD5xAa7AVA=', 'Content-Length': '0'} 2023-05-31T20:18:03.337 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:15:11 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000c72969a9c30da445-006477ab7b-162e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:18:03 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:18:03.338 DEBUG:boto:path=/pklxkd-47/obj1 2023-05-31T20:18:03.339 DEBUG:boto:auth_path=/pklxkd-47/obj1 2023-05-31T20:18:03.339 DEBUG:boto:path=/pklxkd-47/obj1?versionId=null 2023-05-31T20:18:03.339 DEBUG:boto:auth_path=/pklxkd-47/obj1?versionId=null 2023-05-31T20:18:03.340 DEBUG:boto:Method: HEAD 2023-05-31T20:18:03.340 DEBUG:boto:Path: /pklxkd-47/obj1?versionId=null 2023-05-31T20:18:03.341 DEBUG:boto:Data: 2023-05-31T20:18:03.341 DEBUG:boto:Headers: {} 2023-05-31T20:18:03.342 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:18:03.343 DEBUG:boto:Port: 8000 2023-05-31T20:18:03.344 DEBUG:boto:Params: {} 2023-05-31T20:18:03.345 DEBUG:boto:Token: None 2023-05-31T20:18:03.346 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:18:03 GMT /pklxkd-47/obj1?versionId=null 2023-05-31T20:18:03.347 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:n5gRYbkAs2XxoPxJDgD5xAa7AVA= 2023-05-31T20:18:03.348 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:18:03 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:n5gRYbkAs2XxoPxJDgD5xAa7AVA=', 'Content-Length': '0'} 2023-05-31T20:18:03.350 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:15:11 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000006f381aeadbffc341-006477ab7b-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:18:03 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:18:03.351 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2023-05-31T20:18:03.352 DEBUG:boto:path=/pklxkd-47/obj1 2023-05-31T20:18:03.353 DEBUG:boto:auth_path=/pklxkd-47/obj1 2023-05-31T20:18:03.353 DEBUG:boto:path=/pklxkd-47/obj1?versionId=null 2023-05-31T20:18:03.353 DEBUG:boto:auth_path=/pklxkd-47/obj1?versionId=null 2023-05-31T20:18:03.354 DEBUG:boto:Method: GET 2023-05-31T20:18:03.354 DEBUG:boto:Path: /pklxkd-47/obj1?versionId=null 2023-05-31T20:18:03.355 DEBUG:boto:Data: 2023-05-31T20:18:03.355 DEBUG:boto:Headers: {} 2023-05-31T20:18:03.355 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:18:03.356 DEBUG:boto:Port: 8001 2023-05-31T20:18:03.356 DEBUG:boto:Params: {} 2023-05-31T20:18:03.357 DEBUG:boto:Token: None 2023-05-31T20:18:03.358 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:18:03 GMT /pklxkd-47/obj1?versionId=null 2023-05-31T20:18:03.358 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:rWwewHe9JKxuVKVDTt4aQehUEhI= 2023-05-31T20:18:03.358 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:18:03 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:rWwewHe9JKxuVKVDTt4aQehUEhI=', 'Content-Length': '0'} 2023-05-31T20:18:03.360 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:15:11 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000994cd68421fe3897-006477ab7b-162e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:18:03 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:18:03.361 DEBUG:boto:path=/pklxkd-47/obj1 2023-05-31T20:18:03.361 DEBUG:boto:auth_path=/pklxkd-47/obj1 2023-05-31T20:18:03.362 DEBUG:boto:path=/pklxkd-47/obj1?versionId=null 2023-05-31T20:18:03.362 DEBUG:boto:auth_path=/pklxkd-47/obj1?versionId=null 2023-05-31T20:18:03.363 DEBUG:boto:Method: GET 2023-05-31T20:18:03.363 DEBUG:boto:Path: /pklxkd-47/obj1?versionId=null 2023-05-31T20:18:03.363 DEBUG:boto:Data: 2023-05-31T20:18:03.364 DEBUG:boto:Headers: {} 2023-05-31T20:18:03.364 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:18:03.364 DEBUG:boto:Port: 8000 2023-05-31T20:18:03.365 DEBUG:boto:Params: {} 2023-05-31T20:18:03.365 DEBUG:boto:Token: None 2023-05-31T20:18:03.366 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:18:03 GMT /pklxkd-47/obj1?versionId=null 2023-05-31T20:18:03.366 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:rWwewHe9JKxuVKVDTt4aQehUEhI= 2023-05-31T20:18:03.367 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:18:03 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:rWwewHe9JKxuVKVDTt4aQehUEhI=', 'Content-Length': '0'} 2023-05-31T20:18:03.368 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:15:11 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000003592c4f9973956de-006477ab7b-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:18:03 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:18:03.369 DEBUG:boto:path=/pklxkd-47/ 2023-05-31T20:18:03.369 DEBUG:boto:auth_path=/pklxkd-47/ 2023-05-31T20:18:03.370 DEBUG:boto:path=/pklxkd-47/?prefix=obj1 2023-05-31T20:18:03.370 DEBUG:boto:auth_path=/pklxkd-47/?prefix=obj1 2023-05-31T20:18:03.370 DEBUG:boto:Method: GET 2023-05-31T20:18:03.371 DEBUG:boto:Path: /pklxkd-47/?prefix=obj1 2023-05-31T20:18:03.371 DEBUG:boto:Data: 2023-05-31T20:18:03.371 DEBUG:boto:Headers: {} 2023-05-31T20:18:03.372 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:18:03.372 DEBUG:boto:Port: 8001 2023-05-31T20:18:03.372 DEBUG:boto:Params: {} 2023-05-31T20:18:03.373 DEBUG:boto:Token: None 2023-05-31T20:18:03.373 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:18:03 GMT /pklxkd-47/ 2023-05-31T20:18:03.374 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:7MgmJ0JDh4n8NkJuKJQxkQqJnhs= 2023-05-31T20:18:03.374 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:18:03 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:7MgmJ0JDh4n8NkJuKJQxkQqJnhs=', 'Content-Length': '0'} 2023-05-31T20:18:03.376 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000eb36273133b92a90-006477ab7b-162e-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:18:03 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:18:03.377 DEBUG:boto:b'pklxkd-47obj11000falseobj12023-05-31T20:15:11.346Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:18:03.377 DEBUG:boto:path=/pklxkd-47/ 2023-05-31T20:18:03.378 DEBUG:boto:auth_path=/pklxkd-47/ 2023-05-31T20:18:03.378 DEBUG:boto:path=/pklxkd-47/?prefix=obj1 2023-05-31T20:18:03.379 DEBUG:boto:auth_path=/pklxkd-47/?prefix=obj1 2023-05-31T20:18:03.379 DEBUG:boto:Method: GET 2023-05-31T20:18:03.379 DEBUG:boto:Path: /pklxkd-47/?prefix=obj1 2023-05-31T20:18:03.380 DEBUG:boto:Data: 2023-05-31T20:18:03.380 DEBUG:boto:Headers: {} 2023-05-31T20:18:03.380 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:18:03.381 DEBUG:boto:Port: 8000 2023-05-31T20:18:03.381 DEBUG:boto:Params: {} 2023-05-31T20:18:03.382 DEBUG:boto:Token: None 2023-05-31T20:18:03.382 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:18:03 GMT /pklxkd-47/ 2023-05-31T20:18:03.382 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:7MgmJ0JDh4n8NkJuKJQxkQqJnhs= 2023-05-31T20:18:03.383 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:18:03 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:7MgmJ0JDh4n8NkJuKJQxkQqJnhs=', 'Content-Length': '0'} 2023-05-31T20:18:03.385 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b820993ce7d94cb0-006477ab7b-1630-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:18:03 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:18:03.386 DEBUG:boto:b'pklxkd-47obj11000falseobj12023-05-31T20:15:11.346Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:18:03.386 DEBUG:boto:path=/pklxkd-47/obj1 2023-05-31T20:18:03.387 DEBUG:boto:auth_path=/pklxkd-47/obj1 2023-05-31T20:18:03.387 DEBUG:boto:Method: HEAD 2023-05-31T20:18:03.388 DEBUG:boto:Path: /pklxkd-47/obj1 2023-05-31T20:18:03.388 DEBUG:boto:Data: 2023-05-31T20:18:03.388 DEBUG:boto:Headers: {} 2023-05-31T20:18:03.389 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:18:03.389 DEBUG:boto:Port: 8001 2023-05-31T20:18:03.389 DEBUG:boto:Params: {} 2023-05-31T20:18:03.390 DEBUG:boto:Token: None 2023-05-31T20:18:03.390 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:18:03 GMT /pklxkd-47/obj1 2023-05-31T20:18:03.390 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:9GQQE66UDcXvN1z0S0P4kIt2QNE= 2023-05-31T20:18:03.391 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:18:03 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:9GQQE66UDcXvN1z0S0P4kIt2QNE=', 'Content-Length': '0'} 2023-05-31T20:18:03.392 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:15:11 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000002b8b44545a39ade3-006477ab7b-162e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:18:03 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:18:03.393 DEBUG:boto:path=/pklxkd-47/obj1 2023-05-31T20:18:03.393 DEBUG:boto:auth_path=/pklxkd-47/obj1 2023-05-31T20:18:03.394 DEBUG:boto:Method: HEAD 2023-05-31T20:18:03.394 DEBUG:boto:Path: /pklxkd-47/obj1 2023-05-31T20:18:03.394 DEBUG:boto:Data: 2023-05-31T20:18:03.395 DEBUG:boto:Headers: {} 2023-05-31T20:18:03.395 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:18:03.395 DEBUG:boto:Port: 8000 2023-05-31T20:18:03.396 DEBUG:boto:Params: {} 2023-05-31T20:18:03.396 DEBUG:boto:Token: None 2023-05-31T20:18:03.397 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:18:03 GMT /pklxkd-47/obj1 2023-05-31T20:18:03.397 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:9GQQE66UDcXvN1z0S0P4kIt2QNE= 2023-05-31T20:18:03.398 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:18:03 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:9GQQE66UDcXvN1z0S0P4kIt2QNE=', 'Content-Length': '0'} 2023-05-31T20:18:03.399 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:15:11 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000b05c3013a742a239-006477ab7b-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:18:03 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:18:03.400 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2023-05-31T20:18:03.401 DEBUG:boto:path=/pklxkd-47/obj1 2023-05-31T20:18:03.401 DEBUG:boto:auth_path=/pklxkd-47/obj1 2023-05-31T20:18:03.401 DEBUG:boto:Method: GET 2023-05-31T20:18:03.402 DEBUG:boto:Path: /pklxkd-47/obj1 2023-05-31T20:18:03.402 DEBUG:boto:Data: 2023-05-31T20:18:03.402 DEBUG:boto:Headers: {} 2023-05-31T20:18:03.403 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:18:03.403 DEBUG:boto:Port: 8001 2023-05-31T20:18:03.403 DEBUG:boto:Params: {} 2023-05-31T20:18:03.404 DEBUG:boto:Token: None 2023-05-31T20:18:03.404 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:18:03 GMT /pklxkd-47/obj1 2023-05-31T20:18:03.405 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:E1WucHX80BCTYwM+qnrORl44Ioc= 2023-05-31T20:18:03.405 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:18:03 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:E1WucHX80BCTYwM+qnrORl44Ioc=', 'Content-Length': '0'} 2023-05-31T20:18:03.407 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:15:11 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000a4c7ca02de22626e-006477ab7b-162e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:18:03 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:18:03.448 DEBUG:boto:path=/pklxkd-47/obj1 2023-05-31T20:18:03.449 DEBUG:boto:auth_path=/pklxkd-47/obj1 2023-05-31T20:18:03.449 DEBUG:boto:Method: GET 2023-05-31T20:18:03.450 DEBUG:boto:Path: /pklxkd-47/obj1 2023-05-31T20:18:03.450 DEBUG:boto:Data: 2023-05-31T20:18:03.450 DEBUG:boto:Headers: {} 2023-05-31T20:18:03.451 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:18:03.451 DEBUG:boto:Port: 8000 2023-05-31T20:18:03.451 DEBUG:boto:Params: {} 2023-05-31T20:18:03.452 DEBUG:boto:Token: None 2023-05-31T20:18:03.452 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:18:03 GMT /pklxkd-47/obj1 2023-05-31T20:18:03.453 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:E1WucHX80BCTYwM+qnrORl44Ioc= 2023-05-31T20:18:03.454 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:18:03 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:E1WucHX80BCTYwM+qnrORl44Ioc=', 'Content-Length': '0'} 2023-05-31T20:18:03.456 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:15:11 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000a521ecb797f30a1d-006477ab7b-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:18:03 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:18:03.458 DEBUG:boto:path=/pklxkd-47/ 2023-05-31T20:18:03.459 DEBUG:boto:auth_path=/pklxkd-47/ 2023-05-31T20:18:03.460 DEBUG:boto:path=/pklxkd-47/?prefix=obj1 2023-05-31T20:18:03.460 DEBUG:boto:auth_path=/pklxkd-47/?prefix=obj1 2023-05-31T20:18:03.460 DEBUG:boto:Method: GET 2023-05-31T20:18:03.461 DEBUG:boto:Path: /pklxkd-47/?prefix=obj1 2023-05-31T20:18:03.461 DEBUG:boto:Data: 2023-05-31T20:18:03.462 DEBUG:boto:Headers: {} 2023-05-31T20:18:03.462 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:18:03.462 DEBUG:boto:Port: 8001 2023-05-31T20:18:03.463 DEBUG:boto:Params: {} 2023-05-31T20:18:03.463 DEBUG:boto:Token: None 2023-05-31T20:18:03.464 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:18:03 GMT /pklxkd-47/ 2023-05-31T20:18:03.464 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:7MgmJ0JDh4n8NkJuKJQxkQqJnhs= 2023-05-31T20:18:03.465 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:18:03 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:7MgmJ0JDh4n8NkJuKJQxkQqJnhs=', 'Content-Length': '0'} 2023-05-31T20:18:03.467 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000000d285005ee9a82e1-006477ab7b-162e-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:18:03 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:18:03.467 DEBUG:boto:b'pklxkd-47obj11000falseobj12023-05-31T20:15:11.346Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:18:03.468 DEBUG:boto:path=/pklxkd-47/ 2023-05-31T20:18:03.469 DEBUG:boto:auth_path=/pklxkd-47/ 2023-05-31T20:18:03.469 DEBUG:boto:path=/pklxkd-47/?prefix=obj1 2023-05-31T20:18:03.469 DEBUG:boto:auth_path=/pklxkd-47/?prefix=obj1 2023-05-31T20:18:03.470 DEBUG:boto:Method: GET 2023-05-31T20:18:03.470 DEBUG:boto:Path: /pklxkd-47/?prefix=obj1 2023-05-31T20:18:03.470 DEBUG:boto:Data: 2023-05-31T20:18:03.471 DEBUG:boto:Headers: {} 2023-05-31T20:18:03.472 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:18:03.473 DEBUG:boto:Port: 8000 2023-05-31T20:18:03.474 DEBUG:boto:Params: {} 2023-05-31T20:18:03.475 DEBUG:boto:Token: None 2023-05-31T20:18:03.476 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:18:03 GMT /pklxkd-47/ 2023-05-31T20:18:03.478 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:7MgmJ0JDh4n8NkJuKJQxkQqJnhs= 2023-05-31T20:18:03.478 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:18:03 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:7MgmJ0JDh4n8NkJuKJQxkQqJnhs=', 'Content-Length': '0'} 2023-05-31T20:18:03.481 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000080d4955f85e55bcb-006477ab7b-1630-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:18:03 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:18:03.482 DEBUG:boto:b'pklxkd-47obj11000falseobj12023-05-31T20:15:11.346Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:18:03.482 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2023-05-31T20:18:03.483 DEBUG:boto:path=/pklxkd-47/obj2 2023-05-31T20:18:03.483 DEBUG:boto:auth_path=/pklxkd-47/obj2 2023-05-31T20:18:03.484 DEBUG:boto:path=/pklxkd-47/obj2?versionId=null 2023-05-31T20:18:03.484 DEBUG:boto:auth_path=/pklxkd-47/obj2?versionId=null 2023-05-31T20:18:03.485 DEBUG:boto:Method: GET 2023-05-31T20:18:03.485 DEBUG:boto:Path: /pklxkd-47/obj2?versionId=null 2023-05-31T20:18:03.485 DEBUG:boto:Data: 2023-05-31T20:18:03.486 DEBUG:boto:Headers: {} 2023-05-31T20:18:03.486 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:18:03.486 DEBUG:boto:Port: 8001 2023-05-31T20:18:03.487 DEBUG:boto:Params: {} 2023-05-31T20:18:03.487 DEBUG:boto:Token: None 2023-05-31T20:18:03.488 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:18:03 GMT /pklxkd-47/obj2?versionId=null 2023-05-31T20:18:03.489 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:HhlG9JbwN6Tg2Crffhx3vLHqJDQ= 2023-05-31T20:18:03.489 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:18:03 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:HhlG9JbwN6Tg2Crffhx3vLHqJDQ=', 'Content-Length': '0'} 2023-05-31T20:18:03.491 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:15:11 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000084da72e32ff1b26e-006477ab7b-162e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:18:03 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:18:03.492 DEBUG:boto:path=/pklxkd-47/obj2 2023-05-31T20:18:03.492 DEBUG:boto:auth_path=/pklxkd-47/obj2 2023-05-31T20:18:03.493 DEBUG:boto:path=/pklxkd-47/obj2?versionId=null 2023-05-31T20:18:03.493 DEBUG:boto:auth_path=/pklxkd-47/obj2?versionId=null 2023-05-31T20:18:03.493 DEBUG:boto:Method: GET 2023-05-31T20:18:03.494 DEBUG:boto:Path: /pklxkd-47/obj2?versionId=null 2023-05-31T20:18:03.494 DEBUG:boto:Data: 2023-05-31T20:18:03.495 DEBUG:boto:Headers: {} 2023-05-31T20:18:03.496 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:18:03.497 DEBUG:boto:Port: 8000 2023-05-31T20:18:03.498 DEBUG:boto:Params: {} 2023-05-31T20:18:03.499 DEBUG:boto:Token: None 2023-05-31T20:18:03.500 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:18:03 GMT /pklxkd-47/obj2?versionId=null 2023-05-31T20:18:03.501 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:HhlG9JbwN6Tg2Crffhx3vLHqJDQ= 2023-05-31T20:18:03.501 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:18:03 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:HhlG9JbwN6Tg2Crffhx3vLHqJDQ=', 'Content-Length': '0'} 2023-05-31T20:18:03.503 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:15:11 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000000d249eb289cb863a-006477ab7b-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:18:03 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:18:03.503 DEBUG:boto:path=/pklxkd-47/obj2 2023-05-31T20:18:03.504 DEBUG:boto:auth_path=/pklxkd-47/obj2 2023-05-31T20:18:03.504 DEBUG:boto:path=/pklxkd-47/obj2?versionId=null 2023-05-31T20:18:03.504 DEBUG:boto:auth_path=/pklxkd-47/obj2?versionId=null 2023-05-31T20:18:03.505 DEBUG:boto:Method: HEAD 2023-05-31T20:18:03.505 DEBUG:boto:Path: /pklxkd-47/obj2?versionId=null 2023-05-31T20:18:03.505 DEBUG:boto:Data: 2023-05-31T20:18:03.506 DEBUG:boto:Headers: {} 2023-05-31T20:18:03.506 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:18:03.506 DEBUG:boto:Port: 8001 2023-05-31T20:18:03.507 DEBUG:boto:Params: {} 2023-05-31T20:18:03.507 DEBUG:boto:Token: None 2023-05-31T20:18:03.508 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:18:03 GMT /pklxkd-47/obj2?versionId=null 2023-05-31T20:18:03.508 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:Nmm4NDe/21K4e465MTO33SBxHSw= 2023-05-31T20:18:03.509 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:18:03 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:Nmm4NDe/21K4e465MTO33SBxHSw=', 'Content-Length': '0'} 2023-05-31T20:18:03.510 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:15:11 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000c23d92d427f24502-006477ab7b-162e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:18:03 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:18:03.511 DEBUG:boto:path=/pklxkd-47/obj2 2023-05-31T20:18:03.511 DEBUG:boto:auth_path=/pklxkd-47/obj2 2023-05-31T20:18:03.511 DEBUG:boto:path=/pklxkd-47/obj2?versionId=null 2023-05-31T20:18:03.512 DEBUG:boto:auth_path=/pklxkd-47/obj2?versionId=null 2023-05-31T20:18:03.512 DEBUG:boto:Method: HEAD 2023-05-31T20:18:03.513 DEBUG:boto:Path: /pklxkd-47/obj2?versionId=null 2023-05-31T20:18:03.513 DEBUG:boto:Data: 2023-05-31T20:18:03.513 DEBUG:boto:Headers: {} 2023-05-31T20:18:03.514 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:18:03.514 DEBUG:boto:Port: 8000 2023-05-31T20:18:03.514 DEBUG:boto:Params: {} 2023-05-31T20:18:03.515 DEBUG:boto:Token: None 2023-05-31T20:18:03.515 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:18:03 GMT /pklxkd-47/obj2?versionId=null 2023-05-31T20:18:03.515 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:Nmm4NDe/21K4e465MTO33SBxHSw= 2023-05-31T20:18:03.516 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:18:03 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:Nmm4NDe/21K4e465MTO33SBxHSw=', 'Content-Length': '0'} 2023-05-31T20:18:03.517 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:15:11 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000001eb2a396495a317-006477ab7b-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:18:03 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:18:03.518 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2023-05-31T20:18:03.518 DEBUG:boto:path=/pklxkd-47/obj2 2023-05-31T20:18:03.519 DEBUG:boto:auth_path=/pklxkd-47/obj2 2023-05-31T20:18:03.519 DEBUG:boto:path=/pklxkd-47/obj2?versionId=null 2023-05-31T20:18:03.520 DEBUG:boto:auth_path=/pklxkd-47/obj2?versionId=null 2023-05-31T20:18:03.520 DEBUG:boto:Method: GET 2023-05-31T20:18:03.520 DEBUG:boto:Path: /pklxkd-47/obj2?versionId=null 2023-05-31T20:18:03.521 DEBUG:boto:Data: 2023-05-31T20:18:03.521 DEBUG:boto:Headers: {} 2023-05-31T20:18:03.522 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:18:03.522 DEBUG:boto:Port: 8001 2023-05-31T20:18:03.522 DEBUG:boto:Params: {} 2023-05-31T20:18:03.523 DEBUG:boto:Token: None 2023-05-31T20:18:03.523 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:18:03 GMT /pklxkd-47/obj2?versionId=null 2023-05-31T20:18:03.524 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:HhlG9JbwN6Tg2Crffhx3vLHqJDQ= 2023-05-31T20:18:03.524 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:18:03 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:HhlG9JbwN6Tg2Crffhx3vLHqJDQ=', 'Content-Length': '0'} 2023-05-31T20:18:03.526 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:15:11 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000b99c1978082a9043-006477ab7b-162e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:18:03 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:18:03.526 DEBUG:boto:path=/pklxkd-47/obj2 2023-05-31T20:18:03.527 DEBUG:boto:auth_path=/pklxkd-47/obj2 2023-05-31T20:18:03.527 DEBUG:boto:path=/pklxkd-47/obj2?versionId=null 2023-05-31T20:18:03.528 DEBUG:boto:auth_path=/pklxkd-47/obj2?versionId=null 2023-05-31T20:18:03.528 DEBUG:boto:Method: GET 2023-05-31T20:18:03.529 DEBUG:boto:Path: /pklxkd-47/obj2?versionId=null 2023-05-31T20:18:03.529 DEBUG:boto:Data: 2023-05-31T20:18:03.529 DEBUG:boto:Headers: {} 2023-05-31T20:18:03.530 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:18:03.530 DEBUG:boto:Port: 8000 2023-05-31T20:18:03.531 DEBUG:boto:Params: {} 2023-05-31T20:18:03.531 DEBUG:boto:Token: None 2023-05-31T20:18:03.532 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:18:03 GMT /pklxkd-47/obj2?versionId=null 2023-05-31T20:18:03.532 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:HhlG9JbwN6Tg2Crffhx3vLHqJDQ= 2023-05-31T20:18:03.533 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:18:03 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:HhlG9JbwN6Tg2Crffhx3vLHqJDQ=', 'Content-Length': '0'} 2023-05-31T20:18:03.534 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:15:11 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000f54da309a2e2caff-006477ab7b-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:18:03 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:18:03.535 DEBUG:boto:path=/pklxkd-47/ 2023-05-31T20:18:03.536 DEBUG:boto:auth_path=/pklxkd-47/ 2023-05-31T20:18:03.537 DEBUG:boto:path=/pklxkd-47/?prefix=obj2 2023-05-31T20:18:03.538 DEBUG:boto:auth_path=/pklxkd-47/?prefix=obj2 2023-05-31T20:18:03.539 DEBUG:boto:Method: GET 2023-05-31T20:18:03.540 DEBUG:boto:Path: /pklxkd-47/?prefix=obj2 2023-05-31T20:18:03.541 DEBUG:boto:Data: 2023-05-31T20:18:03.541 DEBUG:boto:Headers: {} 2023-05-31T20:18:03.542 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:18:03.543 DEBUG:boto:Port: 8001 2023-05-31T20:18:03.544 DEBUG:boto:Params: {} 2023-05-31T20:18:03.545 DEBUG:boto:Token: None 2023-05-31T20:18:03.545 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:18:03 GMT /pklxkd-47/ 2023-05-31T20:18:03.546 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:7MgmJ0JDh4n8NkJuKJQxkQqJnhs= 2023-05-31T20:18:03.547 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:18:03 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:7MgmJ0JDh4n8NkJuKJQxkQqJnhs=', 'Content-Length': '0'} 2023-05-31T20:18:03.549 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000964c5ecf8699fc30-006477ab7b-162e-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:18:03 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:18:03.550 DEBUG:boto:b'pklxkd-47obj21000falseobj22023-05-31T20:15:11.368Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:18:03.551 DEBUG:boto:path=/pklxkd-47/ 2023-05-31T20:18:03.551 DEBUG:boto:auth_path=/pklxkd-47/ 2023-05-31T20:18:03.551 DEBUG:boto:path=/pklxkd-47/?prefix=obj2 2023-05-31T20:18:03.552 DEBUG:boto:auth_path=/pklxkd-47/?prefix=obj2 2023-05-31T20:18:03.552 DEBUG:boto:Method: GET 2023-05-31T20:18:03.553 DEBUG:boto:Path: /pklxkd-47/?prefix=obj2 2023-05-31T20:18:03.554 DEBUG:boto:Data: 2023-05-31T20:18:03.554 DEBUG:boto:Headers: {} 2023-05-31T20:18:03.554 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:18:03.555 DEBUG:boto:Port: 8000 2023-05-31T20:18:03.555 DEBUG:boto:Params: {} 2023-05-31T20:18:03.556 DEBUG:boto:Token: None 2023-05-31T20:18:03.556 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:18:03 GMT /pklxkd-47/ 2023-05-31T20:18:03.557 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:7MgmJ0JDh4n8NkJuKJQxkQqJnhs= 2023-05-31T20:18:03.557 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:18:03 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:7MgmJ0JDh4n8NkJuKJQxkQqJnhs=', 'Content-Length': '0'} 2023-05-31T20:18:03.560 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000006a1d88d84991cb86-006477ab7b-1630-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:18:03 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:18:03.561 DEBUG:boto:b'pklxkd-47obj21000falseobj22023-05-31T20:15:11.368Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:18:03.562 DEBUG:boto:path=/pklxkd-47/obj2 2023-05-31T20:18:03.563 DEBUG:boto:auth_path=/pklxkd-47/obj2 2023-05-31T20:18:03.564 DEBUG:boto:Method: HEAD 2023-05-31T20:18:03.565 DEBUG:boto:Path: /pklxkd-47/obj2 2023-05-31T20:18:03.565 DEBUG:boto:Data: 2023-05-31T20:18:03.565 DEBUG:boto:Headers: {} 2023-05-31T20:18:03.566 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:18:03.566 DEBUG:boto:Port: 8001 2023-05-31T20:18:03.566 DEBUG:boto:Params: {} 2023-05-31T20:18:03.567 DEBUG:boto:Token: None 2023-05-31T20:18:03.567 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:18:03 GMT /pklxkd-47/obj2 2023-05-31T20:18:03.568 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:pPnMFVsGmUj1UiEtfiMvIwnhX7I= 2023-05-31T20:18:03.568 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:18:03 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:pPnMFVsGmUj1UiEtfiMvIwnhX7I=', 'Content-Length': '0'} 2023-05-31T20:18:03.570 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:15:11 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000d8ae0a9531bcfe76-006477ab7b-162e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:18:03 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:18:03.570 DEBUG:boto:path=/pklxkd-47/obj2 2023-05-31T20:18:03.571 DEBUG:boto:auth_path=/pklxkd-47/obj2 2023-05-31T20:18:03.571 DEBUG:boto:Method: HEAD 2023-05-31T20:18:03.571 DEBUG:boto:Path: /pklxkd-47/obj2 2023-05-31T20:18:03.572 DEBUG:boto:Data: 2023-05-31T20:18:03.572 DEBUG:boto:Headers: {} 2023-05-31T20:18:03.573 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:18:03.573 DEBUG:boto:Port: 8000 2023-05-31T20:18:03.573 DEBUG:boto:Params: {} 2023-05-31T20:18:03.574 DEBUG:boto:Token: None 2023-05-31T20:18:03.574 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:18:03 GMT /pklxkd-47/obj2 2023-05-31T20:18:03.575 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:pPnMFVsGmUj1UiEtfiMvIwnhX7I= 2023-05-31T20:18:03.575 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:18:03 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:pPnMFVsGmUj1UiEtfiMvIwnhX7I=', 'Content-Length': '0'} 2023-05-31T20:18:03.577 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:15:11 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000000d5297cc9d8dd5ed-006477ab7b-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:18:03 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:18:03.577 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2023-05-31T20:18:03.578 DEBUG:boto:path=/pklxkd-47/obj2 2023-05-31T20:18:03.578 DEBUG:boto:auth_path=/pklxkd-47/obj2 2023-05-31T20:18:03.579 DEBUG:boto:Method: GET 2023-05-31T20:18:03.579 DEBUG:boto:Path: /pklxkd-47/obj2 2023-05-31T20:18:03.579 DEBUG:boto:Data: 2023-05-31T20:18:03.580 DEBUG:boto:Headers: {} 2023-05-31T20:18:03.580 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:18:03.581 DEBUG:boto:Port: 8001 2023-05-31T20:18:03.581 DEBUG:boto:Params: {} 2023-05-31T20:18:03.581 DEBUG:boto:Token: None 2023-05-31T20:18:03.582 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:18:03 GMT /pklxkd-47/obj2 2023-05-31T20:18:03.582 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:4Z1VYKjek00onXlL7YdpvZ2jEA8= 2023-05-31T20:18:03.583 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:18:03 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:4Z1VYKjek00onXlL7YdpvZ2jEA8=', 'Content-Length': '0'} 2023-05-31T20:18:03.584 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:15:11 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000004a02d22dd65f5c29-006477ab7b-162e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:18:03 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:18:03.585 DEBUG:boto:path=/pklxkd-47/obj2 2023-05-31T20:18:03.586 DEBUG:boto:auth_path=/pklxkd-47/obj2 2023-05-31T20:18:03.587 DEBUG:boto:Method: GET 2023-05-31T20:18:03.588 DEBUG:boto:Path: /pklxkd-47/obj2 2023-05-31T20:18:03.589 DEBUG:boto:Data: 2023-05-31T20:18:03.590 DEBUG:boto:Headers: {} 2023-05-31T20:18:03.590 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:18:03.590 DEBUG:boto:Port: 8000 2023-05-31T20:18:03.591 DEBUG:boto:Params: {} 2023-05-31T20:18:03.591 DEBUG:boto:Token: None 2023-05-31T20:18:03.592 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:18:03 GMT /pklxkd-47/obj2 2023-05-31T20:18:03.592 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:4Z1VYKjek00onXlL7YdpvZ2jEA8= 2023-05-31T20:18:03.592 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:18:03 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:4Z1VYKjek00onXlL7YdpvZ2jEA8=', 'Content-Length': '0'} 2023-05-31T20:18:03.594 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:15:11 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000a89d7558d092ef39-006477ab7b-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:18:03 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:18:03.636 DEBUG:boto:path=/pklxkd-47/ 2023-05-31T20:18:03.637 DEBUG:boto:auth_path=/pklxkd-47/ 2023-05-31T20:18:03.638 DEBUG:boto:path=/pklxkd-47/?prefix=obj2 2023-05-31T20:18:03.638 DEBUG:boto:auth_path=/pklxkd-47/?prefix=obj2 2023-05-31T20:18:03.639 DEBUG:boto:Method: GET 2023-05-31T20:18:03.639 DEBUG:boto:Path: /pklxkd-47/?prefix=obj2 2023-05-31T20:18:03.640 DEBUG:boto:Data: 2023-05-31T20:18:03.640 DEBUG:boto:Headers: {} 2023-05-31T20:18:03.641 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:18:03.641 DEBUG:boto:Port: 8001 2023-05-31T20:18:03.641 DEBUG:boto:Params: {} 2023-05-31T20:18:03.642 DEBUG:boto:Token: None 2023-05-31T20:18:03.642 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:18:03 GMT /pklxkd-47/ 2023-05-31T20:18:03.643 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:7MgmJ0JDh4n8NkJuKJQxkQqJnhs= 2023-05-31T20:18:03.643 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:18:03 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:7MgmJ0JDh4n8NkJuKJQxkQqJnhs=', 'Content-Length': '0'} 2023-05-31T20:18:03.645 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000094fb91394c0acb7b-006477ab7b-162e-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:18:03 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:18:03.646 DEBUG:boto:b'pklxkd-47obj21000falseobj22023-05-31T20:15:11.368Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:18:03.647 DEBUG:boto:path=/pklxkd-47/ 2023-05-31T20:18:03.647 DEBUG:boto:auth_path=/pklxkd-47/ 2023-05-31T20:18:03.648 DEBUG:boto:path=/pklxkd-47/?prefix=obj2 2023-05-31T20:18:03.648 DEBUG:boto:auth_path=/pklxkd-47/?prefix=obj2 2023-05-31T20:18:03.648 DEBUG:boto:Method: GET 2023-05-31T20:18:03.649 DEBUG:boto:Path: /pklxkd-47/?prefix=obj2 2023-05-31T20:18:03.649 DEBUG:boto:Data: 2023-05-31T20:18:03.650 DEBUG:boto:Headers: {} 2023-05-31T20:18:03.650 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:18:03.651 DEBUG:boto:Port: 8000 2023-05-31T20:18:03.651 DEBUG:boto:Params: {} 2023-05-31T20:18:03.652 DEBUG:boto:Token: None 2023-05-31T20:18:03.653 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:18:03 GMT /pklxkd-47/ 2023-05-31T20:18:03.654 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:7MgmJ0JDh4n8NkJuKJQxkQqJnhs= 2023-05-31T20:18:03.654 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:18:03 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:7MgmJ0JDh4n8NkJuKJQxkQqJnhs=', 'Content-Length': '0'} 2023-05-31T20:18:03.657 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000054fbc150e4b3b22e-006477ab7b-1630-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:18:03 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:18:03.658 DEBUG:boto:b'pklxkd-47obj21000falseobj22023-05-31T20:15:11.368Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:18:03.658 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2023-05-31T20:18:03.659 DEBUG:boto:path=/pklxkd-47/obj3 2023-05-31T20:18:03.659 DEBUG:boto:auth_path=/pklxkd-47/obj3 2023-05-31T20:18:03.660 DEBUG:boto:path=/pklxkd-47/obj3?versionId=null 2023-05-31T20:18:03.660 DEBUG:boto:auth_path=/pklxkd-47/obj3?versionId=null 2023-05-31T20:18:03.661 DEBUG:boto:Method: GET 2023-05-31T20:18:03.661 DEBUG:boto:Path: /pklxkd-47/obj3?versionId=null 2023-05-31T20:18:03.662 DEBUG:boto:Data: 2023-05-31T20:18:03.662 DEBUG:boto:Headers: {} 2023-05-31T20:18:03.663 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:18:03.663 DEBUG:boto:Port: 8001 2023-05-31T20:18:03.664 DEBUG:boto:Params: {} 2023-05-31T20:18:03.665 DEBUG:boto:Token: None 2023-05-31T20:18:03.665 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:18:03 GMT /pklxkd-47/obj3?versionId=null 2023-05-31T20:18:03.666 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:ZxMs9bwNZmgWDBa/dMnWz+AenXA= 2023-05-31T20:18:03.666 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:18:03 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:ZxMs9bwNZmgWDBa/dMnWz+AenXA=', 'Content-Length': '0'} 2023-05-31T20:18:03.668 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:15:11 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000000357264b0fc9b7c1-006477ab7b-162e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:18:03 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:18:03.708 DEBUG:boto:path=/pklxkd-47/obj3 2023-05-31T20:18:03.709 DEBUG:boto:auth_path=/pklxkd-47/obj3 2023-05-31T20:18:03.710 DEBUG:boto:path=/pklxkd-47/obj3?versionId=null 2023-05-31T20:18:03.710 DEBUG:boto:auth_path=/pklxkd-47/obj3?versionId=null 2023-05-31T20:18:03.710 DEBUG:boto:Method: GET 2023-05-31T20:18:03.711 DEBUG:boto:Path: /pklxkd-47/obj3?versionId=null 2023-05-31T20:18:03.712 DEBUG:boto:Data: 2023-05-31T20:18:03.712 DEBUG:boto:Headers: {} 2023-05-31T20:18:03.713 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:18:03.713 DEBUG:boto:Port: 8000 2023-05-31T20:18:03.714 DEBUG:boto:Params: {} 2023-05-31T20:18:03.714 DEBUG:boto:Token: None 2023-05-31T20:18:03.715 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:18:03 GMT /pklxkd-47/obj3?versionId=null 2023-05-31T20:18:03.715 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:ZxMs9bwNZmgWDBa/dMnWz+AenXA= 2023-05-31T20:18:03.715 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:18:03 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:ZxMs9bwNZmgWDBa/dMnWz+AenXA=', 'Content-Length': '0'} 2023-05-31T20:18:03.717 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:15:11 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000091a666e0c0c140fb-006477ab7b-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:18:03 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:18:03.718 DEBUG:boto:path=/pklxkd-47/obj3 2023-05-31T20:18:03.719 DEBUG:boto:auth_path=/pklxkd-47/obj3 2023-05-31T20:18:03.719 DEBUG:boto:path=/pklxkd-47/obj3?versionId=null 2023-05-31T20:18:03.720 DEBUG:boto:auth_path=/pklxkd-47/obj3?versionId=null 2023-05-31T20:18:03.720 DEBUG:boto:Method: HEAD 2023-05-31T20:18:03.721 DEBUG:boto:Path: /pklxkd-47/obj3?versionId=null 2023-05-31T20:18:03.721 DEBUG:boto:Data: 2023-05-31T20:18:03.721 DEBUG:boto:Headers: {} 2023-05-31T20:18:03.722 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:18:03.722 DEBUG:boto:Port: 8001 2023-05-31T20:18:03.723 DEBUG:boto:Params: {} 2023-05-31T20:18:03.723 DEBUG:boto:Token: None 2023-05-31T20:18:03.724 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:18:03 GMT /pklxkd-47/obj3?versionId=null 2023-05-31T20:18:03.724 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:3tMv/eZOn0J707S5X2Hkznjg6KM= 2023-05-31T20:18:03.725 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:18:03 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:3tMv/eZOn0J707S5X2Hkznjg6KM=', 'Content-Length': '0'} 2023-05-31T20:18:03.727 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:15:11 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000001796f6b591e28acf-006477ab7b-162e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:18:03 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:18:03.727 DEBUG:boto:path=/pklxkd-47/obj3 2023-05-31T20:18:03.728 DEBUG:boto:auth_path=/pklxkd-47/obj3 2023-05-31T20:18:03.729 DEBUG:boto:path=/pklxkd-47/obj3?versionId=null 2023-05-31T20:18:03.729 DEBUG:boto:auth_path=/pklxkd-47/obj3?versionId=null 2023-05-31T20:18:03.730 DEBUG:boto:Method: HEAD 2023-05-31T20:18:03.730 DEBUG:boto:Path: /pklxkd-47/obj3?versionId=null 2023-05-31T20:18:03.730 DEBUG:boto:Data: 2023-05-31T20:18:03.731 DEBUG:boto:Headers: {} 2023-05-31T20:18:03.732 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:18:03.733 DEBUG:boto:Port: 8000 2023-05-31T20:18:03.734 DEBUG:boto:Params: {} 2023-05-31T20:18:03.734 DEBUG:boto:Token: None 2023-05-31T20:18:03.735 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:18:03 GMT /pklxkd-47/obj3?versionId=null 2023-05-31T20:18:03.736 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:3tMv/eZOn0J707S5X2Hkznjg6KM= 2023-05-31T20:18:03.737 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:18:03 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:3tMv/eZOn0J707S5X2Hkznjg6KM=', 'Content-Length': '0'} 2023-05-31T20:18:03.739 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:15:11 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000005d69fbf6bc93a0d0-006477ab7b-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:18:03 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:18:03.739 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2023-05-31T20:18:03.740 DEBUG:boto:path=/pklxkd-47/obj3 2023-05-31T20:18:03.740 DEBUG:boto:auth_path=/pklxkd-47/obj3 2023-05-31T20:18:03.741 DEBUG:boto:path=/pklxkd-47/obj3?versionId=null 2023-05-31T20:18:03.742 DEBUG:boto:auth_path=/pklxkd-47/obj3?versionId=null 2023-05-31T20:18:03.742 DEBUG:boto:Method: GET 2023-05-31T20:18:03.742 DEBUG:boto:Path: /pklxkd-47/obj3?versionId=null 2023-05-31T20:18:03.743 DEBUG:boto:Data: 2023-05-31T20:18:03.743 DEBUG:boto:Headers: {} 2023-05-31T20:18:03.743 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:18:03.744 DEBUG:boto:Port: 8001 2023-05-31T20:18:03.744 DEBUG:boto:Params: {} 2023-05-31T20:18:03.745 DEBUG:boto:Token: None 2023-05-31T20:18:03.745 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:18:03 GMT /pklxkd-47/obj3?versionId=null 2023-05-31T20:18:03.746 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:ZxMs9bwNZmgWDBa/dMnWz+AenXA= 2023-05-31T20:18:03.746 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:18:03 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:ZxMs9bwNZmgWDBa/dMnWz+AenXA=', 'Content-Length': '0'} 2023-05-31T20:18:03.748 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:15:11 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000e1512f39aca5cc5f-006477ab7b-162e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:18:03 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:18:03.749 DEBUG:boto:path=/pklxkd-47/obj3 2023-05-31T20:18:03.749 DEBUG:boto:auth_path=/pklxkd-47/obj3 2023-05-31T20:18:03.750 DEBUG:boto:path=/pklxkd-47/obj3?versionId=null 2023-05-31T20:18:03.750 DEBUG:boto:auth_path=/pklxkd-47/obj3?versionId=null 2023-05-31T20:18:03.751 DEBUG:boto:Method: GET 2023-05-31T20:18:03.752 DEBUG:boto:Path: /pklxkd-47/obj3?versionId=null 2023-05-31T20:18:03.753 DEBUG:boto:Data: 2023-05-31T20:18:03.754 DEBUG:boto:Headers: {} 2023-05-31T20:18:03.754 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:18:03.755 DEBUG:boto:Port: 8000 2023-05-31T20:18:03.755 DEBUG:boto:Params: {} 2023-05-31T20:18:03.756 DEBUG:boto:Token: None 2023-05-31T20:18:03.757 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:18:03 GMT /pklxkd-47/obj3?versionId=null 2023-05-31T20:18:03.757 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:ZxMs9bwNZmgWDBa/dMnWz+AenXA= 2023-05-31T20:18:03.758 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:18:03 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:ZxMs9bwNZmgWDBa/dMnWz+AenXA=', 'Content-Length': '0'} 2023-05-31T20:18:03.760 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:15:11 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000005711a863b44738d0-006477ab7b-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:18:03 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:18:03.761 DEBUG:boto:path=/pklxkd-47/ 2023-05-31T20:18:03.761 DEBUG:boto:auth_path=/pklxkd-47/ 2023-05-31T20:18:03.762 DEBUG:boto:path=/pklxkd-47/?prefix=obj3 2023-05-31T20:18:03.762 DEBUG:boto:auth_path=/pklxkd-47/?prefix=obj3 2023-05-31T20:18:03.763 DEBUG:boto:Method: GET 2023-05-31T20:18:03.763 DEBUG:boto:Path: /pklxkd-47/?prefix=obj3 2023-05-31T20:18:03.764 DEBUG:boto:Data: 2023-05-31T20:18:03.764 DEBUG:boto:Headers: {} 2023-05-31T20:18:03.765 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:18:03.766 DEBUG:boto:Port: 8001 2023-05-31T20:18:03.766 DEBUG:boto:Params: {} 2023-05-31T20:18:03.767 DEBUG:boto:Token: None 2023-05-31T20:18:03.767 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:18:03 GMT /pklxkd-47/ 2023-05-31T20:18:03.768 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:7MgmJ0JDh4n8NkJuKJQxkQqJnhs= 2023-05-31T20:18:03.768 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:18:03 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:7MgmJ0JDh4n8NkJuKJQxkQqJnhs=', 'Content-Length': '0'} 2023-05-31T20:18:03.770 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000007ca62c00108ec1bf-006477ab7b-162e-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:18:03 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:18:03.771 DEBUG:boto:b'pklxkd-47obj31000falseobj32023-05-31T20:15:11.402Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:18:03.772 DEBUG:boto:path=/pklxkd-47/ 2023-05-31T20:18:03.772 DEBUG:boto:auth_path=/pklxkd-47/ 2023-05-31T20:18:03.773 DEBUG:boto:path=/pklxkd-47/?prefix=obj3 2023-05-31T20:18:03.773 DEBUG:boto:auth_path=/pklxkd-47/?prefix=obj3 2023-05-31T20:18:03.773 DEBUG:boto:Method: GET 2023-05-31T20:18:03.774 DEBUG:boto:Path: /pklxkd-47/?prefix=obj3 2023-05-31T20:18:03.775 DEBUG:boto:Data: 2023-05-31T20:18:03.775 DEBUG:boto:Headers: {} 2023-05-31T20:18:03.775 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:18:03.776 DEBUG:boto:Port: 8000 2023-05-31T20:18:03.776 DEBUG:boto:Params: {} 2023-05-31T20:18:03.777 DEBUG:boto:Token: None 2023-05-31T20:18:03.777 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:18:03 GMT /pklxkd-47/ 2023-05-31T20:18:03.778 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:7MgmJ0JDh4n8NkJuKJQxkQqJnhs= 2023-05-31T20:18:03.779 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:18:03 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:7MgmJ0JDh4n8NkJuKJQxkQqJnhs=', 'Content-Length': '0'} 2023-05-31T20:18:03.781 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000db43b379066c3d5f-006477ab7b-1630-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:18:03 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:18:03.783 DEBUG:boto:b'pklxkd-47obj31000falseobj32023-05-31T20:15:11.402Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:18:03.784 DEBUG:boto:path=/pklxkd-47/obj3 2023-05-31T20:18:03.785 DEBUG:boto:auth_path=/pklxkd-47/obj3 2023-05-31T20:18:03.785 DEBUG:boto:Method: HEAD 2023-05-31T20:18:03.786 DEBUG:boto:Path: /pklxkd-47/obj3 2023-05-31T20:18:03.786 DEBUG:boto:Data: 2023-05-31T20:18:03.787 DEBUG:boto:Headers: {} 2023-05-31T20:18:03.787 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:18:03.787 DEBUG:boto:Port: 8001 2023-05-31T20:18:03.788 DEBUG:boto:Params: {} 2023-05-31T20:18:03.788 DEBUG:boto:Token: None 2023-05-31T20:18:03.788 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:18:03 GMT /pklxkd-47/obj3 2023-05-31T20:18:03.789 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:yD8a7RUdg3xcgPt9FNWSTGL2jEQ= 2023-05-31T20:18:03.789 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:18:03 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:yD8a7RUdg3xcgPt9FNWSTGL2jEQ=', 'Content-Length': '0'} 2023-05-31T20:18:03.791 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:15:11 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000cee6d6ae6e98bea8-006477ab7b-162e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:18:03 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:18:03.792 DEBUG:boto:path=/pklxkd-47/obj3 2023-05-31T20:18:03.792 DEBUG:boto:auth_path=/pklxkd-47/obj3 2023-05-31T20:18:03.792 DEBUG:boto:Method: HEAD 2023-05-31T20:18:03.793 DEBUG:boto:Path: /pklxkd-47/obj3 2023-05-31T20:18:03.793 DEBUG:boto:Data: 2023-05-31T20:18:03.793 DEBUG:boto:Headers: {} 2023-05-31T20:18:03.794 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:18:03.794 DEBUG:boto:Port: 8000 2023-05-31T20:18:03.794 DEBUG:boto:Params: {} 2023-05-31T20:18:03.795 DEBUG:boto:Token: None 2023-05-31T20:18:03.795 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:18:03 GMT /pklxkd-47/obj3 2023-05-31T20:18:03.796 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:yD8a7RUdg3xcgPt9FNWSTGL2jEQ= 2023-05-31T20:18:03.796 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:18:03 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:yD8a7RUdg3xcgPt9FNWSTGL2jEQ=', 'Content-Length': '0'} 2023-05-31T20:18:03.797 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:15:11 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000912c84933eb82d97-006477ab7b-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:18:03 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:18:03.798 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2023-05-31T20:18:03.798 DEBUG:boto:path=/pklxkd-47/obj3 2023-05-31T20:18:03.799 DEBUG:boto:auth_path=/pklxkd-47/obj3 2023-05-31T20:18:03.799 DEBUG:boto:Method: GET 2023-05-31T20:18:03.800 DEBUG:boto:Path: /pklxkd-47/obj3 2023-05-31T20:18:03.800 DEBUG:boto:Data: 2023-05-31T20:18:03.800 DEBUG:boto:Headers: {} 2023-05-31T20:18:03.801 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:18:03.801 DEBUG:boto:Port: 8001 2023-05-31T20:18:03.801 DEBUG:boto:Params: {} 2023-05-31T20:18:03.802 DEBUG:boto:Token: None 2023-05-31T20:18:03.802 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:18:03 GMT /pklxkd-47/obj3 2023-05-31T20:18:03.803 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:mVY/U2eTb85lRd9cAGNLI9ZfC4I= 2023-05-31T20:18:03.803 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:18:03 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:mVY/U2eTb85lRd9cAGNLI9ZfC4I=', 'Content-Length': '0'} 2023-05-31T20:18:03.805 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:15:11 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000043c68a38b02029f2-006477ab7b-162e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:18:03 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:18:03.805 DEBUG:boto:path=/pklxkd-47/obj3 2023-05-31T20:18:03.806 DEBUG:boto:auth_path=/pklxkd-47/obj3 2023-05-31T20:18:03.806 DEBUG:boto:Method: GET 2023-05-31T20:18:03.806 DEBUG:boto:Path: /pklxkd-47/obj3 2023-05-31T20:18:03.807 DEBUG:boto:Data: 2023-05-31T20:18:03.807 DEBUG:boto:Headers: {} 2023-05-31T20:18:03.808 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:18:03.808 DEBUG:boto:Port: 8000 2023-05-31T20:18:03.808 DEBUG:boto:Params: {} 2023-05-31T20:18:03.809 DEBUG:boto:Token: None 2023-05-31T20:18:03.809 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:18:03 GMT /pklxkd-47/obj3 2023-05-31T20:18:03.810 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:mVY/U2eTb85lRd9cAGNLI9ZfC4I= 2023-05-31T20:18:03.810 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:18:03 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:mVY/U2eTb85lRd9cAGNLI9ZfC4I=', 'Content-Length': '0'} 2023-05-31T20:18:03.812 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:15:11 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000d521b1665988263c-006477ab7b-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:18:03 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:18:03.817 DEBUG:boto:path=/pklxkd-47/ 2023-05-31T20:18:03.817 DEBUG:boto:auth_path=/pklxkd-47/ 2023-05-31T20:18:03.818 DEBUG:boto:path=/pklxkd-47/?prefix=obj3 2023-05-31T20:18:03.818 DEBUG:boto:auth_path=/pklxkd-47/?prefix=obj3 2023-05-31T20:18:03.818 DEBUG:boto:Method: GET 2023-05-31T20:18:03.819 DEBUG:boto:Path: /pklxkd-47/?prefix=obj3 2023-05-31T20:18:03.819 DEBUG:boto:Data: 2023-05-31T20:18:03.819 DEBUG:boto:Headers: {} 2023-05-31T20:18:03.820 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:18:03.820 DEBUG:boto:Port: 8001 2023-05-31T20:18:03.820 DEBUG:boto:Params: {} 2023-05-31T20:18:03.821 DEBUG:boto:Token: None 2023-05-31T20:18:03.821 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:18:03 GMT /pklxkd-47/ 2023-05-31T20:18:03.822 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:7MgmJ0JDh4n8NkJuKJQxkQqJnhs= 2023-05-31T20:18:03.822 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:18:03 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:7MgmJ0JDh4n8NkJuKJQxkQqJnhs=', 'Content-Length': '0'} 2023-05-31T20:18:03.824 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000489c931c1c8b2b2b-006477ab7b-162e-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:18:03 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:18:03.825 DEBUG:boto:b'pklxkd-47obj31000falseobj32023-05-31T20:15:11.402Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:18:03.826 DEBUG:boto:path=/pklxkd-47/ 2023-05-31T20:18:03.826 DEBUG:boto:auth_path=/pklxkd-47/ 2023-05-31T20:18:03.827 DEBUG:boto:path=/pklxkd-47/?prefix=obj3 2023-05-31T20:18:03.827 DEBUG:boto:auth_path=/pklxkd-47/?prefix=obj3 2023-05-31T20:18:03.827 DEBUG:boto:Method: GET 2023-05-31T20:18:03.828 DEBUG:boto:Path: /pklxkd-47/?prefix=obj3 2023-05-31T20:18:03.828 DEBUG:boto:Data: 2023-05-31T20:18:03.828 DEBUG:boto:Headers: {} 2023-05-31T20:18:03.829 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:18:03.829 DEBUG:boto:Port: 8000 2023-05-31T20:18:03.829 DEBUG:boto:Params: {} 2023-05-31T20:18:03.830 DEBUG:boto:Token: None 2023-05-31T20:18:03.830 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:18:03 GMT /pklxkd-47/ 2023-05-31T20:18:03.831 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:7MgmJ0JDh4n8NkJuKJQxkQqJnhs= 2023-05-31T20:18:03.831 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:18:03 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:7MgmJ0JDh4n8NkJuKJQxkQqJnhs=', 'Content-Length': '0'} 2023-05-31T20:18:03.834 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000000d97261c4631a7ae-006477ab7b-1630-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:18:03 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:18:03.877 DEBUG:boto:b'pklxkd-47obj31000falseobj32023-05-31T20:15:11.402Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:18:03.879 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2023-05-31T20:18:03.879 DEBUG:boto:path=/pklxkd-47/obj4 2023-05-31T20:18:03.880 DEBUG:boto:auth_path=/pklxkd-47/obj4 2023-05-31T20:18:03.881 DEBUG:boto:path=/pklxkd-47/obj4?versionId=null 2023-05-31T20:18:03.882 DEBUG:boto:auth_path=/pklxkd-47/obj4?versionId=null 2023-05-31T20:18:03.883 DEBUG:boto:Method: GET 2023-05-31T20:18:03.884 DEBUG:boto:Path: /pklxkd-47/obj4?versionId=null 2023-05-31T20:18:03.884 DEBUG:boto:Data: 2023-05-31T20:18:03.885 DEBUG:boto:Headers: {} 2023-05-31T20:18:03.886 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:18:03.887 DEBUG:boto:Port: 8001 2023-05-31T20:18:03.888 DEBUG:boto:Params: {} 2023-05-31T20:18:03.888 DEBUG:boto:Token: None 2023-05-31T20:18:03.889 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:18:03 GMT /pklxkd-47/obj4?versionId=null 2023-05-31T20:18:03.889 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:ouUH72EFrT7yES4b65ZlSYzwKcY= 2023-05-31T20:18:03.890 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:18:03 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:ouUH72EFrT7yES4b65ZlSYzwKcY=', 'Content-Length': '0'} 2023-05-31T20:18:03.891 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:15:11 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000b485f04038ab3a81-006477ab7b-162e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:18:03 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:18:03.892 DEBUG:boto:path=/pklxkd-47/obj4 2023-05-31T20:18:03.893 DEBUG:boto:auth_path=/pklxkd-47/obj4 2023-05-31T20:18:03.893 DEBUG:boto:path=/pklxkd-47/obj4?versionId=null 2023-05-31T20:18:03.894 DEBUG:boto:auth_path=/pklxkd-47/obj4?versionId=null 2023-05-31T20:18:03.894 DEBUG:boto:Method: GET 2023-05-31T20:18:03.894 DEBUG:boto:Path: /pklxkd-47/obj4?versionId=null 2023-05-31T20:18:03.895 DEBUG:boto:Data: 2023-05-31T20:18:03.895 DEBUG:boto:Headers: {} 2023-05-31T20:18:03.895 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:18:03.896 DEBUG:boto:Port: 8000 2023-05-31T20:18:03.896 DEBUG:boto:Params: {} 2023-05-31T20:18:03.897 DEBUG:boto:Token: None 2023-05-31T20:18:03.897 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:18:03 GMT /pklxkd-47/obj4?versionId=null 2023-05-31T20:18:03.898 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:ouUH72EFrT7yES4b65ZlSYzwKcY= 2023-05-31T20:18:03.898 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:18:03 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:ouUH72EFrT7yES4b65ZlSYzwKcY=', 'Content-Length': '0'} 2023-05-31T20:18:03.900 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:15:11 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000002cb1cb006646a0c9-006477ab7b-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:18:03 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:18:03.901 DEBUG:boto:path=/pklxkd-47/obj4 2023-05-31T20:18:03.901 DEBUG:boto:auth_path=/pklxkd-47/obj4 2023-05-31T20:18:03.902 DEBUG:boto:path=/pklxkd-47/obj4?versionId=null 2023-05-31T20:18:03.902 DEBUG:boto:auth_path=/pklxkd-47/obj4?versionId=null 2023-05-31T20:18:03.902 DEBUG:boto:Method: HEAD 2023-05-31T20:18:03.903 DEBUG:boto:Path: /pklxkd-47/obj4?versionId=null 2023-05-31T20:18:03.903 DEBUG:boto:Data: 2023-05-31T20:18:03.904 DEBUG:boto:Headers: {} 2023-05-31T20:18:03.904 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:18:03.904 DEBUG:boto:Port: 8001 2023-05-31T20:18:03.905 DEBUG:boto:Params: {} 2023-05-31T20:18:03.905 DEBUG:boto:Token: None 2023-05-31T20:18:03.905 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:18:03 GMT /pklxkd-47/obj4?versionId=null 2023-05-31T20:18:03.906 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:F/2kEF3woF1B1ug048lFz4Mw82s= 2023-05-31T20:18:03.906 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:18:03 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:F/2kEF3woF1B1ug048lFz4Mw82s=', 'Content-Length': '0'} 2023-05-31T20:18:03.908 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:15:11 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000964d55ae853575a7-006477ab7b-162e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:18:03 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:18:03.909 DEBUG:boto:path=/pklxkd-47/obj4 2023-05-31T20:18:03.909 DEBUG:boto:auth_path=/pklxkd-47/obj4 2023-05-31T20:18:03.909 DEBUG:boto:path=/pklxkd-47/obj4?versionId=null 2023-05-31T20:18:03.910 DEBUG:boto:auth_path=/pklxkd-47/obj4?versionId=null 2023-05-31T20:18:03.910 DEBUG:boto:Method: HEAD 2023-05-31T20:18:03.910 DEBUG:boto:Path: /pklxkd-47/obj4?versionId=null 2023-05-31T20:18:03.911 DEBUG:boto:Data: 2023-05-31T20:18:03.911 DEBUG:boto:Headers: {} 2023-05-31T20:18:03.911 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:18:03.912 DEBUG:boto:Port: 8000 2023-05-31T20:18:03.912 DEBUG:boto:Params: {} 2023-05-31T20:18:03.912 DEBUG:boto:Token: None 2023-05-31T20:18:03.913 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:18:03 GMT /pklxkd-47/obj4?versionId=null 2023-05-31T20:18:03.914 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:F/2kEF3woF1B1ug048lFz4Mw82s= 2023-05-31T20:18:03.914 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:18:03 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:F/2kEF3woF1B1ug048lFz4Mw82s=', 'Content-Length': '0'} 2023-05-31T20:18:03.916 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:15:11 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000007f699d7a601effd8-006477ab7b-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:18:03 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:18:03.917 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2023-05-31T20:18:03.917 DEBUG:boto:path=/pklxkd-47/obj4 2023-05-31T20:18:03.918 DEBUG:boto:auth_path=/pklxkd-47/obj4 2023-05-31T20:18:03.918 DEBUG:boto:path=/pklxkd-47/obj4?versionId=null 2023-05-31T20:18:03.918 DEBUG:boto:auth_path=/pklxkd-47/obj4?versionId=null 2023-05-31T20:18:03.919 DEBUG:boto:Method: GET 2023-05-31T20:18:03.919 DEBUG:boto:Path: /pklxkd-47/obj4?versionId=null 2023-05-31T20:18:03.919 DEBUG:boto:Data: 2023-05-31T20:18:03.920 DEBUG:boto:Headers: {} 2023-05-31T20:18:03.920 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:18:03.921 DEBUG:boto:Port: 8001 2023-05-31T20:18:03.921 DEBUG:boto:Params: {} 2023-05-31T20:18:03.922 DEBUG:boto:Token: None 2023-05-31T20:18:03.922 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:18:03 GMT /pklxkd-47/obj4?versionId=null 2023-05-31T20:18:03.923 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:ouUH72EFrT7yES4b65ZlSYzwKcY= 2023-05-31T20:18:03.923 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:18:03 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:ouUH72EFrT7yES4b65ZlSYzwKcY=', 'Content-Length': '0'} 2023-05-31T20:18:03.925 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:15:11 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000c6bc4d15cde3a50d-006477ab7b-162e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:18:03 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:18:03.926 DEBUG:boto:path=/pklxkd-47/obj4 2023-05-31T20:18:03.926 DEBUG:boto:auth_path=/pklxkd-47/obj4 2023-05-31T20:18:03.926 DEBUG:boto:path=/pklxkd-47/obj4?versionId=null 2023-05-31T20:18:03.927 DEBUG:boto:auth_path=/pklxkd-47/obj4?versionId=null 2023-05-31T20:18:03.927 DEBUG:boto:Method: GET 2023-05-31T20:18:03.928 DEBUG:boto:Path: /pklxkd-47/obj4?versionId=null 2023-05-31T20:18:03.928 DEBUG:boto:Data: 2023-05-31T20:18:03.929 DEBUG:boto:Headers: {} 2023-05-31T20:18:03.930 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:18:03.930 DEBUG:boto:Port: 8000 2023-05-31T20:18:03.931 DEBUG:boto:Params: {} 2023-05-31T20:18:03.931 DEBUG:boto:Token: None 2023-05-31T20:18:03.932 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:18:03 GMT /pklxkd-47/obj4?versionId=null 2023-05-31T20:18:03.932 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:ouUH72EFrT7yES4b65ZlSYzwKcY= 2023-05-31T20:18:03.933 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:18:03 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:ouUH72EFrT7yES4b65ZlSYzwKcY=', 'Content-Length': '0'} 2023-05-31T20:18:03.935 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:15:11 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000a51435808f3aef29-006477ab7b-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:18:03 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:18:03.935 DEBUG:boto:path=/pklxkd-47/ 2023-05-31T20:18:03.936 DEBUG:boto:auth_path=/pklxkd-47/ 2023-05-31T20:18:03.936 DEBUG:boto:path=/pklxkd-47/?prefix=obj4 2023-05-31T20:18:03.937 DEBUG:boto:auth_path=/pklxkd-47/?prefix=obj4 2023-05-31T20:18:03.937 DEBUG:boto:Method: GET 2023-05-31T20:18:03.937 DEBUG:boto:Path: /pklxkd-47/?prefix=obj4 2023-05-31T20:18:03.938 DEBUG:boto:Data: 2023-05-31T20:18:03.938 DEBUG:boto:Headers: {} 2023-05-31T20:18:03.938 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:18:03.939 DEBUG:boto:Port: 8001 2023-05-31T20:18:03.939 DEBUG:boto:Params: {} 2023-05-31T20:18:03.939 DEBUG:boto:Token: None 2023-05-31T20:18:03.940 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:18:03 GMT /pklxkd-47/ 2023-05-31T20:18:03.940 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:7MgmJ0JDh4n8NkJuKJQxkQqJnhs= 2023-05-31T20:18:03.941 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:18:03 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:7MgmJ0JDh4n8NkJuKJQxkQqJnhs=', 'Content-Length': '0'} 2023-05-31T20:18:03.943 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ba0c8e553479b13e-006477ab7b-162e-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:18:03 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:18:03.988 DEBUG:boto:b'pklxkd-47obj41000falseobj42023-05-31T20:15:11.471Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:18:03.989 DEBUG:boto:path=/pklxkd-47/ 2023-05-31T20:18:03.990 DEBUG:boto:auth_path=/pklxkd-47/ 2023-05-31T20:18:03.990 DEBUG:boto:path=/pklxkd-47/?prefix=obj4 2023-05-31T20:18:03.991 DEBUG:boto:auth_path=/pklxkd-47/?prefix=obj4 2023-05-31T20:18:03.991 DEBUG:boto:Method: GET 2023-05-31T20:18:03.991 DEBUG:boto:Path: /pklxkd-47/?prefix=obj4 2023-05-31T20:18:03.992 DEBUG:boto:Data: 2023-05-31T20:18:03.993 DEBUG:boto:Headers: {} 2023-05-31T20:18:03.994 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:18:03.994 DEBUG:boto:Port: 8000 2023-05-31T20:18:03.994 DEBUG:boto:Params: {} 2023-05-31T20:18:03.995 DEBUG:boto:Token: None 2023-05-31T20:18:03.995 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:18:03 GMT /pklxkd-47/ 2023-05-31T20:18:03.996 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:7MgmJ0JDh4n8NkJuKJQxkQqJnhs= 2023-05-31T20:18:03.996 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:18:03 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:7MgmJ0JDh4n8NkJuKJQxkQqJnhs=', 'Content-Length': '0'} 2023-05-31T20:18:03.998 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f491fa6ee55cfe16-006477ab7b-1630-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:18:03 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:18:03.999 DEBUG:boto:b'pklxkd-47obj41000falseobj42023-05-31T20:15:11.471Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:18:04.000 DEBUG:boto:path=/pklxkd-47/obj4 2023-05-31T20:18:04.000 DEBUG:boto:auth_path=/pklxkd-47/obj4 2023-05-31T20:18:04.000 DEBUG:boto:Method: HEAD 2023-05-31T20:18:04.001 DEBUG:boto:Path: /pklxkd-47/obj4 2023-05-31T20:18:04.001 DEBUG:boto:Data: 2023-05-31T20:18:04.002 DEBUG:boto:Headers: {} 2023-05-31T20:18:04.002 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:18:04.002 DEBUG:boto:Port: 8001 2023-05-31T20:18:04.003 DEBUG:boto:Params: {} 2023-05-31T20:18:04.003 DEBUG:boto:Token: None 2023-05-31T20:18:04.003 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:18:04 GMT /pklxkd-47/obj4 2023-05-31T20:18:04.004 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:UtnbDrKAoAu2iALD/iu6LE79MOs= 2023-05-31T20:18:04.004 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:18:04 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:UtnbDrKAoAu2iALD/iu6LE79MOs=', 'Content-Length': '0'} 2023-05-31T20:18:04.006 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:15:11 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000b7a65d35b7cc5d2d-006477ab7c-162e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:18:04 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:18:04.007 DEBUG:boto:path=/pklxkd-47/obj4 2023-05-31T20:18:04.007 DEBUG:boto:auth_path=/pklxkd-47/obj4 2023-05-31T20:18:04.007 DEBUG:boto:Method: HEAD 2023-05-31T20:18:04.008 DEBUG:boto:Path: /pklxkd-47/obj4 2023-05-31T20:18:04.008 DEBUG:boto:Data: 2023-05-31T20:18:04.009 DEBUG:boto:Headers: {} 2023-05-31T20:18:04.009 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:18:04.009 DEBUG:boto:Port: 8000 2023-05-31T20:18:04.010 DEBUG:boto:Params: {} 2023-05-31T20:18:04.010 DEBUG:boto:Token: None 2023-05-31T20:18:04.011 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:18:04 GMT /pklxkd-47/obj4 2023-05-31T20:18:04.011 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:UtnbDrKAoAu2iALD/iu6LE79MOs= 2023-05-31T20:18:04.012 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:18:04 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:UtnbDrKAoAu2iALD/iu6LE79MOs=', 'Content-Length': '0'} 2023-05-31T20:18:04.013 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:15:11 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000d4b75540cda25278-006477ab7c-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:18:04 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:18:04.014 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2023-05-31T20:18:04.014 DEBUG:boto:path=/pklxkd-47/obj4 2023-05-31T20:18:04.015 DEBUG:boto:auth_path=/pklxkd-47/obj4 2023-05-31T20:18:04.015 DEBUG:boto:Method: GET 2023-05-31T20:18:04.015 DEBUG:boto:Path: /pklxkd-47/obj4 2023-05-31T20:18:04.016 DEBUG:boto:Data: 2023-05-31T20:18:04.017 DEBUG:boto:Headers: {} 2023-05-31T20:18:04.017 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:18:04.018 DEBUG:boto:Port: 8001 2023-05-31T20:18:04.018 DEBUG:boto:Params: {} 2023-05-31T20:18:04.019 DEBUG:boto:Token: None 2023-05-31T20:18:04.020 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:18:04 GMT /pklxkd-47/obj4 2023-05-31T20:18:04.021 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:GNU8VtuvHngahLJSxHIKp1l6PSc= 2023-05-31T20:18:04.021 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:18:04 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:GNU8VtuvHngahLJSxHIKp1l6PSc=', 'Content-Length': '0'} 2023-05-31T20:18:04.023 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:15:11 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000d06f9768dabfe5a4-006477ab7c-162e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:18:04 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:18:04.023 DEBUG:boto:path=/pklxkd-47/obj4 2023-05-31T20:18:04.024 DEBUG:boto:auth_path=/pklxkd-47/obj4 2023-05-31T20:18:04.024 DEBUG:boto:Method: GET 2023-05-31T20:18:04.025 DEBUG:boto:Path: /pklxkd-47/obj4 2023-05-31T20:18:04.025 DEBUG:boto:Data: 2023-05-31T20:18:04.025 DEBUG:boto:Headers: {} 2023-05-31T20:18:04.026 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:18:04.026 DEBUG:boto:Port: 8000 2023-05-31T20:18:04.027 DEBUG:boto:Params: {} 2023-05-31T20:18:04.027 DEBUG:boto:Token: None 2023-05-31T20:18:04.027 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:18:04 GMT /pklxkd-47/obj4 2023-05-31T20:18:04.028 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:GNU8VtuvHngahLJSxHIKp1l6PSc= 2023-05-31T20:18:04.028 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:18:04 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:GNU8VtuvHngahLJSxHIKp1l6PSc=', 'Content-Length': '0'} 2023-05-31T20:18:04.030 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:15:11 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000d6407da86ef4f876-006477ab7c-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:18:04 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:18:04.031 DEBUG:boto:path=/pklxkd-47/ 2023-05-31T20:18:04.033 DEBUG:boto:auth_path=/pklxkd-47/ 2023-05-31T20:18:04.033 DEBUG:boto:path=/pklxkd-47/?prefix=obj4 2023-05-31T20:18:04.033 DEBUG:boto:auth_path=/pklxkd-47/?prefix=obj4 2023-05-31T20:18:04.034 DEBUG:boto:Method: GET 2023-05-31T20:18:04.034 DEBUG:boto:Path: /pklxkd-47/?prefix=obj4 2023-05-31T20:18:04.035 DEBUG:boto:Data: 2023-05-31T20:18:04.035 DEBUG:boto:Headers: {} 2023-05-31T20:18:04.036 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:18:04.036 DEBUG:boto:Port: 8001 2023-05-31T20:18:04.036 DEBUG:boto:Params: {} 2023-05-31T20:18:04.037 DEBUG:boto:Token: None 2023-05-31T20:18:04.038 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:18:04 GMT /pklxkd-47/ 2023-05-31T20:18:04.038 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:piiraNTKk+9LIRTO8zlsdNhJrEw= 2023-05-31T20:18:04.039 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:18:04 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:piiraNTKk+9LIRTO8zlsdNhJrEw=', 'Content-Length': '0'} 2023-05-31T20:18:04.041 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000028642f05f1065676-006477ab7c-162e-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:18:04 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:18:04.042 DEBUG:boto:b'pklxkd-47obj41000falseobj42023-05-31T20:15:11.471Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:18:04.042 DEBUG:boto:path=/pklxkd-47/ 2023-05-31T20:18:04.043 DEBUG:boto:auth_path=/pklxkd-47/ 2023-05-31T20:18:04.043 DEBUG:boto:path=/pklxkd-47/?prefix=obj4 2023-05-31T20:18:04.044 DEBUG:boto:auth_path=/pklxkd-47/?prefix=obj4 2023-05-31T20:18:04.045 DEBUG:boto:Method: GET 2023-05-31T20:18:04.045 DEBUG:boto:Path: /pklxkd-47/?prefix=obj4 2023-05-31T20:18:04.045 DEBUG:boto:Data: 2023-05-31T20:18:04.046 DEBUG:boto:Headers: {} 2023-05-31T20:18:04.046 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:18:04.047 DEBUG:boto:Port: 8000 2023-05-31T20:18:04.047 DEBUG:boto:Params: {} 2023-05-31T20:18:04.048 DEBUG:boto:Token: None 2023-05-31T20:18:04.048 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:18:04 GMT /pklxkd-47/ 2023-05-31T20:18:04.049 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:piiraNTKk+9LIRTO8zlsdNhJrEw= 2023-05-31T20:18:04.049 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:18:04 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:piiraNTKk+9LIRTO8zlsdNhJrEw=', 'Content-Length': '0'} 2023-05-31T20:18:04.052 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b2ec8e694543ebb7-006477ab7c-1630-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:18:04 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:18:04.052 DEBUG:boto:b'pklxkd-47obj41000falseobj42023-05-31T20:15:11.471Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:18:04.053 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj5 2023-05-31T20:18:04.054 DEBUG:boto:path=/pklxkd-47/obj5 2023-05-31T20:18:04.055 DEBUG:boto:auth_path=/pklxkd-47/obj5 2023-05-31T20:18:04.055 DEBUG:boto:path=/pklxkd-47/obj5?versionId=null 2023-05-31T20:18:04.056 DEBUG:boto:auth_path=/pklxkd-47/obj5?versionId=null 2023-05-31T20:18:04.056 DEBUG:boto:Method: GET 2023-05-31T20:18:04.057 DEBUG:boto:Path: /pklxkd-47/obj5?versionId=null 2023-05-31T20:18:04.057 DEBUG:boto:Data: 2023-05-31T20:18:04.058 DEBUG:boto:Headers: {} 2023-05-31T20:18:04.058 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:18:04.059 DEBUG:boto:Port: 8001 2023-05-31T20:18:04.059 DEBUG:boto:Params: {} 2023-05-31T20:18:04.060 DEBUG:boto:Token: None 2023-05-31T20:18:04.060 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:18:04 GMT /pklxkd-47/obj5?versionId=null 2023-05-31T20:18:04.061 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:1tgC2pubwz6JIWRbVmnLijt09dQ= 2023-05-31T20:18:04.062 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:18:04 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:1tgC2pubwz6JIWRbVmnLijt09dQ=', 'Content-Length': '0'} 2023-05-31T20:18:04.064 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:17:25 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000983317c7a3f29f79-006477ab7c-162e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:18:04 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:18:04.065 DEBUG:boto:path=/pklxkd-47/obj5 2023-05-31T20:18:04.065 DEBUG:boto:auth_path=/pklxkd-47/obj5 2023-05-31T20:18:04.065 DEBUG:boto:path=/pklxkd-47/obj5?versionId=null 2023-05-31T20:18:04.066 DEBUG:boto:auth_path=/pklxkd-47/obj5?versionId=null 2023-05-31T20:18:04.066 DEBUG:boto:Method: GET 2023-05-31T20:18:04.067 DEBUG:boto:Path: /pklxkd-47/obj5?versionId=null 2023-05-31T20:18:04.067 DEBUG:boto:Data: 2023-05-31T20:18:04.068 DEBUG:boto:Headers: {} 2023-05-31T20:18:04.068 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:18:04.068 DEBUG:boto:Port: 8000 2023-05-31T20:18:04.069 DEBUG:boto:Params: {} 2023-05-31T20:18:04.069 DEBUG:boto:Token: None 2023-05-31T20:18:04.070 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:18:04 GMT /pklxkd-47/obj5?versionId=null 2023-05-31T20:18:04.070 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:1tgC2pubwz6JIWRbVmnLijt09dQ= 2023-05-31T20:18:04.071 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:18:04 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:1tgC2pubwz6JIWRbVmnLijt09dQ=', 'Content-Length': '0'} 2023-05-31T20:18:04.073 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:17:25 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000a57c0fb9d1eb2f0d-006477ab7c-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:18:04 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:18:04.074 DEBUG:boto:path=/pklxkd-47/obj5 2023-05-31T20:18:04.074 DEBUG:boto:auth_path=/pklxkd-47/obj5 2023-05-31T20:18:04.075 DEBUG:boto:path=/pklxkd-47/obj5?versionId=null 2023-05-31T20:18:04.075 DEBUG:boto:auth_path=/pklxkd-47/obj5?versionId=null 2023-05-31T20:18:04.075 DEBUG:boto:Method: HEAD 2023-05-31T20:18:04.076 DEBUG:boto:Path: /pklxkd-47/obj5?versionId=null 2023-05-31T20:18:04.076 DEBUG:boto:Data: 2023-05-31T20:18:04.076 DEBUG:boto:Headers: {} 2023-05-31T20:18:04.077 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:18:04.077 DEBUG:boto:Port: 8001 2023-05-31T20:18:04.078 DEBUG:boto:Params: {} 2023-05-31T20:18:04.078 DEBUG:boto:Token: None 2023-05-31T20:18:04.078 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:18:04 GMT /pklxkd-47/obj5?versionId=null 2023-05-31T20:18:04.079 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:KaAMXEoIOFppP1Vlc2Q8SAswaGc= 2023-05-31T20:18:04.079 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:18:04 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:KaAMXEoIOFppP1Vlc2Q8SAswaGc=', 'Content-Length': '0'} 2023-05-31T20:18:04.081 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:17:25 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000c18558e3ad0b7340-006477ab7c-162e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:18:04 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:18:04.082 DEBUG:boto:path=/pklxkd-47/obj5 2023-05-31T20:18:04.083 DEBUG:boto:auth_path=/pklxkd-47/obj5 2023-05-31T20:18:04.084 DEBUG:boto:path=/pklxkd-47/obj5?versionId=null 2023-05-31T20:18:04.084 DEBUG:boto:auth_path=/pklxkd-47/obj5?versionId=null 2023-05-31T20:18:04.084 DEBUG:boto:Method: HEAD 2023-05-31T20:18:04.085 DEBUG:boto:Path: /pklxkd-47/obj5?versionId=null 2023-05-31T20:18:04.085 DEBUG:boto:Data: 2023-05-31T20:18:04.085 DEBUG:boto:Headers: {} 2023-05-31T20:18:04.086 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:18:04.086 DEBUG:boto:Port: 8000 2023-05-31T20:18:04.087 DEBUG:boto:Params: {} 2023-05-31T20:18:04.087 DEBUG:boto:Token: None 2023-05-31T20:18:04.087 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:18:04 GMT /pklxkd-47/obj5?versionId=null 2023-05-31T20:18:04.088 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:KaAMXEoIOFppP1Vlc2Q8SAswaGc= 2023-05-31T20:18:04.088 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:18:04 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:KaAMXEoIOFppP1Vlc2Q8SAswaGc=', 'Content-Length': '0'} 2023-05-31T20:18:04.090 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:17:25 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000e638da48a425f0bd-006477ab7c-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:18:04 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:18:04.090 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj5 2023-05-31T20:18:04.091 DEBUG:boto:path=/pklxkd-47/obj5 2023-05-31T20:18:04.091 DEBUG:boto:auth_path=/pklxkd-47/obj5 2023-05-31T20:18:04.092 DEBUG:boto:path=/pklxkd-47/obj5?versionId=null 2023-05-31T20:18:04.092 DEBUG:boto:auth_path=/pklxkd-47/obj5?versionId=null 2023-05-31T20:18:04.093 DEBUG:boto:Method: GET 2023-05-31T20:18:04.093 DEBUG:boto:Path: /pklxkd-47/obj5?versionId=null 2023-05-31T20:18:04.094 DEBUG:boto:Data: 2023-05-31T20:18:04.094 DEBUG:boto:Headers: {} 2023-05-31T20:18:04.094 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:18:04.095 DEBUG:boto:Port: 8001 2023-05-31T20:18:04.095 DEBUG:boto:Params: {} 2023-05-31T20:18:04.096 DEBUG:boto:Token: None 2023-05-31T20:18:04.096 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:18:04 GMT /pklxkd-47/obj5?versionId=null 2023-05-31T20:18:04.097 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:1tgC2pubwz6JIWRbVmnLijt09dQ= 2023-05-31T20:18:04.097 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:18:04 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:1tgC2pubwz6JIWRbVmnLijt09dQ=', 'Content-Length': '0'} 2023-05-31T20:18:04.099 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:17:25 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000023bf3610adf0ba3b-006477ab7c-162e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:18:04 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:18:04.100 DEBUG:boto:path=/pklxkd-47/obj5 2023-05-31T20:18:04.100 DEBUG:boto:auth_path=/pklxkd-47/obj5 2023-05-31T20:18:04.101 DEBUG:boto:path=/pklxkd-47/obj5?versionId=null 2023-05-31T20:18:04.101 DEBUG:boto:auth_path=/pklxkd-47/obj5?versionId=null 2023-05-31T20:18:04.101 DEBUG:boto:Method: GET 2023-05-31T20:18:04.102 DEBUG:boto:Path: /pklxkd-47/obj5?versionId=null 2023-05-31T20:18:04.102 DEBUG:boto:Data: 2023-05-31T20:18:04.102 DEBUG:boto:Headers: {} 2023-05-31T20:18:04.103 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:18:04.103 DEBUG:boto:Port: 8000 2023-05-31T20:18:04.104 DEBUG:boto:Params: {} 2023-05-31T20:18:04.104 DEBUG:boto:Token: None 2023-05-31T20:18:04.104 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:18:04 GMT /pklxkd-47/obj5?versionId=null 2023-05-31T20:18:04.105 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:1tgC2pubwz6JIWRbVmnLijt09dQ= 2023-05-31T20:18:04.105 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:18:04 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:1tgC2pubwz6JIWRbVmnLijt09dQ=', 'Content-Length': '0'} 2023-05-31T20:18:04.107 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:17:25 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000e96064c5c971f31b-006477ab7c-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:18:04 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:18:04.108 DEBUG:boto:path=/pklxkd-47/ 2023-05-31T20:18:04.109 DEBUG:boto:auth_path=/pklxkd-47/ 2023-05-31T20:18:04.109 DEBUG:boto:path=/pklxkd-47/?prefix=obj5 2023-05-31T20:18:04.110 DEBUG:boto:auth_path=/pklxkd-47/?prefix=obj5 2023-05-31T20:18:04.110 DEBUG:boto:Method: GET 2023-05-31T20:18:04.110 DEBUG:boto:Path: /pklxkd-47/?prefix=obj5 2023-05-31T20:18:04.111 DEBUG:boto:Data: 2023-05-31T20:18:04.111 DEBUG:boto:Headers: {} 2023-05-31T20:18:04.111 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:18:04.112 DEBUG:boto:Port: 8001 2023-05-31T20:18:04.112 DEBUG:boto:Params: {} 2023-05-31T20:18:04.112 DEBUG:boto:Token: None 2023-05-31T20:18:04.113 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:18:04 GMT /pklxkd-47/ 2023-05-31T20:18:04.113 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:piiraNTKk+9LIRTO8zlsdNhJrEw= 2023-05-31T20:18:04.114 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:18:04 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:piiraNTKk+9LIRTO8zlsdNhJrEw=', 'Content-Length': '0'} 2023-05-31T20:18:04.116 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000bd84969ba1682cae-006477ab7c-162e-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:18:04 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:18:04.117 DEBUG:boto:b'pklxkd-47obj51000falseobj52023-05-31T20:17:25.752Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:18:04.117 DEBUG:boto:path=/pklxkd-47/ 2023-05-31T20:18:04.119 DEBUG:boto:auth_path=/pklxkd-47/ 2023-05-31T20:18:04.119 DEBUG:boto:path=/pklxkd-47/?prefix=obj5 2023-05-31T20:18:04.120 DEBUG:boto:auth_path=/pklxkd-47/?prefix=obj5 2023-05-31T20:18:04.120 DEBUG:boto:Method: GET 2023-05-31T20:18:04.121 DEBUG:boto:Path: /pklxkd-47/?prefix=obj5 2023-05-31T20:18:04.122 DEBUG:boto:Data: 2023-05-31T20:18:04.123 DEBUG:boto:Headers: {} 2023-05-31T20:18:04.124 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:18:04.125 DEBUG:boto:Port: 8000 2023-05-31T20:18:04.126 DEBUG:boto:Params: {} 2023-05-31T20:18:04.127 DEBUG:boto:Token: None 2023-05-31T20:18:04.128 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:18:04 GMT /pklxkd-47/ 2023-05-31T20:18:04.129 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:piiraNTKk+9LIRTO8zlsdNhJrEw= 2023-05-31T20:18:04.130 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:18:04 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:piiraNTKk+9LIRTO8zlsdNhJrEw=', 'Content-Length': '0'} 2023-05-31T20:18:04.132 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ac22443b4ad4a7ce-006477ab7c-1630-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:18:04 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:18:04.133 DEBUG:boto:b'pklxkd-47obj51000falseobj52023-05-31T20:17:25.752Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:18:04.134 DEBUG:boto:path=/pklxkd-47/obj5 2023-05-31T20:18:04.134 DEBUG:boto:auth_path=/pklxkd-47/obj5 2023-05-31T20:18:04.134 DEBUG:boto:Method: HEAD 2023-05-31T20:18:04.135 DEBUG:boto:Path: /pklxkd-47/obj5 2023-05-31T20:18:04.135 DEBUG:boto:Data: 2023-05-31T20:18:04.135 DEBUG:boto:Headers: {} 2023-05-31T20:18:04.136 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:18:04.136 DEBUG:boto:Port: 8001 2023-05-31T20:18:04.136 DEBUG:boto:Params: {} 2023-05-31T20:18:04.137 DEBUG:boto:Token: None 2023-05-31T20:18:04.137 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:18:04 GMT /pklxkd-47/obj5 2023-05-31T20:18:04.138 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:ntV9hLihVHuU9k30j+5xMIS2JSA= 2023-05-31T20:18:04.138 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:18:04 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:ntV9hLihVHuU9k30j+5xMIS2JSA=', 'Content-Length': '0'} 2023-05-31T20:18:04.140 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:17:25 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000078a9e348506b9078-006477ab7c-162e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:18:04 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:18:04.140 DEBUG:boto:path=/pklxkd-47/obj5 2023-05-31T20:18:04.141 DEBUG:boto:auth_path=/pklxkd-47/obj5 2023-05-31T20:18:04.141 DEBUG:boto:Method: HEAD 2023-05-31T20:18:04.141 DEBUG:boto:Path: /pklxkd-47/obj5 2023-05-31T20:18:04.142 DEBUG:boto:Data: 2023-05-31T20:18:04.142 DEBUG:boto:Headers: {} 2023-05-31T20:18:04.142 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:18:04.143 DEBUG:boto:Port: 8000 2023-05-31T20:18:04.143 DEBUG:boto:Params: {} 2023-05-31T20:18:04.143 DEBUG:boto:Token: None 2023-05-31T20:18:04.144 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:18:04 GMT /pklxkd-47/obj5 2023-05-31T20:18:04.144 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:ntV9hLihVHuU9k30j+5xMIS2JSA= 2023-05-31T20:18:04.145 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:18:04 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:ntV9hLihVHuU9k30j+5xMIS2JSA=', 'Content-Length': '0'} 2023-05-31T20:18:04.149 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:17:25 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000a4ea155c749b3896-006477ab7c-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:18:04 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:18:04.150 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj5 2023-05-31T20:18:04.150 DEBUG:boto:path=/pklxkd-47/obj5 2023-05-31T20:18:04.151 DEBUG:boto:auth_path=/pklxkd-47/obj5 2023-05-31T20:18:04.151 DEBUG:boto:Method: GET 2023-05-31T20:18:04.152 DEBUG:boto:Path: /pklxkd-47/obj5 2023-05-31T20:18:04.152 DEBUG:boto:Data: 2023-05-31T20:18:04.152 DEBUG:boto:Headers: {} 2023-05-31T20:18:04.153 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:18:04.153 DEBUG:boto:Port: 8001 2023-05-31T20:18:04.153 DEBUG:boto:Params: {} 2023-05-31T20:18:04.154 DEBUG:boto:Token: None 2023-05-31T20:18:04.155 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:18:04 GMT /pklxkd-47/obj5 2023-05-31T20:18:04.155 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:a5MbPsGGkxyXQjDIvb5kDMMdwRA= 2023-05-31T20:18:04.155 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:18:04 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:a5MbPsGGkxyXQjDIvb5kDMMdwRA=', 'Content-Length': '0'} 2023-05-31T20:18:04.157 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:17:25 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000006426bf68c7277cad-006477ab7c-162e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:18:04 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:18:04.204 DEBUG:boto:path=/pklxkd-47/obj5 2023-05-31T20:18:04.205 DEBUG:boto:auth_path=/pklxkd-47/obj5 2023-05-31T20:18:04.205 DEBUG:boto:Method: GET 2023-05-31T20:18:04.206 DEBUG:boto:Path: /pklxkd-47/obj5 2023-05-31T20:18:04.206 DEBUG:boto:Data: 2023-05-31T20:18:04.207 DEBUG:boto:Headers: {} 2023-05-31T20:18:04.207 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:18:04.208 DEBUG:boto:Port: 8000 2023-05-31T20:18:04.208 DEBUG:boto:Params: {} 2023-05-31T20:18:04.208 DEBUG:boto:Token: None 2023-05-31T20:18:04.209 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:18:04 GMT /pklxkd-47/obj5 2023-05-31T20:18:04.209 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:a5MbPsGGkxyXQjDIvb5kDMMdwRA= 2023-05-31T20:18:04.210 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:18:04 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:a5MbPsGGkxyXQjDIvb5kDMMdwRA=', 'Content-Length': '0'} 2023-05-31T20:18:04.211 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:17:25 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000c8ad5d9a39063ad8-006477ab7c-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:18:04 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:18:04.212 DEBUG:boto:path=/pklxkd-47/ 2023-05-31T20:18:04.213 DEBUG:boto:auth_path=/pklxkd-47/ 2023-05-31T20:18:04.213 DEBUG:boto:path=/pklxkd-47/?prefix=obj5 2023-05-31T20:18:04.213 DEBUG:boto:auth_path=/pklxkd-47/?prefix=obj5 2023-05-31T20:18:04.214 DEBUG:boto:Method: GET 2023-05-31T20:18:04.214 DEBUG:boto:Path: /pklxkd-47/?prefix=obj5 2023-05-31T20:18:04.214 DEBUG:boto:Data: 2023-05-31T20:18:04.215 DEBUG:boto:Headers: {} 2023-05-31T20:18:04.215 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:18:04.215 DEBUG:boto:Port: 8001 2023-05-31T20:18:04.216 DEBUG:boto:Params: {} 2023-05-31T20:18:04.216 DEBUG:boto:Token: None 2023-05-31T20:18:04.217 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:18:04 GMT /pklxkd-47/ 2023-05-31T20:18:04.217 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:piiraNTKk+9LIRTO8zlsdNhJrEw= 2023-05-31T20:18:04.218 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:18:04 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:piiraNTKk+9LIRTO8zlsdNhJrEw=', 'Content-Length': '0'} 2023-05-31T20:18:04.220 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000075e17ac975045be7-006477ab7c-162e-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:18:04 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:18:04.221 DEBUG:boto:b'pklxkd-47obj51000falseobj52023-05-31T20:17:25.752Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:18:04.221 DEBUG:boto:path=/pklxkd-47/ 2023-05-31T20:18:04.222 DEBUG:boto:auth_path=/pklxkd-47/ 2023-05-31T20:18:04.222 DEBUG:boto:path=/pklxkd-47/?prefix=obj5 2023-05-31T20:18:04.222 DEBUG:boto:auth_path=/pklxkd-47/?prefix=obj5 2023-05-31T20:18:04.223 DEBUG:boto:Method: GET 2023-05-31T20:18:04.223 DEBUG:boto:Path: /pklxkd-47/?prefix=obj5 2023-05-31T20:18:04.224 DEBUG:boto:Data: 2023-05-31T20:18:04.224 DEBUG:boto:Headers: {} 2023-05-31T20:18:04.224 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:18:04.225 DEBUG:boto:Port: 8000 2023-05-31T20:18:04.225 DEBUG:boto:Params: {} 2023-05-31T20:18:04.225 DEBUG:boto:Token: None 2023-05-31T20:18:04.226 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:18:04 GMT /pklxkd-47/ 2023-05-31T20:18:04.226 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:piiraNTKk+9LIRTO8zlsdNhJrEw= 2023-05-31T20:18:04.227 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:18:04 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:piiraNTKk+9LIRTO8zlsdNhJrEw=', 'Content-Length': '0'} 2023-05-31T20:18:04.229 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000099472c00ba19a40e-006477ab7c-1630-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:18:04 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:18:04.229 DEBUG:boto:b'pklxkd-47obj51000falseobj52023-05-31T20:17:25.752Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:18:04.230 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj6 2023-05-31T20:18:04.230 DEBUG:boto:path=/pklxkd-47/obj6 2023-05-31T20:18:04.231 DEBUG:boto:auth_path=/pklxkd-47/obj6 2023-05-31T20:18:04.231 DEBUG:boto:path=/pklxkd-47/obj6?versionId=null 2023-05-31T20:18:04.231 DEBUG:boto:auth_path=/pklxkd-47/obj6?versionId=null 2023-05-31T20:18:04.232 DEBUG:boto:Method: GET 2023-05-31T20:18:04.232 DEBUG:boto:Path: /pklxkd-47/obj6?versionId=null 2023-05-31T20:18:04.233 DEBUG:boto:Data: 2023-05-31T20:18:04.233 DEBUG:boto:Headers: {} 2023-05-31T20:18:04.233 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:18:04.234 DEBUG:boto:Port: 8001 2023-05-31T20:18:04.234 DEBUG:boto:Params: {} 2023-05-31T20:18:04.234 DEBUG:boto:Token: None 2023-05-31T20:18:04.235 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:18:04 GMT /pklxkd-47/obj6?versionId=null 2023-05-31T20:18:04.235 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:dr+iIA6mnLvam5kL9U9zQk7+fGA= 2023-05-31T20:18:04.236 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:18:04 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:dr+iIA6mnLvam5kL9U9zQk7+fGA=', 'Content-Length': '0'} 2023-05-31T20:18:04.237 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:17:25 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000004b18b7a9db2268c4-006477ab7c-162e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:18:04 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:18:04.238 DEBUG:boto:path=/pklxkd-47/obj6 2023-05-31T20:18:04.238 DEBUG:boto:auth_path=/pklxkd-47/obj6 2023-05-31T20:18:04.239 DEBUG:boto:path=/pklxkd-47/obj6?versionId=null 2023-05-31T20:18:04.240 DEBUG:boto:auth_path=/pklxkd-47/obj6?versionId=null 2023-05-31T20:18:04.240 DEBUG:boto:Method: GET 2023-05-31T20:18:04.240 DEBUG:boto:Path: /pklxkd-47/obj6?versionId=null 2023-05-31T20:18:04.241 DEBUG:boto:Data: 2023-05-31T20:18:04.241 DEBUG:boto:Headers: {} 2023-05-31T20:18:04.241 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:18:04.242 DEBUG:boto:Port: 8000 2023-05-31T20:18:04.242 DEBUG:boto:Params: {} 2023-05-31T20:18:04.243 DEBUG:boto:Token: None 2023-05-31T20:18:04.243 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:18:04 GMT /pklxkd-47/obj6?versionId=null 2023-05-31T20:18:04.243 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:dr+iIA6mnLvam5kL9U9zQk7+fGA= 2023-05-31T20:18:04.244 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:18:04 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:dr+iIA6mnLvam5kL9U9zQk7+fGA=', 'Content-Length': '0'} 2023-05-31T20:18:04.246 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:17:25 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000614364fb9a298f35-006477ab7c-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:18:04 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:18:04.246 DEBUG:boto:path=/pklxkd-47/obj6 2023-05-31T20:18:04.247 DEBUG:boto:auth_path=/pklxkd-47/obj6 2023-05-31T20:18:04.247 DEBUG:boto:path=/pklxkd-47/obj6?versionId=null 2023-05-31T20:18:04.247 DEBUG:boto:auth_path=/pklxkd-47/obj6?versionId=null 2023-05-31T20:18:04.248 DEBUG:boto:Method: HEAD 2023-05-31T20:18:04.248 DEBUG:boto:Path: /pklxkd-47/obj6?versionId=null 2023-05-31T20:18:04.248 DEBUG:boto:Data: 2023-05-31T20:18:04.249 DEBUG:boto:Headers: {} 2023-05-31T20:18:04.250 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:18:04.250 DEBUG:boto:Port: 8001 2023-05-31T20:18:04.250 DEBUG:boto:Params: {} 2023-05-31T20:18:04.251 DEBUG:boto:Token: None 2023-05-31T20:18:04.251 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:18:04 GMT /pklxkd-47/obj6?versionId=null 2023-05-31T20:18:04.252 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:gETeQQziBp0XqK9ETkksKpH5JXE= 2023-05-31T20:18:04.252 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:18:04 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:gETeQQziBp0XqK9ETkksKpH5JXE=', 'Content-Length': '0'} 2023-05-31T20:18:04.254 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:17:25 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000014b04444ade361d1-006477ab7c-162e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:18:04 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:18:04.254 DEBUG:boto:path=/pklxkd-47/obj6 2023-05-31T20:18:04.255 DEBUG:boto:auth_path=/pklxkd-47/obj6 2023-05-31T20:18:04.255 DEBUG:boto:path=/pklxkd-47/obj6?versionId=null 2023-05-31T20:18:04.255 DEBUG:boto:auth_path=/pklxkd-47/obj6?versionId=null 2023-05-31T20:18:04.256 DEBUG:boto:Method: HEAD 2023-05-31T20:18:04.256 DEBUG:boto:Path: /pklxkd-47/obj6?versionId=null 2023-05-31T20:18:04.256 DEBUG:boto:Data: 2023-05-31T20:18:04.257 DEBUG:boto:Headers: {} 2023-05-31T20:18:04.257 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:18:04.257 DEBUG:boto:Port: 8000 2023-05-31T20:18:04.258 DEBUG:boto:Params: {} 2023-05-31T20:18:04.258 DEBUG:boto:Token: None 2023-05-31T20:18:04.258 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:18:04 GMT /pklxkd-47/obj6?versionId=null 2023-05-31T20:18:04.259 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:gETeQQziBp0XqK9ETkksKpH5JXE= 2023-05-31T20:18:04.259 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:18:04 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:gETeQQziBp0XqK9ETkksKpH5JXE=', 'Content-Length': '0'} 2023-05-31T20:18:04.261 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:17:25 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000008eef538fc8f335c8-006477ab7c-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:18:04 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:18:04.262 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj6 2023-05-31T20:18:04.262 DEBUG:boto:path=/pklxkd-47/obj6 2023-05-31T20:18:04.263 DEBUG:boto:auth_path=/pklxkd-47/obj6 2023-05-31T20:18:04.263 DEBUG:boto:path=/pklxkd-47/obj6?versionId=null 2023-05-31T20:18:04.263 DEBUG:boto:auth_path=/pklxkd-47/obj6?versionId=null 2023-05-31T20:18:04.264 DEBUG:boto:Method: GET 2023-05-31T20:18:04.264 DEBUG:boto:Path: /pklxkd-47/obj6?versionId=null 2023-05-31T20:18:04.264 DEBUG:boto:Data: 2023-05-31T20:18:04.265 DEBUG:boto:Headers: {} 2023-05-31T20:18:04.266 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:18:04.266 DEBUG:boto:Port: 8001 2023-05-31T20:18:04.266 DEBUG:boto:Params: {} 2023-05-31T20:18:04.267 DEBUG:boto:Token: None 2023-05-31T20:18:04.267 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:18:04 GMT /pklxkd-47/obj6?versionId=null 2023-05-31T20:18:04.268 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:dr+iIA6mnLvam5kL9U9zQk7+fGA= 2023-05-31T20:18:04.268 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:18:04 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:dr+iIA6mnLvam5kL9U9zQk7+fGA=', 'Content-Length': '0'} 2023-05-31T20:18:04.270 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:17:25 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000050251c222820e0a1-006477ab7c-162e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:18:04 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:18:04.271 DEBUG:boto:path=/pklxkd-47/obj6 2023-05-31T20:18:04.271 DEBUG:boto:auth_path=/pklxkd-47/obj6 2023-05-31T20:18:04.272 DEBUG:boto:path=/pklxkd-47/obj6?versionId=null 2023-05-31T20:18:04.272 DEBUG:boto:auth_path=/pklxkd-47/obj6?versionId=null 2023-05-31T20:18:04.273 DEBUG:boto:Method: GET 2023-05-31T20:18:04.273 DEBUG:boto:Path: /pklxkd-47/obj6?versionId=null 2023-05-31T20:18:04.273 DEBUG:boto:Data: 2023-05-31T20:18:04.274 DEBUG:boto:Headers: {} 2023-05-31T20:18:04.274 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:18:04.274 DEBUG:boto:Port: 8000 2023-05-31T20:18:04.275 DEBUG:boto:Params: {} 2023-05-31T20:18:04.275 DEBUG:boto:Token: None 2023-05-31T20:18:04.276 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:18:04 GMT /pklxkd-47/obj6?versionId=null 2023-05-31T20:18:04.276 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:dr+iIA6mnLvam5kL9U9zQk7+fGA= 2023-05-31T20:18:04.276 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:18:04 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:dr+iIA6mnLvam5kL9U9zQk7+fGA=', 'Content-Length': '0'} 2023-05-31T20:18:04.278 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:17:25 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000545504fae2213ea7-006477ab7c-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:18:04 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:18:04.279 DEBUG:boto:path=/pklxkd-47/ 2023-05-31T20:18:04.279 DEBUG:boto:auth_path=/pklxkd-47/ 2023-05-31T20:18:04.279 DEBUG:boto:path=/pklxkd-47/?prefix=obj6 2023-05-31T20:18:04.280 DEBUG:boto:auth_path=/pklxkd-47/?prefix=obj6 2023-05-31T20:18:04.280 DEBUG:boto:Method: GET 2023-05-31T20:18:04.281 DEBUG:boto:Path: /pklxkd-47/?prefix=obj6 2023-05-31T20:18:04.281 DEBUG:boto:Data: 2023-05-31T20:18:04.281 DEBUG:boto:Headers: {} 2023-05-31T20:18:04.282 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:18:04.282 DEBUG:boto:Port: 8001 2023-05-31T20:18:04.282 DEBUG:boto:Params: {} 2023-05-31T20:18:04.283 DEBUG:boto:Token: None 2023-05-31T20:18:04.283 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:18:04 GMT /pklxkd-47/ 2023-05-31T20:18:04.284 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:piiraNTKk+9LIRTO8zlsdNhJrEw= 2023-05-31T20:18:04.284 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:18:04 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:piiraNTKk+9LIRTO8zlsdNhJrEw=', 'Content-Length': '0'} 2023-05-31T20:18:04.286 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000017828f859b2f26da-006477ab7c-162e-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:18:04 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:18:04.287 DEBUG:boto:b'pklxkd-47obj61000falseobj62023-05-31T20:17:25.769Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:18:04.288 DEBUG:boto:path=/pklxkd-47/ 2023-05-31T20:18:04.288 DEBUG:boto:auth_path=/pklxkd-47/ 2023-05-31T20:18:04.288 DEBUG:boto:path=/pklxkd-47/?prefix=obj6 2023-05-31T20:18:04.289 DEBUG:boto:auth_path=/pklxkd-47/?prefix=obj6 2023-05-31T20:18:04.289 DEBUG:boto:Method: GET 2023-05-31T20:18:04.289 DEBUG:boto:Path: /pklxkd-47/?prefix=obj6 2023-05-31T20:18:04.290 DEBUG:boto:Data: 2023-05-31T20:18:04.290 DEBUG:boto:Headers: {} 2023-05-31T20:18:04.291 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:18:04.291 DEBUG:boto:Port: 8000 2023-05-31T20:18:04.291 DEBUG:boto:Params: {} 2023-05-31T20:18:04.292 DEBUG:boto:Token: None 2023-05-31T20:18:04.292 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:18:04 GMT /pklxkd-47/ 2023-05-31T20:18:04.293 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:piiraNTKk+9LIRTO8zlsdNhJrEw= 2023-05-31T20:18:04.293 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:18:04 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:piiraNTKk+9LIRTO8zlsdNhJrEw=', 'Content-Length': '0'} 2023-05-31T20:18:04.295 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000003148b4b5f37e1475-006477ab7c-1630-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:18:04 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:18:04.296 DEBUG:boto:b'pklxkd-47obj61000falseobj62023-05-31T20:17:25.769Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:18:04.296 DEBUG:boto:path=/pklxkd-47/obj6 2023-05-31T20:18:04.297 DEBUG:boto:auth_path=/pklxkd-47/obj6 2023-05-31T20:18:04.297 DEBUG:boto:Method: HEAD 2023-05-31T20:18:04.298 DEBUG:boto:Path: /pklxkd-47/obj6 2023-05-31T20:18:04.298 DEBUG:boto:Data: 2023-05-31T20:18:04.298 DEBUG:boto:Headers: {} 2023-05-31T20:18:04.299 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:18:04.299 DEBUG:boto:Port: 8001 2023-05-31T20:18:04.299 DEBUG:boto:Params: {} 2023-05-31T20:18:04.300 DEBUG:boto:Token: None 2023-05-31T20:18:04.300 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:18:04 GMT /pklxkd-47/obj6 2023-05-31T20:18:04.300 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:0TC6bMUuhbM60nbdvyH864kQD9Q= 2023-05-31T20:18:04.301 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:18:04 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:0TC6bMUuhbM60nbdvyH864kQD9Q=', 'Content-Length': '0'} 2023-05-31T20:18:04.303 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:17:25 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000fff9d8e35838f4f6-006477ab7c-162e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:18:04 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:18:04.304 DEBUG:boto:path=/pklxkd-47/obj6 2023-05-31T20:18:04.305 DEBUG:boto:auth_path=/pklxkd-47/obj6 2023-05-31T20:18:04.305 DEBUG:boto:Method: HEAD 2023-05-31T20:18:04.306 DEBUG:boto:Path: /pklxkd-47/obj6 2023-05-31T20:18:04.307 DEBUG:boto:Data: 2023-05-31T20:18:04.307 DEBUG:boto:Headers: {} 2023-05-31T20:18:04.308 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:18:04.309 DEBUG:boto:Port: 8000 2023-05-31T20:18:04.310 DEBUG:boto:Params: {} 2023-05-31T20:18:04.311 DEBUG:boto:Token: None 2023-05-31T20:18:04.312 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:18:04 GMT /pklxkd-47/obj6 2023-05-31T20:18:04.313 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:0TC6bMUuhbM60nbdvyH864kQD9Q= 2023-05-31T20:18:04.314 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:18:04 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:0TC6bMUuhbM60nbdvyH864kQD9Q=', 'Content-Length': '0'} 2023-05-31T20:18:04.316 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:17:25 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000244335e36001651b-006477ab7c-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:18:04 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:18:04.317 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj6 2023-05-31T20:18:04.318 DEBUG:boto:path=/pklxkd-47/obj6 2023-05-31T20:18:04.318 DEBUG:boto:auth_path=/pklxkd-47/obj6 2023-05-31T20:18:04.319 DEBUG:boto:Method: GET 2023-05-31T20:18:04.320 DEBUG:boto:Path: /pklxkd-47/obj6 2023-05-31T20:18:04.320 DEBUG:boto:Data: 2023-05-31T20:18:04.320 DEBUG:boto:Headers: {} 2023-05-31T20:18:04.321 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:18:04.321 DEBUG:boto:Port: 8001 2023-05-31T20:18:04.322 DEBUG:boto:Params: {} 2023-05-31T20:18:04.322 DEBUG:boto:Token: None 2023-05-31T20:18:04.322 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:18:04 GMT /pklxkd-47/obj6 2023-05-31T20:18:04.323 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:qrA7wSEs6w8R0+5T1sR4EJb2ErU= 2023-05-31T20:18:04.323 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:18:04 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:qrA7wSEs6w8R0+5T1sR4EJb2ErU=', 'Content-Length': '0'} 2023-05-31T20:18:04.325 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:17:25 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000b2e0b00f2f0d5601-006477ab7c-162e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:18:04 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:18:04.326 DEBUG:boto:path=/pklxkd-47/obj6 2023-05-31T20:18:04.326 DEBUG:boto:auth_path=/pklxkd-47/obj6 2023-05-31T20:18:04.326 DEBUG:boto:Method: GET 2023-05-31T20:18:04.327 DEBUG:boto:Path: /pklxkd-47/obj6 2023-05-31T20:18:04.327 DEBUG:boto:Data: 2023-05-31T20:18:04.327 DEBUG:boto:Headers: {} 2023-05-31T20:18:04.328 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:18:04.328 DEBUG:boto:Port: 8000 2023-05-31T20:18:04.329 DEBUG:boto:Params: {} 2023-05-31T20:18:04.329 DEBUG:boto:Token: None 2023-05-31T20:18:04.330 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:18:04 GMT /pklxkd-47/obj6 2023-05-31T20:18:04.330 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:qrA7wSEs6w8R0+5T1sR4EJb2ErU= 2023-05-31T20:18:04.331 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:18:04 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:qrA7wSEs6w8R0+5T1sR4EJb2ErU=', 'Content-Length': '0'} 2023-05-31T20:18:04.333 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:17:25 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000019d9c16d66ed4e51-006477ab7c-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:18:04 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:18:04.333 DEBUG:boto:path=/pklxkd-47/ 2023-05-31T20:18:04.334 DEBUG:boto:auth_path=/pklxkd-47/ 2023-05-31T20:18:04.334 DEBUG:boto:path=/pklxkd-47/?prefix=obj6 2023-05-31T20:18:04.334 DEBUG:boto:auth_path=/pklxkd-47/?prefix=obj6 2023-05-31T20:18:04.335 DEBUG:boto:Method: GET 2023-05-31T20:18:04.335 DEBUG:boto:Path: /pklxkd-47/?prefix=obj6 2023-05-31T20:18:04.335 DEBUG:boto:Data: 2023-05-31T20:18:04.336 DEBUG:boto:Headers: {} 2023-05-31T20:18:04.336 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:18:04.337 DEBUG:boto:Port: 8001 2023-05-31T20:18:04.337 DEBUG:boto:Params: {} 2023-05-31T20:18:04.338 DEBUG:boto:Token: None 2023-05-31T20:18:04.338 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:18:04 GMT /pklxkd-47/ 2023-05-31T20:18:04.339 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:piiraNTKk+9LIRTO8zlsdNhJrEw= 2023-05-31T20:18:04.339 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:18:04 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:piiraNTKk+9LIRTO8zlsdNhJrEw=', 'Content-Length': '0'} 2023-05-31T20:18:04.341 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000008b7ff3e7ccce520c-006477ab7c-162e-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:18:04 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:18:04.384 DEBUG:boto:b'pklxkd-47obj61000falseobj62023-05-31T20:17:25.769Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:18:04.385 DEBUG:boto:path=/pklxkd-47/ 2023-05-31T20:18:04.386 DEBUG:boto:auth_path=/pklxkd-47/ 2023-05-31T20:18:04.386 DEBUG:boto:path=/pklxkd-47/?prefix=obj6 2023-05-31T20:18:04.386 DEBUG:boto:auth_path=/pklxkd-47/?prefix=obj6 2023-05-31T20:18:04.387 DEBUG:boto:Method: GET 2023-05-31T20:18:04.387 DEBUG:boto:Path: /pklxkd-47/?prefix=obj6 2023-05-31T20:18:04.388 DEBUG:boto:Data: 2023-05-31T20:18:04.388 DEBUG:boto:Headers: {} 2023-05-31T20:18:04.388 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:18:04.389 DEBUG:boto:Port: 8000 2023-05-31T20:18:04.390 DEBUG:boto:Params: {} 2023-05-31T20:18:04.391 DEBUG:boto:Token: None 2023-05-31T20:18:04.391 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:18:04 GMT /pklxkd-47/ 2023-05-31T20:18:04.391 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:piiraNTKk+9LIRTO8zlsdNhJrEw= 2023-05-31T20:18:04.392 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:18:04 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:piiraNTKk+9LIRTO8zlsdNhJrEw=', 'Content-Length': '0'} 2023-05-31T20:18:04.394 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000dcb30d746ba7f40b-006477ab7c-1630-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:18:04 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:18:04.395 DEBUG:boto:b'pklxkd-47obj61000falseobj62023-05-31T20:17:25.769Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:18:04.396 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj7 2023-05-31T20:18:04.397 DEBUG:boto:path=/pklxkd-47/obj7 2023-05-31T20:18:04.397 DEBUG:boto:auth_path=/pklxkd-47/obj7 2023-05-31T20:18:04.397 DEBUG:boto:path=/pklxkd-47/obj7?versionId=null 2023-05-31T20:18:04.398 DEBUG:boto:auth_path=/pklxkd-47/obj7?versionId=null 2023-05-31T20:18:04.398 DEBUG:boto:Method: GET 2023-05-31T20:18:04.399 DEBUG:boto:Path: /pklxkd-47/obj7?versionId=null 2023-05-31T20:18:04.399 DEBUG:boto:Data: 2023-05-31T20:18:04.400 DEBUG:boto:Headers: {} 2023-05-31T20:18:04.400 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:18:04.401 DEBUG:boto:Port: 8001 2023-05-31T20:18:04.401 DEBUG:boto:Params: {} 2023-05-31T20:18:04.402 DEBUG:boto:Token: None 2023-05-31T20:18:04.402 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:18:04 GMT /pklxkd-47/obj7?versionId=null 2023-05-31T20:18:04.403 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:6S/wA4vX5WZYTcChnvbEPetgAM0= 2023-05-31T20:18:04.403 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:18:04 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:6S/wA4vX5WZYTcChnvbEPetgAM0=', 'Content-Length': '0'} 2023-05-31T20:18:04.405 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:17:25 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000005748487537fa7297-006477ab7c-162e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:18:04 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:18:04.406 DEBUG:boto:path=/pklxkd-47/obj7 2023-05-31T20:18:04.406 DEBUG:boto:auth_path=/pklxkd-47/obj7 2023-05-31T20:18:04.406 DEBUG:boto:path=/pklxkd-47/obj7?versionId=null 2023-05-31T20:18:04.407 DEBUG:boto:auth_path=/pklxkd-47/obj7?versionId=null 2023-05-31T20:18:04.407 DEBUG:boto:Method: GET 2023-05-31T20:18:04.408 DEBUG:boto:Path: /pklxkd-47/obj7?versionId=null 2023-05-31T20:18:04.408 DEBUG:boto:Data: 2023-05-31T20:18:04.408 DEBUG:boto:Headers: {} 2023-05-31T20:18:04.409 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:18:04.409 DEBUG:boto:Port: 8000 2023-05-31T20:18:04.409 DEBUG:boto:Params: {} 2023-05-31T20:18:04.410 DEBUG:boto:Token: None 2023-05-31T20:18:04.411 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:18:04 GMT /pklxkd-47/obj7?versionId=null 2023-05-31T20:18:04.411 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:6S/wA4vX5WZYTcChnvbEPetgAM0= 2023-05-31T20:18:04.412 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:18:04 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:6S/wA4vX5WZYTcChnvbEPetgAM0=', 'Content-Length': '0'} 2023-05-31T20:18:04.414 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:17:25 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000f82ff72a495c027f-006477ab7c-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:18:04 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:18:04.414 DEBUG:boto:path=/pklxkd-47/obj7 2023-05-31T20:18:04.415 DEBUG:boto:auth_path=/pklxkd-47/obj7 2023-05-31T20:18:04.415 DEBUG:boto:path=/pklxkd-47/obj7?versionId=null 2023-05-31T20:18:04.415 DEBUG:boto:auth_path=/pklxkd-47/obj7?versionId=null 2023-05-31T20:18:04.416 DEBUG:boto:Method: HEAD 2023-05-31T20:18:04.416 DEBUG:boto:Path: /pklxkd-47/obj7?versionId=null 2023-05-31T20:18:04.416 DEBUG:boto:Data: 2023-05-31T20:18:04.417 DEBUG:boto:Headers: {} 2023-05-31T20:18:04.417 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:18:04.417 DEBUG:boto:Port: 8001 2023-05-31T20:18:04.418 DEBUG:boto:Params: {} 2023-05-31T20:18:04.418 DEBUG:boto:Token: None 2023-05-31T20:18:04.419 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:18:04 GMT /pklxkd-47/obj7?versionId=null 2023-05-31T20:18:04.419 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:LkDM4q5XWxguqLNr6m3uCbKgMks= 2023-05-31T20:18:04.419 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:18:04 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:LkDM4q5XWxguqLNr6m3uCbKgMks=', 'Content-Length': '0'} 2023-05-31T20:18:04.421 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:17:25 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000068d1d5622cda8816-006477ab7c-162e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:18:04 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:18:04.422 DEBUG:boto:path=/pklxkd-47/obj7 2023-05-31T20:18:04.422 DEBUG:boto:auth_path=/pklxkd-47/obj7 2023-05-31T20:18:04.423 DEBUG:boto:path=/pklxkd-47/obj7?versionId=null 2023-05-31T20:18:04.423 DEBUG:boto:auth_path=/pklxkd-47/obj7?versionId=null 2023-05-31T20:18:04.424 DEBUG:boto:Method: HEAD 2023-05-31T20:18:04.424 DEBUG:boto:Path: /pklxkd-47/obj7?versionId=null 2023-05-31T20:18:04.424 DEBUG:boto:Data: 2023-05-31T20:18:04.425 DEBUG:boto:Headers: {} 2023-05-31T20:18:04.425 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:18:04.425 DEBUG:boto:Port: 8000 2023-05-31T20:18:04.426 DEBUG:boto:Params: {} 2023-05-31T20:18:04.426 DEBUG:boto:Token: None 2023-05-31T20:18:04.427 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:18:04 GMT /pklxkd-47/obj7?versionId=null 2023-05-31T20:18:04.427 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:LkDM4q5XWxguqLNr6m3uCbKgMks= 2023-05-31T20:18:04.427 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:18:04 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:LkDM4q5XWxguqLNr6m3uCbKgMks=', 'Content-Length': '0'} 2023-05-31T20:18:04.429 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:17:25 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000d68e140db958629d-006477ab7c-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:18:04 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:18:04.430 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj7 2023-05-31T20:18:04.430 DEBUG:boto:path=/pklxkd-47/obj7 2023-05-31T20:18:04.430 DEBUG:boto:auth_path=/pklxkd-47/obj7 2023-05-31T20:18:04.431 DEBUG:boto:path=/pklxkd-47/obj7?versionId=null 2023-05-31T20:18:04.432 DEBUG:boto:auth_path=/pklxkd-47/obj7?versionId=null 2023-05-31T20:18:04.432 DEBUG:boto:Method: GET 2023-05-31T20:18:04.433 DEBUG:boto:Path: /pklxkd-47/obj7?versionId=null 2023-05-31T20:18:04.433 DEBUG:boto:Data: 2023-05-31T20:18:04.434 DEBUG:boto:Headers: {} 2023-05-31T20:18:04.434 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:18:04.434 DEBUG:boto:Port: 8001 2023-05-31T20:18:04.435 DEBUG:boto:Params: {} 2023-05-31T20:18:04.435 DEBUG:boto:Token: None 2023-05-31T20:18:04.436 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:18:04 GMT /pklxkd-47/obj7?versionId=null 2023-05-31T20:18:04.436 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:6S/wA4vX5WZYTcChnvbEPetgAM0= 2023-05-31T20:18:04.436 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:18:04 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:6S/wA4vX5WZYTcChnvbEPetgAM0=', 'Content-Length': '0'} 2023-05-31T20:18:04.439 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:17:25 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000015bc3b7a9ad938da-006477ab7c-162e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:18:04 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:18:04.440 DEBUG:boto:path=/pklxkd-47/obj7 2023-05-31T20:18:04.440 DEBUG:boto:auth_path=/pklxkd-47/obj7 2023-05-31T20:18:04.440 DEBUG:boto:path=/pklxkd-47/obj7?versionId=null 2023-05-31T20:18:04.441 DEBUG:boto:auth_path=/pklxkd-47/obj7?versionId=null 2023-05-31T20:18:04.441 DEBUG:boto:Method: GET 2023-05-31T20:18:04.441 DEBUG:boto:Path: /pklxkd-47/obj7?versionId=null 2023-05-31T20:18:04.442 DEBUG:boto:Data: 2023-05-31T20:18:04.442 DEBUG:boto:Headers: {} 2023-05-31T20:18:04.443 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:18:04.443 DEBUG:boto:Port: 8000 2023-05-31T20:18:04.444 DEBUG:boto:Params: {} 2023-05-31T20:18:04.444 DEBUG:boto:Token: None 2023-05-31T20:18:04.445 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:18:04 GMT /pklxkd-47/obj7?versionId=null 2023-05-31T20:18:04.445 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:6S/wA4vX5WZYTcChnvbEPetgAM0= 2023-05-31T20:18:04.446 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:18:04 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:6S/wA4vX5WZYTcChnvbEPetgAM0=', 'Content-Length': '0'} 2023-05-31T20:18:04.447 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:17:25 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000007786b6807c56457d-006477ab7c-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:18:04 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:18:04.448 DEBUG:boto:path=/pklxkd-47/ 2023-05-31T20:18:04.448 DEBUG:boto:auth_path=/pklxkd-47/ 2023-05-31T20:18:04.449 DEBUG:boto:path=/pklxkd-47/?prefix=obj7 2023-05-31T20:18:04.449 DEBUG:boto:auth_path=/pklxkd-47/?prefix=obj7 2023-05-31T20:18:04.450 DEBUG:boto:Method: GET 2023-05-31T20:18:04.450 DEBUG:boto:Path: /pklxkd-47/?prefix=obj7 2023-05-31T20:18:04.450 DEBUG:boto:Data: 2023-05-31T20:18:04.451 DEBUG:boto:Headers: {} 2023-05-31T20:18:04.451 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:18:04.451 DEBUG:boto:Port: 8001 2023-05-31T20:18:04.452 DEBUG:boto:Params: {} 2023-05-31T20:18:04.452 DEBUG:boto:Token: None 2023-05-31T20:18:04.453 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:18:04 GMT /pklxkd-47/ 2023-05-31T20:18:04.453 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:piiraNTKk+9LIRTO8zlsdNhJrEw= 2023-05-31T20:18:04.454 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:18:04 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:piiraNTKk+9LIRTO8zlsdNhJrEw=', 'Content-Length': '0'} 2023-05-31T20:18:04.456 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000483b43f3b4e88f92-006477ab7c-162e-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:18:04 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:18:04.456 DEBUG:boto:b'pklxkd-47obj71000falseobj72023-05-31T20:17:25.833Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:18:04.457 DEBUG:boto:path=/pklxkd-47/ 2023-05-31T20:18:04.458 DEBUG:boto:auth_path=/pklxkd-47/ 2023-05-31T20:18:04.458 DEBUG:boto:path=/pklxkd-47/?prefix=obj7 2023-05-31T20:18:04.459 DEBUG:boto:auth_path=/pklxkd-47/?prefix=obj7 2023-05-31T20:18:04.459 DEBUG:boto:Method: GET 2023-05-31T20:18:04.459 DEBUG:boto:Path: /pklxkd-47/?prefix=obj7 2023-05-31T20:18:04.460 DEBUG:boto:Data: 2023-05-31T20:18:04.460 DEBUG:boto:Headers: {} 2023-05-31T20:18:04.461 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:18:04.461 DEBUG:boto:Port: 8000 2023-05-31T20:18:04.461 DEBUG:boto:Params: {} 2023-05-31T20:18:04.462 DEBUG:boto:Token: None 2023-05-31T20:18:04.462 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:18:04 GMT /pklxkd-47/ 2023-05-31T20:18:04.463 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:piiraNTKk+9LIRTO8zlsdNhJrEw= 2023-05-31T20:18:04.463 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:18:04 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:piiraNTKk+9LIRTO8zlsdNhJrEw=', 'Content-Length': '0'} 2023-05-31T20:18:04.465 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000496b48781f19ac97-006477ab7c-1630-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:18:04 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:18:04.466 DEBUG:boto:b'pklxkd-47obj71000falseobj72023-05-31T20:17:25.833Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:18:04.467 DEBUG:boto:path=/pklxkd-47/obj7 2023-05-31T20:18:04.467 DEBUG:boto:auth_path=/pklxkd-47/obj7 2023-05-31T20:18:04.467 DEBUG:boto:Method: HEAD 2023-05-31T20:18:04.468 DEBUG:boto:Path: /pklxkd-47/obj7 2023-05-31T20:18:04.468 DEBUG:boto:Data: 2023-05-31T20:18:04.469 DEBUG:boto:Headers: {} 2023-05-31T20:18:04.469 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:18:04.469 DEBUG:boto:Port: 8001 2023-05-31T20:18:04.470 DEBUG:boto:Params: {} 2023-05-31T20:18:04.470 DEBUG:boto:Token: None 2023-05-31T20:18:04.470 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:18:04 GMT /pklxkd-47/obj7 2023-05-31T20:18:04.471 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:zDlu/z1W1kwY8oFf4vPHznwOEow= 2023-05-31T20:18:04.472 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:18:04 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:zDlu/z1W1kwY8oFf4vPHznwOEow=', 'Content-Length': '0'} 2023-05-31T20:18:04.473 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:17:25 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000a152d50784f71d55-006477ab7c-162e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:18:04 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:18:04.474 DEBUG:boto:path=/pklxkd-47/obj7 2023-05-31T20:18:04.474 DEBUG:boto:auth_path=/pklxkd-47/obj7 2023-05-31T20:18:04.475 DEBUG:boto:Method: HEAD 2023-05-31T20:18:04.475 DEBUG:boto:Path: /pklxkd-47/obj7 2023-05-31T20:18:04.475 DEBUG:boto:Data: 2023-05-31T20:18:04.476 DEBUG:boto:Headers: {} 2023-05-31T20:18:04.476 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:18:04.477 DEBUG:boto:Port: 8000 2023-05-31T20:18:04.477 DEBUG:boto:Params: {} 2023-05-31T20:18:04.477 DEBUG:boto:Token: None 2023-05-31T20:18:04.478 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:18:04 GMT /pklxkd-47/obj7 2023-05-31T20:18:04.478 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:zDlu/z1W1kwY8oFf4vPHznwOEow= 2023-05-31T20:18:04.479 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:18:04 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:zDlu/z1W1kwY8oFf4vPHznwOEow=', 'Content-Length': '0'} 2023-05-31T20:18:04.480 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:17:25 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000008e4187f627868c91-006477ab7c-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:18:04 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:18:04.481 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj7 2023-05-31T20:18:04.482 DEBUG:boto:path=/pklxkd-47/obj7 2023-05-31T20:18:04.482 DEBUG:boto:auth_path=/pklxkd-47/obj7 2023-05-31T20:18:04.483 DEBUG:boto:Method: GET 2023-05-31T20:18:04.484 DEBUG:boto:Path: /pklxkd-47/obj7 2023-05-31T20:18:04.484 DEBUG:boto:Data: 2023-05-31T20:18:04.485 DEBUG:boto:Headers: {} 2023-05-31T20:18:04.485 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:18:04.485 DEBUG:boto:Port: 8001 2023-05-31T20:18:04.486 DEBUG:boto:Params: {} 2023-05-31T20:18:04.486 DEBUG:boto:Token: None 2023-05-31T20:18:04.487 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:18:04 GMT /pklxkd-47/obj7 2023-05-31T20:18:04.487 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:sQYlOmaAeO4FyhCjYi4CrA+vNaI= 2023-05-31T20:18:04.488 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:18:04 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:sQYlOmaAeO4FyhCjYi4CrA+vNaI=', 'Content-Length': '0'} 2023-05-31T20:18:04.490 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:17:25 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000000505d4156f369584-006477ab7c-162e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:18:04 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:18:04.490 DEBUG:boto:path=/pklxkd-47/obj7 2023-05-31T20:18:04.491 DEBUG:boto:auth_path=/pklxkd-47/obj7 2023-05-31T20:18:04.491 DEBUG:boto:Method: GET 2023-05-31T20:18:04.491 DEBUG:boto:Path: /pklxkd-47/obj7 2023-05-31T20:18:04.492 DEBUG:boto:Data: 2023-05-31T20:18:04.492 DEBUG:boto:Headers: {} 2023-05-31T20:18:04.493 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:18:04.493 DEBUG:boto:Port: 8000 2023-05-31T20:18:04.493 DEBUG:boto:Params: {} 2023-05-31T20:18:04.494 DEBUG:boto:Token: None 2023-05-31T20:18:04.495 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:18:04 GMT /pklxkd-47/obj7 2023-05-31T20:18:04.495 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:sQYlOmaAeO4FyhCjYi4CrA+vNaI= 2023-05-31T20:18:04.496 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:18:04 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:sQYlOmaAeO4FyhCjYi4CrA+vNaI=', 'Content-Length': '0'} 2023-05-31T20:18:04.497 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:17:25 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000064a3b63fdd96dc53-006477ab7c-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:18:04 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:18:04.540 DEBUG:boto:path=/pklxkd-47/ 2023-05-31T20:18:04.541 DEBUG:boto:auth_path=/pklxkd-47/ 2023-05-31T20:18:04.541 DEBUG:boto:path=/pklxkd-47/?prefix=obj7 2023-05-31T20:18:04.542 DEBUG:boto:auth_path=/pklxkd-47/?prefix=obj7 2023-05-31T20:18:04.542 DEBUG:boto:Method: GET 2023-05-31T20:18:04.542 DEBUG:boto:Path: /pklxkd-47/?prefix=obj7 2023-05-31T20:18:04.543 DEBUG:boto:Data: 2023-05-31T20:18:04.543 DEBUG:boto:Headers: {} 2023-05-31T20:18:04.543 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:18:04.544 DEBUG:boto:Port: 8001 2023-05-31T20:18:04.544 DEBUG:boto:Params: {} 2023-05-31T20:18:04.545 DEBUG:boto:Token: None 2023-05-31T20:18:04.545 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:18:04 GMT /pklxkd-47/ 2023-05-31T20:18:04.546 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:piiraNTKk+9LIRTO8zlsdNhJrEw= 2023-05-31T20:18:04.547 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:18:04 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:piiraNTKk+9LIRTO8zlsdNhJrEw=', 'Content-Length': '0'} 2023-05-31T20:18:04.550 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a6660e0a52b1e4c2-006477ab7c-162e-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:18:04 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:18:04.550 DEBUG:boto:b'pklxkd-47obj71000falseobj72023-05-31T20:17:25.833Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:18:04.551 DEBUG:boto:path=/pklxkd-47/ 2023-05-31T20:18:04.551 DEBUG:boto:auth_path=/pklxkd-47/ 2023-05-31T20:18:04.552 DEBUG:boto:path=/pklxkd-47/?prefix=obj7 2023-05-31T20:18:04.552 DEBUG:boto:auth_path=/pklxkd-47/?prefix=obj7 2023-05-31T20:18:04.553 DEBUG:boto:Method: GET 2023-05-31T20:18:04.553 DEBUG:boto:Path: /pklxkd-47/?prefix=obj7 2023-05-31T20:18:04.553 DEBUG:boto:Data: 2023-05-31T20:18:04.554 DEBUG:boto:Headers: {} 2023-05-31T20:18:04.554 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:18:04.555 DEBUG:boto:Port: 8000 2023-05-31T20:18:04.555 DEBUG:boto:Params: {} 2023-05-31T20:18:04.555 DEBUG:boto:Token: None 2023-05-31T20:18:04.556 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:18:04 GMT /pklxkd-47/ 2023-05-31T20:18:04.556 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:piiraNTKk+9LIRTO8zlsdNhJrEw= 2023-05-31T20:18:04.557 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:18:04 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:piiraNTKk+9LIRTO8zlsdNhJrEw=', 'Content-Length': '0'} 2023-05-31T20:18:04.559 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000cfbb9ee28e8d3a8a-006477ab7c-1630-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:18:04 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:18:04.560 DEBUG:boto:b'pklxkd-47obj71000falseobj72023-05-31T20:17:25.833Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:18:04.561 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj8 2023-05-31T20:18:04.561 DEBUG:boto:path=/pklxkd-47/obj8 2023-05-31T20:18:04.561 DEBUG:boto:auth_path=/pklxkd-47/obj8 2023-05-31T20:18:04.562 DEBUG:boto:path=/pklxkd-47/obj8?versionId=null 2023-05-31T20:18:04.562 DEBUG:boto:auth_path=/pklxkd-47/obj8?versionId=null 2023-05-31T20:18:04.563 DEBUG:boto:Method: GET 2023-05-31T20:18:04.563 DEBUG:boto:Path: /pklxkd-47/obj8?versionId=null 2023-05-31T20:18:04.563 DEBUG:boto:Data: 2023-05-31T20:18:04.564 DEBUG:boto:Headers: {} 2023-05-31T20:18:04.564 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:18:04.564 DEBUG:boto:Port: 8001 2023-05-31T20:18:04.565 DEBUG:boto:Params: {} 2023-05-31T20:18:04.565 DEBUG:boto:Token: None 2023-05-31T20:18:04.566 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:18:04 GMT /pklxkd-47/obj8?versionId=null 2023-05-31T20:18:04.566 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:Y0JZjtnv2dSLAhYvFBWTD5eD7Dg= 2023-05-31T20:18:04.566 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:18:04 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:Y0JZjtnv2dSLAhYvFBWTD5eD7Dg=', 'Content-Length': '0'} 2023-05-31T20:18:04.568 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:17:25 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000b4cf790a6eec2330-006477ab7c-162e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:18:04 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:18:04.612 DEBUG:boto:path=/pklxkd-47/obj8 2023-05-31T20:18:04.613 DEBUG:boto:auth_path=/pklxkd-47/obj8 2023-05-31T20:18:04.613 DEBUG:boto:path=/pklxkd-47/obj8?versionId=null 2023-05-31T20:18:04.614 DEBUG:boto:auth_path=/pklxkd-47/obj8?versionId=null 2023-05-31T20:18:04.614 DEBUG:boto:Method: GET 2023-05-31T20:18:04.614 DEBUG:boto:Path: /pklxkd-47/obj8?versionId=null 2023-05-31T20:18:04.615 DEBUG:boto:Data: 2023-05-31T20:18:04.615 DEBUG:boto:Headers: {} 2023-05-31T20:18:04.615 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:18:04.616 DEBUG:boto:Port: 8000 2023-05-31T20:18:04.616 DEBUG:boto:Params: {} 2023-05-31T20:18:04.617 DEBUG:boto:Token: None 2023-05-31T20:18:04.617 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:18:04 GMT /pklxkd-47/obj8?versionId=null 2023-05-31T20:18:04.618 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:Y0JZjtnv2dSLAhYvFBWTD5eD7Dg= 2023-05-31T20:18:04.618 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:18:04 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:Y0JZjtnv2dSLAhYvFBWTD5eD7Dg=', 'Content-Length': '0'} 2023-05-31T20:18:04.620 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:17:25 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000063478d304ff00df3-006477ab7c-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:18:04 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:18:04.622 DEBUG:boto:path=/pklxkd-47/obj8 2023-05-31T20:18:04.623 DEBUG:boto:auth_path=/pklxkd-47/obj8 2023-05-31T20:18:04.623 DEBUG:boto:path=/pklxkd-47/obj8?versionId=null 2023-05-31T20:18:04.624 DEBUG:boto:auth_path=/pklxkd-47/obj8?versionId=null 2023-05-31T20:18:04.624 DEBUG:boto:Method: HEAD 2023-05-31T20:18:04.625 DEBUG:boto:Path: /pklxkd-47/obj8?versionId=null 2023-05-31T20:18:04.625 DEBUG:boto:Data: 2023-05-31T20:18:04.625 DEBUG:boto:Headers: {} 2023-05-31T20:18:04.626 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:18:04.626 DEBUG:boto:Port: 8001 2023-05-31T20:18:04.626 DEBUG:boto:Params: {} 2023-05-31T20:18:04.627 DEBUG:boto:Token: None 2023-05-31T20:18:04.627 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:18:04 GMT /pklxkd-47/obj8?versionId=null 2023-05-31T20:18:04.628 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:UrnxxLHjIPUK8c8B/4RM3iDJxnQ= 2023-05-31T20:18:04.628 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:18:04 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:UrnxxLHjIPUK8c8B/4RM3iDJxnQ=', 'Content-Length': '0'} 2023-05-31T20:18:04.630 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:17:25 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000add54d2b66285092-006477ab7c-162e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:18:04 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:18:04.631 DEBUG:boto:path=/pklxkd-47/obj8 2023-05-31T20:18:04.631 DEBUG:boto:auth_path=/pklxkd-47/obj8 2023-05-31T20:18:04.631 DEBUG:boto:path=/pklxkd-47/obj8?versionId=null 2023-05-31T20:18:04.632 DEBUG:boto:auth_path=/pklxkd-47/obj8?versionId=null 2023-05-31T20:18:04.632 DEBUG:boto:Method: HEAD 2023-05-31T20:18:04.632 DEBUG:boto:Path: /pklxkd-47/obj8?versionId=null 2023-05-31T20:18:04.633 DEBUG:boto:Data: 2023-05-31T20:18:04.633 DEBUG:boto:Headers: {} 2023-05-31T20:18:04.633 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:18:04.634 DEBUG:boto:Port: 8000 2023-05-31T20:18:04.634 DEBUG:boto:Params: {} 2023-05-31T20:18:04.634 DEBUG:boto:Token: None 2023-05-31T20:18:04.635 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:18:04 GMT /pklxkd-47/obj8?versionId=null 2023-05-31T20:18:04.635 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:UrnxxLHjIPUK8c8B/4RM3iDJxnQ= 2023-05-31T20:18:04.636 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:18:04 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:UrnxxLHjIPUK8c8B/4RM3iDJxnQ=', 'Content-Length': '0'} 2023-05-31T20:18:04.637 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:17:25 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000001a8e1cdae5d8c531-006477ab7c-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:18:04 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:18:04.638 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj8 2023-05-31T20:18:04.638 DEBUG:boto:path=/pklxkd-47/obj8 2023-05-31T20:18:04.639 DEBUG:boto:auth_path=/pklxkd-47/obj8 2023-05-31T20:18:04.639 DEBUG:boto:path=/pklxkd-47/obj8?versionId=null 2023-05-31T20:18:04.639 DEBUG:boto:auth_path=/pklxkd-47/obj8?versionId=null 2023-05-31T20:18:04.640 DEBUG:boto:Method: GET 2023-05-31T20:18:04.640 DEBUG:boto:Path: /pklxkd-47/obj8?versionId=null 2023-05-31T20:18:04.641 DEBUG:boto:Data: 2023-05-31T20:18:04.641 DEBUG:boto:Headers: {} 2023-05-31T20:18:04.641 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:18:04.642 DEBUG:boto:Port: 8001 2023-05-31T20:18:04.642 DEBUG:boto:Params: {} 2023-05-31T20:18:04.642 DEBUG:boto:Token: None 2023-05-31T20:18:04.643 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:18:04 GMT /pklxkd-47/obj8?versionId=null 2023-05-31T20:18:04.643 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:Y0JZjtnv2dSLAhYvFBWTD5eD7Dg= 2023-05-31T20:18:04.644 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:18:04 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:Y0JZjtnv2dSLAhYvFBWTD5eD7Dg=', 'Content-Length': '0'} 2023-05-31T20:18:04.645 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:17:25 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000081b5a84c73a3da8e-006477ab7c-162e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:18:04 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:18:04.646 DEBUG:boto:path=/pklxkd-47/obj8 2023-05-31T20:18:04.647 DEBUG:boto:auth_path=/pklxkd-47/obj8 2023-05-31T20:18:04.647 DEBUG:boto:path=/pklxkd-47/obj8?versionId=null 2023-05-31T20:18:04.647 DEBUG:boto:auth_path=/pklxkd-47/obj8?versionId=null 2023-05-31T20:18:04.648 DEBUG:boto:Method: GET 2023-05-31T20:18:04.648 DEBUG:boto:Path: /pklxkd-47/obj8?versionId=null 2023-05-31T20:18:04.648 DEBUG:boto:Data: 2023-05-31T20:18:04.649 DEBUG:boto:Headers: {} 2023-05-31T20:18:04.649 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:18:04.650 DEBUG:boto:Port: 8000 2023-05-31T20:18:04.650 DEBUG:boto:Params: {} 2023-05-31T20:18:04.650 DEBUG:boto:Token: None 2023-05-31T20:18:04.651 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:18:04 GMT /pklxkd-47/obj8?versionId=null 2023-05-31T20:18:04.651 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:Y0JZjtnv2dSLAhYvFBWTD5eD7Dg= 2023-05-31T20:18:04.652 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:18:04 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:Y0JZjtnv2dSLAhYvFBWTD5eD7Dg=', 'Content-Length': '0'} 2023-05-31T20:18:04.653 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:17:25 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000cdb348cf34185c51-006477ab7c-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:18:04 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:18:04.654 DEBUG:boto:path=/pklxkd-47/ 2023-05-31T20:18:04.654 DEBUG:boto:auth_path=/pklxkd-47/ 2023-05-31T20:18:04.655 DEBUG:boto:path=/pklxkd-47/?prefix=obj8 2023-05-31T20:18:04.655 DEBUG:boto:auth_path=/pklxkd-47/?prefix=obj8 2023-05-31T20:18:04.655 DEBUG:boto:Method: GET 2023-05-31T20:18:04.656 DEBUG:boto:Path: /pklxkd-47/?prefix=obj8 2023-05-31T20:18:04.656 DEBUG:boto:Data: 2023-05-31T20:18:04.657 DEBUG:boto:Headers: {} 2023-05-31T20:18:04.657 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:18:04.657 DEBUG:boto:Port: 8001 2023-05-31T20:18:04.658 DEBUG:boto:Params: {} 2023-05-31T20:18:04.658 DEBUG:boto:Token: None 2023-05-31T20:18:04.659 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:18:04 GMT /pklxkd-47/ 2023-05-31T20:18:04.659 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:piiraNTKk+9LIRTO8zlsdNhJrEw= 2023-05-31T20:18:04.659 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:18:04 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:piiraNTKk+9LIRTO8zlsdNhJrEw=', 'Content-Length': '0'} 2023-05-31T20:18:04.662 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000059863d44922925b9-006477ab7c-162e-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:18:04 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:18:04.662 DEBUG:boto:b'pklxkd-47obj81000falseobj82023-05-31T20:17:25.854Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:18:04.663 DEBUG:boto:path=/pklxkd-47/ 2023-05-31T20:18:04.663 DEBUG:boto:auth_path=/pklxkd-47/ 2023-05-31T20:18:04.664 DEBUG:boto:path=/pklxkd-47/?prefix=obj8 2023-05-31T20:18:04.664 DEBUG:boto:auth_path=/pklxkd-47/?prefix=obj8 2023-05-31T20:18:04.664 DEBUG:boto:Method: GET 2023-05-31T20:18:04.665 DEBUG:boto:Path: /pklxkd-47/?prefix=obj8 2023-05-31T20:18:04.665 DEBUG:boto:Data: 2023-05-31T20:18:04.665 DEBUG:boto:Headers: {} 2023-05-31T20:18:04.666 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:18:04.666 DEBUG:boto:Port: 8000 2023-05-31T20:18:04.667 DEBUG:boto:Params: {} 2023-05-31T20:18:04.667 DEBUG:boto:Token: None 2023-05-31T20:18:04.667 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:18:04 GMT /pklxkd-47/ 2023-05-31T20:18:04.668 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:piiraNTKk+9LIRTO8zlsdNhJrEw= 2023-05-31T20:18:04.668 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:18:04 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:piiraNTKk+9LIRTO8zlsdNhJrEw=', 'Content-Length': '0'} 2023-05-31T20:18:04.670 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000038fbe194dd3f31d1-006477ab7c-1630-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:18:04 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:18:04.671 DEBUG:boto:b'pklxkd-47obj81000falseobj82023-05-31T20:17:25.854Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:18:04.671 DEBUG:boto:path=/pklxkd-47/obj8 2023-05-31T20:18:04.672 DEBUG:boto:auth_path=/pklxkd-47/obj8 2023-05-31T20:18:04.672 DEBUG:boto:Method: HEAD 2023-05-31T20:18:04.673 DEBUG:boto:Path: /pklxkd-47/obj8 2023-05-31T20:18:04.673 DEBUG:boto:Data: 2023-05-31T20:18:04.673 DEBUG:boto:Headers: {} 2023-05-31T20:18:04.674 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:18:04.674 DEBUG:boto:Port: 8001 2023-05-31T20:18:04.675 DEBUG:boto:Params: {} 2023-05-31T20:18:04.675 DEBUG:boto:Token: None 2023-05-31T20:18:04.675 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:18:04 GMT /pklxkd-47/obj8 2023-05-31T20:18:04.676 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:GooBNryZA1AGVLZxJycPbA8O/JI= 2023-05-31T20:18:04.676 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:18:04 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:GooBNryZA1AGVLZxJycPbA8O/JI=', 'Content-Length': '0'} 2023-05-31T20:18:04.678 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:17:25 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000081746b2afe804bc2-006477ab7c-162e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:18:04 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:18:04.678 DEBUG:boto:path=/pklxkd-47/obj8 2023-05-31T20:18:04.679 DEBUG:boto:auth_path=/pklxkd-47/obj8 2023-05-31T20:18:04.679 DEBUG:boto:Method: HEAD 2023-05-31T20:18:04.679 DEBUG:boto:Path: /pklxkd-47/obj8 2023-05-31T20:18:04.680 DEBUG:boto:Data: 2023-05-31T20:18:04.680 DEBUG:boto:Headers: {} 2023-05-31T20:18:04.681 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:18:04.681 DEBUG:boto:Port: 8000 2023-05-31T20:18:04.681 DEBUG:boto:Params: {} 2023-05-31T20:18:04.682 DEBUG:boto:Token: None 2023-05-31T20:18:04.682 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:18:04 GMT /pklxkd-47/obj8 2023-05-31T20:18:04.683 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:GooBNryZA1AGVLZxJycPbA8O/JI= 2023-05-31T20:18:04.683 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:18:04 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:GooBNryZA1AGVLZxJycPbA8O/JI=', 'Content-Length': '0'} 2023-05-31T20:18:04.684 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:17:25 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000c934c8632bcff47d-006477ab7c-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:18:04 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:18:04.685 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj8 2023-05-31T20:18:04.685 DEBUG:boto:path=/pklxkd-47/obj8 2023-05-31T20:18:04.686 DEBUG:boto:auth_path=/pklxkd-47/obj8 2023-05-31T20:18:04.686 DEBUG:boto:Method: GET 2023-05-31T20:18:04.686 DEBUG:boto:Path: /pklxkd-47/obj8 2023-05-31T20:18:04.687 DEBUG:boto:Data: 2023-05-31T20:18:04.687 DEBUG:boto:Headers: {} 2023-05-31T20:18:04.687 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:18:04.688 DEBUG:boto:Port: 8001 2023-05-31T20:18:04.688 DEBUG:boto:Params: {} 2023-05-31T20:18:04.689 DEBUG:boto:Token: None 2023-05-31T20:18:04.689 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:18:04 GMT /pklxkd-47/obj8 2023-05-31T20:18:04.690 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:SMSm8ueqaZ/4Qxru/nUGpf7DZFU= 2023-05-31T20:18:04.690 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:18:04 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:SMSm8ueqaZ/4Qxru/nUGpf7DZFU=', 'Content-Length': '0'} 2023-05-31T20:18:04.691 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:17:25 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000d23bc71a5c2d854b-006477ab7c-162e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:18:04 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:18:04.692 DEBUG:boto:path=/pklxkd-47/obj8 2023-05-31T20:18:04.693 DEBUG:boto:auth_path=/pklxkd-47/obj8 2023-05-31T20:18:04.693 DEBUG:boto:Method: GET 2023-05-31T20:18:04.693 DEBUG:boto:Path: /pklxkd-47/obj8 2023-05-31T20:18:04.694 DEBUG:boto:Data: 2023-05-31T20:18:04.694 DEBUG:boto:Headers: {} 2023-05-31T20:18:04.694 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:18:04.695 DEBUG:boto:Port: 8000 2023-05-31T20:18:04.695 DEBUG:boto:Params: {} 2023-05-31T20:18:04.696 DEBUG:boto:Token: None 2023-05-31T20:18:04.696 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:18:04 GMT /pklxkd-47/obj8 2023-05-31T20:18:04.697 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:SMSm8ueqaZ/4Qxru/nUGpf7DZFU= 2023-05-31T20:18:04.697 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:18:04 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:SMSm8ueqaZ/4Qxru/nUGpf7DZFU=', 'Content-Length': '0'} 2023-05-31T20:18:04.699 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:17:25 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000009f29d941ec4801e1-006477ab7c-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:18:04 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:18:04.700 DEBUG:boto:path=/pklxkd-47/ 2023-05-31T20:18:04.700 DEBUG:boto:auth_path=/pklxkd-47/ 2023-05-31T20:18:04.700 DEBUG:boto:path=/pklxkd-47/?prefix=obj8 2023-05-31T20:18:04.701 DEBUG:boto:auth_path=/pklxkd-47/?prefix=obj8 2023-05-31T20:18:04.701 DEBUG:boto:Method: GET 2023-05-31T20:18:04.701 DEBUG:boto:Path: /pklxkd-47/?prefix=obj8 2023-05-31T20:18:04.702 DEBUG:boto:Data: 2023-05-31T20:18:04.702 DEBUG:boto:Headers: {} 2023-05-31T20:18:04.703 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:18:04.703 DEBUG:boto:Port: 8001 2023-05-31T20:18:04.703 DEBUG:boto:Params: {} 2023-05-31T20:18:04.704 DEBUG:boto:Token: None 2023-05-31T20:18:04.704 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:18:04 GMT /pklxkd-47/ 2023-05-31T20:18:04.705 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:piiraNTKk+9LIRTO8zlsdNhJrEw= 2023-05-31T20:18:04.705 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:18:04 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:piiraNTKk+9LIRTO8zlsdNhJrEw=', 'Content-Length': '0'} 2023-05-31T20:18:04.707 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000080c239689b896e8-006477ab7c-162e-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:18:04 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:18:04.708 DEBUG:boto:b'pklxkd-47obj81000falseobj82023-05-31T20:17:25.854Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:18:04.708 DEBUG:boto:path=/pklxkd-47/ 2023-05-31T20:18:04.709 DEBUG:boto:auth_path=/pklxkd-47/ 2023-05-31T20:18:04.709 DEBUG:boto:path=/pklxkd-47/?prefix=obj8 2023-05-31T20:18:04.709 DEBUG:boto:auth_path=/pklxkd-47/?prefix=obj8 2023-05-31T20:18:04.710 DEBUG:boto:Method: GET 2023-05-31T20:18:04.710 DEBUG:boto:Path: /pklxkd-47/?prefix=obj8 2023-05-31T20:18:04.711 DEBUG:boto:Data: 2023-05-31T20:18:04.711 DEBUG:boto:Headers: {} 2023-05-31T20:18:04.711 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:18:04.712 DEBUG:boto:Port: 8000 2023-05-31T20:18:04.712 DEBUG:boto:Params: {} 2023-05-31T20:18:04.712 DEBUG:boto:Token: None 2023-05-31T20:18:04.713 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:18:04 GMT /pklxkd-47/ 2023-05-31T20:18:04.713 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:piiraNTKk+9LIRTO8zlsdNhJrEw= 2023-05-31T20:18:04.714 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:18:04 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:piiraNTKk+9LIRTO8zlsdNhJrEw=', 'Content-Length': '0'} 2023-05-31T20:18:04.715 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000025d613df7c0dc736-006477ab7c-1630-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:18:04 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:18:04.716 DEBUG:boto:b'pklxkd-47obj81000falseobj82023-05-31T20:17:25.854Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:18:04.717 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=pklxkd-47 zones={a2, a1} 2023-05-31T20:18:04.717 DEBUG:rgw_multi.tests:data checkpoint: source=a1 target=a2 2023-05-31T20:18:04.718 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['datalog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-31T20:18:04.718 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'datalog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-31T20:18:04.719 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 datalog status --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-05-31T20:18:04.768 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2023-05-31T20:18:04.768 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2023-05-31T20:18:04.817 INFO:teuthology.orchestra.run.smithi099.stdout:[ 2023-05-31T20:18:04.817 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T20:18:04.818 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "00000000000000000000:00000000000000003471", 2023-05-31T20:18:04.818 INFO:teuthology.orchestra.run.smithi099.stdout: "last_update": "2023-05-31T20:17:39.809089Z" 2023-05-31T20:18:04.818 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:18:04.818 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T20:18:04.818 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "00000000000000000000:00000000000000002063", 2023-05-31T20:18:04.818 INFO:teuthology.orchestra.run.smithi099.stdout: "last_update": "2023-05-31T20:17:39.850017Z" 2023-05-31T20:18:04.819 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:18:04.819 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T20:18:04.819 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "00000000000000000000:00000000000000007131", 2023-05-31T20:18:04.819 INFO:teuthology.orchestra.run.smithi099.stdout: "last_update": "2023-05-31T20:17:39.976662Z" 2023-05-31T20:18:04.819 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:18:04.819 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T20:18:04.820 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "00000000000000000000:00000000000000005017", 2023-05-31T20:18:04.820 INFO:teuthology.orchestra.run.smithi099.stdout: "last_update": "2023-05-31T20:17:25.552099Z" 2023-05-31T20:18:04.820 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T20:18:04.820 INFO:teuthology.orchestra.run.smithi099.stdout:] 2023-05-31T20:18:04.821 DEBUG:rgw_multi.tests:data markers for zone=a1 markers={0: '00000000000000000000:00000000000000003471', 1: '00000000000000000000:00000000000000002063', 2: '00000000000000000000:00000000000000007131', 3: '00000000000000000000:00000000000000005017'} 2023-05-31T20:18:04.822 INFO:rgw_multi.tests:starting data checkpoint for target_zone=a2 source_zone=a1 2023-05-31T20:18:04.822 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-31T20:18:04.823 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-31T20:18:04.823 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-05-31T20:18:04.874 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2023-05-31T20:18:04.875 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2023-05-31T20:18:04.932 INFO:teuthology.orchestra.run.smithi099.stdout:{ 2023-05-31T20:18:04.933 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_status": { 2023-05-31T20:18:04.933 INFO:teuthology.orchestra.run.smithi099.stdout: "info": { 2023-05-31T20:18:04.933 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "sync", 2023-05-31T20:18:04.933 INFO:teuthology.orchestra.run.smithi099.stdout: "num_shards": 4, 2023-05-31T20:18:04.933 INFO:teuthology.orchestra.run.smithi099.stdout: "instance_id": 949931097149414600 2023-05-31T20:18:04.934 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:18:04.934 INFO:teuthology.orchestra.run.smithi099.stdout: "markers": [ 2023-05-31T20:18:04.934 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T20:18:04.934 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 0, 2023-05-31T20:18:04.934 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T20:18:04.934 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "incremental-sync", 2023-05-31T20:18:04.935 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "00000000000000000000:00000000000000003471", 2023-05-31T20:18:04.935 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T20:18:04.935 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2023-05-31T20:18:04.935 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T20:18:04.935 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2023-05-31T20:17:39.809089Z" 2023-05-31T20:18:04.935 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T20:18:04.935 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:18:04.936 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T20:18:04.936 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 1, 2023-05-31T20:18:04.936 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T20:18:04.936 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "incremental-sync", 2023-05-31T20:18:04.936 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "00000000000000000000:00000000000000002063", 2023-05-31T20:18:04.936 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T20:18:04.937 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2023-05-31T20:18:04.937 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T20:18:04.937 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2023-05-31T20:17:39.850017Z" 2023-05-31T20:18:04.937 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T20:18:04.937 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:18:04.937 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T20:18:04.938 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 2, 2023-05-31T20:18:04.938 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T20:18:04.938 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "incremental-sync", 2023-05-31T20:18:04.938 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "00000000000000000000:00000000000000007131", 2023-05-31T20:18:04.938 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T20:18:04.938 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2023-05-31T20:18:04.938 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T20:18:04.939 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2023-05-31T20:17:39.976662Z" 2023-05-31T20:18:04.939 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T20:18:04.939 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:18:04.940 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T20:18:04.940 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 3, 2023-05-31T20:18:04.940 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T20:18:04.940 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "incremental-sync", 2023-05-31T20:18:04.940 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "00000000000000000000:00000000000000005017", 2023-05-31T20:18:04.940 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T20:18:04.941 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2023-05-31T20:18:04.941 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T20:18:04.941 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2023-05-31T20:17:25.552099Z" 2023-05-31T20:18:04.941 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T20:18:04.941 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T20:18:04.941 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2023-05-31T20:18:04.941 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:18:04.942 INFO:teuthology.orchestra.run.smithi099.stdout: "full_sync": { 2023-05-31T20:18:04.942 INFO:teuthology.orchestra.run.smithi099.stdout: "total": 0, 2023-05-31T20:18:04.942 INFO:teuthology.orchestra.run.smithi099.stdout: "complete": 0 2023-05-31T20:18:04.942 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:18:04.942 INFO:teuthology.orchestra.run.smithi099.stdout: "current_time": "2023-05-31T20:18:04Z" 2023-05-31T20:18:04.942 INFO:teuthology.orchestra.run.smithi099.stdout:} 2023-05-31T20:18:04.943 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 949931097149414600 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000003471", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-05-31T20:17:39.809089Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000002063", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-05-31T20:17:39.850017Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000007131", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-05-31T20:17:39.976662Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000005017", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-05-31T20:17:25.552099Z" } } ] }, "full_sync": { "total": 0, "complete": 0 }, "current_time": "2023-05-31T20:18:04Z" } 2023-05-31T20:18:04.944 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000003471', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-05-31T20:17:39.809089Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000002063', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-05-31T20:17:39.850017Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000007131', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-05-31T20:17:39.976662Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000005017', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-05-31T20:17:25.552099Z'}}] 2023-05-31T20:18:04.944 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000003471', 1: '00000000000000000000:00000000000000002063', 2: '00000000000000000000:00000000000000007131', 3: '00000000000000000000:00000000000000005017'} 2023-05-31T20:18:04.945 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000003471', 1: '00000000000000000000:00000000000000002063', 2: '00000000000000000000:00000000000000007131', 3: '00000000000000000000:00000000000000005017'} 2023-05-31T20:18:04.945 INFO:rgw_multi.tests:finished data checkpoint for target_zone=a2 source_zone=a1 2023-05-31T20:18:04.946 DEBUG:rgw_multi.tests:data checkpoint: source=a2 target=a1 2023-05-31T20:18:04.946 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['datalog', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-31T20:18:04.947 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'datalog', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-31T20:18:04.947 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 datalog status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-05-31T20:18:04.995 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2023-05-31T20:18:04.995 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2023-05-31T20:18:05.048 INFO:teuthology.orchestra.run.smithi099.stdout:[ 2023-05-31T20:18:05.049 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T20:18:05.049 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "00000000000000000000:00000000000000001775", 2023-05-31T20:18:05.049 INFO:teuthology.orchestra.run.smithi099.stdout: "last_update": "2023-05-31T20:17:39.896839Z" 2023-05-31T20:18:05.049 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:18:05.050 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T20:18:05.050 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "00000000000000000000:00000000000000000935", 2023-05-31T20:18:05.050 INFO:teuthology.orchestra.run.smithi099.stdout: "last_update": "2023-05-31T20:17:39.931118Z" 2023-05-31T20:18:05.050 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:18:05.050 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T20:18:05.050 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "00000000000000000000:00000000000000005439", 2023-05-31T20:18:05.051 INFO:teuthology.orchestra.run.smithi099.stdout: "last_update": "2023-05-31T20:17:39.896577Z" 2023-05-31T20:18:05.051 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:18:05.051 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T20:18:05.051 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "00000000000000000000:00000000000000002905", 2023-05-31T20:18:05.051 INFO:teuthology.orchestra.run.smithi099.stdout: "last_update": "2023-05-31T20:07:49.121278Z" 2023-05-31T20:18:05.051 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T20:18:05.051 INFO:teuthology.orchestra.run.smithi099.stdout:] 2023-05-31T20:18:05.052 DEBUG:rgw_multi.tests:data markers for zone=a2 markers={0: '00000000000000000000:00000000000000001775', 1: '00000000000000000000:00000000000000000935', 2: '00000000000000000000:00000000000000005439', 3: '00000000000000000000:00000000000000002905'} 2023-05-31T20:18:05.053 INFO:rgw_multi.tests:starting data checkpoint for target_zone=a1 source_zone=a2 2023-05-31T20:18:05.053 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-31T20:18:05.054 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-31T20:18:05.054 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --source-zone a2 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-05-31T20:18:05.105 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2023-05-31T20:18:05.105 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2023-05-31T20:18:05.148 INFO:teuthology.orchestra.run.smithi099.stdout:{ 2023-05-31T20:18:05.149 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_status": { 2023-05-31T20:18:05.149 INFO:teuthology.orchestra.run.smithi099.stdout: "info": { 2023-05-31T20:18:05.149 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "sync", 2023-05-31T20:18:05.149 INFO:teuthology.orchestra.run.smithi099.stdout: "num_shards": 4, 2023-05-31T20:18:05.149 INFO:teuthology.orchestra.run.smithi099.stdout: "instance_id": 17941554979570305020 2023-05-31T20:18:05.149 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:18:05.150 INFO:teuthology.orchestra.run.smithi099.stdout: "markers": [ 2023-05-31T20:18:05.150 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T20:18:05.150 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 0, 2023-05-31T20:18:05.150 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T20:18:05.150 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "incremental-sync", 2023-05-31T20:18:05.150 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "00000000000000000000:00000000000000001775", 2023-05-31T20:18:05.150 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T20:18:05.151 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2023-05-31T20:18:05.151 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T20:18:05.151 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2023-05-31T20:17:39.896839Z" 2023-05-31T20:18:05.151 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T20:18:05.151 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:18:05.151 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T20:18:05.151 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 1, 2023-05-31T20:18:05.152 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T20:18:05.152 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "incremental-sync", 2023-05-31T20:18:05.152 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "00000000000000000000:00000000000000000935", 2023-05-31T20:18:05.152 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T20:18:05.152 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2023-05-31T20:18:05.152 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T20:18:05.153 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2023-05-31T20:17:39.931118Z" 2023-05-31T20:18:05.153 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T20:18:05.153 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:18:05.153 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T20:18:05.153 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 2, 2023-05-31T20:18:05.153 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T20:18:05.153 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "incremental-sync", 2023-05-31T20:18:05.154 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "00000000000000000000:00000000000000005439", 2023-05-31T20:18:05.154 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T20:18:05.154 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2023-05-31T20:18:05.154 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T20:18:05.154 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2023-05-31T20:17:39.896577Z" 2023-05-31T20:18:05.154 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T20:18:05.154 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:18:05.155 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T20:18:05.155 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 3, 2023-05-31T20:18:05.155 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T20:18:05.155 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "incremental-sync", 2023-05-31T20:18:05.155 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "00000000000000000000:00000000000000002905", 2023-05-31T20:18:05.155 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T20:18:05.155 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2023-05-31T20:18:05.156 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T20:18:05.156 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2023-05-31T20:07:49.121278Z" 2023-05-31T20:18:05.156 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T20:18:05.156 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T20:18:05.156 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2023-05-31T20:18:05.156 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:18:05.156 INFO:teuthology.orchestra.run.smithi099.stdout: "full_sync": { 2023-05-31T20:18:05.157 INFO:teuthology.orchestra.run.smithi099.stdout: "total": 0, 2023-05-31T20:18:05.157 INFO:teuthology.orchestra.run.smithi099.stdout: "complete": 0 2023-05-31T20:18:05.157 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:18:05.157 INFO:teuthology.orchestra.run.smithi099.stdout: "current_time": "2023-05-31T20:18:05Z" 2023-05-31T20:18:05.157 INFO:teuthology.orchestra.run.smithi099.stdout:} 2023-05-31T20:18:05.158 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 17941554979570305020 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000001775", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-05-31T20:17:39.896839Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000000935", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-05-31T20:17:39.931118Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000005439", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-05-31T20:17:39.896577Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000002905", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-05-31T20:07:49.121278Z" } } ] }, "full_sync": { "total": 0, "complete": 0 }, "current_time": "2023-05-31T20:18:05Z" } 2023-05-31T20:18:05.159 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000001775', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-05-31T20:17:39.896839Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000000935', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-05-31T20:17:39.931118Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000005439', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-05-31T20:17:39.896577Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000002905', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-05-31T20:07:49.121278Z'}}] 2023-05-31T20:18:05.159 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000001775', 1: '00000000000000000000:00000000000000000935', 2: '00000000000000000000:00000000000000005439', 3: '00000000000000000000:00000000000000002905'} 2023-05-31T20:18:05.159 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000001775', 1: '00000000000000000000:00000000000000000935', 2: '00000000000000000000:00000000000000005439', 3: '00000000000000000000:00000000000000002905'} 2023-05-31T20:18:05.160 INFO:rgw_multi.tests:finished data checkpoint for target_zone=a1 source_zone=a2 2023-05-31T20:18:05.160 INFO:tasks.rgw_multisite_tests:rgw_multi.tests.test_bucket_sync_enable_right_after_disable ... ok 2023-05-31T20:18:05.161 INFO:rgw_multi.tests:create bucket zone=a1 name=pklxkd-48 2023-05-31T20:18:05.162 DEBUG:boto:path=/pklxkd-48/ 2023-05-31T20:18:05.163 DEBUG:boto:auth_path=/pklxkd-48/ 2023-05-31T20:18:05.163 DEBUG:boto:Method: PUT 2023-05-31T20:18:05.163 DEBUG:boto:Path: /pklxkd-48/ 2023-05-31T20:18:05.164 DEBUG:boto:Data: 2023-05-31T20:18:05.164 DEBUG:boto:Headers: {} 2023-05-31T20:18:05.164 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:18:05.165 DEBUG:boto:Port: 8000 2023-05-31T20:18:05.165 DEBUG:boto:Params: {} 2023-05-31T20:18:05.166 DEBUG:boto:Token: None 2023-05-31T20:18:05.166 DEBUG:boto:StringToSign: PUT Wed, 31 May 2023 20:18:05 GMT /pklxkd-48/ 2023-05-31T20:18:05.166 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:Jis97HyHZqFiCuo0qkMmjyELuus= 2023-05-31T20:18:05.167 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:18:05 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:Jis97HyHZqFiCuo0qkMmjyELuus=', 'Content-Length': '0'} 2023-05-31T20:18:05.197 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000d8c9bdf2e687b019-006477ab7d-1630-a1'), ('Content-Length', '0'), ('Date', 'Wed, 31 May 2023 20:18:05 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:18:05.198 INFO:rgw_multi.tests:create bucket zone=a2 name=pklxkd-49 2023-05-31T20:18:05.198 DEBUG:boto:path=/pklxkd-49/ 2023-05-31T20:18:05.199 DEBUG:boto:auth_path=/pklxkd-49/ 2023-05-31T20:18:05.199 DEBUG:boto:Method: PUT 2023-05-31T20:18:05.200 DEBUG:boto:Path: /pklxkd-49/ 2023-05-31T20:18:05.200 DEBUG:boto:Data: 2023-05-31T20:18:05.201 DEBUG:boto:Headers: {} 2023-05-31T20:18:05.201 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:18:05.201 DEBUG:boto:Port: 8001 2023-05-31T20:18:05.202 DEBUG:boto:Params: {} 2023-05-31T20:18:05.202 DEBUG:boto:Token: None 2023-05-31T20:18:05.202 DEBUG:boto:StringToSign: PUT Wed, 31 May 2023 20:18:05 GMT /pklxkd-49/ 2023-05-31T20:18:05.203 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:gnN6D8a2hMnLv1+lJ4L2g680M9s= 2023-05-31T20:18:05.203 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:18:05 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:gnN6D8a2hMnLv1+lJ4L2g680M9s=', 'Content-Length': '0'} 2023-05-31T20:18:05.245 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx000009f72e3c561c656af-006477ab7d-162e-a2'), ('Content-Length', '0'), ('Date', 'Wed, 31 May 2023 20:18:05 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:18:05.245 DEBUG:boto:path=/pklxkd-48/ 2023-05-31T20:18:05.246 DEBUG:boto:auth_path=/pklxkd-48/ 2023-05-31T20:18:05.246 DEBUG:boto:Method: HEAD 2023-05-31T20:18:05.246 DEBUG:boto:Path: /pklxkd-48/ 2023-05-31T20:18:05.247 DEBUG:boto:Data: 2023-05-31T20:18:05.247 DEBUG:boto:Headers: {} 2023-05-31T20:18:05.247 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:18:05.248 DEBUG:boto:Port: 8000 2023-05-31T20:18:05.248 DEBUG:boto:Params: {} 2023-05-31T20:18:05.249 DEBUG:boto:Token: None 2023-05-31T20:18:05.249 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:18:05 GMT /pklxkd-48/ 2023-05-31T20:18:05.249 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:fBErtd4/wXt9cJSmgEJc9CYn/ps= 2023-05-31T20:18:05.250 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:18:05 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:fBErtd4/wXt9cJSmgEJc9CYn/ps=', 'Content-Length': '0'} 2023-05-31T20:18:05.252 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000008d748d1c07e87abb-006477ab7d-1630-a1'), ('Content-Length', '0'), ('Date', 'Wed, 31 May 2023 20:18:05 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:18:05.253 DEBUG:boto:path=/pklxkd-48/obj1 2023-05-31T20:18:05.253 DEBUG:boto:auth_path=/pklxkd-48/obj1 2023-05-31T20:18:05.254 DEBUG:boto:Method: PUT 2023-05-31T20:18:05.254 DEBUG:boto:Path: /pklxkd-48/obj1 2023-05-31T20:18:05.255 DEBUG:boto:Data: 2023-05-31T20:18:05.255 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2023-05-31T20:18:05.255 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:18:05.256 DEBUG:boto:Port: 8000 2023-05-31T20:18:05.256 DEBUG:boto:Params: {} 2023-05-31T20:18:05.256 DEBUG:boto:Token: None 2023-05-31T20:18:05.257 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Wed, 31 May 2023 20:18:05 GMT /pklxkd-48/obj1 2023-05-31T20:18:05.257 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:L3eNA6OMHFmMgHlmf0kRyAu5q9w= 2023-05-31T20:18:05.258 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Wed, 31 May 2023 20:18:05 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:L3eNA6OMHFmMgHlmf0kRyAu5q9w='} 2023-05-31T20:18:05.268 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000002d374559da77cbb1-006477ab7d-1630-a1'), ('Date', 'Wed, 31 May 2023 20:18:05 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:18:05.269 DEBUG:boto:path=/pklxkd-48/ 2023-05-31T20:18:05.269 DEBUG:boto:auth_path=/pklxkd-48/ 2023-05-31T20:18:05.270 DEBUG:boto:Method: HEAD 2023-05-31T20:18:05.270 DEBUG:boto:Path: /pklxkd-48/ 2023-05-31T20:18:05.270 DEBUG:boto:Data: 2023-05-31T20:18:05.271 DEBUG:boto:Headers: {} 2023-05-31T20:18:05.271 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:18:05.271 DEBUG:boto:Port: 8000 2023-05-31T20:18:05.272 DEBUG:boto:Params: {} 2023-05-31T20:18:05.272 DEBUG:boto:Token: None 2023-05-31T20:18:05.272 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:18:05 GMT /pklxkd-48/ 2023-05-31T20:18:05.273 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:fBErtd4/wXt9cJSmgEJc9CYn/ps= 2023-05-31T20:18:05.273 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:18:05 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:fBErtd4/wXt9cJSmgEJc9CYn/ps=', 'Content-Length': '0'} 2023-05-31T20:18:05.275 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '6'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000b6173c3f068e6f2d-006477ab7d-1630-a1'), ('Content-Length', '0'), ('Date', 'Wed, 31 May 2023 20:18:05 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:18:05.276 DEBUG:boto:path=/pklxkd-48/obj2 2023-05-31T20:18:05.276 DEBUG:boto:auth_path=/pklxkd-48/obj2 2023-05-31T20:18:05.276 DEBUG:boto:Method: PUT 2023-05-31T20:18:05.277 DEBUG:boto:Path: /pklxkd-48/obj2 2023-05-31T20:18:05.277 DEBUG:boto:Data: 2023-05-31T20:18:05.277 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2023-05-31T20:18:05.278 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:18:05.278 DEBUG:boto:Port: 8000 2023-05-31T20:18:05.278 DEBUG:boto:Params: {} 2023-05-31T20:18:05.279 DEBUG:boto:Token: None 2023-05-31T20:18:05.279 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Wed, 31 May 2023 20:18:05 GMT /pklxkd-48/obj2 2023-05-31T20:18:05.280 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:6A8faMQGVBoSpm1NSUBYS3nKhQ4= 2023-05-31T20:18:05.280 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Wed, 31 May 2023 20:18:05 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:6A8faMQGVBoSpm1NSUBYS3nKhQ4='} 2023-05-31T20:18:05.328 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000006a5a631cfeda32b7-006477ab7d-1630-a1'), ('Date', 'Wed, 31 May 2023 20:18:05 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:18:05.329 DEBUG:boto:path=/pklxkd-48/ 2023-05-31T20:18:05.329 DEBUG:boto:auth_path=/pklxkd-48/ 2023-05-31T20:18:05.330 DEBUG:boto:Method: HEAD 2023-05-31T20:18:05.330 DEBUG:boto:Path: /pklxkd-48/ 2023-05-31T20:18:05.330 DEBUG:boto:Data: 2023-05-31T20:18:05.331 DEBUG:boto:Headers: {} 2023-05-31T20:18:05.331 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:18:05.331 DEBUG:boto:Port: 8000 2023-05-31T20:18:05.332 DEBUG:boto:Params: {} 2023-05-31T20:18:05.332 DEBUG:boto:Token: None 2023-05-31T20:18:05.333 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:18:05 GMT /pklxkd-48/ 2023-05-31T20:18:05.334 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:fBErtd4/wXt9cJSmgEJc9CYn/ps= 2023-05-31T20:18:05.334 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:18:05 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:fBErtd4/wXt9cJSmgEJc9CYn/ps=', 'Content-Length': '0'} 2023-05-31T20:18:05.336 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '2'), ('X-RGW-Bytes-Used', '12'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000545541bc01499171-006477ab7d-1630-a1'), ('Content-Length', '0'), ('Date', 'Wed, 31 May 2023 20:18:05 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:18:05.337 DEBUG:boto:path=/pklxkd-48/obj3 2023-05-31T20:18:05.337 DEBUG:boto:auth_path=/pklxkd-48/obj3 2023-05-31T20:18:05.337 DEBUG:boto:Method: PUT 2023-05-31T20:18:05.338 DEBUG:boto:Path: /pklxkd-48/obj3 2023-05-31T20:18:05.338 DEBUG:boto:Data: 2023-05-31T20:18:05.338 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2023-05-31T20:18:05.339 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:18:05.339 DEBUG:boto:Port: 8000 2023-05-31T20:18:05.339 DEBUG:boto:Params: {} 2023-05-31T20:18:05.340 DEBUG:boto:Token: None 2023-05-31T20:18:05.340 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Wed, 31 May 2023 20:18:05 GMT /pklxkd-48/obj3 2023-05-31T20:18:05.341 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:I4AWeycJA6gePKKcZawG6rp5Mu0= 2023-05-31T20:18:05.341 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Wed, 31 May 2023 20:18:05 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:I4AWeycJA6gePKKcZawG6rp5Mu0='} 2023-05-31T20:18:05.346 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000e7f5c242cf75eec9-006477ab7d-1630-a1'), ('Date', 'Wed, 31 May 2023 20:18:05 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:18:05.346 DEBUG:boto:path=/pklxkd-48/ 2023-05-31T20:18:05.347 DEBUG:boto:auth_path=/pklxkd-48/ 2023-05-31T20:18:05.347 DEBUG:boto:Method: HEAD 2023-05-31T20:18:05.347 DEBUG:boto:Path: /pklxkd-48/ 2023-05-31T20:18:05.348 DEBUG:boto:Data: 2023-05-31T20:18:05.348 DEBUG:boto:Headers: {} 2023-05-31T20:18:05.348 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:18:05.349 DEBUG:boto:Port: 8000 2023-05-31T20:18:05.349 DEBUG:boto:Params: {} 2023-05-31T20:18:05.350 DEBUG:boto:Token: None 2023-05-31T20:18:05.350 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:18:05 GMT /pklxkd-48/ 2023-05-31T20:18:05.350 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:fBErtd4/wXt9cJSmgEJc9CYn/ps= 2023-05-31T20:18:05.351 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:18:05 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:fBErtd4/wXt9cJSmgEJc9CYn/ps=', 'Content-Length': '0'} 2023-05-31T20:18:05.353 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '3'), ('X-RGW-Bytes-Used', '18'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000caf78619d52f7232-006477ab7d-1630-a1'), ('Content-Length', '0'), ('Date', 'Wed, 31 May 2023 20:18:05 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:18:05.353 DEBUG:boto:path=/pklxkd-48/obj4 2023-05-31T20:18:05.354 DEBUG:boto:auth_path=/pklxkd-48/obj4 2023-05-31T20:18:05.354 DEBUG:boto:Method: PUT 2023-05-31T20:18:05.354 DEBUG:boto:Path: /pklxkd-48/obj4 2023-05-31T20:18:05.355 DEBUG:boto:Data: 2023-05-31T20:18:05.355 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2023-05-31T20:18:05.355 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:18:05.356 DEBUG:boto:Port: 8000 2023-05-31T20:18:05.356 DEBUG:boto:Params: {} 2023-05-31T20:18:05.357 DEBUG:boto:Token: None 2023-05-31T20:18:05.357 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Wed, 31 May 2023 20:18:05 GMT /pklxkd-48/obj4 2023-05-31T20:18:05.358 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:+yXBZc+n3M6BeC/jsOEwwfhrung= 2023-05-31T20:18:05.358 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Wed, 31 May 2023 20:18:05 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:+yXBZc+n3M6BeC/jsOEwwfhrung='} 2023-05-31T20:18:05.400 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000acba3731b815072b-006477ab7d-1630-a1'), ('Date', 'Wed, 31 May 2023 20:18:05 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:18:05.401 DEBUG:boto:path=/pklxkd-49/ 2023-05-31T20:18:05.401 DEBUG:boto:auth_path=/pklxkd-49/ 2023-05-31T20:18:05.402 DEBUG:boto:Method: HEAD 2023-05-31T20:18:05.402 DEBUG:boto:Path: /pklxkd-49/ 2023-05-31T20:18:05.403 DEBUG:boto:Data: 2023-05-31T20:18:05.403 DEBUG:boto:Headers: {} 2023-05-31T20:18:05.403 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:18:05.403 DEBUG:boto:Port: 8001 2023-05-31T20:18:05.404 DEBUG:boto:Params: {} 2023-05-31T20:18:05.404 DEBUG:boto:Token: None 2023-05-31T20:18:05.405 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:18:05 GMT /pklxkd-49/ 2023-05-31T20:18:05.405 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:iUPG7AiG7TIUMqGyljsqGmKwKLs= 2023-05-31T20:18:05.406 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:18:05 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:iUPG7AiG7TIUMqGyljsqGmKwKLs=', 'Content-Length': '0'} 2023-05-31T20:18:05.408 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000020366c39a91d5975-006477ab7d-162e-a2'), ('Content-Length', '0'), ('Date', 'Wed, 31 May 2023 20:18:05 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:18:05.408 DEBUG:boto:path=/pklxkd-49/obj1 2023-05-31T20:18:05.409 DEBUG:boto:auth_path=/pklxkd-49/obj1 2023-05-31T20:18:05.409 DEBUG:boto:Method: PUT 2023-05-31T20:18:05.410 DEBUG:boto:Path: /pklxkd-49/obj1 2023-05-31T20:18:05.410 DEBUG:boto:Data: 2023-05-31T20:18:05.410 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2023-05-31T20:18:05.410 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:18:05.411 DEBUG:boto:Port: 8001 2023-05-31T20:18:05.411 DEBUG:boto:Params: {} 2023-05-31T20:18:05.412 DEBUG:boto:Token: None 2023-05-31T20:18:05.412 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Wed, 31 May 2023 20:18:05 GMT /pklxkd-49/obj1 2023-05-31T20:18:05.412 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:i4Lnwk+RqEPOw94Yo/GyLcn2XBM= 2023-05-31T20:18:05.413 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Wed, 31 May 2023 20:18:05 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:i4Lnwk+RqEPOw94Yo/GyLcn2XBM='} 2023-05-31T20:18:05.420 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000e32015c1926c098a-006477ab7d-162e-a2'), ('Date', 'Wed, 31 May 2023 20:18:05 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:18:05.421 DEBUG:boto:path=/pklxkd-49/ 2023-05-31T20:18:05.421 DEBUG:boto:auth_path=/pklxkd-49/ 2023-05-31T20:18:05.421 DEBUG:boto:Method: HEAD 2023-05-31T20:18:05.422 DEBUG:boto:Path: /pklxkd-49/ 2023-05-31T20:18:05.422 DEBUG:boto:Data: 2023-05-31T20:18:05.423 DEBUG:boto:Headers: {} 2023-05-31T20:18:05.423 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:18:05.423 DEBUG:boto:Port: 8001 2023-05-31T20:18:05.424 DEBUG:boto:Params: {} 2023-05-31T20:18:05.424 DEBUG:boto:Token: None 2023-05-31T20:18:05.424 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:18:05 GMT /pklxkd-49/ 2023-05-31T20:18:05.425 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:iUPG7AiG7TIUMqGyljsqGmKwKLs= 2023-05-31T20:18:05.425 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:18:05 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:iUPG7AiG7TIUMqGyljsqGmKwKLs=', 'Content-Length': '0'} 2023-05-31T20:18:05.427 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '6'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000002e09f99c699f378-006477ab7d-162e-a2'), ('Content-Length', '0'), ('Date', 'Wed, 31 May 2023 20:18:05 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:18:05.428 DEBUG:boto:path=/pklxkd-49/obj2 2023-05-31T20:18:05.428 DEBUG:boto:auth_path=/pklxkd-49/obj2 2023-05-31T20:18:05.429 DEBUG:boto:Method: PUT 2023-05-31T20:18:05.429 DEBUG:boto:Path: /pklxkd-49/obj2 2023-05-31T20:18:05.430 DEBUG:boto:Data: 2023-05-31T20:18:05.430 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2023-05-31T20:18:05.430 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:18:05.430 DEBUG:boto:Port: 8001 2023-05-31T20:18:05.431 DEBUG:boto:Params: {} 2023-05-31T20:18:05.431 DEBUG:boto:Token: None 2023-05-31T20:18:05.432 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Wed, 31 May 2023 20:18:05 GMT /pklxkd-49/obj2 2023-05-31T20:18:05.432 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:ehQol4+zMZxeZGoq3XPYzBTMOfA= 2023-05-31T20:18:05.432 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Wed, 31 May 2023 20:18:05 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:ehQol4+zMZxeZGoq3XPYzBTMOfA='} 2023-05-31T20:18:05.476 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000b6cc498060acc956-006477ab7d-162e-a2'), ('Date', 'Wed, 31 May 2023 20:18:05 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:18:05.477 DEBUG:boto:path=/pklxkd-49/ 2023-05-31T20:18:05.477 DEBUG:boto:auth_path=/pklxkd-49/ 2023-05-31T20:18:05.478 DEBUG:boto:Method: HEAD 2023-05-31T20:18:05.478 DEBUG:boto:Path: /pklxkd-49/ 2023-05-31T20:18:05.478 DEBUG:boto:Data: 2023-05-31T20:18:05.479 DEBUG:boto:Headers: {} 2023-05-31T20:18:05.479 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:18:05.480 DEBUG:boto:Port: 8001 2023-05-31T20:18:05.480 DEBUG:boto:Params: {} 2023-05-31T20:18:05.481 DEBUG:boto:Token: None 2023-05-31T20:18:05.481 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:18:05 GMT /pklxkd-49/ 2023-05-31T20:18:05.482 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:iUPG7AiG7TIUMqGyljsqGmKwKLs= 2023-05-31T20:18:05.482 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:18:05 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:iUPG7AiG7TIUMqGyljsqGmKwKLs=', 'Content-Length': '0'} 2023-05-31T20:18:05.484 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '2'), ('X-RGW-Bytes-Used', '12'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000baa085ce96e26775-006477ab7d-162e-a2'), ('Content-Length', '0'), ('Date', 'Wed, 31 May 2023 20:18:05 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:18:05.485 DEBUG:boto:path=/pklxkd-49/obj3 2023-05-31T20:18:05.485 DEBUG:boto:auth_path=/pklxkd-49/obj3 2023-05-31T20:18:05.486 DEBUG:boto:Method: PUT 2023-05-31T20:18:05.486 DEBUG:boto:Path: /pklxkd-49/obj3 2023-05-31T20:18:05.487 DEBUG:boto:Data: 2023-05-31T20:18:05.488 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2023-05-31T20:18:05.488 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:18:05.488 DEBUG:boto:Port: 8001 2023-05-31T20:18:05.494 DEBUG:boto:Params: {} 2023-05-31T20:18:05.494 DEBUG:boto:Token: None 2023-05-31T20:18:05.495 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Wed, 31 May 2023 20:18:05 GMT /pklxkd-49/obj3 2023-05-31T20:18:05.495 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:5CbM7VTkkufYcknXSoj6IcrigH4= 2023-05-31T20:18:05.495 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Wed, 31 May 2023 20:18:05 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:5CbM7VTkkufYcknXSoj6IcrigH4='} 2023-05-31T20:18:05.500 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000a7e5de00290ee568-006477ab7d-162e-a2'), ('Date', 'Wed, 31 May 2023 20:18:05 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:18:05.501 DEBUG:boto:path=/pklxkd-49/ 2023-05-31T20:18:05.501 DEBUG:boto:auth_path=/pklxkd-49/ 2023-05-31T20:18:05.502 DEBUG:boto:Method: HEAD 2023-05-31T20:18:05.502 DEBUG:boto:Path: /pklxkd-49/ 2023-05-31T20:18:05.502 DEBUG:boto:Data: 2023-05-31T20:18:05.503 DEBUG:boto:Headers: {} 2023-05-31T20:18:05.503 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:18:05.503 DEBUG:boto:Port: 8001 2023-05-31T20:18:05.504 DEBUG:boto:Params: {} 2023-05-31T20:18:05.504 DEBUG:boto:Token: None 2023-05-31T20:18:05.504 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:18:05 GMT /pklxkd-49/ 2023-05-31T20:18:05.505 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:iUPG7AiG7TIUMqGyljsqGmKwKLs= 2023-05-31T20:18:05.505 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:18:05 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:iUPG7AiG7TIUMqGyljsqGmKwKLs=', 'Content-Length': '0'} 2023-05-31T20:18:05.507 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '3'), ('X-RGW-Bytes-Used', '18'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000dd5fd92309d520ad-006477ab7d-162e-a2'), ('Content-Length', '0'), ('Date', 'Wed, 31 May 2023 20:18:05 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:18:05.508 DEBUG:boto:path=/pklxkd-49/obj4 2023-05-31T20:18:05.508 DEBUG:boto:auth_path=/pklxkd-49/obj4 2023-05-31T20:18:05.508 DEBUG:boto:Method: PUT 2023-05-31T20:18:05.509 DEBUG:boto:Path: /pklxkd-49/obj4 2023-05-31T20:18:05.509 DEBUG:boto:Data: 2023-05-31T20:18:05.509 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2023-05-31T20:18:05.510 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:18:05.510 DEBUG:boto:Port: 8001 2023-05-31T20:18:05.510 DEBUG:boto:Params: {} 2023-05-31T20:18:05.511 DEBUG:boto:Token: None 2023-05-31T20:18:05.511 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Wed, 31 May 2023 20:18:05 GMT /pklxkd-49/obj4 2023-05-31T20:18:05.512 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:bxQQyD3/mMBNhqv9VqXttSGyK/s= 2023-05-31T20:18:05.512 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Wed, 31 May 2023 20:18:05 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:bxQQyD3/mMBNhqv9VqXttSGyK/s='} 2023-05-31T20:18:05.556 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000006be314e3e3566cd0-006477ab7d-162e-a2'), ('Date', 'Wed, 31 May 2023 20:18:05 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:18:05.557 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-31T20:18:05.558 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-31T20:18:05.558 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-05-31T20:18:05.606 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2023-05-31T20:18:05.607 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2023-05-31T20:18:05.640 INFO:teuthology.orchestra.run.smithi099.stderr:No --period given, using current period=2e6123d2-d695-4350-9a01-61a455bc782b 2023-05-31T20:18:05.655 INFO:teuthology.orchestra.run.smithi099.stdout:[ 2023-05-31T20:18:05.656 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T20:18:05.656 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1685562631.516089_59.1", 2023-05-31T20:18:05.656 INFO:teuthology.orchestra.run.smithi099.stdout: "last_update": "2023-05-31T19:50:31.516089Z" 2023-05-31T20:18:05.657 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:18:05.657 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T20:18:05.657 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "", 2023-05-31T20:18:05.657 INFO:teuthology.orchestra.run.smithi099.stdout: "last_update": "0.000000" 2023-05-31T20:18:05.657 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:18:05.657 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T20:18:05.658 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1685562970.274572_93.1", 2023-05-31T20:18:05.658 INFO:teuthology.orchestra.run.smithi099.stdout: "last_update": "2023-05-31T19:56:10.274572Z" 2023-05-31T20:18:05.658 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:18:05.658 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T20:18:05.658 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1685564285.225240_320.1", 2023-05-31T20:18:05.658 INFO:teuthology.orchestra.run.smithi099.stdout: "last_update": "2023-05-31T20:18:05.225240Z" 2023-05-31T20:18:05.658 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T20:18:05.659 INFO:teuthology.orchestra.run.smithi099.stdout:] 2023-05-31T20:18:05.660 DEBUG:rgw_multi.tests:master meta markers={0: '1_1685562631.516089_59.1', 1: '', 2: '1_1685562970.274572_93.1', 3: '1_1685564285.225240_320.1'} 2023-05-31T20:18:05.660 INFO:rgw_multi.tests:starting meta checkpoint for zone=a2 2023-05-31T20:18:05.661 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-31T20:18:05.661 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-31T20:18:05.661 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-05-31T20:18:05.711 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2023-05-31T20:18:05.712 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2023-05-31T20:18:05.753 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:18:05.748+0000 7fdf2f41d780 20 RGW-SYNC:meta: read sync status 2023-05-31T20:18:05.754 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:18:05.749+0000 7fdf2f41d780 20 RGW-SYNC:meta: read sync status 2023-05-31T20:18:05.767 INFO:teuthology.orchestra.run.smithi099.stdout:{ 2023-05-31T20:18:05.768 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_status": { 2023-05-31T20:18:05.768 INFO:teuthology.orchestra.run.smithi099.stdout: "info": { 2023-05-31T20:18:05.768 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "sync", 2023-05-31T20:18:05.768 INFO:teuthology.orchestra.run.smithi099.stdout: "num_shards": 4, 2023-05-31T20:18:05.768 INFO:teuthology.orchestra.run.smithi099.stdout: "period": "2e6123d2-d695-4350-9a01-61a455bc782b", 2023-05-31T20:18:05.768 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2023-05-31T20:18:05.769 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:18:05.769 INFO:teuthology.orchestra.run.smithi099.stdout: "markers": [ 2023-05-31T20:18:05.769 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T20:18:05.769 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 0, 2023-05-31T20:18:05.769 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T20:18:05.769 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2023-05-31T20:18:05.770 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1685562631.516089_59.1", 2023-05-31T20:18:05.770 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T20:18:05.770 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 1, 2023-05-31T20:18:05.770 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T20:18:05.770 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2023-05-31T19:50:31.516089Z", 2023-05-31T20:18:05.770 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2023-05-31T20:18:05.770 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T20:18:05.771 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:18:05.771 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T20:18:05.771 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 1, 2023-05-31T20:18:05.771 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T20:18:05.771 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2023-05-31T20:18:05.771 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "", 2023-05-31T20:18:05.771 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T20:18:05.772 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2023-05-31T20:18:05.772 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T20:18:05.772 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "0.000000", 2023-05-31T20:18:05.772 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2023-05-31T20:18:05.772 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T20:18:05.772 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:18:05.772 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T20:18:05.773 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 2, 2023-05-31T20:18:05.773 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T20:18:05.773 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2023-05-31T20:18:05.773 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1685562970.274572_93.1", 2023-05-31T20:18:05.774 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T20:18:05.774 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2023-05-31T20:18:05.774 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T20:18:05.774 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2023-05-31T19:56:10.274572Z", 2023-05-31T20:18:05.774 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2023-05-31T20:18:05.774 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T20:18:05.774 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:18:05.775 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T20:18:05.775 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 3, 2023-05-31T20:18:05.775 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T20:18:05.775 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2023-05-31T20:18:05.775 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1685564285.225240_320.1", 2023-05-31T20:18:05.775 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T20:18:05.775 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2023-05-31T20:18:05.776 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T20:18:05.776 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2023-05-31T20:18:05.225240Z", 2023-05-31T20:18:05.776 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2023-05-31T20:18:05.776 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T20:18:05.776 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T20:18:05.777 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2023-05-31T20:18:05.777 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:18:05.777 INFO:teuthology.orchestra.run.smithi099.stdout: "full_sync": { 2023-05-31T20:18:05.777 INFO:teuthology.orchestra.run.smithi099.stdout: "total": 1, 2023-05-31T20:18:05.777 INFO:teuthology.orchestra.run.smithi099.stdout: "complete": 1 2023-05-31T20:18:05.777 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:18:05.777 INFO:teuthology.orchestra.run.smithi099.stdout: "current_time": "2023-05-31T20:18:05Z" 2023-05-31T20:18:05.778 INFO:teuthology.orchestra.run.smithi099.stdout:} 2023-05-31T20:18:05.779 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "2e6123d2-d695-4350-9a01-61a455bc782b", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1685562631.516089_59.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-05-31T19:50:31.516089Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1685562970.274572_93.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-05-31T19:56:10.274572Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1685564285.225240_320.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-05-31T20:18:05.225240Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-05-31T20:18:05Z" } 2023-05-31T20:18:05.779 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1685562631.516089_59.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-05-31T19:50:31.516089Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1685562970.274572_93.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-05-31T19:56:10.274572Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1685564285.225240_320.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-05-31T20:18:05.225240Z', 'realm_epoch': 2}}] 2023-05-31T20:18:05.780 DEBUG:rgw_multi.tests:log_status={0: '1_1685562631.516089_59.1', 1: '', 2: '1_1685562970.274572_93.1', 3: '1_1685564285.225240_320.1'} 2023-05-31T20:18:05.780 DEBUG:rgw_multi.tests:sync_status={0: '1_1685562631.516089_59.1', 1: '', 2: '1_1685562970.274572_93.1', 3: '1_1685564285.225240_320.1'} 2023-05-31T20:18:05.780 INFO:rgw_multi.tests:finish meta checkpoint for zone=a2 2023-05-31T20:18:05.780 DEBUG:rgw_multi.tests:bucket checkpoint: source=a1 target=a2 bucket=pklxkd-48 2023-05-31T20:18:05.781 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'pklxkd-48', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-05-31T20:18:05.781 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'pklxkd-48', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-05-31T20:18:05.782 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket pklxkd-48 --source-zone a1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2023-05-31T20:18:05.837 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2023-05-31T20:18:05.882 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2023-05-31T20:18:05.884 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:18:05.879+0000 7f3c6d8a3780 1 waiting to reach incremental sync.. 2023-05-31T20:18:10.884 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:18:10.880+0000 7f3c6d8a3780 1 waiting to reach incremental sync.. 2023-05-31T20:18:15.885 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:18:15.880+0000 7f3c6d8a3780 1 waiting to reach incremental sync.. 2023-05-31T20:18:20.887 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:18:20.882+0000 7f3c6d8a3780 1 bucket sync caught up with source: 2023-05-31T20:18:20.887 INFO:teuthology.orchestra.run.smithi099.stderr: local status: [00000000001.16.6, , 00000000001.47.6, , 00000000001.19.6, , , 00000000001.25.6, , , ] 2023-05-31T20:18:20.887 INFO:teuthology.orchestra.run.smithi099.stderr: remote markers: [00000000001.16.6, , 00000000001.47.6, , 00000000001.19.6, , , 00000000001.25.6, , , ] 2023-05-31T20:18:20.888 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:18:20.882+0000 7f3c6d8a3780 0 bucket checkpoint complete 2023-05-31T20:18:20.910 DEBUG:rgw_multi.tests:bucket checkpoint: source=a2 target=a1 bucket=pklxkd-48 2023-05-31T20:18:20.910 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'pklxkd-48', '--source-zone', 'a2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-05-31T20:18:20.911 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'pklxkd-48', '--source-zone', 'a2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-05-31T20:18:20.911 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket pklxkd-48 --source-zone a2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2023-05-31T20:18:20.978 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2023-05-31T20:18:20.978 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2023-05-31T20:18:21.024 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:18:21.020+0000 7f37d7e6c780 1 waiting to reach incremental sync.. 2023-05-31T20:18:26.025 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:18:26.019+0000 7f37d7e6c780 1 waiting to reach incremental sync.. 2023-05-31T20:18:31.026 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:18:31.020+0000 7f37d7e6c780 1 waiting to reach incremental sync.. 2023-05-31T20:18:36.027 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:18:36.021+0000 7f37d7e6c780 1 waiting to reach incremental sync.. 2023-05-31T20:18:41.027 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:18:41.021+0000 7f37d7e6c780 1 waiting to reach incremental sync.. 2023-05-31T20:18:46.028 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:18:46.022+0000 7f37d7e6c780 1 waiting to reach incremental sync.. 2023-05-31T20:18:51.028 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:18:51.024+0000 7f37d7e6c780 1 waiting to reach incremental sync.. 2023-05-31T20:18:56.029 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:18:56.024+0000 7f37d7e6c780 1 waiting to reach incremental sync.. 2023-05-31T20:19:01.030 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:19:01.025+0000 7f37d7e6c780 1 waiting to reach incremental sync.. 2023-05-31T20:19:06.030 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:19:06.026+0000 7f37d7e6c780 1 waiting to reach incremental sync.. 2023-05-31T20:19:11.031 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:19:11.026+0000 7f37d7e6c780 1 waiting to reach incremental sync.. 2023-05-31T20:19:16.032 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:19:16.026+0000 7f37d7e6c780 1 waiting to reach incremental sync.. 2023-05-31T20:19:21.033 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:19:21.027+0000 7f37d7e6c780 1 waiting to reach incremental sync.. 2023-05-31T20:19:26.033 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:19:26.028+0000 7f37d7e6c780 1 waiting to reach incremental sync.. 2023-05-31T20:19:31.034 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:19:31.028+0000 7f37d7e6c780 1 waiting to reach incremental sync.. 2023-05-31T20:19:36.035 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:19:36.030+0000 7f37d7e6c780 1 waiting to reach incremental sync.. 2023-05-31T20:19:41.035 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:19:41.030+0000 7f37d7e6c780 1 waiting to reach incremental sync.. 2023-05-31T20:19:46.036 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:19:46.031+0000 7f37d7e6c780 1 waiting to reach incremental sync.. 2023-05-31T20:19:51.039 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:19:51.032+0000 7f37d7e6c780 1 waiting to reach incremental sync.. 2023-05-31T20:19:56.037 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:19:56.031+0000 7f37d7e6c780 1 waiting to reach incremental sync.. 2023-05-31T20:20:01.038 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:20:01.032+0000 7f37d7e6c780 1 waiting to reach incremental sync.. 2023-05-31T20:20:06.038 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:20:06.032+0000 7f37d7e6c780 1 waiting to reach incremental sync.. 2023-05-31T20:20:11.038 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:20:11.032+0000 7f37d7e6c780 1 waiting to reach incremental sync.. 2023-05-31T20:20:16.038 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:20:16.034+0000 7f37d7e6c780 1 waiting to reach incremental sync.. 2023-05-31T20:20:21.040 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:20:21.036+0000 7f37d7e6c780 1 bucket sync caught up with source: 2023-05-31T20:20:21.041 INFO:teuthology.orchestra.run.smithi099.stderr: local status: [00000000001.669.6, , 00000000001.1057.6, , 00000000001.1018.6, , , 00000000001.1320.6, , , ] 2023-05-31T20:20:21.041 INFO:teuthology.orchestra.run.smithi099.stderr: remote markers: [00000000001.669.6, , 00000000001.1057.6, , 00000000001.1018.6, , , 00000000001.1320.6, , , ] 2023-05-31T20:20:21.041 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:20:21.036+0000 7f37d7e6c780 0 bucket checkpoint complete 2023-05-31T20:20:21.058 INFO:tasks.rgw_multi.zone_rados:comparing bucket=pklxkd-48 zones={a2, a1} 2023-05-31T20:20:21.058 DEBUG:boto:path=/pklxkd-48/ 2023-05-31T20:20:21.059 DEBUG:boto:auth_path=/pklxkd-48/ 2023-05-31T20:20:21.059 DEBUG:boto:Method: HEAD 2023-05-31T20:20:21.059 DEBUG:boto:Path: /pklxkd-48/ 2023-05-31T20:20:21.060 DEBUG:boto:Data: 2023-05-31T20:20:21.060 DEBUG:boto:Headers: {} 2023-05-31T20:20:21.060 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:20:21.061 DEBUG:boto:Port: 8001 2023-05-31T20:20:21.061 DEBUG:boto:Params: {} 2023-05-31T20:20:21.062 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8001} 2023-05-31T20:20:21.062 DEBUG:boto:Token: None 2023-05-31T20:20:21.063 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:20:21 GMT /pklxkd-48/ 2023-05-31T20:20:21.063 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:+TGBGZPpKLMwPa2hDl4c4OqmQ/8= 2023-05-31T20:20:21.063 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:20:21 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:+TGBGZPpKLMwPa2hDl4c4OqmQ/8=', 'Content-Length': '0'} 2023-05-31T20:20:21.067 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '4'), ('X-RGW-Bytes-Used', '24'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000040b9a79d3671135e-006477ac05-162e-a2'), ('Content-Length', '0'), ('Date', 'Wed, 31 May 2023 20:20:21 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:20:21.067 DEBUG:boto:path=/pklxkd-48/ 2023-05-31T20:20:21.068 DEBUG:boto:auth_path=/pklxkd-48/ 2023-05-31T20:20:21.068 DEBUG:boto:Method: HEAD 2023-05-31T20:20:21.068 DEBUG:boto:Path: /pklxkd-48/ 2023-05-31T20:20:21.069 DEBUG:boto:Data: 2023-05-31T20:20:21.069 DEBUG:boto:Headers: {} 2023-05-31T20:20:21.069 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:20:21.070 DEBUG:boto:Port: 8000 2023-05-31T20:20:21.070 DEBUG:boto:Params: {} 2023-05-31T20:20:21.071 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2023-05-31T20:20:21.071 DEBUG:boto:Token: None 2023-05-31T20:20:21.071 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:20:21 GMT /pklxkd-48/ 2023-05-31T20:20:21.072 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:+TGBGZPpKLMwPa2hDl4c4OqmQ/8= 2023-05-31T20:20:21.072 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:20:21 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:+TGBGZPpKLMwPa2hDl4c4OqmQ/8=', 'Content-Length': '0'} 2023-05-31T20:20:21.075 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '4'), ('X-RGW-Bytes-Used', '24'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000a84f3f87b46d6d7f-006477ac05-1630-a1'), ('Content-Length', '0'), ('Date', 'Wed, 31 May 2023 20:20:21 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:20:21.076 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2023-05-31T20:20:21.076 DEBUG:boto:path=/pklxkd-48/ 2023-05-31T20:20:21.077 DEBUG:boto:auth_path=/pklxkd-48/ 2023-05-31T20:20:21.077 DEBUG:boto:path=/pklxkd-48/?versions&max-keys=999 2023-05-31T20:20:21.078 DEBUG:boto:auth_path=/pklxkd-48/?versions&max-keys=999 2023-05-31T20:20:21.078 DEBUG:boto:Method: GET 2023-05-31T20:20:21.078 DEBUG:boto:Path: /pklxkd-48/?versions&max-keys=999 2023-05-31T20:20:21.079 DEBUG:boto:Data: 2023-05-31T20:20:21.079 DEBUG:boto:Headers: {} 2023-05-31T20:20:21.079 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:20:21.080 DEBUG:boto:Port: 8001 2023-05-31T20:20:21.080 DEBUG:boto:Params: {} 2023-05-31T20:20:21.080 DEBUG:boto:Token: None 2023-05-31T20:20:21.081 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:20:21 GMT /pklxkd-48/?versions 2023-05-31T20:20:21.081 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:KFjnTLVtWLmHces2mi9Og5Q/hrY= 2023-05-31T20:20:21.082 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:20:21 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:KFjnTLVtWLmHces2mi9Og5Q/hrY=', 'Content-Length': '0'} 2023-05-31T20:20:21.084 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000517dcbe06db22a94-006477ac05-162e-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:20:21 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:20:21.085 DEBUG:boto:b'pklxkd-48999falseobj1nulltrue2023-05-31T20:18:05.255Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj2nulltrue2023-05-31T20:18:05.277Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj3nulltrue2023-05-31T20:18:05.338Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj4nulltrue2023-05-31T20:18:05.356Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:20:21.086 DEBUG:tasks.rgw_multi.zone_rados:o=obj1 2023-05-31T20:20:21.086 DEBUG:tasks.rgw_multi.zone_rados:o=obj2 2023-05-31T20:20:21.086 DEBUG:tasks.rgw_multi.zone_rados:o=obj3 2023-05-31T20:20:21.087 DEBUG:tasks.rgw_multi.zone_rados:o=obj4 2023-05-31T20:20:21.087 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2023-05-31T20:20:21.087 DEBUG:boto:path=/pklxkd-48/ 2023-05-31T20:20:21.088 DEBUG:boto:auth_path=/pklxkd-48/ 2023-05-31T20:20:21.088 DEBUG:boto:path=/pklxkd-48/?versions&max-keys=999 2023-05-31T20:20:21.088 DEBUG:boto:auth_path=/pklxkd-48/?versions&max-keys=999 2023-05-31T20:20:21.089 DEBUG:boto:Method: GET 2023-05-31T20:20:21.089 DEBUG:boto:Path: /pklxkd-48/?versions&max-keys=999 2023-05-31T20:20:21.089 DEBUG:boto:Data: 2023-05-31T20:20:21.090 DEBUG:boto:Headers: {} 2023-05-31T20:20:21.090 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:20:21.090 DEBUG:boto:Port: 8000 2023-05-31T20:20:21.091 DEBUG:boto:Params: {} 2023-05-31T20:20:21.091 DEBUG:boto:Token: None 2023-05-31T20:20:21.091 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:20:21 GMT /pklxkd-48/?versions 2023-05-31T20:20:21.092 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:KFjnTLVtWLmHces2mi9Og5Q/hrY= 2023-05-31T20:20:21.092 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:20:21 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:KFjnTLVtWLmHces2mi9Og5Q/hrY=', 'Content-Length': '0'} 2023-05-31T20:20:21.095 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000729acd4c57cf3b69-006477ac05-1630-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:20:21 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:20:21.095 DEBUG:boto:b'pklxkd-48999falseobj1nulltrue2023-05-31T20:18:05.255Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj2nulltrue2023-05-31T20:18:05.277Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj3nulltrue2023-05-31T20:18:05.338Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj4nulltrue2023-05-31T20:18:05.356Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:20:21.096 DEBUG:tasks.rgw_multi.zone_rados:o=obj1 2023-05-31T20:20:21.097 DEBUG:tasks.rgw_multi.zone_rados:o=obj2 2023-05-31T20:20:21.097 DEBUG:tasks.rgw_multi.zone_rados:o=obj3 2023-05-31T20:20:21.097 DEBUG:tasks.rgw_multi.zone_rados:o=obj4 2023-05-31T20:20:21.098 DEBUG:boto:path=/pklxkd-48/ 2023-05-31T20:20:21.098 DEBUG:boto:auth_path=/pklxkd-48/ 2023-05-31T20:20:21.098 DEBUG:boto:path=/pklxkd-48/?versions&max-keys=999 2023-05-31T20:20:21.099 DEBUG:boto:auth_path=/pklxkd-48/?versions&max-keys=999 2023-05-31T20:20:21.099 DEBUG:boto:Method: GET 2023-05-31T20:20:21.099 DEBUG:boto:Path: /pklxkd-48/?versions&max-keys=999 2023-05-31T20:20:21.100 DEBUG:boto:Data: 2023-05-31T20:20:21.100 DEBUG:boto:Headers: {} 2023-05-31T20:20:21.100 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:20:21.101 DEBUG:boto:Port: 8001 2023-05-31T20:20:21.101 DEBUG:boto:Params: {} 2023-05-31T20:20:21.101 DEBUG:boto:Token: None 2023-05-31T20:20:21.102 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:20:21 GMT /pklxkd-48/?versions 2023-05-31T20:20:21.102 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:KFjnTLVtWLmHces2mi9Og5Q/hrY= 2023-05-31T20:20:21.103 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:20:21 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:KFjnTLVtWLmHces2mi9Og5Q/hrY=', 'Content-Length': '0'} 2023-05-31T20:20:21.105 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000cd32458a8a215c80-006477ac05-162e-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:20:21 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:20:21.105 DEBUG:boto:b'pklxkd-48999falseobj1nulltrue2023-05-31T20:18:05.255Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj2nulltrue2023-05-31T20:18:05.277Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj3nulltrue2023-05-31T20:18:05.338Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj4nulltrue2023-05-31T20:18:05.356Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:20:21.107 DEBUG:boto:path=/pklxkd-48/ 2023-05-31T20:20:21.107 DEBUG:boto:auth_path=/pklxkd-48/ 2023-05-31T20:20:21.108 DEBUG:boto:path=/pklxkd-48/?versions&max-keys=999 2023-05-31T20:20:21.109 DEBUG:boto:auth_path=/pklxkd-48/?versions&max-keys=999 2023-05-31T20:20:21.109 DEBUG:boto:Method: GET 2023-05-31T20:20:21.109 DEBUG:boto:Path: /pklxkd-48/?versions&max-keys=999 2023-05-31T20:20:21.110 DEBUG:boto:Data: 2023-05-31T20:20:21.110 DEBUG:boto:Headers: {} 2023-05-31T20:20:21.110 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:20:21.111 DEBUG:boto:Port: 8000 2023-05-31T20:20:21.111 DEBUG:boto:Params: {} 2023-05-31T20:20:21.111 DEBUG:boto:Token: None 2023-05-31T20:20:21.112 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:20:21 GMT /pklxkd-48/?versions 2023-05-31T20:20:21.112 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:KFjnTLVtWLmHces2mi9Og5Q/hrY= 2023-05-31T20:20:21.113 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:20:21 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:KFjnTLVtWLmHces2mi9Og5Q/hrY=', 'Content-Length': '0'} 2023-05-31T20:20:21.115 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ab98e8dfd960adf4-006477ac05-1630-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:20:21 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:20:21.116 DEBUG:boto:b'pklxkd-48999falseobj1nulltrue2023-05-31T20:18:05.255Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj2nulltrue2023-05-31T20:18:05.277Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj3nulltrue2023-05-31T20:18:05.338Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj4nulltrue2023-05-31T20:18:05.356Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:20:21.116 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2023-05-31T20:20:21.117 DEBUG:boto:path=/pklxkd-48/obj1 2023-05-31T20:20:21.117 DEBUG:boto:auth_path=/pklxkd-48/obj1 2023-05-31T20:20:21.118 DEBUG:boto:path=/pklxkd-48/obj1?versionId=null 2023-05-31T20:20:21.118 DEBUG:boto:auth_path=/pklxkd-48/obj1?versionId=null 2023-05-31T20:20:21.118 DEBUG:boto:Method: GET 2023-05-31T20:20:21.119 DEBUG:boto:Path: /pklxkd-48/obj1?versionId=null 2023-05-31T20:20:21.119 DEBUG:boto:Data: 2023-05-31T20:20:21.119 DEBUG:boto:Headers: {} 2023-05-31T20:20:21.120 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:20:21.120 DEBUG:boto:Port: 8001 2023-05-31T20:20:21.121 DEBUG:boto:Params: {} 2023-05-31T20:20:21.121 DEBUG:boto:Token: None 2023-05-31T20:20:21.121 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:20:21 GMT /pklxkd-48/obj1?versionId=null 2023-05-31T20:20:21.122 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:I9m5LAuJxGmBMg3jaKv+v1rT+v8= 2023-05-31T20:20:21.122 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:20:21 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:I9m5LAuJxGmBMg3jaKv+v1rT+v8=', 'Content-Length': '0'} 2023-05-31T20:20:21.124 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:18:05 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000095d5ab0d264d7dfd-006477ac05-162e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:20:21 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:20:21.125 DEBUG:boto:path=/pklxkd-48/obj1 2023-05-31T20:20:21.125 DEBUG:boto:auth_path=/pklxkd-48/obj1 2023-05-31T20:20:21.126 DEBUG:boto:path=/pklxkd-48/obj1?versionId=null 2023-05-31T20:20:21.126 DEBUG:boto:auth_path=/pklxkd-48/obj1?versionId=null 2023-05-31T20:20:21.127 DEBUG:boto:Method: GET 2023-05-31T20:20:21.127 DEBUG:boto:Path: /pklxkd-48/obj1?versionId=null 2023-05-31T20:20:21.127 DEBUG:boto:Data: 2023-05-31T20:20:21.128 DEBUG:boto:Headers: {} 2023-05-31T20:20:21.128 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:20:21.128 DEBUG:boto:Port: 8000 2023-05-31T20:20:21.129 DEBUG:boto:Params: {} 2023-05-31T20:20:21.129 DEBUG:boto:Token: None 2023-05-31T20:20:21.130 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:20:21 GMT /pklxkd-48/obj1?versionId=null 2023-05-31T20:20:21.130 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:I9m5LAuJxGmBMg3jaKv+v1rT+v8= 2023-05-31T20:20:21.130 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:20:21 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:I9m5LAuJxGmBMg3jaKv+v1rT+v8=', 'Content-Length': '0'} 2023-05-31T20:20:21.132 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:18:05 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000062ead770c42285b9-006477ac05-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:20:21 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:20:21.133 DEBUG:boto:path=/pklxkd-48/obj1 2023-05-31T20:20:21.133 DEBUG:boto:auth_path=/pklxkd-48/obj1 2023-05-31T20:20:21.134 DEBUG:boto:path=/pklxkd-48/obj1?versionId=null 2023-05-31T20:20:21.134 DEBUG:boto:auth_path=/pklxkd-48/obj1?versionId=null 2023-05-31T20:20:21.134 DEBUG:boto:Method: HEAD 2023-05-31T20:20:21.135 DEBUG:boto:Path: /pklxkd-48/obj1?versionId=null 2023-05-31T20:20:21.135 DEBUG:boto:Data: 2023-05-31T20:20:21.135 DEBUG:boto:Headers: {} 2023-05-31T20:20:21.135 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:20:21.136 DEBUG:boto:Port: 8001 2023-05-31T20:20:21.136 DEBUG:boto:Params: {} 2023-05-31T20:20:21.137 DEBUG:boto:Token: None 2023-05-31T20:20:21.137 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:20:21 GMT /pklxkd-48/obj1?versionId=null 2023-05-31T20:20:21.137 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:v6h7e8ZWGSvZPZKIo2P/EWx+q3U= 2023-05-31T20:20:21.138 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:20:21 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:v6h7e8ZWGSvZPZKIo2P/EWx+q3U=', 'Content-Length': '0'} 2023-05-31T20:20:21.140 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:18:05 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000ffdc66fd3e6d565c-006477ac05-162e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:20:21 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:20:21.140 DEBUG:boto:path=/pklxkd-48/obj1 2023-05-31T20:20:21.141 DEBUG:boto:auth_path=/pklxkd-48/obj1 2023-05-31T20:20:21.141 DEBUG:boto:path=/pklxkd-48/obj1?versionId=null 2023-05-31T20:20:21.141 DEBUG:boto:auth_path=/pklxkd-48/obj1?versionId=null 2023-05-31T20:20:21.142 DEBUG:boto:Method: HEAD 2023-05-31T20:20:21.142 DEBUG:boto:Path: /pklxkd-48/obj1?versionId=null 2023-05-31T20:20:21.142 DEBUG:boto:Data: 2023-05-31T20:20:21.143 DEBUG:boto:Headers: {} 2023-05-31T20:20:21.143 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:20:21.143 DEBUG:boto:Port: 8000 2023-05-31T20:20:21.144 DEBUG:boto:Params: {} 2023-05-31T20:20:21.144 DEBUG:boto:Token: None 2023-05-31T20:20:21.145 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:20:21 GMT /pklxkd-48/obj1?versionId=null 2023-05-31T20:20:21.145 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:v6h7e8ZWGSvZPZKIo2P/EWx+q3U= 2023-05-31T20:20:21.145 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:20:21 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:v6h7e8ZWGSvZPZKIo2P/EWx+q3U=', 'Content-Length': '0'} 2023-05-31T20:20:21.147 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:18:05 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000080b4f6d34e9fa18d-006477ac05-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:20:21 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:20:21.148 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2023-05-31T20:20:21.148 DEBUG:boto:path=/pklxkd-48/obj1 2023-05-31T20:20:21.148 DEBUG:boto:auth_path=/pklxkd-48/obj1 2023-05-31T20:20:21.149 DEBUG:boto:path=/pklxkd-48/obj1?versionId=null 2023-05-31T20:20:21.149 DEBUG:boto:auth_path=/pklxkd-48/obj1?versionId=null 2023-05-31T20:20:21.149 DEBUG:boto:Method: GET 2023-05-31T20:20:21.150 DEBUG:boto:Path: /pklxkd-48/obj1?versionId=null 2023-05-31T20:20:21.150 DEBUG:boto:Data: 2023-05-31T20:20:21.151 DEBUG:boto:Headers: {} 2023-05-31T20:20:21.151 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:20:21.152 DEBUG:boto:Port: 8001 2023-05-31T20:20:21.152 DEBUG:boto:Params: {} 2023-05-31T20:20:21.152 DEBUG:boto:Token: None 2023-05-31T20:20:21.153 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:20:21 GMT /pklxkd-48/obj1?versionId=null 2023-05-31T20:20:21.153 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:I9m5LAuJxGmBMg3jaKv+v1rT+v8= 2023-05-31T20:20:21.154 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:20:21 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:I9m5LAuJxGmBMg3jaKv+v1rT+v8=', 'Content-Length': '0'} 2023-05-31T20:20:21.156 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:18:05 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000a2f44f9a1f5d84da-006477ac05-162e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:20:21 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:20:21.196 DEBUG:boto:path=/pklxkd-48/obj1 2023-05-31T20:20:21.197 DEBUG:boto:auth_path=/pklxkd-48/obj1 2023-05-31T20:20:21.197 DEBUG:boto:path=/pklxkd-48/obj1?versionId=null 2023-05-31T20:20:21.198 DEBUG:boto:auth_path=/pklxkd-48/obj1?versionId=null 2023-05-31T20:20:21.198 DEBUG:boto:Method: GET 2023-05-31T20:20:21.198 DEBUG:boto:Path: /pklxkd-48/obj1?versionId=null 2023-05-31T20:20:21.199 DEBUG:boto:Data: 2023-05-31T20:20:21.199 DEBUG:boto:Headers: {} 2023-05-31T20:20:21.199 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:20:21.200 DEBUG:boto:Port: 8000 2023-05-31T20:20:21.200 DEBUG:boto:Params: {} 2023-05-31T20:20:21.200 DEBUG:boto:Token: None 2023-05-31T20:20:21.201 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:20:21 GMT /pklxkd-48/obj1?versionId=null 2023-05-31T20:20:21.201 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:I9m5LAuJxGmBMg3jaKv+v1rT+v8= 2023-05-31T20:20:21.202 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:20:21 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:I9m5LAuJxGmBMg3jaKv+v1rT+v8=', 'Content-Length': '0'} 2023-05-31T20:20:21.204 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:18:05 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000328a2b76d02f2c89-006477ac05-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:20:21 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:20:21.204 DEBUG:boto:path=/pklxkd-48/ 2023-05-31T20:20:21.205 DEBUG:boto:auth_path=/pklxkd-48/ 2023-05-31T20:20:21.205 DEBUG:boto:path=/pklxkd-48/?prefix=obj1 2023-05-31T20:20:21.206 DEBUG:boto:auth_path=/pklxkd-48/?prefix=obj1 2023-05-31T20:20:21.206 DEBUG:boto:Method: GET 2023-05-31T20:20:21.206 DEBUG:boto:Path: /pklxkd-48/?prefix=obj1 2023-05-31T20:20:21.207 DEBUG:boto:Data: 2023-05-31T20:20:21.207 DEBUG:boto:Headers: {} 2023-05-31T20:20:21.207 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:20:21.208 DEBUG:boto:Port: 8001 2023-05-31T20:20:21.208 DEBUG:boto:Params: {} 2023-05-31T20:20:21.209 DEBUG:boto:Token: None 2023-05-31T20:20:21.210 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:20:21 GMT /pklxkd-48/ 2023-05-31T20:20:21.210 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:Thv65QkXrDi4etOqQSo2zqkmHRE= 2023-05-31T20:20:21.211 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:20:21 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:Thv65QkXrDi4etOqQSo2zqkmHRE=', 'Content-Length': '0'} 2023-05-31T20:20:21.213 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b6a2c5d2964aaedf-006477ac05-162e-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:20:21 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:20:21.214 DEBUG:boto:b'pklxkd-48obj11000falseobj12023-05-31T20:18:05.255Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:20:21.214 DEBUG:boto:path=/pklxkd-48/ 2023-05-31T20:20:21.215 DEBUG:boto:auth_path=/pklxkd-48/ 2023-05-31T20:20:21.215 DEBUG:boto:path=/pklxkd-48/?prefix=obj1 2023-05-31T20:20:21.215 DEBUG:boto:auth_path=/pklxkd-48/?prefix=obj1 2023-05-31T20:20:21.216 DEBUG:boto:Method: GET 2023-05-31T20:20:21.216 DEBUG:boto:Path: /pklxkd-48/?prefix=obj1 2023-05-31T20:20:21.216 DEBUG:boto:Data: 2023-05-31T20:20:21.217 DEBUG:boto:Headers: {} 2023-05-31T20:20:21.217 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:20:21.218 DEBUG:boto:Port: 8000 2023-05-31T20:20:21.218 DEBUG:boto:Params: {} 2023-05-31T20:20:21.218 DEBUG:boto:Token: None 2023-05-31T20:20:21.219 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:20:21 GMT /pklxkd-48/ 2023-05-31T20:20:21.219 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:Thv65QkXrDi4etOqQSo2zqkmHRE= 2023-05-31T20:20:21.220 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:20:21 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:Thv65QkXrDi4etOqQSo2zqkmHRE=', 'Content-Length': '0'} 2023-05-31T20:20:21.222 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a92dffb8712a0f8e-006477ac05-1630-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:20:21 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:20:21.222 DEBUG:boto:b'pklxkd-48obj11000falseobj12023-05-31T20:18:05.255Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:20:21.223 DEBUG:boto:path=/pklxkd-48/obj1 2023-05-31T20:20:21.223 DEBUG:boto:auth_path=/pklxkd-48/obj1 2023-05-31T20:20:21.224 DEBUG:boto:Method: HEAD 2023-05-31T20:20:21.224 DEBUG:boto:Path: /pklxkd-48/obj1 2023-05-31T20:20:21.224 DEBUG:boto:Data: 2023-05-31T20:20:21.225 DEBUG:boto:Headers: {} 2023-05-31T20:20:21.225 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:20:21.225 DEBUG:boto:Port: 8001 2023-05-31T20:20:21.226 DEBUG:boto:Params: {} 2023-05-31T20:20:21.226 DEBUG:boto:Token: None 2023-05-31T20:20:21.227 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:20:21 GMT /pklxkd-48/obj1 2023-05-31T20:20:21.227 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:suXskWp/ZfvIp0acb/uRv2YOijk= 2023-05-31T20:20:21.227 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:20:21 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:suXskWp/ZfvIp0acb/uRv2YOijk=', 'Content-Length': '0'} 2023-05-31T20:20:21.229 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:18:05 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000312804c53002e5a9-006477ac05-162e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:20:21 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:20:21.230 DEBUG:boto:path=/pklxkd-48/obj1 2023-05-31T20:20:21.230 DEBUG:boto:auth_path=/pklxkd-48/obj1 2023-05-31T20:20:21.230 DEBUG:boto:Method: HEAD 2023-05-31T20:20:21.231 DEBUG:boto:Path: /pklxkd-48/obj1 2023-05-31T20:20:21.231 DEBUG:boto:Data: 2023-05-31T20:20:21.231 DEBUG:boto:Headers: {} 2023-05-31T20:20:21.232 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:20:21.232 DEBUG:boto:Port: 8000 2023-05-31T20:20:21.232 DEBUG:boto:Params: {} 2023-05-31T20:20:21.236 DEBUG:boto:Token: None 2023-05-31T20:20:21.236 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:20:21 GMT /pklxkd-48/obj1 2023-05-31T20:20:21.237 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:suXskWp/ZfvIp0acb/uRv2YOijk= 2023-05-31T20:20:21.237 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:20:21 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:suXskWp/ZfvIp0acb/uRv2YOijk=', 'Content-Length': '0'} 2023-05-31T20:20:21.239 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:18:05 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000002b98fa03e22a8b1-006477ac05-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:20:21 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:20:21.239 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2023-05-31T20:20:21.240 DEBUG:boto:path=/pklxkd-48/obj1 2023-05-31T20:20:21.240 DEBUG:boto:auth_path=/pklxkd-48/obj1 2023-05-31T20:20:21.241 DEBUG:boto:Method: GET 2023-05-31T20:20:21.241 DEBUG:boto:Path: /pklxkd-48/obj1 2023-05-31T20:20:21.241 DEBUG:boto:Data: 2023-05-31T20:20:21.242 DEBUG:boto:Headers: {} 2023-05-31T20:20:21.242 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:20:21.242 DEBUG:boto:Port: 8001 2023-05-31T20:20:21.243 DEBUG:boto:Params: {} 2023-05-31T20:20:21.243 DEBUG:boto:Token: None 2023-05-31T20:20:21.244 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:20:21 GMT /pklxkd-48/obj1 2023-05-31T20:20:21.244 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:VYOiQHTk7sYgxTVfSO9ZPbGIm/Y= 2023-05-31T20:20:21.244 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:20:21 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:VYOiQHTk7sYgxTVfSO9ZPbGIm/Y=', 'Content-Length': '0'} 2023-05-31T20:20:21.246 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:18:05 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000042d783f69ad493e9-006477ac05-162e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:20:21 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:20:21.292 DEBUG:boto:path=/pklxkd-48/obj1 2023-05-31T20:20:21.293 DEBUG:boto:auth_path=/pklxkd-48/obj1 2023-05-31T20:20:21.293 DEBUG:boto:Method: GET 2023-05-31T20:20:21.294 DEBUG:boto:Path: /pklxkd-48/obj1 2023-05-31T20:20:21.294 DEBUG:boto:Data: 2023-05-31T20:20:21.294 DEBUG:boto:Headers: {} 2023-05-31T20:20:21.295 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:20:21.295 DEBUG:boto:Port: 8000 2023-05-31T20:20:21.295 DEBUG:boto:Params: {} 2023-05-31T20:20:21.296 DEBUG:boto:Token: None 2023-05-31T20:20:21.296 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:20:21 GMT /pklxkd-48/obj1 2023-05-31T20:20:21.297 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:VYOiQHTk7sYgxTVfSO9ZPbGIm/Y= 2023-05-31T20:20:21.297 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:20:21 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:VYOiQHTk7sYgxTVfSO9ZPbGIm/Y=', 'Content-Length': '0'} 2023-05-31T20:20:21.299 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:18:05 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000007086b8327734d1c3-006477ac05-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:20:21 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:20:21.300 DEBUG:boto:path=/pklxkd-48/ 2023-05-31T20:20:21.300 DEBUG:boto:auth_path=/pklxkd-48/ 2023-05-31T20:20:21.301 DEBUG:boto:path=/pklxkd-48/?prefix=obj1 2023-05-31T20:20:21.301 DEBUG:boto:auth_path=/pklxkd-48/?prefix=obj1 2023-05-31T20:20:21.301 DEBUG:boto:Method: GET 2023-05-31T20:20:21.302 DEBUG:boto:Path: /pklxkd-48/?prefix=obj1 2023-05-31T20:20:21.302 DEBUG:boto:Data: 2023-05-31T20:20:21.302 DEBUG:boto:Headers: {} 2023-05-31T20:20:21.303 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:20:21.303 DEBUG:boto:Port: 8001 2023-05-31T20:20:21.303 DEBUG:boto:Params: {} 2023-05-31T20:20:21.304 DEBUG:boto:Token: None 2023-05-31T20:20:21.304 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:20:21 GMT /pklxkd-48/ 2023-05-31T20:20:21.305 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:Thv65QkXrDi4etOqQSo2zqkmHRE= 2023-05-31T20:20:21.305 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:20:21 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:Thv65QkXrDi4etOqQSo2zqkmHRE=', 'Content-Length': '0'} 2023-05-31T20:20:21.307 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a3c0ff42986b85e1-006477ac05-162e-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:20:21 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:20:21.308 DEBUG:boto:b'pklxkd-48obj11000falseobj12023-05-31T20:18:05.255Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:20:21.309 DEBUG:boto:path=/pklxkd-48/ 2023-05-31T20:20:21.309 DEBUG:boto:auth_path=/pklxkd-48/ 2023-05-31T20:20:21.309 DEBUG:boto:path=/pklxkd-48/?prefix=obj1 2023-05-31T20:20:21.310 DEBUG:boto:auth_path=/pklxkd-48/?prefix=obj1 2023-05-31T20:20:21.310 DEBUG:boto:Method: GET 2023-05-31T20:20:21.310 DEBUG:boto:Path: /pklxkd-48/?prefix=obj1 2023-05-31T20:20:21.311 DEBUG:boto:Data: 2023-05-31T20:20:21.311 DEBUG:boto:Headers: {} 2023-05-31T20:20:21.311 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:20:21.312 DEBUG:boto:Port: 8000 2023-05-31T20:20:21.312 DEBUG:boto:Params: {} 2023-05-31T20:20:21.313 DEBUG:boto:Token: None 2023-05-31T20:20:21.313 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:20:21 GMT /pklxkd-48/ 2023-05-31T20:20:21.314 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:Thv65QkXrDi4etOqQSo2zqkmHRE= 2023-05-31T20:20:21.314 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:20:21 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:Thv65QkXrDi4etOqQSo2zqkmHRE=', 'Content-Length': '0'} 2023-05-31T20:20:21.316 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000094f270cd1dcfe694-006477ac05-1630-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:20:21 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:20:21.317 DEBUG:boto:b'pklxkd-48obj11000falseobj12023-05-31T20:18:05.255Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:20:21.318 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2023-05-31T20:20:21.318 DEBUG:boto:path=/pklxkd-48/obj2 2023-05-31T20:20:21.318 DEBUG:boto:auth_path=/pklxkd-48/obj2 2023-05-31T20:20:21.319 DEBUG:boto:path=/pklxkd-48/obj2?versionId=null 2023-05-31T20:20:21.319 DEBUG:boto:auth_path=/pklxkd-48/obj2?versionId=null 2023-05-31T20:20:21.319 DEBUG:boto:Method: GET 2023-05-31T20:20:21.320 DEBUG:boto:Path: /pklxkd-48/obj2?versionId=null 2023-05-31T20:20:21.320 DEBUG:boto:Data: 2023-05-31T20:20:21.320 DEBUG:boto:Headers: {} 2023-05-31T20:20:21.321 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:20:21.321 DEBUG:boto:Port: 8001 2023-05-31T20:20:21.321 DEBUG:boto:Params: {} 2023-05-31T20:20:21.322 DEBUG:boto:Token: None 2023-05-31T20:20:21.322 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:20:21 GMT /pklxkd-48/obj2?versionId=null 2023-05-31T20:20:21.323 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:tr0E2TjtU8UNQX1gQo6f/v5DoRo= 2023-05-31T20:20:21.323 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:20:21 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:tr0E2TjtU8UNQX1gQo6f/v5DoRo=', 'Content-Length': '0'} 2023-05-31T20:20:21.325 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:18:05 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000244f230dfdd0f496-006477ac05-162e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:20:21 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:20:21.325 DEBUG:boto:path=/pklxkd-48/obj2 2023-05-31T20:20:21.326 DEBUG:boto:auth_path=/pklxkd-48/obj2 2023-05-31T20:20:21.326 DEBUG:boto:path=/pklxkd-48/obj2?versionId=null 2023-05-31T20:20:21.327 DEBUG:boto:auth_path=/pklxkd-48/obj2?versionId=null 2023-05-31T20:20:21.327 DEBUG:boto:Method: GET 2023-05-31T20:20:21.327 DEBUG:boto:Path: /pklxkd-48/obj2?versionId=null 2023-05-31T20:20:21.328 DEBUG:boto:Data: 2023-05-31T20:20:21.328 DEBUG:boto:Headers: {} 2023-05-31T20:20:21.328 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:20:21.329 DEBUG:boto:Port: 8000 2023-05-31T20:20:21.329 DEBUG:boto:Params: {} 2023-05-31T20:20:21.330 DEBUG:boto:Token: None 2023-05-31T20:20:21.330 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:20:21 GMT /pklxkd-48/obj2?versionId=null 2023-05-31T20:20:21.330 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:tr0E2TjtU8UNQX1gQo6f/v5DoRo= 2023-05-31T20:20:21.331 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:20:21 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:tr0E2TjtU8UNQX1gQo6f/v5DoRo=', 'Content-Length': '0'} 2023-05-31T20:20:21.333 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:18:05 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000c620a335f8f72b53-006477ac05-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:20:21 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:20:21.333 DEBUG:boto:path=/pklxkd-48/obj2 2023-05-31T20:20:21.334 DEBUG:boto:auth_path=/pklxkd-48/obj2 2023-05-31T20:20:21.334 DEBUG:boto:path=/pklxkd-48/obj2?versionId=null 2023-05-31T20:20:21.334 DEBUG:boto:auth_path=/pklxkd-48/obj2?versionId=null 2023-05-31T20:20:21.335 DEBUG:boto:Method: HEAD 2023-05-31T20:20:21.335 DEBUG:boto:Path: /pklxkd-48/obj2?versionId=null 2023-05-31T20:20:21.335 DEBUG:boto:Data: 2023-05-31T20:20:21.336 DEBUG:boto:Headers: {} 2023-05-31T20:20:21.336 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:20:21.336 DEBUG:boto:Port: 8001 2023-05-31T20:20:21.337 DEBUG:boto:Params: {} 2023-05-31T20:20:21.337 DEBUG:boto:Token: None 2023-05-31T20:20:21.337 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:20:21 GMT /pklxkd-48/obj2?versionId=null 2023-05-31T20:20:21.338 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:dYB4P6dCN6L2RIxtOX2NYsEQ5M4= 2023-05-31T20:20:21.338 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:20:21 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:dYB4P6dCN6L2RIxtOX2NYsEQ5M4=', 'Content-Length': '0'} 2023-05-31T20:20:21.340 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:18:05 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000058988dbeaca317af-006477ac05-162e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:20:21 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:20:21.341 DEBUG:boto:path=/pklxkd-48/obj2 2023-05-31T20:20:21.341 DEBUG:boto:auth_path=/pklxkd-48/obj2 2023-05-31T20:20:21.341 DEBUG:boto:path=/pklxkd-48/obj2?versionId=null 2023-05-31T20:20:21.342 DEBUG:boto:auth_path=/pklxkd-48/obj2?versionId=null 2023-05-31T20:20:21.342 DEBUG:boto:Method: HEAD 2023-05-31T20:20:21.342 DEBUG:boto:Path: /pklxkd-48/obj2?versionId=null 2023-05-31T20:20:21.343 DEBUG:boto:Data: 2023-05-31T20:20:21.343 DEBUG:boto:Headers: {} 2023-05-31T20:20:21.343 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:20:21.344 DEBUG:boto:Port: 8000 2023-05-31T20:20:21.344 DEBUG:boto:Params: {} 2023-05-31T20:20:21.344 DEBUG:boto:Token: None 2023-05-31T20:20:21.345 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:20:21 GMT /pklxkd-48/obj2?versionId=null 2023-05-31T20:20:21.345 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:dYB4P6dCN6L2RIxtOX2NYsEQ5M4= 2023-05-31T20:20:21.346 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:20:21 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:dYB4P6dCN6L2RIxtOX2NYsEQ5M4=', 'Content-Length': '0'} 2023-05-31T20:20:21.347 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:18:05 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000093bd9a9a3aee233f-006477ac05-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:20:21 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:20:21.348 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2023-05-31T20:20:21.348 DEBUG:boto:path=/pklxkd-48/obj2 2023-05-31T20:20:21.349 DEBUG:boto:auth_path=/pklxkd-48/obj2 2023-05-31T20:20:21.349 DEBUG:boto:path=/pklxkd-48/obj2?versionId=null 2023-05-31T20:20:21.349 DEBUG:boto:auth_path=/pklxkd-48/obj2?versionId=null 2023-05-31T20:20:21.350 DEBUG:boto:Method: GET 2023-05-31T20:20:21.350 DEBUG:boto:Path: /pklxkd-48/obj2?versionId=null 2023-05-31T20:20:21.351 DEBUG:boto:Data: 2023-05-31T20:20:21.351 DEBUG:boto:Headers: {} 2023-05-31T20:20:21.351 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:20:21.352 DEBUG:boto:Port: 8001 2023-05-31T20:20:21.352 DEBUG:boto:Params: {} 2023-05-31T20:20:21.353 DEBUG:boto:Token: None 2023-05-31T20:20:21.353 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:20:21 GMT /pklxkd-48/obj2?versionId=null 2023-05-31T20:20:21.354 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:tr0E2TjtU8UNQX1gQo6f/v5DoRo= 2023-05-31T20:20:21.354 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:20:21 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:tr0E2TjtU8UNQX1gQo6f/v5DoRo=', 'Content-Length': '0'} 2023-05-31T20:20:21.356 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:18:05 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000002a5d7267908eb72b-006477ac05-162e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:20:21 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:20:21.356 DEBUG:boto:path=/pklxkd-48/obj2 2023-05-31T20:20:21.357 DEBUG:boto:auth_path=/pklxkd-48/obj2 2023-05-31T20:20:21.358 DEBUG:boto:path=/pklxkd-48/obj2?versionId=null 2023-05-31T20:20:21.358 DEBUG:boto:auth_path=/pklxkd-48/obj2?versionId=null 2023-05-31T20:20:21.358 DEBUG:boto:Method: GET 2023-05-31T20:20:21.359 DEBUG:boto:Path: /pklxkd-48/obj2?versionId=null 2023-05-31T20:20:21.359 DEBUG:boto:Data: 2023-05-31T20:20:21.359 DEBUG:boto:Headers: {} 2023-05-31T20:20:21.360 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:20:21.360 DEBUG:boto:Port: 8000 2023-05-31T20:20:21.360 DEBUG:boto:Params: {} 2023-05-31T20:20:21.361 DEBUG:boto:Token: None 2023-05-31T20:20:21.361 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:20:21 GMT /pklxkd-48/obj2?versionId=null 2023-05-31T20:20:21.362 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:tr0E2TjtU8UNQX1gQo6f/v5DoRo= 2023-05-31T20:20:21.362 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:20:21 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:tr0E2TjtU8UNQX1gQo6f/v5DoRo=', 'Content-Length': '0'} 2023-05-31T20:20:21.364 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:18:05 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000c623fa9d2fa8241c-006477ac05-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:20:21 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:20:21.404 DEBUG:boto:path=/pklxkd-48/ 2023-05-31T20:20:21.405 DEBUG:boto:auth_path=/pklxkd-48/ 2023-05-31T20:20:21.405 DEBUG:boto:path=/pklxkd-48/?prefix=obj2 2023-05-31T20:20:21.406 DEBUG:boto:auth_path=/pklxkd-48/?prefix=obj2 2023-05-31T20:20:21.406 DEBUG:boto:Method: GET 2023-05-31T20:20:21.406 DEBUG:boto:Path: /pklxkd-48/?prefix=obj2 2023-05-31T20:20:21.407 DEBUG:boto:Data: 2023-05-31T20:20:21.407 DEBUG:boto:Headers: {} 2023-05-31T20:20:21.407 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:20:21.408 DEBUG:boto:Port: 8001 2023-05-31T20:20:21.408 DEBUG:boto:Params: {} 2023-05-31T20:20:21.409 DEBUG:boto:Token: None 2023-05-31T20:20:21.409 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:20:21 GMT /pklxkd-48/ 2023-05-31T20:20:21.410 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:Thv65QkXrDi4etOqQSo2zqkmHRE= 2023-05-31T20:20:21.410 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:20:21 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:Thv65QkXrDi4etOqQSo2zqkmHRE=', 'Content-Length': '0'} 2023-05-31T20:20:21.413 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000004c8f1c0bdcb99498-006477ac05-162e-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:20:21 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:20:21.413 DEBUG:boto:b'pklxkd-48obj21000falseobj22023-05-31T20:18:05.277Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:20:21.414 DEBUG:boto:path=/pklxkd-48/ 2023-05-31T20:20:21.414 DEBUG:boto:auth_path=/pklxkd-48/ 2023-05-31T20:20:21.415 DEBUG:boto:path=/pklxkd-48/?prefix=obj2 2023-05-31T20:20:21.415 DEBUG:boto:auth_path=/pklxkd-48/?prefix=obj2 2023-05-31T20:20:21.415 DEBUG:boto:Method: GET 2023-05-31T20:20:21.416 DEBUG:boto:Path: /pklxkd-48/?prefix=obj2 2023-05-31T20:20:21.416 DEBUG:boto:Data: 2023-05-31T20:20:21.416 DEBUG:boto:Headers: {} 2023-05-31T20:20:21.417 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:20:21.417 DEBUG:boto:Port: 8000 2023-05-31T20:20:21.418 DEBUG:boto:Params: {} 2023-05-31T20:20:21.418 DEBUG:boto:Token: None 2023-05-31T20:20:21.418 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:20:21 GMT /pklxkd-48/ 2023-05-31T20:20:21.419 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:Thv65QkXrDi4etOqQSo2zqkmHRE= 2023-05-31T20:20:21.419 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:20:21 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:Thv65QkXrDi4etOqQSo2zqkmHRE=', 'Content-Length': '0'} 2023-05-31T20:20:21.423 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a970d9fbba2d7030-006477ac05-1630-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:20:21 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:20:21.423 DEBUG:boto:b'pklxkd-48obj21000falseobj22023-05-31T20:18:05.277Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:20:21.424 DEBUG:boto:path=/pklxkd-48/obj2 2023-05-31T20:20:21.425 DEBUG:boto:auth_path=/pklxkd-48/obj2 2023-05-31T20:20:21.425 DEBUG:boto:Method: HEAD 2023-05-31T20:20:21.425 DEBUG:boto:Path: /pklxkd-48/obj2 2023-05-31T20:20:21.426 DEBUG:boto:Data: 2023-05-31T20:20:21.426 DEBUG:boto:Headers: {} 2023-05-31T20:20:21.426 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:20:21.427 DEBUG:boto:Port: 8001 2023-05-31T20:20:21.427 DEBUG:boto:Params: {} 2023-05-31T20:20:21.427 DEBUG:boto:Token: None 2023-05-31T20:20:21.428 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:20:21 GMT /pklxkd-48/obj2 2023-05-31T20:20:21.428 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:4Iew7K1Yymliq8O4HGmb+ypPYIg= 2023-05-31T20:20:21.429 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:20:21 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:4Iew7K1Yymliq8O4HGmb+ypPYIg=', 'Content-Length': '0'} 2023-05-31T20:20:21.430 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:18:05 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000031fba07535d8f762-006477ac05-162e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:20:21 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:20:21.431 DEBUG:boto:path=/pklxkd-48/obj2 2023-05-31T20:20:21.431 DEBUG:boto:auth_path=/pklxkd-48/obj2 2023-05-31T20:20:21.432 DEBUG:boto:Method: HEAD 2023-05-31T20:20:21.432 DEBUG:boto:Path: /pklxkd-48/obj2 2023-05-31T20:20:21.433 DEBUG:boto:Data: 2023-05-31T20:20:21.433 DEBUG:boto:Headers: {} 2023-05-31T20:20:21.433 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:20:21.434 DEBUG:boto:Port: 8000 2023-05-31T20:20:21.434 DEBUG:boto:Params: {} 2023-05-31T20:20:21.434 DEBUG:boto:Token: None 2023-05-31T20:20:21.435 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:20:21 GMT /pklxkd-48/obj2 2023-05-31T20:20:21.435 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:4Iew7K1Yymliq8O4HGmb+ypPYIg= 2023-05-31T20:20:21.435 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:20:21 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:4Iew7K1Yymliq8O4HGmb+ypPYIg=', 'Content-Length': '0'} 2023-05-31T20:20:21.437 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:18:05 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000dcd55927024e2d47-006477ac05-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:20:21 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:20:21.438 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2023-05-31T20:20:21.439 DEBUG:boto:path=/pklxkd-48/obj2 2023-05-31T20:20:21.439 DEBUG:boto:auth_path=/pklxkd-48/obj2 2023-05-31T20:20:21.440 DEBUG:boto:Method: GET 2023-05-31T20:20:21.440 DEBUG:boto:Path: /pklxkd-48/obj2 2023-05-31T20:20:21.440 DEBUG:boto:Data: 2023-05-31T20:20:21.441 DEBUG:boto:Headers: {} 2023-05-31T20:20:21.441 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:20:21.441 DEBUG:boto:Port: 8001 2023-05-31T20:20:21.442 DEBUG:boto:Params: {} 2023-05-31T20:20:21.442 DEBUG:boto:Token: None 2023-05-31T20:20:21.443 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:20:21 GMT /pklxkd-48/obj2 2023-05-31T20:20:21.443 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:3MDJsk7v1B3H7qDji7XaPjCsNGk= 2023-05-31T20:20:21.443 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:20:21 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:3MDJsk7v1B3H7qDji7XaPjCsNGk=', 'Content-Length': '0'} 2023-05-31T20:20:21.445 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:18:05 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000279d339164d5087d-006477ac05-162e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:20:21 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:20:21.488 DEBUG:boto:path=/pklxkd-48/obj2 2023-05-31T20:20:21.489 DEBUG:boto:auth_path=/pklxkd-48/obj2 2023-05-31T20:20:21.489 DEBUG:boto:Method: GET 2023-05-31T20:20:21.490 DEBUG:boto:Path: /pklxkd-48/obj2 2023-05-31T20:20:21.490 DEBUG:boto:Data: 2023-05-31T20:20:21.491 DEBUG:boto:Headers: {} 2023-05-31T20:20:21.491 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:20:21.491 DEBUG:boto:Port: 8000 2023-05-31T20:20:21.492 DEBUG:boto:Params: {} 2023-05-31T20:20:21.492 DEBUG:boto:Token: None 2023-05-31T20:20:21.493 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:20:21 GMT /pklxkd-48/obj2 2023-05-31T20:20:21.493 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:3MDJsk7v1B3H7qDji7XaPjCsNGk= 2023-05-31T20:20:21.494 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:20:21 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:3MDJsk7v1B3H7qDji7XaPjCsNGk=', 'Content-Length': '0'} 2023-05-31T20:20:21.495 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:18:05 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000007df166c56f608f04-006477ac05-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:20:21 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:20:21.496 DEBUG:boto:path=/pklxkd-48/ 2023-05-31T20:20:21.497 DEBUG:boto:auth_path=/pklxkd-48/ 2023-05-31T20:20:21.497 DEBUG:boto:path=/pklxkd-48/?prefix=obj2 2023-05-31T20:20:21.497 DEBUG:boto:auth_path=/pklxkd-48/?prefix=obj2 2023-05-31T20:20:21.498 DEBUG:boto:Method: GET 2023-05-31T20:20:21.498 DEBUG:boto:Path: /pklxkd-48/?prefix=obj2 2023-05-31T20:20:21.498 DEBUG:boto:Data: 2023-05-31T20:20:21.499 DEBUG:boto:Headers: {} 2023-05-31T20:20:21.499 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:20:21.500 DEBUG:boto:Port: 8001 2023-05-31T20:20:21.500 DEBUG:boto:Params: {} 2023-05-31T20:20:21.500 DEBUG:boto:Token: None 2023-05-31T20:20:21.501 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:20:21 GMT /pklxkd-48/ 2023-05-31T20:20:21.501 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:Thv65QkXrDi4etOqQSo2zqkmHRE= 2023-05-31T20:20:21.502 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:20:21 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:Thv65QkXrDi4etOqQSo2zqkmHRE=', 'Content-Length': '0'} 2023-05-31T20:20:21.504 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f4f64386d3c1676d-006477ac05-162e-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:20:21 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:20:21.505 DEBUG:boto:b'pklxkd-48obj21000falseobj22023-05-31T20:18:05.277Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:20:21.505 DEBUG:boto:path=/pklxkd-48/ 2023-05-31T20:20:21.506 DEBUG:boto:auth_path=/pklxkd-48/ 2023-05-31T20:20:21.506 DEBUG:boto:path=/pklxkd-48/?prefix=obj2 2023-05-31T20:20:21.507 DEBUG:boto:auth_path=/pklxkd-48/?prefix=obj2 2023-05-31T20:20:21.507 DEBUG:boto:Method: GET 2023-05-31T20:20:21.507 DEBUG:boto:Path: /pklxkd-48/?prefix=obj2 2023-05-31T20:20:21.508 DEBUG:boto:Data: 2023-05-31T20:20:21.508 DEBUG:boto:Headers: {} 2023-05-31T20:20:21.509 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:20:21.509 DEBUG:boto:Port: 8000 2023-05-31T20:20:21.509 DEBUG:boto:Params: {} 2023-05-31T20:20:21.510 DEBUG:boto:Token: None 2023-05-31T20:20:21.510 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:20:21 GMT /pklxkd-48/ 2023-05-31T20:20:21.511 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:Thv65QkXrDi4etOqQSo2zqkmHRE= 2023-05-31T20:20:21.511 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:20:21 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:Thv65QkXrDi4etOqQSo2zqkmHRE=', 'Content-Length': '0'} 2023-05-31T20:20:21.514 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d277e9a9333e6e39-006477ac05-1630-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:20:21 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:20:21.514 DEBUG:boto:b'pklxkd-48obj21000falseobj22023-05-31T20:18:05.277Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:20:21.515 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2023-05-31T20:20:21.515 DEBUG:boto:path=/pklxkd-48/obj3 2023-05-31T20:20:21.516 DEBUG:boto:auth_path=/pklxkd-48/obj3 2023-05-31T20:20:21.516 DEBUG:boto:path=/pklxkd-48/obj3?versionId=null 2023-05-31T20:20:21.517 DEBUG:boto:auth_path=/pklxkd-48/obj3?versionId=null 2023-05-31T20:20:21.517 DEBUG:boto:Method: GET 2023-05-31T20:20:21.517 DEBUG:boto:Path: /pklxkd-48/obj3?versionId=null 2023-05-31T20:20:21.518 DEBUG:boto:Data: 2023-05-31T20:20:21.518 DEBUG:boto:Headers: {} 2023-05-31T20:20:21.518 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:20:21.519 DEBUG:boto:Port: 8001 2023-05-31T20:20:21.519 DEBUG:boto:Params: {} 2023-05-31T20:20:21.520 DEBUG:boto:Token: None 2023-05-31T20:20:21.520 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:20:21 GMT /pklxkd-48/obj3?versionId=null 2023-05-31T20:20:21.521 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:kirAtnEn1C+kxsbdnCb18O1HJig= 2023-05-31T20:20:21.521 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:20:21 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:kirAtnEn1C+kxsbdnCb18O1HJig=', 'Content-Length': '0'} 2023-05-31T20:20:21.523 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:18:05 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000187f937b5d592efa-006477ac05-162e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:20:21 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:20:21.524 DEBUG:boto:path=/pklxkd-48/obj3 2023-05-31T20:20:21.524 DEBUG:boto:auth_path=/pklxkd-48/obj3 2023-05-31T20:20:21.525 DEBUG:boto:path=/pklxkd-48/obj3?versionId=null 2023-05-31T20:20:21.525 DEBUG:boto:auth_path=/pklxkd-48/obj3?versionId=null 2023-05-31T20:20:21.525 DEBUG:boto:Method: GET 2023-05-31T20:20:21.526 DEBUG:boto:Path: /pklxkd-48/obj3?versionId=null 2023-05-31T20:20:21.526 DEBUG:boto:Data: 2023-05-31T20:20:21.526 DEBUG:boto:Headers: {} 2023-05-31T20:20:21.527 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:20:21.527 DEBUG:boto:Port: 8000 2023-05-31T20:20:21.527 DEBUG:boto:Params: {} 2023-05-31T20:20:21.528 DEBUG:boto:Token: None 2023-05-31T20:20:21.528 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:20:21 GMT /pklxkd-48/obj3?versionId=null 2023-05-31T20:20:21.529 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:kirAtnEn1C+kxsbdnCb18O1HJig= 2023-05-31T20:20:21.529 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:20:21 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:kirAtnEn1C+kxsbdnCb18O1HJig=', 'Content-Length': '0'} 2023-05-31T20:20:21.531 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:18:05 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000d078f8c72a4924d9-006477ac05-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:20:21 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:20:21.532 DEBUG:boto:path=/pklxkd-48/obj3 2023-05-31T20:20:21.532 DEBUG:boto:auth_path=/pklxkd-48/obj3 2023-05-31T20:20:21.533 DEBUG:boto:path=/pklxkd-48/obj3?versionId=null 2023-05-31T20:20:21.533 DEBUG:boto:auth_path=/pklxkd-48/obj3?versionId=null 2023-05-31T20:20:21.534 DEBUG:boto:Method: HEAD 2023-05-31T20:20:21.534 DEBUG:boto:Path: /pklxkd-48/obj3?versionId=null 2023-05-31T20:20:21.535 DEBUG:boto:Data: 2023-05-31T20:20:21.535 DEBUG:boto:Headers: {} 2023-05-31T20:20:21.535 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:20:21.536 DEBUG:boto:Port: 8001 2023-05-31T20:20:21.536 DEBUG:boto:Params: {} 2023-05-31T20:20:21.537 DEBUG:boto:Token: None 2023-05-31T20:20:21.537 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:20:21 GMT /pklxkd-48/obj3?versionId=null 2023-05-31T20:20:21.537 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:De1GOIqpOl80QRjpdaNelR9458o= 2023-05-31T20:20:21.538 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:20:21 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:De1GOIqpOl80QRjpdaNelR9458o=', 'Content-Length': '0'} 2023-05-31T20:20:21.540 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:18:05 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000f1b9f8ae8a06f806-006477ac05-162e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:20:21 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:20:21.540 DEBUG:boto:path=/pklxkd-48/obj3 2023-05-31T20:20:21.541 DEBUG:boto:auth_path=/pklxkd-48/obj3 2023-05-31T20:20:21.541 DEBUG:boto:path=/pklxkd-48/obj3?versionId=null 2023-05-31T20:20:21.541 DEBUG:boto:auth_path=/pklxkd-48/obj3?versionId=null 2023-05-31T20:20:21.542 DEBUG:boto:Method: HEAD 2023-05-31T20:20:21.542 DEBUG:boto:Path: /pklxkd-48/obj3?versionId=null 2023-05-31T20:20:21.542 DEBUG:boto:Data: 2023-05-31T20:20:21.543 DEBUG:boto:Headers: {} 2023-05-31T20:20:21.543 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:20:21.543 DEBUG:boto:Port: 8000 2023-05-31T20:20:21.544 DEBUG:boto:Params: {} 2023-05-31T20:20:21.544 DEBUG:boto:Token: None 2023-05-31T20:20:21.544 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:20:21 GMT /pklxkd-48/obj3?versionId=null 2023-05-31T20:20:21.545 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:De1GOIqpOl80QRjpdaNelR9458o= 2023-05-31T20:20:21.545 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:20:21 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:De1GOIqpOl80QRjpdaNelR9458o=', 'Content-Length': '0'} 2023-05-31T20:20:21.547 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:18:05 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000004985afe929aa665c-006477ac05-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:20:21 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:20:21.548 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2023-05-31T20:20:21.548 DEBUG:boto:path=/pklxkd-48/obj3 2023-05-31T20:20:21.548 DEBUG:boto:auth_path=/pklxkd-48/obj3 2023-05-31T20:20:21.549 DEBUG:boto:path=/pklxkd-48/obj3?versionId=null 2023-05-31T20:20:21.549 DEBUG:boto:auth_path=/pklxkd-48/obj3?versionId=null 2023-05-31T20:20:21.550 DEBUG:boto:Method: GET 2023-05-31T20:20:21.550 DEBUG:boto:Path: /pklxkd-48/obj3?versionId=null 2023-05-31T20:20:21.550 DEBUG:boto:Data: 2023-05-31T20:20:21.551 DEBUG:boto:Headers: {} 2023-05-31T20:20:21.551 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:20:21.551 DEBUG:boto:Port: 8001 2023-05-31T20:20:21.552 DEBUG:boto:Params: {} 2023-05-31T20:20:21.552 DEBUG:boto:Token: None 2023-05-31T20:20:21.552 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:20:21 GMT /pklxkd-48/obj3?versionId=null 2023-05-31T20:20:21.553 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:kirAtnEn1C+kxsbdnCb18O1HJig= 2023-05-31T20:20:21.553 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:20:21 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:kirAtnEn1C+kxsbdnCb18O1HJig=', 'Content-Length': '0'} 2023-05-31T20:20:21.555 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:18:05 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000abe71c597452bbe3-006477ac05-162e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:20:21 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:20:21.556 DEBUG:boto:path=/pklxkd-48/obj3 2023-05-31T20:20:21.556 DEBUG:boto:auth_path=/pklxkd-48/obj3 2023-05-31T20:20:21.556 DEBUG:boto:path=/pklxkd-48/obj3?versionId=null 2023-05-31T20:20:21.557 DEBUG:boto:auth_path=/pklxkd-48/obj3?versionId=null 2023-05-31T20:20:21.557 DEBUG:boto:Method: GET 2023-05-31T20:20:21.558 DEBUG:boto:Path: /pklxkd-48/obj3?versionId=null 2023-05-31T20:20:21.558 DEBUG:boto:Data: 2023-05-31T20:20:21.558 DEBUG:boto:Headers: {} 2023-05-31T20:20:21.559 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:20:21.559 DEBUG:boto:Port: 8000 2023-05-31T20:20:21.559 DEBUG:boto:Params: {} 2023-05-31T20:20:21.560 DEBUG:boto:Token: None 2023-05-31T20:20:21.560 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:20:21 GMT /pklxkd-48/obj3?versionId=null 2023-05-31T20:20:21.561 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:kirAtnEn1C+kxsbdnCb18O1HJig= 2023-05-31T20:20:21.561 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:20:21 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:kirAtnEn1C+kxsbdnCb18O1HJig=', 'Content-Length': '0'} 2023-05-31T20:20:21.563 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:18:05 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000008170863e7f3b19e1-006477ac05-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:20:21 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:20:21.563 DEBUG:boto:path=/pklxkd-48/ 2023-05-31T20:20:21.564 DEBUG:boto:auth_path=/pklxkd-48/ 2023-05-31T20:20:21.564 DEBUG:boto:path=/pklxkd-48/?prefix=obj3 2023-05-31T20:20:21.565 DEBUG:boto:auth_path=/pklxkd-48/?prefix=obj3 2023-05-31T20:20:21.565 DEBUG:boto:Method: GET 2023-05-31T20:20:21.565 DEBUG:boto:Path: /pklxkd-48/?prefix=obj3 2023-05-31T20:20:21.566 DEBUG:boto:Data: 2023-05-31T20:20:21.566 DEBUG:boto:Headers: {} 2023-05-31T20:20:21.566 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:20:21.567 DEBUG:boto:Port: 8001 2023-05-31T20:20:21.567 DEBUG:boto:Params: {} 2023-05-31T20:20:21.567 DEBUG:boto:Token: None 2023-05-31T20:20:21.568 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:20:21 GMT /pklxkd-48/ 2023-05-31T20:20:21.568 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:Thv65QkXrDi4etOqQSo2zqkmHRE= 2023-05-31T20:20:21.569 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:20:21 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:Thv65QkXrDi4etOqQSo2zqkmHRE=', 'Content-Length': '0'} 2023-05-31T20:20:21.571 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000371007ea9eb0050f-006477ac05-162e-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:20:21 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:20:21.572 DEBUG:boto:b'pklxkd-48obj31000falseobj32023-05-31T20:18:05.338Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:20:21.572 DEBUG:boto:path=/pklxkd-48/ 2023-05-31T20:20:21.573 DEBUG:boto:auth_path=/pklxkd-48/ 2023-05-31T20:20:21.573 DEBUG:boto:path=/pklxkd-48/?prefix=obj3 2023-05-31T20:20:21.574 DEBUG:boto:auth_path=/pklxkd-48/?prefix=obj3 2023-05-31T20:20:21.574 DEBUG:boto:Method: GET 2023-05-31T20:20:21.574 DEBUG:boto:Path: /pklxkd-48/?prefix=obj3 2023-05-31T20:20:21.575 DEBUG:boto:Data: 2023-05-31T20:20:21.575 DEBUG:boto:Headers: {} 2023-05-31T20:20:21.575 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:20:21.576 DEBUG:boto:Port: 8000 2023-05-31T20:20:21.576 DEBUG:boto:Params: {} 2023-05-31T20:20:21.576 DEBUG:boto:Token: None 2023-05-31T20:20:21.577 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:20:21 GMT /pklxkd-48/ 2023-05-31T20:20:21.577 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:Thv65QkXrDi4etOqQSo2zqkmHRE= 2023-05-31T20:20:21.578 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:20:21 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:Thv65QkXrDi4etOqQSo2zqkmHRE=', 'Content-Length': '0'} 2023-05-31T20:20:21.580 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000004e63ad1c2a66b518-006477ac05-1630-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:20:21 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:20:21.580 DEBUG:boto:b'pklxkd-48obj31000falseobj32023-05-31T20:18:05.338Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:20:21.581 DEBUG:boto:path=/pklxkd-48/obj3 2023-05-31T20:20:21.581 DEBUG:boto:auth_path=/pklxkd-48/obj3 2023-05-31T20:20:21.582 DEBUG:boto:Method: HEAD 2023-05-31T20:20:21.582 DEBUG:boto:Path: /pklxkd-48/obj3 2023-05-31T20:20:21.582 DEBUG:boto:Data: 2023-05-31T20:20:21.583 DEBUG:boto:Headers: {} 2023-05-31T20:20:21.583 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:20:21.583 DEBUG:boto:Port: 8001 2023-05-31T20:20:21.584 DEBUG:boto:Params: {} 2023-05-31T20:20:21.584 DEBUG:boto:Token: None 2023-05-31T20:20:21.585 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:20:21 GMT /pklxkd-48/obj3 2023-05-31T20:20:21.585 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:MHrSuL6gnqQtwdBAhVddW/tTcaY= 2023-05-31T20:20:21.585 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:20:21 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:MHrSuL6gnqQtwdBAhVddW/tTcaY=', 'Content-Length': '0'} 2023-05-31T20:20:21.587 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:18:05 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000b33419e57652b1d8-006477ac05-162e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:20:21 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:20:21.588 DEBUG:boto:path=/pklxkd-48/obj3 2023-05-31T20:20:21.588 DEBUG:boto:auth_path=/pklxkd-48/obj3 2023-05-31T20:20:21.588 DEBUG:boto:Method: HEAD 2023-05-31T20:20:21.589 DEBUG:boto:Path: /pklxkd-48/obj3 2023-05-31T20:20:21.589 DEBUG:boto:Data: 2023-05-31T20:20:21.589 DEBUG:boto:Headers: {} 2023-05-31T20:20:21.590 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:20:21.590 DEBUG:boto:Port: 8000 2023-05-31T20:20:21.590 DEBUG:boto:Params: {} 2023-05-31T20:20:21.591 DEBUG:boto:Token: None 2023-05-31T20:20:21.591 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:20:21 GMT /pklxkd-48/obj3 2023-05-31T20:20:21.591 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:MHrSuL6gnqQtwdBAhVddW/tTcaY= 2023-05-31T20:20:21.592 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:20:21 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:MHrSuL6gnqQtwdBAhVddW/tTcaY=', 'Content-Length': '0'} 2023-05-31T20:20:21.593 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:18:05 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000030f0bb405568d7a4-006477ac05-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:20:21 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:20:21.594 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2023-05-31T20:20:21.594 DEBUG:boto:path=/pklxkd-48/obj3 2023-05-31T20:20:21.595 DEBUG:boto:auth_path=/pklxkd-48/obj3 2023-05-31T20:20:21.595 DEBUG:boto:Method: GET 2023-05-31T20:20:21.596 DEBUG:boto:Path: /pklxkd-48/obj3 2023-05-31T20:20:21.596 DEBUG:boto:Data: 2023-05-31T20:20:21.596 DEBUG:boto:Headers: {} 2023-05-31T20:20:21.597 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:20:21.597 DEBUG:boto:Port: 8001 2023-05-31T20:20:21.597 DEBUG:boto:Params: {} 2023-05-31T20:20:21.598 DEBUG:boto:Token: None 2023-05-31T20:20:21.598 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:20:21 GMT /pklxkd-48/obj3 2023-05-31T20:20:21.599 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:Vz2GWUj/oPpWT3sa6TVnUW7tudM= 2023-05-31T20:20:21.599 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:20:21 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:Vz2GWUj/oPpWT3sa6TVnUW7tudM=', 'Content-Length': '0'} 2023-05-31T20:20:21.601 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:18:05 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000022423951ca7cbea-006477ac05-162e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:20:21 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:20:21.601 DEBUG:boto:path=/pklxkd-48/obj3 2023-05-31T20:20:21.602 DEBUG:boto:auth_path=/pklxkd-48/obj3 2023-05-31T20:20:21.602 DEBUG:boto:Method: GET 2023-05-31T20:20:21.602 DEBUG:boto:Path: /pklxkd-48/obj3 2023-05-31T20:20:21.603 DEBUG:boto:Data: 2023-05-31T20:20:21.603 DEBUG:boto:Headers: {} 2023-05-31T20:20:21.603 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:20:21.604 DEBUG:boto:Port: 8000 2023-05-31T20:20:21.604 DEBUG:boto:Params: {} 2023-05-31T20:20:21.605 DEBUG:boto:Token: None 2023-05-31T20:20:21.605 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:20:21 GMT /pklxkd-48/obj3 2023-05-31T20:20:21.605 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:Vz2GWUj/oPpWT3sa6TVnUW7tudM= 2023-05-31T20:20:21.606 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:20:21 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:Vz2GWUj/oPpWT3sa6TVnUW7tudM=', 'Content-Length': '0'} 2023-05-31T20:20:21.608 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:18:05 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000d1e18e3d96fffe0c-006477ac05-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:20:21 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:20:21.608 DEBUG:boto:path=/pklxkd-48/ 2023-05-31T20:20:21.609 DEBUG:boto:auth_path=/pklxkd-48/ 2023-05-31T20:20:21.609 DEBUG:boto:path=/pklxkd-48/?prefix=obj3 2023-05-31T20:20:21.610 DEBUG:boto:auth_path=/pklxkd-48/?prefix=obj3 2023-05-31T20:20:21.610 DEBUG:boto:Method: GET 2023-05-31T20:20:21.610 DEBUG:boto:Path: /pklxkd-48/?prefix=obj3 2023-05-31T20:20:21.611 DEBUG:boto:Data: 2023-05-31T20:20:21.611 DEBUG:boto:Headers: {} 2023-05-31T20:20:21.611 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:20:21.612 DEBUG:boto:Port: 8001 2023-05-31T20:20:21.612 DEBUG:boto:Params: {} 2023-05-31T20:20:21.612 DEBUG:boto:Token: None 2023-05-31T20:20:21.613 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:20:21 GMT /pklxkd-48/ 2023-05-31T20:20:21.613 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:Thv65QkXrDi4etOqQSo2zqkmHRE= 2023-05-31T20:20:21.614 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:20:21 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:Thv65QkXrDi4etOqQSo2zqkmHRE=', 'Content-Length': '0'} 2023-05-31T20:20:21.616 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e386fb7d57b964a6-006477ac05-162e-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:20:21 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:20:21.660 DEBUG:boto:b'pklxkd-48obj31000falseobj32023-05-31T20:18:05.338Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:20:21.661 DEBUG:boto:path=/pklxkd-48/ 2023-05-31T20:20:21.662 DEBUG:boto:auth_path=/pklxkd-48/ 2023-05-31T20:20:21.662 DEBUG:boto:path=/pklxkd-48/?prefix=obj3 2023-05-31T20:20:21.662 DEBUG:boto:auth_path=/pklxkd-48/?prefix=obj3 2023-05-31T20:20:21.663 DEBUG:boto:Method: GET 2023-05-31T20:20:21.663 DEBUG:boto:Path: /pklxkd-48/?prefix=obj3 2023-05-31T20:20:21.663 DEBUG:boto:Data: 2023-05-31T20:20:21.664 DEBUG:boto:Headers: {} 2023-05-31T20:20:21.664 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:20:21.664 DEBUG:boto:Port: 8000 2023-05-31T20:20:21.665 DEBUG:boto:Params: {} 2023-05-31T20:20:21.665 DEBUG:boto:Token: None 2023-05-31T20:20:21.666 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:20:21 GMT /pklxkd-48/ 2023-05-31T20:20:21.666 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:Thv65QkXrDi4etOqQSo2zqkmHRE= 2023-05-31T20:20:21.666 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:20:21 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:Thv65QkXrDi4etOqQSo2zqkmHRE=', 'Content-Length': '0'} 2023-05-31T20:20:21.669 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000081d667063eb42a74-006477ac05-1630-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:20:21 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:20:21.669 DEBUG:boto:b'pklxkd-48obj31000falseobj32023-05-31T20:18:05.338Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:20:21.670 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2023-05-31T20:20:21.670 DEBUG:boto:path=/pklxkd-48/obj4 2023-05-31T20:20:21.671 DEBUG:boto:auth_path=/pklxkd-48/obj4 2023-05-31T20:20:21.671 DEBUG:boto:path=/pklxkd-48/obj4?versionId=null 2023-05-31T20:20:21.672 DEBUG:boto:auth_path=/pklxkd-48/obj4?versionId=null 2023-05-31T20:20:21.672 DEBUG:boto:Method: GET 2023-05-31T20:20:21.672 DEBUG:boto:Path: /pklxkd-48/obj4?versionId=null 2023-05-31T20:20:21.673 DEBUG:boto:Data: 2023-05-31T20:20:21.673 DEBUG:boto:Headers: {} 2023-05-31T20:20:21.673 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:20:21.674 DEBUG:boto:Port: 8001 2023-05-31T20:20:21.674 DEBUG:boto:Params: {} 2023-05-31T20:20:21.674 DEBUG:boto:Token: None 2023-05-31T20:20:21.675 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:20:21 GMT /pklxkd-48/obj4?versionId=null 2023-05-31T20:20:21.675 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:WAKirCNDq2ff7QQOa7SldjYa+x0= 2023-05-31T20:20:21.676 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:20:21 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:WAKirCNDq2ff7QQOa7SldjYa+x0=', 'Content-Length': '0'} 2023-05-31T20:20:21.678 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:18:05 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000016b3fb322991cd09-006477ac05-162e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:20:21 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:20:21.678 DEBUG:boto:path=/pklxkd-48/obj4 2023-05-31T20:20:21.679 DEBUG:boto:auth_path=/pklxkd-48/obj4 2023-05-31T20:20:21.679 DEBUG:boto:path=/pklxkd-48/obj4?versionId=null 2023-05-31T20:20:21.680 DEBUG:boto:auth_path=/pklxkd-48/obj4?versionId=null 2023-05-31T20:20:21.680 DEBUG:boto:Method: GET 2023-05-31T20:20:21.680 DEBUG:boto:Path: /pklxkd-48/obj4?versionId=null 2023-05-31T20:20:21.681 DEBUG:boto:Data: 2023-05-31T20:20:21.681 DEBUG:boto:Headers: {} 2023-05-31T20:20:21.681 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:20:21.682 DEBUG:boto:Port: 8000 2023-05-31T20:20:21.682 DEBUG:boto:Params: {} 2023-05-31T20:20:21.682 DEBUG:boto:Token: None 2023-05-31T20:20:21.683 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:20:21 GMT /pklxkd-48/obj4?versionId=null 2023-05-31T20:20:21.683 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:WAKirCNDq2ff7QQOa7SldjYa+x0= 2023-05-31T20:20:21.684 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:20:21 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:WAKirCNDq2ff7QQOa7SldjYa+x0=', 'Content-Length': '0'} 2023-05-31T20:20:21.686 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:18:05 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000006ac1f86f9e34947a-006477ac05-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:20:21 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:20:21.728 DEBUG:boto:path=/pklxkd-48/obj4 2023-05-31T20:20:21.729 DEBUG:boto:auth_path=/pklxkd-48/obj4 2023-05-31T20:20:21.731 DEBUG:boto:path=/pklxkd-48/obj4?versionId=null 2023-05-31T20:20:21.731 DEBUG:boto:auth_path=/pklxkd-48/obj4?versionId=null 2023-05-31T20:20:21.731 DEBUG:boto:Method: HEAD 2023-05-31T20:20:21.732 DEBUG:boto:Path: /pklxkd-48/obj4?versionId=null 2023-05-31T20:20:21.732 DEBUG:boto:Data: 2023-05-31T20:20:21.732 DEBUG:boto:Headers: {} 2023-05-31T20:20:21.733 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:20:21.733 DEBUG:boto:Port: 8001 2023-05-31T20:20:21.733 DEBUG:boto:Params: {} 2023-05-31T20:20:21.734 DEBUG:boto:Token: None 2023-05-31T20:20:21.734 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:20:21 GMT /pklxkd-48/obj4?versionId=null 2023-05-31T20:20:21.735 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:roSlrNozkvK7YkiLhKBdyvFpRS0= 2023-05-31T20:20:21.735 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:20:21 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:roSlrNozkvK7YkiLhKBdyvFpRS0=', 'Content-Length': '0'} 2023-05-31T20:20:21.737 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:18:05 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000080cf4259cadf4a09-006477ac05-162e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:20:21 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:20:21.737 DEBUG:boto:path=/pklxkd-48/obj4 2023-05-31T20:20:21.738 DEBUG:boto:auth_path=/pklxkd-48/obj4 2023-05-31T20:20:21.738 DEBUG:boto:path=/pklxkd-48/obj4?versionId=null 2023-05-31T20:20:21.738 DEBUG:boto:auth_path=/pklxkd-48/obj4?versionId=null 2023-05-31T20:20:21.739 DEBUG:boto:Method: HEAD 2023-05-31T20:20:21.739 DEBUG:boto:Path: /pklxkd-48/obj4?versionId=null 2023-05-31T20:20:21.740 DEBUG:boto:Data: 2023-05-31T20:20:21.740 DEBUG:boto:Headers: {} 2023-05-31T20:20:21.740 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:20:21.741 DEBUG:boto:Port: 8000 2023-05-31T20:20:21.741 DEBUG:boto:Params: {} 2023-05-31T20:20:21.742 DEBUG:boto:Token: None 2023-05-31T20:20:21.742 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:20:21 GMT /pklxkd-48/obj4?versionId=null 2023-05-31T20:20:21.743 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:roSlrNozkvK7YkiLhKBdyvFpRS0= 2023-05-31T20:20:21.743 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:20:21 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:roSlrNozkvK7YkiLhKBdyvFpRS0=', 'Content-Length': '0'} 2023-05-31T20:20:21.745 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:18:05 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000001234302677b9f2f3-006477ac05-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:20:21 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:20:21.745 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2023-05-31T20:20:21.746 DEBUG:boto:path=/pklxkd-48/obj4 2023-05-31T20:20:21.747 DEBUG:boto:auth_path=/pklxkd-48/obj4 2023-05-31T20:20:21.747 DEBUG:boto:path=/pklxkd-48/obj4?versionId=null 2023-05-31T20:20:21.747 DEBUG:boto:auth_path=/pklxkd-48/obj4?versionId=null 2023-05-31T20:20:21.748 DEBUG:boto:Method: GET 2023-05-31T20:20:21.748 DEBUG:boto:Path: /pklxkd-48/obj4?versionId=null 2023-05-31T20:20:21.748 DEBUG:boto:Data: 2023-05-31T20:20:21.749 DEBUG:boto:Headers: {} 2023-05-31T20:20:21.749 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:20:21.750 DEBUG:boto:Port: 8001 2023-05-31T20:20:21.750 DEBUG:boto:Params: {} 2023-05-31T20:20:21.750 DEBUG:boto:Token: None 2023-05-31T20:20:21.751 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:20:21 GMT /pklxkd-48/obj4?versionId=null 2023-05-31T20:20:21.751 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:WAKirCNDq2ff7QQOa7SldjYa+x0= 2023-05-31T20:20:21.752 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:20:21 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:WAKirCNDq2ff7QQOa7SldjYa+x0=', 'Content-Length': '0'} 2023-05-31T20:20:21.753 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:18:05 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000c885024806a0defb-006477ac05-162e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:20:21 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:20:21.754 DEBUG:boto:path=/pklxkd-48/obj4 2023-05-31T20:20:21.755 DEBUG:boto:auth_path=/pklxkd-48/obj4 2023-05-31T20:20:21.755 DEBUG:boto:path=/pklxkd-48/obj4?versionId=null 2023-05-31T20:20:21.755 DEBUG:boto:auth_path=/pklxkd-48/obj4?versionId=null 2023-05-31T20:20:21.756 DEBUG:boto:Method: GET 2023-05-31T20:20:21.756 DEBUG:boto:Path: /pklxkd-48/obj4?versionId=null 2023-05-31T20:20:21.757 DEBUG:boto:Data: 2023-05-31T20:20:21.757 DEBUG:boto:Headers: {} 2023-05-31T20:20:21.758 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:20:21.758 DEBUG:boto:Port: 8000 2023-05-31T20:20:21.758 DEBUG:boto:Params: {} 2023-05-31T20:20:21.759 DEBUG:boto:Token: None 2023-05-31T20:20:21.759 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:20:21 GMT /pklxkd-48/obj4?versionId=null 2023-05-31T20:20:21.760 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:WAKirCNDq2ff7QQOa7SldjYa+x0= 2023-05-31T20:20:21.761 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:20:21 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:WAKirCNDq2ff7QQOa7SldjYa+x0=', 'Content-Length': '0'} 2023-05-31T20:20:21.763 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:18:05 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000c461dfb759c8a9d2-006477ac05-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:20:21 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:20:21.763 DEBUG:boto:path=/pklxkd-48/ 2023-05-31T20:20:21.764 DEBUG:boto:auth_path=/pklxkd-48/ 2023-05-31T20:20:21.764 DEBUG:boto:path=/pklxkd-48/?prefix=obj4 2023-05-31T20:20:21.765 DEBUG:boto:auth_path=/pklxkd-48/?prefix=obj4 2023-05-31T20:20:21.765 DEBUG:boto:Method: GET 2023-05-31T20:20:21.765 DEBUG:boto:Path: /pklxkd-48/?prefix=obj4 2023-05-31T20:20:21.766 DEBUG:boto:Data: 2023-05-31T20:20:21.766 DEBUG:boto:Headers: {} 2023-05-31T20:20:21.766 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:20:21.767 DEBUG:boto:Port: 8001 2023-05-31T20:20:21.767 DEBUG:boto:Params: {} 2023-05-31T20:20:21.768 DEBUG:boto:Token: None 2023-05-31T20:20:21.768 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:20:21 GMT /pklxkd-48/ 2023-05-31T20:20:21.769 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:Thv65QkXrDi4etOqQSo2zqkmHRE= 2023-05-31T20:20:21.769 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:20:21 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:Thv65QkXrDi4etOqQSo2zqkmHRE=', 'Content-Length': '0'} 2023-05-31T20:20:21.772 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000eca3d81e3ef79789-006477ac05-162e-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:20:21 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:20:21.772 DEBUG:boto:b'pklxkd-48obj41000falseobj42023-05-31T20:18:05.356Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:20:21.773 DEBUG:boto:path=/pklxkd-48/ 2023-05-31T20:20:21.773 DEBUG:boto:auth_path=/pklxkd-48/ 2023-05-31T20:20:21.774 DEBUG:boto:path=/pklxkd-48/?prefix=obj4 2023-05-31T20:20:21.774 DEBUG:boto:auth_path=/pklxkd-48/?prefix=obj4 2023-05-31T20:20:21.774 DEBUG:boto:Method: GET 2023-05-31T20:20:21.775 DEBUG:boto:Path: /pklxkd-48/?prefix=obj4 2023-05-31T20:20:21.775 DEBUG:boto:Data: 2023-05-31T20:20:21.776 DEBUG:boto:Headers: {} 2023-05-31T20:20:21.776 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:20:21.777 DEBUG:boto:Port: 8000 2023-05-31T20:20:21.777 DEBUG:boto:Params: {} 2023-05-31T20:20:21.777 DEBUG:boto:Token: None 2023-05-31T20:20:21.778 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:20:21 GMT /pklxkd-48/ 2023-05-31T20:20:21.779 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:Thv65QkXrDi4etOqQSo2zqkmHRE= 2023-05-31T20:20:21.779 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:20:21 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:Thv65QkXrDi4etOqQSo2zqkmHRE=', 'Content-Length': '0'} 2023-05-31T20:20:21.781 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b98511f338a9965f-006477ac05-1630-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:20:21 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:20:21.782 DEBUG:boto:b'pklxkd-48obj41000falseobj42023-05-31T20:18:05.356Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:20:21.782 DEBUG:boto:path=/pklxkd-48/obj4 2023-05-31T20:20:21.783 DEBUG:boto:auth_path=/pklxkd-48/obj4 2023-05-31T20:20:21.783 DEBUG:boto:Method: HEAD 2023-05-31T20:20:21.784 DEBUG:boto:Path: /pklxkd-48/obj4 2023-05-31T20:20:21.784 DEBUG:boto:Data: 2023-05-31T20:20:21.784 DEBUG:boto:Headers: {} 2023-05-31T20:20:21.785 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:20:21.785 DEBUG:boto:Port: 8001 2023-05-31T20:20:21.785 DEBUG:boto:Params: {} 2023-05-31T20:20:21.786 DEBUG:boto:Token: None 2023-05-31T20:20:21.786 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:20:21 GMT /pklxkd-48/obj4 2023-05-31T20:20:21.787 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:gIy0A7JserflENVOaVyglxcA8OQ= 2023-05-31T20:20:21.787 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:20:21 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:gIy0A7JserflENVOaVyglxcA8OQ=', 'Content-Length': '0'} 2023-05-31T20:20:21.789 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:18:05 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000fe9dc821d7091722-006477ac05-162e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:20:21 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:20:21.789 DEBUG:boto:path=/pklxkd-48/obj4 2023-05-31T20:20:21.790 DEBUG:boto:auth_path=/pklxkd-48/obj4 2023-05-31T20:20:21.790 DEBUG:boto:Method: HEAD 2023-05-31T20:20:21.791 DEBUG:boto:Path: /pklxkd-48/obj4 2023-05-31T20:20:21.791 DEBUG:boto:Data: 2023-05-31T20:20:21.791 DEBUG:boto:Headers: {} 2023-05-31T20:20:21.792 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:20:21.792 DEBUG:boto:Port: 8000 2023-05-31T20:20:21.792 DEBUG:boto:Params: {} 2023-05-31T20:20:21.793 DEBUG:boto:Token: None 2023-05-31T20:20:21.793 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:20:21 GMT /pklxkd-48/obj4 2023-05-31T20:20:21.793 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:gIy0A7JserflENVOaVyglxcA8OQ= 2023-05-31T20:20:21.794 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:20:21 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:gIy0A7JserflENVOaVyglxcA8OQ=', 'Content-Length': '0'} 2023-05-31T20:20:21.796 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:18:05 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000008d1e3a324286c4be-006477ac05-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:20:21 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:20:21.796 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2023-05-31T20:20:21.797 DEBUG:boto:path=/pklxkd-48/obj4 2023-05-31T20:20:21.797 DEBUG:boto:auth_path=/pklxkd-48/obj4 2023-05-31T20:20:21.798 DEBUG:boto:Method: GET 2023-05-31T20:20:21.798 DEBUG:boto:Path: /pklxkd-48/obj4 2023-05-31T20:20:21.798 DEBUG:boto:Data: 2023-05-31T20:20:21.799 DEBUG:boto:Headers: {} 2023-05-31T20:20:21.799 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:20:21.799 DEBUG:boto:Port: 8001 2023-05-31T20:20:21.800 DEBUG:boto:Params: {} 2023-05-31T20:20:21.800 DEBUG:boto:Token: None 2023-05-31T20:20:21.801 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:20:21 GMT /pklxkd-48/obj4 2023-05-31T20:20:21.801 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:a0WlYBsEWKIV1NXkl95lphoerwY= 2023-05-31T20:20:21.801 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:20:21 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:a0WlYBsEWKIV1NXkl95lphoerwY=', 'Content-Length': '0'} 2023-05-31T20:20:21.804 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:18:05 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000e4c7ca8ede6b357e-006477ac05-162e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:20:21 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:20:21.844 DEBUG:boto:path=/pklxkd-48/obj4 2023-05-31T20:20:21.845 DEBUG:boto:auth_path=/pklxkd-48/obj4 2023-05-31T20:20:21.846 DEBUG:boto:Method: GET 2023-05-31T20:20:21.847 DEBUG:boto:Path: /pklxkd-48/obj4 2023-05-31T20:20:21.847 DEBUG:boto:Data: 2023-05-31T20:20:21.848 DEBUG:boto:Headers: {} 2023-05-31T20:20:21.849 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:20:21.849 DEBUG:boto:Port: 8000 2023-05-31T20:20:21.852 DEBUG:boto:Params: {} 2023-05-31T20:20:21.852 DEBUG:boto:Token: None 2023-05-31T20:20:21.853 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:20:21 GMT /pklxkd-48/obj4 2023-05-31T20:20:21.853 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:a0WlYBsEWKIV1NXkl95lphoerwY= 2023-05-31T20:20:21.854 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:20:21 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:a0WlYBsEWKIV1NXkl95lphoerwY=', 'Content-Length': '0'} 2023-05-31T20:20:21.856 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:18:05 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000e7a5c376c8f3e37f-006477ac05-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:20:21 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:20:21.856 DEBUG:boto:path=/pklxkd-48/ 2023-05-31T20:20:21.857 DEBUG:boto:auth_path=/pklxkd-48/ 2023-05-31T20:20:21.857 DEBUG:boto:path=/pklxkd-48/?prefix=obj4 2023-05-31T20:20:21.858 DEBUG:boto:auth_path=/pklxkd-48/?prefix=obj4 2023-05-31T20:20:21.858 DEBUG:boto:Method: GET 2023-05-31T20:20:21.858 DEBUG:boto:Path: /pklxkd-48/?prefix=obj4 2023-05-31T20:20:21.859 DEBUG:boto:Data: 2023-05-31T20:20:21.859 DEBUG:boto:Headers: {} 2023-05-31T20:20:21.859 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:20:21.860 DEBUG:boto:Port: 8001 2023-05-31T20:20:21.860 DEBUG:boto:Params: {} 2023-05-31T20:20:21.861 DEBUG:boto:Token: None 2023-05-31T20:20:21.862 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:20:21 GMT /pklxkd-48/ 2023-05-31T20:20:21.862 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:Thv65QkXrDi4etOqQSo2zqkmHRE= 2023-05-31T20:20:21.863 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:20:21 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:Thv65QkXrDi4etOqQSo2zqkmHRE=', 'Content-Length': '0'} 2023-05-31T20:20:21.866 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d0c88be4f99205d7-006477ac05-162e-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:20:21 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:20:21.866 DEBUG:boto:b'pklxkd-48obj41000falseobj42023-05-31T20:18:05.356Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:20:21.867 DEBUG:boto:path=/pklxkd-48/ 2023-05-31T20:20:21.867 DEBUG:boto:auth_path=/pklxkd-48/ 2023-05-31T20:20:21.868 DEBUG:boto:path=/pklxkd-48/?prefix=obj4 2023-05-31T20:20:21.868 DEBUG:boto:auth_path=/pklxkd-48/?prefix=obj4 2023-05-31T20:20:21.869 DEBUG:boto:Method: GET 2023-05-31T20:20:21.869 DEBUG:boto:Path: /pklxkd-48/?prefix=obj4 2023-05-31T20:20:21.869 DEBUG:boto:Data: 2023-05-31T20:20:21.870 DEBUG:boto:Headers: {} 2023-05-31T20:20:21.870 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:20:21.871 DEBUG:boto:Port: 8000 2023-05-31T20:20:21.871 DEBUG:boto:Params: {} 2023-05-31T20:20:21.872 DEBUG:boto:Token: None 2023-05-31T20:20:21.872 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:20:21 GMT /pklxkd-48/ 2023-05-31T20:20:21.873 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:Thv65QkXrDi4etOqQSo2zqkmHRE= 2023-05-31T20:20:21.873 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:20:21 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:Thv65QkXrDi4etOqQSo2zqkmHRE=', 'Content-Length': '0'} 2023-05-31T20:20:21.876 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a30064795189aa5c-006477ac05-1630-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:20:21 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:20:21.876 DEBUG:boto:b'pklxkd-48obj41000falseobj42023-05-31T20:18:05.356Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:20:21.877 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=pklxkd-48 zones={a2, a1} 2023-05-31T20:20:21.877 DEBUG:rgw_multi.tests:bucket checkpoint: source=a1 target=a2 bucket=pklxkd-49 2023-05-31T20:20:21.878 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'pklxkd-49', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-05-31T20:20:21.878 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'pklxkd-49', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-05-31T20:20:21.879 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket pklxkd-49 --source-zone a1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2023-05-31T20:20:21.943 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2023-05-31T20:20:21.943 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2023-05-31T20:20:22.004 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:20:21.999+0000 7f5891770780 1 bucket sync caught up with source: 2023-05-31T20:20:22.004 INFO:teuthology.orchestra.run.smithi099.stderr: local status: [00000000001.58.6, , 00000000001.27.6, , 00000000001.20.6, , , 00000000001.29.6, , , ] 2023-05-31T20:20:22.005 INFO:teuthology.orchestra.run.smithi099.stderr: remote markers: [00000000001.58.6, , 00000000001.27.6, , 00000000001.20.6, , , 00000000001.29.6, , , ] 2023-05-31T20:20:22.005 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:20:21.999+0000 7f5891770780 0 bucket checkpoint complete 2023-05-31T20:20:22.027 DEBUG:rgw_multi.tests:bucket checkpoint: source=a2 target=a1 bucket=pklxkd-49 2023-05-31T20:20:22.027 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'pklxkd-49', '--source-zone', 'a2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-05-31T20:20:22.028 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'pklxkd-49', '--source-zone', 'a2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-05-31T20:20:22.029 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket pklxkd-49 --source-zone a2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2023-05-31T20:20:22.086 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2023-05-31T20:20:22.087 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2023-05-31T20:20:22.138 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:20:22.133+0000 7f0432cc9780 1 bucket sync caught up with source: 2023-05-31T20:20:22.139 INFO:teuthology.orchestra.run.smithi099.stderr: local status: [00000000001.3833.6, , 00000000001.1318.6, , 00000000001.2234.6, , , 00000000001.1337.6, , , ] 2023-05-31T20:20:22.139 INFO:teuthology.orchestra.run.smithi099.stderr: remote markers: [00000000001.3833.6, , 00000000001.1318.6, , 00000000001.2234.6, , , 00000000001.1337.6, , , ] 2023-05-31T20:20:22.139 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:20:22.133+0000 7f0432cc9780 0 bucket checkpoint complete 2023-05-31T20:20:22.158 INFO:tasks.rgw_multi.zone_rados:comparing bucket=pklxkd-49 zones={a2, a1} 2023-05-31T20:20:22.159 DEBUG:boto:path=/pklxkd-49/ 2023-05-31T20:20:22.159 DEBUG:boto:auth_path=/pklxkd-49/ 2023-05-31T20:20:22.160 DEBUG:boto:Method: HEAD 2023-05-31T20:20:22.160 DEBUG:boto:Path: /pklxkd-49/ 2023-05-31T20:20:22.160 DEBUG:boto:Data: 2023-05-31T20:20:22.161 DEBUG:boto:Headers: {} 2023-05-31T20:20:22.161 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:20:22.161 DEBUG:boto:Port: 8001 2023-05-31T20:20:22.162 DEBUG:boto:Params: {} 2023-05-31T20:20:22.162 DEBUG:boto:Token: None 2023-05-31T20:20:22.162 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:20:22 GMT /pklxkd-49/ 2023-05-31T20:20:22.163 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:J5m5P7CoGDoc/qy0B5CJq2p3dSY= 2023-05-31T20:20:22.163 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:20:22 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:J5m5P7CoGDoc/qy0B5CJq2p3dSY=', 'Content-Length': '0'} 2023-05-31T20:20:22.166 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '4'), ('X-RGW-Bytes-Used', '24'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000083b1e9139bb4acc3-006477ac06-162e-a2'), ('Content-Length', '0'), ('Date', 'Wed, 31 May 2023 20:20:22 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:20:22.166 DEBUG:boto:path=/pklxkd-49/ 2023-05-31T20:20:22.167 DEBUG:boto:auth_path=/pklxkd-49/ 2023-05-31T20:20:22.167 DEBUG:boto:Method: HEAD 2023-05-31T20:20:22.167 DEBUG:boto:Path: /pklxkd-49/ 2023-05-31T20:20:22.168 DEBUG:boto:Data: 2023-05-31T20:20:22.168 DEBUG:boto:Headers: {} 2023-05-31T20:20:22.169 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:20:22.169 DEBUG:boto:Port: 8000 2023-05-31T20:20:22.169 DEBUG:boto:Params: {} 2023-05-31T20:20:22.170 DEBUG:boto:Token: None 2023-05-31T20:20:22.170 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:20:22 GMT /pklxkd-49/ 2023-05-31T20:20:22.170 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:J5m5P7CoGDoc/qy0B5CJq2p3dSY= 2023-05-31T20:20:22.171 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:20:22 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:J5m5P7CoGDoc/qy0B5CJq2p3dSY=', 'Content-Length': '0'} 2023-05-31T20:20:22.174 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '4'), ('X-RGW-Bytes-Used', '24'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000d4838409ba98b35d-006477ac06-1630-a1'), ('Content-Length', '0'), ('Date', 'Wed, 31 May 2023 20:20:22 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:20:22.174 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2023-05-31T20:20:22.175 DEBUG:boto:path=/pklxkd-49/ 2023-05-31T20:20:22.175 DEBUG:boto:auth_path=/pklxkd-49/ 2023-05-31T20:20:22.175 DEBUG:boto:path=/pklxkd-49/?versions&max-keys=999 2023-05-31T20:20:22.176 DEBUG:boto:auth_path=/pklxkd-49/?versions&max-keys=999 2023-05-31T20:20:22.176 DEBUG:boto:Method: GET 2023-05-31T20:20:22.177 DEBUG:boto:Path: /pklxkd-49/?versions&max-keys=999 2023-05-31T20:20:22.177 DEBUG:boto:Data: 2023-05-31T20:20:22.177 DEBUG:boto:Headers: {} 2023-05-31T20:20:22.178 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:20:22.178 DEBUG:boto:Port: 8001 2023-05-31T20:20:22.179 DEBUG:boto:Params: {} 2023-05-31T20:20:22.179 DEBUG:boto:Token: None 2023-05-31T20:20:22.180 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:20:22 GMT /pklxkd-49/?versions 2023-05-31T20:20:22.180 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:ridkIWlV7yJ7+d06it1lpjivzG4= 2023-05-31T20:20:22.181 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:20:22 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:ridkIWlV7yJ7+d06it1lpjivzG4=', 'Content-Length': '0'} 2023-05-31T20:20:22.183 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000009a87c9e62793ba2e-006477ac06-162e-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:20:22 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:20:22.183 DEBUG:boto:b'pklxkd-49999falseobj1nulltrue2023-05-31T20:18:05.410Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj2nulltrue2023-05-31T20:18:05.430Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj3nulltrue2023-05-31T20:18:05.493Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj4nulltrue2023-05-31T20:18:05.509Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:20:22.184 DEBUG:tasks.rgw_multi.zone_rados:o=obj1 2023-05-31T20:20:22.185 DEBUG:tasks.rgw_multi.zone_rados:o=obj2 2023-05-31T20:20:22.185 DEBUG:tasks.rgw_multi.zone_rados:o=obj3 2023-05-31T20:20:22.185 DEBUG:tasks.rgw_multi.zone_rados:o=obj4 2023-05-31T20:20:22.185 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2023-05-31T20:20:22.186 DEBUG:boto:path=/pklxkd-49/ 2023-05-31T20:20:22.186 DEBUG:boto:auth_path=/pklxkd-49/ 2023-05-31T20:20:22.187 DEBUG:boto:path=/pklxkd-49/?versions&max-keys=999 2023-05-31T20:20:22.187 DEBUG:boto:auth_path=/pklxkd-49/?versions&max-keys=999 2023-05-31T20:20:22.187 DEBUG:boto:Method: GET 2023-05-31T20:20:22.188 DEBUG:boto:Path: /pklxkd-49/?versions&max-keys=999 2023-05-31T20:20:22.188 DEBUG:boto:Data: 2023-05-31T20:20:22.188 DEBUG:boto:Headers: {} 2023-05-31T20:20:22.189 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:20:22.189 DEBUG:boto:Port: 8000 2023-05-31T20:20:22.189 DEBUG:boto:Params: {} 2023-05-31T20:20:22.190 DEBUG:boto:Token: None 2023-05-31T20:20:22.190 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:20:22 GMT /pklxkd-49/?versions 2023-05-31T20:20:22.191 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:ridkIWlV7yJ7+d06it1lpjivzG4= 2023-05-31T20:20:22.191 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:20:22 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:ridkIWlV7yJ7+d06it1lpjivzG4=', 'Content-Length': '0'} 2023-05-31T20:20:22.193 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000002d31147a2dab0aea-006477ac06-1630-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:20:22 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:20:22.194 DEBUG:boto:b'pklxkd-49999falseobj1nulltrue2023-05-31T20:18:05.410Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj2nulltrue2023-05-31T20:18:05.430Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj3nulltrue2023-05-31T20:18:05.493Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj4nulltrue2023-05-31T20:18:05.509Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:20:22.195 DEBUG:tasks.rgw_multi.zone_rados:o=obj1 2023-05-31T20:20:22.195 DEBUG:tasks.rgw_multi.zone_rados:o=obj2 2023-05-31T20:20:22.195 DEBUG:tasks.rgw_multi.zone_rados:o=obj3 2023-05-31T20:20:22.196 DEBUG:tasks.rgw_multi.zone_rados:o=obj4 2023-05-31T20:20:22.196 DEBUG:boto:path=/pklxkd-49/ 2023-05-31T20:20:22.197 DEBUG:boto:auth_path=/pklxkd-49/ 2023-05-31T20:20:22.197 DEBUG:boto:path=/pklxkd-49/?versions&max-keys=999 2023-05-31T20:20:22.197 DEBUG:boto:auth_path=/pklxkd-49/?versions&max-keys=999 2023-05-31T20:20:22.198 DEBUG:boto:Method: GET 2023-05-31T20:20:22.198 DEBUG:boto:Path: /pklxkd-49/?versions&max-keys=999 2023-05-31T20:20:22.198 DEBUG:boto:Data: 2023-05-31T20:20:22.199 DEBUG:boto:Headers: {} 2023-05-31T20:20:22.199 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:20:22.199 DEBUG:boto:Port: 8001 2023-05-31T20:20:22.200 DEBUG:boto:Params: {} 2023-05-31T20:20:22.200 DEBUG:boto:Token: None 2023-05-31T20:20:22.201 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:20:22 GMT /pklxkd-49/?versions 2023-05-31T20:20:22.201 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:ridkIWlV7yJ7+d06it1lpjivzG4= 2023-05-31T20:20:22.201 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:20:22 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:ridkIWlV7yJ7+d06it1lpjivzG4=', 'Content-Length': '0'} 2023-05-31T20:20:22.204 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000002f218eb6cbbb998a-006477ac06-162e-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:20:22 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:20:22.205 DEBUG:boto:b'pklxkd-49999falseobj1nulltrue2023-05-31T20:18:05.410Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj2nulltrue2023-05-31T20:18:05.430Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj3nulltrue2023-05-31T20:18:05.493Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj4nulltrue2023-05-31T20:18:05.509Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:20:22.206 DEBUG:boto:path=/pklxkd-49/ 2023-05-31T20:20:22.206 DEBUG:boto:auth_path=/pklxkd-49/ 2023-05-31T20:20:22.207 DEBUG:boto:path=/pklxkd-49/?versions&max-keys=999 2023-05-31T20:20:22.207 DEBUG:boto:auth_path=/pklxkd-49/?versions&max-keys=999 2023-05-31T20:20:22.207 DEBUG:boto:Method: GET 2023-05-31T20:20:22.208 DEBUG:boto:Path: /pklxkd-49/?versions&max-keys=999 2023-05-31T20:20:22.208 DEBUG:boto:Data: 2023-05-31T20:20:22.208 DEBUG:boto:Headers: {} 2023-05-31T20:20:22.209 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:20:22.209 DEBUG:boto:Port: 8000 2023-05-31T20:20:22.210 DEBUG:boto:Params: {} 2023-05-31T20:20:22.210 DEBUG:boto:Token: None 2023-05-31T20:20:22.211 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:20:22 GMT /pklxkd-49/?versions 2023-05-31T20:20:22.211 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:ridkIWlV7yJ7+d06it1lpjivzG4= 2023-05-31T20:20:22.211 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:20:22 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:ridkIWlV7yJ7+d06it1lpjivzG4=', 'Content-Length': '0'} 2023-05-31T20:20:22.214 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000008be70bd42bd94253-006477ac06-1630-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:20:22 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:20:22.215 DEBUG:boto:b'pklxkd-49999falseobj1nulltrue2023-05-31T20:18:05.410Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj2nulltrue2023-05-31T20:18:05.430Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj3nulltrue2023-05-31T20:18:05.493Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj4nulltrue2023-05-31T20:18:05.509Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:20:22.216 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2023-05-31T20:20:22.217 DEBUG:boto:path=/pklxkd-49/obj1 2023-05-31T20:20:22.218 DEBUG:boto:auth_path=/pklxkd-49/obj1 2023-05-31T20:20:22.218 DEBUG:boto:path=/pklxkd-49/obj1?versionId=null 2023-05-31T20:20:22.219 DEBUG:boto:auth_path=/pklxkd-49/obj1?versionId=null 2023-05-31T20:20:22.220 DEBUG:boto:Method: GET 2023-05-31T20:20:22.221 DEBUG:boto:Path: /pklxkd-49/obj1?versionId=null 2023-05-31T20:20:22.221 DEBUG:boto:Data: 2023-05-31T20:20:22.222 DEBUG:boto:Headers: {} 2023-05-31T20:20:22.222 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:20:22.223 DEBUG:boto:Port: 8001 2023-05-31T20:20:22.223 DEBUG:boto:Params: {} 2023-05-31T20:20:22.223 DEBUG:boto:Token: None 2023-05-31T20:20:22.224 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:20:22 GMT /pklxkd-49/obj1?versionId=null 2023-05-31T20:20:22.225 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:8TySsB6SSnduxLwEnX0WLPqwaAc= 2023-05-31T20:20:22.225 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:20:22 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:8TySsB6SSnduxLwEnX0WLPqwaAc=', 'Content-Length': '0'} 2023-05-31T20:20:22.227 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:18:05 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000080bf6e81ace48589-006477ac06-162e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:20:22 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:20:22.227 DEBUG:boto:path=/pklxkd-49/obj1 2023-05-31T20:20:22.228 DEBUG:boto:auth_path=/pklxkd-49/obj1 2023-05-31T20:20:22.228 DEBUG:boto:path=/pklxkd-49/obj1?versionId=null 2023-05-31T20:20:22.228 DEBUG:boto:auth_path=/pklxkd-49/obj1?versionId=null 2023-05-31T20:20:22.229 DEBUG:boto:Method: GET 2023-05-31T20:20:22.229 DEBUG:boto:Path: /pklxkd-49/obj1?versionId=null 2023-05-31T20:20:22.230 DEBUG:boto:Data: 2023-05-31T20:20:22.230 DEBUG:boto:Headers: {} 2023-05-31T20:20:22.230 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:20:22.231 DEBUG:boto:Port: 8000 2023-05-31T20:20:22.231 DEBUG:boto:Params: {} 2023-05-31T20:20:22.231 DEBUG:boto:Token: None 2023-05-31T20:20:22.232 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:20:22 GMT /pklxkd-49/obj1?versionId=null 2023-05-31T20:20:22.232 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:8TySsB6SSnduxLwEnX0WLPqwaAc= 2023-05-31T20:20:22.233 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:20:22 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:8TySsB6SSnduxLwEnX0WLPqwaAc=', 'Content-Length': '0'} 2023-05-31T20:20:22.234 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:18:05 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000041d2abaad4bf28e5-006477ac06-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:20:22 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:20:22.235 DEBUG:boto:path=/pklxkd-49/obj1 2023-05-31T20:20:22.235 DEBUG:boto:auth_path=/pklxkd-49/obj1 2023-05-31T20:20:22.236 DEBUG:boto:path=/pklxkd-49/obj1?versionId=null 2023-05-31T20:20:22.236 DEBUG:boto:auth_path=/pklxkd-49/obj1?versionId=null 2023-05-31T20:20:22.236 DEBUG:boto:Method: HEAD 2023-05-31T20:20:22.237 DEBUG:boto:Path: /pklxkd-49/obj1?versionId=null 2023-05-31T20:20:22.237 DEBUG:boto:Data: 2023-05-31T20:20:22.237 DEBUG:boto:Headers: {} 2023-05-31T20:20:22.238 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:20:22.238 DEBUG:boto:Port: 8001 2023-05-31T20:20:22.238 DEBUG:boto:Params: {} 2023-05-31T20:20:22.239 DEBUG:boto:Token: None 2023-05-31T20:20:22.239 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:20:22 GMT /pklxkd-49/obj1?versionId=null 2023-05-31T20:20:22.240 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:yLt4JfvAAM967bpKAkAr4umOFrI= 2023-05-31T20:20:22.240 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:20:22 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:yLt4JfvAAM967bpKAkAr4umOFrI=', 'Content-Length': '0'} 2023-05-31T20:20:22.241 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:18:05 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000791e7ba31c162c3e-006477ac06-162e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:20:22 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:20:22.242 DEBUG:boto:path=/pklxkd-49/obj1 2023-05-31T20:20:22.242 DEBUG:boto:auth_path=/pklxkd-49/obj1 2023-05-31T20:20:22.243 DEBUG:boto:path=/pklxkd-49/obj1?versionId=null 2023-05-31T20:20:22.243 DEBUG:boto:auth_path=/pklxkd-49/obj1?versionId=null 2023-05-31T20:20:22.243 DEBUG:boto:Method: HEAD 2023-05-31T20:20:22.244 DEBUG:boto:Path: /pklxkd-49/obj1?versionId=null 2023-05-31T20:20:22.244 DEBUG:boto:Data: 2023-05-31T20:20:22.244 DEBUG:boto:Headers: {} 2023-05-31T20:20:22.245 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:20:22.245 DEBUG:boto:Port: 8000 2023-05-31T20:20:22.245 DEBUG:boto:Params: {} 2023-05-31T20:20:22.246 DEBUG:boto:Token: None 2023-05-31T20:20:22.246 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:20:22 GMT /pklxkd-49/obj1?versionId=null 2023-05-31T20:20:22.247 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:yLt4JfvAAM967bpKAkAr4umOFrI= 2023-05-31T20:20:22.247 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:20:22 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:yLt4JfvAAM967bpKAkAr4umOFrI=', 'Content-Length': '0'} 2023-05-31T20:20:22.248 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:18:05 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000001adca09ba9f69e4c-006477ac06-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:20:22 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:20:22.249 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2023-05-31T20:20:22.249 DEBUG:boto:path=/pklxkd-49/obj1 2023-05-31T20:20:22.250 DEBUG:boto:auth_path=/pklxkd-49/obj1 2023-05-31T20:20:22.250 DEBUG:boto:path=/pklxkd-49/obj1?versionId=null 2023-05-31T20:20:22.250 DEBUG:boto:auth_path=/pklxkd-49/obj1?versionId=null 2023-05-31T20:20:22.251 DEBUG:boto:Method: GET 2023-05-31T20:20:22.251 DEBUG:boto:Path: /pklxkd-49/obj1?versionId=null 2023-05-31T20:20:22.251 DEBUG:boto:Data: 2023-05-31T20:20:22.252 DEBUG:boto:Headers: {} 2023-05-31T20:20:22.252 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:20:22.253 DEBUG:boto:Port: 8001 2023-05-31T20:20:22.253 DEBUG:boto:Params: {} 2023-05-31T20:20:22.253 DEBUG:boto:Token: None 2023-05-31T20:20:22.254 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:20:22 GMT /pklxkd-49/obj1?versionId=null 2023-05-31T20:20:22.254 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:8TySsB6SSnduxLwEnX0WLPqwaAc= 2023-05-31T20:20:22.255 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:20:22 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:8TySsB6SSnduxLwEnX0WLPqwaAc=', 'Content-Length': '0'} 2023-05-31T20:20:22.256 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:18:05 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000002a64f004e8009f3a-006477ac06-162e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:20:22 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:20:22.258 DEBUG:boto:path=/pklxkd-49/obj1 2023-05-31T20:20:22.258 DEBUG:boto:auth_path=/pklxkd-49/obj1 2023-05-31T20:20:22.259 DEBUG:boto:path=/pklxkd-49/obj1?versionId=null 2023-05-31T20:20:22.259 DEBUG:boto:auth_path=/pklxkd-49/obj1?versionId=null 2023-05-31T20:20:22.259 DEBUG:boto:Method: GET 2023-05-31T20:20:22.260 DEBUG:boto:Path: /pklxkd-49/obj1?versionId=null 2023-05-31T20:20:22.260 DEBUG:boto:Data: 2023-05-31T20:20:22.261 DEBUG:boto:Headers: {} 2023-05-31T20:20:22.261 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:20:22.261 DEBUG:boto:Port: 8000 2023-05-31T20:20:22.262 DEBUG:boto:Params: {} 2023-05-31T20:20:22.262 DEBUG:boto:Token: None 2023-05-31T20:20:22.263 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:20:22 GMT /pklxkd-49/obj1?versionId=null 2023-05-31T20:20:22.263 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:8TySsB6SSnduxLwEnX0WLPqwaAc= 2023-05-31T20:20:22.263 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:20:22 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:8TySsB6SSnduxLwEnX0WLPqwaAc=', 'Content-Length': '0'} 2023-05-31T20:20:22.265 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:18:05 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000005a2896ce54e8a644-006477ac06-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:20:22 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:20:22.308 DEBUG:boto:path=/pklxkd-49/ 2023-05-31T20:20:22.309 DEBUG:boto:auth_path=/pklxkd-49/ 2023-05-31T20:20:22.309 DEBUG:boto:path=/pklxkd-49/?prefix=obj1 2023-05-31T20:20:22.310 DEBUG:boto:auth_path=/pklxkd-49/?prefix=obj1 2023-05-31T20:20:22.310 DEBUG:boto:Method: GET 2023-05-31T20:20:22.311 DEBUG:boto:Path: /pklxkd-49/?prefix=obj1 2023-05-31T20:20:22.311 DEBUG:boto:Data: 2023-05-31T20:20:22.311 DEBUG:boto:Headers: {} 2023-05-31T20:20:22.312 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:20:22.312 DEBUG:boto:Port: 8001 2023-05-31T20:20:22.312 DEBUG:boto:Params: {} 2023-05-31T20:20:22.313 DEBUG:boto:Token: None 2023-05-31T20:20:22.313 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:20:22 GMT /pklxkd-49/ 2023-05-31T20:20:22.314 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:lW1cZMSyiySZM1lccSseC4NYx4k= 2023-05-31T20:20:22.314 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:20:22 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:lW1cZMSyiySZM1lccSseC4NYx4k=', 'Content-Length': '0'} 2023-05-31T20:20:22.316 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000006c62116703427430-006477ac06-162e-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:20:22 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:20:22.317 DEBUG:boto:b'pklxkd-49obj11000falseobj12023-05-31T20:18:05.410Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:20:22.318 DEBUG:boto:path=/pklxkd-49/ 2023-05-31T20:20:22.318 DEBUG:boto:auth_path=/pklxkd-49/ 2023-05-31T20:20:22.318 DEBUG:boto:path=/pklxkd-49/?prefix=obj1 2023-05-31T20:20:22.319 DEBUG:boto:auth_path=/pklxkd-49/?prefix=obj1 2023-05-31T20:20:22.320 DEBUG:boto:Method: GET 2023-05-31T20:20:22.320 DEBUG:boto:Path: /pklxkd-49/?prefix=obj1 2023-05-31T20:20:22.321 DEBUG:boto:Data: 2023-05-31T20:20:22.322 DEBUG:boto:Headers: {} 2023-05-31T20:20:22.323 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:20:22.324 DEBUG:boto:Port: 8000 2023-05-31T20:20:22.325 DEBUG:boto:Params: {} 2023-05-31T20:20:22.326 DEBUG:boto:Token: None 2023-05-31T20:20:22.327 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:20:22 GMT /pklxkd-49/ 2023-05-31T20:20:22.327 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:lW1cZMSyiySZM1lccSseC4NYx4k= 2023-05-31T20:20:22.328 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:20:22 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:lW1cZMSyiySZM1lccSseC4NYx4k=', 'Content-Length': '0'} 2023-05-31T20:20:22.331 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000577d77fa7eab3e3f-006477ac06-1630-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:20:22 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:20:22.331 DEBUG:boto:b'pklxkd-49obj11000falseobj12023-05-31T20:18:05.410Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:20:22.332 DEBUG:boto:path=/pklxkd-49/obj1 2023-05-31T20:20:22.333 DEBUG:boto:auth_path=/pklxkd-49/obj1 2023-05-31T20:20:22.333 DEBUG:boto:Method: HEAD 2023-05-31T20:20:22.333 DEBUG:boto:Path: /pklxkd-49/obj1 2023-05-31T20:20:22.334 DEBUG:boto:Data: 2023-05-31T20:20:22.334 DEBUG:boto:Headers: {} 2023-05-31T20:20:22.334 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:20:22.335 DEBUG:boto:Port: 8001 2023-05-31T20:20:22.335 DEBUG:boto:Params: {} 2023-05-31T20:20:22.335 DEBUG:boto:Token: None 2023-05-31T20:20:22.336 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:20:22 GMT /pklxkd-49/obj1 2023-05-31T20:20:22.336 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:Aeo/VyR3CHP4bL4pnkeuSvRPMDA= 2023-05-31T20:20:22.337 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:20:22 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:Aeo/VyR3CHP4bL4pnkeuSvRPMDA=', 'Content-Length': '0'} 2023-05-31T20:20:22.339 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:18:05 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000b1acd15acb18d8be-006477ac06-162e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:20:22 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:20:22.339 DEBUG:boto:path=/pklxkd-49/obj1 2023-05-31T20:20:22.340 DEBUG:boto:auth_path=/pklxkd-49/obj1 2023-05-31T20:20:22.340 DEBUG:boto:Method: HEAD 2023-05-31T20:20:22.341 DEBUG:boto:Path: /pklxkd-49/obj1 2023-05-31T20:20:22.341 DEBUG:boto:Data: 2023-05-31T20:20:22.342 DEBUG:boto:Headers: {} 2023-05-31T20:20:22.342 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:20:22.343 DEBUG:boto:Port: 8000 2023-05-31T20:20:22.344 DEBUG:boto:Params: {} 2023-05-31T20:20:22.345 DEBUG:boto:Token: None 2023-05-31T20:20:22.346 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:20:22 GMT /pklxkd-49/obj1 2023-05-31T20:20:22.347 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:Aeo/VyR3CHP4bL4pnkeuSvRPMDA= 2023-05-31T20:20:22.348 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:20:22 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:Aeo/VyR3CHP4bL4pnkeuSvRPMDA=', 'Content-Length': '0'} 2023-05-31T20:20:22.350 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:18:05 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000bfa2ec78c41d17bc-006477ac06-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:20:22 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:20:22.351 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2023-05-31T20:20:22.352 DEBUG:boto:path=/pklxkd-49/obj1 2023-05-31T20:20:22.352 DEBUG:boto:auth_path=/pklxkd-49/obj1 2023-05-31T20:20:22.353 DEBUG:boto:Method: GET 2023-05-31T20:20:22.354 DEBUG:boto:Path: /pklxkd-49/obj1 2023-05-31T20:20:22.355 DEBUG:boto:Data: 2023-05-31T20:20:22.355 DEBUG:boto:Headers: {} 2023-05-31T20:20:22.355 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:20:22.356 DEBUG:boto:Port: 8001 2023-05-31T20:20:22.356 DEBUG:boto:Params: {} 2023-05-31T20:20:22.357 DEBUG:boto:Token: None 2023-05-31T20:20:22.357 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:20:22 GMT /pklxkd-49/obj1 2023-05-31T20:20:22.358 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:iLYwMMDwA94hBLwvo+BdJilBstY= 2023-05-31T20:20:22.358 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:20:22 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:iLYwMMDwA94hBLwvo+BdJilBstY=', 'Content-Length': '0'} 2023-05-31T20:20:22.360 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:18:05 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000016edcb06cfa19e31-006477ac06-162e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:20:22 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:20:22.360 DEBUG:boto:path=/pklxkd-49/obj1 2023-05-31T20:20:22.361 DEBUG:boto:auth_path=/pklxkd-49/obj1 2023-05-31T20:20:22.361 DEBUG:boto:Method: GET 2023-05-31T20:20:22.362 DEBUG:boto:Path: /pklxkd-49/obj1 2023-05-31T20:20:22.362 DEBUG:boto:Data: 2023-05-31T20:20:22.362 DEBUG:boto:Headers: {} 2023-05-31T20:20:22.363 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:20:22.363 DEBUG:boto:Port: 8000 2023-05-31T20:20:22.363 DEBUG:boto:Params: {} 2023-05-31T20:20:22.364 DEBUG:boto:Token: None 2023-05-31T20:20:22.364 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:20:22 GMT /pklxkd-49/obj1 2023-05-31T20:20:22.365 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:iLYwMMDwA94hBLwvo+BdJilBstY= 2023-05-31T20:20:22.365 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:20:22 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:iLYwMMDwA94hBLwvo+BdJilBstY=', 'Content-Length': '0'} 2023-05-31T20:20:22.367 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:18:05 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000efb557b696518644-006477ac06-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:20:22 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:20:22.367 DEBUG:boto:path=/pklxkd-49/ 2023-05-31T20:20:22.368 DEBUG:boto:auth_path=/pklxkd-49/ 2023-05-31T20:20:22.368 DEBUG:boto:path=/pklxkd-49/?prefix=obj1 2023-05-31T20:20:22.368 DEBUG:boto:auth_path=/pklxkd-49/?prefix=obj1 2023-05-31T20:20:22.369 DEBUG:boto:Method: GET 2023-05-31T20:20:22.369 DEBUG:boto:Path: /pklxkd-49/?prefix=obj1 2023-05-31T20:20:22.369 DEBUG:boto:Data: 2023-05-31T20:20:22.370 DEBUG:boto:Headers: {} 2023-05-31T20:20:22.370 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:20:22.371 DEBUG:boto:Port: 8001 2023-05-31T20:20:22.371 DEBUG:boto:Params: {} 2023-05-31T20:20:22.371 DEBUG:boto:Token: None 2023-05-31T20:20:22.372 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:20:22 GMT /pklxkd-49/ 2023-05-31T20:20:22.372 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:lW1cZMSyiySZM1lccSseC4NYx4k= 2023-05-31T20:20:22.373 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:20:22 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:lW1cZMSyiySZM1lccSseC4NYx4k=', 'Content-Length': '0'} 2023-05-31T20:20:22.375 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ed2c7cb1d2aee6de-006477ac06-162e-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:20:22 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:20:22.375 DEBUG:boto:b'pklxkd-49obj11000falseobj12023-05-31T20:18:05.410Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:20:22.376 DEBUG:boto:path=/pklxkd-49/ 2023-05-31T20:20:22.377 DEBUG:boto:auth_path=/pklxkd-49/ 2023-05-31T20:20:22.377 DEBUG:boto:path=/pklxkd-49/?prefix=obj1 2023-05-31T20:20:22.377 DEBUG:boto:auth_path=/pklxkd-49/?prefix=obj1 2023-05-31T20:20:22.378 DEBUG:boto:Method: GET 2023-05-31T20:20:22.378 DEBUG:boto:Path: /pklxkd-49/?prefix=obj1 2023-05-31T20:20:22.378 DEBUG:boto:Data: 2023-05-31T20:20:22.379 DEBUG:boto:Headers: {} 2023-05-31T20:20:22.379 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:20:22.380 DEBUG:boto:Port: 8000 2023-05-31T20:20:22.381 DEBUG:boto:Params: {} 2023-05-31T20:20:22.382 DEBUG:boto:Token: None 2023-05-31T20:20:22.383 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:20:22 GMT /pklxkd-49/ 2023-05-31T20:20:22.384 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:lW1cZMSyiySZM1lccSseC4NYx4k= 2023-05-31T20:20:22.385 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:20:22 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:lW1cZMSyiySZM1lccSseC4NYx4k=', 'Content-Length': '0'} 2023-05-31T20:20:22.387 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000004129e582468f2703-006477ac06-1630-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:20:22 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:20:22.388 DEBUG:boto:b'pklxkd-49obj11000falseobj12023-05-31T20:18:05.410Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:20:22.389 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2023-05-31T20:20:22.389 DEBUG:boto:path=/pklxkd-49/obj2 2023-05-31T20:20:22.389 DEBUG:boto:auth_path=/pklxkd-49/obj2 2023-05-31T20:20:22.390 DEBUG:boto:path=/pklxkd-49/obj2?versionId=null 2023-05-31T20:20:22.390 DEBUG:boto:auth_path=/pklxkd-49/obj2?versionId=null 2023-05-31T20:20:22.391 DEBUG:boto:Method: GET 2023-05-31T20:20:22.391 DEBUG:boto:Path: /pklxkd-49/obj2?versionId=null 2023-05-31T20:20:22.391 DEBUG:boto:Data: 2023-05-31T20:20:22.392 DEBUG:boto:Headers: {} 2023-05-31T20:20:22.392 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:20:22.393 DEBUG:boto:Port: 8001 2023-05-31T20:20:22.393 DEBUG:boto:Params: {} 2023-05-31T20:20:22.393 DEBUG:boto:Token: None 2023-05-31T20:20:22.394 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:20:22 GMT /pklxkd-49/obj2?versionId=null 2023-05-31T20:20:22.394 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:g8eXXodp+EF6QmBljOyAkDxgids= 2023-05-31T20:20:22.395 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:20:22 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:g8eXXodp+EF6QmBljOyAkDxgids=', 'Content-Length': '0'} 2023-05-31T20:20:22.396 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:18:05 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000002a8a1fc8fa31ba08-006477ac06-162e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:20:22 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:20:22.397 DEBUG:boto:path=/pklxkd-49/obj2 2023-05-31T20:20:22.398 DEBUG:boto:auth_path=/pklxkd-49/obj2 2023-05-31T20:20:22.398 DEBUG:boto:path=/pklxkd-49/obj2?versionId=null 2023-05-31T20:20:22.398 DEBUG:boto:auth_path=/pklxkd-49/obj2?versionId=null 2023-05-31T20:20:22.399 DEBUG:boto:Method: GET 2023-05-31T20:20:22.399 DEBUG:boto:Path: /pklxkd-49/obj2?versionId=null 2023-05-31T20:20:22.399 DEBUG:boto:Data: 2023-05-31T20:20:22.400 DEBUG:boto:Headers: {} 2023-05-31T20:20:22.400 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:20:22.401 DEBUG:boto:Port: 8000 2023-05-31T20:20:22.401 DEBUG:boto:Params: {} 2023-05-31T20:20:22.401 DEBUG:boto:Token: None 2023-05-31T20:20:22.402 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:20:22 GMT /pklxkd-49/obj2?versionId=null 2023-05-31T20:20:22.402 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:g8eXXodp+EF6QmBljOyAkDxgids= 2023-05-31T20:20:22.403 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:20:22 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:g8eXXodp+EF6QmBljOyAkDxgids=', 'Content-Length': '0'} 2023-05-31T20:20:22.404 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:18:05 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000020d0ac65c48a6acb-006477ac06-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:20:22 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:20:22.405 DEBUG:boto:path=/pklxkd-49/obj2 2023-05-31T20:20:22.405 DEBUG:boto:auth_path=/pklxkd-49/obj2 2023-05-31T20:20:22.406 DEBUG:boto:path=/pklxkd-49/obj2?versionId=null 2023-05-31T20:20:22.406 DEBUG:boto:auth_path=/pklxkd-49/obj2?versionId=null 2023-05-31T20:20:22.406 DEBUG:boto:Method: HEAD 2023-05-31T20:20:22.407 DEBUG:boto:Path: /pklxkd-49/obj2?versionId=null 2023-05-31T20:20:22.407 DEBUG:boto:Data: 2023-05-31T20:20:22.407 DEBUG:boto:Headers: {} 2023-05-31T20:20:22.408 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:20:22.408 DEBUG:boto:Port: 8001 2023-05-31T20:20:22.408 DEBUG:boto:Params: {} 2023-05-31T20:20:22.409 DEBUG:boto:Token: None 2023-05-31T20:20:22.409 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:20:22 GMT /pklxkd-49/obj2?versionId=null 2023-05-31T20:20:22.409 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:6VGlkvnW/Qg5lV4c3mJLd9p+CkA= 2023-05-31T20:20:22.410 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:20:22 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:6VGlkvnW/Qg5lV4c3mJLd9p+CkA=', 'Content-Length': '0'} 2023-05-31T20:20:22.412 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:18:05 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000050a758a072bb1938-006477ac06-162e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:20:22 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:20:22.412 DEBUG:boto:path=/pklxkd-49/obj2 2023-05-31T20:20:22.413 DEBUG:boto:auth_path=/pklxkd-49/obj2 2023-05-31T20:20:22.413 DEBUG:boto:path=/pklxkd-49/obj2?versionId=null 2023-05-31T20:20:22.413 DEBUG:boto:auth_path=/pklxkd-49/obj2?versionId=null 2023-05-31T20:20:22.414 DEBUG:boto:Method: HEAD 2023-05-31T20:20:22.414 DEBUG:boto:Path: /pklxkd-49/obj2?versionId=null 2023-05-31T20:20:22.414 DEBUG:boto:Data: 2023-05-31T20:20:22.415 DEBUG:boto:Headers: {} 2023-05-31T20:20:22.415 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:20:22.415 DEBUG:boto:Port: 8000 2023-05-31T20:20:22.416 DEBUG:boto:Params: {} 2023-05-31T20:20:22.416 DEBUG:boto:Token: None 2023-05-31T20:20:22.416 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:20:22 GMT /pklxkd-49/obj2?versionId=null 2023-05-31T20:20:22.417 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:6VGlkvnW/Qg5lV4c3mJLd9p+CkA= 2023-05-31T20:20:22.417 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:20:22 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:6VGlkvnW/Qg5lV4c3mJLd9p+CkA=', 'Content-Length': '0'} 2023-05-31T20:20:22.419 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:18:05 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000000fd8c46e812da558-006477ac06-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:20:22 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:20:22.419 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2023-05-31T20:20:22.420 DEBUG:boto:path=/pklxkd-49/obj2 2023-05-31T20:20:22.420 DEBUG:boto:auth_path=/pklxkd-49/obj2 2023-05-31T20:20:22.420 DEBUG:boto:path=/pklxkd-49/obj2?versionId=null 2023-05-31T20:20:22.421 DEBUG:boto:auth_path=/pklxkd-49/obj2?versionId=null 2023-05-31T20:20:22.421 DEBUG:boto:Method: GET 2023-05-31T20:20:22.422 DEBUG:boto:Path: /pklxkd-49/obj2?versionId=null 2023-05-31T20:20:22.422 DEBUG:boto:Data: 2023-05-31T20:20:22.422 DEBUG:boto:Headers: {} 2023-05-31T20:20:22.423 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:20:22.423 DEBUG:boto:Port: 8001 2023-05-31T20:20:22.423 DEBUG:boto:Params: {} 2023-05-31T20:20:22.424 DEBUG:boto:Token: None 2023-05-31T20:20:22.424 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:20:22 GMT /pklxkd-49/obj2?versionId=null 2023-05-31T20:20:22.425 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:g8eXXodp+EF6QmBljOyAkDxgids= 2023-05-31T20:20:22.425 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:20:22 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:g8eXXodp+EF6QmBljOyAkDxgids=', 'Content-Length': '0'} 2023-05-31T20:20:22.427 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:18:05 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000d0a5475063483546-006477ac06-162e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:20:22 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:20:22.468 DEBUG:boto:path=/pklxkd-49/obj2 2023-05-31T20:20:22.469 DEBUG:boto:auth_path=/pklxkd-49/obj2 2023-05-31T20:20:22.469 DEBUG:boto:path=/pklxkd-49/obj2?versionId=null 2023-05-31T20:20:22.470 DEBUG:boto:auth_path=/pklxkd-49/obj2?versionId=null 2023-05-31T20:20:22.470 DEBUG:boto:Method: GET 2023-05-31T20:20:22.470 DEBUG:boto:Path: /pklxkd-49/obj2?versionId=null 2023-05-31T20:20:22.471 DEBUG:boto:Data: 2023-05-31T20:20:22.471 DEBUG:boto:Headers: {} 2023-05-31T20:20:22.471 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:20:22.472 DEBUG:boto:Port: 8000 2023-05-31T20:20:22.472 DEBUG:boto:Params: {} 2023-05-31T20:20:22.473 DEBUG:boto:Token: None 2023-05-31T20:20:22.473 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:20:22 GMT /pklxkd-49/obj2?versionId=null 2023-05-31T20:20:22.473 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:g8eXXodp+EF6QmBljOyAkDxgids= 2023-05-31T20:20:22.474 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:20:22 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:g8eXXodp+EF6QmBljOyAkDxgids=', 'Content-Length': '0'} 2023-05-31T20:20:22.476 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:18:05 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000084815dd20069e829-006477ac06-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:20:22 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:20:22.476 DEBUG:boto:path=/pklxkd-49/ 2023-05-31T20:20:22.477 DEBUG:boto:auth_path=/pklxkd-49/ 2023-05-31T20:20:22.477 DEBUG:boto:path=/pklxkd-49/?prefix=obj2 2023-05-31T20:20:22.477 DEBUG:boto:auth_path=/pklxkd-49/?prefix=obj2 2023-05-31T20:20:22.478 DEBUG:boto:Method: GET 2023-05-31T20:20:22.478 DEBUG:boto:Path: /pklxkd-49/?prefix=obj2 2023-05-31T20:20:22.479 DEBUG:boto:Data: 2023-05-31T20:20:22.479 DEBUG:boto:Headers: {} 2023-05-31T20:20:22.479 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:20:22.480 DEBUG:boto:Port: 8001 2023-05-31T20:20:22.480 DEBUG:boto:Params: {} 2023-05-31T20:20:22.480 DEBUG:boto:Token: None 2023-05-31T20:20:22.481 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:20:22 GMT /pklxkd-49/ 2023-05-31T20:20:22.481 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:lW1cZMSyiySZM1lccSseC4NYx4k= 2023-05-31T20:20:22.482 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:20:22 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:lW1cZMSyiySZM1lccSseC4NYx4k=', 'Content-Length': '0'} 2023-05-31T20:20:22.484 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000005f68f76e4af679e7-006477ac06-162e-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:20:22 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:20:22.484 DEBUG:boto:b'pklxkd-49obj21000falseobj22023-05-31T20:18:05.430Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:20:22.485 DEBUG:boto:path=/pklxkd-49/ 2023-05-31T20:20:22.485 DEBUG:boto:auth_path=/pklxkd-49/ 2023-05-31T20:20:22.486 DEBUG:boto:path=/pklxkd-49/?prefix=obj2 2023-05-31T20:20:22.486 DEBUG:boto:auth_path=/pklxkd-49/?prefix=obj2 2023-05-31T20:20:22.486 DEBUG:boto:Method: GET 2023-05-31T20:20:22.487 DEBUG:boto:Path: /pklxkd-49/?prefix=obj2 2023-05-31T20:20:22.487 DEBUG:boto:Data: 2023-05-31T20:20:22.487 DEBUG:boto:Headers: {} 2023-05-31T20:20:22.488 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:20:22.488 DEBUG:boto:Port: 8000 2023-05-31T20:20:22.489 DEBUG:boto:Params: {} 2023-05-31T20:20:22.489 DEBUG:boto:Token: None 2023-05-31T20:20:22.489 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:20:22 GMT /pklxkd-49/ 2023-05-31T20:20:22.490 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:lW1cZMSyiySZM1lccSseC4NYx4k= 2023-05-31T20:20:22.490 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:20:22 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:lW1cZMSyiySZM1lccSseC4NYx4k=', 'Content-Length': '0'} 2023-05-31T20:20:22.492 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000dde7362975905899-006477ac06-1630-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:20:22 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:20:22.493 DEBUG:boto:b'pklxkd-49obj21000falseobj22023-05-31T20:18:05.430Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:20:22.494 DEBUG:boto:path=/pklxkd-49/obj2 2023-05-31T20:20:22.494 DEBUG:boto:auth_path=/pklxkd-49/obj2 2023-05-31T20:20:22.494 DEBUG:boto:Method: HEAD 2023-05-31T20:20:22.495 DEBUG:boto:Path: /pklxkd-49/obj2 2023-05-31T20:20:22.495 DEBUG:boto:Data: 2023-05-31T20:20:22.495 DEBUG:boto:Headers: {} 2023-05-31T20:20:22.496 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:20:22.496 DEBUG:boto:Port: 8001 2023-05-31T20:20:22.496 DEBUG:boto:Params: {} 2023-05-31T20:20:22.497 DEBUG:boto:Token: None 2023-05-31T20:20:22.497 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:20:22 GMT /pklxkd-49/obj2 2023-05-31T20:20:22.498 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:9QMoOIFGYaMD7shs6cVJ90aa+v8= 2023-05-31T20:20:22.498 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:20:22 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:9QMoOIFGYaMD7shs6cVJ90aa+v8=', 'Content-Length': '0'} 2023-05-31T20:20:22.500 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:18:05 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000851ac9c92f02832c-006477ac06-162e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:20:22 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:20:22.500 DEBUG:boto:path=/pklxkd-49/obj2 2023-05-31T20:20:22.501 DEBUG:boto:auth_path=/pklxkd-49/obj2 2023-05-31T20:20:22.501 DEBUG:boto:Method: HEAD 2023-05-31T20:20:22.501 DEBUG:boto:Path: /pklxkd-49/obj2 2023-05-31T20:20:22.502 DEBUG:boto:Data: 2023-05-31T20:20:22.502 DEBUG:boto:Headers: {} 2023-05-31T20:20:22.502 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:20:22.503 DEBUG:boto:Port: 8000 2023-05-31T20:20:22.503 DEBUG:boto:Params: {} 2023-05-31T20:20:22.503 DEBUG:boto:Token: None 2023-05-31T20:20:22.504 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:20:22 GMT /pklxkd-49/obj2 2023-05-31T20:20:22.504 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:9QMoOIFGYaMD7shs6cVJ90aa+v8= 2023-05-31T20:20:22.505 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:20:22 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:9QMoOIFGYaMD7shs6cVJ90aa+v8=', 'Content-Length': '0'} 2023-05-31T20:20:22.506 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:18:05 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000ca2dbf345a14828c-006477ac06-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:20:22 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:20:22.507 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2023-05-31T20:20:22.507 DEBUG:boto:path=/pklxkd-49/obj2 2023-05-31T20:20:22.508 DEBUG:boto:auth_path=/pklxkd-49/obj2 2023-05-31T20:20:22.508 DEBUG:boto:Method: GET 2023-05-31T20:20:22.508 DEBUG:boto:Path: /pklxkd-49/obj2 2023-05-31T20:20:22.509 DEBUG:boto:Data: 2023-05-31T20:20:22.509 DEBUG:boto:Headers: {} 2023-05-31T20:20:22.509 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:20:22.510 DEBUG:boto:Port: 8001 2023-05-31T20:20:22.510 DEBUG:boto:Params: {} 2023-05-31T20:20:22.511 DEBUG:boto:Token: None 2023-05-31T20:20:22.511 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:20:22 GMT /pklxkd-49/obj2 2023-05-31T20:20:22.512 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:Cu5J7fYNkDcXj0q7FcDXz+RRkK8= 2023-05-31T20:20:22.512 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:20:22 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:Cu5J7fYNkDcXj0q7FcDXz+RRkK8=', 'Content-Length': '0'} 2023-05-31T20:20:22.514 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:18:05 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000b57018ef4f2c8b07-006477ac06-162e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:20:22 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:20:22.514 DEBUG:boto:path=/pklxkd-49/obj2 2023-05-31T20:20:22.515 DEBUG:boto:auth_path=/pklxkd-49/obj2 2023-05-31T20:20:22.515 DEBUG:boto:Method: GET 2023-05-31T20:20:22.515 DEBUG:boto:Path: /pklxkd-49/obj2 2023-05-31T20:20:22.516 DEBUG:boto:Data: 2023-05-31T20:20:22.516 DEBUG:boto:Headers: {} 2023-05-31T20:20:22.516 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:20:22.517 DEBUG:boto:Port: 8000 2023-05-31T20:20:22.517 DEBUG:boto:Params: {} 2023-05-31T20:20:22.518 DEBUG:boto:Token: None 2023-05-31T20:20:22.518 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:20:22 GMT /pklxkd-49/obj2 2023-05-31T20:20:22.518 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:Cu5J7fYNkDcXj0q7FcDXz+RRkK8= 2023-05-31T20:20:22.519 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:20:22 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:Cu5J7fYNkDcXj0q7FcDXz+RRkK8=', 'Content-Length': '0'} 2023-05-31T20:20:22.520 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:18:05 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000aee5e8063432e452-006477ac06-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:20:22 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:20:22.568 DEBUG:boto:path=/pklxkd-49/ 2023-05-31T20:20:22.569 DEBUG:boto:auth_path=/pklxkd-49/ 2023-05-31T20:20:22.569 DEBUG:boto:path=/pklxkd-49/?prefix=obj2 2023-05-31T20:20:22.570 DEBUG:boto:auth_path=/pklxkd-49/?prefix=obj2 2023-05-31T20:20:22.570 DEBUG:boto:Method: GET 2023-05-31T20:20:22.570 DEBUG:boto:Path: /pklxkd-49/?prefix=obj2 2023-05-31T20:20:22.571 DEBUG:boto:Data: 2023-05-31T20:20:22.571 DEBUG:boto:Headers: {} 2023-05-31T20:20:22.571 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:20:22.572 DEBUG:boto:Port: 8001 2023-05-31T20:20:22.572 DEBUG:boto:Params: {} 2023-05-31T20:20:22.573 DEBUG:boto:Token: None 2023-05-31T20:20:22.573 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:20:22 GMT /pklxkd-49/ 2023-05-31T20:20:22.573 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:lW1cZMSyiySZM1lccSseC4NYx4k= 2023-05-31T20:20:22.574 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:20:22 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:lW1cZMSyiySZM1lccSseC4NYx4k=', 'Content-Length': '0'} 2023-05-31T20:20:22.576 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ef53a11ef9a33795-006477ac06-162e-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:20:22 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:20:22.577 DEBUG:boto:b'pklxkd-49obj21000falseobj22023-05-31T20:18:05.430Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:20:22.577 DEBUG:boto:path=/pklxkd-49/ 2023-05-31T20:20:22.578 DEBUG:boto:auth_path=/pklxkd-49/ 2023-05-31T20:20:22.578 DEBUG:boto:path=/pklxkd-49/?prefix=obj2 2023-05-31T20:20:22.579 DEBUG:boto:auth_path=/pklxkd-49/?prefix=obj2 2023-05-31T20:20:22.579 DEBUG:boto:Method: GET 2023-05-31T20:20:22.579 DEBUG:boto:Path: /pklxkd-49/?prefix=obj2 2023-05-31T20:20:22.580 DEBUG:boto:Data: 2023-05-31T20:20:22.580 DEBUG:boto:Headers: {} 2023-05-31T20:20:22.580 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:20:22.581 DEBUG:boto:Port: 8000 2023-05-31T20:20:22.581 DEBUG:boto:Params: {} 2023-05-31T20:20:22.582 DEBUG:boto:Token: None 2023-05-31T20:20:22.582 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:20:22 GMT /pklxkd-49/ 2023-05-31T20:20:22.582 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:lW1cZMSyiySZM1lccSseC4NYx4k= 2023-05-31T20:20:22.583 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:20:22 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:lW1cZMSyiySZM1lccSseC4NYx4k=', 'Content-Length': '0'} 2023-05-31T20:20:22.585 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e51074d922908ed2-006477ac06-1630-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:20:22 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:20:22.586 DEBUG:boto:b'pklxkd-49obj21000falseobj22023-05-31T20:18:05.430Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:20:22.586 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2023-05-31T20:20:22.587 DEBUG:boto:path=/pklxkd-49/obj3 2023-05-31T20:20:22.587 DEBUG:boto:auth_path=/pklxkd-49/obj3 2023-05-31T20:20:22.587 DEBUG:boto:path=/pklxkd-49/obj3?versionId=null 2023-05-31T20:20:22.588 DEBUG:boto:auth_path=/pklxkd-49/obj3?versionId=null 2023-05-31T20:20:22.588 DEBUG:boto:Method: GET 2023-05-31T20:20:22.592 DEBUG:boto:Path: /pklxkd-49/obj3?versionId=null 2023-05-31T20:20:22.592 DEBUG:boto:Data: 2023-05-31T20:20:22.593 DEBUG:boto:Headers: {} 2023-05-31T20:20:22.593 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:20:22.593 DEBUG:boto:Port: 8001 2023-05-31T20:20:22.594 DEBUG:boto:Params: {} 2023-05-31T20:20:22.594 DEBUG:boto:Token: None 2023-05-31T20:20:22.595 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:20:22 GMT /pklxkd-49/obj3?versionId=null 2023-05-31T20:20:22.595 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:5hiwS1dIoDk0RlHGonlUjNeRtiY= 2023-05-31T20:20:22.596 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:20:22 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:5hiwS1dIoDk0RlHGonlUjNeRtiY=', 'Content-Length': '0'} 2023-05-31T20:20:22.598 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:18:05 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000b9913f1977491567-006477ac06-162e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:20:22 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:20:22.599 DEBUG:boto:path=/pklxkd-49/obj3 2023-05-31T20:20:22.600 DEBUG:boto:auth_path=/pklxkd-49/obj3 2023-05-31T20:20:22.600 DEBUG:boto:path=/pklxkd-49/obj3?versionId=null 2023-05-31T20:20:22.601 DEBUG:boto:auth_path=/pklxkd-49/obj3?versionId=null 2023-05-31T20:20:22.601 DEBUG:boto:Method: GET 2023-05-31T20:20:22.602 DEBUG:boto:Path: /pklxkd-49/obj3?versionId=null 2023-05-31T20:20:22.602 DEBUG:boto:Data: 2023-05-31T20:20:22.603 DEBUG:boto:Headers: {} 2023-05-31T20:20:22.604 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:20:22.605 DEBUG:boto:Port: 8000 2023-05-31T20:20:22.606 DEBUG:boto:Params: {} 2023-05-31T20:20:22.607 DEBUG:boto:Token: None 2023-05-31T20:20:22.608 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:20:22 GMT /pklxkd-49/obj3?versionId=null 2023-05-31T20:20:22.609 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:5hiwS1dIoDk0RlHGonlUjNeRtiY= 2023-05-31T20:20:22.610 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:20:22 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:5hiwS1dIoDk0RlHGonlUjNeRtiY=', 'Content-Length': '0'} 2023-05-31T20:20:22.612 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:18:05 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000002a3579ab90b89caa-006477ac06-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:20:22 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:20:22.613 DEBUG:boto:path=/pklxkd-49/obj3 2023-05-31T20:20:22.614 DEBUG:boto:auth_path=/pklxkd-49/obj3 2023-05-31T20:20:22.615 DEBUG:boto:path=/pklxkd-49/obj3?versionId=null 2023-05-31T20:20:22.615 DEBUG:boto:auth_path=/pklxkd-49/obj3?versionId=null 2023-05-31T20:20:22.616 DEBUG:boto:Method: HEAD 2023-05-31T20:20:22.617 DEBUG:boto:Path: /pklxkd-49/obj3?versionId=null 2023-05-31T20:20:22.617 DEBUG:boto:Data: 2023-05-31T20:20:22.618 DEBUG:boto:Headers: {} 2023-05-31T20:20:22.618 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:20:22.618 DEBUG:boto:Port: 8001 2023-05-31T20:20:22.620 DEBUG:boto:Params: {} 2023-05-31T20:20:22.621 DEBUG:boto:Token: None 2023-05-31T20:20:22.624 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:20:22 GMT /pklxkd-49/obj3?versionId=null 2023-05-31T20:20:22.625 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:DWqRUM3ZDfg4GmTYz8oD9J9Bubs= 2023-05-31T20:20:22.626 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:20:22 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:DWqRUM3ZDfg4GmTYz8oD9J9Bubs=', 'Content-Length': '0'} 2023-05-31T20:20:22.628 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:18:05 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000019de6c554bbb6d66-006477ac06-162e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:20:22 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:20:22.629 DEBUG:boto:path=/pklxkd-49/obj3 2023-05-31T20:20:22.629 DEBUG:boto:auth_path=/pklxkd-49/obj3 2023-05-31T20:20:22.630 DEBUG:boto:path=/pklxkd-49/obj3?versionId=null 2023-05-31T20:20:22.630 DEBUG:boto:auth_path=/pklxkd-49/obj3?versionId=null 2023-05-31T20:20:22.630 DEBUG:boto:Method: HEAD 2023-05-31T20:20:22.631 DEBUG:boto:Path: /pklxkd-49/obj3?versionId=null 2023-05-31T20:20:22.632 DEBUG:boto:Data: 2023-05-31T20:20:22.632 DEBUG:boto:Headers: {} 2023-05-31T20:20:22.632 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:20:22.633 DEBUG:boto:Port: 8000 2023-05-31T20:20:22.633 DEBUG:boto:Params: {} 2023-05-31T20:20:22.634 DEBUG:boto:Token: None 2023-05-31T20:20:22.634 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:20:22 GMT /pklxkd-49/obj3?versionId=null 2023-05-31T20:20:22.635 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:DWqRUM3ZDfg4GmTYz8oD9J9Bubs= 2023-05-31T20:20:22.635 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:20:22 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:DWqRUM3ZDfg4GmTYz8oD9J9Bubs=', 'Content-Length': '0'} 2023-05-31T20:20:22.637 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:18:05 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000003993daaf4da99c5-006477ac06-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:20:22 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:20:22.638 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2023-05-31T20:20:22.638 DEBUG:boto:path=/pklxkd-49/obj3 2023-05-31T20:20:22.639 DEBUG:boto:auth_path=/pklxkd-49/obj3 2023-05-31T20:20:22.639 DEBUG:boto:path=/pklxkd-49/obj3?versionId=null 2023-05-31T20:20:22.640 DEBUG:boto:auth_path=/pklxkd-49/obj3?versionId=null 2023-05-31T20:20:22.640 DEBUG:boto:Method: GET 2023-05-31T20:20:22.641 DEBUG:boto:Path: /pklxkd-49/obj3?versionId=null 2023-05-31T20:20:22.641 DEBUG:boto:Data: 2023-05-31T20:20:22.641 DEBUG:boto:Headers: {} 2023-05-31T20:20:22.642 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:20:22.642 DEBUG:boto:Port: 8001 2023-05-31T20:20:22.642 DEBUG:boto:Params: {} 2023-05-31T20:20:22.643 DEBUG:boto:Token: None 2023-05-31T20:20:22.643 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:20:22 GMT /pklxkd-49/obj3?versionId=null 2023-05-31T20:20:22.644 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:5hiwS1dIoDk0RlHGonlUjNeRtiY= 2023-05-31T20:20:22.644 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:20:22 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:5hiwS1dIoDk0RlHGonlUjNeRtiY=', 'Content-Length': '0'} 2023-05-31T20:20:22.646 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:18:05 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000a6f826523f59c228-006477ac06-162e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:20:22 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:20:22.646 DEBUG:boto:path=/pklxkd-49/obj3 2023-05-31T20:20:22.647 DEBUG:boto:auth_path=/pklxkd-49/obj3 2023-05-31T20:20:22.647 DEBUG:boto:path=/pklxkd-49/obj3?versionId=null 2023-05-31T20:20:22.647 DEBUG:boto:auth_path=/pklxkd-49/obj3?versionId=null 2023-05-31T20:20:22.648 DEBUG:boto:Method: GET 2023-05-31T20:20:22.648 DEBUG:boto:Path: /pklxkd-49/obj3?versionId=null 2023-05-31T20:20:22.648 DEBUG:boto:Data: 2023-05-31T20:20:22.649 DEBUG:boto:Headers: {} 2023-05-31T20:20:22.649 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:20:22.650 DEBUG:boto:Port: 8000 2023-05-31T20:20:22.650 DEBUG:boto:Params: {} 2023-05-31T20:20:22.650 DEBUG:boto:Token: None 2023-05-31T20:20:22.651 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:20:22 GMT /pklxkd-49/obj3?versionId=null 2023-05-31T20:20:22.651 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:5hiwS1dIoDk0RlHGonlUjNeRtiY= 2023-05-31T20:20:22.652 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:20:22 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:5hiwS1dIoDk0RlHGonlUjNeRtiY=', 'Content-Length': '0'} 2023-05-31T20:20:22.654 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:18:05 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000daa9dba6a083640e-006477ac06-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:20:22 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:20:22.697 DEBUG:boto:path=/pklxkd-49/ 2023-05-31T20:20:22.698 DEBUG:boto:auth_path=/pklxkd-49/ 2023-05-31T20:20:22.698 DEBUG:boto:path=/pklxkd-49/?prefix=obj3 2023-05-31T20:20:22.698 DEBUG:boto:auth_path=/pklxkd-49/?prefix=obj3 2023-05-31T20:20:22.699 DEBUG:boto:Method: GET 2023-05-31T20:20:22.699 DEBUG:boto:Path: /pklxkd-49/?prefix=obj3 2023-05-31T20:20:22.700 DEBUG:boto:Data: 2023-05-31T20:20:22.700 DEBUG:boto:Headers: {} 2023-05-31T20:20:22.700 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:20:22.701 DEBUG:boto:Port: 8001 2023-05-31T20:20:22.701 DEBUG:boto:Params: {} 2023-05-31T20:20:22.702 DEBUG:boto:Token: None 2023-05-31T20:20:22.702 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:20:22 GMT /pklxkd-49/ 2023-05-31T20:20:22.703 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:lW1cZMSyiySZM1lccSseC4NYx4k= 2023-05-31T20:20:22.703 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:20:22 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:lW1cZMSyiySZM1lccSseC4NYx4k=', 'Content-Length': '0'} 2023-05-31T20:20:22.705 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000086aabbf2cb89c145-006477ac06-162e-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:20:22 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:20:22.706 DEBUG:boto:b'pklxkd-49obj31000falseobj32023-05-31T20:18:05.493Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:20:22.707 DEBUG:boto:path=/pklxkd-49/ 2023-05-31T20:20:22.707 DEBUG:boto:auth_path=/pklxkd-49/ 2023-05-31T20:20:22.707 DEBUG:boto:path=/pklxkd-49/?prefix=obj3 2023-05-31T20:20:22.708 DEBUG:boto:auth_path=/pklxkd-49/?prefix=obj3 2023-05-31T20:20:22.708 DEBUG:boto:Method: GET 2023-05-31T20:20:22.709 DEBUG:boto:Path: /pklxkd-49/?prefix=obj3 2023-05-31T20:20:22.709 DEBUG:boto:Data: 2023-05-31T20:20:22.709 DEBUG:boto:Headers: {} 2023-05-31T20:20:22.710 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:20:22.710 DEBUG:boto:Port: 8000 2023-05-31T20:20:22.710 DEBUG:boto:Params: {} 2023-05-31T20:20:22.711 DEBUG:boto:Token: None 2023-05-31T20:20:22.711 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:20:22 GMT /pklxkd-49/ 2023-05-31T20:20:22.712 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:lW1cZMSyiySZM1lccSseC4NYx4k= 2023-05-31T20:20:22.712 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:20:22 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:lW1cZMSyiySZM1lccSseC4NYx4k=', 'Content-Length': '0'} 2023-05-31T20:20:22.714 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d7fefdffebb2da0b-006477ac06-1630-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:20:22 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:20:22.715 DEBUG:boto:b'pklxkd-49obj31000falseobj32023-05-31T20:18:05.493Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:20:22.715 DEBUG:boto:path=/pklxkd-49/obj3 2023-05-31T20:20:22.716 DEBUG:boto:auth_path=/pklxkd-49/obj3 2023-05-31T20:20:22.717 DEBUG:boto:Method: HEAD 2023-05-31T20:20:22.717 DEBUG:boto:Path: /pklxkd-49/obj3 2023-05-31T20:20:22.718 DEBUG:boto:Data: 2023-05-31T20:20:22.718 DEBUG:boto:Headers: {} 2023-05-31T20:20:22.719 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:20:22.719 DEBUG:boto:Port: 8001 2023-05-31T20:20:22.719 DEBUG:boto:Params: {} 2023-05-31T20:20:22.720 DEBUG:boto:Token: None 2023-05-31T20:20:22.720 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:20:22 GMT /pklxkd-49/obj3 2023-05-31T20:20:22.721 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:I0LZzU/XqF7jXxtJt50C8K18Ea4= 2023-05-31T20:20:22.721 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:20:22 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:I0LZzU/XqF7jXxtJt50C8K18Ea4=', 'Content-Length': '0'} 2023-05-31T20:20:22.723 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:18:05 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000003b786eccd183ae8e-006477ac06-162e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:20:22 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:20:22.723 DEBUG:boto:path=/pklxkd-49/obj3 2023-05-31T20:20:22.724 DEBUG:boto:auth_path=/pklxkd-49/obj3 2023-05-31T20:20:22.724 DEBUG:boto:Method: HEAD 2023-05-31T20:20:22.724 DEBUG:boto:Path: /pklxkd-49/obj3 2023-05-31T20:20:22.725 DEBUG:boto:Data: 2023-05-31T20:20:22.725 DEBUG:boto:Headers: {} 2023-05-31T20:20:22.725 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:20:22.726 DEBUG:boto:Port: 8000 2023-05-31T20:20:22.726 DEBUG:boto:Params: {} 2023-05-31T20:20:22.726 DEBUG:boto:Token: None 2023-05-31T20:20:22.727 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:20:22 GMT /pklxkd-49/obj3 2023-05-31T20:20:22.727 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:I0LZzU/XqF7jXxtJt50C8K18Ea4= 2023-05-31T20:20:22.728 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:20:22 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:I0LZzU/XqF7jXxtJt50C8K18Ea4=', 'Content-Length': '0'} 2023-05-31T20:20:22.729 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:18:05 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000957338e7296f0c9c-006477ac06-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:20:22 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:20:22.730 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2023-05-31T20:20:22.730 DEBUG:boto:path=/pklxkd-49/obj3 2023-05-31T20:20:22.731 DEBUG:boto:auth_path=/pklxkd-49/obj3 2023-05-31T20:20:22.731 DEBUG:boto:Method: GET 2023-05-31T20:20:22.731 DEBUG:boto:Path: /pklxkd-49/obj3 2023-05-31T20:20:22.732 DEBUG:boto:Data: 2023-05-31T20:20:22.732 DEBUG:boto:Headers: {} 2023-05-31T20:20:22.733 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:20:22.733 DEBUG:boto:Port: 8001 2023-05-31T20:20:22.733 DEBUG:boto:Params: {} 2023-05-31T20:20:22.734 DEBUG:boto:Token: None 2023-05-31T20:20:22.734 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:20:22 GMT /pklxkd-49/obj3 2023-05-31T20:20:22.735 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:p0dyVDoxJ3O9T3AybIhebq+1BJs= 2023-05-31T20:20:22.735 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:20:22 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:p0dyVDoxJ3O9T3AybIhebq+1BJs=', 'Content-Length': '0'} 2023-05-31T20:20:22.736 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:18:05 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000bdcb4028de9856b8-006477ac06-162e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:20:22 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:20:22.780 DEBUG:boto:path=/pklxkd-49/obj3 2023-05-31T20:20:22.781 DEBUG:boto:auth_path=/pklxkd-49/obj3 2023-05-31T20:20:22.781 DEBUG:boto:Method: GET 2023-05-31T20:20:22.782 DEBUG:boto:Path: /pklxkd-49/obj3 2023-05-31T20:20:22.782 DEBUG:boto:Data: 2023-05-31T20:20:22.782 DEBUG:boto:Headers: {} 2023-05-31T20:20:22.783 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:20:22.783 DEBUG:boto:Port: 8000 2023-05-31T20:20:22.783 DEBUG:boto:Params: {} 2023-05-31T20:20:22.784 DEBUG:boto:Token: None 2023-05-31T20:20:22.784 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:20:22 GMT /pklxkd-49/obj3 2023-05-31T20:20:22.785 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:p0dyVDoxJ3O9T3AybIhebq+1BJs= 2023-05-31T20:20:22.785 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:20:22 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:p0dyVDoxJ3O9T3AybIhebq+1BJs=', 'Content-Length': '0'} 2023-05-31T20:20:22.787 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:18:05 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000006b59d88f01cb7cb1-006477ac06-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:20:22 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:20:22.788 DEBUG:boto:path=/pklxkd-49/ 2023-05-31T20:20:22.789 DEBUG:boto:auth_path=/pklxkd-49/ 2023-05-31T20:20:22.789 DEBUG:boto:path=/pklxkd-49/?prefix=obj3 2023-05-31T20:20:22.789 DEBUG:boto:auth_path=/pklxkd-49/?prefix=obj3 2023-05-31T20:20:22.790 DEBUG:boto:Method: GET 2023-05-31T20:20:22.790 DEBUG:boto:Path: /pklxkd-49/?prefix=obj3 2023-05-31T20:20:22.791 DEBUG:boto:Data: 2023-05-31T20:20:22.791 DEBUG:boto:Headers: {} 2023-05-31T20:20:22.791 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:20:22.792 DEBUG:boto:Port: 8001 2023-05-31T20:20:22.792 DEBUG:boto:Params: {} 2023-05-31T20:20:22.793 DEBUG:boto:Token: None 2023-05-31T20:20:22.793 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:20:22 GMT /pklxkd-49/ 2023-05-31T20:20:22.793 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:lW1cZMSyiySZM1lccSseC4NYx4k= 2023-05-31T20:20:22.794 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:20:22 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:lW1cZMSyiySZM1lccSseC4NYx4k=', 'Content-Length': '0'} 2023-05-31T20:20:22.796 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000507660a555adf978-006477ac06-162e-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:20:22 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:20:22.797 DEBUG:boto:b'pklxkd-49obj31000falseobj32023-05-31T20:18:05.493Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:20:22.801 DEBUG:boto:path=/pklxkd-49/ 2023-05-31T20:20:22.802 DEBUG:boto:auth_path=/pklxkd-49/ 2023-05-31T20:20:22.802 DEBUG:boto:path=/pklxkd-49/?prefix=obj3 2023-05-31T20:20:22.803 DEBUG:boto:auth_path=/pklxkd-49/?prefix=obj3 2023-05-31T20:20:22.803 DEBUG:boto:Method: GET 2023-05-31T20:20:22.803 DEBUG:boto:Path: /pklxkd-49/?prefix=obj3 2023-05-31T20:20:22.804 DEBUG:boto:Data: 2023-05-31T20:20:22.804 DEBUG:boto:Headers: {} 2023-05-31T20:20:22.805 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:20:22.805 DEBUG:boto:Port: 8000 2023-05-31T20:20:22.805 DEBUG:boto:Params: {} 2023-05-31T20:20:22.806 DEBUG:boto:Token: None 2023-05-31T20:20:22.806 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:20:22 GMT /pklxkd-49/ 2023-05-31T20:20:22.807 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:lW1cZMSyiySZM1lccSseC4NYx4k= 2023-05-31T20:20:22.809 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:20:22 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:lW1cZMSyiySZM1lccSseC4NYx4k=', 'Content-Length': '0'} 2023-05-31T20:20:22.811 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000072e2fdec5d8f29f3-006477ac06-1630-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:20:22 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:20:22.812 DEBUG:boto:b'pklxkd-49obj31000falseobj32023-05-31T20:18:05.493Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:20:22.812 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2023-05-31T20:20:22.813 DEBUG:boto:path=/pklxkd-49/obj4 2023-05-31T20:20:22.813 DEBUG:boto:auth_path=/pklxkd-49/obj4 2023-05-31T20:20:22.814 DEBUG:boto:path=/pklxkd-49/obj4?versionId=null 2023-05-31T20:20:22.814 DEBUG:boto:auth_path=/pklxkd-49/obj4?versionId=null 2023-05-31T20:20:22.814 DEBUG:boto:Method: GET 2023-05-31T20:20:22.815 DEBUG:boto:Path: /pklxkd-49/obj4?versionId=null 2023-05-31T20:20:22.815 DEBUG:boto:Data: 2023-05-31T20:20:22.816 DEBUG:boto:Headers: {} 2023-05-31T20:20:22.816 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:20:22.816 DEBUG:boto:Port: 8001 2023-05-31T20:20:22.817 DEBUG:boto:Params: {} 2023-05-31T20:20:22.817 DEBUG:boto:Token: None 2023-05-31T20:20:22.818 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:20:22 GMT /pklxkd-49/obj4?versionId=null 2023-05-31T20:20:22.819 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:WoePDXrHN5KMrb4L1EVgKip5WIo= 2023-05-31T20:20:22.819 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:20:22 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:WoePDXrHN5KMrb4L1EVgKip5WIo=', 'Content-Length': '0'} 2023-05-31T20:20:22.821 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:18:05 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000001052759169ffc2f3-006477ac06-162e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:20:22 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:20:22.822 DEBUG:boto:path=/pklxkd-49/obj4 2023-05-31T20:20:22.823 DEBUG:boto:auth_path=/pklxkd-49/obj4 2023-05-31T20:20:22.823 DEBUG:boto:path=/pklxkd-49/obj4?versionId=null 2023-05-31T20:20:22.823 DEBUG:boto:auth_path=/pklxkd-49/obj4?versionId=null 2023-05-31T20:20:22.824 DEBUG:boto:Method: GET 2023-05-31T20:20:22.824 DEBUG:boto:Path: /pklxkd-49/obj4?versionId=null 2023-05-31T20:20:22.825 DEBUG:boto:Data: 2023-05-31T20:20:22.825 DEBUG:boto:Headers: {} 2023-05-31T20:20:22.826 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:20:22.826 DEBUG:boto:Port: 8000 2023-05-31T20:20:22.826 DEBUG:boto:Params: {} 2023-05-31T20:20:22.827 DEBUG:boto:Token: None 2023-05-31T20:20:22.827 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:20:22 GMT /pklxkd-49/obj4?versionId=null 2023-05-31T20:20:22.828 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:WoePDXrHN5KMrb4L1EVgKip5WIo= 2023-05-31T20:20:22.828 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:20:22 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:WoePDXrHN5KMrb4L1EVgKip5WIo=', 'Content-Length': '0'} 2023-05-31T20:20:22.831 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:18:05 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000089a51fdd490ba59c-006477ac06-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:20:22 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:20:22.831 DEBUG:boto:path=/pklxkd-49/obj4 2023-05-31T20:20:22.832 DEBUG:boto:auth_path=/pklxkd-49/obj4 2023-05-31T20:20:22.832 DEBUG:boto:path=/pklxkd-49/obj4?versionId=null 2023-05-31T20:20:22.832 DEBUG:boto:auth_path=/pklxkd-49/obj4?versionId=null 2023-05-31T20:20:22.833 DEBUG:boto:Method: HEAD 2023-05-31T20:20:22.833 DEBUG:boto:Path: /pklxkd-49/obj4?versionId=null 2023-05-31T20:20:22.834 DEBUG:boto:Data: 2023-05-31T20:20:22.834 DEBUG:boto:Headers: {} 2023-05-31T20:20:22.834 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:20:22.835 DEBUG:boto:Port: 8001 2023-05-31T20:20:22.835 DEBUG:boto:Params: {} 2023-05-31T20:20:22.835 DEBUG:boto:Token: None 2023-05-31T20:20:22.836 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:20:22 GMT /pklxkd-49/obj4?versionId=null 2023-05-31T20:20:22.836 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:usDNOOcAPdLJU7Xh/eMxoxvQ+Fk= 2023-05-31T20:20:22.837 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:20:22 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:usDNOOcAPdLJU7Xh/eMxoxvQ+Fk=', 'Content-Length': '0'} 2023-05-31T20:20:22.839 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:18:05 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000b4aac1769a5e5642-006477ac06-162e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:20:22 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:20:22.839 DEBUG:boto:path=/pklxkd-49/obj4 2023-05-31T20:20:22.840 DEBUG:boto:auth_path=/pklxkd-49/obj4 2023-05-31T20:20:22.840 DEBUG:boto:path=/pklxkd-49/obj4?versionId=null 2023-05-31T20:20:22.841 DEBUG:boto:auth_path=/pklxkd-49/obj4?versionId=null 2023-05-31T20:20:22.841 DEBUG:boto:Method: HEAD 2023-05-31T20:20:22.841 DEBUG:boto:Path: /pklxkd-49/obj4?versionId=null 2023-05-31T20:20:22.842 DEBUG:boto:Data: 2023-05-31T20:20:22.842 DEBUG:boto:Headers: {} 2023-05-31T20:20:22.842 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:20:22.843 DEBUG:boto:Port: 8000 2023-05-31T20:20:22.843 DEBUG:boto:Params: {} 2023-05-31T20:20:22.843 DEBUG:boto:Token: None 2023-05-31T20:20:22.844 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:20:22 GMT /pklxkd-49/obj4?versionId=null 2023-05-31T20:20:22.844 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:usDNOOcAPdLJU7Xh/eMxoxvQ+Fk= 2023-05-31T20:20:22.845 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:20:22 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:usDNOOcAPdLJU7Xh/eMxoxvQ+Fk=', 'Content-Length': '0'} 2023-05-31T20:20:22.846 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:18:05 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000469158a5c6619f51-006477ac06-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:20:22 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:20:22.847 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2023-05-31T20:20:22.848 DEBUG:boto:path=/pklxkd-49/obj4 2023-05-31T20:20:22.848 DEBUG:boto:auth_path=/pklxkd-49/obj4 2023-05-31T20:20:22.849 DEBUG:boto:path=/pklxkd-49/obj4?versionId=null 2023-05-31T20:20:22.850 DEBUG:boto:auth_path=/pklxkd-49/obj4?versionId=null 2023-05-31T20:20:22.850 DEBUG:boto:Method: GET 2023-05-31T20:20:22.850 DEBUG:boto:Path: /pklxkd-49/obj4?versionId=null 2023-05-31T20:20:22.851 DEBUG:boto:Data: 2023-05-31T20:20:22.851 DEBUG:boto:Headers: {} 2023-05-31T20:20:22.851 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:20:22.852 DEBUG:boto:Port: 8001 2023-05-31T20:20:22.852 DEBUG:boto:Params: {} 2023-05-31T20:20:22.853 DEBUG:boto:Token: None 2023-05-31T20:20:22.853 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:20:22 GMT /pklxkd-49/obj4?versionId=null 2023-05-31T20:20:22.853 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:WoePDXrHN5KMrb4L1EVgKip5WIo= 2023-05-31T20:20:22.854 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:20:22 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:WoePDXrHN5KMrb4L1EVgKip5WIo=', 'Content-Length': '0'} 2023-05-31T20:20:22.856 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:18:05 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000d664af08feafa000-006477ac06-162e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:20:22 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:20:22.857 DEBUG:boto:path=/pklxkd-49/obj4 2023-05-31T20:20:22.858 DEBUG:boto:auth_path=/pklxkd-49/obj4 2023-05-31T20:20:22.858 DEBUG:boto:path=/pklxkd-49/obj4?versionId=null 2023-05-31T20:20:22.859 DEBUG:boto:auth_path=/pklxkd-49/obj4?versionId=null 2023-05-31T20:20:22.859 DEBUG:boto:Method: GET 2023-05-31T20:20:22.859 DEBUG:boto:Path: /pklxkd-49/obj4?versionId=null 2023-05-31T20:20:22.860 DEBUG:boto:Data: 2023-05-31T20:20:22.860 DEBUG:boto:Headers: {} 2023-05-31T20:20:22.861 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:20:22.861 DEBUG:boto:Port: 8000 2023-05-31T20:20:22.861 DEBUG:boto:Params: {} 2023-05-31T20:20:22.862 DEBUG:boto:Token: None 2023-05-31T20:20:22.862 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:20:22 GMT /pklxkd-49/obj4?versionId=null 2023-05-31T20:20:22.863 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:WoePDXrHN5KMrb4L1EVgKip5WIo= 2023-05-31T20:20:22.863 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:20:22 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:WoePDXrHN5KMrb4L1EVgKip5WIo=', 'Content-Length': '0'} 2023-05-31T20:20:22.865 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:18:05 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000005eac8c65a9ba2f3-006477ac06-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:20:22 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:20:22.865 DEBUG:boto:path=/pklxkd-49/ 2023-05-31T20:20:22.866 DEBUG:boto:auth_path=/pklxkd-49/ 2023-05-31T20:20:22.867 DEBUG:boto:path=/pklxkd-49/?prefix=obj4 2023-05-31T20:20:22.867 DEBUG:boto:auth_path=/pklxkd-49/?prefix=obj4 2023-05-31T20:20:22.867 DEBUG:boto:Method: GET 2023-05-31T20:20:22.868 DEBUG:boto:Path: /pklxkd-49/?prefix=obj4 2023-05-31T20:20:22.868 DEBUG:boto:Data: 2023-05-31T20:20:22.868 DEBUG:boto:Headers: {} 2023-05-31T20:20:22.869 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:20:22.869 DEBUG:boto:Port: 8001 2023-05-31T20:20:22.870 DEBUG:boto:Params: {} 2023-05-31T20:20:22.870 DEBUG:boto:Token: None 2023-05-31T20:20:22.871 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:20:22 GMT /pklxkd-49/ 2023-05-31T20:20:22.871 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:lW1cZMSyiySZM1lccSseC4NYx4k= 2023-05-31T20:20:22.874 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:20:22 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:lW1cZMSyiySZM1lccSseC4NYx4k=', 'Content-Length': '0'} 2023-05-31T20:20:22.877 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000004e42d93ed47f6015-006477ac06-162e-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:20:22 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:20:22.878 DEBUG:boto:b'pklxkd-49obj41000falseobj42023-05-31T20:18:05.509Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:20:22.879 DEBUG:boto:path=/pklxkd-49/ 2023-05-31T20:20:22.879 DEBUG:boto:auth_path=/pklxkd-49/ 2023-05-31T20:20:22.879 DEBUG:boto:path=/pklxkd-49/?prefix=obj4 2023-05-31T20:20:22.880 DEBUG:boto:auth_path=/pklxkd-49/?prefix=obj4 2023-05-31T20:20:22.880 DEBUG:boto:Method: GET 2023-05-31T20:20:22.881 DEBUG:boto:Path: /pklxkd-49/?prefix=obj4 2023-05-31T20:20:22.881 DEBUG:boto:Data: 2023-05-31T20:20:22.882 DEBUG:boto:Headers: {} 2023-05-31T20:20:22.882 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:20:22.883 DEBUG:boto:Port: 8000 2023-05-31T20:20:22.883 DEBUG:boto:Params: {} 2023-05-31T20:20:22.884 DEBUG:boto:Token: None 2023-05-31T20:20:22.885 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:20:22 GMT /pklxkd-49/ 2023-05-31T20:20:22.886 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:lW1cZMSyiySZM1lccSseC4NYx4k= 2023-05-31T20:20:22.886 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:20:22 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:lW1cZMSyiySZM1lccSseC4NYx4k=', 'Content-Length': '0'} 2023-05-31T20:20:22.888 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000009a66a0b1a5e7f35f-006477ac06-1630-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:20:22 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:20:22.889 DEBUG:boto:b'pklxkd-49obj41000falseobj42023-05-31T20:18:05.509Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:20:22.889 DEBUG:boto:path=/pklxkd-49/obj4 2023-05-31T20:20:22.890 DEBUG:boto:auth_path=/pklxkd-49/obj4 2023-05-31T20:20:22.893 DEBUG:boto:Method: HEAD 2023-05-31T20:20:22.894 DEBUG:boto:Path: /pklxkd-49/obj4 2023-05-31T20:20:22.895 DEBUG:boto:Data: 2023-05-31T20:20:22.895 DEBUG:boto:Headers: {} 2023-05-31T20:20:22.896 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:20:22.897 DEBUG:boto:Port: 8001 2023-05-31T20:20:22.898 DEBUG:boto:Params: {} 2023-05-31T20:20:22.898 DEBUG:boto:Token: None 2023-05-31T20:20:22.899 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:20:22 GMT /pklxkd-49/obj4 2023-05-31T20:20:22.900 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:jWBC0QhVRaO17Z6DhKOy/k1oROU= 2023-05-31T20:20:22.901 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:20:22 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:jWBC0QhVRaO17Z6DhKOy/k1oROU=', 'Content-Length': '0'} 2023-05-31T20:20:22.903 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:18:05 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000935aece29f82c3a9-006477ac06-162e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:20:22 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:20:22.904 DEBUG:boto:path=/pklxkd-49/obj4 2023-05-31T20:20:22.904 DEBUG:boto:auth_path=/pklxkd-49/obj4 2023-05-31T20:20:22.905 DEBUG:boto:Method: HEAD 2023-05-31T20:20:22.905 DEBUG:boto:Path: /pklxkd-49/obj4 2023-05-31T20:20:22.905 DEBUG:boto:Data: 2023-05-31T20:20:22.906 DEBUG:boto:Headers: {} 2023-05-31T20:20:22.906 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:20:22.906 DEBUG:boto:Port: 8000 2023-05-31T20:20:22.907 DEBUG:boto:Params: {} 2023-05-31T20:20:22.907 DEBUG:boto:Token: None 2023-05-31T20:20:22.907 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:20:22 GMT /pklxkd-49/obj4 2023-05-31T20:20:22.908 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:jWBC0QhVRaO17Z6DhKOy/k1oROU= 2023-05-31T20:20:22.908 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:20:22 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:jWBC0QhVRaO17Z6DhKOy/k1oROU=', 'Content-Length': '0'} 2023-05-31T20:20:22.910 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:18:05 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000009d599127d12dda81-006477ac06-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:20:22 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:20:22.910 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2023-05-31T20:20:22.911 DEBUG:boto:path=/pklxkd-49/obj4 2023-05-31T20:20:22.911 DEBUG:boto:auth_path=/pklxkd-49/obj4 2023-05-31T20:20:22.912 DEBUG:boto:Method: GET 2023-05-31T20:20:22.912 DEBUG:boto:Path: /pklxkd-49/obj4 2023-05-31T20:20:22.912 DEBUG:boto:Data: 2023-05-31T20:20:22.913 DEBUG:boto:Headers: {} 2023-05-31T20:20:22.914 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:20:22.916 DEBUG:boto:Port: 8001 2023-05-31T20:20:22.916 DEBUG:boto:Params: {} 2023-05-31T20:20:22.917 DEBUG:boto:Token: None 2023-05-31T20:20:22.917 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:20:22 GMT /pklxkd-49/obj4 2023-05-31T20:20:22.918 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:/032aWimFrV9AZTry13BSX6542A= 2023-05-31T20:20:22.918 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:20:22 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:/032aWimFrV9AZTry13BSX6542A=', 'Content-Length': '0'} 2023-05-31T20:20:22.920 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:18:05 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000050d99c62b9eef57f-006477ac06-162e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:20:22 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:20:22.960 DEBUG:boto:path=/pklxkd-49/obj4 2023-05-31T20:20:22.961 DEBUG:boto:auth_path=/pklxkd-49/obj4 2023-05-31T20:20:22.962 DEBUG:boto:Method: GET 2023-05-31T20:20:22.962 DEBUG:boto:Path: /pklxkd-49/obj4 2023-05-31T20:20:22.962 DEBUG:boto:Data: 2023-05-31T20:20:22.963 DEBUG:boto:Headers: {} 2023-05-31T20:20:22.963 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:20:22.964 DEBUG:boto:Port: 8000 2023-05-31T20:20:22.964 DEBUG:boto:Params: {} 2023-05-31T20:20:22.965 DEBUG:boto:Token: None 2023-05-31T20:20:22.965 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:20:22 GMT /pklxkd-49/obj4 2023-05-31T20:20:22.966 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:/032aWimFrV9AZTry13BSX6542A= 2023-05-31T20:20:22.966 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:20:22 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:/032aWimFrV9AZTry13BSX6542A=', 'Content-Length': '0'} 2023-05-31T20:20:22.968 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:18:05 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000dc0cb1029a432612-006477ac06-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:20:22 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:20:22.969 DEBUG:boto:path=/pklxkd-49/ 2023-05-31T20:20:22.969 DEBUG:boto:auth_path=/pklxkd-49/ 2023-05-31T20:20:22.969 DEBUG:boto:path=/pklxkd-49/?prefix=obj4 2023-05-31T20:20:22.970 DEBUG:boto:auth_path=/pklxkd-49/?prefix=obj4 2023-05-31T20:20:22.970 DEBUG:boto:Method: GET 2023-05-31T20:20:22.970 DEBUG:boto:Path: /pklxkd-49/?prefix=obj4 2023-05-31T20:20:22.971 DEBUG:boto:Data: 2023-05-31T20:20:22.972 DEBUG:boto:Headers: {} 2023-05-31T20:20:22.972 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:20:22.972 DEBUG:boto:Port: 8001 2023-05-31T20:20:22.973 DEBUG:boto:Params: {} 2023-05-31T20:20:22.973 DEBUG:boto:Token: None 2023-05-31T20:20:22.974 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:20:22 GMT /pklxkd-49/ 2023-05-31T20:20:22.974 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:lW1cZMSyiySZM1lccSseC4NYx4k= 2023-05-31T20:20:22.974 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:20:22 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:lW1cZMSyiySZM1lccSseC4NYx4k=', 'Content-Length': '0'} 2023-05-31T20:20:22.977 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000008d0f7c7ab32f0724-006477ac06-162e-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:20:22 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:20:22.977 DEBUG:boto:b'pklxkd-49obj41000falseobj42023-05-31T20:18:05.509Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:20:22.978 DEBUG:boto:path=/pklxkd-49/ 2023-05-31T20:20:22.978 DEBUG:boto:auth_path=/pklxkd-49/ 2023-05-31T20:20:22.979 DEBUG:boto:path=/pklxkd-49/?prefix=obj4 2023-05-31T20:20:22.979 DEBUG:boto:auth_path=/pklxkd-49/?prefix=obj4 2023-05-31T20:20:22.980 DEBUG:boto:Method: GET 2023-05-31T20:20:22.980 DEBUG:boto:Path: /pklxkd-49/?prefix=obj4 2023-05-31T20:20:22.980 DEBUG:boto:Data: 2023-05-31T20:20:22.981 DEBUG:boto:Headers: {} 2023-05-31T20:20:22.981 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:20:22.982 DEBUG:boto:Port: 8000 2023-05-31T20:20:22.982 DEBUG:boto:Params: {} 2023-05-31T20:20:22.983 DEBUG:boto:Token: None 2023-05-31T20:20:22.985 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:20:22 GMT /pklxkd-49/ 2023-05-31T20:20:22.985 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:lW1cZMSyiySZM1lccSseC4NYx4k= 2023-05-31T20:20:22.986 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:20:22 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:lW1cZMSyiySZM1lccSseC4NYx4k=', 'Content-Length': '0'} 2023-05-31T20:20:22.988 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d7ad41de00cd11e3-006477ac06-1630-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:20:22 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:20:22.989 DEBUG:boto:b'pklxkd-49obj41000falseobj42023-05-31T20:18:05.509Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:20:22.989 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=pklxkd-49 zones={a2, a1} 2023-05-31T20:20:22.990 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'disable', '--bucket', 'pklxkd-48', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-05-31T20:20:22.990 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'disable', '--bucket', 'pklxkd-48', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-05-31T20:20:22.991 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync disable --bucket pklxkd-48 --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 2023-05-31T20:20:23.044 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2023-05-31T20:20:23.044 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2023-05-31T20:20:23.118 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'disable', '--bucket', 'pklxkd-49', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-05-31T20:20:23.119 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'disable', '--bucket', 'pklxkd-49', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-05-31T20:20:23.121 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync disable --bucket pklxkd-49 --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 2023-05-31T20:20:23.174 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2023-05-31T20:20:23.174 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2023-05-31T20:20:23.280 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-31T20:20:23.281 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-31T20:20:23.281 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-05-31T20:20:23.329 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2023-05-31T20:20:23.330 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2023-05-31T20:20:23.363 INFO:teuthology.orchestra.run.smithi099.stderr:No --period given, using current period=2e6123d2-d695-4350-9a01-61a455bc782b 2023-05-31T20:20:23.380 INFO:teuthology.orchestra.run.smithi099.stdout:[ 2023-05-31T20:20:23.380 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T20:20:23.381 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1685562631.516089_59.1", 2023-05-31T20:20:23.381 INFO:teuthology.orchestra.run.smithi099.stdout: "last_update": "2023-05-31T19:50:31.516089Z" 2023-05-31T20:20:23.381 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:20:23.381 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T20:20:23.382 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "", 2023-05-31T20:20:23.382 INFO:teuthology.orchestra.run.smithi099.stdout: "last_update": "0.000000" 2023-05-31T20:20:23.382 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:20:23.382 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T20:20:23.383 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1685562970.274572_93.1", 2023-05-31T20:20:23.383 INFO:teuthology.orchestra.run.smithi099.stdout: "last_update": "2023-05-31T19:56:10.274572Z" 2023-05-31T20:20:23.383 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:20:23.384 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T20:20:23.384 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1685564423.262837_330.1", 2023-05-31T20:20:23.384 INFO:teuthology.orchestra.run.smithi099.stdout: "last_update": "2023-05-31T20:20:23.262837Z" 2023-05-31T20:20:23.385 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T20:20:23.385 INFO:teuthology.orchestra.run.smithi099.stdout:] 2023-05-31T20:20:23.386 DEBUG:rgw_multi.tests:master meta markers={0: '1_1685562631.516089_59.1', 1: '', 2: '1_1685562970.274572_93.1', 3: '1_1685564423.262837_330.1'} 2023-05-31T20:20:23.387 INFO:rgw_multi.tests:starting meta checkpoint for zone=a2 2023-05-31T20:20:23.388 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-31T20:20:23.388 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-31T20:20:23.389 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-05-31T20:20:23.439 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2023-05-31T20:20:23.439 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2023-05-31T20:20:23.488 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:20:23.484+0000 7f6f8297b780 20 RGW-SYNC:meta: read sync status 2023-05-31T20:20:23.490 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:20:23.485+0000 7f6f8297b780 20 RGW-SYNC:meta: read sync status 2023-05-31T20:20:23.507 INFO:teuthology.orchestra.run.smithi099.stdout:{ 2023-05-31T20:20:23.508 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_status": { 2023-05-31T20:20:23.508 INFO:teuthology.orchestra.run.smithi099.stdout: "info": { 2023-05-31T20:20:23.508 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "sync", 2023-05-31T20:20:23.508 INFO:teuthology.orchestra.run.smithi099.stdout: "num_shards": 4, 2023-05-31T20:20:23.509 INFO:teuthology.orchestra.run.smithi099.stdout: "period": "2e6123d2-d695-4350-9a01-61a455bc782b", 2023-05-31T20:20:23.509 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2023-05-31T20:20:23.509 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:20:23.509 INFO:teuthology.orchestra.run.smithi099.stdout: "markers": [ 2023-05-31T20:20:23.510 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T20:20:23.510 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 0, 2023-05-31T20:20:23.510 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T20:20:23.510 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2023-05-31T20:20:23.510 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1685562631.516089_59.1", 2023-05-31T20:20:23.510 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T20:20:23.511 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 1, 2023-05-31T20:20:23.511 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T20:20:23.511 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2023-05-31T19:50:31.516089Z", 2023-05-31T20:20:23.511 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2023-05-31T20:20:23.511 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T20:20:23.511 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:20:23.512 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T20:20:23.512 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 1, 2023-05-31T20:20:23.512 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T20:20:23.512 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2023-05-31T20:20:23.512 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "", 2023-05-31T20:20:23.512 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T20:20:23.513 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2023-05-31T20:20:23.513 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T20:20:23.513 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "0.000000", 2023-05-31T20:20:23.513 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2023-05-31T20:20:23.513 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T20:20:23.513 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:20:23.513 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T20:20:23.514 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 2, 2023-05-31T20:20:23.514 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T20:20:23.514 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2023-05-31T20:20:23.514 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1685562970.274572_93.1", 2023-05-31T20:20:23.514 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T20:20:23.514 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2023-05-31T20:20:23.515 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T20:20:23.515 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2023-05-31T19:56:10.274572Z", 2023-05-31T20:20:23.515 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2023-05-31T20:20:23.515 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T20:20:23.515 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:20:23.515 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T20:20:23.516 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 3, 2023-05-31T20:20:23.516 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T20:20:23.516 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2023-05-31T20:20:23.516 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1685564285.225240_320.1", 2023-05-31T20:20:23.516 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T20:20:23.517 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2023-05-31T20:20:23.517 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T20:20:23.517 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2023-05-31T20:18:05.225240Z", 2023-05-31T20:20:23.518 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2023-05-31T20:20:23.518 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T20:20:23.518 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T20:20:23.519 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2023-05-31T20:20:23.519 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:20:23.520 INFO:teuthology.orchestra.run.smithi099.stdout: "full_sync": { 2023-05-31T20:20:23.520 INFO:teuthology.orchestra.run.smithi099.stdout: "total": 1, 2023-05-31T20:20:23.520 INFO:teuthology.orchestra.run.smithi099.stdout: "complete": 1 2023-05-31T20:20:23.520 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:20:23.520 INFO:teuthology.orchestra.run.smithi099.stdout: "current_time": "2023-05-31T20:20:23Z" 2023-05-31T20:20:23.520 INFO:teuthology.orchestra.run.smithi099.stdout:} 2023-05-31T20:20:23.522 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "2e6123d2-d695-4350-9a01-61a455bc782b", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1685562631.516089_59.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-05-31T19:50:31.516089Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1685562970.274572_93.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-05-31T19:56:10.274572Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1685564285.225240_320.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-05-31T20:18:05.225240Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-05-31T20:20:23Z" } 2023-05-31T20:20:23.522 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1685562631.516089_59.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-05-31T19:50:31.516089Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1685562970.274572_93.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-05-31T19:56:10.274572Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1685564285.225240_320.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-05-31T20:18:05.225240Z', 'realm_epoch': 2}}] 2023-05-31T20:20:23.523 DEBUG:rgw_multi.tests:log_status={0: '1_1685562631.516089_59.1', 1: '', 2: '1_1685562970.274572_93.1', 3: '1_1685564423.262837_330.1'} 2023-05-31T20:20:23.523 DEBUG:rgw_multi.tests:sync_status={0: '1_1685562631.516089_59.1', 1: '', 2: '1_1685562970.274572_93.1', 3: '1_1685564285.225240_320.1'} 2023-05-31T20:20:23.523 WARNING:rgw_multi.tests:zone a2 behind master: shard=3 master=1_1685564423.262837_330.1 target=1_1685564285.225240_320.1 2023-05-31T20:20:28.524 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-31T20:20:28.526 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-31T20:20:28.527 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-05-31T20:20:28.584 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2023-05-31T20:20:28.585 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2023-05-31T20:20:28.624 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:20:28.619+0000 7fb609ef1780 20 RGW-SYNC:meta: read sync status 2023-05-31T20:20:28.625 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:20:28.620+0000 7fb609ef1780 20 RGW-SYNC:meta: read sync status 2023-05-31T20:20:28.640 INFO:teuthology.orchestra.run.smithi099.stdout:{ 2023-05-31T20:20:28.641 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_status": { 2023-05-31T20:20:28.641 INFO:teuthology.orchestra.run.smithi099.stdout: "info": { 2023-05-31T20:20:28.642 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "sync", 2023-05-31T20:20:28.642 INFO:teuthology.orchestra.run.smithi099.stdout: "num_shards": 4, 2023-05-31T20:20:28.642 INFO:teuthology.orchestra.run.smithi099.stdout: "period": "2e6123d2-d695-4350-9a01-61a455bc782b", 2023-05-31T20:20:28.642 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2023-05-31T20:20:28.643 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:20:28.643 INFO:teuthology.orchestra.run.smithi099.stdout: "markers": [ 2023-05-31T20:20:28.643 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T20:20:28.643 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 0, 2023-05-31T20:20:28.644 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T20:20:28.644 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2023-05-31T20:20:28.644 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1685562631.516089_59.1", 2023-05-31T20:20:28.644 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T20:20:28.644 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 1, 2023-05-31T20:20:28.644 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T20:20:28.645 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2023-05-31T19:50:31.516089Z", 2023-05-31T20:20:28.645 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2023-05-31T20:20:28.645 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T20:20:28.645 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:20:28.645 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T20:20:28.645 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 1, 2023-05-31T20:20:28.646 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T20:20:28.646 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2023-05-31T20:20:28.646 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "", 2023-05-31T20:20:28.646 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T20:20:28.646 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2023-05-31T20:20:28.646 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T20:20:28.647 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "0.000000", 2023-05-31T20:20:28.647 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2023-05-31T20:20:28.647 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T20:20:28.647 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:20:28.648 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T20:20:28.648 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 2, 2023-05-31T20:20:28.648 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T20:20:28.648 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2023-05-31T20:20:28.648 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1685562970.274572_93.1", 2023-05-31T20:20:28.649 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T20:20:28.649 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2023-05-31T20:20:28.649 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T20:20:28.649 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2023-05-31T19:56:10.274572Z", 2023-05-31T20:20:28.649 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2023-05-31T20:20:28.649 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T20:20:28.649 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:20:28.650 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T20:20:28.650 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 3, 2023-05-31T20:20:28.650 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T20:20:28.650 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2023-05-31T20:20:28.650 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1685564423.262837_330.1", 2023-05-31T20:20:28.651 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T20:20:28.651 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2023-05-31T20:20:28.652 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T20:20:28.652 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2023-05-31T20:20:23.262837Z", 2023-05-31T20:20:28.652 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2023-05-31T20:20:28.653 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T20:20:28.653 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T20:20:28.653 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2023-05-31T20:20:28.653 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:20:28.654 INFO:teuthology.orchestra.run.smithi099.stdout: "full_sync": { 2023-05-31T20:20:28.654 INFO:teuthology.orchestra.run.smithi099.stdout: "total": 1, 2023-05-31T20:20:28.655 INFO:teuthology.orchestra.run.smithi099.stdout: "complete": 1 2023-05-31T20:20:28.655 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:20:28.655 INFO:teuthology.orchestra.run.smithi099.stdout: "current_time": "2023-05-31T20:20:28Z" 2023-05-31T20:20:28.656 INFO:teuthology.orchestra.run.smithi099.stdout:} 2023-05-31T20:20:28.657 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "2e6123d2-d695-4350-9a01-61a455bc782b", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1685562631.516089_59.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-05-31T19:50:31.516089Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1685562970.274572_93.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-05-31T19:56:10.274572Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1685564423.262837_330.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-05-31T20:20:23.262837Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-05-31T20:20:28Z" } 2023-05-31T20:20:28.658 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1685562631.516089_59.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-05-31T19:50:31.516089Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1685562970.274572_93.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-05-31T19:56:10.274572Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1685564423.262837_330.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-05-31T20:20:23.262837Z', 'realm_epoch': 2}}] 2023-05-31T20:20:28.658 DEBUG:rgw_multi.tests:log_status={0: '1_1685562631.516089_59.1', 1: '', 2: '1_1685562970.274572_93.1', 3: '1_1685564423.262837_330.1'} 2023-05-31T20:20:28.659 DEBUG:rgw_multi.tests:sync_status={0: '1_1685562631.516089_59.1', 1: '', 2: '1_1685562970.274572_93.1', 3: '1_1685564423.262837_330.1'} 2023-05-31T20:20:28.659 INFO:rgw_multi.tests:finish meta checkpoint for zone=a2 2023-05-31T20:20:28.659 DEBUG:boto:path=/pklxkd-48/ 2023-05-31T20:20:28.660 DEBUG:boto:auth_path=/pklxkd-48/ 2023-05-31T20:20:28.660 DEBUG:boto:Method: HEAD 2023-05-31T20:20:28.661 DEBUG:boto:Path: /pklxkd-48/ 2023-05-31T20:20:28.661 DEBUG:boto:Data: 2023-05-31T20:20:28.662 DEBUG:boto:Headers: {} 2023-05-31T20:20:28.662 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:20:28.663 DEBUG:boto:Port: 8000 2023-05-31T20:20:28.663 DEBUG:boto:Params: {} 2023-05-31T20:20:28.663 DEBUG:boto:Token: None 2023-05-31T20:20:28.664 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:20:28 GMT /pklxkd-48/ 2023-05-31T20:20:28.665 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:eC7D9UDIeP25JAL2Fs+oQ7n1hwc= 2023-05-31T20:20:28.665 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:20:28 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:eC7D9UDIeP25JAL2Fs+oQ7n1hwc=', 'Content-Length': '0'} 2023-05-31T20:20:28.668 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '4'), ('X-RGW-Bytes-Used', '24'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000cdd0758eaa6d4a4a-006477ac0c-1630-a1'), ('Content-Length', '0'), ('Date', 'Wed, 31 May 2023 20:20:28 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:20:28.669 DEBUG:boto:path=/pklxkd-48/obj5 2023-05-31T20:20:28.670 DEBUG:boto:auth_path=/pklxkd-48/obj5 2023-05-31T20:20:28.670 DEBUG:boto:Method: PUT 2023-05-31T20:20:28.671 DEBUG:boto:Path: /pklxkd-48/obj5 2023-05-31T20:20:28.671 DEBUG:boto:Data: 2023-05-31T20:20:28.671 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2023-05-31T20:20:28.672 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:20:28.673 DEBUG:boto:Port: 8000 2023-05-31T20:20:28.673 DEBUG:boto:Params: {} 2023-05-31T20:20:28.674 DEBUG:boto:Token: None 2023-05-31T20:20:28.675 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Wed, 31 May 2023 20:20:28 GMT /pklxkd-48/obj5 2023-05-31T20:20:28.675 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:9iduTB/o5pM3DeKVMgqAuCJG7+w= 2023-05-31T20:20:28.675 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Wed, 31 May 2023 20:20:28 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:9iduTB/o5pM3DeKVMgqAuCJG7+w='} 2023-05-31T20:20:28.681 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000d45a5a3c9e4e754b-006477ac0c-1630-a1'), ('Date', 'Wed, 31 May 2023 20:20:28 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:20:28.682 DEBUG:boto:path=/pklxkd-48/ 2023-05-31T20:20:28.683 DEBUG:boto:auth_path=/pklxkd-48/ 2023-05-31T20:20:28.684 DEBUG:boto:Method: HEAD 2023-05-31T20:20:28.685 DEBUG:boto:Path: /pklxkd-48/ 2023-05-31T20:20:28.685 DEBUG:boto:Data: 2023-05-31T20:20:28.686 DEBUG:boto:Headers: {} 2023-05-31T20:20:28.687 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:20:28.687 DEBUG:boto:Port: 8000 2023-05-31T20:20:28.688 DEBUG:boto:Params: {} 2023-05-31T20:20:28.689 DEBUG:boto:Token: None 2023-05-31T20:20:28.690 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:20:28 GMT /pklxkd-48/ 2023-05-31T20:20:28.690 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:eC7D9UDIeP25JAL2Fs+oQ7n1hwc= 2023-05-31T20:20:28.692 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:20:28 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:eC7D9UDIeP25JAL2Fs+oQ7n1hwc=', 'Content-Length': '0'} 2023-05-31T20:20:28.694 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '5'), ('X-RGW-Bytes-Used', '30'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000a2e034ff4ff082f7-006477ac0c-1630-a1'), ('Content-Length', '0'), ('Date', 'Wed, 31 May 2023 20:20:28 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:20:28.695 DEBUG:boto:path=/pklxkd-48/obj6 2023-05-31T20:20:28.696 DEBUG:boto:auth_path=/pklxkd-48/obj6 2023-05-31T20:20:28.697 DEBUG:boto:Method: PUT 2023-05-31T20:20:28.698 DEBUG:boto:Path: /pklxkd-48/obj6 2023-05-31T20:20:28.698 DEBUG:boto:Data: 2023-05-31T20:20:28.699 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2023-05-31T20:20:28.699 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:20:28.699 DEBUG:boto:Port: 8000 2023-05-31T20:20:28.700 DEBUG:boto:Params: {} 2023-05-31T20:20:28.700 DEBUG:boto:Token: None 2023-05-31T20:20:28.701 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Wed, 31 May 2023 20:20:28 GMT /pklxkd-48/obj6 2023-05-31T20:20:28.701 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:bf2LFCccD8S9F48M1fCPAajthwQ= 2023-05-31T20:20:28.701 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Wed, 31 May 2023 20:20:28 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:bf2LFCccD8S9F48M1fCPAajthwQ='} 2023-05-31T20:20:28.708 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000007514e8508aee191b-006477ac0c-1630-a1'), ('Date', 'Wed, 31 May 2023 20:20:28 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:20:28.709 DEBUG:boto:path=/pklxkd-48/ 2023-05-31T20:20:28.709 DEBUG:boto:auth_path=/pklxkd-48/ 2023-05-31T20:20:28.710 DEBUG:boto:Method: HEAD 2023-05-31T20:20:28.710 DEBUG:boto:Path: /pklxkd-48/ 2023-05-31T20:20:28.711 DEBUG:boto:Data: 2023-05-31T20:20:28.712 DEBUG:boto:Headers: {} 2023-05-31T20:20:28.712 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:20:28.713 DEBUG:boto:Port: 8000 2023-05-31T20:20:28.713 DEBUG:boto:Params: {} 2023-05-31T20:20:28.714 DEBUG:boto:Token: None 2023-05-31T20:20:28.714 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:20:28 GMT /pklxkd-48/ 2023-05-31T20:20:28.715 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:eC7D9UDIeP25JAL2Fs+oQ7n1hwc= 2023-05-31T20:20:28.716 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:20:28 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:eC7D9UDIeP25JAL2Fs+oQ7n1hwc=', 'Content-Length': '0'} 2023-05-31T20:20:28.718 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '6'), ('X-RGW-Bytes-Used', '36'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000009acafade9d0f0bc8-006477ac0c-1630-a1'), ('Content-Length', '0'), ('Date', 'Wed, 31 May 2023 20:20:28 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:20:28.718 DEBUG:boto:path=/pklxkd-48/obj7 2023-05-31T20:20:28.719 DEBUG:boto:auth_path=/pklxkd-48/obj7 2023-05-31T20:20:28.719 DEBUG:boto:Method: PUT 2023-05-31T20:20:28.720 DEBUG:boto:Path: /pklxkd-48/obj7 2023-05-31T20:20:28.720 DEBUG:boto:Data: 2023-05-31T20:20:28.720 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2023-05-31T20:20:28.721 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:20:28.721 DEBUG:boto:Port: 8000 2023-05-31T20:20:28.722 DEBUG:boto:Params: {} 2023-05-31T20:20:28.722 DEBUG:boto:Token: None 2023-05-31T20:20:28.723 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Wed, 31 May 2023 20:20:28 GMT /pklxkd-48/obj7 2023-05-31T20:20:28.724 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:+Z4f4Zxd0oGlscBz5CPHZyqjXhY= 2023-05-31T20:20:28.724 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Wed, 31 May 2023 20:20:28 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:+Z4f4Zxd0oGlscBz5CPHZyqjXhY='} 2023-05-31T20:20:28.768 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000004a23e9f64f1bd5c0-006477ac0c-1630-a1'), ('Date', 'Wed, 31 May 2023 20:20:28 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:20:28.769 DEBUG:boto:path=/pklxkd-48/ 2023-05-31T20:20:28.769 DEBUG:boto:auth_path=/pklxkd-48/ 2023-05-31T20:20:28.770 DEBUG:boto:Method: HEAD 2023-05-31T20:20:28.770 DEBUG:boto:Path: /pklxkd-48/ 2023-05-31T20:20:28.770 DEBUG:boto:Data: 2023-05-31T20:20:28.771 DEBUG:boto:Headers: {} 2023-05-31T20:20:28.771 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:20:28.771 DEBUG:boto:Port: 8000 2023-05-31T20:20:28.772 DEBUG:boto:Params: {} 2023-05-31T20:20:28.772 DEBUG:boto:Token: None 2023-05-31T20:20:28.773 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:20:28 GMT /pklxkd-48/ 2023-05-31T20:20:28.773 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:eC7D9UDIeP25JAL2Fs+oQ7n1hwc= 2023-05-31T20:20:28.773 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:20:28 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:eC7D9UDIeP25JAL2Fs+oQ7n1hwc=', 'Content-Length': '0'} 2023-05-31T20:20:28.776 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '7'), ('X-RGW-Bytes-Used', '42'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000031f4122c55dabce6-006477ac0c-1630-a1'), ('Content-Length', '0'), ('Date', 'Wed, 31 May 2023 20:20:28 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:20:28.777 DEBUG:boto:path=/pklxkd-48/obj8 2023-05-31T20:20:28.778 DEBUG:boto:auth_path=/pklxkd-48/obj8 2023-05-31T20:20:28.778 DEBUG:boto:Method: PUT 2023-05-31T20:20:28.778 DEBUG:boto:Path: /pklxkd-48/obj8 2023-05-31T20:20:28.779 DEBUG:boto:Data: 2023-05-31T20:20:28.779 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2023-05-31T20:20:28.779 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:20:28.780 DEBUG:boto:Port: 8000 2023-05-31T20:20:28.780 DEBUG:boto:Params: {} 2023-05-31T20:20:28.780 DEBUG:boto:Token: None 2023-05-31T20:20:28.781 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Wed, 31 May 2023 20:20:28 GMT /pklxkd-48/obj8 2023-05-31T20:20:28.781 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:FOKZUdDjWFRSHQVkxoaeR6ArUrY= 2023-05-31T20:20:28.782 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Wed, 31 May 2023 20:20:28 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:FOKZUdDjWFRSHQVkxoaeR6ArUrY='} 2023-05-31T20:20:28.787 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000c366e21fa176d486-006477ac0c-1630-a1'), ('Date', 'Wed, 31 May 2023 20:20:28 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:20:28.788 DEBUG:boto:path=/pklxkd-49/ 2023-05-31T20:20:28.788 DEBUG:boto:auth_path=/pklxkd-49/ 2023-05-31T20:20:28.788 DEBUG:boto:Method: HEAD 2023-05-31T20:20:28.789 DEBUG:boto:Path: /pklxkd-49/ 2023-05-31T20:20:28.789 DEBUG:boto:Data: 2023-05-31T20:20:28.789 DEBUG:boto:Headers: {} 2023-05-31T20:20:28.790 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:20:28.790 DEBUG:boto:Port: 8001 2023-05-31T20:20:28.790 DEBUG:boto:Params: {} 2023-05-31T20:20:28.791 DEBUG:boto:Token: None 2023-05-31T20:20:28.791 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:20:28 GMT /pklxkd-49/ 2023-05-31T20:20:28.792 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:l0XGnXYvU0yCIfZAnGMsbqsDx8Q= 2023-05-31T20:20:28.792 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:20:28 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:l0XGnXYvU0yCIfZAnGMsbqsDx8Q=', 'Content-Length': '0'} 2023-05-31T20:20:28.794 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '4'), ('X-RGW-Bytes-Used', '24'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000c41cc873eae86588-006477ac0c-162e-a2'), ('Content-Length', '0'), ('Date', 'Wed, 31 May 2023 20:20:28 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:20:28.795 DEBUG:boto:path=/pklxkd-49/obj5 2023-05-31T20:20:28.795 DEBUG:boto:auth_path=/pklxkd-49/obj5 2023-05-31T20:20:28.796 DEBUG:boto:Method: PUT 2023-05-31T20:20:28.796 DEBUG:boto:Path: /pklxkd-49/obj5 2023-05-31T20:20:28.796 DEBUG:boto:Data: 2023-05-31T20:20:28.797 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2023-05-31T20:20:28.797 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:20:28.798 DEBUG:boto:Port: 8001 2023-05-31T20:20:28.798 DEBUG:boto:Params: {} 2023-05-31T20:20:28.798 DEBUG:boto:Token: None 2023-05-31T20:20:28.799 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Wed, 31 May 2023 20:20:28 GMT /pklxkd-49/obj5 2023-05-31T20:20:28.799 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:qWWNHpxGxpG0st0AEpynTcX9RYU= 2023-05-31T20:20:28.800 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Wed, 31 May 2023 20:20:28 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:qWWNHpxGxpG0st0AEpynTcX9RYU='} 2023-05-31T20:20:28.803 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000001aa67afd650fa2c3-006477ac0c-162e-a2'), ('Date', 'Wed, 31 May 2023 20:20:28 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:20:28.804 DEBUG:boto:path=/pklxkd-49/ 2023-05-31T20:20:28.804 DEBUG:boto:auth_path=/pklxkd-49/ 2023-05-31T20:20:28.805 DEBUG:boto:Method: HEAD 2023-05-31T20:20:28.805 DEBUG:boto:Path: /pklxkd-49/ 2023-05-31T20:20:28.805 DEBUG:boto:Data: 2023-05-31T20:20:28.806 DEBUG:boto:Headers: {} 2023-05-31T20:20:28.806 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:20:28.806 DEBUG:boto:Port: 8001 2023-05-31T20:20:28.807 DEBUG:boto:Params: {} 2023-05-31T20:20:28.807 DEBUG:boto:Token: None 2023-05-31T20:20:28.807 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:20:28 GMT /pklxkd-49/ 2023-05-31T20:20:28.808 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:l0XGnXYvU0yCIfZAnGMsbqsDx8Q= 2023-05-31T20:20:28.809 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:20:28 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:l0XGnXYvU0yCIfZAnGMsbqsDx8Q=', 'Content-Length': '0'} 2023-05-31T20:20:28.811 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '5'), ('X-RGW-Bytes-Used', '30'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000bbafb3baa9c572c3-006477ac0c-162e-a2'), ('Content-Length', '0'), ('Date', 'Wed, 31 May 2023 20:20:28 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:20:28.812 DEBUG:boto:path=/pklxkd-49/obj6 2023-05-31T20:20:28.812 DEBUG:boto:auth_path=/pklxkd-49/obj6 2023-05-31T20:20:28.812 DEBUG:boto:Method: PUT 2023-05-31T20:20:28.813 DEBUG:boto:Path: /pklxkd-49/obj6 2023-05-31T20:20:28.813 DEBUG:boto:Data: 2023-05-31T20:20:28.814 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2023-05-31T20:20:28.814 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:20:28.814 DEBUG:boto:Port: 8001 2023-05-31T20:20:28.815 DEBUG:boto:Params: {} 2023-05-31T20:20:28.815 DEBUG:boto:Token: None 2023-05-31T20:20:28.816 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Wed, 31 May 2023 20:20:28 GMT /pklxkd-49/obj6 2023-05-31T20:20:28.816 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:cOjrMdxYT8FH6msx5plLuIPjTpk= 2023-05-31T20:20:28.817 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Wed, 31 May 2023 20:20:28 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:cOjrMdxYT8FH6msx5plLuIPjTpk='} 2023-05-31T20:20:28.821 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000045529582cc624cec-006477ac0c-162e-a2'), ('Date', 'Wed, 31 May 2023 20:20:28 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:20:28.822 DEBUG:boto:path=/pklxkd-49/ 2023-05-31T20:20:28.822 DEBUG:boto:auth_path=/pklxkd-49/ 2023-05-31T20:20:28.822 DEBUG:boto:Method: HEAD 2023-05-31T20:20:28.823 DEBUG:boto:Path: /pklxkd-49/ 2023-05-31T20:20:28.823 DEBUG:boto:Data: 2023-05-31T20:20:28.823 DEBUG:boto:Headers: {} 2023-05-31T20:20:28.824 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:20:28.824 DEBUG:boto:Port: 8001 2023-05-31T20:20:28.824 DEBUG:boto:Params: {} 2023-05-31T20:20:28.825 DEBUG:boto:Token: None 2023-05-31T20:20:28.825 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:20:28 GMT /pklxkd-49/ 2023-05-31T20:20:28.826 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:l0XGnXYvU0yCIfZAnGMsbqsDx8Q= 2023-05-31T20:20:28.826 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:20:28 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:l0XGnXYvU0yCIfZAnGMsbqsDx8Q=', 'Content-Length': '0'} 2023-05-31T20:20:28.828 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '6'), ('X-RGW-Bytes-Used', '36'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000004501960ec2401911-006477ac0c-162e-a2'), ('Content-Length', '0'), ('Date', 'Wed, 31 May 2023 20:20:28 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:20:28.829 DEBUG:boto:path=/pklxkd-49/obj7 2023-05-31T20:20:28.829 DEBUG:boto:auth_path=/pklxkd-49/obj7 2023-05-31T20:20:28.829 DEBUG:boto:Method: PUT 2023-05-31T20:20:28.830 DEBUG:boto:Path: /pklxkd-49/obj7 2023-05-31T20:20:28.830 DEBUG:boto:Data: 2023-05-31T20:20:28.830 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2023-05-31T20:20:28.831 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:20:28.831 DEBUG:boto:Port: 8001 2023-05-31T20:20:28.831 DEBUG:boto:Params: {} 2023-05-31T20:20:28.832 DEBUG:boto:Token: None 2023-05-31T20:20:28.832 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Wed, 31 May 2023 20:20:28 GMT /pklxkd-49/obj7 2023-05-31T20:20:28.833 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:RCo6h4xeyTA4PQI2aTeC7lRP1Ac= 2023-05-31T20:20:28.833 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Wed, 31 May 2023 20:20:28 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:RCo6h4xeyTA4PQI2aTeC7lRP1Ac='} 2023-05-31T20:20:28.837 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000009eb89a9dd87dfa05-006477ac0c-162e-a2'), ('Date', 'Wed, 31 May 2023 20:20:28 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:20:28.837 DEBUG:boto:path=/pklxkd-49/ 2023-05-31T20:20:28.838 DEBUG:boto:auth_path=/pklxkd-49/ 2023-05-31T20:20:28.838 DEBUG:boto:Method: HEAD 2023-05-31T20:20:28.838 DEBUG:boto:Path: /pklxkd-49/ 2023-05-31T20:20:28.839 DEBUG:boto:Data: 2023-05-31T20:20:28.839 DEBUG:boto:Headers: {} 2023-05-31T20:20:28.839 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:20:28.840 DEBUG:boto:Port: 8001 2023-05-31T20:20:28.840 DEBUG:boto:Params: {} 2023-05-31T20:20:28.841 DEBUG:boto:Token: None 2023-05-31T20:20:28.841 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:20:28 GMT /pklxkd-49/ 2023-05-31T20:20:28.841 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:l0XGnXYvU0yCIfZAnGMsbqsDx8Q= 2023-05-31T20:20:28.842 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:20:28 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:l0XGnXYvU0yCIfZAnGMsbqsDx8Q=', 'Content-Length': '0'} 2023-05-31T20:20:28.844 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '7'), ('X-RGW-Bytes-Used', '42'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000b34dbca19c01f04a-006477ac0c-162e-a2'), ('Content-Length', '0'), ('Date', 'Wed, 31 May 2023 20:20:28 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:20:28.844 DEBUG:boto:path=/pklxkd-49/obj8 2023-05-31T20:20:28.845 DEBUG:boto:auth_path=/pklxkd-49/obj8 2023-05-31T20:20:28.845 DEBUG:boto:Method: PUT 2023-05-31T20:20:28.845 DEBUG:boto:Path: /pklxkd-49/obj8 2023-05-31T20:20:28.846 DEBUG:boto:Data: 2023-05-31T20:20:28.846 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2023-05-31T20:20:28.846 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:20:28.847 DEBUG:boto:Port: 8001 2023-05-31T20:20:28.847 DEBUG:boto:Params: {} 2023-05-31T20:20:28.848 DEBUG:boto:Token: None 2023-05-31T20:20:28.848 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Wed, 31 May 2023 20:20:28 GMT /pklxkd-49/obj8 2023-05-31T20:20:28.848 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:x7pEQoc1JZ+xg+qlf0MM6wKkuww= 2023-05-31T20:20:28.849 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Wed, 31 May 2023 20:20:28 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:x7pEQoc1JZ+xg+qlf0MM6wKkuww='} 2023-05-31T20:20:28.892 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000df1a19f958d9a3f4-006477ac0c-162e-a2'), ('Date', 'Wed, 31 May 2023 20:20:28 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:20:28.893 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'enable', '--bucket', 'pklxkd-48', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-05-31T20:20:28.894 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'enable', '--bucket', 'pklxkd-48', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-05-31T20:20:28.894 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync enable --bucket pklxkd-48 --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 2023-05-31T20:20:28.945 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2023-05-31T20:20:28.946 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2023-05-31T20:20:29.075 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'enable', '--bucket', 'pklxkd-49', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-05-31T20:20:29.075 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'enable', '--bucket', 'pklxkd-49', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-05-31T20:20:29.076 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync enable --bucket pklxkd-49 --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 2023-05-31T20:20:29.126 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2023-05-31T20:20:29.127 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2023-05-31T20:20:29.231 DEBUG:rgw_multi.tests:bucket checkpoint: source=a1 target=a2 bucket=pklxkd-48 2023-05-31T20:20:29.231 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'pklxkd-48', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-05-31T20:20:29.232 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'pklxkd-48', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-05-31T20:20:29.232 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket pklxkd-48 --source-zone a1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2023-05-31T20:20:29.288 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2023-05-31T20:20:29.289 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2023-05-31T20:20:29.326 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:20:29.321+0000 7f322078c780 1 waiting for incremental sync to catch up: 2023-05-31T20:20:29.326 INFO:teuthology.orchestra.run.smithi099.stderr: local status: [00000000001.16.6, , 00000000001.47.6, , 00000000001.19.6, , , 00000000001.25.6, , , ] 2023-05-31T20:20:29.326 INFO:teuthology.orchestra.run.smithi099.stderr: remote markers: [00000000004.21.1, 00000000002.30.1, 00000000004.51.1, 00000000002.21.1, 00000000004.23.1, 00000000002.18.1, 00000000002.19.1, 00000000004.33.1, 00000000002.22.1, 00000000002.22.1, 00000000002.30.1] 2023-05-31T20:20:34.327 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:20:34.322+0000 7f322078c780 1 waiting for incremental sync to catch up: 2023-05-31T20:20:34.327 INFO:teuthology.orchestra.run.smithi099.stderr: local status: [00000000001.16.6, , 00000000001.47.6, , 00000000001.19.6, , , 00000000001.25.6, , , ] 2023-05-31T20:20:34.327 INFO:teuthology.orchestra.run.smithi099.stderr: remote markers: [00000000004.21.1, 00000000002.30.1, 00000000004.51.1, 00000000002.21.1, 00000000004.23.1, 00000000002.18.1, 00000000002.19.1, 00000000004.33.1, 00000000002.22.1, 00000000002.22.1, 00000000002.30.1] 2023-05-31T20:20:39.328 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:20:39.322+0000 7f322078c780 1 waiting for incremental sync to catch up: 2023-05-31T20:20:39.328 INFO:teuthology.orchestra.run.smithi099.stderr: local status: [00000000001.16.6, , 00000000001.47.6, , 00000000001.19.6, , , 00000000001.25.6, , , ] 2023-05-31T20:20:39.328 INFO:teuthology.orchestra.run.smithi099.stderr: remote markers: [00000000004.21.1, 00000000002.30.1, 00000000004.51.1, 00000000002.21.1, 00000000004.23.1, 00000000002.18.1, 00000000002.19.1, 00000000004.33.1, 00000000002.22.1, 00000000002.22.1, 00000000002.30.1] 2023-05-31T20:20:44.329 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:20:44.323+0000 7f322078c780 1 bucket sync caught up with source: 2023-05-31T20:20:44.330 INFO:teuthology.orchestra.run.smithi099.stderr: local status: [00000000004.21.1, 00000000002.30.1, 00000000004.51.1, 00000000002.21.1, 00000000004.23.1, 00000000002.18.1, 00000000002.19.1, 00000000004.33.1, 00000000002.22.1, 00000000002.22.1, 00000000002.30.1] 2023-05-31T20:20:44.330 INFO:teuthology.orchestra.run.smithi099.stderr: remote markers: [00000000004.21.1, 00000000002.30.1, 00000000004.51.1, 00000000002.21.1, 00000000004.23.1, 00000000002.18.1, 00000000002.19.1, 00000000004.33.1, 00000000002.22.1, 00000000002.22.1, 00000000002.30.1] 2023-05-31T20:20:44.330 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:20:44.323+0000 7f322078c780 0 bucket checkpoint complete 2023-05-31T20:20:44.351 DEBUG:rgw_multi.tests:bucket checkpoint: source=a2 target=a1 bucket=pklxkd-48 2023-05-31T20:20:44.352 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'pklxkd-48', '--source-zone', 'a2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-05-31T20:20:44.353 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'pklxkd-48', '--source-zone', 'a2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-05-31T20:20:44.353 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket pklxkd-48 --source-zone a2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2023-05-31T20:20:44.414 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2023-05-31T20:20:44.415 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2023-05-31T20:20:44.478 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:20:44.472+0000 7f77fffe7780 1 waiting to reach incremental sync.. 2023-05-31T20:20:49.479 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:20:49.473+0000 7f77fffe7780 1 waiting to reach incremental sync.. 2023-05-31T20:20:54.480 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:20:54.474+0000 7f77fffe7780 1 waiting to reach incremental sync.. 2023-05-31T20:20:59.481 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:20:59.476+0000 7f77fffe7780 1 waiting to reach incremental sync.. 2023-05-31T20:21:04.482 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:21:04.478+0000 7f77fffe7780 1 bucket sync caught up with source: 2023-05-31T20:21:04.483 INFO:teuthology.orchestra.run.smithi099.stderr: local status: [00000000004.676.1, 00000000002.2234.1, 00000000004.1062.1, 00000000002.2251.1, 00000000004.1023.1, 00000000002.1906.1, 00000000002.2270.1, 00000000004.1333.1, 00000000002.677.1, 00000000002.757.1, 00000000002.1984.1] 2023-05-31T20:21:04.483 INFO:teuthology.orchestra.run.smithi099.stderr: remote markers: [00000000002.670.1, 00000000001.2230.1, 00000000002.1058.1, 00000000001.2250.1, 00000000002.1019.1, 00000000001.1902.1, 00000000001.2269.1, 00000000002.1323.1, 00000000001.671.1, 00000000001.756.1, 00000000001.1983.1] 2023-05-31T20:21:04.483 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:21:04.478+0000 7f77fffe7780 0 bucket checkpoint complete 2023-05-31T20:21:04.500 INFO:tasks.rgw_multi.zone_rados:comparing bucket=pklxkd-48 zones={a2, a1} 2023-05-31T20:21:04.500 DEBUG:boto:path=/pklxkd-48/ 2023-05-31T20:21:04.501 DEBUG:boto:auth_path=/pklxkd-48/ 2023-05-31T20:21:04.502 DEBUG:boto:Method: HEAD 2023-05-31T20:21:04.502 DEBUG:boto:Path: /pklxkd-48/ 2023-05-31T20:21:04.503 DEBUG:boto:Data: 2023-05-31T20:21:04.504 DEBUG:boto:Headers: {} 2023-05-31T20:21:04.505 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:21:04.506 DEBUG:boto:Port: 8001 2023-05-31T20:21:04.507 DEBUG:boto:Params: {} 2023-05-31T20:21:04.508 DEBUG:boto:Token: None 2023-05-31T20:21:04.509 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:21:04 GMT /pklxkd-48/ 2023-05-31T20:21:04.509 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:MWYUpo3m/WlE5t31cXV0gx0HQhg= 2023-05-31T20:21:04.510 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:21:04 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:MWYUpo3m/WlE5t31cXV0gx0HQhg=', 'Content-Length': '0'} 2023-05-31T20:21:04.512 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '8'), ('X-RGW-Bytes-Used', '48'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000004fa8a9c9d5a4cab2-006477ac30-162e-a2'), ('Content-Length', '0'), ('Date', 'Wed, 31 May 2023 20:21:04 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:21:04.513 DEBUG:boto:path=/pklxkd-48/ 2023-05-31T20:21:04.514 DEBUG:boto:auth_path=/pklxkd-48/ 2023-05-31T20:21:04.515 DEBUG:boto:Method: HEAD 2023-05-31T20:21:04.516 DEBUG:boto:Path: /pklxkd-48/ 2023-05-31T20:21:04.516 DEBUG:boto:Data: 2023-05-31T20:21:04.517 DEBUG:boto:Headers: {} 2023-05-31T20:21:04.517 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:21:04.518 DEBUG:boto:Port: 8000 2023-05-31T20:21:04.518 DEBUG:boto:Params: {} 2023-05-31T20:21:04.519 DEBUG:boto:Token: None 2023-05-31T20:21:04.519 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:21:04 GMT /pklxkd-48/ 2023-05-31T20:21:04.520 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:MWYUpo3m/WlE5t31cXV0gx0HQhg= 2023-05-31T20:21:04.521 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:21:04 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:MWYUpo3m/WlE5t31cXV0gx0HQhg=', 'Content-Length': '0'} 2023-05-31T20:21:04.524 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '8'), ('X-RGW-Bytes-Used', '48'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000000994e4cd761f9761-006477ac30-1630-a1'), ('Content-Length', '0'), ('Date', 'Wed, 31 May 2023 20:21:04 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:21:04.525 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2023-05-31T20:21:04.526 DEBUG:boto:path=/pklxkd-48/ 2023-05-31T20:21:04.526 DEBUG:boto:auth_path=/pklxkd-48/ 2023-05-31T20:21:04.527 DEBUG:boto:path=/pklxkd-48/?versions&max-keys=999 2023-05-31T20:21:04.527 DEBUG:boto:auth_path=/pklxkd-48/?versions&max-keys=999 2023-05-31T20:21:04.528 DEBUG:boto:Method: GET 2023-05-31T20:21:04.528 DEBUG:boto:Path: /pklxkd-48/?versions&max-keys=999 2023-05-31T20:21:04.528 DEBUG:boto:Data: 2023-05-31T20:21:04.529 DEBUG:boto:Headers: {} 2023-05-31T20:21:04.529 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:21:04.529 DEBUG:boto:Port: 8001 2023-05-31T20:21:04.530 DEBUG:boto:Params: {} 2023-05-31T20:21:04.530 DEBUG:boto:Token: None 2023-05-31T20:21:04.530 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:21:04 GMT /pklxkd-48/?versions 2023-05-31T20:21:04.531 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:9CZqD2rY2DEGmJ02Io3M5gK77TE= 2023-05-31T20:21:04.531 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:21:04 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:9CZqD2rY2DEGmJ02Io3M5gK77TE=', 'Content-Length': '0'} 2023-05-31T20:21:04.533 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b9b60985d5811e6a-006477ac30-162e-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:21:04 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:21:04.534 DEBUG:boto:b'pklxkd-48999falseobj1nulltrue2023-05-31T20:18:05.255Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj2nulltrue2023-05-31T20:18:05.277Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj3nulltrue2023-05-31T20:18:05.338Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj4nulltrue2023-05-31T20:18:05.356Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj5nulltrue2023-05-31T20:20:28.673Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj6nulltrue2023-05-31T20:20:28.699Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj7nulltrue2023-05-31T20:20:28.722Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj8nulltrue2023-05-31T20:20:28.779Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:21:04.535 DEBUG:tasks.rgw_multi.zone_rados:o=obj1 2023-05-31T20:21:04.535 DEBUG:tasks.rgw_multi.zone_rados:o=obj2 2023-05-31T20:21:04.536 DEBUG:tasks.rgw_multi.zone_rados:o=obj3 2023-05-31T20:21:04.536 DEBUG:tasks.rgw_multi.zone_rados:o=obj4 2023-05-31T20:21:04.536 DEBUG:tasks.rgw_multi.zone_rados:o=obj5 2023-05-31T20:21:04.537 DEBUG:tasks.rgw_multi.zone_rados:o=obj6 2023-05-31T20:21:04.537 DEBUG:tasks.rgw_multi.zone_rados:o=obj7 2023-05-31T20:21:04.537 DEBUG:tasks.rgw_multi.zone_rados:o=obj8 2023-05-31T20:21:04.537 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2023-05-31T20:21:04.538 DEBUG:boto:path=/pklxkd-48/ 2023-05-31T20:21:04.538 DEBUG:boto:auth_path=/pklxkd-48/ 2023-05-31T20:21:04.538 DEBUG:boto:path=/pklxkd-48/?versions&max-keys=999 2023-05-31T20:21:04.539 DEBUG:boto:auth_path=/pklxkd-48/?versions&max-keys=999 2023-05-31T20:21:04.539 DEBUG:boto:Method: GET 2023-05-31T20:21:04.539 DEBUG:boto:Path: /pklxkd-48/?versions&max-keys=999 2023-05-31T20:21:04.540 DEBUG:boto:Data: 2023-05-31T20:21:04.540 DEBUG:boto:Headers: {} 2023-05-31T20:21:04.541 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:21:04.541 DEBUG:boto:Port: 8000 2023-05-31T20:21:04.541 DEBUG:boto:Params: {} 2023-05-31T20:21:04.542 DEBUG:boto:Token: None 2023-05-31T20:21:04.542 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:21:04 GMT /pklxkd-48/?versions 2023-05-31T20:21:04.542 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:9CZqD2rY2DEGmJ02Io3M5gK77TE= 2023-05-31T20:21:04.543 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:21:04 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:9CZqD2rY2DEGmJ02Io3M5gK77TE=', 'Content-Length': '0'} 2023-05-31T20:21:04.545 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d742715d393b0ad1-006477ac30-1630-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:21:04 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:21:04.546 DEBUG:boto:b'pklxkd-48999falseobj1nulltrue2023-05-31T20:18:05.255Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj2nulltrue2023-05-31T20:18:05.277Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj3nulltrue2023-05-31T20:18:05.338Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj4nulltrue2023-05-31T20:18:05.356Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj5nulltrue2023-05-31T20:20:28.673Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj6nulltrue2023-05-31T20:20:28.699Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj7nulltrue2023-05-31T20:20:28.722Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj8nulltrue2023-05-31T20:20:28.779Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:21:04.547 DEBUG:tasks.rgw_multi.zone_rados:o=obj1 2023-05-31T20:21:04.547 DEBUG:tasks.rgw_multi.zone_rados:o=obj2 2023-05-31T20:21:04.547 DEBUG:tasks.rgw_multi.zone_rados:o=obj3 2023-05-31T20:21:04.548 DEBUG:tasks.rgw_multi.zone_rados:o=obj4 2023-05-31T20:21:04.548 DEBUG:tasks.rgw_multi.zone_rados:o=obj5 2023-05-31T20:21:04.548 DEBUG:tasks.rgw_multi.zone_rados:o=obj6 2023-05-31T20:21:04.548 DEBUG:tasks.rgw_multi.zone_rados:o=obj7 2023-05-31T20:21:04.549 DEBUG:tasks.rgw_multi.zone_rados:o=obj8 2023-05-31T20:21:04.549 DEBUG:boto:path=/pklxkd-48/ 2023-05-31T20:21:04.550 DEBUG:boto:auth_path=/pklxkd-48/ 2023-05-31T20:21:04.550 DEBUG:boto:path=/pklxkd-48/?versions&max-keys=999 2023-05-31T20:21:04.550 DEBUG:boto:auth_path=/pklxkd-48/?versions&max-keys=999 2023-05-31T20:21:04.551 DEBUG:boto:Method: GET 2023-05-31T20:21:04.551 DEBUG:boto:Path: /pklxkd-48/?versions&max-keys=999 2023-05-31T20:21:04.551 DEBUG:boto:Data: 2023-05-31T20:21:04.552 DEBUG:boto:Headers: {} 2023-05-31T20:21:04.552 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:21:04.552 DEBUG:boto:Port: 8001 2023-05-31T20:21:04.553 DEBUG:boto:Params: {} 2023-05-31T20:21:04.553 DEBUG:boto:Token: None 2023-05-31T20:21:04.554 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:21:04 GMT /pklxkd-48/?versions 2023-05-31T20:21:04.554 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:9CZqD2rY2DEGmJ02Io3M5gK77TE= 2023-05-31T20:21:04.555 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:21:04 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:9CZqD2rY2DEGmJ02Io3M5gK77TE=', 'Content-Length': '0'} 2023-05-31T20:21:04.557 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e25678752ca66b1e-006477ac30-162e-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:21:04 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:21:04.557 DEBUG:boto:b'pklxkd-48999falseobj1nulltrue2023-05-31T20:18:05.255Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj2nulltrue2023-05-31T20:18:05.277Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj3nulltrue2023-05-31T20:18:05.338Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj4nulltrue2023-05-31T20:18:05.356Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj5nulltrue2023-05-31T20:20:28.673Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj6nulltrue2023-05-31T20:20:28.699Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj7nulltrue2023-05-31T20:20:28.722Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj8nulltrue2023-05-31T20:20:28.779Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:21:04.558 DEBUG:boto:path=/pklxkd-48/ 2023-05-31T20:21:04.559 DEBUG:boto:auth_path=/pklxkd-48/ 2023-05-31T20:21:04.559 DEBUG:boto:path=/pklxkd-48/?versions&max-keys=999 2023-05-31T20:21:04.559 DEBUG:boto:auth_path=/pklxkd-48/?versions&max-keys=999 2023-05-31T20:21:04.560 DEBUG:boto:Method: GET 2023-05-31T20:21:04.560 DEBUG:boto:Path: /pklxkd-48/?versions&max-keys=999 2023-05-31T20:21:04.560 DEBUG:boto:Data: 2023-05-31T20:21:04.561 DEBUG:boto:Headers: {} 2023-05-31T20:21:04.561 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:21:04.561 DEBUG:boto:Port: 8000 2023-05-31T20:21:04.562 DEBUG:boto:Params: {} 2023-05-31T20:21:04.562 DEBUG:boto:Token: None 2023-05-31T20:21:04.563 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:21:04 GMT /pklxkd-48/?versions 2023-05-31T20:21:04.563 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:9CZqD2rY2DEGmJ02Io3M5gK77TE= 2023-05-31T20:21:04.563 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:21:04 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:9CZqD2rY2DEGmJ02Io3M5gK77TE=', 'Content-Length': '0'} 2023-05-31T20:21:04.566 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a4755bce17e9a96d-006477ac30-1630-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:21:04 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:21:04.566 DEBUG:boto:b'pklxkd-48999falseobj1nulltrue2023-05-31T20:18:05.255Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj2nulltrue2023-05-31T20:18:05.277Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj3nulltrue2023-05-31T20:18:05.338Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj4nulltrue2023-05-31T20:18:05.356Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj5nulltrue2023-05-31T20:20:28.673Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj6nulltrue2023-05-31T20:20:28.699Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj7nulltrue2023-05-31T20:20:28.722Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj8nulltrue2023-05-31T20:20:28.779Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:21:04.567 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2023-05-31T20:21:04.568 DEBUG:boto:path=/pklxkd-48/obj1 2023-05-31T20:21:04.568 DEBUG:boto:auth_path=/pklxkd-48/obj1 2023-05-31T20:21:04.568 DEBUG:boto:path=/pklxkd-48/obj1?versionId=null 2023-05-31T20:21:04.569 DEBUG:boto:auth_path=/pklxkd-48/obj1?versionId=null 2023-05-31T20:21:04.569 DEBUG:boto:Method: GET 2023-05-31T20:21:04.570 DEBUG:boto:Path: /pklxkd-48/obj1?versionId=null 2023-05-31T20:21:04.570 DEBUG:boto:Data: 2023-05-31T20:21:04.570 DEBUG:boto:Headers: {} 2023-05-31T20:21:04.571 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:21:04.572 DEBUG:boto:Port: 8001 2023-05-31T20:21:04.572 DEBUG:boto:Params: {} 2023-05-31T20:21:04.573 DEBUG:boto:Token: None 2023-05-31T20:21:04.573 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:21:04 GMT /pklxkd-48/obj1?versionId=null 2023-05-31T20:21:04.574 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:x768p2y4SOQJNbPtU8L1DxC336I= 2023-05-31T20:21:04.575 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:21:04 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:x768p2y4SOQJNbPtU8L1DxC336I=', 'Content-Length': '0'} 2023-05-31T20:21:04.576 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:18:05 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000aad04e2778378a51-006477ac30-162e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:21:04 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:21:04.577 DEBUG:boto:path=/pklxkd-48/obj1 2023-05-31T20:21:04.577 DEBUG:boto:auth_path=/pklxkd-48/obj1 2023-05-31T20:21:04.578 DEBUG:boto:path=/pklxkd-48/obj1?versionId=null 2023-05-31T20:21:04.578 DEBUG:boto:auth_path=/pklxkd-48/obj1?versionId=null 2023-05-31T20:21:04.578 DEBUG:boto:Method: GET 2023-05-31T20:21:04.579 DEBUG:boto:Path: /pklxkd-48/obj1?versionId=null 2023-05-31T20:21:04.579 DEBUG:boto:Data: 2023-05-31T20:21:04.580 DEBUG:boto:Headers: {} 2023-05-31T20:21:04.580 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:21:04.581 DEBUG:boto:Port: 8000 2023-05-31T20:21:04.581 DEBUG:boto:Params: {} 2023-05-31T20:21:04.581 DEBUG:boto:Token: None 2023-05-31T20:21:04.582 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:21:04 GMT /pklxkd-48/obj1?versionId=null 2023-05-31T20:21:04.583 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:x768p2y4SOQJNbPtU8L1DxC336I= 2023-05-31T20:21:04.583 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:21:04 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:x768p2y4SOQJNbPtU8L1DxC336I=', 'Content-Length': '0'} 2023-05-31T20:21:04.585 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:18:05 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000e1893f696d22c8d3-006477ac30-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:21:04 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:21:04.586 DEBUG:boto:path=/pklxkd-48/obj1 2023-05-31T20:21:04.586 DEBUG:boto:auth_path=/pklxkd-48/obj1 2023-05-31T20:21:04.586 DEBUG:boto:path=/pklxkd-48/obj1?versionId=null 2023-05-31T20:21:04.587 DEBUG:boto:auth_path=/pklxkd-48/obj1?versionId=null 2023-05-31T20:21:04.587 DEBUG:boto:Method: HEAD 2023-05-31T20:21:04.587 DEBUG:boto:Path: /pklxkd-48/obj1?versionId=null 2023-05-31T20:21:04.588 DEBUG:boto:Data: 2023-05-31T20:21:04.588 DEBUG:boto:Headers: {} 2023-05-31T20:21:04.588 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:21:04.589 DEBUG:boto:Port: 8001 2023-05-31T20:21:04.589 DEBUG:boto:Params: {} 2023-05-31T20:21:04.589 DEBUG:boto:Token: None 2023-05-31T20:21:04.590 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:21:04 GMT /pklxkd-48/obj1?versionId=null 2023-05-31T20:21:04.590 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:m1lLyRoNmhbdXAFN+cyFfmXybO4= 2023-05-31T20:21:04.591 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:21:04 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:m1lLyRoNmhbdXAFN+cyFfmXybO4=', 'Content-Length': '0'} 2023-05-31T20:21:04.593 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:18:05 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000e3b94696a2b126b9-006477ac30-162e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:21:04 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:21:04.593 DEBUG:boto:path=/pklxkd-48/obj1 2023-05-31T20:21:04.594 DEBUG:boto:auth_path=/pklxkd-48/obj1 2023-05-31T20:21:04.594 DEBUG:boto:path=/pklxkd-48/obj1?versionId=null 2023-05-31T20:21:04.594 DEBUG:boto:auth_path=/pklxkd-48/obj1?versionId=null 2023-05-31T20:21:04.595 DEBUG:boto:Method: HEAD 2023-05-31T20:21:04.595 DEBUG:boto:Path: /pklxkd-48/obj1?versionId=null 2023-05-31T20:21:04.596 DEBUG:boto:Data: 2023-05-31T20:21:04.596 DEBUG:boto:Headers: {} 2023-05-31T20:21:04.596 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:21:04.597 DEBUG:boto:Port: 8000 2023-05-31T20:21:04.597 DEBUG:boto:Params: {} 2023-05-31T20:21:04.598 DEBUG:boto:Token: None 2023-05-31T20:21:04.598 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:21:04 GMT /pklxkd-48/obj1?versionId=null 2023-05-31T20:21:04.599 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:m1lLyRoNmhbdXAFN+cyFfmXybO4= 2023-05-31T20:21:04.600 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:21:04 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:m1lLyRoNmhbdXAFN+cyFfmXybO4=', 'Content-Length': '0'} 2023-05-31T20:21:04.601 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:18:05 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000004f0357b10cd843ce-006477ac30-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:21:04 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:21:04.602 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2023-05-31T20:21:04.602 DEBUG:boto:path=/pklxkd-48/obj1 2023-05-31T20:21:04.603 DEBUG:boto:auth_path=/pklxkd-48/obj1 2023-05-31T20:21:04.603 DEBUG:boto:path=/pklxkd-48/obj1?versionId=null 2023-05-31T20:21:04.603 DEBUG:boto:auth_path=/pklxkd-48/obj1?versionId=null 2023-05-31T20:21:04.604 DEBUG:boto:Method: GET 2023-05-31T20:21:04.604 DEBUG:boto:Path: /pklxkd-48/obj1?versionId=null 2023-05-31T20:21:04.605 DEBUG:boto:Data: 2023-05-31T20:21:04.605 DEBUG:boto:Headers: {} 2023-05-31T20:21:04.605 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:21:04.606 DEBUG:boto:Port: 8001 2023-05-31T20:21:04.606 DEBUG:boto:Params: {} 2023-05-31T20:21:04.606 DEBUG:boto:Token: None 2023-05-31T20:21:04.607 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:21:04 GMT /pklxkd-48/obj1?versionId=null 2023-05-31T20:21:04.607 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:x768p2y4SOQJNbPtU8L1DxC336I= 2023-05-31T20:21:04.608 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:21:04 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:x768p2y4SOQJNbPtU8L1DxC336I=', 'Content-Length': '0'} 2023-05-31T20:21:04.610 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:18:05 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000d9acc09652801ec5-006477ac30-162e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:21:04 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:21:04.652 DEBUG:boto:path=/pklxkd-48/obj1 2023-05-31T20:21:04.654 DEBUG:boto:auth_path=/pklxkd-48/obj1 2023-05-31T20:21:04.654 DEBUG:boto:path=/pklxkd-48/obj1?versionId=null 2023-05-31T20:21:04.655 DEBUG:boto:auth_path=/pklxkd-48/obj1?versionId=null 2023-05-31T20:21:04.655 DEBUG:boto:Method: GET 2023-05-31T20:21:04.655 DEBUG:boto:Path: /pklxkd-48/obj1?versionId=null 2023-05-31T20:21:04.656 DEBUG:boto:Data: 2023-05-31T20:21:04.656 DEBUG:boto:Headers: {} 2023-05-31T20:21:04.657 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:21:04.657 DEBUG:boto:Port: 8000 2023-05-31T20:21:04.657 DEBUG:boto:Params: {} 2023-05-31T20:21:04.658 DEBUG:boto:Token: None 2023-05-31T20:21:04.658 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:21:04 GMT /pklxkd-48/obj1?versionId=null 2023-05-31T20:21:04.659 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:x768p2y4SOQJNbPtU8L1DxC336I= 2023-05-31T20:21:04.659 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:21:04 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:x768p2y4SOQJNbPtU8L1DxC336I=', 'Content-Length': '0'} 2023-05-31T20:21:04.661 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:18:05 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000e48343eab9b65dd0-006477ac30-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:21:04 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:21:04.661 DEBUG:boto:path=/pklxkd-48/ 2023-05-31T20:21:04.662 DEBUG:boto:auth_path=/pklxkd-48/ 2023-05-31T20:21:04.662 DEBUG:boto:path=/pklxkd-48/?prefix=obj1 2023-05-31T20:21:04.663 DEBUG:boto:auth_path=/pklxkd-48/?prefix=obj1 2023-05-31T20:21:04.663 DEBUG:boto:Method: GET 2023-05-31T20:21:04.663 DEBUG:boto:Path: /pklxkd-48/?prefix=obj1 2023-05-31T20:21:04.664 DEBUG:boto:Data: 2023-05-31T20:21:04.665 DEBUG:boto:Headers: {} 2023-05-31T20:21:04.665 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:21:04.665 DEBUG:boto:Port: 8001 2023-05-31T20:21:04.666 DEBUG:boto:Params: {} 2023-05-31T20:21:04.666 DEBUG:boto:Token: None 2023-05-31T20:21:04.667 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:21:04 GMT /pklxkd-48/ 2023-05-31T20:21:04.667 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:EwC8smbxQxXdDXo+qklJ5ynQgVM= 2023-05-31T20:21:04.667 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:21:04 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:EwC8smbxQxXdDXo+qklJ5ynQgVM=', 'Content-Length': '0'} 2023-05-31T20:21:04.670 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000fa0fc534da7cf2f4-006477ac30-162e-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:21:04 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:21:04.670 DEBUG:boto:b'pklxkd-48obj11000falseobj12023-05-31T20:18:05.255Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:21:04.671 DEBUG:boto:path=/pklxkd-48/ 2023-05-31T20:21:04.671 DEBUG:boto:auth_path=/pklxkd-48/ 2023-05-31T20:21:04.672 DEBUG:boto:path=/pklxkd-48/?prefix=obj1 2023-05-31T20:21:04.672 DEBUG:boto:auth_path=/pklxkd-48/?prefix=obj1 2023-05-31T20:21:04.673 DEBUG:boto:Method: GET 2023-05-31T20:21:04.673 DEBUG:boto:Path: /pklxkd-48/?prefix=obj1 2023-05-31T20:21:04.673 DEBUG:boto:Data: 2023-05-31T20:21:04.674 DEBUG:boto:Headers: {} 2023-05-31T20:21:04.675 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:21:04.675 DEBUG:boto:Port: 8000 2023-05-31T20:21:04.675 DEBUG:boto:Params: {} 2023-05-31T20:21:04.676 DEBUG:boto:Token: None 2023-05-31T20:21:04.676 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:21:04 GMT /pklxkd-48/ 2023-05-31T20:21:04.677 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:EwC8smbxQxXdDXo+qklJ5ynQgVM= 2023-05-31T20:21:04.677 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:21:04 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:EwC8smbxQxXdDXo+qklJ5ynQgVM=', 'Content-Length': '0'} 2023-05-31T20:21:04.679 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000003864b8e057b90f89-006477ac30-1630-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:21:04 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:21:04.680 DEBUG:boto:b'pklxkd-48obj11000falseobj12023-05-31T20:18:05.255Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:21:04.681 DEBUG:boto:path=/pklxkd-48/obj1 2023-05-31T20:21:04.681 DEBUG:boto:auth_path=/pklxkd-48/obj1 2023-05-31T20:21:04.681 DEBUG:boto:Method: HEAD 2023-05-31T20:21:04.682 DEBUG:boto:Path: /pklxkd-48/obj1 2023-05-31T20:21:04.682 DEBUG:boto:Data: 2023-05-31T20:21:04.682 DEBUG:boto:Headers: {} 2023-05-31T20:21:04.683 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:21:04.683 DEBUG:boto:Port: 8001 2023-05-31T20:21:04.683 DEBUG:boto:Params: {} 2023-05-31T20:21:04.684 DEBUG:boto:Token: None 2023-05-31T20:21:04.684 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:21:04 GMT /pklxkd-48/obj1 2023-05-31T20:21:04.685 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:m5jkQTrA23sJTw4MV3Lx5NXktHw= 2023-05-31T20:21:04.686 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:21:04 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:m5jkQTrA23sJTw4MV3Lx5NXktHw=', 'Content-Length': '0'} 2023-05-31T20:21:04.687 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:18:05 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000006bffdab8a4e359da-006477ac30-162e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:21:04 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:21:04.688 DEBUG:boto:path=/pklxkd-48/obj1 2023-05-31T20:21:04.688 DEBUG:boto:auth_path=/pklxkd-48/obj1 2023-05-31T20:21:04.689 DEBUG:boto:Method: HEAD 2023-05-31T20:21:04.689 DEBUG:boto:Path: /pklxkd-48/obj1 2023-05-31T20:21:04.690 DEBUG:boto:Data: 2023-05-31T20:21:04.690 DEBUG:boto:Headers: {} 2023-05-31T20:21:04.690 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:21:04.691 DEBUG:boto:Port: 8000 2023-05-31T20:21:04.691 DEBUG:boto:Params: {} 2023-05-31T20:21:04.692 DEBUG:boto:Token: None 2023-05-31T20:21:04.692 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:21:04 GMT /pklxkd-48/obj1 2023-05-31T20:21:04.692 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:m5jkQTrA23sJTw4MV3Lx5NXktHw= 2023-05-31T20:21:04.693 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:21:04 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:m5jkQTrA23sJTw4MV3Lx5NXktHw=', 'Content-Length': '0'} 2023-05-31T20:21:04.694 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:18:05 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000cde34fda55954856-006477ac30-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:21:04 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:21:04.696 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2023-05-31T20:21:04.696 DEBUG:boto:path=/pklxkd-48/obj1 2023-05-31T20:21:04.696 DEBUG:boto:auth_path=/pklxkd-48/obj1 2023-05-31T20:21:04.697 DEBUG:boto:Method: GET 2023-05-31T20:21:04.697 DEBUG:boto:Path: /pklxkd-48/obj1 2023-05-31T20:21:04.698 DEBUG:boto:Data: 2023-05-31T20:21:04.698 DEBUG:boto:Headers: {} 2023-05-31T20:21:04.698 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:21:04.699 DEBUG:boto:Port: 8001 2023-05-31T20:21:04.699 DEBUG:boto:Params: {} 2023-05-31T20:21:04.699 DEBUG:boto:Token: None 2023-05-31T20:21:04.700 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:21:04 GMT /pklxkd-48/obj1 2023-05-31T20:21:04.700 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:f4xr/UZsfDsangT3dwaU3ALHByA= 2023-05-31T20:21:04.701 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:21:04 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:f4xr/UZsfDsangT3dwaU3ALHByA=', 'Content-Length': '0'} 2023-05-31T20:21:04.703 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:18:05 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000d2377bfd287fb63f-006477ac30-162e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:21:04 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:21:04.704 DEBUG:boto:path=/pklxkd-48/obj1 2023-05-31T20:21:04.704 DEBUG:boto:auth_path=/pklxkd-48/obj1 2023-05-31T20:21:04.704 DEBUG:boto:Method: GET 2023-05-31T20:21:04.705 DEBUG:boto:Path: /pklxkd-48/obj1 2023-05-31T20:21:04.706 DEBUG:boto:Data: 2023-05-31T20:21:04.706 DEBUG:boto:Headers: {} 2023-05-31T20:21:04.706 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:21:04.707 DEBUG:boto:Port: 8000 2023-05-31T20:21:04.707 DEBUG:boto:Params: {} 2023-05-31T20:21:04.708 DEBUG:boto:Token: None 2023-05-31T20:21:04.708 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:21:04 GMT /pklxkd-48/obj1 2023-05-31T20:21:04.709 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:f4xr/UZsfDsangT3dwaU3ALHByA= 2023-05-31T20:21:04.710 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:21:04 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:f4xr/UZsfDsangT3dwaU3ALHByA=', 'Content-Length': '0'} 2023-05-31T20:21:04.711 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:18:05 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000008c85795febb3916e-006477ac30-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:21:04 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:21:04.712 DEBUG:boto:path=/pklxkd-48/ 2023-05-31T20:21:04.712 DEBUG:boto:auth_path=/pklxkd-48/ 2023-05-31T20:21:04.713 DEBUG:boto:path=/pklxkd-48/?prefix=obj1 2023-05-31T20:21:04.713 DEBUG:boto:auth_path=/pklxkd-48/?prefix=obj1 2023-05-31T20:21:04.714 DEBUG:boto:Method: GET 2023-05-31T20:21:04.714 DEBUG:boto:Path: /pklxkd-48/?prefix=obj1 2023-05-31T20:21:04.714 DEBUG:boto:Data: 2023-05-31T20:21:04.715 DEBUG:boto:Headers: {} 2023-05-31T20:21:04.715 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:21:04.715 DEBUG:boto:Port: 8001 2023-05-31T20:21:04.716 DEBUG:boto:Params: {} 2023-05-31T20:21:04.716 DEBUG:boto:Token: None 2023-05-31T20:21:04.717 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:21:04 GMT /pklxkd-48/ 2023-05-31T20:21:04.717 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:EwC8smbxQxXdDXo+qklJ5ynQgVM= 2023-05-31T20:21:04.717 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:21:04 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:EwC8smbxQxXdDXo+qklJ5ynQgVM=', 'Content-Length': '0'} 2023-05-31T20:21:04.720 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000082448e923cc83d5d-006477ac30-162e-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:21:04 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:21:04.721 DEBUG:boto:b'pklxkd-48obj11000falseobj12023-05-31T20:18:05.255Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:21:04.722 DEBUG:boto:path=/pklxkd-48/ 2023-05-31T20:21:04.722 DEBUG:boto:auth_path=/pklxkd-48/ 2023-05-31T20:21:04.723 DEBUG:boto:path=/pklxkd-48/?prefix=obj1 2023-05-31T20:21:04.723 DEBUG:boto:auth_path=/pklxkd-48/?prefix=obj1 2023-05-31T20:21:04.723 DEBUG:boto:Method: GET 2023-05-31T20:21:04.724 DEBUG:boto:Path: /pklxkd-48/?prefix=obj1 2023-05-31T20:21:04.724 DEBUG:boto:Data: 2023-05-31T20:21:04.724 DEBUG:boto:Headers: {} 2023-05-31T20:21:04.725 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:21:04.725 DEBUG:boto:Port: 8000 2023-05-31T20:21:04.725 DEBUG:boto:Params: {} 2023-05-31T20:21:04.726 DEBUG:boto:Token: None 2023-05-31T20:21:04.726 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:21:04 GMT /pklxkd-48/ 2023-05-31T20:21:04.727 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:EwC8smbxQxXdDXo+qklJ5ynQgVM= 2023-05-31T20:21:04.727 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:21:04 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:EwC8smbxQxXdDXo+qklJ5ynQgVM=', 'Content-Length': '0'} 2023-05-31T20:21:04.730 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000bb6133966781ba65-006477ac30-1630-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:21:04 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:21:04.731 DEBUG:boto:b'pklxkd-48obj11000falseobj12023-05-31T20:18:05.255Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:21:04.731 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2023-05-31T20:21:04.732 DEBUG:boto:path=/pklxkd-48/obj2 2023-05-31T20:21:04.732 DEBUG:boto:auth_path=/pklxkd-48/obj2 2023-05-31T20:21:04.733 DEBUG:boto:path=/pklxkd-48/obj2?versionId=null 2023-05-31T20:21:04.733 DEBUG:boto:auth_path=/pklxkd-48/obj2?versionId=null 2023-05-31T20:21:04.733 DEBUG:boto:Method: GET 2023-05-31T20:21:04.734 DEBUG:boto:Path: /pklxkd-48/obj2?versionId=null 2023-05-31T20:21:04.734 DEBUG:boto:Data: 2023-05-31T20:21:04.734 DEBUG:boto:Headers: {} 2023-05-31T20:21:04.735 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:21:04.735 DEBUG:boto:Port: 8001 2023-05-31T20:21:04.736 DEBUG:boto:Params: {} 2023-05-31T20:21:04.736 DEBUG:boto:Token: None 2023-05-31T20:21:04.736 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:21:04 GMT /pklxkd-48/obj2?versionId=null 2023-05-31T20:21:04.737 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:0nuA1aRKvD0LAkqGK9H9r6jA9QU= 2023-05-31T20:21:04.737 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:21:04 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:0nuA1aRKvD0LAkqGK9H9r6jA9QU=', 'Content-Length': '0'} 2023-05-31T20:21:04.739 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:18:05 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000b885a3093911c7b5-006477ac30-162e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:21:04 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:21:04.740 DEBUG:boto:path=/pklxkd-48/obj2 2023-05-31T20:21:04.740 DEBUG:boto:auth_path=/pklxkd-48/obj2 2023-05-31T20:21:04.741 DEBUG:boto:path=/pklxkd-48/obj2?versionId=null 2023-05-31T20:21:04.741 DEBUG:boto:auth_path=/pklxkd-48/obj2?versionId=null 2023-05-31T20:21:04.742 DEBUG:boto:Method: GET 2023-05-31T20:21:04.742 DEBUG:boto:Path: /pklxkd-48/obj2?versionId=null 2023-05-31T20:21:04.742 DEBUG:boto:Data: 2023-05-31T20:21:04.743 DEBUG:boto:Headers: {} 2023-05-31T20:21:04.743 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:21:04.743 DEBUG:boto:Port: 8000 2023-05-31T20:21:04.744 DEBUG:boto:Params: {} 2023-05-31T20:21:04.744 DEBUG:boto:Token: None 2023-05-31T20:21:04.745 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:21:04 GMT /pklxkd-48/obj2?versionId=null 2023-05-31T20:21:04.745 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:0nuA1aRKvD0LAkqGK9H9r6jA9QU= 2023-05-31T20:21:04.745 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:21:04 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:0nuA1aRKvD0LAkqGK9H9r6jA9QU=', 'Content-Length': '0'} 2023-05-31T20:21:04.747 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:18:05 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000c45b880c99171303-006477ac30-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:21:04 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:21:04.748 DEBUG:boto:path=/pklxkd-48/obj2 2023-05-31T20:21:04.748 DEBUG:boto:auth_path=/pklxkd-48/obj2 2023-05-31T20:21:04.749 DEBUG:boto:path=/pklxkd-48/obj2?versionId=null 2023-05-31T20:21:04.749 DEBUG:boto:auth_path=/pklxkd-48/obj2?versionId=null 2023-05-31T20:21:04.749 DEBUG:boto:Method: HEAD 2023-05-31T20:21:04.750 DEBUG:boto:Path: /pklxkd-48/obj2?versionId=null 2023-05-31T20:21:04.750 DEBUG:boto:Data: 2023-05-31T20:21:04.750 DEBUG:boto:Headers: {} 2023-05-31T20:21:04.751 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:21:04.751 DEBUG:boto:Port: 8001 2023-05-31T20:21:04.751 DEBUG:boto:Params: {} 2023-05-31T20:21:04.752 DEBUG:boto:Token: None 2023-05-31T20:21:04.752 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:21:04 GMT /pklxkd-48/obj2?versionId=null 2023-05-31T20:21:04.753 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:yZtJ7ko67Bs6djywi6ScIk+fUMo= 2023-05-31T20:21:04.753 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:21:04 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:yZtJ7ko67Bs6djywi6ScIk+fUMo=', 'Content-Length': '0'} 2023-05-31T20:21:04.755 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:18:05 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000a6337c83e39c002d-006477ac30-162e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:21:04 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:21:04.755 DEBUG:boto:path=/pklxkd-48/obj2 2023-05-31T20:21:04.756 DEBUG:boto:auth_path=/pklxkd-48/obj2 2023-05-31T20:21:04.756 DEBUG:boto:path=/pklxkd-48/obj2?versionId=null 2023-05-31T20:21:04.757 DEBUG:boto:auth_path=/pklxkd-48/obj2?versionId=null 2023-05-31T20:21:04.757 DEBUG:boto:Method: HEAD 2023-05-31T20:21:04.757 DEBUG:boto:Path: /pklxkd-48/obj2?versionId=null 2023-05-31T20:21:04.758 DEBUG:boto:Data: 2023-05-31T20:21:04.758 DEBUG:boto:Headers: {} 2023-05-31T20:21:04.758 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:21:04.759 DEBUG:boto:Port: 8000 2023-05-31T20:21:04.759 DEBUG:boto:Params: {} 2023-05-31T20:21:04.760 DEBUG:boto:Token: None 2023-05-31T20:21:04.760 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:21:04 GMT /pklxkd-48/obj2?versionId=null 2023-05-31T20:21:04.760 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:yZtJ7ko67Bs6djywi6ScIk+fUMo= 2023-05-31T20:21:04.761 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:21:04 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:yZtJ7ko67Bs6djywi6ScIk+fUMo=', 'Content-Length': '0'} 2023-05-31T20:21:04.763 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:18:05 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000055ecf9869a31531c-006477ac30-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:21:04 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:21:04.763 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2023-05-31T20:21:04.764 DEBUG:boto:path=/pklxkd-48/obj2 2023-05-31T20:21:04.764 DEBUG:boto:auth_path=/pklxkd-48/obj2 2023-05-31T20:21:04.765 DEBUG:boto:path=/pklxkd-48/obj2?versionId=null 2023-05-31T20:21:04.765 DEBUG:boto:auth_path=/pklxkd-48/obj2?versionId=null 2023-05-31T20:21:04.765 DEBUG:boto:Method: GET 2023-05-31T20:21:04.766 DEBUG:boto:Path: /pklxkd-48/obj2?versionId=null 2023-05-31T20:21:04.766 DEBUG:boto:Data: 2023-05-31T20:21:04.766 DEBUG:boto:Headers: {} 2023-05-31T20:21:04.767 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:21:04.767 DEBUG:boto:Port: 8001 2023-05-31T20:21:04.767 DEBUG:boto:Params: {} 2023-05-31T20:21:04.768 DEBUG:boto:Token: None 2023-05-31T20:21:04.768 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:21:04 GMT /pklxkd-48/obj2?versionId=null 2023-05-31T20:21:04.769 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:0nuA1aRKvD0LAkqGK9H9r6jA9QU= 2023-05-31T20:21:04.769 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:21:04 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:0nuA1aRKvD0LAkqGK9H9r6jA9QU=', 'Content-Length': '0'} 2023-05-31T20:21:04.771 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:18:05 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000009d2673d21a649f16-006477ac30-162e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:21:04 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:21:04.772 DEBUG:boto:path=/pklxkd-48/obj2 2023-05-31T20:21:04.772 DEBUG:boto:auth_path=/pklxkd-48/obj2 2023-05-31T20:21:04.773 DEBUG:boto:path=/pklxkd-48/obj2?versionId=null 2023-05-31T20:21:04.773 DEBUG:boto:auth_path=/pklxkd-48/obj2?versionId=null 2023-05-31T20:21:04.773 DEBUG:boto:Method: GET 2023-05-31T20:21:04.774 DEBUG:boto:Path: /pklxkd-48/obj2?versionId=null 2023-05-31T20:21:04.774 DEBUG:boto:Data: 2023-05-31T20:21:04.775 DEBUG:boto:Headers: {} 2023-05-31T20:21:04.775 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:21:04.775 DEBUG:boto:Port: 8000 2023-05-31T20:21:04.776 DEBUG:boto:Params: {} 2023-05-31T20:21:04.776 DEBUG:boto:Token: None 2023-05-31T20:21:04.776 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:21:04 GMT /pklxkd-48/obj2?versionId=null 2023-05-31T20:21:04.777 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:0nuA1aRKvD0LAkqGK9H9r6jA9QU= 2023-05-31T20:21:04.777 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:21:04 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:0nuA1aRKvD0LAkqGK9H9r6jA9QU=', 'Content-Length': '0'} 2023-05-31T20:21:04.779 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:18:05 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000cd23cabf1d1bbb1e-006477ac30-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:21:04 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:21:04.780 DEBUG:boto:path=/pklxkd-48/ 2023-05-31T20:21:04.780 DEBUG:boto:auth_path=/pklxkd-48/ 2023-05-31T20:21:04.781 DEBUG:boto:path=/pklxkd-48/?prefix=obj2 2023-05-31T20:21:04.781 DEBUG:boto:auth_path=/pklxkd-48/?prefix=obj2 2023-05-31T20:21:04.781 DEBUG:boto:Method: GET 2023-05-31T20:21:04.782 DEBUG:boto:Path: /pklxkd-48/?prefix=obj2 2023-05-31T20:21:04.782 DEBUG:boto:Data: 2023-05-31T20:21:04.783 DEBUG:boto:Headers: {} 2023-05-31T20:21:04.783 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:21:04.783 DEBUG:boto:Port: 8001 2023-05-31T20:21:04.784 DEBUG:boto:Params: {} 2023-05-31T20:21:04.784 DEBUG:boto:Token: None 2023-05-31T20:21:04.785 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:21:04 GMT /pklxkd-48/ 2023-05-31T20:21:04.785 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:EwC8smbxQxXdDXo+qklJ5ynQgVM= 2023-05-31T20:21:04.785 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:21:04 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:EwC8smbxQxXdDXo+qklJ5ynQgVM=', 'Content-Length': '0'} 2023-05-31T20:21:04.788 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000019b3bcc7678f042e-006477ac30-162e-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:21:04 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:21:04.788 DEBUG:boto:b'pklxkd-48obj21000falseobj22023-05-31T20:18:05.277Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:21:04.789 DEBUG:boto:path=/pklxkd-48/ 2023-05-31T20:21:04.790 DEBUG:boto:auth_path=/pklxkd-48/ 2023-05-31T20:21:04.790 DEBUG:boto:path=/pklxkd-48/?prefix=obj2 2023-05-31T20:21:04.791 DEBUG:boto:auth_path=/pklxkd-48/?prefix=obj2 2023-05-31T20:21:04.791 DEBUG:boto:Method: GET 2023-05-31T20:21:04.792 DEBUG:boto:Path: /pklxkd-48/?prefix=obj2 2023-05-31T20:21:04.792 DEBUG:boto:Data: 2023-05-31T20:21:04.793 DEBUG:boto:Headers: {} 2023-05-31T20:21:04.793 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:21:04.793 DEBUG:boto:Port: 8000 2023-05-31T20:21:04.794 DEBUG:boto:Params: {} 2023-05-31T20:21:04.794 DEBUG:boto:Token: None 2023-05-31T20:21:04.794 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:21:04 GMT /pklxkd-48/ 2023-05-31T20:21:04.795 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:EwC8smbxQxXdDXo+qklJ5ynQgVM= 2023-05-31T20:21:04.795 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:21:04 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:EwC8smbxQxXdDXo+qklJ5ynQgVM=', 'Content-Length': '0'} 2023-05-31T20:21:04.798 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000029907ff575404b65-006477ac30-1630-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:21:04 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:21:04.798 DEBUG:boto:b'pklxkd-48obj21000falseobj22023-05-31T20:18:05.277Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:21:04.799 DEBUG:boto:path=/pklxkd-48/obj2 2023-05-31T20:21:04.799 DEBUG:boto:auth_path=/pklxkd-48/obj2 2023-05-31T20:21:04.800 DEBUG:boto:Method: HEAD 2023-05-31T20:21:04.800 DEBUG:boto:Path: /pklxkd-48/obj2 2023-05-31T20:21:04.800 DEBUG:boto:Data: 2023-05-31T20:21:04.801 DEBUG:boto:Headers: {} 2023-05-31T20:21:04.801 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:21:04.801 DEBUG:boto:Port: 8001 2023-05-31T20:21:04.802 DEBUG:boto:Params: {} 2023-05-31T20:21:04.802 DEBUG:boto:Token: None 2023-05-31T20:21:04.802 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:21:04 GMT /pklxkd-48/obj2 2023-05-31T20:21:04.803 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:sUQ40XukIBJbebb95ihIRbSGVxo= 2023-05-31T20:21:04.803 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:21:04 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:sUQ40XukIBJbebb95ihIRbSGVxo=', 'Content-Length': '0'} 2023-05-31T20:21:04.805 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:18:05 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000b47a7cb4593b2ffd-006477ac30-162e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:21:04 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:21:04.805 DEBUG:boto:path=/pklxkd-48/obj2 2023-05-31T20:21:04.806 DEBUG:boto:auth_path=/pklxkd-48/obj2 2023-05-31T20:21:04.806 DEBUG:boto:Method: HEAD 2023-05-31T20:21:04.807 DEBUG:boto:Path: /pklxkd-48/obj2 2023-05-31T20:21:04.807 DEBUG:boto:Data: 2023-05-31T20:21:04.807 DEBUG:boto:Headers: {} 2023-05-31T20:21:04.808 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:21:04.808 DEBUG:boto:Port: 8000 2023-05-31T20:21:04.808 DEBUG:boto:Params: {} 2023-05-31T20:21:04.809 DEBUG:boto:Token: None 2023-05-31T20:21:04.809 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:21:04 GMT /pklxkd-48/obj2 2023-05-31T20:21:04.809 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:sUQ40XukIBJbebb95ihIRbSGVxo= 2023-05-31T20:21:04.810 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:21:04 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:sUQ40XukIBJbebb95ihIRbSGVxo=', 'Content-Length': '0'} 2023-05-31T20:21:04.812 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:18:05 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000017658fd53bf62ddd-006477ac30-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:21:04 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:21:04.812 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2023-05-31T20:21:04.813 DEBUG:boto:path=/pklxkd-48/obj2 2023-05-31T20:21:04.813 DEBUG:boto:auth_path=/pklxkd-48/obj2 2023-05-31T20:21:04.814 DEBUG:boto:Method: GET 2023-05-31T20:21:04.814 DEBUG:boto:Path: /pklxkd-48/obj2 2023-05-31T20:21:04.814 DEBUG:boto:Data: 2023-05-31T20:21:04.815 DEBUG:boto:Headers: {} 2023-05-31T20:21:04.815 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:21:04.815 DEBUG:boto:Port: 8001 2023-05-31T20:21:04.816 DEBUG:boto:Params: {} 2023-05-31T20:21:04.816 DEBUG:boto:Token: None 2023-05-31T20:21:04.817 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:21:04 GMT /pklxkd-48/obj2 2023-05-31T20:21:04.817 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:hawowd97e3mCNvhskqhGR60i3fA= 2023-05-31T20:21:04.817 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:21:04 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:hawowd97e3mCNvhskqhGR60i3fA=', 'Content-Length': '0'} 2023-05-31T20:21:04.819 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:18:05 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000003e692e81dd74d7-006477ac30-162e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:21:04 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:21:04.820 DEBUG:boto:path=/pklxkd-48/obj2 2023-05-31T20:21:04.820 DEBUG:boto:auth_path=/pklxkd-48/obj2 2023-05-31T20:21:04.820 DEBUG:boto:Method: GET 2023-05-31T20:21:04.821 DEBUG:boto:Path: /pklxkd-48/obj2 2023-05-31T20:21:04.821 DEBUG:boto:Data: 2023-05-31T20:21:04.822 DEBUG:boto:Headers: {} 2023-05-31T20:21:04.822 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:21:04.822 DEBUG:boto:Port: 8000 2023-05-31T20:21:04.823 DEBUG:boto:Params: {} 2023-05-31T20:21:04.823 DEBUG:boto:Token: None 2023-05-31T20:21:04.823 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:21:04 GMT /pklxkd-48/obj2 2023-05-31T20:21:04.824 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:hawowd97e3mCNvhskqhGR60i3fA= 2023-05-31T20:21:04.824 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:21:04 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:hawowd97e3mCNvhskqhGR60i3fA=', 'Content-Length': '0'} 2023-05-31T20:21:04.826 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:18:05 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000fd3620dc647a2b1a-006477ac30-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:21:04 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:21:04.827 DEBUG:boto:path=/pklxkd-48/ 2023-05-31T20:21:04.827 DEBUG:boto:auth_path=/pklxkd-48/ 2023-05-31T20:21:04.827 DEBUG:boto:path=/pklxkd-48/?prefix=obj2 2023-05-31T20:21:04.828 DEBUG:boto:auth_path=/pklxkd-48/?prefix=obj2 2023-05-31T20:21:04.828 DEBUG:boto:Method: GET 2023-05-31T20:21:04.828 DEBUG:boto:Path: /pklxkd-48/?prefix=obj2 2023-05-31T20:21:04.829 DEBUG:boto:Data: 2023-05-31T20:21:04.829 DEBUG:boto:Headers: {} 2023-05-31T20:21:04.829 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:21:04.830 DEBUG:boto:Port: 8001 2023-05-31T20:21:04.830 DEBUG:boto:Params: {} 2023-05-31T20:21:04.831 DEBUG:boto:Token: None 2023-05-31T20:21:04.831 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:21:04 GMT /pklxkd-48/ 2023-05-31T20:21:04.831 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:EwC8smbxQxXdDXo+qklJ5ynQgVM= 2023-05-31T20:21:04.832 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:21:04 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:EwC8smbxQxXdDXo+qklJ5ynQgVM=', 'Content-Length': '0'} 2023-05-31T20:21:04.834 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000220f2a409b01de60-006477ac30-162e-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:21:04 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:21:04.835 DEBUG:boto:b'pklxkd-48obj21000falseobj22023-05-31T20:18:05.277Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:21:04.835 DEBUG:boto:path=/pklxkd-48/ 2023-05-31T20:21:04.836 DEBUG:boto:auth_path=/pklxkd-48/ 2023-05-31T20:21:04.836 DEBUG:boto:path=/pklxkd-48/?prefix=obj2 2023-05-31T20:21:04.836 DEBUG:boto:auth_path=/pklxkd-48/?prefix=obj2 2023-05-31T20:21:04.837 DEBUG:boto:Method: GET 2023-05-31T20:21:04.837 DEBUG:boto:Path: /pklxkd-48/?prefix=obj2 2023-05-31T20:21:04.838 DEBUG:boto:Data: 2023-05-31T20:21:04.838 DEBUG:boto:Headers: {} 2023-05-31T20:21:04.838 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:21:04.839 DEBUG:boto:Port: 8000 2023-05-31T20:21:04.839 DEBUG:boto:Params: {} 2023-05-31T20:21:04.839 DEBUG:boto:Token: None 2023-05-31T20:21:04.840 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:21:04 GMT /pklxkd-48/ 2023-05-31T20:21:04.840 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:EwC8smbxQxXdDXo+qklJ5ynQgVM= 2023-05-31T20:21:04.841 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:21:04 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:EwC8smbxQxXdDXo+qklJ5ynQgVM=', 'Content-Length': '0'} 2023-05-31T20:21:04.843 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a020c3a68d997311-006477ac30-1630-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:21:04 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:21:04.843 DEBUG:boto:b'pklxkd-48obj21000falseobj22023-05-31T20:18:05.277Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:21:04.844 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2023-05-31T20:21:04.844 DEBUG:boto:path=/pklxkd-48/obj3 2023-05-31T20:21:04.845 DEBUG:boto:auth_path=/pklxkd-48/obj3 2023-05-31T20:21:04.845 DEBUG:boto:path=/pklxkd-48/obj3?versionId=null 2023-05-31T20:21:04.845 DEBUG:boto:auth_path=/pklxkd-48/obj3?versionId=null 2023-05-31T20:21:04.846 DEBUG:boto:Method: GET 2023-05-31T20:21:04.846 DEBUG:boto:Path: /pklxkd-48/obj3?versionId=null 2023-05-31T20:21:04.847 DEBUG:boto:Data: 2023-05-31T20:21:04.847 DEBUG:boto:Headers: {} 2023-05-31T20:21:04.847 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:21:04.848 DEBUG:boto:Port: 8001 2023-05-31T20:21:04.848 DEBUG:boto:Params: {} 2023-05-31T20:21:04.848 DEBUG:boto:Token: None 2023-05-31T20:21:04.849 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:21:04 GMT /pklxkd-48/obj3?versionId=null 2023-05-31T20:21:04.849 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:MF4nbExqkDzD61Q97u3KuYKlAEE= 2023-05-31T20:21:04.850 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:21:04 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:MF4nbExqkDzD61Q97u3KuYKlAEE=', 'Content-Length': '0'} 2023-05-31T20:21:04.852 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:18:05 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000df0ca82cf3bb1949-006477ac30-162e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:21:04 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:21:04.852 DEBUG:boto:path=/pklxkd-48/obj3 2023-05-31T20:21:04.853 DEBUG:boto:auth_path=/pklxkd-48/obj3 2023-05-31T20:21:04.853 DEBUG:boto:path=/pklxkd-48/obj3?versionId=null 2023-05-31T20:21:04.853 DEBUG:boto:auth_path=/pklxkd-48/obj3?versionId=null 2023-05-31T20:21:04.854 DEBUG:boto:Method: GET 2023-05-31T20:21:04.854 DEBUG:boto:Path: /pklxkd-48/obj3?versionId=null 2023-05-31T20:21:04.855 DEBUG:boto:Data: 2023-05-31T20:21:04.855 DEBUG:boto:Headers: {} 2023-05-31T20:21:04.855 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:21:04.856 DEBUG:boto:Port: 8000 2023-05-31T20:21:04.856 DEBUG:boto:Params: {} 2023-05-31T20:21:04.856 DEBUG:boto:Token: None 2023-05-31T20:21:04.857 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:21:04 GMT /pklxkd-48/obj3?versionId=null 2023-05-31T20:21:04.857 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:MF4nbExqkDzD61Q97u3KuYKlAEE= 2023-05-31T20:21:04.858 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:21:04 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:MF4nbExqkDzD61Q97u3KuYKlAEE=', 'Content-Length': '0'} 2023-05-31T20:21:04.859 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:18:05 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000009142d976329c9ea6-006477ac30-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:21:04 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:21:04.860 DEBUG:boto:path=/pklxkd-48/obj3 2023-05-31T20:21:04.860 DEBUG:boto:auth_path=/pklxkd-48/obj3 2023-05-31T20:21:04.861 DEBUG:boto:path=/pklxkd-48/obj3?versionId=null 2023-05-31T20:21:04.861 DEBUG:boto:auth_path=/pklxkd-48/obj3?versionId=null 2023-05-31T20:21:04.861 DEBUG:boto:Method: HEAD 2023-05-31T20:21:04.862 DEBUG:boto:Path: /pklxkd-48/obj3?versionId=null 2023-05-31T20:21:04.862 DEBUG:boto:Data: 2023-05-31T20:21:04.862 DEBUG:boto:Headers: {} 2023-05-31T20:21:04.863 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:21:04.863 DEBUG:boto:Port: 8001 2023-05-31T20:21:04.863 DEBUG:boto:Params: {} 2023-05-31T20:21:04.864 DEBUG:boto:Token: None 2023-05-31T20:21:04.864 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:21:04 GMT /pklxkd-48/obj3?versionId=null 2023-05-31T20:21:04.865 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:iYuFiake20zf9xku1jYl9tnp5xc= 2023-05-31T20:21:04.865 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:21:04 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:iYuFiake20zf9xku1jYl9tnp5xc=', 'Content-Length': '0'} 2023-05-31T20:21:04.867 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:18:05 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000cd1ac5167c9adee0-006477ac30-162e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:21:04 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:21:04.868 DEBUG:boto:path=/pklxkd-48/obj3 2023-05-31T20:21:04.868 DEBUG:boto:auth_path=/pklxkd-48/obj3 2023-05-31T20:21:04.868 DEBUG:boto:path=/pklxkd-48/obj3?versionId=null 2023-05-31T20:21:04.869 DEBUG:boto:auth_path=/pklxkd-48/obj3?versionId=null 2023-05-31T20:21:04.869 DEBUG:boto:Method: HEAD 2023-05-31T20:21:04.869 DEBUG:boto:Path: /pklxkd-48/obj3?versionId=null 2023-05-31T20:21:04.870 DEBUG:boto:Data: 2023-05-31T20:21:04.870 DEBUG:boto:Headers: {} 2023-05-31T20:21:04.870 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:21:04.871 DEBUG:boto:Port: 8000 2023-05-31T20:21:04.871 DEBUG:boto:Params: {} 2023-05-31T20:21:04.872 DEBUG:boto:Token: None 2023-05-31T20:21:04.872 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:21:04 GMT /pklxkd-48/obj3?versionId=null 2023-05-31T20:21:04.872 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:iYuFiake20zf9xku1jYl9tnp5xc= 2023-05-31T20:21:04.873 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:21:04 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:iYuFiake20zf9xku1jYl9tnp5xc=', 'Content-Length': '0'} 2023-05-31T20:21:04.874 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:18:05 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000004cfd1e6147b7c09a-006477ac30-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:21:04 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:21:04.875 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2023-05-31T20:21:04.875 DEBUG:boto:path=/pklxkd-48/obj3 2023-05-31T20:21:04.876 DEBUG:boto:auth_path=/pklxkd-48/obj3 2023-05-31T20:21:04.876 DEBUG:boto:path=/pklxkd-48/obj3?versionId=null 2023-05-31T20:21:04.876 DEBUG:boto:auth_path=/pklxkd-48/obj3?versionId=null 2023-05-31T20:21:04.877 DEBUG:boto:Method: GET 2023-05-31T20:21:04.877 DEBUG:boto:Path: /pklxkd-48/obj3?versionId=null 2023-05-31T20:21:04.878 DEBUG:boto:Data: 2023-05-31T20:21:04.878 DEBUG:boto:Headers: {} 2023-05-31T20:21:04.878 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:21:04.879 DEBUG:boto:Port: 8001 2023-05-31T20:21:04.879 DEBUG:boto:Params: {} 2023-05-31T20:21:04.879 DEBUG:boto:Token: None 2023-05-31T20:21:04.880 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:21:04 GMT /pklxkd-48/obj3?versionId=null 2023-05-31T20:21:04.880 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:MF4nbExqkDzD61Q97u3KuYKlAEE= 2023-05-31T20:21:04.881 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:21:04 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:MF4nbExqkDzD61Q97u3KuYKlAEE=', 'Content-Length': '0'} 2023-05-31T20:21:04.882 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:18:05 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000dbc98a0629e6ab50-006477ac30-162e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:21:04 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:21:04.883 DEBUG:boto:path=/pklxkd-48/obj3 2023-05-31T20:21:04.883 DEBUG:boto:auth_path=/pklxkd-48/obj3 2023-05-31T20:21:04.884 DEBUG:boto:path=/pklxkd-48/obj3?versionId=null 2023-05-31T20:21:04.884 DEBUG:boto:auth_path=/pklxkd-48/obj3?versionId=null 2023-05-31T20:21:04.884 DEBUG:boto:Method: GET 2023-05-31T20:21:04.885 DEBUG:boto:Path: /pklxkd-48/obj3?versionId=null 2023-05-31T20:21:04.885 DEBUG:boto:Data: 2023-05-31T20:21:04.886 DEBUG:boto:Headers: {} 2023-05-31T20:21:04.886 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:21:04.886 DEBUG:boto:Port: 8000 2023-05-31T20:21:04.887 DEBUG:boto:Params: {} 2023-05-31T20:21:04.887 DEBUG:boto:Token: None 2023-05-31T20:21:04.887 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:21:04 GMT /pklxkd-48/obj3?versionId=null 2023-05-31T20:21:04.888 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:MF4nbExqkDzD61Q97u3KuYKlAEE= 2023-05-31T20:21:04.888 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:21:04 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:MF4nbExqkDzD61Q97u3KuYKlAEE=', 'Content-Length': '0'} 2023-05-31T20:21:04.890 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:18:05 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000ebb9d26bba679980-006477ac30-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:21:04 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:21:04.891 DEBUG:boto:path=/pklxkd-48/ 2023-05-31T20:21:04.891 DEBUG:boto:auth_path=/pklxkd-48/ 2023-05-31T20:21:04.891 DEBUG:boto:path=/pklxkd-48/?prefix=obj3 2023-05-31T20:21:04.892 DEBUG:boto:auth_path=/pklxkd-48/?prefix=obj3 2023-05-31T20:21:04.892 DEBUG:boto:Method: GET 2023-05-31T20:21:04.893 DEBUG:boto:Path: /pklxkd-48/?prefix=obj3 2023-05-31T20:21:04.893 DEBUG:boto:Data: 2023-05-31T20:21:04.893 DEBUG:boto:Headers: {} 2023-05-31T20:21:04.894 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:21:04.894 DEBUG:boto:Port: 8001 2023-05-31T20:21:04.894 DEBUG:boto:Params: {} 2023-05-31T20:21:04.895 DEBUG:boto:Token: None 2023-05-31T20:21:04.895 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:21:04 GMT /pklxkd-48/ 2023-05-31T20:21:04.896 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:EwC8smbxQxXdDXo+qklJ5ynQgVM= 2023-05-31T20:21:04.896 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:21:04 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:EwC8smbxQxXdDXo+qklJ5ynQgVM=', 'Content-Length': '0'} 2023-05-31T20:21:04.898 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000009c137e60975096ad-006477ac30-162e-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:21:04 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:21:04.899 DEBUG:boto:b'pklxkd-48obj31000falseobj32023-05-31T20:18:05.338Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:21:04.900 DEBUG:boto:path=/pklxkd-48/ 2023-05-31T20:21:04.900 DEBUG:boto:auth_path=/pklxkd-48/ 2023-05-31T20:21:04.900 DEBUG:boto:path=/pklxkd-48/?prefix=obj3 2023-05-31T20:21:04.901 DEBUG:boto:auth_path=/pklxkd-48/?prefix=obj3 2023-05-31T20:21:04.901 DEBUG:boto:Method: GET 2023-05-31T20:21:04.901 DEBUG:boto:Path: /pklxkd-48/?prefix=obj3 2023-05-31T20:21:04.902 DEBUG:boto:Data: 2023-05-31T20:21:04.902 DEBUG:boto:Headers: {} 2023-05-31T20:21:04.903 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:21:04.903 DEBUG:boto:Port: 8000 2023-05-31T20:21:04.903 DEBUG:boto:Params: {} 2023-05-31T20:21:04.904 DEBUG:boto:Token: None 2023-05-31T20:21:04.904 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:21:04 GMT /pklxkd-48/ 2023-05-31T20:21:04.905 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:EwC8smbxQxXdDXo+qklJ5ynQgVM= 2023-05-31T20:21:04.905 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:21:04 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:EwC8smbxQxXdDXo+qklJ5ynQgVM=', 'Content-Length': '0'} 2023-05-31T20:21:04.907 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000020935cc4d3134b4-006477ac30-1630-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:21:04 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:21:04.908 DEBUG:boto:b'pklxkd-48obj31000falseobj32023-05-31T20:18:05.338Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:21:04.908 DEBUG:boto:path=/pklxkd-48/obj3 2023-05-31T20:21:04.909 DEBUG:boto:auth_path=/pklxkd-48/obj3 2023-05-31T20:21:04.909 DEBUG:boto:Method: HEAD 2023-05-31T20:21:04.909 DEBUG:boto:Path: /pklxkd-48/obj3 2023-05-31T20:21:04.910 DEBUG:boto:Data: 2023-05-31T20:21:04.910 DEBUG:boto:Headers: {} 2023-05-31T20:21:04.910 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:21:04.911 DEBUG:boto:Port: 8001 2023-05-31T20:21:04.911 DEBUG:boto:Params: {} 2023-05-31T20:21:04.911 DEBUG:boto:Token: None 2023-05-31T20:21:04.912 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:21:04 GMT /pklxkd-48/obj3 2023-05-31T20:21:04.912 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:HtCBvXbvsutE/d52Bf6BbkwmYhA= 2023-05-31T20:21:04.913 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:21:04 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:HtCBvXbvsutE/d52Bf6BbkwmYhA=', 'Content-Length': '0'} 2023-05-31T20:21:04.914 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:18:05 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000094155705b9b59d2c-006477ac30-162e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:21:04 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:21:04.915 DEBUG:boto:path=/pklxkd-48/obj3 2023-05-31T20:21:04.915 DEBUG:boto:auth_path=/pklxkd-48/obj3 2023-05-31T20:21:04.916 DEBUG:boto:Method: HEAD 2023-05-31T20:21:04.916 DEBUG:boto:Path: /pklxkd-48/obj3 2023-05-31T20:21:04.916 DEBUG:boto:Data: 2023-05-31T20:21:04.917 DEBUG:boto:Headers: {} 2023-05-31T20:21:04.917 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:21:04.917 DEBUG:boto:Port: 8000 2023-05-31T20:21:04.918 DEBUG:boto:Params: {} 2023-05-31T20:21:04.918 DEBUG:boto:Token: None 2023-05-31T20:21:04.918 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:21:04 GMT /pklxkd-48/obj3 2023-05-31T20:21:04.919 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:HtCBvXbvsutE/d52Bf6BbkwmYhA= 2023-05-31T20:21:04.919 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:21:04 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:HtCBvXbvsutE/d52Bf6BbkwmYhA=', 'Content-Length': '0'} 2023-05-31T20:21:04.921 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:18:05 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000f14f9150785ffc6d-006477ac30-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:21:04 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:21:04.921 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2023-05-31T20:21:04.922 DEBUG:boto:path=/pklxkd-48/obj3 2023-05-31T20:21:04.922 DEBUG:boto:auth_path=/pklxkd-48/obj3 2023-05-31T20:21:04.923 DEBUG:boto:Method: GET 2023-05-31T20:21:04.923 DEBUG:boto:Path: /pklxkd-48/obj3 2023-05-31T20:21:04.923 DEBUG:boto:Data: 2023-05-31T20:21:04.924 DEBUG:boto:Headers: {} 2023-05-31T20:21:04.924 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:21:04.924 DEBUG:boto:Port: 8001 2023-05-31T20:21:04.925 DEBUG:boto:Params: {} 2023-05-31T20:21:04.925 DEBUG:boto:Token: None 2023-05-31T20:21:04.926 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:21:04 GMT /pklxkd-48/obj3 2023-05-31T20:21:04.926 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:+zZNFhVKYd38evzszD/xO2B0khg= 2023-05-31T20:21:04.926 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:21:04 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:+zZNFhVKYd38evzszD/xO2B0khg=', 'Content-Length': '0'} 2023-05-31T20:21:04.928 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:18:05 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000079cd71b556948425-006477ac30-162e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:21:04 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:21:04.929 DEBUG:boto:path=/pklxkd-48/obj3 2023-05-31T20:21:04.929 DEBUG:boto:auth_path=/pklxkd-48/obj3 2023-05-31T20:21:04.930 DEBUG:boto:Method: GET 2023-05-31T20:21:04.930 DEBUG:boto:Path: /pklxkd-48/obj3 2023-05-31T20:21:04.930 DEBUG:boto:Data: 2023-05-31T20:21:04.931 DEBUG:boto:Headers: {} 2023-05-31T20:21:04.931 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:21:04.931 DEBUG:boto:Port: 8000 2023-05-31T20:21:04.932 DEBUG:boto:Params: {} 2023-05-31T20:21:04.932 DEBUG:boto:Token: None 2023-05-31T20:21:04.933 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:21:04 GMT /pklxkd-48/obj3 2023-05-31T20:21:04.933 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:+zZNFhVKYd38evzszD/xO2B0khg= 2023-05-31T20:21:04.934 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:21:04 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:+zZNFhVKYd38evzszD/xO2B0khg=', 'Content-Length': '0'} 2023-05-31T20:21:04.935 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:18:05 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000df5e5bf95a5ab83a-006477ac30-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:21:04 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:21:04.976 DEBUG:boto:path=/pklxkd-48/ 2023-05-31T20:21:04.977 DEBUG:boto:auth_path=/pklxkd-48/ 2023-05-31T20:21:04.977 DEBUG:boto:path=/pklxkd-48/?prefix=obj3 2023-05-31T20:21:04.978 DEBUG:boto:auth_path=/pklxkd-48/?prefix=obj3 2023-05-31T20:21:04.978 DEBUG:boto:Method: GET 2023-05-31T20:21:04.978 DEBUG:boto:Path: /pklxkd-48/?prefix=obj3 2023-05-31T20:21:04.979 DEBUG:boto:Data: 2023-05-31T20:21:04.979 DEBUG:boto:Headers: {} 2023-05-31T20:21:04.979 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:21:04.980 DEBUG:boto:Port: 8001 2023-05-31T20:21:04.980 DEBUG:boto:Params: {} 2023-05-31T20:21:04.981 DEBUG:boto:Token: None 2023-05-31T20:21:04.981 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:21:04 GMT /pklxkd-48/ 2023-05-31T20:21:04.982 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:EwC8smbxQxXdDXo+qklJ5ynQgVM= 2023-05-31T20:21:04.982 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:21:04 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:EwC8smbxQxXdDXo+qklJ5ynQgVM=', 'Content-Length': '0'} 2023-05-31T20:21:04.984 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000370ac8d96de61720-006477ac30-162e-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:21:04 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:21:04.985 DEBUG:boto:b'pklxkd-48obj31000falseobj32023-05-31T20:18:05.338Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:21:04.986 DEBUG:boto:path=/pklxkd-48/ 2023-05-31T20:21:04.986 DEBUG:boto:auth_path=/pklxkd-48/ 2023-05-31T20:21:04.987 DEBUG:boto:path=/pklxkd-48/?prefix=obj3 2023-05-31T20:21:04.987 DEBUG:boto:auth_path=/pklxkd-48/?prefix=obj3 2023-05-31T20:21:04.987 DEBUG:boto:Method: GET 2023-05-31T20:21:04.988 DEBUG:boto:Path: /pklxkd-48/?prefix=obj3 2023-05-31T20:21:04.988 DEBUG:boto:Data: 2023-05-31T20:21:04.988 DEBUG:boto:Headers: {} 2023-05-31T20:21:04.989 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:21:04.989 DEBUG:boto:Port: 8000 2023-05-31T20:21:04.989 DEBUG:boto:Params: {} 2023-05-31T20:21:04.990 DEBUG:boto:Token: None 2023-05-31T20:21:04.990 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:21:04 GMT /pklxkd-48/ 2023-05-31T20:21:04.991 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:EwC8smbxQxXdDXo+qklJ5ynQgVM= 2023-05-31T20:21:04.991 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:21:04 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:EwC8smbxQxXdDXo+qklJ5ynQgVM=', 'Content-Length': '0'} 2023-05-31T20:21:04.994 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000077d3c072feb23d65-006477ac30-1630-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:21:04 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:21:04.995 DEBUG:boto:b'pklxkd-48obj31000falseobj32023-05-31T20:18:05.338Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:21:04.996 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2023-05-31T20:21:04.996 DEBUG:boto:path=/pklxkd-48/obj4 2023-05-31T20:21:04.997 DEBUG:boto:auth_path=/pklxkd-48/obj4 2023-05-31T20:21:04.997 DEBUG:boto:path=/pklxkd-48/obj4?versionId=null 2023-05-31T20:21:04.997 DEBUG:boto:auth_path=/pklxkd-48/obj4?versionId=null 2023-05-31T20:21:04.998 DEBUG:boto:Method: GET 2023-05-31T20:21:04.998 DEBUG:boto:Path: /pklxkd-48/obj4?versionId=null 2023-05-31T20:21:04.998 DEBUG:boto:Data: 2023-05-31T20:21:04.999 DEBUG:boto:Headers: {} 2023-05-31T20:21:04.999 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:21:05.001 DEBUG:boto:Port: 8001 2023-05-31T20:21:05.002 DEBUG:boto:Params: {} 2023-05-31T20:21:05.002 DEBUG:boto:Token: None 2023-05-31T20:21:05.003 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:21:05 GMT /pklxkd-48/obj4?versionId=null 2023-05-31T20:21:05.003 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:GK9om6ytzGDE4Hs9vnEbo1u8pPA= 2023-05-31T20:21:05.004 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:21:05 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:GK9om6ytzGDE4Hs9vnEbo1u8pPA=', 'Content-Length': '0'} 2023-05-31T20:21:05.006 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:18:05 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000056b05f124fa3525d-006477ac31-162e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:21:05 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:21:05.050 DEBUG:boto:path=/pklxkd-48/obj4 2023-05-31T20:21:05.051 DEBUG:boto:auth_path=/pklxkd-48/obj4 2023-05-31T20:21:05.052 DEBUG:boto:path=/pklxkd-48/obj4?versionId=null 2023-05-31T20:21:05.052 DEBUG:boto:auth_path=/pklxkd-48/obj4?versionId=null 2023-05-31T20:21:05.053 DEBUG:boto:Method: GET 2023-05-31T20:21:05.053 DEBUG:boto:Path: /pklxkd-48/obj4?versionId=null 2023-05-31T20:21:05.054 DEBUG:boto:Data: 2023-05-31T20:21:05.055 DEBUG:boto:Headers: {} 2023-05-31T20:21:05.055 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:21:05.056 DEBUG:boto:Port: 8000 2023-05-31T20:21:05.056 DEBUG:boto:Params: {} 2023-05-31T20:21:05.057 DEBUG:boto:Token: None 2023-05-31T20:21:05.057 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:21:05 GMT /pklxkd-48/obj4?versionId=null 2023-05-31T20:21:05.058 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:GK9om6ytzGDE4Hs9vnEbo1u8pPA= 2023-05-31T20:21:05.059 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:21:05 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:GK9om6ytzGDE4Hs9vnEbo1u8pPA=', 'Content-Length': '0'} 2023-05-31T20:21:05.065 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:18:05 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000699e52f68c39d355-006477ac31-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:21:05 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:21:05.066 DEBUG:boto:path=/pklxkd-48/obj4 2023-05-31T20:21:05.067 DEBUG:boto:auth_path=/pklxkd-48/obj4 2023-05-31T20:21:05.067 DEBUG:boto:path=/pklxkd-48/obj4?versionId=null 2023-05-31T20:21:05.068 DEBUG:boto:auth_path=/pklxkd-48/obj4?versionId=null 2023-05-31T20:21:05.068 DEBUG:boto:Method: HEAD 2023-05-31T20:21:05.069 DEBUG:boto:Path: /pklxkd-48/obj4?versionId=null 2023-05-31T20:21:05.069 DEBUG:boto:Data: 2023-05-31T20:21:05.069 DEBUG:boto:Headers: {} 2023-05-31T20:21:05.070 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:21:05.070 DEBUG:boto:Port: 8001 2023-05-31T20:21:05.070 DEBUG:boto:Params: {} 2023-05-31T20:21:05.071 DEBUG:boto:Token: None 2023-05-31T20:21:05.071 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:21:05 GMT /pklxkd-48/obj4?versionId=null 2023-05-31T20:21:05.071 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:FYffMfumDIhfwZU1yB4yZ+cEtWI= 2023-05-31T20:21:05.072 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:21:05 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:FYffMfumDIhfwZU1yB4yZ+cEtWI=', 'Content-Length': '0'} 2023-05-31T20:21:05.076 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:18:05 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000626d36ca569ce197-006477ac31-162e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:21:05 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:21:05.077 DEBUG:boto:path=/pklxkd-48/obj4 2023-05-31T20:21:05.078 DEBUG:boto:auth_path=/pklxkd-48/obj4 2023-05-31T20:21:05.078 DEBUG:boto:path=/pklxkd-48/obj4?versionId=null 2023-05-31T20:21:05.078 DEBUG:boto:auth_path=/pklxkd-48/obj4?versionId=null 2023-05-31T20:21:05.079 DEBUG:boto:Method: HEAD 2023-05-31T20:21:05.079 DEBUG:boto:Path: /pklxkd-48/obj4?versionId=null 2023-05-31T20:21:05.079 DEBUG:boto:Data: 2023-05-31T20:21:05.080 DEBUG:boto:Headers: {} 2023-05-31T20:21:05.080 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:21:05.080 DEBUG:boto:Port: 8000 2023-05-31T20:21:05.081 DEBUG:boto:Params: {} 2023-05-31T20:21:05.081 DEBUG:boto:Token: None 2023-05-31T20:21:05.082 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:21:05 GMT /pklxkd-48/obj4?versionId=null 2023-05-31T20:21:05.082 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:FYffMfumDIhfwZU1yB4yZ+cEtWI= 2023-05-31T20:21:05.082 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:21:05 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:FYffMfumDIhfwZU1yB4yZ+cEtWI=', 'Content-Length': '0'} 2023-05-31T20:21:05.088 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:18:05 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000102a03582bdb51aa-006477ac31-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:21:05 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:21:05.089 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2023-05-31T20:21:05.089 DEBUG:boto:path=/pklxkd-48/obj4 2023-05-31T20:21:05.090 DEBUG:boto:auth_path=/pklxkd-48/obj4 2023-05-31T20:21:05.090 DEBUG:boto:path=/pklxkd-48/obj4?versionId=null 2023-05-31T20:21:05.090 DEBUG:boto:auth_path=/pklxkd-48/obj4?versionId=null 2023-05-31T20:21:05.091 DEBUG:boto:Method: GET 2023-05-31T20:21:05.091 DEBUG:boto:Path: /pklxkd-48/obj4?versionId=null 2023-05-31T20:21:05.091 DEBUG:boto:Data: 2023-05-31T20:21:05.092 DEBUG:boto:Headers: {} 2023-05-31T20:21:05.092 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:21:05.093 DEBUG:boto:Port: 8001 2023-05-31T20:21:05.093 DEBUG:boto:Params: {} 2023-05-31T20:21:05.093 DEBUG:boto:Token: None 2023-05-31T20:21:05.094 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:21:05 GMT /pklxkd-48/obj4?versionId=null 2023-05-31T20:21:05.095 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:GK9om6ytzGDE4Hs9vnEbo1u8pPA= 2023-05-31T20:21:05.095 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:21:05 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:GK9om6ytzGDE4Hs9vnEbo1u8pPA=', 'Content-Length': '0'} 2023-05-31T20:21:05.097 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:18:05 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000bd9ff4692a5172b1-006477ac31-162e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:21:05 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:21:05.099 DEBUG:boto:path=/pklxkd-48/obj4 2023-05-31T20:21:05.099 DEBUG:boto:auth_path=/pklxkd-48/obj4 2023-05-31T20:21:05.100 DEBUG:boto:path=/pklxkd-48/obj4?versionId=null 2023-05-31T20:21:05.100 DEBUG:boto:auth_path=/pklxkd-48/obj4?versionId=null 2023-05-31T20:21:05.100 DEBUG:boto:Method: GET 2023-05-31T20:21:05.101 DEBUG:boto:Path: /pklxkd-48/obj4?versionId=null 2023-05-31T20:21:05.101 DEBUG:boto:Data: 2023-05-31T20:21:05.102 DEBUG:boto:Headers: {} 2023-05-31T20:21:05.102 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:21:05.102 DEBUG:boto:Port: 8000 2023-05-31T20:21:05.103 DEBUG:boto:Params: {} 2023-05-31T20:21:05.103 DEBUG:boto:Token: None 2023-05-31T20:21:05.103 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:21:05 GMT /pklxkd-48/obj4?versionId=null 2023-05-31T20:21:05.104 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:GK9om6ytzGDE4Hs9vnEbo1u8pPA= 2023-05-31T20:21:05.104 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:21:05 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:GK9om6ytzGDE4Hs9vnEbo1u8pPA=', 'Content-Length': '0'} 2023-05-31T20:21:05.108 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:18:05 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000058f6784d56cfd0c0-006477ac31-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:21:05 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:21:05.150 DEBUG:boto:path=/pklxkd-48/ 2023-05-31T20:21:05.151 DEBUG:boto:auth_path=/pklxkd-48/ 2023-05-31T20:21:05.151 DEBUG:boto:path=/pklxkd-48/?prefix=obj4 2023-05-31T20:21:05.152 DEBUG:boto:auth_path=/pklxkd-48/?prefix=obj4 2023-05-31T20:21:05.153 DEBUG:boto:Method: GET 2023-05-31T20:21:05.154 DEBUG:boto:Path: /pklxkd-48/?prefix=obj4 2023-05-31T20:21:05.155 DEBUG:boto:Data: 2023-05-31T20:21:05.156 DEBUG:boto:Headers: {} 2023-05-31T20:21:05.156 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:21:05.157 DEBUG:boto:Port: 8001 2023-05-31T20:21:05.157 DEBUG:boto:Params: {} 2023-05-31T20:21:05.158 DEBUG:boto:Token: None 2023-05-31T20:21:05.158 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:21:05 GMT /pklxkd-48/ 2023-05-31T20:21:05.159 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:3ZoBJqr7pRMjTmfx7u+QyvQHg74= 2023-05-31T20:21:05.159 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:21:05 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:3ZoBJqr7pRMjTmfx7u+QyvQHg74=', 'Content-Length': '0'} 2023-05-31T20:21:05.163 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000030e7a05129fe7fdf-006477ac31-162e-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:21:05 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:21:05.163 DEBUG:boto:b'pklxkd-48obj41000falseobj42023-05-31T20:18:05.356Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:21:05.164 DEBUG:boto:path=/pklxkd-48/ 2023-05-31T20:21:05.165 DEBUG:boto:auth_path=/pklxkd-48/ 2023-05-31T20:21:05.165 DEBUG:boto:path=/pklxkd-48/?prefix=obj4 2023-05-31T20:21:05.166 DEBUG:boto:auth_path=/pklxkd-48/?prefix=obj4 2023-05-31T20:21:05.166 DEBUG:boto:Method: GET 2023-05-31T20:21:05.166 DEBUG:boto:Path: /pklxkd-48/?prefix=obj4 2023-05-31T20:21:05.167 DEBUG:boto:Data: 2023-05-31T20:21:05.167 DEBUG:boto:Headers: {} 2023-05-31T20:21:05.168 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:21:05.168 DEBUG:boto:Port: 8000 2023-05-31T20:21:05.169 DEBUG:boto:Params: {} 2023-05-31T20:21:05.169 DEBUG:boto:Token: None 2023-05-31T20:21:05.170 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:21:05 GMT /pklxkd-48/ 2023-05-31T20:21:05.171 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:3ZoBJqr7pRMjTmfx7u+QyvQHg74= 2023-05-31T20:21:05.171 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:21:05 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:3ZoBJqr7pRMjTmfx7u+QyvQHg74=', 'Content-Length': '0'} 2023-05-31T20:21:05.175 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000871d24dcf84c1504-006477ac31-1630-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:21:05 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:21:05.176 DEBUG:boto:b'pklxkd-48obj41000falseobj42023-05-31T20:18:05.356Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:21:05.177 DEBUG:boto:path=/pklxkd-48/obj4 2023-05-31T20:21:05.177 DEBUG:boto:auth_path=/pklxkd-48/obj4 2023-05-31T20:21:05.178 DEBUG:boto:Method: HEAD 2023-05-31T20:21:05.178 DEBUG:boto:Path: /pklxkd-48/obj4 2023-05-31T20:21:05.179 DEBUG:boto:Data: 2023-05-31T20:21:05.179 DEBUG:boto:Headers: {} 2023-05-31T20:21:05.179 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:21:05.180 DEBUG:boto:Port: 8001 2023-05-31T20:21:05.180 DEBUG:boto:Params: {} 2023-05-31T20:21:05.181 DEBUG:boto:Token: None 2023-05-31T20:21:05.181 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:21:05 GMT /pklxkd-48/obj4 2023-05-31T20:21:05.181 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:ndVmTpgbVDWTT+wjsjlzmGkPAh0= 2023-05-31T20:21:05.182 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:21:05 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:ndVmTpgbVDWTT+wjsjlzmGkPAh0=', 'Content-Length': '0'} 2023-05-31T20:21:05.185 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:18:05 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000065d3870cad421fcd-006477ac31-162e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:21:05 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:21:05.186 DEBUG:boto:path=/pklxkd-48/obj4 2023-05-31T20:21:05.187 DEBUG:boto:auth_path=/pklxkd-48/obj4 2023-05-31T20:21:05.188 DEBUG:boto:Method: HEAD 2023-05-31T20:21:05.188 DEBUG:boto:Path: /pklxkd-48/obj4 2023-05-31T20:21:05.189 DEBUG:boto:Data: 2023-05-31T20:21:05.190 DEBUG:boto:Headers: {} 2023-05-31T20:21:05.190 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:21:05.191 DEBUG:boto:Port: 8000 2023-05-31T20:21:05.191 DEBUG:boto:Params: {} 2023-05-31T20:21:05.192 DEBUG:boto:Token: None 2023-05-31T20:21:05.193 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:21:05 GMT /pklxkd-48/obj4 2023-05-31T20:21:05.194 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:ndVmTpgbVDWTT+wjsjlzmGkPAh0= 2023-05-31T20:21:05.194 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:21:05 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:ndVmTpgbVDWTT+wjsjlzmGkPAh0=', 'Content-Length': '0'} 2023-05-31T20:21:05.197 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:18:05 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000009c03e6ef984ffcf3-006477ac31-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:21:05 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:21:05.197 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2023-05-31T20:21:05.198 DEBUG:boto:path=/pklxkd-48/obj4 2023-05-31T20:21:05.198 DEBUG:boto:auth_path=/pklxkd-48/obj4 2023-05-31T20:21:05.199 DEBUG:boto:Method: GET 2023-05-31T20:21:05.199 DEBUG:boto:Path: /pklxkd-48/obj4 2023-05-31T20:21:05.199 DEBUG:boto:Data: 2023-05-31T20:21:05.200 DEBUG:boto:Headers: {} 2023-05-31T20:21:05.200 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:21:05.200 DEBUG:boto:Port: 8001 2023-05-31T20:21:05.201 DEBUG:boto:Params: {} 2023-05-31T20:21:05.201 DEBUG:boto:Token: None 2023-05-31T20:21:05.202 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:21:05 GMT /pklxkd-48/obj4 2023-05-31T20:21:05.202 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:eXnBKOgFwRMBw6aKgyW/bmtO0dg= 2023-05-31T20:21:05.203 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:21:05 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:eXnBKOgFwRMBw6aKgyW/bmtO0dg=', 'Content-Length': '0'} 2023-05-31T20:21:05.206 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:18:05 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000d06f7e8c4dffe067-006477ac31-162e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:21:05 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:21:05.206 DEBUG:boto:path=/pklxkd-48/obj4 2023-05-31T20:21:05.207 DEBUG:boto:auth_path=/pklxkd-48/obj4 2023-05-31T20:21:05.207 DEBUG:boto:Method: GET 2023-05-31T20:21:05.207 DEBUG:boto:Path: /pklxkd-48/obj4 2023-05-31T20:21:05.208 DEBUG:boto:Data: 2023-05-31T20:21:05.208 DEBUG:boto:Headers: {} 2023-05-31T20:21:05.209 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:21:05.209 DEBUG:boto:Port: 8000 2023-05-31T20:21:05.209 DEBUG:boto:Params: {} 2023-05-31T20:21:05.210 DEBUG:boto:Token: None 2023-05-31T20:21:05.210 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:21:05 GMT /pklxkd-48/obj4 2023-05-31T20:21:05.211 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:eXnBKOgFwRMBw6aKgyW/bmtO0dg= 2023-05-31T20:21:05.211 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:21:05 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:eXnBKOgFwRMBw6aKgyW/bmtO0dg=', 'Content-Length': '0'} 2023-05-31T20:21:05.219 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:18:05 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000092abc4fc4b76887a-006477ac31-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:21:05 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:21:05.222 DEBUG:boto:path=/pklxkd-48/ 2023-05-31T20:21:05.223 DEBUG:boto:auth_path=/pklxkd-48/ 2023-05-31T20:21:05.224 DEBUG:boto:path=/pklxkd-48/?prefix=obj4 2023-05-31T20:21:05.225 DEBUG:boto:auth_path=/pklxkd-48/?prefix=obj4 2023-05-31T20:21:05.226 DEBUG:boto:Method: GET 2023-05-31T20:21:05.227 DEBUG:boto:Path: /pklxkd-48/?prefix=obj4 2023-05-31T20:21:05.228 DEBUG:boto:Data: 2023-05-31T20:21:05.229 DEBUG:boto:Headers: {} 2023-05-31T20:21:05.229 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:21:05.230 DEBUG:boto:Port: 8001 2023-05-31T20:21:05.230 DEBUG:boto:Params: {} 2023-05-31T20:21:05.231 DEBUG:boto:Token: None 2023-05-31T20:21:05.231 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:21:05 GMT /pklxkd-48/ 2023-05-31T20:21:05.231 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:3ZoBJqr7pRMjTmfx7u+QyvQHg74= 2023-05-31T20:21:05.233 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:21:05 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:3ZoBJqr7pRMjTmfx7u+QyvQHg74=', 'Content-Length': '0'} 2023-05-31T20:21:05.238 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f5988c760dfcca29-006477ac31-162e-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:21:05 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:21:05.280 DEBUG:boto:b'pklxkd-48obj41000falseobj42023-05-31T20:18:05.356Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:21:05.282 DEBUG:boto:path=/pklxkd-48/ 2023-05-31T20:21:05.283 DEBUG:boto:auth_path=/pklxkd-48/ 2023-05-31T20:21:05.284 DEBUG:boto:path=/pklxkd-48/?prefix=obj4 2023-05-31T20:21:05.285 DEBUG:boto:auth_path=/pklxkd-48/?prefix=obj4 2023-05-31T20:21:05.286 DEBUG:boto:Method: GET 2023-05-31T20:21:05.286 DEBUG:boto:Path: /pklxkd-48/?prefix=obj4 2023-05-31T20:21:05.287 DEBUG:boto:Data: 2023-05-31T20:21:05.288 DEBUG:boto:Headers: {} 2023-05-31T20:21:05.288 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:21:05.289 DEBUG:boto:Port: 8000 2023-05-31T20:21:05.289 DEBUG:boto:Params: {} 2023-05-31T20:21:05.290 DEBUG:boto:Token: None 2023-05-31T20:21:05.290 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:21:05 GMT /pklxkd-48/ 2023-05-31T20:21:05.291 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:3ZoBJqr7pRMjTmfx7u+QyvQHg74= 2023-05-31T20:21:05.292 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:21:05 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:3ZoBJqr7pRMjTmfx7u+QyvQHg74=', 'Content-Length': '0'} 2023-05-31T20:21:05.296 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b38b26869af2bbbb-006477ac31-1630-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:21:05 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:21:05.298 DEBUG:boto:b'pklxkd-48obj41000falseobj42023-05-31T20:18:05.356Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:21:05.299 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj5 2023-05-31T20:21:05.300 DEBUG:boto:path=/pklxkd-48/obj5 2023-05-31T20:21:05.301 DEBUG:boto:auth_path=/pklxkd-48/obj5 2023-05-31T20:21:05.302 DEBUG:boto:path=/pklxkd-48/obj5?versionId=null 2023-05-31T20:21:05.302 DEBUG:boto:auth_path=/pklxkd-48/obj5?versionId=null 2023-05-31T20:21:05.303 DEBUG:boto:Method: GET 2023-05-31T20:21:05.304 DEBUG:boto:Path: /pklxkd-48/obj5?versionId=null 2023-05-31T20:21:05.305 DEBUG:boto:Data: 2023-05-31T20:21:05.305 DEBUG:boto:Headers: {} 2023-05-31T20:21:05.306 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:21:05.307 DEBUG:boto:Port: 8001 2023-05-31T20:21:05.307 DEBUG:boto:Params: {} 2023-05-31T20:21:05.308 DEBUG:boto:Token: None 2023-05-31T20:21:05.309 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:21:05 GMT /pklxkd-48/obj5?versionId=null 2023-05-31T20:21:05.309 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:ckI0rRVsTmRBahWqnS8JlcBAL1Q= 2023-05-31T20:21:05.309 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:21:05 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:ckI0rRVsTmRBahWqnS8JlcBAL1Q=', 'Content-Length': '0'} 2023-05-31T20:21:05.313 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:20:28 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000773266c13f287af1-006477ac31-162e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:21:05 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:21:05.314 DEBUG:boto:path=/pklxkd-48/obj5 2023-05-31T20:21:05.315 DEBUG:boto:auth_path=/pklxkd-48/obj5 2023-05-31T20:21:05.316 DEBUG:boto:path=/pklxkd-48/obj5?versionId=null 2023-05-31T20:21:05.317 DEBUG:boto:auth_path=/pklxkd-48/obj5?versionId=null 2023-05-31T20:21:05.317 DEBUG:boto:Method: GET 2023-05-31T20:21:05.318 DEBUG:boto:Path: /pklxkd-48/obj5?versionId=null 2023-05-31T20:21:05.319 DEBUG:boto:Data: 2023-05-31T20:21:05.319 DEBUG:boto:Headers: {} 2023-05-31T20:21:05.322 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:21:05.322 DEBUG:boto:Port: 8000 2023-05-31T20:21:05.323 DEBUG:boto:Params: {} 2023-05-31T20:21:05.323 DEBUG:boto:Token: None 2023-05-31T20:21:05.324 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:21:05 GMT /pklxkd-48/obj5?versionId=null 2023-05-31T20:21:05.324 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:ckI0rRVsTmRBahWqnS8JlcBAL1Q= 2023-05-31T20:21:05.325 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:21:05 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:ckI0rRVsTmRBahWqnS8JlcBAL1Q=', 'Content-Length': '0'} 2023-05-31T20:21:05.327 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:20:28 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000006f7a15f83289fc46-006477ac31-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:21:05 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:21:05.375 DEBUG:boto:path=/pklxkd-48/obj5 2023-05-31T20:21:05.376 DEBUG:boto:auth_path=/pklxkd-48/obj5 2023-05-31T20:21:05.377 DEBUG:boto:path=/pklxkd-48/obj5?versionId=null 2023-05-31T20:21:05.378 DEBUG:boto:auth_path=/pklxkd-48/obj5?versionId=null 2023-05-31T20:21:05.379 DEBUG:boto:Method: HEAD 2023-05-31T20:21:05.379 DEBUG:boto:Path: /pklxkd-48/obj5?versionId=null 2023-05-31T20:21:05.380 DEBUG:boto:Data: 2023-05-31T20:21:05.382 DEBUG:boto:Headers: {} 2023-05-31T20:21:05.383 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:21:05.383 DEBUG:boto:Port: 8001 2023-05-31T20:21:05.384 DEBUG:boto:Params: {} 2023-05-31T20:21:05.384 DEBUG:boto:Token: None 2023-05-31T20:21:05.385 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:21:05 GMT /pklxkd-48/obj5?versionId=null 2023-05-31T20:21:05.385 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:p4HVC8MStq5Mn6ShKOKEaubPKHE= 2023-05-31T20:21:05.386 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:21:05 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:p4HVC8MStq5Mn6ShKOKEaubPKHE=', 'Content-Length': '0'} 2023-05-31T20:21:05.388 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:20:28 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000095e49dd9d0de944f-006477ac31-162e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:21:05 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:21:05.389 DEBUG:boto:path=/pklxkd-48/obj5 2023-05-31T20:21:05.389 DEBUG:boto:auth_path=/pklxkd-48/obj5 2023-05-31T20:21:05.389 DEBUG:boto:path=/pklxkd-48/obj5?versionId=null 2023-05-31T20:21:05.390 DEBUG:boto:auth_path=/pklxkd-48/obj5?versionId=null 2023-05-31T20:21:05.391 DEBUG:boto:Method: HEAD 2023-05-31T20:21:05.391 DEBUG:boto:Path: /pklxkd-48/obj5?versionId=null 2023-05-31T20:21:05.391 DEBUG:boto:Data: 2023-05-31T20:21:05.392 DEBUG:boto:Headers: {} 2023-05-31T20:21:05.392 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:21:05.393 DEBUG:boto:Port: 8000 2023-05-31T20:21:05.393 DEBUG:boto:Params: {} 2023-05-31T20:21:05.393 DEBUG:boto:Token: None 2023-05-31T20:21:05.394 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:21:05 GMT /pklxkd-48/obj5?versionId=null 2023-05-31T20:21:05.395 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:p4HVC8MStq5Mn6ShKOKEaubPKHE= 2023-05-31T20:21:05.395 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:21:05 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:p4HVC8MStq5Mn6ShKOKEaubPKHE=', 'Content-Length': '0'} 2023-05-31T20:21:05.398 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:20:28 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000838d0b6db3a0a8a1-006477ac31-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:21:05 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:21:05.398 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj5 2023-05-31T20:21:05.399 DEBUG:boto:path=/pklxkd-48/obj5 2023-05-31T20:21:05.399 DEBUG:boto:auth_path=/pklxkd-48/obj5 2023-05-31T20:21:05.399 DEBUG:boto:path=/pklxkd-48/obj5?versionId=null 2023-05-31T20:21:05.400 DEBUG:boto:auth_path=/pklxkd-48/obj5?versionId=null 2023-05-31T20:21:05.400 DEBUG:boto:Method: GET 2023-05-31T20:21:05.401 DEBUG:boto:Path: /pklxkd-48/obj5?versionId=null 2023-05-31T20:21:05.401 DEBUG:boto:Data: 2023-05-31T20:21:05.401 DEBUG:boto:Headers: {} 2023-05-31T20:21:05.402 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:21:05.402 DEBUG:boto:Port: 8001 2023-05-31T20:21:05.403 DEBUG:boto:Params: {} 2023-05-31T20:21:05.403 DEBUG:boto:Token: None 2023-05-31T20:21:05.403 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:21:05 GMT /pklxkd-48/obj5?versionId=null 2023-05-31T20:21:05.404 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:ckI0rRVsTmRBahWqnS8JlcBAL1Q= 2023-05-31T20:21:05.404 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:21:05 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:ckI0rRVsTmRBahWqnS8JlcBAL1Q=', 'Content-Length': '0'} 2023-05-31T20:21:05.406 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:20:28 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000029edb20440ce2f75-006477ac31-162e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:21:05 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:21:05.407 DEBUG:boto:path=/pklxkd-48/obj5 2023-05-31T20:21:05.408 DEBUG:boto:auth_path=/pklxkd-48/obj5 2023-05-31T20:21:05.408 DEBUG:boto:path=/pklxkd-48/obj5?versionId=null 2023-05-31T20:21:05.408 DEBUG:boto:auth_path=/pklxkd-48/obj5?versionId=null 2023-05-31T20:21:05.409 DEBUG:boto:Method: GET 2023-05-31T20:21:05.409 DEBUG:boto:Path: /pklxkd-48/obj5?versionId=null 2023-05-31T20:21:05.409 DEBUG:boto:Data: 2023-05-31T20:21:05.410 DEBUG:boto:Headers: {} 2023-05-31T20:21:05.410 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:21:05.411 DEBUG:boto:Port: 8000 2023-05-31T20:21:05.411 DEBUG:boto:Params: {} 2023-05-31T20:21:05.411 DEBUG:boto:Token: None 2023-05-31T20:21:05.412 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:21:05 GMT /pklxkd-48/obj5?versionId=null 2023-05-31T20:21:05.412 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:ckI0rRVsTmRBahWqnS8JlcBAL1Q= 2023-05-31T20:21:05.413 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:21:05 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:ckI0rRVsTmRBahWqnS8JlcBAL1Q=', 'Content-Length': '0'} 2023-05-31T20:21:05.415 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:20:28 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000003cd8e3581308c2cb-006477ac31-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:21:05 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:21:05.416 DEBUG:boto:path=/pklxkd-48/ 2023-05-31T20:21:05.418 DEBUG:boto:auth_path=/pklxkd-48/ 2023-05-31T20:21:05.418 DEBUG:boto:path=/pklxkd-48/?prefix=obj5 2023-05-31T20:21:05.419 DEBUG:boto:auth_path=/pklxkd-48/?prefix=obj5 2023-05-31T20:21:05.420 DEBUG:boto:Method: GET 2023-05-31T20:21:05.421 DEBUG:boto:Path: /pklxkd-48/?prefix=obj5 2023-05-31T20:21:05.422 DEBUG:boto:Data: 2023-05-31T20:21:05.423 DEBUG:boto:Headers: {} 2023-05-31T20:21:05.424 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:21:05.424 DEBUG:boto:Port: 8001 2023-05-31T20:21:05.425 DEBUG:boto:Params: {} 2023-05-31T20:21:05.426 DEBUG:boto:Token: None 2023-05-31T20:21:05.427 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:21:05 GMT /pklxkd-48/ 2023-05-31T20:21:05.427 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:3ZoBJqr7pRMjTmfx7u+QyvQHg74= 2023-05-31T20:21:05.428 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:21:05 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:3ZoBJqr7pRMjTmfx7u+QyvQHg74=', 'Content-Length': '0'} 2023-05-31T20:21:05.430 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000097bf0deaa15c731-006477ac31-162e-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:21:05 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:21:05.431 DEBUG:boto:b'pklxkd-48obj51000falseobj52023-05-31T20:20:28.673Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:21:05.431 DEBUG:boto:path=/pklxkd-48/ 2023-05-31T20:21:05.432 DEBUG:boto:auth_path=/pklxkd-48/ 2023-05-31T20:21:05.432 DEBUG:boto:path=/pklxkd-48/?prefix=obj5 2023-05-31T20:21:05.433 DEBUG:boto:auth_path=/pklxkd-48/?prefix=obj5 2023-05-31T20:21:05.433 DEBUG:boto:Method: GET 2023-05-31T20:21:05.433 DEBUG:boto:Path: /pklxkd-48/?prefix=obj5 2023-05-31T20:21:05.434 DEBUG:boto:Data: 2023-05-31T20:21:05.434 DEBUG:boto:Headers: {} 2023-05-31T20:21:05.434 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:21:05.435 DEBUG:boto:Port: 8000 2023-05-31T20:21:05.435 DEBUG:boto:Params: {} 2023-05-31T20:21:05.436 DEBUG:boto:Token: None 2023-05-31T20:21:05.436 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:21:05 GMT /pklxkd-48/ 2023-05-31T20:21:05.437 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:3ZoBJqr7pRMjTmfx7u+QyvQHg74= 2023-05-31T20:21:05.437 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:21:05 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:3ZoBJqr7pRMjTmfx7u+QyvQHg74=', 'Content-Length': '0'} 2023-05-31T20:21:05.440 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000000e2ef7ae2252b77-006477ac31-1630-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:21:05 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:21:05.440 DEBUG:boto:b'pklxkd-48obj51000falseobj52023-05-31T20:20:28.673Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:21:05.441 DEBUG:boto:path=/pklxkd-48/obj5 2023-05-31T20:21:05.441 DEBUG:boto:auth_path=/pklxkd-48/obj5 2023-05-31T20:21:05.442 DEBUG:boto:Method: HEAD 2023-05-31T20:21:05.442 DEBUG:boto:Path: /pklxkd-48/obj5 2023-05-31T20:21:05.443 DEBUG:boto:Data: 2023-05-31T20:21:05.443 DEBUG:boto:Headers: {} 2023-05-31T20:21:05.443 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:21:05.444 DEBUG:boto:Port: 8001 2023-05-31T20:21:05.444 DEBUG:boto:Params: {} 2023-05-31T20:21:05.445 DEBUG:boto:Token: None 2023-05-31T20:21:05.445 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:21:05 GMT /pklxkd-48/obj5 2023-05-31T20:21:05.446 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:kTm8EFM1FZYs2N/7v0e0sKIBzdY= 2023-05-31T20:21:05.446 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:21:05 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:kTm8EFM1FZYs2N/7v0e0sKIBzdY=', 'Content-Length': '0'} 2023-05-31T20:21:05.448 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:20:28 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000103a04c96138d10a-006477ac31-162e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:21:05 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:21:05.450 DEBUG:boto:path=/pklxkd-48/obj5 2023-05-31T20:21:05.451 DEBUG:boto:auth_path=/pklxkd-48/obj5 2023-05-31T20:21:05.451 DEBUG:boto:Method: HEAD 2023-05-31T20:21:05.452 DEBUG:boto:Path: /pklxkd-48/obj5 2023-05-31T20:21:05.453 DEBUG:boto:Data: 2023-05-31T20:21:05.454 DEBUG:boto:Headers: {} 2023-05-31T20:21:05.455 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:21:05.456 DEBUG:boto:Port: 8000 2023-05-31T20:21:05.457 DEBUG:boto:Params: {} 2023-05-31T20:21:05.458 DEBUG:boto:Token: None 2023-05-31T20:21:05.459 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:21:05 GMT /pklxkd-48/obj5 2023-05-31T20:21:05.459 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:kTm8EFM1FZYs2N/7v0e0sKIBzdY= 2023-05-31T20:21:05.459 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:21:05 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:kTm8EFM1FZYs2N/7v0e0sKIBzdY=', 'Content-Length': '0'} 2023-05-31T20:21:05.461 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:20:28 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000055cc52dedb797010-006477ac31-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:21:05 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:21:05.462 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj5 2023-05-31T20:21:05.462 DEBUG:boto:path=/pklxkd-48/obj5 2023-05-31T20:21:05.463 DEBUG:boto:auth_path=/pklxkd-48/obj5 2023-05-31T20:21:05.463 DEBUG:boto:Method: GET 2023-05-31T20:21:05.464 DEBUG:boto:Path: /pklxkd-48/obj5 2023-05-31T20:21:05.464 DEBUG:boto:Data: 2023-05-31T20:21:05.465 DEBUG:boto:Headers: {} 2023-05-31T20:21:05.465 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:21:05.465 DEBUG:boto:Port: 8001 2023-05-31T20:21:05.466 DEBUG:boto:Params: {} 2023-05-31T20:21:05.466 DEBUG:boto:Token: None 2023-05-31T20:21:05.467 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:21:05 GMT /pklxkd-48/obj5 2023-05-31T20:21:05.468 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:3l46duq4KPLgMfpuOn5/u+eikhQ= 2023-05-31T20:21:05.469 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:21:05 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:3l46duq4KPLgMfpuOn5/u+eikhQ=', 'Content-Length': '0'} 2023-05-31T20:21:05.471 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:20:28 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000f1ea74c4678a1894-006477ac31-162e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:21:05 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:21:05.471 DEBUG:boto:path=/pklxkd-48/obj5 2023-05-31T20:21:05.472 DEBUG:boto:auth_path=/pklxkd-48/obj5 2023-05-31T20:21:05.472 DEBUG:boto:Method: GET 2023-05-31T20:21:05.473 DEBUG:boto:Path: /pklxkd-48/obj5 2023-05-31T20:21:05.473 DEBUG:boto:Data: 2023-05-31T20:21:05.474 DEBUG:boto:Headers: {} 2023-05-31T20:21:05.474 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:21:05.474 DEBUG:boto:Port: 8000 2023-05-31T20:21:05.475 DEBUG:boto:Params: {} 2023-05-31T20:21:05.475 DEBUG:boto:Token: None 2023-05-31T20:21:05.475 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:21:05 GMT /pklxkd-48/obj5 2023-05-31T20:21:05.476 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:3l46duq4KPLgMfpuOn5/u+eikhQ= 2023-05-31T20:21:05.476 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:21:05 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:3l46duq4KPLgMfpuOn5/u+eikhQ=', 'Content-Length': '0'} 2023-05-31T20:21:05.478 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:20:28 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000444d8042e70a6760-006477ac31-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:21:05 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:21:05.479 DEBUG:boto:path=/pklxkd-48/ 2023-05-31T20:21:05.479 DEBUG:boto:auth_path=/pklxkd-48/ 2023-05-31T20:21:05.480 DEBUG:boto:path=/pklxkd-48/?prefix=obj5 2023-05-31T20:21:05.480 DEBUG:boto:auth_path=/pklxkd-48/?prefix=obj5 2023-05-31T20:21:05.480 DEBUG:boto:Method: GET 2023-05-31T20:21:05.481 DEBUG:boto:Path: /pklxkd-48/?prefix=obj5 2023-05-31T20:21:05.481 DEBUG:boto:Data: 2023-05-31T20:21:05.482 DEBUG:boto:Headers: {} 2023-05-31T20:21:05.482 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:21:05.482 DEBUG:boto:Port: 8001 2023-05-31T20:21:05.483 DEBUG:boto:Params: {} 2023-05-31T20:21:05.483 DEBUG:boto:Token: None 2023-05-31T20:21:05.484 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:21:05 GMT /pklxkd-48/ 2023-05-31T20:21:05.484 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:3ZoBJqr7pRMjTmfx7u+QyvQHg74= 2023-05-31T20:21:05.485 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:21:05 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:3ZoBJqr7pRMjTmfx7u+QyvQHg74=', 'Content-Length': '0'} 2023-05-31T20:21:05.487 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000fead986bf3312a96-006477ac31-162e-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:21:05 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:21:05.487 DEBUG:boto:b'pklxkd-48obj51000falseobj52023-05-31T20:20:28.673Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:21:05.488 DEBUG:boto:path=/pklxkd-48/ 2023-05-31T20:21:05.489 DEBUG:boto:auth_path=/pklxkd-48/ 2023-05-31T20:21:05.489 DEBUG:boto:path=/pklxkd-48/?prefix=obj5 2023-05-31T20:21:05.489 DEBUG:boto:auth_path=/pklxkd-48/?prefix=obj5 2023-05-31T20:21:05.490 DEBUG:boto:Method: GET 2023-05-31T20:21:05.490 DEBUG:boto:Path: /pklxkd-48/?prefix=obj5 2023-05-31T20:21:05.491 DEBUG:boto:Data: 2023-05-31T20:21:05.491 DEBUG:boto:Headers: {} 2023-05-31T20:21:05.491 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:21:05.492 DEBUG:boto:Port: 8000 2023-05-31T20:21:05.492 DEBUG:boto:Params: {} 2023-05-31T20:21:05.493 DEBUG:boto:Token: None 2023-05-31T20:21:05.493 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:21:05 GMT /pklxkd-48/ 2023-05-31T20:21:05.493 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:3ZoBJqr7pRMjTmfx7u+QyvQHg74= 2023-05-31T20:21:05.494 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:21:05 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:3ZoBJqr7pRMjTmfx7u+QyvQHg74=', 'Content-Length': '0'} 2023-05-31T20:21:05.496 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000001614c13600ded3e8-006477ac31-1630-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:21:05 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:21:05.497 DEBUG:boto:b'pklxkd-48obj51000falseobj52023-05-31T20:20:28.673Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:21:05.498 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj6 2023-05-31T20:21:05.498 DEBUG:boto:path=/pklxkd-48/obj6 2023-05-31T20:21:05.498 DEBUG:boto:auth_path=/pklxkd-48/obj6 2023-05-31T20:21:05.499 DEBUG:boto:path=/pklxkd-48/obj6?versionId=null 2023-05-31T20:21:05.499 DEBUG:boto:auth_path=/pklxkd-48/obj6?versionId=null 2023-05-31T20:21:05.500 DEBUG:boto:Method: GET 2023-05-31T20:21:05.500 DEBUG:boto:Path: /pklxkd-48/obj6?versionId=null 2023-05-31T20:21:05.500 DEBUG:boto:Data: 2023-05-31T20:21:05.501 DEBUG:boto:Headers: {} 2023-05-31T20:21:05.501 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:21:05.501 DEBUG:boto:Port: 8001 2023-05-31T20:21:05.502 DEBUG:boto:Params: {} 2023-05-31T20:21:05.502 DEBUG:boto:Token: None 2023-05-31T20:21:05.503 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:21:05 GMT /pklxkd-48/obj6?versionId=null 2023-05-31T20:21:05.503 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:hFgiB1pvmesaHc5skJddFhBsOjU= 2023-05-31T20:21:05.504 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:21:05 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:hFgiB1pvmesaHc5skJddFhBsOjU=', 'Content-Length': '0'} 2023-05-31T20:21:05.506 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:20:28 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000077032ddb87645644-006477ac31-162e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:21:05 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:21:05.506 DEBUG:boto:path=/pklxkd-48/obj6 2023-05-31T20:21:05.507 DEBUG:boto:auth_path=/pklxkd-48/obj6 2023-05-31T20:21:05.507 DEBUG:boto:path=/pklxkd-48/obj6?versionId=null 2023-05-31T20:21:05.507 DEBUG:boto:auth_path=/pklxkd-48/obj6?versionId=null 2023-05-31T20:21:05.508 DEBUG:boto:Method: GET 2023-05-31T20:21:05.508 DEBUG:boto:Path: /pklxkd-48/obj6?versionId=null 2023-05-31T20:21:05.509 DEBUG:boto:Data: 2023-05-31T20:21:05.509 DEBUG:boto:Headers: {} 2023-05-31T20:21:05.509 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:21:05.510 DEBUG:boto:Port: 8000 2023-05-31T20:21:05.510 DEBUG:boto:Params: {} 2023-05-31T20:21:05.510 DEBUG:boto:Token: None 2023-05-31T20:21:05.511 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:21:05 GMT /pklxkd-48/obj6?versionId=null 2023-05-31T20:21:05.511 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:hFgiB1pvmesaHc5skJddFhBsOjU= 2023-05-31T20:21:05.512 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:21:05 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:hFgiB1pvmesaHc5skJddFhBsOjU=', 'Content-Length': '0'} 2023-05-31T20:21:05.514 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:20:28 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000037fad53d89a0d8fe-006477ac31-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:21:05 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:21:05.515 DEBUG:boto:path=/pklxkd-48/obj6 2023-05-31T20:21:05.515 DEBUG:boto:auth_path=/pklxkd-48/obj6 2023-05-31T20:21:05.515 DEBUG:boto:path=/pklxkd-48/obj6?versionId=null 2023-05-31T20:21:05.516 DEBUG:boto:auth_path=/pklxkd-48/obj6?versionId=null 2023-05-31T20:21:05.516 DEBUG:boto:Method: HEAD 2023-05-31T20:21:05.516 DEBUG:boto:Path: /pklxkd-48/obj6?versionId=null 2023-05-31T20:21:05.517 DEBUG:boto:Data: 2023-05-31T20:21:05.517 DEBUG:boto:Headers: {} 2023-05-31T20:21:05.517 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:21:05.518 DEBUG:boto:Port: 8001 2023-05-31T20:21:05.518 DEBUG:boto:Params: {} 2023-05-31T20:21:05.518 DEBUG:boto:Token: None 2023-05-31T20:21:05.519 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:21:05 GMT /pklxkd-48/obj6?versionId=null 2023-05-31T20:21:05.519 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:YQMqASSX6iRdmhsYT+XvtbJgVlY= 2023-05-31T20:21:05.520 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:21:05 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:YQMqASSX6iRdmhsYT+XvtbJgVlY=', 'Content-Length': '0'} 2023-05-31T20:21:05.521 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:20:28 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000007f460bfaa262b259-006477ac31-162e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:21:05 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:21:05.522 DEBUG:boto:path=/pklxkd-48/obj6 2023-05-31T20:21:05.523 DEBUG:boto:auth_path=/pklxkd-48/obj6 2023-05-31T20:21:05.523 DEBUG:boto:path=/pklxkd-48/obj6?versionId=null 2023-05-31T20:21:05.523 DEBUG:boto:auth_path=/pklxkd-48/obj6?versionId=null 2023-05-31T20:21:05.524 DEBUG:boto:Method: HEAD 2023-05-31T20:21:05.524 DEBUG:boto:Path: /pklxkd-48/obj6?versionId=null 2023-05-31T20:21:05.524 DEBUG:boto:Data: 2023-05-31T20:21:05.525 DEBUG:boto:Headers: {} 2023-05-31T20:21:05.525 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:21:05.525 DEBUG:boto:Port: 8000 2023-05-31T20:21:05.526 DEBUG:boto:Params: {} 2023-05-31T20:21:05.526 DEBUG:boto:Token: None 2023-05-31T20:21:05.527 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:21:05 GMT /pklxkd-48/obj6?versionId=null 2023-05-31T20:21:05.527 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:YQMqASSX6iRdmhsYT+XvtbJgVlY= 2023-05-31T20:21:05.527 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:21:05 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:YQMqASSX6iRdmhsYT+XvtbJgVlY=', 'Content-Length': '0'} 2023-05-31T20:21:05.529 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:20:28 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000000258dfa7a823a0f9-006477ac31-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:21:05 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:21:05.531 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj6 2023-05-31T20:21:05.531 DEBUG:boto:path=/pklxkd-48/obj6 2023-05-31T20:21:05.532 DEBUG:boto:auth_path=/pklxkd-48/obj6 2023-05-31T20:21:05.533 DEBUG:boto:path=/pklxkd-48/obj6?versionId=null 2023-05-31T20:21:05.534 DEBUG:boto:auth_path=/pklxkd-48/obj6?versionId=null 2023-05-31T20:21:05.535 DEBUG:boto:Method: GET 2023-05-31T20:21:05.536 DEBUG:boto:Path: /pklxkd-48/obj6?versionId=null 2023-05-31T20:21:05.536 DEBUG:boto:Data: 2023-05-31T20:21:05.537 DEBUG:boto:Headers: {} 2023-05-31T20:21:05.538 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:21:05.538 DEBUG:boto:Port: 8001 2023-05-31T20:21:05.538 DEBUG:boto:Params: {} 2023-05-31T20:21:05.539 DEBUG:boto:Token: None 2023-05-31T20:21:05.539 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:21:05 GMT /pklxkd-48/obj6?versionId=null 2023-05-31T20:21:05.540 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:hFgiB1pvmesaHc5skJddFhBsOjU= 2023-05-31T20:21:05.540 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:21:05 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:hFgiB1pvmesaHc5skJddFhBsOjU=', 'Content-Length': '0'} 2023-05-31T20:21:05.542 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:20:28 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000007231c7b8fb9923da-006477ac31-162e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:21:05 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:21:05.543 DEBUG:boto:path=/pklxkd-48/obj6 2023-05-31T20:21:05.544 DEBUG:boto:auth_path=/pklxkd-48/obj6 2023-05-31T20:21:05.544 DEBUG:boto:path=/pklxkd-48/obj6?versionId=null 2023-05-31T20:21:05.544 DEBUG:boto:auth_path=/pklxkd-48/obj6?versionId=null 2023-05-31T20:21:05.545 DEBUG:boto:Method: GET 2023-05-31T20:21:05.545 DEBUG:boto:Path: /pklxkd-48/obj6?versionId=null 2023-05-31T20:21:05.545 DEBUG:boto:Data: 2023-05-31T20:21:05.546 DEBUG:boto:Headers: {} 2023-05-31T20:21:05.546 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:21:05.546 DEBUG:boto:Port: 8000 2023-05-31T20:21:05.547 DEBUG:boto:Params: {} 2023-05-31T20:21:05.547 DEBUG:boto:Token: None 2023-05-31T20:21:05.548 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:21:05 GMT /pklxkd-48/obj6?versionId=null 2023-05-31T20:21:05.548 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:hFgiB1pvmesaHc5skJddFhBsOjU= 2023-05-31T20:21:05.549 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:21:05 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:hFgiB1pvmesaHc5skJddFhBsOjU=', 'Content-Length': '0'} 2023-05-31T20:21:05.551 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:20:28 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000003ae4b4a88e8f31e9-006477ac31-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:21:05 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:21:05.593 DEBUG:boto:path=/pklxkd-48/ 2023-05-31T20:21:05.594 DEBUG:boto:auth_path=/pklxkd-48/ 2023-05-31T20:21:05.595 DEBUG:boto:path=/pklxkd-48/?prefix=obj6 2023-05-31T20:21:05.596 DEBUG:boto:auth_path=/pklxkd-48/?prefix=obj6 2023-05-31T20:21:05.597 DEBUG:boto:Method: GET 2023-05-31T20:21:05.597 DEBUG:boto:Path: /pklxkd-48/?prefix=obj6 2023-05-31T20:21:05.598 DEBUG:boto:Data: 2023-05-31T20:21:05.598 DEBUG:boto:Headers: {} 2023-05-31T20:21:05.598 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:21:05.599 DEBUG:boto:Port: 8001 2023-05-31T20:21:05.599 DEBUG:boto:Params: {} 2023-05-31T20:21:05.600 DEBUG:boto:Token: None 2023-05-31T20:21:05.601 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:21:05 GMT /pklxkd-48/ 2023-05-31T20:21:05.601 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:3ZoBJqr7pRMjTmfx7u+QyvQHg74= 2023-05-31T20:21:05.602 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:21:05 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:3ZoBJqr7pRMjTmfx7u+QyvQHg74=', 'Content-Length': '0'} 2023-05-31T20:21:05.604 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000023dd9d67c59c1e80-006477ac31-162e-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:21:05 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:21:05.605 DEBUG:boto:b'pklxkd-48obj61000falseobj62023-05-31T20:20:28.699Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:21:05.606 DEBUG:boto:path=/pklxkd-48/ 2023-05-31T20:21:05.607 DEBUG:boto:auth_path=/pklxkd-48/ 2023-05-31T20:21:05.607 DEBUG:boto:path=/pklxkd-48/?prefix=obj6 2023-05-31T20:21:05.607 DEBUG:boto:auth_path=/pklxkd-48/?prefix=obj6 2023-05-31T20:21:05.608 DEBUG:boto:Method: GET 2023-05-31T20:21:05.608 DEBUG:boto:Path: /pklxkd-48/?prefix=obj6 2023-05-31T20:21:05.609 DEBUG:boto:Data: 2023-05-31T20:21:05.609 DEBUG:boto:Headers: {} 2023-05-31T20:21:05.609 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:21:05.610 DEBUG:boto:Port: 8000 2023-05-31T20:21:05.610 DEBUG:boto:Params: {} 2023-05-31T20:21:05.611 DEBUG:boto:Token: None 2023-05-31T20:21:05.611 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:21:05 GMT /pklxkd-48/ 2023-05-31T20:21:05.612 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:3ZoBJqr7pRMjTmfx7u+QyvQHg74= 2023-05-31T20:21:05.612 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:21:05 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:3ZoBJqr7pRMjTmfx7u+QyvQHg74=', 'Content-Length': '0'} 2023-05-31T20:21:05.615 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000086f110b4dd3d87a1-006477ac31-1630-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:21:05 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:21:05.615 DEBUG:boto:b'pklxkd-48obj61000falseobj62023-05-31T20:20:28.699Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:21:05.616 DEBUG:boto:path=/pklxkd-48/obj6 2023-05-31T20:21:05.616 DEBUG:boto:auth_path=/pklxkd-48/obj6 2023-05-31T20:21:05.617 DEBUG:boto:Method: HEAD 2023-05-31T20:21:05.617 DEBUG:boto:Path: /pklxkd-48/obj6 2023-05-31T20:21:05.618 DEBUG:boto:Data: 2023-05-31T20:21:05.618 DEBUG:boto:Headers: {} 2023-05-31T20:21:05.618 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:21:05.619 DEBUG:boto:Port: 8001 2023-05-31T20:21:05.619 DEBUG:boto:Params: {} 2023-05-31T20:21:05.619 DEBUG:boto:Token: None 2023-05-31T20:21:05.620 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:21:05 GMT /pklxkd-48/obj6 2023-05-31T20:21:05.620 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:4sccVO8UDoZu/6fDwS8WLC8NpCA= 2023-05-31T20:21:05.620 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:21:05 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:4sccVO8UDoZu/6fDwS8WLC8NpCA=', 'Content-Length': '0'} 2023-05-31T20:21:05.622 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:20:28 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000004d72c6181e8ff4c-006477ac31-162e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:21:05 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:21:05.623 DEBUG:boto:path=/pklxkd-48/obj6 2023-05-31T20:21:05.623 DEBUG:boto:auth_path=/pklxkd-48/obj6 2023-05-31T20:21:05.624 DEBUG:boto:Method: HEAD 2023-05-31T20:21:05.624 DEBUG:boto:Path: /pklxkd-48/obj6 2023-05-31T20:21:05.624 DEBUG:boto:Data: 2023-05-31T20:21:05.625 DEBUG:boto:Headers: {} 2023-05-31T20:21:05.625 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:21:05.625 DEBUG:boto:Port: 8000 2023-05-31T20:21:05.626 DEBUG:boto:Params: {} 2023-05-31T20:21:05.626 DEBUG:boto:Token: None 2023-05-31T20:21:05.626 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:21:05 GMT /pklxkd-48/obj6 2023-05-31T20:21:05.627 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:4sccVO8UDoZu/6fDwS8WLC8NpCA= 2023-05-31T20:21:05.627 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:21:05 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:4sccVO8UDoZu/6fDwS8WLC8NpCA=', 'Content-Length': '0'} 2023-05-31T20:21:05.629 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:20:28 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000a0fd307c99026bcf-006477ac31-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:21:05 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:21:05.630 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj6 2023-05-31T20:21:05.630 DEBUG:boto:path=/pklxkd-48/obj6 2023-05-31T20:21:05.630 DEBUG:boto:auth_path=/pklxkd-48/obj6 2023-05-31T20:21:05.631 DEBUG:boto:Method: GET 2023-05-31T20:21:05.631 DEBUG:boto:Path: /pklxkd-48/obj6 2023-05-31T20:21:05.632 DEBUG:boto:Data: 2023-05-31T20:21:05.632 DEBUG:boto:Headers: {} 2023-05-31T20:21:05.632 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:21:05.633 DEBUG:boto:Port: 8001 2023-05-31T20:21:05.633 DEBUG:boto:Params: {} 2023-05-31T20:21:05.633 DEBUG:boto:Token: None 2023-05-31T20:21:05.634 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:21:05 GMT /pklxkd-48/obj6 2023-05-31T20:21:05.634 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:PzxOKt0Sp4h8ofqm+tE6+LhIrzw= 2023-05-31T20:21:05.635 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:21:05 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:PzxOKt0Sp4h8ofqm+tE6+LhIrzw=', 'Content-Length': '0'} 2023-05-31T20:21:05.636 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:20:28 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000dd637837e548dbc3-006477ac31-162e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:21:05 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:21:05.684 DEBUG:boto:path=/pklxkd-48/obj6 2023-05-31T20:21:05.685 DEBUG:boto:auth_path=/pklxkd-48/obj6 2023-05-31T20:21:05.685 DEBUG:boto:Method: GET 2023-05-31T20:21:05.686 DEBUG:boto:Path: /pklxkd-48/obj6 2023-05-31T20:21:05.686 DEBUG:boto:Data: 2023-05-31T20:21:05.687 DEBUG:boto:Headers: {} 2023-05-31T20:21:05.687 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:21:05.687 DEBUG:boto:Port: 8000 2023-05-31T20:21:05.688 DEBUG:boto:Params: {} 2023-05-31T20:21:05.688 DEBUG:boto:Token: None 2023-05-31T20:21:05.689 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:21:05 GMT /pklxkd-48/obj6 2023-05-31T20:21:05.689 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:PzxOKt0Sp4h8ofqm+tE6+LhIrzw= 2023-05-31T20:21:05.690 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:21:05 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:PzxOKt0Sp4h8ofqm+tE6+LhIrzw=', 'Content-Length': '0'} 2023-05-31T20:21:05.692 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:20:28 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000b087fa781c7c90f3-006477ac31-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:21:05 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:21:05.692 DEBUG:boto:path=/pklxkd-48/ 2023-05-31T20:21:05.693 DEBUG:boto:auth_path=/pklxkd-48/ 2023-05-31T20:21:05.693 DEBUG:boto:path=/pklxkd-48/?prefix=obj6 2023-05-31T20:21:05.694 DEBUG:boto:auth_path=/pklxkd-48/?prefix=obj6 2023-05-31T20:21:05.694 DEBUG:boto:Method: GET 2023-05-31T20:21:05.695 DEBUG:boto:Path: /pklxkd-48/?prefix=obj6 2023-05-31T20:21:05.695 DEBUG:boto:Data: 2023-05-31T20:21:05.696 DEBUG:boto:Headers: {} 2023-05-31T20:21:05.696 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:21:05.696 DEBUG:boto:Port: 8001 2023-05-31T20:21:05.697 DEBUG:boto:Params: {} 2023-05-31T20:21:05.697 DEBUG:boto:Token: None 2023-05-31T20:21:05.698 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:21:05 GMT /pklxkd-48/ 2023-05-31T20:21:05.698 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:3ZoBJqr7pRMjTmfx7u+QyvQHg74= 2023-05-31T20:21:05.698 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:21:05 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:3ZoBJqr7pRMjTmfx7u+QyvQHg74=', 'Content-Length': '0'} 2023-05-31T20:21:05.700 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000bd6364fe9b6c7e2a-006477ac31-162e-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:21:05 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:21:05.701 DEBUG:boto:b'pklxkd-48obj61000falseobj62023-05-31T20:20:28.699Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:21:05.702 DEBUG:boto:path=/pklxkd-48/ 2023-05-31T20:21:05.702 DEBUG:boto:auth_path=/pklxkd-48/ 2023-05-31T20:21:05.702 DEBUG:boto:path=/pklxkd-48/?prefix=obj6 2023-05-31T20:21:05.703 DEBUG:boto:auth_path=/pklxkd-48/?prefix=obj6 2023-05-31T20:21:05.703 DEBUG:boto:Method: GET 2023-05-31T20:21:05.704 DEBUG:boto:Path: /pklxkd-48/?prefix=obj6 2023-05-31T20:21:05.704 DEBUG:boto:Data: 2023-05-31T20:21:05.705 DEBUG:boto:Headers: {} 2023-05-31T20:21:05.705 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:21:05.705 DEBUG:boto:Port: 8000 2023-05-31T20:21:05.706 DEBUG:boto:Params: {} 2023-05-31T20:21:05.706 DEBUG:boto:Token: None 2023-05-31T20:21:05.707 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:21:05 GMT /pklxkd-48/ 2023-05-31T20:21:05.707 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:3ZoBJqr7pRMjTmfx7u+QyvQHg74= 2023-05-31T20:21:05.707 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:21:05 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:3ZoBJqr7pRMjTmfx7u+QyvQHg74=', 'Content-Length': '0'} 2023-05-31T20:21:05.710 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a9772bfd77f82acb-006477ac31-1630-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:21:05 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:21:05.710 DEBUG:boto:b'pklxkd-48obj61000falseobj62023-05-31T20:20:28.699Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:21:05.711 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj7 2023-05-31T20:21:05.711 DEBUG:boto:path=/pklxkd-48/obj7 2023-05-31T20:21:05.712 DEBUG:boto:auth_path=/pklxkd-48/obj7 2023-05-31T20:21:05.712 DEBUG:boto:path=/pklxkd-48/obj7?versionId=null 2023-05-31T20:21:05.713 DEBUG:boto:auth_path=/pklxkd-48/obj7?versionId=null 2023-05-31T20:21:05.713 DEBUG:boto:Method: GET 2023-05-31T20:21:05.713 DEBUG:boto:Path: /pklxkd-48/obj7?versionId=null 2023-05-31T20:21:05.714 DEBUG:boto:Data: 2023-05-31T20:21:05.714 DEBUG:boto:Headers: {} 2023-05-31T20:21:05.714 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:21:05.715 DEBUG:boto:Port: 8001 2023-05-31T20:21:05.715 DEBUG:boto:Params: {} 2023-05-31T20:21:05.715 DEBUG:boto:Token: None 2023-05-31T20:21:05.716 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:21:05 GMT /pklxkd-48/obj7?versionId=null 2023-05-31T20:21:05.716 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:Svab8zaQiGfx+c9bFgk4QXXvW0E= 2023-05-31T20:21:05.717 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:21:05 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:Svab8zaQiGfx+c9bFgk4QXXvW0E=', 'Content-Length': '0'} 2023-05-31T20:21:05.719 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:20:28 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000804df69afbbca74e-006477ac31-162e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:21:05 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:21:05.719 DEBUG:boto:path=/pklxkd-48/obj7 2023-05-31T20:21:05.720 DEBUG:boto:auth_path=/pklxkd-48/obj7 2023-05-31T20:21:05.720 DEBUG:boto:path=/pklxkd-48/obj7?versionId=null 2023-05-31T20:21:05.721 DEBUG:boto:auth_path=/pklxkd-48/obj7?versionId=null 2023-05-31T20:21:05.721 DEBUG:boto:Method: GET 2023-05-31T20:21:05.721 DEBUG:boto:Path: /pklxkd-48/obj7?versionId=null 2023-05-31T20:21:05.722 DEBUG:boto:Data: 2023-05-31T20:21:05.722 DEBUG:boto:Headers: {} 2023-05-31T20:21:05.722 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:21:05.723 DEBUG:boto:Port: 8000 2023-05-31T20:21:05.723 DEBUG:boto:Params: {} 2023-05-31T20:21:05.723 DEBUG:boto:Token: None 2023-05-31T20:21:05.724 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:21:05 GMT /pklxkd-48/obj7?versionId=null 2023-05-31T20:21:05.724 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:Svab8zaQiGfx+c9bFgk4QXXvW0E= 2023-05-31T20:21:05.725 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:21:05 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:Svab8zaQiGfx+c9bFgk4QXXvW0E=', 'Content-Length': '0'} 2023-05-31T20:21:05.727 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:20:28 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000f0602888b08d59a7-006477ac31-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:21:05 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:21:05.727 DEBUG:boto:path=/pklxkd-48/obj7 2023-05-31T20:21:05.728 DEBUG:boto:auth_path=/pklxkd-48/obj7 2023-05-31T20:21:05.728 DEBUG:boto:path=/pklxkd-48/obj7?versionId=null 2023-05-31T20:21:05.729 DEBUG:boto:auth_path=/pklxkd-48/obj7?versionId=null 2023-05-31T20:21:05.729 DEBUG:boto:Method: HEAD 2023-05-31T20:21:05.730 DEBUG:boto:Path: /pklxkd-48/obj7?versionId=null 2023-05-31T20:21:05.730 DEBUG:boto:Data: 2023-05-31T20:21:05.730 DEBUG:boto:Headers: {} 2023-05-31T20:21:05.731 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:21:05.731 DEBUG:boto:Port: 8001 2023-05-31T20:21:05.731 DEBUG:boto:Params: {} 2023-05-31T20:21:05.732 DEBUG:boto:Token: None 2023-05-31T20:21:05.733 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:21:05 GMT /pklxkd-48/obj7?versionId=null 2023-05-31T20:21:05.733 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:+Z8r+TmwLrblgpAVZ1vlyNDA5B8= 2023-05-31T20:21:05.734 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:21:05 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:+Z8r+TmwLrblgpAVZ1vlyNDA5B8=', 'Content-Length': '0'} 2023-05-31T20:21:05.735 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:20:28 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000b361e8c6f46de3b6-006477ac31-162e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:21:05 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:21:05.736 DEBUG:boto:path=/pklxkd-48/obj7 2023-05-31T20:21:05.737 DEBUG:boto:auth_path=/pklxkd-48/obj7 2023-05-31T20:21:05.737 DEBUG:boto:path=/pklxkd-48/obj7?versionId=null 2023-05-31T20:21:05.737 DEBUG:boto:auth_path=/pklxkd-48/obj7?versionId=null 2023-05-31T20:21:05.738 DEBUG:boto:Method: HEAD 2023-05-31T20:21:05.738 DEBUG:boto:Path: /pklxkd-48/obj7?versionId=null 2023-05-31T20:21:05.739 DEBUG:boto:Data: 2023-05-31T20:21:05.739 DEBUG:boto:Headers: {} 2023-05-31T20:21:05.739 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:21:05.740 DEBUG:boto:Port: 8000 2023-05-31T20:21:05.740 DEBUG:boto:Params: {} 2023-05-31T20:21:05.740 DEBUG:boto:Token: None 2023-05-31T20:21:05.741 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:21:05 GMT /pklxkd-48/obj7?versionId=null 2023-05-31T20:21:05.741 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:+Z8r+TmwLrblgpAVZ1vlyNDA5B8= 2023-05-31T20:21:05.742 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:21:05 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:+Z8r+TmwLrblgpAVZ1vlyNDA5B8=', 'Content-Length': '0'} 2023-05-31T20:21:05.743 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:20:28 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000050b0b563bfe7f445-006477ac31-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:21:05 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:21:05.744 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj7 2023-05-31T20:21:05.745 DEBUG:boto:path=/pklxkd-48/obj7 2023-05-31T20:21:05.745 DEBUG:boto:auth_path=/pklxkd-48/obj7 2023-05-31T20:21:05.745 DEBUG:boto:path=/pklxkd-48/obj7?versionId=null 2023-05-31T20:21:05.746 DEBUG:boto:auth_path=/pklxkd-48/obj7?versionId=null 2023-05-31T20:21:05.746 DEBUG:boto:Method: GET 2023-05-31T20:21:05.746 DEBUG:boto:Path: /pklxkd-48/obj7?versionId=null 2023-05-31T20:21:05.747 DEBUG:boto:Data: 2023-05-31T20:21:05.747 DEBUG:boto:Headers: {} 2023-05-31T20:21:05.747 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:21:05.748 DEBUG:boto:Port: 8001 2023-05-31T20:21:05.748 DEBUG:boto:Params: {} 2023-05-31T20:21:05.749 DEBUG:boto:Token: None 2023-05-31T20:21:05.749 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:21:05 GMT /pklxkd-48/obj7?versionId=null 2023-05-31T20:21:05.749 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:Svab8zaQiGfx+c9bFgk4QXXvW0E= 2023-05-31T20:21:05.750 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:21:05 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:Svab8zaQiGfx+c9bFgk4QXXvW0E=', 'Content-Length': '0'} 2023-05-31T20:21:05.752 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:20:28 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000a5fd459495d3d1e0-006477ac31-162e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:21:05 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:21:05.753 DEBUG:boto:path=/pklxkd-48/obj7 2023-05-31T20:21:05.753 DEBUG:boto:auth_path=/pklxkd-48/obj7 2023-05-31T20:21:05.754 DEBUG:boto:path=/pklxkd-48/obj7?versionId=null 2023-05-31T20:21:05.754 DEBUG:boto:auth_path=/pklxkd-48/obj7?versionId=null 2023-05-31T20:21:05.754 DEBUG:boto:Method: GET 2023-05-31T20:21:05.755 DEBUG:boto:Path: /pklxkd-48/obj7?versionId=null 2023-05-31T20:21:05.755 DEBUG:boto:Data: 2023-05-31T20:21:05.755 DEBUG:boto:Headers: {} 2023-05-31T20:21:05.756 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:21:05.756 DEBUG:boto:Port: 8000 2023-05-31T20:21:05.757 DEBUG:boto:Params: {} 2023-05-31T20:21:05.758 DEBUG:boto:Token: None 2023-05-31T20:21:05.759 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:21:05 GMT /pklxkd-48/obj7?versionId=null 2023-05-31T20:21:05.761 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:Svab8zaQiGfx+c9bFgk4QXXvW0E= 2023-05-31T20:21:05.762 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:21:05 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:Svab8zaQiGfx+c9bFgk4QXXvW0E=', 'Content-Length': '0'} 2023-05-31T20:21:05.765 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:20:28 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000222431a363dd686e-006477ac31-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:21:05 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:21:05.766 DEBUG:boto:path=/pklxkd-48/ 2023-05-31T20:21:05.767 DEBUG:boto:auth_path=/pklxkd-48/ 2023-05-31T20:21:05.767 DEBUG:boto:path=/pklxkd-48/?prefix=obj7 2023-05-31T20:21:05.768 DEBUG:boto:auth_path=/pklxkd-48/?prefix=obj7 2023-05-31T20:21:05.769 DEBUG:boto:Method: GET 2023-05-31T20:21:05.770 DEBUG:boto:Path: /pklxkd-48/?prefix=obj7 2023-05-31T20:21:05.770 DEBUG:boto:Data: 2023-05-31T20:21:05.771 DEBUG:boto:Headers: {} 2023-05-31T20:21:05.771 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:21:05.772 DEBUG:boto:Port: 8001 2023-05-31T20:21:05.772 DEBUG:boto:Params: {} 2023-05-31T20:21:05.773 DEBUG:boto:Token: None 2023-05-31T20:21:05.773 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:21:05 GMT /pklxkd-48/ 2023-05-31T20:21:05.774 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:3ZoBJqr7pRMjTmfx7u+QyvQHg74= 2023-05-31T20:21:05.774 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:21:05 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:3ZoBJqr7pRMjTmfx7u+QyvQHg74=', 'Content-Length': '0'} 2023-05-31T20:21:05.777 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b80c3c3ff660f479-006477ac31-162e-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:21:05 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:21:05.824 DEBUG:boto:b'pklxkd-48obj71000falseobj72023-05-31T20:20:28.722Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:21:05.826 DEBUG:boto:path=/pklxkd-48/ 2023-05-31T20:21:05.827 DEBUG:boto:auth_path=/pklxkd-48/ 2023-05-31T20:21:05.827 DEBUG:boto:path=/pklxkd-48/?prefix=obj7 2023-05-31T20:21:05.828 DEBUG:boto:auth_path=/pklxkd-48/?prefix=obj7 2023-05-31T20:21:05.829 DEBUG:boto:Method: GET 2023-05-31T20:21:05.830 DEBUG:boto:Path: /pklxkd-48/?prefix=obj7 2023-05-31T20:21:05.830 DEBUG:boto:Data: 2023-05-31T20:21:05.831 DEBUG:boto:Headers: {} 2023-05-31T20:21:05.832 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:21:05.833 DEBUG:boto:Port: 8000 2023-05-31T20:21:05.833 DEBUG:boto:Params: {} 2023-05-31T20:21:05.834 DEBUG:boto:Token: None 2023-05-31T20:21:05.835 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:21:05 GMT /pklxkd-48/ 2023-05-31T20:21:05.836 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:3ZoBJqr7pRMjTmfx7u+QyvQHg74= 2023-05-31T20:21:05.837 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:21:05 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:3ZoBJqr7pRMjTmfx7u+QyvQHg74=', 'Content-Length': '0'} 2023-05-31T20:21:05.840 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c59af61feac0e06e-006477ac31-1630-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:21:05 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:21:05.841 DEBUG:boto:b'pklxkd-48obj71000falseobj72023-05-31T20:20:28.722Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:21:05.843 DEBUG:boto:path=/pklxkd-48/obj7 2023-05-31T20:21:05.844 DEBUG:boto:auth_path=/pklxkd-48/obj7 2023-05-31T20:21:05.845 DEBUG:boto:Method: HEAD 2023-05-31T20:21:05.846 DEBUG:boto:Path: /pklxkd-48/obj7 2023-05-31T20:21:05.846 DEBUG:boto:Data: 2023-05-31T20:21:05.847 DEBUG:boto:Headers: {} 2023-05-31T20:21:05.848 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:21:05.848 DEBUG:boto:Port: 8001 2023-05-31T20:21:05.849 DEBUG:boto:Params: {} 2023-05-31T20:21:05.850 DEBUG:boto:Token: None 2023-05-31T20:21:05.851 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:21:05 GMT /pklxkd-48/obj7 2023-05-31T20:21:05.851 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:nKWba2UoqXIMtckBjcJq6eCPpnM= 2023-05-31T20:21:05.852 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:21:05 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:nKWba2UoqXIMtckBjcJq6eCPpnM=', 'Content-Length': '0'} 2023-05-31T20:21:05.854 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:20:28 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000c172bc6aa1984d33-006477ac31-162e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:21:05 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:21:05.854 DEBUG:boto:path=/pklxkd-48/obj7 2023-05-31T20:21:05.855 DEBUG:boto:auth_path=/pklxkd-48/obj7 2023-05-31T20:21:05.855 DEBUG:boto:Method: HEAD 2023-05-31T20:21:05.855 DEBUG:boto:Path: /pklxkd-48/obj7 2023-05-31T20:21:05.856 DEBUG:boto:Data: 2023-05-31T20:21:05.856 DEBUG:boto:Headers: {} 2023-05-31T20:21:05.857 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:21:05.857 DEBUG:boto:Port: 8000 2023-05-31T20:21:05.857 DEBUG:boto:Params: {} 2023-05-31T20:21:05.858 DEBUG:boto:Token: None 2023-05-31T20:21:05.858 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:21:05 GMT /pklxkd-48/obj7 2023-05-31T20:21:05.859 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:nKWba2UoqXIMtckBjcJq6eCPpnM= 2023-05-31T20:21:05.859 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:21:05 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:nKWba2UoqXIMtckBjcJq6eCPpnM=', 'Content-Length': '0'} 2023-05-31T20:21:05.861 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:20:28 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000239f4a9f4f3b96cb-006477ac31-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:21:05 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:21:05.862 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj7 2023-05-31T20:21:05.862 DEBUG:boto:path=/pklxkd-48/obj7 2023-05-31T20:21:05.862 DEBUG:boto:auth_path=/pklxkd-48/obj7 2023-05-31T20:21:05.863 DEBUG:boto:Method: GET 2023-05-31T20:21:05.863 DEBUG:boto:Path: /pklxkd-48/obj7 2023-05-31T20:21:05.864 DEBUG:boto:Data: 2023-05-31T20:21:05.864 DEBUG:boto:Headers: {} 2023-05-31T20:21:05.864 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:21:05.865 DEBUG:boto:Port: 8001 2023-05-31T20:21:05.865 DEBUG:boto:Params: {} 2023-05-31T20:21:05.866 DEBUG:boto:Token: None 2023-05-31T20:21:05.866 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:21:05 GMT /pklxkd-48/obj7 2023-05-31T20:21:05.867 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:psMMZxNy6LbLAb1fEZLIhPgLblw= 2023-05-31T20:21:05.867 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:21:05 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:psMMZxNy6LbLAb1fEZLIhPgLblw=', 'Content-Length': '0'} 2023-05-31T20:21:05.869 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:20:28 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000017c86e1df04988b7-006477ac31-162e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:21:05 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:21:05.869 DEBUG:boto:path=/pklxkd-48/obj7 2023-05-31T20:21:05.870 DEBUG:boto:auth_path=/pklxkd-48/obj7 2023-05-31T20:21:05.870 DEBUG:boto:Method: GET 2023-05-31T20:21:05.871 DEBUG:boto:Path: /pklxkd-48/obj7 2023-05-31T20:21:05.871 DEBUG:boto:Data: 2023-05-31T20:21:05.872 DEBUG:boto:Headers: {} 2023-05-31T20:21:05.872 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:21:05.872 DEBUG:boto:Port: 8000 2023-05-31T20:21:05.873 DEBUG:boto:Params: {} 2023-05-31T20:21:05.874 DEBUG:boto:Token: None 2023-05-31T20:21:05.874 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:21:05 GMT /pklxkd-48/obj7 2023-05-31T20:21:05.874 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:psMMZxNy6LbLAb1fEZLIhPgLblw= 2023-05-31T20:21:05.875 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:21:05 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:psMMZxNy6LbLAb1fEZLIhPgLblw=', 'Content-Length': '0'} 2023-05-31T20:21:05.877 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:20:28 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000000475f4a62649f035-006477ac31-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:21:05 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:21:05.920 DEBUG:boto:path=/pklxkd-48/ 2023-05-31T20:21:05.921 DEBUG:boto:auth_path=/pklxkd-48/ 2023-05-31T20:21:05.921 DEBUG:boto:path=/pklxkd-48/?prefix=obj7 2023-05-31T20:21:05.922 DEBUG:boto:auth_path=/pklxkd-48/?prefix=obj7 2023-05-31T20:21:05.922 DEBUG:boto:Method: GET 2023-05-31T20:21:05.923 DEBUG:boto:Path: /pklxkd-48/?prefix=obj7 2023-05-31T20:21:05.923 DEBUG:boto:Data: 2023-05-31T20:21:05.924 DEBUG:boto:Headers: {} 2023-05-31T20:21:05.924 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:21:05.925 DEBUG:boto:Port: 8001 2023-05-31T20:21:05.925 DEBUG:boto:Params: {} 2023-05-31T20:21:05.926 DEBUG:boto:Token: None 2023-05-31T20:21:05.927 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:21:05 GMT /pklxkd-48/ 2023-05-31T20:21:05.927 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:3ZoBJqr7pRMjTmfx7u+QyvQHg74= 2023-05-31T20:21:05.928 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:21:05 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:3ZoBJqr7pRMjTmfx7u+QyvQHg74=', 'Content-Length': '0'} 2023-05-31T20:21:05.930 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e58703776fd0e49d-006477ac31-162e-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:21:05 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:21:05.931 DEBUG:boto:b'pklxkd-48obj71000falseobj72023-05-31T20:20:28.722Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:21:05.931 DEBUG:boto:path=/pklxkd-48/ 2023-05-31T20:21:05.932 DEBUG:boto:auth_path=/pklxkd-48/ 2023-05-31T20:21:05.932 DEBUG:boto:path=/pklxkd-48/?prefix=obj7 2023-05-31T20:21:05.933 DEBUG:boto:auth_path=/pklxkd-48/?prefix=obj7 2023-05-31T20:21:05.934 DEBUG:boto:Method: GET 2023-05-31T20:21:05.934 DEBUG:boto:Path: /pklxkd-48/?prefix=obj7 2023-05-31T20:21:05.935 DEBUG:boto:Data: 2023-05-31T20:21:05.936 DEBUG:boto:Headers: {} 2023-05-31T20:21:05.936 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:21:05.937 DEBUG:boto:Port: 8000 2023-05-31T20:21:05.937 DEBUG:boto:Params: {} 2023-05-31T20:21:05.938 DEBUG:boto:Token: None 2023-05-31T20:21:05.938 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:21:05 GMT /pklxkd-48/ 2023-05-31T20:21:05.939 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:3ZoBJqr7pRMjTmfx7u+QyvQHg74= 2023-05-31T20:21:05.939 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:21:05 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:3ZoBJqr7pRMjTmfx7u+QyvQHg74=', 'Content-Length': '0'} 2023-05-31T20:21:05.942 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ff987604667a431c-006477ac31-1630-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:21:05 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:21:05.943 DEBUG:boto:b'pklxkd-48obj71000falseobj72023-05-31T20:20:28.722Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:21:05.944 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj8 2023-05-31T20:21:05.945 DEBUG:boto:path=/pklxkd-48/obj8 2023-05-31T20:21:05.945 DEBUG:boto:auth_path=/pklxkd-48/obj8 2023-05-31T20:21:05.946 DEBUG:boto:path=/pklxkd-48/obj8?versionId=null 2023-05-31T20:21:05.946 DEBUG:boto:auth_path=/pklxkd-48/obj8?versionId=null 2023-05-31T20:21:05.947 DEBUG:boto:Method: GET 2023-05-31T20:21:05.947 DEBUG:boto:Path: /pklxkd-48/obj8?versionId=null 2023-05-31T20:21:05.948 DEBUG:boto:Data: 2023-05-31T20:21:05.948 DEBUG:boto:Headers: {} 2023-05-31T20:21:05.949 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:21:05.949 DEBUG:boto:Port: 8001 2023-05-31T20:21:05.949 DEBUG:boto:Params: {} 2023-05-31T20:21:05.950 DEBUG:boto:Token: None 2023-05-31T20:21:05.951 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:21:05 GMT /pklxkd-48/obj8?versionId=null 2023-05-31T20:21:05.952 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:BmE538Wy++bK3OJljanTkk3AZw8= 2023-05-31T20:21:05.954 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:21:05 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:BmE538Wy++bK3OJljanTkk3AZw8=', 'Content-Length': '0'} 2023-05-31T20:21:05.956 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:20:28 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000433cac6efbc7f0a6-006477ac31-162e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:21:05 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:21:05.957 DEBUG:boto:path=/pklxkd-48/obj8 2023-05-31T20:21:05.958 DEBUG:boto:auth_path=/pklxkd-48/obj8 2023-05-31T20:21:05.958 DEBUG:boto:path=/pklxkd-48/obj8?versionId=null 2023-05-31T20:21:05.958 DEBUG:boto:auth_path=/pklxkd-48/obj8?versionId=null 2023-05-31T20:21:05.959 DEBUG:boto:Method: GET 2023-05-31T20:21:05.959 DEBUG:boto:Path: /pklxkd-48/obj8?versionId=null 2023-05-31T20:21:05.960 DEBUG:boto:Data: 2023-05-31T20:21:05.960 DEBUG:boto:Headers: {} 2023-05-31T20:21:05.960 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:21:05.961 DEBUG:boto:Port: 8000 2023-05-31T20:21:05.962 DEBUG:boto:Params: {} 2023-05-31T20:21:05.963 DEBUG:boto:Token: None 2023-05-31T20:21:05.965 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:21:05 GMT /pklxkd-48/obj8?versionId=null 2023-05-31T20:21:05.966 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:BmE538Wy++bK3OJljanTkk3AZw8= 2023-05-31T20:21:05.967 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:21:05 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:BmE538Wy++bK3OJljanTkk3AZw8=', 'Content-Length': '0'} 2023-05-31T20:21:05.969 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:20:28 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000051d81b84be8e5dfe-006477ac31-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:21:05 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:21:05.970 DEBUG:boto:path=/pklxkd-48/obj8 2023-05-31T20:21:05.970 DEBUG:boto:auth_path=/pklxkd-48/obj8 2023-05-31T20:21:05.971 DEBUG:boto:path=/pklxkd-48/obj8?versionId=null 2023-05-31T20:21:05.971 DEBUG:boto:auth_path=/pklxkd-48/obj8?versionId=null 2023-05-31T20:21:05.971 DEBUG:boto:Method: HEAD 2023-05-31T20:21:05.972 DEBUG:boto:Path: /pklxkd-48/obj8?versionId=null 2023-05-31T20:21:05.972 DEBUG:boto:Data: 2023-05-31T20:21:05.973 DEBUG:boto:Headers: {} 2023-05-31T20:21:05.973 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:21:05.974 DEBUG:boto:Port: 8001 2023-05-31T20:21:05.974 DEBUG:boto:Params: {} 2023-05-31T20:21:05.974 DEBUG:boto:Token: None 2023-05-31T20:21:05.975 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:21:05 GMT /pklxkd-48/obj8?versionId=null 2023-05-31T20:21:05.975 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:J6llwTAT9nYhr0uzKnjZ/4cAMGs= 2023-05-31T20:21:05.976 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:21:05 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:J6llwTAT9nYhr0uzKnjZ/4cAMGs=', 'Content-Length': '0'} 2023-05-31T20:21:05.977 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:20:28 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000071f329b8dea5779a-006477ac31-162e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:21:05 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:21:05.978 DEBUG:boto:path=/pklxkd-48/obj8 2023-05-31T20:21:05.978 DEBUG:boto:auth_path=/pklxkd-48/obj8 2023-05-31T20:21:05.979 DEBUG:boto:path=/pklxkd-48/obj8?versionId=null 2023-05-31T20:21:05.979 DEBUG:boto:auth_path=/pklxkd-48/obj8?versionId=null 2023-05-31T20:21:05.979 DEBUG:boto:Method: HEAD 2023-05-31T20:21:05.980 DEBUG:boto:Path: /pklxkd-48/obj8?versionId=null 2023-05-31T20:21:05.980 DEBUG:boto:Data: 2023-05-31T20:21:05.980 DEBUG:boto:Headers: {} 2023-05-31T20:21:05.981 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:21:05.981 DEBUG:boto:Port: 8000 2023-05-31T20:21:05.982 DEBUG:boto:Params: {} 2023-05-31T20:21:05.982 DEBUG:boto:Token: None 2023-05-31T20:21:05.983 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:21:05 GMT /pklxkd-48/obj8?versionId=null 2023-05-31T20:21:05.983 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:J6llwTAT9nYhr0uzKnjZ/4cAMGs= 2023-05-31T20:21:05.983 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:21:05 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:J6llwTAT9nYhr0uzKnjZ/4cAMGs=', 'Content-Length': '0'} 2023-05-31T20:21:05.985 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:20:28 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000c609bd54bab6538c-006477ac31-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:21:05 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:21:05.986 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj8 2023-05-31T20:21:05.986 DEBUG:boto:path=/pklxkd-48/obj8 2023-05-31T20:21:05.987 DEBUG:boto:auth_path=/pklxkd-48/obj8 2023-05-31T20:21:05.987 DEBUG:boto:path=/pklxkd-48/obj8?versionId=null 2023-05-31T20:21:05.987 DEBUG:boto:auth_path=/pklxkd-48/obj8?versionId=null 2023-05-31T20:21:05.988 DEBUG:boto:Method: GET 2023-05-31T20:21:05.988 DEBUG:boto:Path: /pklxkd-48/obj8?versionId=null 2023-05-31T20:21:05.989 DEBUG:boto:Data: 2023-05-31T20:21:05.989 DEBUG:boto:Headers: {} 2023-05-31T20:21:05.989 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:21:05.990 DEBUG:boto:Port: 8001 2023-05-31T20:21:05.990 DEBUG:boto:Params: {} 2023-05-31T20:21:05.991 DEBUG:boto:Token: None 2023-05-31T20:21:05.991 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:21:05 GMT /pklxkd-48/obj8?versionId=null 2023-05-31T20:21:05.991 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:BmE538Wy++bK3OJljanTkk3AZw8= 2023-05-31T20:21:05.992 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:21:05 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:BmE538Wy++bK3OJljanTkk3AZw8=', 'Content-Length': '0'} 2023-05-31T20:21:05.994 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:20:28 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000adfcdaac2efff95f-006477ac31-162e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:21:05 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:21:05.996 DEBUG:boto:path=/pklxkd-48/obj8 2023-05-31T20:21:05.996 DEBUG:boto:auth_path=/pklxkd-48/obj8 2023-05-31T20:21:05.996 DEBUG:boto:path=/pklxkd-48/obj8?versionId=null 2023-05-31T20:21:05.997 DEBUG:boto:auth_path=/pklxkd-48/obj8?versionId=null 2023-05-31T20:21:05.997 DEBUG:boto:Method: GET 2023-05-31T20:21:05.998 DEBUG:boto:Path: /pklxkd-48/obj8?versionId=null 2023-05-31T20:21:05.998 DEBUG:boto:Data: 2023-05-31T20:21:05.999 DEBUG:boto:Headers: {} 2023-05-31T20:21:05.999 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:21:05.999 DEBUG:boto:Port: 8000 2023-05-31T20:21:06.000 DEBUG:boto:Params: {} 2023-05-31T20:21:06.000 DEBUG:boto:Token: None 2023-05-31T20:21:06.001 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:21:06 GMT /pklxkd-48/obj8?versionId=null 2023-05-31T20:21:06.001 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:dhCn+wAQvihwDgHIBIJeaCIhG14= 2023-05-31T20:21:06.002 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:21:06 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:dhCn+wAQvihwDgHIBIJeaCIhG14=', 'Content-Length': '0'} 2023-05-31T20:21:06.004 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:20:28 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000b2fa3f6b4acd1c59-006477ac31-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:21:06 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:21:06.005 DEBUG:boto:path=/pklxkd-48/ 2023-05-31T20:21:06.006 DEBUG:boto:auth_path=/pklxkd-48/ 2023-05-31T20:21:06.006 DEBUG:boto:path=/pklxkd-48/?prefix=obj8 2023-05-31T20:21:06.006 DEBUG:boto:auth_path=/pklxkd-48/?prefix=obj8 2023-05-31T20:21:06.007 DEBUG:boto:Method: GET 2023-05-31T20:21:06.007 DEBUG:boto:Path: /pklxkd-48/?prefix=obj8 2023-05-31T20:21:06.007 DEBUG:boto:Data: 2023-05-31T20:21:06.008 DEBUG:boto:Headers: {} 2023-05-31T20:21:06.008 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:21:06.009 DEBUG:boto:Port: 8001 2023-05-31T20:21:06.009 DEBUG:boto:Params: {} 2023-05-31T20:21:06.009 DEBUG:boto:Token: None 2023-05-31T20:21:06.010 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:21:06 GMT /pklxkd-48/ 2023-05-31T20:21:06.011 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:+oul6x5kl58ir8XKXBQXKwLGCSw= 2023-05-31T20:21:06.011 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:21:06 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:+oul6x5kl58ir8XKXBQXKwLGCSw=', 'Content-Length': '0'} 2023-05-31T20:21:06.014 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000efc867805ce874bc-006477ac32-162e-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:21:06 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:21:06.015 DEBUG:boto:b'pklxkd-48obj81000falseobj82023-05-31T20:20:28.779Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:21:06.016 DEBUG:boto:path=/pklxkd-48/ 2023-05-31T20:21:06.017 DEBUG:boto:auth_path=/pklxkd-48/ 2023-05-31T20:21:06.017 DEBUG:boto:path=/pklxkd-48/?prefix=obj8 2023-05-31T20:21:06.018 DEBUG:boto:auth_path=/pklxkd-48/?prefix=obj8 2023-05-31T20:21:06.018 DEBUG:boto:Method: GET 2023-05-31T20:21:06.018 DEBUG:boto:Path: /pklxkd-48/?prefix=obj8 2023-05-31T20:21:06.019 DEBUG:boto:Data: 2023-05-31T20:21:06.019 DEBUG:boto:Headers: {} 2023-05-31T20:21:06.019 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:21:06.020 DEBUG:boto:Port: 8000 2023-05-31T20:21:06.020 DEBUG:boto:Params: {} 2023-05-31T20:21:06.021 DEBUG:boto:Token: None 2023-05-31T20:21:06.022 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:21:06 GMT /pklxkd-48/ 2023-05-31T20:21:06.023 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:+oul6x5kl58ir8XKXBQXKwLGCSw= 2023-05-31T20:21:06.024 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:21:06 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:+oul6x5kl58ir8XKXBQXKwLGCSw=', 'Content-Length': '0'} 2023-05-31T20:21:06.027 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c22004497f76315b-006477ac32-1630-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:21:06 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:21:06.068 DEBUG:boto:b'pklxkd-48obj81000falseobj82023-05-31T20:20:28.779Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:21:06.069 DEBUG:boto:path=/pklxkd-48/obj8 2023-05-31T20:21:06.070 DEBUG:boto:auth_path=/pklxkd-48/obj8 2023-05-31T20:21:06.070 DEBUG:boto:Method: HEAD 2023-05-31T20:21:06.070 DEBUG:boto:Path: /pklxkd-48/obj8 2023-05-31T20:21:06.071 DEBUG:boto:Data: 2023-05-31T20:21:06.071 DEBUG:boto:Headers: {} 2023-05-31T20:21:06.072 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:21:06.072 DEBUG:boto:Port: 8001 2023-05-31T20:21:06.072 DEBUG:boto:Params: {} 2023-05-31T20:21:06.073 DEBUG:boto:Token: None 2023-05-31T20:21:06.073 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:21:06 GMT /pklxkd-48/obj8 2023-05-31T20:21:06.073 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:dZB7NoiigX+s1oY6zjeNTDx3S+4= 2023-05-31T20:21:06.074 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:21:06 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:dZB7NoiigX+s1oY6zjeNTDx3S+4=', 'Content-Length': '0'} 2023-05-31T20:21:06.076 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:20:28 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000c753e98f0ec90d48-006477ac32-162e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:21:06 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:21:06.076 DEBUG:boto:path=/pklxkd-48/obj8 2023-05-31T20:21:06.077 DEBUG:boto:auth_path=/pklxkd-48/obj8 2023-05-31T20:21:06.077 DEBUG:boto:Method: HEAD 2023-05-31T20:21:06.077 DEBUG:boto:Path: /pklxkd-48/obj8 2023-05-31T20:21:06.078 DEBUG:boto:Data: 2023-05-31T20:21:06.078 DEBUG:boto:Headers: {} 2023-05-31T20:21:06.078 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:21:06.079 DEBUG:boto:Port: 8000 2023-05-31T20:21:06.079 DEBUG:boto:Params: {} 2023-05-31T20:21:06.079 DEBUG:boto:Token: None 2023-05-31T20:21:06.080 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:21:06 GMT /pklxkd-48/obj8 2023-05-31T20:21:06.080 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:dZB7NoiigX+s1oY6zjeNTDx3S+4= 2023-05-31T20:21:06.081 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:21:06 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:dZB7NoiigX+s1oY6zjeNTDx3S+4=', 'Content-Length': '0'} 2023-05-31T20:21:06.084 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:20:28 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000018eea1e251d20e32-006477ac32-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:21:06 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:21:06.085 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj8 2023-05-31T20:21:06.085 DEBUG:boto:path=/pklxkd-48/obj8 2023-05-31T20:21:06.086 DEBUG:boto:auth_path=/pklxkd-48/obj8 2023-05-31T20:21:06.086 DEBUG:boto:Method: GET 2023-05-31T20:21:06.086 DEBUG:boto:Path: /pklxkd-48/obj8 2023-05-31T20:21:06.087 DEBUG:boto:Data: 2023-05-31T20:21:06.087 DEBUG:boto:Headers: {} 2023-05-31T20:21:06.087 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:21:06.088 DEBUG:boto:Port: 8001 2023-05-31T20:21:06.088 DEBUG:boto:Params: {} 2023-05-31T20:21:06.088 DEBUG:boto:Token: None 2023-05-31T20:21:06.089 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:21:06 GMT /pklxkd-48/obj8 2023-05-31T20:21:06.089 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:ACuWdOoSzYKAs4Vye7RCIOe6M7g= 2023-05-31T20:21:06.090 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:21:06 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:ACuWdOoSzYKAs4Vye7RCIOe6M7g=', 'Content-Length': '0'} 2023-05-31T20:21:06.092 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:20:28 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000005ba4367b055a1526-006477ac32-162e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:21:06 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:21:06.093 DEBUG:boto:path=/pklxkd-48/obj8 2023-05-31T20:21:06.093 DEBUG:boto:auth_path=/pklxkd-48/obj8 2023-05-31T20:21:06.094 DEBUG:boto:Method: GET 2023-05-31T20:21:06.094 DEBUG:boto:Path: /pklxkd-48/obj8 2023-05-31T20:21:06.094 DEBUG:boto:Data: 2023-05-31T20:21:06.095 DEBUG:boto:Headers: {} 2023-05-31T20:21:06.095 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:21:06.096 DEBUG:boto:Port: 8000 2023-05-31T20:21:06.096 DEBUG:boto:Params: {} 2023-05-31T20:21:06.097 DEBUG:boto:Token: None 2023-05-31T20:21:06.097 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:21:06 GMT /pklxkd-48/obj8 2023-05-31T20:21:06.097 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:ACuWdOoSzYKAs4Vye7RCIOe6M7g= 2023-05-31T20:21:06.098 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:21:06 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:ACuWdOoSzYKAs4Vye7RCIOe6M7g=', 'Content-Length': '0'} 2023-05-31T20:21:06.100 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:20:28 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000094df595cd34cd021-006477ac32-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:21:06 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:21:06.101 DEBUG:boto:path=/pklxkd-48/ 2023-05-31T20:21:06.101 DEBUG:boto:auth_path=/pklxkd-48/ 2023-05-31T20:21:06.101 DEBUG:boto:path=/pklxkd-48/?prefix=obj8 2023-05-31T20:21:06.102 DEBUG:boto:auth_path=/pklxkd-48/?prefix=obj8 2023-05-31T20:21:06.102 DEBUG:boto:Method: GET 2023-05-31T20:21:06.103 DEBUG:boto:Path: /pklxkd-48/?prefix=obj8 2023-05-31T20:21:06.103 DEBUG:boto:Data: 2023-05-31T20:21:06.103 DEBUG:boto:Headers: {} 2023-05-31T20:21:06.104 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:21:06.104 DEBUG:boto:Port: 8001 2023-05-31T20:21:06.104 DEBUG:boto:Params: {} 2023-05-31T20:21:06.106 DEBUG:boto:Token: None 2023-05-31T20:21:06.106 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:21:06 GMT /pklxkd-48/ 2023-05-31T20:21:06.106 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:+oul6x5kl58ir8XKXBQXKwLGCSw= 2023-05-31T20:21:06.107 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:21:06 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:+oul6x5kl58ir8XKXBQXKwLGCSw=', 'Content-Length': '0'} 2023-05-31T20:21:06.112 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000027e206c3b0ccab37-006477ac32-162e-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:21:06 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:21:06.156 DEBUG:boto:b'pklxkd-48obj81000falseobj82023-05-31T20:20:28.779Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:21:06.157 DEBUG:boto:path=/pklxkd-48/ 2023-05-31T20:21:06.158 DEBUG:boto:auth_path=/pklxkd-48/ 2023-05-31T20:21:06.158 DEBUG:boto:path=/pklxkd-48/?prefix=obj8 2023-05-31T20:21:06.158 DEBUG:boto:auth_path=/pklxkd-48/?prefix=obj8 2023-05-31T20:21:06.159 DEBUG:boto:Method: GET 2023-05-31T20:21:06.159 DEBUG:boto:Path: /pklxkd-48/?prefix=obj8 2023-05-31T20:21:06.160 DEBUG:boto:Data: 2023-05-31T20:21:06.160 DEBUG:boto:Headers: {} 2023-05-31T20:21:06.160 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:21:06.161 DEBUG:boto:Port: 8000 2023-05-31T20:21:06.161 DEBUG:boto:Params: {} 2023-05-31T20:21:06.161 DEBUG:boto:Token: None 2023-05-31T20:21:06.162 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:21:06 GMT /pklxkd-48/ 2023-05-31T20:21:06.162 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:+oul6x5kl58ir8XKXBQXKwLGCSw= 2023-05-31T20:21:06.163 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:21:06 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:+oul6x5kl58ir8XKXBQXKwLGCSw=', 'Content-Length': '0'} 2023-05-31T20:21:06.165 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000004f74582200bc37ec-006477ac32-1630-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:21:06 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:21:06.166 DEBUG:boto:b'pklxkd-48obj81000falseobj82023-05-31T20:20:28.779Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:21:06.167 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=pklxkd-48 zones={a2, a1} 2023-05-31T20:21:06.167 DEBUG:rgw_multi.tests:bucket checkpoint: source=a1 target=a2 bucket=pklxkd-49 2023-05-31T20:21:06.168 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'pklxkd-49', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-05-31T20:21:06.169 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'pklxkd-49', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-05-31T20:21:06.169 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket pklxkd-49 --source-zone a1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2023-05-31T20:21:06.234 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2023-05-31T20:21:06.235 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2023-05-31T20:21:06.286 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:21:06.277+0000 7f6c053f2780 1 bucket sync caught up with source: 2023-05-31T20:21:06.287 INFO:teuthology.orchestra.run.smithi099.stderr: local status: [00000000004.62.6, 00000000002.18.1, 00000000004.37.6, 00000000002.28.1, 00000000004.24.6, 00000000002.11.1, 00000000002.28.1, 00000000004.33.6, 00000000002.19.1, 00000000002.31.1, 00000000002.35.1] 2023-05-31T20:21:06.287 INFO:teuthology.orchestra.run.smithi099.stderr: remote markers: [00000000004.62.6, 00000000002.18.1, 00000000004.37.6, 00000000002.28.1, 00000000004.24.6, 00000000002.11.1, 00000000002.28.1, 00000000004.33.6, 00000000002.19.1, 00000000002.31.1, 00000000002.35.1] 2023-05-31T20:21:06.287 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:21:06.277+0000 7f6c053f2780 0 bucket checkpoint complete 2023-05-31T20:21:06.299 DEBUG:rgw_multi.tests:bucket checkpoint: source=a2 target=a1 bucket=pklxkd-49 2023-05-31T20:21:06.299 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'pklxkd-49', '--source-zone', 'a2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-05-31T20:21:06.300 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'pklxkd-49', '--source-zone', 'a2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-05-31T20:21:06.300 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket pklxkd-49 --source-zone a2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2023-05-31T20:21:06.357 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2023-05-31T20:21:06.358 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2023-05-31T20:21:06.414 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:21:06.409+0000 7f26da566780 1 bucket sync caught up with source: 2023-05-31T20:21:06.414 INFO:teuthology.orchestra.run.smithi099.stderr: local status: [00000000004.3838.1, 00000000002.982.1, 00000000004.1331.1, 00000000002.2006.1, 00000000004.2239.1, 00000000002.29.1, 00000000002.740.1, 00000000004.1342.1, 00000000002.1904.1, 00000000002.2232.1, 00000000002.1332.1] 2023-05-31T20:21:06.415 INFO:teuthology.orchestra.run.smithi099.stderr: remote markers: [00000000004.3838.1, 00000000002.982.1, 00000000004.1331.1, 00000000002.2006.1, 00000000004.2239.1, 00000000002.29.1, 00000000002.740.1, 00000000004.1342.1, 00000000002.1904.1, 00000000002.2232.1, 00000000002.1332.1] 2023-05-31T20:21:06.415 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:21:06.409+0000 7f26da566780 0 bucket checkpoint complete 2023-05-31T20:21:06.439 INFO:tasks.rgw_multi.zone_rados:comparing bucket=pklxkd-49 zones={a2, a1} 2023-05-31T20:21:06.440 DEBUG:boto:path=/pklxkd-49/ 2023-05-31T20:21:06.440 DEBUG:boto:auth_path=/pklxkd-49/ 2023-05-31T20:21:06.441 DEBUG:boto:Method: HEAD 2023-05-31T20:21:06.441 DEBUG:boto:Path: /pklxkd-49/ 2023-05-31T20:21:06.441 DEBUG:boto:Data: 2023-05-31T20:21:06.442 DEBUG:boto:Headers: {} 2023-05-31T20:21:06.442 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:21:06.442 DEBUG:boto:Port: 8001 2023-05-31T20:21:06.443 DEBUG:boto:Params: {} 2023-05-31T20:21:06.443 DEBUG:boto:Token: None 2023-05-31T20:21:06.443 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:21:06 GMT /pklxkd-49/ 2023-05-31T20:21:06.444 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:dyhvnOnCutJArHvbLY9pm71pgzU= 2023-05-31T20:21:06.444 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:21:06 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:dyhvnOnCutJArHvbLY9pm71pgzU=', 'Content-Length': '0'} 2023-05-31T20:21:06.446 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '8'), ('X-RGW-Bytes-Used', '48'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000069f842d6abf8ca71-006477ac32-162e-a2'), ('Content-Length', '0'), ('Date', 'Wed, 31 May 2023 20:21:06 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:21:06.447 DEBUG:boto:path=/pklxkd-49/ 2023-05-31T20:21:06.447 DEBUG:boto:auth_path=/pklxkd-49/ 2023-05-31T20:21:06.448 DEBUG:boto:Method: HEAD 2023-05-31T20:21:06.448 DEBUG:boto:Path: /pklxkd-49/ 2023-05-31T20:21:06.448 DEBUG:boto:Data: 2023-05-31T20:21:06.449 DEBUG:boto:Headers: {} 2023-05-31T20:21:06.449 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:21:06.449 DEBUG:boto:Port: 8000 2023-05-31T20:21:06.450 DEBUG:boto:Params: {} 2023-05-31T20:21:06.450 DEBUG:boto:Token: None 2023-05-31T20:21:06.450 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:21:06 GMT /pklxkd-49/ 2023-05-31T20:21:06.451 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:dyhvnOnCutJArHvbLY9pm71pgzU= 2023-05-31T20:21:06.451 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:21:06 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:dyhvnOnCutJArHvbLY9pm71pgzU=', 'Content-Length': '0'} 2023-05-31T20:21:06.453 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '8'), ('X-RGW-Bytes-Used', '48'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000000102166cdc14438b-006477ac32-1630-a1'), ('Content-Length', '0'), ('Date', 'Wed, 31 May 2023 20:21:06 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:21:06.454 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2023-05-31T20:21:06.454 DEBUG:boto:path=/pklxkd-49/ 2023-05-31T20:21:06.455 DEBUG:boto:auth_path=/pklxkd-49/ 2023-05-31T20:21:06.455 DEBUG:boto:path=/pklxkd-49/?versions&max-keys=999 2023-05-31T20:21:06.455 DEBUG:boto:auth_path=/pklxkd-49/?versions&max-keys=999 2023-05-31T20:21:06.456 DEBUG:boto:Method: GET 2023-05-31T20:21:06.456 DEBUG:boto:Path: /pklxkd-49/?versions&max-keys=999 2023-05-31T20:21:06.456 DEBUG:boto:Data: 2023-05-31T20:21:06.457 DEBUG:boto:Headers: {} 2023-05-31T20:21:06.457 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:21:06.457 DEBUG:boto:Port: 8001 2023-05-31T20:21:06.458 DEBUG:boto:Params: {} 2023-05-31T20:21:06.458 DEBUG:boto:Token: None 2023-05-31T20:21:06.459 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:21:06 GMT /pklxkd-49/?versions 2023-05-31T20:21:06.459 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:iRRgaBagOQJXpK1fxuh7XIHKADU= 2023-05-31T20:21:06.459 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:21:06 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:iRRgaBagOQJXpK1fxuh7XIHKADU=', 'Content-Length': '0'} 2023-05-31T20:21:06.461 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000075b8472fd3b3e4de-006477ac32-162e-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:21:06 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:21:06.462 DEBUG:boto:b'pklxkd-49999falseobj1nulltrue2023-05-31T20:18:05.410Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj2nulltrue2023-05-31T20:18:05.430Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj3nulltrue2023-05-31T20:18:05.493Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj4nulltrue2023-05-31T20:18:05.509Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj5nulltrue2023-05-31T20:20:28.797Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj6nulltrue2023-05-31T20:20:28.814Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj7nulltrue2023-05-31T20:20:28.830Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj8nulltrue2023-05-31T20:20:28.847Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:21:06.463 DEBUG:tasks.rgw_multi.zone_rados:o=obj1 2023-05-31T20:21:06.463 DEBUG:tasks.rgw_multi.zone_rados:o=obj2 2023-05-31T20:21:06.464 DEBUG:tasks.rgw_multi.zone_rados:o=obj3 2023-05-31T20:21:06.464 DEBUG:tasks.rgw_multi.zone_rados:o=obj4 2023-05-31T20:21:06.464 DEBUG:tasks.rgw_multi.zone_rados:o=obj5 2023-05-31T20:21:06.465 DEBUG:tasks.rgw_multi.zone_rados:o=obj6 2023-05-31T20:21:06.465 DEBUG:tasks.rgw_multi.zone_rados:o=obj7 2023-05-31T20:21:06.465 DEBUG:tasks.rgw_multi.zone_rados:o=obj8 2023-05-31T20:21:06.466 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2023-05-31T20:21:06.466 DEBUG:boto:path=/pklxkd-49/ 2023-05-31T20:21:06.466 DEBUG:boto:auth_path=/pklxkd-49/ 2023-05-31T20:21:06.467 DEBUG:boto:path=/pklxkd-49/?versions&max-keys=999 2023-05-31T20:21:06.467 DEBUG:boto:auth_path=/pklxkd-49/?versions&max-keys=999 2023-05-31T20:21:06.467 DEBUG:boto:Method: GET 2023-05-31T20:21:06.468 DEBUG:boto:Path: /pklxkd-49/?versions&max-keys=999 2023-05-31T20:21:06.468 DEBUG:boto:Data: 2023-05-31T20:21:06.468 DEBUG:boto:Headers: {} 2023-05-31T20:21:06.469 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:21:06.469 DEBUG:boto:Port: 8000 2023-05-31T20:21:06.469 DEBUG:boto:Params: {} 2023-05-31T20:21:06.470 DEBUG:boto:Token: None 2023-05-31T20:21:06.470 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:21:06 GMT /pklxkd-49/?versions 2023-05-31T20:21:06.471 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:iRRgaBagOQJXpK1fxuh7XIHKADU= 2023-05-31T20:21:06.471 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:21:06 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:iRRgaBagOQJXpK1fxuh7XIHKADU=', 'Content-Length': '0'} 2023-05-31T20:21:06.473 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000049af4c9200de379-006477ac32-1630-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:21:06 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:21:06.474 DEBUG:boto:b'pklxkd-49999falseobj1nulltrue2023-05-31T20:18:05.410Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj2nulltrue2023-05-31T20:18:05.430Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj3nulltrue2023-05-31T20:18:05.493Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj4nulltrue2023-05-31T20:18:05.509Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj5nulltrue2023-05-31T20:20:28.797Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj6nulltrue2023-05-31T20:20:28.814Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj7nulltrue2023-05-31T20:20:28.830Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj8nulltrue2023-05-31T20:20:28.847Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:21:06.475 DEBUG:tasks.rgw_multi.zone_rados:o=obj1 2023-05-31T20:21:06.475 DEBUG:tasks.rgw_multi.zone_rados:o=obj2 2023-05-31T20:21:06.475 DEBUG:tasks.rgw_multi.zone_rados:o=obj3 2023-05-31T20:21:06.475 DEBUG:tasks.rgw_multi.zone_rados:o=obj4 2023-05-31T20:21:06.476 DEBUG:tasks.rgw_multi.zone_rados:o=obj5 2023-05-31T20:21:06.476 DEBUG:tasks.rgw_multi.zone_rados:o=obj6 2023-05-31T20:21:06.476 DEBUG:tasks.rgw_multi.zone_rados:o=obj7 2023-05-31T20:21:06.477 DEBUG:tasks.rgw_multi.zone_rados:o=obj8 2023-05-31T20:21:06.477 DEBUG:boto:path=/pklxkd-49/ 2023-05-31T20:21:06.477 DEBUG:boto:auth_path=/pklxkd-49/ 2023-05-31T20:21:06.478 DEBUG:boto:path=/pklxkd-49/?versions&max-keys=999 2023-05-31T20:21:06.478 DEBUG:boto:auth_path=/pklxkd-49/?versions&max-keys=999 2023-05-31T20:21:06.478 DEBUG:boto:Method: GET 2023-05-31T20:21:06.479 DEBUG:boto:Path: /pklxkd-49/?versions&max-keys=999 2023-05-31T20:21:06.479 DEBUG:boto:Data: 2023-05-31T20:21:06.480 DEBUG:boto:Headers: {} 2023-05-31T20:21:06.480 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:21:06.480 DEBUG:boto:Port: 8001 2023-05-31T20:21:06.481 DEBUG:boto:Params: {} 2023-05-31T20:21:06.481 DEBUG:boto:Token: None 2023-05-31T20:21:06.482 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:21:06 GMT /pklxkd-49/?versions 2023-05-31T20:21:06.482 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:iRRgaBagOQJXpK1fxuh7XIHKADU= 2023-05-31T20:21:06.482 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:21:06 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:iRRgaBagOQJXpK1fxuh7XIHKADU=', 'Content-Length': '0'} 2023-05-31T20:21:06.484 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000021ffa90c2768a7a3-006477ac32-162e-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:21:06 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:21:06.485 DEBUG:boto:b'pklxkd-49999falseobj1nulltrue2023-05-31T20:18:05.410Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj2nulltrue2023-05-31T20:18:05.430Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj3nulltrue2023-05-31T20:18:05.493Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj4nulltrue2023-05-31T20:18:05.509Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj5nulltrue2023-05-31T20:20:28.797Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj6nulltrue2023-05-31T20:20:28.814Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj7nulltrue2023-05-31T20:20:28.830Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj8nulltrue2023-05-31T20:20:28.847Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:21:06.486 DEBUG:boto:path=/pklxkd-49/ 2023-05-31T20:21:06.486 DEBUG:boto:auth_path=/pklxkd-49/ 2023-05-31T20:21:06.487 DEBUG:boto:path=/pklxkd-49/?versions&max-keys=999 2023-05-31T20:21:06.487 DEBUG:boto:auth_path=/pklxkd-49/?versions&max-keys=999 2023-05-31T20:21:06.487 DEBUG:boto:Method: GET 2023-05-31T20:21:06.488 DEBUG:boto:Path: /pklxkd-49/?versions&max-keys=999 2023-05-31T20:21:06.488 DEBUG:boto:Data: 2023-05-31T20:21:06.488 DEBUG:boto:Headers: {} 2023-05-31T20:21:06.489 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:21:06.489 DEBUG:boto:Port: 8000 2023-05-31T20:21:06.489 DEBUG:boto:Params: {} 2023-05-31T20:21:06.490 DEBUG:boto:Token: None 2023-05-31T20:21:06.490 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:21:06 GMT /pklxkd-49/?versions 2023-05-31T20:21:06.491 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:iRRgaBagOQJXpK1fxuh7XIHKADU= 2023-05-31T20:21:06.491 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:21:06 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:iRRgaBagOQJXpK1fxuh7XIHKADU=', 'Content-Length': '0'} 2023-05-31T20:21:06.493 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b82ef3e5c463fc99-006477ac32-1630-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:21:06 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:21:06.494 DEBUG:boto:b'pklxkd-49999falseobj1nulltrue2023-05-31T20:18:05.410Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj2nulltrue2023-05-31T20:18:05.430Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj3nulltrue2023-05-31T20:18:05.493Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj4nulltrue2023-05-31T20:18:05.509Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj5nulltrue2023-05-31T20:20:28.797Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj6nulltrue2023-05-31T20:20:28.814Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj7nulltrue2023-05-31T20:20:28.830Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj8nulltrue2023-05-31T20:20:28.847Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:21:06.495 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2023-05-31T20:21:06.495 DEBUG:boto:path=/pklxkd-49/obj1 2023-05-31T20:21:06.496 DEBUG:boto:auth_path=/pklxkd-49/obj1 2023-05-31T20:21:06.496 DEBUG:boto:path=/pklxkd-49/obj1?versionId=null 2023-05-31T20:21:06.496 DEBUG:boto:auth_path=/pklxkd-49/obj1?versionId=null 2023-05-31T20:21:06.497 DEBUG:boto:Method: GET 2023-05-31T20:21:06.497 DEBUG:boto:Path: /pklxkd-49/obj1?versionId=null 2023-05-31T20:21:06.498 DEBUG:boto:Data: 2023-05-31T20:21:06.498 DEBUG:boto:Headers: {} 2023-05-31T20:21:06.498 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:21:06.499 DEBUG:boto:Port: 8001 2023-05-31T20:21:06.499 DEBUG:boto:Params: {} 2023-05-31T20:21:06.500 DEBUG:boto:Token: None 2023-05-31T20:21:06.500 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:21:06 GMT /pklxkd-49/obj1?versionId=null 2023-05-31T20:21:06.501 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:8tsqIcAzVZmkItuo1nqXpWY9O4w= 2023-05-31T20:21:06.501 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:21:06 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:8tsqIcAzVZmkItuo1nqXpWY9O4w=', 'Content-Length': '0'} 2023-05-31T20:21:06.503 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:18:05 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000004838bd9a696e2ed6-006477ac32-162e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:21:06 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:21:06.503 DEBUG:boto:path=/pklxkd-49/obj1 2023-05-31T20:21:06.504 DEBUG:boto:auth_path=/pklxkd-49/obj1 2023-05-31T20:21:06.504 DEBUG:boto:path=/pklxkd-49/obj1?versionId=null 2023-05-31T20:21:06.504 DEBUG:boto:auth_path=/pklxkd-49/obj1?versionId=null 2023-05-31T20:21:06.505 DEBUG:boto:Method: GET 2023-05-31T20:21:06.505 DEBUG:boto:Path: /pklxkd-49/obj1?versionId=null 2023-05-31T20:21:06.505 DEBUG:boto:Data: 2023-05-31T20:21:06.506 DEBUG:boto:Headers: {} 2023-05-31T20:21:06.506 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:21:06.507 DEBUG:boto:Port: 8000 2023-05-31T20:21:06.507 DEBUG:boto:Params: {} 2023-05-31T20:21:06.507 DEBUG:boto:Token: None 2023-05-31T20:21:06.508 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:21:06 GMT /pklxkd-49/obj1?versionId=null 2023-05-31T20:21:06.508 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:8tsqIcAzVZmkItuo1nqXpWY9O4w= 2023-05-31T20:21:06.509 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:21:06 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:8tsqIcAzVZmkItuo1nqXpWY9O4w=', 'Content-Length': '0'} 2023-05-31T20:21:06.511 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:18:05 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000000e363b7fdc20033a-006477ac32-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:21:06 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:21:06.552 DEBUG:boto:path=/pklxkd-49/obj1 2023-05-31T20:21:06.553 DEBUG:boto:auth_path=/pklxkd-49/obj1 2023-05-31T20:21:06.553 DEBUG:boto:path=/pklxkd-49/obj1?versionId=null 2023-05-31T20:21:06.553 DEBUG:boto:auth_path=/pklxkd-49/obj1?versionId=null 2023-05-31T20:21:06.554 DEBUG:boto:Method: HEAD 2023-05-31T20:21:06.554 DEBUG:boto:Path: /pklxkd-49/obj1?versionId=null 2023-05-31T20:21:06.555 DEBUG:boto:Data: 2023-05-31T20:21:06.555 DEBUG:boto:Headers: {} 2023-05-31T20:21:06.555 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:21:06.555 DEBUG:boto:Port: 8001 2023-05-31T20:21:06.556 DEBUG:boto:Params: {} 2023-05-31T20:21:06.556 DEBUG:boto:Token: None 2023-05-31T20:21:06.557 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:21:06 GMT /pklxkd-49/obj1?versionId=null 2023-05-31T20:21:06.557 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:cwYgNdZRfl96uuFZpsdq68WpA/s= 2023-05-31T20:21:06.558 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:21:06 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:cwYgNdZRfl96uuFZpsdq68WpA/s=', 'Content-Length': '0'} 2023-05-31T20:21:06.559 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:18:05 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000372f90cccad97e29-006477ac32-162e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:21:06 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:21:06.560 DEBUG:boto:path=/pklxkd-49/obj1 2023-05-31T20:21:06.560 DEBUG:boto:auth_path=/pklxkd-49/obj1 2023-05-31T20:21:06.561 DEBUG:boto:path=/pklxkd-49/obj1?versionId=null 2023-05-31T20:21:06.561 DEBUG:boto:auth_path=/pklxkd-49/obj1?versionId=null 2023-05-31T20:21:06.561 DEBUG:boto:Method: HEAD 2023-05-31T20:21:06.562 DEBUG:boto:Path: /pklxkd-49/obj1?versionId=null 2023-05-31T20:21:06.562 DEBUG:boto:Data: 2023-05-31T20:21:06.562 DEBUG:boto:Headers: {} 2023-05-31T20:21:06.563 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:21:06.563 DEBUG:boto:Port: 8000 2023-05-31T20:21:06.563 DEBUG:boto:Params: {} 2023-05-31T20:21:06.564 DEBUG:boto:Token: None 2023-05-31T20:21:06.564 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:21:06 GMT /pklxkd-49/obj1?versionId=null 2023-05-31T20:21:06.564 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:cwYgNdZRfl96uuFZpsdq68WpA/s= 2023-05-31T20:21:06.565 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:21:06 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:cwYgNdZRfl96uuFZpsdq68WpA/s=', 'Content-Length': '0'} 2023-05-31T20:21:06.566 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:18:05 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000f857d41e3af2bd30-006477ac32-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:21:06 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:21:06.567 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2023-05-31T20:21:06.567 DEBUG:boto:path=/pklxkd-49/obj1 2023-05-31T20:21:06.568 DEBUG:boto:auth_path=/pklxkd-49/obj1 2023-05-31T20:21:06.568 DEBUG:boto:path=/pklxkd-49/obj1?versionId=null 2023-05-31T20:21:06.568 DEBUG:boto:auth_path=/pklxkd-49/obj1?versionId=null 2023-05-31T20:21:06.569 DEBUG:boto:Method: GET 2023-05-31T20:21:06.569 DEBUG:boto:Path: /pklxkd-49/obj1?versionId=null 2023-05-31T20:21:06.569 DEBUG:boto:Data: 2023-05-31T20:21:06.570 DEBUG:boto:Headers: {} 2023-05-31T20:21:06.570 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:21:06.571 DEBUG:boto:Port: 8001 2023-05-31T20:21:06.571 DEBUG:boto:Params: {} 2023-05-31T20:21:06.571 DEBUG:boto:Token: None 2023-05-31T20:21:06.572 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:21:06 GMT /pklxkd-49/obj1?versionId=null 2023-05-31T20:21:06.572 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:8tsqIcAzVZmkItuo1nqXpWY9O4w= 2023-05-31T20:21:06.573 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:21:06 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:8tsqIcAzVZmkItuo1nqXpWY9O4w=', 'Content-Length': '0'} 2023-05-31T20:21:06.574 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:18:05 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000046fefb9b80769cde-006477ac32-162e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:21:06 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:21:06.575 DEBUG:boto:path=/pklxkd-49/obj1 2023-05-31T20:21:06.575 DEBUG:boto:auth_path=/pklxkd-49/obj1 2023-05-31T20:21:06.576 DEBUG:boto:path=/pklxkd-49/obj1?versionId=null 2023-05-31T20:21:06.576 DEBUG:boto:auth_path=/pklxkd-49/obj1?versionId=null 2023-05-31T20:21:06.576 DEBUG:boto:Method: GET 2023-05-31T20:21:06.577 DEBUG:boto:Path: /pklxkd-49/obj1?versionId=null 2023-05-31T20:21:06.577 DEBUG:boto:Data: 2023-05-31T20:21:06.577 DEBUG:boto:Headers: {} 2023-05-31T20:21:06.578 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:21:06.578 DEBUG:boto:Port: 8000 2023-05-31T20:21:06.578 DEBUG:boto:Params: {} 2023-05-31T20:21:06.579 DEBUG:boto:Token: None 2023-05-31T20:21:06.579 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:21:06 GMT /pklxkd-49/obj1?versionId=null 2023-05-31T20:21:06.580 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:8tsqIcAzVZmkItuo1nqXpWY9O4w= 2023-05-31T20:21:06.580 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:21:06 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:8tsqIcAzVZmkItuo1nqXpWY9O4w=', 'Content-Length': '0'} 2023-05-31T20:21:06.582 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:18:05 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000008904819fe9a4b056-006477ac32-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:21:06 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:21:06.583 DEBUG:boto:path=/pklxkd-49/ 2023-05-31T20:21:06.583 DEBUG:boto:auth_path=/pklxkd-49/ 2023-05-31T20:21:06.583 DEBUG:boto:path=/pklxkd-49/?prefix=obj1 2023-05-31T20:21:06.584 DEBUG:boto:auth_path=/pklxkd-49/?prefix=obj1 2023-05-31T20:21:06.584 DEBUG:boto:Method: GET 2023-05-31T20:21:06.584 DEBUG:boto:Path: /pklxkd-49/?prefix=obj1 2023-05-31T20:21:06.585 DEBUG:boto:Data: 2023-05-31T20:21:06.585 DEBUG:boto:Headers: {} 2023-05-31T20:21:06.586 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:21:06.586 DEBUG:boto:Port: 8001 2023-05-31T20:21:06.586 DEBUG:boto:Params: {} 2023-05-31T20:21:06.587 DEBUG:boto:Token: None 2023-05-31T20:21:06.587 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:21:06 GMT /pklxkd-49/ 2023-05-31T20:21:06.588 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:r3zv403RESwZV+ugf5ZP3UqJm6A= 2023-05-31T20:21:06.588 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:21:06 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:r3zv403RESwZV+ugf5ZP3UqJm6A=', 'Content-Length': '0'} 2023-05-31T20:21:06.590 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000823dfea8df3e2757-006477ac32-162e-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:21:06 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:21:06.591 DEBUG:boto:b'pklxkd-49obj11000falseobj12023-05-31T20:18:05.410Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:21:06.591 DEBUG:boto:path=/pklxkd-49/ 2023-05-31T20:21:06.592 DEBUG:boto:auth_path=/pklxkd-49/ 2023-05-31T20:21:06.592 DEBUG:boto:path=/pklxkd-49/?prefix=obj1 2023-05-31T20:21:06.593 DEBUG:boto:auth_path=/pklxkd-49/?prefix=obj1 2023-05-31T20:21:06.593 DEBUG:boto:Method: GET 2023-05-31T20:21:06.593 DEBUG:boto:Path: /pklxkd-49/?prefix=obj1 2023-05-31T20:21:06.594 DEBUG:boto:Data: 2023-05-31T20:21:06.594 DEBUG:boto:Headers: {} 2023-05-31T20:21:06.594 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:21:06.595 DEBUG:boto:Port: 8000 2023-05-31T20:21:06.595 DEBUG:boto:Params: {} 2023-05-31T20:21:06.595 DEBUG:boto:Token: None 2023-05-31T20:21:06.596 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:21:06 GMT /pklxkd-49/ 2023-05-31T20:21:06.596 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:r3zv403RESwZV+ugf5ZP3UqJm6A= 2023-05-31T20:21:06.597 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:21:06 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:r3zv403RESwZV+ugf5ZP3UqJm6A=', 'Content-Length': '0'} 2023-05-31T20:21:06.599 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000798172b3dd1f08e8-006477ac32-1630-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:21:06 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:21:06.600 DEBUG:boto:b'pklxkd-49obj11000falseobj12023-05-31T20:18:05.410Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:21:06.601 DEBUG:boto:path=/pklxkd-49/obj1 2023-05-31T20:21:06.601 DEBUG:boto:auth_path=/pklxkd-49/obj1 2023-05-31T20:21:06.601 DEBUG:boto:Method: HEAD 2023-05-31T20:21:06.602 DEBUG:boto:Path: /pklxkd-49/obj1 2023-05-31T20:21:06.602 DEBUG:boto:Data: 2023-05-31T20:21:06.602 DEBUG:boto:Headers: {} 2023-05-31T20:21:06.603 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:21:06.603 DEBUG:boto:Port: 8001 2023-05-31T20:21:06.603 DEBUG:boto:Params: {} 2023-05-31T20:21:06.604 DEBUG:boto:Token: None 2023-05-31T20:21:06.604 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:21:06 GMT /pklxkd-49/obj1 2023-05-31T20:21:06.604 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:PTrE9bnhx2qtnILev4Rz1BjVh7M= 2023-05-31T20:21:06.605 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:21:06 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:PTrE9bnhx2qtnILev4Rz1BjVh7M=', 'Content-Length': '0'} 2023-05-31T20:21:06.606 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:18:05 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000297ce5dfe7bcc09a-006477ac32-162e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:21:06 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:21:06.607 DEBUG:boto:path=/pklxkd-49/obj1 2023-05-31T20:21:06.607 DEBUG:boto:auth_path=/pklxkd-49/obj1 2023-05-31T20:21:06.608 DEBUG:boto:Method: HEAD 2023-05-31T20:21:06.608 DEBUG:boto:Path: /pklxkd-49/obj1 2023-05-31T20:21:06.608 DEBUG:boto:Data: 2023-05-31T20:21:06.609 DEBUG:boto:Headers: {} 2023-05-31T20:21:06.609 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:21:06.609 DEBUG:boto:Port: 8000 2023-05-31T20:21:06.610 DEBUG:boto:Params: {} 2023-05-31T20:21:06.610 DEBUG:boto:Token: None 2023-05-31T20:21:06.610 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:21:06 GMT /pklxkd-49/obj1 2023-05-31T20:21:06.611 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:PTrE9bnhx2qtnILev4Rz1BjVh7M= 2023-05-31T20:21:06.611 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:21:06 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:PTrE9bnhx2qtnILev4Rz1BjVh7M=', 'Content-Length': '0'} 2023-05-31T20:21:06.613 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:18:05 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000c0b370e486a6e5b3-006477ac32-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:21:06 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:21:06.613 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2023-05-31T20:21:06.614 DEBUG:boto:path=/pklxkd-49/obj1 2023-05-31T20:21:06.614 DEBUG:boto:auth_path=/pklxkd-49/obj1 2023-05-31T20:21:06.615 DEBUG:boto:Method: GET 2023-05-31T20:21:06.615 DEBUG:boto:Path: /pklxkd-49/obj1 2023-05-31T20:21:06.615 DEBUG:boto:Data: 2023-05-31T20:21:06.616 DEBUG:boto:Headers: {} 2023-05-31T20:21:06.616 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:21:06.617 DEBUG:boto:Port: 8001 2023-05-31T20:21:06.617 DEBUG:boto:Params: {} 2023-05-31T20:21:06.618 DEBUG:boto:Token: None 2023-05-31T20:21:06.618 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:21:06 GMT /pklxkd-49/obj1 2023-05-31T20:21:06.619 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:pnUR/sOwYc1elHqwIFvDI6MnWus= 2023-05-31T20:21:06.619 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:21:06 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:pnUR/sOwYc1elHqwIFvDI6MnWus=', 'Content-Length': '0'} 2023-05-31T20:21:06.621 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:18:05 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000054dc79d3da1410ec-006477ac32-162e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:21:06 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:21:06.622 DEBUG:boto:path=/pklxkd-49/obj1 2023-05-31T20:21:06.622 DEBUG:boto:auth_path=/pklxkd-49/obj1 2023-05-31T20:21:06.622 DEBUG:boto:Method: GET 2023-05-31T20:21:06.623 DEBUG:boto:Path: /pklxkd-49/obj1 2023-05-31T20:21:06.623 DEBUG:boto:Data: 2023-05-31T20:21:06.623 DEBUG:boto:Headers: {} 2023-05-31T20:21:06.624 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:21:06.624 DEBUG:boto:Port: 8000 2023-05-31T20:21:06.625 DEBUG:boto:Params: {} 2023-05-31T20:21:06.625 DEBUG:boto:Token: None 2023-05-31T20:21:06.625 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:21:06 GMT /pklxkd-49/obj1 2023-05-31T20:21:06.626 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:pnUR/sOwYc1elHqwIFvDI6MnWus= 2023-05-31T20:21:06.626 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:21:06 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:pnUR/sOwYc1elHqwIFvDI6MnWus=', 'Content-Length': '0'} 2023-05-31T20:21:06.628 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:18:05 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000693503fd8a5f93ef-006477ac32-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:21:06 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:21:06.629 DEBUG:boto:path=/pklxkd-49/ 2023-05-31T20:21:06.629 DEBUG:boto:auth_path=/pklxkd-49/ 2023-05-31T20:21:06.630 DEBUG:boto:path=/pklxkd-49/?prefix=obj1 2023-05-31T20:21:06.630 DEBUG:boto:auth_path=/pklxkd-49/?prefix=obj1 2023-05-31T20:21:06.630 DEBUG:boto:Method: GET 2023-05-31T20:21:06.631 DEBUG:boto:Path: /pklxkd-49/?prefix=obj1 2023-05-31T20:21:06.631 DEBUG:boto:Data: 2023-05-31T20:21:06.631 DEBUG:boto:Headers: {} 2023-05-31T20:21:06.632 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:21:06.632 DEBUG:boto:Port: 8001 2023-05-31T20:21:06.632 DEBUG:boto:Params: {} 2023-05-31T20:21:06.633 DEBUG:boto:Token: None 2023-05-31T20:21:06.633 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:21:06 GMT /pklxkd-49/ 2023-05-31T20:21:06.634 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:r3zv403RESwZV+ugf5ZP3UqJm6A= 2023-05-31T20:21:06.634 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:21:06 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:r3zv403RESwZV+ugf5ZP3UqJm6A=', 'Content-Length': '0'} 2023-05-31T20:21:06.636 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ec7852ab6d196cc4-006477ac32-162e-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:21:06 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:21:06.637 DEBUG:boto:b'pklxkd-49obj11000falseobj12023-05-31T20:18:05.410Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:21:06.637 DEBUG:boto:path=/pklxkd-49/ 2023-05-31T20:21:06.638 DEBUG:boto:auth_path=/pklxkd-49/ 2023-05-31T20:21:06.638 DEBUG:boto:path=/pklxkd-49/?prefix=obj1 2023-05-31T20:21:06.638 DEBUG:boto:auth_path=/pklxkd-49/?prefix=obj1 2023-05-31T20:21:06.639 DEBUG:boto:Method: GET 2023-05-31T20:21:06.639 DEBUG:boto:Path: /pklxkd-49/?prefix=obj1 2023-05-31T20:21:06.640 DEBUG:boto:Data: 2023-05-31T20:21:06.640 DEBUG:boto:Headers: {} 2023-05-31T20:21:06.640 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:21:06.641 DEBUG:boto:Port: 8000 2023-05-31T20:21:06.641 DEBUG:boto:Params: {} 2023-05-31T20:21:06.641 DEBUG:boto:Token: None 2023-05-31T20:21:06.642 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:21:06 GMT /pklxkd-49/ 2023-05-31T20:21:06.642 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:r3zv403RESwZV+ugf5ZP3UqJm6A= 2023-05-31T20:21:06.643 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:21:06 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:r3zv403RESwZV+ugf5ZP3UqJm6A=', 'Content-Length': '0'} 2023-05-31T20:21:06.645 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000009a401f376684d64e-006477ac32-1630-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:21:06 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:21:06.645 DEBUG:boto:b'pklxkd-49obj11000falseobj12023-05-31T20:18:05.410Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:21:06.646 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2023-05-31T20:21:06.646 DEBUG:boto:path=/pklxkd-49/obj2 2023-05-31T20:21:06.647 DEBUG:boto:auth_path=/pklxkd-49/obj2 2023-05-31T20:21:06.647 DEBUG:boto:path=/pklxkd-49/obj2?versionId=null 2023-05-31T20:21:06.647 DEBUG:boto:auth_path=/pklxkd-49/obj2?versionId=null 2023-05-31T20:21:06.648 DEBUG:boto:Method: GET 2023-05-31T20:21:06.648 DEBUG:boto:Path: /pklxkd-49/obj2?versionId=null 2023-05-31T20:21:06.649 DEBUG:boto:Data: 2023-05-31T20:21:06.649 DEBUG:boto:Headers: {} 2023-05-31T20:21:06.649 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:21:06.650 DEBUG:boto:Port: 8001 2023-05-31T20:21:06.650 DEBUG:boto:Params: {} 2023-05-31T20:21:06.650 DEBUG:boto:Token: None 2023-05-31T20:21:06.651 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:21:06 GMT /pklxkd-49/obj2?versionId=null 2023-05-31T20:21:06.651 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:9eszjwvA5Hn17wZ370WhpFkpAHY= 2023-05-31T20:21:06.652 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:21:06 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:9eszjwvA5Hn17wZ370WhpFkpAHY=', 'Content-Length': '0'} 2023-05-31T20:21:06.653 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:18:05 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000009b31223500b76d0c-006477ac32-162e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:21:06 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:21:06.696 DEBUG:boto:path=/pklxkd-49/obj2 2023-05-31T20:21:06.697 DEBUG:boto:auth_path=/pklxkd-49/obj2 2023-05-31T20:21:06.697 DEBUG:boto:path=/pklxkd-49/obj2?versionId=null 2023-05-31T20:21:06.698 DEBUG:boto:auth_path=/pklxkd-49/obj2?versionId=null 2023-05-31T20:21:06.698 DEBUG:boto:Method: GET 2023-05-31T20:21:06.699 DEBUG:boto:Path: /pklxkd-49/obj2?versionId=null 2023-05-31T20:21:06.699 DEBUG:boto:Data: 2023-05-31T20:21:06.699 DEBUG:boto:Headers: {} 2023-05-31T20:21:06.700 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:21:06.700 DEBUG:boto:Port: 8000 2023-05-31T20:21:06.700 DEBUG:boto:Params: {} 2023-05-31T20:21:06.701 DEBUG:boto:Token: None 2023-05-31T20:21:06.701 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:21:06 GMT /pklxkd-49/obj2?versionId=null 2023-05-31T20:21:06.702 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:9eszjwvA5Hn17wZ370WhpFkpAHY= 2023-05-31T20:21:06.702 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:21:06 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:9eszjwvA5Hn17wZ370WhpFkpAHY=', 'Content-Length': '0'} 2023-05-31T20:21:06.704 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:18:05 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000aae23c37aa388b03-006477ac32-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:21:06 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:21:06.705 DEBUG:boto:path=/pklxkd-49/obj2 2023-05-31T20:21:06.705 DEBUG:boto:auth_path=/pklxkd-49/obj2 2023-05-31T20:21:06.705 DEBUG:boto:path=/pklxkd-49/obj2?versionId=null 2023-05-31T20:21:06.706 DEBUG:boto:auth_path=/pklxkd-49/obj2?versionId=null 2023-05-31T20:21:06.706 DEBUG:boto:Method: HEAD 2023-05-31T20:21:06.706 DEBUG:boto:Path: /pklxkd-49/obj2?versionId=null 2023-05-31T20:21:06.707 DEBUG:boto:Data: 2023-05-31T20:21:06.707 DEBUG:boto:Headers: {} 2023-05-31T20:21:06.707 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:21:06.708 DEBUG:boto:Port: 8001 2023-05-31T20:21:06.708 DEBUG:boto:Params: {} 2023-05-31T20:21:06.709 DEBUG:boto:Token: None 2023-05-31T20:21:06.709 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:21:06 GMT /pklxkd-49/obj2?versionId=null 2023-05-31T20:21:06.709 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:B1F0eGbTBCvQd8aFyCluXfvvt48= 2023-05-31T20:21:06.710 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:21:06 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:B1F0eGbTBCvQd8aFyCluXfvvt48=', 'Content-Length': '0'} 2023-05-31T20:21:06.711 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:18:05 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000349c1a7b16ad51d0-006477ac32-162e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:21:06 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:21:06.712 DEBUG:boto:path=/pklxkd-49/obj2 2023-05-31T20:21:06.712 DEBUG:boto:auth_path=/pklxkd-49/obj2 2023-05-31T20:21:06.713 DEBUG:boto:path=/pklxkd-49/obj2?versionId=null 2023-05-31T20:21:06.713 DEBUG:boto:auth_path=/pklxkd-49/obj2?versionId=null 2023-05-31T20:21:06.713 DEBUG:boto:Method: HEAD 2023-05-31T20:21:06.714 DEBUG:boto:Path: /pklxkd-49/obj2?versionId=null 2023-05-31T20:21:06.714 DEBUG:boto:Data: 2023-05-31T20:21:06.714 DEBUG:boto:Headers: {} 2023-05-31T20:21:06.715 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:21:06.715 DEBUG:boto:Port: 8000 2023-05-31T20:21:06.715 DEBUG:boto:Params: {} 2023-05-31T20:21:06.716 DEBUG:boto:Token: None 2023-05-31T20:21:06.716 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:21:06 GMT /pklxkd-49/obj2?versionId=null 2023-05-31T20:21:06.717 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:B1F0eGbTBCvQd8aFyCluXfvvt48= 2023-05-31T20:21:06.717 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:21:06 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:B1F0eGbTBCvQd8aFyCluXfvvt48=', 'Content-Length': '0'} 2023-05-31T20:21:06.718 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:18:05 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000002fef8d0d706d62de-006477ac32-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:21:06 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:21:06.719 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2023-05-31T20:21:06.719 DEBUG:boto:path=/pklxkd-49/obj2 2023-05-31T20:21:06.720 DEBUG:boto:auth_path=/pklxkd-49/obj2 2023-05-31T20:21:06.720 DEBUG:boto:path=/pklxkd-49/obj2?versionId=null 2023-05-31T20:21:06.721 DEBUG:boto:auth_path=/pklxkd-49/obj2?versionId=null 2023-05-31T20:21:06.721 DEBUG:boto:Method: GET 2023-05-31T20:21:06.721 DEBUG:boto:Path: /pklxkd-49/obj2?versionId=null 2023-05-31T20:21:06.722 DEBUG:boto:Data: 2023-05-31T20:21:06.722 DEBUG:boto:Headers: {} 2023-05-31T20:21:06.722 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:21:06.723 DEBUG:boto:Port: 8001 2023-05-31T20:21:06.723 DEBUG:boto:Params: {} 2023-05-31T20:21:06.723 DEBUG:boto:Token: None 2023-05-31T20:21:06.724 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:21:06 GMT /pklxkd-49/obj2?versionId=null 2023-05-31T20:21:06.724 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:9eszjwvA5Hn17wZ370WhpFkpAHY= 2023-05-31T20:21:06.725 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:21:06 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:9eszjwvA5Hn17wZ370WhpFkpAHY=', 'Content-Length': '0'} 2023-05-31T20:21:06.726 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:18:05 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000062be1fbc55340dde-006477ac32-162e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:21:06 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:21:06.727 DEBUG:boto:path=/pklxkd-49/obj2 2023-05-31T20:21:06.727 DEBUG:boto:auth_path=/pklxkd-49/obj2 2023-05-31T20:21:06.728 DEBUG:boto:path=/pklxkd-49/obj2?versionId=null 2023-05-31T20:21:06.728 DEBUG:boto:auth_path=/pklxkd-49/obj2?versionId=null 2023-05-31T20:21:06.729 DEBUG:boto:Method: GET 2023-05-31T20:21:06.729 DEBUG:boto:Path: /pklxkd-49/obj2?versionId=null 2023-05-31T20:21:06.729 DEBUG:boto:Data: 2023-05-31T20:21:06.730 DEBUG:boto:Headers: {} 2023-05-31T20:21:06.730 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:21:06.730 DEBUG:boto:Port: 8000 2023-05-31T20:21:06.731 DEBUG:boto:Params: {} 2023-05-31T20:21:06.731 DEBUG:boto:Token: None 2023-05-31T20:21:06.731 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:21:06 GMT /pklxkd-49/obj2?versionId=null 2023-05-31T20:21:06.732 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:9eszjwvA5Hn17wZ370WhpFkpAHY= 2023-05-31T20:21:06.732 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:21:06 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:9eszjwvA5Hn17wZ370WhpFkpAHY=', 'Content-Length': '0'} 2023-05-31T20:21:06.734 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:18:05 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000043501f38f5247d1e-006477ac32-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:21:06 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:21:06.734 DEBUG:boto:path=/pklxkd-49/ 2023-05-31T20:21:06.735 DEBUG:boto:auth_path=/pklxkd-49/ 2023-05-31T20:21:06.735 DEBUG:boto:path=/pklxkd-49/?prefix=obj2 2023-05-31T20:21:06.736 DEBUG:boto:auth_path=/pklxkd-49/?prefix=obj2 2023-05-31T20:21:06.736 DEBUG:boto:Method: GET 2023-05-31T20:21:06.736 DEBUG:boto:Path: /pklxkd-49/?prefix=obj2 2023-05-31T20:21:06.737 DEBUG:boto:Data: 2023-05-31T20:21:06.737 DEBUG:boto:Headers: {} 2023-05-31T20:21:06.737 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:21:06.738 DEBUG:boto:Port: 8001 2023-05-31T20:21:06.738 DEBUG:boto:Params: {} 2023-05-31T20:21:06.739 DEBUG:boto:Token: None 2023-05-31T20:21:06.739 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:21:06 GMT /pklxkd-49/ 2023-05-31T20:21:06.739 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:r3zv403RESwZV+ugf5ZP3UqJm6A= 2023-05-31T20:21:06.740 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:21:06 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:r3zv403RESwZV+ugf5ZP3UqJm6A=', 'Content-Length': '0'} 2023-05-31T20:21:06.742 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000000b3222a16436f451-006477ac32-162e-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:21:06 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:21:06.742 DEBUG:boto:b'pklxkd-49obj21000falseobj22023-05-31T20:18:05.430Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:21:06.743 DEBUG:boto:path=/pklxkd-49/ 2023-05-31T20:21:06.743 DEBUG:boto:auth_path=/pklxkd-49/ 2023-05-31T20:21:06.744 DEBUG:boto:path=/pklxkd-49/?prefix=obj2 2023-05-31T20:21:06.744 DEBUG:boto:auth_path=/pklxkd-49/?prefix=obj2 2023-05-31T20:21:06.744 DEBUG:boto:Method: GET 2023-05-31T20:21:06.745 DEBUG:boto:Path: /pklxkd-49/?prefix=obj2 2023-05-31T20:21:06.745 DEBUG:boto:Data: 2023-05-31T20:21:06.745 DEBUG:boto:Headers: {} 2023-05-31T20:21:06.746 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:21:06.746 DEBUG:boto:Port: 8000 2023-05-31T20:21:06.747 DEBUG:boto:Params: {} 2023-05-31T20:21:06.747 DEBUG:boto:Token: None 2023-05-31T20:21:06.747 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:21:06 GMT /pklxkd-49/ 2023-05-31T20:21:06.748 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:r3zv403RESwZV+ugf5ZP3UqJm6A= 2023-05-31T20:21:06.748 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:21:06 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:r3zv403RESwZV+ugf5ZP3UqJm6A=', 'Content-Length': '0'} 2023-05-31T20:21:06.750 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000edd83f7fe904a11a-006477ac32-1630-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:21:06 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:21:06.751 DEBUG:boto:b'pklxkd-49obj21000falseobj22023-05-31T20:18:05.430Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:21:06.752 DEBUG:boto:path=/pklxkd-49/obj2 2023-05-31T20:21:06.752 DEBUG:boto:auth_path=/pklxkd-49/obj2 2023-05-31T20:21:06.752 DEBUG:boto:Method: HEAD 2023-05-31T20:21:06.753 DEBUG:boto:Path: /pklxkd-49/obj2 2023-05-31T20:21:06.753 DEBUG:boto:Data: 2023-05-31T20:21:06.753 DEBUG:boto:Headers: {} 2023-05-31T20:21:06.754 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:21:06.754 DEBUG:boto:Port: 8001 2023-05-31T20:21:06.754 DEBUG:boto:Params: {} 2023-05-31T20:21:06.755 DEBUG:boto:Token: None 2023-05-31T20:21:06.755 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:21:06 GMT /pklxkd-49/obj2 2023-05-31T20:21:06.756 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:znV1xGhs2kjZH7Bs4iNgfgi1mc0= 2023-05-31T20:21:06.756 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:21:06 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:znV1xGhs2kjZH7Bs4iNgfgi1mc0=', 'Content-Length': '0'} 2023-05-31T20:21:06.757 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:18:05 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000153de9485245dfe5-006477ac32-162e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:21:06 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:21:06.758 DEBUG:boto:path=/pklxkd-49/obj2 2023-05-31T20:21:06.758 DEBUG:boto:auth_path=/pklxkd-49/obj2 2023-05-31T20:21:06.759 DEBUG:boto:Method: HEAD 2023-05-31T20:21:06.759 DEBUG:boto:Path: /pklxkd-49/obj2 2023-05-31T20:21:06.759 DEBUG:boto:Data: 2023-05-31T20:21:06.760 DEBUG:boto:Headers: {} 2023-05-31T20:21:06.760 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:21:06.760 DEBUG:boto:Port: 8000 2023-05-31T20:21:06.761 DEBUG:boto:Params: {} 2023-05-31T20:21:06.761 DEBUG:boto:Token: None 2023-05-31T20:21:06.761 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:21:06 GMT /pklxkd-49/obj2 2023-05-31T20:21:06.762 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:znV1xGhs2kjZH7Bs4iNgfgi1mc0= 2023-05-31T20:21:06.762 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:21:06 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:znV1xGhs2kjZH7Bs4iNgfgi1mc0=', 'Content-Length': '0'} 2023-05-31T20:21:06.764 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:18:05 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000843b6bda21600e31-006477ac32-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:21:06 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:21:06.764 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2023-05-31T20:21:06.765 DEBUG:boto:path=/pklxkd-49/obj2 2023-05-31T20:21:06.765 DEBUG:boto:auth_path=/pklxkd-49/obj2 2023-05-31T20:21:06.765 DEBUG:boto:Method: GET 2023-05-31T20:21:06.766 DEBUG:boto:Path: /pklxkd-49/obj2 2023-05-31T20:21:06.766 DEBUG:boto:Data: 2023-05-31T20:21:06.766 DEBUG:boto:Headers: {} 2023-05-31T20:21:06.767 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:21:06.767 DEBUG:boto:Port: 8001 2023-05-31T20:21:06.768 DEBUG:boto:Params: {} 2023-05-31T20:21:06.768 DEBUG:boto:Token: None 2023-05-31T20:21:06.768 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:21:06 GMT /pklxkd-49/obj2 2023-05-31T20:21:06.769 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:rR4aBPClb2ACrvUuWCsjt6ybor8= 2023-05-31T20:21:06.769 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:21:06 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:rR4aBPClb2ACrvUuWCsjt6ybor8=', 'Content-Length': '0'} 2023-05-31T20:21:06.771 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:18:05 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000228d63cad6f01ec5-006477ac32-162e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:21:06 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:21:06.771 DEBUG:boto:path=/pklxkd-49/obj2 2023-05-31T20:21:06.772 DEBUG:boto:auth_path=/pklxkd-49/obj2 2023-05-31T20:21:06.772 DEBUG:boto:Method: GET 2023-05-31T20:21:06.772 DEBUG:boto:Path: /pklxkd-49/obj2 2023-05-31T20:21:06.773 DEBUG:boto:Data: 2023-05-31T20:21:06.773 DEBUG:boto:Headers: {} 2023-05-31T20:21:06.774 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:21:06.774 DEBUG:boto:Port: 8000 2023-05-31T20:21:06.774 DEBUG:boto:Params: {} 2023-05-31T20:21:06.775 DEBUG:boto:Token: None 2023-05-31T20:21:06.775 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:21:06 GMT /pklxkd-49/obj2 2023-05-31T20:21:06.775 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:rR4aBPClb2ACrvUuWCsjt6ybor8= 2023-05-31T20:21:06.776 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:21:06 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:rR4aBPClb2ACrvUuWCsjt6ybor8=', 'Content-Length': '0'} 2023-05-31T20:21:06.777 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:18:05 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000613aaa039ff69b7c-006477ac32-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:21:06 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:21:06.778 DEBUG:boto:path=/pklxkd-49/ 2023-05-31T20:21:06.778 DEBUG:boto:auth_path=/pklxkd-49/ 2023-05-31T20:21:06.779 DEBUG:boto:path=/pklxkd-49/?prefix=obj2 2023-05-31T20:21:06.779 DEBUG:boto:auth_path=/pklxkd-49/?prefix=obj2 2023-05-31T20:21:06.779 DEBUG:boto:Method: GET 2023-05-31T20:21:06.780 DEBUG:boto:Path: /pklxkd-49/?prefix=obj2 2023-05-31T20:21:06.780 DEBUG:boto:Data: 2023-05-31T20:21:06.781 DEBUG:boto:Headers: {} 2023-05-31T20:21:06.781 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:21:06.781 DEBUG:boto:Port: 8001 2023-05-31T20:21:06.782 DEBUG:boto:Params: {} 2023-05-31T20:21:06.782 DEBUG:boto:Token: None 2023-05-31T20:21:06.782 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:21:06 GMT /pklxkd-49/ 2023-05-31T20:21:06.783 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:r3zv403RESwZV+ugf5ZP3UqJm6A= 2023-05-31T20:21:06.783 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:21:06 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:r3zv403RESwZV+ugf5ZP3UqJm6A=', 'Content-Length': '0'} 2023-05-31T20:21:06.785 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000566de1bb364791f3-006477ac32-162e-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:21:06 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:21:06.786 DEBUG:boto:b'pklxkd-49obj21000falseobj22023-05-31T20:18:05.430Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:21:06.786 DEBUG:boto:path=/pklxkd-49/ 2023-05-31T20:21:06.787 DEBUG:boto:auth_path=/pklxkd-49/ 2023-05-31T20:21:06.787 DEBUG:boto:path=/pklxkd-49/?prefix=obj2 2023-05-31T20:21:06.787 DEBUG:boto:auth_path=/pklxkd-49/?prefix=obj2 2023-05-31T20:21:06.788 DEBUG:boto:Method: GET 2023-05-31T20:21:06.788 DEBUG:boto:Path: /pklxkd-49/?prefix=obj2 2023-05-31T20:21:06.789 DEBUG:boto:Data: 2023-05-31T20:21:06.789 DEBUG:boto:Headers: {} 2023-05-31T20:21:06.789 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:21:06.790 DEBUG:boto:Port: 8000 2023-05-31T20:21:06.790 DEBUG:boto:Params: {} 2023-05-31T20:21:06.790 DEBUG:boto:Token: None 2023-05-31T20:21:06.791 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:21:06 GMT /pklxkd-49/ 2023-05-31T20:21:06.791 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:r3zv403RESwZV+ugf5ZP3UqJm6A= 2023-05-31T20:21:06.792 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:21:06 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:r3zv403RESwZV+ugf5ZP3UqJm6A=', 'Content-Length': '0'} 2023-05-31T20:21:06.793 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000007604e4269e2b67ed-006477ac32-1630-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:21:06 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:21:06.794 DEBUG:boto:b'pklxkd-49obj21000falseobj22023-05-31T20:18:05.430Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:21:06.795 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2023-05-31T20:21:06.795 DEBUG:boto:path=/pklxkd-49/obj3 2023-05-31T20:21:06.795 DEBUG:boto:auth_path=/pklxkd-49/obj3 2023-05-31T20:21:06.796 DEBUG:boto:path=/pklxkd-49/obj3?versionId=null 2023-05-31T20:21:06.796 DEBUG:boto:auth_path=/pklxkd-49/obj3?versionId=null 2023-05-31T20:21:06.797 DEBUG:boto:Method: GET 2023-05-31T20:21:06.797 DEBUG:boto:Path: /pklxkd-49/obj3?versionId=null 2023-05-31T20:21:06.797 DEBUG:boto:Data: 2023-05-31T20:21:06.798 DEBUG:boto:Headers: {} 2023-05-31T20:21:06.798 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:21:06.798 DEBUG:boto:Port: 8001 2023-05-31T20:21:06.799 DEBUG:boto:Params: {} 2023-05-31T20:21:06.799 DEBUG:boto:Token: None 2023-05-31T20:21:06.799 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:21:06 GMT /pklxkd-49/obj3?versionId=null 2023-05-31T20:21:06.800 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:bejWGDyLKbXH0bv0aI//Kg/GkKU= 2023-05-31T20:21:06.800 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:21:06 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:bejWGDyLKbXH0bv0aI//Kg/GkKU=', 'Content-Length': '0'} 2023-05-31T20:21:06.802 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:18:05 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000000373b06b32b8051e-006477ac32-162e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:21:06 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:21:06.802 DEBUG:boto:path=/pklxkd-49/obj3 2023-05-31T20:21:06.803 DEBUG:boto:auth_path=/pklxkd-49/obj3 2023-05-31T20:21:06.803 DEBUG:boto:path=/pklxkd-49/obj3?versionId=null 2023-05-31T20:21:06.803 DEBUG:boto:auth_path=/pklxkd-49/obj3?versionId=null 2023-05-31T20:21:06.804 DEBUG:boto:Method: GET 2023-05-31T20:21:06.804 DEBUG:boto:Path: /pklxkd-49/obj3?versionId=null 2023-05-31T20:21:06.805 DEBUG:boto:Data: 2023-05-31T20:21:06.805 DEBUG:boto:Headers: {} 2023-05-31T20:21:06.805 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:21:06.806 DEBUG:boto:Port: 8000 2023-05-31T20:21:06.806 DEBUG:boto:Params: {} 2023-05-31T20:21:06.806 DEBUG:boto:Token: None 2023-05-31T20:21:06.807 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:21:06 GMT /pklxkd-49/obj3?versionId=null 2023-05-31T20:21:06.807 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:bejWGDyLKbXH0bv0aI//Kg/GkKU= 2023-05-31T20:21:06.808 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:21:06 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:bejWGDyLKbXH0bv0aI//Kg/GkKU=', 'Content-Length': '0'} 2023-05-31T20:21:06.809 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:18:05 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000008ce61260f5eb4f1b-006477ac32-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:21:06 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:21:06.810 DEBUG:boto:path=/pklxkd-49/obj3 2023-05-31T20:21:06.810 DEBUG:boto:auth_path=/pklxkd-49/obj3 2023-05-31T20:21:06.810 DEBUG:boto:path=/pklxkd-49/obj3?versionId=null 2023-05-31T20:21:06.811 DEBUG:boto:auth_path=/pklxkd-49/obj3?versionId=null 2023-05-31T20:21:06.811 DEBUG:boto:Method: HEAD 2023-05-31T20:21:06.811 DEBUG:boto:Path: /pklxkd-49/obj3?versionId=null 2023-05-31T20:21:06.812 DEBUG:boto:Data: 2023-05-31T20:21:06.812 DEBUG:boto:Headers: {} 2023-05-31T20:21:06.812 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:21:06.813 DEBUG:boto:Port: 8001 2023-05-31T20:21:06.813 DEBUG:boto:Params: {} 2023-05-31T20:21:06.814 DEBUG:boto:Token: None 2023-05-31T20:21:06.814 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:21:06 GMT /pklxkd-49/obj3?versionId=null 2023-05-31T20:21:06.814 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:OcBmXZWetYR9JbUjl3KPQrN0NE4= 2023-05-31T20:21:06.815 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:21:06 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:OcBmXZWetYR9JbUjl3KPQrN0NE4=', 'Content-Length': '0'} 2023-05-31T20:21:06.816 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:18:05 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000c880679faa27f0c1-006477ac32-162e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:21:06 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:21:06.817 DEBUG:boto:path=/pklxkd-49/obj3 2023-05-31T20:21:06.817 DEBUG:boto:auth_path=/pklxkd-49/obj3 2023-05-31T20:21:06.817 DEBUG:boto:path=/pklxkd-49/obj3?versionId=null 2023-05-31T20:21:06.818 DEBUG:boto:auth_path=/pklxkd-49/obj3?versionId=null 2023-05-31T20:21:06.818 DEBUG:boto:Method: HEAD 2023-05-31T20:21:06.818 DEBUG:boto:Path: /pklxkd-49/obj3?versionId=null 2023-05-31T20:21:06.819 DEBUG:boto:Data: 2023-05-31T20:21:06.819 DEBUG:boto:Headers: {} 2023-05-31T20:21:06.819 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:21:06.820 DEBUG:boto:Port: 8000 2023-05-31T20:21:06.820 DEBUG:boto:Params: {} 2023-05-31T20:21:06.821 DEBUG:boto:Token: None 2023-05-31T20:21:06.821 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:21:06 GMT /pklxkd-49/obj3?versionId=null 2023-05-31T20:21:06.821 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:OcBmXZWetYR9JbUjl3KPQrN0NE4= 2023-05-31T20:21:06.822 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:21:06 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:OcBmXZWetYR9JbUjl3KPQrN0NE4=', 'Content-Length': '0'} 2023-05-31T20:21:06.823 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:18:05 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000006cb64be3b5891203-006477ac32-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:21:06 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:21:06.824 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2023-05-31T20:21:06.824 DEBUG:boto:path=/pklxkd-49/obj3 2023-05-31T20:21:06.825 DEBUG:boto:auth_path=/pklxkd-49/obj3 2023-05-31T20:21:06.825 DEBUG:boto:path=/pklxkd-49/obj3?versionId=null 2023-05-31T20:21:06.825 DEBUG:boto:auth_path=/pklxkd-49/obj3?versionId=null 2023-05-31T20:21:06.826 DEBUG:boto:Method: GET 2023-05-31T20:21:06.826 DEBUG:boto:Path: /pklxkd-49/obj3?versionId=null 2023-05-31T20:21:06.826 DEBUG:boto:Data: 2023-05-31T20:21:06.827 DEBUG:boto:Headers: {} 2023-05-31T20:21:06.827 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:21:06.827 DEBUG:boto:Port: 8001 2023-05-31T20:21:06.828 DEBUG:boto:Params: {} 2023-05-31T20:21:06.828 DEBUG:boto:Token: None 2023-05-31T20:21:06.829 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:21:06 GMT /pklxkd-49/obj3?versionId=null 2023-05-31T20:21:06.829 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:bejWGDyLKbXH0bv0aI//Kg/GkKU= 2023-05-31T20:21:06.830 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:21:06 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:bejWGDyLKbXH0bv0aI//Kg/GkKU=', 'Content-Length': '0'} 2023-05-31T20:21:06.831 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:18:05 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000004d35e053635ee86e-006477ac32-162e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:21:06 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:21:06.872 DEBUG:boto:path=/pklxkd-49/obj3 2023-05-31T20:21:06.873 DEBUG:boto:auth_path=/pklxkd-49/obj3 2023-05-31T20:21:06.873 DEBUG:boto:path=/pklxkd-49/obj3?versionId=null 2023-05-31T20:21:06.874 DEBUG:boto:auth_path=/pklxkd-49/obj3?versionId=null 2023-05-31T20:21:06.874 DEBUG:boto:Method: GET 2023-05-31T20:21:06.875 DEBUG:boto:Path: /pklxkd-49/obj3?versionId=null 2023-05-31T20:21:06.875 DEBUG:boto:Data: 2023-05-31T20:21:06.875 DEBUG:boto:Headers: {} 2023-05-31T20:21:06.876 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:21:06.876 DEBUG:boto:Port: 8000 2023-05-31T20:21:06.876 DEBUG:boto:Params: {} 2023-05-31T20:21:06.877 DEBUG:boto:Token: None 2023-05-31T20:21:06.877 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:21:06 GMT /pklxkd-49/obj3?versionId=null 2023-05-31T20:21:06.878 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:bejWGDyLKbXH0bv0aI//Kg/GkKU= 2023-05-31T20:21:06.878 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:21:06 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:bejWGDyLKbXH0bv0aI//Kg/GkKU=', 'Content-Length': '0'} 2023-05-31T20:21:06.880 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:18:05 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000619b191914aeb896-006477ac32-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:21:06 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:21:06.881 DEBUG:boto:path=/pklxkd-49/ 2023-05-31T20:21:06.882 DEBUG:boto:auth_path=/pklxkd-49/ 2023-05-31T20:21:06.882 DEBUG:boto:path=/pklxkd-49/?prefix=obj3 2023-05-31T20:21:06.882 DEBUG:boto:auth_path=/pklxkd-49/?prefix=obj3 2023-05-31T20:21:06.883 DEBUG:boto:Method: GET 2023-05-31T20:21:06.883 DEBUG:boto:Path: /pklxkd-49/?prefix=obj3 2023-05-31T20:21:06.883 DEBUG:boto:Data: 2023-05-31T20:21:06.884 DEBUG:boto:Headers: {} 2023-05-31T20:21:06.884 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:21:06.884 DEBUG:boto:Port: 8001 2023-05-31T20:21:06.885 DEBUG:boto:Params: {} 2023-05-31T20:21:06.885 DEBUG:boto:Token: None 2023-05-31T20:21:06.886 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:21:06 GMT /pklxkd-49/ 2023-05-31T20:21:06.886 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:r3zv403RESwZV+ugf5ZP3UqJm6A= 2023-05-31T20:21:06.887 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:21:06 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:r3zv403RESwZV+ugf5ZP3UqJm6A=', 'Content-Length': '0'} 2023-05-31T20:21:06.889 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000005a71247d8a622824-006477ac32-162e-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:21:06 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:21:06.889 DEBUG:boto:b'pklxkd-49obj31000falseobj32023-05-31T20:18:05.493Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:21:06.890 DEBUG:boto:path=/pklxkd-49/ 2023-05-31T20:21:06.891 DEBUG:boto:auth_path=/pklxkd-49/ 2023-05-31T20:21:06.891 DEBUG:boto:path=/pklxkd-49/?prefix=obj3 2023-05-31T20:21:06.891 DEBUG:boto:auth_path=/pklxkd-49/?prefix=obj3 2023-05-31T20:21:06.892 DEBUG:boto:Method: GET 2023-05-31T20:21:06.892 DEBUG:boto:Path: /pklxkd-49/?prefix=obj3 2023-05-31T20:21:06.892 DEBUG:boto:Data: 2023-05-31T20:21:06.893 DEBUG:boto:Headers: {} 2023-05-31T20:21:06.893 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:21:06.894 DEBUG:boto:Port: 8000 2023-05-31T20:21:06.894 DEBUG:boto:Params: {} 2023-05-31T20:21:06.895 DEBUG:boto:Token: None 2023-05-31T20:21:06.895 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:21:06 GMT /pklxkd-49/ 2023-05-31T20:21:06.895 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:r3zv403RESwZV+ugf5ZP3UqJm6A= 2023-05-31T20:21:06.896 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:21:06 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:r3zv403RESwZV+ugf5ZP3UqJm6A=', 'Content-Length': '0'} 2023-05-31T20:21:06.898 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000083aa77ff4ab07f4e-006477ac32-1630-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:21:06 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:21:06.940 DEBUG:boto:b'pklxkd-49obj31000falseobj32023-05-31T20:18:05.493Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:21:06.941 DEBUG:boto:path=/pklxkd-49/obj3 2023-05-31T20:21:06.942 DEBUG:boto:auth_path=/pklxkd-49/obj3 2023-05-31T20:21:06.942 DEBUG:boto:Method: HEAD 2023-05-31T20:21:06.942 DEBUG:boto:Path: /pklxkd-49/obj3 2023-05-31T20:21:06.943 DEBUG:boto:Data: 2023-05-31T20:21:06.943 DEBUG:boto:Headers: {} 2023-05-31T20:21:06.943 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:21:06.944 DEBUG:boto:Port: 8001 2023-05-31T20:21:06.944 DEBUG:boto:Params: {} 2023-05-31T20:21:06.945 DEBUG:boto:Token: None 2023-05-31T20:21:06.945 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:21:06 GMT /pklxkd-49/obj3 2023-05-31T20:21:06.945 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:YaubscYiw9Zkzr6BSuLqBuFgDvg= 2023-05-31T20:21:06.946 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:21:06 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:YaubscYiw9Zkzr6BSuLqBuFgDvg=', 'Content-Length': '0'} 2023-05-31T20:21:06.947 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:18:05 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000878930af6193f626-006477ac32-162e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:21:06 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:21:06.948 DEBUG:boto:path=/pklxkd-49/obj3 2023-05-31T20:21:06.948 DEBUG:boto:auth_path=/pklxkd-49/obj3 2023-05-31T20:21:06.949 DEBUG:boto:Method: HEAD 2023-05-31T20:21:06.949 DEBUG:boto:Path: /pklxkd-49/obj3 2023-05-31T20:21:06.950 DEBUG:boto:Data: 2023-05-31T20:21:06.950 DEBUG:boto:Headers: {} 2023-05-31T20:21:06.950 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:21:06.950 DEBUG:boto:Port: 8000 2023-05-31T20:21:06.951 DEBUG:boto:Params: {} 2023-05-31T20:21:06.951 DEBUG:boto:Token: None 2023-05-31T20:21:06.952 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:21:06 GMT /pklxkd-49/obj3 2023-05-31T20:21:06.952 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:YaubscYiw9Zkzr6BSuLqBuFgDvg= 2023-05-31T20:21:06.952 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:21:06 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:YaubscYiw9Zkzr6BSuLqBuFgDvg=', 'Content-Length': '0'} 2023-05-31T20:21:06.954 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:18:05 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000ed7f247c366fc596-006477ac32-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:21:06 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:21:06.955 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2023-05-31T20:21:06.955 DEBUG:boto:path=/pklxkd-49/obj3 2023-05-31T20:21:06.955 DEBUG:boto:auth_path=/pklxkd-49/obj3 2023-05-31T20:21:06.956 DEBUG:boto:Method: GET 2023-05-31T20:21:06.956 DEBUG:boto:Path: /pklxkd-49/obj3 2023-05-31T20:21:06.957 DEBUG:boto:Data: 2023-05-31T20:21:06.957 DEBUG:boto:Headers: {} 2023-05-31T20:21:06.957 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:21:06.958 DEBUG:boto:Port: 8001 2023-05-31T20:21:06.958 DEBUG:boto:Params: {} 2023-05-31T20:21:06.958 DEBUG:boto:Token: None 2023-05-31T20:21:06.959 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:21:06 GMT /pklxkd-49/obj3 2023-05-31T20:21:06.959 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:Wj5pixg9e2LhwUvJJwAhtSF3q8o= 2023-05-31T20:21:06.960 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:21:06 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:Wj5pixg9e2LhwUvJJwAhtSF3q8o=', 'Content-Length': '0'} 2023-05-31T20:21:06.961 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:18:05 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000b2f72d6337afce06-006477ac32-162e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:21:06 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:21:06.962 DEBUG:boto:path=/pklxkd-49/obj3 2023-05-31T20:21:06.963 DEBUG:boto:auth_path=/pklxkd-49/obj3 2023-05-31T20:21:06.963 DEBUG:boto:Method: GET 2023-05-31T20:21:06.964 DEBUG:boto:Path: /pklxkd-49/obj3 2023-05-31T20:21:06.964 DEBUG:boto:Data: 2023-05-31T20:21:06.964 DEBUG:boto:Headers: {} 2023-05-31T20:21:06.965 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:21:06.965 DEBUG:boto:Port: 8000 2023-05-31T20:21:06.965 DEBUG:boto:Params: {} 2023-05-31T20:21:06.966 DEBUG:boto:Token: None 2023-05-31T20:21:06.966 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:21:06 GMT /pklxkd-49/obj3 2023-05-31T20:21:06.967 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:Wj5pixg9e2LhwUvJJwAhtSF3q8o= 2023-05-31T20:21:06.967 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:21:06 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:Wj5pixg9e2LhwUvJJwAhtSF3q8o=', 'Content-Length': '0'} 2023-05-31T20:21:06.969 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:18:05 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000436780c6f2d25a42-006477ac32-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:21:06 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:21:06.970 DEBUG:boto:path=/pklxkd-49/ 2023-05-31T20:21:06.970 DEBUG:boto:auth_path=/pklxkd-49/ 2023-05-31T20:21:06.970 DEBUG:boto:path=/pklxkd-49/?prefix=obj3 2023-05-31T20:21:06.971 DEBUG:boto:auth_path=/pklxkd-49/?prefix=obj3 2023-05-31T20:21:06.971 DEBUG:boto:Method: GET 2023-05-31T20:21:06.971 DEBUG:boto:Path: /pklxkd-49/?prefix=obj3 2023-05-31T20:21:06.972 DEBUG:boto:Data: 2023-05-31T20:21:06.972 DEBUG:boto:Headers: {} 2023-05-31T20:21:06.973 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:21:06.973 DEBUG:boto:Port: 8001 2023-05-31T20:21:06.973 DEBUG:boto:Params: {} 2023-05-31T20:21:06.974 DEBUG:boto:Token: None 2023-05-31T20:21:06.974 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:21:06 GMT /pklxkd-49/ 2023-05-31T20:21:06.975 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:r3zv403RESwZV+ugf5ZP3UqJm6A= 2023-05-31T20:21:06.975 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:21:06 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:r3zv403RESwZV+ugf5ZP3UqJm6A=', 'Content-Length': '0'} 2023-05-31T20:21:06.977 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000007d7a8d497bbb3cb2-006477ac32-162e-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:21:06 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:21:06.978 DEBUG:boto:b'pklxkd-49obj31000falseobj32023-05-31T20:18:05.493Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:21:06.979 DEBUG:boto:path=/pklxkd-49/ 2023-05-31T20:21:06.979 DEBUG:boto:auth_path=/pklxkd-49/ 2023-05-31T20:21:06.979 DEBUG:boto:path=/pklxkd-49/?prefix=obj3 2023-05-31T20:21:06.980 DEBUG:boto:auth_path=/pklxkd-49/?prefix=obj3 2023-05-31T20:21:06.980 DEBUG:boto:Method: GET 2023-05-31T20:21:06.980 DEBUG:boto:Path: /pklxkd-49/?prefix=obj3 2023-05-31T20:21:06.981 DEBUG:boto:Data: 2023-05-31T20:21:06.981 DEBUG:boto:Headers: {} 2023-05-31T20:21:06.982 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:21:06.982 DEBUG:boto:Port: 8000 2023-05-31T20:21:06.982 DEBUG:boto:Params: {} 2023-05-31T20:21:06.983 DEBUG:boto:Token: None 2023-05-31T20:21:06.983 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:21:06 GMT /pklxkd-49/ 2023-05-31T20:21:06.984 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:r3zv403RESwZV+ugf5ZP3UqJm6A= 2023-05-31T20:21:06.984 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:21:06 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:r3zv403RESwZV+ugf5ZP3UqJm6A=', 'Content-Length': '0'} 2023-05-31T20:21:06.987 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000048df43a25026c42b-006477ac32-1630-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:21:06 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:21:06.987 DEBUG:boto:b'pklxkd-49obj31000falseobj32023-05-31T20:18:05.493Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:21:06.988 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2023-05-31T20:21:06.988 DEBUG:boto:path=/pklxkd-49/obj4 2023-05-31T20:21:06.989 DEBUG:boto:auth_path=/pklxkd-49/obj4 2023-05-31T20:21:06.989 DEBUG:boto:path=/pklxkd-49/obj4?versionId=null 2023-05-31T20:21:06.989 DEBUG:boto:auth_path=/pklxkd-49/obj4?versionId=null 2023-05-31T20:21:06.990 DEBUG:boto:Method: GET 2023-05-31T20:21:06.990 DEBUG:boto:Path: /pklxkd-49/obj4?versionId=null 2023-05-31T20:21:06.990 DEBUG:boto:Data: 2023-05-31T20:21:06.991 DEBUG:boto:Headers: {} 2023-05-31T20:21:06.991 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:21:06.991 DEBUG:boto:Port: 8001 2023-05-31T20:21:06.992 DEBUG:boto:Params: {} 2023-05-31T20:21:06.992 DEBUG:boto:Token: None 2023-05-31T20:21:06.993 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:21:06 GMT /pklxkd-49/obj4?versionId=null 2023-05-31T20:21:06.993 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:ChdhVr6Wy8q+t5CNdEXSF+tx8f4= 2023-05-31T20:21:06.994 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:21:06 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:ChdhVr6Wy8q+t5CNdEXSF+tx8f4=', 'Content-Length': '0'} 2023-05-31T20:21:06.995 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:18:05 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000bec76ce43261386e-006477ac32-162e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:21:06 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:21:06.996 DEBUG:boto:path=/pklxkd-49/obj4 2023-05-31T20:21:06.996 DEBUG:boto:auth_path=/pklxkd-49/obj4 2023-05-31T20:21:06.997 DEBUG:boto:path=/pklxkd-49/obj4?versionId=null 2023-05-31T20:21:06.997 DEBUG:boto:auth_path=/pklxkd-49/obj4?versionId=null 2023-05-31T20:21:06.998 DEBUG:boto:Method: GET 2023-05-31T20:21:06.998 DEBUG:boto:Path: /pklxkd-49/obj4?versionId=null 2023-05-31T20:21:06.998 DEBUG:boto:Data: 2023-05-31T20:21:06.999 DEBUG:boto:Headers: {} 2023-05-31T20:21:06.999 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:21:06.999 DEBUG:boto:Port: 8000 2023-05-31T20:21:07.000 DEBUG:boto:Params: {} 2023-05-31T20:21:07.000 DEBUG:boto:Token: None 2023-05-31T20:21:07.000 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:21:07 GMT /pklxkd-49/obj4?versionId=null 2023-05-31T20:21:07.001 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:XinDQW5zDiHaZ7m29JLXHsJOugM= 2023-05-31T20:21:07.001 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:21:07 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:XinDQW5zDiHaZ7m29JLXHsJOugM=', 'Content-Length': '0'} 2023-05-31T20:21:07.003 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:18:05 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000f7b608f641388afa-006477ac32-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:21:06 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:21:07.004 DEBUG:boto:path=/pklxkd-49/obj4 2023-05-31T20:21:07.005 DEBUG:boto:auth_path=/pklxkd-49/obj4 2023-05-31T20:21:07.005 DEBUG:boto:path=/pklxkd-49/obj4?versionId=null 2023-05-31T20:21:07.005 DEBUG:boto:auth_path=/pklxkd-49/obj4?versionId=null 2023-05-31T20:21:07.006 DEBUG:boto:Method: HEAD 2023-05-31T20:21:07.006 DEBUG:boto:Path: /pklxkd-49/obj4?versionId=null 2023-05-31T20:21:07.006 DEBUG:boto:Data: 2023-05-31T20:21:07.007 DEBUG:boto:Headers: {} 2023-05-31T20:21:07.007 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:21:07.007 DEBUG:boto:Port: 8001 2023-05-31T20:21:07.008 DEBUG:boto:Params: {} 2023-05-31T20:21:07.008 DEBUG:boto:Token: None 2023-05-31T20:21:07.008 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:21:07 GMT /pklxkd-49/obj4?versionId=null 2023-05-31T20:21:07.009 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:HoKvDP1xiFgKdsDH+4zInBCOzy0= 2023-05-31T20:21:07.009 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:21:07 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:HoKvDP1xiFgKdsDH+4zInBCOzy0=', 'Content-Length': '0'} 2023-05-31T20:21:07.011 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:18:05 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000004281489fe189e76a-006477ac33-162e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:21:07 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:21:07.011 DEBUG:boto:path=/pklxkd-49/obj4 2023-05-31T20:21:07.012 DEBUG:boto:auth_path=/pklxkd-49/obj4 2023-05-31T20:21:07.012 DEBUG:boto:path=/pklxkd-49/obj4?versionId=null 2023-05-31T20:21:07.012 DEBUG:boto:auth_path=/pklxkd-49/obj4?versionId=null 2023-05-31T20:21:07.013 DEBUG:boto:Method: HEAD 2023-05-31T20:21:07.013 DEBUG:boto:Path: /pklxkd-49/obj4?versionId=null 2023-05-31T20:21:07.013 DEBUG:boto:Data: 2023-05-31T20:21:07.014 DEBUG:boto:Headers: {} 2023-05-31T20:21:07.014 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:21:07.014 DEBUG:boto:Port: 8000 2023-05-31T20:21:07.015 DEBUG:boto:Params: {} 2023-05-31T20:21:07.015 DEBUG:boto:Token: None 2023-05-31T20:21:07.015 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:21:07 GMT /pklxkd-49/obj4?versionId=null 2023-05-31T20:21:07.016 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:HoKvDP1xiFgKdsDH+4zInBCOzy0= 2023-05-31T20:21:07.016 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:21:07 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:HoKvDP1xiFgKdsDH+4zInBCOzy0=', 'Content-Length': '0'} 2023-05-31T20:21:07.018 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:18:05 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000037e56ae98e0720a0-006477ac33-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:21:07 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:21:07.018 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2023-05-31T20:21:07.019 DEBUG:boto:path=/pklxkd-49/obj4 2023-05-31T20:21:07.019 DEBUG:boto:auth_path=/pklxkd-49/obj4 2023-05-31T20:21:07.019 DEBUG:boto:path=/pklxkd-49/obj4?versionId=null 2023-05-31T20:21:07.020 DEBUG:boto:auth_path=/pklxkd-49/obj4?versionId=null 2023-05-31T20:21:07.020 DEBUG:boto:Method: GET 2023-05-31T20:21:07.020 DEBUG:boto:Path: /pklxkd-49/obj4?versionId=null 2023-05-31T20:21:07.021 DEBUG:boto:Data: 2023-05-31T20:21:07.021 DEBUG:boto:Headers: {} 2023-05-31T20:21:07.022 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:21:07.022 DEBUG:boto:Port: 8001 2023-05-31T20:21:07.022 DEBUG:boto:Params: {} 2023-05-31T20:21:07.023 DEBUG:boto:Token: None 2023-05-31T20:21:07.023 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:21:07 GMT /pklxkd-49/obj4?versionId=null 2023-05-31T20:21:07.023 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:XinDQW5zDiHaZ7m29JLXHsJOugM= 2023-05-31T20:21:07.024 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:21:07 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:XinDQW5zDiHaZ7m29JLXHsJOugM=', 'Content-Length': '0'} 2023-05-31T20:21:07.025 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:18:05 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000f3057d108e9963ae-006477ac33-162e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:21:07 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:21:07.026 DEBUG:boto:path=/pklxkd-49/obj4 2023-05-31T20:21:07.026 DEBUG:boto:auth_path=/pklxkd-49/obj4 2023-05-31T20:21:07.027 DEBUG:boto:path=/pklxkd-49/obj4?versionId=null 2023-05-31T20:21:07.027 DEBUG:boto:auth_path=/pklxkd-49/obj4?versionId=null 2023-05-31T20:21:07.028 DEBUG:boto:Method: GET 2023-05-31T20:21:07.028 DEBUG:boto:Path: /pklxkd-49/obj4?versionId=null 2023-05-31T20:21:07.028 DEBUG:boto:Data: 2023-05-31T20:21:07.029 DEBUG:boto:Headers: {} 2023-05-31T20:21:07.029 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:21:07.029 DEBUG:boto:Port: 8000 2023-05-31T20:21:07.030 DEBUG:boto:Params: {} 2023-05-31T20:21:07.030 DEBUG:boto:Token: None 2023-05-31T20:21:07.031 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:21:07 GMT /pklxkd-49/obj4?versionId=null 2023-05-31T20:21:07.031 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:XinDQW5zDiHaZ7m29JLXHsJOugM= 2023-05-31T20:21:07.031 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:21:07 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:XinDQW5zDiHaZ7m29JLXHsJOugM=', 'Content-Length': '0'} 2023-05-31T20:21:07.033 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:18:05 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000079b37bf0864faa18-006477ac33-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:21:07 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:21:07.033 DEBUG:boto:path=/pklxkd-49/ 2023-05-31T20:21:07.034 DEBUG:boto:auth_path=/pklxkd-49/ 2023-05-31T20:21:07.034 DEBUG:boto:path=/pklxkd-49/?prefix=obj4 2023-05-31T20:21:07.035 DEBUG:boto:auth_path=/pklxkd-49/?prefix=obj4 2023-05-31T20:21:07.035 DEBUG:boto:Method: GET 2023-05-31T20:21:07.036 DEBUG:boto:Path: /pklxkd-49/?prefix=obj4 2023-05-31T20:21:07.036 DEBUG:boto:Data: 2023-05-31T20:21:07.036 DEBUG:boto:Headers: {} 2023-05-31T20:21:07.037 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:21:07.037 DEBUG:boto:Port: 8001 2023-05-31T20:21:07.037 DEBUG:boto:Params: {} 2023-05-31T20:21:07.038 DEBUG:boto:Token: None 2023-05-31T20:21:07.038 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:21:07 GMT /pklxkd-49/ 2023-05-31T20:21:07.039 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:hg2FCb+7UP4DNsqWInfYUiAGm9c= 2023-05-31T20:21:07.039 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:21:07 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:hg2FCb+7UP4DNsqWInfYUiAGm9c=', 'Content-Length': '0'} 2023-05-31T20:21:07.041 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000003c1e0ef2c2576c34-006477ac33-162e-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:21:07 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:21:07.042 DEBUG:boto:b'pklxkd-49obj41000falseobj42023-05-31T20:18:05.509Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:21:07.042 DEBUG:boto:path=/pklxkd-49/ 2023-05-31T20:21:07.043 DEBUG:boto:auth_path=/pklxkd-49/ 2023-05-31T20:21:07.043 DEBUG:boto:path=/pklxkd-49/?prefix=obj4 2023-05-31T20:21:07.044 DEBUG:boto:auth_path=/pklxkd-49/?prefix=obj4 2023-05-31T20:21:07.044 DEBUG:boto:Method: GET 2023-05-31T20:21:07.044 DEBUG:boto:Path: /pklxkd-49/?prefix=obj4 2023-05-31T20:21:07.045 DEBUG:boto:Data: 2023-05-31T20:21:07.045 DEBUG:boto:Headers: {} 2023-05-31T20:21:07.045 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:21:07.046 DEBUG:boto:Port: 8000 2023-05-31T20:21:07.046 DEBUG:boto:Params: {} 2023-05-31T20:21:07.047 DEBUG:boto:Token: None 2023-05-31T20:21:07.047 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:21:07 GMT /pklxkd-49/ 2023-05-31T20:21:07.047 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:hg2FCb+7UP4DNsqWInfYUiAGm9c= 2023-05-31T20:21:07.048 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:21:07 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:hg2FCb+7UP4DNsqWInfYUiAGm9c=', 'Content-Length': '0'} 2023-05-31T20:21:07.050 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000091b5edb4373c3308-006477ac33-1630-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:21:07 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:21:07.096 DEBUG:boto:b'pklxkd-49obj41000falseobj42023-05-31T20:18:05.509Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:21:07.097 DEBUG:boto:path=/pklxkd-49/obj4 2023-05-31T20:21:07.097 DEBUG:boto:auth_path=/pklxkd-49/obj4 2023-05-31T20:21:07.098 DEBUG:boto:Method: HEAD 2023-05-31T20:21:07.098 DEBUG:boto:Path: /pklxkd-49/obj4 2023-05-31T20:21:07.098 DEBUG:boto:Data: 2023-05-31T20:21:07.099 DEBUG:boto:Headers: {} 2023-05-31T20:21:07.099 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:21:07.100 DEBUG:boto:Port: 8001 2023-05-31T20:21:07.100 DEBUG:boto:Params: {} 2023-05-31T20:21:07.100 DEBUG:boto:Token: None 2023-05-31T20:21:07.101 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:21:07 GMT /pklxkd-49/obj4 2023-05-31T20:21:07.102 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:caR6vjfcw+6DVqVfPvyJxY0kHpc= 2023-05-31T20:21:07.102 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:21:07 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:caR6vjfcw+6DVqVfPvyJxY0kHpc=', 'Content-Length': '0'} 2023-05-31T20:21:07.104 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:18:05 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000243d7011ea1c0206-006477ac33-162e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:21:07 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:21:07.104 DEBUG:boto:path=/pklxkd-49/obj4 2023-05-31T20:21:07.105 DEBUG:boto:auth_path=/pklxkd-49/obj4 2023-05-31T20:21:07.105 DEBUG:boto:Method: HEAD 2023-05-31T20:21:07.106 DEBUG:boto:Path: /pklxkd-49/obj4 2023-05-31T20:21:07.107 DEBUG:boto:Data: 2023-05-31T20:21:07.108 DEBUG:boto:Headers: {} 2023-05-31T20:21:07.108 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:21:07.109 DEBUG:boto:Port: 8000 2023-05-31T20:21:07.109 DEBUG:boto:Params: {} 2023-05-31T20:21:07.110 DEBUG:boto:Token: None 2023-05-31T20:21:07.110 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:21:07 GMT /pklxkd-49/obj4 2023-05-31T20:21:07.111 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:caR6vjfcw+6DVqVfPvyJxY0kHpc= 2023-05-31T20:21:07.112 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:21:07 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:caR6vjfcw+6DVqVfPvyJxY0kHpc=', 'Content-Length': '0'} 2023-05-31T20:21:07.114 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:18:05 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000063dda3f648f9f5c9-006477ac33-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:21:07 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:21:07.115 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2023-05-31T20:21:07.116 DEBUG:boto:path=/pklxkd-49/obj4 2023-05-31T20:21:07.116 DEBUG:boto:auth_path=/pklxkd-49/obj4 2023-05-31T20:21:07.117 DEBUG:boto:Method: GET 2023-05-31T20:21:07.117 DEBUG:boto:Path: /pklxkd-49/obj4 2023-05-31T20:21:07.118 DEBUG:boto:Data: 2023-05-31T20:21:07.118 DEBUG:boto:Headers: {} 2023-05-31T20:21:07.118 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:21:07.119 DEBUG:boto:Port: 8001 2023-05-31T20:21:07.119 DEBUG:boto:Params: {} 2023-05-31T20:21:07.119 DEBUG:boto:Token: None 2023-05-31T20:21:07.120 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:21:07 GMT /pklxkd-49/obj4 2023-05-31T20:21:07.120 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:yLpYLU1UAy4Y49TykEoZ14nHBtA= 2023-05-31T20:21:07.121 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:21:07 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:yLpYLU1UAy4Y49TykEoZ14nHBtA=', 'Content-Length': '0'} 2023-05-31T20:21:07.123 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:18:05 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000000facc99d731e31c1-006477ac33-162e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:21:07 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:21:07.123 DEBUG:boto:path=/pklxkd-49/obj4 2023-05-31T20:21:07.124 DEBUG:boto:auth_path=/pklxkd-49/obj4 2023-05-31T20:21:07.124 DEBUG:boto:Method: GET 2023-05-31T20:21:07.125 DEBUG:boto:Path: /pklxkd-49/obj4 2023-05-31T20:21:07.125 DEBUG:boto:Data: 2023-05-31T20:21:07.125 DEBUG:boto:Headers: {} 2023-05-31T20:21:07.126 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:21:07.126 DEBUG:boto:Port: 8000 2023-05-31T20:21:07.126 DEBUG:boto:Params: {} 2023-05-31T20:21:07.127 DEBUG:boto:Token: None 2023-05-31T20:21:07.127 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:21:07 GMT /pklxkd-49/obj4 2023-05-31T20:21:07.128 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:yLpYLU1UAy4Y49TykEoZ14nHBtA= 2023-05-31T20:21:07.128 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:21:07 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:yLpYLU1UAy4Y49TykEoZ14nHBtA=', 'Content-Length': '0'} 2023-05-31T20:21:07.130 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:18:05 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000004b9e8bf87d75f37e-006477ac33-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:21:07 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:21:07.130 DEBUG:boto:path=/pklxkd-49/ 2023-05-31T20:21:07.131 DEBUG:boto:auth_path=/pklxkd-49/ 2023-05-31T20:21:07.131 DEBUG:boto:path=/pklxkd-49/?prefix=obj4 2023-05-31T20:21:07.131 DEBUG:boto:auth_path=/pklxkd-49/?prefix=obj4 2023-05-31T20:21:07.132 DEBUG:boto:Method: GET 2023-05-31T20:21:07.132 DEBUG:boto:Path: /pklxkd-49/?prefix=obj4 2023-05-31T20:21:07.133 DEBUG:boto:Data: 2023-05-31T20:21:07.133 DEBUG:boto:Headers: {} 2023-05-31T20:21:07.133 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:21:07.134 DEBUG:boto:Port: 8001 2023-05-31T20:21:07.134 DEBUG:boto:Params: {} 2023-05-31T20:21:07.134 DEBUG:boto:Token: None 2023-05-31T20:21:07.135 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:21:07 GMT /pklxkd-49/ 2023-05-31T20:21:07.135 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:hg2FCb+7UP4DNsqWInfYUiAGm9c= 2023-05-31T20:21:07.136 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:21:07 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:hg2FCb+7UP4DNsqWInfYUiAGm9c=', 'Content-Length': '0'} 2023-05-31T20:21:07.138 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e78c3b5d30ed7f86-006477ac33-162e-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:21:07 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:21:07.138 DEBUG:boto:b'pklxkd-49obj41000falseobj42023-05-31T20:18:05.509Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:21:07.140 DEBUG:boto:path=/pklxkd-49/ 2023-05-31T20:21:07.141 DEBUG:boto:auth_path=/pklxkd-49/ 2023-05-31T20:21:07.141 DEBUG:boto:path=/pklxkd-49/?prefix=obj4 2023-05-31T20:21:07.141 DEBUG:boto:auth_path=/pklxkd-49/?prefix=obj4 2023-05-31T20:21:07.142 DEBUG:boto:Method: GET 2023-05-31T20:21:07.142 DEBUG:boto:Path: /pklxkd-49/?prefix=obj4 2023-05-31T20:21:07.142 DEBUG:boto:Data: 2023-05-31T20:21:07.143 DEBUG:boto:Headers: {} 2023-05-31T20:21:07.143 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:21:07.143 DEBUG:boto:Port: 8000 2023-05-31T20:21:07.144 DEBUG:boto:Params: {} 2023-05-31T20:21:07.144 DEBUG:boto:Token: None 2023-05-31T20:21:07.145 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:21:07 GMT /pklxkd-49/ 2023-05-31T20:21:07.147 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:hg2FCb+7UP4DNsqWInfYUiAGm9c= 2023-05-31T20:21:07.148 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:21:07 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:hg2FCb+7UP4DNsqWInfYUiAGm9c=', 'Content-Length': '0'} 2023-05-31T20:21:07.150 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c9e6638cc18d0bb2-006477ac33-1630-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:21:07 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:21:07.151 DEBUG:boto:b'pklxkd-49obj41000falseobj42023-05-31T20:18:05.509Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:21:07.152 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj5 2023-05-31T20:21:07.152 DEBUG:boto:path=/pklxkd-49/obj5 2023-05-31T20:21:07.153 DEBUG:boto:auth_path=/pklxkd-49/obj5 2023-05-31T20:21:07.154 DEBUG:boto:path=/pklxkd-49/obj5?versionId=null 2023-05-31T20:21:07.154 DEBUG:boto:auth_path=/pklxkd-49/obj5?versionId=null 2023-05-31T20:21:07.155 DEBUG:boto:Method: GET 2023-05-31T20:21:07.155 DEBUG:boto:Path: /pklxkd-49/obj5?versionId=null 2023-05-31T20:21:07.156 DEBUG:boto:Data: 2023-05-31T20:21:07.157 DEBUG:boto:Headers: {} 2023-05-31T20:21:07.157 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:21:07.158 DEBUG:boto:Port: 8001 2023-05-31T20:21:07.159 DEBUG:boto:Params: {} 2023-05-31T20:21:07.159 DEBUG:boto:Token: None 2023-05-31T20:21:07.160 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:21:07 GMT /pklxkd-49/obj5?versionId=null 2023-05-31T20:21:07.161 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:y8lvgx16ooXOTvAW5dMT4y1Ilvc= 2023-05-31T20:21:07.162 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:21:07 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:y8lvgx16ooXOTvAW5dMT4y1Ilvc=', 'Content-Length': '0'} 2023-05-31T20:21:07.164 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:20:28 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000007d40ae4a4de449c1-006477ac33-162e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:21:07 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:21:07.165 DEBUG:boto:path=/pklxkd-49/obj5 2023-05-31T20:21:07.165 DEBUG:boto:auth_path=/pklxkd-49/obj5 2023-05-31T20:21:07.166 DEBUG:boto:path=/pklxkd-49/obj5?versionId=null 2023-05-31T20:21:07.166 DEBUG:boto:auth_path=/pklxkd-49/obj5?versionId=null 2023-05-31T20:21:07.166 DEBUG:boto:Method: GET 2023-05-31T20:21:07.167 DEBUG:boto:Path: /pklxkd-49/obj5?versionId=null 2023-05-31T20:21:07.167 DEBUG:boto:Data: 2023-05-31T20:21:07.167 DEBUG:boto:Headers: {} 2023-05-31T20:21:07.168 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:21:07.168 DEBUG:boto:Port: 8000 2023-05-31T20:21:07.169 DEBUG:boto:Params: {} 2023-05-31T20:21:07.169 DEBUG:boto:Token: None 2023-05-31T20:21:07.170 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:21:07 GMT /pklxkd-49/obj5?versionId=null 2023-05-31T20:21:07.171 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:y8lvgx16ooXOTvAW5dMT4y1Ilvc= 2023-05-31T20:21:07.172 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:21:07 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:y8lvgx16ooXOTvAW5dMT4y1Ilvc=', 'Content-Length': '0'} 2023-05-31T20:21:07.174 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:20:28 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000c3f46449b3f9f33b-006477ac33-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:21:07 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:21:07.176 DEBUG:boto:path=/pklxkd-49/obj5 2023-05-31T20:21:07.176 DEBUG:boto:auth_path=/pklxkd-49/obj5 2023-05-31T20:21:07.177 DEBUG:boto:path=/pklxkd-49/obj5?versionId=null 2023-05-31T20:21:07.178 DEBUG:boto:auth_path=/pklxkd-49/obj5?versionId=null 2023-05-31T20:21:07.179 DEBUG:boto:Method: HEAD 2023-05-31T20:21:07.179 DEBUG:boto:Path: /pklxkd-49/obj5?versionId=null 2023-05-31T20:21:07.180 DEBUG:boto:Data: 2023-05-31T20:21:07.181 DEBUG:boto:Headers: {} 2023-05-31T20:21:07.181 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:21:07.182 DEBUG:boto:Port: 8001 2023-05-31T20:21:07.183 DEBUG:boto:Params: {} 2023-05-31T20:21:07.184 DEBUG:boto:Token: None 2023-05-31T20:21:07.185 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:21:07 GMT /pklxkd-49/obj5?versionId=null 2023-05-31T20:21:07.185 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:3QOUjR3WbRTPS8i0I/678PXzqGA= 2023-05-31T20:21:07.186 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:21:07 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:3QOUjR3WbRTPS8i0I/678PXzqGA=', 'Content-Length': '0'} 2023-05-31T20:21:07.188 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:20:28 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000f47e1d23356163af-006477ac33-162e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:21:07 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:21:07.189 DEBUG:boto:path=/pklxkd-49/obj5 2023-05-31T20:21:07.189 DEBUG:boto:auth_path=/pklxkd-49/obj5 2023-05-31T20:21:07.189 DEBUG:boto:path=/pklxkd-49/obj5?versionId=null 2023-05-31T20:21:07.190 DEBUG:boto:auth_path=/pklxkd-49/obj5?versionId=null 2023-05-31T20:21:07.190 DEBUG:boto:Method: HEAD 2023-05-31T20:21:07.190 DEBUG:boto:Path: /pklxkd-49/obj5?versionId=null 2023-05-31T20:21:07.191 DEBUG:boto:Data: 2023-05-31T20:21:07.191 DEBUG:boto:Headers: {} 2023-05-31T20:21:07.191 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:21:07.192 DEBUG:boto:Port: 8000 2023-05-31T20:21:07.192 DEBUG:boto:Params: {} 2023-05-31T20:21:07.192 DEBUG:boto:Token: None 2023-05-31T20:21:07.193 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:21:07 GMT /pklxkd-49/obj5?versionId=null 2023-05-31T20:21:07.193 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:3QOUjR3WbRTPS8i0I/678PXzqGA= 2023-05-31T20:21:07.194 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:21:07 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:3QOUjR3WbRTPS8i0I/678PXzqGA=', 'Content-Length': '0'} 2023-05-31T20:21:07.195 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:20:28 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000053ae72d8f5339a28-006477ac33-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:21:07 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:21:07.196 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj5 2023-05-31T20:21:07.197 DEBUG:boto:path=/pklxkd-49/obj5 2023-05-31T20:21:07.197 DEBUG:boto:auth_path=/pklxkd-49/obj5 2023-05-31T20:21:07.198 DEBUG:boto:path=/pklxkd-49/obj5?versionId=null 2023-05-31T20:21:07.199 DEBUG:boto:auth_path=/pklxkd-49/obj5?versionId=null 2023-05-31T20:21:07.199 DEBUG:boto:Method: GET 2023-05-31T20:21:07.200 DEBUG:boto:Path: /pklxkd-49/obj5?versionId=null 2023-05-31T20:21:07.200 DEBUG:boto:Data: 2023-05-31T20:21:07.201 DEBUG:boto:Headers: {} 2023-05-31T20:21:07.201 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:21:07.202 DEBUG:boto:Port: 8001 2023-05-31T20:21:07.203 DEBUG:boto:Params: {} 2023-05-31T20:21:07.204 DEBUG:boto:Token: None 2023-05-31T20:21:07.204 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:21:07 GMT /pklxkd-49/obj5?versionId=null 2023-05-31T20:21:07.205 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:y8lvgx16ooXOTvAW5dMT4y1Ilvc= 2023-05-31T20:21:07.206 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:21:07 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:y8lvgx16ooXOTvAW5dMT4y1Ilvc=', 'Content-Length': '0'} 2023-05-31T20:21:07.209 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:20:28 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000e6bcc914cd7b5c3c-006477ac33-162e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:21:07 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:21:07.210 DEBUG:boto:path=/pklxkd-49/obj5 2023-05-31T20:21:07.211 DEBUG:boto:auth_path=/pklxkd-49/obj5 2023-05-31T20:21:07.211 DEBUG:boto:path=/pklxkd-49/obj5?versionId=null 2023-05-31T20:21:07.212 DEBUG:boto:auth_path=/pklxkd-49/obj5?versionId=null 2023-05-31T20:21:07.212 DEBUG:boto:Method: GET 2023-05-31T20:21:07.213 DEBUG:boto:Path: /pklxkd-49/obj5?versionId=null 2023-05-31T20:21:07.213 DEBUG:boto:Data: 2023-05-31T20:21:07.214 DEBUG:boto:Headers: {} 2023-05-31T20:21:07.215 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:21:07.215 DEBUG:boto:Port: 8000 2023-05-31T20:21:07.216 DEBUG:boto:Params: {} 2023-05-31T20:21:07.217 DEBUG:boto:Token: None 2023-05-31T20:21:07.218 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:21:07 GMT /pklxkd-49/obj5?versionId=null 2023-05-31T20:21:07.219 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:y8lvgx16ooXOTvAW5dMT4y1Ilvc= 2023-05-31T20:21:07.220 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:21:07 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:y8lvgx16ooXOTvAW5dMT4y1Ilvc=', 'Content-Length': '0'} 2023-05-31T20:21:07.222 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:20:28 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000004517d4a92ab74151-006477ac33-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:21:07 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:21:07.223 DEBUG:boto:path=/pklxkd-49/ 2023-05-31T20:21:07.224 DEBUG:boto:auth_path=/pklxkd-49/ 2023-05-31T20:21:07.224 DEBUG:boto:path=/pklxkd-49/?prefix=obj5 2023-05-31T20:21:07.225 DEBUG:boto:auth_path=/pklxkd-49/?prefix=obj5 2023-05-31T20:21:07.226 DEBUG:boto:Method: GET 2023-05-31T20:21:07.227 DEBUG:boto:Path: /pklxkd-49/?prefix=obj5 2023-05-31T20:21:07.227 DEBUG:boto:Data: 2023-05-31T20:21:07.228 DEBUG:boto:Headers: {} 2023-05-31T20:21:07.229 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:21:07.229 DEBUG:boto:Port: 8001 2023-05-31T20:21:07.230 DEBUG:boto:Params: {} 2023-05-31T20:21:07.231 DEBUG:boto:Token: None 2023-05-31T20:21:07.232 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:21:07 GMT /pklxkd-49/ 2023-05-31T20:21:07.232 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:hg2FCb+7UP4DNsqWInfYUiAGm9c= 2023-05-31T20:21:07.233 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:21:07 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:hg2FCb+7UP4DNsqWInfYUiAGm9c=', 'Content-Length': '0'} 2023-05-31T20:21:07.235 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000433fd29199349e22-006477ac33-162e-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:21:07 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:21:07.236 DEBUG:boto:b'pklxkd-49obj51000falseobj52023-05-31T20:20:28.797Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:21:07.236 DEBUG:boto:path=/pklxkd-49/ 2023-05-31T20:21:07.237 DEBUG:boto:auth_path=/pklxkd-49/ 2023-05-31T20:21:07.237 DEBUG:boto:path=/pklxkd-49/?prefix=obj5 2023-05-31T20:21:07.237 DEBUG:boto:auth_path=/pklxkd-49/?prefix=obj5 2023-05-31T20:21:07.238 DEBUG:boto:Method: GET 2023-05-31T20:21:07.238 DEBUG:boto:Path: /pklxkd-49/?prefix=obj5 2023-05-31T20:21:07.239 DEBUG:boto:Data: 2023-05-31T20:21:07.239 DEBUG:boto:Headers: {} 2023-05-31T20:21:07.240 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:21:07.240 DEBUG:boto:Port: 8000 2023-05-31T20:21:07.241 DEBUG:boto:Params: {} 2023-05-31T20:21:07.242 DEBUG:boto:Token: None 2023-05-31T20:21:07.243 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:21:07 GMT /pklxkd-49/ 2023-05-31T20:21:07.243 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:hg2FCb+7UP4DNsqWInfYUiAGm9c= 2023-05-31T20:21:07.244 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:21:07 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:hg2FCb+7UP4DNsqWInfYUiAGm9c=', 'Content-Length': '0'} 2023-05-31T20:21:07.246 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000267b24a023572104-006477ac33-1630-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:21:07 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:21:07.246 DEBUG:boto:b'pklxkd-49obj51000falseobj52023-05-31T20:20:28.797Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:21:07.247 DEBUG:boto:path=/pklxkd-49/obj5 2023-05-31T20:21:07.247 DEBUG:boto:auth_path=/pklxkd-49/obj5 2023-05-31T20:21:07.248 DEBUG:boto:Method: HEAD 2023-05-31T20:21:07.248 DEBUG:boto:Path: /pklxkd-49/obj5 2023-05-31T20:21:07.249 DEBUG:boto:Data: 2023-05-31T20:21:07.249 DEBUG:boto:Headers: {} 2023-05-31T20:21:07.249 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:21:07.250 DEBUG:boto:Port: 8001 2023-05-31T20:21:07.250 DEBUG:boto:Params: {} 2023-05-31T20:21:07.250 DEBUG:boto:Token: None 2023-05-31T20:21:07.251 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:21:07 GMT /pklxkd-49/obj5 2023-05-31T20:21:07.251 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:f3uE/ef/kv8GvNdRGlMGTgBNg3A= 2023-05-31T20:21:07.251 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:21:07 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:f3uE/ef/kv8GvNdRGlMGTgBNg3A=', 'Content-Length': '0'} 2023-05-31T20:21:07.253 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:20:28 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000061590cfeb565a929-006477ac33-162e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:21:07 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:21:07.253 DEBUG:boto:path=/pklxkd-49/obj5 2023-05-31T20:21:07.254 DEBUG:boto:auth_path=/pklxkd-49/obj5 2023-05-31T20:21:07.254 DEBUG:boto:Method: HEAD 2023-05-31T20:21:07.254 DEBUG:boto:Path: /pklxkd-49/obj5 2023-05-31T20:21:07.255 DEBUG:boto:Data: 2023-05-31T20:21:07.255 DEBUG:boto:Headers: {} 2023-05-31T20:21:07.255 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:21:07.256 DEBUG:boto:Port: 8000 2023-05-31T20:21:07.256 DEBUG:boto:Params: {} 2023-05-31T20:21:07.257 DEBUG:boto:Token: None 2023-05-31T20:21:07.257 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:21:07 GMT /pklxkd-49/obj5 2023-05-31T20:21:07.257 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:f3uE/ef/kv8GvNdRGlMGTgBNg3A= 2023-05-31T20:21:07.258 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:21:07 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:f3uE/ef/kv8GvNdRGlMGTgBNg3A=', 'Content-Length': '0'} 2023-05-31T20:21:07.259 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:20:28 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000cf020a7361f384b6-006477ac33-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:21:07 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:21:07.260 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj5 2023-05-31T20:21:07.260 DEBUG:boto:path=/pklxkd-49/obj5 2023-05-31T20:21:07.261 DEBUG:boto:auth_path=/pklxkd-49/obj5 2023-05-31T20:21:07.261 DEBUG:boto:Method: GET 2023-05-31T20:21:07.261 DEBUG:boto:Path: /pklxkd-49/obj5 2023-05-31T20:21:07.262 DEBUG:boto:Data: 2023-05-31T20:21:07.262 DEBUG:boto:Headers: {} 2023-05-31T20:21:07.262 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:21:07.263 DEBUG:boto:Port: 8001 2023-05-31T20:21:07.263 DEBUG:boto:Params: {} 2023-05-31T20:21:07.263 DEBUG:boto:Token: None 2023-05-31T20:21:07.264 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:21:07 GMT /pklxkd-49/obj5 2023-05-31T20:21:07.264 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:NZT4XRJLQx8Cc4W6V/cTimBIIpQ= 2023-05-31T20:21:07.265 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:21:07 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:NZT4XRJLQx8Cc4W6V/cTimBIIpQ=', 'Content-Length': '0'} 2023-05-31T20:21:07.266 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:20:28 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000df9e95b6ca7761c3-006477ac33-162e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:21:07 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:21:07.268 DEBUG:boto:path=/pklxkd-49/obj5 2023-05-31T20:21:07.269 DEBUG:boto:auth_path=/pklxkd-49/obj5 2023-05-31T20:21:07.269 DEBUG:boto:Method: GET 2023-05-31T20:21:07.269 DEBUG:boto:Path: /pklxkd-49/obj5 2023-05-31T20:21:07.270 DEBUG:boto:Data: 2023-05-31T20:21:07.270 DEBUG:boto:Headers: {} 2023-05-31T20:21:07.270 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:21:07.271 DEBUG:boto:Port: 8000 2023-05-31T20:21:07.271 DEBUG:boto:Params: {} 2023-05-31T20:21:07.272 DEBUG:boto:Token: None 2023-05-31T20:21:07.272 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:21:07 GMT /pklxkd-49/obj5 2023-05-31T20:21:07.272 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:NZT4XRJLQx8Cc4W6V/cTimBIIpQ= 2023-05-31T20:21:07.273 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:21:07 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:NZT4XRJLQx8Cc4W6V/cTimBIIpQ=', 'Content-Length': '0'} 2023-05-31T20:21:07.278 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:20:28 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000008afb28deb28919b-006477ac33-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:21:07 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:21:07.279 DEBUG:boto:path=/pklxkd-49/ 2023-05-31T20:21:07.280 DEBUG:boto:auth_path=/pklxkd-49/ 2023-05-31T20:21:07.280 DEBUG:boto:path=/pklxkd-49/?prefix=obj5 2023-05-31T20:21:07.280 DEBUG:boto:auth_path=/pklxkd-49/?prefix=obj5 2023-05-31T20:21:07.281 DEBUG:boto:Method: GET 2023-05-31T20:21:07.281 DEBUG:boto:Path: /pklxkd-49/?prefix=obj5 2023-05-31T20:21:07.281 DEBUG:boto:Data: 2023-05-31T20:21:07.282 DEBUG:boto:Headers: {} 2023-05-31T20:21:07.282 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:21:07.282 DEBUG:boto:Port: 8001 2023-05-31T20:21:07.283 DEBUG:boto:Params: {} 2023-05-31T20:21:07.283 DEBUG:boto:Token: None 2023-05-31T20:21:07.284 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:21:07 GMT /pklxkd-49/ 2023-05-31T20:21:07.284 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:hg2FCb+7UP4DNsqWInfYUiAGm9c= 2023-05-31T20:21:07.285 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:21:07 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:hg2FCb+7UP4DNsqWInfYUiAGm9c=', 'Content-Length': '0'} 2023-05-31T20:21:07.286 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000008005cf0d7aa3cf9d-006477ac33-162e-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:21:07 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:21:07.328 DEBUG:boto:b'pklxkd-49obj51000falseobj52023-05-31T20:20:28.797Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:21:07.329 DEBUG:boto:path=/pklxkd-49/ 2023-05-31T20:21:07.330 DEBUG:boto:auth_path=/pklxkd-49/ 2023-05-31T20:21:07.330 DEBUG:boto:path=/pklxkd-49/?prefix=obj5 2023-05-31T20:21:07.331 DEBUG:boto:auth_path=/pklxkd-49/?prefix=obj5 2023-05-31T20:21:07.331 DEBUG:boto:Method: GET 2023-05-31T20:21:07.331 DEBUG:boto:Path: /pklxkd-49/?prefix=obj5 2023-05-31T20:21:07.332 DEBUG:boto:Data: 2023-05-31T20:21:07.332 DEBUG:boto:Headers: {} 2023-05-31T20:21:07.333 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:21:07.333 DEBUG:boto:Port: 8000 2023-05-31T20:21:07.334 DEBUG:boto:Params: {} 2023-05-31T20:21:07.334 DEBUG:boto:Token: None 2023-05-31T20:21:07.334 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:21:07 GMT /pklxkd-49/ 2023-05-31T20:21:07.335 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:hg2FCb+7UP4DNsqWInfYUiAGm9c= 2023-05-31T20:21:07.335 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:21:07 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:hg2FCb+7UP4DNsqWInfYUiAGm9c=', 'Content-Length': '0'} 2023-05-31T20:21:07.338 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000000aac03389b5f246d-006477ac33-1630-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:21:07 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:21:07.338 DEBUG:boto:b'pklxkd-49obj51000falseobj52023-05-31T20:20:28.797Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:21:07.339 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj6 2023-05-31T20:21:07.339 DEBUG:boto:path=/pklxkd-49/obj6 2023-05-31T20:21:07.340 DEBUG:boto:auth_path=/pklxkd-49/obj6 2023-05-31T20:21:07.340 DEBUG:boto:path=/pklxkd-49/obj6?versionId=null 2023-05-31T20:21:07.340 DEBUG:boto:auth_path=/pklxkd-49/obj6?versionId=null 2023-05-31T20:21:07.341 DEBUG:boto:Method: GET 2023-05-31T20:21:07.341 DEBUG:boto:Path: /pklxkd-49/obj6?versionId=null 2023-05-31T20:21:07.341 DEBUG:boto:Data: 2023-05-31T20:21:07.342 DEBUG:boto:Headers: {} 2023-05-31T20:21:07.342 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:21:07.343 DEBUG:boto:Port: 8001 2023-05-31T20:21:07.343 DEBUG:boto:Params: {} 2023-05-31T20:21:07.343 DEBUG:boto:Token: None 2023-05-31T20:21:07.344 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:21:07 GMT /pklxkd-49/obj6?versionId=null 2023-05-31T20:21:07.344 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:4coj4DrIHPT5Dlvrvw6+tLNwg/A= 2023-05-31T20:21:07.345 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:21:07 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:4coj4DrIHPT5Dlvrvw6+tLNwg/A=', 'Content-Length': '0'} 2023-05-31T20:21:07.346 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:20:28 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000d33b7884fde04e9a-006477ac33-162e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:21:07 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:21:07.347 DEBUG:boto:path=/pklxkd-49/obj6 2023-05-31T20:21:07.348 DEBUG:boto:auth_path=/pklxkd-49/obj6 2023-05-31T20:21:07.348 DEBUG:boto:path=/pklxkd-49/obj6?versionId=null 2023-05-31T20:21:07.348 DEBUG:boto:auth_path=/pklxkd-49/obj6?versionId=null 2023-05-31T20:21:07.349 DEBUG:boto:Method: GET 2023-05-31T20:21:07.349 DEBUG:boto:Path: /pklxkd-49/obj6?versionId=null 2023-05-31T20:21:07.349 DEBUG:boto:Data: 2023-05-31T20:21:07.350 DEBUG:boto:Headers: {} 2023-05-31T20:21:07.350 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:21:07.350 DEBUG:boto:Port: 8000 2023-05-31T20:21:07.351 DEBUG:boto:Params: {} 2023-05-31T20:21:07.351 DEBUG:boto:Token: None 2023-05-31T20:21:07.352 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:21:07 GMT /pklxkd-49/obj6?versionId=null 2023-05-31T20:21:07.352 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:4coj4DrIHPT5Dlvrvw6+tLNwg/A= 2023-05-31T20:21:07.353 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:21:07 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:4coj4DrIHPT5Dlvrvw6+tLNwg/A=', 'Content-Length': '0'} 2023-05-31T20:21:07.354 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:20:28 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000003fa945b8059c556e-006477ac33-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:21:07 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:21:07.355 DEBUG:boto:path=/pklxkd-49/obj6 2023-05-31T20:21:07.355 DEBUG:boto:auth_path=/pklxkd-49/obj6 2023-05-31T20:21:07.356 DEBUG:boto:path=/pklxkd-49/obj6?versionId=null 2023-05-31T20:21:07.356 DEBUG:boto:auth_path=/pklxkd-49/obj6?versionId=null 2023-05-31T20:21:07.356 DEBUG:boto:Method: HEAD 2023-05-31T20:21:07.357 DEBUG:boto:Path: /pklxkd-49/obj6?versionId=null 2023-05-31T20:21:07.357 DEBUG:boto:Data: 2023-05-31T20:21:07.357 DEBUG:boto:Headers: {} 2023-05-31T20:21:07.358 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:21:07.358 DEBUG:boto:Port: 8001 2023-05-31T20:21:07.358 DEBUG:boto:Params: {} 2023-05-31T20:21:07.359 DEBUG:boto:Token: None 2023-05-31T20:21:07.359 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:21:07 GMT /pklxkd-49/obj6?versionId=null 2023-05-31T20:21:07.360 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:xBev9kjEQygpPB2Ww9gQDCzJOs8= 2023-05-31T20:21:07.360 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:21:07 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:xBev9kjEQygpPB2Ww9gQDCzJOs8=', 'Content-Length': '0'} 2023-05-31T20:21:07.361 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:20:28 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000005aa94eb7583a0b22-006477ac33-162e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:21:07 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:21:07.362 DEBUG:boto:path=/pklxkd-49/obj6 2023-05-31T20:21:07.362 DEBUG:boto:auth_path=/pklxkd-49/obj6 2023-05-31T20:21:07.363 DEBUG:boto:path=/pklxkd-49/obj6?versionId=null 2023-05-31T20:21:07.363 DEBUG:boto:auth_path=/pklxkd-49/obj6?versionId=null 2023-05-31T20:21:07.363 DEBUG:boto:Method: HEAD 2023-05-31T20:21:07.364 DEBUG:boto:Path: /pklxkd-49/obj6?versionId=null 2023-05-31T20:21:07.364 DEBUG:boto:Data: 2023-05-31T20:21:07.364 DEBUG:boto:Headers: {} 2023-05-31T20:21:07.365 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:21:07.365 DEBUG:boto:Port: 8000 2023-05-31T20:21:07.365 DEBUG:boto:Params: {} 2023-05-31T20:21:07.366 DEBUG:boto:Token: None 2023-05-31T20:21:07.366 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:21:07 GMT /pklxkd-49/obj6?versionId=null 2023-05-31T20:21:07.367 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:xBev9kjEQygpPB2Ww9gQDCzJOs8= 2023-05-31T20:21:07.367 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:21:07 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:xBev9kjEQygpPB2Ww9gQDCzJOs8=', 'Content-Length': '0'} 2023-05-31T20:21:07.369 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:20:28 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000fdb74b8622a57ec3-006477ac33-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:21:07 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:21:07.370 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj6 2023-05-31T20:21:07.370 DEBUG:boto:path=/pklxkd-49/obj6 2023-05-31T20:21:07.371 DEBUG:boto:auth_path=/pklxkd-49/obj6 2023-05-31T20:21:07.371 DEBUG:boto:path=/pklxkd-49/obj6?versionId=null 2023-05-31T20:21:07.371 DEBUG:boto:auth_path=/pklxkd-49/obj6?versionId=null 2023-05-31T20:21:07.372 DEBUG:boto:Method: GET 2023-05-31T20:21:07.372 DEBUG:boto:Path: /pklxkd-49/obj6?versionId=null 2023-05-31T20:21:07.372 DEBUG:boto:Data: 2023-05-31T20:21:07.373 DEBUG:boto:Headers: {} 2023-05-31T20:21:07.373 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:21:07.373 DEBUG:boto:Port: 8001 2023-05-31T20:21:07.374 DEBUG:boto:Params: {} 2023-05-31T20:21:07.374 DEBUG:boto:Token: None 2023-05-31T20:21:07.375 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:21:07 GMT /pklxkd-49/obj6?versionId=null 2023-05-31T20:21:07.375 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:4coj4DrIHPT5Dlvrvw6+tLNwg/A= 2023-05-31T20:21:07.376 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:21:07 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:4coj4DrIHPT5Dlvrvw6+tLNwg/A=', 'Content-Length': '0'} 2023-05-31T20:21:07.377 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:20:28 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000a9ebd7f24daffd6f-006477ac33-162e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:21:07 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:21:07.378 DEBUG:boto:path=/pklxkd-49/obj6 2023-05-31T20:21:07.378 DEBUG:boto:auth_path=/pklxkd-49/obj6 2023-05-31T20:21:07.378 DEBUG:boto:path=/pklxkd-49/obj6?versionId=null 2023-05-31T20:21:07.379 DEBUG:boto:auth_path=/pklxkd-49/obj6?versionId=null 2023-05-31T20:21:07.379 DEBUG:boto:Method: GET 2023-05-31T20:21:07.380 DEBUG:boto:Path: /pklxkd-49/obj6?versionId=null 2023-05-31T20:21:07.380 DEBUG:boto:Data: 2023-05-31T20:21:07.380 DEBUG:boto:Headers: {} 2023-05-31T20:21:07.381 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:21:07.381 DEBUG:boto:Port: 8000 2023-05-31T20:21:07.381 DEBUG:boto:Params: {} 2023-05-31T20:21:07.382 DEBUG:boto:Token: None 2023-05-31T20:21:07.382 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:21:07 GMT /pklxkd-49/obj6?versionId=null 2023-05-31T20:21:07.383 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:4coj4DrIHPT5Dlvrvw6+tLNwg/A= 2023-05-31T20:21:07.383 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:21:07 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:4coj4DrIHPT5Dlvrvw6+tLNwg/A=', 'Content-Length': '0'} 2023-05-31T20:21:07.385 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:20:28 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000609170fe56de4eb7-006477ac33-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:21:07 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:21:07.385 DEBUG:boto:path=/pklxkd-49/ 2023-05-31T20:21:07.386 DEBUG:boto:auth_path=/pklxkd-49/ 2023-05-31T20:21:07.386 DEBUG:boto:path=/pklxkd-49/?prefix=obj6 2023-05-31T20:21:07.386 DEBUG:boto:auth_path=/pklxkd-49/?prefix=obj6 2023-05-31T20:21:07.387 DEBUG:boto:Method: GET 2023-05-31T20:21:07.387 DEBUG:boto:Path: /pklxkd-49/?prefix=obj6 2023-05-31T20:21:07.388 DEBUG:boto:Data: 2023-05-31T20:21:07.388 DEBUG:boto:Headers: {} 2023-05-31T20:21:07.388 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:21:07.389 DEBUG:boto:Port: 8001 2023-05-31T20:21:07.389 DEBUG:boto:Params: {} 2023-05-31T20:21:07.389 DEBUG:boto:Token: None 2023-05-31T20:21:07.390 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:21:07 GMT /pklxkd-49/ 2023-05-31T20:21:07.390 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:hg2FCb+7UP4DNsqWInfYUiAGm9c= 2023-05-31T20:21:07.391 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:21:07 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:hg2FCb+7UP4DNsqWInfYUiAGm9c=', 'Content-Length': '0'} 2023-05-31T20:21:07.393 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b333e089d8e84ae5-006477ac33-162e-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:21:07 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:21:07.393 DEBUG:boto:b'pklxkd-49obj61000falseobj62023-05-31T20:20:28.814Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:21:07.394 DEBUG:boto:path=/pklxkd-49/ 2023-05-31T20:21:07.394 DEBUG:boto:auth_path=/pklxkd-49/ 2023-05-31T20:21:07.395 DEBUG:boto:path=/pklxkd-49/?prefix=obj6 2023-05-31T20:21:07.395 DEBUG:boto:auth_path=/pklxkd-49/?prefix=obj6 2023-05-31T20:21:07.396 DEBUG:boto:Method: GET 2023-05-31T20:21:07.396 DEBUG:boto:Path: /pklxkd-49/?prefix=obj6 2023-05-31T20:21:07.397 DEBUG:boto:Data: 2023-05-31T20:21:07.397 DEBUG:boto:Headers: {} 2023-05-31T20:21:07.397 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:21:07.398 DEBUG:boto:Port: 8000 2023-05-31T20:21:07.398 DEBUG:boto:Params: {} 2023-05-31T20:21:07.399 DEBUG:boto:Token: None 2023-05-31T20:21:07.399 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:21:07 GMT /pklxkd-49/ 2023-05-31T20:21:07.400 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:hg2FCb+7UP4DNsqWInfYUiAGm9c= 2023-05-31T20:21:07.400 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:21:07 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:hg2FCb+7UP4DNsqWInfYUiAGm9c=', 'Content-Length': '0'} 2023-05-31T20:21:07.402 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a2acd2a2f5c1002b-006477ac33-1630-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:21:07 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:21:07.444 DEBUG:boto:b'pklxkd-49obj61000falseobj62023-05-31T20:20:28.814Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:21:07.446 DEBUG:boto:path=/pklxkd-49/obj6 2023-05-31T20:21:07.447 DEBUG:boto:auth_path=/pklxkd-49/obj6 2023-05-31T20:21:07.448 DEBUG:boto:Method: HEAD 2023-05-31T20:21:07.449 DEBUG:boto:Path: /pklxkd-49/obj6 2023-05-31T20:21:07.450 DEBUG:boto:Data: 2023-05-31T20:21:07.451 DEBUG:boto:Headers: {} 2023-05-31T20:21:07.452 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:21:07.453 DEBUG:boto:Port: 8001 2023-05-31T20:21:07.453 DEBUG:boto:Params: {} 2023-05-31T20:21:07.454 DEBUG:boto:Token: None 2023-05-31T20:21:07.455 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:21:07 GMT /pklxkd-49/obj6 2023-05-31T20:21:07.455 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:CcuIW4P3mJFYMHz2M0K2xFU9Ugc= 2023-05-31T20:21:07.456 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:21:07 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:CcuIW4P3mJFYMHz2M0K2xFU9Ugc=', 'Content-Length': '0'} 2023-05-31T20:21:07.458 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:20:28 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000001a87fdb2422209f6-006477ac33-162e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:21:07 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:21:07.458 DEBUG:boto:path=/pklxkd-49/obj6 2023-05-31T20:21:07.459 DEBUG:boto:auth_path=/pklxkd-49/obj6 2023-05-31T20:21:07.459 DEBUG:boto:Method: HEAD 2023-05-31T20:21:07.459 DEBUG:boto:Path: /pklxkd-49/obj6 2023-05-31T20:21:07.460 DEBUG:boto:Data: 2023-05-31T20:21:07.460 DEBUG:boto:Headers: {} 2023-05-31T20:21:07.461 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:21:07.461 DEBUG:boto:Port: 8000 2023-05-31T20:21:07.461 DEBUG:boto:Params: {} 2023-05-31T20:21:07.462 DEBUG:boto:Token: None 2023-05-31T20:21:07.463 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:21:07 GMT /pklxkd-49/obj6 2023-05-31T20:21:07.463 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:CcuIW4P3mJFYMHz2M0K2xFU9Ugc= 2023-05-31T20:21:07.464 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:21:07 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:CcuIW4P3mJFYMHz2M0K2xFU9Ugc=', 'Content-Length': '0'} 2023-05-31T20:21:07.465 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:20:28 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000318c49899c2f545d-006477ac33-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:21:07 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:21:07.466 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj6 2023-05-31T20:21:07.466 DEBUG:boto:path=/pklxkd-49/obj6 2023-05-31T20:21:07.467 DEBUG:boto:auth_path=/pklxkd-49/obj6 2023-05-31T20:21:07.467 DEBUG:boto:Method: GET 2023-05-31T20:21:07.467 DEBUG:boto:Path: /pklxkd-49/obj6 2023-05-31T20:21:07.468 DEBUG:boto:Data: 2023-05-31T20:21:07.468 DEBUG:boto:Headers: {} 2023-05-31T20:21:07.469 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:21:07.469 DEBUG:boto:Port: 8001 2023-05-31T20:21:07.469 DEBUG:boto:Params: {} 2023-05-31T20:21:07.470 DEBUG:boto:Token: None 2023-05-31T20:21:07.470 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:21:07 GMT /pklxkd-49/obj6 2023-05-31T20:21:07.471 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:1ok5W2bup7v7EK+ktaxCxc+7RIA= 2023-05-31T20:21:07.472 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:21:07 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:1ok5W2bup7v7EK+ktaxCxc+7RIA=', 'Content-Length': '0'} 2023-05-31T20:21:07.474 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:20:28 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000587445be114dc43f-006477ac33-162e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:21:07 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:21:07.475 DEBUG:boto:path=/pklxkd-49/obj6 2023-05-31T20:21:07.475 DEBUG:boto:auth_path=/pklxkd-49/obj6 2023-05-31T20:21:07.476 DEBUG:boto:Method: GET 2023-05-31T20:21:07.476 DEBUG:boto:Path: /pklxkd-49/obj6 2023-05-31T20:21:07.476 DEBUG:boto:Data: 2023-05-31T20:21:07.477 DEBUG:boto:Headers: {} 2023-05-31T20:21:07.477 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:21:07.478 DEBUG:boto:Port: 8000 2023-05-31T20:21:07.478 DEBUG:boto:Params: {} 2023-05-31T20:21:07.478 DEBUG:boto:Token: None 2023-05-31T20:21:07.479 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:21:07 GMT /pklxkd-49/obj6 2023-05-31T20:21:07.479 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:1ok5W2bup7v7EK+ktaxCxc+7RIA= 2023-05-31T20:21:07.480 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:21:07 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:1ok5W2bup7v7EK+ktaxCxc+7RIA=', 'Content-Length': '0'} 2023-05-31T20:21:07.481 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:20:28 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000ffb3e90a7a4594f4-006477ac33-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:21:07 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:21:07.482 DEBUG:boto:path=/pklxkd-49/ 2023-05-31T20:21:07.483 DEBUG:boto:auth_path=/pklxkd-49/ 2023-05-31T20:21:07.483 DEBUG:boto:path=/pklxkd-49/?prefix=obj6 2023-05-31T20:21:07.483 DEBUG:boto:auth_path=/pklxkd-49/?prefix=obj6 2023-05-31T20:21:07.484 DEBUG:boto:Method: GET 2023-05-31T20:21:07.484 DEBUG:boto:Path: /pklxkd-49/?prefix=obj6 2023-05-31T20:21:07.485 DEBUG:boto:Data: 2023-05-31T20:21:07.485 DEBUG:boto:Headers: {} 2023-05-31T20:21:07.486 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:21:07.486 DEBUG:boto:Port: 8001 2023-05-31T20:21:07.487 DEBUG:boto:Params: {} 2023-05-31T20:21:07.487 DEBUG:boto:Token: None 2023-05-31T20:21:07.488 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:21:07 GMT /pklxkd-49/ 2023-05-31T20:21:07.488 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:hg2FCb+7UP4DNsqWInfYUiAGm9c= 2023-05-31T20:21:07.489 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:21:07 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:hg2FCb+7UP4DNsqWInfYUiAGm9c=', 'Content-Length': '0'} 2023-05-31T20:21:07.491 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000198bb81c619a2997-006477ac33-162e-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:21:07 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:21:07.491 DEBUG:boto:b'pklxkd-49obj61000falseobj62023-05-31T20:20:28.814Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:21:07.492 DEBUG:boto:path=/pklxkd-49/ 2023-05-31T20:21:07.493 DEBUG:boto:auth_path=/pklxkd-49/ 2023-05-31T20:21:07.493 DEBUG:boto:path=/pklxkd-49/?prefix=obj6 2023-05-31T20:21:07.493 DEBUG:boto:auth_path=/pklxkd-49/?prefix=obj6 2023-05-31T20:21:07.494 DEBUG:boto:Method: GET 2023-05-31T20:21:07.494 DEBUG:boto:Path: /pklxkd-49/?prefix=obj6 2023-05-31T20:21:07.494 DEBUG:boto:Data: 2023-05-31T20:21:07.495 DEBUG:boto:Headers: {} 2023-05-31T20:21:07.495 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:21:07.495 DEBUG:boto:Port: 8000 2023-05-31T20:21:07.496 DEBUG:boto:Params: {} 2023-05-31T20:21:07.497 DEBUG:boto:Token: None 2023-05-31T20:21:07.497 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:21:07 GMT /pklxkd-49/ 2023-05-31T20:21:07.498 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:hg2FCb+7UP4DNsqWInfYUiAGm9c= 2023-05-31T20:21:07.498 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:21:07 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:hg2FCb+7UP4DNsqWInfYUiAGm9c=', 'Content-Length': '0'} 2023-05-31T20:21:07.501 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f77ec9baebe4d10e-006477ac33-1630-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:21:07 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:21:07.501 DEBUG:boto:b'pklxkd-49obj61000falseobj62023-05-31T20:20:28.814Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:21:07.502 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj7 2023-05-31T20:21:07.502 DEBUG:boto:path=/pklxkd-49/obj7 2023-05-31T20:21:07.503 DEBUG:boto:auth_path=/pklxkd-49/obj7 2023-05-31T20:21:07.503 DEBUG:boto:path=/pklxkd-49/obj7?versionId=null 2023-05-31T20:21:07.503 DEBUG:boto:auth_path=/pklxkd-49/obj7?versionId=null 2023-05-31T20:21:07.504 DEBUG:boto:Method: GET 2023-05-31T20:21:07.504 DEBUG:boto:Path: /pklxkd-49/obj7?versionId=null 2023-05-31T20:21:07.505 DEBUG:boto:Data: 2023-05-31T20:21:07.505 DEBUG:boto:Headers: {} 2023-05-31T20:21:07.506 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:21:07.506 DEBUG:boto:Port: 8001 2023-05-31T20:21:07.506 DEBUG:boto:Params: {} 2023-05-31T20:21:07.507 DEBUG:boto:Token: None 2023-05-31T20:21:07.507 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:21:07 GMT /pklxkd-49/obj7?versionId=null 2023-05-31T20:21:07.508 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:L9OGkjlDtTxzcJobqYPM0NQ2LCA= 2023-05-31T20:21:07.508 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:21:07 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:L9OGkjlDtTxzcJobqYPM0NQ2LCA=', 'Content-Length': '0'} 2023-05-31T20:21:07.510 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:20:28 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000063c0ac0c6dbd8e07-006477ac33-162e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:21:07 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:21:07.511 DEBUG:boto:path=/pklxkd-49/obj7 2023-05-31T20:21:07.511 DEBUG:boto:auth_path=/pklxkd-49/obj7 2023-05-31T20:21:07.512 DEBUG:boto:path=/pklxkd-49/obj7?versionId=null 2023-05-31T20:21:07.512 DEBUG:boto:auth_path=/pklxkd-49/obj7?versionId=null 2023-05-31T20:21:07.512 DEBUG:boto:Method: GET 2023-05-31T20:21:07.513 DEBUG:boto:Path: /pklxkd-49/obj7?versionId=null 2023-05-31T20:21:07.513 DEBUG:boto:Data: 2023-05-31T20:21:07.514 DEBUG:boto:Headers: {} 2023-05-31T20:21:07.514 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:21:07.515 DEBUG:boto:Port: 8000 2023-05-31T20:21:07.516 DEBUG:boto:Params: {} 2023-05-31T20:21:07.517 DEBUG:boto:Token: None 2023-05-31T20:21:07.517 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:21:07 GMT /pklxkd-49/obj7?versionId=null 2023-05-31T20:21:07.517 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:L9OGkjlDtTxzcJobqYPM0NQ2LCA= 2023-05-31T20:21:07.518 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:21:07 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:L9OGkjlDtTxzcJobqYPM0NQ2LCA=', 'Content-Length': '0'} 2023-05-31T20:21:07.520 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:20:28 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000009c40f53ee8488d07-006477ac33-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:21:07 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:21:07.521 DEBUG:boto:path=/pklxkd-49/obj7 2023-05-31T20:21:07.521 DEBUG:boto:auth_path=/pklxkd-49/obj7 2023-05-31T20:21:07.522 DEBUG:boto:path=/pklxkd-49/obj7?versionId=null 2023-05-31T20:21:07.522 DEBUG:boto:auth_path=/pklxkd-49/obj7?versionId=null 2023-05-31T20:21:07.522 DEBUG:boto:Method: HEAD 2023-05-31T20:21:07.523 DEBUG:boto:Path: /pklxkd-49/obj7?versionId=null 2023-05-31T20:21:07.523 DEBUG:boto:Data: 2023-05-31T20:21:07.523 DEBUG:boto:Headers: {} 2023-05-31T20:21:07.524 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:21:07.524 DEBUG:boto:Port: 8001 2023-05-31T20:21:07.524 DEBUG:boto:Params: {} 2023-05-31T20:21:07.525 DEBUG:boto:Token: None 2023-05-31T20:21:07.525 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:21:07 GMT /pklxkd-49/obj7?versionId=null 2023-05-31T20:21:07.526 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:GXSfEYm90rPY62mC3QblyblkQ5w= 2023-05-31T20:21:07.526 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:21:07 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:GXSfEYm90rPY62mC3QblyblkQ5w=', 'Content-Length': '0'} 2023-05-31T20:21:07.528 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:20:28 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000b8eb2dedcc7edbaa-006477ac33-162e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:21:07 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:21:07.528 DEBUG:boto:path=/pklxkd-49/obj7 2023-05-31T20:21:07.529 DEBUG:boto:auth_path=/pklxkd-49/obj7 2023-05-31T20:21:07.529 DEBUG:boto:path=/pklxkd-49/obj7?versionId=null 2023-05-31T20:21:07.529 DEBUG:boto:auth_path=/pklxkd-49/obj7?versionId=null 2023-05-31T20:21:07.530 DEBUG:boto:Method: HEAD 2023-05-31T20:21:07.530 DEBUG:boto:Path: /pklxkd-49/obj7?versionId=null 2023-05-31T20:21:07.530 DEBUG:boto:Data: 2023-05-31T20:21:07.531 DEBUG:boto:Headers: {} 2023-05-31T20:21:07.531 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:21:07.531 DEBUG:boto:Port: 8000 2023-05-31T20:21:07.532 DEBUG:boto:Params: {} 2023-05-31T20:21:07.532 DEBUG:boto:Token: None 2023-05-31T20:21:07.532 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:21:07 GMT /pklxkd-49/obj7?versionId=null 2023-05-31T20:21:07.533 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:GXSfEYm90rPY62mC3QblyblkQ5w= 2023-05-31T20:21:07.533 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:21:07 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:GXSfEYm90rPY62mC3QblyblkQ5w=', 'Content-Length': '0'} 2023-05-31T20:21:07.535 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:20:28 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000033dcdfe18469d42d-006477ac33-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:21:07 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:21:07.535 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj7 2023-05-31T20:21:07.536 DEBUG:boto:path=/pklxkd-49/obj7 2023-05-31T20:21:07.536 DEBUG:boto:auth_path=/pklxkd-49/obj7 2023-05-31T20:21:07.537 DEBUG:boto:path=/pklxkd-49/obj7?versionId=null 2023-05-31T20:21:07.537 DEBUG:boto:auth_path=/pklxkd-49/obj7?versionId=null 2023-05-31T20:21:07.537 DEBUG:boto:Method: GET 2023-05-31T20:21:07.538 DEBUG:boto:Path: /pklxkd-49/obj7?versionId=null 2023-05-31T20:21:07.538 DEBUG:boto:Data: 2023-05-31T20:21:07.538 DEBUG:boto:Headers: {} 2023-05-31T20:21:07.539 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:21:07.539 DEBUG:boto:Port: 8001 2023-05-31T20:21:07.540 DEBUG:boto:Params: {} 2023-05-31T20:21:07.540 DEBUG:boto:Token: None 2023-05-31T20:21:07.540 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:21:07 GMT /pklxkd-49/obj7?versionId=null 2023-05-31T20:21:07.541 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:L9OGkjlDtTxzcJobqYPM0NQ2LCA= 2023-05-31T20:21:07.541 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:21:07 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:L9OGkjlDtTxzcJobqYPM0NQ2LCA=', 'Content-Length': '0'} 2023-05-31T20:21:07.543 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:20:28 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000e99103b838d5fce6-006477ac33-162e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:21:07 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:21:07.544 DEBUG:boto:path=/pklxkd-49/obj7 2023-05-31T20:21:07.545 DEBUG:boto:auth_path=/pklxkd-49/obj7 2023-05-31T20:21:07.545 DEBUG:boto:path=/pklxkd-49/obj7?versionId=null 2023-05-31T20:21:07.545 DEBUG:boto:auth_path=/pklxkd-49/obj7?versionId=null 2023-05-31T20:21:07.546 DEBUG:boto:Method: GET 2023-05-31T20:21:07.546 DEBUG:boto:Path: /pklxkd-49/obj7?versionId=null 2023-05-31T20:21:07.546 DEBUG:boto:Data: 2023-05-31T20:21:07.547 DEBUG:boto:Headers: {} 2023-05-31T20:21:07.547 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:21:07.548 DEBUG:boto:Port: 8000 2023-05-31T20:21:07.548 DEBUG:boto:Params: {} 2023-05-31T20:21:07.549 DEBUG:boto:Token: None 2023-05-31T20:21:07.549 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:21:07 GMT /pklxkd-49/obj7?versionId=null 2023-05-31T20:21:07.549 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:L9OGkjlDtTxzcJobqYPM0NQ2LCA= 2023-05-31T20:21:07.550 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:21:07 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:L9OGkjlDtTxzcJobqYPM0NQ2LCA=', 'Content-Length': '0'} 2023-05-31T20:21:07.552 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:20:28 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000003a7ae3f4ca2d031a-006477ac33-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:21:07 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:21:07.553 DEBUG:boto:path=/pklxkd-49/ 2023-05-31T20:21:07.553 DEBUG:boto:auth_path=/pklxkd-49/ 2023-05-31T20:21:07.553 DEBUG:boto:path=/pklxkd-49/?prefix=obj7 2023-05-31T20:21:07.554 DEBUG:boto:auth_path=/pklxkd-49/?prefix=obj7 2023-05-31T20:21:07.554 DEBUG:boto:Method: GET 2023-05-31T20:21:07.555 DEBUG:boto:Path: /pklxkd-49/?prefix=obj7 2023-05-31T20:21:07.555 DEBUG:boto:Data: 2023-05-31T20:21:07.555 DEBUG:boto:Headers: {} 2023-05-31T20:21:07.556 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:21:07.556 DEBUG:boto:Port: 8001 2023-05-31T20:21:07.557 DEBUG:boto:Params: {} 2023-05-31T20:21:07.557 DEBUG:boto:Token: None 2023-05-31T20:21:07.558 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:21:07 GMT /pklxkd-49/ 2023-05-31T20:21:07.558 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:hg2FCb+7UP4DNsqWInfYUiAGm9c= 2023-05-31T20:21:07.559 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:21:07 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:hg2FCb+7UP4DNsqWInfYUiAGm9c=', 'Content-Length': '0'} 2023-05-31T20:21:07.561 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000063f0957f800d241c-006477ac33-162e-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:21:07 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:21:07.561 DEBUG:boto:b'pklxkd-49obj71000falseobj72023-05-31T20:20:28.830Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:21:07.562 DEBUG:boto:path=/pklxkd-49/ 2023-05-31T20:21:07.562 DEBUG:boto:auth_path=/pklxkd-49/ 2023-05-31T20:21:07.563 DEBUG:boto:path=/pklxkd-49/?prefix=obj7 2023-05-31T20:21:07.563 DEBUG:boto:auth_path=/pklxkd-49/?prefix=obj7 2023-05-31T20:21:07.563 DEBUG:boto:Method: GET 2023-05-31T20:21:07.564 DEBUG:boto:Path: /pklxkd-49/?prefix=obj7 2023-05-31T20:21:07.564 DEBUG:boto:Data: 2023-05-31T20:21:07.565 DEBUG:boto:Headers: {} 2023-05-31T20:21:07.565 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:21:07.565 DEBUG:boto:Port: 8000 2023-05-31T20:21:07.566 DEBUG:boto:Params: {} 2023-05-31T20:21:07.566 DEBUG:boto:Token: None 2023-05-31T20:21:07.567 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:21:07 GMT /pklxkd-49/ 2023-05-31T20:21:07.567 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:hg2FCb+7UP4DNsqWInfYUiAGm9c= 2023-05-31T20:21:07.568 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:21:07 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:hg2FCb+7UP4DNsqWInfYUiAGm9c=', 'Content-Length': '0'} 2023-05-31T20:21:07.570 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000090a199c480686032-006477ac33-1630-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:21:07 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:21:07.570 DEBUG:boto:b'pklxkd-49obj71000falseobj72023-05-31T20:20:28.830Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:21:07.571 DEBUG:boto:path=/pklxkd-49/obj7 2023-05-31T20:21:07.572 DEBUG:boto:auth_path=/pklxkd-49/obj7 2023-05-31T20:21:07.572 DEBUG:boto:Method: HEAD 2023-05-31T20:21:07.572 DEBUG:boto:Path: /pklxkd-49/obj7 2023-05-31T20:21:07.573 DEBUG:boto:Data: 2023-05-31T20:21:07.573 DEBUG:boto:Headers: {} 2023-05-31T20:21:07.573 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:21:07.574 DEBUG:boto:Port: 8001 2023-05-31T20:21:07.574 DEBUG:boto:Params: {} 2023-05-31T20:21:07.575 DEBUG:boto:Token: None 2023-05-31T20:21:07.575 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:21:07 GMT /pklxkd-49/obj7 2023-05-31T20:21:07.576 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:afI13GXAKA13ZPlXW40c/BVHJww= 2023-05-31T20:21:07.576 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:21:07 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:afI13GXAKA13ZPlXW40c/BVHJww=', 'Content-Length': '0'} 2023-05-31T20:21:07.577 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:20:28 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000dc75a7406e9bccc4-006477ac33-162e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:21:07 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:21:07.578 DEBUG:boto:path=/pklxkd-49/obj7 2023-05-31T20:21:07.579 DEBUG:boto:auth_path=/pklxkd-49/obj7 2023-05-31T20:21:07.580 DEBUG:boto:Method: HEAD 2023-05-31T20:21:07.580 DEBUG:boto:Path: /pklxkd-49/obj7 2023-05-31T20:21:07.581 DEBUG:boto:Data: 2023-05-31T20:21:07.581 DEBUG:boto:Headers: {} 2023-05-31T20:21:07.582 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:21:07.582 DEBUG:boto:Port: 8000 2023-05-31T20:21:07.582 DEBUG:boto:Params: {} 2023-05-31T20:21:07.583 DEBUG:boto:Token: None 2023-05-31T20:21:07.583 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:21:07 GMT /pklxkd-49/obj7 2023-05-31T20:21:07.584 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:afI13GXAKA13ZPlXW40c/BVHJww= 2023-05-31T20:21:07.584 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:21:07 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:afI13GXAKA13ZPlXW40c/BVHJww=', 'Content-Length': '0'} 2023-05-31T20:21:07.586 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:20:28 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000017d71486f3ad5e20-006477ac33-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:21:07 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:21:07.587 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj7 2023-05-31T20:21:07.587 DEBUG:boto:path=/pklxkd-49/obj7 2023-05-31T20:21:07.587 DEBUG:boto:auth_path=/pklxkd-49/obj7 2023-05-31T20:21:07.588 DEBUG:boto:Method: GET 2023-05-31T20:21:07.588 DEBUG:boto:Path: /pklxkd-49/obj7 2023-05-31T20:21:07.589 DEBUG:boto:Data: 2023-05-31T20:21:07.589 DEBUG:boto:Headers: {} 2023-05-31T20:21:07.589 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:21:07.590 DEBUG:boto:Port: 8001 2023-05-31T20:21:07.590 DEBUG:boto:Params: {} 2023-05-31T20:21:07.591 DEBUG:boto:Token: None 2023-05-31T20:21:07.591 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:21:07 GMT /pklxkd-49/obj7 2023-05-31T20:21:07.592 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:uFirC0JuQL0tOeSCVR/ZcXgpi7Y= 2023-05-31T20:21:07.592 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:21:07 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:uFirC0JuQL0tOeSCVR/ZcXgpi7Y=', 'Content-Length': '0'} 2023-05-31T20:21:07.594 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:20:28 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000ebc68ec1c2760c69-006477ac33-162e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:21:07 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:21:07.595 DEBUG:boto:path=/pklxkd-49/obj7 2023-05-31T20:21:07.596 DEBUG:boto:auth_path=/pklxkd-49/obj7 2023-05-31T20:21:07.596 DEBUG:boto:Method: GET 2023-05-31T20:21:07.596 DEBUG:boto:Path: /pklxkd-49/obj7 2023-05-31T20:21:07.597 DEBUG:boto:Data: 2023-05-31T20:21:07.597 DEBUG:boto:Headers: {} 2023-05-31T20:21:07.597 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:21:07.598 DEBUG:boto:Port: 8000 2023-05-31T20:21:07.598 DEBUG:boto:Params: {} 2023-05-31T20:21:07.599 DEBUG:boto:Token: None 2023-05-31T20:21:07.599 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:21:07 GMT /pklxkd-49/obj7 2023-05-31T20:21:07.600 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:uFirC0JuQL0tOeSCVR/ZcXgpi7Y= 2023-05-31T20:21:07.600 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:21:07 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:uFirC0JuQL0tOeSCVR/ZcXgpi7Y=', 'Content-Length': '0'} 2023-05-31T20:21:07.608 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:20:28 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000001aef7750534aa32-006477ac33-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:21:07 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:21:07.609 DEBUG:boto:path=/pklxkd-49/ 2023-05-31T20:21:07.610 DEBUG:boto:auth_path=/pklxkd-49/ 2023-05-31T20:21:07.610 DEBUG:boto:path=/pklxkd-49/?prefix=obj7 2023-05-31T20:21:07.611 DEBUG:boto:auth_path=/pklxkd-49/?prefix=obj7 2023-05-31T20:21:07.612 DEBUG:boto:Method: GET 2023-05-31T20:21:07.613 DEBUG:boto:Path: /pklxkd-49/?prefix=obj7 2023-05-31T20:21:07.613 DEBUG:boto:Data: 2023-05-31T20:21:07.614 DEBUG:boto:Headers: {} 2023-05-31T20:21:07.615 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:21:07.616 DEBUG:boto:Port: 8001 2023-05-31T20:21:07.617 DEBUG:boto:Params: {} 2023-05-31T20:21:07.619 DEBUG:boto:Token: None 2023-05-31T20:21:07.620 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:21:07 GMT /pklxkd-49/ 2023-05-31T20:21:07.620 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:hg2FCb+7UP4DNsqWInfYUiAGm9c= 2023-05-31T20:21:07.621 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:21:07 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:hg2FCb+7UP4DNsqWInfYUiAGm9c=', 'Content-Length': '0'} 2023-05-31T20:21:07.623 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000059d896294d740740-006477ac33-162e-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:21:07 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:21:07.624 DEBUG:boto:b'pklxkd-49obj71000falseobj72023-05-31T20:20:28.830Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:21:07.625 DEBUG:boto:path=/pklxkd-49/ 2023-05-31T20:21:07.625 DEBUG:boto:auth_path=/pklxkd-49/ 2023-05-31T20:21:07.625 DEBUG:boto:path=/pklxkd-49/?prefix=obj7 2023-05-31T20:21:07.626 DEBUG:boto:auth_path=/pklxkd-49/?prefix=obj7 2023-05-31T20:21:07.626 DEBUG:boto:Method: GET 2023-05-31T20:21:07.627 DEBUG:boto:Path: /pklxkd-49/?prefix=obj7 2023-05-31T20:21:07.627 DEBUG:boto:Data: 2023-05-31T20:21:07.627 DEBUG:boto:Headers: {} 2023-05-31T20:21:07.628 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:21:07.628 DEBUG:boto:Port: 8000 2023-05-31T20:21:07.629 DEBUG:boto:Params: {} 2023-05-31T20:21:07.629 DEBUG:boto:Token: None 2023-05-31T20:21:07.630 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:21:07 GMT /pklxkd-49/ 2023-05-31T20:21:07.631 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:hg2FCb+7UP4DNsqWInfYUiAGm9c= 2023-05-31T20:21:07.632 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:21:07 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:hg2FCb+7UP4DNsqWInfYUiAGm9c=', 'Content-Length': '0'} 2023-05-31T20:21:07.635 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000000ed50f3fd064795d-006477ac33-1630-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:21:07 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:21:07.676 DEBUG:boto:b'pklxkd-49obj71000falseobj72023-05-31T20:20:28.830Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:21:07.677 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj8 2023-05-31T20:21:07.678 DEBUG:boto:path=/pklxkd-49/obj8 2023-05-31T20:21:07.678 DEBUG:boto:auth_path=/pklxkd-49/obj8 2023-05-31T20:21:07.678 DEBUG:boto:path=/pklxkd-49/obj8?versionId=null 2023-05-31T20:21:07.679 DEBUG:boto:auth_path=/pklxkd-49/obj8?versionId=null 2023-05-31T20:21:07.680 DEBUG:boto:Method: GET 2023-05-31T20:21:07.680 DEBUG:boto:Path: /pklxkd-49/obj8?versionId=null 2023-05-31T20:21:07.680 DEBUG:boto:Data: 2023-05-31T20:21:07.681 DEBUG:boto:Headers: {} 2023-05-31T20:21:07.681 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:21:07.682 DEBUG:boto:Port: 8001 2023-05-31T20:21:07.682 DEBUG:boto:Params: {} 2023-05-31T20:21:07.683 DEBUG:boto:Token: None 2023-05-31T20:21:07.684 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:21:07 GMT /pklxkd-49/obj8?versionId=null 2023-05-31T20:21:07.685 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:U2Br/LyQavd48wlKROQNmnncF6E= 2023-05-31T20:21:07.686 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:21:07 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:U2Br/LyQavd48wlKROQNmnncF6E=', 'Content-Length': '0'} 2023-05-31T20:21:07.688 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:20:28 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000081412181b0fccdf-006477ac33-162e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:21:07 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:21:07.689 DEBUG:boto:path=/pklxkd-49/obj8 2023-05-31T20:21:07.690 DEBUG:boto:auth_path=/pklxkd-49/obj8 2023-05-31T20:21:07.690 DEBUG:boto:path=/pklxkd-49/obj8?versionId=null 2023-05-31T20:21:07.691 DEBUG:boto:auth_path=/pklxkd-49/obj8?versionId=null 2023-05-31T20:21:07.692 DEBUG:boto:Method: GET 2023-05-31T20:21:07.692 DEBUG:boto:Path: /pklxkd-49/obj8?versionId=null 2023-05-31T20:21:07.693 DEBUG:boto:Data: 2023-05-31T20:21:07.694 DEBUG:boto:Headers: {} 2023-05-31T20:21:07.695 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:21:07.695 DEBUG:boto:Port: 8000 2023-05-31T20:21:07.696 DEBUG:boto:Params: {} 2023-05-31T20:21:07.697 DEBUG:boto:Token: None 2023-05-31T20:21:07.698 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:21:07 GMT /pklxkd-49/obj8?versionId=null 2023-05-31T20:21:07.699 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:U2Br/LyQavd48wlKROQNmnncF6E= 2023-05-31T20:21:07.700 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:21:07 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:U2Br/LyQavd48wlKROQNmnncF6E=', 'Content-Length': '0'} 2023-05-31T20:21:07.703 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:20:28 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000f81135c40380a73e-006477ac33-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:21:07 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:21:07.704 DEBUG:boto:path=/pklxkd-49/obj8 2023-05-31T20:21:07.705 DEBUG:boto:auth_path=/pklxkd-49/obj8 2023-05-31T20:21:07.705 DEBUG:boto:path=/pklxkd-49/obj8?versionId=null 2023-05-31T20:21:07.705 DEBUG:boto:auth_path=/pklxkd-49/obj8?versionId=null 2023-05-31T20:21:07.706 DEBUG:boto:Method: HEAD 2023-05-31T20:21:07.706 DEBUG:boto:Path: /pklxkd-49/obj8?versionId=null 2023-05-31T20:21:07.707 DEBUG:boto:Data: 2023-05-31T20:21:07.708 DEBUG:boto:Headers: {} 2023-05-31T20:21:07.709 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:21:07.709 DEBUG:boto:Port: 8001 2023-05-31T20:21:07.710 DEBUG:boto:Params: {} 2023-05-31T20:21:07.710 DEBUG:boto:Token: None 2023-05-31T20:21:07.711 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:21:07 GMT /pklxkd-49/obj8?versionId=null 2023-05-31T20:21:07.712 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:jdRnUpxa9fWVFv7biSRhY48euXc= 2023-05-31T20:21:07.712 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:21:07 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:jdRnUpxa9fWVFv7biSRhY48euXc=', 'Content-Length': '0'} 2023-05-31T20:21:07.714 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:20:28 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000f692b55e593b97df-006477ac33-162e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:21:07 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:21:07.714 DEBUG:boto:path=/pklxkd-49/obj8 2023-05-31T20:21:07.715 DEBUG:boto:auth_path=/pklxkd-49/obj8 2023-05-31T20:21:07.715 DEBUG:boto:path=/pklxkd-49/obj8?versionId=null 2023-05-31T20:21:07.715 DEBUG:boto:auth_path=/pklxkd-49/obj8?versionId=null 2023-05-31T20:21:07.716 DEBUG:boto:Method: HEAD 2023-05-31T20:21:07.716 DEBUG:boto:Path: /pklxkd-49/obj8?versionId=null 2023-05-31T20:21:07.717 DEBUG:boto:Data: 2023-05-31T20:21:07.717 DEBUG:boto:Headers: {} 2023-05-31T20:21:07.718 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:21:07.718 DEBUG:boto:Port: 8000 2023-05-31T20:21:07.718 DEBUG:boto:Params: {} 2023-05-31T20:21:07.719 DEBUG:boto:Token: None 2023-05-31T20:21:07.719 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:21:07 GMT /pklxkd-49/obj8?versionId=null 2023-05-31T20:21:07.720 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:jdRnUpxa9fWVFv7biSRhY48euXc= 2023-05-31T20:21:07.720 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:21:07 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:jdRnUpxa9fWVFv7biSRhY48euXc=', 'Content-Length': '0'} 2023-05-31T20:21:07.722 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:20:28 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000a660dc54a8c1a5c6-006477ac33-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:21:07 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:21:07.723 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj8 2023-05-31T20:21:07.723 DEBUG:boto:path=/pklxkd-49/obj8 2023-05-31T20:21:07.724 DEBUG:boto:auth_path=/pklxkd-49/obj8 2023-05-31T20:21:07.725 DEBUG:boto:path=/pklxkd-49/obj8?versionId=null 2023-05-31T20:21:07.725 DEBUG:boto:auth_path=/pklxkd-49/obj8?versionId=null 2023-05-31T20:21:07.726 DEBUG:boto:Method: GET 2023-05-31T20:21:07.726 DEBUG:boto:Path: /pklxkd-49/obj8?versionId=null 2023-05-31T20:21:07.727 DEBUG:boto:Data: 2023-05-31T20:21:07.728 DEBUG:boto:Headers: {} 2023-05-31T20:21:07.728 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:21:07.729 DEBUG:boto:Port: 8001 2023-05-31T20:21:07.729 DEBUG:boto:Params: {} 2023-05-31T20:21:07.729 DEBUG:boto:Token: None 2023-05-31T20:21:07.730 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:21:07 GMT /pklxkd-49/obj8?versionId=null 2023-05-31T20:21:07.731 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:U2Br/LyQavd48wlKROQNmnncF6E= 2023-05-31T20:21:07.731 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:21:07 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:U2Br/LyQavd48wlKROQNmnncF6E=', 'Content-Length': '0'} 2023-05-31T20:21:07.733 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:20:28 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000038c9d98d34ad08f3-006477ac33-162e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:21:07 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:21:07.735 DEBUG:boto:path=/pklxkd-49/obj8 2023-05-31T20:21:07.735 DEBUG:boto:auth_path=/pklxkd-49/obj8 2023-05-31T20:21:07.736 DEBUG:boto:path=/pklxkd-49/obj8?versionId=null 2023-05-31T20:21:07.737 DEBUG:boto:auth_path=/pklxkd-49/obj8?versionId=null 2023-05-31T20:21:07.738 DEBUG:boto:Method: GET 2023-05-31T20:21:07.739 DEBUG:boto:Path: /pklxkd-49/obj8?versionId=null 2023-05-31T20:21:07.739 DEBUG:boto:Data: 2023-05-31T20:21:07.740 DEBUG:boto:Headers: {} 2023-05-31T20:21:07.741 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:21:07.742 DEBUG:boto:Port: 8000 2023-05-31T20:21:07.743 DEBUG:boto:Params: {} 2023-05-31T20:21:07.744 DEBUG:boto:Token: None 2023-05-31T20:21:07.745 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:21:07 GMT /pklxkd-49/obj8?versionId=null 2023-05-31T20:21:07.746 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:U2Br/LyQavd48wlKROQNmnncF6E= 2023-05-31T20:21:07.747 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:21:07 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:U2Br/LyQavd48wlKROQNmnncF6E=', 'Content-Length': '0'} 2023-05-31T20:21:07.749 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:20:28 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000002ff3f198b1259410-006477ac33-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:21:07 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:21:07.750 DEBUG:boto:path=/pklxkd-49/ 2023-05-31T20:21:07.750 DEBUG:boto:auth_path=/pklxkd-49/ 2023-05-31T20:21:07.751 DEBUG:boto:path=/pklxkd-49/?prefix=obj8 2023-05-31T20:21:07.751 DEBUG:boto:auth_path=/pklxkd-49/?prefix=obj8 2023-05-31T20:21:07.751 DEBUG:boto:Method: GET 2023-05-31T20:21:07.752 DEBUG:boto:Path: /pklxkd-49/?prefix=obj8 2023-05-31T20:21:07.752 DEBUG:boto:Data: 2023-05-31T20:21:07.752 DEBUG:boto:Headers: {} 2023-05-31T20:21:07.753 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:21:07.753 DEBUG:boto:Port: 8001 2023-05-31T20:21:07.754 DEBUG:boto:Params: {} 2023-05-31T20:21:07.754 DEBUG:boto:Token: None 2023-05-31T20:21:07.754 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:21:07 GMT /pklxkd-49/ 2023-05-31T20:21:07.755 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:hg2FCb+7UP4DNsqWInfYUiAGm9c= 2023-05-31T20:21:07.755 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:21:07 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:hg2FCb+7UP4DNsqWInfYUiAGm9c=', 'Content-Length': '0'} 2023-05-31T20:21:07.757 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000005a5d1537c0c52d79-006477ac33-162e-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:21:07 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:21:07.804 DEBUG:boto:b'pklxkd-49obj81000falseobj82023-05-31T20:20:28.847Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:21:07.805 DEBUG:boto:path=/pklxkd-49/ 2023-05-31T20:21:07.806 DEBUG:boto:auth_path=/pklxkd-49/ 2023-05-31T20:21:07.806 DEBUG:boto:path=/pklxkd-49/?prefix=obj8 2023-05-31T20:21:07.806 DEBUG:boto:auth_path=/pklxkd-49/?prefix=obj8 2023-05-31T20:21:07.807 DEBUG:boto:Method: GET 2023-05-31T20:21:07.807 DEBUG:boto:Path: /pklxkd-49/?prefix=obj8 2023-05-31T20:21:07.808 DEBUG:boto:Data: 2023-05-31T20:21:07.808 DEBUG:boto:Headers: {} 2023-05-31T20:21:07.809 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:21:07.810 DEBUG:boto:Port: 8000 2023-05-31T20:21:07.811 DEBUG:boto:Params: {} 2023-05-31T20:21:07.811 DEBUG:boto:Token: None 2023-05-31T20:21:07.812 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:21:07 GMT /pklxkd-49/ 2023-05-31T20:21:07.812 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:hg2FCb+7UP4DNsqWInfYUiAGm9c= 2023-05-31T20:21:07.813 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:21:07 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:hg2FCb+7UP4DNsqWInfYUiAGm9c=', 'Content-Length': '0'} 2023-05-31T20:21:07.815 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000fb53d33e0b579dda-006477ac33-1630-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:21:07 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:21:07.815 DEBUG:boto:b'pklxkd-49obj81000falseobj82023-05-31T20:20:28.847Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:21:07.816 DEBUG:boto:path=/pklxkd-49/obj8 2023-05-31T20:21:07.816 DEBUG:boto:auth_path=/pklxkd-49/obj8 2023-05-31T20:21:07.817 DEBUG:boto:Method: HEAD 2023-05-31T20:21:07.817 DEBUG:boto:Path: /pklxkd-49/obj8 2023-05-31T20:21:07.817 DEBUG:boto:Data: 2023-05-31T20:21:07.818 DEBUG:boto:Headers: {} 2023-05-31T20:21:07.818 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:21:07.818 DEBUG:boto:Port: 8001 2023-05-31T20:21:07.819 DEBUG:boto:Params: {} 2023-05-31T20:21:07.819 DEBUG:boto:Token: None 2023-05-31T20:21:07.819 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:21:07 GMT /pklxkd-49/obj8 2023-05-31T20:21:07.820 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:pxlKlDiJHVNGCCNh5xfvRsqcJxc= 2023-05-31T20:21:07.820 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:21:07 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:pxlKlDiJHVNGCCNh5xfvRsqcJxc=', 'Content-Length': '0'} 2023-05-31T20:21:07.822 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:20:28 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000620d1a238c2b6594-006477ac33-162e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:21:07 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:21:07.823 DEBUG:boto:path=/pklxkd-49/obj8 2023-05-31T20:21:07.823 DEBUG:boto:auth_path=/pklxkd-49/obj8 2023-05-31T20:21:07.823 DEBUG:boto:Method: HEAD 2023-05-31T20:21:07.824 DEBUG:boto:Path: /pklxkd-49/obj8 2023-05-31T20:21:07.824 DEBUG:boto:Data: 2023-05-31T20:21:07.825 DEBUG:boto:Headers: {} 2023-05-31T20:21:07.825 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:21:07.825 DEBUG:boto:Port: 8000 2023-05-31T20:21:07.826 DEBUG:boto:Params: {} 2023-05-31T20:21:07.826 DEBUG:boto:Token: None 2023-05-31T20:21:07.828 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:21:07 GMT /pklxkd-49/obj8 2023-05-31T20:21:07.829 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:pxlKlDiJHVNGCCNh5xfvRsqcJxc= 2023-05-31T20:21:07.831 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:21:07 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:pxlKlDiJHVNGCCNh5xfvRsqcJxc=', 'Content-Length': '0'} 2023-05-31T20:21:07.834 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:20:28 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000b05a4ca8af572449-006477ac33-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:21:07 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:21:07.835 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj8 2023-05-31T20:21:07.835 DEBUG:boto:path=/pklxkd-49/obj8 2023-05-31T20:21:07.836 DEBUG:boto:auth_path=/pklxkd-49/obj8 2023-05-31T20:21:07.836 DEBUG:boto:Method: GET 2023-05-31T20:21:07.837 DEBUG:boto:Path: /pklxkd-49/obj8 2023-05-31T20:21:07.837 DEBUG:boto:Data: 2023-05-31T20:21:07.837 DEBUG:boto:Headers: {} 2023-05-31T20:21:07.838 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:21:07.838 DEBUG:boto:Port: 8001 2023-05-31T20:21:07.839 DEBUG:boto:Params: {} 2023-05-31T20:21:07.840 DEBUG:boto:Token: None 2023-05-31T20:21:07.841 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:21:07 GMT /pklxkd-49/obj8 2023-05-31T20:21:07.842 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:1G/FnUX78UVN7m+rHFlTh8T9r0E= 2023-05-31T20:21:07.843 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:21:07 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:1G/FnUX78UVN7m+rHFlTh8T9r0E=', 'Content-Length': '0'} 2023-05-31T20:21:07.845 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:20:28 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000e7a745f30f2950fe-006477ac33-162e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:21:07 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:21:07.845 DEBUG:boto:path=/pklxkd-49/obj8 2023-05-31T20:21:07.846 DEBUG:boto:auth_path=/pklxkd-49/obj8 2023-05-31T20:21:07.847 DEBUG:boto:Method: GET 2023-05-31T20:21:07.847 DEBUG:boto:Path: /pklxkd-49/obj8 2023-05-31T20:21:07.847 DEBUG:boto:Data: 2023-05-31T20:21:07.848 DEBUG:boto:Headers: {} 2023-05-31T20:21:07.848 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:21:07.849 DEBUG:boto:Port: 8000 2023-05-31T20:21:07.849 DEBUG:boto:Params: {} 2023-05-31T20:21:07.849 DEBUG:boto:Token: None 2023-05-31T20:21:07.850 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:21:07 GMT /pklxkd-49/obj8 2023-05-31T20:21:07.850 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:1G/FnUX78UVN7m+rHFlTh8T9r0E= 2023-05-31T20:21:07.851 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:21:07 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:1G/FnUX78UVN7m+rHFlTh8T9r0E=', 'Content-Length': '0'} 2023-05-31T20:21:07.852 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:20:28 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000009799331737703256-006477ac33-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:21:07 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:21:07.853 DEBUG:boto:path=/pklxkd-49/ 2023-05-31T20:21:07.854 DEBUG:boto:auth_path=/pklxkd-49/ 2023-05-31T20:21:07.855 DEBUG:boto:path=/pklxkd-49/?prefix=obj8 2023-05-31T20:21:07.856 DEBUG:boto:auth_path=/pklxkd-49/?prefix=obj8 2023-05-31T20:21:07.857 DEBUG:boto:Method: GET 2023-05-31T20:21:07.858 DEBUG:boto:Path: /pklxkd-49/?prefix=obj8 2023-05-31T20:21:07.858 DEBUG:boto:Data: 2023-05-31T20:21:07.859 DEBUG:boto:Headers: {} 2023-05-31T20:21:07.859 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:21:07.859 DEBUG:boto:Port: 8001 2023-05-31T20:21:07.860 DEBUG:boto:Params: {} 2023-05-31T20:21:07.860 DEBUG:boto:Token: None 2023-05-31T20:21:07.861 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:21:07 GMT /pklxkd-49/ 2023-05-31T20:21:07.861 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:hg2FCb+7UP4DNsqWInfYUiAGm9c= 2023-05-31T20:21:07.862 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:21:07 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:hg2FCb+7UP4DNsqWInfYUiAGm9c=', 'Content-Length': '0'} 2023-05-31T20:21:07.864 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000006b2f4241aa92619e-006477ac33-162e-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:21:07 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:21:07.865 DEBUG:boto:b'pklxkd-49obj81000falseobj82023-05-31T20:20:28.847Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:21:07.865 DEBUG:boto:path=/pklxkd-49/ 2023-05-31T20:21:07.866 DEBUG:boto:auth_path=/pklxkd-49/ 2023-05-31T20:21:07.866 DEBUG:boto:path=/pklxkd-49/?prefix=obj8 2023-05-31T20:21:07.866 DEBUG:boto:auth_path=/pklxkd-49/?prefix=obj8 2023-05-31T20:21:07.867 DEBUG:boto:Method: GET 2023-05-31T20:21:07.867 DEBUG:boto:Path: /pklxkd-49/?prefix=obj8 2023-05-31T20:21:07.868 DEBUG:boto:Data: 2023-05-31T20:21:07.868 DEBUG:boto:Headers: {} 2023-05-31T20:21:07.868 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:21:07.869 DEBUG:boto:Port: 8000 2023-05-31T20:21:07.869 DEBUG:boto:Params: {} 2023-05-31T20:21:07.869 DEBUG:boto:Token: None 2023-05-31T20:21:07.870 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:21:07 GMT /pklxkd-49/ 2023-05-31T20:21:07.870 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:hg2FCb+7UP4DNsqWInfYUiAGm9c= 2023-05-31T20:21:07.871 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:21:07 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:hg2FCb+7UP4DNsqWInfYUiAGm9c=', 'Content-Length': '0'} 2023-05-31T20:21:07.873 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000066a3bd6a7543f1cf-006477ac33-1630-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:21:07 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:21:07.873 DEBUG:boto:b'pklxkd-49obj81000falseobj82023-05-31T20:20:28.847Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:21:07.874 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=pklxkd-49 zones={a2, a1} 2023-05-31T20:21:07.874 DEBUG:rgw_multi.tests:data checkpoint: source=a1 target=a2 2023-05-31T20:21:07.875 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['datalog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-31T20:21:07.875 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'datalog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-31T20:21:07.876 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 datalog status --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-05-31T20:21:07.925 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2023-05-31T20:21:07.925 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2023-05-31T20:21:07.995 INFO:teuthology.orchestra.run.smithi099.stdout:[ 2023-05-31T20:21:07.996 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T20:21:07.996 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "00000000000000000000:00000000000000005731", 2023-05-31T20:21:07.996 INFO:teuthology.orchestra.run.smithi099.stdout: "last_update": "2023-05-31T20:20:59.915684Z" 2023-05-31T20:21:07.996 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:21:07.997 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T20:21:07.997 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "00000000000000000000:00000000000000003755", 2023-05-31T20:21:07.997 INFO:teuthology.orchestra.run.smithi099.stdout: "last_update": "2023-05-31T20:20:59.916728Z" 2023-05-31T20:21:07.997 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:21:07.997 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T20:21:07.998 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "00000000000000000000:00000000000000009951", 2023-05-31T20:21:07.998 INFO:teuthology.orchestra.run.smithi099.stdout: "last_update": "2023-05-31T20:20:59.918574Z" 2023-05-31T20:21:07.998 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:21:07.998 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T20:21:07.998 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "00000000000000000000:00000000000000006709", 2023-05-31T20:21:07.998 INFO:teuthology.orchestra.run.smithi099.stdout: "last_update": "2023-05-31T20:20:29.199765Z" 2023-05-31T20:21:07.999 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T20:21:07.999 INFO:teuthology.orchestra.run.smithi099.stdout:] 2023-05-31T20:21:08.000 DEBUG:rgw_multi.tests:data markers for zone=a1 markers={0: '00000000000000000000:00000000000000005731', 1: '00000000000000000000:00000000000000003755', 2: '00000000000000000000:00000000000000009951', 3: '00000000000000000000:00000000000000006709'} 2023-05-31T20:21:08.000 INFO:rgw_multi.tests:starting data checkpoint for target_zone=a2 source_zone=a1 2023-05-31T20:21:08.001 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-31T20:21:08.001 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-31T20:21:08.002 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-05-31T20:21:08.054 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2023-05-31T20:21:08.055 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2023-05-31T20:21:08.106 INFO:teuthology.orchestra.run.smithi099.stdout:{ 2023-05-31T20:21:08.107 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_status": { 2023-05-31T20:21:08.107 INFO:teuthology.orchestra.run.smithi099.stdout: "info": { 2023-05-31T20:21:08.107 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "sync", 2023-05-31T20:21:08.107 INFO:teuthology.orchestra.run.smithi099.stdout: "num_shards": 4, 2023-05-31T20:21:08.107 INFO:teuthology.orchestra.run.smithi099.stdout: "instance_id": 949931097149414600 2023-05-31T20:21:08.108 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:21:08.108 INFO:teuthology.orchestra.run.smithi099.stdout: "markers": [ 2023-05-31T20:21:08.108 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T20:21:08.108 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 0, 2023-05-31T20:21:08.108 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T20:21:08.108 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "incremental-sync", 2023-05-31T20:21:08.109 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "00000000000000000000:00000000000000005731", 2023-05-31T20:21:08.109 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T20:21:08.109 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2023-05-31T20:21:08.109 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T20:21:08.109 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2023-05-31T20:20:59.915684Z" 2023-05-31T20:21:08.110 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T20:21:08.110 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:21:08.110 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T20:21:08.110 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 1, 2023-05-31T20:21:08.110 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T20:21:08.110 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "incremental-sync", 2023-05-31T20:21:08.111 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "00000000000000000000:00000000000000003755", 2023-05-31T20:21:08.111 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T20:21:08.111 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2023-05-31T20:21:08.111 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T20:21:08.111 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2023-05-31T20:20:59.916728Z" 2023-05-31T20:21:08.111 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T20:21:08.112 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:21:08.112 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T20:21:08.112 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 2, 2023-05-31T20:21:08.112 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T20:21:08.112 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "incremental-sync", 2023-05-31T20:21:08.112 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "00000000000000000000:00000000000000009951", 2023-05-31T20:21:08.112 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T20:21:08.113 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2023-05-31T20:21:08.113 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T20:21:08.113 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2023-05-31T20:20:59.918574Z" 2023-05-31T20:21:08.113 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T20:21:08.113 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:21:08.113 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T20:21:08.114 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 3, 2023-05-31T20:21:08.114 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T20:21:08.114 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "incremental-sync", 2023-05-31T20:21:08.114 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "00000000000000000000:00000000000000006709", 2023-05-31T20:21:08.115 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T20:21:08.115 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2023-05-31T20:21:08.115 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T20:21:08.115 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2023-05-31T20:20:29.199765Z" 2023-05-31T20:21:08.116 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T20:21:08.116 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T20:21:08.116 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2023-05-31T20:21:08.116 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:21:08.117 INFO:teuthology.orchestra.run.smithi099.stdout: "full_sync": { 2023-05-31T20:21:08.117 INFO:teuthology.orchestra.run.smithi099.stdout: "total": 0, 2023-05-31T20:21:08.117 INFO:teuthology.orchestra.run.smithi099.stdout: "complete": 0 2023-05-31T20:21:08.117 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:21:08.117 INFO:teuthology.orchestra.run.smithi099.stdout: "current_time": "2023-05-31T20:21:08Z" 2023-05-31T20:21:08.118 INFO:teuthology.orchestra.run.smithi099.stdout:} 2023-05-31T20:21:08.118 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 949931097149414600 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000005731", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-05-31T20:20:59.915684Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000003755", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-05-31T20:20:59.916728Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000009951", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-05-31T20:20:59.918574Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000006709", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-05-31T20:20:29.199765Z" } } ] }, "full_sync": { "total": 0, "complete": 0 }, "current_time": "2023-05-31T20:21:08Z" } 2023-05-31T20:21:08.119 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000005731', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-05-31T20:20:59.915684Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000003755', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-05-31T20:20:59.916728Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000009951', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-05-31T20:20:59.918574Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000006709', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-05-31T20:20:29.199765Z'}}] 2023-05-31T20:21:08.120 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000005731', 1: '00000000000000000000:00000000000000003755', 2: '00000000000000000000:00000000000000009951', 3: '00000000000000000000:00000000000000006709'} 2023-05-31T20:21:08.120 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000005731', 1: '00000000000000000000:00000000000000003755', 2: '00000000000000000000:00000000000000009951', 3: '00000000000000000000:00000000000000006709'} 2023-05-31T20:21:08.121 INFO:rgw_multi.tests:finished data checkpoint for target_zone=a2 source_zone=a1 2023-05-31T20:21:08.121 DEBUG:rgw_multi.tests:data checkpoint: source=a2 target=a1 2023-05-31T20:21:08.122 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['datalog', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-31T20:21:08.122 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'datalog', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-31T20:21:08.123 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 datalog status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-05-31T20:21:08.171 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2023-05-31T20:21:08.171 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2023-05-31T20:21:08.252 INFO:teuthology.orchestra.run.smithi099.stdout:[ 2023-05-31T20:21:08.253 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T20:21:08.253 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "00000000000000000000:00000000000000003893", 2023-05-31T20:21:08.254 INFO:teuthology.orchestra.run.smithi099.stdout: "last_update": "2023-05-31T20:20:58.573591Z" 2023-05-31T20:21:08.254 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:21:08.254 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T20:21:08.255 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "00000000000000000000:00000000000000002486", 2023-05-31T20:21:08.255 INFO:teuthology.orchestra.run.smithi099.stdout: "last_update": "2023-05-31T20:20:58.575198Z" 2023-05-31T20:21:08.255 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:21:08.256 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T20:21:08.256 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "00000000000000000000:00000000000000007977", 2023-05-31T20:21:08.256 INFO:teuthology.orchestra.run.smithi099.stdout: "last_update": "2023-05-31T20:20:58.576689Z" 2023-05-31T20:21:08.256 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:21:08.257 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T20:21:08.257 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "00000000000000000000:00000000000000004596", 2023-05-31T20:21:08.257 INFO:teuthology.orchestra.run.smithi099.stdout: "last_update": "2023-05-31T20:20:58.579532Z" 2023-05-31T20:21:08.257 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T20:21:08.257 INFO:teuthology.orchestra.run.smithi099.stdout:] 2023-05-31T20:21:08.258 DEBUG:rgw_multi.tests:data markers for zone=a2 markers={0: '00000000000000000000:00000000000000003893', 1: '00000000000000000000:00000000000000002486', 2: '00000000000000000000:00000000000000007977', 3: '00000000000000000000:00000000000000004596'} 2023-05-31T20:21:08.259 INFO:rgw_multi.tests:starting data checkpoint for target_zone=a1 source_zone=a2 2023-05-31T20:21:08.260 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-31T20:21:08.260 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-31T20:21:08.261 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --source-zone a2 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-05-31T20:21:08.312 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2023-05-31T20:21:08.312 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2023-05-31T20:21:08.366 INFO:teuthology.orchestra.run.smithi099.stdout:{ 2023-05-31T20:21:08.367 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_status": { 2023-05-31T20:21:08.367 INFO:teuthology.orchestra.run.smithi099.stdout: "info": { 2023-05-31T20:21:08.367 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "sync", 2023-05-31T20:21:08.367 INFO:teuthology.orchestra.run.smithi099.stdout: "num_shards": 4, 2023-05-31T20:21:08.367 INFO:teuthology.orchestra.run.smithi099.stdout: "instance_id": 17941554979570305020 2023-05-31T20:21:08.368 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:21:08.368 INFO:teuthology.orchestra.run.smithi099.stdout: "markers": [ 2023-05-31T20:21:08.368 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T20:21:08.368 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 0, 2023-05-31T20:21:08.368 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T20:21:08.368 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "incremental-sync", 2023-05-31T20:21:08.369 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "00000000000000000000:00000000000000003893", 2023-05-31T20:21:08.369 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T20:21:08.369 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2023-05-31T20:21:08.369 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T20:21:08.369 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2023-05-31T20:20:58.573591Z" 2023-05-31T20:21:08.369 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T20:21:08.370 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:21:08.370 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T20:21:08.370 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 1, 2023-05-31T20:21:08.370 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T20:21:08.370 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "incremental-sync", 2023-05-31T20:21:08.370 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "00000000000000000000:00000000000000002486", 2023-05-31T20:21:08.370 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T20:21:08.371 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2023-05-31T20:21:08.371 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T20:21:08.371 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2023-05-31T20:20:58.575198Z" 2023-05-31T20:21:08.372 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T20:21:08.372 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:21:08.372 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T20:21:08.372 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 2, 2023-05-31T20:21:08.373 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T20:21:08.373 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "incremental-sync", 2023-05-31T20:21:08.373 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "00000000000000000000:00000000000000007977", 2023-05-31T20:21:08.373 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T20:21:08.373 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2023-05-31T20:21:08.374 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T20:21:08.374 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2023-05-31T20:20:58.576689Z" 2023-05-31T20:21:08.374 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T20:21:08.374 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:21:08.375 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T20:21:08.375 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 3, 2023-05-31T20:21:08.375 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T20:21:08.376 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "incremental-sync", 2023-05-31T20:21:08.376 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "00000000000000000000:00000000000000003750", 2023-05-31T20:21:08.376 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T20:21:08.377 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2023-05-31T20:21:08.377 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T20:21:08.377 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2023-05-31T20:20:25.472599Z" 2023-05-31T20:21:08.378 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T20:21:08.378 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T20:21:08.378 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2023-05-31T20:21:08.379 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:21:08.379 INFO:teuthology.orchestra.run.smithi099.stdout: "full_sync": { 2023-05-31T20:21:08.379 INFO:teuthology.orchestra.run.smithi099.stdout: "total": 0, 2023-05-31T20:21:08.379 INFO:teuthology.orchestra.run.smithi099.stdout: "complete": 0 2023-05-31T20:21:08.380 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:21:08.380 INFO:teuthology.orchestra.run.smithi099.stdout: "current_time": "2023-05-31T20:21:08Z" 2023-05-31T20:21:08.380 INFO:teuthology.orchestra.run.smithi099.stdout:} 2023-05-31T20:21:08.381 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 17941554979570305020 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000003893", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-05-31T20:20:58.573591Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000002486", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-05-31T20:20:58.575198Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000007977", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-05-31T20:20:58.576689Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000003750", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-05-31T20:20:25.472599Z" } } ] }, "full_sync": { "total": 0, "complete": 0 }, "current_time": "2023-05-31T20:21:08Z" } 2023-05-31T20:21:08.381 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000003893', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-05-31T20:20:58.573591Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000002486', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-05-31T20:20:58.575198Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000007977', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-05-31T20:20:58.576689Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000003750', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-05-31T20:20:25.472599Z'}}] 2023-05-31T20:21:08.382 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000003893', 1: '00000000000000000000:00000000000000002486', 2: '00000000000000000000:00000000000000007977', 3: '00000000000000000000:00000000000000004596'} 2023-05-31T20:21:08.382 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000003893', 1: '00000000000000000000:00000000000000002486', 2: '00000000000000000000:00000000000000007977', 3: '00000000000000000000:00000000000000003750'} 2023-05-31T20:21:08.383 WARNING:rgw_multi.tests:data of zone a1 behind zone a2: shard=3 master=00000000000000000000:00000000000000004596 target=00000000000000000000:00000000000000003750 2023-05-31T20:21:13.384 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-31T20:21:13.385 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-31T20:21:13.386 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --source-zone a2 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-05-31T20:21:13.445 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2023-05-31T20:21:13.445 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2023-05-31T20:21:13.516 INFO:teuthology.orchestra.run.smithi099.stdout:{ 2023-05-31T20:21:13.517 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_status": { 2023-05-31T20:21:13.517 INFO:teuthology.orchestra.run.smithi099.stdout: "info": { 2023-05-31T20:21:13.517 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "sync", 2023-05-31T20:21:13.518 INFO:teuthology.orchestra.run.smithi099.stdout: "num_shards": 4, 2023-05-31T20:21:13.518 INFO:teuthology.orchestra.run.smithi099.stdout: "instance_id": 17941554979570305020 2023-05-31T20:21:13.518 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:21:13.518 INFO:teuthology.orchestra.run.smithi099.stdout: "markers": [ 2023-05-31T20:21:13.519 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T20:21:13.519 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 0, 2023-05-31T20:21:13.519 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T20:21:13.519 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "incremental-sync", 2023-05-31T20:21:13.519 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "00000000000000000000:00000000000000003893", 2023-05-31T20:21:13.519 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T20:21:13.520 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2023-05-31T20:21:13.520 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T20:21:13.520 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2023-05-31T20:20:58.573591Z" 2023-05-31T20:21:13.520 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T20:21:13.521 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:21:13.521 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T20:21:13.521 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 1, 2023-05-31T20:21:13.521 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T20:21:13.521 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "incremental-sync", 2023-05-31T20:21:13.521 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "00000000000000000000:00000000000000002486", 2023-05-31T20:21:13.522 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T20:21:13.522 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2023-05-31T20:21:13.522 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T20:21:13.522 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2023-05-31T20:20:58.575198Z" 2023-05-31T20:21:13.522 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T20:21:13.523 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:21:13.523 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T20:21:13.523 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 2, 2023-05-31T20:21:13.524 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T20:21:13.524 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "incremental-sync", 2023-05-31T20:21:13.524 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "00000000000000000000:00000000000000007977", 2023-05-31T20:21:13.525 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T20:21:13.525 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2023-05-31T20:21:13.525 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T20:21:13.525 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2023-05-31T20:20:58.576689Z" 2023-05-31T20:21:13.525 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T20:21:13.526 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:21:13.526 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T20:21:13.526 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 3, 2023-05-31T20:21:13.526 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T20:21:13.526 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "incremental-sync", 2023-05-31T20:21:13.526 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "00000000000000000000:00000000000000004596", 2023-05-31T20:21:13.527 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T20:21:13.527 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2023-05-31T20:21:13.527 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T20:21:13.527 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2023-05-31T20:20:58.579532Z" 2023-05-31T20:21:13.527 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T20:21:13.528 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T20:21:13.528 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2023-05-31T20:21:13.528 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:21:13.528 INFO:teuthology.orchestra.run.smithi099.stdout: "full_sync": { 2023-05-31T20:21:13.529 INFO:teuthology.orchestra.run.smithi099.stdout: "total": 0, 2023-05-31T20:21:13.529 INFO:teuthology.orchestra.run.smithi099.stdout: "complete": 0 2023-05-31T20:21:13.529 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:21:13.530 INFO:teuthology.orchestra.run.smithi099.stdout: "current_time": "2023-05-31T20:21:13Z" 2023-05-31T20:21:13.530 INFO:teuthology.orchestra.run.smithi099.stdout:} 2023-05-31T20:21:13.531 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 17941554979570305020 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000003893", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-05-31T20:20:58.573591Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000002486", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-05-31T20:20:58.575198Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000007977", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-05-31T20:20:58.576689Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000004596", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-05-31T20:20:58.579532Z" } } ] }, "full_sync": { "total": 0, "complete": 0 }, "current_time": "2023-05-31T20:21:13Z" } 2023-05-31T20:21:13.532 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000003893', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-05-31T20:20:58.573591Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000002486', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-05-31T20:20:58.575198Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000007977', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-05-31T20:20:58.576689Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000004596', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-05-31T20:20:58.579532Z'}}] 2023-05-31T20:21:13.533 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000003893', 1: '00000000000000000000:00000000000000002486', 2: '00000000000000000000:00000000000000007977', 3: '00000000000000000000:00000000000000004596'} 2023-05-31T20:21:13.533 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000003893', 1: '00000000000000000000:00000000000000002486', 2: '00000000000000000000:00000000000000007977', 3: '00000000000000000000:00000000000000004596'} 2023-05-31T20:21:13.534 INFO:rgw_multi.tests:finished data checkpoint for target_zone=a1 source_zone=a2 2023-05-31T20:21:13.535 INFO:tasks.rgw_multisite_tests:rgw_multi.tests.test_bucket_sync_disable_enable ... ok 2023-05-31T20:21:13.536 INFO:rgw_multi.tests:create bucket zone=a1 name=pklxkd-50 2023-05-31T20:21:13.537 DEBUG:boto:path=/pklxkd-50/ 2023-05-31T20:21:13.538 DEBUG:boto:auth_path=/pklxkd-50/ 2023-05-31T20:21:13.539 DEBUG:boto:Method: PUT 2023-05-31T20:21:13.539 DEBUG:boto:Path: /pklxkd-50/ 2023-05-31T20:21:13.540 DEBUG:boto:Data: 2023-05-31T20:21:13.540 DEBUG:boto:Headers: {} 2023-05-31T20:21:13.541 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:21:13.541 DEBUG:boto:Port: 8000 2023-05-31T20:21:13.542 DEBUG:boto:Params: {} 2023-05-31T20:21:13.542 DEBUG:boto:Token: None 2023-05-31T20:21:13.543 DEBUG:boto:StringToSign: PUT Wed, 31 May 2023 20:21:13 GMT /pklxkd-50/ 2023-05-31T20:21:13.544 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:0/gX4vy6TGy9cVDN2RvFYBXeUyU= 2023-05-31T20:21:13.545 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:21:13 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:0/gX4vy6TGy9cVDN2RvFYBXeUyU=', 'Content-Length': '0'} 2023-05-31T20:21:13.563 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000fa65076b9403fa8c-006477ac39-1630-a1'), ('Content-Length', '0'), ('Date', 'Wed, 31 May 2023 20:21:13 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:21:13.564 INFO:rgw_multi.tests:create bucket zone=a2 name=pklxkd-51 2023-05-31T20:21:13.565 DEBUG:boto:path=/pklxkd-51/ 2023-05-31T20:21:13.565 DEBUG:boto:auth_path=/pklxkd-51/ 2023-05-31T20:21:13.566 DEBUG:boto:Method: PUT 2023-05-31T20:21:13.567 DEBUG:boto:Path: /pklxkd-51/ 2023-05-31T20:21:13.567 DEBUG:boto:Data: 2023-05-31T20:21:13.568 DEBUG:boto:Headers: {} 2023-05-31T20:21:13.571 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:21:13.571 DEBUG:boto:Port: 8001 2023-05-31T20:21:13.572 DEBUG:boto:Params: {} 2023-05-31T20:21:13.572 DEBUG:boto:Token: None 2023-05-31T20:21:13.572 DEBUG:boto:StringToSign: PUT Wed, 31 May 2023 20:21:13 GMT /pklxkd-51/ 2023-05-31T20:21:13.573 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:jF0bGtJUVUkFvPHBzB2NoCuH4oE= 2023-05-31T20:21:13.573 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:21:13 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:jF0bGtJUVUkFvPHBzB2NoCuH4oE=', 'Content-Length': '0'} 2023-05-31T20:21:13.608 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000d6ae602949a07531-006477ac39-162e-a2'), ('Content-Length', '0'), ('Date', 'Wed, 31 May 2023 20:21:13 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:21:13.610 DEBUG:boto:path=/pklxkd-50/MULTIPART 2023-05-31T20:21:13.611 DEBUG:boto:auth_path=/pklxkd-50/MULTIPART 2023-05-31T20:21:13.611 DEBUG:boto:path=/pklxkd-50/MULTIPART?uploads 2023-05-31T20:21:13.612 DEBUG:boto:auth_path=/pklxkd-50/MULTIPART?uploads 2023-05-31T20:21:13.613 DEBUG:boto:Method: POST 2023-05-31T20:21:13.614 DEBUG:boto:Path: /pklxkd-50/MULTIPART?uploads 2023-05-31T20:21:13.615 DEBUG:boto:Data: 2023-05-31T20:21:13.615 DEBUG:boto:Headers: {} 2023-05-31T20:21:13.616 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:21:13.616 DEBUG:boto:Port: 8000 2023-05-31T20:21:13.617 DEBUG:boto:Params: {} 2023-05-31T20:21:13.618 DEBUG:boto:Token: None 2023-05-31T20:21:13.619 DEBUG:boto:StringToSign: POST Wed, 31 May 2023 20:21:13 GMT /pklxkd-50/MULTIPART?uploads 2023-05-31T20:21:13.619 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:PgJatWHea2jYm5w9dkWavZoUif0= 2023-05-31T20:21:13.620 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:21:13 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:PgJatWHea2jYm5w9dkWavZoUif0=', 'Content-Length': '0'} 2023-05-31T20:21:15.101 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000b77e1d20935ed64c-006477ac39-1630-a1'), ('Content-Type', 'application/xml'), ('Content-Length', '249'), ('Date', 'Wed, 31 May 2023 20:21:15 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:21:15.102 DEBUG:boto:b'pklxkd-50MULTIPART2~wvY4FwQUNMeD1LX8gqy2Ts5j-I2Rc7b' 2023-05-31T20:21:15.144 DEBUG:boto:path=/pklxkd-50/MULTIPART 2023-05-31T20:21:15.145 DEBUG:boto:auth_path=/pklxkd-50/MULTIPART 2023-05-31T20:21:15.145 DEBUG:boto:path=/pklxkd-50/MULTIPART?uploadId=2~wvY4FwQUNMeD1LX8gqy2Ts5j-I2Rc7b&partNumber=1 2023-05-31T20:21:15.146 DEBUG:boto:auth_path=/pklxkd-50/MULTIPART?uploadId=2~wvY4FwQUNMeD1LX8gqy2Ts5j-I2Rc7b&partNumber=1 2023-05-31T20:21:15.146 DEBUG:boto:Method: PUT 2023-05-31T20:21:15.146 DEBUG:boto:Path: /pklxkd-50/MULTIPART?uploadId=2~wvY4FwQUNMeD1LX8gqy2Ts5j-I2Rc7b&partNumber=1 2023-05-31T20:21:15.147 DEBUG:boto:Data: 2023-05-31T20:21:15.147 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'ebKBBg0ze5srhMzzkK3PdA==', 'Content-Length': '5242880', 'Expect': '100-Continue'} 2023-05-31T20:21:15.147 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:21:15.148 DEBUG:boto:Port: 8000 2023-05-31T20:21:15.148 DEBUG:boto:Params: {} 2023-05-31T20:21:15.148 DEBUG:boto:Token: None 2023-05-31T20:21:15.149 DEBUG:boto:StringToSign: PUT ebKBBg0ze5srhMzzkK3PdA== application/octet-stream Wed, 31 May 2023 20:21:15 GMT /pklxkd-50/MULTIPART?partNumber=1&uploadId=2~wvY4FwQUNMeD1LX8gqy2Ts5j-I2Rc7b 2023-05-31T20:21:15.149 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:n7/yFlkNCodWoxqsPbnuaWcreyc= 2023-05-31T20:21:15.150 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'ebKBBg0ze5srhMzzkK3PdA==', 'Content-Length': '5242880', 'Expect': '100-Continue', 'Date': 'Wed, 31 May 2023 20:21:15 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:n7/yFlkNCodWoxqsPbnuaWcreyc='} 2023-05-31T20:21:15.189 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"79b281060d337b9b2b84ccf390adcf74"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000632fa0740ae16b83-006477ac3b-1630-a1'), ('Date', 'Wed, 31 May 2023 20:21:15 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:21:15.230 DEBUG:boto:path=/pklxkd-50/MULTIPART 2023-05-31T20:21:15.231 DEBUG:boto:auth_path=/pklxkd-50/MULTIPART 2023-05-31T20:21:15.232 DEBUG:boto:path=/pklxkd-50/MULTIPART?uploadId=2~wvY4FwQUNMeD1LX8gqy2Ts5j-I2Rc7b&partNumber=2 2023-05-31T20:21:15.233 DEBUG:boto:auth_path=/pklxkd-50/MULTIPART?uploadId=2~wvY4FwQUNMeD1LX8gqy2Ts5j-I2Rc7b&partNumber=2 2023-05-31T20:21:15.233 DEBUG:boto:Method: PUT 2023-05-31T20:21:15.234 DEBUG:boto:Path: /pklxkd-50/MULTIPART?uploadId=2~wvY4FwQUNMeD1LX8gqy2Ts5j-I2Rc7b&partNumber=2 2023-05-31T20:21:15.235 DEBUG:boto:Data: 2023-05-31T20:21:15.235 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'dIQ6OrGTo4m87YmUAtmdXw==', 'Content-Length': '5242880', 'Expect': '100-Continue'} 2023-05-31T20:21:15.236 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:21:15.237 DEBUG:boto:Port: 8000 2023-05-31T20:21:15.237 DEBUG:boto:Params: {} 2023-05-31T20:21:15.238 DEBUG:boto:Token: None 2023-05-31T20:21:15.239 DEBUG:boto:StringToSign: PUT dIQ6OrGTo4m87YmUAtmdXw== application/octet-stream Wed, 31 May 2023 20:21:15 GMT /pklxkd-50/MULTIPART?partNumber=2&uploadId=2~wvY4FwQUNMeD1LX8gqy2Ts5j-I2Rc7b 2023-05-31T20:21:15.239 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:RaW01Hk5EiAN+opQJHmA8AsVSlE= 2023-05-31T20:21:15.240 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'dIQ6OrGTo4m87YmUAtmdXw==', 'Content-Length': '5242880', 'Expect': '100-Continue', 'Date': 'Wed, 31 May 2023 20:21:15 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:RaW01Hk5EiAN+opQJHmA8AsVSlE='} 2023-05-31T20:21:15.277 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"74843a3ab193a389bced899402d99d5f"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000016426f026e02b6b4-006477ac3b-1630-a1'), ('Date', 'Wed, 31 May 2023 20:21:15 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:21:15.316 DEBUG:boto:path=/pklxkd-50/MULTIPART 2023-05-31T20:21:15.317 DEBUG:boto:auth_path=/pklxkd-50/MULTIPART 2023-05-31T20:21:15.317 DEBUG:boto:path=/pklxkd-50/MULTIPART?uploadId=2~wvY4FwQUNMeD1LX8gqy2Ts5j-I2Rc7b&partNumber=3 2023-05-31T20:21:15.318 DEBUG:boto:auth_path=/pklxkd-50/MULTIPART?uploadId=2~wvY4FwQUNMeD1LX8gqy2Ts5j-I2Rc7b&partNumber=3 2023-05-31T20:21:15.318 DEBUG:boto:Method: PUT 2023-05-31T20:21:15.319 DEBUG:boto:Path: /pklxkd-50/MULTIPART?uploadId=2~wvY4FwQUNMeD1LX8gqy2Ts5j-I2Rc7b&partNumber=3 2023-05-31T20:21:15.319 DEBUG:boto:Data: 2023-05-31T20:21:15.319 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'e4RW4edMN49Fhh9TYZ51tg==', 'Content-Length': '5242880', 'Expect': '100-Continue'} 2023-05-31T20:21:15.320 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:21:15.320 DEBUG:boto:Port: 8000 2023-05-31T20:21:15.320 DEBUG:boto:Params: {} 2023-05-31T20:21:15.321 DEBUG:boto:Token: None 2023-05-31T20:21:15.321 DEBUG:boto:StringToSign: PUT e4RW4edMN49Fhh9TYZ51tg== application/octet-stream Wed, 31 May 2023 20:21:15 GMT /pklxkd-50/MULTIPART?partNumber=3&uploadId=2~wvY4FwQUNMeD1LX8gqy2Ts5j-I2Rc7b 2023-05-31T20:21:15.322 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:/fcPHMAolPexMjSXKOzd5AA0ErE= 2023-05-31T20:21:15.322 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'e4RW4edMN49Fhh9TYZ51tg==', 'Content-Length': '5242880', 'Expect': '100-Continue', 'Date': 'Wed, 31 May 2023 20:21:15 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:/fcPHMAolPexMjSXKOzd5AA0ErE='} 2023-05-31T20:21:15.353 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"7b8456e1e74c378f45861f53619e75b6"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000bb7833aab3323dea-006477ac3b-1630-a1'), ('Date', 'Wed, 31 May 2023 20:21:15 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:21:15.373 DEBUG:boto:path=/pklxkd-50/MULTIPART 2023-05-31T20:21:15.374 DEBUG:boto:auth_path=/pklxkd-50/MULTIPART 2023-05-31T20:21:15.375 DEBUG:boto:path=/pklxkd-50/MULTIPART?uploadId=2~wvY4FwQUNMeD1LX8gqy2Ts5j-I2Rc7b&partNumber=4 2023-05-31T20:21:15.376 DEBUG:boto:auth_path=/pklxkd-50/MULTIPART?uploadId=2~wvY4FwQUNMeD1LX8gqy2Ts5j-I2Rc7b&partNumber=4 2023-05-31T20:21:15.376 DEBUG:boto:Method: PUT 2023-05-31T20:21:15.377 DEBUG:boto:Path: /pklxkd-50/MULTIPART?uploadId=2~wvY4FwQUNMeD1LX8gqy2Ts5j-I2Rc7b&partNumber=4 2023-05-31T20:21:15.377 DEBUG:boto:Data: 2023-05-31T20:21:15.378 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'l+jDi4yPcIlVvEiQwDQZTQ==', 'Content-Length': '5242880', 'Expect': '100-Continue'} 2023-05-31T20:21:15.378 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:21:15.379 DEBUG:boto:Port: 8000 2023-05-31T20:21:15.379 DEBUG:boto:Params: {} 2023-05-31T20:21:15.379 DEBUG:boto:Token: None 2023-05-31T20:21:15.380 DEBUG:boto:StringToSign: PUT l+jDi4yPcIlVvEiQwDQZTQ== application/octet-stream Wed, 31 May 2023 20:21:15 GMT /pklxkd-50/MULTIPART?partNumber=4&uploadId=2~wvY4FwQUNMeD1LX8gqy2Ts5j-I2Rc7b 2023-05-31T20:21:15.381 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:w/BUYJVWX9KEaVMAXHCjqvcmKh8= 2023-05-31T20:21:15.381 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'l+jDi4yPcIlVvEiQwDQZTQ==', 'Content-Length': '5242880', 'Expect': '100-Continue', 'Date': 'Wed, 31 May 2023 20:21:15 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:w/BUYJVWX9KEaVMAXHCjqvcmKh8='} 2023-05-31T20:21:15.413 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"97e8c38b8c8f708955bc4890c034194d"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000063277da8deb93a27-006477ac3b-1630-a1'), ('Date', 'Wed, 31 May 2023 20:21:15 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:21:15.413 DEBUG:boto:path=/pklxkd-50/MULTIPART 2023-05-31T20:21:15.414 DEBUG:boto:auth_path=/pklxkd-50/MULTIPART 2023-05-31T20:21:15.414 DEBUG:boto:path=/pklxkd-50/MULTIPART?uploadId=2~wvY4FwQUNMeD1LX8gqy2Ts5j-I2Rc7b 2023-05-31T20:21:15.414 DEBUG:boto:auth_path=/pklxkd-50/MULTIPART?uploadId=2~wvY4FwQUNMeD1LX8gqy2Ts5j-I2Rc7b 2023-05-31T20:21:15.415 DEBUG:boto:Method: GET 2023-05-31T20:21:15.415 DEBUG:boto:Path: /pklxkd-50/MULTIPART?uploadId=2~wvY4FwQUNMeD1LX8gqy2Ts5j-I2Rc7b 2023-05-31T20:21:15.415 DEBUG:boto:Data: 2023-05-31T20:21:15.416 DEBUG:boto:Headers: {} 2023-05-31T20:21:15.416 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:21:15.417 DEBUG:boto:Port: 8000 2023-05-31T20:21:15.417 DEBUG:boto:Params: {} 2023-05-31T20:21:15.418 DEBUG:boto:Token: None 2023-05-31T20:21:15.418 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:21:15 GMT /pklxkd-50/MULTIPART?uploadId=2~wvY4FwQUNMeD1LX8gqy2Ts5j-I2Rc7b 2023-05-31T20:21:15.419 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:OOzvzs/kWkjy5pYT4MMAUdDneSA= 2023-05-31T20:21:15.419 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:21:15 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:OOzvzs/kWkjy5pYT4MMAUdDneSA=', 'Content-Length': '0'} 2023-05-31T20:21:15.422 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ed7d64d150da1ecc-006477ac3b-1630-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:21:15 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:21:15.465 DEBUG:boto:path=/pklxkd-50/MULTIPART 2023-05-31T20:21:15.465 DEBUG:boto:auth_path=/pklxkd-50/MULTIPART 2023-05-31T20:21:15.465 DEBUG:boto:path=/pklxkd-50/MULTIPART?uploadId=2~wvY4FwQUNMeD1LX8gqy2Ts5j-I2Rc7b 2023-05-31T20:21:15.466 DEBUG:boto:auth_path=/pklxkd-50/MULTIPART?uploadId=2~wvY4FwQUNMeD1LX8gqy2Ts5j-I2Rc7b 2023-05-31T20:21:15.466 DEBUG:boto:Method: POST 2023-05-31T20:21:15.467 DEBUG:boto:Path: /pklxkd-50/MULTIPART?uploadId=2~wvY4FwQUNMeD1LX8gqy2Ts5j-I2Rc7b 2023-05-31T20:21:15.467 DEBUG:boto:Data: 1 "79b281060d337b9b2b84ccf390adcf74" 2 "74843a3ab193a389bced899402d99d5f" 3 "7b8456e1e74c378f45861f53619e75b6" 4 "97e8c38b8c8f708955bc4890c034194d" 2023-05-31T20:21:15.467 DEBUG:boto:Headers: {'Content-Type': 'text/xml'} 2023-05-31T20:21:15.468 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:21:15.468 DEBUG:boto:Port: 8000 2023-05-31T20:21:15.469 DEBUG:boto:Params: {} 2023-05-31T20:21:15.470 DEBUG:boto:Token: None 2023-05-31T20:21:15.470 DEBUG:boto:StringToSign: POST text/xml Wed, 31 May 2023 20:21:15 GMT /pklxkd-50/MULTIPART?uploadId=2~wvY4FwQUNMeD1LX8gqy2Ts5j-I2Rc7b 2023-05-31T20:21:15.473 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:4XZMXaxkNk75aaU+48kuEP6xR1Y= 2023-05-31T20:21:15.473 DEBUG:boto:Final headers: {'Content-Type': 'text/xml', 'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:21:15 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:4XZMXaxkNk75aaU+48kuEP6xR1Y=', 'Content-Length': '460'} 2023-05-31T20:21:15.490 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx000008f538573ab710c71-006477ac3b-1630-a1'), ('Content-Type', 'application/xml'), ('Content-Length', '291'), ('Date', 'Wed, 31 May 2023 20:21:15 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:21:15.491 DEBUG:boto:http://smithi099.front.sepia.ceph.com:8000/pklxkd-50/MULTIPARTpklxkd-50MULTIPART 2023-05-31T20:21:15.491 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-31T20:21:15.492 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-31T20:21:15.492 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-05-31T20:21:15.540 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2023-05-31T20:21:15.540 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2023-05-31T20:21:15.594 INFO:teuthology.orchestra.run.smithi099.stderr:No --period given, using current period=2e6123d2-d695-4350-9a01-61a455bc782b 2023-05-31T20:21:15.612 INFO:teuthology.orchestra.run.smithi099.stdout:[ 2023-05-31T20:21:15.613 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T20:21:15.613 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1685562631.516089_59.1", 2023-05-31T20:21:15.613 INFO:teuthology.orchestra.run.smithi099.stdout: "last_update": "2023-05-31T19:50:31.516089Z" 2023-05-31T20:21:15.613 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:21:15.613 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T20:21:15.614 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "", 2023-05-31T20:21:15.614 INFO:teuthology.orchestra.run.smithi099.stdout: "last_update": "0.000000" 2023-05-31T20:21:15.614 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:21:15.614 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T20:21:15.614 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1685562970.274572_93.1", 2023-05-31T20:21:15.614 INFO:teuthology.orchestra.run.smithi099.stdout: "last_update": "2023-05-31T19:56:10.274572Z" 2023-05-31T20:21:15.615 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:21:15.615 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T20:21:15.615 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1685564473.588985_342.1", 2023-05-31T20:21:15.616 INFO:teuthology.orchestra.run.smithi099.stdout: "last_update": "2023-05-31T20:21:13.588985Z" 2023-05-31T20:21:15.616 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T20:21:15.616 INFO:teuthology.orchestra.run.smithi099.stdout:] 2023-05-31T20:21:15.617 DEBUG:rgw_multi.tests:master meta markers={0: '1_1685562631.516089_59.1', 1: '', 2: '1_1685562970.274572_93.1', 3: '1_1685564473.588985_342.1'} 2023-05-31T20:21:15.618 INFO:rgw_multi.tests:starting meta checkpoint for zone=a2 2023-05-31T20:21:15.618 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-31T20:21:15.618 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-31T20:21:15.619 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-05-31T20:21:15.670 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2023-05-31T20:21:15.671 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2023-05-31T20:21:15.704 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:21:15.700+0000 7fd5eed03780 20 RGW-SYNC:meta: read sync status 2023-05-31T20:21:15.705 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:21:15.701+0000 7fd5eed03780 20 RGW-SYNC:meta: read sync status 2023-05-31T20:21:15.718 INFO:teuthology.orchestra.run.smithi099.stdout:{ 2023-05-31T20:21:15.718 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_status": { 2023-05-31T20:21:15.719 INFO:teuthology.orchestra.run.smithi099.stdout: "info": { 2023-05-31T20:21:15.719 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "sync", 2023-05-31T20:21:15.719 INFO:teuthology.orchestra.run.smithi099.stdout: "num_shards": 4, 2023-05-31T20:21:15.719 INFO:teuthology.orchestra.run.smithi099.stdout: "period": "2e6123d2-d695-4350-9a01-61a455bc782b", 2023-05-31T20:21:15.719 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2023-05-31T20:21:15.719 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:21:15.720 INFO:teuthology.orchestra.run.smithi099.stdout: "markers": [ 2023-05-31T20:21:15.720 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T20:21:15.720 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 0, 2023-05-31T20:21:15.720 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T20:21:15.720 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2023-05-31T20:21:15.721 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1685562631.516089_59.1", 2023-05-31T20:21:15.721 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T20:21:15.721 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 1, 2023-05-31T20:21:15.721 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T20:21:15.721 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2023-05-31T19:50:31.516089Z", 2023-05-31T20:21:15.721 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2023-05-31T20:21:15.722 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T20:21:15.722 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:21:15.722 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T20:21:15.722 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 1, 2023-05-31T20:21:15.722 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T20:21:15.723 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2023-05-31T20:21:15.723 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "", 2023-05-31T20:21:15.724 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T20:21:15.724 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2023-05-31T20:21:15.724 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T20:21:15.724 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "0.000000", 2023-05-31T20:21:15.725 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2023-05-31T20:21:15.725 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T20:21:15.725 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:21:15.726 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T20:21:15.726 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 2, 2023-05-31T20:21:15.726 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T20:21:15.726 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2023-05-31T20:21:15.726 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1685562970.274572_93.1", 2023-05-31T20:21:15.727 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T20:21:15.727 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2023-05-31T20:21:15.727 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T20:21:15.728 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2023-05-31T19:56:10.274572Z", 2023-05-31T20:21:15.732 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2023-05-31T20:21:15.732 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T20:21:15.732 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:21:15.732 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T20:21:15.733 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 3, 2023-05-31T20:21:15.733 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T20:21:15.733 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2023-05-31T20:21:15.733 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1685564473.550217_335.1", 2023-05-31T20:21:15.733 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T20:21:15.733 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2023-05-31T20:21:15.734 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T20:21:15.734 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2023-05-31T20:21:13.550217Z", 2023-05-31T20:21:15.735 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2023-05-31T20:21:15.735 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T20:21:15.735 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T20:21:15.735 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2023-05-31T20:21:15.735 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:21:15.735 INFO:teuthology.orchestra.run.smithi099.stdout: "full_sync": { 2023-05-31T20:21:15.736 INFO:teuthology.orchestra.run.smithi099.stdout: "total": 1, 2023-05-31T20:21:15.736 INFO:teuthology.orchestra.run.smithi099.stdout: "complete": 1 2023-05-31T20:21:15.736 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:21:15.736 INFO:teuthology.orchestra.run.smithi099.stdout: "current_time": "2023-05-31T20:21:15Z" 2023-05-31T20:21:15.736 INFO:teuthology.orchestra.run.smithi099.stdout:} 2023-05-31T20:21:15.738 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "2e6123d2-d695-4350-9a01-61a455bc782b", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1685562631.516089_59.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-05-31T19:50:31.516089Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1685562970.274572_93.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-05-31T19:56:10.274572Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1685564473.550217_335.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-05-31T20:21:13.550217Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-05-31T20:21:15Z" } 2023-05-31T20:21:15.738 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1685562631.516089_59.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-05-31T19:50:31.516089Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1685562970.274572_93.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-05-31T19:56:10.274572Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1685564473.550217_335.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-05-31T20:21:13.550217Z', 'realm_epoch': 2}}] 2023-05-31T20:21:15.739 DEBUG:rgw_multi.tests:log_status={0: '1_1685562631.516089_59.1', 1: '', 2: '1_1685562970.274572_93.1', 3: '1_1685564473.588985_342.1'} 2023-05-31T20:21:15.739 DEBUG:rgw_multi.tests:sync_status={0: '1_1685562631.516089_59.1', 1: '', 2: '1_1685562970.274572_93.1', 3: '1_1685564473.550217_335.1'} 2023-05-31T20:21:15.740 WARNING:rgw_multi.tests:zone a2 behind master: shard=3 master=1_1685564473.588985_342.1 target=1_1685564473.550217_335.1 2023-05-31T20:21:20.741 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-31T20:21:20.742 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-31T20:21:20.743 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-05-31T20:21:20.799 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2023-05-31T20:21:20.800 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2023-05-31T20:21:20.837 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:21:20.831+0000 7f1c849cb780 20 RGW-SYNC:meta: read sync status 2023-05-31T20:21:20.838 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:21:20.832+0000 7f1c849cb780 20 RGW-SYNC:meta: read sync status 2023-05-31T20:21:20.852 INFO:teuthology.orchestra.run.smithi099.stdout:{ 2023-05-31T20:21:20.852 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_status": { 2023-05-31T20:21:20.852 INFO:teuthology.orchestra.run.smithi099.stdout: "info": { 2023-05-31T20:21:20.853 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "sync", 2023-05-31T20:21:20.853 INFO:teuthology.orchestra.run.smithi099.stdout: "num_shards": 4, 2023-05-31T20:21:20.853 INFO:teuthology.orchestra.run.smithi099.stdout: "period": "2e6123d2-d695-4350-9a01-61a455bc782b", 2023-05-31T20:21:20.854 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2023-05-31T20:21:20.854 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:21:20.854 INFO:teuthology.orchestra.run.smithi099.stdout: "markers": [ 2023-05-31T20:21:20.854 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T20:21:20.855 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 0, 2023-05-31T20:21:20.855 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T20:21:20.855 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2023-05-31T20:21:20.856 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1685562631.516089_59.1", 2023-05-31T20:21:20.856 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T20:21:20.856 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 1, 2023-05-31T20:21:20.857 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T20:21:20.857 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2023-05-31T19:50:31.516089Z", 2023-05-31T20:21:20.857 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2023-05-31T20:21:20.858 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T20:21:20.858 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:21:20.859 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T20:21:20.859 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 1, 2023-05-31T20:21:20.860 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T20:21:20.860 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2023-05-31T20:21:20.860 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "", 2023-05-31T20:21:20.860 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T20:21:20.860 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2023-05-31T20:21:20.860 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T20:21:20.861 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "0.000000", 2023-05-31T20:21:20.861 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2023-05-31T20:21:20.861 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T20:21:20.861 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:21:20.862 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T20:21:20.862 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 2, 2023-05-31T20:21:20.862 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T20:21:20.862 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2023-05-31T20:21:20.863 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1685562970.274572_93.1", 2023-05-31T20:21:20.863 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T20:21:20.863 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2023-05-31T20:21:20.864 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T20:21:20.864 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2023-05-31T19:56:10.274572Z", 2023-05-31T20:21:20.864 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2023-05-31T20:21:20.864 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T20:21:20.864 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:21:20.865 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T20:21:20.865 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 3, 2023-05-31T20:21:20.865 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T20:21:20.865 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2023-05-31T20:21:20.865 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1685564473.588985_342.1", 2023-05-31T20:21:20.866 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T20:21:20.866 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2023-05-31T20:21:20.866 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T20:21:20.867 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2023-05-31T20:21:13.588985Z", 2023-05-31T20:21:20.867 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2023-05-31T20:21:20.867 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T20:21:20.867 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T20:21:20.868 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2023-05-31T20:21:20.868 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:21:20.869 INFO:teuthology.orchestra.run.smithi099.stdout: "full_sync": { 2023-05-31T20:21:20.870 INFO:teuthology.orchestra.run.smithi099.stdout: "total": 1, 2023-05-31T20:21:20.870 INFO:teuthology.orchestra.run.smithi099.stdout: "complete": 1 2023-05-31T20:21:20.870 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:21:20.870 INFO:teuthology.orchestra.run.smithi099.stdout: "current_time": "2023-05-31T20:21:20Z" 2023-05-31T20:21:20.870 INFO:teuthology.orchestra.run.smithi099.stdout:} 2023-05-31T20:21:20.871 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "2e6123d2-d695-4350-9a01-61a455bc782b", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1685562631.516089_59.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-05-31T19:50:31.516089Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1685562970.274572_93.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-05-31T19:56:10.274572Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1685564473.588985_342.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-05-31T20:21:13.588985Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-05-31T20:21:20Z" } 2023-05-31T20:21:20.872 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1685562631.516089_59.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-05-31T19:50:31.516089Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1685562970.274572_93.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-05-31T19:56:10.274572Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1685564473.588985_342.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-05-31T20:21:13.588985Z', 'realm_epoch': 2}}] 2023-05-31T20:21:20.872 DEBUG:rgw_multi.tests:log_status={0: '1_1685562631.516089_59.1', 1: '', 2: '1_1685562970.274572_93.1', 3: '1_1685564473.588985_342.1'} 2023-05-31T20:21:20.873 DEBUG:rgw_multi.tests:sync_status={0: '1_1685562631.516089_59.1', 1: '', 2: '1_1685562970.274572_93.1', 3: '1_1685564473.588985_342.1'} 2023-05-31T20:21:20.873 INFO:rgw_multi.tests:finish meta checkpoint for zone=a2 2023-05-31T20:21:20.873 DEBUG:rgw_multi.tests:bucket checkpoint: source=a1 target=a2 bucket=pklxkd-50 2023-05-31T20:21:20.874 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'pklxkd-50', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-05-31T20:21:20.874 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'pklxkd-50', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-05-31T20:21:20.874 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket pklxkd-50 --source-zone a1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2023-05-31T20:21:20.932 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2023-05-31T20:21:20.932 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2023-05-31T20:21:20.979 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:21:20.973+0000 7ff75bea0780 1 bucket sync caught up with source: 2023-05-31T20:21:20.980 INFO:teuthology.orchestra.run.smithi099.stderr: local status: [, , , , , 00000000011.66.6, , , , , ] 2023-05-31T20:21:20.980 INFO:teuthology.orchestra.run.smithi099.stderr: remote markers: [, , , , , 00000000011.66.6, , , , , ] 2023-05-31T20:21:20.980 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:21:20.973+0000 7ff75bea0780 0 bucket checkpoint complete 2023-05-31T20:21:21.000 DEBUG:rgw_multi.tests:bucket checkpoint: source=a2 target=a1 bucket=pklxkd-50 2023-05-31T20:21:21.001 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'pklxkd-50', '--source-zone', 'a2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-05-31T20:21:21.001 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'pklxkd-50', '--source-zone', 'a2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-05-31T20:21:21.002 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket pklxkd-50 --source-zone a2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2023-05-31T20:21:21.059 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2023-05-31T20:21:21.060 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2023-05-31T20:21:21.109 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:21:21.103+0000 7f4f8db86780 1 waiting to reach incremental sync.. 2023-05-31T20:21:26.109 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:21:26.104+0000 7f4f8db86780 1 waiting to reach incremental sync.. 2023-05-31T20:21:31.114 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:21:31.106+0000 7f4f8db86780 1 bucket sync caught up with source: 2023-05-31T20:21:31.115 INFO:teuthology.orchestra.run.smithi099.stderr: local status: [, , , , , 00000000001.1069.6, , , , , ] 2023-05-31T20:21:31.115 INFO:teuthology.orchestra.run.smithi099.stderr: remote markers: [, , , , , 00000000001.1069.6, , , , , ] 2023-05-31T20:21:31.115 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:21:31.106+0000 7f4f8db86780 0 bucket checkpoint complete 2023-05-31T20:21:31.135 INFO:tasks.rgw_multi.zone_rados:comparing bucket=pklxkd-50 zones={a2, a1} 2023-05-31T20:21:31.136 DEBUG:boto:path=/pklxkd-50/ 2023-05-31T20:21:31.136 DEBUG:boto:auth_path=/pklxkd-50/ 2023-05-31T20:21:31.137 DEBUG:boto:Method: HEAD 2023-05-31T20:21:31.137 DEBUG:boto:Path: /pklxkd-50/ 2023-05-31T20:21:31.138 DEBUG:boto:Data: 2023-05-31T20:21:31.138 DEBUG:boto:Headers: {} 2023-05-31T20:21:31.139 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:21:31.139 DEBUG:boto:Port: 8001 2023-05-31T20:21:31.139 DEBUG:boto:Params: {} 2023-05-31T20:21:31.140 DEBUG:boto:Token: None 2023-05-31T20:21:31.143 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:21:31 GMT /pklxkd-50/ 2023-05-31T20:21:31.144 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:bqlu8b4lmttQaMYYF9gxqlpWpTY= 2023-05-31T20:21:31.145 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:21:31 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:bqlu8b4lmttQaMYYF9gxqlpWpTY=', 'Content-Length': '0'} 2023-05-31T20:21:31.147 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '20971520'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000ac55acd52837b287-006477ac4b-162e-a2'), ('Content-Length', '0'), ('Date', 'Wed, 31 May 2023 20:21:31 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:21:31.148 DEBUG:boto:path=/pklxkd-50/ 2023-05-31T20:21:31.148 DEBUG:boto:auth_path=/pklxkd-50/ 2023-05-31T20:21:31.149 DEBUG:boto:Method: HEAD 2023-05-31T20:21:31.149 DEBUG:boto:Path: /pklxkd-50/ 2023-05-31T20:21:31.149 DEBUG:boto:Data: 2023-05-31T20:21:31.150 DEBUG:boto:Headers: {} 2023-05-31T20:21:31.150 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:21:31.150 DEBUG:boto:Port: 8000 2023-05-31T20:21:31.151 DEBUG:boto:Params: {} 2023-05-31T20:21:31.151 DEBUG:boto:Token: None 2023-05-31T20:21:31.151 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:21:31 GMT /pklxkd-50/ 2023-05-31T20:21:31.152 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:bqlu8b4lmttQaMYYF9gxqlpWpTY= 2023-05-31T20:21:31.152 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:21:31 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:bqlu8b4lmttQaMYYF9gxqlpWpTY=', 'Content-Length': '0'} 2023-05-31T20:21:31.154 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '20971520'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000853b9d2904a9bd0e-006477ac4b-1630-a1'), ('Content-Length', '0'), ('Date', 'Wed, 31 May 2023 20:21:31 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:21:31.155 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2023-05-31T20:21:31.156 DEBUG:boto:path=/pklxkd-50/ 2023-05-31T20:21:31.156 DEBUG:boto:auth_path=/pklxkd-50/ 2023-05-31T20:21:31.157 DEBUG:boto:path=/pklxkd-50/?versions&max-keys=999 2023-05-31T20:21:31.157 DEBUG:boto:auth_path=/pklxkd-50/?versions&max-keys=999 2023-05-31T20:21:31.158 DEBUG:boto:Method: GET 2023-05-31T20:21:31.159 DEBUG:boto:Path: /pklxkd-50/?versions&max-keys=999 2023-05-31T20:21:31.159 DEBUG:boto:Data: 2023-05-31T20:21:31.159 DEBUG:boto:Headers: {} 2023-05-31T20:21:31.160 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:21:31.160 DEBUG:boto:Port: 8001 2023-05-31T20:21:31.161 DEBUG:boto:Params: {} 2023-05-31T20:21:31.161 DEBUG:boto:Token: None 2023-05-31T20:21:31.161 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:21:31 GMT /pklxkd-50/?versions 2023-05-31T20:21:31.162 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:cjpbESb5v8JFS/aWSOCRPdVKjV4= 2023-05-31T20:21:31.162 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:21:31 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:cjpbESb5v8JFS/aWSOCRPdVKjV4=', 'Content-Length': '0'} 2023-05-31T20:21:31.165 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000008bf1ae60a01a247b-006477ac4b-162e-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:21:31 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:21:31.166 DEBUG:boto:b'pklxkd-50999falseMULTIPARTnulltrue2023-05-31T20:21:15.473Z"d07d411772772538cd4b657da660bfb5-4"20971520STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:21:31.168 DEBUG:tasks.rgw_multi.zone_rados:o=MULTIPART 2023-05-31T20:21:31.169 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2023-05-31T20:21:31.169 DEBUG:boto:path=/pklxkd-50/ 2023-05-31T20:21:31.170 DEBUG:boto:auth_path=/pklxkd-50/ 2023-05-31T20:21:31.171 DEBUG:boto:path=/pklxkd-50/?versions&max-keys=999 2023-05-31T20:21:31.171 DEBUG:boto:auth_path=/pklxkd-50/?versions&max-keys=999 2023-05-31T20:21:31.172 DEBUG:boto:Method: GET 2023-05-31T20:21:31.173 DEBUG:boto:Path: /pklxkd-50/?versions&max-keys=999 2023-05-31T20:21:31.173 DEBUG:boto:Data: 2023-05-31T20:21:31.174 DEBUG:boto:Headers: {} 2023-05-31T20:21:31.174 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:21:31.175 DEBUG:boto:Port: 8000 2023-05-31T20:21:31.175 DEBUG:boto:Params: {} 2023-05-31T20:21:31.175 DEBUG:boto:Token: None 2023-05-31T20:21:31.176 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:21:31 GMT /pklxkd-50/?versions 2023-05-31T20:21:31.176 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:cjpbESb5v8JFS/aWSOCRPdVKjV4= 2023-05-31T20:21:31.177 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:21:31 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:cjpbESb5v8JFS/aWSOCRPdVKjV4=', 'Content-Length': '0'} 2023-05-31T20:21:31.180 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000000628b3ab9bb4fa0a-006477ac4b-1630-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:21:31 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:21:31.181 DEBUG:boto:b'pklxkd-50999falseMULTIPARTnulltrue2023-05-31T20:21:15.473Z"d07d411772772538cd4b657da660bfb5-4"20971520STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:21:31.182 DEBUG:tasks.rgw_multi.zone_rados:o=MULTIPART 2023-05-31T20:21:31.183 DEBUG:boto:path=/pklxkd-50/ 2023-05-31T20:21:31.184 DEBUG:boto:auth_path=/pklxkd-50/ 2023-05-31T20:21:31.184 DEBUG:boto:path=/pklxkd-50/?versions&max-keys=999 2023-05-31T20:21:31.185 DEBUG:boto:auth_path=/pklxkd-50/?versions&max-keys=999 2023-05-31T20:21:31.186 DEBUG:boto:Method: GET 2023-05-31T20:21:31.186 DEBUG:boto:Path: /pklxkd-50/?versions&max-keys=999 2023-05-31T20:21:31.186 DEBUG:boto:Data: 2023-05-31T20:21:31.187 DEBUG:boto:Headers: {} 2023-05-31T20:21:31.188 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:21:31.189 DEBUG:boto:Port: 8001 2023-05-31T20:21:31.189 DEBUG:boto:Params: {} 2023-05-31T20:21:31.190 DEBUG:boto:Token: None 2023-05-31T20:21:31.191 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:21:31 GMT /pklxkd-50/?versions 2023-05-31T20:21:31.191 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:cjpbESb5v8JFS/aWSOCRPdVKjV4= 2023-05-31T20:21:31.192 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:21:31 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:cjpbESb5v8JFS/aWSOCRPdVKjV4=', 'Content-Length': '0'} 2023-05-31T20:21:31.194 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000001b36279613e9d291-006477ac4b-162e-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:21:31 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:21:31.236 DEBUG:boto:b'pklxkd-50999falseMULTIPARTnulltrue2023-05-31T20:21:15.473Z"d07d411772772538cd4b657da660bfb5-4"20971520STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:21:31.237 DEBUG:boto:path=/pklxkd-50/ 2023-05-31T20:21:31.238 DEBUG:boto:auth_path=/pklxkd-50/ 2023-05-31T20:21:31.238 DEBUG:boto:path=/pklxkd-50/?versions&max-keys=999 2023-05-31T20:21:31.239 DEBUG:boto:auth_path=/pklxkd-50/?versions&max-keys=999 2023-05-31T20:21:31.240 DEBUG:boto:Method: GET 2023-05-31T20:21:31.240 DEBUG:boto:Path: /pklxkd-50/?versions&max-keys=999 2023-05-31T20:21:31.241 DEBUG:boto:Data: 2023-05-31T20:21:31.241 DEBUG:boto:Headers: {} 2023-05-31T20:21:31.241 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:21:31.242 DEBUG:boto:Port: 8000 2023-05-31T20:21:31.242 DEBUG:boto:Params: {} 2023-05-31T20:21:31.243 DEBUG:boto:Token: None 2023-05-31T20:21:31.244 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:21:31 GMT /pklxkd-50/?versions 2023-05-31T20:21:31.245 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:cjpbESb5v8JFS/aWSOCRPdVKjV4= 2023-05-31T20:21:31.246 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:21:31 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:cjpbESb5v8JFS/aWSOCRPdVKjV4=', 'Content-Length': '0'} 2023-05-31T20:21:31.249 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000088bae8ec2b3a7002-006477ac4b-1630-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:21:31 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:21:31.250 DEBUG:boto:b'pklxkd-50999falseMULTIPARTnulltrue2023-05-31T20:21:15.473Z"d07d411772772538cd4b657da660bfb5-4"20971520STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:21:31.251 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=MULTIPART 2023-05-31T20:21:31.252 DEBUG:boto:path=/pklxkd-50/MULTIPART 2023-05-31T20:21:31.253 DEBUG:boto:auth_path=/pklxkd-50/MULTIPART 2023-05-31T20:21:31.253 DEBUG:boto:path=/pklxkd-50/MULTIPART?versionId=null 2023-05-31T20:21:31.254 DEBUG:boto:auth_path=/pklxkd-50/MULTIPART?versionId=null 2023-05-31T20:21:31.255 DEBUG:boto:Method: GET 2023-05-31T20:21:31.255 DEBUG:boto:Path: /pklxkd-50/MULTIPART?versionId=null 2023-05-31T20:21:31.256 DEBUG:boto:Data: 2023-05-31T20:21:31.257 DEBUG:boto:Headers: {} 2023-05-31T20:21:31.258 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:21:31.258 DEBUG:boto:Port: 8001 2023-05-31T20:21:31.259 DEBUG:boto:Params: {} 2023-05-31T20:21:31.260 DEBUG:boto:Token: None 2023-05-31T20:21:31.260 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:21:31 GMT /pklxkd-50/MULTIPART?versionId=null 2023-05-31T20:21:31.261 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:NQFaxFqLoqiA7jMndyiorgKSGL8= 2023-05-31T20:21:31.262 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:21:31 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:NQFaxFqLoqiA7jMndyiorgKSGL8=', 'Content-Length': '0'} 2023-05-31T20:21:31.266 DEBUG:boto:Response headers: [('Content-Length', '20971520'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:21:15 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"d07d411772772538cd4b657da660bfb5-4"'), ('x-amz-request-id', 'tx0000082c41e732847f7ea-006477ac4b-162e-a2'), ('Content-Type', 'binary/octet-stream'), ('Date', 'Wed, 31 May 2023 20:21:31 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:21:31.344 DEBUG:boto:path=/pklxkd-50/MULTIPART 2023-05-31T20:21:31.344 DEBUG:boto:auth_path=/pklxkd-50/MULTIPART 2023-05-31T20:21:31.345 DEBUG:boto:path=/pklxkd-50/MULTIPART?versionId=null 2023-05-31T20:21:31.345 DEBUG:boto:auth_path=/pklxkd-50/MULTIPART?versionId=null 2023-05-31T20:21:31.346 DEBUG:boto:Method: GET 2023-05-31T20:21:31.346 DEBUG:boto:Path: /pklxkd-50/MULTIPART?versionId=null 2023-05-31T20:21:31.346 DEBUG:boto:Data: 2023-05-31T20:21:31.347 DEBUG:boto:Headers: {} 2023-05-31T20:21:31.347 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:21:31.348 DEBUG:boto:Port: 8000 2023-05-31T20:21:31.348 DEBUG:boto:Params: {} 2023-05-31T20:21:31.348 DEBUG:boto:Token: None 2023-05-31T20:21:31.349 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:21:31 GMT /pklxkd-50/MULTIPART?versionId=null 2023-05-31T20:21:31.349 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:NQFaxFqLoqiA7jMndyiorgKSGL8= 2023-05-31T20:21:31.350 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:21:31 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:NQFaxFqLoqiA7jMndyiorgKSGL8=', 'Content-Length': '0'} 2023-05-31T20:21:31.354 DEBUG:boto:Response headers: [('Content-Length', '20971520'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:21:15 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d07d411772772538cd4b657da660bfb5-4"'), ('x-amz-request-id', 'tx00000dc15cf5ac76cf38b-006477ac4b-1630-a1'), ('Content-Type', 'binary/octet-stream'), ('Date', 'Wed, 31 May 2023 20:21:31 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:21:31.443 DEBUG:boto:path=/pklxkd-50/MULTIPART 2023-05-31T20:21:31.444 DEBUG:boto:auth_path=/pklxkd-50/MULTIPART 2023-05-31T20:21:31.445 DEBUG:boto:path=/pklxkd-50/MULTIPART?versionId=null 2023-05-31T20:21:31.445 DEBUG:boto:auth_path=/pklxkd-50/MULTIPART?versionId=null 2023-05-31T20:21:31.445 DEBUG:boto:Method: HEAD 2023-05-31T20:21:31.446 DEBUG:boto:Path: /pklxkd-50/MULTIPART?versionId=null 2023-05-31T20:21:31.446 DEBUG:boto:Data: 2023-05-31T20:21:31.446 DEBUG:boto:Headers: {} 2023-05-31T20:21:31.447 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:21:31.447 DEBUG:boto:Port: 8001 2023-05-31T20:21:31.447 DEBUG:boto:Params: {} 2023-05-31T20:21:31.448 DEBUG:boto:Token: None 2023-05-31T20:21:31.448 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:21:31 GMT /pklxkd-50/MULTIPART?versionId=null 2023-05-31T20:21:31.449 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:E9U/5QwOVvw0aBFZ0+4VP5gBAhE= 2023-05-31T20:21:31.449 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:21:31 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:E9U/5QwOVvw0aBFZ0+4VP5gBAhE=', 'Content-Length': '0'} 2023-05-31T20:21:31.452 DEBUG:boto:Response headers: [('Content-Length', '20971520'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:21:15 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"d07d411772772538cd4b657da660bfb5-4"'), ('x-amz-request-id', 'tx000000d8f3790b4d550d3-006477ac4b-162e-a2'), ('Content-Type', 'binary/octet-stream'), ('Date', 'Wed, 31 May 2023 20:21:31 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:21:31.453 DEBUG:boto:path=/pklxkd-50/MULTIPART 2023-05-31T20:21:31.454 DEBUG:boto:auth_path=/pklxkd-50/MULTIPART 2023-05-31T20:21:31.454 DEBUG:boto:path=/pklxkd-50/MULTIPART?versionId=null 2023-05-31T20:21:31.455 DEBUG:boto:auth_path=/pklxkd-50/MULTIPART?versionId=null 2023-05-31T20:21:31.455 DEBUG:boto:Method: HEAD 2023-05-31T20:21:31.456 DEBUG:boto:Path: /pklxkd-50/MULTIPART?versionId=null 2023-05-31T20:21:31.457 DEBUG:boto:Data: 2023-05-31T20:21:31.457 DEBUG:boto:Headers: {} 2023-05-31T20:21:31.458 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:21:31.458 DEBUG:boto:Port: 8000 2023-05-31T20:21:31.459 DEBUG:boto:Params: {} 2023-05-31T20:21:31.460 DEBUG:boto:Token: None 2023-05-31T20:21:31.460 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:21:31 GMT /pklxkd-50/MULTIPART?versionId=null 2023-05-31T20:21:31.461 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:E9U/5QwOVvw0aBFZ0+4VP5gBAhE= 2023-05-31T20:21:31.462 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:21:31 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:E9U/5QwOVvw0aBFZ0+4VP5gBAhE=', 'Content-Length': '0'} 2023-05-31T20:21:31.464 DEBUG:boto:Response headers: [('Content-Length', '20971520'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:21:15 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d07d411772772538cd4b657da660bfb5-4"'), ('x-amz-request-id', 'tx0000043db4c3ad567449b-006477ac4b-1630-a1'), ('Content-Type', 'binary/octet-stream'), ('Date', 'Wed, 31 May 2023 20:21:31 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:21:31.466 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=MULTIPART 2023-05-31T20:21:31.466 DEBUG:boto:path=/pklxkd-50/MULTIPART 2023-05-31T20:21:31.467 DEBUG:boto:auth_path=/pklxkd-50/MULTIPART 2023-05-31T20:21:31.467 DEBUG:boto:path=/pklxkd-50/MULTIPART?versionId=null 2023-05-31T20:21:31.467 DEBUG:boto:auth_path=/pklxkd-50/MULTIPART?versionId=null 2023-05-31T20:21:31.468 DEBUG:boto:Method: GET 2023-05-31T20:21:31.468 DEBUG:boto:Path: /pklxkd-50/MULTIPART?versionId=null 2023-05-31T20:21:31.469 DEBUG:boto:Data: 2023-05-31T20:21:31.469 DEBUG:boto:Headers: {} 2023-05-31T20:21:31.469 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:21:31.470 DEBUG:boto:Port: 8001 2023-05-31T20:21:31.470 DEBUG:boto:Params: {} 2023-05-31T20:21:31.470 DEBUG:boto:Token: None 2023-05-31T20:21:31.471 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:21:31 GMT /pklxkd-50/MULTIPART?versionId=null 2023-05-31T20:21:31.471 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:NQFaxFqLoqiA7jMndyiorgKSGL8= 2023-05-31T20:21:31.472 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:21:31 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:NQFaxFqLoqiA7jMndyiorgKSGL8=', 'Content-Length': '0'} 2023-05-31T20:21:31.475 DEBUG:boto:Response headers: [('Content-Length', '20971520'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:21:15 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"d07d411772772538cd4b657da660bfb5-4"'), ('x-amz-request-id', 'tx00000752e459d8d017877-006477ac4b-162e-a2'), ('Content-Type', 'binary/octet-stream'), ('Date', 'Wed, 31 May 2023 20:21:31 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:21:31.583 DEBUG:boto:path=/pklxkd-50/MULTIPART 2023-05-31T20:21:31.584 DEBUG:boto:auth_path=/pklxkd-50/MULTIPART 2023-05-31T20:21:31.584 DEBUG:boto:path=/pklxkd-50/MULTIPART?versionId=null 2023-05-31T20:21:31.585 DEBUG:boto:auth_path=/pklxkd-50/MULTIPART?versionId=null 2023-05-31T20:21:31.585 DEBUG:boto:Method: GET 2023-05-31T20:21:31.585 DEBUG:boto:Path: /pklxkd-50/MULTIPART?versionId=null 2023-05-31T20:21:31.586 DEBUG:boto:Data: 2023-05-31T20:21:31.586 DEBUG:boto:Headers: {} 2023-05-31T20:21:31.586 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:21:31.587 DEBUG:boto:Port: 8000 2023-05-31T20:21:31.587 DEBUG:boto:Params: {} 2023-05-31T20:21:31.587 DEBUG:boto:Token: None 2023-05-31T20:21:31.588 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:21:31 GMT /pklxkd-50/MULTIPART?versionId=null 2023-05-31T20:21:31.588 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:NQFaxFqLoqiA7jMndyiorgKSGL8= 2023-05-31T20:21:31.589 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:21:31 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:NQFaxFqLoqiA7jMndyiorgKSGL8=', 'Content-Length': '0'} 2023-05-31T20:21:31.598 DEBUG:boto:Response headers: [('Content-Length', '20971520'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:21:15 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d07d411772772538cd4b657da660bfb5-4"'), ('x-amz-request-id', 'tx00000ace98cde7e9b525e-006477ac4b-1630-a1'), ('Content-Type', 'binary/octet-stream'), ('Date', 'Wed, 31 May 2023 20:21:31 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:21:31.688 DEBUG:boto:path=/pklxkd-50/ 2023-05-31T20:21:31.688 DEBUG:boto:auth_path=/pklxkd-50/ 2023-05-31T20:21:31.689 DEBUG:boto:path=/pklxkd-50/?prefix=MULTIPART 2023-05-31T20:21:31.689 DEBUG:boto:auth_path=/pklxkd-50/?prefix=MULTIPART 2023-05-31T20:21:31.690 DEBUG:boto:Method: GET 2023-05-31T20:21:31.690 DEBUG:boto:Path: /pklxkd-50/?prefix=MULTIPART 2023-05-31T20:21:31.690 DEBUG:boto:Data: 2023-05-31T20:21:31.691 DEBUG:boto:Headers: {} 2023-05-31T20:21:31.691 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:21:31.691 DEBUG:boto:Port: 8001 2023-05-31T20:21:31.692 DEBUG:boto:Params: {} 2023-05-31T20:21:31.692 DEBUG:boto:Token: None 2023-05-31T20:21:31.693 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:21:31 GMT /pklxkd-50/ 2023-05-31T20:21:31.693 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:kbV6/FbY3Z4+6JUpovzvFctd8cE= 2023-05-31T20:21:31.694 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:21:31 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:kbV6/FbY3Z4+6JUpovzvFctd8cE=', 'Content-Length': '0'} 2023-05-31T20:21:31.696 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000000dbb6895deb14d18-006477ac4b-162e-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:21:31 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:21:31.697 DEBUG:boto:b'pklxkd-50MULTIPART1000falseMULTIPART2023-05-31T20:21:15.473Z"d07d411772772538cd4b657da660bfb5-4"20971520STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:21:31.698 DEBUG:boto:path=/pklxkd-50/ 2023-05-31T20:21:31.698 DEBUG:boto:auth_path=/pklxkd-50/ 2023-05-31T20:21:31.698 DEBUG:boto:path=/pklxkd-50/?prefix=MULTIPART 2023-05-31T20:21:31.699 DEBUG:boto:auth_path=/pklxkd-50/?prefix=MULTIPART 2023-05-31T20:21:31.699 DEBUG:boto:Method: GET 2023-05-31T20:21:31.699 DEBUG:boto:Path: /pklxkd-50/?prefix=MULTIPART 2023-05-31T20:21:31.700 DEBUG:boto:Data: 2023-05-31T20:21:31.700 DEBUG:boto:Headers: {} 2023-05-31T20:21:31.701 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:21:31.701 DEBUG:boto:Port: 8000 2023-05-31T20:21:31.701 DEBUG:boto:Params: {} 2023-05-31T20:21:31.702 DEBUG:boto:Token: None 2023-05-31T20:21:31.702 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:21:31 GMT /pklxkd-50/ 2023-05-31T20:21:31.703 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:kbV6/FbY3Z4+6JUpovzvFctd8cE= 2023-05-31T20:21:31.703 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:21:31 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:kbV6/FbY3Z4+6JUpovzvFctd8cE=', 'Content-Length': '0'} 2023-05-31T20:21:31.706 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000477e485ac9888996-006477ac4b-1630-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:21:31 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:21:31.706 DEBUG:boto:b'pklxkd-50MULTIPART1000falseMULTIPART2023-05-31T20:21:15.473Z"d07d411772772538cd4b657da660bfb5-4"20971520STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:21:31.707 DEBUG:boto:path=/pklxkd-50/MULTIPART 2023-05-31T20:21:31.707 DEBUG:boto:auth_path=/pklxkd-50/MULTIPART 2023-05-31T20:21:31.708 DEBUG:boto:Method: HEAD 2023-05-31T20:21:31.708 DEBUG:boto:Path: /pklxkd-50/MULTIPART 2023-05-31T20:21:31.708 DEBUG:boto:Data: 2023-05-31T20:21:31.709 DEBUG:boto:Headers: {} 2023-05-31T20:21:31.709 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:21:31.709 DEBUG:boto:Port: 8001 2023-05-31T20:21:31.710 DEBUG:boto:Params: {} 2023-05-31T20:21:31.710 DEBUG:boto:Token: None 2023-05-31T20:21:31.711 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:21:31 GMT /pklxkd-50/MULTIPART 2023-05-31T20:21:31.711 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:foG/8bI0khc9ZnEY0/YXv0GKI5Y= 2023-05-31T20:21:31.711 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:21:31 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:foG/8bI0khc9ZnEY0/YXv0GKI5Y=', 'Content-Length': '0'} 2023-05-31T20:21:31.713 DEBUG:boto:Response headers: [('Content-Length', '20971520'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:21:15 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"d07d411772772538cd4b657da660bfb5-4"'), ('x-amz-request-id', 'tx000008bb12fc98cae4793-006477ac4b-162e-a2'), ('Content-Type', 'binary/octet-stream'), ('Date', 'Wed, 31 May 2023 20:21:31 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:21:31.714 DEBUG:boto:path=/pklxkd-50/MULTIPART 2023-05-31T20:21:31.714 DEBUG:boto:auth_path=/pklxkd-50/MULTIPART 2023-05-31T20:21:31.715 DEBUG:boto:Method: HEAD 2023-05-31T20:21:31.715 DEBUG:boto:Path: /pklxkd-50/MULTIPART 2023-05-31T20:21:31.715 DEBUG:boto:Data: 2023-05-31T20:21:31.716 DEBUG:boto:Headers: {} 2023-05-31T20:21:31.716 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:21:31.716 DEBUG:boto:Port: 8000 2023-05-31T20:21:31.717 DEBUG:boto:Params: {} 2023-05-31T20:21:31.717 DEBUG:boto:Token: None 2023-05-31T20:21:31.717 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:21:31 GMT /pklxkd-50/MULTIPART 2023-05-31T20:21:31.718 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:foG/8bI0khc9ZnEY0/YXv0GKI5Y= 2023-05-31T20:21:31.718 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:21:31 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:foG/8bI0khc9ZnEY0/YXv0GKI5Y=', 'Content-Length': '0'} 2023-05-31T20:21:31.720 DEBUG:boto:Response headers: [('Content-Length', '20971520'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:21:15 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d07d411772772538cd4b657da660bfb5-4"'), ('x-amz-request-id', 'tx000003e958d901eaca0c3-006477ac4b-1630-a1'), ('Content-Type', 'binary/octet-stream'), ('Date', 'Wed, 31 May 2023 20:21:31 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:21:31.721 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=MULTIPART 2023-05-31T20:21:31.721 DEBUG:boto:path=/pklxkd-50/MULTIPART 2023-05-31T20:21:31.721 DEBUG:boto:auth_path=/pklxkd-50/MULTIPART 2023-05-31T20:21:31.722 DEBUG:boto:Method: GET 2023-05-31T20:21:31.722 DEBUG:boto:Path: /pklxkd-50/MULTIPART 2023-05-31T20:21:31.722 DEBUG:boto:Data: 2023-05-31T20:21:31.723 DEBUG:boto:Headers: {} 2023-05-31T20:21:31.723 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:21:31.723 DEBUG:boto:Port: 8001 2023-05-31T20:21:31.724 DEBUG:boto:Params: {} 2023-05-31T20:21:31.724 DEBUG:boto:Token: None 2023-05-31T20:21:31.725 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:21:31 GMT /pklxkd-50/MULTIPART 2023-05-31T20:21:31.725 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:eEyCmv4nE2MpPLbIhtKPYvCml5w= 2023-05-31T20:21:31.726 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:21:31 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:eEyCmv4nE2MpPLbIhtKPYvCml5w=', 'Content-Length': '0'} 2023-05-31T20:21:31.728 DEBUG:boto:Response headers: [('Content-Length', '20971520'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:21:15 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"d07d411772772538cd4b657da660bfb5-4"'), ('x-amz-request-id', 'tx00000ea3e1ed47fbf6955-006477ac4b-162e-a2'), ('Content-Type', 'binary/octet-stream'), ('Date', 'Wed, 31 May 2023 20:21:31 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:21:31.828 DEBUG:boto:path=/pklxkd-50/MULTIPART 2023-05-31T20:21:31.829 DEBUG:boto:auth_path=/pklxkd-50/MULTIPART 2023-05-31T20:21:31.829 DEBUG:boto:Method: GET 2023-05-31T20:21:31.830 DEBUG:boto:Path: /pklxkd-50/MULTIPART 2023-05-31T20:21:31.830 DEBUG:boto:Data: 2023-05-31T20:21:31.830 DEBUG:boto:Headers: {} 2023-05-31T20:21:31.831 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:21:31.831 DEBUG:boto:Port: 8000 2023-05-31T20:21:31.832 DEBUG:boto:Params: {} 2023-05-31T20:21:31.832 DEBUG:boto:Token: None 2023-05-31T20:21:31.833 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:21:31 GMT /pklxkd-50/MULTIPART 2023-05-31T20:21:31.833 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:eEyCmv4nE2MpPLbIhtKPYvCml5w= 2023-05-31T20:21:31.833 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:21:31 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:eEyCmv4nE2MpPLbIhtKPYvCml5w=', 'Content-Length': '0'} 2023-05-31T20:21:31.838 DEBUG:boto:Response headers: [('Content-Length', '20971520'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:21:15 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d07d411772772538cd4b657da660bfb5-4"'), ('x-amz-request-id', 'tx00000b5f5609c2960ca92-006477ac4b-1630-a1'), ('Content-Type', 'binary/octet-stream'), ('Date', 'Wed, 31 May 2023 20:21:31 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:21:31.935 DEBUG:boto:path=/pklxkd-50/ 2023-05-31T20:21:31.936 DEBUG:boto:auth_path=/pklxkd-50/ 2023-05-31T20:21:31.936 DEBUG:boto:path=/pklxkd-50/?prefix=MULTIPART 2023-05-31T20:21:31.937 DEBUG:boto:auth_path=/pklxkd-50/?prefix=MULTIPART 2023-05-31T20:21:31.937 DEBUG:boto:Method: GET 2023-05-31T20:21:31.938 DEBUG:boto:Path: /pklxkd-50/?prefix=MULTIPART 2023-05-31T20:21:31.938 DEBUG:boto:Data: 2023-05-31T20:21:31.938 DEBUG:boto:Headers: {} 2023-05-31T20:21:31.939 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:21:31.939 DEBUG:boto:Port: 8001 2023-05-31T20:21:31.939 DEBUG:boto:Params: {} 2023-05-31T20:21:31.940 DEBUG:boto:Token: None 2023-05-31T20:21:31.940 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:21:31 GMT /pklxkd-50/ 2023-05-31T20:21:31.941 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:kbV6/FbY3Z4+6JUpovzvFctd8cE= 2023-05-31T20:21:31.941 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:21:31 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:kbV6/FbY3Z4+6JUpovzvFctd8cE=', 'Content-Length': '0'} 2023-05-31T20:21:31.944 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a72718d55304e536-006477ac4b-162e-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:21:31 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:21:31.944 DEBUG:boto:b'pklxkd-50MULTIPART1000falseMULTIPART2023-05-31T20:21:15.473Z"d07d411772772538cd4b657da660bfb5-4"20971520STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:21:31.945 DEBUG:boto:path=/pklxkd-50/ 2023-05-31T20:21:31.945 DEBUG:boto:auth_path=/pklxkd-50/ 2023-05-31T20:21:31.946 DEBUG:boto:path=/pklxkd-50/?prefix=MULTIPART 2023-05-31T20:21:31.946 DEBUG:boto:auth_path=/pklxkd-50/?prefix=MULTIPART 2023-05-31T20:21:31.946 DEBUG:boto:Method: GET 2023-05-31T20:21:31.947 DEBUG:boto:Path: /pklxkd-50/?prefix=MULTIPART 2023-05-31T20:21:31.947 DEBUG:boto:Data: 2023-05-31T20:21:31.947 DEBUG:boto:Headers: {} 2023-05-31T20:21:31.948 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:21:31.948 DEBUG:boto:Port: 8000 2023-05-31T20:21:31.949 DEBUG:boto:Params: {} 2023-05-31T20:21:31.949 DEBUG:boto:Token: None 2023-05-31T20:21:31.949 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:21:31 GMT /pklxkd-50/ 2023-05-31T20:21:31.950 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:kbV6/FbY3Z4+6JUpovzvFctd8cE= 2023-05-31T20:21:31.950 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:21:31 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:kbV6/FbY3Z4+6JUpovzvFctd8cE=', 'Content-Length': '0'} 2023-05-31T20:21:31.953 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a2df22d843574eb0-006477ac4b-1630-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:21:31 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:21:31.953 DEBUG:boto:b'pklxkd-50MULTIPART1000falseMULTIPART2023-05-31T20:21:15.473Z"d07d411772772538cd4b657da660bfb5-4"20971520STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:21:31.954 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=pklxkd-50 zones={a2, a1} 2023-05-31T20:21:31.954 INFO:tasks.rgw_multisite_tests:rgw_multi.tests.test_multipart_object_sync ... ok 2023-05-31T20:21:31.955 INFO:rgw_multi.tests:create bucket zone=a1 name=pklxkd-52 2023-05-31T20:21:31.956 DEBUG:boto:path=/pklxkd-52/ 2023-05-31T20:21:31.956 DEBUG:boto:auth_path=/pklxkd-52/ 2023-05-31T20:21:31.956 DEBUG:boto:Method: PUT 2023-05-31T20:21:31.957 DEBUG:boto:Path: /pklxkd-52/ 2023-05-31T20:21:31.957 DEBUG:boto:Data: 2023-05-31T20:21:31.957 DEBUG:boto:Headers: {} 2023-05-31T20:21:31.958 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:21:31.958 DEBUG:boto:Port: 8000 2023-05-31T20:21:31.958 DEBUG:boto:Params: {} 2023-05-31T20:21:31.959 DEBUG:boto:Token: None 2023-05-31T20:21:31.959 DEBUG:boto:StringToSign: PUT Wed, 31 May 2023 20:21:31 GMT /pklxkd-52/ 2023-05-31T20:21:31.959 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:KkRxGeOTtByY3V16EXCkKQ+Z6E4= 2023-05-31T20:21:31.960 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:21:31 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:KkRxGeOTtByY3V16EXCkKQ+Z6E4=', 'Content-Length': '0'} 2023-05-31T20:21:31.977 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx000003ad4692bb338aa5c-006477ac4b-1630-a1'), ('Content-Length', '0'), ('Date', 'Wed, 31 May 2023 20:21:31 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:21:31.977 DEBUG:boto:path=/pklxkd-52/testobj-sse-c 2023-05-31T20:21:31.978 DEBUG:boto:auth_path=/pklxkd-52/testobj-sse-c 2023-05-31T20:21:31.978 DEBUG:boto:Method: PUT 2023-05-31T20:21:31.978 DEBUG:boto:Path: /pklxkd-52/testobj-sse-c 2023-05-31T20:21:31.979 DEBUG:boto:Data: 2023-05-31T20:21:31.979 DEBUG:boto:Headers: {'x-amz-server-side-encryption-customer-algorithm': 'AES256', 'x-amz-server-side-encryption-customer-key': 'pO3upElrwuEXSoFwCfnZPdSsmt/xWeFa0N9KgDijwVs=', 'x-amz-server-side-encryption-customer-key-md5': 'DWygnHRtgiJ77HCm+1rvHw==', 'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': '3FCGuEcom6i4veFJuDiBdQ==', 'Content-Length': '512', 'Expect': '100-Continue'} 2023-05-31T20:21:31.979 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:21:31.980 DEBUG:boto:Port: 8000 2023-05-31T20:21:31.980 DEBUG:boto:Params: {} 2023-05-31T20:21:31.981 DEBUG:boto:Token: None 2023-05-31T20:21:31.981 DEBUG:boto:StringToSign: PUT 3FCGuEcom6i4veFJuDiBdQ== application/octet-stream Wed, 31 May 2023 20:21:31 GMT x-amz-server-side-encryption-customer-algorithm:AES256 x-amz-server-side-encryption-customer-key:pO3upElrwuEXSoFwCfnZPdSsmt/xWeFa0N9KgDijwVs= x-amz-server-side-encryption-customer-key-md5:DWygnHRtgiJ77HCm+1rvHw== /pklxkd-52/testobj-sse-c 2023-05-31T20:21:31.981 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:riRVW3zbKUmgoMIOt87852QFw6A= 2023-05-31T20:21:31.982 DEBUG:boto:Final headers: {'x-amz-server-side-encryption-customer-algorithm': 'AES256', 'x-amz-server-side-encryption-customer-key': 'pO3upElrwuEXSoFwCfnZPdSsmt/xWeFa0N9KgDijwVs=', 'x-amz-server-side-encryption-customer-key-md5': 'DWygnHRtgiJ77HCm+1rvHw==', 'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': '3FCGuEcom6i4veFJuDiBdQ==', 'Content-Length': '512', 'Expect': '100-Continue', 'Date': 'Wed, 31 May 2023 20:21:31 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:riRVW3zbKUmgoMIOt87852QFw6A='} 2023-05-31T20:21:32.028 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"dc5086b847289ba8b8bde149b8388175"'), ('Accept-Ranges', 'bytes'), ('x-amz-server-side-encryption-customer-algorithm', 'AES256'), ('x-amz-server-side-encryption-customer-key-MD5', 'DWygnHRtgiJ77HCm+1rvHw=='), ('x-amz-request-id', 'tx00000d9be278d3f38473d-006477ac4b-1630-a1'), ('Date', 'Wed, 31 May 2023 20:21:31 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:21:32.030 DEBUG:boto:path=/pklxkd-52/testobj-sse-kms 2023-05-31T20:21:32.031 DEBUG:boto:auth_path=/pklxkd-52/testobj-sse-kms 2023-05-31T20:21:32.032 DEBUG:boto:Method: PUT 2023-05-31T20:21:32.033 DEBUG:boto:Path: /pklxkd-52/testobj-sse-kms 2023-05-31T20:21:32.033 DEBUG:boto:Data: 2023-05-31T20:21:32.034 DEBUG:boto:Headers: {'x-amz-server-side-encryption': 'aws:kms', 'x-amz-server-side-encryption-aws-kms-key-id': 'testkey-1', 'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': '3FCGuEcom6i4veFJuDiBdQ==', 'Content-Length': '512', 'Expect': '100-Continue'} 2023-05-31T20:21:32.034 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:21:32.034 DEBUG:boto:Port: 8000 2023-05-31T20:21:32.035 DEBUG:boto:Params: {} 2023-05-31T20:21:32.035 DEBUG:boto:Token: None 2023-05-31T20:21:32.035 DEBUG:boto:StringToSign: PUT 3FCGuEcom6i4veFJuDiBdQ== application/octet-stream Wed, 31 May 2023 20:21:32 GMT x-amz-server-side-encryption:aws:kms x-amz-server-side-encryption-aws-kms-key-id:testkey-1 /pklxkd-52/testobj-sse-kms 2023-05-31T20:21:32.036 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:9Pnqeur1PMlgPoKnLxG3axpxSFQ= 2023-05-31T20:21:32.036 DEBUG:boto:Final headers: {'x-amz-server-side-encryption': 'aws:kms', 'x-amz-server-side-encryption-aws-kms-key-id': 'testkey-1', 'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': '3FCGuEcom6i4veFJuDiBdQ==', 'Content-Length': '512', 'Expect': '100-Continue', 'Date': 'Wed, 31 May 2023 20:21:32 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:9Pnqeur1PMlgPoKnLxG3axpxSFQ='} 2023-05-31T20:21:32.043 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"dc5086b847289ba8b8bde149b8388175"'), ('Accept-Ranges', 'bytes'), ('x-amz-server-side-encryption', 'aws:kms'), ('x-amz-server-side-encryption-aws-kms-key-id', 'testkey-1'), ('x-amz-server-side-encryption-context', 'eyJhd3M6czM6YXJuIjoiYXJuOmF3czpzMzo6cmd3LW11bHRpc2l0ZS10ZXN0LXVzZXI6cGtseGtkLTUyL3Rlc3RvYmotc3NlLWttcyJ9'), ('x-amz-request-id', 'tx00000d0ac52938014bfab-006477ac4c-1630-a1'), ('Date', 'Wed, 31 May 2023 20:21:32 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:21:32.043 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-31T20:21:32.044 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-31T20:21:32.044 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-05-31T20:21:32.098 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2023-05-31T20:21:32.099 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2023-05-31T20:21:32.138 INFO:teuthology.orchestra.run.smithi099.stderr:No --period given, using current period=2e6123d2-d695-4350-9a01-61a455bc782b 2023-05-31T20:21:32.167 INFO:teuthology.orchestra.run.smithi099.stdout:[ 2023-05-31T20:21:32.168 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T20:21:32.168 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1685562631.516089_59.1", 2023-05-31T20:21:32.168 INFO:teuthology.orchestra.run.smithi099.stdout: "last_update": "2023-05-31T19:50:31.516089Z" 2023-05-31T20:21:32.169 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:21:32.169 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T20:21:32.169 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "", 2023-05-31T20:21:32.169 INFO:teuthology.orchestra.run.smithi099.stdout: "last_update": "0.000000" 2023-05-31T20:21:32.170 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:21:32.170 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T20:21:32.170 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1685562970.274572_93.1", 2023-05-31T20:21:32.171 INFO:teuthology.orchestra.run.smithi099.stdout: "last_update": "2023-05-31T19:56:10.274572Z" 2023-05-31T20:21:32.171 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:21:32.171 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T20:21:32.172 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1685564491.970984_346.1", 2023-05-31T20:21:32.172 INFO:teuthology.orchestra.run.smithi099.stdout: "last_update": "2023-05-31T20:21:31.970984Z" 2023-05-31T20:21:32.172 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T20:21:32.173 INFO:teuthology.orchestra.run.smithi099.stdout:] 2023-05-31T20:21:32.175 DEBUG:rgw_multi.tests:master meta markers={0: '1_1685562631.516089_59.1', 1: '', 2: '1_1685562970.274572_93.1', 3: '1_1685564491.970984_346.1'} 2023-05-31T20:21:32.176 INFO:rgw_multi.tests:starting meta checkpoint for zone=a2 2023-05-31T20:21:32.176 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-31T20:21:32.177 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-31T20:21:32.177 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-05-31T20:21:32.226 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2023-05-31T20:21:32.226 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2023-05-31T20:21:32.273 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:21:32.265+0000 7fbcfed50780 20 RGW-SYNC:meta: read sync status 2023-05-31T20:21:32.273 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:21:32.266+0000 7fbcfed50780 20 RGW-SYNC:meta: read sync status 2023-05-31T20:21:32.287 INFO:teuthology.orchestra.run.smithi099.stdout:{ 2023-05-31T20:21:32.287 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_status": { 2023-05-31T20:21:32.287 INFO:teuthology.orchestra.run.smithi099.stdout: "info": { 2023-05-31T20:21:32.287 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "sync", 2023-05-31T20:21:32.288 INFO:teuthology.orchestra.run.smithi099.stdout: "num_shards": 4, 2023-05-31T20:21:32.288 INFO:teuthology.orchestra.run.smithi099.stdout: "period": "2e6123d2-d695-4350-9a01-61a455bc782b", 2023-05-31T20:21:32.288 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2023-05-31T20:21:32.289 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:21:32.289 INFO:teuthology.orchestra.run.smithi099.stdout: "markers": [ 2023-05-31T20:21:32.289 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T20:21:32.289 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 0, 2023-05-31T20:21:32.289 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T20:21:32.289 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2023-05-31T20:21:32.289 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1685562631.516089_59.1", 2023-05-31T20:21:32.290 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T20:21:32.290 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 1, 2023-05-31T20:21:32.290 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T20:21:32.290 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2023-05-31T19:50:31.516089Z", 2023-05-31T20:21:32.290 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2023-05-31T20:21:32.290 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T20:21:32.291 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:21:32.291 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T20:21:32.291 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 1, 2023-05-31T20:21:32.291 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T20:21:32.291 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2023-05-31T20:21:32.291 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "", 2023-05-31T20:21:32.291 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T20:21:32.292 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2023-05-31T20:21:32.292 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T20:21:32.292 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "0.000000", 2023-05-31T20:21:32.292 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2023-05-31T20:21:32.292 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T20:21:32.292 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:21:32.292 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T20:21:32.293 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 2, 2023-05-31T20:21:32.293 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T20:21:32.293 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2023-05-31T20:21:32.293 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1685562970.274572_93.1", 2023-05-31T20:21:32.293 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T20:21:32.293 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2023-05-31T20:21:32.294 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T20:21:32.295 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2023-05-31T19:56:10.274572Z", 2023-05-31T20:21:32.295 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2023-05-31T20:21:32.295 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T20:21:32.295 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:21:32.295 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T20:21:32.296 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 3, 2023-05-31T20:21:32.296 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T20:21:32.296 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2023-05-31T20:21:32.297 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1685564491.970984_346.1", 2023-05-31T20:21:32.297 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T20:21:32.297 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2023-05-31T20:21:32.297 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T20:21:32.297 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2023-05-31T20:21:31.970984Z", 2023-05-31T20:21:32.298 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2023-05-31T20:21:32.298 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T20:21:32.298 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T20:21:32.298 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2023-05-31T20:21:32.298 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:21:32.298 INFO:teuthology.orchestra.run.smithi099.stdout: "full_sync": { 2023-05-31T20:21:32.298 INFO:teuthology.orchestra.run.smithi099.stdout: "total": 1, 2023-05-31T20:21:32.299 INFO:teuthology.orchestra.run.smithi099.stdout: "complete": 1 2023-05-31T20:21:32.299 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:21:32.299 INFO:teuthology.orchestra.run.smithi099.stdout: "current_time": "2023-05-31T20:21:32Z" 2023-05-31T20:21:32.299 INFO:teuthology.orchestra.run.smithi099.stdout:} 2023-05-31T20:21:32.301 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "2e6123d2-d695-4350-9a01-61a455bc782b", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1685562631.516089_59.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-05-31T19:50:31.516089Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1685562970.274572_93.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-05-31T19:56:10.274572Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1685564491.970984_346.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-05-31T20:21:31.970984Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-05-31T20:21:32Z" } 2023-05-31T20:21:32.302 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1685562631.516089_59.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-05-31T19:50:31.516089Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1685562970.274572_93.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-05-31T19:56:10.274572Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1685564491.970984_346.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-05-31T20:21:31.970984Z', 'realm_epoch': 2}}] 2023-05-31T20:21:32.302 DEBUG:rgw_multi.tests:log_status={0: '1_1685562631.516089_59.1', 1: '', 2: '1_1685562970.274572_93.1', 3: '1_1685564491.970984_346.1'} 2023-05-31T20:21:32.303 DEBUG:rgw_multi.tests:sync_status={0: '1_1685562631.516089_59.1', 1: '', 2: '1_1685562970.274572_93.1', 3: '1_1685564491.970984_346.1'} 2023-05-31T20:21:32.303 INFO:rgw_multi.tests:finish meta checkpoint for zone=a2 2023-05-31T20:21:32.303 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'pklxkd-52', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-05-31T20:21:32.304 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'pklxkd-52', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-05-31T20:21:32.304 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket pklxkd-52 --source-zone a1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2023-05-31T20:21:32.362 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2023-05-31T20:21:32.363 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2023-05-31T20:21:32.402 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:21:32.397+0000 7f0d5e99f780 1 waiting to reach incremental sync.. 2023-05-31T20:21:37.403 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:21:37.398+0000 7f0d5e99f780 1 waiting to reach incremental sync.. 2023-05-31T20:21:42.403 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:21:42.399+0000 7f0d5e99f780 1 waiting to reach incremental sync.. 2023-05-31T20:21:47.404 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:21:47.399+0000 7f0d5e99f780 1 waiting to reach incremental sync.. 2023-05-31T20:21:52.404 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:21:52.399+0000 7f0d5e99f780 1 waiting to reach incremental sync.. 2023-05-31T20:21:57.405 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:21:57.400+0000 7f0d5e99f780 1 waiting to reach incremental sync.. 2023-05-31T20:22:02.405 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:22:02.401+0000 7f0d5e99f780 1 waiting to reach incremental sync.. 2023-05-31T20:22:07.406 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:22:07.400+0000 7f0d5e99f780 1 waiting to reach incremental sync.. 2023-05-31T20:22:12.415 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:22:12.401+0000 7f0d5e99f780 1 waiting to reach incremental sync.. 2023-05-31T20:22:17.408 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:22:17.402+0000 7f0d5e99f780 1 waiting to reach incremental sync.. 2023-05-31T20:22:22.408 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:22:22.403+0000 7f0d5e99f780 1 waiting to reach incremental sync.. 2023-05-31T20:22:27.408 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:22:27.404+0000 7f0d5e99f780 1 waiting to reach incremental sync.. 2023-05-31T20:22:32.409 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:22:32.404+0000 7f0d5e99f780 1 waiting to reach incremental sync.. 2023-05-31T20:22:37.410 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:22:37.405+0000 7f0d5e99f780 1 waiting to reach incremental sync.. 2023-05-31T20:22:42.413 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:22:42.405+0000 7f0d5e99f780 1 waiting to reach incremental sync.. 2023-05-31T20:22:47.411 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:22:47.405+0000 7f0d5e99f780 1 waiting to reach incremental sync.. 2023-05-31T20:22:52.411 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:22:52.406+0000 7f0d5e99f780 1 waiting to reach incremental sync.. 2023-05-31T20:22:57.412 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:22:57.406+0000 7f0d5e99f780 1 waiting to reach incremental sync.. 2023-05-31T20:23:02.412 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:23:02.407+0000 7f0d5e99f780 1 waiting to reach incremental sync.. 2023-05-31T20:23:07.414 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:23:07.408+0000 7f0d5e99f780 1 waiting to reach incremental sync.. 2023-05-31T20:23:12.414 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:23:12.409+0000 7f0d5e99f780 1 waiting to reach incremental sync.. 2023-05-31T20:23:17.414 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:23:17.410+0000 7f0d5e99f780 1 waiting to reach incremental sync.. 2023-05-31T20:23:20.059 INFO:tasks.rgw.c1.client.0.smithi099.stdout:tcmalloc: large alloc 1164263424 bytes == 0x55a082e18000 @ 0x7f3b20c7d760 0x7f3b20c9f464 0x7f3b20c9f808 0x7f3b247760fa 0x7f3b24776287 0x7f3b247762d7 0x7f3b2477632e 0x7f39e1dbfff2 0x55a077d38029 0x55a0780073e4 0x55a077fe7a46 0x55a0780296f6 0x55a078029f58 0x55a077dcf83b 0x55a077bc15cc 0x55a077bc4ce4 0x55a077b00516 0x55a077b011e4 0x55a07894e53f 2023-05-31T20:23:22.415 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:23:22.411+0000 7f0d5e99f780 1 waiting to reach incremental sync.. 2023-05-31T20:23:27.416 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:23:27.411+0000 7f0d5e99f780 1 waiting to reach incremental sync.. 2023-05-31T20:23:32.417 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:23:32.411+0000 7f0d5e99f780 1 waiting to reach incremental sync.. 2023-05-31T20:23:37.417 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:23:37.412+0000 7f0d5e99f780 1 waiting to reach incremental sync.. 2023-05-31T20:23:42.418 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:23:42.412+0000 7f0d5e99f780 1 waiting to reach incremental sync.. 2023-05-31T20:23:47.419 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:23:47.413+0000 7f0d5e99f780 1 waiting to reach incremental sync.. 2023-05-31T20:23:52.419 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:23:52.415+0000 7f0d5e99f780 1 waiting to reach incremental sync.. 2023-05-31T20:23:57.420 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:23:57.416+0000 7f0d5e99f780 1 waiting to reach incremental sync.. 2023-05-31T20:24:02.421 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:24:02.416+0000 7f0d5e99f780 1 waiting to reach incremental sync.. 2023-05-31T20:24:07.421 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:24:07.417+0000 7f0d5e99f780 1 waiting to reach incremental sync.. 2023-05-31T20:24:12.422 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:24:12.416+0000 7f0d5e99f780 1 waiting to reach incremental sync.. 2023-05-31T20:24:17.422 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:24:17.417+0000 7f0d5e99f780 1 waiting to reach incremental sync.. 2023-05-31T20:24:22.423 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:24:22.418+0000 7f0d5e99f780 1 waiting to reach incremental sync.. 2023-05-31T20:24:27.424 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:24:27.418+0000 7f0d5e99f780 1 waiting to reach incremental sync.. 2023-05-31T20:24:32.424 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:24:32.419+0000 7f0d5e99f780 1 waiting to reach incremental sync.. 2023-05-31T20:24:37.425 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:24:37.420+0000 7f0d5e99f780 1 waiting to reach incremental sync.. 2023-05-31T20:24:42.425 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:24:42.421+0000 7f0d5e99f780 1 waiting to reach incremental sync.. 2023-05-31T20:24:47.426 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:24:47.421+0000 7f0d5e99f780 1 waiting to reach incremental sync.. 2023-05-31T20:24:52.427 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:24:52.422+0000 7f0d5e99f780 1 waiting to reach incremental sync.. 2023-05-31T20:24:57.428 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:24:57.422+0000 7f0d5e99f780 1 waiting to reach incremental sync.. 2023-05-31T20:25:02.428 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:25:02.423+0000 7f0d5e99f780 1 waiting to reach incremental sync.. 2023-05-31T20:25:07.429 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:25:07.424+0000 7f0d5e99f780 1 waiting to reach incremental sync.. 2023-05-31T20:25:12.430 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:25:12.424+0000 7f0d5e99f780 1 waiting to reach incremental sync.. 2023-05-31T20:25:17.430 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:25:17.425+0000 7f0d5e99f780 1 waiting to reach incremental sync.. 2023-05-31T20:25:22.431 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:25:22.426+0000 7f0d5e99f780 1 waiting to reach incremental sync.. 2023-05-31T20:25:27.431 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:25:27.427+0000 7f0d5e99f780 1 waiting to reach incremental sync.. 2023-05-31T20:25:32.432 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:25:32.427+0000 7f0d5e99f780 1 waiting to reach incremental sync.. 2023-05-31T20:25:37.433 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:25:37.428+0000 7f0d5e99f780 1 waiting to reach incremental sync.. 2023-05-31T20:25:42.433 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:25:42.428+0000 7f0d5e99f780 1 waiting to reach incremental sync.. 2023-05-31T20:25:47.434 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:25:47.428+0000 7f0d5e99f780 1 waiting to reach incremental sync.. 2023-05-31T20:25:52.435 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:25:52.429+0000 7f0d5e99f780 1 waiting to reach incremental sync.. 2023-05-31T20:25:57.435 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:25:57.430+0000 7f0d5e99f780 1 waiting to reach incremental sync.. 2023-05-31T20:26:02.436 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:26:02.431+0000 7f0d5e99f780 1 waiting to reach incremental sync.. 2023-05-31T20:26:07.436 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:26:07.432+0000 7f0d5e99f780 1 waiting to reach incremental sync.. 2023-05-31T20:26:12.437 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:26:12.432+0000 7f0d5e99f780 1 waiting to reach incremental sync.. 2023-05-31T20:26:17.438 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:26:17.433+0000 7f0d5e99f780 1 waiting to reach incremental sync.. 2023-05-31T20:26:22.438 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:26:22.433+0000 7f0d5e99f780 1 waiting to reach incremental sync.. 2023-05-31T20:26:27.439 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:26:27.434+0000 7f0d5e99f780 -1 bucket checkpoint timed out waiting to reach incremental sync 2023-05-31T20:26:27.440 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:26:27.434+0000 7f0d5e99f780 0 bucket sync checkpoint failed: (110) Connection timed out 2023-05-31T20:26:27.440 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:26:27.434+0000 7f0d5e99f780 -1 bucket sync checkpoint failed: (110) Connection timed out 2023-05-31T20:26:27.461 DEBUG:teuthology.orchestra.run:got remote process result: 110 2023-05-31T20:26:27.463 INFO:tasks.rgw_multisite_tests:rgw_multi.tests.test_encrypted_object_sync ... FAIL 2023-05-31T20:26:27.464 INFO:rgw_multi.tests:create bucket zone=a1 name=pklxkd-53 2023-05-31T20:26:27.465 DEBUG:boto:path=/pklxkd-53/ 2023-05-31T20:26:27.465 DEBUG:boto:auth_path=/pklxkd-53/ 2023-05-31T20:26:27.465 DEBUG:boto:Method: PUT 2023-05-31T20:26:27.466 DEBUG:boto:Path: /pklxkd-53/ 2023-05-31T20:26:27.466 DEBUG:boto:Data: 2023-05-31T20:26:27.466 DEBUG:boto:Headers: {} 2023-05-31T20:26:27.467 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:26:27.467 DEBUG:boto:Port: 8000 2023-05-31T20:26:27.467 DEBUG:boto:Params: {} 2023-05-31T20:26:27.468 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2023-05-31T20:26:27.468 DEBUG:boto:Token: None 2023-05-31T20:26:27.469 DEBUG:boto:StringToSign: PUT Wed, 31 May 2023 20:26:27 GMT /pklxkd-53/ 2023-05-31T20:26:27.469 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:CY00FxcRpCp5KH36b06J/kLqxYE= 2023-05-31T20:26:27.470 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:26:27 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:CY00FxcRpCp5KH36b06J/kLqxYE=', 'Content-Length': '0'} 2023-05-31T20:26:27.499 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000ccede962fb36824f-006477ad73-1630-a1'), ('Content-Length', '0'), ('Date', 'Wed, 31 May 2023 20:26:27 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:26:27.500 DEBUG:boto:path=/pklxkd-53/ 2023-05-31T20:26:27.500 DEBUG:boto:auth_path=/pklxkd-53/ 2023-05-31T20:26:27.500 DEBUG:boto:Method: HEAD 2023-05-31T20:26:27.501 DEBUG:boto:Path: /pklxkd-53/ 2023-05-31T20:26:27.501 DEBUG:boto:Data: 2023-05-31T20:26:27.502 DEBUG:boto:Headers: {} 2023-05-31T20:26:27.502 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:26:27.502 DEBUG:boto:Port: 8000 2023-05-31T20:26:27.503 DEBUG:boto:Params: {} 2023-05-31T20:26:27.503 DEBUG:boto:Token: None 2023-05-31T20:26:27.503 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:26:27 GMT /pklxkd-53/ 2023-05-31T20:26:27.504 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:0l2TGne7msTiCqophPyXFhfgjvg= 2023-05-31T20:26:27.504 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:26:27 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:0l2TGne7msTiCqophPyXFhfgjvg=', 'Content-Length': '0'} 2023-05-31T20:26:27.507 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000dce7ec7f0912ccca-006477ad73-1630-a1'), ('Content-Length', '0'), ('Date', 'Wed, 31 May 2023 20:26:27 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:26:27.507 DEBUG:boto:path=/pklxkd-53/a 2023-05-31T20:26:27.508 DEBUG:boto:auth_path=/pklxkd-53/a 2023-05-31T20:26:27.508 DEBUG:boto:Method: PUT 2023-05-31T20:26:27.508 DEBUG:boto:Path: /pklxkd-53/a 2023-05-31T20:26:27.509 DEBUG:boto:Data: 2023-05-31T20:26:27.509 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-05-31T20:26:27.509 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:26:27.510 DEBUG:boto:Port: 8000 2023-05-31T20:26:27.510 DEBUG:boto:Params: {} 2023-05-31T20:26:27.511 DEBUG:boto:Token: None 2023-05-31T20:26:27.511 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Wed, 31 May 2023 20:26:27 GMT /pklxkd-53/a 2023-05-31T20:26:27.511 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:F+kZuHd7tQsEhf02JVrBrhPx6Do= 2023-05-31T20:26:27.512 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Wed, 31 May 2023 20:26:27 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:F+kZuHd7tQsEhf02JVrBrhPx6Do='} 2023-05-31T20:26:27.518 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000e631813e5f75d42c-006477ad73-1630-a1'), ('Date', 'Wed, 31 May 2023 20:26:27 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:26:27.519 DEBUG:boto:path=/pklxkd-53/ 2023-05-31T20:26:27.519 DEBUG:boto:auth_path=/pklxkd-53/ 2023-05-31T20:26:27.520 DEBUG:boto:Method: HEAD 2023-05-31T20:26:27.520 DEBUG:boto:Path: /pklxkd-53/ 2023-05-31T20:26:27.520 DEBUG:boto:Data: 2023-05-31T20:26:27.521 DEBUG:boto:Headers: {} 2023-05-31T20:26:27.521 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:26:27.521 DEBUG:boto:Port: 8000 2023-05-31T20:26:27.522 DEBUG:boto:Params: {} 2023-05-31T20:26:27.522 DEBUG:boto:Token: None 2023-05-31T20:26:27.523 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:26:27 GMT /pklxkd-53/ 2023-05-31T20:26:27.523 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:0l2TGne7msTiCqophPyXFhfgjvg= 2023-05-31T20:26:27.523 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:26:27 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:0l2TGne7msTiCqophPyXFhfgjvg=', 'Content-Length': '0'} 2023-05-31T20:26:27.526 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '3'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000f2884eeb5bcd1d7c-006477ad73-1630-a1'), ('Content-Length', '0'), ('Date', 'Wed, 31 May 2023 20:26:27 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:26:27.526 DEBUG:boto:path=/pklxkd-53/b 2023-05-31T20:26:27.527 DEBUG:boto:auth_path=/pklxkd-53/b 2023-05-31T20:26:27.527 DEBUG:boto:Method: PUT 2023-05-31T20:26:27.528 DEBUG:boto:Path: /pklxkd-53/b 2023-05-31T20:26:27.528 DEBUG:boto:Data: 2023-05-31T20:26:27.528 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-05-31T20:26:27.529 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:26:27.529 DEBUG:boto:Port: 8000 2023-05-31T20:26:27.529 DEBUG:boto:Params: {} 2023-05-31T20:26:27.530 DEBUG:boto:Token: None 2023-05-31T20:26:27.530 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Wed, 31 May 2023 20:26:27 GMT /pklxkd-53/b 2023-05-31T20:26:27.531 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:lk08WyTTctRhMgv4gMdcOT9KlVI= 2023-05-31T20:26:27.532 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Wed, 31 May 2023 20:26:27 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:lk08WyTTctRhMgv4gMdcOT9KlVI='} 2023-05-31T20:26:27.538 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000005a30965aeefd6a69-006477ad73-1630-a1'), ('Date', 'Wed, 31 May 2023 20:26:27 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:26:27.538 DEBUG:boto:path=/pklxkd-53/ 2023-05-31T20:26:27.539 DEBUG:boto:auth_path=/pklxkd-53/ 2023-05-31T20:26:27.539 DEBUG:boto:Method: HEAD 2023-05-31T20:26:27.540 DEBUG:boto:Path: /pklxkd-53/ 2023-05-31T20:26:27.540 DEBUG:boto:Data: 2023-05-31T20:26:27.540 DEBUG:boto:Headers: {} 2023-05-31T20:26:27.541 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:26:27.541 DEBUG:boto:Port: 8000 2023-05-31T20:26:27.541 DEBUG:boto:Params: {} 2023-05-31T20:26:27.542 DEBUG:boto:Token: None 2023-05-31T20:26:27.542 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:26:27 GMT /pklxkd-53/ 2023-05-31T20:26:27.542 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:0l2TGne7msTiCqophPyXFhfgjvg= 2023-05-31T20:26:27.543 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:26:27 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:0l2TGne7msTiCqophPyXFhfgjvg=', 'Content-Length': '0'} 2023-05-31T20:26:27.545 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '2'), ('X-RGW-Bytes-Used', '6'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000b48bd60a3ef604eb-006477ad73-1630-a1'), ('Content-Length', '0'), ('Date', 'Wed, 31 May 2023 20:26:27 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:26:27.546 DEBUG:boto:path=/pklxkd-53/c 2023-05-31T20:26:27.546 DEBUG:boto:auth_path=/pklxkd-53/c 2023-05-31T20:26:27.546 DEBUG:boto:Method: PUT 2023-05-31T20:26:27.547 DEBUG:boto:Path: /pklxkd-53/c 2023-05-31T20:26:27.547 DEBUG:boto:Data: 2023-05-31T20:26:27.547 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-05-31T20:26:27.548 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:26:27.548 DEBUG:boto:Port: 8000 2023-05-31T20:26:27.548 DEBUG:boto:Params: {} 2023-05-31T20:26:27.549 DEBUG:boto:Token: None 2023-05-31T20:26:27.549 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Wed, 31 May 2023 20:26:27 GMT /pklxkd-53/c 2023-05-31T20:26:27.550 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:7tOcD+UlV2YTVsFd2XXEM7dU0xA= 2023-05-31T20:26:27.550 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Wed, 31 May 2023 20:26:27 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:7tOcD+UlV2YTVsFd2XXEM7dU0xA='} 2023-05-31T20:26:27.592 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000891cb6e853e7ef3e-006477ad73-1630-a1'), ('Date', 'Wed, 31 May 2023 20:26:27 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:26:27.593 DEBUG:boto:path=/pklxkd-53/ 2023-05-31T20:26:27.593 DEBUG:boto:auth_path=/pklxkd-53/ 2023-05-31T20:26:27.594 DEBUG:boto:Method: HEAD 2023-05-31T20:26:27.594 DEBUG:boto:Path: /pklxkd-53/ 2023-05-31T20:26:27.594 DEBUG:boto:Data: 2023-05-31T20:26:27.595 DEBUG:boto:Headers: {} 2023-05-31T20:26:27.595 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:26:27.595 DEBUG:boto:Port: 8000 2023-05-31T20:26:27.596 DEBUG:boto:Params: {} 2023-05-31T20:26:27.596 DEBUG:boto:Token: None 2023-05-31T20:26:27.597 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:26:27 GMT /pklxkd-53/ 2023-05-31T20:26:27.597 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:0l2TGne7msTiCqophPyXFhfgjvg= 2023-05-31T20:26:27.597 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:26:27 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:0l2TGne7msTiCqophPyXFhfgjvg=', 'Content-Length': '0'} 2023-05-31T20:26:27.599 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '3'), ('X-RGW-Bytes-Used', '9'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000250dae12b6087226-006477ad73-1630-a1'), ('Content-Length', '0'), ('Date', 'Wed, 31 May 2023 20:26:27 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:26:27.600 DEBUG:boto:path=/pklxkd-53/d 2023-05-31T20:26:27.600 DEBUG:boto:auth_path=/pklxkd-53/d 2023-05-31T20:26:27.601 DEBUG:boto:Method: PUT 2023-05-31T20:26:27.601 DEBUG:boto:Path: /pklxkd-53/d 2023-05-31T20:26:27.601 DEBUG:boto:Data: 2023-05-31T20:26:27.602 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-05-31T20:26:27.602 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:26:27.602 DEBUG:boto:Port: 8000 2023-05-31T20:26:27.603 DEBUG:boto:Params: {} 2023-05-31T20:26:27.603 DEBUG:boto:Token: None 2023-05-31T20:26:27.604 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Wed, 31 May 2023 20:26:27 GMT /pklxkd-53/d 2023-05-31T20:26:27.604 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:jXdLUJQ2tw4QSOV7h5VmBolEwps= 2023-05-31T20:26:27.604 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Wed, 31 May 2023 20:26:27 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:jXdLUJQ2tw4QSOV7h5VmBolEwps='} 2023-05-31T20:26:27.612 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000368ac000445c7ce3-006477ad73-1630-a1'), ('Date', 'Wed, 31 May 2023 20:26:27 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:26:27.612 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-31T20:26:27.613 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-31T20:26:27.613 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-05-31T20:26:27.667 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2023-05-31T20:26:27.667 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2023-05-31T20:26:27.717 INFO:teuthology.orchestra.run.smithi099.stderr:No --period given, using current period=2e6123d2-d695-4350-9a01-61a455bc782b 2023-05-31T20:26:27.738 INFO:teuthology.orchestra.run.smithi099.stdout:[ 2023-05-31T20:26:27.738 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T20:26:27.739 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1685562631.516089_59.1", 2023-05-31T20:26:27.739 INFO:teuthology.orchestra.run.smithi099.stdout: "last_update": "2023-05-31T19:50:31.516089Z" 2023-05-31T20:26:27.739 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:26:27.739 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T20:26:27.739 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "", 2023-05-31T20:26:27.739 INFO:teuthology.orchestra.run.smithi099.stdout: "last_update": "0.000000" 2023-05-31T20:26:27.739 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:26:27.740 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T20:26:27.740 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1685562970.274572_93.1", 2023-05-31T20:26:27.740 INFO:teuthology.orchestra.run.smithi099.stdout: "last_update": "2023-05-31T19:56:10.274572Z" 2023-05-31T20:26:27.740 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:26:27.740 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T20:26:27.740 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1685564787.490753_350.1", 2023-05-31T20:26:27.741 INFO:teuthology.orchestra.run.smithi099.stdout: "last_update": "2023-05-31T20:26:27.490753Z" 2023-05-31T20:26:27.741 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T20:26:27.741 INFO:teuthology.orchestra.run.smithi099.stdout:] 2023-05-31T20:26:27.742 DEBUG:rgw_multi.tests:master meta markers={0: '1_1685562631.516089_59.1', 1: '', 2: '1_1685562970.274572_93.1', 3: '1_1685564787.490753_350.1'} 2023-05-31T20:26:27.742 INFO:rgw_multi.tests:starting meta checkpoint for zone=a2 2023-05-31T20:26:27.743 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-31T20:26:27.743 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-31T20:26:27.743 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-05-31T20:26:27.792 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2023-05-31T20:26:27.793 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2023-05-31T20:26:27.823 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:26:27.817+0000 7f86c071a780 20 RGW-SYNC:meta: read sync status 2023-05-31T20:26:27.823 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:26:27.818+0000 7f86c071a780 20 RGW-SYNC:meta: read sync status 2023-05-31T20:26:27.836 INFO:teuthology.orchestra.run.smithi099.stdout:{ 2023-05-31T20:26:27.837 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_status": { 2023-05-31T20:26:27.837 INFO:teuthology.orchestra.run.smithi099.stdout: "info": { 2023-05-31T20:26:27.837 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "sync", 2023-05-31T20:26:27.837 INFO:teuthology.orchestra.run.smithi099.stdout: "num_shards": 4, 2023-05-31T20:26:27.837 INFO:teuthology.orchestra.run.smithi099.stdout: "period": "2e6123d2-d695-4350-9a01-61a455bc782b", 2023-05-31T20:26:27.837 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2023-05-31T20:26:27.838 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:26:27.838 INFO:teuthology.orchestra.run.smithi099.stdout: "markers": [ 2023-05-31T20:26:27.838 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T20:26:27.839 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 0, 2023-05-31T20:26:27.839 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T20:26:27.839 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2023-05-31T20:26:27.840 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1685562631.516089_59.1", 2023-05-31T20:26:27.840 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T20:26:27.840 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 1, 2023-05-31T20:26:27.840 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T20:26:27.840 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2023-05-31T19:50:31.516089Z", 2023-05-31T20:26:27.840 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2023-05-31T20:26:27.841 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T20:26:27.841 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:26:27.841 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T20:26:27.841 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 1, 2023-05-31T20:26:27.841 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T20:26:27.841 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2023-05-31T20:26:27.841 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "", 2023-05-31T20:26:27.842 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T20:26:27.842 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2023-05-31T20:26:27.842 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T20:26:27.842 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "0.000000", 2023-05-31T20:26:27.842 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2023-05-31T20:26:27.842 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T20:26:27.842 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:26:27.843 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T20:26:27.843 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 2, 2023-05-31T20:26:27.843 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T20:26:27.843 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2023-05-31T20:26:27.843 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1685562970.274572_93.1", 2023-05-31T20:26:27.843 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T20:26:27.843 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2023-05-31T20:26:27.844 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T20:26:27.844 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2023-05-31T19:56:10.274572Z", 2023-05-31T20:26:27.844 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2023-05-31T20:26:27.844 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T20:26:27.844 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:26:27.844 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T20:26:27.844 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 3, 2023-05-31T20:26:27.845 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T20:26:27.845 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2023-05-31T20:26:27.845 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1685564787.490753_350.1", 2023-05-31T20:26:27.845 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T20:26:27.845 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2023-05-31T20:26:27.845 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T20:26:27.846 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2023-05-31T20:26:27.490753Z", 2023-05-31T20:26:27.846 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2023-05-31T20:26:27.846 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T20:26:27.846 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T20:26:27.846 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2023-05-31T20:26:27.846 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:26:27.846 INFO:teuthology.orchestra.run.smithi099.stdout: "full_sync": { 2023-05-31T20:26:27.847 INFO:teuthology.orchestra.run.smithi099.stdout: "total": 1, 2023-05-31T20:26:27.847 INFO:teuthology.orchestra.run.smithi099.stdout: "complete": 1 2023-05-31T20:26:27.847 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:26:27.847 INFO:teuthology.orchestra.run.smithi099.stdout: "current_time": "2023-05-31T20:26:27Z" 2023-05-31T20:26:27.847 INFO:teuthology.orchestra.run.smithi099.stdout:} 2023-05-31T20:26:27.848 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "2e6123d2-d695-4350-9a01-61a455bc782b", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1685562631.516089_59.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-05-31T19:50:31.516089Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1685562970.274572_93.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-05-31T19:56:10.274572Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1685564787.490753_350.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-05-31T20:26:27.490753Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-05-31T20:26:27Z" } 2023-05-31T20:26:27.849 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1685562631.516089_59.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-05-31T19:50:31.516089Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1685562970.274572_93.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-05-31T19:56:10.274572Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1685564787.490753_350.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-05-31T20:26:27.490753Z', 'realm_epoch': 2}}] 2023-05-31T20:26:27.849 DEBUG:rgw_multi.tests:log_status={0: '1_1685562631.516089_59.1', 1: '', 2: '1_1685562970.274572_93.1', 3: '1_1685564787.490753_350.1'} 2023-05-31T20:26:27.849 DEBUG:rgw_multi.tests:sync_status={0: '1_1685562631.516089_59.1', 1: '', 2: '1_1685562970.274572_93.1', 3: '1_1685564787.490753_350.1'} 2023-05-31T20:26:27.850 INFO:rgw_multi.tests:finish meta checkpoint for zone=a2 2023-05-31T20:26:27.850 DEBUG:rgw_multi.tests:bucket checkpoint: source=a1 target=a2 bucket=pklxkd-53 2023-05-31T20:26:27.850 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'pklxkd-53', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-05-31T20:26:27.851 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'pklxkd-53', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-05-31T20:26:27.851 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket pklxkd-53 --source-zone a1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2023-05-31T20:26:27.908 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2023-05-31T20:26:27.908 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2023-05-31T20:26:27.950 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:26:27.944+0000 7f9d962f1780 1 waiting to reach incremental sync.. 2023-05-31T20:26:32.950 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:26:32.945+0000 7f9d962f1780 1 waiting to reach incremental sync.. 2023-05-31T20:26:37.951 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:26:37.946+0000 7f9d962f1780 1 waiting to reach incremental sync.. 2023-05-31T20:26:42.953 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:26:42.947+0000 7f9d962f1780 1 bucket sync caught up with source: 2023-05-31T20:26:42.954 INFO:teuthology.orchestra.run.smithi099.stderr: local status: [, 00000000001.24.6, , , , 00000000001.35.6, , , 00000000001.21.6, , 00000000001.26.6] 2023-05-31T20:26:42.954 INFO:teuthology.orchestra.run.smithi099.stderr: remote markers: [, 00000000001.24.6, , , , 00000000001.35.6, , , 00000000001.21.6, , 00000000001.26.6] 2023-05-31T20:26:42.954 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:26:42.947+0000 7f9d962f1780 0 bucket checkpoint complete 2023-05-31T20:26:42.975 DEBUG:rgw_multi.tests:bucket checkpoint: source=a2 target=a1 bucket=pklxkd-53 2023-05-31T20:26:42.977 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'pklxkd-53', '--source-zone', 'a2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-05-31T20:26:42.977 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'pklxkd-53', '--source-zone', 'a2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-05-31T20:26:42.978 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket pklxkd-53 --source-zone a2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2023-05-31T20:26:43.042 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2023-05-31T20:26:43.043 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2023-05-31T20:26:43.101 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:26:43.095+0000 7fc870ae2780 1 bucket sync caught up with source: 2023-05-31T20:26:43.101 INFO:teuthology.orchestra.run.smithi099.stderr: local status: [, 00000000001.1905.6, , , , 00000000001.3550.6, , , 00000000001.1056.6, , 00000000001.2587.6] 2023-05-31T20:26:43.101 INFO:teuthology.orchestra.run.smithi099.stderr: remote markers: [, 00000000001.1905.6, , , , 00000000001.3550.6, , , 00000000001.1056.6, , 00000000001.2587.6] 2023-05-31T20:26:43.101 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:26:43.095+0000 7fc870ae2780 0 bucket checkpoint complete 2023-05-31T20:26:43.118 INFO:tasks.rgw_multi.zone_rados:comparing bucket=pklxkd-53 zones={a2, a1} 2023-05-31T20:26:43.119 DEBUG:boto:path=/pklxkd-53/ 2023-05-31T20:26:43.119 DEBUG:boto:auth_path=/pklxkd-53/ 2023-05-31T20:26:43.120 DEBUG:boto:Method: HEAD 2023-05-31T20:26:43.120 DEBUG:boto:Path: /pklxkd-53/ 2023-05-31T20:26:43.120 DEBUG:boto:Data: 2023-05-31T20:26:43.121 DEBUG:boto:Headers: {} 2023-05-31T20:26:43.121 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:26:43.121 DEBUG:boto:Port: 8001 2023-05-31T20:26:43.122 DEBUG:boto:Params: {} 2023-05-31T20:26:43.122 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8001} 2023-05-31T20:26:43.123 DEBUG:boto:Token: None 2023-05-31T20:26:43.123 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:26:43 GMT /pklxkd-53/ 2023-05-31T20:26:43.124 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:6pQUhzfkC8vEQoL30EEyH8WWjwQ= 2023-05-31T20:26:43.124 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:26:43 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:6pQUhzfkC8vEQoL30EEyH8WWjwQ=', 'Content-Length': '0'} 2023-05-31T20:26:43.129 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '4'), ('X-RGW-Bytes-Used', '12'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000007be7aeafdf5e5636-006477ad83-162e-a2'), ('Content-Length', '0'), ('Date', 'Wed, 31 May 2023 20:26:43 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:26:43.129 DEBUG:boto:path=/pklxkd-53/ 2023-05-31T20:26:43.130 DEBUG:boto:auth_path=/pklxkd-53/ 2023-05-31T20:26:43.130 DEBUG:boto:Method: HEAD 2023-05-31T20:26:43.130 DEBUG:boto:Path: /pklxkd-53/ 2023-05-31T20:26:43.131 DEBUG:boto:Data: 2023-05-31T20:26:43.131 DEBUG:boto:Headers: {} 2023-05-31T20:26:43.131 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:26:43.132 DEBUG:boto:Port: 8000 2023-05-31T20:26:43.132 DEBUG:boto:Params: {} 2023-05-31T20:26:43.133 DEBUG:boto:Token: None 2023-05-31T20:26:43.133 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:26:43 GMT /pklxkd-53/ 2023-05-31T20:26:43.134 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:6pQUhzfkC8vEQoL30EEyH8WWjwQ= 2023-05-31T20:26:43.134 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:26:43 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:6pQUhzfkC8vEQoL30EEyH8WWjwQ=', 'Content-Length': '0'} 2023-05-31T20:26:43.136 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '4'), ('X-RGW-Bytes-Used', '12'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000f3068abb98e5d81d-006477ad83-1630-a1'), ('Content-Length', '0'), ('Date', 'Wed, 31 May 2023 20:26:43 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:26:43.137 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2023-05-31T20:26:43.137 DEBUG:boto:path=/pklxkd-53/ 2023-05-31T20:26:43.138 DEBUG:boto:auth_path=/pklxkd-53/ 2023-05-31T20:26:43.138 DEBUG:boto:path=/pklxkd-53/?versions&max-keys=999 2023-05-31T20:26:43.139 DEBUG:boto:auth_path=/pklxkd-53/?versions&max-keys=999 2023-05-31T20:26:43.139 DEBUG:boto:Method: GET 2023-05-31T20:26:43.139 DEBUG:boto:Path: /pklxkd-53/?versions&max-keys=999 2023-05-31T20:26:43.140 DEBUG:boto:Data: 2023-05-31T20:26:43.140 DEBUG:boto:Headers: {} 2023-05-31T20:26:43.141 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:26:43.141 DEBUG:boto:Port: 8001 2023-05-31T20:26:43.141 DEBUG:boto:Params: {} 2023-05-31T20:26:43.142 DEBUG:boto:Token: None 2023-05-31T20:26:43.142 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:26:43 GMT /pklxkd-53/?versions 2023-05-31T20:26:43.143 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:qvE9DEE5d4rIRLRvIQ6GUX8BE2w= 2023-05-31T20:26:43.143 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:26:43 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:qvE9DEE5d4rIRLRvIQ6GUX8BE2w=', 'Content-Length': '0'} 2023-05-31T20:26:43.145 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000006cfc803236985c7c-006477ad83-162e-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:26:43 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:26:43.146 DEBUG:boto:b'pklxkd-53999falseanulltrue2023-05-31T20:26:27.510Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalbnulltrue2023-05-31T20:26:27.529Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalcnulltrue2023-05-31T20:26:27.548Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormaldnulltrue2023-05-31T20:26:27.602Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:26:43.147 DEBUG:tasks.rgw_multi.zone_rados:o=a 2023-05-31T20:26:43.147 DEBUG:tasks.rgw_multi.zone_rados:o=b 2023-05-31T20:26:43.148 DEBUG:tasks.rgw_multi.zone_rados:o=c 2023-05-31T20:26:43.148 DEBUG:tasks.rgw_multi.zone_rados:o=d 2023-05-31T20:26:43.148 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2023-05-31T20:26:43.149 DEBUG:boto:path=/pklxkd-53/ 2023-05-31T20:26:43.149 DEBUG:boto:auth_path=/pklxkd-53/ 2023-05-31T20:26:43.150 DEBUG:boto:path=/pklxkd-53/?versions&max-keys=999 2023-05-31T20:26:43.150 DEBUG:boto:auth_path=/pklxkd-53/?versions&max-keys=999 2023-05-31T20:26:43.150 DEBUG:boto:Method: GET 2023-05-31T20:26:43.151 DEBUG:boto:Path: /pklxkd-53/?versions&max-keys=999 2023-05-31T20:26:43.151 DEBUG:boto:Data: 2023-05-31T20:26:43.151 DEBUG:boto:Headers: {} 2023-05-31T20:26:43.152 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:26:43.152 DEBUG:boto:Port: 8000 2023-05-31T20:26:43.153 DEBUG:boto:Params: {} 2023-05-31T20:26:43.153 DEBUG:boto:Token: None 2023-05-31T20:26:43.154 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:26:43 GMT /pklxkd-53/?versions 2023-05-31T20:26:43.154 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:qvE9DEE5d4rIRLRvIQ6GUX8BE2w= 2023-05-31T20:26:43.155 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:26:43 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:qvE9DEE5d4rIRLRvIQ6GUX8BE2w=', 'Content-Length': '0'} 2023-05-31T20:26:43.157 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000829c2c3c2bc9cedb-006477ad83-1630-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:26:43 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:26:43.157 DEBUG:boto:b'pklxkd-53999falseanulltrue2023-05-31T20:26:27.510Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalbnulltrue2023-05-31T20:26:27.529Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalcnulltrue2023-05-31T20:26:27.548Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormaldnulltrue2023-05-31T20:26:27.602Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:26:43.158 DEBUG:tasks.rgw_multi.zone_rados:o=a 2023-05-31T20:26:43.159 DEBUG:tasks.rgw_multi.zone_rados:o=b 2023-05-31T20:26:43.159 DEBUG:tasks.rgw_multi.zone_rados:o=c 2023-05-31T20:26:43.159 DEBUG:tasks.rgw_multi.zone_rados:o=d 2023-05-31T20:26:43.160 DEBUG:boto:path=/pklxkd-53/ 2023-05-31T20:26:43.160 DEBUG:boto:auth_path=/pklxkd-53/ 2023-05-31T20:26:43.161 DEBUG:boto:path=/pklxkd-53/?versions&max-keys=999 2023-05-31T20:26:43.161 DEBUG:boto:auth_path=/pklxkd-53/?versions&max-keys=999 2023-05-31T20:26:43.161 DEBUG:boto:Method: GET 2023-05-31T20:26:43.162 DEBUG:boto:Path: /pklxkd-53/?versions&max-keys=999 2023-05-31T20:26:43.162 DEBUG:boto:Data: 2023-05-31T20:26:43.162 DEBUG:boto:Headers: {} 2023-05-31T20:26:43.163 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:26:43.163 DEBUG:boto:Port: 8001 2023-05-31T20:26:43.164 DEBUG:boto:Params: {} 2023-05-31T20:26:43.164 DEBUG:boto:Token: None 2023-05-31T20:26:43.165 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:26:43 GMT /pklxkd-53/?versions 2023-05-31T20:26:43.165 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:qvE9DEE5d4rIRLRvIQ6GUX8BE2w= 2023-05-31T20:26:43.166 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:26:43 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:qvE9DEE5d4rIRLRvIQ6GUX8BE2w=', 'Content-Length': '0'} 2023-05-31T20:26:43.168 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d1c7cbda61152454-006477ad83-162e-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:26:43 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:26:43.169 DEBUG:boto:b'pklxkd-53999falseanulltrue2023-05-31T20:26:27.510Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalbnulltrue2023-05-31T20:26:27.529Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalcnulltrue2023-05-31T20:26:27.548Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormaldnulltrue2023-05-31T20:26:27.602Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:26:43.170 DEBUG:boto:path=/pklxkd-53/ 2023-05-31T20:26:43.170 DEBUG:boto:auth_path=/pklxkd-53/ 2023-05-31T20:26:43.170 DEBUG:boto:path=/pklxkd-53/?versions&max-keys=999 2023-05-31T20:26:43.171 DEBUG:boto:auth_path=/pklxkd-53/?versions&max-keys=999 2023-05-31T20:26:43.171 DEBUG:boto:Method: GET 2023-05-31T20:26:43.172 DEBUG:boto:Path: /pklxkd-53/?versions&max-keys=999 2023-05-31T20:26:43.172 DEBUG:boto:Data: 2023-05-31T20:26:43.172 DEBUG:boto:Headers: {} 2023-05-31T20:26:43.173 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:26:43.173 DEBUG:boto:Port: 8000 2023-05-31T20:26:43.174 DEBUG:boto:Params: {} 2023-05-31T20:26:43.174 DEBUG:boto:Token: None 2023-05-31T20:26:43.174 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:26:43 GMT /pklxkd-53/?versions 2023-05-31T20:26:43.175 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:qvE9DEE5d4rIRLRvIQ6GUX8BE2w= 2023-05-31T20:26:43.175 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:26:43 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:qvE9DEE5d4rIRLRvIQ6GUX8BE2w=', 'Content-Length': '0'} 2023-05-31T20:26:43.178 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a27e4e699aa55adc-006477ad83-1630-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:26:43 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:26:43.179 DEBUG:boto:b'pklxkd-53999falseanulltrue2023-05-31T20:26:27.510Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalbnulltrue2023-05-31T20:26:27.529Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalcnulltrue2023-05-31T20:26:27.548Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormaldnulltrue2023-05-31T20:26:27.602Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:26:43.180 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=a 2023-05-31T20:26:43.180 DEBUG:boto:path=/pklxkd-53/a 2023-05-31T20:26:43.181 DEBUG:boto:auth_path=/pklxkd-53/a 2023-05-31T20:26:43.181 DEBUG:boto:path=/pklxkd-53/a?versionId=null 2023-05-31T20:26:43.181 DEBUG:boto:auth_path=/pklxkd-53/a?versionId=null 2023-05-31T20:26:43.182 DEBUG:boto:Method: GET 2023-05-31T20:26:43.182 DEBUG:boto:Path: /pklxkd-53/a?versionId=null 2023-05-31T20:26:43.183 DEBUG:boto:Data: 2023-05-31T20:26:43.183 DEBUG:boto:Headers: {} 2023-05-31T20:26:43.183 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:26:43.184 DEBUG:boto:Port: 8001 2023-05-31T20:26:43.184 DEBUG:boto:Params: {} 2023-05-31T20:26:43.185 DEBUG:boto:Token: None 2023-05-31T20:26:43.185 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:26:43 GMT /pklxkd-53/a?versionId=null 2023-05-31T20:26:43.186 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:VQ/+DY39s1E6ZUkVaflBosD52EU= 2023-05-31T20:26:43.186 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:26:43 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:VQ/+DY39s1E6ZUkVaflBosD52EU=', 'Content-Length': '0'} 2023-05-31T20:26:43.188 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:26:27 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000076d95ef1966cc875-006477ad83-162e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:26:43 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:26:43.228 DEBUG:boto:path=/pklxkd-53/a 2023-05-31T20:26:43.229 DEBUG:boto:auth_path=/pklxkd-53/a 2023-05-31T20:26:43.229 DEBUG:boto:path=/pklxkd-53/a?versionId=null 2023-05-31T20:26:43.230 DEBUG:boto:auth_path=/pklxkd-53/a?versionId=null 2023-05-31T20:26:43.230 DEBUG:boto:Method: GET 2023-05-31T20:26:43.230 DEBUG:boto:Path: /pklxkd-53/a?versionId=null 2023-05-31T20:26:43.231 DEBUG:boto:Data: 2023-05-31T20:26:43.231 DEBUG:boto:Headers: {} 2023-05-31T20:26:43.231 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:26:43.232 DEBUG:boto:Port: 8000 2023-05-31T20:26:43.232 DEBUG:boto:Params: {} 2023-05-31T20:26:43.233 DEBUG:boto:Token: None 2023-05-31T20:26:43.233 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:26:43 GMT /pklxkd-53/a?versionId=null 2023-05-31T20:26:43.234 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:VQ/+DY39s1E6ZUkVaflBosD52EU= 2023-05-31T20:26:43.234 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:26:43 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:VQ/+DY39s1E6ZUkVaflBosD52EU=', 'Content-Length': '0'} 2023-05-31T20:26:43.236 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:26:27 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e4e8beff45076fa5-006477ad83-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:26:43 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:26:43.237 DEBUG:boto:path=/pklxkd-53/a 2023-05-31T20:26:43.237 DEBUG:boto:auth_path=/pklxkd-53/a 2023-05-31T20:26:43.238 DEBUG:boto:path=/pklxkd-53/a?versionId=null 2023-05-31T20:26:43.238 DEBUG:boto:auth_path=/pklxkd-53/a?versionId=null 2023-05-31T20:26:43.238 DEBUG:boto:Method: HEAD 2023-05-31T20:26:43.239 DEBUG:boto:Path: /pklxkd-53/a?versionId=null 2023-05-31T20:26:43.239 DEBUG:boto:Data: 2023-05-31T20:26:43.239 DEBUG:boto:Headers: {} 2023-05-31T20:26:43.240 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:26:43.240 DEBUG:boto:Port: 8001 2023-05-31T20:26:43.240 DEBUG:boto:Params: {} 2023-05-31T20:26:43.241 DEBUG:boto:Token: None 2023-05-31T20:26:43.241 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:26:43 GMT /pklxkd-53/a?versionId=null 2023-05-31T20:26:43.242 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:/MbQIuAOrVxVu/M0oteBAeYID1o= 2023-05-31T20:26:43.242 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:26:43 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:/MbQIuAOrVxVu/M0oteBAeYID1o=', 'Content-Length': '0'} 2023-05-31T20:26:43.244 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:26:27 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000048eb5f3a85c5c7cd-006477ad83-162e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:26:43 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:26:43.244 DEBUG:boto:path=/pklxkd-53/a 2023-05-31T20:26:43.245 DEBUG:boto:auth_path=/pklxkd-53/a 2023-05-31T20:26:43.245 DEBUG:boto:path=/pklxkd-53/a?versionId=null 2023-05-31T20:26:43.245 DEBUG:boto:auth_path=/pklxkd-53/a?versionId=null 2023-05-31T20:26:43.246 DEBUG:boto:Method: HEAD 2023-05-31T20:26:43.246 DEBUG:boto:Path: /pklxkd-53/a?versionId=null 2023-05-31T20:26:43.247 DEBUG:boto:Data: 2023-05-31T20:26:43.247 DEBUG:boto:Headers: {} 2023-05-31T20:26:43.247 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:26:43.248 DEBUG:boto:Port: 8000 2023-05-31T20:26:43.248 DEBUG:boto:Params: {} 2023-05-31T20:26:43.248 DEBUG:boto:Token: None 2023-05-31T20:26:43.249 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:26:43 GMT /pklxkd-53/a?versionId=null 2023-05-31T20:26:43.249 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:/MbQIuAOrVxVu/M0oteBAeYID1o= 2023-05-31T20:26:43.250 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:26:43 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:/MbQIuAOrVxVu/M0oteBAeYID1o=', 'Content-Length': '0'} 2023-05-31T20:26:43.251 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:26:27 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000057288453ed6e9c47-006477ad83-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:26:43 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:26:43.252 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=a 2023-05-31T20:26:43.252 DEBUG:boto:path=/pklxkd-53/a 2023-05-31T20:26:43.253 DEBUG:boto:auth_path=/pklxkd-53/a 2023-05-31T20:26:43.253 DEBUG:boto:path=/pklxkd-53/a?versionId=null 2023-05-31T20:26:43.253 DEBUG:boto:auth_path=/pklxkd-53/a?versionId=null 2023-05-31T20:26:43.254 DEBUG:boto:Method: GET 2023-05-31T20:26:43.254 DEBUG:boto:Path: /pklxkd-53/a?versionId=null 2023-05-31T20:26:43.256 DEBUG:boto:Data: 2023-05-31T20:26:43.257 DEBUG:boto:Headers: {} 2023-05-31T20:26:43.257 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:26:43.258 DEBUG:boto:Port: 8001 2023-05-31T20:26:43.258 DEBUG:boto:Params: {} 2023-05-31T20:26:43.259 DEBUG:boto:Token: None 2023-05-31T20:26:43.259 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:26:43 GMT /pklxkd-53/a?versionId=null 2023-05-31T20:26:43.260 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:VQ/+DY39s1E6ZUkVaflBosD52EU= 2023-05-31T20:26:43.260 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:26:43 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:VQ/+DY39s1E6ZUkVaflBosD52EU=', 'Content-Length': '0'} 2023-05-31T20:26:43.262 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:26:27 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000fefd415f4dac9bcb-006477ad83-162e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:26:43 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:26:43.262 DEBUG:boto:path=/pklxkd-53/a 2023-05-31T20:26:43.263 DEBUG:boto:auth_path=/pklxkd-53/a 2023-05-31T20:26:43.263 DEBUG:boto:path=/pklxkd-53/a?versionId=null 2023-05-31T20:26:43.264 DEBUG:boto:auth_path=/pklxkd-53/a?versionId=null 2023-05-31T20:26:43.265 DEBUG:boto:Method: GET 2023-05-31T20:26:43.266 DEBUG:boto:Path: /pklxkd-53/a?versionId=null 2023-05-31T20:26:43.267 DEBUG:boto:Data: 2023-05-31T20:26:43.267 DEBUG:boto:Headers: {} 2023-05-31T20:26:43.267 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:26:43.268 DEBUG:boto:Port: 8000 2023-05-31T20:26:43.268 DEBUG:boto:Params: {} 2023-05-31T20:26:43.269 DEBUG:boto:Token: None 2023-05-31T20:26:43.269 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:26:43 GMT /pklxkd-53/a?versionId=null 2023-05-31T20:26:43.270 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:VQ/+DY39s1E6ZUkVaflBosD52EU= 2023-05-31T20:26:43.270 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:26:43 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:VQ/+DY39s1E6ZUkVaflBosD52EU=', 'Content-Length': '0'} 2023-05-31T20:26:43.273 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:26:27 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000002ad3915ea48c51b-006477ad83-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:26:43 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:26:43.273 DEBUG:boto:path=/pklxkd-53/ 2023-05-31T20:26:43.274 DEBUG:boto:auth_path=/pklxkd-53/ 2023-05-31T20:26:43.274 DEBUG:boto:path=/pklxkd-53/?prefix=a 2023-05-31T20:26:43.275 DEBUG:boto:auth_path=/pklxkd-53/?prefix=a 2023-05-31T20:26:43.275 DEBUG:boto:Method: GET 2023-05-31T20:26:43.275 DEBUG:boto:Path: /pklxkd-53/?prefix=a 2023-05-31T20:26:43.276 DEBUG:boto:Data: 2023-05-31T20:26:43.276 DEBUG:boto:Headers: {} 2023-05-31T20:26:43.277 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:26:43.277 DEBUG:boto:Port: 8001 2023-05-31T20:26:43.278 DEBUG:boto:Params: {} 2023-05-31T20:26:43.278 DEBUG:boto:Token: None 2023-05-31T20:26:43.279 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:26:43 GMT /pklxkd-53/ 2023-05-31T20:26:43.279 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:RtuWiOhPp2v0s+P6tP4ZjzNQme0= 2023-05-31T20:26:43.280 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:26:43 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:RtuWiOhPp2v0s+P6tP4ZjzNQme0=', 'Content-Length': '0'} 2023-05-31T20:26:43.282 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000091f8021c1ac7930-006477ad83-162e-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:26:43 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:26:43.283 DEBUG:boto:b'pklxkd-53a1000falsea2023-05-31T20:26:27.510Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:26:43.283 DEBUG:boto:path=/pklxkd-53/ 2023-05-31T20:26:43.284 DEBUG:boto:auth_path=/pklxkd-53/ 2023-05-31T20:26:43.284 DEBUG:boto:path=/pklxkd-53/?prefix=a 2023-05-31T20:26:43.285 DEBUG:boto:auth_path=/pklxkd-53/?prefix=a 2023-05-31T20:26:43.285 DEBUG:boto:Method: GET 2023-05-31T20:26:43.286 DEBUG:boto:Path: /pklxkd-53/?prefix=a 2023-05-31T20:26:43.286 DEBUG:boto:Data: 2023-05-31T20:26:43.287 DEBUG:boto:Headers: {} 2023-05-31T20:26:43.287 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:26:43.288 DEBUG:boto:Port: 8000 2023-05-31T20:26:43.288 DEBUG:boto:Params: {} 2023-05-31T20:26:43.289 DEBUG:boto:Token: None 2023-05-31T20:26:43.290 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:26:43 GMT /pklxkd-53/ 2023-05-31T20:26:43.291 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:RtuWiOhPp2v0s+P6tP4ZjzNQme0= 2023-05-31T20:26:43.292 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:26:43 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:RtuWiOhPp2v0s+P6tP4ZjzNQme0=', 'Content-Length': '0'} 2023-05-31T20:26:43.295 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000002da4e4a812acc2ad-006477ad83-1630-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:26:43 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:26:43.295 DEBUG:boto:b'pklxkd-53a1000falsea2023-05-31T20:26:27.510Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:26:43.296 DEBUG:boto:path=/pklxkd-53/a 2023-05-31T20:26:43.297 DEBUG:boto:auth_path=/pklxkd-53/a 2023-05-31T20:26:43.298 DEBUG:boto:Method: HEAD 2023-05-31T20:26:43.299 DEBUG:boto:Path: /pklxkd-53/a 2023-05-31T20:26:43.301 DEBUG:boto:Data: 2023-05-31T20:26:43.301 DEBUG:boto:Headers: {} 2023-05-31T20:26:43.302 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:26:43.303 DEBUG:boto:Port: 8001 2023-05-31T20:26:43.304 DEBUG:boto:Params: {} 2023-05-31T20:26:43.305 DEBUG:boto:Token: None 2023-05-31T20:26:43.305 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:26:43 GMT /pklxkd-53/a 2023-05-31T20:26:43.306 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:2DnvNMuUXnVgyEPdWxXtoLTdSOI= 2023-05-31T20:26:43.307 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:26:43 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:2DnvNMuUXnVgyEPdWxXtoLTdSOI=', 'Content-Length': '0'} 2023-05-31T20:26:43.309 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:26:27 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005761e9285c9e7f6f-006477ad83-162e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:26:43 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:26:43.310 DEBUG:boto:path=/pklxkd-53/a 2023-05-31T20:26:43.310 DEBUG:boto:auth_path=/pklxkd-53/a 2023-05-31T20:26:43.311 DEBUG:boto:Method: HEAD 2023-05-31T20:26:43.311 DEBUG:boto:Path: /pklxkd-53/a 2023-05-31T20:26:43.311 DEBUG:boto:Data: 2023-05-31T20:26:43.312 DEBUG:boto:Headers: {} 2023-05-31T20:26:43.312 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:26:43.312 DEBUG:boto:Port: 8000 2023-05-31T20:26:43.313 DEBUG:boto:Params: {} 2023-05-31T20:26:43.314 DEBUG:boto:Token: None 2023-05-31T20:26:43.314 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:26:43 GMT /pklxkd-53/a 2023-05-31T20:26:43.314 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:2DnvNMuUXnVgyEPdWxXtoLTdSOI= 2023-05-31T20:26:43.315 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:26:43 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:2DnvNMuUXnVgyEPdWxXtoLTdSOI=', 'Content-Length': '0'} 2023-05-31T20:26:43.316 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:26:27 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008827bf44325f6cdb-006477ad83-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:26:43 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:26:43.317 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=a 2023-05-31T20:26:43.318 DEBUG:boto:path=/pklxkd-53/a 2023-05-31T20:26:43.318 DEBUG:boto:auth_path=/pklxkd-53/a 2023-05-31T20:26:43.319 DEBUG:boto:Method: GET 2023-05-31T20:26:43.319 DEBUG:boto:Path: /pklxkd-53/a 2023-05-31T20:26:43.320 DEBUG:boto:Data: 2023-05-31T20:26:43.320 DEBUG:boto:Headers: {} 2023-05-31T20:26:43.320 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:26:43.321 DEBUG:boto:Port: 8001 2023-05-31T20:26:43.321 DEBUG:boto:Params: {} 2023-05-31T20:26:43.322 DEBUG:boto:Token: None 2023-05-31T20:26:43.322 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:26:43 GMT /pklxkd-53/a 2023-05-31T20:26:43.323 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:R5NmYe9V4yiWw4OJPUc/5eUbsi8= 2023-05-31T20:26:43.323 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:26:43 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:R5NmYe9V4yiWw4OJPUc/5eUbsi8=', 'Content-Length': '0'} 2023-05-31T20:26:43.325 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:26:27 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a8d82f8feed72a01-006477ad83-162e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:26:43 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:26:43.325 DEBUG:boto:path=/pklxkd-53/a 2023-05-31T20:26:43.326 DEBUG:boto:auth_path=/pklxkd-53/a 2023-05-31T20:26:43.327 DEBUG:boto:Method: GET 2023-05-31T20:26:43.328 DEBUG:boto:Path: /pklxkd-53/a 2023-05-31T20:26:43.329 DEBUG:boto:Data: 2023-05-31T20:26:43.330 DEBUG:boto:Headers: {} 2023-05-31T20:26:43.331 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:26:43.332 DEBUG:boto:Port: 8000 2023-05-31T20:26:43.332 DEBUG:boto:Params: {} 2023-05-31T20:26:43.333 DEBUG:boto:Token: None 2023-05-31T20:26:43.333 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:26:43 GMT /pklxkd-53/a 2023-05-31T20:26:43.334 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:R5NmYe9V4yiWw4OJPUc/5eUbsi8= 2023-05-31T20:26:43.334 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:26:43 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:R5NmYe9V4yiWw4OJPUc/5eUbsi8=', 'Content-Length': '0'} 2023-05-31T20:26:43.336 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:26:27 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000024dee888b9710f8-006477ad83-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:26:43 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:26:43.380 DEBUG:boto:path=/pklxkd-53/ 2023-05-31T20:26:43.381 DEBUG:boto:auth_path=/pklxkd-53/ 2023-05-31T20:26:43.381 DEBUG:boto:path=/pklxkd-53/?prefix=a 2023-05-31T20:26:43.382 DEBUG:boto:auth_path=/pklxkd-53/?prefix=a 2023-05-31T20:26:43.382 DEBUG:boto:Method: GET 2023-05-31T20:26:43.383 DEBUG:boto:Path: /pklxkd-53/?prefix=a 2023-05-31T20:26:43.383 DEBUG:boto:Data: 2023-05-31T20:26:43.384 DEBUG:boto:Headers: {} 2023-05-31T20:26:43.384 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:26:43.384 DEBUG:boto:Port: 8001 2023-05-31T20:26:43.386 DEBUG:boto:Params: {} 2023-05-31T20:26:43.387 DEBUG:boto:Token: None 2023-05-31T20:26:43.387 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:26:43 GMT /pklxkd-53/ 2023-05-31T20:26:43.387 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:RtuWiOhPp2v0s+P6tP4ZjzNQme0= 2023-05-31T20:26:43.388 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:26:43 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:RtuWiOhPp2v0s+P6tP4ZjzNQme0=', 'Content-Length': '0'} 2023-05-31T20:26:43.390 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c69819c4bd2a5c7e-006477ad83-162e-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:26:43 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:26:43.391 DEBUG:boto:b'pklxkd-53a1000falsea2023-05-31T20:26:27.510Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:26:43.391 DEBUG:boto:path=/pklxkd-53/ 2023-05-31T20:26:43.392 DEBUG:boto:auth_path=/pklxkd-53/ 2023-05-31T20:26:43.392 DEBUG:boto:path=/pklxkd-53/?prefix=a 2023-05-31T20:26:43.393 DEBUG:boto:auth_path=/pklxkd-53/?prefix=a 2023-05-31T20:26:43.393 DEBUG:boto:Method: GET 2023-05-31T20:26:43.394 DEBUG:boto:Path: /pklxkd-53/?prefix=a 2023-05-31T20:26:43.394 DEBUG:boto:Data: 2023-05-31T20:26:43.394 DEBUG:boto:Headers: {} 2023-05-31T20:26:43.395 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:26:43.395 DEBUG:boto:Port: 8000 2023-05-31T20:26:43.395 DEBUG:boto:Params: {} 2023-05-31T20:26:43.396 DEBUG:boto:Token: None 2023-05-31T20:26:43.396 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:26:43 GMT /pklxkd-53/ 2023-05-31T20:26:43.397 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:RtuWiOhPp2v0s+P6tP4ZjzNQme0= 2023-05-31T20:26:43.397 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:26:43 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:RtuWiOhPp2v0s+P6tP4ZjzNQme0=', 'Content-Length': '0'} 2023-05-31T20:26:43.400 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000026737100b0329661-006477ad83-1630-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:26:43 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:26:43.400 DEBUG:boto:b'pklxkd-53a1000falsea2023-05-31T20:26:27.510Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:26:43.401 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=b 2023-05-31T20:26:43.401 DEBUG:boto:path=/pklxkd-53/b 2023-05-31T20:26:43.402 DEBUG:boto:auth_path=/pklxkd-53/b 2023-05-31T20:26:43.402 DEBUG:boto:path=/pklxkd-53/b?versionId=null 2023-05-31T20:26:43.402 DEBUG:boto:auth_path=/pklxkd-53/b?versionId=null 2023-05-31T20:26:43.403 DEBUG:boto:Method: GET 2023-05-31T20:26:43.403 DEBUG:boto:Path: /pklxkd-53/b?versionId=null 2023-05-31T20:26:43.404 DEBUG:boto:Data: 2023-05-31T20:26:43.404 DEBUG:boto:Headers: {} 2023-05-31T20:26:43.404 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:26:43.405 DEBUG:boto:Port: 8001 2023-05-31T20:26:43.405 DEBUG:boto:Params: {} 2023-05-31T20:26:43.405 DEBUG:boto:Token: None 2023-05-31T20:26:43.406 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:26:43 GMT /pklxkd-53/b?versionId=null 2023-05-31T20:26:43.406 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:STKRAxIqxCv/YE4WtE40xWuA+a4= 2023-05-31T20:26:43.407 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:26:43 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:STKRAxIqxCv/YE4WtE40xWuA+a4=', 'Content-Length': '0'} 2023-05-31T20:26:43.408 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:26:27 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000e5e54e3ecb5864b-006477ad83-162e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:26:43 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:26:43.409 DEBUG:boto:path=/pklxkd-53/b 2023-05-31T20:26:43.409 DEBUG:boto:auth_path=/pklxkd-53/b 2023-05-31T20:26:43.410 DEBUG:boto:path=/pklxkd-53/b?versionId=null 2023-05-31T20:26:43.410 DEBUG:boto:auth_path=/pklxkd-53/b?versionId=null 2023-05-31T20:26:43.412 DEBUG:boto:Method: GET 2023-05-31T20:26:43.412 DEBUG:boto:Path: /pklxkd-53/b?versionId=null 2023-05-31T20:26:43.412 DEBUG:boto:Data: 2023-05-31T20:26:43.413 DEBUG:boto:Headers: {} 2023-05-31T20:26:43.413 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:26:43.414 DEBUG:boto:Port: 8000 2023-05-31T20:26:43.414 DEBUG:boto:Params: {} 2023-05-31T20:26:43.414 DEBUG:boto:Token: None 2023-05-31T20:26:43.415 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:26:43 GMT /pklxkd-53/b?versionId=null 2023-05-31T20:26:43.415 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:STKRAxIqxCv/YE4WtE40xWuA+a4= 2023-05-31T20:26:43.416 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:26:43 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:STKRAxIqxCv/YE4WtE40xWuA+a4=', 'Content-Length': '0'} 2023-05-31T20:26:43.418 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:26:27 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b2671e9b567b2fa2-006477ad83-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:26:43 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:26:43.419 DEBUG:boto:path=/pklxkd-53/b 2023-05-31T20:26:43.419 DEBUG:boto:auth_path=/pklxkd-53/b 2023-05-31T20:26:43.419 DEBUG:boto:path=/pklxkd-53/b?versionId=null 2023-05-31T20:26:43.420 DEBUG:boto:auth_path=/pklxkd-53/b?versionId=null 2023-05-31T20:26:43.420 DEBUG:boto:Method: HEAD 2023-05-31T20:26:43.420 DEBUG:boto:Path: /pklxkd-53/b?versionId=null 2023-05-31T20:26:43.421 DEBUG:boto:Data: 2023-05-31T20:26:43.421 DEBUG:boto:Headers: {} 2023-05-31T20:26:43.422 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:26:43.422 DEBUG:boto:Port: 8001 2023-05-31T20:26:43.422 DEBUG:boto:Params: {} 2023-05-31T20:26:43.423 DEBUG:boto:Token: None 2023-05-31T20:26:43.423 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:26:43 GMT /pklxkd-53/b?versionId=null 2023-05-31T20:26:43.424 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:qdjlqATWSaCfn1VgFVdmhboo9e8= 2023-05-31T20:26:43.424 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:26:43 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:qdjlqATWSaCfn1VgFVdmhboo9e8=', 'Content-Length': '0'} 2023-05-31T20:26:43.425 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:26:27 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ab7b13d00cb9e735-006477ad83-162e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:26:43 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:26:43.426 DEBUG:boto:path=/pklxkd-53/b 2023-05-31T20:26:43.426 DEBUG:boto:auth_path=/pklxkd-53/b 2023-05-31T20:26:43.427 DEBUG:boto:path=/pklxkd-53/b?versionId=null 2023-05-31T20:26:43.427 DEBUG:boto:auth_path=/pklxkd-53/b?versionId=null 2023-05-31T20:26:43.427 DEBUG:boto:Method: HEAD 2023-05-31T20:26:43.428 DEBUG:boto:Path: /pklxkd-53/b?versionId=null 2023-05-31T20:26:43.428 DEBUG:boto:Data: 2023-05-31T20:26:43.429 DEBUG:boto:Headers: {} 2023-05-31T20:26:43.429 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:26:43.429 DEBUG:boto:Port: 8000 2023-05-31T20:26:43.430 DEBUG:boto:Params: {} 2023-05-31T20:26:43.430 DEBUG:boto:Token: None 2023-05-31T20:26:43.430 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:26:43 GMT /pklxkd-53/b?versionId=null 2023-05-31T20:26:43.431 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:qdjlqATWSaCfn1VgFVdmhboo9e8= 2023-05-31T20:26:43.431 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:26:43 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:qdjlqATWSaCfn1VgFVdmhboo9e8=', 'Content-Length': '0'} 2023-05-31T20:26:43.433 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:26:27 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001e8dbcdb5c9d4af0-006477ad83-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:26:43 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:26:43.433 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=b 2023-05-31T20:26:43.434 DEBUG:boto:path=/pklxkd-53/b 2023-05-31T20:26:43.434 DEBUG:boto:auth_path=/pklxkd-53/b 2023-05-31T20:26:43.435 DEBUG:boto:path=/pklxkd-53/b?versionId=null 2023-05-31T20:26:43.435 DEBUG:boto:auth_path=/pklxkd-53/b?versionId=null 2023-05-31T20:26:43.435 DEBUG:boto:Method: GET 2023-05-31T20:26:43.436 DEBUG:boto:Path: /pklxkd-53/b?versionId=null 2023-05-31T20:26:43.436 DEBUG:boto:Data: 2023-05-31T20:26:43.437 DEBUG:boto:Headers: {} 2023-05-31T20:26:43.437 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:26:43.437 DEBUG:boto:Port: 8001 2023-05-31T20:26:43.438 DEBUG:boto:Params: {} 2023-05-31T20:26:43.438 DEBUG:boto:Token: None 2023-05-31T20:26:43.439 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:26:43 GMT /pklxkd-53/b?versionId=null 2023-05-31T20:26:43.439 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:STKRAxIqxCv/YE4WtE40xWuA+a4= 2023-05-31T20:26:43.440 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:26:43 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:STKRAxIqxCv/YE4WtE40xWuA+a4=', 'Content-Length': '0'} 2023-05-31T20:26:43.442 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:26:27 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f4bbd38bdb8f97a3-006477ad83-162e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:26:43 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:26:43.442 DEBUG:boto:path=/pklxkd-53/b 2023-05-31T20:26:43.443 DEBUG:boto:auth_path=/pklxkd-53/b 2023-05-31T20:26:43.443 DEBUG:boto:path=/pklxkd-53/b?versionId=null 2023-05-31T20:26:43.444 DEBUG:boto:auth_path=/pklxkd-53/b?versionId=null 2023-05-31T20:26:43.444 DEBUG:boto:Method: GET 2023-05-31T20:26:43.444 DEBUG:boto:Path: /pklxkd-53/b?versionId=null 2023-05-31T20:26:43.445 DEBUG:boto:Data: 2023-05-31T20:26:43.445 DEBUG:boto:Headers: {} 2023-05-31T20:26:43.446 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:26:43.446 DEBUG:boto:Port: 8000 2023-05-31T20:26:43.446 DEBUG:boto:Params: {} 2023-05-31T20:26:43.447 DEBUG:boto:Token: None 2023-05-31T20:26:43.447 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:26:43 GMT /pklxkd-53/b?versionId=null 2023-05-31T20:26:43.448 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:STKRAxIqxCv/YE4WtE40xWuA+a4= 2023-05-31T20:26:43.448 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:26:43 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:STKRAxIqxCv/YE4WtE40xWuA+a4=', 'Content-Length': '0'} 2023-05-31T20:26:43.450 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:26:27 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000065822ccaf48e840b-006477ad83-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:26:43 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:26:43.450 DEBUG:boto:path=/pklxkd-53/ 2023-05-31T20:26:43.451 DEBUG:boto:auth_path=/pklxkd-53/ 2023-05-31T20:26:43.451 DEBUG:boto:path=/pklxkd-53/?prefix=b 2023-05-31T20:26:43.451 DEBUG:boto:auth_path=/pklxkd-53/?prefix=b 2023-05-31T20:26:43.452 DEBUG:boto:Method: GET 2023-05-31T20:26:43.452 DEBUG:boto:Path: /pklxkd-53/?prefix=b 2023-05-31T20:26:43.453 DEBUG:boto:Data: 2023-05-31T20:26:43.453 DEBUG:boto:Headers: {} 2023-05-31T20:26:43.453 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:26:43.454 DEBUG:boto:Port: 8001 2023-05-31T20:26:43.454 DEBUG:boto:Params: {} 2023-05-31T20:26:43.454 DEBUG:boto:Token: None 2023-05-31T20:26:43.455 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:26:43 GMT /pklxkd-53/ 2023-05-31T20:26:43.455 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:RtuWiOhPp2v0s+P6tP4ZjzNQme0= 2023-05-31T20:26:43.456 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:26:43 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:RtuWiOhPp2v0s+P6tP4ZjzNQme0=', 'Content-Length': '0'} 2023-05-31T20:26:43.458 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000091fe8e3c76607448-006477ad83-162e-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:26:43 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:26:43.458 DEBUG:boto:b'pklxkd-53b1000falseb2023-05-31T20:26:27.529Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:26:43.460 DEBUG:boto:path=/pklxkd-53/ 2023-05-31T20:26:43.460 DEBUG:boto:auth_path=/pklxkd-53/ 2023-05-31T20:26:43.461 DEBUG:boto:path=/pklxkd-53/?prefix=b 2023-05-31T20:26:43.461 DEBUG:boto:auth_path=/pklxkd-53/?prefix=b 2023-05-31T20:26:43.461 DEBUG:boto:Method: GET 2023-05-31T20:26:43.462 DEBUG:boto:Path: /pklxkd-53/?prefix=b 2023-05-31T20:26:43.462 DEBUG:boto:Data: 2023-05-31T20:26:43.463 DEBUG:boto:Headers: {} 2023-05-31T20:26:43.463 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:26:43.463 DEBUG:boto:Port: 8000 2023-05-31T20:26:43.464 DEBUG:boto:Params: {} 2023-05-31T20:26:43.464 DEBUG:boto:Token: None 2023-05-31T20:26:43.465 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:26:43 GMT /pklxkd-53/ 2023-05-31T20:26:43.465 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:RtuWiOhPp2v0s+P6tP4ZjzNQme0= 2023-05-31T20:26:43.466 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:26:43 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:RtuWiOhPp2v0s+P6tP4ZjzNQme0=', 'Content-Length': '0'} 2023-05-31T20:26:43.468 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e70c6f73a6a41104-006477ad83-1630-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:26:43 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:26:43.468 DEBUG:boto:b'pklxkd-53b1000falseb2023-05-31T20:26:27.529Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:26:43.469 DEBUG:boto:path=/pklxkd-53/b 2023-05-31T20:26:43.470 DEBUG:boto:auth_path=/pklxkd-53/b 2023-05-31T20:26:43.470 DEBUG:boto:Method: HEAD 2023-05-31T20:26:43.470 DEBUG:boto:Path: /pklxkd-53/b 2023-05-31T20:26:43.471 DEBUG:boto:Data: 2023-05-31T20:26:43.471 DEBUG:boto:Headers: {} 2023-05-31T20:26:43.472 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:26:43.472 DEBUG:boto:Port: 8001 2023-05-31T20:26:43.472 DEBUG:boto:Params: {} 2023-05-31T20:26:43.473 DEBUG:boto:Token: None 2023-05-31T20:26:43.473 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:26:43 GMT /pklxkd-53/b 2023-05-31T20:26:43.473 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:zHIWA/WXeZcnX50NCfozKS6IJis= 2023-05-31T20:26:43.474 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:26:43 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:zHIWA/WXeZcnX50NCfozKS6IJis=', 'Content-Length': '0'} 2023-05-31T20:26:43.475 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:26:27 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000e55f32ea049674b-006477ad83-162e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:26:43 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:26:43.476 DEBUG:boto:path=/pklxkd-53/b 2023-05-31T20:26:43.476 DEBUG:boto:auth_path=/pklxkd-53/b 2023-05-31T20:26:43.477 DEBUG:boto:Method: HEAD 2023-05-31T20:26:43.477 DEBUG:boto:Path: /pklxkd-53/b 2023-05-31T20:26:43.477 DEBUG:boto:Data: 2023-05-31T20:26:43.478 DEBUG:boto:Headers: {} 2023-05-31T20:26:43.478 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:26:43.478 DEBUG:boto:Port: 8000 2023-05-31T20:26:43.479 DEBUG:boto:Params: {} 2023-05-31T20:26:43.479 DEBUG:boto:Token: None 2023-05-31T20:26:43.480 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:26:43 GMT /pklxkd-53/b 2023-05-31T20:26:43.480 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:zHIWA/WXeZcnX50NCfozKS6IJis= 2023-05-31T20:26:43.481 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:26:43 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:zHIWA/WXeZcnX50NCfozKS6IJis=', 'Content-Length': '0'} 2023-05-31T20:26:43.482 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:26:27 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e41762c6e651dcf9-006477ad83-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:26:43 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:26:43.483 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=b 2023-05-31T20:26:43.483 DEBUG:boto:path=/pklxkd-53/b 2023-05-31T20:26:43.484 DEBUG:boto:auth_path=/pklxkd-53/b 2023-05-31T20:26:43.484 DEBUG:boto:Method: GET 2023-05-31T20:26:43.484 DEBUG:boto:Path: /pklxkd-53/b 2023-05-31T20:26:43.485 DEBUG:boto:Data: 2023-05-31T20:26:43.485 DEBUG:boto:Headers: {} 2023-05-31T20:26:43.485 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:26:43.486 DEBUG:boto:Port: 8001 2023-05-31T20:26:43.486 DEBUG:boto:Params: {} 2023-05-31T20:26:43.487 DEBUG:boto:Token: None 2023-05-31T20:26:43.487 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:26:43 GMT /pklxkd-53/b 2023-05-31T20:26:43.488 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:yhXmulTkH3JK8ITqhywI/xJxJgM= 2023-05-31T20:26:43.488 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:26:43 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:yhXmulTkH3JK8ITqhywI/xJxJgM=', 'Content-Length': '0'} 2023-05-31T20:26:43.490 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:26:27 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000050a6092fc6b39132-006477ad83-162e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:26:43 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:26:43.491 DEBUG:boto:path=/pklxkd-53/b 2023-05-31T20:26:43.491 DEBUG:boto:auth_path=/pklxkd-53/b 2023-05-31T20:26:43.491 DEBUG:boto:Method: GET 2023-05-31T20:26:43.492 DEBUG:boto:Path: /pklxkd-53/b 2023-05-31T20:26:43.492 DEBUG:boto:Data: 2023-05-31T20:26:43.496 DEBUG:boto:Headers: {} 2023-05-31T20:26:43.496 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:26:43.496 DEBUG:boto:Port: 8000 2023-05-31T20:26:43.497 DEBUG:boto:Params: {} 2023-05-31T20:26:43.497 DEBUG:boto:Token: None 2023-05-31T20:26:43.498 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:26:43 GMT /pklxkd-53/b 2023-05-31T20:26:43.498 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:yhXmulTkH3JK8ITqhywI/xJxJgM= 2023-05-31T20:26:43.499 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:26:43 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:yhXmulTkH3JK8ITqhywI/xJxJgM=', 'Content-Length': '0'} 2023-05-31T20:26:43.500 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:26:27 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006250ad9534271928-006477ad83-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:26:43 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:26:43.501 DEBUG:boto:path=/pklxkd-53/ 2023-05-31T20:26:43.501 DEBUG:boto:auth_path=/pklxkd-53/ 2023-05-31T20:26:43.502 DEBUG:boto:path=/pklxkd-53/?prefix=b 2023-05-31T20:26:43.502 DEBUG:boto:auth_path=/pklxkd-53/?prefix=b 2023-05-31T20:26:43.502 DEBUG:boto:Method: GET 2023-05-31T20:26:43.503 DEBUG:boto:Path: /pklxkd-53/?prefix=b 2023-05-31T20:26:43.503 DEBUG:boto:Data: 2023-05-31T20:26:43.503 DEBUG:boto:Headers: {} 2023-05-31T20:26:43.504 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:26:43.504 DEBUG:boto:Port: 8001 2023-05-31T20:26:43.504 DEBUG:boto:Params: {} 2023-05-31T20:26:43.505 DEBUG:boto:Token: None 2023-05-31T20:26:43.505 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:26:43 GMT /pklxkd-53/ 2023-05-31T20:26:43.506 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:RtuWiOhPp2v0s+P6tP4ZjzNQme0= 2023-05-31T20:26:43.506 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:26:43 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:RtuWiOhPp2v0s+P6tP4ZjzNQme0=', 'Content-Length': '0'} 2023-05-31T20:26:43.508 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a37aa412836b99b9-006477ad83-162e-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:26:43 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:26:43.509 DEBUG:boto:b'pklxkd-53b1000falseb2023-05-31T20:26:27.529Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:26:43.509 DEBUG:boto:path=/pklxkd-53/ 2023-05-31T20:26:43.510 DEBUG:boto:auth_path=/pklxkd-53/ 2023-05-31T20:26:43.510 DEBUG:boto:path=/pklxkd-53/?prefix=b 2023-05-31T20:26:43.510 DEBUG:boto:auth_path=/pklxkd-53/?prefix=b 2023-05-31T20:26:43.511 DEBUG:boto:Method: GET 2023-05-31T20:26:43.511 DEBUG:boto:Path: /pklxkd-53/?prefix=b 2023-05-31T20:26:43.512 DEBUG:boto:Data: 2023-05-31T20:26:43.512 DEBUG:boto:Headers: {} 2023-05-31T20:26:43.512 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:26:43.513 DEBUG:boto:Port: 8000 2023-05-31T20:26:43.513 DEBUG:boto:Params: {} 2023-05-31T20:26:43.514 DEBUG:boto:Token: None 2023-05-31T20:26:43.514 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:26:43 GMT /pklxkd-53/ 2023-05-31T20:26:43.514 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:RtuWiOhPp2v0s+P6tP4ZjzNQme0= 2023-05-31T20:26:43.515 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:26:43 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:RtuWiOhPp2v0s+P6tP4ZjzNQme0=', 'Content-Length': '0'} 2023-05-31T20:26:43.517 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000007a748d7f0a1aa98f-006477ad83-1630-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:26:43 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:26:43.564 DEBUG:boto:b'pklxkd-53b1000falseb2023-05-31T20:26:27.529Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:26:43.565 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=c 2023-05-31T20:26:43.566 DEBUG:boto:path=/pklxkd-53/c 2023-05-31T20:26:43.566 DEBUG:boto:auth_path=/pklxkd-53/c 2023-05-31T20:26:43.566 DEBUG:boto:path=/pklxkd-53/c?versionId=null 2023-05-31T20:26:43.567 DEBUG:boto:auth_path=/pklxkd-53/c?versionId=null 2023-05-31T20:26:43.567 DEBUG:boto:Method: GET 2023-05-31T20:26:43.567 DEBUG:boto:Path: /pklxkd-53/c?versionId=null 2023-05-31T20:26:43.568 DEBUG:boto:Data: 2023-05-31T20:26:43.569 DEBUG:boto:Headers: {} 2023-05-31T20:26:43.569 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:26:43.569 DEBUG:boto:Port: 8001 2023-05-31T20:26:43.570 DEBUG:boto:Params: {} 2023-05-31T20:26:43.570 DEBUG:boto:Token: None 2023-05-31T20:26:43.571 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:26:43 GMT /pklxkd-53/c?versionId=null 2023-05-31T20:26:43.571 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:ZVu6JOnO0eUv3a4F+MKQjh/daAU= 2023-05-31T20:26:43.571 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:26:43 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:ZVu6JOnO0eUv3a4F+MKQjh/daAU=', 'Content-Length': '0'} 2023-05-31T20:26:43.574 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:26:27 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000511183f2e596d6fe-006477ad83-162e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:26:43 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:26:43.575 DEBUG:boto:path=/pklxkd-53/c 2023-05-31T20:26:43.575 DEBUG:boto:auth_path=/pklxkd-53/c 2023-05-31T20:26:43.575 DEBUG:boto:path=/pklxkd-53/c?versionId=null 2023-05-31T20:26:43.576 DEBUG:boto:auth_path=/pklxkd-53/c?versionId=null 2023-05-31T20:26:43.576 DEBUG:boto:Method: GET 2023-05-31T20:26:43.577 DEBUG:boto:Path: /pklxkd-53/c?versionId=null 2023-05-31T20:26:43.577 DEBUG:boto:Data: 2023-05-31T20:26:43.578 DEBUG:boto:Headers: {} 2023-05-31T20:26:43.578 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:26:43.579 DEBUG:boto:Port: 8000 2023-05-31T20:26:43.579 DEBUG:boto:Params: {} 2023-05-31T20:26:43.579 DEBUG:boto:Token: None 2023-05-31T20:26:43.580 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:26:43 GMT /pklxkd-53/c?versionId=null 2023-05-31T20:26:43.580 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:ZVu6JOnO0eUv3a4F+MKQjh/daAU= 2023-05-31T20:26:43.581 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:26:43 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:ZVu6JOnO0eUv3a4F+MKQjh/daAU=', 'Content-Length': '0'} 2023-05-31T20:26:43.583 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:26:27 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f9dcc2b1b1752fe2-006477ad83-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:26:43 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:26:43.584 DEBUG:boto:path=/pklxkd-53/c 2023-05-31T20:26:43.584 DEBUG:boto:auth_path=/pklxkd-53/c 2023-05-31T20:26:43.585 DEBUG:boto:path=/pklxkd-53/c?versionId=null 2023-05-31T20:26:43.585 DEBUG:boto:auth_path=/pklxkd-53/c?versionId=null 2023-05-31T20:26:43.585 DEBUG:boto:Method: HEAD 2023-05-31T20:26:43.586 DEBUG:boto:Path: /pklxkd-53/c?versionId=null 2023-05-31T20:26:43.586 DEBUG:boto:Data: 2023-05-31T20:26:43.586 DEBUG:boto:Headers: {} 2023-05-31T20:26:43.587 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:26:43.587 DEBUG:boto:Port: 8001 2023-05-31T20:26:43.587 DEBUG:boto:Params: {} 2023-05-31T20:26:43.588 DEBUG:boto:Token: None 2023-05-31T20:26:43.588 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:26:43 GMT /pklxkd-53/c?versionId=null 2023-05-31T20:26:43.588 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:UFg2A3CFmECzzN9dXLSQriX4Dmo= 2023-05-31T20:26:43.589 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:26:43 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:UFg2A3CFmECzzN9dXLSQriX4Dmo=', 'Content-Length': '0'} 2023-05-31T20:26:43.591 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:26:27 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000937558972c434371-006477ad83-162e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:26:43 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:26:43.591 DEBUG:boto:path=/pklxkd-53/c 2023-05-31T20:26:43.592 DEBUG:boto:auth_path=/pklxkd-53/c 2023-05-31T20:26:43.593 DEBUG:boto:path=/pklxkd-53/c?versionId=null 2023-05-31T20:26:43.593 DEBUG:boto:auth_path=/pklxkd-53/c?versionId=null 2023-05-31T20:26:43.594 DEBUG:boto:Method: HEAD 2023-05-31T20:26:43.594 DEBUG:boto:Path: /pklxkd-53/c?versionId=null 2023-05-31T20:26:43.594 DEBUG:boto:Data: 2023-05-31T20:26:43.595 DEBUG:boto:Headers: {} 2023-05-31T20:26:43.595 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:26:43.595 DEBUG:boto:Port: 8000 2023-05-31T20:26:43.596 DEBUG:boto:Params: {} 2023-05-31T20:26:43.596 DEBUG:boto:Token: None 2023-05-31T20:26:43.597 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:26:43 GMT /pklxkd-53/c?versionId=null 2023-05-31T20:26:43.597 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:UFg2A3CFmECzzN9dXLSQriX4Dmo= 2023-05-31T20:26:43.598 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:26:43 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:UFg2A3CFmECzzN9dXLSQriX4Dmo=', 'Content-Length': '0'} 2023-05-31T20:26:43.599 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:26:27 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008cd2aa71021f73a7-006477ad83-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:26:43 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:26:43.600 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=c 2023-05-31T20:26:43.600 DEBUG:boto:path=/pklxkd-53/c 2023-05-31T20:26:43.601 DEBUG:boto:auth_path=/pklxkd-53/c 2023-05-31T20:26:43.601 DEBUG:boto:path=/pklxkd-53/c?versionId=null 2023-05-31T20:26:43.601 DEBUG:boto:auth_path=/pklxkd-53/c?versionId=null 2023-05-31T20:26:43.602 DEBUG:boto:Method: GET 2023-05-31T20:26:43.602 DEBUG:boto:Path: /pklxkd-53/c?versionId=null 2023-05-31T20:26:43.602 DEBUG:boto:Data: 2023-05-31T20:26:43.603 DEBUG:boto:Headers: {} 2023-05-31T20:26:43.603 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:26:43.604 DEBUG:boto:Port: 8001 2023-05-31T20:26:43.604 DEBUG:boto:Params: {} 2023-05-31T20:26:43.604 DEBUG:boto:Token: None 2023-05-31T20:26:43.605 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:26:43 GMT /pklxkd-53/c?versionId=null 2023-05-31T20:26:43.605 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:ZVu6JOnO0eUv3a4F+MKQjh/daAU= 2023-05-31T20:26:43.606 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:26:43 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:ZVu6JOnO0eUv3a4F+MKQjh/daAU=', 'Content-Length': '0'} 2023-05-31T20:26:43.607 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:26:27 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007e0888144c1b9cf2-006477ad83-162e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:26:43 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:26:43.608 DEBUG:boto:path=/pklxkd-53/c 2023-05-31T20:26:43.608 DEBUG:boto:auth_path=/pklxkd-53/c 2023-05-31T20:26:43.609 DEBUG:boto:path=/pklxkd-53/c?versionId=null 2023-05-31T20:26:43.609 DEBUG:boto:auth_path=/pklxkd-53/c?versionId=null 2023-05-31T20:26:43.609 DEBUG:boto:Method: GET 2023-05-31T20:26:43.610 DEBUG:boto:Path: /pklxkd-53/c?versionId=null 2023-05-31T20:26:43.610 DEBUG:boto:Data: 2023-05-31T20:26:43.611 DEBUG:boto:Headers: {} 2023-05-31T20:26:43.611 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:26:43.611 DEBUG:boto:Port: 8000 2023-05-31T20:26:43.612 DEBUG:boto:Params: {} 2023-05-31T20:26:43.612 DEBUG:boto:Token: None 2023-05-31T20:26:43.612 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:26:43 GMT /pklxkd-53/c?versionId=null 2023-05-31T20:26:43.613 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:ZVu6JOnO0eUv3a4F+MKQjh/daAU= 2023-05-31T20:26:43.613 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:26:43 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:ZVu6JOnO0eUv3a4F+MKQjh/daAU=', 'Content-Length': '0'} 2023-05-31T20:26:43.615 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:26:27 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a8b7abc0c1fde164-006477ad83-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:26:43 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:26:43.616 DEBUG:boto:path=/pklxkd-53/ 2023-05-31T20:26:43.616 DEBUG:boto:auth_path=/pklxkd-53/ 2023-05-31T20:26:43.616 DEBUG:boto:path=/pklxkd-53/?prefix=c 2023-05-31T20:26:43.617 DEBUG:boto:auth_path=/pklxkd-53/?prefix=c 2023-05-31T20:26:43.617 DEBUG:boto:Method: GET 2023-05-31T20:26:43.617 DEBUG:boto:Path: /pklxkd-53/?prefix=c 2023-05-31T20:26:43.618 DEBUG:boto:Data: 2023-05-31T20:26:43.618 DEBUG:boto:Headers: {} 2023-05-31T20:26:43.619 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:26:43.619 DEBUG:boto:Port: 8001 2023-05-31T20:26:43.619 DEBUG:boto:Params: {} 2023-05-31T20:26:43.620 DEBUG:boto:Token: None 2023-05-31T20:26:43.620 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:26:43 GMT /pklxkd-53/ 2023-05-31T20:26:43.621 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:RtuWiOhPp2v0s+P6tP4ZjzNQme0= 2023-05-31T20:26:43.621 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:26:43 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:RtuWiOhPp2v0s+P6tP4ZjzNQme0=', 'Content-Length': '0'} 2023-05-31T20:26:43.623 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d44ae2fdafd101ab-006477ad83-162e-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:26:43 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:26:43.623 DEBUG:boto:b'pklxkd-53c1000falsec2023-05-31T20:26:27.548Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:26:43.624 DEBUG:boto:path=/pklxkd-53/ 2023-05-31T20:26:43.625 DEBUG:boto:auth_path=/pklxkd-53/ 2023-05-31T20:26:43.625 DEBUG:boto:path=/pklxkd-53/?prefix=c 2023-05-31T20:26:43.626 DEBUG:boto:auth_path=/pklxkd-53/?prefix=c 2023-05-31T20:26:43.626 DEBUG:boto:Method: GET 2023-05-31T20:26:43.626 DEBUG:boto:Path: /pklxkd-53/?prefix=c 2023-05-31T20:26:43.627 DEBUG:boto:Data: 2023-05-31T20:26:43.627 DEBUG:boto:Headers: {} 2023-05-31T20:26:43.627 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:26:43.628 DEBUG:boto:Port: 8000 2023-05-31T20:26:43.628 DEBUG:boto:Params: {} 2023-05-31T20:26:43.629 DEBUG:boto:Token: None 2023-05-31T20:26:43.629 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:26:43 GMT /pklxkd-53/ 2023-05-31T20:26:43.629 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:RtuWiOhPp2v0s+P6tP4ZjzNQme0= 2023-05-31T20:26:43.630 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:26:43 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:RtuWiOhPp2v0s+P6tP4ZjzNQme0=', 'Content-Length': '0'} 2023-05-31T20:26:43.632 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000001fec9e1acf4de733-006477ad83-1630-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:26:43 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:26:43.632 DEBUG:boto:b'pklxkd-53c1000falsec2023-05-31T20:26:27.548Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:26:43.633 DEBUG:boto:path=/pklxkd-53/c 2023-05-31T20:26:43.634 DEBUG:boto:auth_path=/pklxkd-53/c 2023-05-31T20:26:43.634 DEBUG:boto:Method: HEAD 2023-05-31T20:26:43.634 DEBUG:boto:Path: /pklxkd-53/c 2023-05-31T20:26:43.635 DEBUG:boto:Data: 2023-05-31T20:26:43.635 DEBUG:boto:Headers: {} 2023-05-31T20:26:43.635 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:26:43.636 DEBUG:boto:Port: 8001 2023-05-31T20:26:43.636 DEBUG:boto:Params: {} 2023-05-31T20:26:43.637 DEBUG:boto:Token: None 2023-05-31T20:26:43.637 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:26:43 GMT /pklxkd-53/c 2023-05-31T20:26:43.637 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:4dzoxEPvKbKXgSG9JYS4mnOfvYQ= 2023-05-31T20:26:43.638 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:26:43 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:4dzoxEPvKbKXgSG9JYS4mnOfvYQ=', 'Content-Length': '0'} 2023-05-31T20:26:43.640 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:26:27 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000082e1ea3b702c37d8-006477ad83-162e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:26:43 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:26:43.640 DEBUG:boto:path=/pklxkd-53/c 2023-05-31T20:26:43.641 DEBUG:boto:auth_path=/pklxkd-53/c 2023-05-31T20:26:43.641 DEBUG:boto:Method: HEAD 2023-05-31T20:26:43.641 DEBUG:boto:Path: /pklxkd-53/c 2023-05-31T20:26:43.642 DEBUG:boto:Data: 2023-05-31T20:26:43.642 DEBUG:boto:Headers: {} 2023-05-31T20:26:43.642 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:26:43.643 DEBUG:boto:Port: 8000 2023-05-31T20:26:43.643 DEBUG:boto:Params: {} 2023-05-31T20:26:43.644 DEBUG:boto:Token: None 2023-05-31T20:26:43.644 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:26:43 GMT /pklxkd-53/c 2023-05-31T20:26:43.644 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:4dzoxEPvKbKXgSG9JYS4mnOfvYQ= 2023-05-31T20:26:43.645 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:26:43 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:4dzoxEPvKbKXgSG9JYS4mnOfvYQ=', 'Content-Length': '0'} 2023-05-31T20:26:43.646 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:26:27 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007c42ca5b18dd18a7-006477ad83-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:26:43 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:26:43.647 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=c 2023-05-31T20:26:43.647 DEBUG:boto:path=/pklxkd-53/c 2023-05-31T20:26:43.648 DEBUG:boto:auth_path=/pklxkd-53/c 2023-05-31T20:26:43.648 DEBUG:boto:Method: GET 2023-05-31T20:26:43.649 DEBUG:boto:Path: /pklxkd-53/c 2023-05-31T20:26:43.649 DEBUG:boto:Data: 2023-05-31T20:26:43.650 DEBUG:boto:Headers: {} 2023-05-31T20:26:43.650 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:26:43.650 DEBUG:boto:Port: 8001 2023-05-31T20:26:43.651 DEBUG:boto:Params: {} 2023-05-31T20:26:43.651 DEBUG:boto:Token: None 2023-05-31T20:26:43.652 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:26:43 GMT /pklxkd-53/c 2023-05-31T20:26:43.654 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:QV5OoZKtAVhrMQLiIPtNiOo/fiw= 2023-05-31T20:26:43.654 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:26:43 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:QV5OoZKtAVhrMQLiIPtNiOo/fiw=', 'Content-Length': '0'} 2023-05-31T20:26:43.657 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:26:27 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000282232d599e69ead-006477ad83-162e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:26:43 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:26:43.658 DEBUG:boto:path=/pklxkd-53/c 2023-05-31T20:26:43.659 DEBUG:boto:auth_path=/pklxkd-53/c 2023-05-31T20:26:43.660 DEBUG:boto:Method: GET 2023-05-31T20:26:43.661 DEBUG:boto:Path: /pklxkd-53/c 2023-05-31T20:26:43.661 DEBUG:boto:Data: 2023-05-31T20:26:43.662 DEBUG:boto:Headers: {} 2023-05-31T20:26:43.663 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:26:43.663 DEBUG:boto:Port: 8000 2023-05-31T20:26:43.664 DEBUG:boto:Params: {} 2023-05-31T20:26:43.664 DEBUG:boto:Token: None 2023-05-31T20:26:43.665 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:26:43 GMT /pklxkd-53/c 2023-05-31T20:26:43.665 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:QV5OoZKtAVhrMQLiIPtNiOo/fiw= 2023-05-31T20:26:43.666 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:26:43 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:QV5OoZKtAVhrMQLiIPtNiOo/fiw=', 'Content-Length': '0'} 2023-05-31T20:26:43.667 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:26:27 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000be12ba6286cddd1-006477ad83-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:26:43 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:26:43.668 DEBUG:boto:path=/pklxkd-53/ 2023-05-31T20:26:43.669 DEBUG:boto:auth_path=/pklxkd-53/ 2023-05-31T20:26:43.669 DEBUG:boto:path=/pklxkd-53/?prefix=c 2023-05-31T20:26:43.669 DEBUG:boto:auth_path=/pklxkd-53/?prefix=c 2023-05-31T20:26:43.670 DEBUG:boto:Method: GET 2023-05-31T20:26:43.670 DEBUG:boto:Path: /pklxkd-53/?prefix=c 2023-05-31T20:26:43.671 DEBUG:boto:Data: 2023-05-31T20:26:43.671 DEBUG:boto:Headers: {} 2023-05-31T20:26:43.671 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:26:43.672 DEBUG:boto:Port: 8001 2023-05-31T20:26:43.672 DEBUG:boto:Params: {} 2023-05-31T20:26:43.673 DEBUG:boto:Token: None 2023-05-31T20:26:43.674 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:26:43 GMT /pklxkd-53/ 2023-05-31T20:26:43.675 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:RtuWiOhPp2v0s+P6tP4ZjzNQme0= 2023-05-31T20:26:43.675 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:26:43 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:RtuWiOhPp2v0s+P6tP4ZjzNQme0=', 'Content-Length': '0'} 2023-05-31T20:26:43.678 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b729addb46065d04-006477ad83-162e-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:26:43 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:26:43.720 DEBUG:boto:b'pklxkd-53c1000falsec2023-05-31T20:26:27.548Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:26:43.721 DEBUG:boto:path=/pklxkd-53/ 2023-05-31T20:26:43.722 DEBUG:boto:auth_path=/pklxkd-53/ 2023-05-31T20:26:43.722 DEBUG:boto:path=/pklxkd-53/?prefix=c 2023-05-31T20:26:43.723 DEBUG:boto:auth_path=/pklxkd-53/?prefix=c 2023-05-31T20:26:43.723 DEBUG:boto:Method: GET 2023-05-31T20:26:43.723 DEBUG:boto:Path: /pklxkd-53/?prefix=c 2023-05-31T20:26:43.724 DEBUG:boto:Data: 2023-05-31T20:26:43.724 DEBUG:boto:Headers: {} 2023-05-31T20:26:43.725 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:26:43.725 DEBUG:boto:Port: 8000 2023-05-31T20:26:43.726 DEBUG:boto:Params: {} 2023-05-31T20:26:43.726 DEBUG:boto:Token: None 2023-05-31T20:26:43.727 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:26:43 GMT /pklxkd-53/ 2023-05-31T20:26:43.727 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:RtuWiOhPp2v0s+P6tP4ZjzNQme0= 2023-05-31T20:26:43.728 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:26:43 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:RtuWiOhPp2v0s+P6tP4ZjzNQme0=', 'Content-Length': '0'} 2023-05-31T20:26:43.730 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000052421fb19d5658c7-006477ad83-1630-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:26:43 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:26:43.731 DEBUG:boto:b'pklxkd-53c1000falsec2023-05-31T20:26:27.548Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:26:43.731 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=d 2023-05-31T20:26:43.732 DEBUG:boto:path=/pklxkd-53/d 2023-05-31T20:26:43.733 DEBUG:boto:auth_path=/pklxkd-53/d 2023-05-31T20:26:43.733 DEBUG:boto:path=/pklxkd-53/d?versionId=null 2023-05-31T20:26:43.734 DEBUG:boto:auth_path=/pklxkd-53/d?versionId=null 2023-05-31T20:26:43.734 DEBUG:boto:Method: GET 2023-05-31T20:26:43.735 DEBUG:boto:Path: /pklxkd-53/d?versionId=null 2023-05-31T20:26:43.736 DEBUG:boto:Data: 2023-05-31T20:26:43.737 DEBUG:boto:Headers: {} 2023-05-31T20:26:43.738 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:26:43.739 DEBUG:boto:Port: 8001 2023-05-31T20:26:43.739 DEBUG:boto:Params: {} 2023-05-31T20:26:43.740 DEBUG:boto:Token: None 2023-05-31T20:26:43.741 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:26:43 GMT /pklxkd-53/d?versionId=null 2023-05-31T20:26:43.742 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:k9D9WdQoA6fUQWjYd0eIA6j3xkM= 2023-05-31T20:26:43.743 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:26:43 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:k9D9WdQoA6fUQWjYd0eIA6j3xkM=', 'Content-Length': '0'} 2023-05-31T20:26:43.745 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:26:27 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000eeaf61373d7505dc-006477ad83-162e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:26:43 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:26:43.746 DEBUG:boto:path=/pklxkd-53/d 2023-05-31T20:26:43.747 DEBUG:boto:auth_path=/pklxkd-53/d 2023-05-31T20:26:43.747 DEBUG:boto:path=/pklxkd-53/d?versionId=null 2023-05-31T20:26:43.748 DEBUG:boto:auth_path=/pklxkd-53/d?versionId=null 2023-05-31T20:26:43.749 DEBUG:boto:Method: GET 2023-05-31T20:26:43.749 DEBUG:boto:Path: /pklxkd-53/d?versionId=null 2023-05-31T20:26:43.750 DEBUG:boto:Data: 2023-05-31T20:26:43.751 DEBUG:boto:Headers: {} 2023-05-31T20:26:43.751 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:26:43.752 DEBUG:boto:Port: 8000 2023-05-31T20:26:43.753 DEBUG:boto:Params: {} 2023-05-31T20:26:43.753 DEBUG:boto:Token: None 2023-05-31T20:26:43.753 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:26:43 GMT /pklxkd-53/d?versionId=null 2023-05-31T20:26:43.754 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:k9D9WdQoA6fUQWjYd0eIA6j3xkM= 2023-05-31T20:26:43.754 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:26:43 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:k9D9WdQoA6fUQWjYd0eIA6j3xkM=', 'Content-Length': '0'} 2023-05-31T20:26:43.756 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:26:27 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006b120039a9533330-006477ad83-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:26:43 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:26:43.757 DEBUG:boto:path=/pklxkd-53/d 2023-05-31T20:26:43.757 DEBUG:boto:auth_path=/pklxkd-53/d 2023-05-31T20:26:43.758 DEBUG:boto:path=/pklxkd-53/d?versionId=null 2023-05-31T20:26:43.758 DEBUG:boto:auth_path=/pklxkd-53/d?versionId=null 2023-05-31T20:26:43.758 DEBUG:boto:Method: HEAD 2023-05-31T20:26:43.759 DEBUG:boto:Path: /pklxkd-53/d?versionId=null 2023-05-31T20:26:43.759 DEBUG:boto:Data: 2023-05-31T20:26:43.760 DEBUG:boto:Headers: {} 2023-05-31T20:26:43.760 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:26:43.760 DEBUG:boto:Port: 8001 2023-05-31T20:26:43.761 DEBUG:boto:Params: {} 2023-05-31T20:26:43.761 DEBUG:boto:Token: None 2023-05-31T20:26:43.761 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:26:43 GMT /pklxkd-53/d?versionId=null 2023-05-31T20:26:43.762 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:LpAwn2MSCftaRdPnHvErSUy8Ekc= 2023-05-31T20:26:43.762 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:26:43 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:LpAwn2MSCftaRdPnHvErSUy8Ekc=', 'Content-Length': '0'} 2023-05-31T20:26:43.764 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:26:27 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e1f46a65e31c6bbe-006477ad83-162e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:26:43 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:26:43.765 DEBUG:boto:path=/pklxkd-53/d 2023-05-31T20:26:43.765 DEBUG:boto:auth_path=/pklxkd-53/d 2023-05-31T20:26:43.765 DEBUG:boto:path=/pklxkd-53/d?versionId=null 2023-05-31T20:26:43.766 DEBUG:boto:auth_path=/pklxkd-53/d?versionId=null 2023-05-31T20:26:43.766 DEBUG:boto:Method: HEAD 2023-05-31T20:26:43.766 DEBUG:boto:Path: /pklxkd-53/d?versionId=null 2023-05-31T20:26:43.767 DEBUG:boto:Data: 2023-05-31T20:26:43.767 DEBUG:boto:Headers: {} 2023-05-31T20:26:43.767 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:26:43.768 DEBUG:boto:Port: 8000 2023-05-31T20:26:43.768 DEBUG:boto:Params: {} 2023-05-31T20:26:43.768 DEBUG:boto:Token: None 2023-05-31T20:26:43.769 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:26:43 GMT /pklxkd-53/d?versionId=null 2023-05-31T20:26:43.769 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:LpAwn2MSCftaRdPnHvErSUy8Ekc= 2023-05-31T20:26:43.770 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:26:43 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:LpAwn2MSCftaRdPnHvErSUy8Ekc=', 'Content-Length': '0'} 2023-05-31T20:26:43.771 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:26:27 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000092407de9c7158533-006477ad83-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:26:43 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:26:43.772 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=d 2023-05-31T20:26:43.772 DEBUG:boto:path=/pklxkd-53/d 2023-05-31T20:26:43.773 DEBUG:boto:auth_path=/pklxkd-53/d 2023-05-31T20:26:43.774 DEBUG:boto:path=/pklxkd-53/d?versionId=null 2023-05-31T20:26:43.774 DEBUG:boto:auth_path=/pklxkd-53/d?versionId=null 2023-05-31T20:26:43.774 DEBUG:boto:Method: GET 2023-05-31T20:26:43.775 DEBUG:boto:Path: /pklxkd-53/d?versionId=null 2023-05-31T20:26:43.775 DEBUG:boto:Data: 2023-05-31T20:26:43.775 DEBUG:boto:Headers: {} 2023-05-31T20:26:43.776 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:26:43.776 DEBUG:boto:Port: 8001 2023-05-31T20:26:43.776 DEBUG:boto:Params: {} 2023-05-31T20:26:43.777 DEBUG:boto:Token: None 2023-05-31T20:26:43.777 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:26:43 GMT /pklxkd-53/d?versionId=null 2023-05-31T20:26:43.778 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:k9D9WdQoA6fUQWjYd0eIA6j3xkM= 2023-05-31T20:26:43.778 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:26:43 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:k9D9WdQoA6fUQWjYd0eIA6j3xkM=', 'Content-Length': '0'} 2023-05-31T20:26:43.780 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:26:27 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003babb21a1bc5cbaa-006477ad83-162e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:26:43 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:26:43.780 DEBUG:boto:path=/pklxkd-53/d 2023-05-31T20:26:43.781 DEBUG:boto:auth_path=/pklxkd-53/d 2023-05-31T20:26:43.781 DEBUG:boto:path=/pklxkd-53/d?versionId=null 2023-05-31T20:26:43.781 DEBUG:boto:auth_path=/pklxkd-53/d?versionId=null 2023-05-31T20:26:43.782 DEBUG:boto:Method: GET 2023-05-31T20:26:43.782 DEBUG:boto:Path: /pklxkd-53/d?versionId=null 2023-05-31T20:26:43.783 DEBUG:boto:Data: 2023-05-31T20:26:43.783 DEBUG:boto:Headers: {} 2023-05-31T20:26:43.784 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:26:43.784 DEBUG:boto:Port: 8000 2023-05-31T20:26:43.784 DEBUG:boto:Params: {} 2023-05-31T20:26:43.785 DEBUG:boto:Token: None 2023-05-31T20:26:43.785 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:26:43 GMT /pklxkd-53/d?versionId=null 2023-05-31T20:26:43.786 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:k9D9WdQoA6fUQWjYd0eIA6j3xkM= 2023-05-31T20:26:43.786 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:26:43 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:k9D9WdQoA6fUQWjYd0eIA6j3xkM=', 'Content-Length': '0'} 2023-05-31T20:26:43.788 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:26:27 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000dd138d07e8d54f91-006477ad83-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:26:43 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:26:43.789 DEBUG:boto:path=/pklxkd-53/ 2023-05-31T20:26:43.789 DEBUG:boto:auth_path=/pklxkd-53/ 2023-05-31T20:26:43.789 DEBUG:boto:path=/pklxkd-53/?prefix=d 2023-05-31T20:26:43.790 DEBUG:boto:auth_path=/pklxkd-53/?prefix=d 2023-05-31T20:26:43.790 DEBUG:boto:Method: GET 2023-05-31T20:26:43.791 DEBUG:boto:Path: /pklxkd-53/?prefix=d 2023-05-31T20:26:43.791 DEBUG:boto:Data: 2023-05-31T20:26:43.791 DEBUG:boto:Headers: {} 2023-05-31T20:26:43.792 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:26:43.792 DEBUG:boto:Port: 8001 2023-05-31T20:26:43.793 DEBUG:boto:Params: {} 2023-05-31T20:26:43.793 DEBUG:boto:Token: None 2023-05-31T20:26:43.794 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:26:43 GMT /pklxkd-53/ 2023-05-31T20:26:43.794 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:RtuWiOhPp2v0s+P6tP4ZjzNQme0= 2023-05-31T20:26:43.795 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:26:43 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:RtuWiOhPp2v0s+P6tP4ZjzNQme0=', 'Content-Length': '0'} 2023-05-31T20:26:43.797 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000009488ac2b5f9e6af9-006477ad83-162e-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:26:43 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:26:43.797 DEBUG:boto:b'pklxkd-53d1000falsed2023-05-31T20:26:27.602Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:26:43.798 DEBUG:boto:path=/pklxkd-53/ 2023-05-31T20:26:43.798 DEBUG:boto:auth_path=/pklxkd-53/ 2023-05-31T20:26:43.799 DEBUG:boto:path=/pklxkd-53/?prefix=d 2023-05-31T20:26:43.799 DEBUG:boto:auth_path=/pklxkd-53/?prefix=d 2023-05-31T20:26:43.799 DEBUG:boto:Method: GET 2023-05-31T20:26:43.800 DEBUG:boto:Path: /pklxkd-53/?prefix=d 2023-05-31T20:26:43.800 DEBUG:boto:Data: 2023-05-31T20:26:43.800 DEBUG:boto:Headers: {} 2023-05-31T20:26:43.801 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:26:43.801 DEBUG:boto:Port: 8000 2023-05-31T20:26:43.801 DEBUG:boto:Params: {} 2023-05-31T20:26:43.802 DEBUG:boto:Token: None 2023-05-31T20:26:43.802 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:26:43 GMT /pklxkd-53/ 2023-05-31T20:26:43.803 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:RtuWiOhPp2v0s+P6tP4ZjzNQme0= 2023-05-31T20:26:43.803 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:26:43 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:RtuWiOhPp2v0s+P6tP4ZjzNQme0=', 'Content-Length': '0'} 2023-05-31T20:26:43.805 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000072ac1f9339eb5901-006477ad83-1630-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:26:43 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:26:43.806 DEBUG:boto:b'pklxkd-53d1000falsed2023-05-31T20:26:27.602Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:26:43.806 DEBUG:boto:path=/pklxkd-53/d 2023-05-31T20:26:43.807 DEBUG:boto:auth_path=/pklxkd-53/d 2023-05-31T20:26:43.807 DEBUG:boto:Method: HEAD 2023-05-31T20:26:43.808 DEBUG:boto:Path: /pklxkd-53/d 2023-05-31T20:26:43.808 DEBUG:boto:Data: 2023-05-31T20:26:43.808 DEBUG:boto:Headers: {} 2023-05-31T20:26:43.809 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:26:43.809 DEBUG:boto:Port: 8001 2023-05-31T20:26:43.809 DEBUG:boto:Params: {} 2023-05-31T20:26:43.810 DEBUG:boto:Token: None 2023-05-31T20:26:43.810 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:26:43 GMT /pklxkd-53/d 2023-05-31T20:26:43.810 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:kppAhzioTSDsjN9uSenqyISMvVI= 2023-05-31T20:26:43.811 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:26:43 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:kppAhzioTSDsjN9uSenqyISMvVI=', 'Content-Length': '0'} 2023-05-31T20:26:43.812 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:26:27 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e9fb830056c0a023-006477ad83-162e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:26:43 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:26:43.813 DEBUG:boto:path=/pklxkd-53/d 2023-05-31T20:26:43.813 DEBUG:boto:auth_path=/pklxkd-53/d 2023-05-31T20:26:43.814 DEBUG:boto:Method: HEAD 2023-05-31T20:26:43.814 DEBUG:boto:Path: /pklxkd-53/d 2023-05-31T20:26:43.815 DEBUG:boto:Data: 2023-05-31T20:26:43.815 DEBUG:boto:Headers: {} 2023-05-31T20:26:43.815 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:26:43.816 DEBUG:boto:Port: 8000 2023-05-31T20:26:43.816 DEBUG:boto:Params: {} 2023-05-31T20:26:43.817 DEBUG:boto:Token: None 2023-05-31T20:26:43.817 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:26:43 GMT /pklxkd-53/d 2023-05-31T20:26:43.818 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:kppAhzioTSDsjN9uSenqyISMvVI= 2023-05-31T20:26:43.818 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:26:43 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:kppAhzioTSDsjN9uSenqyISMvVI=', 'Content-Length': '0'} 2023-05-31T20:26:43.820 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:26:27 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d0edbd380c8fc0fb-006477ad83-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:26:43 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:26:43.820 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=d 2023-05-31T20:26:43.821 DEBUG:boto:path=/pklxkd-53/d 2023-05-31T20:26:43.821 DEBUG:boto:auth_path=/pklxkd-53/d 2023-05-31T20:26:43.821 DEBUG:boto:Method: GET 2023-05-31T20:26:43.822 DEBUG:boto:Path: /pklxkd-53/d 2023-05-31T20:26:43.822 DEBUG:boto:Data: 2023-05-31T20:26:43.822 DEBUG:boto:Headers: {} 2023-05-31T20:26:43.823 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:26:43.823 DEBUG:boto:Port: 8001 2023-05-31T20:26:43.824 DEBUG:boto:Params: {} 2023-05-31T20:26:43.824 DEBUG:boto:Token: None 2023-05-31T20:26:43.824 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:26:43 GMT /pklxkd-53/d 2023-05-31T20:26:43.825 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:+Go6EtxNUtsoyfTdBTJnhWYwrkI= 2023-05-31T20:26:43.825 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:26:43 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:+Go6EtxNUtsoyfTdBTJnhWYwrkI=', 'Content-Length': '0'} 2023-05-31T20:26:43.827 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:26:27 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ca882bc939c7e355-006477ad83-162e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:26:43 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:26:43.868 DEBUG:boto:path=/pklxkd-53/d 2023-05-31T20:26:43.869 DEBUG:boto:auth_path=/pklxkd-53/d 2023-05-31T20:26:43.869 DEBUG:boto:Method: GET 2023-05-31T20:26:43.870 DEBUG:boto:Path: /pklxkd-53/d 2023-05-31T20:26:43.870 DEBUG:boto:Data: 2023-05-31T20:26:43.871 DEBUG:boto:Headers: {} 2023-05-31T20:26:43.872 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:26:43.873 DEBUG:boto:Port: 8000 2023-05-31T20:26:43.874 DEBUG:boto:Params: {} 2023-05-31T20:26:43.875 DEBUG:boto:Token: None 2023-05-31T20:26:43.876 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:26:43 GMT /pklxkd-53/d 2023-05-31T20:26:43.876 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:+Go6EtxNUtsoyfTdBTJnhWYwrkI= 2023-05-31T20:26:43.877 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:26:43 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:+Go6EtxNUtsoyfTdBTJnhWYwrkI=', 'Content-Length': '0'} 2023-05-31T20:26:43.879 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:26:27 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008205f1d42079b431-006477ad83-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:26:43 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:26:43.880 DEBUG:boto:path=/pklxkd-53/ 2023-05-31T20:26:43.880 DEBUG:boto:auth_path=/pklxkd-53/ 2023-05-31T20:26:43.881 DEBUG:boto:path=/pklxkd-53/?prefix=d 2023-05-31T20:26:43.881 DEBUG:boto:auth_path=/pklxkd-53/?prefix=d 2023-05-31T20:26:43.882 DEBUG:boto:Method: GET 2023-05-31T20:26:43.882 DEBUG:boto:Path: /pklxkd-53/?prefix=d 2023-05-31T20:26:43.883 DEBUG:boto:Data: 2023-05-31T20:26:43.884 DEBUG:boto:Headers: {} 2023-05-31T20:26:43.885 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:26:43.885 DEBUG:boto:Port: 8001 2023-05-31T20:26:43.886 DEBUG:boto:Params: {} 2023-05-31T20:26:43.886 DEBUG:boto:Token: None 2023-05-31T20:26:43.887 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:26:43 GMT /pklxkd-53/ 2023-05-31T20:26:43.887 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:RtuWiOhPp2v0s+P6tP4ZjzNQme0= 2023-05-31T20:26:43.888 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:26:43 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:RtuWiOhPp2v0s+P6tP4ZjzNQme0=', 'Content-Length': '0'} 2023-05-31T20:26:43.890 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000005f50d26ee90d0973-006477ad83-162e-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:26:43 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:26:43.891 DEBUG:boto:b'pklxkd-53d1000falsed2023-05-31T20:26:27.602Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:26:43.892 DEBUG:boto:path=/pklxkd-53/ 2023-05-31T20:26:43.893 DEBUG:boto:auth_path=/pklxkd-53/ 2023-05-31T20:26:43.894 DEBUG:boto:path=/pklxkd-53/?prefix=d 2023-05-31T20:26:43.895 DEBUG:boto:auth_path=/pklxkd-53/?prefix=d 2023-05-31T20:26:43.896 DEBUG:boto:Method: GET 2023-05-31T20:26:43.896 DEBUG:boto:Path: /pklxkd-53/?prefix=d 2023-05-31T20:26:43.897 DEBUG:boto:Data: 2023-05-31T20:26:43.897 DEBUG:boto:Headers: {} 2023-05-31T20:26:43.897 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:26:43.898 DEBUG:boto:Port: 8000 2023-05-31T20:26:43.898 DEBUG:boto:Params: {} 2023-05-31T20:26:43.898 DEBUG:boto:Token: None 2023-05-31T20:26:43.899 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:26:43 GMT /pklxkd-53/ 2023-05-31T20:26:43.899 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:RtuWiOhPp2v0s+P6tP4ZjzNQme0= 2023-05-31T20:26:43.900 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:26:43 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:RtuWiOhPp2v0s+P6tP4ZjzNQme0=', 'Content-Length': '0'} 2023-05-31T20:26:43.902 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c8d6043092056603-006477ad83-1630-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:26:43 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:26:43.902 DEBUG:boto:b'pklxkd-53d1000falsed2023-05-31T20:26:27.602Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:26:43.903 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=pklxkd-53 zones={a2, a1} 2023-05-31T20:26:43.903 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'autotrim', '--rgw-sync-log-trim-max-buckets', '0', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-05-31T20:26:43.904 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'autotrim', '--rgw-sync-log-trim-max-buckets', '0', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-05-31T20:26:43.904 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog autotrim --rgw-sync-log-trim-max-buckets 0 --cluster c1 --debug-rgw 0 --debug-ms 0 2023-05-31T20:26:43.950 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2023-05-31T20:26:43.950 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2023-05-31T20:26:43.999 INFO:rgw_multi.tests:create bucket zone=a1 name=pklxkd-54 2023-05-31T20:26:44.000 DEBUG:boto:path=/pklxkd-54/ 2023-05-31T20:26:44.000 DEBUG:boto:auth_path=/pklxkd-54/ 2023-05-31T20:26:44.000 DEBUG:boto:Method: PUT 2023-05-31T20:26:44.001 DEBUG:boto:Path: /pklxkd-54/ 2023-05-31T20:26:44.001 DEBUG:boto:Data: 2023-05-31T20:26:44.001 DEBUG:boto:Headers: {} 2023-05-31T20:26:44.002 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:26:44.002 DEBUG:boto:Port: 8000 2023-05-31T20:26:44.004 DEBUG:boto:Params: {} 2023-05-31T20:26:44.004 DEBUG:boto:Token: None 2023-05-31T20:26:44.005 DEBUG:boto:StringToSign: PUT Wed, 31 May 2023 20:26:44 GMT /pklxkd-54/ 2023-05-31T20:26:44.006 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:fcthCSqcV+uCRnYnmBV9Cm7EIQI= 2023-05-31T20:26:44.006 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:26:44 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:fcthCSqcV+uCRnYnmBV9Cm7EIQI=', 'Content-Length': '0'} 2023-05-31T20:26:44.021 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx000003b321dafc1ae64e4-006477ad84-1630-a1'), ('Content-Length', '0'), ('Date', 'Wed, 31 May 2023 20:26:44 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:26:44.022 DEBUG:boto:path=/pklxkd-54/ 2023-05-31T20:26:44.022 DEBUG:boto:auth_path=/pklxkd-54/ 2023-05-31T20:26:44.023 DEBUG:boto:Method: HEAD 2023-05-31T20:26:44.023 DEBUG:boto:Path: /pklxkd-54/ 2023-05-31T20:26:44.024 DEBUG:boto:Data: 2023-05-31T20:26:44.024 DEBUG:boto:Headers: {} 2023-05-31T20:26:44.024 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:26:44.025 DEBUG:boto:Port: 8000 2023-05-31T20:26:44.025 DEBUG:boto:Params: {} 2023-05-31T20:26:44.025 DEBUG:boto:Token: None 2023-05-31T20:26:44.026 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:26:44 GMT /pklxkd-54/ 2023-05-31T20:26:44.026 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:4TiMa2FerkBYhPSSMfMdhm9EyUs= 2023-05-31T20:26:44.026 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:26:44 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:4TiMa2FerkBYhPSSMfMdhm9EyUs=', 'Content-Length': '0'} 2023-05-31T20:26:44.029 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000036282f7f37ecb434-006477ad84-1630-a1'), ('Content-Length', '0'), ('Date', 'Wed, 31 May 2023 20:26:44 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:26:44.030 DEBUG:boto:path=/pklxkd-54/a 2023-05-31T20:26:44.031 DEBUG:boto:auth_path=/pklxkd-54/a 2023-05-31T20:26:44.033 DEBUG:boto:Method: PUT 2023-05-31T20:26:44.033 DEBUG:boto:Path: /pklxkd-54/a 2023-05-31T20:26:44.034 DEBUG:boto:Data: 2023-05-31T20:26:44.034 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-05-31T20:26:44.034 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:26:44.035 DEBUG:boto:Port: 8000 2023-05-31T20:26:44.035 DEBUG:boto:Params: {} 2023-05-31T20:26:44.035 DEBUG:boto:Token: None 2023-05-31T20:26:44.036 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Wed, 31 May 2023 20:26:44 GMT /pklxkd-54/a 2023-05-31T20:26:44.036 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:hI8rv9N2yb/mozBGxs26C7GLZE0= 2023-05-31T20:26:44.037 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Wed, 31 May 2023 20:26:44 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:hI8rv9N2yb/mozBGxs26C7GLZE0='} 2023-05-31T20:26:44.044 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000ebd911d040e05a2d-006477ad84-1630-a1'), ('Date', 'Wed, 31 May 2023 20:26:44 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:26:44.045 DEBUG:boto:path=/pklxkd-54/ 2023-05-31T20:26:44.046 DEBUG:boto:auth_path=/pklxkd-54/ 2023-05-31T20:26:44.046 DEBUG:boto:Method: HEAD 2023-05-31T20:26:44.047 DEBUG:boto:Path: /pklxkd-54/ 2023-05-31T20:26:44.048 DEBUG:boto:Data: 2023-05-31T20:26:44.049 DEBUG:boto:Headers: {} 2023-05-31T20:26:44.049 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:26:44.051 DEBUG:boto:Port: 8000 2023-05-31T20:26:44.051 DEBUG:boto:Params: {} 2023-05-31T20:26:44.052 DEBUG:boto:Token: None 2023-05-31T20:26:44.053 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:26:44 GMT /pklxkd-54/ 2023-05-31T20:26:44.053 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:4TiMa2FerkBYhPSSMfMdhm9EyUs= 2023-05-31T20:26:44.054 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:26:44 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:4TiMa2FerkBYhPSSMfMdhm9EyUs=', 'Content-Length': '0'} 2023-05-31T20:26:44.057 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '3'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000ebbd8feb25749edf-006477ad84-1630-a1'), ('Content-Length', '0'), ('Date', 'Wed, 31 May 2023 20:26:44 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:26:44.058 DEBUG:boto:path=/pklxkd-54/b 2023-05-31T20:26:44.059 DEBUG:boto:auth_path=/pklxkd-54/b 2023-05-31T20:26:44.059 DEBUG:boto:Method: PUT 2023-05-31T20:26:44.060 DEBUG:boto:Path: /pklxkd-54/b 2023-05-31T20:26:44.061 DEBUG:boto:Data: 2023-05-31T20:26:44.062 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-05-31T20:26:44.062 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:26:44.063 DEBUG:boto:Port: 8000 2023-05-31T20:26:44.064 DEBUG:boto:Params: {} 2023-05-31T20:26:44.064 DEBUG:boto:Token: None 2023-05-31T20:26:44.065 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Wed, 31 May 2023 20:26:44 GMT /pklxkd-54/b 2023-05-31T20:26:44.066 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:ij5M1kzTHBmXvvMyFEHXGlN0NPw= 2023-05-31T20:26:44.066 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Wed, 31 May 2023 20:26:44 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:ij5M1kzTHBmXvvMyFEHXGlN0NPw='} 2023-05-31T20:26:44.108 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000002623610b20e35898-006477ad84-1630-a1'), ('Date', 'Wed, 31 May 2023 20:26:44 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:26:44.109 DEBUG:boto:path=/pklxkd-54/ 2023-05-31T20:26:44.109 DEBUG:boto:auth_path=/pklxkd-54/ 2023-05-31T20:26:44.110 DEBUG:boto:Method: HEAD 2023-05-31T20:26:44.110 DEBUG:boto:Path: /pklxkd-54/ 2023-05-31T20:26:44.111 DEBUG:boto:Data: 2023-05-31T20:26:44.111 DEBUG:boto:Headers: {} 2023-05-31T20:26:44.111 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:26:44.112 DEBUG:boto:Port: 8000 2023-05-31T20:26:44.112 DEBUG:boto:Params: {} 2023-05-31T20:26:44.112 DEBUG:boto:Token: None 2023-05-31T20:26:44.113 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:26:44 GMT /pklxkd-54/ 2023-05-31T20:26:44.113 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:4TiMa2FerkBYhPSSMfMdhm9EyUs= 2023-05-31T20:26:44.114 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:26:44 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:4TiMa2FerkBYhPSSMfMdhm9EyUs=', 'Content-Length': '0'} 2023-05-31T20:26:44.116 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '2'), ('X-RGW-Bytes-Used', '6'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000a15df5bab9ea2a1e-006477ad84-1630-a1'), ('Content-Length', '0'), ('Date', 'Wed, 31 May 2023 20:26:44 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:26:44.116 DEBUG:boto:path=/pklxkd-54/c 2023-05-31T20:26:44.117 DEBUG:boto:auth_path=/pklxkd-54/c 2023-05-31T20:26:44.117 DEBUG:boto:Method: PUT 2023-05-31T20:26:44.118 DEBUG:boto:Path: /pklxkd-54/c 2023-05-31T20:26:44.118 DEBUG:boto:Data: 2023-05-31T20:26:44.118 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-05-31T20:26:44.119 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:26:44.119 DEBUG:boto:Port: 8000 2023-05-31T20:26:44.119 DEBUG:boto:Params: {} 2023-05-31T20:26:44.120 DEBUG:boto:Token: None 2023-05-31T20:26:44.120 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Wed, 31 May 2023 20:26:44 GMT /pklxkd-54/c 2023-05-31T20:26:44.121 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:rbGbvLv9I3qRX/pPlI32rXs0xH8= 2023-05-31T20:26:44.121 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Wed, 31 May 2023 20:26:44 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:rbGbvLv9I3qRX/pPlI32rXs0xH8='} 2023-05-31T20:26:44.128 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000d3094299bdc57779-006477ad84-1630-a1'), ('Date', 'Wed, 31 May 2023 20:26:44 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:26:44.129 DEBUG:boto:path=/pklxkd-54/ 2023-05-31T20:26:44.129 DEBUG:boto:auth_path=/pklxkd-54/ 2023-05-31T20:26:44.130 DEBUG:boto:Method: HEAD 2023-05-31T20:26:44.130 DEBUG:boto:Path: /pklxkd-54/ 2023-05-31T20:26:44.130 DEBUG:boto:Data: 2023-05-31T20:26:44.131 DEBUG:boto:Headers: {} 2023-05-31T20:26:44.131 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:26:44.131 DEBUG:boto:Port: 8000 2023-05-31T20:26:44.132 DEBUG:boto:Params: {} 2023-05-31T20:26:44.132 DEBUG:boto:Token: None 2023-05-31T20:26:44.133 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:26:44 GMT /pklxkd-54/ 2023-05-31T20:26:44.133 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:4TiMa2FerkBYhPSSMfMdhm9EyUs= 2023-05-31T20:26:44.133 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:26:44 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:4TiMa2FerkBYhPSSMfMdhm9EyUs=', 'Content-Length': '0'} 2023-05-31T20:26:44.137 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '3'), ('X-RGW-Bytes-Used', '9'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000074cdb7c90884bb27-006477ad84-1630-a1'), ('Content-Length', '0'), ('Date', 'Wed, 31 May 2023 20:26:44 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:26:44.137 DEBUG:boto:path=/pklxkd-54/d 2023-05-31T20:26:44.138 DEBUG:boto:auth_path=/pklxkd-54/d 2023-05-31T20:26:44.138 DEBUG:boto:Method: PUT 2023-05-31T20:26:44.139 DEBUG:boto:Path: /pklxkd-54/d 2023-05-31T20:26:44.139 DEBUG:boto:Data: 2023-05-31T20:26:44.139 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-05-31T20:26:44.140 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:26:44.140 DEBUG:boto:Port: 8000 2023-05-31T20:26:44.140 DEBUG:boto:Params: {} 2023-05-31T20:26:44.141 DEBUG:boto:Token: None 2023-05-31T20:26:44.141 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Wed, 31 May 2023 20:26:44 GMT /pklxkd-54/d 2023-05-31T20:26:44.142 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:PNuZZG+vaZ6thcDTkBKGXLGnLJE= 2023-05-31T20:26:44.142 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Wed, 31 May 2023 20:26:44 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:PNuZZG+vaZ6thcDTkBKGXLGnLJE='} 2023-05-31T20:26:44.184 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000009e4e54c758090617-006477ad84-1630-a1'), ('Date', 'Wed, 31 May 2023 20:26:44 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:26:44.185 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-31T20:26:44.186 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-31T20:26:44.186 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-05-31T20:26:44.235 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2023-05-31T20:26:44.235 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2023-05-31T20:26:44.290 INFO:teuthology.orchestra.run.smithi099.stderr:No --period given, using current period=2e6123d2-d695-4350-9a01-61a455bc782b 2023-05-31T20:26:44.312 INFO:teuthology.orchestra.run.smithi099.stdout:[ 2023-05-31T20:26:44.312 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T20:26:44.312 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1685562631.516089_59.1", 2023-05-31T20:26:44.312 INFO:teuthology.orchestra.run.smithi099.stdout: "last_update": "2023-05-31T19:50:31.516089Z" 2023-05-31T20:26:44.313 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:26:44.313 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T20:26:44.313 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "", 2023-05-31T20:26:44.313 INFO:teuthology.orchestra.run.smithi099.stdout: "last_update": "0.000000" 2023-05-31T20:26:44.313 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:26:44.314 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T20:26:44.314 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1685562970.274572_93.1", 2023-05-31T20:26:44.314 INFO:teuthology.orchestra.run.smithi099.stdout: "last_update": "2023-05-31T19:56:10.274572Z" 2023-05-31T20:26:44.314 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:26:44.314 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T20:26:44.314 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1685564804.016433_354.1", 2023-05-31T20:26:44.314 INFO:teuthology.orchestra.run.smithi099.stdout: "last_update": "2023-05-31T20:26:44.016433Z" 2023-05-31T20:26:44.315 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T20:26:44.315 INFO:teuthology.orchestra.run.smithi099.stdout:] 2023-05-31T20:26:44.316 DEBUG:rgw_multi.tests:master meta markers={0: '1_1685562631.516089_59.1', 1: '', 2: '1_1685562970.274572_93.1', 3: '1_1685564804.016433_354.1'} 2023-05-31T20:26:44.316 INFO:rgw_multi.tests:starting meta checkpoint for zone=a2 2023-05-31T20:26:44.317 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-31T20:26:44.317 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-31T20:26:44.317 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-05-31T20:26:44.367 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2023-05-31T20:26:44.367 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2023-05-31T20:26:44.398 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:26:44.394+0000 7f5e09d23780 20 RGW-SYNC:meta: read sync status 2023-05-31T20:26:44.399 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:26:44.395+0000 7f5e09d23780 20 RGW-SYNC:meta: read sync status 2023-05-31T20:26:44.412 INFO:teuthology.orchestra.run.smithi099.stdout:{ 2023-05-31T20:26:44.412 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_status": { 2023-05-31T20:26:44.412 INFO:teuthology.orchestra.run.smithi099.stdout: "info": { 2023-05-31T20:26:44.413 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "sync", 2023-05-31T20:26:44.413 INFO:teuthology.orchestra.run.smithi099.stdout: "num_shards": 4, 2023-05-31T20:26:44.413 INFO:teuthology.orchestra.run.smithi099.stdout: "period": "2e6123d2-d695-4350-9a01-61a455bc782b", 2023-05-31T20:26:44.413 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2023-05-31T20:26:44.413 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:26:44.413 INFO:teuthology.orchestra.run.smithi099.stdout: "markers": [ 2023-05-31T20:26:44.414 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T20:26:44.414 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 0, 2023-05-31T20:26:44.414 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T20:26:44.414 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2023-05-31T20:26:44.414 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1685562631.516089_59.1", 2023-05-31T20:26:44.414 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T20:26:44.414 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 1, 2023-05-31T20:26:44.415 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T20:26:44.415 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2023-05-31T19:50:31.516089Z", 2023-05-31T20:26:44.415 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2023-05-31T20:26:44.415 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T20:26:44.416 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:26:44.416 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T20:26:44.416 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 1, 2023-05-31T20:26:44.416 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T20:26:44.417 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2023-05-31T20:26:44.417 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "", 2023-05-31T20:26:44.417 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T20:26:44.417 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2023-05-31T20:26:44.417 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T20:26:44.418 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "0.000000", 2023-05-31T20:26:44.418 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2023-05-31T20:26:44.418 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T20:26:44.418 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:26:44.418 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T20:26:44.419 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 2, 2023-05-31T20:26:44.419 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T20:26:44.419 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2023-05-31T20:26:44.419 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1685562970.274572_93.1", 2023-05-31T20:26:44.419 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T20:26:44.420 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2023-05-31T20:26:44.420 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T20:26:44.420 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2023-05-31T19:56:10.274572Z", 2023-05-31T20:26:44.420 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2023-05-31T20:26:44.420 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T20:26:44.421 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:26:44.421 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T20:26:44.421 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 3, 2023-05-31T20:26:44.421 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T20:26:44.421 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2023-05-31T20:26:44.422 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1685564804.016433_354.1", 2023-05-31T20:26:44.422 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T20:26:44.423 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2023-05-31T20:26:44.423 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T20:26:44.423 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2023-05-31T20:26:44.016433Z", 2023-05-31T20:26:44.424 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2023-05-31T20:26:44.424 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T20:26:44.424 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T20:26:44.424 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2023-05-31T20:26:44.424 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:26:44.425 INFO:teuthology.orchestra.run.smithi099.stdout: "full_sync": { 2023-05-31T20:26:44.425 INFO:teuthology.orchestra.run.smithi099.stdout: "total": 1, 2023-05-31T20:26:44.425 INFO:teuthology.orchestra.run.smithi099.stdout: "complete": 1 2023-05-31T20:26:44.426 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:26:44.426 INFO:teuthology.orchestra.run.smithi099.stdout: "current_time": "2023-05-31T20:26:44Z" 2023-05-31T20:26:44.426 INFO:teuthology.orchestra.run.smithi099.stdout:} 2023-05-31T20:26:44.428 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "2e6123d2-d695-4350-9a01-61a455bc782b", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1685562631.516089_59.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-05-31T19:50:31.516089Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1685562970.274572_93.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-05-31T19:56:10.274572Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1685564804.016433_354.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-05-31T20:26:44.016433Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-05-31T20:26:44Z" } 2023-05-31T20:26:44.429 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1685562631.516089_59.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-05-31T19:50:31.516089Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1685562970.274572_93.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-05-31T19:56:10.274572Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1685564804.016433_354.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-05-31T20:26:44.016433Z', 'realm_epoch': 2}}] 2023-05-31T20:26:44.429 DEBUG:rgw_multi.tests:log_status={0: '1_1685562631.516089_59.1', 1: '', 2: '1_1685562970.274572_93.1', 3: '1_1685564804.016433_354.1'} 2023-05-31T20:26:44.430 DEBUG:rgw_multi.tests:sync_status={0: '1_1685562631.516089_59.1', 1: '', 2: '1_1685562970.274572_93.1', 3: '1_1685564804.016433_354.1'} 2023-05-31T20:26:44.430 INFO:rgw_multi.tests:finish meta checkpoint for zone=a2 2023-05-31T20:26:44.431 DEBUG:rgw_multi.tests:bucket checkpoint: source=a1 target=a2 bucket=pklxkd-54 2023-05-31T20:26:44.432 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'pklxkd-54', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-05-31T20:26:44.432 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'pklxkd-54', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-05-31T20:26:44.433 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket pklxkd-54 --source-zone a1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2023-05-31T20:26:44.490 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2023-05-31T20:26:44.491 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2023-05-31T20:26:44.528 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:26:44.523+0000 7f8522da3780 1 waiting to reach incremental sync.. 2023-05-31T20:26:49.529 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:26:49.523+0000 7f8522da3780 1 waiting to reach incremental sync.. 2023-05-31T20:26:54.528 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:26:54.524+0000 7f8522da3780 1 waiting to reach incremental sync.. 2023-05-31T20:26:59.529 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:26:59.524+0000 7f8522da3780 1 waiting to reach incremental sync.. 2023-05-31T20:27:04.531 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:27:04.525+0000 7f8522da3780 1 bucket sync caught up with source: 2023-05-31T20:27:04.531 INFO:teuthology.orchestra.run.smithi099.stderr: local status: [, 00000000001.29.6, , , , 00000000001.26.6, , , 00000000001.31.6, , 00000000001.29.6] 2023-05-31T20:27:04.531 INFO:teuthology.orchestra.run.smithi099.stderr: remote markers: [, 00000000001.29.6, , , , 00000000001.26.6, , , 00000000001.31.6, , 00000000001.29.6] 2023-05-31T20:27:04.531 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:27:04.525+0000 7f8522da3780 0 bucket checkpoint complete 2023-05-31T20:27:04.549 DEBUG:rgw_multi.tests:bucket checkpoint: source=a2 target=a1 bucket=pklxkd-54 2023-05-31T20:27:04.549 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'pklxkd-54', '--source-zone', 'a2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-05-31T20:27:04.550 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'pklxkd-54', '--source-zone', 'a2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-05-31T20:27:04.550 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket pklxkd-54 --source-zone a2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2023-05-31T20:27:04.613 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2023-05-31T20:27:04.613 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2023-05-31T20:27:04.662 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:27:04.657+0000 7ffb0a18b780 1 waiting to reach incremental sync.. 2023-05-31T20:27:09.663 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:27:09.657+0000 7ffb0a18b780 1 waiting to reach incremental sync.. 2023-05-31T20:27:14.665 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:27:14.659+0000 7ffb0a18b780 1 bucket sync caught up with source: 2023-05-31T20:27:14.665 INFO:teuthology.orchestra.run.smithi099.stderr: local status: [, 00000000001.2248.6, , , , 00000000001.2258.6, , , 00000000001.745.6, , 00000000001.2590.6] 2023-05-31T20:27:14.666 INFO:teuthology.orchestra.run.smithi099.stderr: remote markers: [, 00000000001.2248.6, , , , 00000000001.2258.6, , , 00000000001.745.6, , 00000000001.2590.6] 2023-05-31T20:27:14.666 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:27:14.659+0000 7ffb0a18b780 0 bucket checkpoint complete 2023-05-31T20:27:14.683 INFO:tasks.rgw_multi.zone_rados:comparing bucket=pklxkd-54 zones={a2, a1} 2023-05-31T20:27:14.684 DEBUG:boto:path=/pklxkd-54/ 2023-05-31T20:27:14.684 DEBUG:boto:auth_path=/pklxkd-54/ 2023-05-31T20:27:14.684 DEBUG:boto:Method: HEAD 2023-05-31T20:27:14.685 DEBUG:boto:Path: /pklxkd-54/ 2023-05-31T20:27:14.685 DEBUG:boto:Data: 2023-05-31T20:27:14.686 DEBUG:boto:Headers: {} 2023-05-31T20:27:14.686 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:27:14.686 DEBUG:boto:Port: 8001 2023-05-31T20:27:14.687 DEBUG:boto:Params: {} 2023-05-31T20:27:14.688 DEBUG:boto:Token: None 2023-05-31T20:27:14.688 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:27:14 GMT /pklxkd-54/ 2023-05-31T20:27:14.689 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:UiVI0X5XJpVWsBOCbLpcjKE+pYs= 2023-05-31T20:27:14.689 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:27:14 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:UiVI0X5XJpVWsBOCbLpcjKE+pYs=', 'Content-Length': '0'} 2023-05-31T20:27:14.691 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '4'), ('X-RGW-Bytes-Used', '12'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000097257a4c5e6ce631-006477ada2-162e-a2'), ('Content-Length', '0'), ('Date', 'Wed, 31 May 2023 20:27:14 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:27:14.692 DEBUG:boto:path=/pklxkd-54/ 2023-05-31T20:27:14.692 DEBUG:boto:auth_path=/pklxkd-54/ 2023-05-31T20:27:14.693 DEBUG:boto:Method: HEAD 2023-05-31T20:27:14.693 DEBUG:boto:Path: /pklxkd-54/ 2023-05-31T20:27:14.693 DEBUG:boto:Data: 2023-05-31T20:27:14.694 DEBUG:boto:Headers: {} 2023-05-31T20:27:14.694 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:27:14.694 DEBUG:boto:Port: 8000 2023-05-31T20:27:14.695 DEBUG:boto:Params: {} 2023-05-31T20:27:14.695 DEBUG:boto:Token: None 2023-05-31T20:27:14.696 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:27:14 GMT /pklxkd-54/ 2023-05-31T20:27:14.696 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:UiVI0X5XJpVWsBOCbLpcjKE+pYs= 2023-05-31T20:27:14.697 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:27:14 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:UiVI0X5XJpVWsBOCbLpcjKE+pYs=', 'Content-Length': '0'} 2023-05-31T20:27:14.699 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '4'), ('X-RGW-Bytes-Used', '12'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000005e05264d501821f0-006477ada2-1630-a1'), ('Content-Length', '0'), ('Date', 'Wed, 31 May 2023 20:27:14 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:27:14.699 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2023-05-31T20:27:14.700 DEBUG:boto:path=/pklxkd-54/ 2023-05-31T20:27:14.700 DEBUG:boto:auth_path=/pklxkd-54/ 2023-05-31T20:27:14.701 DEBUG:boto:path=/pklxkd-54/?versions&max-keys=999 2023-05-31T20:27:14.701 DEBUG:boto:auth_path=/pklxkd-54/?versions&max-keys=999 2023-05-31T20:27:14.701 DEBUG:boto:Method: GET 2023-05-31T20:27:14.702 DEBUG:boto:Path: /pklxkd-54/?versions&max-keys=999 2023-05-31T20:27:14.702 DEBUG:boto:Data: 2023-05-31T20:27:14.702 DEBUG:boto:Headers: {} 2023-05-31T20:27:14.703 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:27:14.703 DEBUG:boto:Port: 8001 2023-05-31T20:27:14.704 DEBUG:boto:Params: {} 2023-05-31T20:27:14.704 DEBUG:boto:Token: None 2023-05-31T20:27:14.704 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:27:14 GMT /pklxkd-54/?versions 2023-05-31T20:27:14.705 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:cYdyCvyzbIbOz/26yZl748X6Mbk= 2023-05-31T20:27:14.705 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:27:14 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:cYdyCvyzbIbOz/26yZl748X6Mbk=', 'Content-Length': '0'} 2023-05-31T20:27:14.708 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000002a7275c1d758feea-006477ada2-162e-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:27:14 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:27:14.708 DEBUG:boto:b'pklxkd-54999falseanulltrue2023-05-31T20:26:44.036Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalbnulltrue2023-05-31T20:26:44.064Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalcnulltrue2023-05-31T20:26:44.119Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormaldnulltrue2023-05-31T20:26:44.140Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:27:14.709 DEBUG:tasks.rgw_multi.zone_rados:o=a 2023-05-31T20:27:14.710 DEBUG:tasks.rgw_multi.zone_rados:o=b 2023-05-31T20:27:14.710 DEBUG:tasks.rgw_multi.zone_rados:o=c 2023-05-31T20:27:14.710 DEBUG:tasks.rgw_multi.zone_rados:o=d 2023-05-31T20:27:14.711 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2023-05-31T20:27:14.711 DEBUG:boto:path=/pklxkd-54/ 2023-05-31T20:27:14.711 DEBUG:boto:auth_path=/pklxkd-54/ 2023-05-31T20:27:14.712 DEBUG:boto:path=/pklxkd-54/?versions&max-keys=999 2023-05-31T20:27:14.712 DEBUG:boto:auth_path=/pklxkd-54/?versions&max-keys=999 2023-05-31T20:27:14.713 DEBUG:boto:Method: GET 2023-05-31T20:27:14.713 DEBUG:boto:Path: /pklxkd-54/?versions&max-keys=999 2023-05-31T20:27:14.713 DEBUG:boto:Data: 2023-05-31T20:27:14.714 DEBUG:boto:Headers: {} 2023-05-31T20:27:14.714 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:27:14.714 DEBUG:boto:Port: 8000 2023-05-31T20:27:14.715 DEBUG:boto:Params: {} 2023-05-31T20:27:14.715 DEBUG:boto:Token: None 2023-05-31T20:27:14.716 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:27:14 GMT /pklxkd-54/?versions 2023-05-31T20:27:14.716 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:cYdyCvyzbIbOz/26yZl748X6Mbk= 2023-05-31T20:27:14.717 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:27:14 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:cYdyCvyzbIbOz/26yZl748X6Mbk=', 'Content-Length': '0'} 2023-05-31T20:27:14.719 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000007b23a48cd5837e4f-006477ada2-1630-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:27:14 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:27:14.719 DEBUG:boto:b'pklxkd-54999falseanulltrue2023-05-31T20:26:44.036Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalbnulltrue2023-05-31T20:26:44.064Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalcnulltrue2023-05-31T20:26:44.119Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormaldnulltrue2023-05-31T20:26:44.140Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:27:14.720 DEBUG:tasks.rgw_multi.zone_rados:o=a 2023-05-31T20:27:14.721 DEBUG:tasks.rgw_multi.zone_rados:o=b 2023-05-31T20:27:14.721 DEBUG:tasks.rgw_multi.zone_rados:o=c 2023-05-31T20:27:14.721 DEBUG:tasks.rgw_multi.zone_rados:o=d 2023-05-31T20:27:14.722 DEBUG:boto:path=/pklxkd-54/ 2023-05-31T20:27:14.722 DEBUG:boto:auth_path=/pklxkd-54/ 2023-05-31T20:27:14.723 DEBUG:boto:path=/pklxkd-54/?versions&max-keys=999 2023-05-31T20:27:14.723 DEBUG:boto:auth_path=/pklxkd-54/?versions&max-keys=999 2023-05-31T20:27:14.723 DEBUG:boto:Method: GET 2023-05-31T20:27:14.724 DEBUG:boto:Path: /pklxkd-54/?versions&max-keys=999 2023-05-31T20:27:14.724 DEBUG:boto:Data: 2023-05-31T20:27:14.724 DEBUG:boto:Headers: {} 2023-05-31T20:27:14.725 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:27:14.725 DEBUG:boto:Port: 8001 2023-05-31T20:27:14.725 DEBUG:boto:Params: {} 2023-05-31T20:27:14.726 DEBUG:boto:Token: None 2023-05-31T20:27:14.726 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:27:14 GMT /pklxkd-54/?versions 2023-05-31T20:27:14.727 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:cYdyCvyzbIbOz/26yZl748X6Mbk= 2023-05-31T20:27:14.727 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:27:14 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:cYdyCvyzbIbOz/26yZl748X6Mbk=', 'Content-Length': '0'} 2023-05-31T20:27:14.729 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000007786a04570efa76f-006477ada2-162e-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:27:14 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:27:14.730 DEBUG:boto:b'pklxkd-54999falseanulltrue2023-05-31T20:26:44.036Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalbnulltrue2023-05-31T20:26:44.064Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalcnulltrue2023-05-31T20:26:44.119Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormaldnulltrue2023-05-31T20:26:44.140Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:27:14.731 DEBUG:boto:path=/pklxkd-54/ 2023-05-31T20:27:14.731 DEBUG:boto:auth_path=/pklxkd-54/ 2023-05-31T20:27:14.731 DEBUG:boto:path=/pklxkd-54/?versions&max-keys=999 2023-05-31T20:27:14.732 DEBUG:boto:auth_path=/pklxkd-54/?versions&max-keys=999 2023-05-31T20:27:14.732 DEBUG:boto:Method: GET 2023-05-31T20:27:14.733 DEBUG:boto:Path: /pklxkd-54/?versions&max-keys=999 2023-05-31T20:27:14.733 DEBUG:boto:Data: 2023-05-31T20:27:14.733 DEBUG:boto:Headers: {} 2023-05-31T20:27:14.734 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:27:14.734 DEBUG:boto:Port: 8000 2023-05-31T20:27:14.734 DEBUG:boto:Params: {} 2023-05-31T20:27:14.735 DEBUG:boto:Token: None 2023-05-31T20:27:14.735 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:27:14 GMT /pklxkd-54/?versions 2023-05-31T20:27:14.736 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:cYdyCvyzbIbOz/26yZl748X6Mbk= 2023-05-31T20:27:14.736 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:27:14 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:cYdyCvyzbIbOz/26yZl748X6Mbk=', 'Content-Length': '0'} 2023-05-31T20:27:14.739 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f6552df14b25ab1e-006477ada2-1630-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:27:14 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:27:14.739 DEBUG:boto:b'pklxkd-54999falseanulltrue2023-05-31T20:26:44.036Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalbnulltrue2023-05-31T20:26:44.064Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalcnulltrue2023-05-31T20:26:44.119Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormaldnulltrue2023-05-31T20:26:44.140Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:27:14.740 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=a 2023-05-31T20:27:14.741 DEBUG:boto:path=/pklxkd-54/a 2023-05-31T20:27:14.741 DEBUG:boto:auth_path=/pklxkd-54/a 2023-05-31T20:27:14.741 DEBUG:boto:path=/pklxkd-54/a?versionId=null 2023-05-31T20:27:14.742 DEBUG:boto:auth_path=/pklxkd-54/a?versionId=null 2023-05-31T20:27:14.742 DEBUG:boto:Method: GET 2023-05-31T20:27:14.742 DEBUG:boto:Path: /pklxkd-54/a?versionId=null 2023-05-31T20:27:14.743 DEBUG:boto:Data: 2023-05-31T20:27:14.743 DEBUG:boto:Headers: {} 2023-05-31T20:27:14.744 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:27:14.744 DEBUG:boto:Port: 8001 2023-05-31T20:27:14.744 DEBUG:boto:Params: {} 2023-05-31T20:27:14.745 DEBUG:boto:Token: None 2023-05-31T20:27:14.745 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:27:14 GMT /pklxkd-54/a?versionId=null 2023-05-31T20:27:14.746 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:xrS2aJsTYfhDrMaJL4pqMC+l9Rk= 2023-05-31T20:27:14.746 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:27:14 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:xrS2aJsTYfhDrMaJL4pqMC+l9Rk=', 'Content-Length': '0'} 2023-05-31T20:27:14.748 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:26:44 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e99769fe8105a0ce-006477ada2-162e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:27:14 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:27:14.749 DEBUG:boto:path=/pklxkd-54/a 2023-05-31T20:27:14.750 DEBUG:boto:auth_path=/pklxkd-54/a 2023-05-31T20:27:14.750 DEBUG:boto:path=/pklxkd-54/a?versionId=null 2023-05-31T20:27:14.751 DEBUG:boto:auth_path=/pklxkd-54/a?versionId=null 2023-05-31T20:27:14.751 DEBUG:boto:Method: GET 2023-05-31T20:27:14.751 DEBUG:boto:Path: /pklxkd-54/a?versionId=null 2023-05-31T20:27:14.752 DEBUG:boto:Data: 2023-05-31T20:27:14.752 DEBUG:boto:Headers: {} 2023-05-31T20:27:14.753 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:27:14.753 DEBUG:boto:Port: 8000 2023-05-31T20:27:14.753 DEBUG:boto:Params: {} 2023-05-31T20:27:14.754 DEBUG:boto:Token: None 2023-05-31T20:27:14.754 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:27:14 GMT /pklxkd-54/a?versionId=null 2023-05-31T20:27:14.755 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:xrS2aJsTYfhDrMaJL4pqMC+l9Rk= 2023-05-31T20:27:14.755 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:27:14 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:xrS2aJsTYfhDrMaJL4pqMC+l9Rk=', 'Content-Length': '0'} 2023-05-31T20:27:14.757 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:26:44 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008cdc78040927192f-006477ada2-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:27:14 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:27:14.758 DEBUG:boto:path=/pklxkd-54/a 2023-05-31T20:27:14.758 DEBUG:boto:auth_path=/pklxkd-54/a 2023-05-31T20:27:14.759 DEBUG:boto:path=/pklxkd-54/a?versionId=null 2023-05-31T20:27:14.759 DEBUG:boto:auth_path=/pklxkd-54/a?versionId=null 2023-05-31T20:27:14.759 DEBUG:boto:Method: HEAD 2023-05-31T20:27:14.760 DEBUG:boto:Path: /pklxkd-54/a?versionId=null 2023-05-31T20:27:14.760 DEBUG:boto:Data: 2023-05-31T20:27:14.760 DEBUG:boto:Headers: {} 2023-05-31T20:27:14.761 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:27:14.761 DEBUG:boto:Port: 8001 2023-05-31T20:27:14.762 DEBUG:boto:Params: {} 2023-05-31T20:27:14.762 DEBUG:boto:Token: None 2023-05-31T20:27:14.762 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:27:14 GMT /pklxkd-54/a?versionId=null 2023-05-31T20:27:14.763 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:QtR5pOFJiuaVmvnBx7w1xRylIiE= 2023-05-31T20:27:14.763 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:27:14 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:QtR5pOFJiuaVmvnBx7w1xRylIiE=', 'Content-Length': '0'} 2023-05-31T20:27:14.765 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:26:44 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000062b485286853a30c-006477ada2-162e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:27:14 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:27:14.766 DEBUG:boto:path=/pklxkd-54/a 2023-05-31T20:27:14.766 DEBUG:boto:auth_path=/pklxkd-54/a 2023-05-31T20:27:14.766 DEBUG:boto:path=/pklxkd-54/a?versionId=null 2023-05-31T20:27:14.767 DEBUG:boto:auth_path=/pklxkd-54/a?versionId=null 2023-05-31T20:27:14.767 DEBUG:boto:Method: HEAD 2023-05-31T20:27:14.767 DEBUG:boto:Path: /pklxkd-54/a?versionId=null 2023-05-31T20:27:14.768 DEBUG:boto:Data: 2023-05-31T20:27:14.768 DEBUG:boto:Headers: {} 2023-05-31T20:27:14.769 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:27:14.769 DEBUG:boto:Port: 8000 2023-05-31T20:27:14.769 DEBUG:boto:Params: {} 2023-05-31T20:27:14.770 DEBUG:boto:Token: None 2023-05-31T20:27:14.770 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:27:14 GMT /pklxkd-54/a?versionId=null 2023-05-31T20:27:14.771 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:QtR5pOFJiuaVmvnBx7w1xRylIiE= 2023-05-31T20:27:14.771 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:27:14 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:QtR5pOFJiuaVmvnBx7w1xRylIiE=', 'Content-Length': '0'} 2023-05-31T20:27:14.773 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:26:44 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008771d42266774ff2-006477ada2-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:27:14 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:27:14.773 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=a 2023-05-31T20:27:14.774 DEBUG:boto:path=/pklxkd-54/a 2023-05-31T20:27:14.774 DEBUG:boto:auth_path=/pklxkd-54/a 2023-05-31T20:27:14.775 DEBUG:boto:path=/pklxkd-54/a?versionId=null 2023-05-31T20:27:14.775 DEBUG:boto:auth_path=/pklxkd-54/a?versionId=null 2023-05-31T20:27:14.775 DEBUG:boto:Method: GET 2023-05-31T20:27:14.776 DEBUG:boto:Path: /pklxkd-54/a?versionId=null 2023-05-31T20:27:14.776 DEBUG:boto:Data: 2023-05-31T20:27:14.777 DEBUG:boto:Headers: {} 2023-05-31T20:27:14.777 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:27:14.777 DEBUG:boto:Port: 8001 2023-05-31T20:27:14.778 DEBUG:boto:Params: {} 2023-05-31T20:27:14.778 DEBUG:boto:Token: None 2023-05-31T20:27:14.779 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:27:14 GMT /pklxkd-54/a?versionId=null 2023-05-31T20:27:14.779 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:xrS2aJsTYfhDrMaJL4pqMC+l9Rk= 2023-05-31T20:27:14.780 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:27:14 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:xrS2aJsTYfhDrMaJL4pqMC+l9Rk=', 'Content-Length': '0'} 2023-05-31T20:27:14.782 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:26:44 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e894a44acb0d5dbc-006477ada2-162e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:27:14 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:27:14.828 DEBUG:boto:path=/pklxkd-54/a 2023-05-31T20:27:14.829 DEBUG:boto:auth_path=/pklxkd-54/a 2023-05-31T20:27:14.829 DEBUG:boto:path=/pklxkd-54/a?versionId=null 2023-05-31T20:27:14.830 DEBUG:boto:auth_path=/pklxkd-54/a?versionId=null 2023-05-31T20:27:14.830 DEBUG:boto:Method: GET 2023-05-31T20:27:14.830 DEBUG:boto:Path: /pklxkd-54/a?versionId=null 2023-05-31T20:27:14.831 DEBUG:boto:Data: 2023-05-31T20:27:14.831 DEBUG:boto:Headers: {} 2023-05-31T20:27:14.832 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:27:14.832 DEBUG:boto:Port: 8000 2023-05-31T20:27:14.832 DEBUG:boto:Params: {} 2023-05-31T20:27:14.833 DEBUG:boto:Token: None 2023-05-31T20:27:14.833 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:27:14 GMT /pklxkd-54/a?versionId=null 2023-05-31T20:27:14.834 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:xrS2aJsTYfhDrMaJL4pqMC+l9Rk= 2023-05-31T20:27:14.834 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:27:14 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:xrS2aJsTYfhDrMaJL4pqMC+l9Rk=', 'Content-Length': '0'} 2023-05-31T20:27:14.837 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:26:44 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000060b9b407fccf8c1f-006477ada2-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:27:14 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:27:14.838 DEBUG:boto:path=/pklxkd-54/ 2023-05-31T20:27:14.838 DEBUG:boto:auth_path=/pklxkd-54/ 2023-05-31T20:27:14.839 DEBUG:boto:path=/pklxkd-54/?prefix=a 2023-05-31T20:27:14.839 DEBUG:boto:auth_path=/pklxkd-54/?prefix=a 2023-05-31T20:27:14.839 DEBUG:boto:Method: GET 2023-05-31T20:27:14.840 DEBUG:boto:Path: /pklxkd-54/?prefix=a 2023-05-31T20:27:14.840 DEBUG:boto:Data: 2023-05-31T20:27:14.841 DEBUG:boto:Headers: {} 2023-05-31T20:27:14.841 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:27:14.841 DEBUG:boto:Port: 8001 2023-05-31T20:27:14.842 DEBUG:boto:Params: {} 2023-05-31T20:27:14.842 DEBUG:boto:Token: None 2023-05-31T20:27:14.843 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:27:14 GMT /pklxkd-54/ 2023-05-31T20:27:14.843 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:il4Mcx+0eVjzcCshsak9uHYr52o= 2023-05-31T20:27:14.843 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:27:14 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:il4Mcx+0eVjzcCshsak9uHYr52o=', 'Content-Length': '0'} 2023-05-31T20:27:14.846 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000bd9c9209ee002492-006477ada2-162e-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:27:14 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:27:14.847 DEBUG:boto:b'pklxkd-54a1000falsea2023-05-31T20:26:44.036Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:27:14.849 DEBUG:boto:path=/pklxkd-54/ 2023-05-31T20:27:14.850 DEBUG:boto:auth_path=/pklxkd-54/ 2023-05-31T20:27:14.851 DEBUG:boto:path=/pklxkd-54/?prefix=a 2023-05-31T20:27:14.852 DEBUG:boto:auth_path=/pklxkd-54/?prefix=a 2023-05-31T20:27:14.852 DEBUG:boto:Method: GET 2023-05-31T20:27:14.853 DEBUG:boto:Path: /pklxkd-54/?prefix=a 2023-05-31T20:27:14.853 DEBUG:boto:Data: 2023-05-31T20:27:14.853 DEBUG:boto:Headers: {} 2023-05-31T20:27:14.854 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:27:14.854 DEBUG:boto:Port: 8000 2023-05-31T20:27:14.854 DEBUG:boto:Params: {} 2023-05-31T20:27:14.855 DEBUG:boto:Token: None 2023-05-31T20:27:14.855 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:27:14 GMT /pklxkd-54/ 2023-05-31T20:27:14.856 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:il4Mcx+0eVjzcCshsak9uHYr52o= 2023-05-31T20:27:14.856 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:27:14 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:il4Mcx+0eVjzcCshsak9uHYr52o=', 'Content-Length': '0'} 2023-05-31T20:27:14.859 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000004c0dadf5fbfa7fb7-006477ada2-1630-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:27:14 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:27:14.859 DEBUG:boto:b'pklxkd-54a1000falsea2023-05-31T20:26:44.036Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:27:14.860 DEBUG:boto:path=/pklxkd-54/a 2023-05-31T20:27:14.860 DEBUG:boto:auth_path=/pklxkd-54/a 2023-05-31T20:27:14.861 DEBUG:boto:Method: HEAD 2023-05-31T20:27:14.861 DEBUG:boto:Path: /pklxkd-54/a 2023-05-31T20:27:14.861 DEBUG:boto:Data: 2023-05-31T20:27:14.862 DEBUG:boto:Headers: {} 2023-05-31T20:27:14.862 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:27:14.862 DEBUG:boto:Port: 8001 2023-05-31T20:27:14.863 DEBUG:boto:Params: {} 2023-05-31T20:27:14.863 DEBUG:boto:Token: None 2023-05-31T20:27:14.864 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:27:14 GMT /pklxkd-54/a 2023-05-31T20:27:14.864 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:lulQhXRYeW/hNv8BHL4zxAumBMw= 2023-05-31T20:27:14.864 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:27:14 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:lulQhXRYeW/hNv8BHL4zxAumBMw=', 'Content-Length': '0'} 2023-05-31T20:27:14.866 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:26:44 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005082552f77454768-006477ada2-162e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:27:14 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:27:14.867 DEBUG:boto:path=/pklxkd-54/a 2023-05-31T20:27:14.867 DEBUG:boto:auth_path=/pklxkd-54/a 2023-05-31T20:27:14.868 DEBUG:boto:Method: HEAD 2023-05-31T20:27:14.868 DEBUG:boto:Path: /pklxkd-54/a 2023-05-31T20:27:14.868 DEBUG:boto:Data: 2023-05-31T20:27:14.869 DEBUG:boto:Headers: {} 2023-05-31T20:27:14.869 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:27:14.869 DEBUG:boto:Port: 8000 2023-05-31T20:27:14.870 DEBUG:boto:Params: {} 2023-05-31T20:27:14.870 DEBUG:boto:Token: None 2023-05-31T20:27:14.870 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:27:14 GMT /pklxkd-54/a 2023-05-31T20:27:14.871 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:lulQhXRYeW/hNv8BHL4zxAumBMw= 2023-05-31T20:27:14.871 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:27:14 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:lulQhXRYeW/hNv8BHL4zxAumBMw=', 'Content-Length': '0'} 2023-05-31T20:27:14.873 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:26:44 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000093f5c7c70a48f7d5-006477ada2-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:27:14 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:27:14.874 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=a 2023-05-31T20:27:14.874 DEBUG:boto:path=/pklxkd-54/a 2023-05-31T20:27:14.874 DEBUG:boto:auth_path=/pklxkd-54/a 2023-05-31T20:27:14.875 DEBUG:boto:Method: GET 2023-05-31T20:27:14.875 DEBUG:boto:Path: /pklxkd-54/a 2023-05-31T20:27:14.876 DEBUG:boto:Data: 2023-05-31T20:27:14.876 DEBUG:boto:Headers: {} 2023-05-31T20:27:14.876 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:27:14.877 DEBUG:boto:Port: 8001 2023-05-31T20:27:14.877 DEBUG:boto:Params: {} 2023-05-31T20:27:14.878 DEBUG:boto:Token: None 2023-05-31T20:27:14.878 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:27:14 GMT /pklxkd-54/a 2023-05-31T20:27:14.878 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:GpgM2TMNPsx6wd3DipT42/YTcIw= 2023-05-31T20:27:14.879 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:27:14 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:GpgM2TMNPsx6wd3DipT42/YTcIw=', 'Content-Length': '0'} 2023-05-31T20:27:14.881 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:26:44 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000096420a2d3a4e9f52-006477ada2-162e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:27:14 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:27:14.881 DEBUG:boto:path=/pklxkd-54/a 2023-05-31T20:27:14.882 DEBUG:boto:auth_path=/pklxkd-54/a 2023-05-31T20:27:14.882 DEBUG:boto:Method: GET 2023-05-31T20:27:14.883 DEBUG:boto:Path: /pklxkd-54/a 2023-05-31T20:27:14.883 DEBUG:boto:Data: 2023-05-31T20:27:14.883 DEBUG:boto:Headers: {} 2023-05-31T20:27:14.884 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:27:14.884 DEBUG:boto:Port: 8000 2023-05-31T20:27:14.884 DEBUG:boto:Params: {} 2023-05-31T20:27:14.885 DEBUG:boto:Token: None 2023-05-31T20:27:14.885 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:27:14 GMT /pklxkd-54/a 2023-05-31T20:27:14.886 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:GpgM2TMNPsx6wd3DipT42/YTcIw= 2023-05-31T20:27:14.886 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:27:14 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:GpgM2TMNPsx6wd3DipT42/YTcIw=', 'Content-Length': '0'} 2023-05-31T20:27:14.888 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:26:44 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009cecf4981dab2a17-006477ada2-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:27:14 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:27:14.889 DEBUG:boto:path=/pklxkd-54/ 2023-05-31T20:27:14.889 DEBUG:boto:auth_path=/pklxkd-54/ 2023-05-31T20:27:14.889 DEBUG:boto:path=/pklxkd-54/?prefix=a 2023-05-31T20:27:14.890 DEBUG:boto:auth_path=/pklxkd-54/?prefix=a 2023-05-31T20:27:14.890 DEBUG:boto:Method: GET 2023-05-31T20:27:14.891 DEBUG:boto:Path: /pklxkd-54/?prefix=a 2023-05-31T20:27:14.891 DEBUG:boto:Data: 2023-05-31T20:27:14.891 DEBUG:boto:Headers: {} 2023-05-31T20:27:14.892 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:27:14.892 DEBUG:boto:Port: 8001 2023-05-31T20:27:14.892 DEBUG:boto:Params: {} 2023-05-31T20:27:14.893 DEBUG:boto:Token: None 2023-05-31T20:27:14.893 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:27:14 GMT /pklxkd-54/ 2023-05-31T20:27:14.894 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:il4Mcx+0eVjzcCshsak9uHYr52o= 2023-05-31T20:27:14.894 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:27:14 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:il4Mcx+0eVjzcCshsak9uHYr52o=', 'Content-Length': '0'} 2023-05-31T20:27:14.896 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000fd2a4f95d4f8992d-006477ada2-162e-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:27:14 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:27:14.897 DEBUG:boto:b'pklxkd-54a1000falsea2023-05-31T20:26:44.036Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:27:14.898 DEBUG:boto:path=/pklxkd-54/ 2023-05-31T20:27:14.898 DEBUG:boto:auth_path=/pklxkd-54/ 2023-05-31T20:27:14.898 DEBUG:boto:path=/pklxkd-54/?prefix=a 2023-05-31T20:27:14.899 DEBUG:boto:auth_path=/pklxkd-54/?prefix=a 2023-05-31T20:27:14.899 DEBUG:boto:Method: GET 2023-05-31T20:27:14.900 DEBUG:boto:Path: /pklxkd-54/?prefix=a 2023-05-31T20:27:14.900 DEBUG:boto:Data: 2023-05-31T20:27:14.900 DEBUG:boto:Headers: {} 2023-05-31T20:27:14.901 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:27:14.901 DEBUG:boto:Port: 8000 2023-05-31T20:27:14.901 DEBUG:boto:Params: {} 2023-05-31T20:27:14.902 DEBUG:boto:Token: None 2023-05-31T20:27:14.902 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:27:14 GMT /pklxkd-54/ 2023-05-31T20:27:14.903 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:il4Mcx+0eVjzcCshsak9uHYr52o= 2023-05-31T20:27:14.903 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:27:14 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:il4Mcx+0eVjzcCshsak9uHYr52o=', 'Content-Length': '0'} 2023-05-31T20:27:14.906 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a40bfacc0ccf986d-006477ada2-1630-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:27:14 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:27:14.948 DEBUG:boto:b'pklxkd-54a1000falsea2023-05-31T20:26:44.036Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:27:14.949 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=b 2023-05-31T20:27:14.949 DEBUG:boto:path=/pklxkd-54/b 2023-05-31T20:27:14.950 DEBUG:boto:auth_path=/pklxkd-54/b 2023-05-31T20:27:14.950 DEBUG:boto:path=/pklxkd-54/b?versionId=null 2023-05-31T20:27:14.951 DEBUG:boto:auth_path=/pklxkd-54/b?versionId=null 2023-05-31T20:27:14.951 DEBUG:boto:Method: GET 2023-05-31T20:27:14.951 DEBUG:boto:Path: /pklxkd-54/b?versionId=null 2023-05-31T20:27:14.952 DEBUG:boto:Data: 2023-05-31T20:27:14.952 DEBUG:boto:Headers: {} 2023-05-31T20:27:14.953 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:27:14.953 DEBUG:boto:Port: 8001 2023-05-31T20:27:14.953 DEBUG:boto:Params: {} 2023-05-31T20:27:14.954 DEBUG:boto:Token: None 2023-05-31T20:27:14.954 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:27:14 GMT /pklxkd-54/b?versionId=null 2023-05-31T20:27:14.955 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:bVsesTfg0UVKS/hHZS1cUf/YLd4= 2023-05-31T20:27:14.955 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:27:14 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:bVsesTfg0UVKS/hHZS1cUf/YLd4=', 'Content-Length': '0'} 2023-05-31T20:27:14.957 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:26:44 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000074fca97f9bc6fd98-006477ada2-162e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:27:14 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:27:14.958 DEBUG:boto:path=/pklxkd-54/b 2023-05-31T20:27:14.958 DEBUG:boto:auth_path=/pklxkd-54/b 2023-05-31T20:27:14.958 DEBUG:boto:path=/pklxkd-54/b?versionId=null 2023-05-31T20:27:14.959 DEBUG:boto:auth_path=/pklxkd-54/b?versionId=null 2023-05-31T20:27:14.959 DEBUG:boto:Method: GET 2023-05-31T20:27:14.960 DEBUG:boto:Path: /pklxkd-54/b?versionId=null 2023-05-31T20:27:14.960 DEBUG:boto:Data: 2023-05-31T20:27:14.960 DEBUG:boto:Headers: {} 2023-05-31T20:27:14.961 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:27:14.961 DEBUG:boto:Port: 8000 2023-05-31T20:27:14.961 DEBUG:boto:Params: {} 2023-05-31T20:27:14.962 DEBUG:boto:Token: None 2023-05-31T20:27:14.962 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:27:14 GMT /pklxkd-54/b?versionId=null 2023-05-31T20:27:14.963 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:bVsesTfg0UVKS/hHZS1cUf/YLd4= 2023-05-31T20:27:14.963 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:27:14 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:bVsesTfg0UVKS/hHZS1cUf/YLd4=', 'Content-Length': '0'} 2023-05-31T20:27:14.965 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:26:44 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c2c0ff718e844390-006477ada2-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:27:14 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:27:14.966 DEBUG:boto:path=/pklxkd-54/b 2023-05-31T20:27:14.966 DEBUG:boto:auth_path=/pklxkd-54/b 2023-05-31T20:27:14.966 DEBUG:boto:path=/pklxkd-54/b?versionId=null 2023-05-31T20:27:14.967 DEBUG:boto:auth_path=/pklxkd-54/b?versionId=null 2023-05-31T20:27:14.967 DEBUG:boto:Method: HEAD 2023-05-31T20:27:14.967 DEBUG:boto:Path: /pklxkd-54/b?versionId=null 2023-05-31T20:27:14.968 DEBUG:boto:Data: 2023-05-31T20:27:14.968 DEBUG:boto:Headers: {} 2023-05-31T20:27:14.969 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:27:14.969 DEBUG:boto:Port: 8001 2023-05-31T20:27:14.970 DEBUG:boto:Params: {} 2023-05-31T20:27:14.971 DEBUG:boto:Token: None 2023-05-31T20:27:14.971 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:27:14 GMT /pklxkd-54/b?versionId=null 2023-05-31T20:27:14.972 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:/etP4XVOgzl/HRbdEb4e8qa0KLA= 2023-05-31T20:27:14.973 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:27:14 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:/etP4XVOgzl/HRbdEb4e8qa0KLA=', 'Content-Length': '0'} 2023-05-31T20:27:14.975 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:26:44 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000d5e6688abc1cc51-006477ada2-162e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:27:14 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:27:14.975 DEBUG:boto:path=/pklxkd-54/b 2023-05-31T20:27:14.976 DEBUG:boto:auth_path=/pklxkd-54/b 2023-05-31T20:27:14.976 DEBUG:boto:path=/pklxkd-54/b?versionId=null 2023-05-31T20:27:14.976 DEBUG:boto:auth_path=/pklxkd-54/b?versionId=null 2023-05-31T20:27:14.977 DEBUG:boto:Method: HEAD 2023-05-31T20:27:14.977 DEBUG:boto:Path: /pklxkd-54/b?versionId=null 2023-05-31T20:27:14.978 DEBUG:boto:Data: 2023-05-31T20:27:14.978 DEBUG:boto:Headers: {} 2023-05-31T20:27:14.978 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:27:14.979 DEBUG:boto:Port: 8000 2023-05-31T20:27:14.979 DEBUG:boto:Params: {} 2023-05-31T20:27:14.980 DEBUG:boto:Token: None 2023-05-31T20:27:14.980 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:27:14 GMT /pklxkd-54/b?versionId=null 2023-05-31T20:27:14.981 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:/etP4XVOgzl/HRbdEb4e8qa0KLA= 2023-05-31T20:27:14.981 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:27:14 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:/etP4XVOgzl/HRbdEb4e8qa0KLA=', 'Content-Length': '0'} 2023-05-31T20:27:14.983 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:26:44 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c554aafd5dfc3f07-006477ada2-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:27:14 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:27:14.983 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=b 2023-05-31T20:27:14.984 DEBUG:boto:path=/pklxkd-54/b 2023-05-31T20:27:14.984 DEBUG:boto:auth_path=/pklxkd-54/b 2023-05-31T20:27:14.985 DEBUG:boto:path=/pklxkd-54/b?versionId=null 2023-05-31T20:27:14.985 DEBUG:boto:auth_path=/pklxkd-54/b?versionId=null 2023-05-31T20:27:14.985 DEBUG:boto:Method: GET 2023-05-31T20:27:14.986 DEBUG:boto:Path: /pklxkd-54/b?versionId=null 2023-05-31T20:27:14.986 DEBUG:boto:Data: 2023-05-31T20:27:14.987 DEBUG:boto:Headers: {} 2023-05-31T20:27:14.987 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:27:14.987 DEBUG:boto:Port: 8001 2023-05-31T20:27:14.988 DEBUG:boto:Params: {} 2023-05-31T20:27:14.988 DEBUG:boto:Token: None 2023-05-31T20:27:14.989 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:27:14 GMT /pklxkd-54/b?versionId=null 2023-05-31T20:27:14.989 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:bVsesTfg0UVKS/hHZS1cUf/YLd4= 2023-05-31T20:27:14.990 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:27:14 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:bVsesTfg0UVKS/hHZS1cUf/YLd4=', 'Content-Length': '0'} 2023-05-31T20:27:14.991 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:26:44 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c3bef3bbcbeebff8-006477ada2-162e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:27:14 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:27:15.032 DEBUG:boto:path=/pklxkd-54/b 2023-05-31T20:27:15.033 DEBUG:boto:auth_path=/pklxkd-54/b 2023-05-31T20:27:15.033 DEBUG:boto:path=/pklxkd-54/b?versionId=null 2023-05-31T20:27:15.034 DEBUG:boto:auth_path=/pklxkd-54/b?versionId=null 2023-05-31T20:27:15.034 DEBUG:boto:Method: GET 2023-05-31T20:27:15.034 DEBUG:boto:Path: /pklxkd-54/b?versionId=null 2023-05-31T20:27:15.035 DEBUG:boto:Data: 2023-05-31T20:27:15.035 DEBUG:boto:Headers: {} 2023-05-31T20:27:15.035 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:27:15.036 DEBUG:boto:Port: 8000 2023-05-31T20:27:15.036 DEBUG:boto:Params: {} 2023-05-31T20:27:15.037 DEBUG:boto:Token: None 2023-05-31T20:27:15.037 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:27:15 GMT /pklxkd-54/b?versionId=null 2023-05-31T20:27:15.038 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:EaP3IsNG+MAmnxJA7q4oXBZ1yTQ= 2023-05-31T20:27:15.038 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:27:15 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:EaP3IsNG+MAmnxJA7q4oXBZ1yTQ=', 'Content-Length': '0'} 2023-05-31T20:27:15.040 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:26:44 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003c0b4ee224a375dc-006477ada3-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:27:15 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:27:15.041 DEBUG:boto:path=/pklxkd-54/ 2023-05-31T20:27:15.041 DEBUG:boto:auth_path=/pklxkd-54/ 2023-05-31T20:27:15.041 DEBUG:boto:path=/pklxkd-54/?prefix=b 2023-05-31T20:27:15.042 DEBUG:boto:auth_path=/pklxkd-54/?prefix=b 2023-05-31T20:27:15.042 DEBUG:boto:Method: GET 2023-05-31T20:27:15.042 DEBUG:boto:Path: /pklxkd-54/?prefix=b 2023-05-31T20:27:15.043 DEBUG:boto:Data: 2023-05-31T20:27:15.043 DEBUG:boto:Headers: {} 2023-05-31T20:27:15.044 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:27:15.044 DEBUG:boto:Port: 8001 2023-05-31T20:27:15.044 DEBUG:boto:Params: {} 2023-05-31T20:27:15.045 DEBUG:boto:Token: None 2023-05-31T20:27:15.045 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:27:15 GMT /pklxkd-54/ 2023-05-31T20:27:15.046 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:nQrx37OAwoLFO+xHyv742191gHM= 2023-05-31T20:27:15.046 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:27:15 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:nQrx37OAwoLFO+xHyv742191gHM=', 'Content-Length': '0'} 2023-05-31T20:27:15.048 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000098592e0b2c5999a3-006477ada3-162e-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:27:15 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:27:15.049 DEBUG:boto:b'pklxkd-54b1000falseb2023-05-31T20:26:44.064Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:27:15.050 DEBUG:boto:path=/pklxkd-54/ 2023-05-31T20:27:15.050 DEBUG:boto:auth_path=/pklxkd-54/ 2023-05-31T20:27:15.050 DEBUG:boto:path=/pklxkd-54/?prefix=b 2023-05-31T20:27:15.051 DEBUG:boto:auth_path=/pklxkd-54/?prefix=b 2023-05-31T20:27:15.051 DEBUG:boto:Method: GET 2023-05-31T20:27:15.051 DEBUG:boto:Path: /pklxkd-54/?prefix=b 2023-05-31T20:27:15.052 DEBUG:boto:Data: 2023-05-31T20:27:15.052 DEBUG:boto:Headers: {} 2023-05-31T20:27:15.052 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:27:15.053 DEBUG:boto:Port: 8000 2023-05-31T20:27:15.053 DEBUG:boto:Params: {} 2023-05-31T20:27:15.054 DEBUG:boto:Token: None 2023-05-31T20:27:15.054 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:27:15 GMT /pklxkd-54/ 2023-05-31T20:27:15.055 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:nQrx37OAwoLFO+xHyv742191gHM= 2023-05-31T20:27:15.055 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:27:15 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:nQrx37OAwoLFO+xHyv742191gHM=', 'Content-Length': '0'} 2023-05-31T20:27:15.057 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000007bf5204f4465ccdf-006477ada3-1630-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:27:15 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:27:15.058 DEBUG:boto:b'pklxkd-54b1000falseb2023-05-31T20:26:44.064Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:27:15.059 DEBUG:boto:path=/pklxkd-54/b 2023-05-31T20:27:15.059 DEBUG:boto:auth_path=/pklxkd-54/b 2023-05-31T20:27:15.059 DEBUG:boto:Method: HEAD 2023-05-31T20:27:15.060 DEBUG:boto:Path: /pklxkd-54/b 2023-05-31T20:27:15.060 DEBUG:boto:Data: 2023-05-31T20:27:15.061 DEBUG:boto:Headers: {} 2023-05-31T20:27:15.061 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:27:15.061 DEBUG:boto:Port: 8001 2023-05-31T20:27:15.062 DEBUG:boto:Params: {} 2023-05-31T20:27:15.062 DEBUG:boto:Token: None 2023-05-31T20:27:15.062 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:27:15 GMT /pklxkd-54/b 2023-05-31T20:27:15.063 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:3B1KRvCMvOQRM6DHRB1v+LYSqjg= 2023-05-31T20:27:15.063 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:27:15 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:3B1KRvCMvOQRM6DHRB1v+LYSqjg=', 'Content-Length': '0'} 2023-05-31T20:27:15.065 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:26:44 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000667891ac5166d111-006477ada3-162e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:27:15 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:27:15.066 DEBUG:boto:path=/pklxkd-54/b 2023-05-31T20:27:15.066 DEBUG:boto:auth_path=/pklxkd-54/b 2023-05-31T20:27:15.067 DEBUG:boto:Method: HEAD 2023-05-31T20:27:15.067 DEBUG:boto:Path: /pklxkd-54/b 2023-05-31T20:27:15.067 DEBUG:boto:Data: 2023-05-31T20:27:15.068 DEBUG:boto:Headers: {} 2023-05-31T20:27:15.068 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:27:15.068 DEBUG:boto:Port: 8000 2023-05-31T20:27:15.069 DEBUG:boto:Params: {} 2023-05-31T20:27:15.069 DEBUG:boto:Token: None 2023-05-31T20:27:15.069 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:27:15 GMT /pklxkd-54/b 2023-05-31T20:27:15.070 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:3B1KRvCMvOQRM6DHRB1v+LYSqjg= 2023-05-31T20:27:15.071 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:27:15 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:3B1KRvCMvOQRM6DHRB1v+LYSqjg=', 'Content-Length': '0'} 2023-05-31T20:27:15.072 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:26:44 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d29696b8eeaba4f9-006477ada3-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:27:15 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:27:15.073 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=b 2023-05-31T20:27:15.073 DEBUG:boto:path=/pklxkd-54/b 2023-05-31T20:27:15.073 DEBUG:boto:auth_path=/pklxkd-54/b 2023-05-31T20:27:15.074 DEBUG:boto:Method: GET 2023-05-31T20:27:15.074 DEBUG:boto:Path: /pklxkd-54/b 2023-05-31T20:27:15.075 DEBUG:boto:Data: 2023-05-31T20:27:15.075 DEBUG:boto:Headers: {} 2023-05-31T20:27:15.076 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:27:15.076 DEBUG:boto:Port: 8001 2023-05-31T20:27:15.077 DEBUG:boto:Params: {} 2023-05-31T20:27:15.077 DEBUG:boto:Token: None 2023-05-31T20:27:15.078 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:27:15 GMT /pklxkd-54/b 2023-05-31T20:27:15.078 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:FUjPKfAgQ83R4koPMx6K7Ss/AO4= 2023-05-31T20:27:15.079 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:27:15 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:FUjPKfAgQ83R4koPMx6K7Ss/AO4=', 'Content-Length': '0'} 2023-05-31T20:27:15.080 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:26:44 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001cc51deaaf4a4617-006477ada3-162e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:27:15 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:27:15.125 DEBUG:boto:path=/pklxkd-54/b 2023-05-31T20:27:15.125 DEBUG:boto:auth_path=/pklxkd-54/b 2023-05-31T20:27:15.126 DEBUG:boto:Method: GET 2023-05-31T20:27:15.126 DEBUG:boto:Path: /pklxkd-54/b 2023-05-31T20:27:15.127 DEBUG:boto:Data: 2023-05-31T20:27:15.127 DEBUG:boto:Headers: {} 2023-05-31T20:27:15.127 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:27:15.128 DEBUG:boto:Port: 8000 2023-05-31T20:27:15.128 DEBUG:boto:Params: {} 2023-05-31T20:27:15.129 DEBUG:boto:Token: None 2023-05-31T20:27:15.129 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:27:15 GMT /pklxkd-54/b 2023-05-31T20:27:15.130 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:FUjPKfAgQ83R4koPMx6K7Ss/AO4= 2023-05-31T20:27:15.130 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:27:15 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:FUjPKfAgQ83R4koPMx6K7Ss/AO4=', 'Content-Length': '0'} 2023-05-31T20:27:15.132 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:26:44 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000042d44de8753dc9da-006477ada3-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:27:15 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:27:15.133 DEBUG:boto:path=/pklxkd-54/ 2023-05-31T20:27:15.134 DEBUG:boto:auth_path=/pklxkd-54/ 2023-05-31T20:27:15.134 DEBUG:boto:path=/pklxkd-54/?prefix=b 2023-05-31T20:27:15.134 DEBUG:boto:auth_path=/pklxkd-54/?prefix=b 2023-05-31T20:27:15.135 DEBUG:boto:Method: GET 2023-05-31T20:27:15.135 DEBUG:boto:Path: /pklxkd-54/?prefix=b 2023-05-31T20:27:15.135 DEBUG:boto:Data: 2023-05-31T20:27:15.136 DEBUG:boto:Headers: {} 2023-05-31T20:27:15.136 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:27:15.137 DEBUG:boto:Port: 8001 2023-05-31T20:27:15.137 DEBUG:boto:Params: {} 2023-05-31T20:27:15.137 DEBUG:boto:Token: None 2023-05-31T20:27:15.138 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:27:15 GMT /pklxkd-54/ 2023-05-31T20:27:15.138 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:nQrx37OAwoLFO+xHyv742191gHM= 2023-05-31T20:27:15.139 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:27:15 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:nQrx37OAwoLFO+xHyv742191gHM=', 'Content-Length': '0'} 2023-05-31T20:27:15.141 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000687a669e55e63714-006477ada3-162e-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:27:15 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:27:15.142 DEBUG:boto:b'pklxkd-54b1000falseb2023-05-31T20:26:44.064Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:27:15.142 DEBUG:boto:path=/pklxkd-54/ 2023-05-31T20:27:15.143 DEBUG:boto:auth_path=/pklxkd-54/ 2023-05-31T20:27:15.143 DEBUG:boto:path=/pklxkd-54/?prefix=b 2023-05-31T20:27:15.143 DEBUG:boto:auth_path=/pklxkd-54/?prefix=b 2023-05-31T20:27:15.144 DEBUG:boto:Method: GET 2023-05-31T20:27:15.144 DEBUG:boto:Path: /pklxkd-54/?prefix=b 2023-05-31T20:27:15.145 DEBUG:boto:Data: 2023-05-31T20:27:15.145 DEBUG:boto:Headers: {} 2023-05-31T20:27:15.145 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:27:15.146 DEBUG:boto:Port: 8000 2023-05-31T20:27:15.146 DEBUG:boto:Params: {} 2023-05-31T20:27:15.146 DEBUG:boto:Token: None 2023-05-31T20:27:15.147 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:27:15 GMT /pklxkd-54/ 2023-05-31T20:27:15.147 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:nQrx37OAwoLFO+xHyv742191gHM= 2023-05-31T20:27:15.148 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:27:15 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:nQrx37OAwoLFO+xHyv742191gHM=', 'Content-Length': '0'} 2023-05-31T20:27:15.151 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000009d254a5a01d5f8bf-006477ada3-1630-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:27:15 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:27:15.151 DEBUG:boto:b'pklxkd-54b1000falseb2023-05-31T20:26:44.064Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:27:15.152 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=c 2023-05-31T20:27:15.152 DEBUG:boto:path=/pklxkd-54/c 2023-05-31T20:27:15.153 DEBUG:boto:auth_path=/pklxkd-54/c 2023-05-31T20:27:15.153 DEBUG:boto:path=/pklxkd-54/c?versionId=null 2023-05-31T20:27:15.153 DEBUG:boto:auth_path=/pklxkd-54/c?versionId=null 2023-05-31T20:27:15.154 DEBUG:boto:Method: GET 2023-05-31T20:27:15.154 DEBUG:boto:Path: /pklxkd-54/c?versionId=null 2023-05-31T20:27:15.155 DEBUG:boto:Data: 2023-05-31T20:27:15.155 DEBUG:boto:Headers: {} 2023-05-31T20:27:15.156 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:27:15.156 DEBUG:boto:Port: 8001 2023-05-31T20:27:15.156 DEBUG:boto:Params: {} 2023-05-31T20:27:15.157 DEBUG:boto:Token: None 2023-05-31T20:27:15.157 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:27:15 GMT /pklxkd-54/c?versionId=null 2023-05-31T20:27:15.158 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:HMS0PITSR4UvV2x11zmk0OEYv0w= 2023-05-31T20:27:15.158 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:27:15 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:HMS0PITSR4UvV2x11zmk0OEYv0w=', 'Content-Length': '0'} 2023-05-31T20:27:15.160 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:26:44 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000285123c82cc45c76-006477ada3-162e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:27:15 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:27:15.161 DEBUG:boto:path=/pklxkd-54/c 2023-05-31T20:27:15.162 DEBUG:boto:auth_path=/pklxkd-54/c 2023-05-31T20:27:15.162 DEBUG:boto:path=/pklxkd-54/c?versionId=null 2023-05-31T20:27:15.162 DEBUG:boto:auth_path=/pklxkd-54/c?versionId=null 2023-05-31T20:27:15.163 DEBUG:boto:Method: GET 2023-05-31T20:27:15.163 DEBUG:boto:Path: /pklxkd-54/c?versionId=null 2023-05-31T20:27:15.164 DEBUG:boto:Data: 2023-05-31T20:27:15.164 DEBUG:boto:Headers: {} 2023-05-31T20:27:15.164 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:27:15.165 DEBUG:boto:Port: 8000 2023-05-31T20:27:15.165 DEBUG:boto:Params: {} 2023-05-31T20:27:15.166 DEBUG:boto:Token: None 2023-05-31T20:27:15.166 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:27:15 GMT /pklxkd-54/c?versionId=null 2023-05-31T20:27:15.167 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:HMS0PITSR4UvV2x11zmk0OEYv0w= 2023-05-31T20:27:15.167 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:27:15 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:HMS0PITSR4UvV2x11zmk0OEYv0w=', 'Content-Length': '0'} 2023-05-31T20:27:15.169 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:26:44 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005eae8327ab90c947-006477ada3-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:27:15 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:27:15.170 DEBUG:boto:path=/pklxkd-54/c 2023-05-31T20:27:15.170 DEBUG:boto:auth_path=/pklxkd-54/c 2023-05-31T20:27:15.171 DEBUG:boto:path=/pklxkd-54/c?versionId=null 2023-05-31T20:27:15.172 DEBUG:boto:auth_path=/pklxkd-54/c?versionId=null 2023-05-31T20:27:15.173 DEBUG:boto:Method: HEAD 2023-05-31T20:27:15.173 DEBUG:boto:Path: /pklxkd-54/c?versionId=null 2023-05-31T20:27:15.174 DEBUG:boto:Data: 2023-05-31T20:27:15.174 DEBUG:boto:Headers: {} 2023-05-31T20:27:15.174 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:27:15.175 DEBUG:boto:Port: 8001 2023-05-31T20:27:15.175 DEBUG:boto:Params: {} 2023-05-31T20:27:15.175 DEBUG:boto:Token: None 2023-05-31T20:27:15.176 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:27:15 GMT /pklxkd-54/c?versionId=null 2023-05-31T20:27:15.176 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:wmAzycNDexgIiMfLk5s3pydiI/A= 2023-05-31T20:27:15.177 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:27:15 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:wmAzycNDexgIiMfLk5s3pydiI/A=', 'Content-Length': '0'} 2023-05-31T20:27:15.179 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:26:44 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a2aa9a60525273ca-006477ada3-162e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:27:15 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:27:15.179 DEBUG:boto:path=/pklxkd-54/c 2023-05-31T20:27:15.180 DEBUG:boto:auth_path=/pklxkd-54/c 2023-05-31T20:27:15.181 DEBUG:boto:path=/pklxkd-54/c?versionId=null 2023-05-31T20:27:15.181 DEBUG:boto:auth_path=/pklxkd-54/c?versionId=null 2023-05-31T20:27:15.181 DEBUG:boto:Method: HEAD 2023-05-31T20:27:15.182 DEBUG:boto:Path: /pklxkd-54/c?versionId=null 2023-05-31T20:27:15.182 DEBUG:boto:Data: 2023-05-31T20:27:15.183 DEBUG:boto:Headers: {} 2023-05-31T20:27:15.183 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:27:15.183 DEBUG:boto:Port: 8000 2023-05-31T20:27:15.184 DEBUG:boto:Params: {} 2023-05-31T20:27:15.184 DEBUG:boto:Token: None 2023-05-31T20:27:15.185 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:27:15 GMT /pklxkd-54/c?versionId=null 2023-05-31T20:27:15.185 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:wmAzycNDexgIiMfLk5s3pydiI/A= 2023-05-31T20:27:15.186 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:27:15 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:wmAzycNDexgIiMfLk5s3pydiI/A=', 'Content-Length': '0'} 2023-05-31T20:27:15.188 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:26:44 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f2ce27632dc2d6ce-006477ada3-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:27:15 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:27:15.189 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=c 2023-05-31T20:27:15.189 DEBUG:boto:path=/pklxkd-54/c 2023-05-31T20:27:15.190 DEBUG:boto:auth_path=/pklxkd-54/c 2023-05-31T20:27:15.190 DEBUG:boto:path=/pklxkd-54/c?versionId=null 2023-05-31T20:27:15.190 DEBUG:boto:auth_path=/pklxkd-54/c?versionId=null 2023-05-31T20:27:15.191 DEBUG:boto:Method: GET 2023-05-31T20:27:15.191 DEBUG:boto:Path: /pklxkd-54/c?versionId=null 2023-05-31T20:27:15.191 DEBUG:boto:Data: 2023-05-31T20:27:15.192 DEBUG:boto:Headers: {} 2023-05-31T20:27:15.192 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:27:15.193 DEBUG:boto:Port: 8001 2023-05-31T20:27:15.193 DEBUG:boto:Params: {} 2023-05-31T20:27:15.193 DEBUG:boto:Token: None 2023-05-31T20:27:15.194 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:27:15 GMT /pklxkd-54/c?versionId=null 2023-05-31T20:27:15.194 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:HMS0PITSR4UvV2x11zmk0OEYv0w= 2023-05-31T20:27:15.195 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:27:15 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:HMS0PITSR4UvV2x11zmk0OEYv0w=', 'Content-Length': '0'} 2023-05-31T20:27:15.197 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:26:44 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000020ea57ee6ebcd9da-006477ada3-162e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:27:15 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:27:15.241 DEBUG:boto:path=/pklxkd-54/c 2023-05-31T20:27:15.241 DEBUG:boto:auth_path=/pklxkd-54/c 2023-05-31T20:27:15.242 DEBUG:boto:path=/pklxkd-54/c?versionId=null 2023-05-31T20:27:15.242 DEBUG:boto:auth_path=/pklxkd-54/c?versionId=null 2023-05-31T20:27:15.242 DEBUG:boto:Method: GET 2023-05-31T20:27:15.243 DEBUG:boto:Path: /pklxkd-54/c?versionId=null 2023-05-31T20:27:15.243 DEBUG:boto:Data: 2023-05-31T20:27:15.244 DEBUG:boto:Headers: {} 2023-05-31T20:27:15.244 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:27:15.244 DEBUG:boto:Port: 8000 2023-05-31T20:27:15.245 DEBUG:boto:Params: {} 2023-05-31T20:27:15.245 DEBUG:boto:Token: None 2023-05-31T20:27:15.246 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:27:15 GMT /pklxkd-54/c?versionId=null 2023-05-31T20:27:15.246 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:HMS0PITSR4UvV2x11zmk0OEYv0w= 2023-05-31T20:27:15.247 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:27:15 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:HMS0PITSR4UvV2x11zmk0OEYv0w=', 'Content-Length': '0'} 2023-05-31T20:27:15.249 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:26:44 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000036bc87b44bf1531e-006477ada3-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:27:15 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:27:15.250 DEBUG:boto:path=/pklxkd-54/ 2023-05-31T20:27:15.250 DEBUG:boto:auth_path=/pklxkd-54/ 2023-05-31T20:27:15.251 DEBUG:boto:path=/pklxkd-54/?prefix=c 2023-05-31T20:27:15.251 DEBUG:boto:auth_path=/pklxkd-54/?prefix=c 2023-05-31T20:27:15.251 DEBUG:boto:Method: GET 2023-05-31T20:27:15.252 DEBUG:boto:Path: /pklxkd-54/?prefix=c 2023-05-31T20:27:15.252 DEBUG:boto:Data: 2023-05-31T20:27:15.252 DEBUG:boto:Headers: {} 2023-05-31T20:27:15.253 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:27:15.253 DEBUG:boto:Port: 8001 2023-05-31T20:27:15.253 DEBUG:boto:Params: {} 2023-05-31T20:27:15.254 DEBUG:boto:Token: None 2023-05-31T20:27:15.254 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:27:15 GMT /pklxkd-54/ 2023-05-31T20:27:15.255 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:nQrx37OAwoLFO+xHyv742191gHM= 2023-05-31T20:27:15.255 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:27:15 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:nQrx37OAwoLFO+xHyv742191gHM=', 'Content-Length': '0'} 2023-05-31T20:27:15.257 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000035e31cc0cb5dcd5c-006477ada3-162e-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:27:15 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:27:15.258 DEBUG:boto:b'pklxkd-54c1000falsec2023-05-31T20:26:44.119Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:27:15.259 DEBUG:boto:path=/pklxkd-54/ 2023-05-31T20:27:15.259 DEBUG:boto:auth_path=/pklxkd-54/ 2023-05-31T20:27:15.260 DEBUG:boto:path=/pklxkd-54/?prefix=c 2023-05-31T20:27:15.260 DEBUG:boto:auth_path=/pklxkd-54/?prefix=c 2023-05-31T20:27:15.260 DEBUG:boto:Method: GET 2023-05-31T20:27:15.261 DEBUG:boto:Path: /pklxkd-54/?prefix=c 2023-05-31T20:27:15.261 DEBUG:boto:Data: 2023-05-31T20:27:15.262 DEBUG:boto:Headers: {} 2023-05-31T20:27:15.262 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:27:15.262 DEBUG:boto:Port: 8000 2023-05-31T20:27:15.263 DEBUG:boto:Params: {} 2023-05-31T20:27:15.263 DEBUG:boto:Token: None 2023-05-31T20:27:15.264 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:27:15 GMT /pklxkd-54/ 2023-05-31T20:27:15.264 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:nQrx37OAwoLFO+xHyv742191gHM= 2023-05-31T20:27:15.266 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:27:15 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:nQrx37OAwoLFO+xHyv742191gHM=', 'Content-Length': '0'} 2023-05-31T20:27:15.268 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000017fe8a58b05fe2fd-006477ada3-1630-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:27:15 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:27:15.308 DEBUG:boto:b'pklxkd-54c1000falsec2023-05-31T20:26:44.119Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:27:15.309 DEBUG:boto:path=/pklxkd-54/c 2023-05-31T20:27:15.310 DEBUG:boto:auth_path=/pklxkd-54/c 2023-05-31T20:27:15.310 DEBUG:boto:Method: HEAD 2023-05-31T20:27:15.311 DEBUG:boto:Path: /pklxkd-54/c 2023-05-31T20:27:15.311 DEBUG:boto:Data: 2023-05-31T20:27:15.311 DEBUG:boto:Headers: {} 2023-05-31T20:27:15.312 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:27:15.312 DEBUG:boto:Port: 8001 2023-05-31T20:27:15.312 DEBUG:boto:Params: {} 2023-05-31T20:27:15.313 DEBUG:boto:Token: None 2023-05-31T20:27:15.313 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:27:15 GMT /pklxkd-54/c 2023-05-31T20:27:15.314 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:GPD9n4DDKQMUJ+ITvAmmTtjHGsk= 2023-05-31T20:27:15.314 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:27:15 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:GPD9n4DDKQMUJ+ITvAmmTtjHGsk=', 'Content-Length': '0'} 2023-05-31T20:27:15.316 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:26:44 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001db4e880a9459869-006477ada3-162e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:27:15 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:27:15.317 DEBUG:boto:path=/pklxkd-54/c 2023-05-31T20:27:15.317 DEBUG:boto:auth_path=/pklxkd-54/c 2023-05-31T20:27:15.318 DEBUG:boto:Method: HEAD 2023-05-31T20:27:15.318 DEBUG:boto:Path: /pklxkd-54/c 2023-05-31T20:27:15.318 DEBUG:boto:Data: 2023-05-31T20:27:15.319 DEBUG:boto:Headers: {} 2023-05-31T20:27:15.319 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:27:15.319 DEBUG:boto:Port: 8000 2023-05-31T20:27:15.320 DEBUG:boto:Params: {} 2023-05-31T20:27:15.320 DEBUG:boto:Token: None 2023-05-31T20:27:15.321 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:27:15 GMT /pklxkd-54/c 2023-05-31T20:27:15.321 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:GPD9n4DDKQMUJ+ITvAmmTtjHGsk= 2023-05-31T20:27:15.321 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:27:15 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:GPD9n4DDKQMUJ+ITvAmmTtjHGsk=', 'Content-Length': '0'} 2023-05-31T20:27:15.323 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:26:44 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000823c9e24aa74b52e-006477ada3-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:27:15 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:27:15.324 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=c 2023-05-31T20:27:15.324 DEBUG:boto:path=/pklxkd-54/c 2023-05-31T20:27:15.325 DEBUG:boto:auth_path=/pklxkd-54/c 2023-05-31T20:27:15.325 DEBUG:boto:Method: GET 2023-05-31T20:27:15.325 DEBUG:boto:Path: /pklxkd-54/c 2023-05-31T20:27:15.326 DEBUG:boto:Data: 2023-05-31T20:27:15.326 DEBUG:boto:Headers: {} 2023-05-31T20:27:15.327 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:27:15.327 DEBUG:boto:Port: 8001 2023-05-31T20:27:15.327 DEBUG:boto:Params: {} 2023-05-31T20:27:15.328 DEBUG:boto:Token: None 2023-05-31T20:27:15.328 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:27:15 GMT /pklxkd-54/c 2023-05-31T20:27:15.329 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:NHowAzndfJrj5fiRMBlao2VDTXI= 2023-05-31T20:27:15.329 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:27:15 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:NHowAzndfJrj5fiRMBlao2VDTXI=', 'Content-Length': '0'} 2023-05-31T20:27:15.331 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:26:44 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000ab055add2d0d943-006477ada3-162e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:27:15 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:27:15.331 DEBUG:boto:path=/pklxkd-54/c 2023-05-31T20:27:15.332 DEBUG:boto:auth_path=/pklxkd-54/c 2023-05-31T20:27:15.332 DEBUG:boto:Method: GET 2023-05-31T20:27:15.333 DEBUG:boto:Path: /pklxkd-54/c 2023-05-31T20:27:15.333 DEBUG:boto:Data: 2023-05-31T20:27:15.334 DEBUG:boto:Headers: {} 2023-05-31T20:27:15.334 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:27:15.334 DEBUG:boto:Port: 8000 2023-05-31T20:27:15.335 DEBUG:boto:Params: {} 2023-05-31T20:27:15.335 DEBUG:boto:Token: None 2023-05-31T20:27:15.335 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:27:15 GMT /pklxkd-54/c 2023-05-31T20:27:15.336 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:NHowAzndfJrj5fiRMBlao2VDTXI= 2023-05-31T20:27:15.336 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:27:15 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:NHowAzndfJrj5fiRMBlao2VDTXI=', 'Content-Length': '0'} 2023-05-31T20:27:15.338 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:26:44 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b5c1f41d49fe70a7-006477ada3-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:27:15 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:27:15.339 DEBUG:boto:path=/pklxkd-54/ 2023-05-31T20:27:15.339 DEBUG:boto:auth_path=/pklxkd-54/ 2023-05-31T20:27:15.340 DEBUG:boto:path=/pklxkd-54/?prefix=c 2023-05-31T20:27:15.340 DEBUG:boto:auth_path=/pklxkd-54/?prefix=c 2023-05-31T20:27:15.340 DEBUG:boto:Method: GET 2023-05-31T20:27:15.341 DEBUG:boto:Path: /pklxkd-54/?prefix=c 2023-05-31T20:27:15.341 DEBUG:boto:Data: 2023-05-31T20:27:15.342 DEBUG:boto:Headers: {} 2023-05-31T20:27:15.342 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:27:15.342 DEBUG:boto:Port: 8001 2023-05-31T20:27:15.343 DEBUG:boto:Params: {} 2023-05-31T20:27:15.343 DEBUG:boto:Token: None 2023-05-31T20:27:15.344 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:27:15 GMT /pklxkd-54/ 2023-05-31T20:27:15.344 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:nQrx37OAwoLFO+xHyv742191gHM= 2023-05-31T20:27:15.345 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:27:15 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:nQrx37OAwoLFO+xHyv742191gHM=', 'Content-Length': '0'} 2023-05-31T20:27:15.347 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a7fc027d8421aa46-006477ada3-162e-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:27:15 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:27:15.347 DEBUG:boto:b'pklxkd-54c1000falsec2023-05-31T20:26:44.119Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:27:15.348 DEBUG:boto:path=/pklxkd-54/ 2023-05-31T20:27:15.348 DEBUG:boto:auth_path=/pklxkd-54/ 2023-05-31T20:27:15.349 DEBUG:boto:path=/pklxkd-54/?prefix=c 2023-05-31T20:27:15.349 DEBUG:boto:auth_path=/pklxkd-54/?prefix=c 2023-05-31T20:27:15.350 DEBUG:boto:Method: GET 2023-05-31T20:27:15.350 DEBUG:boto:Path: /pklxkd-54/?prefix=c 2023-05-31T20:27:15.350 DEBUG:boto:Data: 2023-05-31T20:27:15.351 DEBUG:boto:Headers: {} 2023-05-31T20:27:15.351 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:27:15.352 DEBUG:boto:Port: 8000 2023-05-31T20:27:15.352 DEBUG:boto:Params: {} 2023-05-31T20:27:15.352 DEBUG:boto:Token: None 2023-05-31T20:27:15.353 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:27:15 GMT /pklxkd-54/ 2023-05-31T20:27:15.353 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:nQrx37OAwoLFO+xHyv742191gHM= 2023-05-31T20:27:15.354 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:27:15 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:nQrx37OAwoLFO+xHyv742191gHM=', 'Content-Length': '0'} 2023-05-31T20:27:15.356 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000005d672ff34d5edd4a-006477ada3-1630-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:27:15 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:27:15.357 DEBUG:boto:b'pklxkd-54c1000falsec2023-05-31T20:26:44.119Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:27:15.357 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=d 2023-05-31T20:27:15.358 DEBUG:boto:path=/pklxkd-54/d 2023-05-31T20:27:15.358 DEBUG:boto:auth_path=/pklxkd-54/d 2023-05-31T20:27:15.358 DEBUG:boto:path=/pklxkd-54/d?versionId=null 2023-05-31T20:27:15.359 DEBUG:boto:auth_path=/pklxkd-54/d?versionId=null 2023-05-31T20:27:15.359 DEBUG:boto:Method: GET 2023-05-31T20:27:15.360 DEBUG:boto:Path: /pklxkd-54/d?versionId=null 2023-05-31T20:27:15.360 DEBUG:boto:Data: 2023-05-31T20:27:15.360 DEBUG:boto:Headers: {} 2023-05-31T20:27:15.361 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:27:15.361 DEBUG:boto:Port: 8001 2023-05-31T20:27:15.362 DEBUG:boto:Params: {} 2023-05-31T20:27:15.362 DEBUG:boto:Token: None 2023-05-31T20:27:15.362 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:27:15 GMT /pklxkd-54/d?versionId=null 2023-05-31T20:27:15.363 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:kulLiT/BTT/UfV04YySaTyYGKCk= 2023-05-31T20:27:15.363 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:27:15 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:kulLiT/BTT/UfV04YySaTyYGKCk=', 'Content-Length': '0'} 2023-05-31T20:27:15.365 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:26:44 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000014405967485a24d7-006477ada3-162e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:27:15 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:27:15.366 DEBUG:boto:path=/pklxkd-54/d 2023-05-31T20:27:15.366 DEBUG:boto:auth_path=/pklxkd-54/d 2023-05-31T20:27:15.367 DEBUG:boto:path=/pklxkd-54/d?versionId=null 2023-05-31T20:27:15.367 DEBUG:boto:auth_path=/pklxkd-54/d?versionId=null 2023-05-31T20:27:15.367 DEBUG:boto:Method: GET 2023-05-31T20:27:15.368 DEBUG:boto:Path: /pklxkd-54/d?versionId=null 2023-05-31T20:27:15.368 DEBUG:boto:Data: 2023-05-31T20:27:15.369 DEBUG:boto:Headers: {} 2023-05-31T20:27:15.369 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:27:15.369 DEBUG:boto:Port: 8000 2023-05-31T20:27:15.370 DEBUG:boto:Params: {} 2023-05-31T20:27:15.370 DEBUG:boto:Token: None 2023-05-31T20:27:15.371 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:27:15 GMT /pklxkd-54/d?versionId=null 2023-05-31T20:27:15.371 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:kulLiT/BTT/UfV04YySaTyYGKCk= 2023-05-31T20:27:15.371 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:27:15 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:kulLiT/BTT/UfV04YySaTyYGKCk=', 'Content-Length': '0'} 2023-05-31T20:27:15.374 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:26:44 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000502a5335d4d2086f-006477ada3-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:27:15 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:27:15.374 DEBUG:boto:path=/pklxkd-54/d 2023-05-31T20:27:15.375 DEBUG:boto:auth_path=/pklxkd-54/d 2023-05-31T20:27:15.375 DEBUG:boto:path=/pklxkd-54/d?versionId=null 2023-05-31T20:27:15.375 DEBUG:boto:auth_path=/pklxkd-54/d?versionId=null 2023-05-31T20:27:15.376 DEBUG:boto:Method: HEAD 2023-05-31T20:27:15.376 DEBUG:boto:Path: /pklxkd-54/d?versionId=null 2023-05-31T20:27:15.376 DEBUG:boto:Data: 2023-05-31T20:27:15.377 DEBUG:boto:Headers: {} 2023-05-31T20:27:15.377 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:27:15.377 DEBUG:boto:Port: 8001 2023-05-31T20:27:15.378 DEBUG:boto:Params: {} 2023-05-31T20:27:15.378 DEBUG:boto:Token: None 2023-05-31T20:27:15.379 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:27:15 GMT /pklxkd-54/d?versionId=null 2023-05-31T20:27:15.379 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:WaEQ3ftxd3+H9NdRoHl+oaoDOqY= 2023-05-31T20:27:15.379 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:27:15 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:WaEQ3ftxd3+H9NdRoHl+oaoDOqY=', 'Content-Length': '0'} 2023-05-31T20:27:15.381 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:26:44 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006ab30194f18f69c2-006477ada3-162e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:27:15 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:27:15.382 DEBUG:boto:path=/pklxkd-54/d 2023-05-31T20:27:15.382 DEBUG:boto:auth_path=/pklxkd-54/d 2023-05-31T20:27:15.383 DEBUG:boto:path=/pklxkd-54/d?versionId=null 2023-05-31T20:27:15.383 DEBUG:boto:auth_path=/pklxkd-54/d?versionId=null 2023-05-31T20:27:15.383 DEBUG:boto:Method: HEAD 2023-05-31T20:27:15.384 DEBUG:boto:Path: /pklxkd-54/d?versionId=null 2023-05-31T20:27:15.384 DEBUG:boto:Data: 2023-05-31T20:27:15.384 DEBUG:boto:Headers: {} 2023-05-31T20:27:15.385 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:27:15.385 DEBUG:boto:Port: 8000 2023-05-31T20:27:15.385 DEBUG:boto:Params: {} 2023-05-31T20:27:15.386 DEBUG:boto:Token: None 2023-05-31T20:27:15.386 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:27:15 GMT /pklxkd-54/d?versionId=null 2023-05-31T20:27:15.387 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:WaEQ3ftxd3+H9NdRoHl+oaoDOqY= 2023-05-31T20:27:15.387 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:27:15 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:WaEQ3ftxd3+H9NdRoHl+oaoDOqY=', 'Content-Length': '0'} 2023-05-31T20:27:15.389 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:26:44 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006d8302bb324633a0-006477ada3-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:27:15 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:27:15.389 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=d 2023-05-31T20:27:15.390 DEBUG:boto:path=/pklxkd-54/d 2023-05-31T20:27:15.390 DEBUG:boto:auth_path=/pklxkd-54/d 2023-05-31T20:27:15.391 DEBUG:boto:path=/pklxkd-54/d?versionId=null 2023-05-31T20:27:15.391 DEBUG:boto:auth_path=/pklxkd-54/d?versionId=null 2023-05-31T20:27:15.391 DEBUG:boto:Method: GET 2023-05-31T20:27:15.392 DEBUG:boto:Path: /pklxkd-54/d?versionId=null 2023-05-31T20:27:15.392 DEBUG:boto:Data: 2023-05-31T20:27:15.392 DEBUG:boto:Headers: {} 2023-05-31T20:27:15.393 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:27:15.393 DEBUG:boto:Port: 8001 2023-05-31T20:27:15.393 DEBUG:boto:Params: {} 2023-05-31T20:27:15.394 DEBUG:boto:Token: None 2023-05-31T20:27:15.394 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:27:15 GMT /pklxkd-54/d?versionId=null 2023-05-31T20:27:15.395 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:kulLiT/BTT/UfV04YySaTyYGKCk= 2023-05-31T20:27:15.395 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:27:15 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:kulLiT/BTT/UfV04YySaTyYGKCk=', 'Content-Length': '0'} 2023-05-31T20:27:15.397 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:26:44 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000fbb0dcccc2208e90-006477ada3-162e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:27:15 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:27:15.397 DEBUG:boto:path=/pklxkd-54/d 2023-05-31T20:27:15.398 DEBUG:boto:auth_path=/pklxkd-54/d 2023-05-31T20:27:15.398 DEBUG:boto:path=/pklxkd-54/d?versionId=null 2023-05-31T20:27:15.399 DEBUG:boto:auth_path=/pklxkd-54/d?versionId=null 2023-05-31T20:27:15.399 DEBUG:boto:Method: GET 2023-05-31T20:27:15.399 DEBUG:boto:Path: /pklxkd-54/d?versionId=null 2023-05-31T20:27:15.400 DEBUG:boto:Data: 2023-05-31T20:27:15.400 DEBUG:boto:Headers: {} 2023-05-31T20:27:15.401 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:27:15.401 DEBUG:boto:Port: 8000 2023-05-31T20:27:15.401 DEBUG:boto:Params: {} 2023-05-31T20:27:15.402 DEBUG:boto:Token: None 2023-05-31T20:27:15.402 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:27:15 GMT /pklxkd-54/d?versionId=null 2023-05-31T20:27:15.403 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:kulLiT/BTT/UfV04YySaTyYGKCk= 2023-05-31T20:27:15.403 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:27:15 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:kulLiT/BTT/UfV04YySaTyYGKCk=', 'Content-Length': '0'} 2023-05-31T20:27:15.405 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:26:44 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e4132af2bc883dad-006477ada3-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:27:15 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:27:15.448 DEBUG:boto:path=/pklxkd-54/ 2023-05-31T20:27:15.449 DEBUG:boto:auth_path=/pklxkd-54/ 2023-05-31T20:27:15.449 DEBUG:boto:path=/pklxkd-54/?prefix=d 2023-05-31T20:27:15.450 DEBUG:boto:auth_path=/pklxkd-54/?prefix=d 2023-05-31T20:27:15.450 DEBUG:boto:Method: GET 2023-05-31T20:27:15.450 DEBUG:boto:Path: /pklxkd-54/?prefix=d 2023-05-31T20:27:15.451 DEBUG:boto:Data: 2023-05-31T20:27:15.451 DEBUG:boto:Headers: {} 2023-05-31T20:27:15.451 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:27:15.452 DEBUG:boto:Port: 8001 2023-05-31T20:27:15.452 DEBUG:boto:Params: {} 2023-05-31T20:27:15.453 DEBUG:boto:Token: None 2023-05-31T20:27:15.453 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:27:15 GMT /pklxkd-54/ 2023-05-31T20:27:15.453 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:nQrx37OAwoLFO+xHyv742191gHM= 2023-05-31T20:27:15.454 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:27:15 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:nQrx37OAwoLFO+xHyv742191gHM=', 'Content-Length': '0'} 2023-05-31T20:27:15.457 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000857398bbba946d20-006477ada3-162e-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:27:15 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:27:15.457 DEBUG:boto:b'pklxkd-54d1000falsed2023-05-31T20:26:44.140Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:27:15.458 DEBUG:boto:path=/pklxkd-54/ 2023-05-31T20:27:15.458 DEBUG:boto:auth_path=/pklxkd-54/ 2023-05-31T20:27:15.459 DEBUG:boto:path=/pklxkd-54/?prefix=d 2023-05-31T20:27:15.459 DEBUG:boto:auth_path=/pklxkd-54/?prefix=d 2023-05-31T20:27:15.459 DEBUG:boto:Method: GET 2023-05-31T20:27:15.460 DEBUG:boto:Path: /pklxkd-54/?prefix=d 2023-05-31T20:27:15.460 DEBUG:boto:Data: 2023-05-31T20:27:15.461 DEBUG:boto:Headers: {} 2023-05-31T20:27:15.461 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:27:15.461 DEBUG:boto:Port: 8000 2023-05-31T20:27:15.462 DEBUG:boto:Params: {} 2023-05-31T20:27:15.462 DEBUG:boto:Token: None 2023-05-31T20:27:15.463 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:27:15 GMT /pklxkd-54/ 2023-05-31T20:27:15.463 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:nQrx37OAwoLFO+xHyv742191gHM= 2023-05-31T20:27:15.463 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:27:15 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:nQrx37OAwoLFO+xHyv742191gHM=', 'Content-Length': '0'} 2023-05-31T20:27:15.466 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000080e01fb971072635-006477ada3-1630-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:27:15 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:27:15.466 DEBUG:boto:b'pklxkd-54d1000falsed2023-05-31T20:26:44.140Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:27:15.467 DEBUG:boto:path=/pklxkd-54/d 2023-05-31T20:27:15.467 DEBUG:boto:auth_path=/pklxkd-54/d 2023-05-31T20:27:15.468 DEBUG:boto:Method: HEAD 2023-05-31T20:27:15.468 DEBUG:boto:Path: /pklxkd-54/d 2023-05-31T20:27:15.469 DEBUG:boto:Data: 2023-05-31T20:27:15.469 DEBUG:boto:Headers: {} 2023-05-31T20:27:15.469 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:27:15.470 DEBUG:boto:Port: 8001 2023-05-31T20:27:15.470 DEBUG:boto:Params: {} 2023-05-31T20:27:15.470 DEBUG:boto:Token: None 2023-05-31T20:27:15.471 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:27:15 GMT /pklxkd-54/d 2023-05-31T20:27:15.471 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:Zjax2Xi6srnEgN9f66C6Pe/SI0M= 2023-05-31T20:27:15.471 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:27:15 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:Zjax2Xi6srnEgN9f66C6Pe/SI0M=', 'Content-Length': '0'} 2023-05-31T20:27:15.473 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:26:44 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007a2d7ab116efcfba-006477ada3-162e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:27:15 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:27:15.474 DEBUG:boto:path=/pklxkd-54/d 2023-05-31T20:27:15.474 DEBUG:boto:auth_path=/pklxkd-54/d 2023-05-31T20:27:15.475 DEBUG:boto:Method: HEAD 2023-05-31T20:27:15.475 DEBUG:boto:Path: /pklxkd-54/d 2023-05-31T20:27:15.475 DEBUG:boto:Data: 2023-05-31T20:27:15.476 DEBUG:boto:Headers: {} 2023-05-31T20:27:15.476 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:27:15.476 DEBUG:boto:Port: 8000 2023-05-31T20:27:15.477 DEBUG:boto:Params: {} 2023-05-31T20:27:15.477 DEBUG:boto:Token: None 2023-05-31T20:27:15.478 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:27:15 GMT /pklxkd-54/d 2023-05-31T20:27:15.478 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:Zjax2Xi6srnEgN9f66C6Pe/SI0M= 2023-05-31T20:27:15.478 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:27:15 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:Zjax2Xi6srnEgN9f66C6Pe/SI0M=', 'Content-Length': '0'} 2023-05-31T20:27:15.480 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:26:44 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003d47a27e5edc7638-006477ada3-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:27:15 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:27:15.481 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=d 2023-05-31T20:27:15.481 DEBUG:boto:path=/pklxkd-54/d 2023-05-31T20:27:15.481 DEBUG:boto:auth_path=/pklxkd-54/d 2023-05-31T20:27:15.482 DEBUG:boto:Method: GET 2023-05-31T20:27:15.482 DEBUG:boto:Path: /pklxkd-54/d 2023-05-31T20:27:15.483 DEBUG:boto:Data: 2023-05-31T20:27:15.483 DEBUG:boto:Headers: {} 2023-05-31T20:27:15.484 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:27:15.484 DEBUG:boto:Port: 8001 2023-05-31T20:27:15.485 DEBUG:boto:Params: {} 2023-05-31T20:27:15.485 DEBUG:boto:Token: None 2023-05-31T20:27:15.486 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:27:15 GMT /pklxkd-54/d 2023-05-31T20:27:15.486 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:4p6inQ6W5OpQ9rNoz1X+fF1oi+o= 2023-05-31T20:27:15.486 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:27:15 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:4p6inQ6W5OpQ9rNoz1X+fF1oi+o=', 'Content-Length': '0'} 2023-05-31T20:27:15.488 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:26:44 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000406b8da128919c36-006477ada3-162e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:27:15 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:27:15.489 DEBUG:boto:path=/pklxkd-54/d 2023-05-31T20:27:15.489 DEBUG:boto:auth_path=/pklxkd-54/d 2023-05-31T20:27:15.490 DEBUG:boto:Method: GET 2023-05-31T20:27:15.490 DEBUG:boto:Path: /pklxkd-54/d 2023-05-31T20:27:15.491 DEBUG:boto:Data: 2023-05-31T20:27:15.491 DEBUG:boto:Headers: {} 2023-05-31T20:27:15.491 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:27:15.492 DEBUG:boto:Port: 8000 2023-05-31T20:27:15.492 DEBUG:boto:Params: {} 2023-05-31T20:27:15.493 DEBUG:boto:Token: None 2023-05-31T20:27:15.493 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:27:15 GMT /pklxkd-54/d 2023-05-31T20:27:15.493 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:4p6inQ6W5OpQ9rNoz1X+fF1oi+o= 2023-05-31T20:27:15.494 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:27:15 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:4p6inQ6W5OpQ9rNoz1X+fF1oi+o=', 'Content-Length': '0'} 2023-05-31T20:27:15.496 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:26:44 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000bedcd301abd1d58b-006477ada3-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:27:15 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:27:15.496 DEBUG:boto:path=/pklxkd-54/ 2023-05-31T20:27:15.497 DEBUG:boto:auth_path=/pklxkd-54/ 2023-05-31T20:27:15.497 DEBUG:boto:path=/pklxkd-54/?prefix=d 2023-05-31T20:27:15.497 DEBUG:boto:auth_path=/pklxkd-54/?prefix=d 2023-05-31T20:27:15.498 DEBUG:boto:Method: GET 2023-05-31T20:27:15.498 DEBUG:boto:Path: /pklxkd-54/?prefix=d 2023-05-31T20:27:15.499 DEBUG:boto:Data: 2023-05-31T20:27:15.499 DEBUG:boto:Headers: {} 2023-05-31T20:27:15.499 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:27:15.500 DEBUG:boto:Port: 8001 2023-05-31T20:27:15.500 DEBUG:boto:Params: {} 2023-05-31T20:27:15.500 DEBUG:boto:Token: None 2023-05-31T20:27:15.501 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:27:15 GMT /pklxkd-54/ 2023-05-31T20:27:15.501 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:nQrx37OAwoLFO+xHyv742191gHM= 2023-05-31T20:27:15.502 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:27:15 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:nQrx37OAwoLFO+xHyv742191gHM=', 'Content-Length': '0'} 2023-05-31T20:27:15.504 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000005d129f586eedbf25-006477ada3-162e-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:27:15 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:27:15.548 DEBUG:boto:b'pklxkd-54d1000falsed2023-05-31T20:26:44.140Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:27:15.549 DEBUG:boto:path=/pklxkd-54/ 2023-05-31T20:27:15.549 DEBUG:boto:auth_path=/pklxkd-54/ 2023-05-31T20:27:15.550 DEBUG:boto:path=/pklxkd-54/?prefix=d 2023-05-31T20:27:15.550 DEBUG:boto:auth_path=/pklxkd-54/?prefix=d 2023-05-31T20:27:15.551 DEBUG:boto:Method: GET 2023-05-31T20:27:15.551 DEBUG:boto:Path: /pklxkd-54/?prefix=d 2023-05-31T20:27:15.551 DEBUG:boto:Data: 2023-05-31T20:27:15.552 DEBUG:boto:Headers: {} 2023-05-31T20:27:15.552 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:27:15.553 DEBUG:boto:Port: 8000 2023-05-31T20:27:15.553 DEBUG:boto:Params: {} 2023-05-31T20:27:15.553 DEBUG:boto:Token: None 2023-05-31T20:27:15.554 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:27:15 GMT /pklxkd-54/ 2023-05-31T20:27:15.554 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:nQrx37OAwoLFO+xHyv742191gHM= 2023-05-31T20:27:15.555 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:27:15 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:nQrx37OAwoLFO+xHyv742191gHM=', 'Content-Length': '0'} 2023-05-31T20:27:15.557 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c1dbcba882c36f8b-006477ada3-1630-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:27:15 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:27:15.558 DEBUG:boto:b'pklxkd-54d1000falsed2023-05-31T20:26:44.140Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:27:15.558 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=pklxkd-54 zones={a2, a1} 2023-05-31T20:27:15.559 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'autotrim', '--rgw-sync-log-trim-max-buckets', '1', '--rgw-sync-log-trim-min-cold-buckets', '0', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-05-31T20:27:15.559 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'autotrim', '--rgw-sync-log-trim-max-buckets', '1', '--rgw-sync-log-trim-min-cold-buckets', '0', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-05-31T20:27:15.560 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog autotrim --rgw-sync-log-trim-max-buckets 1 --rgw-sync-log-trim-min-cold-buckets 0 --cluster c1 --debug-rgw 0 --debug-ms 0 2023-05-31T20:27:15.615 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2023-05-31T20:27:15.615 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2023-05-31T20:27:15.681 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'list', '--bucket', 'pklxkd-54', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-31T20:27:15.681 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'list', '--bucket', 'pklxkd-54', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-31T20:27:15.682 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog list --bucket pklxkd-54 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-05-31T20:27:15.730 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2023-05-31T20:27:15.730 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2023-05-31T20:27:15.787 INFO:teuthology.orchestra.run.smithi099.stdout:[] 2023-05-31T20:27:15.789 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'list', '--bucket', 'pklxkd-53', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-31T20:27:15.790 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'list', '--bucket', 'pklxkd-53', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-31T20:27:15.790 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog list --bucket pklxkd-53 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-05-31T20:27:15.837 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2023-05-31T20:27:15.838 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2023-05-31T20:27:15.900 INFO:teuthology.orchestra.run.smithi099.stdout:[ 2023-05-31T20:27:15.900 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T20:27:15.900 INFO:teuthology.orchestra.run.smithi099.stdout: "op_id": "1#00000000001.24.6", 2023-05-31T20:27:15.901 INFO:teuthology.orchestra.run.smithi099.stdout: "op_tag": "3d163b1e-09c7-47e2-ae87-add4aace3f38.5680.16587181007471760428", 2023-05-31T20:27:15.901 INFO:teuthology.orchestra.run.smithi099.stdout: "op": "write", 2023-05-31T20:27:15.901 INFO:teuthology.orchestra.run.smithi099.stdout: "object": "a", 2023-05-31T20:27:15.901 INFO:teuthology.orchestra.run.smithi099.stdout: "instance": "", 2023-05-31T20:27:15.901 INFO:teuthology.orchestra.run.smithi099.stdout: "state": "complete", 2023-05-31T20:27:15.901 INFO:teuthology.orchestra.run.smithi099.stdout: "index_ver": 1, 2023-05-31T20:27:15.902 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2023-05-31T20:26:27.510431020Z", 2023-05-31T20:27:15.902 INFO:teuthology.orchestra.run.smithi099.stdout: "ver": { 2023-05-31T20:27:15.902 INFO:teuthology.orchestra.run.smithi099.stdout: "pool": 9, 2023-05-31T20:27:15.902 INFO:teuthology.orchestra.run.smithi099.stdout: "epoch": 3 2023-05-31T20:27:15.902 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:27:15.902 INFO:teuthology.orchestra.run.smithi099.stdout: "bilog_flags": 0, 2023-05-31T20:27:15.902 INFO:teuthology.orchestra.run.smithi099.stdout: "versioned": false, 2023-05-31T20:27:15.903 INFO:teuthology.orchestra.run.smithi099.stdout: "owner": "", 2023-05-31T20:27:15.903 INFO:teuthology.orchestra.run.smithi099.stdout: "owner_display_name": "", 2023-05-31T20:27:15.903 INFO:teuthology.orchestra.run.smithi099.stdout: "zones_trace": [ 2023-05-31T20:27:15.903 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T20:27:15.903 INFO:teuthology.orchestra.run.smithi099.stdout: "entry": "3d163b1e-09c7-47e2-ae87-add4aace3f38:pklxkd-53:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.18" 2023-05-31T20:27:15.903 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T20:27:15.903 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2023-05-31T20:27:15.904 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:27:15.904 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T20:27:15.904 INFO:teuthology.orchestra.run.smithi099.stdout: "op_id": "5#00000000001.35.6", 2023-05-31T20:27:15.904 INFO:teuthology.orchestra.run.smithi099.stdout: "op_tag": "3d163b1e-09c7-47e2-ae87-add4aace3f38.5680.3930164732213624035", 2023-05-31T20:27:15.904 INFO:teuthology.orchestra.run.smithi099.stdout: "op": "write", 2023-05-31T20:27:15.904 INFO:teuthology.orchestra.run.smithi099.stdout: "object": "d", 2023-05-31T20:27:15.905 INFO:teuthology.orchestra.run.smithi099.stdout: "instance": "", 2023-05-31T20:27:15.905 INFO:teuthology.orchestra.run.smithi099.stdout: "state": "complete", 2023-05-31T20:27:15.905 INFO:teuthology.orchestra.run.smithi099.stdout: "index_ver": 1, 2023-05-31T20:27:15.905 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2023-05-31T20:26:27.602577945Z", 2023-05-31T20:27:15.905 INFO:teuthology.orchestra.run.smithi099.stdout: "ver": { 2023-05-31T20:27:15.905 INFO:teuthology.orchestra.run.smithi099.stdout: "pool": 9, 2023-05-31T20:27:15.905 INFO:teuthology.orchestra.run.smithi099.stdout: "epoch": 9 2023-05-31T20:27:15.906 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:27:15.906 INFO:teuthology.orchestra.run.smithi099.stdout: "bilog_flags": 0, 2023-05-31T20:27:15.906 INFO:teuthology.orchestra.run.smithi099.stdout: "versioned": false, 2023-05-31T20:27:15.906 INFO:teuthology.orchestra.run.smithi099.stdout: "owner": "", 2023-05-31T20:27:15.906 INFO:teuthology.orchestra.run.smithi099.stdout: "owner_display_name": "", 2023-05-31T20:27:15.906 INFO:teuthology.orchestra.run.smithi099.stdout: "zones_trace": [ 2023-05-31T20:27:15.906 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T20:27:15.907 INFO:teuthology.orchestra.run.smithi099.stdout: "entry": "3d163b1e-09c7-47e2-ae87-add4aace3f38:pklxkd-53:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.18" 2023-05-31T20:27:15.907 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T20:27:15.907 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2023-05-31T20:27:15.907 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:27:15.907 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T20:27:15.907 INFO:teuthology.orchestra.run.smithi099.stdout: "op_id": "8#00000000001.21.6", 2023-05-31T20:27:15.907 INFO:teuthology.orchestra.run.smithi099.stdout: "op_tag": "3d163b1e-09c7-47e2-ae87-add4aace3f38.5680.9879972791500402494", 2023-05-31T20:27:15.908 INFO:teuthology.orchestra.run.smithi099.stdout: "op": "write", 2023-05-31T20:27:15.908 INFO:teuthology.orchestra.run.smithi099.stdout: "object": "c", 2023-05-31T20:27:15.908 INFO:teuthology.orchestra.run.smithi099.stdout: "instance": "", 2023-05-31T20:27:15.908 INFO:teuthology.orchestra.run.smithi099.stdout: "state": "complete", 2023-05-31T20:27:15.908 INFO:teuthology.orchestra.run.smithi099.stdout: "index_ver": 1, 2023-05-31T20:27:15.908 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2023-05-31T20:26:27.548111974Z", 2023-05-31T20:27:15.909 INFO:teuthology.orchestra.run.smithi099.stdout: "ver": { 2023-05-31T20:27:15.909 INFO:teuthology.orchestra.run.smithi099.stdout: "pool": 9, 2023-05-31T20:27:15.909 INFO:teuthology.orchestra.run.smithi099.stdout: "epoch": 1 2023-05-31T20:27:15.909 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:27:15.909 INFO:teuthology.orchestra.run.smithi099.stdout: "bilog_flags": 0, 2023-05-31T20:27:15.909 INFO:teuthology.orchestra.run.smithi099.stdout: "versioned": false, 2023-05-31T20:27:15.909 INFO:teuthology.orchestra.run.smithi099.stdout: "owner": "", 2023-05-31T20:27:15.910 INFO:teuthology.orchestra.run.smithi099.stdout: "owner_display_name": "", 2023-05-31T20:27:15.910 INFO:teuthology.orchestra.run.smithi099.stdout: "zones_trace": [ 2023-05-31T20:27:15.910 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T20:27:15.910 INFO:teuthology.orchestra.run.smithi099.stdout: "entry": "3d163b1e-09c7-47e2-ae87-add4aace3f38:pklxkd-53:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.18" 2023-05-31T20:27:15.910 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T20:27:15.910 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2023-05-31T20:27:15.910 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:27:15.911 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T20:27:15.911 INFO:teuthology.orchestra.run.smithi099.stdout: "op_id": "10#00000000001.26.6", 2023-05-31T20:27:15.911 INFO:teuthology.orchestra.run.smithi099.stdout: "op_tag": "3d163b1e-09c7-47e2-ae87-add4aace3f38.5680.6498859579596434025", 2023-05-31T20:27:15.911 INFO:teuthology.orchestra.run.smithi099.stdout: "op": "write", 2023-05-31T20:27:15.911 INFO:teuthology.orchestra.run.smithi099.stdout: "object": "b", 2023-05-31T20:27:15.911 INFO:teuthology.orchestra.run.smithi099.stdout: "instance": "", 2023-05-31T20:27:15.911 INFO:teuthology.orchestra.run.smithi099.stdout: "state": "complete", 2023-05-31T20:27:15.912 INFO:teuthology.orchestra.run.smithi099.stdout: "index_ver": 1, 2023-05-31T20:27:15.912 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2023-05-31T20:26:27.529896878Z", 2023-05-31T20:27:15.912 INFO:teuthology.orchestra.run.smithi099.stdout: "ver": { 2023-05-31T20:27:15.912 INFO:teuthology.orchestra.run.smithi099.stdout: "pool": 9, 2023-05-31T20:27:15.912 INFO:teuthology.orchestra.run.smithi099.stdout: "epoch": 7 2023-05-31T20:27:15.912 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:27:15.912 INFO:teuthology.orchestra.run.smithi099.stdout: "bilog_flags": 0, 2023-05-31T20:27:15.913 INFO:teuthology.orchestra.run.smithi099.stdout: "versioned": false, 2023-05-31T20:27:15.913 INFO:teuthology.orchestra.run.smithi099.stdout: "owner": "", 2023-05-31T20:27:15.913 INFO:teuthology.orchestra.run.smithi099.stdout: "owner_display_name": "", 2023-05-31T20:27:15.913 INFO:teuthology.orchestra.run.smithi099.stdout: "zones_trace": [ 2023-05-31T20:27:15.913 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T20:27:15.913 INFO:teuthology.orchestra.run.smithi099.stdout: "entry": "3d163b1e-09c7-47e2-ae87-add4aace3f38:pklxkd-53:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.18" 2023-05-31T20:27:15.913 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T20:27:15.914 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2023-05-31T20:27:15.914 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T20:27:15.914 INFO:teuthology.orchestra.run.smithi099.stdout:] 2023-05-31T20:27:15.915 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'autotrim', '--rgw-sync-log-trim-max-buckets', '999', '--rgw-sync-log-trim-min-cold-buckets', '999', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-05-31T20:27:15.915 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'autotrim', '--rgw-sync-log-trim-max-buckets', '999', '--rgw-sync-log-trim-min-cold-buckets', '999', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-05-31T20:27:15.916 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog autotrim --rgw-sync-log-trim-max-buckets 999 --rgw-sync-log-trim-min-cold-buckets 999 --cluster c1 --debug-rgw 0 --debug-ms 0 2023-05-31T20:27:15.962 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2023-05-31T20:27:15.963 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2023-05-31T20:27:16.030 INFO:tasks.rgw.c1.client.1.smithi099.stdout:2023-05-31T20:27:16.024+0000 7f3abd8b6700 -1 req 3453176611877135109 0.000999976s :get_bucket_index_log_status ERROR: rgw_read_bucket_full_sync_status() on pipe={s={b=pklxkd-43:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.8,z=3d163b1e-09c7-47e2-ae87-add4aace3f38,az=0},d={b=pklxkd-43:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.8,z=5d8acec7-7e57-4e56-8b32-e3d6b5036001,az=0}} returned ret=-2 2023-05-31T20:27:16.031 INFO:tasks.rgw.c1.client.1.smithi099.stdout:2023-05-31T20:27:16.025+0000 7f3aae898700 -1 req 11729604870742969435 0.001999953s :get_bucket_index_log_status ERROR: rgw_read_bucket_full_sync_status() on pipe={s={b=pklxkd-19:3d163b1e-09c7-47e2-ae87-add4aace3f38.4368.19,z=3d163b1e-09c7-47e2-ae87-add4aace3f38,az=0},d={b=pklxkd-19:3d163b1e-09c7-47e2-ae87-add4aace3f38.4368.19,z=5d8acec7-7e57-4e56-8b32-e3d6b5036001,az=0}} returned ret=-2 2023-05-31T20:27:16.031 INFO:tasks.rgw.c1.client.1.smithi099.stdout:2023-05-31T20:27:16.025+0000 7f3ac70c9700 -1 req 1116871707097087917 0.001999953s :get_bucket_index_log_status ERROR: rgw_read_bucket_full_sync_status() on pipe={s={b=pklxkd-6:3d163b1e-09c7-47e2-ae87-add4aace3f38.4368.6,z=3d163b1e-09c7-47e2-ae87-add4aace3f38,az=0},d={b=pklxkd-6:3d163b1e-09c7-47e2-ae87-add4aace3f38.4368.6,z=5d8acec7-7e57-4e56-8b32-e3d6b5036001,az=0}} returned ret=-2 2023-05-31T20:27:16.035 INFO:tasks.rgw.c1.client.1.smithi099.stdout:2023-05-31T20:27:16.029+0000 7f3a8003b700 -1 req 11528109887574195244 0.001999953s :get_bucket_index_log_status ERROR: rgw_read_bucket_full_sync_status() on pipe={s={b=pklxkd-36:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.1,z=3d163b1e-09c7-47e2-ae87-add4aace3f38,az=0},d={b=pklxkd-36:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.1,z=5d8acec7-7e57-4e56-8b32-e3d6b5036001,az=0}} returned ret=-2 2023-05-31T20:27:16.035 INFO:tasks.rgw.c1.client.1.smithi099.stdout:2023-05-31T20:27:16.029+0000 7f3a8003b700 -1 req 8526306942643765092 0.000999976s :get_bucket_index_log_status ERROR: rgw_read_bucket_full_sync_status() on pipe={s={b=pklxkd-45:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.10,z=3d163b1e-09c7-47e2-ae87-add4aace3f38,az=0},d={b=pklxkd-45:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.10,z=5d8acec7-7e57-4e56-8b32-e3d6b5036001,az=0}} returned ret=-2 2023-05-31T20:27:16.078 INFO:tasks.rgw.c1.client.1.smithi099.stdout:2023-05-31T20:27:16.073+0000 7f3b6da16700 -1 req 4919900347266973246 0.001999954s :get_bucket_index_log_status ERROR: rgw_read_bucket_full_sync_status() on pipe={s={b=pklxkd-3:3d163b1e-09c7-47e2-ae87-add4aace3f38.4368.3,z=3d163b1e-09c7-47e2-ae87-add4aace3f38,az=0},d={b=pklxkd-3:3d163b1e-09c7-47e2-ae87-add4aace3f38.4368.3,z=5d8acec7-7e57-4e56-8b32-e3d6b5036001,az=0}} returned ret=-2 2023-05-31T20:27:16.079 INFO:tasks.rgw.c1.client.1.smithi099.stdout:2023-05-31T20:27:16.073+0000 7f3b6a20f700 -1 req 17210799172395292411 0.001999954s :get_bucket_index_log_status ERROR: rgw_read_bucket_full_sync_status() on pipe={s={b=pklxkd-1:3d163b1e-09c7-47e2-ae87-add4aace3f38.4368.1,z=3d163b1e-09c7-47e2-ae87-add4aace3f38,az=0},d={b=pklxkd-1:3d163b1e-09c7-47e2-ae87-add4aace3f38.4368.1,z=5d8acec7-7e57-4e56-8b32-e3d6b5036001,az=0}} returned ret=-2 2023-05-31T20:27:16.122 INFO:tasks.rgw.c1.client.1.smithi099.stdout:2023-05-31T20:27:16.116+0000 7f3b64203700 -1 req 3104745481554230888 0.000999977s :get_bucket_index_log_status ERROR: rgw_read_bucket_full_sync_status() on pipe={s={b=pklxkd-37:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.2,z=3d163b1e-09c7-47e2-ae87-add4aace3f38,az=0},d={b=pklxkd-37:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.2,z=5d8acec7-7e57-4e56-8b32-e3d6b5036001,az=0}} returned ret=-2 2023-05-31T20:27:16.129 INFO:tasks.rgw.c1.client.1.smithi099.stdout:2023-05-31T20:27:16.124+0000 7f3b589ec700 -1 req 13986075943662778692 0.001999954s :get_bucket_index_log_status ERROR: rgw_read_bucket_full_sync_status() on pipe={s={b=pklxkd-2:3d163b1e-09c7-47e2-ae87-add4aace3f38.4368.2,z=3d163b1e-09c7-47e2-ae87-add4aace3f38,az=0},d={b=pklxkd-2:3d163b1e-09c7-47e2-ae87-add4aace3f38.4368.2,z=5d8acec7-7e57-4e56-8b32-e3d6b5036001,az=0}} returned ret=-2 2023-05-31T20:27:16.165 INFO:tasks.rgw.c1.client.1.smithi099.stdout:2023-05-31T20:27:16.160+0000 7f3b409bc700 -1 req 484950923166673120 0.000999977s :get_bucket_index_log_status ERROR: rgw_read_bucket_full_sync_status() on pipe={s={b=pklxkd-4:3d163b1e-09c7-47e2-ae87-add4aace3f38.4368.4,z=3d163b1e-09c7-47e2-ae87-add4aace3f38,az=0},d={b=pklxkd-4:3d163b1e-09c7-47e2-ae87-add4aace3f38.4368.4,z=5d8acec7-7e57-4e56-8b32-e3d6b5036001,az=0}} returned ret=-2 2023-05-31T20:27:16.173 INFO:tasks.rgw.c1.client.1.smithi099.stdout:2023-05-31T20:27:16.168+0000 7f3b3c1b3700 -1 req 13034923923785718259 0.001999953s :get_bucket_index_log_status ERROR: rgw_read_bucket_full_sync_status() on pipe={s={b=pklxkd-28:3d163b1e-09c7-47e2-ae87-add4aace3f38.4368.28,z=3d163b1e-09c7-47e2-ae87-add4aace3f38,az=0},d={b=pklxkd-28:3d163b1e-09c7-47e2-ae87-add4aace3f38.4368.28,z=5d8acec7-7e57-4e56-8b32-e3d6b5036001,az=0}} returned ret=-2 2023-05-31T20:27:16.178 INFO:tasks.rgw.c1.client.1.smithi099.stdout:2023-05-31T20:27:16.172+0000 7f3b491cd700 -1 req 8984913154891577982 0.001999953s :get_bucket_index_log_status ERROR: rgw_read_bucket_full_sync_status() on pipe={s={b=pklxkd-5:3d163b1e-09c7-47e2-ae87-add4aace3f38.4368.5,z=3d163b1e-09c7-47e2-ae87-add4aace3f38,az=0},d={b=pklxkd-5:3d163b1e-09c7-47e2-ae87-add4aace3f38.4368.5,z=5d8acec7-7e57-4e56-8b32-e3d6b5036001,az=0}} returned ret=-2 2023-05-31T20:27:16.217 INFO:tasks.rgw.c1.client.1.smithi099.stdout:2023-05-31T20:27:16.211+0000 7f3b2b191700 -1 req 16139778180532911310 0.000999977s :get_bucket_index_log_status ERROR: rgw_read_bucket_full_sync_status() on pipe={s={b=pklxkd-40:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.5,z=3d163b1e-09c7-47e2-ae87-add4aace3f38,az=0},d={b=pklxkd-40:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.5,z=5d8acec7-7e57-4e56-8b32-e3d6b5036001,az=0}} returned ret=-2 2023-05-31T20:27:16.257 INFO:tasks.rgw.c1.client.1.smithi099.stdout:2023-05-31T20:27:16.252+0000 7f3b1c974700 -1 req 8699507123419720242 0.001999953s :get_bucket_index_log_status ERROR: rgw_read_bucket_full_sync_status() on pipe={s={b=pklxkd-29:3d163b1e-09c7-47e2-ae87-add4aace3f38.4368.29,z=3d163b1e-09c7-47e2-ae87-add4aace3f38,az=0},d={b=pklxkd-29:3d163b1e-09c7-47e2-ae87-add4aace3f38.4368.29,z=5d8acec7-7e57-4e56-8b32-e3d6b5036001,az=0}} returned ret=-2 2023-05-31T20:27:16.261 INFO:tasks.rgw.c1.client.1.smithi099.stdout:2023-05-31T20:27:16.255+0000 7f3b4f1d9700 -1 req 3055964605031259256 0.001999953s :get_bucket_index_log_status ERROR: rgw_read_bucket_full_sync_status() on pipe={s={b=pklxkd-30:3d163b1e-09c7-47e2-ae87-add4aace3f38.4368.30,z=3d163b1e-09c7-47e2-ae87-add4aace3f38,az=0},d={b=pklxkd-30:3d163b1e-09c7-47e2-ae87-add4aace3f38.4368.30,z=5d8acec7-7e57-4e56-8b32-e3d6b5036001,az=0}} returned ret=-2 2023-05-31T20:27:16.271 INFO:tasks.rgw.c1.client.1.smithi099.stdout:2023-05-31T20:27:16.265+0000 7f3b0f159700 -1 req 10186422718935114999 0.000999977s :get_bucket_index_log_status ERROR: rgw_read_bucket_full_sync_status() on pipe={s={b=pklxkd-51:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.16,z=3d163b1e-09c7-47e2-ae87-add4aace3f38,az=0},d={b=pklxkd-51:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.16,z=5d8acec7-7e57-4e56-8b32-e3d6b5036001,az=0}} returned ret=-2 2023-05-31T20:27:16.320 INFO:tasks.rgw.c1.client.1.smithi099.stdout:2023-05-31T20:27:16.315+0000 7f3ae4904700 -1 req 16129648296340462214 0.000999976s :get_bucket_index_log_status ERROR: rgw_read_bucket_full_sync_status() on pipe={s={b=pklxkd-41:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.6,z=3d163b1e-09c7-47e2-ae87-add4aace3f38,az=0},d={b=pklxkd-41:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.6,z=5d8acec7-7e57-4e56-8b32-e3d6b5036001,az=0}} returned ret=-2 2023-05-31T20:27:16.352 INFO:tasks.rgw.c1.client.1.smithi099.stdout:2023-05-31T20:27:16.346+0000 7f3aed916700 -1 req 6794675827822391834 0.000999977s :get_bucket_index_log_status ERROR: rgw_read_bucket_full_sync_status() on pipe={s={b=pklxkd-22:3d163b1e-09c7-47e2-ae87-add4aace3f38.4368.22,z=3d163b1e-09c7-47e2-ae87-add4aace3f38,az=0},d={b=pklxkd-22:3d163b1e-09c7-47e2-ae87-add4aace3f38.4368.22,z=5d8acec7-7e57-4e56-8b32-e3d6b5036001,az=0}} returned ret=-2 2023-05-31T20:27:16.352 INFO:tasks.rgw.c1.client.1.smithi099.stdout:2023-05-31T20:27:16.346+0000 7f3aef91a700 -1 req 3834070495077343687 0.000999977s :get_bucket_index_log_status ERROR: rgw_read_bucket_full_sync_status() on pipe={s={b=pklxkd-20:3d163b1e-09c7-47e2-ae87-add4aace3f38.4368.20,z=3d163b1e-09c7-47e2-ae87-add4aace3f38,az=0},d={b=pklxkd-20:3d163b1e-09c7-47e2-ae87-add4aace3f38.4368.20,z=5d8acec7-7e57-4e56-8b32-e3d6b5036001,az=0}} returned ret=-2 2023-05-31T20:27:16.361 INFO:tasks.rgw.c1.client.1.smithi099.stdout:2023-05-31T20:27:16.355+0000 7f3ae890c700 -1 req 17659119856060324124 0.000999976s :get_bucket_index_log_status ERROR: rgw_read_bucket_full_sync_status() on pipe={s={b=pklxkd-44:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.9,z=3d163b1e-09c7-47e2-ae87-add4aace3f38,az=0},d={b=pklxkd-44:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.9,z=5d8acec7-7e57-4e56-8b32-e3d6b5036001,az=0}} returned ret=-2 2023-05-31T20:27:16.395 INFO:tasks.rgw.c1.client.1.smithi099.stdout:2023-05-31T20:27:16.389+0000 7f3b04944700 -1 req 10845246651949705405 0.000999977s :get_bucket_index_log_status ERROR: rgw_read_bucket_full_sync_status() on pipe={s={b=pklxkd-42:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.7,z=3d163b1e-09c7-47e2-ae87-add4aace3f38,az=0},d={b=pklxkd-42:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.7,z=5d8acec7-7e57-4e56-8b32-e3d6b5036001,az=0}} returned ret=-2 2023-05-31T20:27:16.395 INFO:tasks.rgw.c1.client.1.smithi099.stdout:2023-05-31T20:27:16.389+0000 7f3ad00db700 -1 req 14691332081347253795 0.000999977s :get_bucket_index_log_status ERROR: rgw_read_bucket_full_sync_status() on pipe={s={b=pklxkd-24:3d163b1e-09c7-47e2-ae87-add4aace3f38.4368.24,z=3d163b1e-09c7-47e2-ae87-add4aace3f38,az=0},d={b=pklxkd-24:3d163b1e-09c7-47e2-ae87-add4aace3f38.4368.24,z=5d8acec7-7e57-4e56-8b32-e3d6b5036001,az=0}} returned ret=-2 2023-05-31T20:27:16.437 INFO:tasks.rgw.c1.client.1.smithi099.stdout:2023-05-31T20:27:16.432+0000 7f3ad30e1700 -1 req 12575930472108703150 0.000999976s :get_bucket_index_log_status ERROR: rgw_read_bucket_full_sync_status() on pipe={s={b=pklxkd-39:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.4,z=3d163b1e-09c7-47e2-ae87-add4aace3f38,az=0},d={b=pklxkd-39:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.4,z=5d8acec7-7e57-4e56-8b32-e3d6b5036001,az=0}} returned ret=-2 2023-05-31T20:27:16.457 INFO:tasks.rgw.c1.client.1.smithi099.stdout:2023-05-31T20:27:16.452+0000 7f3acb8d2700 -1 req 14998201102085404674 0.001999953s :get_bucket_index_log_status ERROR: rgw_read_bucket_full_sync_status() on pipe={s={b=pklxkd-38:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.3,z=3d163b1e-09c7-47e2-ae87-add4aace3f38,az=0},d={b=pklxkd-38:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.3,z=5d8acec7-7e57-4e56-8b32-e3d6b5036001,az=0}} returned ret=-2 2023-05-31T20:27:16.481 INFO:tasks.rgw.c1.client.1.smithi099.stdout:2023-05-31T20:27:16.476+0000 7f3aa2880700 -1 req 113482211147725811 0.001999954s :get_bucket_index_log_status ERROR: rgw_read_bucket_full_sync_status() on pipe={s={b=pklxkd-21:3d163b1e-09c7-47e2-ae87-add4aace3f38.4368.21,z=3d163b1e-09c7-47e2-ae87-add4aace3f38,az=0},d={b=pklxkd-21:3d163b1e-09c7-47e2-ae87-add4aace3f38.4368.21,z=5d8acec7-7e57-4e56-8b32-e3d6b5036001,az=0}} returned ret=-2 2023-05-31T20:27:16.498 INFO:tasks.rgw.c1.client.1.smithi099.stdout:2023-05-31T20:27:16.492+0000 7f3ac98ce700 -1 req 6589385607905304066 0.000999977s :get_bucket_index_log_status ERROR: rgw_read_bucket_full_sync_status() on pipe={s={b=pklxkd-23:3d163b1e-09c7-47e2-ae87-add4aace3f38.4368.23,z=3d163b1e-09c7-47e2-ae87-add4aace3f38,az=0},d={b=pklxkd-23:3d163b1e-09c7-47e2-ae87-add4aace3f38.4368.23,z=5d8acec7-7e57-4e56-8b32-e3d6b5036001,az=0}} returned ret=-2 2023-05-31T20:27:16.501 INFO:tasks.rgw.c1.client.1.smithi099.stdout:2023-05-31T20:27:16.496+0000 7f3a8f059700 -1 req 9415973724574669012 0.000999976s :get_bucket_index_log_status ERROR: rgw_read_bucket_full_sync_status() on pipe={s={b=pklxkd-25:3d163b1e-09c7-47e2-ae87-add4aace3f38.4368.25,z=3d163b1e-09c7-47e2-ae87-add4aace3f38,az=0},d={b=pklxkd-25:3d163b1e-09c7-47e2-ae87-add4aace3f38.4368.25,z=5d8acec7-7e57-4e56-8b32-e3d6b5036001,az=0}} returned ret=-2 2023-05-31T20:27:16.534 INFO:tasks.rgw.c1.client.1.smithi099.stdout:2023-05-31T20:27:16.524+0000 7f3a93061700 -1 req 11660160671844955951 0.001999954s :get_bucket_index_log_status ERROR: rgw_read_bucket_full_sync_status() on pipe={s={b=pklxkd-26:3d163b1e-09c7-47e2-ae87-add4aace3f38.4368.26,z=3d163b1e-09c7-47e2-ae87-add4aace3f38,az=0},d={b=pklxkd-26:3d163b1e-09c7-47e2-ae87-add4aace3f38.4368.26,z=5d8acec7-7e57-4e56-8b32-e3d6b5036001,az=0}} returned ret=-2 2023-05-31T20:27:16.545 INFO:tasks.rgw.c1.client.1.smithi099.stdout:2023-05-31T20:27:16.540+0000 7f3a8c854700 -1 req 7585909194863431183 0.000999976s :get_bucket_index_log_status ERROR: rgw_read_bucket_full_sync_status() on pipe={s={b=pklxkd-27:3d163b1e-09c7-47e2-ae87-add4aace3f38.4368.27,z=3d163b1e-09c7-47e2-ae87-add4aace3f38,az=0},d={b=pklxkd-27:3d163b1e-09c7-47e2-ae87-add4aace3f38.4368.27,z=5d8acec7-7e57-4e56-8b32-e3d6b5036001,az=0}} returned ret=-2 2023-05-31T20:27:16.648 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'list', '--bucket', 'pklxkd-53', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-31T20:27:16.648 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'list', '--bucket', 'pklxkd-53', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-31T20:27:16.649 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog list --bucket pklxkd-53 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-05-31T20:27:16.697 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2023-05-31T20:27:16.697 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2023-05-31T20:27:16.752 INFO:teuthology.orchestra.run.smithi099.stdout:[] 2023-05-31T20:27:16.754 INFO:tasks.rgw_multisite_tests:rgw_multi.tests.test_bucket_index_log_trim ... ok 2023-05-31T20:27:16.756 INFO:rgw_multi.tests:create bucket zone=a1 name=pklxkd-55 2023-05-31T20:27:16.756 DEBUG:boto:path=/pklxkd-55/ 2023-05-31T20:27:16.757 DEBUG:boto:auth_path=/pklxkd-55/ 2023-05-31T20:27:16.757 DEBUG:boto:Method: PUT 2023-05-31T20:27:16.757 DEBUG:boto:Path: /pklxkd-55/ 2023-05-31T20:27:16.758 DEBUG:boto:Data: 2023-05-31T20:27:16.758 DEBUG:boto:Headers: {} 2023-05-31T20:27:16.758 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:27:16.758 DEBUG:boto:Port: 8000 2023-05-31T20:27:16.759 DEBUG:boto:Params: {} 2023-05-31T20:27:16.759 DEBUG:boto:Token: None 2023-05-31T20:27:16.760 DEBUG:boto:StringToSign: PUT Wed, 31 May 2023 20:27:16 GMT /pklxkd-55/ 2023-05-31T20:27:16.760 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:hPHfvLf6VvV0UWrup7p4tutMd7A= 2023-05-31T20:27:16.760 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:27:16 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:hPHfvLf6VvV0UWrup7p4tutMd7A=', 'Content-Length': '0'} 2023-05-31T20:27:16.780 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx0000033a9144774412b74-006477ada4-1630-a1'), ('Content-Length', '0'), ('Date', 'Wed, 31 May 2023 20:27:16 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:27:16.781 DEBUG:boto:path=/pklxkd-55/ 2023-05-31T20:27:16.782 DEBUG:boto:auth_path=/pklxkd-55/ 2023-05-31T20:27:16.782 DEBUG:boto:Method: HEAD 2023-05-31T20:27:16.782 DEBUG:boto:Path: /pklxkd-55/ 2023-05-31T20:27:16.783 DEBUG:boto:Data: 2023-05-31T20:27:16.783 DEBUG:boto:Headers: {} 2023-05-31T20:27:16.783 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:27:16.784 DEBUG:boto:Port: 8000 2023-05-31T20:27:16.784 DEBUG:boto:Params: {} 2023-05-31T20:27:16.784 DEBUG:boto:Token: None 2023-05-31T20:27:16.785 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:27:16 GMT /pklxkd-55/ 2023-05-31T20:27:16.785 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:/gdTTojnB6Hl7PzQ8paUIVDSxl0= 2023-05-31T20:27:16.786 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:27:16 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:/gdTTojnB6Hl7PzQ8paUIVDSxl0=', 'Content-Length': '0'} 2023-05-31T20:27:16.788 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000c7721aa60aa2030d-006477ada4-1630-a1'), ('Content-Length', '0'), ('Date', 'Wed, 31 May 2023 20:27:16 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:27:16.789 DEBUG:boto:path=/pklxkd-55/a 2023-05-31T20:27:16.789 DEBUG:boto:auth_path=/pklxkd-55/a 2023-05-31T20:27:16.790 DEBUG:boto:Method: PUT 2023-05-31T20:27:16.790 DEBUG:boto:Path: /pklxkd-55/a 2023-05-31T20:27:16.790 DEBUG:boto:Data: 2023-05-31T20:27:16.791 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-05-31T20:27:16.791 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:27:16.791 DEBUG:boto:Port: 8000 2023-05-31T20:27:16.792 DEBUG:boto:Params: {} 2023-05-31T20:27:16.792 DEBUG:boto:Token: None 2023-05-31T20:27:16.793 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Wed, 31 May 2023 20:27:16 GMT /pklxkd-55/a 2023-05-31T20:27:16.793 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:Jd87CxZPf/jayeGlTBAOqgrkFT8= 2023-05-31T20:27:16.794 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Wed, 31 May 2023 20:27:16 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:Jd87CxZPf/jayeGlTBAOqgrkFT8='} 2023-05-31T20:27:16.838 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000000caa443b4a2238b1-006477ada4-1630-a1'), ('Date', 'Wed, 31 May 2023 20:27:16 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:27:16.839 DEBUG:boto:path=/pklxkd-55/ 2023-05-31T20:27:16.840 DEBUG:boto:auth_path=/pklxkd-55/ 2023-05-31T20:27:16.840 DEBUG:boto:Method: HEAD 2023-05-31T20:27:16.840 DEBUG:boto:Path: /pklxkd-55/ 2023-05-31T20:27:16.841 DEBUG:boto:Data: 2023-05-31T20:27:16.841 DEBUG:boto:Headers: {} 2023-05-31T20:27:16.841 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:27:16.842 DEBUG:boto:Port: 8000 2023-05-31T20:27:16.842 DEBUG:boto:Params: {} 2023-05-31T20:27:16.842 DEBUG:boto:Token: None 2023-05-31T20:27:16.843 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:27:16 GMT /pklxkd-55/ 2023-05-31T20:27:16.843 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:/gdTTojnB6Hl7PzQ8paUIVDSxl0= 2023-05-31T20:27:16.844 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:27:16 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:/gdTTojnB6Hl7PzQ8paUIVDSxl0=', 'Content-Length': '0'} 2023-05-31T20:27:16.846 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '3'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000002c12c4e858309536-006477ada4-1630-a1'), ('Content-Length', '0'), ('Date', 'Wed, 31 May 2023 20:27:16 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:27:16.846 DEBUG:boto:path=/pklxkd-55/b 2023-05-31T20:27:16.847 DEBUG:boto:auth_path=/pklxkd-55/b 2023-05-31T20:27:16.847 DEBUG:boto:Method: PUT 2023-05-31T20:27:16.847 DEBUG:boto:Path: /pklxkd-55/b 2023-05-31T20:27:16.848 DEBUG:boto:Data: 2023-05-31T20:27:16.849 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-05-31T20:27:16.849 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:27:16.850 DEBUG:boto:Port: 8000 2023-05-31T20:27:16.850 DEBUG:boto:Params: {} 2023-05-31T20:27:16.851 DEBUG:boto:Token: None 2023-05-31T20:27:16.851 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Wed, 31 May 2023 20:27:16 GMT /pklxkd-55/b 2023-05-31T20:27:16.851 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:YNnzYpVdPiyIPZf+ywza8hMU4f4= 2023-05-31T20:27:16.852 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Wed, 31 May 2023 20:27:16 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:YNnzYpVdPiyIPZf+ywza8hMU4f4='} 2023-05-31T20:27:16.859 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000004188f2160578862d-006477ada4-1630-a1'), ('Date', 'Wed, 31 May 2023 20:27:16 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:27:16.859 DEBUG:boto:path=/pklxkd-55/ 2023-05-31T20:27:16.859 DEBUG:boto:auth_path=/pklxkd-55/ 2023-05-31T20:27:16.860 DEBUG:boto:Method: HEAD 2023-05-31T20:27:16.860 DEBUG:boto:Path: /pklxkd-55/ 2023-05-31T20:27:16.861 DEBUG:boto:Data: 2023-05-31T20:27:16.861 DEBUG:boto:Headers: {} 2023-05-31T20:27:16.861 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:27:16.862 DEBUG:boto:Port: 8000 2023-05-31T20:27:16.862 DEBUG:boto:Params: {} 2023-05-31T20:27:16.863 DEBUG:boto:Token: None 2023-05-31T20:27:16.863 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:27:16 GMT /pklxkd-55/ 2023-05-31T20:27:16.864 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:/gdTTojnB6Hl7PzQ8paUIVDSxl0= 2023-05-31T20:27:16.864 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:27:16 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:/gdTTojnB6Hl7PzQ8paUIVDSxl0=', 'Content-Length': '0'} 2023-05-31T20:27:16.866 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '2'), ('X-RGW-Bytes-Used', '6'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000089987bce4d708c39-006477ada4-1630-a1'), ('Content-Length', '0'), ('Date', 'Wed, 31 May 2023 20:27:16 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:27:16.867 DEBUG:boto:path=/pklxkd-55/c 2023-05-31T20:27:16.867 DEBUG:boto:auth_path=/pklxkd-55/c 2023-05-31T20:27:16.868 DEBUG:boto:Method: PUT 2023-05-31T20:27:16.868 DEBUG:boto:Path: /pklxkd-55/c 2023-05-31T20:27:16.868 DEBUG:boto:Data: 2023-05-31T20:27:16.869 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-05-31T20:27:16.869 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:27:16.870 DEBUG:boto:Port: 8000 2023-05-31T20:27:16.870 DEBUG:boto:Params: {} 2023-05-31T20:27:16.870 DEBUG:boto:Token: None 2023-05-31T20:27:16.871 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Wed, 31 May 2023 20:27:16 GMT /pklxkd-55/c 2023-05-31T20:27:16.871 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:BsuxGLmplBlHp3VyZZR3UMY8x3M= 2023-05-31T20:27:16.872 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Wed, 31 May 2023 20:27:16 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:BsuxGLmplBlHp3VyZZR3UMY8x3M='} 2023-05-31T20:27:16.878 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000f45a79d36e823e6a-006477ada4-1630-a1'), ('Date', 'Wed, 31 May 2023 20:27:16 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:27:16.879 DEBUG:boto:path=/pklxkd-55/ 2023-05-31T20:27:16.880 DEBUG:boto:auth_path=/pklxkd-55/ 2023-05-31T20:27:16.881 DEBUG:boto:Method: HEAD 2023-05-31T20:27:16.881 DEBUG:boto:Path: /pklxkd-55/ 2023-05-31T20:27:16.882 DEBUG:boto:Data: 2023-05-31T20:27:16.883 DEBUG:boto:Headers: {} 2023-05-31T20:27:16.883 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:27:16.884 DEBUG:boto:Port: 8000 2023-05-31T20:27:16.885 DEBUG:boto:Params: {} 2023-05-31T20:27:16.886 DEBUG:boto:Token: None 2023-05-31T20:27:16.887 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:27:16 GMT /pklxkd-55/ 2023-05-31T20:27:16.888 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:/gdTTojnB6Hl7PzQ8paUIVDSxl0= 2023-05-31T20:27:16.888 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:27:16 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:/gdTTojnB6Hl7PzQ8paUIVDSxl0=', 'Content-Length': '0'} 2023-05-31T20:27:16.890 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '3'), ('X-RGW-Bytes-Used', '9'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000931f7c1e239f9d1f-006477ada4-1630-a1'), ('Content-Length', '0'), ('Date', 'Wed, 31 May 2023 20:27:16 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:27:16.891 DEBUG:boto:path=/pklxkd-55/d 2023-05-31T20:27:16.891 DEBUG:boto:auth_path=/pklxkd-55/d 2023-05-31T20:27:16.892 DEBUG:boto:Method: PUT 2023-05-31T20:27:16.892 DEBUG:boto:Path: /pklxkd-55/d 2023-05-31T20:27:16.893 DEBUG:boto:Data: 2023-05-31T20:27:16.893 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-05-31T20:27:16.893 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:27:16.894 DEBUG:boto:Port: 8000 2023-05-31T20:27:16.894 DEBUG:boto:Params: {} 2023-05-31T20:27:16.894 DEBUG:boto:Token: None 2023-05-31T20:27:16.895 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Wed, 31 May 2023 20:27:16 GMT /pklxkd-55/d 2023-05-31T20:27:16.895 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:hmT5C+THGJVcaahZJaHyH2ViBO8= 2023-05-31T20:27:16.896 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Wed, 31 May 2023 20:27:16 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:hmT5C+THGJVcaahZJaHyH2ViBO8='} 2023-05-31T20:27:16.940 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000d36174b3f8e74157-006477ada4-1630-a1'), ('Date', 'Wed, 31 May 2023 20:27:16 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:27:16.941 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-31T20:27:16.942 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-31T20:27:16.942 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-05-31T20:27:16.991 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2023-05-31T20:27:16.991 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2023-05-31T20:27:17.033 INFO:teuthology.orchestra.run.smithi099.stderr:No --period given, using current period=2e6123d2-d695-4350-9a01-61a455bc782b 2023-05-31T20:27:17.049 INFO:teuthology.orchestra.run.smithi099.stdout:[ 2023-05-31T20:27:17.050 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T20:27:17.050 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1685562631.516089_59.1", 2023-05-31T20:27:17.051 INFO:teuthology.orchestra.run.smithi099.stdout: "last_update": "2023-05-31T19:50:31.516089Z" 2023-05-31T20:27:17.051 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:27:17.051 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T20:27:17.051 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "", 2023-05-31T20:27:17.052 INFO:teuthology.orchestra.run.smithi099.stdout: "last_update": "0.000000" 2023-05-31T20:27:17.052 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:27:17.052 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T20:27:17.053 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1685562970.274572_93.1", 2023-05-31T20:27:17.053 INFO:teuthology.orchestra.run.smithi099.stdout: "last_update": "2023-05-31T19:56:10.274572Z" 2023-05-31T20:27:17.053 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:27:17.053 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T20:27:17.054 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1685564836.773548_358.1", 2023-05-31T20:27:17.054 INFO:teuthology.orchestra.run.smithi099.stdout: "last_update": "2023-05-31T20:27:16.773548Z" 2023-05-31T20:27:17.054 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T20:27:17.054 INFO:teuthology.orchestra.run.smithi099.stdout:] 2023-05-31T20:27:17.055 DEBUG:rgw_multi.tests:master meta markers={0: '1_1685562631.516089_59.1', 1: '', 2: '1_1685562970.274572_93.1', 3: '1_1685564836.773548_358.1'} 2023-05-31T20:27:17.056 INFO:rgw_multi.tests:starting meta checkpoint for zone=a2 2023-05-31T20:27:17.057 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-31T20:27:17.057 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-31T20:27:17.057 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-05-31T20:27:17.107 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2023-05-31T20:27:17.107 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2023-05-31T20:27:17.139 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:27:17.133+0000 7fa3fc26d780 20 RGW-SYNC:meta: read sync status 2023-05-31T20:27:17.140 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:27:17.134+0000 7fa3fc26d780 20 RGW-SYNC:meta: read sync status 2023-05-31T20:27:17.161 INFO:teuthology.orchestra.run.smithi099.stdout:{ 2023-05-31T20:27:17.161 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_status": { 2023-05-31T20:27:17.162 INFO:teuthology.orchestra.run.smithi099.stdout: "info": { 2023-05-31T20:27:17.162 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "sync", 2023-05-31T20:27:17.162 INFO:teuthology.orchestra.run.smithi099.stdout: "num_shards": 4, 2023-05-31T20:27:17.162 INFO:teuthology.orchestra.run.smithi099.stdout: "period": "2e6123d2-d695-4350-9a01-61a455bc782b", 2023-05-31T20:27:17.162 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2023-05-31T20:27:17.162 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:27:17.163 INFO:teuthology.orchestra.run.smithi099.stdout: "markers": [ 2023-05-31T20:27:17.163 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T20:27:17.163 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 0, 2023-05-31T20:27:17.163 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T20:27:17.163 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2023-05-31T20:27:17.163 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1685562631.516089_59.1", 2023-05-31T20:27:17.164 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T20:27:17.164 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 1, 2023-05-31T20:27:17.164 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T20:27:17.164 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2023-05-31T19:50:31.516089Z", 2023-05-31T20:27:17.164 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2023-05-31T20:27:17.164 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T20:27:17.165 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:27:17.165 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T20:27:17.165 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 1, 2023-05-31T20:27:17.165 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T20:27:17.165 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2023-05-31T20:27:17.165 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "", 2023-05-31T20:27:17.166 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T20:27:17.166 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2023-05-31T20:27:17.166 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T20:27:17.166 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "0.000000", 2023-05-31T20:27:17.166 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2023-05-31T20:27:17.166 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T20:27:17.167 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:27:17.167 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T20:27:17.167 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 2, 2023-05-31T20:27:17.167 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T20:27:17.167 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2023-05-31T20:27:17.167 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1685562970.274572_93.1", 2023-05-31T20:27:17.168 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T20:27:17.168 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2023-05-31T20:27:17.168 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T20:27:17.168 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2023-05-31T19:56:10.274572Z", 2023-05-31T20:27:17.168 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2023-05-31T20:27:17.168 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T20:27:17.169 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:27:17.169 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T20:27:17.169 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 3, 2023-05-31T20:27:17.169 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T20:27:17.169 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2023-05-31T20:27:17.169 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1685564836.773548_358.1", 2023-05-31T20:27:17.170 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T20:27:17.170 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2023-05-31T20:27:17.170 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T20:27:17.170 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2023-05-31T20:27:16.773548Z", 2023-05-31T20:27:17.171 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2023-05-31T20:27:17.171 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T20:27:17.171 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T20:27:17.171 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2023-05-31T20:27:17.172 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:27:17.172 INFO:teuthology.orchestra.run.smithi099.stdout: "full_sync": { 2023-05-31T20:27:17.172 INFO:teuthology.orchestra.run.smithi099.stdout: "total": 1, 2023-05-31T20:27:17.173 INFO:teuthology.orchestra.run.smithi099.stdout: "complete": 1 2023-05-31T20:27:17.173 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:27:17.173 INFO:teuthology.orchestra.run.smithi099.stdout: "current_time": "2023-05-31T20:27:17Z" 2023-05-31T20:27:17.174 INFO:teuthology.orchestra.run.smithi099.stdout:} 2023-05-31T20:27:17.175 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "2e6123d2-d695-4350-9a01-61a455bc782b", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1685562631.516089_59.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-05-31T19:50:31.516089Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1685562970.274572_93.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-05-31T19:56:10.274572Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1685564836.773548_358.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-05-31T20:27:16.773548Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-05-31T20:27:17Z" } 2023-05-31T20:27:17.176 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1685562631.516089_59.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-05-31T19:50:31.516089Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1685562970.274572_93.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-05-31T19:56:10.274572Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1685564836.773548_358.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-05-31T20:27:16.773548Z', 'realm_epoch': 2}}] 2023-05-31T20:27:17.177 DEBUG:rgw_multi.tests:log_status={0: '1_1685562631.516089_59.1', 1: '', 2: '1_1685562970.274572_93.1', 3: '1_1685564836.773548_358.1'} 2023-05-31T20:27:17.178 DEBUG:rgw_multi.tests:sync_status={0: '1_1685562631.516089_59.1', 1: '', 2: '1_1685562970.274572_93.1', 3: '1_1685564836.773548_358.1'} 2023-05-31T20:27:17.178 INFO:rgw_multi.tests:finish meta checkpoint for zone=a2 2023-05-31T20:27:17.179 DEBUG:rgw_multi.tests:bucket checkpoint: source=a1 target=a2 bucket=pklxkd-55 2023-05-31T20:27:17.180 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'pklxkd-55', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-05-31T20:27:17.180 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'pklxkd-55', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-05-31T20:27:17.181 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket pklxkd-55 --source-zone a1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2023-05-31T20:27:17.237 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2023-05-31T20:27:17.238 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2023-05-31T20:27:17.292 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:27:17.287+0000 7fd2fdd08780 1 waiting to reach incremental sync.. 2023-05-31T20:27:22.294 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:27:22.289+0000 7fd2fdd08780 1 bucket sync caught up with source: 2023-05-31T20:27:22.295 INFO:teuthology.orchestra.run.smithi099.stderr: local status: [, 00000000001.28.6, , , , 00000000001.45.6, , , 00000000001.42.6, , 00000000001.108.6] 2023-05-31T20:27:22.295 INFO:teuthology.orchestra.run.smithi099.stderr: remote markers: [, 00000000001.28.6, , , , 00000000001.45.6, , , 00000000001.42.6, , 00000000001.108.6] 2023-05-31T20:27:22.295 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:27:22.289+0000 7fd2fdd08780 0 bucket checkpoint complete 2023-05-31T20:27:22.313 DEBUG:rgw_multi.tests:bucket checkpoint: source=a2 target=a1 bucket=pklxkd-55 2023-05-31T20:27:22.314 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'pklxkd-55', '--source-zone', 'a2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-05-31T20:27:22.314 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'pklxkd-55', '--source-zone', 'a2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-05-31T20:27:22.315 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket pklxkd-55 --source-zone a2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2023-05-31T20:27:22.377 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2023-05-31T20:27:22.378 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2023-05-31T20:27:22.418 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:27:22.413+0000 7f80f47ca780 1 bucket sync caught up with source: 2023-05-31T20:27:22.418 INFO:teuthology.orchestra.run.smithi099.stderr: local status: [, 00000000001.1059.6, , , , 00000000001.1340.6, , , 00000000001.3554.6, , 00000000001.2404.6] 2023-05-31T20:27:22.419 INFO:teuthology.orchestra.run.smithi099.stderr: remote markers: [, 00000000001.1059.6, , , , 00000000001.1340.6, , , 00000000001.3554.6, , 00000000001.2404.6] 2023-05-31T20:27:22.419 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:27:22.413+0000 7f80f47ca780 0 bucket checkpoint complete 2023-05-31T20:27:22.434 INFO:tasks.rgw_multi.zone_rados:comparing bucket=pklxkd-55 zones={a2, a1} 2023-05-31T20:27:22.434 DEBUG:boto:path=/pklxkd-55/ 2023-05-31T20:27:22.435 DEBUG:boto:auth_path=/pklxkd-55/ 2023-05-31T20:27:22.435 DEBUG:boto:Method: HEAD 2023-05-31T20:27:22.436 DEBUG:boto:Path: /pklxkd-55/ 2023-05-31T20:27:22.436 DEBUG:boto:Data: 2023-05-31T20:27:22.436 DEBUG:boto:Headers: {} 2023-05-31T20:27:22.437 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:27:22.437 DEBUG:boto:Port: 8001 2023-05-31T20:27:22.438 DEBUG:boto:Params: {} 2023-05-31T20:27:22.438 DEBUG:boto:Token: None 2023-05-31T20:27:22.439 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:27:22 GMT /pklxkd-55/ 2023-05-31T20:27:22.439 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:YPYI0xrKQ20HAiP/MqnrYwIJejM= 2023-05-31T20:27:22.440 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:27:22 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:YPYI0xrKQ20HAiP/MqnrYwIJejM=', 'Content-Length': '0'} 2023-05-31T20:27:22.442 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '4'), ('X-RGW-Bytes-Used', '12'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000086949aba72247eff-006477adaa-162e-a2'), ('Content-Length', '0'), ('Date', 'Wed, 31 May 2023 20:27:22 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:27:22.443 DEBUG:boto:path=/pklxkd-55/ 2023-05-31T20:27:22.443 DEBUG:boto:auth_path=/pklxkd-55/ 2023-05-31T20:27:22.444 DEBUG:boto:Method: HEAD 2023-05-31T20:27:22.444 DEBUG:boto:Path: /pklxkd-55/ 2023-05-31T20:27:22.444 DEBUG:boto:Data: 2023-05-31T20:27:22.445 DEBUG:boto:Headers: {} 2023-05-31T20:27:22.445 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:27:22.446 DEBUG:boto:Port: 8000 2023-05-31T20:27:22.446 DEBUG:boto:Params: {} 2023-05-31T20:27:22.447 DEBUG:boto:Token: None 2023-05-31T20:27:22.447 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:27:22 GMT /pklxkd-55/ 2023-05-31T20:27:22.447 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:YPYI0xrKQ20HAiP/MqnrYwIJejM= 2023-05-31T20:27:22.448 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:27:22 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:YPYI0xrKQ20HAiP/MqnrYwIJejM=', 'Content-Length': '0'} 2023-05-31T20:27:22.450 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '4'), ('X-RGW-Bytes-Used', '12'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000fe55b73ee1900c3a-006477adaa-1630-a1'), ('Content-Length', '0'), ('Date', 'Wed, 31 May 2023 20:27:22 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:27:22.450 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2023-05-31T20:27:22.451 DEBUG:boto:path=/pklxkd-55/ 2023-05-31T20:27:22.451 DEBUG:boto:auth_path=/pklxkd-55/ 2023-05-31T20:27:22.452 DEBUG:boto:path=/pklxkd-55/?versions&max-keys=999 2023-05-31T20:27:22.452 DEBUG:boto:auth_path=/pklxkd-55/?versions&max-keys=999 2023-05-31T20:27:22.453 DEBUG:boto:Method: GET 2023-05-31T20:27:22.453 DEBUG:boto:Path: /pklxkd-55/?versions&max-keys=999 2023-05-31T20:27:22.454 DEBUG:boto:Data: 2023-05-31T20:27:22.454 DEBUG:boto:Headers: {} 2023-05-31T20:27:22.455 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:27:22.455 DEBUG:boto:Port: 8001 2023-05-31T20:27:22.455 DEBUG:boto:Params: {} 2023-05-31T20:27:22.456 DEBUG:boto:Token: None 2023-05-31T20:27:22.456 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:27:22 GMT /pklxkd-55/?versions 2023-05-31T20:27:22.457 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:rH1yLHJbUp2yS0ZgWD3zz80IhC4= 2023-05-31T20:27:22.458 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:27:22 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:rH1yLHJbUp2yS0ZgWD3zz80IhC4=', 'Content-Length': '0'} 2023-05-31T20:27:22.460 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ede5ee216d55c057-006477adaa-162e-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:27:22 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:27:22.461 DEBUG:boto:b'pklxkd-55999falseanulltrue2023-05-31T20:27:16.792Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalbnulltrue2023-05-31T20:27:16.850Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalcnulltrue2023-05-31T20:27:16.869Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormaldnulltrue2023-05-31T20:27:16.893Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:27:22.461 DEBUG:tasks.rgw_multi.zone_rados:o=a 2023-05-31T20:27:22.462 DEBUG:tasks.rgw_multi.zone_rados:o=b 2023-05-31T20:27:22.462 DEBUG:tasks.rgw_multi.zone_rados:o=c 2023-05-31T20:27:22.463 DEBUG:tasks.rgw_multi.zone_rados:o=d 2023-05-31T20:27:22.463 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2023-05-31T20:27:22.463 DEBUG:boto:path=/pklxkd-55/ 2023-05-31T20:27:22.464 DEBUG:boto:auth_path=/pklxkd-55/ 2023-05-31T20:27:22.464 DEBUG:boto:path=/pklxkd-55/?versions&max-keys=999 2023-05-31T20:27:22.464 DEBUG:boto:auth_path=/pklxkd-55/?versions&max-keys=999 2023-05-31T20:27:22.465 DEBUG:boto:Method: GET 2023-05-31T20:27:22.465 DEBUG:boto:Path: /pklxkd-55/?versions&max-keys=999 2023-05-31T20:27:22.465 DEBUG:boto:Data: 2023-05-31T20:27:22.466 DEBUG:boto:Headers: {} 2023-05-31T20:27:22.466 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:27:22.466 DEBUG:boto:Port: 8000 2023-05-31T20:27:22.467 DEBUG:boto:Params: {} 2023-05-31T20:27:22.467 DEBUG:boto:Token: None 2023-05-31T20:27:22.468 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:27:22 GMT /pklxkd-55/?versions 2023-05-31T20:27:22.468 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:rH1yLHJbUp2yS0ZgWD3zz80IhC4= 2023-05-31T20:27:22.469 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:27:22 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:rH1yLHJbUp2yS0ZgWD3zz80IhC4=', 'Content-Length': '0'} 2023-05-31T20:27:22.471 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000005ac610fc17f38232-006477adaa-1630-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:27:22 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:27:22.471 DEBUG:boto:b'pklxkd-55999falseanulltrue2023-05-31T20:27:16.792Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalbnulltrue2023-05-31T20:27:16.850Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalcnulltrue2023-05-31T20:27:16.869Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormaldnulltrue2023-05-31T20:27:16.893Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:27:22.472 DEBUG:tasks.rgw_multi.zone_rados:o=a 2023-05-31T20:27:22.472 DEBUG:tasks.rgw_multi.zone_rados:o=b 2023-05-31T20:27:22.473 DEBUG:tasks.rgw_multi.zone_rados:o=c 2023-05-31T20:27:22.473 DEBUG:tasks.rgw_multi.zone_rados:o=d 2023-05-31T20:27:22.473 DEBUG:boto:path=/pklxkd-55/ 2023-05-31T20:27:22.474 DEBUG:boto:auth_path=/pklxkd-55/ 2023-05-31T20:27:22.474 DEBUG:boto:path=/pklxkd-55/?versions&max-keys=999 2023-05-31T20:27:22.474 DEBUG:boto:auth_path=/pklxkd-55/?versions&max-keys=999 2023-05-31T20:27:22.475 DEBUG:boto:Method: GET 2023-05-31T20:27:22.475 DEBUG:boto:Path: /pklxkd-55/?versions&max-keys=999 2023-05-31T20:27:22.475 DEBUG:boto:Data: 2023-05-31T20:27:22.476 DEBUG:boto:Headers: {} 2023-05-31T20:27:22.476 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:27:22.476 DEBUG:boto:Port: 8001 2023-05-31T20:27:22.477 DEBUG:boto:Params: {} 2023-05-31T20:27:22.477 DEBUG:boto:Token: None 2023-05-31T20:27:22.478 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:27:22 GMT /pklxkd-55/?versions 2023-05-31T20:27:22.478 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:rH1yLHJbUp2yS0ZgWD3zz80IhC4= 2023-05-31T20:27:22.478 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:27:22 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:rH1yLHJbUp2yS0ZgWD3zz80IhC4=', 'Content-Length': '0'} 2023-05-31T20:27:22.480 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f2655bd4ef4e5557-006477adaa-162e-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:27:22 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:27:22.481 DEBUG:boto:b'pklxkd-55999falseanulltrue2023-05-31T20:27:16.792Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalbnulltrue2023-05-31T20:27:16.850Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalcnulltrue2023-05-31T20:27:16.869Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormaldnulltrue2023-05-31T20:27:16.893Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:27:22.481 DEBUG:boto:path=/pklxkd-55/ 2023-05-31T20:27:22.482 DEBUG:boto:auth_path=/pklxkd-55/ 2023-05-31T20:27:22.482 DEBUG:boto:path=/pklxkd-55/?versions&max-keys=999 2023-05-31T20:27:22.483 DEBUG:boto:auth_path=/pklxkd-55/?versions&max-keys=999 2023-05-31T20:27:22.483 DEBUG:boto:Method: GET 2023-05-31T20:27:22.483 DEBUG:boto:Path: /pklxkd-55/?versions&max-keys=999 2023-05-31T20:27:22.484 DEBUG:boto:Data: 2023-05-31T20:27:22.484 DEBUG:boto:Headers: {} 2023-05-31T20:27:22.484 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:27:22.485 DEBUG:boto:Port: 8000 2023-05-31T20:27:22.485 DEBUG:boto:Params: {} 2023-05-31T20:27:22.486 DEBUG:boto:Token: None 2023-05-31T20:27:22.486 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:27:22 GMT /pklxkd-55/?versions 2023-05-31T20:27:22.486 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:rH1yLHJbUp2yS0ZgWD3zz80IhC4= 2023-05-31T20:27:22.487 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:27:22 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:rH1yLHJbUp2yS0ZgWD3zz80IhC4=', 'Content-Length': '0'} 2023-05-31T20:27:22.489 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ad2e9e00ea56ee1e-006477adaa-1630-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:27:22 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:27:22.489 DEBUG:boto:b'pklxkd-55999falseanulltrue2023-05-31T20:27:16.792Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalbnulltrue2023-05-31T20:27:16.850Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalcnulltrue2023-05-31T20:27:16.869Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormaldnulltrue2023-05-31T20:27:16.893Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:27:22.490 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=a 2023-05-31T20:27:22.490 DEBUG:boto:path=/pklxkd-55/a 2023-05-31T20:27:22.491 DEBUG:boto:auth_path=/pklxkd-55/a 2023-05-31T20:27:22.491 DEBUG:boto:path=/pklxkd-55/a?versionId=null 2023-05-31T20:27:22.492 DEBUG:boto:auth_path=/pklxkd-55/a?versionId=null 2023-05-31T20:27:22.492 DEBUG:boto:Method: GET 2023-05-31T20:27:22.492 DEBUG:boto:Path: /pklxkd-55/a?versionId=null 2023-05-31T20:27:22.493 DEBUG:boto:Data: 2023-05-31T20:27:22.493 DEBUG:boto:Headers: {} 2023-05-31T20:27:22.493 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:27:22.494 DEBUG:boto:Port: 8001 2023-05-31T20:27:22.494 DEBUG:boto:Params: {} 2023-05-31T20:27:22.495 DEBUG:boto:Token: None 2023-05-31T20:27:22.495 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:27:22 GMT /pklxkd-55/a?versionId=null 2023-05-31T20:27:22.496 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:HjVcheEugGytioIwJHyG6X2nsjE= 2023-05-31T20:27:22.496 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:27:22 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:HjVcheEugGytioIwJHyG6X2nsjE=', 'Content-Length': '0'} 2023-05-31T20:27:22.498 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:27:16 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000504e4be1c777ee4c-006477adaa-162e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:27:22 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:27:22.499 DEBUG:boto:path=/pklxkd-55/a 2023-05-31T20:27:22.499 DEBUG:boto:auth_path=/pklxkd-55/a 2023-05-31T20:27:22.500 DEBUG:boto:path=/pklxkd-55/a?versionId=null 2023-05-31T20:27:22.500 DEBUG:boto:auth_path=/pklxkd-55/a?versionId=null 2023-05-31T20:27:22.500 DEBUG:boto:Method: GET 2023-05-31T20:27:22.501 DEBUG:boto:Path: /pklxkd-55/a?versionId=null 2023-05-31T20:27:22.501 DEBUG:boto:Data: 2023-05-31T20:27:22.502 DEBUG:boto:Headers: {} 2023-05-31T20:27:22.502 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:27:22.502 DEBUG:boto:Port: 8000 2023-05-31T20:27:22.503 DEBUG:boto:Params: {} 2023-05-31T20:27:22.504 DEBUG:boto:Token: None 2023-05-31T20:27:22.504 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:27:22 GMT /pklxkd-55/a?versionId=null 2023-05-31T20:27:22.505 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:HjVcheEugGytioIwJHyG6X2nsjE= 2023-05-31T20:27:22.505 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:27:22 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:HjVcheEugGytioIwJHyG6X2nsjE=', 'Content-Length': '0'} 2023-05-31T20:27:22.507 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:27:16 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002502ea0eb09dbcf3-006477adaa-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:27:22 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:27:22.508 DEBUG:boto:path=/pklxkd-55/a 2023-05-31T20:27:22.508 DEBUG:boto:auth_path=/pklxkd-55/a 2023-05-31T20:27:22.508 DEBUG:boto:path=/pklxkd-55/a?versionId=null 2023-05-31T20:27:22.509 DEBUG:boto:auth_path=/pklxkd-55/a?versionId=null 2023-05-31T20:27:22.509 DEBUG:boto:Method: HEAD 2023-05-31T20:27:22.509 DEBUG:boto:Path: /pklxkd-55/a?versionId=null 2023-05-31T20:27:22.510 DEBUG:boto:Data: 2023-05-31T20:27:22.510 DEBUG:boto:Headers: {} 2023-05-31T20:27:22.511 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:27:22.511 DEBUG:boto:Port: 8001 2023-05-31T20:27:22.511 DEBUG:boto:Params: {} 2023-05-31T20:27:22.512 DEBUG:boto:Token: None 2023-05-31T20:27:22.512 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:27:22 GMT /pklxkd-55/a?versionId=null 2023-05-31T20:27:22.513 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:WX/Aeemy7lpWwgAxqKo0HzEntEI= 2023-05-31T20:27:22.513 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:27:22 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:WX/Aeemy7lpWwgAxqKo0HzEntEI=', 'Content-Length': '0'} 2023-05-31T20:27:22.515 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:27:16 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000059dcd5da7b46036e-006477adaa-162e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:27:22 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:27:22.515 DEBUG:boto:path=/pklxkd-55/a 2023-05-31T20:27:22.516 DEBUG:boto:auth_path=/pklxkd-55/a 2023-05-31T20:27:22.516 DEBUG:boto:path=/pklxkd-55/a?versionId=null 2023-05-31T20:27:22.517 DEBUG:boto:auth_path=/pklxkd-55/a?versionId=null 2023-05-31T20:27:22.517 DEBUG:boto:Method: HEAD 2023-05-31T20:27:22.517 DEBUG:boto:Path: /pklxkd-55/a?versionId=null 2023-05-31T20:27:22.518 DEBUG:boto:Data: 2023-05-31T20:27:22.518 DEBUG:boto:Headers: {} 2023-05-31T20:27:22.518 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:27:22.519 DEBUG:boto:Port: 8000 2023-05-31T20:27:22.519 DEBUG:boto:Params: {} 2023-05-31T20:27:22.520 DEBUG:boto:Token: None 2023-05-31T20:27:22.520 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:27:22 GMT /pklxkd-55/a?versionId=null 2023-05-31T20:27:22.520 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:WX/Aeemy7lpWwgAxqKo0HzEntEI= 2023-05-31T20:27:22.521 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:27:22 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:WX/Aeemy7lpWwgAxqKo0HzEntEI=', 'Content-Length': '0'} 2023-05-31T20:27:22.522 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:27:16 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006f493aaa2431ad88-006477adaa-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:27:22 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:27:22.523 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=a 2023-05-31T20:27:22.523 DEBUG:boto:path=/pklxkd-55/a 2023-05-31T20:27:22.524 DEBUG:boto:auth_path=/pklxkd-55/a 2023-05-31T20:27:22.524 DEBUG:boto:path=/pklxkd-55/a?versionId=null 2023-05-31T20:27:22.524 DEBUG:boto:auth_path=/pklxkd-55/a?versionId=null 2023-05-31T20:27:22.525 DEBUG:boto:Method: GET 2023-05-31T20:27:22.525 DEBUG:boto:Path: /pklxkd-55/a?versionId=null 2023-05-31T20:27:22.525 DEBUG:boto:Data: 2023-05-31T20:27:22.526 DEBUG:boto:Headers: {} 2023-05-31T20:27:22.526 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:27:22.526 DEBUG:boto:Port: 8001 2023-05-31T20:27:22.527 DEBUG:boto:Params: {} 2023-05-31T20:27:22.527 DEBUG:boto:Token: None 2023-05-31T20:27:22.528 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:27:22 GMT /pklxkd-55/a?versionId=null 2023-05-31T20:27:22.528 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:HjVcheEugGytioIwJHyG6X2nsjE= 2023-05-31T20:27:22.529 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:27:22 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:HjVcheEugGytioIwJHyG6X2nsjE=', 'Content-Length': '0'} 2023-05-31T20:27:22.530 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:27:16 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b60516f9383d1884-006477adaa-162e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:27:22 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:27:22.531 DEBUG:boto:path=/pklxkd-55/a 2023-05-31T20:27:22.531 DEBUG:boto:auth_path=/pklxkd-55/a 2023-05-31T20:27:22.532 DEBUG:boto:path=/pklxkd-55/a?versionId=null 2023-05-31T20:27:22.532 DEBUG:boto:auth_path=/pklxkd-55/a?versionId=null 2023-05-31T20:27:22.532 DEBUG:boto:Method: GET 2023-05-31T20:27:22.533 DEBUG:boto:Path: /pklxkd-55/a?versionId=null 2023-05-31T20:27:22.533 DEBUG:boto:Data: 2023-05-31T20:27:22.534 DEBUG:boto:Headers: {} 2023-05-31T20:27:22.534 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:27:22.534 DEBUG:boto:Port: 8000 2023-05-31T20:27:22.535 DEBUG:boto:Params: {} 2023-05-31T20:27:22.535 DEBUG:boto:Token: None 2023-05-31T20:27:22.536 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:27:22 GMT /pklxkd-55/a?versionId=null 2023-05-31T20:27:22.536 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:HjVcheEugGytioIwJHyG6X2nsjE= 2023-05-31T20:27:22.537 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:27:22 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:HjVcheEugGytioIwJHyG6X2nsjE=', 'Content-Length': '0'} 2023-05-31T20:27:22.538 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:27:16 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000dd64f19f26d5ea30-006477adaa-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:27:22 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:27:22.539 DEBUG:boto:path=/pklxkd-55/ 2023-05-31T20:27:22.539 DEBUG:boto:auth_path=/pklxkd-55/ 2023-05-31T20:27:22.540 DEBUG:boto:path=/pklxkd-55/?prefix=a 2023-05-31T20:27:22.540 DEBUG:boto:auth_path=/pklxkd-55/?prefix=a 2023-05-31T20:27:22.541 DEBUG:boto:Method: GET 2023-05-31T20:27:22.541 DEBUG:boto:Path: /pklxkd-55/?prefix=a 2023-05-31T20:27:22.541 DEBUG:boto:Data: 2023-05-31T20:27:22.542 DEBUG:boto:Headers: {} 2023-05-31T20:27:22.542 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:27:22.542 DEBUG:boto:Port: 8001 2023-05-31T20:27:22.543 DEBUG:boto:Params: {} 2023-05-31T20:27:22.543 DEBUG:boto:Token: None 2023-05-31T20:27:22.544 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:27:22 GMT /pklxkd-55/ 2023-05-31T20:27:22.544 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:p2p+8G1ZdPifmzf+JcgpB78h0o8= 2023-05-31T20:27:22.545 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:27:22 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:p2p+8G1ZdPifmzf+JcgpB78h0o8=', 'Content-Length': '0'} 2023-05-31T20:27:22.547 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000004baa82331feb0ff3-006477adaa-162e-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:27:22 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:27:22.547 DEBUG:boto:b'pklxkd-55a1000falsea2023-05-31T20:27:16.792Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:27:22.548 DEBUG:boto:path=/pklxkd-55/ 2023-05-31T20:27:22.548 DEBUG:boto:auth_path=/pklxkd-55/ 2023-05-31T20:27:22.549 DEBUG:boto:path=/pklxkd-55/?prefix=a 2023-05-31T20:27:22.549 DEBUG:boto:auth_path=/pklxkd-55/?prefix=a 2023-05-31T20:27:22.549 DEBUG:boto:Method: GET 2023-05-31T20:27:22.550 DEBUG:boto:Path: /pklxkd-55/?prefix=a 2023-05-31T20:27:22.550 DEBUG:boto:Data: 2023-05-31T20:27:22.550 DEBUG:boto:Headers: {} 2023-05-31T20:27:22.551 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:27:22.552 DEBUG:boto:Port: 8000 2023-05-31T20:27:22.552 DEBUG:boto:Params: {} 2023-05-31T20:27:22.552 DEBUG:boto:Token: None 2023-05-31T20:27:22.553 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:27:22 GMT /pklxkd-55/ 2023-05-31T20:27:22.553 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:p2p+8G1ZdPifmzf+JcgpB78h0o8= 2023-05-31T20:27:22.554 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:27:22 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:p2p+8G1ZdPifmzf+JcgpB78h0o8=', 'Content-Length': '0'} 2023-05-31T20:27:22.556 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000009e9d72b6eb393c23-006477adaa-1630-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:27:22 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:27:22.557 DEBUG:boto:b'pklxkd-55a1000falsea2023-05-31T20:27:16.792Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:27:22.558 DEBUG:boto:path=/pklxkd-55/a 2023-05-31T20:27:22.558 DEBUG:boto:auth_path=/pklxkd-55/a 2023-05-31T20:27:22.558 DEBUG:boto:Method: HEAD 2023-05-31T20:27:22.559 DEBUG:boto:Path: /pklxkd-55/a 2023-05-31T20:27:22.559 DEBUG:boto:Data: 2023-05-31T20:27:22.559 DEBUG:boto:Headers: {} 2023-05-31T20:27:22.560 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:27:22.560 DEBUG:boto:Port: 8001 2023-05-31T20:27:22.560 DEBUG:boto:Params: {} 2023-05-31T20:27:22.561 DEBUG:boto:Token: None 2023-05-31T20:27:22.561 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:27:22 GMT /pklxkd-55/a 2023-05-31T20:27:22.562 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:FN5t9rLsnpx97Hrm+q+piTRkijU= 2023-05-31T20:27:22.562 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:27:22 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:FN5t9rLsnpx97Hrm+q+piTRkijU=', 'Content-Length': '0'} 2023-05-31T20:27:22.564 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:27:16 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000103b42fd84696813-006477adaa-162e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:27:22 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:27:22.565 DEBUG:boto:path=/pklxkd-55/a 2023-05-31T20:27:22.565 DEBUG:boto:auth_path=/pklxkd-55/a 2023-05-31T20:27:22.566 DEBUG:boto:Method: HEAD 2023-05-31T20:27:22.566 DEBUG:boto:Path: /pklxkd-55/a 2023-05-31T20:27:22.566 DEBUG:boto:Data: 2023-05-31T20:27:22.567 DEBUG:boto:Headers: {} 2023-05-31T20:27:22.567 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:27:22.567 DEBUG:boto:Port: 8000 2023-05-31T20:27:22.568 DEBUG:boto:Params: {} 2023-05-31T20:27:22.568 DEBUG:boto:Token: None 2023-05-31T20:27:22.568 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:27:22 GMT /pklxkd-55/a 2023-05-31T20:27:22.569 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:FN5t9rLsnpx97Hrm+q+piTRkijU= 2023-05-31T20:27:22.570 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:27:22 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:FN5t9rLsnpx97Hrm+q+piTRkijU=', 'Content-Length': '0'} 2023-05-31T20:27:22.571 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:27:16 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003d0ec68bb5dfca65-006477adaa-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:27:22 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:27:22.572 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=a 2023-05-31T20:27:22.572 DEBUG:boto:path=/pklxkd-55/a 2023-05-31T20:27:22.573 DEBUG:boto:auth_path=/pklxkd-55/a 2023-05-31T20:27:22.573 DEBUG:boto:Method: GET 2023-05-31T20:27:22.574 DEBUG:boto:Path: /pklxkd-55/a 2023-05-31T20:27:22.574 DEBUG:boto:Data: 2023-05-31T20:27:22.574 DEBUG:boto:Headers: {} 2023-05-31T20:27:22.575 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:27:22.575 DEBUG:boto:Port: 8001 2023-05-31T20:27:22.575 DEBUG:boto:Params: {} 2023-05-31T20:27:22.576 DEBUG:boto:Token: None 2023-05-31T20:27:22.576 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:27:22 GMT /pklxkd-55/a 2023-05-31T20:27:22.577 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:QfBq7c12kIVH21Pp/9FHihChbNU= 2023-05-31T20:27:22.577 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:27:22 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:QfBq7c12kIVH21Pp/9FHihChbNU=', 'Content-Length': '0'} 2023-05-31T20:27:22.579 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:27:16 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f0440d4046779ebb-006477adaa-162e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:27:22 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:27:22.580 DEBUG:boto:path=/pklxkd-55/a 2023-05-31T20:27:22.580 DEBUG:boto:auth_path=/pklxkd-55/a 2023-05-31T20:27:22.581 DEBUG:boto:Method: GET 2023-05-31T20:27:22.586 DEBUG:boto:Path: /pklxkd-55/a 2023-05-31T20:27:22.587 DEBUG:boto:Data: 2023-05-31T20:27:22.587 DEBUG:boto:Headers: {} 2023-05-31T20:27:22.588 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:27:22.589 DEBUG:boto:Port: 8000 2023-05-31T20:27:22.590 DEBUG:boto:Params: {} 2023-05-31T20:27:22.590 DEBUG:boto:Token: None 2023-05-31T20:27:22.591 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:27:22 GMT /pklxkd-55/a 2023-05-31T20:27:22.592 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:QfBq7c12kIVH21Pp/9FHihChbNU= 2023-05-31T20:27:22.593 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:27:22 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:QfBq7c12kIVH21Pp/9FHihChbNU=', 'Content-Length': '0'} 2023-05-31T20:27:22.595 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:27:16 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000084daad4fcde81b4b-006477adaa-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:27:22 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:27:22.596 DEBUG:boto:path=/pklxkd-55/ 2023-05-31T20:27:22.596 DEBUG:boto:auth_path=/pklxkd-55/ 2023-05-31T20:27:22.597 DEBUG:boto:path=/pklxkd-55/?prefix=a 2023-05-31T20:27:22.598 DEBUG:boto:auth_path=/pklxkd-55/?prefix=a 2023-05-31T20:27:22.599 DEBUG:boto:Method: GET 2023-05-31T20:27:22.600 DEBUG:boto:Path: /pklxkd-55/?prefix=a 2023-05-31T20:27:22.600 DEBUG:boto:Data: 2023-05-31T20:27:22.601 DEBUG:boto:Headers: {} 2023-05-31T20:27:22.602 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:27:22.603 DEBUG:boto:Port: 8001 2023-05-31T20:27:22.604 DEBUG:boto:Params: {} 2023-05-31T20:27:22.604 DEBUG:boto:Token: None 2023-05-31T20:27:22.605 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:27:22 GMT /pklxkd-55/ 2023-05-31T20:27:22.606 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:p2p+8G1ZdPifmzf+JcgpB78h0o8= 2023-05-31T20:27:22.607 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:27:22 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:p2p+8G1ZdPifmzf+JcgpB78h0o8=', 'Content-Length': '0'} 2023-05-31T20:27:22.609 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000017d0e613c81f17dc-006477adaa-162e-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:27:22 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:27:22.611 DEBUG:boto:b'pklxkd-55a1000falsea2023-05-31T20:27:16.792Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:27:22.611 DEBUG:boto:path=/pklxkd-55/ 2023-05-31T20:27:22.612 DEBUG:boto:auth_path=/pklxkd-55/ 2023-05-31T20:27:22.613 DEBUG:boto:path=/pklxkd-55/?prefix=a 2023-05-31T20:27:22.613 DEBUG:boto:auth_path=/pklxkd-55/?prefix=a 2023-05-31T20:27:22.614 DEBUG:boto:Method: GET 2023-05-31T20:27:22.614 DEBUG:boto:Path: /pklxkd-55/?prefix=a 2023-05-31T20:27:22.615 DEBUG:boto:Data: 2023-05-31T20:27:22.615 DEBUG:boto:Headers: {} 2023-05-31T20:27:22.616 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:27:22.617 DEBUG:boto:Port: 8000 2023-05-31T20:27:22.618 DEBUG:boto:Params: {} 2023-05-31T20:27:22.619 DEBUG:boto:Token: None 2023-05-31T20:27:22.620 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:27:22 GMT /pklxkd-55/ 2023-05-31T20:27:22.621 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:p2p+8G1ZdPifmzf+JcgpB78h0o8= 2023-05-31T20:27:22.622 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:27:22 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:p2p+8G1ZdPifmzf+JcgpB78h0o8=', 'Content-Length': '0'} 2023-05-31T20:27:22.624 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000009827cdc059cd62c2-006477adaa-1630-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:27:22 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:27:22.624 DEBUG:boto:b'pklxkd-55a1000falsea2023-05-31T20:27:16.792Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:27:22.625 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=b 2023-05-31T20:27:22.625 DEBUG:boto:path=/pklxkd-55/b 2023-05-31T20:27:22.626 DEBUG:boto:auth_path=/pklxkd-55/b 2023-05-31T20:27:22.626 DEBUG:boto:path=/pklxkd-55/b?versionId=null 2023-05-31T20:27:22.626 DEBUG:boto:auth_path=/pklxkd-55/b?versionId=null 2023-05-31T20:27:22.627 DEBUG:boto:Method: GET 2023-05-31T20:27:22.627 DEBUG:boto:Path: /pklxkd-55/b?versionId=null 2023-05-31T20:27:22.628 DEBUG:boto:Data: 2023-05-31T20:27:22.628 DEBUG:boto:Headers: {} 2023-05-31T20:27:22.629 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:27:22.630 DEBUG:boto:Port: 8001 2023-05-31T20:27:22.631 DEBUG:boto:Params: {} 2023-05-31T20:27:22.632 DEBUG:boto:Token: None 2023-05-31T20:27:22.633 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:27:22 GMT /pklxkd-55/b?versionId=null 2023-05-31T20:27:22.634 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:QHiV5puDrYlTI9MGI70ZYfdDQxw= 2023-05-31T20:27:22.635 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:27:22 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:QHiV5puDrYlTI9MGI70ZYfdDQxw=', 'Content-Length': '0'} 2023-05-31T20:27:22.636 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:27:16 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003cdf3c8bd483e634-006477adaa-162e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:27:22 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:27:22.684 DEBUG:boto:path=/pklxkd-55/b 2023-05-31T20:27:22.685 DEBUG:boto:auth_path=/pklxkd-55/b 2023-05-31T20:27:22.685 DEBUG:boto:path=/pklxkd-55/b?versionId=null 2023-05-31T20:27:22.686 DEBUG:boto:auth_path=/pklxkd-55/b?versionId=null 2023-05-31T20:27:22.686 DEBUG:boto:Method: GET 2023-05-31T20:27:22.686 DEBUG:boto:Path: /pklxkd-55/b?versionId=null 2023-05-31T20:27:22.687 DEBUG:boto:Data: 2023-05-31T20:27:22.687 DEBUG:boto:Headers: {} 2023-05-31T20:27:22.687 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:27:22.688 DEBUG:boto:Port: 8000 2023-05-31T20:27:22.688 DEBUG:boto:Params: {} 2023-05-31T20:27:22.689 DEBUG:boto:Token: None 2023-05-31T20:27:22.689 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:27:22 GMT /pklxkd-55/b?versionId=null 2023-05-31T20:27:22.690 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:QHiV5puDrYlTI9MGI70ZYfdDQxw= 2023-05-31T20:27:22.690 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:27:22 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:QHiV5puDrYlTI9MGI70ZYfdDQxw=', 'Content-Length': '0'} 2023-05-31T20:27:22.692 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:27:16 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000be0c6f3f4c8ac5d-006477adaa-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:27:22 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:27:22.693 DEBUG:boto:path=/pklxkd-55/b 2023-05-31T20:27:22.693 DEBUG:boto:auth_path=/pklxkd-55/b 2023-05-31T20:27:22.694 DEBUG:boto:path=/pklxkd-55/b?versionId=null 2023-05-31T20:27:22.694 DEBUG:boto:auth_path=/pklxkd-55/b?versionId=null 2023-05-31T20:27:22.694 DEBUG:boto:Method: HEAD 2023-05-31T20:27:22.695 DEBUG:boto:Path: /pklxkd-55/b?versionId=null 2023-05-31T20:27:22.695 DEBUG:boto:Data: 2023-05-31T20:27:22.696 DEBUG:boto:Headers: {} 2023-05-31T20:27:22.696 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:27:22.697 DEBUG:boto:Port: 8001 2023-05-31T20:27:22.697 DEBUG:boto:Params: {} 2023-05-31T20:27:22.697 DEBUG:boto:Token: None 2023-05-31T20:27:22.698 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:27:22 GMT /pklxkd-55/b?versionId=null 2023-05-31T20:27:22.699 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:IpxKqJi7IvrKxtKS7HSWE8HZNqk= 2023-05-31T20:27:22.699 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:27:22 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:IpxKqJi7IvrKxtKS7HSWE8HZNqk=', 'Content-Length': '0'} 2023-05-31T20:27:22.701 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:27:16 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000070b7595de7bcab08-006477adaa-162e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:27:22 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:27:22.702 DEBUG:boto:path=/pklxkd-55/b 2023-05-31T20:27:22.703 DEBUG:boto:auth_path=/pklxkd-55/b 2023-05-31T20:27:22.703 DEBUG:boto:path=/pklxkd-55/b?versionId=null 2023-05-31T20:27:22.704 DEBUG:boto:auth_path=/pklxkd-55/b?versionId=null 2023-05-31T20:27:22.704 DEBUG:boto:Method: HEAD 2023-05-31T20:27:22.705 DEBUG:boto:Path: /pklxkd-55/b?versionId=null 2023-05-31T20:27:22.705 DEBUG:boto:Data: 2023-05-31T20:27:22.705 DEBUG:boto:Headers: {} 2023-05-31T20:27:22.706 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:27:22.706 DEBUG:boto:Port: 8000 2023-05-31T20:27:22.706 DEBUG:boto:Params: {} 2023-05-31T20:27:22.707 DEBUG:boto:Token: None 2023-05-31T20:27:22.708 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:27:22 GMT /pklxkd-55/b?versionId=null 2023-05-31T20:27:22.708 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:IpxKqJi7IvrKxtKS7HSWE8HZNqk= 2023-05-31T20:27:22.709 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:27:22 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:IpxKqJi7IvrKxtKS7HSWE8HZNqk=', 'Content-Length': '0'} 2023-05-31T20:27:22.710 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:27:16 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002a4bda418a9d4a1d-006477adaa-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:27:22 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:27:22.711 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=b 2023-05-31T20:27:22.712 DEBUG:boto:path=/pklxkd-55/b 2023-05-31T20:27:22.713 DEBUG:boto:auth_path=/pklxkd-55/b 2023-05-31T20:27:22.713 DEBUG:boto:path=/pklxkd-55/b?versionId=null 2023-05-31T20:27:22.714 DEBUG:boto:auth_path=/pklxkd-55/b?versionId=null 2023-05-31T20:27:22.715 DEBUG:boto:Method: GET 2023-05-31T20:27:22.715 DEBUG:boto:Path: /pklxkd-55/b?versionId=null 2023-05-31T20:27:22.716 DEBUG:boto:Data: 2023-05-31T20:27:22.717 DEBUG:boto:Headers: {} 2023-05-31T20:27:22.717 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:27:22.718 DEBUG:boto:Port: 8001 2023-05-31T20:27:22.719 DEBUG:boto:Params: {} 2023-05-31T20:27:22.719 DEBUG:boto:Token: None 2023-05-31T20:27:22.720 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:27:22 GMT /pklxkd-55/b?versionId=null 2023-05-31T20:27:22.721 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:QHiV5puDrYlTI9MGI70ZYfdDQxw= 2023-05-31T20:27:22.722 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:27:22 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:QHiV5puDrYlTI9MGI70ZYfdDQxw=', 'Content-Length': '0'} 2023-05-31T20:27:22.724 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:27:16 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000350e9545cc9ab1f4-006477adaa-162e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:27:22 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:27:22.725 DEBUG:boto:path=/pklxkd-55/b 2023-05-31T20:27:22.726 DEBUG:boto:auth_path=/pklxkd-55/b 2023-05-31T20:27:22.726 DEBUG:boto:path=/pklxkd-55/b?versionId=null 2023-05-31T20:27:22.727 DEBUG:boto:auth_path=/pklxkd-55/b?versionId=null 2023-05-31T20:27:22.728 DEBUG:boto:Method: GET 2023-05-31T20:27:22.728 DEBUG:boto:Path: /pklxkd-55/b?versionId=null 2023-05-31T20:27:22.729 DEBUG:boto:Data: 2023-05-31T20:27:22.730 DEBUG:boto:Headers: {} 2023-05-31T20:27:22.730 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:27:22.731 DEBUG:boto:Port: 8000 2023-05-31T20:27:22.732 DEBUG:boto:Params: {} 2023-05-31T20:27:22.732 DEBUG:boto:Token: None 2023-05-31T20:27:22.733 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:27:22 GMT /pklxkd-55/b?versionId=null 2023-05-31T20:27:22.734 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:QHiV5puDrYlTI9MGI70ZYfdDQxw= 2023-05-31T20:27:22.735 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:27:22 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:QHiV5puDrYlTI9MGI70ZYfdDQxw=', 'Content-Length': '0'} 2023-05-31T20:27:22.737 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:27:16 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008bb112c27dcb4e8d-006477adaa-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:27:22 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:27:22.780 DEBUG:boto:path=/pklxkd-55/ 2023-05-31T20:27:22.781 DEBUG:boto:auth_path=/pklxkd-55/ 2023-05-31T20:27:22.781 DEBUG:boto:path=/pklxkd-55/?prefix=b 2023-05-31T20:27:22.782 DEBUG:boto:auth_path=/pklxkd-55/?prefix=b 2023-05-31T20:27:22.782 DEBUG:boto:Method: GET 2023-05-31T20:27:22.783 DEBUG:boto:Path: /pklxkd-55/?prefix=b 2023-05-31T20:27:22.783 DEBUG:boto:Data: 2023-05-31T20:27:22.783 DEBUG:boto:Headers: {} 2023-05-31T20:27:22.784 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:27:22.784 DEBUG:boto:Port: 8001 2023-05-31T20:27:22.785 DEBUG:boto:Params: {} 2023-05-31T20:27:22.785 DEBUG:boto:Token: None 2023-05-31T20:27:22.786 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:27:22 GMT /pklxkd-55/ 2023-05-31T20:27:22.786 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:p2p+8G1ZdPifmzf+JcgpB78h0o8= 2023-05-31T20:27:22.787 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:27:22 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:p2p+8G1ZdPifmzf+JcgpB78h0o8=', 'Content-Length': '0'} 2023-05-31T20:27:22.789 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000067d0b26bcda2484d-006477adaa-162e-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:27:22 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:27:22.790 DEBUG:boto:b'pklxkd-55b1000falseb2023-05-31T20:27:16.850Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:27:22.791 DEBUG:boto:path=/pklxkd-55/ 2023-05-31T20:27:22.791 DEBUG:boto:auth_path=/pklxkd-55/ 2023-05-31T20:27:22.792 DEBUG:boto:path=/pklxkd-55/?prefix=b 2023-05-31T20:27:22.792 DEBUG:boto:auth_path=/pklxkd-55/?prefix=b 2023-05-31T20:27:22.792 DEBUG:boto:Method: GET 2023-05-31T20:27:22.793 DEBUG:boto:Path: /pklxkd-55/?prefix=b 2023-05-31T20:27:22.793 DEBUG:boto:Data: 2023-05-31T20:27:22.794 DEBUG:boto:Headers: {} 2023-05-31T20:27:22.794 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:27:22.794 DEBUG:boto:Port: 8000 2023-05-31T20:27:22.795 DEBUG:boto:Params: {} 2023-05-31T20:27:22.795 DEBUG:boto:Token: None 2023-05-31T20:27:22.796 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:27:22 GMT /pklxkd-55/ 2023-05-31T20:27:22.796 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:p2p+8G1ZdPifmzf+JcgpB78h0o8= 2023-05-31T20:27:22.797 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:27:22 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:p2p+8G1ZdPifmzf+JcgpB78h0o8=', 'Content-Length': '0'} 2023-05-31T20:27:22.799 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000005511508bf33a4a52-006477adaa-1630-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:27:22 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:27:22.800 DEBUG:boto:b'pklxkd-55b1000falseb2023-05-31T20:27:16.850Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:27:22.801 DEBUG:boto:path=/pklxkd-55/b 2023-05-31T20:27:22.801 DEBUG:boto:auth_path=/pklxkd-55/b 2023-05-31T20:27:22.802 DEBUG:boto:Method: HEAD 2023-05-31T20:27:22.802 DEBUG:boto:Path: /pklxkd-55/b 2023-05-31T20:27:22.802 DEBUG:boto:Data: 2023-05-31T20:27:22.803 DEBUG:boto:Headers: {} 2023-05-31T20:27:22.803 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:27:22.803 DEBUG:boto:Port: 8001 2023-05-31T20:27:22.804 DEBUG:boto:Params: {} 2023-05-31T20:27:22.804 DEBUG:boto:Token: None 2023-05-31T20:27:22.804 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:27:22 GMT /pklxkd-55/b 2023-05-31T20:27:22.805 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:DNPJ7OqoWxtHRIezb8NkCfZP7/U= 2023-05-31T20:27:22.805 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:27:22 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:DNPJ7OqoWxtHRIezb8NkCfZP7/U=', 'Content-Length': '0'} 2023-05-31T20:27:22.807 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:27:16 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ee2ade6ec83adf85-006477adaa-162e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:27:22 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:27:22.808 DEBUG:boto:path=/pklxkd-55/b 2023-05-31T20:27:22.808 DEBUG:boto:auth_path=/pklxkd-55/b 2023-05-31T20:27:22.808 DEBUG:boto:Method: HEAD 2023-05-31T20:27:22.809 DEBUG:boto:Path: /pklxkd-55/b 2023-05-31T20:27:22.809 DEBUG:boto:Data: 2023-05-31T20:27:22.810 DEBUG:boto:Headers: {} 2023-05-31T20:27:22.810 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:27:22.810 DEBUG:boto:Port: 8000 2023-05-31T20:27:22.811 DEBUG:boto:Params: {} 2023-05-31T20:27:22.811 DEBUG:boto:Token: None 2023-05-31T20:27:22.811 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:27:22 GMT /pklxkd-55/b 2023-05-31T20:27:22.812 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:DNPJ7OqoWxtHRIezb8NkCfZP7/U= 2023-05-31T20:27:22.812 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:27:22 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:DNPJ7OqoWxtHRIezb8NkCfZP7/U=', 'Content-Length': '0'} 2023-05-31T20:27:22.814 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:27:16 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000246302392ec58be-006477adaa-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:27:22 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:27:22.814 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=b 2023-05-31T20:27:22.815 DEBUG:boto:path=/pklxkd-55/b 2023-05-31T20:27:22.815 DEBUG:boto:auth_path=/pklxkd-55/b 2023-05-31T20:27:22.815 DEBUG:boto:Method: GET 2023-05-31T20:27:22.816 DEBUG:boto:Path: /pklxkd-55/b 2023-05-31T20:27:22.816 DEBUG:boto:Data: 2023-05-31T20:27:22.817 DEBUG:boto:Headers: {} 2023-05-31T20:27:22.817 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:27:22.817 DEBUG:boto:Port: 8001 2023-05-31T20:27:22.818 DEBUG:boto:Params: {} 2023-05-31T20:27:22.818 DEBUG:boto:Token: None 2023-05-31T20:27:22.818 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:27:22 GMT /pklxkd-55/b 2023-05-31T20:27:22.819 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:kwn5VbL4fPr6/U8LpUeyCZxJ52A= 2023-05-31T20:27:22.819 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:27:22 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:kwn5VbL4fPr6/U8LpUeyCZxJ52A=', 'Content-Length': '0'} 2023-05-31T20:27:22.821 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:27:16 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000dd7409e7649fc70f-006477adaa-162e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:27:22 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:27:22.821 DEBUG:boto:path=/pklxkd-55/b 2023-05-31T20:27:22.822 DEBUG:boto:auth_path=/pklxkd-55/b 2023-05-31T20:27:22.822 DEBUG:boto:Method: GET 2023-05-31T20:27:22.823 DEBUG:boto:Path: /pklxkd-55/b 2023-05-31T20:27:22.823 DEBUG:boto:Data: 2023-05-31T20:27:22.823 DEBUG:boto:Headers: {} 2023-05-31T20:27:22.824 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:27:22.824 DEBUG:boto:Port: 8000 2023-05-31T20:27:22.824 DEBUG:boto:Params: {} 2023-05-31T20:27:22.825 DEBUG:boto:Token: None 2023-05-31T20:27:22.825 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:27:22 GMT /pklxkd-55/b 2023-05-31T20:27:22.826 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:kwn5VbL4fPr6/U8LpUeyCZxJ52A= 2023-05-31T20:27:22.826 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:27:22 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:kwn5VbL4fPr6/U8LpUeyCZxJ52A=', 'Content-Length': '0'} 2023-05-31T20:27:22.828 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:27:16 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000761c1a6885a4dd19-006477adaa-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:27:22 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:27:22.828 DEBUG:boto:path=/pklxkd-55/ 2023-05-31T20:27:22.829 DEBUG:boto:auth_path=/pklxkd-55/ 2023-05-31T20:27:22.829 DEBUG:boto:path=/pklxkd-55/?prefix=b 2023-05-31T20:27:22.830 DEBUG:boto:auth_path=/pklxkd-55/?prefix=b 2023-05-31T20:27:22.830 DEBUG:boto:Method: GET 2023-05-31T20:27:22.830 DEBUG:boto:Path: /pklxkd-55/?prefix=b 2023-05-31T20:27:22.831 DEBUG:boto:Data: 2023-05-31T20:27:22.831 DEBUG:boto:Headers: {} 2023-05-31T20:27:22.831 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:27:22.832 DEBUG:boto:Port: 8001 2023-05-31T20:27:22.832 DEBUG:boto:Params: {} 2023-05-31T20:27:22.833 DEBUG:boto:Token: None 2023-05-31T20:27:22.833 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:27:22 GMT /pklxkd-55/ 2023-05-31T20:27:22.833 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:p2p+8G1ZdPifmzf+JcgpB78h0o8= 2023-05-31T20:27:22.834 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:27:22 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:p2p+8G1ZdPifmzf+JcgpB78h0o8=', 'Content-Length': '0'} 2023-05-31T20:27:22.836 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000003e99522e8d609218-006477adaa-162e-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:27:22 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:27:22.876 DEBUG:boto:b'pklxkd-55b1000falseb2023-05-31T20:27:16.850Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:27:22.877 DEBUG:boto:path=/pklxkd-55/ 2023-05-31T20:27:22.878 DEBUG:boto:auth_path=/pklxkd-55/ 2023-05-31T20:27:22.878 DEBUG:boto:path=/pklxkd-55/?prefix=b 2023-05-31T20:27:22.879 DEBUG:boto:auth_path=/pklxkd-55/?prefix=b 2023-05-31T20:27:22.879 DEBUG:boto:Method: GET 2023-05-31T20:27:22.879 DEBUG:boto:Path: /pklxkd-55/?prefix=b 2023-05-31T20:27:22.880 DEBUG:boto:Data: 2023-05-31T20:27:22.880 DEBUG:boto:Headers: {} 2023-05-31T20:27:22.883 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:27:22.883 DEBUG:boto:Port: 8000 2023-05-31T20:27:22.884 DEBUG:boto:Params: {} 2023-05-31T20:27:22.884 DEBUG:boto:Token: None 2023-05-31T20:27:22.885 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:27:22 GMT /pklxkd-55/ 2023-05-31T20:27:22.885 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:p2p+8G1ZdPifmzf+JcgpB78h0o8= 2023-05-31T20:27:22.886 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:27:22 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:p2p+8G1ZdPifmzf+JcgpB78h0o8=', 'Content-Length': '0'} 2023-05-31T20:27:22.888 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000001a3d66db016228de-006477adaa-1630-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:27:22 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:27:22.888 DEBUG:boto:b'pklxkd-55b1000falseb2023-05-31T20:27:16.850Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:27:22.889 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=c 2023-05-31T20:27:22.890 DEBUG:boto:path=/pklxkd-55/c 2023-05-31T20:27:22.890 DEBUG:boto:auth_path=/pklxkd-55/c 2023-05-31T20:27:22.890 DEBUG:boto:path=/pklxkd-55/c?versionId=null 2023-05-31T20:27:22.891 DEBUG:boto:auth_path=/pklxkd-55/c?versionId=null 2023-05-31T20:27:22.891 DEBUG:boto:Method: GET 2023-05-31T20:27:22.892 DEBUG:boto:Path: /pklxkd-55/c?versionId=null 2023-05-31T20:27:22.892 DEBUG:boto:Data: 2023-05-31T20:27:22.892 DEBUG:boto:Headers: {} 2023-05-31T20:27:22.893 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:27:22.893 DEBUG:boto:Port: 8001 2023-05-31T20:27:22.894 DEBUG:boto:Params: {} 2023-05-31T20:27:22.894 DEBUG:boto:Token: None 2023-05-31T20:27:22.894 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:27:22 GMT /pklxkd-55/c?versionId=null 2023-05-31T20:27:22.895 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:Axcde+P5vBM6RZWi3eIHC7wJjlI= 2023-05-31T20:27:22.895 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:27:22 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:Axcde+P5vBM6RZWi3eIHC7wJjlI=', 'Content-Length': '0'} 2023-05-31T20:27:22.897 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:27:16 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000390541f92c487c68-006477adaa-162e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:27:22 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:27:22.898 DEBUG:boto:path=/pklxkd-55/c 2023-05-31T20:27:22.898 DEBUG:boto:auth_path=/pklxkd-55/c 2023-05-31T20:27:22.899 DEBUG:boto:path=/pklxkd-55/c?versionId=null 2023-05-31T20:27:22.899 DEBUG:boto:auth_path=/pklxkd-55/c?versionId=null 2023-05-31T20:27:22.900 DEBUG:boto:Method: GET 2023-05-31T20:27:22.900 DEBUG:boto:Path: /pklxkd-55/c?versionId=null 2023-05-31T20:27:22.900 DEBUG:boto:Data: 2023-05-31T20:27:22.901 DEBUG:boto:Headers: {} 2023-05-31T20:27:22.901 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:27:22.901 DEBUG:boto:Port: 8000 2023-05-31T20:27:22.902 DEBUG:boto:Params: {} 2023-05-31T20:27:22.902 DEBUG:boto:Token: None 2023-05-31T20:27:22.903 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:27:22 GMT /pklxkd-55/c?versionId=null 2023-05-31T20:27:22.903 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:Axcde+P5vBM6RZWi3eIHC7wJjlI= 2023-05-31T20:27:22.904 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:27:22 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:Axcde+P5vBM6RZWi3eIHC7wJjlI=', 'Content-Length': '0'} 2023-05-31T20:27:22.906 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:27:16 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006054613406236015-006477adaa-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:27:22 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:27:22.906 DEBUG:boto:path=/pklxkd-55/c 2023-05-31T20:27:22.907 DEBUG:boto:auth_path=/pklxkd-55/c 2023-05-31T20:27:22.907 DEBUG:boto:path=/pklxkd-55/c?versionId=null 2023-05-31T20:27:22.907 DEBUG:boto:auth_path=/pklxkd-55/c?versionId=null 2023-05-31T20:27:22.908 DEBUG:boto:Method: HEAD 2023-05-31T20:27:22.908 DEBUG:boto:Path: /pklxkd-55/c?versionId=null 2023-05-31T20:27:22.908 DEBUG:boto:Data: 2023-05-31T20:27:22.909 DEBUG:boto:Headers: {} 2023-05-31T20:27:22.909 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:27:22.910 DEBUG:boto:Port: 8001 2023-05-31T20:27:22.910 DEBUG:boto:Params: {} 2023-05-31T20:27:22.910 DEBUG:boto:Token: None 2023-05-31T20:27:22.911 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:27:22 GMT /pklxkd-55/c?versionId=null 2023-05-31T20:27:22.911 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:c1OaLJXgCJtqsCxdKj3F4riZ4YE= 2023-05-31T20:27:22.912 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:27:22 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:c1OaLJXgCJtqsCxdKj3F4riZ4YE=', 'Content-Length': '0'} 2023-05-31T20:27:22.913 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:27:16 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004665bdfd69a2f295-006477adaa-162e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:27:22 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:27:22.914 DEBUG:boto:path=/pklxkd-55/c 2023-05-31T20:27:22.914 DEBUG:boto:auth_path=/pklxkd-55/c 2023-05-31T20:27:22.914 DEBUG:boto:path=/pklxkd-55/c?versionId=null 2023-05-31T20:27:22.915 DEBUG:boto:auth_path=/pklxkd-55/c?versionId=null 2023-05-31T20:27:22.915 DEBUG:boto:Method: HEAD 2023-05-31T20:27:22.915 DEBUG:boto:Path: /pklxkd-55/c?versionId=null 2023-05-31T20:27:22.916 DEBUG:boto:Data: 2023-05-31T20:27:22.916 DEBUG:boto:Headers: {} 2023-05-31T20:27:22.916 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:27:22.917 DEBUG:boto:Port: 8000 2023-05-31T20:27:22.917 DEBUG:boto:Params: {} 2023-05-31T20:27:22.917 DEBUG:boto:Token: None 2023-05-31T20:27:22.918 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:27:22 GMT /pklxkd-55/c?versionId=null 2023-05-31T20:27:22.918 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:c1OaLJXgCJtqsCxdKj3F4riZ4YE= 2023-05-31T20:27:22.919 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:27:22 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:c1OaLJXgCJtqsCxdKj3F4riZ4YE=', 'Content-Length': '0'} 2023-05-31T20:27:22.921 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:27:16 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000052cf79f09b60ac8e-006477adaa-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:27:22 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:27:22.921 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=c 2023-05-31T20:27:22.922 DEBUG:boto:path=/pklxkd-55/c 2023-05-31T20:27:22.922 DEBUG:boto:auth_path=/pklxkd-55/c 2023-05-31T20:27:22.922 DEBUG:boto:path=/pklxkd-55/c?versionId=null 2023-05-31T20:27:22.923 DEBUG:boto:auth_path=/pklxkd-55/c?versionId=null 2023-05-31T20:27:22.923 DEBUG:boto:Method: GET 2023-05-31T20:27:22.923 DEBUG:boto:Path: /pklxkd-55/c?versionId=null 2023-05-31T20:27:22.924 DEBUG:boto:Data: 2023-05-31T20:27:22.924 DEBUG:boto:Headers: {} 2023-05-31T20:27:22.925 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:27:22.925 DEBUG:boto:Port: 8001 2023-05-31T20:27:22.925 DEBUG:boto:Params: {} 2023-05-31T20:27:22.926 DEBUG:boto:Token: None 2023-05-31T20:27:22.926 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:27:22 GMT /pklxkd-55/c?versionId=null 2023-05-31T20:27:22.927 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:Axcde+P5vBM6RZWi3eIHC7wJjlI= 2023-05-31T20:27:22.927 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:27:22 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:Axcde+P5vBM6RZWi3eIHC7wJjlI=', 'Content-Length': '0'} 2023-05-31T20:27:22.929 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:27:16 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005cd40993d632e698-006477adaa-162e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:27:22 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:27:22.930 DEBUG:boto:path=/pklxkd-55/c 2023-05-31T20:27:22.930 DEBUG:boto:auth_path=/pklxkd-55/c 2023-05-31T20:27:22.930 DEBUG:boto:path=/pklxkd-55/c?versionId=null 2023-05-31T20:27:22.931 DEBUG:boto:auth_path=/pklxkd-55/c?versionId=null 2023-05-31T20:27:22.931 DEBUG:boto:Method: GET 2023-05-31T20:27:22.932 DEBUG:boto:Path: /pklxkd-55/c?versionId=null 2023-05-31T20:27:22.932 DEBUG:boto:Data: 2023-05-31T20:27:22.932 DEBUG:boto:Headers: {} 2023-05-31T20:27:22.933 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:27:22.933 DEBUG:boto:Port: 8000 2023-05-31T20:27:22.933 DEBUG:boto:Params: {} 2023-05-31T20:27:22.934 DEBUG:boto:Token: None 2023-05-31T20:27:22.934 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:27:22 GMT /pklxkd-55/c?versionId=null 2023-05-31T20:27:22.935 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:Axcde+P5vBM6RZWi3eIHC7wJjlI= 2023-05-31T20:27:22.935 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:27:22 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:Axcde+P5vBM6RZWi3eIHC7wJjlI=', 'Content-Length': '0'} 2023-05-31T20:27:22.937 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:27:16 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000094e82baf5979d383-006477adaa-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:27:22 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:27:22.937 DEBUG:boto:path=/pklxkd-55/ 2023-05-31T20:27:22.938 DEBUG:boto:auth_path=/pklxkd-55/ 2023-05-31T20:27:22.938 DEBUG:boto:path=/pklxkd-55/?prefix=c 2023-05-31T20:27:22.939 DEBUG:boto:auth_path=/pklxkd-55/?prefix=c 2023-05-31T20:27:22.939 DEBUG:boto:Method: GET 2023-05-31T20:27:22.939 DEBUG:boto:Path: /pklxkd-55/?prefix=c 2023-05-31T20:27:22.940 DEBUG:boto:Data: 2023-05-31T20:27:22.940 DEBUG:boto:Headers: {} 2023-05-31T20:27:22.940 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:27:22.941 DEBUG:boto:Port: 8001 2023-05-31T20:27:22.941 DEBUG:boto:Params: {} 2023-05-31T20:27:22.942 DEBUG:boto:Token: None 2023-05-31T20:27:22.942 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:27:22 GMT /pklxkd-55/ 2023-05-31T20:27:22.942 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:p2p+8G1ZdPifmzf+JcgpB78h0o8= 2023-05-31T20:27:22.943 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:27:22 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:p2p+8G1ZdPifmzf+JcgpB78h0o8=', 'Content-Length': '0'} 2023-05-31T20:27:22.945 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000075a35bcf99357963-006477adaa-162e-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:27:22 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:27:22.945 DEBUG:boto:b'pklxkd-55c1000falsec2023-05-31T20:27:16.869Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:27:22.946 DEBUG:boto:path=/pklxkd-55/ 2023-05-31T20:27:22.947 DEBUG:boto:auth_path=/pklxkd-55/ 2023-05-31T20:27:22.947 DEBUG:boto:path=/pklxkd-55/?prefix=c 2023-05-31T20:27:22.947 DEBUG:boto:auth_path=/pklxkd-55/?prefix=c 2023-05-31T20:27:22.948 DEBUG:boto:Method: GET 2023-05-31T20:27:22.948 DEBUG:boto:Path: /pklxkd-55/?prefix=c 2023-05-31T20:27:22.948 DEBUG:boto:Data: 2023-05-31T20:27:22.949 DEBUG:boto:Headers: {} 2023-05-31T20:27:22.949 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:27:22.949 DEBUG:boto:Port: 8000 2023-05-31T20:27:22.950 DEBUG:boto:Params: {} 2023-05-31T20:27:22.950 DEBUG:boto:Token: None 2023-05-31T20:27:22.951 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:27:22 GMT /pklxkd-55/ 2023-05-31T20:27:22.951 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:p2p+8G1ZdPifmzf+JcgpB78h0o8= 2023-05-31T20:27:22.951 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:27:22 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:p2p+8G1ZdPifmzf+JcgpB78h0o8=', 'Content-Length': '0'} 2023-05-31T20:27:22.953 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000004ec23211a3c2c4f0-006477adaa-1630-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:27:22 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:27:22.996 DEBUG:boto:b'pklxkd-55c1000falsec2023-05-31T20:27:16.869Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:27:22.997 DEBUG:boto:path=/pklxkd-55/c 2023-05-31T20:27:22.998 DEBUG:boto:auth_path=/pklxkd-55/c 2023-05-31T20:27:22.998 DEBUG:boto:Method: HEAD 2023-05-31T20:27:22.999 DEBUG:boto:Path: /pklxkd-55/c 2023-05-31T20:27:22.999 DEBUG:boto:Data: 2023-05-31T20:27:22.999 DEBUG:boto:Headers: {} 2023-05-31T20:27:23.000 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:27:23.000 DEBUG:boto:Port: 8001 2023-05-31T20:27:23.000 DEBUG:boto:Params: {} 2023-05-31T20:27:23.001 DEBUG:boto:Token: None 2023-05-31T20:27:23.001 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:27:23 GMT /pklxkd-55/c 2023-05-31T20:27:23.002 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:hE/j1mXzS//fRGUA9sEWdmEaUWA= 2023-05-31T20:27:23.002 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:27:23 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:hE/j1mXzS//fRGUA9sEWdmEaUWA=', 'Content-Length': '0'} 2023-05-31T20:27:23.004 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:27:16 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d2f715476b53260d-006477adaa-162e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:27:22 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:27:23.005 DEBUG:boto:path=/pklxkd-55/c 2023-05-31T20:27:23.005 DEBUG:boto:auth_path=/pklxkd-55/c 2023-05-31T20:27:23.006 DEBUG:boto:Method: HEAD 2023-05-31T20:27:23.006 DEBUG:boto:Path: /pklxkd-55/c 2023-05-31T20:27:23.007 DEBUG:boto:Data: 2023-05-31T20:27:23.007 DEBUG:boto:Headers: {} 2023-05-31T20:27:23.007 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:27:23.008 DEBUG:boto:Port: 8000 2023-05-31T20:27:23.008 DEBUG:boto:Params: {} 2023-05-31T20:27:23.008 DEBUG:boto:Token: None 2023-05-31T20:27:23.009 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:27:23 GMT /pklxkd-55/c 2023-05-31T20:27:23.009 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:hE/j1mXzS//fRGUA9sEWdmEaUWA= 2023-05-31T20:27:23.010 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:27:23 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:hE/j1mXzS//fRGUA9sEWdmEaUWA=', 'Content-Length': '0'} 2023-05-31T20:27:23.011 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:27:16 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000298f60bfe1f4e56d-006477adab-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:27:23 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:27:23.012 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=c 2023-05-31T20:27:23.012 DEBUG:boto:path=/pklxkd-55/c 2023-05-31T20:27:23.013 DEBUG:boto:auth_path=/pklxkd-55/c 2023-05-31T20:27:23.013 DEBUG:boto:Method: GET 2023-05-31T20:27:23.013 DEBUG:boto:Path: /pklxkd-55/c 2023-05-31T20:27:23.014 DEBUG:boto:Data: 2023-05-31T20:27:23.014 DEBUG:boto:Headers: {} 2023-05-31T20:27:23.014 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:27:23.015 DEBUG:boto:Port: 8001 2023-05-31T20:27:23.015 DEBUG:boto:Params: {} 2023-05-31T20:27:23.016 DEBUG:boto:Token: None 2023-05-31T20:27:23.016 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:27:23 GMT /pklxkd-55/c 2023-05-31T20:27:23.017 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:OLZbK6QwcjGMgpIZWtkjzEEZDJ4= 2023-05-31T20:27:23.017 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:27:23 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:OLZbK6QwcjGMgpIZWtkjzEEZDJ4=', 'Content-Length': '0'} 2023-05-31T20:27:23.019 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:27:16 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000015b3e9865681ac4a-006477adab-162e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:27:23 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:27:23.060 DEBUG:boto:path=/pklxkd-55/c 2023-05-31T20:27:23.061 DEBUG:boto:auth_path=/pklxkd-55/c 2023-05-31T20:27:23.061 DEBUG:boto:Method: GET 2023-05-31T20:27:23.062 DEBUG:boto:Path: /pklxkd-55/c 2023-05-31T20:27:23.062 DEBUG:boto:Data: 2023-05-31T20:27:23.063 DEBUG:boto:Headers: {} 2023-05-31T20:27:23.063 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:27:23.063 DEBUG:boto:Port: 8000 2023-05-31T20:27:23.064 DEBUG:boto:Params: {} 2023-05-31T20:27:23.064 DEBUG:boto:Token: None 2023-05-31T20:27:23.065 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:27:23 GMT /pklxkd-55/c 2023-05-31T20:27:23.065 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:OLZbK6QwcjGMgpIZWtkjzEEZDJ4= 2023-05-31T20:27:23.066 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:27:23 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:OLZbK6QwcjGMgpIZWtkjzEEZDJ4=', 'Content-Length': '0'} 2023-05-31T20:27:23.067 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:27:16 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000dce80df79a3d64c5-006477adab-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:27:23 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:27:23.068 DEBUG:boto:path=/pklxkd-55/ 2023-05-31T20:27:23.069 DEBUG:boto:auth_path=/pklxkd-55/ 2023-05-31T20:27:23.069 DEBUG:boto:path=/pklxkd-55/?prefix=c 2023-05-31T20:27:23.069 DEBUG:boto:auth_path=/pklxkd-55/?prefix=c 2023-05-31T20:27:23.070 DEBUG:boto:Method: GET 2023-05-31T20:27:23.070 DEBUG:boto:Path: /pklxkd-55/?prefix=c 2023-05-31T20:27:23.070 DEBUG:boto:Data: 2023-05-31T20:27:23.071 DEBUG:boto:Headers: {} 2023-05-31T20:27:23.071 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:27:23.072 DEBUG:boto:Port: 8001 2023-05-31T20:27:23.072 DEBUG:boto:Params: {} 2023-05-31T20:27:23.072 DEBUG:boto:Token: None 2023-05-31T20:27:23.073 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:27:23 GMT /pklxkd-55/ 2023-05-31T20:27:23.073 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:h67TlTI878JFBrep/coHQ2+SsZE= 2023-05-31T20:27:23.074 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:27:23 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:h67TlTI878JFBrep/coHQ2+SsZE=', 'Content-Length': '0'} 2023-05-31T20:27:23.076 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c1c337f5e81424b3-006477adab-162e-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:27:23 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:27:23.076 DEBUG:boto:b'pklxkd-55c1000falsec2023-05-31T20:27:16.869Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:27:23.077 DEBUG:boto:path=/pklxkd-55/ 2023-05-31T20:27:23.077 DEBUG:boto:auth_path=/pklxkd-55/ 2023-05-31T20:27:23.078 DEBUG:boto:path=/pklxkd-55/?prefix=c 2023-05-31T20:27:23.078 DEBUG:boto:auth_path=/pklxkd-55/?prefix=c 2023-05-31T20:27:23.078 DEBUG:boto:Method: GET 2023-05-31T20:27:23.079 DEBUG:boto:Path: /pklxkd-55/?prefix=c 2023-05-31T20:27:23.079 DEBUG:boto:Data: 2023-05-31T20:27:23.080 DEBUG:boto:Headers: {} 2023-05-31T20:27:23.080 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:27:23.080 DEBUG:boto:Port: 8000 2023-05-31T20:27:23.081 DEBUG:boto:Params: {} 2023-05-31T20:27:23.081 DEBUG:boto:Token: None 2023-05-31T20:27:23.081 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:27:23 GMT /pklxkd-55/ 2023-05-31T20:27:23.082 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:h67TlTI878JFBrep/coHQ2+SsZE= 2023-05-31T20:27:23.082 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:27:23 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:h67TlTI878JFBrep/coHQ2+SsZE=', 'Content-Length': '0'} 2023-05-31T20:27:23.085 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000006c91edda8ee74844-006477adab-1630-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:27:23 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:27:23.085 DEBUG:boto:b'pklxkd-55c1000falsec2023-05-31T20:27:16.869Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:27:23.086 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=d 2023-05-31T20:27:23.087 DEBUG:boto:path=/pklxkd-55/d 2023-05-31T20:27:23.087 DEBUG:boto:auth_path=/pklxkd-55/d 2023-05-31T20:27:23.087 DEBUG:boto:path=/pklxkd-55/d?versionId=null 2023-05-31T20:27:23.088 DEBUG:boto:auth_path=/pklxkd-55/d?versionId=null 2023-05-31T20:27:23.088 DEBUG:boto:Method: GET 2023-05-31T20:27:23.088 DEBUG:boto:Path: /pklxkd-55/d?versionId=null 2023-05-31T20:27:23.089 DEBUG:boto:Data: 2023-05-31T20:27:23.089 DEBUG:boto:Headers: {} 2023-05-31T20:27:23.090 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:27:23.090 DEBUG:boto:Port: 8001 2023-05-31T20:27:23.090 DEBUG:boto:Params: {} 2023-05-31T20:27:23.091 DEBUG:boto:Token: None 2023-05-31T20:27:23.091 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:27:23 GMT /pklxkd-55/d?versionId=null 2023-05-31T20:27:23.091 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:twzFSNx7ajueQ8o8KAKeg8K/Oa4= 2023-05-31T20:27:23.092 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:27:23 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:twzFSNx7ajueQ8o8KAKeg8K/Oa4=', 'Content-Length': '0'} 2023-05-31T20:27:23.094 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:27:16 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e401a2203e72f942-006477adab-162e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:27:23 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:27:23.094 DEBUG:boto:path=/pklxkd-55/d 2023-05-31T20:27:23.095 DEBUG:boto:auth_path=/pklxkd-55/d 2023-05-31T20:27:23.095 DEBUG:boto:path=/pklxkd-55/d?versionId=null 2023-05-31T20:27:23.095 DEBUG:boto:auth_path=/pklxkd-55/d?versionId=null 2023-05-31T20:27:23.096 DEBUG:boto:Method: GET 2023-05-31T20:27:23.096 DEBUG:boto:Path: /pklxkd-55/d?versionId=null 2023-05-31T20:27:23.097 DEBUG:boto:Data: 2023-05-31T20:27:23.097 DEBUG:boto:Headers: {} 2023-05-31T20:27:23.097 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:27:23.098 DEBUG:boto:Port: 8000 2023-05-31T20:27:23.098 DEBUG:boto:Params: {} 2023-05-31T20:27:23.099 DEBUG:boto:Token: None 2023-05-31T20:27:23.099 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:27:23 GMT /pklxkd-55/d?versionId=null 2023-05-31T20:27:23.099 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:twzFSNx7ajueQ8o8KAKeg8K/Oa4= 2023-05-31T20:27:23.100 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:27:23 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:twzFSNx7ajueQ8o8KAKeg8K/Oa4=', 'Content-Length': '0'} 2023-05-31T20:27:23.101 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:27:16 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000067f445248bff5cec-006477adab-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:27:23 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:27:23.102 DEBUG:boto:path=/pklxkd-55/d 2023-05-31T20:27:23.102 DEBUG:boto:auth_path=/pklxkd-55/d 2023-05-31T20:27:23.103 DEBUG:boto:path=/pklxkd-55/d?versionId=null 2023-05-31T20:27:23.103 DEBUG:boto:auth_path=/pklxkd-55/d?versionId=null 2023-05-31T20:27:23.104 DEBUG:boto:Method: HEAD 2023-05-31T20:27:23.104 DEBUG:boto:Path: /pklxkd-55/d?versionId=null 2023-05-31T20:27:23.104 DEBUG:boto:Data: 2023-05-31T20:27:23.105 DEBUG:boto:Headers: {} 2023-05-31T20:27:23.105 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:27:23.105 DEBUG:boto:Port: 8001 2023-05-31T20:27:23.106 DEBUG:boto:Params: {} 2023-05-31T20:27:23.106 DEBUG:boto:Token: None 2023-05-31T20:27:23.106 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:27:23 GMT /pklxkd-55/d?versionId=null 2023-05-31T20:27:23.107 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:cA98pb1KdBFha1CdV/G2h7uc3nw= 2023-05-31T20:27:23.107 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:27:23 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:cA98pb1KdBFha1CdV/G2h7uc3nw=', 'Content-Length': '0'} 2023-05-31T20:27:23.109 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:27:16 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c7c0b4df5a45c2d7-006477adab-162e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:27:23 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:27:23.110 DEBUG:boto:path=/pklxkd-55/d 2023-05-31T20:27:23.110 DEBUG:boto:auth_path=/pklxkd-55/d 2023-05-31T20:27:23.110 DEBUG:boto:path=/pklxkd-55/d?versionId=null 2023-05-31T20:27:23.111 DEBUG:boto:auth_path=/pklxkd-55/d?versionId=null 2023-05-31T20:27:23.111 DEBUG:boto:Method: HEAD 2023-05-31T20:27:23.111 DEBUG:boto:Path: /pklxkd-55/d?versionId=null 2023-05-31T20:27:23.112 DEBUG:boto:Data: 2023-05-31T20:27:23.112 DEBUG:boto:Headers: {} 2023-05-31T20:27:23.112 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:27:23.113 DEBUG:boto:Port: 8000 2023-05-31T20:27:23.113 DEBUG:boto:Params: {} 2023-05-31T20:27:23.113 DEBUG:boto:Token: None 2023-05-31T20:27:23.114 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:27:23 GMT /pklxkd-55/d?versionId=null 2023-05-31T20:27:23.114 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:cA98pb1KdBFha1CdV/G2h7uc3nw= 2023-05-31T20:27:23.115 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:27:23 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:cA98pb1KdBFha1CdV/G2h7uc3nw=', 'Content-Length': '0'} 2023-05-31T20:27:23.116 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:27:16 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ed938085a76b3cea-006477adab-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:27:23 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:27:23.117 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=d 2023-05-31T20:27:23.117 DEBUG:boto:path=/pklxkd-55/d 2023-05-31T20:27:23.118 DEBUG:boto:auth_path=/pklxkd-55/d 2023-05-31T20:27:23.118 DEBUG:boto:path=/pklxkd-55/d?versionId=null 2023-05-31T20:27:23.118 DEBUG:boto:auth_path=/pklxkd-55/d?versionId=null 2023-05-31T20:27:23.119 DEBUG:boto:Method: GET 2023-05-31T20:27:23.119 DEBUG:boto:Path: /pklxkd-55/d?versionId=null 2023-05-31T20:27:23.119 DEBUG:boto:Data: 2023-05-31T20:27:23.120 DEBUG:boto:Headers: {} 2023-05-31T20:27:23.120 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:27:23.121 DEBUG:boto:Port: 8001 2023-05-31T20:27:23.121 DEBUG:boto:Params: {} 2023-05-31T20:27:23.121 DEBUG:boto:Token: None 2023-05-31T20:27:23.122 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:27:23 GMT /pklxkd-55/d?versionId=null 2023-05-31T20:27:23.122 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:twzFSNx7ajueQ8o8KAKeg8K/Oa4= 2023-05-31T20:27:23.123 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:27:23 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:twzFSNx7ajueQ8o8KAKeg8K/Oa4=', 'Content-Length': '0'} 2023-05-31T20:27:23.125 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:27:16 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006a74cbfd8a276537-006477adab-162e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:27:23 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:27:23.126 DEBUG:boto:path=/pklxkd-55/d 2023-05-31T20:27:23.126 DEBUG:boto:auth_path=/pklxkd-55/d 2023-05-31T20:27:23.126 DEBUG:boto:path=/pklxkd-55/d?versionId=null 2023-05-31T20:27:23.127 DEBUG:boto:auth_path=/pklxkd-55/d?versionId=null 2023-05-31T20:27:23.127 DEBUG:boto:Method: GET 2023-05-31T20:27:23.128 DEBUG:boto:Path: /pklxkd-55/d?versionId=null 2023-05-31T20:27:23.128 DEBUG:boto:Data: 2023-05-31T20:27:23.128 DEBUG:boto:Headers: {} 2023-05-31T20:27:23.129 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:27:23.129 DEBUG:boto:Port: 8000 2023-05-31T20:27:23.130 DEBUG:boto:Params: {} 2023-05-31T20:27:23.130 DEBUG:boto:Token: None 2023-05-31T20:27:23.131 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:27:23 GMT /pklxkd-55/d?versionId=null 2023-05-31T20:27:23.131 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:twzFSNx7ajueQ8o8KAKeg8K/Oa4= 2023-05-31T20:27:23.132 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:27:23 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:twzFSNx7ajueQ8o8KAKeg8K/Oa4=', 'Content-Length': '0'} 2023-05-31T20:27:23.133 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:27:16 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006c78464ce6bf70c3-006477adab-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:27:23 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:27:23.176 DEBUG:boto:path=/pklxkd-55/ 2023-05-31T20:27:23.177 DEBUG:boto:auth_path=/pklxkd-55/ 2023-05-31T20:27:23.177 DEBUG:boto:path=/pklxkd-55/?prefix=d 2023-05-31T20:27:23.178 DEBUG:boto:auth_path=/pklxkd-55/?prefix=d 2023-05-31T20:27:23.178 DEBUG:boto:Method: GET 2023-05-31T20:27:23.179 DEBUG:boto:Path: /pklxkd-55/?prefix=d 2023-05-31T20:27:23.179 DEBUG:boto:Data: 2023-05-31T20:27:23.179 DEBUG:boto:Headers: {} 2023-05-31T20:27:23.180 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:27:23.180 DEBUG:boto:Port: 8001 2023-05-31T20:27:23.181 DEBUG:boto:Params: {} 2023-05-31T20:27:23.181 DEBUG:boto:Token: None 2023-05-31T20:27:23.181 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:27:23 GMT /pklxkd-55/ 2023-05-31T20:27:23.182 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:h67TlTI878JFBrep/coHQ2+SsZE= 2023-05-31T20:27:23.182 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:27:23 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:h67TlTI878JFBrep/coHQ2+SsZE=', 'Content-Length': '0'} 2023-05-31T20:27:23.184 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e29636335d2b7e4b-006477adab-162e-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:27:23 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:27:23.185 DEBUG:boto:b'pklxkd-55d1000falsed2023-05-31T20:27:16.893Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:27:23.186 DEBUG:boto:path=/pklxkd-55/ 2023-05-31T20:27:23.186 DEBUG:boto:auth_path=/pklxkd-55/ 2023-05-31T20:27:23.186 DEBUG:boto:path=/pklxkd-55/?prefix=d 2023-05-31T20:27:23.187 DEBUG:boto:auth_path=/pklxkd-55/?prefix=d 2023-05-31T20:27:23.187 DEBUG:boto:Method: GET 2023-05-31T20:27:23.188 DEBUG:boto:Path: /pklxkd-55/?prefix=d 2023-05-31T20:27:23.188 DEBUG:boto:Data: 2023-05-31T20:27:23.188 DEBUG:boto:Headers: {} 2023-05-31T20:27:23.189 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:27:23.189 DEBUG:boto:Port: 8000 2023-05-31T20:27:23.189 DEBUG:boto:Params: {} 2023-05-31T20:27:23.190 DEBUG:boto:Token: None 2023-05-31T20:27:23.190 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:27:23 GMT /pklxkd-55/ 2023-05-31T20:27:23.191 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:h67TlTI878JFBrep/coHQ2+SsZE= 2023-05-31T20:27:23.191 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:27:23 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:h67TlTI878JFBrep/coHQ2+SsZE=', 'Content-Length': '0'} 2023-05-31T20:27:23.193 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f2333e093b9295d1-006477adab-1630-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:27:23 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:27:23.194 DEBUG:boto:b'pklxkd-55d1000falsed2023-05-31T20:27:16.893Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:27:23.195 DEBUG:boto:path=/pklxkd-55/d 2023-05-31T20:27:23.195 DEBUG:boto:auth_path=/pklxkd-55/d 2023-05-31T20:27:23.195 DEBUG:boto:Method: HEAD 2023-05-31T20:27:23.196 DEBUG:boto:Path: /pklxkd-55/d 2023-05-31T20:27:23.196 DEBUG:boto:Data: 2023-05-31T20:27:23.196 DEBUG:boto:Headers: {} 2023-05-31T20:27:23.197 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:27:23.197 DEBUG:boto:Port: 8001 2023-05-31T20:27:23.197 DEBUG:boto:Params: {} 2023-05-31T20:27:23.198 DEBUG:boto:Token: None 2023-05-31T20:27:23.198 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:27:23 GMT /pklxkd-55/d 2023-05-31T20:27:23.199 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:ixht+ZZRHyFfTSXd83JA6A3PTzA= 2023-05-31T20:27:23.199 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:27:23 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:ixht+ZZRHyFfTSXd83JA6A3PTzA=', 'Content-Length': '0'} 2023-05-31T20:27:23.201 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:27:16 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000cb540719c5ef7aeb-006477adab-162e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:27:23 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:27:23.201 DEBUG:boto:path=/pklxkd-55/d 2023-05-31T20:27:23.202 DEBUG:boto:auth_path=/pklxkd-55/d 2023-05-31T20:27:23.202 DEBUG:boto:Method: HEAD 2023-05-31T20:27:23.203 DEBUG:boto:Path: /pklxkd-55/d 2023-05-31T20:27:23.203 DEBUG:boto:Data: 2023-05-31T20:27:23.203 DEBUG:boto:Headers: {} 2023-05-31T20:27:23.204 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:27:23.204 DEBUG:boto:Port: 8000 2023-05-31T20:27:23.204 DEBUG:boto:Params: {} 2023-05-31T20:27:23.205 DEBUG:boto:Token: None 2023-05-31T20:27:23.205 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:27:23 GMT /pklxkd-55/d 2023-05-31T20:27:23.205 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:ixht+ZZRHyFfTSXd83JA6A3PTzA= 2023-05-31T20:27:23.206 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:27:23 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:ixht+ZZRHyFfTSXd83JA6A3PTzA=', 'Content-Length': '0'} 2023-05-31T20:27:23.207 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:27:16 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006ec1bf2928d4f6b3-006477adab-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:27:23 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:27:23.208 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=d 2023-05-31T20:27:23.208 DEBUG:boto:path=/pklxkd-55/d 2023-05-31T20:27:23.209 DEBUG:boto:auth_path=/pklxkd-55/d 2023-05-31T20:27:23.209 DEBUG:boto:Method: GET 2023-05-31T20:27:23.209 DEBUG:boto:Path: /pklxkd-55/d 2023-05-31T20:27:23.210 DEBUG:boto:Data: 2023-05-31T20:27:23.210 DEBUG:boto:Headers: {} 2023-05-31T20:27:23.211 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:27:23.211 DEBUG:boto:Port: 8001 2023-05-31T20:27:23.211 DEBUG:boto:Params: {} 2023-05-31T20:27:23.212 DEBUG:boto:Token: None 2023-05-31T20:27:23.212 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:27:23 GMT /pklxkd-55/d 2023-05-31T20:27:23.213 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:DjRBoypgizW522NCxCdGLAVZDyI= 2023-05-31T20:27:23.213 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:27:23 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:DjRBoypgizW522NCxCdGLAVZDyI=', 'Content-Length': '0'} 2023-05-31T20:27:23.215 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:27:16 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a537be9948a819e2-006477adab-162e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:27:23 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:27:23.256 DEBUG:boto:path=/pklxkd-55/d 2023-05-31T20:27:23.257 DEBUG:boto:auth_path=/pklxkd-55/d 2023-05-31T20:27:23.257 DEBUG:boto:Method: GET 2023-05-31T20:27:23.258 DEBUG:boto:Path: /pklxkd-55/d 2023-05-31T20:27:23.259 DEBUG:boto:Data: 2023-05-31T20:27:23.259 DEBUG:boto:Headers: {} 2023-05-31T20:27:23.260 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:27:23.260 DEBUG:boto:Port: 8000 2023-05-31T20:27:23.260 DEBUG:boto:Params: {} 2023-05-31T20:27:23.261 DEBUG:boto:Token: None 2023-05-31T20:27:23.261 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:27:23 GMT /pklxkd-55/d 2023-05-31T20:27:23.262 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:DjRBoypgizW522NCxCdGLAVZDyI= 2023-05-31T20:27:23.262 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:27:23 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:DjRBoypgizW522NCxCdGLAVZDyI=', 'Content-Length': '0'} 2023-05-31T20:27:23.264 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:27:16 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000946627bd2bdf9764-006477adab-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:27:23 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:27:23.264 DEBUG:boto:path=/pklxkd-55/ 2023-05-31T20:27:23.265 DEBUG:boto:auth_path=/pklxkd-55/ 2023-05-31T20:27:23.265 DEBUG:boto:path=/pklxkd-55/?prefix=d 2023-05-31T20:27:23.265 DEBUG:boto:auth_path=/pklxkd-55/?prefix=d 2023-05-31T20:27:23.266 DEBUG:boto:Method: GET 2023-05-31T20:27:23.266 DEBUG:boto:Path: /pklxkd-55/?prefix=d 2023-05-31T20:27:23.267 DEBUG:boto:Data: 2023-05-31T20:27:23.267 DEBUG:boto:Headers: {} 2023-05-31T20:27:23.268 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:27:23.268 DEBUG:boto:Port: 8001 2023-05-31T20:27:23.268 DEBUG:boto:Params: {} 2023-05-31T20:27:23.269 DEBUG:boto:Token: None 2023-05-31T20:27:23.269 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:27:23 GMT /pklxkd-55/ 2023-05-31T20:27:23.271 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:h67TlTI878JFBrep/coHQ2+SsZE= 2023-05-31T20:27:23.271 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:27:23 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:h67TlTI878JFBrep/coHQ2+SsZE=', 'Content-Length': '0'} 2023-05-31T20:27:23.273 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000003b622a3165f84704-006477adab-162e-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:27:23 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:27:23.273 DEBUG:boto:b'pklxkd-55d1000falsed2023-05-31T20:27:16.893Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:27:23.274 DEBUG:boto:path=/pklxkd-55/ 2023-05-31T20:27:23.275 DEBUG:boto:auth_path=/pklxkd-55/ 2023-05-31T20:27:23.275 DEBUG:boto:path=/pklxkd-55/?prefix=d 2023-05-31T20:27:23.275 DEBUG:boto:auth_path=/pklxkd-55/?prefix=d 2023-05-31T20:27:23.276 DEBUG:boto:Method: GET 2023-05-31T20:27:23.276 DEBUG:boto:Path: /pklxkd-55/?prefix=d 2023-05-31T20:27:23.276 DEBUG:boto:Data: 2023-05-31T20:27:23.277 DEBUG:boto:Headers: {} 2023-05-31T20:27:23.277 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:27:23.278 DEBUG:boto:Port: 8000 2023-05-31T20:27:23.278 DEBUG:boto:Params: {} 2023-05-31T20:27:23.278 DEBUG:boto:Token: None 2023-05-31T20:27:23.279 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:27:23 GMT /pklxkd-55/ 2023-05-31T20:27:23.279 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:h67TlTI878JFBrep/coHQ2+SsZE= 2023-05-31T20:27:23.280 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:27:23 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:h67TlTI878JFBrep/coHQ2+SsZE=', 'Content-Length': '0'} 2023-05-31T20:27:23.282 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000fe3a7becea55f06a-006477adab-1630-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:27:23 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:27:23.282 DEBUG:boto:b'pklxkd-55d1000falsed2023-05-31T20:27:16.893Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:27:23.283 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=pklxkd-55 zones={a2, a1} 2023-05-31T20:27:23.283 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'layout', '--bucket', 'pklxkd-55', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-05-31T20:27:23.284 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'layout', '--bucket', 'pklxkd-55', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-05-31T20:27:23.284 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket layout --bucket pklxkd-55 --cluster c1 --debug-rgw 0 --debug-ms 0 2023-05-31T20:27:23.332 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2023-05-31T20:27:23.332 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2023-05-31T20:27:23.396 INFO:teuthology.orchestra.run.smithi099.stdout:{ 2023-05-31T20:27:23.397 INFO:teuthology.orchestra.run.smithi099.stdout: "layout": { 2023-05-31T20:27:23.397 INFO:teuthology.orchestra.run.smithi099.stdout: "resharding": "None", 2023-05-31T20:27:23.397 INFO:teuthology.orchestra.run.smithi099.stdout: "current_index": { 2023-05-31T20:27:23.397 INFO:teuthology.orchestra.run.smithi099.stdout: "gen": 0, 2023-05-31T20:27:23.397 INFO:teuthology.orchestra.run.smithi099.stdout: "layout": { 2023-05-31T20:27:23.398 INFO:teuthology.orchestra.run.smithi099.stdout: "type": "Normal", 2023-05-31T20:27:23.398 INFO:teuthology.orchestra.run.smithi099.stdout: "normal": { 2023-05-31T20:27:23.398 INFO:teuthology.orchestra.run.smithi099.stdout: "num_shards": 11, 2023-05-31T20:27:23.398 INFO:teuthology.orchestra.run.smithi099.stdout: "hash_type": "Mod" 2023-05-31T20:27:23.398 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T20:27:23.398 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T20:27:23.398 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:27:23.399 INFO:teuthology.orchestra.run.smithi099.stdout: "logs": [ 2023-05-31T20:27:23.399 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T20:27:23.399 INFO:teuthology.orchestra.run.smithi099.stdout: "gen": 0, 2023-05-31T20:27:23.399 INFO:teuthology.orchestra.run.smithi099.stdout: "layout": { 2023-05-31T20:27:23.399 INFO:teuthology.orchestra.run.smithi099.stdout: "type": "InIndex", 2023-05-31T20:27:23.399 INFO:teuthology.orchestra.run.smithi099.stdout: "in_index": { 2023-05-31T20:27:23.400 INFO:teuthology.orchestra.run.smithi099.stdout: "gen": 0, 2023-05-31T20:27:23.400 INFO:teuthology.orchestra.run.smithi099.stdout: "layout": { 2023-05-31T20:27:23.400 INFO:teuthology.orchestra.run.smithi099.stdout: "num_shards": 11, 2023-05-31T20:27:23.400 INFO:teuthology.orchestra.run.smithi099.stdout: "hash_type": "Mod" 2023-05-31T20:27:23.400 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T20:27:23.400 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T20:27:23.400 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T20:27:23.401 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T20:27:23.401 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2023-05-31T20:27:23.401 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T20:27:23.401 INFO:teuthology.orchestra.run.smithi099.stdout:} 2023-05-31T20:27:23.402 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'list', '--bucket', 'pklxkd-55', '--gen', '0', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-31T20:27:23.403 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'list', '--bucket', 'pklxkd-55', '--gen', '0', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-31T20:27:23.404 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog list --bucket pklxkd-55 --gen 0 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-05-31T20:27:23.454 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2023-05-31T20:27:23.454 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2023-05-31T20:27:23.515 INFO:teuthology.orchestra.run.smithi099.stdout:[ 2023-05-31T20:27:23.516 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T20:27:23.516 INFO:teuthology.orchestra.run.smithi099.stdout: "op_id": "1#00000000001.28.6", 2023-05-31T20:27:23.516 INFO:teuthology.orchestra.run.smithi099.stdout: "op_tag": "3d163b1e-09c7-47e2-ae87-add4aace3f38.5680.912616895933462705", 2023-05-31T20:27:23.516 INFO:teuthology.orchestra.run.smithi099.stdout: "op": "write", 2023-05-31T20:27:23.516 INFO:teuthology.orchestra.run.smithi099.stdout: "object": "a", 2023-05-31T20:27:23.517 INFO:teuthology.orchestra.run.smithi099.stdout: "instance": "", 2023-05-31T20:27:23.517 INFO:teuthology.orchestra.run.smithi099.stdout: "state": "complete", 2023-05-31T20:27:23.517 INFO:teuthology.orchestra.run.smithi099.stdout: "index_ver": 1, 2023-05-31T20:27:23.517 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2023-05-31T20:27:16.792147057Z", 2023-05-31T20:27:23.518 INFO:teuthology.orchestra.run.smithi099.stdout: "ver": { 2023-05-31T20:27:23.518 INFO:teuthology.orchestra.run.smithi099.stdout: "pool": 9, 2023-05-31T20:27:23.518 INFO:teuthology.orchestra.run.smithi099.stdout: "epoch": 1 2023-05-31T20:27:23.518 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:27:23.518 INFO:teuthology.orchestra.run.smithi099.stdout: "bilog_flags": 0, 2023-05-31T20:27:23.518 INFO:teuthology.orchestra.run.smithi099.stdout: "versioned": false, 2023-05-31T20:27:23.519 INFO:teuthology.orchestra.run.smithi099.stdout: "owner": "", 2023-05-31T20:27:23.519 INFO:teuthology.orchestra.run.smithi099.stdout: "owner_display_name": "", 2023-05-31T20:27:23.519 INFO:teuthology.orchestra.run.smithi099.stdout: "zones_trace": [ 2023-05-31T20:27:23.519 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T20:27:23.519 INFO:teuthology.orchestra.run.smithi099.stdout: "entry": "3d163b1e-09c7-47e2-ae87-add4aace3f38:pklxkd-55:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.20" 2023-05-31T20:27:23.520 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T20:27:23.520 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2023-05-31T20:27:23.520 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:27:23.520 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T20:27:23.520 INFO:teuthology.orchestra.run.smithi099.stdout: "op_id": "5#00000000001.45.6", 2023-05-31T20:27:23.521 INFO:teuthology.orchestra.run.smithi099.stdout: "op_tag": "3d163b1e-09c7-47e2-ae87-add4aace3f38.5680.15231583731067601239", 2023-05-31T20:27:23.521 INFO:teuthology.orchestra.run.smithi099.stdout: "op": "write", 2023-05-31T20:27:23.521 INFO:teuthology.orchestra.run.smithi099.stdout: "object": "d", 2023-05-31T20:27:23.521 INFO:teuthology.orchestra.run.smithi099.stdout: "instance": "", 2023-05-31T20:27:23.521 INFO:teuthology.orchestra.run.smithi099.stdout: "state": "complete", 2023-05-31T20:27:23.521 INFO:teuthology.orchestra.run.smithi099.stdout: "index_ver": 1, 2023-05-31T20:27:23.521 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2023-05-31T20:27:16.893760052Z", 2023-05-31T20:27:23.522 INFO:teuthology.orchestra.run.smithi099.stdout: "ver": { 2023-05-31T20:27:23.522 INFO:teuthology.orchestra.run.smithi099.stdout: "pool": 9, 2023-05-31T20:27:23.522 INFO:teuthology.orchestra.run.smithi099.stdout: "epoch": 3 2023-05-31T20:27:23.522 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:27:23.522 INFO:teuthology.orchestra.run.smithi099.stdout: "bilog_flags": 0, 2023-05-31T20:27:23.522 INFO:teuthology.orchestra.run.smithi099.stdout: "versioned": false, 2023-05-31T20:27:23.523 INFO:teuthology.orchestra.run.smithi099.stdout: "owner": "", 2023-05-31T20:27:23.523 INFO:teuthology.orchestra.run.smithi099.stdout: "owner_display_name": "", 2023-05-31T20:27:23.523 INFO:teuthology.orchestra.run.smithi099.stdout: "zones_trace": [ 2023-05-31T20:27:23.523 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T20:27:23.523 INFO:teuthology.orchestra.run.smithi099.stdout: "entry": "3d163b1e-09c7-47e2-ae87-add4aace3f38:pklxkd-55:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.20" 2023-05-31T20:27:23.524 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T20:27:23.524 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2023-05-31T20:27:23.524 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:27:23.524 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T20:27:23.524 INFO:teuthology.orchestra.run.smithi099.stdout: "op_id": "8#00000000001.42.6", 2023-05-31T20:27:23.524 INFO:teuthology.orchestra.run.smithi099.stdout: "op_tag": "3d163b1e-09c7-47e2-ae87-add4aace3f38.5680.17607519642157465194", 2023-05-31T20:27:23.524 INFO:teuthology.orchestra.run.smithi099.stdout: "op": "write", 2023-05-31T20:27:23.525 INFO:teuthology.orchestra.run.smithi099.stdout: "object": "c", 2023-05-31T20:27:23.525 INFO:teuthology.orchestra.run.smithi099.stdout: "instance": "", 2023-05-31T20:27:23.525 INFO:teuthology.orchestra.run.smithi099.stdout: "state": "complete", 2023-05-31T20:27:23.525 INFO:teuthology.orchestra.run.smithi099.stdout: "index_ver": 1, 2023-05-31T20:27:23.525 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2023-05-31T20:27:16.869625413Z", 2023-05-31T20:27:23.526 INFO:teuthology.orchestra.run.smithi099.stdout: "ver": { 2023-05-31T20:27:23.526 INFO:teuthology.orchestra.run.smithi099.stdout: "pool": 9, 2023-05-31T20:27:23.526 INFO:teuthology.orchestra.run.smithi099.stdout: "epoch": 4 2023-05-31T20:27:23.526 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:27:23.526 INFO:teuthology.orchestra.run.smithi099.stdout: "bilog_flags": 0, 2023-05-31T20:27:23.526 INFO:teuthology.orchestra.run.smithi099.stdout: "versioned": false, 2023-05-31T20:27:23.526 INFO:teuthology.orchestra.run.smithi099.stdout: "owner": "", 2023-05-31T20:27:23.527 INFO:teuthology.orchestra.run.smithi099.stdout: "owner_display_name": "", 2023-05-31T20:27:23.527 INFO:teuthology.orchestra.run.smithi099.stdout: "zones_trace": [ 2023-05-31T20:27:23.527 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T20:27:23.527 INFO:teuthology.orchestra.run.smithi099.stdout: "entry": "3d163b1e-09c7-47e2-ae87-add4aace3f38:pklxkd-55:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.20" 2023-05-31T20:27:23.527 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T20:27:23.527 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2023-05-31T20:27:23.528 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:27:23.528 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T20:27:23.528 INFO:teuthology.orchestra.run.smithi099.stdout: "op_id": "10#00000000001.108.6", 2023-05-31T20:27:23.528 INFO:teuthology.orchestra.run.smithi099.stdout: "op_tag": "3d163b1e-09c7-47e2-ae87-add4aace3f38.5680.4722290385692952109", 2023-05-31T20:27:23.528 INFO:teuthology.orchestra.run.smithi099.stdout: "op": "write", 2023-05-31T20:27:23.528 INFO:teuthology.orchestra.run.smithi099.stdout: "object": "b", 2023-05-31T20:27:23.528 INFO:teuthology.orchestra.run.smithi099.stdout: "instance": "", 2023-05-31T20:27:23.529 INFO:teuthology.orchestra.run.smithi099.stdout: "state": "complete", 2023-05-31T20:27:23.529 INFO:teuthology.orchestra.run.smithi099.stdout: "index_ver": 1, 2023-05-31T20:27:23.529 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2023-05-31T20:27:16.850404114Z", 2023-05-31T20:27:23.529 INFO:teuthology.orchestra.run.smithi099.stdout: "ver": { 2023-05-31T20:27:23.529 INFO:teuthology.orchestra.run.smithi099.stdout: "pool": 9, 2023-05-31T20:27:23.530 INFO:teuthology.orchestra.run.smithi099.stdout: "epoch": 2 2023-05-31T20:27:23.530 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:27:23.530 INFO:teuthology.orchestra.run.smithi099.stdout: "bilog_flags": 0, 2023-05-31T20:27:23.530 INFO:teuthology.orchestra.run.smithi099.stdout: "versioned": false, 2023-05-31T20:27:23.530 INFO:teuthology.orchestra.run.smithi099.stdout: "owner": "", 2023-05-31T20:27:23.530 INFO:teuthology.orchestra.run.smithi099.stdout: "owner_display_name": "", 2023-05-31T20:27:23.530 INFO:teuthology.orchestra.run.smithi099.stdout: "zones_trace": [ 2023-05-31T20:27:23.531 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T20:27:23.531 INFO:teuthology.orchestra.run.smithi099.stdout: "entry": "3d163b1e-09c7-47e2-ae87-add4aace3f38:pklxkd-55:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.20" 2023-05-31T20:27:23.531 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T20:27:23.531 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2023-05-31T20:27:23.531 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T20:27:23.531 INFO:teuthology.orchestra.run.smithi099.stdout:] 2023-05-31T20:27:23.532 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'reshard', '--bucket', 'pklxkd-55', '--num-shards', '3', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-05-31T20:27:23.533 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'reshard', '--bucket', 'pklxkd-55', '--num-shards', '3', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-05-31T20:27:23.533 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket reshard --bucket pklxkd-55 --num-shards 3 --yes-i-really-mean-it --cluster c1 --debug-rgw 0 --debug-ms 0 2023-05-31T20:27:23.582 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2023-05-31T20:27:23.583 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2023-05-31T20:27:23.637 INFO:teuthology.orchestra.run.smithi099.stdout:tenant: 2023-05-31T20:27:23.637 INFO:teuthology.orchestra.run.smithi099.stdout:bucket name: pklxkd-55 2023-05-31T20:27:23.641 INFO:teuthology.orchestra.run.smithi099.stdout:total entries: 4 2023-05-31T20:27:23.685 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'layout', '--bucket', 'pklxkd-55', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-05-31T20:27:23.685 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'layout', '--bucket', 'pklxkd-55', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-05-31T20:27:23.686 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket layout --bucket pklxkd-55 --cluster c1 --debug-rgw 0 --debug-ms 0 2023-05-31T20:27:23.733 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2023-05-31T20:27:23.733 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2023-05-31T20:27:23.810 INFO:teuthology.orchestra.run.smithi099.stdout:{ 2023-05-31T20:27:23.811 INFO:teuthology.orchestra.run.smithi099.stdout: "layout": { 2023-05-31T20:27:23.811 INFO:teuthology.orchestra.run.smithi099.stdout: "resharding": "None", 2023-05-31T20:27:23.811 INFO:teuthology.orchestra.run.smithi099.stdout: "current_index": { 2023-05-31T20:27:23.811 INFO:teuthology.orchestra.run.smithi099.stdout: "gen": 1, 2023-05-31T20:27:23.811 INFO:teuthology.orchestra.run.smithi099.stdout: "layout": { 2023-05-31T20:27:23.812 INFO:teuthology.orchestra.run.smithi099.stdout: "type": "Normal", 2023-05-31T20:27:23.812 INFO:teuthology.orchestra.run.smithi099.stdout: "normal": { 2023-05-31T20:27:23.812 INFO:teuthology.orchestra.run.smithi099.stdout: "num_shards": 3, 2023-05-31T20:27:23.812 INFO:teuthology.orchestra.run.smithi099.stdout: "hash_type": "Mod" 2023-05-31T20:27:23.812 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T20:27:23.812 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T20:27:23.812 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:27:23.813 INFO:teuthology.orchestra.run.smithi099.stdout: "logs": [ 2023-05-31T20:27:23.813 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T20:27:23.813 INFO:teuthology.orchestra.run.smithi099.stdout: "gen": 0, 2023-05-31T20:27:23.813 INFO:teuthology.orchestra.run.smithi099.stdout: "layout": { 2023-05-31T20:27:23.813 INFO:teuthology.orchestra.run.smithi099.stdout: "type": "InIndex", 2023-05-31T20:27:23.813 INFO:teuthology.orchestra.run.smithi099.stdout: "in_index": { 2023-05-31T20:27:23.814 INFO:teuthology.orchestra.run.smithi099.stdout: "gen": 0, 2023-05-31T20:27:23.814 INFO:teuthology.orchestra.run.smithi099.stdout: "layout": { 2023-05-31T20:27:23.814 INFO:teuthology.orchestra.run.smithi099.stdout: "num_shards": 11, 2023-05-31T20:27:23.814 INFO:teuthology.orchestra.run.smithi099.stdout: "hash_type": "Mod" 2023-05-31T20:27:23.814 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T20:27:23.814 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T20:27:23.814 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T20:27:23.815 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:27:23.815 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T20:27:23.815 INFO:teuthology.orchestra.run.smithi099.stdout: "gen": 1, 2023-05-31T20:27:23.815 INFO:teuthology.orchestra.run.smithi099.stdout: "layout": { 2023-05-31T20:27:23.815 INFO:teuthology.orchestra.run.smithi099.stdout: "type": "InIndex", 2023-05-31T20:27:23.815 INFO:teuthology.orchestra.run.smithi099.stdout: "in_index": { 2023-05-31T20:27:23.815 INFO:teuthology.orchestra.run.smithi099.stdout: "gen": 1, 2023-05-31T20:27:23.816 INFO:teuthology.orchestra.run.smithi099.stdout: "layout": { 2023-05-31T20:27:23.816 INFO:teuthology.orchestra.run.smithi099.stdout: "num_shards": 3, 2023-05-31T20:27:23.816 INFO:teuthology.orchestra.run.smithi099.stdout: "hash_type": "Mod" 2023-05-31T20:27:23.816 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T20:27:23.816 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T20:27:23.816 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T20:27:23.816 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T20:27:23.817 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2023-05-31T20:27:23.817 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T20:27:23.817 INFO:teuthology.orchestra.run.smithi099.stdout:} 2023-05-31T20:27:23.819 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'list', '--bucket', 'pklxkd-55', '--gen', '1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-31T20:27:23.820 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'list', '--bucket', 'pklxkd-55', '--gen', '1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-31T20:27:23.821 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog list --bucket pklxkd-55 --gen 1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-05-31T20:27:23.870 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2023-05-31T20:27:23.870 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2023-05-31T20:27:23.938 INFO:teuthology.orchestra.run.smithi099.stdout:[] 2023-05-31T20:27:23.940 DEBUG:boto:path=/pklxkd-55/ 2023-05-31T20:27:23.941 DEBUG:boto:auth_path=/pklxkd-55/ 2023-05-31T20:27:23.941 DEBUG:boto:Method: HEAD 2023-05-31T20:27:23.941 DEBUG:boto:Path: /pklxkd-55/ 2023-05-31T20:27:23.942 DEBUG:boto:Data: 2023-05-31T20:27:23.942 DEBUG:boto:Headers: {} 2023-05-31T20:27:23.942 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:27:23.943 DEBUG:boto:Port: 8000 2023-05-31T20:27:23.943 DEBUG:boto:Params: {} 2023-05-31T20:27:23.943 DEBUG:boto:Token: None 2023-05-31T20:27:23.944 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:27:23 GMT /pklxkd-55/ 2023-05-31T20:27:23.949 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:CEgAiulrYDLboHO703TUJQSDrdU= 2023-05-31T20:27:23.949 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:27:23 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:CEgAiulrYDLboHO703TUJQSDrdU=', 'Content-Length': '0'} 2023-05-31T20:27:23.951 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '4'), ('X-RGW-Bytes-Used', '12'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000003602e9329a2cfacb-006477adab-1630-a1'), ('Content-Length', '0'), ('Date', 'Wed, 31 May 2023 20:27:23 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:27:23.952 DEBUG:boto:path=/pklxkd-55/e 2023-05-31T20:27:23.952 DEBUG:boto:auth_path=/pklxkd-55/e 2023-05-31T20:27:23.953 DEBUG:boto:Method: PUT 2023-05-31T20:27:23.953 DEBUG:boto:Path: /pklxkd-55/e 2023-05-31T20:27:23.953 DEBUG:boto:Data: 2023-05-31T20:27:23.954 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-05-31T20:27:23.954 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:27:23.954 DEBUG:boto:Port: 8000 2023-05-31T20:27:23.955 DEBUG:boto:Params: {} 2023-05-31T20:27:23.955 DEBUG:boto:Token: None 2023-05-31T20:27:23.956 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Wed, 31 May 2023 20:27:23 GMT /pklxkd-55/e 2023-05-31T20:27:23.956 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:YG9ufmH3Qho9/nfJWt8bHk8kHpo= 2023-05-31T20:27:23.957 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Wed, 31 May 2023 20:27:23 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:YG9ufmH3Qho9/nfJWt8bHk8kHpo='} 2023-05-31T20:27:23.963 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000c9b76a37aa6279f0-006477adab-1630-a1'), ('Date', 'Wed, 31 May 2023 20:27:23 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:27:23.964 DEBUG:boto:path=/pklxkd-55/ 2023-05-31T20:27:23.964 DEBUG:boto:auth_path=/pklxkd-55/ 2023-05-31T20:27:23.964 DEBUG:boto:Method: HEAD 2023-05-31T20:27:23.965 DEBUG:boto:Path: /pklxkd-55/ 2023-05-31T20:27:23.965 DEBUG:boto:Data: 2023-05-31T20:27:23.965 DEBUG:boto:Headers: {} 2023-05-31T20:27:23.966 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:27:23.966 DEBUG:boto:Port: 8000 2023-05-31T20:27:23.966 DEBUG:boto:Params: {} 2023-05-31T20:27:23.967 DEBUG:boto:Token: None 2023-05-31T20:27:23.968 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:27:23 GMT /pklxkd-55/ 2023-05-31T20:27:23.969 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:CEgAiulrYDLboHO703TUJQSDrdU= 2023-05-31T20:27:23.970 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:27:23 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:CEgAiulrYDLboHO703TUJQSDrdU=', 'Content-Length': '0'} 2023-05-31T20:27:23.974 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '5'), ('X-RGW-Bytes-Used', '15'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000c47b2a42d23d98bf-006477adab-1630-a1'), ('Content-Length', '0'), ('Date', 'Wed, 31 May 2023 20:27:23 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:27:23.974 DEBUG:boto:path=/pklxkd-55/f 2023-05-31T20:27:23.975 DEBUG:boto:auth_path=/pklxkd-55/f 2023-05-31T20:27:23.975 DEBUG:boto:Method: PUT 2023-05-31T20:27:23.976 DEBUG:boto:Path: /pklxkd-55/f 2023-05-31T20:27:23.976 DEBUG:boto:Data: 2023-05-31T20:27:23.977 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-05-31T20:27:23.977 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:27:23.977 DEBUG:boto:Port: 8000 2023-05-31T20:27:23.978 DEBUG:boto:Params: {} 2023-05-31T20:27:23.978 DEBUG:boto:Token: None 2023-05-31T20:27:23.979 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Wed, 31 May 2023 20:27:23 GMT /pklxkd-55/f 2023-05-31T20:27:23.979 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:a1pWMm25Au1uR3xgcR7gQfQNXjU= 2023-05-31T20:27:23.980 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Wed, 31 May 2023 20:27:23 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:a1pWMm25Au1uR3xgcR7gQfQNXjU='} 2023-05-31T20:27:24.024 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000510ec301fc0fbe51-006477adab-1630-a1'), ('Date', 'Wed, 31 May 2023 20:27:23 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:27:24.025 DEBUG:boto:path=/pklxkd-55/ 2023-05-31T20:27:24.025 DEBUG:boto:auth_path=/pklxkd-55/ 2023-05-31T20:27:24.026 DEBUG:boto:Method: HEAD 2023-05-31T20:27:24.027 DEBUG:boto:Path: /pklxkd-55/ 2023-05-31T20:27:24.027 DEBUG:boto:Data: 2023-05-31T20:27:24.027 DEBUG:boto:Headers: {} 2023-05-31T20:27:24.028 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:27:24.028 DEBUG:boto:Port: 8000 2023-05-31T20:27:24.028 DEBUG:boto:Params: {} 2023-05-31T20:27:24.029 DEBUG:boto:Token: None 2023-05-31T20:27:24.029 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:27:24 GMT /pklxkd-55/ 2023-05-31T20:27:24.029 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:Ar+db5NHPQANb45BVa+KAir9ejw= 2023-05-31T20:27:24.030 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:27:24 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:Ar+db5NHPQANb45BVa+KAir9ejw=', 'Content-Length': '0'} 2023-05-31T20:27:24.031 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '6'), ('X-RGW-Bytes-Used', '18'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000ca0be5ef7e1d6577-006477adac-1630-a1'), ('Content-Length', '0'), ('Date', 'Wed, 31 May 2023 20:27:24 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:27:24.032 DEBUG:boto:path=/pklxkd-55/g 2023-05-31T20:27:24.032 DEBUG:boto:auth_path=/pklxkd-55/g 2023-05-31T20:27:24.033 DEBUG:boto:Method: PUT 2023-05-31T20:27:24.033 DEBUG:boto:Path: /pklxkd-55/g 2023-05-31T20:27:24.033 DEBUG:boto:Data: 2023-05-31T20:27:24.034 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-05-31T20:27:24.034 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:27:24.034 DEBUG:boto:Port: 8000 2023-05-31T20:27:24.035 DEBUG:boto:Params: {} 2023-05-31T20:27:24.035 DEBUG:boto:Token: None 2023-05-31T20:27:24.036 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Wed, 31 May 2023 20:27:24 GMT /pklxkd-55/g 2023-05-31T20:27:24.036 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:3qgXKe/W/MZUJUlYldMKkKa7YJ0= 2023-05-31T20:27:24.037 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Wed, 31 May 2023 20:27:24 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:3qgXKe/W/MZUJUlYldMKkKa7YJ0='} 2023-05-31T20:27:24.042 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000fcc73590820242b0-006477adac-1630-a1'), ('Date', 'Wed, 31 May 2023 20:27:24 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:27:24.043 DEBUG:boto:path=/pklxkd-55/ 2023-05-31T20:27:24.043 DEBUG:boto:auth_path=/pklxkd-55/ 2023-05-31T20:27:24.044 DEBUG:boto:Method: HEAD 2023-05-31T20:27:24.044 DEBUG:boto:Path: /pklxkd-55/ 2023-05-31T20:27:24.044 DEBUG:boto:Data: 2023-05-31T20:27:24.045 DEBUG:boto:Headers: {} 2023-05-31T20:27:24.045 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:27:24.045 DEBUG:boto:Port: 8000 2023-05-31T20:27:24.046 DEBUG:boto:Params: {} 2023-05-31T20:27:24.046 DEBUG:boto:Token: None 2023-05-31T20:27:24.046 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:27:24 GMT /pklxkd-55/ 2023-05-31T20:27:24.047 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:Ar+db5NHPQANb45BVa+KAir9ejw= 2023-05-31T20:27:24.048 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:27:24 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:Ar+db5NHPQANb45BVa+KAir9ejw=', 'Content-Length': '0'} 2023-05-31T20:27:24.050 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '7'), ('X-RGW-Bytes-Used', '21'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000096f61cb293d6097c-006477adac-1630-a1'), ('Content-Length', '0'), ('Date', 'Wed, 31 May 2023 20:27:24 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:27:24.050 DEBUG:boto:path=/pklxkd-55/h 2023-05-31T20:27:24.051 DEBUG:boto:auth_path=/pklxkd-55/h 2023-05-31T20:27:24.051 DEBUG:boto:Method: PUT 2023-05-31T20:27:24.051 DEBUG:boto:Path: /pklxkd-55/h 2023-05-31T20:27:24.052 DEBUG:boto:Data: 2023-05-31T20:27:24.052 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-05-31T20:27:24.052 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:27:24.053 DEBUG:boto:Port: 8000 2023-05-31T20:27:24.054 DEBUG:boto:Params: {} 2023-05-31T20:27:24.054 DEBUG:boto:Token: None 2023-05-31T20:27:24.055 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Wed, 31 May 2023 20:27:24 GMT /pklxkd-55/h 2023-05-31T20:27:24.055 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:+TJhWtofpmkyn8DPjTGL0Vvt3WI= 2023-05-31T20:27:24.055 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Wed, 31 May 2023 20:27:24 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:+TJhWtofpmkyn8DPjTGL0Vvt3WI='} 2023-05-31T20:27:24.059 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000cfe08ff100955461-006477adac-1630-a1'), ('Date', 'Wed, 31 May 2023 20:27:24 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:27:24.060 DEBUG:rgw_multi.tests:bucket checkpoint: source=a1 target=a2 bucket=pklxkd-55 2023-05-31T20:27:24.060 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'pklxkd-55', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-05-31T20:27:24.061 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'pklxkd-55', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-05-31T20:27:24.061 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket pklxkd-55 --source-zone a1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2023-05-31T20:27:24.122 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2023-05-31T20:27:24.122 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2023-05-31T20:27:24.173 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:27:24.167+0000 7f0d821a6780 1 waiting to reach latest gen 1, on 0.. 2023-05-31T20:27:29.173 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:27:29.169+0000 7f0d821a6780 1 waiting to reach latest gen 1, on 0.. 2023-05-31T20:27:34.174 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:27:34.170+0000 7f0d821a6780 1 waiting to reach latest gen 1, on 0.. 2023-05-31T20:27:39.175 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:27:39.170+0000 7f0d821a6780 1 waiting to reach latest gen 1, on 0.. 2023-05-31T20:27:44.176 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:27:44.171+0000 7f0d821a6780 1 waiting for incremental sync to catch up: 2023-05-31T20:27:44.176 INFO:teuthology.orchestra.run.smithi099.stderr: local status: [, , ] 2023-05-31T20:27:44.176 INFO:teuthology.orchestra.run.smithi099.stderr: remote markers: [00000000003.52.6, 00000000002.15.6, 00000000002.32.6] 2023-05-31T20:27:49.177 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:27:49.171+0000 7f0d821a6780 1 waiting for incremental sync to catch up: 2023-05-31T20:27:49.177 INFO:teuthology.orchestra.run.smithi099.stderr: local status: [, , ] 2023-05-31T20:27:49.177 INFO:teuthology.orchestra.run.smithi099.stderr: remote markers: [00000000003.52.6, 00000000002.15.6, 00000000002.32.6] 2023-05-31T20:27:54.178 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:27:54.172+0000 7f0d821a6780 1 waiting for incremental sync to catch up: 2023-05-31T20:27:54.178 INFO:teuthology.orchestra.run.smithi099.stderr: local status: [, , ] 2023-05-31T20:27:54.178 INFO:teuthology.orchestra.run.smithi099.stderr: remote markers: [00000000003.52.6, 00000000002.15.6, 00000000002.32.6] 2023-05-31T20:27:59.178 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:27:59.173+0000 7f0d821a6780 1 waiting for incremental sync to catch up: 2023-05-31T20:27:59.179 INFO:teuthology.orchestra.run.smithi099.stderr: local status: [, , ] 2023-05-31T20:27:59.179 INFO:teuthology.orchestra.run.smithi099.stderr: remote markers: [00000000003.52.6, 00000000002.15.6, 00000000002.32.6] 2023-05-31T20:28:04.179 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:28:04.173+0000 7f0d821a6780 1 waiting for incremental sync to catch up: 2023-05-31T20:28:04.179 INFO:teuthology.orchestra.run.smithi099.stderr: local status: [, , ] 2023-05-31T20:28:04.179 INFO:teuthology.orchestra.run.smithi099.stderr: remote markers: [00000000003.52.6, 00000000002.15.6, 00000000002.32.6] 2023-05-31T20:28:09.180 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:28:09.175+0000 7f0d821a6780 1 waiting for incremental sync to catch up: 2023-05-31T20:28:09.180 INFO:teuthology.orchestra.run.smithi099.stderr: local status: [, , ] 2023-05-31T20:28:09.180 INFO:teuthology.orchestra.run.smithi099.stderr: remote markers: [00000000003.52.6, 00000000002.15.6, 00000000002.32.6] 2023-05-31T20:28:14.180 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:28:14.176+0000 7f0d821a6780 1 waiting for incremental sync to catch up: 2023-05-31T20:28:14.181 INFO:teuthology.orchestra.run.smithi099.stderr: local status: [, , ] 2023-05-31T20:28:14.181 INFO:teuthology.orchestra.run.smithi099.stderr: remote markers: [00000000003.52.6, 00000000002.15.6, 00000000002.32.6] 2023-05-31T20:28:19.181 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:28:19.177+0000 7f0d821a6780 1 waiting for incremental sync to catch up: 2023-05-31T20:28:19.182 INFO:teuthology.orchestra.run.smithi099.stderr: local status: [, , ] 2023-05-31T20:28:19.182 INFO:teuthology.orchestra.run.smithi099.stderr: remote markers: [00000000003.52.6, 00000000002.15.6, 00000000002.32.6] 2023-05-31T20:28:24.182 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:28:24.178+0000 7f0d821a6780 1 waiting for incremental sync to catch up: 2023-05-31T20:28:24.183 INFO:teuthology.orchestra.run.smithi099.stderr: local status: [, , ] 2023-05-31T20:28:24.183 INFO:teuthology.orchestra.run.smithi099.stderr: remote markers: [00000000003.52.6, 00000000002.15.6, 00000000002.32.6] 2023-05-31T20:28:29.183 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:28:29.177+0000 7f0d821a6780 1 waiting for incremental sync to catch up: 2023-05-31T20:28:29.183 INFO:teuthology.orchestra.run.smithi099.stderr: local status: [, 00000000002.15.6, ] 2023-05-31T20:28:29.183 INFO:teuthology.orchestra.run.smithi099.stderr: remote markers: [00000000003.52.6, 00000000002.15.6, 00000000002.32.6] 2023-05-31T20:28:34.183 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:28:34.177+0000 7f0d821a6780 1 waiting for incremental sync to catch up: 2023-05-31T20:28:34.183 INFO:teuthology.orchestra.run.smithi099.stderr: local status: [, 00000000002.15.6, ] 2023-05-31T20:28:34.183 INFO:teuthology.orchestra.run.smithi099.stderr: remote markers: [00000000003.52.6, 00000000002.15.6, 00000000002.32.6] 2023-05-31T20:28:39.183 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:28:39.178+0000 7f0d821a6780 1 waiting for incremental sync to catch up: 2023-05-31T20:28:39.184 INFO:teuthology.orchestra.run.smithi099.stderr: local status: [, 00000000002.15.6, ] 2023-05-31T20:28:39.184 INFO:teuthology.orchestra.run.smithi099.stderr: remote markers: [00000000003.52.6, 00000000002.15.6, 00000000002.32.6] 2023-05-31T20:28:44.184 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:28:44.179+0000 7f0d821a6780 1 waiting for incremental sync to catch up: 2023-05-31T20:28:44.185 INFO:teuthology.orchestra.run.smithi099.stderr: local status: [, 00000000002.15.6, 00000000002.32.6] 2023-05-31T20:28:44.185 INFO:teuthology.orchestra.run.smithi099.stderr: remote markers: [00000000003.52.6, 00000000002.15.6, 00000000002.32.6] 2023-05-31T20:28:49.185 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:28:49.180+0000 7f0d821a6780 1 waiting for incremental sync to catch up: 2023-05-31T20:28:49.186 INFO:teuthology.orchestra.run.smithi099.stderr: local status: [, 00000000002.15.6, 00000000002.32.6] 2023-05-31T20:28:49.186 INFO:teuthology.orchestra.run.smithi099.stderr: remote markers: [00000000003.52.6, 00000000002.15.6, 00000000002.32.6] 2023-05-31T20:28:54.186 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:28:54.181+0000 7f0d821a6780 1 waiting for incremental sync to catch up: 2023-05-31T20:28:54.186 INFO:teuthology.orchestra.run.smithi099.stderr: local status: [, 00000000002.15.6, 00000000002.32.6] 2023-05-31T20:28:54.186 INFO:teuthology.orchestra.run.smithi099.stderr: remote markers: [00000000003.52.6, 00000000002.15.6, 00000000002.32.6] 2023-05-31T20:28:59.187 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:28:59.182+0000 7f0d821a6780 1 waiting for incremental sync to catch up: 2023-05-31T20:28:59.187 INFO:teuthology.orchestra.run.smithi099.stderr: local status: [, 00000000002.15.6, 00000000002.32.6] 2023-05-31T20:28:59.188 INFO:teuthology.orchestra.run.smithi099.stderr: remote markers: [00000000003.52.6, 00000000002.15.6, 00000000002.32.6] 2023-05-31T20:29:04.188 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:29:04.183+0000 7f0d821a6780 1 bucket sync caught up with source: 2023-05-31T20:29:04.188 INFO:teuthology.orchestra.run.smithi099.stderr: local status: [00000000003.52.6, 00000000002.15.6, 00000000002.32.6] 2023-05-31T20:29:04.188 INFO:teuthology.orchestra.run.smithi099.stderr: remote markers: [00000000003.52.6, 00000000002.15.6, 00000000002.32.6] 2023-05-31T20:29:04.188 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:29:04.183+0000 7f0d821a6780 0 bucket checkpoint complete 2023-05-31T20:29:04.209 DEBUG:rgw_multi.tests:bucket checkpoint: source=a2 target=a1 bucket=pklxkd-55 2023-05-31T20:29:04.209 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'pklxkd-55', '--source-zone', 'a2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-05-31T20:29:04.210 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'pklxkd-55', '--source-zone', 'a2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-05-31T20:29:04.210 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket pklxkd-55 --source-zone a2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2023-05-31T20:29:04.272 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2023-05-31T20:29:04.273 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2023-05-31T20:29:04.325 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:29:04.320+0000 7f5722510780 1 waiting for incremental sync to catch up: 2023-05-31T20:29:04.325 INFO:teuthology.orchestra.run.smithi099.stderr: local status: [, 00000000002.1063.6, , , , 00000000001.1340.6, , , 00000000002.3558.6, , 00000000001.2404.6] 2023-05-31T20:29:04.325 INFO:teuthology.orchestra.run.smithi099.stderr: remote markers: [, 00000000002.1063.6, , , 00000000001.1666.6, 00000000001.1340.6, , , 00000000002.3558.6, , 00000000002.2408.6] 2023-05-31T20:29:09.326 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:29:09.322+0000 7f5722510780 1 waiting for incremental sync to catch up: 2023-05-31T20:29:09.326 INFO:teuthology.orchestra.run.smithi099.stderr: local status: [, 00000000002.1063.6, , , , 00000000001.1340.6, , , 00000000002.3558.6, , 00000000001.2404.6] 2023-05-31T20:29:09.327 INFO:teuthology.orchestra.run.smithi099.stderr: remote markers: [, 00000000002.1063.6, , , 00000000001.1666.6, 00000000001.1340.6, , , 00000000002.3558.6, , 00000000002.2408.6] 2023-05-31T20:29:14.327 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:29:14.322+0000 7f5722510780 1 waiting for incremental sync to catch up: 2023-05-31T20:29:14.328 INFO:teuthology.orchestra.run.smithi099.stderr: local status: [, 00000000002.1063.6, , , , 00000000001.1340.6, , , 00000000002.3558.6, , 00000000001.2404.6] 2023-05-31T20:29:14.330 INFO:teuthology.orchestra.run.smithi099.stderr: remote markers: [, 00000000002.1063.6, , , 00000000001.1666.6, 00000000001.1340.6, , , 00000000002.3558.6, , 00000000002.2408.6] 2023-05-31T20:29:19.328 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:29:19.323+0000 7f5722510780 1 waiting for incremental sync to catch up: 2023-05-31T20:29:19.329 INFO:teuthology.orchestra.run.smithi099.stderr: local status: [, 00000000002.1063.6, , , , 00000000001.1340.6, , , 00000000002.3558.6, , 00000000001.2404.6] 2023-05-31T20:29:19.329 INFO:teuthology.orchestra.run.smithi099.stderr: remote markers: [, 00000000002.1063.6, , , 00000000001.1666.6, 00000000001.1340.6, , , 00000000002.3558.6, , 00000000002.2408.6] 2023-05-31T20:29:24.330 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:29:24.324+0000 7f5722510780 1 bucket sync caught up with source: 2023-05-31T20:29:24.330 INFO:teuthology.orchestra.run.smithi099.stderr: local status: [, 00000000002.1063.6, , , 00000000001.1666.6, 00000000001.1340.6, , , 00000000002.3558.6, , 00000000002.2408.6] 2023-05-31T20:29:24.330 INFO:teuthology.orchestra.run.smithi099.stderr: remote markers: [, 00000000002.1063.6, , , 00000000001.1666.6, 00000000001.1340.6, , , 00000000002.3558.6, , 00000000002.2408.6] 2023-05-31T20:29:24.331 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:29:24.324+0000 7f5722510780 0 bucket checkpoint complete 2023-05-31T20:29:24.350 INFO:tasks.rgw_multi.zone_rados:comparing bucket=pklxkd-55 zones={a2, a1} 2023-05-31T20:29:24.351 DEBUG:boto:path=/pklxkd-55/ 2023-05-31T20:29:24.351 DEBUG:boto:auth_path=/pklxkd-55/ 2023-05-31T20:29:24.351 DEBUG:boto:Method: HEAD 2023-05-31T20:29:24.352 DEBUG:boto:Path: /pklxkd-55/ 2023-05-31T20:29:24.352 DEBUG:boto:Data: 2023-05-31T20:29:24.353 DEBUG:boto:Headers: {} 2023-05-31T20:29:24.353 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:29:24.353 DEBUG:boto:Port: 8001 2023-05-31T20:29:24.354 DEBUG:boto:Params: {} 2023-05-31T20:29:24.355 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8001} 2023-05-31T20:29:24.356 DEBUG:boto:Token: None 2023-05-31T20:29:24.356 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:29:24 GMT /pklxkd-55/ 2023-05-31T20:29:24.357 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:i72WzALuHR7KCaSfjkxAnzAjKlE= 2023-05-31T20:29:24.358 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:29:24 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:i72WzALuHR7KCaSfjkxAnzAjKlE=', 'Content-Length': '0'} 2023-05-31T20:29:24.361 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '8'), ('X-RGW-Bytes-Used', '24'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000868b23d2978f67e5-006477ae24-162e-a2'), ('Content-Length', '0'), ('Date', 'Wed, 31 May 2023 20:29:24 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:29:24.363 DEBUG:boto:path=/pklxkd-55/ 2023-05-31T20:29:24.363 DEBUG:boto:auth_path=/pklxkd-55/ 2023-05-31T20:29:24.364 DEBUG:boto:Method: HEAD 2023-05-31T20:29:24.365 DEBUG:boto:Path: /pklxkd-55/ 2023-05-31T20:29:24.366 DEBUG:boto:Data: 2023-05-31T20:29:24.366 DEBUG:boto:Headers: {} 2023-05-31T20:29:24.367 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:29:24.367 DEBUG:boto:Port: 8000 2023-05-31T20:29:24.368 DEBUG:boto:Params: {} 2023-05-31T20:29:24.369 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2023-05-31T20:29:24.370 DEBUG:boto:Token: None 2023-05-31T20:29:24.370 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:29:24 GMT /pklxkd-55/ 2023-05-31T20:29:24.371 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:i72WzALuHR7KCaSfjkxAnzAjKlE= 2023-05-31T20:29:24.372 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:29:24 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:i72WzALuHR7KCaSfjkxAnzAjKlE=', 'Content-Length': '0'} 2023-05-31T20:29:24.375 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '8'), ('X-RGW-Bytes-Used', '24'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000074383f9a60d1bb8-006477ae24-1630-a1'), ('Content-Length', '0'), ('Date', 'Wed, 31 May 2023 20:29:24 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:29:24.375 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2023-05-31T20:29:24.376 DEBUG:boto:path=/pklxkd-55/ 2023-05-31T20:29:24.376 DEBUG:boto:auth_path=/pklxkd-55/ 2023-05-31T20:29:24.377 DEBUG:boto:path=/pklxkd-55/?versions&max-keys=999 2023-05-31T20:29:24.377 DEBUG:boto:auth_path=/pklxkd-55/?versions&max-keys=999 2023-05-31T20:29:24.377 DEBUG:boto:Method: GET 2023-05-31T20:29:24.378 DEBUG:boto:Path: /pklxkd-55/?versions&max-keys=999 2023-05-31T20:29:24.378 DEBUG:boto:Data: 2023-05-31T20:29:24.378 DEBUG:boto:Headers: {} 2023-05-31T20:29:24.379 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:29:24.379 DEBUG:boto:Port: 8001 2023-05-31T20:29:24.379 DEBUG:boto:Params: {} 2023-05-31T20:29:24.380 DEBUG:boto:Token: None 2023-05-31T20:29:24.380 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:29:24 GMT /pklxkd-55/?versions 2023-05-31T20:29:24.381 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:+BJqTaR5KWPhPglRdf1Mjt4E5lw= 2023-05-31T20:29:24.382 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:29:24 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:+BJqTaR5KWPhPglRdf1Mjt4E5lw=', 'Content-Length': '0'} 2023-05-31T20:29:24.384 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000000bc1344cc39c9151-006477ae24-162e-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:29:24 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:29:24.385 DEBUG:boto:b'pklxkd-55999falseanulltrue2023-05-31T20:27:16.792Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalbnulltrue2023-05-31T20:27:16.850Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalcnulltrue2023-05-31T20:27:16.869Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormaldnulltrue2023-05-31T20:27:16.893Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalenulltrue2023-05-31T20:27:23.955Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalfnulltrue2023-05-31T20:27:23.977Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalgnulltrue2023-05-31T20:27:24.035Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalhnulltrue2023-05-31T20:27:24.053Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:29:24.386 DEBUG:tasks.rgw_multi.zone_rados:o=a 2023-05-31T20:29:24.388 DEBUG:tasks.rgw_multi.zone_rados:o=b 2023-05-31T20:29:24.388 DEBUG:tasks.rgw_multi.zone_rados:o=c 2023-05-31T20:29:24.389 DEBUG:tasks.rgw_multi.zone_rados:o=d 2023-05-31T20:29:24.389 DEBUG:tasks.rgw_multi.zone_rados:o=e 2023-05-31T20:29:24.389 DEBUG:tasks.rgw_multi.zone_rados:o=f 2023-05-31T20:29:24.390 DEBUG:tasks.rgw_multi.zone_rados:o=g 2023-05-31T20:29:24.390 DEBUG:tasks.rgw_multi.zone_rados:o=h 2023-05-31T20:29:24.390 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2023-05-31T20:29:24.391 DEBUG:boto:path=/pklxkd-55/ 2023-05-31T20:29:24.392 DEBUG:boto:auth_path=/pklxkd-55/ 2023-05-31T20:29:24.393 DEBUG:boto:path=/pklxkd-55/?versions&max-keys=999 2023-05-31T20:29:24.394 DEBUG:boto:auth_path=/pklxkd-55/?versions&max-keys=999 2023-05-31T20:29:24.394 DEBUG:boto:Method: GET 2023-05-31T20:29:24.394 DEBUG:boto:Path: /pklxkd-55/?versions&max-keys=999 2023-05-31T20:29:24.395 DEBUG:boto:Data: 2023-05-31T20:29:24.395 DEBUG:boto:Headers: {} 2023-05-31T20:29:24.396 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:29:24.396 DEBUG:boto:Port: 8000 2023-05-31T20:29:24.396 DEBUG:boto:Params: {} 2023-05-31T20:29:24.397 DEBUG:boto:Token: None 2023-05-31T20:29:24.397 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:29:24 GMT /pklxkd-55/?versions 2023-05-31T20:29:24.398 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:+BJqTaR5KWPhPglRdf1Mjt4E5lw= 2023-05-31T20:29:24.398 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:29:24 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:+BJqTaR5KWPhPglRdf1Mjt4E5lw=', 'Content-Length': '0'} 2023-05-31T20:29:24.400 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ddf9e2b1894011a2-006477ae24-1630-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:29:24 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:29:24.401 DEBUG:boto:b'pklxkd-55999falseanulltrue2023-05-31T20:27:16.792Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalbnulltrue2023-05-31T20:27:16.850Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalcnulltrue2023-05-31T20:27:16.869Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormaldnulltrue2023-05-31T20:27:16.893Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalenulltrue2023-05-31T20:27:23.955Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalfnulltrue2023-05-31T20:27:23.977Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalgnulltrue2023-05-31T20:27:24.035Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalhnulltrue2023-05-31T20:27:24.053Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:29:24.402 DEBUG:tasks.rgw_multi.zone_rados:o=a 2023-05-31T20:29:24.402 DEBUG:tasks.rgw_multi.zone_rados:o=b 2023-05-31T20:29:24.403 DEBUG:tasks.rgw_multi.zone_rados:o=c 2023-05-31T20:29:24.403 DEBUG:tasks.rgw_multi.zone_rados:o=d 2023-05-31T20:29:24.403 DEBUG:tasks.rgw_multi.zone_rados:o=e 2023-05-31T20:29:24.404 DEBUG:tasks.rgw_multi.zone_rados:o=f 2023-05-31T20:29:24.404 DEBUG:tasks.rgw_multi.zone_rados:o=g 2023-05-31T20:29:24.404 DEBUG:tasks.rgw_multi.zone_rados:o=h 2023-05-31T20:29:24.405 DEBUG:boto:path=/pklxkd-55/ 2023-05-31T20:29:24.405 DEBUG:boto:auth_path=/pklxkd-55/ 2023-05-31T20:29:24.405 DEBUG:boto:path=/pklxkd-55/?versions&max-keys=999 2023-05-31T20:29:24.406 DEBUG:boto:auth_path=/pklxkd-55/?versions&max-keys=999 2023-05-31T20:29:24.406 DEBUG:boto:Method: GET 2023-05-31T20:29:24.406 DEBUG:boto:Path: /pklxkd-55/?versions&max-keys=999 2023-05-31T20:29:24.407 DEBUG:boto:Data: 2023-05-31T20:29:24.408 DEBUG:boto:Headers: {} 2023-05-31T20:29:24.408 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:29:24.409 DEBUG:boto:Port: 8001 2023-05-31T20:29:24.409 DEBUG:boto:Params: {} 2023-05-31T20:29:24.409 DEBUG:boto:Token: None 2023-05-31T20:29:24.410 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:29:24 GMT /pklxkd-55/?versions 2023-05-31T20:29:24.410 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:+BJqTaR5KWPhPglRdf1Mjt4E5lw= 2023-05-31T20:29:24.411 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:29:24 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:+BJqTaR5KWPhPglRdf1Mjt4E5lw=', 'Content-Length': '0'} 2023-05-31T20:29:24.413 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ae7ee0fb5dd07b60-006477ae24-162e-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:29:24 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:29:24.413 DEBUG:boto:b'pklxkd-55999falseanulltrue2023-05-31T20:27:16.792Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalbnulltrue2023-05-31T20:27:16.850Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalcnulltrue2023-05-31T20:27:16.869Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormaldnulltrue2023-05-31T20:27:16.893Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalenulltrue2023-05-31T20:27:23.955Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalfnulltrue2023-05-31T20:27:23.977Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalgnulltrue2023-05-31T20:27:24.035Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalhnulltrue2023-05-31T20:27:24.053Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:29:24.414 DEBUG:boto:path=/pklxkd-55/ 2023-05-31T20:29:24.415 DEBUG:boto:auth_path=/pklxkd-55/ 2023-05-31T20:29:24.415 DEBUG:boto:path=/pklxkd-55/?versions&max-keys=999 2023-05-31T20:29:24.415 DEBUG:boto:auth_path=/pklxkd-55/?versions&max-keys=999 2023-05-31T20:29:24.416 DEBUG:boto:Method: GET 2023-05-31T20:29:24.416 DEBUG:boto:Path: /pklxkd-55/?versions&max-keys=999 2023-05-31T20:29:24.417 DEBUG:boto:Data: 2023-05-31T20:29:24.417 DEBUG:boto:Headers: {} 2023-05-31T20:29:24.417 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:29:24.418 DEBUG:boto:Port: 8000 2023-05-31T20:29:24.418 DEBUG:boto:Params: {} 2023-05-31T20:29:24.418 DEBUG:boto:Token: None 2023-05-31T20:29:24.419 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:29:24 GMT /pklxkd-55/?versions 2023-05-31T20:29:24.419 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:+BJqTaR5KWPhPglRdf1Mjt4E5lw= 2023-05-31T20:29:24.420 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:29:24 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:+BJqTaR5KWPhPglRdf1Mjt4E5lw=', 'Content-Length': '0'} 2023-05-31T20:29:24.422 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d92380a36f1a48fa-006477ae24-1630-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:29:24 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:29:24.423 DEBUG:boto:b'pklxkd-55999falseanulltrue2023-05-31T20:27:16.792Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalbnulltrue2023-05-31T20:27:16.850Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalcnulltrue2023-05-31T20:27:16.869Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormaldnulltrue2023-05-31T20:27:16.893Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalenulltrue2023-05-31T20:27:23.955Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalfnulltrue2023-05-31T20:27:23.977Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalgnulltrue2023-05-31T20:27:24.035Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalhnulltrue2023-05-31T20:27:24.053Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:29:24.425 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=a 2023-05-31T20:29:24.426 DEBUG:boto:path=/pklxkd-55/a 2023-05-31T20:29:24.426 DEBUG:boto:auth_path=/pklxkd-55/a 2023-05-31T20:29:24.427 DEBUG:boto:path=/pklxkd-55/a?versionId=null 2023-05-31T20:29:24.428 DEBUG:boto:auth_path=/pklxkd-55/a?versionId=null 2023-05-31T20:29:24.429 DEBUG:boto:Method: GET 2023-05-31T20:29:24.430 DEBUG:boto:Path: /pklxkd-55/a?versionId=null 2023-05-31T20:29:24.430 DEBUG:boto:Data: 2023-05-31T20:29:24.431 DEBUG:boto:Headers: {} 2023-05-31T20:29:24.432 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:29:24.433 DEBUG:boto:Port: 8001 2023-05-31T20:29:24.433 DEBUG:boto:Params: {} 2023-05-31T20:29:24.434 DEBUG:boto:Token: None 2023-05-31T20:29:24.435 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:29:24 GMT /pklxkd-55/a?versionId=null 2023-05-31T20:29:24.436 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:+X92hov5hckyw41lnTNPdKmNFYQ= 2023-05-31T20:29:24.437 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:29:24 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:+X92hov5hckyw41lnTNPdKmNFYQ=', 'Content-Length': '0'} 2023-05-31T20:29:24.439 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:27:16 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e40ceace07e7059a-006477ae24-162e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:29:24 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:29:24.484 DEBUG:boto:path=/pklxkd-55/a 2023-05-31T20:29:24.485 DEBUG:boto:auth_path=/pklxkd-55/a 2023-05-31T20:29:24.486 DEBUG:boto:path=/pklxkd-55/a?versionId=null 2023-05-31T20:29:24.487 DEBUG:boto:auth_path=/pklxkd-55/a?versionId=null 2023-05-31T20:29:24.488 DEBUG:boto:Method: GET 2023-05-31T20:29:24.488 DEBUG:boto:Path: /pklxkd-55/a?versionId=null 2023-05-31T20:29:24.489 DEBUG:boto:Data: 2023-05-31T20:29:24.489 DEBUG:boto:Headers: {} 2023-05-31T20:29:24.490 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:29:24.490 DEBUG:boto:Port: 8000 2023-05-31T20:29:24.491 DEBUG:boto:Params: {} 2023-05-31T20:29:24.491 DEBUG:boto:Token: None 2023-05-31T20:29:24.492 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:29:24 GMT /pklxkd-55/a?versionId=null 2023-05-31T20:29:24.492 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:+X92hov5hckyw41lnTNPdKmNFYQ= 2023-05-31T20:29:24.493 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:29:24 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:+X92hov5hckyw41lnTNPdKmNFYQ=', 'Content-Length': '0'} 2023-05-31T20:29:24.495 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:27:16 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009763aa4e42f7cd05-006477ae24-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:29:24 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:29:24.496 DEBUG:boto:path=/pklxkd-55/a 2023-05-31T20:29:24.496 DEBUG:boto:auth_path=/pklxkd-55/a 2023-05-31T20:29:24.497 DEBUG:boto:path=/pklxkd-55/a?versionId=null 2023-05-31T20:29:24.497 DEBUG:boto:auth_path=/pklxkd-55/a?versionId=null 2023-05-31T20:29:24.497 DEBUG:boto:Method: HEAD 2023-05-31T20:29:24.498 DEBUG:boto:Path: /pklxkd-55/a?versionId=null 2023-05-31T20:29:24.498 DEBUG:boto:Data: 2023-05-31T20:29:24.498 DEBUG:boto:Headers: {} 2023-05-31T20:29:24.499 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:29:24.499 DEBUG:boto:Port: 8001 2023-05-31T20:29:24.500 DEBUG:boto:Params: {} 2023-05-31T20:29:24.500 DEBUG:boto:Token: None 2023-05-31T20:29:24.501 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:29:24 GMT /pklxkd-55/a?versionId=null 2023-05-31T20:29:24.501 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:IU6+aQzj/zu/ThVGrkWnaMaj3/Y= 2023-05-31T20:29:24.502 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:29:24 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:IU6+aQzj/zu/ThVGrkWnaMaj3/Y=', 'Content-Length': '0'} 2023-05-31T20:29:24.503 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:27:16 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000053a87f26e7c712ae-006477ae24-162e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:29:24 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:29:24.504 DEBUG:boto:path=/pklxkd-55/a 2023-05-31T20:29:24.505 DEBUG:boto:auth_path=/pklxkd-55/a 2023-05-31T20:29:24.505 DEBUG:boto:path=/pklxkd-55/a?versionId=null 2023-05-31T20:29:24.506 DEBUG:boto:auth_path=/pklxkd-55/a?versionId=null 2023-05-31T20:29:24.507 DEBUG:boto:Method: HEAD 2023-05-31T20:29:24.507 DEBUG:boto:Path: /pklxkd-55/a?versionId=null 2023-05-31T20:29:24.508 DEBUG:boto:Data: 2023-05-31T20:29:24.509 DEBUG:boto:Headers: {} 2023-05-31T20:29:24.509 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:29:24.509 DEBUG:boto:Port: 8000 2023-05-31T20:29:24.510 DEBUG:boto:Params: {} 2023-05-31T20:29:24.511 DEBUG:boto:Token: None 2023-05-31T20:29:24.511 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:29:24 GMT /pklxkd-55/a?versionId=null 2023-05-31T20:29:24.512 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:IU6+aQzj/zu/ThVGrkWnaMaj3/Y= 2023-05-31T20:29:24.513 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:29:24 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:IU6+aQzj/zu/ThVGrkWnaMaj3/Y=', 'Content-Length': '0'} 2023-05-31T20:29:24.514 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:27:16 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000098ccbb35eefd27e2-006477ae24-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:29:24 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:29:24.515 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=a 2023-05-31T20:29:24.515 DEBUG:boto:path=/pklxkd-55/a 2023-05-31T20:29:24.516 DEBUG:boto:auth_path=/pklxkd-55/a 2023-05-31T20:29:24.516 DEBUG:boto:path=/pklxkd-55/a?versionId=null 2023-05-31T20:29:24.517 DEBUG:boto:auth_path=/pklxkd-55/a?versionId=null 2023-05-31T20:29:24.517 DEBUG:boto:Method: GET 2023-05-31T20:29:24.518 DEBUG:boto:Path: /pklxkd-55/a?versionId=null 2023-05-31T20:29:24.518 DEBUG:boto:Data: 2023-05-31T20:29:24.519 DEBUG:boto:Headers: {} 2023-05-31T20:29:24.519 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:29:24.519 DEBUG:boto:Port: 8001 2023-05-31T20:29:24.520 DEBUG:boto:Params: {} 2023-05-31T20:29:24.521 DEBUG:boto:Token: None 2023-05-31T20:29:24.522 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:29:24 GMT /pklxkd-55/a?versionId=null 2023-05-31T20:29:24.522 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:+X92hov5hckyw41lnTNPdKmNFYQ= 2023-05-31T20:29:24.523 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:29:24 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:+X92hov5hckyw41lnTNPdKmNFYQ=', 'Content-Length': '0'} 2023-05-31T20:29:24.525 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:27:16 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001bd9c2affb92b517-006477ae24-162e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:29:24 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:29:24.526 DEBUG:boto:path=/pklxkd-55/a 2023-05-31T20:29:24.526 DEBUG:boto:auth_path=/pklxkd-55/a 2023-05-31T20:29:24.527 DEBUG:boto:path=/pklxkd-55/a?versionId=null 2023-05-31T20:29:24.527 DEBUG:boto:auth_path=/pklxkd-55/a?versionId=null 2023-05-31T20:29:24.527 DEBUG:boto:Method: GET 2023-05-31T20:29:24.528 DEBUG:boto:Path: /pklxkd-55/a?versionId=null 2023-05-31T20:29:24.528 DEBUG:boto:Data: 2023-05-31T20:29:24.529 DEBUG:boto:Headers: {} 2023-05-31T20:29:24.529 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:29:24.530 DEBUG:boto:Port: 8000 2023-05-31T20:29:24.530 DEBUG:boto:Params: {} 2023-05-31T20:29:24.531 DEBUG:boto:Token: None 2023-05-31T20:29:24.531 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:29:24 GMT /pklxkd-55/a?versionId=null 2023-05-31T20:29:24.531 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:+X92hov5hckyw41lnTNPdKmNFYQ= 2023-05-31T20:29:24.532 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:29:24 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:+X92hov5hckyw41lnTNPdKmNFYQ=', 'Content-Length': '0'} 2023-05-31T20:29:24.534 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:27:16 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003f5b375ffdb4e608-006477ae24-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:29:24 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:29:24.535 DEBUG:boto:path=/pklxkd-55/ 2023-05-31T20:29:24.536 DEBUG:boto:auth_path=/pklxkd-55/ 2023-05-31T20:29:24.536 DEBUG:boto:path=/pklxkd-55/?prefix=a 2023-05-31T20:29:24.537 DEBUG:boto:auth_path=/pklxkd-55/?prefix=a 2023-05-31T20:29:24.537 DEBUG:boto:Method: GET 2023-05-31T20:29:24.538 DEBUG:boto:Path: /pklxkd-55/?prefix=a 2023-05-31T20:29:24.538 DEBUG:boto:Data: 2023-05-31T20:29:24.538 DEBUG:boto:Headers: {} 2023-05-31T20:29:24.539 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:29:24.539 DEBUG:boto:Port: 8001 2023-05-31T20:29:24.539 DEBUG:boto:Params: {} 2023-05-31T20:29:24.540 DEBUG:boto:Token: None 2023-05-31T20:29:24.540 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:29:24 GMT /pklxkd-55/ 2023-05-31T20:29:24.541 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:xCRwQ8iFa3tX64tuk2Bv1zIjyBc= 2023-05-31T20:29:24.541 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:29:24 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:xCRwQ8iFa3tX64tuk2Bv1zIjyBc=', 'Content-Length': '0'} 2023-05-31T20:29:24.543 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000002679f89489768330-006477ae24-162e-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:29:24 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:29:24.544 DEBUG:boto:b'pklxkd-55a1000falsea2023-05-31T20:27:16.792Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:29:24.545 DEBUG:boto:path=/pklxkd-55/ 2023-05-31T20:29:24.546 DEBUG:boto:auth_path=/pklxkd-55/ 2023-05-31T20:29:24.546 DEBUG:boto:path=/pklxkd-55/?prefix=a 2023-05-31T20:29:24.547 DEBUG:boto:auth_path=/pklxkd-55/?prefix=a 2023-05-31T20:29:24.549 DEBUG:boto:Method: GET 2023-05-31T20:29:24.549 DEBUG:boto:Path: /pklxkd-55/?prefix=a 2023-05-31T20:29:24.550 DEBUG:boto:Data: 2023-05-31T20:29:24.550 DEBUG:boto:Headers: {} 2023-05-31T20:29:24.551 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:29:24.551 DEBUG:boto:Port: 8000 2023-05-31T20:29:24.552 DEBUG:boto:Params: {} 2023-05-31T20:29:24.553 DEBUG:boto:Token: None 2023-05-31T20:29:24.553 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:29:24 GMT /pklxkd-55/ 2023-05-31T20:29:24.554 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:xCRwQ8iFa3tX64tuk2Bv1zIjyBc= 2023-05-31T20:29:24.554 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:29:24 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:xCRwQ8iFa3tX64tuk2Bv1zIjyBc=', 'Content-Length': '0'} 2023-05-31T20:29:24.557 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d3ccdf260f6c5cb9-006477ae24-1630-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:29:24 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:29:24.557 DEBUG:boto:b'pklxkd-55a1000falsea2023-05-31T20:27:16.792Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:29:24.558 DEBUG:boto:path=/pklxkd-55/a 2023-05-31T20:29:24.559 DEBUG:boto:auth_path=/pklxkd-55/a 2023-05-31T20:29:24.560 DEBUG:boto:Method: HEAD 2023-05-31T20:29:24.560 DEBUG:boto:Path: /pklxkd-55/a 2023-05-31T20:29:24.561 DEBUG:boto:Data: 2023-05-31T20:29:24.561 DEBUG:boto:Headers: {} 2023-05-31T20:29:24.561 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:29:24.562 DEBUG:boto:Port: 8001 2023-05-31T20:29:24.562 DEBUG:boto:Params: {} 2023-05-31T20:29:24.562 DEBUG:boto:Token: None 2023-05-31T20:29:24.563 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:29:24 GMT /pklxkd-55/a 2023-05-31T20:29:24.563 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:z2RC3sR/xfZmAcFYCxjPlVzZ6Q4= 2023-05-31T20:29:24.564 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:29:24 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:z2RC3sR/xfZmAcFYCxjPlVzZ6Q4=', 'Content-Length': '0'} 2023-05-31T20:29:24.565 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:27:16 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005d208bb192e5dd93-006477ae24-162e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:29:24 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:29:24.566 DEBUG:boto:path=/pklxkd-55/a 2023-05-31T20:29:24.566 DEBUG:boto:auth_path=/pklxkd-55/a 2023-05-31T20:29:24.567 DEBUG:boto:Method: HEAD 2023-05-31T20:29:24.567 DEBUG:boto:Path: /pklxkd-55/a 2023-05-31T20:29:24.568 DEBUG:boto:Data: 2023-05-31T20:29:24.568 DEBUG:boto:Headers: {} 2023-05-31T20:29:24.569 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:29:24.569 DEBUG:boto:Port: 8000 2023-05-31T20:29:24.569 DEBUG:boto:Params: {} 2023-05-31T20:29:24.570 DEBUG:boto:Token: None 2023-05-31T20:29:24.570 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:29:24 GMT /pklxkd-55/a 2023-05-31T20:29:24.571 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:z2RC3sR/xfZmAcFYCxjPlVzZ6Q4= 2023-05-31T20:29:24.571 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:29:24 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:z2RC3sR/xfZmAcFYCxjPlVzZ6Q4=', 'Content-Length': '0'} 2023-05-31T20:29:24.573 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:27:16 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000205edb67e5b206ea-006477ae24-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:29:24 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:29:24.574 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=a 2023-05-31T20:29:24.574 DEBUG:boto:path=/pklxkd-55/a 2023-05-31T20:29:24.575 DEBUG:boto:auth_path=/pklxkd-55/a 2023-05-31T20:29:24.575 DEBUG:boto:Method: GET 2023-05-31T20:29:24.575 DEBUG:boto:Path: /pklxkd-55/a 2023-05-31T20:29:24.576 DEBUG:boto:Data: 2023-05-31T20:29:24.576 DEBUG:boto:Headers: {} 2023-05-31T20:29:24.577 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:29:24.577 DEBUG:boto:Port: 8001 2023-05-31T20:29:24.578 DEBUG:boto:Params: {} 2023-05-31T20:29:24.578 DEBUG:boto:Token: None 2023-05-31T20:29:24.579 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:29:24 GMT /pklxkd-55/a 2023-05-31T20:29:24.579 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:vtnGMPFvk2E8cTyk3oTWDuCLguU= 2023-05-31T20:29:24.580 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:29:24 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:vtnGMPFvk2E8cTyk3oTWDuCLguU=', 'Content-Length': '0'} 2023-05-31T20:29:24.583 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:27:16 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005fc24de09c060edc-006477ae24-162e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:29:24 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:29:24.584 DEBUG:boto:path=/pklxkd-55/a 2023-05-31T20:29:24.585 DEBUG:boto:auth_path=/pklxkd-55/a 2023-05-31T20:29:24.585 DEBUG:boto:Method: GET 2023-05-31T20:29:24.586 DEBUG:boto:Path: /pklxkd-55/a 2023-05-31T20:29:24.586 DEBUG:boto:Data: 2023-05-31T20:29:24.586 DEBUG:boto:Headers: {} 2023-05-31T20:29:24.587 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:29:24.587 DEBUG:boto:Port: 8000 2023-05-31T20:29:24.588 DEBUG:boto:Params: {} 2023-05-31T20:29:24.589 DEBUG:boto:Token: None 2023-05-31T20:29:24.589 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:29:24 GMT /pklxkd-55/a 2023-05-31T20:29:24.590 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:vtnGMPFvk2E8cTyk3oTWDuCLguU= 2023-05-31T20:29:24.590 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:29:24 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:vtnGMPFvk2E8cTyk3oTWDuCLguU=', 'Content-Length': '0'} 2023-05-31T20:29:24.592 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:27:16 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ce6371bb8bf536e3-006477ae24-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:29:24 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:29:24.632 DEBUG:boto:path=/pklxkd-55/ 2023-05-31T20:29:24.633 DEBUG:boto:auth_path=/pklxkd-55/ 2023-05-31T20:29:24.633 DEBUG:boto:path=/pklxkd-55/?prefix=a 2023-05-31T20:29:24.634 DEBUG:boto:auth_path=/pklxkd-55/?prefix=a 2023-05-31T20:29:24.634 DEBUG:boto:Method: GET 2023-05-31T20:29:24.634 DEBUG:boto:Path: /pklxkd-55/?prefix=a 2023-05-31T20:29:24.635 DEBUG:boto:Data: 2023-05-31T20:29:24.635 DEBUG:boto:Headers: {} 2023-05-31T20:29:24.635 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:29:24.636 DEBUG:boto:Port: 8001 2023-05-31T20:29:24.636 DEBUG:boto:Params: {} 2023-05-31T20:29:24.637 DEBUG:boto:Token: None 2023-05-31T20:29:24.637 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:29:24 GMT /pklxkd-55/ 2023-05-31T20:29:24.637 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:xCRwQ8iFa3tX64tuk2Bv1zIjyBc= 2023-05-31T20:29:24.638 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:29:24 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:xCRwQ8iFa3tX64tuk2Bv1zIjyBc=', 'Content-Length': '0'} 2023-05-31T20:29:24.640 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000eb2fead78109ec2b-006477ae24-162e-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:29:24 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:29:24.641 DEBUG:boto:b'pklxkd-55a1000falsea2023-05-31T20:27:16.792Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:29:24.642 DEBUG:boto:path=/pklxkd-55/ 2023-05-31T20:29:24.642 DEBUG:boto:auth_path=/pklxkd-55/ 2023-05-31T20:29:24.642 DEBUG:boto:path=/pklxkd-55/?prefix=a 2023-05-31T20:29:24.643 DEBUG:boto:auth_path=/pklxkd-55/?prefix=a 2023-05-31T20:29:24.643 DEBUG:boto:Method: GET 2023-05-31T20:29:24.643 DEBUG:boto:Path: /pklxkd-55/?prefix=a 2023-05-31T20:29:24.644 DEBUG:boto:Data: 2023-05-31T20:29:24.644 DEBUG:boto:Headers: {} 2023-05-31T20:29:24.644 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:29:24.645 DEBUG:boto:Port: 8000 2023-05-31T20:29:24.645 DEBUG:boto:Params: {} 2023-05-31T20:29:24.646 DEBUG:boto:Token: None 2023-05-31T20:29:24.646 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:29:24 GMT /pklxkd-55/ 2023-05-31T20:29:24.646 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:xCRwQ8iFa3tX64tuk2Bv1zIjyBc= 2023-05-31T20:29:24.647 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:29:24 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:xCRwQ8iFa3tX64tuk2Bv1zIjyBc=', 'Content-Length': '0'} 2023-05-31T20:29:24.649 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000004e9910af8bc9f872-006477ae24-1630-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:29:24 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:29:24.650 DEBUG:boto:b'pklxkd-55a1000falsea2023-05-31T20:27:16.792Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:29:24.650 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=b 2023-05-31T20:29:24.651 DEBUG:boto:path=/pklxkd-55/b 2023-05-31T20:29:24.651 DEBUG:boto:auth_path=/pklxkd-55/b 2023-05-31T20:29:24.651 DEBUG:boto:path=/pklxkd-55/b?versionId=null 2023-05-31T20:29:24.652 DEBUG:boto:auth_path=/pklxkd-55/b?versionId=null 2023-05-31T20:29:24.652 DEBUG:boto:Method: GET 2023-05-31T20:29:24.652 DEBUG:boto:Path: /pklxkd-55/b?versionId=null 2023-05-31T20:29:24.653 DEBUG:boto:Data: 2023-05-31T20:29:24.653 DEBUG:boto:Headers: {} 2023-05-31T20:29:24.654 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:29:24.654 DEBUG:boto:Port: 8001 2023-05-31T20:29:24.654 DEBUG:boto:Params: {} 2023-05-31T20:29:24.655 DEBUG:boto:Token: None 2023-05-31T20:29:24.655 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:29:24 GMT /pklxkd-55/b?versionId=null 2023-05-31T20:29:24.655 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:ju3OtKkKU11EzgC45iZF4622yUc= 2023-05-31T20:29:24.656 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:29:24 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:ju3OtKkKU11EzgC45iZF4622yUc=', 'Content-Length': '0'} 2023-05-31T20:29:24.658 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:27:16 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000700d36d0a6e044f2-006477ae24-162e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:29:24 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:29:24.700 DEBUG:boto:path=/pklxkd-55/b 2023-05-31T20:29:24.701 DEBUG:boto:auth_path=/pklxkd-55/b 2023-05-31T20:29:24.701 DEBUG:boto:path=/pklxkd-55/b?versionId=null 2023-05-31T20:29:24.702 DEBUG:boto:auth_path=/pklxkd-55/b?versionId=null 2023-05-31T20:29:24.702 DEBUG:boto:Method: GET 2023-05-31T20:29:24.703 DEBUG:boto:Path: /pklxkd-55/b?versionId=null 2023-05-31T20:29:24.703 DEBUG:boto:Data: 2023-05-31T20:29:24.704 DEBUG:boto:Headers: {} 2023-05-31T20:29:24.704 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:29:24.705 DEBUG:boto:Port: 8000 2023-05-31T20:29:24.705 DEBUG:boto:Params: {} 2023-05-31T20:29:24.706 DEBUG:boto:Token: None 2023-05-31T20:29:24.706 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:29:24 GMT /pklxkd-55/b?versionId=null 2023-05-31T20:29:24.707 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:ju3OtKkKU11EzgC45iZF4622yUc= 2023-05-31T20:29:24.707 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:29:24 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:ju3OtKkKU11EzgC45iZF4622yUc=', 'Content-Length': '0'} 2023-05-31T20:29:24.709 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:27:16 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003192e70000ac7c25-006477ae24-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:29:24 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:29:24.710 DEBUG:boto:path=/pklxkd-55/b 2023-05-31T20:29:24.710 DEBUG:boto:auth_path=/pklxkd-55/b 2023-05-31T20:29:24.711 DEBUG:boto:path=/pklxkd-55/b?versionId=null 2023-05-31T20:29:24.711 DEBUG:boto:auth_path=/pklxkd-55/b?versionId=null 2023-05-31T20:29:24.711 DEBUG:boto:Method: HEAD 2023-05-31T20:29:24.712 DEBUG:boto:Path: /pklxkd-55/b?versionId=null 2023-05-31T20:29:24.713 DEBUG:boto:Data: 2023-05-31T20:29:24.713 DEBUG:boto:Headers: {} 2023-05-31T20:29:24.714 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:29:24.715 DEBUG:boto:Port: 8001 2023-05-31T20:29:24.716 DEBUG:boto:Params: {} 2023-05-31T20:29:24.716 DEBUG:boto:Token: None 2023-05-31T20:29:24.717 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:29:24 GMT /pklxkd-55/b?versionId=null 2023-05-31T20:29:24.718 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:wjpFMkgIyOzOucNXWIg1exaYuVQ= 2023-05-31T20:29:24.719 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:29:24 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:wjpFMkgIyOzOucNXWIg1exaYuVQ=', 'Content-Length': '0'} 2023-05-31T20:29:24.721 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:27:16 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d44028c8af6e0f4e-006477ae24-162e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:29:24 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:29:24.721 DEBUG:boto:path=/pklxkd-55/b 2023-05-31T20:29:24.722 DEBUG:boto:auth_path=/pklxkd-55/b 2023-05-31T20:29:24.722 DEBUG:boto:path=/pklxkd-55/b?versionId=null 2023-05-31T20:29:24.723 DEBUG:boto:auth_path=/pklxkd-55/b?versionId=null 2023-05-31T20:29:24.723 DEBUG:boto:Method: HEAD 2023-05-31T20:29:24.723 DEBUG:boto:Path: /pklxkd-55/b?versionId=null 2023-05-31T20:29:24.724 DEBUG:boto:Data: 2023-05-31T20:29:24.724 DEBUG:boto:Headers: {} 2023-05-31T20:29:24.725 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:29:24.725 DEBUG:boto:Port: 8000 2023-05-31T20:29:24.725 DEBUG:boto:Params: {} 2023-05-31T20:29:24.726 DEBUG:boto:Token: None 2023-05-31T20:29:24.726 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:29:24 GMT /pklxkd-55/b?versionId=null 2023-05-31T20:29:24.727 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:wjpFMkgIyOzOucNXWIg1exaYuVQ= 2023-05-31T20:29:24.727 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:29:24 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:wjpFMkgIyOzOucNXWIg1exaYuVQ=', 'Content-Length': '0'} 2023-05-31T20:29:24.729 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:27:16 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007787216d332c7500-006477ae24-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:29:24 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:29:24.730 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=b 2023-05-31T20:29:24.730 DEBUG:boto:path=/pklxkd-55/b 2023-05-31T20:29:24.731 DEBUG:boto:auth_path=/pklxkd-55/b 2023-05-31T20:29:24.731 DEBUG:boto:path=/pklxkd-55/b?versionId=null 2023-05-31T20:29:24.732 DEBUG:boto:auth_path=/pklxkd-55/b?versionId=null 2023-05-31T20:29:24.732 DEBUG:boto:Method: GET 2023-05-31T20:29:24.732 DEBUG:boto:Path: /pklxkd-55/b?versionId=null 2023-05-31T20:29:24.733 DEBUG:boto:Data: 2023-05-31T20:29:24.733 DEBUG:boto:Headers: {} 2023-05-31T20:29:24.733 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:29:24.734 DEBUG:boto:Port: 8001 2023-05-31T20:29:24.734 DEBUG:boto:Params: {} 2023-05-31T20:29:24.735 DEBUG:boto:Token: None 2023-05-31T20:29:24.735 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:29:24 GMT /pklxkd-55/b?versionId=null 2023-05-31T20:29:24.735 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:ju3OtKkKU11EzgC45iZF4622yUc= 2023-05-31T20:29:24.736 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:29:24 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:ju3OtKkKU11EzgC45iZF4622yUc=', 'Content-Length': '0'} 2023-05-31T20:29:24.738 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:27:16 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000293b444a3f1448fa-006477ae24-162e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:29:24 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:29:24.738 DEBUG:boto:path=/pklxkd-55/b 2023-05-31T20:29:24.739 DEBUG:boto:auth_path=/pklxkd-55/b 2023-05-31T20:29:24.739 DEBUG:boto:path=/pklxkd-55/b?versionId=null 2023-05-31T20:29:24.739 DEBUG:boto:auth_path=/pklxkd-55/b?versionId=null 2023-05-31T20:29:24.740 DEBUG:boto:Method: GET 2023-05-31T20:29:24.740 DEBUG:boto:Path: /pklxkd-55/b?versionId=null 2023-05-31T20:29:24.741 DEBUG:boto:Data: 2023-05-31T20:29:24.741 DEBUG:boto:Headers: {} 2023-05-31T20:29:24.741 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:29:24.742 DEBUG:boto:Port: 8000 2023-05-31T20:29:24.742 DEBUG:boto:Params: {} 2023-05-31T20:29:24.742 DEBUG:boto:Token: None 2023-05-31T20:29:24.743 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:29:24 GMT /pklxkd-55/b?versionId=null 2023-05-31T20:29:24.743 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:ju3OtKkKU11EzgC45iZF4622yUc= 2023-05-31T20:29:24.744 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:29:24 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:ju3OtKkKU11EzgC45iZF4622yUc=', 'Content-Length': '0'} 2023-05-31T20:29:24.746 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:27:16 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001a139aa291cc36de-006477ae24-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:29:24 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:29:24.792 DEBUG:boto:path=/pklxkd-55/ 2023-05-31T20:29:24.793 DEBUG:boto:auth_path=/pklxkd-55/ 2023-05-31T20:29:24.793 DEBUG:boto:path=/pklxkd-55/?prefix=b 2023-05-31T20:29:24.794 DEBUG:boto:auth_path=/pklxkd-55/?prefix=b 2023-05-31T20:29:24.794 DEBUG:boto:Method: GET 2023-05-31T20:29:24.794 DEBUG:boto:Path: /pklxkd-55/?prefix=b 2023-05-31T20:29:24.795 DEBUG:boto:Data: 2023-05-31T20:29:24.795 DEBUG:boto:Headers: {} 2023-05-31T20:29:24.795 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:29:24.796 DEBUG:boto:Port: 8001 2023-05-31T20:29:24.796 DEBUG:boto:Params: {} 2023-05-31T20:29:24.797 DEBUG:boto:Token: None 2023-05-31T20:29:24.797 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:29:24 GMT /pklxkd-55/ 2023-05-31T20:29:24.798 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:xCRwQ8iFa3tX64tuk2Bv1zIjyBc= 2023-05-31T20:29:24.798 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:29:24 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:xCRwQ8iFa3tX64tuk2Bv1zIjyBc=', 'Content-Length': '0'} 2023-05-31T20:29:24.800 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000047697f32a096ebf1-006477ae24-162e-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:29:24 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:29:24.801 DEBUG:boto:b'pklxkd-55b1000falseb2023-05-31T20:27:16.850Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:29:24.802 DEBUG:boto:path=/pklxkd-55/ 2023-05-31T20:29:24.802 DEBUG:boto:auth_path=/pklxkd-55/ 2023-05-31T20:29:24.802 DEBUG:boto:path=/pklxkd-55/?prefix=b 2023-05-31T20:29:24.803 DEBUG:boto:auth_path=/pklxkd-55/?prefix=b 2023-05-31T20:29:24.803 DEBUG:boto:Method: GET 2023-05-31T20:29:24.803 DEBUG:boto:Path: /pklxkd-55/?prefix=b 2023-05-31T20:29:24.804 DEBUG:boto:Data: 2023-05-31T20:29:24.804 DEBUG:boto:Headers: {} 2023-05-31T20:29:24.804 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:29:24.805 DEBUG:boto:Port: 8000 2023-05-31T20:29:24.806 DEBUG:boto:Params: {} 2023-05-31T20:29:24.807 DEBUG:boto:Token: None 2023-05-31T20:29:24.807 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:29:24 GMT /pklxkd-55/ 2023-05-31T20:29:24.808 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:xCRwQ8iFa3tX64tuk2Bv1zIjyBc= 2023-05-31T20:29:24.808 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:29:24 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:xCRwQ8iFa3tX64tuk2Bv1zIjyBc=', 'Content-Length': '0'} 2023-05-31T20:29:24.810 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b36c382bbe0985f3-006477ae24-1630-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:29:24 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:29:24.811 DEBUG:boto:b'pklxkd-55b1000falseb2023-05-31T20:27:16.850Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:29:24.812 DEBUG:boto:path=/pklxkd-55/b 2023-05-31T20:29:24.812 DEBUG:boto:auth_path=/pklxkd-55/b 2023-05-31T20:29:24.812 DEBUG:boto:Method: HEAD 2023-05-31T20:29:24.813 DEBUG:boto:Path: /pklxkd-55/b 2023-05-31T20:29:24.813 DEBUG:boto:Data: 2023-05-31T20:29:24.813 DEBUG:boto:Headers: {} 2023-05-31T20:29:24.814 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:29:24.814 DEBUG:boto:Port: 8001 2023-05-31T20:29:24.814 DEBUG:boto:Params: {} 2023-05-31T20:29:24.815 DEBUG:boto:Token: None 2023-05-31T20:29:24.815 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:29:24 GMT /pklxkd-55/b 2023-05-31T20:29:24.816 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:1aG6/dzrbbfZto0Qb8afOlzk78c= 2023-05-31T20:29:24.816 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:29:24 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:1aG6/dzrbbfZto0Qb8afOlzk78c=', 'Content-Length': '0'} 2023-05-31T20:29:24.818 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:27:16 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d3f6d06111fd48f2-006477ae24-162e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:29:24 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:29:24.819 DEBUG:boto:path=/pklxkd-55/b 2023-05-31T20:29:24.819 DEBUG:boto:auth_path=/pklxkd-55/b 2023-05-31T20:29:24.820 DEBUG:boto:Method: HEAD 2023-05-31T20:29:24.820 DEBUG:boto:Path: /pklxkd-55/b 2023-05-31T20:29:24.820 DEBUG:boto:Data: 2023-05-31T20:29:24.821 DEBUG:boto:Headers: {} 2023-05-31T20:29:24.821 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:29:24.821 DEBUG:boto:Port: 8000 2023-05-31T20:29:24.822 DEBUG:boto:Params: {} 2023-05-31T20:29:24.822 DEBUG:boto:Token: None 2023-05-31T20:29:24.822 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:29:24 GMT /pklxkd-55/b 2023-05-31T20:29:24.823 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:1aG6/dzrbbfZto0Qb8afOlzk78c= 2023-05-31T20:29:24.823 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:29:24 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:1aG6/dzrbbfZto0Qb8afOlzk78c=', 'Content-Length': '0'} 2023-05-31T20:29:24.825 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:27:16 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006094b18781d552fa-006477ae24-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:29:24 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:29:24.826 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=b 2023-05-31T20:29:24.826 DEBUG:boto:path=/pklxkd-55/b 2023-05-31T20:29:24.827 DEBUG:boto:auth_path=/pklxkd-55/b 2023-05-31T20:29:24.827 DEBUG:boto:Method: GET 2023-05-31T20:29:24.827 DEBUG:boto:Path: /pklxkd-55/b 2023-05-31T20:29:24.828 DEBUG:boto:Data: 2023-05-31T20:29:24.828 DEBUG:boto:Headers: {} 2023-05-31T20:29:24.828 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:29:24.829 DEBUG:boto:Port: 8001 2023-05-31T20:29:24.829 DEBUG:boto:Params: {} 2023-05-31T20:29:24.829 DEBUG:boto:Token: None 2023-05-31T20:29:24.830 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:29:24 GMT /pklxkd-55/b 2023-05-31T20:29:24.830 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:nRgLIEteHGm2wkSOz/c6odfckCQ= 2023-05-31T20:29:24.832 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:29:24 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:nRgLIEteHGm2wkSOz/c6odfckCQ=', 'Content-Length': '0'} 2023-05-31T20:29:24.834 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:27:16 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000cfcd4cb81f9dbad3-006477ae24-162e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:29:24 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:29:24.876 DEBUG:boto:path=/pklxkd-55/b 2023-05-31T20:29:24.877 DEBUG:boto:auth_path=/pklxkd-55/b 2023-05-31T20:29:24.877 DEBUG:boto:Method: GET 2023-05-31T20:29:24.878 DEBUG:boto:Path: /pklxkd-55/b 2023-05-31T20:29:24.878 DEBUG:boto:Data: 2023-05-31T20:29:24.879 DEBUG:boto:Headers: {} 2023-05-31T20:29:24.879 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:29:24.880 DEBUG:boto:Port: 8000 2023-05-31T20:29:24.880 DEBUG:boto:Params: {} 2023-05-31T20:29:24.881 DEBUG:boto:Token: None 2023-05-31T20:29:24.881 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:29:24 GMT /pklxkd-55/b 2023-05-31T20:29:24.882 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:nRgLIEteHGm2wkSOz/c6odfckCQ= 2023-05-31T20:29:24.882 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:29:24 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:nRgLIEteHGm2wkSOz/c6odfckCQ=', 'Content-Length': '0'} 2023-05-31T20:29:24.884 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:27:16 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000081dba989f142aefc-006477ae24-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:29:24 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:29:24.885 DEBUG:boto:path=/pklxkd-55/ 2023-05-31T20:29:24.885 DEBUG:boto:auth_path=/pklxkd-55/ 2023-05-31T20:29:24.886 DEBUG:boto:path=/pklxkd-55/?prefix=b 2023-05-31T20:29:24.886 DEBUG:boto:auth_path=/pklxkd-55/?prefix=b 2023-05-31T20:29:24.887 DEBUG:boto:Method: GET 2023-05-31T20:29:24.887 DEBUG:boto:Path: /pklxkd-55/?prefix=b 2023-05-31T20:29:24.887 DEBUG:boto:Data: 2023-05-31T20:29:24.888 DEBUG:boto:Headers: {} 2023-05-31T20:29:24.888 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:29:24.888 DEBUG:boto:Port: 8001 2023-05-31T20:29:24.889 DEBUG:boto:Params: {} 2023-05-31T20:29:24.889 DEBUG:boto:Token: None 2023-05-31T20:29:24.890 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:29:24 GMT /pklxkd-55/ 2023-05-31T20:29:24.891 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:xCRwQ8iFa3tX64tuk2Bv1zIjyBc= 2023-05-31T20:29:24.891 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:29:24 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:xCRwQ8iFa3tX64tuk2Bv1zIjyBc=', 'Content-Length': '0'} 2023-05-31T20:29:24.893 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000054dfa196f856584d-006477ae24-162e-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:29:24 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:29:24.894 DEBUG:boto:b'pklxkd-55b1000falseb2023-05-31T20:27:16.850Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:29:24.895 DEBUG:boto:path=/pklxkd-55/ 2023-05-31T20:29:24.895 DEBUG:boto:auth_path=/pklxkd-55/ 2023-05-31T20:29:24.895 DEBUG:boto:path=/pklxkd-55/?prefix=b 2023-05-31T20:29:24.896 DEBUG:boto:auth_path=/pklxkd-55/?prefix=b 2023-05-31T20:29:24.896 DEBUG:boto:Method: GET 2023-05-31T20:29:24.897 DEBUG:boto:Path: /pklxkd-55/?prefix=b 2023-05-31T20:29:24.897 DEBUG:boto:Data: 2023-05-31T20:29:24.897 DEBUG:boto:Headers: {} 2023-05-31T20:29:24.898 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:29:24.898 DEBUG:boto:Port: 8000 2023-05-31T20:29:24.899 DEBUG:boto:Params: {} 2023-05-31T20:29:24.899 DEBUG:boto:Token: None 2023-05-31T20:29:24.899 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:29:24 GMT /pklxkd-55/ 2023-05-31T20:29:24.900 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:xCRwQ8iFa3tX64tuk2Bv1zIjyBc= 2023-05-31T20:29:24.900 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:29:24 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:xCRwQ8iFa3tX64tuk2Bv1zIjyBc=', 'Content-Length': '0'} 2023-05-31T20:29:24.902 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000013dad2d637250b85-006477ae24-1630-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:29:24 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:29:24.903 DEBUG:boto:b'pklxkd-55b1000falseb2023-05-31T20:27:16.850Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:29:24.904 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=c 2023-05-31T20:29:24.904 DEBUG:boto:path=/pklxkd-55/c 2023-05-31T20:29:24.905 DEBUG:boto:auth_path=/pklxkd-55/c 2023-05-31T20:29:24.905 DEBUG:boto:path=/pklxkd-55/c?versionId=null 2023-05-31T20:29:24.905 DEBUG:boto:auth_path=/pklxkd-55/c?versionId=null 2023-05-31T20:29:24.906 DEBUG:boto:Method: GET 2023-05-31T20:29:24.906 DEBUG:boto:Path: /pklxkd-55/c?versionId=null 2023-05-31T20:29:24.907 DEBUG:boto:Data: 2023-05-31T20:29:24.907 DEBUG:boto:Headers: {} 2023-05-31T20:29:24.907 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:29:24.908 DEBUG:boto:Port: 8001 2023-05-31T20:29:24.908 DEBUG:boto:Params: {} 2023-05-31T20:29:24.909 DEBUG:boto:Token: None 2023-05-31T20:29:24.909 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:29:24 GMT /pklxkd-55/c?versionId=null 2023-05-31T20:29:24.910 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:dufJB3ZebW9SblWR6TqGAFspARc= 2023-05-31T20:29:24.910 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:29:24 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:dufJB3ZebW9SblWR6TqGAFspARc=', 'Content-Length': '0'} 2023-05-31T20:29:24.912 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:27:16 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000016770863e7b3f199-006477ae24-162e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:29:24 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:29:24.913 DEBUG:boto:path=/pklxkd-55/c 2023-05-31T20:29:24.913 DEBUG:boto:auth_path=/pklxkd-55/c 2023-05-31T20:29:24.914 DEBUG:boto:path=/pklxkd-55/c?versionId=null 2023-05-31T20:29:24.914 DEBUG:boto:auth_path=/pklxkd-55/c?versionId=null 2023-05-31T20:29:24.914 DEBUG:boto:Method: GET 2023-05-31T20:29:24.915 DEBUG:boto:Path: /pklxkd-55/c?versionId=null 2023-05-31T20:29:24.915 DEBUG:boto:Data: 2023-05-31T20:29:24.915 DEBUG:boto:Headers: {} 2023-05-31T20:29:24.916 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:29:24.916 DEBUG:boto:Port: 8000 2023-05-31T20:29:24.917 DEBUG:boto:Params: {} 2023-05-31T20:29:24.917 DEBUG:boto:Token: None 2023-05-31T20:29:24.918 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:29:24 GMT /pklxkd-55/c?versionId=null 2023-05-31T20:29:24.918 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:dufJB3ZebW9SblWR6TqGAFspARc= 2023-05-31T20:29:24.918 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:29:24 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:dufJB3ZebW9SblWR6TqGAFspARc=', 'Content-Length': '0'} 2023-05-31T20:29:24.921 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:27:16 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005a7d3062b7f40908-006477ae24-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:29:24 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:29:24.921 DEBUG:boto:path=/pklxkd-55/c 2023-05-31T20:29:24.922 DEBUG:boto:auth_path=/pklxkd-55/c 2023-05-31T20:29:24.922 DEBUG:boto:path=/pklxkd-55/c?versionId=null 2023-05-31T20:29:24.923 DEBUG:boto:auth_path=/pklxkd-55/c?versionId=null 2023-05-31T20:29:24.923 DEBUG:boto:Method: HEAD 2023-05-31T20:29:24.923 DEBUG:boto:Path: /pklxkd-55/c?versionId=null 2023-05-31T20:29:24.924 DEBUG:boto:Data: 2023-05-31T20:29:24.924 DEBUG:boto:Headers: {} 2023-05-31T20:29:24.925 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:29:24.925 DEBUG:boto:Port: 8001 2023-05-31T20:29:24.925 DEBUG:boto:Params: {} 2023-05-31T20:29:24.926 DEBUG:boto:Token: None 2023-05-31T20:29:24.926 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:29:24 GMT /pklxkd-55/c?versionId=null 2023-05-31T20:29:24.927 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:JDvQi560BJAmCLq/cVNOrKkPhDc= 2023-05-31T20:29:24.927 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:29:24 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:JDvQi560BJAmCLq/cVNOrKkPhDc=', 'Content-Length': '0'} 2023-05-31T20:29:24.929 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:27:16 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f4585029d29076e9-006477ae24-162e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:29:24 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:29:24.930 DEBUG:boto:path=/pklxkd-55/c 2023-05-31T20:29:24.930 DEBUG:boto:auth_path=/pklxkd-55/c 2023-05-31T20:29:24.931 DEBUG:boto:path=/pklxkd-55/c?versionId=null 2023-05-31T20:29:24.931 DEBUG:boto:auth_path=/pklxkd-55/c?versionId=null 2023-05-31T20:29:24.931 DEBUG:boto:Method: HEAD 2023-05-31T20:29:24.932 DEBUG:boto:Path: /pklxkd-55/c?versionId=null 2023-05-31T20:29:24.932 DEBUG:boto:Data: 2023-05-31T20:29:24.933 DEBUG:boto:Headers: {} 2023-05-31T20:29:24.933 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:29:24.933 DEBUG:boto:Port: 8000 2023-05-31T20:29:24.934 DEBUG:boto:Params: {} 2023-05-31T20:29:24.934 DEBUG:boto:Token: None 2023-05-31T20:29:24.934 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:29:24 GMT /pklxkd-55/c?versionId=null 2023-05-31T20:29:24.935 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:JDvQi560BJAmCLq/cVNOrKkPhDc= 2023-05-31T20:29:24.935 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:29:24 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:JDvQi560BJAmCLq/cVNOrKkPhDc=', 'Content-Length': '0'} 2023-05-31T20:29:24.937 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:27:16 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009df90906199d5cf6-006477ae24-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:29:24 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:29:24.938 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=c 2023-05-31T20:29:24.938 DEBUG:boto:path=/pklxkd-55/c 2023-05-31T20:29:24.939 DEBUG:boto:auth_path=/pklxkd-55/c 2023-05-31T20:29:24.939 DEBUG:boto:path=/pklxkd-55/c?versionId=null 2023-05-31T20:29:24.939 DEBUG:boto:auth_path=/pklxkd-55/c?versionId=null 2023-05-31T20:29:24.940 DEBUG:boto:Method: GET 2023-05-31T20:29:24.940 DEBUG:boto:Path: /pklxkd-55/c?versionId=null 2023-05-31T20:29:24.941 DEBUG:boto:Data: 2023-05-31T20:29:24.942 DEBUG:boto:Headers: {} 2023-05-31T20:29:24.942 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:29:24.942 DEBUG:boto:Port: 8001 2023-05-31T20:29:24.943 DEBUG:boto:Params: {} 2023-05-31T20:29:24.945 DEBUG:boto:Token: None 2023-05-31T20:29:24.945 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:29:24 GMT /pklxkd-55/c?versionId=null 2023-05-31T20:29:24.946 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:dufJB3ZebW9SblWR6TqGAFspARc= 2023-05-31T20:29:24.946 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:29:24 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:dufJB3ZebW9SblWR6TqGAFspARc=', 'Content-Length': '0'} 2023-05-31T20:29:24.948 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:27:16 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006df2c29781c17f50-006477ae24-162e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:29:24 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:29:24.992 DEBUG:boto:path=/pklxkd-55/c 2023-05-31T20:29:24.993 DEBUG:boto:auth_path=/pklxkd-55/c 2023-05-31T20:29:24.993 DEBUG:boto:path=/pklxkd-55/c?versionId=null 2023-05-31T20:29:24.994 DEBUG:boto:auth_path=/pklxkd-55/c?versionId=null 2023-05-31T20:29:24.994 DEBUG:boto:Method: GET 2023-05-31T20:29:24.994 DEBUG:boto:Path: /pklxkd-55/c?versionId=null 2023-05-31T20:29:24.995 DEBUG:boto:Data: 2023-05-31T20:29:24.995 DEBUG:boto:Headers: {} 2023-05-31T20:29:24.995 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:29:24.996 DEBUG:boto:Port: 8000 2023-05-31T20:29:24.996 DEBUG:boto:Params: {} 2023-05-31T20:29:24.997 DEBUG:boto:Token: None 2023-05-31T20:29:24.997 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:29:24 GMT /pklxkd-55/c?versionId=null 2023-05-31T20:29:24.998 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:dufJB3ZebW9SblWR6TqGAFspARc= 2023-05-31T20:29:24.998 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:29:24 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:dufJB3ZebW9SblWR6TqGAFspARc=', 'Content-Length': '0'} 2023-05-31T20:29:25.000 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:27:16 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007cd28dea537dcd90-006477ae24-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:29:24 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:29:25.001 DEBUG:boto:path=/pklxkd-55/ 2023-05-31T20:29:25.001 DEBUG:boto:auth_path=/pklxkd-55/ 2023-05-31T20:29:25.001 DEBUG:boto:path=/pklxkd-55/?prefix=c 2023-05-31T20:29:25.002 DEBUG:boto:auth_path=/pklxkd-55/?prefix=c 2023-05-31T20:29:25.003 DEBUG:boto:Method: GET 2023-05-31T20:29:25.003 DEBUG:boto:Path: /pklxkd-55/?prefix=c 2023-05-31T20:29:25.003 DEBUG:boto:Data: 2023-05-31T20:29:25.004 DEBUG:boto:Headers: {} 2023-05-31T20:29:25.004 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:29:25.004 DEBUG:boto:Port: 8001 2023-05-31T20:29:25.005 DEBUG:boto:Params: {} 2023-05-31T20:29:25.005 DEBUG:boto:Token: None 2023-05-31T20:29:25.006 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:29:25 GMT /pklxkd-55/ 2023-05-31T20:29:25.006 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:a0/qPp0bnZeH62hZJf6OA1a38dw= 2023-05-31T20:29:25.006 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:29:25 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:a0/qPp0bnZeH62hZJf6OA1a38dw=', 'Content-Length': '0'} 2023-05-31T20:29:25.009 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a952419d095d0940-006477ae25-162e-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:29:25 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:29:25.009 DEBUG:boto:b'pklxkd-55c1000falsec2023-05-31T20:27:16.869Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:29:25.010 DEBUG:boto:path=/pklxkd-55/ 2023-05-31T20:29:25.011 DEBUG:boto:auth_path=/pklxkd-55/ 2023-05-31T20:29:25.011 DEBUG:boto:path=/pklxkd-55/?prefix=c 2023-05-31T20:29:25.011 DEBUG:boto:auth_path=/pklxkd-55/?prefix=c 2023-05-31T20:29:25.012 DEBUG:boto:Method: GET 2023-05-31T20:29:25.013 DEBUG:boto:Path: /pklxkd-55/?prefix=c 2023-05-31T20:29:25.013 DEBUG:boto:Data: 2023-05-31T20:29:25.013 DEBUG:boto:Headers: {} 2023-05-31T20:29:25.014 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:29:25.014 DEBUG:boto:Port: 8000 2023-05-31T20:29:25.014 DEBUG:boto:Params: {} 2023-05-31T20:29:25.015 DEBUG:boto:Token: None 2023-05-31T20:29:25.015 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:29:25 GMT /pklxkd-55/ 2023-05-31T20:29:25.016 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:a0/qPp0bnZeH62hZJf6OA1a38dw= 2023-05-31T20:29:25.016 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:29:25 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:a0/qPp0bnZeH62hZJf6OA1a38dw=', 'Content-Length': '0'} 2023-05-31T20:29:25.018 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a62928d117cb770a-006477ae25-1630-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:29:25 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:29:25.019 DEBUG:boto:b'pklxkd-55c1000falsec2023-05-31T20:27:16.869Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:29:25.020 DEBUG:boto:path=/pklxkd-55/c 2023-05-31T20:29:25.020 DEBUG:boto:auth_path=/pklxkd-55/c 2023-05-31T20:29:25.021 DEBUG:boto:Method: HEAD 2023-05-31T20:29:25.021 DEBUG:boto:Path: /pklxkd-55/c 2023-05-31T20:29:25.021 DEBUG:boto:Data: 2023-05-31T20:29:25.022 DEBUG:boto:Headers: {} 2023-05-31T20:29:25.022 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:29:25.022 DEBUG:boto:Port: 8001 2023-05-31T20:29:25.023 DEBUG:boto:Params: {} 2023-05-31T20:29:25.023 DEBUG:boto:Token: None 2023-05-31T20:29:25.024 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:29:25 GMT /pklxkd-55/c 2023-05-31T20:29:25.024 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:qyeZ0BO5kT8gimi9XwzG/GXY2sA= 2023-05-31T20:29:25.025 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:29:25 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:qyeZ0BO5kT8gimi9XwzG/GXY2sA=', 'Content-Length': '0'} 2023-05-31T20:29:25.027 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:27:16 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c2d7afcb31510a06-006477ae25-162e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:29:25 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:29:25.027 DEBUG:boto:path=/pklxkd-55/c 2023-05-31T20:29:25.028 DEBUG:boto:auth_path=/pklxkd-55/c 2023-05-31T20:29:25.028 DEBUG:boto:Method: HEAD 2023-05-31T20:29:25.029 DEBUG:boto:Path: /pklxkd-55/c 2023-05-31T20:29:25.029 DEBUG:boto:Data: 2023-05-31T20:29:25.029 DEBUG:boto:Headers: {} 2023-05-31T20:29:25.030 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:29:25.030 DEBUG:boto:Port: 8000 2023-05-31T20:29:25.031 DEBUG:boto:Params: {} 2023-05-31T20:29:25.031 DEBUG:boto:Token: None 2023-05-31T20:29:25.031 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:29:25 GMT /pklxkd-55/c 2023-05-31T20:29:25.032 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:qyeZ0BO5kT8gimi9XwzG/GXY2sA= 2023-05-31T20:29:25.032 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:29:25 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:qyeZ0BO5kT8gimi9XwzG/GXY2sA=', 'Content-Length': '0'} 2023-05-31T20:29:25.034 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:27:16 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003f1903bc8a258fa7-006477ae25-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:29:25 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:29:25.035 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=c 2023-05-31T20:29:25.035 DEBUG:boto:path=/pklxkd-55/c 2023-05-31T20:29:25.036 DEBUG:boto:auth_path=/pklxkd-55/c 2023-05-31T20:29:25.036 DEBUG:boto:Method: GET 2023-05-31T20:29:25.037 DEBUG:boto:Path: /pklxkd-55/c 2023-05-31T20:29:25.037 DEBUG:boto:Data: 2023-05-31T20:29:25.037 DEBUG:boto:Headers: {} 2023-05-31T20:29:25.038 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:29:25.038 DEBUG:boto:Port: 8001 2023-05-31T20:29:25.039 DEBUG:boto:Params: {} 2023-05-31T20:29:25.039 DEBUG:boto:Token: None 2023-05-31T20:29:25.040 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:29:25 GMT /pklxkd-55/c 2023-05-31T20:29:25.041 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:ZwWUW/OF/WaE6lo6AuNjD64oDWo= 2023-05-31T20:29:25.042 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:29:25 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:ZwWUW/OF/WaE6lo6AuNjD64oDWo=', 'Content-Length': '0'} 2023-05-31T20:29:25.044 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:27:16 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c2ef3fe246dc75c4-006477ae25-162e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:29:25 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:29:25.045 DEBUG:boto:path=/pklxkd-55/c 2023-05-31T20:29:25.045 DEBUG:boto:auth_path=/pklxkd-55/c 2023-05-31T20:29:25.045 DEBUG:boto:Method: GET 2023-05-31T20:29:25.046 DEBUG:boto:Path: /pklxkd-55/c 2023-05-31T20:29:25.046 DEBUG:boto:Data: 2023-05-31T20:29:25.046 DEBUG:boto:Headers: {} 2023-05-31T20:29:25.047 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:29:25.047 DEBUG:boto:Port: 8000 2023-05-31T20:29:25.047 DEBUG:boto:Params: {} 2023-05-31T20:29:25.048 DEBUG:boto:Token: None 2023-05-31T20:29:25.049 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:29:25 GMT /pklxkd-55/c 2023-05-31T20:29:25.049 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:ZwWUW/OF/WaE6lo6AuNjD64oDWo= 2023-05-31T20:29:25.049 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:29:25 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:ZwWUW/OF/WaE6lo6AuNjD64oDWo=', 'Content-Length': '0'} 2023-05-31T20:29:25.052 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:27:16 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000032e76ae3a29807c5-006477ae25-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:29:25 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:29:25.053 DEBUG:boto:path=/pklxkd-55/ 2023-05-31T20:29:25.053 DEBUG:boto:auth_path=/pklxkd-55/ 2023-05-31T20:29:25.053 DEBUG:boto:path=/pklxkd-55/?prefix=c 2023-05-31T20:29:25.054 DEBUG:boto:auth_path=/pklxkd-55/?prefix=c 2023-05-31T20:29:25.054 DEBUG:boto:Method: GET 2023-05-31T20:29:25.055 DEBUG:boto:Path: /pklxkd-55/?prefix=c 2023-05-31T20:29:25.055 DEBUG:boto:Data: 2023-05-31T20:29:25.055 DEBUG:boto:Headers: {} 2023-05-31T20:29:25.056 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:29:25.056 DEBUG:boto:Port: 8001 2023-05-31T20:29:25.056 DEBUG:boto:Params: {} 2023-05-31T20:29:25.057 DEBUG:boto:Token: None 2023-05-31T20:29:25.057 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:29:25 GMT /pklxkd-55/ 2023-05-31T20:29:25.058 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:a0/qPp0bnZeH62hZJf6OA1a38dw= 2023-05-31T20:29:25.058 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:29:25 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:a0/qPp0bnZeH62hZJf6OA1a38dw=', 'Content-Length': '0'} 2023-05-31T20:29:25.061 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000004d93bd42868f12e4-006477ae25-162e-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:29:25 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:29:25.061 DEBUG:boto:b'pklxkd-55c1000falsec2023-05-31T20:27:16.869Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:29:25.062 DEBUG:boto:path=/pklxkd-55/ 2023-05-31T20:29:25.063 DEBUG:boto:auth_path=/pklxkd-55/ 2023-05-31T20:29:25.063 DEBUG:boto:path=/pklxkd-55/?prefix=c 2023-05-31T20:29:25.063 DEBUG:boto:auth_path=/pklxkd-55/?prefix=c 2023-05-31T20:29:25.064 DEBUG:boto:Method: GET 2023-05-31T20:29:25.064 DEBUG:boto:Path: /pklxkd-55/?prefix=c 2023-05-31T20:29:25.064 DEBUG:boto:Data: 2023-05-31T20:29:25.065 DEBUG:boto:Headers: {} 2023-05-31T20:29:25.065 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:29:25.065 DEBUG:boto:Port: 8000 2023-05-31T20:29:25.066 DEBUG:boto:Params: {} 2023-05-31T20:29:25.066 DEBUG:boto:Token: None 2023-05-31T20:29:25.067 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:29:25 GMT /pklxkd-55/ 2023-05-31T20:29:25.067 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:a0/qPp0bnZeH62hZJf6OA1a38dw= 2023-05-31T20:29:25.068 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:29:25 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:a0/qPp0bnZeH62hZJf6OA1a38dw=', 'Content-Length': '0'} 2023-05-31T20:29:25.070 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000002e277aeb8b50bd36-006477ae25-1630-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:29:25 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:29:25.071 DEBUG:boto:b'pklxkd-55c1000falsec2023-05-31T20:27:16.869Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:29:25.072 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=d 2023-05-31T20:29:25.073 DEBUG:boto:path=/pklxkd-55/d 2023-05-31T20:29:25.074 DEBUG:boto:auth_path=/pklxkd-55/d 2023-05-31T20:29:25.074 DEBUG:boto:path=/pklxkd-55/d?versionId=null 2023-05-31T20:29:25.075 DEBUG:boto:auth_path=/pklxkd-55/d?versionId=null 2023-05-31T20:29:25.076 DEBUG:boto:Method: GET 2023-05-31T20:29:25.076 DEBUG:boto:Path: /pklxkd-55/d?versionId=null 2023-05-31T20:29:25.077 DEBUG:boto:Data: 2023-05-31T20:29:25.077 DEBUG:boto:Headers: {} 2023-05-31T20:29:25.078 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:29:25.078 DEBUG:boto:Port: 8001 2023-05-31T20:29:25.078 DEBUG:boto:Params: {} 2023-05-31T20:29:25.079 DEBUG:boto:Token: None 2023-05-31T20:29:25.079 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:29:25 GMT /pklxkd-55/d?versionId=null 2023-05-31T20:29:25.080 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:UTL6x5PDnkv/ObW9lagIN65WP+4= 2023-05-31T20:29:25.080 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:29:25 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:UTL6x5PDnkv/ObW9lagIN65WP+4=', 'Content-Length': '0'} 2023-05-31T20:29:25.082 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:27:16 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008ca3940a5c0c8656-006477ae25-162e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:29:25 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:29:25.128 DEBUG:boto:path=/pklxkd-55/d 2023-05-31T20:29:25.129 DEBUG:boto:auth_path=/pklxkd-55/d 2023-05-31T20:29:25.129 DEBUG:boto:path=/pklxkd-55/d?versionId=null 2023-05-31T20:29:25.130 DEBUG:boto:auth_path=/pklxkd-55/d?versionId=null 2023-05-31T20:29:25.130 DEBUG:boto:Method: GET 2023-05-31T20:29:25.131 DEBUG:boto:Path: /pklxkd-55/d?versionId=null 2023-05-31T20:29:25.131 DEBUG:boto:Data: 2023-05-31T20:29:25.131 DEBUG:boto:Headers: {} 2023-05-31T20:29:25.132 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:29:25.132 DEBUG:boto:Port: 8000 2023-05-31T20:29:25.133 DEBUG:boto:Params: {} 2023-05-31T20:29:25.134 DEBUG:boto:Token: None 2023-05-31T20:29:25.135 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:29:25 GMT /pklxkd-55/d?versionId=null 2023-05-31T20:29:25.136 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:UTL6x5PDnkv/ObW9lagIN65WP+4= 2023-05-31T20:29:25.137 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:29:25 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:UTL6x5PDnkv/ObW9lagIN65WP+4=', 'Content-Length': '0'} 2023-05-31T20:29:25.140 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:27:16 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d622e37ec0832bd9-006477ae25-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:29:25 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:29:25.141 DEBUG:boto:path=/pklxkd-55/d 2023-05-31T20:29:25.141 DEBUG:boto:auth_path=/pklxkd-55/d 2023-05-31T20:29:25.142 DEBUG:boto:path=/pklxkd-55/d?versionId=null 2023-05-31T20:29:25.142 DEBUG:boto:auth_path=/pklxkd-55/d?versionId=null 2023-05-31T20:29:25.142 DEBUG:boto:Method: HEAD 2023-05-31T20:29:25.143 DEBUG:boto:Path: /pklxkd-55/d?versionId=null 2023-05-31T20:29:25.143 DEBUG:boto:Data: 2023-05-31T20:29:25.143 DEBUG:boto:Headers: {} 2023-05-31T20:29:25.144 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:29:25.144 DEBUG:boto:Port: 8001 2023-05-31T20:29:25.145 DEBUG:boto:Params: {} 2023-05-31T20:29:25.145 DEBUG:boto:Token: None 2023-05-31T20:29:25.145 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:29:25 GMT /pklxkd-55/d?versionId=null 2023-05-31T20:29:25.146 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:ppfLEA4iI3ZziGNx+2cYKWodLG8= 2023-05-31T20:29:25.146 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:29:25 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:ppfLEA4iI3ZziGNx+2cYKWodLG8=', 'Content-Length': '0'} 2023-05-31T20:29:25.148 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:27:16 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d9ddef9e8bc8cf47-006477ae25-162e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:29:25 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:29:25.149 DEBUG:boto:path=/pklxkd-55/d 2023-05-31T20:29:25.149 DEBUG:boto:auth_path=/pklxkd-55/d 2023-05-31T20:29:25.150 DEBUG:boto:path=/pklxkd-55/d?versionId=null 2023-05-31T20:29:25.150 DEBUG:boto:auth_path=/pklxkd-55/d?versionId=null 2023-05-31T20:29:25.150 DEBUG:boto:Method: HEAD 2023-05-31T20:29:25.151 DEBUG:boto:Path: /pklxkd-55/d?versionId=null 2023-05-31T20:29:25.151 DEBUG:boto:Data: 2023-05-31T20:29:25.151 DEBUG:boto:Headers: {} 2023-05-31T20:29:25.152 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:29:25.152 DEBUG:boto:Port: 8000 2023-05-31T20:29:25.152 DEBUG:boto:Params: {} 2023-05-31T20:29:25.153 DEBUG:boto:Token: None 2023-05-31T20:29:25.153 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:29:25 GMT /pklxkd-55/d?versionId=null 2023-05-31T20:29:25.154 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:ppfLEA4iI3ZziGNx+2cYKWodLG8= 2023-05-31T20:29:25.154 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:29:25 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:ppfLEA4iI3ZziGNx+2cYKWodLG8=', 'Content-Length': '0'} 2023-05-31T20:29:25.156 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:27:16 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b7613e7245136de3-006477ae25-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:29:25 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:29:25.157 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=d 2023-05-31T20:29:25.157 DEBUG:boto:path=/pklxkd-55/d 2023-05-31T20:29:25.157 DEBUG:boto:auth_path=/pklxkd-55/d 2023-05-31T20:29:25.158 DEBUG:boto:path=/pklxkd-55/d?versionId=null 2023-05-31T20:29:25.158 DEBUG:boto:auth_path=/pklxkd-55/d?versionId=null 2023-05-31T20:29:25.158 DEBUG:boto:Method: GET 2023-05-31T20:29:25.159 DEBUG:boto:Path: /pklxkd-55/d?versionId=null 2023-05-31T20:29:25.159 DEBUG:boto:Data: 2023-05-31T20:29:25.159 DEBUG:boto:Headers: {} 2023-05-31T20:29:25.160 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:29:25.160 DEBUG:boto:Port: 8001 2023-05-31T20:29:25.161 DEBUG:boto:Params: {} 2023-05-31T20:29:25.161 DEBUG:boto:Token: None 2023-05-31T20:29:25.161 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:29:25 GMT /pklxkd-55/d?versionId=null 2023-05-31T20:29:25.162 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:UTL6x5PDnkv/ObW9lagIN65WP+4= 2023-05-31T20:29:25.162 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:29:25 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:UTL6x5PDnkv/ObW9lagIN65WP+4=', 'Content-Length': '0'} 2023-05-31T20:29:25.165 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:27:16 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000bd4caeab758f0f6f-006477ae25-162e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:29:25 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:29:25.165 DEBUG:boto:path=/pklxkd-55/d 2023-05-31T20:29:25.166 DEBUG:boto:auth_path=/pklxkd-55/d 2023-05-31T20:29:25.166 DEBUG:boto:path=/pklxkd-55/d?versionId=null 2023-05-31T20:29:25.167 DEBUG:boto:auth_path=/pklxkd-55/d?versionId=null 2023-05-31T20:29:25.167 DEBUG:boto:Method: GET 2023-05-31T20:29:25.167 DEBUG:boto:Path: /pklxkd-55/d?versionId=null 2023-05-31T20:29:25.168 DEBUG:boto:Data: 2023-05-31T20:29:25.168 DEBUG:boto:Headers: {} 2023-05-31T20:29:25.168 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:29:25.169 DEBUG:boto:Port: 8000 2023-05-31T20:29:25.170 DEBUG:boto:Params: {} 2023-05-31T20:29:25.170 DEBUG:boto:Token: None 2023-05-31T20:29:25.170 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:29:25 GMT /pklxkd-55/d?versionId=null 2023-05-31T20:29:25.171 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:UTL6x5PDnkv/ObW9lagIN65WP+4= 2023-05-31T20:29:25.171 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:29:25 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:UTL6x5PDnkv/ObW9lagIN65WP+4=', 'Content-Length': '0'} 2023-05-31T20:29:25.173 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:27:16 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000456554cb69afb174-006477ae25-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:29:25 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:29:25.174 DEBUG:boto:path=/pklxkd-55/ 2023-05-31T20:29:25.175 DEBUG:boto:auth_path=/pklxkd-55/ 2023-05-31T20:29:25.175 DEBUG:boto:path=/pklxkd-55/?prefix=d 2023-05-31T20:29:25.176 DEBUG:boto:auth_path=/pklxkd-55/?prefix=d 2023-05-31T20:29:25.176 DEBUG:boto:Method: GET 2023-05-31T20:29:25.176 DEBUG:boto:Path: /pklxkd-55/?prefix=d 2023-05-31T20:29:25.177 DEBUG:boto:Data: 2023-05-31T20:29:25.177 DEBUG:boto:Headers: {} 2023-05-31T20:29:25.177 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:29:25.178 DEBUG:boto:Port: 8001 2023-05-31T20:29:25.178 DEBUG:boto:Params: {} 2023-05-31T20:29:25.179 DEBUG:boto:Token: None 2023-05-31T20:29:25.179 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:29:25 GMT /pklxkd-55/ 2023-05-31T20:29:25.179 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:a0/qPp0bnZeH62hZJf6OA1a38dw= 2023-05-31T20:29:25.180 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:29:25 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:a0/qPp0bnZeH62hZJf6OA1a38dw=', 'Content-Length': '0'} 2023-05-31T20:29:25.182 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000fc0d4ecdab300639-006477ae25-162e-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:29:25 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:29:25.183 DEBUG:boto:b'pklxkd-55d1000falsed2023-05-31T20:27:16.893Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:29:25.184 DEBUG:boto:path=/pklxkd-55/ 2023-05-31T20:29:25.184 DEBUG:boto:auth_path=/pklxkd-55/ 2023-05-31T20:29:25.185 DEBUG:boto:path=/pklxkd-55/?prefix=d 2023-05-31T20:29:25.185 DEBUG:boto:auth_path=/pklxkd-55/?prefix=d 2023-05-31T20:29:25.185 DEBUG:boto:Method: GET 2023-05-31T20:29:25.186 DEBUG:boto:Path: /pklxkd-55/?prefix=d 2023-05-31T20:29:25.186 DEBUG:boto:Data: 2023-05-31T20:29:25.186 DEBUG:boto:Headers: {} 2023-05-31T20:29:25.187 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:29:25.187 DEBUG:boto:Port: 8000 2023-05-31T20:29:25.188 DEBUG:boto:Params: {} 2023-05-31T20:29:25.188 DEBUG:boto:Token: None 2023-05-31T20:29:25.188 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:29:25 GMT /pklxkd-55/ 2023-05-31T20:29:25.189 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:a0/qPp0bnZeH62hZJf6OA1a38dw= 2023-05-31T20:29:25.189 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:29:25 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:a0/qPp0bnZeH62hZJf6OA1a38dw=', 'Content-Length': '0'} 2023-05-31T20:29:25.191 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000475f353ad9c3a236-006477ae25-1630-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:29:25 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:29:25.192 DEBUG:boto:b'pklxkd-55d1000falsed2023-05-31T20:27:16.893Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:29:25.193 DEBUG:boto:path=/pklxkd-55/d 2023-05-31T20:29:25.193 DEBUG:boto:auth_path=/pklxkd-55/d 2023-05-31T20:29:25.193 DEBUG:boto:Method: HEAD 2023-05-31T20:29:25.194 DEBUG:boto:Path: /pklxkd-55/d 2023-05-31T20:29:25.194 DEBUG:boto:Data: 2023-05-31T20:29:25.194 DEBUG:boto:Headers: {} 2023-05-31T20:29:25.195 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:29:25.195 DEBUG:boto:Port: 8001 2023-05-31T20:29:25.195 DEBUG:boto:Params: {} 2023-05-31T20:29:25.196 DEBUG:boto:Token: None 2023-05-31T20:29:25.196 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:29:25 GMT /pklxkd-55/d 2023-05-31T20:29:25.197 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:/+mjzkh9GSY0ENrh4t2bca2ZSiI= 2023-05-31T20:29:25.197 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:29:25 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:/+mjzkh9GSY0ENrh4t2bca2ZSiI=', 'Content-Length': '0'} 2023-05-31T20:29:25.198 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:27:16 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003fc58f1d87587d62-006477ae25-162e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:29:25 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:29:25.199 DEBUG:boto:path=/pklxkd-55/d 2023-05-31T20:29:25.199 DEBUG:boto:auth_path=/pklxkd-55/d 2023-05-31T20:29:25.200 DEBUG:boto:Method: HEAD 2023-05-31T20:29:25.200 DEBUG:boto:Path: /pklxkd-55/d 2023-05-31T20:29:25.200 DEBUG:boto:Data: 2023-05-31T20:29:25.201 DEBUG:boto:Headers: {} 2023-05-31T20:29:25.201 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:29:25.202 DEBUG:boto:Port: 8000 2023-05-31T20:29:25.202 DEBUG:boto:Params: {} 2023-05-31T20:29:25.202 DEBUG:boto:Token: None 2023-05-31T20:29:25.203 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:29:25 GMT /pklxkd-55/d 2023-05-31T20:29:25.203 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:/+mjzkh9GSY0ENrh4t2bca2ZSiI= 2023-05-31T20:29:25.204 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:29:25 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:/+mjzkh9GSY0ENrh4t2bca2ZSiI=', 'Content-Length': '0'} 2023-05-31T20:29:25.206 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:27:16 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e180f725adc35ac8-006477ae25-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:29:25 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:29:25.206 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=d 2023-05-31T20:29:25.207 DEBUG:boto:path=/pklxkd-55/d 2023-05-31T20:29:25.207 DEBUG:boto:auth_path=/pklxkd-55/d 2023-05-31T20:29:25.207 DEBUG:boto:Method: GET 2023-05-31T20:29:25.208 DEBUG:boto:Path: /pklxkd-55/d 2023-05-31T20:29:25.208 DEBUG:boto:Data: 2023-05-31T20:29:25.209 DEBUG:boto:Headers: {} 2023-05-31T20:29:25.209 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:29:25.209 DEBUG:boto:Port: 8001 2023-05-31T20:29:25.210 DEBUG:boto:Params: {} 2023-05-31T20:29:25.211 DEBUG:boto:Token: None 2023-05-31T20:29:25.211 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:29:25 GMT /pklxkd-55/d 2023-05-31T20:29:25.212 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:n/qR9T6ixiWdO9ktyq98t+xoXn8= 2023-05-31T20:29:25.212 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:29:25 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:n/qR9T6ixiWdO9ktyq98t+xoXn8=', 'Content-Length': '0'} 2023-05-31T20:29:25.214 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:27:16 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b4d026ddc8119cd4-006477ae25-162e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:29:25 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:29:25.215 DEBUG:boto:path=/pklxkd-55/d 2023-05-31T20:29:25.216 DEBUG:boto:auth_path=/pklxkd-55/d 2023-05-31T20:29:25.216 DEBUG:boto:Method: GET 2023-05-31T20:29:25.217 DEBUG:boto:Path: /pklxkd-55/d 2023-05-31T20:29:25.218 DEBUG:boto:Data: 2023-05-31T20:29:25.219 DEBUG:boto:Headers: {} 2023-05-31T20:29:25.220 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:29:25.221 DEBUG:boto:Port: 8000 2023-05-31T20:29:25.222 DEBUG:boto:Params: {} 2023-05-31T20:29:25.223 DEBUG:boto:Token: None 2023-05-31T20:29:25.224 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:29:25 GMT /pklxkd-55/d 2023-05-31T20:29:25.225 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:n/qR9T6ixiWdO9ktyq98t+xoXn8= 2023-05-31T20:29:25.225 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:29:25 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:n/qR9T6ixiWdO9ktyq98t+xoXn8=', 'Content-Length': '0'} 2023-05-31T20:29:25.228 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:27:16 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000423e6993961249ae-006477ae25-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:29:25 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:29:25.228 DEBUG:boto:path=/pklxkd-55/ 2023-05-31T20:29:25.229 DEBUG:boto:auth_path=/pklxkd-55/ 2023-05-31T20:29:25.229 DEBUG:boto:path=/pklxkd-55/?prefix=d 2023-05-31T20:29:25.229 DEBUG:boto:auth_path=/pklxkd-55/?prefix=d 2023-05-31T20:29:25.230 DEBUG:boto:Method: GET 2023-05-31T20:29:25.230 DEBUG:boto:Path: /pklxkd-55/?prefix=d 2023-05-31T20:29:25.231 DEBUG:boto:Data: 2023-05-31T20:29:25.231 DEBUG:boto:Headers: {} 2023-05-31T20:29:25.231 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:29:25.232 DEBUG:boto:Port: 8001 2023-05-31T20:29:25.232 DEBUG:boto:Params: {} 2023-05-31T20:29:25.233 DEBUG:boto:Token: None 2023-05-31T20:29:25.235 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:29:25 GMT /pklxkd-55/ 2023-05-31T20:29:25.236 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:a0/qPp0bnZeH62hZJf6OA1a38dw= 2023-05-31T20:29:25.237 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:29:25 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:a0/qPp0bnZeH62hZJf6OA1a38dw=', 'Content-Length': '0'} 2023-05-31T20:29:25.240 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000007adba8616aaa56f2-006477ae25-162e-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:29:25 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:29:25.240 DEBUG:boto:b'pklxkd-55d1000falsed2023-05-31T20:27:16.893Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:29:25.241 DEBUG:boto:path=/pklxkd-55/ 2023-05-31T20:29:25.241 DEBUG:boto:auth_path=/pklxkd-55/ 2023-05-31T20:29:25.242 DEBUG:boto:path=/pklxkd-55/?prefix=d 2023-05-31T20:29:25.242 DEBUG:boto:auth_path=/pklxkd-55/?prefix=d 2023-05-31T20:29:25.243 DEBUG:boto:Method: GET 2023-05-31T20:29:25.243 DEBUG:boto:Path: /pklxkd-55/?prefix=d 2023-05-31T20:29:25.243 DEBUG:boto:Data: 2023-05-31T20:29:25.244 DEBUG:boto:Headers: {} 2023-05-31T20:29:25.244 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:29:25.245 DEBUG:boto:Port: 8000 2023-05-31T20:29:25.245 DEBUG:boto:Params: {} 2023-05-31T20:29:25.245 DEBUG:boto:Token: None 2023-05-31T20:29:25.246 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:29:25 GMT /pklxkd-55/ 2023-05-31T20:29:25.246 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:a0/qPp0bnZeH62hZJf6OA1a38dw= 2023-05-31T20:29:25.247 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:29:25 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:a0/qPp0bnZeH62hZJf6OA1a38dw=', 'Content-Length': '0'} 2023-05-31T20:29:25.249 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000fec8473da25d2966-006477ae25-1630-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:29:25 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:29:25.250 DEBUG:boto:b'pklxkd-55d1000falsed2023-05-31T20:27:16.893Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:29:25.251 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=e 2023-05-31T20:29:25.252 DEBUG:boto:path=/pklxkd-55/e 2023-05-31T20:29:25.253 DEBUG:boto:auth_path=/pklxkd-55/e 2023-05-31T20:29:25.254 DEBUG:boto:path=/pklxkd-55/e?versionId=null 2023-05-31T20:29:25.254 DEBUG:boto:auth_path=/pklxkd-55/e?versionId=null 2023-05-31T20:29:25.255 DEBUG:boto:Method: GET 2023-05-31T20:29:25.255 DEBUG:boto:Path: /pklxkd-55/e?versionId=null 2023-05-31T20:29:25.256 DEBUG:boto:Data: 2023-05-31T20:29:25.256 DEBUG:boto:Headers: {} 2023-05-31T20:29:25.257 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:29:25.257 DEBUG:boto:Port: 8001 2023-05-31T20:29:25.257 DEBUG:boto:Params: {} 2023-05-31T20:29:25.258 DEBUG:boto:Token: None 2023-05-31T20:29:25.258 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:29:25 GMT /pklxkd-55/e?versionId=null 2023-05-31T20:29:25.259 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:+YJyBGr8MemE+vt1MoE2zrx05v0= 2023-05-31T20:29:25.259 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:29:25 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:+YJyBGr8MemE+vt1MoE2zrx05v0=', 'Content-Length': '0'} 2023-05-31T20:29:25.262 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:27:23 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001ba4ae6a42db484a-006477ae25-162e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:29:25 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:29:25.262 DEBUG:boto:path=/pklxkd-55/e 2023-05-31T20:29:25.263 DEBUG:boto:auth_path=/pklxkd-55/e 2023-05-31T20:29:25.263 DEBUG:boto:path=/pklxkd-55/e?versionId=null 2023-05-31T20:29:25.263 DEBUG:boto:auth_path=/pklxkd-55/e?versionId=null 2023-05-31T20:29:25.264 DEBUG:boto:Method: GET 2023-05-31T20:29:25.264 DEBUG:boto:Path: /pklxkd-55/e?versionId=null 2023-05-31T20:29:25.265 DEBUG:boto:Data: 2023-05-31T20:29:25.265 DEBUG:boto:Headers: {} 2023-05-31T20:29:25.265 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:29:25.266 DEBUG:boto:Port: 8000 2023-05-31T20:29:25.266 DEBUG:boto:Params: {} 2023-05-31T20:29:25.267 DEBUG:boto:Token: None 2023-05-31T20:29:25.267 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:29:25 GMT /pklxkd-55/e?versionId=null 2023-05-31T20:29:25.267 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:+YJyBGr8MemE+vt1MoE2zrx05v0= 2023-05-31T20:29:25.268 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:29:25 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:+YJyBGr8MemE+vt1MoE2zrx05v0=', 'Content-Length': '0'} 2023-05-31T20:29:25.270 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:27:23 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000195d1e3efb6556e0-006477ae25-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:29:25 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:29:25.271 DEBUG:boto:path=/pklxkd-55/e 2023-05-31T20:29:25.271 DEBUG:boto:auth_path=/pklxkd-55/e 2023-05-31T20:29:25.272 DEBUG:boto:path=/pklxkd-55/e?versionId=null 2023-05-31T20:29:25.273 DEBUG:boto:auth_path=/pklxkd-55/e?versionId=null 2023-05-31T20:29:25.274 DEBUG:boto:Method: HEAD 2023-05-31T20:29:25.275 DEBUG:boto:Path: /pklxkd-55/e?versionId=null 2023-05-31T20:29:25.276 DEBUG:boto:Data: 2023-05-31T20:29:25.277 DEBUG:boto:Headers: {} 2023-05-31T20:29:25.278 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:29:25.278 DEBUG:boto:Port: 8001 2023-05-31T20:29:25.279 DEBUG:boto:Params: {} 2023-05-31T20:29:25.280 DEBUG:boto:Token: None 2023-05-31T20:29:25.280 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:29:25 GMT /pklxkd-55/e?versionId=null 2023-05-31T20:29:25.281 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:kQUQG6QGzVq9qxPrQO457i+pbjI= 2023-05-31T20:29:25.282 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:29:25 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:kQUQG6QGzVq9qxPrQO457i+pbjI=', 'Content-Length': '0'} 2023-05-31T20:29:25.284 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:27:23 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000003c2ae36b56c84c2-006477ae25-162e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:29:25 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:29:25.285 DEBUG:boto:path=/pklxkd-55/e 2023-05-31T20:29:25.285 DEBUG:boto:auth_path=/pklxkd-55/e 2023-05-31T20:29:25.286 DEBUG:boto:path=/pklxkd-55/e?versionId=null 2023-05-31T20:29:25.286 DEBUG:boto:auth_path=/pklxkd-55/e?versionId=null 2023-05-31T20:29:25.286 DEBUG:boto:Method: HEAD 2023-05-31T20:29:25.287 DEBUG:boto:Path: /pklxkd-55/e?versionId=null 2023-05-31T20:29:25.287 DEBUG:boto:Data: 2023-05-31T20:29:25.287 DEBUG:boto:Headers: {} 2023-05-31T20:29:25.288 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:29:25.288 DEBUG:boto:Port: 8000 2023-05-31T20:29:25.289 DEBUG:boto:Params: {} 2023-05-31T20:29:25.289 DEBUG:boto:Token: None 2023-05-31T20:29:25.290 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:29:25 GMT /pklxkd-55/e?versionId=null 2023-05-31T20:29:25.290 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:kQUQG6QGzVq9qxPrQO457i+pbjI= 2023-05-31T20:29:25.291 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:29:25 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:kQUQG6QGzVq9qxPrQO457i+pbjI=', 'Content-Length': '0'} 2023-05-31T20:29:25.293 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:27:23 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007e9c679b223c4f90-006477ae25-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:29:25 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:29:25.294 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=e 2023-05-31T20:29:25.295 DEBUG:boto:path=/pklxkd-55/e 2023-05-31T20:29:25.296 DEBUG:boto:auth_path=/pklxkd-55/e 2023-05-31T20:29:25.296 DEBUG:boto:path=/pklxkd-55/e?versionId=null 2023-05-31T20:29:25.297 DEBUG:boto:auth_path=/pklxkd-55/e?versionId=null 2023-05-31T20:29:25.297 DEBUG:boto:Method: GET 2023-05-31T20:29:25.298 DEBUG:boto:Path: /pklxkd-55/e?versionId=null 2023-05-31T20:29:25.299 DEBUG:boto:Data: 2023-05-31T20:29:25.299 DEBUG:boto:Headers: {} 2023-05-31T20:29:25.300 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:29:25.301 DEBUG:boto:Port: 8001 2023-05-31T20:29:25.302 DEBUG:boto:Params: {} 2023-05-31T20:29:25.303 DEBUG:boto:Token: None 2023-05-31T20:29:25.304 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:29:25 GMT /pklxkd-55/e?versionId=null 2023-05-31T20:29:25.304 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:+YJyBGr8MemE+vt1MoE2zrx05v0= 2023-05-31T20:29:25.305 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:29:25 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:+YJyBGr8MemE+vt1MoE2zrx05v0=', 'Content-Length': '0'} 2023-05-31T20:29:25.308 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:27:23 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000030de495e5fcd463d-006477ae25-162e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:29:25 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:29:25.348 DEBUG:boto:path=/pklxkd-55/e 2023-05-31T20:29:25.349 DEBUG:boto:auth_path=/pklxkd-55/e 2023-05-31T20:29:25.349 DEBUG:boto:path=/pklxkd-55/e?versionId=null 2023-05-31T20:29:25.350 DEBUG:boto:auth_path=/pklxkd-55/e?versionId=null 2023-05-31T20:29:25.350 DEBUG:boto:Method: GET 2023-05-31T20:29:25.350 DEBUG:boto:Path: /pklxkd-55/e?versionId=null 2023-05-31T20:29:25.351 DEBUG:boto:Data: 2023-05-31T20:29:25.351 DEBUG:boto:Headers: {} 2023-05-31T20:29:25.351 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:29:25.352 DEBUG:boto:Port: 8000 2023-05-31T20:29:25.352 DEBUG:boto:Params: {} 2023-05-31T20:29:25.353 DEBUG:boto:Token: None 2023-05-31T20:29:25.353 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:29:25 GMT /pklxkd-55/e?versionId=null 2023-05-31T20:29:25.354 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:+YJyBGr8MemE+vt1MoE2zrx05v0= 2023-05-31T20:29:25.354 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:29:25 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:+YJyBGr8MemE+vt1MoE2zrx05v0=', 'Content-Length': '0'} 2023-05-31T20:29:25.356 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:27:23 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d7da4ffc9c0cc262-006477ae25-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:29:25 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:29:25.357 DEBUG:boto:path=/pklxkd-55/ 2023-05-31T20:29:25.357 DEBUG:boto:auth_path=/pklxkd-55/ 2023-05-31T20:29:25.358 DEBUG:boto:path=/pklxkd-55/?prefix=e 2023-05-31T20:29:25.358 DEBUG:boto:auth_path=/pklxkd-55/?prefix=e 2023-05-31T20:29:25.358 DEBUG:boto:Method: GET 2023-05-31T20:29:25.359 DEBUG:boto:Path: /pklxkd-55/?prefix=e 2023-05-31T20:29:25.359 DEBUG:boto:Data: 2023-05-31T20:29:25.359 DEBUG:boto:Headers: {} 2023-05-31T20:29:25.360 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:29:25.360 DEBUG:boto:Port: 8001 2023-05-31T20:29:25.360 DEBUG:boto:Params: {} 2023-05-31T20:29:25.361 DEBUG:boto:Token: None 2023-05-31T20:29:25.362 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:29:25 GMT /pklxkd-55/ 2023-05-31T20:29:25.362 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:a0/qPp0bnZeH62hZJf6OA1a38dw= 2023-05-31T20:29:25.362 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:29:25 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:a0/qPp0bnZeH62hZJf6OA1a38dw=', 'Content-Length': '0'} 2023-05-31T20:29:25.365 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b1450d5593cb5e3c-006477ae25-162e-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:29:25 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:29:25.365 DEBUG:boto:b'pklxkd-55e1000falsee2023-05-31T20:27:23.955Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:29:25.366 DEBUG:boto:path=/pklxkd-55/ 2023-05-31T20:29:25.366 DEBUG:boto:auth_path=/pklxkd-55/ 2023-05-31T20:29:25.367 DEBUG:boto:path=/pklxkd-55/?prefix=e 2023-05-31T20:29:25.367 DEBUG:boto:auth_path=/pklxkd-55/?prefix=e 2023-05-31T20:29:25.367 DEBUG:boto:Method: GET 2023-05-31T20:29:25.368 DEBUG:boto:Path: /pklxkd-55/?prefix=e 2023-05-31T20:29:25.368 DEBUG:boto:Data: 2023-05-31T20:29:25.369 DEBUG:boto:Headers: {} 2023-05-31T20:29:25.369 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:29:25.369 DEBUG:boto:Port: 8000 2023-05-31T20:29:25.370 DEBUG:boto:Params: {} 2023-05-31T20:29:25.370 DEBUG:boto:Token: None 2023-05-31T20:29:25.371 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:29:25 GMT /pklxkd-55/ 2023-05-31T20:29:25.371 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:a0/qPp0bnZeH62hZJf6OA1a38dw= 2023-05-31T20:29:25.372 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:29:25 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:a0/qPp0bnZeH62hZJf6OA1a38dw=', 'Content-Length': '0'} 2023-05-31T20:29:25.374 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000040a5747d5a70ec09-006477ae25-1630-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:29:25 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:29:25.374 DEBUG:boto:b'pklxkd-55e1000falsee2023-05-31T20:27:23.955Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:29:25.375 DEBUG:boto:path=/pklxkd-55/e 2023-05-31T20:29:25.375 DEBUG:boto:auth_path=/pklxkd-55/e 2023-05-31T20:29:25.376 DEBUG:boto:Method: HEAD 2023-05-31T20:29:25.376 DEBUG:boto:Path: /pklxkd-55/e 2023-05-31T20:29:25.376 DEBUG:boto:Data: 2023-05-31T20:29:25.377 DEBUG:boto:Headers: {} 2023-05-31T20:29:25.377 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:29:25.377 DEBUG:boto:Port: 8001 2023-05-31T20:29:25.378 DEBUG:boto:Params: {} 2023-05-31T20:29:25.378 DEBUG:boto:Token: None 2023-05-31T20:29:25.378 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:29:25 GMT /pklxkd-55/e 2023-05-31T20:29:25.379 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:WI/JKAsiEmMNazlf+o8dalAWUSE= 2023-05-31T20:29:25.379 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:29:25 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:WI/JKAsiEmMNazlf+o8dalAWUSE=', 'Content-Length': '0'} 2023-05-31T20:29:25.381 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:27:23 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007fe39288ebc64206-006477ae25-162e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:29:25 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:29:25.382 DEBUG:boto:path=/pklxkd-55/e 2023-05-31T20:29:25.382 DEBUG:boto:auth_path=/pklxkd-55/e 2023-05-31T20:29:25.382 DEBUG:boto:Method: HEAD 2023-05-31T20:29:25.383 DEBUG:boto:Path: /pklxkd-55/e 2023-05-31T20:29:25.383 DEBUG:boto:Data: 2023-05-31T20:29:25.384 DEBUG:boto:Headers: {} 2023-05-31T20:29:25.384 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:29:25.385 DEBUG:boto:Port: 8000 2023-05-31T20:29:25.385 DEBUG:boto:Params: {} 2023-05-31T20:29:25.385 DEBUG:boto:Token: None 2023-05-31T20:29:25.386 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:29:25 GMT /pklxkd-55/e 2023-05-31T20:29:25.386 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:WI/JKAsiEmMNazlf+o8dalAWUSE= 2023-05-31T20:29:25.387 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:29:25 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:WI/JKAsiEmMNazlf+o8dalAWUSE=', 'Content-Length': '0'} 2023-05-31T20:29:25.389 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:27:23 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000068dc75829fdd7306-006477ae25-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:29:25 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:29:25.390 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=e 2023-05-31T20:29:25.390 DEBUG:boto:path=/pklxkd-55/e 2023-05-31T20:29:25.391 DEBUG:boto:auth_path=/pklxkd-55/e 2023-05-31T20:29:25.391 DEBUG:boto:Method: GET 2023-05-31T20:29:25.392 DEBUG:boto:Path: /pklxkd-55/e 2023-05-31T20:29:25.392 DEBUG:boto:Data: 2023-05-31T20:29:25.393 DEBUG:boto:Headers: {} 2023-05-31T20:29:25.394 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:29:25.394 DEBUG:boto:Port: 8001 2023-05-31T20:29:25.395 DEBUG:boto:Params: {} 2023-05-31T20:29:25.395 DEBUG:boto:Token: None 2023-05-31T20:29:25.396 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:29:25 GMT /pklxkd-55/e 2023-05-31T20:29:25.397 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:MSyipDFZixhTHZOKGu7tlz+s3lQ= 2023-05-31T20:29:25.397 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:29:25 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:MSyipDFZixhTHZOKGu7tlz+s3lQ=', 'Content-Length': '0'} 2023-05-31T20:29:25.399 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:27:23 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000ac76b8d29ae8a76-006477ae25-162e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:29:25 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:29:25.444 DEBUG:boto:path=/pklxkd-55/e 2023-05-31T20:29:25.445 DEBUG:boto:auth_path=/pklxkd-55/e 2023-05-31T20:29:25.445 DEBUG:boto:Method: GET 2023-05-31T20:29:25.446 DEBUG:boto:Path: /pklxkd-55/e 2023-05-31T20:29:25.446 DEBUG:boto:Data: 2023-05-31T20:29:25.447 DEBUG:boto:Headers: {} 2023-05-31T20:29:25.447 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:29:25.448 DEBUG:boto:Port: 8000 2023-05-31T20:29:25.448 DEBUG:boto:Params: {} 2023-05-31T20:29:25.449 DEBUG:boto:Token: None 2023-05-31T20:29:25.449 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:29:25 GMT /pklxkd-55/e 2023-05-31T20:29:25.451 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:MSyipDFZixhTHZOKGu7tlz+s3lQ= 2023-05-31T20:29:25.452 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:29:25 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:MSyipDFZixhTHZOKGu7tlz+s3lQ=', 'Content-Length': '0'} 2023-05-31T20:29:25.454 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:27:23 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000da86fe62bf35a242-006477ae25-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:29:25 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:29:25.456 DEBUG:boto:path=/pklxkd-55/ 2023-05-31T20:29:25.456 DEBUG:boto:auth_path=/pklxkd-55/ 2023-05-31T20:29:25.457 DEBUG:boto:path=/pklxkd-55/?prefix=e 2023-05-31T20:29:25.458 DEBUG:boto:auth_path=/pklxkd-55/?prefix=e 2023-05-31T20:29:25.459 DEBUG:boto:Method: GET 2023-05-31T20:29:25.459 DEBUG:boto:Path: /pklxkd-55/?prefix=e 2023-05-31T20:29:25.460 DEBUG:boto:Data: 2023-05-31T20:29:25.461 DEBUG:boto:Headers: {} 2023-05-31T20:29:25.462 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:29:25.462 DEBUG:boto:Port: 8001 2023-05-31T20:29:25.462 DEBUG:boto:Params: {} 2023-05-31T20:29:25.463 DEBUG:boto:Token: None 2023-05-31T20:29:25.465 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:29:25 GMT /pklxkd-55/ 2023-05-31T20:29:25.465 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:a0/qPp0bnZeH62hZJf6OA1a38dw= 2023-05-31T20:29:25.466 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:29:25 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:a0/qPp0bnZeH62hZJf6OA1a38dw=', 'Content-Length': '0'} 2023-05-31T20:29:25.468 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000051b82ab65d1916ac-006477ae25-162e-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:29:25 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:29:25.469 DEBUG:boto:b'pklxkd-55e1000falsee2023-05-31T20:27:23.955Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:29:25.470 DEBUG:boto:path=/pklxkd-55/ 2023-05-31T20:29:25.470 DEBUG:boto:auth_path=/pklxkd-55/ 2023-05-31T20:29:25.471 DEBUG:boto:path=/pklxkd-55/?prefix=e 2023-05-31T20:29:25.471 DEBUG:boto:auth_path=/pklxkd-55/?prefix=e 2023-05-31T20:29:25.471 DEBUG:boto:Method: GET 2023-05-31T20:29:25.472 DEBUG:boto:Path: /pklxkd-55/?prefix=e 2023-05-31T20:29:25.472 DEBUG:boto:Data: 2023-05-31T20:29:25.473 DEBUG:boto:Headers: {} 2023-05-31T20:29:25.473 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:29:25.473 DEBUG:boto:Port: 8000 2023-05-31T20:29:25.474 DEBUG:boto:Params: {} 2023-05-31T20:29:25.474 DEBUG:boto:Token: None 2023-05-31T20:29:25.475 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:29:25 GMT /pklxkd-55/ 2023-05-31T20:29:25.475 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:a0/qPp0bnZeH62hZJf6OA1a38dw= 2023-05-31T20:29:25.476 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:29:25 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:a0/qPp0bnZeH62hZJf6OA1a38dw=', 'Content-Length': '0'} 2023-05-31T20:29:25.478 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000002a61b1ba3e303c88-006477ae25-1630-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:29:25 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:29:25.478 DEBUG:boto:b'pklxkd-55e1000falsee2023-05-31T20:27:23.955Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:29:25.479 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=f 2023-05-31T20:29:25.479 DEBUG:boto:path=/pklxkd-55/f 2023-05-31T20:29:25.480 DEBUG:boto:auth_path=/pklxkd-55/f 2023-05-31T20:29:25.480 DEBUG:boto:path=/pklxkd-55/f?versionId=null 2023-05-31T20:29:25.480 DEBUG:boto:auth_path=/pklxkd-55/f?versionId=null 2023-05-31T20:29:25.481 DEBUG:boto:Method: GET 2023-05-31T20:29:25.481 DEBUG:boto:Path: /pklxkd-55/f?versionId=null 2023-05-31T20:29:25.482 DEBUG:boto:Data: 2023-05-31T20:29:25.482 DEBUG:boto:Headers: {} 2023-05-31T20:29:25.482 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:29:25.483 DEBUG:boto:Port: 8001 2023-05-31T20:29:25.483 DEBUG:boto:Params: {} 2023-05-31T20:29:25.483 DEBUG:boto:Token: None 2023-05-31T20:29:25.484 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:29:25 GMT /pklxkd-55/f?versionId=null 2023-05-31T20:29:25.484 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:7DmZ2WsueqJzxqvgmHkyBb7pLCA= 2023-05-31T20:29:25.485 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:29:25 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:7DmZ2WsueqJzxqvgmHkyBb7pLCA=', 'Content-Length': '0'} 2023-05-31T20:29:25.487 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:27:23 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000052dca59ddc891543-006477ae25-162e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:29:25 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:29:25.487 DEBUG:boto:path=/pklxkd-55/f 2023-05-31T20:29:25.488 DEBUG:boto:auth_path=/pklxkd-55/f 2023-05-31T20:29:25.488 DEBUG:boto:path=/pklxkd-55/f?versionId=null 2023-05-31T20:29:25.489 DEBUG:boto:auth_path=/pklxkd-55/f?versionId=null 2023-05-31T20:29:25.489 DEBUG:boto:Method: GET 2023-05-31T20:29:25.489 DEBUG:boto:Path: /pklxkd-55/f?versionId=null 2023-05-31T20:29:25.490 DEBUG:boto:Data: 2023-05-31T20:29:25.490 DEBUG:boto:Headers: {} 2023-05-31T20:29:25.491 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:29:25.491 DEBUG:boto:Port: 8000 2023-05-31T20:29:25.491 DEBUG:boto:Params: {} 2023-05-31T20:29:25.492 DEBUG:boto:Token: None 2023-05-31T20:29:25.492 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:29:25 GMT /pklxkd-55/f?versionId=null 2023-05-31T20:29:25.493 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:7DmZ2WsueqJzxqvgmHkyBb7pLCA= 2023-05-31T20:29:25.493 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:29:25 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:7DmZ2WsueqJzxqvgmHkyBb7pLCA=', 'Content-Length': '0'} 2023-05-31T20:29:25.495 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:27:23 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009dbe235e65120229-006477ae25-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:29:25 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:29:25.496 DEBUG:boto:path=/pklxkd-55/f 2023-05-31T20:29:25.496 DEBUG:boto:auth_path=/pklxkd-55/f 2023-05-31T20:29:25.497 DEBUG:boto:path=/pklxkd-55/f?versionId=null 2023-05-31T20:29:25.497 DEBUG:boto:auth_path=/pklxkd-55/f?versionId=null 2023-05-31T20:29:25.497 DEBUG:boto:Method: HEAD 2023-05-31T20:29:25.498 DEBUG:boto:Path: /pklxkd-55/f?versionId=null 2023-05-31T20:29:25.498 DEBUG:boto:Data: 2023-05-31T20:29:25.498 DEBUG:boto:Headers: {} 2023-05-31T20:29:25.499 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:29:25.499 DEBUG:boto:Port: 8001 2023-05-31T20:29:25.500 DEBUG:boto:Params: {} 2023-05-31T20:29:25.500 DEBUG:boto:Token: None 2023-05-31T20:29:25.501 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:29:25 GMT /pklxkd-55/f?versionId=null 2023-05-31T20:29:25.501 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:pVjZQX3/jyGJnRQeh0D4CRVFdPg= 2023-05-31T20:29:25.501 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:29:25 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:pVjZQX3/jyGJnRQeh0D4CRVFdPg=', 'Content-Length': '0'} 2023-05-31T20:29:25.503 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:27:23 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c2cbbc0990208f8b-006477ae25-162e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:29:25 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:29:25.504 DEBUG:boto:path=/pklxkd-55/f 2023-05-31T20:29:25.504 DEBUG:boto:auth_path=/pklxkd-55/f 2023-05-31T20:29:25.505 DEBUG:boto:path=/pklxkd-55/f?versionId=null 2023-05-31T20:29:25.505 DEBUG:boto:auth_path=/pklxkd-55/f?versionId=null 2023-05-31T20:29:25.506 DEBUG:boto:Method: HEAD 2023-05-31T20:29:25.506 DEBUG:boto:Path: /pklxkd-55/f?versionId=null 2023-05-31T20:29:25.506 DEBUG:boto:Data: 2023-05-31T20:29:25.507 DEBUG:boto:Headers: {} 2023-05-31T20:29:25.507 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:29:25.507 DEBUG:boto:Port: 8000 2023-05-31T20:29:25.508 DEBUG:boto:Params: {} 2023-05-31T20:29:25.508 DEBUG:boto:Token: None 2023-05-31T20:29:25.509 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:29:25 GMT /pklxkd-55/f?versionId=null 2023-05-31T20:29:25.509 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:pVjZQX3/jyGJnRQeh0D4CRVFdPg= 2023-05-31T20:29:25.509 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:29:25 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:pVjZQX3/jyGJnRQeh0D4CRVFdPg=', 'Content-Length': '0'} 2023-05-31T20:29:25.511 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:27:23 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f89d6e9fed21ab66-006477ae25-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:29:25 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:29:25.512 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=f 2023-05-31T20:29:25.512 DEBUG:boto:path=/pklxkd-55/f 2023-05-31T20:29:25.513 DEBUG:boto:auth_path=/pklxkd-55/f 2023-05-31T20:29:25.513 DEBUG:boto:path=/pklxkd-55/f?versionId=null 2023-05-31T20:29:25.513 DEBUG:boto:auth_path=/pklxkd-55/f?versionId=null 2023-05-31T20:29:25.514 DEBUG:boto:Method: GET 2023-05-31T20:29:25.514 DEBUG:boto:Path: /pklxkd-55/f?versionId=null 2023-05-31T20:29:25.515 DEBUG:boto:Data: 2023-05-31T20:29:25.515 DEBUG:boto:Headers: {} 2023-05-31T20:29:25.515 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:29:25.516 DEBUG:boto:Port: 8001 2023-05-31T20:29:25.516 DEBUG:boto:Params: {} 2023-05-31T20:29:25.516 DEBUG:boto:Token: None 2023-05-31T20:29:25.517 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:29:25 GMT /pklxkd-55/f?versionId=null 2023-05-31T20:29:25.517 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:7DmZ2WsueqJzxqvgmHkyBb7pLCA= 2023-05-31T20:29:25.518 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:29:25 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:7DmZ2WsueqJzxqvgmHkyBb7pLCA=', 'Content-Length': '0'} 2023-05-31T20:29:25.519 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:27:23 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001afa07c63e413033-006477ae25-162e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:29:25 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:29:25.520 DEBUG:boto:path=/pklxkd-55/f 2023-05-31T20:29:25.520 DEBUG:boto:auth_path=/pklxkd-55/f 2023-05-31T20:29:25.521 DEBUG:boto:path=/pklxkd-55/f?versionId=null 2023-05-31T20:29:25.521 DEBUG:boto:auth_path=/pklxkd-55/f?versionId=null 2023-05-31T20:29:25.522 DEBUG:boto:Method: GET 2023-05-31T20:29:25.522 DEBUG:boto:Path: /pklxkd-55/f?versionId=null 2023-05-31T20:29:25.522 DEBUG:boto:Data: 2023-05-31T20:29:25.523 DEBUG:boto:Headers: {} 2023-05-31T20:29:25.523 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:29:25.523 DEBUG:boto:Port: 8000 2023-05-31T20:29:25.524 DEBUG:boto:Params: {} 2023-05-31T20:29:25.524 DEBUG:boto:Token: None 2023-05-31T20:29:25.525 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:29:25 GMT /pklxkd-55/f?versionId=null 2023-05-31T20:29:25.525 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:7DmZ2WsueqJzxqvgmHkyBb7pLCA= 2023-05-31T20:29:25.525 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:29:25 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:7DmZ2WsueqJzxqvgmHkyBb7pLCA=', 'Content-Length': '0'} 2023-05-31T20:29:25.527 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:27:23 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000fe28f17f6b6de431-006477ae25-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:29:25 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:29:25.528 DEBUG:boto:path=/pklxkd-55/ 2023-05-31T20:29:25.528 DEBUG:boto:auth_path=/pklxkd-55/ 2023-05-31T20:29:25.529 DEBUG:boto:path=/pklxkd-55/?prefix=f 2023-05-31T20:29:25.529 DEBUG:boto:auth_path=/pklxkd-55/?prefix=f 2023-05-31T20:29:25.530 DEBUG:boto:Method: GET 2023-05-31T20:29:25.530 DEBUG:boto:Path: /pklxkd-55/?prefix=f 2023-05-31T20:29:25.530 DEBUG:boto:Data: 2023-05-31T20:29:25.531 DEBUG:boto:Headers: {} 2023-05-31T20:29:25.531 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:29:25.531 DEBUG:boto:Port: 8001 2023-05-31T20:29:25.532 DEBUG:boto:Params: {} 2023-05-31T20:29:25.532 DEBUG:boto:Token: None 2023-05-31T20:29:25.533 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:29:25 GMT /pklxkd-55/ 2023-05-31T20:29:25.533 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:a0/qPp0bnZeH62hZJf6OA1a38dw= 2023-05-31T20:29:25.534 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:29:25 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:a0/qPp0bnZeH62hZJf6OA1a38dw=', 'Content-Length': '0'} 2023-05-31T20:29:25.536 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000001f0f7f30e89138ac-006477ae25-162e-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:29:25 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:29:25.537 DEBUG:boto:b'pklxkd-55f1000falsef2023-05-31T20:27:23.977Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:29:25.538 DEBUG:boto:path=/pklxkd-55/ 2023-05-31T20:29:25.538 DEBUG:boto:auth_path=/pklxkd-55/ 2023-05-31T20:29:25.538 DEBUG:boto:path=/pklxkd-55/?prefix=f 2023-05-31T20:29:25.539 DEBUG:boto:auth_path=/pklxkd-55/?prefix=f 2023-05-31T20:29:25.539 DEBUG:boto:Method: GET 2023-05-31T20:29:25.539 DEBUG:boto:Path: /pklxkd-55/?prefix=f 2023-05-31T20:29:25.540 DEBUG:boto:Data: 2023-05-31T20:29:25.540 DEBUG:boto:Headers: {} 2023-05-31T20:29:25.540 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:29:25.541 DEBUG:boto:Port: 8000 2023-05-31T20:29:25.541 DEBUG:boto:Params: {} 2023-05-31T20:29:25.542 DEBUG:boto:Token: None 2023-05-31T20:29:25.542 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:29:25 GMT /pklxkd-55/ 2023-05-31T20:29:25.543 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:a0/qPp0bnZeH62hZJf6OA1a38dw= 2023-05-31T20:29:25.543 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:29:25 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:a0/qPp0bnZeH62hZJf6OA1a38dw=', 'Content-Length': '0'} 2023-05-31T20:29:25.545 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000089745409c2a37fc7-006477ae25-1630-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:29:25 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:29:25.546 DEBUG:boto:b'pklxkd-55f1000falsef2023-05-31T20:27:23.977Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:29:25.547 DEBUG:boto:path=/pklxkd-55/f 2023-05-31T20:29:25.547 DEBUG:boto:auth_path=/pklxkd-55/f 2023-05-31T20:29:25.548 DEBUG:boto:Method: HEAD 2023-05-31T20:29:25.549 DEBUG:boto:Path: /pklxkd-55/f 2023-05-31T20:29:25.549 DEBUG:boto:Data: 2023-05-31T20:29:25.550 DEBUG:boto:Headers: {} 2023-05-31T20:29:25.551 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:29:25.552 DEBUG:boto:Port: 8001 2023-05-31T20:29:25.552 DEBUG:boto:Params: {} 2023-05-31T20:29:25.557 DEBUG:boto:Token: None 2023-05-31T20:29:25.558 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:29:25 GMT /pklxkd-55/f 2023-05-31T20:29:25.558 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:EWNP/WSX9HAk9qDpuzjBhYIpfL0= 2023-05-31T20:29:25.558 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:29:25 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:EWNP/WSX9HAk9qDpuzjBhYIpfL0=', 'Content-Length': '0'} 2023-05-31T20:29:25.560 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:27:23 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000773bbf118a0cf21a-006477ae25-162e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:29:25 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:29:25.561 DEBUG:boto:path=/pklxkd-55/f 2023-05-31T20:29:25.561 DEBUG:boto:auth_path=/pklxkd-55/f 2023-05-31T20:29:25.562 DEBUG:boto:Method: HEAD 2023-05-31T20:29:25.562 DEBUG:boto:Path: /pklxkd-55/f 2023-05-31T20:29:25.562 DEBUG:boto:Data: 2023-05-31T20:29:25.563 DEBUG:boto:Headers: {} 2023-05-31T20:29:25.563 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:29:25.563 DEBUG:boto:Port: 8000 2023-05-31T20:29:25.564 DEBUG:boto:Params: {} 2023-05-31T20:29:25.564 DEBUG:boto:Token: None 2023-05-31T20:29:25.565 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:29:25 GMT /pklxkd-55/f 2023-05-31T20:29:25.565 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:EWNP/WSX9HAk9qDpuzjBhYIpfL0= 2023-05-31T20:29:25.566 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:29:25 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:EWNP/WSX9HAk9qDpuzjBhYIpfL0=', 'Content-Length': '0'} 2023-05-31T20:29:25.567 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:27:23 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003a6f026eb2564052-006477ae25-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:29:25 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:29:25.568 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=f 2023-05-31T20:29:25.568 DEBUG:boto:path=/pklxkd-55/f 2023-05-31T20:29:25.569 DEBUG:boto:auth_path=/pklxkd-55/f 2023-05-31T20:29:25.569 DEBUG:boto:Method: GET 2023-05-31T20:29:25.569 DEBUG:boto:Path: /pklxkd-55/f 2023-05-31T20:29:25.570 DEBUG:boto:Data: 2023-05-31T20:29:25.570 DEBUG:boto:Headers: {} 2023-05-31T20:29:25.571 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:29:25.571 DEBUG:boto:Port: 8001 2023-05-31T20:29:25.571 DEBUG:boto:Params: {} 2023-05-31T20:29:25.572 DEBUG:boto:Token: None 2023-05-31T20:29:25.572 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:29:25 GMT /pklxkd-55/f 2023-05-31T20:29:25.573 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:fS1d9jC0vfS2r1wK4J1RO/REDy4= 2023-05-31T20:29:25.573 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:29:25 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:fS1d9jC0vfS2r1wK4J1RO/REDy4=', 'Content-Length': '0'} 2023-05-31T20:29:25.575 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:27:23 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000007dee78956ea5252-006477ae25-162e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:29:25 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:29:25.616 DEBUG:boto:path=/pklxkd-55/f 2023-05-31T20:29:25.617 DEBUG:boto:auth_path=/pklxkd-55/f 2023-05-31T20:29:25.617 DEBUG:boto:Method: GET 2023-05-31T20:29:25.618 DEBUG:boto:Path: /pklxkd-55/f 2023-05-31T20:29:25.618 DEBUG:boto:Data: 2023-05-31T20:29:25.619 DEBUG:boto:Headers: {} 2023-05-31T20:29:25.619 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:29:25.619 DEBUG:boto:Port: 8000 2023-05-31T20:29:25.620 DEBUG:boto:Params: {} 2023-05-31T20:29:25.620 DEBUG:boto:Token: None 2023-05-31T20:29:25.620 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:29:25 GMT /pklxkd-55/f 2023-05-31T20:29:25.621 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:fS1d9jC0vfS2r1wK4J1RO/REDy4= 2023-05-31T20:29:25.621 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:29:25 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:fS1d9jC0vfS2r1wK4J1RO/REDy4=', 'Content-Length': '0'} 2023-05-31T20:29:25.623 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:27:23 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000092557c5a0b5a61a8-006477ae25-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:29:25 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:29:25.624 DEBUG:boto:path=/pklxkd-55/ 2023-05-31T20:29:25.625 DEBUG:boto:auth_path=/pklxkd-55/ 2023-05-31T20:29:25.625 DEBUG:boto:path=/pklxkd-55/?prefix=f 2023-05-31T20:29:25.625 DEBUG:boto:auth_path=/pklxkd-55/?prefix=f 2023-05-31T20:29:25.626 DEBUG:boto:Method: GET 2023-05-31T20:29:25.626 DEBUG:boto:Path: /pklxkd-55/?prefix=f 2023-05-31T20:29:25.626 DEBUG:boto:Data: 2023-05-31T20:29:25.627 DEBUG:boto:Headers: {} 2023-05-31T20:29:25.627 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:29:25.627 DEBUG:boto:Port: 8001 2023-05-31T20:29:25.628 DEBUG:boto:Params: {} 2023-05-31T20:29:25.628 DEBUG:boto:Token: None 2023-05-31T20:29:25.629 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:29:25 GMT /pklxkd-55/ 2023-05-31T20:29:25.629 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:a0/qPp0bnZeH62hZJf6OA1a38dw= 2023-05-31T20:29:25.629 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:29:25 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:a0/qPp0bnZeH62hZJf6OA1a38dw=', 'Content-Length': '0'} 2023-05-31T20:29:25.632 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000835cae37e5f314d3-006477ae25-162e-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:29:25 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:29:25.632 DEBUG:boto:b'pklxkd-55f1000falsef2023-05-31T20:27:23.977Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:29:25.633 DEBUG:boto:path=/pklxkd-55/ 2023-05-31T20:29:25.634 DEBUG:boto:auth_path=/pklxkd-55/ 2023-05-31T20:29:25.634 DEBUG:boto:path=/pklxkd-55/?prefix=f 2023-05-31T20:29:25.635 DEBUG:boto:auth_path=/pklxkd-55/?prefix=f 2023-05-31T20:29:25.635 DEBUG:boto:Method: GET 2023-05-31T20:29:25.636 DEBUG:boto:Path: /pklxkd-55/?prefix=f 2023-05-31T20:29:25.636 DEBUG:boto:Data: 2023-05-31T20:29:25.637 DEBUG:boto:Headers: {} 2023-05-31T20:29:25.638 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:29:25.638 DEBUG:boto:Port: 8000 2023-05-31T20:29:25.639 DEBUG:boto:Params: {} 2023-05-31T20:29:25.639 DEBUG:boto:Token: None 2023-05-31T20:29:25.640 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:29:25 GMT /pklxkd-55/ 2023-05-31T20:29:25.640 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:a0/qPp0bnZeH62hZJf6OA1a38dw= 2023-05-31T20:29:25.641 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:29:25 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:a0/qPp0bnZeH62hZJf6OA1a38dw=', 'Content-Length': '0'} 2023-05-31T20:29:25.643 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e5af12247ae7a9c8-006477ae25-1630-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:29:25 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:29:25.643 DEBUG:boto:b'pklxkd-55f1000falsef2023-05-31T20:27:23.977Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:29:25.644 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=g 2023-05-31T20:29:25.644 DEBUG:boto:path=/pklxkd-55/g 2023-05-31T20:29:25.645 DEBUG:boto:auth_path=/pklxkd-55/g 2023-05-31T20:29:25.645 DEBUG:boto:path=/pklxkd-55/g?versionId=null 2023-05-31T20:29:25.646 DEBUG:boto:auth_path=/pklxkd-55/g?versionId=null 2023-05-31T20:29:25.646 DEBUG:boto:Method: GET 2023-05-31T20:29:25.647 DEBUG:boto:Path: /pklxkd-55/g?versionId=null 2023-05-31T20:29:25.647 DEBUG:boto:Data: 2023-05-31T20:29:25.648 DEBUG:boto:Headers: {} 2023-05-31T20:29:25.648 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:29:25.648 DEBUG:boto:Port: 8001 2023-05-31T20:29:25.649 DEBUG:boto:Params: {} 2023-05-31T20:29:25.649 DEBUG:boto:Token: None 2023-05-31T20:29:25.650 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:29:25 GMT /pklxkd-55/g?versionId=null 2023-05-31T20:29:25.650 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:qsnSCM4kslUB64GjrB2v0KYUJLc= 2023-05-31T20:29:25.651 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:29:25 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:qsnSCM4kslUB64GjrB2v0KYUJLc=', 'Content-Length': '0'} 2023-05-31T20:29:25.653 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:27:24 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002dd250dede2e2fc2-006477ae25-162e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:29:25 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:29:25.654 DEBUG:boto:path=/pklxkd-55/g 2023-05-31T20:29:25.655 DEBUG:boto:auth_path=/pklxkd-55/g 2023-05-31T20:29:25.656 DEBUG:boto:path=/pklxkd-55/g?versionId=null 2023-05-31T20:29:25.657 DEBUG:boto:auth_path=/pklxkd-55/g?versionId=null 2023-05-31T20:29:25.657 DEBUG:boto:Method: GET 2023-05-31T20:29:25.658 DEBUG:boto:Path: /pklxkd-55/g?versionId=null 2023-05-31T20:29:25.658 DEBUG:boto:Data: 2023-05-31T20:29:25.659 DEBUG:boto:Headers: {} 2023-05-31T20:29:25.659 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:29:25.659 DEBUG:boto:Port: 8000 2023-05-31T20:29:25.660 DEBUG:boto:Params: {} 2023-05-31T20:29:25.660 DEBUG:boto:Token: None 2023-05-31T20:29:25.661 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:29:25 GMT /pklxkd-55/g?versionId=null 2023-05-31T20:29:25.661 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:qsnSCM4kslUB64GjrB2v0KYUJLc= 2023-05-31T20:29:25.662 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:29:25 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:qsnSCM4kslUB64GjrB2v0KYUJLc=', 'Content-Length': '0'} 2023-05-31T20:29:25.664 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:27:24 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000043cf1be21524017a-006477ae25-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:29:25 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:29:25.665 DEBUG:boto:path=/pklxkd-55/g 2023-05-31T20:29:25.666 DEBUG:boto:auth_path=/pklxkd-55/g 2023-05-31T20:29:25.666 DEBUG:boto:path=/pklxkd-55/g?versionId=null 2023-05-31T20:29:25.666 DEBUG:boto:auth_path=/pklxkd-55/g?versionId=null 2023-05-31T20:29:25.667 DEBUG:boto:Method: HEAD 2023-05-31T20:29:25.667 DEBUG:boto:Path: /pklxkd-55/g?versionId=null 2023-05-31T20:29:25.668 DEBUG:boto:Data: 2023-05-31T20:29:25.668 DEBUG:boto:Headers: {} 2023-05-31T20:29:25.669 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:29:25.669 DEBUG:boto:Port: 8001 2023-05-31T20:29:25.669 DEBUG:boto:Params: {} 2023-05-31T20:29:25.670 DEBUG:boto:Token: None 2023-05-31T20:29:25.670 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:29:25 GMT /pklxkd-55/g?versionId=null 2023-05-31T20:29:25.671 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:KE9s1g24EIbZhx5Yr+qs+NeHbco= 2023-05-31T20:29:25.671 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:29:25 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:KE9s1g24EIbZhx5Yr+qs+NeHbco=', 'Content-Length': '0'} 2023-05-31T20:29:25.673 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:27:24 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a64ebb15dc80cc7d-006477ae25-162e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:29:25 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:29:25.674 DEBUG:boto:path=/pklxkd-55/g 2023-05-31T20:29:25.675 DEBUG:boto:auth_path=/pklxkd-55/g 2023-05-31T20:29:25.675 DEBUG:boto:path=/pklxkd-55/g?versionId=null 2023-05-31T20:29:25.675 DEBUG:boto:auth_path=/pklxkd-55/g?versionId=null 2023-05-31T20:29:25.676 DEBUG:boto:Method: HEAD 2023-05-31T20:29:25.676 DEBUG:boto:Path: /pklxkd-55/g?versionId=null 2023-05-31T20:29:25.676 DEBUG:boto:Data: 2023-05-31T20:29:25.677 DEBUG:boto:Headers: {} 2023-05-31T20:29:25.677 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:29:25.678 DEBUG:boto:Port: 8000 2023-05-31T20:29:25.678 DEBUG:boto:Params: {} 2023-05-31T20:29:25.678 DEBUG:boto:Token: None 2023-05-31T20:29:25.679 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:29:25 GMT /pklxkd-55/g?versionId=null 2023-05-31T20:29:25.679 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:KE9s1g24EIbZhx5Yr+qs+NeHbco= 2023-05-31T20:29:25.680 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:29:25 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:KE9s1g24EIbZhx5Yr+qs+NeHbco=', 'Content-Length': '0'} 2023-05-31T20:29:25.681 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:27:24 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000098106713d97f6f4d-006477ae25-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:29:25 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:29:25.682 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=g 2023-05-31T20:29:25.682 DEBUG:boto:path=/pklxkd-55/g 2023-05-31T20:29:25.683 DEBUG:boto:auth_path=/pklxkd-55/g 2023-05-31T20:29:25.683 DEBUG:boto:path=/pklxkd-55/g?versionId=null 2023-05-31T20:29:25.684 DEBUG:boto:auth_path=/pklxkd-55/g?versionId=null 2023-05-31T20:29:25.684 DEBUG:boto:Method: GET 2023-05-31T20:29:25.684 DEBUG:boto:Path: /pklxkd-55/g?versionId=null 2023-05-31T20:29:25.685 DEBUG:boto:Data: 2023-05-31T20:29:25.685 DEBUG:boto:Headers: {} 2023-05-31T20:29:25.685 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:29:25.686 DEBUG:boto:Port: 8001 2023-05-31T20:29:25.686 DEBUG:boto:Params: {} 2023-05-31T20:29:25.687 DEBUG:boto:Token: None 2023-05-31T20:29:25.687 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:29:25 GMT /pklxkd-55/g?versionId=null 2023-05-31T20:29:25.688 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:qsnSCM4kslUB64GjrB2v0KYUJLc= 2023-05-31T20:29:25.688 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:29:25 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:qsnSCM4kslUB64GjrB2v0KYUJLc=', 'Content-Length': '0'} 2023-05-31T20:29:25.690 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:27:24 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a02e02d558c2820f-006477ae25-162e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:29:25 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:29:25.691 DEBUG:boto:path=/pklxkd-55/g 2023-05-31T20:29:25.692 DEBUG:boto:auth_path=/pklxkd-55/g 2023-05-31T20:29:25.692 DEBUG:boto:path=/pklxkd-55/g?versionId=null 2023-05-31T20:29:25.693 DEBUG:boto:auth_path=/pklxkd-55/g?versionId=null 2023-05-31T20:29:25.693 DEBUG:boto:Method: GET 2023-05-31T20:29:25.694 DEBUG:boto:Path: /pklxkd-55/g?versionId=null 2023-05-31T20:29:25.694 DEBUG:boto:Data: 2023-05-31T20:29:25.695 DEBUG:boto:Headers: {} 2023-05-31T20:29:25.695 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:29:25.696 DEBUG:boto:Port: 8000 2023-05-31T20:29:25.696 DEBUG:boto:Params: {} 2023-05-31T20:29:25.697 DEBUG:boto:Token: None 2023-05-31T20:29:25.697 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:29:25 GMT /pklxkd-55/g?versionId=null 2023-05-31T20:29:25.697 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:qsnSCM4kslUB64GjrB2v0KYUJLc= 2023-05-31T20:29:25.698 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:29:25 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:qsnSCM4kslUB64GjrB2v0KYUJLc=', 'Content-Length': '0'} 2023-05-31T20:29:25.700 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:27:24 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ece4a52bfaa107d8-006477ae25-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:29:25 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:29:25.701 DEBUG:boto:path=/pklxkd-55/ 2023-05-31T20:29:25.701 DEBUG:boto:auth_path=/pklxkd-55/ 2023-05-31T20:29:25.702 DEBUG:boto:path=/pklxkd-55/?prefix=g 2023-05-31T20:29:25.702 DEBUG:boto:auth_path=/pklxkd-55/?prefix=g 2023-05-31T20:29:25.703 DEBUG:boto:Method: GET 2023-05-31T20:29:25.704 DEBUG:boto:Path: /pklxkd-55/?prefix=g 2023-05-31T20:29:25.705 DEBUG:boto:Data: 2023-05-31T20:29:25.706 DEBUG:boto:Headers: {} 2023-05-31T20:29:25.706 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:29:25.707 DEBUG:boto:Port: 8001 2023-05-31T20:29:25.708 DEBUG:boto:Params: {} 2023-05-31T20:29:25.709 DEBUG:boto:Token: None 2023-05-31T20:29:25.709 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:29:25 GMT /pklxkd-55/ 2023-05-31T20:29:25.710 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:a0/qPp0bnZeH62hZJf6OA1a38dw= 2023-05-31T20:29:25.711 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:29:25 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:a0/qPp0bnZeH62hZJf6OA1a38dw=', 'Content-Length': '0'} 2023-05-31T20:29:25.714 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000213cfb4550359655-006477ae25-162e-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:29:25 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:29:25.714 DEBUG:boto:b'pklxkd-55g1000falseg2023-05-31T20:27:24.035Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:29:25.716 DEBUG:boto:path=/pklxkd-55/ 2023-05-31T20:29:25.717 DEBUG:boto:auth_path=/pklxkd-55/ 2023-05-31T20:29:25.718 DEBUG:boto:path=/pklxkd-55/?prefix=g 2023-05-31T20:29:25.718 DEBUG:boto:auth_path=/pklxkd-55/?prefix=g 2023-05-31T20:29:25.719 DEBUG:boto:Method: GET 2023-05-31T20:29:25.720 DEBUG:boto:Path: /pklxkd-55/?prefix=g 2023-05-31T20:29:25.721 DEBUG:boto:Data: 2023-05-31T20:29:25.722 DEBUG:boto:Headers: {} 2023-05-31T20:29:25.722 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:29:25.723 DEBUG:boto:Port: 8000 2023-05-31T20:29:25.723 DEBUG:boto:Params: {} 2023-05-31T20:29:25.724 DEBUG:boto:Token: None 2023-05-31T20:29:25.725 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:29:25 GMT /pklxkd-55/ 2023-05-31T20:29:25.725 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:a0/qPp0bnZeH62hZJf6OA1a38dw= 2023-05-31T20:29:25.726 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:29:25 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:a0/qPp0bnZeH62hZJf6OA1a38dw=', 'Content-Length': '0'} 2023-05-31T20:29:25.728 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000090a5490a6429dc72-006477ae25-1630-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:29:25 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:29:25.728 DEBUG:boto:b'pklxkd-55g1000falseg2023-05-31T20:27:24.035Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:29:25.729 DEBUG:boto:path=/pklxkd-55/g 2023-05-31T20:29:25.729 DEBUG:boto:auth_path=/pklxkd-55/g 2023-05-31T20:29:25.730 DEBUG:boto:Method: HEAD 2023-05-31T20:29:25.730 DEBUG:boto:Path: /pklxkd-55/g 2023-05-31T20:29:25.730 DEBUG:boto:Data: 2023-05-31T20:29:25.731 DEBUG:boto:Headers: {} 2023-05-31T20:29:25.731 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:29:25.731 DEBUG:boto:Port: 8001 2023-05-31T20:29:25.732 DEBUG:boto:Params: {} 2023-05-31T20:29:25.732 DEBUG:boto:Token: None 2023-05-31T20:29:25.733 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:29:25 GMT /pklxkd-55/g 2023-05-31T20:29:25.733 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:98XyMXADg1VnamqbYsz7p83l8vU= 2023-05-31T20:29:25.734 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:29:25 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:98XyMXADg1VnamqbYsz7p83l8vU=', 'Content-Length': '0'} 2023-05-31T20:29:25.735 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:27:24 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002a747944dfba0bca-006477ae25-162e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:29:25 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:29:25.736 DEBUG:boto:path=/pklxkd-55/g 2023-05-31T20:29:25.737 DEBUG:boto:auth_path=/pklxkd-55/g 2023-05-31T20:29:25.737 DEBUG:boto:Method: HEAD 2023-05-31T20:29:25.737 DEBUG:boto:Path: /pklxkd-55/g 2023-05-31T20:29:25.738 DEBUG:boto:Data: 2023-05-31T20:29:25.738 DEBUG:boto:Headers: {} 2023-05-31T20:29:25.739 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:29:25.739 DEBUG:boto:Port: 8000 2023-05-31T20:29:25.740 DEBUG:boto:Params: {} 2023-05-31T20:29:25.740 DEBUG:boto:Token: None 2023-05-31T20:29:25.741 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:29:25 GMT /pklxkd-55/g 2023-05-31T20:29:25.741 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:98XyMXADg1VnamqbYsz7p83l8vU= 2023-05-31T20:29:25.742 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:29:25 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:98XyMXADg1VnamqbYsz7p83l8vU=', 'Content-Length': '0'} 2023-05-31T20:29:25.744 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:27:24 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000410c8de93a4089ab-006477ae25-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:29:25 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:29:25.744 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=g 2023-05-31T20:29:25.745 DEBUG:boto:path=/pklxkd-55/g 2023-05-31T20:29:25.745 DEBUG:boto:auth_path=/pklxkd-55/g 2023-05-31T20:29:25.746 DEBUG:boto:Method: GET 2023-05-31T20:29:25.746 DEBUG:boto:Path: /pklxkd-55/g 2023-05-31T20:29:25.747 DEBUG:boto:Data: 2023-05-31T20:29:25.747 DEBUG:boto:Headers: {} 2023-05-31T20:29:25.748 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:29:25.748 DEBUG:boto:Port: 8001 2023-05-31T20:29:25.749 DEBUG:boto:Params: {} 2023-05-31T20:29:25.749 DEBUG:boto:Token: None 2023-05-31T20:29:25.750 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:29:25 GMT /pklxkd-55/g 2023-05-31T20:29:25.750 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:9a77YCWe+i1di4u03C/zuiH2Td8= 2023-05-31T20:29:25.751 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:29:25 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:9a77YCWe+i1di4u03C/zuiH2Td8=', 'Content-Length': '0'} 2023-05-31T20:29:25.752 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:27:24 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001a8147bcd114d078-006477ae25-162e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:29:25 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:29:25.796 DEBUG:boto:path=/pklxkd-55/g 2023-05-31T20:29:25.797 DEBUG:boto:auth_path=/pklxkd-55/g 2023-05-31T20:29:25.797 DEBUG:boto:Method: GET 2023-05-31T20:29:25.798 DEBUG:boto:Path: /pklxkd-55/g 2023-05-31T20:29:25.798 DEBUG:boto:Data: 2023-05-31T20:29:25.798 DEBUG:boto:Headers: {} 2023-05-31T20:29:25.799 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:29:25.799 DEBUG:boto:Port: 8000 2023-05-31T20:29:25.800 DEBUG:boto:Params: {} 2023-05-31T20:29:25.800 DEBUG:boto:Token: None 2023-05-31T20:29:25.800 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:29:25 GMT /pklxkd-55/g 2023-05-31T20:29:25.801 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:9a77YCWe+i1di4u03C/zuiH2Td8= 2023-05-31T20:29:25.801 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:29:25 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:9a77YCWe+i1di4u03C/zuiH2Td8=', 'Content-Length': '0'} 2023-05-31T20:29:25.804 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:27:24 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000044cf4c2b77e79440-006477ae25-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:29:25 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:29:25.804 DEBUG:boto:path=/pklxkd-55/ 2023-05-31T20:29:25.805 DEBUG:boto:auth_path=/pklxkd-55/ 2023-05-31T20:29:25.805 DEBUG:boto:path=/pklxkd-55/?prefix=g 2023-05-31T20:29:25.806 DEBUG:boto:auth_path=/pklxkd-55/?prefix=g 2023-05-31T20:29:25.806 DEBUG:boto:Method: GET 2023-05-31T20:29:25.806 DEBUG:boto:Path: /pklxkd-55/?prefix=g 2023-05-31T20:29:25.807 DEBUG:boto:Data: 2023-05-31T20:29:25.807 DEBUG:boto:Headers: {} 2023-05-31T20:29:25.807 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:29:25.808 DEBUG:boto:Port: 8001 2023-05-31T20:29:25.808 DEBUG:boto:Params: {} 2023-05-31T20:29:25.809 DEBUG:boto:Token: None 2023-05-31T20:29:25.809 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:29:25 GMT /pklxkd-55/ 2023-05-31T20:29:25.809 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:a0/qPp0bnZeH62hZJf6OA1a38dw= 2023-05-31T20:29:25.810 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:29:25 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:a0/qPp0bnZeH62hZJf6OA1a38dw=', 'Content-Length': '0'} 2023-05-31T20:29:25.812 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000092555a0ff9c49a44-006477ae25-162e-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:29:25 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:29:25.813 DEBUG:boto:b'pklxkd-55g1000falseg2023-05-31T20:27:24.035Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:29:25.814 DEBUG:boto:path=/pklxkd-55/ 2023-05-31T20:29:25.814 DEBUG:boto:auth_path=/pklxkd-55/ 2023-05-31T20:29:25.814 DEBUG:boto:path=/pklxkd-55/?prefix=g 2023-05-31T20:29:25.815 DEBUG:boto:auth_path=/pklxkd-55/?prefix=g 2023-05-31T20:29:25.815 DEBUG:boto:Method: GET 2023-05-31T20:29:25.816 DEBUG:boto:Path: /pklxkd-55/?prefix=g 2023-05-31T20:29:25.816 DEBUG:boto:Data: 2023-05-31T20:29:25.816 DEBUG:boto:Headers: {} 2023-05-31T20:29:25.817 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:29:25.817 DEBUG:boto:Port: 8000 2023-05-31T20:29:25.817 DEBUG:boto:Params: {} 2023-05-31T20:29:25.818 DEBUG:boto:Token: None 2023-05-31T20:29:25.818 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:29:25 GMT /pklxkd-55/ 2023-05-31T20:29:25.819 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:a0/qPp0bnZeH62hZJf6OA1a38dw= 2023-05-31T20:29:25.819 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:29:25 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:a0/qPp0bnZeH62hZJf6OA1a38dw=', 'Content-Length': '0'} 2023-05-31T20:29:25.821 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000004dd3d6fce640f28f-006477ae25-1630-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:29:25 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:29:25.822 DEBUG:boto:b'pklxkd-55g1000falseg2023-05-31T20:27:24.035Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:29:25.822 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=h 2023-05-31T20:29:25.823 DEBUG:boto:path=/pklxkd-55/h 2023-05-31T20:29:25.823 DEBUG:boto:auth_path=/pklxkd-55/h 2023-05-31T20:29:25.823 DEBUG:boto:path=/pklxkd-55/h?versionId=null 2023-05-31T20:29:25.824 DEBUG:boto:auth_path=/pklxkd-55/h?versionId=null 2023-05-31T20:29:25.824 DEBUG:boto:Method: GET 2023-05-31T20:29:25.824 DEBUG:boto:Path: /pklxkd-55/h?versionId=null 2023-05-31T20:29:25.825 DEBUG:boto:Data: 2023-05-31T20:29:25.825 DEBUG:boto:Headers: {} 2023-05-31T20:29:25.826 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:29:25.826 DEBUG:boto:Port: 8001 2023-05-31T20:29:25.826 DEBUG:boto:Params: {} 2023-05-31T20:29:25.827 DEBUG:boto:Token: None 2023-05-31T20:29:25.827 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:29:25 GMT /pklxkd-55/h?versionId=null 2023-05-31T20:29:25.828 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:4Y/8L8lDnhHI55GrOpuluMNFBUo= 2023-05-31T20:29:25.828 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:29:25 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:4Y/8L8lDnhHI55GrOpuluMNFBUo=', 'Content-Length': '0'} 2023-05-31T20:29:25.830 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:27:24 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000068bb8c8ebf25405c-006477ae25-162e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:29:25 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:29:25.831 DEBUG:boto:path=/pklxkd-55/h 2023-05-31T20:29:25.831 DEBUG:boto:auth_path=/pklxkd-55/h 2023-05-31T20:29:25.832 DEBUG:boto:path=/pklxkd-55/h?versionId=null 2023-05-31T20:29:25.832 DEBUG:boto:auth_path=/pklxkd-55/h?versionId=null 2023-05-31T20:29:25.833 DEBUG:boto:Method: GET 2023-05-31T20:29:25.833 DEBUG:boto:Path: /pklxkd-55/h?versionId=null 2023-05-31T20:29:25.833 DEBUG:boto:Data: 2023-05-31T20:29:25.834 DEBUG:boto:Headers: {} 2023-05-31T20:29:25.834 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:29:25.835 DEBUG:boto:Port: 8000 2023-05-31T20:29:25.835 DEBUG:boto:Params: {} 2023-05-31T20:29:25.835 DEBUG:boto:Token: None 2023-05-31T20:29:25.836 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:29:25 GMT /pklxkd-55/h?versionId=null 2023-05-31T20:29:25.836 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:4Y/8L8lDnhHI55GrOpuluMNFBUo= 2023-05-31T20:29:25.837 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:29:25 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:4Y/8L8lDnhHI55GrOpuluMNFBUo=', 'Content-Length': '0'} 2023-05-31T20:29:25.839 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:27:24 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ec20a36b751303ea-006477ae25-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:29:25 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:29:25.839 DEBUG:boto:path=/pklxkd-55/h 2023-05-31T20:29:25.840 DEBUG:boto:auth_path=/pklxkd-55/h 2023-05-31T20:29:25.840 DEBUG:boto:path=/pklxkd-55/h?versionId=null 2023-05-31T20:29:25.841 DEBUG:boto:auth_path=/pklxkd-55/h?versionId=null 2023-05-31T20:29:25.841 DEBUG:boto:Method: HEAD 2023-05-31T20:29:25.841 DEBUG:boto:Path: /pklxkd-55/h?versionId=null 2023-05-31T20:29:25.842 DEBUG:boto:Data: 2023-05-31T20:29:25.842 DEBUG:boto:Headers: {} 2023-05-31T20:29:25.843 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:29:25.843 DEBUG:boto:Port: 8001 2023-05-31T20:29:25.843 DEBUG:boto:Params: {} 2023-05-31T20:29:25.844 DEBUG:boto:Token: None 2023-05-31T20:29:25.844 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:29:25 GMT /pklxkd-55/h?versionId=null 2023-05-31T20:29:25.844 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:4efssCLMCr2E9Yl7sdCyB23wbtw= 2023-05-31T20:29:25.845 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:29:25 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:4efssCLMCr2E9Yl7sdCyB23wbtw=', 'Content-Length': '0'} 2023-05-31T20:29:25.847 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:27:24 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000033b93ee33ebb509d-006477ae25-162e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:29:25 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:29:25.847 DEBUG:boto:path=/pklxkd-55/h 2023-05-31T20:29:25.848 DEBUG:boto:auth_path=/pklxkd-55/h 2023-05-31T20:29:25.848 DEBUG:boto:path=/pklxkd-55/h?versionId=null 2023-05-31T20:29:25.849 DEBUG:boto:auth_path=/pklxkd-55/h?versionId=null 2023-05-31T20:29:25.849 DEBUG:boto:Method: HEAD 2023-05-31T20:29:25.849 DEBUG:boto:Path: /pklxkd-55/h?versionId=null 2023-05-31T20:29:25.850 DEBUG:boto:Data: 2023-05-31T20:29:25.850 DEBUG:boto:Headers: {} 2023-05-31T20:29:25.850 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:29:25.851 DEBUG:boto:Port: 8000 2023-05-31T20:29:25.851 DEBUG:boto:Params: {} 2023-05-31T20:29:25.851 DEBUG:boto:Token: None 2023-05-31T20:29:25.852 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:29:25 GMT /pklxkd-55/h?versionId=null 2023-05-31T20:29:25.852 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:4efssCLMCr2E9Yl7sdCyB23wbtw= 2023-05-31T20:29:25.853 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:29:25 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:4efssCLMCr2E9Yl7sdCyB23wbtw=', 'Content-Length': '0'} 2023-05-31T20:29:25.854 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:27:24 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002b74576e185f3faa-006477ae25-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:29:25 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:29:25.855 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=h 2023-05-31T20:29:25.855 DEBUG:boto:path=/pklxkd-55/h 2023-05-31T20:29:25.856 DEBUG:boto:auth_path=/pklxkd-55/h 2023-05-31T20:29:25.856 DEBUG:boto:path=/pklxkd-55/h?versionId=null 2023-05-31T20:29:25.857 DEBUG:boto:auth_path=/pklxkd-55/h?versionId=null 2023-05-31T20:29:25.857 DEBUG:boto:Method: GET 2023-05-31T20:29:25.858 DEBUG:boto:Path: /pklxkd-55/h?versionId=null 2023-05-31T20:29:25.858 DEBUG:boto:Data: 2023-05-31T20:29:25.858 DEBUG:boto:Headers: {} 2023-05-31T20:29:25.859 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:29:25.859 DEBUG:boto:Port: 8001 2023-05-31T20:29:25.859 DEBUG:boto:Params: {} 2023-05-31T20:29:25.860 DEBUG:boto:Token: None 2023-05-31T20:29:25.860 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:29:25 GMT /pklxkd-55/h?versionId=null 2023-05-31T20:29:25.861 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:4Y/8L8lDnhHI55GrOpuluMNFBUo= 2023-05-31T20:29:25.861 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:29:25 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:4Y/8L8lDnhHI55GrOpuluMNFBUo=', 'Content-Length': '0'} 2023-05-31T20:29:25.863 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:27:24 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000023c0be56f71e321e-006477ae25-162e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:29:25 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:29:25.864 DEBUG:boto:path=/pklxkd-55/h 2023-05-31T20:29:25.864 DEBUG:boto:auth_path=/pklxkd-55/h 2023-05-31T20:29:25.865 DEBUG:boto:path=/pklxkd-55/h?versionId=null 2023-05-31T20:29:25.865 DEBUG:boto:auth_path=/pklxkd-55/h?versionId=null 2023-05-31T20:29:25.866 DEBUG:boto:Method: GET 2023-05-31T20:29:25.866 DEBUG:boto:Path: /pklxkd-55/h?versionId=null 2023-05-31T20:29:25.866 DEBUG:boto:Data: 2023-05-31T20:29:25.867 DEBUG:boto:Headers: {} 2023-05-31T20:29:25.867 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:29:25.867 DEBUG:boto:Port: 8000 2023-05-31T20:29:25.868 DEBUG:boto:Params: {} 2023-05-31T20:29:25.868 DEBUG:boto:Token: None 2023-05-31T20:29:25.869 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:29:25 GMT /pklxkd-55/h?versionId=null 2023-05-31T20:29:25.869 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:4Y/8L8lDnhHI55GrOpuluMNFBUo= 2023-05-31T20:29:25.870 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:29:25 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:4Y/8L8lDnhHI55GrOpuluMNFBUo=', 'Content-Length': '0'} 2023-05-31T20:29:25.872 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:27:24 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007cafc6647d568eb0-006477ae25-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:29:25 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:29:25.872 DEBUG:boto:path=/pklxkd-55/ 2023-05-31T20:29:25.873 DEBUG:boto:auth_path=/pklxkd-55/ 2023-05-31T20:29:25.873 DEBUG:boto:path=/pklxkd-55/?prefix=h 2023-05-31T20:29:25.874 DEBUG:boto:auth_path=/pklxkd-55/?prefix=h 2023-05-31T20:29:25.874 DEBUG:boto:Method: GET 2023-05-31T20:29:25.874 DEBUG:boto:Path: /pklxkd-55/?prefix=h 2023-05-31T20:29:25.875 DEBUG:boto:Data: 2023-05-31T20:29:25.875 DEBUG:boto:Headers: {} 2023-05-31T20:29:25.875 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:29:25.876 DEBUG:boto:Port: 8001 2023-05-31T20:29:25.876 DEBUG:boto:Params: {} 2023-05-31T20:29:25.877 DEBUG:boto:Token: None 2023-05-31T20:29:25.877 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:29:25 GMT /pklxkd-55/ 2023-05-31T20:29:25.877 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:a0/qPp0bnZeH62hZJf6OA1a38dw= 2023-05-31T20:29:25.878 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:29:25 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:a0/qPp0bnZeH62hZJf6OA1a38dw=', 'Content-Length': '0'} 2023-05-31T20:29:25.880 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000edbfa582cb1ba6df-006477ae25-162e-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:29:25 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:29:25.881 DEBUG:boto:b'pklxkd-55h1000falseh2023-05-31T20:27:24.053Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:29:25.881 DEBUG:boto:path=/pklxkd-55/ 2023-05-31T20:29:25.882 DEBUG:boto:auth_path=/pklxkd-55/ 2023-05-31T20:29:25.882 DEBUG:boto:path=/pklxkd-55/?prefix=h 2023-05-31T20:29:25.883 DEBUG:boto:auth_path=/pklxkd-55/?prefix=h 2023-05-31T20:29:25.883 DEBUG:boto:Method: GET 2023-05-31T20:29:25.883 DEBUG:boto:Path: /pklxkd-55/?prefix=h 2023-05-31T20:29:25.884 DEBUG:boto:Data: 2023-05-31T20:29:25.884 DEBUG:boto:Headers: {} 2023-05-31T20:29:25.884 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:29:25.885 DEBUG:boto:Port: 8000 2023-05-31T20:29:25.885 DEBUG:boto:Params: {} 2023-05-31T20:29:25.885 DEBUG:boto:Token: None 2023-05-31T20:29:25.886 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:29:25 GMT /pklxkd-55/ 2023-05-31T20:29:25.886 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:a0/qPp0bnZeH62hZJf6OA1a38dw= 2023-05-31T20:29:25.887 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:29:25 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:a0/qPp0bnZeH62hZJf6OA1a38dw=', 'Content-Length': '0'} 2023-05-31T20:29:25.889 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d56fab84570fcd38-006477ae25-1630-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:29:25 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:29:25.889 DEBUG:boto:b'pklxkd-55h1000falseh2023-05-31T20:27:24.053Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:29:25.890 DEBUG:boto:path=/pklxkd-55/h 2023-05-31T20:29:25.890 DEBUG:boto:auth_path=/pklxkd-55/h 2023-05-31T20:29:25.891 DEBUG:boto:Method: HEAD 2023-05-31T20:29:25.891 DEBUG:boto:Path: /pklxkd-55/h 2023-05-31T20:29:25.891 DEBUG:boto:Data: 2023-05-31T20:29:25.892 DEBUG:boto:Headers: {} 2023-05-31T20:29:25.892 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:29:25.892 DEBUG:boto:Port: 8001 2023-05-31T20:29:25.893 DEBUG:boto:Params: {} 2023-05-31T20:29:25.893 DEBUG:boto:Token: None 2023-05-31T20:29:25.894 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:29:25 GMT /pklxkd-55/h 2023-05-31T20:29:25.894 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:EJL0qHt0IGOrq/lgbMCgfQTfo7U= 2023-05-31T20:29:25.894 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:29:25 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:EJL0qHt0IGOrq/lgbMCgfQTfo7U=', 'Content-Length': '0'} 2023-05-31T20:29:25.896 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:27:24 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000221909ccab4120e2-006477ae25-162e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:29:25 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:29:25.897 DEBUG:boto:path=/pklxkd-55/h 2023-05-31T20:29:25.897 DEBUG:boto:auth_path=/pklxkd-55/h 2023-05-31T20:29:25.898 DEBUG:boto:Method: HEAD 2023-05-31T20:29:25.898 DEBUG:boto:Path: /pklxkd-55/h 2023-05-31T20:29:25.898 DEBUG:boto:Data: 2023-05-31T20:29:25.899 DEBUG:boto:Headers: {} 2023-05-31T20:29:25.899 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:29:25.899 DEBUG:boto:Port: 8000 2023-05-31T20:29:25.900 DEBUG:boto:Params: {} 2023-05-31T20:29:25.900 DEBUG:boto:Token: None 2023-05-31T20:29:25.900 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:29:25 GMT /pklxkd-55/h 2023-05-31T20:29:25.901 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:EJL0qHt0IGOrq/lgbMCgfQTfo7U= 2023-05-31T20:29:25.901 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:29:25 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:EJL0qHt0IGOrq/lgbMCgfQTfo7U=', 'Content-Length': '0'} 2023-05-31T20:29:25.903 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:27:24 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000869729a74204d4de-006477ae25-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:29:25 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:29:25.904 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=h 2023-05-31T20:29:25.904 DEBUG:boto:path=/pklxkd-55/h 2023-05-31T20:29:25.904 DEBUG:boto:auth_path=/pklxkd-55/h 2023-05-31T20:29:25.905 DEBUG:boto:Method: GET 2023-05-31T20:29:25.905 DEBUG:boto:Path: /pklxkd-55/h 2023-05-31T20:29:25.905 DEBUG:boto:Data: 2023-05-31T20:29:25.906 DEBUG:boto:Headers: {} 2023-05-31T20:29:25.906 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:29:25.906 DEBUG:boto:Port: 8001 2023-05-31T20:29:25.907 DEBUG:boto:Params: {} 2023-05-31T20:29:25.907 DEBUG:boto:Token: None 2023-05-31T20:29:25.908 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:29:25 GMT /pklxkd-55/h 2023-05-31T20:29:25.908 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:Nr+uUYQjACsxuRi6Tmzu4v68dVo= 2023-05-31T20:29:25.909 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:29:25 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:Nr+uUYQjACsxuRi6Tmzu4v68dVo=', 'Content-Length': '0'} 2023-05-31T20:29:25.911 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:27:24 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000ca95da153df03d2-006477ae25-162e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:29:25 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:29:25.956 DEBUG:boto:path=/pklxkd-55/h 2023-05-31T20:29:25.957 DEBUG:boto:auth_path=/pklxkd-55/h 2023-05-31T20:29:25.957 DEBUG:boto:Method: GET 2023-05-31T20:29:25.958 DEBUG:boto:Path: /pklxkd-55/h 2023-05-31T20:29:25.958 DEBUG:boto:Data: 2023-05-31T20:29:25.959 DEBUG:boto:Headers: {} 2023-05-31T20:29:25.959 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:29:25.960 DEBUG:boto:Port: 8000 2023-05-31T20:29:25.960 DEBUG:boto:Params: {} 2023-05-31T20:29:25.961 DEBUG:boto:Token: None 2023-05-31T20:29:25.961 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:29:25 GMT /pklxkd-55/h 2023-05-31T20:29:25.962 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:Nr+uUYQjACsxuRi6Tmzu4v68dVo= 2023-05-31T20:29:25.962 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:29:25 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:Nr+uUYQjACsxuRi6Tmzu4v68dVo=', 'Content-Length': '0'} 2023-05-31T20:29:25.964 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:27:24 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d849d8934f2f837b-006477ae25-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:29:25 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:29:25.965 DEBUG:boto:path=/pklxkd-55/ 2023-05-31T20:29:25.965 DEBUG:boto:auth_path=/pklxkd-55/ 2023-05-31T20:29:25.966 DEBUG:boto:path=/pklxkd-55/?prefix=h 2023-05-31T20:29:25.966 DEBUG:boto:auth_path=/pklxkd-55/?prefix=h 2023-05-31T20:29:25.966 DEBUG:boto:Method: GET 2023-05-31T20:29:25.967 DEBUG:boto:Path: /pklxkd-55/?prefix=h 2023-05-31T20:29:25.967 DEBUG:boto:Data: 2023-05-31T20:29:25.968 DEBUG:boto:Headers: {} 2023-05-31T20:29:25.968 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:29:25.968 DEBUG:boto:Port: 8001 2023-05-31T20:29:25.969 DEBUG:boto:Params: {} 2023-05-31T20:29:25.969 DEBUG:boto:Token: None 2023-05-31T20:29:25.970 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:29:25 GMT /pklxkd-55/ 2023-05-31T20:29:25.971 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:a0/qPp0bnZeH62hZJf6OA1a38dw= 2023-05-31T20:29:25.971 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:29:25 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:a0/qPp0bnZeH62hZJf6OA1a38dw=', 'Content-Length': '0'} 2023-05-31T20:29:25.973 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e73cffee0e9c841f-006477ae25-162e-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:29:25 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:29:25.974 DEBUG:boto:b'pklxkd-55h1000falseh2023-05-31T20:27:24.053Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:29:25.974 DEBUG:boto:path=/pklxkd-55/ 2023-05-31T20:29:25.975 DEBUG:boto:auth_path=/pklxkd-55/ 2023-05-31T20:29:25.975 DEBUG:boto:path=/pklxkd-55/?prefix=h 2023-05-31T20:29:25.976 DEBUG:boto:auth_path=/pklxkd-55/?prefix=h 2023-05-31T20:29:25.976 DEBUG:boto:Method: GET 2023-05-31T20:29:25.976 DEBUG:boto:Path: /pklxkd-55/?prefix=h 2023-05-31T20:29:25.977 DEBUG:boto:Data: 2023-05-31T20:29:25.977 DEBUG:boto:Headers: {} 2023-05-31T20:29:25.978 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:29:25.978 DEBUG:boto:Port: 8000 2023-05-31T20:29:25.978 DEBUG:boto:Params: {} 2023-05-31T20:29:25.979 DEBUG:boto:Token: None 2023-05-31T20:29:25.979 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:29:25 GMT /pklxkd-55/ 2023-05-31T20:29:25.980 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:a0/qPp0bnZeH62hZJf6OA1a38dw= 2023-05-31T20:29:25.980 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:29:25 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:a0/qPp0bnZeH62hZJf6OA1a38dw=', 'Content-Length': '0'} 2023-05-31T20:29:25.982 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000eb1c80318bda261b-006477ae25-1630-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:29:25 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:29:26.028 DEBUG:boto:b'pklxkd-55h1000falseh2023-05-31T20:27:24.053Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:29:26.029 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=pklxkd-55 zones={a2, a1} 2023-05-31T20:29:26.030 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'reshard', '--bucket', 'pklxkd-55', '--num-shards', '3', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-05-31T20:29:26.030 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'reshard', '--bucket', 'pklxkd-55', '--num-shards', '3', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-05-31T20:29:26.030 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket reshard --bucket pklxkd-55 --num-shards 3 --yes-i-really-mean-it --cluster c1 --debug-rgw 0 --debug-ms 0 2023-05-31T20:29:26.088 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2023-05-31T20:29:26.089 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2023-05-31T20:29:26.151 INFO:teuthology.orchestra.run.smithi099.stdout:tenant: 2023-05-31T20:29:26.151 INFO:teuthology.orchestra.run.smithi099.stdout:bucket name: pklxkd-55 2023-05-31T20:29:26.152 INFO:teuthology.orchestra.run.smithi099.stdout:total entries: 8 2023-05-31T20:29:26.190 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'layout', '--bucket', 'pklxkd-55', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-05-31T20:29:26.191 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'layout', '--bucket', 'pklxkd-55', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-05-31T20:29:26.192 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket layout --bucket pklxkd-55 --cluster c1 --debug-rgw 0 --debug-ms 0 2023-05-31T20:29:26.239 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2023-05-31T20:29:26.239 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2023-05-31T20:29:26.312 INFO:teuthology.orchestra.run.smithi099.stdout:{ 2023-05-31T20:29:26.312 INFO:teuthology.orchestra.run.smithi099.stdout: "layout": { 2023-05-31T20:29:26.312 INFO:teuthology.orchestra.run.smithi099.stdout: "resharding": "None", 2023-05-31T20:29:26.313 INFO:teuthology.orchestra.run.smithi099.stdout: "current_index": { 2023-05-31T20:29:26.313 INFO:teuthology.orchestra.run.smithi099.stdout: "gen": 2, 2023-05-31T20:29:26.313 INFO:teuthology.orchestra.run.smithi099.stdout: "layout": { 2023-05-31T20:29:26.313 INFO:teuthology.orchestra.run.smithi099.stdout: "type": "Normal", 2023-05-31T20:29:26.313 INFO:teuthology.orchestra.run.smithi099.stdout: "normal": { 2023-05-31T20:29:26.313 INFO:teuthology.orchestra.run.smithi099.stdout: "num_shards": 3, 2023-05-31T20:29:26.314 INFO:teuthology.orchestra.run.smithi099.stdout: "hash_type": "Mod" 2023-05-31T20:29:26.314 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T20:29:26.314 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T20:29:26.314 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:29:26.315 INFO:teuthology.orchestra.run.smithi099.stdout: "logs": [ 2023-05-31T20:29:26.315 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T20:29:26.315 INFO:teuthology.orchestra.run.smithi099.stdout: "gen": 0, 2023-05-31T20:29:26.315 INFO:teuthology.orchestra.run.smithi099.stdout: "layout": { 2023-05-31T20:29:26.315 INFO:teuthology.orchestra.run.smithi099.stdout: "type": "InIndex", 2023-05-31T20:29:26.315 INFO:teuthology.orchestra.run.smithi099.stdout: "in_index": { 2023-05-31T20:29:26.315 INFO:teuthology.orchestra.run.smithi099.stdout: "gen": 0, 2023-05-31T20:29:26.316 INFO:teuthology.orchestra.run.smithi099.stdout: "layout": { 2023-05-31T20:29:26.316 INFO:teuthology.orchestra.run.smithi099.stdout: "num_shards": 11, 2023-05-31T20:29:26.316 INFO:teuthology.orchestra.run.smithi099.stdout: "hash_type": "Mod" 2023-05-31T20:29:26.316 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T20:29:26.316 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T20:29:26.316 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T20:29:26.317 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:29:26.317 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T20:29:26.317 INFO:teuthology.orchestra.run.smithi099.stdout: "gen": 1, 2023-05-31T20:29:26.317 INFO:teuthology.orchestra.run.smithi099.stdout: "layout": { 2023-05-31T20:29:26.317 INFO:teuthology.orchestra.run.smithi099.stdout: "type": "InIndex", 2023-05-31T20:29:26.317 INFO:teuthology.orchestra.run.smithi099.stdout: "in_index": { 2023-05-31T20:29:26.317 INFO:teuthology.orchestra.run.smithi099.stdout: "gen": 1, 2023-05-31T20:29:26.318 INFO:teuthology.orchestra.run.smithi099.stdout: "layout": { 2023-05-31T20:29:26.318 INFO:teuthology.orchestra.run.smithi099.stdout: "num_shards": 3, 2023-05-31T20:29:26.318 INFO:teuthology.orchestra.run.smithi099.stdout: "hash_type": "Mod" 2023-05-31T20:29:26.318 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T20:29:26.318 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T20:29:26.318 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T20:29:26.318 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:29:26.319 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T20:29:26.319 INFO:teuthology.orchestra.run.smithi099.stdout: "gen": 2, 2023-05-31T20:29:26.319 INFO:teuthology.orchestra.run.smithi099.stdout: "layout": { 2023-05-31T20:29:26.319 INFO:teuthology.orchestra.run.smithi099.stdout: "type": "InIndex", 2023-05-31T20:29:26.319 INFO:teuthology.orchestra.run.smithi099.stdout: "in_index": { 2023-05-31T20:29:26.319 INFO:teuthology.orchestra.run.smithi099.stdout: "gen": 2, 2023-05-31T20:29:26.319 INFO:teuthology.orchestra.run.smithi099.stdout: "layout": { 2023-05-31T20:29:26.320 INFO:teuthology.orchestra.run.smithi099.stdout: "num_shards": 3, 2023-05-31T20:29:26.320 INFO:teuthology.orchestra.run.smithi099.stdout: "hash_type": "Mod" 2023-05-31T20:29:26.320 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T20:29:26.320 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T20:29:26.320 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T20:29:26.320 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T20:29:26.320 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2023-05-31T20:29:26.321 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T20:29:26.321 INFO:teuthology.orchestra.run.smithi099.stdout:} 2023-05-31T20:29:26.322 DEBUG:rgw_multi.tests:bucket checkpoint: source=a1 target=a2 bucket=pklxkd-55 2023-05-31T20:29:26.322 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'pklxkd-55', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-05-31T20:29:26.323 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'pklxkd-55', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-05-31T20:29:26.323 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket pklxkd-55 --source-zone a1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2023-05-31T20:29:26.380 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2023-05-31T20:29:26.380 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2023-05-31T20:29:26.422 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:29:26.417+0000 7f6086eab780 1 waiting to reach latest gen 2, on 1.. 2023-05-31T20:29:31.423 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:29:31.418+0000 7f6086eab780 1 waiting to reach latest gen 2, on 1.. 2023-05-31T20:29:36.424 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:29:36.419+0000 7f6086eab780 1 waiting to reach latest gen 2, on 1.. 2023-05-31T20:29:41.424 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:29:41.420+0000 7f6086eab780 1 waiting to reach latest gen 2, on 1.. 2023-05-31T20:29:46.425 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:29:46.420+0000 7f6086eab780 1 bucket sync caught up with empty source 2023-05-31T20:29:46.425 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:29:46.420+0000 7f6086eab780 0 bucket checkpoint complete 2023-05-31T20:29:46.443 DEBUG:rgw_multi.tests:bucket checkpoint: source=a2 target=a1 bucket=pklxkd-55 2023-05-31T20:29:46.444 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'pklxkd-55', '--source-zone', 'a2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-05-31T20:29:46.444 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'pklxkd-55', '--source-zone', 'a2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-05-31T20:29:46.445 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket pklxkd-55 --source-zone a2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2023-05-31T20:29:46.506 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2023-05-31T20:29:46.507 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2023-05-31T20:29:46.558 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:29:46.554+0000 7f5c73079780 1 bucket sync caught up with source: 2023-05-31T20:29:46.558 INFO:teuthology.orchestra.run.smithi099.stderr: local status: [, 00000000002.1063.6, , , 00000000001.1666.6, 00000000001.1340.6, , , 00000000002.3558.6, , 00000000002.2408.6] 2023-05-31T20:29:46.559 INFO:teuthology.orchestra.run.smithi099.stderr: remote markers: [, 00000000002.1063.6, , , 00000000001.1666.6, 00000000001.1340.6, , , 00000000002.3558.6, , 00000000002.2408.6] 2023-05-31T20:29:46.559 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:29:46.554+0000 7f5c73079780 0 bucket checkpoint complete 2023-05-31T20:29:46.579 INFO:tasks.rgw_multi.zone_rados:comparing bucket=pklxkd-55 zones={a2, a1} 2023-05-31T20:29:46.580 DEBUG:boto:path=/pklxkd-55/ 2023-05-31T20:29:46.580 DEBUG:boto:auth_path=/pklxkd-55/ 2023-05-31T20:29:46.581 DEBUG:boto:Method: HEAD 2023-05-31T20:29:46.581 DEBUG:boto:Path: /pklxkd-55/ 2023-05-31T20:29:46.581 DEBUG:boto:Data: 2023-05-31T20:29:46.582 DEBUG:boto:Headers: {} 2023-05-31T20:29:46.582 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:29:46.582 DEBUG:boto:Port: 8001 2023-05-31T20:29:46.583 DEBUG:boto:Params: {} 2023-05-31T20:29:46.583 DEBUG:boto:Token: None 2023-05-31T20:29:46.584 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:29:46 GMT /pklxkd-55/ 2023-05-31T20:29:46.584 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:n/yoparVfMjq9ET3vWekmT7kbq0= 2023-05-31T20:29:46.585 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:29:46 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:n/yoparVfMjq9ET3vWekmT7kbq0=', 'Content-Length': '0'} 2023-05-31T20:29:46.587 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '8'), ('X-RGW-Bytes-Used', '24'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000c2632836ee40db21-006477ae3a-162e-a2'), ('Content-Length', '0'), ('Date', 'Wed, 31 May 2023 20:29:46 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:29:46.587 DEBUG:boto:path=/pklxkd-55/ 2023-05-31T20:29:46.588 DEBUG:boto:auth_path=/pklxkd-55/ 2023-05-31T20:29:46.588 DEBUG:boto:Method: HEAD 2023-05-31T20:29:46.589 DEBUG:boto:Path: /pklxkd-55/ 2023-05-31T20:29:46.589 DEBUG:boto:Data: 2023-05-31T20:29:46.589 DEBUG:boto:Headers: {} 2023-05-31T20:29:46.590 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:29:46.590 DEBUG:boto:Port: 8000 2023-05-31T20:29:46.590 DEBUG:boto:Params: {} 2023-05-31T20:29:46.591 DEBUG:boto:Token: None 2023-05-31T20:29:46.591 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:29:46 GMT /pklxkd-55/ 2023-05-31T20:29:46.591 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:n/yoparVfMjq9ET3vWekmT7kbq0= 2023-05-31T20:29:46.592 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:29:46 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:n/yoparVfMjq9ET3vWekmT7kbq0=', 'Content-Length': '0'} 2023-05-31T20:29:46.593 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '8'), ('X-RGW-Bytes-Used', '24'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000079d1e41e1e695184-006477ae3a-1630-a1'), ('Content-Length', '0'), ('Date', 'Wed, 31 May 2023 20:29:46 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:29:46.594 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2023-05-31T20:29:46.594 DEBUG:boto:path=/pklxkd-55/ 2023-05-31T20:29:46.595 DEBUG:boto:auth_path=/pklxkd-55/ 2023-05-31T20:29:46.595 DEBUG:boto:path=/pklxkd-55/?versions&max-keys=999 2023-05-31T20:29:46.595 DEBUG:boto:auth_path=/pklxkd-55/?versions&max-keys=999 2023-05-31T20:29:46.596 DEBUG:boto:Method: GET 2023-05-31T20:29:46.596 DEBUG:boto:Path: /pklxkd-55/?versions&max-keys=999 2023-05-31T20:29:46.597 DEBUG:boto:Data: 2023-05-31T20:29:46.597 DEBUG:boto:Headers: {} 2023-05-31T20:29:46.597 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:29:46.598 DEBUG:boto:Port: 8001 2023-05-31T20:29:46.598 DEBUG:boto:Params: {} 2023-05-31T20:29:46.599 DEBUG:boto:Token: None 2023-05-31T20:29:46.599 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:29:46 GMT /pklxkd-55/?versions 2023-05-31T20:29:46.600 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:rGmKTw6GqSba2RfYy5rKgh3wg8w= 2023-05-31T20:29:46.600 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:29:46 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:rGmKTw6GqSba2RfYy5rKgh3wg8w=', 'Content-Length': '0'} 2023-05-31T20:29:46.602 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000093dd22a1e6a74c0a-006477ae3a-162e-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:29:46 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:29:46.603 DEBUG:boto:b'pklxkd-55999falseanulltrue2023-05-31T20:27:16.792Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalbnulltrue2023-05-31T20:27:16.850Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalcnulltrue2023-05-31T20:27:16.869Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormaldnulltrue2023-05-31T20:27:16.893Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalenulltrue2023-05-31T20:27:23.955Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalfnulltrue2023-05-31T20:27:23.977Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalgnulltrue2023-05-31T20:27:24.035Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalhnulltrue2023-05-31T20:27:24.053Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:29:46.604 DEBUG:tasks.rgw_multi.zone_rados:o=a 2023-05-31T20:29:46.604 DEBUG:tasks.rgw_multi.zone_rados:o=b 2023-05-31T20:29:46.604 DEBUG:tasks.rgw_multi.zone_rados:o=c 2023-05-31T20:29:46.605 DEBUG:tasks.rgw_multi.zone_rados:o=d 2023-05-31T20:29:46.605 DEBUG:tasks.rgw_multi.zone_rados:o=e 2023-05-31T20:29:46.605 DEBUG:tasks.rgw_multi.zone_rados:o=f 2023-05-31T20:29:46.606 DEBUG:tasks.rgw_multi.zone_rados:o=g 2023-05-31T20:29:46.606 DEBUG:tasks.rgw_multi.zone_rados:o=h 2023-05-31T20:29:46.606 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2023-05-31T20:29:46.607 DEBUG:boto:path=/pklxkd-55/ 2023-05-31T20:29:46.607 DEBUG:boto:auth_path=/pklxkd-55/ 2023-05-31T20:29:46.607 DEBUG:boto:path=/pklxkd-55/?versions&max-keys=999 2023-05-31T20:29:46.608 DEBUG:boto:auth_path=/pklxkd-55/?versions&max-keys=999 2023-05-31T20:29:46.608 DEBUG:boto:Method: GET 2023-05-31T20:29:46.608 DEBUG:boto:Path: /pklxkd-55/?versions&max-keys=999 2023-05-31T20:29:46.609 DEBUG:boto:Data: 2023-05-31T20:29:46.609 DEBUG:boto:Headers: {} 2023-05-31T20:29:46.610 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:29:46.610 DEBUG:boto:Port: 8000 2023-05-31T20:29:46.610 DEBUG:boto:Params: {} 2023-05-31T20:29:46.611 DEBUG:boto:Token: None 2023-05-31T20:29:46.611 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:29:46 GMT /pklxkd-55/?versions 2023-05-31T20:29:46.612 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:rGmKTw6GqSba2RfYy5rKgh3wg8w= 2023-05-31T20:29:46.612 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:29:46 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:rGmKTw6GqSba2RfYy5rKgh3wg8w=', 'Content-Length': '0'} 2023-05-31T20:29:46.614 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000085dbbd5b236ca226-006477ae3a-1630-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:29:46 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:29:46.616 DEBUG:boto:b'pklxkd-55999falseanulltrue2023-05-31T20:27:16.792Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalbnulltrue2023-05-31T20:27:16.850Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalcnulltrue2023-05-31T20:27:16.869Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormaldnulltrue2023-05-31T20:27:16.893Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalenulltrue2023-05-31T20:27:23.955Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalfnulltrue2023-05-31T20:27:23.977Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalgnulltrue2023-05-31T20:27:24.035Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalhnulltrue2023-05-31T20:27:24.053Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:29:46.617 DEBUG:tasks.rgw_multi.zone_rados:o=a 2023-05-31T20:29:46.618 DEBUG:tasks.rgw_multi.zone_rados:o=b 2023-05-31T20:29:46.618 DEBUG:tasks.rgw_multi.zone_rados:o=c 2023-05-31T20:29:46.619 DEBUG:tasks.rgw_multi.zone_rados:o=d 2023-05-31T20:29:46.620 DEBUG:tasks.rgw_multi.zone_rados:o=e 2023-05-31T20:29:46.620 DEBUG:tasks.rgw_multi.zone_rados:o=f 2023-05-31T20:29:46.621 DEBUG:tasks.rgw_multi.zone_rados:o=g 2023-05-31T20:29:46.621 DEBUG:tasks.rgw_multi.zone_rados:o=h 2023-05-31T20:29:46.622 DEBUG:boto:path=/pklxkd-55/ 2023-05-31T20:29:46.623 DEBUG:boto:auth_path=/pklxkd-55/ 2023-05-31T20:29:46.623 DEBUG:boto:path=/pklxkd-55/?versions&max-keys=999 2023-05-31T20:29:46.624 DEBUG:boto:auth_path=/pklxkd-55/?versions&max-keys=999 2023-05-31T20:29:46.624 DEBUG:boto:Method: GET 2023-05-31T20:29:46.625 DEBUG:boto:Path: /pklxkd-55/?versions&max-keys=999 2023-05-31T20:29:46.626 DEBUG:boto:Data: 2023-05-31T20:29:46.626 DEBUG:boto:Headers: {} 2023-05-31T20:29:46.627 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:29:46.627 DEBUG:boto:Port: 8001 2023-05-31T20:29:46.628 DEBUG:boto:Params: {} 2023-05-31T20:29:46.629 DEBUG:boto:Token: None 2023-05-31T20:29:46.630 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:29:46 GMT /pklxkd-55/?versions 2023-05-31T20:29:46.631 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:rGmKTw6GqSba2RfYy5rKgh3wg8w= 2023-05-31T20:29:46.632 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:29:46 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:rGmKTw6GqSba2RfYy5rKgh3wg8w=', 'Content-Length': '0'} 2023-05-31T20:29:46.635 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000899bdab387f96b14-006477ae3a-162e-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:29:46 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:29:46.636 DEBUG:boto:b'pklxkd-55999falseanulltrue2023-05-31T20:27:16.792Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalbnulltrue2023-05-31T20:27:16.850Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalcnulltrue2023-05-31T20:27:16.869Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormaldnulltrue2023-05-31T20:27:16.893Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalenulltrue2023-05-31T20:27:23.955Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalfnulltrue2023-05-31T20:27:23.977Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalgnulltrue2023-05-31T20:27:24.035Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalhnulltrue2023-05-31T20:27:24.053Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:29:46.638 DEBUG:boto:path=/pklxkd-55/ 2023-05-31T20:29:46.639 DEBUG:boto:auth_path=/pklxkd-55/ 2023-05-31T20:29:46.639 DEBUG:boto:path=/pklxkd-55/?versions&max-keys=999 2023-05-31T20:29:46.639 DEBUG:boto:auth_path=/pklxkd-55/?versions&max-keys=999 2023-05-31T20:29:46.640 DEBUG:boto:Method: GET 2023-05-31T20:29:46.640 DEBUG:boto:Path: /pklxkd-55/?versions&max-keys=999 2023-05-31T20:29:46.641 DEBUG:boto:Data: 2023-05-31T20:29:46.641 DEBUG:boto:Headers: {} 2023-05-31T20:29:46.641 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:29:46.642 DEBUG:boto:Port: 8000 2023-05-31T20:29:46.642 DEBUG:boto:Params: {} 2023-05-31T20:29:46.642 DEBUG:boto:Token: None 2023-05-31T20:29:46.643 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:29:46 GMT /pklxkd-55/?versions 2023-05-31T20:29:46.643 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:rGmKTw6GqSba2RfYy5rKgh3wg8w= 2023-05-31T20:29:46.645 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:29:46 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:rGmKTw6GqSba2RfYy5rKgh3wg8w=', 'Content-Length': '0'} 2023-05-31T20:29:46.647 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000afa4376b4dd6c8ed-006477ae3a-1630-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:29:46 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:29:46.648 DEBUG:boto:b'pklxkd-55999falseanulltrue2023-05-31T20:27:16.792Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalbnulltrue2023-05-31T20:27:16.850Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalcnulltrue2023-05-31T20:27:16.869Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormaldnulltrue2023-05-31T20:27:16.893Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalenulltrue2023-05-31T20:27:23.955Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalfnulltrue2023-05-31T20:27:23.977Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalgnulltrue2023-05-31T20:27:24.035Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalhnulltrue2023-05-31T20:27:24.053Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:29:46.649 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=a 2023-05-31T20:29:46.650 DEBUG:boto:path=/pklxkd-55/a 2023-05-31T20:29:46.651 DEBUG:boto:auth_path=/pklxkd-55/a 2023-05-31T20:29:46.652 DEBUG:boto:path=/pklxkd-55/a?versionId=null 2023-05-31T20:29:46.652 DEBUG:boto:auth_path=/pklxkd-55/a?versionId=null 2023-05-31T20:29:46.652 DEBUG:boto:Method: GET 2023-05-31T20:29:46.653 DEBUG:boto:Path: /pklxkd-55/a?versionId=null 2023-05-31T20:29:46.653 DEBUG:boto:Data: 2023-05-31T20:29:46.653 DEBUG:boto:Headers: {} 2023-05-31T20:29:46.654 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:29:46.654 DEBUG:boto:Port: 8001 2023-05-31T20:29:46.655 DEBUG:boto:Params: {} 2023-05-31T20:29:46.655 DEBUG:boto:Token: None 2023-05-31T20:29:46.655 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:29:46 GMT /pklxkd-55/a?versionId=null 2023-05-31T20:29:46.656 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:c4UKUgPjYepRbriw4uwPbqdCCx8= 2023-05-31T20:29:46.656 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:29:46 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:c4UKUgPjYepRbriw4uwPbqdCCx8=', 'Content-Length': '0'} 2023-05-31T20:29:46.658 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:27:16 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007dcd78de02405374-006477ae3a-162e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:29:46 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:29:46.659 DEBUG:boto:path=/pklxkd-55/a 2023-05-31T20:29:46.660 DEBUG:boto:auth_path=/pklxkd-55/a 2023-05-31T20:29:46.660 DEBUG:boto:path=/pklxkd-55/a?versionId=null 2023-05-31T20:29:46.661 DEBUG:boto:auth_path=/pklxkd-55/a?versionId=null 2023-05-31T20:29:46.661 DEBUG:boto:Method: GET 2023-05-31T20:29:46.661 DEBUG:boto:Path: /pklxkd-55/a?versionId=null 2023-05-31T20:29:46.662 DEBUG:boto:Data: 2023-05-31T20:29:46.662 DEBUG:boto:Headers: {} 2023-05-31T20:29:46.662 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:29:46.663 DEBUG:boto:Port: 8000 2023-05-31T20:29:46.663 DEBUG:boto:Params: {} 2023-05-31T20:29:46.663 DEBUG:boto:Token: None 2023-05-31T20:29:46.664 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:29:46 GMT /pklxkd-55/a?versionId=null 2023-05-31T20:29:46.664 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:c4UKUgPjYepRbriw4uwPbqdCCx8= 2023-05-31T20:29:46.665 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:29:46 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:c4UKUgPjYepRbriw4uwPbqdCCx8=', 'Content-Length': '0'} 2023-05-31T20:29:46.666 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:27:16 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000080e5d1e45cd8a7a9-006477ae3a-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:29:46 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:29:46.667 DEBUG:boto:path=/pklxkd-55/a 2023-05-31T20:29:46.668 DEBUG:boto:auth_path=/pklxkd-55/a 2023-05-31T20:29:46.668 DEBUG:boto:path=/pklxkd-55/a?versionId=null 2023-05-31T20:29:46.668 DEBUG:boto:auth_path=/pklxkd-55/a?versionId=null 2023-05-31T20:29:46.669 DEBUG:boto:Method: HEAD 2023-05-31T20:29:46.669 DEBUG:boto:Path: /pklxkd-55/a?versionId=null 2023-05-31T20:29:46.669 DEBUG:boto:Data: 2023-05-31T20:29:46.670 DEBUG:boto:Headers: {} 2023-05-31T20:29:46.670 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:29:46.670 DEBUG:boto:Port: 8001 2023-05-31T20:29:46.671 DEBUG:boto:Params: {} 2023-05-31T20:29:46.672 DEBUG:boto:Token: None 2023-05-31T20:29:46.672 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:29:46 GMT /pklxkd-55/a?versionId=null 2023-05-31T20:29:46.673 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:FZdFuuGnv5YAJEQP8rxZw81dAN8= 2023-05-31T20:29:46.673 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:29:46 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:FZdFuuGnv5YAJEQP8rxZw81dAN8=', 'Content-Length': '0'} 2023-05-31T20:29:46.674 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:27:16 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000920f8d56add47fdd-006477ae3a-162e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:29:46 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:29:46.675 DEBUG:boto:path=/pklxkd-55/a 2023-05-31T20:29:46.675 DEBUG:boto:auth_path=/pklxkd-55/a 2023-05-31T20:29:46.676 DEBUG:boto:path=/pklxkd-55/a?versionId=null 2023-05-31T20:29:46.676 DEBUG:boto:auth_path=/pklxkd-55/a?versionId=null 2023-05-31T20:29:46.676 DEBUG:boto:Method: HEAD 2023-05-31T20:29:46.677 DEBUG:boto:Path: /pklxkd-55/a?versionId=null 2023-05-31T20:29:46.677 DEBUG:boto:Data: 2023-05-31T20:29:46.678 DEBUG:boto:Headers: {} 2023-05-31T20:29:46.678 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:29:46.678 DEBUG:boto:Port: 8000 2023-05-31T20:29:46.679 DEBUG:boto:Params: {} 2023-05-31T20:29:46.679 DEBUG:boto:Token: None 2023-05-31T20:29:46.680 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:29:46 GMT /pklxkd-55/a?versionId=null 2023-05-31T20:29:46.680 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:FZdFuuGnv5YAJEQP8rxZw81dAN8= 2023-05-31T20:29:46.680 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:29:46 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:FZdFuuGnv5YAJEQP8rxZw81dAN8=', 'Content-Length': '0'} 2023-05-31T20:29:46.682 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:27:16 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000349696cec2b43e13-006477ae3a-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:29:46 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:29:46.683 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=a 2023-05-31T20:29:46.683 DEBUG:boto:path=/pklxkd-55/a 2023-05-31T20:29:46.684 DEBUG:boto:auth_path=/pklxkd-55/a 2023-05-31T20:29:46.684 DEBUG:boto:path=/pklxkd-55/a?versionId=null 2023-05-31T20:29:46.685 DEBUG:boto:auth_path=/pklxkd-55/a?versionId=null 2023-05-31T20:29:46.685 DEBUG:boto:Method: GET 2023-05-31T20:29:46.685 DEBUG:boto:Path: /pklxkd-55/a?versionId=null 2023-05-31T20:29:46.686 DEBUG:boto:Data: 2023-05-31T20:29:46.686 DEBUG:boto:Headers: {} 2023-05-31T20:29:46.686 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:29:46.687 DEBUG:boto:Port: 8001 2023-05-31T20:29:46.687 DEBUG:boto:Params: {} 2023-05-31T20:29:46.687 DEBUG:boto:Token: None 2023-05-31T20:29:46.688 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:29:46 GMT /pklxkd-55/a?versionId=null 2023-05-31T20:29:46.689 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:c4UKUgPjYepRbriw4uwPbqdCCx8= 2023-05-31T20:29:46.689 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:29:46 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:c4UKUgPjYepRbriw4uwPbqdCCx8=', 'Content-Length': '0'} 2023-05-31T20:29:46.691 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:27:16 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004c61cca21ffd61c5-006477ae3a-162e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:29:46 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:29:46.692 DEBUG:boto:path=/pklxkd-55/a 2023-05-31T20:29:46.693 DEBUG:boto:auth_path=/pklxkd-55/a 2023-05-31T20:29:46.693 DEBUG:boto:path=/pklxkd-55/a?versionId=null 2023-05-31T20:29:46.693 DEBUG:boto:auth_path=/pklxkd-55/a?versionId=null 2023-05-31T20:29:46.694 DEBUG:boto:Method: GET 2023-05-31T20:29:46.694 DEBUG:boto:Path: /pklxkd-55/a?versionId=null 2023-05-31T20:29:46.694 DEBUG:boto:Data: 2023-05-31T20:29:46.695 DEBUG:boto:Headers: {} 2023-05-31T20:29:46.695 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:29:46.696 DEBUG:boto:Port: 8000 2023-05-31T20:29:46.696 DEBUG:boto:Params: {} 2023-05-31T20:29:46.696 DEBUG:boto:Token: None 2023-05-31T20:29:46.697 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:29:46 GMT /pklxkd-55/a?versionId=null 2023-05-31T20:29:46.697 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:c4UKUgPjYepRbriw4uwPbqdCCx8= 2023-05-31T20:29:46.698 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:29:46 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:c4UKUgPjYepRbriw4uwPbqdCCx8=', 'Content-Length': '0'} 2023-05-31T20:29:46.700 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:27:16 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c8dfbb690d9645e2-006477ae3a-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:29:46 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:29:46.701 DEBUG:boto:path=/pklxkd-55/ 2023-05-31T20:29:46.701 DEBUG:boto:auth_path=/pklxkd-55/ 2023-05-31T20:29:46.702 DEBUG:boto:path=/pklxkd-55/?prefix=a 2023-05-31T20:29:46.702 DEBUG:boto:auth_path=/pklxkd-55/?prefix=a 2023-05-31T20:29:46.702 DEBUG:boto:Method: GET 2023-05-31T20:29:46.703 DEBUG:boto:Path: /pklxkd-55/?prefix=a 2023-05-31T20:29:46.703 DEBUG:boto:Data: 2023-05-31T20:29:46.703 DEBUG:boto:Headers: {} 2023-05-31T20:29:46.704 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:29:46.704 DEBUG:boto:Port: 8001 2023-05-31T20:29:46.705 DEBUG:boto:Params: {} 2023-05-31T20:29:46.705 DEBUG:boto:Token: None 2023-05-31T20:29:46.705 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:29:46 GMT /pklxkd-55/ 2023-05-31T20:29:46.706 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:XZ8tDAn1QJf1wBVwJgGN94Jp4hE= 2023-05-31T20:29:46.706 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:29:46 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:XZ8tDAn1QJf1wBVwJgGN94Jp4hE=', 'Content-Length': '0'} 2023-05-31T20:29:46.708 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a9ce75aabbe6fe26-006477ae3a-162e-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:29:46 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:29:46.709 DEBUG:boto:b'pklxkd-55a1000falsea2023-05-31T20:27:16.792Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:29:46.710 DEBUG:boto:path=/pklxkd-55/ 2023-05-31T20:29:46.710 DEBUG:boto:auth_path=/pklxkd-55/ 2023-05-31T20:29:46.711 DEBUG:boto:path=/pklxkd-55/?prefix=a 2023-05-31T20:29:46.711 DEBUG:boto:auth_path=/pklxkd-55/?prefix=a 2023-05-31T20:29:46.711 DEBUG:boto:Method: GET 2023-05-31T20:29:46.712 DEBUG:boto:Path: /pklxkd-55/?prefix=a 2023-05-31T20:29:46.712 DEBUG:boto:Data: 2023-05-31T20:29:46.712 DEBUG:boto:Headers: {} 2023-05-31T20:29:46.713 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:29:46.713 DEBUG:boto:Port: 8000 2023-05-31T20:29:46.714 DEBUG:boto:Params: {} 2023-05-31T20:29:46.714 DEBUG:boto:Token: None 2023-05-31T20:29:46.714 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:29:46 GMT /pklxkd-55/ 2023-05-31T20:29:46.715 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:XZ8tDAn1QJf1wBVwJgGN94Jp4hE= 2023-05-31T20:29:46.715 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:29:46 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:XZ8tDAn1QJf1wBVwJgGN94Jp4hE=', 'Content-Length': '0'} 2023-05-31T20:29:46.717 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ed98b74f53f67154-006477ae3a-1630-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:29:46 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:29:46.717 DEBUG:boto:b'pklxkd-55a1000falsea2023-05-31T20:27:16.792Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:29:46.718 DEBUG:boto:path=/pklxkd-55/a 2023-05-31T20:29:46.718 DEBUG:boto:auth_path=/pklxkd-55/a 2023-05-31T20:29:46.719 DEBUG:boto:Method: HEAD 2023-05-31T20:29:46.719 DEBUG:boto:Path: /pklxkd-55/a 2023-05-31T20:29:46.720 DEBUG:boto:Data: 2023-05-31T20:29:46.720 DEBUG:boto:Headers: {} 2023-05-31T20:29:46.720 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:29:46.721 DEBUG:boto:Port: 8001 2023-05-31T20:29:46.721 DEBUG:boto:Params: {} 2023-05-31T20:29:46.721 DEBUG:boto:Token: None 2023-05-31T20:29:46.722 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:29:46 GMT /pklxkd-55/a 2023-05-31T20:29:46.722 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:oU2fckqlUEqdKJ120jVqGHKOOWg= 2023-05-31T20:29:46.722 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:29:46 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:oU2fckqlUEqdKJ120jVqGHKOOWg=', 'Content-Length': '0'} 2023-05-31T20:29:46.724 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:27:16 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b0fa36fa327894fc-006477ae3a-162e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:29:46 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:29:46.725 DEBUG:boto:path=/pklxkd-55/a 2023-05-31T20:29:46.725 DEBUG:boto:auth_path=/pklxkd-55/a 2023-05-31T20:29:46.725 DEBUG:boto:Method: HEAD 2023-05-31T20:29:46.726 DEBUG:boto:Path: /pklxkd-55/a 2023-05-31T20:29:46.726 DEBUG:boto:Data: 2023-05-31T20:29:46.726 DEBUG:boto:Headers: {} 2023-05-31T20:29:46.727 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:29:46.727 DEBUG:boto:Port: 8000 2023-05-31T20:29:46.727 DEBUG:boto:Params: {} 2023-05-31T20:29:46.728 DEBUG:boto:Token: None 2023-05-31T20:29:46.728 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:29:46 GMT /pklxkd-55/a 2023-05-31T20:29:46.729 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:oU2fckqlUEqdKJ120jVqGHKOOWg= 2023-05-31T20:29:46.729 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:29:46 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:oU2fckqlUEqdKJ120jVqGHKOOWg=', 'Content-Length': '0'} 2023-05-31T20:29:46.731 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:27:16 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f4d186947c72b1ae-006477ae3a-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:29:46 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:29:46.731 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=a 2023-05-31T20:29:46.732 DEBUG:boto:path=/pklxkd-55/a 2023-05-31T20:29:46.732 DEBUG:boto:auth_path=/pklxkd-55/a 2023-05-31T20:29:46.733 DEBUG:boto:Method: GET 2023-05-31T20:29:46.733 DEBUG:boto:Path: /pklxkd-55/a 2023-05-31T20:29:46.733 DEBUG:boto:Data: 2023-05-31T20:29:46.734 DEBUG:boto:Headers: {} 2023-05-31T20:29:46.734 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:29:46.734 DEBUG:boto:Port: 8001 2023-05-31T20:29:46.735 DEBUG:boto:Params: {} 2023-05-31T20:29:46.735 DEBUG:boto:Token: None 2023-05-31T20:29:46.736 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:29:46 GMT /pklxkd-55/a 2023-05-31T20:29:46.736 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:zVsdgueAZ78g4GjCy3kgQ/VWQ6Y= 2023-05-31T20:29:46.737 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:29:46 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:zVsdgueAZ78g4GjCy3kgQ/VWQ6Y=', 'Content-Length': '0'} 2023-05-31T20:29:46.738 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:27:16 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001be3eeafb3076d22-006477ae3a-162e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:29:46 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:29:46.739 DEBUG:boto:path=/pklxkd-55/a 2023-05-31T20:29:46.739 DEBUG:boto:auth_path=/pklxkd-55/a 2023-05-31T20:29:46.740 DEBUG:boto:Method: GET 2023-05-31T20:29:46.740 DEBUG:boto:Path: /pklxkd-55/a 2023-05-31T20:29:46.740 DEBUG:boto:Data: 2023-05-31T20:29:46.741 DEBUG:boto:Headers: {} 2023-05-31T20:29:46.741 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:29:46.741 DEBUG:boto:Port: 8000 2023-05-31T20:29:46.742 DEBUG:boto:Params: {} 2023-05-31T20:29:46.742 DEBUG:boto:Token: None 2023-05-31T20:29:46.743 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:29:46 GMT /pklxkd-55/a 2023-05-31T20:29:46.744 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:zVsdgueAZ78g4GjCy3kgQ/VWQ6Y= 2023-05-31T20:29:46.744 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:29:46 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:zVsdgueAZ78g4GjCy3kgQ/VWQ6Y=', 'Content-Length': '0'} 2023-05-31T20:29:46.746 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:27:16 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b639e34b4c0e1e6e-006477ae3a-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:29:46 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:29:46.747 DEBUG:boto:path=/pklxkd-55/ 2023-05-31T20:29:46.747 DEBUG:boto:auth_path=/pklxkd-55/ 2023-05-31T20:29:46.748 DEBUG:boto:path=/pklxkd-55/?prefix=a 2023-05-31T20:29:46.748 DEBUG:boto:auth_path=/pklxkd-55/?prefix=a 2023-05-31T20:29:46.748 DEBUG:boto:Method: GET 2023-05-31T20:29:46.749 DEBUG:boto:Path: /pklxkd-55/?prefix=a 2023-05-31T20:29:46.749 DEBUG:boto:Data: 2023-05-31T20:29:46.750 DEBUG:boto:Headers: {} 2023-05-31T20:29:46.750 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:29:46.750 DEBUG:boto:Port: 8001 2023-05-31T20:29:46.751 DEBUG:boto:Params: {} 2023-05-31T20:29:46.752 DEBUG:boto:Token: None 2023-05-31T20:29:46.754 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:29:46 GMT /pklxkd-55/ 2023-05-31T20:29:46.755 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:XZ8tDAn1QJf1wBVwJgGN94Jp4hE= 2023-05-31T20:29:46.757 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:29:46 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:XZ8tDAn1QJf1wBVwJgGN94Jp4hE=', 'Content-Length': '0'} 2023-05-31T20:29:46.759 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f7440c964c2076c5-006477ae3a-162e-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:29:46 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:29:46.760 DEBUG:boto:b'pklxkd-55a1000falsea2023-05-31T20:27:16.792Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:29:46.761 DEBUG:boto:path=/pklxkd-55/ 2023-05-31T20:29:46.762 DEBUG:boto:auth_path=/pklxkd-55/ 2023-05-31T20:29:46.762 DEBUG:boto:path=/pklxkd-55/?prefix=a 2023-05-31T20:29:46.762 DEBUG:boto:auth_path=/pklxkd-55/?prefix=a 2023-05-31T20:29:46.763 DEBUG:boto:Method: GET 2023-05-31T20:29:46.763 DEBUG:boto:Path: /pklxkd-55/?prefix=a 2023-05-31T20:29:46.763 DEBUG:boto:Data: 2023-05-31T20:29:46.764 DEBUG:boto:Headers: {} 2023-05-31T20:29:46.764 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:29:46.764 DEBUG:boto:Port: 8000 2023-05-31T20:29:46.765 DEBUG:boto:Params: {} 2023-05-31T20:29:46.765 DEBUG:boto:Token: None 2023-05-31T20:29:46.766 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:29:46 GMT /pklxkd-55/ 2023-05-31T20:29:46.766 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:XZ8tDAn1QJf1wBVwJgGN94Jp4hE= 2023-05-31T20:29:46.767 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:29:46 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:XZ8tDAn1QJf1wBVwJgGN94Jp4hE=', 'Content-Length': '0'} 2023-05-31T20:29:46.769 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000008d349f11be9e4bf0-006477ae3a-1630-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:29:46 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:29:46.770 DEBUG:boto:b'pklxkd-55a1000falsea2023-05-31T20:27:16.792Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:29:46.771 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=b 2023-05-31T20:29:46.771 DEBUG:boto:path=/pklxkd-55/b 2023-05-31T20:29:46.771 DEBUG:boto:auth_path=/pklxkd-55/b 2023-05-31T20:29:46.772 DEBUG:boto:path=/pklxkd-55/b?versionId=null 2023-05-31T20:29:46.772 DEBUG:boto:auth_path=/pklxkd-55/b?versionId=null 2023-05-31T20:29:46.773 DEBUG:boto:Method: GET 2023-05-31T20:29:46.773 DEBUG:boto:Path: /pklxkd-55/b?versionId=null 2023-05-31T20:29:46.773 DEBUG:boto:Data: 2023-05-31T20:29:46.774 DEBUG:boto:Headers: {} 2023-05-31T20:29:46.774 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:29:46.774 DEBUG:boto:Port: 8001 2023-05-31T20:29:46.775 DEBUG:boto:Params: {} 2023-05-31T20:29:46.775 DEBUG:boto:Token: None 2023-05-31T20:29:46.776 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:29:46 GMT /pklxkd-55/b?versionId=null 2023-05-31T20:29:46.776 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:dohegYhb5mm2QkrqEQNUwnnKqu4= 2023-05-31T20:29:46.778 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:29:46 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:dohegYhb5mm2QkrqEQNUwnnKqu4=', 'Content-Length': '0'} 2023-05-31T20:29:46.779 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:27:16 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000049894b68834797bb-006477ae3a-162e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:29:46 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:29:46.780 DEBUG:boto:path=/pklxkd-55/b 2023-05-31T20:29:46.780 DEBUG:boto:auth_path=/pklxkd-55/b 2023-05-31T20:29:46.781 DEBUG:boto:path=/pklxkd-55/b?versionId=null 2023-05-31T20:29:46.781 DEBUG:boto:auth_path=/pklxkd-55/b?versionId=null 2023-05-31T20:29:46.781 DEBUG:boto:Method: GET 2023-05-31T20:29:46.782 DEBUG:boto:Path: /pklxkd-55/b?versionId=null 2023-05-31T20:29:46.782 DEBUG:boto:Data: 2023-05-31T20:29:46.782 DEBUG:boto:Headers: {} 2023-05-31T20:29:46.783 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:29:46.783 DEBUG:boto:Port: 8000 2023-05-31T20:29:46.783 DEBUG:boto:Params: {} 2023-05-31T20:29:46.784 DEBUG:boto:Token: None 2023-05-31T20:29:46.784 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:29:46 GMT /pklxkd-55/b?versionId=null 2023-05-31T20:29:46.785 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:dohegYhb5mm2QkrqEQNUwnnKqu4= 2023-05-31T20:29:46.785 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:29:46 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:dohegYhb5mm2QkrqEQNUwnnKqu4=', 'Content-Length': '0'} 2023-05-31T20:29:46.787 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:27:16 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c3dfe8c7b2dab992-006477ae3a-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:29:46 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:29:46.787 DEBUG:boto:path=/pklxkd-55/b 2023-05-31T20:29:46.789 DEBUG:boto:auth_path=/pklxkd-55/b 2023-05-31T20:29:46.790 DEBUG:boto:path=/pklxkd-55/b?versionId=null 2023-05-31T20:29:46.791 DEBUG:boto:auth_path=/pklxkd-55/b?versionId=null 2023-05-31T20:29:46.792 DEBUG:boto:Method: HEAD 2023-05-31T20:29:46.793 DEBUG:boto:Path: /pklxkd-55/b?versionId=null 2023-05-31T20:29:46.793 DEBUG:boto:Data: 2023-05-31T20:29:46.794 DEBUG:boto:Headers: {} 2023-05-31T20:29:46.794 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:29:46.795 DEBUG:boto:Port: 8001 2023-05-31T20:29:46.795 DEBUG:boto:Params: {} 2023-05-31T20:29:46.796 DEBUG:boto:Token: None 2023-05-31T20:29:46.797 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:29:46 GMT /pklxkd-55/b?versionId=null 2023-05-31T20:29:46.798 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:N8vDB2XHi9UTYW1fpiRhScLt+MU= 2023-05-31T20:29:46.799 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:29:46 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:N8vDB2XHi9UTYW1fpiRhScLt+MU=', 'Content-Length': '0'} 2023-05-31T20:29:46.802 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:27:16 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000089774d4fc910f08-006477ae3a-162e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:29:46 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:29:46.803 DEBUG:boto:path=/pklxkd-55/b 2023-05-31T20:29:46.803 DEBUG:boto:auth_path=/pklxkd-55/b 2023-05-31T20:29:46.804 DEBUG:boto:path=/pklxkd-55/b?versionId=null 2023-05-31T20:29:46.804 DEBUG:boto:auth_path=/pklxkd-55/b?versionId=null 2023-05-31T20:29:46.805 DEBUG:boto:Method: HEAD 2023-05-31T20:29:46.806 DEBUG:boto:Path: /pklxkd-55/b?versionId=null 2023-05-31T20:29:46.806 DEBUG:boto:Data: 2023-05-31T20:29:46.807 DEBUG:boto:Headers: {} 2023-05-31T20:29:46.808 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:29:46.808 DEBUG:boto:Port: 8000 2023-05-31T20:29:46.809 DEBUG:boto:Params: {} 2023-05-31T20:29:46.810 DEBUG:boto:Token: None 2023-05-31T20:29:46.810 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:29:46 GMT /pklxkd-55/b?versionId=null 2023-05-31T20:29:46.811 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:N8vDB2XHi9UTYW1fpiRhScLt+MU= 2023-05-31T20:29:46.812 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:29:46 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:N8vDB2XHi9UTYW1fpiRhScLt+MU=', 'Content-Length': '0'} 2023-05-31T20:29:46.814 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:27:16 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d3007d24be31849e-006477ae3a-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:29:46 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:29:46.814 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=b 2023-05-31T20:29:46.815 DEBUG:boto:path=/pklxkd-55/b 2023-05-31T20:29:46.816 DEBUG:boto:auth_path=/pklxkd-55/b 2023-05-31T20:29:46.817 DEBUG:boto:path=/pklxkd-55/b?versionId=null 2023-05-31T20:29:46.818 DEBUG:boto:auth_path=/pklxkd-55/b?versionId=null 2023-05-31T20:29:46.819 DEBUG:boto:Method: GET 2023-05-31T20:29:46.820 DEBUG:boto:Path: /pklxkd-55/b?versionId=null 2023-05-31T20:29:46.820 DEBUG:boto:Data: 2023-05-31T20:29:46.821 DEBUG:boto:Headers: {} 2023-05-31T20:29:46.821 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:29:46.821 DEBUG:boto:Port: 8001 2023-05-31T20:29:46.822 DEBUG:boto:Params: {} 2023-05-31T20:29:46.822 DEBUG:boto:Token: None 2023-05-31T20:29:46.823 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:29:46 GMT /pklxkd-55/b?versionId=null 2023-05-31T20:29:46.823 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:dohegYhb5mm2QkrqEQNUwnnKqu4= 2023-05-31T20:29:46.824 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:29:46 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:dohegYhb5mm2QkrqEQNUwnnKqu4=', 'Content-Length': '0'} 2023-05-31T20:29:46.825 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:27:16 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b91b038f54a4410a-006477ae3a-162e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:29:46 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:29:46.868 DEBUG:boto:path=/pklxkd-55/b 2023-05-31T20:29:46.869 DEBUG:boto:auth_path=/pklxkd-55/b 2023-05-31T20:29:46.869 DEBUG:boto:path=/pklxkd-55/b?versionId=null 2023-05-31T20:29:46.870 DEBUG:boto:auth_path=/pklxkd-55/b?versionId=null 2023-05-31T20:29:46.870 DEBUG:boto:Method: GET 2023-05-31T20:29:46.871 DEBUG:boto:Path: /pklxkd-55/b?versionId=null 2023-05-31T20:29:46.871 DEBUG:boto:Data: 2023-05-31T20:29:46.871 DEBUG:boto:Headers: {} 2023-05-31T20:29:46.872 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:29:46.872 DEBUG:boto:Port: 8000 2023-05-31T20:29:46.873 DEBUG:boto:Params: {} 2023-05-31T20:29:46.873 DEBUG:boto:Token: None 2023-05-31T20:29:46.873 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:29:46 GMT /pklxkd-55/b?versionId=null 2023-05-31T20:29:46.874 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:dohegYhb5mm2QkrqEQNUwnnKqu4= 2023-05-31T20:29:46.874 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:29:46 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:dohegYhb5mm2QkrqEQNUwnnKqu4=', 'Content-Length': '0'} 2023-05-31T20:29:46.876 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:27:16 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000548aac9ce66b99b2-006477ae3a-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:29:46 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:29:46.877 DEBUG:boto:path=/pklxkd-55/ 2023-05-31T20:29:46.877 DEBUG:boto:auth_path=/pklxkd-55/ 2023-05-31T20:29:46.878 DEBUG:boto:path=/pklxkd-55/?prefix=b 2023-05-31T20:29:46.878 DEBUG:boto:auth_path=/pklxkd-55/?prefix=b 2023-05-31T20:29:46.879 DEBUG:boto:Method: GET 2023-05-31T20:29:46.879 DEBUG:boto:Path: /pklxkd-55/?prefix=b 2023-05-31T20:29:46.880 DEBUG:boto:Data: 2023-05-31T20:29:46.880 DEBUG:boto:Headers: {} 2023-05-31T20:29:46.880 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:29:46.881 DEBUG:boto:Port: 8001 2023-05-31T20:29:46.881 DEBUG:boto:Params: {} 2023-05-31T20:29:46.882 DEBUG:boto:Token: None 2023-05-31T20:29:46.882 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:29:46 GMT /pklxkd-55/ 2023-05-31T20:29:46.883 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:XZ8tDAn1QJf1wBVwJgGN94Jp4hE= 2023-05-31T20:29:46.883 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:29:46 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:XZ8tDAn1QJf1wBVwJgGN94Jp4hE=', 'Content-Length': '0'} 2023-05-31T20:29:46.885 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000089edab617d68c7b4-006477ae3a-162e-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:29:46 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:29:46.886 DEBUG:boto:b'pklxkd-55b1000falseb2023-05-31T20:27:16.850Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:29:46.886 DEBUG:boto:path=/pklxkd-55/ 2023-05-31T20:29:46.887 DEBUG:boto:auth_path=/pklxkd-55/ 2023-05-31T20:29:46.887 DEBUG:boto:path=/pklxkd-55/?prefix=b 2023-05-31T20:29:46.888 DEBUG:boto:auth_path=/pklxkd-55/?prefix=b 2023-05-31T20:29:46.888 DEBUG:boto:Method: GET 2023-05-31T20:29:46.888 DEBUG:boto:Path: /pklxkd-55/?prefix=b 2023-05-31T20:29:46.889 DEBUG:boto:Data: 2023-05-31T20:29:46.889 DEBUG:boto:Headers: {} 2023-05-31T20:29:46.889 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:29:46.890 DEBUG:boto:Port: 8000 2023-05-31T20:29:46.890 DEBUG:boto:Params: {} 2023-05-31T20:29:46.891 DEBUG:boto:Token: None 2023-05-31T20:29:46.891 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:29:46 GMT /pklxkd-55/ 2023-05-31T20:29:46.892 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:XZ8tDAn1QJf1wBVwJgGN94Jp4hE= 2023-05-31T20:29:46.892 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:29:46 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:XZ8tDAn1QJf1wBVwJgGN94Jp4hE=', 'Content-Length': '0'} 2023-05-31T20:29:46.896 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000008085c4ec1a71ff97-006477ae3a-1630-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:29:46 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:29:46.897 DEBUG:boto:b'pklxkd-55b1000falseb2023-05-31T20:27:16.850Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:29:46.897 DEBUG:boto:path=/pklxkd-55/b 2023-05-31T20:29:46.898 DEBUG:boto:auth_path=/pklxkd-55/b 2023-05-31T20:29:46.898 DEBUG:boto:Method: HEAD 2023-05-31T20:29:46.898 DEBUG:boto:Path: /pklxkd-55/b 2023-05-31T20:29:46.899 DEBUG:boto:Data: 2023-05-31T20:29:46.899 DEBUG:boto:Headers: {} 2023-05-31T20:29:46.900 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:29:46.900 DEBUG:boto:Port: 8001 2023-05-31T20:29:46.900 DEBUG:boto:Params: {} 2023-05-31T20:29:46.901 DEBUG:boto:Token: None 2023-05-31T20:29:46.901 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:29:46 GMT /pklxkd-55/b 2023-05-31T20:29:46.902 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:n5uR90dPfv0f03vlYT/EOZ2QVAo= 2023-05-31T20:29:46.902 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:29:46 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:n5uR90dPfv0f03vlYT/EOZ2QVAo=', 'Content-Length': '0'} 2023-05-31T20:29:46.904 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:27:16 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000001dd3f9bc1813902-006477ae3a-162e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:29:46 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:29:46.905 DEBUG:boto:path=/pklxkd-55/b 2023-05-31T20:29:46.905 DEBUG:boto:auth_path=/pklxkd-55/b 2023-05-31T20:29:46.905 DEBUG:boto:Method: HEAD 2023-05-31T20:29:46.906 DEBUG:boto:Path: /pklxkd-55/b 2023-05-31T20:29:46.906 DEBUG:boto:Data: 2023-05-31T20:29:46.907 DEBUG:boto:Headers: {} 2023-05-31T20:29:46.907 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:29:46.907 DEBUG:boto:Port: 8000 2023-05-31T20:29:46.908 DEBUG:boto:Params: {} 2023-05-31T20:29:46.908 DEBUG:boto:Token: None 2023-05-31T20:29:46.908 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:29:46 GMT /pklxkd-55/b 2023-05-31T20:29:46.909 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:n5uR90dPfv0f03vlYT/EOZ2QVAo= 2023-05-31T20:29:46.909 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:29:46 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:n5uR90dPfv0f03vlYT/EOZ2QVAo=', 'Content-Length': '0'} 2023-05-31T20:29:46.911 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:27:16 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000f9608d2c7e2d7a1-006477ae3a-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:29:46 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:29:46.911 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=b 2023-05-31T20:29:46.912 DEBUG:boto:path=/pklxkd-55/b 2023-05-31T20:29:46.912 DEBUG:boto:auth_path=/pklxkd-55/b 2023-05-31T20:29:46.913 DEBUG:boto:Method: GET 2023-05-31T20:29:46.913 DEBUG:boto:Path: /pklxkd-55/b 2023-05-31T20:29:46.913 DEBUG:boto:Data: 2023-05-31T20:29:46.914 DEBUG:boto:Headers: {} 2023-05-31T20:29:46.914 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:29:46.915 DEBUG:boto:Port: 8001 2023-05-31T20:29:46.915 DEBUG:boto:Params: {} 2023-05-31T20:29:46.915 DEBUG:boto:Token: None 2023-05-31T20:29:46.916 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:29:46 GMT /pklxkd-55/b 2023-05-31T20:29:46.916 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:wuXKUJQ/mLdFY/inCZsMAtJ0qfU= 2023-05-31T20:29:46.917 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:29:46 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:wuXKUJQ/mLdFY/inCZsMAtJ0qfU=', 'Content-Length': '0'} 2023-05-31T20:29:46.918 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:27:16 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000843d8ac137cd31f9-006477ae3a-162e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:29:46 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:29:46.919 DEBUG:boto:path=/pklxkd-55/b 2023-05-31T20:29:46.919 DEBUG:boto:auth_path=/pklxkd-55/b 2023-05-31T20:29:46.920 DEBUG:boto:Method: GET 2023-05-31T20:29:46.920 DEBUG:boto:Path: /pklxkd-55/b 2023-05-31T20:29:46.921 DEBUG:boto:Data: 2023-05-31T20:29:46.921 DEBUG:boto:Headers: {} 2023-05-31T20:29:46.921 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:29:46.922 DEBUG:boto:Port: 8000 2023-05-31T20:29:46.922 DEBUG:boto:Params: {} 2023-05-31T20:29:46.922 DEBUG:boto:Token: None 2023-05-31T20:29:46.923 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:29:46 GMT /pklxkd-55/b 2023-05-31T20:29:46.924 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:wuXKUJQ/mLdFY/inCZsMAtJ0qfU= 2023-05-31T20:29:46.924 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:29:46 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:wuXKUJQ/mLdFY/inCZsMAtJ0qfU=', 'Content-Length': '0'} 2023-05-31T20:29:46.926 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:27:16 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009aeb102391373f35-006477ae3a-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:29:46 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:29:46.927 DEBUG:boto:path=/pklxkd-55/ 2023-05-31T20:29:46.927 DEBUG:boto:auth_path=/pklxkd-55/ 2023-05-31T20:29:46.927 DEBUG:boto:path=/pklxkd-55/?prefix=b 2023-05-31T20:29:46.928 DEBUG:boto:auth_path=/pklxkd-55/?prefix=b 2023-05-31T20:29:46.928 DEBUG:boto:Method: GET 2023-05-31T20:29:46.929 DEBUG:boto:Path: /pklxkd-55/?prefix=b 2023-05-31T20:29:46.929 DEBUG:boto:Data: 2023-05-31T20:29:46.929 DEBUG:boto:Headers: {} 2023-05-31T20:29:46.930 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:29:46.930 DEBUG:boto:Port: 8001 2023-05-31T20:29:46.930 DEBUG:boto:Params: {} 2023-05-31T20:29:46.931 DEBUG:boto:Token: None 2023-05-31T20:29:46.931 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:29:46 GMT /pklxkd-55/ 2023-05-31T20:29:46.932 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:XZ8tDAn1QJf1wBVwJgGN94Jp4hE= 2023-05-31T20:29:46.932 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:29:46 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:XZ8tDAn1QJf1wBVwJgGN94Jp4hE=', 'Content-Length': '0'} 2023-05-31T20:29:46.935 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000094264aa1eb44dc55-006477ae3a-162e-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:29:46 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:29:46.935 DEBUG:boto:b'pklxkd-55b1000falseb2023-05-31T20:27:16.850Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:29:46.936 DEBUG:boto:path=/pklxkd-55/ 2023-05-31T20:29:46.936 DEBUG:boto:auth_path=/pklxkd-55/ 2023-05-31T20:29:46.937 DEBUG:boto:path=/pklxkd-55/?prefix=b 2023-05-31T20:29:46.937 DEBUG:boto:auth_path=/pklxkd-55/?prefix=b 2023-05-31T20:29:46.937 DEBUG:boto:Method: GET 2023-05-31T20:29:46.938 DEBUG:boto:Path: /pklxkd-55/?prefix=b 2023-05-31T20:29:46.938 DEBUG:boto:Data: 2023-05-31T20:29:46.939 DEBUG:boto:Headers: {} 2023-05-31T20:29:46.939 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:29:46.939 DEBUG:boto:Port: 8000 2023-05-31T20:29:46.940 DEBUG:boto:Params: {} 2023-05-31T20:29:46.940 DEBUG:boto:Token: None 2023-05-31T20:29:46.941 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:29:46 GMT /pklxkd-55/ 2023-05-31T20:29:46.941 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:XZ8tDAn1QJf1wBVwJgGN94Jp4hE= 2023-05-31T20:29:46.942 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:29:46 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:XZ8tDAn1QJf1wBVwJgGN94Jp4hE=', 'Content-Length': '0'} 2023-05-31T20:29:46.943 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000313a9ec53a3169bc-006477ae3a-1630-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:29:46 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:29:46.944 DEBUG:boto:b'pklxkd-55b1000falseb2023-05-31T20:27:16.850Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:29:46.945 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=c 2023-05-31T20:29:46.945 DEBUG:boto:path=/pklxkd-55/c 2023-05-31T20:29:46.946 DEBUG:boto:auth_path=/pklxkd-55/c 2023-05-31T20:29:46.946 DEBUG:boto:path=/pklxkd-55/c?versionId=null 2023-05-31T20:29:46.946 DEBUG:boto:auth_path=/pklxkd-55/c?versionId=null 2023-05-31T20:29:46.947 DEBUG:boto:Method: GET 2023-05-31T20:29:46.947 DEBUG:boto:Path: /pklxkd-55/c?versionId=null 2023-05-31T20:29:46.947 DEBUG:boto:Data: 2023-05-31T20:29:46.948 DEBUG:boto:Headers: {} 2023-05-31T20:29:46.948 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:29:46.949 DEBUG:boto:Port: 8001 2023-05-31T20:29:46.949 DEBUG:boto:Params: {} 2023-05-31T20:29:46.949 DEBUG:boto:Token: None 2023-05-31T20:29:46.950 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:29:46 GMT /pklxkd-55/c?versionId=null 2023-05-31T20:29:46.950 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:V8bcEaCXTJmgXE/9Lo6ni/7nA3s= 2023-05-31T20:29:46.951 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:29:46 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:V8bcEaCXTJmgXE/9Lo6ni/7nA3s=', 'Content-Length': '0'} 2023-05-31T20:29:46.952 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:27:16 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e70e8edbbd535d16-006477ae3a-162e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:29:46 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:29:46.953 DEBUG:boto:path=/pklxkd-55/c 2023-05-31T20:29:46.954 DEBUG:boto:auth_path=/pklxkd-55/c 2023-05-31T20:29:46.954 DEBUG:boto:path=/pklxkd-55/c?versionId=null 2023-05-31T20:29:46.954 DEBUG:boto:auth_path=/pklxkd-55/c?versionId=null 2023-05-31T20:29:46.955 DEBUG:boto:Method: GET 2023-05-31T20:29:46.955 DEBUG:boto:Path: /pklxkd-55/c?versionId=null 2023-05-31T20:29:46.956 DEBUG:boto:Data: 2023-05-31T20:29:46.956 DEBUG:boto:Headers: {} 2023-05-31T20:29:46.956 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:29:46.957 DEBUG:boto:Port: 8000 2023-05-31T20:29:46.957 DEBUG:boto:Params: {} 2023-05-31T20:29:46.958 DEBUG:boto:Token: None 2023-05-31T20:29:46.958 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:29:46 GMT /pklxkd-55/c?versionId=null 2023-05-31T20:29:46.959 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:V8bcEaCXTJmgXE/9Lo6ni/7nA3s= 2023-05-31T20:29:46.959 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:29:46 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:V8bcEaCXTJmgXE/9Lo6ni/7nA3s=', 'Content-Length': '0'} 2023-05-31T20:29:46.961 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:27:16 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000adfc976e8c841118-006477ae3a-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:29:46 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:29:46.961 DEBUG:boto:path=/pklxkd-55/c 2023-05-31T20:29:46.962 DEBUG:boto:auth_path=/pklxkd-55/c 2023-05-31T20:29:46.962 DEBUG:boto:path=/pklxkd-55/c?versionId=null 2023-05-31T20:29:46.962 DEBUG:boto:auth_path=/pklxkd-55/c?versionId=null 2023-05-31T20:29:46.963 DEBUG:boto:Method: HEAD 2023-05-31T20:29:46.963 DEBUG:boto:Path: /pklxkd-55/c?versionId=null 2023-05-31T20:29:46.963 DEBUG:boto:Data: 2023-05-31T20:29:46.964 DEBUG:boto:Headers: {} 2023-05-31T20:29:46.964 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:29:46.965 DEBUG:boto:Port: 8001 2023-05-31T20:29:46.965 DEBUG:boto:Params: {} 2023-05-31T20:29:46.965 DEBUG:boto:Token: None 2023-05-31T20:29:46.966 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:29:46 GMT /pklxkd-55/c?versionId=null 2023-05-31T20:29:46.966 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:AqMFlyYVuhYdkuTQgQwv/WdMYzM= 2023-05-31T20:29:46.967 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:29:46 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:AqMFlyYVuhYdkuTQgQwv/WdMYzM=', 'Content-Length': '0'} 2023-05-31T20:29:46.968 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:27:16 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f38eddbe82c86087-006477ae3a-162e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:29:46 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:29:46.969 DEBUG:boto:path=/pklxkd-55/c 2023-05-31T20:29:46.969 DEBUG:boto:auth_path=/pklxkd-55/c 2023-05-31T20:29:46.970 DEBUG:boto:path=/pklxkd-55/c?versionId=null 2023-05-31T20:29:46.970 DEBUG:boto:auth_path=/pklxkd-55/c?versionId=null 2023-05-31T20:29:46.970 DEBUG:boto:Method: HEAD 2023-05-31T20:29:46.971 DEBUG:boto:Path: /pklxkd-55/c?versionId=null 2023-05-31T20:29:46.971 DEBUG:boto:Data: 2023-05-31T20:29:46.971 DEBUG:boto:Headers: {} 2023-05-31T20:29:46.972 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:29:46.972 DEBUG:boto:Port: 8000 2023-05-31T20:29:46.972 DEBUG:boto:Params: {} 2023-05-31T20:29:46.973 DEBUG:boto:Token: None 2023-05-31T20:29:46.973 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:29:46 GMT /pklxkd-55/c?versionId=null 2023-05-31T20:29:46.974 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:AqMFlyYVuhYdkuTQgQwv/WdMYzM= 2023-05-31T20:29:46.974 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:29:46 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:AqMFlyYVuhYdkuTQgQwv/WdMYzM=', 'Content-Length': '0'} 2023-05-31T20:29:46.976 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:27:16 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007e437de488f05cf4-006477ae3a-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:29:46 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:29:46.976 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=c 2023-05-31T20:29:46.977 DEBUG:boto:path=/pklxkd-55/c 2023-05-31T20:29:46.977 DEBUG:boto:auth_path=/pklxkd-55/c 2023-05-31T20:29:46.978 DEBUG:boto:path=/pklxkd-55/c?versionId=null 2023-05-31T20:29:46.978 DEBUG:boto:auth_path=/pklxkd-55/c?versionId=null 2023-05-31T20:29:46.978 DEBUG:boto:Method: GET 2023-05-31T20:29:46.979 DEBUG:boto:Path: /pklxkd-55/c?versionId=null 2023-05-31T20:29:46.979 DEBUG:boto:Data: 2023-05-31T20:29:46.979 DEBUG:boto:Headers: {} 2023-05-31T20:29:46.980 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:29:46.980 DEBUG:boto:Port: 8001 2023-05-31T20:29:46.981 DEBUG:boto:Params: {} 2023-05-31T20:29:46.981 DEBUG:boto:Token: None 2023-05-31T20:29:46.981 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:29:46 GMT /pklxkd-55/c?versionId=null 2023-05-31T20:29:46.982 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:V8bcEaCXTJmgXE/9Lo6ni/7nA3s= 2023-05-31T20:29:46.982 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:29:46 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:V8bcEaCXTJmgXE/9Lo6ni/7nA3s=', 'Content-Length': '0'} 2023-05-31T20:29:46.984 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:27:16 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000071f9186ae9818d14-006477ae3a-162e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:29:46 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:29:47.028 DEBUG:boto:path=/pklxkd-55/c 2023-05-31T20:29:47.029 DEBUG:boto:auth_path=/pklxkd-55/c 2023-05-31T20:29:47.029 DEBUG:boto:path=/pklxkd-55/c?versionId=null 2023-05-31T20:29:47.030 DEBUG:boto:auth_path=/pklxkd-55/c?versionId=null 2023-05-31T20:29:47.030 DEBUG:boto:Method: GET 2023-05-31T20:29:47.030 DEBUG:boto:Path: /pklxkd-55/c?versionId=null 2023-05-31T20:29:47.031 DEBUG:boto:Data: 2023-05-31T20:29:47.031 DEBUG:boto:Headers: {} 2023-05-31T20:29:47.032 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:29:47.032 DEBUG:boto:Port: 8000 2023-05-31T20:29:47.033 DEBUG:boto:Params: {} 2023-05-31T20:29:47.033 DEBUG:boto:Token: None 2023-05-31T20:29:47.034 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:29:47 GMT /pklxkd-55/c?versionId=null 2023-05-31T20:29:47.034 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:P1aXLMUH6zOhTLbQHnvGsASl6uk= 2023-05-31T20:29:47.035 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:29:47 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:P1aXLMUH6zOhTLbQHnvGsASl6uk=', 'Content-Length': '0'} 2023-05-31T20:29:47.036 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:27:16 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006ba618b21ca7f01d-006477ae3b-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:29:47 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:29:47.037 DEBUG:boto:path=/pklxkd-55/ 2023-05-31T20:29:47.037 DEBUG:boto:auth_path=/pklxkd-55/ 2023-05-31T20:29:47.038 DEBUG:boto:path=/pklxkd-55/?prefix=c 2023-05-31T20:29:47.038 DEBUG:boto:auth_path=/pklxkd-55/?prefix=c 2023-05-31T20:29:47.038 DEBUG:boto:Method: GET 2023-05-31T20:29:47.039 DEBUG:boto:Path: /pklxkd-55/?prefix=c 2023-05-31T20:29:47.039 DEBUG:boto:Data: 2023-05-31T20:29:47.040 DEBUG:boto:Headers: {} 2023-05-31T20:29:47.040 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:29:47.040 DEBUG:boto:Port: 8001 2023-05-31T20:29:47.041 DEBUG:boto:Params: {} 2023-05-31T20:29:47.041 DEBUG:boto:Token: None 2023-05-31T20:29:47.042 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:29:47 GMT /pklxkd-55/ 2023-05-31T20:29:47.042 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:IBdSyRVlh8qg/aH9XxJ5KZfq+k4= 2023-05-31T20:29:47.043 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:29:47 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:IBdSyRVlh8qg/aH9XxJ5KZfq+k4=', 'Content-Length': '0'} 2023-05-31T20:29:47.045 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000001921d21d924c3074-006477ae3b-162e-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:29:47 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:29:47.045 DEBUG:boto:b'pklxkd-55c1000falsec2023-05-31T20:27:16.869Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:29:47.046 DEBUG:boto:path=/pklxkd-55/ 2023-05-31T20:29:47.046 DEBUG:boto:auth_path=/pklxkd-55/ 2023-05-31T20:29:47.047 DEBUG:boto:path=/pklxkd-55/?prefix=c 2023-05-31T20:29:47.047 DEBUG:boto:auth_path=/pklxkd-55/?prefix=c 2023-05-31T20:29:47.048 DEBUG:boto:Method: GET 2023-05-31T20:29:47.048 DEBUG:boto:Path: /pklxkd-55/?prefix=c 2023-05-31T20:29:47.048 DEBUG:boto:Data: 2023-05-31T20:29:47.049 DEBUG:boto:Headers: {} 2023-05-31T20:29:47.049 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:29:47.050 DEBUG:boto:Port: 8000 2023-05-31T20:29:47.050 DEBUG:boto:Params: {} 2023-05-31T20:29:47.050 DEBUG:boto:Token: None 2023-05-31T20:29:47.051 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:29:47 GMT /pklxkd-55/ 2023-05-31T20:29:47.051 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:IBdSyRVlh8qg/aH9XxJ5KZfq+k4= 2023-05-31T20:29:47.052 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:29:47 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:IBdSyRVlh8qg/aH9XxJ5KZfq+k4=', 'Content-Length': '0'} 2023-05-31T20:29:47.053 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c3c76626a0ec93ef-006477ae3b-1630-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:29:47 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:29:47.054 DEBUG:boto:b'pklxkd-55c1000falsec2023-05-31T20:27:16.869Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:29:47.055 DEBUG:boto:path=/pklxkd-55/c 2023-05-31T20:29:47.055 DEBUG:boto:auth_path=/pklxkd-55/c 2023-05-31T20:29:47.056 DEBUG:boto:Method: HEAD 2023-05-31T20:29:47.056 DEBUG:boto:Path: /pklxkd-55/c 2023-05-31T20:29:47.056 DEBUG:boto:Data: 2023-05-31T20:29:47.057 DEBUG:boto:Headers: {} 2023-05-31T20:29:47.057 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:29:47.058 DEBUG:boto:Port: 8001 2023-05-31T20:29:47.058 DEBUG:boto:Params: {} 2023-05-31T20:29:47.059 DEBUG:boto:Token: None 2023-05-31T20:29:47.059 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:29:47 GMT /pklxkd-55/c 2023-05-31T20:29:47.059 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:vjS8H3RSqPpmqDEM9WRL9MuHsJQ= 2023-05-31T20:29:47.060 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:29:47 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:vjS8H3RSqPpmqDEM9WRL9MuHsJQ=', 'Content-Length': '0'} 2023-05-31T20:29:47.062 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:27:16 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000849f06dc367ce32b-006477ae3b-162e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:29:47 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:29:47.062 DEBUG:boto:path=/pklxkd-55/c 2023-05-31T20:29:47.063 DEBUG:boto:auth_path=/pklxkd-55/c 2023-05-31T20:29:47.063 DEBUG:boto:Method: HEAD 2023-05-31T20:29:47.063 DEBUG:boto:Path: /pklxkd-55/c 2023-05-31T20:29:47.064 DEBUG:boto:Data: 2023-05-31T20:29:47.064 DEBUG:boto:Headers: {} 2023-05-31T20:29:47.065 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:29:47.065 DEBUG:boto:Port: 8000 2023-05-31T20:29:47.065 DEBUG:boto:Params: {} 2023-05-31T20:29:47.066 DEBUG:boto:Token: None 2023-05-31T20:29:47.066 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:29:47 GMT /pklxkd-55/c 2023-05-31T20:29:47.067 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:vjS8H3RSqPpmqDEM9WRL9MuHsJQ= 2023-05-31T20:29:47.067 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:29:47 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:vjS8H3RSqPpmqDEM9WRL9MuHsJQ=', 'Content-Length': '0'} 2023-05-31T20:29:47.070 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:27:16 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f939ed76af465b2d-006477ae3b-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:29:47 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:29:47.071 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=c 2023-05-31T20:29:47.071 DEBUG:boto:path=/pklxkd-55/c 2023-05-31T20:29:47.071 DEBUG:boto:auth_path=/pklxkd-55/c 2023-05-31T20:29:47.072 DEBUG:boto:Method: GET 2023-05-31T20:29:47.072 DEBUG:boto:Path: /pklxkd-55/c 2023-05-31T20:29:47.073 DEBUG:boto:Data: 2023-05-31T20:29:47.073 DEBUG:boto:Headers: {} 2023-05-31T20:29:47.073 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:29:47.074 DEBUG:boto:Port: 8001 2023-05-31T20:29:47.074 DEBUG:boto:Params: {} 2023-05-31T20:29:47.074 DEBUG:boto:Token: None 2023-05-31T20:29:47.075 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:29:47 GMT /pklxkd-55/c 2023-05-31T20:29:47.075 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:KCsebAKjHoJs41UM5GosrCgfQCs= 2023-05-31T20:29:47.076 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:29:47 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:KCsebAKjHoJs41UM5GosrCgfQCs=', 'Content-Length': '0'} 2023-05-31T20:29:47.078 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:27:16 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000019df14a7220e0b1b-006477ae3b-162e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:29:47 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:29:47.079 DEBUG:boto:path=/pklxkd-55/c 2023-05-31T20:29:47.079 DEBUG:boto:auth_path=/pklxkd-55/c 2023-05-31T20:29:47.080 DEBUG:boto:Method: GET 2023-05-31T20:29:47.080 DEBUG:boto:Path: /pklxkd-55/c 2023-05-31T20:29:47.080 DEBUG:boto:Data: 2023-05-31T20:29:47.081 DEBUG:boto:Headers: {} 2023-05-31T20:29:47.081 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:29:47.081 DEBUG:boto:Port: 8000 2023-05-31T20:29:47.082 DEBUG:boto:Params: {} 2023-05-31T20:29:47.082 DEBUG:boto:Token: None 2023-05-31T20:29:47.083 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:29:47 GMT /pklxkd-55/c 2023-05-31T20:29:47.083 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:KCsebAKjHoJs41UM5GosrCgfQCs= 2023-05-31T20:29:47.084 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:29:47 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:KCsebAKjHoJs41UM5GosrCgfQCs=', 'Content-Length': '0'} 2023-05-31T20:29:47.085 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:27:16 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000008809774c2acaf8b-006477ae3b-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:29:47 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:29:47.086 DEBUG:boto:path=/pklxkd-55/ 2023-05-31T20:29:47.086 DEBUG:boto:auth_path=/pklxkd-55/ 2023-05-31T20:29:47.087 DEBUG:boto:path=/pklxkd-55/?prefix=c 2023-05-31T20:29:47.087 DEBUG:boto:auth_path=/pklxkd-55/?prefix=c 2023-05-31T20:29:47.088 DEBUG:boto:Method: GET 2023-05-31T20:29:47.088 DEBUG:boto:Path: /pklxkd-55/?prefix=c 2023-05-31T20:29:47.088 DEBUG:boto:Data: 2023-05-31T20:29:47.089 DEBUG:boto:Headers: {} 2023-05-31T20:29:47.089 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:29:47.089 DEBUG:boto:Port: 8001 2023-05-31T20:29:47.090 DEBUG:boto:Params: {} 2023-05-31T20:29:47.090 DEBUG:boto:Token: None 2023-05-31T20:29:47.091 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:29:47 GMT /pklxkd-55/ 2023-05-31T20:29:47.091 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:IBdSyRVlh8qg/aH9XxJ5KZfq+k4= 2023-05-31T20:29:47.092 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:29:47 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:IBdSyRVlh8qg/aH9XxJ5KZfq+k4=', 'Content-Length': '0'} 2023-05-31T20:29:47.094 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000debb8a925a227acd-006477ae3b-162e-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:29:47 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:29:47.094 DEBUG:boto:b'pklxkd-55c1000falsec2023-05-31T20:27:16.869Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:29:47.095 DEBUG:boto:path=/pklxkd-55/ 2023-05-31T20:29:47.095 DEBUG:boto:auth_path=/pklxkd-55/ 2023-05-31T20:29:47.096 DEBUG:boto:path=/pklxkd-55/?prefix=c 2023-05-31T20:29:47.096 DEBUG:boto:auth_path=/pklxkd-55/?prefix=c 2023-05-31T20:29:47.096 DEBUG:boto:Method: GET 2023-05-31T20:29:47.097 DEBUG:boto:Path: /pklxkd-55/?prefix=c 2023-05-31T20:29:47.097 DEBUG:boto:Data: 2023-05-31T20:29:47.097 DEBUG:boto:Headers: {} 2023-05-31T20:29:47.098 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:29:47.098 DEBUG:boto:Port: 8000 2023-05-31T20:29:47.098 DEBUG:boto:Params: {} 2023-05-31T20:29:47.099 DEBUG:boto:Token: None 2023-05-31T20:29:47.099 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:29:47 GMT /pklxkd-55/ 2023-05-31T20:29:47.100 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:IBdSyRVlh8qg/aH9XxJ5KZfq+k4= 2023-05-31T20:29:47.100 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:29:47 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:IBdSyRVlh8qg/aH9XxJ5KZfq+k4=', 'Content-Length': '0'} 2023-05-31T20:29:47.102 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a1f4a69a81fbeb79-006477ae3b-1630-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:29:47 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:29:47.103 DEBUG:boto:b'pklxkd-55c1000falsec2023-05-31T20:27:16.869Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:29:47.103 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=d 2023-05-31T20:29:47.104 DEBUG:boto:path=/pklxkd-55/d 2023-05-31T20:29:47.104 DEBUG:boto:auth_path=/pklxkd-55/d 2023-05-31T20:29:47.105 DEBUG:boto:path=/pklxkd-55/d?versionId=null 2023-05-31T20:29:47.105 DEBUG:boto:auth_path=/pklxkd-55/d?versionId=null 2023-05-31T20:29:47.106 DEBUG:boto:Method: GET 2023-05-31T20:29:47.106 DEBUG:boto:Path: /pklxkd-55/d?versionId=null 2023-05-31T20:29:47.106 DEBUG:boto:Data: 2023-05-31T20:29:47.107 DEBUG:boto:Headers: {} 2023-05-31T20:29:47.107 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:29:47.107 DEBUG:boto:Port: 8001 2023-05-31T20:29:47.108 DEBUG:boto:Params: {} 2023-05-31T20:29:47.108 DEBUG:boto:Token: None 2023-05-31T20:29:47.109 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:29:47 GMT /pklxkd-55/d?versionId=null 2023-05-31T20:29:47.109 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:2iEJWxnjGeNLda7D6Nei7TwGGLE= 2023-05-31T20:29:47.110 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:29:47 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:2iEJWxnjGeNLda7D6Nei7TwGGLE=', 'Content-Length': '0'} 2023-05-31T20:29:47.111 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:27:16 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000091d5d692949f5efa-006477ae3b-162e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:29:47 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:29:47.112 DEBUG:boto:path=/pklxkd-55/d 2023-05-31T20:29:47.112 DEBUG:boto:auth_path=/pklxkd-55/d 2023-05-31T20:29:47.113 DEBUG:boto:path=/pklxkd-55/d?versionId=null 2023-05-31T20:29:47.113 DEBUG:boto:auth_path=/pklxkd-55/d?versionId=null 2023-05-31T20:29:47.113 DEBUG:boto:Method: GET 2023-05-31T20:29:47.114 DEBUG:boto:Path: /pklxkd-55/d?versionId=null 2023-05-31T20:29:47.114 DEBUG:boto:Data: 2023-05-31T20:29:47.114 DEBUG:boto:Headers: {} 2023-05-31T20:29:47.115 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:29:47.115 DEBUG:boto:Port: 8000 2023-05-31T20:29:47.116 DEBUG:boto:Params: {} 2023-05-31T20:29:47.116 DEBUG:boto:Token: None 2023-05-31T20:29:47.116 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:29:47 GMT /pklxkd-55/d?versionId=null 2023-05-31T20:29:47.117 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:2iEJWxnjGeNLda7D6Nei7TwGGLE= 2023-05-31T20:29:47.117 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:29:47 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:2iEJWxnjGeNLda7D6Nei7TwGGLE=', 'Content-Length': '0'} 2023-05-31T20:29:47.119 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:27:16 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000bc20baa0a0f923b3-006477ae3b-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:29:47 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:29:47.120 DEBUG:boto:path=/pklxkd-55/d 2023-05-31T20:29:47.120 DEBUG:boto:auth_path=/pklxkd-55/d 2023-05-31T20:29:47.120 DEBUG:boto:path=/pklxkd-55/d?versionId=null 2023-05-31T20:29:47.121 DEBUG:boto:auth_path=/pklxkd-55/d?versionId=null 2023-05-31T20:29:47.121 DEBUG:boto:Method: HEAD 2023-05-31T20:29:47.122 DEBUG:boto:Path: /pklxkd-55/d?versionId=null 2023-05-31T20:29:47.122 DEBUG:boto:Data: 2023-05-31T20:29:47.122 DEBUG:boto:Headers: {} 2023-05-31T20:29:47.123 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:29:47.123 DEBUG:boto:Port: 8001 2023-05-31T20:29:47.123 DEBUG:boto:Params: {} 2023-05-31T20:29:47.124 DEBUG:boto:Token: None 2023-05-31T20:29:47.124 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:29:47 GMT /pklxkd-55/d?versionId=null 2023-05-31T20:29:47.125 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:harZR6uuoONBLYwl9WLoBi2jBDY= 2023-05-31T20:29:47.125 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:29:47 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:harZR6uuoONBLYwl9WLoBi2jBDY=', 'Content-Length': '0'} 2023-05-31T20:29:47.127 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:27:16 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004d43da428e4e7492-006477ae3b-162e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:29:47 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:29:47.127 DEBUG:boto:path=/pklxkd-55/d 2023-05-31T20:29:47.128 DEBUG:boto:auth_path=/pklxkd-55/d 2023-05-31T20:29:47.128 DEBUG:boto:path=/pklxkd-55/d?versionId=null 2023-05-31T20:29:47.128 DEBUG:boto:auth_path=/pklxkd-55/d?versionId=null 2023-05-31T20:29:47.129 DEBUG:boto:Method: HEAD 2023-05-31T20:29:47.129 DEBUG:boto:Path: /pklxkd-55/d?versionId=null 2023-05-31T20:29:47.130 DEBUG:boto:Data: 2023-05-31T20:29:47.130 DEBUG:boto:Headers: {} 2023-05-31T20:29:47.130 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:29:47.131 DEBUG:boto:Port: 8000 2023-05-31T20:29:47.131 DEBUG:boto:Params: {} 2023-05-31T20:29:47.131 DEBUG:boto:Token: None 2023-05-31T20:29:47.132 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:29:47 GMT /pklxkd-55/d?versionId=null 2023-05-31T20:29:47.132 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:harZR6uuoONBLYwl9WLoBi2jBDY= 2023-05-31T20:29:47.133 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:29:47 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:harZR6uuoONBLYwl9WLoBi2jBDY=', 'Content-Length': '0'} 2023-05-31T20:29:47.134 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:27:16 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000420c48f5e0e401ce-006477ae3b-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:29:47 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:29:47.135 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=d 2023-05-31T20:29:47.135 DEBUG:boto:path=/pklxkd-55/d 2023-05-31T20:29:47.136 DEBUG:boto:auth_path=/pklxkd-55/d 2023-05-31T20:29:47.136 DEBUG:boto:path=/pklxkd-55/d?versionId=null 2023-05-31T20:29:47.136 DEBUG:boto:auth_path=/pklxkd-55/d?versionId=null 2023-05-31T20:29:47.137 DEBUG:boto:Method: GET 2023-05-31T20:29:47.137 DEBUG:boto:Path: /pklxkd-55/d?versionId=null 2023-05-31T20:29:47.138 DEBUG:boto:Data: 2023-05-31T20:29:47.138 DEBUG:boto:Headers: {} 2023-05-31T20:29:47.138 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:29:47.139 DEBUG:boto:Port: 8001 2023-05-31T20:29:47.139 DEBUG:boto:Params: {} 2023-05-31T20:29:47.139 DEBUG:boto:Token: None 2023-05-31T20:29:47.140 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:29:47 GMT /pklxkd-55/d?versionId=null 2023-05-31T20:29:47.140 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:2iEJWxnjGeNLda7D6Nei7TwGGLE= 2023-05-31T20:29:47.141 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:29:47 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:2iEJWxnjGeNLda7D6Nei7TwGGLE=', 'Content-Length': '0'} 2023-05-31T20:29:47.142 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:27:16 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ed00b7dce04f021f-006477ae3b-162e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:29:47 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:29:47.143 DEBUG:boto:path=/pklxkd-55/d 2023-05-31T20:29:47.143 DEBUG:boto:auth_path=/pklxkd-55/d 2023-05-31T20:29:47.144 DEBUG:boto:path=/pklxkd-55/d?versionId=null 2023-05-31T20:29:47.144 DEBUG:boto:auth_path=/pklxkd-55/d?versionId=null 2023-05-31T20:29:47.144 DEBUG:boto:Method: GET 2023-05-31T20:29:47.145 DEBUG:boto:Path: /pklxkd-55/d?versionId=null 2023-05-31T20:29:47.145 DEBUG:boto:Data: 2023-05-31T20:29:47.146 DEBUG:boto:Headers: {} 2023-05-31T20:29:47.146 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:29:47.146 DEBUG:boto:Port: 8000 2023-05-31T20:29:47.147 DEBUG:boto:Params: {} 2023-05-31T20:29:47.147 DEBUG:boto:Token: None 2023-05-31T20:29:47.148 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:29:47 GMT /pklxkd-55/d?versionId=null 2023-05-31T20:29:47.148 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:2iEJWxnjGeNLda7D6Nei7TwGGLE= 2023-05-31T20:29:47.149 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:29:47 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:2iEJWxnjGeNLda7D6Nei7TwGGLE=', 'Content-Length': '0'} 2023-05-31T20:29:47.150 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:27:16 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009b6945514ae92903-006477ae3b-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:29:47 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:29:47.151 DEBUG:boto:path=/pklxkd-55/ 2023-05-31T20:29:47.151 DEBUG:boto:auth_path=/pklxkd-55/ 2023-05-31T20:29:47.152 DEBUG:boto:path=/pklxkd-55/?prefix=d 2023-05-31T20:29:47.152 DEBUG:boto:auth_path=/pklxkd-55/?prefix=d 2023-05-31T20:29:47.153 DEBUG:boto:Method: GET 2023-05-31T20:29:47.153 DEBUG:boto:Path: /pklxkd-55/?prefix=d 2023-05-31T20:29:47.153 DEBUG:boto:Data: 2023-05-31T20:29:47.154 DEBUG:boto:Headers: {} 2023-05-31T20:29:47.154 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:29:47.155 DEBUG:boto:Port: 8001 2023-05-31T20:29:47.155 DEBUG:boto:Params: {} 2023-05-31T20:29:47.155 DEBUG:boto:Token: None 2023-05-31T20:29:47.156 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:29:47 GMT /pklxkd-55/ 2023-05-31T20:29:47.156 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:IBdSyRVlh8qg/aH9XxJ5KZfq+k4= 2023-05-31T20:29:47.157 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:29:47 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:IBdSyRVlh8qg/aH9XxJ5KZfq+k4=', 'Content-Length': '0'} 2023-05-31T20:29:47.159 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000009e5b2d166e730dc-006477ae3b-162e-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:29:47 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:29:47.159 DEBUG:boto:b'pklxkd-55d1000falsed2023-05-31T20:27:16.893Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:29:47.160 DEBUG:boto:path=/pklxkd-55/ 2023-05-31T20:29:47.161 DEBUG:boto:auth_path=/pklxkd-55/ 2023-05-31T20:29:47.161 DEBUG:boto:path=/pklxkd-55/?prefix=d 2023-05-31T20:29:47.161 DEBUG:boto:auth_path=/pklxkd-55/?prefix=d 2023-05-31T20:29:47.162 DEBUG:boto:Method: GET 2023-05-31T20:29:47.162 DEBUG:boto:Path: /pklxkd-55/?prefix=d 2023-05-31T20:29:47.162 DEBUG:boto:Data: 2023-05-31T20:29:47.163 DEBUG:boto:Headers: {} 2023-05-31T20:29:47.163 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:29:47.164 DEBUG:boto:Port: 8000 2023-05-31T20:29:47.164 DEBUG:boto:Params: {} 2023-05-31T20:29:47.164 DEBUG:boto:Token: None 2023-05-31T20:29:47.165 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:29:47 GMT /pklxkd-55/ 2023-05-31T20:29:47.165 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:IBdSyRVlh8qg/aH9XxJ5KZfq+k4= 2023-05-31T20:29:47.166 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:29:47 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:IBdSyRVlh8qg/aH9XxJ5KZfq+k4=', 'Content-Length': '0'} 2023-05-31T20:29:47.167 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000003ae87ca4fefd366a-006477ae3b-1630-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:29:47 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:29:47.168 DEBUG:boto:b'pklxkd-55d1000falsed2023-05-31T20:27:16.893Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:29:47.169 DEBUG:boto:path=/pklxkd-55/d 2023-05-31T20:29:47.169 DEBUG:boto:auth_path=/pklxkd-55/d 2023-05-31T20:29:47.170 DEBUG:boto:Method: HEAD 2023-05-31T20:29:47.170 DEBUG:boto:Path: /pklxkd-55/d 2023-05-31T20:29:47.170 DEBUG:boto:Data: 2023-05-31T20:29:47.171 DEBUG:boto:Headers: {} 2023-05-31T20:29:47.171 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:29:47.171 DEBUG:boto:Port: 8001 2023-05-31T20:29:47.172 DEBUG:boto:Params: {} 2023-05-31T20:29:47.172 DEBUG:boto:Token: None 2023-05-31T20:29:47.173 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:29:47 GMT /pklxkd-55/d 2023-05-31T20:29:47.173 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:X+9cyPTt250zU6XVmvtAhT2OEYY= 2023-05-31T20:29:47.174 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:29:47 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:X+9cyPTt250zU6XVmvtAhT2OEYY=', 'Content-Length': '0'} 2023-05-31T20:29:47.175 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:27:16 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000083015d841141e613-006477ae3b-162e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:29:47 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:29:47.176 DEBUG:boto:path=/pklxkd-55/d 2023-05-31T20:29:47.177 DEBUG:boto:auth_path=/pklxkd-55/d 2023-05-31T20:29:47.177 DEBUG:boto:Method: HEAD 2023-05-31T20:29:47.177 DEBUG:boto:Path: /pklxkd-55/d 2023-05-31T20:29:47.178 DEBUG:boto:Data: 2023-05-31T20:29:47.178 DEBUG:boto:Headers: {} 2023-05-31T20:29:47.179 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:29:47.179 DEBUG:boto:Port: 8000 2023-05-31T20:29:47.179 DEBUG:boto:Params: {} 2023-05-31T20:29:47.180 DEBUG:boto:Token: None 2023-05-31T20:29:47.180 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:29:47 GMT /pklxkd-55/d 2023-05-31T20:29:47.180 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:X+9cyPTt250zU6XVmvtAhT2OEYY= 2023-05-31T20:29:47.181 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:29:47 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:X+9cyPTt250zU6XVmvtAhT2OEYY=', 'Content-Length': '0'} 2023-05-31T20:29:47.182 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:27:16 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000700134a3fe4ade89-006477ae3b-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:29:47 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:29:47.183 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=d 2023-05-31T20:29:47.183 DEBUG:boto:path=/pklxkd-55/d 2023-05-31T20:29:47.184 DEBUG:boto:auth_path=/pklxkd-55/d 2023-05-31T20:29:47.184 DEBUG:boto:Method: GET 2023-05-31T20:29:47.185 DEBUG:boto:Path: /pklxkd-55/d 2023-05-31T20:29:47.185 DEBUG:boto:Data: 2023-05-31T20:29:47.185 DEBUG:boto:Headers: {} 2023-05-31T20:29:47.186 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:29:47.186 DEBUG:boto:Port: 8001 2023-05-31T20:29:47.186 DEBUG:boto:Params: {} 2023-05-31T20:29:47.187 DEBUG:boto:Token: None 2023-05-31T20:29:47.187 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:29:47 GMT /pklxkd-55/d 2023-05-31T20:29:47.188 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:7NERFesywT7Ehxd1WNnCZkSPCw8= 2023-05-31T20:29:47.188 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:29:47 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:7NERFesywT7Ehxd1WNnCZkSPCw8=', 'Content-Length': '0'} 2023-05-31T20:29:47.190 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:27:16 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000013e2afc8bf7b0126-006477ae3b-162e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:29:47 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:29:47.190 DEBUG:boto:path=/pklxkd-55/d 2023-05-31T20:29:47.191 DEBUG:boto:auth_path=/pklxkd-55/d 2023-05-31T20:29:47.191 DEBUG:boto:Method: GET 2023-05-31T20:29:47.191 DEBUG:boto:Path: /pklxkd-55/d 2023-05-31T20:29:47.192 DEBUG:boto:Data: 2023-05-31T20:29:47.192 DEBUG:boto:Headers: {} 2023-05-31T20:29:47.193 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:29:47.193 DEBUG:boto:Port: 8000 2023-05-31T20:29:47.193 DEBUG:boto:Params: {} 2023-05-31T20:29:47.194 DEBUG:boto:Token: None 2023-05-31T20:29:47.194 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:29:47 GMT /pklxkd-55/d 2023-05-31T20:29:47.195 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:7NERFesywT7Ehxd1WNnCZkSPCw8= 2023-05-31T20:29:47.195 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:29:47 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:7NERFesywT7Ehxd1WNnCZkSPCw8=', 'Content-Length': '0'} 2023-05-31T20:29:47.197 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:27:16 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000bbec8960dcaf26fd-006477ae3b-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:29:47 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:29:47.198 DEBUG:boto:path=/pklxkd-55/ 2023-05-31T20:29:47.198 DEBUG:boto:auth_path=/pklxkd-55/ 2023-05-31T20:29:47.198 DEBUG:boto:path=/pklxkd-55/?prefix=d 2023-05-31T20:29:47.199 DEBUG:boto:auth_path=/pklxkd-55/?prefix=d 2023-05-31T20:29:47.199 DEBUG:boto:Method: GET 2023-05-31T20:29:47.199 DEBUG:boto:Path: /pklxkd-55/?prefix=d 2023-05-31T20:29:47.200 DEBUG:boto:Data: 2023-05-31T20:29:47.201 DEBUG:boto:Headers: {} 2023-05-31T20:29:47.201 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:29:47.201 DEBUG:boto:Port: 8001 2023-05-31T20:29:47.202 DEBUG:boto:Params: {} 2023-05-31T20:29:47.203 DEBUG:boto:Token: None 2023-05-31T20:29:47.204 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:29:47 GMT /pklxkd-55/ 2023-05-31T20:29:47.205 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:IBdSyRVlh8qg/aH9XxJ5KZfq+k4= 2023-05-31T20:29:47.206 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:29:47 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:IBdSyRVlh8qg/aH9XxJ5KZfq+k4=', 'Content-Length': '0'} 2023-05-31T20:29:47.208 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000053ecf5a10431c038-006477ae3b-162e-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:29:47 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:29:47.209 DEBUG:boto:b'pklxkd-55d1000falsed2023-05-31T20:27:16.893Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:29:47.210 DEBUG:boto:path=/pklxkd-55/ 2023-05-31T20:29:47.210 DEBUG:boto:auth_path=/pklxkd-55/ 2023-05-31T20:29:47.211 DEBUG:boto:path=/pklxkd-55/?prefix=d 2023-05-31T20:29:47.211 DEBUG:boto:auth_path=/pklxkd-55/?prefix=d 2023-05-31T20:29:47.212 DEBUG:boto:Method: GET 2023-05-31T20:29:47.212 DEBUG:boto:Path: /pklxkd-55/?prefix=d 2023-05-31T20:29:47.212 DEBUG:boto:Data: 2023-05-31T20:29:47.213 DEBUG:boto:Headers: {} 2023-05-31T20:29:47.213 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:29:47.214 DEBUG:boto:Port: 8000 2023-05-31T20:29:47.214 DEBUG:boto:Params: {} 2023-05-31T20:29:47.215 DEBUG:boto:Token: None 2023-05-31T20:29:47.215 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:29:47 GMT /pklxkd-55/ 2023-05-31T20:29:47.215 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:IBdSyRVlh8qg/aH9XxJ5KZfq+k4= 2023-05-31T20:29:47.216 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:29:47 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:IBdSyRVlh8qg/aH9XxJ5KZfq+k4=', 'Content-Length': '0'} 2023-05-31T20:29:47.218 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c25f6a88831bfcd2-006477ae3b-1630-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:29:47 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:29:47.264 DEBUG:boto:b'pklxkd-55d1000falsed2023-05-31T20:27:16.893Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:29:47.265 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=e 2023-05-31T20:29:47.265 DEBUG:boto:path=/pklxkd-55/e 2023-05-31T20:29:47.266 DEBUG:boto:auth_path=/pklxkd-55/e 2023-05-31T20:29:47.266 DEBUG:boto:path=/pklxkd-55/e?versionId=null 2023-05-31T20:29:47.267 DEBUG:boto:auth_path=/pklxkd-55/e?versionId=null 2023-05-31T20:29:47.267 DEBUG:boto:Method: GET 2023-05-31T20:29:47.267 DEBUG:boto:Path: /pklxkd-55/e?versionId=null 2023-05-31T20:29:47.268 DEBUG:boto:Data: 2023-05-31T20:29:47.269 DEBUG:boto:Headers: {} 2023-05-31T20:29:47.269 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:29:47.269 DEBUG:boto:Port: 8001 2023-05-31T20:29:47.270 DEBUG:boto:Params: {} 2023-05-31T20:29:47.270 DEBUG:boto:Token: None 2023-05-31T20:29:47.271 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:29:47 GMT /pklxkd-55/e?versionId=null 2023-05-31T20:29:47.271 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:dbj3RRF7g23SIKkaPNS7dnpciek= 2023-05-31T20:29:47.272 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:29:47 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:dbj3RRF7g23SIKkaPNS7dnpciek=', 'Content-Length': '0'} 2023-05-31T20:29:47.274 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:27:23 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000094d68dcb50968389-006477ae3b-162e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:29:47 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:29:47.275 DEBUG:boto:path=/pklxkd-55/e 2023-05-31T20:29:47.275 DEBUG:boto:auth_path=/pklxkd-55/e 2023-05-31T20:29:47.275 DEBUG:boto:path=/pklxkd-55/e?versionId=null 2023-05-31T20:29:47.276 DEBUG:boto:auth_path=/pklxkd-55/e?versionId=null 2023-05-31T20:29:47.277 DEBUG:boto:Method: GET 2023-05-31T20:29:47.277 DEBUG:boto:Path: /pklxkd-55/e?versionId=null 2023-05-31T20:29:47.277 DEBUG:boto:Data: 2023-05-31T20:29:47.278 DEBUG:boto:Headers: {} 2023-05-31T20:29:47.278 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:29:47.278 DEBUG:boto:Port: 8000 2023-05-31T20:29:47.279 DEBUG:boto:Params: {} 2023-05-31T20:29:47.279 DEBUG:boto:Token: None 2023-05-31T20:29:47.280 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:29:47 GMT /pklxkd-55/e?versionId=null 2023-05-31T20:29:47.280 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:dbj3RRF7g23SIKkaPNS7dnpciek= 2023-05-31T20:29:47.281 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:29:47 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:dbj3RRF7g23SIKkaPNS7dnpciek=', 'Content-Length': '0'} 2023-05-31T20:29:47.282 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:27:23 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000225c84ef46c560a-006477ae3b-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:29:47 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:29:47.284 DEBUG:boto:path=/pklxkd-55/e 2023-05-31T20:29:47.285 DEBUG:boto:auth_path=/pklxkd-55/e 2023-05-31T20:29:47.287 DEBUG:boto:path=/pklxkd-55/e?versionId=null 2023-05-31T20:29:47.288 DEBUG:boto:auth_path=/pklxkd-55/e?versionId=null 2023-05-31T20:29:47.289 DEBUG:boto:Method: HEAD 2023-05-31T20:29:47.289 DEBUG:boto:Path: /pklxkd-55/e?versionId=null 2023-05-31T20:29:47.290 DEBUG:boto:Data: 2023-05-31T20:29:47.291 DEBUG:boto:Headers: {} 2023-05-31T20:29:47.292 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:29:47.293 DEBUG:boto:Port: 8001 2023-05-31T20:29:47.293 DEBUG:boto:Params: {} 2023-05-31T20:29:47.294 DEBUG:boto:Token: None 2023-05-31T20:29:47.294 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:29:47 GMT /pklxkd-55/e?versionId=null 2023-05-31T20:29:47.295 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:EMisUW3qdqzH3u7tNtCSn9vJCek= 2023-05-31T20:29:47.295 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:29:47 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:EMisUW3qdqzH3u7tNtCSn9vJCek=', 'Content-Length': '0'} 2023-05-31T20:29:47.298 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:27:23 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f4d2ee4fb4efa063-006477ae3b-162e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:29:47 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:29:47.298 DEBUG:boto:path=/pklxkd-55/e 2023-05-31T20:29:47.299 DEBUG:boto:auth_path=/pklxkd-55/e 2023-05-31T20:29:47.299 DEBUG:boto:path=/pklxkd-55/e?versionId=null 2023-05-31T20:29:47.299 DEBUG:boto:auth_path=/pklxkd-55/e?versionId=null 2023-05-31T20:29:47.300 DEBUG:boto:Method: HEAD 2023-05-31T20:29:47.300 DEBUG:boto:Path: /pklxkd-55/e?versionId=null 2023-05-31T20:29:47.301 DEBUG:boto:Data: 2023-05-31T20:29:47.301 DEBUG:boto:Headers: {} 2023-05-31T20:29:47.301 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:29:47.302 DEBUG:boto:Port: 8000 2023-05-31T20:29:47.302 DEBUG:boto:Params: {} 2023-05-31T20:29:47.302 DEBUG:boto:Token: None 2023-05-31T20:29:47.303 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:29:47 GMT /pklxkd-55/e?versionId=null 2023-05-31T20:29:47.304 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:EMisUW3qdqzH3u7tNtCSn9vJCek= 2023-05-31T20:29:47.304 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:29:47 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:EMisUW3qdqzH3u7tNtCSn9vJCek=', 'Content-Length': '0'} 2023-05-31T20:29:47.306 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:27:23 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000285f373464c67f62-006477ae3b-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:29:47 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:29:47.307 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=e 2023-05-31T20:29:47.307 DEBUG:boto:path=/pklxkd-55/e 2023-05-31T20:29:47.307 DEBUG:boto:auth_path=/pklxkd-55/e 2023-05-31T20:29:47.308 DEBUG:boto:path=/pklxkd-55/e?versionId=null 2023-05-31T20:29:47.308 DEBUG:boto:auth_path=/pklxkd-55/e?versionId=null 2023-05-31T20:29:47.309 DEBUG:boto:Method: GET 2023-05-31T20:29:47.309 DEBUG:boto:Path: /pklxkd-55/e?versionId=null 2023-05-31T20:29:47.309 DEBUG:boto:Data: 2023-05-31T20:29:47.310 DEBUG:boto:Headers: {} 2023-05-31T20:29:47.310 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:29:47.310 DEBUG:boto:Port: 8001 2023-05-31T20:29:47.311 DEBUG:boto:Params: {} 2023-05-31T20:29:47.311 DEBUG:boto:Token: None 2023-05-31T20:29:47.312 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:29:47 GMT /pklxkd-55/e?versionId=null 2023-05-31T20:29:47.312 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:dbj3RRF7g23SIKkaPNS7dnpciek= 2023-05-31T20:29:47.313 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:29:47 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:dbj3RRF7g23SIKkaPNS7dnpciek=', 'Content-Length': '0'} 2023-05-31T20:29:47.315 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:27:23 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005f3bd3662479f769-006477ae3b-162e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:29:47 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:29:47.316 DEBUG:boto:path=/pklxkd-55/e 2023-05-31T20:29:47.316 DEBUG:boto:auth_path=/pklxkd-55/e 2023-05-31T20:29:47.317 DEBUG:boto:path=/pklxkd-55/e?versionId=null 2023-05-31T20:29:47.318 DEBUG:boto:auth_path=/pklxkd-55/e?versionId=null 2023-05-31T20:29:47.318 DEBUG:boto:Method: GET 2023-05-31T20:29:47.318 DEBUG:boto:Path: /pklxkd-55/e?versionId=null 2023-05-31T20:29:47.319 DEBUG:boto:Data: 2023-05-31T20:29:47.319 DEBUG:boto:Headers: {} 2023-05-31T20:29:47.320 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:29:47.320 DEBUG:boto:Port: 8000 2023-05-31T20:29:47.320 DEBUG:boto:Params: {} 2023-05-31T20:29:47.321 DEBUG:boto:Token: None 2023-05-31T20:29:47.321 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:29:47 GMT /pklxkd-55/e?versionId=null 2023-05-31T20:29:47.322 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:dbj3RRF7g23SIKkaPNS7dnpciek= 2023-05-31T20:29:47.322 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:29:47 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:dbj3RRF7g23SIKkaPNS7dnpciek=', 'Content-Length': '0'} 2023-05-31T20:29:47.324 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:27:23 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000736a400a3b7da12a-006477ae3b-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:29:47 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:29:47.325 DEBUG:boto:path=/pklxkd-55/ 2023-05-31T20:29:47.326 DEBUG:boto:auth_path=/pklxkd-55/ 2023-05-31T20:29:47.327 DEBUG:boto:path=/pklxkd-55/?prefix=e 2023-05-31T20:29:47.328 DEBUG:boto:auth_path=/pklxkd-55/?prefix=e 2023-05-31T20:29:47.328 DEBUG:boto:Method: GET 2023-05-31T20:29:47.329 DEBUG:boto:Path: /pklxkd-55/?prefix=e 2023-05-31T20:29:47.330 DEBUG:boto:Data: 2023-05-31T20:29:47.331 DEBUG:boto:Headers: {} 2023-05-31T20:29:47.332 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:29:47.332 DEBUG:boto:Port: 8001 2023-05-31T20:29:47.333 DEBUG:boto:Params: {} 2023-05-31T20:29:47.334 DEBUG:boto:Token: None 2023-05-31T20:29:47.334 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:29:47 GMT /pklxkd-55/ 2023-05-31T20:29:47.335 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:IBdSyRVlh8qg/aH9XxJ5KZfq+k4= 2023-05-31T20:29:47.336 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:29:47 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:IBdSyRVlh8qg/aH9XxJ5KZfq+k4=', 'Content-Length': '0'} 2023-05-31T20:29:47.339 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000002ab5d7da6605e3c4-006477ae3b-162e-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:29:47 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:29:47.340 DEBUG:boto:b'pklxkd-55e1000falsee2023-05-31T20:27:23.955Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:29:47.341 DEBUG:boto:path=/pklxkd-55/ 2023-05-31T20:29:47.342 DEBUG:boto:auth_path=/pklxkd-55/ 2023-05-31T20:29:47.343 DEBUG:boto:path=/pklxkd-55/?prefix=e 2023-05-31T20:29:47.344 DEBUG:boto:auth_path=/pklxkd-55/?prefix=e 2023-05-31T20:29:47.344 DEBUG:boto:Method: GET 2023-05-31T20:29:47.345 DEBUG:boto:Path: /pklxkd-55/?prefix=e 2023-05-31T20:29:47.346 DEBUG:boto:Data: 2023-05-31T20:29:47.347 DEBUG:boto:Headers: {} 2023-05-31T20:29:47.348 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:29:47.348 DEBUG:boto:Port: 8000 2023-05-31T20:29:47.349 DEBUG:boto:Params: {} 2023-05-31T20:29:47.352 DEBUG:boto:Token: None 2023-05-31T20:29:47.353 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:29:47 GMT /pklxkd-55/ 2023-05-31T20:29:47.353 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:IBdSyRVlh8qg/aH9XxJ5KZfq+k4= 2023-05-31T20:29:47.354 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:29:47 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:IBdSyRVlh8qg/aH9XxJ5KZfq+k4=', 'Content-Length': '0'} 2023-05-31T20:29:47.355 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000005505522a4c96dd97-006477ae3b-1630-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:29:47 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:29:47.356 DEBUG:boto:b'pklxkd-55e1000falsee2023-05-31T20:27:23.955Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:29:47.357 DEBUG:boto:path=/pklxkd-55/e 2023-05-31T20:29:47.357 DEBUG:boto:auth_path=/pklxkd-55/e 2023-05-31T20:29:47.358 DEBUG:boto:Method: HEAD 2023-05-31T20:29:47.358 DEBUG:boto:Path: /pklxkd-55/e 2023-05-31T20:29:47.358 DEBUG:boto:Data: 2023-05-31T20:29:47.359 DEBUG:boto:Headers: {} 2023-05-31T20:29:47.359 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:29:47.359 DEBUG:boto:Port: 8001 2023-05-31T20:29:47.360 DEBUG:boto:Params: {} 2023-05-31T20:29:47.360 DEBUG:boto:Token: None 2023-05-31T20:29:47.360 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:29:47 GMT /pklxkd-55/e 2023-05-31T20:29:47.361 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:BH7veQe7nRdctzqYNJfbli6Qsqs= 2023-05-31T20:29:47.361 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:29:47 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:BH7veQe7nRdctzqYNJfbli6Qsqs=', 'Content-Length': '0'} 2023-05-31T20:29:47.363 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:27:23 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000013834ba2d3a89bc8-006477ae3b-162e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:29:47 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:29:47.364 DEBUG:boto:path=/pklxkd-55/e 2023-05-31T20:29:47.364 DEBUG:boto:auth_path=/pklxkd-55/e 2023-05-31T20:29:47.365 DEBUG:boto:Method: HEAD 2023-05-31T20:29:47.365 DEBUG:boto:Path: /pklxkd-55/e 2023-05-31T20:29:47.365 DEBUG:boto:Data: 2023-05-31T20:29:47.366 DEBUG:boto:Headers: {} 2023-05-31T20:29:47.366 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:29:47.367 DEBUG:boto:Port: 8000 2023-05-31T20:29:47.367 DEBUG:boto:Params: {} 2023-05-31T20:29:47.367 DEBUG:boto:Token: None 2023-05-31T20:29:47.368 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:29:47 GMT /pklxkd-55/e 2023-05-31T20:29:47.368 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:BH7veQe7nRdctzqYNJfbli6Qsqs= 2023-05-31T20:29:47.369 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:29:47 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:BH7veQe7nRdctzqYNJfbli6Qsqs=', 'Content-Length': '0'} 2023-05-31T20:29:47.371 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:27:23 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a6cd2b6b34890636-006477ae3b-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:29:47 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:29:47.372 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=e 2023-05-31T20:29:47.373 DEBUG:boto:path=/pklxkd-55/e 2023-05-31T20:29:47.374 DEBUG:boto:auth_path=/pklxkd-55/e 2023-05-31T20:29:47.375 DEBUG:boto:Method: GET 2023-05-31T20:29:47.375 DEBUG:boto:Path: /pklxkd-55/e 2023-05-31T20:29:47.376 DEBUG:boto:Data: 2023-05-31T20:29:47.377 DEBUG:boto:Headers: {} 2023-05-31T20:29:47.377 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:29:47.378 DEBUG:boto:Port: 8001 2023-05-31T20:29:47.378 DEBUG:boto:Params: {} 2023-05-31T20:29:47.378 DEBUG:boto:Token: None 2023-05-31T20:29:47.379 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:29:47 GMT /pklxkd-55/e 2023-05-31T20:29:47.379 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:lVX4sPK/Kzs95E32XjhOwYvD868= 2023-05-31T20:29:47.380 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:29:47 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:lVX4sPK/Kzs95E32XjhOwYvD868=', 'Content-Length': '0'} 2023-05-31T20:29:47.381 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:27:23 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003e5fd2a0107bc72c-006477ae3b-162e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:29:47 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:29:47.382 DEBUG:boto:path=/pklxkd-55/e 2023-05-31T20:29:47.382 DEBUG:boto:auth_path=/pklxkd-55/e 2023-05-31T20:29:47.383 DEBUG:boto:Method: GET 2023-05-31T20:29:47.383 DEBUG:boto:Path: /pklxkd-55/e 2023-05-31T20:29:47.383 DEBUG:boto:Data: 2023-05-31T20:29:47.384 DEBUG:boto:Headers: {} 2023-05-31T20:29:47.384 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:29:47.385 DEBUG:boto:Port: 8000 2023-05-31T20:29:47.385 DEBUG:boto:Params: {} 2023-05-31T20:29:47.386 DEBUG:boto:Token: None 2023-05-31T20:29:47.386 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:29:47 GMT /pklxkd-55/e 2023-05-31T20:29:47.386 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:lVX4sPK/Kzs95E32XjhOwYvD868= 2023-05-31T20:29:47.387 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:29:47 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:lVX4sPK/Kzs95E32XjhOwYvD868=', 'Content-Length': '0'} 2023-05-31T20:29:47.388 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:27:23 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005ccb1768bfb0e521-006477ae3b-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:29:47 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:29:47.389 DEBUG:boto:path=/pklxkd-55/ 2023-05-31T20:29:47.390 DEBUG:boto:auth_path=/pklxkd-55/ 2023-05-31T20:29:47.390 DEBUG:boto:path=/pklxkd-55/?prefix=e 2023-05-31T20:29:47.390 DEBUG:boto:auth_path=/pklxkd-55/?prefix=e 2023-05-31T20:29:47.391 DEBUG:boto:Method: GET 2023-05-31T20:29:47.391 DEBUG:boto:Path: /pklxkd-55/?prefix=e 2023-05-31T20:29:47.392 DEBUG:boto:Data: 2023-05-31T20:29:47.392 DEBUG:boto:Headers: {} 2023-05-31T20:29:47.393 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:29:47.393 DEBUG:boto:Port: 8001 2023-05-31T20:29:47.393 DEBUG:boto:Params: {} 2023-05-31T20:29:47.394 DEBUG:boto:Token: None 2023-05-31T20:29:47.394 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:29:47 GMT /pklxkd-55/ 2023-05-31T20:29:47.395 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:IBdSyRVlh8qg/aH9XxJ5KZfq+k4= 2023-05-31T20:29:47.395 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:29:47 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:IBdSyRVlh8qg/aH9XxJ5KZfq+k4=', 'Content-Length': '0'} 2023-05-31T20:29:47.397 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e3d429776c8f9d39-006477ae3b-162e-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:29:47 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:29:47.399 DEBUG:boto:b'pklxkd-55e1000falsee2023-05-31T20:27:23.955Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:29:47.400 DEBUG:boto:path=/pklxkd-55/ 2023-05-31T20:29:47.401 DEBUG:boto:auth_path=/pklxkd-55/ 2023-05-31T20:29:47.402 DEBUG:boto:path=/pklxkd-55/?prefix=e 2023-05-31T20:29:47.403 DEBUG:boto:auth_path=/pklxkd-55/?prefix=e 2023-05-31T20:29:47.404 DEBUG:boto:Method: GET 2023-05-31T20:29:47.405 DEBUG:boto:Path: /pklxkd-55/?prefix=e 2023-05-31T20:29:47.405 DEBUG:boto:Data: 2023-05-31T20:29:47.406 DEBUG:boto:Headers: {} 2023-05-31T20:29:47.407 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:29:47.407 DEBUG:boto:Port: 8000 2023-05-31T20:29:47.408 DEBUG:boto:Params: {} 2023-05-31T20:29:47.409 DEBUG:boto:Token: None 2023-05-31T20:29:47.409 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:29:47 GMT /pklxkd-55/ 2023-05-31T20:29:47.410 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:IBdSyRVlh8qg/aH9XxJ5KZfq+k4= 2023-05-31T20:29:47.411 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:29:47 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:IBdSyRVlh8qg/aH9XxJ5KZfq+k4=', 'Content-Length': '0'} 2023-05-31T20:29:47.412 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000005b6984bd6b6ebba3-006477ae3b-1630-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:29:47 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:29:47.460 DEBUG:boto:b'pklxkd-55e1000falsee2023-05-31T20:27:23.955Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:29:47.461 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=f 2023-05-31T20:29:47.462 DEBUG:boto:path=/pklxkd-55/f 2023-05-31T20:29:47.462 DEBUG:boto:auth_path=/pklxkd-55/f 2023-05-31T20:29:47.463 DEBUG:boto:path=/pklxkd-55/f?versionId=null 2023-05-31T20:29:47.463 DEBUG:boto:auth_path=/pklxkd-55/f?versionId=null 2023-05-31T20:29:47.463 DEBUG:boto:Method: GET 2023-05-31T20:29:47.464 DEBUG:boto:Path: /pklxkd-55/f?versionId=null 2023-05-31T20:29:47.464 DEBUG:boto:Data: 2023-05-31T20:29:47.465 DEBUG:boto:Headers: {} 2023-05-31T20:29:47.465 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:29:47.465 DEBUG:boto:Port: 8001 2023-05-31T20:29:47.466 DEBUG:boto:Params: {} 2023-05-31T20:29:47.466 DEBUG:boto:Token: None 2023-05-31T20:29:47.467 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:29:47 GMT /pklxkd-55/f?versionId=null 2023-05-31T20:29:47.467 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:Mfy/ziqHy+KzjGSncy7KaRwMtxQ= 2023-05-31T20:29:47.468 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:29:47 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:Mfy/ziqHy+KzjGSncy7KaRwMtxQ=', 'Content-Length': '0'} 2023-05-31T20:29:47.469 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:27:23 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000058ff10fe3f6e86f3-006477ae3b-162e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:29:47 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:29:47.470 DEBUG:boto:path=/pklxkd-55/f 2023-05-31T20:29:47.470 DEBUG:boto:auth_path=/pklxkd-55/f 2023-05-31T20:29:47.471 DEBUG:boto:path=/pklxkd-55/f?versionId=null 2023-05-31T20:29:47.471 DEBUG:boto:auth_path=/pklxkd-55/f?versionId=null 2023-05-31T20:29:47.471 DEBUG:boto:Method: GET 2023-05-31T20:29:47.472 DEBUG:boto:Path: /pklxkd-55/f?versionId=null 2023-05-31T20:29:47.472 DEBUG:boto:Data: 2023-05-31T20:29:47.473 DEBUG:boto:Headers: {} 2023-05-31T20:29:47.473 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:29:47.473 DEBUG:boto:Port: 8000 2023-05-31T20:29:47.474 DEBUG:boto:Params: {} 2023-05-31T20:29:47.474 DEBUG:boto:Token: None 2023-05-31T20:29:47.475 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:29:47 GMT /pklxkd-55/f?versionId=null 2023-05-31T20:29:47.475 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:Mfy/ziqHy+KzjGSncy7KaRwMtxQ= 2023-05-31T20:29:47.476 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:29:47 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:Mfy/ziqHy+KzjGSncy7KaRwMtxQ=', 'Content-Length': '0'} 2023-05-31T20:29:47.477 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:27:23 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000097d6e172d97d6b88-006477ae3b-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:29:47 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:29:47.478 DEBUG:boto:path=/pklxkd-55/f 2023-05-31T20:29:47.479 DEBUG:boto:auth_path=/pklxkd-55/f 2023-05-31T20:29:47.479 DEBUG:boto:path=/pklxkd-55/f?versionId=null 2023-05-31T20:29:47.479 DEBUG:boto:auth_path=/pklxkd-55/f?versionId=null 2023-05-31T20:29:47.480 DEBUG:boto:Method: HEAD 2023-05-31T20:29:47.480 DEBUG:boto:Path: /pklxkd-55/f?versionId=null 2023-05-31T20:29:47.480 DEBUG:boto:Data: 2023-05-31T20:29:47.481 DEBUG:boto:Headers: {} 2023-05-31T20:29:47.481 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:29:47.481 DEBUG:boto:Port: 8001 2023-05-31T20:29:47.482 DEBUG:boto:Params: {} 2023-05-31T20:29:47.482 DEBUG:boto:Token: None 2023-05-31T20:29:47.482 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:29:47 GMT /pklxkd-55/f?versionId=null 2023-05-31T20:29:47.483 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:/pgwyH5Q7I5Kk2t5ntqQVa6/9es= 2023-05-31T20:29:47.484 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:29:47 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:/pgwyH5Q7I5Kk2t5ntqQVa6/9es=', 'Content-Length': '0'} 2023-05-31T20:29:47.485 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:27:23 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009518d7cecf9fb350-006477ae3b-162e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:29:47 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:29:47.486 DEBUG:boto:path=/pklxkd-55/f 2023-05-31T20:29:47.486 DEBUG:boto:auth_path=/pklxkd-55/f 2023-05-31T20:29:47.487 DEBUG:boto:path=/pklxkd-55/f?versionId=null 2023-05-31T20:29:47.487 DEBUG:boto:auth_path=/pklxkd-55/f?versionId=null 2023-05-31T20:29:47.488 DEBUG:boto:Method: HEAD 2023-05-31T20:29:47.489 DEBUG:boto:Path: /pklxkd-55/f?versionId=null 2023-05-31T20:29:47.489 DEBUG:boto:Data: 2023-05-31T20:29:47.490 DEBUG:boto:Headers: {} 2023-05-31T20:29:47.491 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:29:47.492 DEBUG:boto:Port: 8000 2023-05-31T20:29:47.493 DEBUG:boto:Params: {} 2023-05-31T20:29:47.493 DEBUG:boto:Token: None 2023-05-31T20:29:47.494 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:29:47 GMT /pklxkd-55/f?versionId=null 2023-05-31T20:29:47.495 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:/pgwyH5Q7I5Kk2t5ntqQVa6/9es= 2023-05-31T20:29:47.495 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:29:47 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:/pgwyH5Q7I5Kk2t5ntqQVa6/9es=', 'Content-Length': '0'} 2023-05-31T20:29:47.497 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:27:23 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e3bd70b245716530-006477ae3b-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:29:47 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:29:47.497 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=f 2023-05-31T20:29:47.498 DEBUG:boto:path=/pklxkd-55/f 2023-05-31T20:29:47.498 DEBUG:boto:auth_path=/pklxkd-55/f 2023-05-31T20:29:47.499 DEBUG:boto:path=/pklxkd-55/f?versionId=null 2023-05-31T20:29:47.499 DEBUG:boto:auth_path=/pklxkd-55/f?versionId=null 2023-05-31T20:29:47.500 DEBUG:boto:Method: GET 2023-05-31T20:29:47.501 DEBUG:boto:Path: /pklxkd-55/f?versionId=null 2023-05-31T20:29:47.501 DEBUG:boto:Data: 2023-05-31T20:29:47.502 DEBUG:boto:Headers: {} 2023-05-31T20:29:47.502 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:29:47.502 DEBUG:boto:Port: 8001 2023-05-31T20:29:47.503 DEBUG:boto:Params: {} 2023-05-31T20:29:47.503 DEBUG:boto:Token: None 2023-05-31T20:29:47.504 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:29:47 GMT /pklxkd-55/f?versionId=null 2023-05-31T20:29:47.505 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:Mfy/ziqHy+KzjGSncy7KaRwMtxQ= 2023-05-31T20:29:47.506 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:29:47 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:Mfy/ziqHy+KzjGSncy7KaRwMtxQ=', 'Content-Length': '0'} 2023-05-31T20:29:47.508 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:27:23 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000488fe1b89a19f9db-006477ae3b-162e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:29:47 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:29:47.509 DEBUG:boto:path=/pklxkd-55/f 2023-05-31T20:29:47.510 DEBUG:boto:auth_path=/pklxkd-55/f 2023-05-31T20:29:47.511 DEBUG:boto:path=/pklxkd-55/f?versionId=null 2023-05-31T20:29:47.512 DEBUG:boto:auth_path=/pklxkd-55/f?versionId=null 2023-05-31T20:29:47.513 DEBUG:boto:Method: GET 2023-05-31T20:29:47.514 DEBUG:boto:Path: /pklxkd-55/f?versionId=null 2023-05-31T20:29:47.515 DEBUG:boto:Data: 2023-05-31T20:29:47.516 DEBUG:boto:Headers: {} 2023-05-31T20:29:47.517 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:29:47.519 DEBUG:boto:Port: 8000 2023-05-31T20:29:47.520 DEBUG:boto:Params: {} 2023-05-31T20:29:47.521 DEBUG:boto:Token: None 2023-05-31T20:29:47.523 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:29:47 GMT /pklxkd-55/f?versionId=null 2023-05-31T20:29:47.524 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:Mfy/ziqHy+KzjGSncy7KaRwMtxQ= 2023-05-31T20:29:47.524 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:29:47 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:Mfy/ziqHy+KzjGSncy7KaRwMtxQ=', 'Content-Length': '0'} 2023-05-31T20:29:47.528 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:27:23 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000848476d6171c2f54-006477ae3b-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:29:47 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:29:47.528 DEBUG:boto:path=/pklxkd-55/ 2023-05-31T20:29:47.529 DEBUG:boto:auth_path=/pklxkd-55/ 2023-05-31T20:29:47.530 DEBUG:boto:path=/pklxkd-55/?prefix=f 2023-05-31T20:29:47.532 DEBUG:boto:auth_path=/pklxkd-55/?prefix=f 2023-05-31T20:29:47.533 DEBUG:boto:Method: GET 2023-05-31T20:29:47.533 DEBUG:boto:Path: /pklxkd-55/?prefix=f 2023-05-31T20:29:47.534 DEBUG:boto:Data: 2023-05-31T20:29:47.535 DEBUG:boto:Headers: {} 2023-05-31T20:29:47.535 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:29:47.536 DEBUG:boto:Port: 8001 2023-05-31T20:29:47.536 DEBUG:boto:Params: {} 2023-05-31T20:29:47.537 DEBUG:boto:Token: None 2023-05-31T20:29:47.538 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:29:47 GMT /pklxkd-55/ 2023-05-31T20:29:47.538 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:IBdSyRVlh8qg/aH9XxJ5KZfq+k4= 2023-05-31T20:29:47.539 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:29:47 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:IBdSyRVlh8qg/aH9XxJ5KZfq+k4=', 'Content-Length': '0'} 2023-05-31T20:29:47.541 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c36ed2d2b3ee014c-006477ae3b-162e-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:29:47 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:29:47.584 DEBUG:boto:b'pklxkd-55f1000falsef2023-05-31T20:27:23.977Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:29:47.585 DEBUG:boto:path=/pklxkd-55/ 2023-05-31T20:29:47.586 DEBUG:boto:auth_path=/pklxkd-55/ 2023-05-31T20:29:47.586 DEBUG:boto:path=/pklxkd-55/?prefix=f 2023-05-31T20:29:47.586 DEBUG:boto:auth_path=/pklxkd-55/?prefix=f 2023-05-31T20:29:47.587 DEBUG:boto:Method: GET 2023-05-31T20:29:47.587 DEBUG:boto:Path: /pklxkd-55/?prefix=f 2023-05-31T20:29:47.588 DEBUG:boto:Data: 2023-05-31T20:29:47.588 DEBUG:boto:Headers: {} 2023-05-31T20:29:47.588 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:29:47.589 DEBUG:boto:Port: 8000 2023-05-31T20:29:47.595 DEBUG:boto:Params: {} 2023-05-31T20:29:47.595 DEBUG:boto:Token: None 2023-05-31T20:29:47.596 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:29:47 GMT /pklxkd-55/ 2023-05-31T20:29:47.596 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:IBdSyRVlh8qg/aH9XxJ5KZfq+k4= 2023-05-31T20:29:47.597 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:29:47 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:IBdSyRVlh8qg/aH9XxJ5KZfq+k4=', 'Content-Length': '0'} 2023-05-31T20:29:47.598 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000015cce64da71492e9-006477ae3b-1630-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:29:47 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:29:47.599 DEBUG:boto:b'pklxkd-55f1000falsef2023-05-31T20:27:23.977Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:29:47.600 DEBUG:boto:path=/pklxkd-55/f 2023-05-31T20:29:47.600 DEBUG:boto:auth_path=/pklxkd-55/f 2023-05-31T20:29:47.601 DEBUG:boto:Method: HEAD 2023-05-31T20:29:47.601 DEBUG:boto:Path: /pklxkd-55/f 2023-05-31T20:29:47.601 DEBUG:boto:Data: 2023-05-31T20:29:47.602 DEBUG:boto:Headers: {} 2023-05-31T20:29:47.602 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:29:47.602 DEBUG:boto:Port: 8001 2023-05-31T20:29:47.603 DEBUG:boto:Params: {} 2023-05-31T20:29:47.603 DEBUG:boto:Token: None 2023-05-31T20:29:47.604 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:29:47 GMT /pklxkd-55/f 2023-05-31T20:29:47.604 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:pWSXwnWTK74NrhdIBQRvMbeg5qs= 2023-05-31T20:29:47.604 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:29:47 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:pWSXwnWTK74NrhdIBQRvMbeg5qs=', 'Content-Length': '0'} 2023-05-31T20:29:47.606 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:27:23 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000060dda93cedae8c96-006477ae3b-162e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:29:47 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:29:47.607 DEBUG:boto:path=/pklxkd-55/f 2023-05-31T20:29:47.607 DEBUG:boto:auth_path=/pklxkd-55/f 2023-05-31T20:29:47.607 DEBUG:boto:Method: HEAD 2023-05-31T20:29:47.608 DEBUG:boto:Path: /pklxkd-55/f 2023-05-31T20:29:47.608 DEBUG:boto:Data: 2023-05-31T20:29:47.609 DEBUG:boto:Headers: {} 2023-05-31T20:29:47.609 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:29:47.609 DEBUG:boto:Port: 8000 2023-05-31T20:29:47.610 DEBUG:boto:Params: {} 2023-05-31T20:29:47.610 DEBUG:boto:Token: None 2023-05-31T20:29:47.610 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:29:47 GMT /pklxkd-55/f 2023-05-31T20:29:47.611 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:pWSXwnWTK74NrhdIBQRvMbeg5qs= 2023-05-31T20:29:47.611 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:29:47 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:pWSXwnWTK74NrhdIBQRvMbeg5qs=', 'Content-Length': '0'} 2023-05-31T20:29:47.613 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:27:23 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f5c9929bb24e0358-006477ae3b-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:29:47 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:29:47.613 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=f 2023-05-31T20:29:47.614 DEBUG:boto:path=/pklxkd-55/f 2023-05-31T20:29:47.614 DEBUG:boto:auth_path=/pklxkd-55/f 2023-05-31T20:29:47.615 DEBUG:boto:Method: GET 2023-05-31T20:29:47.615 DEBUG:boto:Path: /pklxkd-55/f 2023-05-31T20:29:47.616 DEBUG:boto:Data: 2023-05-31T20:29:47.616 DEBUG:boto:Headers: {} 2023-05-31T20:29:47.616 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:29:47.617 DEBUG:boto:Port: 8001 2023-05-31T20:29:47.617 DEBUG:boto:Params: {} 2023-05-31T20:29:47.618 DEBUG:boto:Token: None 2023-05-31T20:29:47.618 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:29:47 GMT /pklxkd-55/f 2023-05-31T20:29:47.618 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:ZqS8ujsq9YKWQDyX4J33P7BIN+Q= 2023-05-31T20:29:47.619 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:29:47 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:ZqS8ujsq9YKWQDyX4J33P7BIN+Q=', 'Content-Length': '0'} 2023-05-31T20:29:47.620 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:27:23 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000afb4b03dc06d2bb1-006477ae3b-162e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:29:47 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:29:47.621 DEBUG:boto:path=/pklxkd-55/f 2023-05-31T20:29:47.622 DEBUG:boto:auth_path=/pklxkd-55/f 2023-05-31T20:29:47.622 DEBUG:boto:Method: GET 2023-05-31T20:29:47.622 DEBUG:boto:Path: /pklxkd-55/f 2023-05-31T20:29:47.623 DEBUG:boto:Data: 2023-05-31T20:29:47.623 DEBUG:boto:Headers: {} 2023-05-31T20:29:47.623 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:29:47.624 DEBUG:boto:Port: 8000 2023-05-31T20:29:47.624 DEBUG:boto:Params: {} 2023-05-31T20:29:47.625 DEBUG:boto:Token: None 2023-05-31T20:29:47.625 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:29:47 GMT /pklxkd-55/f 2023-05-31T20:29:47.625 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:ZqS8ujsq9YKWQDyX4J33P7BIN+Q= 2023-05-31T20:29:47.626 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:29:47 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:ZqS8ujsq9YKWQDyX4J33P7BIN+Q=', 'Content-Length': '0'} 2023-05-31T20:29:47.628 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:27:23 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000035f989404b5e68ce-006477ae3b-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:29:47 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:29:47.668 DEBUG:boto:path=/pklxkd-55/ 2023-05-31T20:29:47.669 DEBUG:boto:auth_path=/pklxkd-55/ 2023-05-31T20:29:47.669 DEBUG:boto:path=/pklxkd-55/?prefix=f 2023-05-31T20:29:47.670 DEBUG:boto:auth_path=/pklxkd-55/?prefix=f 2023-05-31T20:29:47.670 DEBUG:boto:Method: GET 2023-05-31T20:29:47.671 DEBUG:boto:Path: /pklxkd-55/?prefix=f 2023-05-31T20:29:47.671 DEBUG:boto:Data: 2023-05-31T20:29:47.671 DEBUG:boto:Headers: {} 2023-05-31T20:29:47.672 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:29:47.673 DEBUG:boto:Port: 8001 2023-05-31T20:29:47.673 DEBUG:boto:Params: {} 2023-05-31T20:29:47.674 DEBUG:boto:Token: None 2023-05-31T20:29:47.674 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:29:47 GMT /pklxkd-55/ 2023-05-31T20:29:47.674 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:IBdSyRVlh8qg/aH9XxJ5KZfq+k4= 2023-05-31T20:29:47.675 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:29:47 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:IBdSyRVlh8qg/aH9XxJ5KZfq+k4=', 'Content-Length': '0'} 2023-05-31T20:29:47.677 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c853f0a2dc6a1471-006477ae3b-162e-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:29:47 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:29:47.678 DEBUG:boto:b'pklxkd-55f1000falsef2023-05-31T20:27:23.977Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:29:47.679 DEBUG:boto:path=/pklxkd-55/ 2023-05-31T20:29:47.679 DEBUG:boto:auth_path=/pklxkd-55/ 2023-05-31T20:29:47.679 DEBUG:boto:path=/pklxkd-55/?prefix=f 2023-05-31T20:29:47.680 DEBUG:boto:auth_path=/pklxkd-55/?prefix=f 2023-05-31T20:29:47.680 DEBUG:boto:Method: GET 2023-05-31T20:29:47.681 DEBUG:boto:Path: /pklxkd-55/?prefix=f 2023-05-31T20:29:47.681 DEBUG:boto:Data: 2023-05-31T20:29:47.681 DEBUG:boto:Headers: {} 2023-05-31T20:29:47.682 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:29:47.682 DEBUG:boto:Port: 8000 2023-05-31T20:29:47.682 DEBUG:boto:Params: {} 2023-05-31T20:29:47.683 DEBUG:boto:Token: None 2023-05-31T20:29:47.683 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:29:47 GMT /pklxkd-55/ 2023-05-31T20:29:47.684 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:IBdSyRVlh8qg/aH9XxJ5KZfq+k4= 2023-05-31T20:29:47.684 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:29:47 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:IBdSyRVlh8qg/aH9XxJ5KZfq+k4=', 'Content-Length': '0'} 2023-05-31T20:29:47.686 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000688df95bdfa5dc15-006477ae3b-1630-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:29:47 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:29:47.687 DEBUG:boto:b'pklxkd-55f1000falsef2023-05-31T20:27:23.977Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:29:47.687 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=g 2023-05-31T20:29:47.688 DEBUG:boto:path=/pklxkd-55/g 2023-05-31T20:29:47.688 DEBUG:boto:auth_path=/pklxkd-55/g 2023-05-31T20:29:47.689 DEBUG:boto:path=/pklxkd-55/g?versionId=null 2023-05-31T20:29:47.689 DEBUG:boto:auth_path=/pklxkd-55/g?versionId=null 2023-05-31T20:29:47.689 DEBUG:boto:Method: GET 2023-05-31T20:29:47.690 DEBUG:boto:Path: /pklxkd-55/g?versionId=null 2023-05-31T20:29:47.690 DEBUG:boto:Data: 2023-05-31T20:29:47.691 DEBUG:boto:Headers: {} 2023-05-31T20:29:47.691 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:29:47.691 DEBUG:boto:Port: 8001 2023-05-31T20:29:47.692 DEBUG:boto:Params: {} 2023-05-31T20:29:47.692 DEBUG:boto:Token: None 2023-05-31T20:29:47.693 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:29:47 GMT /pklxkd-55/g?versionId=null 2023-05-31T20:29:47.693 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:8rwhJVFIeQEuHOFXFh01O4OeEEE= 2023-05-31T20:29:47.694 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:29:47 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:8rwhJVFIeQEuHOFXFh01O4OeEEE=', 'Content-Length': '0'} 2023-05-31T20:29:47.696 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:27:24 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005763b47dc83006a8-006477ae3b-162e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:29:47 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:29:47.697 DEBUG:boto:path=/pklxkd-55/g 2023-05-31T20:29:47.697 DEBUG:boto:auth_path=/pklxkd-55/g 2023-05-31T20:29:47.697 DEBUG:boto:path=/pklxkd-55/g?versionId=null 2023-05-31T20:29:47.698 DEBUG:boto:auth_path=/pklxkd-55/g?versionId=null 2023-05-31T20:29:47.698 DEBUG:boto:Method: GET 2023-05-31T20:29:47.699 DEBUG:boto:Path: /pklxkd-55/g?versionId=null 2023-05-31T20:29:47.699 DEBUG:boto:Data: 2023-05-31T20:29:47.699 DEBUG:boto:Headers: {} 2023-05-31T20:29:47.700 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:29:47.700 DEBUG:boto:Port: 8000 2023-05-31T20:29:47.701 DEBUG:boto:Params: {} 2023-05-31T20:29:47.701 DEBUG:boto:Token: None 2023-05-31T20:29:47.701 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:29:47 GMT /pklxkd-55/g?versionId=null 2023-05-31T20:29:47.702 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:8rwhJVFIeQEuHOFXFh01O4OeEEE= 2023-05-31T20:29:47.702 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:29:47 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:8rwhJVFIeQEuHOFXFh01O4OeEEE=', 'Content-Length': '0'} 2023-05-31T20:29:47.704 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:27:24 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009fe6cc0cd3241251-006477ae3b-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:29:47 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:29:47.705 DEBUG:boto:path=/pklxkd-55/g 2023-05-31T20:29:47.706 DEBUG:boto:auth_path=/pklxkd-55/g 2023-05-31T20:29:47.706 DEBUG:boto:path=/pklxkd-55/g?versionId=null 2023-05-31T20:29:47.706 DEBUG:boto:auth_path=/pklxkd-55/g?versionId=null 2023-05-31T20:29:47.707 DEBUG:boto:Method: HEAD 2023-05-31T20:29:47.707 DEBUG:boto:Path: /pklxkd-55/g?versionId=null 2023-05-31T20:29:47.708 DEBUG:boto:Data: 2023-05-31T20:29:47.708 DEBUG:boto:Headers: {} 2023-05-31T20:29:47.708 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:29:47.709 DEBUG:boto:Port: 8001 2023-05-31T20:29:47.709 DEBUG:boto:Params: {} 2023-05-31T20:29:47.709 DEBUG:boto:Token: None 2023-05-31T20:29:47.710 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:29:47 GMT /pklxkd-55/g?versionId=null 2023-05-31T20:29:47.710 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:EUQ90RtyUgav9p8DyBg2EaFCQJg= 2023-05-31T20:29:47.711 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:29:47 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:EUQ90RtyUgav9p8DyBg2EaFCQJg=', 'Content-Length': '0'} 2023-05-31T20:29:47.712 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:27:24 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000076f1a43908e465b9-006477ae3b-162e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:29:47 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:29:47.713 DEBUG:boto:path=/pklxkd-55/g 2023-05-31T20:29:47.714 DEBUG:boto:auth_path=/pklxkd-55/g 2023-05-31T20:29:47.714 DEBUG:boto:path=/pklxkd-55/g?versionId=null 2023-05-31T20:29:47.714 DEBUG:boto:auth_path=/pklxkd-55/g?versionId=null 2023-05-31T20:29:47.715 DEBUG:boto:Method: HEAD 2023-05-31T20:29:47.715 DEBUG:boto:Path: /pklxkd-55/g?versionId=null 2023-05-31T20:29:47.715 DEBUG:boto:Data: 2023-05-31T20:29:47.716 DEBUG:boto:Headers: {} 2023-05-31T20:29:47.716 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:29:47.717 DEBUG:boto:Port: 8000 2023-05-31T20:29:47.717 DEBUG:boto:Params: {} 2023-05-31T20:29:47.718 DEBUG:boto:Token: None 2023-05-31T20:29:47.719 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:29:47 GMT /pklxkd-55/g?versionId=null 2023-05-31T20:29:47.719 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:EUQ90RtyUgav9p8DyBg2EaFCQJg= 2023-05-31T20:29:47.720 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:29:47 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:EUQ90RtyUgav9p8DyBg2EaFCQJg=', 'Content-Length': '0'} 2023-05-31T20:29:47.721 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:27:24 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000013227d712e296af0-006477ae3b-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:29:47 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:29:47.722 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=g 2023-05-31T20:29:47.722 DEBUG:boto:path=/pklxkd-55/g 2023-05-31T20:29:47.723 DEBUG:boto:auth_path=/pklxkd-55/g 2023-05-31T20:29:47.723 DEBUG:boto:path=/pklxkd-55/g?versionId=null 2023-05-31T20:29:47.724 DEBUG:boto:auth_path=/pklxkd-55/g?versionId=null 2023-05-31T20:29:47.724 DEBUG:boto:Method: GET 2023-05-31T20:29:47.724 DEBUG:boto:Path: /pklxkd-55/g?versionId=null 2023-05-31T20:29:47.725 DEBUG:boto:Data: 2023-05-31T20:29:47.725 DEBUG:boto:Headers: {} 2023-05-31T20:29:47.725 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:29:47.726 DEBUG:boto:Port: 8001 2023-05-31T20:29:47.726 DEBUG:boto:Params: {} 2023-05-31T20:29:47.727 DEBUG:boto:Token: None 2023-05-31T20:29:47.728 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:29:47 GMT /pklxkd-55/g?versionId=null 2023-05-31T20:29:47.729 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:8rwhJVFIeQEuHOFXFh01O4OeEEE= 2023-05-31T20:29:47.729 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:29:47 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:8rwhJVFIeQEuHOFXFh01O4OeEEE=', 'Content-Length': '0'} 2023-05-31T20:29:47.731 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:27:24 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000194f3d665d538ebd-006477ae3b-162e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:29:47 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:29:47.772 DEBUG:boto:path=/pklxkd-55/g 2023-05-31T20:29:47.773 DEBUG:boto:auth_path=/pklxkd-55/g 2023-05-31T20:29:47.773 DEBUG:boto:path=/pklxkd-55/g?versionId=null 2023-05-31T20:29:47.774 DEBUG:boto:auth_path=/pklxkd-55/g?versionId=null 2023-05-31T20:29:47.774 DEBUG:boto:Method: GET 2023-05-31T20:29:47.775 DEBUG:boto:Path: /pklxkd-55/g?versionId=null 2023-05-31T20:29:47.775 DEBUG:boto:Data: 2023-05-31T20:29:47.775 DEBUG:boto:Headers: {} 2023-05-31T20:29:47.776 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:29:47.776 DEBUG:boto:Port: 8000 2023-05-31T20:29:47.776 DEBUG:boto:Params: {} 2023-05-31T20:29:47.777 DEBUG:boto:Token: None 2023-05-31T20:29:47.777 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:29:47 GMT /pklxkd-55/g?versionId=null 2023-05-31T20:29:47.778 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:8rwhJVFIeQEuHOFXFh01O4OeEEE= 2023-05-31T20:29:47.779 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:29:47 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:8rwhJVFIeQEuHOFXFh01O4OeEEE=', 'Content-Length': '0'} 2023-05-31T20:29:47.780 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:27:24 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000097a43b7adfeb89a4-006477ae3b-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:29:47 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:29:47.781 DEBUG:boto:path=/pklxkd-55/ 2023-05-31T20:29:47.782 DEBUG:boto:auth_path=/pklxkd-55/ 2023-05-31T20:29:47.782 DEBUG:boto:path=/pklxkd-55/?prefix=g 2023-05-31T20:29:47.782 DEBUG:boto:auth_path=/pklxkd-55/?prefix=g 2023-05-31T20:29:47.783 DEBUG:boto:Method: GET 2023-05-31T20:29:47.783 DEBUG:boto:Path: /pklxkd-55/?prefix=g 2023-05-31T20:29:47.784 DEBUG:boto:Data: 2023-05-31T20:29:47.784 DEBUG:boto:Headers: {} 2023-05-31T20:29:47.784 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:29:47.785 DEBUG:boto:Port: 8001 2023-05-31T20:29:47.785 DEBUG:boto:Params: {} 2023-05-31T20:29:47.786 DEBUG:boto:Token: None 2023-05-31T20:29:47.786 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:29:47 GMT /pklxkd-55/ 2023-05-31T20:29:47.786 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:IBdSyRVlh8qg/aH9XxJ5KZfq+k4= 2023-05-31T20:29:47.787 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:29:47 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:IBdSyRVlh8qg/aH9XxJ5KZfq+k4=', 'Content-Length': '0'} 2023-05-31T20:29:47.789 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000de6866aa36ba6168-006477ae3b-162e-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:29:47 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:29:47.790 DEBUG:boto:b'pklxkd-55g1000falseg2023-05-31T20:27:24.035Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:29:47.790 DEBUG:boto:path=/pklxkd-55/ 2023-05-31T20:29:47.791 DEBUG:boto:auth_path=/pklxkd-55/ 2023-05-31T20:29:47.791 DEBUG:boto:path=/pklxkd-55/?prefix=g 2023-05-31T20:29:47.791 DEBUG:boto:auth_path=/pklxkd-55/?prefix=g 2023-05-31T20:29:47.792 DEBUG:boto:Method: GET 2023-05-31T20:29:47.792 DEBUG:boto:Path: /pklxkd-55/?prefix=g 2023-05-31T20:29:47.793 DEBUG:boto:Data: 2023-05-31T20:29:47.793 DEBUG:boto:Headers: {} 2023-05-31T20:29:47.793 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:29:47.794 DEBUG:boto:Port: 8000 2023-05-31T20:29:47.794 DEBUG:boto:Params: {} 2023-05-31T20:29:47.795 DEBUG:boto:Token: None 2023-05-31T20:29:47.795 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:29:47 GMT /pklxkd-55/ 2023-05-31T20:29:47.796 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:IBdSyRVlh8qg/aH9XxJ5KZfq+k4= 2023-05-31T20:29:47.796 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:29:47 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:IBdSyRVlh8qg/aH9XxJ5KZfq+k4=', 'Content-Length': '0'} 2023-05-31T20:29:47.798 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e78fd15707b5c7ce-006477ae3b-1630-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:29:47 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:29:47.845 DEBUG:boto:b'pklxkd-55g1000falseg2023-05-31T20:27:24.035Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:29:47.846 DEBUG:boto:path=/pklxkd-55/g 2023-05-31T20:29:47.846 DEBUG:boto:auth_path=/pklxkd-55/g 2023-05-31T20:29:47.846 DEBUG:boto:Method: HEAD 2023-05-31T20:29:47.847 DEBUG:boto:Path: /pklxkd-55/g 2023-05-31T20:29:47.847 DEBUG:boto:Data: 2023-05-31T20:29:47.848 DEBUG:boto:Headers: {} 2023-05-31T20:29:47.848 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:29:47.849 DEBUG:boto:Port: 8001 2023-05-31T20:29:47.849 DEBUG:boto:Params: {} 2023-05-31T20:29:47.850 DEBUG:boto:Token: None 2023-05-31T20:29:47.850 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:29:47 GMT /pklxkd-55/g 2023-05-31T20:29:47.850 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:zl7xf2Qpq84VG8Y7vB8gy2ps+ZM= 2023-05-31T20:29:47.851 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:29:47 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:zl7xf2Qpq84VG8Y7vB8gy2ps+ZM=', 'Content-Length': '0'} 2023-05-31T20:29:47.853 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:27:24 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000ca190d2d3d02499-006477ae3b-162e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:29:47 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:29:47.853 DEBUG:boto:path=/pklxkd-55/g 2023-05-31T20:29:47.854 DEBUG:boto:auth_path=/pklxkd-55/g 2023-05-31T20:29:47.854 DEBUG:boto:Method: HEAD 2023-05-31T20:29:47.854 DEBUG:boto:Path: /pklxkd-55/g 2023-05-31T20:29:47.855 DEBUG:boto:Data: 2023-05-31T20:29:47.855 DEBUG:boto:Headers: {} 2023-05-31T20:29:47.855 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:29:47.856 DEBUG:boto:Port: 8000 2023-05-31T20:29:47.856 DEBUG:boto:Params: {} 2023-05-31T20:29:47.856 DEBUG:boto:Token: None 2023-05-31T20:29:47.857 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:29:47 GMT /pklxkd-55/g 2023-05-31T20:29:47.857 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:zl7xf2Qpq84VG8Y7vB8gy2ps+ZM= 2023-05-31T20:29:47.858 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:29:47 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:zl7xf2Qpq84VG8Y7vB8gy2ps+ZM=', 'Content-Length': '0'} 2023-05-31T20:29:47.859 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:27:24 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000522253004e669a7d-006477ae3b-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:29:47 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:29:47.860 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=g 2023-05-31T20:29:47.860 DEBUG:boto:path=/pklxkd-55/g 2023-05-31T20:29:47.861 DEBUG:boto:auth_path=/pklxkd-55/g 2023-05-31T20:29:47.861 DEBUG:boto:Method: GET 2023-05-31T20:29:47.861 DEBUG:boto:Path: /pklxkd-55/g 2023-05-31T20:29:47.862 DEBUG:boto:Data: 2023-05-31T20:29:47.862 DEBUG:boto:Headers: {} 2023-05-31T20:29:47.863 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:29:47.863 DEBUG:boto:Port: 8001 2023-05-31T20:29:47.863 DEBUG:boto:Params: {} 2023-05-31T20:29:47.864 DEBUG:boto:Token: None 2023-05-31T20:29:47.864 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:29:47 GMT /pklxkd-55/g 2023-05-31T20:29:47.865 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:FF/Yavj/kG8A7H7WSv3UcpRfHvg= 2023-05-31T20:29:47.865 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:29:47 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:FF/Yavj/kG8A7H7WSv3UcpRfHvg=', 'Content-Length': '0'} 2023-05-31T20:29:47.867 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:27:24 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f58be5a110f5918a-006477ae3b-162e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:29:47 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:29:47.868 DEBUG:boto:path=/pklxkd-55/g 2023-05-31T20:29:47.868 DEBUG:boto:auth_path=/pklxkd-55/g 2023-05-31T20:29:47.868 DEBUG:boto:Method: GET 2023-05-31T20:29:47.869 DEBUG:boto:Path: /pklxkd-55/g 2023-05-31T20:29:47.869 DEBUG:boto:Data: 2023-05-31T20:29:47.870 DEBUG:boto:Headers: {} 2023-05-31T20:29:47.870 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:29:47.870 DEBUG:boto:Port: 8000 2023-05-31T20:29:47.871 DEBUG:boto:Params: {} 2023-05-31T20:29:47.871 DEBUG:boto:Token: None 2023-05-31T20:29:47.872 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:29:47 GMT /pklxkd-55/g 2023-05-31T20:29:47.872 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:FF/Yavj/kG8A7H7WSv3UcpRfHvg= 2023-05-31T20:29:47.872 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:29:47 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:FF/Yavj/kG8A7H7WSv3UcpRfHvg=', 'Content-Length': '0'} 2023-05-31T20:29:47.874 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:27:24 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001c904f5c9a53e6d5-006477ae3b-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:29:47 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:29:47.875 DEBUG:boto:path=/pklxkd-55/ 2023-05-31T20:29:47.875 DEBUG:boto:auth_path=/pklxkd-55/ 2023-05-31T20:29:47.876 DEBUG:boto:path=/pklxkd-55/?prefix=g 2023-05-31T20:29:47.876 DEBUG:boto:auth_path=/pklxkd-55/?prefix=g 2023-05-31T20:29:47.877 DEBUG:boto:Method: GET 2023-05-31T20:29:47.877 DEBUG:boto:Path: /pklxkd-55/?prefix=g 2023-05-31T20:29:47.877 DEBUG:boto:Data: 2023-05-31T20:29:47.878 DEBUG:boto:Headers: {} 2023-05-31T20:29:47.878 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:29:47.879 DEBUG:boto:Port: 8001 2023-05-31T20:29:47.879 DEBUG:boto:Params: {} 2023-05-31T20:29:47.879 DEBUG:boto:Token: None 2023-05-31T20:29:47.880 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:29:47 GMT /pklxkd-55/ 2023-05-31T20:29:47.880 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:IBdSyRVlh8qg/aH9XxJ5KZfq+k4= 2023-05-31T20:29:47.881 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:29:47 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:IBdSyRVlh8qg/aH9XxJ5KZfq+k4=', 'Content-Length': '0'} 2023-05-31T20:29:47.883 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000003fe215c9a8816f37-006477ae3b-162e-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:29:47 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:29:47.883 DEBUG:boto:b'pklxkd-55g1000falseg2023-05-31T20:27:24.035Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:29:47.884 DEBUG:boto:path=/pklxkd-55/ 2023-05-31T20:29:47.884 DEBUG:boto:auth_path=/pklxkd-55/ 2023-05-31T20:29:47.885 DEBUG:boto:path=/pklxkd-55/?prefix=g 2023-05-31T20:29:47.885 DEBUG:boto:auth_path=/pklxkd-55/?prefix=g 2023-05-31T20:29:47.886 DEBUG:boto:Method: GET 2023-05-31T20:29:47.886 DEBUG:boto:Path: /pklxkd-55/?prefix=g 2023-05-31T20:29:47.886 DEBUG:boto:Data: 2023-05-31T20:29:47.887 DEBUG:boto:Headers: {} 2023-05-31T20:29:47.887 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:29:47.887 DEBUG:boto:Port: 8000 2023-05-31T20:29:47.888 DEBUG:boto:Params: {} 2023-05-31T20:29:47.888 DEBUG:boto:Token: None 2023-05-31T20:29:47.889 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:29:47 GMT /pklxkd-55/ 2023-05-31T20:29:47.889 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:IBdSyRVlh8qg/aH9XxJ5KZfq+k4= 2023-05-31T20:29:47.890 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:29:47 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:IBdSyRVlh8qg/aH9XxJ5KZfq+k4=', 'Content-Length': '0'} 2023-05-31T20:29:47.891 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f476bdc1ed70ecf8-006477ae3b-1630-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:29:47 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:29:47.892 DEBUG:boto:b'pklxkd-55g1000falseg2023-05-31T20:27:24.035Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:29:47.892 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=h 2023-05-31T20:29:47.893 DEBUG:boto:path=/pklxkd-55/h 2023-05-31T20:29:47.893 DEBUG:boto:auth_path=/pklxkd-55/h 2023-05-31T20:29:47.894 DEBUG:boto:path=/pklxkd-55/h?versionId=null 2023-05-31T20:29:47.894 DEBUG:boto:auth_path=/pklxkd-55/h?versionId=null 2023-05-31T20:29:47.894 DEBUG:boto:Method: GET 2023-05-31T20:29:47.895 DEBUG:boto:Path: /pklxkd-55/h?versionId=null 2023-05-31T20:29:47.895 DEBUG:boto:Data: 2023-05-31T20:29:47.896 DEBUG:boto:Headers: {} 2023-05-31T20:29:47.896 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:29:47.896 DEBUG:boto:Port: 8001 2023-05-31T20:29:47.897 DEBUG:boto:Params: {} 2023-05-31T20:29:47.897 DEBUG:boto:Token: None 2023-05-31T20:29:47.898 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:29:47 GMT /pklxkd-55/h?versionId=null 2023-05-31T20:29:47.898 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:FCLWQvP+WAiTc4EwDy/2WqLQaSI= 2023-05-31T20:29:47.898 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:29:47 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:FCLWQvP+WAiTc4EwDy/2WqLQaSI=', 'Content-Length': '0'} 2023-05-31T20:29:47.900 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:27:24 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000acc069d765ca9d2e-006477ae3b-162e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:29:47 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:29:47.901 DEBUG:boto:path=/pklxkd-55/h 2023-05-31T20:29:47.901 DEBUG:boto:auth_path=/pklxkd-55/h 2023-05-31T20:29:47.901 DEBUG:boto:path=/pklxkd-55/h?versionId=null 2023-05-31T20:29:47.902 DEBUG:boto:auth_path=/pklxkd-55/h?versionId=null 2023-05-31T20:29:47.902 DEBUG:boto:Method: GET 2023-05-31T20:29:47.903 DEBUG:boto:Path: /pklxkd-55/h?versionId=null 2023-05-31T20:29:47.903 DEBUG:boto:Data: 2023-05-31T20:29:47.903 DEBUG:boto:Headers: {} 2023-05-31T20:29:47.904 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:29:47.904 DEBUG:boto:Port: 8000 2023-05-31T20:29:47.904 DEBUG:boto:Params: {} 2023-05-31T20:29:47.905 DEBUG:boto:Token: None 2023-05-31T20:29:47.905 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:29:47 GMT /pklxkd-55/h?versionId=null 2023-05-31T20:29:47.906 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:FCLWQvP+WAiTc4EwDy/2WqLQaSI= 2023-05-31T20:29:47.906 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:29:47 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:FCLWQvP+WAiTc4EwDy/2WqLQaSI=', 'Content-Length': '0'} 2023-05-31T20:29:47.908 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:27:24 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000ffd5844c36724ff-006477ae3b-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:29:47 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:29:47.908 DEBUG:boto:path=/pklxkd-55/h 2023-05-31T20:29:47.909 DEBUG:boto:auth_path=/pklxkd-55/h 2023-05-31T20:29:47.909 DEBUG:boto:path=/pklxkd-55/h?versionId=null 2023-05-31T20:29:47.910 DEBUG:boto:auth_path=/pklxkd-55/h?versionId=null 2023-05-31T20:29:47.910 DEBUG:boto:Method: HEAD 2023-05-31T20:29:47.910 DEBUG:boto:Path: /pklxkd-55/h?versionId=null 2023-05-31T20:29:47.911 DEBUG:boto:Data: 2023-05-31T20:29:47.911 DEBUG:boto:Headers: {} 2023-05-31T20:29:47.911 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:29:47.912 DEBUG:boto:Port: 8001 2023-05-31T20:29:47.912 DEBUG:boto:Params: {} 2023-05-31T20:29:47.912 DEBUG:boto:Token: None 2023-05-31T20:29:47.913 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:29:47 GMT /pklxkd-55/h?versionId=null 2023-05-31T20:29:47.913 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:zHxzG1sH2HmxravJuhdLofvKy6I= 2023-05-31T20:29:47.914 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:29:47 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:zHxzG1sH2HmxravJuhdLofvKy6I=', 'Content-Length': '0'} 2023-05-31T20:29:47.915 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:27:24 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005364aace1e8225ae-006477ae3b-162e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:29:47 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:29:47.916 DEBUG:boto:path=/pklxkd-55/h 2023-05-31T20:29:47.916 DEBUG:boto:auth_path=/pklxkd-55/h 2023-05-31T20:29:47.917 DEBUG:boto:path=/pklxkd-55/h?versionId=null 2023-05-31T20:29:47.917 DEBUG:boto:auth_path=/pklxkd-55/h?versionId=null 2023-05-31T20:29:47.917 DEBUG:boto:Method: HEAD 2023-05-31T20:29:47.918 DEBUG:boto:Path: /pklxkd-55/h?versionId=null 2023-05-31T20:29:47.918 DEBUG:boto:Data: 2023-05-31T20:29:47.918 DEBUG:boto:Headers: {} 2023-05-31T20:29:47.919 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:29:47.919 DEBUG:boto:Port: 8000 2023-05-31T20:29:47.919 DEBUG:boto:Params: {} 2023-05-31T20:29:47.920 DEBUG:boto:Token: None 2023-05-31T20:29:47.920 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:29:47 GMT /pklxkd-55/h?versionId=null 2023-05-31T20:29:47.921 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:zHxzG1sH2HmxravJuhdLofvKy6I= 2023-05-31T20:29:47.921 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:29:47 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:zHxzG1sH2HmxravJuhdLofvKy6I=', 'Content-Length': '0'} 2023-05-31T20:29:47.923 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:27:24 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000838ce741a113afe9-006477ae3b-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:29:47 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:29:47.923 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=h 2023-05-31T20:29:47.924 DEBUG:boto:path=/pklxkd-55/h 2023-05-31T20:29:47.924 DEBUG:boto:auth_path=/pklxkd-55/h 2023-05-31T20:29:47.924 DEBUG:boto:path=/pklxkd-55/h?versionId=null 2023-05-31T20:29:47.925 DEBUG:boto:auth_path=/pklxkd-55/h?versionId=null 2023-05-31T20:29:47.925 DEBUG:boto:Method: GET 2023-05-31T20:29:47.925 DEBUG:boto:Path: /pklxkd-55/h?versionId=null 2023-05-31T20:29:47.926 DEBUG:boto:Data: 2023-05-31T20:29:47.926 DEBUG:boto:Headers: {} 2023-05-31T20:29:47.927 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:29:47.927 DEBUG:boto:Port: 8001 2023-05-31T20:29:47.927 DEBUG:boto:Params: {} 2023-05-31T20:29:47.928 DEBUG:boto:Token: None 2023-05-31T20:29:47.928 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:29:47 GMT /pklxkd-55/h?versionId=null 2023-05-31T20:29:47.929 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:FCLWQvP+WAiTc4EwDy/2WqLQaSI= 2023-05-31T20:29:47.929 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:29:47 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:FCLWQvP+WAiTc4EwDy/2WqLQaSI=', 'Content-Length': '0'} 2023-05-31T20:29:47.931 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:27:24 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004b23650bd4f1ad23-006477ae3b-162e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:29:47 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:29:47.931 DEBUG:boto:path=/pklxkd-55/h 2023-05-31T20:29:47.932 DEBUG:boto:auth_path=/pklxkd-55/h 2023-05-31T20:29:47.932 DEBUG:boto:path=/pklxkd-55/h?versionId=null 2023-05-31T20:29:47.932 DEBUG:boto:auth_path=/pklxkd-55/h?versionId=null 2023-05-31T20:29:47.933 DEBUG:boto:Method: GET 2023-05-31T20:29:47.933 DEBUG:boto:Path: /pklxkd-55/h?versionId=null 2023-05-31T20:29:47.934 DEBUG:boto:Data: 2023-05-31T20:29:47.934 DEBUG:boto:Headers: {} 2023-05-31T20:29:47.934 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:29:47.935 DEBUG:boto:Port: 8000 2023-05-31T20:29:47.935 DEBUG:boto:Params: {} 2023-05-31T20:29:47.935 DEBUG:boto:Token: None 2023-05-31T20:29:47.936 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:29:47 GMT /pklxkd-55/h?versionId=null 2023-05-31T20:29:47.936 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:FCLWQvP+WAiTc4EwDy/2WqLQaSI= 2023-05-31T20:29:47.937 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:29:47 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:FCLWQvP+WAiTc4EwDy/2WqLQaSI=', 'Content-Length': '0'} 2023-05-31T20:29:47.939 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:27:24 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009a4d2cd898446b8b-006477ae3b-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:29:47 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:29:47.939 DEBUG:boto:path=/pklxkd-55/ 2023-05-31T20:29:47.940 DEBUG:boto:auth_path=/pklxkd-55/ 2023-05-31T20:29:47.940 DEBUG:boto:path=/pklxkd-55/?prefix=h 2023-05-31T20:29:47.940 DEBUG:boto:auth_path=/pklxkd-55/?prefix=h 2023-05-31T20:29:47.941 DEBUG:boto:Method: GET 2023-05-31T20:29:47.941 DEBUG:boto:Path: /pklxkd-55/?prefix=h 2023-05-31T20:29:47.942 DEBUG:boto:Data: 2023-05-31T20:29:47.942 DEBUG:boto:Headers: {} 2023-05-31T20:29:47.943 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:29:47.943 DEBUG:boto:Port: 8001 2023-05-31T20:29:47.943 DEBUG:boto:Params: {} 2023-05-31T20:29:47.944 DEBUG:boto:Token: None 2023-05-31T20:29:47.944 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:29:47 GMT /pklxkd-55/ 2023-05-31T20:29:47.945 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:IBdSyRVlh8qg/aH9XxJ5KZfq+k4= 2023-05-31T20:29:47.945 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:29:47 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:IBdSyRVlh8qg/aH9XxJ5KZfq+k4=', 'Content-Length': '0'} 2023-05-31T20:29:47.947 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000064be796a973247dd-006477ae3b-162e-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:29:47 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:29:47.948 DEBUG:boto:b'pklxkd-55h1000falseh2023-05-31T20:27:24.053Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:29:47.949 DEBUG:boto:path=/pklxkd-55/ 2023-05-31T20:29:47.949 DEBUG:boto:auth_path=/pklxkd-55/ 2023-05-31T20:29:47.949 DEBUG:boto:path=/pklxkd-55/?prefix=h 2023-05-31T20:29:47.950 DEBUG:boto:auth_path=/pklxkd-55/?prefix=h 2023-05-31T20:29:47.950 DEBUG:boto:Method: GET 2023-05-31T20:29:47.951 DEBUG:boto:Path: /pklxkd-55/?prefix=h 2023-05-31T20:29:47.951 DEBUG:boto:Data: 2023-05-31T20:29:47.951 DEBUG:boto:Headers: {} 2023-05-31T20:29:47.952 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:29:47.952 DEBUG:boto:Port: 8000 2023-05-31T20:29:47.953 DEBUG:boto:Params: {} 2023-05-31T20:29:47.953 DEBUG:boto:Token: None 2023-05-31T20:29:47.953 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:29:47 GMT /pklxkd-55/ 2023-05-31T20:29:47.954 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:IBdSyRVlh8qg/aH9XxJ5KZfq+k4= 2023-05-31T20:29:47.954 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:29:47 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:IBdSyRVlh8qg/aH9XxJ5KZfq+k4=', 'Content-Length': '0'} 2023-05-31T20:29:47.956 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e092eb4798e151a8-006477ae3b-1630-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:29:47 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:29:47.956 DEBUG:boto:b'pklxkd-55h1000falseh2023-05-31T20:27:24.053Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:29:47.957 DEBUG:boto:path=/pklxkd-55/h 2023-05-31T20:29:47.958 DEBUG:boto:auth_path=/pklxkd-55/h 2023-05-31T20:29:47.958 DEBUG:boto:Method: HEAD 2023-05-31T20:29:47.958 DEBUG:boto:Path: /pklxkd-55/h 2023-05-31T20:29:47.959 DEBUG:boto:Data: 2023-05-31T20:29:47.959 DEBUG:boto:Headers: {} 2023-05-31T20:29:47.959 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:29:47.960 DEBUG:boto:Port: 8001 2023-05-31T20:29:47.960 DEBUG:boto:Params: {} 2023-05-31T20:29:47.961 DEBUG:boto:Token: None 2023-05-31T20:29:47.961 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:29:47 GMT /pklxkd-55/h 2023-05-31T20:29:47.961 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:tc504Sy/vY0l3VhFKWmlDveznig= 2023-05-31T20:29:47.962 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:29:47 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:tc504Sy/vY0l3VhFKWmlDveznig=', 'Content-Length': '0'} 2023-05-31T20:29:47.963 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:27:24 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000be98e9b5d7ce814d-006477ae3b-162e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:29:47 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:29:47.964 DEBUG:boto:path=/pklxkd-55/h 2023-05-31T20:29:47.964 DEBUG:boto:auth_path=/pklxkd-55/h 2023-05-31T20:29:47.965 DEBUG:boto:Method: HEAD 2023-05-31T20:29:47.965 DEBUG:boto:Path: /pklxkd-55/h 2023-05-31T20:29:47.966 DEBUG:boto:Data: 2023-05-31T20:29:47.966 DEBUG:boto:Headers: {} 2023-05-31T20:29:47.966 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:29:47.967 DEBUG:boto:Port: 8000 2023-05-31T20:29:47.967 DEBUG:boto:Params: {} 2023-05-31T20:29:47.967 DEBUG:boto:Token: None 2023-05-31T20:29:47.968 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:29:47 GMT /pklxkd-55/h 2023-05-31T20:29:47.968 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:tc504Sy/vY0l3VhFKWmlDveznig= 2023-05-31T20:29:47.969 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:29:47 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:tc504Sy/vY0l3VhFKWmlDveznig=', 'Content-Length': '0'} 2023-05-31T20:29:47.970 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:27:24 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f0f5b4e2e29a3bea-006477ae3b-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:29:47 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:29:47.971 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=h 2023-05-31T20:29:47.971 DEBUG:boto:path=/pklxkd-55/h 2023-05-31T20:29:47.972 DEBUG:boto:auth_path=/pklxkd-55/h 2023-05-31T20:29:47.972 DEBUG:boto:Method: GET 2023-05-31T20:29:47.972 DEBUG:boto:Path: /pklxkd-55/h 2023-05-31T20:29:47.973 DEBUG:boto:Data: 2023-05-31T20:29:47.973 DEBUG:boto:Headers: {} 2023-05-31T20:29:47.973 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:29:47.974 DEBUG:boto:Port: 8001 2023-05-31T20:29:47.974 DEBUG:boto:Params: {} 2023-05-31T20:29:47.975 DEBUG:boto:Token: None 2023-05-31T20:29:47.975 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:29:47 GMT /pklxkd-55/h 2023-05-31T20:29:47.975 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:rw0y6VdqLvJVfvH8s+6UcVlWvH8= 2023-05-31T20:29:47.976 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:29:47 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:rw0y6VdqLvJVfvH8s+6UcVlWvH8=', 'Content-Length': '0'} 2023-05-31T20:29:47.977 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:27:24 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000bc3b6175cc0c5c60-006477ae3b-162e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:29:47 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:29:47.978 DEBUG:boto:path=/pklxkd-55/h 2023-05-31T20:29:47.979 DEBUG:boto:auth_path=/pklxkd-55/h 2023-05-31T20:29:47.979 DEBUG:boto:Method: GET 2023-05-31T20:29:47.980 DEBUG:boto:Path: /pklxkd-55/h 2023-05-31T20:29:47.980 DEBUG:boto:Data: 2023-05-31T20:29:47.980 DEBUG:boto:Headers: {} 2023-05-31T20:29:47.981 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:29:47.981 DEBUG:boto:Port: 8000 2023-05-31T20:29:47.981 DEBUG:boto:Params: {} 2023-05-31T20:29:47.982 DEBUG:boto:Token: None 2023-05-31T20:29:47.982 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:29:47 GMT /pklxkd-55/h 2023-05-31T20:29:47.983 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:rw0y6VdqLvJVfvH8s+6UcVlWvH8= 2023-05-31T20:29:47.983 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:29:47 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:rw0y6VdqLvJVfvH8s+6UcVlWvH8=', 'Content-Length': '0'} 2023-05-31T20:29:47.985 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:27:24 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000bef3ab43e3161eb6-006477ae3b-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:29:47 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:29:47.986 DEBUG:boto:path=/pklxkd-55/ 2023-05-31T20:29:47.986 DEBUG:boto:auth_path=/pklxkd-55/ 2023-05-31T20:29:47.987 DEBUG:boto:path=/pklxkd-55/?prefix=h 2023-05-31T20:29:47.987 DEBUG:boto:auth_path=/pklxkd-55/?prefix=h 2023-05-31T20:29:47.987 DEBUG:boto:Method: GET 2023-05-31T20:29:47.988 DEBUG:boto:Path: /pklxkd-55/?prefix=h 2023-05-31T20:29:47.988 DEBUG:boto:Data: 2023-05-31T20:29:47.988 DEBUG:boto:Headers: {} 2023-05-31T20:29:47.989 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:29:47.989 DEBUG:boto:Port: 8001 2023-05-31T20:29:47.990 DEBUG:boto:Params: {} 2023-05-31T20:29:47.990 DEBUG:boto:Token: None 2023-05-31T20:29:47.990 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:29:47 GMT /pklxkd-55/ 2023-05-31T20:29:47.991 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:IBdSyRVlh8qg/aH9XxJ5KZfq+k4= 2023-05-31T20:29:47.991 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:29:47 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:IBdSyRVlh8qg/aH9XxJ5KZfq+k4=', 'Content-Length': '0'} 2023-05-31T20:29:47.993 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000050170fcbf3c14db7-006477ae3b-162e-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:29:47 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:29:47.994 DEBUG:boto:b'pklxkd-55h1000falseh2023-05-31T20:27:24.053Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:29:47.994 DEBUG:boto:path=/pklxkd-55/ 2023-05-31T20:29:47.995 DEBUG:boto:auth_path=/pklxkd-55/ 2023-05-31T20:29:47.996 DEBUG:boto:path=/pklxkd-55/?prefix=h 2023-05-31T20:29:47.996 DEBUG:boto:auth_path=/pklxkd-55/?prefix=h 2023-05-31T20:29:47.996 DEBUG:boto:Method: GET 2023-05-31T20:29:47.997 DEBUG:boto:Path: /pklxkd-55/?prefix=h 2023-05-31T20:29:47.997 DEBUG:boto:Data: 2023-05-31T20:29:47.997 DEBUG:boto:Headers: {} 2023-05-31T20:29:47.998 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:29:47.998 DEBUG:boto:Port: 8000 2023-05-31T20:29:47.999 DEBUG:boto:Params: {} 2023-05-31T20:29:47.999 DEBUG:boto:Token: None 2023-05-31T20:29:47.999 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:29:47 GMT /pklxkd-55/ 2023-05-31T20:29:48.000 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:IBdSyRVlh8qg/aH9XxJ5KZfq+k4= 2023-05-31T20:29:48.000 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:29:47 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:IBdSyRVlh8qg/aH9XxJ5KZfq+k4=', 'Content-Length': '0'} 2023-05-31T20:29:48.002 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000003ee255f8ff6c08c6-006477ae3b-1630-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:29:47 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:29:48.003 DEBUG:boto:b'pklxkd-55h1000falseh2023-05-31T20:27:24.053Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:29:48.003 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=pklxkd-55 zones={a2, a1} 2023-05-31T20:29:48.004 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'autotrim', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-05-31T20:29:48.004 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'autotrim', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-05-31T20:29:48.005 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog autotrim --cluster c1 --debug-rgw 0 --debug-ms 0 2023-05-31T20:29:48.051 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2023-05-31T20:29:48.051 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2023-05-31T20:29:48.102 INFO:tasks.rgw.c1.client.1.smithi099.stdout:2023-05-31T20:29:48.097+0000 7f3b0914d700 -1 req 8445471696962185172 0.000999976s :get_bucket_index_log_status ERROR: rgw_read_bucket_full_sync_status() on pipe={s={b=pklxkd-19:3d163b1e-09c7-47e2-ae87-add4aace3f38.4368.19,z=3d163b1e-09c7-47e2-ae87-add4aace3f38,az=0},d={b=pklxkd-19:3d163b1e-09c7-47e2-ae87-add4aace3f38.4368.19,z=5d8acec7-7e57-4e56-8b32-e3d6b5036001,az=0}} returned ret=-2 2023-05-31T20:29:48.103 INFO:tasks.rgw.c1.client.1.smithi099.stdout:2023-05-31T20:29:48.097+0000 7f3afd135700 -1 req 1138760661668252224 0.000000000s :get_bucket_index_log_status ERROR: rgw_read_bucket_full_sync_status() on pipe={s={b=pklxkd-43:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.8,z=3d163b1e-09c7-47e2-ae87-add4aace3f38,az=0},d={b=pklxkd-43:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.8,z=5d8acec7-7e57-4e56-8b32-e3d6b5036001,az=0}} returned ret=-2 2023-05-31T20:29:48.104 INFO:tasks.rgw.c1.client.1.smithi099.stdout:2023-05-31T20:29:48.099+0000 7f3ad78ea700 -1 req 13455067496672773949 0.000999976s :get_bucket_index_log_status ERROR: rgw_read_bucket_full_sync_status() on pipe={s={b=pklxkd-6:3d163b1e-09c7-47e2-ae87-add4aace3f38.4368.6,z=3d163b1e-09c7-47e2-ae87-add4aace3f38,az=0},d={b=pklxkd-6:3d163b1e-09c7-47e2-ae87-add4aace3f38.4368.6,z=5d8acec7-7e57-4e56-8b32-e3d6b5036001,az=0}} returned ret=-2 2023-05-31T20:29:48.109 INFO:tasks.rgw.c1.client.1.smithi099.stdout:2023-05-31T20:29:48.104+0000 7f3af6928700 -1 req 3739170360535518042 0.000000000s :get_bucket_index_log_status ERROR: rgw_read_bucket_full_sync_status() on pipe={s={b=pklxkd-36:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.1,z=3d163b1e-09c7-47e2-ae87-add4aace3f38,az=0},d={b=pklxkd-36:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.1,z=5d8acec7-7e57-4e56-8b32-e3d6b5036001,az=0}} returned ret=-2 2023-05-31T20:29:48.114 INFO:tasks.rgw.c1.client.1.smithi099.stdout:2023-05-31T20:29:48.110+0000 7f3ace8d8700 -1 req 15092341274871871355 0.000000000s :get_bucket_index_log_status ERROR: rgw_read_bucket_full_sync_status() on pipe={s={b=pklxkd-45:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.10,z=3d163b1e-09c7-47e2-ae87-add4aace3f38,az=0},d={b=pklxkd-45:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.10,z=5d8acec7-7e57-4e56-8b32-e3d6b5036001,az=0}} returned ret=-2 2023-05-31T20:29:48.150 INFO:tasks.rgw.c1.client.1.smithi099.stdout:2023-05-31T20:29:48.145+0000 7f3abf8ba700 -1 req 16689726408984271522 0.000000000s :get_bucket_index_log_status ERROR: rgw_read_bucket_full_sync_status() on pipe={s={b=pklxkd-3:3d163b1e-09c7-47e2-ae87-add4aace3f38.4368.3,z=3d163b1e-09c7-47e2-ae87-add4aace3f38,az=0},d={b=pklxkd-3:3d163b1e-09c7-47e2-ae87-add4aace3f38.4368.3,z=5d8acec7-7e57-4e56-8b32-e3d6b5036001,az=0}} returned ret=-2 2023-05-31T20:29:48.151 INFO:tasks.rgw.c1.client.1.smithi099.stdout:2023-05-31T20:29:48.147+0000 7f3ac20bf700 -1 req 15881427886642416641 0.000000000s :get_bucket_index_log_status ERROR: rgw_read_bucket_full_sync_status() on pipe={s={b=pklxkd-1:3d163b1e-09c7-47e2-ae87-add4aace3f38.4368.1,z=3d163b1e-09c7-47e2-ae87-add4aace3f38,az=0},d={b=pklxkd-1:3d163b1e-09c7-47e2-ae87-add4aace3f38.4368.1,z=5d8acec7-7e57-4e56-8b32-e3d6b5036001,az=0}} returned ret=-2 2023-05-31T20:29:48.194 INFO:tasks.rgw.c1.client.1.smithi099.stdout:2023-05-31T20:29:48.189+0000 7f3abe0b7700 -1 req 7653350138766254582 0.000000000s :get_bucket_index_log_status ERROR: rgw_read_bucket_full_sync_status() on pipe={s={b=pklxkd-37:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.2,z=3d163b1e-09c7-47e2-ae87-add4aace3f38,az=0},d={b=pklxkd-37:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.2,z=5d8acec7-7e57-4e56-8b32-e3d6b5036001,az=0}} returned ret=-2 2023-05-31T20:29:48.271 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'layout', '--bucket', 'pklxkd-55', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-05-31T20:29:48.272 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'layout', '--bucket', 'pklxkd-55', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-05-31T20:29:48.273 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket layout --bucket pklxkd-55 --cluster c1 --debug-rgw 0 --debug-ms 0 2023-05-31T20:29:48.320 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2023-05-31T20:29:48.320 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2023-05-31T20:29:48.376 INFO:teuthology.orchestra.run.smithi099.stdout:{ 2023-05-31T20:29:48.376 INFO:teuthology.orchestra.run.smithi099.stdout: "layout": { 2023-05-31T20:29:48.377 INFO:teuthology.orchestra.run.smithi099.stdout: "resharding": "None", 2023-05-31T20:29:48.377 INFO:teuthology.orchestra.run.smithi099.stdout: "current_index": { 2023-05-31T20:29:48.377 INFO:teuthology.orchestra.run.smithi099.stdout: "gen": 2, 2023-05-31T20:29:48.378 INFO:teuthology.orchestra.run.smithi099.stdout: "layout": { 2023-05-31T20:29:48.378 INFO:teuthology.orchestra.run.smithi099.stdout: "type": "Normal", 2023-05-31T20:29:48.378 INFO:teuthology.orchestra.run.smithi099.stdout: "normal": { 2023-05-31T20:29:48.378 INFO:teuthology.orchestra.run.smithi099.stdout: "num_shards": 3, 2023-05-31T20:29:48.378 INFO:teuthology.orchestra.run.smithi099.stdout: "hash_type": "Mod" 2023-05-31T20:29:48.378 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T20:29:48.378 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T20:29:48.379 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:29:48.379 INFO:teuthology.orchestra.run.smithi099.stdout: "logs": [ 2023-05-31T20:29:48.379 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T20:29:48.379 INFO:teuthology.orchestra.run.smithi099.stdout: "gen": 1, 2023-05-31T20:29:48.379 INFO:teuthology.orchestra.run.smithi099.stdout: "layout": { 2023-05-31T20:29:48.379 INFO:teuthology.orchestra.run.smithi099.stdout: "type": "InIndex", 2023-05-31T20:29:48.380 INFO:teuthology.orchestra.run.smithi099.stdout: "in_index": { 2023-05-31T20:29:48.380 INFO:teuthology.orchestra.run.smithi099.stdout: "gen": 1, 2023-05-31T20:29:48.380 INFO:teuthology.orchestra.run.smithi099.stdout: "layout": { 2023-05-31T20:29:48.380 INFO:teuthology.orchestra.run.smithi099.stdout: "num_shards": 3, 2023-05-31T20:29:48.380 INFO:teuthology.orchestra.run.smithi099.stdout: "hash_type": "Mod" 2023-05-31T20:29:48.381 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T20:29:48.381 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T20:29:48.381 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T20:29:48.381 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:29:48.381 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T20:29:48.382 INFO:teuthology.orchestra.run.smithi099.stdout: "gen": 2, 2023-05-31T20:29:48.382 INFO:teuthology.orchestra.run.smithi099.stdout: "layout": { 2023-05-31T20:29:48.382 INFO:teuthology.orchestra.run.smithi099.stdout: "type": "InIndex", 2023-05-31T20:29:48.382 INFO:teuthology.orchestra.run.smithi099.stdout: "in_index": { 2023-05-31T20:29:48.382 INFO:teuthology.orchestra.run.smithi099.stdout: "gen": 2, 2023-05-31T20:29:48.382 INFO:teuthology.orchestra.run.smithi099.stdout: "layout": { 2023-05-31T20:29:48.382 INFO:teuthology.orchestra.run.smithi099.stdout: "num_shards": 3, 2023-05-31T20:29:48.383 INFO:teuthology.orchestra.run.smithi099.stdout: "hash_type": "Mod" 2023-05-31T20:29:48.383 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T20:29:48.383 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T20:29:48.383 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T20:29:48.383 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T20:29:48.383 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2023-05-31T20:29:48.383 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T20:29:48.384 INFO:teuthology.orchestra.run.smithi099.stdout:} 2023-05-31T20:29:48.385 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'autotrim', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-05-31T20:29:48.385 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'autotrim', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-05-31T20:29:48.386 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog autotrim --cluster c1 --debug-rgw 0 --debug-ms 0 2023-05-31T20:29:48.432 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2023-05-31T20:29:48.432 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2023-05-31T20:29:48.490 INFO:tasks.rgw.c1.client.1.smithi099.stdout:2023-05-31T20:29:48.485+0000 7f3aae898700 -1 req 14616980191073695091 0.000000000s :get_bucket_index_log_status ERROR: rgw_read_bucket_full_sync_status() on pipe={s={b=pklxkd-4:3d163b1e-09c7-47e2-ae87-add4aace3f38.4368.4,z=3d163b1e-09c7-47e2-ae87-add4aace3f38,az=0},d={b=pklxkd-4:3d163b1e-09c7-47e2-ae87-add4aace3f38.4368.4,z=5d8acec7-7e57-4e56-8b32-e3d6b5036001,az=0}} returned ret=-2 2023-05-31T20:29:48.491 INFO:tasks.rgw.c1.client.1.smithi099.stdout:2023-05-31T20:29:48.485+0000 7f3aa808b700 -1 req 12861548945062890721 0.000000000s :get_bucket_index_log_status ERROR: rgw_read_bucket_full_sync_status() on pipe={s={b=pklxkd-2:3d163b1e-09c7-47e2-ae87-add4aace3f38.4368.2,z=3d163b1e-09c7-47e2-ae87-add4aace3f38,az=0},d={b=pklxkd-2:3d163b1e-09c7-47e2-ae87-add4aace3f38.4368.2,z=5d8acec7-7e57-4e56-8b32-e3d6b5036001,az=0}} returned ret=-2 2023-05-31T20:29:48.491 INFO:tasks.rgw.c1.client.1.smithi099.stdout:2023-05-31T20:29:48.485+0000 7f3ab80ab700 -1 req 2137930933412329992 0.000000000s :get_bucket_index_log_status ERROR: rgw_read_bucket_full_sync_status() on pipe={s={b=pklxkd-28:3d163b1e-09c7-47e2-ae87-add4aace3f38.4368.28,z=3d163b1e-09c7-47e2-ae87-add4aace3f38,az=0},d={b=pklxkd-28:3d163b1e-09c7-47e2-ae87-add4aace3f38.4368.28,z=5d8acec7-7e57-4e56-8b32-e3d6b5036001,az=0}} returned ret=-2 2023-05-31T20:29:48.492 INFO:tasks.rgw.c1.client.1.smithi099.stdout:2023-05-31T20:29:48.488+0000 7f3a8f85a700 -1 req 2531218361705108531 0.000999977s :get_bucket_index_log_status ERROR: rgw_read_bucket_full_sync_status() on pipe={s={b=pklxkd-40:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.5,z=3d163b1e-09c7-47e2-ae87-add4aace3f38,az=0},d={b=pklxkd-40:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.5,z=5d8acec7-7e57-4e56-8b32-e3d6b5036001,az=0}} returned ret=-2 2023-05-31T20:29:48.492 INFO:tasks.rgw.c1.client.1.smithi099.stdout:2023-05-31T20:29:48.488+0000 7f3a97069700 -1 req 4722288278995007716 0.000999977s :get_bucket_index_log_status ERROR: rgw_read_bucket_full_sync_status() on pipe={s={b=pklxkd-5:3d163b1e-09c7-47e2-ae87-add4aace3f38.4368.5,z=3d163b1e-09c7-47e2-ae87-add4aace3f38,az=0},d={b=pklxkd-5:3d163b1e-09c7-47e2-ae87-add4aace3f38.4368.5,z=5d8acec7-7e57-4e56-8b32-e3d6b5036001,az=0}} returned ret=-2 2023-05-31T20:29:48.535 INFO:tasks.rgw.c1.client.1.smithi099.stdout:2023-05-31T20:29:48.531+0000 7f3a94063700 -1 req 13620524963349987869 0.000999977s :get_bucket_index_log_status ERROR: rgw_read_bucket_full_sync_status() on pipe={s={b=pklxkd-29:3d163b1e-09c7-47e2-ae87-add4aace3f38.4368.29,z=3d163b1e-09c7-47e2-ae87-add4aace3f38,az=0},d={b=pklxkd-29:3d163b1e-09c7-47e2-ae87-add4aace3f38.4368.29,z=5d8acec7-7e57-4e56-8b32-e3d6b5036001,az=0}} returned ret=-2 2023-05-31T20:29:48.540 INFO:tasks.rgw.c1.client.1.smithi099.stdout:2023-05-31T20:29:48.535+0000 7f3b5a1ef700 -1 req 15780290087924430216 0.000000000s :get_bucket_index_log_status ERROR: rgw_read_bucket_full_sync_status() on pipe={s={b=pklxkd-30:3d163b1e-09c7-47e2-ae87-add4aace3f38.4368.30,z=3d163b1e-09c7-47e2-ae87-add4aace3f38,az=0},d={b=pklxkd-30:3d163b1e-09c7-47e2-ae87-add4aace3f38.4368.30,z=5d8acec7-7e57-4e56-8b32-e3d6b5036001,az=0}} returned ret=-2 2023-05-31T20:29:48.578 INFO:tasks.rgw.c1.client.1.smithi099.stdout:2023-05-31T20:29:48.573+0000 7f3a72820700 -1 req 15950283960328835668 0.000000000s :get_bucket_index_log_status ERROR: rgw_read_bucket_full_sync_status() on pipe={s={b=pklxkd-51:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.16,z=3d163b1e-09c7-47e2-ae87-add4aace3f38,az=0},d={b=pklxkd-51:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.16,z=5d8acec7-7e57-4e56-8b32-e3d6b5036001,az=0}} returned ret=-2 2023-05-31T20:29:48.667 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'layout', '--bucket', 'pklxkd-55', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-05-31T20:29:48.668 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'layout', '--bucket', 'pklxkd-55', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-05-31T20:29:48.669 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket layout --bucket pklxkd-55 --cluster c1 --debug-rgw 0 --debug-ms 0 2023-05-31T20:29:48.717 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2023-05-31T20:29:48.718 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2023-05-31T20:29:48.770 INFO:teuthology.orchestra.run.smithi099.stdout:{ 2023-05-31T20:29:48.770 INFO:teuthology.orchestra.run.smithi099.stdout: "layout": { 2023-05-31T20:29:48.771 INFO:teuthology.orchestra.run.smithi099.stdout: "resharding": "None", 2023-05-31T20:29:48.771 INFO:teuthology.orchestra.run.smithi099.stdout: "current_index": { 2023-05-31T20:29:48.771 INFO:teuthology.orchestra.run.smithi099.stdout: "gen": 2, 2023-05-31T20:29:48.771 INFO:teuthology.orchestra.run.smithi099.stdout: "layout": { 2023-05-31T20:29:48.771 INFO:teuthology.orchestra.run.smithi099.stdout: "type": "Normal", 2023-05-31T20:29:48.772 INFO:teuthology.orchestra.run.smithi099.stdout: "normal": { 2023-05-31T20:29:48.772 INFO:teuthology.orchestra.run.smithi099.stdout: "num_shards": 3, 2023-05-31T20:29:48.772 INFO:teuthology.orchestra.run.smithi099.stdout: "hash_type": "Mod" 2023-05-31T20:29:48.772 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T20:29:48.772 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T20:29:48.772 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:29:48.773 INFO:teuthology.orchestra.run.smithi099.stdout: "logs": [ 2023-05-31T20:29:48.773 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T20:29:48.773 INFO:teuthology.orchestra.run.smithi099.stdout: "gen": 1, 2023-05-31T20:29:48.773 INFO:teuthology.orchestra.run.smithi099.stdout: "layout": { 2023-05-31T20:29:48.773 INFO:teuthology.orchestra.run.smithi099.stdout: "type": "InIndex", 2023-05-31T20:29:48.773 INFO:teuthology.orchestra.run.smithi099.stdout: "in_index": { 2023-05-31T20:29:48.773 INFO:teuthology.orchestra.run.smithi099.stdout: "gen": 1, 2023-05-31T20:29:48.774 INFO:teuthology.orchestra.run.smithi099.stdout: "layout": { 2023-05-31T20:29:48.774 INFO:teuthology.orchestra.run.smithi099.stdout: "num_shards": 3, 2023-05-31T20:29:48.774 INFO:teuthology.orchestra.run.smithi099.stdout: "hash_type": "Mod" 2023-05-31T20:29:48.774 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T20:29:48.774 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T20:29:48.774 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T20:29:48.775 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:29:48.775 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T20:29:48.775 INFO:teuthology.orchestra.run.smithi099.stdout: "gen": 2, 2023-05-31T20:29:48.775 INFO:teuthology.orchestra.run.smithi099.stdout: "layout": { 2023-05-31T20:29:48.775 INFO:teuthology.orchestra.run.smithi099.stdout: "type": "InIndex", 2023-05-31T20:29:48.775 INFO:teuthology.orchestra.run.smithi099.stdout: "in_index": { 2023-05-31T20:29:48.776 INFO:teuthology.orchestra.run.smithi099.stdout: "gen": 2, 2023-05-31T20:29:48.776 INFO:teuthology.orchestra.run.smithi099.stdout: "layout": { 2023-05-31T20:29:48.776 INFO:teuthology.orchestra.run.smithi099.stdout: "num_shards": 3, 2023-05-31T20:29:48.776 INFO:teuthology.orchestra.run.smithi099.stdout: "hash_type": "Mod" 2023-05-31T20:29:48.776 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T20:29:48.777 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T20:29:48.777 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T20:29:48.777 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T20:29:48.777 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2023-05-31T20:29:48.777 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T20:29:48.777 INFO:teuthology.orchestra.run.smithi099.stdout:} 2023-05-31T20:29:48.781 INFO:tasks.rgw_multisite_tests:rgw_multi.tests.test_bucket_reshard_index_log_trim ... FAIL 2023-05-31T20:29:48.783 DEBUG:boto:path=/pklxkd-56/ 2023-05-31T20:29:48.784 DEBUG:boto:auth_path=/pklxkd-56/ 2023-05-31T20:29:48.784 DEBUG:boto:Method: PUT 2023-05-31T20:29:48.785 DEBUG:boto:Path: /pklxkd-56/ 2023-05-31T20:29:48.785 DEBUG:boto:Data: 2023-05-31T20:29:48.786 DEBUG:boto:Headers: {} 2023-05-31T20:29:48.787 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:29:48.787 DEBUG:boto:Port: 8000 2023-05-31T20:29:48.787 DEBUG:boto:Params: {} 2023-05-31T20:29:48.788 DEBUG:boto:Token: None 2023-05-31T20:29:48.788 DEBUG:boto:StringToSign: PUT Wed, 31 May 2023 20:29:48 GMT /pklxkd-56/ 2023-05-31T20:29:48.789 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:i335iI0+9PKUJL/fazcbmliXX1A= 2023-05-31T20:29:48.789 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:29:48 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:i335iI0+9PKUJL/fazcbmliXX1A=', 'Content-Length': '0'} 2023-05-31T20:29:48.811 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000edd0d4ed9b56db39-006477ae3c-1630-a1'), ('Content-Length', '0'), ('Date', 'Wed, 31 May 2023 20:29:48 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:29:48.812 DEBUG:rgw_multi.tests:created bucket=pklxkd-56 2023-05-31T20:29:48.813 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-31T20:29:48.813 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-31T20:29:48.813 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-05-31T20:29:48.865 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2023-05-31T20:29:48.865 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2023-05-31T20:29:48.917 INFO:teuthology.orchestra.run.smithi099.stderr:No --period given, using current period=2e6123d2-d695-4350-9a01-61a455bc782b 2023-05-31T20:29:48.940 INFO:teuthology.orchestra.run.smithi099.stdout:[ 2023-05-31T20:29:48.940 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T20:29:48.941 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1685562631.516089_59.1", 2023-05-31T20:29:48.941 INFO:teuthology.orchestra.run.smithi099.stdout: "last_update": "2023-05-31T19:50:31.516089Z" 2023-05-31T20:29:48.941 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:29:48.941 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T20:29:48.941 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "", 2023-05-31T20:29:48.941 INFO:teuthology.orchestra.run.smithi099.stdout: "last_update": "0.000000" 2023-05-31T20:29:48.942 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:29:48.942 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T20:29:48.942 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1685562970.274572_93.1", 2023-05-31T20:29:48.942 INFO:teuthology.orchestra.run.smithi099.stdout: "last_update": "2023-05-31T19:56:10.274572Z" 2023-05-31T20:29:48.943 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:29:48.943 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T20:29:48.943 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1685564988.802804_373.1", 2023-05-31T20:29:48.943 INFO:teuthology.orchestra.run.smithi099.stdout: "last_update": "2023-05-31T20:29:48.802804Z" 2023-05-31T20:29:48.943 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T20:29:48.943 INFO:teuthology.orchestra.run.smithi099.stdout:] 2023-05-31T20:29:48.944 DEBUG:rgw_multi.tests:master meta markers={0: '1_1685562631.516089_59.1', 1: '', 2: '1_1685562970.274572_93.1', 3: '1_1685564988.802804_373.1'} 2023-05-31T20:29:48.945 INFO:rgw_multi.tests:starting meta checkpoint for zone=a2 2023-05-31T20:29:48.945 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-31T20:29:48.946 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-31T20:29:48.946 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-05-31T20:29:48.996 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2023-05-31T20:29:48.997 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2023-05-31T20:29:49.037 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:29:49.033+0000 7f8af6485780 20 RGW-SYNC:meta: read sync status 2023-05-31T20:29:49.038 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:29:49.034+0000 7f8af6485780 20 RGW-SYNC:meta: read sync status 2023-05-31T20:29:49.051 INFO:teuthology.orchestra.run.smithi099.stdout:{ 2023-05-31T20:29:49.051 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_status": { 2023-05-31T20:29:49.051 INFO:teuthology.orchestra.run.smithi099.stdout: "info": { 2023-05-31T20:29:49.052 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "sync", 2023-05-31T20:29:49.052 INFO:teuthology.orchestra.run.smithi099.stdout: "num_shards": 4, 2023-05-31T20:29:49.052 INFO:teuthology.orchestra.run.smithi099.stdout: "period": "2e6123d2-d695-4350-9a01-61a455bc782b", 2023-05-31T20:29:49.052 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2023-05-31T20:29:49.052 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:29:49.052 INFO:teuthology.orchestra.run.smithi099.stdout: "markers": [ 2023-05-31T20:29:49.053 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T20:29:49.053 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 0, 2023-05-31T20:29:49.053 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T20:29:49.053 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2023-05-31T20:29:49.053 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1685562631.516089_59.1", 2023-05-31T20:29:49.053 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T20:29:49.054 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 1, 2023-05-31T20:29:49.054 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T20:29:49.054 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2023-05-31T19:50:31.516089Z", 2023-05-31T20:29:49.054 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2023-05-31T20:29:49.054 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T20:29:49.054 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:29:49.054 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T20:29:49.055 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 1, 2023-05-31T20:29:49.055 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T20:29:49.055 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2023-05-31T20:29:49.055 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "", 2023-05-31T20:29:49.055 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T20:29:49.055 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2023-05-31T20:29:49.055 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T20:29:49.056 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "0.000000", 2023-05-31T20:29:49.056 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2023-05-31T20:29:49.056 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T20:29:49.056 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:29:49.056 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T20:29:49.056 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 2, 2023-05-31T20:29:49.057 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T20:29:49.057 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2023-05-31T20:29:49.057 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1685562970.274572_93.1", 2023-05-31T20:29:49.057 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T20:29:49.057 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2023-05-31T20:29:49.057 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T20:29:49.057 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2023-05-31T19:56:10.274572Z", 2023-05-31T20:29:49.058 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2023-05-31T20:29:49.058 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T20:29:49.058 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:29:49.058 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T20:29:49.058 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 3, 2023-05-31T20:29:49.058 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T20:29:49.058 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2023-05-31T20:29:49.059 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1685564988.792989_370.1", 2023-05-31T20:29:49.059 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T20:29:49.059 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2023-05-31T20:29:49.059 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T20:29:49.059 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2023-05-31T20:29:48.792989Z", 2023-05-31T20:29:49.059 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2023-05-31T20:29:49.059 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T20:29:49.060 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T20:29:49.060 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2023-05-31T20:29:49.060 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:29:49.060 INFO:teuthology.orchestra.run.smithi099.stdout: "full_sync": { 2023-05-31T20:29:49.060 INFO:teuthology.orchestra.run.smithi099.stdout: "total": 1, 2023-05-31T20:29:49.060 INFO:teuthology.orchestra.run.smithi099.stdout: "complete": 1 2023-05-31T20:29:49.060 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:29:49.061 INFO:teuthology.orchestra.run.smithi099.stdout: "current_time": "2023-05-31T20:29:49Z" 2023-05-31T20:29:49.061 INFO:teuthology.orchestra.run.smithi099.stdout:} 2023-05-31T20:29:49.062 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "2e6123d2-d695-4350-9a01-61a455bc782b", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1685562631.516089_59.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-05-31T19:50:31.516089Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1685562970.274572_93.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-05-31T19:56:10.274572Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1685564988.792989_370.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-05-31T20:29:48.792989Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-05-31T20:29:49Z" } 2023-05-31T20:29:49.062 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1685562631.516089_59.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-05-31T19:50:31.516089Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1685562970.274572_93.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-05-31T19:56:10.274572Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1685564988.792989_370.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-05-31T20:29:48.792989Z', 'realm_epoch': 2}}] 2023-05-31T20:29:49.063 DEBUG:rgw_multi.tests:log_status={0: '1_1685562631.516089_59.1', 1: '', 2: '1_1685562970.274572_93.1', 3: '1_1685564988.802804_373.1'} 2023-05-31T20:29:49.063 DEBUG:rgw_multi.tests:sync_status={0: '1_1685562631.516089_59.1', 1: '', 2: '1_1685562970.274572_93.1', 3: '1_1685564988.792989_370.1'} 2023-05-31T20:29:49.063 WARNING:rgw_multi.tests:zone a2 behind master: shard=3 master=1_1685564988.802804_373.1 target=1_1685564988.792989_370.1 2023-05-31T20:29:54.066 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-31T20:29:54.067 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-31T20:29:54.068 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-05-31T20:29:54.126 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2023-05-31T20:29:54.126 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2023-05-31T20:29:54.162 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:29:54.158+0000 7f98e93fe780 20 RGW-SYNC:meta: read sync status 2023-05-31T20:29:54.163 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:29:54.159+0000 7f98e93fe780 20 RGW-SYNC:meta: read sync status 2023-05-31T20:29:54.177 INFO:teuthology.orchestra.run.smithi099.stdout:{ 2023-05-31T20:29:54.178 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_status": { 2023-05-31T20:29:54.178 INFO:teuthology.orchestra.run.smithi099.stdout: "info": { 2023-05-31T20:29:54.178 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "sync", 2023-05-31T20:29:54.178 INFO:teuthology.orchestra.run.smithi099.stdout: "num_shards": 4, 2023-05-31T20:29:54.179 INFO:teuthology.orchestra.run.smithi099.stdout: "period": "2e6123d2-d695-4350-9a01-61a455bc782b", 2023-05-31T20:29:54.179 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2023-05-31T20:29:54.179 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:29:54.179 INFO:teuthology.orchestra.run.smithi099.stdout: "markers": [ 2023-05-31T20:29:54.179 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T20:29:54.179 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 0, 2023-05-31T20:29:54.180 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T20:29:54.180 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2023-05-31T20:29:54.180 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1685562631.516089_59.1", 2023-05-31T20:29:54.180 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T20:29:54.180 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 1, 2023-05-31T20:29:54.180 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T20:29:54.181 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2023-05-31T19:50:31.516089Z", 2023-05-31T20:29:54.181 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2023-05-31T20:29:54.181 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T20:29:54.181 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:29:54.181 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T20:29:54.181 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 1, 2023-05-31T20:29:54.182 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T20:29:54.182 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2023-05-31T20:29:54.182 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "", 2023-05-31T20:29:54.182 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T20:29:54.182 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2023-05-31T20:29:54.182 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T20:29:54.182 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "0.000000", 2023-05-31T20:29:54.183 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2023-05-31T20:29:54.183 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T20:29:54.183 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:29:54.183 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T20:29:54.183 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 2, 2023-05-31T20:29:54.184 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T20:29:54.184 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2023-05-31T20:29:54.184 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1685562970.274572_93.1", 2023-05-31T20:29:54.185 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T20:29:54.185 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2023-05-31T20:29:54.185 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T20:29:54.185 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2023-05-31T19:56:10.274572Z", 2023-05-31T20:29:54.185 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2023-05-31T20:29:54.186 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T20:29:54.186 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:29:54.186 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T20:29:54.186 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 3, 2023-05-31T20:29:54.187 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T20:29:54.187 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2023-05-31T20:29:54.187 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1685564988.802804_373.1", 2023-05-31T20:29:54.187 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T20:29:54.187 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2023-05-31T20:29:54.187 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T20:29:54.188 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2023-05-31T20:29:48.802804Z", 2023-05-31T20:29:54.188 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2023-05-31T20:29:54.188 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T20:29:54.188 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T20:29:54.189 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2023-05-31T20:29:54.189 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:29:54.189 INFO:teuthology.orchestra.run.smithi099.stdout: "full_sync": { 2023-05-31T20:29:54.189 INFO:teuthology.orchestra.run.smithi099.stdout: "total": 1, 2023-05-31T20:29:54.189 INFO:teuthology.orchestra.run.smithi099.stdout: "complete": 1 2023-05-31T20:29:54.190 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:29:54.190 INFO:teuthology.orchestra.run.smithi099.stdout: "current_time": "2023-05-31T20:29:54Z" 2023-05-31T20:29:54.190 INFO:teuthology.orchestra.run.smithi099.stdout:} 2023-05-31T20:29:54.191 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "2e6123d2-d695-4350-9a01-61a455bc782b", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1685562631.516089_59.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-05-31T19:50:31.516089Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1685562970.274572_93.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-05-31T19:56:10.274572Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1685564988.802804_373.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-05-31T20:29:48.802804Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-05-31T20:29:54Z" } 2023-05-31T20:29:54.191 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1685562631.516089_59.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-05-31T19:50:31.516089Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1685562970.274572_93.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-05-31T19:56:10.274572Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1685564988.802804_373.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-05-31T20:29:48.802804Z', 'realm_epoch': 2}}] 2023-05-31T20:29:54.192 DEBUG:rgw_multi.tests:log_status={0: '1_1685562631.516089_59.1', 1: '', 2: '1_1685562970.274572_93.1', 3: '1_1685564988.802804_373.1'} 2023-05-31T20:29:54.192 DEBUG:rgw_multi.tests:sync_status={0: '1_1685562631.516089_59.1', 1: '', 2: '1_1685562970.274572_93.1', 3: '1_1685564988.802804_373.1'} 2023-05-31T20:29:54.193 INFO:rgw_multi.tests:finish meta checkpoint for zone=a2 2023-05-31T20:29:54.193 DEBUG:boto:path=/pklxkd-56/ 2023-05-31T20:29:54.193 DEBUG:boto:auth_path=/pklxkd-56/ 2023-05-31T20:29:54.194 DEBUG:boto:Method: HEAD 2023-05-31T20:29:54.194 DEBUG:boto:Path: /pklxkd-56/ 2023-05-31T20:29:54.195 DEBUG:boto:Data: 2023-05-31T20:29:54.195 DEBUG:boto:Headers: {} 2023-05-31T20:29:54.196 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:29:54.196 DEBUG:boto:Port: 8000 2023-05-31T20:29:54.197 DEBUG:boto:Params: {} 2023-05-31T20:29:54.197 DEBUG:boto:Token: None 2023-05-31T20:29:54.198 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:29:54 GMT /pklxkd-56/ 2023-05-31T20:29:54.198 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:RuUx9CLI7r/kQG6E0IUHVfNDU1E= 2023-05-31T20:29:54.199 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:29:54 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:RuUx9CLI7r/kQG6E0IUHVfNDU1E=', 'Content-Length': '0'} 2023-05-31T20:29:54.201 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000ee5259f107d81941-006477ae42-1630-a1'), ('Content-Length', '0'), ('Date', 'Wed, 31 May 2023 20:29:54 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:29:54.202 DEBUG:boto:path=/pklxkd-56/a 2023-05-31T20:29:54.202 DEBUG:boto:auth_path=/pklxkd-56/a 2023-05-31T20:29:54.202 DEBUG:boto:Method: PUT 2023-05-31T20:29:54.203 DEBUG:boto:Path: /pklxkd-56/a 2023-05-31T20:29:54.203 DEBUG:boto:Data: 2023-05-31T20:29:54.204 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-05-31T20:29:54.204 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:29:54.204 DEBUG:boto:Port: 8000 2023-05-31T20:29:54.205 DEBUG:boto:Params: {} 2023-05-31T20:29:54.205 DEBUG:boto:Token: None 2023-05-31T20:29:54.206 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Wed, 31 May 2023 20:29:54 GMT /pklxkd-56/a 2023-05-31T20:29:54.206 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:moWETqqDHHutRWTFJ7Pkqd3yr14= 2023-05-31T20:29:54.206 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Wed, 31 May 2023 20:29:54 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:moWETqqDHHutRWTFJ7Pkqd3yr14='} 2023-05-31T20:29:54.248 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000001ec824cf50b133b4-006477ae42-1630-a1'), ('Date', 'Wed, 31 May 2023 20:29:54 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:29:54.249 DEBUG:boto:path=/pklxkd-56/ 2023-05-31T20:29:54.249 DEBUG:boto:auth_path=/pklxkd-56/ 2023-05-31T20:29:54.250 DEBUG:boto:Method: HEAD 2023-05-31T20:29:54.250 DEBUG:boto:Path: /pklxkd-56/ 2023-05-31T20:29:54.250 DEBUG:boto:Data: 2023-05-31T20:29:54.251 DEBUG:boto:Headers: {} 2023-05-31T20:29:54.251 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:29:54.252 DEBUG:boto:Port: 8000 2023-05-31T20:29:54.252 DEBUG:boto:Params: {} 2023-05-31T20:29:54.252 DEBUG:boto:Token: None 2023-05-31T20:29:54.253 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:29:54 GMT /pklxkd-56/ 2023-05-31T20:29:54.253 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:RuUx9CLI7r/kQG6E0IUHVfNDU1E= 2023-05-31T20:29:54.254 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:29:54 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:RuUx9CLI7r/kQG6E0IUHVfNDU1E=', 'Content-Length': '0'} 2023-05-31T20:29:54.256 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '3'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000ffece23db242f892-006477ae42-1630-a1'), ('Content-Length', '0'), ('Date', 'Wed, 31 May 2023 20:29:54 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:29:54.256 DEBUG:boto:path=/pklxkd-56/b 2023-05-31T20:29:54.257 DEBUG:boto:auth_path=/pklxkd-56/b 2023-05-31T20:29:54.257 DEBUG:boto:Method: PUT 2023-05-31T20:29:54.257 DEBUG:boto:Path: /pklxkd-56/b 2023-05-31T20:29:54.258 DEBUG:boto:Data: 2023-05-31T20:29:54.258 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-05-31T20:29:54.258 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:29:54.259 DEBUG:boto:Port: 8000 2023-05-31T20:29:54.259 DEBUG:boto:Params: {} 2023-05-31T20:29:54.260 DEBUG:boto:Token: None 2023-05-31T20:29:54.260 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Wed, 31 May 2023 20:29:54 GMT /pklxkd-56/b 2023-05-31T20:29:54.260 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:YGrGl7HrBfjYrIN+Dr0UCFtKgb4= 2023-05-31T20:29:54.261 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Wed, 31 May 2023 20:29:54 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:YGrGl7HrBfjYrIN+Dr0UCFtKgb4='} 2023-05-31T20:29:54.270 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000071818c0b9a1e1a7e-006477ae42-1630-a1'), ('Date', 'Wed, 31 May 2023 20:29:54 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:29:54.271 DEBUG:boto:path=/pklxkd-56/ 2023-05-31T20:29:54.271 DEBUG:boto:auth_path=/pklxkd-56/ 2023-05-31T20:29:54.272 DEBUG:boto:Method: HEAD 2023-05-31T20:29:54.272 DEBUG:boto:Path: /pklxkd-56/ 2023-05-31T20:29:54.272 DEBUG:boto:Data: 2023-05-31T20:29:54.273 DEBUG:boto:Headers: {} 2023-05-31T20:29:54.273 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:29:54.273 DEBUG:boto:Port: 8000 2023-05-31T20:29:54.274 DEBUG:boto:Params: {} 2023-05-31T20:29:54.274 DEBUG:boto:Token: None 2023-05-31T20:29:54.275 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:29:54 GMT /pklxkd-56/ 2023-05-31T20:29:54.275 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:RuUx9CLI7r/kQG6E0IUHVfNDU1E= 2023-05-31T20:29:54.276 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:29:54 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:RuUx9CLI7r/kQG6E0IUHVfNDU1E=', 'Content-Length': '0'} 2023-05-31T20:29:54.278 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '2'), ('X-RGW-Bytes-Used', '6'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000002095dd411b8adc82-006477ae42-1630-a1'), ('Content-Length', '0'), ('Date', 'Wed, 31 May 2023 20:29:54 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:29:54.279 DEBUG:boto:path=/pklxkd-56/c 2023-05-31T20:29:54.279 DEBUG:boto:auth_path=/pklxkd-56/c 2023-05-31T20:29:54.279 DEBUG:boto:Method: PUT 2023-05-31T20:29:54.280 DEBUG:boto:Path: /pklxkd-56/c 2023-05-31T20:29:54.280 DEBUG:boto:Data: 2023-05-31T20:29:54.281 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-05-31T20:29:54.281 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:29:54.281 DEBUG:boto:Port: 8000 2023-05-31T20:29:54.282 DEBUG:boto:Params: {} 2023-05-31T20:29:54.282 DEBUG:boto:Token: None 2023-05-31T20:29:54.282 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Wed, 31 May 2023 20:29:54 GMT /pklxkd-56/c 2023-05-31T20:29:54.283 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:pPbcOC0W9cuJtyrgRs7CfftmLTI= 2023-05-31T20:29:54.283 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Wed, 31 May 2023 20:29:54 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:pPbcOC0W9cuJtyrgRs7CfftmLTI='} 2023-05-31T20:29:54.295 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000075b3fac79f5c5524-006477ae42-1630-a1'), ('Date', 'Wed, 31 May 2023 20:29:54 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:29:54.295 DEBUG:boto:path=/pklxkd-56/ 2023-05-31T20:29:54.296 DEBUG:boto:auth_path=/pklxkd-56/ 2023-05-31T20:29:54.296 DEBUG:boto:Method: HEAD 2023-05-31T20:29:54.296 DEBUG:boto:Path: /pklxkd-56/ 2023-05-31T20:29:54.297 DEBUG:boto:Data: 2023-05-31T20:29:54.297 DEBUG:boto:Headers: {} 2023-05-31T20:29:54.297 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:29:54.298 DEBUG:boto:Port: 8000 2023-05-31T20:29:54.298 DEBUG:boto:Params: {} 2023-05-31T20:29:54.299 DEBUG:boto:Token: None 2023-05-31T20:29:54.299 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:29:54 GMT /pklxkd-56/ 2023-05-31T20:29:54.300 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:RuUx9CLI7r/kQG6E0IUHVfNDU1E= 2023-05-31T20:29:54.300 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:29:54 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:RuUx9CLI7r/kQG6E0IUHVfNDU1E=', 'Content-Length': '0'} 2023-05-31T20:29:54.302 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '3'), ('X-RGW-Bytes-Used', '9'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000eda4190baa43e5b7-006477ae42-1630-a1'), ('Content-Length', '0'), ('Date', 'Wed, 31 May 2023 20:29:54 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:29:54.303 DEBUG:boto:path=/pklxkd-56/d 2023-05-31T20:29:54.303 DEBUG:boto:auth_path=/pklxkd-56/d 2023-05-31T20:29:54.304 DEBUG:boto:Method: PUT 2023-05-31T20:29:54.304 DEBUG:boto:Path: /pklxkd-56/d 2023-05-31T20:29:54.304 DEBUG:boto:Data: 2023-05-31T20:29:54.305 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-05-31T20:29:54.305 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:29:54.306 DEBUG:boto:Port: 8000 2023-05-31T20:29:54.306 DEBUG:boto:Params: {} 2023-05-31T20:29:54.306 DEBUG:boto:Token: None 2023-05-31T20:29:54.307 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Wed, 31 May 2023 20:29:54 GMT /pklxkd-56/d 2023-05-31T20:29:54.307 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:IrhiS+o+sgGF/k27dYsNQ4ZVyMU= 2023-05-31T20:29:54.308 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Wed, 31 May 2023 20:29:54 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:IrhiS+o+sgGF/k27dYsNQ4ZVyMU='} 2023-05-31T20:29:54.356 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000e31cf2bff69a2698-006477ae42-1630-a1'), ('Date', 'Wed, 31 May 2023 20:29:54 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:29:54.357 DEBUG:rgw_multi.tests:bucket checkpoint: source=a1 target=a2 bucket=pklxkd-56 2023-05-31T20:29:54.357 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'pklxkd-56', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-05-31T20:29:54.358 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'pklxkd-56', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-05-31T20:29:54.358 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket pklxkd-56 --source-zone a1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2023-05-31T20:29:54.416 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2023-05-31T20:29:54.417 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2023-05-31T20:29:54.453 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:29:54.449+0000 7f26d4b8b780 1 waiting to reach incremental sync.. 2023-05-31T20:29:59.454 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:29:59.448+0000 7f26d4b8b780 1 waiting to reach incremental sync.. 2023-05-31T20:30:04.456 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:30:04.450+0000 7f26d4b8b780 1 bucket sync caught up with source: 2023-05-31T20:30:04.456 INFO:teuthology.orchestra.run.smithi099.stderr: local status: [, 00000000001.26.6, , , , 00000000001.50.6, , , 00000000001.18.6, , 00000000001.39.6] 2023-05-31T20:30:04.456 INFO:teuthology.orchestra.run.smithi099.stderr: remote markers: [, 00000000001.26.6, , , , 00000000001.50.6, , , 00000000001.18.6, , 00000000001.39.6] 2023-05-31T20:30:04.457 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:30:04.450+0000 7f26d4b8b780 0 bucket checkpoint complete 2023-05-31T20:30:04.473 DEBUG:rgw_multi.tests:bucket checkpoint: source=a2 target=a1 bucket=pklxkd-56 2023-05-31T20:30:04.474 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'pklxkd-56', '--source-zone', 'a2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-05-31T20:30:04.474 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'pklxkd-56', '--source-zone', 'a2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-05-31T20:30:04.475 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket pklxkd-56 --source-zone a2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2023-05-31T20:30:04.538 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2023-05-31T20:30:04.539 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2023-05-31T20:30:04.584 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:30:04.579+0000 7fcd0bdc9780 1 bucket sync caught up with source: 2023-05-31T20:30:04.585 INFO:teuthology.orchestra.run.smithi099.stderr: local status: [, 00000000001.2276.6, , , , 00000000001.3568.6, , , 00000000001.355.6, , 00000000001.2011.6] 2023-05-31T20:30:04.585 INFO:teuthology.orchestra.run.smithi099.stderr: remote markers: [, 00000000001.2276.6, , , , 00000000001.3568.6, , , 00000000001.355.6, , 00000000001.2011.6] 2023-05-31T20:30:04.585 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:30:04.579+0000 7fcd0bdc9780 0 bucket checkpoint complete 2023-05-31T20:30:04.603 INFO:tasks.rgw_multi.zone_rados:comparing bucket=pklxkd-56 zones={a2, a1} 2023-05-31T20:30:04.604 DEBUG:boto:path=/pklxkd-56/ 2023-05-31T20:30:04.604 DEBUG:boto:auth_path=/pklxkd-56/ 2023-05-31T20:30:04.605 DEBUG:boto:Method: HEAD 2023-05-31T20:30:04.605 DEBUG:boto:Path: /pklxkd-56/ 2023-05-31T20:30:04.605 DEBUG:boto:Data: 2023-05-31T20:30:04.606 DEBUG:boto:Headers: {} 2023-05-31T20:30:04.606 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:30:04.606 DEBUG:boto:Port: 8001 2023-05-31T20:30:04.607 DEBUG:boto:Params: {} 2023-05-31T20:30:04.607 DEBUG:boto:Token: None 2023-05-31T20:30:04.607 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:30:04 GMT /pklxkd-56/ 2023-05-31T20:30:04.608 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:g8YhmVfCXSY8BRXJu1xTzCDzMV8= 2023-05-31T20:30:04.608 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:30:04 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:g8YhmVfCXSY8BRXJu1xTzCDzMV8=', 'Content-Length': '0'} 2023-05-31T20:30:04.611 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '4'), ('X-RGW-Bytes-Used', '12'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000aeae6eb541155faf-006477ae4c-162e-a2'), ('Content-Length', '0'), ('Date', 'Wed, 31 May 2023 20:30:04 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:30:04.611 DEBUG:boto:path=/pklxkd-56/ 2023-05-31T20:30:04.611 DEBUG:boto:auth_path=/pklxkd-56/ 2023-05-31T20:30:04.612 DEBUG:boto:Method: HEAD 2023-05-31T20:30:04.612 DEBUG:boto:Path: /pklxkd-56/ 2023-05-31T20:30:04.613 DEBUG:boto:Data: 2023-05-31T20:30:04.613 DEBUG:boto:Headers: {} 2023-05-31T20:30:04.613 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:30:04.614 DEBUG:boto:Port: 8000 2023-05-31T20:30:04.614 DEBUG:boto:Params: {} 2023-05-31T20:30:04.614 DEBUG:boto:Token: None 2023-05-31T20:30:04.615 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:30:04 GMT /pklxkd-56/ 2023-05-31T20:30:04.615 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:g8YhmVfCXSY8BRXJu1xTzCDzMV8= 2023-05-31T20:30:04.616 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:30:04 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:g8YhmVfCXSY8BRXJu1xTzCDzMV8=', 'Content-Length': '0'} 2023-05-31T20:30:04.618 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '4'), ('X-RGW-Bytes-Used', '12'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000099bf434e50bd0970-006477ae4c-1630-a1'), ('Content-Length', '0'), ('Date', 'Wed, 31 May 2023 20:30:04 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:30:04.618 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2023-05-31T20:30:04.619 DEBUG:boto:path=/pklxkd-56/ 2023-05-31T20:30:04.619 DEBUG:boto:auth_path=/pklxkd-56/ 2023-05-31T20:30:04.620 DEBUG:boto:path=/pklxkd-56/?versions&max-keys=999 2023-05-31T20:30:04.620 DEBUG:boto:auth_path=/pklxkd-56/?versions&max-keys=999 2023-05-31T20:30:04.620 DEBUG:boto:Method: GET 2023-05-31T20:30:04.621 DEBUG:boto:Path: /pklxkd-56/?versions&max-keys=999 2023-05-31T20:30:04.621 DEBUG:boto:Data: 2023-05-31T20:30:04.621 DEBUG:boto:Headers: {} 2023-05-31T20:30:04.622 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:30:04.622 DEBUG:boto:Port: 8001 2023-05-31T20:30:04.622 DEBUG:boto:Params: {} 2023-05-31T20:30:04.623 DEBUG:boto:Token: None 2023-05-31T20:30:04.623 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:30:04 GMT /pklxkd-56/?versions 2023-05-31T20:30:04.624 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:w9umJUQAGJESDVvUwdBGjlMMwlg= 2023-05-31T20:30:04.624 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:30:04 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:w9umJUQAGJESDVvUwdBGjlMMwlg=', 'Content-Length': '0'} 2023-05-31T20:30:04.626 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e5fde6b568774f03-006477ae4c-162e-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:30:04 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:30:04.626 DEBUG:boto:b'pklxkd-56999falseanulltrue2023-05-31T20:29:54.205Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalbnulltrue2023-05-31T20:29:54.259Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalcnulltrue2023-05-31T20:29:54.281Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormaldnulltrue2023-05-31T20:29:54.306Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:30:04.627 DEBUG:tasks.rgw_multi.zone_rados:o=a 2023-05-31T20:30:04.628 DEBUG:tasks.rgw_multi.zone_rados:o=b 2023-05-31T20:30:04.628 DEBUG:tasks.rgw_multi.zone_rados:o=c 2023-05-31T20:30:04.628 DEBUG:tasks.rgw_multi.zone_rados:o=d 2023-05-31T20:30:04.629 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2023-05-31T20:30:04.629 DEBUG:boto:path=/pklxkd-56/ 2023-05-31T20:30:04.629 DEBUG:boto:auth_path=/pklxkd-56/ 2023-05-31T20:30:04.630 DEBUG:boto:path=/pklxkd-56/?versions&max-keys=999 2023-05-31T20:30:04.630 DEBUG:boto:auth_path=/pklxkd-56/?versions&max-keys=999 2023-05-31T20:30:04.630 DEBUG:boto:Method: GET 2023-05-31T20:30:04.631 DEBUG:boto:Path: /pklxkd-56/?versions&max-keys=999 2023-05-31T20:30:04.631 DEBUG:boto:Data: 2023-05-31T20:30:04.631 DEBUG:boto:Headers: {} 2023-05-31T20:30:04.632 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:30:04.632 DEBUG:boto:Port: 8000 2023-05-31T20:30:04.632 DEBUG:boto:Params: {} 2023-05-31T20:30:04.633 DEBUG:boto:Token: None 2023-05-31T20:30:04.633 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:30:04 GMT /pklxkd-56/?versions 2023-05-31T20:30:04.634 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:w9umJUQAGJESDVvUwdBGjlMMwlg= 2023-05-31T20:30:04.634 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:30:04 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:w9umJUQAGJESDVvUwdBGjlMMwlg=', 'Content-Length': '0'} 2023-05-31T20:30:04.636 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000001fec9056c83ea08b-006477ae4c-1630-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:30:04 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:30:04.637 DEBUG:boto:b'pklxkd-56999falseanulltrue2023-05-31T20:29:54.205Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalbnulltrue2023-05-31T20:29:54.259Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalcnulltrue2023-05-31T20:29:54.281Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormaldnulltrue2023-05-31T20:29:54.306Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:30:04.638 DEBUG:tasks.rgw_multi.zone_rados:o=a 2023-05-31T20:30:04.638 DEBUG:tasks.rgw_multi.zone_rados:o=b 2023-05-31T20:30:04.639 DEBUG:tasks.rgw_multi.zone_rados:o=c 2023-05-31T20:30:04.639 DEBUG:tasks.rgw_multi.zone_rados:o=d 2023-05-31T20:30:04.639 DEBUG:boto:path=/pklxkd-56/ 2023-05-31T20:30:04.640 DEBUG:boto:auth_path=/pklxkd-56/ 2023-05-31T20:30:04.640 DEBUG:boto:path=/pklxkd-56/?versions&max-keys=999 2023-05-31T20:30:04.641 DEBUG:boto:auth_path=/pklxkd-56/?versions&max-keys=999 2023-05-31T20:30:04.641 DEBUG:boto:Method: GET 2023-05-31T20:30:04.641 DEBUG:boto:Path: /pklxkd-56/?versions&max-keys=999 2023-05-31T20:30:04.642 DEBUG:boto:Data: 2023-05-31T20:30:04.642 DEBUG:boto:Headers: {} 2023-05-31T20:30:04.642 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:30:04.643 DEBUG:boto:Port: 8001 2023-05-31T20:30:04.643 DEBUG:boto:Params: {} 2023-05-31T20:30:04.643 DEBUG:boto:Token: None 2023-05-31T20:30:04.644 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:30:04 GMT /pklxkd-56/?versions 2023-05-31T20:30:04.644 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:w9umJUQAGJESDVvUwdBGjlMMwlg= 2023-05-31T20:30:04.645 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:30:04 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:w9umJUQAGJESDVvUwdBGjlMMwlg=', 'Content-Length': '0'} 2023-05-31T20:30:04.647 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000dd121349271ba2ff-006477ae4c-162e-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:30:04 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:30:04.647 DEBUG:boto:b'pklxkd-56999falseanulltrue2023-05-31T20:29:54.205Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalbnulltrue2023-05-31T20:29:54.259Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalcnulltrue2023-05-31T20:29:54.281Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormaldnulltrue2023-05-31T20:29:54.306Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:30:04.648 DEBUG:boto:path=/pklxkd-56/ 2023-05-31T20:30:04.648 DEBUG:boto:auth_path=/pklxkd-56/ 2023-05-31T20:30:04.649 DEBUG:boto:path=/pklxkd-56/?versions&max-keys=999 2023-05-31T20:30:04.649 DEBUG:boto:auth_path=/pklxkd-56/?versions&max-keys=999 2023-05-31T20:30:04.649 DEBUG:boto:Method: GET 2023-05-31T20:30:04.650 DEBUG:boto:Path: /pklxkd-56/?versions&max-keys=999 2023-05-31T20:30:04.650 DEBUG:boto:Data: 2023-05-31T20:30:04.651 DEBUG:boto:Headers: {} 2023-05-31T20:30:04.651 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:30:04.651 DEBUG:boto:Port: 8000 2023-05-31T20:30:04.652 DEBUG:boto:Params: {} 2023-05-31T20:30:04.652 DEBUG:boto:Token: None 2023-05-31T20:30:04.653 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:30:04 GMT /pklxkd-56/?versions 2023-05-31T20:30:04.653 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:w9umJUQAGJESDVvUwdBGjlMMwlg= 2023-05-31T20:30:04.653 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:30:04 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:w9umJUQAGJESDVvUwdBGjlMMwlg=', 'Content-Length': '0'} 2023-05-31T20:30:04.656 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c3cf91cd519f6598-006477ae4c-1630-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:30:04 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:30:04.656 DEBUG:boto:b'pklxkd-56999falseanulltrue2023-05-31T20:29:54.205Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalbnulltrue2023-05-31T20:29:54.259Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalcnulltrue2023-05-31T20:29:54.281Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormaldnulltrue2023-05-31T20:29:54.306Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:30:04.657 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=a 2023-05-31T20:30:04.657 DEBUG:boto:path=/pklxkd-56/a 2023-05-31T20:30:04.658 DEBUG:boto:auth_path=/pklxkd-56/a 2023-05-31T20:30:04.658 DEBUG:boto:path=/pklxkd-56/a?versionId=null 2023-05-31T20:30:04.659 DEBUG:boto:auth_path=/pklxkd-56/a?versionId=null 2023-05-31T20:30:04.659 DEBUG:boto:Method: GET 2023-05-31T20:30:04.659 DEBUG:boto:Path: /pklxkd-56/a?versionId=null 2023-05-31T20:30:04.660 DEBUG:boto:Data: 2023-05-31T20:30:04.660 DEBUG:boto:Headers: {} 2023-05-31T20:30:04.660 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:30:04.661 DEBUG:boto:Port: 8001 2023-05-31T20:30:04.661 DEBUG:boto:Params: {} 2023-05-31T20:30:04.662 DEBUG:boto:Token: None 2023-05-31T20:30:04.662 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:30:04 GMT /pklxkd-56/a?versionId=null 2023-05-31T20:30:04.663 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:JA2v559a4O6qYsy+ds5YsXxcUSY= 2023-05-31T20:30:04.663 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:30:04 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:JA2v559a4O6qYsy+ds5YsXxcUSY=', 'Content-Length': '0'} 2023-05-31T20:30:04.665 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:29:54 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b26bef8fceccb3d4-006477ae4c-162e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:30:04 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:30:04.666 DEBUG:boto:path=/pklxkd-56/a 2023-05-31T20:30:04.666 DEBUG:boto:auth_path=/pklxkd-56/a 2023-05-31T20:30:04.666 DEBUG:boto:path=/pklxkd-56/a?versionId=null 2023-05-31T20:30:04.667 DEBUG:boto:auth_path=/pklxkd-56/a?versionId=null 2023-05-31T20:30:04.667 DEBUG:boto:Method: GET 2023-05-31T20:30:04.668 DEBUG:boto:Path: /pklxkd-56/a?versionId=null 2023-05-31T20:30:04.668 DEBUG:boto:Data: 2023-05-31T20:30:04.668 DEBUG:boto:Headers: {} 2023-05-31T20:30:04.669 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:30:04.669 DEBUG:boto:Port: 8000 2023-05-31T20:30:04.669 DEBUG:boto:Params: {} 2023-05-31T20:30:04.670 DEBUG:boto:Token: None 2023-05-31T20:30:04.670 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:30:04 GMT /pklxkd-56/a?versionId=null 2023-05-31T20:30:04.671 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:JA2v559a4O6qYsy+ds5YsXxcUSY= 2023-05-31T20:30:04.671 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:30:04 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:JA2v559a4O6qYsy+ds5YsXxcUSY=', 'Content-Length': '0'} 2023-05-31T20:30:04.673 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:29:54 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000453b9fb105f509c6-006477ae4c-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:30:04 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:30:04.674 DEBUG:boto:path=/pklxkd-56/a 2023-05-31T20:30:04.674 DEBUG:boto:auth_path=/pklxkd-56/a 2023-05-31T20:30:04.674 DEBUG:boto:path=/pklxkd-56/a?versionId=null 2023-05-31T20:30:04.675 DEBUG:boto:auth_path=/pklxkd-56/a?versionId=null 2023-05-31T20:30:04.675 DEBUG:boto:Method: HEAD 2023-05-31T20:30:04.675 DEBUG:boto:Path: /pklxkd-56/a?versionId=null 2023-05-31T20:30:04.676 DEBUG:boto:Data: 2023-05-31T20:30:04.676 DEBUG:boto:Headers: {} 2023-05-31T20:30:04.676 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:30:04.677 DEBUG:boto:Port: 8001 2023-05-31T20:30:04.677 DEBUG:boto:Params: {} 2023-05-31T20:30:04.677 DEBUG:boto:Token: None 2023-05-31T20:30:04.678 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:30:04 GMT /pklxkd-56/a?versionId=null 2023-05-31T20:30:04.678 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:PIYhE5lfrRdSkfxzrsZPibmtH5Q= 2023-05-31T20:30:04.679 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:30:04 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:PIYhE5lfrRdSkfxzrsZPibmtH5Q=', 'Content-Length': '0'} 2023-05-31T20:30:04.680 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:29:54 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000801f0294e74b0b6-006477ae4c-162e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:30:04 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:30:04.681 DEBUG:boto:path=/pklxkd-56/a 2023-05-31T20:30:04.681 DEBUG:boto:auth_path=/pklxkd-56/a 2023-05-31T20:30:04.681 DEBUG:boto:path=/pklxkd-56/a?versionId=null 2023-05-31T20:30:04.682 DEBUG:boto:auth_path=/pklxkd-56/a?versionId=null 2023-05-31T20:30:04.682 DEBUG:boto:Method: HEAD 2023-05-31T20:30:04.682 DEBUG:boto:Path: /pklxkd-56/a?versionId=null 2023-05-31T20:30:04.683 DEBUG:boto:Data: 2023-05-31T20:30:04.683 DEBUG:boto:Headers: {} 2023-05-31T20:30:04.684 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:30:04.684 DEBUG:boto:Port: 8000 2023-05-31T20:30:04.684 DEBUG:boto:Params: {} 2023-05-31T20:30:04.685 DEBUG:boto:Token: None 2023-05-31T20:30:04.685 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:30:04 GMT /pklxkd-56/a?versionId=null 2023-05-31T20:30:04.686 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:PIYhE5lfrRdSkfxzrsZPibmtH5Q= 2023-05-31T20:30:04.686 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:30:04 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:PIYhE5lfrRdSkfxzrsZPibmtH5Q=', 'Content-Length': '0'} 2023-05-31T20:30:04.687 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:29:54 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c49ac61f08227c1d-006477ae4c-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:30:04 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:30:04.688 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=a 2023-05-31T20:30:04.688 DEBUG:boto:path=/pklxkd-56/a 2023-05-31T20:30:04.689 DEBUG:boto:auth_path=/pklxkd-56/a 2023-05-31T20:30:04.689 DEBUG:boto:path=/pklxkd-56/a?versionId=null 2023-05-31T20:30:04.690 DEBUG:boto:auth_path=/pklxkd-56/a?versionId=null 2023-05-31T20:30:04.690 DEBUG:boto:Method: GET 2023-05-31T20:30:04.691 DEBUG:boto:Path: /pklxkd-56/a?versionId=null 2023-05-31T20:30:04.691 DEBUG:boto:Data: 2023-05-31T20:30:04.691 DEBUG:boto:Headers: {} 2023-05-31T20:30:04.692 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:30:04.692 DEBUG:boto:Port: 8001 2023-05-31T20:30:04.692 DEBUG:boto:Params: {} 2023-05-31T20:30:04.693 DEBUG:boto:Token: None 2023-05-31T20:30:04.693 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:30:04 GMT /pklxkd-56/a?versionId=null 2023-05-31T20:30:04.694 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:JA2v559a4O6qYsy+ds5YsXxcUSY= 2023-05-31T20:30:04.694 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:30:04 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:JA2v559a4O6qYsy+ds5YsXxcUSY=', 'Content-Length': '0'} 2023-05-31T20:30:04.696 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:29:54 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ef64338df95c58bc-006477ae4c-162e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:30:04 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:30:04.740 DEBUG:boto:path=/pklxkd-56/a 2023-05-31T20:30:04.741 DEBUG:boto:auth_path=/pklxkd-56/a 2023-05-31T20:30:04.741 DEBUG:boto:path=/pklxkd-56/a?versionId=null 2023-05-31T20:30:04.742 DEBUG:boto:auth_path=/pklxkd-56/a?versionId=null 2023-05-31T20:30:04.742 DEBUG:boto:Method: GET 2023-05-31T20:30:04.743 DEBUG:boto:Path: /pklxkd-56/a?versionId=null 2023-05-31T20:30:04.743 DEBUG:boto:Data: 2023-05-31T20:30:04.743 DEBUG:boto:Headers: {} 2023-05-31T20:30:04.744 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:30:04.744 DEBUG:boto:Port: 8000 2023-05-31T20:30:04.744 DEBUG:boto:Params: {} 2023-05-31T20:30:04.745 DEBUG:boto:Token: None 2023-05-31T20:30:04.745 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:30:04 GMT /pklxkd-56/a?versionId=null 2023-05-31T20:30:04.746 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:JA2v559a4O6qYsy+ds5YsXxcUSY= 2023-05-31T20:30:04.746 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:30:04 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:JA2v559a4O6qYsy+ds5YsXxcUSY=', 'Content-Length': '0'} 2023-05-31T20:30:04.750 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:29:54 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008b78ee32b2db0cb3-006477ae4c-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:30:04 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:30:04.751 DEBUG:boto:path=/pklxkd-56/ 2023-05-31T20:30:04.751 DEBUG:boto:auth_path=/pklxkd-56/ 2023-05-31T20:30:04.752 DEBUG:boto:path=/pklxkd-56/?prefix=a 2023-05-31T20:30:04.752 DEBUG:boto:auth_path=/pklxkd-56/?prefix=a 2023-05-31T20:30:04.753 DEBUG:boto:Method: GET 2023-05-31T20:30:04.753 DEBUG:boto:Path: /pklxkd-56/?prefix=a 2023-05-31T20:30:04.753 DEBUG:boto:Data: 2023-05-31T20:30:04.754 DEBUG:boto:Headers: {} 2023-05-31T20:30:04.754 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:30:04.754 DEBUG:boto:Port: 8001 2023-05-31T20:30:04.755 DEBUG:boto:Params: {} 2023-05-31T20:30:04.755 DEBUG:boto:Token: None 2023-05-31T20:30:04.756 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:30:04 GMT /pklxkd-56/ 2023-05-31T20:30:04.756 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:g3k/VaICMroRazPdg2oO7ngfDXc= 2023-05-31T20:30:04.756 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:30:04 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:g3k/VaICMroRazPdg2oO7ngfDXc=', 'Content-Length': '0'} 2023-05-31T20:30:04.759 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e012fc81fcb6f9e8-006477ae4c-162e-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:30:04 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:30:04.760 DEBUG:boto:b'pklxkd-56a1000falsea2023-05-31T20:29:54.205Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:30:04.760 DEBUG:boto:path=/pklxkd-56/ 2023-05-31T20:30:04.761 DEBUG:boto:auth_path=/pklxkd-56/ 2023-05-31T20:30:04.761 DEBUG:boto:path=/pklxkd-56/?prefix=a 2023-05-31T20:30:04.762 DEBUG:boto:auth_path=/pklxkd-56/?prefix=a 2023-05-31T20:30:04.762 DEBUG:boto:Method: GET 2023-05-31T20:30:04.762 DEBUG:boto:Path: /pklxkd-56/?prefix=a 2023-05-31T20:30:04.763 DEBUG:boto:Data: 2023-05-31T20:30:04.763 DEBUG:boto:Headers: {} 2023-05-31T20:30:04.763 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:30:04.764 DEBUG:boto:Port: 8000 2023-05-31T20:30:04.764 DEBUG:boto:Params: {} 2023-05-31T20:30:04.764 DEBUG:boto:Token: None 2023-05-31T20:30:04.765 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:30:04 GMT /pklxkd-56/ 2023-05-31T20:30:04.765 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:g3k/VaICMroRazPdg2oO7ngfDXc= 2023-05-31T20:30:04.766 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:30:04 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:g3k/VaICMroRazPdg2oO7ngfDXc=', 'Content-Length': '0'} 2023-05-31T20:30:04.768 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c3f4ccff69ad1832-006477ae4c-1630-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:30:04 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:30:04.768 DEBUG:boto:b'pklxkd-56a1000falsea2023-05-31T20:29:54.205Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:30:04.769 DEBUG:boto:path=/pklxkd-56/a 2023-05-31T20:30:04.769 DEBUG:boto:auth_path=/pklxkd-56/a 2023-05-31T20:30:04.770 DEBUG:boto:Method: HEAD 2023-05-31T20:30:04.770 DEBUG:boto:Path: /pklxkd-56/a 2023-05-31T20:30:04.770 DEBUG:boto:Data: 2023-05-31T20:30:04.771 DEBUG:boto:Headers: {} 2023-05-31T20:30:04.771 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:30:04.772 DEBUG:boto:Port: 8001 2023-05-31T20:30:04.772 DEBUG:boto:Params: {} 2023-05-31T20:30:04.773 DEBUG:boto:Token: None 2023-05-31T20:30:04.773 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:30:04 GMT /pklxkd-56/a 2023-05-31T20:30:04.774 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:a9teT4oIn+fUVpYGOqqy30R+kk0= 2023-05-31T20:30:04.774 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:30:04 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:a9teT4oIn+fUVpYGOqqy30R+kk0=', 'Content-Length': '0'} 2023-05-31T20:30:04.776 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:29:54 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c5b4be8423bff970-006477ae4c-162e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:30:04 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:30:04.776 DEBUG:boto:path=/pklxkd-56/a 2023-05-31T20:30:04.777 DEBUG:boto:auth_path=/pklxkd-56/a 2023-05-31T20:30:04.777 DEBUG:boto:Method: HEAD 2023-05-31T20:30:04.777 DEBUG:boto:Path: /pklxkd-56/a 2023-05-31T20:30:04.778 DEBUG:boto:Data: 2023-05-31T20:30:04.778 DEBUG:boto:Headers: {} 2023-05-31T20:30:04.778 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:30:04.779 DEBUG:boto:Port: 8000 2023-05-31T20:30:04.779 DEBUG:boto:Params: {} 2023-05-31T20:30:04.779 DEBUG:boto:Token: None 2023-05-31T20:30:04.780 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:30:04 GMT /pklxkd-56/a 2023-05-31T20:30:04.780 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:a9teT4oIn+fUVpYGOqqy30R+kk0= 2023-05-31T20:30:04.781 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:30:04 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:a9teT4oIn+fUVpYGOqqy30R+kk0=', 'Content-Length': '0'} 2023-05-31T20:30:04.782 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:29:54 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003be96c9f440fc55b-006477ae4c-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:30:04 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:30:04.783 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=a 2023-05-31T20:30:04.783 DEBUG:boto:path=/pklxkd-56/a 2023-05-31T20:30:04.783 DEBUG:boto:auth_path=/pklxkd-56/a 2023-05-31T20:30:04.784 DEBUG:boto:Method: GET 2023-05-31T20:30:04.784 DEBUG:boto:Path: /pklxkd-56/a 2023-05-31T20:30:04.785 DEBUG:boto:Data: 2023-05-31T20:30:04.785 DEBUG:boto:Headers: {} 2023-05-31T20:30:04.785 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:30:04.786 DEBUG:boto:Port: 8001 2023-05-31T20:30:04.786 DEBUG:boto:Params: {} 2023-05-31T20:30:04.786 DEBUG:boto:Token: None 2023-05-31T20:30:04.787 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:30:04 GMT /pklxkd-56/a 2023-05-31T20:30:04.787 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:PNlidQfePo3CBpl8P7hjnFwnU2w= 2023-05-31T20:30:04.788 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:30:04 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:PNlidQfePo3CBpl8P7hjnFwnU2w=', 'Content-Length': '0'} 2023-05-31T20:30:04.789 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:29:54 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000038c3efb42c4d6533-006477ae4c-162e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:30:04 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:30:04.790 DEBUG:boto:path=/pklxkd-56/a 2023-05-31T20:30:04.790 DEBUG:boto:auth_path=/pklxkd-56/a 2023-05-31T20:30:04.791 DEBUG:boto:Method: GET 2023-05-31T20:30:04.791 DEBUG:boto:Path: /pklxkd-56/a 2023-05-31T20:30:04.792 DEBUG:boto:Data: 2023-05-31T20:30:04.792 DEBUG:boto:Headers: {} 2023-05-31T20:30:04.792 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:30:04.793 DEBUG:boto:Port: 8000 2023-05-31T20:30:04.793 DEBUG:boto:Params: {} 2023-05-31T20:30:04.793 DEBUG:boto:Token: None 2023-05-31T20:30:04.794 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:30:04 GMT /pklxkd-56/a 2023-05-31T20:30:04.794 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:PNlidQfePo3CBpl8P7hjnFwnU2w= 2023-05-31T20:30:04.795 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:30:04 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:PNlidQfePo3CBpl8P7hjnFwnU2w=', 'Content-Length': '0'} 2023-05-31T20:30:04.796 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:29:54 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c801536dfe16c570-006477ae4c-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:30:04 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:30:04.797 DEBUG:boto:path=/pklxkd-56/ 2023-05-31T20:30:04.798 DEBUG:boto:auth_path=/pklxkd-56/ 2023-05-31T20:30:04.798 DEBUG:boto:path=/pklxkd-56/?prefix=a 2023-05-31T20:30:04.798 DEBUG:boto:auth_path=/pklxkd-56/?prefix=a 2023-05-31T20:30:04.799 DEBUG:boto:Method: GET 2023-05-31T20:30:04.799 DEBUG:boto:Path: /pklxkd-56/?prefix=a 2023-05-31T20:30:04.799 DEBUG:boto:Data: 2023-05-31T20:30:04.800 DEBUG:boto:Headers: {} 2023-05-31T20:30:04.800 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:30:04.800 DEBUG:boto:Port: 8001 2023-05-31T20:30:04.801 DEBUG:boto:Params: {} 2023-05-31T20:30:04.801 DEBUG:boto:Token: None 2023-05-31T20:30:04.802 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:30:04 GMT /pklxkd-56/ 2023-05-31T20:30:04.802 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:g3k/VaICMroRazPdg2oO7ngfDXc= 2023-05-31T20:30:04.803 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:30:04 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:g3k/VaICMroRazPdg2oO7ngfDXc=', 'Content-Length': '0'} 2023-05-31T20:30:04.805 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f6448a85c2b66b7a-006477ae4c-162e-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:30:04 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:30:04.805 DEBUG:boto:b'pklxkd-56a1000falsea2023-05-31T20:29:54.205Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:30:04.807 DEBUG:boto:path=/pklxkd-56/ 2023-05-31T20:30:04.807 DEBUG:boto:auth_path=/pklxkd-56/ 2023-05-31T20:30:04.807 DEBUG:boto:path=/pklxkd-56/?prefix=a 2023-05-31T20:30:04.808 DEBUG:boto:auth_path=/pklxkd-56/?prefix=a 2023-05-31T20:30:04.808 DEBUG:boto:Method: GET 2023-05-31T20:30:04.809 DEBUG:boto:Path: /pklxkd-56/?prefix=a 2023-05-31T20:30:04.809 DEBUG:boto:Data: 2023-05-31T20:30:04.809 DEBUG:boto:Headers: {} 2023-05-31T20:30:04.810 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:30:04.810 DEBUG:boto:Port: 8000 2023-05-31T20:30:04.810 DEBUG:boto:Params: {} 2023-05-31T20:30:04.811 DEBUG:boto:Token: None 2023-05-31T20:30:04.812 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:30:04 GMT /pklxkd-56/ 2023-05-31T20:30:04.813 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:g3k/VaICMroRazPdg2oO7ngfDXc= 2023-05-31T20:30:04.814 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:30:04 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:g3k/VaICMroRazPdg2oO7ngfDXc=', 'Content-Length': '0'} 2023-05-31T20:30:04.816 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e733aec447628ff1-006477ae4c-1630-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:30:04 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:30:04.860 DEBUG:boto:b'pklxkd-56a1000falsea2023-05-31T20:29:54.205Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:30:04.862 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=b 2023-05-31T20:30:04.863 DEBUG:boto:path=/pklxkd-56/b 2023-05-31T20:30:04.864 DEBUG:boto:auth_path=/pklxkd-56/b 2023-05-31T20:30:04.865 DEBUG:boto:path=/pklxkd-56/b?versionId=null 2023-05-31T20:30:04.865 DEBUG:boto:auth_path=/pklxkd-56/b?versionId=null 2023-05-31T20:30:04.866 DEBUG:boto:Method: GET 2023-05-31T20:30:04.867 DEBUG:boto:Path: /pklxkd-56/b?versionId=null 2023-05-31T20:30:04.868 DEBUG:boto:Data: 2023-05-31T20:30:04.869 DEBUG:boto:Headers: {} 2023-05-31T20:30:04.870 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:30:04.871 DEBUG:boto:Port: 8001 2023-05-31T20:30:04.872 DEBUG:boto:Params: {} 2023-05-31T20:30:04.873 DEBUG:boto:Token: None 2023-05-31T20:30:04.873 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:30:04 GMT /pklxkd-56/b?versionId=null 2023-05-31T20:30:04.874 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:VmVV/2Gd/E/hKKCkr3WKhy/E5rc= 2023-05-31T20:30:04.874 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:30:04 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:VmVV/2Gd/E/hKKCkr3WKhy/E5rc=', 'Content-Length': '0'} 2023-05-31T20:30:04.876 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:29:54 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003462b21a25ab828b-006477ae4c-162e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:30:04 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:30:04.877 DEBUG:boto:path=/pklxkd-56/b 2023-05-31T20:30:04.877 DEBUG:boto:auth_path=/pklxkd-56/b 2023-05-31T20:30:04.878 DEBUG:boto:path=/pklxkd-56/b?versionId=null 2023-05-31T20:30:04.878 DEBUG:boto:auth_path=/pklxkd-56/b?versionId=null 2023-05-31T20:30:04.879 DEBUG:boto:Method: GET 2023-05-31T20:30:04.879 DEBUG:boto:Path: /pklxkd-56/b?versionId=null 2023-05-31T20:30:04.879 DEBUG:boto:Data: 2023-05-31T20:30:04.880 DEBUG:boto:Headers: {} 2023-05-31T20:30:04.880 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:30:04.881 DEBUG:boto:Port: 8000 2023-05-31T20:30:04.881 DEBUG:boto:Params: {} 2023-05-31T20:30:04.882 DEBUG:boto:Token: None 2023-05-31T20:30:04.882 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:30:04 GMT /pklxkd-56/b?versionId=null 2023-05-31T20:30:04.882 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:VmVV/2Gd/E/hKKCkr3WKhy/E5rc= 2023-05-31T20:30:04.883 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:30:04 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:VmVV/2Gd/E/hKKCkr3WKhy/E5rc=', 'Content-Length': '0'} 2023-05-31T20:30:04.885 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:29:54 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001c01d76b09c4644e-006477ae4c-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:30:04 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:30:04.887 DEBUG:boto:path=/pklxkd-56/b 2023-05-31T20:30:04.887 DEBUG:boto:auth_path=/pklxkd-56/b 2023-05-31T20:30:04.888 DEBUG:boto:path=/pklxkd-56/b?versionId=null 2023-05-31T20:30:04.888 DEBUG:boto:auth_path=/pklxkd-56/b?versionId=null 2023-05-31T20:30:04.889 DEBUG:boto:Method: HEAD 2023-05-31T20:30:04.889 DEBUG:boto:Path: /pklxkd-56/b?versionId=null 2023-05-31T20:30:04.889 DEBUG:boto:Data: 2023-05-31T20:30:04.890 DEBUG:boto:Headers: {} 2023-05-31T20:30:04.891 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:30:04.892 DEBUG:boto:Port: 8001 2023-05-31T20:30:04.893 DEBUG:boto:Params: {} 2023-05-31T20:30:04.894 DEBUG:boto:Token: None 2023-05-31T20:30:04.895 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:30:04 GMT /pklxkd-56/b?versionId=null 2023-05-31T20:30:04.896 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:rxfYB4paDOGr6Ee5zQ7hQ2CjoTg= 2023-05-31T20:30:04.897 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:30:04 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:rxfYB4paDOGr6Ee5zQ7hQ2CjoTg=', 'Content-Length': '0'} 2023-05-31T20:30:04.900 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:29:54 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000fe1e20715f534c1a-006477ae4c-162e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:30:04 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:30:04.901 DEBUG:boto:path=/pklxkd-56/b 2023-05-31T20:30:04.901 DEBUG:boto:auth_path=/pklxkd-56/b 2023-05-31T20:30:04.902 DEBUG:boto:path=/pklxkd-56/b?versionId=null 2023-05-31T20:30:04.903 DEBUG:boto:auth_path=/pklxkd-56/b?versionId=null 2023-05-31T20:30:04.903 DEBUG:boto:Method: HEAD 2023-05-31T20:30:04.904 DEBUG:boto:Path: /pklxkd-56/b?versionId=null 2023-05-31T20:30:04.904 DEBUG:boto:Data: 2023-05-31T20:30:04.904 DEBUG:boto:Headers: {} 2023-05-31T20:30:04.905 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:30:04.905 DEBUG:boto:Port: 8000 2023-05-31T20:30:04.905 DEBUG:boto:Params: {} 2023-05-31T20:30:04.906 DEBUG:boto:Token: None 2023-05-31T20:30:04.906 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:30:04 GMT /pklxkd-56/b?versionId=null 2023-05-31T20:30:04.907 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:rxfYB4paDOGr6Ee5zQ7hQ2CjoTg= 2023-05-31T20:30:04.907 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:30:04 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:rxfYB4paDOGr6Ee5zQ7hQ2CjoTg=', 'Content-Length': '0'} 2023-05-31T20:30:04.910 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:29:54 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002e4db8c1a8031f39-006477ae4c-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:30:04 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:30:04.911 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=b 2023-05-31T20:30:04.911 DEBUG:boto:path=/pklxkd-56/b 2023-05-31T20:30:04.912 DEBUG:boto:auth_path=/pklxkd-56/b 2023-05-31T20:30:04.912 DEBUG:boto:path=/pklxkd-56/b?versionId=null 2023-05-31T20:30:04.913 DEBUG:boto:auth_path=/pklxkd-56/b?versionId=null 2023-05-31T20:30:04.913 DEBUG:boto:Method: GET 2023-05-31T20:30:04.913 DEBUG:boto:Path: /pklxkd-56/b?versionId=null 2023-05-31T20:30:04.914 DEBUG:boto:Data: 2023-05-31T20:30:04.914 DEBUG:boto:Headers: {} 2023-05-31T20:30:04.914 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:30:04.915 DEBUG:boto:Port: 8001 2023-05-31T20:30:04.915 DEBUG:boto:Params: {} 2023-05-31T20:30:04.915 DEBUG:boto:Token: None 2023-05-31T20:30:04.916 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:30:04 GMT /pklxkd-56/b?versionId=null 2023-05-31T20:30:04.916 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:VmVV/2Gd/E/hKKCkr3WKhy/E5rc= 2023-05-31T20:30:04.917 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:30:04 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:VmVV/2Gd/E/hKKCkr3WKhy/E5rc=', 'Content-Length': '0'} 2023-05-31T20:30:04.919 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:29:54 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f501acd42df08060-006477ae4c-162e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:30:04 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:30:04.919 DEBUG:boto:path=/pklxkd-56/b 2023-05-31T20:30:04.920 DEBUG:boto:auth_path=/pklxkd-56/b 2023-05-31T20:30:04.920 DEBUG:boto:path=/pklxkd-56/b?versionId=null 2023-05-31T20:30:04.920 DEBUG:boto:auth_path=/pklxkd-56/b?versionId=null 2023-05-31T20:30:04.921 DEBUG:boto:Method: GET 2023-05-31T20:30:04.921 DEBUG:boto:Path: /pklxkd-56/b?versionId=null 2023-05-31T20:30:04.921 DEBUG:boto:Data: 2023-05-31T20:30:04.922 DEBUG:boto:Headers: {} 2023-05-31T20:30:04.922 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:30:04.923 DEBUG:boto:Port: 8000 2023-05-31T20:30:04.923 DEBUG:boto:Params: {} 2023-05-31T20:30:04.923 DEBUG:boto:Token: None 2023-05-31T20:30:04.924 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:30:04 GMT /pklxkd-56/b?versionId=null 2023-05-31T20:30:04.924 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:VmVV/2Gd/E/hKKCkr3WKhy/E5rc= 2023-05-31T20:30:04.925 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:30:04 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:VmVV/2Gd/E/hKKCkr3WKhy/E5rc=', 'Content-Length': '0'} 2023-05-31T20:30:04.926 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:29:54 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e80c9f91895154b7-006477ae4c-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:30:04 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:30:04.927 DEBUG:boto:path=/pklxkd-56/ 2023-05-31T20:30:04.927 DEBUG:boto:auth_path=/pklxkd-56/ 2023-05-31T20:30:04.928 DEBUG:boto:path=/pklxkd-56/?prefix=b 2023-05-31T20:30:04.928 DEBUG:boto:auth_path=/pklxkd-56/?prefix=b 2023-05-31T20:30:04.928 DEBUG:boto:Method: GET 2023-05-31T20:30:04.929 DEBUG:boto:Path: /pklxkd-56/?prefix=b 2023-05-31T20:30:04.929 DEBUG:boto:Data: 2023-05-31T20:30:04.929 DEBUG:boto:Headers: {} 2023-05-31T20:30:04.930 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:30:04.930 DEBUG:boto:Port: 8001 2023-05-31T20:30:04.930 DEBUG:boto:Params: {} 2023-05-31T20:30:04.931 DEBUG:boto:Token: None 2023-05-31T20:30:04.931 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:30:04 GMT /pklxkd-56/ 2023-05-31T20:30:04.932 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:g3k/VaICMroRazPdg2oO7ngfDXc= 2023-05-31T20:30:04.932 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:30:04 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:g3k/VaICMroRazPdg2oO7ngfDXc=', 'Content-Length': '0'} 2023-05-31T20:30:04.935 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ff804816fffc73dc-006477ae4c-162e-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:30:04 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:30:04.935 DEBUG:boto:b'pklxkd-56b1000falseb2023-05-31T20:29:54.259Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:30:04.936 DEBUG:boto:path=/pklxkd-56/ 2023-05-31T20:30:04.936 DEBUG:boto:auth_path=/pklxkd-56/ 2023-05-31T20:30:04.937 DEBUG:boto:path=/pklxkd-56/?prefix=b 2023-05-31T20:30:04.937 DEBUG:boto:auth_path=/pklxkd-56/?prefix=b 2023-05-31T20:30:04.937 DEBUG:boto:Method: GET 2023-05-31T20:30:04.938 DEBUG:boto:Path: /pklxkd-56/?prefix=b 2023-05-31T20:30:04.938 DEBUG:boto:Data: 2023-05-31T20:30:04.939 DEBUG:boto:Headers: {} 2023-05-31T20:30:04.939 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:30:04.939 DEBUG:boto:Port: 8000 2023-05-31T20:30:04.940 DEBUG:boto:Params: {} 2023-05-31T20:30:04.940 DEBUG:boto:Token: None 2023-05-31T20:30:04.940 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:30:04 GMT /pklxkd-56/ 2023-05-31T20:30:04.941 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:g3k/VaICMroRazPdg2oO7ngfDXc= 2023-05-31T20:30:04.941 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:30:04 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:g3k/VaICMroRazPdg2oO7ngfDXc=', 'Content-Length': '0'} 2023-05-31T20:30:04.944 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000002a6e8b92d560feaa-006477ae4c-1630-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:30:04 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:30:04.944 DEBUG:boto:b'pklxkd-56b1000falseb2023-05-31T20:29:54.259Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:30:04.945 DEBUG:boto:path=/pklxkd-56/b 2023-05-31T20:30:04.945 DEBUG:boto:auth_path=/pklxkd-56/b 2023-05-31T20:30:04.946 DEBUG:boto:Method: HEAD 2023-05-31T20:30:04.946 DEBUG:boto:Path: /pklxkd-56/b 2023-05-31T20:30:04.946 DEBUG:boto:Data: 2023-05-31T20:30:04.947 DEBUG:boto:Headers: {} 2023-05-31T20:30:04.947 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:30:04.947 DEBUG:boto:Port: 8001 2023-05-31T20:30:04.948 DEBUG:boto:Params: {} 2023-05-31T20:30:04.948 DEBUG:boto:Token: None 2023-05-31T20:30:04.949 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:30:04 GMT /pklxkd-56/b 2023-05-31T20:30:04.949 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:t+OjSYPjgm4EknHpnVY6XWayndA= 2023-05-31T20:30:04.949 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:30:04 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:t+OjSYPjgm4EknHpnVY6XWayndA=', 'Content-Length': '0'} 2023-05-31T20:30:04.951 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:29:54 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001918120d12323a25-006477ae4c-162e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:30:04 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:30:04.952 DEBUG:boto:path=/pklxkd-56/b 2023-05-31T20:30:04.952 DEBUG:boto:auth_path=/pklxkd-56/b 2023-05-31T20:30:04.952 DEBUG:boto:Method: HEAD 2023-05-31T20:30:04.953 DEBUG:boto:Path: /pklxkd-56/b 2023-05-31T20:30:04.953 DEBUG:boto:Data: 2023-05-31T20:30:04.953 DEBUG:boto:Headers: {} 2023-05-31T20:30:04.954 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:30:04.954 DEBUG:boto:Port: 8000 2023-05-31T20:30:04.954 DEBUG:boto:Params: {} 2023-05-31T20:30:04.955 DEBUG:boto:Token: None 2023-05-31T20:30:04.955 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:30:04 GMT /pklxkd-56/b 2023-05-31T20:30:04.956 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:t+OjSYPjgm4EknHpnVY6XWayndA= 2023-05-31T20:30:04.956 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:30:04 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:t+OjSYPjgm4EknHpnVY6XWayndA=', 'Content-Length': '0'} 2023-05-31T20:30:04.958 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:29:54 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000015f60c1cc152af8f-006477ae4c-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:30:04 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:30:04.958 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=b 2023-05-31T20:30:04.959 DEBUG:boto:path=/pklxkd-56/b 2023-05-31T20:30:04.959 DEBUG:boto:auth_path=/pklxkd-56/b 2023-05-31T20:30:04.960 DEBUG:boto:Method: GET 2023-05-31T20:30:04.960 DEBUG:boto:Path: /pklxkd-56/b 2023-05-31T20:30:04.960 DEBUG:boto:Data: 2023-05-31T20:30:04.961 DEBUG:boto:Headers: {} 2023-05-31T20:30:04.961 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:30:04.961 DEBUG:boto:Port: 8001 2023-05-31T20:30:04.962 DEBUG:boto:Params: {} 2023-05-31T20:30:04.962 DEBUG:boto:Token: None 2023-05-31T20:30:04.963 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:30:04 GMT /pklxkd-56/b 2023-05-31T20:30:04.963 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:519BVVKasDZzHxqJF0IDBK5P8Hg= 2023-05-31T20:30:04.963 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:30:04 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:519BVVKasDZzHxqJF0IDBK5P8Hg=', 'Content-Length': '0'} 2023-05-31T20:30:04.965 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:29:54 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009f420e1e56541f37-006477ae4c-162e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:30:04 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:30:04.966 DEBUG:boto:path=/pklxkd-56/b 2023-05-31T20:30:04.966 DEBUG:boto:auth_path=/pklxkd-56/b 2023-05-31T20:30:04.967 DEBUG:boto:Method: GET 2023-05-31T20:30:04.967 DEBUG:boto:Path: /pklxkd-56/b 2023-05-31T20:30:04.967 DEBUG:boto:Data: 2023-05-31T20:30:04.968 DEBUG:boto:Headers: {} 2023-05-31T20:30:04.968 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:30:04.968 DEBUG:boto:Port: 8000 2023-05-31T20:30:04.969 DEBUG:boto:Params: {} 2023-05-31T20:30:04.969 DEBUG:boto:Token: None 2023-05-31T20:30:04.970 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:30:04 GMT /pklxkd-56/b 2023-05-31T20:30:04.970 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:519BVVKasDZzHxqJF0IDBK5P8Hg= 2023-05-31T20:30:04.971 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:30:04 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:519BVVKasDZzHxqJF0IDBK5P8Hg=', 'Content-Length': '0'} 2023-05-31T20:30:04.972 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:29:54 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c6e6c8fc1792695a-006477ae4c-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:30:04 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:30:04.973 DEBUG:boto:path=/pklxkd-56/ 2023-05-31T20:30:04.973 DEBUG:boto:auth_path=/pklxkd-56/ 2023-05-31T20:30:04.974 DEBUG:boto:path=/pklxkd-56/?prefix=b 2023-05-31T20:30:04.974 DEBUG:boto:auth_path=/pklxkd-56/?prefix=b 2023-05-31T20:30:04.975 DEBUG:boto:Method: GET 2023-05-31T20:30:04.975 DEBUG:boto:Path: /pklxkd-56/?prefix=b 2023-05-31T20:30:04.975 DEBUG:boto:Data: 2023-05-31T20:30:04.976 DEBUG:boto:Headers: {} 2023-05-31T20:30:04.976 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:30:04.977 DEBUG:boto:Port: 8001 2023-05-31T20:30:04.977 DEBUG:boto:Params: {} 2023-05-31T20:30:04.977 DEBUG:boto:Token: None 2023-05-31T20:30:04.978 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:30:04 GMT /pklxkd-56/ 2023-05-31T20:30:04.978 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:g3k/VaICMroRazPdg2oO7ngfDXc= 2023-05-31T20:30:04.979 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:30:04 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:g3k/VaICMroRazPdg2oO7ngfDXc=', 'Content-Length': '0'} 2023-05-31T20:30:04.981 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000086834d181269e674-006477ae4c-162e-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:30:04 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:30:04.981 DEBUG:boto:b'pklxkd-56b1000falseb2023-05-31T20:29:54.259Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:30:04.982 DEBUG:boto:path=/pklxkd-56/ 2023-05-31T20:30:04.982 DEBUG:boto:auth_path=/pklxkd-56/ 2023-05-31T20:30:04.983 DEBUG:boto:path=/pklxkd-56/?prefix=b 2023-05-31T20:30:04.983 DEBUG:boto:auth_path=/pklxkd-56/?prefix=b 2023-05-31T20:30:04.984 DEBUG:boto:Method: GET 2023-05-31T20:30:04.984 DEBUG:boto:Path: /pklxkd-56/?prefix=b 2023-05-31T20:30:04.984 DEBUG:boto:Data: 2023-05-31T20:30:04.985 DEBUG:boto:Headers: {} 2023-05-31T20:30:04.985 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:30:04.985 DEBUG:boto:Port: 8000 2023-05-31T20:30:04.986 DEBUG:boto:Params: {} 2023-05-31T20:30:04.986 DEBUG:boto:Token: None 2023-05-31T20:30:04.987 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:30:04 GMT /pklxkd-56/ 2023-05-31T20:30:04.987 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:g3k/VaICMroRazPdg2oO7ngfDXc= 2023-05-31T20:30:04.988 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:30:04 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:g3k/VaICMroRazPdg2oO7ngfDXc=', 'Content-Length': '0'} 2023-05-31T20:30:04.990 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000002fa7bfb91d99caa7-006477ae4c-1630-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:30:04 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:30:05.036 DEBUG:boto:b'pklxkd-56b1000falseb2023-05-31T20:29:54.259Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:30:05.037 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=c 2023-05-31T20:30:05.037 DEBUG:boto:path=/pklxkd-56/c 2023-05-31T20:30:05.038 DEBUG:boto:auth_path=/pklxkd-56/c 2023-05-31T20:30:05.038 DEBUG:boto:path=/pklxkd-56/c?versionId=null 2023-05-31T20:30:05.039 DEBUG:boto:auth_path=/pklxkd-56/c?versionId=null 2023-05-31T20:30:05.039 DEBUG:boto:Method: GET 2023-05-31T20:30:05.039 DEBUG:boto:Path: /pklxkd-56/c?versionId=null 2023-05-31T20:30:05.040 DEBUG:boto:Data: 2023-05-31T20:30:05.040 DEBUG:boto:Headers: {} 2023-05-31T20:30:05.040 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:30:05.041 DEBUG:boto:Port: 8001 2023-05-31T20:30:05.041 DEBUG:boto:Params: {} 2023-05-31T20:30:05.042 DEBUG:boto:Token: None 2023-05-31T20:30:05.042 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:30:05 GMT /pklxkd-56/c?versionId=null 2023-05-31T20:30:05.042 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:G4h18boT+ZEiAbs3tRaLJUvDMLk= 2023-05-31T20:30:05.043 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:30:05 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:G4h18boT+ZEiAbs3tRaLJUvDMLk=', 'Content-Length': '0'} 2023-05-31T20:30:05.044 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:29:54 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000001eadab2bbccd2b6-006477ae4d-162e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:30:05 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:30:05.045 DEBUG:boto:path=/pklxkd-56/c 2023-05-31T20:30:05.046 DEBUG:boto:auth_path=/pklxkd-56/c 2023-05-31T20:30:05.046 DEBUG:boto:path=/pklxkd-56/c?versionId=null 2023-05-31T20:30:05.046 DEBUG:boto:auth_path=/pklxkd-56/c?versionId=null 2023-05-31T20:30:05.047 DEBUG:boto:Method: GET 2023-05-31T20:30:05.047 DEBUG:boto:Path: /pklxkd-56/c?versionId=null 2023-05-31T20:30:05.047 DEBUG:boto:Data: 2023-05-31T20:30:05.048 DEBUG:boto:Headers: {} 2023-05-31T20:30:05.048 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:30:05.048 DEBUG:boto:Port: 8000 2023-05-31T20:30:05.049 DEBUG:boto:Params: {} 2023-05-31T20:30:05.049 DEBUG:boto:Token: None 2023-05-31T20:30:05.050 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:30:05 GMT /pklxkd-56/c?versionId=null 2023-05-31T20:30:05.050 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:G4h18boT+ZEiAbs3tRaLJUvDMLk= 2023-05-31T20:30:05.051 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:30:05 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:G4h18boT+ZEiAbs3tRaLJUvDMLk=', 'Content-Length': '0'} 2023-05-31T20:30:05.052 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:29:54 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000544cea61ba2d4365-006477ae4d-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:30:05 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:30:05.053 DEBUG:boto:path=/pklxkd-56/c 2023-05-31T20:30:05.053 DEBUG:boto:auth_path=/pklxkd-56/c 2023-05-31T20:30:05.054 DEBUG:boto:path=/pklxkd-56/c?versionId=null 2023-05-31T20:30:05.054 DEBUG:boto:auth_path=/pklxkd-56/c?versionId=null 2023-05-31T20:30:05.055 DEBUG:boto:Method: HEAD 2023-05-31T20:30:05.055 DEBUG:boto:Path: /pklxkd-56/c?versionId=null 2023-05-31T20:30:05.055 DEBUG:boto:Data: 2023-05-31T20:30:05.056 DEBUG:boto:Headers: {} 2023-05-31T20:30:05.056 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:30:05.056 DEBUG:boto:Port: 8001 2023-05-31T20:30:05.057 DEBUG:boto:Params: {} 2023-05-31T20:30:05.057 DEBUG:boto:Token: None 2023-05-31T20:30:05.057 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:30:05 GMT /pklxkd-56/c?versionId=null 2023-05-31T20:30:05.058 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:CG8AnUbaSNJeZaESppEzP3t0GSg= 2023-05-31T20:30:05.058 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:30:05 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:CG8AnUbaSNJeZaESppEzP3t0GSg=', 'Content-Length': '0'} 2023-05-31T20:30:05.060 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:29:54 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000eec93e9c8139f5ce-006477ae4d-162e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:30:05 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:30:05.060 DEBUG:boto:path=/pklxkd-56/c 2023-05-31T20:30:05.061 DEBUG:boto:auth_path=/pklxkd-56/c 2023-05-31T20:30:05.061 DEBUG:boto:path=/pklxkd-56/c?versionId=null 2023-05-31T20:30:05.062 DEBUG:boto:auth_path=/pklxkd-56/c?versionId=null 2023-05-31T20:30:05.062 DEBUG:boto:Method: HEAD 2023-05-31T20:30:05.062 DEBUG:boto:Path: /pklxkd-56/c?versionId=null 2023-05-31T20:30:05.063 DEBUG:boto:Data: 2023-05-31T20:30:05.063 DEBUG:boto:Headers: {} 2023-05-31T20:30:05.063 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:30:05.064 DEBUG:boto:Port: 8000 2023-05-31T20:30:05.064 DEBUG:boto:Params: {} 2023-05-31T20:30:05.064 DEBUG:boto:Token: None 2023-05-31T20:30:05.065 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:30:05 GMT /pklxkd-56/c?versionId=null 2023-05-31T20:30:05.065 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:CG8AnUbaSNJeZaESppEzP3t0GSg= 2023-05-31T20:30:05.066 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:30:05 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:CG8AnUbaSNJeZaESppEzP3t0GSg=', 'Content-Length': '0'} 2023-05-31T20:30:05.067 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:29:54 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003b5b1e7b48f0c261-006477ae4d-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:30:05 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:30:05.068 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=c 2023-05-31T20:30:05.069 DEBUG:boto:path=/pklxkd-56/c 2023-05-31T20:30:05.070 DEBUG:boto:auth_path=/pklxkd-56/c 2023-05-31T20:30:05.070 DEBUG:boto:path=/pklxkd-56/c?versionId=null 2023-05-31T20:30:05.071 DEBUG:boto:auth_path=/pklxkd-56/c?versionId=null 2023-05-31T20:30:05.072 DEBUG:boto:Method: GET 2023-05-31T20:30:05.073 DEBUG:boto:Path: /pklxkd-56/c?versionId=null 2023-05-31T20:30:05.073 DEBUG:boto:Data: 2023-05-31T20:30:05.074 DEBUG:boto:Headers: {} 2023-05-31T20:30:05.075 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:30:05.075 DEBUG:boto:Port: 8001 2023-05-31T20:30:05.076 DEBUG:boto:Params: {} 2023-05-31T20:30:05.076 DEBUG:boto:Token: None 2023-05-31T20:30:05.077 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:30:05 GMT /pklxkd-56/c?versionId=null 2023-05-31T20:30:05.077 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:G4h18boT+ZEiAbs3tRaLJUvDMLk= 2023-05-31T20:30:05.078 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:30:05 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:G4h18boT+ZEiAbs3tRaLJUvDMLk=', 'Content-Length': '0'} 2023-05-31T20:30:05.079 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:29:54 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009114cd4b15131909-006477ae4d-162e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:30:05 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:30:05.120 DEBUG:boto:path=/pklxkd-56/c 2023-05-31T20:30:05.121 DEBUG:boto:auth_path=/pklxkd-56/c 2023-05-31T20:30:05.122 DEBUG:boto:path=/pklxkd-56/c?versionId=null 2023-05-31T20:30:05.123 DEBUG:boto:auth_path=/pklxkd-56/c?versionId=null 2023-05-31T20:30:05.124 DEBUG:boto:Method: GET 2023-05-31T20:30:05.124 DEBUG:boto:Path: /pklxkd-56/c?versionId=null 2023-05-31T20:30:05.125 DEBUG:boto:Data: 2023-05-31T20:30:05.126 DEBUG:boto:Headers: {} 2023-05-31T20:30:05.127 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:30:05.127 DEBUG:boto:Port: 8000 2023-05-31T20:30:05.127 DEBUG:boto:Params: {} 2023-05-31T20:30:05.128 DEBUG:boto:Token: None 2023-05-31T20:30:05.129 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:30:05 GMT /pklxkd-56/c?versionId=null 2023-05-31T20:30:05.130 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:G4h18boT+ZEiAbs3tRaLJUvDMLk= 2023-05-31T20:30:05.131 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:30:05 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:G4h18boT+ZEiAbs3tRaLJUvDMLk=', 'Content-Length': '0'} 2023-05-31T20:30:05.133 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:29:54 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001f6dd65a4a70a78c-006477ae4d-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:30:05 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:30:05.134 DEBUG:boto:path=/pklxkd-56/ 2023-05-31T20:30:05.135 DEBUG:boto:auth_path=/pklxkd-56/ 2023-05-31T20:30:05.135 DEBUG:boto:path=/pklxkd-56/?prefix=c 2023-05-31T20:30:05.136 DEBUG:boto:auth_path=/pklxkd-56/?prefix=c 2023-05-31T20:30:05.137 DEBUG:boto:Method: GET 2023-05-31T20:30:05.137 DEBUG:boto:Path: /pklxkd-56/?prefix=c 2023-05-31T20:30:05.138 DEBUG:boto:Data: 2023-05-31T20:30:05.139 DEBUG:boto:Headers: {} 2023-05-31T20:30:05.139 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:30:05.140 DEBUG:boto:Port: 8001 2023-05-31T20:30:05.140 DEBUG:boto:Params: {} 2023-05-31T20:30:05.141 DEBUG:boto:Token: None 2023-05-31T20:30:05.141 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:30:05 GMT /pklxkd-56/ 2023-05-31T20:30:05.142 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:J7Nnzn6qi5C46akVblD05HEWcKQ= 2023-05-31T20:30:05.143 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:30:05 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:J7Nnzn6qi5C46akVblD05HEWcKQ=', 'Content-Length': '0'} 2023-05-31T20:30:05.146 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000005b5b394896b04e68-006477ae4d-162e-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:30:05 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:30:05.147 DEBUG:boto:b'pklxkd-56c1000falsec2023-05-31T20:29:54.281Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:30:05.148 DEBUG:boto:path=/pklxkd-56/ 2023-05-31T20:30:05.149 DEBUG:boto:auth_path=/pklxkd-56/ 2023-05-31T20:30:05.150 DEBUG:boto:path=/pklxkd-56/?prefix=c 2023-05-31T20:30:05.150 DEBUG:boto:auth_path=/pklxkd-56/?prefix=c 2023-05-31T20:30:05.151 DEBUG:boto:Method: GET 2023-05-31T20:30:05.151 DEBUG:boto:Path: /pklxkd-56/?prefix=c 2023-05-31T20:30:05.152 DEBUG:boto:Data: 2023-05-31T20:30:05.153 DEBUG:boto:Headers: {} 2023-05-31T20:30:05.154 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:30:05.154 DEBUG:boto:Port: 8000 2023-05-31T20:30:05.155 DEBUG:boto:Params: {} 2023-05-31T20:30:05.155 DEBUG:boto:Token: None 2023-05-31T20:30:05.156 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:30:05 GMT /pklxkd-56/ 2023-05-31T20:30:05.156 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:J7Nnzn6qi5C46akVblD05HEWcKQ= 2023-05-31T20:30:05.157 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:30:05 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:J7Nnzn6qi5C46akVblD05HEWcKQ=', 'Content-Length': '0'} 2023-05-31T20:30:05.159 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000db9900294552ba3c-006477ae4d-1630-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:30:05 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:30:05.200 DEBUG:boto:b'pklxkd-56c1000falsec2023-05-31T20:29:54.281Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:30:05.202 DEBUG:boto:path=/pklxkd-56/c 2023-05-31T20:30:05.203 DEBUG:boto:auth_path=/pklxkd-56/c 2023-05-31T20:30:05.203 DEBUG:boto:Method: HEAD 2023-05-31T20:30:05.203 DEBUG:boto:Path: /pklxkd-56/c 2023-05-31T20:30:05.204 DEBUG:boto:Data: 2023-05-31T20:30:05.204 DEBUG:boto:Headers: {} 2023-05-31T20:30:05.204 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:30:05.205 DEBUG:boto:Port: 8001 2023-05-31T20:30:05.205 DEBUG:boto:Params: {} 2023-05-31T20:30:05.206 DEBUG:boto:Token: None 2023-05-31T20:30:05.206 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:30:05 GMT /pklxkd-56/c 2023-05-31T20:30:05.207 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:PfJY/82cQqSPwJzf95LAEmy6U3Y= 2023-05-31T20:30:05.208 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:30:05 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:PfJY/82cQqSPwJzf95LAEmy6U3Y=', 'Content-Length': '0'} 2023-05-31T20:30:05.210 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:29:54 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002ac90ccbbfaa8481-006477ae4d-162e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:30:05 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:30:05.211 DEBUG:boto:path=/pklxkd-56/c 2023-05-31T20:30:05.211 DEBUG:boto:auth_path=/pklxkd-56/c 2023-05-31T20:30:05.212 DEBUG:boto:Method: HEAD 2023-05-31T20:30:05.212 DEBUG:boto:Path: /pklxkd-56/c 2023-05-31T20:30:05.213 DEBUG:boto:Data: 2023-05-31T20:30:05.213 DEBUG:boto:Headers: {} 2023-05-31T20:30:05.214 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:30:05.215 DEBUG:boto:Port: 8000 2023-05-31T20:30:05.215 DEBUG:boto:Params: {} 2023-05-31T20:30:05.215 DEBUG:boto:Token: None 2023-05-31T20:30:05.216 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:30:05 GMT /pklxkd-56/c 2023-05-31T20:30:05.216 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:PfJY/82cQqSPwJzf95LAEmy6U3Y= 2023-05-31T20:30:05.217 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:30:05 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:PfJY/82cQqSPwJzf95LAEmy6U3Y=', 'Content-Length': '0'} 2023-05-31T20:30:05.220 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:29:54 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003506c65fcb793170-006477ae4d-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:30:05 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:30:05.221 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=c 2023-05-31T20:30:05.222 DEBUG:boto:path=/pklxkd-56/c 2023-05-31T20:30:05.223 DEBUG:boto:auth_path=/pklxkd-56/c 2023-05-31T20:30:05.224 DEBUG:boto:Method: GET 2023-05-31T20:30:05.225 DEBUG:boto:Path: /pklxkd-56/c 2023-05-31T20:30:05.226 DEBUG:boto:Data: 2023-05-31T20:30:05.227 DEBUG:boto:Headers: {} 2023-05-31T20:30:05.227 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:30:05.228 DEBUG:boto:Port: 8001 2023-05-31T20:30:05.229 DEBUG:boto:Params: {} 2023-05-31T20:30:05.230 DEBUG:boto:Token: None 2023-05-31T20:30:05.232 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:30:05 GMT /pklxkd-56/c 2023-05-31T20:30:05.233 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:0wzVkEEfxRSiuHDevp5/JfKrCQA= 2023-05-31T20:30:05.234 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:30:05 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:0wzVkEEfxRSiuHDevp5/JfKrCQA=', 'Content-Length': '0'} 2023-05-31T20:30:05.236 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:29:54 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c560f3b22abde266-006477ae4d-162e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:30:05 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:30:05.237 DEBUG:boto:path=/pklxkd-56/c 2023-05-31T20:30:05.238 DEBUG:boto:auth_path=/pklxkd-56/c 2023-05-31T20:30:05.239 DEBUG:boto:Method: GET 2023-05-31T20:30:05.241 DEBUG:boto:Path: /pklxkd-56/c 2023-05-31T20:30:05.242 DEBUG:boto:Data: 2023-05-31T20:30:05.244 DEBUG:boto:Headers: {} 2023-05-31T20:30:05.245 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:30:05.245 DEBUG:boto:Port: 8000 2023-05-31T20:30:05.246 DEBUG:boto:Params: {} 2023-05-31T20:30:05.247 DEBUG:boto:Token: None 2023-05-31T20:30:05.248 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:30:05 GMT /pklxkd-56/c 2023-05-31T20:30:05.249 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:0wzVkEEfxRSiuHDevp5/JfKrCQA= 2023-05-31T20:30:05.250 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:30:05 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:0wzVkEEfxRSiuHDevp5/JfKrCQA=', 'Content-Length': '0'} 2023-05-31T20:30:05.252 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:29:54 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f217ba1d1f1c6dd7-006477ae4d-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:30:05 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:30:05.253 DEBUG:boto:path=/pklxkd-56/ 2023-05-31T20:30:05.254 DEBUG:boto:auth_path=/pklxkd-56/ 2023-05-31T20:30:05.255 DEBUG:boto:path=/pklxkd-56/?prefix=c 2023-05-31T20:30:05.256 DEBUG:boto:auth_path=/pklxkd-56/?prefix=c 2023-05-31T20:30:05.256 DEBUG:boto:Method: GET 2023-05-31T20:30:05.257 DEBUG:boto:Path: /pklxkd-56/?prefix=c 2023-05-31T20:30:05.258 DEBUG:boto:Data: 2023-05-31T20:30:05.258 DEBUG:boto:Headers: {} 2023-05-31T20:30:05.258 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:30:05.259 DEBUG:boto:Port: 8001 2023-05-31T20:30:05.259 DEBUG:boto:Params: {} 2023-05-31T20:30:05.260 DEBUG:boto:Token: None 2023-05-31T20:30:05.260 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:30:05 GMT /pklxkd-56/ 2023-05-31T20:30:05.261 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:J7Nnzn6qi5C46akVblD05HEWcKQ= 2023-05-31T20:30:05.262 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:30:05 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:J7Nnzn6qi5C46akVblD05HEWcKQ=', 'Content-Length': '0'} 2023-05-31T20:30:05.264 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c8f58009377efc38-006477ae4d-162e-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:30:05 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:30:05.266 DEBUG:boto:b'pklxkd-56c1000falsec2023-05-31T20:29:54.281Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:30:05.266 DEBUG:boto:path=/pklxkd-56/ 2023-05-31T20:30:05.267 DEBUG:boto:auth_path=/pklxkd-56/ 2023-05-31T20:30:05.268 DEBUG:boto:path=/pklxkd-56/?prefix=c 2023-05-31T20:30:05.269 DEBUG:boto:auth_path=/pklxkd-56/?prefix=c 2023-05-31T20:30:05.270 DEBUG:boto:Method: GET 2023-05-31T20:30:05.271 DEBUG:boto:Path: /pklxkd-56/?prefix=c 2023-05-31T20:30:05.272 DEBUG:boto:Data: 2023-05-31T20:30:05.273 DEBUG:boto:Headers: {} 2023-05-31T20:30:05.273 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:30:05.274 DEBUG:boto:Port: 8000 2023-05-31T20:30:05.275 DEBUG:boto:Params: {} 2023-05-31T20:30:05.276 DEBUG:boto:Token: None 2023-05-31T20:30:05.277 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:30:05 GMT /pklxkd-56/ 2023-05-31T20:30:05.277 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:J7Nnzn6qi5C46akVblD05HEWcKQ= 2023-05-31T20:30:05.278 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:30:05 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:J7Nnzn6qi5C46akVblD05HEWcKQ=', 'Content-Length': '0'} 2023-05-31T20:30:05.281 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a317c0dbcc90c66d-006477ae4d-1630-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:30:05 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:30:05.282 DEBUG:boto:b'pklxkd-56c1000falsec2023-05-31T20:29:54.281Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:30:05.283 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=d 2023-05-31T20:30:05.284 DEBUG:boto:path=/pklxkd-56/d 2023-05-31T20:30:05.285 DEBUG:boto:auth_path=/pklxkd-56/d 2023-05-31T20:30:05.285 DEBUG:boto:path=/pklxkd-56/d?versionId=null 2023-05-31T20:30:05.286 DEBUG:boto:auth_path=/pklxkd-56/d?versionId=null 2023-05-31T20:30:05.287 DEBUG:boto:Method: GET 2023-05-31T20:30:05.287 DEBUG:boto:Path: /pklxkd-56/d?versionId=null 2023-05-31T20:30:05.288 DEBUG:boto:Data: 2023-05-31T20:30:05.289 DEBUG:boto:Headers: {} 2023-05-31T20:30:05.290 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:30:05.290 DEBUG:boto:Port: 8001 2023-05-31T20:30:05.290 DEBUG:boto:Params: {} 2023-05-31T20:30:05.291 DEBUG:boto:Token: None 2023-05-31T20:30:05.292 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:30:05 GMT /pklxkd-56/d?versionId=null 2023-05-31T20:30:05.293 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:TxvdKJ39/csN5tYgAZMR2rOzun0= 2023-05-31T20:30:05.293 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:30:05 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:TxvdKJ39/csN5tYgAZMR2rOzun0=', 'Content-Length': '0'} 2023-05-31T20:30:05.296 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:29:54 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000369704570ee0817-006477ae4d-162e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:30:05 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:30:05.299 DEBUG:boto:path=/pklxkd-56/d 2023-05-31T20:30:05.300 DEBUG:boto:auth_path=/pklxkd-56/d 2023-05-31T20:30:05.300 DEBUG:boto:path=/pklxkd-56/d?versionId=null 2023-05-31T20:30:05.301 DEBUG:boto:auth_path=/pklxkd-56/d?versionId=null 2023-05-31T20:30:05.302 DEBUG:boto:Method: GET 2023-05-31T20:30:05.303 DEBUG:boto:Path: /pklxkd-56/d?versionId=null 2023-05-31T20:30:05.303 DEBUG:boto:Data: 2023-05-31T20:30:05.304 DEBUG:boto:Headers: {} 2023-05-31T20:30:05.305 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:30:05.305 DEBUG:boto:Port: 8000 2023-05-31T20:30:05.305 DEBUG:boto:Params: {} 2023-05-31T20:30:05.306 DEBUG:boto:Token: None 2023-05-31T20:30:05.306 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:30:05 GMT /pklxkd-56/d?versionId=null 2023-05-31T20:30:05.307 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:TxvdKJ39/csN5tYgAZMR2rOzun0= 2023-05-31T20:30:05.307 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:30:05 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:TxvdKJ39/csN5tYgAZMR2rOzun0=', 'Content-Length': '0'} 2023-05-31T20:30:05.309 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:29:54 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004973d6b6025c848c-006477ae4d-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:30:05 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:30:05.310 DEBUG:boto:path=/pklxkd-56/d 2023-05-31T20:30:05.311 DEBUG:boto:auth_path=/pklxkd-56/d 2023-05-31T20:30:05.311 DEBUG:boto:path=/pklxkd-56/d?versionId=null 2023-05-31T20:30:05.311 DEBUG:boto:auth_path=/pklxkd-56/d?versionId=null 2023-05-31T20:30:05.312 DEBUG:boto:Method: HEAD 2023-05-31T20:30:05.312 DEBUG:boto:Path: /pklxkd-56/d?versionId=null 2023-05-31T20:30:05.312 DEBUG:boto:Data: 2023-05-31T20:30:05.313 DEBUG:boto:Headers: {} 2023-05-31T20:30:05.313 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:30:05.314 DEBUG:boto:Port: 8001 2023-05-31T20:30:05.314 DEBUG:boto:Params: {} 2023-05-31T20:30:05.315 DEBUG:boto:Token: None 2023-05-31T20:30:05.315 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:30:05 GMT /pklxkd-56/d?versionId=null 2023-05-31T20:30:05.316 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:qJ6XPLG6a8XmHQ8rZ+bZ1kHweMs= 2023-05-31T20:30:05.316 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:30:05 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:qJ6XPLG6a8XmHQ8rZ+bZ1kHweMs=', 'Content-Length': '0'} 2023-05-31T20:30:05.318 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:29:54 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000f8ab287f77384e7-006477ae4d-162e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:30:05 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:30:05.319 DEBUG:boto:path=/pklxkd-56/d 2023-05-31T20:30:05.319 DEBUG:boto:auth_path=/pklxkd-56/d 2023-05-31T20:30:05.320 DEBUG:boto:path=/pklxkd-56/d?versionId=null 2023-05-31T20:30:05.320 DEBUG:boto:auth_path=/pklxkd-56/d?versionId=null 2023-05-31T20:30:05.320 DEBUG:boto:Method: HEAD 2023-05-31T20:30:05.321 DEBUG:boto:Path: /pklxkd-56/d?versionId=null 2023-05-31T20:30:05.321 DEBUG:boto:Data: 2023-05-31T20:30:05.321 DEBUG:boto:Headers: {} 2023-05-31T20:30:05.322 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:30:05.322 DEBUG:boto:Port: 8000 2023-05-31T20:30:05.322 DEBUG:boto:Params: {} 2023-05-31T20:30:05.323 DEBUG:boto:Token: None 2023-05-31T20:30:05.323 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:30:05 GMT /pklxkd-56/d?versionId=null 2023-05-31T20:30:05.324 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:qJ6XPLG6a8XmHQ8rZ+bZ1kHweMs= 2023-05-31T20:30:05.324 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:30:05 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:qJ6XPLG6a8XmHQ8rZ+bZ1kHweMs=', 'Content-Length': '0'} 2023-05-31T20:30:05.326 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:29:54 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000536adafb4af35b5c-006477ae4d-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:30:05 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:30:05.327 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=d 2023-05-31T20:30:05.327 DEBUG:boto:path=/pklxkd-56/d 2023-05-31T20:30:05.328 DEBUG:boto:auth_path=/pklxkd-56/d 2023-05-31T20:30:05.328 DEBUG:boto:path=/pklxkd-56/d?versionId=null 2023-05-31T20:30:05.329 DEBUG:boto:auth_path=/pklxkd-56/d?versionId=null 2023-05-31T20:30:05.330 DEBUG:boto:Method: GET 2023-05-31T20:30:05.330 DEBUG:boto:Path: /pklxkd-56/d?versionId=null 2023-05-31T20:30:05.331 DEBUG:boto:Data: 2023-05-31T20:30:05.331 DEBUG:boto:Headers: {} 2023-05-31T20:30:05.331 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:30:05.332 DEBUG:boto:Port: 8001 2023-05-31T20:30:05.332 DEBUG:boto:Params: {} 2023-05-31T20:30:05.333 DEBUG:boto:Token: None 2023-05-31T20:30:05.333 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:30:05 GMT /pklxkd-56/d?versionId=null 2023-05-31T20:30:05.333 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:TxvdKJ39/csN5tYgAZMR2rOzun0= 2023-05-31T20:30:05.334 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:30:05 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:TxvdKJ39/csN5tYgAZMR2rOzun0=', 'Content-Length': '0'} 2023-05-31T20:30:05.335 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:29:54 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000a5e3eb2b953c361-006477ae4d-162e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:30:05 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:30:05.336 DEBUG:boto:path=/pklxkd-56/d 2023-05-31T20:30:05.337 DEBUG:boto:auth_path=/pklxkd-56/d 2023-05-31T20:30:05.337 DEBUG:boto:path=/pklxkd-56/d?versionId=null 2023-05-31T20:30:05.337 DEBUG:boto:auth_path=/pklxkd-56/d?versionId=null 2023-05-31T20:30:05.338 DEBUG:boto:Method: GET 2023-05-31T20:30:05.338 DEBUG:boto:Path: /pklxkd-56/d?versionId=null 2023-05-31T20:30:05.338 DEBUG:boto:Data: 2023-05-31T20:30:05.339 DEBUG:boto:Headers: {} 2023-05-31T20:30:05.339 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:30:05.340 DEBUG:boto:Port: 8000 2023-05-31T20:30:05.340 DEBUG:boto:Params: {} 2023-05-31T20:30:05.340 DEBUG:boto:Token: None 2023-05-31T20:30:05.341 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:30:05 GMT /pklxkd-56/d?versionId=null 2023-05-31T20:30:05.341 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:TxvdKJ39/csN5tYgAZMR2rOzun0= 2023-05-31T20:30:05.342 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:30:05 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:TxvdKJ39/csN5tYgAZMR2rOzun0=', 'Content-Length': '0'} 2023-05-31T20:30:05.343 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:29:54 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000041b3471b3b36a8ca-006477ae4d-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:30:05 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:30:05.344 DEBUG:boto:path=/pklxkd-56/ 2023-05-31T20:30:05.344 DEBUG:boto:auth_path=/pklxkd-56/ 2023-05-31T20:30:05.345 DEBUG:boto:path=/pklxkd-56/?prefix=d 2023-05-31T20:30:05.345 DEBUG:boto:auth_path=/pklxkd-56/?prefix=d 2023-05-31T20:30:05.346 DEBUG:boto:Method: GET 2023-05-31T20:30:05.346 DEBUG:boto:Path: /pklxkd-56/?prefix=d 2023-05-31T20:30:05.346 DEBUG:boto:Data: 2023-05-31T20:30:05.347 DEBUG:boto:Headers: {} 2023-05-31T20:30:05.347 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:30:05.347 DEBUG:boto:Port: 8001 2023-05-31T20:30:05.348 DEBUG:boto:Params: {} 2023-05-31T20:30:05.348 DEBUG:boto:Token: None 2023-05-31T20:30:05.349 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:30:05 GMT /pklxkd-56/ 2023-05-31T20:30:05.349 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:J7Nnzn6qi5C46akVblD05HEWcKQ= 2023-05-31T20:30:05.350 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:30:05 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:J7Nnzn6qi5C46akVblD05HEWcKQ=', 'Content-Length': '0'} 2023-05-31T20:30:05.352 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000071bbf05e8c5593b8-006477ae4d-162e-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:30:05 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:30:05.352 DEBUG:boto:b'pklxkd-56d1000falsed2023-05-31T20:29:54.306Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:30:05.353 DEBUG:boto:path=/pklxkd-56/ 2023-05-31T20:30:05.353 DEBUG:boto:auth_path=/pklxkd-56/ 2023-05-31T20:30:05.354 DEBUG:boto:path=/pklxkd-56/?prefix=d 2023-05-31T20:30:05.354 DEBUG:boto:auth_path=/pklxkd-56/?prefix=d 2023-05-31T20:30:05.354 DEBUG:boto:Method: GET 2023-05-31T20:30:05.355 DEBUG:boto:Path: /pklxkd-56/?prefix=d 2023-05-31T20:30:05.355 DEBUG:boto:Data: 2023-05-31T20:30:05.355 DEBUG:boto:Headers: {} 2023-05-31T20:30:05.356 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:30:05.356 DEBUG:boto:Port: 8000 2023-05-31T20:30:05.357 DEBUG:boto:Params: {} 2023-05-31T20:30:05.357 DEBUG:boto:Token: None 2023-05-31T20:30:05.360 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:30:05 GMT /pklxkd-56/ 2023-05-31T20:30:05.361 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:J7Nnzn6qi5C46akVblD05HEWcKQ= 2023-05-31T20:30:05.362 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:30:05 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:J7Nnzn6qi5C46akVblD05HEWcKQ=', 'Content-Length': '0'} 2023-05-31T20:30:05.364 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000009a11aad71389d0b9-006477ae4d-1630-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:30:05 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:30:05.365 DEBUG:boto:b'pklxkd-56d1000falsed2023-05-31T20:29:54.306Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:30:05.366 DEBUG:boto:path=/pklxkd-56/d 2023-05-31T20:30:05.366 DEBUG:boto:auth_path=/pklxkd-56/d 2023-05-31T20:30:05.367 DEBUG:boto:Method: HEAD 2023-05-31T20:30:05.367 DEBUG:boto:Path: /pklxkd-56/d 2023-05-31T20:30:05.367 DEBUG:boto:Data: 2023-05-31T20:30:05.368 DEBUG:boto:Headers: {} 2023-05-31T20:30:05.368 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:30:05.368 DEBUG:boto:Port: 8001 2023-05-31T20:30:05.369 DEBUG:boto:Params: {} 2023-05-31T20:30:05.369 DEBUG:boto:Token: None 2023-05-31T20:30:05.370 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:30:05 GMT /pklxkd-56/d 2023-05-31T20:30:05.370 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:Lqb/abFrpzxBRvU6qe9D9Ply1hE= 2023-05-31T20:30:05.371 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:30:05 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:Lqb/abFrpzxBRvU6qe9D9Ply1hE=', 'Content-Length': '0'} 2023-05-31T20:30:05.372 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:29:54 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d39fe2ea7d5900d8-006477ae4d-162e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:30:05 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:30:05.373 DEBUG:boto:path=/pklxkd-56/d 2023-05-31T20:30:05.373 DEBUG:boto:auth_path=/pklxkd-56/d 2023-05-31T20:30:05.374 DEBUG:boto:Method: HEAD 2023-05-31T20:30:05.374 DEBUG:boto:Path: /pklxkd-56/d 2023-05-31T20:30:05.374 DEBUG:boto:Data: 2023-05-31T20:30:05.375 DEBUG:boto:Headers: {} 2023-05-31T20:30:05.375 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:30:05.375 DEBUG:boto:Port: 8000 2023-05-31T20:30:05.376 DEBUG:boto:Params: {} 2023-05-31T20:30:05.376 DEBUG:boto:Token: None 2023-05-31T20:30:05.376 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:30:05 GMT /pklxkd-56/d 2023-05-31T20:30:05.377 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:Lqb/abFrpzxBRvU6qe9D9Ply1hE= 2023-05-31T20:30:05.378 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:30:05 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:Lqb/abFrpzxBRvU6qe9D9Ply1hE=', 'Content-Length': '0'} 2023-05-31T20:30:05.379 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:29:54 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000cd140f6f0bb5a5ee-006477ae4d-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:30:05 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:30:05.380 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=d 2023-05-31T20:30:05.380 DEBUG:boto:path=/pklxkd-56/d 2023-05-31T20:30:05.380 DEBUG:boto:auth_path=/pklxkd-56/d 2023-05-31T20:30:05.381 DEBUG:boto:Method: GET 2023-05-31T20:30:05.381 DEBUG:boto:Path: /pklxkd-56/d 2023-05-31T20:30:05.382 DEBUG:boto:Data: 2023-05-31T20:30:05.382 DEBUG:boto:Headers: {} 2023-05-31T20:30:05.382 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:30:05.383 DEBUG:boto:Port: 8001 2023-05-31T20:30:05.383 DEBUG:boto:Params: {} 2023-05-31T20:30:05.383 DEBUG:boto:Token: None 2023-05-31T20:30:05.384 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:30:05 GMT /pklxkd-56/d 2023-05-31T20:30:05.384 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:TG1T5sJwLSkW6GNgm7s5aXYdU3U= 2023-05-31T20:30:05.385 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:30:05 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:TG1T5sJwLSkW6GNgm7s5aXYdU3U=', 'Content-Length': '0'} 2023-05-31T20:30:05.386 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:29:54 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000030c52e127f937e2b-006477ae4d-162e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:30:05 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:30:05.436 DEBUG:boto:path=/pklxkd-56/d 2023-05-31T20:30:05.437 DEBUG:boto:auth_path=/pklxkd-56/d 2023-05-31T20:30:05.437 DEBUG:boto:Method: GET 2023-05-31T20:30:05.438 DEBUG:boto:Path: /pklxkd-56/d 2023-05-31T20:30:05.438 DEBUG:boto:Data: 2023-05-31T20:30:05.438 DEBUG:boto:Headers: {} 2023-05-31T20:30:05.439 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:30:05.439 DEBUG:boto:Port: 8000 2023-05-31T20:30:05.439 DEBUG:boto:Params: {} 2023-05-31T20:30:05.440 DEBUG:boto:Token: None 2023-05-31T20:30:05.440 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:30:05 GMT /pklxkd-56/d 2023-05-31T20:30:05.441 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:TG1T5sJwLSkW6GNgm7s5aXYdU3U= 2023-05-31T20:30:05.441 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:30:05 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:TG1T5sJwLSkW6GNgm7s5aXYdU3U=', 'Content-Length': '0'} 2023-05-31T20:30:05.443 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:29:54 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000418b4dab3b043e41-006477ae4d-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:30:05 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:30:05.444 DEBUG:boto:path=/pklxkd-56/ 2023-05-31T20:30:05.444 DEBUG:boto:auth_path=/pklxkd-56/ 2023-05-31T20:30:05.444 DEBUG:boto:path=/pklxkd-56/?prefix=d 2023-05-31T20:30:05.445 DEBUG:boto:auth_path=/pklxkd-56/?prefix=d 2023-05-31T20:30:05.445 DEBUG:boto:Method: GET 2023-05-31T20:30:05.445 DEBUG:boto:Path: /pklxkd-56/?prefix=d 2023-05-31T20:30:05.446 DEBUG:boto:Data: 2023-05-31T20:30:05.446 DEBUG:boto:Headers: {} 2023-05-31T20:30:05.447 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:30:05.447 DEBUG:boto:Port: 8001 2023-05-31T20:30:05.447 DEBUG:boto:Params: {} 2023-05-31T20:30:05.448 DEBUG:boto:Token: None 2023-05-31T20:30:05.448 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:30:05 GMT /pklxkd-56/ 2023-05-31T20:30:05.449 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:J7Nnzn6qi5C46akVblD05HEWcKQ= 2023-05-31T20:30:05.449 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:30:05 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:J7Nnzn6qi5C46akVblD05HEWcKQ=', 'Content-Length': '0'} 2023-05-31T20:30:05.451 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000000eaf2aa961776235-006477ae4d-162e-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:30:05 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:30:05.452 DEBUG:boto:b'pklxkd-56d1000falsed2023-05-31T20:29:54.306Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:30:05.452 DEBUG:boto:path=/pklxkd-56/ 2023-05-31T20:30:05.453 DEBUG:boto:auth_path=/pklxkd-56/ 2023-05-31T20:30:05.453 DEBUG:boto:path=/pklxkd-56/?prefix=d 2023-05-31T20:30:05.454 DEBUG:boto:auth_path=/pklxkd-56/?prefix=d 2023-05-31T20:30:05.454 DEBUG:boto:Method: GET 2023-05-31T20:30:05.454 DEBUG:boto:Path: /pklxkd-56/?prefix=d 2023-05-31T20:30:05.455 DEBUG:boto:Data: 2023-05-31T20:30:05.455 DEBUG:boto:Headers: {} 2023-05-31T20:30:05.455 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:30:05.456 DEBUG:boto:Port: 8000 2023-05-31T20:30:05.456 DEBUG:boto:Params: {} 2023-05-31T20:30:05.457 DEBUG:boto:Token: None 2023-05-31T20:30:05.457 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:30:05 GMT /pklxkd-56/ 2023-05-31T20:30:05.457 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:J7Nnzn6qi5C46akVblD05HEWcKQ= 2023-05-31T20:30:05.458 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:30:05 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:J7Nnzn6qi5C46akVblD05HEWcKQ=', 'Content-Length': '0'} 2023-05-31T20:30:05.460 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000081d2b29bd84f8427-006477ae4d-1630-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:30:05 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:30:05.460 DEBUG:boto:b'pklxkd-56d1000falsed2023-05-31T20:29:54.306Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:30:05.461 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=pklxkd-56 zones={a2, a1} 2023-05-31T20:30:05.461 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'reshard', '--bucket', 'pklxkd-56', '--num-shards', '3', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-05-31T20:30:05.462 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'reshard', '--bucket', 'pklxkd-56', '--num-shards', '3', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-05-31T20:30:05.462 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket reshard --bucket pklxkd-56 --num-shards 3 --yes-i-really-mean-it --cluster c1 --debug-rgw 0 --debug-ms 0 2023-05-31T20:30:05.515 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2023-05-31T20:30:05.515 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2023-05-31T20:30:05.582 INFO:teuthology.orchestra.run.smithi099.stdout:tenant: 2023-05-31T20:30:05.582 INFO:teuthology.orchestra.run.smithi099.stdout:bucket name: pklxkd-56 2023-05-31T20:30:05.585 INFO:teuthology.orchestra.run.smithi099.stdout:total entries: 4 2023-05-31T20:30:05.649 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'reshard', '--bucket', 'pklxkd-56', '--num-shards', '3', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-05-31T20:30:05.650 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'reshard', '--bucket', 'pklxkd-56', '--num-shards', '3', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-05-31T20:30:05.650 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket reshard --bucket pklxkd-56 --num-shards 3 --yes-i-really-mean-it --cluster c1 --debug-rgw 0 --debug-ms 0 2023-05-31T20:30:05.699 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2023-05-31T20:30:05.700 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2023-05-31T20:30:05.742 INFO:teuthology.orchestra.run.smithi099.stdout:tenant: 2023-05-31T20:30:05.743 INFO:teuthology.orchestra.run.smithi099.stdout:bucket name: pklxkd-56 2023-05-31T20:30:05.743 INFO:teuthology.orchestra.run.smithi099.stdout:total entries: 4 2023-05-31T20:30:05.773 DEBUG:boto:path=/pklxkd-56/ 2023-05-31T20:30:05.774 DEBUG:boto:auth_path=/pklxkd-56/ 2023-05-31T20:30:05.774 DEBUG:boto:Method: HEAD 2023-05-31T20:30:05.775 DEBUG:boto:Path: /pklxkd-56/ 2023-05-31T20:30:05.775 DEBUG:boto:Data: 2023-05-31T20:30:05.776 DEBUG:boto:Headers: {} 2023-05-31T20:30:05.776 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:30:05.776 DEBUG:boto:Port: 8000 2023-05-31T20:30:05.777 DEBUG:boto:Params: {} 2023-05-31T20:30:05.777 DEBUG:boto:Token: None 2023-05-31T20:30:05.778 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:30:05 GMT /pklxkd-56/ 2023-05-31T20:30:05.778 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:4Xs4a4NccmioNYijDjZreb5eUyo= 2023-05-31T20:30:05.779 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:30:05 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:4Xs4a4NccmioNYijDjZreb5eUyo=', 'Content-Length': '0'} 2023-05-31T20:30:05.781 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '4'), ('X-RGW-Bytes-Used', '12'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000cff813608fa36134-006477ae4d-1630-a1'), ('Content-Length', '0'), ('Date', 'Wed, 31 May 2023 20:30:05 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:30:05.781 DEBUG:boto:path=/pklxkd-56/e 2023-05-31T20:30:05.782 DEBUG:boto:auth_path=/pklxkd-56/e 2023-05-31T20:30:05.782 DEBUG:boto:Method: PUT 2023-05-31T20:30:05.783 DEBUG:boto:Path: /pklxkd-56/e 2023-05-31T20:30:05.783 DEBUG:boto:Data: 2023-05-31T20:30:05.783 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-05-31T20:30:05.784 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:30:05.784 DEBUG:boto:Port: 8000 2023-05-31T20:30:05.784 DEBUG:boto:Params: {} 2023-05-31T20:30:05.785 DEBUG:boto:Token: None 2023-05-31T20:30:05.785 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Wed, 31 May 2023 20:30:05 GMT /pklxkd-56/e 2023-05-31T20:30:05.786 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:sRVa9k0TeTLn7OfCb4BJCU03Hyw= 2023-05-31T20:30:05.786 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Wed, 31 May 2023 20:30:05 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:sRVa9k0TeTLn7OfCb4BJCU03Hyw='} 2023-05-31T20:30:05.791 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000ec38f3afed2d4550-006477ae4d-1630-a1'), ('Date', 'Wed, 31 May 2023 20:30:05 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:30:05.792 DEBUG:boto:path=/pklxkd-56/ 2023-05-31T20:30:05.793 DEBUG:boto:auth_path=/pklxkd-56/ 2023-05-31T20:30:05.794 DEBUG:boto:Method: HEAD 2023-05-31T20:30:05.794 DEBUG:boto:Path: /pklxkd-56/ 2023-05-31T20:30:05.794 DEBUG:boto:Data: 2023-05-31T20:30:05.795 DEBUG:boto:Headers: {} 2023-05-31T20:30:05.795 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:30:05.796 DEBUG:boto:Port: 8000 2023-05-31T20:30:05.796 DEBUG:boto:Params: {} 2023-05-31T20:30:05.797 DEBUG:boto:Token: None 2023-05-31T20:30:05.797 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:30:05 GMT /pklxkd-56/ 2023-05-31T20:30:05.798 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:4Xs4a4NccmioNYijDjZreb5eUyo= 2023-05-31T20:30:05.798 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:30:05 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:4Xs4a4NccmioNYijDjZreb5eUyo=', 'Content-Length': '0'} 2023-05-31T20:30:05.800 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '5'), ('X-RGW-Bytes-Used', '15'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000eebf7a39fee8c9cb-006477ae4d-1630-a1'), ('Content-Length', '0'), ('Date', 'Wed, 31 May 2023 20:30:05 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:30:05.801 DEBUG:boto:path=/pklxkd-56/f 2023-05-31T20:30:05.802 DEBUG:boto:auth_path=/pklxkd-56/f 2023-05-31T20:30:05.803 DEBUG:boto:Method: PUT 2023-05-31T20:30:05.804 DEBUG:boto:Path: /pklxkd-56/f 2023-05-31T20:30:05.805 DEBUG:boto:Data: 2023-05-31T20:30:05.805 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-05-31T20:30:05.806 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:30:05.807 DEBUG:boto:Port: 8000 2023-05-31T20:30:05.808 DEBUG:boto:Params: {} 2023-05-31T20:30:05.808 DEBUG:boto:Token: None 2023-05-31T20:30:05.809 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Wed, 31 May 2023 20:30:05 GMT /pklxkd-56/f 2023-05-31T20:30:05.810 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:9+he/3jHamfSmllAcnnalCnlvVE= 2023-05-31T20:30:05.811 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Wed, 31 May 2023 20:30:05 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:9+he/3jHamfSmllAcnnalCnlvVE='} 2023-05-31T20:30:05.820 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000df296453b05171ad-006477ae4d-1630-a1'), ('Date', 'Wed, 31 May 2023 20:30:05 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:30:05.821 DEBUG:boto:path=/pklxkd-56/ 2023-05-31T20:30:05.822 DEBUG:boto:auth_path=/pklxkd-56/ 2023-05-31T20:30:05.822 DEBUG:boto:Method: HEAD 2023-05-31T20:30:05.823 DEBUG:boto:Path: /pklxkd-56/ 2023-05-31T20:30:05.823 DEBUG:boto:Data: 2023-05-31T20:30:05.824 DEBUG:boto:Headers: {} 2023-05-31T20:30:05.825 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:30:05.825 DEBUG:boto:Port: 8000 2023-05-31T20:30:05.826 DEBUG:boto:Params: {} 2023-05-31T20:30:05.827 DEBUG:boto:Token: None 2023-05-31T20:30:05.828 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:30:05 GMT /pklxkd-56/ 2023-05-31T20:30:05.829 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:4Xs4a4NccmioNYijDjZreb5eUyo= 2023-05-31T20:30:05.830 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:30:05 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:4Xs4a4NccmioNYijDjZreb5eUyo=', 'Content-Length': '0'} 2023-05-31T20:30:05.832 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '6'), ('X-RGW-Bytes-Used', '18'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000c09445638a614f49-006477ae4d-1630-a1'), ('Content-Length', '0'), ('Date', 'Wed, 31 May 2023 20:30:05 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:30:05.833 DEBUG:boto:path=/pklxkd-56/g 2023-05-31T20:30:05.834 DEBUG:boto:auth_path=/pklxkd-56/g 2023-05-31T20:30:05.835 DEBUG:boto:Method: PUT 2023-05-31T20:30:05.836 DEBUG:boto:Path: /pklxkd-56/g 2023-05-31T20:30:05.836 DEBUG:boto:Data: 2023-05-31T20:30:05.837 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-05-31T20:30:05.838 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:30:05.839 DEBUG:boto:Port: 8000 2023-05-31T20:30:05.840 DEBUG:boto:Params: {} 2023-05-31T20:30:05.840 DEBUG:boto:Token: None 2023-05-31T20:30:05.841 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Wed, 31 May 2023 20:30:05 GMT /pklxkd-56/g 2023-05-31T20:30:05.842 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:JMSQKQYnCuV89EW3Vv/ohunCTV4= 2023-05-31T20:30:05.843 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Wed, 31 May 2023 20:30:05 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:JMSQKQYnCuV89EW3Vv/ohunCTV4='} 2023-05-31T20:30:05.889 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000dbf7278d8eb24984-006477ae4d-1630-a1'), ('Date', 'Wed, 31 May 2023 20:30:05 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:30:05.890 DEBUG:boto:path=/pklxkd-56/ 2023-05-31T20:30:05.890 DEBUG:boto:auth_path=/pklxkd-56/ 2023-05-31T20:30:05.890 DEBUG:boto:Method: HEAD 2023-05-31T20:30:05.891 DEBUG:boto:Path: /pklxkd-56/ 2023-05-31T20:30:05.891 DEBUG:boto:Data: 2023-05-31T20:30:05.892 DEBUG:boto:Headers: {} 2023-05-31T20:30:05.892 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:30:05.893 DEBUG:boto:Port: 8000 2023-05-31T20:30:05.893 DEBUG:boto:Params: {} 2023-05-31T20:30:05.894 DEBUG:boto:Token: None 2023-05-31T20:30:05.896 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:30:05 GMT /pklxkd-56/ 2023-05-31T20:30:05.897 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:4Xs4a4NccmioNYijDjZreb5eUyo= 2023-05-31T20:30:05.897 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:30:05 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:4Xs4a4NccmioNYijDjZreb5eUyo=', 'Content-Length': '0'} 2023-05-31T20:30:05.899 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '7'), ('X-RGW-Bytes-Used', '21'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000cd8530ff10960a02-006477ae4d-1630-a1'), ('Content-Length', '0'), ('Date', 'Wed, 31 May 2023 20:30:05 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:30:05.900 DEBUG:boto:path=/pklxkd-56/h 2023-05-31T20:30:05.901 DEBUG:boto:auth_path=/pklxkd-56/h 2023-05-31T20:30:05.902 DEBUG:boto:Method: PUT 2023-05-31T20:30:05.903 DEBUG:boto:Path: /pklxkd-56/h 2023-05-31T20:30:05.904 DEBUG:boto:Data: 2023-05-31T20:30:05.905 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-05-31T20:30:05.906 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:30:05.906 DEBUG:boto:Port: 8000 2023-05-31T20:30:05.907 DEBUG:boto:Params: {} 2023-05-31T20:30:05.907 DEBUG:boto:Token: None 2023-05-31T20:30:05.908 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Wed, 31 May 2023 20:30:05 GMT /pklxkd-56/h 2023-05-31T20:30:05.908 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:4bgud/fbYgprQwCu+RgyvStFdcU= 2023-05-31T20:30:05.909 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Wed, 31 May 2023 20:30:05 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:4bgud/fbYgprQwCu+RgyvStFdcU='} 2023-05-31T20:30:05.912 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000009bc5f93d55c6ef9a-006477ae4d-1630-a1'), ('Date', 'Wed, 31 May 2023 20:30:05 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:30:05.914 DEBUG:rgw_multi.tests:bucket checkpoint: source=a1 target=a2 bucket=pklxkd-56 2023-05-31T20:30:05.915 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'pklxkd-56', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-05-31T20:30:05.915 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'pklxkd-56', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-05-31T20:30:05.916 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket pklxkd-56 --source-zone a1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2023-05-31T20:30:05.976 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2023-05-31T20:30:05.977 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2023-05-31T20:30:06.030 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:30:06.025+0000 7f3958492780 1 waiting to reach latest gen 2, on 0.. 2023-05-31T20:30:11.031 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:30:11.026+0000 7f3958492780 1 waiting to reach latest gen 2, on 0.. 2023-05-31T20:30:16.032 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:30:16.027+0000 7f3958492780 1 waiting to reach latest gen 2, on 0.. 2023-05-31T20:30:21.032 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:30:21.028+0000 7f3958492780 1 waiting to reach latest gen 2, on 0.. 2023-05-31T20:30:26.033 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:30:26.029+0000 7f3958492780 1 waiting to reach latest gen 2, on 1.. 2023-05-31T20:30:31.034 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:30:31.029+0000 7f3958492780 1 waiting to reach latest gen 2, on 1.. 2023-05-31T20:30:36.035 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:30:36.030+0000 7f3958492780 1 waiting to reach latest gen 2, on 1.. 2023-05-31T20:30:41.035 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:30:41.030+0000 7f3958492780 1 waiting to reach latest gen 2, on 1.. 2023-05-31T20:30:46.036 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:30:46.031+0000 7f3958492780 1 waiting to reach latest gen 2, on 1.. 2023-05-31T20:30:51.037 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:30:51.031+0000 7f3958492780 1 waiting to reach latest gen 2, on 1.. 2023-05-31T20:30:56.037 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:30:56.032+0000 7f3958492780 1 waiting to reach latest gen 2, on 1.. 2023-05-31T20:31:01.038 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:31:01.034+0000 7f3958492780 1 waiting to reach latest gen 2, on 1.. 2023-05-31T20:31:06.038 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:31:06.034+0000 7f3958492780 1 waiting to reach latest gen 2, on 1.. 2023-05-31T20:31:11.040 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:31:11.035+0000 7f3958492780 1 waiting to reach latest gen 2, on 1.. 2023-05-31T20:31:16.040 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:31:16.035+0000 7f3958492780 1 waiting to reach latest gen 2, on 1.. 2023-05-31T20:31:21.040 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:31:21.035+0000 7f3958492780 1 waiting to reach latest gen 2, on 1.. 2023-05-31T20:31:26.041 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:31:26.036+0000 7f3958492780 1 waiting to reach latest gen 2, on 1.. 2023-05-31T20:31:31.042 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:31:31.036+0000 7f3958492780 1 waiting to reach latest gen 2, on 1.. 2023-05-31T20:31:36.042 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:31:36.037+0000 7f3958492780 1 waiting to reach latest gen 2, on 1.. 2023-05-31T20:31:41.043 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:31:41.038+0000 7f3958492780 1 waiting to reach latest gen 2, on 1.. 2023-05-31T20:31:46.044 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:31:46.040+0000 7f3958492780 1 waiting for incremental sync to catch up: 2023-05-31T20:31:46.045 INFO:teuthology.orchestra.run.smithi099.stderr: local status: [, , ] 2023-05-31T20:31:46.045 INFO:teuthology.orchestra.run.smithi099.stderr: remote markers: [00000000003.69.6, 00000000002.23.6, 00000000002.67.6] 2023-05-31T20:31:51.045 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:31:51.041+0000 7f3958492780 1 waiting for incremental sync to catch up: 2023-05-31T20:31:51.045 INFO:teuthology.orchestra.run.smithi099.stderr: local status: [, , ] 2023-05-31T20:31:51.046 INFO:teuthology.orchestra.run.smithi099.stderr: remote markers: [00000000003.69.6, 00000000002.23.6, 00000000002.67.6] 2023-05-31T20:31:56.046 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:31:56.042+0000 7f3958492780 1 waiting for incremental sync to catch up: 2023-05-31T20:31:56.047 INFO:teuthology.orchestra.run.smithi099.stderr: local status: [, , ] 2023-05-31T20:31:56.047 INFO:teuthology.orchestra.run.smithi099.stderr: remote markers: [00000000003.69.6, 00000000002.23.6, 00000000002.67.6] 2023-05-31T20:32:01.047 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:32:01.043+0000 7f3958492780 1 waiting for incremental sync to catch up: 2023-05-31T20:32:01.047 INFO:teuthology.orchestra.run.smithi099.stderr: local status: [, , ] 2023-05-31T20:32:01.047 INFO:teuthology.orchestra.run.smithi099.stderr: remote markers: [00000000003.69.6, 00000000002.23.6, 00000000002.67.6] 2023-05-31T20:32:06.048 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:32:06.042+0000 7f3958492780 1 waiting for incremental sync to catch up: 2023-05-31T20:32:06.048 INFO:teuthology.orchestra.run.smithi099.stderr: local status: [, , ] 2023-05-31T20:32:06.048 INFO:teuthology.orchestra.run.smithi099.stderr: remote markers: [00000000003.69.6, 00000000002.23.6, 00000000002.67.6] 2023-05-31T20:32:11.048 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:32:11.043+0000 7f3958492780 1 waiting for incremental sync to catch up: 2023-05-31T20:32:11.049 INFO:teuthology.orchestra.run.smithi099.stderr: local status: [, , ] 2023-05-31T20:32:11.049 INFO:teuthology.orchestra.run.smithi099.stderr: remote markers: [00000000003.69.6, 00000000002.23.6, 00000000002.67.6] 2023-05-31T20:32:16.049 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:32:16.044+0000 7f3958492780 1 waiting for incremental sync to catch up: 2023-05-31T20:32:16.050 INFO:teuthology.orchestra.run.smithi099.stderr: local status: [, , ] 2023-05-31T20:32:16.050 INFO:teuthology.orchestra.run.smithi099.stderr: remote markers: [00000000003.69.6, 00000000002.23.6, 00000000002.67.6] 2023-05-31T20:32:21.050 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:32:21.044+0000 7f3958492780 1 waiting for incremental sync to catch up: 2023-05-31T20:32:21.050 INFO:teuthology.orchestra.run.smithi099.stderr: local status: [, , ] 2023-05-31T20:32:21.050 INFO:teuthology.orchestra.run.smithi099.stderr: remote markers: [00000000003.69.6, 00000000002.23.6, 00000000002.67.6] 2023-05-31T20:32:26.050 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:32:26.046+0000 7f3958492780 1 waiting for incremental sync to catch up: 2023-05-31T20:32:26.051 INFO:teuthology.orchestra.run.smithi099.stderr: local status: [, , ] 2023-05-31T20:32:26.051 INFO:teuthology.orchestra.run.smithi099.stderr: remote markers: [00000000003.69.6, 00000000002.23.6, 00000000002.67.6] 2023-05-31T20:32:31.051 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:32:31.047+0000 7f3958492780 1 waiting for incremental sync to catch up: 2023-05-31T20:32:31.052 INFO:teuthology.orchestra.run.smithi099.stderr: local status: [, , ] 2023-05-31T20:32:31.052 INFO:teuthology.orchestra.run.smithi099.stderr: remote markers: [00000000003.69.6, 00000000002.23.6, 00000000002.67.6] 2023-05-31T20:32:36.052 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:32:36.048+0000 7f3958492780 1 waiting for incremental sync to catch up: 2023-05-31T20:32:36.052 INFO:teuthology.orchestra.run.smithi099.stderr: local status: [, , ] 2023-05-31T20:32:36.053 INFO:teuthology.orchestra.run.smithi099.stderr: remote markers: [00000000003.69.6, 00000000002.23.6, 00000000002.67.6] 2023-05-31T20:32:41.053 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:32:41.048+0000 7f3958492780 1 waiting for incremental sync to catch up: 2023-05-31T20:32:41.054 INFO:teuthology.orchestra.run.smithi099.stderr: local status: [, , ] 2023-05-31T20:32:41.054 INFO:teuthology.orchestra.run.smithi099.stderr: remote markers: [00000000003.69.6, 00000000002.23.6, 00000000002.67.6] 2023-05-31T20:32:46.057 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:32:46.049+0000 7f3958492780 1 waiting for incremental sync to catch up: 2023-05-31T20:32:46.057 INFO:teuthology.orchestra.run.smithi099.stderr: local status: [00000000003.69.6, , 00000000002.67.6] 2023-05-31T20:32:46.057 INFO:teuthology.orchestra.run.smithi099.stderr: remote markers: [00000000003.69.6, 00000000002.23.6, 00000000002.67.6] 2023-05-31T20:32:51.054 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:32:51.049+0000 7f3958492780 1 waiting for incremental sync to catch up: 2023-05-31T20:32:51.055 INFO:teuthology.orchestra.run.smithi099.stderr: local status: [00000000003.69.6, , 00000000002.67.6] 2023-05-31T20:32:51.055 INFO:teuthology.orchestra.run.smithi099.stderr: remote markers: [00000000003.69.6, 00000000002.23.6, 00000000002.67.6] 2023-05-31T20:32:56.057 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:32:56.050+0000 7f3958492780 1 waiting for incremental sync to catch up: 2023-05-31T20:32:56.057 INFO:teuthology.orchestra.run.smithi099.stderr: local status: [00000000003.69.6, , 00000000002.67.6] 2023-05-31T20:32:56.057 INFO:teuthology.orchestra.run.smithi099.stderr: remote markers: [00000000003.69.6, 00000000002.23.6, 00000000002.67.6] 2023-05-31T20:33:01.056 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:33:01.051+0000 7f3958492780 1 waiting for incremental sync to catch up: 2023-05-31T20:33:01.057 INFO:teuthology.orchestra.run.smithi099.stderr: local status: [00000000003.69.6, , 00000000002.67.6] 2023-05-31T20:33:01.057 INFO:teuthology.orchestra.run.smithi099.stderr: remote markers: [00000000003.69.6, 00000000002.23.6, 00000000002.67.6] 2023-05-31T20:33:06.057 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:33:06.052+0000 7f3958492780 1 waiting for incremental sync to catch up: 2023-05-31T20:33:06.058 INFO:teuthology.orchestra.run.smithi099.stderr: local status: [00000000003.69.6, , 00000000002.67.6] 2023-05-31T20:33:06.058 INFO:teuthology.orchestra.run.smithi099.stderr: remote markers: [00000000003.69.6, 00000000002.23.6, 00000000002.67.6] 2023-05-31T20:33:11.058 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:33:11.053+0000 7f3958492780 1 waiting for incremental sync to catch up: 2023-05-31T20:33:11.058 INFO:teuthology.orchestra.run.smithi099.stderr: local status: [00000000003.69.6, , 00000000002.67.6] 2023-05-31T20:33:11.058 INFO:teuthology.orchestra.run.smithi099.stderr: remote markers: [00000000003.69.6, 00000000002.23.6, 00000000002.67.6] 2023-05-31T20:33:16.059 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:33:16.054+0000 7f3958492780 1 waiting for incremental sync to catch up: 2023-05-31T20:33:16.059 INFO:teuthology.orchestra.run.smithi099.stderr: local status: [00000000003.69.6, , 00000000002.67.6] 2023-05-31T20:33:16.059 INFO:teuthology.orchestra.run.smithi099.stderr: remote markers: [00000000003.69.6, 00000000002.23.6, 00000000002.67.6] 2023-05-31T20:33:21.059 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:33:21.055+0000 7f3958492780 1 waiting for incremental sync to catch up: 2023-05-31T20:33:21.060 INFO:teuthology.orchestra.run.smithi099.stderr: local status: [00000000003.69.6, , 00000000002.67.6] 2023-05-31T20:33:21.060 INFO:teuthology.orchestra.run.smithi099.stderr: remote markers: [00000000003.69.6, 00000000002.23.6, 00000000002.67.6] 2023-05-31T20:33:26.060 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:33:26.056+0000 7f3958492780 1 waiting for incremental sync to catch up: 2023-05-31T20:33:26.060 INFO:teuthology.orchestra.run.smithi099.stderr: local status: [00000000003.69.6, , 00000000002.67.6] 2023-05-31T20:33:26.061 INFO:teuthology.orchestra.run.smithi099.stderr: remote markers: [00000000003.69.6, 00000000002.23.6, 00000000002.67.6] 2023-05-31T20:33:31.061 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:33:31.056+0000 7f3958492780 1 waiting for incremental sync to catch up: 2023-05-31T20:33:31.061 INFO:teuthology.orchestra.run.smithi099.stderr: local status: [00000000003.69.6, , 00000000002.67.6] 2023-05-31T20:33:31.062 INFO:teuthology.orchestra.run.smithi099.stderr: remote markers: [00000000003.69.6, 00000000002.23.6, 00000000002.67.6] 2023-05-31T20:33:36.062 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:33:36.056+0000 7f3958492780 1 waiting for incremental sync to catch up: 2023-05-31T20:33:36.062 INFO:teuthology.orchestra.run.smithi099.stderr: local status: [00000000003.69.6, , 00000000002.67.6] 2023-05-31T20:33:36.063 INFO:teuthology.orchestra.run.smithi099.stderr: remote markers: [00000000003.69.6, 00000000002.23.6, 00000000002.67.6] 2023-05-31T20:33:41.063 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:33:41.057+0000 7f3958492780 1 waiting for incremental sync to catch up: 2023-05-31T20:33:41.063 INFO:teuthology.orchestra.run.smithi099.stderr: local status: [00000000003.69.6, , 00000000002.67.6] 2023-05-31T20:33:41.063 INFO:teuthology.orchestra.run.smithi099.stderr: remote markers: [00000000003.69.6, 00000000002.23.6, 00000000002.67.6] 2023-05-31T20:33:46.063 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:33:46.058+0000 7f3958492780 1 waiting for incremental sync to catch up: 2023-05-31T20:33:46.064 INFO:teuthology.orchestra.run.smithi099.stderr: local status: [00000000003.69.6, , 00000000002.67.6] 2023-05-31T20:33:46.064 INFO:teuthology.orchestra.run.smithi099.stderr: remote markers: [00000000003.69.6, 00000000002.23.6, 00000000002.67.6] 2023-05-31T20:33:51.064 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:33:51.059+0000 7f3958492780 1 waiting for incremental sync to catch up: 2023-05-31T20:33:51.065 INFO:teuthology.orchestra.run.smithi099.stderr: local status: [00000000003.69.6, , 00000000002.67.6] 2023-05-31T20:33:51.065 INFO:teuthology.orchestra.run.smithi099.stderr: remote markers: [00000000003.69.6, 00000000002.23.6, 00000000002.67.6] 2023-05-31T20:33:56.065 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:33:56.060+0000 7f3958492780 1 waiting for incremental sync to catch up: 2023-05-31T20:33:56.066 INFO:teuthology.orchestra.run.smithi099.stderr: local status: [00000000003.69.6, , 00000000002.67.6] 2023-05-31T20:33:56.066 INFO:teuthology.orchestra.run.smithi099.stderr: remote markers: [00000000003.69.6, 00000000002.23.6, 00000000002.67.6] 2023-05-31T20:34:01.066 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:34:01.061+0000 7f3958492780 1 waiting for incremental sync to catch up: 2023-05-31T20:34:01.066 INFO:teuthology.orchestra.run.smithi099.stderr: local status: [00000000003.69.6, , 00000000002.67.6] 2023-05-31T20:34:01.067 INFO:teuthology.orchestra.run.smithi099.stderr: remote markers: [00000000003.69.6, 00000000002.23.6, 00000000002.67.6] 2023-05-31T20:34:06.067 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:34:06.063+0000 7f3958492780 1 waiting for incremental sync to catch up: 2023-05-31T20:34:06.067 INFO:teuthology.orchestra.run.smithi099.stderr: local status: [00000000003.69.6, , 00000000002.67.6] 2023-05-31T20:34:06.068 INFO:teuthology.orchestra.run.smithi099.stderr: remote markers: [00000000003.69.6, 00000000002.23.6, 00000000002.67.6] 2023-05-31T20:34:11.068 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:34:11.063+0000 7f3958492780 1 waiting for incremental sync to catch up: 2023-05-31T20:34:11.068 INFO:teuthology.orchestra.run.smithi099.stderr: local status: [00000000003.69.6, , 00000000002.67.6] 2023-05-31T20:34:11.068 INFO:teuthology.orchestra.run.smithi099.stderr: remote markers: [00000000003.69.6, 00000000002.23.6, 00000000002.67.6] 2023-05-31T20:34:16.069 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:34:16.063+0000 7f3958492780 1 waiting for incremental sync to catch up: 2023-05-31T20:34:16.069 INFO:teuthology.orchestra.run.smithi099.stderr: local status: [00000000003.69.6, , 00000000002.67.6] 2023-05-31T20:34:16.069 INFO:teuthology.orchestra.run.smithi099.stderr: remote markers: [00000000003.69.6, 00000000002.23.6, 00000000002.67.6] 2023-05-31T20:34:21.069 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:34:21.064+0000 7f3958492780 1 waiting for incremental sync to catch up: 2023-05-31T20:34:21.070 INFO:teuthology.orchestra.run.smithi099.stderr: local status: [00000000003.69.6, , 00000000002.67.6] 2023-05-31T20:34:21.070 INFO:teuthology.orchestra.run.smithi099.stderr: remote markers: [00000000003.69.6, 00000000002.23.6, 00000000002.67.6] 2023-05-31T20:34:26.071 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:34:26.065+0000 7f3958492780 1 waiting for incremental sync to catch up: 2023-05-31T20:34:26.071 INFO:teuthology.orchestra.run.smithi099.stderr: local status: [00000000003.69.6, , 00000000002.67.6] 2023-05-31T20:34:26.071 INFO:teuthology.orchestra.run.smithi099.stderr: remote markers: [00000000003.69.6, 00000000002.23.6, 00000000002.67.6] 2023-05-31T20:34:31.071 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:34:31.066+0000 7f3958492780 1 waiting for incremental sync to catch up: 2023-05-31T20:34:31.071 INFO:teuthology.orchestra.run.smithi099.stderr: local status: [00000000003.69.6, , 00000000002.67.6] 2023-05-31T20:34:31.071 INFO:teuthology.orchestra.run.smithi099.stderr: remote markers: [00000000003.69.6, 00000000002.23.6, 00000000002.67.6] 2023-05-31T20:34:36.072 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:34:36.067+0000 7f3958492780 1 bucket sync caught up with source: 2023-05-31T20:34:36.072 INFO:teuthology.orchestra.run.smithi099.stderr: local status: [00000000003.69.6, 00000000002.23.6, 00000000002.67.6] 2023-05-31T20:34:36.072 INFO:teuthology.orchestra.run.smithi099.stderr: remote markers: [00000000003.69.6, 00000000002.23.6, 00000000002.67.6] 2023-05-31T20:34:36.072 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:34:36.067+0000 7f3958492780 0 bucket checkpoint complete 2023-05-31T20:34:36.093 DEBUG:rgw_multi.tests:bucket checkpoint: source=a2 target=a1 bucket=pklxkd-56 2023-05-31T20:34:36.094 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'pklxkd-56', '--source-zone', 'a2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-05-31T20:34:36.095 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'pklxkd-56', '--source-zone', 'a2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-05-31T20:34:36.095 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket pklxkd-56 --source-zone a2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2023-05-31T20:34:36.157 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2023-05-31T20:34:36.158 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2023-05-31T20:34:36.195 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:34:36.191+0000 7f8af9d42780 1 waiting for incremental sync to catch up: 2023-05-31T20:34:36.196 INFO:teuthology.orchestra.run.smithi099.stderr: local status: [, 00000000001.2276.6, , , 00000000001.998.6, 00000000001.3568.6, , , 00000000002.361.6, , 00000000002.2017.6] 2023-05-31T20:34:36.196 INFO:teuthology.orchestra.run.smithi099.stderr: remote markers: [, 00000000002.2282.6, , , 00000000001.998.6, 00000000001.3568.6, , , 00000000002.361.6, , 00000000002.2017.6] 2023-05-31T20:34:41.196 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:34:41.192+0000 7f8af9d42780 1 waiting for incremental sync to catch up: 2023-05-31T20:34:41.197 INFO:teuthology.orchestra.run.smithi099.stderr: local status: [, 00000000001.2276.6, , , 00000000001.998.6, 00000000001.3568.6, , , 00000000002.361.6, , 00000000002.2017.6] 2023-05-31T20:34:41.197 INFO:teuthology.orchestra.run.smithi099.stderr: remote markers: [, 00000000002.2282.6, , , 00000000001.998.6, 00000000001.3568.6, , , 00000000002.361.6, , 00000000002.2017.6] 2023-05-31T20:34:46.198 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:34:46.193+0000 7f8af9d42780 1 waiting for incremental sync to catch up: 2023-05-31T20:34:46.198 INFO:teuthology.orchestra.run.smithi099.stderr: local status: [, 00000000001.2276.6, , , 00000000001.998.6, 00000000001.3568.6, , , 00000000002.361.6, , 00000000002.2017.6] 2023-05-31T20:34:46.199 INFO:teuthology.orchestra.run.smithi099.stderr: remote markers: [, 00000000002.2282.6, , , 00000000001.998.6, 00000000001.3568.6, , , 00000000002.361.6, , 00000000002.2017.6] 2023-05-31T20:34:51.200 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:34:51.194+0000 7f8af9d42780 1 bucket sync caught up with source: 2023-05-31T20:34:51.200 INFO:teuthology.orchestra.run.smithi099.stderr: local status: [, 00000000002.2282.6, , , 00000000001.998.6, 00000000001.3568.6, , , 00000000002.361.6, , 00000000002.2017.6] 2023-05-31T20:34:51.200 INFO:teuthology.orchestra.run.smithi099.stderr: remote markers: [, 00000000002.2282.6, , , 00000000001.998.6, 00000000001.3568.6, , , 00000000002.361.6, , 00000000002.2017.6] 2023-05-31T20:34:51.200 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:34:51.194+0000 7f8af9d42780 0 bucket checkpoint complete 2023-05-31T20:34:51.219 INFO:tasks.rgw_multi.zone_rados:comparing bucket=pklxkd-56 zones={a2, a1} 2023-05-31T20:34:51.220 DEBUG:boto:path=/pklxkd-56/ 2023-05-31T20:34:51.220 DEBUG:boto:auth_path=/pklxkd-56/ 2023-05-31T20:34:51.221 DEBUG:boto:Method: HEAD 2023-05-31T20:34:51.221 DEBUG:boto:Path: /pklxkd-56/ 2023-05-31T20:34:51.221 DEBUG:boto:Data: 2023-05-31T20:34:51.222 DEBUG:boto:Headers: {} 2023-05-31T20:34:51.222 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:34:51.222 DEBUG:boto:Port: 8001 2023-05-31T20:34:51.223 DEBUG:boto:Params: {} 2023-05-31T20:34:51.223 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8001} 2023-05-31T20:34:51.224 DEBUG:boto:Token: None 2023-05-31T20:34:51.225 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:34:51 GMT /pklxkd-56/ 2023-05-31T20:34:51.225 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:rHOjlTq+o8N4grZnkvlPyjdwORk= 2023-05-31T20:34:51.226 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:34:51 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:rHOjlTq+o8N4grZnkvlPyjdwORk=', 'Content-Length': '0'} 2023-05-31T20:34:51.230 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '8'), ('X-RGW-Bytes-Used', '24'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000005259f6a25737cd8a-006477af6b-162e-a2'), ('Content-Length', '0'), ('Date', 'Wed, 31 May 2023 20:34:51 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:34:51.231 DEBUG:boto:path=/pklxkd-56/ 2023-05-31T20:34:51.231 DEBUG:boto:auth_path=/pklxkd-56/ 2023-05-31T20:34:51.232 DEBUG:boto:Method: HEAD 2023-05-31T20:34:51.232 DEBUG:boto:Path: /pklxkd-56/ 2023-05-31T20:34:51.232 DEBUG:boto:Data: 2023-05-31T20:34:51.233 DEBUG:boto:Headers: {} 2023-05-31T20:34:51.233 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:34:51.234 DEBUG:boto:Port: 8000 2023-05-31T20:34:51.234 DEBUG:boto:Params: {} 2023-05-31T20:34:51.235 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2023-05-31T20:34:51.236 DEBUG:boto:Token: None 2023-05-31T20:34:51.237 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:34:51 GMT /pklxkd-56/ 2023-05-31T20:34:51.238 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:rHOjlTq+o8N4grZnkvlPyjdwORk= 2023-05-31T20:34:51.239 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:34:51 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:rHOjlTq+o8N4grZnkvlPyjdwORk=', 'Content-Length': '0'} 2023-05-31T20:34:51.242 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '8'), ('X-RGW-Bytes-Used', '24'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000004940b1524771896f-006477af6b-1630-a1'), ('Content-Length', '0'), ('Date', 'Wed, 31 May 2023 20:34:51 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:34:51.243 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2023-05-31T20:34:51.244 DEBUG:boto:path=/pklxkd-56/ 2023-05-31T20:34:51.244 DEBUG:boto:auth_path=/pklxkd-56/ 2023-05-31T20:34:51.245 DEBUG:boto:path=/pklxkd-56/?versions&max-keys=999 2023-05-31T20:34:51.245 DEBUG:boto:auth_path=/pklxkd-56/?versions&max-keys=999 2023-05-31T20:34:51.246 DEBUG:boto:Method: GET 2023-05-31T20:34:51.247 DEBUG:boto:Path: /pklxkd-56/?versions&max-keys=999 2023-05-31T20:34:51.247 DEBUG:boto:Data: 2023-05-31T20:34:51.248 DEBUG:boto:Headers: {} 2023-05-31T20:34:51.248 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:34:51.248 DEBUG:boto:Port: 8001 2023-05-31T20:34:51.249 DEBUG:boto:Params: {} 2023-05-31T20:34:51.249 DEBUG:boto:Token: None 2023-05-31T20:34:51.250 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:34:51 GMT /pklxkd-56/?versions 2023-05-31T20:34:51.251 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:CjrqqIr+u2EqUASdtmvBhk2dlMw= 2023-05-31T20:34:51.252 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:34:51 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:CjrqqIr+u2EqUASdtmvBhk2dlMw=', 'Content-Length': '0'} 2023-05-31T20:34:51.255 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000007727ea455e6d0b0e-006477af6b-162e-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:34:51 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:34:51.255 DEBUG:boto:b'pklxkd-56999falseanulltrue2023-05-31T20:29:54.205Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalbnulltrue2023-05-31T20:29:54.259Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalcnulltrue2023-05-31T20:29:54.281Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormaldnulltrue2023-05-31T20:29:54.306Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalenulltrue2023-05-31T20:30:05.784Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalfnulltrue2023-05-31T20:30:05.809Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalgnulltrue2023-05-31T20:30:05.841Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalhnulltrue2023-05-31T20:30:05.906Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:34:51.256 DEBUG:tasks.rgw_multi.zone_rados:o=a 2023-05-31T20:34:51.257 DEBUG:tasks.rgw_multi.zone_rados:o=b 2023-05-31T20:34:51.257 DEBUG:tasks.rgw_multi.zone_rados:o=c 2023-05-31T20:34:51.257 DEBUG:tasks.rgw_multi.zone_rados:o=d 2023-05-31T20:34:51.258 DEBUG:tasks.rgw_multi.zone_rados:o=e 2023-05-31T20:34:51.258 DEBUG:tasks.rgw_multi.zone_rados:o=f 2023-05-31T20:34:51.258 DEBUG:tasks.rgw_multi.zone_rados:o=g 2023-05-31T20:34:51.259 DEBUG:tasks.rgw_multi.zone_rados:o=h 2023-05-31T20:34:51.259 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2023-05-31T20:34:51.259 DEBUG:boto:path=/pklxkd-56/ 2023-05-31T20:34:51.260 DEBUG:boto:auth_path=/pklxkd-56/ 2023-05-31T20:34:51.260 DEBUG:boto:path=/pklxkd-56/?versions&max-keys=999 2023-05-31T20:34:51.261 DEBUG:boto:auth_path=/pklxkd-56/?versions&max-keys=999 2023-05-31T20:34:51.261 DEBUG:boto:Method: GET 2023-05-31T20:34:51.261 DEBUG:boto:Path: /pklxkd-56/?versions&max-keys=999 2023-05-31T20:34:51.262 DEBUG:boto:Data: 2023-05-31T20:34:51.262 DEBUG:boto:Headers: {} 2023-05-31T20:34:51.262 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:34:51.263 DEBUG:boto:Port: 8000 2023-05-31T20:34:51.263 DEBUG:boto:Params: {} 2023-05-31T20:34:51.264 DEBUG:boto:Token: None 2023-05-31T20:34:51.264 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:34:51 GMT /pklxkd-56/?versions 2023-05-31T20:34:51.264 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:CjrqqIr+u2EqUASdtmvBhk2dlMw= 2023-05-31T20:34:51.265 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:34:51 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:CjrqqIr+u2EqUASdtmvBhk2dlMw=', 'Content-Length': '0'} 2023-05-31T20:34:51.267 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000269ef5f601d0a605-006477af6b-1630-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:34:51 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:34:51.268 DEBUG:boto:b'pklxkd-56999falseanulltrue2023-05-31T20:29:54.205Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalbnulltrue2023-05-31T20:29:54.259Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalcnulltrue2023-05-31T20:29:54.281Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormaldnulltrue2023-05-31T20:29:54.306Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalenulltrue2023-05-31T20:30:05.784Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalfnulltrue2023-05-31T20:30:05.809Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalgnulltrue2023-05-31T20:30:05.841Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalhnulltrue2023-05-31T20:30:05.906Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:34:51.269 DEBUG:tasks.rgw_multi.zone_rados:o=a 2023-05-31T20:34:51.269 DEBUG:tasks.rgw_multi.zone_rados:o=b 2023-05-31T20:34:51.269 DEBUG:tasks.rgw_multi.zone_rados:o=c 2023-05-31T20:34:51.270 DEBUG:tasks.rgw_multi.zone_rados:o=d 2023-05-31T20:34:51.270 DEBUG:tasks.rgw_multi.zone_rados:o=e 2023-05-31T20:34:51.270 DEBUG:tasks.rgw_multi.zone_rados:o=f 2023-05-31T20:34:51.271 DEBUG:tasks.rgw_multi.zone_rados:o=g 2023-05-31T20:34:51.271 DEBUG:tasks.rgw_multi.zone_rados:o=h 2023-05-31T20:34:51.272 DEBUG:boto:path=/pklxkd-56/ 2023-05-31T20:34:51.273 DEBUG:boto:auth_path=/pklxkd-56/ 2023-05-31T20:34:51.274 DEBUG:boto:path=/pklxkd-56/?versions&max-keys=999 2023-05-31T20:34:51.275 DEBUG:boto:auth_path=/pklxkd-56/?versions&max-keys=999 2023-05-31T20:34:51.276 DEBUG:boto:Method: GET 2023-05-31T20:34:51.277 DEBUG:boto:Path: /pklxkd-56/?versions&max-keys=999 2023-05-31T20:34:51.278 DEBUG:boto:Data: 2023-05-31T20:34:51.278 DEBUG:boto:Headers: {} 2023-05-31T20:34:51.279 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:34:51.279 DEBUG:boto:Port: 8001 2023-05-31T20:34:51.280 DEBUG:boto:Params: {} 2023-05-31T20:34:51.281 DEBUG:boto:Token: None 2023-05-31T20:34:51.282 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:34:51 GMT /pklxkd-56/?versions 2023-05-31T20:34:51.282 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:CjrqqIr+u2EqUASdtmvBhk2dlMw= 2023-05-31T20:34:51.283 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:34:51 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:CjrqqIr+u2EqUASdtmvBhk2dlMw=', 'Content-Length': '0'} 2023-05-31T20:34:51.286 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000017afd22ad15f1b60-006477af6b-162e-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:34:51 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:34:51.287 DEBUG:boto:b'pklxkd-56999falseanulltrue2023-05-31T20:29:54.205Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalbnulltrue2023-05-31T20:29:54.259Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalcnulltrue2023-05-31T20:29:54.281Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormaldnulltrue2023-05-31T20:29:54.306Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalenulltrue2023-05-31T20:30:05.784Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalfnulltrue2023-05-31T20:30:05.809Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalgnulltrue2023-05-31T20:30:05.841Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalhnulltrue2023-05-31T20:30:05.906Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:34:51.288 DEBUG:boto:path=/pklxkd-56/ 2023-05-31T20:34:51.288 DEBUG:boto:auth_path=/pklxkd-56/ 2023-05-31T20:34:51.289 DEBUG:boto:path=/pklxkd-56/?versions&max-keys=999 2023-05-31T20:34:51.289 DEBUG:boto:auth_path=/pklxkd-56/?versions&max-keys=999 2023-05-31T20:34:51.289 DEBUG:boto:Method: GET 2023-05-31T20:34:51.290 DEBUG:boto:Path: /pklxkd-56/?versions&max-keys=999 2023-05-31T20:34:51.290 DEBUG:boto:Data: 2023-05-31T20:34:51.290 DEBUG:boto:Headers: {} 2023-05-31T20:34:51.291 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:34:51.291 DEBUG:boto:Port: 8000 2023-05-31T20:34:51.292 DEBUG:boto:Params: {} 2023-05-31T20:34:51.292 DEBUG:boto:Token: None 2023-05-31T20:34:51.292 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:34:51 GMT /pklxkd-56/?versions 2023-05-31T20:34:51.293 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:CjrqqIr+u2EqUASdtmvBhk2dlMw= 2023-05-31T20:34:51.293 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:34:51 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:CjrqqIr+u2EqUASdtmvBhk2dlMw=', 'Content-Length': '0'} 2023-05-31T20:34:51.295 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000aa297e466b9f4f9c-006477af6b-1630-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:34:51 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:34:51.296 DEBUG:boto:b'pklxkd-56999falseanulltrue2023-05-31T20:29:54.205Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalbnulltrue2023-05-31T20:29:54.259Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalcnulltrue2023-05-31T20:29:54.281Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormaldnulltrue2023-05-31T20:29:54.306Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalenulltrue2023-05-31T20:30:05.784Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalfnulltrue2023-05-31T20:30:05.809Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalgnulltrue2023-05-31T20:30:05.841Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalhnulltrue2023-05-31T20:30:05.906Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:34:51.297 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=a 2023-05-31T20:34:51.297 DEBUG:boto:path=/pklxkd-56/a 2023-05-31T20:34:51.298 DEBUG:boto:auth_path=/pklxkd-56/a 2023-05-31T20:34:51.298 DEBUG:boto:path=/pklxkd-56/a?versionId=null 2023-05-31T20:34:51.299 DEBUG:boto:auth_path=/pklxkd-56/a?versionId=null 2023-05-31T20:34:51.299 DEBUG:boto:Method: GET 2023-05-31T20:34:51.299 DEBUG:boto:Path: /pklxkd-56/a?versionId=null 2023-05-31T20:34:51.300 DEBUG:boto:Data: 2023-05-31T20:34:51.300 DEBUG:boto:Headers: {} 2023-05-31T20:34:51.301 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:34:51.301 DEBUG:boto:Port: 8001 2023-05-31T20:34:51.301 DEBUG:boto:Params: {} 2023-05-31T20:34:51.302 DEBUG:boto:Token: None 2023-05-31T20:34:51.302 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:34:51 GMT /pklxkd-56/a?versionId=null 2023-05-31T20:34:51.303 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:CaYugzkqWxvgcpFJeuA8/X0BfzE= 2023-05-31T20:34:51.303 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:34:51 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:CaYugzkqWxvgcpFJeuA8/X0BfzE=', 'Content-Length': '0'} 2023-05-31T20:34:51.305 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:29:54 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000021e9fd3c8a39cf12-006477af6b-162e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:34:51 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:34:51.306 DEBUG:boto:path=/pklxkd-56/a 2023-05-31T20:34:51.306 DEBUG:boto:auth_path=/pklxkd-56/a 2023-05-31T20:34:51.307 DEBUG:boto:path=/pklxkd-56/a?versionId=null 2023-05-31T20:34:51.307 DEBUG:boto:auth_path=/pklxkd-56/a?versionId=null 2023-05-31T20:34:51.308 DEBUG:boto:Method: GET 2023-05-31T20:34:51.308 DEBUG:boto:Path: /pklxkd-56/a?versionId=null 2023-05-31T20:34:51.308 DEBUG:boto:Data: 2023-05-31T20:34:51.309 DEBUG:boto:Headers: {} 2023-05-31T20:34:51.309 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:34:51.310 DEBUG:boto:Port: 8000 2023-05-31T20:34:51.310 DEBUG:boto:Params: {} 2023-05-31T20:34:51.310 DEBUG:boto:Token: None 2023-05-31T20:34:51.311 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:34:51 GMT /pklxkd-56/a?versionId=null 2023-05-31T20:34:51.311 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:CaYugzkqWxvgcpFJeuA8/X0BfzE= 2023-05-31T20:34:51.312 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:34:51 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:CaYugzkqWxvgcpFJeuA8/X0BfzE=', 'Content-Length': '0'} 2023-05-31T20:34:51.314 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:29:54 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e898571a309f3c3b-006477af6b-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:34:51 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:34:51.314 DEBUG:boto:path=/pklxkd-56/a 2023-05-31T20:34:51.315 DEBUG:boto:auth_path=/pklxkd-56/a 2023-05-31T20:34:51.315 DEBUG:boto:path=/pklxkd-56/a?versionId=null 2023-05-31T20:34:51.315 DEBUG:boto:auth_path=/pklxkd-56/a?versionId=null 2023-05-31T20:34:51.316 DEBUG:boto:Method: HEAD 2023-05-31T20:34:51.316 DEBUG:boto:Path: /pklxkd-56/a?versionId=null 2023-05-31T20:34:51.317 DEBUG:boto:Data: 2023-05-31T20:34:51.317 DEBUG:boto:Headers: {} 2023-05-31T20:34:51.317 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:34:51.318 DEBUG:boto:Port: 8001 2023-05-31T20:34:51.318 DEBUG:boto:Params: {} 2023-05-31T20:34:51.319 DEBUG:boto:Token: None 2023-05-31T20:34:51.319 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:34:51 GMT /pklxkd-56/a?versionId=null 2023-05-31T20:34:51.319 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:ZcotwG46Rjc2zQg+0Tf+jzHMxdg= 2023-05-31T20:34:51.320 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:34:51 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:ZcotwG46Rjc2zQg+0Tf+jzHMxdg=', 'Content-Length': '0'} 2023-05-31T20:34:51.322 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:29:54 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ebe904caeed307cc-006477af6b-162e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:34:51 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:34:51.323 DEBUG:boto:path=/pklxkd-56/a 2023-05-31T20:34:51.323 DEBUG:boto:auth_path=/pklxkd-56/a 2023-05-31T20:34:51.324 DEBUG:boto:path=/pklxkd-56/a?versionId=null 2023-05-31T20:34:51.324 DEBUG:boto:auth_path=/pklxkd-56/a?versionId=null 2023-05-31T20:34:51.325 DEBUG:boto:Method: HEAD 2023-05-31T20:34:51.325 DEBUG:boto:Path: /pklxkd-56/a?versionId=null 2023-05-31T20:34:51.327 DEBUG:boto:Data: 2023-05-31T20:34:51.327 DEBUG:boto:Headers: {} 2023-05-31T20:34:51.328 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:34:51.328 DEBUG:boto:Port: 8000 2023-05-31T20:34:51.328 DEBUG:boto:Params: {} 2023-05-31T20:34:51.329 DEBUG:boto:Token: None 2023-05-31T20:34:51.329 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:34:51 GMT /pklxkd-56/a?versionId=null 2023-05-31T20:34:51.330 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:ZcotwG46Rjc2zQg+0Tf+jzHMxdg= 2023-05-31T20:34:51.330 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:34:51 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:ZcotwG46Rjc2zQg+0Tf+jzHMxdg=', 'Content-Length': '0'} 2023-05-31T20:34:51.332 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:29:54 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000075693de64b1a98ce-006477af6b-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:34:51 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:34:51.333 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=a 2023-05-31T20:34:51.333 DEBUG:boto:path=/pklxkd-56/a 2023-05-31T20:34:51.333 DEBUG:boto:auth_path=/pklxkd-56/a 2023-05-31T20:34:51.334 DEBUG:boto:path=/pklxkd-56/a?versionId=null 2023-05-31T20:34:51.334 DEBUG:boto:auth_path=/pklxkd-56/a?versionId=null 2023-05-31T20:34:51.336 DEBUG:boto:Method: GET 2023-05-31T20:34:51.336 DEBUG:boto:Path: /pklxkd-56/a?versionId=null 2023-05-31T20:34:51.337 DEBUG:boto:Data: 2023-05-31T20:34:51.337 DEBUG:boto:Headers: {} 2023-05-31T20:34:51.338 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:34:51.338 DEBUG:boto:Port: 8001 2023-05-31T20:34:51.338 DEBUG:boto:Params: {} 2023-05-31T20:34:51.339 DEBUG:boto:Token: None 2023-05-31T20:34:51.339 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:34:51 GMT /pklxkd-56/a?versionId=null 2023-05-31T20:34:51.340 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:CaYugzkqWxvgcpFJeuA8/X0BfzE= 2023-05-31T20:34:51.340 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:34:51 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:CaYugzkqWxvgcpFJeuA8/X0BfzE=', 'Content-Length': '0'} 2023-05-31T20:34:51.342 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:29:54 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003a9ef7f2cad5c4ab-006477af6b-162e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:34:51 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:34:51.343 DEBUG:boto:path=/pklxkd-56/a 2023-05-31T20:34:51.343 DEBUG:boto:auth_path=/pklxkd-56/a 2023-05-31T20:34:51.344 DEBUG:boto:path=/pklxkd-56/a?versionId=null 2023-05-31T20:34:51.344 DEBUG:boto:auth_path=/pklxkd-56/a?versionId=null 2023-05-31T20:34:51.344 DEBUG:boto:Method: GET 2023-05-31T20:34:51.345 DEBUG:boto:Path: /pklxkd-56/a?versionId=null 2023-05-31T20:34:51.345 DEBUG:boto:Data: 2023-05-31T20:34:51.345 DEBUG:boto:Headers: {} 2023-05-31T20:34:51.346 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:34:51.346 DEBUG:boto:Port: 8000 2023-05-31T20:34:51.346 DEBUG:boto:Params: {} 2023-05-31T20:34:51.347 DEBUG:boto:Token: None 2023-05-31T20:34:51.347 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:34:51 GMT /pklxkd-56/a?versionId=null 2023-05-31T20:34:51.348 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:CaYugzkqWxvgcpFJeuA8/X0BfzE= 2023-05-31T20:34:51.348 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:34:51 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:CaYugzkqWxvgcpFJeuA8/X0BfzE=', 'Content-Length': '0'} 2023-05-31T20:34:51.350 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:29:54 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000021ff6f2209b52bff-006477af6b-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:34:51 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:34:51.351 DEBUG:boto:path=/pklxkd-56/ 2023-05-31T20:34:51.351 DEBUG:boto:auth_path=/pklxkd-56/ 2023-05-31T20:34:51.351 DEBUG:boto:path=/pklxkd-56/?prefix=a 2023-05-31T20:34:51.352 DEBUG:boto:auth_path=/pklxkd-56/?prefix=a 2023-05-31T20:34:51.352 DEBUG:boto:Method: GET 2023-05-31T20:34:51.353 DEBUG:boto:Path: /pklxkd-56/?prefix=a 2023-05-31T20:34:51.353 DEBUG:boto:Data: 2023-05-31T20:34:51.353 DEBUG:boto:Headers: {} 2023-05-31T20:34:51.354 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:34:51.354 DEBUG:boto:Port: 8001 2023-05-31T20:34:51.355 DEBUG:boto:Params: {} 2023-05-31T20:34:51.355 DEBUG:boto:Token: None 2023-05-31T20:34:51.355 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:34:51 GMT /pklxkd-56/ 2023-05-31T20:34:51.356 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:sCd5EIDPFEDQoe36mG1HVLn+PZs= 2023-05-31T20:34:51.356 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:34:51 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:sCd5EIDPFEDQoe36mG1HVLn+PZs=', 'Content-Length': '0'} 2023-05-31T20:34:51.359 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000005b47926d829fc060-006477af6b-162e-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:34:51 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:34:51.359 DEBUG:boto:b'pklxkd-56a1000falsea2023-05-31T20:29:54.205Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:34:51.360 DEBUG:boto:path=/pklxkd-56/ 2023-05-31T20:34:51.360 DEBUG:boto:auth_path=/pklxkd-56/ 2023-05-31T20:34:51.361 DEBUG:boto:path=/pklxkd-56/?prefix=a 2023-05-31T20:34:51.361 DEBUG:boto:auth_path=/pklxkd-56/?prefix=a 2023-05-31T20:34:51.361 DEBUG:boto:Method: GET 2023-05-31T20:34:51.362 DEBUG:boto:Path: /pklxkd-56/?prefix=a 2023-05-31T20:34:51.362 DEBUG:boto:Data: 2023-05-31T20:34:51.362 DEBUG:boto:Headers: {} 2023-05-31T20:34:51.363 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:34:51.363 DEBUG:boto:Port: 8000 2023-05-31T20:34:51.364 DEBUG:boto:Params: {} 2023-05-31T20:34:51.364 DEBUG:boto:Token: None 2023-05-31T20:34:51.364 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:34:51 GMT /pklxkd-56/ 2023-05-31T20:34:51.365 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:sCd5EIDPFEDQoe36mG1HVLn+PZs= 2023-05-31T20:34:51.365 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:34:51 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:sCd5EIDPFEDQoe36mG1HVLn+PZs=', 'Content-Length': '0'} 2023-05-31T20:34:51.367 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000061bd85a3bc2cf842-006477af6b-1630-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:34:51 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:34:51.368 DEBUG:boto:b'pklxkd-56a1000falsea2023-05-31T20:29:54.205Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:34:51.369 DEBUG:boto:path=/pklxkd-56/a 2023-05-31T20:34:51.369 DEBUG:boto:auth_path=/pklxkd-56/a 2023-05-31T20:34:51.369 DEBUG:boto:Method: HEAD 2023-05-31T20:34:51.370 DEBUG:boto:Path: /pklxkd-56/a 2023-05-31T20:34:51.370 DEBUG:boto:Data: 2023-05-31T20:34:51.370 DEBUG:boto:Headers: {} 2023-05-31T20:34:51.371 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:34:51.371 DEBUG:boto:Port: 8001 2023-05-31T20:34:51.371 DEBUG:boto:Params: {} 2023-05-31T20:34:51.372 DEBUG:boto:Token: None 2023-05-31T20:34:51.372 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:34:51 GMT /pklxkd-56/a 2023-05-31T20:34:51.373 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:NTwTpJeY9BmU0Ka2RUPY+1ieaMo= 2023-05-31T20:34:51.373 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:34:51 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:NTwTpJeY9BmU0Ka2RUPY+1ieaMo=', 'Content-Length': '0'} 2023-05-31T20:34:51.375 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:29:54 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009222dd02952e8ba9-006477af6b-162e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:34:51 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:34:51.376 DEBUG:boto:path=/pklxkd-56/a 2023-05-31T20:34:51.376 DEBUG:boto:auth_path=/pklxkd-56/a 2023-05-31T20:34:51.376 DEBUG:boto:Method: HEAD 2023-05-31T20:34:51.377 DEBUG:boto:Path: /pklxkd-56/a 2023-05-31T20:34:51.378 DEBUG:boto:Data: 2023-05-31T20:34:51.378 DEBUG:boto:Headers: {} 2023-05-31T20:34:51.378 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:34:51.379 DEBUG:boto:Port: 8000 2023-05-31T20:34:51.379 DEBUG:boto:Params: {} 2023-05-31T20:34:51.379 DEBUG:boto:Token: None 2023-05-31T20:34:51.380 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:34:51 GMT /pklxkd-56/a 2023-05-31T20:34:51.380 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:NTwTpJeY9BmU0Ka2RUPY+1ieaMo= 2023-05-31T20:34:51.381 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:34:51 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:NTwTpJeY9BmU0Ka2RUPY+1ieaMo=', 'Content-Length': '0'} 2023-05-31T20:34:51.382 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:29:54 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e1d9741e8f4ae61d-006477af6b-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:34:51 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:34:51.383 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=a 2023-05-31T20:34:51.383 DEBUG:boto:path=/pklxkd-56/a 2023-05-31T20:34:51.384 DEBUG:boto:auth_path=/pklxkd-56/a 2023-05-31T20:34:51.384 DEBUG:boto:Method: GET 2023-05-31T20:34:51.384 DEBUG:boto:Path: /pklxkd-56/a 2023-05-31T20:34:51.385 DEBUG:boto:Data: 2023-05-31T20:34:51.385 DEBUG:boto:Headers: {} 2023-05-31T20:34:51.386 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:34:51.386 DEBUG:boto:Port: 8001 2023-05-31T20:34:51.386 DEBUG:boto:Params: {} 2023-05-31T20:34:51.387 DEBUG:boto:Token: None 2023-05-31T20:34:51.387 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:34:51 GMT /pklxkd-56/a 2023-05-31T20:34:51.387 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:1G0h5s+fQ+go3sdJDDD88C/mA5E= 2023-05-31T20:34:51.388 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:34:51 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:1G0h5s+fQ+go3sdJDDD88C/mA5E=', 'Content-Length': '0'} 2023-05-31T20:34:51.390 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:29:54 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000190a407ffbf88055-006477af6b-162e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:34:51 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:34:51.390 DEBUG:boto:path=/pklxkd-56/a 2023-05-31T20:34:51.391 DEBUG:boto:auth_path=/pklxkd-56/a 2023-05-31T20:34:51.391 DEBUG:boto:Method: GET 2023-05-31T20:34:51.392 DEBUG:boto:Path: /pklxkd-56/a 2023-05-31T20:34:51.392 DEBUG:boto:Data: 2023-05-31T20:34:51.392 DEBUG:boto:Headers: {} 2023-05-31T20:34:51.393 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:34:51.393 DEBUG:boto:Port: 8000 2023-05-31T20:34:51.393 DEBUG:boto:Params: {} 2023-05-31T20:34:51.394 DEBUG:boto:Token: None 2023-05-31T20:34:51.394 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:34:51 GMT /pklxkd-56/a 2023-05-31T20:34:51.395 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:1G0h5s+fQ+go3sdJDDD88C/mA5E= 2023-05-31T20:34:51.395 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:34:51 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:1G0h5s+fQ+go3sdJDDD88C/mA5E=', 'Content-Length': '0'} 2023-05-31T20:34:51.397 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:29:54 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000cad0250da323feec-006477af6b-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:34:51 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:34:51.398 DEBUG:boto:path=/pklxkd-56/ 2023-05-31T20:34:51.398 DEBUG:boto:auth_path=/pklxkd-56/ 2023-05-31T20:34:51.399 DEBUG:boto:path=/pklxkd-56/?prefix=a 2023-05-31T20:34:51.399 DEBUG:boto:auth_path=/pklxkd-56/?prefix=a 2023-05-31T20:34:51.399 DEBUG:boto:Method: GET 2023-05-31T20:34:51.400 DEBUG:boto:Path: /pklxkd-56/?prefix=a 2023-05-31T20:34:51.400 DEBUG:boto:Data: 2023-05-31T20:34:51.400 DEBUG:boto:Headers: {} 2023-05-31T20:34:51.401 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:34:51.401 DEBUG:boto:Port: 8001 2023-05-31T20:34:51.402 DEBUG:boto:Params: {} 2023-05-31T20:34:51.402 DEBUG:boto:Token: None 2023-05-31T20:34:51.402 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:34:51 GMT /pklxkd-56/ 2023-05-31T20:34:51.403 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:sCd5EIDPFEDQoe36mG1HVLn+PZs= 2023-05-31T20:34:51.403 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:34:51 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:sCd5EIDPFEDQoe36mG1HVLn+PZs=', 'Content-Length': '0'} 2023-05-31T20:34:51.406 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000da24d725e8c2133b-006477af6b-162e-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:34:51 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:34:51.406 DEBUG:boto:b'pklxkd-56a1000falsea2023-05-31T20:29:54.205Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:34:51.407 DEBUG:boto:path=/pklxkd-56/ 2023-05-31T20:34:51.407 DEBUG:boto:auth_path=/pklxkd-56/ 2023-05-31T20:34:51.408 DEBUG:boto:path=/pklxkd-56/?prefix=a 2023-05-31T20:34:51.408 DEBUG:boto:auth_path=/pklxkd-56/?prefix=a 2023-05-31T20:34:51.409 DEBUG:boto:Method: GET 2023-05-31T20:34:51.411 DEBUG:boto:Path: /pklxkd-56/?prefix=a 2023-05-31T20:34:51.412 DEBUG:boto:Data: 2023-05-31T20:34:51.412 DEBUG:boto:Headers: {} 2023-05-31T20:34:51.413 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:34:51.413 DEBUG:boto:Port: 8000 2023-05-31T20:34:51.413 DEBUG:boto:Params: {} 2023-05-31T20:34:51.414 DEBUG:boto:Token: None 2023-05-31T20:34:51.414 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:34:51 GMT /pklxkd-56/ 2023-05-31T20:34:51.415 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:sCd5EIDPFEDQoe36mG1HVLn+PZs= 2023-05-31T20:34:51.415 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:34:51 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:sCd5EIDPFEDQoe36mG1HVLn+PZs=', 'Content-Length': '0'} 2023-05-31T20:34:51.418 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000044b39e39c5e4762f-006477af6b-1630-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:34:51 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:34:51.418 DEBUG:boto:b'pklxkd-56a1000falsea2023-05-31T20:29:54.205Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:34:51.419 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=b 2023-05-31T20:34:51.419 DEBUG:boto:path=/pklxkd-56/b 2023-05-31T20:34:51.420 DEBUG:boto:auth_path=/pklxkd-56/b 2023-05-31T20:34:51.420 DEBUG:boto:path=/pklxkd-56/b?versionId=null 2023-05-31T20:34:51.421 DEBUG:boto:auth_path=/pklxkd-56/b?versionId=null 2023-05-31T20:34:51.421 DEBUG:boto:Method: GET 2023-05-31T20:34:51.421 DEBUG:boto:Path: /pklxkd-56/b?versionId=null 2023-05-31T20:34:51.422 DEBUG:boto:Data: 2023-05-31T20:34:51.422 DEBUG:boto:Headers: {} 2023-05-31T20:34:51.423 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:34:51.423 DEBUG:boto:Port: 8001 2023-05-31T20:34:51.423 DEBUG:boto:Params: {} 2023-05-31T20:34:51.424 DEBUG:boto:Token: None 2023-05-31T20:34:51.424 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:34:51 GMT /pklxkd-56/b?versionId=null 2023-05-31T20:34:51.425 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:jjWKgr2qyS4p9aK5Pg7QfUPoNs8= 2023-05-31T20:34:51.425 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:34:51 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:jjWKgr2qyS4p9aK5Pg7QfUPoNs8=', 'Content-Length': '0'} 2023-05-31T20:34:51.427 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:29:54 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008d46b14a832abb7d-006477af6b-162e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:34:51 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:34:51.428 DEBUG:boto:path=/pklxkd-56/b 2023-05-31T20:34:51.429 DEBUG:boto:auth_path=/pklxkd-56/b 2023-05-31T20:34:51.429 DEBUG:boto:path=/pklxkd-56/b?versionId=null 2023-05-31T20:34:51.430 DEBUG:boto:auth_path=/pklxkd-56/b?versionId=null 2023-05-31T20:34:51.430 DEBUG:boto:Method: GET 2023-05-31T20:34:51.430 DEBUG:boto:Path: /pklxkd-56/b?versionId=null 2023-05-31T20:34:51.431 DEBUG:boto:Data: 2023-05-31T20:34:51.431 DEBUG:boto:Headers: {} 2023-05-31T20:34:51.432 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:34:51.433 DEBUG:boto:Port: 8000 2023-05-31T20:34:51.433 DEBUG:boto:Params: {} 2023-05-31T20:34:51.434 DEBUG:boto:Token: None 2023-05-31T20:34:51.434 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:34:51 GMT /pklxkd-56/b?versionId=null 2023-05-31T20:34:51.435 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:jjWKgr2qyS4p9aK5Pg7QfUPoNs8= 2023-05-31T20:34:51.435 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:34:51 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:jjWKgr2qyS4p9aK5Pg7QfUPoNs8=', 'Content-Length': '0'} 2023-05-31T20:34:51.438 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:29:54 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f8c9744f00c3dd29-006477af6b-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:34:51 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:34:51.440 DEBUG:boto:path=/pklxkd-56/b 2023-05-31T20:34:51.441 DEBUG:boto:auth_path=/pklxkd-56/b 2023-05-31T20:34:51.441 DEBUG:boto:path=/pklxkd-56/b?versionId=null 2023-05-31T20:34:51.442 DEBUG:boto:auth_path=/pklxkd-56/b?versionId=null 2023-05-31T20:34:51.443 DEBUG:boto:Method: HEAD 2023-05-31T20:34:51.443 DEBUG:boto:Path: /pklxkd-56/b?versionId=null 2023-05-31T20:34:51.443 DEBUG:boto:Data: 2023-05-31T20:34:51.444 DEBUG:boto:Headers: {} 2023-05-31T20:34:51.445 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:34:51.447 DEBUG:boto:Port: 8001 2023-05-31T20:34:51.448 DEBUG:boto:Params: {} 2023-05-31T20:34:51.449 DEBUG:boto:Token: None 2023-05-31T20:34:51.450 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:34:51 GMT /pklxkd-56/b?versionId=null 2023-05-31T20:34:51.451 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:SvaYUP1B7ef8fa922zepk8pvPCU= 2023-05-31T20:34:51.452 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:34:51 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:SvaYUP1B7ef8fa922zepk8pvPCU=', 'Content-Length': '0'} 2023-05-31T20:34:51.454 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:29:54 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000098af093f34569303-006477af6b-162e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:34:51 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:34:51.456 DEBUG:boto:path=/pklxkd-56/b 2023-05-31T20:34:51.457 DEBUG:boto:auth_path=/pklxkd-56/b 2023-05-31T20:34:51.458 DEBUG:boto:path=/pklxkd-56/b?versionId=null 2023-05-31T20:34:51.459 DEBUG:boto:auth_path=/pklxkd-56/b?versionId=null 2023-05-31T20:34:51.459 DEBUG:boto:Method: HEAD 2023-05-31T20:34:51.461 DEBUG:boto:Path: /pklxkd-56/b?versionId=null 2023-05-31T20:34:51.462 DEBUG:boto:Data: 2023-05-31T20:34:51.463 DEBUG:boto:Headers: {} 2023-05-31T20:34:51.463 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:34:51.464 DEBUG:boto:Port: 8000 2023-05-31T20:34:51.465 DEBUG:boto:Params: {} 2023-05-31T20:34:51.466 DEBUG:boto:Token: None 2023-05-31T20:34:51.468 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:34:51 GMT /pklxkd-56/b?versionId=null 2023-05-31T20:34:51.469 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:SvaYUP1B7ef8fa922zepk8pvPCU= 2023-05-31T20:34:51.470 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:34:51 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:SvaYUP1B7ef8fa922zepk8pvPCU=', 'Content-Length': '0'} 2023-05-31T20:34:51.473 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:29:54 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000081afacfe62504bfc-006477af6b-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:34:51 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:34:51.475 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=b 2023-05-31T20:34:51.475 DEBUG:boto:path=/pklxkd-56/b 2023-05-31T20:34:51.477 DEBUG:boto:auth_path=/pklxkd-56/b 2023-05-31T20:34:51.478 DEBUG:boto:path=/pklxkd-56/b?versionId=null 2023-05-31T20:34:51.479 DEBUG:boto:auth_path=/pklxkd-56/b?versionId=null 2023-05-31T20:34:51.480 DEBUG:boto:Method: GET 2023-05-31T20:34:51.481 DEBUG:boto:Path: /pklxkd-56/b?versionId=null 2023-05-31T20:34:51.481 DEBUG:boto:Data: 2023-05-31T20:34:51.482 DEBUG:boto:Headers: {} 2023-05-31T20:34:51.482 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:34:51.483 DEBUG:boto:Port: 8001 2023-05-31T20:34:51.483 DEBUG:boto:Params: {} 2023-05-31T20:34:51.484 DEBUG:boto:Token: None 2023-05-31T20:34:51.484 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:34:51 GMT /pklxkd-56/b?versionId=null 2023-05-31T20:34:51.485 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:jjWKgr2qyS4p9aK5Pg7QfUPoNs8= 2023-05-31T20:34:51.486 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:34:51 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:jjWKgr2qyS4p9aK5Pg7QfUPoNs8=', 'Content-Length': '0'} 2023-05-31T20:34:51.488 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:29:54 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f38c9bf4da773fc6-006477af6b-162e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:34:51 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:34:51.489 DEBUG:boto:path=/pklxkd-56/b 2023-05-31T20:34:51.489 DEBUG:boto:auth_path=/pklxkd-56/b 2023-05-31T20:34:51.489 DEBUG:boto:path=/pklxkd-56/b?versionId=null 2023-05-31T20:34:51.490 DEBUG:boto:auth_path=/pklxkd-56/b?versionId=null 2023-05-31T20:34:51.490 DEBUG:boto:Method: GET 2023-05-31T20:34:51.491 DEBUG:boto:Path: /pklxkd-56/b?versionId=null 2023-05-31T20:34:51.491 DEBUG:boto:Data: 2023-05-31T20:34:51.492 DEBUG:boto:Headers: {} 2023-05-31T20:34:51.492 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:34:51.492 DEBUG:boto:Port: 8000 2023-05-31T20:34:51.493 DEBUG:boto:Params: {} 2023-05-31T20:34:51.493 DEBUG:boto:Token: None 2023-05-31T20:34:51.494 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:34:51 GMT /pklxkd-56/b?versionId=null 2023-05-31T20:34:51.494 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:jjWKgr2qyS4p9aK5Pg7QfUPoNs8= 2023-05-31T20:34:51.495 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:34:51 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:jjWKgr2qyS4p9aK5Pg7QfUPoNs8=', 'Content-Length': '0'} 2023-05-31T20:34:51.497 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:29:54 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003d641ab5789b5b3b-006477af6b-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:34:51 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:34:51.497 DEBUG:boto:path=/pklxkd-56/ 2023-05-31T20:34:51.498 DEBUG:boto:auth_path=/pklxkd-56/ 2023-05-31T20:34:51.498 DEBUG:boto:path=/pklxkd-56/?prefix=b 2023-05-31T20:34:51.498 DEBUG:boto:auth_path=/pklxkd-56/?prefix=b 2023-05-31T20:34:51.499 DEBUG:boto:Method: GET 2023-05-31T20:34:51.499 DEBUG:boto:Path: /pklxkd-56/?prefix=b 2023-05-31T20:34:51.500 DEBUG:boto:Data: 2023-05-31T20:34:51.500 DEBUG:boto:Headers: {} 2023-05-31T20:34:51.500 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:34:51.501 DEBUG:boto:Port: 8001 2023-05-31T20:34:51.501 DEBUG:boto:Params: {} 2023-05-31T20:34:51.501 DEBUG:boto:Token: None 2023-05-31T20:34:51.502 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:34:51 GMT /pklxkd-56/ 2023-05-31T20:34:51.503 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:sCd5EIDPFEDQoe36mG1HVLn+PZs= 2023-05-31T20:34:51.503 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:34:51 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:sCd5EIDPFEDQoe36mG1HVLn+PZs=', 'Content-Length': '0'} 2023-05-31T20:34:51.505 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b6ccf96cbae5a934-006477af6b-162e-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:34:51 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:34:51.506 DEBUG:boto:b'pklxkd-56b1000falseb2023-05-31T20:29:54.259Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:34:51.507 DEBUG:boto:path=/pklxkd-56/ 2023-05-31T20:34:51.507 DEBUG:boto:auth_path=/pklxkd-56/ 2023-05-31T20:34:51.507 DEBUG:boto:path=/pklxkd-56/?prefix=b 2023-05-31T20:34:51.508 DEBUG:boto:auth_path=/pklxkd-56/?prefix=b 2023-05-31T20:34:51.508 DEBUG:boto:Method: GET 2023-05-31T20:34:51.508 DEBUG:boto:Path: /pklxkd-56/?prefix=b 2023-05-31T20:34:51.509 DEBUG:boto:Data: 2023-05-31T20:34:51.509 DEBUG:boto:Headers: {} 2023-05-31T20:34:51.510 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:34:51.510 DEBUG:boto:Port: 8000 2023-05-31T20:34:51.511 DEBUG:boto:Params: {} 2023-05-31T20:34:51.511 DEBUG:boto:Token: None 2023-05-31T20:34:51.512 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:34:51 GMT /pklxkd-56/ 2023-05-31T20:34:51.512 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:sCd5EIDPFEDQoe36mG1HVLn+PZs= 2023-05-31T20:34:51.513 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:34:51 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:sCd5EIDPFEDQoe36mG1HVLn+PZs=', 'Content-Length': '0'} 2023-05-31T20:34:51.515 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000002b12093e2df21565-006477af6b-1630-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:34:51 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:34:51.560 DEBUG:boto:b'pklxkd-56b1000falseb2023-05-31T20:29:54.259Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:34:51.561 DEBUG:boto:path=/pklxkd-56/b 2023-05-31T20:34:51.562 DEBUG:boto:auth_path=/pklxkd-56/b 2023-05-31T20:34:51.562 DEBUG:boto:Method: HEAD 2023-05-31T20:34:51.562 DEBUG:boto:Path: /pklxkd-56/b 2023-05-31T20:34:51.563 DEBUG:boto:Data: 2023-05-31T20:34:51.563 DEBUG:boto:Headers: {} 2023-05-31T20:34:51.563 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:34:51.564 DEBUG:boto:Port: 8001 2023-05-31T20:34:51.564 DEBUG:boto:Params: {} 2023-05-31T20:34:51.564 DEBUG:boto:Token: None 2023-05-31T20:34:51.565 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:34:51 GMT /pklxkd-56/b 2023-05-31T20:34:51.565 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:ULG7crENXVY/GVv4VJA49g4O8OQ= 2023-05-31T20:34:51.566 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:34:51 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:ULG7crENXVY/GVv4VJA49g4O8OQ=', 'Content-Length': '0'} 2023-05-31T20:34:51.567 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:29:54 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004ca79ba46d198046-006477af6b-162e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:34:51 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:34:51.568 DEBUG:boto:path=/pklxkd-56/b 2023-05-31T20:34:51.568 DEBUG:boto:auth_path=/pklxkd-56/b 2023-05-31T20:34:51.569 DEBUG:boto:Method: HEAD 2023-05-31T20:34:51.569 DEBUG:boto:Path: /pklxkd-56/b 2023-05-31T20:34:51.569 DEBUG:boto:Data: 2023-05-31T20:34:51.570 DEBUG:boto:Headers: {} 2023-05-31T20:34:51.570 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:34:51.570 DEBUG:boto:Port: 8000 2023-05-31T20:34:51.571 DEBUG:boto:Params: {} 2023-05-31T20:34:51.571 DEBUG:boto:Token: None 2023-05-31T20:34:51.571 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:34:51 GMT /pklxkd-56/b 2023-05-31T20:34:51.572 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:ULG7crENXVY/GVv4VJA49g4O8OQ= 2023-05-31T20:34:51.572 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:34:51 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:ULG7crENXVY/GVv4VJA49g4O8OQ=', 'Content-Length': '0'} 2023-05-31T20:34:51.574 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:29:54 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f57fe9fdebba6b60-006477af6b-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:34:51 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:34:51.575 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=b 2023-05-31T20:34:51.575 DEBUG:boto:path=/pklxkd-56/b 2023-05-31T20:34:51.575 DEBUG:boto:auth_path=/pklxkd-56/b 2023-05-31T20:34:51.576 DEBUG:boto:Method: GET 2023-05-31T20:34:51.576 DEBUG:boto:Path: /pklxkd-56/b 2023-05-31T20:34:51.577 DEBUG:boto:Data: 2023-05-31T20:34:51.577 DEBUG:boto:Headers: {} 2023-05-31T20:34:51.577 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:34:51.578 DEBUG:boto:Port: 8001 2023-05-31T20:34:51.578 DEBUG:boto:Params: {} 2023-05-31T20:34:51.578 DEBUG:boto:Token: None 2023-05-31T20:34:51.579 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:34:51 GMT /pklxkd-56/b 2023-05-31T20:34:51.579 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:JTki/XbO8PHhNBoav9Hyhj/75a4= 2023-05-31T20:34:51.580 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:34:51 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:JTki/XbO8PHhNBoav9Hyhj/75a4=', 'Content-Length': '0'} 2023-05-31T20:34:51.581 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:29:54 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000036a8ba1024004b1-006477af6b-162e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:34:51 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:34:51.582 DEBUG:boto:path=/pklxkd-56/b 2023-05-31T20:34:51.582 DEBUG:boto:auth_path=/pklxkd-56/b 2023-05-31T20:34:51.583 DEBUG:boto:Method: GET 2023-05-31T20:34:51.583 DEBUG:boto:Path: /pklxkd-56/b 2023-05-31T20:34:51.584 DEBUG:boto:Data: 2023-05-31T20:34:51.584 DEBUG:boto:Headers: {} 2023-05-31T20:34:51.584 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:34:51.585 DEBUG:boto:Port: 8000 2023-05-31T20:34:51.585 DEBUG:boto:Params: {} 2023-05-31T20:34:51.585 DEBUG:boto:Token: None 2023-05-31T20:34:51.586 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:34:51 GMT /pklxkd-56/b 2023-05-31T20:34:51.586 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:JTki/XbO8PHhNBoav9Hyhj/75a4= 2023-05-31T20:34:51.587 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:34:51 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:JTki/XbO8PHhNBoav9Hyhj/75a4=', 'Content-Length': '0'} 2023-05-31T20:34:51.588 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:29:54 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000080bd50c0170de718-006477af6b-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:34:51 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:34:51.589 DEBUG:boto:path=/pklxkd-56/ 2023-05-31T20:34:51.589 DEBUG:boto:auth_path=/pklxkd-56/ 2023-05-31T20:34:51.590 DEBUG:boto:path=/pklxkd-56/?prefix=b 2023-05-31T20:34:51.590 DEBUG:boto:auth_path=/pklxkd-56/?prefix=b 2023-05-31T20:34:51.590 DEBUG:boto:Method: GET 2023-05-31T20:34:51.591 DEBUG:boto:Path: /pklxkd-56/?prefix=b 2023-05-31T20:34:51.591 DEBUG:boto:Data: 2023-05-31T20:34:51.591 DEBUG:boto:Headers: {} 2023-05-31T20:34:51.592 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:34:51.592 DEBUG:boto:Port: 8001 2023-05-31T20:34:51.593 DEBUG:boto:Params: {} 2023-05-31T20:34:51.593 DEBUG:boto:Token: None 2023-05-31T20:34:51.593 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:34:51 GMT /pklxkd-56/ 2023-05-31T20:34:51.594 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:sCd5EIDPFEDQoe36mG1HVLn+PZs= 2023-05-31T20:34:51.594 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:34:51 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:sCd5EIDPFEDQoe36mG1HVLn+PZs=', 'Content-Length': '0'} 2023-05-31T20:34:51.597 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f3b842803d4ec6c6-006477af6b-162e-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:34:51 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:34:51.597 DEBUG:boto:b'pklxkd-56b1000falseb2023-05-31T20:29:54.259Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:34:51.598 DEBUG:boto:path=/pklxkd-56/ 2023-05-31T20:34:51.598 DEBUG:boto:auth_path=/pklxkd-56/ 2023-05-31T20:34:51.598 DEBUG:boto:path=/pklxkd-56/?prefix=b 2023-05-31T20:34:51.599 DEBUG:boto:auth_path=/pklxkd-56/?prefix=b 2023-05-31T20:34:51.599 DEBUG:boto:Method: GET 2023-05-31T20:34:51.600 DEBUG:boto:Path: /pklxkd-56/?prefix=b 2023-05-31T20:34:51.600 DEBUG:boto:Data: 2023-05-31T20:34:51.600 DEBUG:boto:Headers: {} 2023-05-31T20:34:51.601 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:34:51.601 DEBUG:boto:Port: 8000 2023-05-31T20:34:51.601 DEBUG:boto:Params: {} 2023-05-31T20:34:51.602 DEBUG:boto:Token: None 2023-05-31T20:34:51.602 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:34:51 GMT /pklxkd-56/ 2023-05-31T20:34:51.603 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:sCd5EIDPFEDQoe36mG1HVLn+PZs= 2023-05-31T20:34:51.603 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:34:51 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:sCd5EIDPFEDQoe36mG1HVLn+PZs=', 'Content-Length': '0'} 2023-05-31T20:34:51.605 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ce5e669d75cfa632-006477af6b-1630-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:34:51 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:34:51.605 DEBUG:boto:b'pklxkd-56b1000falseb2023-05-31T20:29:54.259Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:34:51.606 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=c 2023-05-31T20:34:51.606 DEBUG:boto:path=/pklxkd-56/c 2023-05-31T20:34:51.607 DEBUG:boto:auth_path=/pklxkd-56/c 2023-05-31T20:34:51.607 DEBUG:boto:path=/pklxkd-56/c?versionId=null 2023-05-31T20:34:51.607 DEBUG:boto:auth_path=/pklxkd-56/c?versionId=null 2023-05-31T20:34:51.608 DEBUG:boto:Method: GET 2023-05-31T20:34:51.608 DEBUG:boto:Path: /pklxkd-56/c?versionId=null 2023-05-31T20:34:51.609 DEBUG:boto:Data: 2023-05-31T20:34:51.609 DEBUG:boto:Headers: {} 2023-05-31T20:34:51.609 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:34:51.610 DEBUG:boto:Port: 8001 2023-05-31T20:34:51.610 DEBUG:boto:Params: {} 2023-05-31T20:34:51.610 DEBUG:boto:Token: None 2023-05-31T20:34:51.611 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:34:51 GMT /pklxkd-56/c?versionId=null 2023-05-31T20:34:51.611 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:wJzBZn8HssgVG2Zx5r0zc+DnYzg= 2023-05-31T20:34:51.612 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:34:51 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:wJzBZn8HssgVG2Zx5r0zc+DnYzg=', 'Content-Length': '0'} 2023-05-31T20:34:51.613 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:29:54 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000031cdb02f269db0f7-006477af6b-162e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:34:51 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:34:51.656 DEBUG:boto:path=/pklxkd-56/c 2023-05-31T20:34:51.657 DEBUG:boto:auth_path=/pklxkd-56/c 2023-05-31T20:34:51.658 DEBUG:boto:path=/pklxkd-56/c?versionId=null 2023-05-31T20:34:51.658 DEBUG:boto:auth_path=/pklxkd-56/c?versionId=null 2023-05-31T20:34:51.658 DEBUG:boto:Method: GET 2023-05-31T20:34:51.659 DEBUG:boto:Path: /pklxkd-56/c?versionId=null 2023-05-31T20:34:51.660 DEBUG:boto:Data: 2023-05-31T20:34:51.660 DEBUG:boto:Headers: {} 2023-05-31T20:34:51.661 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:34:51.661 DEBUG:boto:Port: 8000 2023-05-31T20:34:51.661 DEBUG:boto:Params: {} 2023-05-31T20:34:51.662 DEBUG:boto:Token: None 2023-05-31T20:34:51.662 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:34:51 GMT /pklxkd-56/c?versionId=null 2023-05-31T20:34:51.663 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:wJzBZn8HssgVG2Zx5r0zc+DnYzg= 2023-05-31T20:34:51.663 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:34:51 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:wJzBZn8HssgVG2Zx5r0zc+DnYzg=', 'Content-Length': '0'} 2023-05-31T20:34:51.665 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:29:54 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d27be11dbc04d510-006477af6b-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:34:51 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:34:51.666 DEBUG:boto:path=/pklxkd-56/c 2023-05-31T20:34:51.666 DEBUG:boto:auth_path=/pklxkd-56/c 2023-05-31T20:34:51.667 DEBUG:boto:path=/pklxkd-56/c?versionId=null 2023-05-31T20:34:51.667 DEBUG:boto:auth_path=/pklxkd-56/c?versionId=null 2023-05-31T20:34:51.667 DEBUG:boto:Method: HEAD 2023-05-31T20:34:51.668 DEBUG:boto:Path: /pklxkd-56/c?versionId=null 2023-05-31T20:34:51.668 DEBUG:boto:Data: 2023-05-31T20:34:51.668 DEBUG:boto:Headers: {} 2023-05-31T20:34:51.669 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:34:51.669 DEBUG:boto:Port: 8001 2023-05-31T20:34:51.669 DEBUG:boto:Params: {} 2023-05-31T20:34:51.670 DEBUG:boto:Token: None 2023-05-31T20:34:51.670 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:34:51 GMT /pklxkd-56/c?versionId=null 2023-05-31T20:34:51.671 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:ltY+CHsiaYqeTgBQuVa0tz1/nmw= 2023-05-31T20:34:51.671 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:34:51 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:ltY+CHsiaYqeTgBQuVa0tz1/nmw=', 'Content-Length': '0'} 2023-05-31T20:34:51.673 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:29:54 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000430087e0a3b073b7-006477af6b-162e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:34:51 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:34:51.673 DEBUG:boto:path=/pklxkd-56/c 2023-05-31T20:34:51.674 DEBUG:boto:auth_path=/pklxkd-56/c 2023-05-31T20:34:51.674 DEBUG:boto:path=/pklxkd-56/c?versionId=null 2023-05-31T20:34:51.674 DEBUG:boto:auth_path=/pklxkd-56/c?versionId=null 2023-05-31T20:34:51.675 DEBUG:boto:Method: HEAD 2023-05-31T20:34:51.675 DEBUG:boto:Path: /pklxkd-56/c?versionId=null 2023-05-31T20:34:51.675 DEBUG:boto:Data: 2023-05-31T20:34:51.676 DEBUG:boto:Headers: {} 2023-05-31T20:34:51.676 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:34:51.677 DEBUG:boto:Port: 8000 2023-05-31T20:34:51.677 DEBUG:boto:Params: {} 2023-05-31T20:34:51.677 DEBUG:boto:Token: None 2023-05-31T20:34:51.678 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:34:51 GMT /pklxkd-56/c?versionId=null 2023-05-31T20:34:51.678 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:ltY+CHsiaYqeTgBQuVa0tz1/nmw= 2023-05-31T20:34:51.679 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:34:51 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:ltY+CHsiaYqeTgBQuVa0tz1/nmw=', 'Content-Length': '0'} 2023-05-31T20:34:51.680 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:29:54 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000001cbb6e04377c03d-006477af6b-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:34:51 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:34:51.681 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=c 2023-05-31T20:34:51.681 DEBUG:boto:path=/pklxkd-56/c 2023-05-31T20:34:51.682 DEBUG:boto:auth_path=/pklxkd-56/c 2023-05-31T20:34:51.682 DEBUG:boto:path=/pklxkd-56/c?versionId=null 2023-05-31T20:34:51.683 DEBUG:boto:auth_path=/pklxkd-56/c?versionId=null 2023-05-31T20:34:51.683 DEBUG:boto:Method: GET 2023-05-31T20:34:51.684 DEBUG:boto:Path: /pklxkd-56/c?versionId=null 2023-05-31T20:34:51.684 DEBUG:boto:Data: 2023-05-31T20:34:51.684 DEBUG:boto:Headers: {} 2023-05-31T20:34:51.685 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:34:51.685 DEBUG:boto:Port: 8001 2023-05-31T20:34:51.685 DEBUG:boto:Params: {} 2023-05-31T20:34:51.686 DEBUG:boto:Token: None 2023-05-31T20:34:51.686 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:34:51 GMT /pklxkd-56/c?versionId=null 2023-05-31T20:34:51.687 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:wJzBZn8HssgVG2Zx5r0zc+DnYzg= 2023-05-31T20:34:51.687 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:34:51 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:wJzBZn8HssgVG2Zx5r0zc+DnYzg=', 'Content-Length': '0'} 2023-05-31T20:34:51.689 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:29:54 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a79597f47e02c343-006477af6b-162e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:34:51 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:34:51.690 DEBUG:boto:path=/pklxkd-56/c 2023-05-31T20:34:51.690 DEBUG:boto:auth_path=/pklxkd-56/c 2023-05-31T20:34:51.691 DEBUG:boto:path=/pklxkd-56/c?versionId=null 2023-05-31T20:34:51.691 DEBUG:boto:auth_path=/pklxkd-56/c?versionId=null 2023-05-31T20:34:51.691 DEBUG:boto:Method: GET 2023-05-31T20:34:51.692 DEBUG:boto:Path: /pklxkd-56/c?versionId=null 2023-05-31T20:34:51.692 DEBUG:boto:Data: 2023-05-31T20:34:51.692 DEBUG:boto:Headers: {} 2023-05-31T20:34:51.693 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:34:51.693 DEBUG:boto:Port: 8000 2023-05-31T20:34:51.693 DEBUG:boto:Params: {} 2023-05-31T20:34:51.694 DEBUG:boto:Token: None 2023-05-31T20:34:51.694 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:34:51 GMT /pklxkd-56/c?versionId=null 2023-05-31T20:34:51.695 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:wJzBZn8HssgVG2Zx5r0zc+DnYzg= 2023-05-31T20:34:51.695 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:34:51 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:wJzBZn8HssgVG2Zx5r0zc+DnYzg=', 'Content-Length': '0'} 2023-05-31T20:34:51.697 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:29:54 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000108c47f4915f3ccc-006477af6b-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:34:51 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:34:51.740 DEBUG:boto:path=/pklxkd-56/ 2023-05-31T20:34:51.741 DEBUG:boto:auth_path=/pklxkd-56/ 2023-05-31T20:34:51.742 DEBUG:boto:path=/pklxkd-56/?prefix=c 2023-05-31T20:34:51.742 DEBUG:boto:auth_path=/pklxkd-56/?prefix=c 2023-05-31T20:34:51.742 DEBUG:boto:Method: GET 2023-05-31T20:34:51.743 DEBUG:boto:Path: /pklxkd-56/?prefix=c 2023-05-31T20:34:51.743 DEBUG:boto:Data: 2023-05-31T20:34:51.743 DEBUG:boto:Headers: {} 2023-05-31T20:34:51.744 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:34:51.744 DEBUG:boto:Port: 8001 2023-05-31T20:34:51.745 DEBUG:boto:Params: {} 2023-05-31T20:34:51.745 DEBUG:boto:Token: None 2023-05-31T20:34:51.746 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:34:51 GMT /pklxkd-56/ 2023-05-31T20:34:51.746 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:sCd5EIDPFEDQoe36mG1HVLn+PZs= 2023-05-31T20:34:51.747 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:34:51 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:sCd5EIDPFEDQoe36mG1HVLn+PZs=', 'Content-Length': '0'} 2023-05-31T20:34:51.749 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c8ef907dac335b5f-006477af6b-162e-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:34:51 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:34:51.750 DEBUG:boto:b'pklxkd-56c1000falsec2023-05-31T20:29:54.281Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:34:51.752 DEBUG:boto:path=/pklxkd-56/ 2023-05-31T20:34:51.752 DEBUG:boto:auth_path=/pklxkd-56/ 2023-05-31T20:34:51.753 DEBUG:boto:path=/pklxkd-56/?prefix=c 2023-05-31T20:34:51.753 DEBUG:boto:auth_path=/pklxkd-56/?prefix=c 2023-05-31T20:34:51.754 DEBUG:boto:Method: GET 2023-05-31T20:34:51.754 DEBUG:boto:Path: /pklxkd-56/?prefix=c 2023-05-31T20:34:51.755 DEBUG:boto:Data: 2023-05-31T20:34:51.755 DEBUG:boto:Headers: {} 2023-05-31T20:34:51.756 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:34:51.756 DEBUG:boto:Port: 8000 2023-05-31T20:34:51.756 DEBUG:boto:Params: {} 2023-05-31T20:34:51.757 DEBUG:boto:Token: None 2023-05-31T20:34:51.757 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:34:51 GMT /pklxkd-56/ 2023-05-31T20:34:51.758 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:sCd5EIDPFEDQoe36mG1HVLn+PZs= 2023-05-31T20:34:51.758 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:34:51 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:sCd5EIDPFEDQoe36mG1HVLn+PZs=', 'Content-Length': '0'} 2023-05-31T20:34:51.761 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e7fe3423a42cfcdc-006477af6b-1630-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:34:51 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:34:51.761 DEBUG:boto:b'pklxkd-56c1000falsec2023-05-31T20:29:54.281Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:34:51.762 DEBUG:boto:path=/pklxkd-56/c 2023-05-31T20:34:51.762 DEBUG:boto:auth_path=/pklxkd-56/c 2023-05-31T20:34:51.763 DEBUG:boto:Method: HEAD 2023-05-31T20:34:51.763 DEBUG:boto:Path: /pklxkd-56/c 2023-05-31T20:34:51.763 DEBUG:boto:Data: 2023-05-31T20:34:51.764 DEBUG:boto:Headers: {} 2023-05-31T20:34:51.764 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:34:51.764 DEBUG:boto:Port: 8001 2023-05-31T20:34:51.765 DEBUG:boto:Params: {} 2023-05-31T20:34:51.765 DEBUG:boto:Token: None 2023-05-31T20:34:51.766 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:34:51 GMT /pklxkd-56/c 2023-05-31T20:34:51.766 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:exqLd2RPhsTjXQfVLqWXtmndh8s= 2023-05-31T20:34:51.767 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:34:51 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:exqLd2RPhsTjXQfVLqWXtmndh8s=', 'Content-Length': '0'} 2023-05-31T20:34:51.768 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:29:54 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008052fe6d9031e352-006477af6b-162e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:34:51 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:34:51.769 DEBUG:boto:path=/pklxkd-56/c 2023-05-31T20:34:51.769 DEBUG:boto:auth_path=/pklxkd-56/c 2023-05-31T20:34:51.770 DEBUG:boto:Method: HEAD 2023-05-31T20:34:51.770 DEBUG:boto:Path: /pklxkd-56/c 2023-05-31T20:34:51.771 DEBUG:boto:Data: 2023-05-31T20:34:51.771 DEBUG:boto:Headers: {} 2023-05-31T20:34:51.771 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:34:51.772 DEBUG:boto:Port: 8000 2023-05-31T20:34:51.772 DEBUG:boto:Params: {} 2023-05-31T20:34:51.772 DEBUG:boto:Token: None 2023-05-31T20:34:51.773 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:34:51 GMT /pklxkd-56/c 2023-05-31T20:34:51.773 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:exqLd2RPhsTjXQfVLqWXtmndh8s= 2023-05-31T20:34:51.774 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:34:51 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:exqLd2RPhsTjXQfVLqWXtmndh8s=', 'Content-Length': '0'} 2023-05-31T20:34:51.775 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:29:54 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003501c68e3b1f0a5c-006477af6b-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:34:51 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:34:51.776 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=c 2023-05-31T20:34:51.776 DEBUG:boto:path=/pklxkd-56/c 2023-05-31T20:34:51.777 DEBUG:boto:auth_path=/pklxkd-56/c 2023-05-31T20:34:51.777 DEBUG:boto:Method: GET 2023-05-31T20:34:51.778 DEBUG:boto:Path: /pklxkd-56/c 2023-05-31T20:34:51.778 DEBUG:boto:Data: 2023-05-31T20:34:51.778 DEBUG:boto:Headers: {} 2023-05-31T20:34:51.779 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:34:51.779 DEBUG:boto:Port: 8001 2023-05-31T20:34:51.779 DEBUG:boto:Params: {} 2023-05-31T20:34:51.780 DEBUG:boto:Token: None 2023-05-31T20:34:51.780 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:34:51 GMT /pklxkd-56/c 2023-05-31T20:34:51.781 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:ZRS8UOcIXsReT2HFtapzU871xNs= 2023-05-31T20:34:51.781 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:34:51 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:ZRS8UOcIXsReT2HFtapzU871xNs=', 'Content-Length': '0'} 2023-05-31T20:34:51.783 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:29:54 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000032008e7a3d12957f-006477af6b-162e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:34:51 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:34:51.824 DEBUG:boto:path=/pklxkd-56/c 2023-05-31T20:34:51.825 DEBUG:boto:auth_path=/pklxkd-56/c 2023-05-31T20:34:51.825 DEBUG:boto:Method: GET 2023-05-31T20:34:51.826 DEBUG:boto:Path: /pklxkd-56/c 2023-05-31T20:34:51.826 DEBUG:boto:Data: 2023-05-31T20:34:51.827 DEBUG:boto:Headers: {} 2023-05-31T20:34:51.827 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:34:51.827 DEBUG:boto:Port: 8000 2023-05-31T20:34:51.828 DEBUG:boto:Params: {} 2023-05-31T20:34:51.828 DEBUG:boto:Token: None 2023-05-31T20:34:51.829 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:34:51 GMT /pklxkd-56/c 2023-05-31T20:34:51.829 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:ZRS8UOcIXsReT2HFtapzU871xNs= 2023-05-31T20:34:51.830 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:34:51 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:ZRS8UOcIXsReT2HFtapzU871xNs=', 'Content-Length': '0'} 2023-05-31T20:34:51.832 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:29:54 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000057b2394a0cd6cdf0-006477af6b-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:34:51 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:34:51.832 DEBUG:boto:path=/pklxkd-56/ 2023-05-31T20:34:51.833 DEBUG:boto:auth_path=/pklxkd-56/ 2023-05-31T20:34:51.833 DEBUG:boto:path=/pklxkd-56/?prefix=c 2023-05-31T20:34:51.834 DEBUG:boto:auth_path=/pklxkd-56/?prefix=c 2023-05-31T20:34:51.834 DEBUG:boto:Method: GET 2023-05-31T20:34:51.834 DEBUG:boto:Path: /pklxkd-56/?prefix=c 2023-05-31T20:34:51.835 DEBUG:boto:Data: 2023-05-31T20:34:51.835 DEBUG:boto:Headers: {} 2023-05-31T20:34:51.835 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:34:51.836 DEBUG:boto:Port: 8001 2023-05-31T20:34:51.836 DEBUG:boto:Params: {} 2023-05-31T20:34:51.837 DEBUG:boto:Token: None 2023-05-31T20:34:51.838 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:34:51 GMT /pklxkd-56/ 2023-05-31T20:34:51.838 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:sCd5EIDPFEDQoe36mG1HVLn+PZs= 2023-05-31T20:34:51.839 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:34:51 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:sCd5EIDPFEDQoe36mG1HVLn+PZs=', 'Content-Length': '0'} 2023-05-31T20:34:51.841 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000df63f2d238295dfa-006477af6b-162e-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:34:51 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:34:51.842 DEBUG:boto:b'pklxkd-56c1000falsec2023-05-31T20:29:54.281Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:34:51.843 DEBUG:boto:path=/pklxkd-56/ 2023-05-31T20:34:51.843 DEBUG:boto:auth_path=/pklxkd-56/ 2023-05-31T20:34:51.844 DEBUG:boto:path=/pklxkd-56/?prefix=c 2023-05-31T20:34:51.844 DEBUG:boto:auth_path=/pklxkd-56/?prefix=c 2023-05-31T20:34:51.845 DEBUG:boto:Method: GET 2023-05-31T20:34:51.845 DEBUG:boto:Path: /pklxkd-56/?prefix=c 2023-05-31T20:34:51.846 DEBUG:boto:Data: 2023-05-31T20:34:51.846 DEBUG:boto:Headers: {} 2023-05-31T20:34:51.847 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:34:51.848 DEBUG:boto:Port: 8000 2023-05-31T20:34:51.848 DEBUG:boto:Params: {} 2023-05-31T20:34:51.849 DEBUG:boto:Token: None 2023-05-31T20:34:51.850 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:34:51 GMT /pklxkd-56/ 2023-05-31T20:34:51.850 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:sCd5EIDPFEDQoe36mG1HVLn+PZs= 2023-05-31T20:34:51.852 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:34:51 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:sCd5EIDPFEDQoe36mG1HVLn+PZs=', 'Content-Length': '0'} 2023-05-31T20:34:51.854 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000006a7be2662cab58f3-006477af6b-1630-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:34:51 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:34:51.855 DEBUG:boto:b'pklxkd-56c1000falsec2023-05-31T20:29:54.281Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:34:51.855 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=d 2023-05-31T20:34:51.856 DEBUG:boto:path=/pklxkd-56/d 2023-05-31T20:34:51.856 DEBUG:boto:auth_path=/pklxkd-56/d 2023-05-31T20:34:51.857 DEBUG:boto:path=/pklxkd-56/d?versionId=null 2023-05-31T20:34:51.857 DEBUG:boto:auth_path=/pklxkd-56/d?versionId=null 2023-05-31T20:34:51.858 DEBUG:boto:Method: GET 2023-05-31T20:34:51.859 DEBUG:boto:Path: /pklxkd-56/d?versionId=null 2023-05-31T20:34:51.859 DEBUG:boto:Data: 2023-05-31T20:34:51.860 DEBUG:boto:Headers: {} 2023-05-31T20:34:51.860 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:34:51.861 DEBUG:boto:Port: 8001 2023-05-31T20:34:51.861 DEBUG:boto:Params: {} 2023-05-31T20:34:51.862 DEBUG:boto:Token: None 2023-05-31T20:34:51.862 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:34:51 GMT /pklxkd-56/d?versionId=null 2023-05-31T20:34:51.863 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:e+R3q2kBD9Ok1GGN6byFUS4HwPQ= 2023-05-31T20:34:51.864 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:34:51 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:e+R3q2kBD9Ok1GGN6byFUS4HwPQ=', 'Content-Length': '0'} 2023-05-31T20:34:51.865 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:29:54 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000012b63cb129bc7301-006477af6b-162e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:34:51 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:34:51.866 DEBUG:boto:path=/pklxkd-56/d 2023-05-31T20:34:51.867 DEBUG:boto:auth_path=/pklxkd-56/d 2023-05-31T20:34:51.868 DEBUG:boto:path=/pklxkd-56/d?versionId=null 2023-05-31T20:34:51.868 DEBUG:boto:auth_path=/pklxkd-56/d?versionId=null 2023-05-31T20:34:51.869 DEBUG:boto:Method: GET 2023-05-31T20:34:51.869 DEBUG:boto:Path: /pklxkd-56/d?versionId=null 2023-05-31T20:34:51.870 DEBUG:boto:Data: 2023-05-31T20:34:51.870 DEBUG:boto:Headers: {} 2023-05-31T20:34:51.871 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:34:51.871 DEBUG:boto:Port: 8000 2023-05-31T20:34:51.871 DEBUG:boto:Params: {} 2023-05-31T20:34:51.872 DEBUG:boto:Token: None 2023-05-31T20:34:51.872 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:34:51 GMT /pklxkd-56/d?versionId=null 2023-05-31T20:34:51.873 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:e+R3q2kBD9Ok1GGN6byFUS4HwPQ= 2023-05-31T20:34:51.873 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:34:51 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:e+R3q2kBD9Ok1GGN6byFUS4HwPQ=', 'Content-Length': '0'} 2023-05-31T20:34:51.875 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:29:54 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000002c1a5b8e3c7954e-006477af6b-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:34:51 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:34:51.876 DEBUG:boto:path=/pklxkd-56/d 2023-05-31T20:34:51.876 DEBUG:boto:auth_path=/pklxkd-56/d 2023-05-31T20:34:51.877 DEBUG:boto:path=/pklxkd-56/d?versionId=null 2023-05-31T20:34:51.877 DEBUG:boto:auth_path=/pklxkd-56/d?versionId=null 2023-05-31T20:34:51.877 DEBUG:boto:Method: HEAD 2023-05-31T20:34:51.878 DEBUG:boto:Path: /pklxkd-56/d?versionId=null 2023-05-31T20:34:51.878 DEBUG:boto:Data: 2023-05-31T20:34:51.878 DEBUG:boto:Headers: {} 2023-05-31T20:34:51.879 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:34:51.879 DEBUG:boto:Port: 8001 2023-05-31T20:34:51.879 DEBUG:boto:Params: {} 2023-05-31T20:34:51.880 DEBUG:boto:Token: None 2023-05-31T20:34:51.880 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:34:51 GMT /pklxkd-56/d?versionId=null 2023-05-31T20:34:51.881 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:sWLKGj3z1jWFvraoQFS4AqM/p6k= 2023-05-31T20:34:51.881 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:34:51 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:sWLKGj3z1jWFvraoQFS4AqM/p6k=', 'Content-Length': '0'} 2023-05-31T20:34:51.883 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:29:54 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000899d9057e86b1a23-006477af6b-162e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:34:51 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:34:51.883 DEBUG:boto:path=/pklxkd-56/d 2023-05-31T20:34:51.884 DEBUG:boto:auth_path=/pklxkd-56/d 2023-05-31T20:34:51.884 DEBUG:boto:path=/pklxkd-56/d?versionId=null 2023-05-31T20:34:51.885 DEBUG:boto:auth_path=/pklxkd-56/d?versionId=null 2023-05-31T20:34:51.885 DEBUG:boto:Method: HEAD 2023-05-31T20:34:51.885 DEBUG:boto:Path: /pklxkd-56/d?versionId=null 2023-05-31T20:34:51.886 DEBUG:boto:Data: 2023-05-31T20:34:51.886 DEBUG:boto:Headers: {} 2023-05-31T20:34:51.887 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:34:51.887 DEBUG:boto:Port: 8000 2023-05-31T20:34:51.887 DEBUG:boto:Params: {} 2023-05-31T20:34:51.888 DEBUG:boto:Token: None 2023-05-31T20:34:51.888 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:34:51 GMT /pklxkd-56/d?versionId=null 2023-05-31T20:34:51.889 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:sWLKGj3z1jWFvraoQFS4AqM/p6k= 2023-05-31T20:34:51.889 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:34:51 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:sWLKGj3z1jWFvraoQFS4AqM/p6k=', 'Content-Length': '0'} 2023-05-31T20:34:51.891 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:29:54 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000057d3c5a4415bafbc-006477af6b-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:34:51 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:34:51.892 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=d 2023-05-31T20:34:51.892 DEBUG:boto:path=/pklxkd-56/d 2023-05-31T20:34:51.892 DEBUG:boto:auth_path=/pklxkd-56/d 2023-05-31T20:34:51.893 DEBUG:boto:path=/pklxkd-56/d?versionId=null 2023-05-31T20:34:51.893 DEBUG:boto:auth_path=/pklxkd-56/d?versionId=null 2023-05-31T20:34:51.894 DEBUG:boto:Method: GET 2023-05-31T20:34:51.894 DEBUG:boto:Path: /pklxkd-56/d?versionId=null 2023-05-31T20:34:51.894 DEBUG:boto:Data: 2023-05-31T20:34:51.895 DEBUG:boto:Headers: {} 2023-05-31T20:34:51.895 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:34:51.895 DEBUG:boto:Port: 8001 2023-05-31T20:34:51.896 DEBUG:boto:Params: {} 2023-05-31T20:34:51.896 DEBUG:boto:Token: None 2023-05-31T20:34:51.897 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:34:51 GMT /pklxkd-56/d?versionId=null 2023-05-31T20:34:51.897 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:e+R3q2kBD9Ok1GGN6byFUS4HwPQ= 2023-05-31T20:34:51.898 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:34:51 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:e+R3q2kBD9Ok1GGN6byFUS4HwPQ=', 'Content-Length': '0'} 2023-05-31T20:34:51.899 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:29:54 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000fed867c0e7a99854-006477af6b-162e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:34:51 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:34:51.900 DEBUG:boto:path=/pklxkd-56/d 2023-05-31T20:34:51.901 DEBUG:boto:auth_path=/pklxkd-56/d 2023-05-31T20:34:51.901 DEBUG:boto:path=/pklxkd-56/d?versionId=null 2023-05-31T20:34:51.901 DEBUG:boto:auth_path=/pklxkd-56/d?versionId=null 2023-05-31T20:34:51.902 DEBUG:boto:Method: GET 2023-05-31T20:34:51.902 DEBUG:boto:Path: /pklxkd-56/d?versionId=null 2023-05-31T20:34:51.903 DEBUG:boto:Data: 2023-05-31T20:34:51.903 DEBUG:boto:Headers: {} 2023-05-31T20:34:51.903 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:34:51.904 DEBUG:boto:Port: 8000 2023-05-31T20:34:51.904 DEBUG:boto:Params: {} 2023-05-31T20:34:51.905 DEBUG:boto:Token: None 2023-05-31T20:34:51.905 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:34:51 GMT /pklxkd-56/d?versionId=null 2023-05-31T20:34:51.905 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:e+R3q2kBD9Ok1GGN6byFUS4HwPQ= 2023-05-31T20:34:51.906 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:34:51 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:e+R3q2kBD9Ok1GGN6byFUS4HwPQ=', 'Content-Length': '0'} 2023-05-31T20:34:51.908 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:29:54 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000571e24c59b89533-006477af6b-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:34:51 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:34:51.908 DEBUG:boto:path=/pklxkd-56/ 2023-05-31T20:34:51.909 DEBUG:boto:auth_path=/pklxkd-56/ 2023-05-31T20:34:51.909 DEBUG:boto:path=/pklxkd-56/?prefix=d 2023-05-31T20:34:51.910 DEBUG:boto:auth_path=/pklxkd-56/?prefix=d 2023-05-31T20:34:51.910 DEBUG:boto:Method: GET 2023-05-31T20:34:51.910 DEBUG:boto:Path: /pklxkd-56/?prefix=d 2023-05-31T20:34:51.911 DEBUG:boto:Data: 2023-05-31T20:34:51.911 DEBUG:boto:Headers: {} 2023-05-31T20:34:51.911 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:34:51.912 DEBUG:boto:Port: 8001 2023-05-31T20:34:51.912 DEBUG:boto:Params: {} 2023-05-31T20:34:51.913 DEBUG:boto:Token: None 2023-05-31T20:34:51.913 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:34:51 GMT /pklxkd-56/ 2023-05-31T20:34:51.914 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:sCd5EIDPFEDQoe36mG1HVLn+PZs= 2023-05-31T20:34:51.914 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:34:51 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:sCd5EIDPFEDQoe36mG1HVLn+PZs=', 'Content-Length': '0'} 2023-05-31T20:34:51.917 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ce5d1213c72ad9f2-006477af6b-162e-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:34:51 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:34:51.917 DEBUG:boto:b'pklxkd-56d1000falsed2023-05-31T20:29:54.306Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:34:51.918 DEBUG:boto:path=/pklxkd-56/ 2023-05-31T20:34:51.918 DEBUG:boto:auth_path=/pklxkd-56/ 2023-05-31T20:34:51.919 DEBUG:boto:path=/pklxkd-56/?prefix=d 2023-05-31T20:34:51.919 DEBUG:boto:auth_path=/pklxkd-56/?prefix=d 2023-05-31T20:34:51.919 DEBUG:boto:Method: GET 2023-05-31T20:34:51.920 DEBUG:boto:Path: /pklxkd-56/?prefix=d 2023-05-31T20:34:51.920 DEBUG:boto:Data: 2023-05-31T20:34:51.921 DEBUG:boto:Headers: {} 2023-05-31T20:34:51.921 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:34:51.921 DEBUG:boto:Port: 8000 2023-05-31T20:34:51.922 DEBUG:boto:Params: {} 2023-05-31T20:34:51.922 DEBUG:boto:Token: None 2023-05-31T20:34:51.922 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:34:51 GMT /pklxkd-56/ 2023-05-31T20:34:51.923 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:sCd5EIDPFEDQoe36mG1HVLn+PZs= 2023-05-31T20:34:51.923 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:34:51 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:sCd5EIDPFEDQoe36mG1HVLn+PZs=', 'Content-Length': '0'} 2023-05-31T20:34:51.925 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000cedc1e0cef04bb6e-006477af6b-1630-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:34:51 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:34:51.926 DEBUG:boto:b'pklxkd-56d1000falsed2023-05-31T20:29:54.306Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:34:51.927 DEBUG:boto:path=/pklxkd-56/d 2023-05-31T20:34:51.927 DEBUG:boto:auth_path=/pklxkd-56/d 2023-05-31T20:34:51.927 DEBUG:boto:Method: HEAD 2023-05-31T20:34:51.928 DEBUG:boto:Path: /pklxkd-56/d 2023-05-31T20:34:51.928 DEBUG:boto:Data: 2023-05-31T20:34:51.929 DEBUG:boto:Headers: {} 2023-05-31T20:34:51.929 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:34:51.929 DEBUG:boto:Port: 8001 2023-05-31T20:34:51.930 DEBUG:boto:Params: {} 2023-05-31T20:34:51.930 DEBUG:boto:Token: None 2023-05-31T20:34:51.931 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:34:51 GMT /pklxkd-56/d 2023-05-31T20:34:51.931 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:xoY4Ff2f14vJ1+i887Ll7hIdI1g= 2023-05-31T20:34:51.932 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:34:51 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:xoY4Ff2f14vJ1+i887Ll7hIdI1g=', 'Content-Length': '0'} 2023-05-31T20:34:51.933 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:29:54 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000aedc5c9c4f199b4a-006477af6b-162e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:34:51 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:34:51.934 DEBUG:boto:path=/pklxkd-56/d 2023-05-31T20:34:51.934 DEBUG:boto:auth_path=/pklxkd-56/d 2023-05-31T20:34:51.935 DEBUG:boto:Method: HEAD 2023-05-31T20:34:51.935 DEBUG:boto:Path: /pklxkd-56/d 2023-05-31T20:34:51.936 DEBUG:boto:Data: 2023-05-31T20:34:51.936 DEBUG:boto:Headers: {} 2023-05-31T20:34:51.936 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:34:51.937 DEBUG:boto:Port: 8000 2023-05-31T20:34:51.937 DEBUG:boto:Params: {} 2023-05-31T20:34:51.937 DEBUG:boto:Token: None 2023-05-31T20:34:51.938 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:34:51 GMT /pklxkd-56/d 2023-05-31T20:34:51.938 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:xoY4Ff2f14vJ1+i887Ll7hIdI1g= 2023-05-31T20:34:51.939 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:34:51 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:xoY4Ff2f14vJ1+i887Ll7hIdI1g=', 'Content-Length': '0'} 2023-05-31T20:34:51.941 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:29:54 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d178acf2cd65be98-006477af6b-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:34:51 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:34:51.941 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=d 2023-05-31T20:34:51.942 DEBUG:boto:path=/pklxkd-56/d 2023-05-31T20:34:51.942 DEBUG:boto:auth_path=/pklxkd-56/d 2023-05-31T20:34:51.943 DEBUG:boto:Method: GET 2023-05-31T20:34:51.943 DEBUG:boto:Path: /pklxkd-56/d 2023-05-31T20:34:51.943 DEBUG:boto:Data: 2023-05-31T20:34:51.944 DEBUG:boto:Headers: {} 2023-05-31T20:34:51.944 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:34:51.944 DEBUG:boto:Port: 8001 2023-05-31T20:34:51.945 DEBUG:boto:Params: {} 2023-05-31T20:34:51.945 DEBUG:boto:Token: None 2023-05-31T20:34:51.946 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:34:51 GMT /pklxkd-56/d 2023-05-31T20:34:51.946 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:nbKHAYFrK8SebGFnL3qrblhj/U8= 2023-05-31T20:34:51.947 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:34:51 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:nbKHAYFrK8SebGFnL3qrblhj/U8=', 'Content-Length': '0'} 2023-05-31T20:34:51.949 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:29:54 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002320b94d66489e95-006477af6b-162e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:34:51 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:34:51.996 DEBUG:boto:path=/pklxkd-56/d 2023-05-31T20:34:51.997 DEBUG:boto:auth_path=/pklxkd-56/d 2023-05-31T20:34:51.998 DEBUG:boto:Method: GET 2023-05-31T20:34:51.998 DEBUG:boto:Path: /pklxkd-56/d 2023-05-31T20:34:51.999 DEBUG:boto:Data: 2023-05-31T20:34:51.999 DEBUG:boto:Headers: {} 2023-05-31T20:34:52.000 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:34:52.000 DEBUG:boto:Port: 8000 2023-05-31T20:34:52.001 DEBUG:boto:Params: {} 2023-05-31T20:34:52.001 DEBUG:boto:Token: None 2023-05-31T20:34:52.002 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:34:52 GMT /pklxkd-56/d 2023-05-31T20:34:52.003 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:XXmXNWn5XZr2qx9qTI7g2qpfQWg= 2023-05-31T20:34:52.003 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:34:52 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:XXmXNWn5XZr2qx9qTI7g2qpfQWg=', 'Content-Length': '0'} 2023-05-31T20:34:52.005 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:29:54 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005d565e7ead53b40c-006477af6c-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:34:52 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:34:52.006 DEBUG:boto:path=/pklxkd-56/ 2023-05-31T20:34:52.006 DEBUG:boto:auth_path=/pklxkd-56/ 2023-05-31T20:34:52.007 DEBUG:boto:path=/pklxkd-56/?prefix=d 2023-05-31T20:34:52.007 DEBUG:boto:auth_path=/pklxkd-56/?prefix=d 2023-05-31T20:34:52.007 DEBUG:boto:Method: GET 2023-05-31T20:34:52.008 DEBUG:boto:Path: /pklxkd-56/?prefix=d 2023-05-31T20:34:52.008 DEBUG:boto:Data: 2023-05-31T20:34:52.008 DEBUG:boto:Headers: {} 2023-05-31T20:34:52.009 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:34:52.010 DEBUG:boto:Port: 8001 2023-05-31T20:34:52.010 DEBUG:boto:Params: {} 2023-05-31T20:34:52.011 DEBUG:boto:Token: None 2023-05-31T20:34:52.011 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:34:52 GMT /pklxkd-56/ 2023-05-31T20:34:52.012 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:nL4TOhvnP0P54UQXOfqFYR6uCRE= 2023-05-31T20:34:52.012 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:34:52 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:nL4TOhvnP0P54UQXOfqFYR6uCRE=', 'Content-Length': '0'} 2023-05-31T20:34:52.015 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000019f8de34dba5b69c-006477af6c-162e-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:34:52 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:34:52.016 DEBUG:boto:b'pklxkd-56d1000falsed2023-05-31T20:29:54.306Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:34:52.017 DEBUG:boto:path=/pklxkd-56/ 2023-05-31T20:34:52.018 DEBUG:boto:auth_path=/pklxkd-56/ 2023-05-31T20:34:52.019 DEBUG:boto:path=/pklxkd-56/?prefix=d 2023-05-31T20:34:52.019 DEBUG:boto:auth_path=/pklxkd-56/?prefix=d 2023-05-31T20:34:52.020 DEBUG:boto:Method: GET 2023-05-31T20:34:52.021 DEBUG:boto:Path: /pklxkd-56/?prefix=d 2023-05-31T20:34:52.022 DEBUG:boto:Data: 2023-05-31T20:34:52.022 DEBUG:boto:Headers: {} 2023-05-31T20:34:52.023 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:34:52.024 DEBUG:boto:Port: 8000 2023-05-31T20:34:52.025 DEBUG:boto:Params: {} 2023-05-31T20:34:52.026 DEBUG:boto:Token: None 2023-05-31T20:34:52.027 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:34:52 GMT /pklxkd-56/ 2023-05-31T20:34:52.028 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:nL4TOhvnP0P54UQXOfqFYR6uCRE= 2023-05-31T20:34:52.029 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:34:52 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:nL4TOhvnP0P54UQXOfqFYR6uCRE=', 'Content-Length': '0'} 2023-05-31T20:34:52.031 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000063d2fdebd2ee9735-006477af6c-1630-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:34:52 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:34:52.072 DEBUG:boto:b'pklxkd-56d1000falsed2023-05-31T20:29:54.306Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:34:52.073 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=e 2023-05-31T20:34:52.074 DEBUG:boto:path=/pklxkd-56/e 2023-05-31T20:34:52.074 DEBUG:boto:auth_path=/pklxkd-56/e 2023-05-31T20:34:52.074 DEBUG:boto:path=/pklxkd-56/e?versionId=null 2023-05-31T20:34:52.075 DEBUG:boto:auth_path=/pklxkd-56/e?versionId=null 2023-05-31T20:34:52.075 DEBUG:boto:Method: GET 2023-05-31T20:34:52.075 DEBUG:boto:Path: /pklxkd-56/e?versionId=null 2023-05-31T20:34:52.076 DEBUG:boto:Data: 2023-05-31T20:34:52.076 DEBUG:boto:Headers: {} 2023-05-31T20:34:52.079 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:34:52.080 DEBUG:boto:Port: 8001 2023-05-31T20:34:52.080 DEBUG:boto:Params: {} 2023-05-31T20:34:52.081 DEBUG:boto:Token: None 2023-05-31T20:34:52.082 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:34:52 GMT /pklxkd-56/e?versionId=null 2023-05-31T20:34:52.083 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:XWJfBceTkFjJdNKRchzn2Cbuhy8= 2023-05-31T20:34:52.084 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:34:52 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:XWJfBceTkFjJdNKRchzn2Cbuhy8=', 'Content-Length': '0'} 2023-05-31T20:34:52.086 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:30:05 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ca7ee2d8f6c5fdec-006477af6c-162e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:34:52 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:34:52.087 DEBUG:boto:path=/pklxkd-56/e 2023-05-31T20:34:52.088 DEBUG:boto:auth_path=/pklxkd-56/e 2023-05-31T20:34:52.089 DEBUG:boto:path=/pklxkd-56/e?versionId=null 2023-05-31T20:34:52.089 DEBUG:boto:auth_path=/pklxkd-56/e?versionId=null 2023-05-31T20:34:52.090 DEBUG:boto:Method: GET 2023-05-31T20:34:52.091 DEBUG:boto:Path: /pklxkd-56/e?versionId=null 2023-05-31T20:34:52.091 DEBUG:boto:Data: 2023-05-31T20:34:52.092 DEBUG:boto:Headers: {} 2023-05-31T20:34:52.093 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:34:52.094 DEBUG:boto:Port: 8000 2023-05-31T20:34:52.095 DEBUG:boto:Params: {} 2023-05-31T20:34:52.095 DEBUG:boto:Token: None 2023-05-31T20:34:52.097 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:34:52 GMT /pklxkd-56/e?versionId=null 2023-05-31T20:34:52.097 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:XWJfBceTkFjJdNKRchzn2Cbuhy8= 2023-05-31T20:34:52.098 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:34:52 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:XWJfBceTkFjJdNKRchzn2Cbuhy8=', 'Content-Length': '0'} 2023-05-31T20:34:52.101 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:30:05 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000040d7246a55d40dbe-006477af6c-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:34:52 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:34:52.102 DEBUG:boto:path=/pklxkd-56/e 2023-05-31T20:34:52.103 DEBUG:boto:auth_path=/pklxkd-56/e 2023-05-31T20:34:52.103 DEBUG:boto:path=/pklxkd-56/e?versionId=null 2023-05-31T20:34:52.103 DEBUG:boto:auth_path=/pklxkd-56/e?versionId=null 2023-05-31T20:34:52.104 DEBUG:boto:Method: HEAD 2023-05-31T20:34:52.104 DEBUG:boto:Path: /pklxkd-56/e?versionId=null 2023-05-31T20:34:52.105 DEBUG:boto:Data: 2023-05-31T20:34:52.105 DEBUG:boto:Headers: {} 2023-05-31T20:34:52.105 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:34:52.106 DEBUG:boto:Port: 8001 2023-05-31T20:34:52.106 DEBUG:boto:Params: {} 2023-05-31T20:34:52.107 DEBUG:boto:Token: None 2023-05-31T20:34:52.107 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:34:52 GMT /pklxkd-56/e?versionId=null 2023-05-31T20:34:52.107 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:X2+FwjKe1Rhc7F0saMp+S4lZpd0= 2023-05-31T20:34:52.108 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:34:52 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:X2+FwjKe1Rhc7F0saMp+S4lZpd0=', 'Content-Length': '0'} 2023-05-31T20:34:52.111 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:30:05 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000fb02416836233550-006477af6c-162e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:34:52 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:34:52.112 DEBUG:boto:path=/pklxkd-56/e 2023-05-31T20:34:52.112 DEBUG:boto:auth_path=/pklxkd-56/e 2023-05-31T20:34:52.113 DEBUG:boto:path=/pklxkd-56/e?versionId=null 2023-05-31T20:34:52.113 DEBUG:boto:auth_path=/pklxkd-56/e?versionId=null 2023-05-31T20:34:52.113 DEBUG:boto:Method: HEAD 2023-05-31T20:34:52.114 DEBUG:boto:Path: /pklxkd-56/e?versionId=null 2023-05-31T20:34:52.114 DEBUG:boto:Data: 2023-05-31T20:34:52.114 DEBUG:boto:Headers: {} 2023-05-31T20:34:52.115 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:34:52.115 DEBUG:boto:Port: 8000 2023-05-31T20:34:52.115 DEBUG:boto:Params: {} 2023-05-31T20:34:52.116 DEBUG:boto:Token: None 2023-05-31T20:34:52.116 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:34:52 GMT /pklxkd-56/e?versionId=null 2023-05-31T20:34:52.117 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:X2+FwjKe1Rhc7F0saMp+S4lZpd0= 2023-05-31T20:34:52.118 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:34:52 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:X2+FwjKe1Rhc7F0saMp+S4lZpd0=', 'Content-Length': '0'} 2023-05-31T20:34:52.120 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:30:05 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000006544d03f7c7974c-006477af6c-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:34:52 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:34:52.120 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=e 2023-05-31T20:34:52.121 DEBUG:boto:path=/pklxkd-56/e 2023-05-31T20:34:52.122 DEBUG:boto:auth_path=/pklxkd-56/e 2023-05-31T20:34:52.122 DEBUG:boto:path=/pklxkd-56/e?versionId=null 2023-05-31T20:34:52.123 DEBUG:boto:auth_path=/pklxkd-56/e?versionId=null 2023-05-31T20:34:52.123 DEBUG:boto:Method: GET 2023-05-31T20:34:52.123 DEBUG:boto:Path: /pklxkd-56/e?versionId=null 2023-05-31T20:34:52.124 DEBUG:boto:Data: 2023-05-31T20:34:52.124 DEBUG:boto:Headers: {} 2023-05-31T20:34:52.125 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:34:52.125 DEBUG:boto:Port: 8001 2023-05-31T20:34:52.125 DEBUG:boto:Params: {} 2023-05-31T20:34:52.126 DEBUG:boto:Token: None 2023-05-31T20:34:52.126 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:34:52 GMT /pklxkd-56/e?versionId=null 2023-05-31T20:34:52.127 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:XWJfBceTkFjJdNKRchzn2Cbuhy8= 2023-05-31T20:34:52.128 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:34:52 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:XWJfBceTkFjJdNKRchzn2Cbuhy8=', 'Content-Length': '0'} 2023-05-31T20:34:52.130 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:30:05 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b259d5e5866f08f8-006477af6c-162e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:34:52 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:34:52.131 DEBUG:boto:path=/pklxkd-56/e 2023-05-31T20:34:52.132 DEBUG:boto:auth_path=/pklxkd-56/e 2023-05-31T20:34:52.132 DEBUG:boto:path=/pklxkd-56/e?versionId=null 2023-05-31T20:34:52.132 DEBUG:boto:auth_path=/pklxkd-56/e?versionId=null 2023-05-31T20:34:52.133 DEBUG:boto:Method: GET 2023-05-31T20:34:52.133 DEBUG:boto:Path: /pklxkd-56/e?versionId=null 2023-05-31T20:34:52.133 DEBUG:boto:Data: 2023-05-31T20:34:52.134 DEBUG:boto:Headers: {} 2023-05-31T20:34:52.134 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:34:52.135 DEBUG:boto:Port: 8000 2023-05-31T20:34:52.135 DEBUG:boto:Params: {} 2023-05-31T20:34:52.135 DEBUG:boto:Token: None 2023-05-31T20:34:52.136 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:34:52 GMT /pklxkd-56/e?versionId=null 2023-05-31T20:34:52.136 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:XWJfBceTkFjJdNKRchzn2Cbuhy8= 2023-05-31T20:34:52.137 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:34:52 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:XWJfBceTkFjJdNKRchzn2Cbuhy8=', 'Content-Length': '0'} 2023-05-31T20:34:52.139 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:30:05 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000327d4252b1261a6e-006477af6c-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:34:52 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:34:52.139 DEBUG:boto:path=/pklxkd-56/ 2023-05-31T20:34:52.140 DEBUG:boto:auth_path=/pklxkd-56/ 2023-05-31T20:34:52.140 DEBUG:boto:path=/pklxkd-56/?prefix=e 2023-05-31T20:34:52.141 DEBUG:boto:auth_path=/pklxkd-56/?prefix=e 2023-05-31T20:34:52.141 DEBUG:boto:Method: GET 2023-05-31T20:34:52.141 DEBUG:boto:Path: /pklxkd-56/?prefix=e 2023-05-31T20:34:52.142 DEBUG:boto:Data: 2023-05-31T20:34:52.142 DEBUG:boto:Headers: {} 2023-05-31T20:34:52.142 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:34:52.143 DEBUG:boto:Port: 8001 2023-05-31T20:34:52.143 DEBUG:boto:Params: {} 2023-05-31T20:34:52.144 DEBUG:boto:Token: None 2023-05-31T20:34:52.144 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:34:52 GMT /pklxkd-56/ 2023-05-31T20:34:52.145 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:nL4TOhvnP0P54UQXOfqFYR6uCRE= 2023-05-31T20:34:52.145 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:34:52 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:nL4TOhvnP0P54UQXOfqFYR6uCRE=', 'Content-Length': '0'} 2023-05-31T20:34:52.152 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000001be3dce529e7a919-006477af6c-162e-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:34:52 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:34:52.153 DEBUG:boto:b'pklxkd-56e1000falsee2023-05-31T20:30:05.784Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:34:52.154 DEBUG:boto:path=/pklxkd-56/ 2023-05-31T20:34:52.155 DEBUG:boto:auth_path=/pklxkd-56/ 2023-05-31T20:34:52.156 DEBUG:boto:path=/pklxkd-56/?prefix=e 2023-05-31T20:34:52.156 DEBUG:boto:auth_path=/pklxkd-56/?prefix=e 2023-05-31T20:34:52.156 DEBUG:boto:Method: GET 2023-05-31T20:34:52.157 DEBUG:boto:Path: /pklxkd-56/?prefix=e 2023-05-31T20:34:52.158 DEBUG:boto:Data: 2023-05-31T20:34:52.158 DEBUG:boto:Headers: {} 2023-05-31T20:34:52.159 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:34:52.160 DEBUG:boto:Port: 8000 2023-05-31T20:34:52.160 DEBUG:boto:Params: {} 2023-05-31T20:34:52.161 DEBUG:boto:Token: None 2023-05-31T20:34:52.161 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:34:52 GMT /pklxkd-56/ 2023-05-31T20:34:52.162 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:nL4TOhvnP0P54UQXOfqFYR6uCRE= 2023-05-31T20:34:52.162 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:34:52 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:nL4TOhvnP0P54UQXOfqFYR6uCRE=', 'Content-Length': '0'} 2023-05-31T20:34:52.164 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a7663c8db48ca24c-006477af6c-1630-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:34:52 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:34:52.165 DEBUG:boto:b'pklxkd-56e1000falsee2023-05-31T20:30:05.784Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:34:52.166 DEBUG:boto:path=/pklxkd-56/e 2023-05-31T20:34:52.167 DEBUG:boto:auth_path=/pklxkd-56/e 2023-05-31T20:34:52.167 DEBUG:boto:Method: HEAD 2023-05-31T20:34:52.168 DEBUG:boto:Path: /pklxkd-56/e 2023-05-31T20:34:52.168 DEBUG:boto:Data: 2023-05-31T20:34:52.168 DEBUG:boto:Headers: {} 2023-05-31T20:34:52.169 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:34:52.169 DEBUG:boto:Port: 8001 2023-05-31T20:34:52.169 DEBUG:boto:Params: {} 2023-05-31T20:34:52.170 DEBUG:boto:Token: None 2023-05-31T20:34:52.171 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:34:52 GMT /pklxkd-56/e 2023-05-31T20:34:52.171 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:L9+zzTHfiwCdhPMfMSCySj8flfc= 2023-05-31T20:34:52.172 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:34:52 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:L9+zzTHfiwCdhPMfMSCySj8flfc=', 'Content-Length': '0'} 2023-05-31T20:34:52.175 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:30:05 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003d9499b2579b878f-006477af6c-162e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:34:52 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:34:52.176 DEBUG:boto:path=/pklxkd-56/e 2023-05-31T20:34:52.176 DEBUG:boto:auth_path=/pklxkd-56/e 2023-05-31T20:34:52.176 DEBUG:boto:Method: HEAD 2023-05-31T20:34:52.177 DEBUG:boto:Path: /pklxkd-56/e 2023-05-31T20:34:52.177 DEBUG:boto:Data: 2023-05-31T20:34:52.177 DEBUG:boto:Headers: {} 2023-05-31T20:34:52.178 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:34:52.178 DEBUG:boto:Port: 8000 2023-05-31T20:34:52.178 DEBUG:boto:Params: {} 2023-05-31T20:34:52.179 DEBUG:boto:Token: None 2023-05-31T20:34:52.179 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:34:52 GMT /pklxkd-56/e 2023-05-31T20:34:52.180 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:L9+zzTHfiwCdhPMfMSCySj8flfc= 2023-05-31T20:34:52.180 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:34:52 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:L9+zzTHfiwCdhPMfMSCySj8flfc=', 'Content-Length': '0'} 2023-05-31T20:34:52.189 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:30:05 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000aacbed9639892bbd-006477af6c-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:34:52 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:34:52.190 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=e 2023-05-31T20:34:52.191 DEBUG:boto:path=/pklxkd-56/e 2023-05-31T20:34:52.191 DEBUG:boto:auth_path=/pklxkd-56/e 2023-05-31T20:34:52.192 DEBUG:boto:Method: GET 2023-05-31T20:34:52.192 DEBUG:boto:Path: /pklxkd-56/e 2023-05-31T20:34:52.192 DEBUG:boto:Data: 2023-05-31T20:34:52.193 DEBUG:boto:Headers: {} 2023-05-31T20:34:52.193 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:34:52.193 DEBUG:boto:Port: 8001 2023-05-31T20:34:52.194 DEBUG:boto:Params: {} 2023-05-31T20:34:52.195 DEBUG:boto:Token: None 2023-05-31T20:34:52.195 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:34:52 GMT /pklxkd-56/e 2023-05-31T20:34:52.196 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:6Wzcn82GxytjljMoW5RI8VbCFJk= 2023-05-31T20:34:52.196 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:34:52 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:6Wzcn82GxytjljMoW5RI8VbCFJk=', 'Content-Length': '0'} 2023-05-31T20:34:52.202 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:30:05 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008b56df713b75c421-006477af6c-162e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:34:52 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:34:52.203 DEBUG:boto:path=/pklxkd-56/e 2023-05-31T20:34:52.204 DEBUG:boto:auth_path=/pklxkd-56/e 2023-05-31T20:34:52.204 DEBUG:boto:Method: GET 2023-05-31T20:34:52.204 DEBUG:boto:Path: /pklxkd-56/e 2023-05-31T20:34:52.205 DEBUG:boto:Data: 2023-05-31T20:34:52.205 DEBUG:boto:Headers: {} 2023-05-31T20:34:52.206 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:34:52.206 DEBUG:boto:Port: 8000 2023-05-31T20:34:52.207 DEBUG:boto:Params: {} 2023-05-31T20:34:52.207 DEBUG:boto:Token: None 2023-05-31T20:34:52.209 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:34:52 GMT /pklxkd-56/e 2023-05-31T20:34:52.209 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:6Wzcn82GxytjljMoW5RI8VbCFJk= 2023-05-31T20:34:52.210 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:34:52 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:6Wzcn82GxytjljMoW5RI8VbCFJk=', 'Content-Length': '0'} 2023-05-31T20:34:52.212 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:30:05 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f22845c01b593406-006477af6c-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:34:52 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:34:52.252 DEBUG:boto:path=/pklxkd-56/ 2023-05-31T20:34:52.253 DEBUG:boto:auth_path=/pklxkd-56/ 2023-05-31T20:34:52.254 DEBUG:boto:path=/pklxkd-56/?prefix=e 2023-05-31T20:34:52.255 DEBUG:boto:auth_path=/pklxkd-56/?prefix=e 2023-05-31T20:34:52.255 DEBUG:boto:Method: GET 2023-05-31T20:34:52.255 DEBUG:boto:Path: /pklxkd-56/?prefix=e 2023-05-31T20:34:52.256 DEBUG:boto:Data: 2023-05-31T20:34:52.257 DEBUG:boto:Headers: {} 2023-05-31T20:34:52.257 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:34:52.258 DEBUG:boto:Port: 8001 2023-05-31T20:34:52.258 DEBUG:boto:Params: {} 2023-05-31T20:34:52.259 DEBUG:boto:Token: None 2023-05-31T20:34:52.259 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:34:52 GMT /pklxkd-56/ 2023-05-31T20:34:52.260 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:nL4TOhvnP0P54UQXOfqFYR6uCRE= 2023-05-31T20:34:52.260 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:34:52 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:nL4TOhvnP0P54UQXOfqFYR6uCRE=', 'Content-Length': '0'} 2023-05-31T20:34:52.264 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000006f59388bd7bbb6ed-006477af6c-162e-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:34:52 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:34:52.265 DEBUG:boto:b'pklxkd-56e1000falsee2023-05-31T20:30:05.784Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:34:52.266 DEBUG:boto:path=/pklxkd-56/ 2023-05-31T20:34:52.266 DEBUG:boto:auth_path=/pklxkd-56/ 2023-05-31T20:34:52.266 DEBUG:boto:path=/pklxkd-56/?prefix=e 2023-05-31T20:34:52.267 DEBUG:boto:auth_path=/pklxkd-56/?prefix=e 2023-05-31T20:34:52.267 DEBUG:boto:Method: GET 2023-05-31T20:34:52.268 DEBUG:boto:Path: /pklxkd-56/?prefix=e 2023-05-31T20:34:52.268 DEBUG:boto:Data: 2023-05-31T20:34:52.268 DEBUG:boto:Headers: {} 2023-05-31T20:34:52.276 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:34:52.276 DEBUG:boto:Port: 8000 2023-05-31T20:34:52.277 DEBUG:boto:Params: {} 2023-05-31T20:34:52.277 DEBUG:boto:Token: None 2023-05-31T20:34:52.277 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:34:52 GMT /pklxkd-56/ 2023-05-31T20:34:52.278 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:nL4TOhvnP0P54UQXOfqFYR6uCRE= 2023-05-31T20:34:52.279 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:34:52 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:nL4TOhvnP0P54UQXOfqFYR6uCRE=', 'Content-Length': '0'} 2023-05-31T20:34:52.281 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ebaa9f386ecfb3b3-006477af6c-1630-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:34:52 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:34:52.282 DEBUG:boto:b'pklxkd-56e1000falsee2023-05-31T20:30:05.784Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:34:52.283 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=f 2023-05-31T20:34:52.284 DEBUG:boto:path=/pklxkd-56/f 2023-05-31T20:34:52.285 DEBUG:boto:auth_path=/pklxkd-56/f 2023-05-31T20:34:52.286 DEBUG:boto:path=/pklxkd-56/f?versionId=null 2023-05-31T20:34:52.287 DEBUG:boto:auth_path=/pklxkd-56/f?versionId=null 2023-05-31T20:34:52.288 DEBUG:boto:Method: GET 2023-05-31T20:34:52.288 DEBUG:boto:Path: /pklxkd-56/f?versionId=null 2023-05-31T20:34:52.289 DEBUG:boto:Data: 2023-05-31T20:34:52.290 DEBUG:boto:Headers: {} 2023-05-31T20:34:52.291 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:34:52.293 DEBUG:boto:Port: 8001 2023-05-31T20:34:52.294 DEBUG:boto:Params: {} 2023-05-31T20:34:52.296 DEBUG:boto:Token: None 2023-05-31T20:34:52.298 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:34:52 GMT /pklxkd-56/f?versionId=null 2023-05-31T20:34:52.299 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:nnUEr5ToU3UmnXokv1Id4wgiWfQ= 2023-05-31T20:34:52.301 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:34:52 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:nnUEr5ToU3UmnXokv1Id4wgiWfQ=', 'Content-Length': '0'} 2023-05-31T20:34:52.304 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:30:05 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000020598c8e2ccfef86-006477af6c-162e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:34:52 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:34:52.306 DEBUG:boto:path=/pklxkd-56/f 2023-05-31T20:34:52.307 DEBUG:boto:auth_path=/pklxkd-56/f 2023-05-31T20:34:52.307 DEBUG:boto:path=/pklxkd-56/f?versionId=null 2023-05-31T20:34:52.307 DEBUG:boto:auth_path=/pklxkd-56/f?versionId=null 2023-05-31T20:34:52.308 DEBUG:boto:Method: GET 2023-05-31T20:34:52.309 DEBUG:boto:Path: /pklxkd-56/f?versionId=null 2023-05-31T20:34:52.309 DEBUG:boto:Data: 2023-05-31T20:34:52.310 DEBUG:boto:Headers: {} 2023-05-31T20:34:52.310 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:34:52.311 DEBUG:boto:Port: 8000 2023-05-31T20:34:52.311 DEBUG:boto:Params: {} 2023-05-31T20:34:52.311 DEBUG:boto:Token: None 2023-05-31T20:34:52.312 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:34:52 GMT /pklxkd-56/f?versionId=null 2023-05-31T20:34:52.312 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:nnUEr5ToU3UmnXokv1Id4wgiWfQ= 2023-05-31T20:34:52.313 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:34:52 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:nnUEr5ToU3UmnXokv1Id4wgiWfQ=', 'Content-Length': '0'} 2023-05-31T20:34:52.315 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:30:05 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ddb2bfff8deef3e5-006477af6c-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:34:52 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:34:52.316 DEBUG:boto:path=/pklxkd-56/f 2023-05-31T20:34:52.316 DEBUG:boto:auth_path=/pklxkd-56/f 2023-05-31T20:34:52.317 DEBUG:boto:path=/pklxkd-56/f?versionId=null 2023-05-31T20:34:52.317 DEBUG:boto:auth_path=/pklxkd-56/f?versionId=null 2023-05-31T20:34:52.317 DEBUG:boto:Method: HEAD 2023-05-31T20:34:52.318 DEBUG:boto:Path: /pklxkd-56/f?versionId=null 2023-05-31T20:34:52.318 DEBUG:boto:Data: 2023-05-31T20:34:52.318 DEBUG:boto:Headers: {} 2023-05-31T20:34:52.319 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:34:52.319 DEBUG:boto:Port: 8001 2023-05-31T20:34:52.320 DEBUG:boto:Params: {} 2023-05-31T20:34:52.320 DEBUG:boto:Token: None 2023-05-31T20:34:52.321 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:34:52 GMT /pklxkd-56/f?versionId=null 2023-05-31T20:34:52.321 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:IzeiQHolos2I6LdD9XjRnzTMsA0= 2023-05-31T20:34:52.321 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:34:52 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:IzeiQHolos2I6LdD9XjRnzTMsA0=', 'Content-Length': '0'} 2023-05-31T20:34:52.323 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:30:05 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f8704496c390e99e-006477af6c-162e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:34:52 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:34:52.324 DEBUG:boto:path=/pklxkd-56/f 2023-05-31T20:34:52.325 DEBUG:boto:auth_path=/pklxkd-56/f 2023-05-31T20:34:52.325 DEBUG:boto:path=/pklxkd-56/f?versionId=null 2023-05-31T20:34:52.325 DEBUG:boto:auth_path=/pklxkd-56/f?versionId=null 2023-05-31T20:34:52.326 DEBUG:boto:Method: HEAD 2023-05-31T20:34:52.327 DEBUG:boto:Path: /pklxkd-56/f?versionId=null 2023-05-31T20:34:52.327 DEBUG:boto:Data: 2023-05-31T20:34:52.328 DEBUG:boto:Headers: {} 2023-05-31T20:34:52.328 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:34:52.329 DEBUG:boto:Port: 8000 2023-05-31T20:34:52.330 DEBUG:boto:Params: {} 2023-05-31T20:34:52.330 DEBUG:boto:Token: None 2023-05-31T20:34:52.331 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:34:52 GMT /pklxkd-56/f?versionId=null 2023-05-31T20:34:52.332 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:IzeiQHolos2I6LdD9XjRnzTMsA0= 2023-05-31T20:34:52.332 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:34:52 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:IzeiQHolos2I6LdD9XjRnzTMsA0=', 'Content-Length': '0'} 2023-05-31T20:34:52.344 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:30:05 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000013c09864c3a8121b-006477af6c-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:34:52 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:34:52.345 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=f 2023-05-31T20:34:52.346 DEBUG:boto:path=/pklxkd-56/f 2023-05-31T20:34:52.346 DEBUG:boto:auth_path=/pklxkd-56/f 2023-05-31T20:34:52.347 DEBUG:boto:path=/pklxkd-56/f?versionId=null 2023-05-31T20:34:52.347 DEBUG:boto:auth_path=/pklxkd-56/f?versionId=null 2023-05-31T20:34:52.350 DEBUG:boto:Method: GET 2023-05-31T20:34:52.351 DEBUG:boto:Path: /pklxkd-56/f?versionId=null 2023-05-31T20:34:52.351 DEBUG:boto:Data: 2023-05-31T20:34:52.352 DEBUG:boto:Headers: {} 2023-05-31T20:34:52.352 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:34:52.352 DEBUG:boto:Port: 8001 2023-05-31T20:34:52.353 DEBUG:boto:Params: {} 2023-05-31T20:34:52.353 DEBUG:boto:Token: None 2023-05-31T20:34:52.354 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:34:52 GMT /pklxkd-56/f?versionId=null 2023-05-31T20:34:52.354 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:nnUEr5ToU3UmnXokv1Id4wgiWfQ= 2023-05-31T20:34:52.355 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:34:52 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:nnUEr5ToU3UmnXokv1Id4wgiWfQ=', 'Content-Length': '0'} 2023-05-31T20:34:52.356 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:30:05 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000fd8ced1568125eed-006477af6c-162e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:34:52 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:34:52.406 DEBUG:boto:path=/pklxkd-56/f 2023-05-31T20:34:52.407 DEBUG:boto:auth_path=/pklxkd-56/f 2023-05-31T20:34:52.407 DEBUG:boto:path=/pklxkd-56/f?versionId=null 2023-05-31T20:34:52.408 DEBUG:boto:auth_path=/pklxkd-56/f?versionId=null 2023-05-31T20:34:52.408 DEBUG:boto:Method: GET 2023-05-31T20:34:52.409 DEBUG:boto:Path: /pklxkd-56/f?versionId=null 2023-05-31T20:34:52.409 DEBUG:boto:Data: 2023-05-31T20:34:52.409 DEBUG:boto:Headers: {} 2023-05-31T20:34:52.410 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:34:52.410 DEBUG:boto:Port: 8000 2023-05-31T20:34:52.410 DEBUG:boto:Params: {} 2023-05-31T20:34:52.411 DEBUG:boto:Token: None 2023-05-31T20:34:52.411 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:34:52 GMT /pklxkd-56/f?versionId=null 2023-05-31T20:34:52.412 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:nnUEr5ToU3UmnXokv1Id4wgiWfQ= 2023-05-31T20:34:52.412 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:34:52 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:nnUEr5ToU3UmnXokv1Id4wgiWfQ=', 'Content-Length': '0'} 2023-05-31T20:34:52.414 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:30:05 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000716600a8f9d46869-006477af6c-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:34:52 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:34:52.415 DEBUG:boto:path=/pklxkd-56/ 2023-05-31T20:34:52.416 DEBUG:boto:auth_path=/pklxkd-56/ 2023-05-31T20:34:52.416 DEBUG:boto:path=/pklxkd-56/?prefix=f 2023-05-31T20:34:52.417 DEBUG:boto:auth_path=/pklxkd-56/?prefix=f 2023-05-31T20:34:52.417 DEBUG:boto:Method: GET 2023-05-31T20:34:52.418 DEBUG:boto:Path: /pklxkd-56/?prefix=f 2023-05-31T20:34:52.418 DEBUG:boto:Data: 2023-05-31T20:34:52.418 DEBUG:boto:Headers: {} 2023-05-31T20:34:52.419 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:34:52.419 DEBUG:boto:Port: 8001 2023-05-31T20:34:52.420 DEBUG:boto:Params: {} 2023-05-31T20:34:52.420 DEBUG:boto:Token: None 2023-05-31T20:34:52.421 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:34:52 GMT /pklxkd-56/ 2023-05-31T20:34:52.421 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:nL4TOhvnP0P54UQXOfqFYR6uCRE= 2023-05-31T20:34:52.421 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:34:52 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:nL4TOhvnP0P54UQXOfqFYR6uCRE=', 'Content-Length': '0'} 2023-05-31T20:34:52.424 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000009d9d6dcc1eb3cfd4-006477af6c-162e-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:34:52 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:34:52.424 DEBUG:boto:b'pklxkd-56f1000falsef2023-05-31T20:30:05.809Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:34:52.425 DEBUG:boto:path=/pklxkd-56/ 2023-05-31T20:34:52.426 DEBUG:boto:auth_path=/pklxkd-56/ 2023-05-31T20:34:52.426 DEBUG:boto:path=/pklxkd-56/?prefix=f 2023-05-31T20:34:52.426 DEBUG:boto:auth_path=/pklxkd-56/?prefix=f 2023-05-31T20:34:52.427 DEBUG:boto:Method: GET 2023-05-31T20:34:52.427 DEBUG:boto:Path: /pklxkd-56/?prefix=f 2023-05-31T20:34:52.427 DEBUG:boto:Data: 2023-05-31T20:34:52.428 DEBUG:boto:Headers: {} 2023-05-31T20:34:52.428 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:34:52.429 DEBUG:boto:Port: 8000 2023-05-31T20:34:52.429 DEBUG:boto:Params: {} 2023-05-31T20:34:52.430 DEBUG:boto:Token: None 2023-05-31T20:34:52.430 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:34:52 GMT /pklxkd-56/ 2023-05-31T20:34:52.430 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:nL4TOhvnP0P54UQXOfqFYR6uCRE= 2023-05-31T20:34:52.431 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:34:52 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:nL4TOhvnP0P54UQXOfqFYR6uCRE=', 'Content-Length': '0'} 2023-05-31T20:34:52.433 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000068487718b9e5a9ca-006477af6c-1630-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:34:52 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:34:52.434 DEBUG:boto:b'pklxkd-56f1000falsef2023-05-31T20:30:05.809Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:34:52.434 DEBUG:boto:path=/pklxkd-56/f 2023-05-31T20:34:52.436 DEBUG:boto:auth_path=/pklxkd-56/f 2023-05-31T20:34:52.437 DEBUG:boto:Method: HEAD 2023-05-31T20:34:52.437 DEBUG:boto:Path: /pklxkd-56/f 2023-05-31T20:34:52.437 DEBUG:boto:Data: 2023-05-31T20:34:52.438 DEBUG:boto:Headers: {} 2023-05-31T20:34:52.438 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:34:52.438 DEBUG:boto:Port: 8001 2023-05-31T20:34:52.439 DEBUG:boto:Params: {} 2023-05-31T20:34:52.439 DEBUG:boto:Token: None 2023-05-31T20:34:52.439 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:34:52 GMT /pklxkd-56/f 2023-05-31T20:34:52.440 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:yWNhFdVArRquKcEobMO1qYaBckw= 2023-05-31T20:34:52.440 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:34:52 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:yWNhFdVArRquKcEobMO1qYaBckw=', 'Content-Length': '0'} 2023-05-31T20:34:52.442 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:30:05 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d5301802bc33e508-006477af6c-162e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:34:52 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:34:52.443 DEBUG:boto:path=/pklxkd-56/f 2023-05-31T20:34:52.443 DEBUG:boto:auth_path=/pklxkd-56/f 2023-05-31T20:34:52.444 DEBUG:boto:Method: HEAD 2023-05-31T20:34:52.444 DEBUG:boto:Path: /pklxkd-56/f 2023-05-31T20:34:52.445 DEBUG:boto:Data: 2023-05-31T20:34:52.446 DEBUG:boto:Headers: {} 2023-05-31T20:34:52.446 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:34:52.446 DEBUG:boto:Port: 8000 2023-05-31T20:34:52.447 DEBUG:boto:Params: {} 2023-05-31T20:34:52.447 DEBUG:boto:Token: None 2023-05-31T20:34:52.447 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:34:52 GMT /pklxkd-56/f 2023-05-31T20:34:52.448 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:yWNhFdVArRquKcEobMO1qYaBckw= 2023-05-31T20:34:52.448 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:34:52 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:yWNhFdVArRquKcEobMO1qYaBckw=', 'Content-Length': '0'} 2023-05-31T20:34:52.450 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:30:05 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000091c44f94977a5f7c-006477af6c-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:34:52 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:34:52.453 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=f 2023-05-31T20:34:52.453 DEBUG:boto:path=/pklxkd-56/f 2023-05-31T20:34:52.454 DEBUG:boto:auth_path=/pklxkd-56/f 2023-05-31T20:34:52.454 DEBUG:boto:Method: GET 2023-05-31T20:34:52.454 DEBUG:boto:Path: /pklxkd-56/f 2023-05-31T20:34:52.455 DEBUG:boto:Data: 2023-05-31T20:34:52.455 DEBUG:boto:Headers: {} 2023-05-31T20:34:52.455 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:34:52.456 DEBUG:boto:Port: 8001 2023-05-31T20:34:52.456 DEBUG:boto:Params: {} 2023-05-31T20:34:52.457 DEBUG:boto:Token: None 2023-05-31T20:34:52.457 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:34:52 GMT /pklxkd-56/f 2023-05-31T20:34:52.458 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:1w1UcxxZ3cL042IJb6l6JJ2z2fo= 2023-05-31T20:34:52.458 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:34:52 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:1w1UcxxZ3cL042IJb6l6JJ2z2fo=', 'Content-Length': '0'} 2023-05-31T20:34:52.460 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:30:05 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000dbf57a1432713989-006477af6c-162e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:34:52 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:34:52.460 DEBUG:boto:path=/pklxkd-56/f 2023-05-31T20:34:52.461 DEBUG:boto:auth_path=/pklxkd-56/f 2023-05-31T20:34:52.461 DEBUG:boto:Method: GET 2023-05-31T20:34:52.462 DEBUG:boto:Path: /pklxkd-56/f 2023-05-31T20:34:52.462 DEBUG:boto:Data: 2023-05-31T20:34:52.462 DEBUG:boto:Headers: {} 2023-05-31T20:34:52.463 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:34:52.463 DEBUG:boto:Port: 8000 2023-05-31T20:34:52.463 DEBUG:boto:Params: {} 2023-05-31T20:34:52.464 DEBUG:boto:Token: None 2023-05-31T20:34:52.464 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:34:52 GMT /pklxkd-56/f 2023-05-31T20:34:52.465 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:1w1UcxxZ3cL042IJb6l6JJ2z2fo= 2023-05-31T20:34:52.465 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:34:52 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:1w1UcxxZ3cL042IJb6l6JJ2z2fo=', 'Content-Length': '0'} 2023-05-31T20:34:52.468 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:30:05 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ed3f842f15040813-006477af6c-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:34:52 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:34:52.508 DEBUG:boto:path=/pklxkd-56/ 2023-05-31T20:34:52.509 DEBUG:boto:auth_path=/pklxkd-56/ 2023-05-31T20:34:52.510 DEBUG:boto:path=/pklxkd-56/?prefix=f 2023-05-31T20:34:52.510 DEBUG:boto:auth_path=/pklxkd-56/?prefix=f 2023-05-31T20:34:52.511 DEBUG:boto:Method: GET 2023-05-31T20:34:52.511 DEBUG:boto:Path: /pklxkd-56/?prefix=f 2023-05-31T20:34:52.511 DEBUG:boto:Data: 2023-05-31T20:34:52.512 DEBUG:boto:Headers: {} 2023-05-31T20:34:52.512 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:34:52.513 DEBUG:boto:Port: 8001 2023-05-31T20:34:52.514 DEBUG:boto:Params: {} 2023-05-31T20:34:52.514 DEBUG:boto:Token: None 2023-05-31T20:34:52.515 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:34:52 GMT /pklxkd-56/ 2023-05-31T20:34:52.515 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:nL4TOhvnP0P54UQXOfqFYR6uCRE= 2023-05-31T20:34:52.516 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:34:52 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:nL4TOhvnP0P54UQXOfqFYR6uCRE=', 'Content-Length': '0'} 2023-05-31T20:34:52.518 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000387325fc3db9edb0-006477af6c-162e-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:34:52 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:34:52.520 DEBUG:boto:b'pklxkd-56f1000falsef2023-05-31T20:30:05.809Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:34:52.521 DEBUG:boto:path=/pklxkd-56/ 2023-05-31T20:34:52.522 DEBUG:boto:auth_path=/pklxkd-56/ 2023-05-31T20:34:52.523 DEBUG:boto:path=/pklxkd-56/?prefix=f 2023-05-31T20:34:52.524 DEBUG:boto:auth_path=/pklxkd-56/?prefix=f 2023-05-31T20:34:52.525 DEBUG:boto:Method: GET 2023-05-31T20:34:52.526 DEBUG:boto:Path: /pklxkd-56/?prefix=f 2023-05-31T20:34:52.526 DEBUG:boto:Data: 2023-05-31T20:34:52.527 DEBUG:boto:Headers: {} 2023-05-31T20:34:52.528 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:34:52.529 DEBUG:boto:Port: 8000 2023-05-31T20:34:52.529 DEBUG:boto:Params: {} 2023-05-31T20:34:52.530 DEBUG:boto:Token: None 2023-05-31T20:34:52.531 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:34:52 GMT /pklxkd-56/ 2023-05-31T20:34:52.532 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:nL4TOhvnP0P54UQXOfqFYR6uCRE= 2023-05-31T20:34:52.533 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:34:52 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:nL4TOhvnP0P54UQXOfqFYR6uCRE=', 'Content-Length': '0'} 2023-05-31T20:34:52.536 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000008c6a24b5e77a1a2e-006477af6c-1630-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:34:52 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:34:52.537 DEBUG:boto:b'pklxkd-56f1000falsef2023-05-31T20:30:05.809Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:34:52.539 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=g 2023-05-31T20:34:52.541 DEBUG:boto:path=/pklxkd-56/g 2023-05-31T20:34:52.541 DEBUG:boto:auth_path=/pklxkd-56/g 2023-05-31T20:34:52.542 DEBUG:boto:path=/pklxkd-56/g?versionId=null 2023-05-31T20:34:52.542 DEBUG:boto:auth_path=/pklxkd-56/g?versionId=null 2023-05-31T20:34:52.543 DEBUG:boto:Method: GET 2023-05-31T20:34:52.543 DEBUG:boto:Path: /pklxkd-56/g?versionId=null 2023-05-31T20:34:52.543 DEBUG:boto:Data: 2023-05-31T20:34:52.544 DEBUG:boto:Headers: {} 2023-05-31T20:34:52.544 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:34:52.544 DEBUG:boto:Port: 8001 2023-05-31T20:34:52.545 DEBUG:boto:Params: {} 2023-05-31T20:34:52.545 DEBUG:boto:Token: None 2023-05-31T20:34:52.546 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:34:52 GMT /pklxkd-56/g?versionId=null 2023-05-31T20:34:52.546 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:hNEd+mayiw29cmkzljZ8thu1m+Q= 2023-05-31T20:34:52.547 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:34:52 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:hNEd+mayiw29cmkzljZ8thu1m+Q=', 'Content-Length': '0'} 2023-05-31T20:34:52.549 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:30:05 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000d7d3df57ce8d8f3-006477af6c-162e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:34:52 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:34:52.550 DEBUG:boto:path=/pklxkd-56/g 2023-05-31T20:34:52.550 DEBUG:boto:auth_path=/pklxkd-56/g 2023-05-31T20:34:52.551 DEBUG:boto:path=/pklxkd-56/g?versionId=null 2023-05-31T20:34:52.551 DEBUG:boto:auth_path=/pklxkd-56/g?versionId=null 2023-05-31T20:34:52.551 DEBUG:boto:Method: GET 2023-05-31T20:34:52.552 DEBUG:boto:Path: /pklxkd-56/g?versionId=null 2023-05-31T20:34:52.552 DEBUG:boto:Data: 2023-05-31T20:34:52.553 DEBUG:boto:Headers: {} 2023-05-31T20:34:52.553 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:34:52.553 DEBUG:boto:Port: 8000 2023-05-31T20:34:52.554 DEBUG:boto:Params: {} 2023-05-31T20:34:52.554 DEBUG:boto:Token: None 2023-05-31T20:34:52.554 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:34:52 GMT /pklxkd-56/g?versionId=null 2023-05-31T20:34:52.555 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:hNEd+mayiw29cmkzljZ8thu1m+Q= 2023-05-31T20:34:52.555 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:34:52 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:hNEd+mayiw29cmkzljZ8thu1m+Q=', 'Content-Length': '0'} 2023-05-31T20:34:52.557 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:30:05 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000027d49805cd3f6e3d-006477af6c-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:34:52 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:34:52.558 DEBUG:boto:path=/pklxkd-56/g 2023-05-31T20:34:52.558 DEBUG:boto:auth_path=/pklxkd-56/g 2023-05-31T20:34:52.559 DEBUG:boto:path=/pklxkd-56/g?versionId=null 2023-05-31T20:34:52.559 DEBUG:boto:auth_path=/pklxkd-56/g?versionId=null 2023-05-31T20:34:52.559 DEBUG:boto:Method: HEAD 2023-05-31T20:34:52.560 DEBUG:boto:Path: /pklxkd-56/g?versionId=null 2023-05-31T20:34:52.560 DEBUG:boto:Data: 2023-05-31T20:34:52.560 DEBUG:boto:Headers: {} 2023-05-31T20:34:52.561 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:34:52.561 DEBUG:boto:Port: 8001 2023-05-31T20:34:52.561 DEBUG:boto:Params: {} 2023-05-31T20:34:52.562 DEBUG:boto:Token: None 2023-05-31T20:34:52.562 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:34:52 GMT /pklxkd-56/g?versionId=null 2023-05-31T20:34:52.563 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:YHG2wC+R88lP4/v+M1c2EeQpe7k= 2023-05-31T20:34:52.563 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:34:52 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:YHG2wC+R88lP4/v+M1c2EeQpe7k=', 'Content-Length': '0'} 2023-05-31T20:34:52.565 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:30:05 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d3cf49c24cad4379-006477af6c-162e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:34:52 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:34:52.566 DEBUG:boto:path=/pklxkd-56/g 2023-05-31T20:34:52.566 DEBUG:boto:auth_path=/pklxkd-56/g 2023-05-31T20:34:52.566 DEBUG:boto:path=/pklxkd-56/g?versionId=null 2023-05-31T20:34:52.567 DEBUG:boto:auth_path=/pklxkd-56/g?versionId=null 2023-05-31T20:34:52.567 DEBUG:boto:Method: HEAD 2023-05-31T20:34:52.568 DEBUG:boto:Path: /pklxkd-56/g?versionId=null 2023-05-31T20:34:52.568 DEBUG:boto:Data: 2023-05-31T20:34:52.569 DEBUG:boto:Headers: {} 2023-05-31T20:34:52.569 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:34:52.569 DEBUG:boto:Port: 8000 2023-05-31T20:34:52.570 DEBUG:boto:Params: {} 2023-05-31T20:34:52.570 DEBUG:boto:Token: None 2023-05-31T20:34:52.571 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:34:52 GMT /pklxkd-56/g?versionId=null 2023-05-31T20:34:52.571 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:YHG2wC+R88lP4/v+M1c2EeQpe7k= 2023-05-31T20:34:52.571 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:34:52 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:YHG2wC+R88lP4/v+M1c2EeQpe7k=', 'Content-Length': '0'} 2023-05-31T20:34:52.573 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:30:05 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000721972b4810f42f4-006477af6c-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:34:52 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:34:52.574 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=g 2023-05-31T20:34:52.574 DEBUG:boto:path=/pklxkd-56/g 2023-05-31T20:34:52.575 DEBUG:boto:auth_path=/pklxkd-56/g 2023-05-31T20:34:52.575 DEBUG:boto:path=/pklxkd-56/g?versionId=null 2023-05-31T20:34:52.576 DEBUG:boto:auth_path=/pklxkd-56/g?versionId=null 2023-05-31T20:34:52.576 DEBUG:boto:Method: GET 2023-05-31T20:34:52.576 DEBUG:boto:Path: /pklxkd-56/g?versionId=null 2023-05-31T20:34:52.577 DEBUG:boto:Data: 2023-05-31T20:34:52.578 DEBUG:boto:Headers: {} 2023-05-31T20:34:52.578 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:34:52.579 DEBUG:boto:Port: 8001 2023-05-31T20:34:52.579 DEBUG:boto:Params: {} 2023-05-31T20:34:52.579 DEBUG:boto:Token: None 2023-05-31T20:34:52.580 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:34:52 GMT /pklxkd-56/g?versionId=null 2023-05-31T20:34:52.580 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:hNEd+mayiw29cmkzljZ8thu1m+Q= 2023-05-31T20:34:52.581 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:34:52 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:hNEd+mayiw29cmkzljZ8thu1m+Q=', 'Content-Length': '0'} 2023-05-31T20:34:52.583 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:30:05 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007efd6174de1ed552-006477af6c-162e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:34:52 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:34:52.624 DEBUG:boto:path=/pklxkd-56/g 2023-05-31T20:34:52.625 DEBUG:boto:auth_path=/pklxkd-56/g 2023-05-31T20:34:52.625 DEBUG:boto:path=/pklxkd-56/g?versionId=null 2023-05-31T20:34:52.626 DEBUG:boto:auth_path=/pklxkd-56/g?versionId=null 2023-05-31T20:34:52.626 DEBUG:boto:Method: GET 2023-05-31T20:34:52.628 DEBUG:boto:Path: /pklxkd-56/g?versionId=null 2023-05-31T20:34:52.628 DEBUG:boto:Data: 2023-05-31T20:34:52.629 DEBUG:boto:Headers: {} 2023-05-31T20:34:52.629 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:34:52.629 DEBUG:boto:Port: 8000 2023-05-31T20:34:52.630 DEBUG:boto:Params: {} 2023-05-31T20:34:52.631 DEBUG:boto:Token: None 2023-05-31T20:34:52.631 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:34:52 GMT /pklxkd-56/g?versionId=null 2023-05-31T20:34:52.632 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:hNEd+mayiw29cmkzljZ8thu1m+Q= 2023-05-31T20:34:52.632 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:34:52 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:hNEd+mayiw29cmkzljZ8thu1m+Q=', 'Content-Length': '0'} 2023-05-31T20:34:52.634 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:30:05 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f0ca6285aaee9436-006477af6c-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:34:52 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:34:52.635 DEBUG:boto:path=/pklxkd-56/ 2023-05-31T20:34:52.635 DEBUG:boto:auth_path=/pklxkd-56/ 2023-05-31T20:34:52.636 DEBUG:boto:path=/pklxkd-56/?prefix=g 2023-05-31T20:34:52.636 DEBUG:boto:auth_path=/pklxkd-56/?prefix=g 2023-05-31T20:34:52.636 DEBUG:boto:Method: GET 2023-05-31T20:34:52.637 DEBUG:boto:Path: /pklxkd-56/?prefix=g 2023-05-31T20:34:52.637 DEBUG:boto:Data: 2023-05-31T20:34:52.637 DEBUG:boto:Headers: {} 2023-05-31T20:34:52.638 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:34:52.638 DEBUG:boto:Port: 8001 2023-05-31T20:34:52.639 DEBUG:boto:Params: {} 2023-05-31T20:34:52.639 DEBUG:boto:Token: None 2023-05-31T20:34:52.639 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:34:52 GMT /pklxkd-56/ 2023-05-31T20:34:52.640 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:nL4TOhvnP0P54UQXOfqFYR6uCRE= 2023-05-31T20:34:52.640 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:34:52 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:nL4TOhvnP0P54UQXOfqFYR6uCRE=', 'Content-Length': '0'} 2023-05-31T20:34:52.643 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b87a292c566476c7-006477af6c-162e-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:34:52 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:34:52.643 DEBUG:boto:b'pklxkd-56g1000falseg2023-05-31T20:30:05.841Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:34:52.644 DEBUG:boto:path=/pklxkd-56/ 2023-05-31T20:34:52.645 DEBUG:boto:auth_path=/pklxkd-56/ 2023-05-31T20:34:52.645 DEBUG:boto:path=/pklxkd-56/?prefix=g 2023-05-31T20:34:52.645 DEBUG:boto:auth_path=/pklxkd-56/?prefix=g 2023-05-31T20:34:52.646 DEBUG:boto:Method: GET 2023-05-31T20:34:52.646 DEBUG:boto:Path: /pklxkd-56/?prefix=g 2023-05-31T20:34:52.646 DEBUG:boto:Data: 2023-05-31T20:34:52.647 DEBUG:boto:Headers: {} 2023-05-31T20:34:52.647 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:34:52.647 DEBUG:boto:Port: 8000 2023-05-31T20:34:52.648 DEBUG:boto:Params: {} 2023-05-31T20:34:52.649 DEBUG:boto:Token: None 2023-05-31T20:34:52.649 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:34:52 GMT /pklxkd-56/ 2023-05-31T20:34:52.649 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:nL4TOhvnP0P54UQXOfqFYR6uCRE= 2023-05-31T20:34:52.650 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:34:52 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:nL4TOhvnP0P54UQXOfqFYR6uCRE=', 'Content-Length': '0'} 2023-05-31T20:34:52.653 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d9e225263464e09c-006477af6c-1630-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:34:52 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:34:52.653 DEBUG:boto:b'pklxkd-56g1000falseg2023-05-31T20:30:05.841Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:34:52.654 DEBUG:boto:path=/pklxkd-56/g 2023-05-31T20:34:52.654 DEBUG:boto:auth_path=/pklxkd-56/g 2023-05-31T20:34:52.655 DEBUG:boto:Method: HEAD 2023-05-31T20:34:52.655 DEBUG:boto:Path: /pklxkd-56/g 2023-05-31T20:34:52.655 DEBUG:boto:Data: 2023-05-31T20:34:52.656 DEBUG:boto:Headers: {} 2023-05-31T20:34:52.656 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:34:52.656 DEBUG:boto:Port: 8001 2023-05-31T20:34:52.657 DEBUG:boto:Params: {} 2023-05-31T20:34:52.657 DEBUG:boto:Token: None 2023-05-31T20:34:52.658 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:34:52 GMT /pklxkd-56/g 2023-05-31T20:34:52.658 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:/c7G4vsuf/1Qf6BPNQq2cMAfJBk= 2023-05-31T20:34:52.658 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:34:52 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:/c7G4vsuf/1Qf6BPNQq2cMAfJBk=', 'Content-Length': '0'} 2023-05-31T20:34:52.660 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:30:05 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000054f4252a7d98db85-006477af6c-162e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:34:52 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:34:52.661 DEBUG:boto:path=/pklxkd-56/g 2023-05-31T20:34:52.663 DEBUG:boto:auth_path=/pklxkd-56/g 2023-05-31T20:34:52.663 DEBUG:boto:Method: HEAD 2023-05-31T20:34:52.663 DEBUG:boto:Path: /pklxkd-56/g 2023-05-31T20:34:52.664 DEBUG:boto:Data: 2023-05-31T20:34:52.664 DEBUG:boto:Headers: {} 2023-05-31T20:34:52.665 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:34:52.665 DEBUG:boto:Port: 8000 2023-05-31T20:34:52.665 DEBUG:boto:Params: {} 2023-05-31T20:34:52.666 DEBUG:boto:Token: None 2023-05-31T20:34:52.666 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:34:52 GMT /pklxkd-56/g 2023-05-31T20:34:52.667 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:/c7G4vsuf/1Qf6BPNQq2cMAfJBk= 2023-05-31T20:34:52.667 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:34:52 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:/c7G4vsuf/1Qf6BPNQq2cMAfJBk=', 'Content-Length': '0'} 2023-05-31T20:34:52.669 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:30:05 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000adc084febdb76c7a-006477af6c-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:34:52 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:34:52.670 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=g 2023-05-31T20:34:52.670 DEBUG:boto:path=/pklxkd-56/g 2023-05-31T20:34:52.672 DEBUG:boto:auth_path=/pklxkd-56/g 2023-05-31T20:34:52.673 DEBUG:boto:Method: GET 2023-05-31T20:34:52.673 DEBUG:boto:Path: /pklxkd-56/g 2023-05-31T20:34:52.674 DEBUG:boto:Data: 2023-05-31T20:34:52.674 DEBUG:boto:Headers: {} 2023-05-31T20:34:52.675 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:34:52.675 DEBUG:boto:Port: 8001 2023-05-31T20:34:52.675 DEBUG:boto:Params: {} 2023-05-31T20:34:52.676 DEBUG:boto:Token: None 2023-05-31T20:34:52.676 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:34:52 GMT /pklxkd-56/g 2023-05-31T20:34:52.677 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:B2qk5KrMqg/AcD7uD/89rpSgjnA= 2023-05-31T20:34:52.677 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:34:52 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:B2qk5KrMqg/AcD7uD/89rpSgjnA=', 'Content-Length': '0'} 2023-05-31T20:34:52.680 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:30:05 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f9c33b0219a5288d-006477af6c-162e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:34:52 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:34:52.680 DEBUG:boto:path=/pklxkd-56/g 2023-05-31T20:34:52.681 DEBUG:boto:auth_path=/pklxkd-56/g 2023-05-31T20:34:52.681 DEBUG:boto:Method: GET 2023-05-31T20:34:52.682 DEBUG:boto:Path: /pklxkd-56/g 2023-05-31T20:34:52.682 DEBUG:boto:Data: 2023-05-31T20:34:52.683 DEBUG:boto:Headers: {} 2023-05-31T20:34:52.683 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:34:52.683 DEBUG:boto:Port: 8000 2023-05-31T20:34:52.684 DEBUG:boto:Params: {} 2023-05-31T20:34:52.684 DEBUG:boto:Token: None 2023-05-31T20:34:52.685 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:34:52 GMT /pklxkd-56/g 2023-05-31T20:34:52.685 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:B2qk5KrMqg/AcD7uD/89rpSgjnA= 2023-05-31T20:34:52.686 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:34:52 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:B2qk5KrMqg/AcD7uD/89rpSgjnA=', 'Content-Length': '0'} 2023-05-31T20:34:52.688 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:30:05 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000046cac1e1b7d3bee4-006477af6c-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:34:52 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:34:52.728 DEBUG:boto:path=/pklxkd-56/ 2023-05-31T20:34:52.729 DEBUG:boto:auth_path=/pklxkd-56/ 2023-05-31T20:34:52.729 DEBUG:boto:path=/pklxkd-56/?prefix=g 2023-05-31T20:34:52.730 DEBUG:boto:auth_path=/pklxkd-56/?prefix=g 2023-05-31T20:34:52.730 DEBUG:boto:Method: GET 2023-05-31T20:34:52.731 DEBUG:boto:Path: /pklxkd-56/?prefix=g 2023-05-31T20:34:52.731 DEBUG:boto:Data: 2023-05-31T20:34:52.731 DEBUG:boto:Headers: {} 2023-05-31T20:34:52.732 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:34:52.732 DEBUG:boto:Port: 8001 2023-05-31T20:34:52.733 DEBUG:boto:Params: {} 2023-05-31T20:34:52.734 DEBUG:boto:Token: None 2023-05-31T20:34:52.735 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:34:52 GMT /pklxkd-56/ 2023-05-31T20:34:52.735 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:nL4TOhvnP0P54UQXOfqFYR6uCRE= 2023-05-31T20:34:52.736 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:34:52 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:nL4TOhvnP0P54UQXOfqFYR6uCRE=', 'Content-Length': '0'} 2023-05-31T20:34:52.739 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a9d115bb1491b46e-006477af6c-162e-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:34:52 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:34:52.740 DEBUG:boto:b'pklxkd-56g1000falseg2023-05-31T20:30:05.841Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:34:52.742 DEBUG:boto:path=/pklxkd-56/ 2023-05-31T20:34:52.742 DEBUG:boto:auth_path=/pklxkd-56/ 2023-05-31T20:34:52.744 DEBUG:boto:path=/pklxkd-56/?prefix=g 2023-05-31T20:34:52.745 DEBUG:boto:auth_path=/pklxkd-56/?prefix=g 2023-05-31T20:34:52.746 DEBUG:boto:Method: GET 2023-05-31T20:34:52.747 DEBUG:boto:Path: /pklxkd-56/?prefix=g 2023-05-31T20:34:52.747 DEBUG:boto:Data: 2023-05-31T20:34:52.748 DEBUG:boto:Headers: {} 2023-05-31T20:34:52.749 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:34:52.749 DEBUG:boto:Port: 8000 2023-05-31T20:34:52.750 DEBUG:boto:Params: {} 2023-05-31T20:34:52.751 DEBUG:boto:Token: None 2023-05-31T20:34:52.751 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:34:52 GMT /pklxkd-56/ 2023-05-31T20:34:52.752 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:nL4TOhvnP0P54UQXOfqFYR6uCRE= 2023-05-31T20:34:52.753 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:34:52 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:nL4TOhvnP0P54UQXOfqFYR6uCRE=', 'Content-Length': '0'} 2023-05-31T20:34:52.757 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000088314ad2178b6df1-006477af6c-1630-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:34:52 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:34:52.758 DEBUG:boto:b'pklxkd-56g1000falseg2023-05-31T20:30:05.841Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:34:52.759 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=h 2023-05-31T20:34:52.760 DEBUG:boto:path=/pklxkd-56/h 2023-05-31T20:34:52.761 DEBUG:boto:auth_path=/pklxkd-56/h 2023-05-31T20:34:52.762 DEBUG:boto:path=/pklxkd-56/h?versionId=null 2023-05-31T20:34:52.763 DEBUG:boto:auth_path=/pklxkd-56/h?versionId=null 2023-05-31T20:34:52.763 DEBUG:boto:Method: GET 2023-05-31T20:34:52.764 DEBUG:boto:Path: /pklxkd-56/h?versionId=null 2023-05-31T20:34:52.764 DEBUG:boto:Data: 2023-05-31T20:34:52.764 DEBUG:boto:Headers: {} 2023-05-31T20:34:52.765 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:34:52.766 DEBUG:boto:Port: 8001 2023-05-31T20:34:52.768 DEBUG:boto:Params: {} 2023-05-31T20:34:52.769 DEBUG:boto:Token: None 2023-05-31T20:34:52.770 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:34:52 GMT /pklxkd-56/h?versionId=null 2023-05-31T20:34:52.771 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:pz/r5feLgn4/4XkkzghEqmV0pAg= 2023-05-31T20:34:52.772 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:34:52 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:pz/r5feLgn4/4XkkzghEqmV0pAg=', 'Content-Length': '0'} 2023-05-31T20:34:52.774 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:30:05 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000048bf9e821581a8cb-006477af6c-162e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:34:52 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:34:52.775 DEBUG:boto:path=/pklxkd-56/h 2023-05-31T20:34:52.776 DEBUG:boto:auth_path=/pklxkd-56/h 2023-05-31T20:34:52.777 DEBUG:boto:path=/pklxkd-56/h?versionId=null 2023-05-31T20:34:52.778 DEBUG:boto:auth_path=/pklxkd-56/h?versionId=null 2023-05-31T20:34:52.778 DEBUG:boto:Method: GET 2023-05-31T20:34:52.779 DEBUG:boto:Path: /pklxkd-56/h?versionId=null 2023-05-31T20:34:52.779 DEBUG:boto:Data: 2023-05-31T20:34:52.779 DEBUG:boto:Headers: {} 2023-05-31T20:34:52.780 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:34:52.780 DEBUG:boto:Port: 8000 2023-05-31T20:34:52.781 DEBUG:boto:Params: {} 2023-05-31T20:34:52.781 DEBUG:boto:Token: None 2023-05-31T20:34:52.782 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:34:52 GMT /pklxkd-56/h?versionId=null 2023-05-31T20:34:52.782 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:pz/r5feLgn4/4XkkzghEqmV0pAg= 2023-05-31T20:34:52.782 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:34:52 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:pz/r5feLgn4/4XkkzghEqmV0pAg=', 'Content-Length': '0'} 2023-05-31T20:34:52.785 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:30:05 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c0c07421b3120c62-006477af6c-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:34:52 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:34:52.785 DEBUG:boto:path=/pklxkd-56/h 2023-05-31T20:34:52.786 DEBUG:boto:auth_path=/pklxkd-56/h 2023-05-31T20:34:52.786 DEBUG:boto:path=/pklxkd-56/h?versionId=null 2023-05-31T20:34:52.786 DEBUG:boto:auth_path=/pklxkd-56/h?versionId=null 2023-05-31T20:34:52.787 DEBUG:boto:Method: HEAD 2023-05-31T20:34:52.787 DEBUG:boto:Path: /pklxkd-56/h?versionId=null 2023-05-31T20:34:52.787 DEBUG:boto:Data: 2023-05-31T20:34:52.788 DEBUG:boto:Headers: {} 2023-05-31T20:34:52.788 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:34:52.788 DEBUG:boto:Port: 8001 2023-05-31T20:34:52.789 DEBUG:boto:Params: {} 2023-05-31T20:34:52.789 DEBUG:boto:Token: None 2023-05-31T20:34:52.790 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:34:52 GMT /pklxkd-56/h?versionId=null 2023-05-31T20:34:52.790 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:Wdpga5ljOK38tDi+ULfICWYU8Vg= 2023-05-31T20:34:52.790 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:34:52 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:Wdpga5ljOK38tDi+ULfICWYU8Vg=', 'Content-Length': '0'} 2023-05-31T20:34:52.792 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:30:05 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e821ce9de68453ea-006477af6c-162e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:34:52 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:34:52.793 DEBUG:boto:path=/pklxkd-56/h 2023-05-31T20:34:52.793 DEBUG:boto:auth_path=/pklxkd-56/h 2023-05-31T20:34:52.794 DEBUG:boto:path=/pklxkd-56/h?versionId=null 2023-05-31T20:34:52.794 DEBUG:boto:auth_path=/pklxkd-56/h?versionId=null 2023-05-31T20:34:52.794 DEBUG:boto:Method: HEAD 2023-05-31T20:34:52.795 DEBUG:boto:Path: /pklxkd-56/h?versionId=null 2023-05-31T20:34:52.795 DEBUG:boto:Data: 2023-05-31T20:34:52.795 DEBUG:boto:Headers: {} 2023-05-31T20:34:52.796 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:34:52.796 DEBUG:boto:Port: 8000 2023-05-31T20:34:52.796 DEBUG:boto:Params: {} 2023-05-31T20:34:52.797 DEBUG:boto:Token: None 2023-05-31T20:34:52.797 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:34:52 GMT /pklxkd-56/h?versionId=null 2023-05-31T20:34:52.798 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:Wdpga5ljOK38tDi+ULfICWYU8Vg= 2023-05-31T20:34:52.798 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:34:52 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:Wdpga5ljOK38tDi+ULfICWYU8Vg=', 'Content-Length': '0'} 2023-05-31T20:34:52.800 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:30:05 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d4370249fe15f18e-006477af6c-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:34:52 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:34:52.800 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=h 2023-05-31T20:34:52.801 DEBUG:boto:path=/pklxkd-56/h 2023-05-31T20:34:52.801 DEBUG:boto:auth_path=/pklxkd-56/h 2023-05-31T20:34:52.802 DEBUG:boto:path=/pklxkd-56/h?versionId=null 2023-05-31T20:34:52.802 DEBUG:boto:auth_path=/pklxkd-56/h?versionId=null 2023-05-31T20:34:52.802 DEBUG:boto:Method: GET 2023-05-31T20:34:52.803 DEBUG:boto:Path: /pklxkd-56/h?versionId=null 2023-05-31T20:34:52.803 DEBUG:boto:Data: 2023-05-31T20:34:52.803 DEBUG:boto:Headers: {} 2023-05-31T20:34:52.804 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:34:52.804 DEBUG:boto:Port: 8001 2023-05-31T20:34:52.804 DEBUG:boto:Params: {} 2023-05-31T20:34:52.805 DEBUG:boto:Token: None 2023-05-31T20:34:52.805 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:34:52 GMT /pklxkd-56/h?versionId=null 2023-05-31T20:34:52.806 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:pz/r5feLgn4/4XkkzghEqmV0pAg= 2023-05-31T20:34:52.806 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:34:52 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:pz/r5feLgn4/4XkkzghEqmV0pAg=', 'Content-Length': '0'} 2023-05-31T20:34:52.808 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:30:05 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f1470334152e52a7-006477af6c-162e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:34:52 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:34:52.853 DEBUG:boto:path=/pklxkd-56/h 2023-05-31T20:34:52.853 DEBUG:boto:auth_path=/pklxkd-56/h 2023-05-31T20:34:52.854 DEBUG:boto:path=/pklxkd-56/h?versionId=null 2023-05-31T20:34:52.854 DEBUG:boto:auth_path=/pklxkd-56/h?versionId=null 2023-05-31T20:34:52.854 DEBUG:boto:Method: GET 2023-05-31T20:34:52.855 DEBUG:boto:Path: /pklxkd-56/h?versionId=null 2023-05-31T20:34:52.855 DEBUG:boto:Data: 2023-05-31T20:34:52.855 DEBUG:boto:Headers: {} 2023-05-31T20:34:52.856 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:34:52.856 DEBUG:boto:Port: 8000 2023-05-31T20:34:52.856 DEBUG:boto:Params: {} 2023-05-31T20:34:52.857 DEBUG:boto:Token: None 2023-05-31T20:34:52.858 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:34:52 GMT /pklxkd-56/h?versionId=null 2023-05-31T20:34:52.858 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:pz/r5feLgn4/4XkkzghEqmV0pAg= 2023-05-31T20:34:52.859 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:34:52 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:pz/r5feLgn4/4XkkzghEqmV0pAg=', 'Content-Length': '0'} 2023-05-31T20:34:52.861 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:30:05 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b4a8752a6ca96448-006477af6c-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:34:52 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:34:52.862 DEBUG:boto:path=/pklxkd-56/ 2023-05-31T20:34:52.863 DEBUG:boto:auth_path=/pklxkd-56/ 2023-05-31T20:34:52.863 DEBUG:boto:path=/pklxkd-56/?prefix=h 2023-05-31T20:34:52.864 DEBUG:boto:auth_path=/pklxkd-56/?prefix=h 2023-05-31T20:34:52.864 DEBUG:boto:Method: GET 2023-05-31T20:34:52.864 DEBUG:boto:Path: /pklxkd-56/?prefix=h 2023-05-31T20:34:52.865 DEBUG:boto:Data: 2023-05-31T20:34:52.865 DEBUG:boto:Headers: {} 2023-05-31T20:34:52.865 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:34:52.866 DEBUG:boto:Port: 8001 2023-05-31T20:34:52.866 DEBUG:boto:Params: {} 2023-05-31T20:34:52.867 DEBUG:boto:Token: None 2023-05-31T20:34:52.867 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:34:52 GMT /pklxkd-56/ 2023-05-31T20:34:52.867 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:nL4TOhvnP0P54UQXOfqFYR6uCRE= 2023-05-31T20:34:52.868 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:34:52 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:nL4TOhvnP0P54UQXOfqFYR6uCRE=', 'Content-Length': '0'} 2023-05-31T20:34:52.870 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d3367055693ba25a-006477af6c-162e-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:34:52 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:34:52.871 DEBUG:boto:b'pklxkd-56h1000falseh2023-05-31T20:30:05.906Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:34:52.872 DEBUG:boto:path=/pklxkd-56/ 2023-05-31T20:34:52.872 DEBUG:boto:auth_path=/pklxkd-56/ 2023-05-31T20:34:52.872 DEBUG:boto:path=/pklxkd-56/?prefix=h 2023-05-31T20:34:52.873 DEBUG:boto:auth_path=/pklxkd-56/?prefix=h 2023-05-31T20:34:52.873 DEBUG:boto:Method: GET 2023-05-31T20:34:52.873 DEBUG:boto:Path: /pklxkd-56/?prefix=h 2023-05-31T20:34:52.874 DEBUG:boto:Data: 2023-05-31T20:34:52.874 DEBUG:boto:Headers: {} 2023-05-31T20:34:52.875 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:34:52.875 DEBUG:boto:Port: 8000 2023-05-31T20:34:52.875 DEBUG:boto:Params: {} 2023-05-31T20:34:52.876 DEBUG:boto:Token: None 2023-05-31T20:34:52.876 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:34:52 GMT /pklxkd-56/ 2023-05-31T20:34:52.877 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:nL4TOhvnP0P54UQXOfqFYR6uCRE= 2023-05-31T20:34:52.877 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:34:52 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:nL4TOhvnP0P54UQXOfqFYR6uCRE=', 'Content-Length': '0'} 2023-05-31T20:34:52.880 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000001c4ad6f479a19ed8-006477af6c-1630-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:34:52 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:34:52.881 DEBUG:boto:b'pklxkd-56h1000falseh2023-05-31T20:30:05.906Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:34:52.882 DEBUG:boto:path=/pklxkd-56/h 2023-05-31T20:34:52.882 DEBUG:boto:auth_path=/pklxkd-56/h 2023-05-31T20:34:52.883 DEBUG:boto:Method: HEAD 2023-05-31T20:34:52.883 DEBUG:boto:Path: /pklxkd-56/h 2023-05-31T20:34:52.883 DEBUG:boto:Data: 2023-05-31T20:34:52.884 DEBUG:boto:Headers: {} 2023-05-31T20:34:52.884 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:34:52.884 DEBUG:boto:Port: 8001 2023-05-31T20:34:52.885 DEBUG:boto:Params: {} 2023-05-31T20:34:52.886 DEBUG:boto:Token: None 2023-05-31T20:34:52.886 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:34:52 GMT /pklxkd-56/h 2023-05-31T20:34:52.887 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:kNHmeatC1ElHLP78PlTc/gPIdqQ= 2023-05-31T20:34:52.887 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:34:52 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:kNHmeatC1ElHLP78PlTc/gPIdqQ=', 'Content-Length': '0'} 2023-05-31T20:34:52.889 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:30:05 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ba2110c6392738fd-006477af6c-162e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:34:52 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:34:52.889 DEBUG:boto:path=/pklxkd-56/h 2023-05-31T20:34:52.890 DEBUG:boto:auth_path=/pklxkd-56/h 2023-05-31T20:34:52.890 DEBUG:boto:Method: HEAD 2023-05-31T20:34:52.891 DEBUG:boto:Path: /pklxkd-56/h 2023-05-31T20:34:52.891 DEBUG:boto:Data: 2023-05-31T20:34:52.891 DEBUG:boto:Headers: {} 2023-05-31T20:34:52.892 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:34:52.892 DEBUG:boto:Port: 8000 2023-05-31T20:34:52.892 DEBUG:boto:Params: {} 2023-05-31T20:34:52.893 DEBUG:boto:Token: None 2023-05-31T20:34:52.893 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:34:52 GMT /pklxkd-56/h 2023-05-31T20:34:52.893 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:kNHmeatC1ElHLP78PlTc/gPIdqQ= 2023-05-31T20:34:52.894 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:34:52 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:kNHmeatC1ElHLP78PlTc/gPIdqQ=', 'Content-Length': '0'} 2023-05-31T20:34:52.896 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:30:05 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000087e291b1b80e48bb-006477af6c-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:34:52 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:34:52.896 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=h 2023-05-31T20:34:52.897 DEBUG:boto:path=/pklxkd-56/h 2023-05-31T20:34:52.897 DEBUG:boto:auth_path=/pklxkd-56/h 2023-05-31T20:34:52.898 DEBUG:boto:Method: GET 2023-05-31T20:34:52.899 DEBUG:boto:Path: /pklxkd-56/h 2023-05-31T20:34:52.899 DEBUG:boto:Data: 2023-05-31T20:34:52.900 DEBUG:boto:Headers: {} 2023-05-31T20:34:52.900 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:34:52.901 DEBUG:boto:Port: 8001 2023-05-31T20:34:52.901 DEBUG:boto:Params: {} 2023-05-31T20:34:52.901 DEBUG:boto:Token: None 2023-05-31T20:34:52.902 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:34:52 GMT /pklxkd-56/h 2023-05-31T20:34:52.902 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:PgD2QDExpZOHYecHGLY7nvJR8fw= 2023-05-31T20:34:52.903 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:34:52 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:PgD2QDExpZOHYecHGLY7nvJR8fw=', 'Content-Length': '0'} 2023-05-31T20:34:52.905 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:30:05 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000093e5cb76d78d18a5-006477af6c-162e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:34:52 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:34:52.905 DEBUG:boto:path=/pklxkd-56/h 2023-05-31T20:34:52.906 DEBUG:boto:auth_path=/pklxkd-56/h 2023-05-31T20:34:52.907 DEBUG:boto:Method: GET 2023-05-31T20:34:52.907 DEBUG:boto:Path: /pklxkd-56/h 2023-05-31T20:34:52.908 DEBUG:boto:Data: 2023-05-31T20:34:52.909 DEBUG:boto:Headers: {} 2023-05-31T20:34:52.909 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:34:52.910 DEBUG:boto:Port: 8000 2023-05-31T20:34:52.910 DEBUG:boto:Params: {} 2023-05-31T20:34:52.911 DEBUG:boto:Token: None 2023-05-31T20:34:52.911 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:34:52 GMT /pklxkd-56/h 2023-05-31T20:34:52.912 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:PgD2QDExpZOHYecHGLY7nvJR8fw= 2023-05-31T20:34:52.912 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:34:52 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:PgD2QDExpZOHYecHGLY7nvJR8fw=', 'Content-Length': '0'} 2023-05-31T20:34:52.914 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:30:05 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000dc760d0022ca7645-006477af6c-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:34:52 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:34:52.915 DEBUG:boto:path=/pklxkd-56/ 2023-05-31T20:34:52.915 DEBUG:boto:auth_path=/pklxkd-56/ 2023-05-31T20:34:52.916 DEBUG:boto:path=/pklxkd-56/?prefix=h 2023-05-31T20:34:52.916 DEBUG:boto:auth_path=/pklxkd-56/?prefix=h 2023-05-31T20:34:52.917 DEBUG:boto:Method: GET 2023-05-31T20:34:52.917 DEBUG:boto:Path: /pklxkd-56/?prefix=h 2023-05-31T20:34:52.918 DEBUG:boto:Data: 2023-05-31T20:34:52.919 DEBUG:boto:Headers: {} 2023-05-31T20:34:52.920 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:34:52.920 DEBUG:boto:Port: 8001 2023-05-31T20:34:52.920 DEBUG:boto:Params: {} 2023-05-31T20:34:52.921 DEBUG:boto:Token: None 2023-05-31T20:34:52.922 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:34:52 GMT /pklxkd-56/ 2023-05-31T20:34:52.923 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:nL4TOhvnP0P54UQXOfqFYR6uCRE= 2023-05-31T20:34:52.923 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:34:52 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:nL4TOhvnP0P54UQXOfqFYR6uCRE=', 'Content-Length': '0'} 2023-05-31T20:34:52.927 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000002667845812a5f420-006477af6c-162e-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:34:52 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:34:52.968 DEBUG:boto:b'pklxkd-56h1000falseh2023-05-31T20:30:05.906Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:34:52.970 DEBUG:boto:path=/pklxkd-56/ 2023-05-31T20:34:52.971 DEBUG:boto:auth_path=/pklxkd-56/ 2023-05-31T20:34:52.973 DEBUG:boto:path=/pklxkd-56/?prefix=h 2023-05-31T20:34:52.979 DEBUG:boto:auth_path=/pklxkd-56/?prefix=h 2023-05-31T20:34:52.980 DEBUG:boto:Method: GET 2023-05-31T20:34:52.980 DEBUG:boto:Path: /pklxkd-56/?prefix=h 2023-05-31T20:34:52.981 DEBUG:boto:Data: 2023-05-31T20:34:52.981 DEBUG:boto:Headers: {} 2023-05-31T20:34:52.982 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:34:52.982 DEBUG:boto:Port: 8000 2023-05-31T20:34:52.982 DEBUG:boto:Params: {} 2023-05-31T20:34:52.983 DEBUG:boto:Token: None 2023-05-31T20:34:52.984 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:34:52 GMT /pklxkd-56/ 2023-05-31T20:34:52.984 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:nL4TOhvnP0P54UQXOfqFYR6uCRE= 2023-05-31T20:34:52.985 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:34:52 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:nL4TOhvnP0P54UQXOfqFYR6uCRE=', 'Content-Length': '0'} 2023-05-31T20:34:52.990 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000709180cf7efb2630-006477af6c-1630-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:34:52 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:34:52.991 DEBUG:boto:b'pklxkd-56h1000falseh2023-05-31T20:30:05.906Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:34:52.992 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=pklxkd-56 zones={a2, a1} 2023-05-31T20:34:52.992 INFO:tasks.rgw_multisite_tests:rgw_multi.tests.test_bucket_reshard_incremental ... ok 2023-05-31T20:34:52.994 DEBUG:boto:path=/pklxkd-57/ 2023-05-31T20:34:52.994 DEBUG:boto:auth_path=/pklxkd-57/ 2023-05-31T20:34:52.995 DEBUG:boto:Method: PUT 2023-05-31T20:34:52.995 DEBUG:boto:Path: /pklxkd-57/ 2023-05-31T20:34:52.996 DEBUG:boto:Data: 2023-05-31T20:34:52.996 DEBUG:boto:Headers: {} 2023-05-31T20:34:52.997 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:34:52.997 DEBUG:boto:Port: 8000 2023-05-31T20:34:52.997 DEBUG:boto:Params: {} 2023-05-31T20:34:52.998 DEBUG:boto:Token: None 2023-05-31T20:34:52.998 DEBUG:boto:StringToSign: PUT Wed, 31 May 2023 20:34:52 GMT /pklxkd-57/ 2023-05-31T20:34:52.998 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:tHmqC4qVjipeV18C+uhfGNcwcKU= 2023-05-31T20:34:52.999 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:34:52 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:tHmqC4qVjipeV18C+uhfGNcwcKU=', 'Content-Length': '0'} 2023-05-31T20:34:53.026 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000631146c97a39409b-006477af6c-1630-a1'), ('Content-Length', '0'), ('Date', 'Wed, 31 May 2023 20:34:53 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:34:53.026 DEBUG:rgw_multi.tests:created bucket=pklxkd-57 2023-05-31T20:34:53.027 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-31T20:34:53.027 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-31T20:34:53.028 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-05-31T20:34:53.080 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2023-05-31T20:34:53.080 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2023-05-31T20:34:53.141 INFO:teuthology.orchestra.run.smithi099.stderr:No --period given, using current period=2e6123d2-d695-4350-9a01-61a455bc782b 2023-05-31T20:34:53.155 INFO:teuthology.orchestra.run.smithi099.stdout:[ 2023-05-31T20:34:53.156 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T20:34:53.156 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1685562631.516089_59.1", 2023-05-31T20:34:53.156 INFO:teuthology.orchestra.run.smithi099.stdout: "last_update": "2023-05-31T19:50:31.516089Z" 2023-05-31T20:34:53.156 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:34:53.157 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T20:34:53.157 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "", 2023-05-31T20:34:53.157 INFO:teuthology.orchestra.run.smithi099.stdout: "last_update": "0.000000" 2023-05-31T20:34:53.157 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:34:53.157 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T20:34:53.157 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1685562970.274572_93.1", 2023-05-31T20:34:53.157 INFO:teuthology.orchestra.run.smithi099.stdout: "last_update": "2023-05-31T19:56:10.274572Z" 2023-05-31T20:34:53.158 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:34:53.158 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T20:34:53.158 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1685565293.020126_393.1", 2023-05-31T20:34:53.158 INFO:teuthology.orchestra.run.smithi099.stdout: "last_update": "2023-05-31T20:34:53.020126Z" 2023-05-31T20:34:53.158 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T20:34:53.159 INFO:teuthology.orchestra.run.smithi099.stdout:] 2023-05-31T20:34:53.160 DEBUG:rgw_multi.tests:master meta markers={0: '1_1685562631.516089_59.1', 1: '', 2: '1_1685562970.274572_93.1', 3: '1_1685565293.020126_393.1'} 2023-05-31T20:34:53.160 INFO:rgw_multi.tests:starting meta checkpoint for zone=a2 2023-05-31T20:34:53.161 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-31T20:34:53.161 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-31T20:34:53.161 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-05-31T20:34:53.214 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2023-05-31T20:34:53.214 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2023-05-31T20:34:53.245 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:34:53.241+0000 7f4e4608b780 20 RGW-SYNC:meta: read sync status 2023-05-31T20:34:53.246 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:34:53.242+0000 7f4e4608b780 20 RGW-SYNC:meta: read sync status 2023-05-31T20:34:53.259 INFO:teuthology.orchestra.run.smithi099.stdout:{ 2023-05-31T20:34:53.259 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_status": { 2023-05-31T20:34:53.260 INFO:teuthology.orchestra.run.smithi099.stdout: "info": { 2023-05-31T20:34:53.260 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "sync", 2023-05-31T20:34:53.260 INFO:teuthology.orchestra.run.smithi099.stdout: "num_shards": 4, 2023-05-31T20:34:53.260 INFO:teuthology.orchestra.run.smithi099.stdout: "period": "2e6123d2-d695-4350-9a01-61a455bc782b", 2023-05-31T20:34:53.260 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2023-05-31T20:34:53.260 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:34:53.261 INFO:teuthology.orchestra.run.smithi099.stdout: "markers": [ 2023-05-31T20:34:53.261 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T20:34:53.261 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 0, 2023-05-31T20:34:53.261 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T20:34:53.261 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2023-05-31T20:34:53.261 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1685562631.516089_59.1", 2023-05-31T20:34:53.261 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T20:34:53.262 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 1, 2023-05-31T20:34:53.262 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T20:34:53.262 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2023-05-31T19:50:31.516089Z", 2023-05-31T20:34:53.262 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2023-05-31T20:34:53.262 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T20:34:53.262 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:34:53.262 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T20:34:53.263 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 1, 2023-05-31T20:34:53.263 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T20:34:53.263 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2023-05-31T20:34:53.263 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "", 2023-05-31T20:34:53.263 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T20:34:53.263 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2023-05-31T20:34:53.264 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T20:34:53.264 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "0.000000", 2023-05-31T20:34:53.264 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2023-05-31T20:34:53.264 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T20:34:53.264 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:34:53.264 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T20:34:53.264 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 2, 2023-05-31T20:34:53.265 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T20:34:53.265 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2023-05-31T20:34:53.265 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1685562970.274572_93.1", 2023-05-31T20:34:53.265 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T20:34:53.265 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2023-05-31T20:34:53.265 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T20:34:53.266 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2023-05-31T19:56:10.274572Z", 2023-05-31T20:34:53.266 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2023-05-31T20:34:53.266 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T20:34:53.266 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:34:53.266 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T20:34:53.266 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 3, 2023-05-31T20:34:53.266 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T20:34:53.267 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2023-05-31T20:34:53.267 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1685565293.008045_390.1", 2023-05-31T20:34:53.267 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T20:34:53.267 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2023-05-31T20:34:53.267 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T20:34:53.267 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2023-05-31T20:34:53.008045Z", 2023-05-31T20:34:53.267 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2023-05-31T20:34:53.268 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T20:34:53.268 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T20:34:53.268 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2023-05-31T20:34:53.268 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:34:53.268 INFO:teuthology.orchestra.run.smithi099.stdout: "full_sync": { 2023-05-31T20:34:53.268 INFO:teuthology.orchestra.run.smithi099.stdout: "total": 1, 2023-05-31T20:34:53.268 INFO:teuthology.orchestra.run.smithi099.stdout: "complete": 1 2023-05-31T20:34:53.269 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:34:53.269 INFO:teuthology.orchestra.run.smithi099.stdout: "current_time": "2023-05-31T20:34:53Z" 2023-05-31T20:34:53.269 INFO:teuthology.orchestra.run.smithi099.stdout:} 2023-05-31T20:34:53.270 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "2e6123d2-d695-4350-9a01-61a455bc782b", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1685562631.516089_59.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-05-31T19:50:31.516089Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1685562970.274572_93.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-05-31T19:56:10.274572Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1685565293.008045_390.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-05-31T20:34:53.008045Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-05-31T20:34:53Z" } 2023-05-31T20:34:53.270 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1685562631.516089_59.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-05-31T19:50:31.516089Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1685562970.274572_93.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-05-31T19:56:10.274572Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1685565293.008045_390.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-05-31T20:34:53.008045Z', 'realm_epoch': 2}}] 2023-05-31T20:34:53.271 DEBUG:rgw_multi.tests:log_status={0: '1_1685562631.516089_59.1', 1: '', 2: '1_1685562970.274572_93.1', 3: '1_1685565293.020126_393.1'} 2023-05-31T20:34:53.271 DEBUG:rgw_multi.tests:sync_status={0: '1_1685562631.516089_59.1', 1: '', 2: '1_1685562970.274572_93.1', 3: '1_1685565293.008045_390.1'} 2023-05-31T20:34:53.271 WARNING:rgw_multi.tests:zone a2 behind master: shard=3 master=1_1685565293.020126_393.1 target=1_1685565293.008045_390.1 2023-05-31T20:34:58.273 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-31T20:34:58.274 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-31T20:34:58.274 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-05-31T20:34:58.332 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2023-05-31T20:34:58.332 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2023-05-31T20:34:58.389 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:34:58.384+0000 7fc654d40780 20 RGW-SYNC:meta: read sync status 2023-05-31T20:34:58.390 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:34:58.385+0000 7fc654d40780 20 RGW-SYNC:meta: read sync status 2023-05-31T20:34:58.412 INFO:teuthology.orchestra.run.smithi099.stdout:{ 2023-05-31T20:34:58.412 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_status": { 2023-05-31T20:34:58.412 INFO:teuthology.orchestra.run.smithi099.stdout: "info": { 2023-05-31T20:34:58.412 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "sync", 2023-05-31T20:34:58.412 INFO:teuthology.orchestra.run.smithi099.stdout: "num_shards": 4, 2023-05-31T20:34:58.413 INFO:teuthology.orchestra.run.smithi099.stdout: "period": "2e6123d2-d695-4350-9a01-61a455bc782b", 2023-05-31T20:34:58.413 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2023-05-31T20:34:58.413 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:34:58.413 INFO:teuthology.orchestra.run.smithi099.stdout: "markers": [ 2023-05-31T20:34:58.413 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T20:34:58.413 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 0, 2023-05-31T20:34:58.414 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T20:34:58.414 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2023-05-31T20:34:58.414 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1685562631.516089_59.1", 2023-05-31T20:34:58.414 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T20:34:58.414 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 1, 2023-05-31T20:34:58.414 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T20:34:58.414 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2023-05-31T19:50:31.516089Z", 2023-05-31T20:34:58.415 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2023-05-31T20:34:58.415 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T20:34:58.415 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:34:58.416 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T20:34:58.416 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 1, 2023-05-31T20:34:58.416 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T20:34:58.416 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2023-05-31T20:34:58.416 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "", 2023-05-31T20:34:58.416 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T20:34:58.417 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2023-05-31T20:34:58.417 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T20:34:58.417 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "0.000000", 2023-05-31T20:34:58.417 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2023-05-31T20:34:58.417 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T20:34:58.417 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:34:58.417 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T20:34:58.418 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 2, 2023-05-31T20:34:58.418 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T20:34:58.418 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2023-05-31T20:34:58.418 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1685562970.274572_93.1", 2023-05-31T20:34:58.418 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T20:34:58.418 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2023-05-31T20:34:58.419 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T20:34:58.419 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2023-05-31T19:56:10.274572Z", 2023-05-31T20:34:58.419 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2023-05-31T20:34:58.419 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T20:34:58.419 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:34:58.419 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T20:34:58.419 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 3, 2023-05-31T20:34:58.420 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T20:34:58.420 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2023-05-31T20:34:58.420 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1685565293.020126_393.1", 2023-05-31T20:34:58.420 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T20:34:58.420 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2023-05-31T20:34:58.420 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T20:34:58.420 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2023-05-31T20:34:53.020126Z", 2023-05-31T20:34:58.421 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2023-05-31T20:34:58.421 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T20:34:58.421 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T20:34:58.421 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2023-05-31T20:34:58.421 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:34:58.421 INFO:teuthology.orchestra.run.smithi099.stdout: "full_sync": { 2023-05-31T20:34:58.421 INFO:teuthology.orchestra.run.smithi099.stdout: "total": 1, 2023-05-31T20:34:58.422 INFO:teuthology.orchestra.run.smithi099.stdout: "complete": 1 2023-05-31T20:34:58.422 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:34:58.422 INFO:teuthology.orchestra.run.smithi099.stdout: "current_time": "2023-05-31T20:34:58Z" 2023-05-31T20:34:58.422 INFO:teuthology.orchestra.run.smithi099.stdout:} 2023-05-31T20:34:58.423 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "2e6123d2-d695-4350-9a01-61a455bc782b", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1685562631.516089_59.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-05-31T19:50:31.516089Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1685562970.274572_93.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-05-31T19:56:10.274572Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1685565293.020126_393.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-05-31T20:34:53.020126Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-05-31T20:34:58Z" } 2023-05-31T20:34:58.424 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1685562631.516089_59.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-05-31T19:50:31.516089Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1685562970.274572_93.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-05-31T19:56:10.274572Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1685565293.020126_393.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-05-31T20:34:53.020126Z', 'realm_epoch': 2}}] 2023-05-31T20:34:58.424 DEBUG:rgw_multi.tests:log_status={0: '1_1685562631.516089_59.1', 1: '', 2: '1_1685562970.274572_93.1', 3: '1_1685565293.020126_393.1'} 2023-05-31T20:34:58.425 DEBUG:rgw_multi.tests:sync_status={0: '1_1685562631.516089_59.1', 1: '', 2: '1_1685562970.274572_93.1', 3: '1_1685565293.020126_393.1'} 2023-05-31T20:34:58.425 INFO:rgw_multi.tests:finish meta checkpoint for zone=a2 2023-05-31T20:34:58.426 DEBUG:tasks.rgw.c1.client.1:waiting for process to exit 2023-05-31T20:34:58.426 INFO:teuthology.orchestra.run:waiting for 300 2023-05-31T20:34:58.427 INFO:tasks.rgw.c1.client.1.smithi099.stdout:2023-05-31T20:34:58.422+0000 7f3b99d24700 -1 received signal: Terminated from /usr/bin/python3 /bin/daemon-helper term radosgw --rgw-frontends beast port=8001 -n client.1 --cluster c1 -k /etc/ceph/c1.client.1.keyring --log-file /var/log/ceph/rgw.c1.client.1.log --rgw_ops_log_socket_path /home/ubuntu/cephtest/rgw.opslog.c1.client.1.sock --foreground --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm (PID: 113915) UID: 0 2023-05-31T20:34:58.427 INFO:tasks.rgw.c1.client.1.smithi099.stdout:2023-05-31T20:34:58.422+0000 7f3b9df8d7c0 -1 shutting down 2023-05-31T20:35:03.410 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.1 is failed for ~0s 2023-05-31T20:35:04.528 INFO:tasks.rgw.c1.client.1:Stopped 2023-05-31T20:35:04.529 DEBUG:boto:path=/pklxkd-57/ 2023-05-31T20:35:04.529 DEBUG:boto:auth_path=/pklxkd-57/ 2023-05-31T20:35:04.529 DEBUG:boto:Method: HEAD 2023-05-31T20:35:04.530 DEBUG:boto:Path: /pklxkd-57/ 2023-05-31T20:35:04.530 DEBUG:boto:Data: 2023-05-31T20:35:04.530 DEBUG:boto:Headers: {} 2023-05-31T20:35:04.531 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:35:04.531 DEBUG:boto:Port: 8000 2023-05-31T20:35:04.531 DEBUG:boto:Params: {} 2023-05-31T20:35:04.532 DEBUG:boto:Token: None 2023-05-31T20:35:04.533 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:35:04 GMT /pklxkd-57/ 2023-05-31T20:35:04.533 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:JT0Zhmm/NsIRQoR4LoLPgsXHjjk= 2023-05-31T20:35:04.533 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:35:04 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:JT0Zhmm/NsIRQoR4LoLPgsXHjjk=', 'Content-Length': '0'} 2023-05-31T20:35:04.536 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000001929850905146b0f-006477af78-1630-a1'), ('Content-Length', '0'), ('Date', 'Wed, 31 May 2023 20:35:04 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:35:04.537 DEBUG:boto:path=/pklxkd-57/a 2023-05-31T20:35:04.538 DEBUG:boto:auth_path=/pklxkd-57/a 2023-05-31T20:35:04.538 DEBUG:boto:Method: PUT 2023-05-31T20:35:04.539 DEBUG:boto:Path: /pklxkd-57/a 2023-05-31T20:35:04.539 DEBUG:boto:Data: 2023-05-31T20:35:04.539 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-05-31T20:35:04.540 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:35:04.540 DEBUG:boto:Port: 8000 2023-05-31T20:35:04.540 DEBUG:boto:Params: {} 2023-05-31T20:35:04.541 DEBUG:boto:Token: None 2023-05-31T20:35:04.542 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Wed, 31 May 2023 20:35:04 GMT /pklxkd-57/a 2023-05-31T20:35:04.542 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:v4XVC46jkxcdA2xxJIUba60KG9c= 2023-05-31T20:35:04.542 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Wed, 31 May 2023 20:35:04 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:v4XVC46jkxcdA2xxJIUba60KG9c='} 2023-05-31T20:35:04.584 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000020b413a51a1bc189-006477af78-1630-a1'), ('Date', 'Wed, 31 May 2023 20:35:04 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:35:04.585 DEBUG:boto:path=/pklxkd-57/ 2023-05-31T20:35:04.585 DEBUG:boto:auth_path=/pklxkd-57/ 2023-05-31T20:35:04.586 DEBUG:boto:Method: HEAD 2023-05-31T20:35:04.586 DEBUG:boto:Path: /pklxkd-57/ 2023-05-31T20:35:04.587 DEBUG:boto:Data: 2023-05-31T20:35:04.587 DEBUG:boto:Headers: {} 2023-05-31T20:35:04.588 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:35:04.588 DEBUG:boto:Port: 8000 2023-05-31T20:35:04.588 DEBUG:boto:Params: {} 2023-05-31T20:35:04.589 DEBUG:boto:Token: None 2023-05-31T20:35:04.589 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:35:04 GMT /pklxkd-57/ 2023-05-31T20:35:04.590 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:JT0Zhmm/NsIRQoR4LoLPgsXHjjk= 2023-05-31T20:35:04.590 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:35:04 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:JT0Zhmm/NsIRQoR4LoLPgsXHjjk=', 'Content-Length': '0'} 2023-05-31T20:35:04.592 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '3'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000074d073a26f6f5489-006477af78-1630-a1'), ('Content-Length', '0'), ('Date', 'Wed, 31 May 2023 20:35:04 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:35:04.593 DEBUG:boto:path=/pklxkd-57/b 2023-05-31T20:35:04.593 DEBUG:boto:auth_path=/pklxkd-57/b 2023-05-31T20:35:04.594 DEBUG:boto:Method: PUT 2023-05-31T20:35:04.595 DEBUG:boto:Path: /pklxkd-57/b 2023-05-31T20:35:04.596 DEBUG:boto:Data: 2023-05-31T20:35:04.596 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-05-31T20:35:04.597 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:35:04.597 DEBUG:boto:Port: 8000 2023-05-31T20:35:04.598 DEBUG:boto:Params: {} 2023-05-31T20:35:04.599 DEBUG:boto:Token: None 2023-05-31T20:35:04.600 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Wed, 31 May 2023 20:35:04 GMT /pklxkd-57/b 2023-05-31T20:35:04.601 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:U/dL0FztgdeRTgWzvA+REbWr+OY= 2023-05-31T20:35:04.602 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Wed, 31 May 2023 20:35:04 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:U/dL0FztgdeRTgWzvA+REbWr+OY='} 2023-05-31T20:35:04.609 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000aae568144a950e43-006477af78-1630-a1'), ('Date', 'Wed, 31 May 2023 20:35:04 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:35:04.609 DEBUG:boto:path=/pklxkd-57/ 2023-05-31T20:35:04.610 DEBUG:boto:auth_path=/pklxkd-57/ 2023-05-31T20:35:04.610 DEBUG:boto:Method: HEAD 2023-05-31T20:35:04.610 DEBUG:boto:Path: /pklxkd-57/ 2023-05-31T20:35:04.611 DEBUG:boto:Data: 2023-05-31T20:35:04.611 DEBUG:boto:Headers: {} 2023-05-31T20:35:04.611 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:35:04.612 DEBUG:boto:Port: 8000 2023-05-31T20:35:04.612 DEBUG:boto:Params: {} 2023-05-31T20:35:04.613 DEBUG:boto:Token: None 2023-05-31T20:35:04.613 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:35:04 GMT /pklxkd-57/ 2023-05-31T20:35:04.613 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:JT0Zhmm/NsIRQoR4LoLPgsXHjjk= 2023-05-31T20:35:04.614 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:35:04 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:JT0Zhmm/NsIRQoR4LoLPgsXHjjk=', 'Content-Length': '0'} 2023-05-31T20:35:04.617 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '2'), ('X-RGW-Bytes-Used', '6'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000072c36dd82db50665-006477af78-1630-a1'), ('Content-Length', '0'), ('Date', 'Wed, 31 May 2023 20:35:04 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:35:04.617 DEBUG:boto:path=/pklxkd-57/c 2023-05-31T20:35:04.618 DEBUG:boto:auth_path=/pklxkd-57/c 2023-05-31T20:35:04.618 DEBUG:boto:Method: PUT 2023-05-31T20:35:04.618 DEBUG:boto:Path: /pklxkd-57/c 2023-05-31T20:35:04.619 DEBUG:boto:Data: 2023-05-31T20:35:04.619 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-05-31T20:35:04.620 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:35:04.620 DEBUG:boto:Port: 8000 2023-05-31T20:35:04.620 DEBUG:boto:Params: {} 2023-05-31T20:35:04.621 DEBUG:boto:Token: None 2023-05-31T20:35:04.621 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Wed, 31 May 2023 20:35:04 GMT /pklxkd-57/c 2023-05-31T20:35:04.622 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:9UXYyqy6hxlOp4AJsijNc6vMlfw= 2023-05-31T20:35:04.622 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Wed, 31 May 2023 20:35:04 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:9UXYyqy6hxlOp4AJsijNc6vMlfw='} 2023-05-31T20:35:04.668 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000d1172494486bf763-006477af78-1630-a1'), ('Date', 'Wed, 31 May 2023 20:35:04 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:35:04.669 DEBUG:boto:path=/pklxkd-57/ 2023-05-31T20:35:04.669 DEBUG:boto:auth_path=/pklxkd-57/ 2023-05-31T20:35:04.670 DEBUG:boto:Method: HEAD 2023-05-31T20:35:04.670 DEBUG:boto:Path: /pklxkd-57/ 2023-05-31T20:35:04.670 DEBUG:boto:Data: 2023-05-31T20:35:04.671 DEBUG:boto:Headers: {} 2023-05-31T20:35:04.671 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:35:04.671 DEBUG:boto:Port: 8000 2023-05-31T20:35:04.672 DEBUG:boto:Params: {} 2023-05-31T20:35:04.672 DEBUG:boto:Token: None 2023-05-31T20:35:04.673 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:35:04 GMT /pklxkd-57/ 2023-05-31T20:35:04.673 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:JT0Zhmm/NsIRQoR4LoLPgsXHjjk= 2023-05-31T20:35:04.673 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:35:04 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:JT0Zhmm/NsIRQoR4LoLPgsXHjjk=', 'Content-Length': '0'} 2023-05-31T20:35:04.676 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '3'), ('X-RGW-Bytes-Used', '9'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000d171b9bdbcef523f-006477af78-1630-a1'), ('Content-Length', '0'), ('Date', 'Wed, 31 May 2023 20:35:04 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:35:04.677 DEBUG:boto:path=/pklxkd-57/d 2023-05-31T20:35:04.677 DEBUG:boto:auth_path=/pklxkd-57/d 2023-05-31T20:35:04.678 DEBUG:boto:Method: PUT 2023-05-31T20:35:04.678 DEBUG:boto:Path: /pklxkd-57/d 2023-05-31T20:35:04.679 DEBUG:boto:Data: 2023-05-31T20:35:04.679 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-05-31T20:35:04.679 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:35:04.680 DEBUG:boto:Port: 8000 2023-05-31T20:35:04.680 DEBUG:boto:Params: {} 2023-05-31T20:35:04.680 DEBUG:boto:Token: None 2023-05-31T20:35:04.681 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Wed, 31 May 2023 20:35:04 GMT /pklxkd-57/d 2023-05-31T20:35:04.681 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:ax2lyWzwSwDBg6LMTmAqAtqfldo= 2023-05-31T20:35:04.682 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Wed, 31 May 2023 20:35:04 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:ax2lyWzwSwDBg6LMTmAqAtqfldo='} 2023-05-31T20:35:04.688 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000e0e0e724974e36da-006477af78-1630-a1'), ('Date', 'Wed, 31 May 2023 20:35:04 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:35:04.689 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'reshard', '--bucket', 'pklxkd-57', '--num-shards', '3', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-05-31T20:35:04.690 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'reshard', '--bucket', 'pklxkd-57', '--num-shards', '3', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-05-31T20:35:04.691 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket reshard --bucket pklxkd-57 --num-shards 3 --yes-i-really-mean-it --cluster c1 --debug-rgw 0 --debug-ms 0 2023-05-31T20:35:04.748 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2023-05-31T20:35:04.748 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2023-05-31T20:35:04.828 INFO:teuthology.orchestra.run.smithi099.stdout:tenant: 2023-05-31T20:35:04.828 INFO:teuthology.orchestra.run.smithi099.stdout:bucket name: pklxkd-57 2023-05-31T20:35:04.831 INFO:teuthology.orchestra.run.smithi099.stdout:total entries: 4 2023-05-31T20:35:04.894 DEBUG:boto:path=/pklxkd-57/ 2023-05-31T20:35:04.895 DEBUG:boto:auth_path=/pklxkd-57/ 2023-05-31T20:35:04.896 DEBUG:boto:Method: HEAD 2023-05-31T20:35:04.896 DEBUG:boto:Path: /pklxkd-57/ 2023-05-31T20:35:04.896 DEBUG:boto:Data: 2023-05-31T20:35:04.897 DEBUG:boto:Headers: {} 2023-05-31T20:35:04.897 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:35:04.898 DEBUG:boto:Port: 8000 2023-05-31T20:35:04.898 DEBUG:boto:Params: {} 2023-05-31T20:35:04.900 DEBUG:boto:Token: None 2023-05-31T20:35:04.900 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:35:04 GMT /pklxkd-57/ 2023-05-31T20:35:04.901 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:JT0Zhmm/NsIRQoR4LoLPgsXHjjk= 2023-05-31T20:35:04.901 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:35:04 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:JT0Zhmm/NsIRQoR4LoLPgsXHjjk=', 'Content-Length': '0'} 2023-05-31T20:35:04.903 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '4'), ('X-RGW-Bytes-Used', '12'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000eb2aa6dacdb9ce03-006477af78-1630-a1'), ('Content-Length', '0'), ('Date', 'Wed, 31 May 2023 20:35:04 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:35:04.904 DEBUG:boto:path=/pklxkd-57/e 2023-05-31T20:35:04.905 DEBUG:boto:auth_path=/pklxkd-57/e 2023-05-31T20:35:04.905 DEBUG:boto:Method: PUT 2023-05-31T20:35:04.905 DEBUG:boto:Path: /pklxkd-57/e 2023-05-31T20:35:04.906 DEBUG:boto:Data: 2023-05-31T20:35:04.906 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-05-31T20:35:04.906 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:35:04.907 DEBUG:boto:Port: 8000 2023-05-31T20:35:04.907 DEBUG:boto:Params: {} 2023-05-31T20:35:04.907 DEBUG:boto:Token: None 2023-05-31T20:35:04.908 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Wed, 31 May 2023 20:35:04 GMT /pklxkd-57/e 2023-05-31T20:35:04.908 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:50ubIy4pErrri4QjC+r5pbyHB+Y= 2023-05-31T20:35:04.909 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Wed, 31 May 2023 20:35:04 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:50ubIy4pErrri4QjC+r5pbyHB+Y='} 2023-05-31T20:35:04.915 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000007bc29dbadcfa103e-006477af78-1630-a1'), ('Date', 'Wed, 31 May 2023 20:35:04 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:35:04.916 DEBUG:boto:path=/pklxkd-57/ 2023-05-31T20:35:04.916 DEBUG:boto:auth_path=/pklxkd-57/ 2023-05-31T20:35:04.916 DEBUG:boto:Method: HEAD 2023-05-31T20:35:04.917 DEBUG:boto:Path: /pklxkd-57/ 2023-05-31T20:35:04.917 DEBUG:boto:Data: 2023-05-31T20:35:04.917 DEBUG:boto:Headers: {} 2023-05-31T20:35:04.918 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:35:04.918 DEBUG:boto:Port: 8000 2023-05-31T20:35:04.918 DEBUG:boto:Params: {} 2023-05-31T20:35:04.919 DEBUG:boto:Token: None 2023-05-31T20:35:04.919 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:35:04 GMT /pklxkd-57/ 2023-05-31T20:35:04.919 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:JT0Zhmm/NsIRQoR4LoLPgsXHjjk= 2023-05-31T20:35:04.920 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:35:04 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:JT0Zhmm/NsIRQoR4LoLPgsXHjjk=', 'Content-Length': '0'} 2023-05-31T20:35:04.922 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '5'), ('X-RGW-Bytes-Used', '15'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000d382d64d977f0458-006477af78-1630-a1'), ('Content-Length', '0'), ('Date', 'Wed, 31 May 2023 20:35:04 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:35:04.922 DEBUG:boto:path=/pklxkd-57/f 2023-05-31T20:35:04.923 DEBUG:boto:auth_path=/pklxkd-57/f 2023-05-31T20:35:04.923 DEBUG:boto:Method: PUT 2023-05-31T20:35:04.923 DEBUG:boto:Path: /pklxkd-57/f 2023-05-31T20:35:04.924 DEBUG:boto:Data: 2023-05-31T20:35:04.924 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-05-31T20:35:04.924 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:35:04.925 DEBUG:boto:Port: 8000 2023-05-31T20:35:04.925 DEBUG:boto:Params: {} 2023-05-31T20:35:04.925 DEBUG:boto:Token: None 2023-05-31T20:35:04.926 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Wed, 31 May 2023 20:35:04 GMT /pklxkd-57/f 2023-05-31T20:35:04.926 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:RgvP5JOYiMfdm0cHUNSsJgves6U= 2023-05-31T20:35:04.927 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Wed, 31 May 2023 20:35:04 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:RgvP5JOYiMfdm0cHUNSsJgves6U='} 2023-05-31T20:35:04.932 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000909f512bd4995a9b-006477af78-1630-a1'), ('Date', 'Wed, 31 May 2023 20:35:04 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:35:04.932 DEBUG:boto:path=/pklxkd-57/ 2023-05-31T20:35:04.933 DEBUG:boto:auth_path=/pklxkd-57/ 2023-05-31T20:35:04.933 DEBUG:boto:Method: HEAD 2023-05-31T20:35:04.934 DEBUG:boto:Path: /pklxkd-57/ 2023-05-31T20:35:04.934 DEBUG:boto:Data: 2023-05-31T20:35:04.934 DEBUG:boto:Headers: {} 2023-05-31T20:35:04.934 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:35:04.935 DEBUG:boto:Port: 8000 2023-05-31T20:35:04.935 DEBUG:boto:Params: {} 2023-05-31T20:35:04.936 DEBUG:boto:Token: None 2023-05-31T20:35:04.936 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:35:04 GMT /pklxkd-57/ 2023-05-31T20:35:04.936 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:JT0Zhmm/NsIRQoR4LoLPgsXHjjk= 2023-05-31T20:35:04.937 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:35:04 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:JT0Zhmm/NsIRQoR4LoLPgsXHjjk=', 'Content-Length': '0'} 2023-05-31T20:35:04.938 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '6'), ('X-RGW-Bytes-Used', '18'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000c8c6a5ba303d7432-006477af78-1630-a1'), ('Content-Length', '0'), ('Date', 'Wed, 31 May 2023 20:35:04 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:35:04.939 DEBUG:boto:path=/pklxkd-57/g 2023-05-31T20:35:04.939 DEBUG:boto:auth_path=/pklxkd-57/g 2023-05-31T20:35:04.940 DEBUG:boto:Method: PUT 2023-05-31T20:35:04.940 DEBUG:boto:Path: /pklxkd-57/g 2023-05-31T20:35:04.940 DEBUG:boto:Data: 2023-05-31T20:35:04.941 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-05-31T20:35:04.941 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:35:04.941 DEBUG:boto:Port: 8000 2023-05-31T20:35:04.942 DEBUG:boto:Params: {} 2023-05-31T20:35:04.942 DEBUG:boto:Token: None 2023-05-31T20:35:04.943 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Wed, 31 May 2023 20:35:04 GMT /pklxkd-57/g 2023-05-31T20:35:04.943 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:iAO+bWJbpeSt457oGt3x4M8Y8dU= 2023-05-31T20:35:04.944 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Wed, 31 May 2023 20:35:04 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:iAO+bWJbpeSt457oGt3x4M8Y8dU='} 2023-05-31T20:35:04.989 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000509e52d9522a98a9-006477af78-1630-a1'), ('Date', 'Wed, 31 May 2023 20:35:04 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:35:04.990 DEBUG:boto:path=/pklxkd-57/ 2023-05-31T20:35:04.991 DEBUG:boto:auth_path=/pklxkd-57/ 2023-05-31T20:35:04.992 DEBUG:boto:Method: HEAD 2023-05-31T20:35:04.993 DEBUG:boto:Path: /pklxkd-57/ 2023-05-31T20:35:04.993 DEBUG:boto:Data: 2023-05-31T20:35:04.994 DEBUG:boto:Headers: {} 2023-05-31T20:35:04.994 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:35:04.995 DEBUG:boto:Port: 8000 2023-05-31T20:35:04.995 DEBUG:boto:Params: {} 2023-05-31T20:35:04.996 DEBUG:boto:Token: None 2023-05-31T20:35:04.997 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:35:04 GMT /pklxkd-57/ 2023-05-31T20:35:04.997 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:JT0Zhmm/NsIRQoR4LoLPgsXHjjk= 2023-05-31T20:35:04.998 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:35:04 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:JT0Zhmm/NsIRQoR4LoLPgsXHjjk=', 'Content-Length': '0'} 2023-05-31T20:35:05.000 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '7'), ('X-RGW-Bytes-Used', '21'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000dac0163c86ead3db-006477af78-1630-a1'), ('Content-Length', '0'), ('Date', 'Wed, 31 May 2023 20:35:04 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:35:05.001 DEBUG:boto:path=/pklxkd-57/h 2023-05-31T20:35:05.001 DEBUG:boto:auth_path=/pklxkd-57/h 2023-05-31T20:35:05.002 DEBUG:boto:Method: PUT 2023-05-31T20:35:05.003 DEBUG:boto:Path: /pklxkd-57/h 2023-05-31T20:35:05.003 DEBUG:boto:Data: 2023-05-31T20:35:05.003 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-05-31T20:35:05.004 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:35:05.004 DEBUG:boto:Port: 8000 2023-05-31T20:35:05.004 DEBUG:boto:Params: {} 2023-05-31T20:35:05.005 DEBUG:boto:Token: None 2023-05-31T20:35:05.005 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Wed, 31 May 2023 20:35:05 GMT /pklxkd-57/h 2023-05-31T20:35:05.006 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:XwnpysSEbdN/NuHEt0T7I1mbKrw= 2023-05-31T20:35:05.006 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Wed, 31 May 2023 20:35:05 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:XwnpysSEbdN/NuHEt0T7I1mbKrw='} 2023-05-31T20:35:05.010 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000a88c179ee897eb09-006477af79-1630-a1'), ('Date', 'Wed, 31 May 2023 20:35:05 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:35:05.011 INFO:tasks.rgw.c1.client.1:Restarting daemon 2023-05-31T20:35:05.011 DEBUG:teuthology.orchestra.run.smithi099:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term radosgw --rgw-frontends 'beast port=8001' -n client.1 --cluster c1 -k /etc/ceph/c1.client.1.keyring --log-file /var/log/ceph/rgw.c1.client.1.log --rgw_ops_log_socket_path /home/ubuntu/cephtest/rgw.opslog.c1.client.1.sock --foreground --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm | sudo tee /var/log/ceph/rgw.c1.client.1.stdout 2>&1 2023-05-31T20:35:05.014 INFO:tasks.rgw.c1.client.1:Started 2023-05-31T20:35:05.014 DEBUG:teuthology.orchestra.run.smithi099:> curl http://smithi099.front.sepia.ceph.com:8001 2023-05-31T20:35:05.036 DEBUG:teuthology.orchestra.run:got remote process result: 7 2023-05-31T20:35:05.037 INFO:teuthology.orchestra.run.smithi099.stderr: % Total % Received % Xferd Average Speed Time Time Time Current 2023-05-31T20:35:05.038 INFO:teuthology.orchestra.run.smithi099.stderr: Dload Upload Total Spent Left Speed 2023-05-31T20:35:05.038 INFO:teuthology.orchestra.run.smithi099.stderr: 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0curl: (7) Failed to connect to smithi099.front.sepia.ceph.com port 8001: Connection refused 2023-05-31T20:35:06.040 DEBUG:teuthology.orchestra.run.smithi099:> curl http://smithi099.front.sepia.ceph.com:8001 2023-05-31T20:35:06.063 INFO:teuthology.orchestra.run.smithi099.stderr: % Total % Received % Xferd Average Speed Time Time Time Current 2023-05-31T20:35:06.063 INFO:teuthology.orchestra.run.smithi099.stderr: Dload Upload Total Spent Left Speed 2023-05-31T20:35:06.072 INFO:teuthology.orchestra.run.smithi099.stderr: 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 100 214 0 214 0 0 21400 0 --:--:-- --:--:-- --:--:-- 21400 2023-05-31T20:35:06.073 INFO:teuthology.orchestra.run.smithi099.stdout:anonymous 2023-05-31T20:35:06.074 DEBUG:rgw_multi.tests:bucket checkpoint: source=a1 target=a2 bucket=pklxkd-57 2023-05-31T20:35:06.074 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'pklxkd-57', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-05-31T20:35:06.075 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'pklxkd-57', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-05-31T20:35:06.075 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket pklxkd-57 --source-zone a1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2023-05-31T20:35:06.171 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2023-05-31T20:35:06.171 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2023-05-31T20:35:06.217 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:35:06.211+0000 7f42d8769780 1 waiting to reach incremental sync.. 2023-05-31T20:35:09.116 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.1 has been restored 2023-05-31T20:35:11.217 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:35:11.212+0000 7f42d8769780 1 waiting to reach incremental sync.. 2023-05-31T20:35:16.218 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:35:16.213+0000 7f42d8769780 1 waiting to reach incremental sync.. 2023-05-31T20:35:21.218 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:35:21.214+0000 7f42d8769780 1 waiting to reach incremental sync.. 2023-05-31T20:35:26.219 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:35:26.215+0000 7f42d8769780 1 waiting to reach incremental sync.. 2023-05-31T20:35:31.219 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:35:31.215+0000 7f42d8769780 1 waiting to reach incremental sync.. 2023-05-31T20:35:36.220 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:35:36.216+0000 7f42d8769780 1 waiting to reach incremental sync.. 2023-05-31T20:35:41.220 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:35:41.215+0000 7f42d8769780 1 waiting to reach incremental sync.. 2023-05-31T20:35:46.221 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:35:46.216+0000 7f42d8769780 1 waiting to reach incremental sync.. 2023-05-31T20:35:51.221 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:35:51.216+0000 7f42d8769780 1 waiting to reach incremental sync.. 2023-05-31T20:35:56.222 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:35:56.217+0000 7f42d8769780 1 waiting to reach incremental sync.. 2023-05-31T20:36:01.223 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:36:01.218+0000 7f42d8769780 1 waiting to reach incremental sync.. 2023-05-31T20:36:06.223 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:36:06.219+0000 7f42d8769780 1 waiting to reach incremental sync.. 2023-05-31T20:36:11.224 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:36:11.219+0000 7f42d8769780 1 waiting to reach incremental sync.. 2023-05-31T20:36:16.224 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:36:16.220+0000 7f42d8769780 1 waiting to reach incremental sync.. 2023-05-31T20:36:21.225 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:36:21.220+0000 7f42d8769780 1 waiting to reach incremental sync.. 2023-05-31T20:36:26.226 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:36:26.220+0000 7f42d8769780 1 waiting to reach incremental sync.. 2023-05-31T20:36:31.226 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:36:31.221+0000 7f42d8769780 1 waiting to reach incremental sync.. 2023-05-31T20:36:36.227 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:36:36.222+0000 7f42d8769780 1 waiting to reach incremental sync.. 2023-05-31T20:36:41.227 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:36:41.223+0000 7f42d8769780 1 waiting to reach incremental sync.. 2023-05-31T20:36:46.228 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:36:46.224+0000 7f42d8769780 1 waiting to reach incremental sync.. 2023-05-31T20:36:51.229 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:36:51.224+0000 7f42d8769780 1 waiting to reach incremental sync.. 2023-05-31T20:36:56.229 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:36:56.225+0000 7f42d8769780 1 waiting to reach incremental sync.. 2023-05-31T20:37:01.230 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:37:01.226+0000 7f42d8769780 1 waiting to reach incremental sync.. 2023-05-31T20:37:06.230 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:37:06.225+0000 7f42d8769780 1 waiting to reach incremental sync.. 2023-05-31T20:37:11.232 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:37:11.227+0000 7f42d8769780 1 bucket sync caught up with source: 2023-05-31T20:37:11.232 INFO:teuthology.orchestra.run.smithi099.stderr: local status: [00000000003.33.6, 00000000002.40.6, 00000000002.38.6] 2023-05-31T20:37:11.232 INFO:teuthology.orchestra.run.smithi099.stderr: remote markers: [00000000003.33.6, 00000000002.40.6, 00000000002.38.6] 2023-05-31T20:37:11.233 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:37:11.227+0000 7f42d8769780 0 bucket checkpoint complete 2023-05-31T20:37:11.253 DEBUG:rgw_multi.tests:bucket checkpoint: source=a2 target=a1 bucket=pklxkd-57 2023-05-31T20:37:11.254 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'pklxkd-57', '--source-zone', 'a2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-05-31T20:37:11.254 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'pklxkd-57', '--source-zone', 'a2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-05-31T20:37:11.254 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket pklxkd-57 --source-zone a2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2023-05-31T20:37:11.315 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2023-05-31T20:37:11.316 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2023-05-31T20:37:11.365 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:37:11.360+0000 7f65555ef780 1 waiting for incremental sync to catch up: 2023-05-31T20:37:11.365 INFO:teuthology.orchestra.run.smithi099.stderr: local status: [, 00000000002.1979.6, , , , 00000000001.1362.6, , , 00000000002.1363.6, , 00000000001.35.6] 2023-05-31T20:37:11.365 INFO:teuthology.orchestra.run.smithi099.stderr: remote markers: [, 00000000002.1979.6, , , 00000000001.2287.6, 00000000001.1362.6, , , 00000000002.1363.6, , 00000000002.38.6] 2023-05-31T20:37:16.366 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:37:16.361+0000 7f65555ef780 1 waiting for incremental sync to catch up: 2023-05-31T20:37:16.366 INFO:teuthology.orchestra.run.smithi099.stderr: local status: [, 00000000002.1979.6, , , , 00000000001.1362.6, , , 00000000002.1363.6, , 00000000001.35.6] 2023-05-31T20:37:16.367 INFO:teuthology.orchestra.run.smithi099.stderr: remote markers: [, 00000000002.1979.6, , , 00000000001.2287.6, 00000000001.1362.6, , , 00000000002.1363.6, , 00000000002.38.6] 2023-05-31T20:37:21.368 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:37:21.362+0000 7f65555ef780 1 waiting for incremental sync to catch up: 2023-05-31T20:37:21.368 INFO:teuthology.orchestra.run.smithi099.stderr: local status: [, 00000000002.1979.6, , , , 00000000001.1362.6, , , 00000000002.1363.6, , 00000000001.35.6] 2023-05-31T20:37:21.368 INFO:teuthology.orchestra.run.smithi099.stderr: remote markers: [, 00000000002.1979.6, , , 00000000001.2287.6, 00000000001.1362.6, , , 00000000002.1363.6, , 00000000002.38.6] 2023-05-31T20:37:26.369 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:37:26.364+0000 7f65555ef780 1 bucket sync caught up with source: 2023-05-31T20:37:26.369 INFO:teuthology.orchestra.run.smithi099.stderr: local status: [, 00000000002.1979.6, , , 00000000001.2287.6, 00000000001.1362.6, , , 00000000002.1363.6, , 00000000002.38.6] 2023-05-31T20:37:26.370 INFO:teuthology.orchestra.run.smithi099.stderr: remote markers: [, 00000000002.1979.6, , , 00000000001.2287.6, 00000000001.1362.6, , , 00000000002.1363.6, , 00000000002.38.6] 2023-05-31T20:37:26.370 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:37:26.364+0000 7f65555ef780 0 bucket checkpoint complete 2023-05-31T20:37:26.395 INFO:tasks.rgw_multi.zone_rados:comparing bucket=pklxkd-57 zones={a2, a1} 2023-05-31T20:37:26.396 DEBUG:boto:path=/pklxkd-57/ 2023-05-31T20:37:26.396 DEBUG:boto:auth_path=/pklxkd-57/ 2023-05-31T20:37:26.397 DEBUG:boto:Method: HEAD 2023-05-31T20:37:26.397 DEBUG:boto:Path: /pklxkd-57/ 2023-05-31T20:37:26.397 DEBUG:boto:Data: 2023-05-31T20:37:26.398 DEBUG:boto:Headers: {} 2023-05-31T20:37:26.398 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:37:26.398 DEBUG:boto:Port: 8001 2023-05-31T20:37:26.399 DEBUG:boto:Params: {} 2023-05-31T20:37:26.399 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8001} 2023-05-31T20:37:26.400 DEBUG:boto:Token: None 2023-05-31T20:37:26.400 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:37:26 GMT /pklxkd-57/ 2023-05-31T20:37:26.401 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:chnErdCFTiIUnWknC/JUmr+Kmcg= 2023-05-31T20:37:26.401 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:37:26 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:chnErdCFTiIUnWknC/JUmr+Kmcg=', 'Content-Length': '0'} 2023-05-31T20:37:26.406 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '8'), ('X-RGW-Bytes-Used', '24'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000c7d6409a670119a1-006477b006-1b4f-a2'), ('Content-Length', '0'), ('Date', 'Wed, 31 May 2023 20:37:26 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:37:26.406 DEBUG:boto:path=/pklxkd-57/ 2023-05-31T20:37:26.407 DEBUG:boto:auth_path=/pklxkd-57/ 2023-05-31T20:37:26.407 DEBUG:boto:Method: HEAD 2023-05-31T20:37:26.408 DEBUG:boto:Path: /pklxkd-57/ 2023-05-31T20:37:26.408 DEBUG:boto:Data: 2023-05-31T20:37:26.409 DEBUG:boto:Headers: {} 2023-05-31T20:37:26.410 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:37:26.410 DEBUG:boto:Port: 8000 2023-05-31T20:37:26.410 DEBUG:boto:Params: {} 2023-05-31T20:37:26.411 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2023-05-31T20:37:26.411 DEBUG:boto:Token: None 2023-05-31T20:37:26.412 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:37:26 GMT /pklxkd-57/ 2023-05-31T20:37:26.412 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:chnErdCFTiIUnWknC/JUmr+Kmcg= 2023-05-31T20:37:26.412 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:37:26 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:chnErdCFTiIUnWknC/JUmr+Kmcg=', 'Content-Length': '0'} 2023-05-31T20:37:26.416 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '8'), ('X-RGW-Bytes-Used', '24'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000ec0c17b56ba47f39-006477b006-1630-a1'), ('Content-Length', '0'), ('Date', 'Wed, 31 May 2023 20:37:26 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:37:26.416 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2023-05-31T20:37:26.417 DEBUG:boto:path=/pklxkd-57/ 2023-05-31T20:37:26.417 DEBUG:boto:auth_path=/pklxkd-57/ 2023-05-31T20:37:26.418 DEBUG:boto:path=/pklxkd-57/?versions&max-keys=999 2023-05-31T20:37:26.418 DEBUG:boto:auth_path=/pklxkd-57/?versions&max-keys=999 2023-05-31T20:37:26.418 DEBUG:boto:Method: GET 2023-05-31T20:37:26.419 DEBUG:boto:Path: /pklxkd-57/?versions&max-keys=999 2023-05-31T20:37:26.419 DEBUG:boto:Data: 2023-05-31T20:37:26.419 DEBUG:boto:Headers: {} 2023-05-31T20:37:26.420 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:37:26.420 DEBUG:boto:Port: 8001 2023-05-31T20:37:26.420 DEBUG:boto:Params: {} 2023-05-31T20:37:26.421 DEBUG:boto:Token: None 2023-05-31T20:37:26.421 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:37:26 GMT /pklxkd-57/?versions 2023-05-31T20:37:26.422 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:RCEFuj6wj56Q2BsM2N8tkSqu8IM= 2023-05-31T20:37:26.422 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:37:26 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:RCEFuj6wj56Q2BsM2N8tkSqu8IM=', 'Content-Length': '0'} 2023-05-31T20:37:26.425 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000003749f52b53d6dd15-006477b006-1b4f-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:37:26 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:37:26.425 DEBUG:boto:b'pklxkd-57999falseanulltrue2023-05-31T20:35:04.541Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalbnulltrue2023-05-31T20:35:04.600Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalcnulltrue2023-05-31T20:35:04.620Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormaldnulltrue2023-05-31T20:35:04.680Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalenulltrue2023-05-31T20:35:04.906Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalfnulltrue2023-05-31T20:35:04.925Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalgnulltrue2023-05-31T20:35:04.942Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalhnulltrue2023-05-31T20:35:05.004Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:37:26.426 DEBUG:tasks.rgw_multi.zone_rados:o=a 2023-05-31T20:37:26.427 DEBUG:tasks.rgw_multi.zone_rados:o=b 2023-05-31T20:37:26.427 DEBUG:tasks.rgw_multi.zone_rados:o=c 2023-05-31T20:37:26.428 DEBUG:tasks.rgw_multi.zone_rados:o=d 2023-05-31T20:37:26.428 DEBUG:tasks.rgw_multi.zone_rados:o=e 2023-05-31T20:37:26.428 DEBUG:tasks.rgw_multi.zone_rados:o=f 2023-05-31T20:37:26.428 DEBUG:tasks.rgw_multi.zone_rados:o=g 2023-05-31T20:37:26.429 DEBUG:tasks.rgw_multi.zone_rados:o=h 2023-05-31T20:37:26.429 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2023-05-31T20:37:26.429 DEBUG:boto:path=/pklxkd-57/ 2023-05-31T20:37:26.430 DEBUG:boto:auth_path=/pklxkd-57/ 2023-05-31T20:37:26.430 DEBUG:boto:path=/pklxkd-57/?versions&max-keys=999 2023-05-31T20:37:26.431 DEBUG:boto:auth_path=/pklxkd-57/?versions&max-keys=999 2023-05-31T20:37:26.431 DEBUG:boto:Method: GET 2023-05-31T20:37:26.431 DEBUG:boto:Path: /pklxkd-57/?versions&max-keys=999 2023-05-31T20:37:26.432 DEBUG:boto:Data: 2023-05-31T20:37:26.432 DEBUG:boto:Headers: {} 2023-05-31T20:37:26.432 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:37:26.433 DEBUG:boto:Port: 8000 2023-05-31T20:37:26.433 DEBUG:boto:Params: {} 2023-05-31T20:37:26.433 DEBUG:boto:Token: None 2023-05-31T20:37:26.434 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:37:26 GMT /pklxkd-57/?versions 2023-05-31T20:37:26.434 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:RCEFuj6wj56Q2BsM2N8tkSqu8IM= 2023-05-31T20:37:26.435 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:37:26 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:RCEFuj6wj56Q2BsM2N8tkSqu8IM=', 'Content-Length': '0'} 2023-05-31T20:37:26.436 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000363c34c1fc674101-006477b006-1630-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:37:26 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:37:26.437 DEBUG:boto:b'pklxkd-57999falseanulltrue2023-05-31T20:35:04.541Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalbnulltrue2023-05-31T20:35:04.600Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalcnulltrue2023-05-31T20:35:04.620Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormaldnulltrue2023-05-31T20:35:04.680Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalenulltrue2023-05-31T20:35:04.906Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalfnulltrue2023-05-31T20:35:04.925Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalgnulltrue2023-05-31T20:35:04.942Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalhnulltrue2023-05-31T20:35:05.004Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:37:26.438 DEBUG:tasks.rgw_multi.zone_rados:o=a 2023-05-31T20:37:26.439 DEBUG:tasks.rgw_multi.zone_rados:o=b 2023-05-31T20:37:26.439 DEBUG:tasks.rgw_multi.zone_rados:o=c 2023-05-31T20:37:26.439 DEBUG:tasks.rgw_multi.zone_rados:o=d 2023-05-31T20:37:26.440 DEBUG:tasks.rgw_multi.zone_rados:o=e 2023-05-31T20:37:26.440 DEBUG:tasks.rgw_multi.zone_rados:o=f 2023-05-31T20:37:26.440 DEBUG:tasks.rgw_multi.zone_rados:o=g 2023-05-31T20:37:26.440 DEBUG:tasks.rgw_multi.zone_rados:o=h 2023-05-31T20:37:26.441 DEBUG:boto:path=/pklxkd-57/ 2023-05-31T20:37:26.441 DEBUG:boto:auth_path=/pklxkd-57/ 2023-05-31T20:37:26.442 DEBUG:boto:path=/pklxkd-57/?versions&max-keys=999 2023-05-31T20:37:26.442 DEBUG:boto:auth_path=/pklxkd-57/?versions&max-keys=999 2023-05-31T20:37:26.442 DEBUG:boto:Method: GET 2023-05-31T20:37:26.443 DEBUG:boto:Path: /pklxkd-57/?versions&max-keys=999 2023-05-31T20:37:26.443 DEBUG:boto:Data: 2023-05-31T20:37:26.443 DEBUG:boto:Headers: {} 2023-05-31T20:37:26.444 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:37:26.444 DEBUG:boto:Port: 8001 2023-05-31T20:37:26.444 DEBUG:boto:Params: {} 2023-05-31T20:37:26.445 DEBUG:boto:Token: None 2023-05-31T20:37:26.445 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:37:26 GMT /pklxkd-57/?versions 2023-05-31T20:37:26.446 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:RCEFuj6wj56Q2BsM2N8tkSqu8IM= 2023-05-31T20:37:26.446 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:37:26 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:RCEFuj6wj56Q2BsM2N8tkSqu8IM=', 'Content-Length': '0'} 2023-05-31T20:37:26.448 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000085c2ccb35911197c-006477b006-1b4f-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:37:26 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:37:26.449 DEBUG:boto:b'pklxkd-57999falseanulltrue2023-05-31T20:35:04.541Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalbnulltrue2023-05-31T20:35:04.600Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalcnulltrue2023-05-31T20:35:04.620Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormaldnulltrue2023-05-31T20:35:04.680Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalenulltrue2023-05-31T20:35:04.906Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalfnulltrue2023-05-31T20:35:04.925Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalgnulltrue2023-05-31T20:35:04.942Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalhnulltrue2023-05-31T20:35:05.004Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:37:26.450 DEBUG:boto:path=/pklxkd-57/ 2023-05-31T20:37:26.450 DEBUG:boto:auth_path=/pklxkd-57/ 2023-05-31T20:37:26.451 DEBUG:boto:path=/pklxkd-57/?versions&max-keys=999 2023-05-31T20:37:26.451 DEBUG:boto:auth_path=/pklxkd-57/?versions&max-keys=999 2023-05-31T20:37:26.451 DEBUG:boto:Method: GET 2023-05-31T20:37:26.452 DEBUG:boto:Path: /pklxkd-57/?versions&max-keys=999 2023-05-31T20:37:26.452 DEBUG:boto:Data: 2023-05-31T20:37:26.453 DEBUG:boto:Headers: {} 2023-05-31T20:37:26.453 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:37:26.453 DEBUG:boto:Port: 8000 2023-05-31T20:37:26.454 DEBUG:boto:Params: {} 2023-05-31T20:37:26.454 DEBUG:boto:Token: None 2023-05-31T20:37:26.455 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:37:26 GMT /pklxkd-57/?versions 2023-05-31T20:37:26.455 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:RCEFuj6wj56Q2BsM2N8tkSqu8IM= 2023-05-31T20:37:26.456 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:37:26 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:RCEFuj6wj56Q2BsM2N8tkSqu8IM=', 'Content-Length': '0'} 2023-05-31T20:37:26.457 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000047735ef730f32848-006477b006-1630-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:37:26 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:37:26.458 DEBUG:boto:b'pklxkd-57999falseanulltrue2023-05-31T20:35:04.541Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalbnulltrue2023-05-31T20:35:04.600Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalcnulltrue2023-05-31T20:35:04.620Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormaldnulltrue2023-05-31T20:35:04.680Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalenulltrue2023-05-31T20:35:04.906Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalfnulltrue2023-05-31T20:35:04.925Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalgnulltrue2023-05-31T20:35:04.942Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalhnulltrue2023-05-31T20:35:05.004Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:37:26.459 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=a 2023-05-31T20:37:26.459 DEBUG:boto:path=/pklxkd-57/a 2023-05-31T20:37:26.460 DEBUG:boto:auth_path=/pklxkd-57/a 2023-05-31T20:37:26.460 DEBUG:boto:path=/pklxkd-57/a?versionId=null 2023-05-31T20:37:26.461 DEBUG:boto:auth_path=/pklxkd-57/a?versionId=null 2023-05-31T20:37:26.461 DEBUG:boto:Method: GET 2023-05-31T20:37:26.461 DEBUG:boto:Path: /pklxkd-57/a?versionId=null 2023-05-31T20:37:26.462 DEBUG:boto:Data: 2023-05-31T20:37:26.462 DEBUG:boto:Headers: {} 2023-05-31T20:37:26.463 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:37:26.463 DEBUG:boto:Port: 8001 2023-05-31T20:37:26.463 DEBUG:boto:Params: {} 2023-05-31T20:37:26.464 DEBUG:boto:Token: None 2023-05-31T20:37:26.464 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:37:26 GMT /pklxkd-57/a?versionId=null 2023-05-31T20:37:26.465 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:np1bIVnTf4ALRB7OjvtjkVXLQoQ= 2023-05-31T20:37:26.465 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:37:26 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:np1bIVnTf4ALRB7OjvtjkVXLQoQ=', 'Content-Length': '0'} 2023-05-31T20:37:26.467 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:35:04 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000af91fcaef48358dd-006477b006-1b4f-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:37:26 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:37:26.508 DEBUG:boto:path=/pklxkd-57/a 2023-05-31T20:37:26.509 DEBUG:boto:auth_path=/pklxkd-57/a 2023-05-31T20:37:26.509 DEBUG:boto:path=/pklxkd-57/a?versionId=null 2023-05-31T20:37:26.510 DEBUG:boto:auth_path=/pklxkd-57/a?versionId=null 2023-05-31T20:37:26.510 DEBUG:boto:Method: GET 2023-05-31T20:37:26.510 DEBUG:boto:Path: /pklxkd-57/a?versionId=null 2023-05-31T20:37:26.511 DEBUG:boto:Data: 2023-05-31T20:37:26.511 DEBUG:boto:Headers: {} 2023-05-31T20:37:26.512 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:37:26.512 DEBUG:boto:Port: 8000 2023-05-31T20:37:26.512 DEBUG:boto:Params: {} 2023-05-31T20:37:26.513 DEBUG:boto:Token: None 2023-05-31T20:37:26.513 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:37:26 GMT /pklxkd-57/a?versionId=null 2023-05-31T20:37:26.514 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:np1bIVnTf4ALRB7OjvtjkVXLQoQ= 2023-05-31T20:37:26.514 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:37:26 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:np1bIVnTf4ALRB7OjvtjkVXLQoQ=', 'Content-Length': '0'} 2023-05-31T20:37:26.516 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:35:04 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000bf6f15c1ae4161a1-006477b006-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:37:26 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:37:26.517 DEBUG:boto:path=/pklxkd-57/a 2023-05-31T20:37:26.517 DEBUG:boto:auth_path=/pklxkd-57/a 2023-05-31T20:37:26.518 DEBUG:boto:path=/pklxkd-57/a?versionId=null 2023-05-31T20:37:26.518 DEBUG:boto:auth_path=/pklxkd-57/a?versionId=null 2023-05-31T20:37:26.518 DEBUG:boto:Method: HEAD 2023-05-31T20:37:26.519 DEBUG:boto:Path: /pklxkd-57/a?versionId=null 2023-05-31T20:37:26.519 DEBUG:boto:Data: 2023-05-31T20:37:26.519 DEBUG:boto:Headers: {} 2023-05-31T20:37:26.520 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:37:26.520 DEBUG:boto:Port: 8001 2023-05-31T20:37:26.521 DEBUG:boto:Params: {} 2023-05-31T20:37:26.521 DEBUG:boto:Token: None 2023-05-31T20:37:26.521 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:37:26 GMT /pklxkd-57/a?versionId=null 2023-05-31T20:37:26.522 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:Z0y/2TYxV1Nom6qpDjMDHh9134o= 2023-05-31T20:37:26.522 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:37:26 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:Z0y/2TYxV1Nom6qpDjMDHh9134o=', 'Content-Length': '0'} 2023-05-31T20:37:26.524 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:35:04 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000045a949cac3b34215-006477b006-1b4f-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:37:26 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:37:26.525 DEBUG:boto:path=/pklxkd-57/a 2023-05-31T20:37:26.525 DEBUG:boto:auth_path=/pklxkd-57/a 2023-05-31T20:37:26.525 DEBUG:boto:path=/pklxkd-57/a?versionId=null 2023-05-31T20:37:26.526 DEBUG:boto:auth_path=/pklxkd-57/a?versionId=null 2023-05-31T20:37:26.526 DEBUG:boto:Method: HEAD 2023-05-31T20:37:26.526 DEBUG:boto:Path: /pklxkd-57/a?versionId=null 2023-05-31T20:37:26.527 DEBUG:boto:Data: 2023-05-31T20:37:26.527 DEBUG:boto:Headers: {} 2023-05-31T20:37:26.527 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:37:26.528 DEBUG:boto:Port: 8000 2023-05-31T20:37:26.528 DEBUG:boto:Params: {} 2023-05-31T20:37:26.529 DEBUG:boto:Token: None 2023-05-31T20:37:26.529 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:37:26 GMT /pklxkd-57/a?versionId=null 2023-05-31T20:37:26.529 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:Z0y/2TYxV1Nom6qpDjMDHh9134o= 2023-05-31T20:37:26.530 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:37:26 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:Z0y/2TYxV1Nom6qpDjMDHh9134o=', 'Content-Length': '0'} 2023-05-31T20:37:26.532 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:35:04 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000549437c64b64103b-006477b006-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:37:26 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:37:26.533 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=a 2023-05-31T20:37:26.533 DEBUG:boto:path=/pklxkd-57/a 2023-05-31T20:37:26.534 DEBUG:boto:auth_path=/pklxkd-57/a 2023-05-31T20:37:26.534 DEBUG:boto:path=/pklxkd-57/a?versionId=null 2023-05-31T20:37:26.535 DEBUG:boto:auth_path=/pklxkd-57/a?versionId=null 2023-05-31T20:37:26.535 DEBUG:boto:Method: GET 2023-05-31T20:37:26.536 DEBUG:boto:Path: /pklxkd-57/a?versionId=null 2023-05-31T20:37:26.536 DEBUG:boto:Data: 2023-05-31T20:37:26.537 DEBUG:boto:Headers: {} 2023-05-31T20:37:26.537 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:37:26.538 DEBUG:boto:Port: 8001 2023-05-31T20:37:26.538 DEBUG:boto:Params: {} 2023-05-31T20:37:26.539 DEBUG:boto:Token: None 2023-05-31T20:37:26.539 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:37:26 GMT /pklxkd-57/a?versionId=null 2023-05-31T20:37:26.539 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:np1bIVnTf4ALRB7OjvtjkVXLQoQ= 2023-05-31T20:37:26.540 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:37:26 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:np1bIVnTf4ALRB7OjvtjkVXLQoQ=', 'Content-Length': '0'} 2023-05-31T20:37:26.542 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:35:04 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004a8e6b8c6e354f2f-006477b006-1b4f-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:37:26 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:37:26.543 DEBUG:boto:path=/pklxkd-57/a 2023-05-31T20:37:26.543 DEBUG:boto:auth_path=/pklxkd-57/a 2023-05-31T20:37:26.544 DEBUG:boto:path=/pklxkd-57/a?versionId=null 2023-05-31T20:37:26.544 DEBUG:boto:auth_path=/pklxkd-57/a?versionId=null 2023-05-31T20:37:26.544 DEBUG:boto:Method: GET 2023-05-31T20:37:26.545 DEBUG:boto:Path: /pklxkd-57/a?versionId=null 2023-05-31T20:37:26.545 DEBUG:boto:Data: 2023-05-31T20:37:26.545 DEBUG:boto:Headers: {} 2023-05-31T20:37:26.546 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:37:26.546 DEBUG:boto:Port: 8000 2023-05-31T20:37:26.547 DEBUG:boto:Params: {} 2023-05-31T20:37:26.547 DEBUG:boto:Token: None 2023-05-31T20:37:26.547 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:37:26 GMT /pklxkd-57/a?versionId=null 2023-05-31T20:37:26.548 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:np1bIVnTf4ALRB7OjvtjkVXLQoQ= 2023-05-31T20:37:26.548 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:37:26 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:np1bIVnTf4ALRB7OjvtjkVXLQoQ=', 'Content-Length': '0'} 2023-05-31T20:37:26.550 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:35:04 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000aed29dfdcfbd65ef-006477b006-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:37:26 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:37:26.551 DEBUG:boto:path=/pklxkd-57/ 2023-05-31T20:37:26.551 DEBUG:boto:auth_path=/pklxkd-57/ 2023-05-31T20:37:26.552 DEBUG:boto:path=/pklxkd-57/?prefix=a 2023-05-31T20:37:26.552 DEBUG:boto:auth_path=/pklxkd-57/?prefix=a 2023-05-31T20:37:26.552 DEBUG:boto:Method: GET 2023-05-31T20:37:26.553 DEBUG:boto:Path: /pklxkd-57/?prefix=a 2023-05-31T20:37:26.553 DEBUG:boto:Data: 2023-05-31T20:37:26.554 DEBUG:boto:Headers: {} 2023-05-31T20:37:26.554 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:37:26.554 DEBUG:boto:Port: 8001 2023-05-31T20:37:26.555 DEBUG:boto:Params: {} 2023-05-31T20:37:26.555 DEBUG:boto:Token: None 2023-05-31T20:37:26.556 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:37:26 GMT /pklxkd-57/ 2023-05-31T20:37:26.556 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:TOQ9jxSj7BFBWhrNcW+gciGRxKs= 2023-05-31T20:37:26.557 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:37:26 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:TOQ9jxSj7BFBWhrNcW+gciGRxKs=', 'Content-Length': '0'} 2023-05-31T20:37:26.559 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000001e2af130b1d9813-006477b006-1b4f-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:37:26 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:37:26.560 DEBUG:boto:b'pklxkd-57a1000falsea2023-05-31T20:35:04.541Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:37:26.560 DEBUG:boto:path=/pklxkd-57/ 2023-05-31T20:37:26.561 DEBUG:boto:auth_path=/pklxkd-57/ 2023-05-31T20:37:26.561 DEBUG:boto:path=/pklxkd-57/?prefix=a 2023-05-31T20:37:26.561 DEBUG:boto:auth_path=/pklxkd-57/?prefix=a 2023-05-31T20:37:26.562 DEBUG:boto:Method: GET 2023-05-31T20:37:26.562 DEBUG:boto:Path: /pklxkd-57/?prefix=a 2023-05-31T20:37:26.563 DEBUG:boto:Data: 2023-05-31T20:37:26.563 DEBUG:boto:Headers: {} 2023-05-31T20:37:26.563 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:37:26.564 DEBUG:boto:Port: 8000 2023-05-31T20:37:26.564 DEBUG:boto:Params: {} 2023-05-31T20:37:26.564 DEBUG:boto:Token: None 2023-05-31T20:37:26.565 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:37:26 GMT /pklxkd-57/ 2023-05-31T20:37:26.565 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:TOQ9jxSj7BFBWhrNcW+gciGRxKs= 2023-05-31T20:37:26.566 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:37:26 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:TOQ9jxSj7BFBWhrNcW+gciGRxKs=', 'Content-Length': '0'} 2023-05-31T20:37:26.568 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ba2c074dd879d2b3-006477b006-1630-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:37:26 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:37:26.569 DEBUG:boto:b'pklxkd-57a1000falsea2023-05-31T20:35:04.541Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:37:26.569 DEBUG:boto:path=/pklxkd-57/a 2023-05-31T20:37:26.570 DEBUG:boto:auth_path=/pklxkd-57/a 2023-05-31T20:37:26.570 DEBUG:boto:Method: HEAD 2023-05-31T20:37:26.570 DEBUG:boto:Path: /pklxkd-57/a 2023-05-31T20:37:26.571 DEBUG:boto:Data: 2023-05-31T20:37:26.571 DEBUG:boto:Headers: {} 2023-05-31T20:37:26.571 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:37:26.572 DEBUG:boto:Port: 8001 2023-05-31T20:37:26.572 DEBUG:boto:Params: {} 2023-05-31T20:37:26.573 DEBUG:boto:Token: None 2023-05-31T20:37:26.573 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:37:26 GMT /pklxkd-57/a 2023-05-31T20:37:26.573 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:9XBE0FkzhNwdeXwljXfLQUamHYA= 2023-05-31T20:37:26.574 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:37:26 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:9XBE0FkzhNwdeXwljXfLQUamHYA=', 'Content-Length': '0'} 2023-05-31T20:37:26.575 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:35:04 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f2759ff846e0322b-006477b006-1b4f-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:37:26 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:37:26.576 DEBUG:boto:path=/pklxkd-57/a 2023-05-31T20:37:26.577 DEBUG:boto:auth_path=/pklxkd-57/a 2023-05-31T20:37:26.577 DEBUG:boto:Method: HEAD 2023-05-31T20:37:26.577 DEBUG:boto:Path: /pklxkd-57/a 2023-05-31T20:37:26.578 DEBUG:boto:Data: 2023-05-31T20:37:26.578 DEBUG:boto:Headers: {} 2023-05-31T20:37:26.578 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:37:26.579 DEBUG:boto:Port: 8000 2023-05-31T20:37:26.579 DEBUG:boto:Params: {} 2023-05-31T20:37:26.579 DEBUG:boto:Token: None 2023-05-31T20:37:26.580 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:37:26 GMT /pklxkd-57/a 2023-05-31T20:37:26.580 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:9XBE0FkzhNwdeXwljXfLQUamHYA= 2023-05-31T20:37:26.581 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:37:26 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:9XBE0FkzhNwdeXwljXfLQUamHYA=', 'Content-Length': '0'} 2023-05-31T20:37:26.582 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:35:04 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008b1da072dfad947b-006477b006-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:37:26 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:37:26.583 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=a 2023-05-31T20:37:26.583 DEBUG:boto:path=/pklxkd-57/a 2023-05-31T20:37:26.584 DEBUG:boto:auth_path=/pklxkd-57/a 2023-05-31T20:37:26.584 DEBUG:boto:Method: GET 2023-05-31T20:37:26.585 DEBUG:boto:Path: /pklxkd-57/a 2023-05-31T20:37:26.585 DEBUG:boto:Data: 2023-05-31T20:37:26.585 DEBUG:boto:Headers: {} 2023-05-31T20:37:26.586 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:37:26.586 DEBUG:boto:Port: 8001 2023-05-31T20:37:26.586 DEBUG:boto:Params: {} 2023-05-31T20:37:26.587 DEBUG:boto:Token: None 2023-05-31T20:37:26.587 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:37:26 GMT /pklxkd-57/a 2023-05-31T20:37:26.588 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:gOGW1QGxWPot+skJdjHNhF+W10g= 2023-05-31T20:37:26.588 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:37:26 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:gOGW1QGxWPot+skJdjHNhF+W10g=', 'Content-Length': '0'} 2023-05-31T20:37:26.590 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:35:04 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000058f57620526828bd-006477b006-1b4f-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:37:26 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:37:26.591 DEBUG:boto:path=/pklxkd-57/a 2023-05-31T20:37:26.591 DEBUG:boto:auth_path=/pklxkd-57/a 2023-05-31T20:37:26.592 DEBUG:boto:Method: GET 2023-05-31T20:37:26.592 DEBUG:boto:Path: /pklxkd-57/a 2023-05-31T20:37:26.592 DEBUG:boto:Data: 2023-05-31T20:37:26.593 DEBUG:boto:Headers: {} 2023-05-31T20:37:26.593 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:37:26.593 DEBUG:boto:Port: 8000 2023-05-31T20:37:26.594 DEBUG:boto:Params: {} 2023-05-31T20:37:26.594 DEBUG:boto:Token: None 2023-05-31T20:37:26.595 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:37:26 GMT /pklxkd-57/a 2023-05-31T20:37:26.595 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:gOGW1QGxWPot+skJdjHNhF+W10g= 2023-05-31T20:37:26.596 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:37:26 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:gOGW1QGxWPot+skJdjHNhF+W10g=', 'Content-Length': '0'} 2023-05-31T20:37:26.598 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:35:04 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000caedf85feee64c64-006477b006-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:37:26 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:37:26.598 DEBUG:boto:path=/pklxkd-57/ 2023-05-31T20:37:26.599 DEBUG:boto:auth_path=/pklxkd-57/ 2023-05-31T20:37:26.599 DEBUG:boto:path=/pklxkd-57/?prefix=a 2023-05-31T20:37:26.599 DEBUG:boto:auth_path=/pklxkd-57/?prefix=a 2023-05-31T20:37:26.600 DEBUG:boto:Method: GET 2023-05-31T20:37:26.600 DEBUG:boto:Path: /pklxkd-57/?prefix=a 2023-05-31T20:37:26.601 DEBUG:boto:Data: 2023-05-31T20:37:26.601 DEBUG:boto:Headers: {} 2023-05-31T20:37:26.601 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:37:26.602 DEBUG:boto:Port: 8001 2023-05-31T20:37:26.602 DEBUG:boto:Params: {} 2023-05-31T20:37:26.602 DEBUG:boto:Token: None 2023-05-31T20:37:26.603 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:37:26 GMT /pklxkd-57/ 2023-05-31T20:37:26.603 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:TOQ9jxSj7BFBWhrNcW+gciGRxKs= 2023-05-31T20:37:26.604 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:37:26 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:TOQ9jxSj7BFBWhrNcW+gciGRxKs=', 'Content-Length': '0'} 2023-05-31T20:37:26.606 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000084646883a683d3da-006477b006-1b4f-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:37:26 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:37:26.607 DEBUG:boto:b'pklxkd-57a1000falsea2023-05-31T20:35:04.541Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:37:26.608 DEBUG:boto:path=/pklxkd-57/ 2023-05-31T20:37:26.608 DEBUG:boto:auth_path=/pklxkd-57/ 2023-05-31T20:37:26.609 DEBUG:boto:path=/pklxkd-57/?prefix=a 2023-05-31T20:37:26.610 DEBUG:boto:auth_path=/pklxkd-57/?prefix=a 2023-05-31T20:37:26.610 DEBUG:boto:Method: GET 2023-05-31T20:37:26.610 DEBUG:boto:Path: /pklxkd-57/?prefix=a 2023-05-31T20:37:26.611 DEBUG:boto:Data: 2023-05-31T20:37:26.611 DEBUG:boto:Headers: {} 2023-05-31T20:37:26.611 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:37:26.612 DEBUG:boto:Port: 8000 2023-05-31T20:37:26.612 DEBUG:boto:Params: {} 2023-05-31T20:37:26.613 DEBUG:boto:Token: None 2023-05-31T20:37:26.613 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:37:26 GMT /pklxkd-57/ 2023-05-31T20:37:26.614 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:TOQ9jxSj7BFBWhrNcW+gciGRxKs= 2023-05-31T20:37:26.614 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:37:26 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:TOQ9jxSj7BFBWhrNcW+gciGRxKs=', 'Content-Length': '0'} 2023-05-31T20:37:26.616 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e65d3e85cdf62184-006477b006-1630-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:37:26 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:37:26.617 DEBUG:boto:b'pklxkd-57a1000falsea2023-05-31T20:35:04.541Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:37:26.617 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=b 2023-05-31T20:37:26.618 DEBUG:boto:path=/pklxkd-57/b 2023-05-31T20:37:26.618 DEBUG:boto:auth_path=/pklxkd-57/b 2023-05-31T20:37:26.618 DEBUG:boto:path=/pklxkd-57/b?versionId=null 2023-05-31T20:37:26.619 DEBUG:boto:auth_path=/pklxkd-57/b?versionId=null 2023-05-31T20:37:26.619 DEBUG:boto:Method: GET 2023-05-31T20:37:26.620 DEBUG:boto:Path: /pklxkd-57/b?versionId=null 2023-05-31T20:37:26.620 DEBUG:boto:Data: 2023-05-31T20:37:26.620 DEBUG:boto:Headers: {} 2023-05-31T20:37:26.621 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:37:26.621 DEBUG:boto:Port: 8001 2023-05-31T20:37:26.621 DEBUG:boto:Params: {} 2023-05-31T20:37:26.622 DEBUG:boto:Token: None 2023-05-31T20:37:26.622 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:37:26 GMT /pklxkd-57/b?versionId=null 2023-05-31T20:37:26.623 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:z8kOPLvJodIPXUe3z9MF7R3SKt0= 2023-05-31T20:37:26.623 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:37:26 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:z8kOPLvJodIPXUe3z9MF7R3SKt0=', 'Content-Length': '0'} 2023-05-31T20:37:26.625 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:35:04 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a0bb2e90bef21953-006477b006-1b4f-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:37:26 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:37:26.626 DEBUG:boto:path=/pklxkd-57/b 2023-05-31T20:37:26.627 DEBUG:boto:auth_path=/pklxkd-57/b 2023-05-31T20:37:26.627 DEBUG:boto:path=/pklxkd-57/b?versionId=null 2023-05-31T20:37:26.628 DEBUG:boto:auth_path=/pklxkd-57/b?versionId=null 2023-05-31T20:37:26.628 DEBUG:boto:Method: GET 2023-05-31T20:37:26.628 DEBUG:boto:Path: /pklxkd-57/b?versionId=null 2023-05-31T20:37:26.629 DEBUG:boto:Data: 2023-05-31T20:37:26.629 DEBUG:boto:Headers: {} 2023-05-31T20:37:26.629 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:37:26.630 DEBUG:boto:Port: 8000 2023-05-31T20:37:26.630 DEBUG:boto:Params: {} 2023-05-31T20:37:26.631 DEBUG:boto:Token: None 2023-05-31T20:37:26.631 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:37:26 GMT /pklxkd-57/b?versionId=null 2023-05-31T20:37:26.632 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:z8kOPLvJodIPXUe3z9MF7R3SKt0= 2023-05-31T20:37:26.632 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:37:26 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:z8kOPLvJodIPXUe3z9MF7R3SKt0=', 'Content-Length': '0'} 2023-05-31T20:37:26.634 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:35:04 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006d34fe8d2f957404-006477b006-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:37:26 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:37:26.635 DEBUG:boto:path=/pklxkd-57/b 2023-05-31T20:37:26.635 DEBUG:boto:auth_path=/pklxkd-57/b 2023-05-31T20:37:26.635 DEBUG:boto:path=/pklxkd-57/b?versionId=null 2023-05-31T20:37:26.636 DEBUG:boto:auth_path=/pklxkd-57/b?versionId=null 2023-05-31T20:37:26.636 DEBUG:boto:Method: HEAD 2023-05-31T20:37:26.637 DEBUG:boto:Path: /pklxkd-57/b?versionId=null 2023-05-31T20:37:26.637 DEBUG:boto:Data: 2023-05-31T20:37:26.637 DEBUG:boto:Headers: {} 2023-05-31T20:37:26.638 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:37:26.638 DEBUG:boto:Port: 8001 2023-05-31T20:37:26.639 DEBUG:boto:Params: {} 2023-05-31T20:37:26.639 DEBUG:boto:Token: None 2023-05-31T20:37:26.639 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:37:26 GMT /pklxkd-57/b?versionId=null 2023-05-31T20:37:26.640 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:qbRGkXgCZJYS+XFBz8aLMcuZOy0= 2023-05-31T20:37:26.640 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:37:26 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:qbRGkXgCZJYS+XFBz8aLMcuZOy0=', 'Content-Length': '0'} 2023-05-31T20:37:26.642 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:35:04 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000fc4b30c9e7b74d50-006477b006-1b4f-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:37:26 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:37:26.643 DEBUG:boto:path=/pklxkd-57/b 2023-05-31T20:37:26.643 DEBUG:boto:auth_path=/pklxkd-57/b 2023-05-31T20:37:26.644 DEBUG:boto:path=/pklxkd-57/b?versionId=null 2023-05-31T20:37:26.644 DEBUG:boto:auth_path=/pklxkd-57/b?versionId=null 2023-05-31T20:37:26.644 DEBUG:boto:Method: HEAD 2023-05-31T20:37:26.645 DEBUG:boto:Path: /pklxkd-57/b?versionId=null 2023-05-31T20:37:26.645 DEBUG:boto:Data: 2023-05-31T20:37:26.645 DEBUG:boto:Headers: {} 2023-05-31T20:37:26.646 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:37:26.646 DEBUG:boto:Port: 8000 2023-05-31T20:37:26.646 DEBUG:boto:Params: {} 2023-05-31T20:37:26.647 DEBUG:boto:Token: None 2023-05-31T20:37:26.647 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:37:26 GMT /pklxkd-57/b?versionId=null 2023-05-31T20:37:26.648 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:qbRGkXgCZJYS+XFBz8aLMcuZOy0= 2023-05-31T20:37:26.648 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:37:26 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:qbRGkXgCZJYS+XFBz8aLMcuZOy0=', 'Content-Length': '0'} 2023-05-31T20:37:26.650 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:35:04 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001b1b11c7afecd944-006477b006-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:37:26 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:37:26.650 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=b 2023-05-31T20:37:26.651 DEBUG:boto:path=/pklxkd-57/b 2023-05-31T20:37:26.651 DEBUG:boto:auth_path=/pklxkd-57/b 2023-05-31T20:37:26.652 DEBUG:boto:path=/pklxkd-57/b?versionId=null 2023-05-31T20:37:26.652 DEBUG:boto:auth_path=/pklxkd-57/b?versionId=null 2023-05-31T20:37:26.652 DEBUG:boto:Method: GET 2023-05-31T20:37:26.653 DEBUG:boto:Path: /pklxkd-57/b?versionId=null 2023-05-31T20:37:26.653 DEBUG:boto:Data: 2023-05-31T20:37:26.654 DEBUG:boto:Headers: {} 2023-05-31T20:37:26.654 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:37:26.654 DEBUG:boto:Port: 8001 2023-05-31T20:37:26.655 DEBUG:boto:Params: {} 2023-05-31T20:37:26.655 DEBUG:boto:Token: None 2023-05-31T20:37:26.655 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:37:26 GMT /pklxkd-57/b?versionId=null 2023-05-31T20:37:26.656 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:z8kOPLvJodIPXUe3z9MF7R3SKt0= 2023-05-31T20:37:26.656 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:37:26 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:z8kOPLvJodIPXUe3z9MF7R3SKt0=', 'Content-Length': '0'} 2023-05-31T20:37:26.658 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:35:04 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009351910cd2cff769-006477b006-1b4f-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:37:26 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:37:26.659 DEBUG:boto:path=/pklxkd-57/b 2023-05-31T20:37:26.660 DEBUG:boto:auth_path=/pklxkd-57/b 2023-05-31T20:37:26.660 DEBUG:boto:path=/pklxkd-57/b?versionId=null 2023-05-31T20:37:26.660 DEBUG:boto:auth_path=/pklxkd-57/b?versionId=null 2023-05-31T20:37:26.661 DEBUG:boto:Method: GET 2023-05-31T20:37:26.661 DEBUG:boto:Path: /pklxkd-57/b?versionId=null 2023-05-31T20:37:26.661 DEBUG:boto:Data: 2023-05-31T20:37:26.662 DEBUG:boto:Headers: {} 2023-05-31T20:37:26.662 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:37:26.663 DEBUG:boto:Port: 8000 2023-05-31T20:37:26.663 DEBUG:boto:Params: {} 2023-05-31T20:37:26.663 DEBUG:boto:Token: None 2023-05-31T20:37:26.664 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:37:26 GMT /pklxkd-57/b?versionId=null 2023-05-31T20:37:26.664 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:z8kOPLvJodIPXUe3z9MF7R3SKt0= 2023-05-31T20:37:26.665 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:37:26 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:z8kOPLvJodIPXUe3z9MF7R3SKt0=', 'Content-Length': '0'} 2023-05-31T20:37:26.666 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:35:04 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000065919c1edf0003d-006477b006-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:37:26 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:37:26.667 DEBUG:boto:path=/pklxkd-57/ 2023-05-31T20:37:26.668 DEBUG:boto:auth_path=/pklxkd-57/ 2023-05-31T20:37:26.668 DEBUG:boto:path=/pklxkd-57/?prefix=b 2023-05-31T20:37:26.668 DEBUG:boto:auth_path=/pklxkd-57/?prefix=b 2023-05-31T20:37:26.669 DEBUG:boto:Method: GET 2023-05-31T20:37:26.669 DEBUG:boto:Path: /pklxkd-57/?prefix=b 2023-05-31T20:37:26.669 DEBUG:boto:Data: 2023-05-31T20:37:26.670 DEBUG:boto:Headers: {} 2023-05-31T20:37:26.670 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:37:26.671 DEBUG:boto:Port: 8001 2023-05-31T20:37:26.671 DEBUG:boto:Params: {} 2023-05-31T20:37:26.671 DEBUG:boto:Token: None 2023-05-31T20:37:26.672 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:37:26 GMT /pklxkd-57/ 2023-05-31T20:37:26.672 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:TOQ9jxSj7BFBWhrNcW+gciGRxKs= 2023-05-31T20:37:26.673 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:37:26 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:TOQ9jxSj7BFBWhrNcW+gciGRxKs=', 'Content-Length': '0'} 2023-05-31T20:37:26.675 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000dc1ae8e3e4f7702a-006477b006-1b4f-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:37:26 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:37:26.676 DEBUG:boto:b'pklxkd-57b1000falseb2023-05-31T20:35:04.600Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:37:26.676 DEBUG:boto:path=/pklxkd-57/ 2023-05-31T20:37:26.677 DEBUG:boto:auth_path=/pklxkd-57/ 2023-05-31T20:37:26.677 DEBUG:boto:path=/pklxkd-57/?prefix=b 2023-05-31T20:37:26.677 DEBUG:boto:auth_path=/pklxkd-57/?prefix=b 2023-05-31T20:37:26.678 DEBUG:boto:Method: GET 2023-05-31T20:37:26.678 DEBUG:boto:Path: /pklxkd-57/?prefix=b 2023-05-31T20:37:26.679 DEBUG:boto:Data: 2023-05-31T20:37:26.679 DEBUG:boto:Headers: {} 2023-05-31T20:37:26.679 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:37:26.680 DEBUG:boto:Port: 8000 2023-05-31T20:37:26.680 DEBUG:boto:Params: {} 2023-05-31T20:37:26.680 DEBUG:boto:Token: None 2023-05-31T20:37:26.681 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:37:26 GMT /pklxkd-57/ 2023-05-31T20:37:26.681 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:TOQ9jxSj7BFBWhrNcW+gciGRxKs= 2023-05-31T20:37:26.682 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:37:26 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:TOQ9jxSj7BFBWhrNcW+gciGRxKs=', 'Content-Length': '0'} 2023-05-31T20:37:26.683 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000001225dc681ebb4253-006477b006-1630-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:37:26 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:37:26.684 DEBUG:boto:b'pklxkd-57b1000falseb2023-05-31T20:35:04.600Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:37:26.685 DEBUG:boto:path=/pklxkd-57/b 2023-05-31T20:37:26.685 DEBUG:boto:auth_path=/pklxkd-57/b 2023-05-31T20:37:26.685 DEBUG:boto:Method: HEAD 2023-05-31T20:37:26.686 DEBUG:boto:Path: /pklxkd-57/b 2023-05-31T20:37:26.686 DEBUG:boto:Data: 2023-05-31T20:37:26.686 DEBUG:boto:Headers: {} 2023-05-31T20:37:26.687 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:37:26.687 DEBUG:boto:Port: 8001 2023-05-31T20:37:26.687 DEBUG:boto:Params: {} 2023-05-31T20:37:26.688 DEBUG:boto:Token: None 2023-05-31T20:37:26.688 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:37:26 GMT /pklxkd-57/b 2023-05-31T20:37:26.689 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:NoDfVIFtAoFq76QRtY5BQL3tXSM= 2023-05-31T20:37:26.689 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:37:26 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:NoDfVIFtAoFq76QRtY5BQL3tXSM=', 'Content-Length': '0'} 2023-05-31T20:37:26.691 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:35:04 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ae70fd9d3e736680-006477b006-1b4f-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:37:26 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:37:26.691 DEBUG:boto:path=/pklxkd-57/b 2023-05-31T20:37:26.692 DEBUG:boto:auth_path=/pklxkd-57/b 2023-05-31T20:37:26.692 DEBUG:boto:Method: HEAD 2023-05-31T20:37:26.692 DEBUG:boto:Path: /pklxkd-57/b 2023-05-31T20:37:26.693 DEBUG:boto:Data: 2023-05-31T20:37:26.693 DEBUG:boto:Headers: {} 2023-05-31T20:37:26.694 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:37:26.694 DEBUG:boto:Port: 8000 2023-05-31T20:37:26.694 DEBUG:boto:Params: {} 2023-05-31T20:37:26.695 DEBUG:boto:Token: None 2023-05-31T20:37:26.695 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:37:26 GMT /pklxkd-57/b 2023-05-31T20:37:26.695 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:NoDfVIFtAoFq76QRtY5BQL3tXSM= 2023-05-31T20:37:26.696 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:37:26 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:NoDfVIFtAoFq76QRtY5BQL3tXSM=', 'Content-Length': '0'} 2023-05-31T20:37:26.698 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:35:04 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006e3266a2a5188bbf-006477b006-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:37:26 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:37:26.698 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=b 2023-05-31T20:37:26.698 DEBUG:boto:path=/pklxkd-57/b 2023-05-31T20:37:26.699 DEBUG:boto:auth_path=/pklxkd-57/b 2023-05-31T20:37:26.699 DEBUG:boto:Method: GET 2023-05-31T20:37:26.700 DEBUG:boto:Path: /pklxkd-57/b 2023-05-31T20:37:26.700 DEBUG:boto:Data: 2023-05-31T20:37:26.700 DEBUG:boto:Headers: {} 2023-05-31T20:37:26.701 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:37:26.701 DEBUG:boto:Port: 8001 2023-05-31T20:37:26.701 DEBUG:boto:Params: {} 2023-05-31T20:37:26.702 DEBUG:boto:Token: None 2023-05-31T20:37:26.702 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:37:26 GMT /pklxkd-57/b 2023-05-31T20:37:26.703 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:o91VdFdGw7zshr7JxS6/c3Pr9mY= 2023-05-31T20:37:26.703 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:37:26 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:o91VdFdGw7zshr7JxS6/c3Pr9mY=', 'Content-Length': '0'} 2023-05-31T20:37:26.705 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:35:04 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006b795898296b1a6b-006477b006-1b4f-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:37:26 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:37:26.705 DEBUG:boto:path=/pklxkd-57/b 2023-05-31T20:37:26.706 DEBUG:boto:auth_path=/pklxkd-57/b 2023-05-31T20:37:26.706 DEBUG:boto:Method: GET 2023-05-31T20:37:26.707 DEBUG:boto:Path: /pklxkd-57/b 2023-05-31T20:37:26.707 DEBUG:boto:Data: 2023-05-31T20:37:26.707 DEBUG:boto:Headers: {} 2023-05-31T20:37:26.708 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:37:26.708 DEBUG:boto:Port: 8000 2023-05-31T20:37:26.708 DEBUG:boto:Params: {} 2023-05-31T20:37:26.709 DEBUG:boto:Token: None 2023-05-31T20:37:26.709 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:37:26 GMT /pklxkd-57/b 2023-05-31T20:37:26.710 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:o91VdFdGw7zshr7JxS6/c3Pr9mY= 2023-05-31T20:37:26.710 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:37:26 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:o91VdFdGw7zshr7JxS6/c3Pr9mY=', 'Content-Length': '0'} 2023-05-31T20:37:26.712 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:35:04 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a130d65e95d30558-006477b006-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:37:26 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:37:26.712 DEBUG:boto:path=/pklxkd-57/ 2023-05-31T20:37:26.713 DEBUG:boto:auth_path=/pklxkd-57/ 2023-05-31T20:37:26.713 DEBUG:boto:path=/pklxkd-57/?prefix=b 2023-05-31T20:37:26.714 DEBUG:boto:auth_path=/pklxkd-57/?prefix=b 2023-05-31T20:37:26.714 DEBUG:boto:Method: GET 2023-05-31T20:37:26.714 DEBUG:boto:Path: /pklxkd-57/?prefix=b 2023-05-31T20:37:26.715 DEBUG:boto:Data: 2023-05-31T20:37:26.715 DEBUG:boto:Headers: {} 2023-05-31T20:37:26.715 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:37:26.716 DEBUG:boto:Port: 8001 2023-05-31T20:37:26.716 DEBUG:boto:Params: {} 2023-05-31T20:37:26.716 DEBUG:boto:Token: None 2023-05-31T20:37:26.717 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:37:26 GMT /pklxkd-57/ 2023-05-31T20:37:26.717 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:TOQ9jxSj7BFBWhrNcW+gciGRxKs= 2023-05-31T20:37:26.718 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:37:26 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:TOQ9jxSj7BFBWhrNcW+gciGRxKs=', 'Content-Length': '0'} 2023-05-31T20:37:26.720 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000065f9e6d15b5b5c41-006477b006-1b4f-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:37:26 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:37:26.760 DEBUG:boto:b'pklxkd-57b1000falseb2023-05-31T20:35:04.600Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:37:26.761 DEBUG:boto:path=/pklxkd-57/ 2023-05-31T20:37:26.761 DEBUG:boto:auth_path=/pklxkd-57/ 2023-05-31T20:37:26.762 DEBUG:boto:path=/pklxkd-57/?prefix=b 2023-05-31T20:37:26.762 DEBUG:boto:auth_path=/pklxkd-57/?prefix=b 2023-05-31T20:37:26.763 DEBUG:boto:Method: GET 2023-05-31T20:37:26.763 DEBUG:boto:Path: /pklxkd-57/?prefix=b 2023-05-31T20:37:26.763 DEBUG:boto:Data: 2023-05-31T20:37:26.764 DEBUG:boto:Headers: {} 2023-05-31T20:37:26.765 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:37:26.765 DEBUG:boto:Port: 8000 2023-05-31T20:37:26.766 DEBUG:boto:Params: {} 2023-05-31T20:37:26.767 DEBUG:boto:Token: None 2023-05-31T20:37:26.768 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:37:26 GMT /pklxkd-57/ 2023-05-31T20:37:26.769 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:TOQ9jxSj7BFBWhrNcW+gciGRxKs= 2023-05-31T20:37:26.769 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:37:26 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:TOQ9jxSj7BFBWhrNcW+gciGRxKs=', 'Content-Length': '0'} 2023-05-31T20:37:26.772 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d417553b3f5f7fbd-006477b006-1630-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:37:26 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:37:26.773 DEBUG:boto:b'pklxkd-57b1000falseb2023-05-31T20:35:04.600Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:37:26.774 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=c 2023-05-31T20:37:26.775 DEBUG:boto:path=/pklxkd-57/c 2023-05-31T20:37:26.776 DEBUG:boto:auth_path=/pklxkd-57/c 2023-05-31T20:37:26.776 DEBUG:boto:path=/pklxkd-57/c?versionId=null 2023-05-31T20:37:26.777 DEBUG:boto:auth_path=/pklxkd-57/c?versionId=null 2023-05-31T20:37:26.778 DEBUG:boto:Method: GET 2023-05-31T20:37:26.778 DEBUG:boto:Path: /pklxkd-57/c?versionId=null 2023-05-31T20:37:26.779 DEBUG:boto:Data: 2023-05-31T20:37:26.780 DEBUG:boto:Headers: {} 2023-05-31T20:37:26.780 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:37:26.781 DEBUG:boto:Port: 8001 2023-05-31T20:37:26.782 DEBUG:boto:Params: {} 2023-05-31T20:37:26.783 DEBUG:boto:Token: None 2023-05-31T20:37:26.783 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:37:26 GMT /pklxkd-57/c?versionId=null 2023-05-31T20:37:26.784 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:id1hHL2HKZjo+aUVgkAHlx5mW/c= 2023-05-31T20:37:26.785 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:37:26 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:id1hHL2HKZjo+aUVgkAHlx5mW/c=', 'Content-Length': '0'} 2023-05-31T20:37:26.788 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:35:04 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000013b0f2a33d50239c-006477b006-1b4f-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:37:26 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:37:26.789 DEBUG:boto:path=/pklxkd-57/c 2023-05-31T20:37:26.789 DEBUG:boto:auth_path=/pklxkd-57/c 2023-05-31T20:37:26.790 DEBUG:boto:path=/pklxkd-57/c?versionId=null 2023-05-31T20:37:26.790 DEBUG:boto:auth_path=/pklxkd-57/c?versionId=null 2023-05-31T20:37:26.790 DEBUG:boto:Method: GET 2023-05-31T20:37:26.791 DEBUG:boto:Path: /pklxkd-57/c?versionId=null 2023-05-31T20:37:26.791 DEBUG:boto:Data: 2023-05-31T20:37:26.791 DEBUG:boto:Headers: {} 2023-05-31T20:37:26.792 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:37:26.792 DEBUG:boto:Port: 8000 2023-05-31T20:37:26.793 DEBUG:boto:Params: {} 2023-05-31T20:37:26.794 DEBUG:boto:Token: None 2023-05-31T20:37:26.794 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:37:26 GMT /pklxkd-57/c?versionId=null 2023-05-31T20:37:26.795 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:id1hHL2HKZjo+aUVgkAHlx5mW/c= 2023-05-31T20:37:26.795 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:37:26 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:id1hHL2HKZjo+aUVgkAHlx5mW/c=', 'Content-Length': '0'} 2023-05-31T20:37:26.798 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:35:04 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e666bceaac6259c0-006477b006-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:37:26 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:37:26.799 DEBUG:boto:path=/pklxkd-57/c 2023-05-31T20:37:26.800 DEBUG:boto:auth_path=/pklxkd-57/c 2023-05-31T20:37:26.800 DEBUG:boto:path=/pklxkd-57/c?versionId=null 2023-05-31T20:37:26.801 DEBUG:boto:auth_path=/pklxkd-57/c?versionId=null 2023-05-31T20:37:26.801 DEBUG:boto:Method: HEAD 2023-05-31T20:37:26.801 DEBUG:boto:Path: /pklxkd-57/c?versionId=null 2023-05-31T20:37:26.802 DEBUG:boto:Data: 2023-05-31T20:37:26.802 DEBUG:boto:Headers: {} 2023-05-31T20:37:26.802 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:37:26.803 DEBUG:boto:Port: 8001 2023-05-31T20:37:26.803 DEBUG:boto:Params: {} 2023-05-31T20:37:26.804 DEBUG:boto:Token: None 2023-05-31T20:37:26.804 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:37:26 GMT /pklxkd-57/c?versionId=null 2023-05-31T20:37:26.805 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:o0CVzRUA1hz2M01Y8xU9sZ4VXLk= 2023-05-31T20:37:26.805 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:37:26 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:o0CVzRUA1hz2M01Y8xU9sZ4VXLk=', 'Content-Length': '0'} 2023-05-31T20:37:26.807 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:35:04 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ab141f313fc9d3d5-006477b006-1b4f-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:37:26 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:37:26.808 DEBUG:boto:path=/pklxkd-57/c 2023-05-31T20:37:26.809 DEBUG:boto:auth_path=/pklxkd-57/c 2023-05-31T20:37:26.809 DEBUG:boto:path=/pklxkd-57/c?versionId=null 2023-05-31T20:37:26.813 DEBUG:boto:auth_path=/pklxkd-57/c?versionId=null 2023-05-31T20:37:26.813 DEBUG:boto:Method: HEAD 2023-05-31T20:37:26.814 DEBUG:boto:Path: /pklxkd-57/c?versionId=null 2023-05-31T20:37:26.814 DEBUG:boto:Data: 2023-05-31T20:37:26.814 DEBUG:boto:Headers: {} 2023-05-31T20:37:26.815 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:37:26.815 DEBUG:boto:Port: 8000 2023-05-31T20:37:26.815 DEBUG:boto:Params: {} 2023-05-31T20:37:26.816 DEBUG:boto:Token: None 2023-05-31T20:37:26.816 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:37:26 GMT /pklxkd-57/c?versionId=null 2023-05-31T20:37:26.817 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:o0CVzRUA1hz2M01Y8xU9sZ4VXLk= 2023-05-31T20:37:26.817 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:37:26 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:o0CVzRUA1hz2M01Y8xU9sZ4VXLk=', 'Content-Length': '0'} 2023-05-31T20:37:26.819 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:35:04 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007bbecf7be9563504-006477b006-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:37:26 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:37:26.820 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=c 2023-05-31T20:37:26.820 DEBUG:boto:path=/pklxkd-57/c 2023-05-31T20:37:26.821 DEBUG:boto:auth_path=/pklxkd-57/c 2023-05-31T20:37:26.821 DEBUG:boto:path=/pklxkd-57/c?versionId=null 2023-05-31T20:37:26.821 DEBUG:boto:auth_path=/pklxkd-57/c?versionId=null 2023-05-31T20:37:26.823 DEBUG:boto:Method: GET 2023-05-31T20:37:26.823 DEBUG:boto:Path: /pklxkd-57/c?versionId=null 2023-05-31T20:37:26.824 DEBUG:boto:Data: 2023-05-31T20:37:26.824 DEBUG:boto:Headers: {} 2023-05-31T20:37:26.824 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:37:26.825 DEBUG:boto:Port: 8001 2023-05-31T20:37:26.825 DEBUG:boto:Params: {} 2023-05-31T20:37:26.826 DEBUG:boto:Token: None 2023-05-31T20:37:26.826 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:37:26 GMT /pklxkd-57/c?versionId=null 2023-05-31T20:37:26.827 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:id1hHL2HKZjo+aUVgkAHlx5mW/c= 2023-05-31T20:37:26.827 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:37:26 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:id1hHL2HKZjo+aUVgkAHlx5mW/c=', 'Content-Length': '0'} 2023-05-31T20:37:26.829 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:35:04 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f8766573e8278086-006477b006-1b4f-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:37:26 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:37:26.830 DEBUG:boto:path=/pklxkd-57/c 2023-05-31T20:37:26.830 DEBUG:boto:auth_path=/pklxkd-57/c 2023-05-31T20:37:26.831 DEBUG:boto:path=/pklxkd-57/c?versionId=null 2023-05-31T20:37:26.831 DEBUG:boto:auth_path=/pklxkd-57/c?versionId=null 2023-05-31T20:37:26.831 DEBUG:boto:Method: GET 2023-05-31T20:37:26.832 DEBUG:boto:Path: /pklxkd-57/c?versionId=null 2023-05-31T20:37:26.832 DEBUG:boto:Data: 2023-05-31T20:37:26.832 DEBUG:boto:Headers: {} 2023-05-31T20:37:26.833 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:37:26.834 DEBUG:boto:Port: 8000 2023-05-31T20:37:26.834 DEBUG:boto:Params: {} 2023-05-31T20:37:26.835 DEBUG:boto:Token: None 2023-05-31T20:37:26.835 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:37:26 GMT /pklxkd-57/c?versionId=null 2023-05-31T20:37:26.835 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:id1hHL2HKZjo+aUVgkAHlx5mW/c= 2023-05-31T20:37:26.836 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:37:26 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:id1hHL2HKZjo+aUVgkAHlx5mW/c=', 'Content-Length': '0'} 2023-05-31T20:37:26.838 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:35:04 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000dfd093986dd671e-006477b006-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:37:26 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:37:26.839 DEBUG:boto:path=/pklxkd-57/ 2023-05-31T20:37:26.839 DEBUG:boto:auth_path=/pklxkd-57/ 2023-05-31T20:37:26.840 DEBUG:boto:path=/pklxkd-57/?prefix=c 2023-05-31T20:37:26.840 DEBUG:boto:auth_path=/pklxkd-57/?prefix=c 2023-05-31T20:37:26.841 DEBUG:boto:Method: GET 2023-05-31T20:37:26.841 DEBUG:boto:Path: /pklxkd-57/?prefix=c 2023-05-31T20:37:26.842 DEBUG:boto:Data: 2023-05-31T20:37:26.842 DEBUG:boto:Headers: {} 2023-05-31T20:37:26.843 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:37:26.843 DEBUG:boto:Port: 8001 2023-05-31T20:37:26.843 DEBUG:boto:Params: {} 2023-05-31T20:37:26.844 DEBUG:boto:Token: None 2023-05-31T20:37:26.844 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:37:26 GMT /pklxkd-57/ 2023-05-31T20:37:26.845 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:TOQ9jxSj7BFBWhrNcW+gciGRxKs= 2023-05-31T20:37:26.845 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:37:26 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:TOQ9jxSj7BFBWhrNcW+gciGRxKs=', 'Content-Length': '0'} 2023-05-31T20:37:26.848 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000481744dfbef997a5-006477b006-1b4f-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:37:26 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:37:26.848 DEBUG:boto:b'pklxkd-57c1000falsec2023-05-31T20:35:04.620Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:37:26.849 DEBUG:boto:path=/pklxkd-57/ 2023-05-31T20:37:26.850 DEBUG:boto:auth_path=/pklxkd-57/ 2023-05-31T20:37:26.851 DEBUG:boto:path=/pklxkd-57/?prefix=c 2023-05-31T20:37:26.851 DEBUG:boto:auth_path=/pklxkd-57/?prefix=c 2023-05-31T20:37:26.851 DEBUG:boto:Method: GET 2023-05-31T20:37:26.852 DEBUG:boto:Path: /pklxkd-57/?prefix=c 2023-05-31T20:37:26.852 DEBUG:boto:Data: 2023-05-31T20:37:26.853 DEBUG:boto:Headers: {} 2023-05-31T20:37:26.853 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:37:26.853 DEBUG:boto:Port: 8000 2023-05-31T20:37:26.854 DEBUG:boto:Params: {} 2023-05-31T20:37:26.854 DEBUG:boto:Token: None 2023-05-31T20:37:26.854 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:37:26 GMT /pklxkd-57/ 2023-05-31T20:37:26.855 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:TOQ9jxSj7BFBWhrNcW+gciGRxKs= 2023-05-31T20:37:26.855 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:37:26 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:TOQ9jxSj7BFBWhrNcW+gciGRxKs=', 'Content-Length': '0'} 2023-05-31T20:37:26.858 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000078492c6ab6e35b8e-006477b006-1630-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:37:26 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:37:26.859 DEBUG:boto:b'pklxkd-57c1000falsec2023-05-31T20:35:04.620Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:37:26.859 DEBUG:boto:path=/pklxkd-57/c 2023-05-31T20:37:26.860 DEBUG:boto:auth_path=/pklxkd-57/c 2023-05-31T20:37:26.860 DEBUG:boto:Method: HEAD 2023-05-31T20:37:26.861 DEBUG:boto:Path: /pklxkd-57/c 2023-05-31T20:37:26.861 DEBUG:boto:Data: 2023-05-31T20:37:26.861 DEBUG:boto:Headers: {} 2023-05-31T20:37:26.862 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:37:26.862 DEBUG:boto:Port: 8001 2023-05-31T20:37:26.862 DEBUG:boto:Params: {} 2023-05-31T20:37:26.863 DEBUG:boto:Token: None 2023-05-31T20:37:26.863 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:37:26 GMT /pklxkd-57/c 2023-05-31T20:37:26.864 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:9ia3FhBeDCMuw8Wzld7kWwhau5U= 2023-05-31T20:37:26.864 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:37:26 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:9ia3FhBeDCMuw8Wzld7kWwhau5U=', 'Content-Length': '0'} 2023-05-31T20:37:26.868 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:35:04 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d6c5f4b5617509c1-006477b006-1b4f-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:37:26 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:37:26.869 DEBUG:boto:path=/pklxkd-57/c 2023-05-31T20:37:26.869 DEBUG:boto:auth_path=/pklxkd-57/c 2023-05-31T20:37:26.870 DEBUG:boto:Method: HEAD 2023-05-31T20:37:26.870 DEBUG:boto:Path: /pklxkd-57/c 2023-05-31T20:37:26.870 DEBUG:boto:Data: 2023-05-31T20:37:26.871 DEBUG:boto:Headers: {} 2023-05-31T20:37:26.871 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:37:26.871 DEBUG:boto:Port: 8000 2023-05-31T20:37:26.872 DEBUG:boto:Params: {} 2023-05-31T20:37:26.872 DEBUG:boto:Token: None 2023-05-31T20:37:26.873 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:37:26 GMT /pklxkd-57/c 2023-05-31T20:37:26.873 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:9ia3FhBeDCMuw8Wzld7kWwhau5U= 2023-05-31T20:37:26.874 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:37:26 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:9ia3FhBeDCMuw8Wzld7kWwhau5U=', 'Content-Length': '0'} 2023-05-31T20:37:26.875 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:35:04 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000db415ab552528a5a-006477b006-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:37:26 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:37:26.876 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=c 2023-05-31T20:37:26.876 DEBUG:boto:path=/pklxkd-57/c 2023-05-31T20:37:26.877 DEBUG:boto:auth_path=/pklxkd-57/c 2023-05-31T20:37:26.877 DEBUG:boto:Method: GET 2023-05-31T20:37:26.878 DEBUG:boto:Path: /pklxkd-57/c 2023-05-31T20:37:26.878 DEBUG:boto:Data: 2023-05-31T20:37:26.879 DEBUG:boto:Headers: {} 2023-05-31T20:37:26.879 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:37:26.879 DEBUG:boto:Port: 8001 2023-05-31T20:37:26.880 DEBUG:boto:Params: {} 2023-05-31T20:37:26.880 DEBUG:boto:Token: None 2023-05-31T20:37:26.881 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:37:26 GMT /pklxkd-57/c 2023-05-31T20:37:26.881 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:g8d9lojqArsKdvfJL2EpFGAooV8= 2023-05-31T20:37:26.882 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:37:26 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:g8d9lojqArsKdvfJL2EpFGAooV8=', 'Content-Length': '0'} 2023-05-31T20:37:26.884 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:35:04 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c65724c56d050d86-006477b006-1b4f-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:37:26 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:37:26.884 DEBUG:boto:path=/pklxkd-57/c 2023-05-31T20:37:26.885 DEBUG:boto:auth_path=/pklxkd-57/c 2023-05-31T20:37:26.885 DEBUG:boto:Method: GET 2023-05-31T20:37:26.886 DEBUG:boto:Path: /pklxkd-57/c 2023-05-31T20:37:26.886 DEBUG:boto:Data: 2023-05-31T20:37:26.886 DEBUG:boto:Headers: {} 2023-05-31T20:37:26.887 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:37:26.887 DEBUG:boto:Port: 8000 2023-05-31T20:37:26.888 DEBUG:boto:Params: {} 2023-05-31T20:37:26.888 DEBUG:boto:Token: None 2023-05-31T20:37:26.889 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:37:26 GMT /pklxkd-57/c 2023-05-31T20:37:26.889 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:g8d9lojqArsKdvfJL2EpFGAooV8= 2023-05-31T20:37:26.890 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:37:26 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:g8d9lojqArsKdvfJL2EpFGAooV8=', 'Content-Length': '0'} 2023-05-31T20:37:26.891 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:35:04 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000045293373133e05c7-006477b006-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:37:26 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:37:26.892 DEBUG:boto:path=/pklxkd-57/ 2023-05-31T20:37:26.893 DEBUG:boto:auth_path=/pklxkd-57/ 2023-05-31T20:37:26.893 DEBUG:boto:path=/pklxkd-57/?prefix=c 2023-05-31T20:37:26.893 DEBUG:boto:auth_path=/pklxkd-57/?prefix=c 2023-05-31T20:37:26.894 DEBUG:boto:Method: GET 2023-05-31T20:37:26.894 DEBUG:boto:Path: /pklxkd-57/?prefix=c 2023-05-31T20:37:26.895 DEBUG:boto:Data: 2023-05-31T20:37:26.895 DEBUG:boto:Headers: {} 2023-05-31T20:37:26.896 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:37:26.896 DEBUG:boto:Port: 8001 2023-05-31T20:37:26.896 DEBUG:boto:Params: {} 2023-05-31T20:37:26.897 DEBUG:boto:Token: None 2023-05-31T20:37:26.897 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:37:26 GMT /pklxkd-57/ 2023-05-31T20:37:26.898 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:TOQ9jxSj7BFBWhrNcW+gciGRxKs= 2023-05-31T20:37:26.898 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:37:26 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:TOQ9jxSj7BFBWhrNcW+gciGRxKs=', 'Content-Length': '0'} 2023-05-31T20:37:26.901 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a18b790a91440f6d-006477b006-1b4f-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:37:26 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:37:26.902 DEBUG:boto:b'pklxkd-57c1000falsec2023-05-31T20:35:04.620Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:37:26.902 DEBUG:boto:path=/pklxkd-57/ 2023-05-31T20:37:26.903 DEBUG:boto:auth_path=/pklxkd-57/ 2023-05-31T20:37:26.903 DEBUG:boto:path=/pklxkd-57/?prefix=c 2023-05-31T20:37:26.904 DEBUG:boto:auth_path=/pklxkd-57/?prefix=c 2023-05-31T20:37:26.904 DEBUG:boto:Method: GET 2023-05-31T20:37:26.904 DEBUG:boto:Path: /pklxkd-57/?prefix=c 2023-05-31T20:37:26.905 DEBUG:boto:Data: 2023-05-31T20:37:26.905 DEBUG:boto:Headers: {} 2023-05-31T20:37:26.905 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:37:26.906 DEBUG:boto:Port: 8000 2023-05-31T20:37:26.906 DEBUG:boto:Params: {} 2023-05-31T20:37:26.907 DEBUG:boto:Token: None 2023-05-31T20:37:26.907 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:37:26 GMT /pklxkd-57/ 2023-05-31T20:37:26.908 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:TOQ9jxSj7BFBWhrNcW+gciGRxKs= 2023-05-31T20:37:26.909 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:37:26 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:TOQ9jxSj7BFBWhrNcW+gciGRxKs=', 'Content-Length': '0'} 2023-05-31T20:37:26.911 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000009d5aa3abddf6c0aa-006477b006-1630-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:37:26 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:37:26.911 DEBUG:boto:b'pklxkd-57c1000falsec2023-05-31T20:35:04.620Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:37:26.912 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=d 2023-05-31T20:37:26.913 DEBUG:boto:path=/pklxkd-57/d 2023-05-31T20:37:26.913 DEBUG:boto:auth_path=/pklxkd-57/d 2023-05-31T20:37:26.914 DEBUG:boto:path=/pklxkd-57/d?versionId=null 2023-05-31T20:37:26.914 DEBUG:boto:auth_path=/pklxkd-57/d?versionId=null 2023-05-31T20:37:26.915 DEBUG:boto:Method: GET 2023-05-31T20:37:26.915 DEBUG:boto:Path: /pklxkd-57/d?versionId=null 2023-05-31T20:37:26.916 DEBUG:boto:Data: 2023-05-31T20:37:26.916 DEBUG:boto:Headers: {} 2023-05-31T20:37:26.917 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:37:26.917 DEBUG:boto:Port: 8001 2023-05-31T20:37:26.917 DEBUG:boto:Params: {} 2023-05-31T20:37:26.918 DEBUG:boto:Token: None 2023-05-31T20:37:26.918 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:37:26 GMT /pklxkd-57/d?versionId=null 2023-05-31T20:37:26.919 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:Nqb/oiNYsal0Gv3lY2XuJYojxz8= 2023-05-31T20:37:26.920 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:37:26 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:Nqb/oiNYsal0Gv3lY2XuJYojxz8=', 'Content-Length': '0'} 2023-05-31T20:37:26.922 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:35:04 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006ead651bb8de0637-006477b006-1b4f-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:37:26 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:37:26.965 DEBUG:boto:path=/pklxkd-57/d 2023-05-31T20:37:26.966 DEBUG:boto:auth_path=/pklxkd-57/d 2023-05-31T20:37:26.967 DEBUG:boto:path=/pklxkd-57/d?versionId=null 2023-05-31T20:37:26.967 DEBUG:boto:auth_path=/pklxkd-57/d?versionId=null 2023-05-31T20:37:26.968 DEBUG:boto:Method: GET 2023-05-31T20:37:26.968 DEBUG:boto:Path: /pklxkd-57/d?versionId=null 2023-05-31T20:37:26.968 DEBUG:boto:Data: 2023-05-31T20:37:26.969 DEBUG:boto:Headers: {} 2023-05-31T20:37:26.969 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:37:26.970 DEBUG:boto:Port: 8000 2023-05-31T20:37:26.970 DEBUG:boto:Params: {} 2023-05-31T20:37:26.970 DEBUG:boto:Token: None 2023-05-31T20:37:26.971 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:37:26 GMT /pklxkd-57/d?versionId=null 2023-05-31T20:37:26.971 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:Nqb/oiNYsal0Gv3lY2XuJYojxz8= 2023-05-31T20:37:26.972 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:37:26 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:Nqb/oiNYsal0Gv3lY2XuJYojxz8=', 'Content-Length': '0'} 2023-05-31T20:37:26.974 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:35:04 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006a912f8319497a6b-006477b006-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:37:26 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:37:26.975 DEBUG:boto:path=/pklxkd-57/d 2023-05-31T20:37:26.976 DEBUG:boto:auth_path=/pklxkd-57/d 2023-05-31T20:37:26.977 DEBUG:boto:path=/pklxkd-57/d?versionId=null 2023-05-31T20:37:26.977 DEBUG:boto:auth_path=/pklxkd-57/d?versionId=null 2023-05-31T20:37:26.978 DEBUG:boto:Method: HEAD 2023-05-31T20:37:26.979 DEBUG:boto:Path: /pklxkd-57/d?versionId=null 2023-05-31T20:37:26.979 DEBUG:boto:Data: 2023-05-31T20:37:26.980 DEBUG:boto:Headers: {} 2023-05-31T20:37:26.980 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:37:26.981 DEBUG:boto:Port: 8001 2023-05-31T20:37:26.981 DEBUG:boto:Params: {} 2023-05-31T20:37:26.982 DEBUG:boto:Token: None 2023-05-31T20:37:26.982 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:37:26 GMT /pklxkd-57/d?versionId=null 2023-05-31T20:37:26.983 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:qvvxytkAO83VWJR4mWVl8lroLbU= 2023-05-31T20:37:26.983 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:37:26 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:qvvxytkAO83VWJR4mWVl8lroLbU=', 'Content-Length': '0'} 2023-05-31T20:37:26.985 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:35:04 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008196a6140c8a8e3f-006477b006-1b4f-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:37:26 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:37:26.986 DEBUG:boto:path=/pklxkd-57/d 2023-05-31T20:37:26.986 DEBUG:boto:auth_path=/pklxkd-57/d 2023-05-31T20:37:26.987 DEBUG:boto:path=/pklxkd-57/d?versionId=null 2023-05-31T20:37:26.987 DEBUG:boto:auth_path=/pklxkd-57/d?versionId=null 2023-05-31T20:37:26.987 DEBUG:boto:Method: HEAD 2023-05-31T20:37:26.988 DEBUG:boto:Path: /pklxkd-57/d?versionId=null 2023-05-31T20:37:26.988 DEBUG:boto:Data: 2023-05-31T20:37:26.988 DEBUG:boto:Headers: {} 2023-05-31T20:37:26.989 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:37:26.989 DEBUG:boto:Port: 8000 2023-05-31T20:37:26.989 DEBUG:boto:Params: {} 2023-05-31T20:37:26.990 DEBUG:boto:Token: None 2023-05-31T20:37:26.990 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:37:26 GMT /pklxkd-57/d?versionId=null 2023-05-31T20:37:26.991 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:qvvxytkAO83VWJR4mWVl8lroLbU= 2023-05-31T20:37:26.991 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:37:26 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:qvvxytkAO83VWJR4mWVl8lroLbU=', 'Content-Length': '0'} 2023-05-31T20:37:26.993 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:35:04 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000096c5ee455e26d6c9-006477b006-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:37:26 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:37:26.994 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=d 2023-05-31T20:37:26.994 DEBUG:boto:path=/pklxkd-57/d 2023-05-31T20:37:26.995 DEBUG:boto:auth_path=/pklxkd-57/d 2023-05-31T20:37:26.995 DEBUG:boto:path=/pklxkd-57/d?versionId=null 2023-05-31T20:37:26.995 DEBUG:boto:auth_path=/pklxkd-57/d?versionId=null 2023-05-31T20:37:26.996 DEBUG:boto:Method: GET 2023-05-31T20:37:26.996 DEBUG:boto:Path: /pklxkd-57/d?versionId=null 2023-05-31T20:37:26.997 DEBUG:boto:Data: 2023-05-31T20:37:26.997 DEBUG:boto:Headers: {} 2023-05-31T20:37:26.997 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:37:26.998 DEBUG:boto:Port: 8001 2023-05-31T20:37:26.998 DEBUG:boto:Params: {} 2023-05-31T20:37:26.999 DEBUG:boto:Token: None 2023-05-31T20:37:26.999 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:37:26 GMT /pklxkd-57/d?versionId=null 2023-05-31T20:37:27.000 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:Nqb/oiNYsal0Gv3lY2XuJYojxz8= 2023-05-31T20:37:27.000 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:37:26 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:Nqb/oiNYsal0Gv3lY2XuJYojxz8=', 'Content-Length': '0'} 2023-05-31T20:37:27.002 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:35:04 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000064c848131ec83e1c-006477b006-1b4f-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:37:26 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:37:27.003 DEBUG:boto:path=/pklxkd-57/d 2023-05-31T20:37:27.003 DEBUG:boto:auth_path=/pklxkd-57/d 2023-05-31T20:37:27.004 DEBUG:boto:path=/pklxkd-57/d?versionId=null 2023-05-31T20:37:27.004 DEBUG:boto:auth_path=/pklxkd-57/d?versionId=null 2023-05-31T20:37:27.005 DEBUG:boto:Method: GET 2023-05-31T20:37:27.005 DEBUG:boto:Path: /pklxkd-57/d?versionId=null 2023-05-31T20:37:27.005 DEBUG:boto:Data: 2023-05-31T20:37:27.006 DEBUG:boto:Headers: {} 2023-05-31T20:37:27.006 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:37:27.007 DEBUG:boto:Port: 8000 2023-05-31T20:37:27.007 DEBUG:boto:Params: {} 2023-05-31T20:37:27.007 DEBUG:boto:Token: None 2023-05-31T20:37:27.008 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:37:27 GMT /pklxkd-57/d?versionId=null 2023-05-31T20:37:27.009 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:/PYlAyJK7kArdbCkC9y51BqPxrg= 2023-05-31T20:37:27.009 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:37:27 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:/PYlAyJK7kArdbCkC9y51BqPxrg=', 'Content-Length': '0'} 2023-05-31T20:37:27.011 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:35:04 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003fd0a377b5f66d71-006477b007-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:37:27 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:37:27.052 DEBUG:boto:path=/pklxkd-57/ 2023-05-31T20:37:27.053 DEBUG:boto:auth_path=/pklxkd-57/ 2023-05-31T20:37:27.054 DEBUG:boto:path=/pklxkd-57/?prefix=d 2023-05-31T20:37:27.054 DEBUG:boto:auth_path=/pklxkd-57/?prefix=d 2023-05-31T20:37:27.054 DEBUG:boto:Method: GET 2023-05-31T20:37:27.055 DEBUG:boto:Path: /pklxkd-57/?prefix=d 2023-05-31T20:37:27.055 DEBUG:boto:Data: 2023-05-31T20:37:27.056 DEBUG:boto:Headers: {} 2023-05-31T20:37:27.056 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:37:27.057 DEBUG:boto:Port: 8001 2023-05-31T20:37:27.057 DEBUG:boto:Params: {} 2023-05-31T20:37:27.057 DEBUG:boto:Token: None 2023-05-31T20:37:27.058 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:37:27 GMT /pklxkd-57/ 2023-05-31T20:37:27.058 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:kR8XV0FF++frnB9KbRgaOASXdNM= 2023-05-31T20:37:27.059 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:37:27 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:kR8XV0FF++frnB9KbRgaOASXdNM=', 'Content-Length': '0'} 2023-05-31T20:37:27.061 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000000948a866dd2dadf2-006477b007-1b4f-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:37:27 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:37:27.062 DEBUG:boto:b'pklxkd-57d1000falsed2023-05-31T20:35:04.680Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:37:27.063 DEBUG:boto:path=/pklxkd-57/ 2023-05-31T20:37:27.063 DEBUG:boto:auth_path=/pklxkd-57/ 2023-05-31T20:37:27.064 DEBUG:boto:path=/pklxkd-57/?prefix=d 2023-05-31T20:37:27.064 DEBUG:boto:auth_path=/pklxkd-57/?prefix=d 2023-05-31T20:37:27.064 DEBUG:boto:Method: GET 2023-05-31T20:37:27.065 DEBUG:boto:Path: /pklxkd-57/?prefix=d 2023-05-31T20:37:27.065 DEBUG:boto:Data: 2023-05-31T20:37:27.065 DEBUG:boto:Headers: {} 2023-05-31T20:37:27.066 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:37:27.066 DEBUG:boto:Port: 8000 2023-05-31T20:37:27.067 DEBUG:boto:Params: {} 2023-05-31T20:37:27.067 DEBUG:boto:Token: None 2023-05-31T20:37:27.067 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:37:27 GMT /pklxkd-57/ 2023-05-31T20:37:27.068 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:kR8XV0FF++frnB9KbRgaOASXdNM= 2023-05-31T20:37:27.068 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:37:27 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:kR8XV0FF++frnB9KbRgaOASXdNM=', 'Content-Length': '0'} 2023-05-31T20:37:27.070 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000004e8e3c21fde41f00-006477b007-1630-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:37:27 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:37:27.071 DEBUG:boto:b'pklxkd-57d1000falsed2023-05-31T20:35:04.680Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:37:27.072 DEBUG:boto:path=/pklxkd-57/d 2023-05-31T20:37:27.072 DEBUG:boto:auth_path=/pklxkd-57/d 2023-05-31T20:37:27.073 DEBUG:boto:Method: HEAD 2023-05-31T20:37:27.073 DEBUG:boto:Path: /pklxkd-57/d 2023-05-31T20:37:27.073 DEBUG:boto:Data: 2023-05-31T20:37:27.074 DEBUG:boto:Headers: {} 2023-05-31T20:37:27.074 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:37:27.074 DEBUG:boto:Port: 8001 2023-05-31T20:37:27.075 DEBUG:boto:Params: {} 2023-05-31T20:37:27.075 DEBUG:boto:Token: None 2023-05-31T20:37:27.075 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:37:27 GMT /pklxkd-57/d 2023-05-31T20:37:27.076 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:jOBEU22byDvoVJXMk/l40ZgPhPU= 2023-05-31T20:37:27.076 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:37:27 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:jOBEU22byDvoVJXMk/l40ZgPhPU=', 'Content-Length': '0'} 2023-05-31T20:37:27.078 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:35:04 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001d516a71a3017aad-006477b007-1b4f-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:37:27 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:37:27.079 DEBUG:boto:path=/pklxkd-57/d 2023-05-31T20:37:27.080 DEBUG:boto:auth_path=/pklxkd-57/d 2023-05-31T20:37:27.080 DEBUG:boto:Method: HEAD 2023-05-31T20:37:27.080 DEBUG:boto:Path: /pklxkd-57/d 2023-05-31T20:37:27.081 DEBUG:boto:Data: 2023-05-31T20:37:27.081 DEBUG:boto:Headers: {} 2023-05-31T20:37:27.081 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:37:27.082 DEBUG:boto:Port: 8000 2023-05-31T20:37:27.082 DEBUG:boto:Params: {} 2023-05-31T20:37:27.082 DEBUG:boto:Token: None 2023-05-31T20:37:27.083 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:37:27 GMT /pklxkd-57/d 2023-05-31T20:37:27.083 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:jOBEU22byDvoVJXMk/l40ZgPhPU= 2023-05-31T20:37:27.084 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:37:27 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:jOBEU22byDvoVJXMk/l40ZgPhPU=', 'Content-Length': '0'} 2023-05-31T20:37:27.086 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:35:04 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000673ffbf7166fec8c-006477b007-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:37:27 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:37:27.087 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=d 2023-05-31T20:37:27.087 DEBUG:boto:path=/pklxkd-57/d 2023-05-31T20:37:27.087 DEBUG:boto:auth_path=/pklxkd-57/d 2023-05-31T20:37:27.088 DEBUG:boto:Method: GET 2023-05-31T20:37:27.088 DEBUG:boto:Path: /pklxkd-57/d 2023-05-31T20:37:27.089 DEBUG:boto:Data: 2023-05-31T20:37:27.089 DEBUG:boto:Headers: {} 2023-05-31T20:37:27.089 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:37:27.090 DEBUG:boto:Port: 8001 2023-05-31T20:37:27.090 DEBUG:boto:Params: {} 2023-05-31T20:37:27.090 DEBUG:boto:Token: None 2023-05-31T20:37:27.091 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:37:27 GMT /pklxkd-57/d 2023-05-31T20:37:27.091 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:Bz9TwKVgWnuO/qG5DAqbfKqxjxE= 2023-05-31T20:37:27.092 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:37:27 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:Bz9TwKVgWnuO/qG5DAqbfKqxjxE=', 'Content-Length': '0'} 2023-05-31T20:37:27.094 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:35:04 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000ea49accf6cbf6ed-006477b007-1b4f-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:37:27 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:37:27.137 DEBUG:boto:path=/pklxkd-57/d 2023-05-31T20:37:27.137 DEBUG:boto:auth_path=/pklxkd-57/d 2023-05-31T20:37:27.138 DEBUG:boto:Method: GET 2023-05-31T20:37:27.139 DEBUG:boto:Path: /pklxkd-57/d 2023-05-31T20:37:27.139 DEBUG:boto:Data: 2023-05-31T20:37:27.140 DEBUG:boto:Headers: {} 2023-05-31T20:37:27.140 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:37:27.141 DEBUG:boto:Port: 8000 2023-05-31T20:37:27.141 DEBUG:boto:Params: {} 2023-05-31T20:37:27.142 DEBUG:boto:Token: None 2023-05-31T20:37:27.142 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:37:27 GMT /pklxkd-57/d 2023-05-31T20:37:27.143 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:Bz9TwKVgWnuO/qG5DAqbfKqxjxE= 2023-05-31T20:37:27.143 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:37:27 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:Bz9TwKVgWnuO/qG5DAqbfKqxjxE=', 'Content-Length': '0'} 2023-05-31T20:37:27.145 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:35:04 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000590ff2a5e7c877ef-006477b007-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:37:27 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:37:27.146 DEBUG:boto:path=/pklxkd-57/ 2023-05-31T20:37:27.147 DEBUG:boto:auth_path=/pklxkd-57/ 2023-05-31T20:37:27.147 DEBUG:boto:path=/pklxkd-57/?prefix=d 2023-05-31T20:37:27.148 DEBUG:boto:auth_path=/pklxkd-57/?prefix=d 2023-05-31T20:37:27.148 DEBUG:boto:Method: GET 2023-05-31T20:37:27.149 DEBUG:boto:Path: /pklxkd-57/?prefix=d 2023-05-31T20:37:27.149 DEBUG:boto:Data: 2023-05-31T20:37:27.150 DEBUG:boto:Headers: {} 2023-05-31T20:37:27.150 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:37:27.151 DEBUG:boto:Port: 8001 2023-05-31T20:37:27.151 DEBUG:boto:Params: {} 2023-05-31T20:37:27.152 DEBUG:boto:Token: None 2023-05-31T20:37:27.152 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:37:27 GMT /pklxkd-57/ 2023-05-31T20:37:27.153 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:kR8XV0FF++frnB9KbRgaOASXdNM= 2023-05-31T20:37:27.153 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:37:27 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:kR8XV0FF++frnB9KbRgaOASXdNM=', 'Content-Length': '0'} 2023-05-31T20:37:27.156 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000000337a9b4e9eeb09f-006477b007-1b4f-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:37:27 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:37:27.156 DEBUG:boto:b'pklxkd-57d1000falsed2023-05-31T20:35:04.680Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:37:27.157 DEBUG:boto:path=/pklxkd-57/ 2023-05-31T20:37:27.158 DEBUG:boto:auth_path=/pklxkd-57/ 2023-05-31T20:37:27.158 DEBUG:boto:path=/pklxkd-57/?prefix=d 2023-05-31T20:37:27.158 DEBUG:boto:auth_path=/pklxkd-57/?prefix=d 2023-05-31T20:37:27.159 DEBUG:boto:Method: GET 2023-05-31T20:37:27.159 DEBUG:boto:Path: /pklxkd-57/?prefix=d 2023-05-31T20:37:27.160 DEBUG:boto:Data: 2023-05-31T20:37:27.160 DEBUG:boto:Headers: {} 2023-05-31T20:37:27.160 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:37:27.161 DEBUG:boto:Port: 8000 2023-05-31T20:37:27.161 DEBUG:boto:Params: {} 2023-05-31T20:37:27.162 DEBUG:boto:Token: None 2023-05-31T20:37:27.162 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:37:27 GMT /pklxkd-57/ 2023-05-31T20:37:27.163 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:kR8XV0FF++frnB9KbRgaOASXdNM= 2023-05-31T20:37:27.163 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:37:27 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:kR8XV0FF++frnB9KbRgaOASXdNM=', 'Content-Length': '0'} 2023-05-31T20:37:27.165 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000037b0c1af8c3587a6-006477b007-1630-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:37:27 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:37:27.166 DEBUG:boto:b'pklxkd-57d1000falsed2023-05-31T20:35:04.680Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:37:27.167 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=e 2023-05-31T20:37:27.167 DEBUG:boto:path=/pklxkd-57/e 2023-05-31T20:37:27.168 DEBUG:boto:auth_path=/pklxkd-57/e 2023-05-31T20:37:27.169 DEBUG:boto:path=/pklxkd-57/e?versionId=null 2023-05-31T20:37:27.169 DEBUG:boto:auth_path=/pklxkd-57/e?versionId=null 2023-05-31T20:37:27.170 DEBUG:boto:Method: GET 2023-05-31T20:37:27.170 DEBUG:boto:Path: /pklxkd-57/e?versionId=null 2023-05-31T20:37:27.171 DEBUG:boto:Data: 2023-05-31T20:37:27.171 DEBUG:boto:Headers: {} 2023-05-31T20:37:27.171 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:37:27.172 DEBUG:boto:Port: 8001 2023-05-31T20:37:27.172 DEBUG:boto:Params: {} 2023-05-31T20:37:27.173 DEBUG:boto:Token: None 2023-05-31T20:37:27.174 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:37:27 GMT /pklxkd-57/e?versionId=null 2023-05-31T20:37:27.175 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:NBGj3dJd7KNYNDoh1nKDLZsnhDw= 2023-05-31T20:37:27.175 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:37:27 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:NBGj3dJd7KNYNDoh1nKDLZsnhDw=', 'Content-Length': '0'} 2023-05-31T20:37:27.178 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:35:04 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000069ad044395337aa6-006477b007-1b4f-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:37:27 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:37:27.179 DEBUG:boto:path=/pklxkd-57/e 2023-05-31T20:37:27.180 DEBUG:boto:auth_path=/pklxkd-57/e 2023-05-31T20:37:27.182 DEBUG:boto:path=/pklxkd-57/e?versionId=null 2023-05-31T20:37:27.182 DEBUG:boto:auth_path=/pklxkd-57/e?versionId=null 2023-05-31T20:37:27.182 DEBUG:boto:Method: GET 2023-05-31T20:37:27.183 DEBUG:boto:Path: /pklxkd-57/e?versionId=null 2023-05-31T20:37:27.183 DEBUG:boto:Data: 2023-05-31T20:37:27.183 DEBUG:boto:Headers: {} 2023-05-31T20:37:27.184 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:37:27.184 DEBUG:boto:Port: 8000 2023-05-31T20:37:27.184 DEBUG:boto:Params: {} 2023-05-31T20:37:27.185 DEBUG:boto:Token: None 2023-05-31T20:37:27.185 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:37:27 GMT /pklxkd-57/e?versionId=null 2023-05-31T20:37:27.186 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:NBGj3dJd7KNYNDoh1nKDLZsnhDw= 2023-05-31T20:37:27.186 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:37:27 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:NBGj3dJd7KNYNDoh1nKDLZsnhDw=', 'Content-Length': '0'} 2023-05-31T20:37:27.188 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:35:04 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e3b95c3c2dcb1910-006477b007-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:37:27 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:37:27.236 DEBUG:boto:path=/pklxkd-57/e 2023-05-31T20:37:27.239 DEBUG:boto:auth_path=/pklxkd-57/e 2023-05-31T20:37:27.239 DEBUG:boto:path=/pklxkd-57/e?versionId=null 2023-05-31T20:37:27.240 DEBUG:boto:auth_path=/pklxkd-57/e?versionId=null 2023-05-31T20:37:27.240 DEBUG:boto:Method: HEAD 2023-05-31T20:37:27.241 DEBUG:boto:Path: /pklxkd-57/e?versionId=null 2023-05-31T20:37:27.242 DEBUG:boto:Data: 2023-05-31T20:37:27.242 DEBUG:boto:Headers: {} 2023-05-31T20:37:27.242 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:37:27.243 DEBUG:boto:Port: 8001 2023-05-31T20:37:27.243 DEBUG:boto:Params: {} 2023-05-31T20:37:27.245 DEBUG:boto:Token: None 2023-05-31T20:37:27.245 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:37:27 GMT /pklxkd-57/e?versionId=null 2023-05-31T20:37:27.246 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:kztEEmR769kxyv73+PSMXn4AP5E= 2023-05-31T20:37:27.246 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:37:27 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:kztEEmR769kxyv73+PSMXn4AP5E=', 'Content-Length': '0'} 2023-05-31T20:37:27.252 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:35:04 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000886cfa9933bf12b4-006477b007-1b4f-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:37:27 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:37:27.252 DEBUG:boto:path=/pklxkd-57/e 2023-05-31T20:37:27.253 DEBUG:boto:auth_path=/pklxkd-57/e 2023-05-31T20:37:27.253 DEBUG:boto:path=/pklxkd-57/e?versionId=null 2023-05-31T20:37:27.255 DEBUG:boto:auth_path=/pklxkd-57/e?versionId=null 2023-05-31T20:37:27.256 DEBUG:boto:Method: HEAD 2023-05-31T20:37:27.257 DEBUG:boto:Path: /pklxkd-57/e?versionId=null 2023-05-31T20:37:27.257 DEBUG:boto:Data: 2023-05-31T20:37:27.257 DEBUG:boto:Headers: {} 2023-05-31T20:37:27.258 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:37:27.258 DEBUG:boto:Port: 8000 2023-05-31T20:37:27.258 DEBUG:boto:Params: {} 2023-05-31T20:37:27.259 DEBUG:boto:Token: None 2023-05-31T20:37:27.259 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:37:27 GMT /pklxkd-57/e?versionId=null 2023-05-31T20:37:27.260 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:kztEEmR769kxyv73+PSMXn4AP5E= 2023-05-31T20:37:27.260 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:37:27 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:kztEEmR769kxyv73+PSMXn4AP5E=', 'Content-Length': '0'} 2023-05-31T20:37:27.262 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:35:04 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a60d7238615863b3-006477b007-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:37:27 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:37:27.263 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=e 2023-05-31T20:37:27.263 DEBUG:boto:path=/pklxkd-57/e 2023-05-31T20:37:27.264 DEBUG:boto:auth_path=/pklxkd-57/e 2023-05-31T20:37:27.264 DEBUG:boto:path=/pklxkd-57/e?versionId=null 2023-05-31T20:37:27.264 DEBUG:boto:auth_path=/pklxkd-57/e?versionId=null 2023-05-31T20:37:27.265 DEBUG:boto:Method: GET 2023-05-31T20:37:27.265 DEBUG:boto:Path: /pklxkd-57/e?versionId=null 2023-05-31T20:37:27.266 DEBUG:boto:Data: 2023-05-31T20:37:27.266 DEBUG:boto:Headers: {} 2023-05-31T20:37:27.266 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:37:27.267 DEBUG:boto:Port: 8001 2023-05-31T20:37:27.267 DEBUG:boto:Params: {} 2023-05-31T20:37:27.267 DEBUG:boto:Token: None 2023-05-31T20:37:27.268 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:37:27 GMT /pklxkd-57/e?versionId=null 2023-05-31T20:37:27.268 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:NBGj3dJd7KNYNDoh1nKDLZsnhDw= 2023-05-31T20:37:27.269 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:37:27 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:NBGj3dJd7KNYNDoh1nKDLZsnhDw=', 'Content-Length': '0'} 2023-05-31T20:37:27.271 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:35:04 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000066ada0b3076dbe4e-006477b007-1b4f-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:37:27 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:37:27.273 DEBUG:boto:path=/pklxkd-57/e 2023-05-31T20:37:27.273 DEBUG:boto:auth_path=/pklxkd-57/e 2023-05-31T20:37:27.274 DEBUG:boto:path=/pklxkd-57/e?versionId=null 2023-05-31T20:37:27.275 DEBUG:boto:auth_path=/pklxkd-57/e?versionId=null 2023-05-31T20:37:27.276 DEBUG:boto:Method: GET 2023-05-31T20:37:27.276 DEBUG:boto:Path: /pklxkd-57/e?versionId=null 2023-05-31T20:37:27.277 DEBUG:boto:Data: 2023-05-31T20:37:27.278 DEBUG:boto:Headers: {} 2023-05-31T20:37:27.279 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:37:27.281 DEBUG:boto:Port: 8000 2023-05-31T20:37:27.281 DEBUG:boto:Params: {} 2023-05-31T20:37:27.282 DEBUG:boto:Token: None 2023-05-31T20:37:27.282 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:37:27 GMT /pklxkd-57/e?versionId=null 2023-05-31T20:37:27.283 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:NBGj3dJd7KNYNDoh1nKDLZsnhDw= 2023-05-31T20:37:27.283 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:37:27 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:NBGj3dJd7KNYNDoh1nKDLZsnhDw=', 'Content-Length': '0'} 2023-05-31T20:37:27.285 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:35:04 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000862f33a071126313-006477b007-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:37:27 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:37:27.286 DEBUG:boto:path=/pklxkd-57/ 2023-05-31T20:37:27.287 DEBUG:boto:auth_path=/pklxkd-57/ 2023-05-31T20:37:27.287 DEBUG:boto:path=/pklxkd-57/?prefix=e 2023-05-31T20:37:27.287 DEBUG:boto:auth_path=/pklxkd-57/?prefix=e 2023-05-31T20:37:27.288 DEBUG:boto:Method: GET 2023-05-31T20:37:27.288 DEBUG:boto:Path: /pklxkd-57/?prefix=e 2023-05-31T20:37:27.289 DEBUG:boto:Data: 2023-05-31T20:37:27.289 DEBUG:boto:Headers: {} 2023-05-31T20:37:27.289 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:37:27.290 DEBUG:boto:Port: 8001 2023-05-31T20:37:27.290 DEBUG:boto:Params: {} 2023-05-31T20:37:27.290 DEBUG:boto:Token: None 2023-05-31T20:37:27.291 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:37:27 GMT /pklxkd-57/ 2023-05-31T20:37:27.291 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:kR8XV0FF++frnB9KbRgaOASXdNM= 2023-05-31T20:37:27.292 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:37:27 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:kR8XV0FF++frnB9KbRgaOASXdNM=', 'Content-Length': '0'} 2023-05-31T20:37:27.294 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000006c300a135b0c9b5a-006477b007-1b4f-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:37:27 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:37:27.295 DEBUG:boto:b'pklxkd-57e1000falsee2023-05-31T20:35:04.906Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:37:27.295 DEBUG:boto:path=/pklxkd-57/ 2023-05-31T20:37:27.296 DEBUG:boto:auth_path=/pklxkd-57/ 2023-05-31T20:37:27.296 DEBUG:boto:path=/pklxkd-57/?prefix=e 2023-05-31T20:37:27.297 DEBUG:boto:auth_path=/pklxkd-57/?prefix=e 2023-05-31T20:37:27.297 DEBUG:boto:Method: GET 2023-05-31T20:37:27.297 DEBUG:boto:Path: /pklxkd-57/?prefix=e 2023-05-31T20:37:27.298 DEBUG:boto:Data: 2023-05-31T20:37:27.298 DEBUG:boto:Headers: {} 2023-05-31T20:37:27.299 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:37:27.299 DEBUG:boto:Port: 8000 2023-05-31T20:37:27.299 DEBUG:boto:Params: {} 2023-05-31T20:37:27.300 DEBUG:boto:Token: None 2023-05-31T20:37:27.300 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:37:27 GMT /pklxkd-57/ 2023-05-31T20:37:27.301 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:kR8XV0FF++frnB9KbRgaOASXdNM= 2023-05-31T20:37:27.301 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:37:27 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:kR8XV0FF++frnB9KbRgaOASXdNM=', 'Content-Length': '0'} 2023-05-31T20:37:27.303 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000002685ad320a455143-006477b007-1630-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:37:27 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:37:27.304 DEBUG:boto:b'pklxkd-57e1000falsee2023-05-31T20:35:04.906Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:37:27.305 DEBUG:boto:path=/pklxkd-57/e 2023-05-31T20:37:27.305 DEBUG:boto:auth_path=/pklxkd-57/e 2023-05-31T20:37:27.305 DEBUG:boto:Method: HEAD 2023-05-31T20:37:27.306 DEBUG:boto:Path: /pklxkd-57/e 2023-05-31T20:37:27.306 DEBUG:boto:Data: 2023-05-31T20:37:27.307 DEBUG:boto:Headers: {} 2023-05-31T20:37:27.307 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:37:27.307 DEBUG:boto:Port: 8001 2023-05-31T20:37:27.308 DEBUG:boto:Params: {} 2023-05-31T20:37:27.308 DEBUG:boto:Token: None 2023-05-31T20:37:27.308 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:37:27 GMT /pklxkd-57/e 2023-05-31T20:37:27.309 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:mfSQ9GIKpDR56ZZCefA8JGc39Lc= 2023-05-31T20:37:27.309 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:37:27 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:mfSQ9GIKpDR56ZZCefA8JGc39Lc=', 'Content-Length': '0'} 2023-05-31T20:37:27.311 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:35:04 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e475b56145dacd83-006477b007-1b4f-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:37:27 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:37:27.312 DEBUG:boto:path=/pklxkd-57/e 2023-05-31T20:37:27.312 DEBUG:boto:auth_path=/pklxkd-57/e 2023-05-31T20:37:27.313 DEBUG:boto:Method: HEAD 2023-05-31T20:37:27.313 DEBUG:boto:Path: /pklxkd-57/e 2023-05-31T20:37:27.313 DEBUG:boto:Data: 2023-05-31T20:37:27.314 DEBUG:boto:Headers: {} 2023-05-31T20:37:27.314 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:37:27.314 DEBUG:boto:Port: 8000 2023-05-31T20:37:27.315 DEBUG:boto:Params: {} 2023-05-31T20:37:27.315 DEBUG:boto:Token: None 2023-05-31T20:37:27.315 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:37:27 GMT /pklxkd-57/e 2023-05-31T20:37:27.316 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:mfSQ9GIKpDR56ZZCefA8JGc39Lc= 2023-05-31T20:37:27.316 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:37:27 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:mfSQ9GIKpDR56ZZCefA8JGc39Lc=', 'Content-Length': '0'} 2023-05-31T20:37:27.318 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:35:04 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009f56e5e106c346ab-006477b007-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:37:27 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:37:27.319 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=e 2023-05-31T20:37:27.319 DEBUG:boto:path=/pklxkd-57/e 2023-05-31T20:37:27.320 DEBUG:boto:auth_path=/pklxkd-57/e 2023-05-31T20:37:27.320 DEBUG:boto:Method: GET 2023-05-31T20:37:27.321 DEBUG:boto:Path: /pklxkd-57/e 2023-05-31T20:37:27.321 DEBUG:boto:Data: 2023-05-31T20:37:27.321 DEBUG:boto:Headers: {} 2023-05-31T20:37:27.322 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:37:27.322 DEBUG:boto:Port: 8001 2023-05-31T20:37:27.322 DEBUG:boto:Params: {} 2023-05-31T20:37:27.323 DEBUG:boto:Token: None 2023-05-31T20:37:27.323 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:37:27 GMT /pklxkd-57/e 2023-05-31T20:37:27.324 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:8Q4v7K3mehQNFYTrzhyZ4bueHWs= 2023-05-31T20:37:27.324 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:37:27 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:8Q4v7K3mehQNFYTrzhyZ4bueHWs=', 'Content-Length': '0'} 2023-05-31T20:37:27.326 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:35:04 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000cbc9eb68bbd7e73a-006477b007-1b4f-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:37:27 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:37:27.327 DEBUG:boto:path=/pklxkd-57/e 2023-05-31T20:37:27.328 DEBUG:boto:auth_path=/pklxkd-57/e 2023-05-31T20:37:27.329 DEBUG:boto:Method: GET 2023-05-31T20:37:27.329 DEBUG:boto:Path: /pklxkd-57/e 2023-05-31T20:37:27.330 DEBUG:boto:Data: 2023-05-31T20:37:27.330 DEBUG:boto:Headers: {} 2023-05-31T20:37:27.331 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:37:27.331 DEBUG:boto:Port: 8000 2023-05-31T20:37:27.332 DEBUG:boto:Params: {} 2023-05-31T20:37:27.332 DEBUG:boto:Token: None 2023-05-31T20:37:27.333 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:37:27 GMT /pklxkd-57/e 2023-05-31T20:37:27.334 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:8Q4v7K3mehQNFYTrzhyZ4bueHWs= 2023-05-31T20:37:27.335 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:37:27 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:8Q4v7K3mehQNFYTrzhyZ4bueHWs=', 'Content-Length': '0'} 2023-05-31T20:37:27.337 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:35:04 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000077b0c3ba8d42c66c-006477b007-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:37:27 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:37:27.338 DEBUG:boto:path=/pklxkd-57/ 2023-05-31T20:37:27.338 DEBUG:boto:auth_path=/pklxkd-57/ 2023-05-31T20:37:27.339 DEBUG:boto:path=/pklxkd-57/?prefix=e 2023-05-31T20:37:27.339 DEBUG:boto:auth_path=/pklxkd-57/?prefix=e 2023-05-31T20:37:27.339 DEBUG:boto:Method: GET 2023-05-31T20:37:27.340 DEBUG:boto:Path: /pklxkd-57/?prefix=e 2023-05-31T20:37:27.340 DEBUG:boto:Data: 2023-05-31T20:37:27.340 DEBUG:boto:Headers: {} 2023-05-31T20:37:27.341 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:37:27.341 DEBUG:boto:Port: 8001 2023-05-31T20:37:27.341 DEBUG:boto:Params: {} 2023-05-31T20:37:27.342 DEBUG:boto:Token: None 2023-05-31T20:37:27.342 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:37:27 GMT /pklxkd-57/ 2023-05-31T20:37:27.343 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:kR8XV0FF++frnB9KbRgaOASXdNM= 2023-05-31T20:37:27.343 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:37:27 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:kR8XV0FF++frnB9KbRgaOASXdNM=', 'Content-Length': '0'} 2023-05-31T20:37:27.346 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000005ed8066ef5be10f7-006477b007-1b4f-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:37:27 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:37:27.347 DEBUG:boto:b'pklxkd-57e1000falsee2023-05-31T20:35:04.906Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:37:27.348 DEBUG:boto:path=/pklxkd-57/ 2023-05-31T20:37:27.348 DEBUG:boto:auth_path=/pklxkd-57/ 2023-05-31T20:37:27.349 DEBUG:boto:path=/pklxkd-57/?prefix=e 2023-05-31T20:37:27.350 DEBUG:boto:auth_path=/pklxkd-57/?prefix=e 2023-05-31T20:37:27.351 DEBUG:boto:Method: GET 2023-05-31T20:37:27.351 DEBUG:boto:Path: /pklxkd-57/?prefix=e 2023-05-31T20:37:27.352 DEBUG:boto:Data: 2023-05-31T20:37:27.352 DEBUG:boto:Headers: {} 2023-05-31T20:37:27.353 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:37:27.353 DEBUG:boto:Port: 8000 2023-05-31T20:37:27.354 DEBUG:boto:Params: {} 2023-05-31T20:37:27.355 DEBUG:boto:Token: None 2023-05-31T20:37:27.355 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:37:27 GMT /pklxkd-57/ 2023-05-31T20:37:27.356 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:kR8XV0FF++frnB9KbRgaOASXdNM= 2023-05-31T20:37:27.357 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:37:27 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:kR8XV0FF++frnB9KbRgaOASXdNM=', 'Content-Length': '0'} 2023-05-31T20:37:27.359 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ed59848aeab736c6-006477b007-1630-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:37:27 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:37:27.400 DEBUG:boto:b'pklxkd-57e1000falsee2023-05-31T20:35:04.906Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:37:27.401 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=f 2023-05-31T20:37:27.402 DEBUG:boto:path=/pklxkd-57/f 2023-05-31T20:37:27.402 DEBUG:boto:auth_path=/pklxkd-57/f 2023-05-31T20:37:27.403 DEBUG:boto:path=/pklxkd-57/f?versionId=null 2023-05-31T20:37:27.403 DEBUG:boto:auth_path=/pklxkd-57/f?versionId=null 2023-05-31T20:37:27.403 DEBUG:boto:Method: GET 2023-05-31T20:37:27.404 DEBUG:boto:Path: /pklxkd-57/f?versionId=null 2023-05-31T20:37:27.404 DEBUG:boto:Data: 2023-05-31T20:37:27.405 DEBUG:boto:Headers: {} 2023-05-31T20:37:27.405 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:37:27.405 DEBUG:boto:Port: 8001 2023-05-31T20:37:27.406 DEBUG:boto:Params: {} 2023-05-31T20:37:27.406 DEBUG:boto:Token: None 2023-05-31T20:37:27.407 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:37:27 GMT /pklxkd-57/f?versionId=null 2023-05-31T20:37:27.407 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:taZQMODHsXn9wOTrgV5vK0T5PVg= 2023-05-31T20:37:27.408 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:37:27 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:taZQMODHsXn9wOTrgV5vK0T5PVg=', 'Content-Length': '0'} 2023-05-31T20:37:27.410 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:35:04 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e4b926d45c677206-006477b007-1b4f-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:37:27 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:37:27.411 DEBUG:boto:path=/pklxkd-57/f 2023-05-31T20:37:27.411 DEBUG:boto:auth_path=/pklxkd-57/f 2023-05-31T20:37:27.411 DEBUG:boto:path=/pklxkd-57/f?versionId=null 2023-05-31T20:37:27.412 DEBUG:boto:auth_path=/pklxkd-57/f?versionId=null 2023-05-31T20:37:27.412 DEBUG:boto:Method: GET 2023-05-31T20:37:27.413 DEBUG:boto:Path: /pklxkd-57/f?versionId=null 2023-05-31T20:37:27.413 DEBUG:boto:Data: 2023-05-31T20:37:27.413 DEBUG:boto:Headers: {} 2023-05-31T20:37:27.414 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:37:27.414 DEBUG:boto:Port: 8000 2023-05-31T20:37:27.415 DEBUG:boto:Params: {} 2023-05-31T20:37:27.415 DEBUG:boto:Token: None 2023-05-31T20:37:27.415 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:37:27 GMT /pklxkd-57/f?versionId=null 2023-05-31T20:37:27.416 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:taZQMODHsXn9wOTrgV5vK0T5PVg= 2023-05-31T20:37:27.416 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:37:27 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:taZQMODHsXn9wOTrgV5vK0T5PVg=', 'Content-Length': '0'} 2023-05-31T20:37:27.418 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:35:04 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000edb0826407c2a2d9-006477b007-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:37:27 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:37:27.419 DEBUG:boto:path=/pklxkd-57/f 2023-05-31T20:37:27.420 DEBUG:boto:auth_path=/pklxkd-57/f 2023-05-31T20:37:27.420 DEBUG:boto:path=/pklxkd-57/f?versionId=null 2023-05-31T20:37:27.420 DEBUG:boto:auth_path=/pklxkd-57/f?versionId=null 2023-05-31T20:37:27.421 DEBUG:boto:Method: HEAD 2023-05-31T20:37:27.421 DEBUG:boto:Path: /pklxkd-57/f?versionId=null 2023-05-31T20:37:27.421 DEBUG:boto:Data: 2023-05-31T20:37:27.422 DEBUG:boto:Headers: {} 2023-05-31T20:37:27.422 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:37:27.422 DEBUG:boto:Port: 8001 2023-05-31T20:37:27.423 DEBUG:boto:Params: {} 2023-05-31T20:37:27.423 DEBUG:boto:Token: None 2023-05-31T20:37:27.424 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:37:27 GMT /pklxkd-57/f?versionId=null 2023-05-31T20:37:27.424 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:m1JONcMo20uRAcwy4kCBiv3BWIc= 2023-05-31T20:37:27.426 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:37:27 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:m1JONcMo20uRAcwy4kCBiv3BWIc=', 'Content-Length': '0'} 2023-05-31T20:37:27.428 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:35:04 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000082f1271d40bb773c-006477b007-1b4f-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:37:27 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:37:27.429 DEBUG:boto:path=/pklxkd-57/f 2023-05-31T20:37:27.430 DEBUG:boto:auth_path=/pklxkd-57/f 2023-05-31T20:37:27.430 DEBUG:boto:path=/pklxkd-57/f?versionId=null 2023-05-31T20:37:27.430 DEBUG:boto:auth_path=/pklxkd-57/f?versionId=null 2023-05-31T20:37:27.431 DEBUG:boto:Method: HEAD 2023-05-31T20:37:27.431 DEBUG:boto:Path: /pklxkd-57/f?versionId=null 2023-05-31T20:37:27.432 DEBUG:boto:Data: 2023-05-31T20:37:27.432 DEBUG:boto:Headers: {} 2023-05-31T20:37:27.432 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:37:27.433 DEBUG:boto:Port: 8000 2023-05-31T20:37:27.434 DEBUG:boto:Params: {} 2023-05-31T20:37:27.436 DEBUG:boto:Token: None 2023-05-31T20:37:27.437 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:37:27 GMT /pklxkd-57/f?versionId=null 2023-05-31T20:37:27.437 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:m1JONcMo20uRAcwy4kCBiv3BWIc= 2023-05-31T20:37:27.438 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:37:27 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:m1JONcMo20uRAcwy4kCBiv3BWIc=', 'Content-Length': '0'} 2023-05-31T20:37:27.440 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:35:04 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d9e606dbb203ff97-006477b007-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:37:27 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:37:27.441 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=f 2023-05-31T20:37:27.442 DEBUG:boto:path=/pklxkd-57/f 2023-05-31T20:37:27.442 DEBUG:boto:auth_path=/pklxkd-57/f 2023-05-31T20:37:27.442 DEBUG:boto:path=/pklxkd-57/f?versionId=null 2023-05-31T20:37:27.443 DEBUG:boto:auth_path=/pklxkd-57/f?versionId=null 2023-05-31T20:37:27.445 DEBUG:boto:Method: GET 2023-05-31T20:37:27.446 DEBUG:boto:Path: /pklxkd-57/f?versionId=null 2023-05-31T20:37:27.446 DEBUG:boto:Data: 2023-05-31T20:37:27.447 DEBUG:boto:Headers: {} 2023-05-31T20:37:27.447 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:37:27.447 DEBUG:boto:Port: 8001 2023-05-31T20:37:27.448 DEBUG:boto:Params: {} 2023-05-31T20:37:27.448 DEBUG:boto:Token: None 2023-05-31T20:37:27.449 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:37:27 GMT /pklxkd-57/f?versionId=null 2023-05-31T20:37:27.449 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:taZQMODHsXn9wOTrgV5vK0T5PVg= 2023-05-31T20:37:27.450 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:37:27 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:taZQMODHsXn9wOTrgV5vK0T5PVg=', 'Content-Length': '0'} 2023-05-31T20:37:27.452 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:35:04 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e8d9538c04968b72-006477b007-1b4f-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:37:27 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:37:27.453 DEBUG:boto:path=/pklxkd-57/f 2023-05-31T20:37:27.454 DEBUG:boto:auth_path=/pklxkd-57/f 2023-05-31T20:37:27.454 DEBUG:boto:path=/pklxkd-57/f?versionId=null 2023-05-31T20:37:27.454 DEBUG:boto:auth_path=/pklxkd-57/f?versionId=null 2023-05-31T20:37:27.455 DEBUG:boto:Method: GET 2023-05-31T20:37:27.456 DEBUG:boto:Path: /pklxkd-57/f?versionId=null 2023-05-31T20:37:27.456 DEBUG:boto:Data: 2023-05-31T20:37:27.457 DEBUG:boto:Headers: {} 2023-05-31T20:37:27.457 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:37:27.457 DEBUG:boto:Port: 8000 2023-05-31T20:37:27.458 DEBUG:boto:Params: {} 2023-05-31T20:37:27.458 DEBUG:boto:Token: None 2023-05-31T20:37:27.459 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:37:27 GMT /pklxkd-57/f?versionId=null 2023-05-31T20:37:27.459 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:taZQMODHsXn9wOTrgV5vK0T5PVg= 2023-05-31T20:37:27.460 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:37:27 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:taZQMODHsXn9wOTrgV5vK0T5PVg=', 'Content-Length': '0'} 2023-05-31T20:37:27.461 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:35:04 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001545f5fd731a2b3f-006477b007-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:37:27 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:37:27.463 DEBUG:boto:path=/pklxkd-57/ 2023-05-31T20:37:27.464 DEBUG:boto:auth_path=/pklxkd-57/ 2023-05-31T20:37:27.464 DEBUG:boto:path=/pklxkd-57/?prefix=f 2023-05-31T20:37:27.465 DEBUG:boto:auth_path=/pklxkd-57/?prefix=f 2023-05-31T20:37:27.466 DEBUG:boto:Method: GET 2023-05-31T20:37:27.466 DEBUG:boto:Path: /pklxkd-57/?prefix=f 2023-05-31T20:37:27.467 DEBUG:boto:Data: 2023-05-31T20:37:27.467 DEBUG:boto:Headers: {} 2023-05-31T20:37:27.468 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:37:27.469 DEBUG:boto:Port: 8001 2023-05-31T20:37:27.469 DEBUG:boto:Params: {} 2023-05-31T20:37:27.470 DEBUG:boto:Token: None 2023-05-31T20:37:27.471 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:37:27 GMT /pklxkd-57/ 2023-05-31T20:37:27.472 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:kR8XV0FF++frnB9KbRgaOASXdNM= 2023-05-31T20:37:27.473 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:37:27 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:kR8XV0FF++frnB9KbRgaOASXdNM=', 'Content-Length': '0'} 2023-05-31T20:37:27.475 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d7c0f00118995115-006477b007-1b4f-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:37:27 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:37:27.520 DEBUG:boto:b'pklxkd-57f1000falsef2023-05-31T20:35:04.925Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:37:27.522 DEBUG:boto:path=/pklxkd-57/ 2023-05-31T20:37:27.522 DEBUG:boto:auth_path=/pklxkd-57/ 2023-05-31T20:37:27.523 DEBUG:boto:path=/pklxkd-57/?prefix=f 2023-05-31T20:37:27.523 DEBUG:boto:auth_path=/pklxkd-57/?prefix=f 2023-05-31T20:37:27.523 DEBUG:boto:Method: GET 2023-05-31T20:37:27.524 DEBUG:boto:Path: /pklxkd-57/?prefix=f 2023-05-31T20:37:27.524 DEBUG:boto:Data: 2023-05-31T20:37:27.525 DEBUG:boto:Headers: {} 2023-05-31T20:37:27.525 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:37:27.526 DEBUG:boto:Port: 8000 2023-05-31T20:37:27.526 DEBUG:boto:Params: {} 2023-05-31T20:37:27.527 DEBUG:boto:Token: None 2023-05-31T20:37:27.527 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:37:27 GMT /pklxkd-57/ 2023-05-31T20:37:27.528 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:kR8XV0FF++frnB9KbRgaOASXdNM= 2023-05-31T20:37:27.529 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:37:27 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:kR8XV0FF++frnB9KbRgaOASXdNM=', 'Content-Length': '0'} 2023-05-31T20:37:27.531 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000098a1e268f5a99574-006477b007-1630-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:37:27 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:37:27.532 DEBUG:boto:b'pklxkd-57f1000falsef2023-05-31T20:35:04.925Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:37:27.533 DEBUG:boto:path=/pklxkd-57/f 2023-05-31T20:37:27.533 DEBUG:boto:auth_path=/pklxkd-57/f 2023-05-31T20:37:27.533 DEBUG:boto:Method: HEAD 2023-05-31T20:37:27.534 DEBUG:boto:Path: /pklxkd-57/f 2023-05-31T20:37:27.535 DEBUG:boto:Data: 2023-05-31T20:37:27.535 DEBUG:boto:Headers: {} 2023-05-31T20:37:27.535 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:37:27.536 DEBUG:boto:Port: 8001 2023-05-31T20:37:27.536 DEBUG:boto:Params: {} 2023-05-31T20:37:27.537 DEBUG:boto:Token: None 2023-05-31T20:37:27.538 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:37:27 GMT /pklxkd-57/f 2023-05-31T20:37:27.538 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:pg6AwhotqyNsRd1D5In5J8Pu38U= 2023-05-31T20:37:27.539 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:37:27 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:pg6AwhotqyNsRd1D5In5J8Pu38U=', 'Content-Length': '0'} 2023-05-31T20:37:27.541 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:35:04 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000af31608868ae81d9-006477b007-1b4f-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:37:27 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:37:27.542 DEBUG:boto:path=/pklxkd-57/f 2023-05-31T20:37:27.543 DEBUG:boto:auth_path=/pklxkd-57/f 2023-05-31T20:37:27.543 DEBUG:boto:Method: HEAD 2023-05-31T20:37:27.544 DEBUG:boto:Path: /pklxkd-57/f 2023-05-31T20:37:27.544 DEBUG:boto:Data: 2023-05-31T20:37:27.545 DEBUG:boto:Headers: {} 2023-05-31T20:37:27.545 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:37:27.546 DEBUG:boto:Port: 8000 2023-05-31T20:37:27.546 DEBUG:boto:Params: {} 2023-05-31T20:37:27.547 DEBUG:boto:Token: None 2023-05-31T20:37:27.547 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:37:27 GMT /pklxkd-57/f 2023-05-31T20:37:27.548 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:pg6AwhotqyNsRd1D5In5J8Pu38U= 2023-05-31T20:37:27.548 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:37:27 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:pg6AwhotqyNsRd1D5In5J8Pu38U=', 'Content-Length': '0'} 2023-05-31T20:37:27.550 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:35:04 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c1039f12257587ac-006477b007-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:37:27 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:37:27.551 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=f 2023-05-31T20:37:27.551 DEBUG:boto:path=/pklxkd-57/f 2023-05-31T20:37:27.552 DEBUG:boto:auth_path=/pklxkd-57/f 2023-05-31T20:37:27.552 DEBUG:boto:Method: GET 2023-05-31T20:37:27.553 DEBUG:boto:Path: /pklxkd-57/f 2023-05-31T20:37:27.553 DEBUG:boto:Data: 2023-05-31T20:37:27.553 DEBUG:boto:Headers: {} 2023-05-31T20:37:27.554 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:37:27.554 DEBUG:boto:Port: 8001 2023-05-31T20:37:27.554 DEBUG:boto:Params: {} 2023-05-31T20:37:27.555 DEBUG:boto:Token: None 2023-05-31T20:37:27.555 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:37:27 GMT /pklxkd-57/f 2023-05-31T20:37:27.556 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:9k6vVZqQa+ABD0IGAi0TavGS3uE= 2023-05-31T20:37:27.556 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:37:27 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:9k6vVZqQa+ABD0IGAi0TavGS3uE=', 'Content-Length': '0'} 2023-05-31T20:37:27.558 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:35:04 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009bc5a6172b22749e-006477b007-1b4f-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:37:27 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:37:27.559 DEBUG:boto:path=/pklxkd-57/f 2023-05-31T20:37:27.560 DEBUG:boto:auth_path=/pklxkd-57/f 2023-05-31T20:37:27.560 DEBUG:boto:Method: GET 2023-05-31T20:37:27.560 DEBUG:boto:Path: /pklxkd-57/f 2023-05-31T20:37:27.561 DEBUG:boto:Data: 2023-05-31T20:37:27.561 DEBUG:boto:Headers: {} 2023-05-31T20:37:27.561 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:37:27.562 DEBUG:boto:Port: 8000 2023-05-31T20:37:27.562 DEBUG:boto:Params: {} 2023-05-31T20:37:27.563 DEBUG:boto:Token: None 2023-05-31T20:37:27.563 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:37:27 GMT /pklxkd-57/f 2023-05-31T20:37:27.564 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:9k6vVZqQa+ABD0IGAi0TavGS3uE= 2023-05-31T20:37:27.564 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:37:27 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:9k6vVZqQa+ABD0IGAi0TavGS3uE=', 'Content-Length': '0'} 2023-05-31T20:37:27.567 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:35:04 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f661e623d114c144-006477b007-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:37:27 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:37:27.568 DEBUG:boto:path=/pklxkd-57/ 2023-05-31T20:37:27.569 DEBUG:boto:auth_path=/pklxkd-57/ 2023-05-31T20:37:27.570 DEBUG:boto:path=/pklxkd-57/?prefix=f 2023-05-31T20:37:27.571 DEBUG:boto:auth_path=/pklxkd-57/?prefix=f 2023-05-31T20:37:27.571 DEBUG:boto:Method: GET 2023-05-31T20:37:27.572 DEBUG:boto:Path: /pklxkd-57/?prefix=f 2023-05-31T20:37:27.572 DEBUG:boto:Data: 2023-05-31T20:37:27.573 DEBUG:boto:Headers: {} 2023-05-31T20:37:27.573 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:37:27.574 DEBUG:boto:Port: 8001 2023-05-31T20:37:27.574 DEBUG:boto:Params: {} 2023-05-31T20:37:27.575 DEBUG:boto:Token: None 2023-05-31T20:37:27.576 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:37:27 GMT /pklxkd-57/ 2023-05-31T20:37:27.577 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:kR8XV0FF++frnB9KbRgaOASXdNM= 2023-05-31T20:37:27.578 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:37:27 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:kR8XV0FF++frnB9KbRgaOASXdNM=', 'Content-Length': '0'} 2023-05-31T20:37:27.581 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000da4aec0170706d55-006477b007-1b4f-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:37:27 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:37:27.581 DEBUG:boto:b'pklxkd-57f1000falsef2023-05-31T20:35:04.925Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:37:27.582 DEBUG:boto:path=/pklxkd-57/ 2023-05-31T20:37:27.583 DEBUG:boto:auth_path=/pklxkd-57/ 2023-05-31T20:37:27.583 DEBUG:boto:path=/pklxkd-57/?prefix=f 2023-05-31T20:37:27.584 DEBUG:boto:auth_path=/pklxkd-57/?prefix=f 2023-05-31T20:37:27.584 DEBUG:boto:Method: GET 2023-05-31T20:37:27.585 DEBUG:boto:Path: /pklxkd-57/?prefix=f 2023-05-31T20:37:27.585 DEBUG:boto:Data: 2023-05-31T20:37:27.586 DEBUG:boto:Headers: {} 2023-05-31T20:37:27.587 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:37:27.587 DEBUG:boto:Port: 8000 2023-05-31T20:37:27.588 DEBUG:boto:Params: {} 2023-05-31T20:37:27.589 DEBUG:boto:Token: None 2023-05-31T20:37:27.590 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:37:27 GMT /pklxkd-57/ 2023-05-31T20:37:27.590 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:kR8XV0FF++frnB9KbRgaOASXdNM= 2023-05-31T20:37:27.591 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:37:27 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:kR8XV0FF++frnB9KbRgaOASXdNM=', 'Content-Length': '0'} 2023-05-31T20:37:27.593 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000005fc6ccd76a4ae5a1-006477b007-1630-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:37:27 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:37:27.594 DEBUG:boto:b'pklxkd-57f1000falsef2023-05-31T20:35:04.925Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:37:27.595 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=g 2023-05-31T20:37:27.595 DEBUG:boto:path=/pklxkd-57/g 2023-05-31T20:37:27.596 DEBUG:boto:auth_path=/pklxkd-57/g 2023-05-31T20:37:27.596 DEBUG:boto:path=/pklxkd-57/g?versionId=null 2023-05-31T20:37:27.597 DEBUG:boto:auth_path=/pklxkd-57/g?versionId=null 2023-05-31T20:37:27.598 DEBUG:boto:Method: GET 2023-05-31T20:37:27.598 DEBUG:boto:Path: /pklxkd-57/g?versionId=null 2023-05-31T20:37:27.599 DEBUG:boto:Data: 2023-05-31T20:37:27.599 DEBUG:boto:Headers: {} 2023-05-31T20:37:27.599 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:37:27.600 DEBUG:boto:Port: 8001 2023-05-31T20:37:27.601 DEBUG:boto:Params: {} 2023-05-31T20:37:27.601 DEBUG:boto:Token: None 2023-05-31T20:37:27.602 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:37:27 GMT /pklxkd-57/g?versionId=null 2023-05-31T20:37:27.602 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:uE6nx6CjNDHW1i/Um9P2dLrJXnA= 2023-05-31T20:37:27.603 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:37:27 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:uE6nx6CjNDHW1i/Um9P2dLrJXnA=', 'Content-Length': '0'} 2023-05-31T20:37:27.605 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:35:04 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003899cf6f119c8070-006477b007-1b4f-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:37:27 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:37:27.648 DEBUG:boto:path=/pklxkd-57/g 2023-05-31T20:37:27.649 DEBUG:boto:auth_path=/pklxkd-57/g 2023-05-31T20:37:27.650 DEBUG:boto:path=/pklxkd-57/g?versionId=null 2023-05-31T20:37:27.650 DEBUG:boto:auth_path=/pklxkd-57/g?versionId=null 2023-05-31T20:37:27.650 DEBUG:boto:Method: GET 2023-05-31T20:37:27.651 DEBUG:boto:Path: /pklxkd-57/g?versionId=null 2023-05-31T20:37:27.651 DEBUG:boto:Data: 2023-05-31T20:37:27.652 DEBUG:boto:Headers: {} 2023-05-31T20:37:27.652 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:37:27.652 DEBUG:boto:Port: 8000 2023-05-31T20:37:27.653 DEBUG:boto:Params: {} 2023-05-31T20:37:27.654 DEBUG:boto:Token: None 2023-05-31T20:37:27.654 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:37:27 GMT /pklxkd-57/g?versionId=null 2023-05-31T20:37:27.655 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:uE6nx6CjNDHW1i/Um9P2dLrJXnA= 2023-05-31T20:37:27.655 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:37:27 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:uE6nx6CjNDHW1i/Um9P2dLrJXnA=', 'Content-Length': '0'} 2023-05-31T20:37:27.658 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:35:04 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000048fa807c4829e94b-006477b007-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:37:27 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:37:27.659 DEBUG:boto:path=/pklxkd-57/g 2023-05-31T20:37:27.659 DEBUG:boto:auth_path=/pklxkd-57/g 2023-05-31T20:37:27.660 DEBUG:boto:path=/pklxkd-57/g?versionId=null 2023-05-31T20:37:27.660 DEBUG:boto:auth_path=/pklxkd-57/g?versionId=null 2023-05-31T20:37:27.661 DEBUG:boto:Method: HEAD 2023-05-31T20:37:27.661 DEBUG:boto:Path: /pklxkd-57/g?versionId=null 2023-05-31T20:37:27.662 DEBUG:boto:Data: 2023-05-31T20:37:27.662 DEBUG:boto:Headers: {} 2023-05-31T20:37:27.662 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:37:27.663 DEBUG:boto:Port: 8001 2023-05-31T20:37:27.663 DEBUG:boto:Params: {} 2023-05-31T20:37:27.664 DEBUG:boto:Token: None 2023-05-31T20:37:27.664 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:37:27 GMT /pklxkd-57/g?versionId=null 2023-05-31T20:37:27.665 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:HGQmz1SndXJFddp4je5T0WJ1fb4= 2023-05-31T20:37:27.665 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:37:27 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:HGQmz1SndXJFddp4je5T0WJ1fb4=', 'Content-Length': '0'} 2023-05-31T20:37:27.667 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:35:04 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009993a931f445e786-006477b007-1b4f-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:37:27 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:37:27.668 DEBUG:boto:path=/pklxkd-57/g 2023-05-31T20:37:27.669 DEBUG:boto:auth_path=/pklxkd-57/g 2023-05-31T20:37:27.669 DEBUG:boto:path=/pklxkd-57/g?versionId=null 2023-05-31T20:37:27.670 DEBUG:boto:auth_path=/pklxkd-57/g?versionId=null 2023-05-31T20:37:27.670 DEBUG:boto:Method: HEAD 2023-05-31T20:37:27.671 DEBUG:boto:Path: /pklxkd-57/g?versionId=null 2023-05-31T20:37:27.671 DEBUG:boto:Data: 2023-05-31T20:37:27.672 DEBUG:boto:Headers: {} 2023-05-31T20:37:27.672 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:37:27.672 DEBUG:boto:Port: 8000 2023-05-31T20:37:27.673 DEBUG:boto:Params: {} 2023-05-31T20:37:27.673 DEBUG:boto:Token: None 2023-05-31T20:37:27.673 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:37:27 GMT /pklxkd-57/g?versionId=null 2023-05-31T20:37:27.674 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:HGQmz1SndXJFddp4je5T0WJ1fb4= 2023-05-31T20:37:27.674 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:37:27 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:HGQmz1SndXJFddp4je5T0WJ1fb4=', 'Content-Length': '0'} 2023-05-31T20:37:27.677 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:35:04 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000030c24f3106b1ac79-006477b007-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:37:27 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:37:27.677 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=g 2023-05-31T20:37:27.678 DEBUG:boto:path=/pklxkd-57/g 2023-05-31T20:37:27.678 DEBUG:boto:auth_path=/pklxkd-57/g 2023-05-31T20:37:27.679 DEBUG:boto:path=/pklxkd-57/g?versionId=null 2023-05-31T20:37:27.679 DEBUG:boto:auth_path=/pklxkd-57/g?versionId=null 2023-05-31T20:37:27.679 DEBUG:boto:Method: GET 2023-05-31T20:37:27.680 DEBUG:boto:Path: /pklxkd-57/g?versionId=null 2023-05-31T20:37:27.680 DEBUG:boto:Data: 2023-05-31T20:37:27.680 DEBUG:boto:Headers: {} 2023-05-31T20:37:27.681 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:37:27.681 DEBUG:boto:Port: 8001 2023-05-31T20:37:27.682 DEBUG:boto:Params: {} 2023-05-31T20:37:27.682 DEBUG:boto:Token: None 2023-05-31T20:37:27.682 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:37:27 GMT /pklxkd-57/g?versionId=null 2023-05-31T20:37:27.683 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:uE6nx6CjNDHW1i/Um9P2dLrJXnA= 2023-05-31T20:37:27.683 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:37:27 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:uE6nx6CjNDHW1i/Um9P2dLrJXnA=', 'Content-Length': '0'} 2023-05-31T20:37:27.686 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:35:04 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000767272b7cdcb12ed-006477b007-1b4f-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:37:27 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:37:27.686 DEBUG:boto:path=/pklxkd-57/g 2023-05-31T20:37:27.687 DEBUG:boto:auth_path=/pklxkd-57/g 2023-05-31T20:37:27.687 DEBUG:boto:path=/pklxkd-57/g?versionId=null 2023-05-31T20:37:27.687 DEBUG:boto:auth_path=/pklxkd-57/g?versionId=null 2023-05-31T20:37:27.688 DEBUG:boto:Method: GET 2023-05-31T20:37:27.689 DEBUG:boto:Path: /pklxkd-57/g?versionId=null 2023-05-31T20:37:27.689 DEBUG:boto:Data: 2023-05-31T20:37:27.690 DEBUG:boto:Headers: {} 2023-05-31T20:37:27.690 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:37:27.691 DEBUG:boto:Port: 8000 2023-05-31T20:37:27.691 DEBUG:boto:Params: {} 2023-05-31T20:37:27.692 DEBUG:boto:Token: None 2023-05-31T20:37:27.692 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:37:27 GMT /pklxkd-57/g?versionId=null 2023-05-31T20:37:27.693 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:uE6nx6CjNDHW1i/Um9P2dLrJXnA= 2023-05-31T20:37:27.693 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:37:27 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:uE6nx6CjNDHW1i/Um9P2dLrJXnA=', 'Content-Length': '0'} 2023-05-31T20:37:27.695 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:35:04 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005db3f69e2d39cc07-006477b007-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:37:27 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:37:27.696 DEBUG:boto:path=/pklxkd-57/ 2023-05-31T20:37:27.696 DEBUG:boto:auth_path=/pklxkd-57/ 2023-05-31T20:37:27.697 DEBUG:boto:path=/pklxkd-57/?prefix=g 2023-05-31T20:37:27.697 DEBUG:boto:auth_path=/pklxkd-57/?prefix=g 2023-05-31T20:37:27.697 DEBUG:boto:Method: GET 2023-05-31T20:37:27.698 DEBUG:boto:Path: /pklxkd-57/?prefix=g 2023-05-31T20:37:27.698 DEBUG:boto:Data: 2023-05-31T20:37:27.698 DEBUG:boto:Headers: {} 2023-05-31T20:37:27.699 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:37:27.699 DEBUG:boto:Port: 8001 2023-05-31T20:37:27.699 DEBUG:boto:Params: {} 2023-05-31T20:37:27.700 DEBUG:boto:Token: None 2023-05-31T20:37:27.700 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:37:27 GMT /pklxkd-57/ 2023-05-31T20:37:27.701 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:kR8XV0FF++frnB9KbRgaOASXdNM= 2023-05-31T20:37:27.701 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:37:27 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:kR8XV0FF++frnB9KbRgaOASXdNM=', 'Content-Length': '0'} 2023-05-31T20:37:27.704 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000005b38c3996a817b09-006477b007-1b4f-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:37:27 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:37:27.705 DEBUG:boto:b'pklxkd-57g1000falseg2023-05-31T20:35:04.942Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:37:27.707 DEBUG:boto:path=/pklxkd-57/ 2023-05-31T20:37:27.708 DEBUG:boto:auth_path=/pklxkd-57/ 2023-05-31T20:37:27.708 DEBUG:boto:path=/pklxkd-57/?prefix=g 2023-05-31T20:37:27.709 DEBUG:boto:auth_path=/pklxkd-57/?prefix=g 2023-05-31T20:37:27.710 DEBUG:boto:Method: GET 2023-05-31T20:37:27.710 DEBUG:boto:Path: /pklxkd-57/?prefix=g 2023-05-31T20:37:27.711 DEBUG:boto:Data: 2023-05-31T20:37:27.711 DEBUG:boto:Headers: {} 2023-05-31T20:37:27.712 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:37:27.713 DEBUG:boto:Port: 8000 2023-05-31T20:37:27.714 DEBUG:boto:Params: {} 2023-05-31T20:37:27.714 DEBUG:boto:Token: None 2023-05-31T20:37:27.715 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:37:27 GMT /pklxkd-57/ 2023-05-31T20:37:27.716 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:kR8XV0FF++frnB9KbRgaOASXdNM= 2023-05-31T20:37:27.718 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:37:27 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:kR8XV0FF++frnB9KbRgaOASXdNM=', 'Content-Length': '0'} 2023-05-31T20:37:27.720 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d75a06e2d18f9400-006477b007-1630-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:37:27 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:37:27.721 DEBUG:boto:b'pklxkd-57g1000falseg2023-05-31T20:35:04.942Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:37:27.722 DEBUG:boto:path=/pklxkd-57/g 2023-05-31T20:37:27.723 DEBUG:boto:auth_path=/pklxkd-57/g 2023-05-31T20:37:27.724 DEBUG:boto:Method: HEAD 2023-05-31T20:37:27.725 DEBUG:boto:Path: /pklxkd-57/g 2023-05-31T20:37:27.726 DEBUG:boto:Data: 2023-05-31T20:37:27.726 DEBUG:boto:Headers: {} 2023-05-31T20:37:27.727 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:37:27.729 DEBUG:boto:Port: 8001 2023-05-31T20:37:27.729 DEBUG:boto:Params: {} 2023-05-31T20:37:27.730 DEBUG:boto:Token: None 2023-05-31T20:37:27.731 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:37:27 GMT /pklxkd-57/g 2023-05-31T20:37:27.732 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:A3+KAXfulVyKjJ05llnhRrOV8Hs= 2023-05-31T20:37:27.733 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:37:27 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:A3+KAXfulVyKjJ05llnhRrOV8Hs=', 'Content-Length': '0'} 2023-05-31T20:37:27.735 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:35:04 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b9255a95a229bc76-006477b007-1b4f-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:37:27 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:37:27.737 DEBUG:boto:path=/pklxkd-57/g 2023-05-31T20:37:27.737 DEBUG:boto:auth_path=/pklxkd-57/g 2023-05-31T20:37:27.738 DEBUG:boto:Method: HEAD 2023-05-31T20:37:27.739 DEBUG:boto:Path: /pklxkd-57/g 2023-05-31T20:37:27.739 DEBUG:boto:Data: 2023-05-31T20:37:27.740 DEBUG:boto:Headers: {} 2023-05-31T20:37:27.741 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:37:27.741 DEBUG:boto:Port: 8000 2023-05-31T20:37:27.742 DEBUG:boto:Params: {} 2023-05-31T20:37:27.743 DEBUG:boto:Token: None 2023-05-31T20:37:27.743 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:37:27 GMT /pklxkd-57/g 2023-05-31T20:37:27.744 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:A3+KAXfulVyKjJ05llnhRrOV8Hs= 2023-05-31T20:37:27.745 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:37:27 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:A3+KAXfulVyKjJ05llnhRrOV8Hs=', 'Content-Length': '0'} 2023-05-31T20:37:27.747 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:35:04 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f00a4405a3ca4271-006477b007-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:37:27 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:37:27.748 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=g 2023-05-31T20:37:27.749 DEBUG:boto:path=/pklxkd-57/g 2023-05-31T20:37:27.749 DEBUG:boto:auth_path=/pklxkd-57/g 2023-05-31T20:37:27.749 DEBUG:boto:Method: GET 2023-05-31T20:37:27.750 DEBUG:boto:Path: /pklxkd-57/g 2023-05-31T20:37:27.750 DEBUG:boto:Data: 2023-05-31T20:37:27.751 DEBUG:boto:Headers: {} 2023-05-31T20:37:27.751 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:37:27.751 DEBUG:boto:Port: 8001 2023-05-31T20:37:27.752 DEBUG:boto:Params: {} 2023-05-31T20:37:27.752 DEBUG:boto:Token: None 2023-05-31T20:37:27.753 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:37:27 GMT /pklxkd-57/g 2023-05-31T20:37:27.753 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:viLvadsR731xQvXFX9+YfTMz5t8= 2023-05-31T20:37:27.754 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:37:27 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:viLvadsR731xQvXFX9+YfTMz5t8=', 'Content-Length': '0'} 2023-05-31T20:37:27.755 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:35:04 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000069ede2b659c8a127-006477b007-1b4f-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:37:27 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:37:27.756 DEBUG:boto:path=/pklxkd-57/g 2023-05-31T20:37:27.757 DEBUG:boto:auth_path=/pklxkd-57/g 2023-05-31T20:37:27.757 DEBUG:boto:Method: GET 2023-05-31T20:37:27.758 DEBUG:boto:Path: /pklxkd-57/g 2023-05-31T20:37:27.758 DEBUG:boto:Data: 2023-05-31T20:37:27.758 DEBUG:boto:Headers: {} 2023-05-31T20:37:27.759 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:37:27.759 DEBUG:boto:Port: 8000 2023-05-31T20:37:27.759 DEBUG:boto:Params: {} 2023-05-31T20:37:27.760 DEBUG:boto:Token: None 2023-05-31T20:37:27.760 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:37:27 GMT /pklxkd-57/g 2023-05-31T20:37:27.761 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:viLvadsR731xQvXFX9+YfTMz5t8= 2023-05-31T20:37:27.761 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:37:27 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:viLvadsR731xQvXFX9+YfTMz5t8=', 'Content-Length': '0'} 2023-05-31T20:37:27.763 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:35:04 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c4fad078c93146de-006477b007-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:37:27 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:37:27.804 DEBUG:boto:path=/pklxkd-57/ 2023-05-31T20:37:27.805 DEBUG:boto:auth_path=/pklxkd-57/ 2023-05-31T20:37:27.805 DEBUG:boto:path=/pklxkd-57/?prefix=g 2023-05-31T20:37:27.806 DEBUG:boto:auth_path=/pklxkd-57/?prefix=g 2023-05-31T20:37:27.806 DEBUG:boto:Method: GET 2023-05-31T20:37:27.807 DEBUG:boto:Path: /pklxkd-57/?prefix=g 2023-05-31T20:37:27.807 DEBUG:boto:Data: 2023-05-31T20:37:27.808 DEBUG:boto:Headers: {} 2023-05-31T20:37:27.808 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:37:27.808 DEBUG:boto:Port: 8001 2023-05-31T20:37:27.809 DEBUG:boto:Params: {} 2023-05-31T20:37:27.809 DEBUG:boto:Token: None 2023-05-31T20:37:27.809 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:37:27 GMT /pklxkd-57/ 2023-05-31T20:37:27.811 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:kR8XV0FF++frnB9KbRgaOASXdNM= 2023-05-31T20:37:27.811 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:37:27 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:kR8XV0FF++frnB9KbRgaOASXdNM=', 'Content-Length': '0'} 2023-05-31T20:37:27.814 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000096a4b5d4d4aaf4a2-006477b007-1b4f-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:37:27 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:37:27.815 DEBUG:boto:b'pklxkd-57g1000falseg2023-05-31T20:35:04.942Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:37:27.815 DEBUG:boto:path=/pklxkd-57/ 2023-05-31T20:37:27.816 DEBUG:boto:auth_path=/pklxkd-57/ 2023-05-31T20:37:27.816 DEBUG:boto:path=/pklxkd-57/?prefix=g 2023-05-31T20:37:27.817 DEBUG:boto:auth_path=/pklxkd-57/?prefix=g 2023-05-31T20:37:27.817 DEBUG:boto:Method: GET 2023-05-31T20:37:27.817 DEBUG:boto:Path: /pklxkd-57/?prefix=g 2023-05-31T20:37:27.818 DEBUG:boto:Data: 2023-05-31T20:37:27.818 DEBUG:boto:Headers: {} 2023-05-31T20:37:27.818 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:37:27.819 DEBUG:boto:Port: 8000 2023-05-31T20:37:27.819 DEBUG:boto:Params: {} 2023-05-31T20:37:27.819 DEBUG:boto:Token: None 2023-05-31T20:37:27.820 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:37:27 GMT /pklxkd-57/ 2023-05-31T20:37:27.820 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:kR8XV0FF++frnB9KbRgaOASXdNM= 2023-05-31T20:37:27.821 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:37:27 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:kR8XV0FF++frnB9KbRgaOASXdNM=', 'Content-Length': '0'} 2023-05-31T20:37:27.823 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e8e9af9f686f23a6-006477b007-1630-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:37:27 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:37:27.824 DEBUG:boto:b'pklxkd-57g1000falseg2023-05-31T20:35:04.942Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:37:27.824 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=h 2023-05-31T20:37:27.825 DEBUG:boto:path=/pklxkd-57/h 2023-05-31T20:37:27.825 DEBUG:boto:auth_path=/pklxkd-57/h 2023-05-31T20:37:27.825 DEBUG:boto:path=/pklxkd-57/h?versionId=null 2023-05-31T20:37:27.826 DEBUG:boto:auth_path=/pklxkd-57/h?versionId=null 2023-05-31T20:37:27.826 DEBUG:boto:Method: GET 2023-05-31T20:37:27.827 DEBUG:boto:Path: /pklxkd-57/h?versionId=null 2023-05-31T20:37:27.827 DEBUG:boto:Data: 2023-05-31T20:37:27.827 DEBUG:boto:Headers: {} 2023-05-31T20:37:27.828 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:37:27.828 DEBUG:boto:Port: 8001 2023-05-31T20:37:27.828 DEBUG:boto:Params: {} 2023-05-31T20:37:27.829 DEBUG:boto:Token: None 2023-05-31T20:37:27.829 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:37:27 GMT /pklxkd-57/h?versionId=null 2023-05-31T20:37:27.830 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:VMTZKRiBrBljCXj7styI4Cz0xdQ= 2023-05-31T20:37:27.830 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:37:27 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:VMTZKRiBrBljCXj7styI4Cz0xdQ=', 'Content-Length': '0'} 2023-05-31T20:37:27.833 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:35:05 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000321318febf506d5f-006477b007-1b4f-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:37:27 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:37:27.876 DEBUG:boto:path=/pklxkd-57/h 2023-05-31T20:37:27.877 DEBUG:boto:auth_path=/pklxkd-57/h 2023-05-31T20:37:27.877 DEBUG:boto:path=/pklxkd-57/h?versionId=null 2023-05-31T20:37:27.878 DEBUG:boto:auth_path=/pklxkd-57/h?versionId=null 2023-05-31T20:37:27.878 DEBUG:boto:Method: GET 2023-05-31T20:37:27.878 DEBUG:boto:Path: /pklxkd-57/h?versionId=null 2023-05-31T20:37:27.879 DEBUG:boto:Data: 2023-05-31T20:37:27.879 DEBUG:boto:Headers: {} 2023-05-31T20:37:27.879 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:37:27.880 DEBUG:boto:Port: 8000 2023-05-31T20:37:27.880 DEBUG:boto:Params: {} 2023-05-31T20:37:27.881 DEBUG:boto:Token: None 2023-05-31T20:37:27.881 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:37:27 GMT /pklxkd-57/h?versionId=null 2023-05-31T20:37:27.881 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:VMTZKRiBrBljCXj7styI4Cz0xdQ= 2023-05-31T20:37:27.882 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:37:27 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:VMTZKRiBrBljCXj7styI4Cz0xdQ=', 'Content-Length': '0'} 2023-05-31T20:37:27.884 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:35:05 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002a567e89ffc1e09d-006477b007-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:37:27 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:37:27.884 DEBUG:boto:path=/pklxkd-57/h 2023-05-31T20:37:27.885 DEBUG:boto:auth_path=/pklxkd-57/h 2023-05-31T20:37:27.885 DEBUG:boto:path=/pklxkd-57/h?versionId=null 2023-05-31T20:37:27.886 DEBUG:boto:auth_path=/pklxkd-57/h?versionId=null 2023-05-31T20:37:27.886 DEBUG:boto:Method: HEAD 2023-05-31T20:37:27.886 DEBUG:boto:Path: /pklxkd-57/h?versionId=null 2023-05-31T20:37:27.887 DEBUG:boto:Data: 2023-05-31T20:37:27.887 DEBUG:boto:Headers: {} 2023-05-31T20:37:27.887 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:37:27.888 DEBUG:boto:Port: 8001 2023-05-31T20:37:27.888 DEBUG:boto:Params: {} 2023-05-31T20:37:27.888 DEBUG:boto:Token: None 2023-05-31T20:37:27.889 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:37:27 GMT /pklxkd-57/h?versionId=null 2023-05-31T20:37:27.889 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:KApIbitrFJLrPlhllVJe8muCky8= 2023-05-31T20:37:27.890 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:37:27 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:KApIbitrFJLrPlhllVJe8muCky8=', 'Content-Length': '0'} 2023-05-31T20:37:27.891 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:35:05 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a2f969dcbd472398-006477b007-1b4f-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:37:27 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:37:27.892 DEBUG:boto:path=/pklxkd-57/h 2023-05-31T20:37:27.892 DEBUG:boto:auth_path=/pklxkd-57/h 2023-05-31T20:37:27.892 DEBUG:boto:path=/pklxkd-57/h?versionId=null 2023-05-31T20:37:27.893 DEBUG:boto:auth_path=/pklxkd-57/h?versionId=null 2023-05-31T20:37:27.893 DEBUG:boto:Method: HEAD 2023-05-31T20:37:27.893 DEBUG:boto:Path: /pklxkd-57/h?versionId=null 2023-05-31T20:37:27.894 DEBUG:boto:Data: 2023-05-31T20:37:27.894 DEBUG:boto:Headers: {} 2023-05-31T20:37:27.894 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:37:27.895 DEBUG:boto:Port: 8000 2023-05-31T20:37:27.895 DEBUG:boto:Params: {} 2023-05-31T20:37:27.896 DEBUG:boto:Token: None 2023-05-31T20:37:27.896 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:37:27 GMT /pklxkd-57/h?versionId=null 2023-05-31T20:37:27.896 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:KApIbitrFJLrPlhllVJe8muCky8= 2023-05-31T20:37:27.897 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:37:27 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:KApIbitrFJLrPlhllVJe8muCky8=', 'Content-Length': '0'} 2023-05-31T20:37:27.899 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:35:05 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ed54b9707565f198-006477b007-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:37:27 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:37:27.899 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=h 2023-05-31T20:37:27.900 DEBUG:boto:path=/pklxkd-57/h 2023-05-31T20:37:27.900 DEBUG:boto:auth_path=/pklxkd-57/h 2023-05-31T20:37:27.900 DEBUG:boto:path=/pklxkd-57/h?versionId=null 2023-05-31T20:37:27.901 DEBUG:boto:auth_path=/pklxkd-57/h?versionId=null 2023-05-31T20:37:27.901 DEBUG:boto:Method: GET 2023-05-31T20:37:27.902 DEBUG:boto:Path: /pklxkd-57/h?versionId=null 2023-05-31T20:37:27.902 DEBUG:boto:Data: 2023-05-31T20:37:27.902 DEBUG:boto:Headers: {} 2023-05-31T20:37:27.903 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:37:27.903 DEBUG:boto:Port: 8001 2023-05-31T20:37:27.903 DEBUG:boto:Params: {} 2023-05-31T20:37:27.904 DEBUG:boto:Token: None 2023-05-31T20:37:27.905 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:37:27 GMT /pklxkd-57/h?versionId=null 2023-05-31T20:37:27.905 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:VMTZKRiBrBljCXj7styI4Cz0xdQ= 2023-05-31T20:37:27.906 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:37:27 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:VMTZKRiBrBljCXj7styI4Cz0xdQ=', 'Content-Length': '0'} 2023-05-31T20:37:27.909 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:35:05 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000012dce218fb8c7b39-006477b007-1b4f-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:37:27 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:37:27.910 DEBUG:boto:path=/pklxkd-57/h 2023-05-31T20:37:27.910 DEBUG:boto:auth_path=/pklxkd-57/h 2023-05-31T20:37:27.910 DEBUG:boto:path=/pklxkd-57/h?versionId=null 2023-05-31T20:37:27.911 DEBUG:boto:auth_path=/pklxkd-57/h?versionId=null 2023-05-31T20:37:27.911 DEBUG:boto:Method: GET 2023-05-31T20:37:27.912 DEBUG:boto:Path: /pklxkd-57/h?versionId=null 2023-05-31T20:37:27.912 DEBUG:boto:Data: 2023-05-31T20:37:27.913 DEBUG:boto:Headers: {} 2023-05-31T20:37:27.913 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:37:27.913 DEBUG:boto:Port: 8000 2023-05-31T20:37:27.914 DEBUG:boto:Params: {} 2023-05-31T20:37:27.914 DEBUG:boto:Token: None 2023-05-31T20:37:27.915 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:37:27 GMT /pklxkd-57/h?versionId=null 2023-05-31T20:37:27.916 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:VMTZKRiBrBljCXj7styI4Cz0xdQ= 2023-05-31T20:37:27.917 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:37:27 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:VMTZKRiBrBljCXj7styI4Cz0xdQ=', 'Content-Length': '0'} 2023-05-31T20:37:27.919 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:35:05 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007e43ca45b01c13fe-006477b007-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:37:27 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:37:27.920 DEBUG:boto:path=/pklxkd-57/ 2023-05-31T20:37:27.920 DEBUG:boto:auth_path=/pklxkd-57/ 2023-05-31T20:37:27.921 DEBUG:boto:path=/pklxkd-57/?prefix=h 2023-05-31T20:37:27.921 DEBUG:boto:auth_path=/pklxkd-57/?prefix=h 2023-05-31T20:37:27.922 DEBUG:boto:Method: GET 2023-05-31T20:37:27.922 DEBUG:boto:Path: /pklxkd-57/?prefix=h 2023-05-31T20:37:27.922 DEBUG:boto:Data: 2023-05-31T20:37:27.923 DEBUG:boto:Headers: {} 2023-05-31T20:37:27.923 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:37:27.924 DEBUG:boto:Port: 8001 2023-05-31T20:37:27.924 DEBUG:boto:Params: {} 2023-05-31T20:37:27.925 DEBUG:boto:Token: None 2023-05-31T20:37:27.925 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:37:27 GMT /pklxkd-57/ 2023-05-31T20:37:27.926 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:kR8XV0FF++frnB9KbRgaOASXdNM= 2023-05-31T20:37:27.926 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:37:27 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:kR8XV0FF++frnB9KbRgaOASXdNM=', 'Content-Length': '0'} 2023-05-31T20:37:27.930 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000403650315abd899f-006477b007-1b4f-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:37:27 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:37:27.931 DEBUG:boto:b'pklxkd-57h1000falseh2023-05-31T20:35:05.004Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:37:27.931 DEBUG:boto:path=/pklxkd-57/ 2023-05-31T20:37:27.932 DEBUG:boto:auth_path=/pklxkd-57/ 2023-05-31T20:37:27.933 DEBUG:boto:path=/pklxkd-57/?prefix=h 2023-05-31T20:37:27.934 DEBUG:boto:auth_path=/pklxkd-57/?prefix=h 2023-05-31T20:37:27.934 DEBUG:boto:Method: GET 2023-05-31T20:37:27.934 DEBUG:boto:Path: /pklxkd-57/?prefix=h 2023-05-31T20:37:27.935 DEBUG:boto:Data: 2023-05-31T20:37:27.935 DEBUG:boto:Headers: {} 2023-05-31T20:37:27.935 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:37:27.936 DEBUG:boto:Port: 8000 2023-05-31T20:37:27.936 DEBUG:boto:Params: {} 2023-05-31T20:37:27.937 DEBUG:boto:Token: None 2023-05-31T20:37:27.937 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:37:27 GMT /pklxkd-57/ 2023-05-31T20:37:27.938 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:kR8XV0FF++frnB9KbRgaOASXdNM= 2023-05-31T20:37:27.938 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:37:27 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:kR8XV0FF++frnB9KbRgaOASXdNM=', 'Content-Length': '0'} 2023-05-31T20:37:27.940 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000fd286e9dde4ea97b-006477b007-1630-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:37:27 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:37:27.940 DEBUG:boto:b'pklxkd-57h1000falseh2023-05-31T20:35:05.004Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:37:27.941 DEBUG:boto:path=/pklxkd-57/h 2023-05-31T20:37:27.942 DEBUG:boto:auth_path=/pklxkd-57/h 2023-05-31T20:37:27.942 DEBUG:boto:Method: HEAD 2023-05-31T20:37:27.942 DEBUG:boto:Path: /pklxkd-57/h 2023-05-31T20:37:27.943 DEBUG:boto:Data: 2023-05-31T20:37:27.943 DEBUG:boto:Headers: {} 2023-05-31T20:37:27.943 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:37:27.944 DEBUG:boto:Port: 8001 2023-05-31T20:37:27.944 DEBUG:boto:Params: {} 2023-05-31T20:37:27.944 DEBUG:boto:Token: None 2023-05-31T20:37:27.945 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:37:27 GMT /pklxkd-57/h 2023-05-31T20:37:27.945 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:sh36uV0WapzRMOYlFhA2mCW7MXc= 2023-05-31T20:37:27.946 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:37:27 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:sh36uV0WapzRMOYlFhA2mCW7MXc=', 'Content-Length': '0'} 2023-05-31T20:37:27.947 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:35:05 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000165a3aeb223ba181-006477b007-1b4f-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:37:27 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:37:27.948 DEBUG:boto:path=/pklxkd-57/h 2023-05-31T20:37:27.949 DEBUG:boto:auth_path=/pklxkd-57/h 2023-05-31T20:37:27.949 DEBUG:boto:Method: HEAD 2023-05-31T20:37:27.949 DEBUG:boto:Path: /pklxkd-57/h 2023-05-31T20:37:27.950 DEBUG:boto:Data: 2023-05-31T20:37:27.950 DEBUG:boto:Headers: {} 2023-05-31T20:37:27.950 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:37:27.951 DEBUG:boto:Port: 8000 2023-05-31T20:37:27.951 DEBUG:boto:Params: {} 2023-05-31T20:37:27.951 DEBUG:boto:Token: None 2023-05-31T20:37:27.952 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:37:27 GMT /pklxkd-57/h 2023-05-31T20:37:27.952 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:sh36uV0WapzRMOYlFhA2mCW7MXc= 2023-05-31T20:37:27.953 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:37:27 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:sh36uV0WapzRMOYlFhA2mCW7MXc=', 'Content-Length': '0'} 2023-05-31T20:37:27.955 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:35:05 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ebd6767feaa5850b-006477b007-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:37:27 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:37:27.955 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=h 2023-05-31T20:37:27.956 DEBUG:boto:path=/pklxkd-57/h 2023-05-31T20:37:27.956 DEBUG:boto:auth_path=/pklxkd-57/h 2023-05-31T20:37:27.956 DEBUG:boto:Method: GET 2023-05-31T20:37:27.957 DEBUG:boto:Path: /pklxkd-57/h 2023-05-31T20:37:27.957 DEBUG:boto:Data: 2023-05-31T20:37:27.958 DEBUG:boto:Headers: {} 2023-05-31T20:37:27.958 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:37:27.958 DEBUG:boto:Port: 8001 2023-05-31T20:37:27.959 DEBUG:boto:Params: {} 2023-05-31T20:37:27.959 DEBUG:boto:Token: None 2023-05-31T20:37:27.960 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:37:27 GMT /pklxkd-57/h 2023-05-31T20:37:27.960 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:VystQmnrpAoJAEJfWq5RvU1TeR0= 2023-05-31T20:37:27.960 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:37:27 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:VystQmnrpAoJAEJfWq5RvU1TeR0=', 'Content-Length': '0'} 2023-05-31T20:37:27.962 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:35:05 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ec78b21d9e0c59b8-006477b007-1b4f-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:37:27 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:37:28.004 DEBUG:boto:path=/pklxkd-57/h 2023-05-31T20:37:28.005 DEBUG:boto:auth_path=/pklxkd-57/h 2023-05-31T20:37:28.005 DEBUG:boto:Method: GET 2023-05-31T20:37:28.006 DEBUG:boto:Path: /pklxkd-57/h 2023-05-31T20:37:28.006 DEBUG:boto:Data: 2023-05-31T20:37:28.006 DEBUG:boto:Headers: {} 2023-05-31T20:37:28.007 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:37:28.007 DEBUG:boto:Port: 8000 2023-05-31T20:37:28.008 DEBUG:boto:Params: {} 2023-05-31T20:37:28.008 DEBUG:boto:Token: None 2023-05-31T20:37:28.008 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:37:28 GMT /pklxkd-57/h 2023-05-31T20:37:28.009 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:F4aBZhv0yw+9LWkpRkyZJzkAcRI= 2023-05-31T20:37:28.009 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:37:28 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:F4aBZhv0yw+9LWkpRkyZJzkAcRI=', 'Content-Length': '0'} 2023-05-31T20:37:28.011 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:35:05 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000cd1b45563e606fbb-006477b008-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:37:28 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:37:28.012 DEBUG:boto:path=/pklxkd-57/ 2023-05-31T20:37:28.013 DEBUG:boto:auth_path=/pklxkd-57/ 2023-05-31T20:37:28.013 DEBUG:boto:path=/pklxkd-57/?prefix=h 2023-05-31T20:37:28.013 DEBUG:boto:auth_path=/pklxkd-57/?prefix=h 2023-05-31T20:37:28.014 DEBUG:boto:Method: GET 2023-05-31T20:37:28.014 DEBUG:boto:Path: /pklxkd-57/?prefix=h 2023-05-31T20:37:28.014 DEBUG:boto:Data: 2023-05-31T20:37:28.015 DEBUG:boto:Headers: {} 2023-05-31T20:37:28.015 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:37:28.015 DEBUG:boto:Port: 8001 2023-05-31T20:37:28.016 DEBUG:boto:Params: {} 2023-05-31T20:37:28.016 DEBUG:boto:Token: None 2023-05-31T20:37:28.017 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:37:28 GMT /pklxkd-57/ 2023-05-31T20:37:28.017 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:iCTCt+UO1EqWYoeoxV8DdJXMW9w= 2023-05-31T20:37:28.018 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:37:28 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:iCTCt+UO1EqWYoeoxV8DdJXMW9w=', 'Content-Length': '0'} 2023-05-31T20:37:28.020 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000cf39903d5b768f53-006477b008-1b4f-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:37:28 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:37:28.021 DEBUG:boto:b'pklxkd-57h1000falseh2023-05-31T20:35:05.004Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:37:28.021 DEBUG:boto:path=/pklxkd-57/ 2023-05-31T20:37:28.022 DEBUG:boto:auth_path=/pklxkd-57/ 2023-05-31T20:37:28.022 DEBUG:boto:path=/pklxkd-57/?prefix=h 2023-05-31T20:37:28.022 DEBUG:boto:auth_path=/pklxkd-57/?prefix=h 2023-05-31T20:37:28.023 DEBUG:boto:Method: GET 2023-05-31T20:37:28.023 DEBUG:boto:Path: /pklxkd-57/?prefix=h 2023-05-31T20:37:28.024 DEBUG:boto:Data: 2023-05-31T20:37:28.024 DEBUG:boto:Headers: {} 2023-05-31T20:37:28.024 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:37:28.025 DEBUG:boto:Port: 8000 2023-05-31T20:37:28.025 DEBUG:boto:Params: {} 2023-05-31T20:37:28.025 DEBUG:boto:Token: None 2023-05-31T20:37:28.026 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:37:28 GMT /pklxkd-57/ 2023-05-31T20:37:28.026 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:iCTCt+UO1EqWYoeoxV8DdJXMW9w= 2023-05-31T20:37:28.027 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:37:28 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:iCTCt+UO1EqWYoeoxV8DdJXMW9w=', 'Content-Length': '0'} 2023-05-31T20:37:28.029 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000083539008b8de4e34-006477b008-1630-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:37:28 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:37:28.030 DEBUG:boto:b'pklxkd-57h1000falseh2023-05-31T20:35:05.004Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:37:28.031 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=pklxkd-57 zones={a2, a1} 2023-05-31T20:37:28.032 INFO:tasks.rgw_multisite_tests:rgw_multi.tests.test_bucket_reshard_full ... ok 2023-05-31T20:37:28.033 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-31T20:37:28.034 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-31T20:37:28.034 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-05-31T20:37:28.090 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2023-05-31T20:37:28.091 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2023-05-31T20:37:28.132 INFO:teuthology.orchestra.run.smithi099.stderr:No --period given, using current period=2e6123d2-d695-4350-9a01-61a455bc782b 2023-05-31T20:37:28.152 INFO:teuthology.orchestra.run.smithi099.stdout:[ 2023-05-31T20:37:28.153 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T20:37:28.153 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1685562631.516089_59.1", 2023-05-31T20:37:28.153 INFO:teuthology.orchestra.run.smithi099.stdout: "last_update": "2023-05-31T19:50:31.516089Z" 2023-05-31T20:37:28.153 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:37:28.154 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T20:37:28.154 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "", 2023-05-31T20:37:28.154 INFO:teuthology.orchestra.run.smithi099.stdout: "last_update": "0.000000" 2023-05-31T20:37:28.154 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:37:28.154 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T20:37:28.154 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1685562970.274572_93.1", 2023-05-31T20:37:28.154 INFO:teuthology.orchestra.run.smithi099.stdout: "last_update": "2023-05-31T19:56:10.274572Z" 2023-05-31T20:37:28.155 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:37:28.155 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T20:37:28.155 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1685565304.837720_397.1", 2023-05-31T20:37:28.155 INFO:teuthology.orchestra.run.smithi099.stdout: "last_update": "2023-05-31T20:35:04.837720Z" 2023-05-31T20:37:28.155 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T20:37:28.155 INFO:teuthology.orchestra.run.smithi099.stdout:] 2023-05-31T20:37:28.157 DEBUG:rgw_multi.tests:master meta markers={0: '1_1685562631.516089_59.1', 1: '', 2: '1_1685562970.274572_93.1', 3: '1_1685565304.837720_397.1'} 2023-05-31T20:37:28.157 INFO:rgw_multi.tests:starting meta checkpoint for zone=a2 2023-05-31T20:37:28.158 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-31T20:37:28.161 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-31T20:37:28.162 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-05-31T20:37:28.212 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2023-05-31T20:37:28.213 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2023-05-31T20:37:28.266 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:37:28.260+0000 7f6317ab4780 20 RGW-SYNC:meta: read sync status 2023-05-31T20:37:28.266 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:37:28.261+0000 7f6317ab4780 20 RGW-SYNC:meta: read sync status 2023-05-31T20:37:28.285 INFO:teuthology.orchestra.run.smithi099.stdout:{ 2023-05-31T20:37:28.285 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_status": { 2023-05-31T20:37:28.285 INFO:teuthology.orchestra.run.smithi099.stdout: "info": { 2023-05-31T20:37:28.285 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "sync", 2023-05-31T20:37:28.286 INFO:teuthology.orchestra.run.smithi099.stdout: "num_shards": 4, 2023-05-31T20:37:28.286 INFO:teuthology.orchestra.run.smithi099.stdout: "period": "2e6123d2-d695-4350-9a01-61a455bc782b", 2023-05-31T20:37:28.286 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2023-05-31T20:37:28.286 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:37:28.286 INFO:teuthology.orchestra.run.smithi099.stdout: "markers": [ 2023-05-31T20:37:28.286 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T20:37:28.287 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 0, 2023-05-31T20:37:28.288 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T20:37:28.288 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2023-05-31T20:37:28.288 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1685562631.516089_59.1", 2023-05-31T20:37:28.288 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T20:37:28.288 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 1, 2023-05-31T20:37:28.289 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T20:37:28.289 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2023-05-31T19:50:31.516089Z", 2023-05-31T20:37:28.289 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2023-05-31T20:37:28.289 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T20:37:28.289 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:37:28.289 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T20:37:28.289 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 1, 2023-05-31T20:37:28.290 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T20:37:28.290 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2023-05-31T20:37:28.291 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "", 2023-05-31T20:37:28.291 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T20:37:28.291 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2023-05-31T20:37:28.292 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T20:37:28.292 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "0.000000", 2023-05-31T20:37:28.292 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2023-05-31T20:37:28.293 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T20:37:28.293 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:37:28.293 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T20:37:28.293 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 2, 2023-05-31T20:37:28.293 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T20:37:28.293 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2023-05-31T20:37:28.294 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1685562970.274572_93.1", 2023-05-31T20:37:28.294 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T20:37:28.294 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2023-05-31T20:37:28.294 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T20:37:28.294 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2023-05-31T19:56:10.274572Z", 2023-05-31T20:37:28.294 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2023-05-31T20:37:28.294 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T20:37:28.295 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:37:28.295 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T20:37:28.295 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 3, 2023-05-31T20:37:28.295 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T20:37:28.295 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2023-05-31T20:37:28.295 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1685565304.837720_397.1", 2023-05-31T20:37:28.295 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T20:37:28.296 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2023-05-31T20:37:28.296 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T20:37:28.296 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2023-05-31T20:35:04.837720Z", 2023-05-31T20:37:28.296 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2023-05-31T20:37:28.296 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T20:37:28.296 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T20:37:28.297 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2023-05-31T20:37:28.297 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:37:28.297 INFO:teuthology.orchestra.run.smithi099.stdout: "full_sync": { 2023-05-31T20:37:28.297 INFO:teuthology.orchestra.run.smithi099.stdout: "total": 1, 2023-05-31T20:37:28.297 INFO:teuthology.orchestra.run.smithi099.stdout: "complete": 1 2023-05-31T20:37:28.297 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:37:28.298 INFO:teuthology.orchestra.run.smithi099.stdout: "current_time": "2023-05-31T20:37:28Z" 2023-05-31T20:37:28.298 INFO:teuthology.orchestra.run.smithi099.stdout:} 2023-05-31T20:37:28.299 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "2e6123d2-d695-4350-9a01-61a455bc782b", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1685562631.516089_59.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-05-31T19:50:31.516089Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1685562970.274572_93.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-05-31T19:56:10.274572Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1685565304.837720_397.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-05-31T20:35:04.837720Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-05-31T20:37:28Z" } 2023-05-31T20:37:28.299 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1685562631.516089_59.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-05-31T19:50:31.516089Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1685562970.274572_93.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-05-31T19:56:10.274572Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1685565304.837720_397.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-05-31T20:35:04.837720Z', 'realm_epoch': 2}}] 2023-05-31T20:37:28.300 DEBUG:rgw_multi.tests:log_status={0: '1_1685562631.516089_59.1', 1: '', 2: '1_1685562970.274572_93.1', 3: '1_1685565304.837720_397.1'} 2023-05-31T20:37:28.300 DEBUG:rgw_multi.tests:sync_status={0: '1_1685562631.516089_59.1', 1: '', 2: '1_1685562970.274572_93.1', 3: '1_1685565304.837720_397.1'} 2023-05-31T20:37:28.300 INFO:rgw_multi.tests:finish meta checkpoint for zone=a2 2023-05-31T20:37:28.301 DEBUG:boto:path=/ 2023-05-31T20:37:28.301 DEBUG:boto:auth_path=/ 2023-05-31T20:37:28.302 DEBUG:boto:Method: GET 2023-05-31T20:37:28.302 DEBUG:boto:Path: / 2023-05-31T20:37:28.303 DEBUG:boto:Data: 2023-05-31T20:37:28.303 DEBUG:boto:Headers: {} 2023-05-31T20:37:28.304 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:37:28.304 DEBUG:boto:Port: 8000 2023-05-31T20:37:28.304 DEBUG:boto:Params: {} 2023-05-31T20:37:28.305 DEBUG:boto:Token: None 2023-05-31T20:37:28.305 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:37:28 GMT / 2023-05-31T20:37:28.305 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:xANeyzjmEyp21r6/0aSOEd8Z51A= 2023-05-31T20:37:28.306 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:37:28 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:xANeyzjmEyp21r6/0aSOEd8Z51A=', 'Content-Length': '0'} 2023-05-31T20:37:28.309 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000036146ff21731ecb8-006477b008-1630-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:37:28 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:37:28.310 DEBUG:boto:path=/ 2023-05-31T20:37:28.311 DEBUG:boto:auth_path=/ 2023-05-31T20:37:28.311 DEBUG:boto:Method: GET 2023-05-31T20:37:28.311 DEBUG:boto:Path: / 2023-05-31T20:37:28.312 DEBUG:boto:Data: 2023-05-31T20:37:28.312 DEBUG:boto:Headers: {} 2023-05-31T20:37:28.313 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:37:28.313 DEBUG:boto:Port: 8001 2023-05-31T20:37:28.313 DEBUG:boto:Params: {} 2023-05-31T20:37:28.315 DEBUG:boto:Token: None 2023-05-31T20:37:28.316 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:37:28 GMT / 2023-05-31T20:37:28.316 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:xANeyzjmEyp21r6/0aSOEd8Z51A= 2023-05-31T20:37:28.317 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:37:28 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:xANeyzjmEyp21r6/0aSOEd8Z51A=', 'Content-Length': '0'} 2023-05-31T20:37:28.319 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000548567ed32bff072-006477b008-1b4f-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:37:28 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:37:28.320 INFO:tasks.rgw_multisite_tests:rgw_multi.tests.test_bucket_creation_time ... ok 2023-05-31T20:37:28.321 DEBUG:boto:path=/pklxkd-58/ 2023-05-31T20:37:28.322 DEBUG:boto:auth_path=/pklxkd-58/ 2023-05-31T20:37:28.322 DEBUG:boto:Method: PUT 2023-05-31T20:37:28.323 DEBUG:boto:Path: /pklxkd-58/ 2023-05-31T20:37:28.323 DEBUG:boto:Data: 2023-05-31T20:37:28.323 DEBUG:boto:Headers: {} 2023-05-31T20:37:28.324 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:37:28.324 DEBUG:boto:Port: 8000 2023-05-31T20:37:28.324 DEBUG:boto:Params: {} 2023-05-31T20:37:28.325 DEBUG:boto:Token: None 2023-05-31T20:37:28.325 DEBUG:boto:StringToSign: PUT Wed, 31 May 2023 20:37:28 GMT /pklxkd-58/ 2023-05-31T20:37:28.326 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:aBw8bXkIZs2I+OVe9RpPvi6wwxU= 2023-05-31T20:37:28.326 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:37:28 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:aBw8bXkIZs2I+OVe9RpPvi6wwxU=', 'Content-Length': '0'} 2023-05-31T20:37:28.343 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000960021a66d445308-006477b008-1630-a1'), ('Content-Length', '0'), ('Date', 'Wed, 31 May 2023 20:37:28 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:37:28.344 DEBUG:rgw_multi.tests:created bucket=pklxkd-58 2023-05-31T20:37:28.344 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-31T20:37:28.345 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-31T20:37:28.345 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-05-31T20:37:28.394 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2023-05-31T20:37:28.394 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2023-05-31T20:37:28.428 INFO:teuthology.orchestra.run.smithi099.stderr:No --period given, using current period=2e6123d2-d695-4350-9a01-61a455bc782b 2023-05-31T20:37:28.441 INFO:teuthology.orchestra.run.smithi099.stdout:[ 2023-05-31T20:37:28.442 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T20:37:28.442 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1685562631.516089_59.1", 2023-05-31T20:37:28.443 INFO:teuthology.orchestra.run.smithi099.stdout: "last_update": "2023-05-31T19:50:31.516089Z" 2023-05-31T20:37:28.443 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:37:28.443 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T20:37:28.444 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "", 2023-05-31T20:37:28.444 INFO:teuthology.orchestra.run.smithi099.stdout: "last_update": "0.000000" 2023-05-31T20:37:28.445 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:37:28.445 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T20:37:28.445 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1685562970.274572_93.1", 2023-05-31T20:37:28.446 INFO:teuthology.orchestra.run.smithi099.stdout: "last_update": "2023-05-31T19:56:10.274572Z" 2023-05-31T20:37:28.446 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:37:28.446 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T20:37:28.447 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1685565448.338019_401.1", 2023-05-31T20:37:28.447 INFO:teuthology.orchestra.run.smithi099.stdout: "last_update": "2023-05-31T20:37:28.338019Z" 2023-05-31T20:37:28.447 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T20:37:28.448 INFO:teuthology.orchestra.run.smithi099.stdout:] 2023-05-31T20:37:28.449 DEBUG:rgw_multi.tests:master meta markers={0: '1_1685562631.516089_59.1', 1: '', 2: '1_1685562970.274572_93.1', 3: '1_1685565448.338019_401.1'} 2023-05-31T20:37:28.450 INFO:rgw_multi.tests:starting meta checkpoint for zone=a2 2023-05-31T20:37:28.450 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-31T20:37:28.451 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-31T20:37:28.452 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-05-31T20:37:28.503 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2023-05-31T20:37:28.503 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2023-05-31T20:37:28.547 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:37:28.542+0000 7f5da40a5780 20 RGW-SYNC:meta: read sync status 2023-05-31T20:37:28.548 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:37:28.543+0000 7f5da40a5780 20 RGW-SYNC:meta: read sync status 2023-05-31T20:37:28.562 INFO:teuthology.orchestra.run.smithi099.stdout:{ 2023-05-31T20:37:28.563 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_status": { 2023-05-31T20:37:28.563 INFO:teuthology.orchestra.run.smithi099.stdout: "info": { 2023-05-31T20:37:28.563 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "sync", 2023-05-31T20:37:28.563 INFO:teuthology.orchestra.run.smithi099.stdout: "num_shards": 4, 2023-05-31T20:37:28.564 INFO:teuthology.orchestra.run.smithi099.stdout: "period": "2e6123d2-d695-4350-9a01-61a455bc782b", 2023-05-31T20:37:28.564 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2023-05-31T20:37:28.564 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:37:28.564 INFO:teuthology.orchestra.run.smithi099.stdout: "markers": [ 2023-05-31T20:37:28.564 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T20:37:28.564 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 0, 2023-05-31T20:37:28.565 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T20:37:28.565 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2023-05-31T20:37:28.565 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1685562631.516089_59.1", 2023-05-31T20:37:28.565 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T20:37:28.565 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 1, 2023-05-31T20:37:28.565 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T20:37:28.566 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2023-05-31T19:50:31.516089Z", 2023-05-31T20:37:28.566 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2023-05-31T20:37:28.566 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T20:37:28.566 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:37:28.566 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T20:37:28.566 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 1, 2023-05-31T20:37:28.567 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T20:37:28.567 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2023-05-31T20:37:28.567 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "", 2023-05-31T20:37:28.567 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T20:37:28.567 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2023-05-31T20:37:28.567 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T20:37:28.568 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "0.000000", 2023-05-31T20:37:28.568 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2023-05-31T20:37:28.568 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T20:37:28.568 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:37:28.568 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T20:37:28.568 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 2, 2023-05-31T20:37:28.569 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T20:37:28.569 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2023-05-31T20:37:28.569 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1685562970.274572_93.1", 2023-05-31T20:37:28.570 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T20:37:28.571 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2023-05-31T20:37:28.571 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T20:37:28.572 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2023-05-31T19:56:10.274572Z", 2023-05-31T20:37:28.572 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2023-05-31T20:37:28.572 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T20:37:28.572 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:37:28.573 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T20:37:28.573 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 3, 2023-05-31T20:37:28.574 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T20:37:28.574 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2023-05-31T20:37:28.574 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1685565448.338019_401.1", 2023-05-31T20:37:28.574 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T20:37:28.574 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2023-05-31T20:37:28.574 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T20:37:28.575 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2023-05-31T20:37:28.338019Z", 2023-05-31T20:37:28.575 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2023-05-31T20:37:28.575 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T20:37:28.575 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T20:37:28.575 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2023-05-31T20:37:28.575 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:37:28.576 INFO:teuthology.orchestra.run.smithi099.stdout: "full_sync": { 2023-05-31T20:37:28.576 INFO:teuthology.orchestra.run.smithi099.stdout: "total": 1, 2023-05-31T20:37:28.576 INFO:teuthology.orchestra.run.smithi099.stdout: "complete": 1 2023-05-31T20:37:28.576 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:37:28.576 INFO:teuthology.orchestra.run.smithi099.stdout: "current_time": "2023-05-31T20:37:28Z" 2023-05-31T20:37:28.576 INFO:teuthology.orchestra.run.smithi099.stdout:} 2023-05-31T20:37:28.577 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "2e6123d2-d695-4350-9a01-61a455bc782b", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1685562631.516089_59.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-05-31T19:50:31.516089Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1685562970.274572_93.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-05-31T19:56:10.274572Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1685565448.338019_401.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-05-31T20:37:28.338019Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-05-31T20:37:28Z" } 2023-05-31T20:37:28.578 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1685562631.516089_59.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-05-31T19:50:31.516089Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1685562970.274572_93.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-05-31T19:56:10.274572Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1685565448.338019_401.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-05-31T20:37:28.338019Z', 'realm_epoch': 2}}] 2023-05-31T20:37:28.578 DEBUG:rgw_multi.tests:log_status={0: '1_1685562631.516089_59.1', 1: '', 2: '1_1685562970.274572_93.1', 3: '1_1685565448.338019_401.1'} 2023-05-31T20:37:28.579 DEBUG:rgw_multi.tests:sync_status={0: '1_1685562631.516089_59.1', 1: '', 2: '1_1685562970.274572_93.1', 3: '1_1685565448.338019_401.1'} 2023-05-31T20:37:28.579 INFO:rgw_multi.tests:finish meta checkpoint for zone=a2 2023-05-31T20:37:28.579 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'shard', 'objects', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--num-shards', '11', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-31T20:37:28.580 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'shard', 'objects', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--num-shards', '11', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-31T20:37:28.580 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket shard objects --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --num-shards 11 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-05-31T20:37:28.631 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2023-05-31T20:37:28.631 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2023-05-31T20:37:28.695 INFO:teuthology.orchestra.run.smithi099.stdout:{ 2023-05-31T20:37:28.695 INFO:teuthology.orchestra.run.smithi099.stdout: "objs": [ 2023-05-31T20:37:28.695 INFO:teuthology.orchestra.run.smithi099.stdout: "obj00000000000000000001", 2023-05-31T20:37:28.696 INFO:teuthology.orchestra.run.smithi099.stdout: "obj00000000000000000005", 2023-05-31T20:37:28.696 INFO:teuthology.orchestra.run.smithi099.stdout: "obj00000000000000000012", 2023-05-31T20:37:28.696 INFO:teuthology.orchestra.run.smithi099.stdout: "obj00000000000000000004", 2023-05-31T20:37:28.696 INFO:teuthology.orchestra.run.smithi099.stdout: "obj00000000000000000011", 2023-05-31T20:37:28.696 INFO:teuthology.orchestra.run.smithi099.stdout: "obj00000000000000000000", 2023-05-31T20:37:28.696 INFO:teuthology.orchestra.run.smithi099.stdout: "obj00000000000000000010", 2023-05-31T20:37:28.697 INFO:teuthology.orchestra.run.smithi099.stdout: "obj00000000000000000003", 2023-05-31T20:37:28.697 INFO:teuthology.orchestra.run.smithi099.stdout: "obj00000000000000000013", 2023-05-31T20:37:28.697 INFO:teuthology.orchestra.run.smithi099.stdout: "obj00000000000000000002", 2023-05-31T20:37:28.697 INFO:teuthology.orchestra.run.smithi099.stdout: "obj00000000000000000006" 2023-05-31T20:37:28.697 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2023-05-31T20:37:28.697 INFO:teuthology.orchestra.run.smithi099.stdout:} 2023-05-31T20:37:28.698 DEBUG:boto:path=/pklxkd-58/ 2023-05-31T20:37:28.699 DEBUG:boto:auth_path=/pklxkd-58/ 2023-05-31T20:37:28.699 DEBUG:boto:Method: HEAD 2023-05-31T20:37:28.699 DEBUG:boto:Path: /pklxkd-58/ 2023-05-31T20:37:28.700 DEBUG:boto:Data: 2023-05-31T20:37:28.700 DEBUG:boto:Headers: {} 2023-05-31T20:37:28.701 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:37:28.701 DEBUG:boto:Port: 8000 2023-05-31T20:37:28.701 DEBUG:boto:Params: {} 2023-05-31T20:37:28.702 DEBUG:boto:Token: None 2023-05-31T20:37:28.702 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:37:28 GMT /pklxkd-58/ 2023-05-31T20:37:28.703 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:nVb5lEQS0H6uLF/0uEEKACaIQMQ= 2023-05-31T20:37:28.703 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:37:28 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:nVb5lEQS0H6uLF/0uEEKACaIQMQ=', 'Content-Length': '0'} 2023-05-31T20:37:28.705 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000046d4b0bb4fec164d-006477b008-1630-a1'), ('Content-Length', '0'), ('Date', 'Wed, 31 May 2023 20:37:28 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:37:28.706 DEBUG:boto:path=/pklxkd-58/obj00000000000000000010 2023-05-31T20:37:28.706 DEBUG:boto:auth_path=/pklxkd-58/obj00000000000000000010 2023-05-31T20:37:28.707 DEBUG:boto:Method: PUT 2023-05-31T20:37:28.707 DEBUG:boto:Path: /pklxkd-58/obj00000000000000000010 2023-05-31T20:37:28.707 DEBUG:boto:Data: 2023-05-31T20:37:28.708 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-05-31T20:37:28.708 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:37:28.708 DEBUG:boto:Port: 8000 2023-05-31T20:37:28.709 DEBUG:boto:Params: {} 2023-05-31T20:37:28.709 DEBUG:boto:Token: None 2023-05-31T20:37:28.709 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Wed, 31 May 2023 20:37:28 GMT /pklxkd-58/obj00000000000000000010 2023-05-31T20:37:28.710 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:gEB6NaFprgrsCBkQDICdDC0FVQ8= 2023-05-31T20:37:28.710 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Wed, 31 May 2023 20:37:28 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:gEB6NaFprgrsCBkQDICdDC0FVQ8='} 2023-05-31T20:37:28.757 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000117968032d51ed70-006477b008-1630-a1'), ('Date', 'Wed, 31 May 2023 20:37:28 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:37:28.757 DEBUG:boto:path=/pklxkd-58/ 2023-05-31T20:37:28.758 DEBUG:boto:auth_path=/pklxkd-58/ 2023-05-31T20:37:28.758 DEBUG:boto:Method: HEAD 2023-05-31T20:37:28.759 DEBUG:boto:Path: /pklxkd-58/ 2023-05-31T20:37:28.759 DEBUG:boto:Data: 2023-05-31T20:37:28.759 DEBUG:boto:Headers: {} 2023-05-31T20:37:28.759 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:37:28.760 DEBUG:boto:Port: 8000 2023-05-31T20:37:28.761 DEBUG:boto:Params: {} 2023-05-31T20:37:28.761 DEBUG:boto:Token: None 2023-05-31T20:37:28.762 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:37:28 GMT /pklxkd-58/ 2023-05-31T20:37:28.762 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:nVb5lEQS0H6uLF/0uEEKACaIQMQ= 2023-05-31T20:37:28.762 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:37:28 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:nVb5lEQS0H6uLF/0uEEKACaIQMQ=', 'Content-Length': '0'} 2023-05-31T20:37:28.765 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '3'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000004283ed363c867e6e-006477b008-1630-a1'), ('Content-Length', '0'), ('Date', 'Wed, 31 May 2023 20:37:28 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:37:28.765 DEBUG:boto:path=/pklxkd-58/obj00000000000000000002 2023-05-31T20:37:28.766 DEBUG:boto:auth_path=/pklxkd-58/obj00000000000000000002 2023-05-31T20:37:28.766 DEBUG:boto:Method: PUT 2023-05-31T20:37:28.767 DEBUG:boto:Path: /pklxkd-58/obj00000000000000000002 2023-05-31T20:37:28.767 DEBUG:boto:Data: 2023-05-31T20:37:28.767 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-05-31T20:37:28.768 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:37:28.768 DEBUG:boto:Port: 8000 2023-05-31T20:37:28.769 DEBUG:boto:Params: {} 2023-05-31T20:37:28.770 DEBUG:boto:Token: None 2023-05-31T20:37:28.770 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Wed, 31 May 2023 20:37:28 GMT /pklxkd-58/obj00000000000000000002 2023-05-31T20:37:28.771 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:lTBgNe83/ZDwyXaIPvAqabICy3Q= 2023-05-31T20:37:28.771 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Wed, 31 May 2023 20:37:28 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:lTBgNe83/ZDwyXaIPvAqabICy3Q='} 2023-05-31T20:37:28.782 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000893737a4eb71dcfc-006477b008-1630-a1'), ('Date', 'Wed, 31 May 2023 20:37:28 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:37:28.783 DEBUG:boto:path=/pklxkd-58/ 2023-05-31T20:37:28.784 DEBUG:boto:auth_path=/pklxkd-58/ 2023-05-31T20:37:28.786 DEBUG:boto:Method: HEAD 2023-05-31T20:37:28.786 DEBUG:boto:Path: /pklxkd-58/ 2023-05-31T20:37:28.787 DEBUG:boto:Data: 2023-05-31T20:37:28.787 DEBUG:boto:Headers: {} 2023-05-31T20:37:28.788 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:37:28.788 DEBUG:boto:Port: 8000 2023-05-31T20:37:28.789 DEBUG:boto:Params: {} 2023-05-31T20:37:28.790 DEBUG:boto:Token: None 2023-05-31T20:37:28.790 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:37:28 GMT /pklxkd-58/ 2023-05-31T20:37:28.791 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:nVb5lEQS0H6uLF/0uEEKACaIQMQ= 2023-05-31T20:37:28.792 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:37:28 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:nVb5lEQS0H6uLF/0uEEKACaIQMQ=', 'Content-Length': '0'} 2023-05-31T20:37:28.794 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '2'), ('X-RGW-Bytes-Used', '6'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000002b20886a1d365c2c-006477b008-1630-a1'), ('Content-Length', '0'), ('Date', 'Wed, 31 May 2023 20:37:28 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:37:28.795 DEBUG:boto:path=/pklxkd-58/obj00000000000000000011 2023-05-31T20:37:28.796 DEBUG:boto:auth_path=/pklxkd-58/obj00000000000000000011 2023-05-31T20:37:28.796 DEBUG:boto:Method: PUT 2023-05-31T20:37:28.797 DEBUG:boto:Path: /pklxkd-58/obj00000000000000000011 2023-05-31T20:37:28.797 DEBUG:boto:Data: 2023-05-31T20:37:28.798 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-05-31T20:37:28.798 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:37:28.799 DEBUG:boto:Port: 8000 2023-05-31T20:37:28.799 DEBUG:boto:Params: {} 2023-05-31T20:37:28.799 DEBUG:boto:Token: None 2023-05-31T20:37:28.800 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Wed, 31 May 2023 20:37:28 GMT /pklxkd-58/obj00000000000000000011 2023-05-31T20:37:28.801 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:ykLtKIhuwVEIUCrXaIO39DW8o/E= 2023-05-31T20:37:28.801 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Wed, 31 May 2023 20:37:28 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:ykLtKIhuwVEIUCrXaIO39DW8o/E='} 2023-05-31T20:37:28.845 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000ca8774e4dbfc596d-006477b008-1630-a1'), ('Date', 'Wed, 31 May 2023 20:37:28 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:37:28.846 DEBUG:boto:path=/pklxkd-58/ 2023-05-31T20:37:28.847 DEBUG:boto:auth_path=/pklxkd-58/ 2023-05-31T20:37:28.849 DEBUG:boto:Method: HEAD 2023-05-31T20:37:28.850 DEBUG:boto:Path: /pklxkd-58/ 2023-05-31T20:37:28.851 DEBUG:boto:Data: 2023-05-31T20:37:28.852 DEBUG:boto:Headers: {} 2023-05-31T20:37:28.854 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:37:28.855 DEBUG:boto:Port: 8000 2023-05-31T20:37:28.857 DEBUG:boto:Params: {} 2023-05-31T20:37:28.858 DEBUG:boto:Token: None 2023-05-31T20:37:28.859 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:37:28 GMT /pklxkd-58/ 2023-05-31T20:37:28.860 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:nVb5lEQS0H6uLF/0uEEKACaIQMQ= 2023-05-31T20:37:28.861 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:37:28 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:nVb5lEQS0H6uLF/0uEEKACaIQMQ=', 'Content-Length': '0'} 2023-05-31T20:37:28.864 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '3'), ('X-RGW-Bytes-Used', '9'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000071fd793b284ff938-006477b008-1630-a1'), ('Content-Length', '0'), ('Date', 'Wed, 31 May 2023 20:37:28 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:37:28.865 DEBUG:boto:path=/pklxkd-58/obj00000000000000000001 2023-05-31T20:37:28.865 DEBUG:boto:auth_path=/pklxkd-58/obj00000000000000000001 2023-05-31T20:37:28.866 DEBUG:boto:Method: PUT 2023-05-31T20:37:28.866 DEBUG:boto:Path: /pklxkd-58/obj00000000000000000001 2023-05-31T20:37:28.866 DEBUG:boto:Data: 2023-05-31T20:37:28.867 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-05-31T20:37:28.867 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:37:28.867 DEBUG:boto:Port: 8000 2023-05-31T20:37:28.868 DEBUG:boto:Params: {} 2023-05-31T20:37:28.868 DEBUG:boto:Token: None 2023-05-31T20:37:28.869 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Wed, 31 May 2023 20:37:28 GMT /pklxkd-58/obj00000000000000000001 2023-05-31T20:37:28.869 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:JSe3J2ur4d4JNetIUxutGQFLtjM= 2023-05-31T20:37:28.869 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Wed, 31 May 2023 20:37:28 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:JSe3J2ur4d4JNetIUxutGQFLtjM='} 2023-05-31T20:37:28.878 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000932399f04012f37d-006477b008-1630-a1'), ('Date', 'Wed, 31 May 2023 20:37:28 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:37:28.879 DEBUG:boto:path=/pklxkd-58/ 2023-05-31T20:37:28.879 DEBUG:boto:auth_path=/pklxkd-58/ 2023-05-31T20:37:28.879 DEBUG:boto:Method: HEAD 2023-05-31T20:37:28.880 DEBUG:boto:Path: /pklxkd-58/ 2023-05-31T20:37:28.880 DEBUG:boto:Data: 2023-05-31T20:37:28.881 DEBUG:boto:Headers: {} 2023-05-31T20:37:28.881 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:37:28.881 DEBUG:boto:Port: 8000 2023-05-31T20:37:28.882 DEBUG:boto:Params: {} 2023-05-31T20:37:28.882 DEBUG:boto:Token: None 2023-05-31T20:37:28.882 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:37:28 GMT /pklxkd-58/ 2023-05-31T20:37:28.883 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:nVb5lEQS0H6uLF/0uEEKACaIQMQ= 2023-05-31T20:37:28.883 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:37:28 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:nVb5lEQS0H6uLF/0uEEKACaIQMQ=', 'Content-Length': '0'} 2023-05-31T20:37:28.886 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '4'), ('X-RGW-Bytes-Used', '12'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000001dfc8305e20a5224-006477b008-1630-a1'), ('Content-Length', '0'), ('Date', 'Wed, 31 May 2023 20:37:28 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:37:28.886 DEBUG:boto:path=/pklxkd-58/obj00000000000000000004 2023-05-31T20:37:28.887 DEBUG:boto:auth_path=/pklxkd-58/obj00000000000000000004 2023-05-31T20:37:28.887 DEBUG:boto:Method: PUT 2023-05-31T20:37:28.887 DEBUG:boto:Path: /pklxkd-58/obj00000000000000000004 2023-05-31T20:37:28.888 DEBUG:boto:Data: 2023-05-31T20:37:28.888 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-05-31T20:37:28.889 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:37:28.889 DEBUG:boto:Port: 8000 2023-05-31T20:37:28.889 DEBUG:boto:Params: {} 2023-05-31T20:37:28.890 DEBUG:boto:Token: None 2023-05-31T20:37:28.891 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Wed, 31 May 2023 20:37:28 GMT /pklxkd-58/obj00000000000000000004 2023-05-31T20:37:28.892 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:kfrgfvgNKwdljKW8ot7HDOXwgBc= 2023-05-31T20:37:28.894 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Wed, 31 May 2023 20:37:28 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:kfrgfvgNKwdljKW8ot7HDOXwgBc='} 2023-05-31T20:37:28.941 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000a08defb90ab5a85a-006477b008-1630-a1'), ('Date', 'Wed, 31 May 2023 20:37:28 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:37:28.942 DEBUG:boto:path=/pklxkd-58/ 2023-05-31T20:37:28.943 DEBUG:boto:auth_path=/pklxkd-58/ 2023-05-31T20:37:28.943 DEBUG:boto:Method: HEAD 2023-05-31T20:37:28.943 DEBUG:boto:Path: /pklxkd-58/ 2023-05-31T20:37:28.944 DEBUG:boto:Data: 2023-05-31T20:37:28.944 DEBUG:boto:Headers: {} 2023-05-31T20:37:28.945 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:37:28.945 DEBUG:boto:Port: 8000 2023-05-31T20:37:28.945 DEBUG:boto:Params: {} 2023-05-31T20:37:28.946 DEBUG:boto:Token: None 2023-05-31T20:37:28.946 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:37:28 GMT /pklxkd-58/ 2023-05-31T20:37:28.947 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:nVb5lEQS0H6uLF/0uEEKACaIQMQ= 2023-05-31T20:37:28.949 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:37:28 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:nVb5lEQS0H6uLF/0uEEKACaIQMQ=', 'Content-Length': '0'} 2023-05-31T20:37:28.951 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '5'), ('X-RGW-Bytes-Used', '15'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000920264856e23c066-006477b008-1630-a1'), ('Content-Length', '0'), ('Date', 'Wed, 31 May 2023 20:37:28 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:37:28.952 DEBUG:boto:path=/pklxkd-58/obj00000000000000000003 2023-05-31T20:37:28.952 DEBUG:boto:auth_path=/pklxkd-58/obj00000000000000000003 2023-05-31T20:37:28.953 DEBUG:boto:Method: PUT 2023-05-31T20:37:28.954 DEBUG:boto:Path: /pklxkd-58/obj00000000000000000003 2023-05-31T20:37:28.954 DEBUG:boto:Data: 2023-05-31T20:37:28.955 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-05-31T20:37:28.955 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:37:28.956 DEBUG:boto:Port: 8000 2023-05-31T20:37:28.956 DEBUG:boto:Params: {} 2023-05-31T20:37:28.957 DEBUG:boto:Token: None 2023-05-31T20:37:28.957 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Wed, 31 May 2023 20:37:28 GMT /pklxkd-58/obj00000000000000000003 2023-05-31T20:37:28.958 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:F4plCCaNa+NEcrRh93/Rw4gRcYM= 2023-05-31T20:37:28.959 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Wed, 31 May 2023 20:37:28 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:F4plCCaNa+NEcrRh93/Rw4gRcYM='} 2023-05-31T20:37:28.971 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000008c29239b111ef71d-006477b008-1630-a1'), ('Date', 'Wed, 31 May 2023 20:37:28 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:37:28.972 DEBUG:boto:path=/pklxkd-58/ 2023-05-31T20:37:28.973 DEBUG:boto:auth_path=/pklxkd-58/ 2023-05-31T20:37:28.973 DEBUG:boto:Method: HEAD 2023-05-31T20:37:28.974 DEBUG:boto:Path: /pklxkd-58/ 2023-05-31T20:37:28.975 DEBUG:boto:Data: 2023-05-31T20:37:28.975 DEBUG:boto:Headers: {} 2023-05-31T20:37:28.976 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:37:28.976 DEBUG:boto:Port: 8000 2023-05-31T20:37:28.976 DEBUG:boto:Params: {} 2023-05-31T20:37:28.977 DEBUG:boto:Token: None 2023-05-31T20:37:28.978 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:37:28 GMT /pklxkd-58/ 2023-05-31T20:37:28.979 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:nVb5lEQS0H6uLF/0uEEKACaIQMQ= 2023-05-31T20:37:28.979 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:37:28 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:nVb5lEQS0H6uLF/0uEEKACaIQMQ=', 'Content-Length': '0'} 2023-05-31T20:37:28.982 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '6'), ('X-RGW-Bytes-Used', '18'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000cd87173589f07945-006477b008-1630-a1'), ('Content-Length', '0'), ('Date', 'Wed, 31 May 2023 20:37:28 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:37:28.983 DEBUG:boto:path=/pklxkd-58/obj00000000000000000012 2023-05-31T20:37:28.983 DEBUG:boto:auth_path=/pklxkd-58/obj00000000000000000012 2023-05-31T20:37:28.984 DEBUG:boto:Method: PUT 2023-05-31T20:37:28.985 DEBUG:boto:Path: /pklxkd-58/obj00000000000000000012 2023-05-31T20:37:28.985 DEBUG:boto:Data: 2023-05-31T20:37:28.986 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-05-31T20:37:28.986 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:37:28.986 DEBUG:boto:Port: 8000 2023-05-31T20:37:28.987 DEBUG:boto:Params: {} 2023-05-31T20:37:28.987 DEBUG:boto:Token: None 2023-05-31T20:37:28.988 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Wed, 31 May 2023 20:37:28 GMT /pklxkd-58/obj00000000000000000012 2023-05-31T20:37:28.990 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:pMS31KvKFcEMYjvbDSrUrt/UjPo= 2023-05-31T20:37:28.990 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Wed, 31 May 2023 20:37:28 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:pMS31KvKFcEMYjvbDSrUrt/UjPo='} 2023-05-31T20:37:28.996 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000074b216b25d19a70a-006477b008-1630-a1'), ('Date', 'Wed, 31 May 2023 20:37:28 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:37:28.998 DEBUG:boto:path=/pklxkd-58/ 2023-05-31T20:37:28.998 DEBUG:boto:auth_path=/pklxkd-58/ 2023-05-31T20:37:28.999 DEBUG:boto:Method: HEAD 2023-05-31T20:37:28.999 DEBUG:boto:Path: /pklxkd-58/ 2023-05-31T20:37:28.999 DEBUG:boto:Data: 2023-05-31T20:37:29.000 DEBUG:boto:Headers: {} 2023-05-31T20:37:29.000 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:37:29.000 DEBUG:boto:Port: 8000 2023-05-31T20:37:29.001 DEBUG:boto:Params: {} 2023-05-31T20:37:29.002 DEBUG:boto:Token: None 2023-05-31T20:37:29.003 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:37:29 GMT /pklxkd-58/ 2023-05-31T20:37:29.004 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:C3CrqXHJXjHIUqfDoenSZqLJ08M= 2023-05-31T20:37:29.005 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:37:29 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:C3CrqXHJXjHIUqfDoenSZqLJ08M=', 'Content-Length': '0'} 2023-05-31T20:37:29.008 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '7'), ('X-RGW-Bytes-Used', '21'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000b81d8f6658efb12c-006477b009-1630-a1'), ('Content-Length', '0'), ('Date', 'Wed, 31 May 2023 20:37:29 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:37:29.009 DEBUG:boto:path=/pklxkd-58/obj00000000000000000006 2023-05-31T20:37:29.009 DEBUG:boto:auth_path=/pklxkd-58/obj00000000000000000006 2023-05-31T20:37:29.009 DEBUG:boto:Method: PUT 2023-05-31T20:37:29.010 DEBUG:boto:Path: /pklxkd-58/obj00000000000000000006 2023-05-31T20:37:29.010 DEBUG:boto:Data: 2023-05-31T20:37:29.011 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-05-31T20:37:29.011 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:37:29.011 DEBUG:boto:Port: 8000 2023-05-31T20:37:29.012 DEBUG:boto:Params: {} 2023-05-31T20:37:29.012 DEBUG:boto:Token: None 2023-05-31T20:37:29.013 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Wed, 31 May 2023 20:37:29 GMT /pklxkd-58/obj00000000000000000006 2023-05-31T20:37:29.013 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:MxNG2f1ygjYpdHb30ebWa9wsy7M= 2023-05-31T20:37:29.014 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Wed, 31 May 2023 20:37:29 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:MxNG2f1ygjYpdHb30ebWa9wsy7M='} 2023-05-31T20:37:29.061 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000009c0fadb0226bf224-006477b009-1630-a1'), ('Date', 'Wed, 31 May 2023 20:37:29 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:37:29.062 DEBUG:boto:path=/pklxkd-58/ 2023-05-31T20:37:29.062 DEBUG:boto:auth_path=/pklxkd-58/ 2023-05-31T20:37:29.063 DEBUG:boto:Method: HEAD 2023-05-31T20:37:29.063 DEBUG:boto:Path: /pklxkd-58/ 2023-05-31T20:37:29.064 DEBUG:boto:Data: 2023-05-31T20:37:29.065 DEBUG:boto:Headers: {} 2023-05-31T20:37:29.065 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:37:29.066 DEBUG:boto:Port: 8000 2023-05-31T20:37:29.066 DEBUG:boto:Params: {} 2023-05-31T20:37:29.066 DEBUG:boto:Token: None 2023-05-31T20:37:29.067 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:37:29 GMT /pklxkd-58/ 2023-05-31T20:37:29.067 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:C3CrqXHJXjHIUqfDoenSZqLJ08M= 2023-05-31T20:37:29.069 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:37:29 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:C3CrqXHJXjHIUqfDoenSZqLJ08M=', 'Content-Length': '0'} 2023-05-31T20:37:29.072 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '8'), ('X-RGW-Bytes-Used', '24'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000068aa24e6d21251ba-006477b009-1630-a1'), ('Content-Length', '0'), ('Date', 'Wed, 31 May 2023 20:37:29 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:37:29.072 DEBUG:boto:path=/pklxkd-58/obj00000000000000000013 2023-05-31T20:37:29.073 DEBUG:boto:auth_path=/pklxkd-58/obj00000000000000000013 2023-05-31T20:37:29.073 DEBUG:boto:Method: PUT 2023-05-31T20:37:29.074 DEBUG:boto:Path: /pklxkd-58/obj00000000000000000013 2023-05-31T20:37:29.074 DEBUG:boto:Data: 2023-05-31T20:37:29.074 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-05-31T20:37:29.075 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:37:29.076 DEBUG:boto:Port: 8000 2023-05-31T20:37:29.076 DEBUG:boto:Params: {} 2023-05-31T20:37:29.077 DEBUG:boto:Token: None 2023-05-31T20:37:29.078 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Wed, 31 May 2023 20:37:29 GMT /pklxkd-58/obj00000000000000000013 2023-05-31T20:37:29.078 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:2mU1EHMU933HSZz7+540SMEGyLk= 2023-05-31T20:37:29.078 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Wed, 31 May 2023 20:37:29 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:2mU1EHMU933HSZz7+540SMEGyLk='} 2023-05-31T20:37:29.084 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000d0cc40eee04878bf-006477b009-1630-a1'), ('Date', 'Wed, 31 May 2023 20:37:29 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:37:29.085 DEBUG:boto:path=/pklxkd-58/ 2023-05-31T20:37:29.086 DEBUG:boto:auth_path=/pklxkd-58/ 2023-05-31T20:37:29.088 DEBUG:boto:Method: HEAD 2023-05-31T20:37:29.089 DEBUG:boto:Path: /pklxkd-58/ 2023-05-31T20:37:29.089 DEBUG:boto:Data: 2023-05-31T20:37:29.089 DEBUG:boto:Headers: {} 2023-05-31T20:37:29.090 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:37:29.090 DEBUG:boto:Port: 8000 2023-05-31T20:37:29.091 DEBUG:boto:Params: {} 2023-05-31T20:37:29.091 DEBUG:boto:Token: None 2023-05-31T20:37:29.092 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:37:29 GMT /pklxkd-58/ 2023-05-31T20:37:29.092 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:C3CrqXHJXjHIUqfDoenSZqLJ08M= 2023-05-31T20:37:29.093 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:37:29 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:C3CrqXHJXjHIUqfDoenSZqLJ08M=', 'Content-Length': '0'} 2023-05-31T20:37:29.097 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '9'), ('X-RGW-Bytes-Used', '27'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000a50be6d94c947c35-006477b009-1630-a1'), ('Content-Length', '0'), ('Date', 'Wed, 31 May 2023 20:37:29 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:37:29.097 DEBUG:boto:path=/pklxkd-58/obj00000000000000000000 2023-05-31T20:37:29.098 DEBUG:boto:auth_path=/pklxkd-58/obj00000000000000000000 2023-05-31T20:37:29.099 DEBUG:boto:Method: PUT 2023-05-31T20:37:29.099 DEBUG:boto:Path: /pklxkd-58/obj00000000000000000000 2023-05-31T20:37:29.100 DEBUG:boto:Data: 2023-05-31T20:37:29.100 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-05-31T20:37:29.101 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:37:29.101 DEBUG:boto:Port: 8000 2023-05-31T20:37:29.101 DEBUG:boto:Params: {} 2023-05-31T20:37:29.102 DEBUG:boto:Token: None 2023-05-31T20:37:29.102 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Wed, 31 May 2023 20:37:29 GMT /pklxkd-58/obj00000000000000000000 2023-05-31T20:37:29.103 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:zEceMVCNKtwzCscN/RfHWF6zf5E= 2023-05-31T20:37:29.104 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Wed, 31 May 2023 20:37:29 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:zEceMVCNKtwzCscN/RfHWF6zf5E='} 2023-05-31T20:37:29.153 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000005baf3baeac33323a-006477b009-1630-a1'), ('Date', 'Wed, 31 May 2023 20:37:29 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:37:29.155 DEBUG:rgw_multi.tests:bucket checkpoint: source=a1 target=a2 bucket=pklxkd-58 2023-05-31T20:37:29.155 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'pklxkd-58', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-05-31T20:37:29.156 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'pklxkd-58', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-05-31T20:37:29.157 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket pklxkd-58 --source-zone a1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2023-05-31T20:37:29.215 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2023-05-31T20:37:29.216 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2023-05-31T20:37:29.255 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:37:29.248+0000 7feec8105780 1 waiting to reach incremental sync.. 2023-05-31T20:37:34.253 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:37:34.249+0000 7feec8105780 1 waiting to reach incremental sync.. 2023-05-31T20:37:39.254 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:37:39.250+0000 7feec8105780 1 waiting to reach incremental sync.. 2023-05-31T20:37:44.254 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:37:44.249+0000 7feec8105780 1 waiting to reach incremental sync.. 2023-05-31T20:37:49.256 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:37:49.251+0000 7feec8105780 1 bucket sync caught up with source: 2023-05-31T20:37:49.257 INFO:teuthology.orchestra.run.smithi099.stderr: local status: [00000000001.41.6, , 00000000001.37.6, 00000000001.33.6, 00000000001.33.6, 00000000001.33.6, 00000000001.38.6, 00000000001.35.6, 00000000001.39.6, 00000000001.30.6, 00000000001.25.6] 2023-05-31T20:37:49.257 INFO:teuthology.orchestra.run.smithi099.stderr: remote markers: [00000000001.41.6, , 00000000001.37.6, 00000000001.33.6, 00000000001.33.6, 00000000001.33.6, 00000000001.38.6, 00000000001.35.6, 00000000001.39.6, 00000000001.30.6, 00000000001.25.6] 2023-05-31T20:37:49.257 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:37:49.251+0000 7feec8105780 0 bucket checkpoint complete 2023-05-31T20:37:49.277 DEBUG:rgw_multi.tests:bucket checkpoint: source=a2 target=a1 bucket=pklxkd-58 2023-05-31T20:37:49.278 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'pklxkd-58', '--source-zone', 'a2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-05-31T20:37:49.278 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'pklxkd-58', '--source-zone', 'a2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-05-31T20:37:49.279 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket pklxkd-58 --source-zone a2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2023-05-31T20:37:49.340 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2023-05-31T20:37:49.341 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2023-05-31T20:37:49.395 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:37:49.390+0000 7f19ed794780 1 waiting to reach incremental sync.. 2023-05-31T20:37:54.397 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:37:54.392+0000 7f19ed794780 1 bucket sync caught up with source: 2023-05-31T20:37:54.397 INFO:teuthology.orchestra.run.smithi099.stderr: local status: [00000000001.2251.6, , 00000000001.1913.6, 00000000001.776.6, 00000000001.1935.6, 00000000001.1922.6, 00000000001.1676.6, 00000000001.1934.6, 00000000001.1047.6, 00000000001.378.6, 00000000001.976.6] 2023-05-31T20:37:54.398 INFO:teuthology.orchestra.run.smithi099.stderr: remote markers: [00000000001.2251.6, , 00000000001.1913.6, 00000000001.776.6, 00000000001.1935.6, 00000000001.1922.6, 00000000001.1676.6, 00000000001.1934.6, 00000000001.1047.6, 00000000001.378.6, 00000000001.976.6] 2023-05-31T20:37:54.398 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:37:54.392+0000 7f19ed794780 0 bucket checkpoint complete 2023-05-31T20:37:54.416 INFO:tasks.rgw_multi.zone_rados:comparing bucket=pklxkd-58 zones={a2, a1} 2023-05-31T20:37:54.417 DEBUG:boto:path=/pklxkd-58/ 2023-05-31T20:37:54.417 DEBUG:boto:auth_path=/pklxkd-58/ 2023-05-31T20:37:54.418 DEBUG:boto:Method: HEAD 2023-05-31T20:37:54.418 DEBUG:boto:Path: /pklxkd-58/ 2023-05-31T20:37:54.419 DEBUG:boto:Data: 2023-05-31T20:37:54.420 DEBUG:boto:Headers: {} 2023-05-31T20:37:54.420 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:37:54.422 DEBUG:boto:Port: 8001 2023-05-31T20:37:54.423 DEBUG:boto:Params: {} 2023-05-31T20:37:54.424 DEBUG:boto:Token: None 2023-05-31T20:37:54.425 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:37:54 GMT /pklxkd-58/ 2023-05-31T20:37:54.426 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:GyEACRRdHTZwoJXcwwSEpYR5dkI= 2023-05-31T20:37:54.427 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:37:54 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:GyEACRRdHTZwoJXcwwSEpYR5dkI=', 'Content-Length': '0'} 2023-05-31T20:37:54.430 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '10'), ('X-RGW-Bytes-Used', '30'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000b3e931e555f3a292-006477b022-1b4f-a2'), ('Content-Length', '0'), ('Date', 'Wed, 31 May 2023 20:37:54 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:37:54.431 DEBUG:boto:path=/pklxkd-58/ 2023-05-31T20:37:54.431 DEBUG:boto:auth_path=/pklxkd-58/ 2023-05-31T20:37:54.432 DEBUG:boto:Method: HEAD 2023-05-31T20:37:54.432 DEBUG:boto:Path: /pklxkd-58/ 2023-05-31T20:37:54.433 DEBUG:boto:Data: 2023-05-31T20:37:54.433 DEBUG:boto:Headers: {} 2023-05-31T20:37:54.434 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:37:54.435 DEBUG:boto:Port: 8000 2023-05-31T20:37:54.435 DEBUG:boto:Params: {} 2023-05-31T20:37:54.436 DEBUG:boto:Token: None 2023-05-31T20:37:54.436 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:37:54 GMT /pklxkd-58/ 2023-05-31T20:37:54.437 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:GyEACRRdHTZwoJXcwwSEpYR5dkI= 2023-05-31T20:37:54.438 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:37:54 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:GyEACRRdHTZwoJXcwwSEpYR5dkI=', 'Content-Length': '0'} 2023-05-31T20:37:54.441 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '10'), ('X-RGW-Bytes-Used', '30'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000213bbcafacd2fe9d-006477b022-1630-a1'), ('Content-Length', '0'), ('Date', 'Wed, 31 May 2023 20:37:54 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:37:54.442 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2023-05-31T20:37:54.443 DEBUG:boto:path=/pklxkd-58/ 2023-05-31T20:37:54.444 DEBUG:boto:auth_path=/pklxkd-58/ 2023-05-31T20:37:54.444 DEBUG:boto:path=/pklxkd-58/?versions&max-keys=999 2023-05-31T20:37:54.445 DEBUG:boto:auth_path=/pklxkd-58/?versions&max-keys=999 2023-05-31T20:37:54.446 DEBUG:boto:Method: GET 2023-05-31T20:37:54.446 DEBUG:boto:Path: /pklxkd-58/?versions&max-keys=999 2023-05-31T20:37:54.446 DEBUG:boto:Data: 2023-05-31T20:37:54.447 DEBUG:boto:Headers: {} 2023-05-31T20:37:54.447 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:37:54.447 DEBUG:boto:Port: 8001 2023-05-31T20:37:54.448 DEBUG:boto:Params: {} 2023-05-31T20:37:54.448 DEBUG:boto:Token: None 2023-05-31T20:37:54.449 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:37:54 GMT /pklxkd-58/?versions 2023-05-31T20:37:54.449 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:LXLKV6SpqffBVsdUKtIJ0CUnylk= 2023-05-31T20:37:54.449 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:37:54 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:LXLKV6SpqffBVsdUKtIJ0CUnylk=', 'Content-Length': '0'} 2023-05-31T20:37:54.452 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000db5f69babcb2b290-006477b022-1b4f-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:37:54 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:37:54.453 DEBUG:boto:b'pklxkd-58999falseobj00000000000000000000nulltrue2023-05-31T20:37:29.102Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000001nulltrue2023-05-31T20:37:28.869Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000002nulltrue2023-05-31T20:37:28.769Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000003nulltrue2023-05-31T20:37:28.957Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000004nulltrue2023-05-31T20:37:28.895Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000006nulltrue2023-05-31T20:37:29.012Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000010nulltrue2023-05-31T20:37:28.709Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000011nulltrue2023-05-31T20:37:28.799Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000012nulltrue2023-05-31T20:37:28.988Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000013nulltrue2023-05-31T20:37:29.076Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:37:54.454 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000000 2023-05-31T20:37:54.454 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000001 2023-05-31T20:37:54.455 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000002 2023-05-31T20:37:54.455 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000003 2023-05-31T20:37:54.455 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000004 2023-05-31T20:37:54.456 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000006 2023-05-31T20:37:54.456 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000010 2023-05-31T20:37:54.456 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000011 2023-05-31T20:37:54.457 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000012 2023-05-31T20:37:54.457 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000013 2023-05-31T20:37:54.457 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2023-05-31T20:37:54.458 DEBUG:boto:path=/pklxkd-58/ 2023-05-31T20:37:54.458 DEBUG:boto:auth_path=/pklxkd-58/ 2023-05-31T20:37:54.458 DEBUG:boto:path=/pklxkd-58/?versions&max-keys=999 2023-05-31T20:37:54.459 DEBUG:boto:auth_path=/pklxkd-58/?versions&max-keys=999 2023-05-31T20:37:54.459 DEBUG:boto:Method: GET 2023-05-31T20:37:54.459 DEBUG:boto:Path: /pklxkd-58/?versions&max-keys=999 2023-05-31T20:37:54.460 DEBUG:boto:Data: 2023-05-31T20:37:54.461 DEBUG:boto:Headers: {} 2023-05-31T20:37:54.461 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:37:54.462 DEBUG:boto:Port: 8000 2023-05-31T20:37:54.462 DEBUG:boto:Params: {} 2023-05-31T20:37:54.462 DEBUG:boto:Token: None 2023-05-31T20:37:54.463 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:37:54 GMT /pklxkd-58/?versions 2023-05-31T20:37:54.463 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:LXLKV6SpqffBVsdUKtIJ0CUnylk= 2023-05-31T20:37:54.464 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:37:54 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:LXLKV6SpqffBVsdUKtIJ0CUnylk=', 'Content-Length': '0'} 2023-05-31T20:37:54.466 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000024420619c435785b-006477b022-1630-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:37:54 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:37:54.467 DEBUG:boto:b'pklxkd-58999falseobj00000000000000000000nulltrue2023-05-31T20:37:29.102Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000001nulltrue2023-05-31T20:37:28.869Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000002nulltrue2023-05-31T20:37:28.769Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000003nulltrue2023-05-31T20:37:28.957Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000004nulltrue2023-05-31T20:37:28.895Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000006nulltrue2023-05-31T20:37:29.012Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000010nulltrue2023-05-31T20:37:28.709Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000011nulltrue2023-05-31T20:37:28.799Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000012nulltrue2023-05-31T20:37:28.988Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000013nulltrue2023-05-31T20:37:29.076Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:37:54.468 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000000 2023-05-31T20:37:54.468 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000001 2023-05-31T20:37:54.469 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000002 2023-05-31T20:37:54.469 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000003 2023-05-31T20:37:54.469 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000004 2023-05-31T20:37:54.470 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000006 2023-05-31T20:37:54.470 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000010 2023-05-31T20:37:54.470 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000011 2023-05-31T20:37:54.470 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000012 2023-05-31T20:37:54.471 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000013 2023-05-31T20:37:54.472 DEBUG:boto:path=/pklxkd-58/ 2023-05-31T20:37:54.472 DEBUG:boto:auth_path=/pklxkd-58/ 2023-05-31T20:37:54.473 DEBUG:boto:path=/pklxkd-58/?versions&max-keys=999 2023-05-31T20:37:54.473 DEBUG:boto:auth_path=/pklxkd-58/?versions&max-keys=999 2023-05-31T20:37:54.473 DEBUG:boto:Method: GET 2023-05-31T20:37:54.474 DEBUG:boto:Path: /pklxkd-58/?versions&max-keys=999 2023-05-31T20:37:54.474 DEBUG:boto:Data: 2023-05-31T20:37:54.474 DEBUG:boto:Headers: {} 2023-05-31T20:37:54.475 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:37:54.475 DEBUG:boto:Port: 8001 2023-05-31T20:37:54.475 DEBUG:boto:Params: {} 2023-05-31T20:37:54.476 DEBUG:boto:Token: None 2023-05-31T20:37:54.476 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:37:54 GMT /pklxkd-58/?versions 2023-05-31T20:37:54.477 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:LXLKV6SpqffBVsdUKtIJ0CUnylk= 2023-05-31T20:37:54.477 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:37:54 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:LXLKV6SpqffBVsdUKtIJ0CUnylk=', 'Content-Length': '0'} 2023-05-31T20:37:54.480 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000000886ffc28be6c01e-006477b022-1b4f-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:37:54 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:37:54.480 DEBUG:boto:b'pklxkd-58999falseobj00000000000000000000nulltrue2023-05-31T20:37:29.102Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000001nulltrue2023-05-31T20:37:28.869Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000002nulltrue2023-05-31T20:37:28.769Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000003nulltrue2023-05-31T20:37:28.957Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000004nulltrue2023-05-31T20:37:28.895Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000006nulltrue2023-05-31T20:37:29.012Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000010nulltrue2023-05-31T20:37:28.709Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000011nulltrue2023-05-31T20:37:28.799Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000012nulltrue2023-05-31T20:37:28.988Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000013nulltrue2023-05-31T20:37:29.076Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:37:54.482 DEBUG:boto:path=/pklxkd-58/ 2023-05-31T20:37:54.483 DEBUG:boto:auth_path=/pklxkd-58/ 2023-05-31T20:37:54.483 DEBUG:boto:path=/pklxkd-58/?versions&max-keys=999 2023-05-31T20:37:54.484 DEBUG:boto:auth_path=/pklxkd-58/?versions&max-keys=999 2023-05-31T20:37:54.484 DEBUG:boto:Method: GET 2023-05-31T20:37:54.484 DEBUG:boto:Path: /pklxkd-58/?versions&max-keys=999 2023-05-31T20:37:54.485 DEBUG:boto:Data: 2023-05-31T20:37:54.485 DEBUG:boto:Headers: {} 2023-05-31T20:37:54.485 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:37:54.486 DEBUG:boto:Port: 8000 2023-05-31T20:37:54.486 DEBUG:boto:Params: {} 2023-05-31T20:37:54.486 DEBUG:boto:Token: None 2023-05-31T20:37:54.487 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:37:54 GMT /pklxkd-58/?versions 2023-05-31T20:37:54.487 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:LXLKV6SpqffBVsdUKtIJ0CUnylk= 2023-05-31T20:37:54.488 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:37:54 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:LXLKV6SpqffBVsdUKtIJ0CUnylk=', 'Content-Length': '0'} 2023-05-31T20:37:54.490 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000043e9d43c4934603-006477b022-1630-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:37:54 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:37:54.491 DEBUG:boto:b'pklxkd-58999falseobj00000000000000000000nulltrue2023-05-31T20:37:29.102Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000001nulltrue2023-05-31T20:37:28.869Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000002nulltrue2023-05-31T20:37:28.769Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000003nulltrue2023-05-31T20:37:28.957Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000004nulltrue2023-05-31T20:37:28.895Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000006nulltrue2023-05-31T20:37:29.012Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000010nulltrue2023-05-31T20:37:28.709Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000011nulltrue2023-05-31T20:37:28.799Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000012nulltrue2023-05-31T20:37:28.988Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000013nulltrue2023-05-31T20:37:29.076Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:37:54.492 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000000 2023-05-31T20:37:54.493 DEBUG:boto:path=/pklxkd-58/obj00000000000000000000 2023-05-31T20:37:54.493 DEBUG:boto:auth_path=/pklxkd-58/obj00000000000000000000 2023-05-31T20:37:54.493 DEBUG:boto:path=/pklxkd-58/obj00000000000000000000?versionId=null 2023-05-31T20:37:54.494 DEBUG:boto:auth_path=/pklxkd-58/obj00000000000000000000?versionId=null 2023-05-31T20:37:54.494 DEBUG:boto:Method: GET 2023-05-31T20:37:54.494 DEBUG:boto:Path: /pklxkd-58/obj00000000000000000000?versionId=null 2023-05-31T20:37:54.495 DEBUG:boto:Data: 2023-05-31T20:37:54.495 DEBUG:boto:Headers: {} 2023-05-31T20:37:54.495 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:37:54.496 DEBUG:boto:Port: 8001 2023-05-31T20:37:54.496 DEBUG:boto:Params: {} 2023-05-31T20:37:54.497 DEBUG:boto:Token: None 2023-05-31T20:37:54.497 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:37:54 GMT /pklxkd-58/obj00000000000000000000?versionId=null 2023-05-31T20:37:54.498 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:GkFpvB2MSHqFiQLCD0UZYVzjT3E= 2023-05-31T20:37:54.498 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:37:54 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:GkFpvB2MSHqFiQLCD0UZYVzjT3E=', 'Content-Length': '0'} 2023-05-31T20:37:54.500 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:37:29 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000215138273136d5be-006477b022-1b4f-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:37:54 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:37:54.501 DEBUG:boto:path=/pklxkd-58/obj00000000000000000000 2023-05-31T20:37:54.501 DEBUG:boto:auth_path=/pklxkd-58/obj00000000000000000000 2023-05-31T20:37:54.501 DEBUG:boto:path=/pklxkd-58/obj00000000000000000000?versionId=null 2023-05-31T20:37:54.502 DEBUG:boto:auth_path=/pklxkd-58/obj00000000000000000000?versionId=null 2023-05-31T20:37:54.503 DEBUG:boto:Method: GET 2023-05-31T20:37:54.503 DEBUG:boto:Path: /pklxkd-58/obj00000000000000000000?versionId=null 2023-05-31T20:37:54.503 DEBUG:boto:Data: 2023-05-31T20:37:54.504 DEBUG:boto:Headers: {} 2023-05-31T20:37:54.504 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:37:54.504 DEBUG:boto:Port: 8000 2023-05-31T20:37:54.505 DEBUG:boto:Params: {} 2023-05-31T20:37:54.505 DEBUG:boto:Token: None 2023-05-31T20:37:54.506 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:37:54 GMT /pklxkd-58/obj00000000000000000000?versionId=null 2023-05-31T20:37:54.506 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:GkFpvB2MSHqFiQLCD0UZYVzjT3E= 2023-05-31T20:37:54.507 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:37:54 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:GkFpvB2MSHqFiQLCD0UZYVzjT3E=', 'Content-Length': '0'} 2023-05-31T20:37:54.509 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:37:29 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000cf182aa23ad09dbd-006477b022-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:37:54 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:37:54.509 DEBUG:boto:path=/pklxkd-58/obj00000000000000000000 2023-05-31T20:37:54.510 DEBUG:boto:auth_path=/pklxkd-58/obj00000000000000000000 2023-05-31T20:37:54.510 DEBUG:boto:path=/pklxkd-58/obj00000000000000000000?versionId=null 2023-05-31T20:37:54.510 DEBUG:boto:auth_path=/pklxkd-58/obj00000000000000000000?versionId=null 2023-05-31T20:37:54.511 DEBUG:boto:Method: HEAD 2023-05-31T20:37:54.511 DEBUG:boto:Path: /pklxkd-58/obj00000000000000000000?versionId=null 2023-05-31T20:37:54.511 DEBUG:boto:Data: 2023-05-31T20:37:54.512 DEBUG:boto:Headers: {} 2023-05-31T20:37:54.512 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:37:54.512 DEBUG:boto:Port: 8001 2023-05-31T20:37:54.513 DEBUG:boto:Params: {} 2023-05-31T20:37:54.513 DEBUG:boto:Token: None 2023-05-31T20:37:54.514 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:37:54 GMT /pklxkd-58/obj00000000000000000000?versionId=null 2023-05-31T20:37:54.514 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:EmE636LQGe2846jiKSR5A5kzLPo= 2023-05-31T20:37:54.515 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:37:54 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:EmE636LQGe2846jiKSR5A5kzLPo=', 'Content-Length': '0'} 2023-05-31T20:37:54.517 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:37:29 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001df6afd06546e427-006477b022-1b4f-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:37:54 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:37:54.517 DEBUG:boto:path=/pklxkd-58/obj00000000000000000000 2023-05-31T20:37:54.518 DEBUG:boto:auth_path=/pklxkd-58/obj00000000000000000000 2023-05-31T20:37:54.518 DEBUG:boto:path=/pklxkd-58/obj00000000000000000000?versionId=null 2023-05-31T20:37:54.518 DEBUG:boto:auth_path=/pklxkd-58/obj00000000000000000000?versionId=null 2023-05-31T20:37:54.519 DEBUG:boto:Method: HEAD 2023-05-31T20:37:54.519 DEBUG:boto:Path: /pklxkd-58/obj00000000000000000000?versionId=null 2023-05-31T20:37:54.519 DEBUG:boto:Data: 2023-05-31T20:37:54.520 DEBUG:boto:Headers: {} 2023-05-31T20:37:54.520 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:37:54.520 DEBUG:boto:Port: 8000 2023-05-31T20:37:54.521 DEBUG:boto:Params: {} 2023-05-31T20:37:54.521 DEBUG:boto:Token: None 2023-05-31T20:37:54.521 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:37:54 GMT /pklxkd-58/obj00000000000000000000?versionId=null 2023-05-31T20:37:54.522 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:EmE636LQGe2846jiKSR5A5kzLPo= 2023-05-31T20:37:54.522 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:37:54 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:EmE636LQGe2846jiKSR5A5kzLPo=', 'Content-Length': '0'} 2023-05-31T20:37:54.524 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:37:29 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c5f0fac587fee91d-006477b022-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:37:54 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:37:54.525 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000000 2023-05-31T20:37:54.525 DEBUG:boto:path=/pklxkd-58/obj00000000000000000000 2023-05-31T20:37:54.525 DEBUG:boto:auth_path=/pklxkd-58/obj00000000000000000000 2023-05-31T20:37:54.526 DEBUG:boto:path=/pklxkd-58/obj00000000000000000000?versionId=null 2023-05-31T20:37:54.526 DEBUG:boto:auth_path=/pklxkd-58/obj00000000000000000000?versionId=null 2023-05-31T20:37:54.526 DEBUG:boto:Method: GET 2023-05-31T20:37:54.527 DEBUG:boto:Path: /pklxkd-58/obj00000000000000000000?versionId=null 2023-05-31T20:37:54.527 DEBUG:boto:Data: 2023-05-31T20:37:54.527 DEBUG:boto:Headers: {} 2023-05-31T20:37:54.528 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:37:54.528 DEBUG:boto:Port: 8001 2023-05-31T20:37:54.529 DEBUG:boto:Params: {} 2023-05-31T20:37:54.529 DEBUG:boto:Token: None 2023-05-31T20:37:54.529 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:37:54 GMT /pklxkd-58/obj00000000000000000000?versionId=null 2023-05-31T20:37:54.530 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:GkFpvB2MSHqFiQLCD0UZYVzjT3E= 2023-05-31T20:37:54.530 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:37:54 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:GkFpvB2MSHqFiQLCD0UZYVzjT3E=', 'Content-Length': '0'} 2023-05-31T20:37:54.532 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:37:29 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000070bacc5bac93a0d2-006477b022-1b4f-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:37:54 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:37:54.580 DEBUG:boto:path=/pklxkd-58/obj00000000000000000000 2023-05-31T20:37:54.581 DEBUG:boto:auth_path=/pklxkd-58/obj00000000000000000000 2023-05-31T20:37:54.581 DEBUG:boto:path=/pklxkd-58/obj00000000000000000000?versionId=null 2023-05-31T20:37:54.582 DEBUG:boto:auth_path=/pklxkd-58/obj00000000000000000000?versionId=null 2023-05-31T20:37:54.582 DEBUG:boto:Method: GET 2023-05-31T20:37:54.582 DEBUG:boto:Path: /pklxkd-58/obj00000000000000000000?versionId=null 2023-05-31T20:37:54.583 DEBUG:boto:Data: 2023-05-31T20:37:54.583 DEBUG:boto:Headers: {} 2023-05-31T20:37:54.583 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:37:54.584 DEBUG:boto:Port: 8000 2023-05-31T20:37:54.584 DEBUG:boto:Params: {} 2023-05-31T20:37:54.585 DEBUG:boto:Token: None 2023-05-31T20:37:54.585 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:37:54 GMT /pklxkd-58/obj00000000000000000000?versionId=null 2023-05-31T20:37:54.585 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:GkFpvB2MSHqFiQLCD0UZYVzjT3E= 2023-05-31T20:37:54.586 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:37:54 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:GkFpvB2MSHqFiQLCD0UZYVzjT3E=', 'Content-Length': '0'} 2023-05-31T20:37:54.588 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:37:29 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002e088c872e626689-006477b022-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:37:54 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:37:54.589 DEBUG:boto:path=/pklxkd-58/ 2023-05-31T20:37:54.589 DEBUG:boto:auth_path=/pklxkd-58/ 2023-05-31T20:37:54.589 DEBUG:boto:path=/pklxkd-58/?prefix=obj00000000000000000000 2023-05-31T20:37:54.590 DEBUG:boto:auth_path=/pklxkd-58/?prefix=obj00000000000000000000 2023-05-31T20:37:54.590 DEBUG:boto:Method: GET 2023-05-31T20:37:54.590 DEBUG:boto:Path: /pklxkd-58/?prefix=obj00000000000000000000 2023-05-31T20:37:54.591 DEBUG:boto:Data: 2023-05-31T20:37:54.591 DEBUG:boto:Headers: {} 2023-05-31T20:37:54.591 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:37:54.592 DEBUG:boto:Port: 8001 2023-05-31T20:37:54.592 DEBUG:boto:Params: {} 2023-05-31T20:37:54.593 DEBUG:boto:Token: None 2023-05-31T20:37:54.593 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:37:54 GMT /pklxkd-58/ 2023-05-31T20:37:54.593 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:kBU/21bIG8o+fKSyagNAMS+H4qE= 2023-05-31T20:37:54.594 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:37:54 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:kBU/21bIG8o+fKSyagNAMS+H4qE=', 'Content-Length': '0'} 2023-05-31T20:37:54.596 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000092f388df19720fbc-006477b022-1b4f-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:37:54 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:37:54.597 DEBUG:boto:b'pklxkd-58obj000000000000000000001000falseobj000000000000000000002023-05-31T20:37:29.102Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:37:54.598 DEBUG:boto:path=/pklxkd-58/ 2023-05-31T20:37:54.598 DEBUG:boto:auth_path=/pklxkd-58/ 2023-05-31T20:37:54.598 DEBUG:boto:path=/pklxkd-58/?prefix=obj00000000000000000000 2023-05-31T20:37:54.599 DEBUG:boto:auth_path=/pklxkd-58/?prefix=obj00000000000000000000 2023-05-31T20:37:54.599 DEBUG:boto:Method: GET 2023-05-31T20:37:54.600 DEBUG:boto:Path: /pklxkd-58/?prefix=obj00000000000000000000 2023-05-31T20:37:54.600 DEBUG:boto:Data: 2023-05-31T20:37:54.600 DEBUG:boto:Headers: {} 2023-05-31T20:37:54.601 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:37:54.601 DEBUG:boto:Port: 8000 2023-05-31T20:37:54.601 DEBUG:boto:Params: {} 2023-05-31T20:37:54.602 DEBUG:boto:Token: None 2023-05-31T20:37:54.602 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:37:54 GMT /pklxkd-58/ 2023-05-31T20:37:54.603 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:kBU/21bIG8o+fKSyagNAMS+H4qE= 2023-05-31T20:37:54.603 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:37:54 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:kBU/21bIG8o+fKSyagNAMS+H4qE=', 'Content-Length': '0'} 2023-05-31T20:37:54.605 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000008cc32fd891e9fa02-006477b022-1630-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:37:54 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:37:54.606 DEBUG:boto:b'pklxkd-58obj000000000000000000001000falseobj000000000000000000002023-05-31T20:37:29.102Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:37:54.606 DEBUG:boto:path=/pklxkd-58/obj00000000000000000000 2023-05-31T20:37:54.607 DEBUG:boto:auth_path=/pklxkd-58/obj00000000000000000000 2023-05-31T20:37:54.607 DEBUG:boto:Method: HEAD 2023-05-31T20:37:54.608 DEBUG:boto:Path: /pklxkd-58/obj00000000000000000000 2023-05-31T20:37:54.608 DEBUG:boto:Data: 2023-05-31T20:37:54.608 DEBUG:boto:Headers: {} 2023-05-31T20:37:54.609 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:37:54.609 DEBUG:boto:Port: 8001 2023-05-31T20:37:54.609 DEBUG:boto:Params: {} 2023-05-31T20:37:54.610 DEBUG:boto:Token: None 2023-05-31T20:37:54.610 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:37:54 GMT /pklxkd-58/obj00000000000000000000 2023-05-31T20:37:54.610 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:GgZncmUVYwL/VD/w2rcufl+m7Nk= 2023-05-31T20:37:54.611 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:37:54 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:GgZncmUVYwL/VD/w2rcufl+m7Nk=', 'Content-Length': '0'} 2023-05-31T20:37:54.613 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:37:29 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002dad5e6b6b6effb9-006477b022-1b4f-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:37:54 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:37:54.613 DEBUG:boto:path=/pklxkd-58/obj00000000000000000000 2023-05-31T20:37:54.613 DEBUG:boto:auth_path=/pklxkd-58/obj00000000000000000000 2023-05-31T20:37:54.614 DEBUG:boto:Method: HEAD 2023-05-31T20:37:54.614 DEBUG:boto:Path: /pklxkd-58/obj00000000000000000000 2023-05-31T20:37:54.615 DEBUG:boto:Data: 2023-05-31T20:37:54.615 DEBUG:boto:Headers: {} 2023-05-31T20:37:54.615 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:37:54.616 DEBUG:boto:Port: 8000 2023-05-31T20:37:54.616 DEBUG:boto:Params: {} 2023-05-31T20:37:54.616 DEBUG:boto:Token: None 2023-05-31T20:37:54.617 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:37:54 GMT /pklxkd-58/obj00000000000000000000 2023-05-31T20:37:54.617 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:GgZncmUVYwL/VD/w2rcufl+m7Nk= 2023-05-31T20:37:54.617 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:37:54 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:GgZncmUVYwL/VD/w2rcufl+m7Nk=', 'Content-Length': '0'} 2023-05-31T20:37:54.619 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:37:29 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000991667ceb79b5768-006477b022-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:37:54 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:37:54.620 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000000 2023-05-31T20:37:54.620 DEBUG:boto:path=/pklxkd-58/obj00000000000000000000 2023-05-31T20:37:54.621 DEBUG:boto:auth_path=/pklxkd-58/obj00000000000000000000 2023-05-31T20:37:54.621 DEBUG:boto:Method: GET 2023-05-31T20:37:54.621 DEBUG:boto:Path: /pklxkd-58/obj00000000000000000000 2023-05-31T20:37:54.622 DEBUG:boto:Data: 2023-05-31T20:37:54.622 DEBUG:boto:Headers: {} 2023-05-31T20:37:54.622 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:37:54.623 DEBUG:boto:Port: 8001 2023-05-31T20:37:54.623 DEBUG:boto:Params: {} 2023-05-31T20:37:54.623 DEBUG:boto:Token: None 2023-05-31T20:37:54.624 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:37:54 GMT /pklxkd-58/obj00000000000000000000 2023-05-31T20:37:54.624 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:rio6H0poPPvub8gHL37GDP8UHqU= 2023-05-31T20:37:54.625 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:37:54 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:rio6H0poPPvub8gHL37GDP8UHqU=', 'Content-Length': '0'} 2023-05-31T20:37:54.627 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:37:29 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000adaaad848a23b520-006477b022-1b4f-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:37:54 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:37:54.669 DEBUG:boto:path=/pklxkd-58/obj00000000000000000000 2023-05-31T20:37:54.669 DEBUG:boto:auth_path=/pklxkd-58/obj00000000000000000000 2023-05-31T20:37:54.670 DEBUG:boto:Method: GET 2023-05-31T20:37:54.670 DEBUG:boto:Path: /pklxkd-58/obj00000000000000000000 2023-05-31T20:37:54.670 DEBUG:boto:Data: 2023-05-31T20:37:54.671 DEBUG:boto:Headers: {} 2023-05-31T20:37:54.671 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:37:54.671 DEBUG:boto:Port: 8000 2023-05-31T20:37:54.672 DEBUG:boto:Params: {} 2023-05-31T20:37:54.672 DEBUG:boto:Token: None 2023-05-31T20:37:54.673 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:37:54 GMT /pklxkd-58/obj00000000000000000000 2023-05-31T20:37:54.673 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:rio6H0poPPvub8gHL37GDP8UHqU= 2023-05-31T20:37:54.674 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:37:54 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:rio6H0poPPvub8gHL37GDP8UHqU=', 'Content-Length': '0'} 2023-05-31T20:37:54.676 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:37:29 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000794ba4974518def4-006477b022-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:37:54 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:37:54.677 DEBUG:boto:path=/pklxkd-58/ 2023-05-31T20:37:54.678 DEBUG:boto:auth_path=/pklxkd-58/ 2023-05-31T20:37:54.679 DEBUG:boto:path=/pklxkd-58/?prefix=obj00000000000000000000 2023-05-31T20:37:54.679 DEBUG:boto:auth_path=/pklxkd-58/?prefix=obj00000000000000000000 2023-05-31T20:37:54.680 DEBUG:boto:Method: GET 2023-05-31T20:37:54.681 DEBUG:boto:Path: /pklxkd-58/?prefix=obj00000000000000000000 2023-05-31T20:37:54.682 DEBUG:boto:Data: 2023-05-31T20:37:54.682 DEBUG:boto:Headers: {} 2023-05-31T20:37:54.683 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:37:54.684 DEBUG:boto:Port: 8001 2023-05-31T20:37:54.685 DEBUG:boto:Params: {} 2023-05-31T20:37:54.685 DEBUG:boto:Token: None 2023-05-31T20:37:54.686 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:37:54 GMT /pklxkd-58/ 2023-05-31T20:37:54.687 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:kBU/21bIG8o+fKSyagNAMS+H4qE= 2023-05-31T20:37:54.688 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:37:54 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:kBU/21bIG8o+fKSyagNAMS+H4qE=', 'Content-Length': '0'} 2023-05-31T20:37:54.691 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000dd23ffba5847cebb-006477b022-1b4f-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:37:54 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:37:54.693 DEBUG:boto:b'pklxkd-58obj000000000000000000001000falseobj000000000000000000002023-05-31T20:37:29.102Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:37:54.693 DEBUG:boto:path=/pklxkd-58/ 2023-05-31T20:37:54.694 DEBUG:boto:auth_path=/pklxkd-58/ 2023-05-31T20:37:54.694 DEBUG:boto:path=/pklxkd-58/?prefix=obj00000000000000000000 2023-05-31T20:37:54.695 DEBUG:boto:auth_path=/pklxkd-58/?prefix=obj00000000000000000000 2023-05-31T20:37:54.695 DEBUG:boto:Method: GET 2023-05-31T20:37:54.696 DEBUG:boto:Path: /pklxkd-58/?prefix=obj00000000000000000000 2023-05-31T20:37:54.696 DEBUG:boto:Data: 2023-05-31T20:37:54.697 DEBUG:boto:Headers: {} 2023-05-31T20:37:54.698 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:37:54.698 DEBUG:boto:Port: 8000 2023-05-31T20:37:54.699 DEBUG:boto:Params: {} 2023-05-31T20:37:54.700 DEBUG:boto:Token: None 2023-05-31T20:37:54.701 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:37:54 GMT /pklxkd-58/ 2023-05-31T20:37:54.704 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:kBU/21bIG8o+fKSyagNAMS+H4qE= 2023-05-31T20:37:54.705 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:37:54 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:kBU/21bIG8o+fKSyagNAMS+H4qE=', 'Content-Length': '0'} 2023-05-31T20:37:54.707 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a4b098bdf938170a-006477b022-1630-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:37:54 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:37:54.708 DEBUG:boto:b'pklxkd-58obj000000000000000000001000falseobj000000000000000000002023-05-31T20:37:29.102Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:37:54.709 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000001 2023-05-31T20:37:54.709 DEBUG:boto:path=/pklxkd-58/obj00000000000000000001 2023-05-31T20:37:54.709 DEBUG:boto:auth_path=/pklxkd-58/obj00000000000000000001 2023-05-31T20:37:54.710 DEBUG:boto:path=/pklxkd-58/obj00000000000000000001?versionId=null 2023-05-31T20:37:54.710 DEBUG:boto:auth_path=/pklxkd-58/obj00000000000000000001?versionId=null 2023-05-31T20:37:54.710 DEBUG:boto:Method: GET 2023-05-31T20:37:54.711 DEBUG:boto:Path: /pklxkd-58/obj00000000000000000001?versionId=null 2023-05-31T20:37:54.711 DEBUG:boto:Data: 2023-05-31T20:37:54.712 DEBUG:boto:Headers: {} 2023-05-31T20:37:54.712 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:37:54.712 DEBUG:boto:Port: 8001 2023-05-31T20:37:54.713 DEBUG:boto:Params: {} 2023-05-31T20:37:54.713 DEBUG:boto:Token: None 2023-05-31T20:37:54.713 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:37:54 GMT /pklxkd-58/obj00000000000000000001?versionId=null 2023-05-31T20:37:54.714 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:m93kt09GlIrx+mDS+8aAmYWWXNI= 2023-05-31T20:37:54.714 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:37:54 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:m93kt09GlIrx+mDS+8aAmYWWXNI=', 'Content-Length': '0'} 2023-05-31T20:37:54.716 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:37:28 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c9f2bbd3164aa3a0-006477b022-1b4f-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:37:54 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:37:54.717 DEBUG:boto:path=/pklxkd-58/obj00000000000000000001 2023-05-31T20:37:54.717 DEBUG:boto:auth_path=/pklxkd-58/obj00000000000000000001 2023-05-31T20:37:54.718 DEBUG:boto:path=/pklxkd-58/obj00000000000000000001?versionId=null 2023-05-31T20:37:54.718 DEBUG:boto:auth_path=/pklxkd-58/obj00000000000000000001?versionId=null 2023-05-31T20:37:54.718 DEBUG:boto:Method: GET 2023-05-31T20:37:54.719 DEBUG:boto:Path: /pklxkd-58/obj00000000000000000001?versionId=null 2023-05-31T20:37:54.719 DEBUG:boto:Data: 2023-05-31T20:37:54.719 DEBUG:boto:Headers: {} 2023-05-31T20:37:54.720 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:37:54.720 DEBUG:boto:Port: 8000 2023-05-31T20:37:54.721 DEBUG:boto:Params: {} 2023-05-31T20:37:54.721 DEBUG:boto:Token: None 2023-05-31T20:37:54.722 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:37:54 GMT /pklxkd-58/obj00000000000000000001?versionId=null 2023-05-31T20:37:54.722 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:m93kt09GlIrx+mDS+8aAmYWWXNI= 2023-05-31T20:37:54.722 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:37:54 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:m93kt09GlIrx+mDS+8aAmYWWXNI=', 'Content-Length': '0'} 2023-05-31T20:37:54.725 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:37:28 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f871fb1aa47537c8-006477b022-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:37:54 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:37:54.725 DEBUG:boto:path=/pklxkd-58/obj00000000000000000001 2023-05-31T20:37:54.726 DEBUG:boto:auth_path=/pklxkd-58/obj00000000000000000001 2023-05-31T20:37:54.726 DEBUG:boto:path=/pklxkd-58/obj00000000000000000001?versionId=null 2023-05-31T20:37:54.726 DEBUG:boto:auth_path=/pklxkd-58/obj00000000000000000001?versionId=null 2023-05-31T20:37:54.727 DEBUG:boto:Method: HEAD 2023-05-31T20:37:54.727 DEBUG:boto:Path: /pklxkd-58/obj00000000000000000001?versionId=null 2023-05-31T20:37:54.727 DEBUG:boto:Data: 2023-05-31T20:37:54.728 DEBUG:boto:Headers: {} 2023-05-31T20:37:54.728 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:37:54.728 DEBUG:boto:Port: 8001 2023-05-31T20:37:54.729 DEBUG:boto:Params: {} 2023-05-31T20:37:54.729 DEBUG:boto:Token: None 2023-05-31T20:37:54.729 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:37:54 GMT /pklxkd-58/obj00000000000000000001?versionId=null 2023-05-31T20:37:54.730 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:l8JjvKENJP608BGuVamoq7Olfsw= 2023-05-31T20:37:54.730 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:37:54 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:l8JjvKENJP608BGuVamoq7Olfsw=', 'Content-Length': '0'} 2023-05-31T20:37:54.733 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:37:28 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000014e124513eaa8b20-006477b022-1b4f-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:37:54 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:37:54.733 DEBUG:boto:path=/pklxkd-58/obj00000000000000000001 2023-05-31T20:37:54.734 DEBUG:boto:auth_path=/pklxkd-58/obj00000000000000000001 2023-05-31T20:37:54.734 DEBUG:boto:path=/pklxkd-58/obj00000000000000000001?versionId=null 2023-05-31T20:37:54.734 DEBUG:boto:auth_path=/pklxkd-58/obj00000000000000000001?versionId=null 2023-05-31T20:37:54.735 DEBUG:boto:Method: HEAD 2023-05-31T20:37:54.735 DEBUG:boto:Path: /pklxkd-58/obj00000000000000000001?versionId=null 2023-05-31T20:37:54.735 DEBUG:boto:Data: 2023-05-31T20:37:54.736 DEBUG:boto:Headers: {} 2023-05-31T20:37:54.736 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:37:54.736 DEBUG:boto:Port: 8000 2023-05-31T20:37:54.737 DEBUG:boto:Params: {} 2023-05-31T20:37:54.737 DEBUG:boto:Token: None 2023-05-31T20:37:54.738 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:37:54 GMT /pklxkd-58/obj00000000000000000001?versionId=null 2023-05-31T20:37:54.738 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:l8JjvKENJP608BGuVamoq7Olfsw= 2023-05-31T20:37:54.738 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:37:54 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:l8JjvKENJP608BGuVamoq7Olfsw=', 'Content-Length': '0'} 2023-05-31T20:37:54.740 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:37:28 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c2beebdcfde59d21-006477b022-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:37:54 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:37:54.741 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000001 2023-05-31T20:37:54.741 DEBUG:boto:path=/pklxkd-58/obj00000000000000000001 2023-05-31T20:37:54.741 DEBUG:boto:auth_path=/pklxkd-58/obj00000000000000000001 2023-05-31T20:37:54.742 DEBUG:boto:path=/pklxkd-58/obj00000000000000000001?versionId=null 2023-05-31T20:37:54.742 DEBUG:boto:auth_path=/pklxkd-58/obj00000000000000000001?versionId=null 2023-05-31T20:37:54.742 DEBUG:boto:Method: GET 2023-05-31T20:37:54.743 DEBUG:boto:Path: /pklxkd-58/obj00000000000000000001?versionId=null 2023-05-31T20:37:54.743 DEBUG:boto:Data: 2023-05-31T20:37:54.743 DEBUG:boto:Headers: {} 2023-05-31T20:37:54.744 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:37:54.744 DEBUG:boto:Port: 8001 2023-05-31T20:37:54.745 DEBUG:boto:Params: {} 2023-05-31T20:37:54.745 DEBUG:boto:Token: None 2023-05-31T20:37:54.745 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:37:54 GMT /pklxkd-58/obj00000000000000000001?versionId=null 2023-05-31T20:37:54.746 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:m93kt09GlIrx+mDS+8aAmYWWXNI= 2023-05-31T20:37:54.746 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:37:54 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:m93kt09GlIrx+mDS+8aAmYWWXNI=', 'Content-Length': '0'} 2023-05-31T20:37:54.748 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:37:28 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000594dbc9497faa79-006477b022-1b4f-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:37:54 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:37:54.788 DEBUG:boto:path=/pklxkd-58/obj00000000000000000001 2023-05-31T20:37:54.789 DEBUG:boto:auth_path=/pklxkd-58/obj00000000000000000001 2023-05-31T20:37:54.790 DEBUG:boto:path=/pklxkd-58/obj00000000000000000001?versionId=null 2023-05-31T20:37:54.790 DEBUG:boto:auth_path=/pklxkd-58/obj00000000000000000001?versionId=null 2023-05-31T20:37:54.790 DEBUG:boto:Method: GET 2023-05-31T20:37:54.791 DEBUG:boto:Path: /pklxkd-58/obj00000000000000000001?versionId=null 2023-05-31T20:37:54.791 DEBUG:boto:Data: 2023-05-31T20:37:54.791 DEBUG:boto:Headers: {} 2023-05-31T20:37:54.792 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:37:54.792 DEBUG:boto:Port: 8000 2023-05-31T20:37:54.793 DEBUG:boto:Params: {} 2023-05-31T20:37:54.793 DEBUG:boto:Token: None 2023-05-31T20:37:54.794 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:37:54 GMT /pklxkd-58/obj00000000000000000001?versionId=null 2023-05-31T20:37:54.794 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:m93kt09GlIrx+mDS+8aAmYWWXNI= 2023-05-31T20:37:54.794 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:37:54 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:m93kt09GlIrx+mDS+8aAmYWWXNI=', 'Content-Length': '0'} 2023-05-31T20:37:54.796 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:37:28 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006b404acce6de3c83-006477b022-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:37:54 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:37:54.797 DEBUG:boto:path=/pklxkd-58/ 2023-05-31T20:37:54.798 DEBUG:boto:auth_path=/pklxkd-58/ 2023-05-31T20:37:54.798 DEBUG:boto:path=/pklxkd-58/?prefix=obj00000000000000000001 2023-05-31T20:37:54.798 DEBUG:boto:auth_path=/pklxkd-58/?prefix=obj00000000000000000001 2023-05-31T20:37:54.799 DEBUG:boto:Method: GET 2023-05-31T20:37:54.799 DEBUG:boto:Path: /pklxkd-58/?prefix=obj00000000000000000001 2023-05-31T20:37:54.799 DEBUG:boto:Data: 2023-05-31T20:37:54.800 DEBUG:boto:Headers: {} 2023-05-31T20:37:54.800 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:37:54.801 DEBUG:boto:Port: 8001 2023-05-31T20:37:54.801 DEBUG:boto:Params: {} 2023-05-31T20:37:54.802 DEBUG:boto:Token: None 2023-05-31T20:37:54.802 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:37:54 GMT /pklxkd-58/ 2023-05-31T20:37:54.803 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:kBU/21bIG8o+fKSyagNAMS+H4qE= 2023-05-31T20:37:54.803 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:37:54 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:kBU/21bIG8o+fKSyagNAMS+H4qE=', 'Content-Length': '0'} 2023-05-31T20:37:54.806 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f1d01afa63f9c615-006477b022-1b4f-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:37:54 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:37:54.807 DEBUG:boto:b'pklxkd-58obj000000000000000000011000falseobj000000000000000000012023-05-31T20:37:28.869Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:37:54.808 DEBUG:boto:path=/pklxkd-58/ 2023-05-31T20:37:54.808 DEBUG:boto:auth_path=/pklxkd-58/ 2023-05-31T20:37:54.809 DEBUG:boto:path=/pklxkd-58/?prefix=obj00000000000000000001 2023-05-31T20:37:54.809 DEBUG:boto:auth_path=/pklxkd-58/?prefix=obj00000000000000000001 2023-05-31T20:37:54.810 DEBUG:boto:Method: GET 2023-05-31T20:37:54.810 DEBUG:boto:Path: /pklxkd-58/?prefix=obj00000000000000000001 2023-05-31T20:37:54.810 DEBUG:boto:Data: 2023-05-31T20:37:54.811 DEBUG:boto:Headers: {} 2023-05-31T20:37:54.811 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:37:54.811 DEBUG:boto:Port: 8000 2023-05-31T20:37:54.812 DEBUG:boto:Params: {} 2023-05-31T20:37:54.812 DEBUG:boto:Token: None 2023-05-31T20:37:54.813 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:37:54 GMT /pklxkd-58/ 2023-05-31T20:37:54.813 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:kBU/21bIG8o+fKSyagNAMS+H4qE= 2023-05-31T20:37:54.814 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:37:54 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:kBU/21bIG8o+fKSyagNAMS+H4qE=', 'Content-Length': '0'} 2023-05-31T20:37:54.816 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000005999bfc19392d4ef-006477b022-1630-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:37:54 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:37:54.817 DEBUG:boto:b'pklxkd-58obj000000000000000000011000falseobj000000000000000000012023-05-31T20:37:28.869Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:37:54.818 DEBUG:boto:path=/pklxkd-58/obj00000000000000000001 2023-05-31T20:37:54.818 DEBUG:boto:auth_path=/pklxkd-58/obj00000000000000000001 2023-05-31T20:37:54.819 DEBUG:boto:Method: HEAD 2023-05-31T20:37:54.819 DEBUG:boto:Path: /pklxkd-58/obj00000000000000000001 2023-05-31T20:37:54.819 DEBUG:boto:Data: 2023-05-31T20:37:54.820 DEBUG:boto:Headers: {} 2023-05-31T20:37:54.820 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:37:54.820 DEBUG:boto:Port: 8001 2023-05-31T20:37:54.821 DEBUG:boto:Params: {} 2023-05-31T20:37:54.821 DEBUG:boto:Token: None 2023-05-31T20:37:54.821 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:37:54 GMT /pklxkd-58/obj00000000000000000001 2023-05-31T20:37:54.822 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:xXkEGanhQ8Sqgn8G/78a68o41dA= 2023-05-31T20:37:54.822 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:37:54 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:xXkEGanhQ8Sqgn8G/78a68o41dA=', 'Content-Length': '0'} 2023-05-31T20:37:54.824 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:37:28 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004b30ac6ea44fb729-006477b022-1b4f-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:37:54 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:37:54.825 DEBUG:boto:path=/pklxkd-58/obj00000000000000000001 2023-05-31T20:37:54.825 DEBUG:boto:auth_path=/pklxkd-58/obj00000000000000000001 2023-05-31T20:37:54.826 DEBUG:boto:Method: HEAD 2023-05-31T20:37:54.826 DEBUG:boto:Path: /pklxkd-58/obj00000000000000000001 2023-05-31T20:37:54.826 DEBUG:boto:Data: 2023-05-31T20:37:54.827 DEBUG:boto:Headers: {} 2023-05-31T20:37:54.827 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:37:54.827 DEBUG:boto:Port: 8000 2023-05-31T20:37:54.828 DEBUG:boto:Params: {} 2023-05-31T20:37:54.828 DEBUG:boto:Token: None 2023-05-31T20:37:54.829 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:37:54 GMT /pklxkd-58/obj00000000000000000001 2023-05-31T20:37:54.829 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:xXkEGanhQ8Sqgn8G/78a68o41dA= 2023-05-31T20:37:54.830 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:37:54 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:xXkEGanhQ8Sqgn8G/78a68o41dA=', 'Content-Length': '0'} 2023-05-31T20:37:54.831 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:37:28 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008d1680004c039de8-006477b022-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:37:54 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:37:54.832 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000001 2023-05-31T20:37:54.833 DEBUG:boto:path=/pklxkd-58/obj00000000000000000001 2023-05-31T20:37:54.833 DEBUG:boto:auth_path=/pklxkd-58/obj00000000000000000001 2023-05-31T20:37:54.833 DEBUG:boto:Method: GET 2023-05-31T20:37:54.834 DEBUG:boto:Path: /pklxkd-58/obj00000000000000000001 2023-05-31T20:37:54.834 DEBUG:boto:Data: 2023-05-31T20:37:54.835 DEBUG:boto:Headers: {} 2023-05-31T20:37:54.835 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:37:54.835 DEBUG:boto:Port: 8001 2023-05-31T20:37:54.836 DEBUG:boto:Params: {} 2023-05-31T20:37:54.836 DEBUG:boto:Token: None 2023-05-31T20:37:54.837 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:37:54 GMT /pklxkd-58/obj00000000000000000001 2023-05-31T20:37:54.837 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:dH7ImHmn+zctDAP23DYCLjvPkcA= 2023-05-31T20:37:54.837 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:37:54 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:dH7ImHmn+zctDAP23DYCLjvPkcA=', 'Content-Length': '0'} 2023-05-31T20:37:54.839 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:37:28 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000da029db7db7a2f4c-006477b022-1b4f-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:37:54 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:37:54.840 DEBUG:boto:path=/pklxkd-58/obj00000000000000000001 2023-05-31T20:37:54.840 DEBUG:boto:auth_path=/pklxkd-58/obj00000000000000000001 2023-05-31T20:37:54.841 DEBUG:boto:Method: GET 2023-05-31T20:37:54.841 DEBUG:boto:Path: /pklxkd-58/obj00000000000000000001 2023-05-31T20:37:54.842 DEBUG:boto:Data: 2023-05-31T20:37:54.842 DEBUG:boto:Headers: {} 2023-05-31T20:37:54.842 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:37:54.843 DEBUG:boto:Port: 8000 2023-05-31T20:37:54.843 DEBUG:boto:Params: {} 2023-05-31T20:37:54.844 DEBUG:boto:Token: None 2023-05-31T20:37:54.844 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:37:54 GMT /pklxkd-58/obj00000000000000000001 2023-05-31T20:37:54.845 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:dH7ImHmn+zctDAP23DYCLjvPkcA= 2023-05-31T20:37:54.845 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:37:54 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:dH7ImHmn+zctDAP23DYCLjvPkcA=', 'Content-Length': '0'} 2023-05-31T20:37:54.847 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:37:28 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001cb2aab0b8fe4940-006477b022-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:37:54 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:37:54.892 DEBUG:boto:path=/pklxkd-58/ 2023-05-31T20:37:54.893 DEBUG:boto:auth_path=/pklxkd-58/ 2023-05-31T20:37:54.893 DEBUG:boto:path=/pklxkd-58/?prefix=obj00000000000000000001 2023-05-31T20:37:54.894 DEBUG:boto:auth_path=/pklxkd-58/?prefix=obj00000000000000000001 2023-05-31T20:37:54.894 DEBUG:boto:Method: GET 2023-05-31T20:37:54.894 DEBUG:boto:Path: /pklxkd-58/?prefix=obj00000000000000000001 2023-05-31T20:37:54.895 DEBUG:boto:Data: 2023-05-31T20:37:54.895 DEBUG:boto:Headers: {} 2023-05-31T20:37:54.895 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:37:54.896 DEBUG:boto:Port: 8001 2023-05-31T20:37:54.896 DEBUG:boto:Params: {} 2023-05-31T20:37:54.897 DEBUG:boto:Token: None 2023-05-31T20:37:54.897 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:37:54 GMT /pklxkd-58/ 2023-05-31T20:37:54.898 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:kBU/21bIG8o+fKSyagNAMS+H4qE= 2023-05-31T20:37:54.898 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:37:54 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:kBU/21bIG8o+fKSyagNAMS+H4qE=', 'Content-Length': '0'} 2023-05-31T20:37:54.900 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e8e97eddcf6dc591-006477b022-1b4f-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:37:54 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:37:54.901 DEBUG:boto:b'pklxkd-58obj000000000000000000011000falseobj000000000000000000012023-05-31T20:37:28.869Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:37:54.902 DEBUG:boto:path=/pklxkd-58/ 2023-05-31T20:37:54.902 DEBUG:boto:auth_path=/pklxkd-58/ 2023-05-31T20:37:54.903 DEBUG:boto:path=/pklxkd-58/?prefix=obj00000000000000000001 2023-05-31T20:37:54.903 DEBUG:boto:auth_path=/pklxkd-58/?prefix=obj00000000000000000001 2023-05-31T20:37:54.903 DEBUG:boto:Method: GET 2023-05-31T20:37:54.904 DEBUG:boto:Path: /pklxkd-58/?prefix=obj00000000000000000001 2023-05-31T20:37:54.904 DEBUG:boto:Data: 2023-05-31T20:37:54.904 DEBUG:boto:Headers: {} 2023-05-31T20:37:54.905 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:37:54.905 DEBUG:boto:Port: 8000 2023-05-31T20:37:54.905 DEBUG:boto:Params: {} 2023-05-31T20:37:54.906 DEBUG:boto:Token: None 2023-05-31T20:37:54.906 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:37:54 GMT /pklxkd-58/ 2023-05-31T20:37:54.907 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:kBU/21bIG8o+fKSyagNAMS+H4qE= 2023-05-31T20:37:54.907 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:37:54 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:kBU/21bIG8o+fKSyagNAMS+H4qE=', 'Content-Length': '0'} 2023-05-31T20:37:54.910 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e36d4f422d2b6a7e-006477b022-1630-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:37:54 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:37:54.910 DEBUG:boto:b'pklxkd-58obj000000000000000000011000falseobj000000000000000000012023-05-31T20:37:28.869Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:37:54.911 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000002 2023-05-31T20:37:54.911 DEBUG:boto:path=/pklxkd-58/obj00000000000000000002 2023-05-31T20:37:54.912 DEBUG:boto:auth_path=/pklxkd-58/obj00000000000000000002 2023-05-31T20:37:54.912 DEBUG:boto:path=/pklxkd-58/obj00000000000000000002?versionId=null 2023-05-31T20:37:54.912 DEBUG:boto:auth_path=/pklxkd-58/obj00000000000000000002?versionId=null 2023-05-31T20:37:54.913 DEBUG:boto:Method: GET 2023-05-31T20:37:54.913 DEBUG:boto:Path: /pklxkd-58/obj00000000000000000002?versionId=null 2023-05-31T20:37:54.913 DEBUG:boto:Data: 2023-05-31T20:37:54.914 DEBUG:boto:Headers: {} 2023-05-31T20:37:54.914 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:37:54.915 DEBUG:boto:Port: 8001 2023-05-31T20:37:54.915 DEBUG:boto:Params: {} 2023-05-31T20:37:54.915 DEBUG:boto:Token: None 2023-05-31T20:37:54.916 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:37:54 GMT /pklxkd-58/obj00000000000000000002?versionId=null 2023-05-31T20:37:54.916 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:g4ZtlHY7Zfea278BqMwGlPMeNqI= 2023-05-31T20:37:54.917 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:37:54 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:g4ZtlHY7Zfea278BqMwGlPMeNqI=', 'Content-Length': '0'} 2023-05-31T20:37:54.918 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:37:28 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c71adc32cc74dad0-006477b022-1b4f-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:37:54 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:37:54.960 DEBUG:boto:path=/pklxkd-58/obj00000000000000000002 2023-05-31T20:37:54.961 DEBUG:boto:auth_path=/pklxkd-58/obj00000000000000000002 2023-05-31T20:37:54.962 DEBUG:boto:path=/pklxkd-58/obj00000000000000000002?versionId=null 2023-05-31T20:37:54.962 DEBUG:boto:auth_path=/pklxkd-58/obj00000000000000000002?versionId=null 2023-05-31T20:37:54.962 DEBUG:boto:Method: GET 2023-05-31T20:37:54.963 DEBUG:boto:Path: /pklxkd-58/obj00000000000000000002?versionId=null 2023-05-31T20:37:54.963 DEBUG:boto:Data: 2023-05-31T20:37:54.963 DEBUG:boto:Headers: {} 2023-05-31T20:37:54.964 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:37:54.964 DEBUG:boto:Port: 8000 2023-05-31T20:37:54.965 DEBUG:boto:Params: {} 2023-05-31T20:37:54.965 DEBUG:boto:Token: None 2023-05-31T20:37:54.965 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:37:54 GMT /pklxkd-58/obj00000000000000000002?versionId=null 2023-05-31T20:37:54.966 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:g4ZtlHY7Zfea278BqMwGlPMeNqI= 2023-05-31T20:37:54.966 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:37:54 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:g4ZtlHY7Zfea278BqMwGlPMeNqI=', 'Content-Length': '0'} 2023-05-31T20:37:54.969 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:37:28 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000fb17658c8dbe13cd-006477b022-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:37:54 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:37:54.969 DEBUG:boto:path=/pklxkd-58/obj00000000000000000002 2023-05-31T20:37:54.970 DEBUG:boto:auth_path=/pklxkd-58/obj00000000000000000002 2023-05-31T20:37:54.970 DEBUG:boto:path=/pklxkd-58/obj00000000000000000002?versionId=null 2023-05-31T20:37:54.970 DEBUG:boto:auth_path=/pklxkd-58/obj00000000000000000002?versionId=null 2023-05-31T20:37:54.971 DEBUG:boto:Method: HEAD 2023-05-31T20:37:54.971 DEBUG:boto:Path: /pklxkd-58/obj00000000000000000002?versionId=null 2023-05-31T20:37:54.971 DEBUG:boto:Data: 2023-05-31T20:37:54.972 DEBUG:boto:Headers: {} 2023-05-31T20:37:54.972 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:37:54.972 DEBUG:boto:Port: 8001 2023-05-31T20:37:54.973 DEBUG:boto:Params: {} 2023-05-31T20:37:54.973 DEBUG:boto:Token: None 2023-05-31T20:37:54.974 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:37:54 GMT /pklxkd-58/obj00000000000000000002?versionId=null 2023-05-31T20:37:54.974 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:KFk4N10ImJH/UrnqS+mWT/v8w78= 2023-05-31T20:37:54.974 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:37:54 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:KFk4N10ImJH/UrnqS+mWT/v8w78=', 'Content-Length': '0'} 2023-05-31T20:37:54.978 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:37:28 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002f83c3cf294a3ade-006477b022-1b4f-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:37:54 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:37:54.980 DEBUG:boto:path=/pklxkd-58/obj00000000000000000002 2023-05-31T20:37:54.981 DEBUG:boto:auth_path=/pklxkd-58/obj00000000000000000002 2023-05-31T20:37:54.981 DEBUG:boto:path=/pklxkd-58/obj00000000000000000002?versionId=null 2023-05-31T20:37:54.981 DEBUG:boto:auth_path=/pklxkd-58/obj00000000000000000002?versionId=null 2023-05-31T20:37:54.982 DEBUG:boto:Method: HEAD 2023-05-31T20:37:54.982 DEBUG:boto:Path: /pklxkd-58/obj00000000000000000002?versionId=null 2023-05-31T20:37:54.982 DEBUG:boto:Data: 2023-05-31T20:37:54.983 DEBUG:boto:Headers: {} 2023-05-31T20:37:54.984 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:37:54.985 DEBUG:boto:Port: 8000 2023-05-31T20:37:54.985 DEBUG:boto:Params: {} 2023-05-31T20:37:54.985 DEBUG:boto:Token: None 2023-05-31T20:37:54.986 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:37:54 GMT /pklxkd-58/obj00000000000000000002?versionId=null 2023-05-31T20:37:54.986 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:KFk4N10ImJH/UrnqS+mWT/v8w78= 2023-05-31T20:37:54.987 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:37:54 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:KFk4N10ImJH/UrnqS+mWT/v8w78=', 'Content-Length': '0'} 2023-05-31T20:37:54.990 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:37:28 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c92c1c4fb9403c51-006477b022-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:37:54 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:37:54.990 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000002 2023-05-31T20:37:54.991 DEBUG:boto:path=/pklxkd-58/obj00000000000000000002 2023-05-31T20:37:54.991 DEBUG:boto:auth_path=/pklxkd-58/obj00000000000000000002 2023-05-31T20:37:54.992 DEBUG:boto:path=/pklxkd-58/obj00000000000000000002?versionId=null 2023-05-31T20:37:54.992 DEBUG:boto:auth_path=/pklxkd-58/obj00000000000000000002?versionId=null 2023-05-31T20:37:54.992 DEBUG:boto:Method: GET 2023-05-31T20:37:54.993 DEBUG:boto:Path: /pklxkd-58/obj00000000000000000002?versionId=null 2023-05-31T20:37:54.994 DEBUG:boto:Data: 2023-05-31T20:37:54.994 DEBUG:boto:Headers: {} 2023-05-31T20:37:54.995 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:37:54.995 DEBUG:boto:Port: 8001 2023-05-31T20:37:54.995 DEBUG:boto:Params: {} 2023-05-31T20:37:54.996 DEBUG:boto:Token: None 2023-05-31T20:37:54.996 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:37:54 GMT /pklxkd-58/obj00000000000000000002?versionId=null 2023-05-31T20:37:54.997 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:g4ZtlHY7Zfea278BqMwGlPMeNqI= 2023-05-31T20:37:54.997 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:37:54 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:g4ZtlHY7Zfea278BqMwGlPMeNqI=', 'Content-Length': '0'} 2023-05-31T20:37:55.004 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:37:28 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000823184f8f95e2c10-006477b022-1b4f-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:37:54 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:37:55.008 DEBUG:boto:path=/pklxkd-58/obj00000000000000000002 2023-05-31T20:37:55.010 DEBUG:boto:auth_path=/pklxkd-58/obj00000000000000000002 2023-05-31T20:37:55.010 DEBUG:boto:path=/pklxkd-58/obj00000000000000000002?versionId=null 2023-05-31T20:37:55.011 DEBUG:boto:auth_path=/pklxkd-58/obj00000000000000000002?versionId=null 2023-05-31T20:37:55.011 DEBUG:boto:Method: GET 2023-05-31T20:37:55.012 DEBUG:boto:Path: /pklxkd-58/obj00000000000000000002?versionId=null 2023-05-31T20:37:55.013 DEBUG:boto:Data: 2023-05-31T20:37:55.014 DEBUG:boto:Headers: {} 2023-05-31T20:37:55.014 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:37:55.015 DEBUG:boto:Port: 8000 2023-05-31T20:37:55.015 DEBUG:boto:Params: {} 2023-05-31T20:37:55.016 DEBUG:boto:Token: None 2023-05-31T20:37:55.017 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:37:55 GMT /pklxkd-58/obj00000000000000000002?versionId=null 2023-05-31T20:37:55.018 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:IpN6vs8uJLK5yWRocqViafc1Lmk= 2023-05-31T20:37:55.019 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:37:55 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:IpN6vs8uJLK5yWRocqViafc1Lmk=', 'Content-Length': '0'} 2023-05-31T20:37:55.023 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:37:28 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006fd2df5abb419a9a-006477b023-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:37:55 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:37:55.065 DEBUG:boto:path=/pklxkd-58/ 2023-05-31T20:37:55.066 DEBUG:boto:auth_path=/pklxkd-58/ 2023-05-31T20:37:55.066 DEBUG:boto:path=/pklxkd-58/?prefix=obj00000000000000000002 2023-05-31T20:37:55.067 DEBUG:boto:auth_path=/pklxkd-58/?prefix=obj00000000000000000002 2023-05-31T20:37:55.067 DEBUG:boto:Method: GET 2023-05-31T20:37:55.067 DEBUG:boto:Path: /pklxkd-58/?prefix=obj00000000000000000002 2023-05-31T20:37:55.068 DEBUG:boto:Data: 2023-05-31T20:37:55.068 DEBUG:boto:Headers: {} 2023-05-31T20:37:55.069 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:37:55.069 DEBUG:boto:Port: 8001 2023-05-31T20:37:55.069 DEBUG:boto:Params: {} 2023-05-31T20:37:55.070 DEBUG:boto:Token: None 2023-05-31T20:37:55.070 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:37:55 GMT /pklxkd-58/ 2023-05-31T20:37:55.071 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:lno1rwvBXdlscvtz78c1xk4QhQw= 2023-05-31T20:37:55.071 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:37:55 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:lno1rwvBXdlscvtz78c1xk4QhQw=', 'Content-Length': '0'} 2023-05-31T20:37:55.074 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ea936b70be9b62c4-006477b023-1b4f-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:37:55 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:37:55.075 DEBUG:boto:b'pklxkd-58obj000000000000000000021000falseobj000000000000000000022023-05-31T20:37:28.769Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:37:55.077 DEBUG:boto:path=/pklxkd-58/ 2023-05-31T20:37:55.077 DEBUG:boto:auth_path=/pklxkd-58/ 2023-05-31T20:37:55.077 DEBUG:boto:path=/pklxkd-58/?prefix=obj00000000000000000002 2023-05-31T20:37:55.078 DEBUG:boto:auth_path=/pklxkd-58/?prefix=obj00000000000000000002 2023-05-31T20:37:55.078 DEBUG:boto:Method: GET 2023-05-31T20:37:55.078 DEBUG:boto:Path: /pklxkd-58/?prefix=obj00000000000000000002 2023-05-31T20:37:55.079 DEBUG:boto:Data: 2023-05-31T20:37:55.079 DEBUG:boto:Headers: {} 2023-05-31T20:37:55.080 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:37:55.080 DEBUG:boto:Port: 8000 2023-05-31T20:37:55.081 DEBUG:boto:Params: {} 2023-05-31T20:37:55.081 DEBUG:boto:Token: None 2023-05-31T20:37:55.082 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:37:55 GMT /pklxkd-58/ 2023-05-31T20:37:55.082 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:lno1rwvBXdlscvtz78c1xk4QhQw= 2023-05-31T20:37:55.083 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:37:55 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:lno1rwvBXdlscvtz78c1xk4QhQw=', 'Content-Length': '0'} 2023-05-31T20:37:55.086 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d7c96bf7f27c8b84-006477b023-1630-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:37:55 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:37:55.088 DEBUG:boto:b'pklxkd-58obj000000000000000000021000falseobj000000000000000000022023-05-31T20:37:28.769Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:37:55.089 DEBUG:boto:path=/pklxkd-58/obj00000000000000000002 2023-05-31T20:37:55.090 DEBUG:boto:auth_path=/pklxkd-58/obj00000000000000000002 2023-05-31T20:37:55.091 DEBUG:boto:Method: HEAD 2023-05-31T20:37:55.091 DEBUG:boto:Path: /pklxkd-58/obj00000000000000000002 2023-05-31T20:37:55.092 DEBUG:boto:Data: 2023-05-31T20:37:55.092 DEBUG:boto:Headers: {} 2023-05-31T20:37:55.092 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:37:55.093 DEBUG:boto:Port: 8001 2023-05-31T20:37:55.093 DEBUG:boto:Params: {} 2023-05-31T20:37:55.093 DEBUG:boto:Token: None 2023-05-31T20:37:55.094 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:37:55 GMT /pklxkd-58/obj00000000000000000002 2023-05-31T20:37:55.094 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:CszJJUwsN4wwtCiCO3WQ4oH2Xbc= 2023-05-31T20:37:55.095 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:37:55 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:CszJJUwsN4wwtCiCO3WQ4oH2Xbc=', 'Content-Length': '0'} 2023-05-31T20:37:55.098 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:37:28 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d09a04051c971ee7-006477b023-1b4f-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:37:55 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:37:55.099 DEBUG:boto:path=/pklxkd-58/obj00000000000000000002 2023-05-31T20:37:55.099 DEBUG:boto:auth_path=/pklxkd-58/obj00000000000000000002 2023-05-31T20:37:55.100 DEBUG:boto:Method: HEAD 2023-05-31T20:37:55.101 DEBUG:boto:Path: /pklxkd-58/obj00000000000000000002 2023-05-31T20:37:55.102 DEBUG:boto:Data: 2023-05-31T20:37:55.103 DEBUG:boto:Headers: {} 2023-05-31T20:37:55.104 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:37:55.105 DEBUG:boto:Port: 8000 2023-05-31T20:37:55.106 DEBUG:boto:Params: {} 2023-05-31T20:37:55.106 DEBUG:boto:Token: None 2023-05-31T20:37:55.107 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:37:55 GMT /pklxkd-58/obj00000000000000000002 2023-05-31T20:37:55.108 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:CszJJUwsN4wwtCiCO3WQ4oH2Xbc= 2023-05-31T20:37:55.109 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:37:55 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:CszJJUwsN4wwtCiCO3WQ4oH2Xbc=', 'Content-Length': '0'} 2023-05-31T20:37:55.112 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:37:28 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f580f64df554ea81-006477b023-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:37:55 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:37:55.113 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000002 2023-05-31T20:37:55.114 DEBUG:boto:path=/pklxkd-58/obj00000000000000000002 2023-05-31T20:37:55.115 DEBUG:boto:auth_path=/pklxkd-58/obj00000000000000000002 2023-05-31T20:37:55.116 DEBUG:boto:Method: GET 2023-05-31T20:37:55.116 DEBUG:boto:Path: /pklxkd-58/obj00000000000000000002 2023-05-31T20:37:55.117 DEBUG:boto:Data: 2023-05-31T20:37:55.118 DEBUG:boto:Headers: {} 2023-05-31T20:37:55.118 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:37:55.119 DEBUG:boto:Port: 8001 2023-05-31T20:37:55.120 DEBUG:boto:Params: {} 2023-05-31T20:37:55.120 DEBUG:boto:Token: None 2023-05-31T20:37:55.121 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:37:55 GMT /pklxkd-58/obj00000000000000000002 2023-05-31T20:37:55.122 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:uBhCmnSjok13bFoI3o4A/EfUIJA= 2023-05-31T20:37:55.122 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:37:55 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:uBhCmnSjok13bFoI3o4A/EfUIJA=', 'Content-Length': '0'} 2023-05-31T20:37:55.126 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:37:28 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ae0ef5dcf825e89c-006477b023-1b4f-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:37:55 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:37:55.168 DEBUG:boto:path=/pklxkd-58/obj00000000000000000002 2023-05-31T20:37:55.169 DEBUG:boto:auth_path=/pklxkd-58/obj00000000000000000002 2023-05-31T20:37:55.170 DEBUG:boto:Method: GET 2023-05-31T20:37:55.170 DEBUG:boto:Path: /pklxkd-58/obj00000000000000000002 2023-05-31T20:37:55.171 DEBUG:boto:Data: 2023-05-31T20:37:55.171 DEBUG:boto:Headers: {} 2023-05-31T20:37:55.172 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:37:55.172 DEBUG:boto:Port: 8000 2023-05-31T20:37:55.172 DEBUG:boto:Params: {} 2023-05-31T20:37:55.173 DEBUG:boto:Token: None 2023-05-31T20:37:55.174 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:37:55 GMT /pklxkd-58/obj00000000000000000002 2023-05-31T20:37:55.174 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:uBhCmnSjok13bFoI3o4A/EfUIJA= 2023-05-31T20:37:55.174 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:37:55 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:uBhCmnSjok13bFoI3o4A/EfUIJA=', 'Content-Length': '0'} 2023-05-31T20:37:55.178 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:37:28 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000045ec0b8400212e79-006477b023-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:37:55 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:37:55.179 DEBUG:boto:path=/pklxkd-58/ 2023-05-31T20:37:55.180 DEBUG:boto:auth_path=/pklxkd-58/ 2023-05-31T20:37:55.181 DEBUG:boto:path=/pklxkd-58/?prefix=obj00000000000000000002 2023-05-31T20:37:55.182 DEBUG:boto:auth_path=/pklxkd-58/?prefix=obj00000000000000000002 2023-05-31T20:37:55.183 DEBUG:boto:Method: GET 2023-05-31T20:37:55.183 DEBUG:boto:Path: /pklxkd-58/?prefix=obj00000000000000000002 2023-05-31T20:37:55.184 DEBUG:boto:Data: 2023-05-31T20:37:55.184 DEBUG:boto:Headers: {} 2023-05-31T20:37:55.185 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:37:55.185 DEBUG:boto:Port: 8001 2023-05-31T20:37:55.186 DEBUG:boto:Params: {} 2023-05-31T20:37:55.186 DEBUG:boto:Token: None 2023-05-31T20:37:55.187 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:37:55 GMT /pklxkd-58/ 2023-05-31T20:37:55.187 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:lno1rwvBXdlscvtz78c1xk4QhQw= 2023-05-31T20:37:55.188 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:37:55 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:lno1rwvBXdlscvtz78c1xk4QhQw=', 'Content-Length': '0'} 2023-05-31T20:37:55.190 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d006e8ff2b3d3c94-006477b023-1b4f-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:37:55 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:37:55.192 DEBUG:boto:b'pklxkd-58obj000000000000000000021000falseobj000000000000000000022023-05-31T20:37:28.769Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:37:55.193 DEBUG:boto:path=/pklxkd-58/ 2023-05-31T20:37:55.194 DEBUG:boto:auth_path=/pklxkd-58/ 2023-05-31T20:37:55.194 DEBUG:boto:path=/pklxkd-58/?prefix=obj00000000000000000002 2023-05-31T20:37:55.195 DEBUG:boto:auth_path=/pklxkd-58/?prefix=obj00000000000000000002 2023-05-31T20:37:55.195 DEBUG:boto:Method: GET 2023-05-31T20:37:55.196 DEBUG:boto:Path: /pklxkd-58/?prefix=obj00000000000000000002 2023-05-31T20:37:55.196 DEBUG:boto:Data: 2023-05-31T20:37:55.197 DEBUG:boto:Headers: {} 2023-05-31T20:37:55.197 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:37:55.197 DEBUG:boto:Port: 8000 2023-05-31T20:37:55.198 DEBUG:boto:Params: {} 2023-05-31T20:37:55.198 DEBUG:boto:Token: None 2023-05-31T20:37:55.198 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:37:55 GMT /pklxkd-58/ 2023-05-31T20:37:55.199 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:lno1rwvBXdlscvtz78c1xk4QhQw= 2023-05-31T20:37:55.199 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:37:55 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:lno1rwvBXdlscvtz78c1xk4QhQw=', 'Content-Length': '0'} 2023-05-31T20:37:55.202 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c63ead0ee0afcd86-006477b023-1630-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:37:55 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:37:55.244 DEBUG:boto:b'pklxkd-58obj000000000000000000021000falseobj000000000000000000022023-05-31T20:37:28.769Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:37:55.245 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000003 2023-05-31T20:37:55.246 DEBUG:boto:path=/pklxkd-58/obj00000000000000000003 2023-05-31T20:37:55.246 DEBUG:boto:auth_path=/pklxkd-58/obj00000000000000000003 2023-05-31T20:37:55.247 DEBUG:boto:path=/pklxkd-58/obj00000000000000000003?versionId=null 2023-05-31T20:37:55.247 DEBUG:boto:auth_path=/pklxkd-58/obj00000000000000000003?versionId=null 2023-05-31T20:37:55.247 DEBUG:boto:Method: GET 2023-05-31T20:37:55.248 DEBUG:boto:Path: /pklxkd-58/obj00000000000000000003?versionId=null 2023-05-31T20:37:55.248 DEBUG:boto:Data: 2023-05-31T20:37:55.249 DEBUG:boto:Headers: {} 2023-05-31T20:37:55.249 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:37:55.249 DEBUG:boto:Port: 8001 2023-05-31T20:37:55.250 DEBUG:boto:Params: {} 2023-05-31T20:37:55.250 DEBUG:boto:Token: None 2023-05-31T20:37:55.251 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:37:55 GMT /pklxkd-58/obj00000000000000000003?versionId=null 2023-05-31T20:37:55.251 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:qklVr6n0LbdcQpURYudrgnEow+0= 2023-05-31T20:37:55.252 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:37:55 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:qklVr6n0LbdcQpURYudrgnEow+0=', 'Content-Length': '0'} 2023-05-31T20:37:55.254 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:37:28 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000229f8cd1272e4e39-006477b023-1b4f-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:37:55 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:37:55.254 DEBUG:boto:path=/pklxkd-58/obj00000000000000000003 2023-05-31T20:37:55.255 DEBUG:boto:auth_path=/pklxkd-58/obj00000000000000000003 2023-05-31T20:37:55.255 DEBUG:boto:path=/pklxkd-58/obj00000000000000000003?versionId=null 2023-05-31T20:37:55.256 DEBUG:boto:auth_path=/pklxkd-58/obj00000000000000000003?versionId=null 2023-05-31T20:37:55.256 DEBUG:boto:Method: GET 2023-05-31T20:37:55.256 DEBUG:boto:Path: /pklxkd-58/obj00000000000000000003?versionId=null 2023-05-31T20:37:55.257 DEBUG:boto:Data: 2023-05-31T20:37:55.257 DEBUG:boto:Headers: {} 2023-05-31T20:37:55.258 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:37:55.258 DEBUG:boto:Port: 8000 2023-05-31T20:37:55.258 DEBUG:boto:Params: {} 2023-05-31T20:37:55.259 DEBUG:boto:Token: None 2023-05-31T20:37:55.259 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:37:55 GMT /pklxkd-58/obj00000000000000000003?versionId=null 2023-05-31T20:37:55.260 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:qklVr6n0LbdcQpURYudrgnEow+0= 2023-05-31T20:37:55.260 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:37:55 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:qklVr6n0LbdcQpURYudrgnEow+0=', 'Content-Length': '0'} 2023-05-31T20:37:55.262 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:37:28 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000caaa467fb74effb9-006477b023-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:37:55 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:37:55.264 DEBUG:boto:path=/pklxkd-58/obj00000000000000000003 2023-05-31T20:37:55.265 DEBUG:boto:auth_path=/pklxkd-58/obj00000000000000000003 2023-05-31T20:37:55.265 DEBUG:boto:path=/pklxkd-58/obj00000000000000000003?versionId=null 2023-05-31T20:37:55.265 DEBUG:boto:auth_path=/pklxkd-58/obj00000000000000000003?versionId=null 2023-05-31T20:37:55.266 DEBUG:boto:Method: HEAD 2023-05-31T20:37:55.266 DEBUG:boto:Path: /pklxkd-58/obj00000000000000000003?versionId=null 2023-05-31T20:37:55.266 DEBUG:boto:Data: 2023-05-31T20:37:55.267 DEBUG:boto:Headers: {} 2023-05-31T20:37:55.267 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:37:55.267 DEBUG:boto:Port: 8001 2023-05-31T20:37:55.268 DEBUG:boto:Params: {} 2023-05-31T20:37:55.268 DEBUG:boto:Token: None 2023-05-31T20:37:55.269 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:37:55 GMT /pklxkd-58/obj00000000000000000003?versionId=null 2023-05-31T20:37:55.269 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:EjeF506zRT/b33TFACkLZ4/XmvA= 2023-05-31T20:37:55.270 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:37:55 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:EjeF506zRT/b33TFACkLZ4/XmvA=', 'Content-Length': '0'} 2023-05-31T20:37:55.271 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:37:28 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000067c8bc61b870189-006477b023-1b4f-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:37:55 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:37:55.272 DEBUG:boto:path=/pklxkd-58/obj00000000000000000003 2023-05-31T20:37:55.273 DEBUG:boto:auth_path=/pklxkd-58/obj00000000000000000003 2023-05-31T20:37:55.273 DEBUG:boto:path=/pklxkd-58/obj00000000000000000003?versionId=null 2023-05-31T20:37:55.273 DEBUG:boto:auth_path=/pklxkd-58/obj00000000000000000003?versionId=null 2023-05-31T20:37:55.274 DEBUG:boto:Method: HEAD 2023-05-31T20:37:55.274 DEBUG:boto:Path: /pklxkd-58/obj00000000000000000003?versionId=null 2023-05-31T20:37:55.274 DEBUG:boto:Data: 2023-05-31T20:37:55.275 DEBUG:boto:Headers: {} 2023-05-31T20:37:55.275 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:37:55.276 DEBUG:boto:Port: 8000 2023-05-31T20:37:55.276 DEBUG:boto:Params: {} 2023-05-31T20:37:55.277 DEBUG:boto:Token: None 2023-05-31T20:37:55.278 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:37:55 GMT /pklxkd-58/obj00000000000000000003?versionId=null 2023-05-31T20:37:55.279 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:EjeF506zRT/b33TFACkLZ4/XmvA= 2023-05-31T20:37:55.280 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:37:55 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:EjeF506zRT/b33TFACkLZ4/XmvA=', 'Content-Length': '0'} 2023-05-31T20:37:55.282 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:37:28 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f2589b53e51cad00-006477b023-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:37:55 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:37:55.283 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000003 2023-05-31T20:37:55.284 DEBUG:boto:path=/pklxkd-58/obj00000000000000000003 2023-05-31T20:37:55.284 DEBUG:boto:auth_path=/pklxkd-58/obj00000000000000000003 2023-05-31T20:37:55.284 DEBUG:boto:path=/pklxkd-58/obj00000000000000000003?versionId=null 2023-05-31T20:37:55.285 DEBUG:boto:auth_path=/pklxkd-58/obj00000000000000000003?versionId=null 2023-05-31T20:37:55.285 DEBUG:boto:Method: GET 2023-05-31T20:37:55.285 DEBUG:boto:Path: /pklxkd-58/obj00000000000000000003?versionId=null 2023-05-31T20:37:55.286 DEBUG:boto:Data: 2023-05-31T20:37:55.287 DEBUG:boto:Headers: {} 2023-05-31T20:37:55.288 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:37:55.289 DEBUG:boto:Port: 8001 2023-05-31T20:37:55.290 DEBUG:boto:Params: {} 2023-05-31T20:37:55.291 DEBUG:boto:Token: None 2023-05-31T20:37:55.292 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:37:55 GMT /pklxkd-58/obj00000000000000000003?versionId=null 2023-05-31T20:37:55.293 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:qklVr6n0LbdcQpURYudrgnEow+0= 2023-05-31T20:37:55.293 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:37:55 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:qklVr6n0LbdcQpURYudrgnEow+0=', 'Content-Length': '0'} 2023-05-31T20:37:55.296 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:37:28 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c6e17ea460ffe3e4-006477b023-1b4f-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:37:55 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:37:55.297 DEBUG:boto:path=/pklxkd-58/obj00000000000000000003 2023-05-31T20:37:55.297 DEBUG:boto:auth_path=/pklxkd-58/obj00000000000000000003 2023-05-31T20:37:55.298 DEBUG:boto:path=/pklxkd-58/obj00000000000000000003?versionId=null 2023-05-31T20:37:55.298 DEBUG:boto:auth_path=/pklxkd-58/obj00000000000000000003?versionId=null 2023-05-31T20:37:55.299 DEBUG:boto:Method: GET 2023-05-31T20:37:55.299 DEBUG:boto:Path: /pklxkd-58/obj00000000000000000003?versionId=null 2023-05-31T20:37:55.300 DEBUG:boto:Data: 2023-05-31T20:37:55.300 DEBUG:boto:Headers: {} 2023-05-31T20:37:55.300 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:37:55.301 DEBUG:boto:Port: 8000 2023-05-31T20:37:55.301 DEBUG:boto:Params: {} 2023-05-31T20:37:55.302 DEBUG:boto:Token: None 2023-05-31T20:37:55.302 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:37:55 GMT /pklxkd-58/obj00000000000000000003?versionId=null 2023-05-31T20:37:55.302 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:qklVr6n0LbdcQpURYudrgnEow+0= 2023-05-31T20:37:55.303 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:37:55 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:qklVr6n0LbdcQpURYudrgnEow+0=', 'Content-Length': '0'} 2023-05-31T20:37:55.305 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:37:28 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009cc855ed55e3cb83-006477b023-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:37:55 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:37:55.306 DEBUG:boto:path=/pklxkd-58/ 2023-05-31T20:37:55.307 DEBUG:boto:auth_path=/pklxkd-58/ 2023-05-31T20:37:55.307 DEBUG:boto:path=/pklxkd-58/?prefix=obj00000000000000000003 2023-05-31T20:37:55.307 DEBUG:boto:auth_path=/pklxkd-58/?prefix=obj00000000000000000003 2023-05-31T20:37:55.308 DEBUG:boto:Method: GET 2023-05-31T20:37:55.308 DEBUG:boto:Path: /pklxkd-58/?prefix=obj00000000000000000003 2023-05-31T20:37:55.309 DEBUG:boto:Data: 2023-05-31T20:37:55.309 DEBUG:boto:Headers: {} 2023-05-31T20:37:55.309 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:37:55.310 DEBUG:boto:Port: 8001 2023-05-31T20:37:55.310 DEBUG:boto:Params: {} 2023-05-31T20:37:55.310 DEBUG:boto:Token: None 2023-05-31T20:37:55.311 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:37:55 GMT /pklxkd-58/ 2023-05-31T20:37:55.311 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:lno1rwvBXdlscvtz78c1xk4QhQw= 2023-05-31T20:37:55.312 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:37:55 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:lno1rwvBXdlscvtz78c1xk4QhQw=', 'Content-Length': '0'} 2023-05-31T20:37:55.314 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000064c0d498cc892d2b-006477b023-1b4f-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:37:55 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:37:55.315 DEBUG:boto:b'pklxkd-58obj000000000000000000031000falseobj000000000000000000032023-05-31T20:37:28.957Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:37:55.315 DEBUG:boto:path=/pklxkd-58/ 2023-05-31T20:37:55.316 DEBUG:boto:auth_path=/pklxkd-58/ 2023-05-31T20:37:55.316 DEBUG:boto:path=/pklxkd-58/?prefix=obj00000000000000000003 2023-05-31T20:37:55.317 DEBUG:boto:auth_path=/pklxkd-58/?prefix=obj00000000000000000003 2023-05-31T20:37:55.317 DEBUG:boto:Method: GET 2023-05-31T20:37:55.317 DEBUG:boto:Path: /pklxkd-58/?prefix=obj00000000000000000003 2023-05-31T20:37:55.318 DEBUG:boto:Data: 2023-05-31T20:37:55.318 DEBUG:boto:Headers: {} 2023-05-31T20:37:55.318 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:37:55.319 DEBUG:boto:Port: 8000 2023-05-31T20:37:55.319 DEBUG:boto:Params: {} 2023-05-31T20:37:55.319 DEBUG:boto:Token: None 2023-05-31T20:37:55.320 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:37:55 GMT /pklxkd-58/ 2023-05-31T20:37:55.320 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:lno1rwvBXdlscvtz78c1xk4QhQw= 2023-05-31T20:37:55.321 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:37:55 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:lno1rwvBXdlscvtz78c1xk4QhQw=', 'Content-Length': '0'} 2023-05-31T20:37:55.323 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000bff85c51150b3629-006477b023-1630-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:37:55 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:37:55.324 DEBUG:boto:b'pklxkd-58obj000000000000000000031000falseobj000000000000000000032023-05-31T20:37:28.957Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:37:55.324 DEBUG:boto:path=/pklxkd-58/obj00000000000000000003 2023-05-31T20:37:55.325 DEBUG:boto:auth_path=/pklxkd-58/obj00000000000000000003 2023-05-31T20:37:55.325 DEBUG:boto:Method: HEAD 2023-05-31T20:37:55.325 DEBUG:boto:Path: /pklxkd-58/obj00000000000000000003 2023-05-31T20:37:55.326 DEBUG:boto:Data: 2023-05-31T20:37:55.326 DEBUG:boto:Headers: {} 2023-05-31T20:37:55.326 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:37:55.327 DEBUG:boto:Port: 8001 2023-05-31T20:37:55.327 DEBUG:boto:Params: {} 2023-05-31T20:37:55.328 DEBUG:boto:Token: None 2023-05-31T20:37:55.328 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:37:55 GMT /pklxkd-58/obj00000000000000000003 2023-05-31T20:37:55.328 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:wgObU7tAvtDh/T59hJ7coclxc7k= 2023-05-31T20:37:55.329 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:37:55 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:wgObU7tAvtDh/T59hJ7coclxc7k=', 'Content-Length': '0'} 2023-05-31T20:37:55.331 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:37:28 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000041ac4b6aa666963e-006477b023-1b4f-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:37:55 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:37:55.331 DEBUG:boto:path=/pklxkd-58/obj00000000000000000003 2023-05-31T20:37:55.332 DEBUG:boto:auth_path=/pklxkd-58/obj00000000000000000003 2023-05-31T20:37:55.332 DEBUG:boto:Method: HEAD 2023-05-31T20:37:55.332 DEBUG:boto:Path: /pklxkd-58/obj00000000000000000003 2023-05-31T20:37:55.333 DEBUG:boto:Data: 2023-05-31T20:37:55.333 DEBUG:boto:Headers: {} 2023-05-31T20:37:55.333 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:37:55.334 DEBUG:boto:Port: 8000 2023-05-31T20:37:55.334 DEBUG:boto:Params: {} 2023-05-31T20:37:55.334 DEBUG:boto:Token: None 2023-05-31T20:37:55.335 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:37:55 GMT /pklxkd-58/obj00000000000000000003 2023-05-31T20:37:55.335 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:wgObU7tAvtDh/T59hJ7coclxc7k= 2023-05-31T20:37:55.336 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:37:55 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:wgObU7tAvtDh/T59hJ7coclxc7k=', 'Content-Length': '0'} 2023-05-31T20:37:55.337 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:37:28 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e4fb91f7ec6dd6b3-006477b023-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:37:55 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:37:55.338 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000003 2023-05-31T20:37:55.338 DEBUG:boto:path=/pklxkd-58/obj00000000000000000003 2023-05-31T20:37:55.339 DEBUG:boto:auth_path=/pklxkd-58/obj00000000000000000003 2023-05-31T20:37:55.339 DEBUG:boto:Method: GET 2023-05-31T20:37:55.339 DEBUG:boto:Path: /pklxkd-58/obj00000000000000000003 2023-05-31T20:37:55.340 DEBUG:boto:Data: 2023-05-31T20:37:55.340 DEBUG:boto:Headers: {} 2023-05-31T20:37:55.341 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:37:55.341 DEBUG:boto:Port: 8001 2023-05-31T20:37:55.341 DEBUG:boto:Params: {} 2023-05-31T20:37:55.342 DEBUG:boto:Token: None 2023-05-31T20:37:55.342 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:37:55 GMT /pklxkd-58/obj00000000000000000003 2023-05-31T20:37:55.343 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:0tyvmerNC3Y+AVv3COgkSNlOgv4= 2023-05-31T20:37:55.343 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:37:55 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:0tyvmerNC3Y+AVv3COgkSNlOgv4=', 'Content-Length': '0'} 2023-05-31T20:37:55.345 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:37:28 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000446e5a6c402cb401-006477b023-1b4f-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:37:55 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:37:55.346 DEBUG:boto:path=/pklxkd-58/obj00000000000000000003 2023-05-31T20:37:55.346 DEBUG:boto:auth_path=/pklxkd-58/obj00000000000000000003 2023-05-31T20:37:55.347 DEBUG:boto:Method: GET 2023-05-31T20:37:55.347 DEBUG:boto:Path: /pklxkd-58/obj00000000000000000003 2023-05-31T20:37:55.347 DEBUG:boto:Data: 2023-05-31T20:37:55.348 DEBUG:boto:Headers: {} 2023-05-31T20:37:55.348 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:37:55.348 DEBUG:boto:Port: 8000 2023-05-31T20:37:55.349 DEBUG:boto:Params: {} 2023-05-31T20:37:55.349 DEBUG:boto:Token: None 2023-05-31T20:37:55.350 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:37:55 GMT /pklxkd-58/obj00000000000000000003 2023-05-31T20:37:55.350 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:0tyvmerNC3Y+AVv3COgkSNlOgv4= 2023-05-31T20:37:55.350 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:37:55 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:0tyvmerNC3Y+AVv3COgkSNlOgv4=', 'Content-Length': '0'} 2023-05-31T20:37:55.352 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:37:28 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000077d331a83577575c-006477b023-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:37:55 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:37:55.353 DEBUG:boto:path=/pklxkd-58/ 2023-05-31T20:37:55.353 DEBUG:boto:auth_path=/pklxkd-58/ 2023-05-31T20:37:55.354 DEBUG:boto:path=/pklxkd-58/?prefix=obj00000000000000000003 2023-05-31T20:37:55.354 DEBUG:boto:auth_path=/pklxkd-58/?prefix=obj00000000000000000003 2023-05-31T20:37:55.355 DEBUG:boto:Method: GET 2023-05-31T20:37:55.355 DEBUG:boto:Path: /pklxkd-58/?prefix=obj00000000000000000003 2023-05-31T20:37:55.355 DEBUG:boto:Data: 2023-05-31T20:37:55.356 DEBUG:boto:Headers: {} 2023-05-31T20:37:55.356 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:37:55.356 DEBUG:boto:Port: 8001 2023-05-31T20:37:55.357 DEBUG:boto:Params: {} 2023-05-31T20:37:55.357 DEBUG:boto:Token: None 2023-05-31T20:37:55.358 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:37:55 GMT /pklxkd-58/ 2023-05-31T20:37:55.358 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:lno1rwvBXdlscvtz78c1xk4QhQw= 2023-05-31T20:37:55.358 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:37:55 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:lno1rwvBXdlscvtz78c1xk4QhQw=', 'Content-Length': '0'} 2023-05-31T20:37:55.361 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000165dda927dfa7e70-006477b023-1b4f-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:37:55 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:37:55.361 DEBUG:boto:b'pklxkd-58obj000000000000000000031000falseobj000000000000000000032023-05-31T20:37:28.957Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:37:55.362 DEBUG:boto:path=/pklxkd-58/ 2023-05-31T20:37:55.362 DEBUG:boto:auth_path=/pklxkd-58/ 2023-05-31T20:37:55.363 DEBUG:boto:path=/pklxkd-58/?prefix=obj00000000000000000003 2023-05-31T20:37:55.363 DEBUG:boto:auth_path=/pklxkd-58/?prefix=obj00000000000000000003 2023-05-31T20:37:55.364 DEBUG:boto:Method: GET 2023-05-31T20:37:55.364 DEBUG:boto:Path: /pklxkd-58/?prefix=obj00000000000000000003 2023-05-31T20:37:55.364 DEBUG:boto:Data: 2023-05-31T20:37:55.365 DEBUG:boto:Headers: {} 2023-05-31T20:37:55.365 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:37:55.365 DEBUG:boto:Port: 8000 2023-05-31T20:37:55.366 DEBUG:boto:Params: {} 2023-05-31T20:37:55.366 DEBUG:boto:Token: None 2023-05-31T20:37:55.367 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:37:55 GMT /pklxkd-58/ 2023-05-31T20:37:55.367 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:lno1rwvBXdlscvtz78c1xk4QhQw= 2023-05-31T20:37:55.367 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:37:55 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:lno1rwvBXdlscvtz78c1xk4QhQw=', 'Content-Length': '0'} 2023-05-31T20:37:55.370 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000feddaaeb1a8dc223-006477b023-1630-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:37:55 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:37:55.416 DEBUG:boto:b'pklxkd-58obj000000000000000000031000falseobj000000000000000000032023-05-31T20:37:28.957Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:37:55.417 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000004 2023-05-31T20:37:55.417 DEBUG:boto:path=/pklxkd-58/obj00000000000000000004 2023-05-31T20:37:55.418 DEBUG:boto:auth_path=/pklxkd-58/obj00000000000000000004 2023-05-31T20:37:55.418 DEBUG:boto:path=/pklxkd-58/obj00000000000000000004?versionId=null 2023-05-31T20:37:55.419 DEBUG:boto:auth_path=/pklxkd-58/obj00000000000000000004?versionId=null 2023-05-31T20:37:55.419 DEBUG:boto:Method: GET 2023-05-31T20:37:55.419 DEBUG:boto:Path: /pklxkd-58/obj00000000000000000004?versionId=null 2023-05-31T20:37:55.420 DEBUG:boto:Data: 2023-05-31T20:37:55.420 DEBUG:boto:Headers: {} 2023-05-31T20:37:55.420 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:37:55.421 DEBUG:boto:Port: 8001 2023-05-31T20:37:55.421 DEBUG:boto:Params: {} 2023-05-31T20:37:55.422 DEBUG:boto:Token: None 2023-05-31T20:37:55.422 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:37:55 GMT /pklxkd-58/obj00000000000000000004?versionId=null 2023-05-31T20:37:55.422 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:lXJaFAN8nNSPldhC13mD1pdArDw= 2023-05-31T20:37:55.423 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:37:55 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:lXJaFAN8nNSPldhC13mD1pdArDw=', 'Content-Length': '0'} 2023-05-31T20:37:55.425 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:37:28 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000efea4d6ebf5db5aa-006477b023-1b4f-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:37:55 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:37:55.425 DEBUG:boto:path=/pklxkd-58/obj00000000000000000004 2023-05-31T20:37:55.426 DEBUG:boto:auth_path=/pklxkd-58/obj00000000000000000004 2023-05-31T20:37:55.426 DEBUG:boto:path=/pklxkd-58/obj00000000000000000004?versionId=null 2023-05-31T20:37:55.427 DEBUG:boto:auth_path=/pklxkd-58/obj00000000000000000004?versionId=null 2023-05-31T20:37:55.427 DEBUG:boto:Method: GET 2023-05-31T20:37:55.427 DEBUG:boto:Path: /pklxkd-58/obj00000000000000000004?versionId=null 2023-05-31T20:37:55.428 DEBUG:boto:Data: 2023-05-31T20:37:55.428 DEBUG:boto:Headers: {} 2023-05-31T20:37:55.428 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:37:55.429 DEBUG:boto:Port: 8000 2023-05-31T20:37:55.429 DEBUG:boto:Params: {} 2023-05-31T20:37:55.429 DEBUG:boto:Token: None 2023-05-31T20:37:55.430 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:37:55 GMT /pklxkd-58/obj00000000000000000004?versionId=null 2023-05-31T20:37:55.430 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:lXJaFAN8nNSPldhC13mD1pdArDw= 2023-05-31T20:37:55.431 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:37:55 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:lXJaFAN8nNSPldhC13mD1pdArDw=', 'Content-Length': '0'} 2023-05-31T20:37:55.432 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:37:28 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006b645f0de7a95a72-006477b023-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:37:55 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:37:55.433 DEBUG:boto:path=/pklxkd-58/obj00000000000000000004 2023-05-31T20:37:55.434 DEBUG:boto:auth_path=/pklxkd-58/obj00000000000000000004 2023-05-31T20:37:55.434 DEBUG:boto:path=/pklxkd-58/obj00000000000000000004?versionId=null 2023-05-31T20:37:55.434 DEBUG:boto:auth_path=/pklxkd-58/obj00000000000000000004?versionId=null 2023-05-31T20:37:55.435 DEBUG:boto:Method: HEAD 2023-05-31T20:37:55.435 DEBUG:boto:Path: /pklxkd-58/obj00000000000000000004?versionId=null 2023-05-31T20:37:55.435 DEBUG:boto:Data: 2023-05-31T20:37:55.436 DEBUG:boto:Headers: {} 2023-05-31T20:37:55.436 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:37:55.436 DEBUG:boto:Port: 8001 2023-05-31T20:37:55.437 DEBUG:boto:Params: {} 2023-05-31T20:37:55.437 DEBUG:boto:Token: None 2023-05-31T20:37:55.437 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:37:55 GMT /pklxkd-58/obj00000000000000000004?versionId=null 2023-05-31T20:37:55.438 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:nLACpKeGWLZcRY4mPAwqXFVY+jM= 2023-05-31T20:37:55.438 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:37:55 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:nLACpKeGWLZcRY4mPAwqXFVY+jM=', 'Content-Length': '0'} 2023-05-31T20:37:55.440 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:37:28 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000828966c58b511707-006477b023-1b4f-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:37:55 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:37:55.441 DEBUG:boto:path=/pklxkd-58/obj00000000000000000004 2023-05-31T20:37:55.441 DEBUG:boto:auth_path=/pklxkd-58/obj00000000000000000004 2023-05-31T20:37:55.441 DEBUG:boto:path=/pklxkd-58/obj00000000000000000004?versionId=null 2023-05-31T20:37:55.442 DEBUG:boto:auth_path=/pklxkd-58/obj00000000000000000004?versionId=null 2023-05-31T20:37:55.442 DEBUG:boto:Method: HEAD 2023-05-31T20:37:55.442 DEBUG:boto:Path: /pklxkd-58/obj00000000000000000004?versionId=null 2023-05-31T20:37:55.443 DEBUG:boto:Data: 2023-05-31T20:37:55.443 DEBUG:boto:Headers: {} 2023-05-31T20:37:55.443 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:37:55.444 DEBUG:boto:Port: 8000 2023-05-31T20:37:55.444 DEBUG:boto:Params: {} 2023-05-31T20:37:55.444 DEBUG:boto:Token: None 2023-05-31T20:37:55.445 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:37:55 GMT /pklxkd-58/obj00000000000000000004?versionId=null 2023-05-31T20:37:55.445 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:nLACpKeGWLZcRY4mPAwqXFVY+jM= 2023-05-31T20:37:55.446 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:37:55 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:nLACpKeGWLZcRY4mPAwqXFVY+jM=', 'Content-Length': '0'} 2023-05-31T20:37:55.447 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:37:28 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005bcad06ed14a71d6-006477b023-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:37:55 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:37:55.448 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000004 2023-05-31T20:37:55.448 DEBUG:boto:path=/pklxkd-58/obj00000000000000000004 2023-05-31T20:37:55.449 DEBUG:boto:auth_path=/pklxkd-58/obj00000000000000000004 2023-05-31T20:37:55.449 DEBUG:boto:path=/pklxkd-58/obj00000000000000000004?versionId=null 2023-05-31T20:37:55.450 DEBUG:boto:auth_path=/pklxkd-58/obj00000000000000000004?versionId=null 2023-05-31T20:37:55.450 DEBUG:boto:Method: GET 2023-05-31T20:37:55.450 DEBUG:boto:Path: /pklxkd-58/obj00000000000000000004?versionId=null 2023-05-31T20:37:55.451 DEBUG:boto:Data: 2023-05-31T20:37:55.451 DEBUG:boto:Headers: {} 2023-05-31T20:37:55.451 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:37:55.452 DEBUG:boto:Port: 8001 2023-05-31T20:37:55.452 DEBUG:boto:Params: {} 2023-05-31T20:37:55.453 DEBUG:boto:Token: None 2023-05-31T20:37:55.453 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:37:55 GMT /pklxkd-58/obj00000000000000000004?versionId=null 2023-05-31T20:37:55.454 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:lXJaFAN8nNSPldhC13mD1pdArDw= 2023-05-31T20:37:55.454 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:37:55 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:lXJaFAN8nNSPldhC13mD1pdArDw=', 'Content-Length': '0'} 2023-05-31T20:37:55.456 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:37:28 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000060cfab1d4098f8ed-006477b023-1b4f-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:37:55 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:37:55.496 DEBUG:boto:path=/pklxkd-58/obj00000000000000000004 2023-05-31T20:37:55.497 DEBUG:boto:auth_path=/pklxkd-58/obj00000000000000000004 2023-05-31T20:37:55.497 DEBUG:boto:path=/pklxkd-58/obj00000000000000000004?versionId=null 2023-05-31T20:37:55.498 DEBUG:boto:auth_path=/pklxkd-58/obj00000000000000000004?versionId=null 2023-05-31T20:37:55.498 DEBUG:boto:Method: GET 2023-05-31T20:37:55.499 DEBUG:boto:Path: /pklxkd-58/obj00000000000000000004?versionId=null 2023-05-31T20:37:55.499 DEBUG:boto:Data: 2023-05-31T20:37:55.499 DEBUG:boto:Headers: {} 2023-05-31T20:37:55.500 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:37:55.500 DEBUG:boto:Port: 8000 2023-05-31T20:37:55.500 DEBUG:boto:Params: {} 2023-05-31T20:37:55.501 DEBUG:boto:Token: None 2023-05-31T20:37:55.501 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:37:55 GMT /pklxkd-58/obj00000000000000000004?versionId=null 2023-05-31T20:37:55.502 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:lXJaFAN8nNSPldhC13mD1pdArDw= 2023-05-31T20:37:55.502 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:37:55 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:lXJaFAN8nNSPldhC13mD1pdArDw=', 'Content-Length': '0'} 2023-05-31T20:37:55.504 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:37:28 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000029b2fd19a541e3f0-006477b023-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:37:55 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:37:55.505 DEBUG:boto:path=/pklxkd-58/ 2023-05-31T20:37:55.505 DEBUG:boto:auth_path=/pklxkd-58/ 2023-05-31T20:37:55.506 DEBUG:boto:path=/pklxkd-58/?prefix=obj00000000000000000004 2023-05-31T20:37:55.506 DEBUG:boto:auth_path=/pklxkd-58/?prefix=obj00000000000000000004 2023-05-31T20:37:55.506 DEBUG:boto:Method: GET 2023-05-31T20:37:55.507 DEBUG:boto:Path: /pklxkd-58/?prefix=obj00000000000000000004 2023-05-31T20:37:55.507 DEBUG:boto:Data: 2023-05-31T20:37:55.507 DEBUG:boto:Headers: {} 2023-05-31T20:37:55.508 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:37:55.508 DEBUG:boto:Port: 8001 2023-05-31T20:37:55.509 DEBUG:boto:Params: {} 2023-05-31T20:37:55.509 DEBUG:boto:Token: None 2023-05-31T20:37:55.509 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:37:55 GMT /pklxkd-58/ 2023-05-31T20:37:55.510 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:lno1rwvBXdlscvtz78c1xk4QhQw= 2023-05-31T20:37:55.510 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:37:55 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:lno1rwvBXdlscvtz78c1xk4QhQw=', 'Content-Length': '0'} 2023-05-31T20:37:55.513 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a6aac52490416232-006477b023-1b4f-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:37:55 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:37:55.513 DEBUG:boto:b'pklxkd-58obj000000000000000000041000falseobj000000000000000000042023-05-31T20:37:28.895Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:37:55.514 DEBUG:boto:path=/pklxkd-58/ 2023-05-31T20:37:55.514 DEBUG:boto:auth_path=/pklxkd-58/ 2023-05-31T20:37:55.515 DEBUG:boto:path=/pklxkd-58/?prefix=obj00000000000000000004 2023-05-31T20:37:55.515 DEBUG:boto:auth_path=/pklxkd-58/?prefix=obj00000000000000000004 2023-05-31T20:37:55.515 DEBUG:boto:Method: GET 2023-05-31T20:37:55.516 DEBUG:boto:Path: /pklxkd-58/?prefix=obj00000000000000000004 2023-05-31T20:37:55.516 DEBUG:boto:Data: 2023-05-31T20:37:55.516 DEBUG:boto:Headers: {} 2023-05-31T20:37:55.517 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:37:55.517 DEBUG:boto:Port: 8000 2023-05-31T20:37:55.517 DEBUG:boto:Params: {} 2023-05-31T20:37:55.518 DEBUG:boto:Token: None 2023-05-31T20:37:55.518 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:37:55 GMT /pklxkd-58/ 2023-05-31T20:37:55.519 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:lno1rwvBXdlscvtz78c1xk4QhQw= 2023-05-31T20:37:55.519 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:37:55 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:lno1rwvBXdlscvtz78c1xk4QhQw=', 'Content-Length': '0'} 2023-05-31T20:37:55.522 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000628090bc433bb79f-006477b023-1630-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:37:55 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:37:55.523 DEBUG:boto:b'pklxkd-58obj000000000000000000041000falseobj000000000000000000042023-05-31T20:37:28.895Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:37:55.523 DEBUG:boto:path=/pklxkd-58/obj00000000000000000004 2023-05-31T20:37:55.524 DEBUG:boto:auth_path=/pklxkd-58/obj00000000000000000004 2023-05-31T20:37:55.524 DEBUG:boto:Method: HEAD 2023-05-31T20:37:55.525 DEBUG:boto:Path: /pklxkd-58/obj00000000000000000004 2023-05-31T20:37:55.525 DEBUG:boto:Data: 2023-05-31T20:37:55.525 DEBUG:boto:Headers: {} 2023-05-31T20:37:55.526 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:37:55.526 DEBUG:boto:Port: 8001 2023-05-31T20:37:55.526 DEBUG:boto:Params: {} 2023-05-31T20:37:55.527 DEBUG:boto:Token: None 2023-05-31T20:37:55.527 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:37:55 GMT /pklxkd-58/obj00000000000000000004 2023-05-31T20:37:55.528 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:03tWsUXGqHqMm3r4GNNRIpDme2g= 2023-05-31T20:37:55.528 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:37:55 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:03tWsUXGqHqMm3r4GNNRIpDme2g=', 'Content-Length': '0'} 2023-05-31T20:37:55.530 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:37:28 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b177bd1992637918-006477b023-1b4f-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:37:55 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:37:55.530 DEBUG:boto:path=/pklxkd-58/obj00000000000000000004 2023-05-31T20:37:55.531 DEBUG:boto:auth_path=/pklxkd-58/obj00000000000000000004 2023-05-31T20:37:55.531 DEBUG:boto:Method: HEAD 2023-05-31T20:37:55.532 DEBUG:boto:Path: /pklxkd-58/obj00000000000000000004 2023-05-31T20:37:55.532 DEBUG:boto:Data: 2023-05-31T20:37:55.532 DEBUG:boto:Headers: {} 2023-05-31T20:37:55.533 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:37:55.533 DEBUG:boto:Port: 8000 2023-05-31T20:37:55.533 DEBUG:boto:Params: {} 2023-05-31T20:37:55.534 DEBUG:boto:Token: None 2023-05-31T20:37:55.534 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:37:55 GMT /pklxkd-58/obj00000000000000000004 2023-05-31T20:37:55.535 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:03tWsUXGqHqMm3r4GNNRIpDme2g= 2023-05-31T20:37:55.535 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:37:55 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:03tWsUXGqHqMm3r4GNNRIpDme2g=', 'Content-Length': '0'} 2023-05-31T20:37:55.537 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:37:28 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b6541fe440a5edc3-006477b023-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:37:55 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:37:55.537 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000004 2023-05-31T20:37:55.538 DEBUG:boto:path=/pklxkd-58/obj00000000000000000004 2023-05-31T20:37:55.538 DEBUG:boto:auth_path=/pklxkd-58/obj00000000000000000004 2023-05-31T20:37:55.539 DEBUG:boto:Method: GET 2023-05-31T20:37:55.539 DEBUG:boto:Path: /pklxkd-58/obj00000000000000000004 2023-05-31T20:37:55.539 DEBUG:boto:Data: 2023-05-31T20:37:55.540 DEBUG:boto:Headers: {} 2023-05-31T20:37:55.540 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:37:55.541 DEBUG:boto:Port: 8001 2023-05-31T20:37:55.541 DEBUG:boto:Params: {} 2023-05-31T20:37:55.541 DEBUG:boto:Token: None 2023-05-31T20:37:55.542 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:37:55 GMT /pklxkd-58/obj00000000000000000004 2023-05-31T20:37:55.542 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:fU40ULiBpf3znJgRnDqFVAXr3MU= 2023-05-31T20:37:55.543 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:37:55 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:fU40ULiBpf3znJgRnDqFVAXr3MU=', 'Content-Length': '0'} 2023-05-31T20:37:55.545 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:37:28 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b0ea77141eb86a70-006477b023-1b4f-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:37:55 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:37:55.545 DEBUG:boto:path=/pklxkd-58/obj00000000000000000004 2023-05-31T20:37:55.546 DEBUG:boto:auth_path=/pklxkd-58/obj00000000000000000004 2023-05-31T20:37:55.546 DEBUG:boto:Method: GET 2023-05-31T20:37:55.547 DEBUG:boto:Path: /pklxkd-58/obj00000000000000000004 2023-05-31T20:37:55.547 DEBUG:boto:Data: 2023-05-31T20:37:55.547 DEBUG:boto:Headers: {} 2023-05-31T20:37:55.548 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:37:55.548 DEBUG:boto:Port: 8000 2023-05-31T20:37:55.549 DEBUG:boto:Params: {} 2023-05-31T20:37:55.550 DEBUG:boto:Token: None 2023-05-31T20:37:55.550 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:37:55 GMT /pklxkd-58/obj00000000000000000004 2023-05-31T20:37:55.551 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:fU40ULiBpf3znJgRnDqFVAXr3MU= 2023-05-31T20:37:55.552 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:37:55 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:fU40ULiBpf3znJgRnDqFVAXr3MU=', 'Content-Length': '0'} 2023-05-31T20:37:55.555 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:37:28 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003818afba590ff83f-006477b023-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:37:55 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:37:55.596 DEBUG:boto:path=/pklxkd-58/ 2023-05-31T20:37:55.597 DEBUG:boto:auth_path=/pklxkd-58/ 2023-05-31T20:37:55.597 DEBUG:boto:path=/pklxkd-58/?prefix=obj00000000000000000004 2023-05-31T20:37:55.598 DEBUG:boto:auth_path=/pklxkd-58/?prefix=obj00000000000000000004 2023-05-31T20:37:55.598 DEBUG:boto:Method: GET 2023-05-31T20:37:55.598 DEBUG:boto:Path: /pklxkd-58/?prefix=obj00000000000000000004 2023-05-31T20:37:55.599 DEBUG:boto:Data: 2023-05-31T20:37:55.599 DEBUG:boto:Headers: {} 2023-05-31T20:37:55.599 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:37:55.600 DEBUG:boto:Port: 8001 2023-05-31T20:37:55.600 DEBUG:boto:Params: {} 2023-05-31T20:37:55.601 DEBUG:boto:Token: None 2023-05-31T20:37:55.601 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:37:55 GMT /pklxkd-58/ 2023-05-31T20:37:55.602 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:lno1rwvBXdlscvtz78c1xk4QhQw= 2023-05-31T20:37:55.605 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:37:55 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:lno1rwvBXdlscvtz78c1xk4QhQw=', 'Content-Length': '0'} 2023-05-31T20:37:55.607 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000002ae2cd8c74beff66-006477b023-1b4f-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:37:55 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:37:55.608 DEBUG:boto:b'pklxkd-58obj000000000000000000041000falseobj000000000000000000042023-05-31T20:37:28.895Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:37:55.609 DEBUG:boto:path=/pklxkd-58/ 2023-05-31T20:37:55.609 DEBUG:boto:auth_path=/pklxkd-58/ 2023-05-31T20:37:55.610 DEBUG:boto:path=/pklxkd-58/?prefix=obj00000000000000000004 2023-05-31T20:37:55.610 DEBUG:boto:auth_path=/pklxkd-58/?prefix=obj00000000000000000004 2023-05-31T20:37:55.611 DEBUG:boto:Method: GET 2023-05-31T20:37:55.611 DEBUG:boto:Path: /pklxkd-58/?prefix=obj00000000000000000004 2023-05-31T20:37:55.612 DEBUG:boto:Data: 2023-05-31T20:37:55.613 DEBUG:boto:Headers: {} 2023-05-31T20:37:55.613 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:37:55.614 DEBUG:boto:Port: 8000 2023-05-31T20:37:55.614 DEBUG:boto:Params: {} 2023-05-31T20:37:55.615 DEBUG:boto:Token: None 2023-05-31T20:37:55.615 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:37:55 GMT /pklxkd-58/ 2023-05-31T20:37:55.616 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:lno1rwvBXdlscvtz78c1xk4QhQw= 2023-05-31T20:37:55.616 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:37:55 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:lno1rwvBXdlscvtz78c1xk4QhQw=', 'Content-Length': '0'} 2023-05-31T20:37:55.619 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000374a38dff68fccc2-006477b023-1630-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:37:55 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:37:55.619 DEBUG:boto:b'pklxkd-58obj000000000000000000041000falseobj000000000000000000042023-05-31T20:37:28.895Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:37:55.620 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000006 2023-05-31T20:37:55.621 DEBUG:boto:path=/pklxkd-58/obj00000000000000000006 2023-05-31T20:37:55.621 DEBUG:boto:auth_path=/pklxkd-58/obj00000000000000000006 2023-05-31T20:37:55.622 DEBUG:boto:path=/pklxkd-58/obj00000000000000000006?versionId=null 2023-05-31T20:37:55.623 DEBUG:boto:auth_path=/pklxkd-58/obj00000000000000000006?versionId=null 2023-05-31T20:37:55.624 DEBUG:boto:Method: GET 2023-05-31T20:37:55.626 DEBUG:boto:Path: /pklxkd-58/obj00000000000000000006?versionId=null 2023-05-31T20:37:55.627 DEBUG:boto:Data: 2023-05-31T20:37:55.628 DEBUG:boto:Headers: {} 2023-05-31T20:37:55.628 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:37:55.629 DEBUG:boto:Port: 8001 2023-05-31T20:37:55.629 DEBUG:boto:Params: {} 2023-05-31T20:37:55.630 DEBUG:boto:Token: None 2023-05-31T20:37:55.630 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:37:55 GMT /pklxkd-58/obj00000000000000000006?versionId=null 2023-05-31T20:37:55.631 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:qKf6wNDC62Xm0HqkuoDcJf7jvUk= 2023-05-31T20:37:55.631 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:37:55 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:qKf6wNDC62Xm0HqkuoDcJf7jvUk=', 'Content-Length': '0'} 2023-05-31T20:37:55.633 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:37:29 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004f05024e67a22dd7-006477b023-1b4f-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:37:55 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:37:55.634 DEBUG:boto:path=/pklxkd-58/obj00000000000000000006 2023-05-31T20:37:55.635 DEBUG:boto:auth_path=/pklxkd-58/obj00000000000000000006 2023-05-31T20:37:55.635 DEBUG:boto:path=/pklxkd-58/obj00000000000000000006?versionId=null 2023-05-31T20:37:55.635 DEBUG:boto:auth_path=/pklxkd-58/obj00000000000000000006?versionId=null 2023-05-31T20:37:55.636 DEBUG:boto:Method: GET 2023-05-31T20:37:55.636 DEBUG:boto:Path: /pklxkd-58/obj00000000000000000006?versionId=null 2023-05-31T20:37:55.637 DEBUG:boto:Data: 2023-05-31T20:37:55.637 DEBUG:boto:Headers: {} 2023-05-31T20:37:55.637 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:37:55.638 DEBUG:boto:Port: 8000 2023-05-31T20:37:55.640 DEBUG:boto:Params: {} 2023-05-31T20:37:55.641 DEBUG:boto:Token: None 2023-05-31T20:37:55.641 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:37:55 GMT /pklxkd-58/obj00000000000000000006?versionId=null 2023-05-31T20:37:55.642 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:qKf6wNDC62Xm0HqkuoDcJf7jvUk= 2023-05-31T20:37:55.643 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:37:55 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:qKf6wNDC62Xm0HqkuoDcJf7jvUk=', 'Content-Length': '0'} 2023-05-31T20:37:55.646 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:37:29 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e3f8c30db1bc0660-006477b023-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:37:55 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:37:55.646 DEBUG:boto:path=/pklxkd-58/obj00000000000000000006 2023-05-31T20:37:55.647 DEBUG:boto:auth_path=/pklxkd-58/obj00000000000000000006 2023-05-31T20:37:55.647 DEBUG:boto:path=/pklxkd-58/obj00000000000000000006?versionId=null 2023-05-31T20:37:55.647 DEBUG:boto:auth_path=/pklxkd-58/obj00000000000000000006?versionId=null 2023-05-31T20:37:55.648 DEBUG:boto:Method: HEAD 2023-05-31T20:37:55.648 DEBUG:boto:Path: /pklxkd-58/obj00000000000000000006?versionId=null 2023-05-31T20:37:55.648 DEBUG:boto:Data: 2023-05-31T20:37:55.649 DEBUG:boto:Headers: {} 2023-05-31T20:37:55.650 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:37:55.650 DEBUG:boto:Port: 8001 2023-05-31T20:37:55.651 DEBUG:boto:Params: {} 2023-05-31T20:37:55.651 DEBUG:boto:Token: None 2023-05-31T20:37:55.651 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:37:55 GMT /pklxkd-58/obj00000000000000000006?versionId=null 2023-05-31T20:37:55.652 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:5NVIK62TTrOj1z2MHBwDWnOUKbU= 2023-05-31T20:37:55.653 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:37:55 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:5NVIK62TTrOj1z2MHBwDWnOUKbU=', 'Content-Length': '0'} 2023-05-31T20:37:55.655 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:37:29 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d0727d17d71ed73d-006477b023-1b4f-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:37:55 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:37:55.656 DEBUG:boto:path=/pklxkd-58/obj00000000000000000006 2023-05-31T20:37:55.657 DEBUG:boto:auth_path=/pklxkd-58/obj00000000000000000006 2023-05-31T20:37:55.657 DEBUG:boto:path=/pklxkd-58/obj00000000000000000006?versionId=null 2023-05-31T20:37:55.659 DEBUG:boto:auth_path=/pklxkd-58/obj00000000000000000006?versionId=null 2023-05-31T20:37:55.659 DEBUG:boto:Method: HEAD 2023-05-31T20:37:55.660 DEBUG:boto:Path: /pklxkd-58/obj00000000000000000006?versionId=null 2023-05-31T20:37:55.661 DEBUG:boto:Data: 2023-05-31T20:37:55.662 DEBUG:boto:Headers: {} 2023-05-31T20:37:55.662 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:37:55.662 DEBUG:boto:Port: 8000 2023-05-31T20:37:55.663 DEBUG:boto:Params: {} 2023-05-31T20:37:55.663 DEBUG:boto:Token: None 2023-05-31T20:37:55.663 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:37:55 GMT /pklxkd-58/obj00000000000000000006?versionId=null 2023-05-31T20:37:55.664 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:5NVIK62TTrOj1z2MHBwDWnOUKbU= 2023-05-31T20:37:55.665 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:37:55 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:5NVIK62TTrOj1z2MHBwDWnOUKbU=', 'Content-Length': '0'} 2023-05-31T20:37:55.666 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:37:29 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000058a5e11982247f65-006477b023-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:37:55 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:37:55.667 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000006 2023-05-31T20:37:55.667 DEBUG:boto:path=/pklxkd-58/obj00000000000000000006 2023-05-31T20:37:55.668 DEBUG:boto:auth_path=/pklxkd-58/obj00000000000000000006 2023-05-31T20:37:55.668 DEBUG:boto:path=/pklxkd-58/obj00000000000000000006?versionId=null 2023-05-31T20:37:55.669 DEBUG:boto:auth_path=/pklxkd-58/obj00000000000000000006?versionId=null 2023-05-31T20:37:55.669 DEBUG:boto:Method: GET 2023-05-31T20:37:55.669 DEBUG:boto:Path: /pklxkd-58/obj00000000000000000006?versionId=null 2023-05-31T20:37:55.670 DEBUG:boto:Data: 2023-05-31T20:37:55.670 DEBUG:boto:Headers: {} 2023-05-31T20:37:55.671 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:37:55.671 DEBUG:boto:Port: 8001 2023-05-31T20:37:55.672 DEBUG:boto:Params: {} 2023-05-31T20:37:55.672 DEBUG:boto:Token: None 2023-05-31T20:37:55.673 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:37:55 GMT /pklxkd-58/obj00000000000000000006?versionId=null 2023-05-31T20:37:55.673 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:qKf6wNDC62Xm0HqkuoDcJf7jvUk= 2023-05-31T20:37:55.674 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:37:55 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:qKf6wNDC62Xm0HqkuoDcJf7jvUk=', 'Content-Length': '0'} 2023-05-31T20:37:55.675 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:37:29 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000714e320bc43857ef-006477b023-1b4f-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:37:55 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:37:55.720 DEBUG:boto:path=/pklxkd-58/obj00000000000000000006 2023-05-31T20:37:55.721 DEBUG:boto:auth_path=/pklxkd-58/obj00000000000000000006 2023-05-31T20:37:55.721 DEBUG:boto:path=/pklxkd-58/obj00000000000000000006?versionId=null 2023-05-31T20:37:55.722 DEBUG:boto:auth_path=/pklxkd-58/obj00000000000000000006?versionId=null 2023-05-31T20:37:55.722 DEBUG:boto:Method: GET 2023-05-31T20:37:55.723 DEBUG:boto:Path: /pklxkd-58/obj00000000000000000006?versionId=null 2023-05-31T20:37:55.723 DEBUG:boto:Data: 2023-05-31T20:37:55.723 DEBUG:boto:Headers: {} 2023-05-31T20:37:55.724 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:37:55.724 DEBUG:boto:Port: 8000 2023-05-31T20:37:55.724 DEBUG:boto:Params: {} 2023-05-31T20:37:55.725 DEBUG:boto:Token: None 2023-05-31T20:37:55.726 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:37:55 GMT /pklxkd-58/obj00000000000000000006?versionId=null 2023-05-31T20:37:55.726 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:qKf6wNDC62Xm0HqkuoDcJf7jvUk= 2023-05-31T20:37:55.727 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:37:55 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:qKf6wNDC62Xm0HqkuoDcJf7jvUk=', 'Content-Length': '0'} 2023-05-31T20:37:55.729 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:37:29 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c3492fcf66a217a2-006477b023-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:37:55 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:37:55.729 DEBUG:boto:path=/pklxkd-58/ 2023-05-31T20:37:55.730 DEBUG:boto:auth_path=/pklxkd-58/ 2023-05-31T20:37:55.730 DEBUG:boto:path=/pklxkd-58/?prefix=obj00000000000000000006 2023-05-31T20:37:55.731 DEBUG:boto:auth_path=/pklxkd-58/?prefix=obj00000000000000000006 2023-05-31T20:37:55.731 DEBUG:boto:Method: GET 2023-05-31T20:37:55.732 DEBUG:boto:Path: /pklxkd-58/?prefix=obj00000000000000000006 2023-05-31T20:37:55.732 DEBUG:boto:Data: 2023-05-31T20:37:55.733 DEBUG:boto:Headers: {} 2023-05-31T20:37:55.733 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:37:55.733 DEBUG:boto:Port: 8001 2023-05-31T20:37:55.734 DEBUG:boto:Params: {} 2023-05-31T20:37:55.734 DEBUG:boto:Token: None 2023-05-31T20:37:55.735 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:37:55 GMT /pklxkd-58/ 2023-05-31T20:37:55.735 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:lno1rwvBXdlscvtz78c1xk4QhQw= 2023-05-31T20:37:55.736 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:37:55 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:lno1rwvBXdlscvtz78c1xk4QhQw=', 'Content-Length': '0'} 2023-05-31T20:37:55.738 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000032482680f48fdb29-006477b023-1b4f-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:37:55 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:37:55.739 DEBUG:boto:b'pklxkd-58obj000000000000000000061000falseobj000000000000000000062023-05-31T20:37:29.012Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:37:55.740 DEBUG:boto:path=/pklxkd-58/ 2023-05-31T20:37:55.740 DEBUG:boto:auth_path=/pklxkd-58/ 2023-05-31T20:37:55.741 DEBUG:boto:path=/pklxkd-58/?prefix=obj00000000000000000006 2023-05-31T20:37:55.741 DEBUG:boto:auth_path=/pklxkd-58/?prefix=obj00000000000000000006 2023-05-31T20:37:55.742 DEBUG:boto:Method: GET 2023-05-31T20:37:55.742 DEBUG:boto:Path: /pklxkd-58/?prefix=obj00000000000000000006 2023-05-31T20:37:55.742 DEBUG:boto:Data: 2023-05-31T20:37:55.743 DEBUG:boto:Headers: {} 2023-05-31T20:37:55.743 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:37:55.744 DEBUG:boto:Port: 8000 2023-05-31T20:37:55.745 DEBUG:boto:Params: {} 2023-05-31T20:37:55.745 DEBUG:boto:Token: None 2023-05-31T20:37:55.746 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:37:55 GMT /pklxkd-58/ 2023-05-31T20:37:55.747 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:lno1rwvBXdlscvtz78c1xk4QhQw= 2023-05-31T20:37:55.748 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:37:55 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:lno1rwvBXdlscvtz78c1xk4QhQw=', 'Content-Length': '0'} 2023-05-31T20:37:55.752 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000618c2cf46d8786d4-006477b023-1630-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:37:55 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:37:55.753 DEBUG:boto:b'pklxkd-58obj000000000000000000061000falseobj000000000000000000062023-05-31T20:37:29.012Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:37:55.753 DEBUG:boto:path=/pklxkd-58/obj00000000000000000006 2023-05-31T20:37:55.754 DEBUG:boto:auth_path=/pklxkd-58/obj00000000000000000006 2023-05-31T20:37:55.754 DEBUG:boto:Method: HEAD 2023-05-31T20:37:55.755 DEBUG:boto:Path: /pklxkd-58/obj00000000000000000006 2023-05-31T20:37:55.755 DEBUG:boto:Data: 2023-05-31T20:37:55.756 DEBUG:boto:Headers: {} 2023-05-31T20:37:55.756 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:37:55.756 DEBUG:boto:Port: 8001 2023-05-31T20:37:55.757 DEBUG:boto:Params: {} 2023-05-31T20:37:55.758 DEBUG:boto:Token: None 2023-05-31T20:37:55.759 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:37:55 GMT /pklxkd-58/obj00000000000000000006 2023-05-31T20:37:55.760 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:W/Qj9+T7Dw+bVmywN6iCYDOI7CU= 2023-05-31T20:37:55.761 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:37:55 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:W/Qj9+T7Dw+bVmywN6iCYDOI7CU=', 'Content-Length': '0'} 2023-05-31T20:37:55.763 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:37:29 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d92f0bbdda9f0e2f-006477b023-1b4f-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:37:55 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:37:55.765 DEBUG:boto:path=/pklxkd-58/obj00000000000000000006 2023-05-31T20:37:55.765 DEBUG:boto:auth_path=/pklxkd-58/obj00000000000000000006 2023-05-31T20:37:55.766 DEBUG:boto:Method: HEAD 2023-05-31T20:37:55.766 DEBUG:boto:Path: /pklxkd-58/obj00000000000000000006 2023-05-31T20:37:55.767 DEBUG:boto:Data: 2023-05-31T20:37:55.768 DEBUG:boto:Headers: {} 2023-05-31T20:37:55.768 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:37:55.768 DEBUG:boto:Port: 8000 2023-05-31T20:37:55.769 DEBUG:boto:Params: {} 2023-05-31T20:37:55.769 DEBUG:boto:Token: None 2023-05-31T20:37:55.769 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:37:55 GMT /pklxkd-58/obj00000000000000000006 2023-05-31T20:37:55.770 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:W/Qj9+T7Dw+bVmywN6iCYDOI7CU= 2023-05-31T20:37:55.770 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:37:55 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:W/Qj9+T7Dw+bVmywN6iCYDOI7CU=', 'Content-Length': '0'} 2023-05-31T20:37:55.772 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:37:29 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000dbe2f691b3cb6e3d-006477b023-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:37:55 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:37:55.773 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000006 2023-05-31T20:37:55.773 DEBUG:boto:path=/pklxkd-58/obj00000000000000000006 2023-05-31T20:37:55.774 DEBUG:boto:auth_path=/pklxkd-58/obj00000000000000000006 2023-05-31T20:37:55.774 DEBUG:boto:Method: GET 2023-05-31T20:37:55.774 DEBUG:boto:Path: /pklxkd-58/obj00000000000000000006 2023-05-31T20:37:55.775 DEBUG:boto:Data: 2023-05-31T20:37:55.775 DEBUG:boto:Headers: {} 2023-05-31T20:37:55.776 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:37:55.776 DEBUG:boto:Port: 8001 2023-05-31T20:37:55.776 DEBUG:boto:Params: {} 2023-05-31T20:37:55.777 DEBUG:boto:Token: None 2023-05-31T20:37:55.777 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:37:55 GMT /pklxkd-58/obj00000000000000000006 2023-05-31T20:37:55.778 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:iEMxSwbNF94DfAvlmhoiYjXnF3Q= 2023-05-31T20:37:55.778 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:37:55 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:iEMxSwbNF94DfAvlmhoiYjXnF3Q=', 'Content-Length': '0'} 2023-05-31T20:37:55.780 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:37:29 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005d98804117b7fe09-006477b023-1b4f-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:37:55 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:37:55.780 DEBUG:boto:path=/pklxkd-58/obj00000000000000000006 2023-05-31T20:37:55.781 DEBUG:boto:auth_path=/pklxkd-58/obj00000000000000000006 2023-05-31T20:37:55.781 DEBUG:boto:Method: GET 2023-05-31T20:37:55.781 DEBUG:boto:Path: /pklxkd-58/obj00000000000000000006 2023-05-31T20:37:55.782 DEBUG:boto:Data: 2023-05-31T20:37:55.782 DEBUG:boto:Headers: {} 2023-05-31T20:37:55.783 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:37:55.783 DEBUG:boto:Port: 8000 2023-05-31T20:37:55.783 DEBUG:boto:Params: {} 2023-05-31T20:37:55.784 DEBUG:boto:Token: None 2023-05-31T20:37:55.784 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:37:55 GMT /pklxkd-58/obj00000000000000000006 2023-05-31T20:37:55.785 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:iEMxSwbNF94DfAvlmhoiYjXnF3Q= 2023-05-31T20:37:55.785 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:37:55 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:iEMxSwbNF94DfAvlmhoiYjXnF3Q=', 'Content-Length': '0'} 2023-05-31T20:37:55.787 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:37:29 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b745bef1aae1d122-006477b023-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:37:55 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:37:55.832 DEBUG:boto:path=/pklxkd-58/ 2023-05-31T20:37:55.833 DEBUG:boto:auth_path=/pklxkd-58/ 2023-05-31T20:37:55.833 DEBUG:boto:path=/pklxkd-58/?prefix=obj00000000000000000006 2023-05-31T20:37:55.834 DEBUG:boto:auth_path=/pklxkd-58/?prefix=obj00000000000000000006 2023-05-31T20:37:55.834 DEBUG:boto:Method: GET 2023-05-31T20:37:55.835 DEBUG:boto:Path: /pklxkd-58/?prefix=obj00000000000000000006 2023-05-31T20:37:55.835 DEBUG:boto:Data: 2023-05-31T20:37:55.836 DEBUG:boto:Headers: {} 2023-05-31T20:37:55.836 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:37:55.836 DEBUG:boto:Port: 8001 2023-05-31T20:37:55.837 DEBUG:boto:Params: {} 2023-05-31T20:37:55.837 DEBUG:boto:Token: None 2023-05-31T20:37:55.838 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:37:55 GMT /pklxkd-58/ 2023-05-31T20:37:55.839 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:lno1rwvBXdlscvtz78c1xk4QhQw= 2023-05-31T20:37:55.840 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:37:55 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:lno1rwvBXdlscvtz78c1xk4QhQw=', 'Content-Length': '0'} 2023-05-31T20:37:55.843 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000005e76288bc4e8c3f5-006477b023-1b4f-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:37:55 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:37:55.843 DEBUG:boto:b'pklxkd-58obj000000000000000000061000falseobj000000000000000000062023-05-31T20:37:29.012Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:37:55.844 DEBUG:boto:path=/pklxkd-58/ 2023-05-31T20:37:55.845 DEBUG:boto:auth_path=/pklxkd-58/ 2023-05-31T20:37:55.845 DEBUG:boto:path=/pklxkd-58/?prefix=obj00000000000000000006 2023-05-31T20:37:55.845 DEBUG:boto:auth_path=/pklxkd-58/?prefix=obj00000000000000000006 2023-05-31T20:37:55.846 DEBUG:boto:Method: GET 2023-05-31T20:37:55.846 DEBUG:boto:Path: /pklxkd-58/?prefix=obj00000000000000000006 2023-05-31T20:37:55.847 DEBUG:boto:Data: 2023-05-31T20:37:55.847 DEBUG:boto:Headers: {} 2023-05-31T20:37:55.847 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:37:55.848 DEBUG:boto:Port: 8000 2023-05-31T20:37:55.848 DEBUG:boto:Params: {} 2023-05-31T20:37:55.849 DEBUG:boto:Token: None 2023-05-31T20:37:55.849 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:37:55 GMT /pklxkd-58/ 2023-05-31T20:37:55.850 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:lno1rwvBXdlscvtz78c1xk4QhQw= 2023-05-31T20:37:55.850 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:37:55 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:lno1rwvBXdlscvtz78c1xk4QhQw=', 'Content-Length': '0'} 2023-05-31T20:37:55.853 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000061bb7f633264281b-006477b023-1630-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:37:55 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:37:55.853 DEBUG:boto:b'pklxkd-58obj000000000000000000061000falseobj000000000000000000062023-05-31T20:37:29.012Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:37:55.854 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000010 2023-05-31T20:37:55.855 DEBUG:boto:path=/pklxkd-58/obj00000000000000000010 2023-05-31T20:37:55.855 DEBUG:boto:auth_path=/pklxkd-58/obj00000000000000000010 2023-05-31T20:37:55.855 DEBUG:boto:path=/pklxkd-58/obj00000000000000000010?versionId=null 2023-05-31T20:37:55.856 DEBUG:boto:auth_path=/pklxkd-58/obj00000000000000000010?versionId=null 2023-05-31T20:37:55.856 DEBUG:boto:Method: GET 2023-05-31T20:37:55.857 DEBUG:boto:Path: /pklxkd-58/obj00000000000000000010?versionId=null 2023-05-31T20:37:55.857 DEBUG:boto:Data: 2023-05-31T20:37:55.857 DEBUG:boto:Headers: {} 2023-05-31T20:37:55.858 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:37:55.858 DEBUG:boto:Port: 8001 2023-05-31T20:37:55.859 DEBUG:boto:Params: {} 2023-05-31T20:37:55.859 DEBUG:boto:Token: None 2023-05-31T20:37:55.860 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:37:55 GMT /pklxkd-58/obj00000000000000000010?versionId=null 2023-05-31T20:37:55.861 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:EvuELma8bZUtmk3Or387d7Xeo6Y= 2023-05-31T20:37:55.861 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:37:55 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:EvuELma8bZUtmk3Or387d7Xeo6Y=', 'Content-Length': '0'} 2023-05-31T20:37:55.863 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:37:28 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000876a679a97c95f3-006477b023-1b4f-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:37:55 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:37:55.864 DEBUG:boto:path=/pklxkd-58/obj00000000000000000010 2023-05-31T20:37:55.864 DEBUG:boto:auth_path=/pklxkd-58/obj00000000000000000010 2023-05-31T20:37:55.865 DEBUG:boto:path=/pklxkd-58/obj00000000000000000010?versionId=null 2023-05-31T20:37:55.865 DEBUG:boto:auth_path=/pklxkd-58/obj00000000000000000010?versionId=null 2023-05-31T20:37:55.865 DEBUG:boto:Method: GET 2023-05-31T20:37:55.866 DEBUG:boto:Path: /pklxkd-58/obj00000000000000000010?versionId=null 2023-05-31T20:37:55.866 DEBUG:boto:Data: 2023-05-31T20:37:55.867 DEBUG:boto:Headers: {} 2023-05-31T20:37:55.867 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:37:55.868 DEBUG:boto:Port: 8000 2023-05-31T20:37:55.868 DEBUG:boto:Params: {} 2023-05-31T20:37:55.869 DEBUG:boto:Token: None 2023-05-31T20:37:55.870 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:37:55 GMT /pklxkd-58/obj00000000000000000010?versionId=null 2023-05-31T20:37:55.870 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:EvuELma8bZUtmk3Or387d7Xeo6Y= 2023-05-31T20:37:55.871 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:37:55 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:EvuELma8bZUtmk3Or387d7Xeo6Y=', 'Content-Length': '0'} 2023-05-31T20:37:55.873 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:37:28 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000033e1f8f6a1b8d344-006477b023-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:37:55 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:37:55.873 DEBUG:boto:path=/pklxkd-58/obj00000000000000000010 2023-05-31T20:37:55.875 DEBUG:boto:auth_path=/pklxkd-58/obj00000000000000000010 2023-05-31T20:37:55.875 DEBUG:boto:path=/pklxkd-58/obj00000000000000000010?versionId=null 2023-05-31T20:37:55.876 DEBUG:boto:auth_path=/pklxkd-58/obj00000000000000000010?versionId=null 2023-05-31T20:37:55.876 DEBUG:boto:Method: HEAD 2023-05-31T20:37:55.876 DEBUG:boto:Path: /pklxkd-58/obj00000000000000000010?versionId=null 2023-05-31T20:37:55.877 DEBUG:boto:Data: 2023-05-31T20:37:55.877 DEBUG:boto:Headers: {} 2023-05-31T20:37:55.877 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:37:55.878 DEBUG:boto:Port: 8001 2023-05-31T20:37:55.878 DEBUG:boto:Params: {} 2023-05-31T20:37:55.879 DEBUG:boto:Token: None 2023-05-31T20:37:55.879 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:37:55 GMT /pklxkd-58/obj00000000000000000010?versionId=null 2023-05-31T20:37:55.879 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:V1+dPl1/39NeCR4YeH4uu4IPBPU= 2023-05-31T20:37:55.880 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:37:55 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:V1+dPl1/39NeCR4YeH4uu4IPBPU=', 'Content-Length': '0'} 2023-05-31T20:37:55.882 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:37:28 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000fe6e2595691634d4-006477b023-1b4f-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:37:55 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:37:55.883 DEBUG:boto:path=/pklxkd-58/obj00000000000000000010 2023-05-31T20:37:55.883 DEBUG:boto:auth_path=/pklxkd-58/obj00000000000000000010 2023-05-31T20:37:55.883 DEBUG:boto:path=/pklxkd-58/obj00000000000000000010?versionId=null 2023-05-31T20:37:55.884 DEBUG:boto:auth_path=/pklxkd-58/obj00000000000000000010?versionId=null 2023-05-31T20:37:55.884 DEBUG:boto:Method: HEAD 2023-05-31T20:37:55.885 DEBUG:boto:Path: /pklxkd-58/obj00000000000000000010?versionId=null 2023-05-31T20:37:55.885 DEBUG:boto:Data: 2023-05-31T20:37:55.885 DEBUG:boto:Headers: {} 2023-05-31T20:37:55.886 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:37:55.886 DEBUG:boto:Port: 8000 2023-05-31T20:37:55.886 DEBUG:boto:Params: {} 2023-05-31T20:37:55.887 DEBUG:boto:Token: None 2023-05-31T20:37:55.887 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:37:55 GMT /pklxkd-58/obj00000000000000000010?versionId=null 2023-05-31T20:37:55.888 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:V1+dPl1/39NeCR4YeH4uu4IPBPU= 2023-05-31T20:37:55.888 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:37:55 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:V1+dPl1/39NeCR4YeH4uu4IPBPU=', 'Content-Length': '0'} 2023-05-31T20:37:55.890 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:37:28 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001705b3be12a0882f-006477b023-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:37:55 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:37:55.891 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000010 2023-05-31T20:37:55.891 DEBUG:boto:path=/pklxkd-58/obj00000000000000000010 2023-05-31T20:37:55.891 DEBUG:boto:auth_path=/pklxkd-58/obj00000000000000000010 2023-05-31T20:37:55.892 DEBUG:boto:path=/pklxkd-58/obj00000000000000000010?versionId=null 2023-05-31T20:37:55.892 DEBUG:boto:auth_path=/pklxkd-58/obj00000000000000000010?versionId=null 2023-05-31T20:37:55.893 DEBUG:boto:Method: GET 2023-05-31T20:37:55.893 DEBUG:boto:Path: /pklxkd-58/obj00000000000000000010?versionId=null 2023-05-31T20:37:55.893 DEBUG:boto:Data: 2023-05-31T20:37:55.894 DEBUG:boto:Headers: {} 2023-05-31T20:37:55.894 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:37:55.895 DEBUG:boto:Port: 8001 2023-05-31T20:37:55.895 DEBUG:boto:Params: {} 2023-05-31T20:37:55.895 DEBUG:boto:Token: None 2023-05-31T20:37:55.896 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:37:55 GMT /pklxkd-58/obj00000000000000000010?versionId=null 2023-05-31T20:37:55.896 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:EvuELma8bZUtmk3Or387d7Xeo6Y= 2023-05-31T20:37:55.897 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:37:55 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:EvuELma8bZUtmk3Or387d7Xeo6Y=', 'Content-Length': '0'} 2023-05-31T20:37:55.899 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:37:28 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a89002f36b262efa-006477b023-1b4f-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:37:55 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:37:55.900 DEBUG:boto:path=/pklxkd-58/obj00000000000000000010 2023-05-31T20:37:55.900 DEBUG:boto:auth_path=/pklxkd-58/obj00000000000000000010 2023-05-31T20:37:55.900 DEBUG:boto:path=/pklxkd-58/obj00000000000000000010?versionId=null 2023-05-31T20:37:55.901 DEBUG:boto:auth_path=/pklxkd-58/obj00000000000000000010?versionId=null 2023-05-31T20:37:55.901 DEBUG:boto:Method: GET 2023-05-31T20:37:55.902 DEBUG:boto:Path: /pklxkd-58/obj00000000000000000010?versionId=null 2023-05-31T20:37:55.902 DEBUG:boto:Data: 2023-05-31T20:37:55.903 DEBUG:boto:Headers: {} 2023-05-31T20:37:55.903 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:37:55.904 DEBUG:boto:Port: 8000 2023-05-31T20:37:55.904 DEBUG:boto:Params: {} 2023-05-31T20:37:55.905 DEBUG:boto:Token: None 2023-05-31T20:37:55.906 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:37:55 GMT /pklxkd-58/obj00000000000000000010?versionId=null 2023-05-31T20:37:55.906 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:EvuELma8bZUtmk3Or387d7Xeo6Y= 2023-05-31T20:37:55.907 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:37:55 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:EvuELma8bZUtmk3Or387d7Xeo6Y=', 'Content-Length': '0'} 2023-05-31T20:37:55.910 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:37:28 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b9ec582d2f73918d-006477b023-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:37:55 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:37:55.952 DEBUG:boto:path=/pklxkd-58/ 2023-05-31T20:37:55.953 DEBUG:boto:auth_path=/pklxkd-58/ 2023-05-31T20:37:55.953 DEBUG:boto:path=/pklxkd-58/?prefix=obj00000000000000000010 2023-05-31T20:37:55.954 DEBUG:boto:auth_path=/pklxkd-58/?prefix=obj00000000000000000010 2023-05-31T20:37:55.954 DEBUG:boto:Method: GET 2023-05-31T20:37:55.955 DEBUG:boto:Path: /pklxkd-58/?prefix=obj00000000000000000010 2023-05-31T20:37:55.955 DEBUG:boto:Data: 2023-05-31T20:37:55.955 DEBUG:boto:Headers: {} 2023-05-31T20:37:55.956 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:37:55.956 DEBUG:boto:Port: 8001 2023-05-31T20:37:55.957 DEBUG:boto:Params: {} 2023-05-31T20:37:55.958 DEBUG:boto:Token: None 2023-05-31T20:37:55.959 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:37:55 GMT /pklxkd-58/ 2023-05-31T20:37:55.959 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:lno1rwvBXdlscvtz78c1xk4QhQw= 2023-05-31T20:37:55.960 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:37:55 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:lno1rwvBXdlscvtz78c1xk4QhQw=', 'Content-Length': '0'} 2023-05-31T20:37:55.963 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d2a5dbe6437becb0-006477b023-1b4f-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:37:55 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:37:55.964 DEBUG:boto:b'pklxkd-58obj000000000000000000101000falseobj000000000000000000102023-05-31T20:37:28.709Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:37:55.965 DEBUG:boto:path=/pklxkd-58/ 2023-05-31T20:37:55.966 DEBUG:boto:auth_path=/pklxkd-58/ 2023-05-31T20:37:55.967 DEBUG:boto:path=/pklxkd-58/?prefix=obj00000000000000000010 2023-05-31T20:37:55.968 DEBUG:boto:auth_path=/pklxkd-58/?prefix=obj00000000000000000010 2023-05-31T20:37:55.969 DEBUG:boto:Method: GET 2023-05-31T20:37:55.970 DEBUG:boto:Path: /pklxkd-58/?prefix=obj00000000000000000010 2023-05-31T20:37:55.970 DEBUG:boto:Data: 2023-05-31T20:37:55.971 DEBUG:boto:Headers: {} 2023-05-31T20:37:55.972 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:37:55.973 DEBUG:boto:Port: 8000 2023-05-31T20:37:55.973 DEBUG:boto:Params: {} 2023-05-31T20:37:55.974 DEBUG:boto:Token: None 2023-05-31T20:37:55.975 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:37:55 GMT /pklxkd-58/ 2023-05-31T20:37:55.976 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:lno1rwvBXdlscvtz78c1xk4QhQw= 2023-05-31T20:37:55.977 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:37:55 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:lno1rwvBXdlscvtz78c1xk4QhQw=', 'Content-Length': '0'} 2023-05-31T20:37:55.979 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ec267f6676560b7e-006477b023-1630-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:37:55 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:37:55.980 DEBUG:boto:b'pklxkd-58obj000000000000000000101000falseobj000000000000000000102023-05-31T20:37:28.709Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:37:55.981 DEBUG:boto:path=/pklxkd-58/obj00000000000000000010 2023-05-31T20:37:55.981 DEBUG:boto:auth_path=/pklxkd-58/obj00000000000000000010 2023-05-31T20:37:55.981 DEBUG:boto:Method: HEAD 2023-05-31T20:37:55.982 DEBUG:boto:Path: /pklxkd-58/obj00000000000000000010 2023-05-31T20:37:55.982 DEBUG:boto:Data: 2023-05-31T20:37:55.982 DEBUG:boto:Headers: {} 2023-05-31T20:37:55.983 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:37:55.983 DEBUG:boto:Port: 8001 2023-05-31T20:37:55.984 DEBUG:boto:Params: {} 2023-05-31T20:37:55.985 DEBUG:boto:Token: None 2023-05-31T20:37:55.986 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:37:55 GMT /pklxkd-58/obj00000000000000000010 2023-05-31T20:37:55.987 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:6BfMkMA9jLKCFI5Yp+pkkPxgCVU= 2023-05-31T20:37:55.988 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:37:55 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:6BfMkMA9jLKCFI5Yp+pkkPxgCVU=', 'Content-Length': '0'} 2023-05-31T20:37:55.991 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:37:28 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000cad059c7be2e6ab0-006477b023-1b4f-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:37:55 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:37:55.991 DEBUG:boto:path=/pklxkd-58/obj00000000000000000010 2023-05-31T20:37:55.992 DEBUG:boto:auth_path=/pklxkd-58/obj00000000000000000010 2023-05-31T20:37:55.992 DEBUG:boto:Method: HEAD 2023-05-31T20:37:55.993 DEBUG:boto:Path: /pklxkd-58/obj00000000000000000010 2023-05-31T20:37:55.993 DEBUG:boto:Data: 2023-05-31T20:37:55.993 DEBUG:boto:Headers: {} 2023-05-31T20:37:55.994 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:37:55.995 DEBUG:boto:Port: 8000 2023-05-31T20:37:55.995 DEBUG:boto:Params: {} 2023-05-31T20:37:55.996 DEBUG:boto:Token: None 2023-05-31T20:37:55.997 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:37:55 GMT /pklxkd-58/obj00000000000000000010 2023-05-31T20:37:55.997 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:6BfMkMA9jLKCFI5Yp+pkkPxgCVU= 2023-05-31T20:37:55.998 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:37:55 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:6BfMkMA9jLKCFI5Yp+pkkPxgCVU=', 'Content-Length': '0'} 2023-05-31T20:37:56.000 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:37:28 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ca8d7e7674a09202-006477b023-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:37:55 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:37:56.001 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000010 2023-05-31T20:37:56.001 DEBUG:boto:path=/pklxkd-58/obj00000000000000000010 2023-05-31T20:37:56.002 DEBUG:boto:auth_path=/pklxkd-58/obj00000000000000000010 2023-05-31T20:37:56.002 DEBUG:boto:Method: GET 2023-05-31T20:37:56.003 DEBUG:boto:Path: /pklxkd-58/obj00000000000000000010 2023-05-31T20:37:56.003 DEBUG:boto:Data: 2023-05-31T20:37:56.003 DEBUG:boto:Headers: {} 2023-05-31T20:37:56.004 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:37:56.004 DEBUG:boto:Port: 8001 2023-05-31T20:37:56.005 DEBUG:boto:Params: {} 2023-05-31T20:37:56.005 DEBUG:boto:Token: None 2023-05-31T20:37:56.005 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:37:56 GMT /pklxkd-58/obj00000000000000000010 2023-05-31T20:37:56.006 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:rlmudinwrglcBwe656uJH+uxwv0= 2023-05-31T20:37:56.006 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:37:56 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:rlmudinwrglcBwe656uJH+uxwv0=', 'Content-Length': '0'} 2023-05-31T20:37:56.008 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:37:28 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000721c39fa6dd5af33-006477b024-1b4f-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:37:56 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:37:56.010 DEBUG:boto:path=/pklxkd-58/obj00000000000000000010 2023-05-31T20:37:56.011 DEBUG:boto:auth_path=/pklxkd-58/obj00000000000000000010 2023-05-31T20:37:56.012 DEBUG:boto:Method: GET 2023-05-31T20:37:56.012 DEBUG:boto:Path: /pklxkd-58/obj00000000000000000010 2023-05-31T20:37:56.013 DEBUG:boto:Data: 2023-05-31T20:37:56.014 DEBUG:boto:Headers: {} 2023-05-31T20:37:56.014 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:37:56.014 DEBUG:boto:Port: 8000 2023-05-31T20:37:56.015 DEBUG:boto:Params: {} 2023-05-31T20:37:56.015 DEBUG:boto:Token: None 2023-05-31T20:37:56.016 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:37:56 GMT /pklxkd-58/obj00000000000000000010 2023-05-31T20:37:56.017 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:rlmudinwrglcBwe656uJH+uxwv0= 2023-05-31T20:37:56.018 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:37:56 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:rlmudinwrglcBwe656uJH+uxwv0=', 'Content-Length': '0'} 2023-05-31T20:37:56.021 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:37:28 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000981718c7c860383-006477b024-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:37:56 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:37:56.022 DEBUG:boto:path=/pklxkd-58/ 2023-05-31T20:37:56.023 DEBUG:boto:auth_path=/pklxkd-58/ 2023-05-31T20:37:56.024 DEBUG:boto:path=/pklxkd-58/?prefix=obj00000000000000000010 2023-05-31T20:37:56.025 DEBUG:boto:auth_path=/pklxkd-58/?prefix=obj00000000000000000010 2023-05-31T20:37:56.026 DEBUG:boto:Method: GET 2023-05-31T20:37:56.027 DEBUG:boto:Path: /pklxkd-58/?prefix=obj00000000000000000010 2023-05-31T20:37:56.027 DEBUG:boto:Data: 2023-05-31T20:37:56.028 DEBUG:boto:Headers: {} 2023-05-31T20:37:56.028 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:37:56.028 DEBUG:boto:Port: 8001 2023-05-31T20:37:56.029 DEBUG:boto:Params: {} 2023-05-31T20:37:56.029 DEBUG:boto:Token: None 2023-05-31T20:37:56.030 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:37:56 GMT /pklxkd-58/ 2023-05-31T20:37:56.030 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:w9W22pmUKEAPpzk1jAPDCAnED+8= 2023-05-31T20:37:56.031 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:37:56 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:w9W22pmUKEAPpzk1jAPDCAnED+8=', 'Content-Length': '0'} 2023-05-31T20:37:56.033 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e3d860ca1617143c-006477b024-1b4f-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:37:56 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:37:56.034 DEBUG:boto:b'pklxkd-58obj000000000000000000101000falseobj000000000000000000102023-05-31T20:37:28.709Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:37:56.035 DEBUG:boto:path=/pklxkd-58/ 2023-05-31T20:37:56.036 DEBUG:boto:auth_path=/pklxkd-58/ 2023-05-31T20:37:56.037 DEBUG:boto:path=/pklxkd-58/?prefix=obj00000000000000000010 2023-05-31T20:37:56.037 DEBUG:boto:auth_path=/pklxkd-58/?prefix=obj00000000000000000010 2023-05-31T20:37:56.037 DEBUG:boto:Method: GET 2023-05-31T20:37:56.038 DEBUG:boto:Path: /pklxkd-58/?prefix=obj00000000000000000010 2023-05-31T20:37:56.038 DEBUG:boto:Data: 2023-05-31T20:37:56.038 DEBUG:boto:Headers: {} 2023-05-31T20:37:56.039 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:37:56.039 DEBUG:boto:Port: 8000 2023-05-31T20:37:56.039 DEBUG:boto:Params: {} 2023-05-31T20:37:56.040 DEBUG:boto:Token: None 2023-05-31T20:37:56.040 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:37:56 GMT /pklxkd-58/ 2023-05-31T20:37:56.041 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:w9W22pmUKEAPpzk1jAPDCAnED+8= 2023-05-31T20:37:56.041 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:37:56 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:w9W22pmUKEAPpzk1jAPDCAnED+8=', 'Content-Length': '0'} 2023-05-31T20:37:56.044 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e9bbeaabb96b59f5-006477b024-1630-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:37:56 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:37:56.088 DEBUG:boto:b'pklxkd-58obj000000000000000000101000falseobj000000000000000000102023-05-31T20:37:28.709Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:37:56.090 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000011 2023-05-31T20:37:56.091 DEBUG:boto:path=/pklxkd-58/obj00000000000000000011 2023-05-31T20:37:56.092 DEBUG:boto:auth_path=/pklxkd-58/obj00000000000000000011 2023-05-31T20:37:56.093 DEBUG:boto:path=/pklxkd-58/obj00000000000000000011?versionId=null 2023-05-31T20:37:56.094 DEBUG:boto:auth_path=/pklxkd-58/obj00000000000000000011?versionId=null 2023-05-31T20:37:56.094 DEBUG:boto:Method: GET 2023-05-31T20:37:56.095 DEBUG:boto:Path: /pklxkd-58/obj00000000000000000011?versionId=null 2023-05-31T20:37:56.096 DEBUG:boto:Data: 2023-05-31T20:37:56.096 DEBUG:boto:Headers: {} 2023-05-31T20:37:56.097 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:37:56.097 DEBUG:boto:Port: 8001 2023-05-31T20:37:56.098 DEBUG:boto:Params: {} 2023-05-31T20:37:56.098 DEBUG:boto:Token: None 2023-05-31T20:37:56.099 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:37:56 GMT /pklxkd-58/obj00000000000000000011?versionId=null 2023-05-31T20:37:56.099 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:JWewtAEEu5SzJg0e36Fk9TlFxDQ= 2023-05-31T20:37:56.100 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:37:56 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:JWewtAEEu5SzJg0e36Fk9TlFxDQ=', 'Content-Length': '0'} 2023-05-31T20:37:56.102 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:37:28 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004594a9d72fe36b1d-006477b024-1b4f-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:37:56 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:37:56.102 DEBUG:boto:path=/pklxkd-58/obj00000000000000000011 2023-05-31T20:37:56.103 DEBUG:boto:auth_path=/pklxkd-58/obj00000000000000000011 2023-05-31T20:37:56.103 DEBUG:boto:path=/pklxkd-58/obj00000000000000000011?versionId=null 2023-05-31T20:37:56.103 DEBUG:boto:auth_path=/pklxkd-58/obj00000000000000000011?versionId=null 2023-05-31T20:37:56.104 DEBUG:boto:Method: GET 2023-05-31T20:37:56.104 DEBUG:boto:Path: /pklxkd-58/obj00000000000000000011?versionId=null 2023-05-31T20:37:56.105 DEBUG:boto:Data: 2023-05-31T20:37:56.105 DEBUG:boto:Headers: {} 2023-05-31T20:37:56.105 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:37:56.106 DEBUG:boto:Port: 8000 2023-05-31T20:37:56.106 DEBUG:boto:Params: {} 2023-05-31T20:37:56.107 DEBUG:boto:Token: None 2023-05-31T20:37:56.107 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:37:56 GMT /pklxkd-58/obj00000000000000000011?versionId=null 2023-05-31T20:37:56.108 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:JWewtAEEu5SzJg0e36Fk9TlFxDQ= 2023-05-31T20:37:56.108 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:37:56 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:JWewtAEEu5SzJg0e36Fk9TlFxDQ=', 'Content-Length': '0'} 2023-05-31T20:37:56.110 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:37:28 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000392ef66c8fad7ed0-006477b024-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:37:56 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:37:56.111 DEBUG:boto:path=/pklxkd-58/obj00000000000000000011 2023-05-31T20:37:56.111 DEBUG:boto:auth_path=/pklxkd-58/obj00000000000000000011 2023-05-31T20:37:56.112 DEBUG:boto:path=/pklxkd-58/obj00000000000000000011?versionId=null 2023-05-31T20:37:56.112 DEBUG:boto:auth_path=/pklxkd-58/obj00000000000000000011?versionId=null 2023-05-31T20:37:56.112 DEBUG:boto:Method: HEAD 2023-05-31T20:37:56.113 DEBUG:boto:Path: /pklxkd-58/obj00000000000000000011?versionId=null 2023-05-31T20:37:56.113 DEBUG:boto:Data: 2023-05-31T20:37:56.113 DEBUG:boto:Headers: {} 2023-05-31T20:37:56.114 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:37:56.114 DEBUG:boto:Port: 8001 2023-05-31T20:37:56.114 DEBUG:boto:Params: {} 2023-05-31T20:37:56.115 DEBUG:boto:Token: None 2023-05-31T20:37:56.115 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:37:56 GMT /pklxkd-58/obj00000000000000000011?versionId=null 2023-05-31T20:37:56.116 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:w+PqtwsG3df0o/s6OHD5Synq8UM= 2023-05-31T20:37:56.116 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:37:56 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:w+PqtwsG3df0o/s6OHD5Synq8UM=', 'Content-Length': '0'} 2023-05-31T20:37:56.118 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:37:28 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000057914416a5d6d411-006477b024-1b4f-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:37:56 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:37:56.118 DEBUG:boto:path=/pklxkd-58/obj00000000000000000011 2023-05-31T20:37:56.119 DEBUG:boto:auth_path=/pklxkd-58/obj00000000000000000011 2023-05-31T20:37:56.120 DEBUG:boto:path=/pklxkd-58/obj00000000000000000011?versionId=null 2023-05-31T20:37:56.120 DEBUG:boto:auth_path=/pklxkd-58/obj00000000000000000011?versionId=null 2023-05-31T20:37:56.121 DEBUG:boto:Method: HEAD 2023-05-31T20:37:56.122 DEBUG:boto:Path: /pklxkd-58/obj00000000000000000011?versionId=null 2023-05-31T20:37:56.123 DEBUG:boto:Data: 2023-05-31T20:37:56.124 DEBUG:boto:Headers: {} 2023-05-31T20:37:56.125 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:37:56.126 DEBUG:boto:Port: 8000 2023-05-31T20:37:56.126 DEBUG:boto:Params: {} 2023-05-31T20:37:56.127 DEBUG:boto:Token: None 2023-05-31T20:37:56.128 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:37:56 GMT /pklxkd-58/obj00000000000000000011?versionId=null 2023-05-31T20:37:56.129 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:w+PqtwsG3df0o/s6OHD5Synq8UM= 2023-05-31T20:37:56.130 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:37:56 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:w+PqtwsG3df0o/s6OHD5Synq8UM=', 'Content-Length': '0'} 2023-05-31T20:37:56.133 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:37:28 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007618f810ea2a294b-006477b024-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:37:56 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:37:56.134 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000011 2023-05-31T20:37:56.134 DEBUG:boto:path=/pklxkd-58/obj00000000000000000011 2023-05-31T20:37:56.135 DEBUG:boto:auth_path=/pklxkd-58/obj00000000000000000011 2023-05-31T20:37:56.136 DEBUG:boto:path=/pklxkd-58/obj00000000000000000011?versionId=null 2023-05-31T20:37:56.136 DEBUG:boto:auth_path=/pklxkd-58/obj00000000000000000011?versionId=null 2023-05-31T20:37:56.137 DEBUG:boto:Method: GET 2023-05-31T20:37:56.137 DEBUG:boto:Path: /pklxkd-58/obj00000000000000000011?versionId=null 2023-05-31T20:37:56.137 DEBUG:boto:Data: 2023-05-31T20:37:56.138 DEBUG:boto:Headers: {} 2023-05-31T20:37:56.138 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:37:56.139 DEBUG:boto:Port: 8001 2023-05-31T20:37:56.139 DEBUG:boto:Params: {} 2023-05-31T20:37:56.139 DEBUG:boto:Token: None 2023-05-31T20:37:56.140 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:37:56 GMT /pklxkd-58/obj00000000000000000011?versionId=null 2023-05-31T20:37:56.140 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:JWewtAEEu5SzJg0e36Fk9TlFxDQ= 2023-05-31T20:37:56.141 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:37:56 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:JWewtAEEu5SzJg0e36Fk9TlFxDQ=', 'Content-Length': '0'} 2023-05-31T20:37:56.143 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:37:28 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000235b792d0773b592-006477b024-1b4f-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:37:56 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:37:56.144 DEBUG:boto:path=/pklxkd-58/obj00000000000000000011 2023-05-31T20:37:56.144 DEBUG:boto:auth_path=/pklxkd-58/obj00000000000000000011 2023-05-31T20:37:56.145 DEBUG:boto:path=/pklxkd-58/obj00000000000000000011?versionId=null 2023-05-31T20:37:56.145 DEBUG:boto:auth_path=/pklxkd-58/obj00000000000000000011?versionId=null 2023-05-31T20:37:56.145 DEBUG:boto:Method: GET 2023-05-31T20:37:56.146 DEBUG:boto:Path: /pklxkd-58/obj00000000000000000011?versionId=null 2023-05-31T20:37:56.146 DEBUG:boto:Data: 2023-05-31T20:37:56.147 DEBUG:boto:Headers: {} 2023-05-31T20:37:56.147 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:37:56.147 DEBUG:boto:Port: 8000 2023-05-31T20:37:56.148 DEBUG:boto:Params: {} 2023-05-31T20:37:56.149 DEBUG:boto:Token: None 2023-05-31T20:37:56.150 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:37:56 GMT /pklxkd-58/obj00000000000000000011?versionId=null 2023-05-31T20:37:56.151 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:JWewtAEEu5SzJg0e36Fk9TlFxDQ= 2023-05-31T20:37:56.152 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:37:56 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:JWewtAEEu5SzJg0e36Fk9TlFxDQ=', 'Content-Length': '0'} 2023-05-31T20:37:56.154 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:37:28 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000018ba5e6e0431f0cf-006477b024-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:37:56 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:37:56.155 DEBUG:boto:path=/pklxkd-58/ 2023-05-31T20:37:56.156 DEBUG:boto:auth_path=/pklxkd-58/ 2023-05-31T20:37:56.156 DEBUG:boto:path=/pklxkd-58/?prefix=obj00000000000000000011 2023-05-31T20:37:56.157 DEBUG:boto:auth_path=/pklxkd-58/?prefix=obj00000000000000000011 2023-05-31T20:37:56.159 DEBUG:boto:Method: GET 2023-05-31T20:37:56.159 DEBUG:boto:Path: /pklxkd-58/?prefix=obj00000000000000000011 2023-05-31T20:37:56.160 DEBUG:boto:Data: 2023-05-31T20:37:56.160 DEBUG:boto:Headers: {} 2023-05-31T20:37:56.161 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:37:56.161 DEBUG:boto:Port: 8001 2023-05-31T20:37:56.161 DEBUG:boto:Params: {} 2023-05-31T20:37:56.162 DEBUG:boto:Token: None 2023-05-31T20:37:56.162 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:37:56 GMT /pklxkd-58/ 2023-05-31T20:37:56.163 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:w9W22pmUKEAPpzk1jAPDCAnED+8= 2023-05-31T20:37:56.163 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:37:56 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:w9W22pmUKEAPpzk1jAPDCAnED+8=', 'Content-Length': '0'} 2023-05-31T20:37:56.165 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e7b832ba3d379795-006477b024-1b4f-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:37:56 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:37:56.166 DEBUG:boto:b'pklxkd-58obj000000000000000000111000falseobj000000000000000000112023-05-31T20:37:28.799Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:37:56.167 DEBUG:boto:path=/pklxkd-58/ 2023-05-31T20:37:56.167 DEBUG:boto:auth_path=/pklxkd-58/ 2023-05-31T20:37:56.167 DEBUG:boto:path=/pklxkd-58/?prefix=obj00000000000000000011 2023-05-31T20:37:56.168 DEBUG:boto:auth_path=/pklxkd-58/?prefix=obj00000000000000000011 2023-05-31T20:37:56.168 DEBUG:boto:Method: GET 2023-05-31T20:37:56.169 DEBUG:boto:Path: /pklxkd-58/?prefix=obj00000000000000000011 2023-05-31T20:37:56.169 DEBUG:boto:Data: 2023-05-31T20:37:56.170 DEBUG:boto:Headers: {} 2023-05-31T20:37:56.170 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:37:56.170 DEBUG:boto:Port: 8000 2023-05-31T20:37:56.171 DEBUG:boto:Params: {} 2023-05-31T20:37:56.171 DEBUG:boto:Token: None 2023-05-31T20:37:56.172 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:37:56 GMT /pklxkd-58/ 2023-05-31T20:37:56.172 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:w9W22pmUKEAPpzk1jAPDCAnED+8= 2023-05-31T20:37:56.173 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:37:56 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:w9W22pmUKEAPpzk1jAPDCAnED+8=', 'Content-Length': '0'} 2023-05-31T20:37:56.175 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d651bc87471d41fe-006477b024-1630-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:37:56 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:37:56.176 DEBUG:boto:b'pklxkd-58obj000000000000000000111000falseobj000000000000000000112023-05-31T20:37:28.799Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:37:56.176 DEBUG:boto:path=/pklxkd-58/obj00000000000000000011 2023-05-31T20:37:56.177 DEBUG:boto:auth_path=/pklxkd-58/obj00000000000000000011 2023-05-31T20:37:56.177 DEBUG:boto:Method: HEAD 2023-05-31T20:37:56.177 DEBUG:boto:Path: /pklxkd-58/obj00000000000000000011 2023-05-31T20:37:56.178 DEBUG:boto:Data: 2023-05-31T20:37:56.178 DEBUG:boto:Headers: {} 2023-05-31T20:37:56.178 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:37:56.179 DEBUG:boto:Port: 8001 2023-05-31T20:37:56.179 DEBUG:boto:Params: {} 2023-05-31T20:37:56.179 DEBUG:boto:Token: None 2023-05-31T20:37:56.180 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:37:56 GMT /pklxkd-58/obj00000000000000000011 2023-05-31T20:37:56.180 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:QLAP7rvJDsEDANVErJrU2wTyOXo= 2023-05-31T20:37:56.181 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:37:56 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:QLAP7rvJDsEDANVErJrU2wTyOXo=', 'Content-Length': '0'} 2023-05-31T20:37:56.182 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:37:28 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004f281bb267c5b0e3-006477b024-1b4f-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:37:56 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:37:56.183 DEBUG:boto:path=/pklxkd-58/obj00000000000000000011 2023-05-31T20:37:56.183 DEBUG:boto:auth_path=/pklxkd-58/obj00000000000000000011 2023-05-31T20:37:56.184 DEBUG:boto:Method: HEAD 2023-05-31T20:37:56.184 DEBUG:boto:Path: /pklxkd-58/obj00000000000000000011 2023-05-31T20:37:56.184 DEBUG:boto:Data: 2023-05-31T20:37:56.185 DEBUG:boto:Headers: {} 2023-05-31T20:37:56.185 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:37:56.185 DEBUG:boto:Port: 8000 2023-05-31T20:37:56.186 DEBUG:boto:Params: {} 2023-05-31T20:37:56.186 DEBUG:boto:Token: None 2023-05-31T20:37:56.187 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:37:56 GMT /pklxkd-58/obj00000000000000000011 2023-05-31T20:37:56.187 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:QLAP7rvJDsEDANVErJrU2wTyOXo= 2023-05-31T20:37:56.187 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:37:56 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:QLAP7rvJDsEDANVErJrU2wTyOXo=', 'Content-Length': '0'} 2023-05-31T20:37:56.189 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:37:28 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000064a3e7ec7710bf56-006477b024-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:37:56 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:37:56.190 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000011 2023-05-31T20:37:56.190 DEBUG:boto:path=/pklxkd-58/obj00000000000000000011 2023-05-31T20:37:56.190 DEBUG:boto:auth_path=/pklxkd-58/obj00000000000000000011 2023-05-31T20:37:56.191 DEBUG:boto:Method: GET 2023-05-31T20:37:56.191 DEBUG:boto:Path: /pklxkd-58/obj00000000000000000011 2023-05-31T20:37:56.191 DEBUG:boto:Data: 2023-05-31T20:37:56.192 DEBUG:boto:Headers: {} 2023-05-31T20:37:56.192 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:37:56.193 DEBUG:boto:Port: 8001 2023-05-31T20:37:56.193 DEBUG:boto:Params: {} 2023-05-31T20:37:56.193 DEBUG:boto:Token: None 2023-05-31T20:37:56.194 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:37:56 GMT /pklxkd-58/obj00000000000000000011 2023-05-31T20:37:56.194 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:ubFCcZlAsKggma4qnerly/Lc/Vc= 2023-05-31T20:37:56.195 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:37:56 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:ubFCcZlAsKggma4qnerly/Lc/Vc=', 'Content-Length': '0'} 2023-05-31T20:37:56.197 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:37:28 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000baa86bcaa3dad655-006477b024-1b4f-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:37:56 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:37:56.244 DEBUG:boto:path=/pklxkd-58/obj00000000000000000011 2023-05-31T20:37:56.245 DEBUG:boto:auth_path=/pklxkd-58/obj00000000000000000011 2023-05-31T20:37:56.245 DEBUG:boto:Method: GET 2023-05-31T20:37:56.246 DEBUG:boto:Path: /pklxkd-58/obj00000000000000000011 2023-05-31T20:37:56.246 DEBUG:boto:Data: 2023-05-31T20:37:56.248 DEBUG:boto:Headers: {} 2023-05-31T20:37:56.250 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:37:56.251 DEBUG:boto:Port: 8000 2023-05-31T20:37:56.251 DEBUG:boto:Params: {} 2023-05-31T20:37:56.252 DEBUG:boto:Token: None 2023-05-31T20:37:56.252 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:37:56 GMT /pklxkd-58/obj00000000000000000011 2023-05-31T20:37:56.253 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:ubFCcZlAsKggma4qnerly/Lc/Vc= 2023-05-31T20:37:56.253 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:37:56 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:ubFCcZlAsKggma4qnerly/Lc/Vc=', 'Content-Length': '0'} 2023-05-31T20:37:56.255 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:37:28 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000023806c6b7a9b0149-006477b024-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:37:56 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:37:56.256 DEBUG:boto:path=/pklxkd-58/ 2023-05-31T20:37:56.256 DEBUG:boto:auth_path=/pklxkd-58/ 2023-05-31T20:37:56.257 DEBUG:boto:path=/pklxkd-58/?prefix=obj00000000000000000011 2023-05-31T20:37:56.257 DEBUG:boto:auth_path=/pklxkd-58/?prefix=obj00000000000000000011 2023-05-31T20:37:56.258 DEBUG:boto:Method: GET 2023-05-31T20:37:56.259 DEBUG:boto:Path: /pklxkd-58/?prefix=obj00000000000000000011 2023-05-31T20:37:56.260 DEBUG:boto:Data: 2023-05-31T20:37:56.261 DEBUG:boto:Headers: {} 2023-05-31T20:37:56.262 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:37:56.263 DEBUG:boto:Port: 8001 2023-05-31T20:37:56.264 DEBUG:boto:Params: {} 2023-05-31T20:37:56.265 DEBUG:boto:Token: None 2023-05-31T20:37:56.265 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:37:56 GMT /pklxkd-58/ 2023-05-31T20:37:56.266 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:w9W22pmUKEAPpzk1jAPDCAnED+8= 2023-05-31T20:37:56.267 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:37:56 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:w9W22pmUKEAPpzk1jAPDCAnED+8=', 'Content-Length': '0'} 2023-05-31T20:37:56.270 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d7f2cb042a156acc-006477b024-1b4f-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:37:56 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:37:56.270 DEBUG:boto:b'pklxkd-58obj000000000000000000111000falseobj000000000000000000112023-05-31T20:37:28.799Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:37:56.271 DEBUG:boto:path=/pklxkd-58/ 2023-05-31T20:37:56.271 DEBUG:boto:auth_path=/pklxkd-58/ 2023-05-31T20:37:56.272 DEBUG:boto:path=/pklxkd-58/?prefix=obj00000000000000000011 2023-05-31T20:37:56.272 DEBUG:boto:auth_path=/pklxkd-58/?prefix=obj00000000000000000011 2023-05-31T20:37:56.273 DEBUG:boto:Method: GET 2023-05-31T20:37:56.273 DEBUG:boto:Path: /pklxkd-58/?prefix=obj00000000000000000011 2023-05-31T20:37:56.273 DEBUG:boto:Data: 2023-05-31T20:37:56.274 DEBUG:boto:Headers: {} 2023-05-31T20:37:56.274 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:37:56.274 DEBUG:boto:Port: 8000 2023-05-31T20:37:56.275 DEBUG:boto:Params: {} 2023-05-31T20:37:56.276 DEBUG:boto:Token: None 2023-05-31T20:37:56.277 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:37:56 GMT /pklxkd-58/ 2023-05-31T20:37:56.278 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:w9W22pmUKEAPpzk1jAPDCAnED+8= 2023-05-31T20:37:56.279 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:37:56 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:w9W22pmUKEAPpzk1jAPDCAnED+8=', 'Content-Length': '0'} 2023-05-31T20:37:56.282 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e0f3ba1f3eb562d6-006477b024-1630-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:37:56 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:37:56.283 DEBUG:boto:b'pklxkd-58obj000000000000000000111000falseobj000000000000000000112023-05-31T20:37:28.799Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:37:56.284 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000012 2023-05-31T20:37:56.285 DEBUG:boto:path=/pklxkd-58/obj00000000000000000012 2023-05-31T20:37:56.286 DEBUG:boto:auth_path=/pklxkd-58/obj00000000000000000012 2023-05-31T20:37:56.287 DEBUG:boto:path=/pklxkd-58/obj00000000000000000012?versionId=null 2023-05-31T20:37:56.287 DEBUG:boto:auth_path=/pklxkd-58/obj00000000000000000012?versionId=null 2023-05-31T20:37:56.288 DEBUG:boto:Method: GET 2023-05-31T20:37:56.289 DEBUG:boto:Path: /pklxkd-58/obj00000000000000000012?versionId=null 2023-05-31T20:37:56.290 DEBUG:boto:Data: 2023-05-31T20:37:56.291 DEBUG:boto:Headers: {} 2023-05-31T20:37:56.291 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:37:56.292 DEBUG:boto:Port: 8001 2023-05-31T20:37:56.293 DEBUG:boto:Params: {} 2023-05-31T20:37:56.294 DEBUG:boto:Token: None 2023-05-31T20:37:56.295 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:37:56 GMT /pklxkd-58/obj00000000000000000012?versionId=null 2023-05-31T20:37:56.295 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:QPR6/RLjvNQr31GDLVIHAbuv2/Q= 2023-05-31T20:37:56.296 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:37:56 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:QPR6/RLjvNQr31GDLVIHAbuv2/Q=', 'Content-Length': '0'} 2023-05-31T20:37:56.299 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:37:28 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009ce3a2ee562f90e9-006477b024-1b4f-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:37:56 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:37:56.300 DEBUG:boto:path=/pklxkd-58/obj00000000000000000012 2023-05-31T20:37:56.301 DEBUG:boto:auth_path=/pklxkd-58/obj00000000000000000012 2023-05-31T20:37:56.302 DEBUG:boto:path=/pklxkd-58/obj00000000000000000012?versionId=null 2023-05-31T20:37:56.302 DEBUG:boto:auth_path=/pklxkd-58/obj00000000000000000012?versionId=null 2023-05-31T20:37:56.303 DEBUG:boto:Method: GET 2023-05-31T20:37:56.304 DEBUG:boto:Path: /pklxkd-58/obj00000000000000000012?versionId=null 2023-05-31T20:37:56.305 DEBUG:boto:Data: 2023-05-31T20:37:56.306 DEBUG:boto:Headers: {} 2023-05-31T20:37:56.307 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:37:56.308 DEBUG:boto:Port: 8000 2023-05-31T20:37:56.309 DEBUG:boto:Params: {} 2023-05-31T20:37:56.310 DEBUG:boto:Token: None 2023-05-31T20:37:56.310 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:37:56 GMT /pklxkd-58/obj00000000000000000012?versionId=null 2023-05-31T20:37:56.311 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:QPR6/RLjvNQr31GDLVIHAbuv2/Q= 2023-05-31T20:37:56.312 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:37:56 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:QPR6/RLjvNQr31GDLVIHAbuv2/Q=', 'Content-Length': '0'} 2023-05-31T20:37:56.314 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:37:28 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a86ec952610e234e-006477b024-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:37:56 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:37:56.356 DEBUG:boto:path=/pklxkd-58/obj00000000000000000012 2023-05-31T20:37:56.357 DEBUG:boto:auth_path=/pklxkd-58/obj00000000000000000012 2023-05-31T20:37:56.357 DEBUG:boto:path=/pklxkd-58/obj00000000000000000012?versionId=null 2023-05-31T20:37:56.358 DEBUG:boto:auth_path=/pklxkd-58/obj00000000000000000012?versionId=null 2023-05-31T20:37:56.359 DEBUG:boto:Method: HEAD 2023-05-31T20:37:56.359 DEBUG:boto:Path: /pklxkd-58/obj00000000000000000012?versionId=null 2023-05-31T20:37:56.360 DEBUG:boto:Data: 2023-05-31T20:37:56.360 DEBUG:boto:Headers: {} 2023-05-31T20:37:56.360 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:37:56.361 DEBUG:boto:Port: 8001 2023-05-31T20:37:56.361 DEBUG:boto:Params: {} 2023-05-31T20:37:56.362 DEBUG:boto:Token: None 2023-05-31T20:37:56.363 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:37:56 GMT /pklxkd-58/obj00000000000000000012?versionId=null 2023-05-31T20:37:56.363 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:DXxeuFT9qKvN3nv0uFlcdLxglqo= 2023-05-31T20:37:56.364 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:37:56 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:DXxeuFT9qKvN3nv0uFlcdLxglqo=', 'Content-Length': '0'} 2023-05-31T20:37:56.366 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:37:28 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000077de6c2e77a8d423-006477b024-1b4f-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:37:56 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:37:56.366 DEBUG:boto:path=/pklxkd-58/obj00000000000000000012 2023-05-31T20:37:56.367 DEBUG:boto:auth_path=/pklxkd-58/obj00000000000000000012 2023-05-31T20:37:56.367 DEBUG:boto:path=/pklxkd-58/obj00000000000000000012?versionId=null 2023-05-31T20:37:56.367 DEBUG:boto:auth_path=/pklxkd-58/obj00000000000000000012?versionId=null 2023-05-31T20:37:56.368 DEBUG:boto:Method: HEAD 2023-05-31T20:37:56.368 DEBUG:boto:Path: /pklxkd-58/obj00000000000000000012?versionId=null 2023-05-31T20:37:56.369 DEBUG:boto:Data: 2023-05-31T20:37:56.369 DEBUG:boto:Headers: {} 2023-05-31T20:37:56.370 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:37:56.370 DEBUG:boto:Port: 8000 2023-05-31T20:37:56.370 DEBUG:boto:Params: {} 2023-05-31T20:37:56.371 DEBUG:boto:Token: None 2023-05-31T20:37:56.371 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:37:56 GMT /pklxkd-58/obj00000000000000000012?versionId=null 2023-05-31T20:37:56.372 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:DXxeuFT9qKvN3nv0uFlcdLxglqo= 2023-05-31T20:37:56.372 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:37:56 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:DXxeuFT9qKvN3nv0uFlcdLxglqo=', 'Content-Length': '0'} 2023-05-31T20:37:56.374 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:37:28 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000041c25f3078e65deb-006477b024-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:37:56 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:37:56.374 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000012 2023-05-31T20:37:56.375 DEBUG:boto:path=/pklxkd-58/obj00000000000000000012 2023-05-31T20:37:56.375 DEBUG:boto:auth_path=/pklxkd-58/obj00000000000000000012 2023-05-31T20:37:56.376 DEBUG:boto:path=/pklxkd-58/obj00000000000000000012?versionId=null 2023-05-31T20:37:56.376 DEBUG:boto:auth_path=/pklxkd-58/obj00000000000000000012?versionId=null 2023-05-31T20:37:56.376 DEBUG:boto:Method: GET 2023-05-31T20:37:56.377 DEBUG:boto:Path: /pklxkd-58/obj00000000000000000012?versionId=null 2023-05-31T20:37:56.377 DEBUG:boto:Data: 2023-05-31T20:37:56.377 DEBUG:boto:Headers: {} 2023-05-31T20:37:56.378 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:37:56.378 DEBUG:boto:Port: 8001 2023-05-31T20:37:56.379 DEBUG:boto:Params: {} 2023-05-31T20:37:56.379 DEBUG:boto:Token: None 2023-05-31T20:37:56.380 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:37:56 GMT /pklxkd-58/obj00000000000000000012?versionId=null 2023-05-31T20:37:56.380 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:QPR6/RLjvNQr31GDLVIHAbuv2/Q= 2023-05-31T20:37:56.381 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:37:56 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:QPR6/RLjvNQr31GDLVIHAbuv2/Q=', 'Content-Length': '0'} 2023-05-31T20:37:56.383 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:37:28 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b266cb24abad2a3b-006477b024-1b4f-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:37:56 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:37:56.383 DEBUG:boto:path=/pklxkd-58/obj00000000000000000012 2023-05-31T20:37:56.384 DEBUG:boto:auth_path=/pklxkd-58/obj00000000000000000012 2023-05-31T20:37:56.384 DEBUG:boto:path=/pklxkd-58/obj00000000000000000012?versionId=null 2023-05-31T20:37:56.385 DEBUG:boto:auth_path=/pklxkd-58/obj00000000000000000012?versionId=null 2023-05-31T20:37:56.385 DEBUG:boto:Method: GET 2023-05-31T20:37:56.385 DEBUG:boto:Path: /pklxkd-58/obj00000000000000000012?versionId=null 2023-05-31T20:37:56.386 DEBUG:boto:Data: 2023-05-31T20:37:56.386 DEBUG:boto:Headers: {} 2023-05-31T20:37:56.386 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:37:56.387 DEBUG:boto:Port: 8000 2023-05-31T20:37:56.387 DEBUG:boto:Params: {} 2023-05-31T20:37:56.388 DEBUG:boto:Token: None 2023-05-31T20:37:56.388 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:37:56 GMT /pklxkd-58/obj00000000000000000012?versionId=null 2023-05-31T20:37:56.388 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:QPR6/RLjvNQr31GDLVIHAbuv2/Q= 2023-05-31T20:37:56.389 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:37:56 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:QPR6/RLjvNQr31GDLVIHAbuv2/Q=', 'Content-Length': '0'} 2023-05-31T20:37:56.391 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:37:28 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f6b23d5304eb022c-006477b024-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:37:56 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:37:56.392 DEBUG:boto:path=/pklxkd-58/ 2023-05-31T20:37:56.393 DEBUG:boto:auth_path=/pklxkd-58/ 2023-05-31T20:37:56.393 DEBUG:boto:path=/pklxkd-58/?prefix=obj00000000000000000012 2023-05-31T20:37:56.393 DEBUG:boto:auth_path=/pklxkd-58/?prefix=obj00000000000000000012 2023-05-31T20:37:56.394 DEBUG:boto:Method: GET 2023-05-31T20:37:56.394 DEBUG:boto:Path: /pklxkd-58/?prefix=obj00000000000000000012 2023-05-31T20:37:56.394 DEBUG:boto:Data: 2023-05-31T20:37:56.395 DEBUG:boto:Headers: {} 2023-05-31T20:37:56.395 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:37:56.395 DEBUG:boto:Port: 8001 2023-05-31T20:37:56.396 DEBUG:boto:Params: {} 2023-05-31T20:37:56.396 DEBUG:boto:Token: None 2023-05-31T20:37:56.397 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:37:56 GMT /pklxkd-58/ 2023-05-31T20:37:56.397 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:w9W22pmUKEAPpzk1jAPDCAnED+8= 2023-05-31T20:37:56.398 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:37:56 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:w9W22pmUKEAPpzk1jAPDCAnED+8=', 'Content-Length': '0'} 2023-05-31T20:37:56.400 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000003fb71f549ff85239-006477b024-1b4f-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:37:56 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:37:56.400 DEBUG:boto:b'pklxkd-58obj000000000000000000121000falseobj000000000000000000122023-05-31T20:37:28.988Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:37:56.401 DEBUG:boto:path=/pklxkd-58/ 2023-05-31T20:37:56.402 DEBUG:boto:auth_path=/pklxkd-58/ 2023-05-31T20:37:56.402 DEBUG:boto:path=/pklxkd-58/?prefix=obj00000000000000000012 2023-05-31T20:37:56.402 DEBUG:boto:auth_path=/pklxkd-58/?prefix=obj00000000000000000012 2023-05-31T20:37:56.403 DEBUG:boto:Method: GET 2023-05-31T20:37:56.403 DEBUG:boto:Path: /pklxkd-58/?prefix=obj00000000000000000012 2023-05-31T20:37:56.403 DEBUG:boto:Data: 2023-05-31T20:37:56.404 DEBUG:boto:Headers: {} 2023-05-31T20:37:56.404 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:37:56.405 DEBUG:boto:Port: 8000 2023-05-31T20:37:56.405 DEBUG:boto:Params: {} 2023-05-31T20:37:56.406 DEBUG:boto:Token: None 2023-05-31T20:37:56.406 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:37:56 GMT /pklxkd-58/ 2023-05-31T20:37:56.407 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:w9W22pmUKEAPpzk1jAPDCAnED+8= 2023-05-31T20:37:56.407 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:37:56 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:w9W22pmUKEAPpzk1jAPDCAnED+8=', 'Content-Length': '0'} 2023-05-31T20:37:56.409 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a120959abffc009f-006477b024-1630-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:37:56 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:37:56.410 DEBUG:boto:b'pklxkd-58obj000000000000000000121000falseobj000000000000000000122023-05-31T20:37:28.988Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:37:56.411 DEBUG:boto:path=/pklxkd-58/obj00000000000000000012 2023-05-31T20:37:56.412 DEBUG:boto:auth_path=/pklxkd-58/obj00000000000000000012 2023-05-31T20:37:56.412 DEBUG:boto:Method: HEAD 2023-05-31T20:37:56.412 DEBUG:boto:Path: /pklxkd-58/obj00000000000000000012 2023-05-31T20:37:56.413 DEBUG:boto:Data: 2023-05-31T20:37:56.413 DEBUG:boto:Headers: {} 2023-05-31T20:37:56.413 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:37:56.414 DEBUG:boto:Port: 8001 2023-05-31T20:37:56.414 DEBUG:boto:Params: {} 2023-05-31T20:37:56.415 DEBUG:boto:Token: None 2023-05-31T20:37:56.415 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:37:56 GMT /pklxkd-58/obj00000000000000000012 2023-05-31T20:37:56.416 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:F33QJvmE6KzoXpxuwFyi1Z2JSVI= 2023-05-31T20:37:56.416 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:37:56 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:F33QJvmE6KzoXpxuwFyi1Z2JSVI=', 'Content-Length': '0'} 2023-05-31T20:37:56.418 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:37:28 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004ca833c13b1c0e03-006477b024-1b4f-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:37:56 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:37:56.418 DEBUG:boto:path=/pklxkd-58/obj00000000000000000012 2023-05-31T20:37:56.419 DEBUG:boto:auth_path=/pklxkd-58/obj00000000000000000012 2023-05-31T20:37:56.419 DEBUG:boto:Method: HEAD 2023-05-31T20:37:56.420 DEBUG:boto:Path: /pklxkd-58/obj00000000000000000012 2023-05-31T20:37:56.420 DEBUG:boto:Data: 2023-05-31T20:37:56.421 DEBUG:boto:Headers: {} 2023-05-31T20:37:56.421 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:37:56.421 DEBUG:boto:Port: 8000 2023-05-31T20:37:56.422 DEBUG:boto:Params: {} 2023-05-31T20:37:56.422 DEBUG:boto:Token: None 2023-05-31T20:37:56.423 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:37:56 GMT /pklxkd-58/obj00000000000000000012 2023-05-31T20:37:56.423 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:F33QJvmE6KzoXpxuwFyi1Z2JSVI= 2023-05-31T20:37:56.423 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:37:56 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:F33QJvmE6KzoXpxuwFyi1Z2JSVI=', 'Content-Length': '0'} 2023-05-31T20:37:56.425 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:37:28 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d721027d13d41c49-006477b024-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:37:56 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:37:56.426 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000012 2023-05-31T20:37:56.426 DEBUG:boto:path=/pklxkd-58/obj00000000000000000012 2023-05-31T20:37:56.426 DEBUG:boto:auth_path=/pklxkd-58/obj00000000000000000012 2023-05-31T20:37:56.427 DEBUG:boto:Method: GET 2023-05-31T20:37:56.428 DEBUG:boto:Path: /pklxkd-58/obj00000000000000000012 2023-05-31T20:37:56.428 DEBUG:boto:Data: 2023-05-31T20:37:56.428 DEBUG:boto:Headers: {} 2023-05-31T20:37:56.429 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:37:56.429 DEBUG:boto:Port: 8001 2023-05-31T20:37:56.430 DEBUG:boto:Params: {} 2023-05-31T20:37:56.430 DEBUG:boto:Token: None 2023-05-31T20:37:56.430 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:37:56 GMT /pklxkd-58/obj00000000000000000012 2023-05-31T20:37:56.431 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:0Ejq7xvsGG7LpJFlIBpCENaSJ80= 2023-05-31T20:37:56.431 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:37:56 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:0Ejq7xvsGG7LpJFlIBpCENaSJ80=', 'Content-Length': '0'} 2023-05-31T20:37:56.434 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:37:28 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000289363b8a6a2e9bf-006477b024-1b4f-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:37:56 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:37:56.434 DEBUG:boto:path=/pklxkd-58/obj00000000000000000012 2023-05-31T20:37:56.435 DEBUG:boto:auth_path=/pklxkd-58/obj00000000000000000012 2023-05-31T20:37:56.435 DEBUG:boto:Method: GET 2023-05-31T20:37:56.435 DEBUG:boto:Path: /pklxkd-58/obj00000000000000000012 2023-05-31T20:37:56.436 DEBUG:boto:Data: 2023-05-31T20:37:56.436 DEBUG:boto:Headers: {} 2023-05-31T20:37:56.437 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:37:56.437 DEBUG:boto:Port: 8000 2023-05-31T20:37:56.437 DEBUG:boto:Params: {} 2023-05-31T20:37:56.438 DEBUG:boto:Token: None 2023-05-31T20:37:56.438 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:37:56 GMT /pklxkd-58/obj00000000000000000012 2023-05-31T20:37:56.439 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:0Ejq7xvsGG7LpJFlIBpCENaSJ80= 2023-05-31T20:37:56.439 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:37:56 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:0Ejq7xvsGG7LpJFlIBpCENaSJ80=', 'Content-Length': '0'} 2023-05-31T20:37:56.442 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:37:28 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005c9fa37176c9b8e9-006477b024-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:37:56 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:37:56.442 DEBUG:boto:path=/pklxkd-58/ 2023-05-31T20:37:56.443 DEBUG:boto:auth_path=/pklxkd-58/ 2023-05-31T20:37:56.443 DEBUG:boto:path=/pklxkd-58/?prefix=obj00000000000000000012 2023-05-31T20:37:56.443 DEBUG:boto:auth_path=/pklxkd-58/?prefix=obj00000000000000000012 2023-05-31T20:37:56.444 DEBUG:boto:Method: GET 2023-05-31T20:37:56.444 DEBUG:boto:Path: /pklxkd-58/?prefix=obj00000000000000000012 2023-05-31T20:37:56.445 DEBUG:boto:Data: 2023-05-31T20:37:56.445 DEBUG:boto:Headers: {} 2023-05-31T20:37:56.445 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:37:56.446 DEBUG:boto:Port: 8001 2023-05-31T20:37:56.446 DEBUG:boto:Params: {} 2023-05-31T20:37:56.446 DEBUG:boto:Token: None 2023-05-31T20:37:56.447 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:37:56 GMT /pklxkd-58/ 2023-05-31T20:37:56.447 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:w9W22pmUKEAPpzk1jAPDCAnED+8= 2023-05-31T20:37:56.448 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:37:56 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:w9W22pmUKEAPpzk1jAPDCAnED+8=', 'Content-Length': '0'} 2023-05-31T20:37:56.450 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000078ec7965f67c0dc3-006477b024-1b4f-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:37:56 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:37:56.451 DEBUG:boto:b'pklxkd-58obj000000000000000000121000falseobj000000000000000000122023-05-31T20:37:28.988Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:37:56.451 DEBUG:boto:path=/pklxkd-58/ 2023-05-31T20:37:56.452 DEBUG:boto:auth_path=/pklxkd-58/ 2023-05-31T20:37:56.452 DEBUG:boto:path=/pklxkd-58/?prefix=obj00000000000000000012 2023-05-31T20:37:56.453 DEBUG:boto:auth_path=/pklxkd-58/?prefix=obj00000000000000000012 2023-05-31T20:37:56.453 DEBUG:boto:Method: GET 2023-05-31T20:37:56.453 DEBUG:boto:Path: /pklxkd-58/?prefix=obj00000000000000000012 2023-05-31T20:37:56.454 DEBUG:boto:Data: 2023-05-31T20:37:56.454 DEBUG:boto:Headers: {} 2023-05-31T20:37:56.454 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:37:56.455 DEBUG:boto:Port: 8000 2023-05-31T20:37:56.455 DEBUG:boto:Params: {} 2023-05-31T20:37:56.455 DEBUG:boto:Token: None 2023-05-31T20:37:56.456 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:37:56 GMT /pklxkd-58/ 2023-05-31T20:37:56.456 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:w9W22pmUKEAPpzk1jAPDCAnED+8= 2023-05-31T20:37:56.457 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:37:56 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:w9W22pmUKEAPpzk1jAPDCAnED+8=', 'Content-Length': '0'} 2023-05-31T20:37:56.459 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e2de834af184cde0-006477b024-1630-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:37:56 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:37:56.500 DEBUG:boto:b'pklxkd-58obj000000000000000000121000falseobj000000000000000000122023-05-31T20:37:28.988Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:37:56.501 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000013 2023-05-31T20:37:56.502 DEBUG:boto:path=/pklxkd-58/obj00000000000000000013 2023-05-31T20:37:56.502 DEBUG:boto:auth_path=/pklxkd-58/obj00000000000000000013 2023-05-31T20:37:56.502 DEBUG:boto:path=/pklxkd-58/obj00000000000000000013?versionId=null 2023-05-31T20:37:56.503 DEBUG:boto:auth_path=/pklxkd-58/obj00000000000000000013?versionId=null 2023-05-31T20:37:56.503 DEBUG:boto:Method: GET 2023-05-31T20:37:56.504 DEBUG:boto:Path: /pklxkd-58/obj00000000000000000013?versionId=null 2023-05-31T20:37:56.505 DEBUG:boto:Data: 2023-05-31T20:37:56.505 DEBUG:boto:Headers: {} 2023-05-31T20:37:56.505 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:37:56.506 DEBUG:boto:Port: 8001 2023-05-31T20:37:56.507 DEBUG:boto:Params: {} 2023-05-31T20:37:56.508 DEBUG:boto:Token: None 2023-05-31T20:37:56.509 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:37:56 GMT /pklxkd-58/obj00000000000000000013?versionId=null 2023-05-31T20:37:56.510 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:tipM9EZEIaf3te9TjgaCD7j633A= 2023-05-31T20:37:56.511 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:37:56 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:tipM9EZEIaf3te9TjgaCD7j633A=', 'Content-Length': '0'} 2023-05-31T20:37:56.515 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:37:29 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e2fbdacd7dc822dd-006477b024-1b4f-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:37:56 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:37:56.515 DEBUG:boto:path=/pklxkd-58/obj00000000000000000013 2023-05-31T20:37:56.516 DEBUG:boto:auth_path=/pklxkd-58/obj00000000000000000013 2023-05-31T20:37:56.516 DEBUG:boto:path=/pklxkd-58/obj00000000000000000013?versionId=null 2023-05-31T20:37:56.516 DEBUG:boto:auth_path=/pklxkd-58/obj00000000000000000013?versionId=null 2023-05-31T20:37:56.517 DEBUG:boto:Method: GET 2023-05-31T20:37:56.517 DEBUG:boto:Path: /pklxkd-58/obj00000000000000000013?versionId=null 2023-05-31T20:37:56.518 DEBUG:boto:Data: 2023-05-31T20:37:56.518 DEBUG:boto:Headers: {} 2023-05-31T20:37:56.519 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:37:56.519 DEBUG:boto:Port: 8000 2023-05-31T20:37:56.519 DEBUG:boto:Params: {} 2023-05-31T20:37:56.520 DEBUG:boto:Token: None 2023-05-31T20:37:56.520 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:37:56 GMT /pklxkd-58/obj00000000000000000013?versionId=null 2023-05-31T20:37:56.521 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:tipM9EZEIaf3te9TjgaCD7j633A= 2023-05-31T20:37:56.521 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:37:56 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:tipM9EZEIaf3te9TjgaCD7j633A=', 'Content-Length': '0'} 2023-05-31T20:37:56.524 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:37:29 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000d953a1fc9e17cd5-006477b024-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:37:56 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:37:56.525 DEBUG:boto:path=/pklxkd-58/obj00000000000000000013 2023-05-31T20:37:56.525 DEBUG:boto:auth_path=/pklxkd-58/obj00000000000000000013 2023-05-31T20:37:56.526 DEBUG:boto:path=/pklxkd-58/obj00000000000000000013?versionId=null 2023-05-31T20:37:56.526 DEBUG:boto:auth_path=/pklxkd-58/obj00000000000000000013?versionId=null 2023-05-31T20:37:56.526 DEBUG:boto:Method: HEAD 2023-05-31T20:37:56.527 DEBUG:boto:Path: /pklxkd-58/obj00000000000000000013?versionId=null 2023-05-31T20:37:56.527 DEBUG:boto:Data: 2023-05-31T20:37:56.528 DEBUG:boto:Headers: {} 2023-05-31T20:37:56.528 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:37:56.529 DEBUG:boto:Port: 8001 2023-05-31T20:37:56.529 DEBUG:boto:Params: {} 2023-05-31T20:37:56.529 DEBUG:boto:Token: None 2023-05-31T20:37:56.530 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:37:56 GMT /pklxkd-58/obj00000000000000000013?versionId=null 2023-05-31T20:37:56.530 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:hQBN6wY3bQSwhvv9a8dIjSu3RY0= 2023-05-31T20:37:56.531 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:37:56 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:hQBN6wY3bQSwhvv9a8dIjSu3RY0=', 'Content-Length': '0'} 2023-05-31T20:37:56.533 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:37:29 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001a718ad93edfcc25-006477b024-1b4f-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:37:56 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:37:56.533 DEBUG:boto:path=/pklxkd-58/obj00000000000000000013 2023-05-31T20:37:56.534 DEBUG:boto:auth_path=/pklxkd-58/obj00000000000000000013 2023-05-31T20:37:56.534 DEBUG:boto:path=/pklxkd-58/obj00000000000000000013?versionId=null 2023-05-31T20:37:56.535 DEBUG:boto:auth_path=/pklxkd-58/obj00000000000000000013?versionId=null 2023-05-31T20:37:56.536 DEBUG:boto:Method: HEAD 2023-05-31T20:37:56.536 DEBUG:boto:Path: /pklxkd-58/obj00000000000000000013?versionId=null 2023-05-31T20:37:56.536 DEBUG:boto:Data: 2023-05-31T20:37:56.537 DEBUG:boto:Headers: {} 2023-05-31T20:37:56.537 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:37:56.538 DEBUG:boto:Port: 8000 2023-05-31T20:37:56.538 DEBUG:boto:Params: {} 2023-05-31T20:37:56.539 DEBUG:boto:Token: None 2023-05-31T20:37:56.539 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:37:56 GMT /pklxkd-58/obj00000000000000000013?versionId=null 2023-05-31T20:37:56.540 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:hQBN6wY3bQSwhvv9a8dIjSu3RY0= 2023-05-31T20:37:56.541 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:37:56 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:hQBN6wY3bQSwhvv9a8dIjSu3RY0=', 'Content-Length': '0'} 2023-05-31T20:37:56.543 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:37:29 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000019a870f91c80b097-006477b024-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:37:56 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:37:56.543 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000013 2023-05-31T20:37:56.544 DEBUG:boto:path=/pklxkd-58/obj00000000000000000013 2023-05-31T20:37:56.544 DEBUG:boto:auth_path=/pklxkd-58/obj00000000000000000013 2023-05-31T20:37:56.545 DEBUG:boto:path=/pklxkd-58/obj00000000000000000013?versionId=null 2023-05-31T20:37:56.545 DEBUG:boto:auth_path=/pklxkd-58/obj00000000000000000013?versionId=null 2023-05-31T20:37:56.546 DEBUG:boto:Method: GET 2023-05-31T20:37:56.546 DEBUG:boto:Path: /pklxkd-58/obj00000000000000000013?versionId=null 2023-05-31T20:37:56.546 DEBUG:boto:Data: 2023-05-31T20:37:56.547 DEBUG:boto:Headers: {} 2023-05-31T20:37:56.547 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:37:56.548 DEBUG:boto:Port: 8001 2023-05-31T20:37:56.548 DEBUG:boto:Params: {} 2023-05-31T20:37:56.549 DEBUG:boto:Token: None 2023-05-31T20:37:56.550 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:37:56 GMT /pklxkd-58/obj00000000000000000013?versionId=null 2023-05-31T20:37:56.550 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:tipM9EZEIaf3te9TjgaCD7j633A= 2023-05-31T20:37:56.551 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:37:56 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:tipM9EZEIaf3te9TjgaCD7j633A=', 'Content-Length': '0'} 2023-05-31T20:37:56.553 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:37:29 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000979e2fe8c6407b6a-006477b024-1b4f-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:37:56 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:37:56.553 DEBUG:boto:path=/pklxkd-58/obj00000000000000000013 2023-05-31T20:37:56.554 DEBUG:boto:auth_path=/pklxkd-58/obj00000000000000000013 2023-05-31T20:37:56.554 DEBUG:boto:path=/pklxkd-58/obj00000000000000000013?versionId=null 2023-05-31T20:37:56.554 DEBUG:boto:auth_path=/pklxkd-58/obj00000000000000000013?versionId=null 2023-05-31T20:37:56.555 DEBUG:boto:Method: GET 2023-05-31T20:37:56.555 DEBUG:boto:Path: /pklxkd-58/obj00000000000000000013?versionId=null 2023-05-31T20:37:56.556 DEBUG:boto:Data: 2023-05-31T20:37:56.556 DEBUG:boto:Headers: {} 2023-05-31T20:37:56.556 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:37:56.557 DEBUG:boto:Port: 8000 2023-05-31T20:37:56.557 DEBUG:boto:Params: {} 2023-05-31T20:37:56.558 DEBUG:boto:Token: None 2023-05-31T20:37:56.558 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:37:56 GMT /pklxkd-58/obj00000000000000000013?versionId=null 2023-05-31T20:37:56.558 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:tipM9EZEIaf3te9TjgaCD7j633A= 2023-05-31T20:37:56.559 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:37:56 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:tipM9EZEIaf3te9TjgaCD7j633A=', 'Content-Length': '0'} 2023-05-31T20:37:56.561 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:37:29 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f7e846e4856f3bf9-006477b024-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:37:56 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:37:56.562 DEBUG:boto:path=/pklxkd-58/ 2023-05-31T20:37:56.562 DEBUG:boto:auth_path=/pklxkd-58/ 2023-05-31T20:37:56.562 DEBUG:boto:path=/pklxkd-58/?prefix=obj00000000000000000013 2023-05-31T20:37:56.563 DEBUG:boto:auth_path=/pklxkd-58/?prefix=obj00000000000000000013 2023-05-31T20:37:56.563 DEBUG:boto:Method: GET 2023-05-31T20:37:56.564 DEBUG:boto:Path: /pklxkd-58/?prefix=obj00000000000000000013 2023-05-31T20:37:56.564 DEBUG:boto:Data: 2023-05-31T20:37:56.564 DEBUG:boto:Headers: {} 2023-05-31T20:37:56.565 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:37:56.565 DEBUG:boto:Port: 8001 2023-05-31T20:37:56.565 DEBUG:boto:Params: {} 2023-05-31T20:37:56.566 DEBUG:boto:Token: None 2023-05-31T20:37:56.566 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:37:56 GMT /pklxkd-58/ 2023-05-31T20:37:56.567 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:w9W22pmUKEAPpzk1jAPDCAnED+8= 2023-05-31T20:37:56.567 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:37:56 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:w9W22pmUKEAPpzk1jAPDCAnED+8=', 'Content-Length': '0'} 2023-05-31T20:37:56.569 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000000343338a54ccc8e2-006477b024-1b4f-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:37:56 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:37:56.612 DEBUG:boto:b'pklxkd-58obj000000000000000000131000falseobj000000000000000000132023-05-31T20:37:29.076Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:37:56.613 DEBUG:boto:path=/pklxkd-58/ 2023-05-31T20:37:56.614 DEBUG:boto:auth_path=/pklxkd-58/ 2023-05-31T20:37:56.614 DEBUG:boto:path=/pklxkd-58/?prefix=obj00000000000000000013 2023-05-31T20:37:56.615 DEBUG:boto:auth_path=/pklxkd-58/?prefix=obj00000000000000000013 2023-05-31T20:37:56.615 DEBUG:boto:Method: GET 2023-05-31T20:37:56.615 DEBUG:boto:Path: /pklxkd-58/?prefix=obj00000000000000000013 2023-05-31T20:37:56.616 DEBUG:boto:Data: 2023-05-31T20:37:56.616 DEBUG:boto:Headers: {} 2023-05-31T20:37:56.616 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:37:56.617 DEBUG:boto:Port: 8000 2023-05-31T20:37:56.617 DEBUG:boto:Params: {} 2023-05-31T20:37:56.618 DEBUG:boto:Token: None 2023-05-31T20:37:56.618 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:37:56 GMT /pklxkd-58/ 2023-05-31T20:37:56.618 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:w9W22pmUKEAPpzk1jAPDCAnED+8= 2023-05-31T20:37:56.619 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:37:56 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:w9W22pmUKEAPpzk1jAPDCAnED+8=', 'Content-Length': '0'} 2023-05-31T20:37:56.622 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000008ddcea260daf7e0a-006477b024-1630-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:37:56 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:37:56.622 DEBUG:boto:b'pklxkd-58obj000000000000000000131000falseobj000000000000000000132023-05-31T20:37:29.076Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:37:56.623 DEBUG:boto:path=/pklxkd-58/obj00000000000000000013 2023-05-31T20:37:56.623 DEBUG:boto:auth_path=/pklxkd-58/obj00000000000000000013 2023-05-31T20:37:56.624 DEBUG:boto:Method: HEAD 2023-05-31T20:37:56.624 DEBUG:boto:Path: /pklxkd-58/obj00000000000000000013 2023-05-31T20:37:56.624 DEBUG:boto:Data: 2023-05-31T20:37:56.625 DEBUG:boto:Headers: {} 2023-05-31T20:37:56.625 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:37:56.625 DEBUG:boto:Port: 8001 2023-05-31T20:37:56.626 DEBUG:boto:Params: {} 2023-05-31T20:37:56.626 DEBUG:boto:Token: None 2023-05-31T20:37:56.626 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:37:56 GMT /pklxkd-58/obj00000000000000000013 2023-05-31T20:37:56.627 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:n2LQ5fdIqastCamLpD6SwNBKmxU= 2023-05-31T20:37:56.627 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:37:56 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:n2LQ5fdIqastCamLpD6SwNBKmxU=', 'Content-Length': '0'} 2023-05-31T20:37:56.629 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:37:29 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000053920690e193753b-006477b024-1b4f-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:37:56 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:37:56.630 DEBUG:boto:path=/pklxkd-58/obj00000000000000000013 2023-05-31T20:37:56.630 DEBUG:boto:auth_path=/pklxkd-58/obj00000000000000000013 2023-05-31T20:37:56.630 DEBUG:boto:Method: HEAD 2023-05-31T20:37:56.631 DEBUG:boto:Path: /pklxkd-58/obj00000000000000000013 2023-05-31T20:37:56.631 DEBUG:boto:Data: 2023-05-31T20:37:56.631 DEBUG:boto:Headers: {} 2023-05-31T20:37:56.632 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:37:56.632 DEBUG:boto:Port: 8000 2023-05-31T20:37:56.633 DEBUG:boto:Params: {} 2023-05-31T20:37:56.633 DEBUG:boto:Token: None 2023-05-31T20:37:56.633 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:37:56 GMT /pklxkd-58/obj00000000000000000013 2023-05-31T20:37:56.634 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:n2LQ5fdIqastCamLpD6SwNBKmxU= 2023-05-31T20:37:56.634 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:37:56 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:n2LQ5fdIqastCamLpD6SwNBKmxU=', 'Content-Length': '0'} 2023-05-31T20:37:56.636 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:37:29 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000032fc387d4dd1445f-006477b024-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:37:56 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:37:56.637 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000013 2023-05-31T20:37:56.637 DEBUG:boto:path=/pklxkd-58/obj00000000000000000013 2023-05-31T20:37:56.637 DEBUG:boto:auth_path=/pklxkd-58/obj00000000000000000013 2023-05-31T20:37:56.638 DEBUG:boto:Method: GET 2023-05-31T20:37:56.638 DEBUG:boto:Path: /pklxkd-58/obj00000000000000000013 2023-05-31T20:37:56.638 DEBUG:boto:Data: 2023-05-31T20:37:56.639 DEBUG:boto:Headers: {} 2023-05-31T20:37:56.639 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:37:56.640 DEBUG:boto:Port: 8001 2023-05-31T20:37:56.640 DEBUG:boto:Params: {} 2023-05-31T20:37:56.640 DEBUG:boto:Token: None 2023-05-31T20:37:56.641 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:37:56 GMT /pklxkd-58/obj00000000000000000013 2023-05-31T20:37:56.641 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:fPBkFLSg9nkkwKe4mcbZgO5IVTg= 2023-05-31T20:37:56.642 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:37:56 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:fPBkFLSg9nkkwKe4mcbZgO5IVTg=', 'Content-Length': '0'} 2023-05-31T20:37:56.644 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:37:29 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002c5de7bd60bba615-006477b024-1b4f-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:37:56 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:37:56.644 DEBUG:boto:path=/pklxkd-58/obj00000000000000000013 2023-05-31T20:37:56.645 DEBUG:boto:auth_path=/pklxkd-58/obj00000000000000000013 2023-05-31T20:37:56.645 DEBUG:boto:Method: GET 2023-05-31T20:37:56.646 DEBUG:boto:Path: /pklxkd-58/obj00000000000000000013 2023-05-31T20:37:56.646 DEBUG:boto:Data: 2023-05-31T20:37:56.646 DEBUG:boto:Headers: {} 2023-05-31T20:37:56.647 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:37:56.647 DEBUG:boto:Port: 8000 2023-05-31T20:37:56.648 DEBUG:boto:Params: {} 2023-05-31T20:37:56.648 DEBUG:boto:Token: None 2023-05-31T20:37:56.649 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:37:56 GMT /pklxkd-58/obj00000000000000000013 2023-05-31T20:37:56.649 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:fPBkFLSg9nkkwKe4mcbZgO5IVTg= 2023-05-31T20:37:56.649 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:37:56 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:fPBkFLSg9nkkwKe4mcbZgO5IVTg=', 'Content-Length': '0'} 2023-05-31T20:37:56.651 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:37:29 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000040b7a3ba2ddecabe-006477b024-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:37:56 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:37:56.652 DEBUG:boto:path=/pklxkd-58/ 2023-05-31T20:37:56.652 DEBUG:boto:auth_path=/pklxkd-58/ 2023-05-31T20:37:56.653 DEBUG:boto:path=/pklxkd-58/?prefix=obj00000000000000000013 2023-05-31T20:37:56.653 DEBUG:boto:auth_path=/pklxkd-58/?prefix=obj00000000000000000013 2023-05-31T20:37:56.654 DEBUG:boto:Method: GET 2023-05-31T20:37:56.654 DEBUG:boto:Path: /pklxkd-58/?prefix=obj00000000000000000013 2023-05-31T20:37:56.654 DEBUG:boto:Data: 2023-05-31T20:37:56.655 DEBUG:boto:Headers: {} 2023-05-31T20:37:56.655 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:37:56.656 DEBUG:boto:Port: 8001 2023-05-31T20:37:56.656 DEBUG:boto:Params: {} 2023-05-31T20:37:56.656 DEBUG:boto:Token: None 2023-05-31T20:37:56.657 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:37:56 GMT /pklxkd-58/ 2023-05-31T20:37:56.657 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:w9W22pmUKEAPpzk1jAPDCAnED+8= 2023-05-31T20:37:56.658 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:37:56 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:w9W22pmUKEAPpzk1jAPDCAnED+8=', 'Content-Length': '0'} 2023-05-31T20:37:56.660 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000eafc2387f314a07b-006477b024-1b4f-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:37:56 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:37:56.661 DEBUG:boto:b'pklxkd-58obj000000000000000000131000falseobj000000000000000000132023-05-31T20:37:29.076Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:37:56.661 DEBUG:boto:path=/pklxkd-58/ 2023-05-31T20:37:56.662 DEBUG:boto:auth_path=/pklxkd-58/ 2023-05-31T20:37:56.663 DEBUG:boto:path=/pklxkd-58/?prefix=obj00000000000000000013 2023-05-31T20:37:56.664 DEBUG:boto:auth_path=/pklxkd-58/?prefix=obj00000000000000000013 2023-05-31T20:37:56.664 DEBUG:boto:Method: GET 2023-05-31T20:37:56.666 DEBUG:boto:Path: /pklxkd-58/?prefix=obj00000000000000000013 2023-05-31T20:37:56.666 DEBUG:boto:Data: 2023-05-31T20:37:56.667 DEBUG:boto:Headers: {} 2023-05-31T20:37:56.667 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:37:56.667 DEBUG:boto:Port: 8000 2023-05-31T20:37:56.669 DEBUG:boto:Params: {} 2023-05-31T20:37:56.669 DEBUG:boto:Token: None 2023-05-31T20:37:56.669 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:37:56 GMT /pklxkd-58/ 2023-05-31T20:37:56.670 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:w9W22pmUKEAPpzk1jAPDCAnED+8= 2023-05-31T20:37:56.670 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:37:56 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:w9W22pmUKEAPpzk1jAPDCAnED+8=', 'Content-Length': '0'} 2023-05-31T20:37:56.673 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000004cfa1f973e78bc33-006477b024-1630-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:37:56 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:37:56.673 DEBUG:boto:b'pklxkd-58obj000000000000000000131000falseobj000000000000000000132023-05-31T20:37:29.076Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:37:56.674 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=pklxkd-58 zones={a2, a1} 2023-05-31T20:37:56.674 DEBUG:tasks.rgw.c1.client.1:waiting for process to exit 2023-05-31T20:37:56.675 INFO:teuthology.orchestra.run:waiting for 300 2023-05-31T20:37:56.676 INFO:tasks.rgw.c1.client.1.smithi099.stdout:2023-05-31T20:37:56.670+0000 7fe3a35a2700 -1 received signal: Terminated from /usr/bin/python3 /bin/daemon-helper term radosgw --rgw-frontends beast port=8001 -n client.1 --cluster c1 -k /etc/ceph/c1.client.1.keyring --log-file /var/log/ceph/rgw.c1.client.1.log --rgw_ops_log_socket_path /home/ubuntu/cephtest/rgw.opslog.c1.client.1.sock --foreground --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm (PID: 150919) UID: 0 2023-05-31T20:37:56.676 INFO:tasks.rgw.c1.client.1.smithi099.stdout:2023-05-31T20:37:56.670+0000 7fe3a780b7c0 -1 shutting down 2023-05-31T20:38:11.678 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.1 is failed for ~0s 2023-05-31T20:38:14.978 INFO:tasks.rgw.c1.client.1:Stopped 2023-05-31T20:38:14.979 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'reshard', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'pklxkd-58', '--num-shards', '17', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-05-31T20:38:14.979 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'reshard', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'pklxkd-58', '--num-shards', '17', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-05-31T20:38:14.980 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket reshard --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --bucket pklxkd-58 --num-shards 17 --yes-i-really-mean-it --cluster c1 --debug-rgw 0 --debug-ms 0 2023-05-31T20:38:15.041 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2023-05-31T20:38:15.041 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2023-05-31T20:38:15.118 INFO:teuthology.orchestra.run.smithi099.stdout:tenant: 2023-05-31T20:38:15.118 INFO:teuthology.orchestra.run.smithi099.stdout:bucket name: pklxkd-58 2023-05-31T20:38:15.122 INFO:teuthology.orchestra.run.smithi099.stdout:total entries: 10 2023-05-31T20:38:15.170 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'shard', 'objects', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--num-shards', '17', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-31T20:38:15.171 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'shard', 'objects', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--num-shards', '17', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-31T20:38:15.171 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket shard objects --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --num-shards 17 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-05-31T20:38:15.223 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2023-05-31T20:38:15.223 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2023-05-31T20:38:15.273 INFO:teuthology.orchestra.run.smithi099.stdout:{ 2023-05-31T20:38:15.273 INFO:teuthology.orchestra.run.smithi099.stdout: "objs": [ 2023-05-31T20:38:15.274 INFO:teuthology.orchestra.run.smithi099.stdout: "obj00000000000000000007", 2023-05-31T20:38:15.274 INFO:teuthology.orchestra.run.smithi099.stdout: "obj00000000000000000016", 2023-05-31T20:38:15.274 INFO:teuthology.orchestra.run.smithi099.stdout: "obj00000000000000000024", 2023-05-31T20:38:15.275 INFO:teuthology.orchestra.run.smithi099.stdout: "obj00000000000000000020", 2023-05-31T20:38:15.275 INFO:teuthology.orchestra.run.smithi099.stdout: "obj00000000000000000028", 2023-05-31T20:38:15.275 INFO:teuthology.orchestra.run.smithi099.stdout: "obj00000000000000000000", 2023-05-31T20:38:15.276 INFO:teuthology.orchestra.run.smithi099.stdout: "obj00000000000000000032", 2023-05-31T20:38:15.276 INFO:teuthology.orchestra.run.smithi099.stdout: "obj00000000000000000001", 2023-05-31T20:38:15.276 INFO:teuthology.orchestra.run.smithi099.stdout: "obj00000000000000000010", 2023-05-31T20:38:15.276 INFO:teuthology.orchestra.run.smithi099.stdout: "obj00000000000000000019", 2023-05-31T20:38:15.277 INFO:teuthology.orchestra.run.smithi099.stdout: "obj00000000000000000013", 2023-05-31T20:38:15.277 INFO:teuthology.orchestra.run.smithi099.stdout: "obj00000000000000000021", 2023-05-31T20:38:15.277 INFO:teuthology.orchestra.run.smithi099.stdout: "obj00000000000000000030", 2023-05-31T20:38:15.277 INFO:teuthology.orchestra.run.smithi099.stdout: "obj00000000000000000026", 2023-05-31T20:38:15.277 INFO:teuthology.orchestra.run.smithi099.stdout: "obj00000000000000000033", 2023-05-31T20:38:15.277 INFO:teuthology.orchestra.run.smithi099.stdout: "obj00000000000000000004", 2023-05-31T20:38:15.278 INFO:teuthology.orchestra.run.smithi099.stdout: "obj00000000000000000076" 2023-05-31T20:38:15.278 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2023-05-31T20:38:15.278 INFO:teuthology.orchestra.run.smithi099.stdout:} 2023-05-31T20:38:15.279 DEBUG:boto:path=/pklxkd-58/ 2023-05-31T20:38:15.280 DEBUG:boto:auth_path=/pklxkd-58/ 2023-05-31T20:38:15.280 DEBUG:boto:Method: HEAD 2023-05-31T20:38:15.280 DEBUG:boto:Path: /pklxkd-58/ 2023-05-31T20:38:15.281 DEBUG:boto:Data: 2023-05-31T20:38:15.281 DEBUG:boto:Headers: {} 2023-05-31T20:38:15.281 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:38:15.282 DEBUG:boto:Port: 8000 2023-05-31T20:38:15.283 DEBUG:boto:Params: {} 2023-05-31T20:38:15.284 DEBUG:boto:Token: None 2023-05-31T20:38:15.285 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:38:15 GMT /pklxkd-58/ 2023-05-31T20:38:15.286 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:zOj+u86Oolgn2P4vg/jZJGKSDpk= 2023-05-31T20:38:15.288 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:38:15 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:zOj+u86Oolgn2P4vg/jZJGKSDpk=', 'Content-Length': '0'} 2023-05-31T20:38:15.291 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '10'), ('X-RGW-Bytes-Used', '30'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000008d98cdde1f8678f4-006477b037-1630-a1'), ('Content-Length', '0'), ('Date', 'Wed, 31 May 2023 20:38:15 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:38:15.292 DEBUG:boto:path=/pklxkd-58/obj00000000000000000033 2023-05-31T20:38:15.292 DEBUG:boto:auth_path=/pklxkd-58/obj00000000000000000033 2023-05-31T20:38:15.293 DEBUG:boto:Method: PUT 2023-05-31T20:38:15.293 DEBUG:boto:Path: /pklxkd-58/obj00000000000000000033 2023-05-31T20:38:15.294 DEBUG:boto:Data: 2023-05-31T20:38:15.294 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-05-31T20:38:15.294 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:38:15.295 DEBUG:boto:Port: 8000 2023-05-31T20:38:15.295 DEBUG:boto:Params: {} 2023-05-31T20:38:15.295 DEBUG:boto:Token: None 2023-05-31T20:38:15.296 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Wed, 31 May 2023 20:38:15 GMT /pklxkd-58/obj00000000000000000033 2023-05-31T20:38:15.296 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:hWdQ8lkkkGDOunFikmI+fQlHBEE= 2023-05-31T20:38:15.297 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Wed, 31 May 2023 20:38:15 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:hWdQ8lkkkGDOunFikmI+fQlHBEE='} 2023-05-31T20:38:15.302 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000025ac16af735b072b-006477b037-1630-a1'), ('Date', 'Wed, 31 May 2023 20:38:15 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:38:15.303 DEBUG:boto:path=/pklxkd-58/ 2023-05-31T20:38:15.304 DEBUG:boto:auth_path=/pklxkd-58/ 2023-05-31T20:38:15.304 DEBUG:boto:Method: HEAD 2023-05-31T20:38:15.305 DEBUG:boto:Path: /pklxkd-58/ 2023-05-31T20:38:15.306 DEBUG:boto:Data: 2023-05-31T20:38:15.306 DEBUG:boto:Headers: {} 2023-05-31T20:38:15.306 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:38:15.307 DEBUG:boto:Port: 8000 2023-05-31T20:38:15.307 DEBUG:boto:Params: {} 2023-05-31T20:38:15.309 DEBUG:boto:Token: None 2023-05-31T20:38:15.309 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:38:15 GMT /pklxkd-58/ 2023-05-31T20:38:15.310 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:zOj+u86Oolgn2P4vg/jZJGKSDpk= 2023-05-31T20:38:15.310 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:38:15 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:zOj+u86Oolgn2P4vg/jZJGKSDpk=', 'Content-Length': '0'} 2023-05-31T20:38:15.312 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '11'), ('X-RGW-Bytes-Used', '33'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000f4f6f226ad4fc1ae-006477b037-1630-a1'), ('Content-Length', '0'), ('Date', 'Wed, 31 May 2023 20:38:15 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:38:15.313 DEBUG:boto:path=/pklxkd-58/obj00000000000000000028 2023-05-31T20:38:15.313 DEBUG:boto:auth_path=/pklxkd-58/obj00000000000000000028 2023-05-31T20:38:15.314 DEBUG:boto:Method: PUT 2023-05-31T20:38:15.314 DEBUG:boto:Path: /pklxkd-58/obj00000000000000000028 2023-05-31T20:38:15.315 DEBUG:boto:Data: 2023-05-31T20:38:15.315 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-05-31T20:38:15.315 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:38:15.316 DEBUG:boto:Port: 8000 2023-05-31T20:38:15.316 DEBUG:boto:Params: {} 2023-05-31T20:38:15.316 DEBUG:boto:Token: None 2023-05-31T20:38:15.317 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Wed, 31 May 2023 20:38:15 GMT /pklxkd-58/obj00000000000000000028 2023-05-31T20:38:15.318 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:LqgriJpJGEDMxxJKoTuferfBAwc= 2023-05-31T20:38:15.318 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Wed, 31 May 2023 20:38:15 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:LqgriJpJGEDMxxJKoTuferfBAwc='} 2023-05-31T20:38:15.364 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000008184ab89d42342a3-006477b037-1630-a1'), ('Date', 'Wed, 31 May 2023 20:38:15 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:38:15.365 DEBUG:boto:path=/pklxkd-58/ 2023-05-31T20:38:15.365 DEBUG:boto:auth_path=/pklxkd-58/ 2023-05-31T20:38:15.366 DEBUG:boto:Method: HEAD 2023-05-31T20:38:15.366 DEBUG:boto:Path: /pklxkd-58/ 2023-05-31T20:38:15.367 DEBUG:boto:Data: 2023-05-31T20:38:15.367 DEBUG:boto:Headers: {} 2023-05-31T20:38:15.367 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:38:15.368 DEBUG:boto:Port: 8000 2023-05-31T20:38:15.368 DEBUG:boto:Params: {} 2023-05-31T20:38:15.368 DEBUG:boto:Token: None 2023-05-31T20:38:15.369 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:38:15 GMT /pklxkd-58/ 2023-05-31T20:38:15.369 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:zOj+u86Oolgn2P4vg/jZJGKSDpk= 2023-05-31T20:38:15.370 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:38:15 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:zOj+u86Oolgn2P4vg/jZJGKSDpk=', 'Content-Length': '0'} 2023-05-31T20:38:15.372 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '12'), ('X-RGW-Bytes-Used', '36'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000081b1add7695395b8-006477b037-1630-a1'), ('Content-Length', '0'), ('Date', 'Wed, 31 May 2023 20:38:15 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:38:15.373 DEBUG:boto:path=/pklxkd-58/obj00000000000000000024 2023-05-31T20:38:15.373 DEBUG:boto:auth_path=/pklxkd-58/obj00000000000000000024 2023-05-31T20:38:15.373 DEBUG:boto:Method: PUT 2023-05-31T20:38:15.374 DEBUG:boto:Path: /pklxkd-58/obj00000000000000000024 2023-05-31T20:38:15.374 DEBUG:boto:Data: 2023-05-31T20:38:15.374 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-05-31T20:38:15.375 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:38:15.375 DEBUG:boto:Port: 8000 2023-05-31T20:38:15.375 DEBUG:boto:Params: {} 2023-05-31T20:38:15.376 DEBUG:boto:Token: None 2023-05-31T20:38:15.376 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Wed, 31 May 2023 20:38:15 GMT /pklxkd-58/obj00000000000000000024 2023-05-31T20:38:15.377 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:IAGp8k8X/eoEJmdiVFDxQQMYuhQ= 2023-05-31T20:38:15.377 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Wed, 31 May 2023 20:38:15 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:IAGp8k8X/eoEJmdiVFDxQQMYuhQ='} 2023-05-31T20:38:15.383 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000b299529395961d85-006477b037-1630-a1'), ('Date', 'Wed, 31 May 2023 20:38:15 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:38:15.384 DEBUG:boto:path=/pklxkd-58/ 2023-05-31T20:38:15.384 DEBUG:boto:auth_path=/pklxkd-58/ 2023-05-31T20:38:15.384 DEBUG:boto:Method: HEAD 2023-05-31T20:38:15.385 DEBUG:boto:Path: /pklxkd-58/ 2023-05-31T20:38:15.385 DEBUG:boto:Data: 2023-05-31T20:38:15.385 DEBUG:boto:Headers: {} 2023-05-31T20:38:15.386 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:38:15.386 DEBUG:boto:Port: 8000 2023-05-31T20:38:15.386 DEBUG:boto:Params: {} 2023-05-31T20:38:15.387 DEBUG:boto:Token: None 2023-05-31T20:38:15.387 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:38:15 GMT /pklxkd-58/ 2023-05-31T20:38:15.388 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:zOj+u86Oolgn2P4vg/jZJGKSDpk= 2023-05-31T20:38:15.388 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:38:15 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:zOj+u86Oolgn2P4vg/jZJGKSDpk=', 'Content-Length': '0'} 2023-05-31T20:38:15.391 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '13'), ('X-RGW-Bytes-Used', '39'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000003310947dbec81e19-006477b037-1630-a1'), ('Content-Length', '0'), ('Date', 'Wed, 31 May 2023 20:38:15 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:38:15.392 DEBUG:boto:path=/pklxkd-58/obj00000000000000000013 2023-05-31T20:38:15.392 DEBUG:boto:auth_path=/pklxkd-58/obj00000000000000000013 2023-05-31T20:38:15.393 DEBUG:boto:Method: PUT 2023-05-31T20:38:15.393 DEBUG:boto:Path: /pklxkd-58/obj00000000000000000013 2023-05-31T20:38:15.393 DEBUG:boto:Data: 2023-05-31T20:38:15.394 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-05-31T20:38:15.394 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:38:15.394 DEBUG:boto:Port: 8000 2023-05-31T20:38:15.395 DEBUG:boto:Params: {} 2023-05-31T20:38:15.395 DEBUG:boto:Token: None 2023-05-31T20:38:15.396 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Wed, 31 May 2023 20:38:15 GMT /pklxkd-58/obj00000000000000000013 2023-05-31T20:38:15.396 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:Mz8qMqJJQNsTByF6M7LAP0XEbS8= 2023-05-31T20:38:15.396 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Wed, 31 May 2023 20:38:15 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:Mz8qMqJJQNsTByF6M7LAP0XEbS8='} 2023-05-31T20:38:15.403 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000763dfd9f66b5b022-006477b037-1630-a1'), ('Date', 'Wed, 31 May 2023 20:38:15 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:38:15.403 DEBUG:boto:path=/pklxkd-58/ 2023-05-31T20:38:15.404 DEBUG:boto:auth_path=/pklxkd-58/ 2023-05-31T20:38:15.404 DEBUG:boto:Method: HEAD 2023-05-31T20:38:15.404 DEBUG:boto:Path: /pklxkd-58/ 2023-05-31T20:38:15.405 DEBUG:boto:Data: 2023-05-31T20:38:15.405 DEBUG:boto:Headers: {} 2023-05-31T20:38:15.405 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:38:15.406 DEBUG:boto:Port: 8000 2023-05-31T20:38:15.406 DEBUG:boto:Params: {} 2023-05-31T20:38:15.406 DEBUG:boto:Token: None 2023-05-31T20:38:15.407 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:38:15 GMT /pklxkd-58/ 2023-05-31T20:38:15.407 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:zOj+u86Oolgn2P4vg/jZJGKSDpk= 2023-05-31T20:38:15.408 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:38:15 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:zOj+u86Oolgn2P4vg/jZJGKSDpk=', 'Content-Length': '0'} 2023-05-31T20:38:15.410 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '13'), ('X-RGW-Bytes-Used', '39'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000008e96dbb56a0bc0c4-006477b037-1630-a1'), ('Content-Length', '0'), ('Date', 'Wed, 31 May 2023 20:38:15 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:38:15.410 DEBUG:boto:path=/pklxkd-58/obj00000000000000000019 2023-05-31T20:38:15.411 DEBUG:boto:auth_path=/pklxkd-58/obj00000000000000000019 2023-05-31T20:38:15.411 DEBUG:boto:Method: PUT 2023-05-31T20:38:15.412 DEBUG:boto:Path: /pklxkd-58/obj00000000000000000019 2023-05-31T20:38:15.412 DEBUG:boto:Data: 2023-05-31T20:38:15.413 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-05-31T20:38:15.413 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:38:15.413 DEBUG:boto:Port: 8000 2023-05-31T20:38:15.414 DEBUG:boto:Params: {} 2023-05-31T20:38:15.414 DEBUG:boto:Token: None 2023-05-31T20:38:15.414 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Wed, 31 May 2023 20:38:15 GMT /pklxkd-58/obj00000000000000000019 2023-05-31T20:38:15.415 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:o8yxSMZBC3Q5IKpvFMPeAhMKp9Y= 2023-05-31T20:38:15.415 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Wed, 31 May 2023 20:38:15 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:o8yxSMZBC3Q5IKpvFMPeAhMKp9Y='} 2023-05-31T20:38:15.460 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000003be22bdb8bd2c0ab-006477b037-1630-a1'), ('Date', 'Wed, 31 May 2023 20:38:15 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:38:15.461 DEBUG:boto:path=/pklxkd-58/ 2023-05-31T20:38:15.461 DEBUG:boto:auth_path=/pklxkd-58/ 2023-05-31T20:38:15.462 DEBUG:boto:Method: HEAD 2023-05-31T20:38:15.462 DEBUG:boto:Path: /pklxkd-58/ 2023-05-31T20:38:15.462 DEBUG:boto:Data: 2023-05-31T20:38:15.463 DEBUG:boto:Headers: {} 2023-05-31T20:38:15.463 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:38:15.463 DEBUG:boto:Port: 8000 2023-05-31T20:38:15.464 DEBUG:boto:Params: {} 2023-05-31T20:38:15.464 DEBUG:boto:Token: None 2023-05-31T20:38:15.465 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:38:15 GMT /pklxkd-58/ 2023-05-31T20:38:15.465 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:zOj+u86Oolgn2P4vg/jZJGKSDpk= 2023-05-31T20:38:15.465 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:38:15 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:zOj+u86Oolgn2P4vg/jZJGKSDpk=', 'Content-Length': '0'} 2023-05-31T20:38:15.468 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '14'), ('X-RGW-Bytes-Used', '42'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000005ec7a25f8dcd7791-006477b037-1630-a1'), ('Content-Length', '0'), ('Date', 'Wed, 31 May 2023 20:38:15 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:38:15.469 DEBUG:boto:path=/pklxkd-58/obj00000000000000000021 2023-05-31T20:38:15.469 DEBUG:boto:auth_path=/pklxkd-58/obj00000000000000000021 2023-05-31T20:38:15.469 DEBUG:boto:Method: PUT 2023-05-31T20:38:15.470 DEBUG:boto:Path: /pklxkd-58/obj00000000000000000021 2023-05-31T20:38:15.470 DEBUG:boto:Data: 2023-05-31T20:38:15.470 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-05-31T20:38:15.471 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:38:15.471 DEBUG:boto:Port: 8000 2023-05-31T20:38:15.471 DEBUG:boto:Params: {} 2023-05-31T20:38:15.472 DEBUG:boto:Token: None 2023-05-31T20:38:15.472 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Wed, 31 May 2023 20:38:15 GMT /pklxkd-58/obj00000000000000000021 2023-05-31T20:38:15.473 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:0Z2J0+eLI+CQZNArrnhDpiCnl5k= 2023-05-31T20:38:15.473 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Wed, 31 May 2023 20:38:15 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:0Z2J0+eLI+CQZNArrnhDpiCnl5k='} 2023-05-31T20:38:15.478 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000003c7fe78ef077ddb0-006477b037-1630-a1'), ('Date', 'Wed, 31 May 2023 20:38:15 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:38:15.479 DEBUG:boto:path=/pklxkd-58/ 2023-05-31T20:38:15.479 DEBUG:boto:auth_path=/pklxkd-58/ 2023-05-31T20:38:15.479 DEBUG:boto:Method: HEAD 2023-05-31T20:38:15.480 DEBUG:boto:Path: /pklxkd-58/ 2023-05-31T20:38:15.480 DEBUG:boto:Data: 2023-05-31T20:38:15.480 DEBUG:boto:Headers: {} 2023-05-31T20:38:15.481 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:38:15.481 DEBUG:boto:Port: 8000 2023-05-31T20:38:15.481 DEBUG:boto:Params: {} 2023-05-31T20:38:15.482 DEBUG:boto:Token: None 2023-05-31T20:38:15.482 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:38:15 GMT /pklxkd-58/ 2023-05-31T20:38:15.483 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:zOj+u86Oolgn2P4vg/jZJGKSDpk= 2023-05-31T20:38:15.483 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:38:15 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:zOj+u86Oolgn2P4vg/jZJGKSDpk=', 'Content-Length': '0'} 2023-05-31T20:38:15.486 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '15'), ('X-RGW-Bytes-Used', '45'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000006a381b626a3d072b-006477b037-1630-a1'), ('Content-Length', '0'), ('Date', 'Wed, 31 May 2023 20:38:15 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:38:15.486 DEBUG:boto:path=/pklxkd-58/obj00000000000000000026 2023-05-31T20:38:15.487 DEBUG:boto:auth_path=/pklxkd-58/obj00000000000000000026 2023-05-31T20:38:15.487 DEBUG:boto:Method: PUT 2023-05-31T20:38:15.487 DEBUG:boto:Path: /pklxkd-58/obj00000000000000000026 2023-05-31T20:38:15.488 DEBUG:boto:Data: 2023-05-31T20:38:15.488 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-05-31T20:38:15.488 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:38:15.489 DEBUG:boto:Port: 8000 2023-05-31T20:38:15.489 DEBUG:boto:Params: {} 2023-05-31T20:38:15.490 DEBUG:boto:Token: None 2023-05-31T20:38:15.490 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Wed, 31 May 2023 20:38:15 GMT /pklxkd-58/obj00000000000000000026 2023-05-31T20:38:15.490 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:ybzDp5KnxNpaOnIuAXaW9xloTME= 2023-05-31T20:38:15.491 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Wed, 31 May 2023 20:38:15 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:ybzDp5KnxNpaOnIuAXaW9xloTME='} 2023-05-31T20:38:15.537 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000289f49584093e866-006477b037-1630-a1'), ('Date', 'Wed, 31 May 2023 20:38:15 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:38:15.538 DEBUG:boto:path=/pklxkd-58/ 2023-05-31T20:38:15.538 DEBUG:boto:auth_path=/pklxkd-58/ 2023-05-31T20:38:15.539 DEBUG:boto:Method: HEAD 2023-05-31T20:38:15.540 DEBUG:boto:Path: /pklxkd-58/ 2023-05-31T20:38:15.540 DEBUG:boto:Data: 2023-05-31T20:38:15.541 DEBUG:boto:Headers: {} 2023-05-31T20:38:15.542 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:38:15.542 DEBUG:boto:Port: 8000 2023-05-31T20:38:15.543 DEBUG:boto:Params: {} 2023-05-31T20:38:15.544 DEBUG:boto:Token: None 2023-05-31T20:38:15.545 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:38:15 GMT /pklxkd-58/ 2023-05-31T20:38:15.546 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:zOj+u86Oolgn2P4vg/jZJGKSDpk= 2023-05-31T20:38:15.547 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:38:15 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:zOj+u86Oolgn2P4vg/jZJGKSDpk=', 'Content-Length': '0'} 2023-05-31T20:38:15.550 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '16'), ('X-RGW-Bytes-Used', '48'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000021bffb01fef13a7e-006477b037-1630-a1'), ('Content-Length', '0'), ('Date', 'Wed, 31 May 2023 20:38:15 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:38:15.551 DEBUG:boto:path=/pklxkd-58/obj00000000000000000000 2023-05-31T20:38:15.553 DEBUG:boto:auth_path=/pklxkd-58/obj00000000000000000000 2023-05-31T20:38:15.553 DEBUG:boto:Method: PUT 2023-05-31T20:38:15.553 DEBUG:boto:Path: /pklxkd-58/obj00000000000000000000 2023-05-31T20:38:15.554 DEBUG:boto:Data: 2023-05-31T20:38:15.554 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-05-31T20:38:15.555 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:38:15.555 DEBUG:boto:Port: 8000 2023-05-31T20:38:15.556 DEBUG:boto:Params: {} 2023-05-31T20:38:15.557 DEBUG:boto:Token: None 2023-05-31T20:38:15.558 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Wed, 31 May 2023 20:38:15 GMT /pklxkd-58/obj00000000000000000000 2023-05-31T20:38:15.559 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:P86Q+hsbjbsNXtpXqEKalhnuF/U= 2023-05-31T20:38:15.560 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Wed, 31 May 2023 20:38:15 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:P86Q+hsbjbsNXtpXqEKalhnuF/U='} 2023-05-31T20:38:15.568 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000e3aaca71b4fa54f8-006477b037-1630-a1'), ('Date', 'Wed, 31 May 2023 20:38:15 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:38:15.569 DEBUG:boto:path=/pklxkd-58/ 2023-05-31T20:38:15.570 DEBUG:boto:auth_path=/pklxkd-58/ 2023-05-31T20:38:15.571 DEBUG:boto:Method: HEAD 2023-05-31T20:38:15.572 DEBUG:boto:Path: /pklxkd-58/ 2023-05-31T20:38:15.573 DEBUG:boto:Data: 2023-05-31T20:38:15.573 DEBUG:boto:Headers: {} 2023-05-31T20:38:15.574 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:38:15.574 DEBUG:boto:Port: 8000 2023-05-31T20:38:15.574 DEBUG:boto:Params: {} 2023-05-31T20:38:15.575 DEBUG:boto:Token: None 2023-05-31T20:38:15.575 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:38:15 GMT /pklxkd-58/ 2023-05-31T20:38:15.576 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:zOj+u86Oolgn2P4vg/jZJGKSDpk= 2023-05-31T20:38:15.576 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:38:15 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:zOj+u86Oolgn2P4vg/jZJGKSDpk=', 'Content-Length': '0'} 2023-05-31T20:38:15.580 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '16'), ('X-RGW-Bytes-Used', '48'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000aa79cfc19c88ce7c-006477b037-1630-a1'), ('Content-Length', '0'), ('Date', 'Wed, 31 May 2023 20:38:15 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:38:15.581 DEBUG:boto:path=/pklxkd-58/obj00000000000000000001 2023-05-31T20:38:15.581 DEBUG:boto:auth_path=/pklxkd-58/obj00000000000000000001 2023-05-31T20:38:15.581 DEBUG:boto:Method: PUT 2023-05-31T20:38:15.582 DEBUG:boto:Path: /pklxkd-58/obj00000000000000000001 2023-05-31T20:38:15.582 DEBUG:boto:Data: 2023-05-31T20:38:15.582 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-05-31T20:38:15.583 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:38:15.583 DEBUG:boto:Port: 8000 2023-05-31T20:38:15.583 DEBUG:boto:Params: {} 2023-05-31T20:38:15.584 DEBUG:boto:Token: None 2023-05-31T20:38:15.584 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Wed, 31 May 2023 20:38:15 GMT /pklxkd-58/obj00000000000000000001 2023-05-31T20:38:15.585 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:qbWZyYmgU6fTloNb3OPMWSYnfmE= 2023-05-31T20:38:15.585 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Wed, 31 May 2023 20:38:15 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:qbWZyYmgU6fTloNb3OPMWSYnfmE='} 2023-05-31T20:38:15.628 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000c1e5c1741e6b9ee8-006477b037-1630-a1'), ('Date', 'Wed, 31 May 2023 20:38:15 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:38:15.629 DEBUG:boto:path=/pklxkd-58/ 2023-05-31T20:38:15.630 DEBUG:boto:auth_path=/pklxkd-58/ 2023-05-31T20:38:15.630 DEBUG:boto:Method: HEAD 2023-05-31T20:38:15.630 DEBUG:boto:Path: /pklxkd-58/ 2023-05-31T20:38:15.631 DEBUG:boto:Data: 2023-05-31T20:38:15.631 DEBUG:boto:Headers: {} 2023-05-31T20:38:15.631 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:38:15.632 DEBUG:boto:Port: 8000 2023-05-31T20:38:15.632 DEBUG:boto:Params: {} 2023-05-31T20:38:15.632 DEBUG:boto:Token: None 2023-05-31T20:38:15.633 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:38:15 GMT /pklxkd-58/ 2023-05-31T20:38:15.633 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:zOj+u86Oolgn2P4vg/jZJGKSDpk= 2023-05-31T20:38:15.634 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:38:15 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:zOj+u86Oolgn2P4vg/jZJGKSDpk=', 'Content-Length': '0'} 2023-05-31T20:38:15.639 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '16'), ('X-RGW-Bytes-Used', '48'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000009a030bf11400302e-006477b037-1630-a1'), ('Content-Length', '0'), ('Date', 'Wed, 31 May 2023 20:38:15 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:38:15.640 DEBUG:boto:path=/pklxkd-58/obj00000000000000000007 2023-05-31T20:38:15.641 DEBUG:boto:auth_path=/pklxkd-58/obj00000000000000000007 2023-05-31T20:38:15.642 DEBUG:boto:Method: PUT 2023-05-31T20:38:15.643 DEBUG:boto:Path: /pklxkd-58/obj00000000000000000007 2023-05-31T20:38:15.644 DEBUG:boto:Data: 2023-05-31T20:38:15.645 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-05-31T20:38:15.646 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:38:15.647 DEBUG:boto:Port: 8000 2023-05-31T20:38:15.648 DEBUG:boto:Params: {} 2023-05-31T20:38:15.649 DEBUG:boto:Token: None 2023-05-31T20:38:15.649 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Wed, 31 May 2023 20:38:15 GMT /pklxkd-58/obj00000000000000000007 2023-05-31T20:38:15.650 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:7FAdhXmDSd7SHEv1koIbOHL08lo= 2023-05-31T20:38:15.651 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Wed, 31 May 2023 20:38:15 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:7FAdhXmDSd7SHEv1koIbOHL08lo='} 2023-05-31T20:38:15.658 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000ab3a52d0d73fa03d-006477b037-1630-a1'), ('Date', 'Wed, 31 May 2023 20:38:15 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:38:15.658 DEBUG:boto:path=/pklxkd-58/ 2023-05-31T20:38:15.659 DEBUG:boto:auth_path=/pklxkd-58/ 2023-05-31T20:38:15.659 DEBUG:boto:Method: HEAD 2023-05-31T20:38:15.659 DEBUG:boto:Path: /pklxkd-58/ 2023-05-31T20:38:15.660 DEBUG:boto:Data: 2023-05-31T20:38:15.660 DEBUG:boto:Headers: {} 2023-05-31T20:38:15.660 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:38:15.661 DEBUG:boto:Port: 8000 2023-05-31T20:38:15.661 DEBUG:boto:Params: {} 2023-05-31T20:38:15.661 DEBUG:boto:Token: None 2023-05-31T20:38:15.662 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:38:15 GMT /pklxkd-58/ 2023-05-31T20:38:15.662 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:zOj+u86Oolgn2P4vg/jZJGKSDpk= 2023-05-31T20:38:15.663 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:38:15 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:zOj+u86Oolgn2P4vg/jZJGKSDpk=', 'Content-Length': '0'} 2023-05-31T20:38:15.668 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '17'), ('X-RGW-Bytes-Used', '51'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000008960629788525ebf-006477b037-1630-a1'), ('Content-Length', '0'), ('Date', 'Wed, 31 May 2023 20:38:15 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:38:15.669 DEBUG:boto:path=/pklxkd-58/obj00000000000000000016 2023-05-31T20:38:15.669 DEBUG:boto:auth_path=/pklxkd-58/obj00000000000000000016 2023-05-31T20:38:15.670 DEBUG:boto:Method: PUT 2023-05-31T20:38:15.670 DEBUG:boto:Path: /pklxkd-58/obj00000000000000000016 2023-05-31T20:38:15.671 DEBUG:boto:Data: 2023-05-31T20:38:15.671 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-05-31T20:38:15.671 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:38:15.672 DEBUG:boto:Port: 8000 2023-05-31T20:38:15.672 DEBUG:boto:Params: {} 2023-05-31T20:38:15.672 DEBUG:boto:Token: None 2023-05-31T20:38:15.673 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Wed, 31 May 2023 20:38:15 GMT /pklxkd-58/obj00000000000000000016 2023-05-31T20:38:15.673 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:WyOs75VUL4nRHqTm7Qy9AINIpQk= 2023-05-31T20:38:15.674 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Wed, 31 May 2023 20:38:15 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:WyOs75VUL4nRHqTm7Qy9AINIpQk='} 2023-05-31T20:38:15.717 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000c266cfbb4e0814f2-006477b037-1630-a1'), ('Date', 'Wed, 31 May 2023 20:38:15 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:38:15.717 DEBUG:boto:path=/pklxkd-58/ 2023-05-31T20:38:15.718 DEBUG:boto:auth_path=/pklxkd-58/ 2023-05-31T20:38:15.718 DEBUG:boto:Method: HEAD 2023-05-31T20:38:15.718 DEBUG:boto:Path: /pklxkd-58/ 2023-05-31T20:38:15.719 DEBUG:boto:Data: 2023-05-31T20:38:15.719 DEBUG:boto:Headers: {} 2023-05-31T20:38:15.719 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:38:15.720 DEBUG:boto:Port: 8000 2023-05-31T20:38:15.720 DEBUG:boto:Params: {} 2023-05-31T20:38:15.721 DEBUG:boto:Token: None 2023-05-31T20:38:15.722 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:38:15 GMT /pklxkd-58/ 2023-05-31T20:38:15.723 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:zOj+u86Oolgn2P4vg/jZJGKSDpk= 2023-05-31T20:38:15.724 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:38:15 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:zOj+u86Oolgn2P4vg/jZJGKSDpk=', 'Content-Length': '0'} 2023-05-31T20:38:15.727 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '18'), ('X-RGW-Bytes-Used', '54'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000005f02696a32a56056-006477b037-1630-a1'), ('Content-Length', '0'), ('Date', 'Wed, 31 May 2023 20:38:15 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:38:15.729 DEBUG:boto:path=/pklxkd-58/obj00000000000000000030 2023-05-31T20:38:15.729 DEBUG:boto:auth_path=/pklxkd-58/obj00000000000000000030 2023-05-31T20:38:15.730 DEBUG:boto:Method: PUT 2023-05-31T20:38:15.730 DEBUG:boto:Path: /pklxkd-58/obj00000000000000000030 2023-05-31T20:38:15.731 DEBUG:boto:Data: 2023-05-31T20:38:15.732 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-05-31T20:38:15.732 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:38:15.732 DEBUG:boto:Port: 8000 2023-05-31T20:38:15.733 DEBUG:boto:Params: {} 2023-05-31T20:38:15.733 DEBUG:boto:Token: None 2023-05-31T20:38:15.734 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Wed, 31 May 2023 20:38:15 GMT /pklxkd-58/obj00000000000000000030 2023-05-31T20:38:15.734 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:NEIWCL4R2thj/4q9YB5hcwd4u3o= 2023-05-31T20:38:15.734 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Wed, 31 May 2023 20:38:15 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:NEIWCL4R2thj/4q9YB5hcwd4u3o='} 2023-05-31T20:38:15.740 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000009e859c2cb18c9a27-006477b037-1630-a1'), ('Date', 'Wed, 31 May 2023 20:38:15 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:38:15.741 DEBUG:boto:path=/pklxkd-58/ 2023-05-31T20:38:15.741 DEBUG:boto:auth_path=/pklxkd-58/ 2023-05-31T20:38:15.741 DEBUG:boto:Method: HEAD 2023-05-31T20:38:15.742 DEBUG:boto:Path: /pklxkd-58/ 2023-05-31T20:38:15.742 DEBUG:boto:Data: 2023-05-31T20:38:15.742 DEBUG:boto:Headers: {} 2023-05-31T20:38:15.743 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:38:15.743 DEBUG:boto:Port: 8000 2023-05-31T20:38:15.743 DEBUG:boto:Params: {} 2023-05-31T20:38:15.744 DEBUG:boto:Token: None 2023-05-31T20:38:15.744 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:38:15 GMT /pklxkd-58/ 2023-05-31T20:38:15.745 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:zOj+u86Oolgn2P4vg/jZJGKSDpk= 2023-05-31T20:38:15.745 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:38:15 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:zOj+u86Oolgn2P4vg/jZJGKSDpk=', 'Content-Length': '0'} 2023-05-31T20:38:15.747 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '19'), ('X-RGW-Bytes-Used', '57'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000a26d281d4f17232d-006477b037-1630-a1'), ('Content-Length', '0'), ('Date', 'Wed, 31 May 2023 20:38:15 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:38:15.748 DEBUG:boto:path=/pklxkd-58/obj00000000000000000020 2023-05-31T20:38:15.748 DEBUG:boto:auth_path=/pklxkd-58/obj00000000000000000020 2023-05-31T20:38:15.749 DEBUG:boto:Method: PUT 2023-05-31T20:38:15.750 DEBUG:boto:Path: /pklxkd-58/obj00000000000000000020 2023-05-31T20:38:15.750 DEBUG:boto:Data: 2023-05-31T20:38:15.750 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-05-31T20:38:15.751 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:38:15.751 DEBUG:boto:Port: 8000 2023-05-31T20:38:15.752 DEBUG:boto:Params: {} 2023-05-31T20:38:15.753 DEBUG:boto:Token: None 2023-05-31T20:38:15.754 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Wed, 31 May 2023 20:38:15 GMT /pklxkd-58/obj00000000000000000020 2023-05-31T20:38:15.755 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:WyYsHkOeXXysomeUK1AwRKpKJkQ= 2023-05-31T20:38:15.756 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Wed, 31 May 2023 20:38:15 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:WyYsHkOeXXysomeUK1AwRKpKJkQ='} 2023-05-31T20:38:15.765 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000004994874fc62e1d8c-006477b037-1630-a1'), ('Date', 'Wed, 31 May 2023 20:38:15 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:38:15.765 DEBUG:boto:path=/pklxkd-58/ 2023-05-31T20:38:15.766 DEBUG:boto:auth_path=/pklxkd-58/ 2023-05-31T20:38:15.766 DEBUG:boto:Method: HEAD 2023-05-31T20:38:15.767 DEBUG:boto:Path: /pklxkd-58/ 2023-05-31T20:38:15.767 DEBUG:boto:Data: 2023-05-31T20:38:15.767 DEBUG:boto:Headers: {} 2023-05-31T20:38:15.767 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:38:15.768 DEBUG:boto:Port: 8000 2023-05-31T20:38:15.768 DEBUG:boto:Params: {} 2023-05-31T20:38:15.769 DEBUG:boto:Token: None 2023-05-31T20:38:15.769 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:38:15 GMT /pklxkd-58/ 2023-05-31T20:38:15.770 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:zOj+u86Oolgn2P4vg/jZJGKSDpk= 2023-05-31T20:38:15.770 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:38:15 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:zOj+u86Oolgn2P4vg/jZJGKSDpk=', 'Content-Length': '0'} 2023-05-31T20:38:15.773 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '20'), ('X-RGW-Bytes-Used', '60'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000020214663d0a3e783-006477b037-1630-a1'), ('Content-Length', '0'), ('Date', 'Wed, 31 May 2023 20:38:15 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:38:15.773 DEBUG:boto:path=/pklxkd-58/obj00000000000000000010 2023-05-31T20:38:15.774 DEBUG:boto:auth_path=/pklxkd-58/obj00000000000000000010 2023-05-31T20:38:15.774 DEBUG:boto:Method: PUT 2023-05-31T20:38:15.775 DEBUG:boto:Path: /pklxkd-58/obj00000000000000000010 2023-05-31T20:38:15.775 DEBUG:boto:Data: 2023-05-31T20:38:15.775 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-05-31T20:38:15.776 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:38:15.776 DEBUG:boto:Port: 8000 2023-05-31T20:38:15.777 DEBUG:boto:Params: {} 2023-05-31T20:38:15.777 DEBUG:boto:Token: None 2023-05-31T20:38:15.778 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Wed, 31 May 2023 20:38:15 GMT /pklxkd-58/obj00000000000000000010 2023-05-31T20:38:15.778 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:xaIDOvIaPFhT5TL++KF3e5xXASM= 2023-05-31T20:38:15.779 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Wed, 31 May 2023 20:38:15 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:xaIDOvIaPFhT5TL++KF3e5xXASM='} 2023-05-31T20:38:15.820 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000007c1b729ceb973fed-006477b037-1630-a1'), ('Date', 'Wed, 31 May 2023 20:38:15 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:38:15.821 DEBUG:boto:path=/pklxkd-58/ 2023-05-31T20:38:15.822 DEBUG:boto:auth_path=/pklxkd-58/ 2023-05-31T20:38:15.823 DEBUG:boto:Method: HEAD 2023-05-31T20:38:15.823 DEBUG:boto:Path: /pklxkd-58/ 2023-05-31T20:38:15.824 DEBUG:boto:Data: 2023-05-31T20:38:15.825 DEBUG:boto:Headers: {} 2023-05-31T20:38:15.825 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:38:15.826 DEBUG:boto:Port: 8000 2023-05-31T20:38:15.826 DEBUG:boto:Params: {} 2023-05-31T20:38:15.827 DEBUG:boto:Token: None 2023-05-31T20:38:15.827 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:38:15 GMT /pklxkd-58/ 2023-05-31T20:38:15.828 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:zOj+u86Oolgn2P4vg/jZJGKSDpk= 2023-05-31T20:38:15.829 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:38:15 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:zOj+u86Oolgn2P4vg/jZJGKSDpk=', 'Content-Length': '0'} 2023-05-31T20:38:15.831 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '20'), ('X-RGW-Bytes-Used', '60'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000007c14031d1b62c68b-006477b037-1630-a1'), ('Content-Length', '0'), ('Date', 'Wed, 31 May 2023 20:38:15 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:38:15.833 DEBUG:boto:path=/pklxkd-58/obj00000000000000000004 2023-05-31T20:38:15.833 DEBUG:boto:auth_path=/pklxkd-58/obj00000000000000000004 2023-05-31T20:38:15.834 DEBUG:boto:Method: PUT 2023-05-31T20:38:15.835 DEBUG:boto:Path: /pklxkd-58/obj00000000000000000004 2023-05-31T20:38:15.835 DEBUG:boto:Data: 2023-05-31T20:38:15.836 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-05-31T20:38:15.836 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:38:15.837 DEBUG:boto:Port: 8000 2023-05-31T20:38:15.837 DEBUG:boto:Params: {} 2023-05-31T20:38:15.837 DEBUG:boto:Token: None 2023-05-31T20:38:15.838 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Wed, 31 May 2023 20:38:15 GMT /pklxkd-58/obj00000000000000000004 2023-05-31T20:38:15.839 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:jVaXvANLA10wrLAPgahX1u7/hr0= 2023-05-31T20:38:15.839 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Wed, 31 May 2023 20:38:15 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:jVaXvANLA10wrLAPgahX1u7/hr0='} 2023-05-31T20:38:15.847 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000032bdafbfbbc4c4d8-006477b037-1630-a1'), ('Date', 'Wed, 31 May 2023 20:38:15 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:38:15.848 DEBUG:boto:path=/pklxkd-58/ 2023-05-31T20:38:15.849 DEBUG:boto:auth_path=/pklxkd-58/ 2023-05-31T20:38:15.849 DEBUG:boto:Method: HEAD 2023-05-31T20:38:15.849 DEBUG:boto:Path: /pklxkd-58/ 2023-05-31T20:38:15.850 DEBUG:boto:Data: 2023-05-31T20:38:15.850 DEBUG:boto:Headers: {} 2023-05-31T20:38:15.851 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:38:15.851 DEBUG:boto:Port: 8000 2023-05-31T20:38:15.852 DEBUG:boto:Params: {} 2023-05-31T20:38:15.852 DEBUG:boto:Token: None 2023-05-31T20:38:15.853 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:38:15 GMT /pklxkd-58/ 2023-05-31T20:38:15.853 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:zOj+u86Oolgn2P4vg/jZJGKSDpk= 2023-05-31T20:38:15.854 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:38:15 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:zOj+u86Oolgn2P4vg/jZJGKSDpk=', 'Content-Length': '0'} 2023-05-31T20:38:15.856 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '20'), ('X-RGW-Bytes-Used', '60'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000005ba2f81f62153c32-006477b037-1630-a1'), ('Content-Length', '0'), ('Date', 'Wed, 31 May 2023 20:38:15 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:38:15.857 DEBUG:boto:path=/pklxkd-58/obj00000000000000000032 2023-05-31T20:38:15.858 DEBUG:boto:auth_path=/pklxkd-58/obj00000000000000000032 2023-05-31T20:38:15.858 DEBUG:boto:Method: PUT 2023-05-31T20:38:15.859 DEBUG:boto:Path: /pklxkd-58/obj00000000000000000032 2023-05-31T20:38:15.859 DEBUG:boto:Data: 2023-05-31T20:38:15.860 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-05-31T20:38:15.860 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:38:15.861 DEBUG:boto:Port: 8000 2023-05-31T20:38:15.861 DEBUG:boto:Params: {} 2023-05-31T20:38:15.862 DEBUG:boto:Token: None 2023-05-31T20:38:15.862 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Wed, 31 May 2023 20:38:15 GMT /pklxkd-58/obj00000000000000000032 2023-05-31T20:38:15.863 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:IDVhVnahHNE1x6Q2xCDfW/udbBA= 2023-05-31T20:38:15.864 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Wed, 31 May 2023 20:38:15 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:IDVhVnahHNE1x6Q2xCDfW/udbBA='} 2023-05-31T20:38:15.873 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000b8a14dd06974fa57-006477b037-1630-a1'), ('Date', 'Wed, 31 May 2023 20:38:15 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:38:15.873 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'reshard', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'pklxkd-58', '--num-shards', '19', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-05-31T20:38:15.874 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'reshard', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'pklxkd-58', '--num-shards', '19', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-05-31T20:38:15.874 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket reshard --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --bucket pklxkd-58 --num-shards 19 --yes-i-really-mean-it --cluster c1 --debug-rgw 0 --debug-ms 0 2023-05-31T20:38:15.927 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2023-05-31T20:38:15.928 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2023-05-31T20:38:15.982 INFO:teuthology.orchestra.run.smithi099.stdout:tenant: 2023-05-31T20:38:15.982 INFO:teuthology.orchestra.run.smithi099.stdout:bucket name: pklxkd-58 2023-05-31T20:38:15.989 INFO:teuthology.orchestra.run.smithi099.stdout:total entries: 21 2023-05-31T20:38:16.061 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'shard', 'objects', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--num-shards', '19', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-31T20:38:16.062 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'shard', 'objects', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--num-shards', '19', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-31T20:38:16.063 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket shard objects --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --num-shards 19 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-05-31T20:38:16.116 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2023-05-31T20:38:16.116 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2023-05-31T20:38:16.192 INFO:teuthology.orchestra.run.smithi099.stdout:{ 2023-05-31T20:38:16.192 INFO:teuthology.orchestra.run.smithi099.stdout: "objs": [ 2023-05-31T20:38:16.192 INFO:teuthology.orchestra.run.smithi099.stdout: "obj00000000000000000005", 2023-05-31T20:38:16.193 INFO:teuthology.orchestra.run.smithi099.stdout: "obj00000000000000000010", 2023-05-31T20:38:16.193 INFO:teuthology.orchestra.run.smithi099.stdout: "obj00000000000000000016", 2023-05-31T20:38:16.193 INFO:teuthology.orchestra.run.smithi099.stdout: "obj00000000000000000021", 2023-05-31T20:38:16.193 INFO:teuthology.orchestra.run.smithi099.stdout: "obj00000000000000000004", 2023-05-31T20:38:16.193 INFO:teuthology.orchestra.run.smithi099.stdout: "obj00000000000000000025", 2023-05-31T20:38:16.194 INFO:teuthology.orchestra.run.smithi099.stdout: "obj00000000000000000030", 2023-05-31T20:38:16.194 INFO:teuthology.orchestra.run.smithi099.stdout: "obj00000000000000000015", 2023-05-31T20:38:16.194 INFO:teuthology.orchestra.run.smithi099.stdout: "obj00000000000000000019", 2023-05-31T20:38:16.194 INFO:teuthology.orchestra.run.smithi099.stdout: "obj00000000000000000003", 2023-05-31T20:38:16.194 INFO:teuthology.orchestra.run.smithi099.stdout: "obj00000000000000000009", 2023-05-31T20:38:16.194 INFO:teuthology.orchestra.run.smithi099.stdout: "obj00000000000000000014", 2023-05-31T20:38:16.195 INFO:teuthology.orchestra.run.smithi099.stdout: "obj00000000000000000012", 2023-05-31T20:38:16.195 INFO:teuthology.orchestra.run.smithi099.stdout: "obj00000000000000000002", 2023-05-31T20:38:16.195 INFO:teuthology.orchestra.run.smithi099.stdout: "obj00000000000000000000", 2023-05-31T20:38:16.195 INFO:teuthology.orchestra.run.smithi099.stdout: "obj00000000000000000006", 2023-05-31T20:38:16.195 INFO:teuthology.orchestra.run.smithi099.stdout: "obj00000000000000000011", 2023-05-31T20:38:16.195 INFO:teuthology.orchestra.run.smithi099.stdout: "obj00000000000000000001", 2023-05-31T20:38:16.196 INFO:teuthology.orchestra.run.smithi099.stdout: "obj00000000000000000007" 2023-05-31T20:38:16.196 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2023-05-31T20:38:16.196 INFO:teuthology.orchestra.run.smithi099.stdout:} 2023-05-31T20:38:16.197 DEBUG:boto:path=/pklxkd-58/ 2023-05-31T20:38:16.198 DEBUG:boto:auth_path=/pklxkd-58/ 2023-05-31T20:38:16.199 DEBUG:boto:Method: HEAD 2023-05-31T20:38:16.200 DEBUG:boto:Path: /pklxkd-58/ 2023-05-31T20:38:16.201 DEBUG:boto:Data: 2023-05-31T20:38:16.202 DEBUG:boto:Headers: {} 2023-05-31T20:38:16.203 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:38:16.204 DEBUG:boto:Port: 8000 2023-05-31T20:38:16.205 DEBUG:boto:Params: {} 2023-05-31T20:38:16.206 DEBUG:boto:Token: None 2023-05-31T20:38:16.206 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:38:16 GMT /pklxkd-58/ 2023-05-31T20:38:16.207 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:CQRtozsOi7RN4Myv9sdbVJvzIYc= 2023-05-31T20:38:16.208 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:38:16 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:CQRtozsOi7RN4Myv9sdbVJvzIYc=', 'Content-Length': '0'} 2023-05-31T20:38:16.211 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '21'), ('X-RGW-Bytes-Used', '63'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000005f535089304d0c9f-006477b038-1630-a1'), ('Content-Length', '0'), ('Date', 'Wed, 31 May 2023 20:38:16 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:38:16.212 DEBUG:boto:path=/pklxkd-58/obj00000000000000000011 2023-05-31T20:38:16.213 DEBUG:boto:auth_path=/pklxkd-58/obj00000000000000000011 2023-05-31T20:38:16.213 DEBUG:boto:Method: PUT 2023-05-31T20:38:16.214 DEBUG:boto:Path: /pklxkd-58/obj00000000000000000011 2023-05-31T20:38:16.214 DEBUG:boto:Data: 2023-05-31T20:38:16.215 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-05-31T20:38:16.215 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:38:16.215 DEBUG:boto:Port: 8000 2023-05-31T20:38:16.216 DEBUG:boto:Params: {} 2023-05-31T20:38:16.216 DEBUG:boto:Token: None 2023-05-31T20:38:16.217 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Wed, 31 May 2023 20:38:16 GMT /pklxkd-58/obj00000000000000000011 2023-05-31T20:38:16.218 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:J3WmFTWUjhNsZGrmPEG3o5VkBlc= 2023-05-31T20:38:16.218 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Wed, 31 May 2023 20:38:16 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:J3WmFTWUjhNsZGrmPEG3o5VkBlc='} 2023-05-31T20:38:16.226 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000b3918cae6312bb52-006477b038-1630-a1'), ('Date', 'Wed, 31 May 2023 20:38:16 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:38:16.227 DEBUG:boto:path=/pklxkd-58/ 2023-05-31T20:38:16.227 DEBUG:boto:auth_path=/pklxkd-58/ 2023-05-31T20:38:16.228 DEBUG:boto:Method: HEAD 2023-05-31T20:38:16.228 DEBUG:boto:Path: /pklxkd-58/ 2023-05-31T20:38:16.228 DEBUG:boto:Data: 2023-05-31T20:38:16.229 DEBUG:boto:Headers: {} 2023-05-31T20:38:16.229 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:38:16.230 DEBUG:boto:Port: 8000 2023-05-31T20:38:16.230 DEBUG:boto:Params: {} 2023-05-31T20:38:16.231 DEBUG:boto:Token: None 2023-05-31T20:38:16.231 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:38:16 GMT /pklxkd-58/ 2023-05-31T20:38:16.231 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:CQRtozsOi7RN4Myv9sdbVJvzIYc= 2023-05-31T20:38:16.232 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:38:16 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:CQRtozsOi7RN4Myv9sdbVJvzIYc=', 'Content-Length': '0'} 2023-05-31T20:38:16.235 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '21'), ('X-RGW-Bytes-Used', '63'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000034db8aee9f8eedf4-006477b038-1630-a1'), ('Content-Length', '0'), ('Date', 'Wed, 31 May 2023 20:38:16 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:38:16.236 DEBUG:boto:path=/pklxkd-58/obj00000000000000000015 2023-05-31T20:38:16.236 DEBUG:boto:auth_path=/pklxkd-58/obj00000000000000000015 2023-05-31T20:38:16.236 DEBUG:boto:Method: PUT 2023-05-31T20:38:16.237 DEBUG:boto:Path: /pklxkd-58/obj00000000000000000015 2023-05-31T20:38:16.237 DEBUG:boto:Data: 2023-05-31T20:38:16.238 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-05-31T20:38:16.238 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:38:16.238 DEBUG:boto:Port: 8000 2023-05-31T20:38:16.239 DEBUG:boto:Params: {} 2023-05-31T20:38:16.239 DEBUG:boto:Token: None 2023-05-31T20:38:16.240 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Wed, 31 May 2023 20:38:16 GMT /pklxkd-58/obj00000000000000000015 2023-05-31T20:38:16.240 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:If/mWCtW3qIObp/ckmW1RmALpm4= 2023-05-31T20:38:16.241 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Wed, 31 May 2023 20:38:16 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:If/mWCtW3qIObp/ckmW1RmALpm4='} 2023-05-31T20:38:16.284 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000066bb547f0e651b54-006477b038-1630-a1'), ('Date', 'Wed, 31 May 2023 20:38:16 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:38:16.285 DEBUG:boto:path=/pklxkd-58/ 2023-05-31T20:38:16.285 DEBUG:boto:auth_path=/pklxkd-58/ 2023-05-31T20:38:16.286 DEBUG:boto:Method: HEAD 2023-05-31T20:38:16.286 DEBUG:boto:Path: /pklxkd-58/ 2023-05-31T20:38:16.286 DEBUG:boto:Data: 2023-05-31T20:38:16.287 DEBUG:boto:Headers: {} 2023-05-31T20:38:16.287 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:38:16.288 DEBUG:boto:Port: 8000 2023-05-31T20:38:16.288 DEBUG:boto:Params: {} 2023-05-31T20:38:16.288 DEBUG:boto:Token: None 2023-05-31T20:38:16.289 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:38:16 GMT /pklxkd-58/ 2023-05-31T20:38:16.289 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:CQRtozsOi7RN4Myv9sdbVJvzIYc= 2023-05-31T20:38:16.290 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:38:16 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:CQRtozsOi7RN4Myv9sdbVJvzIYc=', 'Content-Length': '0'} 2023-05-31T20:38:16.292 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '22'), ('X-RGW-Bytes-Used', '66'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000a3bd664ec01c7355-006477b038-1630-a1'), ('Content-Length', '0'), ('Date', 'Wed, 31 May 2023 20:38:16 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:38:16.293 DEBUG:boto:path=/pklxkd-58/obj00000000000000000021 2023-05-31T20:38:16.294 DEBUG:boto:auth_path=/pklxkd-58/obj00000000000000000021 2023-05-31T20:38:16.294 DEBUG:boto:Method: PUT 2023-05-31T20:38:16.294 DEBUG:boto:Path: /pklxkd-58/obj00000000000000000021 2023-05-31T20:38:16.295 DEBUG:boto:Data: 2023-05-31T20:38:16.296 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-05-31T20:38:16.296 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:38:16.297 DEBUG:boto:Port: 8000 2023-05-31T20:38:16.297 DEBUG:boto:Params: {} 2023-05-31T20:38:16.298 DEBUG:boto:Token: None 2023-05-31T20:38:16.298 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Wed, 31 May 2023 20:38:16 GMT /pklxkd-58/obj00000000000000000021 2023-05-31T20:38:16.299 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:ycEfFhgZnVCyC9hLwJIIRkaWZ5M= 2023-05-31T20:38:16.299 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Wed, 31 May 2023 20:38:16 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:ycEfFhgZnVCyC9hLwJIIRkaWZ5M='} 2023-05-31T20:38:16.305 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000003282d5890d178bb5-006477b038-1630-a1'), ('Date', 'Wed, 31 May 2023 20:38:16 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:38:16.306 DEBUG:boto:path=/pklxkd-58/ 2023-05-31T20:38:16.307 DEBUG:boto:auth_path=/pklxkd-58/ 2023-05-31T20:38:16.307 DEBUG:boto:Method: HEAD 2023-05-31T20:38:16.308 DEBUG:boto:Path: /pklxkd-58/ 2023-05-31T20:38:16.308 DEBUG:boto:Data: 2023-05-31T20:38:16.308 DEBUG:boto:Headers: {} 2023-05-31T20:38:16.309 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:38:16.309 DEBUG:boto:Port: 8000 2023-05-31T20:38:16.310 DEBUG:boto:Params: {} 2023-05-31T20:38:16.310 DEBUG:boto:Token: None 2023-05-31T20:38:16.310 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:38:16 GMT /pklxkd-58/ 2023-05-31T20:38:16.311 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:CQRtozsOi7RN4Myv9sdbVJvzIYc= 2023-05-31T20:38:16.311 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:38:16 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:CQRtozsOi7RN4Myv9sdbVJvzIYc=', 'Content-Length': '0'} 2023-05-31T20:38:16.314 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '22'), ('X-RGW-Bytes-Used', '66'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000004a761de7f773b009-006477b038-1630-a1'), ('Content-Length', '0'), ('Date', 'Wed, 31 May 2023 20:38:16 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:38:16.314 DEBUG:boto:path=/pklxkd-58/obj00000000000000000030 2023-05-31T20:38:16.315 DEBUG:boto:auth_path=/pklxkd-58/obj00000000000000000030 2023-05-31T20:38:16.315 DEBUG:boto:Method: PUT 2023-05-31T20:38:16.316 DEBUG:boto:Path: /pklxkd-58/obj00000000000000000030 2023-05-31T20:38:16.316 DEBUG:boto:Data: 2023-05-31T20:38:16.316 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-05-31T20:38:16.317 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:38:16.317 DEBUG:boto:Port: 8000 2023-05-31T20:38:16.318 DEBUG:boto:Params: {} 2023-05-31T20:38:16.318 DEBUG:boto:Token: None 2023-05-31T20:38:16.319 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Wed, 31 May 2023 20:38:16 GMT /pklxkd-58/obj00000000000000000030 2023-05-31T20:38:16.319 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:tUMh8p+DW6YBD9c6gGY5SA5+Rco= 2023-05-31T20:38:16.320 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Wed, 31 May 2023 20:38:16 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:tUMh8p+DW6YBD9c6gGY5SA5+Rco='} 2023-05-31T20:38:16.364 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000007940e650429f8963-006477b038-1630-a1'), ('Date', 'Wed, 31 May 2023 20:38:16 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:38:16.366 DEBUG:boto:path=/pklxkd-58/ 2023-05-31T20:38:16.366 DEBUG:boto:auth_path=/pklxkd-58/ 2023-05-31T20:38:16.367 DEBUG:boto:Method: HEAD 2023-05-31T20:38:16.368 DEBUG:boto:Path: /pklxkd-58/ 2023-05-31T20:38:16.368 DEBUG:boto:Data: 2023-05-31T20:38:16.369 DEBUG:boto:Headers: {} 2023-05-31T20:38:16.369 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:38:16.370 DEBUG:boto:Port: 8000 2023-05-31T20:38:16.370 DEBUG:boto:Params: {} 2023-05-31T20:38:16.370 DEBUG:boto:Token: None 2023-05-31T20:38:16.371 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:38:16 GMT /pklxkd-58/ 2023-05-31T20:38:16.371 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:CQRtozsOi7RN4Myv9sdbVJvzIYc= 2023-05-31T20:38:16.372 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:38:16 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:CQRtozsOi7RN4Myv9sdbVJvzIYc=', 'Content-Length': '0'} 2023-05-31T20:38:16.374 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '22'), ('X-RGW-Bytes-Used', '66'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000054e2c20712d708b0-006477b038-1630-a1'), ('Content-Length', '0'), ('Date', 'Wed, 31 May 2023 20:38:16 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:38:16.375 DEBUG:boto:path=/pklxkd-58/obj00000000000000000012 2023-05-31T20:38:16.375 DEBUG:boto:auth_path=/pklxkd-58/obj00000000000000000012 2023-05-31T20:38:16.375 DEBUG:boto:Method: PUT 2023-05-31T20:38:16.376 DEBUG:boto:Path: /pklxkd-58/obj00000000000000000012 2023-05-31T20:38:16.376 DEBUG:boto:Data: 2023-05-31T20:38:16.376 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-05-31T20:38:16.377 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:38:16.377 DEBUG:boto:Port: 8000 2023-05-31T20:38:16.378 DEBUG:boto:Params: {} 2023-05-31T20:38:16.378 DEBUG:boto:Token: None 2023-05-31T20:38:16.378 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Wed, 31 May 2023 20:38:16 GMT /pklxkd-58/obj00000000000000000012 2023-05-31T20:38:16.379 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:qURgO3toBenROlOAAHDDnAKUJQ0= 2023-05-31T20:38:16.379 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Wed, 31 May 2023 20:38:16 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:qURgO3toBenROlOAAHDDnAKUJQ0='} 2023-05-31T20:38:16.385 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000007604455accbb5e9a-006477b038-1630-a1'), ('Date', 'Wed, 31 May 2023 20:38:16 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:38:16.386 DEBUG:boto:path=/pklxkd-58/ 2023-05-31T20:38:16.386 DEBUG:boto:auth_path=/pklxkd-58/ 2023-05-31T20:38:16.387 DEBUG:boto:Method: HEAD 2023-05-31T20:38:16.387 DEBUG:boto:Path: /pklxkd-58/ 2023-05-31T20:38:16.387 DEBUG:boto:Data: 2023-05-31T20:38:16.388 DEBUG:boto:Headers: {} 2023-05-31T20:38:16.388 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:38:16.389 DEBUG:boto:Port: 8000 2023-05-31T20:38:16.389 DEBUG:boto:Params: {} 2023-05-31T20:38:16.389 DEBUG:boto:Token: None 2023-05-31T20:38:16.390 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:38:16 GMT /pklxkd-58/ 2023-05-31T20:38:16.390 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:CQRtozsOi7RN4Myv9sdbVJvzIYc= 2023-05-31T20:38:16.390 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:38:16 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:CQRtozsOi7RN4Myv9sdbVJvzIYc=', 'Content-Length': '0'} 2023-05-31T20:38:16.393 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '22'), ('X-RGW-Bytes-Used', '66'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000801be81ccfd9fcaf-006477b038-1630-a1'), ('Content-Length', '0'), ('Date', 'Wed, 31 May 2023 20:38:16 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:38:16.394 DEBUG:boto:path=/pklxkd-58/obj00000000000000000001 2023-05-31T20:38:16.394 DEBUG:boto:auth_path=/pklxkd-58/obj00000000000000000001 2023-05-31T20:38:16.394 DEBUG:boto:Method: PUT 2023-05-31T20:38:16.395 DEBUG:boto:Path: /pklxkd-58/obj00000000000000000001 2023-05-31T20:38:16.395 DEBUG:boto:Data: 2023-05-31T20:38:16.395 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-05-31T20:38:16.396 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:38:16.396 DEBUG:boto:Port: 8000 2023-05-31T20:38:16.396 DEBUG:boto:Params: {} 2023-05-31T20:38:16.397 DEBUG:boto:Token: None 2023-05-31T20:38:16.398 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Wed, 31 May 2023 20:38:16 GMT /pklxkd-58/obj00000000000000000001 2023-05-31T20:38:16.398 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:isTeMXve/VCaB/s2+EuUYbDSOTA= 2023-05-31T20:38:16.399 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Wed, 31 May 2023 20:38:16 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:isTeMXve/VCaB/s2+EuUYbDSOTA='} 2023-05-31T20:38:16.444 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000394718ea704df155-006477b038-1630-a1'), ('Date', 'Wed, 31 May 2023 20:38:16 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:38:16.445 DEBUG:boto:path=/pklxkd-58/ 2023-05-31T20:38:16.445 DEBUG:boto:auth_path=/pklxkd-58/ 2023-05-31T20:38:16.446 DEBUG:boto:Method: HEAD 2023-05-31T20:38:16.446 DEBUG:boto:Path: /pklxkd-58/ 2023-05-31T20:38:16.446 DEBUG:boto:Data: 2023-05-31T20:38:16.447 DEBUG:boto:Headers: {} 2023-05-31T20:38:16.447 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:38:16.447 DEBUG:boto:Port: 8000 2023-05-31T20:38:16.448 DEBUG:boto:Params: {} 2023-05-31T20:38:16.448 DEBUG:boto:Token: None 2023-05-31T20:38:16.449 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:38:16 GMT /pklxkd-58/ 2023-05-31T20:38:16.449 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:CQRtozsOi7RN4Myv9sdbVJvzIYc= 2023-05-31T20:38:16.449 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:38:16 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:CQRtozsOi7RN4Myv9sdbVJvzIYc=', 'Content-Length': '0'} 2023-05-31T20:38:16.452 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '22'), ('X-RGW-Bytes-Used', '66'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000076b51f9e7f8b0152-006477b038-1630-a1'), ('Content-Length', '0'), ('Date', 'Wed, 31 May 2023 20:38:16 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:38:16.452 DEBUG:boto:path=/pklxkd-58/obj00000000000000000010 2023-05-31T20:38:16.453 DEBUG:boto:auth_path=/pklxkd-58/obj00000000000000000010 2023-05-31T20:38:16.453 DEBUG:boto:Method: PUT 2023-05-31T20:38:16.454 DEBUG:boto:Path: /pklxkd-58/obj00000000000000000010 2023-05-31T20:38:16.454 DEBUG:boto:Data: 2023-05-31T20:38:16.454 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-05-31T20:38:16.455 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:38:16.455 DEBUG:boto:Port: 8000 2023-05-31T20:38:16.455 DEBUG:boto:Params: {} 2023-05-31T20:38:16.456 DEBUG:boto:Token: None 2023-05-31T20:38:16.456 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Wed, 31 May 2023 20:38:16 GMT /pklxkd-58/obj00000000000000000010 2023-05-31T20:38:16.457 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:Gpt5tHd8HgXuvq2ScfoYn+XZnGQ= 2023-05-31T20:38:16.457 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Wed, 31 May 2023 20:38:16 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:Gpt5tHd8HgXuvq2ScfoYn+XZnGQ='} 2023-05-31T20:38:16.466 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000061ede7d50c68af43-006477b038-1630-a1'), ('Date', 'Wed, 31 May 2023 20:38:16 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:38:16.466 DEBUG:boto:path=/pklxkd-58/ 2023-05-31T20:38:16.467 DEBUG:boto:auth_path=/pklxkd-58/ 2023-05-31T20:38:16.467 DEBUG:boto:Method: HEAD 2023-05-31T20:38:16.468 DEBUG:boto:Path: /pklxkd-58/ 2023-05-31T20:38:16.468 DEBUG:boto:Data: 2023-05-31T20:38:16.469 DEBUG:boto:Headers: {} 2023-05-31T20:38:16.469 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:38:16.469 DEBUG:boto:Port: 8000 2023-05-31T20:38:16.470 DEBUG:boto:Params: {} 2023-05-31T20:38:16.470 DEBUG:boto:Token: None 2023-05-31T20:38:16.471 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:38:16 GMT /pklxkd-58/ 2023-05-31T20:38:16.472 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:CQRtozsOi7RN4Myv9sdbVJvzIYc= 2023-05-31T20:38:16.472 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:38:16 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:CQRtozsOi7RN4Myv9sdbVJvzIYc=', 'Content-Length': '0'} 2023-05-31T20:38:16.475 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '22'), ('X-RGW-Bytes-Used', '66'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000ad273a56a033df4d-006477b038-1630-a1'), ('Content-Length', '0'), ('Date', 'Wed, 31 May 2023 20:38:16 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:38:16.476 DEBUG:boto:path=/pklxkd-58/obj00000000000000000019 2023-05-31T20:38:16.477 DEBUG:boto:auth_path=/pklxkd-58/obj00000000000000000019 2023-05-31T20:38:16.477 DEBUG:boto:Method: PUT 2023-05-31T20:38:16.478 DEBUG:boto:Path: /pklxkd-58/obj00000000000000000019 2023-05-31T20:38:16.479 DEBUG:boto:Data: 2023-05-31T20:38:16.479 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-05-31T20:38:16.480 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:38:16.480 DEBUG:boto:Port: 8000 2023-05-31T20:38:16.481 DEBUG:boto:Params: {} 2023-05-31T20:38:16.482 DEBUG:boto:Token: None 2023-05-31T20:38:16.482 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Wed, 31 May 2023 20:38:16 GMT /pklxkd-58/obj00000000000000000019 2023-05-31T20:38:16.483 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:fonzi29DY2G5+jKtIqNJPu3XVG8= 2023-05-31T20:38:16.484 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Wed, 31 May 2023 20:38:16 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:fonzi29DY2G5+jKtIqNJPu3XVG8='} 2023-05-31T20:38:16.490 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000003184832d9f6ee9bd-006477b038-1630-a1'), ('Date', 'Wed, 31 May 2023 20:38:16 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:38:16.491 DEBUG:boto:path=/pklxkd-58/ 2023-05-31T20:38:16.491 DEBUG:boto:auth_path=/pklxkd-58/ 2023-05-31T20:38:16.491 DEBUG:boto:Method: HEAD 2023-05-31T20:38:16.492 DEBUG:boto:Path: /pklxkd-58/ 2023-05-31T20:38:16.493 DEBUG:boto:Data: 2023-05-31T20:38:16.493 DEBUG:boto:Headers: {} 2023-05-31T20:38:16.494 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:38:16.494 DEBUG:boto:Port: 8000 2023-05-31T20:38:16.494 DEBUG:boto:Params: {} 2023-05-31T20:38:16.495 DEBUG:boto:Token: None 2023-05-31T20:38:16.495 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:38:16 GMT /pklxkd-58/ 2023-05-31T20:38:16.495 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:CQRtozsOi7RN4Myv9sdbVJvzIYc= 2023-05-31T20:38:16.496 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:38:16 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:CQRtozsOi7RN4Myv9sdbVJvzIYc=', 'Content-Length': '0'} 2023-05-31T20:38:16.499 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '22'), ('X-RGW-Bytes-Used', '66'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000bcee8c9a2eea8325-006477b038-1630-a1'), ('Content-Length', '0'), ('Date', 'Wed, 31 May 2023 20:38:16 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:38:16.499 DEBUG:boto:path=/pklxkd-58/obj00000000000000000025 2023-05-31T20:38:16.500 DEBUG:boto:auth_path=/pklxkd-58/obj00000000000000000025 2023-05-31T20:38:16.500 DEBUG:boto:Method: PUT 2023-05-31T20:38:16.501 DEBUG:boto:Path: /pklxkd-58/obj00000000000000000025 2023-05-31T20:38:16.501 DEBUG:boto:Data: 2023-05-31T20:38:16.502 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-05-31T20:38:16.503 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:38:16.504 DEBUG:boto:Port: 8000 2023-05-31T20:38:16.504 DEBUG:boto:Params: {} 2023-05-31T20:38:16.505 DEBUG:boto:Token: None 2023-05-31T20:38:16.506 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Wed, 31 May 2023 20:38:16 GMT /pklxkd-58/obj00000000000000000025 2023-05-31T20:38:16.506 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:PQgKe3pWrD4l0qzHIYaql4TiFaI= 2023-05-31T20:38:16.507 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Wed, 31 May 2023 20:38:16 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:PQgKe3pWrD4l0qzHIYaql4TiFaI='} 2023-05-31T20:38:16.557 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000fcdd8a2d75c30f29-006477b038-1630-a1'), ('Date', 'Wed, 31 May 2023 20:38:16 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:38:16.558 DEBUG:boto:path=/pklxkd-58/ 2023-05-31T20:38:16.558 DEBUG:boto:auth_path=/pklxkd-58/ 2023-05-31T20:38:16.559 DEBUG:boto:Method: HEAD 2023-05-31T20:38:16.559 DEBUG:boto:Path: /pklxkd-58/ 2023-05-31T20:38:16.559 DEBUG:boto:Data: 2023-05-31T20:38:16.560 DEBUG:boto:Headers: {} 2023-05-31T20:38:16.560 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:38:16.561 DEBUG:boto:Port: 8000 2023-05-31T20:38:16.561 DEBUG:boto:Params: {} 2023-05-31T20:38:16.561 DEBUG:boto:Token: None 2023-05-31T20:38:16.562 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:38:16 GMT /pklxkd-58/ 2023-05-31T20:38:16.563 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:CQRtozsOi7RN4Myv9sdbVJvzIYc= 2023-05-31T20:38:16.563 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:38:16 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:CQRtozsOi7RN4Myv9sdbVJvzIYc=', 'Content-Length': '0'} 2023-05-31T20:38:16.566 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '23'), ('X-RGW-Bytes-Used', '69'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000005b1d79856bb5d525-006477b038-1630-a1'), ('Content-Length', '0'), ('Date', 'Wed, 31 May 2023 20:38:16 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:38:16.567 DEBUG:boto:path=/pklxkd-58/obj00000000000000000016 2023-05-31T20:38:16.568 DEBUG:boto:auth_path=/pklxkd-58/obj00000000000000000016 2023-05-31T20:38:16.569 DEBUG:boto:Method: PUT 2023-05-31T20:38:16.570 DEBUG:boto:Path: /pklxkd-58/obj00000000000000000016 2023-05-31T20:38:16.571 DEBUG:boto:Data: 2023-05-31T20:38:16.572 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-05-31T20:38:16.572 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:38:16.573 DEBUG:boto:Port: 8000 2023-05-31T20:38:16.573 DEBUG:boto:Params: {} 2023-05-31T20:38:16.574 DEBUG:boto:Token: None 2023-05-31T20:38:16.574 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Wed, 31 May 2023 20:38:16 GMT /pklxkd-58/obj00000000000000000016 2023-05-31T20:38:16.575 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:ik4AZ5nNYCwMrC+hKD6q0pwQIqc= 2023-05-31T20:38:16.576 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Wed, 31 May 2023 20:38:16 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:ik4AZ5nNYCwMrC+hKD6q0pwQIqc='} 2023-05-31T20:38:16.586 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000026b625c565fba698-006477b038-1630-a1'), ('Date', 'Wed, 31 May 2023 20:38:16 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:38:16.587 DEBUG:boto:path=/pklxkd-58/ 2023-05-31T20:38:16.588 DEBUG:boto:auth_path=/pklxkd-58/ 2023-05-31T20:38:16.588 DEBUG:boto:Method: HEAD 2023-05-31T20:38:16.588 DEBUG:boto:Path: /pklxkd-58/ 2023-05-31T20:38:16.589 DEBUG:boto:Data: 2023-05-31T20:38:16.589 DEBUG:boto:Headers: {} 2023-05-31T20:38:16.589 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:38:16.590 DEBUG:boto:Port: 8000 2023-05-31T20:38:16.590 DEBUG:boto:Params: {} 2023-05-31T20:38:16.591 DEBUG:boto:Token: None 2023-05-31T20:38:16.591 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:38:16 GMT /pklxkd-58/ 2023-05-31T20:38:16.591 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:CQRtozsOi7RN4Myv9sdbVJvzIYc= 2023-05-31T20:38:16.592 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:38:16 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:CQRtozsOi7RN4Myv9sdbVJvzIYc=', 'Content-Length': '0'} 2023-05-31T20:38:16.594 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '23'), ('X-RGW-Bytes-Used', '69'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000083bb897b8b9242dd-006477b038-1630-a1'), ('Content-Length', '0'), ('Date', 'Wed, 31 May 2023 20:38:16 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:38:16.595 DEBUG:boto:path=/pklxkd-58/obj00000000000000000005 2023-05-31T20:38:16.595 DEBUG:boto:auth_path=/pklxkd-58/obj00000000000000000005 2023-05-31T20:38:16.596 DEBUG:boto:Method: PUT 2023-05-31T20:38:16.596 DEBUG:boto:Path: /pklxkd-58/obj00000000000000000005 2023-05-31T20:38:16.596 DEBUG:boto:Data: 2023-05-31T20:38:16.597 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-05-31T20:38:16.597 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:38:16.597 DEBUG:boto:Port: 8000 2023-05-31T20:38:16.598 DEBUG:boto:Params: {} 2023-05-31T20:38:16.598 DEBUG:boto:Token: None 2023-05-31T20:38:16.599 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Wed, 31 May 2023 20:38:16 GMT /pklxkd-58/obj00000000000000000005 2023-05-31T20:38:16.599 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:8KD6px6hWd22lLZOmiLPvasi5GE= 2023-05-31T20:38:16.599 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Wed, 31 May 2023 20:38:16 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:8KD6px6hWd22lLZOmiLPvasi5GE='} 2023-05-31T20:38:16.644 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000002513a57bc55e7d1-006477b038-1630-a1'), ('Date', 'Wed, 31 May 2023 20:38:16 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:38:16.645 DEBUG:boto:path=/pklxkd-58/ 2023-05-31T20:38:16.645 DEBUG:boto:auth_path=/pklxkd-58/ 2023-05-31T20:38:16.646 DEBUG:boto:Method: HEAD 2023-05-31T20:38:16.646 DEBUG:boto:Path: /pklxkd-58/ 2023-05-31T20:38:16.646 DEBUG:boto:Data: 2023-05-31T20:38:16.647 DEBUG:boto:Headers: {} 2023-05-31T20:38:16.647 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:38:16.647 DEBUG:boto:Port: 8000 2023-05-31T20:38:16.648 DEBUG:boto:Params: {} 2023-05-31T20:38:16.648 DEBUG:boto:Token: None 2023-05-31T20:38:16.649 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:38:16 GMT /pklxkd-58/ 2023-05-31T20:38:16.649 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:CQRtozsOi7RN4Myv9sdbVJvzIYc= 2023-05-31T20:38:16.649 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:38:16 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:CQRtozsOi7RN4Myv9sdbVJvzIYc=', 'Content-Length': '0'} 2023-05-31T20:38:16.652 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '24'), ('X-RGW-Bytes-Used', '72'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000001adf396a7ee8757f-006477b038-1630-a1'), ('Content-Length', '0'), ('Date', 'Wed, 31 May 2023 20:38:16 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:38:16.653 DEBUG:boto:path=/pklxkd-58/obj00000000000000000009 2023-05-31T20:38:16.653 DEBUG:boto:auth_path=/pklxkd-58/obj00000000000000000009 2023-05-31T20:38:16.653 DEBUG:boto:Method: PUT 2023-05-31T20:38:16.654 DEBUG:boto:Path: /pklxkd-58/obj00000000000000000009 2023-05-31T20:38:16.654 DEBUG:boto:Data: 2023-05-31T20:38:16.654 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-05-31T20:38:16.655 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:38:16.655 DEBUG:boto:Port: 8000 2023-05-31T20:38:16.656 DEBUG:boto:Params: {} 2023-05-31T20:38:16.656 DEBUG:boto:Token: None 2023-05-31T20:38:16.657 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Wed, 31 May 2023 20:38:16 GMT /pklxkd-58/obj00000000000000000009 2023-05-31T20:38:16.657 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:f4h8s2kqlz+vUpjRasYn27rk64s= 2023-05-31T20:38:16.657 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Wed, 31 May 2023 20:38:16 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:f4h8s2kqlz+vUpjRasYn27rk64s='} 2023-05-31T20:38:16.668 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000006800d5b47d9ad699-006477b038-1630-a1'), ('Date', 'Wed, 31 May 2023 20:38:16 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:38:16.669 DEBUG:boto:path=/pklxkd-58/ 2023-05-31T20:38:16.670 DEBUG:boto:auth_path=/pklxkd-58/ 2023-05-31T20:38:16.670 DEBUG:boto:Method: HEAD 2023-05-31T20:38:16.670 DEBUG:boto:Path: /pklxkd-58/ 2023-05-31T20:38:16.671 DEBUG:boto:Data: 2023-05-31T20:38:16.671 DEBUG:boto:Headers: {} 2023-05-31T20:38:16.671 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:38:16.672 DEBUG:boto:Port: 8000 2023-05-31T20:38:16.672 DEBUG:boto:Params: {} 2023-05-31T20:38:16.673 DEBUG:boto:Token: None 2023-05-31T20:38:16.673 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:38:16 GMT /pklxkd-58/ 2023-05-31T20:38:16.674 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:CQRtozsOi7RN4Myv9sdbVJvzIYc= 2023-05-31T20:38:16.674 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:38:16 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:CQRtozsOi7RN4Myv9sdbVJvzIYc=', 'Content-Length': '0'} 2023-05-31T20:38:16.677 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '25'), ('X-RGW-Bytes-Used', '75'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000840d41b87478bbd3-006477b038-1630-a1'), ('Content-Length', '0'), ('Date', 'Wed, 31 May 2023 20:38:16 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:38:16.677 DEBUG:boto:path=/pklxkd-58/obj00000000000000000003 2023-05-31T20:38:16.678 DEBUG:boto:auth_path=/pklxkd-58/obj00000000000000000003 2023-05-31T20:38:16.679 DEBUG:boto:Method: PUT 2023-05-31T20:38:16.679 DEBUG:boto:Path: /pklxkd-58/obj00000000000000000003 2023-05-31T20:38:16.679 DEBUG:boto:Data: 2023-05-31T20:38:16.680 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-05-31T20:38:16.680 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:38:16.680 DEBUG:boto:Port: 8000 2023-05-31T20:38:16.681 DEBUG:boto:Params: {} 2023-05-31T20:38:16.681 DEBUG:boto:Token: None 2023-05-31T20:38:16.682 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Wed, 31 May 2023 20:38:16 GMT /pklxkd-58/obj00000000000000000003 2023-05-31T20:38:16.682 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:HlOGS1NdMNhkKaBF0C63nGEkAu4= 2023-05-31T20:38:16.682 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Wed, 31 May 2023 20:38:16 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:HlOGS1NdMNhkKaBF0C63nGEkAu4='} 2023-05-31T20:38:16.728 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000002679c90bf115ec3b-006477b038-1630-a1'), ('Date', 'Wed, 31 May 2023 20:38:16 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:38:16.729 DEBUG:boto:path=/pklxkd-58/ 2023-05-31T20:38:16.730 DEBUG:boto:auth_path=/pklxkd-58/ 2023-05-31T20:38:16.730 DEBUG:boto:Method: HEAD 2023-05-31T20:38:16.730 DEBUG:boto:Path: /pklxkd-58/ 2023-05-31T20:38:16.731 DEBUG:boto:Data: 2023-05-31T20:38:16.731 DEBUG:boto:Headers: {} 2023-05-31T20:38:16.731 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:38:16.732 DEBUG:boto:Port: 8000 2023-05-31T20:38:16.732 DEBUG:boto:Params: {} 2023-05-31T20:38:16.732 DEBUG:boto:Token: None 2023-05-31T20:38:16.733 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:38:16 GMT /pklxkd-58/ 2023-05-31T20:38:16.733 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:CQRtozsOi7RN4Myv9sdbVJvzIYc= 2023-05-31T20:38:16.734 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:38:16 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:CQRtozsOi7RN4Myv9sdbVJvzIYc=', 'Content-Length': '0'} 2023-05-31T20:38:16.737 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '25'), ('X-RGW-Bytes-Used', '75'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000003f29893c033c236d-006477b038-1630-a1'), ('Content-Length', '0'), ('Date', 'Wed, 31 May 2023 20:38:16 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:38:16.737 DEBUG:boto:path=/pklxkd-58/obj00000000000000000014 2023-05-31T20:38:16.738 DEBUG:boto:auth_path=/pklxkd-58/obj00000000000000000014 2023-05-31T20:38:16.738 DEBUG:boto:Method: PUT 2023-05-31T20:38:16.738 DEBUG:boto:Path: /pklxkd-58/obj00000000000000000014 2023-05-31T20:38:16.739 DEBUG:boto:Data: 2023-05-31T20:38:16.739 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-05-31T20:38:16.740 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:38:16.741 DEBUG:boto:Port: 8000 2023-05-31T20:38:16.742 DEBUG:boto:Params: {} 2023-05-31T20:38:16.743 DEBUG:boto:Token: None 2023-05-31T20:38:16.744 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Wed, 31 May 2023 20:38:16 GMT /pklxkd-58/obj00000000000000000014 2023-05-31T20:38:16.745 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:EbQXs5UP0qRXGmxY3S3+xZVv0Rs= 2023-05-31T20:38:16.746 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Wed, 31 May 2023 20:38:16 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:EbQXs5UP0qRXGmxY3S3+xZVv0Rs='} 2023-05-31T20:38:16.757 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000e90854131f71c7a8-006477b038-1630-a1'), ('Date', 'Wed, 31 May 2023 20:38:16 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:38:16.758 DEBUG:boto:path=/pklxkd-58/ 2023-05-31T20:38:16.759 DEBUG:boto:auth_path=/pklxkd-58/ 2023-05-31T20:38:16.759 DEBUG:boto:Method: HEAD 2023-05-31T20:38:16.760 DEBUG:boto:Path: /pklxkd-58/ 2023-05-31T20:38:16.760 DEBUG:boto:Data: 2023-05-31T20:38:16.761 DEBUG:boto:Headers: {} 2023-05-31T20:38:16.761 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:38:16.761 DEBUG:boto:Port: 8000 2023-05-31T20:38:16.762 DEBUG:boto:Params: {} 2023-05-31T20:38:16.763 DEBUG:boto:Token: None 2023-05-31T20:38:16.763 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:38:16 GMT /pklxkd-58/ 2023-05-31T20:38:16.764 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:CQRtozsOi7RN4Myv9sdbVJvzIYc= 2023-05-31T20:38:16.764 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:38:16 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:CQRtozsOi7RN4Myv9sdbVJvzIYc=', 'Content-Length': '0'} 2023-05-31T20:38:16.767 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '26'), ('X-RGW-Bytes-Used', '78'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000c862a5fd677811cf-006477b038-1630-a1'), ('Content-Length', '0'), ('Date', 'Wed, 31 May 2023 20:38:16 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:38:16.768 DEBUG:boto:path=/pklxkd-58/obj00000000000000000006 2023-05-31T20:38:16.769 DEBUG:boto:auth_path=/pklxkd-58/obj00000000000000000006 2023-05-31T20:38:16.769 DEBUG:boto:Method: PUT 2023-05-31T20:38:16.769 DEBUG:boto:Path: /pklxkd-58/obj00000000000000000006 2023-05-31T20:38:16.770 DEBUG:boto:Data: 2023-05-31T20:38:16.770 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-05-31T20:38:16.770 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:38:16.771 DEBUG:boto:Port: 8000 2023-05-31T20:38:16.771 DEBUG:boto:Params: {} 2023-05-31T20:38:16.771 DEBUG:boto:Token: None 2023-05-31T20:38:16.772 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Wed, 31 May 2023 20:38:16 GMT /pklxkd-58/obj00000000000000000006 2023-05-31T20:38:16.772 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:fJn9qicKjesoRsOuBfd3lSCtdZE= 2023-05-31T20:38:16.773 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Wed, 31 May 2023 20:38:16 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:fJn9qicKjesoRsOuBfd3lSCtdZE='} 2023-05-31T20:38:16.817 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000395ed7671da4f890-006477b038-1630-a1'), ('Date', 'Wed, 31 May 2023 20:38:16 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:38:16.818 DEBUG:boto:path=/pklxkd-58/ 2023-05-31T20:38:16.820 DEBUG:boto:auth_path=/pklxkd-58/ 2023-05-31T20:38:16.820 DEBUG:boto:Method: HEAD 2023-05-31T20:38:16.821 DEBUG:boto:Path: /pklxkd-58/ 2023-05-31T20:38:16.821 DEBUG:boto:Data: 2023-05-31T20:38:16.822 DEBUG:boto:Headers: {} 2023-05-31T20:38:16.822 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:38:16.823 DEBUG:boto:Port: 8000 2023-05-31T20:38:16.824 DEBUG:boto:Params: {} 2023-05-31T20:38:16.824 DEBUG:boto:Token: None 2023-05-31T20:38:16.825 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:38:16 GMT /pklxkd-58/ 2023-05-31T20:38:16.826 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:CQRtozsOi7RN4Myv9sdbVJvzIYc= 2023-05-31T20:38:16.826 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:38:16 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:CQRtozsOi7RN4Myv9sdbVJvzIYc=', 'Content-Length': '0'} 2023-05-31T20:38:16.829 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '26'), ('X-RGW-Bytes-Used', '78'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000005b93ec262a9b5b29-006477b038-1630-a1'), ('Content-Length', '0'), ('Date', 'Wed, 31 May 2023 20:38:16 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:38:16.830 DEBUG:boto:path=/pklxkd-58/obj00000000000000000000 2023-05-31T20:38:16.830 DEBUG:boto:auth_path=/pklxkd-58/obj00000000000000000000 2023-05-31T20:38:16.831 DEBUG:boto:Method: PUT 2023-05-31T20:38:16.831 DEBUG:boto:Path: /pklxkd-58/obj00000000000000000000 2023-05-31T20:38:16.832 DEBUG:boto:Data: 2023-05-31T20:38:16.832 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-05-31T20:38:16.833 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:38:16.834 DEBUG:boto:Port: 8000 2023-05-31T20:38:16.834 DEBUG:boto:Params: {} 2023-05-31T20:38:16.834 DEBUG:boto:Token: None 2023-05-31T20:38:16.835 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Wed, 31 May 2023 20:38:16 GMT /pklxkd-58/obj00000000000000000000 2023-05-31T20:38:16.835 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:BKqXelrc03pzY0OF75yZ89e6bjU= 2023-05-31T20:38:16.836 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Wed, 31 May 2023 20:38:16 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:BKqXelrc03pzY0OF75yZ89e6bjU='} 2023-05-31T20:38:16.850 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000bb021e2fa41216e3-006477b038-1630-a1'), ('Date', 'Wed, 31 May 2023 20:38:16 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:38:16.851 DEBUG:boto:path=/pklxkd-58/ 2023-05-31T20:38:16.852 DEBUG:boto:auth_path=/pklxkd-58/ 2023-05-31T20:38:16.852 DEBUG:boto:Method: HEAD 2023-05-31T20:38:16.853 DEBUG:boto:Path: /pklxkd-58/ 2023-05-31T20:38:16.853 DEBUG:boto:Data: 2023-05-31T20:38:16.853 DEBUG:boto:Headers: {} 2023-05-31T20:38:16.854 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:38:16.854 DEBUG:boto:Port: 8000 2023-05-31T20:38:16.854 DEBUG:boto:Params: {} 2023-05-31T20:38:16.855 DEBUG:boto:Token: None 2023-05-31T20:38:16.855 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:38:16 GMT /pklxkd-58/ 2023-05-31T20:38:16.856 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:CQRtozsOi7RN4Myv9sdbVJvzIYc= 2023-05-31T20:38:16.857 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:38:16 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:CQRtozsOi7RN4Myv9sdbVJvzIYc=', 'Content-Length': '0'} 2023-05-31T20:38:16.861 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '26'), ('X-RGW-Bytes-Used', '78'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000009a947b2fa9f795fa-006477b038-1630-a1'), ('Content-Length', '0'), ('Date', 'Wed, 31 May 2023 20:38:16 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:38:16.862 DEBUG:boto:path=/pklxkd-58/obj00000000000000000007 2023-05-31T20:38:16.862 DEBUG:boto:auth_path=/pklxkd-58/obj00000000000000000007 2023-05-31T20:38:16.863 DEBUG:boto:Method: PUT 2023-05-31T20:38:16.863 DEBUG:boto:Path: /pklxkd-58/obj00000000000000000007 2023-05-31T20:38:16.863 DEBUG:boto:Data: 2023-05-31T20:38:16.864 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-05-31T20:38:16.864 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:38:16.864 DEBUG:boto:Port: 8000 2023-05-31T20:38:16.865 DEBUG:boto:Params: {} 2023-05-31T20:38:16.865 DEBUG:boto:Token: None 2023-05-31T20:38:16.866 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Wed, 31 May 2023 20:38:16 GMT /pklxkd-58/obj00000000000000000007 2023-05-31T20:38:16.866 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:Ecv7m14q2RykFhD4AeKHf8+c/ZY= 2023-05-31T20:38:16.867 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Wed, 31 May 2023 20:38:16 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:Ecv7m14q2RykFhD4AeKHf8+c/ZY='} 2023-05-31T20:38:16.908 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000e88f35ac81e08a0b-006477b038-1630-a1'), ('Date', 'Wed, 31 May 2023 20:38:16 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:38:16.909 DEBUG:boto:path=/pklxkd-58/ 2023-05-31T20:38:16.910 DEBUG:boto:auth_path=/pklxkd-58/ 2023-05-31T20:38:16.910 DEBUG:boto:Method: HEAD 2023-05-31T20:38:16.910 DEBUG:boto:Path: /pklxkd-58/ 2023-05-31T20:38:16.911 DEBUG:boto:Data: 2023-05-31T20:38:16.911 DEBUG:boto:Headers: {} 2023-05-31T20:38:16.911 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:38:16.912 DEBUG:boto:Port: 8000 2023-05-31T20:38:16.913 DEBUG:boto:Params: {} 2023-05-31T20:38:16.913 DEBUG:boto:Token: None 2023-05-31T20:38:16.914 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:38:16 GMT /pklxkd-58/ 2023-05-31T20:38:16.914 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:CQRtozsOi7RN4Myv9sdbVJvzIYc= 2023-05-31T20:38:16.915 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:38:16 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:CQRtozsOi7RN4Myv9sdbVJvzIYc=', 'Content-Length': '0'} 2023-05-31T20:38:16.917 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '26'), ('X-RGW-Bytes-Used', '78'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000a730a181b6b6ac9d-006477b038-1630-a1'), ('Content-Length', '0'), ('Date', 'Wed, 31 May 2023 20:38:16 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:38:16.918 DEBUG:boto:path=/pklxkd-58/obj00000000000000000004 2023-05-31T20:38:16.918 DEBUG:boto:auth_path=/pklxkd-58/obj00000000000000000004 2023-05-31T20:38:16.919 DEBUG:boto:Method: PUT 2023-05-31T20:38:16.919 DEBUG:boto:Path: /pklxkd-58/obj00000000000000000004 2023-05-31T20:38:16.920 DEBUG:boto:Data: 2023-05-31T20:38:16.920 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-05-31T20:38:16.921 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:38:16.921 DEBUG:boto:Port: 8000 2023-05-31T20:38:16.921 DEBUG:boto:Params: {} 2023-05-31T20:38:16.922 DEBUG:boto:Token: None 2023-05-31T20:38:16.923 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Wed, 31 May 2023 20:38:16 GMT /pklxkd-58/obj00000000000000000004 2023-05-31T20:38:16.923 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:cNkMciJ+ebsJ8hzbcjUjjkvl0MU= 2023-05-31T20:38:16.924 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Wed, 31 May 2023 20:38:16 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:cNkMciJ+ebsJ8hzbcjUjjkvl0MU='} 2023-05-31T20:38:16.931 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000098737dc3aac00e93-006477b038-1630-a1'), ('Date', 'Wed, 31 May 2023 20:38:16 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:38:16.932 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'reshard', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'pklxkd-58', '--num-shards', '23', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-05-31T20:38:16.934 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'reshard', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'pklxkd-58', '--num-shards', '23', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-05-31T20:38:16.935 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket reshard --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --bucket pklxkd-58 --num-shards 23 --yes-i-really-mean-it --cluster c1 --debug-rgw 0 --debug-ms 0 2023-05-31T20:38:16.992 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2023-05-31T20:38:16.992 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2023-05-31T20:38:17.042 INFO:teuthology.orchestra.run.smithi099.stdout:tenant: 2023-05-31T20:38:17.042 INFO:teuthology.orchestra.run.smithi099.stdout:bucket name: pklxkd-58 2023-05-31T20:38:17.049 INFO:teuthology.orchestra.run.smithi099.stdout:total entries: 26 2023-05-31T20:38:17.122 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'shard', 'objects', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--num-shards', '23', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-31T20:38:17.123 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'shard', 'objects', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--num-shards', '23', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-31T20:38:17.124 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket shard objects --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --num-shards 23 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-05-31T20:38:17.175 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2023-05-31T20:38:17.175 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2023-05-31T20:38:17.253 INFO:teuthology.orchestra.run.smithi099.stdout:{ 2023-05-31T20:38:17.253 INFO:teuthology.orchestra.run.smithi099.stdout: "objs": [ 2023-05-31T20:38:17.254 INFO:teuthology.orchestra.run.smithi099.stdout: "obj00000000000000000058", 2023-05-31T20:38:17.254 INFO:teuthology.orchestra.run.smithi099.stdout: "obj00000000000000000026", 2023-05-31T20:38:17.254 INFO:teuthology.orchestra.run.smithi099.stdout: "obj00000000000000000011", 2023-05-31T20:38:17.254 INFO:teuthology.orchestra.run.smithi099.stdout: "obj00000000000000000046", 2023-05-31T20:38:17.254 INFO:teuthology.orchestra.run.smithi099.stdout: "obj00000000000000000014", 2023-05-31T20:38:17.254 INFO:teuthology.orchestra.run.smithi099.stdout: "obj00000000000000000050", 2023-05-31T20:38:17.255 INFO:teuthology.orchestra.run.smithi099.stdout: "obj00000000000000000032", 2023-05-31T20:38:17.255 INFO:teuthology.orchestra.run.smithi099.stdout: "obj00000000000000000002", 2023-05-31T20:38:17.255 INFO:teuthology.orchestra.run.smithi099.stdout: "obj00000000000000000039", 2023-05-31T20:38:17.255 INFO:teuthology.orchestra.run.smithi099.stdout: "obj00000000000000000007", 2023-05-31T20:38:17.255 INFO:teuthology.orchestra.run.smithi099.stdout: "obj00000000000000000006", 2023-05-31T20:38:17.255 INFO:teuthology.orchestra.run.smithi099.stdout: "obj00000000000000000040", 2023-05-31T20:38:17.256 INFO:teuthology.orchestra.run.smithi099.stdout: "obj00000000000000000009", 2023-05-31T20:38:17.256 INFO:teuthology.orchestra.run.smithi099.stdout: "obj00000000000000000045", 2023-05-31T20:38:17.256 INFO:teuthology.orchestra.run.smithi099.stdout: "obj00000000000000000013", 2023-05-31T20:38:17.256 INFO:teuthology.orchestra.run.smithi099.stdout: "obj00000000000000000012", 2023-05-31T20:38:17.257 INFO:teuthology.orchestra.run.smithi099.stdout: "obj00000000000000000033", 2023-05-31T20:38:17.257 INFO:teuthology.orchestra.run.smithi099.stdout: "obj00000000000000000001", 2023-05-31T20:38:17.257 INFO:teuthology.orchestra.run.smithi099.stdout: "obj00000000000000000000", 2023-05-31T20:38:17.258 INFO:teuthology.orchestra.run.smithi099.stdout: "obj00000000000000000021", 2023-05-31T20:38:17.258 INFO:teuthology.orchestra.run.smithi099.stdout: "obj00000000000000000003", 2023-05-31T20:38:17.258 INFO:teuthology.orchestra.run.smithi099.stdout: "obj00000000000000000055", 2023-05-31T20:38:17.259 INFO:teuthology.orchestra.run.smithi099.stdout: "obj00000000000000000008" 2023-05-31T20:38:17.259 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2023-05-31T20:38:17.259 INFO:teuthology.orchestra.run.smithi099.stdout:} 2023-05-31T20:38:17.261 DEBUG:boto:path=/pklxkd-58/ 2023-05-31T20:38:17.262 DEBUG:boto:auth_path=/pklxkd-58/ 2023-05-31T20:38:17.263 DEBUG:boto:Method: HEAD 2023-05-31T20:38:17.264 DEBUG:boto:Path: /pklxkd-58/ 2023-05-31T20:38:17.264 DEBUG:boto:Data: 2023-05-31T20:38:17.265 DEBUG:boto:Headers: {} 2023-05-31T20:38:17.265 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:38:17.265 DEBUG:boto:Port: 8000 2023-05-31T20:38:17.266 DEBUG:boto:Params: {} 2023-05-31T20:38:17.266 DEBUG:boto:Token: None 2023-05-31T20:38:17.267 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:38:17 GMT /pklxkd-58/ 2023-05-31T20:38:17.267 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:jg/JHBeyhCfiZ+Bpv2lLfE21uGU= 2023-05-31T20:38:17.267 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:38:17 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:jg/JHBeyhCfiZ+Bpv2lLfE21uGU=', 'Content-Length': '0'} 2023-05-31T20:38:17.271 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '26'), ('X-RGW-Bytes-Used', '78'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000004e732b1c9c5b62cd-006477b039-1630-a1'), ('Content-Length', '0'), ('Date', 'Wed, 31 May 2023 20:38:17 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:38:17.272 DEBUG:boto:path=/pklxkd-58/obj00000000000000000026 2023-05-31T20:38:17.273 DEBUG:boto:auth_path=/pklxkd-58/obj00000000000000000026 2023-05-31T20:38:17.274 DEBUG:boto:Method: PUT 2023-05-31T20:38:17.274 DEBUG:boto:Path: /pklxkd-58/obj00000000000000000026 2023-05-31T20:38:17.274 DEBUG:boto:Data: 2023-05-31T20:38:17.275 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-05-31T20:38:17.275 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:38:17.275 DEBUG:boto:Port: 8000 2023-05-31T20:38:17.276 DEBUG:boto:Params: {} 2023-05-31T20:38:17.276 DEBUG:boto:Token: None 2023-05-31T20:38:17.277 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Wed, 31 May 2023 20:38:17 GMT /pklxkd-58/obj00000000000000000026 2023-05-31T20:38:17.277 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:rFkuJUvw+vnlXNNDbdMDwEDhRac= 2023-05-31T20:38:17.277 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Wed, 31 May 2023 20:38:17 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:rFkuJUvw+vnlXNNDbdMDwEDhRac='} 2023-05-31T20:38:17.285 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000cec7a00c6c9e0db4-006477b039-1630-a1'), ('Date', 'Wed, 31 May 2023 20:38:17 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:38:17.286 DEBUG:boto:path=/pklxkd-58/ 2023-05-31T20:38:17.286 DEBUG:boto:auth_path=/pklxkd-58/ 2023-05-31T20:38:17.287 DEBUG:boto:Method: HEAD 2023-05-31T20:38:17.287 DEBUG:boto:Path: /pklxkd-58/ 2023-05-31T20:38:17.287 DEBUG:boto:Data: 2023-05-31T20:38:17.288 DEBUG:boto:Headers: {} 2023-05-31T20:38:17.288 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:38:17.289 DEBUG:boto:Port: 8000 2023-05-31T20:38:17.289 DEBUG:boto:Params: {} 2023-05-31T20:38:17.289 DEBUG:boto:Token: None 2023-05-31T20:38:17.290 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:38:17 GMT /pklxkd-58/ 2023-05-31T20:38:17.290 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:jg/JHBeyhCfiZ+Bpv2lLfE21uGU= 2023-05-31T20:38:17.291 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:38:17 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:jg/JHBeyhCfiZ+Bpv2lLfE21uGU=', 'Content-Length': '0'} 2023-05-31T20:38:17.291 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.1 has been restored 2023-05-31T20:38:17.293 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '26'), ('X-RGW-Bytes-Used', '78'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000002e3a0e45121621a8-006477b039-1630-a1'), ('Content-Length', '0'), ('Date', 'Wed, 31 May 2023 20:38:17 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:38:17.294 DEBUG:boto:path=/pklxkd-58/obj00000000000000000013 2023-05-31T20:38:17.294 DEBUG:boto:auth_path=/pklxkd-58/obj00000000000000000013 2023-05-31T20:38:17.294 DEBUG:boto:Method: PUT 2023-05-31T20:38:17.295 DEBUG:boto:Path: /pklxkd-58/obj00000000000000000013 2023-05-31T20:38:17.295 DEBUG:boto:Data: 2023-05-31T20:38:17.295 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-05-31T20:38:17.296 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:38:17.296 DEBUG:boto:Port: 8000 2023-05-31T20:38:17.297 DEBUG:boto:Params: {} 2023-05-31T20:38:17.297 DEBUG:boto:Token: None 2023-05-31T20:38:17.297 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Wed, 31 May 2023 20:38:17 GMT /pklxkd-58/obj00000000000000000013 2023-05-31T20:38:17.298 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:FH7wJ8+jAVPdVExwiWsSznzgPKI= 2023-05-31T20:38:17.298 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Wed, 31 May 2023 20:38:17 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:FH7wJ8+jAVPdVExwiWsSznzgPKI='} 2023-05-31T20:38:17.341 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000018439d1c1cf118a1-006477b039-1630-a1'), ('Date', 'Wed, 31 May 2023 20:38:17 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:38:17.342 DEBUG:boto:path=/pklxkd-58/ 2023-05-31T20:38:17.343 DEBUG:boto:auth_path=/pklxkd-58/ 2023-05-31T20:38:17.343 DEBUG:boto:Method: HEAD 2023-05-31T20:38:17.344 DEBUG:boto:Path: /pklxkd-58/ 2023-05-31T20:38:17.345 DEBUG:boto:Data: 2023-05-31T20:38:17.346 DEBUG:boto:Headers: {} 2023-05-31T20:38:17.346 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:38:17.347 DEBUG:boto:Port: 8000 2023-05-31T20:38:17.348 DEBUG:boto:Params: {} 2023-05-31T20:38:17.348 DEBUG:boto:Token: None 2023-05-31T20:38:17.349 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:38:17 GMT /pklxkd-58/ 2023-05-31T20:38:17.349 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:jg/JHBeyhCfiZ+Bpv2lLfE21uGU= 2023-05-31T20:38:17.350 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:38:17 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:jg/JHBeyhCfiZ+Bpv2lLfE21uGU=', 'Content-Length': '0'} 2023-05-31T20:38:17.352 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '26'), ('X-RGW-Bytes-Used', '78'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000621c172dbbfca887-006477b039-1630-a1'), ('Content-Length', '0'), ('Date', 'Wed, 31 May 2023 20:38:17 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:38:17.353 DEBUG:boto:path=/pklxkd-58/obj00000000000000000046 2023-05-31T20:38:17.353 DEBUG:boto:auth_path=/pklxkd-58/obj00000000000000000046 2023-05-31T20:38:17.354 DEBUG:boto:Method: PUT 2023-05-31T20:38:17.354 DEBUG:boto:Path: /pklxkd-58/obj00000000000000000046 2023-05-31T20:38:17.354 DEBUG:boto:Data: 2023-05-31T20:38:17.355 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-05-31T20:38:17.355 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:38:17.355 DEBUG:boto:Port: 8000 2023-05-31T20:38:17.356 DEBUG:boto:Params: {} 2023-05-31T20:38:17.357 DEBUG:boto:Token: None 2023-05-31T20:38:17.358 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Wed, 31 May 2023 20:38:17 GMT /pklxkd-58/obj00000000000000000046 2023-05-31T20:38:17.358 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:revvxVLNvSAz1T1EG0+Sdis/fy8= 2023-05-31T20:38:17.359 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Wed, 31 May 2023 20:38:17 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:revvxVLNvSAz1T1EG0+Sdis/fy8='} 2023-05-31T20:38:17.365 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000006d09fc9ea129d967-006477b039-1630-a1'), ('Date', 'Wed, 31 May 2023 20:38:17 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:38:17.366 DEBUG:boto:path=/pklxkd-58/ 2023-05-31T20:38:17.366 DEBUG:boto:auth_path=/pklxkd-58/ 2023-05-31T20:38:17.367 DEBUG:boto:Method: HEAD 2023-05-31T20:38:17.367 DEBUG:boto:Path: /pklxkd-58/ 2023-05-31T20:38:17.368 DEBUG:boto:Data: 2023-05-31T20:38:17.369 DEBUG:boto:Headers: {} 2023-05-31T20:38:17.369 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:38:17.369 DEBUG:boto:Port: 8000 2023-05-31T20:38:17.370 DEBUG:boto:Params: {} 2023-05-31T20:38:17.371 DEBUG:boto:Token: None 2023-05-31T20:38:17.371 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:38:17 GMT /pklxkd-58/ 2023-05-31T20:38:17.372 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:jg/JHBeyhCfiZ+Bpv2lLfE21uGU= 2023-05-31T20:38:17.373 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:38:17 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:jg/JHBeyhCfiZ+Bpv2lLfE21uGU=', 'Content-Length': '0'} 2023-05-31T20:38:17.376 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '27'), ('X-RGW-Bytes-Used', '81'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000000cc6fda505510f73-006477b039-1630-a1'), ('Content-Length', '0'), ('Date', 'Wed, 31 May 2023 20:38:17 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:38:17.377 DEBUG:boto:path=/pklxkd-58/obj00000000000000000039 2023-05-31T20:38:17.377 DEBUG:boto:auth_path=/pklxkd-58/obj00000000000000000039 2023-05-31T20:38:17.378 DEBUG:boto:Method: PUT 2023-05-31T20:38:17.378 DEBUG:boto:Path: /pklxkd-58/obj00000000000000000039 2023-05-31T20:38:17.379 DEBUG:boto:Data: 2023-05-31T20:38:17.379 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-05-31T20:38:17.379 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:38:17.380 DEBUG:boto:Port: 8000 2023-05-31T20:38:17.380 DEBUG:boto:Params: {} 2023-05-31T20:38:17.381 DEBUG:boto:Token: None 2023-05-31T20:38:17.381 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Wed, 31 May 2023 20:38:17 GMT /pklxkd-58/obj00000000000000000039 2023-05-31T20:38:17.381 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:HWWkkivNp7yY9aHJA1n84b+BvaQ= 2023-05-31T20:38:17.382 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Wed, 31 May 2023 20:38:17 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:HWWkkivNp7yY9aHJA1n84b+BvaQ='} 2023-05-31T20:38:17.387 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000f19ec0ea949f8d42-006477b039-1630-a1'), ('Date', 'Wed, 31 May 2023 20:38:17 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:38:17.388 DEBUG:boto:path=/pklxkd-58/ 2023-05-31T20:38:17.388 DEBUG:boto:auth_path=/pklxkd-58/ 2023-05-31T20:38:17.388 DEBUG:boto:Method: HEAD 2023-05-31T20:38:17.389 DEBUG:boto:Path: /pklxkd-58/ 2023-05-31T20:38:17.389 DEBUG:boto:Data: 2023-05-31T20:38:17.389 DEBUG:boto:Headers: {} 2023-05-31T20:38:17.390 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:38:17.390 DEBUG:boto:Port: 8000 2023-05-31T20:38:17.390 DEBUG:boto:Params: {} 2023-05-31T20:38:17.391 DEBUG:boto:Token: None 2023-05-31T20:38:17.391 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:38:17 GMT /pklxkd-58/ 2023-05-31T20:38:17.392 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:jg/JHBeyhCfiZ+Bpv2lLfE21uGU= 2023-05-31T20:38:17.392 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:38:17 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:jg/JHBeyhCfiZ+Bpv2lLfE21uGU=', 'Content-Length': '0'} 2023-05-31T20:38:17.395 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '28'), ('X-RGW-Bytes-Used', '84'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000ba2315a5c23b001a-006477b039-1630-a1'), ('Content-Length', '0'), ('Date', 'Wed, 31 May 2023 20:38:17 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:38:17.395 DEBUG:boto:path=/pklxkd-58/obj00000000000000000009 2023-05-31T20:38:17.396 DEBUG:boto:auth_path=/pklxkd-58/obj00000000000000000009 2023-05-31T20:38:17.396 DEBUG:boto:Method: PUT 2023-05-31T20:38:17.396 DEBUG:boto:Path: /pklxkd-58/obj00000000000000000009 2023-05-31T20:38:17.397 DEBUG:boto:Data: 2023-05-31T20:38:17.397 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-05-31T20:38:17.397 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:38:17.398 DEBUG:boto:Port: 8000 2023-05-31T20:38:17.398 DEBUG:boto:Params: {} 2023-05-31T20:38:17.398 DEBUG:boto:Token: None 2023-05-31T20:38:17.399 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Wed, 31 May 2023 20:38:17 GMT /pklxkd-58/obj00000000000000000009 2023-05-31T20:38:17.399 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:EgBfKq1SgPFNQMS0uVl8rZkRzAI= 2023-05-31T20:38:17.400 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Wed, 31 May 2023 20:38:17 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:EgBfKq1SgPFNQMS0uVl8rZkRzAI='} 2023-05-31T20:38:17.449 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000e1ae85ff0b07f256-006477b039-1630-a1'), ('Date', 'Wed, 31 May 2023 20:38:17 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:38:17.450 DEBUG:boto:path=/pklxkd-58/ 2023-05-31T20:38:17.451 DEBUG:boto:auth_path=/pklxkd-58/ 2023-05-31T20:38:17.452 DEBUG:boto:Method: HEAD 2023-05-31T20:38:17.452 DEBUG:boto:Path: /pklxkd-58/ 2023-05-31T20:38:17.453 DEBUG:boto:Data: 2023-05-31T20:38:17.453 DEBUG:boto:Headers: {} 2023-05-31T20:38:17.454 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:38:17.454 DEBUG:boto:Port: 8000 2023-05-31T20:38:17.454 DEBUG:boto:Params: {} 2023-05-31T20:38:17.455 DEBUG:boto:Token: None 2023-05-31T20:38:17.456 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:38:17 GMT /pklxkd-58/ 2023-05-31T20:38:17.457 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:jg/JHBeyhCfiZ+Bpv2lLfE21uGU= 2023-05-31T20:38:17.458 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:38:17 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:jg/JHBeyhCfiZ+Bpv2lLfE21uGU=', 'Content-Length': '0'} 2023-05-31T20:38:17.463 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '28'), ('X-RGW-Bytes-Used', '84'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000b3e2afd17f81ccc3-006477b039-1630-a1'), ('Content-Length', '0'), ('Date', 'Wed, 31 May 2023 20:38:17 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:38:17.465 DEBUG:boto:path=/pklxkd-58/obj00000000000000000008 2023-05-31T20:38:17.465 DEBUG:boto:auth_path=/pklxkd-58/obj00000000000000000008 2023-05-31T20:38:17.466 DEBUG:boto:Method: PUT 2023-05-31T20:38:17.467 DEBUG:boto:Path: /pklxkd-58/obj00000000000000000008 2023-05-31T20:38:17.467 DEBUG:boto:Data: 2023-05-31T20:38:17.468 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-05-31T20:38:17.468 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:38:17.469 DEBUG:boto:Port: 8000 2023-05-31T20:38:17.469 DEBUG:boto:Params: {} 2023-05-31T20:38:17.470 DEBUG:boto:Token: None 2023-05-31T20:38:17.470 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Wed, 31 May 2023 20:38:17 GMT /pklxkd-58/obj00000000000000000008 2023-05-31T20:38:17.471 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:JIcibbwXf7o/5Eft7cx8SPMM84U= 2023-05-31T20:38:17.472 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Wed, 31 May 2023 20:38:17 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:JIcibbwXf7o/5Eft7cx8SPMM84U='} 2023-05-31T20:38:17.478 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000002d4baba350b4f985-006477b039-1630-a1'), ('Date', 'Wed, 31 May 2023 20:38:17 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:38:17.479 DEBUG:boto:path=/pklxkd-58/ 2023-05-31T20:38:17.479 DEBUG:boto:auth_path=/pklxkd-58/ 2023-05-31T20:38:17.480 DEBUG:boto:Method: HEAD 2023-05-31T20:38:17.481 DEBUG:boto:Path: /pklxkd-58/ 2023-05-31T20:38:17.481 DEBUG:boto:Data: 2023-05-31T20:38:17.482 DEBUG:boto:Headers: {} 2023-05-31T20:38:17.482 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:38:17.483 DEBUG:boto:Port: 8000 2023-05-31T20:38:17.483 DEBUG:boto:Params: {} 2023-05-31T20:38:17.484 DEBUG:boto:Token: None 2023-05-31T20:38:17.485 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:38:17 GMT /pklxkd-58/ 2023-05-31T20:38:17.486 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:jg/JHBeyhCfiZ+Bpv2lLfE21uGU= 2023-05-31T20:38:17.486 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:38:17 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:jg/JHBeyhCfiZ+Bpv2lLfE21uGU=', 'Content-Length': '0'} 2023-05-31T20:38:17.489 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '29'), ('X-RGW-Bytes-Used', '87'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000005128f097e8f4adc8-006477b039-1630-a1'), ('Content-Length', '0'), ('Date', 'Wed, 31 May 2023 20:38:17 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:38:17.490 DEBUG:boto:path=/pklxkd-58/obj00000000000000000002 2023-05-31T20:38:17.491 DEBUG:boto:auth_path=/pklxkd-58/obj00000000000000000002 2023-05-31T20:38:17.492 DEBUG:boto:Method: PUT 2023-05-31T20:38:17.492 DEBUG:boto:Path: /pklxkd-58/obj00000000000000000002 2023-05-31T20:38:17.493 DEBUG:boto:Data: 2023-05-31T20:38:17.493 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-05-31T20:38:17.493 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:38:17.494 DEBUG:boto:Port: 8000 2023-05-31T20:38:17.494 DEBUG:boto:Params: {} 2023-05-31T20:38:17.495 DEBUG:boto:Token: None 2023-05-31T20:38:17.495 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Wed, 31 May 2023 20:38:17 GMT /pklxkd-58/obj00000000000000000002 2023-05-31T20:38:17.495 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:4og58Qn0+fyRd/sdyLeU4pAC4wY= 2023-05-31T20:38:17.496 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Wed, 31 May 2023 20:38:17 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:4og58Qn0+fyRd/sdyLeU4pAC4wY='} 2023-05-31T20:38:17.540 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000023544d4e59ee6d6-006477b039-1630-a1'), ('Date', 'Wed, 31 May 2023 20:38:17 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:38:17.541 DEBUG:boto:path=/pklxkd-58/ 2023-05-31T20:38:17.541 DEBUG:boto:auth_path=/pklxkd-58/ 2023-05-31T20:38:17.542 DEBUG:boto:Method: HEAD 2023-05-31T20:38:17.542 DEBUG:boto:Path: /pklxkd-58/ 2023-05-31T20:38:17.542 DEBUG:boto:Data: 2023-05-31T20:38:17.543 DEBUG:boto:Headers: {} 2023-05-31T20:38:17.543 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:38:17.543 DEBUG:boto:Port: 8000 2023-05-31T20:38:17.544 DEBUG:boto:Params: {} 2023-05-31T20:38:17.544 DEBUG:boto:Token: None 2023-05-31T20:38:17.545 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:38:17 GMT /pklxkd-58/ 2023-05-31T20:38:17.545 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:jg/JHBeyhCfiZ+Bpv2lLfE21uGU= 2023-05-31T20:38:17.546 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:38:17 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:jg/JHBeyhCfiZ+Bpv2lLfE21uGU=', 'Content-Length': '0'} 2023-05-31T20:38:17.548 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '29'), ('X-RGW-Bytes-Used', '87'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000019757e64fba493f5-006477b039-1630-a1'), ('Content-Length', '0'), ('Date', 'Wed, 31 May 2023 20:38:17 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:38:17.549 DEBUG:boto:path=/pklxkd-58/obj00000000000000000040 2023-05-31T20:38:17.549 DEBUG:boto:auth_path=/pklxkd-58/obj00000000000000000040 2023-05-31T20:38:17.550 DEBUG:boto:Method: PUT 2023-05-31T20:38:17.550 DEBUG:boto:Path: /pklxkd-58/obj00000000000000000040 2023-05-31T20:38:17.551 DEBUG:boto:Data: 2023-05-31T20:38:17.551 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-05-31T20:38:17.551 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:38:17.552 DEBUG:boto:Port: 8000 2023-05-31T20:38:17.552 DEBUG:boto:Params: {} 2023-05-31T20:38:17.552 DEBUG:boto:Token: None 2023-05-31T20:38:17.553 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Wed, 31 May 2023 20:38:17 GMT /pklxkd-58/obj00000000000000000040 2023-05-31T20:38:17.553 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:rBB7R3F2sWGPV0ctwQ0wC+Ok54s= 2023-05-31T20:38:17.554 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Wed, 31 May 2023 20:38:17 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:rBB7R3F2sWGPV0ctwQ0wC+Ok54s='} 2023-05-31T20:38:17.559 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000007735790d5268a8ec-006477b039-1630-a1'), ('Date', 'Wed, 31 May 2023 20:38:17 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:38:17.560 DEBUG:boto:path=/pklxkd-58/ 2023-05-31T20:38:17.560 DEBUG:boto:auth_path=/pklxkd-58/ 2023-05-31T20:38:17.561 DEBUG:boto:Method: HEAD 2023-05-31T20:38:17.561 DEBUG:boto:Path: /pklxkd-58/ 2023-05-31T20:38:17.561 DEBUG:boto:Data: 2023-05-31T20:38:17.562 DEBUG:boto:Headers: {} 2023-05-31T20:38:17.562 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:38:17.562 DEBUG:boto:Port: 8000 2023-05-31T20:38:17.563 DEBUG:boto:Params: {} 2023-05-31T20:38:17.564 DEBUG:boto:Token: None 2023-05-31T20:38:17.564 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:38:17 GMT /pklxkd-58/ 2023-05-31T20:38:17.565 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:jg/JHBeyhCfiZ+Bpv2lLfE21uGU= 2023-05-31T20:38:17.565 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:38:17 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:jg/JHBeyhCfiZ+Bpv2lLfE21uGU=', 'Content-Length': '0'} 2023-05-31T20:38:17.567 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '30'), ('X-RGW-Bytes-Used', '90'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000309e4157092f1d50-006477b039-1630-a1'), ('Content-Length', '0'), ('Date', 'Wed, 31 May 2023 20:38:17 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:38:17.568 DEBUG:boto:path=/pklxkd-58/obj00000000000000000033 2023-05-31T20:38:17.568 DEBUG:boto:auth_path=/pklxkd-58/obj00000000000000000033 2023-05-31T20:38:17.569 DEBUG:boto:Method: PUT 2023-05-31T20:38:17.569 DEBUG:boto:Path: /pklxkd-58/obj00000000000000000033 2023-05-31T20:38:17.570 DEBUG:boto:Data: 2023-05-31T20:38:17.570 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-05-31T20:38:17.570 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:38:17.571 DEBUG:boto:Port: 8000 2023-05-31T20:38:17.571 DEBUG:boto:Params: {} 2023-05-31T20:38:17.571 DEBUG:boto:Token: None 2023-05-31T20:38:17.572 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Wed, 31 May 2023 20:38:17 GMT /pklxkd-58/obj00000000000000000033 2023-05-31T20:38:17.572 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:ii2lswQpWtb9eW8CfJqiVj+GWxE= 2023-05-31T20:38:17.573 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Wed, 31 May 2023 20:38:17 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:ii2lswQpWtb9eW8CfJqiVj+GWxE='} 2023-05-31T20:38:17.620 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000d655f3be70965ce8-006477b039-1630-a1'), ('Date', 'Wed, 31 May 2023 20:38:17 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:38:17.621 DEBUG:boto:path=/pklxkd-58/ 2023-05-31T20:38:17.621 DEBUG:boto:auth_path=/pklxkd-58/ 2023-05-31T20:38:17.622 DEBUG:boto:Method: HEAD 2023-05-31T20:38:17.622 DEBUG:boto:Path: /pklxkd-58/ 2023-05-31T20:38:17.622 DEBUG:boto:Data: 2023-05-31T20:38:17.623 DEBUG:boto:Headers: {} 2023-05-31T20:38:17.623 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:38:17.623 DEBUG:boto:Port: 8000 2023-05-31T20:38:17.624 DEBUG:boto:Params: {} 2023-05-31T20:38:17.624 DEBUG:boto:Token: None 2023-05-31T20:38:17.625 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:38:17 GMT /pklxkd-58/ 2023-05-31T20:38:17.625 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:jg/JHBeyhCfiZ+Bpv2lLfE21uGU= 2023-05-31T20:38:17.625 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:38:17 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:jg/JHBeyhCfiZ+Bpv2lLfE21uGU=', 'Content-Length': '0'} 2023-05-31T20:38:17.628 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '30'), ('X-RGW-Bytes-Used', '90'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000004093c8684a466ee5-006477b039-1630-a1'), ('Content-Length', '0'), ('Date', 'Wed, 31 May 2023 20:38:17 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:38:17.629 DEBUG:boto:path=/pklxkd-58/obj00000000000000000058 2023-05-31T20:38:17.629 DEBUG:boto:auth_path=/pklxkd-58/obj00000000000000000058 2023-05-31T20:38:17.630 DEBUG:boto:Method: PUT 2023-05-31T20:38:17.630 DEBUG:boto:Path: /pklxkd-58/obj00000000000000000058 2023-05-31T20:38:17.630 DEBUG:boto:Data: 2023-05-31T20:38:17.631 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-05-31T20:38:17.631 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:38:17.631 DEBUG:boto:Port: 8000 2023-05-31T20:38:17.632 DEBUG:boto:Params: {} 2023-05-31T20:38:17.632 DEBUG:boto:Token: None 2023-05-31T20:38:17.633 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Wed, 31 May 2023 20:38:17 GMT /pklxkd-58/obj00000000000000000058 2023-05-31T20:38:17.633 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:LQidbYpadeIoViAYtiVKbMBZ6dI= 2023-05-31T20:38:17.633 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Wed, 31 May 2023 20:38:17 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:LQidbYpadeIoViAYtiVKbMBZ6dI='} 2023-05-31T20:38:17.639 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000009c7d9d1303b28c8d-006477b039-1630-a1'), ('Date', 'Wed, 31 May 2023 20:38:17 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:38:17.640 DEBUG:boto:path=/pklxkd-58/ 2023-05-31T20:38:17.640 DEBUG:boto:auth_path=/pklxkd-58/ 2023-05-31T20:38:17.641 DEBUG:boto:Method: HEAD 2023-05-31T20:38:17.641 DEBUG:boto:Path: /pklxkd-58/ 2023-05-31T20:38:17.642 DEBUG:boto:Data: 2023-05-31T20:38:17.642 DEBUG:boto:Headers: {} 2023-05-31T20:38:17.642 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:38:17.643 DEBUG:boto:Port: 8000 2023-05-31T20:38:17.643 DEBUG:boto:Params: {} 2023-05-31T20:38:17.643 DEBUG:boto:Token: None 2023-05-31T20:38:17.644 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:38:17 GMT /pklxkd-58/ 2023-05-31T20:38:17.644 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:jg/JHBeyhCfiZ+Bpv2lLfE21uGU= 2023-05-31T20:38:17.645 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:38:17 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:jg/JHBeyhCfiZ+Bpv2lLfE21uGU=', 'Content-Length': '0'} 2023-05-31T20:38:17.647 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '31'), ('X-RGW-Bytes-Used', '93'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000cc9e47b1a473f02e-006477b039-1630-a1'), ('Content-Length', '0'), ('Date', 'Wed, 31 May 2023 20:38:17 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:38:17.648 DEBUG:boto:path=/pklxkd-58/obj00000000000000000021 2023-05-31T20:38:17.649 DEBUG:boto:auth_path=/pklxkd-58/obj00000000000000000021 2023-05-31T20:38:17.649 DEBUG:boto:Method: PUT 2023-05-31T20:38:17.649 DEBUG:boto:Path: /pklxkd-58/obj00000000000000000021 2023-05-31T20:38:17.650 DEBUG:boto:Data: 2023-05-31T20:38:17.650 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-05-31T20:38:17.650 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:38:17.651 DEBUG:boto:Port: 8000 2023-05-31T20:38:17.651 DEBUG:boto:Params: {} 2023-05-31T20:38:17.652 DEBUG:boto:Token: None 2023-05-31T20:38:17.652 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Wed, 31 May 2023 20:38:17 GMT /pklxkd-58/obj00000000000000000021 2023-05-31T20:38:17.653 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:jp01WLFZEqJIUpgQSutRtqzMEFw= 2023-05-31T20:38:17.653 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Wed, 31 May 2023 20:38:17 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:jp01WLFZEqJIUpgQSutRtqzMEFw='} 2023-05-31T20:38:17.696 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000008391bdd01c44a09f-006477b039-1630-a1'), ('Date', 'Wed, 31 May 2023 20:38:17 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:38:17.698 DEBUG:boto:path=/pklxkd-58/ 2023-05-31T20:38:17.698 DEBUG:boto:auth_path=/pklxkd-58/ 2023-05-31T20:38:17.699 DEBUG:boto:Method: HEAD 2023-05-31T20:38:17.700 DEBUG:boto:Path: /pklxkd-58/ 2023-05-31T20:38:17.700 DEBUG:boto:Data: 2023-05-31T20:38:17.701 DEBUG:boto:Headers: {} 2023-05-31T20:38:17.702 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:38:17.702 DEBUG:boto:Port: 8000 2023-05-31T20:38:17.703 DEBUG:boto:Params: {} 2023-05-31T20:38:17.704 DEBUG:boto:Token: None 2023-05-31T20:38:17.705 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:38:17 GMT /pklxkd-58/ 2023-05-31T20:38:17.706 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:jg/JHBeyhCfiZ+Bpv2lLfE21uGU= 2023-05-31T20:38:17.706 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:38:17 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:jg/JHBeyhCfiZ+Bpv2lLfE21uGU=', 'Content-Length': '0'} 2023-05-31T20:38:17.709 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '31'), ('X-RGW-Bytes-Used', '93'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000e6c2bf345c269fdd-006477b039-1630-a1'), ('Content-Length', '0'), ('Date', 'Wed, 31 May 2023 20:38:17 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:38:17.710 DEBUG:boto:path=/pklxkd-58/obj00000000000000000014 2023-05-31T20:38:17.711 DEBUG:boto:auth_path=/pklxkd-58/obj00000000000000000014 2023-05-31T20:38:17.712 DEBUG:boto:Method: PUT 2023-05-31T20:38:17.712 DEBUG:boto:Path: /pklxkd-58/obj00000000000000000014 2023-05-31T20:38:17.713 DEBUG:boto:Data: 2023-05-31T20:38:17.715 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-05-31T20:38:17.715 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:38:17.716 DEBUG:boto:Port: 8000 2023-05-31T20:38:17.717 DEBUG:boto:Params: {} 2023-05-31T20:38:17.718 DEBUG:boto:Token: None 2023-05-31T20:38:17.718 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Wed, 31 May 2023 20:38:17 GMT /pklxkd-58/obj00000000000000000014 2023-05-31T20:38:17.719 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:AB0AAOA7zQsGZO7G0pkorCR9qm8= 2023-05-31T20:38:17.719 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Wed, 31 May 2023 20:38:17 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:AB0AAOA7zQsGZO7G0pkorCR9qm8='} 2023-05-31T20:38:17.729 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000d28bcd8c669bb891-006477b039-1630-a1'), ('Date', 'Wed, 31 May 2023 20:38:17 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:38:17.730 DEBUG:boto:path=/pklxkd-58/ 2023-05-31T20:38:17.731 DEBUG:boto:auth_path=/pklxkd-58/ 2023-05-31T20:38:17.732 DEBUG:boto:Method: HEAD 2023-05-31T20:38:17.732 DEBUG:boto:Path: /pklxkd-58/ 2023-05-31T20:38:17.733 DEBUG:boto:Data: 2023-05-31T20:38:17.733 DEBUG:boto:Headers: {} 2023-05-31T20:38:17.734 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:38:17.735 DEBUG:boto:Port: 8000 2023-05-31T20:38:17.735 DEBUG:boto:Params: {} 2023-05-31T20:38:17.736 DEBUG:boto:Token: None 2023-05-31T20:38:17.737 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:38:17 GMT /pklxkd-58/ 2023-05-31T20:38:17.737 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:jg/JHBeyhCfiZ+Bpv2lLfE21uGU= 2023-05-31T20:38:17.738 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:38:17 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:jg/JHBeyhCfiZ+Bpv2lLfE21uGU=', 'Content-Length': '0'} 2023-05-31T20:38:17.740 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '31'), ('X-RGW-Bytes-Used', '93'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000cbd59418b688ba26-006477b039-1630-a1'), ('Content-Length', '0'), ('Date', 'Wed, 31 May 2023 20:38:17 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:38:17.741 DEBUG:boto:path=/pklxkd-58/obj00000000000000000007 2023-05-31T20:38:17.742 DEBUG:boto:auth_path=/pklxkd-58/obj00000000000000000007 2023-05-31T20:38:17.743 DEBUG:boto:Method: PUT 2023-05-31T20:38:17.744 DEBUG:boto:Path: /pklxkd-58/obj00000000000000000007 2023-05-31T20:38:17.745 DEBUG:boto:Data: 2023-05-31T20:38:17.746 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-05-31T20:38:17.747 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:38:17.747 DEBUG:boto:Port: 8000 2023-05-31T20:38:17.747 DEBUG:boto:Params: {} 2023-05-31T20:38:17.748 DEBUG:boto:Token: None 2023-05-31T20:38:17.748 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Wed, 31 May 2023 20:38:17 GMT /pklxkd-58/obj00000000000000000007 2023-05-31T20:38:17.749 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:SDgpa4IydY6w+ECT+rrKyKY4aCE= 2023-05-31T20:38:17.749 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Wed, 31 May 2023 20:38:17 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:SDgpa4IydY6w+ECT+rrKyKY4aCE='} 2023-05-31T20:38:17.792 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000cf44eb7d2e706e39-006477b039-1630-a1'), ('Date', 'Wed, 31 May 2023 20:38:17 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:38:17.793 DEBUG:boto:path=/pklxkd-58/ 2023-05-31T20:38:17.794 DEBUG:boto:auth_path=/pklxkd-58/ 2023-05-31T20:38:17.794 DEBUG:boto:Method: HEAD 2023-05-31T20:38:17.794 DEBUG:boto:Path: /pklxkd-58/ 2023-05-31T20:38:17.795 DEBUG:boto:Data: 2023-05-31T20:38:17.795 DEBUG:boto:Headers: {} 2023-05-31T20:38:17.795 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:38:17.796 DEBUG:boto:Port: 8000 2023-05-31T20:38:17.796 DEBUG:boto:Params: {} 2023-05-31T20:38:17.797 DEBUG:boto:Token: None 2023-05-31T20:38:17.797 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:38:17 GMT /pklxkd-58/ 2023-05-31T20:38:17.798 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:jg/JHBeyhCfiZ+Bpv2lLfE21uGU= 2023-05-31T20:38:17.798 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:38:17 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:jg/JHBeyhCfiZ+Bpv2lLfE21uGU=', 'Content-Length': '0'} 2023-05-31T20:38:17.801 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '31'), ('X-RGW-Bytes-Used', '93'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000009b009fc21ff12042-006477b039-1630-a1'), ('Content-Length', '0'), ('Date', 'Wed, 31 May 2023 20:38:17 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:38:17.801 DEBUG:boto:path=/pklxkd-58/obj00000000000000000001 2023-05-31T20:38:17.802 DEBUG:boto:auth_path=/pklxkd-58/obj00000000000000000001 2023-05-31T20:38:17.802 DEBUG:boto:Method: PUT 2023-05-31T20:38:17.802 DEBUG:boto:Path: /pklxkd-58/obj00000000000000000001 2023-05-31T20:38:17.803 DEBUG:boto:Data: 2023-05-31T20:38:17.803 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-05-31T20:38:17.804 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:38:17.804 DEBUG:boto:Port: 8000 2023-05-31T20:38:17.804 DEBUG:boto:Params: {} 2023-05-31T20:38:17.805 DEBUG:boto:Token: None 2023-05-31T20:38:17.806 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Wed, 31 May 2023 20:38:17 GMT /pklxkd-58/obj00000000000000000001 2023-05-31T20:38:17.806 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:+qR7cxmIAO/OhvA3smwuTRICF4I= 2023-05-31T20:38:17.807 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Wed, 31 May 2023 20:38:17 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:+qR7cxmIAO/OhvA3smwuTRICF4I='} 2023-05-31T20:38:17.818 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000003d08d600aa112131-006477b039-1630-a1'), ('Date', 'Wed, 31 May 2023 20:38:17 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:38:17.818 DEBUG:boto:path=/pklxkd-58/ 2023-05-31T20:38:17.819 DEBUG:boto:auth_path=/pklxkd-58/ 2023-05-31T20:38:17.819 DEBUG:boto:Method: HEAD 2023-05-31T20:38:17.820 DEBUG:boto:Path: /pklxkd-58/ 2023-05-31T20:38:17.820 DEBUG:boto:Data: 2023-05-31T20:38:17.821 DEBUG:boto:Headers: {} 2023-05-31T20:38:17.821 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:38:17.821 DEBUG:boto:Port: 8000 2023-05-31T20:38:17.822 DEBUG:boto:Params: {} 2023-05-31T20:38:17.822 DEBUG:boto:Token: None 2023-05-31T20:38:17.823 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:38:17 GMT /pklxkd-58/ 2023-05-31T20:38:17.823 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:jg/JHBeyhCfiZ+Bpv2lLfE21uGU= 2023-05-31T20:38:17.824 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:38:17 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:jg/JHBeyhCfiZ+Bpv2lLfE21uGU=', 'Content-Length': '0'} 2023-05-31T20:38:17.827 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '31'), ('X-RGW-Bytes-Used', '93'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000f81367432377349c-006477b039-1630-a1'), ('Content-Length', '0'), ('Date', 'Wed, 31 May 2023 20:38:17 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:38:17.827 DEBUG:boto:path=/pklxkd-58/obj00000000000000000045 2023-05-31T20:38:17.828 DEBUG:boto:auth_path=/pklxkd-58/obj00000000000000000045 2023-05-31T20:38:17.828 DEBUG:boto:Method: PUT 2023-05-31T20:38:17.829 DEBUG:boto:Path: /pklxkd-58/obj00000000000000000045 2023-05-31T20:38:17.829 DEBUG:boto:Data: 2023-05-31T20:38:17.830 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-05-31T20:38:17.830 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:38:17.831 DEBUG:boto:Port: 8000 2023-05-31T20:38:17.831 DEBUG:boto:Params: {} 2023-05-31T20:38:17.831 DEBUG:boto:Token: None 2023-05-31T20:38:17.832 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Wed, 31 May 2023 20:38:17 GMT /pklxkd-58/obj00000000000000000045 2023-05-31T20:38:17.833 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:cJpNiMo8eT9CGDrtqYc3ePQ40v4= 2023-05-31T20:38:17.833 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Wed, 31 May 2023 20:38:17 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:cJpNiMo8eT9CGDrtqYc3ePQ40v4='} 2023-05-31T20:38:17.844 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000069dc3bfd4c5d82f5-006477b039-1630-a1'), ('Date', 'Wed, 31 May 2023 20:38:17 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:38:17.846 DEBUG:boto:path=/pklxkd-58/ 2023-05-31T20:38:17.846 DEBUG:boto:auth_path=/pklxkd-58/ 2023-05-31T20:38:17.847 DEBUG:boto:Method: HEAD 2023-05-31T20:38:17.848 DEBUG:boto:Path: /pklxkd-58/ 2023-05-31T20:38:17.849 DEBUG:boto:Data: 2023-05-31T20:38:17.850 DEBUG:boto:Headers: {} 2023-05-31T20:38:17.851 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:38:17.851 DEBUG:boto:Port: 8000 2023-05-31T20:38:17.852 DEBUG:boto:Params: {} 2023-05-31T20:38:17.852 DEBUG:boto:Token: None 2023-05-31T20:38:17.853 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:38:17 GMT /pklxkd-58/ 2023-05-31T20:38:17.854 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:jg/JHBeyhCfiZ+Bpv2lLfE21uGU= 2023-05-31T20:38:17.854 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:38:17 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:jg/JHBeyhCfiZ+Bpv2lLfE21uGU=', 'Content-Length': '0'} 2023-05-31T20:38:17.857 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '32'), ('X-RGW-Bytes-Used', '96'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000e9a4663429d5e8d2-006477b039-1630-a1'), ('Content-Length', '0'), ('Date', 'Wed, 31 May 2023 20:38:17 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:38:17.858 DEBUG:boto:path=/pklxkd-58/obj00000000000000000055 2023-05-31T20:38:17.859 DEBUG:boto:auth_path=/pklxkd-58/obj00000000000000000055 2023-05-31T20:38:17.859 DEBUG:boto:Method: PUT 2023-05-31T20:38:17.860 DEBUG:boto:Path: /pklxkd-58/obj00000000000000000055 2023-05-31T20:38:17.861 DEBUG:boto:Data: 2023-05-31T20:38:17.861 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-05-31T20:38:17.862 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:38:17.862 DEBUG:boto:Port: 8000 2023-05-31T20:38:17.863 DEBUG:boto:Params: {} 2023-05-31T20:38:17.863 DEBUG:boto:Token: None 2023-05-31T20:38:17.864 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Wed, 31 May 2023 20:38:17 GMT /pklxkd-58/obj00000000000000000055 2023-05-31T20:38:17.864 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:q+G/Z3EtBkV0VAAFDUEVHMQHoLs= 2023-05-31T20:38:17.865 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Wed, 31 May 2023 20:38:17 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:q+G/Z3EtBkV0VAAFDUEVHMQHoLs='} 2023-05-31T20:38:17.908 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000007aaa8d0cbb7c32d3-006477b039-1630-a1'), ('Date', 'Wed, 31 May 2023 20:38:17 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:38:17.909 DEBUG:boto:path=/pklxkd-58/ 2023-05-31T20:38:17.910 DEBUG:boto:auth_path=/pklxkd-58/ 2023-05-31T20:38:17.910 DEBUG:boto:Method: HEAD 2023-05-31T20:38:17.911 DEBUG:boto:Path: /pklxkd-58/ 2023-05-31T20:38:17.911 DEBUG:boto:Data: 2023-05-31T20:38:17.911 DEBUG:boto:Headers: {} 2023-05-31T20:38:17.912 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:38:17.912 DEBUG:boto:Port: 8000 2023-05-31T20:38:17.913 DEBUG:boto:Params: {} 2023-05-31T20:38:17.913 DEBUG:boto:Token: None 2023-05-31T20:38:17.913 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:38:17 GMT /pklxkd-58/ 2023-05-31T20:38:17.914 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:jg/JHBeyhCfiZ+Bpv2lLfE21uGU= 2023-05-31T20:38:17.915 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:38:17 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:jg/JHBeyhCfiZ+Bpv2lLfE21uGU=', 'Content-Length': '0'} 2023-05-31T20:38:17.918 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '33'), ('X-RGW-Bytes-Used', '99'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000048fecf8cfdde773a-006477b039-1630-a1'), ('Content-Length', '0'), ('Date', 'Wed, 31 May 2023 20:38:17 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:38:17.918 DEBUG:boto:path=/pklxkd-58/obj00000000000000000050 2023-05-31T20:38:17.919 DEBUG:boto:auth_path=/pklxkd-58/obj00000000000000000050 2023-05-31T20:38:17.919 DEBUG:boto:Method: PUT 2023-05-31T20:38:17.919 DEBUG:boto:Path: /pklxkd-58/obj00000000000000000050 2023-05-31T20:38:17.920 DEBUG:boto:Data: 2023-05-31T20:38:17.920 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-05-31T20:38:17.921 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:38:17.921 DEBUG:boto:Port: 8000 2023-05-31T20:38:17.921 DEBUG:boto:Params: {} 2023-05-31T20:38:17.922 DEBUG:boto:Token: None 2023-05-31T20:38:17.923 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Wed, 31 May 2023 20:38:17 GMT /pklxkd-58/obj00000000000000000050 2023-05-31T20:38:17.923 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:CMm0k5hcGSlYFxkHbSt/KbmYsDQ= 2023-05-31T20:38:17.923 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Wed, 31 May 2023 20:38:17 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:CMm0k5hcGSlYFxkHbSt/KbmYsDQ='} 2023-05-31T20:38:17.931 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000001d4d3358f997c26e-006477b039-1630-a1'), ('Date', 'Wed, 31 May 2023 20:38:17 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:38:17.931 DEBUG:boto:path=/pklxkd-58/ 2023-05-31T20:38:17.932 DEBUG:boto:auth_path=/pklxkd-58/ 2023-05-31T20:38:17.932 DEBUG:boto:Method: HEAD 2023-05-31T20:38:17.932 DEBUG:boto:Path: /pklxkd-58/ 2023-05-31T20:38:17.933 DEBUG:boto:Data: 2023-05-31T20:38:17.933 DEBUG:boto:Headers: {} 2023-05-31T20:38:17.933 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:38:17.934 DEBUG:boto:Port: 8000 2023-05-31T20:38:17.934 DEBUG:boto:Params: {} 2023-05-31T20:38:17.934 DEBUG:boto:Token: None 2023-05-31T20:38:17.935 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:38:17 GMT /pklxkd-58/ 2023-05-31T20:38:17.935 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:jg/JHBeyhCfiZ+Bpv2lLfE21uGU= 2023-05-31T20:38:17.936 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:38:17 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:jg/JHBeyhCfiZ+Bpv2lLfE21uGU=', 'Content-Length': '0'} 2023-05-31T20:38:17.939 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '34'), ('X-RGW-Bytes-Used', '102'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000007dc4411b480dadb2-006477b039-1630-a1'), ('Content-Length', '0'), ('Date', 'Wed, 31 May 2023 20:38:17 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:38:17.939 DEBUG:boto:path=/pklxkd-58/obj00000000000000000000 2023-05-31T20:38:17.940 DEBUG:boto:auth_path=/pklxkd-58/obj00000000000000000000 2023-05-31T20:38:17.940 DEBUG:boto:Method: PUT 2023-05-31T20:38:17.940 DEBUG:boto:Path: /pklxkd-58/obj00000000000000000000 2023-05-31T20:38:17.941 DEBUG:boto:Data: 2023-05-31T20:38:17.941 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-05-31T20:38:17.941 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:38:17.942 DEBUG:boto:Port: 8000 2023-05-31T20:38:17.942 DEBUG:boto:Params: {} 2023-05-31T20:38:17.942 DEBUG:boto:Token: None 2023-05-31T20:38:17.943 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Wed, 31 May 2023 20:38:17 GMT /pklxkd-58/obj00000000000000000000 2023-05-31T20:38:17.943 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:5t0L/Mn38HmYYZaf8bWazhK21fE= 2023-05-31T20:38:17.944 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Wed, 31 May 2023 20:38:17 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:5t0L/Mn38HmYYZaf8bWazhK21fE='} 2023-05-31T20:38:17.988 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000bb301f8f3c125b2d-006477b039-1630-a1'), ('Date', 'Wed, 31 May 2023 20:38:17 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:38:17.989 DEBUG:boto:path=/pklxkd-58/ 2023-05-31T20:38:17.990 DEBUG:boto:auth_path=/pklxkd-58/ 2023-05-31T20:38:17.990 DEBUG:boto:Method: HEAD 2023-05-31T20:38:17.990 DEBUG:boto:Path: /pklxkd-58/ 2023-05-31T20:38:17.991 DEBUG:boto:Data: 2023-05-31T20:38:17.991 DEBUG:boto:Headers: {} 2023-05-31T20:38:17.991 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:38:17.992 DEBUG:boto:Port: 8000 2023-05-31T20:38:17.992 DEBUG:boto:Params: {} 2023-05-31T20:38:17.992 DEBUG:boto:Token: None 2023-05-31T20:38:17.993 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:38:17 GMT /pklxkd-58/ 2023-05-31T20:38:17.993 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:jg/JHBeyhCfiZ+Bpv2lLfE21uGU= 2023-05-31T20:38:17.994 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:38:17 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:jg/JHBeyhCfiZ+Bpv2lLfE21uGU=', 'Content-Length': '0'} 2023-05-31T20:38:17.997 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '34'), ('X-RGW-Bytes-Used', '102'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000007ff62989746b29be-006477b039-1630-a1'), ('Content-Length', '0'), ('Date', 'Wed, 31 May 2023 20:38:17 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:38:17.997 DEBUG:boto:path=/pklxkd-58/obj00000000000000000003 2023-05-31T20:38:17.998 DEBUG:boto:auth_path=/pklxkd-58/obj00000000000000000003 2023-05-31T20:38:17.998 DEBUG:boto:Method: PUT 2023-05-31T20:38:17.999 DEBUG:boto:Path: /pklxkd-58/obj00000000000000000003 2023-05-31T20:38:18.000 DEBUG:boto:Data: 2023-05-31T20:38:18.001 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-05-31T20:38:18.002 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:38:18.003 DEBUG:boto:Port: 8000 2023-05-31T20:38:18.003 DEBUG:boto:Params: {} 2023-05-31T20:38:18.005 DEBUG:boto:Token: None 2023-05-31T20:38:18.006 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Wed, 31 May 2023 20:38:18 GMT /pklxkd-58/obj00000000000000000003 2023-05-31T20:38:18.007 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:AKRkgMylJ7Mnm9jz0+xJs4RKm+o= 2023-05-31T20:38:18.009 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Wed, 31 May 2023 20:38:18 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:AKRkgMylJ7Mnm9jz0+xJs4RKm+o='} 2023-05-31T20:38:18.018 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000dcbf1a4919b4a8aa-006477b03a-1630-a1'), ('Date', 'Wed, 31 May 2023 20:38:18 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:38:18.020 DEBUG:boto:path=/pklxkd-58/ 2023-05-31T20:38:18.021 DEBUG:boto:auth_path=/pklxkd-58/ 2023-05-31T20:38:18.021 DEBUG:boto:Method: HEAD 2023-05-31T20:38:18.022 DEBUG:boto:Path: /pklxkd-58/ 2023-05-31T20:38:18.022 DEBUG:boto:Data: 2023-05-31T20:38:18.023 DEBUG:boto:Headers: {} 2023-05-31T20:38:18.023 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:38:18.024 DEBUG:boto:Port: 8000 2023-05-31T20:38:18.024 DEBUG:boto:Params: {} 2023-05-31T20:38:18.025 DEBUG:boto:Token: None 2023-05-31T20:38:18.025 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:38:18 GMT /pklxkd-58/ 2023-05-31T20:38:18.026 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:mPIo1OW7xVEcvBZnJtgeMHBCsAQ= 2023-05-31T20:38:18.027 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:38:18 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:mPIo1OW7xVEcvBZnJtgeMHBCsAQ=', 'Content-Length': '0'} 2023-05-31T20:38:18.029 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '34'), ('X-RGW-Bytes-Used', '102'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000a7bb7c10ecb14927-006477b03a-1630-a1'), ('Content-Length', '0'), ('Date', 'Wed, 31 May 2023 20:38:18 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:38:18.030 DEBUG:boto:path=/pklxkd-58/obj00000000000000000006 2023-05-31T20:38:18.031 DEBUG:boto:auth_path=/pklxkd-58/obj00000000000000000006 2023-05-31T20:38:18.032 DEBUG:boto:Method: PUT 2023-05-31T20:38:18.032 DEBUG:boto:Path: /pklxkd-58/obj00000000000000000006 2023-05-31T20:38:18.033 DEBUG:boto:Data: 2023-05-31T20:38:18.033 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-05-31T20:38:18.034 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:38:18.034 DEBUG:boto:Port: 8000 2023-05-31T20:38:18.034 DEBUG:boto:Params: {} 2023-05-31T20:38:18.035 DEBUG:boto:Token: None 2023-05-31T20:38:18.035 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Wed, 31 May 2023 20:38:18 GMT /pklxkd-58/obj00000000000000000006 2023-05-31T20:38:18.036 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:+7uZ4yyUS2AgFDwe2yrIlyoVdAE= 2023-05-31T20:38:18.037 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Wed, 31 May 2023 20:38:18 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:+7uZ4yyUS2AgFDwe2yrIlyoVdAE='} 2023-05-31T20:38:18.081 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000522152fc4b1a8d54-006477b03a-1630-a1'), ('Date', 'Wed, 31 May 2023 20:38:18 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:38:18.082 DEBUG:boto:path=/pklxkd-58/ 2023-05-31T20:38:18.082 DEBUG:boto:auth_path=/pklxkd-58/ 2023-05-31T20:38:18.083 DEBUG:boto:Method: HEAD 2023-05-31T20:38:18.084 DEBUG:boto:Path: /pklxkd-58/ 2023-05-31T20:38:18.084 DEBUG:boto:Data: 2023-05-31T20:38:18.085 DEBUG:boto:Headers: {} 2023-05-31T20:38:18.085 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:38:18.086 DEBUG:boto:Port: 8000 2023-05-31T20:38:18.087 DEBUG:boto:Params: {} 2023-05-31T20:38:18.087 DEBUG:boto:Token: None 2023-05-31T20:38:18.088 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:38:18 GMT /pklxkd-58/ 2023-05-31T20:38:18.088 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:mPIo1OW7xVEcvBZnJtgeMHBCsAQ= 2023-05-31T20:38:18.089 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:38:18 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:mPIo1OW7xVEcvBZnJtgeMHBCsAQ=', 'Content-Length': '0'} 2023-05-31T20:38:18.092 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '34'), ('X-RGW-Bytes-Used', '102'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000003215d3dda5ba4d32-006477b03a-1630-a1'), ('Content-Length', '0'), ('Date', 'Wed, 31 May 2023 20:38:18 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:38:18.092 DEBUG:boto:path=/pklxkd-58/obj00000000000000000012 2023-05-31T20:38:18.093 DEBUG:boto:auth_path=/pklxkd-58/obj00000000000000000012 2023-05-31T20:38:18.093 DEBUG:boto:Method: PUT 2023-05-31T20:38:18.093 DEBUG:boto:Path: /pklxkd-58/obj00000000000000000012 2023-05-31T20:38:18.094 DEBUG:boto:Data: 2023-05-31T20:38:18.094 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-05-31T20:38:18.094 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:38:18.095 DEBUG:boto:Port: 8000 2023-05-31T20:38:18.096 DEBUG:boto:Params: {} 2023-05-31T20:38:18.096 DEBUG:boto:Token: None 2023-05-31T20:38:18.096 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Wed, 31 May 2023 20:38:18 GMT /pklxkd-58/obj00000000000000000012 2023-05-31T20:38:18.097 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:/RG34Q7moqpoaUqBK3UPpTfjgjA= 2023-05-31T20:38:18.097 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Wed, 31 May 2023 20:38:18 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:/RG34Q7moqpoaUqBK3UPpTfjgjA='} 2023-05-31T20:38:18.105 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000f1d60b926fcf1a0b-006477b03a-1630-a1'), ('Date', 'Wed, 31 May 2023 20:38:18 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:38:18.106 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'reshard', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'pklxkd-58', '--num-shards', '29', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-05-31T20:38:18.106 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'reshard', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'pklxkd-58', '--num-shards', '29', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-05-31T20:38:18.107 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket reshard --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --bucket pklxkd-58 --num-shards 29 --yes-i-really-mean-it --cluster c1 --debug-rgw 0 --debug-ms 0 2023-05-31T20:38:18.160 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2023-05-31T20:38:18.160 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2023-05-31T20:38:18.230 INFO:teuthology.orchestra.run.smithi099.stdout:tenant: 2023-05-31T20:38:18.230 INFO:teuthology.orchestra.run.smithi099.stdout:bucket name: pklxkd-58 2023-05-31T20:38:18.239 INFO:teuthology.orchestra.run.smithi099.stdout:total entries: 34 2023-05-31T20:38:18.292 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'shard', 'objects', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--num-shards', '29', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-31T20:38:18.293 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'shard', 'objects', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--num-shards', '29', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-31T20:38:18.293 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket shard objects --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --num-shards 29 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-05-31T20:38:18.343 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2023-05-31T20:38:18.344 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2023-05-31T20:38:18.412 INFO:teuthology.orchestra.run.smithi099.stdout:{ 2023-05-31T20:38:18.413 INFO:teuthology.orchestra.run.smithi099.stdout: "objs": [ 2023-05-31T20:38:18.413 INFO:teuthology.orchestra.run.smithi099.stdout: "obj00000000000000000006", 2023-05-31T20:38:18.413 INFO:teuthology.orchestra.run.smithi099.stdout: "obj00000000000000000059", 2023-05-31T20:38:18.414 INFO:teuthology.orchestra.run.smithi099.stdout: "obj00000000000000000027", 2023-05-31T20:38:18.414 INFO:teuthology.orchestra.run.smithi099.stdout: "obj00000000000000000004", 2023-05-31T20:38:18.414 INFO:teuthology.orchestra.run.smithi099.stdout: "obj00000000000000000011", 2023-05-31T20:38:18.415 INFO:teuthology.orchestra.run.smithi099.stdout: "obj00000000000000000002", 2023-05-31T20:38:18.415 INFO:teuthology.orchestra.run.smithi099.stdout: "obj00000000000000000019", 2023-05-31T20:38:18.415 INFO:teuthology.orchestra.run.smithi099.stdout: "obj00000000000000000049", 2023-05-31T20:38:18.415 INFO:teuthology.orchestra.run.smithi099.stdout: "obj00000000000000000017", 2023-05-31T20:38:18.416 INFO:teuthology.orchestra.run.smithi099.stdout: "obj00000000000000000008", 2023-05-31T20:38:18.416 INFO:teuthology.orchestra.run.smithi099.stdout: "obj00000000000000000015", 2023-05-31T20:38:18.416 INFO:teuthology.orchestra.run.smithi099.stdout: "obj00000000000000000031", 2023-05-31T20:38:18.417 INFO:teuthology.orchestra.run.smithi099.stdout: "obj00000000000000000022", 2023-05-31T20:38:18.417 INFO:teuthology.orchestra.run.smithi099.stdout: "obj00000000000000000013", 2023-05-31T20:38:18.417 INFO:teuthology.orchestra.run.smithi099.stdout: "obj00000000000000000020", 2023-05-31T20:38:18.418 INFO:teuthology.orchestra.run.smithi099.stdout: "obj00000000000000000037", 2023-05-31T20:38:18.418 INFO:teuthology.orchestra.run.smithi099.stdout: "obj00000000000000000005", 2023-05-31T20:38:18.418 INFO:teuthology.orchestra.run.smithi099.stdout: "obj00000000000000000012", 2023-05-31T20:38:18.419 INFO:teuthology.orchestra.run.smithi099.stdout: "obj00000000000000000003", 2023-05-31T20:38:18.419 INFO:teuthology.orchestra.run.smithi099.stdout: "obj00000000000000000042", 2023-05-31T20:38:18.419 INFO:teuthology.orchestra.run.smithi099.stdout: "obj00000000000000000010", 2023-05-31T20:38:18.420 INFO:teuthology.orchestra.run.smithi099.stdout: "obj00000000000000000001", 2023-05-31T20:38:18.420 INFO:teuthology.orchestra.run.smithi099.stdout: "obj00000000000000000009", 2023-05-31T20:38:18.420 INFO:teuthology.orchestra.run.smithi099.stdout: "obj00000000000000000025", 2023-05-31T20:38:18.420 INFO:teuthology.orchestra.run.smithi099.stdout: "obj00000000000000000016", 2023-05-31T20:38:18.420 INFO:teuthology.orchestra.run.smithi099.stdout: "obj00000000000000000000", 2023-05-31T20:38:18.420 INFO:teuthology.orchestra.run.smithi099.stdout: "obj00000000000000000014", 2023-05-31T20:38:18.421 INFO:teuthology.orchestra.run.smithi099.stdout: "obj00000000000000000030", 2023-05-31T20:38:18.421 INFO:teuthology.orchestra.run.smithi099.stdout: "obj00000000000000000021" 2023-05-31T20:38:18.421 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2023-05-31T20:38:18.421 INFO:teuthology.orchestra.run.smithi099.stdout:} 2023-05-31T20:38:18.422 DEBUG:boto:path=/pklxkd-58/ 2023-05-31T20:38:18.423 DEBUG:boto:auth_path=/pklxkd-58/ 2023-05-31T20:38:18.423 DEBUG:boto:Method: HEAD 2023-05-31T20:38:18.424 DEBUG:boto:Path: /pklxkd-58/ 2023-05-31T20:38:18.424 DEBUG:boto:Data: 2023-05-31T20:38:18.424 DEBUG:boto:Headers: {} 2023-05-31T20:38:18.425 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:38:18.425 DEBUG:boto:Port: 8000 2023-05-31T20:38:18.426 DEBUG:boto:Params: {} 2023-05-31T20:38:18.426 DEBUG:boto:Token: None 2023-05-31T20:38:18.426 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:38:18 GMT /pklxkd-58/ 2023-05-31T20:38:18.427 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:mPIo1OW7xVEcvBZnJtgeMHBCsAQ= 2023-05-31T20:38:18.427 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:38:18 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:mPIo1OW7xVEcvBZnJtgeMHBCsAQ=', 'Content-Length': '0'} 2023-05-31T20:38:18.431 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '34'), ('X-RGW-Bytes-Used', '102'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000b77e2af49a765194-006477b03a-1630-a1'), ('Content-Length', '0'), ('Date', 'Wed, 31 May 2023 20:38:18 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:38:18.431 DEBUG:boto:path=/pklxkd-58/obj00000000000000000031 2023-05-31T20:38:18.433 DEBUG:boto:auth_path=/pklxkd-58/obj00000000000000000031 2023-05-31T20:38:18.434 DEBUG:boto:Method: PUT 2023-05-31T20:38:18.434 DEBUG:boto:Path: /pklxkd-58/obj00000000000000000031 2023-05-31T20:38:18.435 DEBUG:boto:Data: 2023-05-31T20:38:18.436 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-05-31T20:38:18.437 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:38:18.438 DEBUG:boto:Port: 8000 2023-05-31T20:38:18.439 DEBUG:boto:Params: {} 2023-05-31T20:38:18.440 DEBUG:boto:Token: None 2023-05-31T20:38:18.440 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Wed, 31 May 2023 20:38:18 GMT /pklxkd-58/obj00000000000000000031 2023-05-31T20:38:18.441 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:eiOsOW+ORKESxdArYRLaq3QRSA8= 2023-05-31T20:38:18.442 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Wed, 31 May 2023 20:38:18 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:eiOsOW+ORKESxdArYRLaq3QRSA8='} 2023-05-31T20:38:18.451 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000172cdfbea977c3a2-006477b03a-1630-a1'), ('Date', 'Wed, 31 May 2023 20:38:18 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:38:18.452 DEBUG:boto:path=/pklxkd-58/ 2023-05-31T20:38:18.453 DEBUG:boto:auth_path=/pklxkd-58/ 2023-05-31T20:38:18.453 DEBUG:boto:Method: HEAD 2023-05-31T20:38:18.453 DEBUG:boto:Path: /pklxkd-58/ 2023-05-31T20:38:18.454 DEBUG:boto:Data: 2023-05-31T20:38:18.454 DEBUG:boto:Headers: {} 2023-05-31T20:38:18.455 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:38:18.455 DEBUG:boto:Port: 8000 2023-05-31T20:38:18.455 DEBUG:boto:Params: {} 2023-05-31T20:38:18.456 DEBUG:boto:Token: None 2023-05-31T20:38:18.456 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:38:18 GMT /pklxkd-58/ 2023-05-31T20:38:18.457 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:mPIo1OW7xVEcvBZnJtgeMHBCsAQ= 2023-05-31T20:38:18.457 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:38:18 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:mPIo1OW7xVEcvBZnJtgeMHBCsAQ=', 'Content-Length': '0'} 2023-05-31T20:38:18.460 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '35'), ('X-RGW-Bytes-Used', '105'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000002f3458c4ee8f2ba9-006477b03a-1630-a1'), ('Content-Length', '0'), ('Date', 'Wed, 31 May 2023 20:38:18 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:38:18.461 DEBUG:boto:path=/pklxkd-58/obj00000000000000000017 2023-05-31T20:38:18.461 DEBUG:boto:auth_path=/pklxkd-58/obj00000000000000000017 2023-05-31T20:38:18.461 DEBUG:boto:Method: PUT 2023-05-31T20:38:18.462 DEBUG:boto:Path: /pklxkd-58/obj00000000000000000017 2023-05-31T20:38:18.462 DEBUG:boto:Data: 2023-05-31T20:38:18.463 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-05-31T20:38:18.463 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:38:18.463 DEBUG:boto:Port: 8000 2023-05-31T20:38:18.464 DEBUG:boto:Params: {} 2023-05-31T20:38:18.464 DEBUG:boto:Token: None 2023-05-31T20:38:18.464 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Wed, 31 May 2023 20:38:18 GMT /pklxkd-58/obj00000000000000000017 2023-05-31T20:38:18.465 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:XQfBxFNgDnXe+8OwWufpx3fBacY= 2023-05-31T20:38:18.465 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Wed, 31 May 2023 20:38:18 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:XQfBxFNgDnXe+8OwWufpx3fBacY='} 2023-05-31T20:38:18.512 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000b98e24a1dfd84859-006477b03a-1630-a1'), ('Date', 'Wed, 31 May 2023 20:38:18 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:38:18.513 DEBUG:boto:path=/pklxkd-58/ 2023-05-31T20:38:18.514 DEBUG:boto:auth_path=/pklxkd-58/ 2023-05-31T20:38:18.514 DEBUG:boto:Method: HEAD 2023-05-31T20:38:18.514 DEBUG:boto:Path: /pklxkd-58/ 2023-05-31T20:38:18.515 DEBUG:boto:Data: 2023-05-31T20:38:18.515 DEBUG:boto:Headers: {} 2023-05-31T20:38:18.515 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:38:18.516 DEBUG:boto:Port: 8000 2023-05-31T20:38:18.516 DEBUG:boto:Params: {} 2023-05-31T20:38:18.516 DEBUG:boto:Token: None 2023-05-31T20:38:18.517 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:38:18 GMT /pklxkd-58/ 2023-05-31T20:38:18.517 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:mPIo1OW7xVEcvBZnJtgeMHBCsAQ= 2023-05-31T20:38:18.518 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:38:18 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:mPIo1OW7xVEcvBZnJtgeMHBCsAQ=', 'Content-Length': '0'} 2023-05-31T20:38:18.520 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '36'), ('X-RGW-Bytes-Used', '108'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000ec75ead0b1425673-006477b03a-1630-a1'), ('Content-Length', '0'), ('Date', 'Wed, 31 May 2023 20:38:18 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:38:18.522 DEBUG:boto:path=/pklxkd-58/obj00000000000000000049 2023-05-31T20:38:18.522 DEBUG:boto:auth_path=/pklxkd-58/obj00000000000000000049 2023-05-31T20:38:18.523 DEBUG:boto:Method: PUT 2023-05-31T20:38:18.524 DEBUG:boto:Path: /pklxkd-58/obj00000000000000000049 2023-05-31T20:38:18.525 DEBUG:boto:Data: 2023-05-31T20:38:18.526 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-05-31T20:38:18.526 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:38:18.527 DEBUG:boto:Port: 8000 2023-05-31T20:38:18.527 DEBUG:boto:Params: {} 2023-05-31T20:38:18.528 DEBUG:boto:Token: None 2023-05-31T20:38:18.528 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Wed, 31 May 2023 20:38:18 GMT /pklxkd-58/obj00000000000000000049 2023-05-31T20:38:18.529 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:askDEQg1brceNUfOQuMTLaUr/bo= 2023-05-31T20:38:18.529 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Wed, 31 May 2023 20:38:18 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:askDEQg1brceNUfOQuMTLaUr/bo='} 2023-05-31T20:38:18.540 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000009facce13956a7ec-006477b03a-1630-a1'), ('Date', 'Wed, 31 May 2023 20:38:18 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:38:18.541 DEBUG:boto:path=/pklxkd-58/ 2023-05-31T20:38:18.542 DEBUG:boto:auth_path=/pklxkd-58/ 2023-05-31T20:38:18.543 DEBUG:boto:Method: HEAD 2023-05-31T20:38:18.544 DEBUG:boto:Path: /pklxkd-58/ 2023-05-31T20:38:18.545 DEBUG:boto:Data: 2023-05-31T20:38:18.546 DEBUG:boto:Headers: {} 2023-05-31T20:38:18.546 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:38:18.547 DEBUG:boto:Port: 8000 2023-05-31T20:38:18.548 DEBUG:boto:Params: {} 2023-05-31T20:38:18.548 DEBUG:boto:Token: None 2023-05-31T20:38:18.549 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:38:18 GMT /pklxkd-58/ 2023-05-31T20:38:18.550 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:mPIo1OW7xVEcvBZnJtgeMHBCsAQ= 2023-05-31T20:38:18.550 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:38:18 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:mPIo1OW7xVEcvBZnJtgeMHBCsAQ=', 'Content-Length': '0'} 2023-05-31T20:38:18.553 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '37'), ('X-RGW-Bytes-Used', '111'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000a58c3b1f1cccaaf5-006477b03a-1630-a1'), ('Content-Length', '0'), ('Date', 'Wed, 31 May 2023 20:38:18 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:38:18.554 DEBUG:boto:path=/pklxkd-58/obj00000000000000000059 2023-05-31T20:38:18.555 DEBUG:boto:auth_path=/pklxkd-58/obj00000000000000000059 2023-05-31T20:38:18.555 DEBUG:boto:Method: PUT 2023-05-31T20:38:18.556 DEBUG:boto:Path: /pklxkd-58/obj00000000000000000059 2023-05-31T20:38:18.557 DEBUG:boto:Data: 2023-05-31T20:38:18.557 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-05-31T20:38:18.558 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:38:18.559 DEBUG:boto:Port: 8000 2023-05-31T20:38:18.560 DEBUG:boto:Params: {} 2023-05-31T20:38:18.561 DEBUG:boto:Token: None 2023-05-31T20:38:18.561 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Wed, 31 May 2023 20:38:18 GMT /pklxkd-58/obj00000000000000000059 2023-05-31T20:38:18.562 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:ULMIwBh7QwwrP+pXACAdNjbuwRk= 2023-05-31T20:38:18.562 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Wed, 31 May 2023 20:38:18 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:ULMIwBh7QwwrP+pXACAdNjbuwRk='} 2023-05-31T20:38:18.604 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000abd7cd301a639d54-006477b03a-1630-a1'), ('Date', 'Wed, 31 May 2023 20:38:18 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:38:18.606 DEBUG:boto:path=/pklxkd-58/ 2023-05-31T20:38:18.606 DEBUG:boto:auth_path=/pklxkd-58/ 2023-05-31T20:38:18.607 DEBUG:boto:Method: HEAD 2023-05-31T20:38:18.608 DEBUG:boto:Path: /pklxkd-58/ 2023-05-31T20:38:18.609 DEBUG:boto:Data: 2023-05-31T20:38:18.609 DEBUG:boto:Headers: {} 2023-05-31T20:38:18.610 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:38:18.611 DEBUG:boto:Port: 8000 2023-05-31T20:38:18.612 DEBUG:boto:Params: {} 2023-05-31T20:38:18.613 DEBUG:boto:Token: None 2023-05-31T20:38:18.613 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:38:18 GMT /pklxkd-58/ 2023-05-31T20:38:18.614 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:mPIo1OW7xVEcvBZnJtgeMHBCsAQ= 2023-05-31T20:38:18.615 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:38:18 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:mPIo1OW7xVEcvBZnJtgeMHBCsAQ=', 'Content-Length': '0'} 2023-05-31T20:38:18.618 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '38'), ('X-RGW-Bytes-Used', '114'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000005cec61b3d254bff1-006477b03a-1630-a1'), ('Content-Length', '0'), ('Date', 'Wed, 31 May 2023 20:38:18 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:38:18.619 DEBUG:boto:path=/pklxkd-58/obj00000000000000000010 2023-05-31T20:38:18.620 DEBUG:boto:auth_path=/pklxkd-58/obj00000000000000000010 2023-05-31T20:38:18.620 DEBUG:boto:Method: PUT 2023-05-31T20:38:18.621 DEBUG:boto:Path: /pklxkd-58/obj00000000000000000010 2023-05-31T20:38:18.621 DEBUG:boto:Data: 2023-05-31T20:38:18.622 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-05-31T20:38:18.622 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:38:18.623 DEBUG:boto:Port: 8000 2023-05-31T20:38:18.623 DEBUG:boto:Params: {} 2023-05-31T20:38:18.624 DEBUG:boto:Token: None 2023-05-31T20:38:18.624 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Wed, 31 May 2023 20:38:18 GMT /pklxkd-58/obj00000000000000000010 2023-05-31T20:38:18.625 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:hDZKe3p5GKAcOdmc/AYNSfi1w+w= 2023-05-31T20:38:18.626 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Wed, 31 May 2023 20:38:18 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:hDZKe3p5GKAcOdmc/AYNSfi1w+w='} 2023-05-31T20:38:18.640 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000bf035c77ef4cf3ce-006477b03a-1630-a1'), ('Date', 'Wed, 31 May 2023 20:38:18 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:38:18.641 DEBUG:boto:path=/pklxkd-58/ 2023-05-31T20:38:18.641 DEBUG:boto:auth_path=/pklxkd-58/ 2023-05-31T20:38:18.642 DEBUG:boto:Method: HEAD 2023-05-31T20:38:18.642 DEBUG:boto:Path: /pklxkd-58/ 2023-05-31T20:38:18.642 DEBUG:boto:Data: 2023-05-31T20:38:18.643 DEBUG:boto:Headers: {} 2023-05-31T20:38:18.643 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:38:18.643 DEBUG:boto:Port: 8000 2023-05-31T20:38:18.644 DEBUG:boto:Params: {} 2023-05-31T20:38:18.644 DEBUG:boto:Token: None 2023-05-31T20:38:18.645 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:38:18 GMT /pklxkd-58/ 2023-05-31T20:38:18.645 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:mPIo1OW7xVEcvBZnJtgeMHBCsAQ= 2023-05-31T20:38:18.645 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:38:18 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:mPIo1OW7xVEcvBZnJtgeMHBCsAQ=', 'Content-Length': '0'} 2023-05-31T20:38:18.648 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '38'), ('X-RGW-Bytes-Used', '114'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000ff9970cce7175a92-006477b03a-1630-a1'), ('Content-Length', '0'), ('Date', 'Wed, 31 May 2023 20:38:18 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:38:18.650 DEBUG:boto:path=/pklxkd-58/obj00000000000000000019 2023-05-31T20:38:18.650 DEBUG:boto:auth_path=/pklxkd-58/obj00000000000000000019 2023-05-31T20:38:18.651 DEBUG:boto:Method: PUT 2023-05-31T20:38:18.652 DEBUG:boto:Path: /pklxkd-58/obj00000000000000000019 2023-05-31T20:38:18.653 DEBUG:boto:Data: 2023-05-31T20:38:18.654 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-05-31T20:38:18.655 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:38:18.656 DEBUG:boto:Port: 8000 2023-05-31T20:38:18.657 DEBUG:boto:Params: {} 2023-05-31T20:38:18.657 DEBUG:boto:Token: None 2023-05-31T20:38:18.658 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Wed, 31 May 2023 20:38:18 GMT /pklxkd-58/obj00000000000000000019 2023-05-31T20:38:18.658 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:TchnSJrIYbpR4jral4/EBoYSSRU= 2023-05-31T20:38:18.660 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Wed, 31 May 2023 20:38:18 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:TchnSJrIYbpR4jral4/EBoYSSRU='} 2023-05-31T20:38:18.708 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000004e92845430129378-006477b03a-1630-a1'), ('Date', 'Wed, 31 May 2023 20:38:18 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:38:18.710 DEBUG:boto:path=/pklxkd-58/ 2023-05-31T20:38:18.711 DEBUG:boto:auth_path=/pklxkd-58/ 2023-05-31T20:38:18.712 DEBUG:boto:Method: HEAD 2023-05-31T20:38:18.713 DEBUG:boto:Path: /pklxkd-58/ 2023-05-31T20:38:18.714 DEBUG:boto:Data: 2023-05-31T20:38:18.714 DEBUG:boto:Headers: {} 2023-05-31T20:38:18.715 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:38:18.715 DEBUG:boto:Port: 8000 2023-05-31T20:38:18.716 DEBUG:boto:Params: {} 2023-05-31T20:38:18.717 DEBUG:boto:Token: None 2023-05-31T20:38:18.718 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:38:18 GMT /pklxkd-58/ 2023-05-31T20:38:18.718 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:mPIo1OW7xVEcvBZnJtgeMHBCsAQ= 2023-05-31T20:38:18.719 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:38:18 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:mPIo1OW7xVEcvBZnJtgeMHBCsAQ=', 'Content-Length': '0'} 2023-05-31T20:38:18.722 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '38'), ('X-RGW-Bytes-Used', '114'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000001398d0f35d1956a3-006477b03a-1630-a1'), ('Content-Length', '0'), ('Date', 'Wed, 31 May 2023 20:38:18 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:38:18.723 DEBUG:boto:path=/pklxkd-58/obj00000000000000000004 2023-05-31T20:38:18.723 DEBUG:boto:auth_path=/pklxkd-58/obj00000000000000000004 2023-05-31T20:38:18.724 DEBUG:boto:Method: PUT 2023-05-31T20:38:18.724 DEBUG:boto:Path: /pklxkd-58/obj00000000000000000004 2023-05-31T20:38:18.725 DEBUG:boto:Data: 2023-05-31T20:38:18.725 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-05-31T20:38:18.726 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:38:18.726 DEBUG:boto:Port: 8000 2023-05-31T20:38:18.727 DEBUG:boto:Params: {} 2023-05-31T20:38:18.727 DEBUG:boto:Token: None 2023-05-31T20:38:18.728 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Wed, 31 May 2023 20:38:18 GMT /pklxkd-58/obj00000000000000000004 2023-05-31T20:38:18.728 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:Fw6dvohOsL6SskbKGcRb4rtf72M= 2023-05-31T20:38:18.729 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Wed, 31 May 2023 20:38:18 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:Fw6dvohOsL6SskbKGcRb4rtf72M='} 2023-05-31T20:38:18.739 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000007ff5809b9dc8cc97-006477b03a-1630-a1'), ('Date', 'Wed, 31 May 2023 20:38:18 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:38:18.740 DEBUG:boto:path=/pklxkd-58/ 2023-05-31T20:38:18.740 DEBUG:boto:auth_path=/pklxkd-58/ 2023-05-31T20:38:18.741 DEBUG:boto:Method: HEAD 2023-05-31T20:38:18.741 DEBUG:boto:Path: /pklxkd-58/ 2023-05-31T20:38:18.742 DEBUG:boto:Data: 2023-05-31T20:38:18.742 DEBUG:boto:Headers: {} 2023-05-31T20:38:18.743 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:38:18.743 DEBUG:boto:Port: 8000 2023-05-31T20:38:18.744 DEBUG:boto:Params: {} 2023-05-31T20:38:18.744 DEBUG:boto:Token: None 2023-05-31T20:38:18.745 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:38:18 GMT /pklxkd-58/ 2023-05-31T20:38:18.746 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:mPIo1OW7xVEcvBZnJtgeMHBCsAQ= 2023-05-31T20:38:18.747 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:38:18 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:mPIo1OW7xVEcvBZnJtgeMHBCsAQ=', 'Content-Length': '0'} 2023-05-31T20:38:18.750 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '38'), ('X-RGW-Bytes-Used', '114'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000024409fc60c7129b7-006477b03a-1630-a1'), ('Content-Length', '0'), ('Date', 'Wed, 31 May 2023 20:38:18 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:38:18.751 DEBUG:boto:path=/pklxkd-58/obj00000000000000000021 2023-05-31T20:38:18.752 DEBUG:boto:auth_path=/pklxkd-58/obj00000000000000000021 2023-05-31T20:38:18.752 DEBUG:boto:Method: PUT 2023-05-31T20:38:18.753 DEBUG:boto:Path: /pklxkd-58/obj00000000000000000021 2023-05-31T20:38:18.753 DEBUG:boto:Data: 2023-05-31T20:38:18.754 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-05-31T20:38:18.755 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:38:18.755 DEBUG:boto:Port: 8000 2023-05-31T20:38:18.755 DEBUG:boto:Params: {} 2023-05-31T20:38:18.756 DEBUG:boto:Token: None 2023-05-31T20:38:18.756 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Wed, 31 May 2023 20:38:18 GMT /pklxkd-58/obj00000000000000000021 2023-05-31T20:38:18.757 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:SNwWtWtHK6POUoYoS3ikOn8g4dM= 2023-05-31T20:38:18.758 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Wed, 31 May 2023 20:38:18 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:SNwWtWtHK6POUoYoS3ikOn8g4dM='} 2023-05-31T20:38:18.804 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000008b02e6aa2ae20545-006477b03a-1630-a1'), ('Date', 'Wed, 31 May 2023 20:38:18 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:38:18.805 DEBUG:boto:path=/pklxkd-58/ 2023-05-31T20:38:18.806 DEBUG:boto:auth_path=/pklxkd-58/ 2023-05-31T20:38:18.806 DEBUG:boto:Method: HEAD 2023-05-31T20:38:18.807 DEBUG:boto:Path: /pklxkd-58/ 2023-05-31T20:38:18.807 DEBUG:boto:Data: 2023-05-31T20:38:18.807 DEBUG:boto:Headers: {} 2023-05-31T20:38:18.808 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:38:18.808 DEBUG:boto:Port: 8000 2023-05-31T20:38:18.809 DEBUG:boto:Params: {} 2023-05-31T20:38:18.809 DEBUG:boto:Token: None 2023-05-31T20:38:18.810 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:38:18 GMT /pklxkd-58/ 2023-05-31T20:38:18.810 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:mPIo1OW7xVEcvBZnJtgeMHBCsAQ= 2023-05-31T20:38:18.811 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:38:18 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:mPIo1OW7xVEcvBZnJtgeMHBCsAQ=', 'Content-Length': '0'} 2023-05-31T20:38:18.814 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '38'), ('X-RGW-Bytes-Used', '114'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000049b49674b733325-006477b03a-1630-a1'), ('Content-Length', '0'), ('Date', 'Wed, 31 May 2023 20:38:18 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:38:18.815 DEBUG:boto:path=/pklxkd-58/obj00000000000000000015 2023-05-31T20:38:18.815 DEBUG:boto:auth_path=/pklxkd-58/obj00000000000000000015 2023-05-31T20:38:18.816 DEBUG:boto:Method: PUT 2023-05-31T20:38:18.816 DEBUG:boto:Path: /pklxkd-58/obj00000000000000000015 2023-05-31T20:38:18.817 DEBUG:boto:Data: 2023-05-31T20:38:18.817 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-05-31T20:38:18.817 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:38:18.818 DEBUG:boto:Port: 8000 2023-05-31T20:38:18.818 DEBUG:boto:Params: {} 2023-05-31T20:38:18.819 DEBUG:boto:Token: None 2023-05-31T20:38:18.819 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Wed, 31 May 2023 20:38:18 GMT /pklxkd-58/obj00000000000000000015 2023-05-31T20:38:18.820 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:rO4faUX7RHhl5BOVsUkGomM1X2s= 2023-05-31T20:38:18.820 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Wed, 31 May 2023 20:38:18 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:rO4faUX7RHhl5BOVsUkGomM1X2s='} 2023-05-31T20:38:18.827 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000099802620f2905188-006477b03a-1630-a1'), ('Date', 'Wed, 31 May 2023 20:38:18 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:38:18.828 DEBUG:boto:path=/pklxkd-58/ 2023-05-31T20:38:18.828 DEBUG:boto:auth_path=/pklxkd-58/ 2023-05-31T20:38:18.829 DEBUG:boto:Method: HEAD 2023-05-31T20:38:18.829 DEBUG:boto:Path: /pklxkd-58/ 2023-05-31T20:38:18.829 DEBUG:boto:Data: 2023-05-31T20:38:18.830 DEBUG:boto:Headers: {} 2023-05-31T20:38:18.830 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:38:18.830 DEBUG:boto:Port: 8000 2023-05-31T20:38:18.831 DEBUG:boto:Params: {} 2023-05-31T20:38:18.831 DEBUG:boto:Token: None 2023-05-31T20:38:18.832 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:38:18 GMT /pklxkd-58/ 2023-05-31T20:38:18.832 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:mPIo1OW7xVEcvBZnJtgeMHBCsAQ= 2023-05-31T20:38:18.833 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:38:18 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:mPIo1OW7xVEcvBZnJtgeMHBCsAQ=', 'Content-Length': '0'} 2023-05-31T20:38:18.836 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '38'), ('X-RGW-Bytes-Used', '114'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000834f7d2d220a883d-006477b03a-1630-a1'), ('Content-Length', '0'), ('Date', 'Wed, 31 May 2023 20:38:18 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:38:18.836 DEBUG:boto:path=/pklxkd-58/obj00000000000000000030 2023-05-31T20:38:18.837 DEBUG:boto:auth_path=/pklxkd-58/obj00000000000000000030 2023-05-31T20:38:18.837 DEBUG:boto:Method: PUT 2023-05-31T20:38:18.838 DEBUG:boto:Path: /pklxkd-58/obj00000000000000000030 2023-05-31T20:38:18.838 DEBUG:boto:Data: 2023-05-31T20:38:18.839 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-05-31T20:38:18.839 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:38:18.839 DEBUG:boto:Port: 8000 2023-05-31T20:38:18.840 DEBUG:boto:Params: {} 2023-05-31T20:38:18.840 DEBUG:boto:Token: None 2023-05-31T20:38:18.840 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Wed, 31 May 2023 20:38:18 GMT /pklxkd-58/obj00000000000000000030 2023-05-31T20:38:18.841 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:TJ6uhOTyDw9xDuYkYXkSuMmXxqU= 2023-05-31T20:38:18.841 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Wed, 31 May 2023 20:38:18 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:TJ6uhOTyDw9xDuYkYXkSuMmXxqU='} 2023-05-31T20:38:18.884 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000001bb77c7ee7bf57ca-006477b03a-1630-a1'), ('Date', 'Wed, 31 May 2023 20:38:18 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:38:18.885 DEBUG:boto:path=/pklxkd-58/ 2023-05-31T20:38:18.885 DEBUG:boto:auth_path=/pklxkd-58/ 2023-05-31T20:38:18.886 DEBUG:boto:Method: HEAD 2023-05-31T20:38:18.886 DEBUG:boto:Path: /pklxkd-58/ 2023-05-31T20:38:18.886 DEBUG:boto:Data: 2023-05-31T20:38:18.887 DEBUG:boto:Headers: {} 2023-05-31T20:38:18.887 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:38:18.887 DEBUG:boto:Port: 8000 2023-05-31T20:38:18.888 DEBUG:boto:Params: {} 2023-05-31T20:38:18.888 DEBUG:boto:Token: None 2023-05-31T20:38:18.889 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:38:18 GMT /pklxkd-58/ 2023-05-31T20:38:18.889 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:mPIo1OW7xVEcvBZnJtgeMHBCsAQ= 2023-05-31T20:38:18.890 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:38:18 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:mPIo1OW7xVEcvBZnJtgeMHBCsAQ=', 'Content-Length': '0'} 2023-05-31T20:38:18.892 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '38'), ('X-RGW-Bytes-Used', '114'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000877ecc130d170d04-006477b03a-1630-a1'), ('Content-Length', '0'), ('Date', 'Wed, 31 May 2023 20:38:18 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:38:18.893 DEBUG:boto:path=/pklxkd-58/obj00000000000000000008 2023-05-31T20:38:18.894 DEBUG:boto:auth_path=/pklxkd-58/obj00000000000000000008 2023-05-31T20:38:18.894 DEBUG:boto:Method: PUT 2023-05-31T20:38:18.895 DEBUG:boto:Path: /pklxkd-58/obj00000000000000000008 2023-05-31T20:38:18.895 DEBUG:boto:Data: 2023-05-31T20:38:18.895 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-05-31T20:38:18.896 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:38:18.896 DEBUG:boto:Port: 8000 2023-05-31T20:38:18.896 DEBUG:boto:Params: {} 2023-05-31T20:38:18.897 DEBUG:boto:Token: None 2023-05-31T20:38:18.897 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Wed, 31 May 2023 20:38:18 GMT /pklxkd-58/obj00000000000000000008 2023-05-31T20:38:18.898 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:a4kixzn0zPPP7v+xCEJEN1kf1nw= 2023-05-31T20:38:18.898 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Wed, 31 May 2023 20:38:18 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:a4kixzn0zPPP7v+xCEJEN1kf1nw='} 2023-05-31T20:38:18.905 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000f762a37d7c304aaa-006477b03a-1630-a1'), ('Date', 'Wed, 31 May 2023 20:38:18 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:38:18.905 DEBUG:boto:path=/pklxkd-58/ 2023-05-31T20:38:18.906 DEBUG:boto:auth_path=/pklxkd-58/ 2023-05-31T20:38:18.906 DEBUG:boto:Method: HEAD 2023-05-31T20:38:18.906 DEBUG:boto:Path: /pklxkd-58/ 2023-05-31T20:38:18.907 DEBUG:boto:Data: 2023-05-31T20:38:18.907 DEBUG:boto:Headers: {} 2023-05-31T20:38:18.907 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:38:18.908 DEBUG:boto:Port: 8000 2023-05-31T20:38:18.908 DEBUG:boto:Params: {} 2023-05-31T20:38:18.909 DEBUG:boto:Token: None 2023-05-31T20:38:18.909 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:38:18 GMT /pklxkd-58/ 2023-05-31T20:38:18.909 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:mPIo1OW7xVEcvBZnJtgeMHBCsAQ= 2023-05-31T20:38:18.910 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:38:18 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:mPIo1OW7xVEcvBZnJtgeMHBCsAQ=', 'Content-Length': '0'} 2023-05-31T20:38:18.913 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '38'), ('X-RGW-Bytes-Used', '114'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000014c20334c7b96343-006477b03a-1630-a1'), ('Content-Length', '0'), ('Date', 'Wed, 31 May 2023 20:38:18 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:38:18.913 DEBUG:boto:path=/pklxkd-58/obj00000000000000000013 2023-05-31T20:38:18.914 DEBUG:boto:auth_path=/pklxkd-58/obj00000000000000000013 2023-05-31T20:38:18.914 DEBUG:boto:Method: PUT 2023-05-31T20:38:18.914 DEBUG:boto:Path: /pklxkd-58/obj00000000000000000013 2023-05-31T20:38:18.915 DEBUG:boto:Data: 2023-05-31T20:38:18.915 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-05-31T20:38:18.915 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:38:18.916 DEBUG:boto:Port: 8000 2023-05-31T20:38:18.916 DEBUG:boto:Params: {} 2023-05-31T20:38:18.917 DEBUG:boto:Token: None 2023-05-31T20:38:18.917 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Wed, 31 May 2023 20:38:18 GMT /pklxkd-58/obj00000000000000000013 2023-05-31T20:38:18.917 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:WIYe9NFzIlQb6jKJoPt3rfkVPSw= 2023-05-31T20:38:18.918 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Wed, 31 May 2023 20:38:18 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:WIYe9NFzIlQb6jKJoPt3rfkVPSw='} 2023-05-31T20:38:18.964 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000e61f4ca80949debe-006477b03a-1630-a1'), ('Date', 'Wed, 31 May 2023 20:38:18 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:38:18.965 DEBUG:boto:path=/pklxkd-58/ 2023-05-31T20:38:18.966 DEBUG:boto:auth_path=/pklxkd-58/ 2023-05-31T20:38:18.966 DEBUG:boto:Method: HEAD 2023-05-31T20:38:18.967 DEBUG:boto:Path: /pklxkd-58/ 2023-05-31T20:38:18.967 DEBUG:boto:Data: 2023-05-31T20:38:18.967 DEBUG:boto:Headers: {} 2023-05-31T20:38:18.968 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:38:18.968 DEBUG:boto:Port: 8000 2023-05-31T20:38:18.969 DEBUG:boto:Params: {} 2023-05-31T20:38:18.969 DEBUG:boto:Token: None 2023-05-31T20:38:18.969 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:38:18 GMT /pklxkd-58/ 2023-05-31T20:38:18.970 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:mPIo1OW7xVEcvBZnJtgeMHBCsAQ= 2023-05-31T20:38:18.970 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:38:18 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:mPIo1OW7xVEcvBZnJtgeMHBCsAQ=', 'Content-Length': '0'} 2023-05-31T20:38:18.973 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '38'), ('X-RGW-Bytes-Used', '114'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000972c2a824c8777d4-006477b03a-1630-a1'), ('Content-Length', '0'), ('Date', 'Wed, 31 May 2023 20:38:18 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:38:18.974 DEBUG:boto:path=/pklxkd-58/obj00000000000000000042 2023-05-31T20:38:18.974 DEBUG:boto:auth_path=/pklxkd-58/obj00000000000000000042 2023-05-31T20:38:18.974 DEBUG:boto:Method: PUT 2023-05-31T20:38:18.975 DEBUG:boto:Path: /pklxkd-58/obj00000000000000000042 2023-05-31T20:38:18.975 DEBUG:boto:Data: 2023-05-31T20:38:18.976 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-05-31T20:38:18.976 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:38:18.976 DEBUG:boto:Port: 8000 2023-05-31T20:38:18.977 DEBUG:boto:Params: {} 2023-05-31T20:38:18.977 DEBUG:boto:Token: None 2023-05-31T20:38:18.978 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Wed, 31 May 2023 20:38:18 GMT /pklxkd-58/obj00000000000000000042 2023-05-31T20:38:18.978 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:DjMfFlojfOMlXf5YKJGRMBeto+Q= 2023-05-31T20:38:18.979 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Wed, 31 May 2023 20:38:18 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:DjMfFlojfOMlXf5YKJGRMBeto+Q='} 2023-05-31T20:38:18.984 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000e11fc066d2a5ca2e-006477b03a-1630-a1'), ('Date', 'Wed, 31 May 2023 20:38:18 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:38:18.985 DEBUG:boto:path=/pklxkd-58/ 2023-05-31T20:38:18.985 DEBUG:boto:auth_path=/pklxkd-58/ 2023-05-31T20:38:18.986 DEBUG:boto:Method: HEAD 2023-05-31T20:38:18.986 DEBUG:boto:Path: /pklxkd-58/ 2023-05-31T20:38:18.987 DEBUG:boto:Data: 2023-05-31T20:38:18.987 DEBUG:boto:Headers: {} 2023-05-31T20:38:18.987 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:38:18.988 DEBUG:boto:Port: 8000 2023-05-31T20:38:18.988 DEBUG:boto:Params: {} 2023-05-31T20:38:18.989 DEBUG:boto:Token: None 2023-05-31T20:38:18.989 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:38:18 GMT /pklxkd-58/ 2023-05-31T20:38:18.990 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:mPIo1OW7xVEcvBZnJtgeMHBCsAQ= 2023-05-31T20:38:18.991 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:38:18 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:mPIo1OW7xVEcvBZnJtgeMHBCsAQ=', 'Content-Length': '0'} 2023-05-31T20:38:18.995 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '39'), ('X-RGW-Bytes-Used', '117'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000bfc49bc68a8eae6b-006477b03a-1630-a1'), ('Content-Length', '0'), ('Date', 'Wed, 31 May 2023 20:38:18 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:38:18.995 DEBUG:boto:path=/pklxkd-58/obj00000000000000000025 2023-05-31T20:38:18.996 DEBUG:boto:auth_path=/pklxkd-58/obj00000000000000000025 2023-05-31T20:38:18.996 DEBUG:boto:Method: PUT 2023-05-31T20:38:18.997 DEBUG:boto:Path: /pklxkd-58/obj00000000000000000025 2023-05-31T20:38:18.997 DEBUG:boto:Data: 2023-05-31T20:38:18.997 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-05-31T20:38:18.998 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:38:18.998 DEBUG:boto:Port: 8000 2023-05-31T20:38:19.000 DEBUG:boto:Params: {} 2023-05-31T20:38:19.000 DEBUG:boto:Token: None 2023-05-31T20:38:19.001 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Wed, 31 May 2023 20:38:19 GMT /pklxkd-58/obj00000000000000000025 2023-05-31T20:38:19.002 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:kX3yuON5olWinB5ufjeOTp71P1c= 2023-05-31T20:38:19.003 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Wed, 31 May 2023 20:38:19 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:kX3yuON5olWinB5ufjeOTp71P1c='} 2023-05-31T20:38:19.048 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000f7ab8fc1622b629d-006477b03b-1630-a1'), ('Date', 'Wed, 31 May 2023 20:38:19 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:38:19.049 DEBUG:boto:path=/pklxkd-58/ 2023-05-31T20:38:19.049 DEBUG:boto:auth_path=/pklxkd-58/ 2023-05-31T20:38:19.050 DEBUG:boto:Method: HEAD 2023-05-31T20:38:19.050 DEBUG:boto:Path: /pklxkd-58/ 2023-05-31T20:38:19.051 DEBUG:boto:Data: 2023-05-31T20:38:19.051 DEBUG:boto:Headers: {} 2023-05-31T20:38:19.051 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:38:19.052 DEBUG:boto:Port: 8000 2023-05-31T20:38:19.055 DEBUG:boto:Params: {} 2023-05-31T20:38:19.056 DEBUG:boto:Token: None 2023-05-31T20:38:19.056 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:38:19 GMT /pklxkd-58/ 2023-05-31T20:38:19.057 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:eV79GzMOeufr774jk4ITvUwXq9g= 2023-05-31T20:38:19.057 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:38:19 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:eV79GzMOeufr774jk4ITvUwXq9g=', 'Content-Length': '0'} 2023-05-31T20:38:19.061 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '39'), ('X-RGW-Bytes-Used', '117'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000060fcd7c7eb8321e4-006477b03b-1630-a1'), ('Content-Length', '0'), ('Date', 'Wed, 31 May 2023 20:38:19 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:38:19.063 DEBUG:boto:path=/pklxkd-58/obj00000000000000000027 2023-05-31T20:38:19.064 DEBUG:boto:auth_path=/pklxkd-58/obj00000000000000000027 2023-05-31T20:38:19.065 DEBUG:boto:Method: PUT 2023-05-31T20:38:19.065 DEBUG:boto:Path: /pklxkd-58/obj00000000000000000027 2023-05-31T20:38:19.070 DEBUG:boto:Data: 2023-05-31T20:38:19.070 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-05-31T20:38:19.071 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:38:19.071 DEBUG:boto:Port: 8000 2023-05-31T20:38:19.072 DEBUG:boto:Params: {} 2023-05-31T20:38:19.073 DEBUG:boto:Token: None 2023-05-31T20:38:19.074 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Wed, 31 May 2023 20:38:19 GMT /pklxkd-58/obj00000000000000000027 2023-05-31T20:38:19.074 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:6WYPKS/iS+S9yD/ugj+8+gFWujY= 2023-05-31T20:38:19.076 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Wed, 31 May 2023 20:38:19 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:6WYPKS/iS+S9yD/ugj+8+gFWujY='} 2023-05-31T20:38:19.088 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000091f6459ab6d08f6b-006477b03b-1630-a1'), ('Date', 'Wed, 31 May 2023 20:38:19 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:38:19.089 DEBUG:boto:path=/pklxkd-58/ 2023-05-31T20:38:19.090 DEBUG:boto:auth_path=/pklxkd-58/ 2023-05-31T20:38:19.090 DEBUG:boto:Method: HEAD 2023-05-31T20:38:19.091 DEBUG:boto:Path: /pklxkd-58/ 2023-05-31T20:38:19.091 DEBUG:boto:Data: 2023-05-31T20:38:19.092 DEBUG:boto:Headers: {} 2023-05-31T20:38:19.092 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:38:19.092 DEBUG:boto:Port: 8000 2023-05-31T20:38:19.093 DEBUG:boto:Params: {} 2023-05-31T20:38:19.093 DEBUG:boto:Token: None 2023-05-31T20:38:19.093 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:38:19 GMT /pklxkd-58/ 2023-05-31T20:38:19.094 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:eV79GzMOeufr774jk4ITvUwXq9g= 2023-05-31T20:38:19.094 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:38:19 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:eV79GzMOeufr774jk4ITvUwXq9g=', 'Content-Length': '0'} 2023-05-31T20:38:19.098 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '40'), ('X-RGW-Bytes-Used', '120'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000053cac5d1fbd9013f-006477b03b-1630-a1'), ('Content-Length', '0'), ('Date', 'Wed, 31 May 2023 20:38:19 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:38:19.098 DEBUG:boto:path=/pklxkd-58/obj00000000000000000006 2023-05-31T20:38:19.099 DEBUG:boto:auth_path=/pklxkd-58/obj00000000000000000006 2023-05-31T20:38:19.100 DEBUG:boto:Method: PUT 2023-05-31T20:38:19.100 DEBUG:boto:Path: /pklxkd-58/obj00000000000000000006 2023-05-31T20:38:19.101 DEBUG:boto:Data: 2023-05-31T20:38:19.102 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-05-31T20:38:19.103 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:38:19.103 DEBUG:boto:Port: 8000 2023-05-31T20:38:19.104 DEBUG:boto:Params: {} 2023-05-31T20:38:19.105 DEBUG:boto:Token: None 2023-05-31T20:38:19.105 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Wed, 31 May 2023 20:38:19 GMT /pklxkd-58/obj00000000000000000006 2023-05-31T20:38:19.106 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:IQ29aeeSEOjx2wRZAiQz1rQDVxw= 2023-05-31T20:38:19.106 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Wed, 31 May 2023 20:38:19 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:IQ29aeeSEOjx2wRZAiQz1rQDVxw='} 2023-05-31T20:38:19.152 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000003cb857ca6236c19d-006477b03b-1630-a1'), ('Date', 'Wed, 31 May 2023 20:38:19 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:38:19.153 DEBUG:boto:path=/pklxkd-58/ 2023-05-31T20:38:19.154 DEBUG:boto:auth_path=/pklxkd-58/ 2023-05-31T20:38:19.155 DEBUG:boto:Method: HEAD 2023-05-31T20:38:19.156 DEBUG:boto:Path: /pklxkd-58/ 2023-05-31T20:38:19.156 DEBUG:boto:Data: 2023-05-31T20:38:19.157 DEBUG:boto:Headers: {} 2023-05-31T20:38:19.158 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:38:19.159 DEBUG:boto:Port: 8000 2023-05-31T20:38:19.160 DEBUG:boto:Params: {} 2023-05-31T20:38:19.161 DEBUG:boto:Token: None 2023-05-31T20:38:19.161 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:38:19 GMT /pklxkd-58/ 2023-05-31T20:38:19.162 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:eV79GzMOeufr774jk4ITvUwXq9g= 2023-05-31T20:38:19.163 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:38:19 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:eV79GzMOeufr774jk4ITvUwXq9g=', 'Content-Length': '0'} 2023-05-31T20:38:19.166 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '40'), ('X-RGW-Bytes-Used', '120'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000038d4dea55b788ee-006477b03b-1630-a1'), ('Content-Length', '0'), ('Date', 'Wed, 31 May 2023 20:38:19 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:38:19.167 DEBUG:boto:path=/pklxkd-58/obj00000000000000000005 2023-05-31T20:38:19.167 DEBUG:boto:auth_path=/pklxkd-58/obj00000000000000000005 2023-05-31T20:38:19.168 DEBUG:boto:Method: PUT 2023-05-31T20:38:19.168 DEBUG:boto:Path: /pklxkd-58/obj00000000000000000005 2023-05-31T20:38:19.169 DEBUG:boto:Data: 2023-05-31T20:38:19.169 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-05-31T20:38:19.169 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:38:19.170 DEBUG:boto:Port: 8000 2023-05-31T20:38:19.170 DEBUG:boto:Params: {} 2023-05-31T20:38:19.171 DEBUG:boto:Token: None 2023-05-31T20:38:19.171 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Wed, 31 May 2023 20:38:19 GMT /pklxkd-58/obj00000000000000000005 2023-05-31T20:38:19.171 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:9NfsBMAEu/K77GH2sf3oqpvcOEw= 2023-05-31T20:38:19.172 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Wed, 31 May 2023 20:38:19 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:9NfsBMAEu/K77GH2sf3oqpvcOEw='} 2023-05-31T20:38:19.184 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000000a77da525b5d9fbc-006477b03b-1630-a1'), ('Date', 'Wed, 31 May 2023 20:38:19 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:38:19.186 DEBUG:boto:path=/pklxkd-58/ 2023-05-31T20:38:19.186 DEBUG:boto:auth_path=/pklxkd-58/ 2023-05-31T20:38:19.187 DEBUG:boto:Method: HEAD 2023-05-31T20:38:19.189 DEBUG:boto:Path: /pklxkd-58/ 2023-05-31T20:38:19.190 DEBUG:boto:Data: 2023-05-31T20:38:19.190 DEBUG:boto:Headers: {} 2023-05-31T20:38:19.190 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:38:19.191 DEBUG:boto:Port: 8000 2023-05-31T20:38:19.191 DEBUG:boto:Params: {} 2023-05-31T20:38:19.192 DEBUG:boto:Token: None 2023-05-31T20:38:19.192 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:38:19 GMT /pklxkd-58/ 2023-05-31T20:38:19.193 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:eV79GzMOeufr774jk4ITvUwXq9g= 2023-05-31T20:38:19.193 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:38:19 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:eV79GzMOeufr774jk4ITvUwXq9g=', 'Content-Length': '0'} 2023-05-31T20:38:19.196 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '40'), ('X-RGW-Bytes-Used', '120'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000041968a2e7e3b06d0-006477b03b-1630-a1'), ('Content-Length', '0'), ('Date', 'Wed, 31 May 2023 20:38:19 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:38:19.197 DEBUG:boto:path=/pklxkd-58/obj00000000000000000022 2023-05-31T20:38:19.197 DEBUG:boto:auth_path=/pklxkd-58/obj00000000000000000022 2023-05-31T20:38:19.198 DEBUG:boto:Method: PUT 2023-05-31T20:38:19.198 DEBUG:boto:Path: /pklxkd-58/obj00000000000000000022 2023-05-31T20:38:19.199 DEBUG:boto:Data: 2023-05-31T20:38:19.199 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-05-31T20:38:19.200 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:38:19.200 DEBUG:boto:Port: 8000 2023-05-31T20:38:19.201 DEBUG:boto:Params: {} 2023-05-31T20:38:19.202 DEBUG:boto:Token: None 2023-05-31T20:38:19.203 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Wed, 31 May 2023 20:38:19 GMT /pklxkd-58/obj00000000000000000022 2023-05-31T20:38:19.203 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:tAhC1wbYnQ8m+zsvy0fggf4FnLY= 2023-05-31T20:38:19.204 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Wed, 31 May 2023 20:38:19 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:tAhC1wbYnQ8m+zsvy0fggf4FnLY='} 2023-05-31T20:38:19.248 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000072b76282006cbd7d-006477b03b-1630-a1'), ('Date', 'Wed, 31 May 2023 20:38:19 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:38:19.249 DEBUG:boto:path=/pklxkd-58/ 2023-05-31T20:38:19.250 DEBUG:boto:auth_path=/pklxkd-58/ 2023-05-31T20:38:19.250 DEBUG:boto:Method: HEAD 2023-05-31T20:38:19.251 DEBUG:boto:Path: /pklxkd-58/ 2023-05-31T20:38:19.251 DEBUG:boto:Data: 2023-05-31T20:38:19.251 DEBUG:boto:Headers: {} 2023-05-31T20:38:19.252 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:38:19.252 DEBUG:boto:Port: 8000 2023-05-31T20:38:19.253 DEBUG:boto:Params: {} 2023-05-31T20:38:19.253 DEBUG:boto:Token: None 2023-05-31T20:38:19.254 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:38:19 GMT /pklxkd-58/ 2023-05-31T20:38:19.254 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:eV79GzMOeufr774jk4ITvUwXq9g= 2023-05-31T20:38:19.255 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:38:19 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:eV79GzMOeufr774jk4ITvUwXq9g=', 'Content-Length': '0'} 2023-05-31T20:38:19.258 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '41'), ('X-RGW-Bytes-Used', '123'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000051cf50cb3c46d7d6-006477b03b-1630-a1'), ('Content-Length', '0'), ('Date', 'Wed, 31 May 2023 20:38:19 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:38:19.259 DEBUG:boto:path=/pklxkd-58/obj00000000000000000009 2023-05-31T20:38:19.260 DEBUG:boto:auth_path=/pklxkd-58/obj00000000000000000009 2023-05-31T20:38:19.261 DEBUG:boto:Method: PUT 2023-05-31T20:38:19.261 DEBUG:boto:Path: /pklxkd-58/obj00000000000000000009 2023-05-31T20:38:19.262 DEBUG:boto:Data: 2023-05-31T20:38:19.262 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-05-31T20:38:19.263 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:38:19.263 DEBUG:boto:Port: 8000 2023-05-31T20:38:19.263 DEBUG:boto:Params: {} 2023-05-31T20:38:19.264 DEBUG:boto:Token: None 2023-05-31T20:38:19.264 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Wed, 31 May 2023 20:38:19 GMT /pklxkd-58/obj00000000000000000009 2023-05-31T20:38:19.265 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:cJvncclxD9rcV0LstH+Mgy/9bCE= 2023-05-31T20:38:19.265 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Wed, 31 May 2023 20:38:19 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:cJvncclxD9rcV0LstH+Mgy/9bCE='} 2023-05-31T20:38:19.279 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000cb842a5275dfd497-006477b03b-1630-a1'), ('Date', 'Wed, 31 May 2023 20:38:19 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:38:19.280 DEBUG:boto:path=/pklxkd-58/ 2023-05-31T20:38:19.281 DEBUG:boto:auth_path=/pklxkd-58/ 2023-05-31T20:38:19.282 DEBUG:boto:Method: HEAD 2023-05-31T20:38:19.283 DEBUG:boto:Path: /pklxkd-58/ 2023-05-31T20:38:19.284 DEBUG:boto:Data: 2023-05-31T20:38:19.285 DEBUG:boto:Headers: {} 2023-05-31T20:38:19.285 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:38:19.286 DEBUG:boto:Port: 8000 2023-05-31T20:38:19.287 DEBUG:boto:Params: {} 2023-05-31T20:38:19.287 DEBUG:boto:Token: None 2023-05-31T20:38:19.288 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:38:19 GMT /pklxkd-58/ 2023-05-31T20:38:19.288 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:eV79GzMOeufr774jk4ITvUwXq9g= 2023-05-31T20:38:19.288 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:38:19 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:eV79GzMOeufr774jk4ITvUwXq9g=', 'Content-Length': '0'} 2023-05-31T20:38:19.291 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '41'), ('X-RGW-Bytes-Used', '123'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000007d5125ecbfdf064b-006477b03b-1630-a1'), ('Content-Length', '0'), ('Date', 'Wed, 31 May 2023 20:38:19 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:38:19.292 DEBUG:boto:path=/pklxkd-58/obj00000000000000000016 2023-05-31T20:38:19.293 DEBUG:boto:auth_path=/pklxkd-58/obj00000000000000000016 2023-05-31T20:38:19.293 DEBUG:boto:Method: PUT 2023-05-31T20:38:19.293 DEBUG:boto:Path: /pklxkd-58/obj00000000000000000016 2023-05-31T20:38:19.294 DEBUG:boto:Data: 2023-05-31T20:38:19.294 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-05-31T20:38:19.295 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:38:19.295 DEBUG:boto:Port: 8000 2023-05-31T20:38:19.295 DEBUG:boto:Params: {} 2023-05-31T20:38:19.296 DEBUG:boto:Token: None 2023-05-31T20:38:19.296 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Wed, 31 May 2023 20:38:19 GMT /pklxkd-58/obj00000000000000000016 2023-05-31T20:38:19.297 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:nmGycqfob/hsKntFVJpOGG9JaQc= 2023-05-31T20:38:19.297 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Wed, 31 May 2023 20:38:19 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:nmGycqfob/hsKntFVJpOGG9JaQc='} 2023-05-31T20:38:19.344 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000712c564266f6b30d-006477b03b-1630-a1'), ('Date', 'Wed, 31 May 2023 20:38:19 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:38:19.345 DEBUG:boto:path=/pklxkd-58/ 2023-05-31T20:38:19.345 DEBUG:boto:auth_path=/pklxkd-58/ 2023-05-31T20:38:19.346 DEBUG:boto:Method: HEAD 2023-05-31T20:38:19.346 DEBUG:boto:Path: /pklxkd-58/ 2023-05-31T20:38:19.347 DEBUG:boto:Data: 2023-05-31T20:38:19.347 DEBUG:boto:Headers: {} 2023-05-31T20:38:19.347 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:38:19.348 DEBUG:boto:Port: 8000 2023-05-31T20:38:19.348 DEBUG:boto:Params: {} 2023-05-31T20:38:19.349 DEBUG:boto:Token: None 2023-05-31T20:38:19.349 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:38:19 GMT /pklxkd-58/ 2023-05-31T20:38:19.349 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:eV79GzMOeufr774jk4ITvUwXq9g= 2023-05-31T20:38:19.350 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:38:19 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:eV79GzMOeufr774jk4ITvUwXq9g=', 'Content-Length': '0'} 2023-05-31T20:38:19.353 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '41'), ('X-RGW-Bytes-Used', '123'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000066c0c8445a36614b-006477b03b-1630-a1'), ('Content-Length', '0'), ('Date', 'Wed, 31 May 2023 20:38:19 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:38:19.354 DEBUG:boto:path=/pklxkd-58/obj00000000000000000037 2023-05-31T20:38:19.354 DEBUG:boto:auth_path=/pklxkd-58/obj00000000000000000037 2023-05-31T20:38:19.355 DEBUG:boto:Method: PUT 2023-05-31T20:38:19.356 DEBUG:boto:Path: /pklxkd-58/obj00000000000000000037 2023-05-31T20:38:19.356 DEBUG:boto:Data: 2023-05-31T20:38:19.357 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-05-31T20:38:19.357 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:38:19.358 DEBUG:boto:Port: 8000 2023-05-31T20:38:19.358 DEBUG:boto:Params: {} 2023-05-31T20:38:19.358 DEBUG:boto:Token: None 2023-05-31T20:38:19.359 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Wed, 31 May 2023 20:38:19 GMT /pklxkd-58/obj00000000000000000037 2023-05-31T20:38:19.359 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:Q80yU8dWLMSO7zlc9mtdIOHTCyI= 2023-05-31T20:38:19.360 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Wed, 31 May 2023 20:38:19 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:Q80yU8dWLMSO7zlc9mtdIOHTCyI='} 2023-05-31T20:38:19.365 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000930c9a0a747a48e9-006477b03b-1630-a1'), ('Date', 'Wed, 31 May 2023 20:38:19 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:38:19.365 DEBUG:boto:path=/pklxkd-58/ 2023-05-31T20:38:19.366 DEBUG:boto:auth_path=/pklxkd-58/ 2023-05-31T20:38:19.366 DEBUG:boto:Method: HEAD 2023-05-31T20:38:19.366 DEBUG:boto:Path: /pklxkd-58/ 2023-05-31T20:38:19.367 DEBUG:boto:Data: 2023-05-31T20:38:19.367 DEBUG:boto:Headers: {} 2023-05-31T20:38:19.367 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:38:19.368 DEBUG:boto:Port: 8000 2023-05-31T20:38:19.368 DEBUG:boto:Params: {} 2023-05-31T20:38:19.368 DEBUG:boto:Token: None 2023-05-31T20:38:19.369 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:38:19 GMT /pklxkd-58/ 2023-05-31T20:38:19.370 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:eV79GzMOeufr774jk4ITvUwXq9g= 2023-05-31T20:38:19.370 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:38:19 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:eV79GzMOeufr774jk4ITvUwXq9g=', 'Content-Length': '0'} 2023-05-31T20:38:19.373 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '42'), ('X-RGW-Bytes-Used', '126'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000a8bfd35370998aeb-006477b03b-1630-a1'), ('Content-Length', '0'), ('Date', 'Wed, 31 May 2023 20:38:19 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:38:19.374 DEBUG:boto:path=/pklxkd-58/obj00000000000000000020 2023-05-31T20:38:19.375 DEBUG:boto:auth_path=/pklxkd-58/obj00000000000000000020 2023-05-31T20:38:19.375 DEBUG:boto:Method: PUT 2023-05-31T20:38:19.376 DEBUG:boto:Path: /pklxkd-58/obj00000000000000000020 2023-05-31T20:38:19.377 DEBUG:boto:Data: 2023-05-31T20:38:19.377 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-05-31T20:38:19.378 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:38:19.379 DEBUG:boto:Port: 8000 2023-05-31T20:38:19.379 DEBUG:boto:Params: {} 2023-05-31T20:38:19.381 DEBUG:boto:Token: None 2023-05-31T20:38:19.382 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Wed, 31 May 2023 20:38:19 GMT /pklxkd-58/obj00000000000000000020 2023-05-31T20:38:19.383 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:BDj++7wFV1tdBSIJgb1aPV6d28c= 2023-05-31T20:38:19.384 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Wed, 31 May 2023 20:38:19 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:BDj++7wFV1tdBSIJgb1aPV6d28c='} 2023-05-31T20:38:19.429 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000032cdf50493ead835-006477b03b-1630-a1'), ('Date', 'Wed, 31 May 2023 20:38:19 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:38:19.430 DEBUG:boto:path=/pklxkd-58/ 2023-05-31T20:38:19.430 DEBUG:boto:auth_path=/pklxkd-58/ 2023-05-31T20:38:19.430 DEBUG:boto:Method: HEAD 2023-05-31T20:38:19.431 DEBUG:boto:Path: /pklxkd-58/ 2023-05-31T20:38:19.431 DEBUG:boto:Data: 2023-05-31T20:38:19.432 DEBUG:boto:Headers: {} 2023-05-31T20:38:19.432 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:38:19.432 DEBUG:boto:Port: 8000 2023-05-31T20:38:19.433 DEBUG:boto:Params: {} 2023-05-31T20:38:19.433 DEBUG:boto:Token: None 2023-05-31T20:38:19.433 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:38:19 GMT /pklxkd-58/ 2023-05-31T20:38:19.434 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:eV79GzMOeufr774jk4ITvUwXq9g= 2023-05-31T20:38:19.434 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:38:19 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:eV79GzMOeufr774jk4ITvUwXq9g=', 'Content-Length': '0'} 2023-05-31T20:38:19.437 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '42'), ('X-RGW-Bytes-Used', '126'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000e140b60db9e14196-006477b03b-1630-a1'), ('Content-Length', '0'), ('Date', 'Wed, 31 May 2023 20:38:19 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:38:19.438 DEBUG:boto:path=/pklxkd-58/obj00000000000000000012 2023-05-31T20:38:19.439 DEBUG:boto:auth_path=/pklxkd-58/obj00000000000000000012 2023-05-31T20:38:19.439 DEBUG:boto:Method: PUT 2023-05-31T20:38:19.440 DEBUG:boto:Path: /pklxkd-58/obj00000000000000000012 2023-05-31T20:38:19.441 DEBUG:boto:Data: 2023-05-31T20:38:19.441 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-05-31T20:38:19.442 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:38:19.442 DEBUG:boto:Port: 8000 2023-05-31T20:38:19.443 DEBUG:boto:Params: {} 2023-05-31T20:38:19.444 DEBUG:boto:Token: None 2023-05-31T20:38:19.444 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Wed, 31 May 2023 20:38:19 GMT /pklxkd-58/obj00000000000000000012 2023-05-31T20:38:19.445 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:ORS2VoJj4iEZ1kug5NjODQpUPvU= 2023-05-31T20:38:19.446 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Wed, 31 May 2023 20:38:19 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:ORS2VoJj4iEZ1kug5NjODQpUPvU='} 2023-05-31T20:38:19.454 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000005c60a106265644bb-006477b03b-1630-a1'), ('Date', 'Wed, 31 May 2023 20:38:19 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:38:19.455 DEBUG:boto:path=/pklxkd-58/ 2023-05-31T20:38:19.455 DEBUG:boto:auth_path=/pklxkd-58/ 2023-05-31T20:38:19.456 DEBUG:boto:Method: HEAD 2023-05-31T20:38:19.456 DEBUG:boto:Path: /pklxkd-58/ 2023-05-31T20:38:19.457 DEBUG:boto:Data: 2023-05-31T20:38:19.457 DEBUG:boto:Headers: {} 2023-05-31T20:38:19.458 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:38:19.459 DEBUG:boto:Port: 8000 2023-05-31T20:38:19.460 DEBUG:boto:Params: {} 2023-05-31T20:38:19.461 DEBUG:boto:Token: None 2023-05-31T20:38:19.462 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:38:19 GMT /pklxkd-58/ 2023-05-31T20:38:19.463 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:eV79GzMOeufr774jk4ITvUwXq9g= 2023-05-31T20:38:19.464 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:38:19 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:eV79GzMOeufr774jk4ITvUwXq9g=', 'Content-Length': '0'} 2023-05-31T20:38:19.468 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '42'), ('X-RGW-Bytes-Used', '126'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000067c5bb3a2df87f73-006477b03b-1630-a1'), ('Content-Length', '0'), ('Date', 'Wed, 31 May 2023 20:38:19 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:38:19.469 DEBUG:boto:path=/pklxkd-58/obj00000000000000000014 2023-05-31T20:38:19.470 DEBUG:boto:auth_path=/pklxkd-58/obj00000000000000000014 2023-05-31T20:38:19.470 DEBUG:boto:Method: PUT 2023-05-31T20:38:19.471 DEBUG:boto:Path: /pklxkd-58/obj00000000000000000014 2023-05-31T20:38:19.471 DEBUG:boto:Data: 2023-05-31T20:38:19.471 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-05-31T20:38:19.472 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:38:19.472 DEBUG:boto:Port: 8000 2023-05-31T20:38:19.473 DEBUG:boto:Params: {} 2023-05-31T20:38:19.475 DEBUG:boto:Token: None 2023-05-31T20:38:19.475 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Wed, 31 May 2023 20:38:19 GMT /pklxkd-58/obj00000000000000000014 2023-05-31T20:38:19.476 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:SqocQ6BuVBtcjXWFuQuLoLoF1Iw= 2023-05-31T20:38:19.476 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Wed, 31 May 2023 20:38:19 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:SqocQ6BuVBtcjXWFuQuLoLoF1Iw='} 2023-05-31T20:38:19.524 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000ebda7dd07b8dd3d4-006477b03b-1630-a1'), ('Date', 'Wed, 31 May 2023 20:38:19 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:38:19.525 DEBUG:boto:path=/pklxkd-58/ 2023-05-31T20:38:19.526 DEBUG:boto:auth_path=/pklxkd-58/ 2023-05-31T20:38:19.526 DEBUG:boto:Method: HEAD 2023-05-31T20:38:19.526 DEBUG:boto:Path: /pklxkd-58/ 2023-05-31T20:38:19.527 DEBUG:boto:Data: 2023-05-31T20:38:19.527 DEBUG:boto:Headers: {} 2023-05-31T20:38:19.527 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:38:19.528 DEBUG:boto:Port: 8000 2023-05-31T20:38:19.528 DEBUG:boto:Params: {} 2023-05-31T20:38:19.528 DEBUG:boto:Token: None 2023-05-31T20:38:19.529 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:38:19 GMT /pklxkd-58/ 2023-05-31T20:38:19.531 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:eV79GzMOeufr774jk4ITvUwXq9g= 2023-05-31T20:38:19.531 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:38:19 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:eV79GzMOeufr774jk4ITvUwXq9g=', 'Content-Length': '0'} 2023-05-31T20:38:19.534 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '42'), ('X-RGW-Bytes-Used', '126'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000000d22025ab3094cc7-006477b03b-1630-a1'), ('Content-Length', '0'), ('Date', 'Wed, 31 May 2023 20:38:19 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:38:19.535 DEBUG:boto:path=/pklxkd-58/obj00000000000000000000 2023-05-31T20:38:19.535 DEBUG:boto:auth_path=/pklxkd-58/obj00000000000000000000 2023-05-31T20:38:19.536 DEBUG:boto:Method: PUT 2023-05-31T20:38:19.536 DEBUG:boto:Path: /pklxkd-58/obj00000000000000000000 2023-05-31T20:38:19.536 DEBUG:boto:Data: 2023-05-31T20:38:19.537 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-05-31T20:38:19.537 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:38:19.537 DEBUG:boto:Port: 8000 2023-05-31T20:38:19.538 DEBUG:boto:Params: {} 2023-05-31T20:38:19.538 DEBUG:boto:Token: None 2023-05-31T20:38:19.539 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Wed, 31 May 2023 20:38:19 GMT /pklxkd-58/obj00000000000000000000 2023-05-31T20:38:19.539 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:4BETYxJnhH4JyGmKlRuwuVef/iY= 2023-05-31T20:38:19.540 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Wed, 31 May 2023 20:38:19 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:4BETYxJnhH4JyGmKlRuwuVef/iY='} 2023-05-31T20:38:19.547 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000bc0ded1efc60360b-006477b03b-1630-a1'), ('Date', 'Wed, 31 May 2023 20:38:19 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:38:19.548 DEBUG:boto:path=/pklxkd-58/ 2023-05-31T20:38:19.548 DEBUG:boto:auth_path=/pklxkd-58/ 2023-05-31T20:38:19.549 DEBUG:boto:Method: HEAD 2023-05-31T20:38:19.550 DEBUG:boto:Path: /pklxkd-58/ 2023-05-31T20:38:19.551 DEBUG:boto:Data: 2023-05-31T20:38:19.552 DEBUG:boto:Headers: {} 2023-05-31T20:38:19.553 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:38:19.553 DEBUG:boto:Port: 8000 2023-05-31T20:38:19.554 DEBUG:boto:Params: {} 2023-05-31T20:38:19.555 DEBUG:boto:Token: None 2023-05-31T20:38:19.556 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:38:19 GMT /pklxkd-58/ 2023-05-31T20:38:19.558 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:eV79GzMOeufr774jk4ITvUwXq9g= 2023-05-31T20:38:19.558 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:38:19 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:eV79GzMOeufr774jk4ITvUwXq9g=', 'Content-Length': '0'} 2023-05-31T20:38:19.562 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '42'), ('X-RGW-Bytes-Used', '126'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000a64dfc6cfab97648-006477b03b-1630-a1'), ('Content-Length', '0'), ('Date', 'Wed, 31 May 2023 20:38:19 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:38:19.563 DEBUG:boto:path=/pklxkd-58/obj00000000000000000003 2023-05-31T20:38:19.563 DEBUG:boto:auth_path=/pklxkd-58/obj00000000000000000003 2023-05-31T20:38:19.564 DEBUG:boto:Method: PUT 2023-05-31T20:38:19.564 DEBUG:boto:Path: /pklxkd-58/obj00000000000000000003 2023-05-31T20:38:19.565 DEBUG:boto:Data: 2023-05-31T20:38:19.565 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-05-31T20:38:19.566 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:38:19.566 DEBUG:boto:Port: 8000 2023-05-31T20:38:19.567 DEBUG:boto:Params: {} 2023-05-31T20:38:19.569 DEBUG:boto:Token: None 2023-05-31T20:38:19.570 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Wed, 31 May 2023 20:38:19 GMT /pklxkd-58/obj00000000000000000003 2023-05-31T20:38:19.570 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:OOY8RI3wczL3uxlSrqUQvMpAhhA= 2023-05-31T20:38:19.571 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Wed, 31 May 2023 20:38:19 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:OOY8RI3wczL3uxlSrqUQvMpAhhA='} 2023-05-31T20:38:19.616 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000b7e5181be39215e9-006477b03b-1630-a1'), ('Date', 'Wed, 31 May 2023 20:38:19 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:38:19.617 DEBUG:boto:path=/pklxkd-58/ 2023-05-31T20:38:19.618 DEBUG:boto:auth_path=/pklxkd-58/ 2023-05-31T20:38:19.618 DEBUG:boto:Method: HEAD 2023-05-31T20:38:19.618 DEBUG:boto:Path: /pklxkd-58/ 2023-05-31T20:38:19.619 DEBUG:boto:Data: 2023-05-31T20:38:19.619 DEBUG:boto:Headers: {} 2023-05-31T20:38:19.620 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:38:19.621 DEBUG:boto:Port: 8000 2023-05-31T20:38:19.622 DEBUG:boto:Params: {} 2023-05-31T20:38:19.623 DEBUG:boto:Token: None 2023-05-31T20:38:19.623 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:38:19 GMT /pklxkd-58/ 2023-05-31T20:38:19.625 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:eV79GzMOeufr774jk4ITvUwXq9g= 2023-05-31T20:38:19.626 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:38:19 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:eV79GzMOeufr774jk4ITvUwXq9g=', 'Content-Length': '0'} 2023-05-31T20:38:19.630 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '42'), ('X-RGW-Bytes-Used', '126'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000b795ab2ca7de4b82-006477b03b-1630-a1'), ('Content-Length', '0'), ('Date', 'Wed, 31 May 2023 20:38:19 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:38:19.631 DEBUG:boto:path=/pklxkd-58/obj00000000000000000001 2023-05-31T20:38:19.631 DEBUG:boto:auth_path=/pklxkd-58/obj00000000000000000001 2023-05-31T20:38:19.632 DEBUG:boto:Method: PUT 2023-05-31T20:38:19.632 DEBUG:boto:Path: /pklxkd-58/obj00000000000000000001 2023-05-31T20:38:19.633 DEBUG:boto:Data: 2023-05-31T20:38:19.633 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-05-31T20:38:19.633 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:38:19.634 DEBUG:boto:Port: 8000 2023-05-31T20:38:19.634 DEBUG:boto:Params: {} 2023-05-31T20:38:19.634 DEBUG:boto:Token: None 2023-05-31T20:38:19.635 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Wed, 31 May 2023 20:38:19 GMT /pklxkd-58/obj00000000000000000001 2023-05-31T20:38:19.636 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:zvzYR2vGdCU7JeEv0wtpR2Fe1aI= 2023-05-31T20:38:19.636 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Wed, 31 May 2023 20:38:19 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:zvzYR2vGdCU7JeEv0wtpR2Fe1aI='} 2023-05-31T20:38:19.645 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000007890d43e767b197a-006477b03b-1630-a1'), ('Date', 'Wed, 31 May 2023 20:38:19 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:38:19.646 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'reshard', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'pklxkd-58', '--num-shards', '31', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-05-31T20:38:19.646 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'reshard', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'pklxkd-58', '--num-shards', '31', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-05-31T20:38:19.647 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket reshard --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --bucket pklxkd-58 --num-shards 31 --yes-i-really-mean-it --cluster c1 --debug-rgw 0 --debug-ms 0 2023-05-31T20:38:19.699 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2023-05-31T20:38:19.700 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2023-05-31T20:38:19.753 INFO:teuthology.orchestra.run.smithi099.stdout:tenant: 2023-05-31T20:38:19.753 INFO:teuthology.orchestra.run.smithi099.stdout:bucket name: pklxkd-58 2023-05-31T20:38:19.763 INFO:teuthology.orchestra.run.smithi099.stdout:total entries: 42 2023-05-31T20:38:19.868 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'shard', 'objects', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--num-shards', '31', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-31T20:38:19.868 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'shard', 'objects', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--num-shards', '31', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-31T20:38:19.869 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket shard objects --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --num-shards 31 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-05-31T20:38:19.921 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2023-05-31T20:38:19.921 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2023-05-31T20:38:19.978 INFO:teuthology.orchestra.run.smithi099.stdout:{ 2023-05-31T20:38:19.979 INFO:teuthology.orchestra.run.smithi099.stdout: "objs": [ 2023-05-31T20:38:19.979 INFO:teuthology.orchestra.run.smithi099.stdout: "obj00000000000000000019", 2023-05-31T20:38:19.979 INFO:teuthology.orchestra.run.smithi099.stdout: "obj00000000000000000006", 2023-05-31T20:38:19.979 INFO:teuthology.orchestra.run.smithi099.stdout: "obj00000000000000000467", 2023-05-31T20:38:19.979 INFO:teuthology.orchestra.run.smithi099.stdout: "obj00000000000000000005", 2023-05-31T20:38:19.980 INFO:teuthology.orchestra.run.smithi099.stdout: "obj00000000000000000056", 2023-05-31T20:38:19.980 INFO:teuthology.orchestra.run.smithi099.stdout: "obj00000000000000000004", 2023-05-31T20:38:19.980 INFO:teuthology.orchestra.run.smithi099.stdout: "obj00000000000000000023", 2023-05-31T20:38:19.980 INFO:teuthology.orchestra.run.smithi099.stdout: "obj00000000000000000317", 2023-05-31T20:38:19.980 INFO:teuthology.orchestra.run.smithi099.stdout: "obj00000000000000000022", 2023-05-31T20:38:19.980 INFO:teuthology.orchestra.run.smithi099.stdout: "obj00000000000000000000", 2023-05-31T20:38:19.981 INFO:teuthology.orchestra.run.smithi099.stdout: "obj00000000000000000021", 2023-05-31T20:38:19.981 INFO:teuthology.orchestra.run.smithi099.stdout: "obj00000000000000000009", 2023-05-31T20:38:19.981 INFO:teuthology.orchestra.run.smithi099.stdout: "obj00000000000000000271", 2023-05-31T20:38:19.981 INFO:teuthology.orchestra.run.smithi099.stdout: "obj00000000000000000008", 2023-05-31T20:38:19.981 INFO:teuthology.orchestra.run.smithi099.stdout: "obj00000000000000000018", 2023-05-31T20:38:19.981 INFO:teuthology.orchestra.run.smithi099.stdout: "obj00000000000000000007", 2023-05-31T20:38:19.982 INFO:teuthology.orchestra.run.smithi099.stdout: "obj00000000000000000017", 2023-05-31T20:38:19.982 INFO:teuthology.orchestra.run.smithi099.stdout: "obj00000000000000000162", 2023-05-31T20:38:19.982 INFO:teuthology.orchestra.run.smithi099.stdout: "obj00000000000000000016", 2023-05-31T20:38:19.982 INFO:teuthology.orchestra.run.smithi099.stdout: "obj00000000000000000003", 2023-05-31T20:38:19.982 INFO:teuthology.orchestra.run.smithi099.stdout: "obj00000000000000000423", 2023-05-31T20:38:19.982 INFO:teuthology.orchestra.run.smithi099.stdout: "obj00000000000000000002", 2023-05-31T20:38:19.983 INFO:teuthology.orchestra.run.smithi099.stdout: "obj00000000000000000012", 2023-05-31T20:38:19.983 INFO:teuthology.orchestra.run.smithi099.stdout: "obj00000000000000000001", 2023-05-31T20:38:19.983 INFO:teuthology.orchestra.run.smithi099.stdout: "obj00000000000000000011", 2023-05-31T20:38:19.983 INFO:teuthology.orchestra.run.smithi099.stdout: "obj00000000000000000283", 2023-05-31T20:38:19.983 INFO:teuthology.orchestra.run.smithi099.stdout: "obj00000000000000000010", 2023-05-31T20:38:19.983 INFO:teuthology.orchestra.run.smithi099.stdout: "obj00000000000000000061", 2023-05-31T20:38:19.983 INFO:teuthology.orchestra.run.smithi099.stdout: "obj00000000000000000050", 2023-05-31T20:38:19.984 INFO:teuthology.orchestra.run.smithi099.stdout: "obj00000000000000000029", 2023-05-31T20:38:19.984 INFO:teuthology.orchestra.run.smithi099.stdout: "obj00000000000000000133" 2023-05-31T20:38:19.984 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2023-05-31T20:38:19.984 INFO:teuthology.orchestra.run.smithi099.stdout:} 2023-05-31T20:38:19.985 DEBUG:boto:path=/pklxkd-58/ 2023-05-31T20:38:19.986 DEBUG:boto:auth_path=/pklxkd-58/ 2023-05-31T20:38:19.986 DEBUG:boto:Method: HEAD 2023-05-31T20:38:19.986 DEBUG:boto:Path: /pklxkd-58/ 2023-05-31T20:38:19.987 DEBUG:boto:Data: 2023-05-31T20:38:19.987 DEBUG:boto:Headers: {} 2023-05-31T20:38:19.988 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:38:19.988 DEBUG:boto:Port: 8000 2023-05-31T20:38:19.989 DEBUG:boto:Params: {} 2023-05-31T20:38:19.989 DEBUG:boto:Token: None 2023-05-31T20:38:19.990 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:38:19 GMT /pklxkd-58/ 2023-05-31T20:38:19.990 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:eV79GzMOeufr774jk4ITvUwXq9g= 2023-05-31T20:38:19.990 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:38:19 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:eV79GzMOeufr774jk4ITvUwXq9g=', 'Content-Length': '0'} 2023-05-31T20:38:19.994 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '42'), ('X-RGW-Bytes-Used', '126'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000bc30f84af538f4e7-006477b03b-1630-a1'), ('Content-Length', '0'), ('Date', 'Wed, 31 May 2023 20:38:19 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:38:19.994 DEBUG:boto:path=/pklxkd-58/obj00000000000000000133 2023-05-31T20:38:19.995 DEBUG:boto:auth_path=/pklxkd-58/obj00000000000000000133 2023-05-31T20:38:19.995 DEBUG:boto:Method: PUT 2023-05-31T20:38:19.995 DEBUG:boto:Path: /pklxkd-58/obj00000000000000000133 2023-05-31T20:38:19.996 DEBUG:boto:Data: 2023-05-31T20:38:19.996 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-05-31T20:38:19.997 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:38:19.997 DEBUG:boto:Port: 8000 2023-05-31T20:38:19.997 DEBUG:boto:Params: {} 2023-05-31T20:38:19.998 DEBUG:boto:Token: None 2023-05-31T20:38:19.998 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Wed, 31 May 2023 20:38:19 GMT /pklxkd-58/obj00000000000000000133 2023-05-31T20:38:19.999 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:OBy6ka3uRcj5wxO8wf2LYuUxK00= 2023-05-31T20:38:19.999 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Wed, 31 May 2023 20:38:19 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:OBy6ka3uRcj5wxO8wf2LYuUxK00='} 2023-05-31T20:38:20.044 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000ed89cbe32d2dde67-006477b03b-1630-a1'), ('Date', 'Wed, 31 May 2023 20:38:20 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:38:20.045 DEBUG:boto:path=/pklxkd-58/ 2023-05-31T20:38:20.046 DEBUG:boto:auth_path=/pklxkd-58/ 2023-05-31T20:38:20.046 DEBUG:boto:Method: HEAD 2023-05-31T20:38:20.046 DEBUG:boto:Path: /pklxkd-58/ 2023-05-31T20:38:20.047 DEBUG:boto:Data: 2023-05-31T20:38:20.047 DEBUG:boto:Headers: {} 2023-05-31T20:38:20.047 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:38:20.048 DEBUG:boto:Port: 8000 2023-05-31T20:38:20.050 DEBUG:boto:Params: {} 2023-05-31T20:38:20.050 DEBUG:boto:Token: None 2023-05-31T20:38:20.051 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:38:20 GMT /pklxkd-58/ 2023-05-31T20:38:20.051 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:4Eer1z/FcQvT7jk1J7ab2Xq4gug= 2023-05-31T20:38:20.052 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:38:20 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:4Eer1z/FcQvT7jk1J7ab2Xq4gug=', 'Content-Length': '0'} 2023-05-31T20:38:20.056 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '43'), ('X-RGW-Bytes-Used', '129'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000005fc4d2e766d2d63f-006477b03c-1630-a1'), ('Content-Length', '0'), ('Date', 'Wed, 31 May 2023 20:38:20 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:38:20.057 DEBUG:boto:path=/pklxkd-58/obj00000000000000000317 2023-05-31T20:38:20.058 DEBUG:boto:auth_path=/pklxkd-58/obj00000000000000000317 2023-05-31T20:38:20.059 DEBUG:boto:Method: PUT 2023-05-31T20:38:20.060 DEBUG:boto:Path: /pklxkd-58/obj00000000000000000317 2023-05-31T20:38:20.061 DEBUG:boto:Data: 2023-05-31T20:38:20.062 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-05-31T20:38:20.063 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:38:20.063 DEBUG:boto:Port: 8000 2023-05-31T20:38:20.064 DEBUG:boto:Params: {} 2023-05-31T20:38:20.065 DEBUG:boto:Token: None 2023-05-31T20:38:20.066 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Wed, 31 May 2023 20:38:20 GMT /pklxkd-58/obj00000000000000000317 2023-05-31T20:38:20.066 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:bQNbNQLLUKQ7L5zo/3krncBezfM= 2023-05-31T20:38:20.067 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Wed, 31 May 2023 20:38:20 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:bQNbNQLLUKQ7L5zo/3krncBezfM='} 2023-05-31T20:38:20.073 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000040198f899bd82297-006477b03c-1630-a1'), ('Date', 'Wed, 31 May 2023 20:38:20 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:38:20.074 DEBUG:boto:path=/pklxkd-58/ 2023-05-31T20:38:20.075 DEBUG:boto:auth_path=/pklxkd-58/ 2023-05-31T20:38:20.075 DEBUG:boto:Method: HEAD 2023-05-31T20:38:20.076 DEBUG:boto:Path: /pklxkd-58/ 2023-05-31T20:38:20.077 DEBUG:boto:Data: 2023-05-31T20:38:20.077 DEBUG:boto:Headers: {} 2023-05-31T20:38:20.078 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:38:20.078 DEBUG:boto:Port: 8000 2023-05-31T20:38:20.079 DEBUG:boto:Params: {} 2023-05-31T20:38:20.080 DEBUG:boto:Token: None 2023-05-31T20:38:20.080 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:38:20 GMT /pklxkd-58/ 2023-05-31T20:38:20.081 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:4Eer1z/FcQvT7jk1J7ab2Xq4gug= 2023-05-31T20:38:20.082 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:38:20 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:4Eer1z/FcQvT7jk1J7ab2Xq4gug=', 'Content-Length': '0'} 2023-05-31T20:38:20.086 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '44'), ('X-RGW-Bytes-Used', '132'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000e227e4fd49b9623e-006477b03c-1630-a1'), ('Content-Length', '0'), ('Date', 'Wed, 31 May 2023 20:38:20 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:38:20.087 DEBUG:boto:path=/pklxkd-58/obj00000000000000000008 2023-05-31T20:38:20.087 DEBUG:boto:auth_path=/pklxkd-58/obj00000000000000000008 2023-05-31T20:38:20.088 DEBUG:boto:Method: PUT 2023-05-31T20:38:20.089 DEBUG:boto:Path: /pklxkd-58/obj00000000000000000008 2023-05-31T20:38:20.089 DEBUG:boto:Data: 2023-05-31T20:38:20.090 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-05-31T20:38:20.091 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:38:20.091 DEBUG:boto:Port: 8000 2023-05-31T20:38:20.092 DEBUG:boto:Params: {} 2023-05-31T20:38:20.093 DEBUG:boto:Token: None 2023-05-31T20:38:20.094 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Wed, 31 May 2023 20:38:20 GMT /pklxkd-58/obj00000000000000000008 2023-05-31T20:38:20.094 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:11j5YvPr8ZWdavezmBIFh+DrPs4= 2023-05-31T20:38:20.095 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Wed, 31 May 2023 20:38:20 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:11j5YvPr8ZWdavezmBIFh+DrPs4='} 2023-05-31T20:38:20.141 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000d82bcc836653d6bc-006477b03c-1630-a1'), ('Date', 'Wed, 31 May 2023 20:38:20 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:38:20.142 DEBUG:boto:path=/pklxkd-58/ 2023-05-31T20:38:20.142 DEBUG:boto:auth_path=/pklxkd-58/ 2023-05-31T20:38:20.143 DEBUG:boto:Method: HEAD 2023-05-31T20:38:20.144 DEBUG:boto:Path: /pklxkd-58/ 2023-05-31T20:38:20.144 DEBUG:boto:Data: 2023-05-31T20:38:20.145 DEBUG:boto:Headers: {} 2023-05-31T20:38:20.145 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:38:20.146 DEBUG:boto:Port: 8000 2023-05-31T20:38:20.146 DEBUG:boto:Params: {} 2023-05-31T20:38:20.147 DEBUG:boto:Token: None 2023-05-31T20:38:20.147 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:38:20 GMT /pklxkd-58/ 2023-05-31T20:38:20.148 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:4Eer1z/FcQvT7jk1J7ab2Xq4gug= 2023-05-31T20:38:20.148 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:38:20 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:4Eer1z/FcQvT7jk1J7ab2Xq4gug=', 'Content-Length': '0'} 2023-05-31T20:38:20.152 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '44'), ('X-RGW-Bytes-Used', '132'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000e1fd707f83d77d4c-006477b03c-1630-a1'), ('Content-Length', '0'), ('Date', 'Wed, 31 May 2023 20:38:20 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:38:20.152 DEBUG:boto:path=/pklxkd-58/obj00000000000000000016 2023-05-31T20:38:20.153 DEBUG:boto:auth_path=/pklxkd-58/obj00000000000000000016 2023-05-31T20:38:20.153 DEBUG:boto:Method: PUT 2023-05-31T20:38:20.154 DEBUG:boto:Path: /pklxkd-58/obj00000000000000000016 2023-05-31T20:38:20.155 DEBUG:boto:Data: 2023-05-31T20:38:20.155 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-05-31T20:38:20.156 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:38:20.156 DEBUG:boto:Port: 8000 2023-05-31T20:38:20.157 DEBUG:boto:Params: {} 2023-05-31T20:38:20.157 DEBUG:boto:Token: None 2023-05-31T20:38:20.158 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Wed, 31 May 2023 20:38:20 GMT /pklxkd-58/obj00000000000000000016 2023-05-31T20:38:20.159 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:/KjJ6XCwkVXBXEWnD92/l+Iqfwk= 2023-05-31T20:38:20.159 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Wed, 31 May 2023 20:38:20 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:/KjJ6XCwkVXBXEWnD92/l+Iqfwk='} 2023-05-31T20:38:20.166 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000007a895adbb5349d87-006477b03c-1630-a1'), ('Date', 'Wed, 31 May 2023 20:38:20 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:38:20.167 DEBUG:boto:path=/pklxkd-58/ 2023-05-31T20:38:20.168 DEBUG:boto:auth_path=/pklxkd-58/ 2023-05-31T20:38:20.168 DEBUG:boto:Method: HEAD 2023-05-31T20:38:20.169 DEBUG:boto:Path: /pklxkd-58/ 2023-05-31T20:38:20.169 DEBUG:boto:Data: 2023-05-31T20:38:20.169 DEBUG:boto:Headers: {} 2023-05-31T20:38:20.170 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:38:20.171 DEBUG:boto:Port: 8000 2023-05-31T20:38:20.171 DEBUG:boto:Params: {} 2023-05-31T20:38:20.172 DEBUG:boto:Token: None 2023-05-31T20:38:20.173 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:38:20 GMT /pklxkd-58/ 2023-05-31T20:38:20.174 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:4Eer1z/FcQvT7jk1J7ab2Xq4gug= 2023-05-31T20:38:20.175 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:38:20 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:4Eer1z/FcQvT7jk1J7ab2Xq4gug=', 'Content-Length': '0'} 2023-05-31T20:38:20.178 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '44'), ('X-RGW-Bytes-Used', '132'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000007ee901feeb61db21-006477b03c-1630-a1'), ('Content-Length', '0'), ('Date', 'Wed, 31 May 2023 20:38:20 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:38:20.179 DEBUG:boto:path=/pklxkd-58/obj00000000000000000050 2023-05-31T20:38:20.180 DEBUG:boto:auth_path=/pklxkd-58/obj00000000000000000050 2023-05-31T20:38:20.180 DEBUG:boto:Method: PUT 2023-05-31T20:38:20.181 DEBUG:boto:Path: /pklxkd-58/obj00000000000000000050 2023-05-31T20:38:20.181 DEBUG:boto:Data: 2023-05-31T20:38:20.181 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-05-31T20:38:20.182 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:38:20.182 DEBUG:boto:Port: 8000 2023-05-31T20:38:20.183 DEBUG:boto:Params: {} 2023-05-31T20:38:20.183 DEBUG:boto:Token: None 2023-05-31T20:38:20.183 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Wed, 31 May 2023 20:38:20 GMT /pklxkd-58/obj00000000000000000050 2023-05-31T20:38:20.184 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:0AZ8mg17kzmK9tjHcyH14XCfh+Y= 2023-05-31T20:38:20.185 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Wed, 31 May 2023 20:38:20 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:0AZ8mg17kzmK9tjHcyH14XCfh+Y='} 2023-05-31T20:38:20.228 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000403de045bba7ad14-006477b03c-1630-a1'), ('Date', 'Wed, 31 May 2023 20:38:20 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:38:20.229 DEBUG:boto:path=/pklxkd-58/ 2023-05-31T20:38:20.230 DEBUG:boto:auth_path=/pklxkd-58/ 2023-05-31T20:38:20.230 DEBUG:boto:Method: HEAD 2023-05-31T20:38:20.230 DEBUG:boto:Path: /pklxkd-58/ 2023-05-31T20:38:20.231 DEBUG:boto:Data: 2023-05-31T20:38:20.231 DEBUG:boto:Headers: {} 2023-05-31T20:38:20.231 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:38:20.232 DEBUG:boto:Port: 8000 2023-05-31T20:38:20.232 DEBUG:boto:Params: {} 2023-05-31T20:38:20.233 DEBUG:boto:Token: None 2023-05-31T20:38:20.233 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:38:20 GMT /pklxkd-58/ 2023-05-31T20:38:20.233 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:4Eer1z/FcQvT7jk1J7ab2Xq4gug= 2023-05-31T20:38:20.234 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:38:20 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:4Eer1z/FcQvT7jk1J7ab2Xq4gug=', 'Content-Length': '0'} 2023-05-31T20:38:20.237 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '44'), ('X-RGW-Bytes-Used', '132'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000002cdbb82c66c2b3ac-006477b03c-1630-a1'), ('Content-Length', '0'), ('Date', 'Wed, 31 May 2023 20:38:20 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:38:20.238 DEBUG:boto:path=/pklxkd-58/obj00000000000000000271 2023-05-31T20:38:20.238 DEBUG:boto:auth_path=/pklxkd-58/obj00000000000000000271 2023-05-31T20:38:20.238 DEBUG:boto:Method: PUT 2023-05-31T20:38:20.239 DEBUG:boto:Path: /pklxkd-58/obj00000000000000000271 2023-05-31T20:38:20.239 DEBUG:boto:Data: 2023-05-31T20:38:20.240 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-05-31T20:38:20.240 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:38:20.240 DEBUG:boto:Port: 8000 2023-05-31T20:38:20.241 DEBUG:boto:Params: {} 2023-05-31T20:38:20.241 DEBUG:boto:Token: None 2023-05-31T20:38:20.242 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Wed, 31 May 2023 20:38:20 GMT /pklxkd-58/obj00000000000000000271 2023-05-31T20:38:20.242 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:gxi4EcZnG0x2CYjLr9rD/qyXQk0= 2023-05-31T20:38:20.242 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Wed, 31 May 2023 20:38:20 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:gxi4EcZnG0x2CYjLr9rD/qyXQk0='} 2023-05-31T20:38:20.248 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000a47c45642370f0d0-006477b03c-1630-a1'), ('Date', 'Wed, 31 May 2023 20:38:20 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:38:20.249 DEBUG:boto:path=/pklxkd-58/ 2023-05-31T20:38:20.250 DEBUG:boto:auth_path=/pklxkd-58/ 2023-05-31T20:38:20.251 DEBUG:boto:Method: HEAD 2023-05-31T20:38:20.251 DEBUG:boto:Path: /pklxkd-58/ 2023-05-31T20:38:20.252 DEBUG:boto:Data: 2023-05-31T20:38:20.253 DEBUG:boto:Headers: {} 2023-05-31T20:38:20.253 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:38:20.254 DEBUG:boto:Port: 8000 2023-05-31T20:38:20.255 DEBUG:boto:Params: {} 2023-05-31T20:38:20.255 DEBUG:boto:Token: None 2023-05-31T20:38:20.256 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:38:20 GMT /pklxkd-58/ 2023-05-31T20:38:20.257 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:4Eer1z/FcQvT7jk1J7ab2Xq4gug= 2023-05-31T20:38:20.257 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:38:20 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:4Eer1z/FcQvT7jk1J7ab2Xq4gug=', 'Content-Length': '0'} 2023-05-31T20:38:20.260 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '45'), ('X-RGW-Bytes-Used', '135'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000002ba946103dc920b4-006477b03c-1630-a1'), ('Content-Length', '0'), ('Date', 'Wed, 31 May 2023 20:38:20 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:38:20.261 DEBUG:boto:path=/pklxkd-58/obj00000000000000000029 2023-05-31T20:38:20.261 DEBUG:boto:auth_path=/pklxkd-58/obj00000000000000000029 2023-05-31T20:38:20.262 DEBUG:boto:Method: PUT 2023-05-31T20:38:20.262 DEBUG:boto:Path: /pklxkd-58/obj00000000000000000029 2023-05-31T20:38:20.262 DEBUG:boto:Data: 2023-05-31T20:38:20.263 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-05-31T20:38:20.263 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:38:20.263 DEBUG:boto:Port: 8000 2023-05-31T20:38:20.264 DEBUG:boto:Params: {} 2023-05-31T20:38:20.265 DEBUG:boto:Token: None 2023-05-31T20:38:20.265 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Wed, 31 May 2023 20:38:20 GMT /pklxkd-58/obj00000000000000000029 2023-05-31T20:38:20.266 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:6wlTcYCIcSmByIpSklxx01+vFb0= 2023-05-31T20:38:20.267 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Wed, 31 May 2023 20:38:20 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:6wlTcYCIcSmByIpSklxx01+vFb0='} 2023-05-31T20:38:20.312 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000eb78d86c67b1d888-006477b03c-1630-a1'), ('Date', 'Wed, 31 May 2023 20:38:20 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:38:20.313 DEBUG:boto:path=/pklxkd-58/ 2023-05-31T20:38:20.313 DEBUG:boto:auth_path=/pklxkd-58/ 2023-05-31T20:38:20.314 DEBUG:boto:Method: HEAD 2023-05-31T20:38:20.314 DEBUG:boto:Path: /pklxkd-58/ 2023-05-31T20:38:20.315 DEBUG:boto:Data: 2023-05-31T20:38:20.315 DEBUG:boto:Headers: {} 2023-05-31T20:38:20.316 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:38:20.316 DEBUG:boto:Port: 8000 2023-05-31T20:38:20.317 DEBUG:boto:Params: {} 2023-05-31T20:38:20.317 DEBUG:boto:Token: None 2023-05-31T20:38:20.318 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:38:20 GMT /pklxkd-58/ 2023-05-31T20:38:20.318 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:4Eer1z/FcQvT7jk1J7ab2Xq4gug= 2023-05-31T20:38:20.319 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:38:20 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:4Eer1z/FcQvT7jk1J7ab2Xq4gug=', 'Content-Length': '0'} 2023-05-31T20:38:20.323 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '46'), ('X-RGW-Bytes-Used', '138'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000a54ea47906e82cba-006477b03c-1630-a1'), ('Content-Length', '0'), ('Date', 'Wed, 31 May 2023 20:38:20 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:38:20.324 DEBUG:boto:path=/pklxkd-58/obj00000000000000000467 2023-05-31T20:38:20.324 DEBUG:boto:auth_path=/pklxkd-58/obj00000000000000000467 2023-05-31T20:38:20.325 DEBUG:boto:Method: PUT 2023-05-31T20:38:20.325 DEBUG:boto:Path: /pklxkd-58/obj00000000000000000467 2023-05-31T20:38:20.325 DEBUG:boto:Data: 2023-05-31T20:38:20.326 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-05-31T20:38:20.326 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:38:20.326 DEBUG:boto:Port: 8000 2023-05-31T20:38:20.328 DEBUG:boto:Params: {} 2023-05-31T20:38:20.328 DEBUG:boto:Token: None 2023-05-31T20:38:20.328 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Wed, 31 May 2023 20:38:20 GMT /pklxkd-58/obj00000000000000000467 2023-05-31T20:38:20.329 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:zEzrEA866Jtfp6xq7aH5d9ckIHg= 2023-05-31T20:38:20.329 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Wed, 31 May 2023 20:38:20 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:zEzrEA866Jtfp6xq7aH5d9ckIHg='} 2023-05-31T20:38:20.335 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000080a9f8b72b69d96a-006477b03c-1630-a1'), ('Date', 'Wed, 31 May 2023 20:38:20 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:38:20.336 DEBUG:boto:path=/pklxkd-58/ 2023-05-31T20:38:20.336 DEBUG:boto:auth_path=/pklxkd-58/ 2023-05-31T20:38:20.337 DEBUG:boto:Method: HEAD 2023-05-31T20:38:20.337 DEBUG:boto:Path: /pklxkd-58/ 2023-05-31T20:38:20.337 DEBUG:boto:Data: 2023-05-31T20:38:20.338 DEBUG:boto:Headers: {} 2023-05-31T20:38:20.339 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:38:20.339 DEBUG:boto:Port: 8000 2023-05-31T20:38:20.339 DEBUG:boto:Params: {} 2023-05-31T20:38:20.340 DEBUG:boto:Token: None 2023-05-31T20:38:20.340 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:38:20 GMT /pklxkd-58/ 2023-05-31T20:38:20.341 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:4Eer1z/FcQvT7jk1J7ab2Xq4gug= 2023-05-31T20:38:20.341 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:38:20 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:4Eer1z/FcQvT7jk1J7ab2Xq4gug=', 'Content-Length': '0'} 2023-05-31T20:38:20.344 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '47'), ('X-RGW-Bytes-Used', '141'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000a6c220108a886e82-006477b03c-1630-a1'), ('Content-Length', '0'), ('Date', 'Wed, 31 May 2023 20:38:20 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:38:20.345 DEBUG:boto:path=/pklxkd-58/obj00000000000000000023 2023-05-31T20:38:20.345 DEBUG:boto:auth_path=/pklxkd-58/obj00000000000000000023 2023-05-31T20:38:20.345 DEBUG:boto:Method: PUT 2023-05-31T20:38:20.346 DEBUG:boto:Path: /pklxkd-58/obj00000000000000000023 2023-05-31T20:38:20.346 DEBUG:boto:Data: 2023-05-31T20:38:20.346 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-05-31T20:38:20.347 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:38:20.347 DEBUG:boto:Port: 8000 2023-05-31T20:38:20.347 DEBUG:boto:Params: {} 2023-05-31T20:38:20.348 DEBUG:boto:Token: None 2023-05-31T20:38:20.348 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Wed, 31 May 2023 20:38:20 GMT /pklxkd-58/obj00000000000000000023 2023-05-31T20:38:20.349 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:/WqI9t3rNocOlL0m5Of8ja4Iqp8= 2023-05-31T20:38:20.349 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Wed, 31 May 2023 20:38:20 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:/WqI9t3rNocOlL0m5Of8ja4Iqp8='} 2023-05-31T20:38:20.358 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000410733d77b9d4a71-006477b03c-1630-a1'), ('Date', 'Wed, 31 May 2023 20:38:20 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:38:20.358 DEBUG:boto:path=/pklxkd-58/ 2023-05-31T20:38:20.359 DEBUG:boto:auth_path=/pklxkd-58/ 2023-05-31T20:38:20.359 DEBUG:boto:Method: HEAD 2023-05-31T20:38:20.360 DEBUG:boto:Path: /pklxkd-58/ 2023-05-31T20:38:20.360 DEBUG:boto:Data: 2023-05-31T20:38:20.360 DEBUG:boto:Headers: {} 2023-05-31T20:38:20.361 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:38:20.361 DEBUG:boto:Port: 8000 2023-05-31T20:38:20.361 DEBUG:boto:Params: {} 2023-05-31T20:38:20.362 DEBUG:boto:Token: None 2023-05-31T20:38:20.362 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:38:20 GMT /pklxkd-58/ 2023-05-31T20:38:20.363 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:4Eer1z/FcQvT7jk1J7ab2Xq4gug= 2023-05-31T20:38:20.363 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:38:20 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:4Eer1z/FcQvT7jk1J7ab2Xq4gug=', 'Content-Length': '0'} 2023-05-31T20:38:20.366 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '48'), ('X-RGW-Bytes-Used', '144'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000a1fcd28fb2d6ff53-006477b03c-1630-a1'), ('Content-Length', '0'), ('Date', 'Wed, 31 May 2023 20:38:20 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:38:20.367 DEBUG:boto:path=/pklxkd-58/obj00000000000000000056 2023-05-31T20:38:20.367 DEBUG:boto:auth_path=/pklxkd-58/obj00000000000000000056 2023-05-31T20:38:20.368 DEBUG:boto:Method: PUT 2023-05-31T20:38:20.368 DEBUG:boto:Path: /pklxkd-58/obj00000000000000000056 2023-05-31T20:38:20.369 DEBUG:boto:Data: 2023-05-31T20:38:20.369 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-05-31T20:38:20.370 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:38:20.370 DEBUG:boto:Port: 8000 2023-05-31T20:38:20.370 DEBUG:boto:Params: {} 2023-05-31T20:38:20.371 DEBUG:boto:Token: None 2023-05-31T20:38:20.371 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Wed, 31 May 2023 20:38:20 GMT /pklxkd-58/obj00000000000000000056 2023-05-31T20:38:20.372 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:swOcURs048SyJwZmxHz8GRoLs6E= 2023-05-31T20:38:20.372 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Wed, 31 May 2023 20:38:20 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:swOcURs048SyJwZmxHz8GRoLs6E='} 2023-05-31T20:38:20.420 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000009f30187b2070b507-006477b03c-1630-a1'), ('Date', 'Wed, 31 May 2023 20:38:20 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:38:20.421 DEBUG:boto:path=/pklxkd-58/ 2023-05-31T20:38:20.422 DEBUG:boto:auth_path=/pklxkd-58/ 2023-05-31T20:38:20.422 DEBUG:boto:Method: HEAD 2023-05-31T20:38:20.423 DEBUG:boto:Path: /pklxkd-58/ 2023-05-31T20:38:20.423 DEBUG:boto:Data: 2023-05-31T20:38:20.423 DEBUG:boto:Headers: {} 2023-05-31T20:38:20.424 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:38:20.424 DEBUG:boto:Port: 8000 2023-05-31T20:38:20.424 DEBUG:boto:Params: {} 2023-05-31T20:38:20.425 DEBUG:boto:Token: None 2023-05-31T20:38:20.425 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:38:20 GMT /pklxkd-58/ 2023-05-31T20:38:20.426 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:4Eer1z/FcQvT7jk1J7ab2Xq4gug= 2023-05-31T20:38:20.427 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:38:20 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:4Eer1z/FcQvT7jk1J7ab2Xq4gug=', 'Content-Length': '0'} 2023-05-31T20:38:20.430 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '49'), ('X-RGW-Bytes-Used', '147'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000006f2cec95a00e9dfe-006477b03c-1630-a1'), ('Content-Length', '0'), ('Date', 'Wed, 31 May 2023 20:38:20 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:38:20.431 DEBUG:boto:path=/pklxkd-58/obj00000000000000000423 2023-05-31T20:38:20.431 DEBUG:boto:auth_path=/pklxkd-58/obj00000000000000000423 2023-05-31T20:38:20.431 DEBUG:boto:Method: PUT 2023-05-31T20:38:20.432 DEBUG:boto:Path: /pklxkd-58/obj00000000000000000423 2023-05-31T20:38:20.432 DEBUG:boto:Data: 2023-05-31T20:38:20.433 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-05-31T20:38:20.433 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:38:20.433 DEBUG:boto:Port: 8000 2023-05-31T20:38:20.434 DEBUG:boto:Params: {} 2023-05-31T20:38:20.434 DEBUG:boto:Token: None 2023-05-31T20:38:20.435 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Wed, 31 May 2023 20:38:20 GMT /pklxkd-58/obj00000000000000000423 2023-05-31T20:38:20.435 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:DwANT8U0rTWsPpjI7Kx19P7VM/g= 2023-05-31T20:38:20.436 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Wed, 31 May 2023 20:38:20 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:DwANT8U0rTWsPpjI7Kx19P7VM/g='} 2023-05-31T20:38:20.442 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000004c3f7f6d889132f5-006477b03c-1630-a1'), ('Date', 'Wed, 31 May 2023 20:38:20 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:38:20.443 DEBUG:boto:path=/pklxkd-58/ 2023-05-31T20:38:20.444 DEBUG:boto:auth_path=/pklxkd-58/ 2023-05-31T20:38:20.444 DEBUG:boto:Method: HEAD 2023-05-31T20:38:20.444 DEBUG:boto:Path: /pklxkd-58/ 2023-05-31T20:38:20.445 DEBUG:boto:Data: 2023-05-31T20:38:20.445 DEBUG:boto:Headers: {} 2023-05-31T20:38:20.445 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:38:20.446 DEBUG:boto:Port: 8000 2023-05-31T20:38:20.446 DEBUG:boto:Params: {} 2023-05-31T20:38:20.447 DEBUG:boto:Token: None 2023-05-31T20:38:20.447 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:38:20 GMT /pklxkd-58/ 2023-05-31T20:38:20.448 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:4Eer1z/FcQvT7jk1J7ab2Xq4gug= 2023-05-31T20:38:20.448 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:38:20 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:4Eer1z/FcQvT7jk1J7ab2Xq4gug=', 'Content-Length': '0'} 2023-05-31T20:38:20.451 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '50'), ('X-RGW-Bytes-Used', '150'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000007625edc40fa3c816-006477b03c-1630-a1'), ('Content-Length', '0'), ('Date', 'Wed, 31 May 2023 20:38:20 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:38:20.452 DEBUG:boto:path=/pklxkd-58/obj00000000000000000004 2023-05-31T20:38:20.452 DEBUG:boto:auth_path=/pklxkd-58/obj00000000000000000004 2023-05-31T20:38:20.453 DEBUG:boto:Method: PUT 2023-05-31T20:38:20.453 DEBUG:boto:Path: /pklxkd-58/obj00000000000000000004 2023-05-31T20:38:20.454 DEBUG:boto:Data: 2023-05-31T20:38:20.454 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-05-31T20:38:20.454 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:38:20.455 DEBUG:boto:Port: 8000 2023-05-31T20:38:20.455 DEBUG:boto:Params: {} 2023-05-31T20:38:20.456 DEBUG:boto:Token: None 2023-05-31T20:38:20.456 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Wed, 31 May 2023 20:38:20 GMT /pklxkd-58/obj00000000000000000004 2023-05-31T20:38:20.457 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:1+RdUjqmwnA/M+kvEC5N5kKM5Cg= 2023-05-31T20:38:20.457 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Wed, 31 May 2023 20:38:20 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:1+RdUjqmwnA/M+kvEC5N5kKM5Cg='} 2023-05-31T20:38:20.500 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000000d3dd210f65dbb25-006477b03c-1630-a1'), ('Date', 'Wed, 31 May 2023 20:38:20 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:38:20.501 DEBUG:boto:path=/pklxkd-58/ 2023-05-31T20:38:20.501 DEBUG:boto:auth_path=/pklxkd-58/ 2023-05-31T20:38:20.502 DEBUG:boto:Method: HEAD 2023-05-31T20:38:20.502 DEBUG:boto:Path: /pklxkd-58/ 2023-05-31T20:38:20.502 DEBUG:boto:Data: 2023-05-31T20:38:20.503 DEBUG:boto:Headers: {} 2023-05-31T20:38:20.503 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:38:20.503 DEBUG:boto:Port: 8000 2023-05-31T20:38:20.504 DEBUG:boto:Params: {} 2023-05-31T20:38:20.504 DEBUG:boto:Token: None 2023-05-31T20:38:20.504 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:38:20 GMT /pklxkd-58/ 2023-05-31T20:38:20.505 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:4Eer1z/FcQvT7jk1J7ab2Xq4gug= 2023-05-31T20:38:20.505 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:38:20 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:4Eer1z/FcQvT7jk1J7ab2Xq4gug=', 'Content-Length': '0'} 2023-05-31T20:38:20.508 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '50'), ('X-RGW-Bytes-Used', '150'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000002b3f47c2e2043706-006477b03c-1630-a1'), ('Content-Length', '0'), ('Date', 'Wed, 31 May 2023 20:38:20 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:38:20.509 DEBUG:boto:path=/pklxkd-58/obj00000000000000000011 2023-05-31T20:38:20.509 DEBUG:boto:auth_path=/pklxkd-58/obj00000000000000000011 2023-05-31T20:38:20.509 DEBUG:boto:Method: PUT 2023-05-31T20:38:20.510 DEBUG:boto:Path: /pklxkd-58/obj00000000000000000011 2023-05-31T20:38:20.510 DEBUG:boto:Data: 2023-05-31T20:38:20.510 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-05-31T20:38:20.511 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:38:20.511 DEBUG:boto:Port: 8000 2023-05-31T20:38:20.511 DEBUG:boto:Params: {} 2023-05-31T20:38:20.512 DEBUG:boto:Token: None 2023-05-31T20:38:20.512 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Wed, 31 May 2023 20:38:20 GMT /pklxkd-58/obj00000000000000000011 2023-05-31T20:38:20.513 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:V6Xi2Ii49+rptLURhzxN8j7XEbw= 2023-05-31T20:38:20.513 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Wed, 31 May 2023 20:38:20 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:V6Xi2Ii49+rptLURhzxN8j7XEbw='} 2023-05-31T20:38:20.522 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000e5fb5e869c0be46a-006477b03c-1630-a1'), ('Date', 'Wed, 31 May 2023 20:38:20 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:38:20.522 DEBUG:boto:path=/pklxkd-58/ 2023-05-31T20:38:20.522 DEBUG:boto:auth_path=/pklxkd-58/ 2023-05-31T20:38:20.523 DEBUG:boto:Method: HEAD 2023-05-31T20:38:20.523 DEBUG:boto:Path: /pklxkd-58/ 2023-05-31T20:38:20.524 DEBUG:boto:Data: 2023-05-31T20:38:20.524 DEBUG:boto:Headers: {} 2023-05-31T20:38:20.524 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:38:20.525 DEBUG:boto:Port: 8000 2023-05-31T20:38:20.525 DEBUG:boto:Params: {} 2023-05-31T20:38:20.525 DEBUG:boto:Token: None 2023-05-31T20:38:20.526 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:38:20 GMT /pklxkd-58/ 2023-05-31T20:38:20.526 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:4Eer1z/FcQvT7jk1J7ab2Xq4gug= 2023-05-31T20:38:20.526 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:38:20 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:4Eer1z/FcQvT7jk1J7ab2Xq4gug=', 'Content-Length': '0'} 2023-05-31T20:38:20.529 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '50'), ('X-RGW-Bytes-Used', '150'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000051c6268308fd8557-006477b03c-1630-a1'), ('Content-Length', '0'), ('Date', 'Wed, 31 May 2023 20:38:20 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:38:20.530 DEBUG:boto:path=/pklxkd-58/obj00000000000000000162 2023-05-31T20:38:20.533 DEBUG:boto:auth_path=/pklxkd-58/obj00000000000000000162 2023-05-31T20:38:20.534 DEBUG:boto:Method: PUT 2023-05-31T20:38:20.534 DEBUG:boto:Path: /pklxkd-58/obj00000000000000000162 2023-05-31T20:38:20.535 DEBUG:boto:Data: 2023-05-31T20:38:20.535 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-05-31T20:38:20.536 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:38:20.537 DEBUG:boto:Port: 8000 2023-05-31T20:38:20.537 DEBUG:boto:Params: {} 2023-05-31T20:38:20.538 DEBUG:boto:Token: None 2023-05-31T20:38:20.538 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Wed, 31 May 2023 20:38:20 GMT /pklxkd-58/obj00000000000000000162 2023-05-31T20:38:20.539 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:HwzkqyiHeIq5WMFyCYdQbF1pKjo= 2023-05-31T20:38:20.540 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Wed, 31 May 2023 20:38:20 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:HwzkqyiHeIq5WMFyCYdQbF1pKjo='} 2023-05-31T20:38:20.584 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000adccfac621968eb2-006477b03c-1630-a1'), ('Date', 'Wed, 31 May 2023 20:38:20 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:38:20.585 DEBUG:boto:path=/pklxkd-58/ 2023-05-31T20:38:20.586 DEBUG:boto:auth_path=/pklxkd-58/ 2023-05-31T20:38:20.586 DEBUG:boto:Method: HEAD 2023-05-31T20:38:20.586 DEBUG:boto:Path: /pklxkd-58/ 2023-05-31T20:38:20.587 DEBUG:boto:Data: 2023-05-31T20:38:20.587 DEBUG:boto:Headers: {} 2023-05-31T20:38:20.587 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:38:20.588 DEBUG:boto:Port: 8000 2023-05-31T20:38:20.588 DEBUG:boto:Params: {} 2023-05-31T20:38:20.589 DEBUG:boto:Token: None 2023-05-31T20:38:20.589 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:38:20 GMT /pklxkd-58/ 2023-05-31T20:38:20.589 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:4Eer1z/FcQvT7jk1J7ab2Xq4gug= 2023-05-31T20:38:20.590 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:38:20 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:4Eer1z/FcQvT7jk1J7ab2Xq4gug=', 'Content-Length': '0'} 2023-05-31T20:38:20.593 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '51'), ('X-RGW-Bytes-Used', '153'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000cfe95ac002e7e140-006477b03c-1630-a1'), ('Content-Length', '0'), ('Date', 'Wed, 31 May 2023 20:38:20 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:38:20.594 DEBUG:boto:path=/pklxkd-58/obj00000000000000000022 2023-05-31T20:38:20.594 DEBUG:boto:auth_path=/pklxkd-58/obj00000000000000000022 2023-05-31T20:38:20.595 DEBUG:boto:Method: PUT 2023-05-31T20:38:20.596 DEBUG:boto:Path: /pklxkd-58/obj00000000000000000022 2023-05-31T20:38:20.596 DEBUG:boto:Data: 2023-05-31T20:38:20.597 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-05-31T20:38:20.597 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:38:20.598 DEBUG:boto:Port: 8000 2023-05-31T20:38:20.598 DEBUG:boto:Params: {} 2023-05-31T20:38:20.598 DEBUG:boto:Token: None 2023-05-31T20:38:20.599 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Wed, 31 May 2023 20:38:20 GMT /pklxkd-58/obj00000000000000000022 2023-05-31T20:38:20.599 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:qQQk8IP4jXEQPdupfImJJbtfxA0= 2023-05-31T20:38:20.600 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Wed, 31 May 2023 20:38:20 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:qQQk8IP4jXEQPdupfImJJbtfxA0='} 2023-05-31T20:38:20.614 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000b0ff9ea5f81590ba-006477b03c-1630-a1'), ('Date', 'Wed, 31 May 2023 20:38:20 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:38:20.615 DEBUG:boto:path=/pklxkd-58/ 2023-05-31T20:38:20.616 DEBUG:boto:auth_path=/pklxkd-58/ 2023-05-31T20:38:20.616 DEBUG:boto:Method: HEAD 2023-05-31T20:38:20.617 DEBUG:boto:Path: /pklxkd-58/ 2023-05-31T20:38:20.617 DEBUG:boto:Data: 2023-05-31T20:38:20.617 DEBUG:boto:Headers: {} 2023-05-31T20:38:20.618 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:38:20.618 DEBUG:boto:Port: 8000 2023-05-31T20:38:20.618 DEBUG:boto:Params: {} 2023-05-31T20:38:20.619 DEBUG:boto:Token: None 2023-05-31T20:38:20.619 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:38:20 GMT /pklxkd-58/ 2023-05-31T20:38:20.620 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:4Eer1z/FcQvT7jk1J7ab2Xq4gug= 2023-05-31T20:38:20.620 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:38:20 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:4Eer1z/FcQvT7jk1J7ab2Xq4gug=', 'Content-Length': '0'} 2023-05-31T20:38:20.623 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '51'), ('X-RGW-Bytes-Used', '153'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000007805e0cb33bce0ae-006477b03c-1630-a1'), ('Content-Length', '0'), ('Date', 'Wed, 31 May 2023 20:38:20 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:38:20.624 DEBUG:boto:path=/pklxkd-58/obj00000000000000000012 2023-05-31T20:38:20.624 DEBUG:boto:auth_path=/pklxkd-58/obj00000000000000000012 2023-05-31T20:38:20.625 DEBUG:boto:Method: PUT 2023-05-31T20:38:20.625 DEBUG:boto:Path: /pklxkd-58/obj00000000000000000012 2023-05-31T20:38:20.625 DEBUG:boto:Data: 2023-05-31T20:38:20.626 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-05-31T20:38:20.626 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:38:20.626 DEBUG:boto:Port: 8000 2023-05-31T20:38:20.627 DEBUG:boto:Params: {} 2023-05-31T20:38:20.627 DEBUG:boto:Token: None 2023-05-31T20:38:20.627 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Wed, 31 May 2023 20:38:20 GMT /pklxkd-58/obj00000000000000000012 2023-05-31T20:38:20.628 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:i7WgvP8P5KoX7l/ESZAO9tpOI0Q= 2023-05-31T20:38:20.628 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Wed, 31 May 2023 20:38:20 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:i7WgvP8P5KoX7l/ESZAO9tpOI0Q='} 2023-05-31T20:38:20.672 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000070b15c82559d410f-006477b03c-1630-a1'), ('Date', 'Wed, 31 May 2023 20:38:20 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:38:20.673 DEBUG:boto:path=/pklxkd-58/ 2023-05-31T20:38:20.673 DEBUG:boto:auth_path=/pklxkd-58/ 2023-05-31T20:38:20.674 DEBUG:boto:Method: HEAD 2023-05-31T20:38:20.674 DEBUG:boto:Path: /pklxkd-58/ 2023-05-31T20:38:20.675 DEBUG:boto:Data: 2023-05-31T20:38:20.675 DEBUG:boto:Headers: {} 2023-05-31T20:38:20.675 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:38:20.676 DEBUG:boto:Port: 8000 2023-05-31T20:38:20.676 DEBUG:boto:Params: {} 2023-05-31T20:38:20.676 DEBUG:boto:Token: None 2023-05-31T20:38:20.677 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:38:20 GMT /pklxkd-58/ 2023-05-31T20:38:20.677 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:4Eer1z/FcQvT7jk1J7ab2Xq4gug= 2023-05-31T20:38:20.678 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:38:20 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:4Eer1z/FcQvT7jk1J7ab2Xq4gug=', 'Content-Length': '0'} 2023-05-31T20:38:20.681 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '51'), ('X-RGW-Bytes-Used', '153'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000a190c74dc9d8f7d0-006477b03c-1630-a1'), ('Content-Length', '0'), ('Date', 'Wed, 31 May 2023 20:38:20 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:38:20.681 DEBUG:boto:path=/pklxkd-58/obj00000000000000000018 2023-05-31T20:38:20.682 DEBUG:boto:auth_path=/pklxkd-58/obj00000000000000000018 2023-05-31T20:38:20.682 DEBUG:boto:Method: PUT 2023-05-31T20:38:20.683 DEBUG:boto:Path: /pklxkd-58/obj00000000000000000018 2023-05-31T20:38:20.684 DEBUG:boto:Data: 2023-05-31T20:38:20.684 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-05-31T20:38:20.685 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:38:20.686 DEBUG:boto:Port: 8000 2023-05-31T20:38:20.687 DEBUG:boto:Params: {} 2023-05-31T20:38:20.688 DEBUG:boto:Token: None 2023-05-31T20:38:20.689 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Wed, 31 May 2023 20:38:20 GMT /pklxkd-58/obj00000000000000000018 2023-05-31T20:38:20.690 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:MTZYNyN15aRP9Iwe79Mqzpt2iB8= 2023-05-31T20:38:20.691 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Wed, 31 May 2023 20:38:20 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:MTZYNyN15aRP9Iwe79Mqzpt2iB8='} 2023-05-31T20:38:20.701 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000335afb14e6c90b0b-006477b03c-1630-a1'), ('Date', 'Wed, 31 May 2023 20:38:20 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:38:20.702 DEBUG:boto:path=/pklxkd-58/ 2023-05-31T20:38:20.702 DEBUG:boto:auth_path=/pklxkd-58/ 2023-05-31T20:38:20.703 DEBUG:boto:Method: HEAD 2023-05-31T20:38:20.703 DEBUG:boto:Path: /pklxkd-58/ 2023-05-31T20:38:20.704 DEBUG:boto:Data: 2023-05-31T20:38:20.705 DEBUG:boto:Headers: {} 2023-05-31T20:38:20.705 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:38:20.705 DEBUG:boto:Port: 8000 2023-05-31T20:38:20.706 DEBUG:boto:Params: {} 2023-05-31T20:38:20.706 DEBUG:boto:Token: None 2023-05-31T20:38:20.707 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:38:20 GMT /pklxkd-58/ 2023-05-31T20:38:20.707 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:4Eer1z/FcQvT7jk1J7ab2Xq4gug= 2023-05-31T20:38:20.707 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:38:20 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:4Eer1z/FcQvT7jk1J7ab2Xq4gug=', 'Content-Length': '0'} 2023-05-31T20:38:20.710 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '52'), ('X-RGW-Bytes-Used', '156'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000fdc8a1e886dc9d28-006477b03c-1630-a1'), ('Content-Length', '0'), ('Date', 'Wed, 31 May 2023 20:38:20 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:38:20.711 DEBUG:boto:path=/pklxkd-58/obj00000000000000000017 2023-05-31T20:38:20.711 DEBUG:boto:auth_path=/pklxkd-58/obj00000000000000000017 2023-05-31T20:38:20.712 DEBUG:boto:Method: PUT 2023-05-31T20:38:20.712 DEBUG:boto:Path: /pklxkd-58/obj00000000000000000017 2023-05-31T20:38:20.713 DEBUG:boto:Data: 2023-05-31T20:38:20.713 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-05-31T20:38:20.713 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:38:20.714 DEBUG:boto:Port: 8000 2023-05-31T20:38:20.714 DEBUG:boto:Params: {} 2023-05-31T20:38:20.715 DEBUG:boto:Token: None 2023-05-31T20:38:20.715 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Wed, 31 May 2023 20:38:20 GMT /pklxkd-58/obj00000000000000000017 2023-05-31T20:38:20.716 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:uek82h1MAutaSQeosgDsCvj4j+E= 2023-05-31T20:38:20.716 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Wed, 31 May 2023 20:38:20 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:uek82h1MAutaSQeosgDsCvj4j+E='} 2023-05-31T20:38:20.764 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000093540c12187b15dd-006477b03c-1630-a1'), ('Date', 'Wed, 31 May 2023 20:38:20 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:38:20.765 DEBUG:boto:path=/pklxkd-58/ 2023-05-31T20:38:20.766 DEBUG:boto:auth_path=/pklxkd-58/ 2023-05-31T20:38:20.766 DEBUG:boto:Method: HEAD 2023-05-31T20:38:20.767 DEBUG:boto:Path: /pklxkd-58/ 2023-05-31T20:38:20.767 DEBUG:boto:Data: 2023-05-31T20:38:20.768 DEBUG:boto:Headers: {} 2023-05-31T20:38:20.768 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:38:20.768 DEBUG:boto:Port: 8000 2023-05-31T20:38:20.769 DEBUG:boto:Params: {} 2023-05-31T20:38:20.769 DEBUG:boto:Token: None 2023-05-31T20:38:20.770 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:38:20 GMT /pklxkd-58/ 2023-05-31T20:38:20.770 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:4Eer1z/FcQvT7jk1J7ab2Xq4gug= 2023-05-31T20:38:20.770 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:38:20 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:4Eer1z/FcQvT7jk1J7ab2Xq4gug=', 'Content-Length': '0'} 2023-05-31T20:38:20.774 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '52'), ('X-RGW-Bytes-Used', '156'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000007f548b6f11cc2795-006477b03c-1630-a1'), ('Content-Length', '0'), ('Date', 'Wed, 31 May 2023 20:38:20 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:38:20.775 DEBUG:boto:path=/pklxkd-58/obj00000000000000000006 2023-05-31T20:38:20.775 DEBUG:boto:auth_path=/pklxkd-58/obj00000000000000000006 2023-05-31T20:38:20.776 DEBUG:boto:Method: PUT 2023-05-31T20:38:20.776 DEBUG:boto:Path: /pklxkd-58/obj00000000000000000006 2023-05-31T20:38:20.776 DEBUG:boto:Data: 2023-05-31T20:38:20.777 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-05-31T20:38:20.777 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:38:20.777 DEBUG:boto:Port: 8000 2023-05-31T20:38:20.778 DEBUG:boto:Params: {} 2023-05-31T20:38:20.778 DEBUG:boto:Token: None 2023-05-31T20:38:20.779 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Wed, 31 May 2023 20:38:20 GMT /pklxkd-58/obj00000000000000000006 2023-05-31T20:38:20.780 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:pRV2TxnJ0fL3fV6De3LKsS7nurw= 2023-05-31T20:38:20.780 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Wed, 31 May 2023 20:38:20 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:pRV2TxnJ0fL3fV6De3LKsS7nurw='} 2023-05-31T20:38:20.787 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000dd1eb0b49f4f269f-006477b03c-1630-a1'), ('Date', 'Wed, 31 May 2023 20:38:20 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:38:20.788 DEBUG:boto:path=/pklxkd-58/ 2023-05-31T20:38:20.788 DEBUG:boto:auth_path=/pklxkd-58/ 2023-05-31T20:38:20.789 DEBUG:boto:Method: HEAD 2023-05-31T20:38:20.790 DEBUG:boto:Path: /pklxkd-58/ 2023-05-31T20:38:20.791 DEBUG:boto:Data: 2023-05-31T20:38:20.791 DEBUG:boto:Headers: {} 2023-05-31T20:38:20.792 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:38:20.793 DEBUG:boto:Port: 8000 2023-05-31T20:38:20.794 DEBUG:boto:Params: {} 2023-05-31T20:38:20.794 DEBUG:boto:Token: None 2023-05-31T20:38:20.794 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:38:20 GMT /pklxkd-58/ 2023-05-31T20:38:20.795 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:4Eer1z/FcQvT7jk1J7ab2Xq4gug= 2023-05-31T20:38:20.795 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:38:20 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:4Eer1z/FcQvT7jk1J7ab2Xq4gug=', 'Content-Length': '0'} 2023-05-31T20:38:20.798 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '52'), ('X-RGW-Bytes-Used', '156'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000085406eb87d4abf0e-006477b03c-1630-a1'), ('Content-Length', '0'), ('Date', 'Wed, 31 May 2023 20:38:20 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:38:20.799 DEBUG:boto:path=/pklxkd-58/obj00000000000000000021 2023-05-31T20:38:20.799 DEBUG:boto:auth_path=/pklxkd-58/obj00000000000000000021 2023-05-31T20:38:20.800 DEBUG:boto:Method: PUT 2023-05-31T20:38:20.800 DEBUG:boto:Path: /pklxkd-58/obj00000000000000000021 2023-05-31T20:38:20.800 DEBUG:boto:Data: 2023-05-31T20:38:20.801 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-05-31T20:38:20.801 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:38:20.801 DEBUG:boto:Port: 8000 2023-05-31T20:38:20.802 DEBUG:boto:Params: {} 2023-05-31T20:38:20.802 DEBUG:boto:Token: None 2023-05-31T20:38:20.803 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Wed, 31 May 2023 20:38:20 GMT /pklxkd-58/obj00000000000000000021 2023-05-31T20:38:20.803 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:5XFbgVamzfEKeZmpFMKmlHMRdOw= 2023-05-31T20:38:20.804 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Wed, 31 May 2023 20:38:20 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:5XFbgVamzfEKeZmpFMKmlHMRdOw='} 2023-05-31T20:38:20.852 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000d955e0ddc4e80997-006477b03c-1630-a1'), ('Date', 'Wed, 31 May 2023 20:38:20 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:38:20.853 DEBUG:boto:path=/pklxkd-58/ 2023-05-31T20:38:20.854 DEBUG:boto:auth_path=/pklxkd-58/ 2023-05-31T20:38:20.854 DEBUG:boto:Method: HEAD 2023-05-31T20:38:20.854 DEBUG:boto:Path: /pklxkd-58/ 2023-05-31T20:38:20.855 DEBUG:boto:Data: 2023-05-31T20:38:20.856 DEBUG:boto:Headers: {} 2023-05-31T20:38:20.856 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:38:20.857 DEBUG:boto:Port: 8000 2023-05-31T20:38:20.858 DEBUG:boto:Params: {} 2023-05-31T20:38:20.859 DEBUG:boto:Token: None 2023-05-31T20:38:20.860 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:38:20 GMT /pklxkd-58/ 2023-05-31T20:38:20.861 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:4Eer1z/FcQvT7jk1J7ab2Xq4gug= 2023-05-31T20:38:20.862 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:38:20 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:4Eer1z/FcQvT7jk1J7ab2Xq4gug=', 'Content-Length': '0'} 2023-05-31T20:38:20.865 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '52'), ('X-RGW-Bytes-Used', '156'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000008df6594918e88742-006477b03c-1630-a1'), ('Content-Length', '0'), ('Date', 'Wed, 31 May 2023 20:38:20 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:38:20.865 DEBUG:boto:path=/pklxkd-58/obj00000000000000000005 2023-05-31T20:38:20.866 DEBUG:boto:auth_path=/pklxkd-58/obj00000000000000000005 2023-05-31T20:38:20.866 DEBUG:boto:Method: PUT 2023-05-31T20:38:20.867 DEBUG:boto:Path: /pklxkd-58/obj00000000000000000005 2023-05-31T20:38:20.867 DEBUG:boto:Data: 2023-05-31T20:38:20.867 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-05-31T20:38:20.868 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:38:20.868 DEBUG:boto:Port: 8000 2023-05-31T20:38:20.869 DEBUG:boto:Params: {} 2023-05-31T20:38:20.869 DEBUG:boto:Token: None 2023-05-31T20:38:20.870 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Wed, 31 May 2023 20:38:20 GMT /pklxkd-58/obj00000000000000000005 2023-05-31T20:38:20.871 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:IlBB/cG+KKFTqLpEYxIsj9oEoUM= 2023-05-31T20:38:20.872 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Wed, 31 May 2023 20:38:20 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:IlBB/cG+KKFTqLpEYxIsj9oEoUM='} 2023-05-31T20:38:20.879 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000006ae29a455a73de78-006477b03c-1630-a1'), ('Date', 'Wed, 31 May 2023 20:38:20 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:38:20.880 DEBUG:boto:path=/pklxkd-58/ 2023-05-31T20:38:20.880 DEBUG:boto:auth_path=/pklxkd-58/ 2023-05-31T20:38:20.880 DEBUG:boto:Method: HEAD 2023-05-31T20:38:20.881 DEBUG:boto:Path: /pklxkd-58/ 2023-05-31T20:38:20.881 DEBUG:boto:Data: 2023-05-31T20:38:20.882 DEBUG:boto:Headers: {} 2023-05-31T20:38:20.882 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:38:20.883 DEBUG:boto:Port: 8000 2023-05-31T20:38:20.883 DEBUG:boto:Params: {} 2023-05-31T20:38:20.883 DEBUG:boto:Token: None 2023-05-31T20:38:20.884 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:38:20 GMT /pklxkd-58/ 2023-05-31T20:38:20.884 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:4Eer1z/FcQvT7jk1J7ab2Xq4gug= 2023-05-31T20:38:20.885 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:38:20 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:4Eer1z/FcQvT7jk1J7ab2Xq4gug=', 'Content-Length': '0'} 2023-05-31T20:38:20.888 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '52'), ('X-RGW-Bytes-Used', '156'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000079f349443ed300a2-006477b03c-1630-a1'), ('Content-Length', '0'), ('Date', 'Wed, 31 May 2023 20:38:20 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:38:20.889 DEBUG:boto:path=/pklxkd-58/obj00000000000000000009 2023-05-31T20:38:20.891 DEBUG:boto:auth_path=/pklxkd-58/obj00000000000000000009 2023-05-31T20:38:20.892 DEBUG:boto:Method: PUT 2023-05-31T20:38:20.892 DEBUG:boto:Path: /pklxkd-58/obj00000000000000000009 2023-05-31T20:38:20.892 DEBUG:boto:Data: 2023-05-31T20:38:20.893 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-05-31T20:38:20.893 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:38:20.893 DEBUG:boto:Port: 8000 2023-05-31T20:38:20.894 DEBUG:boto:Params: {} 2023-05-31T20:38:20.894 DEBUG:boto:Token: None 2023-05-31T20:38:20.895 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Wed, 31 May 2023 20:38:20 GMT /pklxkd-58/obj00000000000000000009 2023-05-31T20:38:20.895 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:/5aljS2LbdeppphQHk2ydNZXJjE= 2023-05-31T20:38:20.896 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Wed, 31 May 2023 20:38:20 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:/5aljS2LbdeppphQHk2ydNZXJjE='} 2023-05-31T20:38:20.941 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000009601236f7178b223-006477b03c-1630-a1'), ('Date', 'Wed, 31 May 2023 20:38:20 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:38:20.942 DEBUG:boto:path=/pklxkd-58/ 2023-05-31T20:38:20.942 DEBUG:boto:auth_path=/pklxkd-58/ 2023-05-31T20:38:20.942 DEBUG:boto:Method: HEAD 2023-05-31T20:38:20.943 DEBUG:boto:Path: /pklxkd-58/ 2023-05-31T20:38:20.943 DEBUG:boto:Data: 2023-05-31T20:38:20.943 DEBUG:boto:Headers: {} 2023-05-31T20:38:20.944 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:38:20.944 DEBUG:boto:Port: 8000 2023-05-31T20:38:20.944 DEBUG:boto:Params: {} 2023-05-31T20:38:20.945 DEBUG:boto:Token: None 2023-05-31T20:38:20.945 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:38:20 GMT /pklxkd-58/ 2023-05-31T20:38:20.946 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:4Eer1z/FcQvT7jk1J7ab2Xq4gug= 2023-05-31T20:38:20.946 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:38:20 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:4Eer1z/FcQvT7jk1J7ab2Xq4gug=', 'Content-Length': '0'} 2023-05-31T20:38:20.949 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '52'), ('X-RGW-Bytes-Used', '156'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000c6036afffcc14999-006477b03c-1630-a1'), ('Content-Length', '0'), ('Date', 'Wed, 31 May 2023 20:38:20 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:38:20.950 DEBUG:boto:path=/pklxkd-58/obj00000000000000000283 2023-05-31T20:38:20.950 DEBUG:boto:auth_path=/pklxkd-58/obj00000000000000000283 2023-05-31T20:38:20.950 DEBUG:boto:Method: PUT 2023-05-31T20:38:20.951 DEBUG:boto:Path: /pklxkd-58/obj00000000000000000283 2023-05-31T20:38:20.951 DEBUG:boto:Data: 2023-05-31T20:38:20.951 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-05-31T20:38:20.952 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:38:20.952 DEBUG:boto:Port: 8000 2023-05-31T20:38:20.953 DEBUG:boto:Params: {} 2023-05-31T20:38:20.953 DEBUG:boto:Token: None 2023-05-31T20:38:20.953 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Wed, 31 May 2023 20:38:20 GMT /pklxkd-58/obj00000000000000000283 2023-05-31T20:38:20.954 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:moRObdzmCbwaRICNVrf7gfH05Hg= 2023-05-31T20:38:20.955 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Wed, 31 May 2023 20:38:20 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:moRObdzmCbwaRICNVrf7gfH05Hg='} 2023-05-31T20:38:20.961 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000dd6b360fddab443b-006477b03c-1630-a1'), ('Date', 'Wed, 31 May 2023 20:38:20 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:38:20.961 DEBUG:boto:path=/pklxkd-58/ 2023-05-31T20:38:20.962 DEBUG:boto:auth_path=/pklxkd-58/ 2023-05-31T20:38:20.962 DEBUG:boto:Method: HEAD 2023-05-31T20:38:20.962 DEBUG:boto:Path: /pklxkd-58/ 2023-05-31T20:38:20.963 DEBUG:boto:Data: 2023-05-31T20:38:20.964 DEBUG:boto:Headers: {} 2023-05-31T20:38:20.964 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:38:20.964 DEBUG:boto:Port: 8000 2023-05-31T20:38:20.965 DEBUG:boto:Params: {} 2023-05-31T20:38:20.965 DEBUG:boto:Token: None 2023-05-31T20:38:20.965 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:38:20 GMT /pklxkd-58/ 2023-05-31T20:38:20.966 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:4Eer1z/FcQvT7jk1J7ab2Xq4gug= 2023-05-31T20:38:20.967 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:38:20 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:4Eer1z/FcQvT7jk1J7ab2Xq4gug=', 'Content-Length': '0'} 2023-05-31T20:38:20.970 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '53'), ('X-RGW-Bytes-Used', '159'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000037d4af587935b5a5-006477b03c-1630-a1'), ('Content-Length', '0'), ('Date', 'Wed, 31 May 2023 20:38:20 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:38:20.970 DEBUG:boto:path=/pklxkd-58/obj00000000000000000000 2023-05-31T20:38:20.971 DEBUG:boto:auth_path=/pklxkd-58/obj00000000000000000000 2023-05-31T20:38:20.971 DEBUG:boto:Method: PUT 2023-05-31T20:38:20.972 DEBUG:boto:Path: /pklxkd-58/obj00000000000000000000 2023-05-31T20:38:20.972 DEBUG:boto:Data: 2023-05-31T20:38:20.972 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-05-31T20:38:20.973 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:38:20.973 DEBUG:boto:Port: 8000 2023-05-31T20:38:20.974 DEBUG:boto:Params: {} 2023-05-31T20:38:20.974 DEBUG:boto:Token: None 2023-05-31T20:38:20.975 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Wed, 31 May 2023 20:38:20 GMT /pklxkd-58/obj00000000000000000000 2023-05-31T20:38:20.975 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:UKlh49W2CHcYZRttNPqv3aD+mLI= 2023-05-31T20:38:20.975 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Wed, 31 May 2023 20:38:20 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:UKlh49W2CHcYZRttNPqv3aD+mLI='} 2023-05-31T20:38:21.024 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000092716b9ededddab7-006477b03c-1630-a1'), ('Date', 'Wed, 31 May 2023 20:38:20 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:38:21.025 DEBUG:boto:path=/pklxkd-58/ 2023-05-31T20:38:21.025 DEBUG:boto:auth_path=/pklxkd-58/ 2023-05-31T20:38:21.026 DEBUG:boto:Method: HEAD 2023-05-31T20:38:21.027 DEBUG:boto:Path: /pklxkd-58/ 2023-05-31T20:38:21.027 DEBUG:boto:Data: 2023-05-31T20:38:21.027 DEBUG:boto:Headers: {} 2023-05-31T20:38:21.028 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:38:21.028 DEBUG:boto:Port: 8000 2023-05-31T20:38:21.028 DEBUG:boto:Params: {} 2023-05-31T20:38:21.029 DEBUG:boto:Token: None 2023-05-31T20:38:21.030 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:38:21 GMT /pklxkd-58/ 2023-05-31T20:38:21.030 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:s31FU4vdTVg+6vxPE5KFsQQx3XQ= 2023-05-31T20:38:21.030 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:38:21 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:s31FU4vdTVg+6vxPE5KFsQQx3XQ=', 'Content-Length': '0'} 2023-05-31T20:38:21.034 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '53'), ('X-RGW-Bytes-Used', '159'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000009676f4d45dade8ce-006477b03d-1630-a1'), ('Content-Length', '0'), ('Date', 'Wed, 31 May 2023 20:38:21 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:38:21.034 DEBUG:boto:path=/pklxkd-58/obj00000000000000000001 2023-05-31T20:38:21.035 DEBUG:boto:auth_path=/pklxkd-58/obj00000000000000000001 2023-05-31T20:38:21.035 DEBUG:boto:Method: PUT 2023-05-31T20:38:21.036 DEBUG:boto:Path: /pklxkd-58/obj00000000000000000001 2023-05-31T20:38:21.036 DEBUG:boto:Data: 2023-05-31T20:38:21.036 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-05-31T20:38:21.037 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:38:21.037 DEBUG:boto:Port: 8000 2023-05-31T20:38:21.037 DEBUG:boto:Params: {} 2023-05-31T20:38:21.038 DEBUG:boto:Token: None 2023-05-31T20:38:21.038 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Wed, 31 May 2023 20:38:21 GMT /pklxkd-58/obj00000000000000000001 2023-05-31T20:38:21.039 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:GKOUxheR4LVB0ffhyQc0elul/9A= 2023-05-31T20:38:21.040 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Wed, 31 May 2023 20:38:21 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:GKOUxheR4LVB0ffhyQc0elul/9A='} 2023-05-31T20:38:21.047 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000002431e4e57e8e4343-006477b03d-1630-a1'), ('Date', 'Wed, 31 May 2023 20:38:21 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:38:21.047 DEBUG:boto:path=/pklxkd-58/ 2023-05-31T20:38:21.049 DEBUG:boto:auth_path=/pklxkd-58/ 2023-05-31T20:38:21.049 DEBUG:boto:Method: HEAD 2023-05-31T20:38:21.049 DEBUG:boto:Path: /pklxkd-58/ 2023-05-31T20:38:21.050 DEBUG:boto:Data: 2023-05-31T20:38:21.050 DEBUG:boto:Headers: {} 2023-05-31T20:38:21.050 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:38:21.051 DEBUG:boto:Port: 8000 2023-05-31T20:38:21.051 DEBUG:boto:Params: {} 2023-05-31T20:38:21.052 DEBUG:boto:Token: None 2023-05-31T20:38:21.052 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:38:21 GMT /pklxkd-58/ 2023-05-31T20:38:21.053 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:s31FU4vdTVg+6vxPE5KFsQQx3XQ= 2023-05-31T20:38:21.053 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:38:21 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:s31FU4vdTVg+6vxPE5KFsQQx3XQ=', 'Content-Length': '0'} 2023-05-31T20:38:21.056 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '53'), ('X-RGW-Bytes-Used', '159'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000abdabbfd1b6f8b99-006477b03d-1630-a1'), ('Content-Length', '0'), ('Date', 'Wed, 31 May 2023 20:38:21 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:38:21.057 DEBUG:boto:path=/pklxkd-58/obj00000000000000000007 2023-05-31T20:38:21.058 DEBUG:boto:auth_path=/pklxkd-58/obj00000000000000000007 2023-05-31T20:38:21.059 DEBUG:boto:Method: PUT 2023-05-31T20:38:21.060 DEBUG:boto:Path: /pklxkd-58/obj00000000000000000007 2023-05-31T20:38:21.060 DEBUG:boto:Data: 2023-05-31T20:38:21.061 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-05-31T20:38:21.062 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:38:21.062 DEBUG:boto:Port: 8000 2023-05-31T20:38:21.063 DEBUG:boto:Params: {} 2023-05-31T20:38:21.064 DEBUG:boto:Token: None 2023-05-31T20:38:21.064 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Wed, 31 May 2023 20:38:21 GMT /pklxkd-58/obj00000000000000000007 2023-05-31T20:38:21.065 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:8l3lEO5ENF6yBCNRrJA0fVnFgxg= 2023-05-31T20:38:21.065 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Wed, 31 May 2023 20:38:21 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:8l3lEO5ENF6yBCNRrJA0fVnFgxg='} 2023-05-31T20:38:21.108 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000003655558adf719635-006477b03d-1630-a1'), ('Date', 'Wed, 31 May 2023 20:38:21 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:38:21.109 DEBUG:boto:path=/pklxkd-58/ 2023-05-31T20:38:21.110 DEBUG:boto:auth_path=/pklxkd-58/ 2023-05-31T20:38:21.111 DEBUG:boto:Method: HEAD 2023-05-31T20:38:21.112 DEBUG:boto:Path: /pklxkd-58/ 2023-05-31T20:38:21.113 DEBUG:boto:Data: 2023-05-31T20:38:21.114 DEBUG:boto:Headers: {} 2023-05-31T20:38:21.115 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:38:21.116 DEBUG:boto:Port: 8000 2023-05-31T20:38:21.117 DEBUG:boto:Params: {} 2023-05-31T20:38:21.118 DEBUG:boto:Token: None 2023-05-31T20:38:21.119 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:38:21 GMT /pklxkd-58/ 2023-05-31T20:38:21.119 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:s31FU4vdTVg+6vxPE5KFsQQx3XQ= 2023-05-31T20:38:21.120 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:38:21 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:s31FU4vdTVg+6vxPE5KFsQQx3XQ=', 'Content-Length': '0'} 2023-05-31T20:38:21.124 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '53'), ('X-RGW-Bytes-Used', '159'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000008dfe965152858540-006477b03d-1630-a1'), ('Content-Length', '0'), ('Date', 'Wed, 31 May 2023 20:38:21 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:38:21.124 DEBUG:boto:path=/pklxkd-58/obj00000000000000000010 2023-05-31T20:38:21.125 DEBUG:boto:auth_path=/pklxkd-58/obj00000000000000000010 2023-05-31T20:38:21.125 DEBUG:boto:Method: PUT 2023-05-31T20:38:21.126 DEBUG:boto:Path: /pklxkd-58/obj00000000000000000010 2023-05-31T20:38:21.126 DEBUG:boto:Data: 2023-05-31T20:38:21.126 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-05-31T20:38:21.127 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:38:21.127 DEBUG:boto:Port: 8000 2023-05-31T20:38:21.128 DEBUG:boto:Params: {} 2023-05-31T20:38:21.128 DEBUG:boto:Token: None 2023-05-31T20:38:21.128 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Wed, 31 May 2023 20:38:21 GMT /pklxkd-58/obj00000000000000000010 2023-05-31T20:38:21.129 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:mpG6i+5lKXdFR3KYwBWDcJJGxbI= 2023-05-31T20:38:21.129 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Wed, 31 May 2023 20:38:21 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:mpG6i+5lKXdFR3KYwBWDcJJGxbI='} 2023-05-31T20:38:21.141 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000008a2f124b07fa03f6-006477b03d-1630-a1'), ('Date', 'Wed, 31 May 2023 20:38:21 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:38:21.142 DEBUG:boto:path=/pklxkd-58/ 2023-05-31T20:38:21.142 DEBUG:boto:auth_path=/pklxkd-58/ 2023-05-31T20:38:21.143 DEBUG:boto:Method: HEAD 2023-05-31T20:38:21.143 DEBUG:boto:Path: /pklxkd-58/ 2023-05-31T20:38:21.143 DEBUG:boto:Data: 2023-05-31T20:38:21.144 DEBUG:boto:Headers: {} 2023-05-31T20:38:21.144 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:38:21.144 DEBUG:boto:Port: 8000 2023-05-31T20:38:21.145 DEBUG:boto:Params: {} 2023-05-31T20:38:21.145 DEBUG:boto:Token: None 2023-05-31T20:38:21.146 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:38:21 GMT /pklxkd-58/ 2023-05-31T20:38:21.146 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:s31FU4vdTVg+6vxPE5KFsQQx3XQ= 2023-05-31T20:38:21.147 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:38:21 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:s31FU4vdTVg+6vxPE5KFsQQx3XQ=', 'Content-Length': '0'} 2023-05-31T20:38:21.150 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '53'), ('X-RGW-Bytes-Used', '159'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000008ea0e635905249ce-006477b03d-1630-a1'), ('Content-Length', '0'), ('Date', 'Wed, 31 May 2023 20:38:21 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:38:21.151 DEBUG:boto:path=/pklxkd-58/obj00000000000000000061 2023-05-31T20:38:21.151 DEBUG:boto:auth_path=/pklxkd-58/obj00000000000000000061 2023-05-31T20:38:21.151 DEBUG:boto:Method: PUT 2023-05-31T20:38:21.152 DEBUG:boto:Path: /pklxkd-58/obj00000000000000000061 2023-05-31T20:38:21.152 DEBUG:boto:Data: 2023-05-31T20:38:21.152 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-05-31T20:38:21.153 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:38:21.153 DEBUG:boto:Port: 8000 2023-05-31T20:38:21.154 DEBUG:boto:Params: {} 2023-05-31T20:38:21.154 DEBUG:boto:Token: None 2023-05-31T20:38:21.154 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Wed, 31 May 2023 20:38:21 GMT /pklxkd-58/obj00000000000000000061 2023-05-31T20:38:21.155 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:fkdUcEoxeMxymdouml5jXVHB+dw= 2023-05-31T20:38:21.155 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Wed, 31 May 2023 20:38:21 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:fkdUcEoxeMxymdouml5jXVHB+dw='} 2023-05-31T20:38:21.200 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000009688cb22037f83c7-006477b03d-1630-a1'), ('Date', 'Wed, 31 May 2023 20:38:21 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:38:21.201 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'reshard', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'pklxkd-58', '--num-shards', '37', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-05-31T20:38:21.202 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'reshard', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'pklxkd-58', '--num-shards', '37', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-05-31T20:38:21.202 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket reshard --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --bucket pklxkd-58 --num-shards 37 --yes-i-really-mean-it --cluster c1 --debug-rgw 0 --debug-ms 0 2023-05-31T20:38:21.255 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2023-05-31T20:38:21.256 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2023-05-31T20:38:21.334 INFO:teuthology.orchestra.run.smithi099.stdout:tenant: 2023-05-31T20:38:21.334 INFO:teuthology.orchestra.run.smithi099.stdout:bucket name: pklxkd-58 2023-05-31T20:38:21.345 INFO:teuthology.orchestra.run.smithi099.stdout:total entries: 54 2023-05-31T20:38:21.411 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'shard', 'objects', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--num-shards', '37', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-31T20:38:21.411 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'shard', 'objects', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--num-shards', '37', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-31T20:38:21.412 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket shard objects --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --num-shards 37 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-05-31T20:38:21.462 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2023-05-31T20:38:21.463 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2023-05-31T20:38:21.540 INFO:teuthology.orchestra.run.smithi099.stdout:{ 2023-05-31T20:38:21.541 INFO:teuthology.orchestra.run.smithi099.stdout: "objs": [ 2023-05-31T20:38:21.541 INFO:teuthology.orchestra.run.smithi099.stdout: "obj00000000000000000060", 2023-05-31T20:38:21.541 INFO:teuthology.orchestra.run.smithi099.stdout: "obj00000000000000000000", 2023-05-31T20:38:21.541 INFO:teuthology.orchestra.run.smithi099.stdout: "obj00000000000000000011", 2023-05-31T20:38:21.542 INFO:teuthology.orchestra.run.smithi099.stdout: "obj00000000000000000001", 2023-05-31T20:38:21.542 INFO:teuthology.orchestra.run.smithi099.stdout: "obj00000000000000000033", 2023-05-31T20:38:21.542 INFO:teuthology.orchestra.run.smithi099.stdout: "obj00000000000000000065", 2023-05-31T20:38:21.542 INFO:teuthology.orchestra.run.smithi099.stdout: "obj00000000000000000005", 2023-05-31T20:38:21.542 INFO:teuthology.orchestra.run.smithi099.stdout: "obj00000000000000000016", 2023-05-31T20:38:21.542 INFO:teuthology.orchestra.run.smithi099.stdout: "obj00000000000000000048", 2023-05-31T20:38:21.542 INFO:teuthology.orchestra.run.smithi099.stdout: "obj00000000000000000015", 2023-05-31T20:38:21.543 INFO:teuthology.orchestra.run.smithi099.stdout: "obj00000000000000000026", 2023-05-31T20:38:21.543 INFO:teuthology.orchestra.run.smithi099.stdout: "obj00000000000000000030", 2023-05-31T20:38:21.543 INFO:teuthology.orchestra.run.smithi099.stdout: "obj00000000000000000062", 2023-05-31T20:38:21.543 INFO:teuthology.orchestra.run.smithi099.stdout: "obj00000000000000000009", 2023-05-31T20:38:21.543 INFO:teuthology.orchestra.run.smithi099.stdout: "obj00000000000000000040", 2023-05-31T20:38:21.543 INFO:teuthology.orchestra.run.smithi099.stdout: "obj00000000000000000024", 2023-05-31T20:38:21.543 INFO:teuthology.orchestra.run.smithi099.stdout: "obj00000000000000000012", 2023-05-31T20:38:21.544 INFO:teuthology.orchestra.run.smithi099.stdout: "obj00000000000000000002", 2023-05-31T20:38:21.544 INFO:teuthology.orchestra.run.smithi099.stdout: "obj00000000000000000013", 2023-05-31T20:38:21.544 INFO:teuthology.orchestra.run.smithi099.stdout: "obj00000000000000000045", 2023-05-31T20:38:21.544 INFO:teuthology.orchestra.run.smithi099.stdout: "obj00000000000000000006", 2023-05-31T20:38:21.544 INFO:teuthology.orchestra.run.smithi099.stdout: "obj00000000000000000017", 2023-05-31T20:38:21.544 INFO:teuthology.orchestra.run.smithi099.stdout: "obj00000000000000000007", 2023-05-31T20:38:21.545 INFO:teuthology.orchestra.run.smithi099.stdout: "obj00000000000000000039", 2023-05-31T20:38:21.545 INFO:teuthology.orchestra.run.smithi099.stdout: "obj00000000000000000027", 2023-05-31T20:38:21.545 INFO:teuthology.orchestra.run.smithi099.stdout: "obj00000000000000000010", 2023-05-31T20:38:21.545 INFO:teuthology.orchestra.run.smithi099.stdout: "obj00000000000000000021", 2023-05-31T20:38:21.545 INFO:teuthology.orchestra.run.smithi099.stdout: "obj00000000000000000053", 2023-05-31T20:38:21.545 INFO:teuthology.orchestra.run.smithi099.stdout: "obj00000000000000000020", 2023-05-31T20:38:21.545 INFO:teuthology.orchestra.run.smithi099.stdout: "obj00000000000000000004", 2023-05-31T20:38:21.546 INFO:teuthology.orchestra.run.smithi099.stdout: "obj00000000000000000036", 2023-05-31T20:38:21.546 INFO:teuthology.orchestra.run.smithi099.stdout: "obj00000000000000000003", 2023-05-31T20:38:21.546 INFO:teuthology.orchestra.run.smithi099.stdout: "obj00000000000000000014", 2023-05-31T20:38:21.546 INFO:teuthology.orchestra.run.smithi099.stdout: "obj00000000000000000019", 2023-05-31T20:38:21.546 INFO:teuthology.orchestra.run.smithi099.stdout: "obj00000000000000000057", 2023-05-31T20:38:21.547 INFO:teuthology.orchestra.run.smithi099.stdout: "obj00000000000000000018", 2023-05-31T20:38:21.547 INFO:teuthology.orchestra.run.smithi099.stdout: "obj00000000000000000008" 2023-05-31T20:38:21.547 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2023-05-31T20:38:21.547 INFO:teuthology.orchestra.run.smithi099.stdout:} 2023-05-31T20:38:21.548 DEBUG:boto:path=/pklxkd-58/ 2023-05-31T20:38:21.549 DEBUG:boto:auth_path=/pklxkd-58/ 2023-05-31T20:38:21.549 DEBUG:boto:Method: HEAD 2023-05-31T20:38:21.549 DEBUG:boto:Path: /pklxkd-58/ 2023-05-31T20:38:21.550 DEBUG:boto:Data: 2023-05-31T20:38:21.550 DEBUG:boto:Headers: {} 2023-05-31T20:38:21.550 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:38:21.551 DEBUG:boto:Port: 8000 2023-05-31T20:38:21.551 DEBUG:boto:Params: {} 2023-05-31T20:38:21.551 DEBUG:boto:Token: None 2023-05-31T20:38:21.552 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:38:21 GMT /pklxkd-58/ 2023-05-31T20:38:21.552 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:s31FU4vdTVg+6vxPE5KFsQQx3XQ= 2023-05-31T20:38:21.553 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:38:21 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:s31FU4vdTVg+6vxPE5KFsQQx3XQ=', 'Content-Length': '0'} 2023-05-31T20:38:21.556 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '54'), ('X-RGW-Bytes-Used', '162'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000006c08d9d0c2065ed3-006477b03d-1630-a1'), ('Content-Length', '0'), ('Date', 'Wed, 31 May 2023 20:38:21 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:38:21.556 DEBUG:boto:path=/pklxkd-58/obj00000000000000000019 2023-05-31T20:38:21.557 DEBUG:boto:auth_path=/pklxkd-58/obj00000000000000000019 2023-05-31T20:38:21.557 DEBUG:boto:Method: PUT 2023-05-31T20:38:21.558 DEBUG:boto:Path: /pklxkd-58/obj00000000000000000019 2023-05-31T20:38:21.558 DEBUG:boto:Data: 2023-05-31T20:38:21.558 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-05-31T20:38:21.559 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:38:21.559 DEBUG:boto:Port: 8000 2023-05-31T20:38:21.559 DEBUG:boto:Params: {} 2023-05-31T20:38:21.560 DEBUG:boto:Token: None 2023-05-31T20:38:21.560 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Wed, 31 May 2023 20:38:21 GMT /pklxkd-58/obj00000000000000000019 2023-05-31T20:38:21.560 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:MPSuFoF/mqAz8G07cV4dkvA2zMI= 2023-05-31T20:38:21.561 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Wed, 31 May 2023 20:38:21 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:MPSuFoF/mqAz8G07cV4dkvA2zMI='} 2023-05-31T20:38:21.574 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000ef9e011e1e427497-006477b03d-1630-a1'), ('Date', 'Wed, 31 May 2023 20:38:21 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:38:21.575 DEBUG:boto:path=/pklxkd-58/ 2023-05-31T20:38:21.576 DEBUG:boto:auth_path=/pklxkd-58/ 2023-05-31T20:38:21.576 DEBUG:boto:Method: HEAD 2023-05-31T20:38:21.577 DEBUG:boto:Path: /pklxkd-58/ 2023-05-31T20:38:21.577 DEBUG:boto:Data: 2023-05-31T20:38:21.578 DEBUG:boto:Headers: {} 2023-05-31T20:38:21.578 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:38:21.578 DEBUG:boto:Port: 8000 2023-05-31T20:38:21.579 DEBUG:boto:Params: {} 2023-05-31T20:38:21.579 DEBUG:boto:Token: None 2023-05-31T20:38:21.579 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:38:21 GMT /pklxkd-58/ 2023-05-31T20:38:21.580 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:s31FU4vdTVg+6vxPE5KFsQQx3XQ= 2023-05-31T20:38:21.580 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:38:21 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:s31FU4vdTVg+6vxPE5KFsQQx3XQ=', 'Content-Length': '0'} 2023-05-31T20:38:21.584 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '54'), ('X-RGW-Bytes-Used', '162'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000b85b09c66054312b-006477b03d-1630-a1'), ('Content-Length', '0'), ('Date', 'Wed, 31 May 2023 20:38:21 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:38:21.584 DEBUG:boto:path=/pklxkd-58/obj00000000000000000009 2023-05-31T20:38:21.585 DEBUG:boto:auth_path=/pklxkd-58/obj00000000000000000009 2023-05-31T20:38:21.585 DEBUG:boto:Method: PUT 2023-05-31T20:38:21.586 DEBUG:boto:Path: /pklxkd-58/obj00000000000000000009 2023-05-31T20:38:21.586 DEBUG:boto:Data: 2023-05-31T20:38:21.587 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-05-31T20:38:21.587 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:38:21.588 DEBUG:boto:Port: 8000 2023-05-31T20:38:21.588 DEBUG:boto:Params: {} 2023-05-31T20:38:21.588 DEBUG:boto:Token: None 2023-05-31T20:38:21.589 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Wed, 31 May 2023 20:38:21 GMT /pklxkd-58/obj00000000000000000009 2023-05-31T20:38:21.589 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:2U9HUOFT1/gZhsuUI0zFpOr9Dm8= 2023-05-31T20:38:21.590 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Wed, 31 May 2023 20:38:21 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:2U9HUOFT1/gZhsuUI0zFpOr9Dm8='} 2023-05-31T20:38:21.601 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000baa97e85b0bff164-006477b03d-1630-a1'), ('Date', 'Wed, 31 May 2023 20:38:21 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:38:21.602 DEBUG:boto:path=/pklxkd-58/ 2023-05-31T20:38:21.602 DEBUG:boto:auth_path=/pklxkd-58/ 2023-05-31T20:38:21.603 DEBUG:boto:Method: HEAD 2023-05-31T20:38:21.603 DEBUG:boto:Path: /pklxkd-58/ 2023-05-31T20:38:21.603 DEBUG:boto:Data: 2023-05-31T20:38:21.604 DEBUG:boto:Headers: {} 2023-05-31T20:38:21.604 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:38:21.604 DEBUG:boto:Port: 8000 2023-05-31T20:38:21.605 DEBUG:boto:Params: {} 2023-05-31T20:38:21.605 DEBUG:boto:Token: None 2023-05-31T20:38:21.606 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:38:21 GMT /pklxkd-58/ 2023-05-31T20:38:21.606 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:s31FU4vdTVg+6vxPE5KFsQQx3XQ= 2023-05-31T20:38:21.606 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:38:21 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:s31FU4vdTVg+6vxPE5KFsQQx3XQ=', 'Content-Length': '0'} 2023-05-31T20:38:21.611 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '54'), ('X-RGW-Bytes-Used', '162'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000d040414a93ff6522-006477b03d-1630-a1'), ('Content-Length', '0'), ('Date', 'Wed, 31 May 2023 20:38:21 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:38:21.612 DEBUG:boto:path=/pklxkd-58/obj00000000000000000027 2023-05-31T20:38:21.612 DEBUG:boto:auth_path=/pklxkd-58/obj00000000000000000027 2023-05-31T20:38:21.613 DEBUG:boto:Method: PUT 2023-05-31T20:38:21.613 DEBUG:boto:Path: /pklxkd-58/obj00000000000000000027 2023-05-31T20:38:21.614 DEBUG:boto:Data: 2023-05-31T20:38:21.614 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-05-31T20:38:21.615 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:38:21.615 DEBUG:boto:Port: 8000 2023-05-31T20:38:21.615 DEBUG:boto:Params: {} 2023-05-31T20:38:21.616 DEBUG:boto:Token: None 2023-05-31T20:38:21.616 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Wed, 31 May 2023 20:38:21 GMT /pklxkd-58/obj00000000000000000027 2023-05-31T20:38:21.617 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:QOkBMwW7jYtVsLkgaTHls7LZWhE= 2023-05-31T20:38:21.617 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Wed, 31 May 2023 20:38:21 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:QOkBMwW7jYtVsLkgaTHls7LZWhE='} 2023-05-31T20:38:21.628 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000005a090c49a7e8ce57-006477b03d-1630-a1'), ('Date', 'Wed, 31 May 2023 20:38:21 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:38:21.628 DEBUG:boto:path=/pklxkd-58/ 2023-05-31T20:38:21.629 DEBUG:boto:auth_path=/pklxkd-58/ 2023-05-31T20:38:21.629 DEBUG:boto:Method: HEAD 2023-05-31T20:38:21.630 DEBUG:boto:Path: /pklxkd-58/ 2023-05-31T20:38:21.630 DEBUG:boto:Data: 2023-05-31T20:38:21.631 DEBUG:boto:Headers: {} 2023-05-31T20:38:21.631 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:38:21.631 DEBUG:boto:Port: 8000 2023-05-31T20:38:21.632 DEBUG:boto:Params: {} 2023-05-31T20:38:21.632 DEBUG:boto:Token: None 2023-05-31T20:38:21.633 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:38:21 GMT /pklxkd-58/ 2023-05-31T20:38:21.633 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:s31FU4vdTVg+6vxPE5KFsQQx3XQ= 2023-05-31T20:38:21.634 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:38:21 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:s31FU4vdTVg+6vxPE5KFsQQx3XQ=', 'Content-Length': '0'} 2023-05-31T20:38:21.637 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '54'), ('X-RGW-Bytes-Used', '162'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000002efa92e94d9ca77b-006477b03d-1630-a1'), ('Content-Length', '0'), ('Date', 'Wed, 31 May 2023 20:38:21 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:38:21.638 DEBUG:boto:path=/pklxkd-58/obj00000000000000000030 2023-05-31T20:38:21.638 DEBUG:boto:auth_path=/pklxkd-58/obj00000000000000000030 2023-05-31T20:38:21.639 DEBUG:boto:Method: PUT 2023-05-31T20:38:21.639 DEBUG:boto:Path: /pklxkd-58/obj00000000000000000030 2023-05-31T20:38:21.640 DEBUG:boto:Data: 2023-05-31T20:38:21.640 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-05-31T20:38:21.641 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:38:21.641 DEBUG:boto:Port: 8000 2023-05-31T20:38:21.641 DEBUG:boto:Params: {} 2023-05-31T20:38:21.642 DEBUG:boto:Token: None 2023-05-31T20:38:21.642 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Wed, 31 May 2023 20:38:21 GMT /pklxkd-58/obj00000000000000000030 2023-05-31T20:38:21.643 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:HOpfDxUgywgS6WeqTTw+KF7uL84= 2023-05-31T20:38:21.643 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Wed, 31 May 2023 20:38:21 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:HOpfDxUgywgS6WeqTTw+KF7uL84='} 2023-05-31T20:38:21.693 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000d4ed284be82ff4dd-006477b03d-1630-a1'), ('Date', 'Wed, 31 May 2023 20:38:21 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:38:21.694 DEBUG:boto:path=/pklxkd-58/ 2023-05-31T20:38:21.694 DEBUG:boto:auth_path=/pklxkd-58/ 2023-05-31T20:38:21.695 DEBUG:boto:Method: HEAD 2023-05-31T20:38:21.696 DEBUG:boto:Path: /pklxkd-58/ 2023-05-31T20:38:21.696 DEBUG:boto:Data: 2023-05-31T20:38:21.697 DEBUG:boto:Headers: {} 2023-05-31T20:38:21.698 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:38:21.699 DEBUG:boto:Port: 8000 2023-05-31T20:38:21.699 DEBUG:boto:Params: {} 2023-05-31T20:38:21.700 DEBUG:boto:Token: None 2023-05-31T20:38:21.701 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:38:21 GMT /pklxkd-58/ 2023-05-31T20:38:21.702 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:s31FU4vdTVg+6vxPE5KFsQQx3XQ= 2023-05-31T20:38:21.702 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:38:21 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:s31FU4vdTVg+6vxPE5KFsQQx3XQ=', 'Content-Length': '0'} 2023-05-31T20:38:21.705 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '54'), ('X-RGW-Bytes-Used', '162'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000e8abeca9bbd0e64a-006477b03d-1630-a1'), ('Content-Length', '0'), ('Date', 'Wed, 31 May 2023 20:38:21 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:38:21.706 DEBUG:boto:path=/pklxkd-58/obj00000000000000000005 2023-05-31T20:38:21.707 DEBUG:boto:auth_path=/pklxkd-58/obj00000000000000000005 2023-05-31T20:38:21.707 DEBUG:boto:Method: PUT 2023-05-31T20:38:21.707 DEBUG:boto:Path: /pklxkd-58/obj00000000000000000005 2023-05-31T20:38:21.708 DEBUG:boto:Data: 2023-05-31T20:38:21.708 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-05-31T20:38:21.708 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:38:21.709 DEBUG:boto:Port: 8000 2023-05-31T20:38:21.709 DEBUG:boto:Params: {} 2023-05-31T20:38:21.709 DEBUG:boto:Token: None 2023-05-31T20:38:21.710 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Wed, 31 May 2023 20:38:21 GMT /pklxkd-58/obj00000000000000000005 2023-05-31T20:38:21.711 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:dBMjuZSiijoPYc4LHUJA5lrq7DA= 2023-05-31T20:38:21.711 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Wed, 31 May 2023 20:38:21 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:dBMjuZSiijoPYc4LHUJA5lrq7DA='} 2023-05-31T20:38:21.721 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000870bd58197af51ff-006477b03d-1630-a1'), ('Date', 'Wed, 31 May 2023 20:38:21 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:38:21.722 DEBUG:boto:path=/pklxkd-58/ 2023-05-31T20:38:21.722 DEBUG:boto:auth_path=/pklxkd-58/ 2023-05-31T20:38:21.723 DEBUG:boto:Method: HEAD 2023-05-31T20:38:21.723 DEBUG:boto:Path: /pklxkd-58/ 2023-05-31T20:38:21.723 DEBUG:boto:Data: 2023-05-31T20:38:21.724 DEBUG:boto:Headers: {} 2023-05-31T20:38:21.724 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:38:21.725 DEBUG:boto:Port: 8000 2023-05-31T20:38:21.725 DEBUG:boto:Params: {} 2023-05-31T20:38:21.725 DEBUG:boto:Token: None 2023-05-31T20:38:21.726 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:38:21 GMT /pklxkd-58/ 2023-05-31T20:38:21.726 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:s31FU4vdTVg+6vxPE5KFsQQx3XQ= 2023-05-31T20:38:21.727 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:38:21 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:s31FU4vdTVg+6vxPE5KFsQQx3XQ=', 'Content-Length': '0'} 2023-05-31T20:38:21.730 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '54'), ('X-RGW-Bytes-Used', '162'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000013470cecee2758ee-006477b03d-1630-a1'), ('Content-Length', '0'), ('Date', 'Wed, 31 May 2023 20:38:21 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:38:21.731 DEBUG:boto:path=/pklxkd-58/obj00000000000000000006 2023-05-31T20:38:21.731 DEBUG:boto:auth_path=/pklxkd-58/obj00000000000000000006 2023-05-31T20:38:21.732 DEBUG:boto:Method: PUT 2023-05-31T20:38:21.732 DEBUG:boto:Path: /pklxkd-58/obj00000000000000000006 2023-05-31T20:38:21.733 DEBUG:boto:Data: 2023-05-31T20:38:21.733 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-05-31T20:38:21.733 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:38:21.734 DEBUG:boto:Port: 8000 2023-05-31T20:38:21.734 DEBUG:boto:Params: {} 2023-05-31T20:38:21.735 DEBUG:boto:Token: None 2023-05-31T20:38:21.735 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Wed, 31 May 2023 20:38:21 GMT /pklxkd-58/obj00000000000000000006 2023-05-31T20:38:21.736 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:S8tWq7MWdV9fX6qWnDhvZ2vfCDo= 2023-05-31T20:38:21.737 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Wed, 31 May 2023 20:38:21 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:S8tWq7MWdV9fX6qWnDhvZ2vfCDo='} 2023-05-31T20:38:21.780 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000e8b2bc583a0f3a42-006477b03d-1630-a1'), ('Date', 'Wed, 31 May 2023 20:38:21 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:38:21.781 DEBUG:boto:path=/pklxkd-58/ 2023-05-31T20:38:21.782 DEBUG:boto:auth_path=/pklxkd-58/ 2023-05-31T20:38:21.782 DEBUG:boto:Method: HEAD 2023-05-31T20:38:21.782 DEBUG:boto:Path: /pklxkd-58/ 2023-05-31T20:38:21.783 DEBUG:boto:Data: 2023-05-31T20:38:21.783 DEBUG:boto:Headers: {} 2023-05-31T20:38:21.783 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:38:21.784 DEBUG:boto:Port: 8000 2023-05-31T20:38:21.784 DEBUG:boto:Params: {} 2023-05-31T20:38:21.785 DEBUG:boto:Token: None 2023-05-31T20:38:21.785 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:38:21 GMT /pklxkd-58/ 2023-05-31T20:38:21.785 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:s31FU4vdTVg+6vxPE5KFsQQx3XQ= 2023-05-31T20:38:21.786 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:38:21 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:s31FU4vdTVg+6vxPE5KFsQQx3XQ=', 'Content-Length': '0'} 2023-05-31T20:38:21.789 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '54'), ('X-RGW-Bytes-Used', '162'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000006da6249336bf8434-006477b03d-1630-a1'), ('Content-Length', '0'), ('Date', 'Wed, 31 May 2023 20:38:21 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:38:21.790 DEBUG:boto:path=/pklxkd-58/obj00000000000000000062 2023-05-31T20:38:21.790 DEBUG:boto:auth_path=/pklxkd-58/obj00000000000000000062 2023-05-31T20:38:21.790 DEBUG:boto:Method: PUT 2023-05-31T20:38:21.791 DEBUG:boto:Path: /pklxkd-58/obj00000000000000000062 2023-05-31T20:38:21.791 DEBUG:boto:Data: 2023-05-31T20:38:21.792 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-05-31T20:38:21.792 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:38:21.792 DEBUG:boto:Port: 8000 2023-05-31T20:38:21.793 DEBUG:boto:Params: {} 2023-05-31T20:38:21.793 DEBUG:boto:Token: None 2023-05-31T20:38:21.794 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Wed, 31 May 2023 20:38:21 GMT /pklxkd-58/obj00000000000000000062 2023-05-31T20:38:21.795 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:BwE1k8ayjT2IleRZfXGcc4CxTfg= 2023-05-31T20:38:21.795 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Wed, 31 May 2023 20:38:21 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:BwE1k8ayjT2IleRZfXGcc4CxTfg='} 2023-05-31T20:38:21.802 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000008d0a372b7f319149-006477b03d-1630-a1'), ('Date', 'Wed, 31 May 2023 20:38:21 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:38:21.803 DEBUG:boto:path=/pklxkd-58/ 2023-05-31T20:38:21.803 DEBUG:boto:auth_path=/pklxkd-58/ 2023-05-31T20:38:21.804 DEBUG:boto:Method: HEAD 2023-05-31T20:38:21.804 DEBUG:boto:Path: /pklxkd-58/ 2023-05-31T20:38:21.805 DEBUG:boto:Data: 2023-05-31T20:38:21.805 DEBUG:boto:Headers: {} 2023-05-31T20:38:21.805 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:38:21.806 DEBUG:boto:Port: 8000 2023-05-31T20:38:21.806 DEBUG:boto:Params: {} 2023-05-31T20:38:21.807 DEBUG:boto:Token: None 2023-05-31T20:38:21.807 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:38:21 GMT /pklxkd-58/ 2023-05-31T20:38:21.807 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:s31FU4vdTVg+6vxPE5KFsQQx3XQ= 2023-05-31T20:38:21.808 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:38:21 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:s31FU4vdTVg+6vxPE5KFsQQx3XQ=', 'Content-Length': '0'} 2023-05-31T20:38:21.815 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '55'), ('X-RGW-Bytes-Used', '165'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000008b85ff07e63f2af8-006477b03d-1630-a1'), ('Content-Length', '0'), ('Date', 'Wed, 31 May 2023 20:38:21 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:38:21.816 DEBUG:boto:path=/pklxkd-58/obj00000000000000000057 2023-05-31T20:38:21.816 DEBUG:boto:auth_path=/pklxkd-58/obj00000000000000000057 2023-05-31T20:38:21.817 DEBUG:boto:Method: PUT 2023-05-31T20:38:21.817 DEBUG:boto:Path: /pklxkd-58/obj00000000000000000057 2023-05-31T20:38:21.817 DEBUG:boto:Data: 2023-05-31T20:38:21.818 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-05-31T20:38:21.818 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:38:21.818 DEBUG:boto:Port: 8000 2023-05-31T20:38:21.819 DEBUG:boto:Params: {} 2023-05-31T20:38:21.819 DEBUG:boto:Token: None 2023-05-31T20:38:21.819 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Wed, 31 May 2023 20:38:21 GMT /pklxkd-58/obj00000000000000000057 2023-05-31T20:38:21.820 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:TGYLshEQcfSHpo3BHawuZuFxd3U= 2023-05-31T20:38:21.820 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Wed, 31 May 2023 20:38:21 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:TGYLshEQcfSHpo3BHawuZuFxd3U='} 2023-05-31T20:38:21.865 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000c0ed1f731ba6469f-006477b03d-1630-a1'), ('Date', 'Wed, 31 May 2023 20:38:21 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:38:21.866 DEBUG:boto:path=/pklxkd-58/ 2023-05-31T20:38:21.867 DEBUG:boto:auth_path=/pklxkd-58/ 2023-05-31T20:38:21.868 DEBUG:boto:Method: HEAD 2023-05-31T20:38:21.869 DEBUG:boto:Path: /pklxkd-58/ 2023-05-31T20:38:21.869 DEBUG:boto:Data: 2023-05-31T20:38:21.870 DEBUG:boto:Headers: {} 2023-05-31T20:38:21.871 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:38:21.871 DEBUG:boto:Port: 8000 2023-05-31T20:38:21.872 DEBUG:boto:Params: {} 2023-05-31T20:38:21.873 DEBUG:boto:Token: None 2023-05-31T20:38:21.874 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:38:21 GMT /pklxkd-58/ 2023-05-31T20:38:21.874 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:s31FU4vdTVg+6vxPE5KFsQQx3XQ= 2023-05-31T20:38:21.875 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:38:21 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:s31FU4vdTVg+6vxPE5KFsQQx3XQ=', 'Content-Length': '0'} 2023-05-31T20:38:21.879 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '56'), ('X-RGW-Bytes-Used', '168'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000d6c6845f2b228ee6-006477b03d-1630-a1'), ('Content-Length', '0'), ('Date', 'Wed, 31 May 2023 20:38:21 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:38:21.880 DEBUG:boto:path=/pklxkd-58/obj00000000000000000018 2023-05-31T20:38:21.881 DEBUG:boto:auth_path=/pklxkd-58/obj00000000000000000018 2023-05-31T20:38:21.882 DEBUG:boto:Method: PUT 2023-05-31T20:38:21.882 DEBUG:boto:Path: /pklxkd-58/obj00000000000000000018 2023-05-31T20:38:21.883 DEBUG:boto:Data: 2023-05-31T20:38:21.883 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-05-31T20:38:21.883 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:38:21.884 DEBUG:boto:Port: 8000 2023-05-31T20:38:21.885 DEBUG:boto:Params: {} 2023-05-31T20:38:21.886 DEBUG:boto:Token: None 2023-05-31T20:38:21.887 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Wed, 31 May 2023 20:38:21 GMT /pklxkd-58/obj00000000000000000018 2023-05-31T20:38:21.888 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:/Zkq/5VKdYMQnBFFtcr0d/IBz/Q= 2023-05-31T20:38:21.889 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Wed, 31 May 2023 20:38:21 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:/Zkq/5VKdYMQnBFFtcr0d/IBz/Q='} 2023-05-31T20:38:21.897 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000896a560022c1347e-006477b03d-1630-a1'), ('Date', 'Wed, 31 May 2023 20:38:21 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:38:21.897 DEBUG:boto:path=/pklxkd-58/ 2023-05-31T20:38:21.898 DEBUG:boto:auth_path=/pklxkd-58/ 2023-05-31T20:38:21.899 DEBUG:boto:Method: HEAD 2023-05-31T20:38:21.899 DEBUG:boto:Path: /pklxkd-58/ 2023-05-31T20:38:21.900 DEBUG:boto:Data: 2023-05-31T20:38:21.900 DEBUG:boto:Headers: {} 2023-05-31T20:38:21.900 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:38:21.901 DEBUG:boto:Port: 8000 2023-05-31T20:38:21.901 DEBUG:boto:Params: {} 2023-05-31T20:38:21.902 DEBUG:boto:Token: None 2023-05-31T20:38:21.902 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:38:21 GMT /pklxkd-58/ 2023-05-31T20:38:21.902 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:s31FU4vdTVg+6vxPE5KFsQQx3XQ= 2023-05-31T20:38:21.903 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:38:21 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:s31FU4vdTVg+6vxPE5KFsQQx3XQ=', 'Content-Length': '0'} 2023-05-31T20:38:21.906 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '56'), ('X-RGW-Bytes-Used', '168'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000ebbf2278f3e7a5ed-006477b03d-1630-a1'), ('Content-Length', '0'), ('Date', 'Wed, 31 May 2023 20:38:21 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:38:21.907 DEBUG:boto:path=/pklxkd-58/obj00000000000000000003 2023-05-31T20:38:21.907 DEBUG:boto:auth_path=/pklxkd-58/obj00000000000000000003 2023-05-31T20:38:21.908 DEBUG:boto:Method: PUT 2023-05-31T20:38:21.908 DEBUG:boto:Path: /pklxkd-58/obj00000000000000000003 2023-05-31T20:38:21.908 DEBUG:boto:Data: 2023-05-31T20:38:21.909 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-05-31T20:38:21.909 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:38:21.910 DEBUG:boto:Port: 8000 2023-05-31T20:38:21.910 DEBUG:boto:Params: {} 2023-05-31T20:38:21.911 DEBUG:boto:Token: None 2023-05-31T20:38:21.911 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Wed, 31 May 2023 20:38:21 GMT /pklxkd-58/obj00000000000000000003 2023-05-31T20:38:21.911 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:LOuxS36gboQkBGY+5pMFvNR2dKw= 2023-05-31T20:38:21.912 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Wed, 31 May 2023 20:38:21 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:LOuxS36gboQkBGY+5pMFvNR2dKw='} 2023-05-31T20:38:21.956 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000087613ef36555234b-006477b03d-1630-a1'), ('Date', 'Wed, 31 May 2023 20:38:21 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:38:21.957 DEBUG:boto:path=/pklxkd-58/ 2023-05-31T20:38:21.958 DEBUG:boto:auth_path=/pklxkd-58/ 2023-05-31T20:38:21.959 DEBUG:boto:Method: HEAD 2023-05-31T20:38:21.959 DEBUG:boto:Path: /pklxkd-58/ 2023-05-31T20:38:21.959 DEBUG:boto:Data: 2023-05-31T20:38:21.960 DEBUG:boto:Headers: {} 2023-05-31T20:38:21.960 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:38:21.961 DEBUG:boto:Port: 8000 2023-05-31T20:38:21.961 DEBUG:boto:Params: {} 2023-05-31T20:38:21.961 DEBUG:boto:Token: None 2023-05-31T20:38:21.962 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:38:21 GMT /pklxkd-58/ 2023-05-31T20:38:21.962 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:s31FU4vdTVg+6vxPE5KFsQQx3XQ= 2023-05-31T20:38:21.963 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:38:21 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:s31FU4vdTVg+6vxPE5KFsQQx3XQ=', 'Content-Length': '0'} 2023-05-31T20:38:21.966 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '56'), ('X-RGW-Bytes-Used', '168'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000d115d0ac9a19fe51-006477b03d-1630-a1'), ('Content-Length', '0'), ('Date', 'Wed, 31 May 2023 20:38:21 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:38:21.967 DEBUG:boto:path=/pklxkd-58/obj00000000000000000000 2023-05-31T20:38:21.967 DEBUG:boto:auth_path=/pklxkd-58/obj00000000000000000000 2023-05-31T20:38:21.968 DEBUG:boto:Method: PUT 2023-05-31T20:38:21.968 DEBUG:boto:Path: /pklxkd-58/obj00000000000000000000 2023-05-31T20:38:21.969 DEBUG:boto:Data: 2023-05-31T20:38:21.969 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-05-31T20:38:21.969 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:38:21.970 DEBUG:boto:Port: 8000 2023-05-31T20:38:21.971 DEBUG:boto:Params: {} 2023-05-31T20:38:21.972 DEBUG:boto:Token: None 2023-05-31T20:38:21.972 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Wed, 31 May 2023 20:38:21 GMT /pklxkd-58/obj00000000000000000000 2023-05-31T20:38:21.973 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:OzbskrbdQ4QjKByXmz9mz+BuDmA= 2023-05-31T20:38:21.974 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Wed, 31 May 2023 20:38:21 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:OzbskrbdQ4QjKByXmz9mz+BuDmA='} 2023-05-31T20:38:21.982 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000c80d2a2e90675aa1-006477b03d-1630-a1'), ('Date', 'Wed, 31 May 2023 20:38:21 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:38:21.983 DEBUG:boto:path=/pklxkd-58/ 2023-05-31T20:38:21.984 DEBUG:boto:auth_path=/pklxkd-58/ 2023-05-31T20:38:21.985 DEBUG:boto:Method: HEAD 2023-05-31T20:38:21.986 DEBUG:boto:Path: /pklxkd-58/ 2023-05-31T20:38:21.987 DEBUG:boto:Data: 2023-05-31T20:38:21.987 DEBUG:boto:Headers: {} 2023-05-31T20:38:21.987 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:38:21.988 DEBUG:boto:Port: 8000 2023-05-31T20:38:21.988 DEBUG:boto:Params: {} 2023-05-31T20:38:21.989 DEBUG:boto:Token: None 2023-05-31T20:38:21.989 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:38:21 GMT /pklxkd-58/ 2023-05-31T20:38:21.990 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:s31FU4vdTVg+6vxPE5KFsQQx3XQ= 2023-05-31T20:38:21.990 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:38:21 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:s31FU4vdTVg+6vxPE5KFsQQx3XQ=', 'Content-Length': '0'} 2023-05-31T20:38:21.993 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '56'), ('X-RGW-Bytes-Used', '168'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000063e58f8958d0474f-006477b03d-1630-a1'), ('Content-Length', '0'), ('Date', 'Wed, 31 May 2023 20:38:21 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:38:21.994 DEBUG:boto:path=/pklxkd-58/obj00000000000000000065 2023-05-31T20:38:21.994 DEBUG:boto:auth_path=/pklxkd-58/obj00000000000000000065 2023-05-31T20:38:21.995 DEBUG:boto:Method: PUT 2023-05-31T20:38:21.995 DEBUG:boto:Path: /pklxkd-58/obj00000000000000000065 2023-05-31T20:38:21.996 DEBUG:boto:Data: 2023-05-31T20:38:21.997 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-05-31T20:38:21.998 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:38:21.999 DEBUG:boto:Port: 8000 2023-05-31T20:38:22.000 DEBUG:boto:Params: {} 2023-05-31T20:38:22.001 DEBUG:boto:Token: None 2023-05-31T20:38:22.002 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Wed, 31 May 2023 20:38:22 GMT /pklxkd-58/obj00000000000000000065 2023-05-31T20:38:22.003 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:erQv5BXzXcs3387NjtrsvzlVF6I= 2023-05-31T20:38:22.005 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Wed, 31 May 2023 20:38:22 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:erQv5BXzXcs3387NjtrsvzlVF6I='} 2023-05-31T20:38:22.048 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000084aa89b29dc478b8-006477b03e-1630-a1'), ('Date', 'Wed, 31 May 2023 20:38:22 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:38:22.049 DEBUG:boto:path=/pklxkd-58/ 2023-05-31T20:38:22.050 DEBUG:boto:auth_path=/pklxkd-58/ 2023-05-31T20:38:22.050 DEBUG:boto:Method: HEAD 2023-05-31T20:38:22.050 DEBUG:boto:Path: /pklxkd-58/ 2023-05-31T20:38:22.051 DEBUG:boto:Data: 2023-05-31T20:38:22.051 DEBUG:boto:Headers: {} 2023-05-31T20:38:22.051 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:38:22.052 DEBUG:boto:Port: 8000 2023-05-31T20:38:22.052 DEBUG:boto:Params: {} 2023-05-31T20:38:22.052 DEBUG:boto:Token: None 2023-05-31T20:38:22.053 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:38:22 GMT /pklxkd-58/ 2023-05-31T20:38:22.053 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:yibMkgDYmA8uZlGmaKDLCUn3DTQ= 2023-05-31T20:38:22.054 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:38:22 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:yibMkgDYmA8uZlGmaKDLCUn3DTQ=', 'Content-Length': '0'} 2023-05-31T20:38:22.057 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '57'), ('X-RGW-Bytes-Used', '171'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000f832088758d79e63-006477b03e-1630-a1'), ('Content-Length', '0'), ('Date', 'Wed, 31 May 2023 20:38:22 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:38:22.058 DEBUG:boto:path=/pklxkd-58/obj00000000000000000002 2023-05-31T20:38:22.058 DEBUG:boto:auth_path=/pklxkd-58/obj00000000000000000002 2023-05-31T20:38:22.059 DEBUG:boto:Method: PUT 2023-05-31T20:38:22.059 DEBUG:boto:Path: /pklxkd-58/obj00000000000000000002 2023-05-31T20:38:22.060 DEBUG:boto:Data: 2023-05-31T20:38:22.060 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-05-31T20:38:22.061 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:38:22.061 DEBUG:boto:Port: 8000 2023-05-31T20:38:22.061 DEBUG:boto:Params: {} 2023-05-31T20:38:22.062 DEBUG:boto:Token: None 2023-05-31T20:38:22.062 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Wed, 31 May 2023 20:38:22 GMT /pklxkd-58/obj00000000000000000002 2023-05-31T20:38:22.063 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:wr0jGHZWvJeGfaaQ6JD6nVve3Yg= 2023-05-31T20:38:22.063 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Wed, 31 May 2023 20:38:22 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:wr0jGHZWvJeGfaaQ6JD6nVve3Yg='} 2023-05-31T20:38:22.071 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000005afec7c62de1b6d0-006477b03e-1630-a1'), ('Date', 'Wed, 31 May 2023 20:38:22 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:38:22.072 DEBUG:boto:path=/pklxkd-58/ 2023-05-31T20:38:22.072 DEBUG:boto:auth_path=/pklxkd-58/ 2023-05-31T20:38:22.073 DEBUG:boto:Method: HEAD 2023-05-31T20:38:22.073 DEBUG:boto:Path: /pklxkd-58/ 2023-05-31T20:38:22.073 DEBUG:boto:Data: 2023-05-31T20:38:22.074 DEBUG:boto:Headers: {} 2023-05-31T20:38:22.074 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:38:22.074 DEBUG:boto:Port: 8000 2023-05-31T20:38:22.075 DEBUG:boto:Params: {} 2023-05-31T20:38:22.075 DEBUG:boto:Token: None 2023-05-31T20:38:22.076 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:38:22 GMT /pklxkd-58/ 2023-05-31T20:38:22.076 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:yibMkgDYmA8uZlGmaKDLCUn3DTQ= 2023-05-31T20:38:22.077 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:38:22 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:yibMkgDYmA8uZlGmaKDLCUn3DTQ=', 'Content-Length': '0'} 2023-05-31T20:38:22.080 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '57'), ('X-RGW-Bytes-Used', '171'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000c81af98b2581a0ec-006477b03e-1630-a1'), ('Content-Length', '0'), ('Date', 'Wed, 31 May 2023 20:38:22 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:38:22.081 DEBUG:boto:path=/pklxkd-58/obj00000000000000000033 2023-05-31T20:38:22.081 DEBUG:boto:auth_path=/pklxkd-58/obj00000000000000000033 2023-05-31T20:38:22.082 DEBUG:boto:Method: PUT 2023-05-31T20:38:22.082 DEBUG:boto:Path: /pklxkd-58/obj00000000000000000033 2023-05-31T20:38:22.083 DEBUG:boto:Data: 2023-05-31T20:38:22.083 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-05-31T20:38:22.083 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:38:22.084 DEBUG:boto:Port: 8000 2023-05-31T20:38:22.084 DEBUG:boto:Params: {} 2023-05-31T20:38:22.084 DEBUG:boto:Token: None 2023-05-31T20:38:22.085 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Wed, 31 May 2023 20:38:22 GMT /pklxkd-58/obj00000000000000000033 2023-05-31T20:38:22.085 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:GGhFGXJM2iMw6TnRu/U9vTy+Ypk= 2023-05-31T20:38:22.086 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Wed, 31 May 2023 20:38:22 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:GGhFGXJM2iMw6TnRu/U9vTy+Ypk='} 2023-05-31T20:38:22.128 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000008c1b2ec1e8cb2555-006477b03e-1630-a1'), ('Date', 'Wed, 31 May 2023 20:38:22 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:38:22.129 DEBUG:boto:path=/pklxkd-58/ 2023-05-31T20:38:22.129 DEBUG:boto:auth_path=/pklxkd-58/ 2023-05-31T20:38:22.130 DEBUG:boto:Method: HEAD 2023-05-31T20:38:22.130 DEBUG:boto:Path: /pklxkd-58/ 2023-05-31T20:38:22.131 DEBUG:boto:Data: 2023-05-31T20:38:22.131 DEBUG:boto:Headers: {} 2023-05-31T20:38:22.131 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:38:22.132 DEBUG:boto:Port: 8000 2023-05-31T20:38:22.132 DEBUG:boto:Params: {} 2023-05-31T20:38:22.133 DEBUG:boto:Token: None 2023-05-31T20:38:22.133 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:38:22 GMT /pklxkd-58/ 2023-05-31T20:38:22.133 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:yibMkgDYmA8uZlGmaKDLCUn3DTQ= 2023-05-31T20:38:22.134 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:38:22 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:yibMkgDYmA8uZlGmaKDLCUn3DTQ=', 'Content-Length': '0'} 2023-05-31T20:38:22.137 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '57'), ('X-RGW-Bytes-Used', '171'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000000701eec803c37f6-006477b03e-1630-a1'), ('Content-Length', '0'), ('Date', 'Wed, 31 May 2023 20:38:22 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:38:22.138 DEBUG:boto:path=/pklxkd-58/obj00000000000000000016 2023-05-31T20:38:22.138 DEBUG:boto:auth_path=/pklxkd-58/obj00000000000000000016 2023-05-31T20:38:22.139 DEBUG:boto:Method: PUT 2023-05-31T20:38:22.139 DEBUG:boto:Path: /pklxkd-58/obj00000000000000000016 2023-05-31T20:38:22.139 DEBUG:boto:Data: 2023-05-31T20:38:22.140 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-05-31T20:38:22.140 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:38:22.140 DEBUG:boto:Port: 8000 2023-05-31T20:38:22.141 DEBUG:boto:Params: {} 2023-05-31T20:38:22.141 DEBUG:boto:Token: None 2023-05-31T20:38:22.141 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Wed, 31 May 2023 20:38:22 GMT /pklxkd-58/obj00000000000000000016 2023-05-31T20:38:22.142 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:n3r3yS/yaHqDj0dmVMfGrgj3Cco= 2023-05-31T20:38:22.142 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Wed, 31 May 2023 20:38:22 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:n3r3yS/yaHqDj0dmVMfGrgj3Cco='} 2023-05-31T20:38:22.156 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000e1e0be6ecc6d5144-006477b03e-1630-a1'), ('Date', 'Wed, 31 May 2023 20:38:22 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:38:22.157 DEBUG:boto:path=/pklxkd-58/ 2023-05-31T20:38:22.158 DEBUG:boto:auth_path=/pklxkd-58/ 2023-05-31T20:38:22.158 DEBUG:boto:Method: HEAD 2023-05-31T20:38:22.158 DEBUG:boto:Path: /pklxkd-58/ 2023-05-31T20:38:22.159 DEBUG:boto:Data: 2023-05-31T20:38:22.159 DEBUG:boto:Headers: {} 2023-05-31T20:38:22.159 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:38:22.160 DEBUG:boto:Port: 8000 2023-05-31T20:38:22.160 DEBUG:boto:Params: {} 2023-05-31T20:38:22.160 DEBUG:boto:Token: None 2023-05-31T20:38:22.161 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:38:22 GMT /pklxkd-58/ 2023-05-31T20:38:22.161 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:yibMkgDYmA8uZlGmaKDLCUn3DTQ= 2023-05-31T20:38:22.162 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:38:22 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:yibMkgDYmA8uZlGmaKDLCUn3DTQ=', 'Content-Length': '0'} 2023-05-31T20:38:22.165 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '57'), ('X-RGW-Bytes-Used', '171'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000006b06a2b2552e151f-006477b03e-1630-a1'), ('Content-Length', '0'), ('Date', 'Wed, 31 May 2023 20:38:22 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:38:22.166 DEBUG:boto:path=/pklxkd-58/obj00000000000000000001 2023-05-31T20:38:22.166 DEBUG:boto:auth_path=/pklxkd-58/obj00000000000000000001 2023-05-31T20:38:22.167 DEBUG:boto:Method: PUT 2023-05-31T20:38:22.167 DEBUG:boto:Path: /pklxkd-58/obj00000000000000000001 2023-05-31T20:38:22.168 DEBUG:boto:Data: 2023-05-31T20:38:22.168 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-05-31T20:38:22.168 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:38:22.169 DEBUG:boto:Port: 8000 2023-05-31T20:38:22.169 DEBUG:boto:Params: {} 2023-05-31T20:38:22.169 DEBUG:boto:Token: None 2023-05-31T20:38:22.170 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Wed, 31 May 2023 20:38:22 GMT /pklxkd-58/obj00000000000000000001 2023-05-31T20:38:22.170 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:wUiM61akZz8M3FDh/6LLK2jWRYQ= 2023-05-31T20:38:22.171 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Wed, 31 May 2023 20:38:22 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:wUiM61akZz8M3FDh/6LLK2jWRYQ='} 2023-05-31T20:38:22.213 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000a9fd2c19b0c2876c-006477b03e-1630-a1'), ('Date', 'Wed, 31 May 2023 20:38:22 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:38:22.214 DEBUG:boto:path=/pklxkd-58/ 2023-05-31T20:38:22.214 DEBUG:boto:auth_path=/pklxkd-58/ 2023-05-31T20:38:22.214 DEBUG:boto:Method: HEAD 2023-05-31T20:38:22.215 DEBUG:boto:Path: /pklxkd-58/ 2023-05-31T20:38:22.215 DEBUG:boto:Data: 2023-05-31T20:38:22.215 DEBUG:boto:Headers: {} 2023-05-31T20:38:22.216 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:38:22.216 DEBUG:boto:Port: 8000 2023-05-31T20:38:22.217 DEBUG:boto:Params: {} 2023-05-31T20:38:22.217 DEBUG:boto:Token: None 2023-05-31T20:38:22.218 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:38:22 GMT /pklxkd-58/ 2023-05-31T20:38:22.219 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:yibMkgDYmA8uZlGmaKDLCUn3DTQ= 2023-05-31T20:38:22.219 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:38:22 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:yibMkgDYmA8uZlGmaKDLCUn3DTQ=', 'Content-Length': '0'} 2023-05-31T20:38:22.223 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '57'), ('X-RGW-Bytes-Used', '171'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000005b9b0d11d28361c3-006477b03e-1630-a1'), ('Content-Length', '0'), ('Date', 'Wed, 31 May 2023 20:38:22 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:38:22.224 DEBUG:boto:path=/pklxkd-58/obj00000000000000000004 2023-05-31T20:38:22.224 DEBUG:boto:auth_path=/pklxkd-58/obj00000000000000000004 2023-05-31T20:38:22.224 DEBUG:boto:Method: PUT 2023-05-31T20:38:22.225 DEBUG:boto:Path: /pklxkd-58/obj00000000000000000004 2023-05-31T20:38:22.225 DEBUG:boto:Data: 2023-05-31T20:38:22.226 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-05-31T20:38:22.226 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:38:22.227 DEBUG:boto:Port: 8000 2023-05-31T20:38:22.227 DEBUG:boto:Params: {} 2023-05-31T20:38:22.228 DEBUG:boto:Token: None 2023-05-31T20:38:22.228 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Wed, 31 May 2023 20:38:22 GMT /pklxkd-58/obj00000000000000000004 2023-05-31T20:38:22.229 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:qN73qJCajAb0W8ReadTRKZaE3oE= 2023-05-31T20:38:22.229 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Wed, 31 May 2023 20:38:22 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:qN73qJCajAb0W8ReadTRKZaE3oE='} 2023-05-31T20:38:22.236 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000054013fbcebfb8065-006477b03e-1630-a1'), ('Date', 'Wed, 31 May 2023 20:38:22 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:38:22.237 DEBUG:boto:path=/pklxkd-58/ 2023-05-31T20:38:22.238 DEBUG:boto:auth_path=/pklxkd-58/ 2023-05-31T20:38:22.238 DEBUG:boto:Method: HEAD 2023-05-31T20:38:22.238 DEBUG:boto:Path: /pklxkd-58/ 2023-05-31T20:38:22.240 DEBUG:boto:Data: 2023-05-31T20:38:22.240 DEBUG:boto:Headers: {} 2023-05-31T20:38:22.241 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:38:22.241 DEBUG:boto:Port: 8000 2023-05-31T20:38:22.241 DEBUG:boto:Params: {} 2023-05-31T20:38:22.242 DEBUG:boto:Token: None 2023-05-31T20:38:22.242 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:38:22 GMT /pklxkd-58/ 2023-05-31T20:38:22.242 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:yibMkgDYmA8uZlGmaKDLCUn3DTQ= 2023-05-31T20:38:22.243 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:38:22 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:yibMkgDYmA8uZlGmaKDLCUn3DTQ=', 'Content-Length': '0'} 2023-05-31T20:38:22.246 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '57'), ('X-RGW-Bytes-Used', '171'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000b752c1e65727c95c-006477b03e-1630-a1'), ('Content-Length', '0'), ('Date', 'Wed, 31 May 2023 20:38:22 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:38:22.247 DEBUG:boto:path=/pklxkd-58/obj00000000000000000045 2023-05-31T20:38:22.247 DEBUG:boto:auth_path=/pklxkd-58/obj00000000000000000045 2023-05-31T20:38:22.248 DEBUG:boto:Method: PUT 2023-05-31T20:38:22.248 DEBUG:boto:Path: /pklxkd-58/obj00000000000000000045 2023-05-31T20:38:22.248 DEBUG:boto:Data: 2023-05-31T20:38:22.249 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-05-31T20:38:22.249 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:38:22.250 DEBUG:boto:Port: 8000 2023-05-31T20:38:22.250 DEBUG:boto:Params: {} 2023-05-31T20:38:22.251 DEBUG:boto:Token: None 2023-05-31T20:38:22.251 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Wed, 31 May 2023 20:38:22 GMT /pklxkd-58/obj00000000000000000045 2023-05-31T20:38:22.252 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:eVRu9MbekbNChKiHOkbkPmL9hYM= 2023-05-31T20:38:22.252 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Wed, 31 May 2023 20:38:22 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:eVRu9MbekbNChKiHOkbkPmL9hYM='} 2023-05-31T20:38:22.296 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000d6f1672da997ca43-006477b03e-1630-a1'), ('Date', 'Wed, 31 May 2023 20:38:22 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:38:22.297 DEBUG:boto:path=/pklxkd-58/ 2023-05-31T20:38:22.297 DEBUG:boto:auth_path=/pklxkd-58/ 2023-05-31T20:38:22.298 DEBUG:boto:Method: HEAD 2023-05-31T20:38:22.298 DEBUG:boto:Path: /pklxkd-58/ 2023-05-31T20:38:22.299 DEBUG:boto:Data: 2023-05-31T20:38:22.299 DEBUG:boto:Headers: {} 2023-05-31T20:38:22.299 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:38:22.300 DEBUG:boto:Port: 8000 2023-05-31T20:38:22.300 DEBUG:boto:Params: {} 2023-05-31T20:38:22.300 DEBUG:boto:Token: None 2023-05-31T20:38:22.301 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:38:22 GMT /pklxkd-58/ 2023-05-31T20:38:22.301 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:yibMkgDYmA8uZlGmaKDLCUn3DTQ= 2023-05-31T20:38:22.302 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:38:22 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:yibMkgDYmA8uZlGmaKDLCUn3DTQ=', 'Content-Length': '0'} 2023-05-31T20:38:22.305 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '57'), ('X-RGW-Bytes-Used', '171'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000dd637597e3f03983-006477b03e-1630-a1'), ('Content-Length', '0'), ('Date', 'Wed, 31 May 2023 20:38:22 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:38:22.305 DEBUG:boto:path=/pklxkd-58/obj00000000000000000012 2023-05-31T20:38:22.306 DEBUG:boto:auth_path=/pklxkd-58/obj00000000000000000012 2023-05-31T20:38:22.306 DEBUG:boto:Method: PUT 2023-05-31T20:38:22.307 DEBUG:boto:Path: /pklxkd-58/obj00000000000000000012 2023-05-31T20:38:22.307 DEBUG:boto:Data: 2023-05-31T20:38:22.307 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-05-31T20:38:22.308 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:38:22.308 DEBUG:boto:Port: 8000 2023-05-31T20:38:22.308 DEBUG:boto:Params: {} 2023-05-31T20:38:22.309 DEBUG:boto:Token: None 2023-05-31T20:38:22.309 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Wed, 31 May 2023 20:38:22 GMT /pklxkd-58/obj00000000000000000012 2023-05-31T20:38:22.310 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:TgBc6NUDFsqAzBpTeE/EO6yAW/0= 2023-05-31T20:38:22.310 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Wed, 31 May 2023 20:38:22 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:TgBc6NUDFsqAzBpTeE/EO6yAW/0='} 2023-05-31T20:38:22.316 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000093a690e3ea4cf0e6-006477b03e-1630-a1'), ('Date', 'Wed, 31 May 2023 20:38:22 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:38:22.317 DEBUG:boto:path=/pklxkd-58/ 2023-05-31T20:38:22.317 DEBUG:boto:auth_path=/pklxkd-58/ 2023-05-31T20:38:22.317 DEBUG:boto:Method: HEAD 2023-05-31T20:38:22.318 DEBUG:boto:Path: /pklxkd-58/ 2023-05-31T20:38:22.318 DEBUG:boto:Data: 2023-05-31T20:38:22.318 DEBUG:boto:Headers: {} 2023-05-31T20:38:22.319 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:38:22.319 DEBUG:boto:Port: 8000 2023-05-31T20:38:22.319 DEBUG:boto:Params: {} 2023-05-31T20:38:22.320 DEBUG:boto:Token: None 2023-05-31T20:38:22.320 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:38:22 GMT /pklxkd-58/ 2023-05-31T20:38:22.320 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:yibMkgDYmA8uZlGmaKDLCUn3DTQ= 2023-05-31T20:38:22.321 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:38:22 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:yibMkgDYmA8uZlGmaKDLCUn3DTQ=', 'Content-Length': '0'} 2023-05-31T20:38:22.324 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '57'), ('X-RGW-Bytes-Used', '171'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000007bc0095e8a4fd7fe-006477b03e-1630-a1'), ('Content-Length', '0'), ('Date', 'Wed, 31 May 2023 20:38:22 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:38:22.325 DEBUG:boto:path=/pklxkd-58/obj00000000000000000040 2023-05-31T20:38:22.325 DEBUG:boto:auth_path=/pklxkd-58/obj00000000000000000040 2023-05-31T20:38:22.325 DEBUG:boto:Method: PUT 2023-05-31T20:38:22.326 DEBUG:boto:Path: /pklxkd-58/obj00000000000000000040 2023-05-31T20:38:22.326 DEBUG:boto:Data: 2023-05-31T20:38:22.326 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-05-31T20:38:22.327 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:38:22.327 DEBUG:boto:Port: 8000 2023-05-31T20:38:22.327 DEBUG:boto:Params: {} 2023-05-31T20:38:22.328 DEBUG:boto:Token: None 2023-05-31T20:38:22.328 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Wed, 31 May 2023 20:38:22 GMT /pklxkd-58/obj00000000000000000040 2023-05-31T20:38:22.329 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:9kKk+ElaM6Iivr1634PXsQGwh/s= 2023-05-31T20:38:22.329 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Wed, 31 May 2023 20:38:22 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:9kKk+ElaM6Iivr1634PXsQGwh/s='} 2023-05-31T20:38:22.372 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000bfd31342f5e1ac51-006477b03e-1630-a1'), ('Date', 'Wed, 31 May 2023 20:38:22 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:38:22.373 DEBUG:boto:path=/pklxkd-58/ 2023-05-31T20:38:22.373 DEBUG:boto:auth_path=/pklxkd-58/ 2023-05-31T20:38:22.374 DEBUG:boto:Method: HEAD 2023-05-31T20:38:22.374 DEBUG:boto:Path: /pklxkd-58/ 2023-05-31T20:38:22.374 DEBUG:boto:Data: 2023-05-31T20:38:22.375 DEBUG:boto:Headers: {} 2023-05-31T20:38:22.375 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:38:22.375 DEBUG:boto:Port: 8000 2023-05-31T20:38:22.376 DEBUG:boto:Params: {} 2023-05-31T20:38:22.376 DEBUG:boto:Token: None 2023-05-31T20:38:22.377 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:38:22 GMT /pklxkd-58/ 2023-05-31T20:38:22.377 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:yibMkgDYmA8uZlGmaKDLCUn3DTQ= 2023-05-31T20:38:22.377 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:38:22 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:yibMkgDYmA8uZlGmaKDLCUn3DTQ=', 'Content-Length': '0'} 2023-05-31T20:38:22.380 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '57'), ('X-RGW-Bytes-Used', '171'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000025931efa25618c1a-006477b03e-1630-a1'), ('Content-Length', '0'), ('Date', 'Wed, 31 May 2023 20:38:22 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:38:22.381 DEBUG:boto:path=/pklxkd-58/obj00000000000000000039 2023-05-31T20:38:22.381 DEBUG:boto:auth_path=/pklxkd-58/obj00000000000000000039 2023-05-31T20:38:22.382 DEBUG:boto:Method: PUT 2023-05-31T20:38:22.382 DEBUG:boto:Path: /pklxkd-58/obj00000000000000000039 2023-05-31T20:38:22.383 DEBUG:boto:Data: 2023-05-31T20:38:22.383 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-05-31T20:38:22.383 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:38:22.383 DEBUG:boto:Port: 8000 2023-05-31T20:38:22.384 DEBUG:boto:Params: {} 2023-05-31T20:38:22.384 DEBUG:boto:Token: None 2023-05-31T20:38:22.385 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Wed, 31 May 2023 20:38:22 GMT /pklxkd-58/obj00000000000000000039 2023-05-31T20:38:22.385 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:RbqnbS71OmI4mftywHVFIF+9AFM= 2023-05-31T20:38:22.385 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Wed, 31 May 2023 20:38:22 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:RbqnbS71OmI4mftywHVFIF+9AFM='} 2023-05-31T20:38:22.395 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000d6ac868056b60f55-006477b03e-1630-a1'), ('Date', 'Wed, 31 May 2023 20:38:22 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:38:22.395 DEBUG:boto:path=/pklxkd-58/ 2023-05-31T20:38:22.396 DEBUG:boto:auth_path=/pklxkd-58/ 2023-05-31T20:38:22.396 DEBUG:boto:Method: HEAD 2023-05-31T20:38:22.397 DEBUG:boto:Path: /pklxkd-58/ 2023-05-31T20:38:22.397 DEBUG:boto:Data: 2023-05-31T20:38:22.397 DEBUG:boto:Headers: {} 2023-05-31T20:38:22.398 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:38:22.398 DEBUG:boto:Port: 8000 2023-05-31T20:38:22.398 DEBUG:boto:Params: {} 2023-05-31T20:38:22.399 DEBUG:boto:Token: None 2023-05-31T20:38:22.399 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:38:22 GMT /pklxkd-58/ 2023-05-31T20:38:22.400 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:yibMkgDYmA8uZlGmaKDLCUn3DTQ= 2023-05-31T20:38:22.400 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:38:22 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:yibMkgDYmA8uZlGmaKDLCUn3DTQ=', 'Content-Length': '0'} 2023-05-31T20:38:22.403 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '57'), ('X-RGW-Bytes-Used', '171'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000000fbf5013737ed6-006477b03e-1630-a1'), ('Content-Length', '0'), ('Date', 'Wed, 31 May 2023 20:38:22 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:38:22.404 DEBUG:boto:path=/pklxkd-58/obj00000000000000000026 2023-05-31T20:38:22.404 DEBUG:boto:auth_path=/pklxkd-58/obj00000000000000000026 2023-05-31T20:38:22.405 DEBUG:boto:Method: PUT 2023-05-31T20:38:22.405 DEBUG:boto:Path: /pklxkd-58/obj00000000000000000026 2023-05-31T20:38:22.406 DEBUG:boto:Data: 2023-05-31T20:38:22.406 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-05-31T20:38:22.407 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:38:22.407 DEBUG:boto:Port: 8000 2023-05-31T20:38:22.407 DEBUG:boto:Params: {} 2023-05-31T20:38:22.408 DEBUG:boto:Token: None 2023-05-31T20:38:22.408 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Wed, 31 May 2023 20:38:22 GMT /pklxkd-58/obj00000000000000000026 2023-05-31T20:38:22.409 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:haUap5ZxFivyqqDOlmn0nyl9v/0= 2023-05-31T20:38:22.409 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Wed, 31 May 2023 20:38:22 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:haUap5ZxFivyqqDOlmn0nyl9v/0='} 2023-05-31T20:38:22.452 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000a14f02d80f4e0d16-006477b03e-1630-a1'), ('Date', 'Wed, 31 May 2023 20:38:22 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:38:22.453 DEBUG:boto:path=/pklxkd-58/ 2023-05-31T20:38:22.453 DEBUG:boto:auth_path=/pklxkd-58/ 2023-05-31T20:38:22.454 DEBUG:boto:Method: HEAD 2023-05-31T20:38:22.454 DEBUG:boto:Path: /pklxkd-58/ 2023-05-31T20:38:22.455 DEBUG:boto:Data: 2023-05-31T20:38:22.455 DEBUG:boto:Headers: {} 2023-05-31T20:38:22.455 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:38:22.456 DEBUG:boto:Port: 8000 2023-05-31T20:38:22.456 DEBUG:boto:Params: {} 2023-05-31T20:38:22.456 DEBUG:boto:Token: None 2023-05-31T20:38:22.457 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:38:22 GMT /pklxkd-58/ 2023-05-31T20:38:22.457 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:yibMkgDYmA8uZlGmaKDLCUn3DTQ= 2023-05-31T20:38:22.458 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:38:22 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:yibMkgDYmA8uZlGmaKDLCUn3DTQ=', 'Content-Length': '0'} 2023-05-31T20:38:22.461 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '57'), ('X-RGW-Bytes-Used', '171'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000e9fb5be342b17ac1-006477b03e-1630-a1'), ('Content-Length', '0'), ('Date', 'Wed, 31 May 2023 20:38:22 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:38:22.462 DEBUG:boto:path=/pklxkd-58/obj00000000000000000020 2023-05-31T20:38:22.462 DEBUG:boto:auth_path=/pklxkd-58/obj00000000000000000020 2023-05-31T20:38:22.462 DEBUG:boto:Method: PUT 2023-05-31T20:38:22.463 DEBUG:boto:Path: /pklxkd-58/obj00000000000000000020 2023-05-31T20:38:22.463 DEBUG:boto:Data: 2023-05-31T20:38:22.463 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-05-31T20:38:22.464 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:38:22.464 DEBUG:boto:Port: 8000 2023-05-31T20:38:22.464 DEBUG:boto:Params: {} 2023-05-31T20:38:22.465 DEBUG:boto:Token: None 2023-05-31T20:38:22.465 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Wed, 31 May 2023 20:38:22 GMT /pklxkd-58/obj00000000000000000020 2023-05-31T20:38:22.466 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:b5aNPLUdZGeN3JVekoqN0tS5E50= 2023-05-31T20:38:22.467 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Wed, 31 May 2023 20:38:22 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:b5aNPLUdZGeN3JVekoqN0tS5E50='} 2023-05-31T20:38:22.476 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000238c88173043c368-006477b03e-1630-a1'), ('Date', 'Wed, 31 May 2023 20:38:22 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:38:22.477 DEBUG:boto:path=/pklxkd-58/ 2023-05-31T20:38:22.477 DEBUG:boto:auth_path=/pklxkd-58/ 2023-05-31T20:38:22.478 DEBUG:boto:Method: HEAD 2023-05-31T20:38:22.478 DEBUG:boto:Path: /pklxkd-58/ 2023-05-31T20:38:22.478 DEBUG:boto:Data: 2023-05-31T20:38:22.479 DEBUG:boto:Headers: {} 2023-05-31T20:38:22.480 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:38:22.480 DEBUG:boto:Port: 8000 2023-05-31T20:38:22.481 DEBUG:boto:Params: {} 2023-05-31T20:38:22.481 DEBUG:boto:Token: None 2023-05-31T20:38:22.482 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:38:22 GMT /pklxkd-58/ 2023-05-31T20:38:22.482 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:yibMkgDYmA8uZlGmaKDLCUn3DTQ= 2023-05-31T20:38:22.482 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:38:22 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:yibMkgDYmA8uZlGmaKDLCUn3DTQ=', 'Content-Length': '0'} 2023-05-31T20:38:22.486 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '57'), ('X-RGW-Bytes-Used', '171'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000059fa015070f1f5a8-006477b03e-1630-a1'), ('Content-Length', '0'), ('Date', 'Wed, 31 May 2023 20:38:22 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:38:22.486 DEBUG:boto:path=/pklxkd-58/obj00000000000000000048 2023-05-31T20:38:22.487 DEBUG:boto:auth_path=/pklxkd-58/obj00000000000000000048 2023-05-31T20:38:22.487 DEBUG:boto:Method: PUT 2023-05-31T20:38:22.488 DEBUG:boto:Path: /pklxkd-58/obj00000000000000000048 2023-05-31T20:38:22.488 DEBUG:boto:Data: 2023-05-31T20:38:22.488 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-05-31T20:38:22.489 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:38:22.489 DEBUG:boto:Port: 8000 2023-05-31T20:38:22.489 DEBUG:boto:Params: {} 2023-05-31T20:38:22.490 DEBUG:boto:Token: None 2023-05-31T20:38:22.490 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Wed, 31 May 2023 20:38:22 GMT /pklxkd-58/obj00000000000000000048 2023-05-31T20:38:22.490 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:/Zm4aTE2f6VhzgKjRn7YnHJwRDI= 2023-05-31T20:38:22.491 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Wed, 31 May 2023 20:38:22 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:/Zm4aTE2f6VhzgKjRn7YnHJwRDI='} 2023-05-31T20:38:22.532 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000ae3f26fa2f35f22f-006477b03e-1630-a1'), ('Date', 'Wed, 31 May 2023 20:38:22 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:38:22.533 DEBUG:boto:path=/pklxkd-58/ 2023-05-31T20:38:22.534 DEBUG:boto:auth_path=/pklxkd-58/ 2023-05-31T20:38:22.534 DEBUG:boto:Method: HEAD 2023-05-31T20:38:22.534 DEBUG:boto:Path: /pklxkd-58/ 2023-05-31T20:38:22.535 DEBUG:boto:Data: 2023-05-31T20:38:22.535 DEBUG:boto:Headers: {} 2023-05-31T20:38:22.536 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:38:22.537 DEBUG:boto:Port: 8000 2023-05-31T20:38:22.538 DEBUG:boto:Params: {} 2023-05-31T20:38:22.538 DEBUG:boto:Token: None 2023-05-31T20:38:22.539 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:38:22 GMT /pklxkd-58/ 2023-05-31T20:38:22.540 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:yibMkgDYmA8uZlGmaKDLCUn3DTQ= 2023-05-31T20:38:22.542 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:38:22 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:yibMkgDYmA8uZlGmaKDLCUn3DTQ=', 'Content-Length': '0'} 2023-05-31T20:38:22.545 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '58'), ('X-RGW-Bytes-Used', '174'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000eaed4e8bd7ce1b94-006477b03e-1630-a1'), ('Content-Length', '0'), ('Date', 'Wed, 31 May 2023 20:38:22 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:38:22.546 DEBUG:boto:path=/pklxkd-58/obj00000000000000000017 2023-05-31T20:38:22.546 DEBUG:boto:auth_path=/pklxkd-58/obj00000000000000000017 2023-05-31T20:38:22.546 DEBUG:boto:Method: PUT 2023-05-31T20:38:22.547 DEBUG:boto:Path: /pklxkd-58/obj00000000000000000017 2023-05-31T20:38:22.547 DEBUG:boto:Data: 2023-05-31T20:38:22.547 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-05-31T20:38:22.548 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:38:22.548 DEBUG:boto:Port: 8000 2023-05-31T20:38:22.548 DEBUG:boto:Params: {} 2023-05-31T20:38:22.549 DEBUG:boto:Token: None 2023-05-31T20:38:22.549 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Wed, 31 May 2023 20:38:22 GMT /pklxkd-58/obj00000000000000000017 2023-05-31T20:38:22.550 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:zlJYJPbI0fudo/N0MziRQ7B5rH4= 2023-05-31T20:38:22.550 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Wed, 31 May 2023 20:38:22 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:zlJYJPbI0fudo/N0MziRQ7B5rH4='} 2023-05-31T20:38:22.562 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000002e104f1ca9332648-006477b03e-1630-a1'), ('Date', 'Wed, 31 May 2023 20:38:22 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:38:22.563 DEBUG:boto:path=/pklxkd-58/ 2023-05-31T20:38:22.563 DEBUG:boto:auth_path=/pklxkd-58/ 2023-05-31T20:38:22.564 DEBUG:boto:Method: HEAD 2023-05-31T20:38:22.564 DEBUG:boto:Path: /pklxkd-58/ 2023-05-31T20:38:22.564 DEBUG:boto:Data: 2023-05-31T20:38:22.565 DEBUG:boto:Headers: {} 2023-05-31T20:38:22.565 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:38:22.565 DEBUG:boto:Port: 8000 2023-05-31T20:38:22.566 DEBUG:boto:Params: {} 2023-05-31T20:38:22.566 DEBUG:boto:Token: None 2023-05-31T20:38:22.567 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:38:22 GMT /pklxkd-58/ 2023-05-31T20:38:22.567 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:yibMkgDYmA8uZlGmaKDLCUn3DTQ= 2023-05-31T20:38:22.567 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:38:22 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:yibMkgDYmA8uZlGmaKDLCUn3DTQ=', 'Content-Length': '0'} 2023-05-31T20:38:22.571 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '58'), ('X-RGW-Bytes-Used', '174'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000049777ab96057b813-006477b03e-1630-a1'), ('Content-Length', '0'), ('Date', 'Wed, 31 May 2023 20:38:22 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:38:22.572 DEBUG:boto:path=/pklxkd-58/obj00000000000000000060 2023-05-31T20:38:22.572 DEBUG:boto:auth_path=/pklxkd-58/obj00000000000000000060 2023-05-31T20:38:22.573 DEBUG:boto:Method: PUT 2023-05-31T20:38:22.573 DEBUG:boto:Path: /pklxkd-58/obj00000000000000000060 2023-05-31T20:38:22.573 DEBUG:boto:Data: 2023-05-31T20:38:22.574 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-05-31T20:38:22.574 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:38:22.575 DEBUG:boto:Port: 8000 2023-05-31T20:38:22.575 DEBUG:boto:Params: {} 2023-05-31T20:38:22.575 DEBUG:boto:Token: None 2023-05-31T20:38:22.576 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Wed, 31 May 2023 20:38:22 GMT /pklxkd-58/obj00000000000000000060 2023-05-31T20:38:22.577 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:YhcbV4Xw6v2+z2DeKIRU2eFKMIk= 2023-05-31T20:38:22.577 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Wed, 31 May 2023 20:38:22 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:YhcbV4Xw6v2+z2DeKIRU2eFKMIk='} 2023-05-31T20:38:22.621 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000085588001d5181940-006477b03e-1630-a1'), ('Date', 'Wed, 31 May 2023 20:38:22 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:38:22.622 DEBUG:boto:path=/pklxkd-58/ 2023-05-31T20:38:22.622 DEBUG:boto:auth_path=/pklxkd-58/ 2023-05-31T20:38:22.623 DEBUG:boto:Method: HEAD 2023-05-31T20:38:22.624 DEBUG:boto:Path: /pklxkd-58/ 2023-05-31T20:38:22.624 DEBUG:boto:Data: 2023-05-31T20:38:22.625 DEBUG:boto:Headers: {} 2023-05-31T20:38:22.625 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:38:22.625 DEBUG:boto:Port: 8000 2023-05-31T20:38:22.626 DEBUG:boto:Params: {} 2023-05-31T20:38:22.626 DEBUG:boto:Token: None 2023-05-31T20:38:22.627 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:38:22 GMT /pklxkd-58/ 2023-05-31T20:38:22.627 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:yibMkgDYmA8uZlGmaKDLCUn3DTQ= 2023-05-31T20:38:22.627 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:38:22 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:yibMkgDYmA8uZlGmaKDLCUn3DTQ=', 'Content-Length': '0'} 2023-05-31T20:38:22.631 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '59'), ('X-RGW-Bytes-Used', '177'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000005da49c964db76c0-006477b03e-1630-a1'), ('Content-Length', '0'), ('Date', 'Wed, 31 May 2023 20:38:22 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:38:22.632 DEBUG:boto:path=/pklxkd-58/obj00000000000000000036 2023-05-31T20:38:22.633 DEBUG:boto:auth_path=/pklxkd-58/obj00000000000000000036 2023-05-31T20:38:22.633 DEBUG:boto:Method: PUT 2023-05-31T20:38:22.634 DEBUG:boto:Path: /pklxkd-58/obj00000000000000000036 2023-05-31T20:38:22.634 DEBUG:boto:Data: 2023-05-31T20:38:22.635 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-05-31T20:38:22.635 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:38:22.635 DEBUG:boto:Port: 8000 2023-05-31T20:38:22.636 DEBUG:boto:Params: {} 2023-05-31T20:38:22.636 DEBUG:boto:Token: None 2023-05-31T20:38:22.637 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Wed, 31 May 2023 20:38:22 GMT /pklxkd-58/obj00000000000000000036 2023-05-31T20:38:22.637 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:zBspOf/3YQQrhUurjEW3ujlegew= 2023-05-31T20:38:22.638 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Wed, 31 May 2023 20:38:22 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:zBspOf/3YQQrhUurjEW3ujlegew='} 2023-05-31T20:38:22.643 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000015456cae2caf0d69-006477b03e-1630-a1'), ('Date', 'Wed, 31 May 2023 20:38:22 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:38:22.644 DEBUG:boto:path=/pklxkd-58/ 2023-05-31T20:38:22.644 DEBUG:boto:auth_path=/pklxkd-58/ 2023-05-31T20:38:22.645 DEBUG:boto:Method: HEAD 2023-05-31T20:38:22.645 DEBUG:boto:Path: /pklxkd-58/ 2023-05-31T20:38:22.646 DEBUG:boto:Data: 2023-05-31T20:38:22.646 DEBUG:boto:Headers: {} 2023-05-31T20:38:22.646 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:38:22.647 DEBUG:boto:Port: 8000 2023-05-31T20:38:22.647 DEBUG:boto:Params: {} 2023-05-31T20:38:22.647 DEBUG:boto:Token: None 2023-05-31T20:38:22.648 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:38:22 GMT /pklxkd-58/ 2023-05-31T20:38:22.648 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:yibMkgDYmA8uZlGmaKDLCUn3DTQ= 2023-05-31T20:38:22.649 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:38:22 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:yibMkgDYmA8uZlGmaKDLCUn3DTQ=', 'Content-Length': '0'} 2023-05-31T20:38:22.652 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '60'), ('X-RGW-Bytes-Used', '180'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000024f402a70be4e279-006477b03e-1630-a1'), ('Content-Length', '0'), ('Date', 'Wed, 31 May 2023 20:38:22 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:38:22.653 DEBUG:boto:path=/pklxkd-58/obj00000000000000000015 2023-05-31T20:38:22.653 DEBUG:boto:auth_path=/pklxkd-58/obj00000000000000000015 2023-05-31T20:38:22.654 DEBUG:boto:Method: PUT 2023-05-31T20:38:22.654 DEBUG:boto:Path: /pklxkd-58/obj00000000000000000015 2023-05-31T20:38:22.654 DEBUG:boto:Data: 2023-05-31T20:38:22.655 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-05-31T20:38:22.655 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:38:22.655 DEBUG:boto:Port: 8000 2023-05-31T20:38:22.656 DEBUG:boto:Params: {} 2023-05-31T20:38:22.656 DEBUG:boto:Token: None 2023-05-31T20:38:22.657 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Wed, 31 May 2023 20:38:22 GMT /pklxkd-58/obj00000000000000000015 2023-05-31T20:38:22.658 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:DO1ub15canuSEo5L+gCV4eutvjA= 2023-05-31T20:38:22.658 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Wed, 31 May 2023 20:38:22 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:DO1ub15canuSEo5L+gCV4eutvjA='} 2023-05-31T20:38:22.700 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000000bfaaea3bdd473b2-006477b03e-1630-a1'), ('Date', 'Wed, 31 May 2023 20:38:22 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:38:22.701 DEBUG:boto:path=/pklxkd-58/ 2023-05-31T20:38:22.701 DEBUG:boto:auth_path=/pklxkd-58/ 2023-05-31T20:38:22.702 DEBUG:boto:Method: HEAD 2023-05-31T20:38:22.702 DEBUG:boto:Path: /pklxkd-58/ 2023-05-31T20:38:22.703 DEBUG:boto:Data: 2023-05-31T20:38:22.703 DEBUG:boto:Headers: {} 2023-05-31T20:38:22.703 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:38:22.703 DEBUG:boto:Port: 8000 2023-05-31T20:38:22.704 DEBUG:boto:Params: {} 2023-05-31T20:38:22.704 DEBUG:boto:Token: None 2023-05-31T20:38:22.705 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:38:22 GMT /pklxkd-58/ 2023-05-31T20:38:22.705 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:yibMkgDYmA8uZlGmaKDLCUn3DTQ= 2023-05-31T20:38:22.705 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:38:22 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:yibMkgDYmA8uZlGmaKDLCUn3DTQ=', 'Content-Length': '0'} 2023-05-31T20:38:22.709 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '60'), ('X-RGW-Bytes-Used', '180'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000000dab98c1c2c57c1a-006477b03e-1630-a1'), ('Content-Length', '0'), ('Date', 'Wed, 31 May 2023 20:38:22 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:38:22.709 DEBUG:boto:path=/pklxkd-58/obj00000000000000000011 2023-05-31T20:38:22.710 DEBUG:boto:auth_path=/pklxkd-58/obj00000000000000000011 2023-05-31T20:38:22.710 DEBUG:boto:Method: PUT 2023-05-31T20:38:22.711 DEBUG:boto:Path: /pklxkd-58/obj00000000000000000011 2023-05-31T20:38:22.711 DEBUG:boto:Data: 2023-05-31T20:38:22.711 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-05-31T20:38:22.712 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:38:22.712 DEBUG:boto:Port: 8000 2023-05-31T20:38:22.712 DEBUG:boto:Params: {} 2023-05-31T20:38:22.713 DEBUG:boto:Token: None 2023-05-31T20:38:22.713 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Wed, 31 May 2023 20:38:22 GMT /pklxkd-58/obj00000000000000000011 2023-05-31T20:38:22.713 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:1YRFz7Psd4WQutnCuax4j34ya5M= 2023-05-31T20:38:22.714 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Wed, 31 May 2023 20:38:22 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:1YRFz7Psd4WQutnCuax4j34ya5M='} 2023-05-31T20:38:22.720 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000751c74c469fc0eab-006477b03e-1630-a1'), ('Date', 'Wed, 31 May 2023 20:38:22 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:38:22.722 DEBUG:boto:path=/pklxkd-58/ 2023-05-31T20:38:22.722 DEBUG:boto:auth_path=/pklxkd-58/ 2023-05-31T20:38:22.722 DEBUG:boto:Method: HEAD 2023-05-31T20:38:22.723 DEBUG:boto:Path: /pklxkd-58/ 2023-05-31T20:38:22.723 DEBUG:boto:Data: 2023-05-31T20:38:22.723 DEBUG:boto:Headers: {} 2023-05-31T20:38:22.724 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:38:22.724 DEBUG:boto:Port: 8000 2023-05-31T20:38:22.724 DEBUG:boto:Params: {} 2023-05-31T20:38:22.725 DEBUG:boto:Token: None 2023-05-31T20:38:22.725 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:38:22 GMT /pklxkd-58/ 2023-05-31T20:38:22.726 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:yibMkgDYmA8uZlGmaKDLCUn3DTQ= 2023-05-31T20:38:22.726 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:38:22 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:yibMkgDYmA8uZlGmaKDLCUn3DTQ=', 'Content-Length': '0'} 2023-05-31T20:38:22.729 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '60'), ('X-RGW-Bytes-Used', '180'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000b1fbd52750e197ac-006477b03e-1630-a1'), ('Content-Length', '0'), ('Date', 'Wed, 31 May 2023 20:38:22 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:38:22.730 DEBUG:boto:path=/pklxkd-58/obj00000000000000000013 2023-05-31T20:38:22.730 DEBUG:boto:auth_path=/pklxkd-58/obj00000000000000000013 2023-05-31T20:38:22.731 DEBUG:boto:Method: PUT 2023-05-31T20:38:22.731 DEBUG:boto:Path: /pklxkd-58/obj00000000000000000013 2023-05-31T20:38:22.731 DEBUG:boto:Data: 2023-05-31T20:38:22.732 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-05-31T20:38:22.732 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:38:22.732 DEBUG:boto:Port: 8000 2023-05-31T20:38:22.733 DEBUG:boto:Params: {} 2023-05-31T20:38:22.734 DEBUG:boto:Token: None 2023-05-31T20:38:22.734 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Wed, 31 May 2023 20:38:22 GMT /pklxkd-58/obj00000000000000000013 2023-05-31T20:38:22.736 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:onw0jwAQmaw9tdFkdXGUsnQFMQI= 2023-05-31T20:38:22.736 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Wed, 31 May 2023 20:38:22 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:onw0jwAQmaw9tdFkdXGUsnQFMQI='} 2023-05-31T20:38:22.780 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000220e3f90a2a5f77f-006477b03e-1630-a1'), ('Date', 'Wed, 31 May 2023 20:38:22 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:38:22.781 DEBUG:boto:path=/pklxkd-58/ 2023-05-31T20:38:22.781 DEBUG:boto:auth_path=/pklxkd-58/ 2023-05-31T20:38:22.782 DEBUG:boto:Method: HEAD 2023-05-31T20:38:22.782 DEBUG:boto:Path: /pklxkd-58/ 2023-05-31T20:38:22.783 DEBUG:boto:Data: 2023-05-31T20:38:22.783 DEBUG:boto:Headers: {} 2023-05-31T20:38:22.783 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:38:22.784 DEBUG:boto:Port: 8000 2023-05-31T20:38:22.784 DEBUG:boto:Params: {} 2023-05-31T20:38:22.784 DEBUG:boto:Token: None 2023-05-31T20:38:22.785 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:38:22 GMT /pklxkd-58/ 2023-05-31T20:38:22.786 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:yibMkgDYmA8uZlGmaKDLCUn3DTQ= 2023-05-31T20:38:22.786 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:38:22 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:yibMkgDYmA8uZlGmaKDLCUn3DTQ=', 'Content-Length': '0'} 2023-05-31T20:38:22.789 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '60'), ('X-RGW-Bytes-Used', '180'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000bd848d8d9cc364a2-006477b03e-1630-a1'), ('Content-Length', '0'), ('Date', 'Wed, 31 May 2023 20:38:22 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:38:22.790 DEBUG:boto:path=/pklxkd-58/obj00000000000000000007 2023-05-31T20:38:22.790 DEBUG:boto:auth_path=/pklxkd-58/obj00000000000000000007 2023-05-31T20:38:22.791 DEBUG:boto:Method: PUT 2023-05-31T20:38:22.791 DEBUG:boto:Path: /pklxkd-58/obj00000000000000000007 2023-05-31T20:38:22.792 DEBUG:boto:Data: 2023-05-31T20:38:22.792 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-05-31T20:38:22.792 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:38:22.793 DEBUG:boto:Port: 8000 2023-05-31T20:38:22.793 DEBUG:boto:Params: {} 2023-05-31T20:38:22.794 DEBUG:boto:Token: None 2023-05-31T20:38:22.794 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Wed, 31 May 2023 20:38:22 GMT /pklxkd-58/obj00000000000000000007 2023-05-31T20:38:22.795 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:cV/EE5e7ghwIZmf5dHd366WfwEU= 2023-05-31T20:38:22.795 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Wed, 31 May 2023 20:38:22 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:cV/EE5e7ghwIZmf5dHd366WfwEU='} 2023-05-31T20:38:22.802 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000002ebf64277393dd2a-006477b03e-1630-a1'), ('Date', 'Wed, 31 May 2023 20:38:22 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:38:22.802 DEBUG:boto:path=/pklxkd-58/ 2023-05-31T20:38:22.803 DEBUG:boto:auth_path=/pklxkd-58/ 2023-05-31T20:38:22.803 DEBUG:boto:Method: HEAD 2023-05-31T20:38:22.803 DEBUG:boto:Path: /pklxkd-58/ 2023-05-31T20:38:22.804 DEBUG:boto:Data: 2023-05-31T20:38:22.804 DEBUG:boto:Headers: {} 2023-05-31T20:38:22.805 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:38:22.805 DEBUG:boto:Port: 8000 2023-05-31T20:38:22.805 DEBUG:boto:Params: {} 2023-05-31T20:38:22.806 DEBUG:boto:Token: None 2023-05-31T20:38:22.806 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:38:22 GMT /pklxkd-58/ 2023-05-31T20:38:22.806 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:yibMkgDYmA8uZlGmaKDLCUn3DTQ= 2023-05-31T20:38:22.807 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:38:22 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:yibMkgDYmA8uZlGmaKDLCUn3DTQ=', 'Content-Length': '0'} 2023-05-31T20:38:22.810 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '60'), ('X-RGW-Bytes-Used', '180'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000787ca89c9e52d837-006477b03e-1630-a1'), ('Content-Length', '0'), ('Date', 'Wed, 31 May 2023 20:38:22 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:38:22.811 DEBUG:boto:path=/pklxkd-58/obj00000000000000000024 2023-05-31T20:38:22.811 DEBUG:boto:auth_path=/pklxkd-58/obj00000000000000000024 2023-05-31T20:38:22.811 DEBUG:boto:Method: PUT 2023-05-31T20:38:22.812 DEBUG:boto:Path: /pklxkd-58/obj00000000000000000024 2023-05-31T20:38:22.812 DEBUG:boto:Data: 2023-05-31T20:38:22.812 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-05-31T20:38:22.813 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:38:22.813 DEBUG:boto:Port: 8000 2023-05-31T20:38:22.813 DEBUG:boto:Params: {} 2023-05-31T20:38:22.814 DEBUG:boto:Token: None 2023-05-31T20:38:22.814 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Wed, 31 May 2023 20:38:22 GMT /pklxkd-58/obj00000000000000000024 2023-05-31T20:38:22.815 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:Jx8bDqpB1xsPl4q24THPjmow7xE= 2023-05-31T20:38:22.815 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Wed, 31 May 2023 20:38:22 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:Jx8bDqpB1xsPl4q24THPjmow7xE='} 2023-05-31T20:38:22.860 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000f2136686e7168277-006477b03e-1630-a1'), ('Date', 'Wed, 31 May 2023 20:38:22 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:38:22.861 DEBUG:boto:path=/pklxkd-58/ 2023-05-31T20:38:22.861 DEBUG:boto:auth_path=/pklxkd-58/ 2023-05-31T20:38:22.862 DEBUG:boto:Method: HEAD 2023-05-31T20:38:22.862 DEBUG:boto:Path: /pklxkd-58/ 2023-05-31T20:38:22.862 DEBUG:boto:Data: 2023-05-31T20:38:22.863 DEBUG:boto:Headers: {} 2023-05-31T20:38:22.863 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:38:22.863 DEBUG:boto:Port: 8000 2023-05-31T20:38:22.864 DEBUG:boto:Params: {} 2023-05-31T20:38:22.864 DEBUG:boto:Token: None 2023-05-31T20:38:22.865 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:38:22 GMT /pklxkd-58/ 2023-05-31T20:38:22.865 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:yibMkgDYmA8uZlGmaKDLCUn3DTQ= 2023-05-31T20:38:22.865 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:38:22 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:yibMkgDYmA8uZlGmaKDLCUn3DTQ=', 'Content-Length': '0'} 2023-05-31T20:38:22.869 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '60'), ('X-RGW-Bytes-Used', '180'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000041c13a5f62bac680-006477b03e-1630-a1'), ('Content-Length', '0'), ('Date', 'Wed, 31 May 2023 20:38:22 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:38:22.870 DEBUG:boto:path=/pklxkd-58/obj00000000000000000053 2023-05-31T20:38:22.870 DEBUG:boto:auth_path=/pklxkd-58/obj00000000000000000053 2023-05-31T20:38:22.870 DEBUG:boto:Method: PUT 2023-05-31T20:38:22.871 DEBUG:boto:Path: /pklxkd-58/obj00000000000000000053 2023-05-31T20:38:22.871 DEBUG:boto:Data: 2023-05-31T20:38:22.871 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-05-31T20:38:22.872 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:38:22.872 DEBUG:boto:Port: 8000 2023-05-31T20:38:22.872 DEBUG:boto:Params: {} 2023-05-31T20:38:22.873 DEBUG:boto:Token: None 2023-05-31T20:38:22.874 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Wed, 31 May 2023 20:38:22 GMT /pklxkd-58/obj00000000000000000053 2023-05-31T20:38:22.874 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:HRPESdwkolRYei/2qmchRlP5C78= 2023-05-31T20:38:22.875 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Wed, 31 May 2023 20:38:22 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:HRPESdwkolRYei/2qmchRlP5C78='} 2023-05-31T20:38:22.884 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000bbf256c856787d6e-006477b03e-1630-a1'), ('Date', 'Wed, 31 May 2023 20:38:22 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:38:22.885 DEBUG:boto:path=/pklxkd-58/ 2023-05-31T20:38:22.885 DEBUG:boto:auth_path=/pklxkd-58/ 2023-05-31T20:38:22.885 DEBUG:boto:Method: HEAD 2023-05-31T20:38:22.886 DEBUG:boto:Path: /pklxkd-58/ 2023-05-31T20:38:22.886 DEBUG:boto:Data: 2023-05-31T20:38:22.886 DEBUG:boto:Headers: {} 2023-05-31T20:38:22.887 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:38:22.887 DEBUG:boto:Port: 8000 2023-05-31T20:38:22.887 DEBUG:boto:Params: {} 2023-05-31T20:38:22.888 DEBUG:boto:Token: None 2023-05-31T20:38:22.888 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:38:22 GMT /pklxkd-58/ 2023-05-31T20:38:22.889 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:yibMkgDYmA8uZlGmaKDLCUn3DTQ= 2023-05-31T20:38:22.889 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:38:22 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:yibMkgDYmA8uZlGmaKDLCUn3DTQ=', 'Content-Length': '0'} 2023-05-31T20:38:22.892 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '61'), ('X-RGW-Bytes-Used', '183'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000b62f7a9d757ad217-006477b03e-1630-a1'), ('Content-Length', '0'), ('Date', 'Wed, 31 May 2023 20:38:22 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:38:22.893 DEBUG:boto:path=/pklxkd-58/obj00000000000000000014 2023-05-31T20:38:22.893 DEBUG:boto:auth_path=/pklxkd-58/obj00000000000000000014 2023-05-31T20:38:22.894 DEBUG:boto:Method: PUT 2023-05-31T20:38:22.894 DEBUG:boto:Path: /pklxkd-58/obj00000000000000000014 2023-05-31T20:38:22.895 DEBUG:boto:Data: 2023-05-31T20:38:22.896 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-05-31T20:38:22.896 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:38:22.896 DEBUG:boto:Port: 8000 2023-05-31T20:38:22.897 DEBUG:boto:Params: {} 2023-05-31T20:38:22.897 DEBUG:boto:Token: None 2023-05-31T20:38:22.897 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Wed, 31 May 2023 20:38:22 GMT /pklxkd-58/obj00000000000000000014 2023-05-31T20:38:22.898 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:rQkIfV9QXpmrVUSFbv6GcHJQZMo= 2023-05-31T20:38:22.898 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Wed, 31 May 2023 20:38:22 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:rQkIfV9QXpmrVUSFbv6GcHJQZMo='} 2023-05-31T20:38:22.941 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000aef9ebf67800196c-006477b03e-1630-a1'), ('Date', 'Wed, 31 May 2023 20:38:22 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:38:22.942 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'run', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'pklxkd-58', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-05-31T20:38:22.942 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'run', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'pklxkd-58', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-05-31T20:38:22.943 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync run --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --bucket pklxkd-58 --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 2023-05-31T20:38:22.996 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2023-05-31T20:38:22.997 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2023-05-31T20:38:23.114 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:23.107+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:0]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-05-31T20:38:23.114 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:23.107+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:3]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-05-31T20:38:23.116 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:23.107+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:4]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-05-31T20:38:23.116 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:23.107+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:0]: sync status for source bucket shard: 2 2023-05-31T20:38:23.117 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:23.107+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:0]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:0]: listing bilog for incremental sync; position=00000000001.41.6 2023-05-31T20:38:23.117 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:23.107+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:2]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-05-31T20:38:23.117 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:23.107+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:3]: sync status for source bucket shard: 2 2023-05-31T20:38:23.118 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:23.107+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:3]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:3]: listing bilog for incremental sync; position=00000000001.33.6 2023-05-31T20:38:23.118 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:23.107+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:4]: sync status for source bucket shard: 2 2023-05-31T20:38:23.118 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:23.108+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:4]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:4]: listing bilog for incremental sync; position=00000000001.33.6 2023-05-31T20:38:23.119 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:23.108+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:5]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-05-31T20:38:23.119 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:23.108+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:1]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-05-31T20:38:23.119 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:23.108+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:2]: sync status for source bucket shard: 2 2023-05-31T20:38:23.120 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:23.108+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:2]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:2]: listing bilog for incremental sync; position=00000000001.37.6 2023-05-31T20:38:23.120 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:23.108+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:9]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-05-31T20:38:23.120 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:23.109+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:5]: sync status for source bucket shard: 2 2023-05-31T20:38:23.121 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:23.109+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:7]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-05-31T20:38:23.121 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:23.109+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:8]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-05-31T20:38:23.122 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:23.109+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:5]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:5]: listing bilog for incremental sync; position=00000000001.33.6 2023-05-31T20:38:23.122 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:23.109+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:9]: sync status for source bucket shard: 2 2023-05-31T20:38:23.122 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:23.109+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:1]: sync status for source bucket shard: 2 2023-05-31T20:38:23.123 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:23.109+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:9]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:9]: listing bilog for incremental sync; position=00000000001.30.6 2023-05-31T20:38:23.123 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:23.109+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:1]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:1]: listing bilog for incremental sync; position= 2023-05-31T20:38:23.123 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:23.109+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:6]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-05-31T20:38:23.124 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:23.109+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:10]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-05-31T20:38:23.125 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:23.110+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:6]: sync status for source bucket shard: 2 2023-05-31T20:38:23.125 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:23.110+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:10]: sync status for source bucket shard: 2 2023-05-31T20:38:23.126 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:23.110+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:6]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:6]: listing bilog for incremental sync; position=00000000001.38.6 2023-05-31T20:38:23.126 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:23.110+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:10]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:10]: listing bilog for incremental sync; position=00000000001.25.6 2023-05-31T20:38:23.126 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:23.110+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:8]: sync status for source bucket shard: 2 2023-05-31T20:38:23.126 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:23.110+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:8]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:8]: listing bilog for incremental sync; position=00000000001.39.6 2023-05-31T20:38:23.127 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:23.110+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:7]: sync status for source bucket shard: 2 2023-05-31T20:38:23.127 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:23.110+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:7]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:7]: listing bilog for incremental sync; position=00000000001.35.6 2023-05-31T20:38:23.177 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:23.172+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:0]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-05-31T20:38:23.177 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:23.172+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:1]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-05-31T20:38:23.177 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:23.172+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:3]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-05-31T20:38:23.177 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:23.172+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:0]: sync status for source bucket shard: 0 2023-05-31T20:38:23.177 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:23.172+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:0]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:0]: listing bilog for incremental sync; position= 2023-05-31T20:38:23.178 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:23.172+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:5]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-05-31T20:38:23.178 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:23.172+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:1]: sync status for source bucket shard: 0 2023-05-31T20:38:23.178 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:23.172+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:1]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:1]: listing bilog for incremental sync; position= 2023-05-31T20:38:23.178 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:23.172+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:3]: sync status for source bucket shard: 0 2023-05-31T20:38:23.178 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:23.172+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:3]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:3]: listing bilog for incremental sync; position= 2023-05-31T20:38:23.178 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:23.173+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:5]: sync status for source bucket shard: 0 2023-05-31T20:38:23.179 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:23.173+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:5]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:5]: listing bilog for incremental sync; position= 2023-05-31T20:38:23.182 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:23.173+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:2]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-05-31T20:38:23.182 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:23.173+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:4]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-05-31T20:38:23.182 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:23.173+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:6]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-05-31T20:38:23.183 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:23.173+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:9]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-05-31T20:38:23.183 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:23.174+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:2]: sync status for source bucket shard: 0 2023-05-31T20:38:23.183 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:23.174+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:2]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:2]: listing bilog for incremental sync; position= 2023-05-31T20:38:23.183 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:23.174+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:4]: sync status for source bucket shard: 0 2023-05-31T20:38:23.183 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:23.174+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:4]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:4]: listing bilog for incremental sync; position= 2023-05-31T20:38:23.184 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:23.174+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:10]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-05-31T20:38:23.184 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:23.174+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:9]: sync status for source bucket shard: 0 2023-05-31T20:38:23.184 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:23.174+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:7]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-05-31T20:38:23.184 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:23.174+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:9]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:9]: listing bilog for incremental sync; position= 2023-05-31T20:38:23.184 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:23.174+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:6]: sync status for source bucket shard: 0 2023-05-31T20:38:23.184 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:23.174+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:6]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:6]: listing bilog for incremental sync; position= 2023-05-31T20:38:23.185 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:23.174+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:11]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-05-31T20:38:23.185 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:23.174+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:8]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-05-31T20:38:23.185 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:23.174+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:12]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-05-31T20:38:23.185 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:23.174+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:15]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-05-31T20:38:23.185 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:23.175+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:13]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-05-31T20:38:23.185 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:23.175+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:7]: sync status for source bucket shard: 0 2023-05-31T20:38:23.186 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:23.175+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:7]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:7]: listing bilog for incremental sync; position= 2023-05-31T20:38:23.186 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:23.175+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:10]: sync status for source bucket shard: 0 2023-05-31T20:38:23.186 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:23.175+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:13]: sync status for source bucket shard: 0 2023-05-31T20:38:23.186 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:23.175+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:10]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:10]: listing bilog for incremental sync; position= 2023-05-31T20:38:23.186 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:23.175+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:13]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:13]: listing bilog for incremental sync; position= 2023-05-31T20:38:23.186 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:23.175+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:16]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-05-31T20:38:23.187 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:23.175+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:8]: sync status for source bucket shard: 0 2023-05-31T20:38:23.187 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:23.175+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:15]: sync status for source bucket shard: 0 2023-05-31T20:38:23.187 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:23.175+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:8]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:8]: listing bilog for incremental sync; position= 2023-05-31T20:38:23.187 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:23.175+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:15]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:15]: listing bilog for incremental sync; position= 2023-05-31T20:38:23.187 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:23.175+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:11]: sync status for source bucket shard: 0 2023-05-31T20:38:23.187 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:23.176+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:12]: sync status for source bucket shard: 0 2023-05-31T20:38:23.188 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:23.176+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:11]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:11]: listing bilog for incremental sync; position= 2023-05-31T20:38:23.188 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:23.176+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:12]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:12]: listing bilog for incremental sync; position= 2023-05-31T20:38:23.188 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:23.176+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:14]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-05-31T20:38:23.188 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:23.176+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:2]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:2]: parsed entry: id=00000000001.75.6 iter->object=obj00000000000000000024 iter->instance= name=obj00000000000000000024 instance= ns= 2023-05-31T20:38:23.188 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:23.176+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:2]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:2]: syncing object: pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:2/obj00000000000000000024 2023-05-31T20:38:23.188 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:23.176+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:2]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:2]: entry->timestamp=2023-05-31T20:38:15.375872+0000 2023-05-31T20:38:23.189 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:23.176+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:2]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:2]:entry[obj00000000000000000024]: bucket sync single entry (source_zone=3d163b1e-09c7-47e2-ae87-add4aace3f38) b=bucket sync single entry (source_zone=3d163b1e-09c7-47e2-ae87-add4aace3f38) b= log_entry=00000000001.75.6 op=0 op_state=1 2023-05-31T20:38:23.189 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:23.176+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:4]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:4]: parsed entry: id=00000000001.43.6 iter->object=obj00000000000000000028 iter->instance= name=obj00000000000000000028 instance= ns= 2023-05-31T20:38:23.189 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:23.176+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:4]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:4]: syncing object: pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:4/obj00000000000000000028 2023-05-31T20:38:23.189 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:23.176+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:4]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:4]: entry->timestamp=2023-05-31T20:38:15.316632+0000 2023-05-31T20:38:23.189 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:23.176+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:4]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:4]:entry[obj00000000000000000028]: bucket sync single entry (source_zone=3d163b1e-09c7-47e2-ae87-add4aace3f38) b=bucket sync single entry (source_zone=3d163b1e-09c7-47e2-ae87-add4aace3f38) b= log_entry=00000000001.43.6 op=0 op_state=1 2023-05-31T20:38:23.189 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:23.176+0000 7f22572d6780 5 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:2]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:2]:entry[obj00000000000000000024]: bucket sync: sync obj: 3d163b1e-09c7-47e2-ae87-add4aace3f38/:pklxkd-58[3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23])/obj00000000000000000024[0] 2023-05-31T20:38:23.189 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:23.176+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:2]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:2]: listing bilog for incremental sync; position=00000000001.75.6 2023-05-31T20:38:23.190 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:23.176+0000 7f22572d6780 0 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:4]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:4]:entry[obj00000000000000000028]: : injecting data sync error on key=obj00000000000000000028 2023-05-31T20:38:23.190 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:23.176+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:4]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:4]: listing bilog for incremental sync; position=00000000001.43.6 2023-05-31T20:38:23.190 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:23.176+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:16]: sync status for source bucket shard: 0 2023-05-31T20:38:23.190 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:23.176+0000 7f22572d6780 10 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:4]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:4]:entry[obj00000000000000000028]: failed, retcode=-5 ((5) Input/output error) 2023-05-31T20:38:23.190 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:23.176+0000 7f22572d6780 0 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:4]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:4]:entry[obj00000000000000000028]: ERROR: failed to sync object: pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:4/obj00000000000000000028 2023-05-31T20:38:23.190 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:23.176+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:16]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:16]: listing bilog for incremental sync; position= 2023-05-31T20:38:23.193 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:23.178+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:8]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:8]: parsed entry: id=00000000002.50.6 iter->object=obj00000000000000000010 iter->instance= name=obj00000000000000000010 instance= ns= 2023-05-31T20:38:23.193 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:23.178+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:8]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:8]: syncing object: pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:8/obj00000000000000000010 2023-05-31T20:38:23.193 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:23.178+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:8]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:8]: entry->timestamp=2023-05-31T20:38:15.777030+0000 2023-05-31T20:38:23.193 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:23.178+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:8]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:8]:entry[obj00000000000000000010]: bucket sync single entry (source_zone=3d163b1e-09c7-47e2-ae87-add4aace3f38) b=bucket sync single entry (source_zone=3d163b1e-09c7-47e2-ae87-add4aace3f38) b= log_entry=00000000002.50.6 op=0 op_state=1 2023-05-31T20:38:23.193 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:23.178+0000 7f22572d6780 5 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:8]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:8]:entry[obj00000000000000000010]: bucket sync: sync obj: 3d163b1e-09c7-47e2-ae87-add4aace3f38/:pklxkd-58[3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23])/obj00000000000000000010[0] 2023-05-31T20:38:23.194 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:23.178+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:8]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:8]: listing bilog for incremental sync; position=00000000002.50.6 2023-05-31T20:38:23.194 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:23.178+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:9]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:9]: parsed entry: id=00000000001.118.6 iter->object=obj00000000000000000019 iter->instance= name=obj00000000000000000019 instance= ns= 2023-05-31T20:38:23.194 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:23.178+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:9]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:9]: syncing object: pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:9/obj00000000000000000019 2023-05-31T20:38:23.194 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:23.178+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:9]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:9]: entry->timestamp=2023-05-31T20:38:15.414071+0000 2023-05-31T20:38:23.194 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:23.178+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:9]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:9]:entry[obj00000000000000000019]: bucket sync single entry (source_zone=3d163b1e-09c7-47e2-ae87-add4aace3f38) b=bucket sync single entry (source_zone=3d163b1e-09c7-47e2-ae87-add4aace3f38) b= log_entry=00000000001.118.6 op=0 op_state=1 2023-05-31T20:38:23.194 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:23.178+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:6]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:6]: parsed entry: id=00000000001.52.6 iter->object=obj00000000000000000032 iter->instance= name=obj00000000000000000032 instance= ns= 2023-05-31T20:38:23.194 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:23.178+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:6]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:6]: syncing object: pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:6/obj00000000000000000032 2023-05-31T20:38:23.195 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:23.178+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:6]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:6]: entry->timestamp=2023-05-31T20:38:15.861929+0000 2023-05-31T20:38:23.195 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:23.178+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:6]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:6]:entry[obj00000000000000000032]: bucket sync single entry (source_zone=3d163b1e-09c7-47e2-ae87-add4aace3f38) b=bucket sync single entry (source_zone=3d163b1e-09c7-47e2-ae87-add4aace3f38) b= log_entry=00000000001.52.6 op=0 op_state=1 2023-05-31T20:38:23.195 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:23.178+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:10]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:10]: parsed entry: id=00000000002.38.6 iter->object=obj00000000000000000013 iter->instance= name=obj00000000000000000013 instance= ns= 2023-05-31T20:38:23.195 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:23.178+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:10]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:10]: syncing object: pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:10/obj00000000000000000013 2023-05-31T20:38:23.195 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:23.178+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:10]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:10]: entry->timestamp=2023-05-31T20:38:15.394812+0000 2023-05-31T20:38:23.195 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:23.178+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:10]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:10]:entry[obj00000000000000000013]: bucket sync single entry (source_zone=3d163b1e-09c7-47e2-ae87-add4aace3f38) b=bucket sync single entry (source_zone=3d163b1e-09c7-47e2-ae87-add4aace3f38) b= log_entry=00000000002.38.6 op=0 op_state=1 2023-05-31T20:38:23.196 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:23.178+0000 7f22572d6780 5 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:9]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:9]:entry[obj00000000000000000019]: bucket sync: sync obj: 3d163b1e-09c7-47e2-ae87-add4aace3f38/:pklxkd-58[3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23])/obj00000000000000000019[0] 2023-05-31T20:38:23.196 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:23.178+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:9]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:9]: listing bilog for incremental sync; position=00000000001.118.6 2023-05-31T20:38:23.196 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:23.178+0000 7f22572d6780 5 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:6]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:6]:entry[obj00000000000000000032]: bucket sync: sync obj: 3d163b1e-09c7-47e2-ae87-add4aace3f38/:pklxkd-58[3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23])/obj00000000000000000032[0] 2023-05-31T20:38:23.196 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:23.178+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:6]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:6]: listing bilog for incremental sync; position=00000000001.52.6 2023-05-31T20:38:23.196 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:23.178+0000 7f22572d6780 5 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:10]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:10]:entry[obj00000000000000000013]: bucket sync: sync obj: 3d163b1e-09c7-47e2-ae87-add4aace3f38/:pklxkd-58[3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23])/obj00000000000000000013[0] 2023-05-31T20:38:23.196 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:23.178+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:10]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:10]: listing bilog for incremental sync; position=00000000002.38.6 2023-05-31T20:38:23.197 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:23.178+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:14]: sync status for source bucket shard: 0 2023-05-31T20:38:23.197 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:23.178+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:14]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:14]: listing bilog for incremental sync; position= 2023-05-31T20:38:23.197 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:23.179+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:15]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:15]: parsed entry: id=00000000002.27.6 iter->object=obj00000000000000000004 iter->instance= name=obj00000000000000000004 instance= ns= 2023-05-31T20:38:23.197 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:23.179+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:15]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:15]: syncing object: pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:15/obj00000000000000000004 2023-05-31T20:38:23.197 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:23.179+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:15]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:15]: entry->timestamp=2023-05-31T20:38:15.837602+0000 2023-05-31T20:38:23.198 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:23.179+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:15]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:15]:entry[obj00000000000000000004]: bucket sync single entry (source_zone=3d163b1e-09c7-47e2-ae87-add4aace3f38) b=bucket sync single entry (source_zone=3d163b1e-09c7-47e2-ae87-add4aace3f38) b= log_entry=00000000002.27.6 op=0 op_state=1 2023-05-31T20:38:23.198 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:23.179+0000 7f22572d6780 5 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:15]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:15]:entry[obj00000000000000000004]: bucket sync: sync obj: 3d163b1e-09c7-47e2-ae87-add4aace3f38/:pklxkd-58[3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23])/obj00000000000000000004[0] 2023-05-31T20:38:23.198 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:23.179+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:15]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:15]: listing bilog for incremental sync; position=00000000002.27.6 2023-05-31T20:38:23.198 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:23.179+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:11]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:11]: parsed entry: id=00000000001.120.6 iter->object=obj00000000000000000021 iter->instance= name=obj00000000000000000021 instance= ns= 2023-05-31T20:38:23.198 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:23.179+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:11]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:11]: syncing object: pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:11/obj00000000000000000021 2023-05-31T20:38:23.199 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:23.179+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:11]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:11]: entry->timestamp=2023-05-31T20:38:15.471618+0000 2023-05-31T20:38:23.199 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:23.179+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:11]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:11]:entry[obj00000000000000000021]: bucket sync single entry (source_zone=3d163b1e-09c7-47e2-ae87-add4aace3f38) b=bucket sync single entry (source_zone=3d163b1e-09c7-47e2-ae87-add4aace3f38) b= log_entry=00000000001.120.6 op=0 op_state=1 2023-05-31T20:38:23.199 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:23.179+0000 7f22572d6780 0 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:11]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:11]:entry[obj00000000000000000021]: : injecting data sync error on key=obj00000000000000000021 2023-05-31T20:38:23.199 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:23.179+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:11]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:11]: listing bilog for incremental sync; position=00000000001.120.6 2023-05-31T20:38:23.199 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:23.179+0000 7f22572d6780 10 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:11]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:11]:entry[obj00000000000000000021]: failed, retcode=-5 ((5) Input/output error) 2023-05-31T20:38:23.200 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:23.179+0000 7f22572d6780 0 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:11]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:11]:entry[obj00000000000000000021]: ERROR: failed to sync object: pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:11/obj00000000000000000021 2023-05-31T20:38:23.200 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:23.180+0000 7f22572d6780 10 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:4]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:4]: a sync operation returned error 2023-05-31T20:38:23.200 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:23.180+0000 7f22572d6780 10 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:4]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:4]: backing out with sync_status=-5 2023-05-31T20:38:23.200 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:23.180+0000 7f22572d6780 5 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:4]: incremental sync on bucket failed, retcode=-5 2023-05-31T20:38:23.200 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:23.180+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:4]: ERROR: incremental sync failed. error: -5 2023-05-31T20:38:23.200 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:23.182+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:12]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:12]: parsed entry: id=00000000001.48.6 iter->object=obj00000000000000000030 iter->instance= name=obj00000000000000000030 instance= ns= 2023-05-31T20:38:23.200 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:23.182+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:12]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:12]: syncing object: pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:12/obj00000000000000000030 2023-05-31T20:38:23.201 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:23.182+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:12]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:12]: entry->timestamp=2023-05-31T20:38:15.732923+0000 2023-05-31T20:38:23.201 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:23.182+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:12]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:12]:entry[obj00000000000000000030]: bucket sync single entry (source_zone=3d163b1e-09c7-47e2-ae87-add4aace3f38) b=bucket sync single entry (source_zone=3d163b1e-09c7-47e2-ae87-add4aace3f38) b= log_entry=00000000001.48.6 op=0 op_state=1 2023-05-31T20:38:23.201 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:23.182+0000 7f22572d6780 5 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:12]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:12]:entry[obj00000000000000000030]: bucket sync: sync obj: 3d163b1e-09c7-47e2-ae87-add4aace3f38/:pklxkd-58[3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23])/obj00000000000000000030[0] 2023-05-31T20:38:23.201 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:23.182+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:12]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:12]: listing bilog for incremental sync; position=00000000001.48.6 2023-05-31T20:38:23.202 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:23.184+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:4]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-05-31T20:38:23.202 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:23.185+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:4]: sync status for source bucket shard: 0 2023-05-31T20:38:23.202 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:23.185+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:4]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:4]: listing bilog for incremental sync; position= 2023-05-31T20:38:23.203 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:23.193+0000 7f22572d6780 10 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:15]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:15]:entry[obj00000000000000000004]: success 2023-05-31T20:38:23.203 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:23.194+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:15]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:15]: updating marker marker_oid=bucket.sync-status.3d163b1e-09c7-47e2-ae87-add4aace3f38:pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:15:1 marker=00000000002.27.6 timestamp=2023-05-31T20:38:15.837602+0000 2023-05-31T20:38:23.204 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:23.194+0000 7f22572d6780 10 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:6]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:6]:entry[obj00000000000000000032]: success 2023-05-31T20:38:23.204 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:23.194+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:6]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:6]: updating marker marker_oid=bucket.sync-status.3d163b1e-09c7-47e2-ae87-add4aace3f38:pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:6:1 marker=00000000001.52.6 timestamp=2023-05-31T20:38:15.861929+0000 2023-05-31T20:38:23.204 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:23.194+0000 7f22572d6780 10 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:8]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:8]:entry[obj00000000000000000010]: success 2023-05-31T20:38:23.204 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:23.194+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:8]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:8]: updating marker marker_oid=bucket.sync-status.3d163b1e-09c7-47e2-ae87-add4aace3f38:pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:8:1 marker=00000000002.50.6 timestamp=2023-05-31T20:38:15.777030+0000 2023-05-31T20:38:23.204 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:23.195+0000 7f22572d6780 10 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:2]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:2]:entry[obj00000000000000000024]: success 2023-05-31T20:38:23.204 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:23.195+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:2]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:2]: updating marker marker_oid=bucket.sync-status.3d163b1e-09c7-47e2-ae87-add4aace3f38:pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:2:1 marker=00000000001.75.6 timestamp=2023-05-31T20:38:15.375872+0000 2023-05-31T20:38:23.205 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:23.195+0000 7f22572d6780 10 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:10]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:10]:entry[obj00000000000000000013]: success 2023-05-31T20:38:23.205 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:23.195+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:10]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:10]: updating marker marker_oid=bucket.sync-status.3d163b1e-09c7-47e2-ae87-add4aace3f38:pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:10:1 marker=00000000002.38.6 timestamp=2023-05-31T20:38:15.394812+0000 2023-05-31T20:38:23.218 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:23.214+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:0]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:0]: parsed entry: id=00000000001.32.6 iter->object=obj00000000000000000007 iter->instance= name=obj00000000000000000007 instance= ns= 2023-05-31T20:38:23.219 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:23.214+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:0]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:0]: syncing object: pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:0/obj00000000000000000007 2023-05-31T20:38:23.219 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:23.214+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:0]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:0]: entry->timestamp=2023-05-31T20:38:15.650113+0000 2023-05-31T20:38:23.219 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:23.214+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:0]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:0]:entry[obj00000000000000000007]: bucket sync single entry (source_zone=3d163b1e-09c7-47e2-ae87-add4aace3f38) b=bucket sync single entry (source_zone=3d163b1e-09c7-47e2-ae87-add4aace3f38) b= log_entry=00000000001.32.6 op=0 op_state=1 2023-05-31T20:38:23.219 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:23.214+0000 7f22572d6780 5 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:0]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:0]:entry[obj00000000000000000007]: bucket sync: sync obj: 3d163b1e-09c7-47e2-ae87-add4aace3f38/:pklxkd-58[3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23])/obj00000000000000000007[0] 2023-05-31T20:38:23.219 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:23.214+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:0]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:0]: listing bilog for incremental sync; position=00000000001.32.6 2023-05-31T20:38:23.220 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:23.215+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:3]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:3]: parsed entry: id=00000000001.31.6 iter->object=obj00000000000000000020 iter->instance= name=obj00000000000000000020 instance= ns= 2023-05-31T20:38:23.220 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:23.215+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:3]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:3]: syncing object: pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:3/obj00000000000000000020 2023-05-31T20:38:23.221 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:23.215+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:3]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:3]: entry->timestamp=2023-05-31T20:38:15.755225+0000 2023-05-31T20:38:23.221 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:23.215+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:3]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:3]:entry[obj00000000000000000020]: bucket sync single entry (source_zone=3d163b1e-09c7-47e2-ae87-add4aace3f38) b=bucket sync single entry (source_zone=3d163b1e-09c7-47e2-ae87-add4aace3f38) b= log_entry=00000000001.31.6 op=0 op_state=1 2023-05-31T20:38:23.221 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:23.215+0000 7f22572d6780 5 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:3]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:3]:entry[obj00000000000000000020]: bucket sync: sync obj: 3d163b1e-09c7-47e2-ae87-add4aace3f38/:pklxkd-58[3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23])/obj00000000000000000020[0] 2023-05-31T20:38:23.221 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:23.215+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:3]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:3]: listing bilog for incremental sync; position=00000000001.31.6 2023-05-31T20:38:23.221 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:23.216+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:1]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:1]: parsed entry: id=00000000001.45.6 iter->object=obj00000000000000000016 iter->instance= name=obj00000000000000000016 instance= ns= 2023-05-31T20:38:23.221 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:23.216+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:1]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:1]: syncing object: pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:1/obj00000000000000000016 2023-05-31T20:38:23.222 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:23.216+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:1]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:1]: entry->timestamp=2023-05-31T20:38:15.672186+0000 2023-05-31T20:38:23.222 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:23.216+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:1]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:1]:entry[obj00000000000000000016]: bucket sync single entry (source_zone=3d163b1e-09c7-47e2-ae87-add4aace3f38) b=bucket sync single entry (source_zone=3d163b1e-09c7-47e2-ae87-add4aace3f38) b= log_entry=00000000001.45.6 op=0 op_state=1 2023-05-31T20:38:23.222 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:23.216+0000 7f22572d6780 0 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:1]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:1]:entry[obj00000000000000000016]: : injecting data sync error on key=obj00000000000000000016 2023-05-31T20:38:23.222 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:23.216+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:1]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:1]: listing bilog for incremental sync; position=00000000001.45.6 2023-05-31T20:38:23.222 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:23.216+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:5]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:5]: parsed entry: id=00000000002.122.6 iter->object=obj00000000000000000000 iter->instance= name=obj00000000000000000000 instance= ns= 2023-05-31T20:38:23.222 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:23.216+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:5]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:5]: syncing object: pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:5/obj00000000000000000000 2023-05-31T20:38:23.223 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:23.216+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:5]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:5]: entry->timestamp=2023-05-31T20:38:15.559188+0000 2023-05-31T20:38:23.223 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:23.216+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:5]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:5]:entry[obj00000000000000000000]: bucket sync single entry (source_zone=3d163b1e-09c7-47e2-ae87-add4aace3f38) b=bucket sync single entry (source_zone=3d163b1e-09c7-47e2-ae87-add4aace3f38) b= log_entry=00000000002.122.6 op=0 op_state=1 2023-05-31T20:38:23.223 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:23.216+0000 7f22572d6780 10 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:1]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:1]:entry[obj00000000000000000016]: failed, retcode=-5 ((5) Input/output error) 2023-05-31T20:38:23.223 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:23.216+0000 7f22572d6780 0 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:1]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:1]:entry[obj00000000000000000016]: ERROR: failed to sync object: pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:1/obj00000000000000000016 2023-05-31T20:38:23.223 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:23.216+0000 7f22572d6780 5 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:5]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:5]:entry[obj00000000000000000000]: bucket sync: sync obj: 3d163b1e-09c7-47e2-ae87-add4aace3f38/:pklxkd-58[3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23])/obj00000000000000000000[0] 2023-05-31T20:38:23.223 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:23.216+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:5]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:5]: listing bilog for incremental sync; position=00000000002.122.6 2023-05-31T20:38:23.224 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:23.217+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:13]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:13]: parsed entry: id=00000000001.36.6 iter->object=obj00000000000000000026 iter->instance= name=obj00000000000000000026 instance= ns= 2023-05-31T20:38:23.225 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:23.217+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:13]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:13]: syncing object: pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:13/obj00000000000000000026 2023-05-31T20:38:23.225 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:23.217+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:13]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:13]: entry->timestamp=2023-05-31T20:38:15.489620+0000 2023-05-31T20:38:23.225 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:23.217+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:13]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:13]:entry[obj00000000000000000026]: bucket sync single entry (source_zone=3d163b1e-09c7-47e2-ae87-add4aace3f38) b=bucket sync single entry (source_zone=3d163b1e-09c7-47e2-ae87-add4aace3f38) b= log_entry=00000000001.36.6 op=0 op_state=1 2023-05-31T20:38:23.225 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:23.217+0000 7f22572d6780 0 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:13]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:13]:entry[obj00000000000000000026]: : injecting data sync error on key=obj00000000000000000026 2023-05-31T20:38:23.225 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:23.217+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:13]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:13]: listing bilog for incremental sync; position=00000000001.36.6 2023-05-31T20:38:23.225 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:23.217+0000 7f22572d6780 10 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:13]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:13]:entry[obj00000000000000000026]: failed, retcode=-5 ((5) Input/output error) 2023-05-31T20:38:23.226 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:23.217+0000 7f22572d6780 0 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:13]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:13]:entry[obj00000000000000000026]: ERROR: failed to sync object: pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:13/obj00000000000000000026 2023-05-31T20:38:23.226 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:23.218+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:7]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:7]: parsed entry: id=00000000002.38.6 iter->object=obj00000000000000000001 iter->instance= name=obj00000000000000000001 instance= ns= 2023-05-31T20:38:23.226 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:23.218+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:7]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:7]: syncing object: pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:7/obj00000000000000000001 2023-05-31T20:38:23.226 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:23.218+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:7]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:7]: entry->timestamp=2023-05-31T20:38:15.583844+0000 2023-05-31T20:38:23.226 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:23.218+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:7]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:7]:entry[obj00000000000000000001]: bucket sync single entry (source_zone=3d163b1e-09c7-47e2-ae87-add4aace3f38) b=bucket sync single entry (source_zone=3d163b1e-09c7-47e2-ae87-add4aace3f38) b= log_entry=00000000002.38.6 op=0 op_state=1 2023-05-31T20:38:23.226 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:23.218+0000 7f22572d6780 5 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:7]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:7]:entry[obj00000000000000000001]: bucket sync: sync obj: 3d163b1e-09c7-47e2-ae87-add4aace3f38/:pklxkd-58[3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23])/obj00000000000000000001[0] 2023-05-31T20:38:23.227 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:23.218+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:7]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:7]: listing bilog for incremental sync; position=00000000002.38.6 2023-05-31T20:38:23.227 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:23.221+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:14]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:14]: parsed entry: id=00000000001.41.6 iter->object=obj00000000000000000033 iter->instance= name=obj00000000000000000033 instance= ns= 2023-05-31T20:38:23.227 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:23.221+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:14]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:14]: syncing object: pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:14/obj00000000000000000033 2023-05-31T20:38:23.228 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:23.221+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:14]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:14]: entry->timestamp=2023-05-31T20:38:15.295616+0000 2023-05-31T20:38:23.228 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:23.221+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:14]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:14]:entry[obj00000000000000000033]: bucket sync single entry (source_zone=3d163b1e-09c7-47e2-ae87-add4aace3f38) b=bucket sync single entry (source_zone=3d163b1e-09c7-47e2-ae87-add4aace3f38) b= log_entry=00000000001.41.6 op=0 op_state=1 2023-05-31T20:38:23.228 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:23.221+0000 7f22572d6780 5 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:14]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:14]:entry[obj00000000000000000033]: bucket sync: sync obj: 3d163b1e-09c7-47e2-ae87-add4aace3f38/:pklxkd-58[3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23])/obj00000000000000000033[0] 2023-05-31T20:38:23.228 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:23.221+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:14]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:14]: listing bilog for incremental sync; position=00000000001.41.6 2023-05-31T20:38:23.229 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:23.224+0000 7f22572d6780 10 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:11]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:11]: a sync operation returned error 2023-05-31T20:38:23.229 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:23.224+0000 7f22572d6780 10 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:11]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:11]: backing out with sync_status=-5 2023-05-31T20:38:23.229 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:23.224+0000 7f22572d6780 5 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:11]: incremental sync on bucket failed, retcode=-5 2023-05-31T20:38:23.229 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:23.224+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:11]: ERROR: incremental sync failed. error: -5 2023-05-31T20:38:23.231 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:23.227+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:4]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:4]: parsed entry: id=00000000001.43.6 iter->object=obj00000000000000000028 iter->instance= name=obj00000000000000000028 instance= ns= 2023-05-31T20:38:23.231 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:23.227+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:4]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:4]: syncing object: pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:4/obj00000000000000000028 2023-05-31T20:38:23.231 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:23.227+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:4]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:4]: entry->timestamp=2023-05-31T20:38:15.316632+0000 2023-05-31T20:38:23.231 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:23.227+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:4]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:4]:entry[obj00000000000000000028]: bucket sync single entry (source_zone=3d163b1e-09c7-47e2-ae87-add4aace3f38) b=bucket sync single entry (source_zone=3d163b1e-09c7-47e2-ae87-add4aace3f38) b= log_entry=00000000001.43.6 op=0 op_state=1 2023-05-31T20:38:23.231 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:23.227+0000 7f22572d6780 0 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:4]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:4]:entry[obj00000000000000000028]: : injecting data sync error on key=obj00000000000000000028 2023-05-31T20:38:23.232 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:23.227+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:4]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:4]: listing bilog for incremental sync; position=00000000001.43.6 2023-05-31T20:38:23.232 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:23.227+0000 7f22572d6780 10 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:4]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:4]:entry[obj00000000000000000028]: failed, retcode=-5 ((5) Input/output error) 2023-05-31T20:38:23.232 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:23.227+0000 7f22572d6780 0 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:4]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:4]:entry[obj00000000000000000028]: ERROR: failed to sync object: pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:4/obj00000000000000000028 2023-05-31T20:38:23.233 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:23.229+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:11]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-05-31T20:38:23.234 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:23.229+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:11]: sync status for source bucket shard: 0 2023-05-31T20:38:23.234 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:23.229+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:11]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:11]: listing bilog for incremental sync; position= 2023-05-31T20:38:23.241 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:23.237+0000 7f22572d6780 10 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:7]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:7]:entry[obj00000000000000000001]: success 2023-05-31T20:38:23.242 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:23.237+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:7]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:7]: updating marker marker_oid=bucket.sync-status.3d163b1e-09c7-47e2-ae87-add4aace3f38:pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:7:1 marker=00000000002.38.6 timestamp=2023-05-31T20:38:15.583844+0000 2023-05-31T20:38:23.242 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:23.237+0000 7f22572d6780 10 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:12]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:12]:entry[obj00000000000000000030]: success 2023-05-31T20:38:23.242 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:23.237+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:12]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:12]: updating marker marker_oid=bucket.sync-status.3d163b1e-09c7-47e2-ae87-add4aace3f38:pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:12:1 marker=00000000001.48.6 timestamp=2023-05-31T20:38:15.732923+0000 2023-05-31T20:38:23.242 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:23.237+0000 7f22572d6780 10 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:9]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:9]:entry[obj00000000000000000019]: success 2023-05-31T20:38:23.242 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:23.237+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:9]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:9]: updating marker marker_oid=bucket.sync-status.3d163b1e-09c7-47e2-ae87-add4aace3f38:pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:9:1 marker=00000000001.118.6 timestamp=2023-05-31T20:38:15.414071+0000 2023-05-31T20:38:23.243 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:23.239+0000 7f22572d6780 10 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:14]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:14]:entry[obj00000000000000000033]: success 2023-05-31T20:38:23.244 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:23.239+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:14]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:14]: updating marker marker_oid=bucket.sync-status.3d163b1e-09c7-47e2-ae87-add4aace3f38:pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:14:1 marker=00000000001.41.6 timestamp=2023-05-31T20:38:15.295616+0000 2023-05-31T20:38:23.262 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:23.258+0000 7f22572d6780 10 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:1]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:1]: a sync operation returned error 2023-05-31T20:38:23.262 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:23.258+0000 7f22572d6780 10 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:1]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:1]: backing out with sync_status=-5 2023-05-31T20:38:23.262 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:23.258+0000 7f22572d6780 5 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:1]: incremental sync on bucket failed, retcode=-5 2023-05-31T20:38:23.263 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:23.258+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:1]: ERROR: incremental sync failed. error: -5 2023-05-31T20:38:23.264 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:23.258+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:1]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-05-31T20:38:23.264 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:23.258+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:1]: sync status for source bucket shard: 0 2023-05-31T20:38:23.264 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:23.258+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:1]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:1]: listing bilog for incremental sync; position= 2023-05-31T20:38:23.265 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:23.259+0000 7f22572d6780 10 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:13]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:13]: a sync operation returned error 2023-05-31T20:38:23.265 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:23.259+0000 7f22572d6780 10 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:13]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:13]: backing out with sync_status=-5 2023-05-31T20:38:23.265 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:23.259+0000 7f22572d6780 5 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:13]: incremental sync on bucket failed, retcode=-5 2023-05-31T20:38:23.265 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:23.259+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:13]: ERROR: incremental sync failed. error: -5 2023-05-31T20:38:23.266 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:23.259+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:13]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-05-31T20:38:23.266 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:23.259+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:13]: sync status for source bucket shard: 0 2023-05-31T20:38:23.266 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:23.259+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:13]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:13]: listing bilog for incremental sync; position= 2023-05-31T20:38:23.273 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:23.269+0000 7f22572d6780 10 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:4]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:4]: a sync operation returned error 2023-05-31T20:38:23.273 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:23.269+0000 7f22572d6780 10 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:4]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:4]: backing out with sync_status=-5 2023-05-31T20:38:23.274 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:23.269+0000 7f22572d6780 5 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:4]: incremental sync on bucket failed, retcode=-5 2023-05-31T20:38:23.274 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:23.269+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:4]: ERROR: incremental sync failed. error: -5 2023-05-31T20:38:23.275 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:23.269+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:4]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-05-31T20:38:23.275 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:23.270+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:4]: sync status for source bucket shard: 0 2023-05-31T20:38:23.275 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:23.270+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:4]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:4]: listing bilog for incremental sync; position= 2023-05-31T20:38:23.277 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:23.273+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:11]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:11]: parsed entry: id=00000000001.120.6 iter->object=obj00000000000000000021 iter->instance= name=obj00000000000000000021 instance= ns= 2023-05-31T20:38:23.277 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:23.273+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:11]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:11]: syncing object: pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:11/obj00000000000000000021 2023-05-31T20:38:23.278 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:23.273+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:11]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:11]: entry->timestamp=2023-05-31T20:38:15.471618+0000 2023-05-31T20:38:23.278 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:23.273+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:11]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:11]:entry[obj00000000000000000021]: bucket sync single entry (source_zone=3d163b1e-09c7-47e2-ae87-add4aace3f38) b=bucket sync single entry (source_zone=3d163b1e-09c7-47e2-ae87-add4aace3f38) b= log_entry=00000000001.120.6 op=0 op_state=1 2023-05-31T20:38:23.278 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:23.273+0000 7f22572d6780 5 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:11]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:11]:entry[obj00000000000000000021]: bucket sync: sync obj: 3d163b1e-09c7-47e2-ae87-add4aace3f38/:pklxkd-58[3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23])/obj00000000000000000021[0] 2023-05-31T20:38:23.279 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:23.273+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:11]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:11]: listing bilog for incremental sync; position=00000000001.120.6 2023-05-31T20:38:23.279 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:23.275+0000 7f22572d6780 10 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:0]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:0]:entry[obj00000000000000000007]: success 2023-05-31T20:38:23.280 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:23.275+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:0]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:0]: updating marker marker_oid=bucket.sync-status.3d163b1e-09c7-47e2-ae87-add4aace3f38:pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:0:1 marker=00000000001.32.6 timestamp=2023-05-31T20:38:15.650113+0000 2023-05-31T20:38:23.280 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:23.275+0000 7f22572d6780 10 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:5]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:5]:entry[obj00000000000000000000]: success 2023-05-31T20:38:23.280 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:23.275+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:5]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:5]: updating marker marker_oid=bucket.sync-status.3d163b1e-09c7-47e2-ae87-add4aace3f38:pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:5:1 marker=00000000002.122.6 timestamp=2023-05-31T20:38:15.559188+0000 2023-05-31T20:38:23.284 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:23.280+0000 7f22572d6780 10 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:3]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:3]:entry[obj00000000000000000020]: success 2023-05-31T20:38:23.284 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:23.280+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:3]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:3]: updating marker marker_oid=bucket.sync-status.3d163b1e-09c7-47e2-ae87-add4aace3f38:pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:3:1 marker=00000000001.31.6 timestamp=2023-05-31T20:38:15.755225+0000 2023-05-31T20:38:23.311 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:23.301+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:1]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:1]: parsed entry: id=00000000001.45.6 iter->object=obj00000000000000000016 iter->instance= name=obj00000000000000000016 instance= ns= 2023-05-31T20:38:23.312 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:23.301+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:1]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:1]: syncing object: pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:1/obj00000000000000000016 2023-05-31T20:38:23.312 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:23.301+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:1]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:1]: entry->timestamp=2023-05-31T20:38:15.672186+0000 2023-05-31T20:38:23.312 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:23.301+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:1]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:1]:entry[obj00000000000000000016]: bucket sync single entry (source_zone=3d163b1e-09c7-47e2-ae87-add4aace3f38) b=bucket sync single entry (source_zone=3d163b1e-09c7-47e2-ae87-add4aace3f38) b= log_entry=00000000001.45.6 op=0 op_state=1 2023-05-31T20:38:23.312 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:23.301+0000 7f22572d6780 5 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:1]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:1]:entry[obj00000000000000000016]: bucket sync: sync obj: 3d163b1e-09c7-47e2-ae87-add4aace3f38/:pklxkd-58[3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23])/obj00000000000000000016[0] 2023-05-31T20:38:23.313 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:23.301+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:1]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:1]: listing bilog for incremental sync; position=00000000001.45.6 2023-05-31T20:38:23.313 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:23.301+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:13]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:13]: parsed entry: id=00000000001.36.6 iter->object=obj00000000000000000026 iter->instance= name=obj00000000000000000026 instance= ns= 2023-05-31T20:38:23.313 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:23.301+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:13]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:13]: syncing object: pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:13/obj00000000000000000026 2023-05-31T20:38:23.313 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:23.301+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:13]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:13]: entry->timestamp=2023-05-31T20:38:15.489620+0000 2023-05-31T20:38:23.313 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:23.301+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:13]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:13]:entry[obj00000000000000000026]: bucket sync single entry (source_zone=3d163b1e-09c7-47e2-ae87-add4aace3f38) b=bucket sync single entry (source_zone=3d163b1e-09c7-47e2-ae87-add4aace3f38) b= log_entry=00000000001.36.6 op=0 op_state=1 2023-05-31T20:38:23.314 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:23.301+0000 7f22572d6780 5 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:13]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:13]:entry[obj00000000000000000026]: bucket sync: sync obj: 3d163b1e-09c7-47e2-ae87-add4aace3f38/:pklxkd-58[3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23])/obj00000000000000000026[0] 2023-05-31T20:38:23.314 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:23.301+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:13]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:13]: listing bilog for incremental sync; position=00000000001.36.6 2023-05-31T20:38:23.314 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:23.308+0000 7f22572d6780 10 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:13]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:13]:entry[obj00000000000000000026]: success 2023-05-31T20:38:23.315 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:23.308+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:13]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:13]: updating marker marker_oid=bucket.sync-status.3d163b1e-09c7-47e2-ae87-add4aace3f38:pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:13:1 marker=00000000001.36.6 timestamp=2023-05-31T20:38:15.489620+0000 2023-05-31T20:38:23.316 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:23.312+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:4]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:4]: parsed entry: id=00000000001.43.6 iter->object=obj00000000000000000028 iter->instance= name=obj00000000000000000028 instance= ns= 2023-05-31T20:38:23.317 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:23.312+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:4]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:4]: syncing object: pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:4/obj00000000000000000028 2023-05-31T20:38:23.317 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:23.312+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:4]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:4]: entry->timestamp=2023-05-31T20:38:15.316632+0000 2023-05-31T20:38:23.317 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:23.312+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:4]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:4]:entry[obj00000000000000000028]: bucket sync single entry (source_zone=3d163b1e-09c7-47e2-ae87-add4aace3f38) b=bucket sync single entry (source_zone=3d163b1e-09c7-47e2-ae87-add4aace3f38) b= log_entry=00000000001.43.6 op=0 op_state=1 2023-05-31T20:38:23.317 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:23.312+0000 7f22572d6780 5 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:4]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:4]:entry[obj00000000000000000028]: bucket sync: sync obj: 3d163b1e-09c7-47e2-ae87-add4aace3f38/:pklxkd-58[3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23])/obj00000000000000000028[0] 2023-05-31T20:38:23.318 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:23.312+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:4]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:4]: listing bilog for incremental sync; position=00000000001.43.6 2023-05-31T20:38:23.318 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:23.313+0000 7f22572d6780 10 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:1]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:1]:entry[obj00000000000000000016]: success 2023-05-31T20:38:23.318 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:23.313+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:1]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:1]: updating marker marker_oid=bucket.sync-status.3d163b1e-09c7-47e2-ae87-add4aace3f38:pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:1:1 marker=00000000001.45.6 timestamp=2023-05-31T20:38:15.672186+0000 2023-05-31T20:38:23.323 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:23.319+0000 7f22572d6780 10 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:11]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:11]:entry[obj00000000000000000021]: success 2023-05-31T20:38:23.324 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:23.319+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:11]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:11]: updating marker marker_oid=bucket.sync-status.3d163b1e-09c7-47e2-ae87-add4aace3f38:pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:11:1 marker=00000000001.120.6 timestamp=2023-05-31T20:38:15.471618+0000 2023-05-31T20:38:23.362 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:23.357+0000 7f22572d6780 10 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:4]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:4]:entry[obj00000000000000000028]: success 2023-05-31T20:38:23.362 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:23.357+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:4]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:4]: updating marker marker_oid=bucket.sync-status.3d163b1e-09c7-47e2-ae87-add4aace3f38:pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:4:1 marker=00000000001.43.6 timestamp=2023-05-31T20:38:15.316632+0000 2023-05-31T20:38:23.373 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:23.368+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:0]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-05-31T20:38:23.374 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:23.368+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:4]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-05-31T20:38:23.374 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:23.368+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:2]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-05-31T20:38:23.374 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:23.369+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:4]: sync status for source bucket shard: 0 2023-05-31T20:38:23.374 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:23.369+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:4]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:4]: listing bilog for incremental sync; position= 2023-05-31T20:38:23.374 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:23.369+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:1]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-05-31T20:38:23.375 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:23.369+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:2]: sync status for source bucket shard: 0 2023-05-31T20:38:23.375 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:23.369+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:2]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:2]: listing bilog for incremental sync; position= 2023-05-31T20:38:23.375 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:23.369+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:3]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-05-31T20:38:23.375 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:23.369+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:6]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-05-31T20:38:23.375 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:23.369+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:3]: sync status for source bucket shard: 0 2023-05-31T20:38:23.376 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:23.369+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:3]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:3]: listing bilog for incremental sync; position= 2023-05-31T20:38:23.376 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:23.369+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:6]: sync status for source bucket shard: 0 2023-05-31T20:38:23.376 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:23.369+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:6]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:6]: listing bilog for incremental sync; position= 2023-05-31T20:38:23.376 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:23.369+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:7]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-05-31T20:38:23.384 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:23.370+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:5]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-05-31T20:38:23.384 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:23.370+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:8]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-05-31T20:38:23.385 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:23.370+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:9]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-05-31T20:38:23.385 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:23.370+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:7]: sync status for source bucket shard: 0 2023-05-31T20:38:23.385 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:23.370+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:7]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:7]: listing bilog for incremental sync; position= 2023-05-31T20:38:23.385 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:23.370+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:10]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-05-31T20:38:23.386 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:23.371+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:5]: sync status for source bucket shard: 0 2023-05-31T20:38:23.386 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:23.371+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:8]: sync status for source bucket shard: 0 2023-05-31T20:38:23.386 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:23.371+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:9]: sync status for source bucket shard: 0 2023-05-31T20:38:23.387 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:23.371+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:5]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:5]: listing bilog for incremental sync; position= 2023-05-31T20:38:23.387 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:23.371+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:8]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:8]: listing bilog for incremental sync; position= 2023-05-31T20:38:23.387 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:23.371+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:9]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:9]: listing bilog for incremental sync; position= 2023-05-31T20:38:23.388 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:23.371+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:11]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-05-31T20:38:23.388 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:23.371+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:12]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-05-31T20:38:23.388 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:23.371+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:10]: sync status for source bucket shard: 0 2023-05-31T20:38:23.389 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:23.371+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:10]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:10]: listing bilog for incremental sync; position= 2023-05-31T20:38:23.389 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:23.371+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:14]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-05-31T20:38:23.389 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:23.371+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:13]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-05-31T20:38:23.391 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:23.371+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:15]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-05-31T20:38:23.391 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:23.371+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:11]: sync status for source bucket shard: 0 2023-05-31T20:38:23.392 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:23.371+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:11]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:11]: listing bilog for incremental sync; position= 2023-05-31T20:38:23.392 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:23.371+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:16]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-05-31T20:38:23.393 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:23.371+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:14]: sync status for source bucket shard: 0 2023-05-31T20:38:23.393 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:23.371+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:14]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:14]: listing bilog for incremental sync; position= 2023-05-31T20:38:23.393 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:23.372+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:12]: sync status for source bucket shard: 0 2023-05-31T20:38:23.394 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:23.372+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:15]: sync status for source bucket shard: 0 2023-05-31T20:38:23.394 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:23.372+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:13]: sync status for source bucket shard: 0 2023-05-31T20:38:23.394 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:23.372+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:12]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:12]: listing bilog for incremental sync; position= 2023-05-31T20:38:23.394 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:23.372+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:15]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:15]: listing bilog for incremental sync; position= 2023-05-31T20:38:23.395 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:23.372+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:13]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:13]: listing bilog for incremental sync; position= 2023-05-31T20:38:23.395 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:23.372+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:6]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:6]: parsed entry: id=00000000002.76.6 iter->object=obj00000000000000000030 iter->instance= name=obj00000000000000000030 instance= ns= 2023-05-31T20:38:23.396 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:23.372+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:6]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:6]: syncing object: pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:6/obj00000000000000000030 2023-05-31T20:38:23.396 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:23.372+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:6]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:6]: entry->timestamp=2023-05-31T20:38:16.319144+0000 2023-05-31T20:38:23.396 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:23.372+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:6]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:6]:entry[obj00000000000000000030]: bucket sync single entry (source_zone=3d163b1e-09c7-47e2-ae87-add4aace3f38) b=bucket sync single entry (source_zone=3d163b1e-09c7-47e2-ae87-add4aace3f38) b= log_entry=00000000002.76.6 op=0 op_state=1 2023-05-31T20:38:23.396 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:23.372+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:0]: sync status for source bucket shard: 0 2023-05-31T20:38:23.397 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:23.372+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:17]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-05-31T20:38:23.398 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:23.372+0000 7f22572d6780 5 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:6]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:6]:entry[obj00000000000000000030]: bucket sync: sync obj: 3d163b1e-09c7-47e2-ae87-add4aace3f38/:pklxkd-58[3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23])/obj00000000000000000030[0] 2023-05-31T20:38:23.398 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:23.372+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:6]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:6]: listing bilog for incremental sync; position=00000000002.76.6 2023-05-31T20:38:23.398 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:23.372+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:0]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:0]: listing bilog for incremental sync; position= 2023-05-31T20:38:23.399 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:23.372+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:18]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-05-31T20:38:23.399 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:23.374+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:16]: sync status for source bucket shard: 0 2023-05-31T20:38:23.399 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:23.374+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:16]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:16]: listing bilog for incremental sync; position= 2023-05-31T20:38:23.399 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:23.374+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:8]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:8]: parsed entry: id=00000000002.129.6 iter->object=obj00000000000000000019 iter->instance= name=obj00000000000000000019 instance= ns= 2023-05-31T20:38:23.400 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:23.374+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:8]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:8]: syncing object: pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:8/obj00000000000000000019 2023-05-31T20:38:23.400 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:23.374+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:8]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:8]: entry->timestamp=2023-05-31T20:38:16.482523+0000 2023-05-31T20:38:23.400 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:23.374+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:8]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:8]:entry[obj00000000000000000019]: bucket sync single entry (source_zone=3d163b1e-09c7-47e2-ae87-add4aace3f38) b=bucket sync single entry (source_zone=3d163b1e-09c7-47e2-ae87-add4aace3f38) b= log_entry=00000000002.129.6 op=0 op_state=1 2023-05-31T20:38:23.401 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:23.374+0000 7f22572d6780 5 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:8]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:8]:entry[obj00000000000000000019]: bucket sync: sync obj: 3d163b1e-09c7-47e2-ae87-add4aace3f38/:pklxkd-58[3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23])/obj00000000000000000019[0] 2023-05-31T20:38:23.401 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:23.374+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:8]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:8]: listing bilog for incremental sync; position=00000000002.129.6 2023-05-31T20:38:23.402 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:23.374+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:10]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:10]: parsed entry: id=00000000001.59.6 iter->object=obj00000000000000000009 iter->instance= name=obj00000000000000000009 instance= ns= 2023-05-31T20:38:23.402 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:23.374+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:10]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:10]: syncing object: pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:10/obj00000000000000000009 2023-05-31T20:38:23.402 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:23.374+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:10]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:10]: entry->timestamp=2023-05-31T20:38:16.655883+0000 2023-05-31T20:38:23.403 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:23.374+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:10]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:10]:entry[obj00000000000000000009]: bucket sync single entry (source_zone=3d163b1e-09c7-47e2-ae87-add4aace3f38) b=bucket sync single entry (source_zone=3d163b1e-09c7-47e2-ae87-add4aace3f38) b= log_entry=00000000001.59.6 op=0 op_state=1 2023-05-31T20:38:23.403 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:23.374+0000 7f22572d6780 5 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:10]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:10]:entry[obj00000000000000000009]: bucket sync: sync obj: 3d163b1e-09c7-47e2-ae87-add4aace3f38/:pklxkd-58[3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23])/obj00000000000000000009[0] 2023-05-31T20:38:23.403 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:23.374+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:10]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:10]: listing bilog for incremental sync; position=00000000001.59.6 2023-05-31T20:38:23.403 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:23.374+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:11]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:11]: parsed entry: id=00000000001.80.6 iter->object=obj00000000000000000014 iter->instance= name=obj00000000000000000014 instance= ns= 2023-05-31T20:38:23.403 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:23.374+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:11]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:11]: syncing object: pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:11/obj00000000000000000014 2023-05-31T20:38:23.403 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:23.374+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:11]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:11]: entry->timestamp=2023-05-31T20:38:16.744677+0000 2023-05-31T20:38:23.404 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:23.374+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:11]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:11]:entry[obj00000000000000000014]: bucket sync single entry (source_zone=3d163b1e-09c7-47e2-ae87-add4aace3f38) b=bucket sync single entry (source_zone=3d163b1e-09c7-47e2-ae87-add4aace3f38) b= log_entry=00000000001.80.6 op=0 op_state=1 2023-05-31T20:38:23.404 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:23.374+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:12]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:12]: parsed entry: id=00000000002.37.6 iter->object=obj00000000000000000012 iter->instance= name=obj00000000000000000012 instance= ns= 2023-05-31T20:38:23.404 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:23.374+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:12]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:12]: syncing object: pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:12/obj00000000000000000012 2023-05-31T20:38:23.404 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:23.374+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:12]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:12]: entry->timestamp=2023-05-31T20:38:16.377712+0000 2023-05-31T20:38:23.404 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:23.374+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:12]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:12]:entry[obj00000000000000000012]: bucket sync single entry (source_zone=3d163b1e-09c7-47e2-ae87-add4aace3f38) b=bucket sync single entry (source_zone=3d163b1e-09c7-47e2-ae87-add4aace3f38) b= log_entry=00000000002.37.6 op=0 op_state=1 2023-05-31T20:38:23.404 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:23.374+0000 7f22572d6780 5 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:11]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:11]:entry[obj00000000000000000014]: bucket sync: sync obj: 3d163b1e-09c7-47e2-ae87-add4aace3f38/:pklxkd-58[3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23])/obj00000000000000000014[0] 2023-05-31T20:38:23.405 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:23.374+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:11]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:11]: listing bilog for incremental sync; position=00000000001.80.6 2023-05-31T20:38:23.405 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:23.374+0000 7f22572d6780 5 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:12]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:12]:entry[obj00000000000000000012]: bucket sync: sync obj: 3d163b1e-09c7-47e2-ae87-add4aace3f38/:pklxkd-58[3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23])/obj00000000000000000012[0] 2023-05-31T20:38:23.406 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:23.374+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:12]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:12]: listing bilog for incremental sync; position=00000000002.37.6 2023-05-31T20:38:23.406 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:23.374+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:5]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:5]: parsed entry: id=00000000001.40.6 iter->object=obj00000000000000000025 iter->instance= name=obj00000000000000000025 instance= ns= 2023-05-31T20:38:23.406 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:23.374+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:5]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:5]: syncing object: pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:5/obj00000000000000000025 2023-05-31T20:38:23.406 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:23.374+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:5]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:5]: entry->timestamp=2023-05-31T20:38:16.505818+0000 2023-05-31T20:38:23.406 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:23.374+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:5]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:5]:entry[obj00000000000000000025]: bucket sync single entry (source_zone=3d163b1e-09c7-47e2-ae87-add4aace3f38) b=bucket sync single entry (source_zone=3d163b1e-09c7-47e2-ae87-add4aace3f38) b= log_entry=00000000001.40.6 op=0 op_state=1 2023-05-31T20:38:23.407 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:23.374+0000 7f22572d6780 5 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:5]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:5]:entry[obj00000000000000000025]: bucket sync: sync obj: 3d163b1e-09c7-47e2-ae87-add4aace3f38/:pklxkd-58[3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23])/obj00000000000000000025[0] 2023-05-31T20:38:23.407 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:23.374+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:5]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:5]: listing bilog for incremental sync; position=00000000001.40.6 2023-05-31T20:38:23.407 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:23.375+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:17]: sync status for source bucket shard: 0 2023-05-31T20:38:23.407 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:23.375+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:17]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:17]: listing bilog for incremental sync; position= 2023-05-31T20:38:23.407 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:23.375+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:0]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:0]: parsed entry: id=00000000002.78.6 iter->object=obj00000000000000000005 iter->instance= name=obj00000000000000000005 instance= ns= 2023-05-31T20:38:23.407 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:23.375+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:0]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:0]: syncing object: pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:0/obj00000000000000000005 2023-05-31T20:38:23.408 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:23.375+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:0]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:0]: entry->timestamp=2023-05-31T20:38:16.597889+0000 2023-05-31T20:38:23.408 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:23.375+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:0]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:0]:entry[obj00000000000000000005]: bucket sync single entry (source_zone=3d163b1e-09c7-47e2-ae87-add4aace3f38) b=bucket sync single entry (source_zone=3d163b1e-09c7-47e2-ae87-add4aace3f38) b= log_entry=00000000002.78.6 op=0 op_state=1 2023-05-31T20:38:23.408 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:23.375+0000 7f22572d6780 5 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:0]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:0]:entry[obj00000000000000000005]: bucket sync: sync obj: 3d163b1e-09c7-47e2-ae87-add4aace3f38/:pklxkd-58[3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23])/obj00000000000000000005[0] 2023-05-31T20:38:23.408 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:23.375+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:0]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:0]: listing bilog for incremental sync; position=00000000002.78.6 2023-05-31T20:38:23.408 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:23.375+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:9]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:9]: parsed entry: id=00000000002.42.6 iter->object=obj00000000000000000003 iter->instance= name=obj00000000000000000003 instance= ns= 2023-05-31T20:38:23.408 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:23.375+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:9]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:9]: syncing object: pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:9/obj00000000000000000003 2023-05-31T20:38:23.409 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:23.375+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:9]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:9]: entry->timestamp=2023-05-31T20:38:16.680803+0000 2023-05-31T20:38:23.409 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:23.375+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:9]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:9]:entry[obj00000000000000000003]: bucket sync single entry (source_zone=3d163b1e-09c7-47e2-ae87-add4aace3f38) b=bucket sync single entry (source_zone=3d163b1e-09c7-47e2-ae87-add4aace3f38) b= log_entry=00000000002.42.6 op=0 op_state=1 2023-05-31T20:38:23.409 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:23.375+0000 7f22572d6780 5 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:9]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:9]:entry[obj00000000000000000003]: bucket sync: sync obj: 3d163b1e-09c7-47e2-ae87-add4aace3f38/:pklxkd-58[3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23])/obj00000000000000000003[0] 2023-05-31T20:38:23.409 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:23.375+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:9]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:9]: listing bilog for incremental sync; position=00000000002.42.6 2023-05-31T20:38:23.409 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:23.375+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:1]: sync status for source bucket shard: 0 2023-05-31T20:38:23.409 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:23.375+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:1]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:1]: listing bilog for incremental sync; position= 2023-05-31T20:38:23.409 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:23.376+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:18]: sync status for source bucket shard: 0 2023-05-31T20:38:23.410 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:23.376+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:15]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:15]: parsed entry: id=00000000002.63.6 iter->object=obj00000000000000000006 iter->instance= name=obj00000000000000000006 instance= ns= 2023-05-31T20:38:23.410 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:23.376+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:15]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:15]: syncing object: pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:15/obj00000000000000000006 2023-05-31T20:38:23.410 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:23.376+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:15]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:15]: entry->timestamp=2023-05-31T20:38:16.771109+0000 2023-05-31T20:38:23.410 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:23.376+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:15]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:15]:entry[obj00000000000000000006]: bucket sync single entry (source_zone=3d163b1e-09c7-47e2-ae87-add4aace3f38) b=bucket sync single entry (source_zone=3d163b1e-09c7-47e2-ae87-add4aace3f38) b= log_entry=00000000002.63.6 op=0 op_state=1 2023-05-31T20:38:23.410 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:23.376+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:18]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:18]: listing bilog for incremental sync; position= 2023-05-31T20:38:23.410 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:23.376+0000 7f22572d6780 5 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:15]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:15]:entry[obj00000000000000000006]: bucket sync: sync obj: 3d163b1e-09c7-47e2-ae87-add4aace3f38/:pklxkd-58[3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23])/obj00000000000000000006[0] 2023-05-31T20:38:23.411 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:23.376+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:15]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:15]: listing bilog for incremental sync; position=00000000002.63.6 2023-05-31T20:38:23.411 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:23.376+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:14]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:14]: parsed entry: id=00000000002.46.6 iter->object=obj00000000000000000000 iter->instance= name=obj00000000000000000000 instance= ns= 2023-05-31T20:38:23.411 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:23.376+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:14]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:14]: syncing object: pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:14/obj00000000000000000000 2023-05-31T20:38:23.411 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:23.376+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:14]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:14]: entry->timestamp=2023-05-31T20:38:16.835308+0000 2023-05-31T20:38:23.411 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:23.376+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:14]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:14]:entry[obj00000000000000000000]: bucket sync single entry (source_zone=3d163b1e-09c7-47e2-ae87-add4aace3f38) b=bucket sync single entry (source_zone=3d163b1e-09c7-47e2-ae87-add4aace3f38) b= log_entry=00000000002.46.6 op=0 op_state=1 2023-05-31T20:38:23.412 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:23.376+0000 7f22572d6780 5 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:14]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:14]:entry[obj00000000000000000000]: bucket sync: sync obj: 3d163b1e-09c7-47e2-ae87-add4aace3f38/:pklxkd-58[3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23])/obj00000000000000000000[0] 2023-05-31T20:38:23.412 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:23.376+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:14]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:14]: listing bilog for incremental sync; position=00000000002.46.6 2023-05-31T20:38:23.413 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:23.380+0000 7f22572d6780 10 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:6]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:6]:entry[obj00000000000000000030]: success 2023-05-31T20:38:23.413 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:23.380+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:6]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:6]: updating marker marker_oid=bucket.sync-status.3d163b1e-09c7-47e2-ae87-add4aace3f38:pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:6:2 marker=00000000002.76.6 timestamp=2023-05-31T20:38:16.319144+0000 2023-05-31T20:38:23.414 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:23.382+0000 7f22572d6780 10 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:8]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:8]:entry[obj00000000000000000019]: success 2023-05-31T20:38:23.415 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:23.383+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:8]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:8]: updating marker marker_oid=bucket.sync-status.3d163b1e-09c7-47e2-ae87-add4aace3f38:pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:8:2 marker=00000000002.129.6 timestamp=2023-05-31T20:38:16.482523+0000 2023-05-31T20:38:23.415 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:23.383+0000 7f22572d6780 10 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:14]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:14]:entry[obj00000000000000000000]: success 2023-05-31T20:38:23.415 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:23.383+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:14]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:14]: updating marker marker_oid=bucket.sync-status.3d163b1e-09c7-47e2-ae87-add4aace3f38:pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:14:2 marker=00000000002.46.6 timestamp=2023-05-31T20:38:16.835308+0000 2023-05-31T20:38:23.415 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:23.390+0000 7f22572d6780 10 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:5]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:5]:entry[obj00000000000000000025]: success 2023-05-31T20:38:23.415 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:23.390+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:5]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:5]: updating marker marker_oid=bucket.sync-status.3d163b1e-09c7-47e2-ae87-add4aace3f38:pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:5:2 marker=00000000001.40.6 timestamp=2023-05-31T20:38:16.505818+0000 2023-05-31T20:38:23.415 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:23.392+0000 7f22572d6780 10 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:12]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:12]:entry[obj00000000000000000012]: success 2023-05-31T20:38:23.416 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:23.392+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:12]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:12]: updating marker marker_oid=bucket.sync-status.3d163b1e-09c7-47e2-ae87-add4aace3f38:pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:12:2 marker=00000000002.37.6 timestamp=2023-05-31T20:38:16.377712+0000 2023-05-31T20:38:23.416 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:23.395+0000 7f22572d6780 10 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:15]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:15]:entry[obj00000000000000000006]: success 2023-05-31T20:38:23.416 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:23.395+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:15]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:15]: updating marker marker_oid=bucket.sync-status.3d163b1e-09c7-47e2-ae87-add4aace3f38:pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:15:2 marker=00000000002.63.6 timestamp=2023-05-31T20:38:16.771109+0000 2023-05-31T20:38:23.416 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:23.395+0000 7f22572d6780 10 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:0]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:0]:entry[obj00000000000000000005]: success 2023-05-31T20:38:23.416 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:23.395+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:0]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:0]: updating marker marker_oid=bucket.sync-status.3d163b1e-09c7-47e2-ae87-add4aace3f38:pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:0:2 marker=00000000002.78.6 timestamp=2023-05-31T20:38:16.597889+0000 2023-05-31T20:38:23.416 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:23.396+0000 7f22572d6780 10 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:11]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:11]:entry[obj00000000000000000014]: success 2023-05-31T20:38:23.417 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:23.396+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:11]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:11]: updating marker marker_oid=bucket.sync-status.3d163b1e-09c7-47e2-ae87-add4aace3f38:pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:11:2 marker=00000000001.80.6 timestamp=2023-05-31T20:38:16.744677+0000 2023-05-31T20:38:23.417 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:23.401+0000 7f22572d6780 10 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:9]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:9]:entry[obj00000000000000000003]: success 2023-05-31T20:38:23.417 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:23.401+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:9]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:9]: updating marker marker_oid=bucket.sync-status.3d163b1e-09c7-47e2-ae87-add4aace3f38:pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:9:2 marker=00000000002.42.6 timestamp=2023-05-31T20:38:16.680803+0000 2023-05-31T20:38:23.418 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:23.411+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:4]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:4]: parsed entry: id=00000000002.65.6 iter->object=obj00000000000000000004 iter->instance= name=obj00000000000000000004 instance= ns= 2023-05-31T20:38:23.418 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:23.411+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:4]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:4]: syncing object: pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:4/obj00000000000000000004 2023-05-31T20:38:23.419 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:23.411+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:4]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:4]: entry->timestamp=2023-05-31T20:38:16.922480+0000 2023-05-31T20:38:23.419 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:23.411+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:4]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:4]:entry[obj00000000000000000004]: bucket sync single entry (source_zone=3d163b1e-09c7-47e2-ae87-add4aace3f38) b=bucket sync single entry (source_zone=3d163b1e-09c7-47e2-ae87-add4aace3f38) b= log_entry=00000000002.65.6 op=0 op_state=1 2023-05-31T20:38:23.419 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:23.411+0000 7f22572d6780 5 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:4]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:4]:entry[obj00000000000000000004]: bucket sync: sync obj: 3d163b1e-09c7-47e2-ae87-add4aace3f38/:pklxkd-58[3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23])/obj00000000000000000004[0] 2023-05-31T20:38:23.419 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:23.411+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:4]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:4]: listing bilog for incremental sync; position=00000000002.65.6 2023-05-31T20:38:23.419 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:23.411+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:2]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:2]: parsed entry: id=00000000002.78.6 iter->object=obj00000000000000000016 iter->instance= name=obj00000000000000000016 instance= ns= 2023-05-31T20:38:23.419 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:23.411+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:2]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:2]: syncing object: pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:2/obj00000000000000000016 2023-05-31T20:38:23.419 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:23.411+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:2]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:2]: entry->timestamp=2023-05-31T20:38:16.574683+0000 2023-05-31T20:38:23.420 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:23.411+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:2]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:2]:entry[obj00000000000000000016]: bucket sync single entry (source_zone=3d163b1e-09c7-47e2-ae87-add4aace3f38) b=bucket sync single entry (source_zone=3d163b1e-09c7-47e2-ae87-add4aace3f38) b= log_entry=00000000002.78.6 op=0 op_state=1 2023-05-31T20:38:23.420 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:23.411+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:3]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:3]: parsed entry: id=00000000002.43.6 iter->object=obj00000000000000000021 iter->instance= name=obj00000000000000000021 instance= ns= 2023-05-31T20:38:23.420 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:23.411+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:3]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:3]: syncing object: pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:3/obj00000000000000000021 2023-05-31T20:38:23.420 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:23.411+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:3]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:3]: entry->timestamp=2023-05-31T20:38:16.297578+0000 2023-05-31T20:38:23.420 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:23.411+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:3]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:3]:entry[obj00000000000000000021]: bucket sync single entry (source_zone=3d163b1e-09c7-47e2-ae87-add4aace3f38) b=bucket sync single entry (source_zone=3d163b1e-09c7-47e2-ae87-add4aace3f38) b= log_entry=00000000002.43.6 op=0 op_state=1 2023-05-31T20:38:23.420 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:23.411+0000 7f22572d6780 5 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:2]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:2]:entry[obj00000000000000000016]: bucket sync: sync obj: 3d163b1e-09c7-47e2-ae87-add4aace3f38/:pklxkd-58[3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23])/obj00000000000000000016[0] 2023-05-31T20:38:23.421 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:23.411+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:2]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:2]: listing bilog for incremental sync; position=00000000002.78.6 2023-05-31T20:38:23.421 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:23.411+0000 7f22572d6780 5 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:3]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:3]:entry[obj00000000000000000021]: bucket sync: sync obj: 3d163b1e-09c7-47e2-ae87-add4aace3f38/:pklxkd-58[3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23])/obj00000000000000000021[0] 2023-05-31T20:38:23.421 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:23.411+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:3]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:3]: listing bilog for incremental sync; position=00000000002.43.6 2023-05-31T20:38:23.421 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:23.413+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:7]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:7]: parsed entry: id=00000000001.37.6 iter->object=obj00000000000000000015 iter->instance= name=obj00000000000000000015 instance= ns= 2023-05-31T20:38:23.421 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:23.413+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:7]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:7]: syncing object: pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:7/obj00000000000000000015 2023-05-31T20:38:23.421 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:23.413+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:7]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:7]: entry->timestamp=2023-05-31T20:38:16.238986+0000 2023-05-31T20:38:23.422 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:23.413+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:7]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:7]:entry[obj00000000000000000015]: bucket sync single entry (source_zone=3d163b1e-09c7-47e2-ae87-add4aace3f38) b=bucket sync single entry (source_zone=3d163b1e-09c7-47e2-ae87-add4aace3f38) b= log_entry=00000000001.37.6 op=0 op_state=1 2023-05-31T20:38:23.422 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:23.413+0000 7f22572d6780 5 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:7]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:7]:entry[obj00000000000000000015]: bucket sync: sync obj: 3d163b1e-09c7-47e2-ae87-add4aace3f38/:pklxkd-58[3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23])/obj00000000000000000015[0] 2023-05-31T20:38:23.422 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:23.413+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:7]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:7]: listing bilog for incremental sync; position=00000000001.37.6 2023-05-31T20:38:23.422 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:23.413+0000 7f22572d6780 10 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:4]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:4]:entry[obj00000000000000000004]: success 2023-05-31T20:38:23.422 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:23.413+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:4]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:4]: updating marker marker_oid=bucket.sync-status.3d163b1e-09c7-47e2-ae87-add4aace3f38:pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:4:2 marker=00000000002.65.6 timestamp=2023-05-31T20:38:16.922480+0000 2023-05-31T20:38:23.422 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:23.413+0000 7f22572d6780 10 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:3]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:3]:entry[obj00000000000000000021]: success 2023-05-31T20:38:23.422 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:23.413+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:3]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:3]: updating marker marker_oid=bucket.sync-status.3d163b1e-09c7-47e2-ae87-add4aace3f38:pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:3:2 marker=00000000002.43.6 timestamp=2023-05-31T20:38:16.297578+0000 2023-05-31T20:38:23.423 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:23.413+0000 7f22572d6780 10 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:2]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:2]:entry[obj00000000000000000016]: success 2023-05-31T20:38:23.423 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:23.413+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:2]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:2]: updating marker marker_oid=bucket.sync-status.3d163b1e-09c7-47e2-ae87-add4aace3f38:pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:2:2 marker=00000000002.78.6 timestamp=2023-05-31T20:38:16.574683+0000 2023-05-31T20:38:23.425 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:23.421+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:16]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:16]: parsed entry: id=00000000002.61.6 iter->object=obj00000000000000000011 iter->instance= name=obj00000000000000000011 instance= ns= 2023-05-31T20:38:23.425 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:23.421+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:16]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:16]: syncing object: pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:16/obj00000000000000000011 2023-05-31T20:38:23.426 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:23.421+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:16]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:16]: entry->timestamp=2023-05-31T20:38:16.217487+0000 2023-05-31T20:38:23.426 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:23.421+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:16]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:16]:entry[obj00000000000000000011]: bucket sync single entry (source_zone=3d163b1e-09c7-47e2-ae87-add4aace3f38) b=bucket sync single entry (source_zone=3d163b1e-09c7-47e2-ae87-add4aace3f38) b= log_entry=00000000002.61.6 op=0 op_state=1 2023-05-31T20:38:23.426 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:23.421+0000 7f22572d6780 5 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:16]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:16]:entry[obj00000000000000000011]: bucket sync: sync obj: 3d163b1e-09c7-47e2-ae87-add4aace3f38/:pklxkd-58[3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23])/obj00000000000000000011[0] 2023-05-31T20:38:23.426 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:23.421+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:16]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:16]: listing bilog for incremental sync; position=00000000002.61.6 2023-05-31T20:38:23.427 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:23.422+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:17]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:17]: parsed entry: id=00000000002.57.6 iter->object=obj00000000000000000001 iter->instance= name=obj00000000000000000001 instance= ns= 2023-05-31T20:38:23.427 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:23.422+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:17]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:17]: syncing object: pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:17/obj00000000000000000001 2023-05-31T20:38:23.427 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:23.422+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:17]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:17]: entry->timestamp=2023-05-31T20:38:16.397222+0000 2023-05-31T20:38:23.427 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:23.422+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:17]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:17]:entry[obj00000000000000000001]: bucket sync single entry (source_zone=3d163b1e-09c7-47e2-ae87-add4aace3f38) b=bucket sync single entry (source_zone=3d163b1e-09c7-47e2-ae87-add4aace3f38) b= log_entry=00000000002.57.6 op=0 op_state=1 2023-05-31T20:38:23.427 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:23.422+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:1]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:1]: parsed entry: id=00000000002.80.6 iter->object=obj00000000000000000010 iter->instance= name=obj00000000000000000010 instance= ns= 2023-05-31T20:38:23.428 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:23.422+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:1]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:1]: syncing object: pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:1/obj00000000000000000010 2023-05-31T20:38:23.428 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:23.422+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:1]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:1]: entry->timestamp=2023-05-31T20:38:16.455609+0000 2023-05-31T20:38:23.428 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:23.422+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:1]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:1]:entry[obj00000000000000000010]: bucket sync single entry (source_zone=3d163b1e-09c7-47e2-ae87-add4aace3f38) b=bucket sync single entry (source_zone=3d163b1e-09c7-47e2-ae87-add4aace3f38) b= log_entry=00000000002.80.6 op=0 op_state=1 2023-05-31T20:38:23.428 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:23.422+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:18]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:18]: parsed entry: id=00000000002.48.6 iter->object=obj00000000000000000007 iter->instance= name=obj00000000000000000007 instance= ns= 2023-05-31T20:38:23.428 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:23.422+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:18]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:18]: syncing object: pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:18/obj00000000000000000007 2023-05-31T20:38:23.429 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:23.422+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:18]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:18]: entry->timestamp=2023-05-31T20:38:16.864971+0000 2023-05-31T20:38:23.429 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:23.422+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:18]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:18]:entry[obj00000000000000000007]: bucket sync single entry (source_zone=3d163b1e-09c7-47e2-ae87-add4aace3f38) b=bucket sync single entry (source_zone=3d163b1e-09c7-47e2-ae87-add4aace3f38) b= log_entry=00000000002.48.6 op=0 op_state=1 2023-05-31T20:38:23.429 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:23.422+0000 7f22572d6780 5 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:17]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:17]:entry[obj00000000000000000001]: bucket sync: sync obj: 3d163b1e-09c7-47e2-ae87-add4aace3f38/:pklxkd-58[3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23])/obj00000000000000000001[0] 2023-05-31T20:38:23.429 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:23.422+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:17]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:17]: listing bilog for incremental sync; position=00000000002.57.6 2023-05-31T20:38:23.429 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:23.422+0000 7f22572d6780 5 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:1]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:1]:entry[obj00000000000000000010]: bucket sync: sync obj: 3d163b1e-09c7-47e2-ae87-add4aace3f38/:pklxkd-58[3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23])/obj00000000000000000010[0] 2023-05-31T20:38:23.429 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:23.422+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:1]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:1]: listing bilog for incremental sync; position=00000000002.80.6 2023-05-31T20:38:23.430 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:23.422+0000 7f22572d6780 5 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:18]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:18]:entry[obj00000000000000000007]: bucket sync: sync obj: 3d163b1e-09c7-47e2-ae87-add4aace3f38/:pklxkd-58[3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23])/obj00000000000000000007[0] 2023-05-31T20:38:23.430 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:23.422+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:18]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:18]: listing bilog for incremental sync; position=00000000002.48.6 2023-05-31T20:38:23.430 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:23.425+0000 7f22572d6780 10 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:18]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:18]:entry[obj00000000000000000007]: success 2023-05-31T20:38:23.431 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:23.425+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:18]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:18]: updating marker marker_oid=bucket.sync-status.3d163b1e-09c7-47e2-ae87-add4aace3f38:pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:18:2 marker=00000000002.48.6 timestamp=2023-05-31T20:38:16.864971+0000 2023-05-31T20:38:23.431 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:23.426+0000 7f22572d6780 10 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:1]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:1]:entry[obj00000000000000000010]: success 2023-05-31T20:38:23.431 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:23.426+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:1]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:1]: updating marker marker_oid=bucket.sync-status.3d163b1e-09c7-47e2-ae87-add4aace3f38:pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:1:2 marker=00000000002.80.6 timestamp=2023-05-31T20:38:16.455609+0000 2023-05-31T20:38:23.431 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:23.426+0000 7f22572d6780 10 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:17]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:17]:entry[obj00000000000000000001]: success 2023-05-31T20:38:23.431 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:23.426+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:17]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:17]: updating marker marker_oid=bucket.sync-status.3d163b1e-09c7-47e2-ae87-add4aace3f38:pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:17:2 marker=00000000002.57.6 timestamp=2023-05-31T20:38:16.397222+0000 2023-05-31T20:38:23.432 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:23.427+0000 7f22572d6780 10 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:10]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:10]:entry[obj00000000000000000009]: success 2023-05-31T20:38:23.432 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:23.428+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:10]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:10]: updating marker marker_oid=bucket.sync-status.3d163b1e-09c7-47e2-ae87-add4aace3f38:pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:10:2 marker=00000000001.59.6 timestamp=2023-05-31T20:38:16.655883+0000 2023-05-31T20:38:23.472 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:23.468+0000 7f22572d6780 10 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:7]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:7]:entry[obj00000000000000000015]: success 2023-05-31T20:38:23.472 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:23.468+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:7]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:7]: updating marker marker_oid=bucket.sync-status.3d163b1e-09c7-47e2-ae87-add4aace3f38:pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:7:2 marker=00000000001.37.6 timestamp=2023-05-31T20:38:16.238986+0000 2023-05-31T20:38:23.481 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:23.477+0000 7f22572d6780 10 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:16]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:16]:entry[obj00000000000000000011]: success 2023-05-31T20:38:23.481 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:23.477+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:16]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:16]: updating marker marker_oid=bucket.sync-status.3d163b1e-09c7-47e2-ae87-add4aace3f38:pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:16:2 marker=00000000002.61.6 timestamp=2023-05-31T20:38:16.217487+0000 2023-05-31T20:38:23.509 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:23.503+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:2]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-05-31T20:38:23.509 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:23.503+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:1]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-05-31T20:38:23.509 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:23.504+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:3]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-05-31T20:38:23.509 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:23.504+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:2]: sync status for source bucket shard: 0 2023-05-31T20:38:23.509 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:23.504+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:2]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:2]: listing bilog for incremental sync; position= 2023-05-31T20:38:23.510 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:23.504+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:1]: sync status for source bucket shard: 0 2023-05-31T20:38:23.510 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:23.504+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:1]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:1]: listing bilog for incremental sync; position= 2023-05-31T20:38:23.510 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:23.504+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:0]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-05-31T20:38:23.510 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:23.504+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:3]: sync status for source bucket shard: 0 2023-05-31T20:38:23.510 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:23.504+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:3]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:3]: listing bilog for incremental sync; position= 2023-05-31T20:38:23.510 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:23.504+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:0]: sync status for source bucket shard: 0 2023-05-31T20:38:23.510 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:23.504+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:5]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-05-31T20:38:23.511 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:23.504+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:0]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:0]: listing bilog for incremental sync; position= 2023-05-31T20:38:23.511 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:23.504+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:4]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-05-31T20:38:23.511 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:23.505+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:6]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-05-31T20:38:23.514 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:23.505+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:4]: sync status for source bucket shard: 0 2023-05-31T20:38:23.515 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:23.505+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:4]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:4]: listing bilog for incremental sync; position= 2023-05-31T20:38:23.515 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:23.505+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:6]: sync status for source bucket shard: 0 2023-05-31T20:38:23.515 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:23.505+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:5]: sync status for source bucket shard: 0 2023-05-31T20:38:23.515 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:23.505+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:6]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:6]: listing bilog for incremental sync; position= 2023-05-31T20:38:23.516 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:23.505+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:5]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:5]: listing bilog for incremental sync; position= 2023-05-31T20:38:23.516 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:23.505+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:7]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-05-31T20:38:23.516 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:23.505+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:3]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:3]: parsed entry: id=00000000001.64.6 iter->object=obj00000000000000000046 iter->instance= name=obj00000000000000000046 instance= ns= 2023-05-31T20:38:23.516 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:23.505+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:3]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:3]: syncing object: pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:3/obj00000000000000000046 2023-05-31T20:38:23.516 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:23.505+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:3]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:3]: entry->timestamp=2023-05-31T20:38:17.357924+0000 2023-05-31T20:38:23.516 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:23.505+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:3]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:3]:entry[obj00000000000000000046]: bucket sync single entry (source_zone=3d163b1e-09c7-47e2-ae87-add4aace3f38) b=bucket sync single entry (source_zone=3d163b1e-09c7-47e2-ae87-add4aace3f38) b= log_entry=00000000001.64.6 op=0 op_state=1 2023-05-31T20:38:23.517 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:23.505+0000 7f22572d6780 5 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:3]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:3]:entry[obj00000000000000000046]: bucket sync: sync obj: 3d163b1e-09c7-47e2-ae87-add4aace3f38/:pklxkd-58[3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23])/obj00000000000000000046[0] 2023-05-31T20:38:23.517 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:23.505+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:3]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:3]: listing bilog for incremental sync; position=00000000001.64.6 2023-05-31T20:38:23.517 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:23.506+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:7]: sync status for source bucket shard: 0 2023-05-31T20:38:23.518 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:23.506+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:8]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-05-31T20:38:23.518 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:23.506+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:7]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:7]: listing bilog for incremental sync; position= 2023-05-31T20:38:23.518 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:23.506+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:9]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-05-31T20:38:23.518 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:23.506+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:11]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-05-31T20:38:23.518 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:23.506+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:10]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-05-31T20:38:23.518 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:23.506+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:12]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-05-31T20:38:23.519 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:23.506+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:14]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-05-31T20:38:23.519 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:23.507+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:12]: sync status for source bucket shard: 0 2023-05-31T20:38:23.519 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:23.507+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:13]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-05-31T20:38:23.520 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:23.507+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:12]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:12]: listing bilog for incremental sync; position= 2023-05-31T20:38:23.520 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:23.507+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:0]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:0]: parsed entry: id=00000000001.42.6 iter->object=obj00000000000000000058 iter->instance= name=obj00000000000000000058 instance= ns= 2023-05-31T20:38:23.521 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:23.507+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:0]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:0]: syncing object: pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:0/obj00000000000000000058 2023-05-31T20:38:23.521 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:23.507+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:0]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:0]: entry->timestamp=2023-05-31T20:38:17.631771+0000 2023-05-31T20:38:23.521 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:23.507+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:0]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:0]:entry[obj00000000000000000058]: bucket sync single entry (source_zone=3d163b1e-09c7-47e2-ae87-add4aace3f38) b=bucket sync single entry (source_zone=3d163b1e-09c7-47e2-ae87-add4aace3f38) b= log_entry=00000000001.42.6 op=0 op_state=1 2023-05-31T20:38:23.521 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:23.507+0000 7f22572d6780 5 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:0]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:0]:entry[obj00000000000000000058]: bucket sync: sync obj: 3d163b1e-09c7-47e2-ae87-add4aace3f38/:pklxkd-58[3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23])/obj00000000000000000058[0] 2023-05-31T20:38:23.521 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:23.507+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:0]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:0]: listing bilog for incremental sync; position=00000000001.42.6 2023-05-31T20:38:23.522 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:23.508+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:9]: sync status for source bucket shard: 0 2023-05-31T20:38:23.522 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:23.508+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:15]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-05-31T20:38:23.522 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:23.508+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:9]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:9]: listing bilog for incremental sync; position= 2023-05-31T20:38:23.522 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:23.508+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:16]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-05-31T20:38:23.523 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:23.508+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:17]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-05-31T20:38:23.523 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:23.508+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:20]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-05-31T20:38:23.523 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:23.508+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:19]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-05-31T20:38:23.523 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:23.508+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:1]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:1]: parsed entry: id=00000000002.44.6 iter->object=obj00000000000000000026 iter->instance= name=obj00000000000000000026 instance= ns= 2023-05-31T20:38:23.524 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:23.508+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:1]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:1]: syncing object: pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:1/obj00000000000000000026 2023-05-31T20:38:23.525 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:23.508+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:1]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:1]: entry->timestamp=2023-05-31T20:38:17.275761+0000 2023-05-31T20:38:23.525 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:23.508+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:1]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:1]:entry[obj00000000000000000026]: bucket sync single entry (source_zone=3d163b1e-09c7-47e2-ae87-add4aace3f38) b=bucket sync single entry (source_zone=3d163b1e-09c7-47e2-ae87-add4aace3f38) b= log_entry=00000000002.44.6 op=0 op_state=1 2023-05-31T20:38:23.525 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:23.508+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:18]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-05-31T20:38:23.526 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:23.508+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:4]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:4]: parsed entry: id=00000000002.73.6 iter->object=obj00000000000000000014 iter->instance= name=obj00000000000000000014 instance= ns= 2023-05-31T20:38:23.526 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:23.508+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:4]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:4]: syncing object: pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:4/obj00000000000000000014 2023-05-31T20:38:23.526 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:23.508+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:4]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:4]: entry->timestamp=2023-05-31T20:38:17.717419+0000 2023-05-31T20:38:23.527 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:23.508+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:4]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:4]:entry[obj00000000000000000014]: bucket sync single entry (source_zone=3d163b1e-09c7-47e2-ae87-add4aace3f38) b=bucket sync single entry (source_zone=3d163b1e-09c7-47e2-ae87-add4aace3f38) b= log_entry=00000000002.73.6 op=0 op_state=1 2023-05-31T20:38:23.527 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:23.508+0000 7f22572d6780 5 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:1]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:1]:entry[obj00000000000000000026]: bucket sync: sync obj: 3d163b1e-09c7-47e2-ae87-add4aace3f38/:pklxkd-58[3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23])/obj00000000000000000026[0] 2023-05-31T20:38:23.527 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:23.508+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:1]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:1]: listing bilog for incremental sync; position=00000000002.44.6 2023-05-31T20:38:23.528 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:23.508+0000 7f22572d6780 5 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:4]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:4]:entry[obj00000000000000000014]: bucket sync: sync obj: 3d163b1e-09c7-47e2-ae87-add4aace3f38/:pklxkd-58[3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23])/obj00000000000000000014[0] 2023-05-31T20:38:23.528 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:23.508+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:4]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:4]: listing bilog for incremental sync; position=00000000002.73.6 2023-05-31T20:38:23.528 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:23.508+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:22]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-05-31T20:38:23.528 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:23.509+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:17]: sync status for source bucket shard: 0 2023-05-31T20:38:23.528 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:23.509+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:17]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:17]: listing bilog for incremental sync; position= 2023-05-31T20:38:23.529 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:23.509+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:5]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:5]: parsed entry: id=00000000001.46.6 iter->object=obj00000000000000000050 iter->instance= name=obj00000000000000000050 instance= ns= 2023-05-31T20:38:23.529 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:23.509+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:5]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:5]: syncing object: pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:5/obj00000000000000000050 2023-05-31T20:38:23.529 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:23.509+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:5]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:5]: entry->timestamp=2023-05-31T20:38:17.921897+0000 2023-05-31T20:38:23.529 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:23.509+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:5]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:5]:entry[obj00000000000000000050]: bucket sync single entry (source_zone=3d163b1e-09c7-47e2-ae87-add4aace3f38) b=bucket sync single entry (source_zone=3d163b1e-09c7-47e2-ae87-add4aace3f38) b= log_entry=00000000001.46.6 op=0 op_state=1 2023-05-31T20:38:23.529 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:23.509+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:21]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-05-31T20:38:23.530 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:23.509+0000 7f22572d6780 5 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:5]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:5]:entry[obj00000000000000000050]: bucket sync: sync obj: 3d163b1e-09c7-47e2-ae87-add4aace3f38/:pklxkd-58[3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23])/obj00000000000000000050[0] 2023-05-31T20:38:23.530 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:23.509+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:5]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:5]: listing bilog for incremental sync; position=00000000001.46.6 2023-05-31T20:38:23.530 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:23.509+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:16]: sync status for source bucket shard: 0 2023-05-31T20:38:23.530 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:23.509+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:22]: sync status for source bucket shard: 0 2023-05-31T20:38:23.530 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:23.509+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:16]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:16]: listing bilog for incremental sync; position= 2023-05-31T20:38:23.531 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:23.509+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:22]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:22]: listing bilog for incremental sync; position= 2023-05-31T20:38:23.531 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:23.510+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:14]: sync status for source bucket shard: 0 2023-05-31T20:38:23.531 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:23.510+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:14]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:14]: listing bilog for incremental sync; position= 2023-05-31T20:38:23.531 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:23.510+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:10]: sync status for source bucket shard: 0 2023-05-31T20:38:23.532 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:23.510+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:10]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:10]: listing bilog for incremental sync; position= 2023-05-31T20:38:23.535 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:23.511+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:8]: sync status for source bucket shard: 0 2023-05-31T20:38:23.536 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:23.511+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:8]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:8]: listing bilog for incremental sync; position= 2023-05-31T20:38:23.536 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:23.511+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:11]: sync status for source bucket shard: 0 2023-05-31T20:38:23.536 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:23.511+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:13]: sync status for source bucket shard: 0 2023-05-31T20:38:23.536 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:23.511+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:15]: sync status for source bucket shard: 0 2023-05-31T20:38:23.536 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:23.511+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:19]: sync status for source bucket shard: 0 2023-05-31T20:38:23.537 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:23.511+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:20]: sync status for source bucket shard: 0 2023-05-31T20:38:23.537 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:23.511+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:18]: sync status for source bucket shard: 0 2023-05-31T20:38:23.537 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:23.511+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:11]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:11]: listing bilog for incremental sync; position= 2023-05-31T20:38:23.537 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:23.511+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:13]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:13]: listing bilog for incremental sync; position= 2023-05-31T20:38:23.537 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:23.511+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:15]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:15]: listing bilog for incremental sync; position= 2023-05-31T20:38:23.537 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:23.511+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:19]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:19]: listing bilog for incremental sync; position= 2023-05-31T20:38:23.537 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:23.511+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:20]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:20]: listing bilog for incremental sync; position= 2023-05-31T20:38:23.538 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:23.511+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:18]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:18]: listing bilog for incremental sync; position= 2023-05-31T20:38:23.538 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:23.511+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:21]: sync status for source bucket shard: 0 2023-05-31T20:38:23.538 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:23.511+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:21]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:21]: listing bilog for incremental sync; position= 2023-05-31T20:38:23.539 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:23.512+0000 7f22572d6780 10 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:4]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:4]:entry[obj00000000000000000014]: success 2023-05-31T20:38:23.539 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:23.512+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:4]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:4]: updating marker marker_oid=bucket.sync-status.3d163b1e-09c7-47e2-ae87-add4aace3f38:pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:4:3 marker=00000000002.73.6 timestamp=2023-05-31T20:38:17.717419+0000 2023-05-31T20:38:23.539 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:23.513+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:10]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:10]: parsed entry: id=00000000002.50.6 iter->object=obj00000000000000000006 iter->instance= name=obj00000000000000000006 instance= ns= 2023-05-31T20:38:23.539 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:23.513+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:10]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:10]: syncing object: pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:10/obj00000000000000000006 2023-05-31T20:38:23.539 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:23.513+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:10]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:10]: entry->timestamp=2023-05-31T20:38:18.035690+0000 2023-05-31T20:38:23.540 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:23.513+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:10]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:10]:entry[obj00000000000000000006]: bucket sync single entry (source_zone=3d163b1e-09c7-47e2-ae87-add4aace3f38) b=bucket sync single entry (source_zone=3d163b1e-09c7-47e2-ae87-add4aace3f38) b= log_entry=00000000002.50.6 op=0 op_state=1 2023-05-31T20:38:23.540 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:23.513+0000 7f22572d6780 5 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:10]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:10]:entry[obj00000000000000000006]: bucket sync: sync obj: 3d163b1e-09c7-47e2-ae87-add4aace3f38/:pklxkd-58[3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23])/obj00000000000000000006[0] 2023-05-31T20:38:23.540 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:23.513+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:10]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:10]: listing bilog for incremental sync; position=00000000002.50.6 2023-05-31T20:38:23.540 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:23.513+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:14]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:14]: parsed entry: id=00000000002.42.6 iter->object=obj00000000000000000013 iter->instance= name=obj00000000000000000013 instance= ns= 2023-05-31T20:38:23.541 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:23.513+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:14]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:14]: syncing object: pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:14/obj00000000000000000013 2023-05-31T20:38:23.541 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:23.513+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:14]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:14]: entry->timestamp=2023-05-31T20:38:17.296633+0000 2023-05-31T20:38:23.541 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:23.513+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:14]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:14]:entry[obj00000000000000000013]: bucket sync single entry (source_zone=3d163b1e-09c7-47e2-ae87-add4aace3f38) b=bucket sync single entry (source_zone=3d163b1e-09c7-47e2-ae87-add4aace3f38) b= log_entry=00000000002.42.6 op=0 op_state=1 2023-05-31T20:38:23.541 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:23.513+0000 7f22572d6780 5 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:14]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:14]:entry[obj00000000000000000013]: bucket sync: sync obj: 3d163b1e-09c7-47e2-ae87-add4aace3f38/:pklxkd-58[3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23])/obj00000000000000000013[0] 2023-05-31T20:38:23.542 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:23.513+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:14]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:14]: listing bilog for incremental sync; position=00000000002.42.6 2023-05-31T20:38:23.542 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:23.514+0000 7f22572d6780 10 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:1]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:1]:entry[obj00000000000000000026]: success 2023-05-31T20:38:23.542 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:23.514+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:1]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:1]: updating marker marker_oid=bucket.sync-status.3d163b1e-09c7-47e2-ae87-add4aace3f38:pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:1:3 marker=00000000002.44.6 timestamp=2023-05-31T20:38:17.275761+0000 2023-05-31T20:38:23.542 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:23.515+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:16]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:16]: parsed entry: id=00000000002.46.6 iter->object=obj00000000000000000033 iter->instance= name=obj00000000000000000033 instance= ns= 2023-05-31T20:38:23.543 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:23.515+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:16]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:16]: syncing object: pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:16/obj00000000000000000033 2023-05-31T20:38:23.543 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:23.515+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:16]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:16]: entry->timestamp=2023-05-31T20:38:17.570858+0000 2023-05-31T20:38:23.543 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:23.515+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:16]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:16]:entry[obj00000000000000000033]: bucket sync single entry (source_zone=3d163b1e-09c7-47e2-ae87-add4aace3f38) b=bucket sync single entry (source_zone=3d163b1e-09c7-47e2-ae87-add4aace3f38) b= log_entry=00000000002.46.6 op=0 op_state=1 2023-05-31T20:38:23.543 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:23.515+0000 7f22572d6780 5 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:16]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:16]:entry[obj00000000000000000033]: bucket sync: sync obj: 3d163b1e-09c7-47e2-ae87-add4aace3f38/:pklxkd-58[3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23])/obj00000000000000000033[0] 2023-05-31T20:38:23.544 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:23.515+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:16]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:16]: listing bilog for incremental sync; position=00000000002.46.6 2023-05-31T20:38:23.544 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:23.515+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:22]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:22]: parsed entry: id=00000000002.49.6 iter->object=obj00000000000000000008 iter->instance= name=obj00000000000000000008 instance= ns= 2023-05-31T20:38:23.544 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:23.515+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:22]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:22]: syncing object: pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:22/obj00000000000000000008 2023-05-31T20:38:23.545 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:23.515+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:22]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:22]: entry->timestamp=2023-05-31T20:38:17.470631+0000 2023-05-31T20:38:23.545 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:23.515+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:22]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:22]:entry[obj00000000000000000008]: bucket sync single entry (source_zone=3d163b1e-09c7-47e2-ae87-add4aace3f38) b=bucket sync single entry (source_zone=3d163b1e-09c7-47e2-ae87-add4aace3f38) b= log_entry=00000000002.49.6 op=0 op_state=1 2023-05-31T20:38:23.545 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:23.515+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:8]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:8]: parsed entry: id=00000000001.56.6 iter->object=obj00000000000000000039 iter->instance= name=obj00000000000000000039 instance= ns= 2023-05-31T20:38:23.545 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:23.515+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:8]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:8]: syncing object: pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:8/obj00000000000000000039 2023-05-31T20:38:23.546 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:23.515+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:8]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:8]: entry->timestamp=2023-05-31T20:38:17.380290+0000 2023-05-31T20:38:23.546 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:23.515+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:8]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:8]:entry[obj00000000000000000039]: bucket sync single entry (source_zone=3d163b1e-09c7-47e2-ae87-add4aace3f38) b=bucket sync single entry (source_zone=3d163b1e-09c7-47e2-ae87-add4aace3f38) b= log_entry=00000000001.56.6 op=0 op_state=1 2023-05-31T20:38:23.546 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:23.515+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:13]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:13]: parsed entry: id=00000000001.20.6 iter->object=obj00000000000000000045 iter->instance= name=obj00000000000000000045 instance= ns= 2023-05-31T20:38:23.546 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:23.515+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:13]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:13]: syncing object: pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:13/obj00000000000000000045 2023-05-31T20:38:23.546 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:23.515+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:13]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:13]: entry->timestamp=2023-05-31T20:38:17.831535+0000 2023-05-31T20:38:23.547 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:23.515+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:13]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:13]:entry[obj00000000000000000045]: bucket sync single entry (source_zone=3d163b1e-09c7-47e2-ae87-add4aace3f38) b=bucket sync single entry (source_zone=3d163b1e-09c7-47e2-ae87-add4aace3f38) b= log_entry=00000000001.20.6 op=0 op_state=1 2023-05-31T20:38:23.547 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:23.515+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:20]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:20]: parsed entry: id=00000000002.33.6 iter->object=obj00000000000000000003 iter->instance= name=obj00000000000000000003 instance= ns= 2023-05-31T20:38:23.547 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:23.515+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:20]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:20]: syncing object: pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:20/obj00000000000000000003 2023-05-31T20:38:23.548 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:23.515+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:20]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:20]: entry->timestamp=2023-05-31T20:38:18.007509+0000 2023-05-31T20:38:23.548 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:23.516+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:20]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:20]:entry[obj00000000000000000003]: bucket sync single entry (source_zone=3d163b1e-09c7-47e2-ae87-add4aace3f38) b=bucket sync single entry (source_zone=3d163b1e-09c7-47e2-ae87-add4aace3f38) b= log_entry=00000000002.33.6 op=0 op_state=1 2023-05-31T20:38:23.548 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:23.516+0000 7f22572d6780 5 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:22]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:22]:entry[obj00000000000000000008]: bucket sync: sync obj: 3d163b1e-09c7-47e2-ae87-add4aace3f38/:pklxkd-58[3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23])/obj00000000000000000008[0] 2023-05-31T20:38:23.548 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:23.516+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:22]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:22]: listing bilog for incremental sync; position=00000000002.49.6 2023-05-31T20:38:23.549 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:23.516+0000 7f22572d6780 5 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:8]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:8]:entry[obj00000000000000000039]: bucket sync: sync obj: 3d163b1e-09c7-47e2-ae87-add4aace3f38/:pklxkd-58[3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23])/obj00000000000000000039[0] 2023-05-31T20:38:23.549 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:23.516+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:8]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:8]: listing bilog for incremental sync; position=00000000001.56.6 2023-05-31T20:38:23.549 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:23.516+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:17]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:17]: parsed entry: id=00000000002.45.6 iter->object=obj00000000000000000001 iter->instance= name=obj00000000000000000001 instance= ns= 2023-05-31T20:38:23.549 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:23.516+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:17]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:17]: syncing object: pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:17/obj00000000000000000001 2023-05-31T20:38:23.550 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:23.516+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:17]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:17]: entry->timestamp=2023-05-31T20:38:17.805124+0000 2023-05-31T20:38:23.550 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:23.516+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:17]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:17]:entry[obj00000000000000000001]: bucket sync single entry (source_zone=3d163b1e-09c7-47e2-ae87-add4aace3f38) b=bucket sync single entry (source_zone=3d163b1e-09c7-47e2-ae87-add4aace3f38) b= log_entry=00000000002.45.6 op=0 op_state=1 2023-05-31T20:38:23.550 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:23.516+0000 7f22572d6780 5 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:13]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:13]:entry[obj00000000000000000045]: bucket sync: sync obj: 3d163b1e-09c7-47e2-ae87-add4aace3f38/:pklxkd-58[3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23])/obj00000000000000000045[0] 2023-05-31T20:38:23.550 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:23.516+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:13]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:13]: listing bilog for incremental sync; position=00000000001.20.6 2023-05-31T20:38:23.551 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:23.516+0000 7f22572d6780 5 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:20]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:20]:entry[obj00000000000000000003]: bucket sync: sync obj: 3d163b1e-09c7-47e2-ae87-add4aace3f38/:pklxkd-58[3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23])/obj00000000000000000003[0] 2023-05-31T20:38:23.551 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:23.516+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:20]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:20]: listing bilog for incremental sync; position=00000000002.33.6 2023-05-31T20:38:23.551 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:23.516+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:18]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:18]: parsed entry: id=00000000002.52.6 iter->object=obj00000000000000000000 iter->instance= name=obj00000000000000000000 instance= ns= 2023-05-31T20:38:23.552 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:23.516+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:18]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:18]: syncing object: pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:18/obj00000000000000000000 2023-05-31T20:38:23.552 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:23.516+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:18]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:18]: entry->timestamp=2023-05-31T20:38:17.942191+0000 2023-05-31T20:38:23.552 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:23.516+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:18]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:18]:entry[obj00000000000000000000]: bucket sync single entry (source_zone=3d163b1e-09c7-47e2-ae87-add4aace3f38) b=bucket sync single entry (source_zone=3d163b1e-09c7-47e2-ae87-add4aace3f38) b= log_entry=00000000002.52.6 op=0 op_state=1 2023-05-31T20:38:23.552 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:23.516+0000 7f22572d6780 5 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:17]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:17]:entry[obj00000000000000000001]: bucket sync: sync obj: 3d163b1e-09c7-47e2-ae87-add4aace3f38/:pklxkd-58[3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23])/obj00000000000000000001[0] 2023-05-31T20:38:23.553 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:23.516+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:17]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:17]: listing bilog for incremental sync; position=00000000002.45.6 2023-05-31T20:38:23.553 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:23.516+0000 7f22572d6780 5 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:18]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:18]:entry[obj00000000000000000000]: bucket sync: sync obj: 3d163b1e-09c7-47e2-ae87-add4aace3f38/:pklxkd-58[3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23])/obj00000000000000000000[0] 2023-05-31T20:38:23.553 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:23.516+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:18]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:18]: listing bilog for incremental sync; position=00000000002.52.6 2023-05-31T20:38:23.554 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:23.517+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:19]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:19]: parsed entry: id=00000000002.50.6 iter->object=obj00000000000000000021 iter->instance= name=obj00000000000000000021 instance= ns= 2023-05-31T20:38:23.554 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:23.517+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:19]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:19]: syncing object: pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:19/obj00000000000000000021 2023-05-31T20:38:23.554 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:23.517+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:19]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:19]: entry->timestamp=2023-05-31T20:38:17.652062+0000 2023-05-31T20:38:23.554 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:23.517+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:19]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:19]:entry[obj00000000000000000021]: bucket sync single entry (source_zone=3d163b1e-09c7-47e2-ae87-add4aace3f38) b=bucket sync single entry (source_zone=3d163b1e-09c7-47e2-ae87-add4aace3f38) b= log_entry=00000000002.50.6 op=0 op_state=1 2023-05-31T20:38:23.554 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:23.517+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:15]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:15]: parsed entry: id=00000000002.46.6 iter->object=obj00000000000000000012 iter->instance= name=obj00000000000000000012 instance= ns= 2023-05-31T20:38:23.554 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:23.517+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:15]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:15]: syncing object: pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:15/obj00000000000000000012 2023-05-31T20:38:23.555 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:23.517+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:15]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:15]: entry->timestamp=2023-05-31T20:38:18.095753+0000 2023-05-31T20:38:23.555 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:23.517+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:15]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:15]:entry[obj00000000000000000012]: bucket sync single entry (source_zone=3d163b1e-09c7-47e2-ae87-add4aace3f38) b=bucket sync single entry (source_zone=3d163b1e-09c7-47e2-ae87-add4aace3f38) b= log_entry=00000000002.46.6 op=0 op_state=1 2023-05-31T20:38:23.555 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:23.517+0000 7f22572d6780 5 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:19]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:19]:entry[obj00000000000000000021]: bucket sync: sync obj: 3d163b1e-09c7-47e2-ae87-add4aace3f38/:pklxkd-58[3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23])/obj00000000000000000021[0] 2023-05-31T20:38:23.555 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:23.517+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:19]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:19]: listing bilog for incremental sync; position=00000000002.50.6 2023-05-31T20:38:23.556 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:23.517+0000 7f22572d6780 5 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:15]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:15]:entry[obj00000000000000000012]: bucket sync: sync obj: 3d163b1e-09c7-47e2-ae87-add4aace3f38/:pklxkd-58[3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23])/obj00000000000000000012[0] 2023-05-31T20:38:23.556 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:23.517+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:15]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:15]: listing bilog for incremental sync; position=00000000002.46.6 2023-05-31T20:38:23.556 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:23.517+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:21]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:21]: parsed entry: id=00000000001.82.6 iter->object=obj00000000000000000055 iter->instance= name=obj00000000000000000055 instance= ns= 2023-05-31T20:38:23.556 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:23.517+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:21]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:21]: syncing object: pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:21/obj00000000000000000055 2023-05-31T20:38:23.556 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:23.517+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:21]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:21]: entry->timestamp=2023-05-31T20:38:17.862999+0000 2023-05-31T20:38:23.557 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:23.517+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:21]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:21]:entry[obj00000000000000000055]: bucket sync single entry (source_zone=3d163b1e-09c7-47e2-ae87-add4aace3f38) b=bucket sync single entry (source_zone=3d163b1e-09c7-47e2-ae87-add4aace3f38) b= log_entry=00000000001.82.6 op=0 op_state=1 2023-05-31T20:38:23.557 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:23.517+0000 7f22572d6780 5 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:21]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:21]:entry[obj00000000000000000055]: bucket sync: sync obj: 3d163b1e-09c7-47e2-ae87-add4aace3f38/:pklxkd-58[3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23])/obj00000000000000000055[0] 2023-05-31T20:38:23.557 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:23.517+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:21]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:21]: listing bilog for incremental sync; position=00000000001.82.6 2023-05-31T20:38:23.557 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:23.521+0000 7f22572d6780 10 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:17]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:17]:entry[obj00000000000000000001]: success 2023-05-31T20:38:23.557 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:23.521+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:17]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:17]: updating marker marker_oid=bucket.sync-status.3d163b1e-09c7-47e2-ae87-add4aace3f38:pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:17:3 marker=00000000002.45.6 timestamp=2023-05-31T20:38:17.805124+0000 2023-05-31T20:38:23.558 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:23.521+0000 7f22572d6780 10 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:18]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:18]:entry[obj00000000000000000000]: success 2023-05-31T20:38:23.558 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:23.521+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:18]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:18]: updating marker marker_oid=bucket.sync-status.3d163b1e-09c7-47e2-ae87-add4aace3f38:pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:18:3 marker=00000000002.52.6 timestamp=2023-05-31T20:38:17.942191+0000 2023-05-31T20:38:23.558 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:23.521+0000 7f22572d6780 10 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:19]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:19]:entry[obj00000000000000000021]: success 2023-05-31T20:38:23.558 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:23.521+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:19]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:19]: updating marker marker_oid=bucket.sync-status.3d163b1e-09c7-47e2-ae87-add4aace3f38:pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:19:3 marker=00000000002.50.6 timestamp=2023-05-31T20:38:17.652062+0000 2023-05-31T20:38:23.558 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:23.521+0000 7f22572d6780 10 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:14]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:14]:entry[obj00000000000000000013]: success 2023-05-31T20:38:23.558 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:23.521+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:14]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:14]: updating marker marker_oid=bucket.sync-status.3d163b1e-09c7-47e2-ae87-add4aace3f38:pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:14:3 marker=00000000002.42.6 timestamp=2023-05-31T20:38:17.296633+0000 2023-05-31T20:38:23.559 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:23.522+0000 7f22572d6780 10 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:16]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:16]:entry[obj00000000000000000033]: success 2023-05-31T20:38:23.559 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:23.522+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:16]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:16]: updating marker marker_oid=bucket.sync-status.3d163b1e-09c7-47e2-ae87-add4aace3f38:pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:16:3 marker=00000000002.46.6 timestamp=2023-05-31T20:38:17.570858+0000 2023-05-31T20:38:23.559 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:23.523+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:11]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:11]: parsed entry: id=00000000001.51.6 iter->object=obj00000000000000000040 iter->instance= name=obj00000000000000000040 instance= ns= 2023-05-31T20:38:23.559 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:23.524+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:11]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:11]: syncing object: pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:11/obj00000000000000000040 2023-05-31T20:38:23.559 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:23.524+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:11]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:11]: entry->timestamp=2023-05-31T20:38:17.552106+0000 2023-05-31T20:38:23.559 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:23.524+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:11]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:11]:entry[obj00000000000000000040]: bucket sync single entry (source_zone=3d163b1e-09c7-47e2-ae87-add4aace3f38) b=bucket sync single entry (source_zone=3d163b1e-09c7-47e2-ae87-add4aace3f38) b= log_entry=00000000001.51.6 op=0 op_state=1 2023-05-31T20:38:23.559 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:23.524+0000 7f22572d6780 5 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:11]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:11]:entry[obj00000000000000000040]: bucket sync: sync obj: 3d163b1e-09c7-47e2-ae87-add4aace3f38/:pklxkd-58[3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23])/obj00000000000000000040[0] 2023-05-31T20:38:23.560 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:23.524+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:11]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:11]: listing bilog for incremental sync; position=00000000001.51.6 2023-05-31T20:38:23.560 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:23.524+0000 7f22572d6780 10 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:3]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:3]:entry[obj00000000000000000046]: success 2023-05-31T20:38:23.560 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:23.524+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:3]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:3]: updating marker marker_oid=bucket.sync-status.3d163b1e-09c7-47e2-ae87-add4aace3f38:pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:3:3 marker=00000000001.64.6 timestamp=2023-05-31T20:38:17.357924+0000 2023-05-31T20:38:23.560 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:23.525+0000 7f22572d6780 10 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:10]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:10]:entry[obj00000000000000000006]: success 2023-05-31T20:38:23.560 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:23.525+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:10]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:10]: updating marker marker_oid=bucket.sync-status.3d163b1e-09c7-47e2-ae87-add4aace3f38:pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:10:3 marker=00000000002.50.6 timestamp=2023-05-31T20:38:18.035690+0000 2023-05-31T20:38:23.561 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:23.525+0000 7f22572d6780 10 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:15]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:15]:entry[obj00000000000000000012]: success 2023-05-31T20:38:23.561 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:23.525+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:15]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:15]: updating marker marker_oid=bucket.sync-status.3d163b1e-09c7-47e2-ae87-add4aace3f38:pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:15:3 marker=00000000002.46.6 timestamp=2023-05-31T20:38:18.095753+0000 2023-05-31T20:38:23.561 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:23.525+0000 7f22572d6780 10 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:20]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:20]:entry[obj00000000000000000003]: success 2023-05-31T20:38:23.561 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:23.525+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:20]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:20]: updating marker marker_oid=bucket.sync-status.3d163b1e-09c7-47e2-ae87-add4aace3f38:pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:20:3 marker=00000000002.33.6 timestamp=2023-05-31T20:38:18.007509+0000 2023-05-31T20:38:23.561 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:23.530+0000 7f22572d6780 10 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:0]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:0]:entry[obj00000000000000000058]: success 2023-05-31T20:38:23.562 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:23.530+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:0]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:0]: updating marker marker_oid=bucket.sync-status.3d163b1e-09c7-47e2-ae87-add4aace3f38:pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:0:3 marker=00000000001.42.6 timestamp=2023-05-31T20:38:17.631771+0000 2023-05-31T20:38:23.563 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:23.536+0000 7f22572d6780 10 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:8]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:8]:entry[obj00000000000000000039]: success 2023-05-31T20:38:23.563 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:23.536+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:8]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:8]: updating marker marker_oid=bucket.sync-status.3d163b1e-09c7-47e2-ae87-add4aace3f38:pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:8:3 marker=00000000001.56.6 timestamp=2023-05-31T20:38:17.380290+0000 2023-05-31T20:38:23.564 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:23.536+0000 7f22572d6780 10 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:13]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:13]:entry[obj00000000000000000045]: success 2023-05-31T20:38:23.564 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:23.536+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:13]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:13]: updating marker marker_oid=bucket.sync-status.3d163b1e-09c7-47e2-ae87-add4aace3f38:pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:13:3 marker=00000000001.20.6 timestamp=2023-05-31T20:38:17.831535+0000 2023-05-31T20:38:23.564 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:23.549+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:12]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:12]: parsed entry: id=00000000002.44.6 iter->object=obj00000000000000000009 iter->instance= name=obj00000000000000000009 instance= ns= 2023-05-31T20:38:23.564 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:23.549+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:12]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:12]: syncing object: pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:12/obj00000000000000000009 2023-05-31T20:38:23.564 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:23.549+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:12]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:12]: entry->timestamp=2023-05-31T20:38:17.398121+0000 2023-05-31T20:38:23.565 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:23.549+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:12]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:12]:entry[obj00000000000000000009]: bucket sync single entry (source_zone=3d163b1e-09c7-47e2-ae87-add4aace3f38) b=bucket sync single entry (source_zone=3d163b1e-09c7-47e2-ae87-add4aace3f38) b= log_entry=00000000002.44.6 op=0 op_state=1 2023-05-31T20:38:23.565 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:23.549+0000 7f22572d6780 5 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:12]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:12]:entry[obj00000000000000000009]: bucket sync: sync obj: 3d163b1e-09c7-47e2-ae87-add4aace3f38/:pklxkd-58[3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23])/obj00000000000000000009[0] 2023-05-31T20:38:23.565 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:23.549+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:12]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:12]: listing bilog for incremental sync; position=00000000002.44.6 2023-05-31T20:38:23.565 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:23.550+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:7]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:7]: parsed entry: id=00000000002.20.6 iter->object=obj00000000000000000002 iter->instance= name=obj00000000000000000002 instance= ns= 2023-05-31T20:38:23.565 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:23.550+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:7]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:7]: syncing object: pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:7/obj00000000000000000002 2023-05-31T20:38:23.565 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:23.550+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:7]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:7]: entry->timestamp=2023-05-31T20:38:17.494297+0000 2023-05-31T20:38:23.566 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:23.550+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:7]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:7]:entry[obj00000000000000000002]: bucket sync single entry (source_zone=3d163b1e-09c7-47e2-ae87-add4aace3f38) b=bucket sync single entry (source_zone=3d163b1e-09c7-47e2-ae87-add4aace3f38) b= log_entry=00000000002.20.6 op=0 op_state=1 2023-05-31T20:38:23.566 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:23.550+0000 7f22572d6780 5 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:7]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:7]:entry[obj00000000000000000002]: bucket sync: sync obj: 3d163b1e-09c7-47e2-ae87-add4aace3f38/:pklxkd-58[3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23])/obj00000000000000000002[0] 2023-05-31T20:38:23.566 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:23.550+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:7]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:7]: listing bilog for incremental sync; position=00000000002.20.6 2023-05-31T20:38:23.566 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:23.551+0000 7f22572d6780 10 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:12]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:12]:entry[obj00000000000000000009]: success 2023-05-31T20:38:23.566 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:23.551+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:12]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:12]: updating marker marker_oid=bucket.sync-status.3d163b1e-09c7-47e2-ae87-add4aace3f38:pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:12:3 marker=00000000002.44.6 timestamp=2023-05-31T20:38:17.398121+0000 2023-05-31T20:38:23.566 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:23.554+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:9]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:9]: parsed entry: id=00000000002.32.6 iter->object=obj00000000000000000007 iter->instance= name=obj00000000000000000007 instance= ns= 2023-05-31T20:38:23.567 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:23.554+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:9]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:9]: syncing object: pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:9/obj00000000000000000007 2023-05-31T20:38:23.567 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:23.554+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:9]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:9]: entry->timestamp=2023-05-31T20:38:17.747773+0000 2023-05-31T20:38:23.567 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:23.554+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:9]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:9]:entry[obj00000000000000000007]: bucket sync single entry (source_zone=3d163b1e-09c7-47e2-ae87-add4aace3f38) b=bucket sync single entry (source_zone=3d163b1e-09c7-47e2-ae87-add4aace3f38) b= log_entry=00000000002.32.6 op=0 op_state=1 2023-05-31T20:38:23.567 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:23.554+0000 7f22572d6780 5 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:9]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:9]:entry[obj00000000000000000007]: bucket sync: sync obj: 3d163b1e-09c7-47e2-ae87-add4aace3f38/:pklxkd-58[3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23])/obj00000000000000000007[0] 2023-05-31T20:38:23.567 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:23.554+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:9]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:9]: listing bilog for incremental sync; position=00000000002.32.6 2023-05-31T20:38:23.567 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:23.555+0000 7f22572d6780 10 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:9]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:9]:entry[obj00000000000000000007]: success 2023-05-31T20:38:23.567 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:23.555+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:9]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:9]: updating marker marker_oid=bucket.sync-status.3d163b1e-09c7-47e2-ae87-add4aace3f38:pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:9:3 marker=00000000002.32.6 timestamp=2023-05-31T20:38:17.747773+0000 2023-05-31T20:38:23.576 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:23.571+0000 7f22572d6780 10 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:5]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:5]:entry[obj00000000000000000050]: success 2023-05-31T20:38:23.577 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:23.571+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:5]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:5]: updating marker marker_oid=bucket.sync-status.3d163b1e-09c7-47e2-ae87-add4aace3f38:pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:5:3 marker=00000000001.46.6 timestamp=2023-05-31T20:38:17.921897+0000 2023-05-31T20:38:23.577 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:23.573+0000 7f22572d6780 10 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:22]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:22]:entry[obj00000000000000000008]: success 2023-05-31T20:38:23.578 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:23.573+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:22]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:22]: updating marker marker_oid=bucket.sync-status.3d163b1e-09c7-47e2-ae87-add4aace3f38:pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:22:3 marker=00000000002.49.6 timestamp=2023-05-31T20:38:17.470631+0000 2023-05-31T20:38:23.579 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:23.575+0000 7f22572d6780 10 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:21]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:21]:entry[obj00000000000000000055]: success 2023-05-31T20:38:23.579 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:23.575+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:21]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:21]: updating marker marker_oid=bucket.sync-status.3d163b1e-09c7-47e2-ae87-add4aace3f38:pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:21:3 marker=00000000001.82.6 timestamp=2023-05-31T20:38:17.862999+0000 2023-05-31T20:38:23.581 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:23.576+0000 7f22572d6780 10 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:11]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:11]:entry[obj00000000000000000040]: success 2023-05-31T20:38:23.581 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:23.576+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:11]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:11]: updating marker marker_oid=bucket.sync-status.3d163b1e-09c7-47e2-ae87-add4aace3f38:pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:11:3 marker=00000000001.51.6 timestamp=2023-05-31T20:38:17.552106+0000 2023-05-31T20:38:23.608 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:23.604+0000 7f22572d6780 10 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:7]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:7]:entry[obj00000000000000000002]: success 2023-05-31T20:38:23.608 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:23.604+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:7]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:7]: updating marker marker_oid=bucket.sync-status.3d163b1e-09c7-47e2-ae87-add4aace3f38:pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:7:3 marker=00000000002.20.6 timestamp=2023-05-31T20:38:17.494297+0000 2023-05-31T20:38:23.666 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:23.660+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:1]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-05-31T20:38:23.666 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:23.660+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:0]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-05-31T20:38:23.667 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:23.661+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:2]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-05-31T20:38:23.667 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:23.661+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:1]: sync status for source bucket shard: 0 2023-05-31T20:38:23.668 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:23.661+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:1]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:1]: listing bilog for incremental sync; position= 2023-05-31T20:38:23.668 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:23.661+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:0]: sync status for source bucket shard: 0 2023-05-31T20:38:23.668 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:23.661+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:0]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:0]: listing bilog for incremental sync; position= 2023-05-31T20:38:23.669 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:23.661+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:3]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-05-31T20:38:23.669 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:23.661+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:2]: sync status for source bucket shard: 0 2023-05-31T20:38:23.669 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:23.661+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:2]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:2]: listing bilog for incremental sync; position= 2023-05-31T20:38:23.670 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:23.661+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:5]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-05-31T20:38:23.673 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:23.661+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:3]: sync status for source bucket shard: 0 2023-05-31T20:38:23.673 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:23.661+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:3]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:3]: listing bilog for incremental sync; position= 2023-05-31T20:38:23.673 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:23.661+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:4]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-05-31T20:38:23.673 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:23.662+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:4]: sync status for source bucket shard: 0 2023-05-31T20:38:23.673 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:23.662+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:7]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-05-31T20:38:23.674 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:23.662+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:4]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:4]: listing bilog for incremental sync; position= 2023-05-31T20:38:23.674 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:23.662+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:1]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:1]: parsed entry: id=00000000001.42.6 iter->object=obj00000000000000000059 iter->instance= name=obj00000000000000000059 instance= ns= 2023-05-31T20:38:23.674 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:23.662+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:1]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:1]: syncing object: pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:1/obj00000000000000000059 2023-05-31T20:38:23.674 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:23.662+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:1]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:1]: entry->timestamp=2023-05-31T20:38:18.560629+0000 2023-05-31T20:38:23.674 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:23.662+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:1]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:1]:entry[obj00000000000000000059]: bucket sync single entry (source_zone=3d163b1e-09c7-47e2-ae87-add4aace3f38) b=bucket sync single entry (source_zone=3d163b1e-09c7-47e2-ae87-add4aace3f38) b= log_entry=00000000001.42.6 op=0 op_state=1 2023-05-31T20:38:23.674 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:23.662+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:6]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-05-31T20:38:23.675 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:23.662+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:10]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-05-31T20:38:23.675 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:23.662+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:8]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-05-31T20:38:23.675 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:23.662+0000 7f22572d6780 5 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:1]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:1]:entry[obj00000000000000000059]: bucket sync: sync obj: 3d163b1e-09c7-47e2-ae87-add4aace3f38/:pklxkd-58[3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23])/obj00000000000000000059[0] 2023-05-31T20:38:23.675 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:23.662+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:1]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:1]: listing bilog for incremental sync; position=00000000001.42.6 2023-05-31T20:38:23.675 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:23.663+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:5]: sync status for source bucket shard: 0 2023-05-31T20:38:23.675 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:23.663+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:7]: sync status for source bucket shard: 0 2023-05-31T20:38:23.676 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:23.663+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:0]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:0]: parsed entry: id=00000000002.39.6 iter->object=obj00000000000000000006 iter->instance= name=obj00000000000000000006 instance= ns= 2023-05-31T20:38:23.676 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:23.663+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:0]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:0]: syncing object: pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:0/obj00000000000000000006 2023-05-31T20:38:23.677 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:23.663+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:0]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:0]: entry->timestamp=2023-05-31T20:38:19.104820+0000 2023-05-31T20:38:23.677 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:23.663+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:0]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:0]:entry[obj00000000000000000006]: bucket sync single entry (source_zone=3d163b1e-09c7-47e2-ae87-add4aace3f38) b=bucket sync single entry (source_zone=3d163b1e-09c7-47e2-ae87-add4aace3f38) b= log_entry=00000000002.39.6 op=0 op_state=1 2023-05-31T20:38:23.677 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:23.663+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:5]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:5]: listing bilog for incremental sync; position= 2023-05-31T20:38:23.677 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:23.663+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:7]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:7]: listing bilog for incremental sync; position= 2023-05-31T20:38:23.677 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:23.663+0000 7f22572d6780 0 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:0]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:0]:entry[obj00000000000000000006]: : injecting data sync error on key=obj00000000000000000006 2023-05-31T20:38:23.678 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:23.663+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:0]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:0]: listing bilog for incremental sync; position=00000000002.39.6 2023-05-31T20:38:23.678 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:23.663+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:2]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:2]: parsed entry: id=00000000001.49.6 iter->object=obj00000000000000000027 iter->instance= name=obj00000000000000000027 instance= ns= 2023-05-31T20:38:23.678 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:23.663+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:2]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:2]: syncing object: pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:2/obj00000000000000000027 2023-05-31T20:38:23.678 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:23.663+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:2]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:2]: entry->timestamp=2023-05-31T20:38:19.074615+0000 2023-05-31T20:38:23.678 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:23.663+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:2]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:2]:entry[obj00000000000000000027]: bucket sync single entry (source_zone=3d163b1e-09c7-47e2-ae87-add4aace3f38) b=bucket sync single entry (source_zone=3d163b1e-09c7-47e2-ae87-add4aace3f38) b= log_entry=00000000001.49.6 op=0 op_state=1 2023-05-31T20:38:23.679 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:23.663+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:3]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:3]: parsed entry: id=00000000002.40.6 iter->object=obj00000000000000000004 iter->instance= name=obj00000000000000000004 instance= ns= 2023-05-31T20:38:23.679 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:23.663+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:3]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:3]: syncing object: pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:3/obj00000000000000000004 2023-05-31T20:38:23.679 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:23.663+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:3]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:3]: entry->timestamp=2023-05-31T20:38:18.727945+0000 2023-05-31T20:38:23.679 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:23.663+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:3]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:3]:entry[obj00000000000000000004]: bucket sync single entry (source_zone=3d163b1e-09c7-47e2-ae87-add4aace3f38) b=bucket sync single entry (source_zone=3d163b1e-09c7-47e2-ae87-add4aace3f38) b= log_entry=00000000002.40.6 op=0 op_state=1 2023-05-31T20:38:23.679 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:23.663+0000 7f22572d6780 10 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:0]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:0]:entry[obj00000000000000000006]: failed, retcode=-5 ((5) Input/output error) 2023-05-31T20:38:23.679 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:23.663+0000 7f22572d6780 0 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:0]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:0]:entry[obj00000000000000000006]: ERROR: failed to sync object: pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:0/obj00000000000000000006 2023-05-31T20:38:23.679 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:23.663+0000 7f22572d6780 5 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:2]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:2]:entry[obj00000000000000000027]: bucket sync: sync obj: 3d163b1e-09c7-47e2-ae87-add4aace3f38/:pklxkd-58[3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23])/obj00000000000000000027[0] 2023-05-31T20:38:23.680 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:23.663+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:2]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:2]: listing bilog for incremental sync; position=00000000001.49.6 2023-05-31T20:38:23.680 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:23.663+0000 7f22572d6780 5 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:3]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:3]:entry[obj00000000000000000004]: bucket sync: sync obj: 3d163b1e-09c7-47e2-ae87-add4aace3f38/:pklxkd-58[3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23])/obj00000000000000000004[0] 2023-05-31T20:38:23.680 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:23.663+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:3]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:3]: listing bilog for incremental sync; position=00000000002.40.6 2023-05-31T20:38:23.680 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:23.665+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:6]: sync status for source bucket shard: 0 2023-05-31T20:38:23.681 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:23.665+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:8]: sync status for source bucket shard: 0 2023-05-31T20:38:23.681 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:23.665+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:9]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-05-31T20:38:23.681 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:23.665+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:6]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:6]: listing bilog for incremental sync; position= 2023-05-31T20:38:23.681 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:23.665+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:8]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:8]: listing bilog for incremental sync; position= 2023-05-31T20:38:23.681 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:23.665+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:11]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-05-31T20:38:23.681 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:23.665+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:12]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-05-31T20:38:23.682 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:23.665+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:13]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-05-31T20:38:23.682 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:23.665+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:15]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-05-31T20:38:23.682 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:23.665+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:14]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-05-31T20:38:23.683 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:23.665+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:17]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-05-31T20:38:23.683 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:23.665+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:16]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-05-31T20:38:23.683 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:23.665+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:18]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-05-31T20:38:23.683 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:23.665+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:19]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-05-31T20:38:23.683 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:23.665+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:20]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-05-31T20:38:23.683 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:23.666+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:21]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-05-31T20:38:23.683 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:23.666+0000 7f22572d6780 10 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:3]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:3]:entry[obj00000000000000000004]: success 2023-05-31T20:38:23.684 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:23.666+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:3]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:3]: updating marker marker_oid=bucket.sync-status.3d163b1e-09c7-47e2-ae87-add4aace3f38:pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:3:4 marker=00000000002.40.6 timestamp=2023-05-31T20:38:18.727945+0000 2023-05-31T20:38:23.684 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:23.667+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:13]: sync status for source bucket shard: 0 2023-05-31T20:38:23.684 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:23.667+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:20]: sync status for source bucket shard: 0 2023-05-31T20:38:23.684 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:23.667+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:17]: sync status for source bucket shard: 0 2023-05-31T20:38:23.684 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:23.667+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:23]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-05-31T20:38:23.685 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:23.667+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:13]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:13]: listing bilog for incremental sync; position= 2023-05-31T20:38:23.685 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:23.667+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:20]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:20]: listing bilog for incremental sync; position= 2023-05-31T20:38:23.685 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:23.667+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:17]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:17]: listing bilog for incremental sync; position= 2023-05-31T20:38:23.685 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:23.667+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:25]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-05-31T20:38:23.685 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:23.667+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:24]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-05-31T20:38:23.686 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:23.667+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:18]: sync status for source bucket shard: 0 2023-05-31T20:38:23.686 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:23.667+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:18]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:18]: listing bilog for incremental sync; position= 2023-05-31T20:38:23.686 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:23.667+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:21]: sync status for source bucket shard: 0 2023-05-31T20:38:23.686 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:23.667+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:21]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:21]: listing bilog for incremental sync; position= 2023-05-31T20:38:23.686 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:23.667+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:8]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:8]: parsed entry: id=00000000001.52.6 iter->object=obj00000000000000000017 iter->instance= name=obj00000000000000000017 instance= ns= 2023-05-31T20:38:23.686 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:23.667+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:8]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:8]: syncing object: pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:8/obj00000000000000000017 2023-05-31T20:38:23.687 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:23.667+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:8]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:8]: entry->timestamp=2023-05-31T20:38:18.463637+0000 2023-05-31T20:38:23.687 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:23.667+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:8]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:8]:entry[obj00000000000000000017]: bucket sync single entry (source_zone=3d163b1e-09c7-47e2-ae87-add4aace3f38) b=bucket sync single entry (source_zone=3d163b1e-09c7-47e2-ae87-add4aace3f38) b= log_entry=00000000001.52.6 op=0 op_state=1 2023-05-31T20:38:23.687 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:23.668+0000 7f22572d6780 0 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:8]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:8]:entry[obj00000000000000000017]: : injecting data sync error on key=obj00000000000000000017 2023-05-31T20:38:23.687 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:23.668+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:8]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:8]: listing bilog for incremental sync; position=00000000001.52.6 2023-05-31T20:38:23.687 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:23.668+0000 7f22572d6780 10 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:8]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:8]:entry[obj00000000000000000017]: failed, retcode=-5 ((5) Input/output error) 2023-05-31T20:38:23.688 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:23.668+0000 7f22572d6780 0 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:8]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:8]:entry[obj00000000000000000017]: ERROR: failed to sync object: pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:8/obj00000000000000000017 2023-05-31T20:38:23.688 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:23.668+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:9]: sync status for source bucket shard: 0 2023-05-31T20:38:23.688 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:23.668+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:9]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:9]: listing bilog for incremental sync; position= 2023-05-31T20:38:23.688 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:23.668+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:26]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-05-31T20:38:23.688 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:23.668+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:22]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-05-31T20:38:23.691 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:23.668+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:25]: sync status for source bucket shard: 0 2023-05-31T20:38:23.691 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:23.668+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:25]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:25]: listing bilog for incremental sync; position= 2023-05-31T20:38:23.691 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:23.668+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:24]: sync status for source bucket shard: 0 2023-05-31T20:38:23.692 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:23.668+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:24]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:24]: listing bilog for incremental sync; position= 2023-05-31T20:38:23.692 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:23.669+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:10]: sync status for source bucket shard: 0 2023-05-31T20:38:23.692 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:23.669+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:11]: sync status for source bucket shard: 0 2023-05-31T20:38:23.692 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:23.669+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:14]: sync status for source bucket shard: 0 2023-05-31T20:38:23.692 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:23.669+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:12]: sync status for source bucket shard: 0 2023-05-31T20:38:23.692 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:23.669+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:19]: sync status for source bucket shard: 0 2023-05-31T20:38:23.693 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:23.669+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:15]: sync status for source bucket shard: 0 2023-05-31T20:38:23.693 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:23.669+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:16]: sync status for source bucket shard: 0 2023-05-31T20:38:23.693 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:23.669+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:27]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-05-31T20:38:23.693 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:23.669+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:10]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:10]: listing bilog for incremental sync; position= 2023-05-31T20:38:23.693 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:23.669+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:11]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:11]: listing bilog for incremental sync; position= 2023-05-31T20:38:23.693 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:23.669+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:14]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:14]: listing bilog for incremental sync; position= 2023-05-31T20:38:23.693 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:23.669+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:12]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:12]: listing bilog for incremental sync; position= 2023-05-31T20:38:23.694 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:23.669+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:19]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:19]: listing bilog for incremental sync; position= 2023-05-31T20:38:23.694 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:23.669+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:15]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:15]: listing bilog for incremental sync; position= 2023-05-31T20:38:23.694 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:23.669+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:16]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:16]: listing bilog for incremental sync; position= 2023-05-31T20:38:23.694 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:23.669+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:28]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-05-31T20:38:23.694 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:23.669+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:17]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:17]: parsed entry: id=00000000002.58.6 iter->object=obj00000000000000000012 iter->instance= name=obj00000000000000000012 instance= ns= 2023-05-31T20:38:23.695 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:23.669+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:17]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:17]: syncing object: pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:17/obj00000000000000000012 2023-05-31T20:38:23.695 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:23.669+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:17]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:17]: entry->timestamp=2023-05-31T20:38:19.444473+0000 2023-05-31T20:38:23.695 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:23.669+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:17]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:17]:entry[obj00000000000000000012]: bucket sync single entry (source_zone=3d163b1e-09c7-47e2-ae87-add4aace3f38) b=bucket sync single entry (source_zone=3d163b1e-09c7-47e2-ae87-add4aace3f38) b= log_entry=00000000002.58.6 op=0 op_state=1 2023-05-31T20:38:23.695 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:23.669+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:23]: sync status for source bucket shard: 0 2023-05-31T20:38:23.695 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:23.670+0000 7f22572d6780 5 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:17]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:17]:entry[obj00000000000000000012]: bucket sync: sync obj: 3d163b1e-09c7-47e2-ae87-add4aace3f38/:pklxkd-58[3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23])/obj00000000000000000012[0] 2023-05-31T20:38:23.696 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:23.670+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:17]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:17]: listing bilog for incremental sync; position=00000000002.58.6 2023-05-31T20:38:23.696 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:23.670+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:23]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:23]: listing bilog for incremental sync; position= 2023-05-31T20:38:23.696 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:23.670+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:22]: sync status for source bucket shard: 0 2023-05-31T20:38:23.696 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:23.670+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:26]: sync status for source bucket shard: 0 2023-05-31T20:38:23.696 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:23.671+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:22]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:22]: listing bilog for incremental sync; position= 2023-05-31T20:38:23.697 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:23.671+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:26]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:26]: listing bilog for incremental sync; position= 2023-05-31T20:38:23.697 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:23.671+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:27]: sync status for source bucket shard: 0 2023-05-31T20:38:23.697 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:23.672+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:27]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:27]: listing bilog for incremental sync; position= 2023-05-31T20:38:23.697 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:23.672+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:13]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:13]: parsed entry: id=00000000002.69.6 iter->object=obj00000000000000000013 iter->instance= name=obj00000000000000000013 instance= ns= 2023-05-31T20:38:23.697 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:23.672+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:13]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:13]: syncing object: pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:13/obj00000000000000000013 2023-05-31T20:38:23.698 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:23.672+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:13]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:13]: entry->timestamp=2023-05-31T20:38:18.916325+0000 2023-05-31T20:38:23.698 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:23.672+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:13]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:13]:entry[obj00000000000000000013]: bucket sync single entry (source_zone=3d163b1e-09c7-47e2-ae87-add4aace3f38) b=bucket sync single entry (source_zone=3d163b1e-09c7-47e2-ae87-add4aace3f38) b= log_entry=00000000002.69.6 op=0 op_state=1 2023-05-31T20:38:23.698 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:23.672+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:18]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:18]: parsed entry: id=00000000002.51.6 iter->object=obj00000000000000000003 iter->instance= name=obj00000000000000000003 instance= ns= 2023-05-31T20:38:23.698 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:23.672+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:18]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:18]: syncing object: pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:18/obj00000000000000000003 2023-05-31T20:38:23.698 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:23.672+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:18]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:18]: entry->timestamp=2023-05-31T20:38:19.569846+0000 2023-05-31T20:38:23.698 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:23.672+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:18]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:18]:entry[obj00000000000000000003]: bucket sync single entry (source_zone=3d163b1e-09c7-47e2-ae87-add4aace3f38) b=bucket sync single entry (source_zone=3d163b1e-09c7-47e2-ae87-add4aace3f38) b= log_entry=00000000002.51.6 op=0 op_state=1 2023-05-31T20:38:23.699 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:23.672+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:21]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:21]: parsed entry: id=00000000002.60.6 iter->object=obj00000000000000000001 iter->instance= name=obj00000000000000000001 instance= ns= 2023-05-31T20:38:23.699 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:23.672+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:21]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:21]: syncing object: pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:21/obj00000000000000000001 2023-05-31T20:38:23.699 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:23.672+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:21]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:21]: entry->timestamp=2023-05-31T20:38:19.634831+0000 2023-05-31T20:38:23.700 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:23.672+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:21]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:21]:entry[obj00000000000000000001]: bucket sync single entry (source_zone=3d163b1e-09c7-47e2-ae87-add4aace3f38) b=bucket sync single entry (source_zone=3d163b1e-09c7-47e2-ae87-add4aace3f38) b= log_entry=00000000002.60.6 op=0 op_state=1 2023-05-31T20:38:23.700 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:23.672+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:9]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:9]: parsed entry: id=00000000002.38.6 iter->object=obj00000000000000000008 iter->instance= name=obj00000000000000000008 instance= ns= 2023-05-31T20:38:23.700 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:23.672+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:9]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:9]: syncing object: pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:9/obj00000000000000000008 2023-05-31T20:38:23.701 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:23.672+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:9]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:9]: entry->timestamp=2023-05-31T20:38:18.896585+0000 2023-05-31T20:38:23.701 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:23.672+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:9]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:9]:entry[obj00000000000000000008]: bucket sync single entry (source_zone=3d163b1e-09c7-47e2-ae87-add4aace3f38) b=bucket sync single entry (source_zone=3d163b1e-09c7-47e2-ae87-add4aace3f38) b= log_entry=00000000002.38.6 op=0 op_state=1 2023-05-31T20:38:23.701 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:23.672+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:24]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:24]: parsed entry: id=00000000002.52.6 iter->object=obj00000000000000000016 iter->instance= name=obj00000000000000000016 instance= ns= 2023-05-31T20:38:23.702 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:23.672+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:24]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:24]: syncing object: pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:24/obj00000000000000000016 2023-05-31T20:38:23.702 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:23.672+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:24]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:24]: entry->timestamp=2023-05-31T20:38:19.295411+0000 2023-05-31T20:38:23.702 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:23.672+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:24]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:24]:entry[obj00000000000000000016]: bucket sync single entry (source_zone=3d163b1e-09c7-47e2-ae87-add4aace3f38) b=bucket sync single entry (source_zone=3d163b1e-09c7-47e2-ae87-add4aace3f38) b= log_entry=00000000002.52.6 op=0 op_state=1 2023-05-31T20:38:23.703 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:23.672+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:28]: sync status for source bucket shard: 0 2023-05-31T20:38:23.703 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:23.672+0000 7f22572d6780 0 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:13]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:13]:entry[obj00000000000000000013]: : injecting data sync error on key=obj00000000000000000013 2023-05-31T20:38:23.703 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:23.672+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:13]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:13]: listing bilog for incremental sync; position=00000000002.69.6 2023-05-31T20:38:23.704 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:23.672+0000 7f22572d6780 5 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:18]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:18]:entry[obj00000000000000000003]: bucket sync: sync obj: 3d163b1e-09c7-47e2-ae87-add4aace3f38/:pklxkd-58[3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23])/obj00000000000000000003[0] 2023-05-31T20:38:23.704 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:23.672+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:18]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:18]: listing bilog for incremental sync; position=00000000002.51.6 2023-05-31T20:38:23.704 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:23.672+0000 7f22572d6780 5 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:21]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:21]:entry[obj00000000000000000001]: bucket sync: sync obj: 3d163b1e-09c7-47e2-ae87-add4aace3f38/:pklxkd-58[3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23])/obj00000000000000000001[0] 2023-05-31T20:38:23.705 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:23.672+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:21]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:21]: listing bilog for incremental sync; position=00000000002.60.6 2023-05-31T20:38:23.705 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:23.672+0000 7f22572d6780 5 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:9]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:9]:entry[obj00000000000000000008]: bucket sync: sync obj: 3d163b1e-09c7-47e2-ae87-add4aace3f38/:pklxkd-58[3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23])/obj00000000000000000008[0] 2023-05-31T20:38:23.705 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:23.672+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:9]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:9]: listing bilog for incremental sync; position=00000000002.38.6 2023-05-31T20:38:23.706 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:23.672+0000 7f22572d6780 5 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:24]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:24]:entry[obj00000000000000000016]: bucket sync: sync obj: 3d163b1e-09c7-47e2-ae87-add4aace3f38/:pklxkd-58[3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23])/obj00000000000000000016[0] 2023-05-31T20:38:23.706 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:23.672+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:24]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:24]: listing bilog for incremental sync; position=00000000002.52.6 2023-05-31T20:38:23.707 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:23.672+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:28]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:28]: listing bilog for incremental sync; position= 2023-05-31T20:38:23.707 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:23.672+0000 7f22572d6780 10 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:13]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:13]:entry[obj00000000000000000013]: failed, retcode=-5 ((5) Input/output error) 2023-05-31T20:38:23.707 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:23.672+0000 7f22572d6780 0 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:13]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:13]:entry[obj00000000000000000013]: ERROR: failed to sync object: pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:13/obj00000000000000000013 2023-05-31T20:38:23.707 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:23.674+0000 7f22572d6780 10 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:17]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:17]:entry[obj00000000000000000012]: success 2023-05-31T20:38:23.708 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:23.674+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:17]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:17]: updating marker marker_oid=bucket.sync-status.3d163b1e-09c7-47e2-ae87-add4aace3f38:pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:17:4 marker=00000000002.58.6 timestamp=2023-05-31T20:38:19.444473+0000 2023-05-31T20:38:23.708 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:23.674+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:25]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:25]: parsed entry: id=00000000002.46.6 iter->object=obj00000000000000000000 iter->instance= name=obj00000000000000000000 instance= ns= 2023-05-31T20:38:23.708 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:23.674+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:25]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:25]: syncing object: pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:25/obj00000000000000000000 2023-05-31T20:38:23.708 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:23.674+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:25]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:25]: entry->timestamp=2023-05-31T20:38:19.538360+0000 2023-05-31T20:38:23.709 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:23.674+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:25]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:25]:entry[obj00000000000000000000]: bucket sync single entry (source_zone=3d163b1e-09c7-47e2-ae87-add4aace3f38) b=bucket sync single entry (source_zone=3d163b1e-09c7-47e2-ae87-add4aace3f38) b= log_entry=00000000002.46.6 op=0 op_state=1 2023-05-31T20:38:23.709 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:23.674+0000 7f22572d6780 5 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:25]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:25]:entry[obj00000000000000000000]: bucket sync: sync obj: 3d163b1e-09c7-47e2-ae87-add4aace3f38/:pklxkd-58[3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23])/obj00000000000000000000[0] 2023-05-31T20:38:23.709 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:23.674+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:25]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:25]: listing bilog for incremental sync; position=00000000002.46.6 2023-05-31T20:38:23.710 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:23.675+0000 7f22572d6780 10 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:8]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:8]: a sync operation returned error 2023-05-31T20:38:23.710 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:23.675+0000 7f22572d6780 10 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:8]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:8]: backing out with sync_status=-5 2023-05-31T20:38:23.710 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:23.675+0000 7f22572d6780 5 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:8]: incremental sync on bucket failed, retcode=-5 2023-05-31T20:38:23.710 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:23.675+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:8]: ERROR: incremental sync failed. error: -5 2023-05-31T20:38:23.711 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:23.677+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:26]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:26]: parsed entry: id=00000000002.78.6 iter->object=obj00000000000000000014 iter->instance= name=obj00000000000000000014 instance= ns= 2023-05-31T20:38:23.711 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:23.677+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:26]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:26]: syncing object: pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:26/obj00000000000000000014 2023-05-31T20:38:23.711 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:23.677+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:26]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:26]: entry->timestamp=2023-05-31T20:38:19.474789+0000 2023-05-31T20:38:23.711 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:23.677+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:26]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:26]:entry[obj00000000000000000014]: bucket sync single entry (source_zone=3d163b1e-09c7-47e2-ae87-add4aace3f38) b=bucket sync single entry (source_zone=3d163b1e-09c7-47e2-ae87-add4aace3f38) b= log_entry=00000000002.78.6 op=0 op_state=1 2023-05-31T20:38:23.711 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:23.677+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:23]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:23]: parsed entry: id=00000000002.49.6 iter->object=obj00000000000000000025 iter->instance= name=obj00000000000000000025 instance= ns= 2023-05-31T20:38:23.711 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:23.677+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:23]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:23]: syncing object: pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:23/obj00000000000000000025 2023-05-31T20:38:23.712 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:23.677+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:23]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:23]: entry->timestamp=2023-05-31T20:38:19.001294+0000 2023-05-31T20:38:23.712 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:23.677+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:23]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:23]:entry[obj00000000000000000025]: bucket sync single entry (source_zone=3d163b1e-09c7-47e2-ae87-add4aace3f38) b=bucket sync single entry (source_zone=3d163b1e-09c7-47e2-ae87-add4aace3f38) b= log_entry=00000000002.49.6 op=0 op_state=1 2023-05-31T20:38:23.712 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:23.677+0000 7f22572d6780 5 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:26]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:26]:entry[obj00000000000000000014]: bucket sync: sync obj: 3d163b1e-09c7-47e2-ae87-add4aace3f38/:pklxkd-58[3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23])/obj00000000000000000014[0] 2023-05-31T20:38:23.712 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:23.677+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:26]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:26]: listing bilog for incremental sync; position=00000000002.78.6 2023-05-31T20:38:23.712 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:23.677+0000 7f22572d6780 0 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:23]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:23]:entry[obj00000000000000000025]: : injecting data sync error on key=obj00000000000000000025 2023-05-31T20:38:23.712 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:23.677+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:23]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:23]: listing bilog for incremental sync; position=00000000002.49.6 2023-05-31T20:38:23.713 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:23.677+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:22]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:22]: parsed entry: id=00000000002.56.6 iter->object=obj00000000000000000009 iter->instance= name=obj00000000000000000009 instance= ns= 2023-05-31T20:38:23.713 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:23.677+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:22]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:22]: syncing object: pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:22/obj00000000000000000009 2023-05-31T20:38:23.713 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:23.677+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:22]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:22]: entry->timestamp=2023-05-31T20:38:19.263885+0000 2023-05-31T20:38:23.713 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:23.677+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:22]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:22]:entry[obj00000000000000000009]: bucket sync single entry (source_zone=3d163b1e-09c7-47e2-ae87-add4aace3f38) b=bucket sync single entry (source_zone=3d163b1e-09c7-47e2-ae87-add4aace3f38) b= log_entry=00000000002.56.6 op=0 op_state=1 2023-05-31T20:38:23.713 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:23.677+0000 7f22572d6780 10 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:23]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:23]:entry[obj00000000000000000025]: failed, retcode=-5 ((5) Input/output error) 2023-05-31T20:38:23.713 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:23.677+0000 7f22572d6780 0 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:23]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:23]:entry[obj00000000000000000025]: ERROR: failed to sync object: pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:23/obj00000000000000000025 2023-05-31T20:38:23.714 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:23.677+0000 7f22572d6780 5 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:22]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:22]:entry[obj00000000000000000009]: bucket sync: sync obj: 3d163b1e-09c7-47e2-ae87-add4aace3f38/:pklxkd-58[3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23])/obj00000000000000000009[0] 2023-05-31T20:38:23.714 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:23.677+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:22]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:22]: listing bilog for incremental sync; position=00000000002.56.6 2023-05-31T20:38:23.714 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:23.678+0000 7f22572d6780 10 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:9]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:9]:entry[obj00000000000000000008]: success 2023-05-31T20:38:23.714 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:23.678+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:9]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:9]: updating marker marker_oid=bucket.sync-status.3d163b1e-09c7-47e2-ae87-add4aace3f38:pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:9:4 marker=00000000002.38.6 timestamp=2023-05-31T20:38:18.896585+0000 2023-05-31T20:38:23.714 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:23.678+0000 7f22572d6780 10 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:24]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:24]:entry[obj00000000000000000016]: success 2023-05-31T20:38:23.714 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:23.678+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:24]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:24]: updating marker marker_oid=bucket.sync-status.3d163b1e-09c7-47e2-ae87-add4aace3f38:pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:24:4 marker=00000000002.52.6 timestamp=2023-05-31T20:38:19.295411+0000 2023-05-31T20:38:23.714 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:23.678+0000 7f22572d6780 10 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:13]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:13]: a sync operation returned error 2023-05-31T20:38:23.715 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:23.678+0000 7f22572d6780 10 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:13]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:13]: backing out with sync_status=-5 2023-05-31T20:38:23.715 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:23.678+0000 7f22572d6780 5 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:13]: incremental sync on bucket failed, retcode=-5 2023-05-31T20:38:23.715 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:23.678+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:13]: ERROR: incremental sync failed. error: -5 2023-05-31T20:38:23.715 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:23.679+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:28]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:28]: parsed entry: id=00000000002.92.6 iter->object=obj00000000000000000021 iter->instance= name=obj00000000000000000021 instance= ns= 2023-05-31T20:38:23.715 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:23.679+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:28]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:28]: syncing object: pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:28/obj00000000000000000021 2023-05-31T20:38:23.716 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:23.679+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:28]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:28]: entry->timestamp=2023-05-31T20:38:18.756566+0000 2023-05-31T20:38:23.716 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:23.679+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:28]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:28]:entry[obj00000000000000000021]: bucket sync single entry (source_zone=3d163b1e-09c7-47e2-ae87-add4aace3f38) b=bucket sync single entry (source_zone=3d163b1e-09c7-47e2-ae87-add4aace3f38) b= log_entry=00000000002.92.6 op=0 op_state=1 2023-05-31T20:38:23.716 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:23.679+0000 7f22572d6780 10 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:21]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:21]:entry[obj00000000000000000001]: success 2023-05-31T20:38:23.716 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:23.679+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:21]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:21]: updating marker marker_oid=bucket.sync-status.3d163b1e-09c7-47e2-ae87-add4aace3f38:pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:21:4 marker=00000000002.60.6 timestamp=2023-05-31T20:38:19.634831+0000 2023-05-31T20:38:23.717 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:23.679+0000 7f22572d6780 10 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:18]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:18]:entry[obj00000000000000000003]: success 2023-05-31T20:38:23.717 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:23.679+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:18]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:18]: updating marker marker_oid=bucket.sync-status.3d163b1e-09c7-47e2-ae87-add4aace3f38:pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:18:4 marker=00000000002.51.6 timestamp=2023-05-31T20:38:19.569846+0000 2023-05-31T20:38:23.717 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:23.679+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:27]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:27]: parsed entry: id=00000000002.94.6 iter->object=obj00000000000000000030 iter->instance= name=obj00000000000000000030 instance= ns= 2023-05-31T20:38:23.717 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:23.679+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:27]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:27]: syncing object: pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:27/obj00000000000000000030 2023-05-31T20:38:23.717 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:23.679+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:27]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:27]: entry->timestamp=2023-05-31T20:38:18.839732+0000 2023-05-31T20:38:23.717 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:23.679+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:27]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:27]:entry[obj00000000000000000030]: bucket sync single entry (source_zone=3d163b1e-09c7-47e2-ae87-add4aace3f38) b=bucket sync single entry (source_zone=3d163b1e-09c7-47e2-ae87-add4aace3f38) b= log_entry=00000000002.94.6 op=0 op_state=1 2023-05-31T20:38:23.718 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:23.679+0000 7f22572d6780 5 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:28]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:28]:entry[obj00000000000000000021]: bucket sync: sync obj: 3d163b1e-09c7-47e2-ae87-add4aace3f38/:pklxkd-58[3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23])/obj00000000000000000021[0] 2023-05-31T20:38:23.718 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:23.679+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:28]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:28]: listing bilog for incremental sync; position=00000000002.92.6 2023-05-31T20:38:23.718 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:23.679+0000 7f22572d6780 5 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:27]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:27]:entry[obj00000000000000000030]: bucket sync: sync obj: 3d163b1e-09c7-47e2-ae87-add4aace3f38/:pklxkd-58[3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23])/obj00000000000000000030[0] 2023-05-31T20:38:23.718 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:23.679+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:27]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:27]: listing bilog for incremental sync; position=00000000002.94.6 2023-05-31T20:38:23.718 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:23.680+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:8]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-05-31T20:38:23.719 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:23.680+0000 7f22572d6780 10 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:26]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:26]:entry[obj00000000000000000014]: success 2023-05-31T20:38:23.719 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:23.680+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:26]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:26]: updating marker marker_oid=bucket.sync-status.3d163b1e-09c7-47e2-ae87-add4aace3f38:pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:26:4 marker=00000000002.78.6 timestamp=2023-05-31T20:38:19.474789+0000 2023-05-31T20:38:23.719 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:23.681+0000 7f22572d6780 10 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:25]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:25]:entry[obj00000000000000000000]: success 2023-05-31T20:38:23.719 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:23.681+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:25]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:25]: updating marker marker_oid=bucket.sync-status.3d163b1e-09c7-47e2-ae87-add4aace3f38:pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:25:4 marker=00000000002.46.6 timestamp=2023-05-31T20:38:19.538360+0000 2023-05-31T20:38:23.720 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:23.681+0000 7f22572d6780 10 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:22]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:22]:entry[obj00000000000000000009]: success 2023-05-31T20:38:23.720 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:23.681+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:22]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:22]: updating marker marker_oid=bucket.sync-status.3d163b1e-09c7-47e2-ae87-add4aace3f38:pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:22:4 marker=00000000002.56.6 timestamp=2023-05-31T20:38:19.263885+0000 2023-05-31T20:38:23.720 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:23.683+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:8]: sync status for source bucket shard: 0 2023-05-31T20:38:23.720 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:23.683+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:8]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:8]: listing bilog for incremental sync; position= 2023-05-31T20:38:23.721 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:23.683+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:13]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-05-31T20:38:23.721 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:23.683+0000 7f22572d6780 10 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:1]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:1]:entry[obj00000000000000000059]: success 2023-05-31T20:38:23.721 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:23.683+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:1]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:1]: updating marker marker_oid=bucket.sync-status.3d163b1e-09c7-47e2-ae87-add4aace3f38:pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:1:4 marker=00000000001.42.6 timestamp=2023-05-31T20:38:18.560629+0000 2023-05-31T20:38:23.721 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:23.683+0000 7f22572d6780 10 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:27]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:27]:entry[obj00000000000000000030]: success 2023-05-31T20:38:23.721 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:23.683+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:27]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:27]: updating marker marker_oid=bucket.sync-status.3d163b1e-09c7-47e2-ae87-add4aace3f38:pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:27:4 marker=00000000002.94.6 timestamp=2023-05-31T20:38:18.839732+0000 2023-05-31T20:38:23.722 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:23.683+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:13]: sync status for source bucket shard: 0 2023-05-31T20:38:23.722 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:23.683+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:13]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:13]: listing bilog for incremental sync; position= 2023-05-31T20:38:23.722 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:23.684+0000 7f22572d6780 10 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:28]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:28]:entry[obj00000000000000000021]: success 2023-05-31T20:38:23.722 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:23.684+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:28]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:28]: updating marker marker_oid=bucket.sync-status.3d163b1e-09c7-47e2-ae87-add4aace3f38:pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:28:4 marker=00000000002.92.6 timestamp=2023-05-31T20:38:18.756566+0000 2023-05-31T20:38:23.722 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:23.685+0000 7f22572d6780 10 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:2]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:2]:entry[obj00000000000000000027]: success 2023-05-31T20:38:23.722 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:23.685+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:2]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:2]: updating marker marker_oid=bucket.sync-status.3d163b1e-09c7-47e2-ae87-add4aace3f38:pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:2:4 marker=00000000001.49.6 timestamp=2023-05-31T20:38:19.074615+0000 2023-05-31T20:38:23.725 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:23.706+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:7]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:7]: parsed entry: id=00000000001.90.6 iter->object=obj00000000000000000049 iter->instance= name=obj00000000000000000049 instance= ns= 2023-05-31T20:38:23.726 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:23.706+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:7]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:7]: syncing object: pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:7/obj00000000000000000049 2023-05-31T20:38:23.726 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:23.706+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:7]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:7]: entry->timestamp=2023-05-31T20:38:18.527639+0000 2023-05-31T20:38:23.726 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:23.706+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:7]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:7]:entry[obj00000000000000000049]: bucket sync single entry (source_zone=3d163b1e-09c7-47e2-ae87-add4aace3f38) b=bucket sync single entry (source_zone=3d163b1e-09c7-47e2-ae87-add4aace3f38) b= log_entry=00000000001.90.6 op=0 op_state=1 2023-05-31T20:38:23.726 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:23.706+0000 7f22572d6780 0 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:7]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:7]:entry[obj00000000000000000049]: : injecting data sync error on key=obj00000000000000000049 2023-05-31T20:38:23.726 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:23.706+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:7]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:7]: listing bilog for incremental sync; position=00000000001.90.6 2023-05-31T20:38:23.726 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:23.706+0000 7f22572d6780 10 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:7]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:7]:entry[obj00000000000000000049]: failed, retcode=-5 ((5) Input/output error) 2023-05-31T20:38:23.727 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:23.706+0000 7f22572d6780 0 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:7]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:7]:entry[obj00000000000000000049]: ERROR: failed to sync object: pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:7/obj00000000000000000049 2023-05-31T20:38:23.727 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:23.707+0000 7f22572d6780 10 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:0]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:0]: a sync operation returned error 2023-05-31T20:38:23.727 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:23.707+0000 7f22572d6780 10 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:0]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:0]: backing out with sync_status=-5 2023-05-31T20:38:23.727 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:23.707+0000 7f22572d6780 5 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:0]: incremental sync on bucket failed, retcode=-5 2023-05-31T20:38:23.727 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:23.707+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:0]: ERROR: incremental sync failed. error: -5 2023-05-31T20:38:23.727 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:23.708+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:6]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:6]: parsed entry: id=00000000002.63.6 iter->object=obj00000000000000000019 iter->instance= name=obj00000000000000000019 instance= ns= 2023-05-31T20:38:23.728 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:23.708+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:6]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:6]: syncing object: pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:6/obj00000000000000000019 2023-05-31T20:38:23.728 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:23.708+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:6]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:6]: entry->timestamp=2023-05-31T20:38:18.658033+0000 2023-05-31T20:38:23.728 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:23.708+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:6]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:6]:entry[obj00000000000000000019]: bucket sync single entry (source_zone=3d163b1e-09c7-47e2-ae87-add4aace3f38) b=bucket sync single entry (source_zone=3d163b1e-09c7-47e2-ae87-add4aace3f38) b= log_entry=00000000002.63.6 op=0 op_state=1 2023-05-31T20:38:23.728 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:23.708+0000 7f22572d6780 5 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:6]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:6]:entry[obj00000000000000000019]: bucket sync: sync obj: 3d163b1e-09c7-47e2-ae87-add4aace3f38/:pklxkd-58[3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23])/obj00000000000000000019[0] 2023-05-31T20:38:23.728 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:23.708+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:6]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:6]: listing bilog for incremental sync; position=00000000002.63.6 2023-05-31T20:38:23.728 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:23.709+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:0]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-05-31T20:38:23.729 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:23.709+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:0]: sync status for source bucket shard: 0 2023-05-31T20:38:23.729 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:23.709+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:0]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:0]: listing bilog for incremental sync; position= 2023-05-31T20:38:23.729 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:23.710+0000 7f22572d6780 10 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:6]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:6]:entry[obj00000000000000000019]: success 2023-05-31T20:38:23.729 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:23.710+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:6]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:6]: updating marker marker_oid=bucket.sync-status.3d163b1e-09c7-47e2-ae87-add4aace3f38:pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:6:4 marker=00000000002.63.6 timestamp=2023-05-31T20:38:18.658033+0000 2023-05-31T20:38:23.729 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:23.711+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:20]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:20]: parsed entry: id=00000000002.35.6 iter->object=obj00000000000000000010 iter->instance= name=obj00000000000000000010 instance= ns= 2023-05-31T20:38:23.729 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:23.711+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:20]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:20]: syncing object: pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:20/obj00000000000000000010 2023-05-31T20:38:23.730 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:23.711+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:20]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:20]: entry->timestamp=2023-05-31T20:38:18.624687+0000 2023-05-31T20:38:23.730 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:23.711+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:20]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:20]:entry[obj00000000000000000010]: bucket sync single entry (source_zone=3d163b1e-09c7-47e2-ae87-add4aace3f38) b=bucket sync single entry (source_zone=3d163b1e-09c7-47e2-ae87-add4aace3f38) b= log_entry=00000000002.35.6 op=0 op_state=1 2023-05-31T20:38:23.730 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:23.711+0000 7f22572d6780 5 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:20]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:20]:entry[obj00000000000000000010]: bucket sync: sync obj: 3d163b1e-09c7-47e2-ae87-add4aace3f38/:pklxkd-58[3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23])/obj00000000000000000010[0] 2023-05-31T20:38:23.731 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:23.711+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:20]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:20]: listing bilog for incremental sync; position=00000000002.35.6 2023-05-31T20:38:23.731 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:23.712+0000 7f22572d6780 10 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:20]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:20]:entry[obj00000000000000000010]: success 2023-05-31T20:38:23.731 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:23.712+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:20]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:20]: updating marker marker_oid=bucket.sync-status.3d163b1e-09c7-47e2-ae87-add4aace3f38:pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:20:4 marker=00000000002.35.6 timestamp=2023-05-31T20:38:18.624687+0000 2023-05-31T20:38:23.731 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:23.713+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:11]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:11]: parsed entry: id=00000000001.26.6 iter->object=obj00000000000000000031 iter->instance= name=obj00000000000000000031 instance= ns= 2023-05-31T20:38:23.731 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:23.713+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:11]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:11]: syncing object: pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:11/obj00000000000000000031 2023-05-31T20:38:23.732 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:23.713+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:11]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:11]: entry->timestamp=2023-05-31T20:38:18.440797+0000 2023-05-31T20:38:23.732 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:23.713+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:11]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:11]:entry[obj00000000000000000031]: bucket sync single entry (source_zone=3d163b1e-09c7-47e2-ae87-add4aace3f38) b=bucket sync single entry (source_zone=3d163b1e-09c7-47e2-ae87-add4aace3f38) b= log_entry=00000000001.26.6 op=0 op_state=1 2023-05-31T20:38:23.732 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:23.713+0000 7f22572d6780 5 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:11]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:11]:entry[obj00000000000000000031]: bucket sync: sync obj: 3d163b1e-09c7-47e2-ae87-add4aace3f38/:pklxkd-58[3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23])/obj00000000000000000031[0] 2023-05-31T20:38:23.732 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:23.713+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:11]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:11]: listing bilog for incremental sync; position=00000000001.26.6 2023-05-31T20:38:23.733 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:23.713+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:16]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:16]: parsed entry: id=00000000002.28.6 iter->object=obj00000000000000000005 iter->instance= name=obj00000000000000000005 instance= ns= 2023-05-31T20:38:23.733 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:23.713+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:16]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:16]: syncing object: pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:16/obj00000000000000000005 2023-05-31T20:38:23.733 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:23.713+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:16]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:16]: entry->timestamp=2023-05-31T20:38:19.170574+0000 2023-05-31T20:38:23.733 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:23.713+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:16]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:16]:entry[obj00000000000000000005]: bucket sync single entry (source_zone=3d163b1e-09c7-47e2-ae87-add4aace3f38) b=bucket sync single entry (source_zone=3d163b1e-09c7-47e2-ae87-add4aace3f38) b= log_entry=00000000002.28.6 op=0 op_state=1 2023-05-31T20:38:23.734 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:23.713+0000 7f22572d6780 5 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:16]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:16]:entry[obj00000000000000000005]: bucket sync: sync obj: 3d163b1e-09c7-47e2-ae87-add4aace3f38/:pklxkd-58[3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23])/obj00000000000000000005[0] 2023-05-31T20:38:23.734 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:23.713+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:16]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:16]: listing bilog for incremental sync; position=00000000002.28.6 2023-05-31T20:38:23.734 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:23.715+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:14]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:14]: parsed entry: id=00000000002.41.6 iter->object=obj00000000000000000020 iter->instance= name=obj00000000000000000020 instance= ns= 2023-05-31T20:38:23.734 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:23.715+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:14]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:14]: syncing object: pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:14/obj00000000000000000020 2023-05-31T20:38:23.735 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:23.715+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:14]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:14]: entry->timestamp=2023-05-31T20:38:19.382645+0000 2023-05-31T20:38:23.735 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:23.715+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:14]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:14]:entry[obj00000000000000000020]: bucket sync single entry (source_zone=3d163b1e-09c7-47e2-ae87-add4aace3f38) b=bucket sync single entry (source_zone=3d163b1e-09c7-47e2-ae87-add4aace3f38) b= log_entry=00000000002.41.6 op=0 op_state=1 2023-05-31T20:38:23.735 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:23.715+0000 7f22572d6780 5 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:14]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:14]:entry[obj00000000000000000020]: bucket sync: sync obj: 3d163b1e-09c7-47e2-ae87-add4aace3f38/:pklxkd-58[3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23])/obj00000000000000000020[0] 2023-05-31T20:38:23.736 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:23.715+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:14]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:14]: listing bilog for incremental sync; position=00000000002.41.6 2023-05-31T20:38:23.736 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:23.715+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:12]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:12]: parsed entry: id=00000000002.44.6 iter->object=obj00000000000000000022 iter->instance= name=obj00000000000000000022 instance= ns= 2023-05-31T20:38:23.736 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:23.715+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:12]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:12]: syncing object: pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:12/obj00000000000000000022 2023-05-31T20:38:23.736 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:23.715+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:12]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:12]: entry->timestamp=2023-05-31T20:38:19.202516+0000 2023-05-31T20:38:23.737 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:23.715+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:12]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:12]:entry[obj00000000000000000022]: bucket sync single entry (source_zone=3d163b1e-09c7-47e2-ae87-add4aace3f38) b=bucket sync single entry (source_zone=3d163b1e-09c7-47e2-ae87-add4aace3f38) b= log_entry=00000000002.44.6 op=0 op_state=1 2023-05-31T20:38:23.737 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:23.715+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:19]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:19]: parsed entry: id=00000000001.37.6 iter->object=obj00000000000000000042 iter->instance= name=obj00000000000000000042 instance= ns= 2023-05-31T20:38:23.737 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:23.715+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:19]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:19]: syncing object: pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:19/obj00000000000000000042 2023-05-31T20:38:23.737 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:23.715+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:19]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:19]: entry->timestamp=2023-05-31T20:38:18.977382+0000 2023-05-31T20:38:23.738 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:23.715+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:19]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:19]:entry[obj00000000000000000042]: bucket sync single entry (source_zone=3d163b1e-09c7-47e2-ae87-add4aace3f38) b=bucket sync single entry (source_zone=3d163b1e-09c7-47e2-ae87-add4aace3f38) b= log_entry=00000000001.37.6 op=0 op_state=1 2023-05-31T20:38:23.738 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:23.715+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:15]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:15]: parsed entry: id=00000000001.40.6 iter->object=obj00000000000000000037 iter->instance= name=obj00000000000000000037 instance= ns= 2023-05-31T20:38:23.738 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:23.715+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:15]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:15]: syncing object: pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:15/obj00000000000000000037 2023-05-31T20:38:23.738 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:23.715+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:15]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:15]: entry->timestamp=2023-05-31T20:38:19.358015+0000 2023-05-31T20:38:23.739 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:23.715+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:15]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:15]:entry[obj00000000000000000037]: bucket sync single entry (source_zone=3d163b1e-09c7-47e2-ae87-add4aace3f38) b=bucket sync single entry (source_zone=3d163b1e-09c7-47e2-ae87-add4aace3f38) b= log_entry=00000000001.40.6 op=0 op_state=1 2023-05-31T20:38:23.739 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:23.715+0000 7f22572d6780 10 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:16]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:16]:entry[obj00000000000000000005]: success 2023-05-31T20:38:23.739 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:23.715+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:16]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:16]: updating marker marker_oid=bucket.sync-status.3d163b1e-09c7-47e2-ae87-add4aace3f38:pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:16:4 marker=00000000002.28.6 timestamp=2023-05-31T20:38:19.170574+0000 2023-05-31T20:38:23.739 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:23.715+0000 7f22572d6780 5 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:12]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:12]:entry[obj00000000000000000022]: bucket sync: sync obj: 3d163b1e-09c7-47e2-ae87-add4aace3f38/:pklxkd-58[3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23])/obj00000000000000000022[0] 2023-05-31T20:38:23.740 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:23.715+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:12]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:12]: listing bilog for incremental sync; position=00000000002.44.6 2023-05-31T20:38:23.740 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:23.715+0000 7f22572d6780 5 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:19]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:19]:entry[obj00000000000000000042]: bucket sync: sync obj: 3d163b1e-09c7-47e2-ae87-add4aace3f38/:pklxkd-58[3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23])/obj00000000000000000042[0] 2023-05-31T20:38:23.740 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:23.715+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:19]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:19]: listing bilog for incremental sync; position=00000000001.37.6 2023-05-31T20:38:23.740 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:23.715+0000 7f22572d6780 5 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:15]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:15]:entry[obj00000000000000000037]: bucket sync: sync obj: 3d163b1e-09c7-47e2-ae87-add4aace3f38/:pklxkd-58[3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23])/obj00000000000000000037[0] 2023-05-31T20:38:23.741 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:23.715+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:15]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:15]: listing bilog for incremental sync; position=00000000001.40.6 2023-05-31T20:38:23.741 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:23.717+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:10]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:10]: parsed entry: id=00000000002.42.6 iter->object=obj00000000000000000015 iter->instance= name=obj00000000000000000015 instance= ns= 2023-05-31T20:38:23.741 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:23.717+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:10]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:10]: syncing object: pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:10/obj00000000000000000015 2023-05-31T20:38:23.741 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:23.717+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:10]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:10]: entry->timestamp=2023-05-31T20:38:18.818606+0000 2023-05-31T20:38:23.742 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:23.717+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:10]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:10]:entry[obj00000000000000000015]: bucket sync single entry (source_zone=3d163b1e-09c7-47e2-ae87-add4aace3f38) b=bucket sync single entry (source_zone=3d163b1e-09c7-47e2-ae87-add4aace3f38) b= log_entry=00000000002.42.6 op=0 op_state=1 2023-05-31T20:38:23.742 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:23.717+0000 7f22572d6780 5 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:10]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:10]:entry[obj00000000000000000015]: bucket sync: sync obj: 3d163b1e-09c7-47e2-ae87-add4aace3f38/:pklxkd-58[3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23])/obj00000000000000000015[0] 2023-05-31T20:38:23.742 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:23.717+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:10]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:10]: listing bilog for incremental sync; position=00000000002.42.6 2023-05-31T20:38:23.742 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:23.719+0000 7f22572d6780 10 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:14]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:14]:entry[obj00000000000000000020]: success 2023-05-31T20:38:23.742 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:23.719+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:14]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:14]: updating marker marker_oid=bucket.sync-status.3d163b1e-09c7-47e2-ae87-add4aace3f38:pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:14:4 marker=00000000002.41.6 timestamp=2023-05-31T20:38:19.382645+0000 2023-05-31T20:38:23.742 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:23.719+0000 7f22572d6780 10 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:10]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:10]:entry[obj00000000000000000015]: success 2023-05-31T20:38:23.743 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:23.719+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:10]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:10]: updating marker marker_oid=bucket.sync-status.3d163b1e-09c7-47e2-ae87-add4aace3f38:pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:10:4 marker=00000000002.42.6 timestamp=2023-05-31T20:38:18.818606+0000 2023-05-31T20:38:23.744 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:23.722+0000 7f22572d6780 10 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:23]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:23]: a sync operation returned error 2023-05-31T20:38:23.744 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:23.722+0000 7f22572d6780 10 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:23]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:23]: backing out with sync_status=-5 2023-05-31T20:38:23.745 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:23.722+0000 7f22572d6780 5 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:23]: incremental sync on bucket failed, retcode=-5 2023-05-31T20:38:23.745 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:23.722+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:23]: ERROR: incremental sync failed. error: -5 2023-05-31T20:38:23.745 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:23.724+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:23]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-05-31T20:38:23.745 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:23.724+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:23]: sync status for source bucket shard: 0 2023-05-31T20:38:23.745 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:23.724+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:23]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:23]: listing bilog for incremental sync; position= 2023-05-31T20:38:23.745 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:23.726+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:8]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:8]: parsed entry: id=00000000001.52.6 iter->object=obj00000000000000000017 iter->instance= name=obj00000000000000000017 instance= ns= 2023-05-31T20:38:23.746 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:23.726+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:8]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:8]: syncing object: pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:8/obj00000000000000000017 2023-05-31T20:38:23.746 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:23.726+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:8]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:8]: entry->timestamp=2023-05-31T20:38:18.463637+0000 2023-05-31T20:38:23.746 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:23.726+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:8]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:8]:entry[obj00000000000000000017]: bucket sync single entry (source_zone=3d163b1e-09c7-47e2-ae87-add4aace3f38) b=bucket sync single entry (source_zone=3d163b1e-09c7-47e2-ae87-add4aace3f38) b= log_entry=00000000001.52.6 op=0 op_state=1 2023-05-31T20:38:23.746 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:23.726+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:13]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:13]: parsed entry: id=00000000002.69.6 iter->object=obj00000000000000000013 iter->instance= name=obj00000000000000000013 instance= ns= 2023-05-31T20:38:23.746 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:23.726+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:13]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:13]: syncing object: pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:13/obj00000000000000000013 2023-05-31T20:38:23.746 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:23.726+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:13]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:13]: entry->timestamp=2023-05-31T20:38:18.916325+0000 2023-05-31T20:38:23.746 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:23.726+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:13]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:13]:entry[obj00000000000000000013]: bucket sync single entry (source_zone=3d163b1e-09c7-47e2-ae87-add4aace3f38) b=bucket sync single entry (source_zone=3d163b1e-09c7-47e2-ae87-add4aace3f38) b= log_entry=00000000002.69.6 op=0 op_state=1 2023-05-31T20:38:23.747 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:23.726+0000 7f22572d6780 5 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:8]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:8]:entry[obj00000000000000000017]: bucket sync: sync obj: 3d163b1e-09c7-47e2-ae87-add4aace3f38/:pklxkd-58[3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23])/obj00000000000000000017[0] 2023-05-31T20:38:23.747 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:23.726+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:8]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:8]: listing bilog for incremental sync; position=00000000001.52.6 2023-05-31T20:38:23.747 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:23.726+0000 7f22572d6780 5 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:13]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:13]:entry[obj00000000000000000013]: bucket sync: sync obj: 3d163b1e-09c7-47e2-ae87-add4aace3f38/:pklxkd-58[3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23])/obj00000000000000000013[0] 2023-05-31T20:38:23.747 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:23.726+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:13]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:13]: listing bilog for incremental sync; position=00000000002.69.6 2023-05-31T20:38:23.747 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:23.728+0000 7f22572d6780 10 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:13]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:13]:entry[obj00000000000000000013]: success 2023-05-31T20:38:23.747 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:23.728+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:13]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:13]: updating marker marker_oid=bucket.sync-status.3d163b1e-09c7-47e2-ae87-add4aace3f38:pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:13:4 marker=00000000002.69.6 timestamp=2023-05-31T20:38:18.916325+0000 2023-05-31T20:38:23.752 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:23.748+0000 7f22572d6780 10 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:7]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:7]: a sync operation returned error 2023-05-31T20:38:23.752 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:23.748+0000 7f22572d6780 10 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:7]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:7]: backing out with sync_status=-5 2023-05-31T20:38:23.752 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:23.748+0000 7f22572d6780 5 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:7]: incremental sync on bucket failed, retcode=-5 2023-05-31T20:38:23.753 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:23.748+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:7]: ERROR: incremental sync failed. error: -5 2023-05-31T20:38:23.753 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:23.748+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:7]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-05-31T20:38:23.753 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:23.748+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:7]: sync status for source bucket shard: 0 2023-05-31T20:38:23.754 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:23.748+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:7]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:7]: listing bilog for incremental sync; position= 2023-05-31T20:38:23.755 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:23.751+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:0]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:0]: parsed entry: id=00000000002.39.6 iter->object=obj00000000000000000006 iter->instance= name=obj00000000000000000006 instance= ns= 2023-05-31T20:38:23.755 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:23.751+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:0]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:0]: syncing object: pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:0/obj00000000000000000006 2023-05-31T20:38:23.755 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:23.751+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:0]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:0]: entry->timestamp=2023-05-31T20:38:19.104820+0000 2023-05-31T20:38:23.756 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:23.751+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:0]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:0]:entry[obj00000000000000000006]: bucket sync single entry (source_zone=3d163b1e-09c7-47e2-ae87-add4aace3f38) b=bucket sync single entry (source_zone=3d163b1e-09c7-47e2-ae87-add4aace3f38) b= log_entry=00000000002.39.6 op=0 op_state=1 2023-05-31T20:38:23.756 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:23.751+0000 7f22572d6780 5 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:0]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:0]:entry[obj00000000000000000006]: bucket sync: sync obj: 3d163b1e-09c7-47e2-ae87-add4aace3f38/:pklxkd-58[3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23])/obj00000000000000000006[0] 2023-05-31T20:38:23.756 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:23.751+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:0]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:0]: listing bilog for incremental sync; position=00000000002.39.6 2023-05-31T20:38:23.756 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:23.752+0000 7f22572d6780 10 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:0]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:0]:entry[obj00000000000000000006]: success 2023-05-31T20:38:23.757 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:23.752+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:0]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:0]: updating marker marker_oid=bucket.sync-status.3d163b1e-09c7-47e2-ae87-add4aace3f38:pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:0:4 marker=00000000002.39.6 timestamp=2023-05-31T20:38:19.104820+0000 2023-05-31T20:38:23.766 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:23.762+0000 7f22572d6780 10 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:11]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:11]:entry[obj00000000000000000031]: success 2023-05-31T20:38:23.766 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:23.762+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:11]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:11]: updating marker marker_oid=bucket.sync-status.3d163b1e-09c7-47e2-ae87-add4aace3f38:pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:11:4 marker=00000000001.26.6 timestamp=2023-05-31T20:38:18.440797+0000 2023-05-31T20:38:23.770 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:23.766+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:23]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:23]: parsed entry: id=00000000002.49.6 iter->object=obj00000000000000000025 iter->instance= name=obj00000000000000000025 instance= ns= 2023-05-31T20:38:23.770 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:23.766+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:23]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:23]: syncing object: pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:23/obj00000000000000000025 2023-05-31T20:38:23.771 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:23.766+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:23]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:23]: entry->timestamp=2023-05-31T20:38:19.001294+0000 2023-05-31T20:38:23.771 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:23.766+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:23]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:23]:entry[obj00000000000000000025]: bucket sync single entry (source_zone=3d163b1e-09c7-47e2-ae87-add4aace3f38) b=bucket sync single entry (source_zone=3d163b1e-09c7-47e2-ae87-add4aace3f38) b= log_entry=00000000002.49.6 op=0 op_state=1 2023-05-31T20:38:23.771 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:23.766+0000 7f22572d6780 5 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:23]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:23]:entry[obj00000000000000000025]: bucket sync: sync obj: 3d163b1e-09c7-47e2-ae87-add4aace3f38/:pklxkd-58[3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23])/obj00000000000000000025[0] 2023-05-31T20:38:23.771 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:23.766+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:23]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:23]: listing bilog for incremental sync; position=00000000002.49.6 2023-05-31T20:38:23.772 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:23.767+0000 7f22572d6780 10 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:23]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:23]:entry[obj00000000000000000025]: success 2023-05-31T20:38:23.772 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:23.767+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:23]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:23]: updating marker marker_oid=bucket.sync-status.3d163b1e-09c7-47e2-ae87-add4aace3f38:pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:23:4 marker=00000000002.49.6 timestamp=2023-05-31T20:38:19.001294+0000 2023-05-31T20:38:23.774 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:23.770+0000 7f22572d6780 10 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:12]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:12]:entry[obj00000000000000000022]: success 2023-05-31T20:38:23.775 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:23.770+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:12]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:12]: updating marker marker_oid=bucket.sync-status.3d163b1e-09c7-47e2-ae87-add4aace3f38:pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:12:4 marker=00000000002.44.6 timestamp=2023-05-31T20:38:19.202516+0000 2023-05-31T20:38:23.775 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:23.770+0000 7f22572d6780 10 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:19]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:19]:entry[obj00000000000000000042]: success 2023-05-31T20:38:23.775 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:23.770+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:19]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:19]: updating marker marker_oid=bucket.sync-status.3d163b1e-09c7-47e2-ae87-add4aace3f38:pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:19:4 marker=00000000001.37.6 timestamp=2023-05-31T20:38:18.977382+0000 2023-05-31T20:38:23.780 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:23.776+0000 7f22572d6780 10 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:15]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:15]:entry[obj00000000000000000037]: success 2023-05-31T20:38:23.781 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:23.776+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:15]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:15]: updating marker marker_oid=bucket.sync-status.3d163b1e-09c7-47e2-ae87-add4aace3f38:pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:15:4 marker=00000000001.40.6 timestamp=2023-05-31T20:38:19.358015+0000 2023-05-31T20:38:23.786 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:23.782+0000 7f22572d6780 10 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:8]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:8]:entry[obj00000000000000000017]: success 2023-05-31T20:38:23.786 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:23.782+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:8]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:8]: updating marker marker_oid=bucket.sync-status.3d163b1e-09c7-47e2-ae87-add4aace3f38:pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:8:4 marker=00000000001.52.6 timestamp=2023-05-31T20:38:18.463637+0000 2023-05-31T20:38:23.794 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:23.790+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:7]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:7]: parsed entry: id=00000000001.90.6 iter->object=obj00000000000000000049 iter->instance= name=obj00000000000000000049 instance= ns= 2023-05-31T20:38:23.794 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:23.790+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:7]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:7]: syncing object: pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:7/obj00000000000000000049 2023-05-31T20:38:23.795 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:23.790+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:7]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:7]: entry->timestamp=2023-05-31T20:38:18.527639+0000 2023-05-31T20:38:23.795 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:23.790+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:7]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:7]:entry[obj00000000000000000049]: bucket sync single entry (source_zone=3d163b1e-09c7-47e2-ae87-add4aace3f38) b=bucket sync single entry (source_zone=3d163b1e-09c7-47e2-ae87-add4aace3f38) b= log_entry=00000000001.90.6 op=0 op_state=1 2023-05-31T20:38:23.795 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:23.790+0000 7f22572d6780 5 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:7]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:7]:entry[obj00000000000000000049]: bucket sync: sync obj: 3d163b1e-09c7-47e2-ae87-add4aace3f38/:pklxkd-58[3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23])/obj00000000000000000049[0] 2023-05-31T20:38:23.795 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:23.790+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:7]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:7]: listing bilog for incremental sync; position=00000000001.90.6 2023-05-31T20:38:23.841 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:23.837+0000 7f22572d6780 10 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:7]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:7]:entry[obj00000000000000000049]: success 2023-05-31T20:38:23.842 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:23.837+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:7]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:7]: updating marker marker_oid=bucket.sync-status.3d163b1e-09c7-47e2-ae87-add4aace3f38:pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:7:4 marker=00000000001.90.6 timestamp=2023-05-31T20:38:18.527639+0000 2023-05-31T20:38:23.850 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:23.845+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:0]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-05-31T20:38:23.850 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:23.845+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:1]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-05-31T20:38:23.851 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:23.845+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:2]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-05-31T20:38:23.851 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:23.846+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:3]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-05-31T20:38:23.851 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:23.846+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:1]: sync status for source bucket shard: 0 2023-05-31T20:38:23.852 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:23.846+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:1]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:1]: listing bilog for incremental sync; position= 2023-05-31T20:38:23.852 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:23.846+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:0]: sync status for source bucket shard: 0 2023-05-31T20:38:23.852 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:23.846+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:0]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:0]: listing bilog for incremental sync; position= 2023-05-31T20:38:23.852 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:23.846+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:3]: sync status for source bucket shard: 0 2023-05-31T20:38:23.852 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:23.846+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:5]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-05-31T20:38:23.852 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:23.846+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:3]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:3]: listing bilog for incremental sync; position= 2023-05-31T20:38:23.853 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:23.846+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:7]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-05-31T20:38:23.855 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:23.846+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:4]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-05-31T20:38:23.855 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:23.847+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:7]: sync status for source bucket shard: 0 2023-05-31T20:38:23.855 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:23.847+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:6]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-05-31T20:38:23.855 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:23.847+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:7]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:7]: listing bilog for incremental sync; position= 2023-05-31T20:38:23.856 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:23.847+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:8]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-05-31T20:38:23.856 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:23.847+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:4]: sync status for source bucket shard: 0 2023-05-31T20:38:23.856 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:23.847+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:4]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:4]: listing bilog for incremental sync; position= 2023-05-31T20:38:23.856 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:23.847+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:5]: sync status for source bucket shard: 0 2023-05-31T20:38:23.857 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:23.847+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:5]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:5]: listing bilog for incremental sync; position= 2023-05-31T20:38:23.857 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:23.847+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:9]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-05-31T20:38:23.857 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:23.848+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:8]: sync status for source bucket shard: 0 2023-05-31T20:38:23.857 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:23.848+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:8]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:8]: listing bilog for incremental sync; position= 2023-05-31T20:38:23.857 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:23.848+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:6]: sync status for source bucket shard: 0 2023-05-31T20:38:23.858 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:23.848+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:6]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:6]: listing bilog for incremental sync; position= 2023-05-31T20:38:23.858 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:23.848+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:10]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-05-31T20:38:23.858 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:23.848+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:12]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-05-31T20:38:23.858 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:23.848+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:11]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-05-31T20:38:23.859 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:23.848+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:14]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-05-31T20:38:23.859 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:23.848+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:13]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-05-31T20:38:23.859 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:23.848+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:15]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-05-31T20:38:23.859 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:23.848+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:17]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-05-31T20:38:23.860 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:23.849+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:12]: sync status for source bucket shard: 0 2023-05-31T20:38:23.860 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:23.849+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:7]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:7]: parsed entry: id=00000000001.82.6 iter->object=obj00000000000000000317 iter->instance= name=obj00000000000000000317 instance= ns= 2023-05-31T20:38:23.860 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:23.849+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:7]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:7]: syncing object: pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:7/obj00000000000000000317 2023-05-31T20:38:23.860 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:23.849+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:7]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:7]: entry->timestamp=2023-05-31T20:38:20.065984+0000 2023-05-31T20:38:23.860 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:23.849+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:7]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:7]:entry[obj00000000000000000317]: bucket sync single entry (source_zone=3d163b1e-09c7-47e2-ae87-add4aace3f38) b=bucket sync single entry (source_zone=3d163b1e-09c7-47e2-ae87-add4aace3f38) b= log_entry=00000000001.82.6 op=0 op_state=1 2023-05-31T20:38:23.861 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:23.849+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:12]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:12]: listing bilog for incremental sync; position= 2023-05-31T20:38:23.861 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:23.849+0000 7f22572d6780 5 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:7]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:7]:entry[obj00000000000000000317]: bucket sync: sync obj: 3d163b1e-09c7-47e2-ae87-add4aace3f38/:pklxkd-58[3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23])/obj00000000000000000317[0] 2023-05-31T20:38:23.862 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:23.849+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:7]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:7]: listing bilog for incremental sync; position=00000000001.82.6 2023-05-31T20:38:23.862 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:23.849+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:4]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:4]: parsed entry: id=00000000001.45.6 iter->object=obj00000000000000000056 iter->instance= name=obj00000000000000000056 instance= ns= 2023-05-31T20:38:23.862 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:23.849+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:4]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:4]: syncing object: pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:4/obj00000000000000000056 2023-05-31T20:38:23.862 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:23.849+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:4]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:4]: entry->timestamp=2023-05-31T20:38:20.370757+0000 2023-05-31T20:38:23.862 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:23.849+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:4]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:4]:entry[obj00000000000000000056]: bucket sync single entry (source_zone=3d163b1e-09c7-47e2-ae87-add4aace3f38) b=bucket sync single entry (source_zone=3d163b1e-09c7-47e2-ae87-add4aace3f38) b= log_entry=00000000001.45.6 op=0 op_state=1 2023-05-31T20:38:23.862 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:23.849+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:5]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:5]: parsed entry: id=00000000002.25.6 iter->object=obj00000000000000000004 iter->instance= name=obj00000000000000000004 instance= ns= 2023-05-31T20:38:23.863 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:23.849+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:5]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:5]: syncing object: pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:5/obj00000000000000000004 2023-05-31T20:38:23.863 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:23.849+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:5]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:5]: entry->timestamp=2023-05-31T20:38:20.455466+0000 2023-05-31T20:38:23.863 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:23.849+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:5]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:5]:entry[obj00000000000000000004]: bucket sync single entry (source_zone=3d163b1e-09c7-47e2-ae87-add4aace3f38) b=bucket sync single entry (source_zone=3d163b1e-09c7-47e2-ae87-add4aace3f38) b= log_entry=00000000002.25.6 op=0 op_state=1 2023-05-31T20:38:23.863 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:23.849+0000 7f22572d6780 5 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:4]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:4]:entry[obj00000000000000000056]: bucket sync: sync obj: 3d163b1e-09c7-47e2-ae87-add4aace3f38/:pklxkd-58[3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23])/obj00000000000000000056[0] 2023-05-31T20:38:23.863 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:23.849+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:4]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:4]: listing bilog for incremental sync; position=00000000001.45.6 2023-05-31T20:38:23.863 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:23.849+0000 7f22572d6780 5 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:5]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:5]:entry[obj00000000000000000004]: bucket sync: sync obj: 3d163b1e-09c7-47e2-ae87-add4aace3f38/:pklxkd-58[3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23])/obj00000000000000000004[0] 2023-05-31T20:38:23.864 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:23.849+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:5]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:5]: listing bilog for incremental sync; position=00000000002.25.6 2023-05-31T20:38:23.868 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:23.850+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:13]: sync status for source bucket shard: 0 2023-05-31T20:38:23.868 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:23.850+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:17]: sync status for source bucket shard: 0 2023-05-31T20:38:23.868 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:23.850+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:11]: sync status for source bucket shard: 0 2023-05-31T20:38:23.868 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:23.850+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:13]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:13]: listing bilog for incremental sync; position= 2023-05-31T20:38:23.868 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:23.850+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:17]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:17]: listing bilog for incremental sync; position= 2023-05-31T20:38:23.869 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:23.850+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:11]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:11]: listing bilog for incremental sync; position= 2023-05-31T20:38:23.869 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:23.850+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:14]: sync status for source bucket shard: 0 2023-05-31T20:38:23.869 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:23.850+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:14]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:14]: listing bilog for incremental sync; position= 2023-05-31T20:38:23.869 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:23.850+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:10]: sync status for source bucket shard: 0 2023-05-31T20:38:23.869 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:23.850+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:15]: sync status for source bucket shard: 0 2023-05-31T20:38:23.870 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:23.850+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:10]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:10]: listing bilog for incremental sync; position= 2023-05-31T20:38:23.870 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:23.850+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:15]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:15]: listing bilog for incremental sync; position= 2023-05-31T20:38:23.870 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:23.850+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:16]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-05-31T20:38:23.870 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:23.850+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:18]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-05-31T20:38:23.870 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:23.850+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:19]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-05-31T20:38:23.870 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:23.850+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:20]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-05-31T20:38:23.871 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:23.850+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:21]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-05-31T20:38:23.871 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:23.850+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:24]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-05-31T20:38:23.871 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:23.851+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:23]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-05-31T20:38:23.871 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:23.851+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:22]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-05-31T20:38:23.871 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:23.851+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:25]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-05-31T20:38:23.872 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:23.852+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:16]: sync status for source bucket shard: 0 2023-05-31T20:38:23.872 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:23.852+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:16]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:16]: listing bilog for incremental sync; position= 2023-05-31T20:38:23.873 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:23.852+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:2]: sync status for source bucket shard: 0 2023-05-31T20:38:23.873 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:23.852+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:20]: sync status for source bucket shard: 0 2023-05-31T20:38:23.873 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:23.852+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:2]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:2]: listing bilog for incremental sync; position= 2023-05-31T20:38:23.873 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:23.852+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:20]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:20]: listing bilog for incremental sync; position= 2023-05-31T20:38:23.873 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:23.852+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:19]: sync status for source bucket shard: 0 2023-05-31T20:38:23.873 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:23.852+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:26]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-05-31T20:38:23.874 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:23.852+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:27]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-05-31T20:38:23.874 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:23.852+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:28]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-05-31T20:38:23.874 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:23.852+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:29]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-05-31T20:38:23.875 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:23.852+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:19]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:19]: listing bilog for incremental sync; position= 2023-05-31T20:38:23.875 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:23.852+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:21]: sync status for source bucket shard: 0 2023-05-31T20:38:23.875 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:23.852+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:21]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:21]: listing bilog for incremental sync; position= 2023-05-31T20:38:23.875 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:23.853+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:30]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-05-31T20:38:23.875 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:23.853+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:22]: sync status for source bucket shard: 0 2023-05-31T20:38:23.876 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:23.853+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:23]: sync status for source bucket shard: 0 2023-05-31T20:38:23.876 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:23.853+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:25]: sync status for source bucket shard: 0 2023-05-31T20:38:23.876 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:23.853+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:22]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:22]: listing bilog for incremental sync; position= 2023-05-31T20:38:23.876 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:23.853+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:23]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:23]: listing bilog for incremental sync; position= 2023-05-31T20:38:23.876 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:23.853+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:25]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:25]: listing bilog for incremental sync; position= 2023-05-31T20:38:23.877 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:23.853+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:13]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:13]: parsed entry: id=00000000002.57.6 iter->object=obj00000000000000000008 iter->instance= name=obj00000000000000000008 instance= ns= 2023-05-31T20:38:23.877 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:23.853+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:13]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:13]: syncing object: pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:13/obj00000000000000000008 2023-05-31T20:38:23.877 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:23.853+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:13]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:13]: entry->timestamp=2023-05-31T20:38:20.093833+0000 2023-05-31T20:38:23.877 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:23.853+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:13]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:13]:entry[obj00000000000000000008]: bucket sync single entry (source_zone=3d163b1e-09c7-47e2-ae87-add4aace3f38) b=bucket sync single entry (source_zone=3d163b1e-09c7-47e2-ae87-add4aace3f38) b= log_entry=00000000002.57.6 op=0 op_state=1 2023-05-31T20:38:23.877 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:23.853+0000 7f22572d6780 0 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:13]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:13]:entry[obj00000000000000000008]: : injecting data sync error on key=obj00000000000000000008 2023-05-31T20:38:23.877 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:23.853+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:13]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:13]: listing bilog for incremental sync; position=00000000002.57.6 2023-05-31T20:38:23.878 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:23.853+0000 7f22572d6780 10 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:13]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:13]:entry[obj00000000000000000008]: failed, retcode=-5 ((5) Input/output error) 2023-05-31T20:38:23.878 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:23.853+0000 7f22572d6780 0 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:13]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:13]:entry[obj00000000000000000008]: ERROR: failed to sync object: pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:13/obj00000000000000000008 2023-05-31T20:38:23.878 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:23.854+0000 7f22572d6780 10 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:5]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:5]:entry[obj00000000000000000004]: success 2023-05-31T20:38:23.878 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:23.854+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:5]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:5]: updating marker marker_oid=bucket.sync-status.3d163b1e-09c7-47e2-ae87-add4aace3f38:pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:5:5 marker=00000000002.25.6 timestamp=2023-05-31T20:38:20.455466+0000 2023-05-31T20:38:23.878 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:23.855+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:28]: sync status for source bucket shard: 0 2023-05-31T20:38:23.878 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:23.855+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:29]: sync status for source bucket shard: 0 2023-05-31T20:38:23.879 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:23.855+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:26]: sync status for source bucket shard: 0 2023-05-31T20:38:23.879 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:23.855+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:27]: sync status for source bucket shard: 0 2023-05-31T20:38:23.879 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:23.855+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:9]: sync status for source bucket shard: 0 2023-05-31T20:38:23.879 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:23.855+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:18]: sync status for source bucket shard: 0 2023-05-31T20:38:23.880 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:23.855+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:24]: sync status for source bucket shard: 0 2023-05-31T20:38:23.881 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:23.855+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:28]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:28]: listing bilog for incremental sync; position= 2023-05-31T20:38:23.881 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:23.855+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:29]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:29]: listing bilog for incremental sync; position= 2023-05-31T20:38:23.881 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:23.855+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:17]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:17]: parsed entry: id=00000000001.42.6 iter->object=obj00000000000000000162 iter->instance= name=obj00000000000000000162 instance= ns= 2023-05-31T20:38:23.881 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:23.855+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:17]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:17]: syncing object: pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:17/obj00000000000000000162 2023-05-31T20:38:23.881 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:23.855+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:17]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:17]: entry->timestamp=2023-05-31T20:38:20.538625+0000 2023-05-31T20:38:23.882 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:23.855+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:17]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:17]:entry[obj00000000000000000162]: bucket sync single entry (source_zone=3d163b1e-09c7-47e2-ae87-add4aace3f38) b=bucket sync single entry (source_zone=3d163b1e-09c7-47e2-ae87-add4aace3f38) b= log_entry=00000000001.42.6 op=0 op_state=1 2023-05-31T20:38:23.882 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:23.856+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:26]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:26]: listing bilog for incremental sync; position= 2023-05-31T20:38:23.882 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:23.856+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:27]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:27]: listing bilog for incremental sync; position= 2023-05-31T20:38:23.882 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:23.856+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:9]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:9]: listing bilog for incremental sync; position= 2023-05-31T20:38:23.882 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:23.856+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:18]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:18]: listing bilog for incremental sync; position= 2023-05-31T20:38:23.882 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:23.856+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:24]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:24]: listing bilog for incremental sync; position= 2023-05-31T20:38:23.882 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:23.856+0000 7f22572d6780 5 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:17]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:17]:entry[obj00000000000000000162]: bucket sync: sync obj: 3d163b1e-09c7-47e2-ae87-add4aace3f38/:pklxkd-58[3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23])/obj00000000000000000162[0] 2023-05-31T20:38:23.883 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:23.856+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:17]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:17]: listing bilog for incremental sync; position=00000000001.42.6 2023-05-31T20:38:23.883 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:23.856+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:11]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:11]: parsed entry: id=00000000002.51.6 iter->object=obj00000000000000000009 iter->instance= name=obj00000000000000000009 instance= ns= 2023-05-31T20:38:23.883 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:23.856+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:11]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:11]: syncing object: pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:11/obj00000000000000000009 2023-05-31T20:38:23.883 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:23.856+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:11]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:11]: entry->timestamp=2023-05-31T20:38:20.894560+0000 2023-05-31T20:38:23.883 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:23.856+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:11]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:11]:entry[obj00000000000000000009]: bucket sync single entry (source_zone=3d163b1e-09c7-47e2-ae87-add4aace3f38) b=bucket sync single entry (source_zone=3d163b1e-09c7-47e2-ae87-add4aace3f38) b= log_entry=00000000002.51.6 op=0 op_state=1 2023-05-31T20:38:23.883 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:23.856+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:14]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:14]: parsed entry: id=00000000002.43.6 iter->object=obj00000000000000000018 iter->instance= name=obj00000000000000000018 instance= ns= 2023-05-31T20:38:23.883 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:23.856+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:14]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:14]: syncing object: pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:14/obj00000000000000000018 2023-05-31T20:38:23.884 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:23.856+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:14]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:14]: entry->timestamp=2023-05-31T20:38:20.689463+0000 2023-05-31T20:38:23.884 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:23.856+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:14]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:14]:entry[obj00000000000000000018]: bucket sync single entry (source_zone=3d163b1e-09c7-47e2-ae87-add4aace3f38) b=bucket sync single entry (source_zone=3d163b1e-09c7-47e2-ae87-add4aace3f38) b= log_entry=00000000002.43.6 op=0 op_state=1 2023-05-31T20:38:23.884 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:23.856+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:10]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:10]: parsed entry: id=00000000002.57.6 iter->object=obj00000000000000000021 iter->instance= name=obj00000000000000000021 instance= ns= 2023-05-31T20:38:23.884 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:23.856+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:10]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:10]: syncing object: pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:10/obj00000000000000000021 2023-05-31T20:38:23.884 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:23.856+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:10]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:10]: entry->timestamp=2023-05-31T20:38:20.802133+0000 2023-05-31T20:38:23.884 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:23.856+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:10]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:10]:entry[obj00000000000000000021]: bucket sync single entry (source_zone=3d163b1e-09c7-47e2-ae87-add4aace3f38) b=bucket sync single entry (source_zone=3d163b1e-09c7-47e2-ae87-add4aace3f38) b= log_entry=00000000002.57.6 op=0 op_state=1 2023-05-31T20:38:23.885 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:23.856+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:15]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:15]: parsed entry: id=00000000002.44.6 iter->object=obj00000000000000000007 iter->instance= name=obj00000000000000000007 instance= ns= 2023-05-31T20:38:23.885 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:23.856+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:15]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:15]: syncing object: pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:15/obj00000000000000000007 2023-05-31T20:38:23.885 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:23.856+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:15]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:15]: entry->timestamp=2023-05-31T20:38:21.063553+0000 2023-05-31T20:38:23.885 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:23.856+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:15]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:15]:entry[obj00000000000000000007]: bucket sync single entry (source_zone=3d163b1e-09c7-47e2-ae87-add4aace3f38) b=bucket sync single entry (source_zone=3d163b1e-09c7-47e2-ae87-add4aace3f38) b= log_entry=00000000002.44.6 op=0 op_state=1 2023-05-31T20:38:23.885 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:23.856+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:2]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:2]: parsed entry: id=00000000001.50.6 iter->object=obj00000000000000000467 iter->instance= name=obj00000000000000000467 instance= ns= 2023-05-31T20:38:23.886 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:23.856+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:2]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:2]: syncing object: pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:2/obj00000000000000000467 2023-05-31T20:38:23.886 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:23.856+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:2]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:2]: entry->timestamp=2023-05-31T20:38:20.327690+0000 2023-05-31T20:38:23.886 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:23.856+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:2]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:2]:entry[obj00000000000000000467]: bucket sync single entry (source_zone=3d163b1e-09c7-47e2-ae87-add4aace3f38) b=bucket sync single entry (source_zone=3d163b1e-09c7-47e2-ae87-add4aace3f38) b= log_entry=00000000001.50.6 op=0 op_state=1 2023-05-31T20:38:23.886 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:23.856+0000 7f22572d6780 5 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:11]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:11]:entry[obj00000000000000000009]: bucket sync: sync obj: 3d163b1e-09c7-47e2-ae87-add4aace3f38/:pklxkd-58[3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23])/obj00000000000000000009[0] 2023-05-31T20:38:23.886 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:23.856+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:11]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:11]: listing bilog for incremental sync; position=00000000002.51.6 2023-05-31T20:38:23.887 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:23.856+0000 7f22572d6780 5 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:14]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:14]:entry[obj00000000000000000018]: bucket sync: sync obj: 3d163b1e-09c7-47e2-ae87-add4aace3f38/:pklxkd-58[3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23])/obj00000000000000000018[0] 2023-05-31T20:38:23.887 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:23.856+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:14]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:14]: listing bilog for incremental sync; position=00000000002.43.6 2023-05-31T20:38:23.887 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:23.856+0000 7f22572d6780 5 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:10]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:10]:entry[obj00000000000000000021]: bucket sync: sync obj: 3d163b1e-09c7-47e2-ae87-add4aace3f38/:pklxkd-58[3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23])/obj00000000000000000021[0] 2023-05-31T20:38:23.887 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:23.856+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:10]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:10]: listing bilog for incremental sync; position=00000000002.57.6 2023-05-31T20:38:23.887 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:23.856+0000 7f22572d6780 0 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:15]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:15]:entry[obj00000000000000000007]: : injecting data sync error on key=obj00000000000000000007 2023-05-31T20:38:23.887 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:23.856+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:15]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:15]: listing bilog for incremental sync; position=00000000002.44.6 2023-05-31T20:38:23.888 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:23.856+0000 7f22572d6780 0 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:2]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:2]:entry[obj00000000000000000467]: : injecting data sync error on key=obj00000000000000000467 2023-05-31T20:38:23.888 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:23.856+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:2]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:2]: listing bilog for incremental sync; position=00000000001.50.6 2023-05-31T20:38:23.888 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:23.856+0000 7f22572d6780 10 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:15]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:15]:entry[obj00000000000000000007]: failed, retcode=-5 ((5) Input/output error) 2023-05-31T20:38:23.888 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:23.856+0000 7f22572d6780 0 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:15]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:15]:entry[obj00000000000000000007]: ERROR: failed to sync object: pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:15/obj00000000000000000007 2023-05-31T20:38:23.888 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:23.856+0000 7f22572d6780 10 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:2]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:2]:entry[obj00000000000000000467]: failed, retcode=-5 ((5) Input/output error) 2023-05-31T20:38:23.888 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:23.856+0000 7f22572d6780 0 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:2]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:2]:entry[obj00000000000000000467]: ERROR: failed to sync object: pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:2/obj00000000000000000467 2023-05-31T20:38:23.888 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:23.857+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:30]: sync status for source bucket shard: 0 2023-05-31T20:38:23.889 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:23.857+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:30]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:30]: listing bilog for incremental sync; position= 2023-05-31T20:38:23.889 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:23.859+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:24]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:24]: parsed entry: id=00000000002.52.6 iter->object=obj00000000000000000011 iter->instance= name=obj00000000000000000011 instance= ns= 2023-05-31T20:38:23.889 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:23.859+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:24]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:24]: syncing object: pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:24/obj00000000000000000011 2023-05-31T20:38:23.889 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:23.859+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:24]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:24]: entry->timestamp=2023-05-31T20:38:20.511590+0000 2023-05-31T20:38:23.889 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:23.859+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:24]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:24]:entry[obj00000000000000000011]: bucket sync single entry (source_zone=3d163b1e-09c7-47e2-ae87-add4aace3f38) b=bucket sync single entry (source_zone=3d163b1e-09c7-47e2-ae87-add4aace3f38) b= log_entry=00000000002.52.6 op=0 op_state=1 2023-05-31T20:38:23.889 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:23.859+0000 7f22572d6780 5 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:24]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:24]:entry[obj00000000000000000011]: bucket sync: sync obj: 3d163b1e-09c7-47e2-ae87-add4aace3f38/:pklxkd-58[3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23])/obj00000000000000000011[0] 2023-05-31T20:38:23.890 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:23.859+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:24]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:24]: listing bilog for incremental sync; position=00000000002.52.6 2023-05-31T20:38:23.890 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:23.861+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:27]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:27]: parsed entry: id=00000000001.71.6 iter->object=obj00000000000000000061 iter->instance= name=obj00000000000000000061 instance= ns= 2023-05-31T20:38:23.890 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:23.861+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:27]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:27]: syncing object: pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:27/obj00000000000000000061 2023-05-31T20:38:23.890 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:23.861+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:27]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:27]: entry->timestamp=2023-05-31T20:38:21.154093+0000 2023-05-31T20:38:23.890 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:23.861+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:27]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:27]:entry[obj00000000000000000061]: bucket sync single entry (source_zone=3d163b1e-09c7-47e2-ae87-add4aace3f38) b=bucket sync single entry (source_zone=3d163b1e-09c7-47e2-ae87-add4aace3f38) b= log_entry=00000000001.71.6 op=0 op_state=1 2023-05-31T20:38:23.890 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:23.861+0000 7f22572d6780 0 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:27]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:27]:entry[obj00000000000000000061]: : injecting data sync error on key=obj00000000000000000061 2023-05-31T20:38:23.891 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:23.861+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:27]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:27]: listing bilog for incremental sync; position=00000000001.71.6 2023-05-31T20:38:23.891 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:23.861+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:9]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:9]: parsed entry: id=00000000002.49.6 iter->object=obj00000000000000000000 iter->instance= name=obj00000000000000000000 instance= ns= 2023-05-31T20:38:23.892 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:23.861+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:9]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:9]: syncing object: pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:9/obj00000000000000000000 2023-05-31T20:38:23.892 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:23.861+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:9]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:9]: entry->timestamp=2023-05-31T20:38:20.973923+0000 2023-05-31T20:38:23.892 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:23.861+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:9]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:9]:entry[obj00000000000000000000]: bucket sync single entry (source_zone=3d163b1e-09c7-47e2-ae87-add4aace3f38) b=bucket sync single entry (source_zone=3d163b1e-09c7-47e2-ae87-add4aace3f38) b= log_entry=00000000002.49.6 op=0 op_state=1 2023-05-31T20:38:23.892 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:23.861+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:18]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:18]: parsed entry: id=00000000002.40.6 iter->object=obj00000000000000000016 iter->instance= name=obj00000000000000000016 instance= ns= 2023-05-31T20:38:23.892 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:23.861+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:18]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:18]: syncing object: pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:18/obj00000000000000000016 2023-05-31T20:38:23.892 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:23.861+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:18]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:18]: entry->timestamp=2023-05-31T20:38:20.157954+0000 2023-05-31T20:38:23.893 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:23.861+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:18]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:18]:entry[obj00000000000000000016]: bucket sync single entry (source_zone=3d163b1e-09c7-47e2-ae87-add4aace3f38) b=bucket sync single entry (source_zone=3d163b1e-09c7-47e2-ae87-add4aace3f38) b= log_entry=00000000002.40.6 op=0 op_state=1 2023-05-31T20:38:23.893 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:23.861+0000 7f22572d6780 10 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:27]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:27]:entry[obj00000000000000000061]: failed, retcode=-5 ((5) Input/output error) 2023-05-31T20:38:23.893 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:23.861+0000 7f22572d6780 0 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:27]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:27]:entry[obj00000000000000000061]: ERROR: failed to sync object: pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:27/obj00000000000000000061 2023-05-31T20:38:23.893 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:23.861+0000 7f22572d6780 5 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:9]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:9]:entry[obj00000000000000000000]: bucket sync: sync obj: 3d163b1e-09c7-47e2-ae87-add4aace3f38/:pklxkd-58[3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23])/obj00000000000000000000[0] 2023-05-31T20:38:23.893 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:23.861+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:9]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:9]: listing bilog for incremental sync; position=00000000002.49.6 2023-05-31T20:38:23.894 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:23.861+0000 7f22572d6780 5 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:18]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:18]:entry[obj00000000000000000016]: bucket sync: sync obj: 3d163b1e-09c7-47e2-ae87-add4aace3f38/:pklxkd-58[3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23])/obj00000000000000000016[0] 2023-05-31T20:38:23.894 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:23.861+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:18]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:18]: listing bilog for incremental sync; position=00000000002.40.6 2023-05-31T20:38:23.894 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:23.862+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:30]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:30]: parsed entry: id=00000000001.47.6 iter->object=obj00000000000000000133 iter->instance= name=obj00000000000000000133 instance= ns= 2023-05-31T20:38:23.894 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:23.862+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:30]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:30]: syncing object: pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:30/obj00000000000000000133 2023-05-31T20:38:23.894 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:23.862+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:30]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:30]: entry->timestamp=2023-05-31T20:38:19.997509+0000 2023-05-31T20:38:23.894 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:23.862+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:30]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:30]:entry[obj00000000000000000133]: bucket sync single entry (source_zone=3d163b1e-09c7-47e2-ae87-add4aace3f38) b=bucket sync single entry (source_zone=3d163b1e-09c7-47e2-ae87-add4aace3f38) b= log_entry=00000000001.47.6 op=0 op_state=1 2023-05-31T20:38:23.895 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:23.862+0000 7f22572d6780 0 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:30]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:30]:entry[obj00000000000000000133]: : injecting data sync error on key=obj00000000000000000133 2023-05-31T20:38:23.895 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:23.862+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:30]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:30]: listing bilog for incremental sync; position=00000000001.47.6 2023-05-31T20:38:23.895 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:23.862+0000 7f22572d6780 10 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:30]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:30]:entry[obj00000000000000000133]: failed, retcode=-5 ((5) Input/output error) 2023-05-31T20:38:23.895 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:23.862+0000 7f22572d6780 0 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:30]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:30]:entry[obj00000000000000000133]: ERROR: failed to sync object: pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:30/obj00000000000000000133 2023-05-31T20:38:23.895 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:23.863+0000 7f22572d6780 10 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:24]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:24]:entry[obj00000000000000000011]: success 2023-05-31T20:38:23.895 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:23.863+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:24]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:24]: updating marker marker_oid=bucket.sync-status.3d163b1e-09c7-47e2-ae87-add4aace3f38:pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:24:5 marker=00000000002.52.6 timestamp=2023-05-31T20:38:20.511590+0000 2023-05-31T20:38:23.895 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:23.863+0000 7f22572d6780 10 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:10]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:10]:entry[obj00000000000000000021]: success 2023-05-31T20:38:23.896 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:23.863+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:10]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:10]: updating marker marker_oid=bucket.sync-status.3d163b1e-09c7-47e2-ae87-add4aace3f38:pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:10:5 marker=00000000002.57.6 timestamp=2023-05-31T20:38:20.802133+0000 2023-05-31T20:38:23.896 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:23.863+0000 7f22572d6780 10 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:11]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:11]:entry[obj00000000000000000009]: success 2023-05-31T20:38:23.896 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:23.863+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:11]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:11]: updating marker marker_oid=bucket.sync-status.3d163b1e-09c7-47e2-ae87-add4aace3f38:pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:11:5 marker=00000000002.51.6 timestamp=2023-05-31T20:38:20.894560+0000 2023-05-31T20:38:23.899 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:23.866+0000 7f22572d6780 10 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:15]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:15]: a sync operation returned error 2023-05-31T20:38:23.899 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:23.866+0000 7f22572d6780 10 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:2]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:2]: a sync operation returned error 2023-05-31T20:38:23.899 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:23.866+0000 7f22572d6780 10 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:15]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:15]: backing out with sync_status=-5 2023-05-31T20:38:23.899 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:23.866+0000 7f22572d6780 5 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:15]: incremental sync on bucket failed, retcode=-5 2023-05-31T20:38:23.899 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:23.866+0000 7f22572d6780 10 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:2]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:2]: backing out with sync_status=-5 2023-05-31T20:38:23.900 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:23.866+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:15]: ERROR: incremental sync failed. error: -5 2023-05-31T20:38:23.900 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:23.866+0000 7f22572d6780 5 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:2]: incremental sync on bucket failed, retcode=-5 2023-05-31T20:38:23.900 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:23.866+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:2]: ERROR: incremental sync failed. error: -5 2023-05-31T20:38:23.900 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:23.867+0000 7f22572d6780 10 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:9]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:9]:entry[obj00000000000000000000]: success 2023-05-31T20:38:23.900 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:23.867+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:9]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:9]: updating marker marker_oid=bucket.sync-status.3d163b1e-09c7-47e2-ae87-add4aace3f38:pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:9:5 marker=00000000002.49.6 timestamp=2023-05-31T20:38:20.973923+0000 2023-05-31T20:38:23.900 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:23.867+0000 7f22572d6780 10 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:18]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:18]:entry[obj00000000000000000016]: success 2023-05-31T20:38:23.901 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:23.867+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:18]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:18]: updating marker marker_oid=bucket.sync-status.3d163b1e-09c7-47e2-ae87-add4aace3f38:pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:18:5 marker=00000000002.40.6 timestamp=2023-05-31T20:38:20.157954+0000 2023-05-31T20:38:23.901 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:23.868+0000 7f22572d6780 10 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:4]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:4]:entry[obj00000000000000000056]: success 2023-05-31T20:38:23.901 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:23.868+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:4]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:4]: updating marker marker_oid=bucket.sync-status.3d163b1e-09c7-47e2-ae87-add4aace3f38:pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:4:5 marker=00000000001.45.6 timestamp=2023-05-31T20:38:20.370757+0000 2023-05-31T20:38:23.901 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:23.872+0000 7f22572d6780 10 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:7]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:7]:entry[obj00000000000000000317]: success 2023-05-31T20:38:23.901 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:23.872+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:7]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:7]: updating marker marker_oid=bucket.sync-status.3d163b1e-09c7-47e2-ae87-add4aace3f38:pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:7:5 marker=00000000001.82.6 timestamp=2023-05-31T20:38:20.065984+0000 2023-05-31T20:38:23.901 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:23.873+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:15]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-05-31T20:38:23.901 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:23.873+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:2]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-05-31T20:38:23.902 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:23.874+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:15]: sync status for source bucket shard: 0 2023-05-31T20:38:23.902 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:23.874+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:15]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:15]: listing bilog for incremental sync; position= 2023-05-31T20:38:23.902 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:23.874+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:2]: sync status for source bucket shard: 0 2023-05-31T20:38:23.902 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:23.874+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:2]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:2]: listing bilog for incremental sync; position= 2023-05-31T20:38:23.902 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:23.888+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:1]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:1]: parsed entry: id=00000000002.47.6 iter->object=obj00000000000000000006 iter->instance= name=obj00000000000000000006 instance= ns= 2023-05-31T20:38:23.902 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:23.888+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:1]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:1]: syncing object: pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:1/obj00000000000000000006 2023-05-31T20:38:23.903 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:23.888+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:1]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:1]: entry->timestamp=2023-05-31T20:38:20.778444+0000 2023-05-31T20:38:23.903 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:23.888+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:1]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:1]:entry[obj00000000000000000006]: bucket sync single entry (source_zone=3d163b1e-09c7-47e2-ae87-add4aace3f38) b=bucket sync single entry (source_zone=3d163b1e-09c7-47e2-ae87-add4aace3f38) b= log_entry=00000000002.47.6 op=0 op_state=1 2023-05-31T20:38:23.903 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:23.888+0000 7f22572d6780 5 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:1]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:1]:entry[obj00000000000000000006]: bucket sync: sync obj: 3d163b1e-09c7-47e2-ae87-add4aace3f38/:pklxkd-58[3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23])/obj00000000000000000006[0] 2023-05-31T20:38:23.903 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:23.888+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:1]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:1]: listing bilog for incremental sync; position=00000000002.47.6 2023-05-31T20:38:23.903 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:23.888+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:3]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:3]: parsed entry: id=00000000002.57.6 iter->object=obj00000000000000000005 iter->instance= name=obj00000000000000000005 instance= ns= 2023-05-31T20:38:23.904 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:23.888+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:3]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:3]: syncing object: pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:3/obj00000000000000000005 2023-05-31T20:38:23.904 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:23.888+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:3]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:3]: entry->timestamp=2023-05-31T20:38:20.871317+0000 2023-05-31T20:38:23.904 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:23.888+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:3]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:3]:entry[obj00000000000000000005]: bucket sync single entry (source_zone=3d163b1e-09c7-47e2-ae87-add4aace3f38) b=bucket sync single entry (source_zone=3d163b1e-09c7-47e2-ae87-add4aace3f38) b= log_entry=00000000002.57.6 op=0 op_state=1 2023-05-31T20:38:23.904 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:23.888+0000 7f22572d6780 5 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:3]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:3]:entry[obj00000000000000000005]: bucket sync: sync obj: 3d163b1e-09c7-47e2-ae87-add4aace3f38/:pklxkd-58[3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23])/obj00000000000000000005[0] 2023-05-31T20:38:23.904 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:23.888+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:3]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:3]: listing bilog for incremental sync; position=00000000002.57.6 2023-05-31T20:38:23.904 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:23.890+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:8]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:8]: parsed entry: id=00000000002.134.6 iter->object=obj00000000000000000022 iter->instance= name=obj00000000000000000022 instance= ns= 2023-05-31T20:38:23.905 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:23.890+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:8]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:8]: syncing object: pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:8/obj00000000000000000022 2023-05-31T20:38:23.905 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:23.890+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:8]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:8]: entry->timestamp=2023-05-31T20:38:20.598399+0000 2023-05-31T20:38:23.905 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:23.890+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:8]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:8]:entry[obj00000000000000000022]: bucket sync single entry (source_zone=3d163b1e-09c7-47e2-ae87-add4aace3f38) b=bucket sync single entry (source_zone=3d163b1e-09c7-47e2-ae87-add4aace3f38) b= log_entry=00000000002.134.6 op=0 op_state=1 2023-05-31T20:38:23.905 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:23.890+0000 7f22572d6780 0 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:8]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:8]:entry[obj00000000000000000022]: : injecting data sync error on key=obj00000000000000000022 2023-05-31T20:38:23.905 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:23.890+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:8]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:8]: listing bilog for incremental sync; position=00000000002.134.6 2023-05-31T20:38:23.905 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:23.890+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:6]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:6]: parsed entry: id=00000000002.49.6 iter->object=obj00000000000000000023 iter->instance= name=obj00000000000000000023 instance= ns= 2023-05-31T20:38:23.905 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:23.890+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:6]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:6]: syncing object: pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:6/obj00000000000000000023 2023-05-31T20:38:23.906 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:23.890+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:6]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:6]: entry->timestamp=2023-05-31T20:38:20.347602+0000 2023-05-31T20:38:23.906 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:23.890+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:6]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:6]:entry[obj00000000000000000023]: bucket sync single entry (source_zone=3d163b1e-09c7-47e2-ae87-add4aace3f38) b=bucket sync single entry (source_zone=3d163b1e-09c7-47e2-ae87-add4aace3f38) b= log_entry=00000000002.49.6 op=0 op_state=1 2023-05-31T20:38:23.906 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:23.890+0000 7f22572d6780 10 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:8]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:8]:entry[obj00000000000000000022]: failed, retcode=-5 ((5) Input/output error) 2023-05-31T20:38:23.906 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:23.890+0000 7f22572d6780 0 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:8]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:8]:entry[obj00000000000000000022]: ERROR: failed to sync object: pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:8/obj00000000000000000022 2023-05-31T20:38:23.906 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:23.890+0000 7f22572d6780 5 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:6]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:6]:entry[obj00000000000000000023]: bucket sync: sync obj: 3d163b1e-09c7-47e2-ae87-add4aace3f38/:pklxkd-58[3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23])/obj00000000000000000023[0] 2023-05-31T20:38:23.906 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:23.890+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:6]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:6]: listing bilog for incremental sync; position=00000000002.49.6 2023-05-31T20:38:23.907 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:23.890+0000 7f22572d6780 10 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:1]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:1]:entry[obj00000000000000000006]: success 2023-05-31T20:38:23.907 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:23.890+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:1]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:1]: updating marker marker_oid=bucket.sync-status.3d163b1e-09c7-47e2-ae87-add4aace3f38:pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:1:5 marker=00000000002.47.6 timestamp=2023-05-31T20:38:20.778444+0000 2023-05-31T20:38:23.907 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:23.891+0000 7f22572d6780 10 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:3]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:3]:entry[obj00000000000000000005]: success 2023-05-31T20:38:23.907 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:23.891+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:3]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:3]: updating marker marker_oid=bucket.sync-status.3d163b1e-09c7-47e2-ae87-add4aace3f38:pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:3:5 marker=00000000002.57.6 timestamp=2023-05-31T20:38:20.871317+0000 2023-05-31T20:38:23.907 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:23.892+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:12]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:12]: parsed entry: id=00000000001.58.6 iter->object=obj00000000000000000271 iter->instance= name=obj00000000000000000271 instance= ns= 2023-05-31T20:38:23.907 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:23.892+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:12]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:12]: syncing object: pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:12/obj00000000000000000271 2023-05-31T20:38:23.907 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:23.892+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:12]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:12]: entry->timestamp=2023-05-31T20:38:20.240943+0000 2023-05-31T20:38:23.908 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:23.892+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:12]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:12]:entry[obj00000000000000000271]: bucket sync single entry (source_zone=3d163b1e-09c7-47e2-ae87-add4aace3f38) b=bucket sync single entry (source_zone=3d163b1e-09c7-47e2-ae87-add4aace3f38) b= log_entry=00000000001.58.6 op=0 op_state=1 2023-05-31T20:38:23.908 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:23.892+0000 7f22572d6780 0 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:12]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:12]:entry[obj00000000000000000271]: : injecting data sync error on key=obj00000000000000000271 2023-05-31T20:38:23.908 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:23.892+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:12]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:12]: listing bilog for incremental sync; position=00000000001.58.6 2023-05-31T20:38:23.908 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:23.892+0000 7f22572d6780 10 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:12]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:12]:entry[obj00000000000000000271]: failed, retcode=-5 ((5) Input/output error) 2023-05-31T20:38:23.908 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:23.892+0000 7f22572d6780 0 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:12]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:12]:entry[obj00000000000000000271]: ERROR: failed to sync object: pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:12/obj00000000000000000271 2023-05-31T20:38:23.908 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:23.895+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:16]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:16]: parsed entry: id=00000000002.45.6 iter->object=obj00000000000000000017 iter->instance= name=obj00000000000000000017 instance= ns= 2023-05-31T20:38:23.909 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:23.895+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:16]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:16]: syncing object: pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:16/obj00000000000000000017 2023-05-31T20:38:23.909 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:23.895+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:16]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:16]: entry->timestamp=2023-05-31T20:38:20.714951+0000 2023-05-31T20:38:23.909 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:23.895+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:16]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:16]:entry[obj00000000000000000017]: bucket sync single entry (source_zone=3d163b1e-09c7-47e2-ae87-add4aace3f38) b=bucket sync single entry (source_zone=3d163b1e-09c7-47e2-ae87-add4aace3f38) b= log_entry=00000000002.45.6 op=0 op_state=1 2023-05-31T20:38:23.909 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:23.895+0000 7f22572d6780 5 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:16]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:16]:entry[obj00000000000000000017]: bucket sync: sync obj: 3d163b1e-09c7-47e2-ae87-add4aace3f38/:pklxkd-58[3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23])/obj00000000000000000017[0] 2023-05-31T20:38:23.909 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:23.895+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:16]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:16]: listing bilog for incremental sync; position=00000000002.45.6 2023-05-31T20:38:23.912 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:23.896+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:22]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:22]: parsed entry: id=00000000002.53.6 iter->object=obj00000000000000000012 iter->instance= name=obj00000000000000000012 instance= ns= 2023-05-31T20:38:23.912 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:23.896+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:22]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:22]: syncing object: pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:22/obj00000000000000000012 2023-05-31T20:38:23.912 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:23.896+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:22]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:22]: entry->timestamp=2023-05-31T20:38:20.626928+0000 2023-05-31T20:38:23.913 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:23.896+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:22]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:22]:entry[obj00000000000000000012]: bucket sync single entry (source_zone=3d163b1e-09c7-47e2-ae87-add4aace3f38) b=bucket sync single entry (source_zone=3d163b1e-09c7-47e2-ae87-add4aace3f38) b= log_entry=00000000002.53.6 op=0 op_state=1 2023-05-31T20:38:23.913 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:23.896+0000 7f22572d6780 0 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:22]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:22]:entry[obj00000000000000000012]: : injecting data sync error on key=obj00000000000000000012 2023-05-31T20:38:23.913 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:23.896+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:22]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:22]: listing bilog for incremental sync; position=00000000002.53.6 2023-05-31T20:38:23.913 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:23.896+0000 7f22572d6780 10 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:22]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:22]:entry[obj00000000000000000012]: failed, retcode=-5 ((5) Input/output error) 2023-05-31T20:38:23.913 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:23.896+0000 7f22572d6780 0 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:22]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:22]:entry[obj00000000000000000012]: ERROR: failed to sync object: pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:22/obj00000000000000000012 2023-05-31T20:38:23.913 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:23.896+0000 7f22572d6780 10 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:16]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:16]:entry[obj00000000000000000017]: success 2023-05-31T20:38:23.914 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:23.896+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:16]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:16]: updating marker marker_oid=bucket.sync-status.3d163b1e-09c7-47e2-ae87-add4aace3f38:pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:16:5 marker=00000000002.45.6 timestamp=2023-05-31T20:38:20.714951+0000 2023-05-31T20:38:23.914 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:23.897+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:20]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:20]: parsed entry: id=00000000001.60.6 iter->object=obj00000000000000000423 iter->instance= name=obj00000000000000000423 instance= ns= 2023-05-31T20:38:23.914 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:23.897+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:20]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:20]: syncing object: pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:20/obj00000000000000000423 2023-05-31T20:38:23.914 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:23.897+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:20]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:20]: entry->timestamp=2023-05-31T20:38:20.434035+0000 2023-05-31T20:38:23.914 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:23.897+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:20]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:20]:entry[obj00000000000000000423]: bucket sync single entry (source_zone=3d163b1e-09c7-47e2-ae87-add4aace3f38) b=bucket sync single entry (source_zone=3d163b1e-09c7-47e2-ae87-add4aace3f38) b= log_entry=00000000001.60.6 op=0 op_state=1 2023-05-31T20:38:23.914 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:23.897+0000 7f22572d6780 5 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:20]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:20]:entry[obj00000000000000000423]: bucket sync: sync obj: 3d163b1e-09c7-47e2-ae87-add4aace3f38/:pklxkd-58[3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23])/obj00000000000000000423[0] 2023-05-31T20:38:23.915 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:23.897+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:20]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:20]: listing bilog for incremental sync; position=00000000001.60.6 2023-05-31T20:38:23.915 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:23.897+0000 7f22572d6780 10 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:13]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:13]: a sync operation returned error 2023-05-31T20:38:23.915 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:23.897+0000 7f22572d6780 10 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:13]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:13]: backing out with sync_status=-5 2023-05-31T20:38:23.915 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:23.897+0000 7f22572d6780 5 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:13]: incremental sync on bucket failed, retcode=-5 2023-05-31T20:38:23.915 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:23.897+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:13]: ERROR: incremental sync failed. error: -5 2023-05-31T20:38:23.915 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:23.899+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:23]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:23]: parsed entry: id=00000000002.51.6 iter->object=obj00000000000000000001 iter->instance= name=obj00000000000000000001 instance= ns= 2023-05-31T20:38:23.915 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:23.899+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:23]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:23]: syncing object: pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:23/obj00000000000000000001 2023-05-31T20:38:23.916 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:23.899+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:23]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:23]: entry->timestamp=2023-05-31T20:38:21.038260+0000 2023-05-31T20:38:23.916 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:23.899+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:23]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:23]:entry[obj00000000000000000001]: bucket sync single entry (source_zone=3d163b1e-09c7-47e2-ae87-add4aace3f38) b=bucket sync single entry (source_zone=3d163b1e-09c7-47e2-ae87-add4aace3f38) b= log_entry=00000000002.51.6 op=0 op_state=1 2023-05-31T20:38:23.916 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:23.899+0000 7f22572d6780 5 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:23]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:23]:entry[obj00000000000000000001]: bucket sync: sync obj: 3d163b1e-09c7-47e2-ae87-add4aace3f38/:pklxkd-58[3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23])/obj00000000000000000001[0] 2023-05-31T20:38:23.916 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:23.899+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:23]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:23]: listing bilog for incremental sync; position=00000000002.51.6 2023-05-31T20:38:23.916 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:23.899+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:25]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:25]: parsed entry: id=00000000001.69.6 iter->object=obj00000000000000000283 iter->instance= name=obj00000000000000000283 instance= ns= 2023-05-31T20:38:23.916 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:23.899+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:25]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:25]: syncing object: pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:25/obj00000000000000000283 2023-05-31T20:38:23.917 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:23.899+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:25]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:25]: entry->timestamp=2023-05-31T20:38:20.953383+0000 2023-05-31T20:38:23.917 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:23.899+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:25]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:25]:entry[obj00000000000000000283]: bucket sync single entry (source_zone=3d163b1e-09c7-47e2-ae87-add4aace3f38) b=bucket sync single entry (source_zone=3d163b1e-09c7-47e2-ae87-add4aace3f38) b= log_entry=00000000001.69.6 op=0 op_state=1 2023-05-31T20:38:23.917 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:23.899+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:29]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:29]: parsed entry: id=00000000001.50.6 iter->object=obj00000000000000000029 iter->instance= name=obj00000000000000000029 instance= ns= 2023-05-31T20:38:23.917 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:23.899+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:29]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:29]: syncing object: pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:29/obj00000000000000000029 2023-05-31T20:38:23.917 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:23.899+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:29]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:29]: entry->timestamp=2023-05-31T20:38:20.265007+0000 2023-05-31T20:38:23.917 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:23.899+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:29]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:29]:entry[obj00000000000000000029]: bucket sync single entry (source_zone=3d163b1e-09c7-47e2-ae87-add4aace3f38) b=bucket sync single entry (source_zone=3d163b1e-09c7-47e2-ae87-add4aace3f38) b= log_entry=00000000001.50.6 op=0 op_state=1 2023-05-31T20:38:23.917 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:23.899+0000 7f22572d6780 5 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:25]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:25]:entry[obj00000000000000000283]: bucket sync: sync obj: 3d163b1e-09c7-47e2-ae87-add4aace3f38/:pklxkd-58[3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23])/obj00000000000000000283[0] 2023-05-31T20:38:23.918 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:23.899+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:25]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:25]: listing bilog for incremental sync; position=00000000001.69.6 2023-05-31T20:38:23.918 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:23.899+0000 7f22572d6780 5 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:29]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:29]:entry[obj00000000000000000029]: bucket sync: sync obj: 3d163b1e-09c7-47e2-ae87-add4aace3f38/:pklxkd-58[3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23])/obj00000000000000000029[0] 2023-05-31T20:38:23.918 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:23.899+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:29]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:29]: listing bilog for incremental sync; position=00000000001.50.6 2023-05-31T20:38:23.918 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:23.899+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:13]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-05-31T20:38:23.918 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:23.901+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:13]: sync status for source bucket shard: 0 2023-05-31T20:38:23.918 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:23.901+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:13]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:13]: listing bilog for incremental sync; position= 2023-05-31T20:38:23.919 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:23.901+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:28]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:28]: parsed entry: id=00000000002.42.6 iter->object=obj00000000000000000050 iter->instance= name=obj00000000000000000050 instance= ns= 2023-05-31T20:38:23.919 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:23.901+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:28]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:28]: syncing object: pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:28/obj00000000000000000050 2023-05-31T20:38:23.919 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:23.901+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:28]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:28]: entry->timestamp=2023-05-31T20:38:20.183372+0000 2023-05-31T20:38:23.919 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:23.901+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:28]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:28]:entry[obj00000000000000000050]: bucket sync single entry (source_zone=3d163b1e-09c7-47e2-ae87-add4aace3f38) b=bucket sync single entry (source_zone=3d163b1e-09c7-47e2-ae87-add4aace3f38) b= log_entry=00000000002.42.6 op=0 op_state=1 2023-05-31T20:38:23.919 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:23.901+0000 7f22572d6780 5 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:28]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:28]:entry[obj00000000000000000050]: bucket sync: sync obj: 3d163b1e-09c7-47e2-ae87-add4aace3f38/:pklxkd-58[3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23])/obj00000000000000000050[0] 2023-05-31T20:38:23.919 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:23.901+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:28]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:28]: listing bilog for incremental sync; position=00000000002.42.6 2023-05-31T20:38:23.919 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:23.901+0000 7f22572d6780 10 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:23]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:23]:entry[obj00000000000000000001]: success 2023-05-31T20:38:23.920 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:23.901+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:23]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:23]: updating marker marker_oid=bucket.sync-status.3d163b1e-09c7-47e2-ae87-add4aace3f38:pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:23:5 marker=00000000002.51.6 timestamp=2023-05-31T20:38:21.038260+0000 2023-05-31T20:38:23.920 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:23.903+0000 7f22572d6780 10 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:28]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:28]:entry[obj00000000000000000050]: success 2023-05-31T20:38:23.920 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:23.903+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:28]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:28]: updating marker marker_oid=bucket.sync-status.3d163b1e-09c7-47e2-ae87-add4aace3f38:pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:28:5 marker=00000000002.42.6 timestamp=2023-05-31T20:38:20.183372+0000 2023-05-31T20:38:23.920 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:23.905+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:26]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:26]: parsed entry: id=00000000002.36.6 iter->object=obj00000000000000000010 iter->instance= name=obj00000000000000000010 instance= ns= 2023-05-31T20:38:23.920 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:23.905+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:26]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:26]: syncing object: pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:26/obj00000000000000000010 2023-05-31T20:38:23.920 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:23.905+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:26]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:26]: entry->timestamp=2023-05-31T20:38:21.127706+0000 2023-05-31T20:38:23.921 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:23.905+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:26]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:26]:entry[obj00000000000000000010]: bucket sync single entry (source_zone=3d163b1e-09c7-47e2-ae87-add4aace3f38) b=bucket sync single entry (source_zone=3d163b1e-09c7-47e2-ae87-add4aace3f38) b= log_entry=00000000002.36.6 op=0 op_state=1 2023-05-31T20:38:23.921 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:23.905+0000 7f22572d6780 5 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:26]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:26]:entry[obj00000000000000000010]: bucket sync: sync obj: 3d163b1e-09c7-47e2-ae87-add4aace3f38/:pklxkd-58[3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23])/obj00000000000000000010[0] 2023-05-31T20:38:23.921 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:23.905+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:26]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:26]: listing bilog for incremental sync; position=00000000002.36.6 2023-05-31T20:38:23.921 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:23.905+0000 7f22572d6780 10 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:27]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:27]: a sync operation returned error 2023-05-31T20:38:23.921 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:23.905+0000 7f22572d6780 10 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:30]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:30]: a sync operation returned error 2023-05-31T20:38:23.921 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:23.905+0000 7f22572d6780 10 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:27]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:27]: backing out with sync_status=-5 2023-05-31T20:38:23.921 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:23.905+0000 7f22572d6780 10 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:30]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:30]: backing out with sync_status=-5 2023-05-31T20:38:23.922 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:23.905+0000 7f22572d6780 5 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:27]: incremental sync on bucket failed, retcode=-5 2023-05-31T20:38:23.922 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:23.905+0000 7f22572d6780 5 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:30]: incremental sync on bucket failed, retcode=-5 2023-05-31T20:38:23.922 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:23.905+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:27]: ERROR: incremental sync failed. error: -5 2023-05-31T20:38:23.922 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:23.905+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:30]: ERROR: incremental sync failed. error: -5 2023-05-31T20:38:23.922 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:23.907+0000 7f22572d6780 10 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:25]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:25]:entry[obj00000000000000000283]: success 2023-05-31T20:38:23.922 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:23.907+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:25]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:25]: updating marker marker_oid=bucket.sync-status.3d163b1e-09c7-47e2-ae87-add4aace3f38:pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:25:5 marker=00000000001.69.6 timestamp=2023-05-31T20:38:20.953383+0000 2023-05-31T20:38:23.923 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:23.907+0000 7f22572d6780 10 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:26]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:26]:entry[obj00000000000000000010]: success 2023-05-31T20:38:23.923 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:23.907+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:26]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:26]: updating marker marker_oid=bucket.sync-status.3d163b1e-09c7-47e2-ae87-add4aace3f38:pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:26:5 marker=00000000002.36.6 timestamp=2023-05-31T20:38:21.127706+0000 2023-05-31T20:38:23.923 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:23.908+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:30]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-05-31T20:38:23.924 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:23.909+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:27]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-05-31T20:38:23.925 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:23.909+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:27]: sync status for source bucket shard: 0 2023-05-31T20:38:23.925 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:23.909+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:27]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:27]: listing bilog for incremental sync; position= 2023-05-31T20:38:23.925 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:23.909+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:30]: sync status for source bucket shard: 0 2023-05-31T20:38:23.925 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:23.909+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:30]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:30]: listing bilog for incremental sync; position= 2023-05-31T20:38:23.925 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:23.911+0000 7f22572d6780 10 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:17]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:17]:entry[obj00000000000000000162]: success 2023-05-31T20:38:23.925 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:23.911+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:17]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:17]: updating marker marker_oid=bucket.sync-status.3d163b1e-09c7-47e2-ae87-add4aace3f38:pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:17:5 marker=00000000001.42.6 timestamp=2023-05-31T20:38:20.538625+0000 2023-05-31T20:38:23.926 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:23.912+0000 7f22572d6780 10 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:14]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:14]:entry[obj00000000000000000018]: success 2023-05-31T20:38:23.926 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:23.912+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:14]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:14]: updating marker marker_oid=bucket.sync-status.3d163b1e-09c7-47e2-ae87-add4aace3f38:pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:14:5 marker=00000000002.43.6 timestamp=2023-05-31T20:38:20.689463+0000 2023-05-31T20:38:23.926 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:23.917+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:15]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:15]: parsed entry: id=00000000002.44.6 iter->object=obj00000000000000000007 iter->instance= name=obj00000000000000000007 instance= ns= 2023-05-31T20:38:23.926 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:23.917+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:15]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:15]: syncing object: pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:15/obj00000000000000000007 2023-05-31T20:38:23.926 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:23.917+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:15]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:15]: entry->timestamp=2023-05-31T20:38:21.063553+0000 2023-05-31T20:38:23.926 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:23.917+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:15]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:15]:entry[obj00000000000000000007]: bucket sync single entry (source_zone=3d163b1e-09c7-47e2-ae87-add4aace3f38) b=bucket sync single entry (source_zone=3d163b1e-09c7-47e2-ae87-add4aace3f38) b= log_entry=00000000002.44.6 op=0 op_state=1 2023-05-31T20:38:23.926 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:23.917+0000 7f22572d6780 5 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:15]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:15]:entry[obj00000000000000000007]: bucket sync: sync obj: 3d163b1e-09c7-47e2-ae87-add4aace3f38/:pklxkd-58[3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23])/obj00000000000000000007[0] 2023-05-31T20:38:23.927 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:23.917+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:15]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:15]: listing bilog for incremental sync; position=00000000002.44.6 2023-05-31T20:38:23.927 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:23.917+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:2]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:2]: parsed entry: id=00000000001.50.6 iter->object=obj00000000000000000467 iter->instance= name=obj00000000000000000467 instance= ns= 2023-05-31T20:38:23.927 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:23.917+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:2]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:2]: syncing object: pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:2/obj00000000000000000467 2023-05-31T20:38:23.927 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:23.917+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:2]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:2]: entry->timestamp=2023-05-31T20:38:20.327690+0000 2023-05-31T20:38:23.927 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:23.917+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:2]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:2]:entry[obj00000000000000000467]: bucket sync single entry (source_zone=3d163b1e-09c7-47e2-ae87-add4aace3f38) b=bucket sync single entry (source_zone=3d163b1e-09c7-47e2-ae87-add4aace3f38) b= log_entry=00000000001.50.6 op=0 op_state=1 2023-05-31T20:38:23.927 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:23.917+0000 7f22572d6780 5 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:2]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:2]:entry[obj00000000000000000467]: bucket sync: sync obj: 3d163b1e-09c7-47e2-ae87-add4aace3f38/:pklxkd-58[3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23])/obj00000000000000000467[0] 2023-05-31T20:38:23.927 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:23.917+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:2]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:2]: listing bilog for incremental sync; position=00000000001.50.6 2023-05-31T20:38:23.928 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:23.919+0000 7f22572d6780 10 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:15]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:15]:entry[obj00000000000000000007]: success 2023-05-31T20:38:23.928 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:23.919+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:15]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:15]: updating marker marker_oid=bucket.sync-status.3d163b1e-09c7-47e2-ae87-add4aace3f38:pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:15:5 marker=00000000002.44.6 timestamp=2023-05-31T20:38:21.063553+0000 2023-05-31T20:38:23.937 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:23.933+0000 7f22572d6780 10 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:8]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:8]: a sync operation returned error 2023-05-31T20:38:23.937 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:23.933+0000 7f22572d6780 10 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:8]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:8]: backing out with sync_status=-5 2023-05-31T20:38:23.937 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:23.933+0000 7f22572d6780 5 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:8]: incremental sync on bucket failed, retcode=-5 2023-05-31T20:38:23.938 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:23.933+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:8]: ERROR: incremental sync failed. error: -5 2023-05-31T20:38:23.938 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:23.933+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:8]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-05-31T20:38:23.939 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:23.933+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:8]: sync status for source bucket shard: 0 2023-05-31T20:38:23.939 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:23.933+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:8]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:8]: listing bilog for incremental sync; position= 2023-05-31T20:38:23.939 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:23.934+0000 7f22572d6780 10 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:12]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:12]: a sync operation returned error 2023-05-31T20:38:23.939 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:23.934+0000 7f22572d6780 10 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:12]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:12]: backing out with sync_status=-5 2023-05-31T20:38:23.939 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:23.934+0000 7f22572d6780 5 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:12]: incremental sync on bucket failed, retcode=-5 2023-05-31T20:38:23.939 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:23.934+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:12]: ERROR: incremental sync failed. error: -5 2023-05-31T20:38:23.940 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:23.934+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:12]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-05-31T20:38:23.940 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:23.934+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:12]: sync status for source bucket shard: 0 2023-05-31T20:38:23.940 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:23.934+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:12]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:12]: listing bilog for incremental sync; position= 2023-05-31T20:38:23.941 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:23.936+0000 7f22572d6780 10 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:6]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:6]:entry[obj00000000000000000023]: success 2023-05-31T20:38:23.941 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:23.936+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:6]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:6]: updating marker marker_oid=bucket.sync-status.3d163b1e-09c7-47e2-ae87-add4aace3f38:pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:6:5 marker=00000000002.49.6 timestamp=2023-05-31T20:38:20.347602+0000 2023-05-31T20:38:23.942 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:23.938+0000 7f22572d6780 10 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:22]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:22]: a sync operation returned error 2023-05-31T20:38:23.942 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:23.938+0000 7f22572d6780 10 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:22]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:22]: backing out with sync_status=-5 2023-05-31T20:38:23.942 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:23.938+0000 7f22572d6780 5 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:22]: incremental sync on bucket failed, retcode=-5 2023-05-31T20:38:23.942 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:23.938+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:22]: ERROR: incremental sync failed. error: -5 2023-05-31T20:38:23.945 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:23.941+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:22]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-05-31T20:38:23.946 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:23.941+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:22]: sync status for source bucket shard: 0 2023-05-31T20:38:23.946 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:23.941+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:22]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:22]: listing bilog for incremental sync; position= 2023-05-31T20:38:23.948 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:23.944+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:13]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:13]: parsed entry: id=00000000002.57.6 iter->object=obj00000000000000000008 iter->instance= name=obj00000000000000000008 instance= ns= 2023-05-31T20:38:23.948 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:23.944+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:13]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:13]: syncing object: pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:13/obj00000000000000000008 2023-05-31T20:38:23.948 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:23.944+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:13]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:13]: entry->timestamp=2023-05-31T20:38:20.093833+0000 2023-05-31T20:38:23.948 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:23.944+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:13]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:13]:entry[obj00000000000000000008]: bucket sync single entry (source_zone=3d163b1e-09c7-47e2-ae87-add4aace3f38) b=bucket sync single entry (source_zone=3d163b1e-09c7-47e2-ae87-add4aace3f38) b= log_entry=00000000002.57.6 op=0 op_state=1 2023-05-31T20:38:23.949 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:23.944+0000 7f22572d6780 5 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:13]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:13]:entry[obj00000000000000000008]: bucket sync: sync obj: 3d163b1e-09c7-47e2-ae87-add4aace3f38/:pklxkd-58[3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23])/obj00000000000000000008[0] 2023-05-31T20:38:23.949 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:23.944+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:13]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:13]: listing bilog for incremental sync; position=00000000002.57.6 2023-05-31T20:38:23.949 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:23.945+0000 7f22572d6780 10 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:13]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:13]:entry[obj00000000000000000008]: success 2023-05-31T20:38:23.950 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:23.945+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:13]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:13]: updating marker marker_oid=bucket.sync-status.3d163b1e-09c7-47e2-ae87-add4aace3f38:pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:13:5 marker=00000000002.57.6 timestamp=2023-05-31T20:38:20.093833+0000 2023-05-31T20:38:23.955 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:23.951+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:27]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:27]: parsed entry: id=00000000001.71.6 iter->object=obj00000000000000000061 iter->instance= name=obj00000000000000000061 instance= ns= 2023-05-31T20:38:23.956 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:23.951+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:27]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:27]: syncing object: pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:27/obj00000000000000000061 2023-05-31T20:38:23.957 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:23.951+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:27]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:27]: entry->timestamp=2023-05-31T20:38:21.154093+0000 2023-05-31T20:38:23.957 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:23.951+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:27]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:27]:entry[obj00000000000000000061]: bucket sync single entry (source_zone=3d163b1e-09c7-47e2-ae87-add4aace3f38) b=bucket sync single entry (source_zone=3d163b1e-09c7-47e2-ae87-add4aace3f38) b= log_entry=00000000001.71.6 op=0 op_state=1 2023-05-31T20:38:23.957 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:23.951+0000 7f22572d6780 5 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:27]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:27]:entry[obj00000000000000000061]: bucket sync: sync obj: 3d163b1e-09c7-47e2-ae87-add4aace3f38/:pklxkd-58[3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23])/obj00000000000000000061[0] 2023-05-31T20:38:23.957 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:23.951+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:27]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:27]: listing bilog for incremental sync; position=00000000001.71.6 2023-05-31T20:38:23.957 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:23.951+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:30]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:30]: parsed entry: id=00000000001.47.6 iter->object=obj00000000000000000133 iter->instance= name=obj00000000000000000133 instance= ns= 2023-05-31T20:38:23.958 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:23.951+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:30]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:30]: syncing object: pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:30/obj00000000000000000133 2023-05-31T20:38:23.958 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:23.951+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:30]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:30]: entry->timestamp=2023-05-31T20:38:19.997509+0000 2023-05-31T20:38:23.958 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:23.951+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:30]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:30]:entry[obj00000000000000000133]: bucket sync single entry (source_zone=3d163b1e-09c7-47e2-ae87-add4aace3f38) b=bucket sync single entry (source_zone=3d163b1e-09c7-47e2-ae87-add4aace3f38) b= log_entry=00000000001.47.6 op=0 op_state=1 2023-05-31T20:38:23.958 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:23.951+0000 7f22572d6780 5 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:30]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:30]:entry[obj00000000000000000133]: bucket sync: sync obj: 3d163b1e-09c7-47e2-ae87-add4aace3f38/:pklxkd-58[3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23])/obj00000000000000000133[0] 2023-05-31T20:38:23.958 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:23.951+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:30]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:30]: listing bilog for incremental sync; position=00000000001.47.6 2023-05-31T20:38:23.959 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:23.953+0000 7f22572d6780 10 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:29]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:29]:entry[obj00000000000000000029]: success 2023-05-31T20:38:23.959 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:23.953+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:29]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:29]: updating marker marker_oid=bucket.sync-status.3d163b1e-09c7-47e2-ae87-add4aace3f38:pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:29:5 marker=00000000001.50.6 timestamp=2023-05-31T20:38:20.265007+0000 2023-05-31T20:38:23.959 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:23.954+0000 7f22572d6780 10 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:20]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:20]:entry[obj00000000000000000423]: success 2023-05-31T20:38:23.959 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:23.954+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:20]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:20]: updating marker marker_oid=bucket.sync-status.3d163b1e-09c7-47e2-ae87-add4aace3f38:pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:20:5 marker=00000000001.60.6 timestamp=2023-05-31T20:38:20.434035+0000 2023-05-31T20:38:23.974 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:23.970+0000 7f22572d6780 10 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:2]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:2]:entry[obj00000000000000000467]: success 2023-05-31T20:38:23.974 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:23.970+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:2]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:2]: updating marker marker_oid=bucket.sync-status.3d163b1e-09c7-47e2-ae87-add4aace3f38:pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:2:5 marker=00000000001.50.6 timestamp=2023-05-31T20:38:20.327690+0000 2023-05-31T20:38:23.981 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:23.976+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:8]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:8]: parsed entry: id=00000000002.134.6 iter->object=obj00000000000000000022 iter->instance= name=obj00000000000000000022 instance= ns= 2023-05-31T20:38:23.981 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:23.976+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:8]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:8]: syncing object: pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:8/obj00000000000000000022 2023-05-31T20:38:23.981 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:23.976+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:8]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:8]: entry->timestamp=2023-05-31T20:38:20.598399+0000 2023-05-31T20:38:23.982 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:23.976+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:8]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:8]:entry[obj00000000000000000022]: bucket sync single entry (source_zone=3d163b1e-09c7-47e2-ae87-add4aace3f38) b=bucket sync single entry (source_zone=3d163b1e-09c7-47e2-ae87-add4aace3f38) b= log_entry=00000000002.134.6 op=0 op_state=1 2023-05-31T20:38:23.982 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:23.976+0000 7f22572d6780 5 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:8]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:8]:entry[obj00000000000000000022]: bucket sync: sync obj: 3d163b1e-09c7-47e2-ae87-add4aace3f38/:pklxkd-58[3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23])/obj00000000000000000022[0] 2023-05-31T20:38:23.982 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:23.976+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:8]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:8]: listing bilog for incremental sync; position=00000000002.134.6 2023-05-31T20:38:23.982 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:23.976+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:12]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:12]: parsed entry: id=00000000001.58.6 iter->object=obj00000000000000000271 iter->instance= name=obj00000000000000000271 instance= ns= 2023-05-31T20:38:23.982 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:23.976+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:12]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:12]: syncing object: pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:12/obj00000000000000000271 2023-05-31T20:38:23.982 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:23.976+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:12]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:12]: entry->timestamp=2023-05-31T20:38:20.240943+0000 2023-05-31T20:38:23.982 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:23.976+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:12]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:12]:entry[obj00000000000000000271]: bucket sync single entry (source_zone=3d163b1e-09c7-47e2-ae87-add4aace3f38) b=bucket sync single entry (source_zone=3d163b1e-09c7-47e2-ae87-add4aace3f38) b= log_entry=00000000001.58.6 op=0 op_state=1 2023-05-31T20:38:23.983 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:23.976+0000 7f22572d6780 5 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:12]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:12]:entry[obj00000000000000000271]: bucket sync: sync obj: 3d163b1e-09c7-47e2-ae87-add4aace3f38/:pklxkd-58[3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23])/obj00000000000000000271[0] 2023-05-31T20:38:23.983 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:23.976+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:12]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:12]: listing bilog for incremental sync; position=00000000001.58.6 2023-05-31T20:38:23.983 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:23.978+0000 7f22572d6780 10 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:8]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:8]:entry[obj00000000000000000022]: success 2023-05-31T20:38:23.983 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:23.978+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:8]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:8]: updating marker marker_oid=bucket.sync-status.3d163b1e-09c7-47e2-ae87-add4aace3f38:pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:8:5 marker=00000000002.134.6 timestamp=2023-05-31T20:38:20.598399+0000 2023-05-31T20:38:23.987 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:23.983+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:22]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:22]: parsed entry: id=00000000002.53.6 iter->object=obj00000000000000000012 iter->instance= name=obj00000000000000000012 instance= ns= 2023-05-31T20:38:23.987 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:23.983+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:22]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:22]: syncing object: pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:22/obj00000000000000000012 2023-05-31T20:38:23.988 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:23.983+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:22]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:22]: entry->timestamp=2023-05-31T20:38:20.626928+0000 2023-05-31T20:38:23.988 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:23.983+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:22]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:22]:entry[obj00000000000000000012]: bucket sync single entry (source_zone=3d163b1e-09c7-47e2-ae87-add4aace3f38) b=bucket sync single entry (source_zone=3d163b1e-09c7-47e2-ae87-add4aace3f38) b= log_entry=00000000002.53.6 op=0 op_state=1 2023-05-31T20:38:23.988 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:23.983+0000 7f22572d6780 0 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:22]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:22]:entry[obj00000000000000000012]: : injecting data sync error on key=obj00000000000000000012 2023-05-31T20:38:23.988 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:23.983+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:22]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:22]: listing bilog for incremental sync; position=00000000002.53.6 2023-05-31T20:38:23.988 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:23.983+0000 7f22572d6780 10 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:22]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:22]:entry[obj00000000000000000012]: failed, retcode=-5 ((5) Input/output error) 2023-05-31T20:38:23.988 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:23.983+0000 7f22572d6780 0 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:22]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:22]:entry[obj00000000000000000012]: ERROR: failed to sync object: pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:22/obj00000000000000000012 2023-05-31T20:38:24.008 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:24.003+0000 7f22572d6780 10 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:27]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:27]:entry[obj00000000000000000061]: success 2023-05-31T20:38:24.009 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:24.003+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:27]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:27]: updating marker marker_oid=bucket.sync-status.3d163b1e-09c7-47e2-ae87-add4aace3f38:pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:27:5 marker=00000000001.71.6 timestamp=2023-05-31T20:38:21.154093+0000 2023-05-31T20:38:24.010 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:24.006+0000 7f22572d6780 10 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:30]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:30]:entry[obj00000000000000000133]: success 2023-05-31T20:38:24.010 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:24.006+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:30]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:30]: updating marker marker_oid=bucket.sync-status.3d163b1e-09c7-47e2-ae87-add4aace3f38:pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:30:5 marker=00000000001.47.6 timestamp=2023-05-31T20:38:19.997509+0000 2023-05-31T20:38:24.029 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:24.025+0000 7f22572d6780 10 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:22]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:22]: a sync operation returned error 2023-05-31T20:38:24.029 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:24.025+0000 7f22572d6780 10 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:22]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:22]: backing out with sync_status=-5 2023-05-31T20:38:24.029 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:24.025+0000 7f22572d6780 5 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:22]: incremental sync on bucket failed, retcode=-5 2023-05-31T20:38:24.030 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:24.025+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:22]: ERROR: incremental sync failed. error: -5 2023-05-31T20:38:24.030 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:24.026+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:22]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-05-31T20:38:24.030 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:24.026+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:22]: sync status for source bucket shard: 0 2023-05-31T20:38:24.031 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:24.026+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:22]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:22]: listing bilog for incremental sync; position= 2023-05-31T20:38:24.032 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:24.028+0000 7f22572d6780 10 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:12]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:12]:entry[obj00000000000000000271]: success 2023-05-31T20:38:24.033 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:24.028+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:12]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:12]: updating marker marker_oid=bucket.sync-status.3d163b1e-09c7-47e2-ae87-add4aace3f38:pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:12:5 marker=00000000001.58.6 timestamp=2023-05-31T20:38:20.240943+0000 2023-05-31T20:38:24.073 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:24.068+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:22]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:22]: parsed entry: id=00000000002.53.6 iter->object=obj00000000000000000012 iter->instance= name=obj00000000000000000012 instance= ns= 2023-05-31T20:38:24.073 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:24.068+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:22]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:22]: syncing object: pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:22/obj00000000000000000012 2023-05-31T20:38:24.073 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:24.068+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:22]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:22]: entry->timestamp=2023-05-31T20:38:20.626928+0000 2023-05-31T20:38:24.074 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:24.068+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:22]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:22]:entry[obj00000000000000000012]: bucket sync single entry (source_zone=3d163b1e-09c7-47e2-ae87-add4aace3f38) b=bucket sync single entry (source_zone=3d163b1e-09c7-47e2-ae87-add4aace3f38) b= log_entry=00000000002.53.6 op=0 op_state=1 2023-05-31T20:38:24.074 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:24.068+0000 7f22572d6780 0 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:22]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:22]:entry[obj00000000000000000012]: : injecting data sync error on key=obj00000000000000000012 2023-05-31T20:38:24.074 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:24.068+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:22]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:22]: listing bilog for incremental sync; position=00000000002.53.6 2023-05-31T20:38:24.074 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:24.068+0000 7f22572d6780 10 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:22]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:22]:entry[obj00000000000000000012]: failed, retcode=-5 ((5) Input/output error) 2023-05-31T20:38:24.074 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:24.068+0000 7f22572d6780 0 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:22]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:22]:entry[obj00000000000000000012]: ERROR: failed to sync object: pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:22/obj00000000000000000012 2023-05-31T20:38:24.114 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:24.110+0000 7f22572d6780 10 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:22]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:22]: a sync operation returned error 2023-05-31T20:38:24.115 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:24.110+0000 7f22572d6780 10 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:22]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:22]: backing out with sync_status=-5 2023-05-31T20:38:24.115 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:24.110+0000 7f22572d6780 5 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:22]: incremental sync on bucket failed, retcode=-5 2023-05-31T20:38:24.116 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:24.110+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:22]: ERROR: incremental sync failed. error: -5 2023-05-31T20:38:24.116 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:24.110+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:22]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-05-31T20:38:24.116 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:24.110+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:22]: sync status for source bucket shard: 0 2023-05-31T20:38:24.116 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:24.110+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:22]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:22]: listing bilog for incremental sync; position= 2023-05-31T20:38:24.159 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:24.152+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:22]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:22]: parsed entry: id=00000000002.53.6 iter->object=obj00000000000000000012 iter->instance= name=obj00000000000000000012 instance= ns= 2023-05-31T20:38:24.160 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:24.152+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:22]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:22]: syncing object: pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:22/obj00000000000000000012 2023-05-31T20:38:24.160 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:24.152+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:22]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:22]: entry->timestamp=2023-05-31T20:38:20.626928+0000 2023-05-31T20:38:24.161 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:24.152+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:22]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:22]:entry[obj00000000000000000012]: bucket sync single entry (source_zone=3d163b1e-09c7-47e2-ae87-add4aace3f38) b=bucket sync single entry (source_zone=3d163b1e-09c7-47e2-ae87-add4aace3f38) b= log_entry=00000000002.53.6 op=0 op_state=1 2023-05-31T20:38:24.161 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:24.152+0000 7f22572d6780 5 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:22]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:22]:entry[obj00000000000000000012]: bucket sync: sync obj: 3d163b1e-09c7-47e2-ae87-add4aace3f38/:pklxkd-58[3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23])/obj00000000000000000012[0] 2023-05-31T20:38:24.161 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:24.152+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:22]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:22]: listing bilog for incremental sync; position=00000000002.53.6 2023-05-31T20:38:24.161 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:24.153+0000 7f22572d6780 10 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:22]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:22]:entry[obj00000000000000000012]: success 2023-05-31T20:38:24.162 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:24.153+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:22]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:22]: updating marker marker_oid=bucket.sync-status.3d163b1e-09c7-47e2-ae87-add4aace3f38:pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:22:5 marker=00000000002.53.6 timestamp=2023-05-31T20:38:20.626928+0000 2023-05-31T20:38:24.204 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:24.199+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:0]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-05-31T20:38:24.204 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:24.199+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:1]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-05-31T20:38:24.204 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:24.199+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:2]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-05-31T20:38:24.205 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:24.199+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:3]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-05-31T20:38:24.205 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:24.199+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:4]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-05-31T20:38:24.205 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:24.199+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:5]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-05-31T20:38:24.205 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:24.199+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:7]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-05-31T20:38:24.205 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:24.199+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:6]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-05-31T20:38:24.205 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:24.199+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:2]: sync status for source bucket shard: 0 2023-05-31T20:38:24.206 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:24.199+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:0]: sync status for source bucket shard: 0 2023-05-31T20:38:24.206 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:24.199+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:2]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:2]: listing bilog for incremental sync; position= 2023-05-31T20:38:24.206 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:24.199+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:0]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:0]: listing bilog for incremental sync; position= 2023-05-31T20:38:24.206 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:24.199+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:5]: sync status for source bucket shard: 0 2023-05-31T20:38:24.206 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:24.199+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:1]: sync status for source bucket shard: 0 2023-05-31T20:38:24.206 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:24.199+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:5]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:5]: listing bilog for incremental sync; position= 2023-05-31T20:38:24.206 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:24.199+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:1]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:1]: listing bilog for incremental sync; position= 2023-05-31T20:38:24.207 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:24.199+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:4]: sync status for source bucket shard: 0 2023-05-31T20:38:24.207 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:24.200+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:4]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:4]: listing bilog for incremental sync; position= 2023-05-31T20:38:24.207 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:24.200+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:8]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-05-31T20:38:24.207 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:24.200+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:6]: sync status for source bucket shard: 0 2023-05-31T20:38:24.207 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:24.200+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:7]: sync status for source bucket shard: 0 2023-05-31T20:38:24.207 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:24.200+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:6]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:6]: listing bilog for incremental sync; position= 2023-05-31T20:38:24.208 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:24.200+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:7]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:7]: listing bilog for incremental sync; position= 2023-05-31T20:38:24.212 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:24.200+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:9]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-05-31T20:38:24.213 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:24.200+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:11]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-05-31T20:38:24.213 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:24.200+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:10]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-05-31T20:38:24.215 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:24.201+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:12]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-05-31T20:38:24.215 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:24.201+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:11]: sync status for source bucket shard: 0 2023-05-31T20:38:24.215 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:24.201+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:11]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:11]: listing bilog for incremental sync; position= 2023-05-31T20:38:24.215 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:24.201+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:0]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:0]: parsed entry: id=00000000001.58.6 iter->object=obj00000000000000000060 iter->instance= name=obj00000000000000000060 instance= ns= 2023-05-31T20:38:24.215 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:24.201+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:0]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:0]: syncing object: pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:0/obj00000000000000000060 2023-05-31T20:38:24.216 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:24.201+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:0]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:0]: entry->timestamp=2023-05-31T20:38:22.575673+0000 2023-05-31T20:38:24.216 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:24.201+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:0]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:0]:entry[obj00000000000000000060]: bucket sync single entry (source_zone=3d163b1e-09c7-47e2-ae87-add4aace3f38) b=bucket sync single entry (source_zone=3d163b1e-09c7-47e2-ae87-add4aace3f38) b= log_entry=00000000001.58.6 op=0 op_state=1 2023-05-31T20:38:24.216 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:24.201+0000 7f22572d6780 5 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:0]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:0]:entry[obj00000000000000000060]: bucket sync: sync obj: 3d163b1e-09c7-47e2-ae87-add4aace3f38/:pklxkd-58[3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23])/obj00000000000000000060[0] 2023-05-31T20:38:24.216 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:24.201+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:0]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:0]: listing bilog for incremental sync; position=00000000001.58.6 2023-05-31T20:38:24.216 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:24.202+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:8]: sync status for source bucket shard: 0 2023-05-31T20:38:24.216 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:24.202+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:9]: sync status for source bucket shard: 0 2023-05-31T20:38:24.217 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:24.202+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:8]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:8]: listing bilog for incremental sync; position= 2023-05-31T20:38:24.217 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:24.202+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:9]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:9]: listing bilog for incremental sync; position= 2023-05-31T20:38:24.217 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:24.202+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:5]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:5]: parsed entry: id=00000000001.58.6 iter->object=obj00000000000000000065 iter->instance= name=obj00000000000000000065 instance= ns= 2023-05-31T20:38:24.217 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:24.202+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:5]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:5]: syncing object: pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:5/obj00000000000000000065 2023-05-31T20:38:24.217 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:24.202+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:5]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:5]: entry->timestamp=2023-05-31T20:38:22.003695+0000 2023-05-31T20:38:24.217 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:24.202+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:5]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:5]:entry[obj00000000000000000065]: bucket sync single entry (source_zone=3d163b1e-09c7-47e2-ae87-add4aace3f38) b=bucket sync single entry (source_zone=3d163b1e-09c7-47e2-ae87-add4aace3f38) b= log_entry=00000000001.58.6 op=0 op_state=1 2023-05-31T20:38:24.217 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:24.202+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:13]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-05-31T20:38:24.218 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:24.202+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:14]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-05-31T20:38:24.218 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:24.202+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:15]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-05-31T20:38:24.218 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:24.202+0000 7f22572d6780 5 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:5]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:5]:entry[obj00000000000000000065]: bucket sync: sync obj: 3d163b1e-09c7-47e2-ae87-add4aace3f38/:pklxkd-58[3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23])/obj00000000000000000065[0] 2023-05-31T20:38:24.218 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:24.202+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:5]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:5]: listing bilog for incremental sync; position=00000000001.58.6 2023-05-31T20:38:24.218 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:24.202+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:1]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:1]: parsed entry: id=00000000002.74.6 iter->object=obj00000000000000000000 iter->instance= name=obj00000000000000000000 instance= ns= 2023-05-31T20:38:24.218 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:24.202+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:1]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:1]: syncing object: pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:1/obj00000000000000000000 2023-05-31T20:38:24.219 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:24.202+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:1]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:1]: entry->timestamp=2023-05-31T20:38:21.972620+0000 2023-05-31T20:38:24.219 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:24.202+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:1]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:1]:entry[obj00000000000000000000]: bucket sync single entry (source_zone=3d163b1e-09c7-47e2-ae87-add4aace3f38) b=bucket sync single entry (source_zone=3d163b1e-09c7-47e2-ae87-add4aace3f38) b= log_entry=00000000002.74.6 op=0 op_state=1 2023-05-31T20:38:24.219 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:24.202+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:16]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-05-31T20:38:24.219 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:24.202+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:18]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-05-31T20:38:24.219 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:24.202+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:4]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:4]: parsed entry: id=00000000002.91.6 iter->object=obj00000000000000000033 iter->instance= name=obj00000000000000000033 instance= ns= 2023-05-31T20:38:24.219 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:24.202+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:4]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:4]: syncing object: pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:4/obj00000000000000000033 2023-05-31T20:38:24.219 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:24.202+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:4]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:4]: entry->timestamp=2023-05-31T20:38:22.084294+0000 2023-05-31T20:38:24.220 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:24.202+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:4]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:4]:entry[obj00000000000000000033]: bucket sync single entry (source_zone=3d163b1e-09c7-47e2-ae87-add4aace3f38) b=bucket sync single entry (source_zone=3d163b1e-09c7-47e2-ae87-add4aace3f38) b= log_entry=00000000002.91.6 op=0 op_state=1 2023-05-31T20:38:24.220 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:24.202+0000 7f22572d6780 5 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:1]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:1]:entry[obj00000000000000000000]: bucket sync: sync obj: 3d163b1e-09c7-47e2-ae87-add4aace3f38/:pklxkd-58[3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23])/obj00000000000000000000[0] 2023-05-31T20:38:24.220 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:24.202+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:1]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:1]: listing bilog for incremental sync; position=00000000002.74.6 2023-05-31T20:38:24.220 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:24.202+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:6]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:6]: parsed entry: id=00000000002.62.6 iter->object=obj00000000000000000005 iter->instance= name=obj00000000000000000005 instance= ns= 2023-05-31T20:38:24.220 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:24.202+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:6]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:6]: syncing object: pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:6/obj00000000000000000005 2023-05-31T20:38:24.220 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:24.202+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:6]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:6]: entry->timestamp=2023-05-31T20:38:21.709914+0000 2023-05-31T20:38:24.221 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:24.202+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:6]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:6]:entry[obj00000000000000000005]: bucket sync single entry (source_zone=3d163b1e-09c7-47e2-ae87-add4aace3f38) b=bucket sync single entry (source_zone=3d163b1e-09c7-47e2-ae87-add4aace3f38) b= log_entry=00000000002.62.6 op=0 op_state=1 2023-05-31T20:38:24.221 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:24.202+0000 7f22572d6780 5 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:4]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:4]:entry[obj00000000000000000033]: bucket sync: sync obj: 3d163b1e-09c7-47e2-ae87-add4aace3f38/:pklxkd-58[3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23])/obj00000000000000000033[0] 2023-05-31T20:38:24.221 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:24.202+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:4]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:4]: listing bilog for incremental sync; position=00000000002.91.6 2023-05-31T20:38:24.221 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:24.202+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:7]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:7]: parsed entry: id=00000000002.41.6 iter->object=obj00000000000000000016 iter->instance= name=obj00000000000000000016 instance= ns= 2023-05-31T20:38:24.221 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:24.202+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:7]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:7]: syncing object: pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:7/obj00000000000000000016 2023-05-31T20:38:24.221 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:24.202+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:7]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:7]: entry->timestamp=2023-05-31T20:38:22.140716+0000 2023-05-31T20:38:24.221 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:24.202+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:7]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:7]:entry[obj00000000000000000016]: bucket sync single entry (source_zone=3d163b1e-09c7-47e2-ae87-add4aace3f38) b=bucket sync single entry (source_zone=3d163b1e-09c7-47e2-ae87-add4aace3f38) b= log_entry=00000000002.41.6 op=0 op_state=1 2023-05-31T20:38:24.222 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:24.203+0000 7f22572d6780 5 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:6]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:6]:entry[obj00000000000000000005]: bucket sync: sync obj: 3d163b1e-09c7-47e2-ae87-add4aace3f38/:pklxkd-58[3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23])/obj00000000000000000005[0] 2023-05-31T20:38:24.222 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:24.203+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:6]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:6]: listing bilog for incremental sync; position=00000000002.62.6 2023-05-31T20:38:24.222 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:24.203+0000 7f22572d6780 5 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:7]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:7]:entry[obj00000000000000000016]: bucket sync: sync obj: 3d163b1e-09c7-47e2-ae87-add4aace3f38/:pklxkd-58[3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23])/obj00000000000000000016[0] 2023-05-31T20:38:24.222 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:24.203+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:7]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:7]: listing bilog for incremental sync; position=00000000002.41.6 2023-05-31T20:38:24.222 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:24.203+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:17]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-05-31T20:38:24.222 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:24.203+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:19]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-05-31T20:38:24.223 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:24.203+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:20]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-05-31T20:38:24.223 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:24.203+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:21]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-05-31T20:38:24.223 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:24.203+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:22]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-05-31T20:38:24.223 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:24.204+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:13]: sync status for source bucket shard: 0 2023-05-31T20:38:24.223 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:24.204+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:16]: sync status for source bucket shard: 0 2023-05-31T20:38:24.223 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:24.204+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:18]: sync status for source bucket shard: 0 2023-05-31T20:38:24.223 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:24.204+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:23]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-05-31T20:38:24.224 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:24.204+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:13]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:13]: listing bilog for incremental sync; position= 2023-05-31T20:38:24.224 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:24.204+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:16]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:16]: listing bilog for incremental sync; position= 2023-05-31T20:38:24.224 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:24.204+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:18]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:18]: listing bilog for incremental sync; position= 2023-05-31T20:38:24.224 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:24.204+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:24]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-05-31T20:38:24.224 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:24.204+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:25]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-05-31T20:38:24.224 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:24.204+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:26]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-05-31T20:38:24.225 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:24.204+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:20]: sync status for source bucket shard: 0 2023-05-31T20:38:24.225 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:24.204+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:20]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:20]: listing bilog for incremental sync; position= 2023-05-31T20:38:24.225 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:24.204+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:27]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-05-31T20:38:24.225 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:24.205+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:14]: sync status for source bucket shard: 0 2023-05-31T20:38:24.225 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:24.205+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:14]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:14]: listing bilog for incremental sync; position= 2023-05-31T20:38:24.225 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:24.205+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:21]: sync status for source bucket shard: 0 2023-05-31T20:38:24.225 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:24.205+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:22]: sync status for source bucket shard: 0 2023-05-31T20:38:24.226 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:24.205+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:17]: sync status for source bucket shard: 0 2023-05-31T20:38:24.226 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:24.205+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:21]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:21]: listing bilog for incremental sync; position= 2023-05-31T20:38:24.226 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:24.205+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:22]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:22]: listing bilog for incremental sync; position= 2023-05-31T20:38:24.226 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:24.205+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:17]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:17]: listing bilog for incremental sync; position= 2023-05-31T20:38:24.226 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:24.206+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:25]: sync status for source bucket shard: 0 2023-05-31T20:38:24.226 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:24.206+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:26]: sync status for source bucket shard: 0 2023-05-31T20:38:24.226 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:24.206+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:23]: sync status for source bucket shard: 0 2023-05-31T20:38:24.227 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:24.206+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:27]: sync status for source bucket shard: 0 2023-05-31T20:38:24.227 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:24.206+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:25]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:25]: listing bilog for incremental sync; position= 2023-05-31T20:38:24.227 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:24.206+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:26]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:26]: listing bilog for incremental sync; position= 2023-05-31T20:38:24.227 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:24.206+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:23]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:23]: listing bilog for incremental sync; position= 2023-05-31T20:38:24.228 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:24.206+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:27]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:27]: listing bilog for incremental sync; position= 2023-05-31T20:38:24.228 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:24.206+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:28]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-05-31T20:38:24.228 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:24.206+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:29]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-05-31T20:38:24.228 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:24.206+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:30]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-05-31T20:38:24.228 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:24.206+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:31]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-05-31T20:38:24.229 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:24.206+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:32]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-05-31T20:38:24.229 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:24.206+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:33]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-05-31T20:38:24.229 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:24.206+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:10]: sync status for source bucket shard: 0 2023-05-31T20:38:24.229 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:24.206+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:34]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-05-31T20:38:24.229 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:24.206+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:35]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-05-31T20:38:24.229 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:24.206+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:10]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:10]: listing bilog for incremental sync; position= 2023-05-31T20:38:24.230 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:24.206+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:3]: sync status for source bucket shard: 0 2023-05-31T20:38:24.230 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:24.206+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:36]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-05-31T20:38:24.230 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:24.207+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:12]: sync status for source bucket shard: 0 2023-05-31T20:38:24.230 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:24.207+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:3]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:3]: listing bilog for incremental sync; position= 2023-05-31T20:38:24.230 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:24.207+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:12]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:12]: listing bilog for incremental sync; position= 2023-05-31T20:38:24.230 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:24.207+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:19]: sync status for source bucket shard: 0 2023-05-31T20:38:24.230 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:24.207+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:19]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:19]: listing bilog for incremental sync; position= 2023-05-31T20:38:24.231 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:24.207+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:29]: sync status for source bucket shard: 0 2023-05-31T20:38:24.231 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:24.207+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:33]: sync status for source bucket shard: 0 2023-05-31T20:38:24.231 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:24.207+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:29]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:29]: listing bilog for incremental sync; position= 2023-05-31T20:38:24.231 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:24.207+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:33]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:33]: listing bilog for incremental sync; position= 2023-05-31T20:38:24.231 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:24.207+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:34]: sync status for source bucket shard: 0 2023-05-31T20:38:24.231 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:24.207+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:34]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:34]: listing bilog for incremental sync; position= 2023-05-31T20:38:24.232 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:24.207+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:35]: sync status for source bucket shard: 0 2023-05-31T20:38:24.232 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:24.207+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:35]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:35]: listing bilog for incremental sync; position= 2023-05-31T20:38:24.232 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:24.208+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:28]: sync status for source bucket shard: 0 2023-05-31T20:38:24.232 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:24.208+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:28]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:28]: listing bilog for incremental sync; position= 2023-05-31T20:38:24.234 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:24.212+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:30]: sync status for source bucket shard: 0 2023-05-31T20:38:24.234 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:24.212+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:30]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:30]: listing bilog for incremental sync; position= 2023-05-31T20:38:24.235 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:24.212+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:36]: sync status for source bucket shard: 0 2023-05-31T20:38:24.235 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:24.212+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:24]: sync status for source bucket shard: 0 2023-05-31T20:38:24.235 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:24.212+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:31]: sync status for source bucket shard: 0 2023-05-31T20:38:24.235 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:24.212+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:15]: sync status for source bucket shard: 0 2023-05-31T20:38:24.235 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:24.212+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:32]: sync status for source bucket shard: 0 2023-05-31T20:38:24.235 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:24.212+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:36]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:36]: listing bilog for incremental sync; position= 2023-05-31T20:38:24.236 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:24.212+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:24]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:24]: listing bilog for incremental sync; position= 2023-05-31T20:38:24.236 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:24.212+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:31]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:31]: listing bilog for incremental sync; position= 2023-05-31T20:38:24.236 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:24.212+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:15]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:15]: listing bilog for incremental sync; position= 2023-05-31T20:38:24.236 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:24.212+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:32]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:32]: listing bilog for incremental sync; position= 2023-05-31T20:38:24.236 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:24.216+0000 7f22572d6780 10 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:1]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:1]:entry[obj00000000000000000000]: success 2023-05-31T20:38:24.236 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:24.216+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:1]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:1]: updating marker marker_oid=bucket.sync-status.3d163b1e-09c7-47e2-ae87-add4aace3f38:pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:1:6 marker=00000000002.74.6 timestamp=2023-05-31T20:38:21.972620+0000 2023-05-31T20:38:24.236 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:24.216+0000 7f22572d6780 10 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:7]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:7]:entry[obj00000000000000000016]: success 2023-05-31T20:38:24.237 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:24.216+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:7]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:7]: updating marker marker_oid=bucket.sync-status.3d163b1e-09c7-47e2-ae87-add4aace3f38:pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:7:6 marker=00000000002.41.6 timestamp=2023-05-31T20:38:22.140716+0000 2023-05-31T20:38:24.237 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:24.216+0000 7f22572d6780 10 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:6]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:6]:entry[obj00000000000000000005]: success 2023-05-31T20:38:24.237 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:24.216+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:6]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:6]: updating marker marker_oid=bucket.sync-status.3d163b1e-09c7-47e2-ae87-add4aace3f38:pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:6:6 marker=00000000002.62.6 timestamp=2023-05-31T20:38:21.709914+0000 2023-05-31T20:38:24.237 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:24.216+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:13]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:13]: parsed entry: id=00000000002.60.6 iter->object=obj00000000000000000009 iter->instance= name=obj00000000000000000009 instance= ns= 2023-05-31T20:38:24.237 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:24.216+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:13]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:13]: syncing object: pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:13/obj00000000000000000009 2023-05-31T20:38:24.237 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:24.216+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:13]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:13]: entry->timestamp=2023-05-31T20:38:21.588236+0000 2023-05-31T20:38:24.238 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:24.216+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:13]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:13]:entry[obj00000000000000000009]: bucket sync single entry (source_zone=3d163b1e-09c7-47e2-ae87-add4aace3f38) b=bucket sync single entry (source_zone=3d163b1e-09c7-47e2-ae87-add4aace3f38) b= log_entry=00000000002.60.6 op=0 op_state=1 2023-05-31T20:38:24.238 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:24.216+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:16]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:16]: parsed entry: id=00000000002.47.6 iter->object=obj00000000000000000012 iter->instance= name=obj00000000000000000012 instance= ns= 2023-05-31T20:38:24.238 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:24.216+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:16]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:16]: syncing object: pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:16/obj00000000000000000012 2023-05-31T20:38:24.238 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:24.216+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:16]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:16]: entry->timestamp=2023-05-31T20:38:22.308395+0000 2023-05-31T20:38:24.239 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:24.216+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:16]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:16]:entry[obj00000000000000000012]: bucket sync single entry (source_zone=3d163b1e-09c7-47e2-ae87-add4aace3f38) b=bucket sync single entry (source_zone=3d163b1e-09c7-47e2-ae87-add4aace3f38) b= log_entry=00000000002.47.6 op=0 op_state=1 2023-05-31T20:38:24.239 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:24.216+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:18]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:18]: parsed entry: id=00000000002.53.6 iter->object=obj00000000000000000013 iter->instance= name=obj00000000000000000013 instance= ns= 2023-05-31T20:38:24.239 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:24.216+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:18]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:18]: syncing object: pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:18/obj00000000000000000013 2023-05-31T20:38:24.239 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:24.216+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:18]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:18]: entry->timestamp=2023-05-31T20:38:22.736493+0000 2023-05-31T20:38:24.239 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:24.216+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:18]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:18]:entry[obj00000000000000000013]: bucket sync single entry (source_zone=3d163b1e-09c7-47e2-ae87-add4aace3f38) b=bucket sync single entry (source_zone=3d163b1e-09c7-47e2-ae87-add4aace3f38) b= log_entry=00000000002.53.6 op=0 op_state=1 2023-05-31T20:38:24.239 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:24.216+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:20]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:20]: parsed entry: id=00000000002.70.6 iter->object=obj00000000000000000006 iter->instance= name=obj00000000000000000006 instance= ns= 2023-05-31T20:38:24.240 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:24.216+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:20]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:20]: syncing object: pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:20/obj00000000000000000006 2023-05-31T20:38:24.240 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:24.216+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:20]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:20]: entry->timestamp=2023-05-31T20:38:21.735843+0000 2023-05-31T20:38:24.240 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:24.216+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:20]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:20]:entry[obj00000000000000000006]: bucket sync single entry (source_zone=3d163b1e-09c7-47e2-ae87-add4aace3f38) b=bucket sync single entry (source_zone=3d163b1e-09c7-47e2-ae87-add4aace3f38) b= log_entry=00000000002.70.6 op=0 op_state=1 2023-05-31T20:38:24.240 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:24.216+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:14]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:14]: parsed entry: id=00000000002.62.6 iter->object=obj00000000000000000040 iter->instance= name=obj00000000000000000040 instance= ns= 2023-05-31T20:38:24.240 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:24.216+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:14]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:14]: syncing object: pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:14/obj00000000000000000040 2023-05-31T20:38:24.240 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:24.216+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:14]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:14]: entry->timestamp=2023-05-31T20:38:22.327307+0000 2023-05-31T20:38:24.240 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:24.216+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:14]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:14]:entry[obj00000000000000000040]: bucket sync single entry (source_zone=3d163b1e-09c7-47e2-ae87-add4aace3f38) b=bucket sync single entry (source_zone=3d163b1e-09c7-47e2-ae87-add4aace3f38) b= log_entry=00000000002.62.6 op=0 op_state=1 2023-05-31T20:38:24.241 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:24.216+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:21]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:21]: parsed entry: id=00000000002.88.6 iter->object=obj00000000000000000017 iter->instance= name=obj00000000000000000017 instance= ns= 2023-05-31T20:38:24.241 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:24.216+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:21]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:21]: syncing object: pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:21/obj00000000000000000017 2023-05-31T20:38:24.241 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:24.216+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:21]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:21]: entry->timestamp=2023-05-31T20:38:22.548865+0000 2023-05-31T20:38:24.241 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:24.216+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:21]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:21]:entry[obj00000000000000000017]: bucket sync single entry (source_zone=3d163b1e-09c7-47e2-ae87-add4aace3f38) b=bucket sync single entry (source_zone=3d163b1e-09c7-47e2-ae87-add4aace3f38) b= log_entry=00000000002.88.6 op=0 op_state=1 2023-05-31T20:38:24.241 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:24.216+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:22]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:22]: parsed entry: id=00000000002.55.6 iter->object=obj00000000000000000007 iter->instance= name=obj00000000000000000007 instance= ns= 2023-05-31T20:38:24.242 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:24.216+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:22]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:22]: syncing object: pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:22/obj00000000000000000007 2023-05-31T20:38:24.242 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:24.216+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:22]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:22]: entry->timestamp=2023-05-31T20:38:22.793692+0000 2023-05-31T20:38:24.242 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:24.216+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:22]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:22]:entry[obj00000000000000000007]: bucket sync single entry (source_zone=3d163b1e-09c7-47e2-ae87-add4aace3f38) b=bucket sync single entry (source_zone=3d163b1e-09c7-47e2-ae87-add4aace3f38) b= log_entry=00000000002.55.6 op=0 op_state=1 2023-05-31T20:38:24.242 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:24.216+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:17]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:17]: parsed entry: id=00000000002.58.6 iter->object=obj00000000000000000002 iter->instance= name=obj00000000000000000002 instance= ns= 2023-05-31T20:38:24.242 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:24.216+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:17]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:17]: syncing object: pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:17/obj00000000000000000002 2023-05-31T20:38:24.243 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:24.216+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:17]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:17]: entry->timestamp=2023-05-31T20:38:22.061689+0000 2023-05-31T20:38:24.243 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:24.216+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:17]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:17]:entry[obj00000000000000000002]: bucket sync single entry (source_zone=3d163b1e-09c7-47e2-ae87-add4aace3f38) b=bucket sync single entry (source_zone=3d163b1e-09c7-47e2-ae87-add4aace3f38) b= log_entry=00000000002.58.6 op=0 op_state=1 2023-05-31T20:38:24.243 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:24.216+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:23]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:23]: parsed entry: id=00000000002.59.6 iter->object=obj00000000000000000039 iter->instance= name=obj00000000000000000039 instance= ns= 2023-05-31T20:38:24.243 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:24.216+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:23]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:23]: syncing object: pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:23/obj00000000000000000039 2023-05-31T20:38:24.243 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:24.216+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:23]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:23]: entry->timestamp=2023-05-31T20:38:22.383805+0000 2023-05-31T20:38:24.243 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:24.216+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:23]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:23]:entry[obj00000000000000000039]: bucket sync single entry (source_zone=3d163b1e-09c7-47e2-ae87-add4aace3f38) b=bucket sync single entry (source_zone=3d163b1e-09c7-47e2-ae87-add4aace3f38) b= log_entry=00000000002.59.6 op=0 op_state=1 2023-05-31T20:38:24.243 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:24.216+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:27]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:27]: parsed entry: id=00000000002.93.6 iter->object=obj00000000000000000053 iter->instance= name=obj00000000000000000053 instance= ns= 2023-05-31T20:38:24.244 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:24.216+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:27]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:27]: syncing object: pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:27/obj00000000000000000053 2023-05-31T20:38:24.244 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:24.216+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:27]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:27]: entry->timestamp=2023-05-31T20:38:22.873398+0000 2023-05-31T20:38:24.244 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:24.216+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:27]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:27]:entry[obj00000000000000000053]: bucket sync single entry (source_zone=3d163b1e-09c7-47e2-ae87-add4aace3f38) b=bucket sync single entry (source_zone=3d163b1e-09c7-47e2-ae87-add4aace3f38) b= log_entry=00000000002.93.6 op=0 op_state=1 2023-05-31T20:38:24.244 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:24.216+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:10]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:10]: parsed entry: id=00000000002.56.6 iter->object=obj00000000000000000026 iter->instance= name=obj00000000000000000026 instance= ns= 2023-05-31T20:38:24.244 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:24.216+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:10]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:10]: syncing object: pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:10/obj00000000000000000026 2023-05-31T20:38:24.244 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:24.216+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:10]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:10]: entry->timestamp=2023-05-31T20:38:22.407332+0000 2023-05-31T20:38:24.245 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:24.216+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:10]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:10]:entry[obj00000000000000000026]: bucket sync single entry (source_zone=3d163b1e-09c7-47e2-ae87-add4aace3f38) b=bucket sync single entry (source_zone=3d163b1e-09c7-47e2-ae87-add4aace3f38) b= log_entry=00000000002.56.6 op=0 op_state=1 2023-05-31T20:38:24.245 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:24.216+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:3]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:3]: parsed entry: id=00000000002.88.6 iter->object=obj00000000000000000001 iter->instance= name=obj00000000000000000001 instance= ns= 2023-05-31T20:38:24.245 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:24.216+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:3]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:3]: syncing object: pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:3/obj00000000000000000001 2023-05-31T20:38:24.245 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:24.216+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:3]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:3]: entry->timestamp=2023-05-31T20:38:22.169017+0000 2023-05-31T20:38:24.245 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:24.216+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:3]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:3]:entry[obj00000000000000000001]: bucket sync single entry (source_zone=3d163b1e-09c7-47e2-ae87-add4aace3f38) b=bucket sync single entry (source_zone=3d163b1e-09c7-47e2-ae87-add4aace3f38) b= log_entry=00000000002.88.6 op=0 op_state=1 2023-05-31T20:38:24.245 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:24.216+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:12]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:12]: parsed entry: id=00000000001.86.6 iter->object=obj00000000000000000062 iter->instance= name=obj00000000000000000062 instance= ns= 2023-05-31T20:38:24.245 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:24.216+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:12]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:12]: syncing object: pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:12/obj00000000000000000062 2023-05-31T20:38:24.246 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:24.216+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:12]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:12]: entry->timestamp=2023-05-31T20:38:21.794173+0000 2023-05-31T20:38:24.246 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:24.216+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:12]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:12]:entry[obj00000000000000000062]: bucket sync single entry (source_zone=3d163b1e-09c7-47e2-ae87-add4aace3f38) b=bucket sync single entry (source_zone=3d163b1e-09c7-47e2-ae87-add4aace3f38) b= log_entry=00000000001.86.6 op=0 op_state=1 2023-05-31T20:38:24.246 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:24.216+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:19]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:19]: parsed entry: id=00000000002.47.6 iter->object=obj00000000000000000045 iter->instance= name=obj00000000000000000045 instance= ns= 2023-05-31T20:38:24.246 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:24.217+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:19]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:19]: syncing object: pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:19/obj00000000000000000045 2023-05-31T20:38:24.246 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:24.217+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:19]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:19]: entry->timestamp=2023-05-31T20:38:22.250679+0000 2023-05-31T20:38:24.246 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:24.217+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:19]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:19]:entry[obj00000000000000000045]: bucket sync single entry (source_zone=3d163b1e-09c7-47e2-ae87-add4aace3f38) b=bucket sync single entry (source_zone=3d163b1e-09c7-47e2-ae87-add4aace3f38) b= log_entry=00000000002.47.6 op=0 op_state=1 2023-05-31T20:38:24.247 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:24.217+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:29]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:29]: parsed entry: id=00000000002.51.6 iter->object=obj00000000000000000004 iter->instance= name=obj00000000000000000004 instance= ns= 2023-05-31T20:38:24.247 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:24.217+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:29]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:29]: syncing object: pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:29/obj00000000000000000004 2023-05-31T20:38:24.247 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:24.217+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:29]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:29]: entry->timestamp=2023-05-31T20:38:22.227799+0000 2023-05-31T20:38:24.247 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:24.217+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:29]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:29]:entry[obj00000000000000000004]: bucket sync single entry (source_zone=3d163b1e-09c7-47e2-ae87-add4aace3f38) b=bucket sync single entry (source_zone=3d163b1e-09c7-47e2-ae87-add4aace3f38) b= log_entry=00000000002.51.6 op=0 op_state=1 2023-05-31T20:38:24.247 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:24.217+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:33]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:33]: parsed entry: id=00000000002.50.6 iter->object=obj00000000000000000019 iter->instance= name=obj00000000000000000019 instance= ns= 2023-05-31T20:38:24.247 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:24.217+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:33]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:33]: syncing object: pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:33/obj00000000000000000019 2023-05-31T20:38:24.247 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:24.217+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:33]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:33]: entry->timestamp=2023-05-31T20:38:21.559227+0000 2023-05-31T20:38:24.248 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:24.217+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:33]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:33]:entry[obj00000000000000000019]: bucket sync single entry (source_zone=3d163b1e-09c7-47e2-ae87-add4aace3f38) b=bucket sync single entry (source_zone=3d163b1e-09c7-47e2-ae87-add4aace3f38) b= log_entry=00000000002.50.6 op=0 op_state=1 2023-05-31T20:38:24.248 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:24.217+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:34]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:34]: parsed entry: id=00000000001.56.6 iter->object=obj00000000000000000057 iter->instance= name=obj00000000000000000057 instance= ns= 2023-05-31T20:38:24.248 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:24.217+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:34]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:34]: syncing object: pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:34/obj00000000000000000057 2023-05-31T20:38:24.248 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:24.217+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:34]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:34]: entry->timestamp=2023-05-31T20:38:21.818642+0000 2023-05-31T20:38:24.248 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:24.217+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:34]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:34]:entry[obj00000000000000000057]: bucket sync single entry (source_zone=3d163b1e-09c7-47e2-ae87-add4aace3f38) b=bucket sync single entry (source_zone=3d163b1e-09c7-47e2-ae87-add4aace3f38) b= log_entry=00000000001.56.6 op=0 op_state=1 2023-05-31T20:38:24.248 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:24.217+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:35]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:35]: parsed entry: id=00000000002.52.6 iter->object=obj00000000000000000018 iter->instance= name=obj00000000000000000018 instance= ns= 2023-05-31T20:38:24.249 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:24.217+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:35]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:35]: syncing object: pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:35/obj00000000000000000018 2023-05-31T20:38:24.249 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:24.217+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:35]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:35]: entry->timestamp=2023-05-31T20:38:21.887881+0000 2023-05-31T20:38:24.249 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:24.217+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:35]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:35]:entry[obj00000000000000000018]: bucket sync single entry (source_zone=3d163b1e-09c7-47e2-ae87-add4aace3f38) b=bucket sync single entry (source_zone=3d163b1e-09c7-47e2-ae87-add4aace3f38) b= log_entry=00000000002.52.6 op=0 op_state=1 2023-05-31T20:38:24.249 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:24.217+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:28]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:28]: parsed entry: id=00000000002.58.6 iter->object=obj00000000000000000020 iter->instance= name=obj00000000000000000020 instance= ns= 2023-05-31T20:38:24.249 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:24.217+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:28]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:28]: syncing object: pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:28/obj00000000000000000020 2023-05-31T20:38:24.249 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:24.217+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:28]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:28]: entry->timestamp=2023-05-31T20:38:22.465273+0000 2023-05-31T20:38:24.249 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:24.217+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:28]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:28]:entry[obj00000000000000000020]: bucket sync single entry (source_zone=3d163b1e-09c7-47e2-ae87-add4aace3f38) b=bucket sync single entry (source_zone=3d163b1e-09c7-47e2-ae87-add4aace3f38) b= log_entry=00000000002.58.6 op=0 op_state=1 2023-05-31T20:38:24.250 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:24.217+0000 7f22572d6780 10 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:4]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:4]:entry[obj00000000000000000033]: success 2023-05-31T20:38:24.250 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:24.217+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:4]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:4]: updating marker marker_oid=bucket.sync-status.3d163b1e-09c7-47e2-ae87-add4aace3f38:pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:4:6 marker=00000000002.91.6 timestamp=2023-05-31T20:38:22.084294+0000 2023-05-31T20:38:24.250 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:24.217+0000 7f22572d6780 5 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:13]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:13]:entry[obj00000000000000000009]: bucket sync: sync obj: 3d163b1e-09c7-47e2-ae87-add4aace3f38/:pklxkd-58[3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23])/obj00000000000000000009[0] 2023-05-31T20:38:24.250 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:24.217+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:13]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:13]: listing bilog for incremental sync; position=00000000002.60.6 2023-05-31T20:38:24.251 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:24.217+0000 7f22572d6780 5 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:16]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:16]:entry[obj00000000000000000012]: bucket sync: sync obj: 3d163b1e-09c7-47e2-ae87-add4aace3f38/:pklxkd-58[3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23])/obj00000000000000000012[0] 2023-05-31T20:38:24.251 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:24.217+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:16]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:16]: listing bilog for incremental sync; position=00000000002.47.6 2023-05-31T20:38:24.251 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:24.217+0000 7f22572d6780 5 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:18]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:18]:entry[obj00000000000000000013]: bucket sync: sync obj: 3d163b1e-09c7-47e2-ae87-add4aace3f38/:pklxkd-58[3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23])/obj00000000000000000013[0] 2023-05-31T20:38:24.251 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:24.217+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:18]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:18]: listing bilog for incremental sync; position=00000000002.53.6 2023-05-31T20:38:24.251 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:24.217+0000 7f22572d6780 5 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:20]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:20]:entry[obj00000000000000000006]: bucket sync: sync obj: 3d163b1e-09c7-47e2-ae87-add4aace3f38/:pklxkd-58[3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23])/obj00000000000000000006[0] 2023-05-31T20:38:24.252 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:24.217+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:20]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:20]: listing bilog for incremental sync; position=00000000002.70.6 2023-05-31T20:38:24.252 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:24.217+0000 7f22572d6780 5 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:14]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:14]:entry[obj00000000000000000040]: bucket sync: sync obj: 3d163b1e-09c7-47e2-ae87-add4aace3f38/:pklxkd-58[3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23])/obj00000000000000000040[0] 2023-05-31T20:38:24.252 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:24.217+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:14]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:14]: listing bilog for incremental sync; position=00000000002.62.6 2023-05-31T20:38:24.252 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:24.217+0000 7f22572d6780 5 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:21]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:21]:entry[obj00000000000000000017]: bucket sync: sync obj: 3d163b1e-09c7-47e2-ae87-add4aace3f38/:pklxkd-58[3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23])/obj00000000000000000017[0] 2023-05-31T20:38:24.252 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:24.217+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:21]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:21]: listing bilog for incremental sync; position=00000000002.88.6 2023-05-31T20:38:24.252 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:24.217+0000 7f22572d6780 5 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:22]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:22]:entry[obj00000000000000000007]: bucket sync: sync obj: 3d163b1e-09c7-47e2-ae87-add4aace3f38/:pklxkd-58[3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23])/obj00000000000000000007[0] 2023-05-31T20:38:24.253 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:24.217+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:22]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:22]: listing bilog for incremental sync; position=00000000002.55.6 2023-05-31T20:38:24.253 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:24.217+0000 7f22572d6780 5 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:17]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:17]:entry[obj00000000000000000002]: bucket sync: sync obj: 3d163b1e-09c7-47e2-ae87-add4aace3f38/:pklxkd-58[3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23])/obj00000000000000000002[0] 2023-05-31T20:38:24.253 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:24.217+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:17]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:17]: listing bilog for incremental sync; position=00000000002.58.6 2023-05-31T20:38:24.253 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:24.217+0000 7f22572d6780 5 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:23]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:23]:entry[obj00000000000000000039]: bucket sync: sync obj: 3d163b1e-09c7-47e2-ae87-add4aace3f38/:pklxkd-58[3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23])/obj00000000000000000039[0] 2023-05-31T20:38:24.253 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:24.217+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:23]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:23]: listing bilog for incremental sync; position=00000000002.59.6 2023-05-31T20:38:24.253 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:24.217+0000 7f22572d6780 0 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:27]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:27]:entry[obj00000000000000000053]: : injecting data sync error on key=obj00000000000000000053 2023-05-31T20:38:24.254 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:24.217+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:27]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:27]: listing bilog for incremental sync; position=00000000002.93.6 2023-05-31T20:38:24.254 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:24.217+0000 7f22572d6780 5 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:10]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:10]:entry[obj00000000000000000026]: bucket sync: sync obj: 3d163b1e-09c7-47e2-ae87-add4aace3f38/:pklxkd-58[3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23])/obj00000000000000000026[0] 2023-05-31T20:38:24.254 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:24.217+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:10]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:10]: listing bilog for incremental sync; position=00000000002.56.6 2023-05-31T20:38:24.254 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:24.217+0000 7f22572d6780 5 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:3]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:3]:entry[obj00000000000000000001]: bucket sync: sync obj: 3d163b1e-09c7-47e2-ae87-add4aace3f38/:pklxkd-58[3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23])/obj00000000000000000001[0] 2023-05-31T20:38:24.254 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:24.217+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:3]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:3]: listing bilog for incremental sync; position=00000000002.88.6 2023-05-31T20:38:24.254 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:24.217+0000 7f22572d6780 5 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:12]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:12]:entry[obj00000000000000000062]: bucket sync: sync obj: 3d163b1e-09c7-47e2-ae87-add4aace3f38/:pklxkd-58[3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23])/obj00000000000000000062[0] 2023-05-31T20:38:24.254 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:24.217+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:12]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:12]: listing bilog for incremental sync; position=00000000001.86.6 2023-05-31T20:38:24.255 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:24.217+0000 7f22572d6780 5 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:19]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:19]:entry[obj00000000000000000045]: bucket sync: sync obj: 3d163b1e-09c7-47e2-ae87-add4aace3f38/:pklxkd-58[3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23])/obj00000000000000000045[0] 2023-05-31T20:38:24.255 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:24.217+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:19]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:19]: listing bilog for incremental sync; position=00000000002.47.6 2023-05-31T20:38:24.255 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:24.217+0000 7f22572d6780 5 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:29]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:29]:entry[obj00000000000000000004]: bucket sync: sync obj: 3d163b1e-09c7-47e2-ae87-add4aace3f38/:pklxkd-58[3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23])/obj00000000000000000004[0] 2023-05-31T20:38:24.255 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:24.217+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:29]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:29]: listing bilog for incremental sync; position=00000000002.51.6 2023-05-31T20:38:24.255 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:24.217+0000 7f22572d6780 5 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:33]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:33]:entry[obj00000000000000000019]: bucket sync: sync obj: 3d163b1e-09c7-47e2-ae87-add4aace3f38/:pklxkd-58[3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23])/obj00000000000000000019[0] 2023-05-31T20:38:24.255 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:24.217+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:33]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:33]: listing bilog for incremental sync; position=00000000002.50.6 2023-05-31T20:38:24.255 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:24.217+0000 7f22572d6780 5 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:34]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:34]:entry[obj00000000000000000057]: bucket sync: sync obj: 3d163b1e-09c7-47e2-ae87-add4aace3f38/:pklxkd-58[3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23])/obj00000000000000000057[0] 2023-05-31T20:38:24.256 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:24.217+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:34]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:34]: listing bilog for incremental sync; position=00000000001.56.6 2023-05-31T20:38:24.256 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:24.217+0000 7f22572d6780 0 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:35]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:35]:entry[obj00000000000000000018]: : injecting data sync error on key=obj00000000000000000018 2023-05-31T20:38:24.256 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:24.217+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:35]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:35]: listing bilog for incremental sync; position=00000000002.52.6 2023-05-31T20:38:24.256 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:24.217+0000 7f22572d6780 5 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:28]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:28]:entry[obj00000000000000000020]: bucket sync: sync obj: 3d163b1e-09c7-47e2-ae87-add4aace3f38/:pklxkd-58[3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23])/obj00000000000000000020[0] 2023-05-31T20:38:24.256 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:24.217+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:28]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:28]: listing bilog for incremental sync; position=00000000002.58.6 2023-05-31T20:38:24.256 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:24.217+0000 7f22572d6780 10 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:27]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:27]:entry[obj00000000000000000053]: failed, retcode=-5 ((5) Input/output error) 2023-05-31T20:38:24.257 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:24.217+0000 7f22572d6780 0 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:27]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:27]:entry[obj00000000000000000053]: ERROR: failed to sync object: pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:27/obj00000000000000000053 2023-05-31T20:38:24.257 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:24.217+0000 7f22572d6780 10 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:35]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:35]:entry[obj00000000000000000018]: failed, retcode=-5 ((5) Input/output error) 2023-05-31T20:38:24.257 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:24.217+0000 7f22572d6780 0 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:35]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:35]:entry[obj00000000000000000018]: ERROR: failed to sync object: pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:35/obj00000000000000000018 2023-05-31T20:38:24.257 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:24.220+0000 7f22572d6780 10 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:5]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:5]:entry[obj00000000000000000065]: success 2023-05-31T20:38:24.257 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:24.220+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:5]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:5]: updating marker marker_oid=bucket.sync-status.3d163b1e-09c7-47e2-ae87-add4aace3f38:pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:5:6 marker=00000000001.58.6 timestamp=2023-05-31T20:38:22.003695+0000 2023-05-31T20:38:24.257 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:24.226+0000 7f22572d6780 10 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:13]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:13]:entry[obj00000000000000000009]: success 2023-05-31T20:38:24.258 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:24.226+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:13]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:13]: updating marker marker_oid=bucket.sync-status.3d163b1e-09c7-47e2-ae87-add4aace3f38:pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:13:6 marker=00000000002.60.6 timestamp=2023-05-31T20:38:21.588236+0000 2023-05-31T20:38:24.258 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:24.226+0000 7f22572d6780 10 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:18]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:18]:entry[obj00000000000000000013]: success 2023-05-31T20:38:24.258 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:24.226+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:18]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:18]: updating marker marker_oid=bucket.sync-status.3d163b1e-09c7-47e2-ae87-add4aace3f38:pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:18:6 marker=00000000002.53.6 timestamp=2023-05-31T20:38:22.736493+0000 2023-05-31T20:38:24.258 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:24.226+0000 7f22572d6780 10 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:16]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:16]:entry[obj00000000000000000012]: success 2023-05-31T20:38:24.258 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:24.227+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:16]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:16]: updating marker marker_oid=bucket.sync-status.3d163b1e-09c7-47e2-ae87-add4aace3f38:pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:16:6 marker=00000000002.47.6 timestamp=2023-05-31T20:38:22.308395+0000 2023-05-31T20:38:24.258 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:24.227+0000 7f22572d6780 10 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:21]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:21]:entry[obj00000000000000000017]: success 2023-05-31T20:38:24.258 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:24.227+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:21]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:21]: updating marker marker_oid=bucket.sync-status.3d163b1e-09c7-47e2-ae87-add4aace3f38:pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:21:6 marker=00000000002.88.6 timestamp=2023-05-31T20:38:22.548865+0000 2023-05-31T20:38:24.259 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:24.227+0000 7f22572d6780 10 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:23]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:23]:entry[obj00000000000000000039]: success 2023-05-31T20:38:24.259 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:24.227+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:23]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:23]: updating marker marker_oid=bucket.sync-status.3d163b1e-09c7-47e2-ae87-add4aace3f38:pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:23:6 marker=00000000002.59.6 timestamp=2023-05-31T20:38:22.383805+0000 2023-05-31T20:38:24.259 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:24.227+0000 7f22572d6780 10 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:20]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:20]:entry[obj00000000000000000006]: success 2023-05-31T20:38:24.259 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:24.227+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:20]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:20]: updating marker marker_oid=bucket.sync-status.3d163b1e-09c7-47e2-ae87-add4aace3f38:pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:20:6 marker=00000000002.70.6 timestamp=2023-05-31T20:38:21.735843+0000 2023-05-31T20:38:24.259 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:24.229+0000 7f22572d6780 10 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:14]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:14]:entry[obj00000000000000000040]: success 2023-05-31T20:38:24.259 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:24.229+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:14]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:14]: updating marker marker_oid=bucket.sync-status.3d163b1e-09c7-47e2-ae87-add4aace3f38:pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:14:6 marker=00000000002.62.6 timestamp=2023-05-31T20:38:22.327307+0000 2023-05-31T20:38:24.259 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:24.229+0000 7f22572d6780 10 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:17]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:17]:entry[obj00000000000000000002]: success 2023-05-31T20:38:24.260 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:24.229+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:17]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:17]: updating marker marker_oid=bucket.sync-status.3d163b1e-09c7-47e2-ae87-add4aace3f38:pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:17:6 marker=00000000002.58.6 timestamp=2023-05-31T20:38:22.061689+0000 2023-05-31T20:38:24.260 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:24.229+0000 7f22572d6780 10 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:3]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:3]:entry[obj00000000000000000001]: success 2023-05-31T20:38:24.260 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:24.229+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:3]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:3]: updating marker marker_oid=bucket.sync-status.3d163b1e-09c7-47e2-ae87-add4aace3f38:pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:3:6 marker=00000000002.88.6 timestamp=2023-05-31T20:38:22.169017+0000 2023-05-31T20:38:24.260 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:24.229+0000 7f22572d6780 10 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:10]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:10]:entry[obj00000000000000000026]: success 2023-05-31T20:38:24.260 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:24.229+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:10]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:10]: updating marker marker_oid=bucket.sync-status.3d163b1e-09c7-47e2-ae87-add4aace3f38:pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:10:6 marker=00000000002.56.6 timestamp=2023-05-31T20:38:22.407332+0000 2023-05-31T20:38:24.260 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:24.229+0000 7f22572d6780 10 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:19]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:19]:entry[obj00000000000000000045]: success 2023-05-31T20:38:24.261 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:24.229+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:19]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:19]: updating marker marker_oid=bucket.sync-status.3d163b1e-09c7-47e2-ae87-add4aace3f38:pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:19:6 marker=00000000002.47.6 timestamp=2023-05-31T20:38:22.250679+0000 2023-05-31T20:38:24.261 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:24.230+0000 7f22572d6780 10 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:33]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:33]:entry[obj00000000000000000019]: success 2023-05-31T20:38:24.261 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:24.230+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:33]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:33]: updating marker marker_oid=bucket.sync-status.3d163b1e-09c7-47e2-ae87-add4aace3f38:pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:33:6 marker=00000000002.50.6 timestamp=2023-05-31T20:38:21.559227+0000 2023-05-31T20:38:24.263 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:24.235+0000 7f22572d6780 10 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:22]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:22]:entry[obj00000000000000000007]: success 2023-05-31T20:38:24.264 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:24.235+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:22]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:22]: updating marker marker_oid=bucket.sync-status.3d163b1e-09c7-47e2-ae87-add4aace3f38:pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:22:6 marker=00000000002.55.6 timestamp=2023-05-31T20:38:22.793692+0000 2023-05-31T20:38:24.264 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:24.235+0000 7f22572d6780 10 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:28]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:28]:entry[obj00000000000000000020]: success 2023-05-31T20:38:24.264 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:24.235+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:28]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:28]: updating marker marker_oid=bucket.sync-status.3d163b1e-09c7-47e2-ae87-add4aace3f38:pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:28:6 marker=00000000002.58.6 timestamp=2023-05-31T20:38:22.465273+0000 2023-05-31T20:38:24.264 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:24.235+0000 7f22572d6780 10 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:29]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:29]:entry[obj00000000000000000004]: success 2023-05-31T20:38:24.264 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:24.235+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:29]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:29]: updating marker marker_oid=bucket.sync-status.3d163b1e-09c7-47e2-ae87-add4aace3f38:pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:29:6 marker=00000000002.51.6 timestamp=2023-05-31T20:38:22.227799+0000 2023-05-31T20:38:24.265 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:24.239+0000 7f22572d6780 10 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:12]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:12]:entry[obj00000000000000000062]: success 2023-05-31T20:38:24.265 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:24.239+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:12]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:12]: updating marker marker_oid=bucket.sync-status.3d163b1e-09c7-47e2-ae87-add4aace3f38:pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:12:6 marker=00000000001.86.6 timestamp=2023-05-31T20:38:21.794173+0000 2023-05-31T20:38:24.265 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:24.243+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:2]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:2]: parsed entry: id=00000000002.52.6 iter->object=obj00000000000000000011 iter->instance= name=obj00000000000000000011 instance= ns= 2023-05-31T20:38:24.265 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:24.243+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:2]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:2]: syncing object: pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:2/obj00000000000000000011 2023-05-31T20:38:24.265 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:24.243+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:2]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:2]: entry->timestamp=2023-05-31T20:38:22.712231+0000 2023-05-31T20:38:24.265 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:24.243+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:2]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:2]:entry[obj00000000000000000011]: bucket sync single entry (source_zone=3d163b1e-09c7-47e2-ae87-add4aace3f38) b=bucket sync single entry (source_zone=3d163b1e-09c7-47e2-ae87-add4aace3f38) b= log_entry=00000000002.52.6 op=0 op_state=1 2023-05-31T20:38:24.265 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:24.243+0000 7f22572d6780 5 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:2]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:2]:entry[obj00000000000000000011]: bucket sync: sync obj: 3d163b1e-09c7-47e2-ae87-add4aace3f38/:pklxkd-58[3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23])/obj00000000000000000011[0] 2023-05-31T20:38:24.266 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:24.243+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:2]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:2]: listing bilog for incremental sync; position=00000000002.52.6 2023-05-31T20:38:24.266 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:24.243+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:11]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:11]: parsed entry: id=00000000002.37.6 iter->object=obj00000000000000000030 iter->instance= name=obj00000000000000000030 instance= ns= 2023-05-31T20:38:24.266 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:24.243+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:11]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:11]: syncing object: pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:11/obj00000000000000000030 2023-05-31T20:38:24.266 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:24.243+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:11]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:11]: entry->timestamp=2023-05-31T20:38:21.641789+0000 2023-05-31T20:38:24.266 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:24.243+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:11]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:11]:entry[obj00000000000000000030]: bucket sync single entry (source_zone=3d163b1e-09c7-47e2-ae87-add4aace3f38) b=bucket sync single entry (source_zone=3d163b1e-09c7-47e2-ae87-add4aace3f38) b= log_entry=00000000002.37.6 op=0 op_state=1 2023-05-31T20:38:24.266 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:24.243+0000 7f22572d6780 5 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:11]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:11]:entry[obj00000000000000000030]: bucket sync: sync obj: 3d163b1e-09c7-47e2-ae87-add4aace3f38/:pklxkd-58[3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23])/obj00000000000000000030[0] 2023-05-31T20:38:24.267 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:24.243+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:11]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:11]: listing bilog for incremental sync; position=00000000002.37.6 2023-05-31T20:38:24.267 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:24.244+0000 7f22572d6780 10 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:11]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:11]:entry[obj00000000000000000030]: success 2023-05-31T20:38:24.267 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:24.244+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:11]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:11]: updating marker marker_oid=bucket.sync-status.3d163b1e-09c7-47e2-ae87-add4aace3f38:pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:11:6 marker=00000000002.37.6 timestamp=2023-05-31T20:38:21.641789+0000 2023-05-31T20:38:24.267 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:24.245+0000 7f22572d6780 10 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:2]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:2]:entry[obj00000000000000000011]: success 2023-05-31T20:38:24.267 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:24.245+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:2]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:2]: updating marker marker_oid=bucket.sync-status.3d163b1e-09c7-47e2-ae87-add4aace3f38:pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:2:6 marker=00000000002.52.6 timestamp=2023-05-31T20:38:22.712231+0000 2023-05-31T20:38:24.267 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:24.246+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:8]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:8]: parsed entry: id=00000000001.48.6 iter->object=obj00000000000000000048 iter->instance= name=obj00000000000000000048 instance= ns= 2023-05-31T20:38:24.267 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:24.246+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:8]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:8]: syncing object: pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:8/obj00000000000000000048 2023-05-31T20:38:24.268 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:24.246+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:8]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:8]: entry->timestamp=2023-05-31T20:38:22.489163+0000 2023-05-31T20:38:24.268 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:24.246+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:8]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:8]:entry[obj00000000000000000048]: bucket sync single entry (source_zone=3d163b1e-09c7-47e2-ae87-add4aace3f38) b=bucket sync single entry (source_zone=3d163b1e-09c7-47e2-ae87-add4aace3f38) b= log_entry=00000000001.48.6 op=0 op_state=1 2023-05-31T20:38:24.268 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:24.246+0000 7f22572d6780 5 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:8]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:8]:entry[obj00000000000000000048]: bucket sync: sync obj: 3d163b1e-09c7-47e2-ae87-add4aace3f38/:pklxkd-58[3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23])/obj00000000000000000048[0] 2023-05-31T20:38:24.268 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:24.246+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:8]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:8]: listing bilog for incremental sync; position=00000000001.48.6 2023-05-31T20:38:24.268 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:24.246+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:9]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:9]: parsed entry: id=00000000002.30.6 iter->object=obj00000000000000000015 iter->instance= name=obj00000000000000000015 instance= ns= 2023-05-31T20:38:24.268 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:24.246+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:9]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:9]: syncing object: pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:9/obj00000000000000000015 2023-05-31T20:38:24.269 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:24.246+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:9]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:9]: entry->timestamp=2023-05-31T20:38:22.656305+0000 2023-05-31T20:38:24.269 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:24.246+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:9]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:9]:entry[obj00000000000000000015]: bucket sync single entry (source_zone=3d163b1e-09c7-47e2-ae87-add4aace3f38) b=bucket sync single entry (source_zone=3d163b1e-09c7-47e2-ae87-add4aace3f38) b= log_entry=00000000002.30.6 op=0 op_state=1 2023-05-31T20:38:24.269 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:24.246+0000 7f22572d6780 5 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:9]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:9]:entry[obj00000000000000000015]: bucket sync: sync obj: 3d163b1e-09c7-47e2-ae87-add4aace3f38/:pklxkd-58[3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23])/obj00000000000000000015[0] 2023-05-31T20:38:24.269 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:24.246+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:9]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:9]: listing bilog for incremental sync; position=00000000002.30.6 2023-05-31T20:38:24.269 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:24.248+0000 7f22572d6780 10 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:9]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:9]:entry[obj00000000000000000015]: success 2023-05-31T20:38:24.269 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:24.248+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:9]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:9]: updating marker marker_oid=bucket.sync-status.3d163b1e-09c7-47e2-ae87-add4aace3f38:pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:9:6 marker=00000000002.30.6 timestamp=2023-05-31T20:38:22.656305+0000 2023-05-31T20:38:24.269 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:24.255+0000 7f22572d6780 10 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:0]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:0]:entry[obj00000000000000000060]: success 2023-05-31T20:38:24.270 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:24.255+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:0]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:0]: updating marker marker_oid=bucket.sync-status.3d163b1e-09c7-47e2-ae87-add4aace3f38:pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:0:6 marker=00000000001.58.6 timestamp=2023-05-31T20:38:22.575673+0000 2023-05-31T20:38:24.270 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:24.256+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:30]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:30]: parsed entry: id=00000000001.90.6 iter->object=obj00000000000000000036 iter->instance= name=obj00000000000000000036 instance= ns= 2023-05-31T20:38:24.270 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:24.256+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:30]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:30]: syncing object: pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:30/obj00000000000000000036 2023-05-31T20:38:24.270 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:24.256+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:30]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:30]: entry->timestamp=2023-05-31T20:38:22.636301+0000 2023-05-31T20:38:24.270 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:24.256+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:30]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:30]:entry[obj00000000000000000036]: bucket sync single entry (source_zone=3d163b1e-09c7-47e2-ae87-add4aace3f38) b=bucket sync single entry (source_zone=3d163b1e-09c7-47e2-ae87-add4aace3f38) b= log_entry=00000000001.90.6 op=0 op_state=1 2023-05-31T20:38:24.270 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:24.256+0000 7f22572d6780 5 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:30]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:30]:entry[obj00000000000000000036]: bucket sync: sync obj: 3d163b1e-09c7-47e2-ae87-add4aace3f38/:pklxkd-58[3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23])/obj00000000000000000036[0] 2023-05-31T20:38:24.271 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:24.256+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:30]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:30]: listing bilog for incremental sync; position=00000000001.90.6 2023-05-31T20:38:24.271 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:24.258+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:31]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:31]: parsed entry: id=00000000002.46.6 iter->object=obj00000000000000000003 iter->instance= name=obj00000000000000000003 instance= ns= 2023-05-31T20:38:24.271 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:24.258+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:31]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:31]: syncing object: pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:31/obj00000000000000000003 2023-05-31T20:38:24.271 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:24.258+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:31]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:31]: entry->timestamp=2023-05-31T20:38:21.910544+0000 2023-05-31T20:38:24.271 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:24.258+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:31]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:31]:entry[obj00000000000000000003]: bucket sync single entry (source_zone=3d163b1e-09c7-47e2-ae87-add4aace3f38) b=bucket sync single entry (source_zone=3d163b1e-09c7-47e2-ae87-add4aace3f38) b= log_entry=00000000002.46.6 op=0 op_state=1 2023-05-31T20:38:24.271 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:24.258+0000 7f22572d6780 5 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:31]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:31]:entry[obj00000000000000000003]: bucket sync: sync obj: 3d163b1e-09c7-47e2-ae87-add4aace3f38/:pklxkd-58[3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23])/obj00000000000000000003[0] 2023-05-31T20:38:24.271 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:24.258+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:31]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:31]: listing bilog for incremental sync; position=00000000002.46.6 2023-05-31T20:38:24.272 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:24.259+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:24]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:24]: parsed entry: id=00000000002.55.6 iter->object=obj00000000000000000027 iter->instance= name=obj00000000000000000027 instance= ns= 2023-05-31T20:38:24.272 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:24.259+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:24]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:24]: syncing object: pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:24/obj00000000000000000027 2023-05-31T20:38:24.272 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:24.259+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:24]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:24]: entry->timestamp=2023-05-31T20:38:21.615565+0000 2023-05-31T20:38:24.272 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:24.259+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:24]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:24]:entry[obj00000000000000000027]: bucket sync single entry (source_zone=3d163b1e-09c7-47e2-ae87-add4aace3f38) b=bucket sync single entry (source_zone=3d163b1e-09c7-47e2-ae87-add4aace3f38) b= log_entry=00000000002.55.6 op=0 op_state=1 2023-05-31T20:38:24.272 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:24.259+0000 7f22572d6780 5 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:24]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:24]:entry[obj00000000000000000027]: bucket sync: sync obj: 3d163b1e-09c7-47e2-ae87-add4aace3f38/:pklxkd-58[3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23])/obj00000000000000000027[0] 2023-05-31T20:38:24.272 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:24.259+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:24]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:24]: listing bilog for incremental sync; position=00000000002.55.6 2023-05-31T20:38:24.273 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:24.260+0000 7f22572d6780 10 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:31]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:31]:entry[obj00000000000000000003]: success 2023-05-31T20:38:24.273 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:24.260+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:31]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:31]: updating marker marker_oid=bucket.sync-status.3d163b1e-09c7-47e2-ae87-add4aace3f38:pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:31:6 marker=00000000002.46.6 timestamp=2023-05-31T20:38:21.910544+0000 2023-05-31T20:38:24.274 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:24.260+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:15]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:15]: parsed entry: id=00000000002.61.6 iter->object=obj00000000000000000024 iter->instance= name=obj00000000000000000024 instance= ns= 2023-05-31T20:38:24.275 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:24.260+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:15]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:15]: syncing object: pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:15/obj00000000000000000024 2023-05-31T20:38:24.275 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:24.260+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:15]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:15]: entry->timestamp=2023-05-31T20:38:22.813906+0000 2023-05-31T20:38:24.275 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:24.260+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:15]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:15]:entry[obj00000000000000000024]: bucket sync single entry (source_zone=3d163b1e-09c7-47e2-ae87-add4aace3f38) b=bucket sync single entry (source_zone=3d163b1e-09c7-47e2-ae87-add4aace3f38) b= log_entry=00000000002.61.6 op=0 op_state=1 2023-05-31T20:38:24.275 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:24.260+0000 7f22572d6780 5 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:15]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:15]:entry[obj00000000000000000024]: bucket sync: sync obj: 3d163b1e-09c7-47e2-ae87-add4aace3f38/:pklxkd-58[3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23])/obj00000000000000000024[0] 2023-05-31T20:38:24.275 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:24.260+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:15]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:15]: listing bilog for incremental sync; position=00000000002.61.6 2023-05-31T20:38:24.276 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:24.260+0000 7f22572d6780 10 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:24]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:24]:entry[obj00000000000000000027]: success 2023-05-31T20:38:24.276 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:24.260+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:24]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:24]: updating marker marker_oid=bucket.sync-status.3d163b1e-09c7-47e2-ae87-add4aace3f38:pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:24:6 marker=00000000002.55.6 timestamp=2023-05-31T20:38:21.615565+0000 2023-05-31T20:38:24.276 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:24.261+0000 7f22572d6780 10 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:15]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:15]:entry[obj00000000000000000024]: success 2023-05-31T20:38:24.276 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:24.261+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:15]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:15]: updating marker marker_oid=bucket.sync-status.3d163b1e-09c7-47e2-ae87-add4aace3f38:pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:15:6 marker=00000000002.61.6 timestamp=2023-05-31T20:38:22.813906+0000 2023-05-31T20:38:24.276 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:24.265+0000 7f22572d6780 10 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:35]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:35]: a sync operation returned error 2023-05-31T20:38:24.276 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:24.265+0000 7f22572d6780 10 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:35]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:35]: backing out with sync_status=-5 2023-05-31T20:38:24.277 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:24.265+0000 7f22572d6780 5 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:35]: incremental sync on bucket failed, retcode=-5 2023-05-31T20:38:24.277 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:24.265+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:35]: ERROR: incremental sync failed. error: -5 2023-05-31T20:38:24.277 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:24.265+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:35]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-05-31T20:38:24.277 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:24.265+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:35]: sync status for source bucket shard: 0 2023-05-31T20:38:24.277 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:24.265+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:35]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:35]: listing bilog for incremental sync; position= 2023-05-31T20:38:24.277 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:24.268+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:32]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:32]: parsed entry: id=00000000002.60.6 iter->object=obj00000000000000000014 iter->instance= name=obj00000000000000000014 instance= ns= 2023-05-31T20:38:24.277 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:24.268+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:32]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:32]: syncing object: pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:32/obj00000000000000000014 2023-05-31T20:38:24.278 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:24.268+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:32]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:32]: entry->timestamp=2023-05-31T20:38:22.896668+0000 2023-05-31T20:38:24.278 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:24.268+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:32]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:32]:entry[obj00000000000000000014]: bucket sync single entry (source_zone=3d163b1e-09c7-47e2-ae87-add4aace3f38) b=bucket sync single entry (source_zone=3d163b1e-09c7-47e2-ae87-add4aace3f38) b= log_entry=00000000002.60.6 op=0 op_state=1 2023-05-31T20:38:24.278 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:24.268+0000 7f22572d6780 5 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:32]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:32]:entry[obj00000000000000000014]: bucket sync: sync obj: 3d163b1e-09c7-47e2-ae87-add4aace3f38/:pklxkd-58[3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23])/obj00000000000000000014[0] 2023-05-31T20:38:24.278 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:24.268+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:32]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:32]: listing bilog for incremental sync; position=00000000002.60.6 2023-05-31T20:38:24.278 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:24.268+0000 7f22572d6780 10 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:27]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:27]: a sync operation returned error 2023-05-31T20:38:24.278 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:24.268+0000 7f22572d6780 10 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:27]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:27]: backing out with sync_status=-5 2023-05-31T20:38:24.279 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:24.268+0000 7f22572d6780 5 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:27]: incremental sync on bucket failed, retcode=-5 2023-05-31T20:38:24.279 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:24.268+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:27]: ERROR: incremental sync failed. error: -5 2023-05-31T20:38:24.279 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:24.268+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:27]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-05-31T20:38:24.279 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:24.269+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:27]: sync status for source bucket shard: 0 2023-05-31T20:38:24.280 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:24.269+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:27]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:27]: listing bilog for incremental sync; position= 2023-05-31T20:38:24.280 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:24.269+0000 7f22572d6780 10 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:32]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:32]:entry[obj00000000000000000014]: success 2023-05-31T20:38:24.280 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:24.269+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:32]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:32]: updating marker marker_oid=bucket.sync-status.3d163b1e-09c7-47e2-ae87-add4aace3f38:pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:32:6 marker=00000000002.60.6 timestamp=2023-05-31T20:38:22.896668+0000 2023-05-31T20:38:24.281 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:24.274+0000 7f22572d6780 10 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:34]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:34]:entry[obj00000000000000000057]: success 2023-05-31T20:38:24.281 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:24.274+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:34]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:34]: updating marker marker_oid=bucket.sync-status.3d163b1e-09c7-47e2-ae87-add4aace3f38:pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:34:6 marker=00000000001.56.6 timestamp=2023-05-31T20:38:21.818642+0000 2023-05-31T20:38:24.295 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:24.291+0000 7f22572d6780 10 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:8]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:8]:entry[obj00000000000000000048]: success 2023-05-31T20:38:24.295 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:24.291+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:8]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:8]: updating marker marker_oid=bucket.sync-status.3d163b1e-09c7-47e2-ae87-add4aace3f38:pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:8:6 marker=00000000001.48.6 timestamp=2023-05-31T20:38:22.489163+0000 2023-05-31T20:38:24.305 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:24.301+0000 7f22572d6780 10 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:30]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:30]:entry[obj00000000000000000036]: success 2023-05-31T20:38:24.306 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:24.301+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:30]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:30]: updating marker marker_oid=bucket.sync-status.3d163b1e-09c7-47e2-ae87-add4aace3f38:pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:30:6 marker=00000000001.90.6 timestamp=2023-05-31T20:38:22.636301+0000 2023-05-31T20:38:24.311 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:24.307+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:35]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:35]: parsed entry: id=00000000002.52.6 iter->object=obj00000000000000000018 iter->instance= name=obj00000000000000000018 instance= ns= 2023-05-31T20:38:24.311 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:24.307+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:35]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:35]: syncing object: pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:35/obj00000000000000000018 2023-05-31T20:38:24.311 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:24.307+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:35]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:35]: entry->timestamp=2023-05-31T20:38:21.887881+0000 2023-05-31T20:38:24.312 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:24.307+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:35]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:35]:entry[obj00000000000000000018]: bucket sync single entry (source_zone=3d163b1e-09c7-47e2-ae87-add4aace3f38) b=bucket sync single entry (source_zone=3d163b1e-09c7-47e2-ae87-add4aace3f38) b= log_entry=00000000002.52.6 op=0 op_state=1 2023-05-31T20:38:24.312 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:24.307+0000 7f22572d6780 5 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:35]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:35]:entry[obj00000000000000000018]: bucket sync: sync obj: 3d163b1e-09c7-47e2-ae87-add4aace3f38/:pklxkd-58[3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23])/obj00000000000000000018[0] 2023-05-31T20:38:24.312 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:24.307+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:35]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:35]: listing bilog for incremental sync; position=00000000002.52.6 2023-05-31T20:38:24.313 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:24.308+0000 7f22572d6780 10 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:35]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:35]:entry[obj00000000000000000018]: success 2023-05-31T20:38:24.313 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:24.308+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:35]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:35]: updating marker marker_oid=bucket.sync-status.3d163b1e-09c7-47e2-ae87-add4aace3f38:pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:35:6 marker=00000000002.52.6 timestamp=2023-05-31T20:38:21.887881+0000 2023-05-31T20:38:24.315 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:24.311+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:27]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:27]: parsed entry: id=00000000002.93.6 iter->object=obj00000000000000000053 iter->instance= name=obj00000000000000000053 instance= ns= 2023-05-31T20:38:24.316 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:24.311+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:27]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:27]: syncing object: pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:27/obj00000000000000000053 2023-05-31T20:38:24.316 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:24.311+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:27]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:27]: entry->timestamp=2023-05-31T20:38:22.873398+0000 2023-05-31T20:38:24.316 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:24.311+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:27]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:27]:entry[obj00000000000000000053]: bucket sync single entry (source_zone=3d163b1e-09c7-47e2-ae87-add4aace3f38) b=bucket sync single entry (source_zone=3d163b1e-09c7-47e2-ae87-add4aace3f38) b= log_entry=00000000002.93.6 op=0 op_state=1 2023-05-31T20:38:24.316 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:24.311+0000 7f22572d6780 5 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:27]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:27]:entry[obj00000000000000000053]: bucket sync: sync obj: 3d163b1e-09c7-47e2-ae87-add4aace3f38/:pklxkd-58[3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23])/obj00000000000000000053[0] 2023-05-31T20:38:24.316 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:24.311+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:27]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:27]: listing bilog for incremental sync; position=00000000002.93.6 2023-05-31T20:38:24.360 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:24.356+0000 7f22572d6780 10 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:27]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:27]:entry[obj00000000000000000053]: success 2023-05-31T20:38:24.361 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:24.356+0000 7f22572d6780 20 RGW-SYNC:bucket[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23<-pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:27]:inc_sync[pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:27]: updating marker marker_oid=bucket.sync-status.3d163b1e-09c7-47e2-ae87-add4aace3f38:pklxkd-58:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.23:27:6 marker=00000000002.93.6 timestamp=2023-05-31T20:38:22.873398+0000 2023-05-31T20:38:24.400 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'list', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'pklxkd-58', '--max-entries', '10000', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-31T20:38:24.401 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'list', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'pklxkd-58', '--max-entries', '10000', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-31T20:38:24.402 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket list --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --bucket pklxkd-58 --max-entries 10000 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-05-31T20:38:24.493 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2023-05-31T20:38:24.493 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2023-05-31T20:38:24.554 INFO:teuthology.orchestra.run.smithi099.stdout:[ 2023-05-31T20:38:24.555 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T20:38:24.555 INFO:teuthology.orchestra.run.smithi099.stdout: "name": "obj00000000000000000000", 2023-05-31T20:38:24.555 INFO:teuthology.orchestra.run.smithi099.stdout: "instance": "", 2023-05-31T20:38:24.555 INFO:teuthology.orchestra.run.smithi099.stdout: "ver": { 2023-05-31T20:38:24.556 INFO:teuthology.orchestra.run.smithi099.stdout: "pool": 9, 2023-05-31T20:38:24.556 INFO:teuthology.orchestra.run.smithi099.stdout: "epoch": 19 2023-05-31T20:38:24.556 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:38:24.556 INFO:teuthology.orchestra.run.smithi099.stdout: "locator": "", 2023-05-31T20:38:24.556 INFO:teuthology.orchestra.run.smithi099.stdout: "exists": true, 2023-05-31T20:38:24.556 INFO:teuthology.orchestra.run.smithi099.stdout: "meta": { 2023-05-31T20:38:24.557 INFO:teuthology.orchestra.run.smithi099.stdout: "category": 1, 2023-05-31T20:38:24.557 INFO:teuthology.orchestra.run.smithi099.stdout: "size": 16, 2023-05-31T20:38:24.557 INFO:teuthology.orchestra.run.smithi099.stdout: "mtime": "2023-05-31T20:38:21.972620Z", 2023-05-31T20:38:24.557 INFO:teuthology.orchestra.run.smithi099.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-05-31T20:38:24.557 INFO:teuthology.orchestra.run.smithi099.stdout: "storage_class": "", 2023-05-31T20:38:24.557 INFO:teuthology.orchestra.run.smithi099.stdout: "owner": "rgw-multisite-test-user", 2023-05-31T20:38:24.557 INFO:teuthology.orchestra.run.smithi099.stdout: "owner_display_name": "Multisite Test User", 2023-05-31T20:38:24.558 INFO:teuthology.orchestra.run.smithi099.stdout: "content_type": "application/octet-stream", 2023-05-31T20:38:24.558 INFO:teuthology.orchestra.run.smithi099.stdout: "accounted_size": 3, 2023-05-31T20:38:24.558 INFO:teuthology.orchestra.run.smithi099.stdout: "user_data": "", 2023-05-31T20:38:24.558 INFO:teuthology.orchestra.run.smithi099.stdout: "appendable": false 2023-05-31T20:38:24.558 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:38:24.558 INFO:teuthology.orchestra.run.smithi099.stdout: "tag": "3d163b1e-09c7-47e2-ae87-add4aace3f38.5680.14415224361762380449", 2023-05-31T20:38:24.558 INFO:teuthology.orchestra.run.smithi099.stdout: "flags": 0, 2023-05-31T20:38:24.559 INFO:teuthology.orchestra.run.smithi099.stdout: "pending_map": [], 2023-05-31T20:38:24.559 INFO:teuthology.orchestra.run.smithi099.stdout: "versioned_epoch": 0 2023-05-31T20:38:24.559 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:38:24.559 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T20:38:24.559 INFO:teuthology.orchestra.run.smithi099.stdout: "name": "obj00000000000000000001", 2023-05-31T20:38:24.559 INFO:teuthology.orchestra.run.smithi099.stdout: "instance": "", 2023-05-31T20:38:24.560 INFO:teuthology.orchestra.run.smithi099.stdout: "ver": { 2023-05-31T20:38:24.560 INFO:teuthology.orchestra.run.smithi099.stdout: "pool": 9, 2023-05-31T20:38:24.560 INFO:teuthology.orchestra.run.smithi099.stdout: "epoch": 22 2023-05-31T20:38:24.560 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:38:24.560 INFO:teuthology.orchestra.run.smithi099.stdout: "locator": "", 2023-05-31T20:38:24.560 INFO:teuthology.orchestra.run.smithi099.stdout: "exists": true, 2023-05-31T20:38:24.561 INFO:teuthology.orchestra.run.smithi099.stdout: "meta": { 2023-05-31T20:38:24.561 INFO:teuthology.orchestra.run.smithi099.stdout: "category": 1, 2023-05-31T20:38:24.561 INFO:teuthology.orchestra.run.smithi099.stdout: "size": 16, 2023-05-31T20:38:24.561 INFO:teuthology.orchestra.run.smithi099.stdout: "mtime": "2023-05-31T20:38:22.169017Z", 2023-05-31T20:38:24.561 INFO:teuthology.orchestra.run.smithi099.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-05-31T20:38:24.561 INFO:teuthology.orchestra.run.smithi099.stdout: "storage_class": "", 2023-05-31T20:38:24.561 INFO:teuthology.orchestra.run.smithi099.stdout: "owner": "rgw-multisite-test-user", 2023-05-31T20:38:24.562 INFO:teuthology.orchestra.run.smithi099.stdout: "owner_display_name": "Multisite Test User", 2023-05-31T20:38:24.562 INFO:teuthology.orchestra.run.smithi099.stdout: "content_type": "application/octet-stream", 2023-05-31T20:38:24.562 INFO:teuthology.orchestra.run.smithi099.stdout: "accounted_size": 3, 2023-05-31T20:38:24.562 INFO:teuthology.orchestra.run.smithi099.stdout: "user_data": "", 2023-05-31T20:38:24.562 INFO:teuthology.orchestra.run.smithi099.stdout: "appendable": false 2023-05-31T20:38:24.562 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:38:24.563 INFO:teuthology.orchestra.run.smithi099.stdout: "tag": "3d163b1e-09c7-47e2-ae87-add4aace3f38.5680.12248995050368960364", 2023-05-31T20:38:24.563 INFO:teuthology.orchestra.run.smithi099.stdout: "flags": 0, 2023-05-31T20:38:24.563 INFO:teuthology.orchestra.run.smithi099.stdout: "pending_map": [], 2023-05-31T20:38:24.563 INFO:teuthology.orchestra.run.smithi099.stdout: "versioned_epoch": 0 2023-05-31T20:38:24.563 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:38:24.563 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T20:38:24.563 INFO:teuthology.orchestra.run.smithi099.stdout: "name": "obj00000000000000000002", 2023-05-31T20:38:24.564 INFO:teuthology.orchestra.run.smithi099.stdout: "instance": "", 2023-05-31T20:38:24.564 INFO:teuthology.orchestra.run.smithi099.stdout: "ver": { 2023-05-31T20:38:24.564 INFO:teuthology.orchestra.run.smithi099.stdout: "pool": 9, 2023-05-31T20:38:24.564 INFO:teuthology.orchestra.run.smithi099.stdout: "epoch": 8 2023-05-31T20:38:24.564 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:38:24.564 INFO:teuthology.orchestra.run.smithi099.stdout: "locator": "", 2023-05-31T20:38:24.565 INFO:teuthology.orchestra.run.smithi099.stdout: "exists": true, 2023-05-31T20:38:24.566 INFO:teuthology.orchestra.run.smithi099.stdout: "meta": { 2023-05-31T20:38:24.566 INFO:teuthology.orchestra.run.smithi099.stdout: "category": 1, 2023-05-31T20:38:24.566 INFO:teuthology.orchestra.run.smithi099.stdout: "size": 16, 2023-05-31T20:38:24.566 INFO:teuthology.orchestra.run.smithi099.stdout: "mtime": "2023-05-31T20:38:22.061689Z", 2023-05-31T20:38:24.567 INFO:teuthology.orchestra.run.smithi099.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-05-31T20:38:24.567 INFO:teuthology.orchestra.run.smithi099.stdout: "storage_class": "", 2023-05-31T20:38:24.567 INFO:teuthology.orchestra.run.smithi099.stdout: "owner": "rgw-multisite-test-user", 2023-05-31T20:38:24.567 INFO:teuthology.orchestra.run.smithi099.stdout: "owner_display_name": "Multisite Test User", 2023-05-31T20:38:24.567 INFO:teuthology.orchestra.run.smithi099.stdout: "content_type": "application/octet-stream", 2023-05-31T20:38:24.568 INFO:teuthology.orchestra.run.smithi099.stdout: "accounted_size": 3, 2023-05-31T20:38:24.568 INFO:teuthology.orchestra.run.smithi099.stdout: "user_data": "", 2023-05-31T20:38:24.568 INFO:teuthology.orchestra.run.smithi099.stdout: "appendable": false 2023-05-31T20:38:24.568 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:38:24.568 INFO:teuthology.orchestra.run.smithi099.stdout: "tag": "3d163b1e-09c7-47e2-ae87-add4aace3f38.5680.6556897761485240016", 2023-05-31T20:38:24.568 INFO:teuthology.orchestra.run.smithi099.stdout: "flags": 0, 2023-05-31T20:38:24.569 INFO:teuthology.orchestra.run.smithi099.stdout: "pending_map": [], 2023-05-31T20:38:24.569 INFO:teuthology.orchestra.run.smithi099.stdout: "versioned_epoch": 0 2023-05-31T20:38:24.569 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:38:24.569 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T20:38:24.569 INFO:teuthology.orchestra.run.smithi099.stdout: "name": "obj00000000000000000003", 2023-05-31T20:38:24.570 INFO:teuthology.orchestra.run.smithi099.stdout: "instance": "", 2023-05-31T20:38:24.570 INFO:teuthology.orchestra.run.smithi099.stdout: "ver": { 2023-05-31T20:38:24.570 INFO:teuthology.orchestra.run.smithi099.stdout: "pool": 9, 2023-05-31T20:38:24.570 INFO:teuthology.orchestra.run.smithi099.stdout: "epoch": 11 2023-05-31T20:38:24.570 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:38:24.570 INFO:teuthology.orchestra.run.smithi099.stdout: "locator": "", 2023-05-31T20:38:24.571 INFO:teuthology.orchestra.run.smithi099.stdout: "exists": true, 2023-05-31T20:38:24.571 INFO:teuthology.orchestra.run.smithi099.stdout: "meta": { 2023-05-31T20:38:24.571 INFO:teuthology.orchestra.run.smithi099.stdout: "category": 1, 2023-05-31T20:38:24.571 INFO:teuthology.orchestra.run.smithi099.stdout: "size": 6, 2023-05-31T20:38:24.571 INFO:teuthology.orchestra.run.smithi099.stdout: "mtime": "2023-05-31T20:38:21.910544Z", 2023-05-31T20:38:24.571 INFO:teuthology.orchestra.run.smithi099.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-05-31T20:38:24.572 INFO:teuthology.orchestra.run.smithi099.stdout: "storage_class": "", 2023-05-31T20:38:24.572 INFO:teuthology.orchestra.run.smithi099.stdout: "owner": "rgw-multisite-test-user", 2023-05-31T20:38:24.572 INFO:teuthology.orchestra.run.smithi099.stdout: "owner_display_name": "Multisite Test User", 2023-05-31T20:38:24.572 INFO:teuthology.orchestra.run.smithi099.stdout: "content_type": "application/octet-stream", 2023-05-31T20:38:24.572 INFO:teuthology.orchestra.run.smithi099.stdout: "accounted_size": 3, 2023-05-31T20:38:24.572 INFO:teuthology.orchestra.run.smithi099.stdout: "user_data": "", 2023-05-31T20:38:24.573 INFO:teuthology.orchestra.run.smithi099.stdout: "appendable": false 2023-05-31T20:38:24.573 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:38:24.573 INFO:teuthology.orchestra.run.smithi099.stdout: "tag": "3d163b1e-09c7-47e2-ae87-add4aace3f38.5680.9755147482959258443", 2023-05-31T20:38:24.573 INFO:teuthology.orchestra.run.smithi099.stdout: "flags": 0, 2023-05-31T20:38:24.573 INFO:teuthology.orchestra.run.smithi099.stdout: "pending_map": [], 2023-05-31T20:38:24.573 INFO:teuthology.orchestra.run.smithi099.stdout: "versioned_epoch": 0 2023-05-31T20:38:24.574 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:38:24.574 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T20:38:24.574 INFO:teuthology.orchestra.run.smithi099.stdout: "name": "obj00000000000000000004", 2023-05-31T20:38:24.574 INFO:teuthology.orchestra.run.smithi099.stdout: "instance": "", 2023-05-31T20:38:24.574 INFO:teuthology.orchestra.run.smithi099.stdout: "ver": { 2023-05-31T20:38:24.574 INFO:teuthology.orchestra.run.smithi099.stdout: "pool": 9, 2023-05-31T20:38:24.574 INFO:teuthology.orchestra.run.smithi099.stdout: "epoch": 14 2023-05-31T20:38:24.575 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:38:24.575 INFO:teuthology.orchestra.run.smithi099.stdout: "locator": "", 2023-05-31T20:38:24.575 INFO:teuthology.orchestra.run.smithi099.stdout: "exists": true, 2023-05-31T20:38:24.575 INFO:teuthology.orchestra.run.smithi099.stdout: "meta": { 2023-05-31T20:38:24.575 INFO:teuthology.orchestra.run.smithi099.stdout: "category": 1, 2023-05-31T20:38:24.575 INFO:teuthology.orchestra.run.smithi099.stdout: "size": 6, 2023-05-31T20:38:24.576 INFO:teuthology.orchestra.run.smithi099.stdout: "mtime": "2023-05-31T20:38:22.227799Z", 2023-05-31T20:38:24.576 INFO:teuthology.orchestra.run.smithi099.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-05-31T20:38:24.576 INFO:teuthology.orchestra.run.smithi099.stdout: "storage_class": "", 2023-05-31T20:38:24.576 INFO:teuthology.orchestra.run.smithi099.stdout: "owner": "rgw-multisite-test-user", 2023-05-31T20:38:24.576 INFO:teuthology.orchestra.run.smithi099.stdout: "owner_display_name": "Multisite Test User", 2023-05-31T20:38:24.576 INFO:teuthology.orchestra.run.smithi099.stdout: "content_type": "application/octet-stream", 2023-05-31T20:38:24.577 INFO:teuthology.orchestra.run.smithi099.stdout: "accounted_size": 3, 2023-05-31T20:38:24.577 INFO:teuthology.orchestra.run.smithi099.stdout: "user_data": "", 2023-05-31T20:38:24.577 INFO:teuthology.orchestra.run.smithi099.stdout: "appendable": false 2023-05-31T20:38:24.577 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:38:24.577 INFO:teuthology.orchestra.run.smithi099.stdout: "tag": "3d163b1e-09c7-47e2-ae87-add4aace3f38.5680.6053189454808186981", 2023-05-31T20:38:24.577 INFO:teuthology.orchestra.run.smithi099.stdout: "flags": 0, 2023-05-31T20:38:24.578 INFO:teuthology.orchestra.run.smithi099.stdout: "pending_map": [], 2023-05-31T20:38:24.578 INFO:teuthology.orchestra.run.smithi099.stdout: "versioned_epoch": 0 2023-05-31T20:38:24.578 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:38:24.578 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T20:38:24.578 INFO:teuthology.orchestra.run.smithi099.stdout: "name": "obj00000000000000000005", 2023-05-31T20:38:24.578 INFO:teuthology.orchestra.run.smithi099.stdout: "instance": "", 2023-05-31T20:38:24.579 INFO:teuthology.orchestra.run.smithi099.stdout: "ver": { 2023-05-31T20:38:24.579 INFO:teuthology.orchestra.run.smithi099.stdout: "pool": 9, 2023-05-31T20:38:24.579 INFO:teuthology.orchestra.run.smithi099.stdout: "epoch": 11 2023-05-31T20:38:24.579 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:38:24.579 INFO:teuthology.orchestra.run.smithi099.stdout: "locator": "", 2023-05-31T20:38:24.579 INFO:teuthology.orchestra.run.smithi099.stdout: "exists": true, 2023-05-31T20:38:24.580 INFO:teuthology.orchestra.run.smithi099.stdout: "meta": { 2023-05-31T20:38:24.580 INFO:teuthology.orchestra.run.smithi099.stdout: "category": 1, 2023-05-31T20:38:24.580 INFO:teuthology.orchestra.run.smithi099.stdout: "size": 16, 2023-05-31T20:38:24.580 INFO:teuthology.orchestra.run.smithi099.stdout: "mtime": "2023-05-31T20:38:21.709914Z", 2023-05-31T20:38:24.580 INFO:teuthology.orchestra.run.smithi099.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-05-31T20:38:24.580 INFO:teuthology.orchestra.run.smithi099.stdout: "storage_class": "", 2023-05-31T20:38:24.581 INFO:teuthology.orchestra.run.smithi099.stdout: "owner": "rgw-multisite-test-user", 2023-05-31T20:38:24.581 INFO:teuthology.orchestra.run.smithi099.stdout: "owner_display_name": "Multisite Test User", 2023-05-31T20:38:24.581 INFO:teuthology.orchestra.run.smithi099.stdout: "content_type": "application/octet-stream", 2023-05-31T20:38:24.581 INFO:teuthology.orchestra.run.smithi099.stdout: "accounted_size": 3, 2023-05-31T20:38:24.581 INFO:teuthology.orchestra.run.smithi099.stdout: "user_data": "", 2023-05-31T20:38:24.581 INFO:teuthology.orchestra.run.smithi099.stdout: "appendable": false 2023-05-31T20:38:24.582 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:38:24.582 INFO:teuthology.orchestra.run.smithi099.stdout: "tag": "3d163b1e-09c7-47e2-ae87-add4aace3f38.5680.9731106172436435455", 2023-05-31T20:38:24.582 INFO:teuthology.orchestra.run.smithi099.stdout: "flags": 0, 2023-05-31T20:38:24.582 INFO:teuthology.orchestra.run.smithi099.stdout: "pending_map": [], 2023-05-31T20:38:24.582 INFO:teuthology.orchestra.run.smithi099.stdout: "versioned_epoch": 0 2023-05-31T20:38:24.582 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:38:24.582 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T20:38:24.583 INFO:teuthology.orchestra.run.smithi099.stdout: "name": "obj00000000000000000006", 2023-05-31T20:38:24.583 INFO:teuthology.orchestra.run.smithi099.stdout: "instance": "", 2023-05-31T20:38:24.583 INFO:teuthology.orchestra.run.smithi099.stdout: "ver": { 2023-05-31T20:38:24.583 INFO:teuthology.orchestra.run.smithi099.stdout: "pool": 9, 2023-05-31T20:38:24.583 INFO:teuthology.orchestra.run.smithi099.stdout: "epoch": 14 2023-05-31T20:38:24.583 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:38:24.584 INFO:teuthology.orchestra.run.smithi099.stdout: "locator": "", 2023-05-31T20:38:24.584 INFO:teuthology.orchestra.run.smithi099.stdout: "exists": true, 2023-05-31T20:38:24.584 INFO:teuthology.orchestra.run.smithi099.stdout: "meta": { 2023-05-31T20:38:24.584 INFO:teuthology.orchestra.run.smithi099.stdout: "category": 1, 2023-05-31T20:38:24.584 INFO:teuthology.orchestra.run.smithi099.stdout: "size": 16, 2023-05-31T20:38:24.584 INFO:teuthology.orchestra.run.smithi099.stdout: "mtime": "2023-05-31T20:38:21.735843Z", 2023-05-31T20:38:24.585 INFO:teuthology.orchestra.run.smithi099.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-05-31T20:38:24.585 INFO:teuthology.orchestra.run.smithi099.stdout: "storage_class": "", 2023-05-31T20:38:24.585 INFO:teuthology.orchestra.run.smithi099.stdout: "owner": "rgw-multisite-test-user", 2023-05-31T20:38:24.585 INFO:teuthology.orchestra.run.smithi099.stdout: "owner_display_name": "Multisite Test User", 2023-05-31T20:38:24.585 INFO:teuthology.orchestra.run.smithi099.stdout: "content_type": "application/octet-stream", 2023-05-31T20:38:24.585 INFO:teuthology.orchestra.run.smithi099.stdout: "accounted_size": 3, 2023-05-31T20:38:24.585 INFO:teuthology.orchestra.run.smithi099.stdout: "user_data": "", 2023-05-31T20:38:24.586 INFO:teuthology.orchestra.run.smithi099.stdout: "appendable": false 2023-05-31T20:38:24.586 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:38:24.586 INFO:teuthology.orchestra.run.smithi099.stdout: "tag": "3d163b1e-09c7-47e2-ae87-add4aace3f38.5680.16767671449770998338", 2023-05-31T20:38:24.586 INFO:teuthology.orchestra.run.smithi099.stdout: "flags": 0, 2023-05-31T20:38:24.586 INFO:teuthology.orchestra.run.smithi099.stdout: "pending_map": [], 2023-05-31T20:38:24.587 INFO:teuthology.orchestra.run.smithi099.stdout: "versioned_epoch": 0 2023-05-31T20:38:24.587 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:38:24.587 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T20:38:24.587 INFO:teuthology.orchestra.run.smithi099.stdout: "name": "obj00000000000000000007", 2023-05-31T20:38:24.587 INFO:teuthology.orchestra.run.smithi099.stdout: "instance": "", 2023-05-31T20:38:24.587 INFO:teuthology.orchestra.run.smithi099.stdout: "ver": { 2023-05-31T20:38:24.588 INFO:teuthology.orchestra.run.smithi099.stdout: "pool": 9, 2023-05-31T20:38:24.588 INFO:teuthology.orchestra.run.smithi099.stdout: "epoch": 12 2023-05-31T20:38:24.588 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:38:24.588 INFO:teuthology.orchestra.run.smithi099.stdout: "locator": "", 2023-05-31T20:38:24.588 INFO:teuthology.orchestra.run.smithi099.stdout: "exists": true, 2023-05-31T20:38:24.588 INFO:teuthology.orchestra.run.smithi099.stdout: "meta": { 2023-05-31T20:38:24.589 INFO:teuthology.orchestra.run.smithi099.stdout: "category": 1, 2023-05-31T20:38:24.589 INFO:teuthology.orchestra.run.smithi099.stdout: "size": 16, 2023-05-31T20:38:24.589 INFO:teuthology.orchestra.run.smithi099.stdout: "mtime": "2023-05-31T20:38:22.793692Z", 2023-05-31T20:38:24.589 INFO:teuthology.orchestra.run.smithi099.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-05-31T20:38:24.589 INFO:teuthology.orchestra.run.smithi099.stdout: "storage_class": "", 2023-05-31T20:38:24.589 INFO:teuthology.orchestra.run.smithi099.stdout: "owner": "rgw-multisite-test-user", 2023-05-31T20:38:24.589 INFO:teuthology.orchestra.run.smithi099.stdout: "owner_display_name": "Multisite Test User", 2023-05-31T20:38:24.590 INFO:teuthology.orchestra.run.smithi099.stdout: "content_type": "application/octet-stream", 2023-05-31T20:38:24.590 INFO:teuthology.orchestra.run.smithi099.stdout: "accounted_size": 3, 2023-05-31T20:38:24.590 INFO:teuthology.orchestra.run.smithi099.stdout: "user_data": "", 2023-05-31T20:38:24.590 INFO:teuthology.orchestra.run.smithi099.stdout: "appendable": false 2023-05-31T20:38:24.590 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:38:24.590 INFO:teuthology.orchestra.run.smithi099.stdout: "tag": "3d163b1e-09c7-47e2-ae87-add4aace3f38.5680.3368521166901992746", 2023-05-31T20:38:24.591 INFO:teuthology.orchestra.run.smithi099.stdout: "flags": 0, 2023-05-31T20:38:24.591 INFO:teuthology.orchestra.run.smithi099.stdout: "pending_map": [], 2023-05-31T20:38:24.591 INFO:teuthology.orchestra.run.smithi099.stdout: "versioned_epoch": 0 2023-05-31T20:38:24.591 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:38:24.591 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T20:38:24.591 INFO:teuthology.orchestra.run.smithi099.stdout: "name": "obj00000000000000000008", 2023-05-31T20:38:24.591 INFO:teuthology.orchestra.run.smithi099.stdout: "instance": "", 2023-05-31T20:38:24.592 INFO:teuthology.orchestra.run.smithi099.stdout: "ver": { 2023-05-31T20:38:24.592 INFO:teuthology.orchestra.run.smithi099.stdout: "pool": 9, 2023-05-31T20:38:24.592 INFO:teuthology.orchestra.run.smithi099.stdout: "epoch": 9 2023-05-31T20:38:24.592 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:38:24.592 INFO:teuthology.orchestra.run.smithi099.stdout: "locator": "", 2023-05-31T20:38:24.592 INFO:teuthology.orchestra.run.smithi099.stdout: "exists": true, 2023-05-31T20:38:24.593 INFO:teuthology.orchestra.run.smithi099.stdout: "meta": { 2023-05-31T20:38:24.593 INFO:teuthology.orchestra.run.smithi099.stdout: "category": 1, 2023-05-31T20:38:24.593 INFO:teuthology.orchestra.run.smithi099.stdout: "size": 3, 2023-05-31T20:38:24.593 INFO:teuthology.orchestra.run.smithi099.stdout: "mtime": "2023-05-31T20:38:20.093833Z", 2023-05-31T20:38:24.593 INFO:teuthology.orchestra.run.smithi099.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-05-31T20:38:24.593 INFO:teuthology.orchestra.run.smithi099.stdout: "storage_class": "", 2023-05-31T20:38:24.594 INFO:teuthology.orchestra.run.smithi099.stdout: "owner": "rgw-multisite-test-user", 2023-05-31T20:38:24.594 INFO:teuthology.orchestra.run.smithi099.stdout: "owner_display_name": "Multisite Test User", 2023-05-31T20:38:24.594 INFO:teuthology.orchestra.run.smithi099.stdout: "content_type": "application/octet-stream", 2023-05-31T20:38:24.594 INFO:teuthology.orchestra.run.smithi099.stdout: "accounted_size": 3, 2023-05-31T20:38:24.594 INFO:teuthology.orchestra.run.smithi099.stdout: "user_data": "", 2023-05-31T20:38:24.594 INFO:teuthology.orchestra.run.smithi099.stdout: "appendable": false 2023-05-31T20:38:24.595 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:38:24.595 INFO:teuthology.orchestra.run.smithi099.stdout: "tag": "3d163b1e-09c7-47e2-ae87-add4aace3f38.5680.15576768600920544956", 2023-05-31T20:38:24.595 INFO:teuthology.orchestra.run.smithi099.stdout: "flags": 0, 2023-05-31T20:38:24.595 INFO:teuthology.orchestra.run.smithi099.stdout: "pending_map": [], 2023-05-31T20:38:24.595 INFO:teuthology.orchestra.run.smithi099.stdout: "versioned_epoch": 0 2023-05-31T20:38:24.595 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:38:24.595 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T20:38:24.596 INFO:teuthology.orchestra.run.smithi099.stdout: "name": "obj00000000000000000009", 2023-05-31T20:38:24.596 INFO:teuthology.orchestra.run.smithi099.stdout: "instance": "", 2023-05-31T20:38:24.596 INFO:teuthology.orchestra.run.smithi099.stdout: "ver": { 2023-05-31T20:38:24.596 INFO:teuthology.orchestra.run.smithi099.stdout: "pool": 9, 2023-05-31T20:38:24.596 INFO:teuthology.orchestra.run.smithi099.stdout: "epoch": 12 2023-05-31T20:38:24.596 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:38:24.597 INFO:teuthology.orchestra.run.smithi099.stdout: "locator": "", 2023-05-31T20:38:24.597 INFO:teuthology.orchestra.run.smithi099.stdout: "exists": true, 2023-05-31T20:38:24.597 INFO:teuthology.orchestra.run.smithi099.stdout: "meta": { 2023-05-31T20:38:24.597 INFO:teuthology.orchestra.run.smithi099.stdout: "category": 1, 2023-05-31T20:38:24.597 INFO:teuthology.orchestra.run.smithi099.stdout: "size": 6, 2023-05-31T20:38:24.597 INFO:teuthology.orchestra.run.smithi099.stdout: "mtime": "2023-05-31T20:38:21.588236Z", 2023-05-31T20:38:24.598 INFO:teuthology.orchestra.run.smithi099.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-05-31T20:38:24.598 INFO:teuthology.orchestra.run.smithi099.stdout: "storage_class": "", 2023-05-31T20:38:24.598 INFO:teuthology.orchestra.run.smithi099.stdout: "owner": "rgw-multisite-test-user", 2023-05-31T20:38:24.598 INFO:teuthology.orchestra.run.smithi099.stdout: "owner_display_name": "Multisite Test User", 2023-05-31T20:38:24.598 INFO:teuthology.orchestra.run.smithi099.stdout: "content_type": "application/octet-stream", 2023-05-31T20:38:24.598 INFO:teuthology.orchestra.run.smithi099.stdout: "accounted_size": 3, 2023-05-31T20:38:24.598 INFO:teuthology.orchestra.run.smithi099.stdout: "user_data": "", 2023-05-31T20:38:24.599 INFO:teuthology.orchestra.run.smithi099.stdout: "appendable": false 2023-05-31T20:38:24.599 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:38:24.599 INFO:teuthology.orchestra.run.smithi099.stdout: "tag": "3d163b1e-09c7-47e2-ae87-add4aace3f38.5680.13450420874779816292", 2023-05-31T20:38:24.599 INFO:teuthology.orchestra.run.smithi099.stdout: "flags": 0, 2023-05-31T20:38:24.599 INFO:teuthology.orchestra.run.smithi099.stdout: "pending_map": [], 2023-05-31T20:38:24.599 INFO:teuthology.orchestra.run.smithi099.stdout: "versioned_epoch": 0 2023-05-31T20:38:24.600 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:38:24.600 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T20:38:24.600 INFO:teuthology.orchestra.run.smithi099.stdout: "name": "obj00000000000000000010", 2023-05-31T20:38:24.600 INFO:teuthology.orchestra.run.smithi099.stdout: "instance": "", 2023-05-31T20:38:24.600 INFO:teuthology.orchestra.run.smithi099.stdout: "ver": { 2023-05-31T20:38:24.600 INFO:teuthology.orchestra.run.smithi099.stdout: "pool": 9, 2023-05-31T20:38:24.601 INFO:teuthology.orchestra.run.smithi099.stdout: "epoch": 22 2023-05-31T20:38:24.601 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:38:24.601 INFO:teuthology.orchestra.run.smithi099.stdout: "locator": "", 2023-05-31T20:38:24.601 INFO:teuthology.orchestra.run.smithi099.stdout: "exists": true, 2023-05-31T20:38:24.601 INFO:teuthology.orchestra.run.smithi099.stdout: "meta": { 2023-05-31T20:38:24.601 INFO:teuthology.orchestra.run.smithi099.stdout: "category": 1, 2023-05-31T20:38:24.602 INFO:teuthology.orchestra.run.smithi099.stdout: "size": 3, 2023-05-31T20:38:24.602 INFO:teuthology.orchestra.run.smithi099.stdout: "mtime": "2023-05-31T20:38:21.127706Z", 2023-05-31T20:38:24.602 INFO:teuthology.orchestra.run.smithi099.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-05-31T20:38:24.602 INFO:teuthology.orchestra.run.smithi099.stdout: "storage_class": "", 2023-05-31T20:38:24.602 INFO:teuthology.orchestra.run.smithi099.stdout: "owner": "rgw-multisite-test-user", 2023-05-31T20:38:24.602 INFO:teuthology.orchestra.run.smithi099.stdout: "owner_display_name": "Multisite Test User", 2023-05-31T20:38:24.602 INFO:teuthology.orchestra.run.smithi099.stdout: "content_type": "application/octet-stream", 2023-05-31T20:38:24.603 INFO:teuthology.orchestra.run.smithi099.stdout: "accounted_size": 3, 2023-05-31T20:38:24.603 INFO:teuthology.orchestra.run.smithi099.stdout: "user_data": "", 2023-05-31T20:38:24.603 INFO:teuthology.orchestra.run.smithi099.stdout: "appendable": false 2023-05-31T20:38:24.603 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:38:24.603 INFO:teuthology.orchestra.run.smithi099.stdout: "tag": "3d163b1e-09c7-47e2-ae87-add4aace3f38.5680.9957197414605128694", 2023-05-31T20:38:24.603 INFO:teuthology.orchestra.run.smithi099.stdout: "flags": 0, 2023-05-31T20:38:24.604 INFO:teuthology.orchestra.run.smithi099.stdout: "pending_map": [], 2023-05-31T20:38:24.604 INFO:teuthology.orchestra.run.smithi099.stdout: "versioned_epoch": 0 2023-05-31T20:38:24.604 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:38:24.604 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T20:38:24.604 INFO:teuthology.orchestra.run.smithi099.stdout: "name": "obj00000000000000000011", 2023-05-31T20:38:24.604 INFO:teuthology.orchestra.run.smithi099.stdout: "instance": "", 2023-05-31T20:38:24.605 INFO:teuthology.orchestra.run.smithi099.stdout: "ver": { 2023-05-31T20:38:24.605 INFO:teuthology.orchestra.run.smithi099.stdout: "pool": 9, 2023-05-31T20:38:24.605 INFO:teuthology.orchestra.run.smithi099.stdout: "epoch": 21 2023-05-31T20:38:24.605 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:38:24.605 INFO:teuthology.orchestra.run.smithi099.stdout: "locator": "", 2023-05-31T20:38:24.605 INFO:teuthology.orchestra.run.smithi099.stdout: "exists": true, 2023-05-31T20:38:24.605 INFO:teuthology.orchestra.run.smithi099.stdout: "meta": { 2023-05-31T20:38:24.606 INFO:teuthology.orchestra.run.smithi099.stdout: "category": 1, 2023-05-31T20:38:24.606 INFO:teuthology.orchestra.run.smithi099.stdout: "size": 5, 2023-05-31T20:38:24.606 INFO:teuthology.orchestra.run.smithi099.stdout: "mtime": "2023-05-31T20:38:22.712231Z", 2023-05-31T20:38:24.606 INFO:teuthology.orchestra.run.smithi099.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-05-31T20:38:24.606 INFO:teuthology.orchestra.run.smithi099.stdout: "storage_class": "", 2023-05-31T20:38:24.606 INFO:teuthology.orchestra.run.smithi099.stdout: "owner": "rgw-multisite-test-user", 2023-05-31T20:38:24.607 INFO:teuthology.orchestra.run.smithi099.stdout: "owner_display_name": "Multisite Test User", 2023-05-31T20:38:24.607 INFO:teuthology.orchestra.run.smithi099.stdout: "content_type": "application/octet-stream", 2023-05-31T20:38:24.607 INFO:teuthology.orchestra.run.smithi099.stdout: "accounted_size": 3, 2023-05-31T20:38:24.607 INFO:teuthology.orchestra.run.smithi099.stdout: "user_data": "", 2023-05-31T20:38:24.607 INFO:teuthology.orchestra.run.smithi099.stdout: "appendable": false 2023-05-31T20:38:24.607 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:38:24.608 INFO:teuthology.orchestra.run.smithi099.stdout: "tag": "3d163b1e-09c7-47e2-ae87-add4aace3f38.5680.8438748188726005419", 2023-05-31T20:38:24.608 INFO:teuthology.orchestra.run.smithi099.stdout: "flags": 0, 2023-05-31T20:38:24.608 INFO:teuthology.orchestra.run.smithi099.stdout: "pending_map": [], 2023-05-31T20:38:24.608 INFO:teuthology.orchestra.run.smithi099.stdout: "versioned_epoch": 0 2023-05-31T20:38:24.608 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:38:24.608 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T20:38:24.608 INFO:teuthology.orchestra.run.smithi099.stdout: "name": "obj00000000000000000012", 2023-05-31T20:38:24.609 INFO:teuthology.orchestra.run.smithi099.stdout: "instance": "", 2023-05-31T20:38:24.609 INFO:teuthology.orchestra.run.smithi099.stdout: "ver": { 2023-05-31T20:38:24.609 INFO:teuthology.orchestra.run.smithi099.stdout: "pool": 9, 2023-05-31T20:38:24.609 INFO:teuthology.orchestra.run.smithi099.stdout: "epoch": 24 2023-05-31T20:38:24.609 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:38:24.609 INFO:teuthology.orchestra.run.smithi099.stdout: "locator": "", 2023-05-31T20:38:24.610 INFO:teuthology.orchestra.run.smithi099.stdout: "exists": true, 2023-05-31T20:38:24.610 INFO:teuthology.orchestra.run.smithi099.stdout: "meta": { 2023-05-31T20:38:24.610 INFO:teuthology.orchestra.run.smithi099.stdout: "category": 1, 2023-05-31T20:38:24.610 INFO:teuthology.orchestra.run.smithi099.stdout: "size": 16, 2023-05-31T20:38:24.610 INFO:teuthology.orchestra.run.smithi099.stdout: "mtime": "2023-05-31T20:38:22.308395Z", 2023-05-31T20:38:24.610 INFO:teuthology.orchestra.run.smithi099.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-05-31T20:38:24.611 INFO:teuthology.orchestra.run.smithi099.stdout: "storage_class": "", 2023-05-31T20:38:24.611 INFO:teuthology.orchestra.run.smithi099.stdout: "owner": "rgw-multisite-test-user", 2023-05-31T20:38:24.611 INFO:teuthology.orchestra.run.smithi099.stdout: "owner_display_name": "Multisite Test User", 2023-05-31T20:38:24.611 INFO:teuthology.orchestra.run.smithi099.stdout: "content_type": "application/octet-stream", 2023-05-31T20:38:24.611 INFO:teuthology.orchestra.run.smithi099.stdout: "accounted_size": 3, 2023-05-31T20:38:24.611 INFO:teuthology.orchestra.run.smithi099.stdout: "user_data": "", 2023-05-31T20:38:24.611 INFO:teuthology.orchestra.run.smithi099.stdout: "appendable": false 2023-05-31T20:38:24.612 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:38:24.612 INFO:teuthology.orchestra.run.smithi099.stdout: "tag": "3d163b1e-09c7-47e2-ae87-add4aace3f38.5680.10639350478272262374", 2023-05-31T20:38:24.612 INFO:teuthology.orchestra.run.smithi099.stdout: "flags": 0, 2023-05-31T20:38:24.612 INFO:teuthology.orchestra.run.smithi099.stdout: "pending_map": [], 2023-05-31T20:38:24.612 INFO:teuthology.orchestra.run.smithi099.stdout: "versioned_epoch": 0 2023-05-31T20:38:24.612 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:38:24.613 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T20:38:24.613 INFO:teuthology.orchestra.run.smithi099.stdout: "name": "obj00000000000000000013", 2023-05-31T20:38:24.613 INFO:teuthology.orchestra.run.smithi099.stdout: "instance": "", 2023-05-31T20:38:24.613 INFO:teuthology.orchestra.run.smithi099.stdout: "ver": { 2023-05-31T20:38:24.613 INFO:teuthology.orchestra.run.smithi099.stdout: "pool": 9, 2023-05-31T20:38:24.613 INFO:teuthology.orchestra.run.smithi099.stdout: "epoch": 44 2023-05-31T20:38:24.614 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:38:24.614 INFO:teuthology.orchestra.run.smithi099.stdout: "locator": "", 2023-05-31T20:38:24.614 INFO:teuthology.orchestra.run.smithi099.stdout: "exists": true, 2023-05-31T20:38:24.614 INFO:teuthology.orchestra.run.smithi099.stdout: "meta": { 2023-05-31T20:38:24.614 INFO:teuthology.orchestra.run.smithi099.stdout: "category": 1, 2023-05-31T20:38:24.614 INFO:teuthology.orchestra.run.smithi099.stdout: "size": 3, 2023-05-31T20:38:24.614 INFO:teuthology.orchestra.run.smithi099.stdout: "mtime": "2023-05-31T20:38:22.736493Z", 2023-05-31T20:38:24.615 INFO:teuthology.orchestra.run.smithi099.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-05-31T20:38:24.615 INFO:teuthology.orchestra.run.smithi099.stdout: "storage_class": "", 2023-05-31T20:38:24.615 INFO:teuthology.orchestra.run.smithi099.stdout: "owner": "rgw-multisite-test-user", 2023-05-31T20:38:24.615 INFO:teuthology.orchestra.run.smithi099.stdout: "owner_display_name": "Multisite Test User", 2023-05-31T20:38:24.615 INFO:teuthology.orchestra.run.smithi099.stdout: "content_type": "application/octet-stream", 2023-05-31T20:38:24.615 INFO:teuthology.orchestra.run.smithi099.stdout: "accounted_size": 3, 2023-05-31T20:38:24.616 INFO:teuthology.orchestra.run.smithi099.stdout: "user_data": "", 2023-05-31T20:38:24.616 INFO:teuthology.orchestra.run.smithi099.stdout: "appendable": false 2023-05-31T20:38:24.616 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:38:24.616 INFO:teuthology.orchestra.run.smithi099.stdout: "tag": "3d163b1e-09c7-47e2-ae87-add4aace3f38.5680.2453968737400125311", 2023-05-31T20:38:24.616 INFO:teuthology.orchestra.run.smithi099.stdout: "flags": 0, 2023-05-31T20:38:24.616 INFO:teuthology.orchestra.run.smithi099.stdout: "pending_map": [], 2023-05-31T20:38:24.617 INFO:teuthology.orchestra.run.smithi099.stdout: "versioned_epoch": 0 2023-05-31T20:38:24.617 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:38:24.617 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T20:38:24.617 INFO:teuthology.orchestra.run.smithi099.stdout: "name": "obj00000000000000000014", 2023-05-31T20:38:24.617 INFO:teuthology.orchestra.run.smithi099.stdout: "instance": "", 2023-05-31T20:38:24.617 INFO:teuthology.orchestra.run.smithi099.stdout: "ver": { 2023-05-31T20:38:24.618 INFO:teuthology.orchestra.run.smithi099.stdout: "pool": 9, 2023-05-31T20:38:24.618 INFO:teuthology.orchestra.run.smithi099.stdout: "epoch": 10 2023-05-31T20:38:24.618 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:38:24.618 INFO:teuthology.orchestra.run.smithi099.stdout: "locator": "", 2023-05-31T20:38:24.618 INFO:teuthology.orchestra.run.smithi099.stdout: "exists": true, 2023-05-31T20:38:24.618 INFO:teuthology.orchestra.run.smithi099.stdout: "meta": { 2023-05-31T20:38:24.618 INFO:teuthology.orchestra.run.smithi099.stdout: "category": 1, 2023-05-31T20:38:24.619 INFO:teuthology.orchestra.run.smithi099.stdout: "size": 6, 2023-05-31T20:38:24.619 INFO:teuthology.orchestra.run.smithi099.stdout: "mtime": "2023-05-31T20:38:22.896668Z", 2023-05-31T20:38:24.619 INFO:teuthology.orchestra.run.smithi099.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-05-31T20:38:24.619 INFO:teuthology.orchestra.run.smithi099.stdout: "storage_class": "", 2023-05-31T20:38:24.619 INFO:teuthology.orchestra.run.smithi099.stdout: "owner": "rgw-multisite-test-user", 2023-05-31T20:38:24.619 INFO:teuthology.orchestra.run.smithi099.stdout: "owner_display_name": "Multisite Test User", 2023-05-31T20:38:24.620 INFO:teuthology.orchestra.run.smithi099.stdout: "content_type": "application/octet-stream", 2023-05-31T20:38:24.620 INFO:teuthology.orchestra.run.smithi099.stdout: "accounted_size": 3, 2023-05-31T20:38:24.620 INFO:teuthology.orchestra.run.smithi099.stdout: "user_data": "", 2023-05-31T20:38:24.620 INFO:teuthology.orchestra.run.smithi099.stdout: "appendable": false 2023-05-31T20:38:24.620 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:38:24.620 INFO:teuthology.orchestra.run.smithi099.stdout: "tag": "3d163b1e-09c7-47e2-ae87-add4aace3f38.5680.12608368075608168812", 2023-05-31T20:38:24.621 INFO:teuthology.orchestra.run.smithi099.stdout: "flags": 0, 2023-05-31T20:38:24.621 INFO:teuthology.orchestra.run.smithi099.stdout: "pending_map": [], 2023-05-31T20:38:24.621 INFO:teuthology.orchestra.run.smithi099.stdout: "versioned_epoch": 0 2023-05-31T20:38:24.621 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:38:24.621 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T20:38:24.621 INFO:teuthology.orchestra.run.smithi099.stdout: "name": "obj00000000000000000015", 2023-05-31T20:38:24.622 INFO:teuthology.orchestra.run.smithi099.stdout: "instance": "", 2023-05-31T20:38:24.622 INFO:teuthology.orchestra.run.smithi099.stdout: "ver": { 2023-05-31T20:38:24.622 INFO:teuthology.orchestra.run.smithi099.stdout: "pool": 9, 2023-05-31T20:38:24.622 INFO:teuthology.orchestra.run.smithi099.stdout: "epoch": 8 2023-05-31T20:38:24.622 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:38:24.622 INFO:teuthology.orchestra.run.smithi099.stdout: "locator": "", 2023-05-31T20:38:24.622 INFO:teuthology.orchestra.run.smithi099.stdout: "exists": true, 2023-05-31T20:38:24.623 INFO:teuthology.orchestra.run.smithi099.stdout: "meta": { 2023-05-31T20:38:24.623 INFO:teuthology.orchestra.run.smithi099.stdout: "category": 1, 2023-05-31T20:38:24.623 INFO:teuthology.orchestra.run.smithi099.stdout: "size": 5, 2023-05-31T20:38:24.623 INFO:teuthology.orchestra.run.smithi099.stdout: "mtime": "2023-05-31T20:38:22.656305Z", 2023-05-31T20:38:24.623 INFO:teuthology.orchestra.run.smithi099.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-05-31T20:38:24.623 INFO:teuthology.orchestra.run.smithi099.stdout: "storage_class": "", 2023-05-31T20:38:24.624 INFO:teuthology.orchestra.run.smithi099.stdout: "owner": "rgw-multisite-test-user", 2023-05-31T20:38:24.624 INFO:teuthology.orchestra.run.smithi099.stdout: "owner_display_name": "Multisite Test User", 2023-05-31T20:38:24.624 INFO:teuthology.orchestra.run.smithi099.stdout: "content_type": "application/octet-stream", 2023-05-31T20:38:24.624 INFO:teuthology.orchestra.run.smithi099.stdout: "accounted_size": 3, 2023-05-31T20:38:24.624 INFO:teuthology.orchestra.run.smithi099.stdout: "user_data": "", 2023-05-31T20:38:24.624 INFO:teuthology.orchestra.run.smithi099.stdout: "appendable": false 2023-05-31T20:38:24.625 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:38:24.625 INFO:teuthology.orchestra.run.smithi099.stdout: "tag": "3d163b1e-09c7-47e2-ae87-add4aace3f38.5680.863194296882590642", 2023-05-31T20:38:24.625 INFO:teuthology.orchestra.run.smithi099.stdout: "flags": 0, 2023-05-31T20:38:24.625 INFO:teuthology.orchestra.run.smithi099.stdout: "pending_map": [], 2023-05-31T20:38:24.625 INFO:teuthology.orchestra.run.smithi099.stdout: "versioned_epoch": 0 2023-05-31T20:38:24.625 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:38:24.625 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T20:38:24.626 INFO:teuthology.orchestra.run.smithi099.stdout: "name": "obj00000000000000000016", 2023-05-31T20:38:24.626 INFO:teuthology.orchestra.run.smithi099.stdout: "instance": "", 2023-05-31T20:38:24.626 INFO:teuthology.orchestra.run.smithi099.stdout: "ver": { 2023-05-31T20:38:24.626 INFO:teuthology.orchestra.run.smithi099.stdout: "pool": 9, 2023-05-31T20:38:24.626 INFO:teuthology.orchestra.run.smithi099.stdout: "epoch": 15 2023-05-31T20:38:24.626 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:38:24.627 INFO:teuthology.orchestra.run.smithi099.stdout: "locator": "", 2023-05-31T20:38:24.627 INFO:teuthology.orchestra.run.smithi099.stdout: "exists": true, 2023-05-31T20:38:24.627 INFO:teuthology.orchestra.run.smithi099.stdout: "meta": { 2023-05-31T20:38:24.627 INFO:teuthology.orchestra.run.smithi099.stdout: "category": 1, 2023-05-31T20:38:24.627 INFO:teuthology.orchestra.run.smithi099.stdout: "size": 6, 2023-05-31T20:38:24.627 INFO:teuthology.orchestra.run.smithi099.stdout: "mtime": "2023-05-31T20:38:22.140716Z", 2023-05-31T20:38:24.628 INFO:teuthology.orchestra.run.smithi099.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-05-31T20:38:24.628 INFO:teuthology.orchestra.run.smithi099.stdout: "storage_class": "", 2023-05-31T20:38:24.628 INFO:teuthology.orchestra.run.smithi099.stdout: "owner": "rgw-multisite-test-user", 2023-05-31T20:38:24.628 INFO:teuthology.orchestra.run.smithi099.stdout: "owner_display_name": "Multisite Test User", 2023-05-31T20:38:24.628 INFO:teuthology.orchestra.run.smithi099.stdout: "content_type": "application/octet-stream", 2023-05-31T20:38:24.628 INFO:teuthology.orchestra.run.smithi099.stdout: "accounted_size": 3, 2023-05-31T20:38:24.629 INFO:teuthology.orchestra.run.smithi099.stdout: "user_data": "", 2023-05-31T20:38:24.629 INFO:teuthology.orchestra.run.smithi099.stdout: "appendable": false 2023-05-31T20:38:24.629 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:38:24.629 INFO:teuthology.orchestra.run.smithi099.stdout: "tag": "3d163b1e-09c7-47e2-ae87-add4aace3f38.5680.16276218436402368836", 2023-05-31T20:38:24.629 INFO:teuthology.orchestra.run.smithi099.stdout: "flags": 0, 2023-05-31T20:38:24.629 INFO:teuthology.orchestra.run.smithi099.stdout: "pending_map": [], 2023-05-31T20:38:24.629 INFO:teuthology.orchestra.run.smithi099.stdout: "versioned_epoch": 0 2023-05-31T20:38:24.630 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:38:24.630 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T20:38:24.630 INFO:teuthology.orchestra.run.smithi099.stdout: "name": "obj00000000000000000017", 2023-05-31T20:38:24.630 INFO:teuthology.orchestra.run.smithi099.stdout: "instance": "", 2023-05-31T20:38:24.630 INFO:teuthology.orchestra.run.smithi099.stdout: "ver": { 2023-05-31T20:38:24.630 INFO:teuthology.orchestra.run.smithi099.stdout: "pool": 9, 2023-05-31T20:38:24.631 INFO:teuthology.orchestra.run.smithi099.stdout: "epoch": 13 2023-05-31T20:38:24.631 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:38:24.631 INFO:teuthology.orchestra.run.smithi099.stdout: "locator": "", 2023-05-31T20:38:24.631 INFO:teuthology.orchestra.run.smithi099.stdout: "exists": true, 2023-05-31T20:38:24.631 INFO:teuthology.orchestra.run.smithi099.stdout: "meta": { 2023-05-31T20:38:24.631 INFO:teuthology.orchestra.run.smithi099.stdout: "category": 1, 2023-05-31T20:38:24.631 INFO:teuthology.orchestra.run.smithi099.stdout: "size": 5, 2023-05-31T20:38:24.632 INFO:teuthology.orchestra.run.smithi099.stdout: "mtime": "2023-05-31T20:38:22.548865Z", 2023-05-31T20:38:24.632 INFO:teuthology.orchestra.run.smithi099.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-05-31T20:38:24.632 INFO:teuthology.orchestra.run.smithi099.stdout: "storage_class": "", 2023-05-31T20:38:24.632 INFO:teuthology.orchestra.run.smithi099.stdout: "owner": "rgw-multisite-test-user", 2023-05-31T20:38:24.632 INFO:teuthology.orchestra.run.smithi099.stdout: "owner_display_name": "Multisite Test User", 2023-05-31T20:38:24.632 INFO:teuthology.orchestra.run.smithi099.stdout: "content_type": "application/octet-stream", 2023-05-31T20:38:24.633 INFO:teuthology.orchestra.run.smithi099.stdout: "accounted_size": 3, 2023-05-31T20:38:24.633 INFO:teuthology.orchestra.run.smithi099.stdout: "user_data": "", 2023-05-31T20:38:24.633 INFO:teuthology.orchestra.run.smithi099.stdout: "appendable": false 2023-05-31T20:38:24.633 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:38:24.633 INFO:teuthology.orchestra.run.smithi099.stdout: "tag": "3d163b1e-09c7-47e2-ae87-add4aace3f38.5680.3319239909888435784", 2023-05-31T20:38:24.633 INFO:teuthology.orchestra.run.smithi099.stdout: "flags": 0, 2023-05-31T20:38:24.634 INFO:teuthology.orchestra.run.smithi099.stdout: "pending_map": [], 2023-05-31T20:38:24.634 INFO:teuthology.orchestra.run.smithi099.stdout: "versioned_epoch": 0 2023-05-31T20:38:24.634 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:38:24.634 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T20:38:24.634 INFO:teuthology.orchestra.run.smithi099.stdout: "name": "obj00000000000000000018", 2023-05-31T20:38:24.634 INFO:teuthology.orchestra.run.smithi099.stdout: "instance": "", 2023-05-31T20:38:24.635 INFO:teuthology.orchestra.run.smithi099.stdout: "ver": { 2023-05-31T20:38:24.635 INFO:teuthology.orchestra.run.smithi099.stdout: "pool": 9, 2023-05-31T20:38:24.635 INFO:teuthology.orchestra.run.smithi099.stdout: "epoch": 5 2023-05-31T20:38:24.635 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:38:24.635 INFO:teuthology.orchestra.run.smithi099.stdout: "locator": "", 2023-05-31T20:38:24.635 INFO:teuthology.orchestra.run.smithi099.stdout: "exists": true, 2023-05-31T20:38:24.635 INFO:teuthology.orchestra.run.smithi099.stdout: "meta": { 2023-05-31T20:38:24.636 INFO:teuthology.orchestra.run.smithi099.stdout: "category": 1, 2023-05-31T20:38:24.636 INFO:teuthology.orchestra.run.smithi099.stdout: "size": 6, 2023-05-31T20:38:24.636 INFO:teuthology.orchestra.run.smithi099.stdout: "mtime": "2023-05-31T20:38:21.887881Z", 2023-05-31T20:38:24.636 INFO:teuthology.orchestra.run.smithi099.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-05-31T20:38:24.636 INFO:teuthology.orchestra.run.smithi099.stdout: "storage_class": "", 2023-05-31T20:38:24.637 INFO:teuthology.orchestra.run.smithi099.stdout: "owner": "rgw-multisite-test-user", 2023-05-31T20:38:24.637 INFO:teuthology.orchestra.run.smithi099.stdout: "owner_display_name": "Multisite Test User", 2023-05-31T20:38:24.637 INFO:teuthology.orchestra.run.smithi099.stdout: "content_type": "application/octet-stream", 2023-05-31T20:38:24.637 INFO:teuthology.orchestra.run.smithi099.stdout: "accounted_size": 3, 2023-05-31T20:38:24.637 INFO:teuthology.orchestra.run.smithi099.stdout: "user_data": "", 2023-05-31T20:38:24.637 INFO:teuthology.orchestra.run.smithi099.stdout: "appendable": false 2023-05-31T20:38:24.637 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:38:24.638 INFO:teuthology.orchestra.run.smithi099.stdout: "tag": "3d163b1e-09c7-47e2-ae87-add4aace3f38.5680.9901821289310532734", 2023-05-31T20:38:24.638 INFO:teuthology.orchestra.run.smithi099.stdout: "flags": 0, 2023-05-31T20:38:24.638 INFO:teuthology.orchestra.run.smithi099.stdout: "pending_map": [], 2023-05-31T20:38:24.638 INFO:teuthology.orchestra.run.smithi099.stdout: "versioned_epoch": 0 2023-05-31T20:38:24.638 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:38:24.638 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T20:38:24.639 INFO:teuthology.orchestra.run.smithi099.stdout: "name": "obj00000000000000000019", 2023-05-31T20:38:24.639 INFO:teuthology.orchestra.run.smithi099.stdout: "instance": "", 2023-05-31T20:38:24.639 INFO:teuthology.orchestra.run.smithi099.stdout: "ver": { 2023-05-31T20:38:24.639 INFO:teuthology.orchestra.run.smithi099.stdout: "pool": 9, 2023-05-31T20:38:24.639 INFO:teuthology.orchestra.run.smithi099.stdout: "epoch": 8 2023-05-31T20:38:24.639 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:38:24.640 INFO:teuthology.orchestra.run.smithi099.stdout: "locator": "", 2023-05-31T20:38:24.640 INFO:teuthology.orchestra.run.smithi099.stdout: "exists": true, 2023-05-31T20:38:24.640 INFO:teuthology.orchestra.run.smithi099.stdout: "meta": { 2023-05-31T20:38:24.640 INFO:teuthology.orchestra.run.smithi099.stdout: "category": 1, 2023-05-31T20:38:24.640 INFO:teuthology.orchestra.run.smithi099.stdout: "size": 16, 2023-05-31T20:38:24.640 INFO:teuthology.orchestra.run.smithi099.stdout: "mtime": "2023-05-31T20:38:21.559227Z", 2023-05-31T20:38:24.641 INFO:teuthology.orchestra.run.smithi099.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-05-31T20:38:24.641 INFO:teuthology.orchestra.run.smithi099.stdout: "storage_class": "", 2023-05-31T20:38:24.641 INFO:teuthology.orchestra.run.smithi099.stdout: "owner": "rgw-multisite-test-user", 2023-05-31T20:38:24.641 INFO:teuthology.orchestra.run.smithi099.stdout: "owner_display_name": "Multisite Test User", 2023-05-31T20:38:24.641 INFO:teuthology.orchestra.run.smithi099.stdout: "content_type": "application/octet-stream", 2023-05-31T20:38:24.641 INFO:teuthology.orchestra.run.smithi099.stdout: "accounted_size": 3, 2023-05-31T20:38:24.642 INFO:teuthology.orchestra.run.smithi099.stdout: "user_data": "", 2023-05-31T20:38:24.642 INFO:teuthology.orchestra.run.smithi099.stdout: "appendable": false 2023-05-31T20:38:24.642 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:38:24.642 INFO:teuthology.orchestra.run.smithi099.stdout: "tag": "3d163b1e-09c7-47e2-ae87-add4aace3f38.5680.17266239250253378711", 2023-05-31T20:38:24.642 INFO:teuthology.orchestra.run.smithi099.stdout: "flags": 0, 2023-05-31T20:38:24.642 INFO:teuthology.orchestra.run.smithi099.stdout: "pending_map": [], 2023-05-31T20:38:24.642 INFO:teuthology.orchestra.run.smithi099.stdout: "versioned_epoch": 0 2023-05-31T20:38:24.643 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:38:24.643 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T20:38:24.643 INFO:teuthology.orchestra.run.smithi099.stdout: "name": "obj00000000000000000020", 2023-05-31T20:38:24.643 INFO:teuthology.orchestra.run.smithi099.stdout: "instance": "", 2023-05-31T20:38:24.643 INFO:teuthology.orchestra.run.smithi099.stdout: "ver": { 2023-05-31T20:38:24.643 INFO:teuthology.orchestra.run.smithi099.stdout: "pool": 9, 2023-05-31T20:38:24.644 INFO:teuthology.orchestra.run.smithi099.stdout: "epoch": 11 2023-05-31T20:38:24.644 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:38:24.644 INFO:teuthology.orchestra.run.smithi099.stdout: "locator": "", 2023-05-31T20:38:24.644 INFO:teuthology.orchestra.run.smithi099.stdout: "exists": true, 2023-05-31T20:38:24.644 INFO:teuthology.orchestra.run.smithi099.stdout: "meta": { 2023-05-31T20:38:24.645 INFO:teuthology.orchestra.run.smithi099.stdout: "category": 1, 2023-05-31T20:38:24.645 INFO:teuthology.orchestra.run.smithi099.stdout: "size": 5, 2023-05-31T20:38:24.645 INFO:teuthology.orchestra.run.smithi099.stdout: "mtime": "2023-05-31T20:38:22.465273Z", 2023-05-31T20:38:24.645 INFO:teuthology.orchestra.run.smithi099.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-05-31T20:38:24.645 INFO:teuthology.orchestra.run.smithi099.stdout: "storage_class": "", 2023-05-31T20:38:24.646 INFO:teuthology.orchestra.run.smithi099.stdout: "owner": "rgw-multisite-test-user", 2023-05-31T20:38:24.646 INFO:teuthology.orchestra.run.smithi099.stdout: "owner_display_name": "Multisite Test User", 2023-05-31T20:38:24.646 INFO:teuthology.orchestra.run.smithi099.stdout: "content_type": "application/octet-stream", 2023-05-31T20:38:24.646 INFO:teuthology.orchestra.run.smithi099.stdout: "accounted_size": 3, 2023-05-31T20:38:24.646 INFO:teuthology.orchestra.run.smithi099.stdout: "user_data": "", 2023-05-31T20:38:24.646 INFO:teuthology.orchestra.run.smithi099.stdout: "appendable": false 2023-05-31T20:38:24.647 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:38:24.647 INFO:teuthology.orchestra.run.smithi099.stdout: "tag": "3d163b1e-09c7-47e2-ae87-add4aace3f38.5680.2561571921242342248", 2023-05-31T20:38:24.647 INFO:teuthology.orchestra.run.smithi099.stdout: "flags": 0, 2023-05-31T20:38:24.647 INFO:teuthology.orchestra.run.smithi099.stdout: "pending_map": [], 2023-05-31T20:38:24.647 INFO:teuthology.orchestra.run.smithi099.stdout: "versioned_epoch": 0 2023-05-31T20:38:24.647 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:38:24.648 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T20:38:24.648 INFO:teuthology.orchestra.run.smithi099.stdout: "name": "obj00000000000000000021", 2023-05-31T20:38:24.648 INFO:teuthology.orchestra.run.smithi099.stdout: "instance": "", 2023-05-31T20:38:24.648 INFO:teuthology.orchestra.run.smithi099.stdout: "ver": { 2023-05-31T20:38:24.648 INFO:teuthology.orchestra.run.smithi099.stdout: "pool": 9, 2023-05-31T20:38:24.648 INFO:teuthology.orchestra.run.smithi099.stdout: "epoch": 12 2023-05-31T20:38:24.648 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:38:24.649 INFO:teuthology.orchestra.run.smithi099.stdout: "locator": "", 2023-05-31T20:38:24.649 INFO:teuthology.orchestra.run.smithi099.stdout: "exists": true, 2023-05-31T20:38:24.649 INFO:teuthology.orchestra.run.smithi099.stdout: "meta": { 2023-05-31T20:38:24.649 INFO:teuthology.orchestra.run.smithi099.stdout: "category": 1, 2023-05-31T20:38:24.649 INFO:teuthology.orchestra.run.smithi099.stdout: "size": 5, 2023-05-31T20:38:24.649 INFO:teuthology.orchestra.run.smithi099.stdout: "mtime": "2023-05-31T20:38:20.802133Z", 2023-05-31T20:38:24.650 INFO:teuthology.orchestra.run.smithi099.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-05-31T20:38:24.650 INFO:teuthology.orchestra.run.smithi099.stdout: "storage_class": "", 2023-05-31T20:38:24.650 INFO:teuthology.orchestra.run.smithi099.stdout: "owner": "rgw-multisite-test-user", 2023-05-31T20:38:24.650 INFO:teuthology.orchestra.run.smithi099.stdout: "owner_display_name": "Multisite Test User", 2023-05-31T20:38:24.650 INFO:teuthology.orchestra.run.smithi099.stdout: "content_type": "application/octet-stream", 2023-05-31T20:38:24.650 INFO:teuthology.orchestra.run.smithi099.stdout: "accounted_size": 3, 2023-05-31T20:38:24.650 INFO:teuthology.orchestra.run.smithi099.stdout: "user_data": "", 2023-05-31T20:38:24.651 INFO:teuthology.orchestra.run.smithi099.stdout: "appendable": false 2023-05-31T20:38:24.651 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:38:24.651 INFO:teuthology.orchestra.run.smithi099.stdout: "tag": "3d163b1e-09c7-47e2-ae87-add4aace3f38.5680.15660670522346703255", 2023-05-31T20:38:24.651 INFO:teuthology.orchestra.run.smithi099.stdout: "flags": 0, 2023-05-31T20:38:24.651 INFO:teuthology.orchestra.run.smithi099.stdout: "pending_map": [], 2023-05-31T20:38:24.651 INFO:teuthology.orchestra.run.smithi099.stdout: "versioned_epoch": 0 2023-05-31T20:38:24.652 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:38:24.652 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T20:38:24.652 INFO:teuthology.orchestra.run.smithi099.stdout: "name": "obj00000000000000000022", 2023-05-31T20:38:24.652 INFO:teuthology.orchestra.run.smithi099.stdout: "instance": "", 2023-05-31T20:38:24.652 INFO:teuthology.orchestra.run.smithi099.stdout: "ver": { 2023-05-31T20:38:24.652 INFO:teuthology.orchestra.run.smithi099.stdout: "pool": 9, 2023-05-31T20:38:24.652 INFO:teuthology.orchestra.run.smithi099.stdout: "epoch": 3 2023-05-31T20:38:24.653 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:38:24.653 INFO:teuthology.orchestra.run.smithi099.stdout: "locator": "", 2023-05-31T20:38:24.653 INFO:teuthology.orchestra.run.smithi099.stdout: "exists": true, 2023-05-31T20:38:24.653 INFO:teuthology.orchestra.run.smithi099.stdout: "meta": { 2023-05-31T20:38:24.653 INFO:teuthology.orchestra.run.smithi099.stdout: "category": 1, 2023-05-31T20:38:24.653 INFO:teuthology.orchestra.run.smithi099.stdout: "size": 6, 2023-05-31T20:38:24.653 INFO:teuthology.orchestra.run.smithi099.stdout: "mtime": "2023-05-31T20:38:20.598399Z", 2023-05-31T20:38:24.654 INFO:teuthology.orchestra.run.smithi099.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-05-31T20:38:24.654 INFO:teuthology.orchestra.run.smithi099.stdout: "storage_class": "", 2023-05-31T20:38:24.654 INFO:teuthology.orchestra.run.smithi099.stdout: "owner": "rgw-multisite-test-user", 2023-05-31T20:38:24.654 INFO:teuthology.orchestra.run.smithi099.stdout: "owner_display_name": "Multisite Test User", 2023-05-31T20:38:24.654 INFO:teuthology.orchestra.run.smithi099.stdout: "content_type": "application/octet-stream", 2023-05-31T20:38:24.654 INFO:teuthology.orchestra.run.smithi099.stdout: "accounted_size": 3, 2023-05-31T20:38:24.654 INFO:teuthology.orchestra.run.smithi099.stdout: "user_data": "", 2023-05-31T20:38:24.655 INFO:teuthology.orchestra.run.smithi099.stdout: "appendable": false 2023-05-31T20:38:24.655 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:38:24.655 INFO:teuthology.orchestra.run.smithi099.stdout: "tag": "3d163b1e-09c7-47e2-ae87-add4aace3f38.5680.12754087105405489338", 2023-05-31T20:38:24.655 INFO:teuthology.orchestra.run.smithi099.stdout: "flags": 0, 2023-05-31T20:38:24.655 INFO:teuthology.orchestra.run.smithi099.stdout: "pending_map": [], 2023-05-31T20:38:24.655 INFO:teuthology.orchestra.run.smithi099.stdout: "versioned_epoch": 0 2023-05-31T20:38:24.655 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:38:24.656 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T20:38:24.656 INFO:teuthology.orchestra.run.smithi099.stdout: "name": "obj00000000000000000023", 2023-05-31T20:38:24.656 INFO:teuthology.orchestra.run.smithi099.stdout: "instance": "", 2023-05-31T20:38:24.656 INFO:teuthology.orchestra.run.smithi099.stdout: "ver": { 2023-05-31T20:38:24.656 INFO:teuthology.orchestra.run.smithi099.stdout: "pool": 9, 2023-05-31T20:38:24.656 INFO:teuthology.orchestra.run.smithi099.stdout: "epoch": 4 2023-05-31T20:38:24.657 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:38:24.657 INFO:teuthology.orchestra.run.smithi099.stdout: "locator": "", 2023-05-31T20:38:24.657 INFO:teuthology.orchestra.run.smithi099.stdout: "exists": true, 2023-05-31T20:38:24.657 INFO:teuthology.orchestra.run.smithi099.stdout: "meta": { 2023-05-31T20:38:24.657 INFO:teuthology.orchestra.run.smithi099.stdout: "category": 1, 2023-05-31T20:38:24.657 INFO:teuthology.orchestra.run.smithi099.stdout: "size": 3, 2023-05-31T20:38:24.657 INFO:teuthology.orchestra.run.smithi099.stdout: "mtime": "2023-05-31T20:38:20.347602Z", 2023-05-31T20:38:24.658 INFO:teuthology.orchestra.run.smithi099.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-05-31T20:38:24.658 INFO:teuthology.orchestra.run.smithi099.stdout: "storage_class": "", 2023-05-31T20:38:24.658 INFO:teuthology.orchestra.run.smithi099.stdout: "owner": "rgw-multisite-test-user", 2023-05-31T20:38:24.658 INFO:teuthology.orchestra.run.smithi099.stdout: "owner_display_name": "Multisite Test User", 2023-05-31T20:38:24.658 INFO:teuthology.orchestra.run.smithi099.stdout: "content_type": "application/octet-stream", 2023-05-31T20:38:24.658 INFO:teuthology.orchestra.run.smithi099.stdout: "accounted_size": 3, 2023-05-31T20:38:24.659 INFO:teuthology.orchestra.run.smithi099.stdout: "user_data": "", 2023-05-31T20:38:24.659 INFO:teuthology.orchestra.run.smithi099.stdout: "appendable": false 2023-05-31T20:38:24.659 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:38:24.659 INFO:teuthology.orchestra.run.smithi099.stdout: "tag": "3d163b1e-09c7-47e2-ae87-add4aace3f38.5680.4685770937887181425", 2023-05-31T20:38:24.659 INFO:teuthology.orchestra.run.smithi099.stdout: "flags": 0, 2023-05-31T20:38:24.659 INFO:teuthology.orchestra.run.smithi099.stdout: "pending_map": [], 2023-05-31T20:38:24.659 INFO:teuthology.orchestra.run.smithi099.stdout: "versioned_epoch": 0 2023-05-31T20:38:24.660 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:38:24.660 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T20:38:24.660 INFO:teuthology.orchestra.run.smithi099.stdout: "name": "obj00000000000000000024", 2023-05-31T20:38:24.660 INFO:teuthology.orchestra.run.smithi099.stdout: "instance": "", 2023-05-31T20:38:24.660 INFO:teuthology.orchestra.run.smithi099.stdout: "ver": { 2023-05-31T20:38:24.660 INFO:teuthology.orchestra.run.smithi099.stdout: "pool": 9, 2023-05-31T20:38:24.661 INFO:teuthology.orchestra.run.smithi099.stdout: "epoch": 9 2023-05-31T20:38:24.661 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:38:24.661 INFO:teuthology.orchestra.run.smithi099.stdout: "locator": "", 2023-05-31T20:38:24.661 INFO:teuthology.orchestra.run.smithi099.stdout: "exists": true, 2023-05-31T20:38:24.661 INFO:teuthology.orchestra.run.smithi099.stdout: "meta": { 2023-05-31T20:38:24.661 INFO:teuthology.orchestra.run.smithi099.stdout: "category": 1, 2023-05-31T20:38:24.661 INFO:teuthology.orchestra.run.smithi099.stdout: "size": 16, 2023-05-31T20:38:24.662 INFO:teuthology.orchestra.run.smithi099.stdout: "mtime": "2023-05-31T20:38:22.813906Z", 2023-05-31T20:38:24.662 INFO:teuthology.orchestra.run.smithi099.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-05-31T20:38:24.662 INFO:teuthology.orchestra.run.smithi099.stdout: "storage_class": "", 2023-05-31T20:38:24.662 INFO:teuthology.orchestra.run.smithi099.stdout: "owner": "rgw-multisite-test-user", 2023-05-31T20:38:24.662 INFO:teuthology.orchestra.run.smithi099.stdout: "owner_display_name": "Multisite Test User", 2023-05-31T20:38:24.663 INFO:teuthology.orchestra.run.smithi099.stdout: "content_type": "application/octet-stream", 2023-05-31T20:38:24.663 INFO:teuthology.orchestra.run.smithi099.stdout: "accounted_size": 3, 2023-05-31T20:38:24.663 INFO:teuthology.orchestra.run.smithi099.stdout: "user_data": "", 2023-05-31T20:38:24.663 INFO:teuthology.orchestra.run.smithi099.stdout: "appendable": false 2023-05-31T20:38:24.663 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:38:24.663 INFO:teuthology.orchestra.run.smithi099.stdout: "tag": "3d163b1e-09c7-47e2-ae87-add4aace3f38.5680.17443398511324725879", 2023-05-31T20:38:24.664 INFO:teuthology.orchestra.run.smithi099.stdout: "flags": 0, 2023-05-31T20:38:24.664 INFO:teuthology.orchestra.run.smithi099.stdout: "pending_map": [], 2023-05-31T20:38:24.664 INFO:teuthology.orchestra.run.smithi099.stdout: "versioned_epoch": 0 2023-05-31T20:38:24.664 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:38:24.664 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T20:38:24.664 INFO:teuthology.orchestra.run.smithi099.stdout: "name": "obj00000000000000000025", 2023-05-31T20:38:24.665 INFO:teuthology.orchestra.run.smithi099.stdout: "instance": "", 2023-05-31T20:38:24.665 INFO:teuthology.orchestra.run.smithi099.stdout: "ver": { 2023-05-31T20:38:24.665 INFO:teuthology.orchestra.run.smithi099.stdout: "pool": 9, 2023-05-31T20:38:24.665 INFO:teuthology.orchestra.run.smithi099.stdout: "epoch": 11 2023-05-31T20:38:24.665 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:38:24.665 INFO:teuthology.orchestra.run.smithi099.stdout: "locator": "", 2023-05-31T20:38:24.665 INFO:teuthology.orchestra.run.smithi099.stdout: "exists": true, 2023-05-31T20:38:24.666 INFO:teuthology.orchestra.run.smithi099.stdout: "meta": { 2023-05-31T20:38:24.666 INFO:teuthology.orchestra.run.smithi099.stdout: "category": 1, 2023-05-31T20:38:24.666 INFO:teuthology.orchestra.run.smithi099.stdout: "size": 3, 2023-05-31T20:38:24.666 INFO:teuthology.orchestra.run.smithi099.stdout: "mtime": "2023-05-31T20:38:19.001294Z", 2023-05-31T20:38:24.666 INFO:teuthology.orchestra.run.smithi099.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-05-31T20:38:24.666 INFO:teuthology.orchestra.run.smithi099.stdout: "storage_class": "", 2023-05-31T20:38:24.666 INFO:teuthology.orchestra.run.smithi099.stdout: "owner": "rgw-multisite-test-user", 2023-05-31T20:38:24.667 INFO:teuthology.orchestra.run.smithi099.stdout: "owner_display_name": "Multisite Test User", 2023-05-31T20:38:24.667 INFO:teuthology.orchestra.run.smithi099.stdout: "content_type": "application/octet-stream", 2023-05-31T20:38:24.667 INFO:teuthology.orchestra.run.smithi099.stdout: "accounted_size": 3, 2023-05-31T20:38:24.667 INFO:teuthology.orchestra.run.smithi099.stdout: "user_data": "", 2023-05-31T20:38:24.667 INFO:teuthology.orchestra.run.smithi099.stdout: "appendable": false 2023-05-31T20:38:24.667 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:38:24.668 INFO:teuthology.orchestra.run.smithi099.stdout: "tag": "3d163b1e-09c7-47e2-ae87-add4aace3f38.5680.17846516009124192925", 2023-05-31T20:38:24.668 INFO:teuthology.orchestra.run.smithi099.stdout: "flags": 0, 2023-05-31T20:38:24.668 INFO:teuthology.orchestra.run.smithi099.stdout: "pending_map": [], 2023-05-31T20:38:24.668 INFO:teuthology.orchestra.run.smithi099.stdout: "versioned_epoch": 0 2023-05-31T20:38:24.668 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:38:24.668 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T20:38:24.668 INFO:teuthology.orchestra.run.smithi099.stdout: "name": "obj00000000000000000026", 2023-05-31T20:38:24.669 INFO:teuthology.orchestra.run.smithi099.stdout: "instance": "", 2023-05-31T20:38:24.669 INFO:teuthology.orchestra.run.smithi099.stdout: "ver": { 2023-05-31T20:38:24.669 INFO:teuthology.orchestra.run.smithi099.stdout: "pool": 9, 2023-05-31T20:38:24.669 INFO:teuthology.orchestra.run.smithi099.stdout: "epoch": 24 2023-05-31T20:38:24.669 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:38:24.670 INFO:teuthology.orchestra.run.smithi099.stdout: "locator": "", 2023-05-31T20:38:24.670 INFO:teuthology.orchestra.run.smithi099.stdout: "exists": true, 2023-05-31T20:38:24.670 INFO:teuthology.orchestra.run.smithi099.stdout: "meta": { 2023-05-31T20:38:24.670 INFO:teuthology.orchestra.run.smithi099.stdout: "category": 1, 2023-05-31T20:38:24.670 INFO:teuthology.orchestra.run.smithi099.stdout: "size": 16, 2023-05-31T20:38:24.670 INFO:teuthology.orchestra.run.smithi099.stdout: "mtime": "2023-05-31T20:38:22.407332Z", 2023-05-31T20:38:24.670 INFO:teuthology.orchestra.run.smithi099.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-05-31T20:38:24.671 INFO:teuthology.orchestra.run.smithi099.stdout: "storage_class": "", 2023-05-31T20:38:24.671 INFO:teuthology.orchestra.run.smithi099.stdout: "owner": "rgw-multisite-test-user", 2023-05-31T20:38:24.671 INFO:teuthology.orchestra.run.smithi099.stdout: "owner_display_name": "Multisite Test User", 2023-05-31T20:38:24.671 INFO:teuthology.orchestra.run.smithi099.stdout: "content_type": "application/octet-stream", 2023-05-31T20:38:24.671 INFO:teuthology.orchestra.run.smithi099.stdout: "accounted_size": 3, 2023-05-31T20:38:24.671 INFO:teuthology.orchestra.run.smithi099.stdout: "user_data": "", 2023-05-31T20:38:24.672 INFO:teuthology.orchestra.run.smithi099.stdout: "appendable": false 2023-05-31T20:38:24.672 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:38:24.672 INFO:teuthology.orchestra.run.smithi099.stdout: "tag": "3d163b1e-09c7-47e2-ae87-add4aace3f38.5680.11623512290259504406", 2023-05-31T20:38:24.672 INFO:teuthology.orchestra.run.smithi099.stdout: "flags": 0, 2023-05-31T20:38:24.672 INFO:teuthology.orchestra.run.smithi099.stdout: "pending_map": [], 2023-05-31T20:38:24.672 INFO:teuthology.orchestra.run.smithi099.stdout: "versioned_epoch": 0 2023-05-31T20:38:24.673 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:38:24.673 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T20:38:24.673 INFO:teuthology.orchestra.run.smithi099.stdout: "name": "obj00000000000000000027", 2023-05-31T20:38:24.673 INFO:teuthology.orchestra.run.smithi099.stdout: "instance": "", 2023-05-31T20:38:24.673 INFO:teuthology.orchestra.run.smithi099.stdout: "ver": { 2023-05-31T20:38:24.673 INFO:teuthology.orchestra.run.smithi099.stdout: "pool": 9, 2023-05-31T20:38:24.673 INFO:teuthology.orchestra.run.smithi099.stdout: "epoch": 7 2023-05-31T20:38:24.674 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:38:24.674 INFO:teuthology.orchestra.run.smithi099.stdout: "locator": "", 2023-05-31T20:38:24.674 INFO:teuthology.orchestra.run.smithi099.stdout: "exists": true, 2023-05-31T20:38:24.674 INFO:teuthology.orchestra.run.smithi099.stdout: "meta": { 2023-05-31T20:38:24.674 INFO:teuthology.orchestra.run.smithi099.stdout: "category": 1, 2023-05-31T20:38:24.674 INFO:teuthology.orchestra.run.smithi099.stdout: "size": 6, 2023-05-31T20:38:24.675 INFO:teuthology.orchestra.run.smithi099.stdout: "mtime": "2023-05-31T20:38:21.615565Z", 2023-05-31T20:38:24.675 INFO:teuthology.orchestra.run.smithi099.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-05-31T20:38:24.675 INFO:teuthology.orchestra.run.smithi099.stdout: "storage_class": "", 2023-05-31T20:38:24.675 INFO:teuthology.orchestra.run.smithi099.stdout: "owner": "rgw-multisite-test-user", 2023-05-31T20:38:24.675 INFO:teuthology.orchestra.run.smithi099.stdout: "owner_display_name": "Multisite Test User", 2023-05-31T20:38:24.675 INFO:teuthology.orchestra.run.smithi099.stdout: "content_type": "application/octet-stream", 2023-05-31T20:38:24.675 INFO:teuthology.orchestra.run.smithi099.stdout: "accounted_size": 3, 2023-05-31T20:38:24.676 INFO:teuthology.orchestra.run.smithi099.stdout: "user_data": "", 2023-05-31T20:38:24.676 INFO:teuthology.orchestra.run.smithi099.stdout: "appendable": false 2023-05-31T20:38:24.676 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:38:24.676 INFO:teuthology.orchestra.run.smithi099.stdout: "tag": "3d163b1e-09c7-47e2-ae87-add4aace3f38.5680.6487730248693108311", 2023-05-31T20:38:24.676 INFO:teuthology.orchestra.run.smithi099.stdout: "flags": 0, 2023-05-31T20:38:24.676 INFO:teuthology.orchestra.run.smithi099.stdout: "pending_map": [], 2023-05-31T20:38:24.676 INFO:teuthology.orchestra.run.smithi099.stdout: "versioned_epoch": 0 2023-05-31T20:38:24.677 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:38:24.677 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T20:38:24.677 INFO:teuthology.orchestra.run.smithi099.stdout: "name": "obj00000000000000000028", 2023-05-31T20:38:24.677 INFO:teuthology.orchestra.run.smithi099.stdout: "instance": "", 2023-05-31T20:38:24.677 INFO:teuthology.orchestra.run.smithi099.stdout: "ver": { 2023-05-31T20:38:24.677 INFO:teuthology.orchestra.run.smithi099.stdout: "pool": 9, 2023-05-31T20:38:24.677 INFO:teuthology.orchestra.run.smithi099.stdout: "epoch": 8 2023-05-31T20:38:24.678 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:38:24.678 INFO:teuthology.orchestra.run.smithi099.stdout: "locator": "", 2023-05-31T20:38:24.678 INFO:teuthology.orchestra.run.smithi099.stdout: "exists": true, 2023-05-31T20:38:24.678 INFO:teuthology.orchestra.run.smithi099.stdout: "meta": { 2023-05-31T20:38:24.678 INFO:teuthology.orchestra.run.smithi099.stdout: "category": 1, 2023-05-31T20:38:24.678 INFO:teuthology.orchestra.run.smithi099.stdout: "size": 3, 2023-05-31T20:38:24.678 INFO:teuthology.orchestra.run.smithi099.stdout: "mtime": "2023-05-31T20:38:15.316632Z", 2023-05-31T20:38:24.679 INFO:teuthology.orchestra.run.smithi099.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-05-31T20:38:24.679 INFO:teuthology.orchestra.run.smithi099.stdout: "storage_class": "", 2023-05-31T20:38:24.679 INFO:teuthology.orchestra.run.smithi099.stdout: "owner": "rgw-multisite-test-user", 2023-05-31T20:38:24.679 INFO:teuthology.orchestra.run.smithi099.stdout: "owner_display_name": "Multisite Test User", 2023-05-31T20:38:24.679 INFO:teuthology.orchestra.run.smithi099.stdout: "content_type": "application/octet-stream", 2023-05-31T20:38:24.679 INFO:teuthology.orchestra.run.smithi099.stdout: "accounted_size": 3, 2023-05-31T20:38:24.679 INFO:teuthology.orchestra.run.smithi099.stdout: "user_data": "", 2023-05-31T20:38:24.680 INFO:teuthology.orchestra.run.smithi099.stdout: "appendable": false 2023-05-31T20:38:24.680 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:38:24.680 INFO:teuthology.orchestra.run.smithi099.stdout: "tag": "3d163b1e-09c7-47e2-ae87-add4aace3f38.5680.9332772936276460195", 2023-05-31T20:38:24.680 INFO:teuthology.orchestra.run.smithi099.stdout: "flags": 0, 2023-05-31T20:38:24.680 INFO:teuthology.orchestra.run.smithi099.stdout: "pending_map": [], 2023-05-31T20:38:24.680 INFO:teuthology.orchestra.run.smithi099.stdout: "versioned_epoch": 0 2023-05-31T20:38:24.681 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:38:24.681 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T20:38:24.681 INFO:teuthology.orchestra.run.smithi099.stdout: "name": "obj00000000000000000029", 2023-05-31T20:38:24.681 INFO:teuthology.orchestra.run.smithi099.stdout: "instance": "", 2023-05-31T20:38:24.681 INFO:teuthology.orchestra.run.smithi099.stdout: "ver": { 2023-05-31T20:38:24.681 INFO:teuthology.orchestra.run.smithi099.stdout: "pool": 9, 2023-05-31T20:38:24.681 INFO:teuthology.orchestra.run.smithi099.stdout: "epoch": 8 2023-05-31T20:38:24.682 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:38:24.682 INFO:teuthology.orchestra.run.smithi099.stdout: "locator": "", 2023-05-31T20:38:24.682 INFO:teuthology.orchestra.run.smithi099.stdout: "exists": true, 2023-05-31T20:38:24.682 INFO:teuthology.orchestra.run.smithi099.stdout: "meta": { 2023-05-31T20:38:24.682 INFO:teuthology.orchestra.run.smithi099.stdout: "category": 1, 2023-05-31T20:38:24.682 INFO:teuthology.orchestra.run.smithi099.stdout: "size": 16, 2023-05-31T20:38:24.682 INFO:teuthology.orchestra.run.smithi099.stdout: "mtime": "2023-05-31T20:38:20.265007Z", 2023-05-31T20:38:24.683 INFO:teuthology.orchestra.run.smithi099.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-05-31T20:38:24.683 INFO:teuthology.orchestra.run.smithi099.stdout: "storage_class": "", 2023-05-31T20:38:24.683 INFO:teuthology.orchestra.run.smithi099.stdout: "owner": "rgw-multisite-test-user", 2023-05-31T20:38:24.683 INFO:teuthology.orchestra.run.smithi099.stdout: "owner_display_name": "Multisite Test User", 2023-05-31T20:38:24.683 INFO:teuthology.orchestra.run.smithi099.stdout: "content_type": "application/octet-stream", 2023-05-31T20:38:24.683 INFO:teuthology.orchestra.run.smithi099.stdout: "accounted_size": 3, 2023-05-31T20:38:24.683 INFO:teuthology.orchestra.run.smithi099.stdout: "user_data": "", 2023-05-31T20:38:24.684 INFO:teuthology.orchestra.run.smithi099.stdout: "appendable": false 2023-05-31T20:38:24.684 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:38:24.684 INFO:teuthology.orchestra.run.smithi099.stdout: "tag": "3d163b1e-09c7-47e2-ae87-add4aace3f38.5680.16967549556226119816", 2023-05-31T20:38:24.684 INFO:teuthology.orchestra.run.smithi099.stdout: "flags": 0, 2023-05-31T20:38:24.684 INFO:teuthology.orchestra.run.smithi099.stdout: "pending_map": [], 2023-05-31T20:38:24.684 INFO:teuthology.orchestra.run.smithi099.stdout: "versioned_epoch": 0 2023-05-31T20:38:24.685 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:38:24.685 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T20:38:24.685 INFO:teuthology.orchestra.run.smithi099.stdout: "name": "obj00000000000000000030", 2023-05-31T20:38:24.685 INFO:teuthology.orchestra.run.smithi099.stdout: "instance": "", 2023-05-31T20:38:24.685 INFO:teuthology.orchestra.run.smithi099.stdout: "ver": { 2023-05-31T20:38:24.685 INFO:teuthology.orchestra.run.smithi099.stdout: "pool": 9, 2023-05-31T20:38:24.685 INFO:teuthology.orchestra.run.smithi099.stdout: "epoch": 9 2023-05-31T20:38:24.686 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:38:24.686 INFO:teuthology.orchestra.run.smithi099.stdout: "locator": "", 2023-05-31T20:38:24.686 INFO:teuthology.orchestra.run.smithi099.stdout: "exists": true, 2023-05-31T20:38:24.686 INFO:teuthology.orchestra.run.smithi099.stdout: "meta": { 2023-05-31T20:38:24.686 INFO:teuthology.orchestra.run.smithi099.stdout: "category": 1, 2023-05-31T20:38:24.686 INFO:teuthology.orchestra.run.smithi099.stdout: "size": 5, 2023-05-31T20:38:24.686 INFO:teuthology.orchestra.run.smithi099.stdout: "mtime": "2023-05-31T20:38:21.641789Z", 2023-05-31T20:38:24.687 INFO:teuthology.orchestra.run.smithi099.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-05-31T20:38:24.687 INFO:teuthology.orchestra.run.smithi099.stdout: "storage_class": "", 2023-05-31T20:38:24.687 INFO:teuthology.orchestra.run.smithi099.stdout: "owner": "rgw-multisite-test-user", 2023-05-31T20:38:24.687 INFO:teuthology.orchestra.run.smithi099.stdout: "owner_display_name": "Multisite Test User", 2023-05-31T20:38:24.687 INFO:teuthology.orchestra.run.smithi099.stdout: "content_type": "application/octet-stream", 2023-05-31T20:38:24.687 INFO:teuthology.orchestra.run.smithi099.stdout: "accounted_size": 3, 2023-05-31T20:38:24.687 INFO:teuthology.orchestra.run.smithi099.stdout: "user_data": "", 2023-05-31T20:38:24.688 INFO:teuthology.orchestra.run.smithi099.stdout: "appendable": false 2023-05-31T20:38:24.688 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:38:24.688 INFO:teuthology.orchestra.run.smithi099.stdout: "tag": "3d163b1e-09c7-47e2-ae87-add4aace3f38.5680.15342963812004263133", 2023-05-31T20:38:24.688 INFO:teuthology.orchestra.run.smithi099.stdout: "flags": 0, 2023-05-31T20:38:24.688 INFO:teuthology.orchestra.run.smithi099.stdout: "pending_map": [], 2023-05-31T20:38:24.688 INFO:teuthology.orchestra.run.smithi099.stdout: "versioned_epoch": 0 2023-05-31T20:38:24.688 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:38:24.689 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T20:38:24.689 INFO:teuthology.orchestra.run.smithi099.stdout: "name": "obj00000000000000000031", 2023-05-31T20:38:24.689 INFO:teuthology.orchestra.run.smithi099.stdout: "instance": "", 2023-05-31T20:38:24.689 INFO:teuthology.orchestra.run.smithi099.stdout: "ver": { 2023-05-31T20:38:24.689 INFO:teuthology.orchestra.run.smithi099.stdout: "pool": 9, 2023-05-31T20:38:24.689 INFO:teuthology.orchestra.run.smithi099.stdout: "epoch": 5 2023-05-31T20:38:24.690 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:38:24.690 INFO:teuthology.orchestra.run.smithi099.stdout: "locator": "", 2023-05-31T20:38:24.690 INFO:teuthology.orchestra.run.smithi099.stdout: "exists": true, 2023-05-31T20:38:24.690 INFO:teuthology.orchestra.run.smithi099.stdout: "meta": { 2023-05-31T20:38:24.690 INFO:teuthology.orchestra.run.smithi099.stdout: "category": 1, 2023-05-31T20:38:24.690 INFO:teuthology.orchestra.run.smithi099.stdout: "size": 3, 2023-05-31T20:38:24.690 INFO:teuthology.orchestra.run.smithi099.stdout: "mtime": "2023-05-31T20:38:18.440797Z", 2023-05-31T20:38:24.691 INFO:teuthology.orchestra.run.smithi099.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-05-31T20:38:24.691 INFO:teuthology.orchestra.run.smithi099.stdout: "storage_class": "", 2023-05-31T20:38:24.691 INFO:teuthology.orchestra.run.smithi099.stdout: "owner": "rgw-multisite-test-user", 2023-05-31T20:38:24.691 INFO:teuthology.orchestra.run.smithi099.stdout: "owner_display_name": "Multisite Test User", 2023-05-31T20:38:24.691 INFO:teuthology.orchestra.run.smithi099.stdout: "content_type": "application/octet-stream", 2023-05-31T20:38:24.691 INFO:teuthology.orchestra.run.smithi099.stdout: "accounted_size": 3, 2023-05-31T20:38:24.691 INFO:teuthology.orchestra.run.smithi099.stdout: "user_data": "", 2023-05-31T20:38:24.692 INFO:teuthology.orchestra.run.smithi099.stdout: "appendable": false 2023-05-31T20:38:24.692 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:38:24.692 INFO:teuthology.orchestra.run.smithi099.stdout: "tag": "3d163b1e-09c7-47e2-ae87-add4aace3f38.5680.1669955571827590050", 2023-05-31T20:38:24.692 INFO:teuthology.orchestra.run.smithi099.stdout: "flags": 0, 2023-05-31T20:38:24.692 INFO:teuthology.orchestra.run.smithi099.stdout: "pending_map": [], 2023-05-31T20:38:24.692 INFO:teuthology.orchestra.run.smithi099.stdout: "versioned_epoch": 0 2023-05-31T20:38:24.692 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:38:24.693 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T20:38:24.693 INFO:teuthology.orchestra.run.smithi099.stdout: "name": "obj00000000000000000032", 2023-05-31T20:38:24.693 INFO:teuthology.orchestra.run.smithi099.stdout: "instance": "", 2023-05-31T20:38:24.693 INFO:teuthology.orchestra.run.smithi099.stdout: "ver": { 2023-05-31T20:38:24.693 INFO:teuthology.orchestra.run.smithi099.stdout: "pool": 9, 2023-05-31T20:38:24.693 INFO:teuthology.orchestra.run.smithi099.stdout: "epoch": 3 2023-05-31T20:38:24.694 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:38:24.694 INFO:teuthology.orchestra.run.smithi099.stdout: "locator": "", 2023-05-31T20:38:24.694 INFO:teuthology.orchestra.run.smithi099.stdout: "exists": true, 2023-05-31T20:38:24.694 INFO:teuthology.orchestra.run.smithi099.stdout: "meta": { 2023-05-31T20:38:24.694 INFO:teuthology.orchestra.run.smithi099.stdout: "category": 1, 2023-05-31T20:38:24.694 INFO:teuthology.orchestra.run.smithi099.stdout: "size": 3, 2023-05-31T20:38:24.694 INFO:teuthology.orchestra.run.smithi099.stdout: "mtime": "2023-05-31T20:38:15.861929Z", 2023-05-31T20:38:24.695 INFO:teuthology.orchestra.run.smithi099.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-05-31T20:38:24.695 INFO:teuthology.orchestra.run.smithi099.stdout: "storage_class": "", 2023-05-31T20:38:24.695 INFO:teuthology.orchestra.run.smithi099.stdout: "owner": "rgw-multisite-test-user", 2023-05-31T20:38:24.695 INFO:teuthology.orchestra.run.smithi099.stdout: "owner_display_name": "Multisite Test User", 2023-05-31T20:38:24.695 INFO:teuthology.orchestra.run.smithi099.stdout: "content_type": "application/octet-stream", 2023-05-31T20:38:24.695 INFO:teuthology.orchestra.run.smithi099.stdout: "accounted_size": 3, 2023-05-31T20:38:24.695 INFO:teuthology.orchestra.run.smithi099.stdout: "user_data": "", 2023-05-31T20:38:24.696 INFO:teuthology.orchestra.run.smithi099.stdout: "appendable": false 2023-05-31T20:38:24.696 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:38:24.696 INFO:teuthology.orchestra.run.smithi099.stdout: "tag": "3d163b1e-09c7-47e2-ae87-add4aace3f38.5680.13304000331746966103", 2023-05-31T20:38:24.696 INFO:teuthology.orchestra.run.smithi099.stdout: "flags": 0, 2023-05-31T20:38:24.696 INFO:teuthology.orchestra.run.smithi099.stdout: "pending_map": [], 2023-05-31T20:38:24.696 INFO:teuthology.orchestra.run.smithi099.stdout: "versioned_epoch": 0 2023-05-31T20:38:24.696 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:38:24.697 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T20:38:24.697 INFO:teuthology.orchestra.run.smithi099.stdout: "name": "obj00000000000000000033", 2023-05-31T20:38:24.697 INFO:teuthology.orchestra.run.smithi099.stdout: "instance": "", 2023-05-31T20:38:24.697 INFO:teuthology.orchestra.run.smithi099.stdout: "ver": { 2023-05-31T20:38:24.697 INFO:teuthology.orchestra.run.smithi099.stdout: "pool": 9, 2023-05-31T20:38:24.697 INFO:teuthology.orchestra.run.smithi099.stdout: "epoch": 10 2023-05-31T20:38:24.697 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:38:24.698 INFO:teuthology.orchestra.run.smithi099.stdout: "locator": "", 2023-05-31T20:38:24.698 INFO:teuthology.orchestra.run.smithi099.stdout: "exists": true, 2023-05-31T20:38:24.698 INFO:teuthology.orchestra.run.smithi099.stdout: "meta": { 2023-05-31T20:38:24.698 INFO:teuthology.orchestra.run.smithi099.stdout: "category": 1, 2023-05-31T20:38:24.698 INFO:teuthology.orchestra.run.smithi099.stdout: "size": 5, 2023-05-31T20:38:24.698 INFO:teuthology.orchestra.run.smithi099.stdout: "mtime": "2023-05-31T20:38:22.084294Z", 2023-05-31T20:38:24.698 INFO:teuthology.orchestra.run.smithi099.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-05-31T20:38:24.699 INFO:teuthology.orchestra.run.smithi099.stdout: "storage_class": "", 2023-05-31T20:38:24.699 INFO:teuthology.orchestra.run.smithi099.stdout: "owner": "rgw-multisite-test-user", 2023-05-31T20:38:24.699 INFO:teuthology.orchestra.run.smithi099.stdout: "owner_display_name": "Multisite Test User", 2023-05-31T20:38:24.699 INFO:teuthology.orchestra.run.smithi099.stdout: "content_type": "application/octet-stream", 2023-05-31T20:38:24.699 INFO:teuthology.orchestra.run.smithi099.stdout: "accounted_size": 3, 2023-05-31T20:38:24.699 INFO:teuthology.orchestra.run.smithi099.stdout: "user_data": "", 2023-05-31T20:38:24.699 INFO:teuthology.orchestra.run.smithi099.stdout: "appendable": false 2023-05-31T20:38:24.700 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:38:24.700 INFO:teuthology.orchestra.run.smithi099.stdout: "tag": "3d163b1e-09c7-47e2-ae87-add4aace3f38.5680.10095714400050292053", 2023-05-31T20:38:24.700 INFO:teuthology.orchestra.run.smithi099.stdout: "flags": 0, 2023-05-31T20:38:24.700 INFO:teuthology.orchestra.run.smithi099.stdout: "pending_map": [], 2023-05-31T20:38:24.700 INFO:teuthology.orchestra.run.smithi099.stdout: "versioned_epoch": 0 2023-05-31T20:38:24.700 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:38:24.701 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T20:38:24.701 INFO:teuthology.orchestra.run.smithi099.stdout: "name": "obj00000000000000000036", 2023-05-31T20:38:24.701 INFO:teuthology.orchestra.run.smithi099.stdout: "instance": "", 2023-05-31T20:38:24.701 INFO:teuthology.orchestra.run.smithi099.stdout: "ver": { 2023-05-31T20:38:24.701 INFO:teuthology.orchestra.run.smithi099.stdout: "pool": 9, 2023-05-31T20:38:24.701 INFO:teuthology.orchestra.run.smithi099.stdout: "epoch": 2 2023-05-31T20:38:24.701 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:38:24.702 INFO:teuthology.orchestra.run.smithi099.stdout: "locator": "", 2023-05-31T20:38:24.702 INFO:teuthology.orchestra.run.smithi099.stdout: "exists": true, 2023-05-31T20:38:24.702 INFO:teuthology.orchestra.run.smithi099.stdout: "meta": { 2023-05-31T20:38:24.702 INFO:teuthology.orchestra.run.smithi099.stdout: "category": 1, 2023-05-31T20:38:24.702 INFO:teuthology.orchestra.run.smithi099.stdout: "size": 5, 2023-05-31T20:38:24.702 INFO:teuthology.orchestra.run.smithi099.stdout: "mtime": "2023-05-31T20:38:22.636301Z", 2023-05-31T20:38:24.702 INFO:teuthology.orchestra.run.smithi099.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-05-31T20:38:24.703 INFO:teuthology.orchestra.run.smithi099.stdout: "storage_class": "", 2023-05-31T20:38:24.703 INFO:teuthology.orchestra.run.smithi099.stdout: "owner": "rgw-multisite-test-user", 2023-05-31T20:38:24.703 INFO:teuthology.orchestra.run.smithi099.stdout: "owner_display_name": "Multisite Test User", 2023-05-31T20:38:24.703 INFO:teuthology.orchestra.run.smithi099.stdout: "content_type": "application/octet-stream", 2023-05-31T20:38:24.703 INFO:teuthology.orchestra.run.smithi099.stdout: "accounted_size": 3, 2023-05-31T20:38:24.703 INFO:teuthology.orchestra.run.smithi099.stdout: "user_data": "", 2023-05-31T20:38:24.703 INFO:teuthology.orchestra.run.smithi099.stdout: "appendable": false 2023-05-31T20:38:24.704 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:38:24.704 INFO:teuthology.orchestra.run.smithi099.stdout: "tag": "3d163b1e-09c7-47e2-ae87-add4aace3f38.5680.1532750743519300969", 2023-05-31T20:38:24.704 INFO:teuthology.orchestra.run.smithi099.stdout: "flags": 0, 2023-05-31T20:38:24.704 INFO:teuthology.orchestra.run.smithi099.stdout: "pending_map": [], 2023-05-31T20:38:24.704 INFO:teuthology.orchestra.run.smithi099.stdout: "versioned_epoch": 0 2023-05-31T20:38:24.704 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:38:24.705 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T20:38:24.705 INFO:teuthology.orchestra.run.smithi099.stdout: "name": "obj00000000000000000037", 2023-05-31T20:38:24.705 INFO:teuthology.orchestra.run.smithi099.stdout: "instance": "", 2023-05-31T20:38:24.705 INFO:teuthology.orchestra.run.smithi099.stdout: "ver": { 2023-05-31T20:38:24.705 INFO:teuthology.orchestra.run.smithi099.stdout: "pool": 9, 2023-05-31T20:38:24.705 INFO:teuthology.orchestra.run.smithi099.stdout: "epoch": 7 2023-05-31T20:38:24.705 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:38:24.706 INFO:teuthology.orchestra.run.smithi099.stdout: "locator": "", 2023-05-31T20:38:24.706 INFO:teuthology.orchestra.run.smithi099.stdout: "exists": true, 2023-05-31T20:38:24.706 INFO:teuthology.orchestra.run.smithi099.stdout: "meta": { 2023-05-31T20:38:24.706 INFO:teuthology.orchestra.run.smithi099.stdout: "category": 1, 2023-05-31T20:38:24.706 INFO:teuthology.orchestra.run.smithi099.stdout: "size": 3, 2023-05-31T20:38:24.706 INFO:teuthology.orchestra.run.smithi099.stdout: "mtime": "2023-05-31T20:38:19.358015Z", 2023-05-31T20:38:24.706 INFO:teuthology.orchestra.run.smithi099.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-05-31T20:38:24.707 INFO:teuthology.orchestra.run.smithi099.stdout: "storage_class": "", 2023-05-31T20:38:24.707 INFO:teuthology.orchestra.run.smithi099.stdout: "owner": "rgw-multisite-test-user", 2023-05-31T20:38:24.707 INFO:teuthology.orchestra.run.smithi099.stdout: "owner_display_name": "Multisite Test User", 2023-05-31T20:38:24.707 INFO:teuthology.orchestra.run.smithi099.stdout: "content_type": "application/octet-stream", 2023-05-31T20:38:24.707 INFO:teuthology.orchestra.run.smithi099.stdout: "accounted_size": 3, 2023-05-31T20:38:24.707 INFO:teuthology.orchestra.run.smithi099.stdout: "user_data": "", 2023-05-31T20:38:24.707 INFO:teuthology.orchestra.run.smithi099.stdout: "appendable": false 2023-05-31T20:38:24.708 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:38:24.708 INFO:teuthology.orchestra.run.smithi099.stdout: "tag": "3d163b1e-09c7-47e2-ae87-add4aace3f38.5680.10596013392990456041", 2023-05-31T20:38:24.708 INFO:teuthology.orchestra.run.smithi099.stdout: "flags": 0, 2023-05-31T20:38:24.708 INFO:teuthology.orchestra.run.smithi099.stdout: "pending_map": [], 2023-05-31T20:38:24.708 INFO:teuthology.orchestra.run.smithi099.stdout: "versioned_epoch": 0 2023-05-31T20:38:24.709 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:38:24.709 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T20:38:24.709 INFO:teuthology.orchestra.run.smithi099.stdout: "name": "obj00000000000000000039", 2023-05-31T20:38:24.709 INFO:teuthology.orchestra.run.smithi099.stdout: "instance": "", 2023-05-31T20:38:24.709 INFO:teuthology.orchestra.run.smithi099.stdout: "ver": { 2023-05-31T20:38:24.709 INFO:teuthology.orchestra.run.smithi099.stdout: "pool": 9, 2023-05-31T20:38:24.709 INFO:teuthology.orchestra.run.smithi099.stdout: "epoch": 3 2023-05-31T20:38:24.710 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:38:24.710 INFO:teuthology.orchestra.run.smithi099.stdout: "locator": "", 2023-05-31T20:38:24.710 INFO:teuthology.orchestra.run.smithi099.stdout: "exists": true, 2023-05-31T20:38:24.710 INFO:teuthology.orchestra.run.smithi099.stdout: "meta": { 2023-05-31T20:38:24.710 INFO:teuthology.orchestra.run.smithi099.stdout: "category": 1, 2023-05-31T20:38:24.710 INFO:teuthology.orchestra.run.smithi099.stdout: "size": 5, 2023-05-31T20:38:24.710 INFO:teuthology.orchestra.run.smithi099.stdout: "mtime": "2023-05-31T20:38:22.383805Z", 2023-05-31T20:38:24.711 INFO:teuthology.orchestra.run.smithi099.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-05-31T20:38:24.711 INFO:teuthology.orchestra.run.smithi099.stdout: "storage_class": "", 2023-05-31T20:38:24.711 INFO:teuthology.orchestra.run.smithi099.stdout: "owner": "rgw-multisite-test-user", 2023-05-31T20:38:24.711 INFO:teuthology.orchestra.run.smithi099.stdout: "owner_display_name": "Multisite Test User", 2023-05-31T20:38:24.711 INFO:teuthology.orchestra.run.smithi099.stdout: "content_type": "application/octet-stream", 2023-05-31T20:38:24.711 INFO:teuthology.orchestra.run.smithi099.stdout: "accounted_size": 3, 2023-05-31T20:38:24.711 INFO:teuthology.orchestra.run.smithi099.stdout: "user_data": "", 2023-05-31T20:38:24.712 INFO:teuthology.orchestra.run.smithi099.stdout: "appendable": false 2023-05-31T20:38:24.712 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:38:24.712 INFO:teuthology.orchestra.run.smithi099.stdout: "tag": "3d163b1e-09c7-47e2-ae87-add4aace3f38.5680.15468886705879519061", 2023-05-31T20:38:24.712 INFO:teuthology.orchestra.run.smithi099.stdout: "flags": 0, 2023-05-31T20:38:24.712 INFO:teuthology.orchestra.run.smithi099.stdout: "pending_map": [], 2023-05-31T20:38:24.712 INFO:teuthology.orchestra.run.smithi099.stdout: "versioned_epoch": 0 2023-05-31T20:38:24.713 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:38:24.713 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T20:38:24.713 INFO:teuthology.orchestra.run.smithi099.stdout: "name": "obj00000000000000000040", 2023-05-31T20:38:24.713 INFO:teuthology.orchestra.run.smithi099.stdout: "instance": "", 2023-05-31T20:38:24.713 INFO:teuthology.orchestra.run.smithi099.stdout: "ver": { 2023-05-31T20:38:24.713 INFO:teuthology.orchestra.run.smithi099.stdout: "pool": 9, 2023-05-31T20:38:24.713 INFO:teuthology.orchestra.run.smithi099.stdout: "epoch": 6 2023-05-31T20:38:24.714 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:38:24.714 INFO:teuthology.orchestra.run.smithi099.stdout: "locator": "", 2023-05-31T20:38:24.714 INFO:teuthology.orchestra.run.smithi099.stdout: "exists": true, 2023-05-31T20:38:24.714 INFO:teuthology.orchestra.run.smithi099.stdout: "meta": { 2023-05-31T20:38:24.714 INFO:teuthology.orchestra.run.smithi099.stdout: "category": 1, 2023-05-31T20:38:24.714 INFO:teuthology.orchestra.run.smithi099.stdout: "size": 5, 2023-05-31T20:38:24.714 INFO:teuthology.orchestra.run.smithi099.stdout: "mtime": "2023-05-31T20:38:22.327307Z", 2023-05-31T20:38:24.715 INFO:teuthology.orchestra.run.smithi099.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-05-31T20:38:24.715 INFO:teuthology.orchestra.run.smithi099.stdout: "storage_class": "", 2023-05-31T20:38:24.715 INFO:teuthology.orchestra.run.smithi099.stdout: "owner": "rgw-multisite-test-user", 2023-05-31T20:38:24.715 INFO:teuthology.orchestra.run.smithi099.stdout: "owner_display_name": "Multisite Test User", 2023-05-31T20:38:24.715 INFO:teuthology.orchestra.run.smithi099.stdout: "content_type": "application/octet-stream", 2023-05-31T20:38:24.715 INFO:teuthology.orchestra.run.smithi099.stdout: "accounted_size": 3, 2023-05-31T20:38:24.715 INFO:teuthology.orchestra.run.smithi099.stdout: "user_data": "", 2023-05-31T20:38:24.716 INFO:teuthology.orchestra.run.smithi099.stdout: "appendable": false 2023-05-31T20:38:24.716 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:38:24.716 INFO:teuthology.orchestra.run.smithi099.stdout: "tag": "3d163b1e-09c7-47e2-ae87-add4aace3f38.5680.13822412859644161105", 2023-05-31T20:38:24.716 INFO:teuthology.orchestra.run.smithi099.stdout: "flags": 0, 2023-05-31T20:38:24.716 INFO:teuthology.orchestra.run.smithi099.stdout: "pending_map": [], 2023-05-31T20:38:24.716 INFO:teuthology.orchestra.run.smithi099.stdout: "versioned_epoch": 0 2023-05-31T20:38:24.716 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:38:24.717 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T20:38:24.717 INFO:teuthology.orchestra.run.smithi099.stdout: "name": "obj00000000000000000042", 2023-05-31T20:38:24.717 INFO:teuthology.orchestra.run.smithi099.stdout: "instance": "", 2023-05-31T20:38:24.717 INFO:teuthology.orchestra.run.smithi099.stdout: "ver": { 2023-05-31T20:38:24.717 INFO:teuthology.orchestra.run.smithi099.stdout: "pool": 9, 2023-05-31T20:38:24.717 INFO:teuthology.orchestra.run.smithi099.stdout: "epoch": 1 2023-05-31T20:38:24.717 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:38:24.718 INFO:teuthology.orchestra.run.smithi099.stdout: "locator": "", 2023-05-31T20:38:24.718 INFO:teuthology.orchestra.run.smithi099.stdout: "exists": true, 2023-05-31T20:38:24.718 INFO:teuthology.orchestra.run.smithi099.stdout: "meta": { 2023-05-31T20:38:24.718 INFO:teuthology.orchestra.run.smithi099.stdout: "category": 1, 2023-05-31T20:38:24.718 INFO:teuthology.orchestra.run.smithi099.stdout: "size": 5, 2023-05-31T20:38:24.718 INFO:teuthology.orchestra.run.smithi099.stdout: "mtime": "2023-05-31T20:38:18.977382Z", 2023-05-31T20:38:24.718 INFO:teuthology.orchestra.run.smithi099.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-05-31T20:38:24.719 INFO:teuthology.orchestra.run.smithi099.stdout: "storage_class": "", 2023-05-31T20:38:24.719 INFO:teuthology.orchestra.run.smithi099.stdout: "owner": "rgw-multisite-test-user", 2023-05-31T20:38:24.719 INFO:teuthology.orchestra.run.smithi099.stdout: "owner_display_name": "Multisite Test User", 2023-05-31T20:38:24.719 INFO:teuthology.orchestra.run.smithi099.stdout: "content_type": "application/octet-stream", 2023-05-31T20:38:24.719 INFO:teuthology.orchestra.run.smithi099.stdout: "accounted_size": 3, 2023-05-31T20:38:24.719 INFO:teuthology.orchestra.run.smithi099.stdout: "user_data": "", 2023-05-31T20:38:24.719 INFO:teuthology.orchestra.run.smithi099.stdout: "appendable": false 2023-05-31T20:38:24.720 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:38:24.720 INFO:teuthology.orchestra.run.smithi099.stdout: "tag": "3d163b1e-09c7-47e2-ae87-add4aace3f38.5680.16221895930665093678", 2023-05-31T20:38:24.720 INFO:teuthology.orchestra.run.smithi099.stdout: "flags": 0, 2023-05-31T20:38:24.720 INFO:teuthology.orchestra.run.smithi099.stdout: "pending_map": [], 2023-05-31T20:38:24.720 INFO:teuthology.orchestra.run.smithi099.stdout: "versioned_epoch": 0 2023-05-31T20:38:24.720 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:38:24.721 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T20:38:24.721 INFO:teuthology.orchestra.run.smithi099.stdout: "name": "obj00000000000000000045", 2023-05-31T20:38:24.721 INFO:teuthology.orchestra.run.smithi099.stdout: "instance": "", 2023-05-31T20:38:24.721 INFO:teuthology.orchestra.run.smithi099.stdout: "ver": { 2023-05-31T20:38:24.721 INFO:teuthology.orchestra.run.smithi099.stdout: "pool": 9, 2023-05-31T20:38:24.721 INFO:teuthology.orchestra.run.smithi099.stdout: "epoch": 16 2023-05-31T20:38:24.721 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:38:24.722 INFO:teuthology.orchestra.run.smithi099.stdout: "locator": "", 2023-05-31T20:38:24.722 INFO:teuthology.orchestra.run.smithi099.stdout: "exists": true, 2023-05-31T20:38:24.722 INFO:teuthology.orchestra.run.smithi099.stdout: "meta": { 2023-05-31T20:38:24.722 INFO:teuthology.orchestra.run.smithi099.stdout: "category": 1, 2023-05-31T20:38:24.722 INFO:teuthology.orchestra.run.smithi099.stdout: "size": 6, 2023-05-31T20:38:24.722 INFO:teuthology.orchestra.run.smithi099.stdout: "mtime": "2023-05-31T20:38:22.250679Z", 2023-05-31T20:38:24.722 INFO:teuthology.orchestra.run.smithi099.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-05-31T20:38:24.723 INFO:teuthology.orchestra.run.smithi099.stdout: "storage_class": "", 2023-05-31T20:38:24.723 INFO:teuthology.orchestra.run.smithi099.stdout: "owner": "rgw-multisite-test-user", 2023-05-31T20:38:24.723 INFO:teuthology.orchestra.run.smithi099.stdout: "owner_display_name": "Multisite Test User", 2023-05-31T20:38:24.723 INFO:teuthology.orchestra.run.smithi099.stdout: "content_type": "application/octet-stream", 2023-05-31T20:38:24.723 INFO:teuthology.orchestra.run.smithi099.stdout: "accounted_size": 3, 2023-05-31T20:38:24.723 INFO:teuthology.orchestra.run.smithi099.stdout: "user_data": "", 2023-05-31T20:38:24.724 INFO:teuthology.orchestra.run.smithi099.stdout: "appendable": false 2023-05-31T20:38:24.724 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:38:24.724 INFO:teuthology.orchestra.run.smithi099.stdout: "tag": "3d163b1e-09c7-47e2-ae87-add4aace3f38.5680.15488274039320332867", 2023-05-31T20:38:24.724 INFO:teuthology.orchestra.run.smithi099.stdout: "flags": 0, 2023-05-31T20:38:24.724 INFO:teuthology.orchestra.run.smithi099.stdout: "pending_map": [], 2023-05-31T20:38:24.724 INFO:teuthology.orchestra.run.smithi099.stdout: "versioned_epoch": 0 2023-05-31T20:38:24.725 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:38:24.725 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T20:38:24.725 INFO:teuthology.orchestra.run.smithi099.stdout: "name": "obj00000000000000000046", 2023-05-31T20:38:24.725 INFO:teuthology.orchestra.run.smithi099.stdout: "instance": "", 2023-05-31T20:38:24.725 INFO:teuthology.orchestra.run.smithi099.stdout: "ver": { 2023-05-31T20:38:24.725 INFO:teuthology.orchestra.run.smithi099.stdout: "pool": 9, 2023-05-31T20:38:24.725 INFO:teuthology.orchestra.run.smithi099.stdout: "epoch": 4 2023-05-31T20:38:24.726 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:38:24.726 INFO:teuthology.orchestra.run.smithi099.stdout: "locator": "", 2023-05-31T20:38:24.726 INFO:teuthology.orchestra.run.smithi099.stdout: "exists": true, 2023-05-31T20:38:24.726 INFO:teuthology.orchestra.run.smithi099.stdout: "meta": { 2023-05-31T20:38:24.726 INFO:teuthology.orchestra.run.smithi099.stdout: "category": 1, 2023-05-31T20:38:24.726 INFO:teuthology.orchestra.run.smithi099.stdout: "size": 16, 2023-05-31T20:38:24.726 INFO:teuthology.orchestra.run.smithi099.stdout: "mtime": "2023-05-31T20:38:17.357924Z", 2023-05-31T20:38:24.727 INFO:teuthology.orchestra.run.smithi099.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-05-31T20:38:24.727 INFO:teuthology.orchestra.run.smithi099.stdout: "storage_class": "", 2023-05-31T20:38:24.727 INFO:teuthology.orchestra.run.smithi099.stdout: "owner": "rgw-multisite-test-user", 2023-05-31T20:38:24.727 INFO:teuthology.orchestra.run.smithi099.stdout: "owner_display_name": "Multisite Test User", 2023-05-31T20:38:24.727 INFO:teuthology.orchestra.run.smithi099.stdout: "content_type": "application/octet-stream", 2023-05-31T20:38:24.727 INFO:teuthology.orchestra.run.smithi099.stdout: "accounted_size": 3, 2023-05-31T20:38:24.727 INFO:teuthology.orchestra.run.smithi099.stdout: "user_data": "", 2023-05-31T20:38:24.728 INFO:teuthology.orchestra.run.smithi099.stdout: "appendable": false 2023-05-31T20:38:24.728 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:38:24.728 INFO:teuthology.orchestra.run.smithi099.stdout: "tag": "3d163b1e-09c7-47e2-ae87-add4aace3f38.5680.7857088783163447655", 2023-05-31T20:38:24.728 INFO:teuthology.orchestra.run.smithi099.stdout: "flags": 0, 2023-05-31T20:38:24.728 INFO:teuthology.orchestra.run.smithi099.stdout: "pending_map": [], 2023-05-31T20:38:24.728 INFO:teuthology.orchestra.run.smithi099.stdout: "versioned_epoch": 0 2023-05-31T20:38:24.728 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:38:24.729 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T20:38:24.729 INFO:teuthology.orchestra.run.smithi099.stdout: "name": "obj00000000000000000048", 2023-05-31T20:38:24.729 INFO:teuthology.orchestra.run.smithi099.stdout: "instance": "", 2023-05-31T20:38:24.729 INFO:teuthology.orchestra.run.smithi099.stdout: "ver": { 2023-05-31T20:38:24.729 INFO:teuthology.orchestra.run.smithi099.stdout: "pool": 9, 2023-05-31T20:38:24.729 INFO:teuthology.orchestra.run.smithi099.stdout: "epoch": 6 2023-05-31T20:38:24.729 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:38:24.730 INFO:teuthology.orchestra.run.smithi099.stdout: "locator": "", 2023-05-31T20:38:24.730 INFO:teuthology.orchestra.run.smithi099.stdout: "exists": true, 2023-05-31T20:38:24.730 INFO:teuthology.orchestra.run.smithi099.stdout: "meta": { 2023-05-31T20:38:24.730 INFO:teuthology.orchestra.run.smithi099.stdout: "category": 1, 2023-05-31T20:38:24.730 INFO:teuthology.orchestra.run.smithi099.stdout: "size": 16, 2023-05-31T20:38:24.730 INFO:teuthology.orchestra.run.smithi099.stdout: "mtime": "2023-05-31T20:38:22.489163Z", 2023-05-31T20:38:24.730 INFO:teuthology.orchestra.run.smithi099.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-05-31T20:38:24.731 INFO:teuthology.orchestra.run.smithi099.stdout: "storage_class": "", 2023-05-31T20:38:24.731 INFO:teuthology.orchestra.run.smithi099.stdout: "owner": "rgw-multisite-test-user", 2023-05-31T20:38:24.731 INFO:teuthology.orchestra.run.smithi099.stdout: "owner_display_name": "Multisite Test User", 2023-05-31T20:38:24.731 INFO:teuthology.orchestra.run.smithi099.stdout: "content_type": "application/octet-stream", 2023-05-31T20:38:24.731 INFO:teuthology.orchestra.run.smithi099.stdout: "accounted_size": 3, 2023-05-31T20:38:24.731 INFO:teuthology.orchestra.run.smithi099.stdout: "user_data": "", 2023-05-31T20:38:24.731 INFO:teuthology.orchestra.run.smithi099.stdout: "appendable": false 2023-05-31T20:38:24.732 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:38:24.732 INFO:teuthology.orchestra.run.smithi099.stdout: "tag": "3d163b1e-09c7-47e2-ae87-add4aace3f38.5680.12555797142107976239", 2023-05-31T20:38:24.732 INFO:teuthology.orchestra.run.smithi099.stdout: "flags": 0, 2023-05-31T20:38:24.732 INFO:teuthology.orchestra.run.smithi099.stdout: "pending_map": [], 2023-05-31T20:38:24.732 INFO:teuthology.orchestra.run.smithi099.stdout: "versioned_epoch": 0 2023-05-31T20:38:24.732 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:38:24.733 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T20:38:24.733 INFO:teuthology.orchestra.run.smithi099.stdout: "name": "obj00000000000000000049", 2023-05-31T20:38:24.733 INFO:teuthology.orchestra.run.smithi099.stdout: "instance": "", 2023-05-31T20:38:24.733 INFO:teuthology.orchestra.run.smithi099.stdout: "ver": { 2023-05-31T20:38:24.733 INFO:teuthology.orchestra.run.smithi099.stdout: "pool": 9, 2023-05-31T20:38:24.733 INFO:teuthology.orchestra.run.smithi099.stdout: "epoch": 2 2023-05-31T20:38:24.733 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:38:24.734 INFO:teuthology.orchestra.run.smithi099.stdout: "locator": "", 2023-05-31T20:38:24.734 INFO:teuthology.orchestra.run.smithi099.stdout: "exists": true, 2023-05-31T20:38:24.734 INFO:teuthology.orchestra.run.smithi099.stdout: "meta": { 2023-05-31T20:38:24.734 INFO:teuthology.orchestra.run.smithi099.stdout: "category": 1, 2023-05-31T20:38:24.734 INFO:teuthology.orchestra.run.smithi099.stdout: "size": 5, 2023-05-31T20:38:24.734 INFO:teuthology.orchestra.run.smithi099.stdout: "mtime": "2023-05-31T20:38:18.527639Z", 2023-05-31T20:38:24.734 INFO:teuthology.orchestra.run.smithi099.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-05-31T20:38:24.735 INFO:teuthology.orchestra.run.smithi099.stdout: "storage_class": "", 2023-05-31T20:38:24.735 INFO:teuthology.orchestra.run.smithi099.stdout: "owner": "rgw-multisite-test-user", 2023-05-31T20:38:24.735 INFO:teuthology.orchestra.run.smithi099.stdout: "owner_display_name": "Multisite Test User", 2023-05-31T20:38:24.735 INFO:teuthology.orchestra.run.smithi099.stdout: "content_type": "application/octet-stream", 2023-05-31T20:38:24.735 INFO:teuthology.orchestra.run.smithi099.stdout: "accounted_size": 3, 2023-05-31T20:38:24.735 INFO:teuthology.orchestra.run.smithi099.stdout: "user_data": "", 2023-05-31T20:38:24.735 INFO:teuthology.orchestra.run.smithi099.stdout: "appendable": false 2023-05-31T20:38:24.736 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:38:24.736 INFO:teuthology.orchestra.run.smithi099.stdout: "tag": "3d163b1e-09c7-47e2-ae87-add4aace3f38.5680.719112358220703724", 2023-05-31T20:38:24.736 INFO:teuthology.orchestra.run.smithi099.stdout: "flags": 0, 2023-05-31T20:38:24.736 INFO:teuthology.orchestra.run.smithi099.stdout: "pending_map": [], 2023-05-31T20:38:24.736 INFO:teuthology.orchestra.run.smithi099.stdout: "versioned_epoch": 0 2023-05-31T20:38:24.736 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:38:24.736 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T20:38:24.737 INFO:teuthology.orchestra.run.smithi099.stdout: "name": "obj00000000000000000050", 2023-05-31T20:38:24.737 INFO:teuthology.orchestra.run.smithi099.stdout: "instance": "", 2023-05-31T20:38:24.737 INFO:teuthology.orchestra.run.smithi099.stdout: "ver": { 2023-05-31T20:38:24.737 INFO:teuthology.orchestra.run.smithi099.stdout: "pool": 9, 2023-05-31T20:38:24.737 INFO:teuthology.orchestra.run.smithi099.stdout: "epoch": 8 2023-05-31T20:38:24.737 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:38:24.738 INFO:teuthology.orchestra.run.smithi099.stdout: "locator": "", 2023-05-31T20:38:24.738 INFO:teuthology.orchestra.run.smithi099.stdout: "exists": true, 2023-05-31T20:38:24.738 INFO:teuthology.orchestra.run.smithi099.stdout: "meta": { 2023-05-31T20:38:24.738 INFO:teuthology.orchestra.run.smithi099.stdout: "category": 1, 2023-05-31T20:38:24.738 INFO:teuthology.orchestra.run.smithi099.stdout: "size": 16, 2023-05-31T20:38:24.738 INFO:teuthology.orchestra.run.smithi099.stdout: "mtime": "2023-05-31T20:38:20.183372Z", 2023-05-31T20:38:24.738 INFO:teuthology.orchestra.run.smithi099.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-05-31T20:38:24.739 INFO:teuthology.orchestra.run.smithi099.stdout: "storage_class": "", 2023-05-31T20:38:24.739 INFO:teuthology.orchestra.run.smithi099.stdout: "owner": "rgw-multisite-test-user", 2023-05-31T20:38:24.739 INFO:teuthology.orchestra.run.smithi099.stdout: "owner_display_name": "Multisite Test User", 2023-05-31T20:38:24.739 INFO:teuthology.orchestra.run.smithi099.stdout: "content_type": "application/octet-stream", 2023-05-31T20:38:24.739 INFO:teuthology.orchestra.run.smithi099.stdout: "accounted_size": 3, 2023-05-31T20:38:24.739 INFO:teuthology.orchestra.run.smithi099.stdout: "user_data": "", 2023-05-31T20:38:24.739 INFO:teuthology.orchestra.run.smithi099.stdout: "appendable": false 2023-05-31T20:38:24.739 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:38:24.740 INFO:teuthology.orchestra.run.smithi099.stdout: "tag": "3d163b1e-09c7-47e2-ae87-add4aace3f38.5680.4629102582112431380", 2023-05-31T20:38:24.740 INFO:teuthology.orchestra.run.smithi099.stdout: "flags": 0, 2023-05-31T20:38:24.740 INFO:teuthology.orchestra.run.smithi099.stdout: "pending_map": [], 2023-05-31T20:38:24.740 INFO:teuthology.orchestra.run.smithi099.stdout: "versioned_epoch": 0 2023-05-31T20:38:24.740 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:38:24.740 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T20:38:24.741 INFO:teuthology.orchestra.run.smithi099.stdout: "name": "obj00000000000000000053", 2023-05-31T20:38:24.741 INFO:teuthology.orchestra.run.smithi099.stdout: "instance": "", 2023-05-31T20:38:24.741 INFO:teuthology.orchestra.run.smithi099.stdout: "ver": { 2023-05-31T20:38:24.741 INFO:teuthology.orchestra.run.smithi099.stdout: "pool": 9, 2023-05-31T20:38:24.741 INFO:teuthology.orchestra.run.smithi099.stdout: "epoch": 13 2023-05-31T20:38:24.741 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:38:24.741 INFO:teuthology.orchestra.run.smithi099.stdout: "locator": "", 2023-05-31T20:38:24.742 INFO:teuthology.orchestra.run.smithi099.stdout: "exists": true, 2023-05-31T20:38:24.742 INFO:teuthology.orchestra.run.smithi099.stdout: "meta": { 2023-05-31T20:38:24.742 INFO:teuthology.orchestra.run.smithi099.stdout: "category": 1, 2023-05-31T20:38:24.742 INFO:teuthology.orchestra.run.smithi099.stdout: "size": 3, 2023-05-31T20:38:24.742 INFO:teuthology.orchestra.run.smithi099.stdout: "mtime": "2023-05-31T20:38:22.873398Z", 2023-05-31T20:38:24.742 INFO:teuthology.orchestra.run.smithi099.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-05-31T20:38:24.742 INFO:teuthology.orchestra.run.smithi099.stdout: "storage_class": "", 2023-05-31T20:38:24.743 INFO:teuthology.orchestra.run.smithi099.stdout: "owner": "rgw-multisite-test-user", 2023-05-31T20:38:24.743 INFO:teuthology.orchestra.run.smithi099.stdout: "owner_display_name": "Multisite Test User", 2023-05-31T20:38:24.743 INFO:teuthology.orchestra.run.smithi099.stdout: "content_type": "application/octet-stream", 2023-05-31T20:38:24.743 INFO:teuthology.orchestra.run.smithi099.stdout: "accounted_size": 3, 2023-05-31T20:38:24.743 INFO:teuthology.orchestra.run.smithi099.stdout: "user_data": "", 2023-05-31T20:38:24.743 INFO:teuthology.orchestra.run.smithi099.stdout: "appendable": false 2023-05-31T20:38:24.743 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:38:24.744 INFO:teuthology.orchestra.run.smithi099.stdout: "tag": "3d163b1e-09c7-47e2-ae87-add4aace3f38.5680.13542982447900687726", 2023-05-31T20:38:24.744 INFO:teuthology.orchestra.run.smithi099.stdout: "flags": 0, 2023-05-31T20:38:24.744 INFO:teuthology.orchestra.run.smithi099.stdout: "pending_map": [], 2023-05-31T20:38:24.744 INFO:teuthology.orchestra.run.smithi099.stdout: "versioned_epoch": 0 2023-05-31T20:38:24.744 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:38:24.744 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T20:38:24.744 INFO:teuthology.orchestra.run.smithi099.stdout: "name": "obj00000000000000000055", 2023-05-31T20:38:24.745 INFO:teuthology.orchestra.run.smithi099.stdout: "instance": "", 2023-05-31T20:38:24.745 INFO:teuthology.orchestra.run.smithi099.stdout: "ver": { 2023-05-31T20:38:24.745 INFO:teuthology.orchestra.run.smithi099.stdout: "pool": 9, 2023-05-31T20:38:24.745 INFO:teuthology.orchestra.run.smithi099.stdout: "epoch": 2 2023-05-31T20:38:24.745 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:38:24.745 INFO:teuthology.orchestra.run.smithi099.stdout: "locator": "", 2023-05-31T20:38:24.745 INFO:teuthology.orchestra.run.smithi099.stdout: "exists": true, 2023-05-31T20:38:24.746 INFO:teuthology.orchestra.run.smithi099.stdout: "meta": { 2023-05-31T20:38:24.746 INFO:teuthology.orchestra.run.smithi099.stdout: "category": 1, 2023-05-31T20:38:24.746 INFO:teuthology.orchestra.run.smithi099.stdout: "size": 5, 2023-05-31T20:38:24.746 INFO:teuthology.orchestra.run.smithi099.stdout: "mtime": "2023-05-31T20:38:17.862999Z", 2023-05-31T20:38:24.746 INFO:teuthology.orchestra.run.smithi099.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-05-31T20:38:24.746 INFO:teuthology.orchestra.run.smithi099.stdout: "storage_class": "", 2023-05-31T20:38:24.746 INFO:teuthology.orchestra.run.smithi099.stdout: "owner": "rgw-multisite-test-user", 2023-05-31T20:38:24.747 INFO:teuthology.orchestra.run.smithi099.stdout: "owner_display_name": "Multisite Test User", 2023-05-31T20:38:24.747 INFO:teuthology.orchestra.run.smithi099.stdout: "content_type": "application/octet-stream", 2023-05-31T20:38:24.747 INFO:teuthology.orchestra.run.smithi099.stdout: "accounted_size": 3, 2023-05-31T20:38:24.747 INFO:teuthology.orchestra.run.smithi099.stdout: "user_data": "", 2023-05-31T20:38:24.747 INFO:teuthology.orchestra.run.smithi099.stdout: "appendable": false 2023-05-31T20:38:24.747 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:38:24.747 INFO:teuthology.orchestra.run.smithi099.stdout: "tag": "3d163b1e-09c7-47e2-ae87-add4aace3f38.5680.8839032304492622547", 2023-05-31T20:38:24.748 INFO:teuthology.orchestra.run.smithi099.stdout: "flags": 0, 2023-05-31T20:38:24.748 INFO:teuthology.orchestra.run.smithi099.stdout: "pending_map": [], 2023-05-31T20:38:24.748 INFO:teuthology.orchestra.run.smithi099.stdout: "versioned_epoch": 0 2023-05-31T20:38:24.748 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:38:24.748 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T20:38:24.748 INFO:teuthology.orchestra.run.smithi099.stdout: "name": "obj00000000000000000056", 2023-05-31T20:38:24.748 INFO:teuthology.orchestra.run.smithi099.stdout: "instance": "", 2023-05-31T20:38:24.749 INFO:teuthology.orchestra.run.smithi099.stdout: "ver": { 2023-05-31T20:38:24.749 INFO:teuthology.orchestra.run.smithi099.stdout: "pool": 9, 2023-05-31T20:38:24.749 INFO:teuthology.orchestra.run.smithi099.stdout: "epoch": 4 2023-05-31T20:38:24.749 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:38:24.749 INFO:teuthology.orchestra.run.smithi099.stdout: "locator": "", 2023-05-31T20:38:24.749 INFO:teuthology.orchestra.run.smithi099.stdout: "exists": true, 2023-05-31T20:38:24.749 INFO:teuthology.orchestra.run.smithi099.stdout: "meta": { 2023-05-31T20:38:24.750 INFO:teuthology.orchestra.run.smithi099.stdout: "category": 1, 2023-05-31T20:38:24.750 INFO:teuthology.orchestra.run.smithi099.stdout: "size": 3, 2023-05-31T20:38:24.750 INFO:teuthology.orchestra.run.smithi099.stdout: "mtime": "2023-05-31T20:38:20.370757Z", 2023-05-31T20:38:24.750 INFO:teuthology.orchestra.run.smithi099.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-05-31T20:38:24.750 INFO:teuthology.orchestra.run.smithi099.stdout: "storage_class": "", 2023-05-31T20:38:24.750 INFO:teuthology.orchestra.run.smithi099.stdout: "owner": "rgw-multisite-test-user", 2023-05-31T20:38:24.750 INFO:teuthology.orchestra.run.smithi099.stdout: "owner_display_name": "Multisite Test User", 2023-05-31T20:38:24.751 INFO:teuthology.orchestra.run.smithi099.stdout: "content_type": "application/octet-stream", 2023-05-31T20:38:24.751 INFO:teuthology.orchestra.run.smithi099.stdout: "accounted_size": 3, 2023-05-31T20:38:24.751 INFO:teuthology.orchestra.run.smithi099.stdout: "user_data": "", 2023-05-31T20:38:24.751 INFO:teuthology.orchestra.run.smithi099.stdout: "appendable": false 2023-05-31T20:38:24.751 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:38:24.751 INFO:teuthology.orchestra.run.smithi099.stdout: "tag": "3d163b1e-09c7-47e2-ae87-add4aace3f38.5680.11470695168016954631", 2023-05-31T20:38:24.751 INFO:teuthology.orchestra.run.smithi099.stdout: "flags": 0, 2023-05-31T20:38:24.752 INFO:teuthology.orchestra.run.smithi099.stdout: "pending_map": [], 2023-05-31T20:38:24.752 INFO:teuthology.orchestra.run.smithi099.stdout: "versioned_epoch": 0 2023-05-31T20:38:24.752 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:38:24.752 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T20:38:24.752 INFO:teuthology.orchestra.run.smithi099.stdout: "name": "obj00000000000000000057", 2023-05-31T20:38:24.752 INFO:teuthology.orchestra.run.smithi099.stdout: "instance": "", 2023-05-31T20:38:24.752 INFO:teuthology.orchestra.run.smithi099.stdout: "ver": { 2023-05-31T20:38:24.753 INFO:teuthology.orchestra.run.smithi099.stdout: "pool": 9, 2023-05-31T20:38:24.753 INFO:teuthology.orchestra.run.smithi099.stdout: "epoch": 9 2023-05-31T20:38:24.753 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:38:24.753 INFO:teuthology.orchestra.run.smithi099.stdout: "locator": "", 2023-05-31T20:38:24.753 INFO:teuthology.orchestra.run.smithi099.stdout: "exists": true, 2023-05-31T20:38:24.753 INFO:teuthology.orchestra.run.smithi099.stdout: "meta": { 2023-05-31T20:38:24.754 INFO:teuthology.orchestra.run.smithi099.stdout: "category": 1, 2023-05-31T20:38:24.754 INFO:teuthology.orchestra.run.smithi099.stdout: "size": 3, 2023-05-31T20:38:24.754 INFO:teuthology.orchestra.run.smithi099.stdout: "mtime": "2023-05-31T20:38:21.818642Z", 2023-05-31T20:38:24.754 INFO:teuthology.orchestra.run.smithi099.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-05-31T20:38:24.754 INFO:teuthology.orchestra.run.smithi099.stdout: "storage_class": "", 2023-05-31T20:38:24.754 INFO:teuthology.orchestra.run.smithi099.stdout: "owner": "rgw-multisite-test-user", 2023-05-31T20:38:24.754 INFO:teuthology.orchestra.run.smithi099.stdout: "owner_display_name": "Multisite Test User", 2023-05-31T20:38:24.755 INFO:teuthology.orchestra.run.smithi099.stdout: "content_type": "application/octet-stream", 2023-05-31T20:38:24.755 INFO:teuthology.orchestra.run.smithi099.stdout: "accounted_size": 3, 2023-05-31T20:38:24.755 INFO:teuthology.orchestra.run.smithi099.stdout: "user_data": "", 2023-05-31T20:38:24.755 INFO:teuthology.orchestra.run.smithi099.stdout: "appendable": false 2023-05-31T20:38:24.755 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:38:24.755 INFO:teuthology.orchestra.run.smithi099.stdout: "tag": "3d163b1e-09c7-47e2-ae87-add4aace3f38.5680.13901802204008171167", 2023-05-31T20:38:24.755 INFO:teuthology.orchestra.run.smithi099.stdout: "flags": 0, 2023-05-31T20:38:24.756 INFO:teuthology.orchestra.run.smithi099.stdout: "pending_map": [], 2023-05-31T20:38:24.756 INFO:teuthology.orchestra.run.smithi099.stdout: "versioned_epoch": 0 2023-05-31T20:38:24.756 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:38:24.756 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T20:38:24.756 INFO:teuthology.orchestra.run.smithi099.stdout: "name": "obj00000000000000000058", 2023-05-31T20:38:24.756 INFO:teuthology.orchestra.run.smithi099.stdout: "instance": "", 2023-05-31T20:38:24.757 INFO:teuthology.orchestra.run.smithi099.stdout: "ver": { 2023-05-31T20:38:24.757 INFO:teuthology.orchestra.run.smithi099.stdout: "pool": 9, 2023-05-31T20:38:24.757 INFO:teuthology.orchestra.run.smithi099.stdout: "epoch": 5 2023-05-31T20:38:24.757 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:38:24.757 INFO:teuthology.orchestra.run.smithi099.stdout: "locator": "", 2023-05-31T20:38:24.757 INFO:teuthology.orchestra.run.smithi099.stdout: "exists": true, 2023-05-31T20:38:24.757 INFO:teuthology.orchestra.run.smithi099.stdout: "meta": { 2023-05-31T20:38:24.758 INFO:teuthology.orchestra.run.smithi099.stdout: "category": 1, 2023-05-31T20:38:24.758 INFO:teuthology.orchestra.run.smithi099.stdout: "size": 5, 2023-05-31T20:38:24.758 INFO:teuthology.orchestra.run.smithi099.stdout: "mtime": "2023-05-31T20:38:17.631771Z", 2023-05-31T20:38:24.758 INFO:teuthology.orchestra.run.smithi099.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-05-31T20:38:24.758 INFO:teuthology.orchestra.run.smithi099.stdout: "storage_class": "", 2023-05-31T20:38:24.758 INFO:teuthology.orchestra.run.smithi099.stdout: "owner": "rgw-multisite-test-user", 2023-05-31T20:38:24.758 INFO:teuthology.orchestra.run.smithi099.stdout: "owner_display_name": "Multisite Test User", 2023-05-31T20:38:24.759 INFO:teuthology.orchestra.run.smithi099.stdout: "content_type": "application/octet-stream", 2023-05-31T20:38:24.759 INFO:teuthology.orchestra.run.smithi099.stdout: "accounted_size": 3, 2023-05-31T20:38:24.759 INFO:teuthology.orchestra.run.smithi099.stdout: "user_data": "", 2023-05-31T20:38:24.759 INFO:teuthology.orchestra.run.smithi099.stdout: "appendable": false 2023-05-31T20:38:24.759 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:38:24.759 INFO:teuthology.orchestra.run.smithi099.stdout: "tag": "3d163b1e-09c7-47e2-ae87-add4aace3f38.5680.11276341746997562509", 2023-05-31T20:38:24.759 INFO:teuthology.orchestra.run.smithi099.stdout: "flags": 0, 2023-05-31T20:38:24.760 INFO:teuthology.orchestra.run.smithi099.stdout: "pending_map": [], 2023-05-31T20:38:24.760 INFO:teuthology.orchestra.run.smithi099.stdout: "versioned_epoch": 0 2023-05-31T20:38:24.760 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:38:24.760 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T20:38:24.760 INFO:teuthology.orchestra.run.smithi099.stdout: "name": "obj00000000000000000059", 2023-05-31T20:38:24.760 INFO:teuthology.orchestra.run.smithi099.stdout: "instance": "", 2023-05-31T20:38:24.760 INFO:teuthology.orchestra.run.smithi099.stdout: "ver": { 2023-05-31T20:38:24.761 INFO:teuthology.orchestra.run.smithi099.stdout: "pool": 9, 2023-05-31T20:38:24.761 INFO:teuthology.orchestra.run.smithi099.stdout: "epoch": 1 2023-05-31T20:38:24.761 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:38:24.761 INFO:teuthology.orchestra.run.smithi099.stdout: "locator": "", 2023-05-31T20:38:24.761 INFO:teuthology.orchestra.run.smithi099.stdout: "exists": true, 2023-05-31T20:38:24.761 INFO:teuthology.orchestra.run.smithi099.stdout: "meta": { 2023-05-31T20:38:24.761 INFO:teuthology.orchestra.run.smithi099.stdout: "category": 1, 2023-05-31T20:38:24.762 INFO:teuthology.orchestra.run.smithi099.stdout: "size": 3, 2023-05-31T20:38:24.762 INFO:teuthology.orchestra.run.smithi099.stdout: "mtime": "2023-05-31T20:38:18.560629Z", 2023-05-31T20:38:24.762 INFO:teuthology.orchestra.run.smithi099.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-05-31T20:38:24.762 INFO:teuthology.orchestra.run.smithi099.stdout: "storage_class": "", 2023-05-31T20:38:24.763 INFO:teuthology.orchestra.run.smithi099.stdout: "owner": "rgw-multisite-test-user", 2023-05-31T20:38:24.763 INFO:teuthology.orchestra.run.smithi099.stdout: "owner_display_name": "Multisite Test User", 2023-05-31T20:38:24.763 INFO:teuthology.orchestra.run.smithi099.stdout: "content_type": "application/octet-stream", 2023-05-31T20:38:24.763 INFO:teuthology.orchestra.run.smithi099.stdout: "accounted_size": 3, 2023-05-31T20:38:24.763 INFO:teuthology.orchestra.run.smithi099.stdout: "user_data": "", 2023-05-31T20:38:24.763 INFO:teuthology.orchestra.run.smithi099.stdout: "appendable": false 2023-05-31T20:38:24.763 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:38:24.764 INFO:teuthology.orchestra.run.smithi099.stdout: "tag": "3d163b1e-09c7-47e2-ae87-add4aace3f38.5680.12382591306963328340", 2023-05-31T20:38:24.764 INFO:teuthology.orchestra.run.smithi099.stdout: "flags": 0, 2023-05-31T20:38:24.764 INFO:teuthology.orchestra.run.smithi099.stdout: "pending_map": [], 2023-05-31T20:38:24.764 INFO:teuthology.orchestra.run.smithi099.stdout: "versioned_epoch": 0 2023-05-31T20:38:24.764 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:38:24.764 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T20:38:24.764 INFO:teuthology.orchestra.run.smithi099.stdout: "name": "obj00000000000000000060", 2023-05-31T20:38:24.765 INFO:teuthology.orchestra.run.smithi099.stdout: "instance": "", 2023-05-31T20:38:24.765 INFO:teuthology.orchestra.run.smithi099.stdout: "ver": { 2023-05-31T20:38:24.765 INFO:teuthology.orchestra.run.smithi099.stdout: "pool": 9, 2023-05-31T20:38:24.765 INFO:teuthology.orchestra.run.smithi099.stdout: "epoch": 4 2023-05-31T20:38:24.765 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:38:24.765 INFO:teuthology.orchestra.run.smithi099.stdout: "locator": "", 2023-05-31T20:38:24.765 INFO:teuthology.orchestra.run.smithi099.stdout: "exists": true, 2023-05-31T20:38:24.766 INFO:teuthology.orchestra.run.smithi099.stdout: "meta": { 2023-05-31T20:38:24.766 INFO:teuthology.orchestra.run.smithi099.stdout: "category": 1, 2023-05-31T20:38:24.766 INFO:teuthology.orchestra.run.smithi099.stdout: "size": 16, 2023-05-31T20:38:24.766 INFO:teuthology.orchestra.run.smithi099.stdout: "mtime": "2023-05-31T20:38:22.575673Z", 2023-05-31T20:38:24.766 INFO:teuthology.orchestra.run.smithi099.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-05-31T20:38:24.766 INFO:teuthology.orchestra.run.smithi099.stdout: "storage_class": "", 2023-05-31T20:38:24.766 INFO:teuthology.orchestra.run.smithi099.stdout: "owner": "rgw-multisite-test-user", 2023-05-31T20:38:24.767 INFO:teuthology.orchestra.run.smithi099.stdout: "owner_display_name": "Multisite Test User", 2023-05-31T20:38:24.767 INFO:teuthology.orchestra.run.smithi099.stdout: "content_type": "application/octet-stream", 2023-05-31T20:38:24.767 INFO:teuthology.orchestra.run.smithi099.stdout: "accounted_size": 3, 2023-05-31T20:38:24.767 INFO:teuthology.orchestra.run.smithi099.stdout: "user_data": "", 2023-05-31T20:38:24.767 INFO:teuthology.orchestra.run.smithi099.stdout: "appendable": false 2023-05-31T20:38:24.767 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:38:24.768 INFO:teuthology.orchestra.run.smithi099.stdout: "tag": "3d163b1e-09c7-47e2-ae87-add4aace3f38.5680.9608570550353402176", 2023-05-31T20:38:24.768 INFO:teuthology.orchestra.run.smithi099.stdout: "flags": 0, 2023-05-31T20:38:24.768 INFO:teuthology.orchestra.run.smithi099.stdout: "pending_map": [], 2023-05-31T20:38:24.768 INFO:teuthology.orchestra.run.smithi099.stdout: "versioned_epoch": 0 2023-05-31T20:38:24.768 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:38:24.768 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T20:38:24.768 INFO:teuthology.orchestra.run.smithi099.stdout: "name": "obj00000000000000000061", 2023-05-31T20:38:24.769 INFO:teuthology.orchestra.run.smithi099.stdout: "instance": "", 2023-05-31T20:38:24.769 INFO:teuthology.orchestra.run.smithi099.stdout: "ver": { 2023-05-31T20:38:24.769 INFO:teuthology.orchestra.run.smithi099.stdout: "pool": 9, 2023-05-31T20:38:24.769 INFO:teuthology.orchestra.run.smithi099.stdout: "epoch": 4 2023-05-31T20:38:24.769 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:38:24.769 INFO:teuthology.orchestra.run.smithi099.stdout: "locator": "", 2023-05-31T20:38:24.770 INFO:teuthology.orchestra.run.smithi099.stdout: "exists": true, 2023-05-31T20:38:24.770 INFO:teuthology.orchestra.run.smithi099.stdout: "meta": { 2023-05-31T20:38:24.770 INFO:teuthology.orchestra.run.smithi099.stdout: "category": 1, 2023-05-31T20:38:24.770 INFO:teuthology.orchestra.run.smithi099.stdout: "size": 16, 2023-05-31T20:38:24.770 INFO:teuthology.orchestra.run.smithi099.stdout: "mtime": "2023-05-31T20:38:21.154093Z", 2023-05-31T20:38:24.770 INFO:teuthology.orchestra.run.smithi099.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-05-31T20:38:24.770 INFO:teuthology.orchestra.run.smithi099.stdout: "storage_class": "", 2023-05-31T20:38:24.771 INFO:teuthology.orchestra.run.smithi099.stdout: "owner": "rgw-multisite-test-user", 2023-05-31T20:38:24.771 INFO:teuthology.orchestra.run.smithi099.stdout: "owner_display_name": "Multisite Test User", 2023-05-31T20:38:24.771 INFO:teuthology.orchestra.run.smithi099.stdout: "content_type": "application/octet-stream", 2023-05-31T20:38:24.771 INFO:teuthology.orchestra.run.smithi099.stdout: "accounted_size": 3, 2023-05-31T20:38:24.771 INFO:teuthology.orchestra.run.smithi099.stdout: "user_data": "", 2023-05-31T20:38:24.771 INFO:teuthology.orchestra.run.smithi099.stdout: "appendable": false 2023-05-31T20:38:24.771 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:38:24.772 INFO:teuthology.orchestra.run.smithi099.stdout: "tag": "3d163b1e-09c7-47e2-ae87-add4aace3f38.5680.10847143049469854663", 2023-05-31T20:38:24.772 INFO:teuthology.orchestra.run.smithi099.stdout: "flags": 0, 2023-05-31T20:38:24.772 INFO:teuthology.orchestra.run.smithi099.stdout: "pending_map": [], 2023-05-31T20:38:24.772 INFO:teuthology.orchestra.run.smithi099.stdout: "versioned_epoch": 0 2023-05-31T20:38:24.772 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:38:24.772 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T20:38:24.772 INFO:teuthology.orchestra.run.smithi099.stdout: "name": "obj00000000000000000062", 2023-05-31T20:38:24.773 INFO:teuthology.orchestra.run.smithi099.stdout: "instance": "", 2023-05-31T20:38:24.773 INFO:teuthology.orchestra.run.smithi099.stdout: "ver": { 2023-05-31T20:38:24.773 INFO:teuthology.orchestra.run.smithi099.stdout: "pool": 9, 2023-05-31T20:38:24.773 INFO:teuthology.orchestra.run.smithi099.stdout: "epoch": 3 2023-05-31T20:38:24.773 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:38:24.773 INFO:teuthology.orchestra.run.smithi099.stdout: "locator": "", 2023-05-31T20:38:24.773 INFO:teuthology.orchestra.run.smithi099.stdout: "exists": true, 2023-05-31T20:38:24.774 INFO:teuthology.orchestra.run.smithi099.stdout: "meta": { 2023-05-31T20:38:24.774 INFO:teuthology.orchestra.run.smithi099.stdout: "category": 1, 2023-05-31T20:38:24.774 INFO:teuthology.orchestra.run.smithi099.stdout: "size": 16, 2023-05-31T20:38:24.774 INFO:teuthology.orchestra.run.smithi099.stdout: "mtime": "2023-05-31T20:38:21.794173Z", 2023-05-31T20:38:24.774 INFO:teuthology.orchestra.run.smithi099.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-05-31T20:38:24.774 INFO:teuthology.orchestra.run.smithi099.stdout: "storage_class": "", 2023-05-31T20:38:24.774 INFO:teuthology.orchestra.run.smithi099.stdout: "owner": "rgw-multisite-test-user", 2023-05-31T20:38:24.775 INFO:teuthology.orchestra.run.smithi099.stdout: "owner_display_name": "Multisite Test User", 2023-05-31T20:38:24.775 INFO:teuthology.orchestra.run.smithi099.stdout: "content_type": "application/octet-stream", 2023-05-31T20:38:24.775 INFO:teuthology.orchestra.run.smithi099.stdout: "accounted_size": 3, 2023-05-31T20:38:24.775 INFO:teuthology.orchestra.run.smithi099.stdout: "user_data": "", 2023-05-31T20:38:24.775 INFO:teuthology.orchestra.run.smithi099.stdout: "appendable": false 2023-05-31T20:38:24.775 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:38:24.775 INFO:teuthology.orchestra.run.smithi099.stdout: "tag": "3d163b1e-09c7-47e2-ae87-add4aace3f38.5680.10162996169072021833", 2023-05-31T20:38:24.776 INFO:teuthology.orchestra.run.smithi099.stdout: "flags": 0, 2023-05-31T20:38:24.776 INFO:teuthology.orchestra.run.smithi099.stdout: "pending_map": [], 2023-05-31T20:38:24.776 INFO:teuthology.orchestra.run.smithi099.stdout: "versioned_epoch": 0 2023-05-31T20:38:24.776 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:38:24.776 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T20:38:24.776 INFO:teuthology.orchestra.run.smithi099.stdout: "name": "obj00000000000000000065", 2023-05-31T20:38:24.776 INFO:teuthology.orchestra.run.smithi099.stdout: "instance": "", 2023-05-31T20:38:24.777 INFO:teuthology.orchestra.run.smithi099.stdout: "ver": { 2023-05-31T20:38:24.777 INFO:teuthology.orchestra.run.smithi099.stdout: "pool": 9, 2023-05-31T20:38:24.777 INFO:teuthology.orchestra.run.smithi099.stdout: "epoch": 9 2023-05-31T20:38:24.777 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:38:24.777 INFO:teuthology.orchestra.run.smithi099.stdout: "locator": "", 2023-05-31T20:38:24.777 INFO:teuthology.orchestra.run.smithi099.stdout: "exists": true, 2023-05-31T20:38:24.777 INFO:teuthology.orchestra.run.smithi099.stdout: "meta": { 2023-05-31T20:38:24.778 INFO:teuthology.orchestra.run.smithi099.stdout: "category": 1, 2023-05-31T20:38:24.778 INFO:teuthology.orchestra.run.smithi099.stdout: "size": 5, 2023-05-31T20:38:24.778 INFO:teuthology.orchestra.run.smithi099.stdout: "mtime": "2023-05-31T20:38:22.003695Z", 2023-05-31T20:38:24.778 INFO:teuthology.orchestra.run.smithi099.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-05-31T20:38:24.778 INFO:teuthology.orchestra.run.smithi099.stdout: "storage_class": "", 2023-05-31T20:38:24.778 INFO:teuthology.orchestra.run.smithi099.stdout: "owner": "rgw-multisite-test-user", 2023-05-31T20:38:24.779 INFO:teuthology.orchestra.run.smithi099.stdout: "owner_display_name": "Multisite Test User", 2023-05-31T20:38:24.779 INFO:teuthology.orchestra.run.smithi099.stdout: "content_type": "application/octet-stream", 2023-05-31T20:38:24.779 INFO:teuthology.orchestra.run.smithi099.stdout: "accounted_size": 3, 2023-05-31T20:38:24.779 INFO:teuthology.orchestra.run.smithi099.stdout: "user_data": "", 2023-05-31T20:38:24.779 INFO:teuthology.orchestra.run.smithi099.stdout: "appendable": false 2023-05-31T20:38:24.779 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:38:24.779 INFO:teuthology.orchestra.run.smithi099.stdout: "tag": "3d163b1e-09c7-47e2-ae87-add4aace3f38.5680.9559604559291381944", 2023-05-31T20:38:24.780 INFO:teuthology.orchestra.run.smithi099.stdout: "flags": 0, 2023-05-31T20:38:24.780 INFO:teuthology.orchestra.run.smithi099.stdout: "pending_map": [], 2023-05-31T20:38:24.780 INFO:teuthology.orchestra.run.smithi099.stdout: "versioned_epoch": 0 2023-05-31T20:38:24.780 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:38:24.780 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T20:38:24.780 INFO:teuthology.orchestra.run.smithi099.stdout: "name": "obj00000000000000000133", 2023-05-31T20:38:24.780 INFO:teuthology.orchestra.run.smithi099.stdout: "instance": "", 2023-05-31T20:38:24.781 INFO:teuthology.orchestra.run.smithi099.stdout: "ver": { 2023-05-31T20:38:24.781 INFO:teuthology.orchestra.run.smithi099.stdout: "pool": 9, 2023-05-31T20:38:24.781 INFO:teuthology.orchestra.run.smithi099.stdout: "epoch": 11 2023-05-31T20:38:24.781 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:38:24.781 INFO:teuthology.orchestra.run.smithi099.stdout: "locator": "", 2023-05-31T20:38:24.781 INFO:teuthology.orchestra.run.smithi099.stdout: "exists": true, 2023-05-31T20:38:24.781 INFO:teuthology.orchestra.run.smithi099.stdout: "meta": { 2023-05-31T20:38:24.782 INFO:teuthology.orchestra.run.smithi099.stdout: "category": 1, 2023-05-31T20:38:24.782 INFO:teuthology.orchestra.run.smithi099.stdout: "size": 16, 2023-05-31T20:38:24.782 INFO:teuthology.orchestra.run.smithi099.stdout: "mtime": "2023-05-31T20:38:19.997509Z", 2023-05-31T20:38:24.782 INFO:teuthology.orchestra.run.smithi099.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-05-31T20:38:24.782 INFO:teuthology.orchestra.run.smithi099.stdout: "storage_class": "", 2023-05-31T20:38:24.782 INFO:teuthology.orchestra.run.smithi099.stdout: "owner": "rgw-multisite-test-user", 2023-05-31T20:38:24.782 INFO:teuthology.orchestra.run.smithi099.stdout: "owner_display_name": "Multisite Test User", 2023-05-31T20:38:24.783 INFO:teuthology.orchestra.run.smithi099.stdout: "content_type": "application/octet-stream", 2023-05-31T20:38:24.783 INFO:teuthology.orchestra.run.smithi099.stdout: "accounted_size": 3, 2023-05-31T20:38:24.783 INFO:teuthology.orchestra.run.smithi099.stdout: "user_data": "", 2023-05-31T20:38:24.783 INFO:teuthology.orchestra.run.smithi099.stdout: "appendable": false 2023-05-31T20:38:24.783 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:38:24.783 INFO:teuthology.orchestra.run.smithi099.stdout: "tag": "3d163b1e-09c7-47e2-ae87-add4aace3f38.5680.17116436035374276199", 2023-05-31T20:38:24.783 INFO:teuthology.orchestra.run.smithi099.stdout: "flags": 0, 2023-05-31T20:38:24.784 INFO:teuthology.orchestra.run.smithi099.stdout: "pending_map": [], 2023-05-31T20:38:24.784 INFO:teuthology.orchestra.run.smithi099.stdout: "versioned_epoch": 0 2023-05-31T20:38:24.784 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:38:24.784 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T20:38:24.784 INFO:teuthology.orchestra.run.smithi099.stdout: "name": "obj00000000000000000162", 2023-05-31T20:38:24.784 INFO:teuthology.orchestra.run.smithi099.stdout: "instance": "", 2023-05-31T20:38:24.784 INFO:teuthology.orchestra.run.smithi099.stdout: "ver": { 2023-05-31T20:38:24.785 INFO:teuthology.orchestra.run.smithi099.stdout: "pool": 9, 2023-05-31T20:38:24.785 INFO:teuthology.orchestra.run.smithi099.stdout: "epoch": 7 2023-05-31T20:38:24.785 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:38:24.785 INFO:teuthology.orchestra.run.smithi099.stdout: "locator": "", 2023-05-31T20:38:24.785 INFO:teuthology.orchestra.run.smithi099.stdout: "exists": true, 2023-05-31T20:38:24.785 INFO:teuthology.orchestra.run.smithi099.stdout: "meta": { 2023-05-31T20:38:24.785 INFO:teuthology.orchestra.run.smithi099.stdout: "category": 1, 2023-05-31T20:38:24.786 INFO:teuthology.orchestra.run.smithi099.stdout: "size": 16, 2023-05-31T20:38:24.786 INFO:teuthology.orchestra.run.smithi099.stdout: "mtime": "2023-05-31T20:38:20.538625Z", 2023-05-31T20:38:24.786 INFO:teuthology.orchestra.run.smithi099.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-05-31T20:38:24.786 INFO:teuthology.orchestra.run.smithi099.stdout: "storage_class": "", 2023-05-31T20:38:24.786 INFO:teuthology.orchestra.run.smithi099.stdout: "owner": "rgw-multisite-test-user", 2023-05-31T20:38:24.786 INFO:teuthology.orchestra.run.smithi099.stdout: "owner_display_name": "Multisite Test User", 2023-05-31T20:38:24.786 INFO:teuthology.orchestra.run.smithi099.stdout: "content_type": "application/octet-stream", 2023-05-31T20:38:24.787 INFO:teuthology.orchestra.run.smithi099.stdout: "accounted_size": 3, 2023-05-31T20:38:24.787 INFO:teuthology.orchestra.run.smithi099.stdout: "user_data": "", 2023-05-31T20:38:24.787 INFO:teuthology.orchestra.run.smithi099.stdout: "appendable": false 2023-05-31T20:38:24.787 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:38:24.787 INFO:teuthology.orchestra.run.smithi099.stdout: "tag": "3d163b1e-09c7-47e2-ae87-add4aace3f38.5680.12523660392684490418", 2023-05-31T20:38:24.787 INFO:teuthology.orchestra.run.smithi099.stdout: "flags": 0, 2023-05-31T20:38:24.787 INFO:teuthology.orchestra.run.smithi099.stdout: "pending_map": [], 2023-05-31T20:38:24.788 INFO:teuthology.orchestra.run.smithi099.stdout: "versioned_epoch": 0 2023-05-31T20:38:24.788 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:38:24.788 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T20:38:24.788 INFO:teuthology.orchestra.run.smithi099.stdout: "name": "obj00000000000000000271", 2023-05-31T20:38:24.788 INFO:teuthology.orchestra.run.smithi099.stdout: "instance": "", 2023-05-31T20:38:24.788 INFO:teuthology.orchestra.run.smithi099.stdout: "ver": { 2023-05-31T20:38:24.789 INFO:teuthology.orchestra.run.smithi099.stdout: "pool": 9, 2023-05-31T20:38:24.789 INFO:teuthology.orchestra.run.smithi099.stdout: "epoch": 7 2023-05-31T20:38:24.789 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:38:24.789 INFO:teuthology.orchestra.run.smithi099.stdout: "locator": "", 2023-05-31T20:38:24.789 INFO:teuthology.orchestra.run.smithi099.stdout: "exists": true, 2023-05-31T20:38:24.789 INFO:teuthology.orchestra.run.smithi099.stdout: "meta": { 2023-05-31T20:38:24.789 INFO:teuthology.orchestra.run.smithi099.stdout: "category": 1, 2023-05-31T20:38:24.790 INFO:teuthology.orchestra.run.smithi099.stdout: "size": 6, 2023-05-31T20:38:24.790 INFO:teuthology.orchestra.run.smithi099.stdout: "mtime": "2023-05-31T20:38:20.240943Z", 2023-05-31T20:38:24.790 INFO:teuthology.orchestra.run.smithi099.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-05-31T20:38:24.790 INFO:teuthology.orchestra.run.smithi099.stdout: "storage_class": "", 2023-05-31T20:38:24.790 INFO:teuthology.orchestra.run.smithi099.stdout: "owner": "rgw-multisite-test-user", 2023-05-31T20:38:24.790 INFO:teuthology.orchestra.run.smithi099.stdout: "owner_display_name": "Multisite Test User", 2023-05-31T20:38:24.790 INFO:teuthology.orchestra.run.smithi099.stdout: "content_type": "application/octet-stream", 2023-05-31T20:38:24.791 INFO:teuthology.orchestra.run.smithi099.stdout: "accounted_size": 3, 2023-05-31T20:38:24.791 INFO:teuthology.orchestra.run.smithi099.stdout: "user_data": "", 2023-05-31T20:38:24.791 INFO:teuthology.orchestra.run.smithi099.stdout: "appendable": false 2023-05-31T20:38:24.791 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:38:24.791 INFO:teuthology.orchestra.run.smithi099.stdout: "tag": "3d163b1e-09c7-47e2-ae87-add4aace3f38.5680.11852424615725953232", 2023-05-31T20:38:24.791 INFO:teuthology.orchestra.run.smithi099.stdout: "flags": 0, 2023-05-31T20:38:24.791 INFO:teuthology.orchestra.run.smithi099.stdout: "pending_map": [], 2023-05-31T20:38:24.792 INFO:teuthology.orchestra.run.smithi099.stdout: "versioned_epoch": 0 2023-05-31T20:38:24.792 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:38:24.792 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T20:38:24.792 INFO:teuthology.orchestra.run.smithi099.stdout: "name": "obj00000000000000000283", 2023-05-31T20:38:24.792 INFO:teuthology.orchestra.run.smithi099.stdout: "instance": "", 2023-05-31T20:38:24.792 INFO:teuthology.orchestra.run.smithi099.stdout: "ver": { 2023-05-31T20:38:24.792 INFO:teuthology.orchestra.run.smithi099.stdout: "pool": 9, 2023-05-31T20:38:24.793 INFO:teuthology.orchestra.run.smithi099.stdout: "epoch": 12 2023-05-31T20:38:24.793 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:38:24.793 INFO:teuthology.orchestra.run.smithi099.stdout: "locator": "", 2023-05-31T20:38:24.793 INFO:teuthology.orchestra.run.smithi099.stdout: "exists": true, 2023-05-31T20:38:24.793 INFO:teuthology.orchestra.run.smithi099.stdout: "meta": { 2023-05-31T20:38:24.793 INFO:teuthology.orchestra.run.smithi099.stdout: "category": 1, 2023-05-31T20:38:24.793 INFO:teuthology.orchestra.run.smithi099.stdout: "size": 6, 2023-05-31T20:38:24.794 INFO:teuthology.orchestra.run.smithi099.stdout: "mtime": "2023-05-31T20:38:20.953383Z", 2023-05-31T20:38:24.794 INFO:teuthology.orchestra.run.smithi099.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-05-31T20:38:24.794 INFO:teuthology.orchestra.run.smithi099.stdout: "storage_class": "", 2023-05-31T20:38:24.794 INFO:teuthology.orchestra.run.smithi099.stdout: "owner": "rgw-multisite-test-user", 2023-05-31T20:38:24.794 INFO:teuthology.orchestra.run.smithi099.stdout: "owner_display_name": "Multisite Test User", 2023-05-31T20:38:24.794 INFO:teuthology.orchestra.run.smithi099.stdout: "content_type": "application/octet-stream", 2023-05-31T20:38:24.794 INFO:teuthology.orchestra.run.smithi099.stdout: "accounted_size": 3, 2023-05-31T20:38:24.795 INFO:teuthology.orchestra.run.smithi099.stdout: "user_data": "", 2023-05-31T20:38:24.795 INFO:teuthology.orchestra.run.smithi099.stdout: "appendable": false 2023-05-31T20:38:24.795 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:38:24.795 INFO:teuthology.orchestra.run.smithi099.stdout: "tag": "3d163b1e-09c7-47e2-ae87-add4aace3f38.5680.15954905546661512251", 2023-05-31T20:38:24.795 INFO:teuthology.orchestra.run.smithi099.stdout: "flags": 0, 2023-05-31T20:38:24.795 INFO:teuthology.orchestra.run.smithi099.stdout: "pending_map": [], 2023-05-31T20:38:24.795 INFO:teuthology.orchestra.run.smithi099.stdout: "versioned_epoch": 0 2023-05-31T20:38:24.796 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:38:24.796 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T20:38:24.796 INFO:teuthology.orchestra.run.smithi099.stdout: "name": "obj00000000000000000317", 2023-05-31T20:38:24.796 INFO:teuthology.orchestra.run.smithi099.stdout: "instance": "", 2023-05-31T20:38:24.796 INFO:teuthology.orchestra.run.smithi099.stdout: "ver": { 2023-05-31T20:38:24.796 INFO:teuthology.orchestra.run.smithi099.stdout: "pool": 9, 2023-05-31T20:38:24.797 INFO:teuthology.orchestra.run.smithi099.stdout: "epoch": 2 2023-05-31T20:38:24.797 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:38:24.797 INFO:teuthology.orchestra.run.smithi099.stdout: "locator": "", 2023-05-31T20:38:24.797 INFO:teuthology.orchestra.run.smithi099.stdout: "exists": true, 2023-05-31T20:38:24.797 INFO:teuthology.orchestra.run.smithi099.stdout: "meta": { 2023-05-31T20:38:24.797 INFO:teuthology.orchestra.run.smithi099.stdout: "category": 1, 2023-05-31T20:38:24.797 INFO:teuthology.orchestra.run.smithi099.stdout: "size": 6, 2023-05-31T20:38:24.798 INFO:teuthology.orchestra.run.smithi099.stdout: "mtime": "2023-05-31T20:38:20.065984Z", 2023-05-31T20:38:24.798 INFO:teuthology.orchestra.run.smithi099.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-05-31T20:38:24.798 INFO:teuthology.orchestra.run.smithi099.stdout: "storage_class": "", 2023-05-31T20:38:24.798 INFO:teuthology.orchestra.run.smithi099.stdout: "owner": "rgw-multisite-test-user", 2023-05-31T20:38:24.798 INFO:teuthology.orchestra.run.smithi099.stdout: "owner_display_name": "Multisite Test User", 2023-05-31T20:38:24.798 INFO:teuthology.orchestra.run.smithi099.stdout: "content_type": "application/octet-stream", 2023-05-31T20:38:24.798 INFO:teuthology.orchestra.run.smithi099.stdout: "accounted_size": 3, 2023-05-31T20:38:24.799 INFO:teuthology.orchestra.run.smithi099.stdout: "user_data": "", 2023-05-31T20:38:24.799 INFO:teuthology.orchestra.run.smithi099.stdout: "appendable": false 2023-05-31T20:38:24.799 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:38:24.799 INFO:teuthology.orchestra.run.smithi099.stdout: "tag": "3d163b1e-09c7-47e2-ae87-add4aace3f38.5680.4618880714033078935", 2023-05-31T20:38:24.799 INFO:teuthology.orchestra.run.smithi099.stdout: "flags": 0, 2023-05-31T20:38:24.799 INFO:teuthology.orchestra.run.smithi099.stdout: "pending_map": [], 2023-05-31T20:38:24.800 INFO:teuthology.orchestra.run.smithi099.stdout: "versioned_epoch": 0 2023-05-31T20:38:24.801 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:38:24.801 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T20:38:24.801 INFO:teuthology.orchestra.run.smithi099.stdout: "name": "obj00000000000000000423", 2023-05-31T20:38:24.801 INFO:teuthology.orchestra.run.smithi099.stdout: "instance": "", 2023-05-31T20:38:24.801 INFO:teuthology.orchestra.run.smithi099.stdout: "ver": { 2023-05-31T20:38:24.801 INFO:teuthology.orchestra.run.smithi099.stdout: "pool": 9, 2023-05-31T20:38:24.802 INFO:teuthology.orchestra.run.smithi099.stdout: "epoch": 2 2023-05-31T20:38:24.802 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:38:24.802 INFO:teuthology.orchestra.run.smithi099.stdout: "locator": "", 2023-05-31T20:38:24.802 INFO:teuthology.orchestra.run.smithi099.stdout: "exists": true, 2023-05-31T20:38:24.802 INFO:teuthology.orchestra.run.smithi099.stdout: "meta": { 2023-05-31T20:38:24.802 INFO:teuthology.orchestra.run.smithi099.stdout: "category": 1, 2023-05-31T20:38:24.803 INFO:teuthology.orchestra.run.smithi099.stdout: "size": 16, 2023-05-31T20:38:24.803 INFO:teuthology.orchestra.run.smithi099.stdout: "mtime": "2023-05-31T20:38:20.434035Z", 2023-05-31T20:38:24.803 INFO:teuthology.orchestra.run.smithi099.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-05-31T20:38:24.803 INFO:teuthology.orchestra.run.smithi099.stdout: "storage_class": "", 2023-05-31T20:38:24.803 INFO:teuthology.orchestra.run.smithi099.stdout: "owner": "rgw-multisite-test-user", 2023-05-31T20:38:24.803 INFO:teuthology.orchestra.run.smithi099.stdout: "owner_display_name": "Multisite Test User", 2023-05-31T20:38:24.803 INFO:teuthology.orchestra.run.smithi099.stdout: "content_type": "application/octet-stream", 2023-05-31T20:38:24.804 INFO:teuthology.orchestra.run.smithi099.stdout: "accounted_size": 3, 2023-05-31T20:38:24.804 INFO:teuthology.orchestra.run.smithi099.stdout: "user_data": "", 2023-05-31T20:38:24.804 INFO:teuthology.orchestra.run.smithi099.stdout: "appendable": false 2023-05-31T20:38:24.804 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:38:24.804 INFO:teuthology.orchestra.run.smithi099.stdout: "tag": "3d163b1e-09c7-47e2-ae87-add4aace3f38.5680.5494250178834674421", 2023-05-31T20:38:24.804 INFO:teuthology.orchestra.run.smithi099.stdout: "flags": 0, 2023-05-31T20:38:24.804 INFO:teuthology.orchestra.run.smithi099.stdout: "pending_map": [], 2023-05-31T20:38:24.805 INFO:teuthology.orchestra.run.smithi099.stdout: "versioned_epoch": 0 2023-05-31T20:38:24.805 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:38:24.805 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T20:38:24.805 INFO:teuthology.orchestra.run.smithi099.stdout: "name": "obj00000000000000000467", 2023-05-31T20:38:24.805 INFO:teuthology.orchestra.run.smithi099.stdout: "instance": "", 2023-05-31T20:38:24.805 INFO:teuthology.orchestra.run.smithi099.stdout: "ver": { 2023-05-31T20:38:24.805 INFO:teuthology.orchestra.run.smithi099.stdout: "pool": 9, 2023-05-31T20:38:24.806 INFO:teuthology.orchestra.run.smithi099.stdout: "epoch": 3 2023-05-31T20:38:24.806 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:38:24.806 INFO:teuthology.orchestra.run.smithi099.stdout: "locator": "", 2023-05-31T20:38:24.806 INFO:teuthology.orchestra.run.smithi099.stdout: "exists": true, 2023-05-31T20:38:24.806 INFO:teuthology.orchestra.run.smithi099.stdout: "meta": { 2023-05-31T20:38:24.806 INFO:teuthology.orchestra.run.smithi099.stdout: "category": 1, 2023-05-31T20:38:24.806 INFO:teuthology.orchestra.run.smithi099.stdout: "size": 3, 2023-05-31T20:38:24.807 INFO:teuthology.orchestra.run.smithi099.stdout: "mtime": "2023-05-31T20:38:20.327690Z", 2023-05-31T20:38:24.807 INFO:teuthology.orchestra.run.smithi099.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-05-31T20:38:24.807 INFO:teuthology.orchestra.run.smithi099.stdout: "storage_class": "", 2023-05-31T20:38:24.807 INFO:teuthology.orchestra.run.smithi099.stdout: "owner": "rgw-multisite-test-user", 2023-05-31T20:38:24.807 INFO:teuthology.orchestra.run.smithi099.stdout: "owner_display_name": "Multisite Test User", 2023-05-31T20:38:24.807 INFO:teuthology.orchestra.run.smithi099.stdout: "content_type": "application/octet-stream", 2023-05-31T20:38:24.808 INFO:teuthology.orchestra.run.smithi099.stdout: "accounted_size": 3, 2023-05-31T20:38:24.808 INFO:teuthology.orchestra.run.smithi099.stdout: "user_data": "", 2023-05-31T20:38:24.808 INFO:teuthology.orchestra.run.smithi099.stdout: "appendable": false 2023-05-31T20:38:24.808 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:38:24.808 INFO:teuthology.orchestra.run.smithi099.stdout: "tag": "3d163b1e-09c7-47e2-ae87-add4aace3f38.5680.9271214773509937514", 2023-05-31T20:38:24.808 INFO:teuthology.orchestra.run.smithi099.stdout: "flags": 0, 2023-05-31T20:38:24.808 INFO:teuthology.orchestra.run.smithi099.stdout: "pending_map": [], 2023-05-31T20:38:24.809 INFO:teuthology.orchestra.run.smithi099.stdout: "versioned_epoch": 0 2023-05-31T20:38:24.809 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T20:38:24.809 INFO:teuthology.orchestra.run.smithi099.stdout:] 2023-05-31T20:38:24.810 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'list', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'pklxkd-58', '--max-entries', '10000', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-31T20:38:24.810 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'list', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'pklxkd-58', '--max-entries', '10000', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-31T20:38:24.811 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket list --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --bucket pklxkd-58 --max-entries 10000 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-05-31T20:38:24.864 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2023-05-31T20:38:24.865 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2023-05-31T20:38:24.900 INFO:teuthology.orchestra.run.smithi099.stdout:[ 2023-05-31T20:38:24.900 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T20:38:24.900 INFO:teuthology.orchestra.run.smithi099.stdout: "name": "obj00000000000000000000", 2023-05-31T20:38:24.901 INFO:teuthology.orchestra.run.smithi099.stdout: "instance": "", 2023-05-31T20:38:24.901 INFO:teuthology.orchestra.run.smithi099.stdout: "ver": { 2023-05-31T20:38:24.901 INFO:teuthology.orchestra.run.smithi099.stdout: "pool": 13, 2023-05-31T20:38:24.901 INFO:teuthology.orchestra.run.smithi099.stdout: "epoch": 4 2023-05-31T20:38:24.901 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:38:24.901 INFO:teuthology.orchestra.run.smithi099.stdout: "locator": "", 2023-05-31T20:38:24.902 INFO:teuthology.orchestra.run.smithi099.stdout: "exists": true, 2023-05-31T20:38:24.902 INFO:teuthology.orchestra.run.smithi099.stdout: "meta": { 2023-05-31T20:38:24.902 INFO:teuthology.orchestra.run.smithi099.stdout: "category": 1, 2023-05-31T20:38:24.902 INFO:teuthology.orchestra.run.smithi099.stdout: "size": 16, 2023-05-31T20:38:24.902 INFO:teuthology.orchestra.run.smithi099.stdout: "mtime": "2023-05-31T20:38:21.972620Z", 2023-05-31T20:38:24.902 INFO:teuthology.orchestra.run.smithi099.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-05-31T20:38:24.902 INFO:teuthology.orchestra.run.smithi099.stdout: "storage_class": "", 2023-05-31T20:38:24.903 INFO:teuthology.orchestra.run.smithi099.stdout: "owner": "rgw-multisite-test-user", 2023-05-31T20:38:24.903 INFO:teuthology.orchestra.run.smithi099.stdout: "owner_display_name": "Multisite Test User", 2023-05-31T20:38:24.903 INFO:teuthology.orchestra.run.smithi099.stdout: "content_type": "application/octet-stream", 2023-05-31T20:38:24.903 INFO:teuthology.orchestra.run.smithi099.stdout: "accounted_size": 3, 2023-05-31T20:38:24.903 INFO:teuthology.orchestra.run.smithi099.stdout: "user_data": "", 2023-05-31T20:38:24.903 INFO:teuthology.orchestra.run.smithi099.stdout: "appendable": false 2023-05-31T20:38:24.904 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:38:24.904 INFO:teuthology.orchestra.run.smithi099.stdout: "tag": "_wMGlBLcNrWzcNs2y6UeXwlQEcSeA7pT", 2023-05-31T20:38:24.904 INFO:teuthology.orchestra.run.smithi099.stdout: "flags": 0, 2023-05-31T20:38:24.904 INFO:teuthology.orchestra.run.smithi099.stdout: "pending_map": [], 2023-05-31T20:38:24.904 INFO:teuthology.orchestra.run.smithi099.stdout: "versioned_epoch": 0 2023-05-31T20:38:24.904 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:38:24.904 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T20:38:24.905 INFO:teuthology.orchestra.run.smithi099.stdout: "name": "obj00000000000000000001", 2023-05-31T20:38:24.905 INFO:teuthology.orchestra.run.smithi099.stdout: "instance": "", 2023-05-31T20:38:24.905 INFO:teuthology.orchestra.run.smithi099.stdout: "ver": { 2023-05-31T20:38:24.905 INFO:teuthology.orchestra.run.smithi099.stdout: "pool": 13, 2023-05-31T20:38:24.905 INFO:teuthology.orchestra.run.smithi099.stdout: "epoch": 3 2023-05-31T20:38:24.905 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:38:24.905 INFO:teuthology.orchestra.run.smithi099.stdout: "locator": "", 2023-05-31T20:38:24.906 INFO:teuthology.orchestra.run.smithi099.stdout: "exists": true, 2023-05-31T20:38:24.906 INFO:teuthology.orchestra.run.smithi099.stdout: "meta": { 2023-05-31T20:38:24.906 INFO:teuthology.orchestra.run.smithi099.stdout: "category": 1, 2023-05-31T20:38:24.906 INFO:teuthology.orchestra.run.smithi099.stdout: "size": 6, 2023-05-31T20:38:24.906 INFO:teuthology.orchestra.run.smithi099.stdout: "mtime": "2023-05-31T20:38:22.169017Z", 2023-05-31T20:38:24.906 INFO:teuthology.orchestra.run.smithi099.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-05-31T20:38:24.906 INFO:teuthology.orchestra.run.smithi099.stdout: "storage_class": "", 2023-05-31T20:38:24.907 INFO:teuthology.orchestra.run.smithi099.stdout: "owner": "rgw-multisite-test-user", 2023-05-31T20:38:24.907 INFO:teuthology.orchestra.run.smithi099.stdout: "owner_display_name": "Multisite Test User", 2023-05-31T20:38:24.907 INFO:teuthology.orchestra.run.smithi099.stdout: "content_type": "application/octet-stream", 2023-05-31T20:38:24.907 INFO:teuthology.orchestra.run.smithi099.stdout: "accounted_size": 3, 2023-05-31T20:38:24.907 INFO:teuthology.orchestra.run.smithi099.stdout: "user_data": "", 2023-05-31T20:38:24.907 INFO:teuthology.orchestra.run.smithi099.stdout: "appendable": false 2023-05-31T20:38:24.907 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:38:24.908 INFO:teuthology.orchestra.run.smithi099.stdout: "tag": "_TCCrSRK27AtEE8gJRIaXZ5ThG24iDkc", 2023-05-31T20:38:24.908 INFO:teuthology.orchestra.run.smithi099.stdout: "flags": 0, 2023-05-31T20:38:24.908 INFO:teuthology.orchestra.run.smithi099.stdout: "pending_map": [], 2023-05-31T20:38:24.908 INFO:teuthology.orchestra.run.smithi099.stdout: "versioned_epoch": 0 2023-05-31T20:38:24.908 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:38:24.908 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T20:38:24.909 INFO:teuthology.orchestra.run.smithi099.stdout: "name": "obj00000000000000000002", 2023-05-31T20:38:24.909 INFO:teuthology.orchestra.run.smithi099.stdout: "instance": "", 2023-05-31T20:38:24.909 INFO:teuthology.orchestra.run.smithi099.stdout: "ver": { 2023-05-31T20:38:24.909 INFO:teuthology.orchestra.run.smithi099.stdout: "pool": 13, 2023-05-31T20:38:24.909 INFO:teuthology.orchestra.run.smithi099.stdout: "epoch": 7 2023-05-31T20:38:24.909 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:38:24.909 INFO:teuthology.orchestra.run.smithi099.stdout: "locator": "", 2023-05-31T20:38:24.910 INFO:teuthology.orchestra.run.smithi099.stdout: "exists": true, 2023-05-31T20:38:24.910 INFO:teuthology.orchestra.run.smithi099.stdout: "meta": { 2023-05-31T20:38:24.910 INFO:teuthology.orchestra.run.smithi099.stdout: "category": 1, 2023-05-31T20:38:24.910 INFO:teuthology.orchestra.run.smithi099.stdout: "size": 6, 2023-05-31T20:38:24.910 INFO:teuthology.orchestra.run.smithi099.stdout: "mtime": "2023-05-31T20:38:22.061689Z", 2023-05-31T20:38:24.910 INFO:teuthology.orchestra.run.smithi099.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-05-31T20:38:24.910 INFO:teuthology.orchestra.run.smithi099.stdout: "storage_class": "", 2023-05-31T20:38:24.911 INFO:teuthology.orchestra.run.smithi099.stdout: "owner": "rgw-multisite-test-user", 2023-05-31T20:38:24.911 INFO:teuthology.orchestra.run.smithi099.stdout: "owner_display_name": "Multisite Test User", 2023-05-31T20:38:24.911 INFO:teuthology.orchestra.run.smithi099.stdout: "content_type": "application/octet-stream", 2023-05-31T20:38:24.911 INFO:teuthology.orchestra.run.smithi099.stdout: "accounted_size": 3, 2023-05-31T20:38:24.911 INFO:teuthology.orchestra.run.smithi099.stdout: "user_data": "", 2023-05-31T20:38:24.911 INFO:teuthology.orchestra.run.smithi099.stdout: "appendable": false 2023-05-31T20:38:24.911 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:38:24.912 INFO:teuthology.orchestra.run.smithi099.stdout: "tag": "_G9ABX_JMIWbh81BW8zlRXl5Qp4YLTFq", 2023-05-31T20:38:24.912 INFO:teuthology.orchestra.run.smithi099.stdout: "flags": 0, 2023-05-31T20:38:24.912 INFO:teuthology.orchestra.run.smithi099.stdout: "pending_map": [], 2023-05-31T20:38:24.912 INFO:teuthology.orchestra.run.smithi099.stdout: "versioned_epoch": 0 2023-05-31T20:38:24.913 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:38:24.913 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T20:38:24.913 INFO:teuthology.orchestra.run.smithi099.stdout: "name": "obj00000000000000000003", 2023-05-31T20:38:24.913 INFO:teuthology.orchestra.run.smithi099.stdout: "instance": "", 2023-05-31T20:38:24.913 INFO:teuthology.orchestra.run.smithi099.stdout: "ver": { 2023-05-31T20:38:24.913 INFO:teuthology.orchestra.run.smithi099.stdout: "pool": 13, 2023-05-31T20:38:24.914 INFO:teuthology.orchestra.run.smithi099.stdout: "epoch": 3 2023-05-31T20:38:24.914 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:38:24.914 INFO:teuthology.orchestra.run.smithi099.stdout: "locator": "", 2023-05-31T20:38:24.914 INFO:teuthology.orchestra.run.smithi099.stdout: "exists": true, 2023-05-31T20:38:24.914 INFO:teuthology.orchestra.run.smithi099.stdout: "meta": { 2023-05-31T20:38:24.914 INFO:teuthology.orchestra.run.smithi099.stdout: "category": 1, 2023-05-31T20:38:24.915 INFO:teuthology.orchestra.run.smithi099.stdout: "size": 5, 2023-05-31T20:38:24.915 INFO:teuthology.orchestra.run.smithi099.stdout: "mtime": "2023-05-31T20:38:21.910544Z", 2023-05-31T20:38:24.915 INFO:teuthology.orchestra.run.smithi099.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-05-31T20:38:24.915 INFO:teuthology.orchestra.run.smithi099.stdout: "storage_class": "", 2023-05-31T20:38:24.915 INFO:teuthology.orchestra.run.smithi099.stdout: "owner": "rgw-multisite-test-user", 2023-05-31T20:38:24.915 INFO:teuthology.orchestra.run.smithi099.stdout: "owner_display_name": "Multisite Test User", 2023-05-31T20:38:24.915 INFO:teuthology.orchestra.run.smithi099.stdout: "content_type": "application/octet-stream", 2023-05-31T20:38:24.916 INFO:teuthology.orchestra.run.smithi099.stdout: "accounted_size": 3, 2023-05-31T20:38:24.916 INFO:teuthology.orchestra.run.smithi099.stdout: "user_data": "", 2023-05-31T20:38:24.916 INFO:teuthology.orchestra.run.smithi099.stdout: "appendable": false 2023-05-31T20:38:24.916 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:38:24.916 INFO:teuthology.orchestra.run.smithi099.stdout: "tag": "_1fQHzHj1p94GyHY_TFtnMPUHjtOuh-s", 2023-05-31T20:38:24.916 INFO:teuthology.orchestra.run.smithi099.stdout: "flags": 0, 2023-05-31T20:38:24.917 INFO:teuthology.orchestra.run.smithi099.stdout: "pending_map": [], 2023-05-31T20:38:24.917 INFO:teuthology.orchestra.run.smithi099.stdout: "versioned_epoch": 0 2023-05-31T20:38:24.917 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:38:24.917 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T20:38:24.917 INFO:teuthology.orchestra.run.smithi099.stdout: "name": "obj00000000000000000004", 2023-05-31T20:38:24.917 INFO:teuthology.orchestra.run.smithi099.stdout: "instance": "", 2023-05-31T20:38:24.917 INFO:teuthology.orchestra.run.smithi099.stdout: "ver": { 2023-05-31T20:38:24.918 INFO:teuthology.orchestra.run.smithi099.stdout: "pool": 13, 2023-05-31T20:38:24.918 INFO:teuthology.orchestra.run.smithi099.stdout: "epoch": 4 2023-05-31T20:38:24.918 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:38:24.918 INFO:teuthology.orchestra.run.smithi099.stdout: "locator": "", 2023-05-31T20:38:24.918 INFO:teuthology.orchestra.run.smithi099.stdout: "exists": true, 2023-05-31T20:38:24.918 INFO:teuthology.orchestra.run.smithi099.stdout: "meta": { 2023-05-31T20:38:24.918 INFO:teuthology.orchestra.run.smithi099.stdout: "category": 1, 2023-05-31T20:38:24.919 INFO:teuthology.orchestra.run.smithi099.stdout: "size": 16, 2023-05-31T20:38:24.919 INFO:teuthology.orchestra.run.smithi099.stdout: "mtime": "2023-05-31T20:38:22.227799Z", 2023-05-31T20:38:24.919 INFO:teuthology.orchestra.run.smithi099.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-05-31T20:38:24.919 INFO:teuthology.orchestra.run.smithi099.stdout: "storage_class": "", 2023-05-31T20:38:24.919 INFO:teuthology.orchestra.run.smithi099.stdout: "owner": "rgw-multisite-test-user", 2023-05-31T20:38:24.919 INFO:teuthology.orchestra.run.smithi099.stdout: "owner_display_name": "Multisite Test User", 2023-05-31T20:38:24.919 INFO:teuthology.orchestra.run.smithi099.stdout: "content_type": "application/octet-stream", 2023-05-31T20:38:24.920 INFO:teuthology.orchestra.run.smithi099.stdout: "accounted_size": 3, 2023-05-31T20:38:24.920 INFO:teuthology.orchestra.run.smithi099.stdout: "user_data": "", 2023-05-31T20:38:24.920 INFO:teuthology.orchestra.run.smithi099.stdout: "appendable": false 2023-05-31T20:38:24.920 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:38:24.920 INFO:teuthology.orchestra.run.smithi099.stdout: "tag": "_MXrHV9n2v6KRERWEOARc_wbqC0rEJPZ", 2023-05-31T20:38:24.920 INFO:teuthology.orchestra.run.smithi099.stdout: "flags": 0, 2023-05-31T20:38:24.920 INFO:teuthology.orchestra.run.smithi099.stdout: "pending_map": [], 2023-05-31T20:38:24.921 INFO:teuthology.orchestra.run.smithi099.stdout: "versioned_epoch": 0 2023-05-31T20:38:24.921 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:38:24.921 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T20:38:24.921 INFO:teuthology.orchestra.run.smithi099.stdout: "name": "obj00000000000000000005", 2023-05-31T20:38:24.921 INFO:teuthology.orchestra.run.smithi099.stdout: "instance": "", 2023-05-31T20:38:24.921 INFO:teuthology.orchestra.run.smithi099.stdout: "ver": { 2023-05-31T20:38:24.921 INFO:teuthology.orchestra.run.smithi099.stdout: "pool": 13, 2023-05-31T20:38:24.922 INFO:teuthology.orchestra.run.smithi099.stdout: "epoch": 2 2023-05-31T20:38:24.922 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:38:24.922 INFO:teuthology.orchestra.run.smithi099.stdout: "locator": "", 2023-05-31T20:38:24.922 INFO:teuthology.orchestra.run.smithi099.stdout: "exists": true, 2023-05-31T20:38:24.922 INFO:teuthology.orchestra.run.smithi099.stdout: "meta": { 2023-05-31T20:38:24.922 INFO:teuthology.orchestra.run.smithi099.stdout: "category": 1, 2023-05-31T20:38:24.922 INFO:teuthology.orchestra.run.smithi099.stdout: "size": 5, 2023-05-31T20:38:24.923 INFO:teuthology.orchestra.run.smithi099.stdout: "mtime": "2023-05-31T20:38:21.709914Z", 2023-05-31T20:38:24.923 INFO:teuthology.orchestra.run.smithi099.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-05-31T20:38:24.923 INFO:teuthology.orchestra.run.smithi099.stdout: "storage_class": "", 2023-05-31T20:38:24.923 INFO:teuthology.orchestra.run.smithi099.stdout: "owner": "rgw-multisite-test-user", 2023-05-31T20:38:24.923 INFO:teuthology.orchestra.run.smithi099.stdout: "owner_display_name": "Multisite Test User", 2023-05-31T20:38:24.923 INFO:teuthology.orchestra.run.smithi099.stdout: "content_type": "application/octet-stream", 2023-05-31T20:38:24.924 INFO:teuthology.orchestra.run.smithi099.stdout: "accounted_size": 3, 2023-05-31T20:38:24.924 INFO:teuthology.orchestra.run.smithi099.stdout: "user_data": "", 2023-05-31T20:38:24.924 INFO:teuthology.orchestra.run.smithi099.stdout: "appendable": false 2023-05-31T20:38:24.924 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:38:24.924 INFO:teuthology.orchestra.run.smithi099.stdout: "tag": "_IsCxE0YzawNe722c9HcSmDkHXbVq7HW", 2023-05-31T20:38:24.924 INFO:teuthology.orchestra.run.smithi099.stdout: "flags": 0, 2023-05-31T20:38:24.924 INFO:teuthology.orchestra.run.smithi099.stdout: "pending_map": [], 2023-05-31T20:38:24.925 INFO:teuthology.orchestra.run.smithi099.stdout: "versioned_epoch": 0 2023-05-31T20:38:24.925 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:38:24.925 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T20:38:24.925 INFO:teuthology.orchestra.run.smithi099.stdout: "name": "obj00000000000000000006", 2023-05-31T20:38:24.925 INFO:teuthology.orchestra.run.smithi099.stdout: "instance": "", 2023-05-31T20:38:24.925 INFO:teuthology.orchestra.run.smithi099.stdout: "ver": { 2023-05-31T20:38:24.925 INFO:teuthology.orchestra.run.smithi099.stdout: "pool": 13, 2023-05-31T20:38:24.926 INFO:teuthology.orchestra.run.smithi099.stdout: "epoch": 5 2023-05-31T20:38:24.926 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:38:24.926 INFO:teuthology.orchestra.run.smithi099.stdout: "locator": "", 2023-05-31T20:38:24.926 INFO:teuthology.orchestra.run.smithi099.stdout: "exists": true, 2023-05-31T20:38:24.926 INFO:teuthology.orchestra.run.smithi099.stdout: "meta": { 2023-05-31T20:38:24.926 INFO:teuthology.orchestra.run.smithi099.stdout: "category": 1, 2023-05-31T20:38:24.926 INFO:teuthology.orchestra.run.smithi099.stdout: "size": 6, 2023-05-31T20:38:24.927 INFO:teuthology.orchestra.run.smithi099.stdout: "mtime": "2023-05-31T20:38:21.735843Z", 2023-05-31T20:38:24.927 INFO:teuthology.orchestra.run.smithi099.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-05-31T20:38:24.927 INFO:teuthology.orchestra.run.smithi099.stdout: "storage_class": "", 2023-05-31T20:38:24.927 INFO:teuthology.orchestra.run.smithi099.stdout: "owner": "rgw-multisite-test-user", 2023-05-31T20:38:24.927 INFO:teuthology.orchestra.run.smithi099.stdout: "owner_display_name": "Multisite Test User", 2023-05-31T20:38:24.927 INFO:teuthology.orchestra.run.smithi099.stdout: "content_type": "application/octet-stream", 2023-05-31T20:38:24.927 INFO:teuthology.orchestra.run.smithi099.stdout: "accounted_size": 3, 2023-05-31T20:38:24.928 INFO:teuthology.orchestra.run.smithi099.stdout: "user_data": "", 2023-05-31T20:38:24.928 INFO:teuthology.orchestra.run.smithi099.stdout: "appendable": false 2023-05-31T20:38:24.928 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:38:24.928 INFO:teuthology.orchestra.run.smithi099.stdout: "tag": "_kZPkXNrXZa62UgmlIZdnYWXwZY4tqmH", 2023-05-31T20:38:24.929 INFO:teuthology.orchestra.run.smithi099.stdout: "flags": 0, 2023-05-31T20:38:24.929 INFO:teuthology.orchestra.run.smithi099.stdout: "pending_map": [], 2023-05-31T20:38:24.929 INFO:teuthology.orchestra.run.smithi099.stdout: "versioned_epoch": 0 2023-05-31T20:38:24.929 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:38:24.929 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T20:38:24.929 INFO:teuthology.orchestra.run.smithi099.stdout: "name": "obj00000000000000000007", 2023-05-31T20:38:24.929 INFO:teuthology.orchestra.run.smithi099.stdout: "instance": "", 2023-05-31T20:38:24.930 INFO:teuthology.orchestra.run.smithi099.stdout: "ver": { 2023-05-31T20:38:24.930 INFO:teuthology.orchestra.run.smithi099.stdout: "pool": 13, 2023-05-31T20:38:24.930 INFO:teuthology.orchestra.run.smithi099.stdout: "epoch": 4 2023-05-31T20:38:24.930 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:38:24.930 INFO:teuthology.orchestra.run.smithi099.stdout: "locator": "", 2023-05-31T20:38:24.930 INFO:teuthology.orchestra.run.smithi099.stdout: "exists": true, 2023-05-31T20:38:24.930 INFO:teuthology.orchestra.run.smithi099.stdout: "meta": { 2023-05-31T20:38:24.931 INFO:teuthology.orchestra.run.smithi099.stdout: "category": 1, 2023-05-31T20:38:24.931 INFO:teuthology.orchestra.run.smithi099.stdout: "size": 3, 2023-05-31T20:38:24.931 INFO:teuthology.orchestra.run.smithi099.stdout: "mtime": "2023-05-31T20:38:22.793692Z", 2023-05-31T20:38:24.931 INFO:teuthology.orchestra.run.smithi099.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-05-31T20:38:24.931 INFO:teuthology.orchestra.run.smithi099.stdout: "storage_class": "", 2023-05-31T20:38:24.931 INFO:teuthology.orchestra.run.smithi099.stdout: "owner": "rgw-multisite-test-user", 2023-05-31T20:38:24.931 INFO:teuthology.orchestra.run.smithi099.stdout: "owner_display_name": "Multisite Test User", 2023-05-31T20:38:24.932 INFO:teuthology.orchestra.run.smithi099.stdout: "content_type": "application/octet-stream", 2023-05-31T20:38:24.932 INFO:teuthology.orchestra.run.smithi099.stdout: "accounted_size": 3, 2023-05-31T20:38:24.932 INFO:teuthology.orchestra.run.smithi099.stdout: "user_data": "", 2023-05-31T20:38:24.932 INFO:teuthology.orchestra.run.smithi099.stdout: "appendable": false 2023-05-31T20:38:24.932 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:38:24.932 INFO:teuthology.orchestra.run.smithi099.stdout: "tag": "_5VFZ1SIrPRdj2U5DzjXHPvZeBOY_Mlq", 2023-05-31T20:38:24.932 INFO:teuthology.orchestra.run.smithi099.stdout: "flags": 0, 2023-05-31T20:38:24.933 INFO:teuthology.orchestra.run.smithi099.stdout: "pending_map": [], 2023-05-31T20:38:24.933 INFO:teuthology.orchestra.run.smithi099.stdout: "versioned_epoch": 0 2023-05-31T20:38:24.933 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:38:24.933 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T20:38:24.933 INFO:teuthology.orchestra.run.smithi099.stdout: "name": "obj00000000000000000008", 2023-05-31T20:38:24.933 INFO:teuthology.orchestra.run.smithi099.stdout: "instance": "", 2023-05-31T20:38:24.934 INFO:teuthology.orchestra.run.smithi099.stdout: "ver": { 2023-05-31T20:38:24.934 INFO:teuthology.orchestra.run.smithi099.stdout: "pool": 13, 2023-05-31T20:38:24.934 INFO:teuthology.orchestra.run.smithi099.stdout: "epoch": 5 2023-05-31T20:38:24.934 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:38:24.934 INFO:teuthology.orchestra.run.smithi099.stdout: "locator": "", 2023-05-31T20:38:24.934 INFO:teuthology.orchestra.run.smithi099.stdout: "exists": true, 2023-05-31T20:38:24.934 INFO:teuthology.orchestra.run.smithi099.stdout: "meta": { 2023-05-31T20:38:24.935 INFO:teuthology.orchestra.run.smithi099.stdout: "category": 1, 2023-05-31T20:38:24.935 INFO:teuthology.orchestra.run.smithi099.stdout: "size": 16, 2023-05-31T20:38:24.935 INFO:teuthology.orchestra.run.smithi099.stdout: "mtime": "2023-05-31T20:38:20.093833Z", 2023-05-31T20:38:24.935 INFO:teuthology.orchestra.run.smithi099.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-05-31T20:38:24.935 INFO:teuthology.orchestra.run.smithi099.stdout: "storage_class": "", 2023-05-31T20:38:24.935 INFO:teuthology.orchestra.run.smithi099.stdout: "owner": "rgw-multisite-test-user", 2023-05-31T20:38:24.935 INFO:teuthology.orchestra.run.smithi099.stdout: "owner_display_name": "Multisite Test User", 2023-05-31T20:38:24.936 INFO:teuthology.orchestra.run.smithi099.stdout: "content_type": "application/octet-stream", 2023-05-31T20:38:24.936 INFO:teuthology.orchestra.run.smithi099.stdout: "accounted_size": 3, 2023-05-31T20:38:24.936 INFO:teuthology.orchestra.run.smithi099.stdout: "user_data": "", 2023-05-31T20:38:24.936 INFO:teuthology.orchestra.run.smithi099.stdout: "appendable": false 2023-05-31T20:38:24.936 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:38:24.937 INFO:teuthology.orchestra.run.smithi099.stdout: "tag": "_jF8scqweHP1ZoBElowasBTuRR4bySib", 2023-05-31T20:38:24.937 INFO:teuthology.orchestra.run.smithi099.stdout: "flags": 0, 2023-05-31T20:38:24.937 INFO:teuthology.orchestra.run.smithi099.stdout: "pending_map": [], 2023-05-31T20:38:24.937 INFO:teuthology.orchestra.run.smithi099.stdout: "versioned_epoch": 0 2023-05-31T20:38:24.937 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:38:24.937 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T20:38:24.937 INFO:teuthology.orchestra.run.smithi099.stdout: "name": "obj00000000000000000009", 2023-05-31T20:38:24.938 INFO:teuthology.orchestra.run.smithi099.stdout: "instance": "", 2023-05-31T20:38:24.938 INFO:teuthology.orchestra.run.smithi099.stdout: "ver": { 2023-05-31T20:38:24.938 INFO:teuthology.orchestra.run.smithi099.stdout: "pool": 13, 2023-05-31T20:38:24.938 INFO:teuthology.orchestra.run.smithi099.stdout: "epoch": 4 2023-05-31T20:38:24.938 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:38:24.938 INFO:teuthology.orchestra.run.smithi099.stdout: "locator": "", 2023-05-31T20:38:24.939 INFO:teuthology.orchestra.run.smithi099.stdout: "exists": true, 2023-05-31T20:38:24.939 INFO:teuthology.orchestra.run.smithi099.stdout: "meta": { 2023-05-31T20:38:24.939 INFO:teuthology.orchestra.run.smithi099.stdout: "category": 1, 2023-05-31T20:38:24.939 INFO:teuthology.orchestra.run.smithi099.stdout: "size": 16, 2023-05-31T20:38:24.939 INFO:teuthology.orchestra.run.smithi099.stdout: "mtime": "2023-05-31T20:38:21.588236Z", 2023-05-31T20:38:24.939 INFO:teuthology.orchestra.run.smithi099.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-05-31T20:38:24.939 INFO:teuthology.orchestra.run.smithi099.stdout: "storage_class": "", 2023-05-31T20:38:24.940 INFO:teuthology.orchestra.run.smithi099.stdout: "owner": "rgw-multisite-test-user", 2023-05-31T20:38:24.940 INFO:teuthology.orchestra.run.smithi099.stdout: "owner_display_name": "Multisite Test User", 2023-05-31T20:38:24.940 INFO:teuthology.orchestra.run.smithi099.stdout: "content_type": "application/octet-stream", 2023-05-31T20:38:24.940 INFO:teuthology.orchestra.run.smithi099.stdout: "accounted_size": 3, 2023-05-31T20:38:24.940 INFO:teuthology.orchestra.run.smithi099.stdout: "user_data": "", 2023-05-31T20:38:24.940 INFO:teuthology.orchestra.run.smithi099.stdout: "appendable": false 2023-05-31T20:38:24.940 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:38:24.941 INFO:teuthology.orchestra.run.smithi099.stdout: "tag": "_CdU68LNJENzb5hKdzYvzy4jwBaaUxpG", 2023-05-31T20:38:24.941 INFO:teuthology.orchestra.run.smithi099.stdout: "flags": 0, 2023-05-31T20:38:24.941 INFO:teuthology.orchestra.run.smithi099.stdout: "pending_map": [], 2023-05-31T20:38:24.941 INFO:teuthology.orchestra.run.smithi099.stdout: "versioned_epoch": 0 2023-05-31T20:38:24.941 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:38:24.941 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T20:38:24.941 INFO:teuthology.orchestra.run.smithi099.stdout: "name": "obj00000000000000000010", 2023-05-31T20:38:24.942 INFO:teuthology.orchestra.run.smithi099.stdout: "instance": "", 2023-05-31T20:38:24.942 INFO:teuthology.orchestra.run.smithi099.stdout: "ver": { 2023-05-31T20:38:24.942 INFO:teuthology.orchestra.run.smithi099.stdout: "pool": 13, 2023-05-31T20:38:24.942 INFO:teuthology.orchestra.run.smithi099.stdout: "epoch": 4 2023-05-31T20:38:24.942 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:38:24.942 INFO:teuthology.orchestra.run.smithi099.stdout: "locator": "", 2023-05-31T20:38:24.943 INFO:teuthology.orchestra.run.smithi099.stdout: "exists": true, 2023-05-31T20:38:24.943 INFO:teuthology.orchestra.run.smithi099.stdout: "meta": { 2023-05-31T20:38:24.943 INFO:teuthology.orchestra.run.smithi099.stdout: "category": 1, 2023-05-31T20:38:24.943 INFO:teuthology.orchestra.run.smithi099.stdout: "size": 16, 2023-05-31T20:38:24.943 INFO:teuthology.orchestra.run.smithi099.stdout: "mtime": "2023-05-31T20:38:21.127706Z", 2023-05-31T20:38:24.943 INFO:teuthology.orchestra.run.smithi099.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-05-31T20:38:24.943 INFO:teuthology.orchestra.run.smithi099.stdout: "storage_class": "", 2023-05-31T20:38:24.944 INFO:teuthology.orchestra.run.smithi099.stdout: "owner": "rgw-multisite-test-user", 2023-05-31T20:38:24.944 INFO:teuthology.orchestra.run.smithi099.stdout: "owner_display_name": "Multisite Test User", 2023-05-31T20:38:24.944 INFO:teuthology.orchestra.run.smithi099.stdout: "content_type": "application/octet-stream", 2023-05-31T20:38:24.944 INFO:teuthology.orchestra.run.smithi099.stdout: "accounted_size": 3, 2023-05-31T20:38:24.944 INFO:teuthology.orchestra.run.smithi099.stdout: "user_data": "", 2023-05-31T20:38:24.944 INFO:teuthology.orchestra.run.smithi099.stdout: "appendable": false 2023-05-31T20:38:24.944 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:38:24.945 INFO:teuthology.orchestra.run.smithi099.stdout: "tag": "_HLvA0Tdg9hjUbJtacqinJ0Lp-vk0y-Q", 2023-05-31T20:38:24.945 INFO:teuthology.orchestra.run.smithi099.stdout: "flags": 0, 2023-05-31T20:38:24.945 INFO:teuthology.orchestra.run.smithi099.stdout: "pending_map": [], 2023-05-31T20:38:24.945 INFO:teuthology.orchestra.run.smithi099.stdout: "versioned_epoch": 0 2023-05-31T20:38:24.945 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:38:24.945 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T20:38:24.945 INFO:teuthology.orchestra.run.smithi099.stdout: "name": "obj00000000000000000011", 2023-05-31T20:38:24.946 INFO:teuthology.orchestra.run.smithi099.stdout: "instance": "", 2023-05-31T20:38:24.946 INFO:teuthology.orchestra.run.smithi099.stdout: "ver": { 2023-05-31T20:38:24.946 INFO:teuthology.orchestra.run.smithi099.stdout: "pool": 13, 2023-05-31T20:38:24.946 INFO:teuthology.orchestra.run.smithi099.stdout: "epoch": 5 2023-05-31T20:38:24.946 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:38:24.946 INFO:teuthology.orchestra.run.smithi099.stdout: "locator": "", 2023-05-31T20:38:24.946 INFO:teuthology.orchestra.run.smithi099.stdout: "exists": true, 2023-05-31T20:38:24.947 INFO:teuthology.orchestra.run.smithi099.stdout: "meta": { 2023-05-31T20:38:24.947 INFO:teuthology.orchestra.run.smithi099.stdout: "category": 1, 2023-05-31T20:38:24.947 INFO:teuthology.orchestra.run.smithi099.stdout: "size": 5, 2023-05-31T20:38:24.947 INFO:teuthology.orchestra.run.smithi099.stdout: "mtime": "2023-05-31T20:38:22.712231Z", 2023-05-31T20:38:24.947 INFO:teuthology.orchestra.run.smithi099.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-05-31T20:38:24.947 INFO:teuthology.orchestra.run.smithi099.stdout: "storage_class": "", 2023-05-31T20:38:24.947 INFO:teuthology.orchestra.run.smithi099.stdout: "owner": "rgw-multisite-test-user", 2023-05-31T20:38:24.948 INFO:teuthology.orchestra.run.smithi099.stdout: "owner_display_name": "Multisite Test User", 2023-05-31T20:38:24.948 INFO:teuthology.orchestra.run.smithi099.stdout: "content_type": "application/octet-stream", 2023-05-31T20:38:24.948 INFO:teuthology.orchestra.run.smithi099.stdout: "accounted_size": 3, 2023-05-31T20:38:24.948 INFO:teuthology.orchestra.run.smithi099.stdout: "user_data": "", 2023-05-31T20:38:24.948 INFO:teuthology.orchestra.run.smithi099.stdout: "appendable": false 2023-05-31T20:38:24.948 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:38:24.948 INFO:teuthology.orchestra.run.smithi099.stdout: "tag": "_SkF5WQXKqJeUQH-YPCOMA7Lti1oulGF", 2023-05-31T20:38:24.949 INFO:teuthology.orchestra.run.smithi099.stdout: "flags": 0, 2023-05-31T20:38:24.949 INFO:teuthology.orchestra.run.smithi099.stdout: "pending_map": [], 2023-05-31T20:38:24.949 INFO:teuthology.orchestra.run.smithi099.stdout: "versioned_epoch": 0 2023-05-31T20:38:24.949 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:38:24.949 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T20:38:24.949 INFO:teuthology.orchestra.run.smithi099.stdout: "name": "obj00000000000000000012", 2023-05-31T20:38:24.949 INFO:teuthology.orchestra.run.smithi099.stdout: "instance": "", 2023-05-31T20:38:24.950 INFO:teuthology.orchestra.run.smithi099.stdout: "ver": { 2023-05-31T20:38:24.950 INFO:teuthology.orchestra.run.smithi099.stdout: "pool": 13, 2023-05-31T20:38:24.950 INFO:teuthology.orchestra.run.smithi099.stdout: "epoch": 4 2023-05-31T20:38:24.950 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:38:24.950 INFO:teuthology.orchestra.run.smithi099.stdout: "locator": "", 2023-05-31T20:38:24.950 INFO:teuthology.orchestra.run.smithi099.stdout: "exists": true, 2023-05-31T20:38:24.950 INFO:teuthology.orchestra.run.smithi099.stdout: "meta": { 2023-05-31T20:38:24.951 INFO:teuthology.orchestra.run.smithi099.stdout: "category": 1, 2023-05-31T20:38:24.951 INFO:teuthology.orchestra.run.smithi099.stdout: "size": 3, 2023-05-31T20:38:24.951 INFO:teuthology.orchestra.run.smithi099.stdout: "mtime": "2023-05-31T20:38:22.308395Z", 2023-05-31T20:38:24.951 INFO:teuthology.orchestra.run.smithi099.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-05-31T20:38:24.951 INFO:teuthology.orchestra.run.smithi099.stdout: "storage_class": "", 2023-05-31T20:38:24.951 INFO:teuthology.orchestra.run.smithi099.stdout: "owner": "rgw-multisite-test-user", 2023-05-31T20:38:24.951 INFO:teuthology.orchestra.run.smithi099.stdout: "owner_display_name": "Multisite Test User", 2023-05-31T20:38:24.952 INFO:teuthology.orchestra.run.smithi099.stdout: "content_type": "application/octet-stream", 2023-05-31T20:38:24.952 INFO:teuthology.orchestra.run.smithi099.stdout: "accounted_size": 3, 2023-05-31T20:38:24.952 INFO:teuthology.orchestra.run.smithi099.stdout: "user_data": "", 2023-05-31T20:38:24.952 INFO:teuthology.orchestra.run.smithi099.stdout: "appendable": false 2023-05-31T20:38:24.952 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:38:24.952 INFO:teuthology.orchestra.run.smithi099.stdout: "tag": "_8r-XtxQ-P65ZxuxMZMLcpLxct_Ykw2Y", 2023-05-31T20:38:24.953 INFO:teuthology.orchestra.run.smithi099.stdout: "flags": 0, 2023-05-31T20:38:24.953 INFO:teuthology.orchestra.run.smithi099.stdout: "pending_map": [], 2023-05-31T20:38:24.953 INFO:teuthology.orchestra.run.smithi099.stdout: "versioned_epoch": 0 2023-05-31T20:38:24.953 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:38:24.953 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T20:38:24.953 INFO:teuthology.orchestra.run.smithi099.stdout: "name": "obj00000000000000000013", 2023-05-31T20:38:24.953 INFO:teuthology.orchestra.run.smithi099.stdout: "instance": "", 2023-05-31T20:38:24.954 INFO:teuthology.orchestra.run.smithi099.stdout: "ver": { 2023-05-31T20:38:24.954 INFO:teuthology.orchestra.run.smithi099.stdout: "pool": 13, 2023-05-31T20:38:24.954 INFO:teuthology.orchestra.run.smithi099.stdout: "epoch": 37 2023-05-31T20:38:24.954 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:38:24.954 INFO:teuthology.orchestra.run.smithi099.stdout: "locator": "", 2023-05-31T20:38:24.954 INFO:teuthology.orchestra.run.smithi099.stdout: "exists": true, 2023-05-31T20:38:24.954 INFO:teuthology.orchestra.run.smithi099.stdout: "meta": { 2023-05-31T20:38:24.955 INFO:teuthology.orchestra.run.smithi099.stdout: "category": 1, 2023-05-31T20:38:24.955 INFO:teuthology.orchestra.run.smithi099.stdout: "size": 6, 2023-05-31T20:38:24.955 INFO:teuthology.orchestra.run.smithi099.stdout: "mtime": "2023-05-31T20:38:22.736493Z", 2023-05-31T20:38:24.955 INFO:teuthology.orchestra.run.smithi099.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-05-31T20:38:24.955 INFO:teuthology.orchestra.run.smithi099.stdout: "storage_class": "", 2023-05-31T20:38:24.955 INFO:teuthology.orchestra.run.smithi099.stdout: "owner": "rgw-multisite-test-user", 2023-05-31T20:38:24.955 INFO:teuthology.orchestra.run.smithi099.stdout: "owner_display_name": "Multisite Test User", 2023-05-31T20:38:24.956 INFO:teuthology.orchestra.run.smithi099.stdout: "content_type": "application/octet-stream", 2023-05-31T20:38:24.956 INFO:teuthology.orchestra.run.smithi099.stdout: "accounted_size": 3, 2023-05-31T20:38:24.956 INFO:teuthology.orchestra.run.smithi099.stdout: "user_data": "", 2023-05-31T20:38:24.956 INFO:teuthology.orchestra.run.smithi099.stdout: "appendable": false 2023-05-31T20:38:24.956 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:38:24.956 INFO:teuthology.orchestra.run.smithi099.stdout: "tag": "_yCZrohVKclRtpUAw922XRzJMsJ2OjcU", 2023-05-31T20:38:24.956 INFO:teuthology.orchestra.run.smithi099.stdout: "flags": 0, 2023-05-31T20:38:24.957 INFO:teuthology.orchestra.run.smithi099.stdout: "pending_map": [], 2023-05-31T20:38:24.957 INFO:teuthology.orchestra.run.smithi099.stdout: "versioned_epoch": 0 2023-05-31T20:38:24.957 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:38:24.957 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T20:38:24.957 INFO:teuthology.orchestra.run.smithi099.stdout: "name": "obj00000000000000000014", 2023-05-31T20:38:24.957 INFO:teuthology.orchestra.run.smithi099.stdout: "instance": "", 2023-05-31T20:38:24.957 INFO:teuthology.orchestra.run.smithi099.stdout: "ver": { 2023-05-31T20:38:24.958 INFO:teuthology.orchestra.run.smithi099.stdout: "pool": 13, 2023-05-31T20:38:24.958 INFO:teuthology.orchestra.run.smithi099.stdout: "epoch": 2 2023-05-31T20:38:24.958 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:38:24.958 INFO:teuthology.orchestra.run.smithi099.stdout: "locator": "", 2023-05-31T20:38:24.958 INFO:teuthology.orchestra.run.smithi099.stdout: "exists": true, 2023-05-31T20:38:24.958 INFO:teuthology.orchestra.run.smithi099.stdout: "meta": { 2023-05-31T20:38:24.958 INFO:teuthology.orchestra.run.smithi099.stdout: "category": 1, 2023-05-31T20:38:24.959 INFO:teuthology.orchestra.run.smithi099.stdout: "size": 6, 2023-05-31T20:38:24.959 INFO:teuthology.orchestra.run.smithi099.stdout: "mtime": "2023-05-31T20:38:22.896668Z", 2023-05-31T20:38:24.959 INFO:teuthology.orchestra.run.smithi099.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-05-31T20:38:24.959 INFO:teuthology.orchestra.run.smithi099.stdout: "storage_class": "", 2023-05-31T20:38:24.959 INFO:teuthology.orchestra.run.smithi099.stdout: "owner": "rgw-multisite-test-user", 2023-05-31T20:38:24.959 INFO:teuthology.orchestra.run.smithi099.stdout: "owner_display_name": "Multisite Test User", 2023-05-31T20:38:24.960 INFO:teuthology.orchestra.run.smithi099.stdout: "content_type": "application/octet-stream", 2023-05-31T20:38:24.960 INFO:teuthology.orchestra.run.smithi099.stdout: "accounted_size": 3, 2023-05-31T20:38:24.960 INFO:teuthology.orchestra.run.smithi099.stdout: "user_data": "", 2023-05-31T20:38:24.960 INFO:teuthology.orchestra.run.smithi099.stdout: "appendable": false 2023-05-31T20:38:24.960 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:38:24.960 INFO:teuthology.orchestra.run.smithi099.stdout: "tag": "_hDpwuEsUV0SCcDVvolBFxTaLqlv0gZ6", 2023-05-31T20:38:24.960 INFO:teuthology.orchestra.run.smithi099.stdout: "flags": 0, 2023-05-31T20:38:24.961 INFO:teuthology.orchestra.run.smithi099.stdout: "pending_map": [], 2023-05-31T20:38:24.961 INFO:teuthology.orchestra.run.smithi099.stdout: "versioned_epoch": 0 2023-05-31T20:38:24.961 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:38:24.961 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T20:38:24.961 INFO:teuthology.orchestra.run.smithi099.stdout: "name": "obj00000000000000000015", 2023-05-31T20:38:24.961 INFO:teuthology.orchestra.run.smithi099.stdout: "instance": "", 2023-05-31T20:38:24.961 INFO:teuthology.orchestra.run.smithi099.stdout: "ver": { 2023-05-31T20:38:24.962 INFO:teuthology.orchestra.run.smithi099.stdout: "pool": 13, 2023-05-31T20:38:24.962 INFO:teuthology.orchestra.run.smithi099.stdout: "epoch": 4 2023-05-31T20:38:24.962 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:38:24.962 INFO:teuthology.orchestra.run.smithi099.stdout: "locator": "", 2023-05-31T20:38:24.962 INFO:teuthology.orchestra.run.smithi099.stdout: "exists": true, 2023-05-31T20:38:24.962 INFO:teuthology.orchestra.run.smithi099.stdout: "meta": { 2023-05-31T20:38:24.962 INFO:teuthology.orchestra.run.smithi099.stdout: "category": 1, 2023-05-31T20:38:24.963 INFO:teuthology.orchestra.run.smithi099.stdout: "size": 5, 2023-05-31T20:38:24.963 INFO:teuthology.orchestra.run.smithi099.stdout: "mtime": "2023-05-31T20:38:22.656305Z", 2023-05-31T20:38:24.963 INFO:teuthology.orchestra.run.smithi099.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-05-31T20:38:24.963 INFO:teuthology.orchestra.run.smithi099.stdout: "storage_class": "", 2023-05-31T20:38:24.963 INFO:teuthology.orchestra.run.smithi099.stdout: "owner": "rgw-multisite-test-user", 2023-05-31T20:38:24.963 INFO:teuthology.orchestra.run.smithi099.stdout: "owner_display_name": "Multisite Test User", 2023-05-31T20:38:24.963 INFO:teuthology.orchestra.run.smithi099.stdout: "content_type": "application/octet-stream", 2023-05-31T20:38:24.964 INFO:teuthology.orchestra.run.smithi099.stdout: "accounted_size": 3, 2023-05-31T20:38:24.964 INFO:teuthology.orchestra.run.smithi099.stdout: "user_data": "", 2023-05-31T20:38:24.964 INFO:teuthology.orchestra.run.smithi099.stdout: "appendable": false 2023-05-31T20:38:24.964 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:38:24.964 INFO:teuthology.orchestra.run.smithi099.stdout: "tag": "_dlAuQih204IpB-upY70CkutKx8aT2Mu", 2023-05-31T20:38:24.964 INFO:teuthology.orchestra.run.smithi099.stdout: "flags": 0, 2023-05-31T20:38:24.964 INFO:teuthology.orchestra.run.smithi099.stdout: "pending_map": [], 2023-05-31T20:38:24.965 INFO:teuthology.orchestra.run.smithi099.stdout: "versioned_epoch": 0 2023-05-31T20:38:24.965 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:38:24.965 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T20:38:24.965 INFO:teuthology.orchestra.run.smithi099.stdout: "name": "obj00000000000000000016", 2023-05-31T20:38:24.965 INFO:teuthology.orchestra.run.smithi099.stdout: "instance": "", 2023-05-31T20:38:24.965 INFO:teuthology.orchestra.run.smithi099.stdout: "ver": { 2023-05-31T20:38:24.966 INFO:teuthology.orchestra.run.smithi099.stdout: "pool": 13, 2023-05-31T20:38:24.966 INFO:teuthology.orchestra.run.smithi099.stdout: "epoch": 5 2023-05-31T20:38:24.966 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:38:24.966 INFO:teuthology.orchestra.run.smithi099.stdout: "locator": "", 2023-05-31T20:38:24.966 INFO:teuthology.orchestra.run.smithi099.stdout: "exists": true, 2023-05-31T20:38:24.966 INFO:teuthology.orchestra.run.smithi099.stdout: "meta": { 2023-05-31T20:38:24.966 INFO:teuthology.orchestra.run.smithi099.stdout: "category": 1, 2023-05-31T20:38:24.967 INFO:teuthology.orchestra.run.smithi099.stdout: "size": 6, 2023-05-31T20:38:24.967 INFO:teuthology.orchestra.run.smithi099.stdout: "mtime": "2023-05-31T20:38:22.140716Z", 2023-05-31T20:38:24.967 INFO:teuthology.orchestra.run.smithi099.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-05-31T20:38:24.967 INFO:teuthology.orchestra.run.smithi099.stdout: "storage_class": "", 2023-05-31T20:38:24.967 INFO:teuthology.orchestra.run.smithi099.stdout: "owner": "rgw-multisite-test-user", 2023-05-31T20:38:24.967 INFO:teuthology.orchestra.run.smithi099.stdout: "owner_display_name": "Multisite Test User", 2023-05-31T20:38:24.967 INFO:teuthology.orchestra.run.smithi099.stdout: "content_type": "application/octet-stream", 2023-05-31T20:38:24.968 INFO:teuthology.orchestra.run.smithi099.stdout: "accounted_size": 3, 2023-05-31T20:38:24.968 INFO:teuthology.orchestra.run.smithi099.stdout: "user_data": "", 2023-05-31T20:38:24.968 INFO:teuthology.orchestra.run.smithi099.stdout: "appendable": false 2023-05-31T20:38:24.968 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:38:24.968 INFO:teuthology.orchestra.run.smithi099.stdout: "tag": "_Sb4-H5YbJHAaTHjVvkf7eeLsuev48ca", 2023-05-31T20:38:24.968 INFO:teuthology.orchestra.run.smithi099.stdout: "flags": 0, 2023-05-31T20:38:24.968 INFO:teuthology.orchestra.run.smithi099.stdout: "pending_map": [], 2023-05-31T20:38:24.969 INFO:teuthology.orchestra.run.smithi099.stdout: "versioned_epoch": 0 2023-05-31T20:38:24.969 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:38:24.969 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T20:38:24.969 INFO:teuthology.orchestra.run.smithi099.stdout: "name": "obj00000000000000000017", 2023-05-31T20:38:24.969 INFO:teuthology.orchestra.run.smithi099.stdout: "instance": "", 2023-05-31T20:38:24.969 INFO:teuthology.orchestra.run.smithi099.stdout: "ver": { 2023-05-31T20:38:24.969 INFO:teuthology.orchestra.run.smithi099.stdout: "pool": 13, 2023-05-31T20:38:24.970 INFO:teuthology.orchestra.run.smithi099.stdout: "epoch": 3 2023-05-31T20:38:24.970 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:38:24.970 INFO:teuthology.orchestra.run.smithi099.stdout: "locator": "", 2023-05-31T20:38:24.970 INFO:teuthology.orchestra.run.smithi099.stdout: "exists": true, 2023-05-31T20:38:24.970 INFO:teuthology.orchestra.run.smithi099.stdout: "meta": { 2023-05-31T20:38:24.970 INFO:teuthology.orchestra.run.smithi099.stdout: "category": 1, 2023-05-31T20:38:24.970 INFO:teuthology.orchestra.run.smithi099.stdout: "size": 5, 2023-05-31T20:38:24.971 INFO:teuthology.orchestra.run.smithi099.stdout: "mtime": "2023-05-31T20:38:22.548865Z", 2023-05-31T20:38:24.971 INFO:teuthology.orchestra.run.smithi099.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-05-31T20:38:24.971 INFO:teuthology.orchestra.run.smithi099.stdout: "storage_class": "", 2023-05-31T20:38:24.971 INFO:teuthology.orchestra.run.smithi099.stdout: "owner": "rgw-multisite-test-user", 2023-05-31T20:38:24.971 INFO:teuthology.orchestra.run.smithi099.stdout: "owner_display_name": "Multisite Test User", 2023-05-31T20:38:24.971 INFO:teuthology.orchestra.run.smithi099.stdout: "content_type": "application/octet-stream", 2023-05-31T20:38:24.971 INFO:teuthology.orchestra.run.smithi099.stdout: "accounted_size": 3, 2023-05-31T20:38:24.972 INFO:teuthology.orchestra.run.smithi099.stdout: "user_data": "", 2023-05-31T20:38:24.972 INFO:teuthology.orchestra.run.smithi099.stdout: "appendable": false 2023-05-31T20:38:24.972 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:38:24.972 INFO:teuthology.orchestra.run.smithi099.stdout: "tag": "_a3fUDVEchpV2Qd6EE-WYLLFXr30jU7N", 2023-05-31T20:38:24.972 INFO:teuthology.orchestra.run.smithi099.stdout: "flags": 0, 2023-05-31T20:38:24.972 INFO:teuthology.orchestra.run.smithi099.stdout: "pending_map": [], 2023-05-31T20:38:24.972 INFO:teuthology.orchestra.run.smithi099.stdout: "versioned_epoch": 0 2023-05-31T20:38:24.973 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:38:24.973 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T20:38:24.973 INFO:teuthology.orchestra.run.smithi099.stdout: "name": "obj00000000000000000018", 2023-05-31T20:38:24.973 INFO:teuthology.orchestra.run.smithi099.stdout: "instance": "", 2023-05-31T20:38:24.973 INFO:teuthology.orchestra.run.smithi099.stdout: "ver": { 2023-05-31T20:38:24.973 INFO:teuthology.orchestra.run.smithi099.stdout: "pool": 13, 2023-05-31T20:38:24.973 INFO:teuthology.orchestra.run.smithi099.stdout: "epoch": 3 2023-05-31T20:38:24.974 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:38:24.974 INFO:teuthology.orchestra.run.smithi099.stdout: "locator": "", 2023-05-31T20:38:24.974 INFO:teuthology.orchestra.run.smithi099.stdout: "exists": true, 2023-05-31T20:38:24.974 INFO:teuthology.orchestra.run.smithi099.stdout: "meta": { 2023-05-31T20:38:24.974 INFO:teuthology.orchestra.run.smithi099.stdout: "category": 1, 2023-05-31T20:38:24.974 INFO:teuthology.orchestra.run.smithi099.stdout: "size": 16, 2023-05-31T20:38:24.974 INFO:teuthology.orchestra.run.smithi099.stdout: "mtime": "2023-05-31T20:38:21.887881Z", 2023-05-31T20:38:24.975 INFO:teuthology.orchestra.run.smithi099.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-05-31T20:38:24.975 INFO:teuthology.orchestra.run.smithi099.stdout: "storage_class": "", 2023-05-31T20:38:24.975 INFO:teuthology.orchestra.run.smithi099.stdout: "owner": "rgw-multisite-test-user", 2023-05-31T20:38:24.975 INFO:teuthology.orchestra.run.smithi099.stdout: "owner_display_name": "Multisite Test User", 2023-05-31T20:38:24.975 INFO:teuthology.orchestra.run.smithi099.stdout: "content_type": "application/octet-stream", 2023-05-31T20:38:24.975 INFO:teuthology.orchestra.run.smithi099.stdout: "accounted_size": 3, 2023-05-31T20:38:24.975 INFO:teuthology.orchestra.run.smithi099.stdout: "user_data": "", 2023-05-31T20:38:24.976 INFO:teuthology.orchestra.run.smithi099.stdout: "appendable": false 2023-05-31T20:38:24.976 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:38:24.976 INFO:teuthology.orchestra.run.smithi099.stdout: "tag": "_BWKwbgIRQLnCUuqULn2M2T_JC0rGgef", 2023-05-31T20:38:24.976 INFO:teuthology.orchestra.run.smithi099.stdout: "flags": 0, 2023-05-31T20:38:24.976 INFO:teuthology.orchestra.run.smithi099.stdout: "pending_map": [], 2023-05-31T20:38:24.976 INFO:teuthology.orchestra.run.smithi099.stdout: "versioned_epoch": 0 2023-05-31T20:38:24.977 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:38:24.977 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T20:38:24.977 INFO:teuthology.orchestra.run.smithi099.stdout: "name": "obj00000000000000000019", 2023-05-31T20:38:24.977 INFO:teuthology.orchestra.run.smithi099.stdout: "instance": "", 2023-05-31T20:38:24.977 INFO:teuthology.orchestra.run.smithi099.stdout: "ver": { 2023-05-31T20:38:24.977 INFO:teuthology.orchestra.run.smithi099.stdout: "pool": 13, 2023-05-31T20:38:24.977 INFO:teuthology.orchestra.run.smithi099.stdout: "epoch": 4 2023-05-31T20:38:24.978 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:38:24.978 INFO:teuthology.orchestra.run.smithi099.stdout: "locator": "", 2023-05-31T20:38:24.978 INFO:teuthology.orchestra.run.smithi099.stdout: "exists": true, 2023-05-31T20:38:24.978 INFO:teuthology.orchestra.run.smithi099.stdout: "meta": { 2023-05-31T20:38:24.978 INFO:teuthology.orchestra.run.smithi099.stdout: "category": 1, 2023-05-31T20:38:24.978 INFO:teuthology.orchestra.run.smithi099.stdout: "size": 5, 2023-05-31T20:38:24.978 INFO:teuthology.orchestra.run.smithi099.stdout: "mtime": "2023-05-31T20:38:21.559227Z", 2023-05-31T20:38:24.979 INFO:teuthology.orchestra.run.smithi099.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-05-31T20:38:24.979 INFO:teuthology.orchestra.run.smithi099.stdout: "storage_class": "", 2023-05-31T20:38:24.979 INFO:teuthology.orchestra.run.smithi099.stdout: "owner": "rgw-multisite-test-user", 2023-05-31T20:38:24.979 INFO:teuthology.orchestra.run.smithi099.stdout: "owner_display_name": "Multisite Test User", 2023-05-31T20:38:24.979 INFO:teuthology.orchestra.run.smithi099.stdout: "content_type": "application/octet-stream", 2023-05-31T20:38:24.979 INFO:teuthology.orchestra.run.smithi099.stdout: "accounted_size": 3, 2023-05-31T20:38:24.979 INFO:teuthology.orchestra.run.smithi099.stdout: "user_data": "", 2023-05-31T20:38:24.980 INFO:teuthology.orchestra.run.smithi099.stdout: "appendable": false 2023-05-31T20:38:24.980 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:38:24.980 INFO:teuthology.orchestra.run.smithi099.stdout: "tag": "_9onqz8SdyhnywUFE-wqBp_bfOsQQmSw", 2023-05-31T20:38:24.980 INFO:teuthology.orchestra.run.smithi099.stdout: "flags": 0, 2023-05-31T20:38:24.980 INFO:teuthology.orchestra.run.smithi099.stdout: "pending_map": [], 2023-05-31T20:38:24.980 INFO:teuthology.orchestra.run.smithi099.stdout: "versioned_epoch": 0 2023-05-31T20:38:24.980 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:38:24.981 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T20:38:24.981 INFO:teuthology.orchestra.run.smithi099.stdout: "name": "obj00000000000000000020", 2023-05-31T20:38:24.981 INFO:teuthology.orchestra.run.smithi099.stdout: "instance": "", 2023-05-31T20:38:24.981 INFO:teuthology.orchestra.run.smithi099.stdout: "ver": { 2023-05-31T20:38:24.981 INFO:teuthology.orchestra.run.smithi099.stdout: "pool": 13, 2023-05-31T20:38:24.981 INFO:teuthology.orchestra.run.smithi099.stdout: "epoch": 3 2023-05-31T20:38:24.981 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:38:24.982 INFO:teuthology.orchestra.run.smithi099.stdout: "locator": "", 2023-05-31T20:38:24.982 INFO:teuthology.orchestra.run.smithi099.stdout: "exists": true, 2023-05-31T20:38:24.982 INFO:teuthology.orchestra.run.smithi099.stdout: "meta": { 2023-05-31T20:38:24.982 INFO:teuthology.orchestra.run.smithi099.stdout: "category": 1, 2023-05-31T20:38:24.982 INFO:teuthology.orchestra.run.smithi099.stdout: "size": 3, 2023-05-31T20:38:24.982 INFO:teuthology.orchestra.run.smithi099.stdout: "mtime": "2023-05-31T20:38:22.465273Z", 2023-05-31T20:38:24.982 INFO:teuthology.orchestra.run.smithi099.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-05-31T20:38:24.983 INFO:teuthology.orchestra.run.smithi099.stdout: "storage_class": "", 2023-05-31T20:38:24.983 INFO:teuthology.orchestra.run.smithi099.stdout: "owner": "rgw-multisite-test-user", 2023-05-31T20:38:24.983 INFO:teuthology.orchestra.run.smithi099.stdout: "owner_display_name": "Multisite Test User", 2023-05-31T20:38:24.983 INFO:teuthology.orchestra.run.smithi099.stdout: "content_type": "application/octet-stream", 2023-05-31T20:38:24.983 INFO:teuthology.orchestra.run.smithi099.stdout: "accounted_size": 3, 2023-05-31T20:38:24.983 INFO:teuthology.orchestra.run.smithi099.stdout: "user_data": "", 2023-05-31T20:38:24.984 INFO:teuthology.orchestra.run.smithi099.stdout: "appendable": false 2023-05-31T20:38:24.984 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:38:24.984 INFO:teuthology.orchestra.run.smithi099.stdout: "tag": "__hxLHzGG2gxs2QwyIj80lbP-rutLJwM", 2023-05-31T20:38:24.984 INFO:teuthology.orchestra.run.smithi099.stdout: "flags": 0, 2023-05-31T20:38:24.984 INFO:teuthology.orchestra.run.smithi099.stdout: "pending_map": [], 2023-05-31T20:38:24.984 INFO:teuthology.orchestra.run.smithi099.stdout: "versioned_epoch": 0 2023-05-31T20:38:24.984 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:38:24.985 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T20:38:24.985 INFO:teuthology.orchestra.run.smithi099.stdout: "name": "obj00000000000000000021", 2023-05-31T20:38:24.985 INFO:teuthology.orchestra.run.smithi099.stdout: "instance": "", 2023-05-31T20:38:24.985 INFO:teuthology.orchestra.run.smithi099.stdout: "ver": { 2023-05-31T20:38:24.985 INFO:teuthology.orchestra.run.smithi099.stdout: "pool": 13, 2023-05-31T20:38:24.985 INFO:teuthology.orchestra.run.smithi099.stdout: "epoch": 4 2023-05-31T20:38:24.985 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:38:24.986 INFO:teuthology.orchestra.run.smithi099.stdout: "locator": "", 2023-05-31T20:38:24.986 INFO:teuthology.orchestra.run.smithi099.stdout: "exists": true, 2023-05-31T20:38:24.986 INFO:teuthology.orchestra.run.smithi099.stdout: "meta": { 2023-05-31T20:38:24.986 INFO:teuthology.orchestra.run.smithi099.stdout: "category": 1, 2023-05-31T20:38:24.986 INFO:teuthology.orchestra.run.smithi099.stdout: "size": 6, 2023-05-31T20:38:24.986 INFO:teuthology.orchestra.run.smithi099.stdout: "mtime": "2023-05-31T20:38:20.802133Z", 2023-05-31T20:38:24.986 INFO:teuthology.orchestra.run.smithi099.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-05-31T20:38:24.987 INFO:teuthology.orchestra.run.smithi099.stdout: "storage_class": "", 2023-05-31T20:38:24.987 INFO:teuthology.orchestra.run.smithi099.stdout: "owner": "rgw-multisite-test-user", 2023-05-31T20:38:24.987 INFO:teuthology.orchestra.run.smithi099.stdout: "owner_display_name": "Multisite Test User", 2023-05-31T20:38:24.987 INFO:teuthology.orchestra.run.smithi099.stdout: "content_type": "application/octet-stream", 2023-05-31T20:38:24.987 INFO:teuthology.orchestra.run.smithi099.stdout: "accounted_size": 3, 2023-05-31T20:38:24.987 INFO:teuthology.orchestra.run.smithi099.stdout: "user_data": "", 2023-05-31T20:38:24.987 INFO:teuthology.orchestra.run.smithi099.stdout: "appendable": false 2023-05-31T20:38:24.988 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:38:24.988 INFO:teuthology.orchestra.run.smithi099.stdout: "tag": "_wILXyFqov9o7AVaFVOtY328wXJvjia_", 2023-05-31T20:38:24.988 INFO:teuthology.orchestra.run.smithi099.stdout: "flags": 0, 2023-05-31T20:38:24.988 INFO:teuthology.orchestra.run.smithi099.stdout: "pending_map": [], 2023-05-31T20:38:24.988 INFO:teuthology.orchestra.run.smithi099.stdout: "versioned_epoch": 0 2023-05-31T20:38:24.988 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:38:24.988 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T20:38:24.989 INFO:teuthology.orchestra.run.smithi099.stdout: "name": "obj00000000000000000022", 2023-05-31T20:38:24.989 INFO:teuthology.orchestra.run.smithi099.stdout: "instance": "", 2023-05-31T20:38:24.989 INFO:teuthology.orchestra.run.smithi099.stdout: "ver": { 2023-05-31T20:38:24.989 INFO:teuthology.orchestra.run.smithi099.stdout: "pool": 13, 2023-05-31T20:38:24.989 INFO:teuthology.orchestra.run.smithi099.stdout: "epoch": 1 2023-05-31T20:38:24.989 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:38:24.989 INFO:teuthology.orchestra.run.smithi099.stdout: "locator": "", 2023-05-31T20:38:24.990 INFO:teuthology.orchestra.run.smithi099.stdout: "exists": true, 2023-05-31T20:38:24.990 INFO:teuthology.orchestra.run.smithi099.stdout: "meta": { 2023-05-31T20:38:24.990 INFO:teuthology.orchestra.run.smithi099.stdout: "category": 1, 2023-05-31T20:38:24.990 INFO:teuthology.orchestra.run.smithi099.stdout: "size": 3, 2023-05-31T20:38:24.990 INFO:teuthology.orchestra.run.smithi099.stdout: "mtime": "2023-05-31T20:38:20.598399Z", 2023-05-31T20:38:24.990 INFO:teuthology.orchestra.run.smithi099.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-05-31T20:38:24.991 INFO:teuthology.orchestra.run.smithi099.stdout: "storage_class": "", 2023-05-31T20:38:24.991 INFO:teuthology.orchestra.run.smithi099.stdout: "owner": "rgw-multisite-test-user", 2023-05-31T20:38:24.991 INFO:teuthology.orchestra.run.smithi099.stdout: "owner_display_name": "Multisite Test User", 2023-05-31T20:38:24.991 INFO:teuthology.orchestra.run.smithi099.stdout: "content_type": "application/octet-stream", 2023-05-31T20:38:24.991 INFO:teuthology.orchestra.run.smithi099.stdout: "accounted_size": 3, 2023-05-31T20:38:24.991 INFO:teuthology.orchestra.run.smithi099.stdout: "user_data": "", 2023-05-31T20:38:24.991 INFO:teuthology.orchestra.run.smithi099.stdout: "appendable": false 2023-05-31T20:38:24.992 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:38:24.992 INFO:teuthology.orchestra.run.smithi099.stdout: "tag": "_-OH5kKmx6ItCmkhpdhcoEztd6Oygx-0", 2023-05-31T20:38:24.992 INFO:teuthology.orchestra.run.smithi099.stdout: "flags": 0, 2023-05-31T20:38:24.992 INFO:teuthology.orchestra.run.smithi099.stdout: "pending_map": [], 2023-05-31T20:38:24.992 INFO:teuthology.orchestra.run.smithi099.stdout: "versioned_epoch": 0 2023-05-31T20:38:24.992 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:38:24.992 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T20:38:24.993 INFO:teuthology.orchestra.run.smithi099.stdout: "name": "obj00000000000000000023", 2023-05-31T20:38:24.993 INFO:teuthology.orchestra.run.smithi099.stdout: "instance": "", 2023-05-31T20:38:24.993 INFO:teuthology.orchestra.run.smithi099.stdout: "ver": { 2023-05-31T20:38:24.993 INFO:teuthology.orchestra.run.smithi099.stdout: "pool": 13, 2023-05-31T20:38:24.993 INFO:teuthology.orchestra.run.smithi099.stdout: "epoch": 4 2023-05-31T20:38:24.993 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:38:24.993 INFO:teuthology.orchestra.run.smithi099.stdout: "locator": "", 2023-05-31T20:38:24.994 INFO:teuthology.orchestra.run.smithi099.stdout: "exists": true, 2023-05-31T20:38:24.994 INFO:teuthology.orchestra.run.smithi099.stdout: "meta": { 2023-05-31T20:38:24.994 INFO:teuthology.orchestra.run.smithi099.stdout: "category": 1, 2023-05-31T20:38:24.994 INFO:teuthology.orchestra.run.smithi099.stdout: "size": 16, 2023-05-31T20:38:24.994 INFO:teuthology.orchestra.run.smithi099.stdout: "mtime": "2023-05-31T20:38:20.347602Z", 2023-05-31T20:38:24.994 INFO:teuthology.orchestra.run.smithi099.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-05-31T20:38:24.994 INFO:teuthology.orchestra.run.smithi099.stdout: "storage_class": "", 2023-05-31T20:38:24.995 INFO:teuthology.orchestra.run.smithi099.stdout: "owner": "rgw-multisite-test-user", 2023-05-31T20:38:24.995 INFO:teuthology.orchestra.run.smithi099.stdout: "owner_display_name": "Multisite Test User", 2023-05-31T20:38:24.995 INFO:teuthology.orchestra.run.smithi099.stdout: "content_type": "application/octet-stream", 2023-05-31T20:38:24.995 INFO:teuthology.orchestra.run.smithi099.stdout: "accounted_size": 3, 2023-05-31T20:38:24.995 INFO:teuthology.orchestra.run.smithi099.stdout: "user_data": "", 2023-05-31T20:38:24.995 INFO:teuthology.orchestra.run.smithi099.stdout: "appendable": false 2023-05-31T20:38:24.995 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:38:24.996 INFO:teuthology.orchestra.run.smithi099.stdout: "tag": "_u8ZaJHeAz274QWZux0T77OWyg1Gn-tt", 2023-05-31T20:38:24.996 INFO:teuthology.orchestra.run.smithi099.stdout: "flags": 0, 2023-05-31T20:38:24.996 INFO:teuthology.orchestra.run.smithi099.stdout: "pending_map": [], 2023-05-31T20:38:24.996 INFO:teuthology.orchestra.run.smithi099.stdout: "versioned_epoch": 0 2023-05-31T20:38:24.996 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:38:24.996 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T20:38:24.996 INFO:teuthology.orchestra.run.smithi099.stdout: "name": "obj00000000000000000024", 2023-05-31T20:38:24.997 INFO:teuthology.orchestra.run.smithi099.stdout: "instance": "", 2023-05-31T20:38:24.997 INFO:teuthology.orchestra.run.smithi099.stdout: "ver": { 2023-05-31T20:38:24.997 INFO:teuthology.orchestra.run.smithi099.stdout: "pool": 13, 2023-05-31T20:38:24.997 INFO:teuthology.orchestra.run.smithi099.stdout: "epoch": 3 2023-05-31T20:38:24.997 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:38:24.997 INFO:teuthology.orchestra.run.smithi099.stdout: "locator": "", 2023-05-31T20:38:24.997 INFO:teuthology.orchestra.run.smithi099.stdout: "exists": true, 2023-05-31T20:38:24.998 INFO:teuthology.orchestra.run.smithi099.stdout: "meta": { 2023-05-31T20:38:24.998 INFO:teuthology.orchestra.run.smithi099.stdout: "category": 1, 2023-05-31T20:38:24.998 INFO:teuthology.orchestra.run.smithi099.stdout: "size": 16, 2023-05-31T20:38:24.998 INFO:teuthology.orchestra.run.smithi099.stdout: "mtime": "2023-05-31T20:38:22.813906Z", 2023-05-31T20:38:24.998 INFO:teuthology.orchestra.run.smithi099.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-05-31T20:38:24.998 INFO:teuthology.orchestra.run.smithi099.stdout: "storage_class": "", 2023-05-31T20:38:24.998 INFO:teuthology.orchestra.run.smithi099.stdout: "owner": "rgw-multisite-test-user", 2023-05-31T20:38:24.999 INFO:teuthology.orchestra.run.smithi099.stdout: "owner_display_name": "Multisite Test User", 2023-05-31T20:38:24.999 INFO:teuthology.orchestra.run.smithi099.stdout: "content_type": "application/octet-stream", 2023-05-31T20:38:24.999 INFO:teuthology.orchestra.run.smithi099.stdout: "accounted_size": 3, 2023-05-31T20:38:24.999 INFO:teuthology.orchestra.run.smithi099.stdout: "user_data": "", 2023-05-31T20:38:24.999 INFO:teuthology.orchestra.run.smithi099.stdout: "appendable": false 2023-05-31T20:38:24.999 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:38:24.999 INFO:teuthology.orchestra.run.smithi099.stdout: "tag": "_YKDhX00lV6lVQI7eNlgNQ9AyuKFeFKL", 2023-05-31T20:38:25.000 INFO:teuthology.orchestra.run.smithi099.stdout: "flags": 0, 2023-05-31T20:38:25.000 INFO:teuthology.orchestra.run.smithi099.stdout: "pending_map": [], 2023-05-31T20:38:25.000 INFO:teuthology.orchestra.run.smithi099.stdout: "versioned_epoch": 0 2023-05-31T20:38:25.000 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:38:25.000 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T20:38:25.000 INFO:teuthology.orchestra.run.smithi099.stdout: "name": "obj00000000000000000025", 2023-05-31T20:38:25.000 INFO:teuthology.orchestra.run.smithi099.stdout: "instance": "", 2023-05-31T20:38:25.001 INFO:teuthology.orchestra.run.smithi099.stdout: "ver": { 2023-05-31T20:38:25.001 INFO:teuthology.orchestra.run.smithi099.stdout: "pool": 13, 2023-05-31T20:38:25.001 INFO:teuthology.orchestra.run.smithi099.stdout: "epoch": 7 2023-05-31T20:38:25.001 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:38:25.001 INFO:teuthology.orchestra.run.smithi099.stdout: "locator": "", 2023-05-31T20:38:25.002 INFO:teuthology.orchestra.run.smithi099.stdout: "exists": true, 2023-05-31T20:38:25.002 INFO:teuthology.orchestra.run.smithi099.stdout: "meta": { 2023-05-31T20:38:25.002 INFO:teuthology.orchestra.run.smithi099.stdout: "category": 1, 2023-05-31T20:38:25.002 INFO:teuthology.orchestra.run.smithi099.stdout: "size": 3, 2023-05-31T20:38:25.002 INFO:teuthology.orchestra.run.smithi099.stdout: "mtime": "2023-05-31T20:38:19.001294Z", 2023-05-31T20:38:25.002 INFO:teuthology.orchestra.run.smithi099.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-05-31T20:38:25.002 INFO:teuthology.orchestra.run.smithi099.stdout: "storage_class": "", 2023-05-31T20:38:25.003 INFO:teuthology.orchestra.run.smithi099.stdout: "owner": "rgw-multisite-test-user", 2023-05-31T20:38:25.003 INFO:teuthology.orchestra.run.smithi099.stdout: "owner_display_name": "Multisite Test User", 2023-05-31T20:38:25.003 INFO:teuthology.orchestra.run.smithi099.stdout: "content_type": "application/octet-stream", 2023-05-31T20:38:25.003 INFO:teuthology.orchestra.run.smithi099.stdout: "accounted_size": 3, 2023-05-31T20:38:25.003 INFO:teuthology.orchestra.run.smithi099.stdout: "user_data": "", 2023-05-31T20:38:25.003 INFO:teuthology.orchestra.run.smithi099.stdout: "appendable": false 2023-05-31T20:38:25.004 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:38:25.004 INFO:teuthology.orchestra.run.smithi099.stdout: "tag": "_LwPsAUvct4Sost7oBUYhwGXGvGyZQOA", 2023-05-31T20:38:25.004 INFO:teuthology.orchestra.run.smithi099.stdout: "flags": 0, 2023-05-31T20:38:25.004 INFO:teuthology.orchestra.run.smithi099.stdout: "pending_map": [], 2023-05-31T20:38:25.004 INFO:teuthology.orchestra.run.smithi099.stdout: "versioned_epoch": 0 2023-05-31T20:38:25.004 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:38:25.004 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T20:38:25.005 INFO:teuthology.orchestra.run.smithi099.stdout: "name": "obj00000000000000000026", 2023-05-31T20:38:25.005 INFO:teuthology.orchestra.run.smithi099.stdout: "instance": "", 2023-05-31T20:38:25.005 INFO:teuthology.orchestra.run.smithi099.stdout: "ver": { 2023-05-31T20:38:25.005 INFO:teuthology.orchestra.run.smithi099.stdout: "pool": 13, 2023-05-31T20:38:25.005 INFO:teuthology.orchestra.run.smithi099.stdout: "epoch": 5 2023-05-31T20:38:25.005 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:38:25.005 INFO:teuthology.orchestra.run.smithi099.stdout: "locator": "", 2023-05-31T20:38:25.006 INFO:teuthology.orchestra.run.smithi099.stdout: "exists": true, 2023-05-31T20:38:25.006 INFO:teuthology.orchestra.run.smithi099.stdout: "meta": { 2023-05-31T20:38:25.006 INFO:teuthology.orchestra.run.smithi099.stdout: "category": 1, 2023-05-31T20:38:25.006 INFO:teuthology.orchestra.run.smithi099.stdout: "size": 3, 2023-05-31T20:38:25.006 INFO:teuthology.orchestra.run.smithi099.stdout: "mtime": "2023-05-31T20:38:22.407332Z", 2023-05-31T20:38:25.006 INFO:teuthology.orchestra.run.smithi099.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-05-31T20:38:25.007 INFO:teuthology.orchestra.run.smithi099.stdout: "storage_class": "", 2023-05-31T20:38:25.007 INFO:teuthology.orchestra.run.smithi099.stdout: "owner": "rgw-multisite-test-user", 2023-05-31T20:38:25.007 INFO:teuthology.orchestra.run.smithi099.stdout: "owner_display_name": "Multisite Test User", 2023-05-31T20:38:25.007 INFO:teuthology.orchestra.run.smithi099.stdout: "content_type": "application/octet-stream", 2023-05-31T20:38:25.007 INFO:teuthology.orchestra.run.smithi099.stdout: "accounted_size": 3, 2023-05-31T20:38:25.007 INFO:teuthology.orchestra.run.smithi099.stdout: "user_data": "", 2023-05-31T20:38:25.007 INFO:teuthology.orchestra.run.smithi099.stdout: "appendable": false 2023-05-31T20:38:25.008 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:38:25.008 INFO:teuthology.orchestra.run.smithi099.stdout: "tag": "_ReSkABHmgZbZT2NY3pXFAw6MaKLGtok", 2023-05-31T20:38:25.008 INFO:teuthology.orchestra.run.smithi099.stdout: "flags": 0, 2023-05-31T20:38:25.008 INFO:teuthology.orchestra.run.smithi099.stdout: "pending_map": [], 2023-05-31T20:38:25.008 INFO:teuthology.orchestra.run.smithi099.stdout: "versioned_epoch": 0 2023-05-31T20:38:25.008 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:38:25.008 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T20:38:25.009 INFO:teuthology.orchestra.run.smithi099.stdout: "name": "obj00000000000000000027", 2023-05-31T20:38:25.009 INFO:teuthology.orchestra.run.smithi099.stdout: "instance": "", 2023-05-31T20:38:25.009 INFO:teuthology.orchestra.run.smithi099.stdout: "ver": { 2023-05-31T20:38:25.009 INFO:teuthology.orchestra.run.smithi099.stdout: "pool": 13, 2023-05-31T20:38:25.009 INFO:teuthology.orchestra.run.smithi099.stdout: "epoch": 5 2023-05-31T20:38:25.009 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:38:25.009 INFO:teuthology.orchestra.run.smithi099.stdout: "locator": "", 2023-05-31T20:38:25.010 INFO:teuthology.orchestra.run.smithi099.stdout: "exists": true, 2023-05-31T20:38:25.010 INFO:teuthology.orchestra.run.smithi099.stdout: "meta": { 2023-05-31T20:38:25.010 INFO:teuthology.orchestra.run.smithi099.stdout: "category": 1, 2023-05-31T20:38:25.010 INFO:teuthology.orchestra.run.smithi099.stdout: "size": 6, 2023-05-31T20:38:25.010 INFO:teuthology.orchestra.run.smithi099.stdout: "mtime": "2023-05-31T20:38:21.615565Z", 2023-05-31T20:38:25.010 INFO:teuthology.orchestra.run.smithi099.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-05-31T20:38:25.011 INFO:teuthology.orchestra.run.smithi099.stdout: "storage_class": "", 2023-05-31T20:38:25.011 INFO:teuthology.orchestra.run.smithi099.stdout: "owner": "rgw-multisite-test-user", 2023-05-31T20:38:25.011 INFO:teuthology.orchestra.run.smithi099.stdout: "owner_display_name": "Multisite Test User", 2023-05-31T20:38:25.011 INFO:teuthology.orchestra.run.smithi099.stdout: "content_type": "application/octet-stream", 2023-05-31T20:38:25.011 INFO:teuthology.orchestra.run.smithi099.stdout: "accounted_size": 3, 2023-05-31T20:38:25.011 INFO:teuthology.orchestra.run.smithi099.stdout: "user_data": "", 2023-05-31T20:38:25.011 INFO:teuthology.orchestra.run.smithi099.stdout: "appendable": false 2023-05-31T20:38:25.012 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:38:25.012 INFO:teuthology.orchestra.run.smithi099.stdout: "tag": "_4xS4db7I0q2Y2zuyWMZapz9ZXxpw_An", 2023-05-31T20:38:25.012 INFO:teuthology.orchestra.run.smithi099.stdout: "flags": 0, 2023-05-31T20:38:25.012 INFO:teuthology.orchestra.run.smithi099.stdout: "pending_map": [], 2023-05-31T20:38:25.012 INFO:teuthology.orchestra.run.smithi099.stdout: "versioned_epoch": 0 2023-05-31T20:38:25.012 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:38:25.012 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T20:38:25.013 INFO:teuthology.orchestra.run.smithi099.stdout: "name": "obj00000000000000000028", 2023-05-31T20:38:25.013 INFO:teuthology.orchestra.run.smithi099.stdout: "instance": "", 2023-05-31T20:38:25.013 INFO:teuthology.orchestra.run.smithi099.stdout: "ver": { 2023-05-31T20:38:25.013 INFO:teuthology.orchestra.run.smithi099.stdout: "pool": 13, 2023-05-31T20:38:25.013 INFO:teuthology.orchestra.run.smithi099.stdout: "epoch": 6 2023-05-31T20:38:25.013 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:38:25.013 INFO:teuthology.orchestra.run.smithi099.stdout: "locator": "", 2023-05-31T20:38:25.014 INFO:teuthology.orchestra.run.smithi099.stdout: "exists": true, 2023-05-31T20:38:25.014 INFO:teuthology.orchestra.run.smithi099.stdout: "meta": { 2023-05-31T20:38:25.014 INFO:teuthology.orchestra.run.smithi099.stdout: "category": 1, 2023-05-31T20:38:25.014 INFO:teuthology.orchestra.run.smithi099.stdout: "size": 3, 2023-05-31T20:38:25.014 INFO:teuthology.orchestra.run.smithi099.stdout: "mtime": "2023-05-31T20:38:15.316632Z", 2023-05-31T20:38:25.014 INFO:teuthology.orchestra.run.smithi099.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-05-31T20:38:25.014 INFO:teuthology.orchestra.run.smithi099.stdout: "storage_class": "", 2023-05-31T20:38:25.015 INFO:teuthology.orchestra.run.smithi099.stdout: "owner": "rgw-multisite-test-user", 2023-05-31T20:38:25.015 INFO:teuthology.orchestra.run.smithi099.stdout: "owner_display_name": "Multisite Test User", 2023-05-31T20:38:25.015 INFO:teuthology.orchestra.run.smithi099.stdout: "content_type": "application/octet-stream", 2023-05-31T20:38:25.015 INFO:teuthology.orchestra.run.smithi099.stdout: "accounted_size": 3, 2023-05-31T20:38:25.015 INFO:teuthology.orchestra.run.smithi099.stdout: "user_data": "", 2023-05-31T20:38:25.015 INFO:teuthology.orchestra.run.smithi099.stdout: "appendable": false 2023-05-31T20:38:25.016 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:38:25.016 INFO:teuthology.orchestra.run.smithi099.stdout: "tag": "_ndRU4STTfXdEfJa3fb4stSC6EwUWVpk", 2023-05-31T20:38:25.016 INFO:teuthology.orchestra.run.smithi099.stdout: "flags": 0, 2023-05-31T20:38:25.016 INFO:teuthology.orchestra.run.smithi099.stdout: "pending_map": [], 2023-05-31T20:38:25.016 INFO:teuthology.orchestra.run.smithi099.stdout: "versioned_epoch": 0 2023-05-31T20:38:25.016 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:38:25.016 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T20:38:25.017 INFO:teuthology.orchestra.run.smithi099.stdout: "name": "obj00000000000000000029", 2023-05-31T20:38:25.017 INFO:teuthology.orchestra.run.smithi099.stdout: "instance": "", 2023-05-31T20:38:25.017 INFO:teuthology.orchestra.run.smithi099.stdout: "ver": { 2023-05-31T20:38:25.017 INFO:teuthology.orchestra.run.smithi099.stdout: "pool": 13, 2023-05-31T20:38:25.017 INFO:teuthology.orchestra.run.smithi099.stdout: "epoch": 9 2023-05-31T20:38:25.017 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:38:25.017 INFO:teuthology.orchestra.run.smithi099.stdout: "locator": "", 2023-05-31T20:38:25.018 INFO:teuthology.orchestra.run.smithi099.stdout: "exists": true, 2023-05-31T20:38:25.018 INFO:teuthology.orchestra.run.smithi099.stdout: "meta": { 2023-05-31T20:38:25.018 INFO:teuthology.orchestra.run.smithi099.stdout: "category": 1, 2023-05-31T20:38:25.018 INFO:teuthology.orchestra.run.smithi099.stdout: "size": 16, 2023-05-31T20:38:25.018 INFO:teuthology.orchestra.run.smithi099.stdout: "mtime": "2023-05-31T20:38:20.265007Z", 2023-05-31T20:38:25.018 INFO:teuthology.orchestra.run.smithi099.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-05-31T20:38:25.018 INFO:teuthology.orchestra.run.smithi099.stdout: "storage_class": "", 2023-05-31T20:38:25.019 INFO:teuthology.orchestra.run.smithi099.stdout: "owner": "rgw-multisite-test-user", 2023-05-31T20:38:25.019 INFO:teuthology.orchestra.run.smithi099.stdout: "owner_display_name": "Multisite Test User", 2023-05-31T20:38:25.019 INFO:teuthology.orchestra.run.smithi099.stdout: "content_type": "application/octet-stream", 2023-05-31T20:38:25.019 INFO:teuthology.orchestra.run.smithi099.stdout: "accounted_size": 3, 2023-05-31T20:38:25.019 INFO:teuthology.orchestra.run.smithi099.stdout: "user_data": "", 2023-05-31T20:38:25.019 INFO:teuthology.orchestra.run.smithi099.stdout: "appendable": false 2023-05-31T20:38:25.019 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:38:25.020 INFO:teuthology.orchestra.run.smithi099.stdout: "tag": "_QwlOKa-9bRHSi_7Dmw8JzytdNGe3czx", 2023-05-31T20:38:25.020 INFO:teuthology.orchestra.run.smithi099.stdout: "flags": 0, 2023-05-31T20:38:25.020 INFO:teuthology.orchestra.run.smithi099.stdout: "pending_map": [], 2023-05-31T20:38:25.020 INFO:teuthology.orchestra.run.smithi099.stdout: "versioned_epoch": 0 2023-05-31T20:38:25.020 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:38:25.020 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T20:38:25.020 INFO:teuthology.orchestra.run.smithi099.stdout: "name": "obj00000000000000000030", 2023-05-31T20:38:25.021 INFO:teuthology.orchestra.run.smithi099.stdout: "instance": "", 2023-05-31T20:38:25.021 INFO:teuthology.orchestra.run.smithi099.stdout: "ver": { 2023-05-31T20:38:25.021 INFO:teuthology.orchestra.run.smithi099.stdout: "pool": 13, 2023-05-31T20:38:25.021 INFO:teuthology.orchestra.run.smithi099.stdout: "epoch": 2 2023-05-31T20:38:25.021 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:38:25.021 INFO:teuthology.orchestra.run.smithi099.stdout: "locator": "", 2023-05-31T20:38:25.022 INFO:teuthology.orchestra.run.smithi099.stdout: "exists": true, 2023-05-31T20:38:25.022 INFO:teuthology.orchestra.run.smithi099.stdout: "meta": { 2023-05-31T20:38:25.022 INFO:teuthology.orchestra.run.smithi099.stdout: "category": 1, 2023-05-31T20:38:25.022 INFO:teuthology.orchestra.run.smithi099.stdout: "size": 16, 2023-05-31T20:38:25.022 INFO:teuthology.orchestra.run.smithi099.stdout: "mtime": "2023-05-31T20:38:21.641789Z", 2023-05-31T20:38:25.022 INFO:teuthology.orchestra.run.smithi099.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-05-31T20:38:25.022 INFO:teuthology.orchestra.run.smithi099.stdout: "storage_class": "", 2023-05-31T20:38:25.023 INFO:teuthology.orchestra.run.smithi099.stdout: "owner": "rgw-multisite-test-user", 2023-05-31T20:38:25.023 INFO:teuthology.orchestra.run.smithi099.stdout: "owner_display_name": "Multisite Test User", 2023-05-31T20:38:25.023 INFO:teuthology.orchestra.run.smithi099.stdout: "content_type": "application/octet-stream", 2023-05-31T20:38:25.023 INFO:teuthology.orchestra.run.smithi099.stdout: "accounted_size": 3, 2023-05-31T20:38:25.023 INFO:teuthology.orchestra.run.smithi099.stdout: "user_data": "", 2023-05-31T20:38:25.023 INFO:teuthology.orchestra.run.smithi099.stdout: "appendable": false 2023-05-31T20:38:25.023 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:38:25.024 INFO:teuthology.orchestra.run.smithi099.stdout: "tag": "_sq-lnamE8ok8eJeToHQFUAj2Q3Pyl4c", 2023-05-31T20:38:25.024 INFO:teuthology.orchestra.run.smithi099.stdout: "flags": 0, 2023-05-31T20:38:25.024 INFO:teuthology.orchestra.run.smithi099.stdout: "pending_map": [], 2023-05-31T20:38:25.024 INFO:teuthology.orchestra.run.smithi099.stdout: "versioned_epoch": 0 2023-05-31T20:38:25.024 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:38:25.024 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T20:38:25.025 INFO:teuthology.orchestra.run.smithi099.stdout: "name": "obj00000000000000000031", 2023-05-31T20:38:25.025 INFO:teuthology.orchestra.run.smithi099.stdout: "instance": "", 2023-05-31T20:38:25.025 INFO:teuthology.orchestra.run.smithi099.stdout: "ver": { 2023-05-31T20:38:25.025 INFO:teuthology.orchestra.run.smithi099.stdout: "pool": 13, 2023-05-31T20:38:25.025 INFO:teuthology.orchestra.run.smithi099.stdout: "epoch": 3 2023-05-31T20:38:25.025 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:38:25.025 INFO:teuthology.orchestra.run.smithi099.stdout: "locator": "", 2023-05-31T20:38:25.026 INFO:teuthology.orchestra.run.smithi099.stdout: "exists": true, 2023-05-31T20:38:25.026 INFO:teuthology.orchestra.run.smithi099.stdout: "meta": { 2023-05-31T20:38:25.026 INFO:teuthology.orchestra.run.smithi099.stdout: "category": 1, 2023-05-31T20:38:25.026 INFO:teuthology.orchestra.run.smithi099.stdout: "size": 16, 2023-05-31T20:38:25.026 INFO:teuthology.orchestra.run.smithi099.stdout: "mtime": "2023-05-31T20:38:18.440797Z", 2023-05-31T20:38:25.026 INFO:teuthology.orchestra.run.smithi099.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-05-31T20:38:25.026 INFO:teuthology.orchestra.run.smithi099.stdout: "storage_class": "", 2023-05-31T20:38:25.027 INFO:teuthology.orchestra.run.smithi099.stdout: "owner": "rgw-multisite-test-user", 2023-05-31T20:38:25.027 INFO:teuthology.orchestra.run.smithi099.stdout: "owner_display_name": "Multisite Test User", 2023-05-31T20:38:25.027 INFO:teuthology.orchestra.run.smithi099.stdout: "content_type": "application/octet-stream", 2023-05-31T20:38:25.027 INFO:teuthology.orchestra.run.smithi099.stdout: "accounted_size": 3, 2023-05-31T20:38:25.027 INFO:teuthology.orchestra.run.smithi099.stdout: "user_data": "", 2023-05-31T20:38:25.027 INFO:teuthology.orchestra.run.smithi099.stdout: "appendable": false 2023-05-31T20:38:25.027 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:38:25.028 INFO:teuthology.orchestra.run.smithi099.stdout: "tag": "_AQph0qWP2DEKmBLqdFWvZnbzuzGyvHD", 2023-05-31T20:38:25.028 INFO:teuthology.orchestra.run.smithi099.stdout: "flags": 0, 2023-05-31T20:38:25.028 INFO:teuthology.orchestra.run.smithi099.stdout: "pending_map": [], 2023-05-31T20:38:25.028 INFO:teuthology.orchestra.run.smithi099.stdout: "versioned_epoch": 0 2023-05-31T20:38:25.028 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:38:25.028 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T20:38:25.028 INFO:teuthology.orchestra.run.smithi099.stdout: "name": "obj00000000000000000032", 2023-05-31T20:38:25.029 INFO:teuthology.orchestra.run.smithi099.stdout: "instance": "", 2023-05-31T20:38:25.029 INFO:teuthology.orchestra.run.smithi099.stdout: "ver": { 2023-05-31T20:38:25.029 INFO:teuthology.orchestra.run.smithi099.stdout: "pool": 13, 2023-05-31T20:38:25.029 INFO:teuthology.orchestra.run.smithi099.stdout: "epoch": 3 2023-05-31T20:38:25.029 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:38:25.029 INFO:teuthology.orchestra.run.smithi099.stdout: "locator": "", 2023-05-31T20:38:25.029 INFO:teuthology.orchestra.run.smithi099.stdout: "exists": true, 2023-05-31T20:38:25.030 INFO:teuthology.orchestra.run.smithi099.stdout: "meta": { 2023-05-31T20:38:25.030 INFO:teuthology.orchestra.run.smithi099.stdout: "category": 1, 2023-05-31T20:38:25.030 INFO:teuthology.orchestra.run.smithi099.stdout: "size": 16, 2023-05-31T20:38:25.030 INFO:teuthology.orchestra.run.smithi099.stdout: "mtime": "2023-05-31T20:38:15.861929Z", 2023-05-31T20:38:25.030 INFO:teuthology.orchestra.run.smithi099.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-05-31T20:38:25.030 INFO:teuthology.orchestra.run.smithi099.stdout: "storage_class": "", 2023-05-31T20:38:25.030 INFO:teuthology.orchestra.run.smithi099.stdout: "owner": "rgw-multisite-test-user", 2023-05-31T20:38:25.031 INFO:teuthology.orchestra.run.smithi099.stdout: "owner_display_name": "Multisite Test User", 2023-05-31T20:38:25.031 INFO:teuthology.orchestra.run.smithi099.stdout: "content_type": "application/octet-stream", 2023-05-31T20:38:25.031 INFO:teuthology.orchestra.run.smithi099.stdout: "accounted_size": 3, 2023-05-31T20:38:25.031 INFO:teuthology.orchestra.run.smithi099.stdout: "user_data": "", 2023-05-31T20:38:25.031 INFO:teuthology.orchestra.run.smithi099.stdout: "appendable": false 2023-05-31T20:38:25.031 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:38:25.032 INFO:teuthology.orchestra.run.smithi099.stdout: "tag": "_lUI8t64BdUbc-W6rS9hwWjKJkKjfaNL", 2023-05-31T20:38:25.032 INFO:teuthology.orchestra.run.smithi099.stdout: "flags": 0, 2023-05-31T20:38:25.032 INFO:teuthology.orchestra.run.smithi099.stdout: "pending_map": [], 2023-05-31T20:38:25.032 INFO:teuthology.orchestra.run.smithi099.stdout: "versioned_epoch": 0 2023-05-31T20:38:25.032 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:38:25.032 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T20:38:25.032 INFO:teuthology.orchestra.run.smithi099.stdout: "name": "obj00000000000000000033", 2023-05-31T20:38:25.033 INFO:teuthology.orchestra.run.smithi099.stdout: "instance": "", 2023-05-31T20:38:25.033 INFO:teuthology.orchestra.run.smithi099.stdout: "ver": { 2023-05-31T20:38:25.033 INFO:teuthology.orchestra.run.smithi099.stdout: "pool": 13, 2023-05-31T20:38:25.033 INFO:teuthology.orchestra.run.smithi099.stdout: "epoch": 8 2023-05-31T20:38:25.033 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:38:25.033 INFO:teuthology.orchestra.run.smithi099.stdout: "locator": "", 2023-05-31T20:38:25.033 INFO:teuthology.orchestra.run.smithi099.stdout: "exists": true, 2023-05-31T20:38:25.034 INFO:teuthology.orchestra.run.smithi099.stdout: "meta": { 2023-05-31T20:38:25.034 INFO:teuthology.orchestra.run.smithi099.stdout: "category": 1, 2023-05-31T20:38:25.034 INFO:teuthology.orchestra.run.smithi099.stdout: "size": 5, 2023-05-31T20:38:25.034 INFO:teuthology.orchestra.run.smithi099.stdout: "mtime": "2023-05-31T20:38:22.084294Z", 2023-05-31T20:38:25.034 INFO:teuthology.orchestra.run.smithi099.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-05-31T20:38:25.034 INFO:teuthology.orchestra.run.smithi099.stdout: "storage_class": "", 2023-05-31T20:38:25.034 INFO:teuthology.orchestra.run.smithi099.stdout: "owner": "rgw-multisite-test-user", 2023-05-31T20:38:25.035 INFO:teuthology.orchestra.run.smithi099.stdout: "owner_display_name": "Multisite Test User", 2023-05-31T20:38:25.035 INFO:teuthology.orchestra.run.smithi099.stdout: "content_type": "application/octet-stream", 2023-05-31T20:38:25.035 INFO:teuthology.orchestra.run.smithi099.stdout: "accounted_size": 3, 2023-05-31T20:38:25.035 INFO:teuthology.orchestra.run.smithi099.stdout: "user_data": "", 2023-05-31T20:38:25.035 INFO:teuthology.orchestra.run.smithi099.stdout: "appendable": false 2023-05-31T20:38:25.035 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:38:25.035 INFO:teuthology.orchestra.run.smithi099.stdout: "tag": "_EBFj_FSC5ZNZADjrICT4ofEi3cuWkS3", 2023-05-31T20:38:25.036 INFO:teuthology.orchestra.run.smithi099.stdout: "flags": 0, 2023-05-31T20:38:25.036 INFO:teuthology.orchestra.run.smithi099.stdout: "pending_map": [], 2023-05-31T20:38:25.036 INFO:teuthology.orchestra.run.smithi099.stdout: "versioned_epoch": 0 2023-05-31T20:38:25.036 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:38:25.036 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T20:38:25.036 INFO:teuthology.orchestra.run.smithi099.stdout: "name": "obj00000000000000000036", 2023-05-31T20:38:25.036 INFO:teuthology.orchestra.run.smithi099.stdout: "instance": "", 2023-05-31T20:38:25.037 INFO:teuthology.orchestra.run.smithi099.stdout: "ver": { 2023-05-31T20:38:25.037 INFO:teuthology.orchestra.run.smithi099.stdout: "pool": 13, 2023-05-31T20:38:25.037 INFO:teuthology.orchestra.run.smithi099.stdout: "epoch": 2 2023-05-31T20:38:25.037 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:38:25.037 INFO:teuthology.orchestra.run.smithi099.stdout: "locator": "", 2023-05-31T20:38:25.037 INFO:teuthology.orchestra.run.smithi099.stdout: "exists": true, 2023-05-31T20:38:25.037 INFO:teuthology.orchestra.run.smithi099.stdout: "meta": { 2023-05-31T20:38:25.038 INFO:teuthology.orchestra.run.smithi099.stdout: "category": 1, 2023-05-31T20:38:25.038 INFO:teuthology.orchestra.run.smithi099.stdout: "size": 5, 2023-05-31T20:38:25.038 INFO:teuthology.orchestra.run.smithi099.stdout: "mtime": "2023-05-31T20:38:22.636301Z", 2023-05-31T20:38:25.038 INFO:teuthology.orchestra.run.smithi099.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-05-31T20:38:25.038 INFO:teuthology.orchestra.run.smithi099.stdout: "storage_class": "", 2023-05-31T20:38:25.038 INFO:teuthology.orchestra.run.smithi099.stdout: "owner": "rgw-multisite-test-user", 2023-05-31T20:38:25.038 INFO:teuthology.orchestra.run.smithi099.stdout: "owner_display_name": "Multisite Test User", 2023-05-31T20:38:25.039 INFO:teuthology.orchestra.run.smithi099.stdout: "content_type": "application/octet-stream", 2023-05-31T20:38:25.039 INFO:teuthology.orchestra.run.smithi099.stdout: "accounted_size": 3, 2023-05-31T20:38:25.039 INFO:teuthology.orchestra.run.smithi099.stdout: "user_data": "", 2023-05-31T20:38:25.039 INFO:teuthology.orchestra.run.smithi099.stdout: "appendable": false 2023-05-31T20:38:25.039 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:38:25.039 INFO:teuthology.orchestra.run.smithi099.stdout: "tag": "_ZhoHdU8cz9uqtKuLv8dDtfMEiB609Ao", 2023-05-31T20:38:25.040 INFO:teuthology.orchestra.run.smithi099.stdout: "flags": 0, 2023-05-31T20:38:25.040 INFO:teuthology.orchestra.run.smithi099.stdout: "pending_map": [], 2023-05-31T20:38:25.040 INFO:teuthology.orchestra.run.smithi099.stdout: "versioned_epoch": 0 2023-05-31T20:38:25.040 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:38:25.040 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T20:38:25.040 INFO:teuthology.orchestra.run.smithi099.stdout: "name": "obj00000000000000000037", 2023-05-31T20:38:25.040 INFO:teuthology.orchestra.run.smithi099.stdout: "instance": "", 2023-05-31T20:38:25.041 INFO:teuthology.orchestra.run.smithi099.stdout: "ver": { 2023-05-31T20:38:25.041 INFO:teuthology.orchestra.run.smithi099.stdout: "pool": 13, 2023-05-31T20:38:25.041 INFO:teuthology.orchestra.run.smithi099.stdout: "epoch": 4 2023-05-31T20:38:25.041 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:38:25.041 INFO:teuthology.orchestra.run.smithi099.stdout: "locator": "", 2023-05-31T20:38:25.041 INFO:teuthology.orchestra.run.smithi099.stdout: "exists": true, 2023-05-31T20:38:25.041 INFO:teuthology.orchestra.run.smithi099.stdout: "meta": { 2023-05-31T20:38:25.042 INFO:teuthology.orchestra.run.smithi099.stdout: "category": 1, 2023-05-31T20:38:25.042 INFO:teuthology.orchestra.run.smithi099.stdout: "size": 6, 2023-05-31T20:38:25.042 INFO:teuthology.orchestra.run.smithi099.stdout: "mtime": "2023-05-31T20:38:19.358015Z", 2023-05-31T20:38:25.042 INFO:teuthology.orchestra.run.smithi099.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-05-31T20:38:25.042 INFO:teuthology.orchestra.run.smithi099.stdout: "storage_class": "", 2023-05-31T20:38:25.042 INFO:teuthology.orchestra.run.smithi099.stdout: "owner": "rgw-multisite-test-user", 2023-05-31T20:38:25.042 INFO:teuthology.orchestra.run.smithi099.stdout: "owner_display_name": "Multisite Test User", 2023-05-31T20:38:25.043 INFO:teuthology.orchestra.run.smithi099.stdout: "content_type": "application/octet-stream", 2023-05-31T20:38:25.043 INFO:teuthology.orchestra.run.smithi099.stdout: "accounted_size": 3, 2023-05-31T20:38:25.043 INFO:teuthology.orchestra.run.smithi099.stdout: "user_data": "", 2023-05-31T20:38:25.043 INFO:teuthology.orchestra.run.smithi099.stdout: "appendable": false 2023-05-31T20:38:25.043 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:38:25.043 INFO:teuthology.orchestra.run.smithi099.stdout: "tag": "_TN5eK_91ABW6qOGWg-ZQ1836ZhjD2LB", 2023-05-31T20:38:25.043 INFO:teuthology.orchestra.run.smithi099.stdout: "flags": 0, 2023-05-31T20:38:25.044 INFO:teuthology.orchestra.run.smithi099.stdout: "pending_map": [], 2023-05-31T20:38:25.044 INFO:teuthology.orchestra.run.smithi099.stdout: "versioned_epoch": 0 2023-05-31T20:38:25.044 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:38:25.044 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T20:38:25.044 INFO:teuthology.orchestra.run.smithi099.stdout: "name": "obj00000000000000000039", 2023-05-31T20:38:25.044 INFO:teuthology.orchestra.run.smithi099.stdout: "instance": "", 2023-05-31T20:38:25.044 INFO:teuthology.orchestra.run.smithi099.stdout: "ver": { 2023-05-31T20:38:25.045 INFO:teuthology.orchestra.run.smithi099.stdout: "pool": 13, 2023-05-31T20:38:25.045 INFO:teuthology.orchestra.run.smithi099.stdout: "epoch": 1 2023-05-31T20:38:25.045 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:38:25.045 INFO:teuthology.orchestra.run.smithi099.stdout: "locator": "", 2023-05-31T20:38:25.045 INFO:teuthology.orchestra.run.smithi099.stdout: "exists": true, 2023-05-31T20:38:25.045 INFO:teuthology.orchestra.run.smithi099.stdout: "meta": { 2023-05-31T20:38:25.045 INFO:teuthology.orchestra.run.smithi099.stdout: "category": 1, 2023-05-31T20:38:25.046 INFO:teuthology.orchestra.run.smithi099.stdout: "size": 5, 2023-05-31T20:38:25.046 INFO:teuthology.orchestra.run.smithi099.stdout: "mtime": "2023-05-31T20:38:22.383805Z", 2023-05-31T20:38:25.046 INFO:teuthology.orchestra.run.smithi099.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-05-31T20:38:25.046 INFO:teuthology.orchestra.run.smithi099.stdout: "storage_class": "", 2023-05-31T20:38:25.046 INFO:teuthology.orchestra.run.smithi099.stdout: "owner": "rgw-multisite-test-user", 2023-05-31T20:38:25.046 INFO:teuthology.orchestra.run.smithi099.stdout: "owner_display_name": "Multisite Test User", 2023-05-31T20:38:25.047 INFO:teuthology.orchestra.run.smithi099.stdout: "content_type": "application/octet-stream", 2023-05-31T20:38:25.047 INFO:teuthology.orchestra.run.smithi099.stdout: "accounted_size": 3, 2023-05-31T20:38:25.047 INFO:teuthology.orchestra.run.smithi099.stdout: "user_data": "", 2023-05-31T20:38:25.047 INFO:teuthology.orchestra.run.smithi099.stdout: "appendable": false 2023-05-31T20:38:25.047 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:38:25.047 INFO:teuthology.orchestra.run.smithi099.stdout: "tag": "_8KZ7F8na1iubUnUhtiwZTEM2hLXqK4L", 2023-05-31T20:38:25.047 INFO:teuthology.orchestra.run.smithi099.stdout: "flags": 0, 2023-05-31T20:38:25.048 INFO:teuthology.orchestra.run.smithi099.stdout: "pending_map": [], 2023-05-31T20:38:25.048 INFO:teuthology.orchestra.run.smithi099.stdout: "versioned_epoch": 0 2023-05-31T20:38:25.048 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:38:25.048 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T20:38:25.048 INFO:teuthology.orchestra.run.smithi099.stdout: "name": "obj00000000000000000040", 2023-05-31T20:38:25.048 INFO:teuthology.orchestra.run.smithi099.stdout: "instance": "", 2023-05-31T20:38:25.048 INFO:teuthology.orchestra.run.smithi099.stdout: "ver": { 2023-05-31T20:38:25.049 INFO:teuthology.orchestra.run.smithi099.stdout: "pool": 13, 2023-05-31T20:38:25.049 INFO:teuthology.orchestra.run.smithi099.stdout: "epoch": 4 2023-05-31T20:38:25.049 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:38:25.049 INFO:teuthology.orchestra.run.smithi099.stdout: "locator": "", 2023-05-31T20:38:25.049 INFO:teuthology.orchestra.run.smithi099.stdout: "exists": true, 2023-05-31T20:38:25.049 INFO:teuthology.orchestra.run.smithi099.stdout: "meta": { 2023-05-31T20:38:25.049 INFO:teuthology.orchestra.run.smithi099.stdout: "category": 1, 2023-05-31T20:38:25.050 INFO:teuthology.orchestra.run.smithi099.stdout: "size": 3, 2023-05-31T20:38:25.050 INFO:teuthology.orchestra.run.smithi099.stdout: "mtime": "2023-05-31T20:38:22.327307Z", 2023-05-31T20:38:25.050 INFO:teuthology.orchestra.run.smithi099.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-05-31T20:38:25.050 INFO:teuthology.orchestra.run.smithi099.stdout: "storage_class": "", 2023-05-31T20:38:25.050 INFO:teuthology.orchestra.run.smithi099.stdout: "owner": "rgw-multisite-test-user", 2023-05-31T20:38:25.050 INFO:teuthology.orchestra.run.smithi099.stdout: "owner_display_name": "Multisite Test User", 2023-05-31T20:38:25.050 INFO:teuthology.orchestra.run.smithi099.stdout: "content_type": "application/octet-stream", 2023-05-31T20:38:25.051 INFO:teuthology.orchestra.run.smithi099.stdout: "accounted_size": 3, 2023-05-31T20:38:25.051 INFO:teuthology.orchestra.run.smithi099.stdout: "user_data": "", 2023-05-31T20:38:25.051 INFO:teuthology.orchestra.run.smithi099.stdout: "appendable": false 2023-05-31T20:38:25.051 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:38:25.051 INFO:teuthology.orchestra.run.smithi099.stdout: "tag": "_dqklxd5yRv6AeFVq1F_v_UbEh1AYzzE", 2023-05-31T20:38:25.051 INFO:teuthology.orchestra.run.smithi099.stdout: "flags": 0, 2023-05-31T20:38:25.051 INFO:teuthology.orchestra.run.smithi099.stdout: "pending_map": [], 2023-05-31T20:38:25.052 INFO:teuthology.orchestra.run.smithi099.stdout: "versioned_epoch": 0 2023-05-31T20:38:25.052 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:38:25.052 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T20:38:25.052 INFO:teuthology.orchestra.run.smithi099.stdout: "name": "obj00000000000000000042", 2023-05-31T20:38:25.052 INFO:teuthology.orchestra.run.smithi099.stdout: "instance": "", 2023-05-31T20:38:25.052 INFO:teuthology.orchestra.run.smithi099.stdout: "ver": { 2023-05-31T20:38:25.053 INFO:teuthology.orchestra.run.smithi099.stdout: "pool": 13, 2023-05-31T20:38:25.053 INFO:teuthology.orchestra.run.smithi099.stdout: "epoch": 1 2023-05-31T20:38:25.053 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:38:25.053 INFO:teuthology.orchestra.run.smithi099.stdout: "locator": "", 2023-05-31T20:38:25.053 INFO:teuthology.orchestra.run.smithi099.stdout: "exists": true, 2023-05-31T20:38:25.053 INFO:teuthology.orchestra.run.smithi099.stdout: "meta": { 2023-05-31T20:38:25.053 INFO:teuthology.orchestra.run.smithi099.stdout: "category": 1, 2023-05-31T20:38:25.054 INFO:teuthology.orchestra.run.smithi099.stdout: "size": 5, 2023-05-31T20:38:25.054 INFO:teuthology.orchestra.run.smithi099.stdout: "mtime": "2023-05-31T20:38:18.977382Z", 2023-05-31T20:38:25.054 INFO:teuthology.orchestra.run.smithi099.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-05-31T20:38:25.054 INFO:teuthology.orchestra.run.smithi099.stdout: "storage_class": "", 2023-05-31T20:38:25.054 INFO:teuthology.orchestra.run.smithi099.stdout: "owner": "rgw-multisite-test-user", 2023-05-31T20:38:25.054 INFO:teuthology.orchestra.run.smithi099.stdout: "owner_display_name": "Multisite Test User", 2023-05-31T20:38:25.054 INFO:teuthology.orchestra.run.smithi099.stdout: "content_type": "application/octet-stream", 2023-05-31T20:38:25.055 INFO:teuthology.orchestra.run.smithi099.stdout: "accounted_size": 3, 2023-05-31T20:38:25.055 INFO:teuthology.orchestra.run.smithi099.stdout: "user_data": "", 2023-05-31T20:38:25.055 INFO:teuthology.orchestra.run.smithi099.stdout: "appendable": false 2023-05-31T20:38:25.055 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:38:25.055 INFO:teuthology.orchestra.run.smithi099.stdout: "tag": "_S1GFcBeFKiCoASTKh1CJSk48aLrOuPc", 2023-05-31T20:38:25.055 INFO:teuthology.orchestra.run.smithi099.stdout: "flags": 0, 2023-05-31T20:38:25.055 INFO:teuthology.orchestra.run.smithi099.stdout: "pending_map": [], 2023-05-31T20:38:25.056 INFO:teuthology.orchestra.run.smithi099.stdout: "versioned_epoch": 0 2023-05-31T20:38:25.056 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:38:25.056 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T20:38:25.056 INFO:teuthology.orchestra.run.smithi099.stdout: "name": "obj00000000000000000045", 2023-05-31T20:38:25.056 INFO:teuthology.orchestra.run.smithi099.stdout: "instance": "", 2023-05-31T20:38:25.056 INFO:teuthology.orchestra.run.smithi099.stdout: "ver": { 2023-05-31T20:38:25.056 INFO:teuthology.orchestra.run.smithi099.stdout: "pool": 13, 2023-05-31T20:38:25.057 INFO:teuthology.orchestra.run.smithi099.stdout: "epoch": 5 2023-05-31T20:38:25.057 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:38:25.057 INFO:teuthology.orchestra.run.smithi099.stdout: "locator": "", 2023-05-31T20:38:25.057 INFO:teuthology.orchestra.run.smithi099.stdout: "exists": true, 2023-05-31T20:38:25.057 INFO:teuthology.orchestra.run.smithi099.stdout: "meta": { 2023-05-31T20:38:25.057 INFO:teuthology.orchestra.run.smithi099.stdout: "category": 1, 2023-05-31T20:38:25.057 INFO:teuthology.orchestra.run.smithi099.stdout: "size": 16, 2023-05-31T20:38:25.058 INFO:teuthology.orchestra.run.smithi099.stdout: "mtime": "2023-05-31T20:38:22.250679Z", 2023-05-31T20:38:25.058 INFO:teuthology.orchestra.run.smithi099.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-05-31T20:38:25.058 INFO:teuthology.orchestra.run.smithi099.stdout: "storage_class": "", 2023-05-31T20:38:25.058 INFO:teuthology.orchestra.run.smithi099.stdout: "owner": "rgw-multisite-test-user", 2023-05-31T20:38:25.058 INFO:teuthology.orchestra.run.smithi099.stdout: "owner_display_name": "Multisite Test User", 2023-05-31T20:38:25.058 INFO:teuthology.orchestra.run.smithi099.stdout: "content_type": "application/octet-stream", 2023-05-31T20:38:25.058 INFO:teuthology.orchestra.run.smithi099.stdout: "accounted_size": 3, 2023-05-31T20:38:25.059 INFO:teuthology.orchestra.run.smithi099.stdout: "user_data": "", 2023-05-31T20:38:25.059 INFO:teuthology.orchestra.run.smithi099.stdout: "appendable": false 2023-05-31T20:38:25.059 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:38:25.059 INFO:teuthology.orchestra.run.smithi099.stdout: "tag": "_SnfOXfWj0MNOQlX--KErQITboKeKgBa", 2023-05-31T20:38:25.059 INFO:teuthology.orchestra.run.smithi099.stdout: "flags": 0, 2023-05-31T20:38:25.059 INFO:teuthology.orchestra.run.smithi099.stdout: "pending_map": [], 2023-05-31T20:38:25.060 INFO:teuthology.orchestra.run.smithi099.stdout: "versioned_epoch": 0 2023-05-31T20:38:25.060 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:38:25.060 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T20:38:25.060 INFO:teuthology.orchestra.run.smithi099.stdout: "name": "obj00000000000000000046", 2023-05-31T20:38:25.060 INFO:teuthology.orchestra.run.smithi099.stdout: "instance": "", 2023-05-31T20:38:25.060 INFO:teuthology.orchestra.run.smithi099.stdout: "ver": { 2023-05-31T20:38:25.060 INFO:teuthology.orchestra.run.smithi099.stdout: "pool": 13, 2023-05-31T20:38:25.061 INFO:teuthology.orchestra.run.smithi099.stdout: "epoch": 4 2023-05-31T20:38:25.061 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:38:25.061 INFO:teuthology.orchestra.run.smithi099.stdout: "locator": "", 2023-05-31T20:38:25.061 INFO:teuthology.orchestra.run.smithi099.stdout: "exists": true, 2023-05-31T20:38:25.061 INFO:teuthology.orchestra.run.smithi099.stdout: "meta": { 2023-05-31T20:38:25.061 INFO:teuthology.orchestra.run.smithi099.stdout: "category": 1, 2023-05-31T20:38:25.061 INFO:teuthology.orchestra.run.smithi099.stdout: "size": 5, 2023-05-31T20:38:25.062 INFO:teuthology.orchestra.run.smithi099.stdout: "mtime": "2023-05-31T20:38:17.357924Z", 2023-05-31T20:38:25.062 INFO:teuthology.orchestra.run.smithi099.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-05-31T20:38:25.062 INFO:teuthology.orchestra.run.smithi099.stdout: "storage_class": "", 2023-05-31T20:38:25.062 INFO:teuthology.orchestra.run.smithi099.stdout: "owner": "rgw-multisite-test-user", 2023-05-31T20:38:25.062 INFO:teuthology.orchestra.run.smithi099.stdout: "owner_display_name": "Multisite Test User", 2023-05-31T20:38:25.062 INFO:teuthology.orchestra.run.smithi099.stdout: "content_type": "application/octet-stream", 2023-05-31T20:38:25.062 INFO:teuthology.orchestra.run.smithi099.stdout: "accounted_size": 3, 2023-05-31T20:38:25.063 INFO:teuthology.orchestra.run.smithi099.stdout: "user_data": "", 2023-05-31T20:38:25.063 INFO:teuthology.orchestra.run.smithi099.stdout: "appendable": false 2023-05-31T20:38:25.063 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:38:25.063 INFO:teuthology.orchestra.run.smithi099.stdout: "tag": "_NdqwYql4Img12H__nL9vJ2OPSdeSWlS", 2023-05-31T20:38:25.063 INFO:teuthology.orchestra.run.smithi099.stdout: "flags": 0, 2023-05-31T20:38:25.063 INFO:teuthology.orchestra.run.smithi099.stdout: "pending_map": [], 2023-05-31T20:38:25.063 INFO:teuthology.orchestra.run.smithi099.stdout: "versioned_epoch": 0 2023-05-31T20:38:25.064 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:38:25.064 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T20:38:25.064 INFO:teuthology.orchestra.run.smithi099.stdout: "name": "obj00000000000000000048", 2023-05-31T20:38:25.064 INFO:teuthology.orchestra.run.smithi099.stdout: "instance": "", 2023-05-31T20:38:25.064 INFO:teuthology.orchestra.run.smithi099.stdout: "ver": { 2023-05-31T20:38:25.064 INFO:teuthology.orchestra.run.smithi099.stdout: "pool": 13, 2023-05-31T20:38:25.064 INFO:teuthology.orchestra.run.smithi099.stdout: "epoch": 4 2023-05-31T20:38:25.065 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:38:25.065 INFO:teuthology.orchestra.run.smithi099.stdout: "locator": "", 2023-05-31T20:38:25.065 INFO:teuthology.orchestra.run.smithi099.stdout: "exists": true, 2023-05-31T20:38:25.065 INFO:teuthology.orchestra.run.smithi099.stdout: "meta": { 2023-05-31T20:38:25.065 INFO:teuthology.orchestra.run.smithi099.stdout: "category": 1, 2023-05-31T20:38:25.065 INFO:teuthology.orchestra.run.smithi099.stdout: "size": 16, 2023-05-31T20:38:25.065 INFO:teuthology.orchestra.run.smithi099.stdout: "mtime": "2023-05-31T20:38:22.489163Z", 2023-05-31T20:38:25.066 INFO:teuthology.orchestra.run.smithi099.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-05-31T20:38:25.066 INFO:teuthology.orchestra.run.smithi099.stdout: "storage_class": "", 2023-05-31T20:38:25.066 INFO:teuthology.orchestra.run.smithi099.stdout: "owner": "rgw-multisite-test-user", 2023-05-31T20:38:25.066 INFO:teuthology.orchestra.run.smithi099.stdout: "owner_display_name": "Multisite Test User", 2023-05-31T20:38:25.066 INFO:teuthology.orchestra.run.smithi099.stdout: "content_type": "application/octet-stream", 2023-05-31T20:38:25.066 INFO:teuthology.orchestra.run.smithi099.stdout: "accounted_size": 3, 2023-05-31T20:38:25.067 INFO:teuthology.orchestra.run.smithi099.stdout: "user_data": "", 2023-05-31T20:38:25.067 INFO:teuthology.orchestra.run.smithi099.stdout: "appendable": false 2023-05-31T20:38:25.067 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:38:25.067 INFO:teuthology.orchestra.run.smithi099.stdout: "tag": "_2RFrw_esgvukfvinYjnFT7SR8mcxEJs", 2023-05-31T20:38:25.067 INFO:teuthology.orchestra.run.smithi099.stdout: "flags": 0, 2023-05-31T20:38:25.067 INFO:teuthology.orchestra.run.smithi099.stdout: "pending_map": [], 2023-05-31T20:38:25.067 INFO:teuthology.orchestra.run.smithi099.stdout: "versioned_epoch": 0 2023-05-31T20:38:25.068 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:38:25.068 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T20:38:25.068 INFO:teuthology.orchestra.run.smithi099.stdout: "name": "obj00000000000000000049", 2023-05-31T20:38:25.068 INFO:teuthology.orchestra.run.smithi099.stdout: "instance": "", 2023-05-31T20:38:25.068 INFO:teuthology.orchestra.run.smithi099.stdout: "ver": { 2023-05-31T20:38:25.068 INFO:teuthology.orchestra.run.smithi099.stdout: "pool": 13, 2023-05-31T20:38:25.069 INFO:teuthology.orchestra.run.smithi099.stdout: "epoch": 2 2023-05-31T20:38:25.069 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:38:25.069 INFO:teuthology.orchestra.run.smithi099.stdout: "locator": "", 2023-05-31T20:38:25.069 INFO:teuthology.orchestra.run.smithi099.stdout: "exists": true, 2023-05-31T20:38:25.069 INFO:teuthology.orchestra.run.smithi099.stdout: "meta": { 2023-05-31T20:38:25.069 INFO:teuthology.orchestra.run.smithi099.stdout: "category": 1, 2023-05-31T20:38:25.069 INFO:teuthology.orchestra.run.smithi099.stdout: "size": 16, 2023-05-31T20:38:25.070 INFO:teuthology.orchestra.run.smithi099.stdout: "mtime": "2023-05-31T20:38:18.527639Z", 2023-05-31T20:38:25.070 INFO:teuthology.orchestra.run.smithi099.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-05-31T20:38:25.070 INFO:teuthology.orchestra.run.smithi099.stdout: "storage_class": "", 2023-05-31T20:38:25.070 INFO:teuthology.orchestra.run.smithi099.stdout: "owner": "rgw-multisite-test-user", 2023-05-31T20:38:25.070 INFO:teuthology.orchestra.run.smithi099.stdout: "owner_display_name": "Multisite Test User", 2023-05-31T20:38:25.070 INFO:teuthology.orchestra.run.smithi099.stdout: "content_type": "application/octet-stream", 2023-05-31T20:38:25.070 INFO:teuthology.orchestra.run.smithi099.stdout: "accounted_size": 3, 2023-05-31T20:38:25.071 INFO:teuthology.orchestra.run.smithi099.stdout: "user_data": "", 2023-05-31T20:38:25.071 INFO:teuthology.orchestra.run.smithi099.stdout: "appendable": false 2023-05-31T20:38:25.071 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:38:25.071 INFO:teuthology.orchestra.run.smithi099.stdout: "tag": "_avPUGBCpDHed3FK9Ecv70eOs9fJaiGN", 2023-05-31T20:38:25.071 INFO:teuthology.orchestra.run.smithi099.stdout: "flags": 0, 2023-05-31T20:38:25.071 INFO:teuthology.orchestra.run.smithi099.stdout: "pending_map": [], 2023-05-31T20:38:25.071 INFO:teuthology.orchestra.run.smithi099.stdout: "versioned_epoch": 0 2023-05-31T20:38:25.072 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:38:25.072 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T20:38:25.072 INFO:teuthology.orchestra.run.smithi099.stdout: "name": "obj00000000000000000050", 2023-05-31T20:38:25.072 INFO:teuthology.orchestra.run.smithi099.stdout: "instance": "", 2023-05-31T20:38:25.072 INFO:teuthology.orchestra.run.smithi099.stdout: "ver": { 2023-05-31T20:38:25.072 INFO:teuthology.orchestra.run.smithi099.stdout: "pool": 13, 2023-05-31T20:38:25.072 INFO:teuthology.orchestra.run.smithi099.stdout: "epoch": 4 2023-05-31T20:38:25.073 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:38:25.073 INFO:teuthology.orchestra.run.smithi099.stdout: "locator": "", 2023-05-31T20:38:25.073 INFO:teuthology.orchestra.run.smithi099.stdout: "exists": true, 2023-05-31T20:38:25.073 INFO:teuthology.orchestra.run.smithi099.stdout: "meta": { 2023-05-31T20:38:25.073 INFO:teuthology.orchestra.run.smithi099.stdout: "category": 1, 2023-05-31T20:38:25.073 INFO:teuthology.orchestra.run.smithi099.stdout: "size": 5, 2023-05-31T20:38:25.074 INFO:teuthology.orchestra.run.smithi099.stdout: "mtime": "2023-05-31T20:38:20.183372Z", 2023-05-31T20:38:25.074 INFO:teuthology.orchestra.run.smithi099.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-05-31T20:38:25.074 INFO:teuthology.orchestra.run.smithi099.stdout: "storage_class": "", 2023-05-31T20:38:25.074 INFO:teuthology.orchestra.run.smithi099.stdout: "owner": "rgw-multisite-test-user", 2023-05-31T20:38:25.074 INFO:teuthology.orchestra.run.smithi099.stdout: "owner_display_name": "Multisite Test User", 2023-05-31T20:38:25.074 INFO:teuthology.orchestra.run.smithi099.stdout: "content_type": "application/octet-stream", 2023-05-31T20:38:25.074 INFO:teuthology.orchestra.run.smithi099.stdout: "accounted_size": 3, 2023-05-31T20:38:25.075 INFO:teuthology.orchestra.run.smithi099.stdout: "user_data": "", 2023-05-31T20:38:25.075 INFO:teuthology.orchestra.run.smithi099.stdout: "appendable": false 2023-05-31T20:38:25.075 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:38:25.075 INFO:teuthology.orchestra.run.smithi099.stdout: "tag": "_hk2OYXXyrbWMpLNSeTVPNdrF33KPc97", 2023-05-31T20:38:25.075 INFO:teuthology.orchestra.run.smithi099.stdout: "flags": 0, 2023-05-31T20:38:25.075 INFO:teuthology.orchestra.run.smithi099.stdout: "pending_map": [], 2023-05-31T20:38:25.075 INFO:teuthology.orchestra.run.smithi099.stdout: "versioned_epoch": 0 2023-05-31T20:38:25.076 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:38:25.076 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T20:38:25.076 INFO:teuthology.orchestra.run.smithi099.stdout: "name": "obj00000000000000000053", 2023-05-31T20:38:25.076 INFO:teuthology.orchestra.run.smithi099.stdout: "instance": "", 2023-05-31T20:38:25.076 INFO:teuthology.orchestra.run.smithi099.stdout: "ver": { 2023-05-31T20:38:25.076 INFO:teuthology.orchestra.run.smithi099.stdout: "pool": 13, 2023-05-31T20:38:25.076 INFO:teuthology.orchestra.run.smithi099.stdout: "epoch": 8 2023-05-31T20:38:25.077 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:38:25.077 INFO:teuthology.orchestra.run.smithi099.stdout: "locator": "", 2023-05-31T20:38:25.077 INFO:teuthology.orchestra.run.smithi099.stdout: "exists": true, 2023-05-31T20:38:25.077 INFO:teuthology.orchestra.run.smithi099.stdout: "meta": { 2023-05-31T20:38:25.077 INFO:teuthology.orchestra.run.smithi099.stdout: "category": 1, 2023-05-31T20:38:25.077 INFO:teuthology.orchestra.run.smithi099.stdout: "size": 6, 2023-05-31T20:38:25.078 INFO:teuthology.orchestra.run.smithi099.stdout: "mtime": "2023-05-31T20:38:22.873398Z", 2023-05-31T20:38:25.078 INFO:teuthology.orchestra.run.smithi099.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-05-31T20:38:25.078 INFO:teuthology.orchestra.run.smithi099.stdout: "storage_class": "", 2023-05-31T20:38:25.078 INFO:teuthology.orchestra.run.smithi099.stdout: "owner": "rgw-multisite-test-user", 2023-05-31T20:38:25.078 INFO:teuthology.orchestra.run.smithi099.stdout: "owner_display_name": "Multisite Test User", 2023-05-31T20:38:25.078 INFO:teuthology.orchestra.run.smithi099.stdout: "content_type": "application/octet-stream", 2023-05-31T20:38:25.078 INFO:teuthology.orchestra.run.smithi099.stdout: "accounted_size": 3, 2023-05-31T20:38:25.079 INFO:teuthology.orchestra.run.smithi099.stdout: "user_data": "", 2023-05-31T20:38:25.079 INFO:teuthology.orchestra.run.smithi099.stdout: "appendable": false 2023-05-31T20:38:25.079 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:38:25.079 INFO:teuthology.orchestra.run.smithi099.stdout: "tag": "_AGH4aBiWshl5_wfj3LXNEBRQiGHDya1", 2023-05-31T20:38:25.079 INFO:teuthology.orchestra.run.smithi099.stdout: "flags": 0, 2023-05-31T20:38:25.079 INFO:teuthology.orchestra.run.smithi099.stdout: "pending_map": [], 2023-05-31T20:38:25.079 INFO:teuthology.orchestra.run.smithi099.stdout: "versioned_epoch": 0 2023-05-31T20:38:25.080 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:38:25.080 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T20:38:25.080 INFO:teuthology.orchestra.run.smithi099.stdout: "name": "obj00000000000000000055", 2023-05-31T20:38:25.080 INFO:teuthology.orchestra.run.smithi099.stdout: "instance": "", 2023-05-31T20:38:25.080 INFO:teuthology.orchestra.run.smithi099.stdout: "ver": { 2023-05-31T20:38:25.080 INFO:teuthology.orchestra.run.smithi099.stdout: "pool": 13, 2023-05-31T20:38:25.080 INFO:teuthology.orchestra.run.smithi099.stdout: "epoch": 2 2023-05-31T20:38:25.081 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:38:25.081 INFO:teuthology.orchestra.run.smithi099.stdout: "locator": "", 2023-05-31T20:38:25.081 INFO:teuthology.orchestra.run.smithi099.stdout: "exists": true, 2023-05-31T20:38:25.081 INFO:teuthology.orchestra.run.smithi099.stdout: "meta": { 2023-05-31T20:38:25.081 INFO:teuthology.orchestra.run.smithi099.stdout: "category": 1, 2023-05-31T20:38:25.081 INFO:teuthology.orchestra.run.smithi099.stdout: "size": 3, 2023-05-31T20:38:25.081 INFO:teuthology.orchestra.run.smithi099.stdout: "mtime": "2023-05-31T20:38:17.862999Z", 2023-05-31T20:38:25.082 INFO:teuthology.orchestra.run.smithi099.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-05-31T20:38:25.082 INFO:teuthology.orchestra.run.smithi099.stdout: "storage_class": "", 2023-05-31T20:38:25.082 INFO:teuthology.orchestra.run.smithi099.stdout: "owner": "rgw-multisite-test-user", 2023-05-31T20:38:25.082 INFO:teuthology.orchestra.run.smithi099.stdout: "owner_display_name": "Multisite Test User", 2023-05-31T20:38:25.082 INFO:teuthology.orchestra.run.smithi099.stdout: "content_type": "application/octet-stream", 2023-05-31T20:38:25.082 INFO:teuthology.orchestra.run.smithi099.stdout: "accounted_size": 3, 2023-05-31T20:38:25.082 INFO:teuthology.orchestra.run.smithi099.stdout: "user_data": "", 2023-05-31T20:38:25.083 INFO:teuthology.orchestra.run.smithi099.stdout: "appendable": false 2023-05-31T20:38:25.083 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:38:25.083 INFO:teuthology.orchestra.run.smithi099.stdout: "tag": "_OSdo56a1N3G0IkmMbv1vSaoakS_dWai", 2023-05-31T20:38:25.083 INFO:teuthology.orchestra.run.smithi099.stdout: "flags": 0, 2023-05-31T20:38:25.083 INFO:teuthology.orchestra.run.smithi099.stdout: "pending_map": [], 2023-05-31T20:38:25.083 INFO:teuthology.orchestra.run.smithi099.stdout: "versioned_epoch": 0 2023-05-31T20:38:25.084 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:38:25.084 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T20:38:25.084 INFO:teuthology.orchestra.run.smithi099.stdout: "name": "obj00000000000000000056", 2023-05-31T20:38:25.084 INFO:teuthology.orchestra.run.smithi099.stdout: "instance": "", 2023-05-31T20:38:25.084 INFO:teuthology.orchestra.run.smithi099.stdout: "ver": { 2023-05-31T20:38:25.084 INFO:teuthology.orchestra.run.smithi099.stdout: "pool": 13, 2023-05-31T20:38:25.084 INFO:teuthology.orchestra.run.smithi099.stdout: "epoch": 3 2023-05-31T20:38:25.085 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:38:25.085 INFO:teuthology.orchestra.run.smithi099.stdout: "locator": "", 2023-05-31T20:38:25.085 INFO:teuthology.orchestra.run.smithi099.stdout: "exists": true, 2023-05-31T20:38:25.085 INFO:teuthology.orchestra.run.smithi099.stdout: "meta": { 2023-05-31T20:38:25.085 INFO:teuthology.orchestra.run.smithi099.stdout: "category": 1, 2023-05-31T20:38:25.085 INFO:teuthology.orchestra.run.smithi099.stdout: "size": 16, 2023-05-31T20:38:25.085 INFO:teuthology.orchestra.run.smithi099.stdout: "mtime": "2023-05-31T20:38:20.370757Z", 2023-05-31T20:38:25.086 INFO:teuthology.orchestra.run.smithi099.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-05-31T20:38:25.086 INFO:teuthology.orchestra.run.smithi099.stdout: "storage_class": "", 2023-05-31T20:38:25.086 INFO:teuthology.orchestra.run.smithi099.stdout: "owner": "rgw-multisite-test-user", 2023-05-31T20:38:25.086 INFO:teuthology.orchestra.run.smithi099.stdout: "owner_display_name": "Multisite Test User", 2023-05-31T20:38:25.086 INFO:teuthology.orchestra.run.smithi099.stdout: "content_type": "application/octet-stream", 2023-05-31T20:38:25.086 INFO:teuthology.orchestra.run.smithi099.stdout: "accounted_size": 3, 2023-05-31T20:38:25.087 INFO:teuthology.orchestra.run.smithi099.stdout: "user_data": "", 2023-05-31T20:38:25.087 INFO:teuthology.orchestra.run.smithi099.stdout: "appendable": false 2023-05-31T20:38:25.087 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:38:25.087 INFO:teuthology.orchestra.run.smithi099.stdout: "tag": "_j_68uaEVVBuW-NC4U9hKz5c-xXVWryc", 2023-05-31T20:38:25.087 INFO:teuthology.orchestra.run.smithi099.stdout: "flags": 0, 2023-05-31T20:38:25.087 INFO:teuthology.orchestra.run.smithi099.stdout: "pending_map": [], 2023-05-31T20:38:25.087 INFO:teuthology.orchestra.run.smithi099.stdout: "versioned_epoch": 0 2023-05-31T20:38:25.088 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:38:25.088 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T20:38:25.088 INFO:teuthology.orchestra.run.smithi099.stdout: "name": "obj00000000000000000057", 2023-05-31T20:38:25.088 INFO:teuthology.orchestra.run.smithi099.stdout: "instance": "", 2023-05-31T20:38:25.088 INFO:teuthology.orchestra.run.smithi099.stdout: "ver": { 2023-05-31T20:38:25.088 INFO:teuthology.orchestra.run.smithi099.stdout: "pool": 13, 2023-05-31T20:38:25.089 INFO:teuthology.orchestra.run.smithi099.stdout: "epoch": 5 2023-05-31T20:38:25.089 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:38:25.089 INFO:teuthology.orchestra.run.smithi099.stdout: "locator": "", 2023-05-31T20:38:25.089 INFO:teuthology.orchestra.run.smithi099.stdout: "exists": true, 2023-05-31T20:38:25.089 INFO:teuthology.orchestra.run.smithi099.stdout: "meta": { 2023-05-31T20:38:25.089 INFO:teuthology.orchestra.run.smithi099.stdout: "category": 1, 2023-05-31T20:38:25.089 INFO:teuthology.orchestra.run.smithi099.stdout: "size": 5, 2023-05-31T20:38:25.090 INFO:teuthology.orchestra.run.smithi099.stdout: "mtime": "2023-05-31T20:38:21.818642Z", 2023-05-31T20:38:25.090 INFO:teuthology.orchestra.run.smithi099.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-05-31T20:38:25.090 INFO:teuthology.orchestra.run.smithi099.stdout: "storage_class": "", 2023-05-31T20:38:25.090 INFO:teuthology.orchestra.run.smithi099.stdout: "owner": "rgw-multisite-test-user", 2023-05-31T20:38:25.090 INFO:teuthology.orchestra.run.smithi099.stdout: "owner_display_name": "Multisite Test User", 2023-05-31T20:38:25.090 INFO:teuthology.orchestra.run.smithi099.stdout: "content_type": "application/octet-stream", 2023-05-31T20:38:25.090 INFO:teuthology.orchestra.run.smithi099.stdout: "accounted_size": 3, 2023-05-31T20:38:25.091 INFO:teuthology.orchestra.run.smithi099.stdout: "user_data": "", 2023-05-31T20:38:25.091 INFO:teuthology.orchestra.run.smithi099.stdout: "appendable": false 2023-05-31T20:38:25.091 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:38:25.091 INFO:teuthology.orchestra.run.smithi099.stdout: "tag": "__iEbTFz7PkPpTUsew1vG_J8Sdjn4QaA", 2023-05-31T20:38:25.091 INFO:teuthology.orchestra.run.smithi099.stdout: "flags": 0, 2023-05-31T20:38:25.091 INFO:teuthology.orchestra.run.smithi099.stdout: "pending_map": [], 2023-05-31T20:38:25.091 INFO:teuthology.orchestra.run.smithi099.stdout: "versioned_epoch": 0 2023-05-31T20:38:25.092 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:38:25.092 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T20:38:25.092 INFO:teuthology.orchestra.run.smithi099.stdout: "name": "obj00000000000000000058", 2023-05-31T20:38:25.092 INFO:teuthology.orchestra.run.smithi099.stdout: "instance": "", 2023-05-31T20:38:25.092 INFO:teuthology.orchestra.run.smithi099.stdout: "ver": { 2023-05-31T20:38:25.092 INFO:teuthology.orchestra.run.smithi099.stdout: "pool": 13, 2023-05-31T20:38:25.093 INFO:teuthology.orchestra.run.smithi099.stdout: "epoch": 7 2023-05-31T20:38:25.093 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:38:25.093 INFO:teuthology.orchestra.run.smithi099.stdout: "locator": "", 2023-05-31T20:38:25.093 INFO:teuthology.orchestra.run.smithi099.stdout: "exists": true, 2023-05-31T20:38:25.093 INFO:teuthology.orchestra.run.smithi099.stdout: "meta": { 2023-05-31T20:38:25.093 INFO:teuthology.orchestra.run.smithi099.stdout: "category": 1, 2023-05-31T20:38:25.093 INFO:teuthology.orchestra.run.smithi099.stdout: "size": 5, 2023-05-31T20:38:25.094 INFO:teuthology.orchestra.run.smithi099.stdout: "mtime": "2023-05-31T20:38:17.631771Z", 2023-05-31T20:38:25.094 INFO:teuthology.orchestra.run.smithi099.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-05-31T20:38:25.094 INFO:teuthology.orchestra.run.smithi099.stdout: "storage_class": "", 2023-05-31T20:38:25.094 INFO:teuthology.orchestra.run.smithi099.stdout: "owner": "rgw-multisite-test-user", 2023-05-31T20:38:25.094 INFO:teuthology.orchestra.run.smithi099.stdout: "owner_display_name": "Multisite Test User", 2023-05-31T20:38:25.094 INFO:teuthology.orchestra.run.smithi099.stdout: "content_type": "application/octet-stream", 2023-05-31T20:38:25.094 INFO:teuthology.orchestra.run.smithi099.stdout: "accounted_size": 3, 2023-05-31T20:38:25.095 INFO:teuthology.orchestra.run.smithi099.stdout: "user_data": "", 2023-05-31T20:38:25.095 INFO:teuthology.orchestra.run.smithi099.stdout: "appendable": false 2023-05-31T20:38:25.095 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:38:25.095 INFO:teuthology.orchestra.run.smithi099.stdout: "tag": "_g2UWgbocPL0RZaTfgJnbx_oHbCb-Mvw", 2023-05-31T20:38:25.095 INFO:teuthology.orchestra.run.smithi099.stdout: "flags": 0, 2023-05-31T20:38:25.095 INFO:teuthology.orchestra.run.smithi099.stdout: "pending_map": [], 2023-05-31T20:38:25.095 INFO:teuthology.orchestra.run.smithi099.stdout: "versioned_epoch": 0 2023-05-31T20:38:25.096 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:38:25.096 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T20:38:25.096 INFO:teuthology.orchestra.run.smithi099.stdout: "name": "obj00000000000000000059", 2023-05-31T20:38:25.096 INFO:teuthology.orchestra.run.smithi099.stdout: "instance": "", 2023-05-31T20:38:25.096 INFO:teuthology.orchestra.run.smithi099.stdout: "ver": { 2023-05-31T20:38:25.096 INFO:teuthology.orchestra.run.smithi099.stdout: "pool": 13, 2023-05-31T20:38:25.096 INFO:teuthology.orchestra.run.smithi099.stdout: "epoch": 1 2023-05-31T20:38:25.097 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:38:25.097 INFO:teuthology.orchestra.run.smithi099.stdout: "locator": "", 2023-05-31T20:38:25.097 INFO:teuthology.orchestra.run.smithi099.stdout: "exists": true, 2023-05-31T20:38:25.097 INFO:teuthology.orchestra.run.smithi099.stdout: "meta": { 2023-05-31T20:38:25.097 INFO:teuthology.orchestra.run.smithi099.stdout: "category": 1, 2023-05-31T20:38:25.097 INFO:teuthology.orchestra.run.smithi099.stdout: "size": 16, 2023-05-31T20:38:25.097 INFO:teuthology.orchestra.run.smithi099.stdout: "mtime": "2023-05-31T20:38:18.560629Z", 2023-05-31T20:38:25.098 INFO:teuthology.orchestra.run.smithi099.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-05-31T20:38:25.098 INFO:teuthology.orchestra.run.smithi099.stdout: "storage_class": "", 2023-05-31T20:38:25.098 INFO:teuthology.orchestra.run.smithi099.stdout: "owner": "rgw-multisite-test-user", 2023-05-31T20:38:25.098 INFO:teuthology.orchestra.run.smithi099.stdout: "owner_display_name": "Multisite Test User", 2023-05-31T20:38:25.098 INFO:teuthology.orchestra.run.smithi099.stdout: "content_type": "application/octet-stream", 2023-05-31T20:38:25.098 INFO:teuthology.orchestra.run.smithi099.stdout: "accounted_size": 3, 2023-05-31T20:38:25.098 INFO:teuthology.orchestra.run.smithi099.stdout: "user_data": "", 2023-05-31T20:38:25.099 INFO:teuthology.orchestra.run.smithi099.stdout: "appendable": false 2023-05-31T20:38:25.099 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:38:25.099 INFO:teuthology.orchestra.run.smithi099.stdout: "tag": "_cgin_-nMUfHIVZjNXGlflKhQ4lt_q2l", 2023-05-31T20:38:25.099 INFO:teuthology.orchestra.run.smithi099.stdout: "flags": 0, 2023-05-31T20:38:25.099 INFO:teuthology.orchestra.run.smithi099.stdout: "pending_map": [], 2023-05-31T20:38:25.099 INFO:teuthology.orchestra.run.smithi099.stdout: "versioned_epoch": 0 2023-05-31T20:38:25.099 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:38:25.100 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T20:38:25.100 INFO:teuthology.orchestra.run.smithi099.stdout: "name": "obj00000000000000000060", 2023-05-31T20:38:25.100 INFO:teuthology.orchestra.run.smithi099.stdout: "instance": "", 2023-05-31T20:38:25.100 INFO:teuthology.orchestra.run.smithi099.stdout: "ver": { 2023-05-31T20:38:25.100 INFO:teuthology.orchestra.run.smithi099.stdout: "pool": 13, 2023-05-31T20:38:25.101 INFO:teuthology.orchestra.run.smithi099.stdout: "epoch": 4 2023-05-31T20:38:25.101 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:38:25.101 INFO:teuthology.orchestra.run.smithi099.stdout: "locator": "", 2023-05-31T20:38:25.101 INFO:teuthology.orchestra.run.smithi099.stdout: "exists": true, 2023-05-31T20:38:25.101 INFO:teuthology.orchestra.run.smithi099.stdout: "meta": { 2023-05-31T20:38:25.101 INFO:teuthology.orchestra.run.smithi099.stdout: "category": 1, 2023-05-31T20:38:25.102 INFO:teuthology.orchestra.run.smithi099.stdout: "size": 16, 2023-05-31T20:38:25.102 INFO:teuthology.orchestra.run.smithi099.stdout: "mtime": "2023-05-31T20:38:22.575673Z", 2023-05-31T20:38:25.102 INFO:teuthology.orchestra.run.smithi099.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-05-31T20:38:25.102 INFO:teuthology.orchestra.run.smithi099.stdout: "storage_class": "", 2023-05-31T20:38:25.102 INFO:teuthology.orchestra.run.smithi099.stdout: "owner": "rgw-multisite-test-user", 2023-05-31T20:38:25.102 INFO:teuthology.orchestra.run.smithi099.stdout: "owner_display_name": "Multisite Test User", 2023-05-31T20:38:25.102 INFO:teuthology.orchestra.run.smithi099.stdout: "content_type": "application/octet-stream", 2023-05-31T20:38:25.103 INFO:teuthology.orchestra.run.smithi099.stdout: "accounted_size": 3, 2023-05-31T20:38:25.103 INFO:teuthology.orchestra.run.smithi099.stdout: "user_data": "", 2023-05-31T20:38:25.103 INFO:teuthology.orchestra.run.smithi099.stdout: "appendable": false 2023-05-31T20:38:25.103 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:38:25.103 INFO:teuthology.orchestra.run.smithi099.stdout: "tag": "_zgmISPavJ-X0KMemW_7ZBwXeKIdv32U", 2023-05-31T20:38:25.104 INFO:teuthology.orchestra.run.smithi099.stdout: "flags": 0, 2023-05-31T20:38:25.104 INFO:teuthology.orchestra.run.smithi099.stdout: "pending_map": [], 2023-05-31T20:38:25.104 INFO:teuthology.orchestra.run.smithi099.stdout: "versioned_epoch": 0 2023-05-31T20:38:25.104 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:38:25.104 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T20:38:25.104 INFO:teuthology.orchestra.run.smithi099.stdout: "name": "obj00000000000000000061", 2023-05-31T20:38:25.104 INFO:teuthology.orchestra.run.smithi099.stdout: "instance": "", 2023-05-31T20:38:25.105 INFO:teuthology.orchestra.run.smithi099.stdout: "ver": { 2023-05-31T20:38:25.105 INFO:teuthology.orchestra.run.smithi099.stdout: "pool": 13, 2023-05-31T20:38:25.105 INFO:teuthology.orchestra.run.smithi099.stdout: "epoch": 2 2023-05-31T20:38:25.105 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:38:25.105 INFO:teuthology.orchestra.run.smithi099.stdout: "locator": "", 2023-05-31T20:38:25.105 INFO:teuthology.orchestra.run.smithi099.stdout: "exists": true, 2023-05-31T20:38:25.105 INFO:teuthology.orchestra.run.smithi099.stdout: "meta": { 2023-05-31T20:38:25.106 INFO:teuthology.orchestra.run.smithi099.stdout: "category": 1, 2023-05-31T20:38:25.106 INFO:teuthology.orchestra.run.smithi099.stdout: "size": 5, 2023-05-31T20:38:25.106 INFO:teuthology.orchestra.run.smithi099.stdout: "mtime": "2023-05-31T20:38:21.154093Z", 2023-05-31T20:38:25.106 INFO:teuthology.orchestra.run.smithi099.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-05-31T20:38:25.106 INFO:teuthology.orchestra.run.smithi099.stdout: "storage_class": "", 2023-05-31T20:38:25.106 INFO:teuthology.orchestra.run.smithi099.stdout: "owner": "rgw-multisite-test-user", 2023-05-31T20:38:25.107 INFO:teuthology.orchestra.run.smithi099.stdout: "owner_display_name": "Multisite Test User", 2023-05-31T20:38:25.107 INFO:teuthology.orchestra.run.smithi099.stdout: "content_type": "application/octet-stream", 2023-05-31T20:38:25.107 INFO:teuthology.orchestra.run.smithi099.stdout: "accounted_size": 3, 2023-05-31T20:38:25.107 INFO:teuthology.orchestra.run.smithi099.stdout: "user_data": "", 2023-05-31T20:38:25.107 INFO:teuthology.orchestra.run.smithi099.stdout: "appendable": false 2023-05-31T20:38:25.107 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:38:25.108 INFO:teuthology.orchestra.run.smithi099.stdout: "tag": "_lKSrZtjWfWtIojTtp3A4fEwgazQV9Bq", 2023-05-31T20:38:25.108 INFO:teuthology.orchestra.run.smithi099.stdout: "flags": 0, 2023-05-31T20:38:25.108 INFO:teuthology.orchestra.run.smithi099.stdout: "pending_map": [], 2023-05-31T20:38:25.108 INFO:teuthology.orchestra.run.smithi099.stdout: "versioned_epoch": 0 2023-05-31T20:38:25.108 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:38:25.108 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T20:38:25.108 INFO:teuthology.orchestra.run.smithi099.stdout: "name": "obj00000000000000000062", 2023-05-31T20:38:25.109 INFO:teuthology.orchestra.run.smithi099.stdout: "instance": "", 2023-05-31T20:38:25.109 INFO:teuthology.orchestra.run.smithi099.stdout: "ver": { 2023-05-31T20:38:25.109 INFO:teuthology.orchestra.run.smithi099.stdout: "pool": 13, 2023-05-31T20:38:25.109 INFO:teuthology.orchestra.run.smithi099.stdout: "epoch": 3 2023-05-31T20:38:25.109 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:38:25.109 INFO:teuthology.orchestra.run.smithi099.stdout: "locator": "", 2023-05-31T20:38:25.109 INFO:teuthology.orchestra.run.smithi099.stdout: "exists": true, 2023-05-31T20:38:25.110 INFO:teuthology.orchestra.run.smithi099.stdout: "meta": { 2023-05-31T20:38:25.110 INFO:teuthology.orchestra.run.smithi099.stdout: "category": 1, 2023-05-31T20:38:25.110 INFO:teuthology.orchestra.run.smithi099.stdout: "size": 5, 2023-05-31T20:38:25.110 INFO:teuthology.orchestra.run.smithi099.stdout: "mtime": "2023-05-31T20:38:21.794173Z", 2023-05-31T20:38:25.110 INFO:teuthology.orchestra.run.smithi099.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-05-31T20:38:25.110 INFO:teuthology.orchestra.run.smithi099.stdout: "storage_class": "", 2023-05-31T20:38:25.110 INFO:teuthology.orchestra.run.smithi099.stdout: "owner": "rgw-multisite-test-user", 2023-05-31T20:38:25.111 INFO:teuthology.orchestra.run.smithi099.stdout: "owner_display_name": "Multisite Test User", 2023-05-31T20:38:25.111 INFO:teuthology.orchestra.run.smithi099.stdout: "content_type": "application/octet-stream", 2023-05-31T20:38:25.111 INFO:teuthology.orchestra.run.smithi099.stdout: "accounted_size": 3, 2023-05-31T20:38:25.111 INFO:teuthology.orchestra.run.smithi099.stdout: "user_data": "", 2023-05-31T20:38:25.111 INFO:teuthology.orchestra.run.smithi099.stdout: "appendable": false 2023-05-31T20:38:25.111 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:38:25.111 INFO:teuthology.orchestra.run.smithi099.stdout: "tag": "_fpkckLHjwPki8JVhmVut7aibkhXXEC3", 2023-05-31T20:38:25.112 INFO:teuthology.orchestra.run.smithi099.stdout: "flags": 0, 2023-05-31T20:38:25.112 INFO:teuthology.orchestra.run.smithi099.stdout: "pending_map": [], 2023-05-31T20:38:25.112 INFO:teuthology.orchestra.run.smithi099.stdout: "versioned_epoch": 0 2023-05-31T20:38:25.112 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:38:25.112 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T20:38:25.112 INFO:teuthology.orchestra.run.smithi099.stdout: "name": "obj00000000000000000065", 2023-05-31T20:38:25.112 INFO:teuthology.orchestra.run.smithi099.stdout: "instance": "", 2023-05-31T20:38:25.113 INFO:teuthology.orchestra.run.smithi099.stdout: "ver": { 2023-05-31T20:38:25.113 INFO:teuthology.orchestra.run.smithi099.stdout: "pool": 13, 2023-05-31T20:38:25.113 INFO:teuthology.orchestra.run.smithi099.stdout: "epoch": 5 2023-05-31T20:38:25.113 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:38:25.113 INFO:teuthology.orchestra.run.smithi099.stdout: "locator": "", 2023-05-31T20:38:25.113 INFO:teuthology.orchestra.run.smithi099.stdout: "exists": true, 2023-05-31T20:38:25.113 INFO:teuthology.orchestra.run.smithi099.stdout: "meta": { 2023-05-31T20:38:25.114 INFO:teuthology.orchestra.run.smithi099.stdout: "category": 1, 2023-05-31T20:38:25.114 INFO:teuthology.orchestra.run.smithi099.stdout: "size": 3, 2023-05-31T20:38:25.114 INFO:teuthology.orchestra.run.smithi099.stdout: "mtime": "2023-05-31T20:38:22.003695Z", 2023-05-31T20:38:25.114 INFO:teuthology.orchestra.run.smithi099.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-05-31T20:38:25.114 INFO:teuthology.orchestra.run.smithi099.stdout: "storage_class": "", 2023-05-31T20:38:25.114 INFO:teuthology.orchestra.run.smithi099.stdout: "owner": "rgw-multisite-test-user", 2023-05-31T20:38:25.114 INFO:teuthology.orchestra.run.smithi099.stdout: "owner_display_name": "Multisite Test User", 2023-05-31T20:38:25.115 INFO:teuthology.orchestra.run.smithi099.stdout: "content_type": "application/octet-stream", 2023-05-31T20:38:25.115 INFO:teuthology.orchestra.run.smithi099.stdout: "accounted_size": 3, 2023-05-31T20:38:25.115 INFO:teuthology.orchestra.run.smithi099.stdout: "user_data": "", 2023-05-31T20:38:25.115 INFO:teuthology.orchestra.run.smithi099.stdout: "appendable": false 2023-05-31T20:38:25.115 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:38:25.115 INFO:teuthology.orchestra.run.smithi099.stdout: "tag": "_Y7tnOQpDAWi9pTD8n2xYWdvsJXFCHQw", 2023-05-31T20:38:25.116 INFO:teuthology.orchestra.run.smithi099.stdout: "flags": 0, 2023-05-31T20:38:25.116 INFO:teuthology.orchestra.run.smithi099.stdout: "pending_map": [], 2023-05-31T20:38:25.116 INFO:teuthology.orchestra.run.smithi099.stdout: "versioned_epoch": 0 2023-05-31T20:38:25.116 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:38:25.116 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T20:38:25.116 INFO:teuthology.orchestra.run.smithi099.stdout: "name": "obj00000000000000000133", 2023-05-31T20:38:25.117 INFO:teuthology.orchestra.run.smithi099.stdout: "instance": "", 2023-05-31T20:38:25.117 INFO:teuthology.orchestra.run.smithi099.stdout: "ver": { 2023-05-31T20:38:25.117 INFO:teuthology.orchestra.run.smithi099.stdout: "pool": 13, 2023-05-31T20:38:25.117 INFO:teuthology.orchestra.run.smithi099.stdout: "epoch": 7 2023-05-31T20:38:25.117 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:38:25.117 INFO:teuthology.orchestra.run.smithi099.stdout: "locator": "", 2023-05-31T20:38:25.117 INFO:teuthology.orchestra.run.smithi099.stdout: "exists": true, 2023-05-31T20:38:25.118 INFO:teuthology.orchestra.run.smithi099.stdout: "meta": { 2023-05-31T20:38:25.118 INFO:teuthology.orchestra.run.smithi099.stdout: "category": 1, 2023-05-31T20:38:25.118 INFO:teuthology.orchestra.run.smithi099.stdout: "size": 3, 2023-05-31T20:38:25.118 INFO:teuthology.orchestra.run.smithi099.stdout: "mtime": "2023-05-31T20:38:19.997509Z", 2023-05-31T20:38:25.118 INFO:teuthology.orchestra.run.smithi099.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-05-31T20:38:25.118 INFO:teuthology.orchestra.run.smithi099.stdout: "storage_class": "", 2023-05-31T20:38:25.119 INFO:teuthology.orchestra.run.smithi099.stdout: "owner": "rgw-multisite-test-user", 2023-05-31T20:38:25.119 INFO:teuthology.orchestra.run.smithi099.stdout: "owner_display_name": "Multisite Test User", 2023-05-31T20:38:25.119 INFO:teuthology.orchestra.run.smithi099.stdout: "content_type": "application/octet-stream", 2023-05-31T20:38:25.119 INFO:teuthology.orchestra.run.smithi099.stdout: "accounted_size": 3, 2023-05-31T20:38:25.119 INFO:teuthology.orchestra.run.smithi099.stdout: "user_data": "", 2023-05-31T20:38:25.119 INFO:teuthology.orchestra.run.smithi099.stdout: "appendable": false 2023-05-31T20:38:25.119 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:38:25.120 INFO:teuthology.orchestra.run.smithi099.stdout: "tag": "_OfmsfpYR6smrdVeUgSq1QR1kdm7-uWw", 2023-05-31T20:38:25.120 INFO:teuthology.orchestra.run.smithi099.stdout: "flags": 0, 2023-05-31T20:38:25.120 INFO:teuthology.orchestra.run.smithi099.stdout: "pending_map": [], 2023-05-31T20:38:25.120 INFO:teuthology.orchestra.run.smithi099.stdout: "versioned_epoch": 0 2023-05-31T20:38:25.120 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:38:25.120 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T20:38:25.120 INFO:teuthology.orchestra.run.smithi099.stdout: "name": "obj00000000000000000162", 2023-05-31T20:38:25.121 INFO:teuthology.orchestra.run.smithi099.stdout: "instance": "", 2023-05-31T20:38:25.121 INFO:teuthology.orchestra.run.smithi099.stdout: "ver": { 2023-05-31T20:38:25.121 INFO:teuthology.orchestra.run.smithi099.stdout: "pool": 13, 2023-05-31T20:38:25.121 INFO:teuthology.orchestra.run.smithi099.stdout: "epoch": 3 2023-05-31T20:38:25.121 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:38:25.121 INFO:teuthology.orchestra.run.smithi099.stdout: "locator": "", 2023-05-31T20:38:25.122 INFO:teuthology.orchestra.run.smithi099.stdout: "exists": true, 2023-05-31T20:38:25.122 INFO:teuthology.orchestra.run.smithi099.stdout: "meta": { 2023-05-31T20:38:25.122 INFO:teuthology.orchestra.run.smithi099.stdout: "category": 1, 2023-05-31T20:38:25.122 INFO:teuthology.orchestra.run.smithi099.stdout: "size": 3, 2023-05-31T20:38:25.122 INFO:teuthology.orchestra.run.smithi099.stdout: "mtime": "2023-05-31T20:38:20.538625Z", 2023-05-31T20:38:25.122 INFO:teuthology.orchestra.run.smithi099.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-05-31T20:38:25.123 INFO:teuthology.orchestra.run.smithi099.stdout: "storage_class": "", 2023-05-31T20:38:25.123 INFO:teuthology.orchestra.run.smithi099.stdout: "owner": "rgw-multisite-test-user", 2023-05-31T20:38:25.123 INFO:teuthology.orchestra.run.smithi099.stdout: "owner_display_name": "Multisite Test User", 2023-05-31T20:38:25.123 INFO:teuthology.orchestra.run.smithi099.stdout: "content_type": "application/octet-stream", 2023-05-31T20:38:25.123 INFO:teuthology.orchestra.run.smithi099.stdout: "accounted_size": 3, 2023-05-31T20:38:25.123 INFO:teuthology.orchestra.run.smithi099.stdout: "user_data": "", 2023-05-31T20:38:25.123 INFO:teuthology.orchestra.run.smithi099.stdout: "appendable": false 2023-05-31T20:38:25.124 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:38:25.124 INFO:teuthology.orchestra.run.smithi099.stdout: "tag": "_QdRwwlgapxCkpXyfKPUCPErWRjprgbJ", 2023-05-31T20:38:25.124 INFO:teuthology.orchestra.run.smithi099.stdout: "flags": 0, 2023-05-31T20:38:25.124 INFO:teuthology.orchestra.run.smithi099.stdout: "pending_map": [], 2023-05-31T20:38:25.124 INFO:teuthology.orchestra.run.smithi099.stdout: "versioned_epoch": 0 2023-05-31T20:38:25.124 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:38:25.124 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T20:38:25.126 INFO:teuthology.orchestra.run.smithi099.stdout: "name": "obj00000000000000000271", 2023-05-31T20:38:25.126 INFO:teuthology.orchestra.run.smithi099.stdout: "instance": "", 2023-05-31T20:38:25.126 INFO:teuthology.orchestra.run.smithi099.stdout: "ver": { 2023-05-31T20:38:25.126 INFO:teuthology.orchestra.run.smithi099.stdout: "pool": 13, 2023-05-31T20:38:25.126 INFO:teuthology.orchestra.run.smithi099.stdout: "epoch": 3 2023-05-31T20:38:25.127 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:38:25.127 INFO:teuthology.orchestra.run.smithi099.stdout: "locator": "", 2023-05-31T20:38:25.127 INFO:teuthology.orchestra.run.smithi099.stdout: "exists": true, 2023-05-31T20:38:25.127 INFO:teuthology.orchestra.run.smithi099.stdout: "meta": { 2023-05-31T20:38:25.127 INFO:teuthology.orchestra.run.smithi099.stdout: "category": 1, 2023-05-31T20:38:25.127 INFO:teuthology.orchestra.run.smithi099.stdout: "size": 16, 2023-05-31T20:38:25.127 INFO:teuthology.orchestra.run.smithi099.stdout: "mtime": "2023-05-31T20:38:20.240943Z", 2023-05-31T20:38:25.128 INFO:teuthology.orchestra.run.smithi099.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-05-31T20:38:25.128 INFO:teuthology.orchestra.run.smithi099.stdout: "storage_class": "", 2023-05-31T20:38:25.128 INFO:teuthology.orchestra.run.smithi099.stdout: "owner": "rgw-multisite-test-user", 2023-05-31T20:38:25.128 INFO:teuthology.orchestra.run.smithi099.stdout: "owner_display_name": "Multisite Test User", 2023-05-31T20:38:25.128 INFO:teuthology.orchestra.run.smithi099.stdout: "content_type": "application/octet-stream", 2023-05-31T20:38:25.129 INFO:teuthology.orchestra.run.smithi099.stdout: "accounted_size": 3, 2023-05-31T20:38:25.129 INFO:teuthology.orchestra.run.smithi099.stdout: "user_data": "", 2023-05-31T20:38:25.129 INFO:teuthology.orchestra.run.smithi099.stdout: "appendable": false 2023-05-31T20:38:25.129 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:38:25.129 INFO:teuthology.orchestra.run.smithi099.stdout: "tag": "_nF5C9MhDphFYD6ajF-nOW-J_6f3_lbl", 2023-05-31T20:38:25.129 INFO:teuthology.orchestra.run.smithi099.stdout: "flags": 0, 2023-05-31T20:38:25.129 INFO:teuthology.orchestra.run.smithi099.stdout: "pending_map": [], 2023-05-31T20:38:25.130 INFO:teuthology.orchestra.run.smithi099.stdout: "versioned_epoch": 0 2023-05-31T20:38:25.130 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:38:25.130 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T20:38:25.130 INFO:teuthology.orchestra.run.smithi099.stdout: "name": "obj00000000000000000283", 2023-05-31T20:38:25.130 INFO:teuthology.orchestra.run.smithi099.stdout: "instance": "", 2023-05-31T20:38:25.130 INFO:teuthology.orchestra.run.smithi099.stdout: "ver": { 2023-05-31T20:38:25.130 INFO:teuthology.orchestra.run.smithi099.stdout: "pool": 13, 2023-05-31T20:38:25.131 INFO:teuthology.orchestra.run.smithi099.stdout: "epoch": 6 2023-05-31T20:38:25.131 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:38:25.131 INFO:teuthology.orchestra.run.smithi099.stdout: "locator": "", 2023-05-31T20:38:25.131 INFO:teuthology.orchestra.run.smithi099.stdout: "exists": true, 2023-05-31T20:38:25.131 INFO:teuthology.orchestra.run.smithi099.stdout: "meta": { 2023-05-31T20:38:25.131 INFO:teuthology.orchestra.run.smithi099.stdout: "category": 1, 2023-05-31T20:38:25.132 INFO:teuthology.orchestra.run.smithi099.stdout: "size": 5, 2023-05-31T20:38:25.132 INFO:teuthology.orchestra.run.smithi099.stdout: "mtime": "2023-05-31T20:38:20.953383Z", 2023-05-31T20:38:25.132 INFO:teuthology.orchestra.run.smithi099.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-05-31T20:38:25.132 INFO:teuthology.orchestra.run.smithi099.stdout: "storage_class": "", 2023-05-31T20:38:25.132 INFO:teuthology.orchestra.run.smithi099.stdout: "owner": "rgw-multisite-test-user", 2023-05-31T20:38:25.132 INFO:teuthology.orchestra.run.smithi099.stdout: "owner_display_name": "Multisite Test User", 2023-05-31T20:38:25.133 INFO:teuthology.orchestra.run.smithi099.stdout: "content_type": "application/octet-stream", 2023-05-31T20:38:25.133 INFO:teuthology.orchestra.run.smithi099.stdout: "accounted_size": 3, 2023-05-31T20:38:25.133 INFO:teuthology.orchestra.run.smithi099.stdout: "user_data": "", 2023-05-31T20:38:25.133 INFO:teuthology.orchestra.run.smithi099.stdout: "appendable": false 2023-05-31T20:38:25.133 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:38:25.133 INFO:teuthology.orchestra.run.smithi099.stdout: "tag": "_0UTEf2kMKIGX5y5t6NI9qLSFSm9VSwP", 2023-05-31T20:38:25.134 INFO:teuthology.orchestra.run.smithi099.stdout: "flags": 0, 2023-05-31T20:38:25.134 INFO:teuthology.orchestra.run.smithi099.stdout: "pending_map": [], 2023-05-31T20:38:25.134 INFO:teuthology.orchestra.run.smithi099.stdout: "versioned_epoch": 0 2023-05-31T20:38:25.134 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:38:25.134 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T20:38:25.134 INFO:teuthology.orchestra.run.smithi099.stdout: "name": "obj00000000000000000317", 2023-05-31T20:38:25.134 INFO:teuthology.orchestra.run.smithi099.stdout: "instance": "", 2023-05-31T20:38:25.135 INFO:teuthology.orchestra.run.smithi099.stdout: "ver": { 2023-05-31T20:38:25.135 INFO:teuthology.orchestra.run.smithi099.stdout: "pool": 13, 2023-05-31T20:38:25.135 INFO:teuthology.orchestra.run.smithi099.stdout: "epoch": 4 2023-05-31T20:38:25.135 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:38:25.135 INFO:teuthology.orchestra.run.smithi099.stdout: "locator": "", 2023-05-31T20:38:25.135 INFO:teuthology.orchestra.run.smithi099.stdout: "exists": true, 2023-05-31T20:38:25.135 INFO:teuthology.orchestra.run.smithi099.stdout: "meta": { 2023-05-31T20:38:25.136 INFO:teuthology.orchestra.run.smithi099.stdout: "category": 1, 2023-05-31T20:38:25.136 INFO:teuthology.orchestra.run.smithi099.stdout: "size": 16, 2023-05-31T20:38:25.136 INFO:teuthology.orchestra.run.smithi099.stdout: "mtime": "2023-05-31T20:38:20.065984Z", 2023-05-31T20:38:25.136 INFO:teuthology.orchestra.run.smithi099.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-05-31T20:38:25.136 INFO:teuthology.orchestra.run.smithi099.stdout: "storage_class": "", 2023-05-31T20:38:25.136 INFO:teuthology.orchestra.run.smithi099.stdout: "owner": "rgw-multisite-test-user", 2023-05-31T20:38:25.137 INFO:teuthology.orchestra.run.smithi099.stdout: "owner_display_name": "Multisite Test User", 2023-05-31T20:38:25.137 INFO:teuthology.orchestra.run.smithi099.stdout: "content_type": "application/octet-stream", 2023-05-31T20:38:25.137 INFO:teuthology.orchestra.run.smithi099.stdout: "accounted_size": 3, 2023-05-31T20:38:25.137 INFO:teuthology.orchestra.run.smithi099.stdout: "user_data": "", 2023-05-31T20:38:25.137 INFO:teuthology.orchestra.run.smithi099.stdout: "appendable": false 2023-05-31T20:38:25.137 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:38:25.137 INFO:teuthology.orchestra.run.smithi099.stdout: "tag": "_sNFXQOZNhTusUnLmrP95bLPaKWo9cF7", 2023-05-31T20:38:25.138 INFO:teuthology.orchestra.run.smithi099.stdout: "flags": 0, 2023-05-31T20:38:25.138 INFO:teuthology.orchestra.run.smithi099.stdout: "pending_map": [], 2023-05-31T20:38:25.138 INFO:teuthology.orchestra.run.smithi099.stdout: "versioned_epoch": 0 2023-05-31T20:38:25.138 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:38:25.138 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T20:38:25.138 INFO:teuthology.orchestra.run.smithi099.stdout: "name": "obj00000000000000000423", 2023-05-31T20:38:25.138 INFO:teuthology.orchestra.run.smithi099.stdout: "instance": "", 2023-05-31T20:38:25.139 INFO:teuthology.orchestra.run.smithi099.stdout: "ver": { 2023-05-31T20:38:25.139 INFO:teuthology.orchestra.run.smithi099.stdout: "pool": 13, 2023-05-31T20:38:25.139 INFO:teuthology.orchestra.run.smithi099.stdout: "epoch": 2 2023-05-31T20:38:25.139 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:38:25.139 INFO:teuthology.orchestra.run.smithi099.stdout: "locator": "", 2023-05-31T20:38:25.139 INFO:teuthology.orchestra.run.smithi099.stdout: "exists": true, 2023-05-31T20:38:25.139 INFO:teuthology.orchestra.run.smithi099.stdout: "meta": { 2023-05-31T20:38:25.140 INFO:teuthology.orchestra.run.smithi099.stdout: "category": 1, 2023-05-31T20:38:25.140 INFO:teuthology.orchestra.run.smithi099.stdout: "size": 16, 2023-05-31T20:38:25.140 INFO:teuthology.orchestra.run.smithi099.stdout: "mtime": "2023-05-31T20:38:20.434035Z", 2023-05-31T20:38:25.140 INFO:teuthology.orchestra.run.smithi099.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-05-31T20:38:25.140 INFO:teuthology.orchestra.run.smithi099.stdout: "storage_class": "", 2023-05-31T20:38:25.140 INFO:teuthology.orchestra.run.smithi099.stdout: "owner": "rgw-multisite-test-user", 2023-05-31T20:38:25.141 INFO:teuthology.orchestra.run.smithi099.stdout: "owner_display_name": "Multisite Test User", 2023-05-31T20:38:25.141 INFO:teuthology.orchestra.run.smithi099.stdout: "content_type": "application/octet-stream", 2023-05-31T20:38:25.141 INFO:teuthology.orchestra.run.smithi099.stdout: "accounted_size": 3, 2023-05-31T20:38:25.141 INFO:teuthology.orchestra.run.smithi099.stdout: "user_data": "", 2023-05-31T20:38:25.141 INFO:teuthology.orchestra.run.smithi099.stdout: "appendable": false 2023-05-31T20:38:25.141 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:38:25.141 INFO:teuthology.orchestra.run.smithi099.stdout: "tag": "_7mxz3NCMBtq2-wWTtOzaB1La2_p3_cF", 2023-05-31T20:38:25.142 INFO:teuthology.orchestra.run.smithi099.stdout: "flags": 0, 2023-05-31T20:38:25.142 INFO:teuthology.orchestra.run.smithi099.stdout: "pending_map": [], 2023-05-31T20:38:25.142 INFO:teuthology.orchestra.run.smithi099.stdout: "versioned_epoch": 0 2023-05-31T20:38:25.142 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:38:25.142 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T20:38:25.142 INFO:teuthology.orchestra.run.smithi099.stdout: "name": "obj00000000000000000467", 2023-05-31T20:38:25.142 INFO:teuthology.orchestra.run.smithi099.stdout: "instance": "", 2023-05-31T20:38:25.143 INFO:teuthology.orchestra.run.smithi099.stdout: "ver": { 2023-05-31T20:38:25.143 INFO:teuthology.orchestra.run.smithi099.stdout: "pool": 13, 2023-05-31T20:38:25.143 INFO:teuthology.orchestra.run.smithi099.stdout: "epoch": 4 2023-05-31T20:38:25.143 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:38:25.143 INFO:teuthology.orchestra.run.smithi099.stdout: "locator": "", 2023-05-31T20:38:25.143 INFO:teuthology.orchestra.run.smithi099.stdout: "exists": true, 2023-05-31T20:38:25.144 INFO:teuthology.orchestra.run.smithi099.stdout: "meta": { 2023-05-31T20:38:25.144 INFO:teuthology.orchestra.run.smithi099.stdout: "category": 1, 2023-05-31T20:38:25.144 INFO:teuthology.orchestra.run.smithi099.stdout: "size": 5, 2023-05-31T20:38:25.144 INFO:teuthology.orchestra.run.smithi099.stdout: "mtime": "2023-05-31T20:38:20.327690Z", 2023-05-31T20:38:25.144 INFO:teuthology.orchestra.run.smithi099.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-05-31T20:38:25.144 INFO:teuthology.orchestra.run.smithi099.stdout: "storage_class": "", 2023-05-31T20:38:25.144 INFO:teuthology.orchestra.run.smithi099.stdout: "owner": "rgw-multisite-test-user", 2023-05-31T20:38:25.145 INFO:teuthology.orchestra.run.smithi099.stdout: "owner_display_name": "Multisite Test User", 2023-05-31T20:38:25.145 INFO:teuthology.orchestra.run.smithi099.stdout: "content_type": "application/octet-stream", 2023-05-31T20:38:25.145 INFO:teuthology.orchestra.run.smithi099.stdout: "accounted_size": 3, 2023-05-31T20:38:25.145 INFO:teuthology.orchestra.run.smithi099.stdout: "user_data": "", 2023-05-31T20:38:25.145 INFO:teuthology.orchestra.run.smithi099.stdout: "appendable": false 2023-05-31T20:38:25.145 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:38:25.146 INFO:teuthology.orchestra.run.smithi099.stdout: "tag": "_mpejFbdOk-YVB9zWmoiVVuhYy6uG-Ct", 2023-05-31T20:38:25.146 INFO:teuthology.orchestra.run.smithi099.stdout: "flags": 0, 2023-05-31T20:38:25.146 INFO:teuthology.orchestra.run.smithi099.stdout: "pending_map": [], 2023-05-31T20:38:25.146 INFO:teuthology.orchestra.run.smithi099.stdout: "versioned_epoch": 0 2023-05-31T20:38:25.146 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T20:38:25.146 INFO:teuthology.orchestra.run.smithi099.stdout:] 2023-05-31T20:38:25.147 INFO:tasks.rgw.c1.client.1:Restarting daemon 2023-05-31T20:38:25.148 DEBUG:teuthology.orchestra.run.smithi099:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term radosgw --rgw-frontends 'beast port=8001' -n client.1 --cluster c1 -k /etc/ceph/c1.client.1.keyring --log-file /var/log/ceph/rgw.c1.client.1.log --rgw_ops_log_socket_path /home/ubuntu/cephtest/rgw.opslog.c1.client.1.sock --foreground --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm | sudo tee /var/log/ceph/rgw.c1.client.1.stdout 2>&1 2023-05-31T20:38:25.151 INFO:tasks.rgw.c1.client.1:Started 2023-05-31T20:38:25.151 DEBUG:teuthology.orchestra.run.smithi099:> curl http://smithi099.front.sepia.ceph.com:8001 2023-05-31T20:38:25.172 DEBUG:teuthology.orchestra.run:got remote process result: 7 2023-05-31T20:38:25.172 INFO:teuthology.orchestra.run.smithi099.stderr: % Total % Received % Xferd Average Speed Time Time Time Current 2023-05-31T20:38:25.173 INFO:teuthology.orchestra.run.smithi099.stderr: Dload Upload Total Spent Left Speed 2023-05-31T20:38:25.173 INFO:teuthology.orchestra.run.smithi099.stderr: 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0curl: (7) Failed to connect to smithi099.front.sepia.ceph.com port 8001: Connection refused 2023-05-31T20:38:26.174 DEBUG:teuthology.orchestra.run.smithi099:> curl http://smithi099.front.sepia.ceph.com:8001 2023-05-31T20:38:26.195 INFO:teuthology.orchestra.run.smithi099.stderr: % Total % Received % Xferd Average Speed Time Time Time Current 2023-05-31T20:38:26.196 INFO:teuthology.orchestra.run.smithi099.stderr: Dload Upload Total Spent Left Speed 2023-05-31T20:38:26.196 INFO:teuthology.orchestra.run.smithi099.stderr: 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 100 214 0 214 0 0 208k 0 --:--:-- --:--:-- --:--:-- 208k 2023-05-31T20:38:26.197 INFO:teuthology.orchestra.run.smithi099.stdout:anonymous 2023-05-31T20:38:26.197 DEBUG:rgw_multi.tests:bucket checkpoint: source=a1 target=a2 bucket=pklxkd-58 2023-05-31T20:38:26.198 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'pklxkd-58', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-05-31T20:38:26.198 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'pklxkd-58', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-05-31T20:38:26.198 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket pklxkd-58 --source-zone a1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2023-05-31T20:38:26.293 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2023-05-31T20:38:26.294 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2023-05-31T20:38:26.340 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:26.335+0000 7f887c9fe780 1 bucket sync caught up with source: 2023-05-31T20:38:26.340 INFO:teuthology.orchestra.run.smithi099.stderr: local status: [00000000001.58.6, 00000000002.74.6, 00000000002.52.6, 00000000002.88.6, 00000000002.91.6, 00000000001.58.6, 00000000002.62.6, 00000000002.41.6, 00000000001.48.6, 00000000002.30.6, 00000000002.56.6, 00000000002.37.6, 00000000001.86.6, 00000000002.60.6, 00000000002.62.6, 00000000002.61.6, 00000000002.47.6, 00000000002.58.6, 00000000002.53.6, 00000000002.47.6, 00000000002.70.6, 00000000002.88.6, 00000000002.55.6, 00000000002.59.6, 00000000002.55.6, , , 00000000002.93.6, 00000000002.58.6, 00000000002.51.6, 00000000001.90.6, 00000000002.46.6, 00000000002.60.6, 00000000002.50.6, 00000000001.56.6, 00000000002.52.6, ] 2023-05-31T20:38:26.340 INFO:teuthology.orchestra.run.smithi099.stderr: remote markers: [00000000001.58.6, 00000000002.74.6, 00000000002.52.6, 00000000002.88.6, 00000000002.91.6, 00000000001.58.6, 00000000002.62.6, 00000000002.41.6, 00000000001.48.6, 00000000002.30.6, 00000000002.56.6, 00000000002.37.6, 00000000001.86.6, 00000000002.60.6, 00000000002.62.6, 00000000002.61.6, 00000000002.47.6, 00000000002.58.6, 00000000002.53.6, 00000000002.47.6, 00000000002.70.6, 00000000002.88.6, 00000000002.55.6, 00000000002.59.6, 00000000002.55.6, , , 00000000002.93.6, 00000000002.58.6, 00000000002.51.6, 00000000001.90.6, 00000000002.46.6, 00000000002.60.6, 00000000002.50.6, 00000000001.56.6, 00000000002.52.6, ] 2023-05-31T20:38:26.341 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:26.335+0000 7f887c9fe780 0 bucket checkpoint complete 2023-05-31T20:38:26.354 DEBUG:rgw_multi.tests:bucket checkpoint: source=a2 target=a1 bucket=pklxkd-58 2023-05-31T20:38:26.354 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'pklxkd-58', '--source-zone', 'a2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-05-31T20:38:26.354 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'pklxkd-58', '--source-zone', 'a2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-05-31T20:38:26.355 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket pklxkd-58 --source-zone a2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2023-05-31T20:38:26.411 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2023-05-31T20:38:26.412 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2023-05-31T20:38:26.450 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:26.445+0000 7fb4da1b3780 1 waiting for incremental sync to catch up: 2023-05-31T20:38:26.451 INFO:teuthology.orchestra.run.smithi099.stderr: local status: [00000000005.2259.6, 00000000006.3857.6, 00000000006.1923.6, 00000000001.776.6, 00000000005.1949.6, 00000000007.1934.6, 00000000008.1690.6, 00000000001.1934.6, 00000000007.1059.6, 00000000007.390.6, 00000000004.982.6] 2023-05-31T20:38:26.451 INFO:teuthology.orchestra.run.smithi099.stderr: remote markers: [00000000005.2259.6, 00000000006.3857.6, 00000000006.1923.6, 00000000010.794.6, 00000000005.1949.6, 00000000007.1934.6, 00000000008.1690.6, 00000000006.1953.6, 00000000007.1059.6, 00000000007.390.6, 00000000004.982.6] 2023-05-31T20:38:31.451 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:31.446+0000 7fb4da1b3780 1 waiting for incremental sync to catch up: 2023-05-31T20:38:31.451 INFO:teuthology.orchestra.run.smithi099.stderr: local status: [00000000005.2259.6, 00000000006.3857.6, 00000000006.1923.6, 00000000001.776.6, 00000000005.1949.6, 00000000007.1934.6, 00000000008.1690.6, 00000000001.1934.6, 00000000007.1059.6, 00000000007.390.6, 00000000004.982.6] 2023-05-31T20:38:31.452 INFO:teuthology.orchestra.run.smithi099.stderr: remote markers: [00000000005.2259.6, 00000000006.3857.6, 00000000006.1923.6, 00000000010.794.6, 00000000005.1949.6, 00000000007.1934.6, 00000000008.1690.6, 00000000006.1953.6, 00000000007.1059.6, 00000000007.390.6, 00000000004.982.6] 2023-05-31T20:38:36.452 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:36.447+0000 7fb4da1b3780 1 waiting for incremental sync to catch up: 2023-05-31T20:38:36.453 INFO:teuthology.orchestra.run.smithi099.stderr: local status: [00000000005.2259.6, 00000000006.3857.6, 00000000006.1923.6, 00000000001.776.6, 00000000005.1949.6, 00000000007.1934.6, 00000000008.1690.6, 00000000001.1934.6, 00000000007.1059.6, 00000000007.390.6, 00000000004.982.6] 2023-05-31T20:38:36.453 INFO:teuthology.orchestra.run.smithi099.stderr: remote markers: [00000000005.2259.6, 00000000006.3857.6, 00000000006.1923.6, 00000000010.794.6, 00000000005.1949.6, 00000000007.1934.6, 00000000008.1690.6, 00000000006.1953.6, 00000000007.1059.6, 00000000007.390.6, 00000000004.982.6] 2023-05-31T20:38:41.454 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:41.448+0000 7fb4da1b3780 1 waiting for incremental sync to catch up: 2023-05-31T20:38:41.454 INFO:teuthology.orchestra.run.smithi099.stderr: local status: [00000000005.2259.6, 00000000006.3857.6, 00000000006.1923.6, 00000000001.776.6, 00000000005.1949.6, 00000000007.1934.6, 00000000008.1690.6, 00000000001.1934.6, 00000000007.1059.6, 00000000007.390.6, 00000000004.982.6] 2023-05-31T20:38:41.454 INFO:teuthology.orchestra.run.smithi099.stderr: remote markers: [00000000005.2259.6, 00000000006.3857.6, 00000000006.1923.6, 00000000010.794.6, 00000000005.1949.6, 00000000007.1934.6, 00000000008.1690.6, 00000000006.1953.6, 00000000007.1059.6, 00000000007.390.6, 00000000004.982.6] 2023-05-31T20:38:46.455 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:46.450+0000 7fb4da1b3780 1 waiting for incremental sync to catch up: 2023-05-31T20:38:46.456 INFO:teuthology.orchestra.run.smithi099.stderr: local status: [00000000005.2259.6, 00000000006.3857.6, 00000000006.1923.6, 00000000001.776.6, 00000000005.1949.6, 00000000007.1934.6, 00000000008.1690.6, 00000000001.1934.6, 00000000007.1059.6, 00000000007.390.6, 00000000004.982.6] 2023-05-31T20:38:46.456 INFO:teuthology.orchestra.run.smithi099.stderr: remote markers: [00000000005.2259.6, 00000000006.3857.6, 00000000006.1923.6, 00000000010.794.6, 00000000005.1949.6, 00000000007.1934.6, 00000000008.1690.6, 00000000006.1953.6, 00000000007.1059.6, 00000000007.390.6, 00000000004.982.6] 2023-05-31T20:38:51.456 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:51.452+0000 7fb4da1b3780 1 waiting for incremental sync to catch up: 2023-05-31T20:38:51.457 INFO:teuthology.orchestra.run.smithi099.stderr: local status: [00000000005.2259.6, 00000000006.3857.6, 00000000006.1923.6, 00000000001.776.6, 00000000005.1949.6, 00000000007.1934.6, 00000000008.1690.6, 00000000001.1934.6, 00000000007.1059.6, 00000000007.390.6, 00000000004.982.6] 2023-05-31T20:38:51.457 INFO:teuthology.orchestra.run.smithi099.stderr: remote markers: [00000000005.2259.6, 00000000006.3857.6, 00000000006.1923.6, 00000000010.794.6, 00000000005.1949.6, 00000000007.1934.6, 00000000008.1690.6, 00000000006.1953.6, 00000000007.1059.6, 00000000007.390.6, 00000000004.982.6] 2023-05-31T20:38:56.458 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:38:56.453+0000 7fb4da1b3780 1 waiting for incremental sync to catch up: 2023-05-31T20:38:56.459 INFO:teuthology.orchestra.run.smithi099.stderr: local status: [00000000005.2259.6, 00000000006.3857.6, 00000000006.1923.6, 00000000001.776.6, 00000000005.1949.6, 00000000007.1934.6, 00000000008.1690.6, 00000000001.1934.6, 00000000007.1059.6, 00000000007.390.6, 00000000004.982.6] 2023-05-31T20:38:56.459 INFO:teuthology.orchestra.run.smithi099.stderr: remote markers: [00000000005.2259.6, 00000000006.3857.6, 00000000006.1923.6, 00000000010.794.6, 00000000005.1949.6, 00000000007.1934.6, 00000000008.1690.6, 00000000006.1953.6, 00000000007.1059.6, 00000000007.390.6, 00000000004.982.6] 2023-05-31T20:39:01.459 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:39:01.455+0000 7fb4da1b3780 1 waiting for incremental sync to catch up: 2023-05-31T20:39:01.460 INFO:teuthology.orchestra.run.smithi099.stderr: local status: [00000000005.2259.6, 00000000006.3857.6, 00000000006.1923.6, 00000000001.776.6, 00000000005.1949.6, 00000000007.1934.6, 00000000008.1690.6, 00000000001.1934.6, 00000000007.1059.6, 00000000007.390.6, 00000000004.982.6] 2023-05-31T20:39:01.460 INFO:teuthology.orchestra.run.smithi099.stderr: remote markers: [00000000005.2259.6, 00000000006.3857.6, 00000000006.1923.6, 00000000010.794.6, 00000000005.1949.6, 00000000007.1934.6, 00000000008.1690.6, 00000000006.1953.6, 00000000007.1059.6, 00000000007.390.6, 00000000004.982.6] 2023-05-31T20:39:06.460 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:39:06.456+0000 7fb4da1b3780 1 bucket sync caught up with source: 2023-05-31T20:39:06.461 INFO:teuthology.orchestra.run.smithi099.stderr: local status: [00000000005.2259.6, 00000000006.3857.6, 00000000006.1923.6, 00000000010.794.6, 00000000005.1949.6, 00000000007.1934.6, 00000000008.1690.6, 00000000006.1953.6, 00000000007.1059.6, 00000000007.390.6, 00000000004.982.6] 2023-05-31T20:39:06.461 INFO:teuthology.orchestra.run.smithi099.stderr: remote markers: [00000000005.2259.6, 00000000006.3857.6, 00000000006.1923.6, 00000000010.794.6, 00000000005.1949.6, 00000000007.1934.6, 00000000008.1690.6, 00000000006.1953.6, 00000000007.1059.6, 00000000007.390.6, 00000000004.982.6] 2023-05-31T20:39:06.461 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:39:06.456+0000 7fb4da1b3780 0 bucket checkpoint complete 2023-05-31T20:39:06.479 INFO:tasks.rgw_multi.zone_rados:comparing bucket=pklxkd-58 zones={a2, a1} 2023-05-31T20:39:06.480 DEBUG:boto:path=/pklxkd-58/ 2023-05-31T20:39:06.480 DEBUG:boto:auth_path=/pklxkd-58/ 2023-05-31T20:39:06.481 DEBUG:boto:Method: HEAD 2023-05-31T20:39:06.482 DEBUG:boto:Path: /pklxkd-58/ 2023-05-31T20:39:06.483 DEBUG:boto:Data: 2023-05-31T20:39:06.484 DEBUG:boto:Headers: {} 2023-05-31T20:39:06.486 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:39:06.486 DEBUG:boto:Port: 8001 2023-05-31T20:39:06.487 DEBUG:boto:Params: {} 2023-05-31T20:39:06.487 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8001} 2023-05-31T20:39:06.488 DEBUG:boto:Token: None 2023-05-31T20:39:06.488 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:39:06 GMT /pklxkd-58/ 2023-05-31T20:39:06.489 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:2ZMcPU22O5q9UpjGaX7+SoAJs7Y= 2023-05-31T20:39:06.489 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:39:06 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:2ZMcPU22O5q9UpjGaX7+SoAJs7Y=', 'Content-Length': '0'} 2023-05-31T20:39:06.495 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '61'), ('X-RGW-Bytes-Used', '183'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000000fe0b75794c717ba-006477b06a-1c17-a2'), ('Content-Length', '0'), ('Date', 'Wed, 31 May 2023 20:39:06 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:39:06.496 DEBUG:boto:path=/pklxkd-58/ 2023-05-31T20:39:06.496 DEBUG:boto:auth_path=/pklxkd-58/ 2023-05-31T20:39:06.497 DEBUG:boto:Method: HEAD 2023-05-31T20:39:06.497 DEBUG:boto:Path: /pklxkd-58/ 2023-05-31T20:39:06.497 DEBUG:boto:Data: 2023-05-31T20:39:06.498 DEBUG:boto:Headers: {} 2023-05-31T20:39:06.498 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:39:06.498 DEBUG:boto:Port: 8000 2023-05-31T20:39:06.499 DEBUG:boto:Params: {} 2023-05-31T20:39:06.499 DEBUG:boto:Token: None 2023-05-31T20:39:06.499 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:39:06 GMT /pklxkd-58/ 2023-05-31T20:39:06.500 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:2ZMcPU22O5q9UpjGaX7+SoAJs7Y= 2023-05-31T20:39:06.500 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:39:06 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:2ZMcPU22O5q9UpjGaX7+SoAJs7Y=', 'Content-Length': '0'} 2023-05-31T20:39:06.504 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '61'), ('X-RGW-Bytes-Used', '183'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000037537027b08736b-006477b06a-1630-a1'), ('Content-Length', '0'), ('Date', 'Wed, 31 May 2023 20:39:06 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:39:06.505 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2023-05-31T20:39:06.505 DEBUG:boto:path=/pklxkd-58/ 2023-05-31T20:39:06.505 DEBUG:boto:auth_path=/pklxkd-58/ 2023-05-31T20:39:06.506 DEBUG:boto:path=/pklxkd-58/?versions&max-keys=999 2023-05-31T20:39:06.506 DEBUG:boto:auth_path=/pklxkd-58/?versions&max-keys=999 2023-05-31T20:39:06.507 DEBUG:boto:Method: GET 2023-05-31T20:39:06.507 DEBUG:boto:Path: /pklxkd-58/?versions&max-keys=999 2023-05-31T20:39:06.507 DEBUG:boto:Data: 2023-05-31T20:39:06.508 DEBUG:boto:Headers: {} 2023-05-31T20:39:06.508 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:39:06.508 DEBUG:boto:Port: 8001 2023-05-31T20:39:06.509 DEBUG:boto:Params: {} 2023-05-31T20:39:06.509 DEBUG:boto:Token: None 2023-05-31T20:39:06.509 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:39:06 GMT /pklxkd-58/?versions 2023-05-31T20:39:06.510 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:zJ+RcxA7u8JmnQTXwxhXJG9x6RQ= 2023-05-31T20:39:06.510 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:39:06 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:zJ+RcxA7u8JmnQTXwxhXJG9x6RQ=', 'Content-Length': '0'} 2023-05-31T20:39:06.513 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000004556d7327b3a59d1-006477b06a-1c17-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:39:06 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:39:06.514 DEBUG:boto:b'pklxkd-58999falseobj00000000000000000000nulltrue2023-05-31T20:38:21.972Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000001nulltrue2023-05-31T20:38:22.169Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000002nulltrue2023-05-31T20:38:22.061Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000003nulltrue2023-05-31T20:38:21.910Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000004nulltrue2023-05-31T20:38:22.227Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000005nulltrue2023-05-31T20:38:21.709Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000006nulltrue2023-05-31T20:38:21.735Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000007nulltrue2023-05-31T20:38:22.793Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000008nulltrue2023-05-31T20:38:20.093Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000009nulltrue2023-05-31T20:38:21.588Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000010nulltrue2023-05-31T20:38:21.127Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000011nulltrue2023-05-31T20:38:22.712Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000012nulltrue2023-05-31T20:38:22.308Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000013nulltrue2023-05-31T20:38:22.736Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000014nulltrue2023-05-31T20:38:22.896Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000015nulltrue2023-05-31T20:38:22.656Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000016nulltrue2023-05-31T20:38:22.140Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000017nulltrue2023-05-31T20:38:22.548Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000018nulltrue2023-05-31T20:38:21.887Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000019nulltrue2023-05-31T20:38:21.559Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000020nulltrue2023-05-31T20:38:22.465Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000021nulltrue2023-05-31T20:38:20.802Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000022nulltrue2023-05-31T20:38:20.598Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000023nulltrue2023-05-31T20:38:20.347Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000024nulltrue2023-05-31T20:38:22.813Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000025nulltrue2023-05-31T20:38:19.001Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000026nulltrue2023-05-31T20:38:22.407Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000027nulltrue2023-05-31T20:38:21.615Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000028nulltrue2023-05-31T20:38:15.316Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000029nulltrue2023-05-31T20:38:20.265Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000030nulltrue2023-05-31T20:38:21.641Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000031nulltrue2023-05-31T20:38:18.440Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000032nulltrue2023-05-31T20:38:15.861Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000033nulltrue2023-05-31T20:38:22.084Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000036nulltrue2023-05-31T20:38:22.636Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000037nulltrue2023-05-31T20:38:19.358Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000039nulltrue2023-05-31T20:38:22.383Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000040nulltrue2023-05-31T20:38:22.327Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000042nulltrue2023-05-31T20:38:18.977Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000045nulltrue2023-05-31T20:38:22.250Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000046nulltrue2023-05-31T20:38:17.357Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000048nulltrue2023-05-31T20:38:22.489Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000049nulltrue2023-05-31T20:38:18.527Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000050nulltrue2023-05-31T20:38:20.183Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000053nulltrue2023-05-31T20:38:22.873Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000055nulltrue2023-05-31T20:38:17.862Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000056nulltrue2023-05-31T20:38:20.370Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000057nulltrue2023-05-31T20:38:21.818Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000058nulltrue2023-05-31T20:38:17.631Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000059nulltrue2023-05-31T20:38:18.560Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000060nulltrue2023-05-31T20:38:22.575Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000061nulltrue2023-05-31T20:38:21.154Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000062nulltrue2023-05-31T20:38:21.794Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000065nulltrue2023-05-31T20:38:22.003Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000133nulltrue2023-05-31T20:38:19.997Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000162nulltrue2023-05-31T20:38:20.538Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000271nulltrue2023-05-31T20:38:20.240Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000283nulltrue2023-05-31T20:38:20.953Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000317nulltrue2023-05-31T20:38:20.065Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000423nulltrue2023-05-31T20:38:20.434Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000467nulltrue2023-05-31T20:38:20.327Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:39:06.517 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000000 2023-05-31T20:39:06.518 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000001 2023-05-31T20:39:06.518 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000002 2023-05-31T20:39:06.518 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000003 2023-05-31T20:39:06.519 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000004 2023-05-31T20:39:06.519 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000005 2023-05-31T20:39:06.519 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000006 2023-05-31T20:39:06.519 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000007 2023-05-31T20:39:06.520 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000008 2023-05-31T20:39:06.520 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000009 2023-05-31T20:39:06.520 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000010 2023-05-31T20:39:06.521 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000011 2023-05-31T20:39:06.521 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000012 2023-05-31T20:39:06.521 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000013 2023-05-31T20:39:06.522 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000014 2023-05-31T20:39:06.522 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000015 2023-05-31T20:39:06.522 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000016 2023-05-31T20:39:06.522 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000017 2023-05-31T20:39:06.523 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000018 2023-05-31T20:39:06.523 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000019 2023-05-31T20:39:06.524 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000020 2023-05-31T20:39:06.524 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000021 2023-05-31T20:39:06.524 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000022 2023-05-31T20:39:06.525 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000023 2023-05-31T20:39:06.525 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000024 2023-05-31T20:39:06.525 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000025 2023-05-31T20:39:06.526 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000026 2023-05-31T20:39:06.526 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000027 2023-05-31T20:39:06.526 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000028 2023-05-31T20:39:06.526 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000029 2023-05-31T20:39:06.527 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000030 2023-05-31T20:39:06.527 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000031 2023-05-31T20:39:06.527 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000032 2023-05-31T20:39:06.528 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000033 2023-05-31T20:39:06.528 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000036 2023-05-31T20:39:06.528 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000037 2023-05-31T20:39:06.528 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000039 2023-05-31T20:39:06.529 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000040 2023-05-31T20:39:06.529 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000042 2023-05-31T20:39:06.529 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000045 2023-05-31T20:39:06.530 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000046 2023-05-31T20:39:06.530 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000048 2023-05-31T20:39:06.530 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000049 2023-05-31T20:39:06.531 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000050 2023-05-31T20:39:06.531 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000053 2023-05-31T20:39:06.531 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000055 2023-05-31T20:39:06.531 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000056 2023-05-31T20:39:06.532 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000057 2023-05-31T20:39:06.532 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000058 2023-05-31T20:39:06.532 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000059 2023-05-31T20:39:06.533 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000060 2023-05-31T20:39:06.533 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000061 2023-05-31T20:39:06.534 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000062 2023-05-31T20:39:06.534 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000065 2023-05-31T20:39:06.534 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000133 2023-05-31T20:39:06.535 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000162 2023-05-31T20:39:06.535 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000271 2023-05-31T20:39:06.535 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000283 2023-05-31T20:39:06.536 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000317 2023-05-31T20:39:06.536 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000423 2023-05-31T20:39:06.536 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000467 2023-05-31T20:39:06.537 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2023-05-31T20:39:06.537 DEBUG:boto:path=/pklxkd-58/ 2023-05-31T20:39:06.537 DEBUG:boto:auth_path=/pklxkd-58/ 2023-05-31T20:39:06.538 DEBUG:boto:path=/pklxkd-58/?versions&max-keys=999 2023-05-31T20:39:06.538 DEBUG:boto:auth_path=/pklxkd-58/?versions&max-keys=999 2023-05-31T20:39:06.538 DEBUG:boto:Method: GET 2023-05-31T20:39:06.539 DEBUG:boto:Path: /pklxkd-58/?versions&max-keys=999 2023-05-31T20:39:06.539 DEBUG:boto:Data: 2023-05-31T20:39:06.539 DEBUG:boto:Headers: {} 2023-05-31T20:39:06.540 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:39:06.540 DEBUG:boto:Port: 8000 2023-05-31T20:39:06.540 DEBUG:boto:Params: {} 2023-05-31T20:39:06.541 DEBUG:boto:Token: None 2023-05-31T20:39:06.541 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:39:06 GMT /pklxkd-58/?versions 2023-05-31T20:39:06.542 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:zJ+RcxA7u8JmnQTXwxhXJG9x6RQ= 2023-05-31T20:39:06.542 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:39:06 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:zJ+RcxA7u8JmnQTXwxhXJG9x6RQ=', 'Content-Length': '0'} 2023-05-31T20:39:06.547 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000004b08f424c586939c-006477b06a-1630-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:39:06 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:39:06.548 DEBUG:boto:b'pklxkd-58999falseobj00000000000000000000nulltrue2023-05-31T20:38:21.972Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000001nulltrue2023-05-31T20:38:22.169Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000002nulltrue2023-05-31T20:38:22.061Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000003nulltrue2023-05-31T20:38:21.910Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000004nulltrue2023-05-31T20:38:22.227Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000005nulltrue2023-05-31T20:38:21.709Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000006nulltrue2023-05-31T20:38:21.735Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000007nulltrue2023-05-31T20:38:22.793Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000008nulltrue2023-05-31T20:38:20.093Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000009nulltrue2023-05-31T20:38:21.588Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000010nulltrue2023-05-31T20:38:21.127Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000011nulltrue2023-05-31T20:38:22.712Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000012nulltrue2023-05-31T20:38:22.308Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000013nulltrue2023-05-31T20:38:22.736Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000014nulltrue2023-05-31T20:38:22.896Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000015nulltrue2023-05-31T20:38:22.656Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000016nulltrue2023-05-31T20:38:22.140Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000017nulltrue2023-05-31T20:38:22.548Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000018nulltrue2023-05-31T20:38:21.887Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000019nulltrue2023-05-31T20:38:21.559Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000020nulltrue2023-05-31T20:38:22.465Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000021nulltrue2023-05-31T20:38:20.802Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000022nulltrue2023-05-31T20:38:20.598Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000023nulltrue2023-05-31T20:38:20.347Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000024nulltrue2023-05-31T20:38:22.813Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000025nulltrue2023-05-31T20:38:19.001Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000026nulltrue2023-05-31T20:38:22.407Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000027nulltrue2023-05-31T20:38:21.615Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000028nulltrue2023-05-31T20:38:15.316Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000029nulltrue2023-05-31T20:38:20.265Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000030nulltrue2023-05-31T20:38:21.641Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000031nulltrue2023-05-31T20:38:18.440Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000032nulltrue2023-05-31T20:38:15.861Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000033nulltrue2023-05-31T20:38:22.084Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000036nulltrue2023-05-31T20:38:22.636Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000037nulltrue2023-05-31T20:38:19.358Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000039nulltrue2023-05-31T20:38:22.383Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000040nulltrue2023-05-31T20:38:22.327Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000042nulltrue2023-05-31T20:38:18.977Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000045nulltrue2023-05-31T20:38:22.250Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000046nulltrue2023-05-31T20:38:17.357Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000048nulltrue2023-05-31T20:38:22.489Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000049nulltrue2023-05-31T20:38:18.527Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000050nulltrue2023-05-31T20:38:20.183Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000053nulltrue2023-05-31T20:38:22.873Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000055nulltrue2023-05-31T20:38:17.862Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000056nulltrue2023-05-31T20:38:20.370Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000057nulltrue2023-05-31T20:38:21.818Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000058nulltrue2023-05-31T20:38:17.631Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000059nulltrue2023-05-31T20:38:18.560Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000060nulltrue2023-05-31T20:38:22.575Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000061nulltrue2023-05-31T20:38:21.154Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000062nulltrue2023-05-31T20:38:21.794Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000065nulltrue2023-05-31T20:38:22.003Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000133nulltrue2023-05-31T20:38:19.997Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000162nulltrue2023-05-31T20:38:20.538Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000271nulltrue2023-05-31T20:38:20.240Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000283nulltrue2023-05-31T20:38:20.953Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000317nulltrue2023-05-31T20:38:20.065Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000423nulltrue2023-05-31T20:38:20.434Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000467nulltrue2023-05-31T20:38:20.327Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:39:06.551 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000000 2023-05-31T20:39:06.551 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000001 2023-05-31T20:39:06.551 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000002 2023-05-31T20:39:06.552 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000003 2023-05-31T20:39:06.552 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000004 2023-05-31T20:39:06.552 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000005 2023-05-31T20:39:06.553 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000006 2023-05-31T20:39:06.553 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000007 2023-05-31T20:39:06.553 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000008 2023-05-31T20:39:06.553 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000009 2023-05-31T20:39:06.554 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000010 2023-05-31T20:39:06.554 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000011 2023-05-31T20:39:06.555 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000012 2023-05-31T20:39:06.555 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000013 2023-05-31T20:39:06.555 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000014 2023-05-31T20:39:06.556 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000015 2023-05-31T20:39:06.556 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000016 2023-05-31T20:39:06.556 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000017 2023-05-31T20:39:06.557 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000018 2023-05-31T20:39:06.557 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000019 2023-05-31T20:39:06.557 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000020 2023-05-31T20:39:06.557 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000021 2023-05-31T20:39:06.558 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000022 2023-05-31T20:39:06.558 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000023 2023-05-31T20:39:06.558 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000024 2023-05-31T20:39:06.559 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000025 2023-05-31T20:39:06.559 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000026 2023-05-31T20:39:06.559 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000027 2023-05-31T20:39:06.559 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000028 2023-05-31T20:39:06.560 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000029 2023-05-31T20:39:06.560 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000030 2023-05-31T20:39:06.560 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000031 2023-05-31T20:39:06.561 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000032 2023-05-31T20:39:06.561 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000033 2023-05-31T20:39:06.561 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000036 2023-05-31T20:39:06.562 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000037 2023-05-31T20:39:06.562 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000039 2023-05-31T20:39:06.562 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000040 2023-05-31T20:39:06.563 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000042 2023-05-31T20:39:06.563 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000045 2023-05-31T20:39:06.563 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000046 2023-05-31T20:39:06.563 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000048 2023-05-31T20:39:06.564 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000049 2023-05-31T20:39:06.564 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000050 2023-05-31T20:39:06.565 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000053 2023-05-31T20:39:06.565 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000055 2023-05-31T20:39:06.565 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000056 2023-05-31T20:39:06.566 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000057 2023-05-31T20:39:06.566 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000058 2023-05-31T20:39:06.566 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000059 2023-05-31T20:39:06.567 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000060 2023-05-31T20:39:06.567 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000061 2023-05-31T20:39:06.567 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000062 2023-05-31T20:39:06.567 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000065 2023-05-31T20:39:06.568 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000133 2023-05-31T20:39:06.568 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000162 2023-05-31T20:39:06.568 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000271 2023-05-31T20:39:06.569 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000283 2023-05-31T20:39:06.569 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000317 2023-05-31T20:39:06.569 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000423 2023-05-31T20:39:06.569 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000467 2023-05-31T20:39:06.570 DEBUG:boto:path=/pklxkd-58/ 2023-05-31T20:39:06.570 DEBUG:boto:auth_path=/pklxkd-58/ 2023-05-31T20:39:06.571 DEBUG:boto:path=/pklxkd-58/?versions&max-keys=999 2023-05-31T20:39:06.571 DEBUG:boto:auth_path=/pklxkd-58/?versions&max-keys=999 2023-05-31T20:39:06.572 DEBUG:boto:Method: GET 2023-05-31T20:39:06.572 DEBUG:boto:Path: /pklxkd-58/?versions&max-keys=999 2023-05-31T20:39:06.573 DEBUG:boto:Data: 2023-05-31T20:39:06.573 DEBUG:boto:Headers: {} 2023-05-31T20:39:06.573 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:39:06.574 DEBUG:boto:Port: 8001 2023-05-31T20:39:06.574 DEBUG:boto:Params: {} 2023-05-31T20:39:06.575 DEBUG:boto:Token: None 2023-05-31T20:39:06.575 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:39:06 GMT /pklxkd-58/?versions 2023-05-31T20:39:06.576 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:zJ+RcxA7u8JmnQTXwxhXJG9x6RQ= 2023-05-31T20:39:06.576 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:39:06 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:zJ+RcxA7u8JmnQTXwxhXJG9x6RQ=', 'Content-Length': '0'} 2023-05-31T20:39:06.579 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000021f6d75f21908498-006477b06a-1c17-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:39:06 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:39:06.580 DEBUG:boto:b'pklxkd-58999falseobj00000000000000000000nulltrue2023-05-31T20:38:21.972Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000001nulltrue2023-05-31T20:38:22.169Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000002nulltrue2023-05-31T20:38:22.061Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000003nulltrue2023-05-31T20:38:21.910Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000004nulltrue2023-05-31T20:38:22.227Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000005nulltrue2023-05-31T20:38:21.709Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000006nulltrue2023-05-31T20:38:21.735Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000007nulltrue2023-05-31T20:38:22.793Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000008nulltrue2023-05-31T20:38:20.093Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000009nulltrue2023-05-31T20:38:21.588Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000010nulltrue2023-05-31T20:38:21.127Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000011nulltrue2023-05-31T20:38:22.712Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000012nulltrue2023-05-31T20:38:22.308Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000013nulltrue2023-05-31T20:38:22.736Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000014nulltrue2023-05-31T20:38:22.896Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000015nulltrue2023-05-31T20:38:22.656Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000016nulltrue2023-05-31T20:38:22.140Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000017nulltrue2023-05-31T20:38:22.548Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000018nulltrue2023-05-31T20:38:21.887Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000019nulltrue2023-05-31T20:38:21.559Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000020nulltrue2023-05-31T20:38:22.465Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000021nulltrue2023-05-31T20:38:20.802Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000022nulltrue2023-05-31T20:38:20.598Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000023nulltrue2023-05-31T20:38:20.347Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000024nulltrue2023-05-31T20:38:22.813Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000025nulltrue2023-05-31T20:38:19.001Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000026nulltrue2023-05-31T20:38:22.407Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000027nulltrue2023-05-31T20:38:21.615Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000028nulltrue2023-05-31T20:38:15.316Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000029nulltrue2023-05-31T20:38:20.265Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000030nulltrue2023-05-31T20:38:21.641Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000031nulltrue2023-05-31T20:38:18.440Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000032nulltrue2023-05-31T20:38:15.861Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000033nulltrue2023-05-31T20:38:22.084Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000036nulltrue2023-05-31T20:38:22.636Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000037nulltrue2023-05-31T20:38:19.358Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000039nulltrue2023-05-31T20:38:22.383Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000040nulltrue2023-05-31T20:38:22.327Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000042nulltrue2023-05-31T20:38:18.977Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000045nulltrue2023-05-31T20:38:22.250Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000046nulltrue2023-05-31T20:38:17.357Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000048nulltrue2023-05-31T20:38:22.489Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000049nulltrue2023-05-31T20:38:18.527Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000050nulltrue2023-05-31T20:38:20.183Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000053nulltrue2023-05-31T20:38:22.873Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000055nulltrue2023-05-31T20:38:17.862Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000056nulltrue2023-05-31T20:38:20.370Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000057nulltrue2023-05-31T20:38:21.818Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000058nulltrue2023-05-31T20:38:17.631Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000059nulltrue2023-05-31T20:38:18.560Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000060nulltrue2023-05-31T20:38:22.575Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000061nulltrue2023-05-31T20:38:21.154Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000062nulltrue2023-05-31T20:38:21.794Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000065nulltrue2023-05-31T20:38:22.003Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000133nulltrue2023-05-31T20:38:19.997Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000162nulltrue2023-05-31T20:38:20.538Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000271nulltrue2023-05-31T20:38:20.240Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000283nulltrue2023-05-31T20:38:20.953Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000317nulltrue2023-05-31T20:38:20.065Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000423nulltrue2023-05-31T20:38:20.434Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000467nulltrue2023-05-31T20:38:20.327Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:39:06.585 DEBUG:boto:path=/pklxkd-58/ 2023-05-31T20:39:06.586 DEBUG:boto:auth_path=/pklxkd-58/ 2023-05-31T20:39:06.587 DEBUG:boto:path=/pklxkd-58/?versions&max-keys=999 2023-05-31T20:39:06.587 DEBUG:boto:auth_path=/pklxkd-58/?versions&max-keys=999 2023-05-31T20:39:06.588 DEBUG:boto:Method: GET 2023-05-31T20:39:06.589 DEBUG:boto:Path: /pklxkd-58/?versions&max-keys=999 2023-05-31T20:39:06.589 DEBUG:boto:Data: 2023-05-31T20:39:06.590 DEBUG:boto:Headers: {} 2023-05-31T20:39:06.590 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:39:06.591 DEBUG:boto:Port: 8000 2023-05-31T20:39:06.592 DEBUG:boto:Params: {} 2023-05-31T20:39:06.593 DEBUG:boto:Token: None 2023-05-31T20:39:06.593 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:39:06 GMT /pklxkd-58/?versions 2023-05-31T20:39:06.594 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:zJ+RcxA7u8JmnQTXwxhXJG9x6RQ= 2023-05-31T20:39:06.594 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:39:06 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:zJ+RcxA7u8JmnQTXwxhXJG9x6RQ=', 'Content-Length': '0'} 2023-05-31T20:39:06.599 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000355f68413fee617a-006477b06a-1630-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:39:06 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:39:06.600 DEBUG:boto:b'pklxkd-58999falseobj00000000000000000000nulltrue2023-05-31T20:38:21.972Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000001nulltrue2023-05-31T20:38:22.169Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000002nulltrue2023-05-31T20:38:22.061Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000003nulltrue2023-05-31T20:38:21.910Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000004nulltrue2023-05-31T20:38:22.227Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000005nulltrue2023-05-31T20:38:21.709Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000006nulltrue2023-05-31T20:38:21.735Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000007nulltrue2023-05-31T20:38:22.793Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000008nulltrue2023-05-31T20:38:20.093Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000009nulltrue2023-05-31T20:38:21.588Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000010nulltrue2023-05-31T20:38:21.127Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000011nulltrue2023-05-31T20:38:22.712Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000012nulltrue2023-05-31T20:38:22.308Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000013nulltrue2023-05-31T20:38:22.736Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000014nulltrue2023-05-31T20:38:22.896Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000015nulltrue2023-05-31T20:38:22.656Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000016nulltrue2023-05-31T20:38:22.140Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000017nulltrue2023-05-31T20:38:22.548Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000018nulltrue2023-05-31T20:38:21.887Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000019nulltrue2023-05-31T20:38:21.559Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000020nulltrue2023-05-31T20:38:22.465Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000021nulltrue2023-05-31T20:38:20.802Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000022nulltrue2023-05-31T20:38:20.598Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000023nulltrue2023-05-31T20:38:20.347Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000024nulltrue2023-05-31T20:38:22.813Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000025nulltrue2023-05-31T20:38:19.001Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000026nulltrue2023-05-31T20:38:22.407Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000027nulltrue2023-05-31T20:38:21.615Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000028nulltrue2023-05-31T20:38:15.316Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000029nulltrue2023-05-31T20:38:20.265Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000030nulltrue2023-05-31T20:38:21.641Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000031nulltrue2023-05-31T20:38:18.440Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000032nulltrue2023-05-31T20:38:15.861Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000033nulltrue2023-05-31T20:38:22.084Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000036nulltrue2023-05-31T20:38:22.636Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000037nulltrue2023-05-31T20:38:19.358Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000039nulltrue2023-05-31T20:38:22.383Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000040nulltrue2023-05-31T20:38:22.327Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000042nulltrue2023-05-31T20:38:18.977Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000045nulltrue2023-05-31T20:38:22.250Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000046nulltrue2023-05-31T20:38:17.357Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000048nulltrue2023-05-31T20:38:22.489Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000049nulltrue2023-05-31T20:38:18.527Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000050nulltrue2023-05-31T20:38:20.183Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000053nulltrue2023-05-31T20:38:22.873Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000055nulltrue2023-05-31T20:38:17.862Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000056nulltrue2023-05-31T20:38:20.370Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000057nulltrue2023-05-31T20:38:21.818Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000058nulltrue2023-05-31T20:38:17.631Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000059nulltrue2023-05-31T20:38:18.560Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000060nulltrue2023-05-31T20:38:22.575Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000061nulltrue2023-05-31T20:38:21.154Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000062nulltrue2023-05-31T20:38:21.794Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000065nulltrue2023-05-31T20:38:22.003Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000133nulltrue2023-05-31T20:38:19.997Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000162nulltrue2023-05-31T20:38:20.538Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000271nulltrue2023-05-31T20:38:20.240Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000283nulltrue2023-05-31T20:38:20.953Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000317nulltrue2023-05-31T20:38:20.065Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000423nulltrue2023-05-31T20:38:20.434Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000467nulltrue2023-05-31T20:38:20.327Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:39:06.605 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000000 2023-05-31T20:39:06.605 DEBUG:boto:path=/pklxkd-58/obj00000000000000000000 2023-05-31T20:39:06.606 DEBUG:boto:auth_path=/pklxkd-58/obj00000000000000000000 2023-05-31T20:39:06.606 DEBUG:boto:path=/pklxkd-58/obj00000000000000000000?versionId=null 2023-05-31T20:39:06.607 DEBUG:boto:auth_path=/pklxkd-58/obj00000000000000000000?versionId=null 2023-05-31T20:39:06.607 DEBUG:boto:Method: GET 2023-05-31T20:39:06.607 DEBUG:boto:Path: /pklxkd-58/obj00000000000000000000?versionId=null 2023-05-31T20:39:06.608 DEBUG:boto:Data: 2023-05-31T20:39:06.609 DEBUG:boto:Headers: {} 2023-05-31T20:39:06.610 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:39:06.610 DEBUG:boto:Port: 8001 2023-05-31T20:39:06.611 DEBUG:boto:Params: {} 2023-05-31T20:39:06.611 DEBUG:boto:Token: None 2023-05-31T20:39:06.612 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:39:06 GMT /pklxkd-58/obj00000000000000000000?versionId=null 2023-05-31T20:39:06.612 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:EFbqf+C8u+eG+xkwupf1PXAnxfM= 2023-05-31T20:39:06.613 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:39:06 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:EFbqf+C8u+eG+xkwupf1PXAnxfM=', 'Content-Length': '0'} 2023-05-31T20:39:06.616 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:38:21 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000135b0b422f145c94-006477b06a-1c17-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:39:06 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:39:06.617 DEBUG:boto:path=/pklxkd-58/obj00000000000000000000 2023-05-31T20:39:06.618 DEBUG:boto:auth_path=/pklxkd-58/obj00000000000000000000 2023-05-31T20:39:06.618 DEBUG:boto:path=/pklxkd-58/obj00000000000000000000?versionId=null 2023-05-31T20:39:06.618 DEBUG:boto:auth_path=/pklxkd-58/obj00000000000000000000?versionId=null 2023-05-31T20:39:06.619 DEBUG:boto:Method: GET 2023-05-31T20:39:06.619 DEBUG:boto:Path: /pklxkd-58/obj00000000000000000000?versionId=null 2023-05-31T20:39:06.620 DEBUG:boto:Data: 2023-05-31T20:39:06.620 DEBUG:boto:Headers: {} 2023-05-31T20:39:06.620 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:39:06.621 DEBUG:boto:Port: 8000 2023-05-31T20:39:06.621 DEBUG:boto:Params: {} 2023-05-31T20:39:06.622 DEBUG:boto:Token: None 2023-05-31T20:39:06.622 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:39:06 GMT /pklxkd-58/obj00000000000000000000?versionId=null 2023-05-31T20:39:06.623 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:EFbqf+C8u+eG+xkwupf1PXAnxfM= 2023-05-31T20:39:06.623 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:39:06 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:EFbqf+C8u+eG+xkwupf1PXAnxfM=', 'Content-Length': '0'} 2023-05-31T20:39:06.626 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:38:21 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007c18976af4994d76-006477b06a-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:39:06 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:39:06.627 DEBUG:boto:path=/pklxkd-58/obj00000000000000000000 2023-05-31T20:39:06.627 DEBUG:boto:auth_path=/pklxkd-58/obj00000000000000000000 2023-05-31T20:39:06.628 DEBUG:boto:path=/pklxkd-58/obj00000000000000000000?versionId=null 2023-05-31T20:39:06.628 DEBUG:boto:auth_path=/pklxkd-58/obj00000000000000000000?versionId=null 2023-05-31T20:39:06.628 DEBUG:boto:Method: HEAD 2023-05-31T20:39:06.629 DEBUG:boto:Path: /pklxkd-58/obj00000000000000000000?versionId=null 2023-05-31T20:39:06.629 DEBUG:boto:Data: 2023-05-31T20:39:06.630 DEBUG:boto:Headers: {} 2023-05-31T20:39:06.630 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:39:06.630 DEBUG:boto:Port: 8001 2023-05-31T20:39:06.631 DEBUG:boto:Params: {} 2023-05-31T20:39:06.631 DEBUG:boto:Token: None 2023-05-31T20:39:06.632 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:39:06 GMT /pklxkd-58/obj00000000000000000000?versionId=null 2023-05-31T20:39:06.632 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:ytgEPwGPHxoWLFd1qGZJ9seXVNk= 2023-05-31T20:39:06.633 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:39:06 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:ytgEPwGPHxoWLFd1qGZJ9seXVNk=', 'Content-Length': '0'} 2023-05-31T20:39:06.635 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:38:21 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d1240159e11cd907-006477b06a-1c17-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:39:06 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:39:06.635 DEBUG:boto:path=/pklxkd-58/obj00000000000000000000 2023-05-31T20:39:06.636 DEBUG:boto:auth_path=/pklxkd-58/obj00000000000000000000 2023-05-31T20:39:06.637 DEBUG:boto:path=/pklxkd-58/obj00000000000000000000?versionId=null 2023-05-31T20:39:06.637 DEBUG:boto:auth_path=/pklxkd-58/obj00000000000000000000?versionId=null 2023-05-31T20:39:06.637 DEBUG:boto:Method: HEAD 2023-05-31T20:39:06.638 DEBUG:boto:Path: /pklxkd-58/obj00000000000000000000?versionId=null 2023-05-31T20:39:06.640 DEBUG:boto:Data: 2023-05-31T20:39:06.641 DEBUG:boto:Headers: {} 2023-05-31T20:39:06.642 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:39:06.643 DEBUG:boto:Port: 8000 2023-05-31T20:39:06.644 DEBUG:boto:Params: {} 2023-05-31T20:39:06.645 DEBUG:boto:Token: None 2023-05-31T20:39:06.645 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:39:06 GMT /pklxkd-58/obj00000000000000000000?versionId=null 2023-05-31T20:39:06.646 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:ytgEPwGPHxoWLFd1qGZJ9seXVNk= 2023-05-31T20:39:06.647 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:39:06 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:ytgEPwGPHxoWLFd1qGZJ9seXVNk=', 'Content-Length': '0'} 2023-05-31T20:39:06.650 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:38:21 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000454e96e33ff09e3-006477b06a-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:39:06 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:39:06.651 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000000 2023-05-31T20:39:06.651 DEBUG:boto:path=/pklxkd-58/obj00000000000000000000 2023-05-31T20:39:06.652 DEBUG:boto:auth_path=/pklxkd-58/obj00000000000000000000 2023-05-31T20:39:06.653 DEBUG:boto:path=/pklxkd-58/obj00000000000000000000?versionId=null 2023-05-31T20:39:06.654 DEBUG:boto:auth_path=/pklxkd-58/obj00000000000000000000?versionId=null 2023-05-31T20:39:06.655 DEBUG:boto:Method: GET 2023-05-31T20:39:06.656 DEBUG:boto:Path: /pklxkd-58/obj00000000000000000000?versionId=null 2023-05-31T20:39:06.657 DEBUG:boto:Data: 2023-05-31T20:39:06.658 DEBUG:boto:Headers: {} 2023-05-31T20:39:06.659 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:39:06.659 DEBUG:boto:Port: 8001 2023-05-31T20:39:06.660 DEBUG:boto:Params: {} 2023-05-31T20:39:06.661 DEBUG:boto:Token: None 2023-05-31T20:39:06.662 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:39:06 GMT /pklxkd-58/obj00000000000000000000?versionId=null 2023-05-31T20:39:06.662 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:EFbqf+C8u+eG+xkwupf1PXAnxfM= 2023-05-31T20:39:06.663 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:39:06 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:EFbqf+C8u+eG+xkwupf1PXAnxfM=', 'Content-Length': '0'} 2023-05-31T20:39:06.665 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:38:21 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008109a736bd5d8927-006477b06a-1c17-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:39:06 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:39:06.666 DEBUG:boto:path=/pklxkd-58/obj00000000000000000000 2023-05-31T20:39:06.666 DEBUG:boto:auth_path=/pklxkd-58/obj00000000000000000000 2023-05-31T20:39:06.667 DEBUG:boto:path=/pklxkd-58/obj00000000000000000000?versionId=null 2023-05-31T20:39:06.667 DEBUG:boto:auth_path=/pklxkd-58/obj00000000000000000000?versionId=null 2023-05-31T20:39:06.668 DEBUG:boto:Method: GET 2023-05-31T20:39:06.668 DEBUG:boto:Path: /pklxkd-58/obj00000000000000000000?versionId=null 2023-05-31T20:39:06.668 DEBUG:boto:Data: 2023-05-31T20:39:06.669 DEBUG:boto:Headers: {} 2023-05-31T20:39:06.669 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:39:06.670 DEBUG:boto:Port: 8000 2023-05-31T20:39:06.671 DEBUG:boto:Params: {} 2023-05-31T20:39:06.671 DEBUG:boto:Token: None 2023-05-31T20:39:06.672 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:39:06 GMT /pklxkd-58/obj00000000000000000000?versionId=null 2023-05-31T20:39:06.672 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:EFbqf+C8u+eG+xkwupf1PXAnxfM= 2023-05-31T20:39:06.673 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:39:06 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:EFbqf+C8u+eG+xkwupf1PXAnxfM=', 'Content-Length': '0'} 2023-05-31T20:39:06.676 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:38:21 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000001853f8420052b5d-006477b06a-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:39:06 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:39:06.677 DEBUG:boto:path=/pklxkd-58/ 2023-05-31T20:39:06.677 DEBUG:boto:auth_path=/pklxkd-58/ 2023-05-31T20:39:06.678 DEBUG:boto:path=/pklxkd-58/?prefix=obj00000000000000000000 2023-05-31T20:39:06.679 DEBUG:boto:auth_path=/pklxkd-58/?prefix=obj00000000000000000000 2023-05-31T20:39:06.679 DEBUG:boto:Method: GET 2023-05-31T20:39:06.680 DEBUG:boto:Path: /pklxkd-58/?prefix=obj00000000000000000000 2023-05-31T20:39:06.681 DEBUG:boto:Data: 2023-05-31T20:39:06.681 DEBUG:boto:Headers: {} 2023-05-31T20:39:06.682 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:39:06.683 DEBUG:boto:Port: 8001 2023-05-31T20:39:06.684 DEBUG:boto:Params: {} 2023-05-31T20:39:06.684 DEBUG:boto:Token: None 2023-05-31T20:39:06.685 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:39:06 GMT /pklxkd-58/ 2023-05-31T20:39:06.686 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:Kt8rOlTVlv8Ps0Xuh2zVVbpYKiE= 2023-05-31T20:39:06.687 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:39:06 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:Kt8rOlTVlv8Ps0Xuh2zVVbpYKiE=', 'Content-Length': '0'} 2023-05-31T20:39:06.690 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000007c85ef2688199c99-006477b06a-1c17-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:39:06 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:39:06.691 DEBUG:boto:b'pklxkd-58obj000000000000000000001000falseobj000000000000000000002023-05-31T20:38:21.972Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:39:06.692 DEBUG:boto:path=/pklxkd-58/ 2023-05-31T20:39:06.692 DEBUG:boto:auth_path=/pklxkd-58/ 2023-05-31T20:39:06.693 DEBUG:boto:path=/pklxkd-58/?prefix=obj00000000000000000000 2023-05-31T20:39:06.694 DEBUG:boto:auth_path=/pklxkd-58/?prefix=obj00000000000000000000 2023-05-31T20:39:06.695 DEBUG:boto:Method: GET 2023-05-31T20:39:06.695 DEBUG:boto:Path: /pklxkd-58/?prefix=obj00000000000000000000 2023-05-31T20:39:06.696 DEBUG:boto:Data: 2023-05-31T20:39:06.697 DEBUG:boto:Headers: {} 2023-05-31T20:39:06.697 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:39:06.698 DEBUG:boto:Port: 8000 2023-05-31T20:39:06.699 DEBUG:boto:Params: {} 2023-05-31T20:39:06.700 DEBUG:boto:Token: None 2023-05-31T20:39:06.700 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:39:06 GMT /pklxkd-58/ 2023-05-31T20:39:06.701 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:Kt8rOlTVlv8Ps0Xuh2zVVbpYKiE= 2023-05-31T20:39:06.702 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:39:06 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:Kt8rOlTVlv8Ps0Xuh2zVVbpYKiE=', 'Content-Length': '0'} 2023-05-31T20:39:06.712 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000039138b808009d27c-006477b06a-1630-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:39:06 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:39:06.714 DEBUG:boto:b'pklxkd-58obj000000000000000000001000falseobj000000000000000000002023-05-31T20:38:21.972Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:39:06.715 DEBUG:boto:path=/pklxkd-58/obj00000000000000000000 2023-05-31T20:39:06.716 DEBUG:boto:auth_path=/pklxkd-58/obj00000000000000000000 2023-05-31T20:39:06.717 DEBUG:boto:Method: HEAD 2023-05-31T20:39:06.718 DEBUG:boto:Path: /pklxkd-58/obj00000000000000000000 2023-05-31T20:39:06.719 DEBUG:boto:Data: 2023-05-31T20:39:06.719 DEBUG:boto:Headers: {} 2023-05-31T20:39:06.720 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:39:06.721 DEBUG:boto:Port: 8001 2023-05-31T20:39:06.721 DEBUG:boto:Params: {} 2023-05-31T20:39:06.722 DEBUG:boto:Token: None 2023-05-31T20:39:06.723 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:39:06 GMT /pklxkd-58/obj00000000000000000000 2023-05-31T20:39:06.724 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:PggoyNrZJkA+S1VJqogzZZSW1IM= 2023-05-31T20:39:06.725 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:39:06 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:PggoyNrZJkA+S1VJqogzZZSW1IM=', 'Content-Length': '0'} 2023-05-31T20:39:06.727 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:38:21 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000019f7373882b83c1d-006477b06a-1c17-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:39:06 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:39:06.728 DEBUG:boto:path=/pklxkd-58/obj00000000000000000000 2023-05-31T20:39:06.728 DEBUG:boto:auth_path=/pklxkd-58/obj00000000000000000000 2023-05-31T20:39:06.728 DEBUG:boto:Method: HEAD 2023-05-31T20:39:06.729 DEBUG:boto:Path: /pklxkd-58/obj00000000000000000000 2023-05-31T20:39:06.729 DEBUG:boto:Data: 2023-05-31T20:39:06.730 DEBUG:boto:Headers: {} 2023-05-31T20:39:06.730 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:39:06.731 DEBUG:boto:Port: 8000 2023-05-31T20:39:06.732 DEBUG:boto:Params: {} 2023-05-31T20:39:06.732 DEBUG:boto:Token: None 2023-05-31T20:39:06.733 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:39:06 GMT /pklxkd-58/obj00000000000000000000 2023-05-31T20:39:06.734 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:PggoyNrZJkA+S1VJqogzZZSW1IM= 2023-05-31T20:39:06.735 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:39:06 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:PggoyNrZJkA+S1VJqogzZZSW1IM=', 'Content-Length': '0'} 2023-05-31T20:39:06.737 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:38:21 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e02d4548778d96e4-006477b06a-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:39:06 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:39:06.738 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000000 2023-05-31T20:39:06.739 DEBUG:boto:path=/pklxkd-58/obj00000000000000000000 2023-05-31T20:39:06.740 DEBUG:boto:auth_path=/pklxkd-58/obj00000000000000000000 2023-05-31T20:39:06.741 DEBUG:boto:Method: GET 2023-05-31T20:39:06.741 DEBUG:boto:Path: /pklxkd-58/obj00000000000000000000 2023-05-31T20:39:06.742 DEBUG:boto:Data: 2023-05-31T20:39:06.743 DEBUG:boto:Headers: {} 2023-05-31T20:39:06.743 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:39:06.744 DEBUG:boto:Port: 8001 2023-05-31T20:39:06.745 DEBUG:boto:Params: {} 2023-05-31T20:39:06.746 DEBUG:boto:Token: None 2023-05-31T20:39:06.746 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:39:06 GMT /pklxkd-58/obj00000000000000000000 2023-05-31T20:39:06.747 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:18NqZRvdyEhhMljhYtprCYkrEoI= 2023-05-31T20:39:06.748 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:39:06 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:18NqZRvdyEhhMljhYtprCYkrEoI=', 'Content-Length': '0'} 2023-05-31T20:39:06.751 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:38:21 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c302a25cb7db9cc1-006477b06a-1c17-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:39:06 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:39:06.752 DEBUG:boto:path=/pklxkd-58/obj00000000000000000000 2023-05-31T20:39:06.753 DEBUG:boto:auth_path=/pklxkd-58/obj00000000000000000000 2023-05-31T20:39:06.753 DEBUG:boto:Method: GET 2023-05-31T20:39:06.754 DEBUG:boto:Path: /pklxkd-58/obj00000000000000000000 2023-05-31T20:39:06.755 DEBUG:boto:Data: 2023-05-31T20:39:06.755 DEBUG:boto:Headers: {} 2023-05-31T20:39:06.756 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:39:06.757 DEBUG:boto:Port: 8000 2023-05-31T20:39:06.757 DEBUG:boto:Params: {} 2023-05-31T20:39:06.758 DEBUG:boto:Token: None 2023-05-31T20:39:06.758 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:39:06 GMT /pklxkd-58/obj00000000000000000000 2023-05-31T20:39:06.760 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:18NqZRvdyEhhMljhYtprCYkrEoI= 2023-05-31T20:39:06.761 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:39:06 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:18NqZRvdyEhhMljhYtprCYkrEoI=', 'Content-Length': '0'} 2023-05-31T20:39:06.763 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:38:21 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000015b21138644cd2d-006477b06a-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:39:06 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:39:06.808 DEBUG:boto:path=/pklxkd-58/ 2023-05-31T20:39:06.809 DEBUG:boto:auth_path=/pklxkd-58/ 2023-05-31T20:39:06.809 DEBUG:boto:path=/pklxkd-58/?prefix=obj00000000000000000000 2023-05-31T20:39:06.810 DEBUG:boto:auth_path=/pklxkd-58/?prefix=obj00000000000000000000 2023-05-31T20:39:06.810 DEBUG:boto:Method: GET 2023-05-31T20:39:06.811 DEBUG:boto:Path: /pklxkd-58/?prefix=obj00000000000000000000 2023-05-31T20:39:06.812 DEBUG:boto:Data: 2023-05-31T20:39:06.813 DEBUG:boto:Headers: {} 2023-05-31T20:39:06.813 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:39:06.814 DEBUG:boto:Port: 8001 2023-05-31T20:39:06.815 DEBUG:boto:Params: {} 2023-05-31T20:39:06.816 DEBUG:boto:Token: None 2023-05-31T20:39:06.817 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:39:06 GMT /pklxkd-58/ 2023-05-31T20:39:06.818 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:Kt8rOlTVlv8Ps0Xuh2zVVbpYKiE= 2023-05-31T20:39:06.819 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:39:06 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:Kt8rOlTVlv8Ps0Xuh2zVVbpYKiE=', 'Content-Length': '0'} 2023-05-31T20:39:06.822 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000dceef37d23af0de3-006477b06a-1c17-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:39:06 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:39:06.823 DEBUG:boto:b'pklxkd-58obj000000000000000000001000falseobj000000000000000000002023-05-31T20:38:21.972Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:39:06.823 DEBUG:boto:path=/pklxkd-58/ 2023-05-31T20:39:06.824 DEBUG:boto:auth_path=/pklxkd-58/ 2023-05-31T20:39:06.824 DEBUG:boto:path=/pklxkd-58/?prefix=obj00000000000000000000 2023-05-31T20:39:06.825 DEBUG:boto:auth_path=/pklxkd-58/?prefix=obj00000000000000000000 2023-05-31T20:39:06.825 DEBUG:boto:Method: GET 2023-05-31T20:39:06.826 DEBUG:boto:Path: /pklxkd-58/?prefix=obj00000000000000000000 2023-05-31T20:39:06.826 DEBUG:boto:Data: 2023-05-31T20:39:06.827 DEBUG:boto:Headers: {} 2023-05-31T20:39:06.828 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:39:06.829 DEBUG:boto:Port: 8000 2023-05-31T20:39:06.829 DEBUG:boto:Params: {} 2023-05-31T20:39:06.830 DEBUG:boto:Token: None 2023-05-31T20:39:06.831 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:39:06 GMT /pklxkd-58/ 2023-05-31T20:39:06.832 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:Kt8rOlTVlv8Ps0Xuh2zVVbpYKiE= 2023-05-31T20:39:06.833 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:39:06 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:Kt8rOlTVlv8Ps0Xuh2zVVbpYKiE=', 'Content-Length': '0'} 2023-05-31T20:39:06.837 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000104fb1257eafb7b8-006477b06a-1630-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:39:06 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:39:06.838 DEBUG:boto:b'pklxkd-58obj000000000000000000001000falseobj000000000000000000002023-05-31T20:38:21.972Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:39:06.839 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000001 2023-05-31T20:39:06.839 DEBUG:boto:path=/pklxkd-58/obj00000000000000000001 2023-05-31T20:39:06.840 DEBUG:boto:auth_path=/pklxkd-58/obj00000000000000000001 2023-05-31T20:39:06.840 DEBUG:boto:path=/pklxkd-58/obj00000000000000000001?versionId=null 2023-05-31T20:39:06.841 DEBUG:boto:auth_path=/pklxkd-58/obj00000000000000000001?versionId=null 2023-05-31T20:39:06.841 DEBUG:boto:Method: GET 2023-05-31T20:39:06.842 DEBUG:boto:Path: /pklxkd-58/obj00000000000000000001?versionId=null 2023-05-31T20:39:06.842 DEBUG:boto:Data: 2023-05-31T20:39:06.843 DEBUG:boto:Headers: {} 2023-05-31T20:39:06.843 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:39:06.844 DEBUG:boto:Port: 8001 2023-05-31T20:39:06.845 DEBUG:boto:Params: {} 2023-05-31T20:39:06.845 DEBUG:boto:Token: None 2023-05-31T20:39:06.846 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:39:06 GMT /pklxkd-58/obj00000000000000000001?versionId=null 2023-05-31T20:39:06.847 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:h33n5qLzQ2FO+QsF6BkwgO+Qy+Y= 2023-05-31T20:39:06.848 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:39:06 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:h33n5qLzQ2FO+QsF6BkwgO+Qy+Y=', 'Content-Length': '0'} 2023-05-31T20:39:06.851 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:38:22 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009cd5f8c8d42b3184-006477b06a-1c17-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:39:06 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:39:06.852 DEBUG:boto:path=/pklxkd-58/obj00000000000000000001 2023-05-31T20:39:06.853 DEBUG:boto:auth_path=/pklxkd-58/obj00000000000000000001 2023-05-31T20:39:06.853 DEBUG:boto:path=/pklxkd-58/obj00000000000000000001?versionId=null 2023-05-31T20:39:06.854 DEBUG:boto:auth_path=/pklxkd-58/obj00000000000000000001?versionId=null 2023-05-31T20:39:06.855 DEBUG:boto:Method: GET 2023-05-31T20:39:06.856 DEBUG:boto:Path: /pklxkd-58/obj00000000000000000001?versionId=null 2023-05-31T20:39:06.856 DEBUG:boto:Data: 2023-05-31T20:39:06.857 DEBUG:boto:Headers: {} 2023-05-31T20:39:06.858 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:39:06.859 DEBUG:boto:Port: 8000 2023-05-31T20:39:06.859 DEBUG:boto:Params: {} 2023-05-31T20:39:06.860 DEBUG:boto:Token: None 2023-05-31T20:39:06.860 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:39:06 GMT /pklxkd-58/obj00000000000000000001?versionId=null 2023-05-31T20:39:06.861 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:h33n5qLzQ2FO+QsF6BkwgO+Qy+Y= 2023-05-31T20:39:06.861 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:39:06 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:h33n5qLzQ2FO+QsF6BkwgO+Qy+Y=', 'Content-Length': '0'} 2023-05-31T20:39:06.863 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:38:22 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001cabbbff2ad8756a-006477b06a-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:39:06 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:39:06.864 DEBUG:boto:path=/pklxkd-58/obj00000000000000000001 2023-05-31T20:39:06.865 DEBUG:boto:auth_path=/pklxkd-58/obj00000000000000000001 2023-05-31T20:39:06.865 DEBUG:boto:path=/pklxkd-58/obj00000000000000000001?versionId=null 2023-05-31T20:39:06.866 DEBUG:boto:auth_path=/pklxkd-58/obj00000000000000000001?versionId=null 2023-05-31T20:39:06.866 DEBUG:boto:Method: HEAD 2023-05-31T20:39:06.866 DEBUG:boto:Path: /pklxkd-58/obj00000000000000000001?versionId=null 2023-05-31T20:39:06.867 DEBUG:boto:Data: 2023-05-31T20:39:06.867 DEBUG:boto:Headers: {} 2023-05-31T20:39:06.867 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:39:06.868 DEBUG:boto:Port: 8001 2023-05-31T20:39:06.868 DEBUG:boto:Params: {} 2023-05-31T20:39:06.868 DEBUG:boto:Token: None 2023-05-31T20:39:06.869 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:39:06 GMT /pklxkd-58/obj00000000000000000001?versionId=null 2023-05-31T20:39:06.869 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:PEwMW6ejjvBZLazHzlW6MXtChKs= 2023-05-31T20:39:06.870 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:39:06 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:PEwMW6ejjvBZLazHzlW6MXtChKs=', 'Content-Length': '0'} 2023-05-31T20:39:06.872 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:38:22 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000fdf68784109fd070-006477b06a-1c17-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:39:06 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:39:06.873 DEBUG:boto:path=/pklxkd-58/obj00000000000000000001 2023-05-31T20:39:06.873 DEBUG:boto:auth_path=/pklxkd-58/obj00000000000000000001 2023-05-31T20:39:06.873 DEBUG:boto:path=/pklxkd-58/obj00000000000000000001?versionId=null 2023-05-31T20:39:06.874 DEBUG:boto:auth_path=/pklxkd-58/obj00000000000000000001?versionId=null 2023-05-31T20:39:06.874 DEBUG:boto:Method: HEAD 2023-05-31T20:39:06.875 DEBUG:boto:Path: /pklxkd-58/obj00000000000000000001?versionId=null 2023-05-31T20:39:06.875 DEBUG:boto:Data: 2023-05-31T20:39:06.876 DEBUG:boto:Headers: {} 2023-05-31T20:39:06.876 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:39:06.877 DEBUG:boto:Port: 8000 2023-05-31T20:39:06.877 DEBUG:boto:Params: {} 2023-05-31T20:39:06.878 DEBUG:boto:Token: None 2023-05-31T20:39:06.878 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:39:06 GMT /pklxkd-58/obj00000000000000000001?versionId=null 2023-05-31T20:39:06.878 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:PEwMW6ejjvBZLazHzlW6MXtChKs= 2023-05-31T20:39:06.879 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:39:06 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:PEwMW6ejjvBZLazHzlW6MXtChKs=', 'Content-Length': '0'} 2023-05-31T20:39:06.881 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:38:22 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006284011806b9463b-006477b06a-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:39:06 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:39:06.883 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000001 2023-05-31T20:39:06.883 DEBUG:boto:path=/pklxkd-58/obj00000000000000000001 2023-05-31T20:39:06.884 DEBUG:boto:auth_path=/pklxkd-58/obj00000000000000000001 2023-05-31T20:39:06.884 DEBUG:boto:path=/pklxkd-58/obj00000000000000000001?versionId=null 2023-05-31T20:39:06.885 DEBUG:boto:auth_path=/pklxkd-58/obj00000000000000000001?versionId=null 2023-05-31T20:39:06.885 DEBUG:boto:Method: GET 2023-05-31T20:39:06.885 DEBUG:boto:Path: /pklxkd-58/obj00000000000000000001?versionId=null 2023-05-31T20:39:06.886 DEBUG:boto:Data: 2023-05-31T20:39:06.886 DEBUG:boto:Headers: {} 2023-05-31T20:39:06.886 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:39:06.887 DEBUG:boto:Port: 8001 2023-05-31T20:39:06.887 DEBUG:boto:Params: {} 2023-05-31T20:39:06.888 DEBUG:boto:Token: None 2023-05-31T20:39:06.888 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:39:06 GMT /pklxkd-58/obj00000000000000000001?versionId=null 2023-05-31T20:39:06.889 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:h33n5qLzQ2FO+QsF6BkwgO+Qy+Y= 2023-05-31T20:39:06.889 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:39:06 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:h33n5qLzQ2FO+QsF6BkwgO+Qy+Y=', 'Content-Length': '0'} 2023-05-31T20:39:06.891 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:38:22 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007d33f0624507aca4-006477b06a-1c17-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:39:06 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:39:06.892 DEBUG:boto:path=/pklxkd-58/obj00000000000000000001 2023-05-31T20:39:06.892 DEBUG:boto:auth_path=/pklxkd-58/obj00000000000000000001 2023-05-31T20:39:06.893 DEBUG:boto:path=/pklxkd-58/obj00000000000000000001?versionId=null 2023-05-31T20:39:06.894 DEBUG:boto:auth_path=/pklxkd-58/obj00000000000000000001?versionId=null 2023-05-31T20:39:06.894 DEBUG:boto:Method: GET 2023-05-31T20:39:06.895 DEBUG:boto:Path: /pklxkd-58/obj00000000000000000001?versionId=null 2023-05-31T20:39:06.896 DEBUG:boto:Data: 2023-05-31T20:39:06.896 DEBUG:boto:Headers: {} 2023-05-31T20:39:06.897 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:39:06.897 DEBUG:boto:Port: 8000 2023-05-31T20:39:06.897 DEBUG:boto:Params: {} 2023-05-31T20:39:06.898 DEBUG:boto:Token: None 2023-05-31T20:39:06.898 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:39:06 GMT /pklxkd-58/obj00000000000000000001?versionId=null 2023-05-31T20:39:06.899 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:h33n5qLzQ2FO+QsF6BkwgO+Qy+Y= 2023-05-31T20:39:06.899 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:39:06 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:h33n5qLzQ2FO+QsF6BkwgO+Qy+Y=', 'Content-Length': '0'} 2023-05-31T20:39:06.901 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:38:22 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000950e26a7d3e188ff-006477b06a-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:39:06 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:39:06.948 DEBUG:boto:path=/pklxkd-58/ 2023-05-31T20:39:06.949 DEBUG:boto:auth_path=/pklxkd-58/ 2023-05-31T20:39:06.950 DEBUG:boto:path=/pklxkd-58/?prefix=obj00000000000000000001 2023-05-31T20:39:06.951 DEBUG:boto:auth_path=/pklxkd-58/?prefix=obj00000000000000000001 2023-05-31T20:39:06.951 DEBUG:boto:Method: GET 2023-05-31T20:39:06.952 DEBUG:boto:Path: /pklxkd-58/?prefix=obj00000000000000000001 2023-05-31T20:39:06.954 DEBUG:boto:Data: 2023-05-31T20:39:06.954 DEBUG:boto:Headers: {} 2023-05-31T20:39:06.955 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:39:06.955 DEBUG:boto:Port: 8001 2023-05-31T20:39:06.956 DEBUG:boto:Params: {} 2023-05-31T20:39:06.956 DEBUG:boto:Token: None 2023-05-31T20:39:06.957 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:39:06 GMT /pklxkd-58/ 2023-05-31T20:39:06.957 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:Kt8rOlTVlv8Ps0Xuh2zVVbpYKiE= 2023-05-31T20:39:06.958 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:39:06 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:Kt8rOlTVlv8Ps0Xuh2zVVbpYKiE=', 'Content-Length': '0'} 2023-05-31T20:39:06.961 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000033a3728a90dcb3cc-006477b06a-1c17-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:39:06 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:39:06.962 DEBUG:boto:b'pklxkd-58obj000000000000000000011000falseobj000000000000000000012023-05-31T20:38:22.169Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:39:06.963 DEBUG:boto:path=/pklxkd-58/ 2023-05-31T20:39:06.964 DEBUG:boto:auth_path=/pklxkd-58/ 2023-05-31T20:39:06.965 DEBUG:boto:path=/pklxkd-58/?prefix=obj00000000000000000001 2023-05-31T20:39:06.966 DEBUG:boto:auth_path=/pklxkd-58/?prefix=obj00000000000000000001 2023-05-31T20:39:06.967 DEBUG:boto:Method: GET 2023-05-31T20:39:06.967 DEBUG:boto:Path: /pklxkd-58/?prefix=obj00000000000000000001 2023-05-31T20:39:06.968 DEBUG:boto:Data: 2023-05-31T20:39:06.969 DEBUG:boto:Headers: {} 2023-05-31T20:39:06.970 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:39:06.972 DEBUG:boto:Port: 8000 2023-05-31T20:39:06.972 DEBUG:boto:Params: {} 2023-05-31T20:39:06.973 DEBUG:boto:Token: None 2023-05-31T20:39:06.974 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:39:06 GMT /pklxkd-58/ 2023-05-31T20:39:06.975 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:Kt8rOlTVlv8Ps0Xuh2zVVbpYKiE= 2023-05-31T20:39:06.976 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:39:06 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:Kt8rOlTVlv8Ps0Xuh2zVVbpYKiE=', 'Content-Length': '0'} 2023-05-31T20:39:06.980 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d86b3c0feebefb1a-006477b06a-1630-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:39:06 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:39:06.982 DEBUG:boto:b'pklxkd-58obj000000000000000000011000falseobj000000000000000000012023-05-31T20:38:22.169Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:39:06.983 DEBUG:boto:path=/pklxkd-58/obj00000000000000000001 2023-05-31T20:39:06.984 DEBUG:boto:auth_path=/pklxkd-58/obj00000000000000000001 2023-05-31T20:39:06.985 DEBUG:boto:Method: HEAD 2023-05-31T20:39:06.986 DEBUG:boto:Path: /pklxkd-58/obj00000000000000000001 2023-05-31T20:39:06.986 DEBUG:boto:Data: 2023-05-31T20:39:06.987 DEBUG:boto:Headers: {} 2023-05-31T20:39:06.988 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:39:06.988 DEBUG:boto:Port: 8001 2023-05-31T20:39:06.989 DEBUG:boto:Params: {} 2023-05-31T20:39:06.989 DEBUG:boto:Token: None 2023-05-31T20:39:06.989 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:39:06 GMT /pklxkd-58/obj00000000000000000001 2023-05-31T20:39:06.994 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:PCas9RyAegKZ+1SCqfybegu6fSI= 2023-05-31T20:39:06.995 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:39:06 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:PCas9RyAegKZ+1SCqfybegu6fSI=', 'Content-Length': '0'} 2023-05-31T20:39:06.999 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:38:22 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000ccfebff19de0380-006477b06a-1c17-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:39:06 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:39:07.000 DEBUG:boto:path=/pklxkd-58/obj00000000000000000001 2023-05-31T20:39:07.001 DEBUG:boto:auth_path=/pklxkd-58/obj00000000000000000001 2023-05-31T20:39:07.002 DEBUG:boto:Method: HEAD 2023-05-31T20:39:07.003 DEBUG:boto:Path: /pklxkd-58/obj00000000000000000001 2023-05-31T20:39:07.003 DEBUG:boto:Data: 2023-05-31T20:39:07.006 DEBUG:boto:Headers: {} 2023-05-31T20:39:07.007 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:39:07.007 DEBUG:boto:Port: 8000 2023-05-31T20:39:07.008 DEBUG:boto:Params: {} 2023-05-31T20:39:07.008 DEBUG:boto:Token: None 2023-05-31T20:39:07.009 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:39:07 GMT /pklxkd-58/obj00000000000000000001 2023-05-31T20:39:07.009 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:kMlMOPoFGfbSnpjiWTxJCWenQ3I= 2023-05-31T20:39:07.009 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:39:07 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:kMlMOPoFGfbSnpjiWTxJCWenQ3I=', 'Content-Length': '0'} 2023-05-31T20:39:07.011 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:38:22 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c7e51c18859e061c-006477b06b-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:39:07 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:39:07.012 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000001 2023-05-31T20:39:07.012 DEBUG:boto:path=/pklxkd-58/obj00000000000000000001 2023-05-31T20:39:07.013 DEBUG:boto:auth_path=/pklxkd-58/obj00000000000000000001 2023-05-31T20:39:07.013 DEBUG:boto:Method: GET 2023-05-31T20:39:07.014 DEBUG:boto:Path: /pklxkd-58/obj00000000000000000001 2023-05-31T20:39:07.014 DEBUG:boto:Data: 2023-05-31T20:39:07.015 DEBUG:boto:Headers: {} 2023-05-31T20:39:07.015 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:39:07.016 DEBUG:boto:Port: 8001 2023-05-31T20:39:07.016 DEBUG:boto:Params: {} 2023-05-31T20:39:07.017 DEBUG:boto:Token: None 2023-05-31T20:39:07.017 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:39:07 GMT /pklxkd-58/obj00000000000000000001 2023-05-31T20:39:07.018 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:/oeZpZJLrJY0wI/qz8tVORAFtyU= 2023-05-31T20:39:07.018 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:39:07 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:/oeZpZJLrJY0wI/qz8tVORAFtyU=', 'Content-Length': '0'} 2023-05-31T20:39:07.020 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:38:22 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ebd4cdfa10ee3c4d-006477b06b-1c17-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:39:07 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:39:07.068 DEBUG:boto:path=/pklxkd-58/obj00000000000000000001 2023-05-31T20:39:07.068 DEBUG:boto:auth_path=/pklxkd-58/obj00000000000000000001 2023-05-31T20:39:07.069 DEBUG:boto:Method: GET 2023-05-31T20:39:07.069 DEBUG:boto:Path: /pklxkd-58/obj00000000000000000001 2023-05-31T20:39:07.069 DEBUG:boto:Data: 2023-05-31T20:39:07.070 DEBUG:boto:Headers: {} 2023-05-31T20:39:07.070 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:39:07.071 DEBUG:boto:Port: 8000 2023-05-31T20:39:07.072 DEBUG:boto:Params: {} 2023-05-31T20:39:07.072 DEBUG:boto:Token: None 2023-05-31T20:39:07.073 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:39:07 GMT /pklxkd-58/obj00000000000000000001 2023-05-31T20:39:07.073 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:/oeZpZJLrJY0wI/qz8tVORAFtyU= 2023-05-31T20:39:07.073 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:39:07 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:/oeZpZJLrJY0wI/qz8tVORAFtyU=', 'Content-Length': '0'} 2023-05-31T20:39:07.082 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:38:22 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009e979d29ed425a5b-006477b06b-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:39:07 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:39:07.083 DEBUG:boto:path=/pklxkd-58/ 2023-05-31T20:39:07.084 DEBUG:boto:auth_path=/pklxkd-58/ 2023-05-31T20:39:07.084 DEBUG:boto:path=/pklxkd-58/?prefix=obj00000000000000000001 2023-05-31T20:39:07.084 DEBUG:boto:auth_path=/pklxkd-58/?prefix=obj00000000000000000001 2023-05-31T20:39:07.085 DEBUG:boto:Method: GET 2023-05-31T20:39:07.085 DEBUG:boto:Path: /pklxkd-58/?prefix=obj00000000000000000001 2023-05-31T20:39:07.085 DEBUG:boto:Data: 2023-05-31T20:39:07.086 DEBUG:boto:Headers: {} 2023-05-31T20:39:07.086 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:39:07.086 DEBUG:boto:Port: 8001 2023-05-31T20:39:07.087 DEBUG:boto:Params: {} 2023-05-31T20:39:07.087 DEBUG:boto:Token: None 2023-05-31T20:39:07.088 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:39:07 GMT /pklxkd-58/ 2023-05-31T20:39:07.088 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:097vQnzeWddc3TithBo9whiVZBE= 2023-05-31T20:39:07.089 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:39:07 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:097vQnzeWddc3TithBo9whiVZBE=', 'Content-Length': '0'} 2023-05-31T20:39:07.091 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ebf7b22e75cb111e-006477b06b-1c17-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:39:07 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:39:07.092 DEBUG:boto:b'pklxkd-58obj000000000000000000011000falseobj000000000000000000012023-05-31T20:38:22.169Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:39:07.093 DEBUG:boto:path=/pklxkd-58/ 2023-05-31T20:39:07.094 DEBUG:boto:auth_path=/pklxkd-58/ 2023-05-31T20:39:07.094 DEBUG:boto:path=/pklxkd-58/?prefix=obj00000000000000000001 2023-05-31T20:39:07.095 DEBUG:boto:auth_path=/pklxkd-58/?prefix=obj00000000000000000001 2023-05-31T20:39:07.095 DEBUG:boto:Method: GET 2023-05-31T20:39:07.095 DEBUG:boto:Path: /pklxkd-58/?prefix=obj00000000000000000001 2023-05-31T20:39:07.096 DEBUG:boto:Data: 2023-05-31T20:39:07.096 DEBUG:boto:Headers: {} 2023-05-31T20:39:07.097 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:39:07.097 DEBUG:boto:Port: 8000 2023-05-31T20:39:07.098 DEBUG:boto:Params: {} 2023-05-31T20:39:07.098 DEBUG:boto:Token: None 2023-05-31T20:39:07.099 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:39:07 GMT /pklxkd-58/ 2023-05-31T20:39:07.099 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:097vQnzeWddc3TithBo9whiVZBE= 2023-05-31T20:39:07.100 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:39:07 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:097vQnzeWddc3TithBo9whiVZBE=', 'Content-Length': '0'} 2023-05-31T20:39:07.115 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000072d94fd8e6aeddc6-006477b06b-1630-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:39:07 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:39:07.152 DEBUG:boto:b'pklxkd-58obj000000000000000000011000falseobj000000000000000000012023-05-31T20:38:22.169Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:39:07.158 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000002 2023-05-31T20:39:07.159 DEBUG:boto:path=/pklxkd-58/obj00000000000000000002 2023-05-31T20:39:07.159 DEBUG:boto:auth_path=/pklxkd-58/obj00000000000000000002 2023-05-31T20:39:07.160 DEBUG:boto:path=/pklxkd-58/obj00000000000000000002?versionId=null 2023-05-31T20:39:07.160 DEBUG:boto:auth_path=/pklxkd-58/obj00000000000000000002?versionId=null 2023-05-31T20:39:07.160 DEBUG:boto:Method: GET 2023-05-31T20:39:07.161 DEBUG:boto:Path: /pklxkd-58/obj00000000000000000002?versionId=null 2023-05-31T20:39:07.161 DEBUG:boto:Data: 2023-05-31T20:39:07.161 DEBUG:boto:Headers: {} 2023-05-31T20:39:07.162 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:39:07.162 DEBUG:boto:Port: 8001 2023-05-31T20:39:07.163 DEBUG:boto:Params: {} 2023-05-31T20:39:07.163 DEBUG:boto:Token: None 2023-05-31T20:39:07.163 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:39:07 GMT /pklxkd-58/obj00000000000000000002?versionId=null 2023-05-31T20:39:07.164 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:SwabPADVvmy62PYzJ/azJ6Ovjb4= 2023-05-31T20:39:07.165 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:39:07 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:SwabPADVvmy62PYzJ/azJ6Ovjb4=', 'Content-Length': '0'} 2023-05-31T20:39:07.167 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:38:22 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000093fbe4142c888d1d-006477b06b-1c17-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:39:07 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:39:07.168 DEBUG:boto:path=/pklxkd-58/obj00000000000000000002 2023-05-31T20:39:07.168 DEBUG:boto:auth_path=/pklxkd-58/obj00000000000000000002 2023-05-31T20:39:07.169 DEBUG:boto:path=/pklxkd-58/obj00000000000000000002?versionId=null 2023-05-31T20:39:07.169 DEBUG:boto:auth_path=/pklxkd-58/obj00000000000000000002?versionId=null 2023-05-31T20:39:07.169 DEBUG:boto:Method: GET 2023-05-31T20:39:07.170 DEBUG:boto:Path: /pklxkd-58/obj00000000000000000002?versionId=null 2023-05-31T20:39:07.170 DEBUG:boto:Data: 2023-05-31T20:39:07.170 DEBUG:boto:Headers: {} 2023-05-31T20:39:07.171 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:39:07.172 DEBUG:boto:Port: 8000 2023-05-31T20:39:07.172 DEBUG:boto:Params: {} 2023-05-31T20:39:07.172 DEBUG:boto:Token: None 2023-05-31T20:39:07.173 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:39:07 GMT /pklxkd-58/obj00000000000000000002?versionId=null 2023-05-31T20:39:07.173 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:SwabPADVvmy62PYzJ/azJ6Ovjb4= 2023-05-31T20:39:07.174 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:39:07 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:SwabPADVvmy62PYzJ/azJ6Ovjb4=', 'Content-Length': '0'} 2023-05-31T20:39:07.177 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:38:22 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000041c76acb0f4c09bf-006477b06b-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:39:07 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:39:07.180 DEBUG:boto:path=/pklxkd-58/obj00000000000000000002 2023-05-31T20:39:07.180 DEBUG:boto:auth_path=/pklxkd-58/obj00000000000000000002 2023-05-31T20:39:07.184 DEBUG:boto:path=/pklxkd-58/obj00000000000000000002?versionId=null 2023-05-31T20:39:07.185 DEBUG:boto:auth_path=/pklxkd-58/obj00000000000000000002?versionId=null 2023-05-31T20:39:07.185 DEBUG:boto:Method: HEAD 2023-05-31T20:39:07.188 DEBUG:boto:Path: /pklxkd-58/obj00000000000000000002?versionId=null 2023-05-31T20:39:07.189 DEBUG:boto:Data: 2023-05-31T20:39:07.189 DEBUG:boto:Headers: {} 2023-05-31T20:39:07.190 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:39:07.190 DEBUG:boto:Port: 8001 2023-05-31T20:39:07.190 DEBUG:boto:Params: {} 2023-05-31T20:39:07.191 DEBUG:boto:Token: None 2023-05-31T20:39:07.191 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:39:07 GMT /pklxkd-58/obj00000000000000000002?versionId=null 2023-05-31T20:39:07.191 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:cUclLodZdV2PHQUn8eroNryZAKU= 2023-05-31T20:39:07.192 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:39:07 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:cUclLodZdV2PHQUn8eroNryZAKU=', 'Content-Length': '0'} 2023-05-31T20:39:07.195 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:38:22 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000cc998d70b7c6d65f-006477b06b-1c17-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:39:07 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:39:07.196 DEBUG:boto:path=/pklxkd-58/obj00000000000000000002 2023-05-31T20:39:07.196 DEBUG:boto:auth_path=/pklxkd-58/obj00000000000000000002 2023-05-31T20:39:07.196 DEBUG:boto:path=/pklxkd-58/obj00000000000000000002?versionId=null 2023-05-31T20:39:07.197 DEBUG:boto:auth_path=/pklxkd-58/obj00000000000000000002?versionId=null 2023-05-31T20:39:07.197 DEBUG:boto:Method: HEAD 2023-05-31T20:39:07.197 DEBUG:boto:Path: /pklxkd-58/obj00000000000000000002?versionId=null 2023-05-31T20:39:07.198 DEBUG:boto:Data: 2023-05-31T20:39:07.198 DEBUG:boto:Headers: {} 2023-05-31T20:39:07.198 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:39:07.199 DEBUG:boto:Port: 8000 2023-05-31T20:39:07.199 DEBUG:boto:Params: {} 2023-05-31T20:39:07.199 DEBUG:boto:Token: None 2023-05-31T20:39:07.200 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:39:07 GMT /pklxkd-58/obj00000000000000000002?versionId=null 2023-05-31T20:39:07.202 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:cUclLodZdV2PHQUn8eroNryZAKU= 2023-05-31T20:39:07.203 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:39:07 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:cUclLodZdV2PHQUn8eroNryZAKU=', 'Content-Length': '0'} 2023-05-31T20:39:07.207 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:38:22 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b11db37ae570956b-006477b06b-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:39:07 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:39:07.207 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000002 2023-05-31T20:39:07.208 DEBUG:boto:path=/pklxkd-58/obj00000000000000000002 2023-05-31T20:39:07.208 DEBUG:boto:auth_path=/pklxkd-58/obj00000000000000000002 2023-05-31T20:39:07.209 DEBUG:boto:path=/pklxkd-58/obj00000000000000000002?versionId=null 2023-05-31T20:39:07.209 DEBUG:boto:auth_path=/pklxkd-58/obj00000000000000000002?versionId=null 2023-05-31T20:39:07.210 DEBUG:boto:Method: GET 2023-05-31T20:39:07.210 DEBUG:boto:Path: /pklxkd-58/obj00000000000000000002?versionId=null 2023-05-31T20:39:07.211 DEBUG:boto:Data: 2023-05-31T20:39:07.211 DEBUG:boto:Headers: {} 2023-05-31T20:39:07.211 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:39:07.212 DEBUG:boto:Port: 8001 2023-05-31T20:39:07.212 DEBUG:boto:Params: {} 2023-05-31T20:39:07.213 DEBUG:boto:Token: None 2023-05-31T20:39:07.213 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:39:07 GMT /pklxkd-58/obj00000000000000000002?versionId=null 2023-05-31T20:39:07.214 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:SwabPADVvmy62PYzJ/azJ6Ovjb4= 2023-05-31T20:39:07.214 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:39:07 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:SwabPADVvmy62PYzJ/azJ6Ovjb4=', 'Content-Length': '0'} 2023-05-31T20:39:07.216 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:38:22 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000084d8bad34e424dcc-006477b06b-1c17-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:39:07 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:39:07.217 DEBUG:boto:path=/pklxkd-58/obj00000000000000000002 2023-05-31T20:39:07.218 DEBUG:boto:auth_path=/pklxkd-58/obj00000000000000000002 2023-05-31T20:39:07.218 DEBUG:boto:path=/pklxkd-58/obj00000000000000000002?versionId=null 2023-05-31T20:39:07.219 DEBUG:boto:auth_path=/pklxkd-58/obj00000000000000000002?versionId=null 2023-05-31T20:39:07.219 DEBUG:boto:Method: GET 2023-05-31T20:39:07.220 DEBUG:boto:Path: /pklxkd-58/obj00000000000000000002?versionId=null 2023-05-31T20:39:07.220 DEBUG:boto:Data: 2023-05-31T20:39:07.221 DEBUG:boto:Headers: {} 2023-05-31T20:39:07.225 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:39:07.225 DEBUG:boto:Port: 8000 2023-05-31T20:39:07.225 DEBUG:boto:Params: {} 2023-05-31T20:39:07.226 DEBUG:boto:Token: None 2023-05-31T20:39:07.226 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:39:07 GMT /pklxkd-58/obj00000000000000000002?versionId=null 2023-05-31T20:39:07.227 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:SwabPADVvmy62PYzJ/azJ6Ovjb4= 2023-05-31T20:39:07.227 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:39:07 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:SwabPADVvmy62PYzJ/azJ6Ovjb4=', 'Content-Length': '0'} 2023-05-31T20:39:07.229 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:38:22 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008cc70c258cc30f48-006477b06b-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:39:07 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:39:07.272 DEBUG:boto:path=/pklxkd-58/ 2023-05-31T20:39:07.273 DEBUG:boto:auth_path=/pklxkd-58/ 2023-05-31T20:39:07.274 DEBUG:boto:path=/pklxkd-58/?prefix=obj00000000000000000002 2023-05-31T20:39:07.274 DEBUG:boto:auth_path=/pklxkd-58/?prefix=obj00000000000000000002 2023-05-31T20:39:07.274 DEBUG:boto:Method: GET 2023-05-31T20:39:07.275 DEBUG:boto:Path: /pklxkd-58/?prefix=obj00000000000000000002 2023-05-31T20:39:07.275 DEBUG:boto:Data: 2023-05-31T20:39:07.276 DEBUG:boto:Headers: {} 2023-05-31T20:39:07.276 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:39:07.276 DEBUG:boto:Port: 8001 2023-05-31T20:39:07.277 DEBUG:boto:Params: {} 2023-05-31T20:39:07.277 DEBUG:boto:Token: None 2023-05-31T20:39:07.278 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:39:07 GMT /pklxkd-58/ 2023-05-31T20:39:07.278 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:097vQnzeWddc3TithBo9whiVZBE= 2023-05-31T20:39:07.279 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:39:07 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:097vQnzeWddc3TithBo9whiVZBE=', 'Content-Length': '0'} 2023-05-31T20:39:07.281 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000004edd309cfbd1bbdf-006477b06b-1c17-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:39:07 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:39:07.283 DEBUG:boto:b'pklxkd-58obj000000000000000000021000falseobj000000000000000000022023-05-31T20:38:22.061Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:39:07.284 DEBUG:boto:path=/pklxkd-58/ 2023-05-31T20:39:07.285 DEBUG:boto:auth_path=/pklxkd-58/ 2023-05-31T20:39:07.286 DEBUG:boto:path=/pklxkd-58/?prefix=obj00000000000000000002 2023-05-31T20:39:07.286 DEBUG:boto:auth_path=/pklxkd-58/?prefix=obj00000000000000000002 2023-05-31T20:39:07.287 DEBUG:boto:Method: GET 2023-05-31T20:39:07.288 DEBUG:boto:Path: /pklxkd-58/?prefix=obj00000000000000000002 2023-05-31T20:39:07.289 DEBUG:boto:Data: 2023-05-31T20:39:07.290 DEBUG:boto:Headers: {} 2023-05-31T20:39:07.292 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:39:07.293 DEBUG:boto:Port: 8000 2023-05-31T20:39:07.293 DEBUG:boto:Params: {} 2023-05-31T20:39:07.294 DEBUG:boto:Token: None 2023-05-31T20:39:07.295 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:39:07 GMT /pklxkd-58/ 2023-05-31T20:39:07.295 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:097vQnzeWddc3TithBo9whiVZBE= 2023-05-31T20:39:07.296 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:39:07 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:097vQnzeWddc3TithBo9whiVZBE=', 'Content-Length': '0'} 2023-05-31T20:39:07.300 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000062de8c547625fee-006477b06b-1630-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:39:07 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:39:07.301 DEBUG:boto:b'pklxkd-58obj000000000000000000021000falseobj000000000000000000022023-05-31T20:38:22.061Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:39:07.301 DEBUG:boto:path=/pklxkd-58/obj00000000000000000002 2023-05-31T20:39:07.302 DEBUG:boto:auth_path=/pklxkd-58/obj00000000000000000002 2023-05-31T20:39:07.302 DEBUG:boto:Method: HEAD 2023-05-31T20:39:07.305 DEBUG:boto:Path: /pklxkd-58/obj00000000000000000002 2023-05-31T20:39:07.307 DEBUG:boto:Data: 2023-05-31T20:39:07.307 DEBUG:boto:Headers: {} 2023-05-31T20:39:07.307 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:39:07.308 DEBUG:boto:Port: 8001 2023-05-31T20:39:07.308 DEBUG:boto:Params: {} 2023-05-31T20:39:07.308 DEBUG:boto:Token: None 2023-05-31T20:39:07.309 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:39:07 GMT /pklxkd-58/obj00000000000000000002 2023-05-31T20:39:07.309 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:DzRvZbjSpxbqtmu9X8++U/cHRBU= 2023-05-31T20:39:07.310 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:39:07 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:DzRvZbjSpxbqtmu9X8++U/cHRBU=', 'Content-Length': '0'} 2023-05-31T20:39:07.312 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:38:22 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000079338eadeaec0cdc-006477b06b-1c17-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:39:07 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:39:07.313 DEBUG:boto:path=/pklxkd-58/obj00000000000000000002 2023-05-31T20:39:07.313 DEBUG:boto:auth_path=/pklxkd-58/obj00000000000000000002 2023-05-31T20:39:07.314 DEBUG:boto:Method: HEAD 2023-05-31T20:39:07.314 DEBUG:boto:Path: /pklxkd-58/obj00000000000000000002 2023-05-31T20:39:07.314 DEBUG:boto:Data: 2023-05-31T20:39:07.315 DEBUG:boto:Headers: {} 2023-05-31T20:39:07.315 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:39:07.315 DEBUG:boto:Port: 8000 2023-05-31T20:39:07.316 DEBUG:boto:Params: {} 2023-05-31T20:39:07.317 DEBUG:boto:Token: None 2023-05-31T20:39:07.317 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:39:07 GMT /pklxkd-58/obj00000000000000000002 2023-05-31T20:39:07.318 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:DzRvZbjSpxbqtmu9X8++U/cHRBU= 2023-05-31T20:39:07.318 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:39:07 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:DzRvZbjSpxbqtmu9X8++U/cHRBU=', 'Content-Length': '0'} 2023-05-31T20:39:07.320 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:38:22 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000a4eaced3d3b8fba-006477b06b-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:39:07 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:39:07.321 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000002 2023-05-31T20:39:07.321 DEBUG:boto:path=/pklxkd-58/obj00000000000000000002 2023-05-31T20:39:07.321 DEBUG:boto:auth_path=/pklxkd-58/obj00000000000000000002 2023-05-31T20:39:07.322 DEBUG:boto:Method: GET 2023-05-31T20:39:07.322 DEBUG:boto:Path: /pklxkd-58/obj00000000000000000002 2023-05-31T20:39:07.323 DEBUG:boto:Data: 2023-05-31T20:39:07.323 DEBUG:boto:Headers: {} 2023-05-31T20:39:07.324 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:39:07.324 DEBUG:boto:Port: 8001 2023-05-31T20:39:07.324 DEBUG:boto:Params: {} 2023-05-31T20:39:07.325 DEBUG:boto:Token: None 2023-05-31T20:39:07.325 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:39:07 GMT /pklxkd-58/obj00000000000000000002 2023-05-31T20:39:07.326 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:zcZIarlL6YsHyNuVGbvOVB95DoU= 2023-05-31T20:39:07.327 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:39:07 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:zcZIarlL6YsHyNuVGbvOVB95DoU=', 'Content-Length': '0'} 2023-05-31T20:39:07.332 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:38:22 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000015e94772b4fed85e-006477b06b-1c17-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:39:07 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:39:07.333 DEBUG:boto:path=/pklxkd-58/obj00000000000000000002 2023-05-31T20:39:07.333 DEBUG:boto:auth_path=/pklxkd-58/obj00000000000000000002 2023-05-31T20:39:07.334 DEBUG:boto:Method: GET 2023-05-31T20:39:07.336 DEBUG:boto:Path: /pklxkd-58/obj00000000000000000002 2023-05-31T20:39:07.337 DEBUG:boto:Data: 2023-05-31T20:39:07.339 DEBUG:boto:Headers: {} 2023-05-31T20:39:07.340 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:39:07.340 DEBUG:boto:Port: 8000 2023-05-31T20:39:07.341 DEBUG:boto:Params: {} 2023-05-31T20:39:07.342 DEBUG:boto:Token: None 2023-05-31T20:39:07.343 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:39:07 GMT /pklxkd-58/obj00000000000000000002 2023-05-31T20:39:07.344 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:zcZIarlL6YsHyNuVGbvOVB95DoU= 2023-05-31T20:39:07.345 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:39:07 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:zcZIarlL6YsHyNuVGbvOVB95DoU=', 'Content-Length': '0'} 2023-05-31T20:39:07.350 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:38:22 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ec0a0aa5de5dca3c-006477b06b-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:39:07 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:39:07.351 DEBUG:boto:path=/pklxkd-58/ 2023-05-31T20:39:07.351 DEBUG:boto:auth_path=/pklxkd-58/ 2023-05-31T20:39:07.352 DEBUG:boto:path=/pklxkd-58/?prefix=obj00000000000000000002 2023-05-31T20:39:07.352 DEBUG:boto:auth_path=/pklxkd-58/?prefix=obj00000000000000000002 2023-05-31T20:39:07.353 DEBUG:boto:Method: GET 2023-05-31T20:39:07.353 DEBUG:boto:Path: /pklxkd-58/?prefix=obj00000000000000000002 2023-05-31T20:39:07.354 DEBUG:boto:Data: 2023-05-31T20:39:07.354 DEBUG:boto:Headers: {} 2023-05-31T20:39:07.354 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:39:07.355 DEBUG:boto:Port: 8001 2023-05-31T20:39:07.355 DEBUG:boto:Params: {} 2023-05-31T20:39:07.357 DEBUG:boto:Token: None 2023-05-31T20:39:07.357 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:39:07 GMT /pklxkd-58/ 2023-05-31T20:39:07.358 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:097vQnzeWddc3TithBo9whiVZBE= 2023-05-31T20:39:07.359 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:39:07 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:097vQnzeWddc3TithBo9whiVZBE=', 'Content-Length': '0'} 2023-05-31T20:39:07.361 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000332b1ee2db45ce66-006477b06b-1c17-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:39:07 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:39:07.404 DEBUG:boto:b'pklxkd-58obj000000000000000000021000falseobj000000000000000000022023-05-31T20:38:22.061Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:39:07.407 DEBUG:boto:path=/pklxkd-58/ 2023-05-31T20:39:07.408 DEBUG:boto:auth_path=/pklxkd-58/ 2023-05-31T20:39:07.408 DEBUG:boto:path=/pklxkd-58/?prefix=obj00000000000000000002 2023-05-31T20:39:07.409 DEBUG:boto:auth_path=/pklxkd-58/?prefix=obj00000000000000000002 2023-05-31T20:39:07.411 DEBUG:boto:Method: GET 2023-05-31T20:39:07.412 DEBUG:boto:Path: /pklxkd-58/?prefix=obj00000000000000000002 2023-05-31T20:39:07.413 DEBUG:boto:Data: 2023-05-31T20:39:07.414 DEBUG:boto:Headers: {} 2023-05-31T20:39:07.420 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:39:07.420 DEBUG:boto:Port: 8000 2023-05-31T20:39:07.421 DEBUG:boto:Params: {} 2023-05-31T20:39:07.421 DEBUG:boto:Token: None 2023-05-31T20:39:07.422 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:39:07 GMT /pklxkd-58/ 2023-05-31T20:39:07.422 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:097vQnzeWddc3TithBo9whiVZBE= 2023-05-31T20:39:07.423 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:39:07 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:097vQnzeWddc3TithBo9whiVZBE=', 'Content-Length': '0'} 2023-05-31T20:39:07.427 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000fd389966a8dfe2a6-006477b06b-1630-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:39:07 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:39:07.427 DEBUG:boto:b'pklxkd-58obj000000000000000000021000falseobj000000000000000000022023-05-31T20:38:22.061Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:39:07.428 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000003 2023-05-31T20:39:07.428 DEBUG:boto:path=/pklxkd-58/obj00000000000000000003 2023-05-31T20:39:07.429 DEBUG:boto:auth_path=/pklxkd-58/obj00000000000000000003 2023-05-31T20:39:07.429 DEBUG:boto:path=/pklxkd-58/obj00000000000000000003?versionId=null 2023-05-31T20:39:07.430 DEBUG:boto:auth_path=/pklxkd-58/obj00000000000000000003?versionId=null 2023-05-31T20:39:07.430 DEBUG:boto:Method: GET 2023-05-31T20:39:07.430 DEBUG:boto:Path: /pklxkd-58/obj00000000000000000003?versionId=null 2023-05-31T20:39:07.431 DEBUG:boto:Data: 2023-05-31T20:39:07.431 DEBUG:boto:Headers: {} 2023-05-31T20:39:07.431 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:39:07.432 DEBUG:boto:Port: 8001 2023-05-31T20:39:07.432 DEBUG:boto:Params: {} 2023-05-31T20:39:07.433 DEBUG:boto:Token: None 2023-05-31T20:39:07.433 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:39:07 GMT /pklxkd-58/obj00000000000000000003?versionId=null 2023-05-31T20:39:07.433 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:RjHurN8trqR2p05Xxh6wHmqUhDg= 2023-05-31T20:39:07.434 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:39:07 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:RjHurN8trqR2p05Xxh6wHmqUhDg=', 'Content-Length': '0'} 2023-05-31T20:39:07.436 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:38:21 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001ad3e725b089a027-006477b06b-1c17-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:39:07 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:39:07.437 DEBUG:boto:path=/pklxkd-58/obj00000000000000000003 2023-05-31T20:39:07.438 DEBUG:boto:auth_path=/pklxkd-58/obj00000000000000000003 2023-05-31T20:39:07.438 DEBUG:boto:path=/pklxkd-58/obj00000000000000000003?versionId=null 2023-05-31T20:39:07.439 DEBUG:boto:auth_path=/pklxkd-58/obj00000000000000000003?versionId=null 2023-05-31T20:39:07.439 DEBUG:boto:Method: GET 2023-05-31T20:39:07.439 DEBUG:boto:Path: /pklxkd-58/obj00000000000000000003?versionId=null 2023-05-31T20:39:07.440 DEBUG:boto:Data: 2023-05-31T20:39:07.440 DEBUG:boto:Headers: {} 2023-05-31T20:39:07.440 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:39:07.441 DEBUG:boto:Port: 8000 2023-05-31T20:39:07.441 DEBUG:boto:Params: {} 2023-05-31T20:39:07.442 DEBUG:boto:Token: None 2023-05-31T20:39:07.442 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:39:07 GMT /pklxkd-58/obj00000000000000000003?versionId=null 2023-05-31T20:39:07.443 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:RjHurN8trqR2p05Xxh6wHmqUhDg= 2023-05-31T20:39:07.443 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:39:07 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:RjHurN8trqR2p05Xxh6wHmqUhDg=', 'Content-Length': '0'} 2023-05-31T20:39:07.447 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:38:21 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003b66b327b95e01ca-006477b06b-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:39:07 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:39:07.448 DEBUG:boto:path=/pklxkd-58/obj00000000000000000003 2023-05-31T20:39:07.449 DEBUG:boto:auth_path=/pklxkd-58/obj00000000000000000003 2023-05-31T20:39:07.450 DEBUG:boto:path=/pklxkd-58/obj00000000000000000003?versionId=null 2023-05-31T20:39:07.451 DEBUG:boto:auth_path=/pklxkd-58/obj00000000000000000003?versionId=null 2023-05-31T20:39:07.452 DEBUG:boto:Method: HEAD 2023-05-31T20:39:07.454 DEBUG:boto:Path: /pklxkd-58/obj00000000000000000003?versionId=null 2023-05-31T20:39:07.454 DEBUG:boto:Data: 2023-05-31T20:39:07.455 DEBUG:boto:Headers: {} 2023-05-31T20:39:07.455 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:39:07.455 DEBUG:boto:Port: 8001 2023-05-31T20:39:07.456 DEBUG:boto:Params: {} 2023-05-31T20:39:07.456 DEBUG:boto:Token: None 2023-05-31T20:39:07.457 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:39:07 GMT /pklxkd-58/obj00000000000000000003?versionId=null 2023-05-31T20:39:07.457 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:coZ6OR1F0YsvENUreacd0i+pDgc= 2023-05-31T20:39:07.458 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:39:07 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:coZ6OR1F0YsvENUreacd0i+pDgc=', 'Content-Length': '0'} 2023-05-31T20:39:07.460 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:38:21 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000cb65321813e6240e-006477b06b-1c17-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:39:07 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:39:07.460 DEBUG:boto:path=/pklxkd-58/obj00000000000000000003 2023-05-31T20:39:07.461 DEBUG:boto:auth_path=/pklxkd-58/obj00000000000000000003 2023-05-31T20:39:07.461 DEBUG:boto:path=/pklxkd-58/obj00000000000000000003?versionId=null 2023-05-31T20:39:07.462 DEBUG:boto:auth_path=/pklxkd-58/obj00000000000000000003?versionId=null 2023-05-31T20:39:07.462 DEBUG:boto:Method: HEAD 2023-05-31T20:39:07.462 DEBUG:boto:Path: /pklxkd-58/obj00000000000000000003?versionId=null 2023-05-31T20:39:07.463 DEBUG:boto:Data: 2023-05-31T20:39:07.463 DEBUG:boto:Headers: {} 2023-05-31T20:39:07.464 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:39:07.464 DEBUG:boto:Port: 8000 2023-05-31T20:39:07.464 DEBUG:boto:Params: {} 2023-05-31T20:39:07.465 DEBUG:boto:Token: None 2023-05-31T20:39:07.465 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:39:07 GMT /pklxkd-58/obj00000000000000000003?versionId=null 2023-05-31T20:39:07.466 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:coZ6OR1F0YsvENUreacd0i+pDgc= 2023-05-31T20:39:07.466 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:39:07 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:coZ6OR1F0YsvENUreacd0i+pDgc=', 'Content-Length': '0'} 2023-05-31T20:39:07.468 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:38:21 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006e775e8183df4f7c-006477b06b-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:39:07 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:39:07.469 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000003 2023-05-31T20:39:07.470 DEBUG:boto:path=/pklxkd-58/obj00000000000000000003 2023-05-31T20:39:07.470 DEBUG:boto:auth_path=/pklxkd-58/obj00000000000000000003 2023-05-31T20:39:07.470 DEBUG:boto:path=/pklxkd-58/obj00000000000000000003?versionId=null 2023-05-31T20:39:07.471 DEBUG:boto:auth_path=/pklxkd-58/obj00000000000000000003?versionId=null 2023-05-31T20:39:07.471 DEBUG:boto:Method: GET 2023-05-31T20:39:07.471 DEBUG:boto:Path: /pklxkd-58/obj00000000000000000003?versionId=null 2023-05-31T20:39:07.472 DEBUG:boto:Data: 2023-05-31T20:39:07.472 DEBUG:boto:Headers: {} 2023-05-31T20:39:07.473 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:39:07.473 DEBUG:boto:Port: 8001 2023-05-31T20:39:07.473 DEBUG:boto:Params: {} 2023-05-31T20:39:07.474 DEBUG:boto:Token: None 2023-05-31T20:39:07.474 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:39:07 GMT /pklxkd-58/obj00000000000000000003?versionId=null 2023-05-31T20:39:07.475 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:RjHurN8trqR2p05Xxh6wHmqUhDg= 2023-05-31T20:39:07.475 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:39:07 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:RjHurN8trqR2p05Xxh6wHmqUhDg=', 'Content-Length': '0'} 2023-05-31T20:39:07.477 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:38:21 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e8e5c41de5c9cbb3-006477b06b-1c17-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:39:07 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:39:07.477 DEBUG:boto:path=/pklxkd-58/obj00000000000000000003 2023-05-31T20:39:07.478 DEBUG:boto:auth_path=/pklxkd-58/obj00000000000000000003 2023-05-31T20:39:07.479 DEBUG:boto:path=/pklxkd-58/obj00000000000000000003?versionId=null 2023-05-31T20:39:07.479 DEBUG:boto:auth_path=/pklxkd-58/obj00000000000000000003?versionId=null 2023-05-31T20:39:07.480 DEBUG:boto:Method: GET 2023-05-31T20:39:07.480 DEBUG:boto:Path: /pklxkd-58/obj00000000000000000003?versionId=null 2023-05-31T20:39:07.481 DEBUG:boto:Data: 2023-05-31T20:39:07.481 DEBUG:boto:Headers: {} 2023-05-31T20:39:07.481 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:39:07.482 DEBUG:boto:Port: 8000 2023-05-31T20:39:07.482 DEBUG:boto:Params: {} 2023-05-31T20:39:07.482 DEBUG:boto:Token: None 2023-05-31T20:39:07.483 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:39:07 GMT /pklxkd-58/obj00000000000000000003?versionId=null 2023-05-31T20:39:07.483 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:RjHurN8trqR2p05Xxh6wHmqUhDg= 2023-05-31T20:39:07.484 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:39:07 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:RjHurN8trqR2p05Xxh6wHmqUhDg=', 'Content-Length': '0'} 2023-05-31T20:39:07.486 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:38:21 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008eb8b5aa465d20b9-006477b06b-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:39:07 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:39:07.528 DEBUG:boto:path=/pklxkd-58/ 2023-05-31T20:39:07.529 DEBUG:boto:auth_path=/pklxkd-58/ 2023-05-31T20:39:07.529 DEBUG:boto:path=/pklxkd-58/?prefix=obj00000000000000000003 2023-05-31T20:39:07.530 DEBUG:boto:auth_path=/pklxkd-58/?prefix=obj00000000000000000003 2023-05-31T20:39:07.530 DEBUG:boto:Method: GET 2023-05-31T20:39:07.530 DEBUG:boto:Path: /pklxkd-58/?prefix=obj00000000000000000003 2023-05-31T20:39:07.531 DEBUG:boto:Data: 2023-05-31T20:39:07.531 DEBUG:boto:Headers: {} 2023-05-31T20:39:07.532 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:39:07.532 DEBUG:boto:Port: 8001 2023-05-31T20:39:07.532 DEBUG:boto:Params: {} 2023-05-31T20:39:07.533 DEBUG:boto:Token: None 2023-05-31T20:39:07.533 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:39:07 GMT /pklxkd-58/ 2023-05-31T20:39:07.534 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:097vQnzeWddc3TithBo9whiVZBE= 2023-05-31T20:39:07.534 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:39:07 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:097vQnzeWddc3TithBo9whiVZBE=', 'Content-Length': '0'} 2023-05-31T20:39:07.537 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000009bb93f3a4ee9a1af-006477b06b-1c17-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:39:07 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:39:07.538 DEBUG:boto:b'pklxkd-58obj000000000000000000031000falseobj000000000000000000032023-05-31T20:38:21.910Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:39:07.539 DEBUG:boto:path=/pklxkd-58/ 2023-05-31T20:39:07.539 DEBUG:boto:auth_path=/pklxkd-58/ 2023-05-31T20:39:07.539 DEBUG:boto:path=/pklxkd-58/?prefix=obj00000000000000000003 2023-05-31T20:39:07.540 DEBUG:boto:auth_path=/pklxkd-58/?prefix=obj00000000000000000003 2023-05-31T20:39:07.540 DEBUG:boto:Method: GET 2023-05-31T20:39:07.541 DEBUG:boto:Path: /pklxkd-58/?prefix=obj00000000000000000003 2023-05-31T20:39:07.541 DEBUG:boto:Data: 2023-05-31T20:39:07.541 DEBUG:boto:Headers: {} 2023-05-31T20:39:07.542 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:39:07.542 DEBUG:boto:Port: 8000 2023-05-31T20:39:07.542 DEBUG:boto:Params: {} 2023-05-31T20:39:07.543 DEBUG:boto:Token: None 2023-05-31T20:39:07.543 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:39:07 GMT /pklxkd-58/ 2023-05-31T20:39:07.544 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:097vQnzeWddc3TithBo9whiVZBE= 2023-05-31T20:39:07.544 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:39:07 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:097vQnzeWddc3TithBo9whiVZBE=', 'Content-Length': '0'} 2023-05-31T20:39:07.549 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b976bb695e3bf46c-006477b06b-1630-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:39:07 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:39:07.549 DEBUG:boto:b'pklxkd-58obj000000000000000000031000falseobj000000000000000000032023-05-31T20:38:21.910Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:39:07.550 DEBUG:boto:path=/pklxkd-58/obj00000000000000000003 2023-05-31T20:39:07.550 DEBUG:boto:auth_path=/pklxkd-58/obj00000000000000000003 2023-05-31T20:39:07.551 DEBUG:boto:Method: HEAD 2023-05-31T20:39:07.551 DEBUG:boto:Path: /pklxkd-58/obj00000000000000000003 2023-05-31T20:39:07.551 DEBUG:boto:Data: 2023-05-31T20:39:07.552 DEBUG:boto:Headers: {} 2023-05-31T20:39:07.552 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:39:07.552 DEBUG:boto:Port: 8001 2023-05-31T20:39:07.553 DEBUG:boto:Params: {} 2023-05-31T20:39:07.553 DEBUG:boto:Token: None 2023-05-31T20:39:07.554 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:39:07 GMT /pklxkd-58/obj00000000000000000003 2023-05-31T20:39:07.554 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:RrbpeKUlwghmGSwHof1oX2u1isY= 2023-05-31T20:39:07.554 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:39:07 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:RrbpeKUlwghmGSwHof1oX2u1isY=', 'Content-Length': '0'} 2023-05-31T20:39:07.556 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:38:21 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000062c3f9c6d57c0b8-006477b06b-1c17-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:39:07 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:39:07.557 DEBUG:boto:path=/pklxkd-58/obj00000000000000000003 2023-05-31T20:39:07.558 DEBUG:boto:auth_path=/pklxkd-58/obj00000000000000000003 2023-05-31T20:39:07.558 DEBUG:boto:Method: HEAD 2023-05-31T20:39:07.559 DEBUG:boto:Path: /pklxkd-58/obj00000000000000000003 2023-05-31T20:39:07.559 DEBUG:boto:Data: 2023-05-31T20:39:07.560 DEBUG:boto:Headers: {} 2023-05-31T20:39:07.560 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:39:07.561 DEBUG:boto:Port: 8000 2023-05-31T20:39:07.561 DEBUG:boto:Params: {} 2023-05-31T20:39:07.561 DEBUG:boto:Token: None 2023-05-31T20:39:07.562 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:39:07 GMT /pklxkd-58/obj00000000000000000003 2023-05-31T20:39:07.562 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:RrbpeKUlwghmGSwHof1oX2u1isY= 2023-05-31T20:39:07.563 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:39:07 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:RrbpeKUlwghmGSwHof1oX2u1isY=', 'Content-Length': '0'} 2023-05-31T20:39:07.565 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:38:21 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f92246588deb70c3-006477b06b-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:39:07 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:39:07.565 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000003 2023-05-31T20:39:07.566 DEBUG:boto:path=/pklxkd-58/obj00000000000000000003 2023-05-31T20:39:07.566 DEBUG:boto:auth_path=/pklxkd-58/obj00000000000000000003 2023-05-31T20:39:07.566 DEBUG:boto:Method: GET 2023-05-31T20:39:07.567 DEBUG:boto:Path: /pklxkd-58/obj00000000000000000003 2023-05-31T20:39:07.567 DEBUG:boto:Data: 2023-05-31T20:39:07.567 DEBUG:boto:Headers: {} 2023-05-31T20:39:07.568 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:39:07.568 DEBUG:boto:Port: 8001 2023-05-31T20:39:07.569 DEBUG:boto:Params: {} 2023-05-31T20:39:07.569 DEBUG:boto:Token: None 2023-05-31T20:39:07.569 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:39:07 GMT /pklxkd-58/obj00000000000000000003 2023-05-31T20:39:07.570 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:VbJsCsORlmAP2Fta4XnRihURzOQ= 2023-05-31T20:39:07.570 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:39:07 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:VbJsCsORlmAP2Fta4XnRihURzOQ=', 'Content-Length': '0'} 2023-05-31T20:39:07.572 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:38:21 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c598e443db03e12e-006477b06b-1c17-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:39:07 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:39:07.616 DEBUG:boto:path=/pklxkd-58/obj00000000000000000003 2023-05-31T20:39:07.618 DEBUG:boto:auth_path=/pklxkd-58/obj00000000000000000003 2023-05-31T20:39:07.619 DEBUG:boto:Method: GET 2023-05-31T20:39:07.619 DEBUG:boto:Path: /pklxkd-58/obj00000000000000000003 2023-05-31T20:39:07.620 DEBUG:boto:Data: 2023-05-31T20:39:07.621 DEBUG:boto:Headers: {} 2023-05-31T20:39:07.622 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:39:07.622 DEBUG:boto:Port: 8000 2023-05-31T20:39:07.623 DEBUG:boto:Params: {} 2023-05-31T20:39:07.623 DEBUG:boto:Token: None 2023-05-31T20:39:07.624 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:39:07 GMT /pklxkd-58/obj00000000000000000003 2023-05-31T20:39:07.625 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:VbJsCsORlmAP2Fta4XnRihURzOQ= 2023-05-31T20:39:07.626 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:39:07 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:VbJsCsORlmAP2Fta4XnRihURzOQ=', 'Content-Length': '0'} 2023-05-31T20:39:07.630 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:38:21 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000be580bd29383346d-006477b06b-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:39:07 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:39:07.631 DEBUG:boto:path=/pklxkd-58/ 2023-05-31T20:39:07.634 DEBUG:boto:auth_path=/pklxkd-58/ 2023-05-31T20:39:07.635 DEBUG:boto:path=/pklxkd-58/?prefix=obj00000000000000000003 2023-05-31T20:39:07.637 DEBUG:boto:auth_path=/pklxkd-58/?prefix=obj00000000000000000003 2023-05-31T20:39:07.638 DEBUG:boto:Method: GET 2023-05-31T20:39:07.638 DEBUG:boto:Path: /pklxkd-58/?prefix=obj00000000000000000003 2023-05-31T20:39:07.639 DEBUG:boto:Data: 2023-05-31T20:39:07.640 DEBUG:boto:Headers: {} 2023-05-31T20:39:07.644 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:39:07.645 DEBUG:boto:Port: 8001 2023-05-31T20:39:07.645 DEBUG:boto:Params: {} 2023-05-31T20:39:07.645 DEBUG:boto:Token: None 2023-05-31T20:39:07.646 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:39:07 GMT /pklxkd-58/ 2023-05-31T20:39:07.647 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:097vQnzeWddc3TithBo9whiVZBE= 2023-05-31T20:39:07.647 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:39:07 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:097vQnzeWddc3TithBo9whiVZBE=', 'Content-Length': '0'} 2023-05-31T20:39:07.654 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000079cfed799ca1d875-006477b06b-1c17-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:39:07 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:39:07.654 DEBUG:boto:b'pklxkd-58obj000000000000000000031000falseobj000000000000000000032023-05-31T20:38:21.910Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:39:07.655 DEBUG:boto:path=/pklxkd-58/ 2023-05-31T20:39:07.656 DEBUG:boto:auth_path=/pklxkd-58/ 2023-05-31T20:39:07.656 DEBUG:boto:path=/pklxkd-58/?prefix=obj00000000000000000003 2023-05-31T20:39:07.656 DEBUG:boto:auth_path=/pklxkd-58/?prefix=obj00000000000000000003 2023-05-31T20:39:07.657 DEBUG:boto:Method: GET 2023-05-31T20:39:07.657 DEBUG:boto:Path: /pklxkd-58/?prefix=obj00000000000000000003 2023-05-31T20:39:07.658 DEBUG:boto:Data: 2023-05-31T20:39:07.658 DEBUG:boto:Headers: {} 2023-05-31T20:39:07.658 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:39:07.659 DEBUG:boto:Port: 8000 2023-05-31T20:39:07.659 DEBUG:boto:Params: {} 2023-05-31T20:39:07.659 DEBUG:boto:Token: None 2023-05-31T20:39:07.660 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:39:07 GMT /pklxkd-58/ 2023-05-31T20:39:07.664 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:097vQnzeWddc3TithBo9whiVZBE= 2023-05-31T20:39:07.664 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:39:07 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:097vQnzeWddc3TithBo9whiVZBE=', 'Content-Length': '0'} 2023-05-31T20:39:07.669 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c82d1457ee744741-006477b06b-1630-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:39:07 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:39:07.714 DEBUG:boto:b'pklxkd-58obj000000000000000000031000falseobj000000000000000000032023-05-31T20:38:21.910Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:39:07.715 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000004 2023-05-31T20:39:07.716 DEBUG:boto:path=/pklxkd-58/obj00000000000000000004 2023-05-31T20:39:07.716 DEBUG:boto:auth_path=/pklxkd-58/obj00000000000000000004 2023-05-31T20:39:07.717 DEBUG:boto:path=/pklxkd-58/obj00000000000000000004?versionId=null 2023-05-31T20:39:07.717 DEBUG:boto:auth_path=/pklxkd-58/obj00000000000000000004?versionId=null 2023-05-31T20:39:07.718 DEBUG:boto:Method: GET 2023-05-31T20:39:07.718 DEBUG:boto:Path: /pklxkd-58/obj00000000000000000004?versionId=null 2023-05-31T20:39:07.719 DEBUG:boto:Data: 2023-05-31T20:39:07.720 DEBUG:boto:Headers: {} 2023-05-31T20:39:07.720 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:39:07.720 DEBUG:boto:Port: 8001 2023-05-31T20:39:07.721 DEBUG:boto:Params: {} 2023-05-31T20:39:07.722 DEBUG:boto:Token: None 2023-05-31T20:39:07.722 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:39:07 GMT /pklxkd-58/obj00000000000000000004?versionId=null 2023-05-31T20:39:07.723 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:++4mhuLr2VksLe4wiw72IChk2qI= 2023-05-31T20:39:07.723 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:39:07 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:++4mhuLr2VksLe4wiw72IChk2qI=', 'Content-Length': '0'} 2023-05-31T20:39:07.727 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:38:22 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c4fa145368b3b386-006477b06b-1c17-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:39:07 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:39:07.730 DEBUG:boto:path=/pklxkd-58/obj00000000000000000004 2023-05-31T20:39:07.732 DEBUG:boto:auth_path=/pklxkd-58/obj00000000000000000004 2023-05-31T20:39:07.733 DEBUG:boto:path=/pklxkd-58/obj00000000000000000004?versionId=null 2023-05-31T20:39:07.734 DEBUG:boto:auth_path=/pklxkd-58/obj00000000000000000004?versionId=null 2023-05-31T20:39:07.734 DEBUG:boto:Method: GET 2023-05-31T20:39:07.735 DEBUG:boto:Path: /pklxkd-58/obj00000000000000000004?versionId=null 2023-05-31T20:39:07.736 DEBUG:boto:Data: 2023-05-31T20:39:07.737 DEBUG:boto:Headers: {} 2023-05-31T20:39:07.738 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:39:07.739 DEBUG:boto:Port: 8000 2023-05-31T20:39:07.739 DEBUG:boto:Params: {} 2023-05-31T20:39:07.740 DEBUG:boto:Token: None 2023-05-31T20:39:07.741 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:39:07 GMT /pklxkd-58/obj00000000000000000004?versionId=null 2023-05-31T20:39:07.742 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:++4mhuLr2VksLe4wiw72IChk2qI= 2023-05-31T20:39:07.744 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:39:07 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:++4mhuLr2VksLe4wiw72IChk2qI=', 'Content-Length': '0'} 2023-05-31T20:39:07.746 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:38:22 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000afb830268be4600f-006477b06b-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:39:07 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:39:07.747 DEBUG:boto:path=/pklxkd-58/obj00000000000000000004 2023-05-31T20:39:07.748 DEBUG:boto:auth_path=/pklxkd-58/obj00000000000000000004 2023-05-31T20:39:07.748 DEBUG:boto:path=/pklxkd-58/obj00000000000000000004?versionId=null 2023-05-31T20:39:07.748 DEBUG:boto:auth_path=/pklxkd-58/obj00000000000000000004?versionId=null 2023-05-31T20:39:07.749 DEBUG:boto:Method: HEAD 2023-05-31T20:39:07.749 DEBUG:boto:Path: /pklxkd-58/obj00000000000000000004?versionId=null 2023-05-31T20:39:07.749 DEBUG:boto:Data: 2023-05-31T20:39:07.750 DEBUG:boto:Headers: {} 2023-05-31T20:39:07.750 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:39:07.750 DEBUG:boto:Port: 8001 2023-05-31T20:39:07.751 DEBUG:boto:Params: {} 2023-05-31T20:39:07.751 DEBUG:boto:Token: None 2023-05-31T20:39:07.752 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:39:07 GMT /pklxkd-58/obj00000000000000000004?versionId=null 2023-05-31T20:39:07.752 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:mdIEPdD6ZU2PMZlk8criaeg82mU= 2023-05-31T20:39:07.752 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:39:07 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:mdIEPdD6ZU2PMZlk8criaeg82mU=', 'Content-Length': '0'} 2023-05-31T20:39:07.756 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:38:22 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001da608d68884b190-006477b06b-1c17-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:39:07 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:39:07.757 DEBUG:boto:path=/pklxkd-58/obj00000000000000000004 2023-05-31T20:39:07.757 DEBUG:boto:auth_path=/pklxkd-58/obj00000000000000000004 2023-05-31T20:39:07.758 DEBUG:boto:path=/pklxkd-58/obj00000000000000000004?versionId=null 2023-05-31T20:39:07.758 DEBUG:boto:auth_path=/pklxkd-58/obj00000000000000000004?versionId=null 2023-05-31T20:39:07.759 DEBUG:boto:Method: HEAD 2023-05-31T20:39:07.759 DEBUG:boto:Path: /pklxkd-58/obj00000000000000000004?versionId=null 2023-05-31T20:39:07.760 DEBUG:boto:Data: 2023-05-31T20:39:07.760 DEBUG:boto:Headers: {} 2023-05-31T20:39:07.761 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:39:07.761 DEBUG:boto:Port: 8000 2023-05-31T20:39:07.762 DEBUG:boto:Params: {} 2023-05-31T20:39:07.763 DEBUG:boto:Token: None 2023-05-31T20:39:07.764 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:39:07 GMT /pklxkd-58/obj00000000000000000004?versionId=null 2023-05-31T20:39:07.765 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:mdIEPdD6ZU2PMZlk8criaeg82mU= 2023-05-31T20:39:07.766 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:39:07 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:mdIEPdD6ZU2PMZlk8criaeg82mU=', 'Content-Length': '0'} 2023-05-31T20:39:07.768 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:38:22 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000cb0ee4175109146b-006477b06b-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:39:07 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:39:07.769 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000004 2023-05-31T20:39:07.770 DEBUG:boto:path=/pklxkd-58/obj00000000000000000004 2023-05-31T20:39:07.770 DEBUG:boto:auth_path=/pklxkd-58/obj00000000000000000004 2023-05-31T20:39:07.770 DEBUG:boto:path=/pklxkd-58/obj00000000000000000004?versionId=null 2023-05-31T20:39:07.771 DEBUG:boto:auth_path=/pklxkd-58/obj00000000000000000004?versionId=null 2023-05-31T20:39:07.771 DEBUG:boto:Method: GET 2023-05-31T20:39:07.771 DEBUG:boto:Path: /pklxkd-58/obj00000000000000000004?versionId=null 2023-05-31T20:39:07.772 DEBUG:boto:Data: 2023-05-31T20:39:07.772 DEBUG:boto:Headers: {} 2023-05-31T20:39:07.773 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:39:07.773 DEBUG:boto:Port: 8001 2023-05-31T20:39:07.774 DEBUG:boto:Params: {} 2023-05-31T20:39:07.774 DEBUG:boto:Token: None 2023-05-31T20:39:07.776 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:39:07 GMT /pklxkd-58/obj00000000000000000004?versionId=null 2023-05-31T20:39:07.776 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:++4mhuLr2VksLe4wiw72IChk2qI= 2023-05-31T20:39:07.777 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:39:07 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:++4mhuLr2VksLe4wiw72IChk2qI=', 'Content-Length': '0'} 2023-05-31T20:39:07.779 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:38:22 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000133a10aa45506628-006477b06b-1c17-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:39:07 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:39:07.780 DEBUG:boto:path=/pklxkd-58/obj00000000000000000004 2023-05-31T20:39:07.781 DEBUG:boto:auth_path=/pklxkd-58/obj00000000000000000004 2023-05-31T20:39:07.782 DEBUG:boto:path=/pklxkd-58/obj00000000000000000004?versionId=null 2023-05-31T20:39:07.782 DEBUG:boto:auth_path=/pklxkd-58/obj00000000000000000004?versionId=null 2023-05-31T20:39:07.782 DEBUG:boto:Method: GET 2023-05-31T20:39:07.783 DEBUG:boto:Path: /pklxkd-58/obj00000000000000000004?versionId=null 2023-05-31T20:39:07.783 DEBUG:boto:Data: 2023-05-31T20:39:07.783 DEBUG:boto:Headers: {} 2023-05-31T20:39:07.784 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:39:07.785 DEBUG:boto:Port: 8000 2023-05-31T20:39:07.785 DEBUG:boto:Params: {} 2023-05-31T20:39:07.786 DEBUG:boto:Token: None 2023-05-31T20:39:07.786 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:39:07 GMT /pklxkd-58/obj00000000000000000004?versionId=null 2023-05-31T20:39:07.787 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:++4mhuLr2VksLe4wiw72IChk2qI= 2023-05-31T20:39:07.788 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:39:07 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:++4mhuLr2VksLe4wiw72IChk2qI=', 'Content-Length': '0'} 2023-05-31T20:39:07.791 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:38:22 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008f05db7457006b6c-006477b06b-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:39:07 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:39:07.791 DEBUG:boto:path=/pklxkd-58/ 2023-05-31T20:39:07.792 DEBUG:boto:auth_path=/pklxkd-58/ 2023-05-31T20:39:07.792 DEBUG:boto:path=/pklxkd-58/?prefix=obj00000000000000000004 2023-05-31T20:39:07.793 DEBUG:boto:auth_path=/pklxkd-58/?prefix=obj00000000000000000004 2023-05-31T20:39:07.793 DEBUG:boto:Method: GET 2023-05-31T20:39:07.793 DEBUG:boto:Path: /pklxkd-58/?prefix=obj00000000000000000004 2023-05-31T20:39:07.794 DEBUG:boto:Data: 2023-05-31T20:39:07.795 DEBUG:boto:Headers: {} 2023-05-31T20:39:07.795 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:39:07.795 DEBUG:boto:Port: 8001 2023-05-31T20:39:07.796 DEBUG:boto:Params: {} 2023-05-31T20:39:07.796 DEBUG:boto:Token: None 2023-05-31T20:39:07.797 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:39:07 GMT /pklxkd-58/ 2023-05-31T20:39:07.797 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:097vQnzeWddc3TithBo9whiVZBE= 2023-05-31T20:39:07.798 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:39:07 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:097vQnzeWddc3TithBo9whiVZBE=', 'Content-Length': '0'} 2023-05-31T20:39:07.801 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e7a1783315329778-006477b06b-1c17-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:39:07 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:39:07.801 DEBUG:boto:b'pklxkd-58obj000000000000000000041000falseobj000000000000000000042023-05-31T20:38:22.227Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:39:07.803 DEBUG:boto:path=/pklxkd-58/ 2023-05-31T20:39:07.803 DEBUG:boto:auth_path=/pklxkd-58/ 2023-05-31T20:39:07.804 DEBUG:boto:path=/pklxkd-58/?prefix=obj00000000000000000004 2023-05-31T20:39:07.804 DEBUG:boto:auth_path=/pklxkd-58/?prefix=obj00000000000000000004 2023-05-31T20:39:07.804 DEBUG:boto:Method: GET 2023-05-31T20:39:07.805 DEBUG:boto:Path: /pklxkd-58/?prefix=obj00000000000000000004 2023-05-31T20:39:07.805 DEBUG:boto:Data: 2023-05-31T20:39:07.806 DEBUG:boto:Headers: {} 2023-05-31T20:39:07.806 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:39:07.806 DEBUG:boto:Port: 8000 2023-05-31T20:39:07.807 DEBUG:boto:Params: {} 2023-05-31T20:39:07.807 DEBUG:boto:Token: None 2023-05-31T20:39:07.808 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:39:07 GMT /pklxkd-58/ 2023-05-31T20:39:07.809 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:097vQnzeWddc3TithBo9whiVZBE= 2023-05-31T20:39:07.809 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:39:07 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:097vQnzeWddc3TithBo9whiVZBE=', 'Content-Length': '0'} 2023-05-31T20:39:07.814 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000dd747e532054b248-006477b06b-1630-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:39:07 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:39:07.814 DEBUG:boto:b'pklxkd-58obj000000000000000000041000falseobj000000000000000000042023-05-31T20:38:22.227Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:39:07.815 DEBUG:boto:path=/pklxkd-58/obj00000000000000000004 2023-05-31T20:39:07.816 DEBUG:boto:auth_path=/pklxkd-58/obj00000000000000000004 2023-05-31T20:39:07.816 DEBUG:boto:Method: HEAD 2023-05-31T20:39:07.816 DEBUG:boto:Path: /pklxkd-58/obj00000000000000000004 2023-05-31T20:39:07.817 DEBUG:boto:Data: 2023-05-31T20:39:07.817 DEBUG:boto:Headers: {} 2023-05-31T20:39:07.817 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:39:07.818 DEBUG:boto:Port: 8001 2023-05-31T20:39:07.818 DEBUG:boto:Params: {} 2023-05-31T20:39:07.818 DEBUG:boto:Token: None 2023-05-31T20:39:07.819 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:39:07 GMT /pklxkd-58/obj00000000000000000004 2023-05-31T20:39:07.819 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:QbkgnI26jP3j5ytwaSkBF9bOiks= 2023-05-31T20:39:07.820 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:39:07 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:QbkgnI26jP3j5ytwaSkBF9bOiks=', 'Content-Length': '0'} 2023-05-31T20:39:07.822 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:38:22 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000953054595b2492f1-006477b06b-1c17-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:39:07 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:39:07.822 DEBUG:boto:path=/pklxkd-58/obj00000000000000000004 2023-05-31T20:39:07.823 DEBUG:boto:auth_path=/pklxkd-58/obj00000000000000000004 2023-05-31T20:39:07.823 DEBUG:boto:Method: HEAD 2023-05-31T20:39:07.823 DEBUG:boto:Path: /pklxkd-58/obj00000000000000000004 2023-05-31T20:39:07.824 DEBUG:boto:Data: 2023-05-31T20:39:07.824 DEBUG:boto:Headers: {} 2023-05-31T20:39:07.825 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:39:07.825 DEBUG:boto:Port: 8000 2023-05-31T20:39:07.825 DEBUG:boto:Params: {} 2023-05-31T20:39:07.826 DEBUG:boto:Token: None 2023-05-31T20:39:07.826 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:39:07 GMT /pklxkd-58/obj00000000000000000004 2023-05-31T20:39:07.827 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:QbkgnI26jP3j5ytwaSkBF9bOiks= 2023-05-31T20:39:07.827 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:39:07 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:QbkgnI26jP3j5ytwaSkBF9bOiks=', 'Content-Length': '0'} 2023-05-31T20:39:07.829 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:38:22 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000fb47d291e3876816-006477b06b-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:39:07 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:39:07.830 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000004 2023-05-31T20:39:07.830 DEBUG:boto:path=/pklxkd-58/obj00000000000000000004 2023-05-31T20:39:07.830 DEBUG:boto:auth_path=/pklxkd-58/obj00000000000000000004 2023-05-31T20:39:07.831 DEBUG:boto:Method: GET 2023-05-31T20:39:07.831 DEBUG:boto:Path: /pklxkd-58/obj00000000000000000004 2023-05-31T20:39:07.832 DEBUG:boto:Data: 2023-05-31T20:39:07.832 DEBUG:boto:Headers: {} 2023-05-31T20:39:07.833 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:39:07.833 DEBUG:boto:Port: 8001 2023-05-31T20:39:07.833 DEBUG:boto:Params: {} 2023-05-31T20:39:07.834 DEBUG:boto:Token: None 2023-05-31T20:39:07.834 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:39:07 GMT /pklxkd-58/obj00000000000000000004 2023-05-31T20:39:07.835 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:sX1TRV1K3vqODWrxdqOYByl5kjM= 2023-05-31T20:39:07.835 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:39:07 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:sX1TRV1K3vqODWrxdqOYByl5kjM=', 'Content-Length': '0'} 2023-05-31T20:39:07.837 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:38:22 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004091569c89566790-006477b06b-1c17-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:39:07 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:39:07.880 DEBUG:boto:path=/pklxkd-58/obj00000000000000000004 2023-05-31T20:39:07.882 DEBUG:boto:auth_path=/pklxkd-58/obj00000000000000000004 2023-05-31T20:39:07.882 DEBUG:boto:Method: GET 2023-05-31T20:39:07.883 DEBUG:boto:Path: /pklxkd-58/obj00000000000000000004 2023-05-31T20:39:07.884 DEBUG:boto:Data: 2023-05-31T20:39:07.884 DEBUG:boto:Headers: {} 2023-05-31T20:39:07.885 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:39:07.885 DEBUG:boto:Port: 8000 2023-05-31T20:39:07.886 DEBUG:boto:Params: {} 2023-05-31T20:39:07.886 DEBUG:boto:Token: None 2023-05-31T20:39:07.887 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:39:07 GMT /pklxkd-58/obj00000000000000000004 2023-05-31T20:39:07.887 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:sX1TRV1K3vqODWrxdqOYByl5kjM= 2023-05-31T20:39:07.888 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:39:07 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:sX1TRV1K3vqODWrxdqOYByl5kjM=', 'Content-Length': '0'} 2023-05-31T20:39:07.890 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:38:22 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000089ce4dede785a9b1-006477b06b-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:39:07 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:39:07.891 DEBUG:boto:path=/pklxkd-58/ 2023-05-31T20:39:07.891 DEBUG:boto:auth_path=/pklxkd-58/ 2023-05-31T20:39:07.892 DEBUG:boto:path=/pklxkd-58/?prefix=obj00000000000000000004 2023-05-31T20:39:07.892 DEBUG:boto:auth_path=/pklxkd-58/?prefix=obj00000000000000000004 2023-05-31T20:39:07.893 DEBUG:boto:Method: GET 2023-05-31T20:39:07.893 DEBUG:boto:Path: /pklxkd-58/?prefix=obj00000000000000000004 2023-05-31T20:39:07.894 DEBUG:boto:Data: 2023-05-31T20:39:07.895 DEBUG:boto:Headers: {} 2023-05-31T20:39:07.895 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:39:07.896 DEBUG:boto:Port: 8001 2023-05-31T20:39:07.896 DEBUG:boto:Params: {} 2023-05-31T20:39:07.897 DEBUG:boto:Token: None 2023-05-31T20:39:07.898 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:39:07 GMT /pklxkd-58/ 2023-05-31T20:39:07.899 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:097vQnzeWddc3TithBo9whiVZBE= 2023-05-31T20:39:07.899 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:39:07 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:097vQnzeWddc3TithBo9whiVZBE=', 'Content-Length': '0'} 2023-05-31T20:39:07.902 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000003fb89c7be3717480-006477b06b-1c17-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:39:07 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:39:07.903 DEBUG:boto:b'pklxkd-58obj000000000000000000041000falseobj000000000000000000042023-05-31T20:38:22.227Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:39:07.904 DEBUG:boto:path=/pklxkd-58/ 2023-05-31T20:39:07.904 DEBUG:boto:auth_path=/pklxkd-58/ 2023-05-31T20:39:07.905 DEBUG:boto:path=/pklxkd-58/?prefix=obj00000000000000000004 2023-05-31T20:39:07.905 DEBUG:boto:auth_path=/pklxkd-58/?prefix=obj00000000000000000004 2023-05-31T20:39:07.905 DEBUG:boto:Method: GET 2023-05-31T20:39:07.906 DEBUG:boto:Path: /pklxkd-58/?prefix=obj00000000000000000004 2023-05-31T20:39:07.906 DEBUG:boto:Data: 2023-05-31T20:39:07.907 DEBUG:boto:Headers: {} 2023-05-31T20:39:07.907 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:39:07.907 DEBUG:boto:Port: 8000 2023-05-31T20:39:07.908 DEBUG:boto:Params: {} 2023-05-31T20:39:07.908 DEBUG:boto:Token: None 2023-05-31T20:39:07.909 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:39:07 GMT /pklxkd-58/ 2023-05-31T20:39:07.910 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:097vQnzeWddc3TithBo9whiVZBE= 2023-05-31T20:39:07.910 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:39:07 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:097vQnzeWddc3TithBo9whiVZBE=', 'Content-Length': '0'} 2023-05-31T20:39:07.916 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000063eec9831425f79d-006477b06b-1630-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:39:07 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:39:07.916 DEBUG:boto:b'pklxkd-58obj000000000000000000041000falseobj000000000000000000042023-05-31T20:38:22.227Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:39:07.917 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000005 2023-05-31T20:39:07.917 DEBUG:boto:path=/pklxkd-58/obj00000000000000000005 2023-05-31T20:39:07.918 DEBUG:boto:auth_path=/pklxkd-58/obj00000000000000000005 2023-05-31T20:39:07.918 DEBUG:boto:path=/pklxkd-58/obj00000000000000000005?versionId=null 2023-05-31T20:39:07.918 DEBUG:boto:auth_path=/pklxkd-58/obj00000000000000000005?versionId=null 2023-05-31T20:39:07.919 DEBUG:boto:Method: GET 2023-05-31T20:39:07.919 DEBUG:boto:Path: /pklxkd-58/obj00000000000000000005?versionId=null 2023-05-31T20:39:07.920 DEBUG:boto:Data: 2023-05-31T20:39:07.920 DEBUG:boto:Headers: {} 2023-05-31T20:39:07.920 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:39:07.921 DEBUG:boto:Port: 8001 2023-05-31T20:39:07.921 DEBUG:boto:Params: {} 2023-05-31T20:39:07.921 DEBUG:boto:Token: None 2023-05-31T20:39:07.922 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:39:07 GMT /pklxkd-58/obj00000000000000000005?versionId=null 2023-05-31T20:39:07.922 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:crf07j7/XChovMEaWI+MvGU1tfI= 2023-05-31T20:39:07.923 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:39:07 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:crf07j7/XChovMEaWI+MvGU1tfI=', 'Content-Length': '0'} 2023-05-31T20:39:07.925 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:38:21 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000094a87db6b10b5fed-006477b06b-1c17-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:39:07 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:39:07.925 DEBUG:boto:path=/pklxkd-58/obj00000000000000000005 2023-05-31T20:39:07.926 DEBUG:boto:auth_path=/pklxkd-58/obj00000000000000000005 2023-05-31T20:39:07.926 DEBUG:boto:path=/pklxkd-58/obj00000000000000000005?versionId=null 2023-05-31T20:39:07.927 DEBUG:boto:auth_path=/pklxkd-58/obj00000000000000000005?versionId=null 2023-05-31T20:39:07.927 DEBUG:boto:Method: GET 2023-05-31T20:39:07.927 DEBUG:boto:Path: /pklxkd-58/obj00000000000000000005?versionId=null 2023-05-31T20:39:07.928 DEBUG:boto:Data: 2023-05-31T20:39:07.928 DEBUG:boto:Headers: {} 2023-05-31T20:39:07.928 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:39:07.929 DEBUG:boto:Port: 8000 2023-05-31T20:39:07.929 DEBUG:boto:Params: {} 2023-05-31T20:39:07.929 DEBUG:boto:Token: None 2023-05-31T20:39:07.930 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:39:07 GMT /pklxkd-58/obj00000000000000000005?versionId=null 2023-05-31T20:39:07.930 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:crf07j7/XChovMEaWI+MvGU1tfI= 2023-05-31T20:39:07.931 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:39:07 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:crf07j7/XChovMEaWI+MvGU1tfI=', 'Content-Length': '0'} 2023-05-31T20:39:07.932 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:38:21 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004508e37280f6a401-006477b06b-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:39:07 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:39:07.933 DEBUG:boto:path=/pklxkd-58/obj00000000000000000005 2023-05-31T20:39:07.933 DEBUG:boto:auth_path=/pklxkd-58/obj00000000000000000005 2023-05-31T20:39:07.934 DEBUG:boto:path=/pklxkd-58/obj00000000000000000005?versionId=null 2023-05-31T20:39:07.934 DEBUG:boto:auth_path=/pklxkd-58/obj00000000000000000005?versionId=null 2023-05-31T20:39:07.934 DEBUG:boto:Method: HEAD 2023-05-31T20:39:07.935 DEBUG:boto:Path: /pklxkd-58/obj00000000000000000005?versionId=null 2023-05-31T20:39:07.935 DEBUG:boto:Data: 2023-05-31T20:39:07.935 DEBUG:boto:Headers: {} 2023-05-31T20:39:07.936 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:39:07.936 DEBUG:boto:Port: 8001 2023-05-31T20:39:07.936 DEBUG:boto:Params: {} 2023-05-31T20:39:07.937 DEBUG:boto:Token: None 2023-05-31T20:39:07.937 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:39:07 GMT /pklxkd-58/obj00000000000000000005?versionId=null 2023-05-31T20:39:07.938 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:A5n8xDvWgZ11pe1xGZKv8rfba6s= 2023-05-31T20:39:07.938 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:39:07 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:A5n8xDvWgZ11pe1xGZKv8rfba6s=', 'Content-Length': '0'} 2023-05-31T20:39:07.940 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:38:21 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e78838e42280dd88-006477b06b-1c17-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:39:07 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:39:07.940 DEBUG:boto:path=/pklxkd-58/obj00000000000000000005 2023-05-31T20:39:07.941 DEBUG:boto:auth_path=/pklxkd-58/obj00000000000000000005 2023-05-31T20:39:07.941 DEBUG:boto:path=/pklxkd-58/obj00000000000000000005?versionId=null 2023-05-31T20:39:07.941 DEBUG:boto:auth_path=/pklxkd-58/obj00000000000000000005?versionId=null 2023-05-31T20:39:07.942 DEBUG:boto:Method: HEAD 2023-05-31T20:39:07.942 DEBUG:boto:Path: /pklxkd-58/obj00000000000000000005?versionId=null 2023-05-31T20:39:07.942 DEBUG:boto:Data: 2023-05-31T20:39:07.943 DEBUG:boto:Headers: {} 2023-05-31T20:39:07.943 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:39:07.943 DEBUG:boto:Port: 8000 2023-05-31T20:39:07.944 DEBUG:boto:Params: {} 2023-05-31T20:39:07.944 DEBUG:boto:Token: None 2023-05-31T20:39:07.944 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:39:07 GMT /pklxkd-58/obj00000000000000000005?versionId=null 2023-05-31T20:39:07.945 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:A5n8xDvWgZ11pe1xGZKv8rfba6s= 2023-05-31T20:39:07.945 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:39:07 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:A5n8xDvWgZ11pe1xGZKv8rfba6s=', 'Content-Length': '0'} 2023-05-31T20:39:07.947 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:38:21 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b9b14dffe82d57a9-006477b06b-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:39:07 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:39:07.948 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000005 2023-05-31T20:39:07.948 DEBUG:boto:path=/pklxkd-58/obj00000000000000000005 2023-05-31T20:39:07.948 DEBUG:boto:auth_path=/pklxkd-58/obj00000000000000000005 2023-05-31T20:39:07.949 DEBUG:boto:path=/pklxkd-58/obj00000000000000000005?versionId=null 2023-05-31T20:39:07.949 DEBUG:boto:auth_path=/pklxkd-58/obj00000000000000000005?versionId=null 2023-05-31T20:39:07.949 DEBUG:boto:Method: GET 2023-05-31T20:39:07.950 DEBUG:boto:Path: /pklxkd-58/obj00000000000000000005?versionId=null 2023-05-31T20:39:07.950 DEBUG:boto:Data: 2023-05-31T20:39:07.951 DEBUG:boto:Headers: {} 2023-05-31T20:39:07.951 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:39:07.951 DEBUG:boto:Port: 8001 2023-05-31T20:39:07.952 DEBUG:boto:Params: {} 2023-05-31T20:39:07.952 DEBUG:boto:Token: None 2023-05-31T20:39:07.952 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:39:07 GMT /pklxkd-58/obj00000000000000000005?versionId=null 2023-05-31T20:39:07.953 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:crf07j7/XChovMEaWI+MvGU1tfI= 2023-05-31T20:39:07.953 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:39:07 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:crf07j7/XChovMEaWI+MvGU1tfI=', 'Content-Length': '0'} 2023-05-31T20:39:07.955 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:38:21 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000061c821af2d17f620-006477b06b-1c17-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:39:07 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:39:07.956 DEBUG:boto:path=/pklxkd-58/obj00000000000000000005 2023-05-31T20:39:07.956 DEBUG:boto:auth_path=/pklxkd-58/obj00000000000000000005 2023-05-31T20:39:07.957 DEBUG:boto:path=/pklxkd-58/obj00000000000000000005?versionId=null 2023-05-31T20:39:07.957 DEBUG:boto:auth_path=/pklxkd-58/obj00000000000000000005?versionId=null 2023-05-31T20:39:07.957 DEBUG:boto:Method: GET 2023-05-31T20:39:07.958 DEBUG:boto:Path: /pklxkd-58/obj00000000000000000005?versionId=null 2023-05-31T20:39:07.958 DEBUG:boto:Data: 2023-05-31T20:39:07.958 DEBUG:boto:Headers: {} 2023-05-31T20:39:07.959 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:39:07.959 DEBUG:boto:Port: 8000 2023-05-31T20:39:07.960 DEBUG:boto:Params: {} 2023-05-31T20:39:07.961 DEBUG:boto:Token: None 2023-05-31T20:39:07.962 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:39:07 GMT /pklxkd-58/obj00000000000000000005?versionId=null 2023-05-31T20:39:07.962 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:crf07j7/XChovMEaWI+MvGU1tfI= 2023-05-31T20:39:07.962 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:39:07 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:crf07j7/XChovMEaWI+MvGU1tfI=', 'Content-Length': '0'} 2023-05-31T20:39:07.964 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:38:21 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005771173125906919-006477b06b-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:39:07 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:39:07.965 DEBUG:boto:path=/pklxkd-58/ 2023-05-31T20:39:07.965 DEBUG:boto:auth_path=/pklxkd-58/ 2023-05-31T20:39:07.966 DEBUG:boto:path=/pklxkd-58/?prefix=obj00000000000000000005 2023-05-31T20:39:07.966 DEBUG:boto:auth_path=/pklxkd-58/?prefix=obj00000000000000000005 2023-05-31T20:39:07.966 DEBUG:boto:Method: GET 2023-05-31T20:39:07.967 DEBUG:boto:Path: /pklxkd-58/?prefix=obj00000000000000000005 2023-05-31T20:39:07.968 DEBUG:boto:Data: 2023-05-31T20:39:07.969 DEBUG:boto:Headers: {} 2023-05-31T20:39:07.970 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:39:07.971 DEBUG:boto:Port: 8001 2023-05-31T20:39:07.972 DEBUG:boto:Params: {} 2023-05-31T20:39:07.973 DEBUG:boto:Token: None 2023-05-31T20:39:07.974 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:39:07 GMT /pklxkd-58/ 2023-05-31T20:39:07.975 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:097vQnzeWddc3TithBo9whiVZBE= 2023-05-31T20:39:07.976 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:39:07 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:097vQnzeWddc3TithBo9whiVZBE=', 'Content-Length': '0'} 2023-05-31T20:39:07.979 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000607f07f2e5cb0ed1-006477b06b-1c17-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:39:07 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:39:07.980 DEBUG:boto:b'pklxkd-58obj000000000000000000051000falseobj000000000000000000052023-05-31T20:38:21.709Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:39:07.980 DEBUG:boto:path=/pklxkd-58/ 2023-05-31T20:39:07.981 DEBUG:boto:auth_path=/pklxkd-58/ 2023-05-31T20:39:07.981 DEBUG:boto:path=/pklxkd-58/?prefix=obj00000000000000000005 2023-05-31T20:39:07.982 DEBUG:boto:auth_path=/pklxkd-58/?prefix=obj00000000000000000005 2023-05-31T20:39:07.982 DEBUG:boto:Method: GET 2023-05-31T20:39:07.982 DEBUG:boto:Path: /pklxkd-58/?prefix=obj00000000000000000005 2023-05-31T20:39:07.983 DEBUG:boto:Data: 2023-05-31T20:39:07.983 DEBUG:boto:Headers: {} 2023-05-31T20:39:07.983 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:39:07.984 DEBUG:boto:Port: 8000 2023-05-31T20:39:07.984 DEBUG:boto:Params: {} 2023-05-31T20:39:07.985 DEBUG:boto:Token: None 2023-05-31T20:39:07.985 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:39:07 GMT /pklxkd-58/ 2023-05-31T20:39:07.986 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:097vQnzeWddc3TithBo9whiVZBE= 2023-05-31T20:39:07.986 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:39:07 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:097vQnzeWddc3TithBo9whiVZBE=', 'Content-Length': '0'} 2023-05-31T20:39:07.991 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000aadb0f587c2f7586-006477b06b-1630-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:39:07 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:39:08.034 DEBUG:boto:b'pklxkd-58obj000000000000000000051000falseobj000000000000000000052023-05-31T20:38:21.709Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:39:08.035 DEBUG:boto:path=/pklxkd-58/obj00000000000000000005 2023-05-31T20:39:08.035 DEBUG:boto:auth_path=/pklxkd-58/obj00000000000000000005 2023-05-31T20:39:08.036 DEBUG:boto:Method: HEAD 2023-05-31T20:39:08.037 DEBUG:boto:Path: /pklxkd-58/obj00000000000000000005 2023-05-31T20:39:08.037 DEBUG:boto:Data: 2023-05-31T20:39:08.037 DEBUG:boto:Headers: {} 2023-05-31T20:39:08.038 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:39:08.038 DEBUG:boto:Port: 8001 2023-05-31T20:39:08.038 DEBUG:boto:Params: {} 2023-05-31T20:39:08.039 DEBUG:boto:Token: None 2023-05-31T20:39:08.039 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:39:08 GMT /pklxkd-58/obj00000000000000000005 2023-05-31T20:39:08.040 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:88kVJ9LlYlE7s0dRCOIp/vgjIn4= 2023-05-31T20:39:08.040 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:39:08 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:88kVJ9LlYlE7s0dRCOIp/vgjIn4=', 'Content-Length': '0'} 2023-05-31T20:39:08.042 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:38:21 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000eb8f52cb6ecfd515-006477b06c-1c17-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:39:08 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:39:08.043 DEBUG:boto:path=/pklxkd-58/obj00000000000000000005 2023-05-31T20:39:08.044 DEBUG:boto:auth_path=/pklxkd-58/obj00000000000000000005 2023-05-31T20:39:08.044 DEBUG:boto:Method: HEAD 2023-05-31T20:39:08.044 DEBUG:boto:Path: /pklxkd-58/obj00000000000000000005 2023-05-31T20:39:08.045 DEBUG:boto:Data: 2023-05-31T20:39:08.045 DEBUG:boto:Headers: {} 2023-05-31T20:39:08.045 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:39:08.046 DEBUG:boto:Port: 8000 2023-05-31T20:39:08.046 DEBUG:boto:Params: {} 2023-05-31T20:39:08.046 DEBUG:boto:Token: None 2023-05-31T20:39:08.047 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:39:08 GMT /pklxkd-58/obj00000000000000000005 2023-05-31T20:39:08.047 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:88kVJ9LlYlE7s0dRCOIp/vgjIn4= 2023-05-31T20:39:08.048 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:39:08 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:88kVJ9LlYlE7s0dRCOIp/vgjIn4=', 'Content-Length': '0'} 2023-05-31T20:39:08.050 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:38:21 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a29136714cd8439a-006477b06c-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:39:08 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:39:08.051 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000005 2023-05-31T20:39:08.051 DEBUG:boto:path=/pklxkd-58/obj00000000000000000005 2023-05-31T20:39:08.051 DEBUG:boto:auth_path=/pklxkd-58/obj00000000000000000005 2023-05-31T20:39:08.052 DEBUG:boto:Method: GET 2023-05-31T20:39:08.052 DEBUG:boto:Path: /pklxkd-58/obj00000000000000000005 2023-05-31T20:39:08.053 DEBUG:boto:Data: 2023-05-31T20:39:08.053 DEBUG:boto:Headers: {} 2023-05-31T20:39:08.053 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:39:08.054 DEBUG:boto:Port: 8001 2023-05-31T20:39:08.054 DEBUG:boto:Params: {} 2023-05-31T20:39:08.054 DEBUG:boto:Token: None 2023-05-31T20:39:08.055 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:39:08 GMT /pklxkd-58/obj00000000000000000005 2023-05-31T20:39:08.055 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:VZmxdvm8m1yvNbsoom3w+39CBh0= 2023-05-31T20:39:08.058 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:39:08 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:VZmxdvm8m1yvNbsoom3w+39CBh0=', 'Content-Length': '0'} 2023-05-31T20:39:08.064 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:38:21 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000693a6eb9518e60a7-006477b06c-1c17-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:39:08 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:39:08.066 DEBUG:boto:path=/pklxkd-58/obj00000000000000000005 2023-05-31T20:39:08.067 DEBUG:boto:auth_path=/pklxkd-58/obj00000000000000000005 2023-05-31T20:39:08.068 DEBUG:boto:Method: GET 2023-05-31T20:39:08.068 DEBUG:boto:Path: /pklxkd-58/obj00000000000000000005 2023-05-31T20:39:08.069 DEBUG:boto:Data: 2023-05-31T20:39:08.069 DEBUG:boto:Headers: {} 2023-05-31T20:39:08.070 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:39:08.071 DEBUG:boto:Port: 8000 2023-05-31T20:39:08.071 DEBUG:boto:Params: {} 2023-05-31T20:39:08.072 DEBUG:boto:Token: None 2023-05-31T20:39:08.074 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:39:08 GMT /pklxkd-58/obj00000000000000000005 2023-05-31T20:39:08.074 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:VZmxdvm8m1yvNbsoom3w+39CBh0= 2023-05-31T20:39:08.075 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:39:08 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:VZmxdvm8m1yvNbsoom3w+39CBh0=', 'Content-Length': '0'} 2023-05-31T20:39:08.077 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:38:21 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e4ae5a9f52c94ba9-006477b06c-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:39:08 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:39:08.078 DEBUG:boto:path=/pklxkd-58/ 2023-05-31T20:39:08.078 DEBUG:boto:auth_path=/pklxkd-58/ 2023-05-31T20:39:08.079 DEBUG:boto:path=/pklxkd-58/?prefix=obj00000000000000000005 2023-05-31T20:39:08.079 DEBUG:boto:auth_path=/pklxkd-58/?prefix=obj00000000000000000005 2023-05-31T20:39:08.080 DEBUG:boto:Method: GET 2023-05-31T20:39:08.080 DEBUG:boto:Path: /pklxkd-58/?prefix=obj00000000000000000005 2023-05-31T20:39:08.080 DEBUG:boto:Data: 2023-05-31T20:39:08.081 DEBUG:boto:Headers: {} 2023-05-31T20:39:08.081 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:39:08.082 DEBUG:boto:Port: 8001 2023-05-31T20:39:08.082 DEBUG:boto:Params: {} 2023-05-31T20:39:08.082 DEBUG:boto:Token: None 2023-05-31T20:39:08.085 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:39:08 GMT /pklxkd-58/ 2023-05-31T20:39:08.085 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:ZA0HAdOWomTD+dvweZNsEimG+Ok= 2023-05-31T20:39:08.085 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:39:08 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:ZA0HAdOWomTD+dvweZNsEimG+Ok=', 'Content-Length': '0'} 2023-05-31T20:39:08.092 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c44fab5283b61733-006477b06c-1c17-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:39:08 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:39:08.092 DEBUG:boto:b'pklxkd-58obj000000000000000000051000falseobj000000000000000000052023-05-31T20:38:21.709Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:39:08.093 DEBUG:boto:path=/pklxkd-58/ 2023-05-31T20:39:08.094 DEBUG:boto:auth_path=/pklxkd-58/ 2023-05-31T20:39:08.094 DEBUG:boto:path=/pklxkd-58/?prefix=obj00000000000000000005 2023-05-31T20:39:08.094 DEBUG:boto:auth_path=/pklxkd-58/?prefix=obj00000000000000000005 2023-05-31T20:39:08.095 DEBUG:boto:Method: GET 2023-05-31T20:39:08.095 DEBUG:boto:Path: /pklxkd-58/?prefix=obj00000000000000000005 2023-05-31T20:39:08.095 DEBUG:boto:Data: 2023-05-31T20:39:08.096 DEBUG:boto:Headers: {} 2023-05-31T20:39:08.096 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:39:08.096 DEBUG:boto:Port: 8000 2023-05-31T20:39:08.097 DEBUG:boto:Params: {} 2023-05-31T20:39:08.097 DEBUG:boto:Token: None 2023-05-31T20:39:08.098 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:39:08 GMT /pklxkd-58/ 2023-05-31T20:39:08.098 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:ZA0HAdOWomTD+dvweZNsEimG+Ok= 2023-05-31T20:39:08.099 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:39:08 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:ZA0HAdOWomTD+dvweZNsEimG+Ok=', 'Content-Length': '0'} 2023-05-31T20:39:08.107 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c7ed19f31ed436cd-006477b06c-1630-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:39:08 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:39:08.108 DEBUG:boto:b'pklxkd-58obj000000000000000000051000falseobj000000000000000000052023-05-31T20:38:21.709Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:39:08.109 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000006 2023-05-31T20:39:08.109 DEBUG:boto:path=/pklxkd-58/obj00000000000000000006 2023-05-31T20:39:08.110 DEBUG:boto:auth_path=/pklxkd-58/obj00000000000000000006 2023-05-31T20:39:08.110 DEBUG:boto:path=/pklxkd-58/obj00000000000000000006?versionId=null 2023-05-31T20:39:08.110 DEBUG:boto:auth_path=/pklxkd-58/obj00000000000000000006?versionId=null 2023-05-31T20:39:08.111 DEBUG:boto:Method: GET 2023-05-31T20:39:08.111 DEBUG:boto:Path: /pklxkd-58/obj00000000000000000006?versionId=null 2023-05-31T20:39:08.111 DEBUG:boto:Data: 2023-05-31T20:39:08.112 DEBUG:boto:Headers: {} 2023-05-31T20:39:08.112 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:39:08.112 DEBUG:boto:Port: 8001 2023-05-31T20:39:08.113 DEBUG:boto:Params: {} 2023-05-31T20:39:08.113 DEBUG:boto:Token: None 2023-05-31T20:39:08.114 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:39:08 GMT /pklxkd-58/obj00000000000000000006?versionId=null 2023-05-31T20:39:08.114 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:TQnpGznrM4hQXQLECUS3TkEktAE= 2023-05-31T20:39:08.115 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:39:08 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:TQnpGznrM4hQXQLECUS3TkEktAE=', 'Content-Length': '0'} 2023-05-31T20:39:08.117 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:38:21 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000036d25caffe3d9667-006477b06c-1c17-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:39:08 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:39:08.164 DEBUG:boto:path=/pklxkd-58/obj00000000000000000006 2023-05-31T20:39:08.165 DEBUG:boto:auth_path=/pklxkd-58/obj00000000000000000006 2023-05-31T20:39:08.166 DEBUG:boto:path=/pklxkd-58/obj00000000000000000006?versionId=null 2023-05-31T20:39:08.167 DEBUG:boto:auth_path=/pklxkd-58/obj00000000000000000006?versionId=null 2023-05-31T20:39:08.167 DEBUG:boto:Method: GET 2023-05-31T20:39:08.168 DEBUG:boto:Path: /pklxkd-58/obj00000000000000000006?versionId=null 2023-05-31T20:39:08.169 DEBUG:boto:Data: 2023-05-31T20:39:08.169 DEBUG:boto:Headers: {} 2023-05-31T20:39:08.170 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:39:08.171 DEBUG:boto:Port: 8000 2023-05-31T20:39:08.172 DEBUG:boto:Params: {} 2023-05-31T20:39:08.172 DEBUG:boto:Token: None 2023-05-31T20:39:08.174 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:39:08 GMT /pklxkd-58/obj00000000000000000006?versionId=null 2023-05-31T20:39:08.175 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:TQnpGznrM4hQXQLECUS3TkEktAE= 2023-05-31T20:39:08.176 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:39:08 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:TQnpGznrM4hQXQLECUS3TkEktAE=', 'Content-Length': '0'} 2023-05-31T20:39:08.186 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:38:21 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002aaf0a5068e0dbd9-006477b06c-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:39:08 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:39:08.187 DEBUG:boto:path=/pklxkd-58/obj00000000000000000006 2023-05-31T20:39:08.187 DEBUG:boto:auth_path=/pklxkd-58/obj00000000000000000006 2023-05-31T20:39:08.188 DEBUG:boto:path=/pklxkd-58/obj00000000000000000006?versionId=null 2023-05-31T20:39:08.188 DEBUG:boto:auth_path=/pklxkd-58/obj00000000000000000006?versionId=null 2023-05-31T20:39:08.188 DEBUG:boto:Method: HEAD 2023-05-31T20:39:08.189 DEBUG:boto:Path: /pklxkd-58/obj00000000000000000006?versionId=null 2023-05-31T20:39:08.189 DEBUG:boto:Data: 2023-05-31T20:39:08.189 DEBUG:boto:Headers: {} 2023-05-31T20:39:08.190 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:39:08.190 DEBUG:boto:Port: 8001 2023-05-31T20:39:08.190 DEBUG:boto:Params: {} 2023-05-31T20:39:08.191 DEBUG:boto:Token: None 2023-05-31T20:39:08.191 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:39:08 GMT /pklxkd-58/obj00000000000000000006?versionId=null 2023-05-31T20:39:08.192 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:v7tkMtpxX3c2bMtf4dQNhAilE7Y= 2023-05-31T20:39:08.193 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:39:08 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:v7tkMtpxX3c2bMtf4dQNhAilE7Y=', 'Content-Length': '0'} 2023-05-31T20:39:08.195 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:38:21 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a00eb42c4a754a15-006477b06c-1c17-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:39:08 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:39:08.199 DEBUG:boto:path=/pklxkd-58/obj00000000000000000006 2023-05-31T20:39:08.200 DEBUG:boto:auth_path=/pklxkd-58/obj00000000000000000006 2023-05-31T20:39:08.200 DEBUG:boto:path=/pklxkd-58/obj00000000000000000006?versionId=null 2023-05-31T20:39:08.201 DEBUG:boto:auth_path=/pklxkd-58/obj00000000000000000006?versionId=null 2023-05-31T20:39:08.201 DEBUG:boto:Method: HEAD 2023-05-31T20:39:08.201 DEBUG:boto:Path: /pklxkd-58/obj00000000000000000006?versionId=null 2023-05-31T20:39:08.202 DEBUG:boto:Data: 2023-05-31T20:39:08.202 DEBUG:boto:Headers: {} 2023-05-31T20:39:08.202 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:39:08.203 DEBUG:boto:Port: 8000 2023-05-31T20:39:08.203 DEBUG:boto:Params: {} 2023-05-31T20:39:08.203 DEBUG:boto:Token: None 2023-05-31T20:39:08.204 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:39:08 GMT /pklxkd-58/obj00000000000000000006?versionId=null 2023-05-31T20:39:08.205 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:v7tkMtpxX3c2bMtf4dQNhAilE7Y= 2023-05-31T20:39:08.205 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:39:08 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:v7tkMtpxX3c2bMtf4dQNhAilE7Y=', 'Content-Length': '0'} 2023-05-31T20:39:08.207 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:38:21 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e869da7859684f37-006477b06c-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:39:08 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:39:08.208 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000006 2023-05-31T20:39:08.208 DEBUG:boto:path=/pklxkd-58/obj00000000000000000006 2023-05-31T20:39:08.209 DEBUG:boto:auth_path=/pklxkd-58/obj00000000000000000006 2023-05-31T20:39:08.209 DEBUG:boto:path=/pklxkd-58/obj00000000000000000006?versionId=null 2023-05-31T20:39:08.209 DEBUG:boto:auth_path=/pklxkd-58/obj00000000000000000006?versionId=null 2023-05-31T20:39:08.210 DEBUG:boto:Method: GET 2023-05-31T20:39:08.210 DEBUG:boto:Path: /pklxkd-58/obj00000000000000000006?versionId=null 2023-05-31T20:39:08.210 DEBUG:boto:Data: 2023-05-31T20:39:08.211 DEBUG:boto:Headers: {} 2023-05-31T20:39:08.211 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:39:08.211 DEBUG:boto:Port: 8001 2023-05-31T20:39:08.212 DEBUG:boto:Params: {} 2023-05-31T20:39:08.212 DEBUG:boto:Token: None 2023-05-31T20:39:08.213 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:39:08 GMT /pklxkd-58/obj00000000000000000006?versionId=null 2023-05-31T20:39:08.213 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:TQnpGznrM4hQXQLECUS3TkEktAE= 2023-05-31T20:39:08.214 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:39:08 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:TQnpGznrM4hQXQLECUS3TkEktAE=', 'Content-Length': '0'} 2023-05-31T20:39:08.216 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:38:21 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004a3604a197fd4de6-006477b06c-1c17-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:39:08 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:39:08.217 DEBUG:boto:path=/pklxkd-58/obj00000000000000000006 2023-05-31T20:39:08.217 DEBUG:boto:auth_path=/pklxkd-58/obj00000000000000000006 2023-05-31T20:39:08.218 DEBUG:boto:path=/pklxkd-58/obj00000000000000000006?versionId=null 2023-05-31T20:39:08.218 DEBUG:boto:auth_path=/pklxkd-58/obj00000000000000000006?versionId=null 2023-05-31T20:39:08.218 DEBUG:boto:Method: GET 2023-05-31T20:39:08.219 DEBUG:boto:Path: /pklxkd-58/obj00000000000000000006?versionId=null 2023-05-31T20:39:08.220 DEBUG:boto:Data: 2023-05-31T20:39:08.221 DEBUG:boto:Headers: {} 2023-05-31T20:39:08.221 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:39:08.221 DEBUG:boto:Port: 8000 2023-05-31T20:39:08.222 DEBUG:boto:Params: {} 2023-05-31T20:39:08.222 DEBUG:boto:Token: None 2023-05-31T20:39:08.223 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:39:08 GMT /pklxkd-58/obj00000000000000000006?versionId=null 2023-05-31T20:39:08.224 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:TQnpGznrM4hQXQLECUS3TkEktAE= 2023-05-31T20:39:08.224 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:39:08 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:TQnpGznrM4hQXQLECUS3TkEktAE=', 'Content-Length': '0'} 2023-05-31T20:39:08.226 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:38:21 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a3a0f21b286c81d5-006477b06c-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:39:08 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:39:08.268 DEBUG:boto:path=/pklxkd-58/ 2023-05-31T20:39:08.269 DEBUG:boto:auth_path=/pklxkd-58/ 2023-05-31T20:39:08.270 DEBUG:boto:path=/pklxkd-58/?prefix=obj00000000000000000006 2023-05-31T20:39:08.270 DEBUG:boto:auth_path=/pklxkd-58/?prefix=obj00000000000000000006 2023-05-31T20:39:08.270 DEBUG:boto:Method: GET 2023-05-31T20:39:08.271 DEBUG:boto:Path: /pklxkd-58/?prefix=obj00000000000000000006 2023-05-31T20:39:08.271 DEBUG:boto:Data: 2023-05-31T20:39:08.271 DEBUG:boto:Headers: {} 2023-05-31T20:39:08.272 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:39:08.272 DEBUG:boto:Port: 8001 2023-05-31T20:39:08.273 DEBUG:boto:Params: {} 2023-05-31T20:39:08.273 DEBUG:boto:Token: None 2023-05-31T20:39:08.273 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:39:08 GMT /pklxkd-58/ 2023-05-31T20:39:08.274 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:ZA0HAdOWomTD+dvweZNsEimG+Ok= 2023-05-31T20:39:08.274 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:39:08 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:ZA0HAdOWomTD+dvweZNsEimG+Ok=', 'Content-Length': '0'} 2023-05-31T20:39:08.277 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000097df146b0df1d43-006477b06c-1c17-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:39:08 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:39:08.278 DEBUG:boto:b'pklxkd-58obj000000000000000000061000falseobj000000000000000000062023-05-31T20:38:21.735Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:39:08.279 DEBUG:boto:path=/pklxkd-58/ 2023-05-31T20:39:08.279 DEBUG:boto:auth_path=/pklxkd-58/ 2023-05-31T20:39:08.279 DEBUG:boto:path=/pklxkd-58/?prefix=obj00000000000000000006 2023-05-31T20:39:08.280 DEBUG:boto:auth_path=/pklxkd-58/?prefix=obj00000000000000000006 2023-05-31T20:39:08.280 DEBUG:boto:Method: GET 2023-05-31T20:39:08.280 DEBUG:boto:Path: /pklxkd-58/?prefix=obj00000000000000000006 2023-05-31T20:39:08.281 DEBUG:boto:Data: 2023-05-31T20:39:08.281 DEBUG:boto:Headers: {} 2023-05-31T20:39:08.282 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:39:08.282 DEBUG:boto:Port: 8000 2023-05-31T20:39:08.282 DEBUG:boto:Params: {} 2023-05-31T20:39:08.283 DEBUG:boto:Token: None 2023-05-31T20:39:08.283 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:39:08 GMT /pklxkd-58/ 2023-05-31T20:39:08.284 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:ZA0HAdOWomTD+dvweZNsEimG+Ok= 2023-05-31T20:39:08.285 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:39:08 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:ZA0HAdOWomTD+dvweZNsEimG+Ok=', 'Content-Length': '0'} 2023-05-31T20:39:08.290 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000003992b7f5b88ccd76-006477b06c-1630-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:39:08 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:39:08.291 DEBUG:boto:b'pklxkd-58obj000000000000000000061000falseobj000000000000000000062023-05-31T20:38:21.735Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:39:08.292 DEBUG:boto:path=/pklxkd-58/obj00000000000000000006 2023-05-31T20:39:08.293 DEBUG:boto:auth_path=/pklxkd-58/obj00000000000000000006 2023-05-31T20:39:08.293 DEBUG:boto:Method: HEAD 2023-05-31T20:39:08.293 DEBUG:boto:Path: /pklxkd-58/obj00000000000000000006 2023-05-31T20:39:08.294 DEBUG:boto:Data: 2023-05-31T20:39:08.294 DEBUG:boto:Headers: {} 2023-05-31T20:39:08.294 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:39:08.295 DEBUG:boto:Port: 8001 2023-05-31T20:39:08.295 DEBUG:boto:Params: {} 2023-05-31T20:39:08.295 DEBUG:boto:Token: None 2023-05-31T20:39:08.296 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:39:08 GMT /pklxkd-58/obj00000000000000000006 2023-05-31T20:39:08.296 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:vGmvFbeospzZzB0llJurXUJPK6s= 2023-05-31T20:39:08.297 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:39:08 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:vGmvFbeospzZzB0llJurXUJPK6s=', 'Content-Length': '0'} 2023-05-31T20:39:08.299 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:38:21 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ad66666d0606218f-006477b06c-1c17-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:39:08 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:39:08.300 DEBUG:boto:path=/pklxkd-58/obj00000000000000000006 2023-05-31T20:39:08.300 DEBUG:boto:auth_path=/pklxkd-58/obj00000000000000000006 2023-05-31T20:39:08.301 DEBUG:boto:Method: HEAD 2023-05-31T20:39:08.302 DEBUG:boto:Path: /pklxkd-58/obj00000000000000000006 2023-05-31T20:39:08.302 DEBUG:boto:Data: 2023-05-31T20:39:08.302 DEBUG:boto:Headers: {} 2023-05-31T20:39:08.303 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:39:08.303 DEBUG:boto:Port: 8000 2023-05-31T20:39:08.303 DEBUG:boto:Params: {} 2023-05-31T20:39:08.304 DEBUG:boto:Token: None 2023-05-31T20:39:08.304 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:39:08 GMT /pklxkd-58/obj00000000000000000006 2023-05-31T20:39:08.305 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:vGmvFbeospzZzB0llJurXUJPK6s= 2023-05-31T20:39:08.305 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:39:08 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:vGmvFbeospzZzB0llJurXUJPK6s=', 'Content-Length': '0'} 2023-05-31T20:39:08.307 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:38:21 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f7ff5263d6facd6c-006477b06c-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:39:08 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:39:08.308 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000006 2023-05-31T20:39:08.309 DEBUG:boto:path=/pklxkd-58/obj00000000000000000006 2023-05-31T20:39:08.310 DEBUG:boto:auth_path=/pklxkd-58/obj00000000000000000006 2023-05-31T20:39:08.313 DEBUG:boto:Method: GET 2023-05-31T20:39:08.314 DEBUG:boto:Path: /pklxkd-58/obj00000000000000000006 2023-05-31T20:39:08.314 DEBUG:boto:Data: 2023-05-31T20:39:08.314 DEBUG:boto:Headers: {} 2023-05-31T20:39:08.315 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:39:08.315 DEBUG:boto:Port: 8001 2023-05-31T20:39:08.315 DEBUG:boto:Params: {} 2023-05-31T20:39:08.316 DEBUG:boto:Token: None 2023-05-31T20:39:08.317 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:39:08 GMT /pklxkd-58/obj00000000000000000006 2023-05-31T20:39:08.317 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:MLhoCiRnxpV0VV0jr2wSLscSCe8= 2023-05-31T20:39:08.317 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:39:08 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:MLhoCiRnxpV0VV0jr2wSLscSCe8=', 'Content-Length': '0'} 2023-05-31T20:39:08.320 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:38:21 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000081975b9d69a422d9-006477b06c-1c17-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:39:08 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:39:08.320 DEBUG:boto:path=/pklxkd-58/obj00000000000000000006 2023-05-31T20:39:08.321 DEBUG:boto:auth_path=/pklxkd-58/obj00000000000000000006 2023-05-31T20:39:08.321 DEBUG:boto:Method: GET 2023-05-31T20:39:08.321 DEBUG:boto:Path: /pklxkd-58/obj00000000000000000006 2023-05-31T20:39:08.322 DEBUG:boto:Data: 2023-05-31T20:39:08.322 DEBUG:boto:Headers: {} 2023-05-31T20:39:08.323 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:39:08.323 DEBUG:boto:Port: 8000 2023-05-31T20:39:08.323 DEBUG:boto:Params: {} 2023-05-31T20:39:08.324 DEBUG:boto:Token: None 2023-05-31T20:39:08.324 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:39:08 GMT /pklxkd-58/obj00000000000000000006 2023-05-31T20:39:08.325 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:MLhoCiRnxpV0VV0jr2wSLscSCe8= 2023-05-31T20:39:08.325 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:39:08 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:MLhoCiRnxpV0VV0jr2wSLscSCe8=', 'Content-Length': '0'} 2023-05-31T20:39:08.327 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:38:21 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b58b3187b497b0b8-006477b06c-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:39:08 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:39:08.328 DEBUG:boto:path=/pklxkd-58/ 2023-05-31T20:39:08.328 DEBUG:boto:auth_path=/pklxkd-58/ 2023-05-31T20:39:08.329 DEBUG:boto:path=/pklxkd-58/?prefix=obj00000000000000000006 2023-05-31T20:39:08.329 DEBUG:boto:auth_path=/pklxkd-58/?prefix=obj00000000000000000006 2023-05-31T20:39:08.329 DEBUG:boto:Method: GET 2023-05-31T20:39:08.330 DEBUG:boto:Path: /pklxkd-58/?prefix=obj00000000000000000006 2023-05-31T20:39:08.330 DEBUG:boto:Data: 2023-05-31T20:39:08.330 DEBUG:boto:Headers: {} 2023-05-31T20:39:08.331 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:39:08.331 DEBUG:boto:Port: 8001 2023-05-31T20:39:08.332 DEBUG:boto:Params: {} 2023-05-31T20:39:08.332 DEBUG:boto:Token: None 2023-05-31T20:39:08.332 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:39:08 GMT /pklxkd-58/ 2023-05-31T20:39:08.333 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:ZA0HAdOWomTD+dvweZNsEimG+Ok= 2023-05-31T20:39:08.333 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:39:08 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:ZA0HAdOWomTD+dvweZNsEimG+Ok=', 'Content-Length': '0'} 2023-05-31T20:39:08.337 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f9e8e97b6cd182ad-006477b06c-1c17-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:39:08 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:39:08.337 DEBUG:boto:b'pklxkd-58obj000000000000000000061000falseobj000000000000000000062023-05-31T20:38:21.735Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:39:08.338 DEBUG:boto:path=/pklxkd-58/ 2023-05-31T20:39:08.339 DEBUG:boto:auth_path=/pklxkd-58/ 2023-05-31T20:39:08.339 DEBUG:boto:path=/pklxkd-58/?prefix=obj00000000000000000006 2023-05-31T20:39:08.340 DEBUG:boto:auth_path=/pklxkd-58/?prefix=obj00000000000000000006 2023-05-31T20:39:08.340 DEBUG:boto:Method: GET 2023-05-31T20:39:08.340 DEBUG:boto:Path: /pklxkd-58/?prefix=obj00000000000000000006 2023-05-31T20:39:08.341 DEBUG:boto:Data: 2023-05-31T20:39:08.341 DEBUG:boto:Headers: {} 2023-05-31T20:39:08.342 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:39:08.342 DEBUG:boto:Port: 8000 2023-05-31T20:39:08.342 DEBUG:boto:Params: {} 2023-05-31T20:39:08.343 DEBUG:boto:Token: None 2023-05-31T20:39:08.343 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:39:08 GMT /pklxkd-58/ 2023-05-31T20:39:08.344 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:ZA0HAdOWomTD+dvweZNsEimG+Ok= 2023-05-31T20:39:08.344 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:39:08 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:ZA0HAdOWomTD+dvweZNsEimG+Ok=', 'Content-Length': '0'} 2023-05-31T20:39:08.349 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000129ac1bc91dfcab8-006477b06c-1630-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:39:08 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:39:08.350 DEBUG:boto:b'pklxkd-58obj000000000000000000061000falseobj000000000000000000062023-05-31T20:38:21.735Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:39:08.351 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000007 2023-05-31T20:39:08.351 DEBUG:boto:path=/pklxkd-58/obj00000000000000000007 2023-05-31T20:39:08.352 DEBUG:boto:auth_path=/pklxkd-58/obj00000000000000000007 2023-05-31T20:39:08.352 DEBUG:boto:path=/pklxkd-58/obj00000000000000000007?versionId=null 2023-05-31T20:39:08.353 DEBUG:boto:auth_path=/pklxkd-58/obj00000000000000000007?versionId=null 2023-05-31T20:39:08.353 DEBUG:boto:Method: GET 2023-05-31T20:39:08.353 DEBUG:boto:Path: /pklxkd-58/obj00000000000000000007?versionId=null 2023-05-31T20:39:08.354 DEBUG:boto:Data: 2023-05-31T20:39:08.354 DEBUG:boto:Headers: {} 2023-05-31T20:39:08.354 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:39:08.355 DEBUG:boto:Port: 8001 2023-05-31T20:39:08.355 DEBUG:boto:Params: {} 2023-05-31T20:39:08.356 DEBUG:boto:Token: None 2023-05-31T20:39:08.356 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:39:08 GMT /pklxkd-58/obj00000000000000000007?versionId=null 2023-05-31T20:39:08.357 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:Y/HEd2hCzo8Pp/M+RPAOi22VoNA= 2023-05-31T20:39:08.357 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:39:08 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:Y/HEd2hCzo8Pp/M+RPAOi22VoNA=', 'Content-Length': '0'} 2023-05-31T20:39:08.361 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:38:22 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006170b16ed56081a1-006477b06c-1c17-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:39:08 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:39:08.362 DEBUG:boto:path=/pklxkd-58/obj00000000000000000007 2023-05-31T20:39:08.363 DEBUG:boto:auth_path=/pklxkd-58/obj00000000000000000007 2023-05-31T20:39:08.364 DEBUG:boto:path=/pklxkd-58/obj00000000000000000007?versionId=null 2023-05-31T20:39:08.365 DEBUG:boto:auth_path=/pklxkd-58/obj00000000000000000007?versionId=null 2023-05-31T20:39:08.366 DEBUG:boto:Method: GET 2023-05-31T20:39:08.367 DEBUG:boto:Path: /pklxkd-58/obj00000000000000000007?versionId=null 2023-05-31T20:39:08.368 DEBUG:boto:Data: 2023-05-31T20:39:08.369 DEBUG:boto:Headers: {} 2023-05-31T20:39:08.370 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:39:08.371 DEBUG:boto:Port: 8000 2023-05-31T20:39:08.372 DEBUG:boto:Params: {} 2023-05-31T20:39:08.373 DEBUG:boto:Token: None 2023-05-31T20:39:08.374 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:39:08 GMT /pklxkd-58/obj00000000000000000007?versionId=null 2023-05-31T20:39:08.374 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:Y/HEd2hCzo8Pp/M+RPAOi22VoNA= 2023-05-31T20:39:08.375 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:39:08 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:Y/HEd2hCzo8Pp/M+RPAOi22VoNA=', 'Content-Length': '0'} 2023-05-31T20:39:08.379 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:38:22 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003df3bebd2310de1f-006477b06c-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:39:08 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:39:08.421 DEBUG:boto:path=/pklxkd-58/obj00000000000000000007 2023-05-31T20:39:08.422 DEBUG:boto:auth_path=/pklxkd-58/obj00000000000000000007 2023-05-31T20:39:08.422 DEBUG:boto:path=/pklxkd-58/obj00000000000000000007?versionId=null 2023-05-31T20:39:08.422 DEBUG:boto:auth_path=/pklxkd-58/obj00000000000000000007?versionId=null 2023-05-31T20:39:08.423 DEBUG:boto:Method: HEAD 2023-05-31T20:39:08.423 DEBUG:boto:Path: /pklxkd-58/obj00000000000000000007?versionId=null 2023-05-31T20:39:08.423 DEBUG:boto:Data: 2023-05-31T20:39:08.424 DEBUG:boto:Headers: {} 2023-05-31T20:39:08.424 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:39:08.425 DEBUG:boto:Port: 8001 2023-05-31T20:39:08.425 DEBUG:boto:Params: {} 2023-05-31T20:39:08.425 DEBUG:boto:Token: None 2023-05-31T20:39:08.427 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:39:08 GMT /pklxkd-58/obj00000000000000000007?versionId=null 2023-05-31T20:39:08.427 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:cfzxW6EpZ92ecEHwbIXrmZSTedo= 2023-05-31T20:39:08.427 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:39:08 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:cfzxW6EpZ92ecEHwbIXrmZSTedo=', 'Content-Length': '0'} 2023-05-31T20:39:08.429 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:38:22 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000155bb2bd98f8fad5-006477b06c-1c17-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:39:08 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:39:08.430 DEBUG:boto:path=/pklxkd-58/obj00000000000000000007 2023-05-31T20:39:08.430 DEBUG:boto:auth_path=/pklxkd-58/obj00000000000000000007 2023-05-31T20:39:08.431 DEBUG:boto:path=/pklxkd-58/obj00000000000000000007?versionId=null 2023-05-31T20:39:08.431 DEBUG:boto:auth_path=/pklxkd-58/obj00000000000000000007?versionId=null 2023-05-31T20:39:08.431 DEBUG:boto:Method: HEAD 2023-05-31T20:39:08.432 DEBUG:boto:Path: /pklxkd-58/obj00000000000000000007?versionId=null 2023-05-31T20:39:08.432 DEBUG:boto:Data: 2023-05-31T20:39:08.433 DEBUG:boto:Headers: {} 2023-05-31T20:39:08.433 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:39:08.434 DEBUG:boto:Port: 8000 2023-05-31T20:39:08.434 DEBUG:boto:Params: {} 2023-05-31T20:39:08.435 DEBUG:boto:Token: None 2023-05-31T20:39:08.435 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:39:08 GMT /pklxkd-58/obj00000000000000000007?versionId=null 2023-05-31T20:39:08.435 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:cfzxW6EpZ92ecEHwbIXrmZSTedo= 2023-05-31T20:39:08.436 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:39:08 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:cfzxW6EpZ92ecEHwbIXrmZSTedo=', 'Content-Length': '0'} 2023-05-31T20:39:08.438 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:38:22 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000adba408c86e14d80-006477b06c-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:39:08 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:39:08.438 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000007 2023-05-31T20:39:08.439 DEBUG:boto:path=/pklxkd-58/obj00000000000000000007 2023-05-31T20:39:08.439 DEBUG:boto:auth_path=/pklxkd-58/obj00000000000000000007 2023-05-31T20:39:08.440 DEBUG:boto:path=/pklxkd-58/obj00000000000000000007?versionId=null 2023-05-31T20:39:08.440 DEBUG:boto:auth_path=/pklxkd-58/obj00000000000000000007?versionId=null 2023-05-31T20:39:08.441 DEBUG:boto:Method: GET 2023-05-31T20:39:08.441 DEBUG:boto:Path: /pklxkd-58/obj00000000000000000007?versionId=null 2023-05-31T20:39:08.441 DEBUG:boto:Data: 2023-05-31T20:39:08.442 DEBUG:boto:Headers: {} 2023-05-31T20:39:08.442 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:39:08.442 DEBUG:boto:Port: 8001 2023-05-31T20:39:08.443 DEBUG:boto:Params: {} 2023-05-31T20:39:08.443 DEBUG:boto:Token: None 2023-05-31T20:39:08.444 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:39:08 GMT /pklxkd-58/obj00000000000000000007?versionId=null 2023-05-31T20:39:08.444 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:Y/HEd2hCzo8Pp/M+RPAOi22VoNA= 2023-05-31T20:39:08.445 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:39:08 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:Y/HEd2hCzo8Pp/M+RPAOi22VoNA=', 'Content-Length': '0'} 2023-05-31T20:39:08.446 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:38:22 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f76ec5f4b55567c2-006477b06c-1c17-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:39:08 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:39:08.447 DEBUG:boto:path=/pklxkd-58/obj00000000000000000007 2023-05-31T20:39:08.448 DEBUG:boto:auth_path=/pklxkd-58/obj00000000000000000007 2023-05-31T20:39:08.448 DEBUG:boto:path=/pklxkd-58/obj00000000000000000007?versionId=null 2023-05-31T20:39:08.448 DEBUG:boto:auth_path=/pklxkd-58/obj00000000000000000007?versionId=null 2023-05-31T20:39:08.449 DEBUG:boto:Method: GET 2023-05-31T20:39:08.449 DEBUG:boto:Path: /pklxkd-58/obj00000000000000000007?versionId=null 2023-05-31T20:39:08.450 DEBUG:boto:Data: 2023-05-31T20:39:08.450 DEBUG:boto:Headers: {} 2023-05-31T20:39:08.450 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:39:08.451 DEBUG:boto:Port: 8000 2023-05-31T20:39:08.451 DEBUG:boto:Params: {} 2023-05-31T20:39:08.452 DEBUG:boto:Token: None 2023-05-31T20:39:08.453 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:39:08 GMT /pklxkd-58/obj00000000000000000007?versionId=null 2023-05-31T20:39:08.453 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:Y/HEd2hCzo8Pp/M+RPAOi22VoNA= 2023-05-31T20:39:08.453 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:39:08 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:Y/HEd2hCzo8Pp/M+RPAOi22VoNA=', 'Content-Length': '0'} 2023-05-31T20:39:08.456 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:38:22 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008d0fde03e5349564-006477b06c-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:39:08 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:39:08.456 DEBUG:boto:path=/pklxkd-58/ 2023-05-31T20:39:08.457 DEBUG:boto:auth_path=/pklxkd-58/ 2023-05-31T20:39:08.457 DEBUG:boto:path=/pklxkd-58/?prefix=obj00000000000000000007 2023-05-31T20:39:08.458 DEBUG:boto:auth_path=/pklxkd-58/?prefix=obj00000000000000000007 2023-05-31T20:39:08.458 DEBUG:boto:Method: GET 2023-05-31T20:39:08.459 DEBUG:boto:Path: /pklxkd-58/?prefix=obj00000000000000000007 2023-05-31T20:39:08.460 DEBUG:boto:Data: 2023-05-31T20:39:08.460 DEBUG:boto:Headers: {} 2023-05-31T20:39:08.461 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:39:08.461 DEBUG:boto:Port: 8001 2023-05-31T20:39:08.461 DEBUG:boto:Params: {} 2023-05-31T20:39:08.462 DEBUG:boto:Token: None 2023-05-31T20:39:08.462 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:39:08 GMT /pklxkd-58/ 2023-05-31T20:39:08.463 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:ZA0HAdOWomTD+dvweZNsEimG+Ok= 2023-05-31T20:39:08.463 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:39:08 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:ZA0HAdOWomTD+dvweZNsEimG+Ok=', 'Content-Length': '0'} 2023-05-31T20:39:08.465 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000bbb459de8c21a38a-006477b06c-1c17-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:39:08 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:39:08.466 DEBUG:boto:b'pklxkd-58obj000000000000000000071000falseobj000000000000000000072023-05-31T20:38:22.793Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:39:08.467 DEBUG:boto:path=/pklxkd-58/ 2023-05-31T20:39:08.467 DEBUG:boto:auth_path=/pklxkd-58/ 2023-05-31T20:39:08.468 DEBUG:boto:path=/pklxkd-58/?prefix=obj00000000000000000007 2023-05-31T20:39:08.468 DEBUG:boto:auth_path=/pklxkd-58/?prefix=obj00000000000000000007 2023-05-31T20:39:08.469 DEBUG:boto:Method: GET 2023-05-31T20:39:08.469 DEBUG:boto:Path: /pklxkd-58/?prefix=obj00000000000000000007 2023-05-31T20:39:08.470 DEBUG:boto:Data: 2023-05-31T20:39:08.470 DEBUG:boto:Headers: {} 2023-05-31T20:39:08.470 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:39:08.471 DEBUG:boto:Port: 8000 2023-05-31T20:39:08.471 DEBUG:boto:Params: {} 2023-05-31T20:39:08.471 DEBUG:boto:Token: None 2023-05-31T20:39:08.472 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:39:08 GMT /pklxkd-58/ 2023-05-31T20:39:08.472 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:ZA0HAdOWomTD+dvweZNsEimG+Ok= 2023-05-31T20:39:08.473 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:39:08 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:ZA0HAdOWomTD+dvweZNsEimG+Ok=', 'Content-Length': '0'} 2023-05-31T20:39:08.476 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000000a0b9dd4591fa8e1-006477b06c-1630-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:39:08 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:39:08.477 DEBUG:boto:b'pklxkd-58obj000000000000000000071000falseobj000000000000000000072023-05-31T20:38:22.793Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:39:08.478 DEBUG:boto:path=/pklxkd-58/obj00000000000000000007 2023-05-31T20:39:08.478 DEBUG:boto:auth_path=/pklxkd-58/obj00000000000000000007 2023-05-31T20:39:08.478 DEBUG:boto:Method: HEAD 2023-05-31T20:39:08.479 DEBUG:boto:Path: /pklxkd-58/obj00000000000000000007 2023-05-31T20:39:08.479 DEBUG:boto:Data: 2023-05-31T20:39:08.479 DEBUG:boto:Headers: {} 2023-05-31T20:39:08.480 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:39:08.480 DEBUG:boto:Port: 8001 2023-05-31T20:39:08.480 DEBUG:boto:Params: {} 2023-05-31T20:39:08.481 DEBUG:boto:Token: None 2023-05-31T20:39:08.481 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:39:08 GMT /pklxkd-58/obj00000000000000000007 2023-05-31T20:39:08.482 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:RhfwhE6AzoB+61eyMYjhsEO9AME= 2023-05-31T20:39:08.482 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:39:08 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:RhfwhE6AzoB+61eyMYjhsEO9AME=', 'Content-Length': '0'} 2023-05-31T20:39:08.484 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:38:22 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000091ee25f48c457767-006477b06c-1c17-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:39:08 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:39:08.484 DEBUG:boto:path=/pklxkd-58/obj00000000000000000007 2023-05-31T20:39:08.485 DEBUG:boto:auth_path=/pklxkd-58/obj00000000000000000007 2023-05-31T20:39:08.485 DEBUG:boto:Method: HEAD 2023-05-31T20:39:08.486 DEBUG:boto:Path: /pklxkd-58/obj00000000000000000007 2023-05-31T20:39:08.486 DEBUG:boto:Data: 2023-05-31T20:39:08.486 DEBUG:boto:Headers: {} 2023-05-31T20:39:08.487 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:39:08.487 DEBUG:boto:Port: 8000 2023-05-31T20:39:08.487 DEBUG:boto:Params: {} 2023-05-31T20:39:08.488 DEBUG:boto:Token: None 2023-05-31T20:39:08.488 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:39:08 GMT /pklxkd-58/obj00000000000000000007 2023-05-31T20:39:08.489 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:RhfwhE6AzoB+61eyMYjhsEO9AME= 2023-05-31T20:39:08.489 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:39:08 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:RhfwhE6AzoB+61eyMYjhsEO9AME=', 'Content-Length': '0'} 2023-05-31T20:39:08.492 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:38:22 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000353e6d4a55f970f2-006477b06c-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:39:08 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:39:08.492 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000007 2023-05-31T20:39:08.493 DEBUG:boto:path=/pklxkd-58/obj00000000000000000007 2023-05-31T20:39:08.493 DEBUG:boto:auth_path=/pklxkd-58/obj00000000000000000007 2023-05-31T20:39:08.494 DEBUG:boto:Method: GET 2023-05-31T20:39:08.494 DEBUG:boto:Path: /pklxkd-58/obj00000000000000000007 2023-05-31T20:39:08.494 DEBUG:boto:Data: 2023-05-31T20:39:08.495 DEBUG:boto:Headers: {} 2023-05-31T20:39:08.495 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:39:08.495 DEBUG:boto:Port: 8001 2023-05-31T20:39:08.496 DEBUG:boto:Params: {} 2023-05-31T20:39:08.496 DEBUG:boto:Token: None 2023-05-31T20:39:08.497 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:39:08 GMT /pklxkd-58/obj00000000000000000007 2023-05-31T20:39:08.497 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:teAuR3wP7RdCgdvOOR166pO9CLo= 2023-05-31T20:39:08.498 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:39:08 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:teAuR3wP7RdCgdvOOR166pO9CLo=', 'Content-Length': '0'} 2023-05-31T20:39:08.499 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:38:22 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003290ce3fb76715aa-006477b06c-1c17-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:39:08 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:39:08.545 DEBUG:boto:path=/pklxkd-58/obj00000000000000000007 2023-05-31T20:39:08.548 DEBUG:boto:auth_path=/pklxkd-58/obj00000000000000000007 2023-05-31T20:39:08.548 DEBUG:boto:Method: GET 2023-05-31T20:39:08.549 DEBUG:boto:Path: /pklxkd-58/obj00000000000000000007 2023-05-31T20:39:08.549 DEBUG:boto:Data: 2023-05-31T20:39:08.550 DEBUG:boto:Headers: {} 2023-05-31T20:39:08.550 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:39:08.550 DEBUG:boto:Port: 8000 2023-05-31T20:39:08.551 DEBUG:boto:Params: {} 2023-05-31T20:39:08.551 DEBUG:boto:Token: None 2023-05-31T20:39:08.552 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:39:08 GMT /pklxkd-58/obj00000000000000000007 2023-05-31T20:39:08.552 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:teAuR3wP7RdCgdvOOR166pO9CLo= 2023-05-31T20:39:08.553 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:39:08 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:teAuR3wP7RdCgdvOOR166pO9CLo=', 'Content-Length': '0'} 2023-05-31T20:39:08.555 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:38:22 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a475d1b6d461b822-006477b06c-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:39:08 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:39:08.556 DEBUG:boto:path=/pklxkd-58/ 2023-05-31T20:39:08.556 DEBUG:boto:auth_path=/pklxkd-58/ 2023-05-31T20:39:08.557 DEBUG:boto:path=/pklxkd-58/?prefix=obj00000000000000000007 2023-05-31T20:39:08.559 DEBUG:boto:auth_path=/pklxkd-58/?prefix=obj00000000000000000007 2023-05-31T20:39:08.559 DEBUG:boto:Method: GET 2023-05-31T20:39:08.560 DEBUG:boto:Path: /pklxkd-58/?prefix=obj00000000000000000007 2023-05-31T20:39:08.560 DEBUG:boto:Data: 2023-05-31T20:39:08.561 DEBUG:boto:Headers: {} 2023-05-31T20:39:08.561 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:39:08.561 DEBUG:boto:Port: 8001 2023-05-31T20:39:08.562 DEBUG:boto:Params: {} 2023-05-31T20:39:08.562 DEBUG:boto:Token: None 2023-05-31T20:39:08.563 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:39:08 GMT /pklxkd-58/ 2023-05-31T20:39:08.563 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:ZA0HAdOWomTD+dvweZNsEimG+Ok= 2023-05-31T20:39:08.563 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:39:08 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:ZA0HAdOWomTD+dvweZNsEimG+Ok=', 'Content-Length': '0'} 2023-05-31T20:39:08.566 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a2ef9ac3357739af-006477b06c-1c17-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:39:08 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:39:08.568 DEBUG:boto:b'pklxkd-58obj000000000000000000071000falseobj000000000000000000072023-05-31T20:38:22.793Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:39:08.569 DEBUG:boto:path=/pklxkd-58/ 2023-05-31T20:39:08.570 DEBUG:boto:auth_path=/pklxkd-58/ 2023-05-31T20:39:08.571 DEBUG:boto:path=/pklxkd-58/?prefix=obj00000000000000000007 2023-05-31T20:39:08.572 DEBUG:boto:auth_path=/pklxkd-58/?prefix=obj00000000000000000007 2023-05-31T20:39:08.572 DEBUG:boto:Method: GET 2023-05-31T20:39:08.574 DEBUG:boto:Path: /pklxkd-58/?prefix=obj00000000000000000007 2023-05-31T20:39:08.574 DEBUG:boto:Data: 2023-05-31T20:39:08.575 DEBUG:boto:Headers: {} 2023-05-31T20:39:08.576 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:39:08.577 DEBUG:boto:Port: 8000 2023-05-31T20:39:08.577 DEBUG:boto:Params: {} 2023-05-31T20:39:08.578 DEBUG:boto:Token: None 2023-05-31T20:39:08.579 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:39:08 GMT /pklxkd-58/ 2023-05-31T20:39:08.580 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:ZA0HAdOWomTD+dvweZNsEimG+Ok= 2023-05-31T20:39:08.581 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:39:08 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:ZA0HAdOWomTD+dvweZNsEimG+Ok=', 'Content-Length': '0'} 2023-05-31T20:39:08.586 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000032c10c66f5270221-006477b06c-1630-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:39:08 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:39:08.587 DEBUG:boto:b'pklxkd-58obj000000000000000000071000falseobj000000000000000000072023-05-31T20:38:22.793Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:39:08.588 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000008 2023-05-31T20:39:08.588 DEBUG:boto:path=/pklxkd-58/obj00000000000000000008 2023-05-31T20:39:08.589 DEBUG:boto:auth_path=/pklxkd-58/obj00000000000000000008 2023-05-31T20:39:08.589 DEBUG:boto:path=/pklxkd-58/obj00000000000000000008?versionId=null 2023-05-31T20:39:08.590 DEBUG:boto:auth_path=/pklxkd-58/obj00000000000000000008?versionId=null 2023-05-31T20:39:08.590 DEBUG:boto:Method: GET 2023-05-31T20:39:08.590 DEBUG:boto:Path: /pklxkd-58/obj00000000000000000008?versionId=null 2023-05-31T20:39:08.591 DEBUG:boto:Data: 2023-05-31T20:39:08.591 DEBUG:boto:Headers: {} 2023-05-31T20:39:08.591 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:39:08.592 DEBUG:boto:Port: 8001 2023-05-31T20:39:08.592 DEBUG:boto:Params: {} 2023-05-31T20:39:08.593 DEBUG:boto:Token: None 2023-05-31T20:39:08.593 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:39:08 GMT /pklxkd-58/obj00000000000000000008?versionId=null 2023-05-31T20:39:08.594 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:MEHRo6NkwC5QSZayMl9r2tKcSZM= 2023-05-31T20:39:08.595 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:39:08 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:MEHRo6NkwC5QSZayMl9r2tKcSZM=', 'Content-Length': '0'} 2023-05-31T20:39:08.597 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:38:20 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000067f9c585de481bfb-006477b06c-1c17-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:39:08 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:39:08.598 DEBUG:boto:path=/pklxkd-58/obj00000000000000000008 2023-05-31T20:39:08.599 DEBUG:boto:auth_path=/pklxkd-58/obj00000000000000000008 2023-05-31T20:39:08.599 DEBUG:boto:path=/pklxkd-58/obj00000000000000000008?versionId=null 2023-05-31T20:39:08.599 DEBUG:boto:auth_path=/pklxkd-58/obj00000000000000000008?versionId=null 2023-05-31T20:39:08.600 DEBUG:boto:Method: GET 2023-05-31T20:39:08.600 DEBUG:boto:Path: /pklxkd-58/obj00000000000000000008?versionId=null 2023-05-31T20:39:08.601 DEBUG:boto:Data: 2023-05-31T20:39:08.601 DEBUG:boto:Headers: {} 2023-05-31T20:39:08.601 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:39:08.602 DEBUG:boto:Port: 8000 2023-05-31T20:39:08.602 DEBUG:boto:Params: {} 2023-05-31T20:39:08.603 DEBUG:boto:Token: None 2023-05-31T20:39:08.603 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:39:08 GMT /pklxkd-58/obj00000000000000000008?versionId=null 2023-05-31T20:39:08.604 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:MEHRo6NkwC5QSZayMl9r2tKcSZM= 2023-05-31T20:39:08.604 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:39:08 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:MEHRo6NkwC5QSZayMl9r2tKcSZM=', 'Content-Length': '0'} 2023-05-31T20:39:08.606 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:38:20 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007cb980e0fa942c37-006477b06c-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:39:08 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:39:08.653 DEBUG:boto:path=/pklxkd-58/obj00000000000000000008 2023-05-31T20:39:08.653 DEBUG:boto:auth_path=/pklxkd-58/obj00000000000000000008 2023-05-31T20:39:08.654 DEBUG:boto:path=/pklxkd-58/obj00000000000000000008?versionId=null 2023-05-31T20:39:08.655 DEBUG:boto:auth_path=/pklxkd-58/obj00000000000000000008?versionId=null 2023-05-31T20:39:08.655 DEBUG:boto:Method: HEAD 2023-05-31T20:39:08.656 DEBUG:boto:Path: /pklxkd-58/obj00000000000000000008?versionId=null 2023-05-31T20:39:08.657 DEBUG:boto:Data: 2023-05-31T20:39:08.657 DEBUG:boto:Headers: {} 2023-05-31T20:39:08.658 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:39:08.658 DEBUG:boto:Port: 8001 2023-05-31T20:39:08.658 DEBUG:boto:Params: {} 2023-05-31T20:39:08.659 DEBUG:boto:Token: None 2023-05-31T20:39:08.659 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:39:08 GMT /pklxkd-58/obj00000000000000000008?versionId=null 2023-05-31T20:39:08.660 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:/qHUnCPKpR5/4lJJVPb0e0/Opq4= 2023-05-31T20:39:08.660 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:39:08 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:/qHUnCPKpR5/4lJJVPb0e0/Opq4=', 'Content-Length': '0'} 2023-05-31T20:39:08.664 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:38:20 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000bf5811f3e0a7ddfd-006477b06c-1c17-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:39:08 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:39:08.665 DEBUG:boto:path=/pklxkd-58/obj00000000000000000008 2023-05-31T20:39:08.666 DEBUG:boto:auth_path=/pklxkd-58/obj00000000000000000008 2023-05-31T20:39:08.666 DEBUG:boto:path=/pklxkd-58/obj00000000000000000008?versionId=null 2023-05-31T20:39:08.666 DEBUG:boto:auth_path=/pklxkd-58/obj00000000000000000008?versionId=null 2023-05-31T20:39:08.667 DEBUG:boto:Method: HEAD 2023-05-31T20:39:08.667 DEBUG:boto:Path: /pklxkd-58/obj00000000000000000008?versionId=null 2023-05-31T20:39:08.667 DEBUG:boto:Data: 2023-05-31T20:39:08.668 DEBUG:boto:Headers: {} 2023-05-31T20:39:08.668 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:39:08.669 DEBUG:boto:Port: 8000 2023-05-31T20:39:08.669 DEBUG:boto:Params: {} 2023-05-31T20:39:08.670 DEBUG:boto:Token: None 2023-05-31T20:39:08.670 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:39:08 GMT /pklxkd-58/obj00000000000000000008?versionId=null 2023-05-31T20:39:08.671 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:/qHUnCPKpR5/4lJJVPb0e0/Opq4= 2023-05-31T20:39:08.671 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:39:08 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:/qHUnCPKpR5/4lJJVPb0e0/Opq4=', 'Content-Length': '0'} 2023-05-31T20:39:08.673 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:38:20 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d9dc158998a747d9-006477b06c-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:39:08 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:39:08.673 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000008 2023-05-31T20:39:08.674 DEBUG:boto:path=/pklxkd-58/obj00000000000000000008 2023-05-31T20:39:08.674 DEBUG:boto:auth_path=/pklxkd-58/obj00000000000000000008 2023-05-31T20:39:08.675 DEBUG:boto:path=/pklxkd-58/obj00000000000000000008?versionId=null 2023-05-31T20:39:08.675 DEBUG:boto:auth_path=/pklxkd-58/obj00000000000000000008?versionId=null 2023-05-31T20:39:08.675 DEBUG:boto:Method: GET 2023-05-31T20:39:08.676 DEBUG:boto:Path: /pklxkd-58/obj00000000000000000008?versionId=null 2023-05-31T20:39:08.676 DEBUG:boto:Data: 2023-05-31T20:39:08.677 DEBUG:boto:Headers: {} 2023-05-31T20:39:08.677 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:39:08.677 DEBUG:boto:Port: 8001 2023-05-31T20:39:08.678 DEBUG:boto:Params: {} 2023-05-31T20:39:08.678 DEBUG:boto:Token: None 2023-05-31T20:39:08.678 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:39:08 GMT /pklxkd-58/obj00000000000000000008?versionId=null 2023-05-31T20:39:08.679 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:MEHRo6NkwC5QSZayMl9r2tKcSZM= 2023-05-31T20:39:08.679 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:39:08 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:MEHRo6NkwC5QSZayMl9r2tKcSZM=', 'Content-Length': '0'} 2023-05-31T20:39:08.681 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:38:20 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006c5b4b7a8a05384c-006477b06c-1c17-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:39:08 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:39:08.682 DEBUG:boto:path=/pklxkd-58/obj00000000000000000008 2023-05-31T20:39:08.682 DEBUG:boto:auth_path=/pklxkd-58/obj00000000000000000008 2023-05-31T20:39:08.683 DEBUG:boto:path=/pklxkd-58/obj00000000000000000008?versionId=null 2023-05-31T20:39:08.683 DEBUG:boto:auth_path=/pklxkd-58/obj00000000000000000008?versionId=null 2023-05-31T20:39:08.683 DEBUG:boto:Method: GET 2023-05-31T20:39:08.684 DEBUG:boto:Path: /pklxkd-58/obj00000000000000000008?versionId=null 2023-05-31T20:39:08.684 DEBUG:boto:Data: 2023-05-31T20:39:08.685 DEBUG:boto:Headers: {} 2023-05-31T20:39:08.685 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:39:08.685 DEBUG:boto:Port: 8000 2023-05-31T20:39:08.686 DEBUG:boto:Params: {} 2023-05-31T20:39:08.686 DEBUG:boto:Token: None 2023-05-31T20:39:08.686 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:39:08 GMT /pklxkd-58/obj00000000000000000008?versionId=null 2023-05-31T20:39:08.687 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:MEHRo6NkwC5QSZayMl9r2tKcSZM= 2023-05-31T20:39:08.687 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:39:08 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:MEHRo6NkwC5QSZayMl9r2tKcSZM=', 'Content-Length': '0'} 2023-05-31T20:39:08.689 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:38:20 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003e70edd4d65981ea-006477b06c-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:39:08 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:39:08.690 DEBUG:boto:path=/pklxkd-58/ 2023-05-31T20:39:08.690 DEBUG:boto:auth_path=/pklxkd-58/ 2023-05-31T20:39:08.691 DEBUG:boto:path=/pklxkd-58/?prefix=obj00000000000000000008 2023-05-31T20:39:08.691 DEBUG:boto:auth_path=/pklxkd-58/?prefix=obj00000000000000000008 2023-05-31T20:39:08.691 DEBUG:boto:Method: GET 2023-05-31T20:39:08.692 DEBUG:boto:Path: /pklxkd-58/?prefix=obj00000000000000000008 2023-05-31T20:39:08.692 DEBUG:boto:Data: 2023-05-31T20:39:08.693 DEBUG:boto:Headers: {} 2023-05-31T20:39:08.693 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:39:08.693 DEBUG:boto:Port: 8001 2023-05-31T20:39:08.694 DEBUG:boto:Params: {} 2023-05-31T20:39:08.694 DEBUG:boto:Token: None 2023-05-31T20:39:08.695 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:39:08 GMT /pklxkd-58/ 2023-05-31T20:39:08.695 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:ZA0HAdOWomTD+dvweZNsEimG+Ok= 2023-05-31T20:39:08.696 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:39:08 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:ZA0HAdOWomTD+dvweZNsEimG+Ok=', 'Content-Length': '0'} 2023-05-31T20:39:08.702 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000089e8f5d113235a6a-006477b06c-1c17-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:39:08 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:39:08.702 DEBUG:boto:b'pklxkd-58obj000000000000000000081000falseobj000000000000000000082023-05-31T20:38:20.093Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:39:08.703 DEBUG:boto:path=/pklxkd-58/ 2023-05-31T20:39:08.704 DEBUG:boto:auth_path=/pklxkd-58/ 2023-05-31T20:39:08.704 DEBUG:boto:path=/pklxkd-58/?prefix=obj00000000000000000008 2023-05-31T20:39:08.704 DEBUG:boto:auth_path=/pklxkd-58/?prefix=obj00000000000000000008 2023-05-31T20:39:08.705 DEBUG:boto:Method: GET 2023-05-31T20:39:08.705 DEBUG:boto:Path: /pklxkd-58/?prefix=obj00000000000000000008 2023-05-31T20:39:08.705 DEBUG:boto:Data: 2023-05-31T20:39:08.706 DEBUG:boto:Headers: {} 2023-05-31T20:39:08.706 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:39:08.707 DEBUG:boto:Port: 8000 2023-05-31T20:39:08.707 DEBUG:boto:Params: {} 2023-05-31T20:39:08.707 DEBUG:boto:Token: None 2023-05-31T20:39:08.708 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:39:08 GMT /pklxkd-58/ 2023-05-31T20:39:08.708 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:ZA0HAdOWomTD+dvweZNsEimG+Ok= 2023-05-31T20:39:08.709 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:39:08 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:ZA0HAdOWomTD+dvweZNsEimG+Ok=', 'Content-Length': '0'} 2023-05-31T20:39:08.713 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000305790c6486621c3-006477b06c-1630-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:39:08 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:39:08.756 DEBUG:boto:b'pklxkd-58obj000000000000000000081000falseobj000000000000000000082023-05-31T20:38:20.093Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:39:08.758 DEBUG:boto:path=/pklxkd-58/obj00000000000000000008 2023-05-31T20:39:08.758 DEBUG:boto:auth_path=/pklxkd-58/obj00000000000000000008 2023-05-31T20:39:08.759 DEBUG:boto:Method: HEAD 2023-05-31T20:39:08.759 DEBUG:boto:Path: /pklxkd-58/obj00000000000000000008 2023-05-31T20:39:08.760 DEBUG:boto:Data: 2023-05-31T20:39:08.760 DEBUG:boto:Headers: {} 2023-05-31T20:39:08.760 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:39:08.761 DEBUG:boto:Port: 8001 2023-05-31T20:39:08.761 DEBUG:boto:Params: {} 2023-05-31T20:39:08.762 DEBUG:boto:Token: None 2023-05-31T20:39:08.762 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:39:08 GMT /pklxkd-58/obj00000000000000000008 2023-05-31T20:39:08.763 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:aStBpck4u/e1Hs4VckdxpMCtt7M= 2023-05-31T20:39:08.763 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:39:08 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:aStBpck4u/e1Hs4VckdxpMCtt7M=', 'Content-Length': '0'} 2023-05-31T20:39:08.765 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:38:20 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d62141b96a425840-006477b06c-1c17-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:39:08 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:39:08.766 DEBUG:boto:path=/pklxkd-58/obj00000000000000000008 2023-05-31T20:39:08.767 DEBUG:boto:auth_path=/pklxkd-58/obj00000000000000000008 2023-05-31T20:39:08.767 DEBUG:boto:Method: HEAD 2023-05-31T20:39:08.767 DEBUG:boto:Path: /pklxkd-58/obj00000000000000000008 2023-05-31T20:39:08.768 DEBUG:boto:Data: 2023-05-31T20:39:08.768 DEBUG:boto:Headers: {} 2023-05-31T20:39:08.768 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:39:08.769 DEBUG:boto:Port: 8000 2023-05-31T20:39:08.769 DEBUG:boto:Params: {} 2023-05-31T20:39:08.770 DEBUG:boto:Token: None 2023-05-31T20:39:08.770 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:39:08 GMT /pklxkd-58/obj00000000000000000008 2023-05-31T20:39:08.771 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:aStBpck4u/e1Hs4VckdxpMCtt7M= 2023-05-31T20:39:08.771 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:39:08 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:aStBpck4u/e1Hs4VckdxpMCtt7M=', 'Content-Length': '0'} 2023-05-31T20:39:08.773 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:38:20 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000bdc90adfbcb0949c-006477b06c-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:39:08 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:39:08.773 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000008 2023-05-31T20:39:08.774 DEBUG:boto:path=/pklxkd-58/obj00000000000000000008 2023-05-31T20:39:08.774 DEBUG:boto:auth_path=/pklxkd-58/obj00000000000000000008 2023-05-31T20:39:08.775 DEBUG:boto:Method: GET 2023-05-31T20:39:08.775 DEBUG:boto:Path: /pklxkd-58/obj00000000000000000008 2023-05-31T20:39:08.775 DEBUG:boto:Data: 2023-05-31T20:39:08.776 DEBUG:boto:Headers: {} 2023-05-31T20:39:08.776 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:39:08.777 DEBUG:boto:Port: 8001 2023-05-31T20:39:08.777 DEBUG:boto:Params: {} 2023-05-31T20:39:08.777 DEBUG:boto:Token: None 2023-05-31T20:39:08.778 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:39:08 GMT /pklxkd-58/obj00000000000000000008 2023-05-31T20:39:08.778 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:THarYTgczVWq6bRkKDFUO59EfQw= 2023-05-31T20:39:08.779 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:39:08 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:THarYTgczVWq6bRkKDFUO59EfQw=', 'Content-Length': '0'} 2023-05-31T20:39:08.781 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:38:20 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009ed3b0963c907d76-006477b06c-1c17-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:39:08 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:39:08.782 DEBUG:boto:path=/pklxkd-58/obj00000000000000000008 2023-05-31T20:39:08.782 DEBUG:boto:auth_path=/pklxkd-58/obj00000000000000000008 2023-05-31T20:39:08.783 DEBUG:boto:Method: GET 2023-05-31T20:39:08.783 DEBUG:boto:Path: /pklxkd-58/obj00000000000000000008 2023-05-31T20:39:08.783 DEBUG:boto:Data: 2023-05-31T20:39:08.784 DEBUG:boto:Headers: {} 2023-05-31T20:39:08.784 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:39:08.785 DEBUG:boto:Port: 8000 2023-05-31T20:39:08.785 DEBUG:boto:Params: {} 2023-05-31T20:39:08.785 DEBUG:boto:Token: None 2023-05-31T20:39:08.786 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:39:08 GMT /pklxkd-58/obj00000000000000000008 2023-05-31T20:39:08.786 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:THarYTgczVWq6bRkKDFUO59EfQw= 2023-05-31T20:39:08.787 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:39:08 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:THarYTgczVWq6bRkKDFUO59EfQw=', 'Content-Length': '0'} 2023-05-31T20:39:08.789 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:38:20 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000bcfe60b24d061a25-006477b06c-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:39:08 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:39:08.790 DEBUG:boto:path=/pklxkd-58/ 2023-05-31T20:39:08.790 DEBUG:boto:auth_path=/pklxkd-58/ 2023-05-31T20:39:08.791 DEBUG:boto:path=/pklxkd-58/?prefix=obj00000000000000000008 2023-05-31T20:39:08.791 DEBUG:boto:auth_path=/pklxkd-58/?prefix=obj00000000000000000008 2023-05-31T20:39:08.792 DEBUG:boto:Method: GET 2023-05-31T20:39:08.792 DEBUG:boto:Path: /pklxkd-58/?prefix=obj00000000000000000008 2023-05-31T20:39:08.792 DEBUG:boto:Data: 2023-05-31T20:39:08.793 DEBUG:boto:Headers: {} 2023-05-31T20:39:08.793 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:39:08.794 DEBUG:boto:Port: 8001 2023-05-31T20:39:08.794 DEBUG:boto:Params: {} 2023-05-31T20:39:08.795 DEBUG:boto:Token: None 2023-05-31T20:39:08.795 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:39:08 GMT /pklxkd-58/ 2023-05-31T20:39:08.796 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:ZA0HAdOWomTD+dvweZNsEimG+Ok= 2023-05-31T20:39:08.796 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:39:08 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:ZA0HAdOWomTD+dvweZNsEimG+Ok=', 'Content-Length': '0'} 2023-05-31T20:39:08.799 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000de843815faeeed17-006477b06c-1c17-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:39:08 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:39:08.844 DEBUG:boto:b'pklxkd-58obj000000000000000000081000falseobj000000000000000000082023-05-31T20:38:20.093Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:39:08.845 DEBUG:boto:path=/pklxkd-58/ 2023-05-31T20:39:08.846 DEBUG:boto:auth_path=/pklxkd-58/ 2023-05-31T20:39:08.846 DEBUG:boto:path=/pklxkd-58/?prefix=obj00000000000000000008 2023-05-31T20:39:08.846 DEBUG:boto:auth_path=/pklxkd-58/?prefix=obj00000000000000000008 2023-05-31T20:39:08.847 DEBUG:boto:Method: GET 2023-05-31T20:39:08.847 DEBUG:boto:Path: /pklxkd-58/?prefix=obj00000000000000000008 2023-05-31T20:39:08.848 DEBUG:boto:Data: 2023-05-31T20:39:08.848 DEBUG:boto:Headers: {} 2023-05-31T20:39:08.849 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:39:08.849 DEBUG:boto:Port: 8000 2023-05-31T20:39:08.850 DEBUG:boto:Params: {} 2023-05-31T20:39:08.850 DEBUG:boto:Token: None 2023-05-31T20:39:08.850 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:39:08 GMT /pklxkd-58/ 2023-05-31T20:39:08.851 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:ZA0HAdOWomTD+dvweZNsEimG+Ok= 2023-05-31T20:39:08.852 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:39:08 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:ZA0HAdOWomTD+dvweZNsEimG+Ok=', 'Content-Length': '0'} 2023-05-31T20:39:08.856 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000000def8ad59b49b228-006477b06c-1630-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:39:08 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:39:08.856 DEBUG:boto:b'pklxkd-58obj000000000000000000081000falseobj000000000000000000082023-05-31T20:38:20.093Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:39:08.857 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000009 2023-05-31T20:39:08.857 DEBUG:boto:path=/pklxkd-58/obj00000000000000000009 2023-05-31T20:39:08.858 DEBUG:boto:auth_path=/pklxkd-58/obj00000000000000000009 2023-05-31T20:39:08.858 DEBUG:boto:path=/pklxkd-58/obj00000000000000000009?versionId=null 2023-05-31T20:39:08.859 DEBUG:boto:auth_path=/pklxkd-58/obj00000000000000000009?versionId=null 2023-05-31T20:39:08.859 DEBUG:boto:Method: GET 2023-05-31T20:39:08.859 DEBUG:boto:Path: /pklxkd-58/obj00000000000000000009?versionId=null 2023-05-31T20:39:08.860 DEBUG:boto:Data: 2023-05-31T20:39:08.860 DEBUG:boto:Headers: {} 2023-05-31T20:39:08.861 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:39:08.861 DEBUG:boto:Port: 8001 2023-05-31T20:39:08.861 DEBUG:boto:Params: {} 2023-05-31T20:39:08.862 DEBUG:boto:Token: None 2023-05-31T20:39:08.862 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:39:08 GMT /pklxkd-58/obj00000000000000000009?versionId=null 2023-05-31T20:39:08.863 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:iZ8nV2Ju4mRIPEh0nSolO6z40SE= 2023-05-31T20:39:08.863 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:39:08 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:iZ8nV2Ju4mRIPEh0nSolO6z40SE=', 'Content-Length': '0'} 2023-05-31T20:39:08.866 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:38:21 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008706bb5779d243d6-006477b06c-1c17-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:39:08 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:39:08.866 DEBUG:boto:path=/pklxkd-58/obj00000000000000000009 2023-05-31T20:39:08.867 DEBUG:boto:auth_path=/pklxkd-58/obj00000000000000000009 2023-05-31T20:39:08.867 DEBUG:boto:path=/pklxkd-58/obj00000000000000000009?versionId=null 2023-05-31T20:39:08.867 DEBUG:boto:auth_path=/pklxkd-58/obj00000000000000000009?versionId=null 2023-05-31T20:39:08.868 DEBUG:boto:Method: GET 2023-05-31T20:39:08.868 DEBUG:boto:Path: /pklxkd-58/obj00000000000000000009?versionId=null 2023-05-31T20:39:08.869 DEBUG:boto:Data: 2023-05-31T20:39:08.869 DEBUG:boto:Headers: {} 2023-05-31T20:39:08.870 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:39:08.870 DEBUG:boto:Port: 8000 2023-05-31T20:39:08.870 DEBUG:boto:Params: {} 2023-05-31T20:39:08.871 DEBUG:boto:Token: None 2023-05-31T20:39:08.871 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:39:08 GMT /pklxkd-58/obj00000000000000000009?versionId=null 2023-05-31T20:39:08.872 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:iZ8nV2Ju4mRIPEh0nSolO6z40SE= 2023-05-31T20:39:08.872 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:39:08 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:iZ8nV2Ju4mRIPEh0nSolO6z40SE=', 'Content-Length': '0'} 2023-05-31T20:39:08.874 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:38:21 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000dbff82f7d7e435d6-006477b06c-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:39:08 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:39:08.875 DEBUG:boto:path=/pklxkd-58/obj00000000000000000009 2023-05-31T20:39:08.875 DEBUG:boto:auth_path=/pklxkd-58/obj00000000000000000009 2023-05-31T20:39:08.876 DEBUG:boto:path=/pklxkd-58/obj00000000000000000009?versionId=null 2023-05-31T20:39:08.876 DEBUG:boto:auth_path=/pklxkd-58/obj00000000000000000009?versionId=null 2023-05-31T20:39:08.876 DEBUG:boto:Method: HEAD 2023-05-31T20:39:08.877 DEBUG:boto:Path: /pklxkd-58/obj00000000000000000009?versionId=null 2023-05-31T20:39:08.877 DEBUG:boto:Data: 2023-05-31T20:39:08.877 DEBUG:boto:Headers: {} 2023-05-31T20:39:08.878 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:39:08.878 DEBUG:boto:Port: 8001 2023-05-31T20:39:08.878 DEBUG:boto:Params: {} 2023-05-31T20:39:08.879 DEBUG:boto:Token: None 2023-05-31T20:39:08.879 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:39:08 GMT /pklxkd-58/obj00000000000000000009?versionId=null 2023-05-31T20:39:08.880 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:en+2nFEAv7XeiQRMz9S+lRMWVb0= 2023-05-31T20:39:08.880 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:39:08 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:en+2nFEAv7XeiQRMz9S+lRMWVb0=', 'Content-Length': '0'} 2023-05-31T20:39:08.882 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:38:21 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c2de119277326438-006477b06c-1c17-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:39:08 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:39:08.883 DEBUG:boto:path=/pklxkd-58/obj00000000000000000009 2023-05-31T20:39:08.883 DEBUG:boto:auth_path=/pklxkd-58/obj00000000000000000009 2023-05-31T20:39:08.883 DEBUG:boto:path=/pklxkd-58/obj00000000000000000009?versionId=null 2023-05-31T20:39:08.884 DEBUG:boto:auth_path=/pklxkd-58/obj00000000000000000009?versionId=null 2023-05-31T20:39:08.884 DEBUG:boto:Method: HEAD 2023-05-31T20:39:08.885 DEBUG:boto:Path: /pklxkd-58/obj00000000000000000009?versionId=null 2023-05-31T20:39:08.885 DEBUG:boto:Data: 2023-05-31T20:39:08.885 DEBUG:boto:Headers: {} 2023-05-31T20:39:08.886 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:39:08.886 DEBUG:boto:Port: 8000 2023-05-31T20:39:08.886 DEBUG:boto:Params: {} 2023-05-31T20:39:08.887 DEBUG:boto:Token: None 2023-05-31T20:39:08.887 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:39:08 GMT /pklxkd-58/obj00000000000000000009?versionId=null 2023-05-31T20:39:08.888 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:en+2nFEAv7XeiQRMz9S+lRMWVb0= 2023-05-31T20:39:08.888 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:39:08 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:en+2nFEAv7XeiQRMz9S+lRMWVb0=', 'Content-Length': '0'} 2023-05-31T20:39:08.890 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:38:21 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007ad1f8e5f3116ab5-006477b06c-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:39:08 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:39:08.891 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000009 2023-05-31T20:39:08.891 DEBUG:boto:path=/pklxkd-58/obj00000000000000000009 2023-05-31T20:39:08.891 DEBUG:boto:auth_path=/pklxkd-58/obj00000000000000000009 2023-05-31T20:39:08.892 DEBUG:boto:path=/pklxkd-58/obj00000000000000000009?versionId=null 2023-05-31T20:39:08.892 DEBUG:boto:auth_path=/pklxkd-58/obj00000000000000000009?versionId=null 2023-05-31T20:39:08.893 DEBUG:boto:Method: GET 2023-05-31T20:39:08.893 DEBUG:boto:Path: /pklxkd-58/obj00000000000000000009?versionId=null 2023-05-31T20:39:08.894 DEBUG:boto:Data: 2023-05-31T20:39:08.894 DEBUG:boto:Headers: {} 2023-05-31T20:39:08.894 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:39:08.895 DEBUG:boto:Port: 8001 2023-05-31T20:39:08.895 DEBUG:boto:Params: {} 2023-05-31T20:39:08.896 DEBUG:boto:Token: None 2023-05-31T20:39:08.896 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:39:08 GMT /pklxkd-58/obj00000000000000000009?versionId=null 2023-05-31T20:39:08.897 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:iZ8nV2Ju4mRIPEh0nSolO6z40SE= 2023-05-31T20:39:08.897 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:39:08 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:iZ8nV2Ju4mRIPEh0nSolO6z40SE=', 'Content-Length': '0'} 2023-05-31T20:39:08.899 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:38:21 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007aae117809433ac4-006477b06c-1c17-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:39:08 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:39:08.900 DEBUG:boto:path=/pklxkd-58/obj00000000000000000009 2023-05-31T20:39:08.901 DEBUG:boto:auth_path=/pklxkd-58/obj00000000000000000009 2023-05-31T20:39:08.901 DEBUG:boto:path=/pklxkd-58/obj00000000000000000009?versionId=null 2023-05-31T20:39:08.901 DEBUG:boto:auth_path=/pklxkd-58/obj00000000000000000009?versionId=null 2023-05-31T20:39:08.902 DEBUG:boto:Method: GET 2023-05-31T20:39:08.902 DEBUG:boto:Path: /pklxkd-58/obj00000000000000000009?versionId=null 2023-05-31T20:39:08.903 DEBUG:boto:Data: 2023-05-31T20:39:08.903 DEBUG:boto:Headers: {} 2023-05-31T20:39:08.904 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:39:08.904 DEBUG:boto:Port: 8000 2023-05-31T20:39:08.904 DEBUG:boto:Params: {} 2023-05-31T20:39:08.905 DEBUG:boto:Token: None 2023-05-31T20:39:08.905 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:39:08 GMT /pklxkd-58/obj00000000000000000009?versionId=null 2023-05-31T20:39:08.906 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:iZ8nV2Ju4mRIPEh0nSolO6z40SE= 2023-05-31T20:39:08.906 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:39:08 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:iZ8nV2Ju4mRIPEh0nSolO6z40SE=', 'Content-Length': '0'} 2023-05-31T20:39:08.908 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:38:21 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006a2508b4a22a96cb-006477b06c-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:39:08 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:39:08.909 DEBUG:boto:path=/pklxkd-58/ 2023-05-31T20:39:08.910 DEBUG:boto:auth_path=/pklxkd-58/ 2023-05-31T20:39:08.910 DEBUG:boto:path=/pklxkd-58/?prefix=obj00000000000000000009 2023-05-31T20:39:08.911 DEBUG:boto:auth_path=/pklxkd-58/?prefix=obj00000000000000000009 2023-05-31T20:39:08.911 DEBUG:boto:Method: GET 2023-05-31T20:39:08.911 DEBUG:boto:Path: /pklxkd-58/?prefix=obj00000000000000000009 2023-05-31T20:39:08.912 DEBUG:boto:Data: 2023-05-31T20:39:08.912 DEBUG:boto:Headers: {} 2023-05-31T20:39:08.913 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:39:08.913 DEBUG:boto:Port: 8001 2023-05-31T20:39:08.914 DEBUG:boto:Params: {} 2023-05-31T20:39:08.914 DEBUG:boto:Token: None 2023-05-31T20:39:08.915 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:39:08 GMT /pklxkd-58/ 2023-05-31T20:39:08.915 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:ZA0HAdOWomTD+dvweZNsEimG+Ok= 2023-05-31T20:39:08.915 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:39:08 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:ZA0HAdOWomTD+dvweZNsEimG+Ok=', 'Content-Length': '0'} 2023-05-31T20:39:08.918 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000068526e07d62357ea-006477b06c-1c17-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:39:08 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:39:08.919 DEBUG:boto:b'pklxkd-58obj000000000000000000091000falseobj000000000000000000092023-05-31T20:38:21.588Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:39:08.919 DEBUG:boto:path=/pklxkd-58/ 2023-05-31T20:39:08.920 DEBUG:boto:auth_path=/pklxkd-58/ 2023-05-31T20:39:08.920 DEBUG:boto:path=/pklxkd-58/?prefix=obj00000000000000000009 2023-05-31T20:39:08.921 DEBUG:boto:auth_path=/pklxkd-58/?prefix=obj00000000000000000009 2023-05-31T20:39:08.921 DEBUG:boto:Method: GET 2023-05-31T20:39:08.922 DEBUG:boto:Path: /pklxkd-58/?prefix=obj00000000000000000009 2023-05-31T20:39:08.922 DEBUG:boto:Data: 2023-05-31T20:39:08.922 DEBUG:boto:Headers: {} 2023-05-31T20:39:08.923 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:39:08.923 DEBUG:boto:Port: 8000 2023-05-31T20:39:08.924 DEBUG:boto:Params: {} 2023-05-31T20:39:08.924 DEBUG:boto:Token: None 2023-05-31T20:39:08.925 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:39:08 GMT /pklxkd-58/ 2023-05-31T20:39:08.925 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:ZA0HAdOWomTD+dvweZNsEimG+Ok= 2023-05-31T20:39:08.926 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:39:08 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:ZA0HAdOWomTD+dvweZNsEimG+Ok=', 'Content-Length': '0'} 2023-05-31T20:39:08.930 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000004e5e8ded89f6e622-006477b06c-1630-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:39:08 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:39:08.972 DEBUG:boto:b'pklxkd-58obj000000000000000000091000falseobj000000000000000000092023-05-31T20:38:21.588Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:39:08.973 DEBUG:boto:path=/pklxkd-58/obj00000000000000000009 2023-05-31T20:39:08.974 DEBUG:boto:auth_path=/pklxkd-58/obj00000000000000000009 2023-05-31T20:39:08.974 DEBUG:boto:Method: HEAD 2023-05-31T20:39:08.974 DEBUG:boto:Path: /pklxkd-58/obj00000000000000000009 2023-05-31T20:39:08.975 DEBUG:boto:Data: 2023-05-31T20:39:08.975 DEBUG:boto:Headers: {} 2023-05-31T20:39:08.976 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:39:08.976 DEBUG:boto:Port: 8001 2023-05-31T20:39:08.977 DEBUG:boto:Params: {} 2023-05-31T20:39:08.977 DEBUG:boto:Token: None 2023-05-31T20:39:08.978 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:39:08 GMT /pklxkd-58/obj00000000000000000009 2023-05-31T20:39:08.978 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:iRXlrWyiKURAyYh4YAD7gDrtySE= 2023-05-31T20:39:08.979 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:39:08 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:iRXlrWyiKURAyYh4YAD7gDrtySE=', 'Content-Length': '0'} 2023-05-31T20:39:08.981 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:38:21 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ca636fefe3098b76-006477b06c-1c17-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:39:08 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:39:08.981 DEBUG:boto:path=/pklxkd-58/obj00000000000000000009 2023-05-31T20:39:08.982 DEBUG:boto:auth_path=/pklxkd-58/obj00000000000000000009 2023-05-31T20:39:08.982 DEBUG:boto:Method: HEAD 2023-05-31T20:39:08.982 DEBUG:boto:Path: /pklxkd-58/obj00000000000000000009 2023-05-31T20:39:08.983 DEBUG:boto:Data: 2023-05-31T20:39:08.983 DEBUG:boto:Headers: {} 2023-05-31T20:39:08.983 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:39:08.984 DEBUG:boto:Port: 8000 2023-05-31T20:39:08.984 DEBUG:boto:Params: {} 2023-05-31T20:39:08.984 DEBUG:boto:Token: None 2023-05-31T20:39:08.985 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:39:08 GMT /pklxkd-58/obj00000000000000000009 2023-05-31T20:39:08.985 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:iRXlrWyiKURAyYh4YAD7gDrtySE= 2023-05-31T20:39:08.986 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:39:08 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:iRXlrWyiKURAyYh4YAD7gDrtySE=', 'Content-Length': '0'} 2023-05-31T20:39:08.988 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:38:21 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000038a593101a1e7006-006477b06c-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:39:08 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:39:08.988 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000009 2023-05-31T20:39:08.989 DEBUG:boto:path=/pklxkd-58/obj00000000000000000009 2023-05-31T20:39:08.989 DEBUG:boto:auth_path=/pklxkd-58/obj00000000000000000009 2023-05-31T20:39:08.990 DEBUG:boto:Method: GET 2023-05-31T20:39:08.990 DEBUG:boto:Path: /pklxkd-58/obj00000000000000000009 2023-05-31T20:39:08.990 DEBUG:boto:Data: 2023-05-31T20:39:08.991 DEBUG:boto:Headers: {} 2023-05-31T20:39:08.991 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:39:08.991 DEBUG:boto:Port: 8001 2023-05-31T20:39:08.992 DEBUG:boto:Params: {} 2023-05-31T20:39:08.992 DEBUG:boto:Token: None 2023-05-31T20:39:08.993 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:39:08 GMT /pklxkd-58/obj00000000000000000009 2023-05-31T20:39:08.993 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:I2oOxXH0uJIHMQe9QoGLpMhwolc= 2023-05-31T20:39:08.994 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:39:08 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:I2oOxXH0uJIHMQe9QoGLpMhwolc=', 'Content-Length': '0'} 2023-05-31T20:39:08.995 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:38:21 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000fa7d52b7099160f8-006477b06c-1c17-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:39:08 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:39:09.040 DEBUG:boto:path=/pklxkd-58/obj00000000000000000009 2023-05-31T20:39:09.041 DEBUG:boto:auth_path=/pklxkd-58/obj00000000000000000009 2023-05-31T20:39:09.041 DEBUG:boto:Method: GET 2023-05-31T20:39:09.042 DEBUG:boto:Path: /pklxkd-58/obj00000000000000000009 2023-05-31T20:39:09.042 DEBUG:boto:Data: 2023-05-31T20:39:09.043 DEBUG:boto:Headers: {} 2023-05-31T20:39:09.043 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:39:09.043 DEBUG:boto:Port: 8000 2023-05-31T20:39:09.044 DEBUG:boto:Params: {} 2023-05-31T20:39:09.044 DEBUG:boto:Token: None 2023-05-31T20:39:09.045 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:39:09 GMT /pklxkd-58/obj00000000000000000009 2023-05-31T20:39:09.045 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:04BqOkDvnXfAa7QCZhGaeEaa224= 2023-05-31T20:39:09.046 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:39:09 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:04BqOkDvnXfAa7QCZhGaeEaa224=', 'Content-Length': '0'} 2023-05-31T20:39:09.048 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:38:21 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000005d7ec90ae789ae9-006477b06d-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:39:09 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:39:09.048 DEBUG:boto:path=/pklxkd-58/ 2023-05-31T20:39:09.049 DEBUG:boto:auth_path=/pklxkd-58/ 2023-05-31T20:39:09.049 DEBUG:boto:path=/pklxkd-58/?prefix=obj00000000000000000009 2023-05-31T20:39:09.050 DEBUG:boto:auth_path=/pklxkd-58/?prefix=obj00000000000000000009 2023-05-31T20:39:09.050 DEBUG:boto:Method: GET 2023-05-31T20:39:09.050 DEBUG:boto:Path: /pklxkd-58/?prefix=obj00000000000000000009 2023-05-31T20:39:09.051 DEBUG:boto:Data: 2023-05-31T20:39:09.051 DEBUG:boto:Headers: {} 2023-05-31T20:39:09.051 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:39:09.052 DEBUG:boto:Port: 8001 2023-05-31T20:39:09.052 DEBUG:boto:Params: {} 2023-05-31T20:39:09.053 DEBUG:boto:Token: None 2023-05-31T20:39:09.053 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:39:09 GMT /pklxkd-58/ 2023-05-31T20:39:09.054 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:gwqyY/Id1F/1VD2W4hCRK4WEzXM= 2023-05-31T20:39:09.054 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:39:09 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:gwqyY/Id1F/1VD2W4hCRK4WEzXM=', 'Content-Length': '0'} 2023-05-31T20:39:09.057 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000008fa25fa99c1054-006477b06d-1c17-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:39:09 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:39:09.057 DEBUG:boto:b'pklxkd-58obj000000000000000000091000falseobj000000000000000000092023-05-31T20:38:21.588Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:39:09.058 DEBUG:boto:path=/pklxkd-58/ 2023-05-31T20:39:09.058 DEBUG:boto:auth_path=/pklxkd-58/ 2023-05-31T20:39:09.059 DEBUG:boto:path=/pklxkd-58/?prefix=obj00000000000000000009 2023-05-31T20:39:09.059 DEBUG:boto:auth_path=/pklxkd-58/?prefix=obj00000000000000000009 2023-05-31T20:39:09.059 DEBUG:boto:Method: GET 2023-05-31T20:39:09.060 DEBUG:boto:Path: /pklxkd-58/?prefix=obj00000000000000000009 2023-05-31T20:39:09.060 DEBUG:boto:Data: 2023-05-31T20:39:09.061 DEBUG:boto:Headers: {} 2023-05-31T20:39:09.061 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:39:09.061 DEBUG:boto:Port: 8000 2023-05-31T20:39:09.062 DEBUG:boto:Params: {} 2023-05-31T20:39:09.063 DEBUG:boto:Token: None 2023-05-31T20:39:09.064 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:39:09 GMT /pklxkd-58/ 2023-05-31T20:39:09.064 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:gwqyY/Id1F/1VD2W4hCRK4WEzXM= 2023-05-31T20:39:09.065 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:39:09 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:gwqyY/Id1F/1VD2W4hCRK4WEzXM=', 'Content-Length': '0'} 2023-05-31T20:39:09.069 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000039349dc8f5f43513-006477b06d-1630-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:39:09 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:39:09.070 DEBUG:boto:b'pklxkd-58obj000000000000000000091000falseobj000000000000000000092023-05-31T20:38:21.588Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:39:09.071 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000010 2023-05-31T20:39:09.071 DEBUG:boto:path=/pklxkd-58/obj00000000000000000010 2023-05-31T20:39:09.072 DEBUG:boto:auth_path=/pklxkd-58/obj00000000000000000010 2023-05-31T20:39:09.072 DEBUG:boto:path=/pklxkd-58/obj00000000000000000010?versionId=null 2023-05-31T20:39:09.073 DEBUG:boto:auth_path=/pklxkd-58/obj00000000000000000010?versionId=null 2023-05-31T20:39:09.073 DEBUG:boto:Method: GET 2023-05-31T20:39:09.074 DEBUG:boto:Path: /pklxkd-58/obj00000000000000000010?versionId=null 2023-05-31T20:39:09.074 DEBUG:boto:Data: 2023-05-31T20:39:09.074 DEBUG:boto:Headers: {} 2023-05-31T20:39:09.075 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:39:09.075 DEBUG:boto:Port: 8001 2023-05-31T20:39:09.075 DEBUG:boto:Params: {} 2023-05-31T20:39:09.076 DEBUG:boto:Token: None 2023-05-31T20:39:09.077 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:39:09 GMT /pklxkd-58/obj00000000000000000010?versionId=null 2023-05-31T20:39:09.078 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:yDPuUJBSTv7tqbUDJjROcFIPC8Y= 2023-05-31T20:39:09.078 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:39:09 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:yDPuUJBSTv7tqbUDJjROcFIPC8Y=', 'Content-Length': '0'} 2023-05-31T20:39:09.081 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:38:21 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b8eb97fd02e33100-006477b06d-1c17-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:39:09 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:39:09.081 DEBUG:boto:path=/pklxkd-58/obj00000000000000000010 2023-05-31T20:39:09.082 DEBUG:boto:auth_path=/pklxkd-58/obj00000000000000000010 2023-05-31T20:39:09.082 DEBUG:boto:path=/pklxkd-58/obj00000000000000000010?versionId=null 2023-05-31T20:39:09.082 DEBUG:boto:auth_path=/pklxkd-58/obj00000000000000000010?versionId=null 2023-05-31T20:39:09.083 DEBUG:boto:Method: GET 2023-05-31T20:39:09.087 DEBUG:boto:Path: /pklxkd-58/obj00000000000000000010?versionId=null 2023-05-31T20:39:09.088 DEBUG:boto:Data: 2023-05-31T20:39:09.088 DEBUG:boto:Headers: {} 2023-05-31T20:39:09.089 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:39:09.089 DEBUG:boto:Port: 8000 2023-05-31T20:39:09.089 DEBUG:boto:Params: {} 2023-05-31T20:39:09.090 DEBUG:boto:Token: None 2023-05-31T20:39:09.090 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:39:09 GMT /pklxkd-58/obj00000000000000000010?versionId=null 2023-05-31T20:39:09.091 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:yDPuUJBSTv7tqbUDJjROcFIPC8Y= 2023-05-31T20:39:09.091 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:39:09 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:yDPuUJBSTv7tqbUDJjROcFIPC8Y=', 'Content-Length': '0'} 2023-05-31T20:39:09.093 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:38:21 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001c147e009a674dbe-006477b06d-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:39:09 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:39:09.094 DEBUG:boto:path=/pklxkd-58/obj00000000000000000010 2023-05-31T20:39:09.095 DEBUG:boto:auth_path=/pklxkd-58/obj00000000000000000010 2023-05-31T20:39:09.095 DEBUG:boto:path=/pklxkd-58/obj00000000000000000010?versionId=null 2023-05-31T20:39:09.095 DEBUG:boto:auth_path=/pklxkd-58/obj00000000000000000010?versionId=null 2023-05-31T20:39:09.096 DEBUG:boto:Method: HEAD 2023-05-31T20:39:09.096 DEBUG:boto:Path: /pklxkd-58/obj00000000000000000010?versionId=null 2023-05-31T20:39:09.097 DEBUG:boto:Data: 2023-05-31T20:39:09.098 DEBUG:boto:Headers: {} 2023-05-31T20:39:09.098 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:39:09.098 DEBUG:boto:Port: 8001 2023-05-31T20:39:09.099 DEBUG:boto:Params: {} 2023-05-31T20:39:09.099 DEBUG:boto:Token: None 2023-05-31T20:39:09.100 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:39:09 GMT /pklxkd-58/obj00000000000000000010?versionId=null 2023-05-31T20:39:09.100 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:n2bmBIHdmHm5avXZ7E64CkLPUno= 2023-05-31T20:39:09.101 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:39:09 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:n2bmBIHdmHm5avXZ7E64CkLPUno=', 'Content-Length': '0'} 2023-05-31T20:39:09.103 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:38:21 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a779093bd8b8d6ab-006477b06d-1c17-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:39:09 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:39:09.104 DEBUG:boto:path=/pklxkd-58/obj00000000000000000010 2023-05-31T20:39:09.104 DEBUG:boto:auth_path=/pklxkd-58/obj00000000000000000010 2023-05-31T20:39:09.105 DEBUG:boto:path=/pklxkd-58/obj00000000000000000010?versionId=null 2023-05-31T20:39:09.105 DEBUG:boto:auth_path=/pklxkd-58/obj00000000000000000010?versionId=null 2023-05-31T20:39:09.105 DEBUG:boto:Method: HEAD 2023-05-31T20:39:09.106 DEBUG:boto:Path: /pklxkd-58/obj00000000000000000010?versionId=null 2023-05-31T20:39:09.106 DEBUG:boto:Data: 2023-05-31T20:39:09.107 DEBUG:boto:Headers: {} 2023-05-31T20:39:09.107 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:39:09.107 DEBUG:boto:Port: 8000 2023-05-31T20:39:09.108 DEBUG:boto:Params: {} 2023-05-31T20:39:09.108 DEBUG:boto:Token: None 2023-05-31T20:39:09.109 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:39:09 GMT /pklxkd-58/obj00000000000000000010?versionId=null 2023-05-31T20:39:09.109 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:n2bmBIHdmHm5avXZ7E64CkLPUno= 2023-05-31T20:39:09.113 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:39:09 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:n2bmBIHdmHm5avXZ7E64CkLPUno=', 'Content-Length': '0'} 2023-05-31T20:39:09.115 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:38:21 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c2e4283a168ca729-006477b06d-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:39:09 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:39:09.116 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000010 2023-05-31T20:39:09.116 DEBUG:boto:path=/pklxkd-58/obj00000000000000000010 2023-05-31T20:39:09.117 DEBUG:boto:auth_path=/pklxkd-58/obj00000000000000000010 2023-05-31T20:39:09.117 DEBUG:boto:path=/pklxkd-58/obj00000000000000000010?versionId=null 2023-05-31T20:39:09.117 DEBUG:boto:auth_path=/pklxkd-58/obj00000000000000000010?versionId=null 2023-05-31T20:39:09.118 DEBUG:boto:Method: GET 2023-05-31T20:39:09.118 DEBUG:boto:Path: /pklxkd-58/obj00000000000000000010?versionId=null 2023-05-31T20:39:09.118 DEBUG:boto:Data: 2023-05-31T20:39:09.119 DEBUG:boto:Headers: {} 2023-05-31T20:39:09.119 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:39:09.120 DEBUG:boto:Port: 8001 2023-05-31T20:39:09.120 DEBUG:boto:Params: {} 2023-05-31T20:39:09.121 DEBUG:boto:Token: None 2023-05-31T20:39:09.121 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:39:09 GMT /pklxkd-58/obj00000000000000000010?versionId=null 2023-05-31T20:39:09.122 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:yDPuUJBSTv7tqbUDJjROcFIPC8Y= 2023-05-31T20:39:09.122 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:39:09 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:yDPuUJBSTv7tqbUDJjROcFIPC8Y=', 'Content-Length': '0'} 2023-05-31T20:39:09.124 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:38:21 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000fc142c24b06f8666-006477b06d-1c17-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:39:09 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:39:09.169 DEBUG:boto:path=/pklxkd-58/obj00000000000000000010 2023-05-31T20:39:09.170 DEBUG:boto:auth_path=/pklxkd-58/obj00000000000000000010 2023-05-31T20:39:09.170 DEBUG:boto:path=/pklxkd-58/obj00000000000000000010?versionId=null 2023-05-31T20:39:09.170 DEBUG:boto:auth_path=/pklxkd-58/obj00000000000000000010?versionId=null 2023-05-31T20:39:09.171 DEBUG:boto:Method: GET 2023-05-31T20:39:09.171 DEBUG:boto:Path: /pklxkd-58/obj00000000000000000010?versionId=null 2023-05-31T20:39:09.172 DEBUG:boto:Data: 2023-05-31T20:39:09.172 DEBUG:boto:Headers: {} 2023-05-31T20:39:09.172 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:39:09.173 DEBUG:boto:Port: 8000 2023-05-31T20:39:09.173 DEBUG:boto:Params: {} 2023-05-31T20:39:09.174 DEBUG:boto:Token: None 2023-05-31T20:39:09.174 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:39:09 GMT /pklxkd-58/obj00000000000000000010?versionId=null 2023-05-31T20:39:09.175 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:yDPuUJBSTv7tqbUDJjROcFIPC8Y= 2023-05-31T20:39:09.175 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:39:09 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:yDPuUJBSTv7tqbUDJjROcFIPC8Y=', 'Content-Length': '0'} 2023-05-31T20:39:09.177 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:38:21 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e60b704959c78e33-006477b06d-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:39:09 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:39:09.178 DEBUG:boto:path=/pklxkd-58/ 2023-05-31T20:39:09.179 DEBUG:boto:auth_path=/pklxkd-58/ 2023-05-31T20:39:09.180 DEBUG:boto:path=/pklxkd-58/?prefix=obj00000000000000000010 2023-05-31T20:39:09.180 DEBUG:boto:auth_path=/pklxkd-58/?prefix=obj00000000000000000010 2023-05-31T20:39:09.180 DEBUG:boto:Method: GET 2023-05-31T20:39:09.181 DEBUG:boto:Path: /pklxkd-58/?prefix=obj00000000000000000010 2023-05-31T20:39:09.181 DEBUG:boto:Data: 2023-05-31T20:39:09.182 DEBUG:boto:Headers: {} 2023-05-31T20:39:09.182 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:39:09.182 DEBUG:boto:Port: 8001 2023-05-31T20:39:09.183 DEBUG:boto:Params: {} 2023-05-31T20:39:09.183 DEBUG:boto:Token: None 2023-05-31T20:39:09.184 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:39:09 GMT /pklxkd-58/ 2023-05-31T20:39:09.184 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:gwqyY/Id1F/1VD2W4hCRK4WEzXM= 2023-05-31T20:39:09.185 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:39:09 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:gwqyY/Id1F/1VD2W4hCRK4WEzXM=', 'Content-Length': '0'} 2023-05-31T20:39:09.187 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000001872e90b5e789a8d-006477b06d-1c17-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:39:09 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:39:09.188 DEBUG:boto:b'pklxkd-58obj000000000000000000101000falseobj000000000000000000102023-05-31T20:38:21.127Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:39:09.188 DEBUG:boto:path=/pklxkd-58/ 2023-05-31T20:39:09.189 DEBUG:boto:auth_path=/pklxkd-58/ 2023-05-31T20:39:09.190 DEBUG:boto:path=/pklxkd-58/?prefix=obj00000000000000000010 2023-05-31T20:39:09.190 DEBUG:boto:auth_path=/pklxkd-58/?prefix=obj00000000000000000010 2023-05-31T20:39:09.191 DEBUG:boto:Method: GET 2023-05-31T20:39:09.191 DEBUG:boto:Path: /pklxkd-58/?prefix=obj00000000000000000010 2023-05-31T20:39:09.192 DEBUG:boto:Data: 2023-05-31T20:39:09.192 DEBUG:boto:Headers: {} 2023-05-31T20:39:09.192 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:39:09.193 DEBUG:boto:Port: 8000 2023-05-31T20:39:09.193 DEBUG:boto:Params: {} 2023-05-31T20:39:09.193 DEBUG:boto:Token: None 2023-05-31T20:39:09.194 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:39:09 GMT /pklxkd-58/ 2023-05-31T20:39:09.194 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:gwqyY/Id1F/1VD2W4hCRK4WEzXM= 2023-05-31T20:39:09.195 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:39:09 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:gwqyY/Id1F/1VD2W4hCRK4WEzXM=', 'Content-Length': '0'} 2023-05-31T20:39:09.200 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000519e16af7d4fb78b-006477b06d-1630-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:39:09 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:39:09.202 DEBUG:boto:b'pklxkd-58obj000000000000000000101000falseobj000000000000000000102023-05-31T20:38:21.127Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:39:09.203 DEBUG:boto:path=/pklxkd-58/obj00000000000000000010 2023-05-31T20:39:09.204 DEBUG:boto:auth_path=/pklxkd-58/obj00000000000000000010 2023-05-31T20:39:09.205 DEBUG:boto:Method: HEAD 2023-05-31T20:39:09.205 DEBUG:boto:Path: /pklxkd-58/obj00000000000000000010 2023-05-31T20:39:09.205 DEBUG:boto:Data: 2023-05-31T20:39:09.206 DEBUG:boto:Headers: {} 2023-05-31T20:39:09.206 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:39:09.207 DEBUG:boto:Port: 8001 2023-05-31T20:39:09.207 DEBUG:boto:Params: {} 2023-05-31T20:39:09.208 DEBUG:boto:Token: None 2023-05-31T20:39:09.208 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:39:09 GMT /pklxkd-58/obj00000000000000000010 2023-05-31T20:39:09.209 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:Qw6sVNcY6Xf9mfaiAdTsAkkT+Rw= 2023-05-31T20:39:09.210 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:39:09 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:Qw6sVNcY6Xf9mfaiAdTsAkkT+Rw=', 'Content-Length': '0'} 2023-05-31T20:39:09.212 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:38:21 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f49e3594ab2a4779-006477b06d-1c17-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:39:09 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:39:09.213 DEBUG:boto:path=/pklxkd-58/obj00000000000000000010 2023-05-31T20:39:09.214 DEBUG:boto:auth_path=/pklxkd-58/obj00000000000000000010 2023-05-31T20:39:09.215 DEBUG:boto:Method: HEAD 2023-05-31T20:39:09.216 DEBUG:boto:Path: /pklxkd-58/obj00000000000000000010 2023-05-31T20:39:09.217 DEBUG:boto:Data: 2023-05-31T20:39:09.218 DEBUG:boto:Headers: {} 2023-05-31T20:39:09.218 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:39:09.218 DEBUG:boto:Port: 8000 2023-05-31T20:39:09.219 DEBUG:boto:Params: {} 2023-05-31T20:39:09.220 DEBUG:boto:Token: None 2023-05-31T20:39:09.220 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:39:09 GMT /pklxkd-58/obj00000000000000000010 2023-05-31T20:39:09.221 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:Qw6sVNcY6Xf9mfaiAdTsAkkT+Rw= 2023-05-31T20:39:09.222 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:39:09 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:Qw6sVNcY6Xf9mfaiAdTsAkkT+Rw=', 'Content-Length': '0'} 2023-05-31T20:39:09.224 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:38:21 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001526b16d69faa97f-006477b06d-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:39:09 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:39:09.226 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000010 2023-05-31T20:39:09.226 DEBUG:boto:path=/pklxkd-58/obj00000000000000000010 2023-05-31T20:39:09.226 DEBUG:boto:auth_path=/pklxkd-58/obj00000000000000000010 2023-05-31T20:39:09.227 DEBUG:boto:Method: GET 2023-05-31T20:39:09.227 DEBUG:boto:Path: /pklxkd-58/obj00000000000000000010 2023-05-31T20:39:09.228 DEBUG:boto:Data: 2023-05-31T20:39:09.228 DEBUG:boto:Headers: {} 2023-05-31T20:39:09.228 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:39:09.229 DEBUG:boto:Port: 8001 2023-05-31T20:39:09.229 DEBUG:boto:Params: {} 2023-05-31T20:39:09.229 DEBUG:boto:Token: None 2023-05-31T20:39:09.230 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:39:09 GMT /pklxkd-58/obj00000000000000000010 2023-05-31T20:39:09.230 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:vnMDEqGQcEZcBm8TsGGfr+EHjl4= 2023-05-31T20:39:09.231 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:39:09 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:vnMDEqGQcEZcBm8TsGGfr+EHjl4=', 'Content-Length': '0'} 2023-05-31T20:39:09.233 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:38:21 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007ff9c012cb7d0a01-006477b06d-1c17-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:39:09 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:39:09.234 DEBUG:boto:path=/pklxkd-58/obj00000000000000000010 2023-05-31T20:39:09.234 DEBUG:boto:auth_path=/pklxkd-58/obj00000000000000000010 2023-05-31T20:39:09.234 DEBUG:boto:Method: GET 2023-05-31T20:39:09.235 DEBUG:boto:Path: /pklxkd-58/obj00000000000000000010 2023-05-31T20:39:09.235 DEBUG:boto:Data: 2023-05-31T20:39:09.235 DEBUG:boto:Headers: {} 2023-05-31T20:39:09.236 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:39:09.236 DEBUG:boto:Port: 8000 2023-05-31T20:39:09.237 DEBUG:boto:Params: {} 2023-05-31T20:39:09.237 DEBUG:boto:Token: None 2023-05-31T20:39:09.237 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:39:09 GMT /pklxkd-58/obj00000000000000000010 2023-05-31T20:39:09.238 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:vnMDEqGQcEZcBm8TsGGfr+EHjl4= 2023-05-31T20:39:09.238 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:39:09 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:vnMDEqGQcEZcBm8TsGGfr+EHjl4=', 'Content-Length': '0'} 2023-05-31T20:39:09.240 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:38:21 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000fe2d40b1a133a04f-006477b06d-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:39:09 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:39:09.284 DEBUG:boto:path=/pklxkd-58/ 2023-05-31T20:39:09.285 DEBUG:boto:auth_path=/pklxkd-58/ 2023-05-31T20:39:09.285 DEBUG:boto:path=/pklxkd-58/?prefix=obj00000000000000000010 2023-05-31T20:39:09.286 DEBUG:boto:auth_path=/pklxkd-58/?prefix=obj00000000000000000010 2023-05-31T20:39:09.286 DEBUG:boto:Method: GET 2023-05-31T20:39:09.286 DEBUG:boto:Path: /pklxkd-58/?prefix=obj00000000000000000010 2023-05-31T20:39:09.287 DEBUG:boto:Data: 2023-05-31T20:39:09.287 DEBUG:boto:Headers: {} 2023-05-31T20:39:09.287 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:39:09.288 DEBUG:boto:Port: 8001 2023-05-31T20:39:09.288 DEBUG:boto:Params: {} 2023-05-31T20:39:09.289 DEBUG:boto:Token: None 2023-05-31T20:39:09.289 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:39:09 GMT /pklxkd-58/ 2023-05-31T20:39:09.290 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:gwqyY/Id1F/1VD2W4hCRK4WEzXM= 2023-05-31T20:39:09.290 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:39:09 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:gwqyY/Id1F/1VD2W4hCRK4WEzXM=', 'Content-Length': '0'} 2023-05-31T20:39:09.293 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ce17ea89cff1cdc7-006477b06d-1c17-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:39:09 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:39:09.294 DEBUG:boto:b'pklxkd-58obj000000000000000000101000falseobj000000000000000000102023-05-31T20:38:21.127Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:39:09.295 DEBUG:boto:path=/pklxkd-58/ 2023-05-31T20:39:09.297 DEBUG:boto:auth_path=/pklxkd-58/ 2023-05-31T20:39:09.297 DEBUG:boto:path=/pklxkd-58/?prefix=obj00000000000000000010 2023-05-31T20:39:09.297 DEBUG:boto:auth_path=/pklxkd-58/?prefix=obj00000000000000000010 2023-05-31T20:39:09.298 DEBUG:boto:Method: GET 2023-05-31T20:39:09.298 DEBUG:boto:Path: /pklxkd-58/?prefix=obj00000000000000000010 2023-05-31T20:39:09.299 DEBUG:boto:Data: 2023-05-31T20:39:09.299 DEBUG:boto:Headers: {} 2023-05-31T20:39:09.299 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:39:09.300 DEBUG:boto:Port: 8000 2023-05-31T20:39:09.300 DEBUG:boto:Params: {} 2023-05-31T20:39:09.300 DEBUG:boto:Token: None 2023-05-31T20:39:09.301 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:39:09 GMT /pklxkd-58/ 2023-05-31T20:39:09.301 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:gwqyY/Id1F/1VD2W4hCRK4WEzXM= 2023-05-31T20:39:09.302 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:39:09 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:gwqyY/Id1F/1VD2W4hCRK4WEzXM=', 'Content-Length': '0'} 2023-05-31T20:39:09.306 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a53b355b8f6df59f-006477b06d-1630-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:39:09 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:39:09.307 DEBUG:boto:b'pklxkd-58obj000000000000000000101000falseobj000000000000000000102023-05-31T20:38:21.127Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:39:09.308 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000011 2023-05-31T20:39:09.308 DEBUG:boto:path=/pklxkd-58/obj00000000000000000011 2023-05-31T20:39:09.309 DEBUG:boto:auth_path=/pklxkd-58/obj00000000000000000011 2023-05-31T20:39:09.309 DEBUG:boto:path=/pklxkd-58/obj00000000000000000011?versionId=null 2023-05-31T20:39:09.309 DEBUG:boto:auth_path=/pklxkd-58/obj00000000000000000011?versionId=null 2023-05-31T20:39:09.310 DEBUG:boto:Method: GET 2023-05-31T20:39:09.310 DEBUG:boto:Path: /pklxkd-58/obj00000000000000000011?versionId=null 2023-05-31T20:39:09.310 DEBUG:boto:Data: 2023-05-31T20:39:09.311 DEBUG:boto:Headers: {} 2023-05-31T20:39:09.311 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:39:09.311 DEBUG:boto:Port: 8001 2023-05-31T20:39:09.312 DEBUG:boto:Params: {} 2023-05-31T20:39:09.312 DEBUG:boto:Token: None 2023-05-31T20:39:09.313 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:39:09 GMT /pklxkd-58/obj00000000000000000011?versionId=null 2023-05-31T20:39:09.313 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:Quqb2nHHv79kFpKdumIr4FfQ+Ec= 2023-05-31T20:39:09.313 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:39:09 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:Quqb2nHHv79kFpKdumIr4FfQ+Ec=', 'Content-Length': '0'} 2023-05-31T20:39:09.316 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:38:22 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000750384569fbe44c5-006477b06d-1c17-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:39:09 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:39:09.360 DEBUG:boto:path=/pklxkd-58/obj00000000000000000011 2023-05-31T20:39:09.361 DEBUG:boto:auth_path=/pklxkd-58/obj00000000000000000011 2023-05-31T20:39:09.362 DEBUG:boto:path=/pklxkd-58/obj00000000000000000011?versionId=null 2023-05-31T20:39:09.362 DEBUG:boto:auth_path=/pklxkd-58/obj00000000000000000011?versionId=null 2023-05-31T20:39:09.362 DEBUG:boto:Method: GET 2023-05-31T20:39:09.363 DEBUG:boto:Path: /pklxkd-58/obj00000000000000000011?versionId=null 2023-05-31T20:39:09.363 DEBUG:boto:Data: 2023-05-31T20:39:09.363 DEBUG:boto:Headers: {} 2023-05-31T20:39:09.364 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:39:09.364 DEBUG:boto:Port: 8000 2023-05-31T20:39:09.365 DEBUG:boto:Params: {} 2023-05-31T20:39:09.365 DEBUG:boto:Token: None 2023-05-31T20:39:09.365 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:39:09 GMT /pklxkd-58/obj00000000000000000011?versionId=null 2023-05-31T20:39:09.366 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:Quqb2nHHv79kFpKdumIr4FfQ+Ec= 2023-05-31T20:39:09.366 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:39:09 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:Quqb2nHHv79kFpKdumIr4FfQ+Ec=', 'Content-Length': '0'} 2023-05-31T20:39:09.370 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:38:22 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d6961ffd59c76460-006477b06d-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:39:09 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:39:09.371 DEBUG:boto:path=/pklxkd-58/obj00000000000000000011 2023-05-31T20:39:09.371 DEBUG:boto:auth_path=/pklxkd-58/obj00000000000000000011 2023-05-31T20:39:09.371 DEBUG:boto:path=/pklxkd-58/obj00000000000000000011?versionId=null 2023-05-31T20:39:09.372 DEBUG:boto:auth_path=/pklxkd-58/obj00000000000000000011?versionId=null 2023-05-31T20:39:09.372 DEBUG:boto:Method: HEAD 2023-05-31T20:39:09.372 DEBUG:boto:Path: /pklxkd-58/obj00000000000000000011?versionId=null 2023-05-31T20:39:09.373 DEBUG:boto:Data: 2023-05-31T20:39:09.373 DEBUG:boto:Headers: {} 2023-05-31T20:39:09.373 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:39:09.374 DEBUG:boto:Port: 8001 2023-05-31T20:39:09.374 DEBUG:boto:Params: {} 2023-05-31T20:39:09.374 DEBUG:boto:Token: None 2023-05-31T20:39:09.375 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:39:09 GMT /pklxkd-58/obj00000000000000000011?versionId=null 2023-05-31T20:39:09.375 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:BQhV51VdS4CsMRD4dOeKYWkrmFw= 2023-05-31T20:39:09.376 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:39:09 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:BQhV51VdS4CsMRD4dOeKYWkrmFw=', 'Content-Length': '0'} 2023-05-31T20:39:09.378 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:38:22 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001627c87d43188527-006477b06d-1c17-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:39:09 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:39:09.379 DEBUG:boto:path=/pklxkd-58/obj00000000000000000011 2023-05-31T20:39:09.379 DEBUG:boto:auth_path=/pklxkd-58/obj00000000000000000011 2023-05-31T20:39:09.380 DEBUG:boto:path=/pklxkd-58/obj00000000000000000011?versionId=null 2023-05-31T20:39:09.381 DEBUG:boto:auth_path=/pklxkd-58/obj00000000000000000011?versionId=null 2023-05-31T20:39:09.381 DEBUG:boto:Method: HEAD 2023-05-31T20:39:09.382 DEBUG:boto:Path: /pklxkd-58/obj00000000000000000011?versionId=null 2023-05-31T20:39:09.382 DEBUG:boto:Data: 2023-05-31T20:39:09.382 DEBUG:boto:Headers: {} 2023-05-31T20:39:09.383 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:39:09.383 DEBUG:boto:Port: 8000 2023-05-31T20:39:09.383 DEBUG:boto:Params: {} 2023-05-31T20:39:09.384 DEBUG:boto:Token: None 2023-05-31T20:39:09.384 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:39:09 GMT /pklxkd-58/obj00000000000000000011?versionId=null 2023-05-31T20:39:09.385 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:BQhV51VdS4CsMRD4dOeKYWkrmFw= 2023-05-31T20:39:09.385 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:39:09 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:BQhV51VdS4CsMRD4dOeKYWkrmFw=', 'Content-Length': '0'} 2023-05-31T20:39:09.387 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:38:22 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000080f569655be17603-006477b06d-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:39:09 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:39:09.391 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000011 2023-05-31T20:39:09.392 DEBUG:boto:path=/pklxkd-58/obj00000000000000000011 2023-05-31T20:39:09.392 DEBUG:boto:auth_path=/pklxkd-58/obj00000000000000000011 2023-05-31T20:39:09.393 DEBUG:boto:path=/pklxkd-58/obj00000000000000000011?versionId=null 2023-05-31T20:39:09.393 DEBUG:boto:auth_path=/pklxkd-58/obj00000000000000000011?versionId=null 2023-05-31T20:39:09.393 DEBUG:boto:Method: GET 2023-05-31T20:39:09.394 DEBUG:boto:Path: /pklxkd-58/obj00000000000000000011?versionId=null 2023-05-31T20:39:09.394 DEBUG:boto:Data: 2023-05-31T20:39:09.395 DEBUG:boto:Headers: {} 2023-05-31T20:39:09.395 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:39:09.395 DEBUG:boto:Port: 8001 2023-05-31T20:39:09.396 DEBUG:boto:Params: {} 2023-05-31T20:39:09.397 DEBUG:boto:Token: None 2023-05-31T20:39:09.397 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:39:09 GMT /pklxkd-58/obj00000000000000000011?versionId=null 2023-05-31T20:39:09.398 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:Quqb2nHHv79kFpKdumIr4FfQ+Ec= 2023-05-31T20:39:09.398 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:39:09 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:Quqb2nHHv79kFpKdumIr4FfQ+Ec=', 'Content-Length': '0'} 2023-05-31T20:39:09.401 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:38:22 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000760e46236e0738b-006477b06d-1c17-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:39:09 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:39:09.402 DEBUG:boto:path=/pklxkd-58/obj00000000000000000011 2023-05-31T20:39:09.402 DEBUG:boto:auth_path=/pklxkd-58/obj00000000000000000011 2023-05-31T20:39:09.403 DEBUG:boto:path=/pklxkd-58/obj00000000000000000011?versionId=null 2023-05-31T20:39:09.403 DEBUG:boto:auth_path=/pklxkd-58/obj00000000000000000011?versionId=null 2023-05-31T20:39:09.403 DEBUG:boto:Method: GET 2023-05-31T20:39:09.404 DEBUG:boto:Path: /pklxkd-58/obj00000000000000000011?versionId=null 2023-05-31T20:39:09.405 DEBUG:boto:Data: 2023-05-31T20:39:09.405 DEBUG:boto:Headers: {} 2023-05-31T20:39:09.405 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:39:09.406 DEBUG:boto:Port: 8000 2023-05-31T20:39:09.406 DEBUG:boto:Params: {} 2023-05-31T20:39:09.407 DEBUG:boto:Token: None 2023-05-31T20:39:09.407 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:39:09 GMT /pklxkd-58/obj00000000000000000011?versionId=null 2023-05-31T20:39:09.408 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:Quqb2nHHv79kFpKdumIr4FfQ+Ec= 2023-05-31T20:39:09.408 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:39:09 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:Quqb2nHHv79kFpKdumIr4FfQ+Ec=', 'Content-Length': '0'} 2023-05-31T20:39:09.410 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:38:22 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004443476728fda563-006477b06d-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:39:09 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:39:09.412 DEBUG:boto:path=/pklxkd-58/ 2023-05-31T20:39:09.412 DEBUG:boto:auth_path=/pklxkd-58/ 2023-05-31T20:39:09.412 DEBUG:boto:path=/pklxkd-58/?prefix=obj00000000000000000011 2023-05-31T20:39:09.413 DEBUG:boto:auth_path=/pklxkd-58/?prefix=obj00000000000000000011 2023-05-31T20:39:09.413 DEBUG:boto:Method: GET 2023-05-31T20:39:09.414 DEBUG:boto:Path: /pklxkd-58/?prefix=obj00000000000000000011 2023-05-31T20:39:09.414 DEBUG:boto:Data: 2023-05-31T20:39:09.414 DEBUG:boto:Headers: {} 2023-05-31T20:39:09.415 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:39:09.415 DEBUG:boto:Port: 8001 2023-05-31T20:39:09.415 DEBUG:boto:Params: {} 2023-05-31T20:39:09.416 DEBUG:boto:Token: None 2023-05-31T20:39:09.416 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:39:09 GMT /pklxkd-58/ 2023-05-31T20:39:09.417 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:gwqyY/Id1F/1VD2W4hCRK4WEzXM= 2023-05-31T20:39:09.417 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:39:09 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:gwqyY/Id1F/1VD2W4hCRK4WEzXM=', 'Content-Length': '0'} 2023-05-31T20:39:09.420 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f0c832f68bf6a7de-006477b06d-1c17-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:39:09 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:39:09.423 DEBUG:boto:b'pklxkd-58obj000000000000000000111000falseobj000000000000000000112023-05-31T20:38:22.712Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:39:09.424 DEBUG:boto:path=/pklxkd-58/ 2023-05-31T20:39:09.425 DEBUG:boto:auth_path=/pklxkd-58/ 2023-05-31T20:39:09.425 DEBUG:boto:path=/pklxkd-58/?prefix=obj00000000000000000011 2023-05-31T20:39:09.426 DEBUG:boto:auth_path=/pklxkd-58/?prefix=obj00000000000000000011 2023-05-31T20:39:09.426 DEBUG:boto:Method: GET 2023-05-31T20:39:09.426 DEBUG:boto:Path: /pklxkd-58/?prefix=obj00000000000000000011 2023-05-31T20:39:09.427 DEBUG:boto:Data: 2023-05-31T20:39:09.427 DEBUG:boto:Headers: {} 2023-05-31T20:39:09.427 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:39:09.428 DEBUG:boto:Port: 8000 2023-05-31T20:39:09.428 DEBUG:boto:Params: {} 2023-05-31T20:39:09.429 DEBUG:boto:Token: None 2023-05-31T20:39:09.429 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:39:09 GMT /pklxkd-58/ 2023-05-31T20:39:09.430 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:gwqyY/Id1F/1VD2W4hCRK4WEzXM= 2023-05-31T20:39:09.430 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:39:09 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:gwqyY/Id1F/1VD2W4hCRK4WEzXM=', 'Content-Length': '0'} 2023-05-31T20:39:09.435 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d19acc154d316634-006477b06d-1630-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:39:09 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:39:09.437 DEBUG:boto:b'pklxkd-58obj000000000000000000111000falseobj000000000000000000112023-05-31T20:38:22.712Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:39:09.438 DEBUG:boto:path=/pklxkd-58/obj00000000000000000011 2023-05-31T20:39:09.438 DEBUG:boto:auth_path=/pklxkd-58/obj00000000000000000011 2023-05-31T20:39:09.439 DEBUG:boto:Method: HEAD 2023-05-31T20:39:09.439 DEBUG:boto:Path: /pklxkd-58/obj00000000000000000011 2023-05-31T20:39:09.439 DEBUG:boto:Data: 2023-05-31T20:39:09.440 DEBUG:boto:Headers: {} 2023-05-31T20:39:09.440 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:39:09.440 DEBUG:boto:Port: 8001 2023-05-31T20:39:09.441 DEBUG:boto:Params: {} 2023-05-31T20:39:09.441 DEBUG:boto:Token: None 2023-05-31T20:39:09.442 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:39:09 GMT /pklxkd-58/obj00000000000000000011 2023-05-31T20:39:09.442 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:GiDfJUm21T2vzmRilaRU6YAY10w= 2023-05-31T20:39:09.443 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:39:09 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:GiDfJUm21T2vzmRilaRU6YAY10w=', 'Content-Length': '0'} 2023-05-31T20:39:09.445 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:38:22 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b97bf4ff0c361a28-006477b06d-1c17-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:39:09 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:39:09.446 DEBUG:boto:path=/pklxkd-58/obj00000000000000000011 2023-05-31T20:39:09.447 DEBUG:boto:auth_path=/pklxkd-58/obj00000000000000000011 2023-05-31T20:39:09.447 DEBUG:boto:Method: HEAD 2023-05-31T20:39:09.448 DEBUG:boto:Path: /pklxkd-58/obj00000000000000000011 2023-05-31T20:39:09.448 DEBUG:boto:Data: 2023-05-31T20:39:09.448 DEBUG:boto:Headers: {} 2023-05-31T20:39:09.449 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:39:09.449 DEBUG:boto:Port: 8000 2023-05-31T20:39:09.449 DEBUG:boto:Params: {} 2023-05-31T20:39:09.450 DEBUG:boto:Token: None 2023-05-31T20:39:09.450 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:39:09 GMT /pklxkd-58/obj00000000000000000011 2023-05-31T20:39:09.451 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:GiDfJUm21T2vzmRilaRU6YAY10w= 2023-05-31T20:39:09.451 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:39:09 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:GiDfJUm21T2vzmRilaRU6YAY10w=', 'Content-Length': '0'} 2023-05-31T20:39:09.453 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:38:22 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ddf33bdd9ff55c53-006477b06d-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:39:09 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:39:09.454 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000011 2023-05-31T20:39:09.454 DEBUG:boto:path=/pklxkd-58/obj00000000000000000011 2023-05-31T20:39:09.455 DEBUG:boto:auth_path=/pklxkd-58/obj00000000000000000011 2023-05-31T20:39:09.456 DEBUG:boto:Method: GET 2023-05-31T20:39:09.456 DEBUG:boto:Path: /pklxkd-58/obj00000000000000000011 2023-05-31T20:39:09.457 DEBUG:boto:Data: 2023-05-31T20:39:09.457 DEBUG:boto:Headers: {} 2023-05-31T20:39:09.457 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:39:09.458 DEBUG:boto:Port: 8001 2023-05-31T20:39:09.458 DEBUG:boto:Params: {} 2023-05-31T20:39:09.459 DEBUG:boto:Token: None 2023-05-31T20:39:09.459 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:39:09 GMT /pklxkd-58/obj00000000000000000011 2023-05-31T20:39:09.460 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:D3wD2w4ofeA41Xc1cK7D5B0F5vI= 2023-05-31T20:39:09.460 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:39:09 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:D3wD2w4ofeA41Xc1cK7D5B0F5vI=', 'Content-Length': '0'} 2023-05-31T20:39:09.463 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:38:22 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b7e32034e1253821-006477b06d-1c17-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:39:09 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:39:09.504 DEBUG:boto:path=/pklxkd-58/obj00000000000000000011 2023-05-31T20:39:09.505 DEBUG:boto:auth_path=/pklxkd-58/obj00000000000000000011 2023-05-31T20:39:09.505 DEBUG:boto:Method: GET 2023-05-31T20:39:09.506 DEBUG:boto:Path: /pklxkd-58/obj00000000000000000011 2023-05-31T20:39:09.506 DEBUG:boto:Data: 2023-05-31T20:39:09.506 DEBUG:boto:Headers: {} 2023-05-31T20:39:09.507 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:39:09.507 DEBUG:boto:Port: 8000 2023-05-31T20:39:09.508 DEBUG:boto:Params: {} 2023-05-31T20:39:09.508 DEBUG:boto:Token: None 2023-05-31T20:39:09.508 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:39:09 GMT /pklxkd-58/obj00000000000000000011 2023-05-31T20:39:09.509 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:D3wD2w4ofeA41Xc1cK7D5B0F5vI= 2023-05-31T20:39:09.509 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:39:09 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:D3wD2w4ofeA41Xc1cK7D5B0F5vI=', 'Content-Length': '0'} 2023-05-31T20:39:09.511 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:38:22 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005fca9847dc7e45aa-006477b06d-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:39:09 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:39:09.512 DEBUG:boto:path=/pklxkd-58/ 2023-05-31T20:39:09.512 DEBUG:boto:auth_path=/pklxkd-58/ 2023-05-31T20:39:09.513 DEBUG:boto:path=/pklxkd-58/?prefix=obj00000000000000000011 2023-05-31T20:39:09.513 DEBUG:boto:auth_path=/pklxkd-58/?prefix=obj00000000000000000011 2023-05-31T20:39:09.513 DEBUG:boto:Method: GET 2023-05-31T20:39:09.514 DEBUG:boto:Path: /pklxkd-58/?prefix=obj00000000000000000011 2023-05-31T20:39:09.515 DEBUG:boto:Data: 2023-05-31T20:39:09.516 DEBUG:boto:Headers: {} 2023-05-31T20:39:09.516 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:39:09.517 DEBUG:boto:Port: 8001 2023-05-31T20:39:09.517 DEBUG:boto:Params: {} 2023-05-31T20:39:09.518 DEBUG:boto:Token: None 2023-05-31T20:39:09.519 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:39:09 GMT /pklxkd-58/ 2023-05-31T20:39:09.519 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:gwqyY/Id1F/1VD2W4hCRK4WEzXM= 2023-05-31T20:39:09.520 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:39:09 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:gwqyY/Id1F/1VD2W4hCRK4WEzXM=', 'Content-Length': '0'} 2023-05-31T20:39:09.523 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000084c8cb51a217cdcd-006477b06d-1c17-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:39:09 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:39:09.524 DEBUG:boto:b'pklxkd-58obj000000000000000000111000falseobj000000000000000000112023-05-31T20:38:22.712Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:39:09.524 DEBUG:boto:path=/pklxkd-58/ 2023-05-31T20:39:09.525 DEBUG:boto:auth_path=/pklxkd-58/ 2023-05-31T20:39:09.525 DEBUG:boto:path=/pklxkd-58/?prefix=obj00000000000000000011 2023-05-31T20:39:09.526 DEBUG:boto:auth_path=/pklxkd-58/?prefix=obj00000000000000000011 2023-05-31T20:39:09.526 DEBUG:boto:Method: GET 2023-05-31T20:39:09.526 DEBUG:boto:Path: /pklxkd-58/?prefix=obj00000000000000000011 2023-05-31T20:39:09.527 DEBUG:boto:Data: 2023-05-31T20:39:09.527 DEBUG:boto:Headers: {} 2023-05-31T20:39:09.528 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:39:09.528 DEBUG:boto:Port: 8000 2023-05-31T20:39:09.528 DEBUG:boto:Params: {} 2023-05-31T20:39:09.529 DEBUG:boto:Token: None 2023-05-31T20:39:09.529 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:39:09 GMT /pklxkd-58/ 2023-05-31T20:39:09.530 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:gwqyY/Id1F/1VD2W4hCRK4WEzXM= 2023-05-31T20:39:09.530 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:39:09 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:gwqyY/Id1F/1VD2W4hCRK4WEzXM=', 'Content-Length': '0'} 2023-05-31T20:39:09.534 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000fd7280ab5af4ca2e-006477b06d-1630-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:39:09 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:39:09.535 DEBUG:boto:b'pklxkd-58obj000000000000000000111000falseobj000000000000000000112023-05-31T20:38:22.712Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:39:09.535 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000012 2023-05-31T20:39:09.536 DEBUG:boto:path=/pklxkd-58/obj00000000000000000012 2023-05-31T20:39:09.536 DEBUG:boto:auth_path=/pklxkd-58/obj00000000000000000012 2023-05-31T20:39:09.537 DEBUG:boto:path=/pklxkd-58/obj00000000000000000012?versionId=null 2023-05-31T20:39:09.537 DEBUG:boto:auth_path=/pklxkd-58/obj00000000000000000012?versionId=null 2023-05-31T20:39:09.537 DEBUG:boto:Method: GET 2023-05-31T20:39:09.538 DEBUG:boto:Path: /pklxkd-58/obj00000000000000000012?versionId=null 2023-05-31T20:39:09.538 DEBUG:boto:Data: 2023-05-31T20:39:09.538 DEBUG:boto:Headers: {} 2023-05-31T20:39:09.539 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:39:09.539 DEBUG:boto:Port: 8001 2023-05-31T20:39:09.540 DEBUG:boto:Params: {} 2023-05-31T20:39:09.540 DEBUG:boto:Token: None 2023-05-31T20:39:09.541 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:39:09 GMT /pklxkd-58/obj00000000000000000012?versionId=null 2023-05-31T20:39:09.541 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:gEGRChe2Ya2KrJSi5pBSiX1SnF8= 2023-05-31T20:39:09.541 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:39:09 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:gEGRChe2Ya2KrJSi5pBSiX1SnF8=', 'Content-Length': '0'} 2023-05-31T20:39:09.544 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:38:22 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003069c9232081c212-006477b06d-1c17-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:39:09 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:39:09.544 DEBUG:boto:path=/pklxkd-58/obj00000000000000000012 2023-05-31T20:39:09.545 DEBUG:boto:auth_path=/pklxkd-58/obj00000000000000000012 2023-05-31T20:39:09.545 DEBUG:boto:path=/pklxkd-58/obj00000000000000000012?versionId=null 2023-05-31T20:39:09.546 DEBUG:boto:auth_path=/pklxkd-58/obj00000000000000000012?versionId=null 2023-05-31T20:39:09.546 DEBUG:boto:Method: GET 2023-05-31T20:39:09.547 DEBUG:boto:Path: /pklxkd-58/obj00000000000000000012?versionId=null 2023-05-31T20:39:09.547 DEBUG:boto:Data: 2023-05-31T20:39:09.547 DEBUG:boto:Headers: {} 2023-05-31T20:39:09.548 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:39:09.548 DEBUG:boto:Port: 8000 2023-05-31T20:39:09.548 DEBUG:boto:Params: {} 2023-05-31T20:39:09.549 DEBUG:boto:Token: None 2023-05-31T20:39:09.549 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:39:09 GMT /pklxkd-58/obj00000000000000000012?versionId=null 2023-05-31T20:39:09.550 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:gEGRChe2Ya2KrJSi5pBSiX1SnF8= 2023-05-31T20:39:09.550 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:39:09 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:gEGRChe2Ya2KrJSi5pBSiX1SnF8=', 'Content-Length': '0'} 2023-05-31T20:39:09.552 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:38:22 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000279b18f4509abe73-006477b06d-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:39:09 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:39:09.596 DEBUG:boto:path=/pklxkd-58/obj00000000000000000012 2023-05-31T20:39:09.597 DEBUG:boto:auth_path=/pklxkd-58/obj00000000000000000012 2023-05-31T20:39:09.597 DEBUG:boto:path=/pklxkd-58/obj00000000000000000012?versionId=null 2023-05-31T20:39:09.598 DEBUG:boto:auth_path=/pklxkd-58/obj00000000000000000012?versionId=null 2023-05-31T20:39:09.598 DEBUG:boto:Method: HEAD 2023-05-31T20:39:09.598 DEBUG:boto:Path: /pklxkd-58/obj00000000000000000012?versionId=null 2023-05-31T20:39:09.599 DEBUG:boto:Data: 2023-05-31T20:39:09.599 DEBUG:boto:Headers: {} 2023-05-31T20:39:09.599 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:39:09.600 DEBUG:boto:Port: 8001 2023-05-31T20:39:09.600 DEBUG:boto:Params: {} 2023-05-31T20:39:09.601 DEBUG:boto:Token: None 2023-05-31T20:39:09.601 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:39:09 GMT /pklxkd-58/obj00000000000000000012?versionId=null 2023-05-31T20:39:09.601 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:N60fViPZWihsVYiWVr7ggw/1SiA= 2023-05-31T20:39:09.602 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:39:09 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:N60fViPZWihsVYiWVr7ggw/1SiA=', 'Content-Length': '0'} 2023-05-31T20:39:09.604 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:38:22 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000bd862241ec507cab-006477b06d-1c17-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:39:09 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:39:09.604 DEBUG:boto:path=/pklxkd-58/obj00000000000000000012 2023-05-31T20:39:09.605 DEBUG:boto:auth_path=/pklxkd-58/obj00000000000000000012 2023-05-31T20:39:09.605 DEBUG:boto:path=/pklxkd-58/obj00000000000000000012?versionId=null 2023-05-31T20:39:09.606 DEBUG:boto:auth_path=/pklxkd-58/obj00000000000000000012?versionId=null 2023-05-31T20:39:09.606 DEBUG:boto:Method: HEAD 2023-05-31T20:39:09.606 DEBUG:boto:Path: /pklxkd-58/obj00000000000000000012?versionId=null 2023-05-31T20:39:09.607 DEBUG:boto:Data: 2023-05-31T20:39:09.607 DEBUG:boto:Headers: {} 2023-05-31T20:39:09.607 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:39:09.608 DEBUG:boto:Port: 8000 2023-05-31T20:39:09.608 DEBUG:boto:Params: {} 2023-05-31T20:39:09.608 DEBUG:boto:Token: None 2023-05-31T20:39:09.609 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:39:09 GMT /pklxkd-58/obj00000000000000000012?versionId=null 2023-05-31T20:39:09.609 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:N60fViPZWihsVYiWVr7ggw/1SiA= 2023-05-31T20:39:09.610 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:39:09 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:N60fViPZWihsVYiWVr7ggw/1SiA=', 'Content-Length': '0'} 2023-05-31T20:39:09.611 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:38:22 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000884055e8ce912601-006477b06d-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:39:09 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:39:09.612 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000012 2023-05-31T20:39:09.612 DEBUG:boto:path=/pklxkd-58/obj00000000000000000012 2023-05-31T20:39:09.613 DEBUG:boto:auth_path=/pklxkd-58/obj00000000000000000012 2023-05-31T20:39:09.613 DEBUG:boto:path=/pklxkd-58/obj00000000000000000012?versionId=null 2023-05-31T20:39:09.614 DEBUG:boto:auth_path=/pklxkd-58/obj00000000000000000012?versionId=null 2023-05-31T20:39:09.614 DEBUG:boto:Method: GET 2023-05-31T20:39:09.614 DEBUG:boto:Path: /pklxkd-58/obj00000000000000000012?versionId=null 2023-05-31T20:39:09.615 DEBUG:boto:Data: 2023-05-31T20:39:09.615 DEBUG:boto:Headers: {} 2023-05-31T20:39:09.615 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:39:09.616 DEBUG:boto:Port: 8001 2023-05-31T20:39:09.616 DEBUG:boto:Params: {} 2023-05-31T20:39:09.616 DEBUG:boto:Token: None 2023-05-31T20:39:09.617 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:39:09 GMT /pklxkd-58/obj00000000000000000012?versionId=null 2023-05-31T20:39:09.617 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:gEGRChe2Ya2KrJSi5pBSiX1SnF8= 2023-05-31T20:39:09.618 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:39:09 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:gEGRChe2Ya2KrJSi5pBSiX1SnF8=', 'Content-Length': '0'} 2023-05-31T20:39:09.620 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:38:22 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b5fa35f8a3f48b2e-006477b06d-1c17-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:39:09 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:39:09.621 DEBUG:boto:path=/pklxkd-58/obj00000000000000000012 2023-05-31T20:39:09.621 DEBUG:boto:auth_path=/pklxkd-58/obj00000000000000000012 2023-05-31T20:39:09.621 DEBUG:boto:path=/pklxkd-58/obj00000000000000000012?versionId=null 2023-05-31T20:39:09.622 DEBUG:boto:auth_path=/pklxkd-58/obj00000000000000000012?versionId=null 2023-05-31T20:39:09.622 DEBUG:boto:Method: GET 2023-05-31T20:39:09.622 DEBUG:boto:Path: /pklxkd-58/obj00000000000000000012?versionId=null 2023-05-31T20:39:09.623 DEBUG:boto:Data: 2023-05-31T20:39:09.623 DEBUG:boto:Headers: {} 2023-05-31T20:39:09.624 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:39:09.624 DEBUG:boto:Port: 8000 2023-05-31T20:39:09.624 DEBUG:boto:Params: {} 2023-05-31T20:39:09.625 DEBUG:boto:Token: None 2023-05-31T20:39:09.625 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:39:09 GMT /pklxkd-58/obj00000000000000000012?versionId=null 2023-05-31T20:39:09.626 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:gEGRChe2Ya2KrJSi5pBSiX1SnF8= 2023-05-31T20:39:09.626 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:39:09 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:gEGRChe2Ya2KrJSi5pBSiX1SnF8=', 'Content-Length': '0'} 2023-05-31T20:39:09.628 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:38:22 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d804fc91feb32645-006477b06d-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:39:09 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:39:09.629 DEBUG:boto:path=/pklxkd-58/ 2023-05-31T20:39:09.630 DEBUG:boto:auth_path=/pklxkd-58/ 2023-05-31T20:39:09.630 DEBUG:boto:path=/pklxkd-58/?prefix=obj00000000000000000012 2023-05-31T20:39:09.630 DEBUG:boto:auth_path=/pklxkd-58/?prefix=obj00000000000000000012 2023-05-31T20:39:09.631 DEBUG:boto:Method: GET 2023-05-31T20:39:09.631 DEBUG:boto:Path: /pklxkd-58/?prefix=obj00000000000000000012 2023-05-31T20:39:09.631 DEBUG:boto:Data: 2023-05-31T20:39:09.632 DEBUG:boto:Headers: {} 2023-05-31T20:39:09.632 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:39:09.633 DEBUG:boto:Port: 8001 2023-05-31T20:39:09.633 DEBUG:boto:Params: {} 2023-05-31T20:39:09.633 DEBUG:boto:Token: None 2023-05-31T20:39:09.634 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:39:09 GMT /pklxkd-58/ 2023-05-31T20:39:09.634 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:gwqyY/Id1F/1VD2W4hCRK4WEzXM= 2023-05-31T20:39:09.635 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:39:09 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:gwqyY/Id1F/1VD2W4hCRK4WEzXM=', 'Content-Length': '0'} 2023-05-31T20:39:09.637 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000401e5b2c916f68ba-006477b06d-1c17-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:39:09 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:39:09.638 DEBUG:boto:b'pklxkd-58obj000000000000000000121000falseobj000000000000000000122023-05-31T20:38:22.308Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:39:09.639 DEBUG:boto:path=/pklxkd-58/ 2023-05-31T20:39:09.639 DEBUG:boto:auth_path=/pklxkd-58/ 2023-05-31T20:39:09.639 DEBUG:boto:path=/pklxkd-58/?prefix=obj00000000000000000012 2023-05-31T20:39:09.640 DEBUG:boto:auth_path=/pklxkd-58/?prefix=obj00000000000000000012 2023-05-31T20:39:09.640 DEBUG:boto:Method: GET 2023-05-31T20:39:09.641 DEBUG:boto:Path: /pklxkd-58/?prefix=obj00000000000000000012 2023-05-31T20:39:09.641 DEBUG:boto:Data: 2023-05-31T20:39:09.641 DEBUG:boto:Headers: {} 2023-05-31T20:39:09.642 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:39:09.642 DEBUG:boto:Port: 8000 2023-05-31T20:39:09.642 DEBUG:boto:Params: {} 2023-05-31T20:39:09.643 DEBUG:boto:Token: None 2023-05-31T20:39:09.643 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:39:09 GMT /pklxkd-58/ 2023-05-31T20:39:09.644 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:gwqyY/Id1F/1VD2W4hCRK4WEzXM= 2023-05-31T20:39:09.644 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:39:09 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:gwqyY/Id1F/1VD2W4hCRK4WEzXM=', 'Content-Length': '0'} 2023-05-31T20:39:09.651 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000009ca074cb6228d4d-006477b06d-1630-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:39:09 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:39:09.652 DEBUG:boto:b'pklxkd-58obj000000000000000000121000falseobj000000000000000000122023-05-31T20:38:22.308Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:39:09.653 DEBUG:boto:path=/pklxkd-58/obj00000000000000000012 2023-05-31T20:39:09.654 DEBUG:boto:auth_path=/pklxkd-58/obj00000000000000000012 2023-05-31T20:39:09.654 DEBUG:boto:Method: HEAD 2023-05-31T20:39:09.654 DEBUG:boto:Path: /pklxkd-58/obj00000000000000000012 2023-05-31T20:39:09.655 DEBUG:boto:Data: 2023-05-31T20:39:09.655 DEBUG:boto:Headers: {} 2023-05-31T20:39:09.655 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:39:09.656 DEBUG:boto:Port: 8001 2023-05-31T20:39:09.656 DEBUG:boto:Params: {} 2023-05-31T20:39:09.657 DEBUG:boto:Token: None 2023-05-31T20:39:09.657 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:39:09 GMT /pklxkd-58/obj00000000000000000012 2023-05-31T20:39:09.658 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:hvqnpPGreXCG9oEFMwO8zUKBehU= 2023-05-31T20:39:09.658 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:39:09 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:hvqnpPGreXCG9oEFMwO8zUKBehU=', 'Content-Length': '0'} 2023-05-31T20:39:09.660 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:38:22 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b9b300e5c736c027-006477b06d-1c17-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:39:09 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:39:09.661 DEBUG:boto:path=/pklxkd-58/obj00000000000000000012 2023-05-31T20:39:09.662 DEBUG:boto:auth_path=/pklxkd-58/obj00000000000000000012 2023-05-31T20:39:09.662 DEBUG:boto:Method: HEAD 2023-05-31T20:39:09.662 DEBUG:boto:Path: /pklxkd-58/obj00000000000000000012 2023-05-31T20:39:09.663 DEBUG:boto:Data: 2023-05-31T20:39:09.663 DEBUG:boto:Headers: {} 2023-05-31T20:39:09.663 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:39:09.664 DEBUG:boto:Port: 8000 2023-05-31T20:39:09.664 DEBUG:boto:Params: {} 2023-05-31T20:39:09.665 DEBUG:boto:Token: None 2023-05-31T20:39:09.665 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:39:09 GMT /pklxkd-58/obj00000000000000000012 2023-05-31T20:39:09.665 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:hvqnpPGreXCG9oEFMwO8zUKBehU= 2023-05-31T20:39:09.666 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:39:09 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:hvqnpPGreXCG9oEFMwO8zUKBehU=', 'Content-Length': '0'} 2023-05-31T20:39:09.668 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:38:22 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000050018bcd6afdea90-006477b06d-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:39:09 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:39:09.670 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000012 2023-05-31T20:39:09.671 DEBUG:boto:path=/pklxkd-58/obj00000000000000000012 2023-05-31T20:39:09.671 DEBUG:boto:auth_path=/pklxkd-58/obj00000000000000000012 2023-05-31T20:39:09.672 DEBUG:boto:Method: GET 2023-05-31T20:39:09.672 DEBUG:boto:Path: /pklxkd-58/obj00000000000000000012 2023-05-31T20:39:09.672 DEBUG:boto:Data: 2023-05-31T20:39:09.673 DEBUG:boto:Headers: {} 2023-05-31T20:39:09.673 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:39:09.674 DEBUG:boto:Port: 8001 2023-05-31T20:39:09.674 DEBUG:boto:Params: {} 2023-05-31T20:39:09.674 DEBUG:boto:Token: None 2023-05-31T20:39:09.675 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:39:09 GMT /pklxkd-58/obj00000000000000000012 2023-05-31T20:39:09.675 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:RKA5g4fF7AaXeMXUGzG/hZWRUFs= 2023-05-31T20:39:09.676 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:39:09 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:RKA5g4fF7AaXeMXUGzG/hZWRUFs=', 'Content-Length': '0'} 2023-05-31T20:39:09.678 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:38:22 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d9ec87f987f76aa4-006477b06d-1c17-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:39:09 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:39:09.679 DEBUG:boto:path=/pklxkd-58/obj00000000000000000012 2023-05-31T20:39:09.682 DEBUG:boto:auth_path=/pklxkd-58/obj00000000000000000012 2023-05-31T20:39:09.682 DEBUG:boto:Method: GET 2023-05-31T20:39:09.683 DEBUG:boto:Path: /pklxkd-58/obj00000000000000000012 2023-05-31T20:39:09.683 DEBUG:boto:Data: 2023-05-31T20:39:09.683 DEBUG:boto:Headers: {} 2023-05-31T20:39:09.684 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:39:09.684 DEBUG:boto:Port: 8000 2023-05-31T20:39:09.685 DEBUG:boto:Params: {} 2023-05-31T20:39:09.685 DEBUG:boto:Token: None 2023-05-31T20:39:09.686 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:39:09 GMT /pklxkd-58/obj00000000000000000012 2023-05-31T20:39:09.686 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:RKA5g4fF7AaXeMXUGzG/hZWRUFs= 2023-05-31T20:39:09.687 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:39:09 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:RKA5g4fF7AaXeMXUGzG/hZWRUFs=', 'Content-Length': '0'} 2023-05-31T20:39:09.689 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:38:22 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000090bd7f3ecee66538-006477b06d-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:39:09 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:39:09.736 DEBUG:boto:path=/pklxkd-58/ 2023-05-31T20:39:09.736 DEBUG:boto:auth_path=/pklxkd-58/ 2023-05-31T20:39:09.737 DEBUG:boto:path=/pklxkd-58/?prefix=obj00000000000000000012 2023-05-31T20:39:09.737 DEBUG:boto:auth_path=/pklxkd-58/?prefix=obj00000000000000000012 2023-05-31T20:39:09.738 DEBUG:boto:Method: GET 2023-05-31T20:39:09.738 DEBUG:boto:Path: /pklxkd-58/?prefix=obj00000000000000000012 2023-05-31T20:39:09.738 DEBUG:boto:Data: 2023-05-31T20:39:09.739 DEBUG:boto:Headers: {} 2023-05-31T20:39:09.739 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:39:09.740 DEBUG:boto:Port: 8001 2023-05-31T20:39:09.740 DEBUG:boto:Params: {} 2023-05-31T20:39:09.740 DEBUG:boto:Token: None 2023-05-31T20:39:09.741 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:39:09 GMT /pklxkd-58/ 2023-05-31T20:39:09.741 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:gwqyY/Id1F/1VD2W4hCRK4WEzXM= 2023-05-31T20:39:09.742 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:39:09 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:gwqyY/Id1F/1VD2W4hCRK4WEzXM=', 'Content-Length': '0'} 2023-05-31T20:39:09.745 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c5944109fa26460b-006477b06d-1c17-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:39:09 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:39:09.746 DEBUG:boto:b'pklxkd-58obj000000000000000000121000falseobj000000000000000000122023-05-31T20:38:22.308Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:39:09.747 DEBUG:boto:path=/pklxkd-58/ 2023-05-31T20:39:09.747 DEBUG:boto:auth_path=/pklxkd-58/ 2023-05-31T20:39:09.748 DEBUG:boto:path=/pklxkd-58/?prefix=obj00000000000000000012 2023-05-31T20:39:09.748 DEBUG:boto:auth_path=/pklxkd-58/?prefix=obj00000000000000000012 2023-05-31T20:39:09.748 DEBUG:boto:Method: GET 2023-05-31T20:39:09.749 DEBUG:boto:Path: /pklxkd-58/?prefix=obj00000000000000000012 2023-05-31T20:39:09.749 DEBUG:boto:Data: 2023-05-31T20:39:09.750 DEBUG:boto:Headers: {} 2023-05-31T20:39:09.750 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:39:09.750 DEBUG:boto:Port: 8000 2023-05-31T20:39:09.751 DEBUG:boto:Params: {} 2023-05-31T20:39:09.751 DEBUG:boto:Token: None 2023-05-31T20:39:09.752 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:39:09 GMT /pklxkd-58/ 2023-05-31T20:39:09.752 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:gwqyY/Id1F/1VD2W4hCRK4WEzXM= 2023-05-31T20:39:09.753 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:39:09 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:gwqyY/Id1F/1VD2W4hCRK4WEzXM=', 'Content-Length': '0'} 2023-05-31T20:39:09.757 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000021bdedc1f1916456-006477b06d-1630-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:39:09 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:39:09.758 DEBUG:boto:b'pklxkd-58obj000000000000000000121000falseobj000000000000000000122023-05-31T20:38:22.308Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:39:09.759 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000013 2023-05-31T20:39:09.759 DEBUG:boto:path=/pklxkd-58/obj00000000000000000013 2023-05-31T20:39:09.760 DEBUG:boto:auth_path=/pklxkd-58/obj00000000000000000013 2023-05-31T20:39:09.760 DEBUG:boto:path=/pklxkd-58/obj00000000000000000013?versionId=null 2023-05-31T20:39:09.761 DEBUG:boto:auth_path=/pklxkd-58/obj00000000000000000013?versionId=null 2023-05-31T20:39:09.761 DEBUG:boto:Method: GET 2023-05-31T20:39:09.761 DEBUG:boto:Path: /pklxkd-58/obj00000000000000000013?versionId=null 2023-05-31T20:39:09.762 DEBUG:boto:Data: 2023-05-31T20:39:09.762 DEBUG:boto:Headers: {} 2023-05-31T20:39:09.763 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:39:09.763 DEBUG:boto:Port: 8001 2023-05-31T20:39:09.763 DEBUG:boto:Params: {} 2023-05-31T20:39:09.764 DEBUG:boto:Token: None 2023-05-31T20:39:09.764 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:39:09 GMT /pklxkd-58/obj00000000000000000013?versionId=null 2023-05-31T20:39:09.765 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:C2BWiOt2CRoDWh9yuWEGRJb2+0M= 2023-05-31T20:39:09.765 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:39:09 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:C2BWiOt2CRoDWh9yuWEGRJb2+0M=', 'Content-Length': '0'} 2023-05-31T20:39:09.768 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:38:22 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b0c81547fcbcc155-006477b06d-1c17-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:39:09 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:39:09.808 DEBUG:boto:path=/pklxkd-58/obj00000000000000000013 2023-05-31T20:39:09.809 DEBUG:boto:auth_path=/pklxkd-58/obj00000000000000000013 2023-05-31T20:39:09.810 DEBUG:boto:path=/pklxkd-58/obj00000000000000000013?versionId=null 2023-05-31T20:39:09.810 DEBUG:boto:auth_path=/pklxkd-58/obj00000000000000000013?versionId=null 2023-05-31T20:39:09.810 DEBUG:boto:Method: GET 2023-05-31T20:39:09.811 DEBUG:boto:Path: /pklxkd-58/obj00000000000000000013?versionId=null 2023-05-31T20:39:09.811 DEBUG:boto:Data: 2023-05-31T20:39:09.811 DEBUG:boto:Headers: {} 2023-05-31T20:39:09.812 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:39:09.812 DEBUG:boto:Port: 8000 2023-05-31T20:39:09.813 DEBUG:boto:Params: {} 2023-05-31T20:39:09.813 DEBUG:boto:Token: None 2023-05-31T20:39:09.814 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:39:09 GMT /pklxkd-58/obj00000000000000000013?versionId=null 2023-05-31T20:39:09.814 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:C2BWiOt2CRoDWh9yuWEGRJb2+0M= 2023-05-31T20:39:09.814 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:39:09 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:C2BWiOt2CRoDWh9yuWEGRJb2+0M=', 'Content-Length': '0'} 2023-05-31T20:39:09.817 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:38:22 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000046cac4deb95c1306-006477b06d-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:39:09 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:39:09.818 DEBUG:boto:path=/pklxkd-58/obj00000000000000000013 2023-05-31T20:39:09.818 DEBUG:boto:auth_path=/pklxkd-58/obj00000000000000000013 2023-05-31T20:39:09.818 DEBUG:boto:path=/pklxkd-58/obj00000000000000000013?versionId=null 2023-05-31T20:39:09.819 DEBUG:boto:auth_path=/pklxkd-58/obj00000000000000000013?versionId=null 2023-05-31T20:39:09.819 DEBUG:boto:Method: HEAD 2023-05-31T20:39:09.820 DEBUG:boto:Path: /pklxkd-58/obj00000000000000000013?versionId=null 2023-05-31T20:39:09.820 DEBUG:boto:Data: 2023-05-31T20:39:09.820 DEBUG:boto:Headers: {} 2023-05-31T20:39:09.821 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:39:09.821 DEBUG:boto:Port: 8001 2023-05-31T20:39:09.821 DEBUG:boto:Params: {} 2023-05-31T20:39:09.822 DEBUG:boto:Token: None 2023-05-31T20:39:09.822 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:39:09 GMT /pklxkd-58/obj00000000000000000013?versionId=null 2023-05-31T20:39:09.823 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:dGqnKB7srpbeEFHm1tiv+rT5cQs= 2023-05-31T20:39:09.823 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:39:09 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:dGqnKB7srpbeEFHm1tiv+rT5cQs=', 'Content-Length': '0'} 2023-05-31T20:39:09.825 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:38:22 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007d3d8c96b2bdb15b-006477b06d-1c17-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:39:09 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:39:09.826 DEBUG:boto:path=/pklxkd-58/obj00000000000000000013 2023-05-31T20:39:09.827 DEBUG:boto:auth_path=/pklxkd-58/obj00000000000000000013 2023-05-31T20:39:09.828 DEBUG:boto:path=/pklxkd-58/obj00000000000000000013?versionId=null 2023-05-31T20:39:09.829 DEBUG:boto:auth_path=/pklxkd-58/obj00000000000000000013?versionId=null 2023-05-31T20:39:09.830 DEBUG:boto:Method: HEAD 2023-05-31T20:39:09.830 DEBUG:boto:Path: /pklxkd-58/obj00000000000000000013?versionId=null 2023-05-31T20:39:09.831 DEBUG:boto:Data: 2023-05-31T20:39:09.832 DEBUG:boto:Headers: {} 2023-05-31T20:39:09.834 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:39:09.835 DEBUG:boto:Port: 8000 2023-05-31T20:39:09.836 DEBUG:boto:Params: {} 2023-05-31T20:39:09.836 DEBUG:boto:Token: None 2023-05-31T20:39:09.837 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:39:09 GMT /pklxkd-58/obj00000000000000000013?versionId=null 2023-05-31T20:39:09.838 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:dGqnKB7srpbeEFHm1tiv+rT5cQs= 2023-05-31T20:39:09.839 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:39:09 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:dGqnKB7srpbeEFHm1tiv+rT5cQs=', 'Content-Length': '0'} 2023-05-31T20:39:09.840 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:38:22 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000072d2473fcb401685-006477b06d-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:39:09 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:39:09.841 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000013 2023-05-31T20:39:09.842 DEBUG:boto:path=/pklxkd-58/obj00000000000000000013 2023-05-31T20:39:09.842 DEBUG:boto:auth_path=/pklxkd-58/obj00000000000000000013 2023-05-31T20:39:09.842 DEBUG:boto:path=/pklxkd-58/obj00000000000000000013?versionId=null 2023-05-31T20:39:09.843 DEBUG:boto:auth_path=/pklxkd-58/obj00000000000000000013?versionId=null 2023-05-31T20:39:09.843 DEBUG:boto:Method: GET 2023-05-31T20:39:09.844 DEBUG:boto:Path: /pklxkd-58/obj00000000000000000013?versionId=null 2023-05-31T20:39:09.845 DEBUG:boto:Data: 2023-05-31T20:39:09.845 DEBUG:boto:Headers: {} 2023-05-31T20:39:09.846 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:39:09.846 DEBUG:boto:Port: 8001 2023-05-31T20:39:09.846 DEBUG:boto:Params: {} 2023-05-31T20:39:09.847 DEBUG:boto:Token: None 2023-05-31T20:39:09.848 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:39:09 GMT /pklxkd-58/obj00000000000000000013?versionId=null 2023-05-31T20:39:09.849 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:C2BWiOt2CRoDWh9yuWEGRJb2+0M= 2023-05-31T20:39:09.849 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:39:09 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:C2BWiOt2CRoDWh9yuWEGRJb2+0M=', 'Content-Length': '0'} 2023-05-31T20:39:09.852 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:38:22 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000046a0a5549f0b56f7-006477b06d-1c17-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:39:09 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:39:09.853 DEBUG:boto:path=/pklxkd-58/obj00000000000000000013 2023-05-31T20:39:09.854 DEBUG:boto:auth_path=/pklxkd-58/obj00000000000000000013 2023-05-31T20:39:09.854 DEBUG:boto:path=/pklxkd-58/obj00000000000000000013?versionId=null 2023-05-31T20:39:09.855 DEBUG:boto:auth_path=/pklxkd-58/obj00000000000000000013?versionId=null 2023-05-31T20:39:09.856 DEBUG:boto:Method: GET 2023-05-31T20:39:09.857 DEBUG:boto:Path: /pklxkd-58/obj00000000000000000013?versionId=null 2023-05-31T20:39:09.858 DEBUG:boto:Data: 2023-05-31T20:39:09.859 DEBUG:boto:Headers: {} 2023-05-31T20:39:09.860 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:39:09.861 DEBUG:boto:Port: 8000 2023-05-31T20:39:09.861 DEBUG:boto:Params: {} 2023-05-31T20:39:09.861 DEBUG:boto:Token: None 2023-05-31T20:39:09.862 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:39:09 GMT /pklxkd-58/obj00000000000000000013?versionId=null 2023-05-31T20:39:09.863 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:C2BWiOt2CRoDWh9yuWEGRJb2+0M= 2023-05-31T20:39:09.863 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:39:09 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:C2BWiOt2CRoDWh9yuWEGRJb2+0M=', 'Content-Length': '0'} 2023-05-31T20:39:09.866 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:38:22 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000feb14622d6260680-006477b06d-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:39:09 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:39:09.908 DEBUG:boto:path=/pklxkd-58/ 2023-05-31T20:39:09.910 DEBUG:boto:auth_path=/pklxkd-58/ 2023-05-31T20:39:09.910 DEBUG:boto:path=/pklxkd-58/?prefix=obj00000000000000000013 2023-05-31T20:39:09.911 DEBUG:boto:auth_path=/pklxkd-58/?prefix=obj00000000000000000013 2023-05-31T20:39:09.912 DEBUG:boto:Method: GET 2023-05-31T20:39:09.912 DEBUG:boto:Path: /pklxkd-58/?prefix=obj00000000000000000013 2023-05-31T20:39:09.913 DEBUG:boto:Data: 2023-05-31T20:39:09.913 DEBUG:boto:Headers: {} 2023-05-31T20:39:09.914 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:39:09.915 DEBUG:boto:Port: 8001 2023-05-31T20:39:09.915 DEBUG:boto:Params: {} 2023-05-31T20:39:09.916 DEBUG:boto:Token: None 2023-05-31T20:39:09.916 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:39:09 GMT /pklxkd-58/ 2023-05-31T20:39:09.917 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:gwqyY/Id1F/1VD2W4hCRK4WEzXM= 2023-05-31T20:39:09.917 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:39:09 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:gwqyY/Id1F/1VD2W4hCRK4WEzXM=', 'Content-Length': '0'} 2023-05-31T20:39:09.920 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000350a8ffe3cbd94c5-006477b06d-1c17-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:39:09 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:39:09.921 DEBUG:boto:b'pklxkd-58obj000000000000000000131000falseobj000000000000000000132023-05-31T20:38:22.736Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:39:09.922 DEBUG:boto:path=/pklxkd-58/ 2023-05-31T20:39:09.922 DEBUG:boto:auth_path=/pklxkd-58/ 2023-05-31T20:39:09.923 DEBUG:boto:path=/pklxkd-58/?prefix=obj00000000000000000013 2023-05-31T20:39:09.923 DEBUG:boto:auth_path=/pklxkd-58/?prefix=obj00000000000000000013 2023-05-31T20:39:09.927 DEBUG:boto:Method: GET 2023-05-31T20:39:09.927 DEBUG:boto:Path: /pklxkd-58/?prefix=obj00000000000000000013 2023-05-31T20:39:09.928 DEBUG:boto:Data: 2023-05-31T20:39:09.928 DEBUG:boto:Headers: {} 2023-05-31T20:39:09.929 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:39:09.929 DEBUG:boto:Port: 8000 2023-05-31T20:39:09.930 DEBUG:boto:Params: {} 2023-05-31T20:39:09.930 DEBUG:boto:Token: None 2023-05-31T20:39:09.931 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:39:09 GMT /pklxkd-58/ 2023-05-31T20:39:09.931 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:gwqyY/Id1F/1VD2W4hCRK4WEzXM= 2023-05-31T20:39:09.931 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:39:09 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:gwqyY/Id1F/1VD2W4hCRK4WEzXM=', 'Content-Length': '0'} 2023-05-31T20:39:09.937 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000011b8bd1a282b4b57-006477b06d-1630-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:39:09 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:39:09.937 DEBUG:boto:b'pklxkd-58obj000000000000000000131000falseobj000000000000000000132023-05-31T20:38:22.736Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:39:09.938 DEBUG:boto:path=/pklxkd-58/obj00000000000000000013 2023-05-31T20:39:09.938 DEBUG:boto:auth_path=/pklxkd-58/obj00000000000000000013 2023-05-31T20:39:09.939 DEBUG:boto:Method: HEAD 2023-05-31T20:39:09.939 DEBUG:boto:Path: /pklxkd-58/obj00000000000000000013 2023-05-31T20:39:09.939 DEBUG:boto:Data: 2023-05-31T20:39:09.940 DEBUG:boto:Headers: {} 2023-05-31T20:39:09.940 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:39:09.940 DEBUG:boto:Port: 8001 2023-05-31T20:39:09.941 DEBUG:boto:Params: {} 2023-05-31T20:39:09.941 DEBUG:boto:Token: None 2023-05-31T20:39:09.942 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:39:09 GMT /pklxkd-58/obj00000000000000000013 2023-05-31T20:39:09.942 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:Ql2AZeZ6ZMnkc1+7S8okgzDRsVM= 2023-05-31T20:39:09.942 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:39:09 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:Ql2AZeZ6ZMnkc1+7S8okgzDRsVM=', 'Content-Length': '0'} 2023-05-31T20:39:09.944 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:38:22 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000028449f23414595b7-006477b06d-1c17-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:39:09 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:39:09.945 DEBUG:boto:path=/pklxkd-58/obj00000000000000000013 2023-05-31T20:39:09.945 DEBUG:boto:auth_path=/pklxkd-58/obj00000000000000000013 2023-05-31T20:39:09.946 DEBUG:boto:Method: HEAD 2023-05-31T20:39:09.946 DEBUG:boto:Path: /pklxkd-58/obj00000000000000000013 2023-05-31T20:39:09.946 DEBUG:boto:Data: 2023-05-31T20:39:09.947 DEBUG:boto:Headers: {} 2023-05-31T20:39:09.947 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:39:09.947 DEBUG:boto:Port: 8000 2023-05-31T20:39:09.948 DEBUG:boto:Params: {} 2023-05-31T20:39:09.948 DEBUG:boto:Token: None 2023-05-31T20:39:09.949 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:39:09 GMT /pklxkd-58/obj00000000000000000013 2023-05-31T20:39:09.949 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:Ql2AZeZ6ZMnkc1+7S8okgzDRsVM= 2023-05-31T20:39:09.949 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:39:09 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:Ql2AZeZ6ZMnkc1+7S8okgzDRsVM=', 'Content-Length': '0'} 2023-05-31T20:39:09.951 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:38:22 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b024e2b84ab0b4b3-006477b06d-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:39:09 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:39:09.952 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000013 2023-05-31T20:39:09.952 DEBUG:boto:path=/pklxkd-58/obj00000000000000000013 2023-05-31T20:39:09.952 DEBUG:boto:auth_path=/pklxkd-58/obj00000000000000000013 2023-05-31T20:39:09.953 DEBUG:boto:Method: GET 2023-05-31T20:39:09.953 DEBUG:boto:Path: /pklxkd-58/obj00000000000000000013 2023-05-31T20:39:09.954 DEBUG:boto:Data: 2023-05-31T20:39:09.954 DEBUG:boto:Headers: {} 2023-05-31T20:39:09.954 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:39:09.955 DEBUG:boto:Port: 8001 2023-05-31T20:39:09.955 DEBUG:boto:Params: {} 2023-05-31T20:39:09.955 DEBUG:boto:Token: None 2023-05-31T20:39:09.956 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:39:09 GMT /pklxkd-58/obj00000000000000000013 2023-05-31T20:39:09.956 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:47j209FZaqGoaKgpfShudc5pw4Q= 2023-05-31T20:39:09.957 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:39:09 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:47j209FZaqGoaKgpfShudc5pw4Q=', 'Content-Length': '0'} 2023-05-31T20:39:09.959 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:38:22 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000afa3a57302fa6d97-006477b06d-1c17-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:39:09 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:39:10.000 DEBUG:boto:path=/pklxkd-58/obj00000000000000000013 2023-05-31T20:39:10.001 DEBUG:boto:auth_path=/pklxkd-58/obj00000000000000000013 2023-05-31T20:39:10.002 DEBUG:boto:Method: GET 2023-05-31T20:39:10.002 DEBUG:boto:Path: /pklxkd-58/obj00000000000000000013 2023-05-31T20:39:10.002 DEBUG:boto:Data: 2023-05-31T20:39:10.003 DEBUG:boto:Headers: {} 2023-05-31T20:39:10.003 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:39:10.003 DEBUG:boto:Port: 8000 2023-05-31T20:39:10.004 DEBUG:boto:Params: {} 2023-05-31T20:39:10.004 DEBUG:boto:Token: None 2023-05-31T20:39:10.005 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:39:10 GMT /pklxkd-58/obj00000000000000000013 2023-05-31T20:39:10.005 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:vwl3R+6BvX+q1CMlAYwhRcGrn5U= 2023-05-31T20:39:10.006 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:39:10 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:vwl3R+6BvX+q1CMlAYwhRcGrn5U=', 'Content-Length': '0'} 2023-05-31T20:39:10.008 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:38:22 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000048fd3d00af6c351b-006477b06e-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:39:10 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:39:10.009 DEBUG:boto:path=/pklxkd-58/ 2023-05-31T20:39:10.010 DEBUG:boto:auth_path=/pklxkd-58/ 2023-05-31T20:39:10.010 DEBUG:boto:path=/pklxkd-58/?prefix=obj00000000000000000013 2023-05-31T20:39:10.010 DEBUG:boto:auth_path=/pklxkd-58/?prefix=obj00000000000000000013 2023-05-31T20:39:10.011 DEBUG:boto:Method: GET 2023-05-31T20:39:10.011 DEBUG:boto:Path: /pklxkd-58/?prefix=obj00000000000000000013 2023-05-31T20:39:10.012 DEBUG:boto:Data: 2023-05-31T20:39:10.012 DEBUG:boto:Headers: {} 2023-05-31T20:39:10.013 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:39:10.013 DEBUG:boto:Port: 8001 2023-05-31T20:39:10.013 DEBUG:boto:Params: {} 2023-05-31T20:39:10.014 DEBUG:boto:Token: None 2023-05-31T20:39:10.014 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:39:10 GMT /pklxkd-58/ 2023-05-31T20:39:10.015 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:rToF82M8NN6MyNReCsDOOu4EGBY= 2023-05-31T20:39:10.015 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:39:10 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:rToF82M8NN6MyNReCsDOOu4EGBY=', 'Content-Length': '0'} 2023-05-31T20:39:10.019 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000034894d697250e85-006477b06e-1c17-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:39:10 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:39:10.019 DEBUG:boto:b'pklxkd-58obj000000000000000000131000falseobj000000000000000000132023-05-31T20:38:22.736Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:39:10.020 DEBUG:boto:path=/pklxkd-58/ 2023-05-31T20:39:10.021 DEBUG:boto:auth_path=/pklxkd-58/ 2023-05-31T20:39:10.021 DEBUG:boto:path=/pklxkd-58/?prefix=obj00000000000000000013 2023-05-31T20:39:10.021 DEBUG:boto:auth_path=/pklxkd-58/?prefix=obj00000000000000000013 2023-05-31T20:39:10.022 DEBUG:boto:Method: GET 2023-05-31T20:39:10.022 DEBUG:boto:Path: /pklxkd-58/?prefix=obj00000000000000000013 2023-05-31T20:39:10.023 DEBUG:boto:Data: 2023-05-31T20:39:10.023 DEBUG:boto:Headers: {} 2023-05-31T20:39:10.023 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:39:10.024 DEBUG:boto:Port: 8000 2023-05-31T20:39:10.025 DEBUG:boto:Params: {} 2023-05-31T20:39:10.026 DEBUG:boto:Token: None 2023-05-31T20:39:10.027 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:39:10 GMT /pklxkd-58/ 2023-05-31T20:39:10.027 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:rToF82M8NN6MyNReCsDOOu4EGBY= 2023-05-31T20:39:10.028 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:39:10 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:rToF82M8NN6MyNReCsDOOu4EGBY=', 'Content-Length': '0'} 2023-05-31T20:39:10.032 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000003790a1671267191-006477b06e-1630-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:39:10 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:39:10.033 DEBUG:boto:b'pklxkd-58obj000000000000000000131000falseobj000000000000000000132023-05-31T20:38:22.736Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:39:10.034 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000014 2023-05-31T20:39:10.034 DEBUG:boto:path=/pklxkd-58/obj00000000000000000014 2023-05-31T20:39:10.034 DEBUG:boto:auth_path=/pklxkd-58/obj00000000000000000014 2023-05-31T20:39:10.035 DEBUG:boto:path=/pklxkd-58/obj00000000000000000014?versionId=null 2023-05-31T20:39:10.035 DEBUG:boto:auth_path=/pklxkd-58/obj00000000000000000014?versionId=null 2023-05-31T20:39:10.036 DEBUG:boto:Method: GET 2023-05-31T20:39:10.036 DEBUG:boto:Path: /pklxkd-58/obj00000000000000000014?versionId=null 2023-05-31T20:39:10.037 DEBUG:boto:Data: 2023-05-31T20:39:10.037 DEBUG:boto:Headers: {} 2023-05-31T20:39:10.037 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:39:10.038 DEBUG:boto:Port: 8001 2023-05-31T20:39:10.038 DEBUG:boto:Params: {} 2023-05-31T20:39:10.039 DEBUG:boto:Token: None 2023-05-31T20:39:10.039 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:39:10 GMT /pklxkd-58/obj00000000000000000014?versionId=null 2023-05-31T20:39:10.040 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:tdOv9ITi5K7lnU2BjEVqH1BFYok= 2023-05-31T20:39:10.041 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:39:10 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:tdOv9ITi5K7lnU2BjEVqH1BFYok=', 'Content-Length': '0'} 2023-05-31T20:39:10.043 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:38:22 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006d2c12949c4114a6-006477b06e-1c17-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:39:10 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:39:10.044 DEBUG:boto:path=/pklxkd-58/obj00000000000000000014 2023-05-31T20:39:10.044 DEBUG:boto:auth_path=/pklxkd-58/obj00000000000000000014 2023-05-31T20:39:10.045 DEBUG:boto:path=/pklxkd-58/obj00000000000000000014?versionId=null 2023-05-31T20:39:10.046 DEBUG:boto:auth_path=/pklxkd-58/obj00000000000000000014?versionId=null 2023-05-31T20:39:10.046 DEBUG:boto:Method: GET 2023-05-31T20:39:10.046 DEBUG:boto:Path: /pklxkd-58/obj00000000000000000014?versionId=null 2023-05-31T20:39:10.047 DEBUG:boto:Data: 2023-05-31T20:39:10.047 DEBUG:boto:Headers: {} 2023-05-31T20:39:10.048 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:39:10.048 DEBUG:boto:Port: 8000 2023-05-31T20:39:10.048 DEBUG:boto:Params: {} 2023-05-31T20:39:10.049 DEBUG:boto:Token: None 2023-05-31T20:39:10.049 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:39:10 GMT /pklxkd-58/obj00000000000000000014?versionId=null 2023-05-31T20:39:10.050 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:tdOv9ITi5K7lnU2BjEVqH1BFYok= 2023-05-31T20:39:10.050 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:39:10 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:tdOv9ITi5K7lnU2BjEVqH1BFYok=', 'Content-Length': '0'} 2023-05-31T20:39:10.053 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:38:22 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006161fcb24aa3cd95-006477b06e-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:39:10 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:39:10.096 DEBUG:boto:path=/pklxkd-58/obj00000000000000000014 2023-05-31T20:39:10.097 DEBUG:boto:auth_path=/pklxkd-58/obj00000000000000000014 2023-05-31T20:39:10.098 DEBUG:boto:path=/pklxkd-58/obj00000000000000000014?versionId=null 2023-05-31T20:39:10.098 DEBUG:boto:auth_path=/pklxkd-58/obj00000000000000000014?versionId=null 2023-05-31T20:39:10.098 DEBUG:boto:Method: HEAD 2023-05-31T20:39:10.099 DEBUG:boto:Path: /pklxkd-58/obj00000000000000000014?versionId=null 2023-05-31T20:39:10.099 DEBUG:boto:Data: 2023-05-31T20:39:10.099 DEBUG:boto:Headers: {} 2023-05-31T20:39:10.100 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:39:10.100 DEBUG:boto:Port: 8001 2023-05-31T20:39:10.100 DEBUG:boto:Params: {} 2023-05-31T20:39:10.101 DEBUG:boto:Token: None 2023-05-31T20:39:10.101 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:39:10 GMT /pklxkd-58/obj00000000000000000014?versionId=null 2023-05-31T20:39:10.102 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:IMMSAfMtUTzFCYpc6WDpUkSkvSw= 2023-05-31T20:39:10.102 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:39:10 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:IMMSAfMtUTzFCYpc6WDpUkSkvSw=', 'Content-Length': '0'} 2023-05-31T20:39:10.104 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:38:22 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007caf57bf449823fe-006477b06e-1c17-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:39:10 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:39:10.106 DEBUG:boto:path=/pklxkd-58/obj00000000000000000014 2023-05-31T20:39:10.106 DEBUG:boto:auth_path=/pklxkd-58/obj00000000000000000014 2023-05-31T20:39:10.107 DEBUG:boto:path=/pklxkd-58/obj00000000000000000014?versionId=null 2023-05-31T20:39:10.108 DEBUG:boto:auth_path=/pklxkd-58/obj00000000000000000014?versionId=null 2023-05-31T20:39:10.108 DEBUG:boto:Method: HEAD 2023-05-31T20:39:10.109 DEBUG:boto:Path: /pklxkd-58/obj00000000000000000014?versionId=null 2023-05-31T20:39:10.110 DEBUG:boto:Data: 2023-05-31T20:39:10.110 DEBUG:boto:Headers: {} 2023-05-31T20:39:10.111 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:39:10.112 DEBUG:boto:Port: 8000 2023-05-31T20:39:10.113 DEBUG:boto:Params: {} 2023-05-31T20:39:10.114 DEBUG:boto:Token: None 2023-05-31T20:39:10.115 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:39:10 GMT /pklxkd-58/obj00000000000000000014?versionId=null 2023-05-31T20:39:10.116 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:IMMSAfMtUTzFCYpc6WDpUkSkvSw= 2023-05-31T20:39:10.117 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:39:10 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:IMMSAfMtUTzFCYpc6WDpUkSkvSw=', 'Content-Length': '0'} 2023-05-31T20:39:10.119 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:38:22 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000efeeaabefc60e074-006477b06e-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:39:10 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:39:10.120 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000014 2023-05-31T20:39:10.120 DEBUG:boto:path=/pklxkd-58/obj00000000000000000014 2023-05-31T20:39:10.121 DEBUG:boto:auth_path=/pklxkd-58/obj00000000000000000014 2023-05-31T20:39:10.121 DEBUG:boto:path=/pklxkd-58/obj00000000000000000014?versionId=null 2023-05-31T20:39:10.121 DEBUG:boto:auth_path=/pklxkd-58/obj00000000000000000014?versionId=null 2023-05-31T20:39:10.122 DEBUG:boto:Method: GET 2023-05-31T20:39:10.122 DEBUG:boto:Path: /pklxkd-58/obj00000000000000000014?versionId=null 2023-05-31T20:39:10.122 DEBUG:boto:Data: 2023-05-31T20:39:10.123 DEBUG:boto:Headers: {} 2023-05-31T20:39:10.123 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:39:10.123 DEBUG:boto:Port: 8001 2023-05-31T20:39:10.124 DEBUG:boto:Params: {} 2023-05-31T20:39:10.124 DEBUG:boto:Token: None 2023-05-31T20:39:10.125 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:39:10 GMT /pklxkd-58/obj00000000000000000014?versionId=null 2023-05-31T20:39:10.125 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:tdOv9ITi5K7lnU2BjEVqH1BFYok= 2023-05-31T20:39:10.126 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:39:10 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:tdOv9ITi5K7lnU2BjEVqH1BFYok=', 'Content-Length': '0'} 2023-05-31T20:39:10.128 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:38:22 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000fcea4bc2008201a3-006477b06e-1c17-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:39:10 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:39:10.129 DEBUG:boto:path=/pklxkd-58/obj00000000000000000014 2023-05-31T20:39:10.129 DEBUG:boto:auth_path=/pklxkd-58/obj00000000000000000014 2023-05-31T20:39:10.129 DEBUG:boto:path=/pklxkd-58/obj00000000000000000014?versionId=null 2023-05-31T20:39:10.130 DEBUG:boto:auth_path=/pklxkd-58/obj00000000000000000014?versionId=null 2023-05-31T20:39:10.130 DEBUG:boto:Method: GET 2023-05-31T20:39:10.130 DEBUG:boto:Path: /pklxkd-58/obj00000000000000000014?versionId=null 2023-05-31T20:39:10.131 DEBUG:boto:Data: 2023-05-31T20:39:10.131 DEBUG:boto:Headers: {} 2023-05-31T20:39:10.132 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:39:10.132 DEBUG:boto:Port: 8000 2023-05-31T20:39:10.132 DEBUG:boto:Params: {} 2023-05-31T20:39:10.133 DEBUG:boto:Token: None 2023-05-31T20:39:10.133 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:39:10 GMT /pklxkd-58/obj00000000000000000014?versionId=null 2023-05-31T20:39:10.134 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:tdOv9ITi5K7lnU2BjEVqH1BFYok= 2023-05-31T20:39:10.134 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:39:10 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:tdOv9ITi5K7lnU2BjEVqH1BFYok=', 'Content-Length': '0'} 2023-05-31T20:39:10.136 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:38:22 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000fbe61bba90b88993-006477b06e-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:39:10 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:39:10.137 DEBUG:boto:path=/pklxkd-58/ 2023-05-31T20:39:10.137 DEBUG:boto:auth_path=/pklxkd-58/ 2023-05-31T20:39:10.138 DEBUG:boto:path=/pklxkd-58/?prefix=obj00000000000000000014 2023-05-31T20:39:10.138 DEBUG:boto:auth_path=/pklxkd-58/?prefix=obj00000000000000000014 2023-05-31T20:39:10.138 DEBUG:boto:Method: GET 2023-05-31T20:39:10.139 DEBUG:boto:Path: /pklxkd-58/?prefix=obj00000000000000000014 2023-05-31T20:39:10.139 DEBUG:boto:Data: 2023-05-31T20:39:10.140 DEBUG:boto:Headers: {} 2023-05-31T20:39:10.140 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:39:10.140 DEBUG:boto:Port: 8001 2023-05-31T20:39:10.141 DEBUG:boto:Params: {} 2023-05-31T20:39:10.141 DEBUG:boto:Token: None 2023-05-31T20:39:10.142 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:39:10 GMT /pklxkd-58/ 2023-05-31T20:39:10.142 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:rToF82M8NN6MyNReCsDOOu4EGBY= 2023-05-31T20:39:10.143 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:39:10 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:rToF82M8NN6MyNReCsDOOu4EGBY=', 'Content-Length': '0'} 2023-05-31T20:39:10.145 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a6379dd581da97e6-006477b06e-1c17-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:39:10 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:39:10.146 DEBUG:boto:b'pklxkd-58obj000000000000000000141000falseobj000000000000000000142023-05-31T20:38:22.896Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:39:10.147 DEBUG:boto:path=/pklxkd-58/ 2023-05-31T20:39:10.147 DEBUG:boto:auth_path=/pklxkd-58/ 2023-05-31T20:39:10.147 DEBUG:boto:path=/pklxkd-58/?prefix=obj00000000000000000014 2023-05-31T20:39:10.148 DEBUG:boto:auth_path=/pklxkd-58/?prefix=obj00000000000000000014 2023-05-31T20:39:10.148 DEBUG:boto:Method: GET 2023-05-31T20:39:10.149 DEBUG:boto:Path: /pklxkd-58/?prefix=obj00000000000000000014 2023-05-31T20:39:10.150 DEBUG:boto:Data: 2023-05-31T20:39:10.150 DEBUG:boto:Headers: {} 2023-05-31T20:39:10.150 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:39:10.151 DEBUG:boto:Port: 8000 2023-05-31T20:39:10.151 DEBUG:boto:Params: {} 2023-05-31T20:39:10.152 DEBUG:boto:Token: None 2023-05-31T20:39:10.152 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:39:10 GMT /pklxkd-58/ 2023-05-31T20:39:10.153 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:rToF82M8NN6MyNReCsDOOu4EGBY= 2023-05-31T20:39:10.153 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:39:10 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:rToF82M8NN6MyNReCsDOOu4EGBY=', 'Content-Length': '0'} 2023-05-31T20:39:10.158 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d8bc12f3641383b0-006477b06e-1630-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:39:10 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:39:10.159 DEBUG:boto:b'pklxkd-58obj000000000000000000141000falseobj000000000000000000142023-05-31T20:38:22.896Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:39:10.160 DEBUG:boto:path=/pklxkd-58/obj00000000000000000014 2023-05-31T20:39:10.161 DEBUG:boto:auth_path=/pklxkd-58/obj00000000000000000014 2023-05-31T20:39:10.161 DEBUG:boto:Method: HEAD 2023-05-31T20:39:10.161 DEBUG:boto:Path: /pklxkd-58/obj00000000000000000014 2023-05-31T20:39:10.162 DEBUG:boto:Data: 2023-05-31T20:39:10.162 DEBUG:boto:Headers: {} 2023-05-31T20:39:10.162 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:39:10.163 DEBUG:boto:Port: 8001 2023-05-31T20:39:10.163 DEBUG:boto:Params: {} 2023-05-31T20:39:10.163 DEBUG:boto:Token: None 2023-05-31T20:39:10.164 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:39:10 GMT /pklxkd-58/obj00000000000000000014 2023-05-31T20:39:10.164 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:P20PY4dHuJ4B5xb5UZk22jTa6rI= 2023-05-31T20:39:10.165 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:39:10 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:P20PY4dHuJ4B5xb5UZk22jTa6rI=', 'Content-Length': '0'} 2023-05-31T20:39:10.167 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:38:22 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000058766957aa23dcc6-006477b06e-1c17-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:39:10 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:39:10.167 DEBUG:boto:path=/pklxkd-58/obj00000000000000000014 2023-05-31T20:39:10.168 DEBUG:boto:auth_path=/pklxkd-58/obj00000000000000000014 2023-05-31T20:39:10.168 DEBUG:boto:Method: HEAD 2023-05-31T20:39:10.168 DEBUG:boto:Path: /pklxkd-58/obj00000000000000000014 2023-05-31T20:39:10.169 DEBUG:boto:Data: 2023-05-31T20:39:10.169 DEBUG:boto:Headers: {} 2023-05-31T20:39:10.169 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:39:10.170 DEBUG:boto:Port: 8000 2023-05-31T20:39:10.170 DEBUG:boto:Params: {} 2023-05-31T20:39:10.170 DEBUG:boto:Token: None 2023-05-31T20:39:10.171 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:39:10 GMT /pklxkd-58/obj00000000000000000014 2023-05-31T20:39:10.171 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:P20PY4dHuJ4B5xb5UZk22jTa6rI= 2023-05-31T20:39:10.172 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:39:10 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:P20PY4dHuJ4B5xb5UZk22jTa6rI=', 'Content-Length': '0'} 2023-05-31T20:39:10.174 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:38:22 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000039811c1630b26049-006477b06e-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:39:10 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:39:10.174 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000014 2023-05-31T20:39:10.175 DEBUG:boto:path=/pklxkd-58/obj00000000000000000014 2023-05-31T20:39:10.175 DEBUG:boto:auth_path=/pklxkd-58/obj00000000000000000014 2023-05-31T20:39:10.175 DEBUG:boto:Method: GET 2023-05-31T20:39:10.176 DEBUG:boto:Path: /pklxkd-58/obj00000000000000000014 2023-05-31T20:39:10.176 DEBUG:boto:Data: 2023-05-31T20:39:10.177 DEBUG:boto:Headers: {} 2023-05-31T20:39:10.177 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:39:10.177 DEBUG:boto:Port: 8001 2023-05-31T20:39:10.178 DEBUG:boto:Params: {} 2023-05-31T20:39:10.178 DEBUG:boto:Token: None 2023-05-31T20:39:10.179 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:39:10 GMT /pklxkd-58/obj00000000000000000014 2023-05-31T20:39:10.179 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:4zdTHQwqq0TvFlL9FfC5PbiiEO8= 2023-05-31T20:39:10.179 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:39:10 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:4zdTHQwqq0TvFlL9FfC5PbiiEO8=', 'Content-Length': '0'} 2023-05-31T20:39:10.181 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:38:22 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000233691f69a77fc2b-006477b06e-1c17-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:39:10 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:39:10.224 DEBUG:boto:path=/pklxkd-58/obj00000000000000000014 2023-05-31T20:39:10.225 DEBUG:boto:auth_path=/pklxkd-58/obj00000000000000000014 2023-05-31T20:39:10.225 DEBUG:boto:Method: GET 2023-05-31T20:39:10.226 DEBUG:boto:Path: /pklxkd-58/obj00000000000000000014 2023-05-31T20:39:10.226 DEBUG:boto:Data: 2023-05-31T20:39:10.226 DEBUG:boto:Headers: {} 2023-05-31T20:39:10.227 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:39:10.227 DEBUG:boto:Port: 8000 2023-05-31T20:39:10.227 DEBUG:boto:Params: {} 2023-05-31T20:39:10.228 DEBUG:boto:Token: None 2023-05-31T20:39:10.229 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:39:10 GMT /pklxkd-58/obj00000000000000000014 2023-05-31T20:39:10.229 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:4zdTHQwqq0TvFlL9FfC5PbiiEO8= 2023-05-31T20:39:10.230 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:39:10 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:4zdTHQwqq0TvFlL9FfC5PbiiEO8=', 'Content-Length': '0'} 2023-05-31T20:39:10.232 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:38:22 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004f34d1d4eca5fe69-006477b06e-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:39:10 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:39:10.232 DEBUG:boto:path=/pklxkd-58/ 2023-05-31T20:39:10.233 DEBUG:boto:auth_path=/pklxkd-58/ 2023-05-31T20:39:10.233 DEBUG:boto:path=/pklxkd-58/?prefix=obj00000000000000000014 2023-05-31T20:39:10.234 DEBUG:boto:auth_path=/pklxkd-58/?prefix=obj00000000000000000014 2023-05-31T20:39:10.234 DEBUG:boto:Method: GET 2023-05-31T20:39:10.234 DEBUG:boto:Path: /pklxkd-58/?prefix=obj00000000000000000014 2023-05-31T20:39:10.235 DEBUG:boto:Data: 2023-05-31T20:39:10.235 DEBUG:boto:Headers: {} 2023-05-31T20:39:10.235 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:39:10.236 DEBUG:boto:Port: 8001 2023-05-31T20:39:10.236 DEBUG:boto:Params: {} 2023-05-31T20:39:10.237 DEBUG:boto:Token: None 2023-05-31T20:39:10.237 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:39:10 GMT /pklxkd-58/ 2023-05-31T20:39:10.237 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:rToF82M8NN6MyNReCsDOOu4EGBY= 2023-05-31T20:39:10.238 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:39:10 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:rToF82M8NN6MyNReCsDOOu4EGBY=', 'Content-Length': '0'} 2023-05-31T20:39:10.241 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000626b55485fac2c01-006477b06e-1c17-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:39:10 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:39:10.241 DEBUG:boto:b'pklxkd-58obj000000000000000000141000falseobj000000000000000000142023-05-31T20:38:22.896Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:39:10.242 DEBUG:boto:path=/pklxkd-58/ 2023-05-31T20:39:10.242 DEBUG:boto:auth_path=/pklxkd-58/ 2023-05-31T20:39:10.243 DEBUG:boto:path=/pklxkd-58/?prefix=obj00000000000000000014 2023-05-31T20:39:10.243 DEBUG:boto:auth_path=/pklxkd-58/?prefix=obj00000000000000000014 2023-05-31T20:39:10.243 DEBUG:boto:Method: GET 2023-05-31T20:39:10.244 DEBUG:boto:Path: /pklxkd-58/?prefix=obj00000000000000000014 2023-05-31T20:39:10.244 DEBUG:boto:Data: 2023-05-31T20:39:10.245 DEBUG:boto:Headers: {} 2023-05-31T20:39:10.245 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:39:10.245 DEBUG:boto:Port: 8000 2023-05-31T20:39:10.246 DEBUG:boto:Params: {} 2023-05-31T20:39:10.246 DEBUG:boto:Token: None 2023-05-31T20:39:10.246 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:39:10 GMT /pklxkd-58/ 2023-05-31T20:39:10.247 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:rToF82M8NN6MyNReCsDOOu4EGBY= 2023-05-31T20:39:10.247 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:39:10 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:rToF82M8NN6MyNReCsDOOu4EGBY=', 'Content-Length': '0'} 2023-05-31T20:39:10.252 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000008f28cdf44331c2ea-006477b06e-1630-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:39:10 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:39:10.253 DEBUG:boto:b'pklxkd-58obj000000000000000000141000falseobj000000000000000000142023-05-31T20:38:22.896Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:39:10.253 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000015 2023-05-31T20:39:10.254 DEBUG:boto:path=/pklxkd-58/obj00000000000000000015 2023-05-31T20:39:10.255 DEBUG:boto:auth_path=/pklxkd-58/obj00000000000000000015 2023-05-31T20:39:10.256 DEBUG:boto:path=/pklxkd-58/obj00000000000000000015?versionId=null 2023-05-31T20:39:10.256 DEBUG:boto:auth_path=/pklxkd-58/obj00000000000000000015?versionId=null 2023-05-31T20:39:10.257 DEBUG:boto:Method: GET 2023-05-31T20:39:10.258 DEBUG:boto:Path: /pklxkd-58/obj00000000000000000015?versionId=null 2023-05-31T20:39:10.259 DEBUG:boto:Data: 2023-05-31T20:39:10.260 DEBUG:boto:Headers: {} 2023-05-31T20:39:10.261 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:39:10.261 DEBUG:boto:Port: 8001 2023-05-31T20:39:10.262 DEBUG:boto:Params: {} 2023-05-31T20:39:10.262 DEBUG:boto:Token: None 2023-05-31T20:39:10.263 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:39:10 GMT /pklxkd-58/obj00000000000000000015?versionId=null 2023-05-31T20:39:10.264 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:7XXt8MToq/o9UJpJFYWU0zl7T88= 2023-05-31T20:39:10.265 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:39:10 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:7XXt8MToq/o9UJpJFYWU0zl7T88=', 'Content-Length': '0'} 2023-05-31T20:39:10.268 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:38:22 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000faae855844b2a85a-006477b06e-1c17-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:39:10 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:39:10.269 DEBUG:boto:path=/pklxkd-58/obj00000000000000000015 2023-05-31T20:39:10.270 DEBUG:boto:auth_path=/pklxkd-58/obj00000000000000000015 2023-05-31T20:39:10.271 DEBUG:boto:path=/pklxkd-58/obj00000000000000000015?versionId=null 2023-05-31T20:39:10.272 DEBUG:boto:auth_path=/pklxkd-58/obj00000000000000000015?versionId=null 2023-05-31T20:39:10.273 DEBUG:boto:Method: GET 2023-05-31T20:39:10.274 DEBUG:boto:Path: /pklxkd-58/obj00000000000000000015?versionId=null 2023-05-31T20:39:10.274 DEBUG:boto:Data: 2023-05-31T20:39:10.275 DEBUG:boto:Headers: {} 2023-05-31T20:39:10.276 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:39:10.277 DEBUG:boto:Port: 8000 2023-05-31T20:39:10.277 DEBUG:boto:Params: {} 2023-05-31T20:39:10.278 DEBUG:boto:Token: None 2023-05-31T20:39:10.280 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:39:10 GMT /pklxkd-58/obj00000000000000000015?versionId=null 2023-05-31T20:39:10.281 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:7XXt8MToq/o9UJpJFYWU0zl7T88= 2023-05-31T20:39:10.281 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:39:10 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:7XXt8MToq/o9UJpJFYWU0zl7T88=', 'Content-Length': '0'} 2023-05-31T20:39:10.284 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:38:22 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f2513c311d4c43da-006477b06e-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:39:10 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:39:10.285 DEBUG:boto:path=/pklxkd-58/obj00000000000000000015 2023-05-31T20:39:10.286 DEBUG:boto:auth_path=/pklxkd-58/obj00000000000000000015 2023-05-31T20:39:10.286 DEBUG:boto:path=/pklxkd-58/obj00000000000000000015?versionId=null 2023-05-31T20:39:10.287 DEBUG:boto:auth_path=/pklxkd-58/obj00000000000000000015?versionId=null 2023-05-31T20:39:10.288 DEBUG:boto:Method: HEAD 2023-05-31T20:39:10.289 DEBUG:boto:Path: /pklxkd-58/obj00000000000000000015?versionId=null 2023-05-31T20:39:10.290 DEBUG:boto:Data: 2023-05-31T20:39:10.290 DEBUG:boto:Headers: {} 2023-05-31T20:39:10.291 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:39:10.291 DEBUG:boto:Port: 8001 2023-05-31T20:39:10.292 DEBUG:boto:Params: {} 2023-05-31T20:39:10.293 DEBUG:boto:Token: None 2023-05-31T20:39:10.294 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:39:10 GMT /pklxkd-58/obj00000000000000000015?versionId=null 2023-05-31T20:39:10.295 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:jvEib8rn0y2il9FRyEXm/1H6h6k= 2023-05-31T20:39:10.295 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:39:10 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:jvEib8rn0y2il9FRyEXm/1H6h6k=', 'Content-Length': '0'} 2023-05-31T20:39:10.298 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:38:22 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006856230abcd633cc-006477b06e-1c17-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:39:10 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:39:10.299 DEBUG:boto:path=/pklxkd-58/obj00000000000000000015 2023-05-31T20:39:10.299 DEBUG:boto:auth_path=/pklxkd-58/obj00000000000000000015 2023-05-31T20:39:10.300 DEBUG:boto:path=/pklxkd-58/obj00000000000000000015?versionId=null 2023-05-31T20:39:10.301 DEBUG:boto:auth_path=/pklxkd-58/obj00000000000000000015?versionId=null 2023-05-31T20:39:10.301 DEBUG:boto:Method: HEAD 2023-05-31T20:39:10.302 DEBUG:boto:Path: /pklxkd-58/obj00000000000000000015?versionId=null 2023-05-31T20:39:10.302 DEBUG:boto:Data: 2023-05-31T20:39:10.303 DEBUG:boto:Headers: {} 2023-05-31T20:39:10.304 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:39:10.304 DEBUG:boto:Port: 8000 2023-05-31T20:39:10.305 DEBUG:boto:Params: {} 2023-05-31T20:39:10.305 DEBUG:boto:Token: None 2023-05-31T20:39:10.306 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:39:10 GMT /pklxkd-58/obj00000000000000000015?versionId=null 2023-05-31T20:39:10.307 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:jvEib8rn0y2il9FRyEXm/1H6h6k= 2023-05-31T20:39:10.307 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:39:10 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:jvEib8rn0y2il9FRyEXm/1H6h6k=', 'Content-Length': '0'} 2023-05-31T20:39:10.310 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:38:22 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000054b851814efe316e-006477b06e-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:39:10 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:39:10.311 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000015 2023-05-31T20:39:10.312 DEBUG:boto:path=/pklxkd-58/obj00000000000000000015 2023-05-31T20:39:10.312 DEBUG:boto:auth_path=/pklxkd-58/obj00000000000000000015 2023-05-31T20:39:10.313 DEBUG:boto:path=/pklxkd-58/obj00000000000000000015?versionId=null 2023-05-31T20:39:10.314 DEBUG:boto:auth_path=/pklxkd-58/obj00000000000000000015?versionId=null 2023-05-31T20:39:10.314 DEBUG:boto:Method: GET 2023-05-31T20:39:10.315 DEBUG:boto:Path: /pklxkd-58/obj00000000000000000015?versionId=null 2023-05-31T20:39:10.315 DEBUG:boto:Data: 2023-05-31T20:39:10.316 DEBUG:boto:Headers: {} 2023-05-31T20:39:10.316 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:39:10.317 DEBUG:boto:Port: 8001 2023-05-31T20:39:10.317 DEBUG:boto:Params: {} 2023-05-31T20:39:10.318 DEBUG:boto:Token: None 2023-05-31T20:39:10.318 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:39:10 GMT /pklxkd-58/obj00000000000000000015?versionId=null 2023-05-31T20:39:10.319 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:7XXt8MToq/o9UJpJFYWU0zl7T88= 2023-05-31T20:39:10.320 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:39:10 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:7XXt8MToq/o9UJpJFYWU0zl7T88=', 'Content-Length': '0'} 2023-05-31T20:39:10.323 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:38:22 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005e877d4ca84a06d1-006477b06e-1c17-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:39:10 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:39:10.324 DEBUG:boto:path=/pklxkd-58/obj00000000000000000015 2023-05-31T20:39:10.325 DEBUG:boto:auth_path=/pklxkd-58/obj00000000000000000015 2023-05-31T20:39:10.325 DEBUG:boto:path=/pklxkd-58/obj00000000000000000015?versionId=null 2023-05-31T20:39:10.326 DEBUG:boto:auth_path=/pklxkd-58/obj00000000000000000015?versionId=null 2023-05-31T20:39:10.327 DEBUG:boto:Method: GET 2023-05-31T20:39:10.327 DEBUG:boto:Path: /pklxkd-58/obj00000000000000000015?versionId=null 2023-05-31T20:39:10.328 DEBUG:boto:Data: 2023-05-31T20:39:10.329 DEBUG:boto:Headers: {} 2023-05-31T20:39:10.330 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:39:10.331 DEBUG:boto:Port: 8000 2023-05-31T20:39:10.332 DEBUG:boto:Params: {} 2023-05-31T20:39:10.333 DEBUG:boto:Token: None 2023-05-31T20:39:10.333 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:39:10 GMT /pklxkd-58/obj00000000000000000015?versionId=null 2023-05-31T20:39:10.334 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:7XXt8MToq/o9UJpJFYWU0zl7T88= 2023-05-31T20:39:10.334 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:39:10 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:7XXt8MToq/o9UJpJFYWU0zl7T88=', 'Content-Length': '0'} 2023-05-31T20:39:10.337 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:38:22 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000692970621a7e5670-006477b06e-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:39:10 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:39:10.338 DEBUG:boto:path=/pklxkd-58/ 2023-05-31T20:39:10.339 DEBUG:boto:auth_path=/pklxkd-58/ 2023-05-31T20:39:10.340 DEBUG:boto:path=/pklxkd-58/?prefix=obj00000000000000000015 2023-05-31T20:39:10.341 DEBUG:boto:auth_path=/pklxkd-58/?prefix=obj00000000000000000015 2023-05-31T20:39:10.342 DEBUG:boto:Method: GET 2023-05-31T20:39:10.342 DEBUG:boto:Path: /pklxkd-58/?prefix=obj00000000000000000015 2023-05-31T20:39:10.343 DEBUG:boto:Data: 2023-05-31T20:39:10.344 DEBUG:boto:Headers: {} 2023-05-31T20:39:10.345 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:39:10.345 DEBUG:boto:Port: 8001 2023-05-31T20:39:10.345 DEBUG:boto:Params: {} 2023-05-31T20:39:10.346 DEBUG:boto:Token: None 2023-05-31T20:39:10.346 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:39:10 GMT /pklxkd-58/ 2023-05-31T20:39:10.347 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:rToF82M8NN6MyNReCsDOOu4EGBY= 2023-05-31T20:39:10.347 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:39:10 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:rToF82M8NN6MyNReCsDOOu4EGBY=', 'Content-Length': '0'} 2023-05-31T20:39:10.349 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000010d68daca8848c48-006477b06e-1c17-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:39:10 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:39:10.350 DEBUG:boto:b'pklxkd-58obj000000000000000000151000falseobj000000000000000000152023-05-31T20:38:22.656Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:39:10.351 DEBUG:boto:path=/pklxkd-58/ 2023-05-31T20:39:10.351 DEBUG:boto:auth_path=/pklxkd-58/ 2023-05-31T20:39:10.352 DEBUG:boto:path=/pklxkd-58/?prefix=obj00000000000000000015 2023-05-31T20:39:10.352 DEBUG:boto:auth_path=/pklxkd-58/?prefix=obj00000000000000000015 2023-05-31T20:39:10.352 DEBUG:boto:Method: GET 2023-05-31T20:39:10.353 DEBUG:boto:Path: /pklxkd-58/?prefix=obj00000000000000000015 2023-05-31T20:39:10.353 DEBUG:boto:Data: 2023-05-31T20:39:10.353 DEBUG:boto:Headers: {} 2023-05-31T20:39:10.354 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:39:10.354 DEBUG:boto:Port: 8000 2023-05-31T20:39:10.354 DEBUG:boto:Params: {} 2023-05-31T20:39:10.355 DEBUG:boto:Token: None 2023-05-31T20:39:10.355 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:39:10 GMT /pklxkd-58/ 2023-05-31T20:39:10.356 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:rToF82M8NN6MyNReCsDOOu4EGBY= 2023-05-31T20:39:10.356 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:39:10 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:rToF82M8NN6MyNReCsDOOu4EGBY=', 'Content-Length': '0'} 2023-05-31T20:39:10.360 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d8968f0fa6bdf947-006477b06e-1630-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:39:10 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:39:10.361 DEBUG:boto:b'pklxkd-58obj000000000000000000151000falseobj000000000000000000152023-05-31T20:38:22.656Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:39:10.361 DEBUG:boto:path=/pklxkd-58/obj00000000000000000015 2023-05-31T20:39:10.362 DEBUG:boto:auth_path=/pklxkd-58/obj00000000000000000015 2023-05-31T20:39:10.362 DEBUG:boto:Method: HEAD 2023-05-31T20:39:10.363 DEBUG:boto:Path: /pklxkd-58/obj00000000000000000015 2023-05-31T20:39:10.363 DEBUG:boto:Data: 2023-05-31T20:39:10.363 DEBUG:boto:Headers: {} 2023-05-31T20:39:10.363 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:39:10.364 DEBUG:boto:Port: 8001 2023-05-31T20:39:10.364 DEBUG:boto:Params: {} 2023-05-31T20:39:10.365 DEBUG:boto:Token: None 2023-05-31T20:39:10.365 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:39:10 GMT /pklxkd-58/obj00000000000000000015 2023-05-31T20:39:10.365 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:UXmkG6ppovTRG/uY+B749RX3ymk= 2023-05-31T20:39:10.366 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:39:10 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:UXmkG6ppovTRG/uY+B749RX3ymk=', 'Content-Length': '0'} 2023-05-31T20:39:10.368 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:38:22 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000aad5c8b1c781943a-006477b06e-1c17-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:39:10 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:39:10.368 DEBUG:boto:path=/pklxkd-58/obj00000000000000000015 2023-05-31T20:39:10.369 DEBUG:boto:auth_path=/pklxkd-58/obj00000000000000000015 2023-05-31T20:39:10.369 DEBUG:boto:Method: HEAD 2023-05-31T20:39:10.370 DEBUG:boto:Path: /pklxkd-58/obj00000000000000000015 2023-05-31T20:39:10.370 DEBUG:boto:Data: 2023-05-31T20:39:10.370 DEBUG:boto:Headers: {} 2023-05-31T20:39:10.371 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:39:10.371 DEBUG:boto:Port: 8000 2023-05-31T20:39:10.371 DEBUG:boto:Params: {} 2023-05-31T20:39:10.372 DEBUG:boto:Token: None 2023-05-31T20:39:10.372 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:39:10 GMT /pklxkd-58/obj00000000000000000015 2023-05-31T20:39:10.372 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:UXmkG6ppovTRG/uY+B749RX3ymk= 2023-05-31T20:39:10.373 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:39:10 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:UXmkG6ppovTRG/uY+B749RX3ymk=', 'Content-Length': '0'} 2023-05-31T20:39:10.375 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:38:22 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008485ec65d6b09950-006477b06e-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:39:10 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:39:10.375 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000015 2023-05-31T20:39:10.376 DEBUG:boto:path=/pklxkd-58/obj00000000000000000015 2023-05-31T20:39:10.376 DEBUG:boto:auth_path=/pklxkd-58/obj00000000000000000015 2023-05-31T20:39:10.376 DEBUG:boto:Method: GET 2023-05-31T20:39:10.377 DEBUG:boto:Path: /pklxkd-58/obj00000000000000000015 2023-05-31T20:39:10.377 DEBUG:boto:Data: 2023-05-31T20:39:10.378 DEBUG:boto:Headers: {} 2023-05-31T20:39:10.378 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:39:10.378 DEBUG:boto:Port: 8001 2023-05-31T20:39:10.379 DEBUG:boto:Params: {} 2023-05-31T20:39:10.379 DEBUG:boto:Token: None 2023-05-31T20:39:10.379 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:39:10 GMT /pklxkd-58/obj00000000000000000015 2023-05-31T20:39:10.380 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:AI/cKvgbujTnIRK7l0qgPk6vZmU= 2023-05-31T20:39:10.380 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:39:10 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:AI/cKvgbujTnIRK7l0qgPk6vZmU=', 'Content-Length': '0'} 2023-05-31T20:39:10.382 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:38:22 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000013f4232d3292c9c5-006477b06e-1c17-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:39:10 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:39:10.428 DEBUG:boto:path=/pklxkd-58/obj00000000000000000015 2023-05-31T20:39:10.429 DEBUG:boto:auth_path=/pklxkd-58/obj00000000000000000015 2023-05-31T20:39:10.430 DEBUG:boto:Method: GET 2023-05-31T20:39:10.430 DEBUG:boto:Path: /pklxkd-58/obj00000000000000000015 2023-05-31T20:39:10.431 DEBUG:boto:Data: 2023-05-31T20:39:10.432 DEBUG:boto:Headers: {} 2023-05-31T20:39:10.432 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:39:10.432 DEBUG:boto:Port: 8000 2023-05-31T20:39:10.433 DEBUG:boto:Params: {} 2023-05-31T20:39:10.433 DEBUG:boto:Token: None 2023-05-31T20:39:10.434 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:39:10 GMT /pklxkd-58/obj00000000000000000015 2023-05-31T20:39:10.434 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:AI/cKvgbujTnIRK7l0qgPk6vZmU= 2023-05-31T20:39:10.434 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:39:10 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:AI/cKvgbujTnIRK7l0qgPk6vZmU=', 'Content-Length': '0'} 2023-05-31T20:39:10.437 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:38:22 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005d592edb1ec3444c-006477b06e-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:39:10 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:39:10.438 DEBUG:boto:path=/pklxkd-58/ 2023-05-31T20:39:10.438 DEBUG:boto:auth_path=/pklxkd-58/ 2023-05-31T20:39:10.439 DEBUG:boto:path=/pklxkd-58/?prefix=obj00000000000000000015 2023-05-31T20:39:10.439 DEBUG:boto:auth_path=/pklxkd-58/?prefix=obj00000000000000000015 2023-05-31T20:39:10.439 DEBUG:boto:Method: GET 2023-05-31T20:39:10.440 DEBUG:boto:Path: /pklxkd-58/?prefix=obj00000000000000000015 2023-05-31T20:39:10.440 DEBUG:boto:Data: 2023-05-31T20:39:10.440 DEBUG:boto:Headers: {} 2023-05-31T20:39:10.441 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:39:10.442 DEBUG:boto:Port: 8001 2023-05-31T20:39:10.442 DEBUG:boto:Params: {} 2023-05-31T20:39:10.442 DEBUG:boto:Token: None 2023-05-31T20:39:10.443 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:39:10 GMT /pklxkd-58/ 2023-05-31T20:39:10.443 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:rToF82M8NN6MyNReCsDOOu4EGBY= 2023-05-31T20:39:10.444 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:39:10 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:rToF82M8NN6MyNReCsDOOu4EGBY=', 'Content-Length': '0'} 2023-05-31T20:39:10.446 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000039230001ed1fc47b-006477b06e-1c17-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:39:10 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:39:10.447 DEBUG:boto:b'pklxkd-58obj000000000000000000151000falseobj000000000000000000152023-05-31T20:38:22.656Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:39:10.447 DEBUG:boto:path=/pklxkd-58/ 2023-05-31T20:39:10.448 DEBUG:boto:auth_path=/pklxkd-58/ 2023-05-31T20:39:10.448 DEBUG:boto:path=/pklxkd-58/?prefix=obj00000000000000000015 2023-05-31T20:39:10.449 DEBUG:boto:auth_path=/pklxkd-58/?prefix=obj00000000000000000015 2023-05-31T20:39:10.449 DEBUG:boto:Method: GET 2023-05-31T20:39:10.449 DEBUG:boto:Path: /pklxkd-58/?prefix=obj00000000000000000015 2023-05-31T20:39:10.450 DEBUG:boto:Data: 2023-05-31T20:39:10.450 DEBUG:boto:Headers: {} 2023-05-31T20:39:10.450 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:39:10.451 DEBUG:boto:Port: 8000 2023-05-31T20:39:10.451 DEBUG:boto:Params: {} 2023-05-31T20:39:10.452 DEBUG:boto:Token: None 2023-05-31T20:39:10.452 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:39:10 GMT /pklxkd-58/ 2023-05-31T20:39:10.452 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:rToF82M8NN6MyNReCsDOOu4EGBY= 2023-05-31T20:39:10.453 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:39:10 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:rToF82M8NN6MyNReCsDOOu4EGBY=', 'Content-Length': '0'} 2023-05-31T20:39:10.457 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000af1d8ce7bda7f23a-006477b06e-1630-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:39:10 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:39:10.458 DEBUG:boto:b'pklxkd-58obj000000000000000000151000falseobj000000000000000000152023-05-31T20:38:22.656Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:39:10.458 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000016 2023-05-31T20:39:10.459 DEBUG:boto:path=/pklxkd-58/obj00000000000000000016 2023-05-31T20:39:10.459 DEBUG:boto:auth_path=/pklxkd-58/obj00000000000000000016 2023-05-31T20:39:10.459 DEBUG:boto:path=/pklxkd-58/obj00000000000000000016?versionId=null 2023-05-31T20:39:10.460 DEBUG:boto:auth_path=/pklxkd-58/obj00000000000000000016?versionId=null 2023-05-31T20:39:10.460 DEBUG:boto:Method: GET 2023-05-31T20:39:10.460 DEBUG:boto:Path: /pklxkd-58/obj00000000000000000016?versionId=null 2023-05-31T20:39:10.461 DEBUG:boto:Data: 2023-05-31T20:39:10.461 DEBUG:boto:Headers: {} 2023-05-31T20:39:10.462 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:39:10.462 DEBUG:boto:Port: 8001 2023-05-31T20:39:10.462 DEBUG:boto:Params: {} 2023-05-31T20:39:10.463 DEBUG:boto:Token: None 2023-05-31T20:39:10.463 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:39:10 GMT /pklxkd-58/obj00000000000000000016?versionId=null 2023-05-31T20:39:10.463 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:e5lmQrwz9loKuWqG/2zjDJSXgUU= 2023-05-31T20:39:10.464 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:39:10 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:e5lmQrwz9loKuWqG/2zjDJSXgUU=', 'Content-Length': '0'} 2023-05-31T20:39:10.466 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:38:22 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000af73e00403238e50-006477b06e-1c17-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:39:10 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:39:10.467 DEBUG:boto:path=/pklxkd-58/obj00000000000000000016 2023-05-31T20:39:10.467 DEBUG:boto:auth_path=/pklxkd-58/obj00000000000000000016 2023-05-31T20:39:10.467 DEBUG:boto:path=/pklxkd-58/obj00000000000000000016?versionId=null 2023-05-31T20:39:10.468 DEBUG:boto:auth_path=/pklxkd-58/obj00000000000000000016?versionId=null 2023-05-31T20:39:10.468 DEBUG:boto:Method: GET 2023-05-31T20:39:10.469 DEBUG:boto:Path: /pklxkd-58/obj00000000000000000016?versionId=null 2023-05-31T20:39:10.469 DEBUG:boto:Data: 2023-05-31T20:39:10.469 DEBUG:boto:Headers: {} 2023-05-31T20:39:10.470 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:39:10.470 DEBUG:boto:Port: 8000 2023-05-31T20:39:10.470 DEBUG:boto:Params: {} 2023-05-31T20:39:10.471 DEBUG:boto:Token: None 2023-05-31T20:39:10.471 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:39:10 GMT /pklxkd-58/obj00000000000000000016?versionId=null 2023-05-31T20:39:10.472 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:e5lmQrwz9loKuWqG/2zjDJSXgUU= 2023-05-31T20:39:10.472 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:39:10 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:e5lmQrwz9loKuWqG/2zjDJSXgUU=', 'Content-Length': '0'} 2023-05-31T20:39:10.474 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:38:22 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e118b847b7759431-006477b06e-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:39:10 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:39:10.475 DEBUG:boto:path=/pklxkd-58/obj00000000000000000016 2023-05-31T20:39:10.475 DEBUG:boto:auth_path=/pklxkd-58/obj00000000000000000016 2023-05-31T20:39:10.476 DEBUG:boto:path=/pklxkd-58/obj00000000000000000016?versionId=null 2023-05-31T20:39:10.476 DEBUG:boto:auth_path=/pklxkd-58/obj00000000000000000016?versionId=null 2023-05-31T20:39:10.476 DEBUG:boto:Method: HEAD 2023-05-31T20:39:10.477 DEBUG:boto:Path: /pklxkd-58/obj00000000000000000016?versionId=null 2023-05-31T20:39:10.477 DEBUG:boto:Data: 2023-05-31T20:39:10.477 DEBUG:boto:Headers: {} 2023-05-31T20:39:10.478 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:39:10.478 DEBUG:boto:Port: 8001 2023-05-31T20:39:10.478 DEBUG:boto:Params: {} 2023-05-31T20:39:10.479 DEBUG:boto:Token: None 2023-05-31T20:39:10.479 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:39:10 GMT /pklxkd-58/obj00000000000000000016?versionId=null 2023-05-31T20:39:10.479 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:+NgLRM666zH+Z4A2yj9QTTgrtRE= 2023-05-31T20:39:10.480 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:39:10 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:+NgLRM666zH+Z4A2yj9QTTgrtRE=', 'Content-Length': '0'} 2023-05-31T20:39:10.482 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:38:22 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009b676d4458d20e3f-006477b06e-1c17-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:39:10 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:39:10.482 DEBUG:boto:path=/pklxkd-58/obj00000000000000000016 2023-05-31T20:39:10.483 DEBUG:boto:auth_path=/pklxkd-58/obj00000000000000000016 2023-05-31T20:39:10.483 DEBUG:boto:path=/pklxkd-58/obj00000000000000000016?versionId=null 2023-05-31T20:39:10.483 DEBUG:boto:auth_path=/pklxkd-58/obj00000000000000000016?versionId=null 2023-05-31T20:39:10.484 DEBUG:boto:Method: HEAD 2023-05-31T20:39:10.484 DEBUG:boto:Path: /pklxkd-58/obj00000000000000000016?versionId=null 2023-05-31T20:39:10.484 DEBUG:boto:Data: 2023-05-31T20:39:10.485 DEBUG:boto:Headers: {} 2023-05-31T20:39:10.485 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:39:10.485 DEBUG:boto:Port: 8000 2023-05-31T20:39:10.486 DEBUG:boto:Params: {} 2023-05-31T20:39:10.486 DEBUG:boto:Token: None 2023-05-31T20:39:10.487 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:39:10 GMT /pklxkd-58/obj00000000000000000016?versionId=null 2023-05-31T20:39:10.487 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:+NgLRM666zH+Z4A2yj9QTTgrtRE= 2023-05-31T20:39:10.487 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:39:10 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:+NgLRM666zH+Z4A2yj9QTTgrtRE=', 'Content-Length': '0'} 2023-05-31T20:39:10.489 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:38:22 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006bb047ca55b316bd-006477b06e-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:39:10 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:39:10.490 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000016 2023-05-31T20:39:10.490 DEBUG:boto:path=/pklxkd-58/obj00000000000000000016 2023-05-31T20:39:10.490 DEBUG:boto:auth_path=/pklxkd-58/obj00000000000000000016 2023-05-31T20:39:10.491 DEBUG:boto:path=/pklxkd-58/obj00000000000000000016?versionId=null 2023-05-31T20:39:10.491 DEBUG:boto:auth_path=/pklxkd-58/obj00000000000000000016?versionId=null 2023-05-31T20:39:10.491 DEBUG:boto:Method: GET 2023-05-31T20:39:10.492 DEBUG:boto:Path: /pklxkd-58/obj00000000000000000016?versionId=null 2023-05-31T20:39:10.492 DEBUG:boto:Data: 2023-05-31T20:39:10.492 DEBUG:boto:Headers: {} 2023-05-31T20:39:10.493 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:39:10.493 DEBUG:boto:Port: 8001 2023-05-31T20:39:10.494 DEBUG:boto:Params: {} 2023-05-31T20:39:10.494 DEBUG:boto:Token: None 2023-05-31T20:39:10.494 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:39:10 GMT /pklxkd-58/obj00000000000000000016?versionId=null 2023-05-31T20:39:10.495 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:e5lmQrwz9loKuWqG/2zjDJSXgUU= 2023-05-31T20:39:10.495 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:39:10 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:e5lmQrwz9loKuWqG/2zjDJSXgUU=', 'Content-Length': '0'} 2023-05-31T20:39:10.497 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:38:22 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000019d6b743d7d921cf-006477b06e-1c17-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:39:10 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:39:10.498 DEBUG:boto:path=/pklxkd-58/obj00000000000000000016 2023-05-31T20:39:10.498 DEBUG:boto:auth_path=/pklxkd-58/obj00000000000000000016 2023-05-31T20:39:10.499 DEBUG:boto:path=/pklxkd-58/obj00000000000000000016?versionId=null 2023-05-31T20:39:10.499 DEBUG:boto:auth_path=/pklxkd-58/obj00000000000000000016?versionId=null 2023-05-31T20:39:10.499 DEBUG:boto:Method: GET 2023-05-31T20:39:10.500 DEBUG:boto:Path: /pklxkd-58/obj00000000000000000016?versionId=null 2023-05-31T20:39:10.500 DEBUG:boto:Data: 2023-05-31T20:39:10.500 DEBUG:boto:Headers: {} 2023-05-31T20:39:10.501 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:39:10.501 DEBUG:boto:Port: 8000 2023-05-31T20:39:10.501 DEBUG:boto:Params: {} 2023-05-31T20:39:10.502 DEBUG:boto:Token: None 2023-05-31T20:39:10.502 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:39:10 GMT /pklxkd-58/obj00000000000000000016?versionId=null 2023-05-31T20:39:10.503 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:e5lmQrwz9loKuWqG/2zjDJSXgUU= 2023-05-31T20:39:10.503 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:39:10 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:e5lmQrwz9loKuWqG/2zjDJSXgUU=', 'Content-Length': '0'} 2023-05-31T20:39:10.505 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:38:22 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000293bbed9fdef669b-006477b06e-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:39:10 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:39:10.552 DEBUG:boto:path=/pklxkd-58/ 2023-05-31T20:39:10.553 DEBUG:boto:auth_path=/pklxkd-58/ 2023-05-31T20:39:10.553 DEBUG:boto:path=/pklxkd-58/?prefix=obj00000000000000000016 2023-05-31T20:39:10.554 DEBUG:boto:auth_path=/pklxkd-58/?prefix=obj00000000000000000016 2023-05-31T20:39:10.554 DEBUG:boto:Method: GET 2023-05-31T20:39:10.554 DEBUG:boto:Path: /pklxkd-58/?prefix=obj00000000000000000016 2023-05-31T20:39:10.555 DEBUG:boto:Data: 2023-05-31T20:39:10.555 DEBUG:boto:Headers: {} 2023-05-31T20:39:10.555 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:39:10.556 DEBUG:boto:Port: 8001 2023-05-31T20:39:10.557 DEBUG:boto:Params: {} 2023-05-31T20:39:10.557 DEBUG:boto:Token: None 2023-05-31T20:39:10.558 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:39:10 GMT /pklxkd-58/ 2023-05-31T20:39:10.558 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:rToF82M8NN6MyNReCsDOOu4EGBY= 2023-05-31T20:39:10.558 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:39:10 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:rToF82M8NN6MyNReCsDOOu4EGBY=', 'Content-Length': '0'} 2023-05-31T20:39:10.561 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000005813f272aad64f2f-006477b06e-1c17-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:39:10 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:39:10.562 DEBUG:boto:b'pklxkd-58obj000000000000000000161000falseobj000000000000000000162023-05-31T20:38:22.140Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:39:10.563 DEBUG:boto:path=/pklxkd-58/ 2023-05-31T20:39:10.564 DEBUG:boto:auth_path=/pklxkd-58/ 2023-05-31T20:39:10.564 DEBUG:boto:path=/pklxkd-58/?prefix=obj00000000000000000016 2023-05-31T20:39:10.564 DEBUG:boto:auth_path=/pklxkd-58/?prefix=obj00000000000000000016 2023-05-31T20:39:10.565 DEBUG:boto:Method: GET 2023-05-31T20:39:10.565 DEBUG:boto:Path: /pklxkd-58/?prefix=obj00000000000000000016 2023-05-31T20:39:10.565 DEBUG:boto:Data: 2023-05-31T20:39:10.566 DEBUG:boto:Headers: {} 2023-05-31T20:39:10.566 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:39:10.567 DEBUG:boto:Port: 8000 2023-05-31T20:39:10.567 DEBUG:boto:Params: {} 2023-05-31T20:39:10.567 DEBUG:boto:Token: None 2023-05-31T20:39:10.568 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:39:10 GMT /pklxkd-58/ 2023-05-31T20:39:10.568 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:rToF82M8NN6MyNReCsDOOu4EGBY= 2023-05-31T20:39:10.569 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:39:10 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:rToF82M8NN6MyNReCsDOOu4EGBY=', 'Content-Length': '0'} 2023-05-31T20:39:10.573 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f1008dfdf2d5cec3-006477b06e-1630-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:39:10 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:39:10.573 DEBUG:boto:b'pklxkd-58obj000000000000000000161000falseobj000000000000000000162023-05-31T20:38:22.140Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:39:10.574 DEBUG:boto:path=/pklxkd-58/obj00000000000000000016 2023-05-31T20:39:10.574 DEBUG:boto:auth_path=/pklxkd-58/obj00000000000000000016 2023-05-31T20:39:10.575 DEBUG:boto:Method: HEAD 2023-05-31T20:39:10.575 DEBUG:boto:Path: /pklxkd-58/obj00000000000000000016 2023-05-31T20:39:10.576 DEBUG:boto:Data: 2023-05-31T20:39:10.576 DEBUG:boto:Headers: {} 2023-05-31T20:39:10.576 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:39:10.577 DEBUG:boto:Port: 8001 2023-05-31T20:39:10.577 DEBUG:boto:Params: {} 2023-05-31T20:39:10.577 DEBUG:boto:Token: None 2023-05-31T20:39:10.578 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:39:10 GMT /pklxkd-58/obj00000000000000000016 2023-05-31T20:39:10.578 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:trix8P1AAnbxjgxQtTuzNT/9p3A= 2023-05-31T20:39:10.579 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:39:10 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:trix8P1AAnbxjgxQtTuzNT/9p3A=', 'Content-Length': '0'} 2023-05-31T20:39:10.581 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:38:22 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009ae89856cd32f896-006477b06e-1c17-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:39:10 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:39:10.581 DEBUG:boto:path=/pklxkd-58/obj00000000000000000016 2023-05-31T20:39:10.582 DEBUG:boto:auth_path=/pklxkd-58/obj00000000000000000016 2023-05-31T20:39:10.582 DEBUG:boto:Method: HEAD 2023-05-31T20:39:10.583 DEBUG:boto:Path: /pklxkd-58/obj00000000000000000016 2023-05-31T20:39:10.583 DEBUG:boto:Data: 2023-05-31T20:39:10.583 DEBUG:boto:Headers: {} 2023-05-31T20:39:10.584 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:39:10.584 DEBUG:boto:Port: 8000 2023-05-31T20:39:10.585 DEBUG:boto:Params: {} 2023-05-31T20:39:10.585 DEBUG:boto:Token: None 2023-05-31T20:39:10.585 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:39:10 GMT /pklxkd-58/obj00000000000000000016 2023-05-31T20:39:10.586 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:trix8P1AAnbxjgxQtTuzNT/9p3A= 2023-05-31T20:39:10.586 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:39:10 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:trix8P1AAnbxjgxQtTuzNT/9p3A=', 'Content-Length': '0'} 2023-05-31T20:39:10.589 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:38:22 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004e3a8a3811a7f306-006477b06e-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:39:10 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:39:10.589 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000016 2023-05-31T20:39:10.590 DEBUG:boto:path=/pklxkd-58/obj00000000000000000016 2023-05-31T20:39:10.590 DEBUG:boto:auth_path=/pklxkd-58/obj00000000000000000016 2023-05-31T20:39:10.590 DEBUG:boto:Method: GET 2023-05-31T20:39:10.591 DEBUG:boto:Path: /pklxkd-58/obj00000000000000000016 2023-05-31T20:39:10.591 DEBUG:boto:Data: 2023-05-31T20:39:10.591 DEBUG:boto:Headers: {} 2023-05-31T20:39:10.592 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:39:10.592 DEBUG:boto:Port: 8001 2023-05-31T20:39:10.593 DEBUG:boto:Params: {} 2023-05-31T20:39:10.593 DEBUG:boto:Token: None 2023-05-31T20:39:10.593 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:39:10 GMT /pklxkd-58/obj00000000000000000016 2023-05-31T20:39:10.594 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:IunClj705YMMydCHTErkQCegQ9M= 2023-05-31T20:39:10.594 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:39:10 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:IunClj705YMMydCHTErkQCegQ9M=', 'Content-Length': '0'} 2023-05-31T20:39:10.596 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:38:22 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e6ac8a3df8a16eab-006477b06e-1c17-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:39:10 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:39:10.644 DEBUG:boto:path=/pklxkd-58/obj00000000000000000016 2023-05-31T20:39:10.645 DEBUG:boto:auth_path=/pklxkd-58/obj00000000000000000016 2023-05-31T20:39:10.646 DEBUG:boto:Method: GET 2023-05-31T20:39:10.646 DEBUG:boto:Path: /pklxkd-58/obj00000000000000000016 2023-05-31T20:39:10.646 DEBUG:boto:Data: 2023-05-31T20:39:10.647 DEBUG:boto:Headers: {} 2023-05-31T20:39:10.647 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:39:10.648 DEBUG:boto:Port: 8000 2023-05-31T20:39:10.648 DEBUG:boto:Params: {} 2023-05-31T20:39:10.648 DEBUG:boto:Token: None 2023-05-31T20:39:10.649 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:39:10 GMT /pklxkd-58/obj00000000000000000016 2023-05-31T20:39:10.649 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:IunClj705YMMydCHTErkQCegQ9M= 2023-05-31T20:39:10.651 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:39:10 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:IunClj705YMMydCHTErkQCegQ9M=', 'Content-Length': '0'} 2023-05-31T20:39:10.658 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:38:22 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000071e51284a4b2e4de-006477b06e-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:39:10 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:39:10.659 DEBUG:boto:path=/pklxkd-58/ 2023-05-31T20:39:10.660 DEBUG:boto:auth_path=/pklxkd-58/ 2023-05-31T20:39:10.661 DEBUG:boto:path=/pklxkd-58/?prefix=obj00000000000000000016 2023-05-31T20:39:10.662 DEBUG:boto:auth_path=/pklxkd-58/?prefix=obj00000000000000000016 2023-05-31T20:39:10.663 DEBUG:boto:Method: GET 2023-05-31T20:39:10.664 DEBUG:boto:Path: /pklxkd-58/?prefix=obj00000000000000000016 2023-05-31T20:39:10.665 DEBUG:boto:Data: 2023-05-31T20:39:10.666 DEBUG:boto:Headers: {} 2023-05-31T20:39:10.668 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:39:10.668 DEBUG:boto:Port: 8001 2023-05-31T20:39:10.668 DEBUG:boto:Params: {} 2023-05-31T20:39:10.669 DEBUG:boto:Token: None 2023-05-31T20:39:10.669 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:39:10 GMT /pklxkd-58/ 2023-05-31T20:39:10.670 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:rToF82M8NN6MyNReCsDOOu4EGBY= 2023-05-31T20:39:10.671 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:39:10 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:rToF82M8NN6MyNReCsDOOu4EGBY=', 'Content-Length': '0'} 2023-05-31T20:39:10.673 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c4da32638eded5ab-006477b06e-1c17-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:39:10 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:39:10.674 DEBUG:boto:b'pklxkd-58obj000000000000000000161000falseobj000000000000000000162023-05-31T20:38:22.140Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:39:10.675 DEBUG:boto:path=/pklxkd-58/ 2023-05-31T20:39:10.675 DEBUG:boto:auth_path=/pklxkd-58/ 2023-05-31T20:39:10.676 DEBUG:boto:path=/pklxkd-58/?prefix=obj00000000000000000016 2023-05-31T20:39:10.676 DEBUG:boto:auth_path=/pklxkd-58/?prefix=obj00000000000000000016 2023-05-31T20:39:10.677 DEBUG:boto:Method: GET 2023-05-31T20:39:10.677 DEBUG:boto:Path: /pklxkd-58/?prefix=obj00000000000000000016 2023-05-31T20:39:10.677 DEBUG:boto:Data: 2023-05-31T20:39:10.678 DEBUG:boto:Headers: {} 2023-05-31T20:39:10.678 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:39:10.678 DEBUG:boto:Port: 8000 2023-05-31T20:39:10.679 DEBUG:boto:Params: {} 2023-05-31T20:39:10.679 DEBUG:boto:Token: None 2023-05-31T20:39:10.680 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:39:10 GMT /pklxkd-58/ 2023-05-31T20:39:10.680 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:rToF82M8NN6MyNReCsDOOu4EGBY= 2023-05-31T20:39:10.681 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:39:10 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:rToF82M8NN6MyNReCsDOOu4EGBY=', 'Content-Length': '0'} 2023-05-31T20:39:10.685 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f1151c06fd757872-006477b06e-1630-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:39:10 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:39:10.728 DEBUG:boto:b'pklxkd-58obj000000000000000000161000falseobj000000000000000000162023-05-31T20:38:22.140Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:39:10.729 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000017 2023-05-31T20:39:10.729 DEBUG:boto:path=/pklxkd-58/obj00000000000000000017 2023-05-31T20:39:10.730 DEBUG:boto:auth_path=/pklxkd-58/obj00000000000000000017 2023-05-31T20:39:10.730 DEBUG:boto:path=/pklxkd-58/obj00000000000000000017?versionId=null 2023-05-31T20:39:10.731 DEBUG:boto:auth_path=/pklxkd-58/obj00000000000000000017?versionId=null 2023-05-31T20:39:10.731 DEBUG:boto:Method: GET 2023-05-31T20:39:10.731 DEBUG:boto:Path: /pklxkd-58/obj00000000000000000017?versionId=null 2023-05-31T20:39:10.732 DEBUG:boto:Data: 2023-05-31T20:39:10.733 DEBUG:boto:Headers: {} 2023-05-31T20:39:10.733 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:39:10.733 DEBUG:boto:Port: 8001 2023-05-31T20:39:10.734 DEBUG:boto:Params: {} 2023-05-31T20:39:10.734 DEBUG:boto:Token: None 2023-05-31T20:39:10.735 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:39:10 GMT /pklxkd-58/obj00000000000000000017?versionId=null 2023-05-31T20:39:10.735 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:Sl5Be0/iuTgv8OmhauX5m0ewRSo= 2023-05-31T20:39:10.735 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:39:10 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:Sl5Be0/iuTgv8OmhauX5m0ewRSo=', 'Content-Length': '0'} 2023-05-31T20:39:10.738 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:38:22 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004fe0f70c3573e5ff-006477b06e-1c17-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:39:10 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:39:10.739 DEBUG:boto:path=/pklxkd-58/obj00000000000000000017 2023-05-31T20:39:10.739 DEBUG:boto:auth_path=/pklxkd-58/obj00000000000000000017 2023-05-31T20:39:10.739 DEBUG:boto:path=/pklxkd-58/obj00000000000000000017?versionId=null 2023-05-31T20:39:10.740 DEBUG:boto:auth_path=/pklxkd-58/obj00000000000000000017?versionId=null 2023-05-31T20:39:10.740 DEBUG:boto:Method: GET 2023-05-31T20:39:10.741 DEBUG:boto:Path: /pklxkd-58/obj00000000000000000017?versionId=null 2023-05-31T20:39:10.741 DEBUG:boto:Data: 2023-05-31T20:39:10.741 DEBUG:boto:Headers: {} 2023-05-31T20:39:10.742 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:39:10.742 DEBUG:boto:Port: 8000 2023-05-31T20:39:10.742 DEBUG:boto:Params: {} 2023-05-31T20:39:10.743 DEBUG:boto:Token: None 2023-05-31T20:39:10.743 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:39:10 GMT /pklxkd-58/obj00000000000000000017?versionId=null 2023-05-31T20:39:10.744 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:Sl5Be0/iuTgv8OmhauX5m0ewRSo= 2023-05-31T20:39:10.744 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:39:10 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:Sl5Be0/iuTgv8OmhauX5m0ewRSo=', 'Content-Length': '0'} 2023-05-31T20:39:10.746 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:38:22 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000186a873911fd9c2b-006477b06e-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:39:10 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:39:10.747 DEBUG:boto:path=/pklxkd-58/obj00000000000000000017 2023-05-31T20:39:10.747 DEBUG:boto:auth_path=/pklxkd-58/obj00000000000000000017 2023-05-31T20:39:10.748 DEBUG:boto:path=/pklxkd-58/obj00000000000000000017?versionId=null 2023-05-31T20:39:10.748 DEBUG:boto:auth_path=/pklxkd-58/obj00000000000000000017?versionId=null 2023-05-31T20:39:10.748 DEBUG:boto:Method: HEAD 2023-05-31T20:39:10.749 DEBUG:boto:Path: /pklxkd-58/obj00000000000000000017?versionId=null 2023-05-31T20:39:10.749 DEBUG:boto:Data: 2023-05-31T20:39:10.749 DEBUG:boto:Headers: {} 2023-05-31T20:39:10.750 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:39:10.750 DEBUG:boto:Port: 8001 2023-05-31T20:39:10.750 DEBUG:boto:Params: {} 2023-05-31T20:39:10.751 DEBUG:boto:Token: None 2023-05-31T20:39:10.751 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:39:10 GMT /pklxkd-58/obj00000000000000000017?versionId=null 2023-05-31T20:39:10.752 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:1Y6AhzA+OasnZeHwiqhAm4e7d5Q= 2023-05-31T20:39:10.752 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:39:10 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:1Y6AhzA+OasnZeHwiqhAm4e7d5Q=', 'Content-Length': '0'} 2023-05-31T20:39:10.754 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:38:22 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000ad9c0df10a42a3d-006477b06e-1c17-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:39:10 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:39:10.755 DEBUG:boto:path=/pklxkd-58/obj00000000000000000017 2023-05-31T20:39:10.755 DEBUG:boto:auth_path=/pklxkd-58/obj00000000000000000017 2023-05-31T20:39:10.756 DEBUG:boto:path=/pklxkd-58/obj00000000000000000017?versionId=null 2023-05-31T20:39:10.756 DEBUG:boto:auth_path=/pklxkd-58/obj00000000000000000017?versionId=null 2023-05-31T20:39:10.757 DEBUG:boto:Method: HEAD 2023-05-31T20:39:10.757 DEBUG:boto:Path: /pklxkd-58/obj00000000000000000017?versionId=null 2023-05-31T20:39:10.757 DEBUG:boto:Data: 2023-05-31T20:39:10.758 DEBUG:boto:Headers: {} 2023-05-31T20:39:10.758 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:39:10.758 DEBUG:boto:Port: 8000 2023-05-31T20:39:10.759 DEBUG:boto:Params: {} 2023-05-31T20:39:10.759 DEBUG:boto:Token: None 2023-05-31T20:39:10.759 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:39:10 GMT /pklxkd-58/obj00000000000000000017?versionId=null 2023-05-31T20:39:10.760 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:1Y6AhzA+OasnZeHwiqhAm4e7d5Q= 2023-05-31T20:39:10.760 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:39:10 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:1Y6AhzA+OasnZeHwiqhAm4e7d5Q=', 'Content-Length': '0'} 2023-05-31T20:39:10.762 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:38:22 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000a98ac0db02cf0eb-006477b06e-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:39:10 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:39:10.763 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000017 2023-05-31T20:39:10.763 DEBUG:boto:path=/pklxkd-58/obj00000000000000000017 2023-05-31T20:39:10.764 DEBUG:boto:auth_path=/pklxkd-58/obj00000000000000000017 2023-05-31T20:39:10.764 DEBUG:boto:path=/pklxkd-58/obj00000000000000000017?versionId=null 2023-05-31T20:39:10.765 DEBUG:boto:auth_path=/pklxkd-58/obj00000000000000000017?versionId=null 2023-05-31T20:39:10.765 DEBUG:boto:Method: GET 2023-05-31T20:39:10.765 DEBUG:boto:Path: /pklxkd-58/obj00000000000000000017?versionId=null 2023-05-31T20:39:10.766 DEBUG:boto:Data: 2023-05-31T20:39:10.766 DEBUG:boto:Headers: {} 2023-05-31T20:39:10.766 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:39:10.767 DEBUG:boto:Port: 8001 2023-05-31T20:39:10.767 DEBUG:boto:Params: {} 2023-05-31T20:39:10.768 DEBUG:boto:Token: None 2023-05-31T20:39:10.768 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:39:10 GMT /pklxkd-58/obj00000000000000000017?versionId=null 2023-05-31T20:39:10.769 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:Sl5Be0/iuTgv8OmhauX5m0ewRSo= 2023-05-31T20:39:10.769 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:39:10 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:Sl5Be0/iuTgv8OmhauX5m0ewRSo=', 'Content-Length': '0'} 2023-05-31T20:39:10.771 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:38:22 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001c2ca7c97ec4fb4b-006477b06e-1c17-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:39:10 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:39:10.772 DEBUG:boto:path=/pklxkd-58/obj00000000000000000017 2023-05-31T20:39:10.772 DEBUG:boto:auth_path=/pklxkd-58/obj00000000000000000017 2023-05-31T20:39:10.772 DEBUG:boto:path=/pklxkd-58/obj00000000000000000017?versionId=null 2023-05-31T20:39:10.773 DEBUG:boto:auth_path=/pklxkd-58/obj00000000000000000017?versionId=null 2023-05-31T20:39:10.773 DEBUG:boto:Method: GET 2023-05-31T20:39:10.774 DEBUG:boto:Path: /pklxkd-58/obj00000000000000000017?versionId=null 2023-05-31T20:39:10.774 DEBUG:boto:Data: 2023-05-31T20:39:10.774 DEBUG:boto:Headers: {} 2023-05-31T20:39:10.775 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:39:10.775 DEBUG:boto:Port: 8000 2023-05-31T20:39:10.776 DEBUG:boto:Params: {} 2023-05-31T20:39:10.776 DEBUG:boto:Token: None 2023-05-31T20:39:10.777 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:39:10 GMT /pklxkd-58/obj00000000000000000017?versionId=null 2023-05-31T20:39:10.777 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:Sl5Be0/iuTgv8OmhauX5m0ewRSo= 2023-05-31T20:39:10.777 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:39:10 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:Sl5Be0/iuTgv8OmhauX5m0ewRSo=', 'Content-Length': '0'} 2023-05-31T20:39:10.779 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:38:22 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008b2a69bd6e60e391-006477b06e-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:39:10 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:39:10.780 DEBUG:boto:path=/pklxkd-58/ 2023-05-31T20:39:10.781 DEBUG:boto:auth_path=/pklxkd-58/ 2023-05-31T20:39:10.781 DEBUG:boto:path=/pklxkd-58/?prefix=obj00000000000000000017 2023-05-31T20:39:10.781 DEBUG:boto:auth_path=/pklxkd-58/?prefix=obj00000000000000000017 2023-05-31T20:39:10.782 DEBUG:boto:Method: GET 2023-05-31T20:39:10.782 DEBUG:boto:Path: /pklxkd-58/?prefix=obj00000000000000000017 2023-05-31T20:39:10.782 DEBUG:boto:Data: 2023-05-31T20:39:10.783 DEBUG:boto:Headers: {} 2023-05-31T20:39:10.783 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:39:10.783 DEBUG:boto:Port: 8001 2023-05-31T20:39:10.784 DEBUG:boto:Params: {} 2023-05-31T20:39:10.784 DEBUG:boto:Token: None 2023-05-31T20:39:10.785 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:39:10 GMT /pklxkd-58/ 2023-05-31T20:39:10.785 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:rToF82M8NN6MyNReCsDOOu4EGBY= 2023-05-31T20:39:10.786 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:39:10 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:rToF82M8NN6MyNReCsDOOu4EGBY=', 'Content-Length': '0'} 2023-05-31T20:39:10.788 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000099a8f881e204c4bc-006477b06e-1c17-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:39:10 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:39:10.789 DEBUG:boto:b'pklxkd-58obj000000000000000000171000falseobj000000000000000000172023-05-31T20:38:22.548Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:39:10.789 DEBUG:boto:path=/pklxkd-58/ 2023-05-31T20:39:10.790 DEBUG:boto:auth_path=/pklxkd-58/ 2023-05-31T20:39:10.790 DEBUG:boto:path=/pklxkd-58/?prefix=obj00000000000000000017 2023-05-31T20:39:10.791 DEBUG:boto:auth_path=/pklxkd-58/?prefix=obj00000000000000000017 2023-05-31T20:39:10.791 DEBUG:boto:Method: GET 2023-05-31T20:39:10.791 DEBUG:boto:Path: /pklxkd-58/?prefix=obj00000000000000000017 2023-05-31T20:39:10.792 DEBUG:boto:Data: 2023-05-31T20:39:10.792 DEBUG:boto:Headers: {} 2023-05-31T20:39:10.792 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:39:10.793 DEBUG:boto:Port: 8000 2023-05-31T20:39:10.793 DEBUG:boto:Params: {} 2023-05-31T20:39:10.794 DEBUG:boto:Token: None 2023-05-31T20:39:10.794 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:39:10 GMT /pklxkd-58/ 2023-05-31T20:39:10.794 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:rToF82M8NN6MyNReCsDOOu4EGBY= 2023-05-31T20:39:10.795 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:39:10 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:rToF82M8NN6MyNReCsDOOu4EGBY=', 'Content-Length': '0'} 2023-05-31T20:39:10.799 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f982ddda975ec2ba-006477b06e-1630-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:39:10 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:39:10.799 DEBUG:boto:b'pklxkd-58obj000000000000000000171000falseobj000000000000000000172023-05-31T20:38:22.548Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:39:10.800 DEBUG:boto:path=/pklxkd-58/obj00000000000000000017 2023-05-31T20:39:10.801 DEBUG:boto:auth_path=/pklxkd-58/obj00000000000000000017 2023-05-31T20:39:10.801 DEBUG:boto:Method: HEAD 2023-05-31T20:39:10.801 DEBUG:boto:Path: /pklxkd-58/obj00000000000000000017 2023-05-31T20:39:10.802 DEBUG:boto:Data: 2023-05-31T20:39:10.802 DEBUG:boto:Headers: {} 2023-05-31T20:39:10.802 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:39:10.803 DEBUG:boto:Port: 8001 2023-05-31T20:39:10.803 DEBUG:boto:Params: {} 2023-05-31T20:39:10.803 DEBUG:boto:Token: None 2023-05-31T20:39:10.804 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:39:10 GMT /pklxkd-58/obj00000000000000000017 2023-05-31T20:39:10.804 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:8Rr7XKo+cbSEZFW9cnuD7PqpeF8= 2023-05-31T20:39:10.805 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:39:10 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:8Rr7XKo+cbSEZFW9cnuD7PqpeF8=', 'Content-Length': '0'} 2023-05-31T20:39:10.806 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:38:22 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000266685208fc2c794-006477b06e-1c17-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:39:10 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:39:10.807 DEBUG:boto:path=/pklxkd-58/obj00000000000000000017 2023-05-31T20:39:10.808 DEBUG:boto:auth_path=/pklxkd-58/obj00000000000000000017 2023-05-31T20:39:10.808 DEBUG:boto:Method: HEAD 2023-05-31T20:39:10.809 DEBUG:boto:Path: /pklxkd-58/obj00000000000000000017 2023-05-31T20:39:10.809 DEBUG:boto:Data: 2023-05-31T20:39:10.809 DEBUG:boto:Headers: {} 2023-05-31T20:39:10.810 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:39:10.810 DEBUG:boto:Port: 8000 2023-05-31T20:39:10.810 DEBUG:boto:Params: {} 2023-05-31T20:39:10.811 DEBUG:boto:Token: None 2023-05-31T20:39:10.811 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:39:10 GMT /pklxkd-58/obj00000000000000000017 2023-05-31T20:39:10.812 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:8Rr7XKo+cbSEZFW9cnuD7PqpeF8= 2023-05-31T20:39:10.812 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:39:10 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:8Rr7XKo+cbSEZFW9cnuD7PqpeF8=', 'Content-Length': '0'} 2023-05-31T20:39:10.814 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:38:22 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000401a89f39c44c33c-006477b06e-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:39:10 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:39:10.815 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000017 2023-05-31T20:39:10.815 DEBUG:boto:path=/pklxkd-58/obj00000000000000000017 2023-05-31T20:39:10.815 DEBUG:boto:auth_path=/pklxkd-58/obj00000000000000000017 2023-05-31T20:39:10.816 DEBUG:boto:Method: GET 2023-05-31T20:39:10.816 DEBUG:boto:Path: /pklxkd-58/obj00000000000000000017 2023-05-31T20:39:10.817 DEBUG:boto:Data: 2023-05-31T20:39:10.817 DEBUG:boto:Headers: {} 2023-05-31T20:39:10.817 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:39:10.818 DEBUG:boto:Port: 8001 2023-05-31T20:39:10.818 DEBUG:boto:Params: {} 2023-05-31T20:39:10.818 DEBUG:boto:Token: None 2023-05-31T20:39:10.819 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:39:10 GMT /pklxkd-58/obj00000000000000000017 2023-05-31T20:39:10.819 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:B9dS2CmdufPYhpSaJQ5ZToyYOc4= 2023-05-31T20:39:10.820 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:39:10 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:B9dS2CmdufPYhpSaJQ5ZToyYOc4=', 'Content-Length': '0'} 2023-05-31T20:39:10.822 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:38:22 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d0e1c24a67b1afc8-006477b06e-1c17-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:39:10 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:39:10.823 DEBUG:boto:path=/pklxkd-58/obj00000000000000000017 2023-05-31T20:39:10.823 DEBUG:boto:auth_path=/pklxkd-58/obj00000000000000000017 2023-05-31T20:39:10.823 DEBUG:boto:Method: GET 2023-05-31T20:39:10.824 DEBUG:boto:Path: /pklxkd-58/obj00000000000000000017 2023-05-31T20:39:10.824 DEBUG:boto:Data: 2023-05-31T20:39:10.825 DEBUG:boto:Headers: {} 2023-05-31T20:39:10.825 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:39:10.825 DEBUG:boto:Port: 8000 2023-05-31T20:39:10.826 DEBUG:boto:Params: {} 2023-05-31T20:39:10.826 DEBUG:boto:Token: None 2023-05-31T20:39:10.827 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:39:10 GMT /pklxkd-58/obj00000000000000000017 2023-05-31T20:39:10.827 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:B9dS2CmdufPYhpSaJQ5ZToyYOc4= 2023-05-31T20:39:10.827 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:39:10 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:B9dS2CmdufPYhpSaJQ5ZToyYOc4=', 'Content-Length': '0'} 2023-05-31T20:39:10.830 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:38:22 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e66431c65e7273b0-006477b06e-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:39:10 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:39:10.831 DEBUG:boto:path=/pklxkd-58/ 2023-05-31T20:39:10.831 DEBUG:boto:auth_path=/pklxkd-58/ 2023-05-31T20:39:10.831 DEBUG:boto:path=/pklxkd-58/?prefix=obj00000000000000000017 2023-05-31T20:39:10.832 DEBUG:boto:auth_path=/pklxkd-58/?prefix=obj00000000000000000017 2023-05-31T20:39:10.833 DEBUG:boto:Method: GET 2023-05-31T20:39:10.833 DEBUG:boto:Path: /pklxkd-58/?prefix=obj00000000000000000017 2023-05-31T20:39:10.833 DEBUG:boto:Data: 2023-05-31T20:39:10.834 DEBUG:boto:Headers: {} 2023-05-31T20:39:10.834 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:39:10.835 DEBUG:boto:Port: 8001 2023-05-31T20:39:10.835 DEBUG:boto:Params: {} 2023-05-31T20:39:10.835 DEBUG:boto:Token: None 2023-05-31T20:39:10.836 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:39:10 GMT /pklxkd-58/ 2023-05-31T20:39:10.837 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:rToF82M8NN6MyNReCsDOOu4EGBY= 2023-05-31T20:39:10.837 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:39:10 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:rToF82M8NN6MyNReCsDOOu4EGBY=', 'Content-Length': '0'} 2023-05-31T20:39:10.840 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ab030d717ec5b3e9-006477b06e-1c17-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:39:10 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:39:10.841 DEBUG:boto:b'pklxkd-58obj000000000000000000171000falseobj000000000000000000172023-05-31T20:38:22.548Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:39:10.842 DEBUG:boto:path=/pklxkd-58/ 2023-05-31T20:39:10.843 DEBUG:boto:auth_path=/pklxkd-58/ 2023-05-31T20:39:10.844 DEBUG:boto:path=/pklxkd-58/?prefix=obj00000000000000000017 2023-05-31T20:39:10.845 DEBUG:boto:auth_path=/pklxkd-58/?prefix=obj00000000000000000017 2023-05-31T20:39:10.846 DEBUG:boto:Method: GET 2023-05-31T20:39:10.847 DEBUG:boto:Path: /pklxkd-58/?prefix=obj00000000000000000017 2023-05-31T20:39:10.848 DEBUG:boto:Data: 2023-05-31T20:39:10.848 DEBUG:boto:Headers: {} 2023-05-31T20:39:10.850 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:39:10.850 DEBUG:boto:Port: 8000 2023-05-31T20:39:10.851 DEBUG:boto:Params: {} 2023-05-31T20:39:10.852 DEBUG:boto:Token: None 2023-05-31T20:39:10.853 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:39:10 GMT /pklxkd-58/ 2023-05-31T20:39:10.854 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:rToF82M8NN6MyNReCsDOOu4EGBY= 2023-05-31T20:39:10.854 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:39:10 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:rToF82M8NN6MyNReCsDOOu4EGBY=', 'Content-Length': '0'} 2023-05-31T20:39:10.859 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000002cfede84ed8b12e5-006477b06e-1630-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:39:10 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:39:10.860 DEBUG:boto:b'pklxkd-58obj000000000000000000171000falseobj000000000000000000172023-05-31T20:38:22.548Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:39:10.861 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000018 2023-05-31T20:39:10.862 DEBUG:boto:path=/pklxkd-58/obj00000000000000000018 2023-05-31T20:39:10.863 DEBUG:boto:auth_path=/pklxkd-58/obj00000000000000000018 2023-05-31T20:39:10.863 DEBUG:boto:path=/pklxkd-58/obj00000000000000000018?versionId=null 2023-05-31T20:39:10.864 DEBUG:boto:auth_path=/pklxkd-58/obj00000000000000000018?versionId=null 2023-05-31T20:39:10.865 DEBUG:boto:Method: GET 2023-05-31T20:39:10.866 DEBUG:boto:Path: /pklxkd-58/obj00000000000000000018?versionId=null 2023-05-31T20:39:10.866 DEBUG:boto:Data: 2023-05-31T20:39:10.867 DEBUG:boto:Headers: {} 2023-05-31T20:39:10.867 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:39:10.868 DEBUG:boto:Port: 8001 2023-05-31T20:39:10.868 DEBUG:boto:Params: {} 2023-05-31T20:39:10.869 DEBUG:boto:Token: None 2023-05-31T20:39:10.869 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:39:10 GMT /pklxkd-58/obj00000000000000000018?versionId=null 2023-05-31T20:39:10.870 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:hD9WagDglD1cYC/V3gMyRSNZXdU= 2023-05-31T20:39:10.870 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:39:10 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:hD9WagDglD1cYC/V3gMyRSNZXdU=', 'Content-Length': '0'} 2023-05-31T20:39:10.873 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:38:21 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000cdf44ee30d005a5c-006477b06e-1c17-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:39:10 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:39:10.874 DEBUG:boto:path=/pklxkd-58/obj00000000000000000018 2023-05-31T20:39:10.875 DEBUG:boto:auth_path=/pklxkd-58/obj00000000000000000018 2023-05-31T20:39:10.876 DEBUG:boto:path=/pklxkd-58/obj00000000000000000018?versionId=null 2023-05-31T20:39:10.876 DEBUG:boto:auth_path=/pklxkd-58/obj00000000000000000018?versionId=null 2023-05-31T20:39:10.877 DEBUG:boto:Method: GET 2023-05-31T20:39:10.878 DEBUG:boto:Path: /pklxkd-58/obj00000000000000000018?versionId=null 2023-05-31T20:39:10.878 DEBUG:boto:Data: 2023-05-31T20:39:10.879 DEBUG:boto:Headers: {} 2023-05-31T20:39:10.880 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:39:10.881 DEBUG:boto:Port: 8000 2023-05-31T20:39:10.882 DEBUG:boto:Params: {} 2023-05-31T20:39:10.883 DEBUG:boto:Token: None 2023-05-31T20:39:10.883 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:39:10 GMT /pklxkd-58/obj00000000000000000018?versionId=null 2023-05-31T20:39:10.884 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:hD9WagDglD1cYC/V3gMyRSNZXdU= 2023-05-31T20:39:10.885 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:39:10 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:hD9WagDglD1cYC/V3gMyRSNZXdU=', 'Content-Length': '0'} 2023-05-31T20:39:10.887 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:38:21 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001a7a61fec7862d85-006477b06e-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:39:10 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:39:10.928 DEBUG:boto:path=/pklxkd-58/obj00000000000000000018 2023-05-31T20:39:10.929 DEBUG:boto:auth_path=/pklxkd-58/obj00000000000000000018 2023-05-31T20:39:10.930 DEBUG:boto:path=/pklxkd-58/obj00000000000000000018?versionId=null 2023-05-31T20:39:10.930 DEBUG:boto:auth_path=/pklxkd-58/obj00000000000000000018?versionId=null 2023-05-31T20:39:10.931 DEBUG:boto:Method: HEAD 2023-05-31T20:39:10.931 DEBUG:boto:Path: /pklxkd-58/obj00000000000000000018?versionId=null 2023-05-31T20:39:10.931 DEBUG:boto:Data: 2023-05-31T20:39:10.932 DEBUG:boto:Headers: {} 2023-05-31T20:39:10.932 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:39:10.933 DEBUG:boto:Port: 8001 2023-05-31T20:39:10.934 DEBUG:boto:Params: {} 2023-05-31T20:39:10.935 DEBUG:boto:Token: None 2023-05-31T20:39:10.936 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:39:10 GMT /pklxkd-58/obj00000000000000000018?versionId=null 2023-05-31T20:39:10.936 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:5Xx4rsgBOeI59BIzBParn8ramqg= 2023-05-31T20:39:10.937 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:39:10 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:5Xx4rsgBOeI59BIzBParn8ramqg=', 'Content-Length': '0'} 2023-05-31T20:39:10.939 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:38:21 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009be8ed1ec2add36d-006477b06e-1c17-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:39:10 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:39:10.940 DEBUG:boto:path=/pklxkd-58/obj00000000000000000018 2023-05-31T20:39:10.940 DEBUG:boto:auth_path=/pklxkd-58/obj00000000000000000018 2023-05-31T20:39:10.941 DEBUG:boto:path=/pklxkd-58/obj00000000000000000018?versionId=null 2023-05-31T20:39:10.941 DEBUG:boto:auth_path=/pklxkd-58/obj00000000000000000018?versionId=null 2023-05-31T20:39:10.942 DEBUG:boto:Method: HEAD 2023-05-31T20:39:10.942 DEBUG:boto:Path: /pklxkd-58/obj00000000000000000018?versionId=null 2023-05-31T20:39:10.943 DEBUG:boto:Data: 2023-05-31T20:39:10.943 DEBUG:boto:Headers: {} 2023-05-31T20:39:10.943 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:39:10.944 DEBUG:boto:Port: 8000 2023-05-31T20:39:10.944 DEBUG:boto:Params: {} 2023-05-31T20:39:10.945 DEBUG:boto:Token: None 2023-05-31T20:39:10.946 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:39:10 GMT /pklxkd-58/obj00000000000000000018?versionId=null 2023-05-31T20:39:10.947 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:5Xx4rsgBOeI59BIzBParn8ramqg= 2023-05-31T20:39:10.948 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:39:10 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:5Xx4rsgBOeI59BIzBParn8ramqg=', 'Content-Length': '0'} 2023-05-31T20:39:10.951 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:38:21 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000be37d5da8e95b29c-006477b06e-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:39:10 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:39:10.951 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000018 2023-05-31T20:39:10.952 DEBUG:boto:path=/pklxkd-58/obj00000000000000000018 2023-05-31T20:39:10.953 DEBUG:boto:auth_path=/pklxkd-58/obj00000000000000000018 2023-05-31T20:39:10.953 DEBUG:boto:path=/pklxkd-58/obj00000000000000000018?versionId=null 2023-05-31T20:39:10.953 DEBUG:boto:auth_path=/pklxkd-58/obj00000000000000000018?versionId=null 2023-05-31T20:39:10.954 DEBUG:boto:Method: GET 2023-05-31T20:39:10.954 DEBUG:boto:Path: /pklxkd-58/obj00000000000000000018?versionId=null 2023-05-31T20:39:10.955 DEBUG:boto:Data: 2023-05-31T20:39:10.956 DEBUG:boto:Headers: {} 2023-05-31T20:39:10.957 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:39:10.957 DEBUG:boto:Port: 8001 2023-05-31T20:39:10.958 DEBUG:boto:Params: {} 2023-05-31T20:39:10.959 DEBUG:boto:Token: None 2023-05-31T20:39:10.960 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:39:10 GMT /pklxkd-58/obj00000000000000000018?versionId=null 2023-05-31T20:39:10.960 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:hD9WagDglD1cYC/V3gMyRSNZXdU= 2023-05-31T20:39:10.961 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:39:10 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:hD9WagDglD1cYC/V3gMyRSNZXdU=', 'Content-Length': '0'} 2023-05-31T20:39:10.963 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:38:21 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002a8e3e83971c0b75-006477b06e-1c17-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:39:10 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:39:10.964 DEBUG:boto:path=/pklxkd-58/obj00000000000000000018 2023-05-31T20:39:10.964 DEBUG:boto:auth_path=/pklxkd-58/obj00000000000000000018 2023-05-31T20:39:10.965 DEBUG:boto:path=/pklxkd-58/obj00000000000000000018?versionId=null 2023-05-31T20:39:10.965 DEBUG:boto:auth_path=/pklxkd-58/obj00000000000000000018?versionId=null 2023-05-31T20:39:10.965 DEBUG:boto:Method: GET 2023-05-31T20:39:10.966 DEBUG:boto:Path: /pklxkd-58/obj00000000000000000018?versionId=null 2023-05-31T20:39:10.966 DEBUG:boto:Data: 2023-05-31T20:39:10.967 DEBUG:boto:Headers: {} 2023-05-31T20:39:10.967 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:39:10.967 DEBUG:boto:Port: 8000 2023-05-31T20:39:10.968 DEBUG:boto:Params: {} 2023-05-31T20:39:10.968 DEBUG:boto:Token: None 2023-05-31T20:39:10.969 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:39:10 GMT /pklxkd-58/obj00000000000000000018?versionId=null 2023-05-31T20:39:10.969 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:hD9WagDglD1cYC/V3gMyRSNZXdU= 2023-05-31T20:39:10.970 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:39:10 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:hD9WagDglD1cYC/V3gMyRSNZXdU=', 'Content-Length': '0'} 2023-05-31T20:39:10.972 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:38:21 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c91a48411495848e-006477b06e-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:39:10 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:39:10.973 DEBUG:boto:path=/pklxkd-58/ 2023-05-31T20:39:10.973 DEBUG:boto:auth_path=/pklxkd-58/ 2023-05-31T20:39:10.974 DEBUG:boto:path=/pklxkd-58/?prefix=obj00000000000000000018 2023-05-31T20:39:10.974 DEBUG:boto:auth_path=/pklxkd-58/?prefix=obj00000000000000000018 2023-05-31T20:39:10.975 DEBUG:boto:Method: GET 2023-05-31T20:39:10.975 DEBUG:boto:Path: /pklxkd-58/?prefix=obj00000000000000000018 2023-05-31T20:39:10.975 DEBUG:boto:Data: 2023-05-31T20:39:10.976 DEBUG:boto:Headers: {} 2023-05-31T20:39:10.976 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:39:10.977 DEBUG:boto:Port: 8001 2023-05-31T20:39:10.977 DEBUG:boto:Params: {} 2023-05-31T20:39:10.978 DEBUG:boto:Token: None 2023-05-31T20:39:10.978 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:39:10 GMT /pklxkd-58/ 2023-05-31T20:39:10.979 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:rToF82M8NN6MyNReCsDOOu4EGBY= 2023-05-31T20:39:10.979 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:39:10 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:rToF82M8NN6MyNReCsDOOu4EGBY=', 'Content-Length': '0'} 2023-05-31T20:39:10.981 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000011790225f914648e-006477b06e-1c17-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:39:10 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:39:10.982 DEBUG:boto:b'pklxkd-58obj000000000000000000181000falseobj000000000000000000182023-05-31T20:38:21.887Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:39:10.983 DEBUG:boto:path=/pklxkd-58/ 2023-05-31T20:39:10.983 DEBUG:boto:auth_path=/pklxkd-58/ 2023-05-31T20:39:10.983 DEBUG:boto:path=/pklxkd-58/?prefix=obj00000000000000000018 2023-05-31T20:39:10.984 DEBUG:boto:auth_path=/pklxkd-58/?prefix=obj00000000000000000018 2023-05-31T20:39:10.984 DEBUG:boto:Method: GET 2023-05-31T20:39:10.984 DEBUG:boto:Path: /pklxkd-58/?prefix=obj00000000000000000018 2023-05-31T20:39:10.985 DEBUG:boto:Data: 2023-05-31T20:39:10.985 DEBUG:boto:Headers: {} 2023-05-31T20:39:10.986 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:39:10.986 DEBUG:boto:Port: 8000 2023-05-31T20:39:10.986 DEBUG:boto:Params: {} 2023-05-31T20:39:10.987 DEBUG:boto:Token: None 2023-05-31T20:39:10.987 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:39:10 GMT /pklxkd-58/ 2023-05-31T20:39:10.988 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:rToF82M8NN6MyNReCsDOOu4EGBY= 2023-05-31T20:39:10.988 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:39:10 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:rToF82M8NN6MyNReCsDOOu4EGBY=', 'Content-Length': '0'} 2023-05-31T20:39:10.992 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000009d2fc847e34f2996-006477b06e-1630-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:39:10 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:39:10.993 DEBUG:boto:b'pklxkd-58obj000000000000000000181000falseobj000000000000000000182023-05-31T20:38:21.887Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:39:10.993 DEBUG:boto:path=/pklxkd-58/obj00000000000000000018 2023-05-31T20:39:10.994 DEBUG:boto:auth_path=/pklxkd-58/obj00000000000000000018 2023-05-31T20:39:10.994 DEBUG:boto:Method: HEAD 2023-05-31T20:39:10.994 DEBUG:boto:Path: /pklxkd-58/obj00000000000000000018 2023-05-31T20:39:10.995 DEBUG:boto:Data: 2023-05-31T20:39:10.995 DEBUG:boto:Headers: {} 2023-05-31T20:39:10.995 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:39:10.996 DEBUG:boto:Port: 8001 2023-05-31T20:39:10.996 DEBUG:boto:Params: {} 2023-05-31T20:39:10.997 DEBUG:boto:Token: None 2023-05-31T20:39:10.997 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:39:10 GMT /pklxkd-58/obj00000000000000000018 2023-05-31T20:39:10.997 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:E/vZSo20yCe2XbBjA3wrtbDYIPk= 2023-05-31T20:39:10.998 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:39:10 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:E/vZSo20yCe2XbBjA3wrtbDYIPk=', 'Content-Length': '0'} 2023-05-31T20:39:11.000 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:38:21 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c97da9860c15f1e6-006477b06e-1c17-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:39:10 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:39:11.001 DEBUG:boto:path=/pklxkd-58/obj00000000000000000018 2023-05-31T20:39:11.001 DEBUG:boto:auth_path=/pklxkd-58/obj00000000000000000018 2023-05-31T20:39:11.001 DEBUG:boto:Method: HEAD 2023-05-31T20:39:11.002 DEBUG:boto:Path: /pklxkd-58/obj00000000000000000018 2023-05-31T20:39:11.002 DEBUG:boto:Data: 2023-05-31T20:39:11.002 DEBUG:boto:Headers: {} 2023-05-31T20:39:11.003 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:39:11.003 DEBUG:boto:Port: 8000 2023-05-31T20:39:11.004 DEBUG:boto:Params: {} 2023-05-31T20:39:11.004 DEBUG:boto:Token: None 2023-05-31T20:39:11.004 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:39:11 GMT /pklxkd-58/obj00000000000000000018 2023-05-31T20:39:11.005 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:Z/RGPlzi/ipNB+un0GgdmWQSpOU= 2023-05-31T20:39:11.005 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:39:11 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:Z/RGPlzi/ipNB+un0GgdmWQSpOU=', 'Content-Length': '0'} 2023-05-31T20:39:11.007 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:38:21 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000543676e63697af1b-006477b06f-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:39:11 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:39:11.008 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000018 2023-05-31T20:39:11.008 DEBUG:boto:path=/pklxkd-58/obj00000000000000000018 2023-05-31T20:39:11.009 DEBUG:boto:auth_path=/pklxkd-58/obj00000000000000000018 2023-05-31T20:39:11.009 DEBUG:boto:Method: GET 2023-05-31T20:39:11.010 DEBUG:boto:Path: /pklxkd-58/obj00000000000000000018 2023-05-31T20:39:11.010 DEBUG:boto:Data: 2023-05-31T20:39:11.011 DEBUG:boto:Headers: {} 2023-05-31T20:39:11.011 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:39:11.011 DEBUG:boto:Port: 8001 2023-05-31T20:39:11.012 DEBUG:boto:Params: {} 2023-05-31T20:39:11.012 DEBUG:boto:Token: None 2023-05-31T20:39:11.013 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:39:11 GMT /pklxkd-58/obj00000000000000000018 2023-05-31T20:39:11.013 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:eOwC47OcUxhcIoy7ClmabL6CE/g= 2023-05-31T20:39:11.013 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:39:11 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:eOwC47OcUxhcIoy7ClmabL6CE/g=', 'Content-Length': '0'} 2023-05-31T20:39:11.015 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:38:21 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a35d390eed279e72-006477b06f-1c17-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:39:11 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:39:11.056 DEBUG:boto:path=/pklxkd-58/obj00000000000000000018 2023-05-31T20:39:11.057 DEBUG:boto:auth_path=/pklxkd-58/obj00000000000000000018 2023-05-31T20:39:11.058 DEBUG:boto:Method: GET 2023-05-31T20:39:11.058 DEBUG:boto:Path: /pklxkd-58/obj00000000000000000018 2023-05-31T20:39:11.058 DEBUG:boto:Data: 2023-05-31T20:39:11.059 DEBUG:boto:Headers: {} 2023-05-31T20:39:11.059 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:39:11.059 DEBUG:boto:Port: 8000 2023-05-31T20:39:11.060 DEBUG:boto:Params: {} 2023-05-31T20:39:11.060 DEBUG:boto:Token: None 2023-05-31T20:39:11.061 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:39:11 GMT /pklxkd-58/obj00000000000000000018 2023-05-31T20:39:11.061 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:eOwC47OcUxhcIoy7ClmabL6CE/g= 2023-05-31T20:39:11.062 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:39:11 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:eOwC47OcUxhcIoy7ClmabL6CE/g=', 'Content-Length': '0'} 2023-05-31T20:39:11.064 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:38:21 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000081141a2a9936a65-006477b06f-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:39:11 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:39:11.065 DEBUG:boto:path=/pklxkd-58/ 2023-05-31T20:39:11.066 DEBUG:boto:auth_path=/pklxkd-58/ 2023-05-31T20:39:11.066 DEBUG:boto:path=/pklxkd-58/?prefix=obj00000000000000000018 2023-05-31T20:39:11.066 DEBUG:boto:auth_path=/pklxkd-58/?prefix=obj00000000000000000018 2023-05-31T20:39:11.067 DEBUG:boto:Method: GET 2023-05-31T20:39:11.067 DEBUG:boto:Path: /pklxkd-58/?prefix=obj00000000000000000018 2023-05-31T20:39:11.068 DEBUG:boto:Data: 2023-05-31T20:39:11.068 DEBUG:boto:Headers: {} 2023-05-31T20:39:11.069 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:39:11.069 DEBUG:boto:Port: 8001 2023-05-31T20:39:11.069 DEBUG:boto:Params: {} 2023-05-31T20:39:11.070 DEBUG:boto:Token: None 2023-05-31T20:39:11.070 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:39:11 GMT /pklxkd-58/ 2023-05-31T20:39:11.071 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:GHNSu1S8h4v2Qp+durSNxgCJG9E= 2023-05-31T20:39:11.071 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:39:11 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:GHNSu1S8h4v2Qp+durSNxgCJG9E=', 'Content-Length': '0'} 2023-05-31T20:39:11.074 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000000e96dc486a551712-006477b06f-1c17-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:39:11 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:39:11.075 DEBUG:boto:b'pklxkd-58obj000000000000000000181000falseobj000000000000000000182023-05-31T20:38:21.887Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:39:11.075 DEBUG:boto:path=/pklxkd-58/ 2023-05-31T20:39:11.076 DEBUG:boto:auth_path=/pklxkd-58/ 2023-05-31T20:39:11.077 DEBUG:boto:path=/pklxkd-58/?prefix=obj00000000000000000018 2023-05-31T20:39:11.077 DEBUG:boto:auth_path=/pklxkd-58/?prefix=obj00000000000000000018 2023-05-31T20:39:11.077 DEBUG:boto:Method: GET 2023-05-31T20:39:11.078 DEBUG:boto:Path: /pklxkd-58/?prefix=obj00000000000000000018 2023-05-31T20:39:11.078 DEBUG:boto:Data: 2023-05-31T20:39:11.079 DEBUG:boto:Headers: {} 2023-05-31T20:39:11.079 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:39:11.079 DEBUG:boto:Port: 8000 2023-05-31T20:39:11.080 DEBUG:boto:Params: {} 2023-05-31T20:39:11.080 DEBUG:boto:Token: None 2023-05-31T20:39:11.081 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:39:11 GMT /pklxkd-58/ 2023-05-31T20:39:11.081 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:GHNSu1S8h4v2Qp+durSNxgCJG9E= 2023-05-31T20:39:11.081 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:39:11 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:GHNSu1S8h4v2Qp+durSNxgCJG9E=', 'Content-Length': '0'} 2023-05-31T20:39:11.086 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e8987dc644746df3-006477b06f-1630-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:39:11 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:39:11.087 DEBUG:boto:b'pklxkd-58obj000000000000000000181000falseobj000000000000000000182023-05-31T20:38:21.887Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:39:11.087 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000019 2023-05-31T20:39:11.088 DEBUG:boto:path=/pklxkd-58/obj00000000000000000019 2023-05-31T20:39:11.089 DEBUG:boto:auth_path=/pklxkd-58/obj00000000000000000019 2023-05-31T20:39:11.089 DEBUG:boto:path=/pklxkd-58/obj00000000000000000019?versionId=null 2023-05-31T20:39:11.090 DEBUG:boto:auth_path=/pklxkd-58/obj00000000000000000019?versionId=null 2023-05-31T20:39:11.090 DEBUG:boto:Method: GET 2023-05-31T20:39:11.091 DEBUG:boto:Path: /pklxkd-58/obj00000000000000000019?versionId=null 2023-05-31T20:39:11.091 DEBUG:boto:Data: 2023-05-31T20:39:11.091 DEBUG:boto:Headers: {} 2023-05-31T20:39:11.092 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:39:11.092 DEBUG:boto:Port: 8001 2023-05-31T20:39:11.093 DEBUG:boto:Params: {} 2023-05-31T20:39:11.093 DEBUG:boto:Token: None 2023-05-31T20:39:11.094 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:39:11 GMT /pklxkd-58/obj00000000000000000019?versionId=null 2023-05-31T20:39:11.094 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:OO/JN+SpJiUagU7LmC2HGDpStsk= 2023-05-31T20:39:11.095 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:39:11 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:OO/JN+SpJiUagU7LmC2HGDpStsk=', 'Content-Length': '0'} 2023-05-31T20:39:11.097 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:38:21 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b5955d936c124e95-006477b06f-1c17-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:39:11 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:39:11.098 DEBUG:boto:path=/pklxkd-58/obj00000000000000000019 2023-05-31T20:39:11.098 DEBUG:boto:auth_path=/pklxkd-58/obj00000000000000000019 2023-05-31T20:39:11.099 DEBUG:boto:path=/pklxkd-58/obj00000000000000000019?versionId=null 2023-05-31T20:39:11.099 DEBUG:boto:auth_path=/pklxkd-58/obj00000000000000000019?versionId=null 2023-05-31T20:39:11.100 DEBUG:boto:Method: GET 2023-05-31T20:39:11.100 DEBUG:boto:Path: /pklxkd-58/obj00000000000000000019?versionId=null 2023-05-31T20:39:11.101 DEBUG:boto:Data: 2023-05-31T20:39:11.101 DEBUG:boto:Headers: {} 2023-05-31T20:39:11.101 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:39:11.102 DEBUG:boto:Port: 8000 2023-05-31T20:39:11.102 DEBUG:boto:Params: {} 2023-05-31T20:39:11.103 DEBUG:boto:Token: None 2023-05-31T20:39:11.103 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:39:11 GMT /pklxkd-58/obj00000000000000000019?versionId=null 2023-05-31T20:39:11.104 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:OO/JN+SpJiUagU7LmC2HGDpStsk= 2023-05-31T20:39:11.104 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:39:11 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:OO/JN+SpJiUagU7LmC2HGDpStsk=', 'Content-Length': '0'} 2023-05-31T20:39:11.106 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:38:21 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003d6b76db3ee3703f-006477b06f-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:39:11 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:39:11.107 DEBUG:boto:path=/pklxkd-58/obj00000000000000000019 2023-05-31T20:39:11.108 DEBUG:boto:auth_path=/pklxkd-58/obj00000000000000000019 2023-05-31T20:39:11.108 DEBUG:boto:path=/pklxkd-58/obj00000000000000000019?versionId=null 2023-05-31T20:39:11.109 DEBUG:boto:auth_path=/pklxkd-58/obj00000000000000000019?versionId=null 2023-05-31T20:39:11.109 DEBUG:boto:Method: HEAD 2023-05-31T20:39:11.110 DEBUG:boto:Path: /pklxkd-58/obj00000000000000000019?versionId=null 2023-05-31T20:39:11.110 DEBUG:boto:Data: 2023-05-31T20:39:11.110 DEBUG:boto:Headers: {} 2023-05-31T20:39:11.111 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:39:11.111 DEBUG:boto:Port: 8001 2023-05-31T20:39:11.111 DEBUG:boto:Params: {} 2023-05-31T20:39:11.112 DEBUG:boto:Token: None 2023-05-31T20:39:11.112 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:39:11 GMT /pklxkd-58/obj00000000000000000019?versionId=null 2023-05-31T20:39:11.113 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:QpZXf3zLsfUD04MOe94/08U/CIo= 2023-05-31T20:39:11.113 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:39:11 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:QpZXf3zLsfUD04MOe94/08U/CIo=', 'Content-Length': '0'} 2023-05-31T20:39:11.115 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:38:21 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c4be6366cbe9896c-006477b06f-1c17-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:39:11 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:39:11.115 DEBUG:boto:path=/pklxkd-58/obj00000000000000000019 2023-05-31T20:39:11.116 DEBUG:boto:auth_path=/pklxkd-58/obj00000000000000000019 2023-05-31T20:39:11.116 DEBUG:boto:path=/pklxkd-58/obj00000000000000000019?versionId=null 2023-05-31T20:39:11.117 DEBUG:boto:auth_path=/pklxkd-58/obj00000000000000000019?versionId=null 2023-05-31T20:39:11.117 DEBUG:boto:Method: HEAD 2023-05-31T20:39:11.119 DEBUG:boto:Path: /pklxkd-58/obj00000000000000000019?versionId=null 2023-05-31T20:39:11.119 DEBUG:boto:Data: 2023-05-31T20:39:11.120 DEBUG:boto:Headers: {} 2023-05-31T20:39:11.120 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:39:11.120 DEBUG:boto:Port: 8000 2023-05-31T20:39:11.121 DEBUG:boto:Params: {} 2023-05-31T20:39:11.121 DEBUG:boto:Token: None 2023-05-31T20:39:11.122 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:39:11 GMT /pklxkd-58/obj00000000000000000019?versionId=null 2023-05-31T20:39:11.122 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:QpZXf3zLsfUD04MOe94/08U/CIo= 2023-05-31T20:39:11.122 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:39:11 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:QpZXf3zLsfUD04MOe94/08U/CIo=', 'Content-Length': '0'} 2023-05-31T20:39:11.124 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:38:21 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b0f28cdc44344fee-006477b06f-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:39:11 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:39:11.125 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000019 2023-05-31T20:39:11.125 DEBUG:boto:path=/pklxkd-58/obj00000000000000000019 2023-05-31T20:39:11.126 DEBUG:boto:auth_path=/pklxkd-58/obj00000000000000000019 2023-05-31T20:39:11.126 DEBUG:boto:path=/pklxkd-58/obj00000000000000000019?versionId=null 2023-05-31T20:39:11.127 DEBUG:boto:auth_path=/pklxkd-58/obj00000000000000000019?versionId=null 2023-05-31T20:39:11.127 DEBUG:boto:Method: GET 2023-05-31T20:39:11.127 DEBUG:boto:Path: /pklxkd-58/obj00000000000000000019?versionId=null 2023-05-31T20:39:11.128 DEBUG:boto:Data: 2023-05-31T20:39:11.128 DEBUG:boto:Headers: {} 2023-05-31T20:39:11.129 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:39:11.129 DEBUG:boto:Port: 8001 2023-05-31T20:39:11.129 DEBUG:boto:Params: {} 2023-05-31T20:39:11.130 DEBUG:boto:Token: None 2023-05-31T20:39:11.130 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:39:11 GMT /pklxkd-58/obj00000000000000000019?versionId=null 2023-05-31T20:39:11.131 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:OO/JN+SpJiUagU7LmC2HGDpStsk= 2023-05-31T20:39:11.131 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:39:11 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:OO/JN+SpJiUagU7LmC2HGDpStsk=', 'Content-Length': '0'} 2023-05-31T20:39:11.133 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:38:21 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000bc5f7728cd953b43-006477b06f-1c17-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:39:11 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:39:11.134 DEBUG:boto:path=/pklxkd-58/obj00000000000000000019 2023-05-31T20:39:11.134 DEBUG:boto:auth_path=/pklxkd-58/obj00000000000000000019 2023-05-31T20:39:11.135 DEBUG:boto:path=/pklxkd-58/obj00000000000000000019?versionId=null 2023-05-31T20:39:11.135 DEBUG:boto:auth_path=/pklxkd-58/obj00000000000000000019?versionId=null 2023-05-31T20:39:11.136 DEBUG:boto:Method: GET 2023-05-31T20:39:11.136 DEBUG:boto:Path: /pklxkd-58/obj00000000000000000019?versionId=null 2023-05-31T20:39:11.137 DEBUG:boto:Data: 2023-05-31T20:39:11.137 DEBUG:boto:Headers: {} 2023-05-31T20:39:11.137 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:39:11.138 DEBUG:boto:Port: 8000 2023-05-31T20:39:11.138 DEBUG:boto:Params: {} 2023-05-31T20:39:11.139 DEBUG:boto:Token: None 2023-05-31T20:39:11.139 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:39:11 GMT /pklxkd-58/obj00000000000000000019?versionId=null 2023-05-31T20:39:11.140 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:OO/JN+SpJiUagU7LmC2HGDpStsk= 2023-05-31T20:39:11.140 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:39:11 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:OO/JN+SpJiUagU7LmC2HGDpStsk=', 'Content-Length': '0'} 2023-05-31T20:39:11.143 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:38:21 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000469e2e565b70b5e3-006477b06f-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:39:11 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:39:11.184 DEBUG:boto:path=/pklxkd-58/ 2023-05-31T20:39:11.185 DEBUG:boto:auth_path=/pklxkd-58/ 2023-05-31T20:39:11.186 DEBUG:boto:path=/pklxkd-58/?prefix=obj00000000000000000019 2023-05-31T20:39:11.186 DEBUG:boto:auth_path=/pklxkd-58/?prefix=obj00000000000000000019 2023-05-31T20:39:11.187 DEBUG:boto:Method: GET 2023-05-31T20:39:11.187 DEBUG:boto:Path: /pklxkd-58/?prefix=obj00000000000000000019 2023-05-31T20:39:11.187 DEBUG:boto:Data: 2023-05-31T20:39:11.188 DEBUG:boto:Headers: {} 2023-05-31T20:39:11.188 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:39:11.189 DEBUG:boto:Port: 8001 2023-05-31T20:39:11.189 DEBUG:boto:Params: {} 2023-05-31T20:39:11.189 DEBUG:boto:Token: None 2023-05-31T20:39:11.190 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:39:11 GMT /pklxkd-58/ 2023-05-31T20:39:11.190 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:GHNSu1S8h4v2Qp+durSNxgCJG9E= 2023-05-31T20:39:11.191 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:39:11 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:GHNSu1S8h4v2Qp+durSNxgCJG9E=', 'Content-Length': '0'} 2023-05-31T20:39:11.193 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000000271e80afeaaf856-006477b06f-1c17-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:39:11 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:39:11.194 DEBUG:boto:b'pklxkd-58obj000000000000000000191000falseobj000000000000000000192023-05-31T20:38:21.559Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:39:11.195 DEBUG:boto:path=/pklxkd-58/ 2023-05-31T20:39:11.196 DEBUG:boto:auth_path=/pklxkd-58/ 2023-05-31T20:39:11.196 DEBUG:boto:path=/pklxkd-58/?prefix=obj00000000000000000019 2023-05-31T20:39:11.196 DEBUG:boto:auth_path=/pklxkd-58/?prefix=obj00000000000000000019 2023-05-31T20:39:11.197 DEBUG:boto:Method: GET 2023-05-31T20:39:11.197 DEBUG:boto:Path: /pklxkd-58/?prefix=obj00000000000000000019 2023-05-31T20:39:11.197 DEBUG:boto:Data: 2023-05-31T20:39:11.198 DEBUG:boto:Headers: {} 2023-05-31T20:39:11.198 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:39:11.199 DEBUG:boto:Port: 8000 2023-05-31T20:39:11.199 DEBUG:boto:Params: {} 2023-05-31T20:39:11.199 DEBUG:boto:Token: None 2023-05-31T20:39:11.200 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:39:11 GMT /pklxkd-58/ 2023-05-31T20:39:11.200 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:GHNSu1S8h4v2Qp+durSNxgCJG9E= 2023-05-31T20:39:11.201 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:39:11 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:GHNSu1S8h4v2Qp+durSNxgCJG9E=', 'Content-Length': '0'} 2023-05-31T20:39:11.205 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000650745bd6ccbd126-006477b06f-1630-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:39:11 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:39:11.205 DEBUG:boto:b'pklxkd-58obj000000000000000000191000falseobj000000000000000000192023-05-31T20:38:21.559Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:39:11.206 DEBUG:boto:path=/pklxkd-58/obj00000000000000000019 2023-05-31T20:39:11.207 DEBUG:boto:auth_path=/pklxkd-58/obj00000000000000000019 2023-05-31T20:39:11.207 DEBUG:boto:Method: HEAD 2023-05-31T20:39:11.207 DEBUG:boto:Path: /pklxkd-58/obj00000000000000000019 2023-05-31T20:39:11.208 DEBUG:boto:Data: 2023-05-31T20:39:11.208 DEBUG:boto:Headers: {} 2023-05-31T20:39:11.209 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:39:11.209 DEBUG:boto:Port: 8001 2023-05-31T20:39:11.209 DEBUG:boto:Params: {} 2023-05-31T20:39:11.210 DEBUG:boto:Token: None 2023-05-31T20:39:11.210 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:39:11 GMT /pklxkd-58/obj00000000000000000019 2023-05-31T20:39:11.211 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:dZZ81RZgNYLhzhsbgufHrqrBxS8= 2023-05-31T20:39:11.211 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:39:11 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:dZZ81RZgNYLhzhsbgufHrqrBxS8=', 'Content-Length': '0'} 2023-05-31T20:39:11.213 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:38:21 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000014136737bb6dd46e-006477b06f-1c17-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:39:11 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:39:11.214 DEBUG:boto:path=/pklxkd-58/obj00000000000000000019 2023-05-31T20:39:11.215 DEBUG:boto:auth_path=/pklxkd-58/obj00000000000000000019 2023-05-31T20:39:11.215 DEBUG:boto:Method: HEAD 2023-05-31T20:39:11.215 DEBUG:boto:Path: /pklxkd-58/obj00000000000000000019 2023-05-31T20:39:11.216 DEBUG:boto:Data: 2023-05-31T20:39:11.216 DEBUG:boto:Headers: {} 2023-05-31T20:39:11.217 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:39:11.217 DEBUG:boto:Port: 8000 2023-05-31T20:39:11.217 DEBUG:boto:Params: {} 2023-05-31T20:39:11.218 DEBUG:boto:Token: None 2023-05-31T20:39:11.218 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:39:11 GMT /pklxkd-58/obj00000000000000000019 2023-05-31T20:39:11.219 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:dZZ81RZgNYLhzhsbgufHrqrBxS8= 2023-05-31T20:39:11.219 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:39:11 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:dZZ81RZgNYLhzhsbgufHrqrBxS8=', 'Content-Length': '0'} 2023-05-31T20:39:11.221 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:38:21 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000cf4ccf093a28d8b9-006477b06f-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:39:11 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:39:11.221 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000019 2023-05-31T20:39:11.222 DEBUG:boto:path=/pklxkd-58/obj00000000000000000019 2023-05-31T20:39:11.222 DEBUG:boto:auth_path=/pklxkd-58/obj00000000000000000019 2023-05-31T20:39:11.223 DEBUG:boto:Method: GET 2023-05-31T20:39:11.223 DEBUG:boto:Path: /pklxkd-58/obj00000000000000000019 2023-05-31T20:39:11.223 DEBUG:boto:Data: 2023-05-31T20:39:11.224 DEBUG:boto:Headers: {} 2023-05-31T20:39:11.224 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:39:11.225 DEBUG:boto:Port: 8001 2023-05-31T20:39:11.225 DEBUG:boto:Params: {} 2023-05-31T20:39:11.225 DEBUG:boto:Token: None 2023-05-31T20:39:11.226 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:39:11 GMT /pklxkd-58/obj00000000000000000019 2023-05-31T20:39:11.226 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:hu4j3GTwn/P1YhePoE5Gf9Xnqo8= 2023-05-31T20:39:11.227 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:39:11 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:hu4j3GTwn/P1YhePoE5Gf9Xnqo8=', 'Content-Length': '0'} 2023-05-31T20:39:11.229 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:38:21 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000064bf5e080c70f5e5-006477b06f-1c17-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:39:11 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:39:11.272 DEBUG:boto:path=/pklxkd-58/obj00000000000000000019 2023-05-31T20:39:11.273 DEBUG:boto:auth_path=/pklxkd-58/obj00000000000000000019 2023-05-31T20:39:11.273 DEBUG:boto:Method: GET 2023-05-31T20:39:11.274 DEBUG:boto:Path: /pklxkd-58/obj00000000000000000019 2023-05-31T20:39:11.274 DEBUG:boto:Data: 2023-05-31T20:39:11.275 DEBUG:boto:Headers: {} 2023-05-31T20:39:11.275 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:39:11.275 DEBUG:boto:Port: 8000 2023-05-31T20:39:11.276 DEBUG:boto:Params: {} 2023-05-31T20:39:11.276 DEBUG:boto:Token: None 2023-05-31T20:39:11.277 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:39:11 GMT /pklxkd-58/obj00000000000000000019 2023-05-31T20:39:11.277 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:hu4j3GTwn/P1YhePoE5Gf9Xnqo8= 2023-05-31T20:39:11.278 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:39:11 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:hu4j3GTwn/P1YhePoE5Gf9Xnqo8=', 'Content-Length': '0'} 2023-05-31T20:39:11.280 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:38:21 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007019c72d986196e1-006477b06f-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:39:11 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:39:11.280 DEBUG:boto:path=/pklxkd-58/ 2023-05-31T20:39:11.281 DEBUG:boto:auth_path=/pklxkd-58/ 2023-05-31T20:39:11.281 DEBUG:boto:path=/pklxkd-58/?prefix=obj00000000000000000019 2023-05-31T20:39:11.282 DEBUG:boto:auth_path=/pklxkd-58/?prefix=obj00000000000000000019 2023-05-31T20:39:11.282 DEBUG:boto:Method: GET 2023-05-31T20:39:11.282 DEBUG:boto:Path: /pklxkd-58/?prefix=obj00000000000000000019 2023-05-31T20:39:11.283 DEBUG:boto:Data: 2023-05-31T20:39:11.283 DEBUG:boto:Headers: {} 2023-05-31T20:39:11.283 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:39:11.284 DEBUG:boto:Port: 8001 2023-05-31T20:39:11.284 DEBUG:boto:Params: {} 2023-05-31T20:39:11.285 DEBUG:boto:Token: None 2023-05-31T20:39:11.285 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:39:11 GMT /pklxkd-58/ 2023-05-31T20:39:11.285 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:GHNSu1S8h4v2Qp+durSNxgCJG9E= 2023-05-31T20:39:11.286 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:39:11 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:GHNSu1S8h4v2Qp+durSNxgCJG9E=', 'Content-Length': '0'} 2023-05-31T20:39:11.288 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000033ffb0addf52d4ee-006477b06f-1c17-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:39:11 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:39:11.289 DEBUG:boto:b'pklxkd-58obj000000000000000000191000falseobj000000000000000000192023-05-31T20:38:21.559Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:39:11.290 DEBUG:boto:path=/pklxkd-58/ 2023-05-31T20:39:11.290 DEBUG:boto:auth_path=/pklxkd-58/ 2023-05-31T20:39:11.290 DEBUG:boto:path=/pklxkd-58/?prefix=obj00000000000000000019 2023-05-31T20:39:11.291 DEBUG:boto:auth_path=/pklxkd-58/?prefix=obj00000000000000000019 2023-05-31T20:39:11.291 DEBUG:boto:Method: GET 2023-05-31T20:39:11.291 DEBUG:boto:Path: /pklxkd-58/?prefix=obj00000000000000000019 2023-05-31T20:39:11.292 DEBUG:boto:Data: 2023-05-31T20:39:11.293 DEBUG:boto:Headers: {} 2023-05-31T20:39:11.293 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:39:11.293 DEBUG:boto:Port: 8000 2023-05-31T20:39:11.294 DEBUG:boto:Params: {} 2023-05-31T20:39:11.294 DEBUG:boto:Token: None 2023-05-31T20:39:11.295 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:39:11 GMT /pklxkd-58/ 2023-05-31T20:39:11.295 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:GHNSu1S8h4v2Qp+durSNxgCJG9E= 2023-05-31T20:39:11.295 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:39:11 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:GHNSu1S8h4v2Qp+durSNxgCJG9E=', 'Content-Length': '0'} 2023-05-31T20:39:11.300 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000071a855897e6fd312-006477b06f-1630-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:39:11 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:39:11.300 DEBUG:boto:b'pklxkd-58obj000000000000000000191000falseobj000000000000000000192023-05-31T20:38:21.559Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:39:11.301 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000020 2023-05-31T20:39:11.301 DEBUG:boto:path=/pklxkd-58/obj00000000000000000020 2023-05-31T20:39:11.302 DEBUG:boto:auth_path=/pklxkd-58/obj00000000000000000020 2023-05-31T20:39:11.302 DEBUG:boto:path=/pklxkd-58/obj00000000000000000020?versionId=null 2023-05-31T20:39:11.303 DEBUG:boto:auth_path=/pklxkd-58/obj00000000000000000020?versionId=null 2023-05-31T20:39:11.303 DEBUG:boto:Method: GET 2023-05-31T20:39:11.303 DEBUG:boto:Path: /pklxkd-58/obj00000000000000000020?versionId=null 2023-05-31T20:39:11.304 DEBUG:boto:Data: 2023-05-31T20:39:11.304 DEBUG:boto:Headers: {} 2023-05-31T20:39:11.304 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:39:11.305 DEBUG:boto:Port: 8001 2023-05-31T20:39:11.305 DEBUG:boto:Params: {} 2023-05-31T20:39:11.306 DEBUG:boto:Token: None 2023-05-31T20:39:11.306 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:39:11 GMT /pklxkd-58/obj00000000000000000020?versionId=null 2023-05-31T20:39:11.307 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:/f1lggtfEa3XqDq3hCAbw7rn8Jo= 2023-05-31T20:39:11.308 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:39:11 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:/f1lggtfEa3XqDq3hCAbw7rn8Jo=', 'Content-Length': '0'} 2023-05-31T20:39:11.310 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:38:22 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000033a09b9a9c0dc269-006477b06f-1c17-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:39:11 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:39:11.310 DEBUG:boto:path=/pklxkd-58/obj00000000000000000020 2023-05-31T20:39:11.311 DEBUG:boto:auth_path=/pklxkd-58/obj00000000000000000020 2023-05-31T20:39:11.311 DEBUG:boto:path=/pklxkd-58/obj00000000000000000020?versionId=null 2023-05-31T20:39:11.312 DEBUG:boto:auth_path=/pklxkd-58/obj00000000000000000020?versionId=null 2023-05-31T20:39:11.312 DEBUG:boto:Method: GET 2023-05-31T20:39:11.312 DEBUG:boto:Path: /pklxkd-58/obj00000000000000000020?versionId=null 2023-05-31T20:39:11.313 DEBUG:boto:Data: 2023-05-31T20:39:11.313 DEBUG:boto:Headers: {} 2023-05-31T20:39:11.313 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:39:11.314 DEBUG:boto:Port: 8000 2023-05-31T20:39:11.314 DEBUG:boto:Params: {} 2023-05-31T20:39:11.315 DEBUG:boto:Token: None 2023-05-31T20:39:11.315 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:39:11 GMT /pklxkd-58/obj00000000000000000020?versionId=null 2023-05-31T20:39:11.315 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:/f1lggtfEa3XqDq3hCAbw7rn8Jo= 2023-05-31T20:39:11.316 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:39:11 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:/f1lggtfEa3XqDq3hCAbw7rn8Jo=', 'Content-Length': '0'} 2023-05-31T20:39:11.318 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:38:22 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008d7543ee65ab0b15-006477b06f-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:39:11 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:39:11.319 DEBUG:boto:path=/pklxkd-58/obj00000000000000000020 2023-05-31T20:39:11.319 DEBUG:boto:auth_path=/pklxkd-58/obj00000000000000000020 2023-05-31T20:39:11.319 DEBUG:boto:path=/pklxkd-58/obj00000000000000000020?versionId=null 2023-05-31T20:39:11.320 DEBUG:boto:auth_path=/pklxkd-58/obj00000000000000000020?versionId=null 2023-05-31T20:39:11.320 DEBUG:boto:Method: HEAD 2023-05-31T20:39:11.321 DEBUG:boto:Path: /pklxkd-58/obj00000000000000000020?versionId=null 2023-05-31T20:39:11.321 DEBUG:boto:Data: 2023-05-31T20:39:11.321 DEBUG:boto:Headers: {} 2023-05-31T20:39:11.322 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:39:11.322 DEBUG:boto:Port: 8001 2023-05-31T20:39:11.322 DEBUG:boto:Params: {} 2023-05-31T20:39:11.323 DEBUG:boto:Token: None 2023-05-31T20:39:11.323 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:39:11 GMT /pklxkd-58/obj00000000000000000020?versionId=null 2023-05-31T20:39:11.324 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:uiavPINHLr9Vg27ztsO1G9LIQ0c= 2023-05-31T20:39:11.324 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:39:11 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:uiavPINHLr9Vg27ztsO1G9LIQ0c=', 'Content-Length': '0'} 2023-05-31T20:39:11.326 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:38:22 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f2a5b3bb6ee2b1ba-006477b06f-1c17-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:39:11 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:39:11.326 DEBUG:boto:path=/pklxkd-58/obj00000000000000000020 2023-05-31T20:39:11.327 DEBUG:boto:auth_path=/pklxkd-58/obj00000000000000000020 2023-05-31T20:39:11.327 DEBUG:boto:path=/pklxkd-58/obj00000000000000000020?versionId=null 2023-05-31T20:39:11.327 DEBUG:boto:auth_path=/pklxkd-58/obj00000000000000000020?versionId=null 2023-05-31T20:39:11.328 DEBUG:boto:Method: HEAD 2023-05-31T20:39:11.328 DEBUG:boto:Path: /pklxkd-58/obj00000000000000000020?versionId=null 2023-05-31T20:39:11.329 DEBUG:boto:Data: 2023-05-31T20:39:11.329 DEBUG:boto:Headers: {} 2023-05-31T20:39:11.329 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:39:11.330 DEBUG:boto:Port: 8000 2023-05-31T20:39:11.330 DEBUG:boto:Params: {} 2023-05-31T20:39:11.330 DEBUG:boto:Token: None 2023-05-31T20:39:11.331 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:39:11 GMT /pklxkd-58/obj00000000000000000020?versionId=null 2023-05-31T20:39:11.331 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:uiavPINHLr9Vg27ztsO1G9LIQ0c= 2023-05-31T20:39:11.332 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:39:11 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:uiavPINHLr9Vg27ztsO1G9LIQ0c=', 'Content-Length': '0'} 2023-05-31T20:39:11.334 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:38:22 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e3bbbd46e623859b-006477b06f-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:39:11 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:39:11.335 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000020 2023-05-31T20:39:11.335 DEBUG:boto:path=/pklxkd-58/obj00000000000000000020 2023-05-31T20:39:11.336 DEBUG:boto:auth_path=/pklxkd-58/obj00000000000000000020 2023-05-31T20:39:11.337 DEBUG:boto:path=/pklxkd-58/obj00000000000000000020?versionId=null 2023-05-31T20:39:11.337 DEBUG:boto:auth_path=/pklxkd-58/obj00000000000000000020?versionId=null 2023-05-31T20:39:11.337 DEBUG:boto:Method: GET 2023-05-31T20:39:11.338 DEBUG:boto:Path: /pklxkd-58/obj00000000000000000020?versionId=null 2023-05-31T20:39:11.338 DEBUG:boto:Data: 2023-05-31T20:39:11.338 DEBUG:boto:Headers: {} 2023-05-31T20:39:11.339 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:39:11.339 DEBUG:boto:Port: 8001 2023-05-31T20:39:11.339 DEBUG:boto:Params: {} 2023-05-31T20:39:11.340 DEBUG:boto:Token: None 2023-05-31T20:39:11.340 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:39:11 GMT /pklxkd-58/obj00000000000000000020?versionId=null 2023-05-31T20:39:11.341 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:/f1lggtfEa3XqDq3hCAbw7rn8Jo= 2023-05-31T20:39:11.341 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:39:11 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:/f1lggtfEa3XqDq3hCAbw7rn8Jo=', 'Content-Length': '0'} 2023-05-31T20:39:11.343 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:38:22 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d73bd21dd3d383ef-006477b06f-1c17-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:39:11 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:39:11.344 DEBUG:boto:path=/pklxkd-58/obj00000000000000000020 2023-05-31T20:39:11.344 DEBUG:boto:auth_path=/pklxkd-58/obj00000000000000000020 2023-05-31T20:39:11.345 DEBUG:boto:path=/pklxkd-58/obj00000000000000000020?versionId=null 2023-05-31T20:39:11.345 DEBUG:boto:auth_path=/pklxkd-58/obj00000000000000000020?versionId=null 2023-05-31T20:39:11.345 DEBUG:boto:Method: GET 2023-05-31T20:39:11.346 DEBUG:boto:Path: /pklxkd-58/obj00000000000000000020?versionId=null 2023-05-31T20:39:11.346 DEBUG:boto:Data: 2023-05-31T20:39:11.346 DEBUG:boto:Headers: {} 2023-05-31T20:39:11.347 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:39:11.347 DEBUG:boto:Port: 8000 2023-05-31T20:39:11.348 DEBUG:boto:Params: {} 2023-05-31T20:39:11.348 DEBUG:boto:Token: None 2023-05-31T20:39:11.348 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:39:11 GMT /pklxkd-58/obj00000000000000000020?versionId=null 2023-05-31T20:39:11.349 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:/f1lggtfEa3XqDq3hCAbw7rn8Jo= 2023-05-31T20:39:11.349 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:39:11 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:/f1lggtfEa3XqDq3hCAbw7rn8Jo=', 'Content-Length': '0'} 2023-05-31T20:39:11.351 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:38:22 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000017695df2af88dae8-006477b06f-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:39:11 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:39:11.352 DEBUG:boto:path=/pklxkd-58/ 2023-05-31T20:39:11.352 DEBUG:boto:auth_path=/pklxkd-58/ 2023-05-31T20:39:11.353 DEBUG:boto:path=/pklxkd-58/?prefix=obj00000000000000000020 2023-05-31T20:39:11.353 DEBUG:boto:auth_path=/pklxkd-58/?prefix=obj00000000000000000020 2023-05-31T20:39:11.354 DEBUG:boto:Method: GET 2023-05-31T20:39:11.354 DEBUG:boto:Path: /pklxkd-58/?prefix=obj00000000000000000020 2023-05-31T20:39:11.354 DEBUG:boto:Data: 2023-05-31T20:39:11.355 DEBUG:boto:Headers: {} 2023-05-31T20:39:11.355 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:39:11.355 DEBUG:boto:Port: 8001 2023-05-31T20:39:11.356 DEBUG:boto:Params: {} 2023-05-31T20:39:11.356 DEBUG:boto:Token: None 2023-05-31T20:39:11.357 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:39:11 GMT /pklxkd-58/ 2023-05-31T20:39:11.357 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:GHNSu1S8h4v2Qp+durSNxgCJG9E= 2023-05-31T20:39:11.357 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:39:11 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:GHNSu1S8h4v2Qp+durSNxgCJG9E=', 'Content-Length': '0'} 2023-05-31T20:39:11.360 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000001e79a6128f261b67-006477b06f-1c17-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:39:11 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:39:11.362 DEBUG:boto:b'pklxkd-58obj000000000000000000201000falseobj000000000000000000202023-05-31T20:38:22.465Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:39:11.364 DEBUG:boto:path=/pklxkd-58/ 2023-05-31T20:39:11.364 DEBUG:boto:auth_path=/pklxkd-58/ 2023-05-31T20:39:11.364 DEBUG:boto:path=/pklxkd-58/?prefix=obj00000000000000000020 2023-05-31T20:39:11.365 DEBUG:boto:auth_path=/pklxkd-58/?prefix=obj00000000000000000020 2023-05-31T20:39:11.365 DEBUG:boto:Method: GET 2023-05-31T20:39:11.366 DEBUG:boto:Path: /pklxkd-58/?prefix=obj00000000000000000020 2023-05-31T20:39:11.366 DEBUG:boto:Data: 2023-05-31T20:39:11.366 DEBUG:boto:Headers: {} 2023-05-31T20:39:11.367 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:39:11.367 DEBUG:boto:Port: 8000 2023-05-31T20:39:11.367 DEBUG:boto:Params: {} 2023-05-31T20:39:11.368 DEBUG:boto:Token: None 2023-05-31T20:39:11.368 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:39:11 GMT /pklxkd-58/ 2023-05-31T20:39:11.369 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:GHNSu1S8h4v2Qp+durSNxgCJG9E= 2023-05-31T20:39:11.369 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:39:11 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:GHNSu1S8h4v2Qp+durSNxgCJG9E=', 'Content-Length': '0'} 2023-05-31T20:39:11.374 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000185f82436b448ca9-006477b06f-1630-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:39:11 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:39:11.376 DEBUG:boto:b'pklxkd-58obj000000000000000000201000falseobj000000000000000000202023-05-31T20:38:22.465Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:39:11.377 DEBUG:boto:path=/pklxkd-58/obj00000000000000000020 2023-05-31T20:39:11.378 DEBUG:boto:auth_path=/pklxkd-58/obj00000000000000000020 2023-05-31T20:39:11.378 DEBUG:boto:Method: HEAD 2023-05-31T20:39:11.378 DEBUG:boto:Path: /pklxkd-58/obj00000000000000000020 2023-05-31T20:39:11.379 DEBUG:boto:Data: 2023-05-31T20:39:11.379 DEBUG:boto:Headers: {} 2023-05-31T20:39:11.379 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:39:11.380 DEBUG:boto:Port: 8001 2023-05-31T20:39:11.380 DEBUG:boto:Params: {} 2023-05-31T20:39:11.380 DEBUG:boto:Token: None 2023-05-31T20:39:11.381 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:39:11 GMT /pklxkd-58/obj00000000000000000020 2023-05-31T20:39:11.381 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:0WqfeCCkxixpMj1t7dHGGL0YDQ4= 2023-05-31T20:39:11.382 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:39:11 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:0WqfeCCkxixpMj1t7dHGGL0YDQ4=', 'Content-Length': '0'} 2023-05-31T20:39:11.384 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:38:22 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e082523815c2249c-006477b06f-1c17-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:39:11 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:39:11.385 DEBUG:boto:path=/pklxkd-58/obj00000000000000000020 2023-05-31T20:39:11.385 DEBUG:boto:auth_path=/pklxkd-58/obj00000000000000000020 2023-05-31T20:39:11.386 DEBUG:boto:Method: HEAD 2023-05-31T20:39:11.387 DEBUG:boto:Path: /pklxkd-58/obj00000000000000000020 2023-05-31T20:39:11.387 DEBUG:boto:Data: 2023-05-31T20:39:11.387 DEBUG:boto:Headers: {} 2023-05-31T20:39:11.388 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:39:11.388 DEBUG:boto:Port: 8000 2023-05-31T20:39:11.388 DEBUG:boto:Params: {} 2023-05-31T20:39:11.389 DEBUG:boto:Token: None 2023-05-31T20:39:11.390 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:39:11 GMT /pklxkd-58/obj00000000000000000020 2023-05-31T20:39:11.390 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:0WqfeCCkxixpMj1t7dHGGL0YDQ4= 2023-05-31T20:39:11.391 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:39:11 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:0WqfeCCkxixpMj1t7dHGGL0YDQ4=', 'Content-Length': '0'} 2023-05-31T20:39:11.393 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:38:22 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000430eacf4e4c0678d-006477b06f-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:39:11 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:39:11.393 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000020 2023-05-31T20:39:11.394 DEBUG:boto:path=/pklxkd-58/obj00000000000000000020 2023-05-31T20:39:11.394 DEBUG:boto:auth_path=/pklxkd-58/obj00000000000000000020 2023-05-31T20:39:11.395 DEBUG:boto:Method: GET 2023-05-31T20:39:11.395 DEBUG:boto:Path: /pklxkd-58/obj00000000000000000020 2023-05-31T20:39:11.396 DEBUG:boto:Data: 2023-05-31T20:39:11.396 DEBUG:boto:Headers: {} 2023-05-31T20:39:11.396 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:39:11.397 DEBUG:boto:Port: 8001 2023-05-31T20:39:11.397 DEBUG:boto:Params: {} 2023-05-31T20:39:11.398 DEBUG:boto:Token: None 2023-05-31T20:39:11.398 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:39:11 GMT /pklxkd-58/obj00000000000000000020 2023-05-31T20:39:11.398 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:BbTnKt7KztUI6JOBHmjD/4TD+uE= 2023-05-31T20:39:11.399 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:39:11 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:BbTnKt7KztUI6JOBHmjD/4TD+uE=', 'Content-Length': '0'} 2023-05-31T20:39:11.401 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:38:22 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c0a0ce6f7821cf28-006477b06f-1c17-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:39:11 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:39:11.444 DEBUG:boto:path=/pklxkd-58/obj00000000000000000020 2023-05-31T20:39:11.445 DEBUG:boto:auth_path=/pklxkd-58/obj00000000000000000020 2023-05-31T20:39:11.445 DEBUG:boto:Method: GET 2023-05-31T20:39:11.446 DEBUG:boto:Path: /pklxkd-58/obj00000000000000000020 2023-05-31T20:39:11.446 DEBUG:boto:Data: 2023-05-31T20:39:11.447 DEBUG:boto:Headers: {} 2023-05-31T20:39:11.447 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:39:11.448 DEBUG:boto:Port: 8000 2023-05-31T20:39:11.449 DEBUG:boto:Params: {} 2023-05-31T20:39:11.449 DEBUG:boto:Token: None 2023-05-31T20:39:11.449 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:39:11 GMT /pklxkd-58/obj00000000000000000020 2023-05-31T20:39:11.450 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:BbTnKt7KztUI6JOBHmjD/4TD+uE= 2023-05-31T20:39:11.450 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:39:11 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:BbTnKt7KztUI6JOBHmjD/4TD+uE=', 'Content-Length': '0'} 2023-05-31T20:39:11.452 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:38:22 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000288f41df881f1601-006477b06f-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:39:11 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:39:11.453 DEBUG:boto:path=/pklxkd-58/ 2023-05-31T20:39:11.454 DEBUG:boto:auth_path=/pklxkd-58/ 2023-05-31T20:39:11.454 DEBUG:boto:path=/pklxkd-58/?prefix=obj00000000000000000020 2023-05-31T20:39:11.454 DEBUG:boto:auth_path=/pklxkd-58/?prefix=obj00000000000000000020 2023-05-31T20:39:11.455 DEBUG:boto:Method: GET 2023-05-31T20:39:11.455 DEBUG:boto:Path: /pklxkd-58/?prefix=obj00000000000000000020 2023-05-31T20:39:11.455 DEBUG:boto:Data: 2023-05-31T20:39:11.456 DEBUG:boto:Headers: {} 2023-05-31T20:39:11.456 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:39:11.457 DEBUG:boto:Port: 8001 2023-05-31T20:39:11.457 DEBUG:boto:Params: {} 2023-05-31T20:39:11.457 DEBUG:boto:Token: None 2023-05-31T20:39:11.458 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:39:11 GMT /pklxkd-58/ 2023-05-31T20:39:11.458 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:GHNSu1S8h4v2Qp+durSNxgCJG9E= 2023-05-31T20:39:11.459 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:39:11 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:GHNSu1S8h4v2Qp+durSNxgCJG9E=', 'Content-Length': '0'} 2023-05-31T20:39:11.462 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d97c284be685f98a-006477b06f-1c17-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:39:11 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:39:11.462 DEBUG:boto:b'pklxkd-58obj000000000000000000201000falseobj000000000000000000202023-05-31T20:38:22.465Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:39:11.463 DEBUG:boto:path=/pklxkd-58/ 2023-05-31T20:39:11.463 DEBUG:boto:auth_path=/pklxkd-58/ 2023-05-31T20:39:11.464 DEBUG:boto:path=/pklxkd-58/?prefix=obj00000000000000000020 2023-05-31T20:39:11.464 DEBUG:boto:auth_path=/pklxkd-58/?prefix=obj00000000000000000020 2023-05-31T20:39:11.465 DEBUG:boto:Method: GET 2023-05-31T20:39:11.465 DEBUG:boto:Path: /pklxkd-58/?prefix=obj00000000000000000020 2023-05-31T20:39:11.465 DEBUG:boto:Data: 2023-05-31T20:39:11.466 DEBUG:boto:Headers: {} 2023-05-31T20:39:11.466 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:39:11.467 DEBUG:boto:Port: 8000 2023-05-31T20:39:11.467 DEBUG:boto:Params: {} 2023-05-31T20:39:11.468 DEBUG:boto:Token: None 2023-05-31T20:39:11.468 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:39:11 GMT /pklxkd-58/ 2023-05-31T20:39:11.468 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:GHNSu1S8h4v2Qp+durSNxgCJG9E= 2023-05-31T20:39:11.469 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:39:11 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:GHNSu1S8h4v2Qp+durSNxgCJG9E=', 'Content-Length': '0'} 2023-05-31T20:39:11.474 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000006a1fecf8837f5e6e-006477b06f-1630-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:39:11 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:39:11.474 DEBUG:boto:b'pklxkd-58obj000000000000000000201000falseobj000000000000000000202023-05-31T20:38:22.465Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:39:11.475 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000021 2023-05-31T20:39:11.475 DEBUG:boto:path=/pklxkd-58/obj00000000000000000021 2023-05-31T20:39:11.476 DEBUG:boto:auth_path=/pklxkd-58/obj00000000000000000021 2023-05-31T20:39:11.476 DEBUG:boto:path=/pklxkd-58/obj00000000000000000021?versionId=null 2023-05-31T20:39:11.476 DEBUG:boto:auth_path=/pklxkd-58/obj00000000000000000021?versionId=null 2023-05-31T20:39:11.477 DEBUG:boto:Method: GET 2023-05-31T20:39:11.477 DEBUG:boto:Path: /pklxkd-58/obj00000000000000000021?versionId=null 2023-05-31T20:39:11.478 DEBUG:boto:Data: 2023-05-31T20:39:11.478 DEBUG:boto:Headers: {} 2023-05-31T20:39:11.478 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:39:11.479 DEBUG:boto:Port: 8001 2023-05-31T20:39:11.479 DEBUG:boto:Params: {} 2023-05-31T20:39:11.479 DEBUG:boto:Token: None 2023-05-31T20:39:11.480 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:39:11 GMT /pklxkd-58/obj00000000000000000021?versionId=null 2023-05-31T20:39:11.480 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:18xun5JY1xni/VqWU8C/ZBnhP9E= 2023-05-31T20:39:11.481 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:39:11 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:18xun5JY1xni/VqWU8C/ZBnhP9E=', 'Content-Length': '0'} 2023-05-31T20:39:11.483 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:38:20 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000688069ce6b44f4a0-006477b06f-1c17-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:39:11 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:39:11.484 DEBUG:boto:path=/pklxkd-58/obj00000000000000000021 2023-05-31T20:39:11.486 DEBUG:boto:auth_path=/pklxkd-58/obj00000000000000000021 2023-05-31T20:39:11.487 DEBUG:boto:path=/pklxkd-58/obj00000000000000000021?versionId=null 2023-05-31T20:39:11.487 DEBUG:boto:auth_path=/pklxkd-58/obj00000000000000000021?versionId=null 2023-05-31T20:39:11.488 DEBUG:boto:Method: GET 2023-05-31T20:39:11.488 DEBUG:boto:Path: /pklxkd-58/obj00000000000000000021?versionId=null 2023-05-31T20:39:11.489 DEBUG:boto:Data: 2023-05-31T20:39:11.489 DEBUG:boto:Headers: {} 2023-05-31T20:39:11.490 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:39:11.490 DEBUG:boto:Port: 8000 2023-05-31T20:39:11.490 DEBUG:boto:Params: {} 2023-05-31T20:39:11.491 DEBUG:boto:Token: None 2023-05-31T20:39:11.491 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:39:11 GMT /pklxkd-58/obj00000000000000000021?versionId=null 2023-05-31T20:39:11.492 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:18xun5JY1xni/VqWU8C/ZBnhP9E= 2023-05-31T20:39:11.492 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:39:11 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:18xun5JY1xni/VqWU8C/ZBnhP9E=', 'Content-Length': '0'} 2023-05-31T20:39:11.494 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:38:20 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000074b11faf5e6bb72b-006477b06f-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:39:11 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:39:11.495 DEBUG:boto:path=/pklxkd-58/obj00000000000000000021 2023-05-31T20:39:11.495 DEBUG:boto:auth_path=/pklxkd-58/obj00000000000000000021 2023-05-31T20:39:11.496 DEBUG:boto:path=/pklxkd-58/obj00000000000000000021?versionId=null 2023-05-31T20:39:11.496 DEBUG:boto:auth_path=/pklxkd-58/obj00000000000000000021?versionId=null 2023-05-31T20:39:11.497 DEBUG:boto:Method: HEAD 2023-05-31T20:39:11.497 DEBUG:boto:Path: /pklxkd-58/obj00000000000000000021?versionId=null 2023-05-31T20:39:11.497 DEBUG:boto:Data: 2023-05-31T20:39:11.498 DEBUG:boto:Headers: {} 2023-05-31T20:39:11.498 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:39:11.499 DEBUG:boto:Port: 8001 2023-05-31T20:39:11.499 DEBUG:boto:Params: {} 2023-05-31T20:39:11.499 DEBUG:boto:Token: None 2023-05-31T20:39:11.500 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:39:11 GMT /pklxkd-58/obj00000000000000000021?versionId=null 2023-05-31T20:39:11.500 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:RcKsOewRw70d06l5Mfltdn06o7U= 2023-05-31T20:39:11.501 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:39:11 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:RcKsOewRw70d06l5Mfltdn06o7U=', 'Content-Length': '0'} 2023-05-31T20:39:11.502 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:38:20 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006d8efef7d7d0544f-006477b06f-1c17-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:39:11 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:39:11.503 DEBUG:boto:path=/pklxkd-58/obj00000000000000000021 2023-05-31T20:39:11.503 DEBUG:boto:auth_path=/pklxkd-58/obj00000000000000000021 2023-05-31T20:39:11.504 DEBUG:boto:path=/pklxkd-58/obj00000000000000000021?versionId=null 2023-05-31T20:39:11.504 DEBUG:boto:auth_path=/pklxkd-58/obj00000000000000000021?versionId=null 2023-05-31T20:39:11.504 DEBUG:boto:Method: HEAD 2023-05-31T20:39:11.505 DEBUG:boto:Path: /pklxkd-58/obj00000000000000000021?versionId=null 2023-05-31T20:39:11.506 DEBUG:boto:Data: 2023-05-31T20:39:11.506 DEBUG:boto:Headers: {} 2023-05-31T20:39:11.507 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:39:11.507 DEBUG:boto:Port: 8000 2023-05-31T20:39:11.507 DEBUG:boto:Params: {} 2023-05-31T20:39:11.508 DEBUG:boto:Token: None 2023-05-31T20:39:11.508 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:39:11 GMT /pklxkd-58/obj00000000000000000021?versionId=null 2023-05-31T20:39:11.509 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:RcKsOewRw70d06l5Mfltdn06o7U= 2023-05-31T20:39:11.509 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:39:11 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:RcKsOewRw70d06l5Mfltdn06o7U=', 'Content-Length': '0'} 2023-05-31T20:39:11.511 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:38:20 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000fb84460a641772a9-006477b06f-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:39:11 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:39:11.512 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000021 2023-05-31T20:39:11.512 DEBUG:boto:path=/pklxkd-58/obj00000000000000000021 2023-05-31T20:39:11.513 DEBUG:boto:auth_path=/pklxkd-58/obj00000000000000000021 2023-05-31T20:39:11.513 DEBUG:boto:path=/pklxkd-58/obj00000000000000000021?versionId=null 2023-05-31T20:39:11.514 DEBUG:boto:auth_path=/pklxkd-58/obj00000000000000000021?versionId=null 2023-05-31T20:39:11.514 DEBUG:boto:Method: GET 2023-05-31T20:39:11.514 DEBUG:boto:Path: /pklxkd-58/obj00000000000000000021?versionId=null 2023-05-31T20:39:11.515 DEBUG:boto:Data: 2023-05-31T20:39:11.515 DEBUG:boto:Headers: {} 2023-05-31T20:39:11.516 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:39:11.516 DEBUG:boto:Port: 8001 2023-05-31T20:39:11.516 DEBUG:boto:Params: {} 2023-05-31T20:39:11.517 DEBUG:boto:Token: None 2023-05-31T20:39:11.517 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:39:11 GMT /pklxkd-58/obj00000000000000000021?versionId=null 2023-05-31T20:39:11.518 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:18xun5JY1xni/VqWU8C/ZBnhP9E= 2023-05-31T20:39:11.518 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:39:11 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:18xun5JY1xni/VqWU8C/ZBnhP9E=', 'Content-Length': '0'} 2023-05-31T20:39:11.520 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:38:20 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d855ee6501d80696-006477b06f-1c17-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:39:11 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:39:11.521 DEBUG:boto:path=/pklxkd-58/obj00000000000000000021 2023-05-31T20:39:11.521 DEBUG:boto:auth_path=/pklxkd-58/obj00000000000000000021 2023-05-31T20:39:11.522 DEBUG:boto:path=/pklxkd-58/obj00000000000000000021?versionId=null 2023-05-31T20:39:11.522 DEBUG:boto:auth_path=/pklxkd-58/obj00000000000000000021?versionId=null 2023-05-31T20:39:11.522 DEBUG:boto:Method: GET 2023-05-31T20:39:11.523 DEBUG:boto:Path: /pklxkd-58/obj00000000000000000021?versionId=null 2023-05-31T20:39:11.523 DEBUG:boto:Data: 2023-05-31T20:39:11.524 DEBUG:boto:Headers: {} 2023-05-31T20:39:11.524 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:39:11.524 DEBUG:boto:Port: 8000 2023-05-31T20:39:11.525 DEBUG:boto:Params: {} 2023-05-31T20:39:11.525 DEBUG:boto:Token: None 2023-05-31T20:39:11.526 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:39:11 GMT /pklxkd-58/obj00000000000000000021?versionId=null 2023-05-31T20:39:11.526 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:18xun5JY1xni/VqWU8C/ZBnhP9E= 2023-05-31T20:39:11.527 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:39:11 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:18xun5JY1xni/VqWU8C/ZBnhP9E=', 'Content-Length': '0'} 2023-05-31T20:39:11.529 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:38:20 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001fe0bcf8b7aa6d53-006477b06f-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:39:11 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:39:11.530 DEBUG:boto:path=/pklxkd-58/ 2023-05-31T20:39:11.531 DEBUG:boto:auth_path=/pklxkd-58/ 2023-05-31T20:39:11.531 DEBUG:boto:path=/pklxkd-58/?prefix=obj00000000000000000021 2023-05-31T20:39:11.531 DEBUG:boto:auth_path=/pklxkd-58/?prefix=obj00000000000000000021 2023-05-31T20:39:11.532 DEBUG:boto:Method: GET 2023-05-31T20:39:11.532 DEBUG:boto:Path: /pklxkd-58/?prefix=obj00000000000000000021 2023-05-31T20:39:11.533 DEBUG:boto:Data: 2023-05-31T20:39:11.533 DEBUG:boto:Headers: {} 2023-05-31T20:39:11.533 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:39:11.534 DEBUG:boto:Port: 8001 2023-05-31T20:39:11.534 DEBUG:boto:Params: {} 2023-05-31T20:39:11.535 DEBUG:boto:Token: None 2023-05-31T20:39:11.535 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:39:11 GMT /pklxkd-58/ 2023-05-31T20:39:11.536 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:GHNSu1S8h4v2Qp+durSNxgCJG9E= 2023-05-31T20:39:11.536 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:39:11 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:GHNSu1S8h4v2Qp+durSNxgCJG9E=', 'Content-Length': '0'} 2023-05-31T20:39:11.539 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000002aa1668e6a233fb7-006477b06f-1c17-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:39:11 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:39:11.540 DEBUG:boto:b'pklxkd-58obj000000000000000000211000falseobj000000000000000000212023-05-31T20:38:20.802Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:39:11.541 DEBUG:boto:path=/pklxkd-58/ 2023-05-31T20:39:11.541 DEBUG:boto:auth_path=/pklxkd-58/ 2023-05-31T20:39:11.542 DEBUG:boto:path=/pklxkd-58/?prefix=obj00000000000000000021 2023-05-31T20:39:11.542 DEBUG:boto:auth_path=/pklxkd-58/?prefix=obj00000000000000000021 2023-05-31T20:39:11.542 DEBUG:boto:Method: GET 2023-05-31T20:39:11.543 DEBUG:boto:Path: /pklxkd-58/?prefix=obj00000000000000000021 2023-05-31T20:39:11.543 DEBUG:boto:Data: 2023-05-31T20:39:11.543 DEBUG:boto:Headers: {} 2023-05-31T20:39:11.544 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:39:11.544 DEBUG:boto:Port: 8000 2023-05-31T20:39:11.545 DEBUG:boto:Params: {} 2023-05-31T20:39:11.545 DEBUG:boto:Token: None 2023-05-31T20:39:11.545 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:39:11 GMT /pklxkd-58/ 2023-05-31T20:39:11.546 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:GHNSu1S8h4v2Qp+durSNxgCJG9E= 2023-05-31T20:39:11.546 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:39:11 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:GHNSu1S8h4v2Qp+durSNxgCJG9E=', 'Content-Length': '0'} 2023-05-31T20:39:11.551 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000046446deed40a211c-006477b06f-1630-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:39:11 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:39:11.596 DEBUG:boto:b'pklxkd-58obj000000000000000000211000falseobj000000000000000000212023-05-31T20:38:20.802Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:39:11.597 DEBUG:boto:path=/pklxkd-58/obj00000000000000000021 2023-05-31T20:39:11.598 DEBUG:boto:auth_path=/pklxkd-58/obj00000000000000000021 2023-05-31T20:39:11.598 DEBUG:boto:Method: HEAD 2023-05-31T20:39:11.599 DEBUG:boto:Path: /pklxkd-58/obj00000000000000000021 2023-05-31T20:39:11.599 DEBUG:boto:Data: 2023-05-31T20:39:11.599 DEBUG:boto:Headers: {} 2023-05-31T20:39:11.600 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:39:11.600 DEBUG:boto:Port: 8001 2023-05-31T20:39:11.600 DEBUG:boto:Params: {} 2023-05-31T20:39:11.601 DEBUG:boto:Token: None 2023-05-31T20:39:11.601 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:39:11 GMT /pklxkd-58/obj00000000000000000021 2023-05-31T20:39:11.602 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:Oqw+HxahCHMtkcTJkJX8pZ6/n9E= 2023-05-31T20:39:11.602 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:39:11 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:Oqw+HxahCHMtkcTJkJX8pZ6/n9E=', 'Content-Length': '0'} 2023-05-31T20:39:11.605 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:38:20 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000463a68adb0745948-006477b06f-1c17-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:39:11 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:39:11.606 DEBUG:boto:path=/pklxkd-58/obj00000000000000000021 2023-05-31T20:39:11.607 DEBUG:boto:auth_path=/pklxkd-58/obj00000000000000000021 2023-05-31T20:39:11.607 DEBUG:boto:Method: HEAD 2023-05-31T20:39:11.607 DEBUG:boto:Path: /pklxkd-58/obj00000000000000000021 2023-05-31T20:39:11.608 DEBUG:boto:Data: 2023-05-31T20:39:11.608 DEBUG:boto:Headers: {} 2023-05-31T20:39:11.608 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:39:11.609 DEBUG:boto:Port: 8000 2023-05-31T20:39:11.609 DEBUG:boto:Params: {} 2023-05-31T20:39:11.609 DEBUG:boto:Token: None 2023-05-31T20:39:11.610 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:39:11 GMT /pklxkd-58/obj00000000000000000021 2023-05-31T20:39:11.610 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:Oqw+HxahCHMtkcTJkJX8pZ6/n9E= 2023-05-31T20:39:11.611 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:39:11 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:Oqw+HxahCHMtkcTJkJX8pZ6/n9E=', 'Content-Length': '0'} 2023-05-31T20:39:11.613 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:38:20 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000970fb77e35732c8f-006477b06f-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:39:11 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:39:11.613 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000021 2023-05-31T20:39:11.614 DEBUG:boto:path=/pklxkd-58/obj00000000000000000021 2023-05-31T20:39:11.614 DEBUG:boto:auth_path=/pklxkd-58/obj00000000000000000021 2023-05-31T20:39:11.615 DEBUG:boto:Method: GET 2023-05-31T20:39:11.615 DEBUG:boto:Path: /pklxkd-58/obj00000000000000000021 2023-05-31T20:39:11.615 DEBUG:boto:Data: 2023-05-31T20:39:11.616 DEBUG:boto:Headers: {} 2023-05-31T20:39:11.616 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:39:11.616 DEBUG:boto:Port: 8001 2023-05-31T20:39:11.617 DEBUG:boto:Params: {} 2023-05-31T20:39:11.617 DEBUG:boto:Token: None 2023-05-31T20:39:11.618 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:39:11 GMT /pklxkd-58/obj00000000000000000021 2023-05-31T20:39:11.618 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:1VqsPupJEoIooPv8pNUWumtUT6s= 2023-05-31T20:39:11.619 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:39:11 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:1VqsPupJEoIooPv8pNUWumtUT6s=', 'Content-Length': '0'} 2023-05-31T20:39:11.622 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:38:20 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000080d29a6901e6ad08-006477b06f-1c17-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:39:11 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:39:11.664 DEBUG:boto:path=/pklxkd-58/obj00000000000000000021 2023-05-31T20:39:11.665 DEBUG:boto:auth_path=/pklxkd-58/obj00000000000000000021 2023-05-31T20:39:11.666 DEBUG:boto:Method: GET 2023-05-31T20:39:11.666 DEBUG:boto:Path: /pklxkd-58/obj00000000000000000021 2023-05-31T20:39:11.666 DEBUG:boto:Data: 2023-05-31T20:39:11.667 DEBUG:boto:Headers: {} 2023-05-31T20:39:11.667 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:39:11.668 DEBUG:boto:Port: 8000 2023-05-31T20:39:11.668 DEBUG:boto:Params: {} 2023-05-31T20:39:11.668 DEBUG:boto:Token: None 2023-05-31T20:39:11.669 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:39:11 GMT /pklxkd-58/obj00000000000000000021 2023-05-31T20:39:11.670 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:1VqsPupJEoIooPv8pNUWumtUT6s= 2023-05-31T20:39:11.670 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:39:11 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:1VqsPupJEoIooPv8pNUWumtUT6s=', 'Content-Length': '0'} 2023-05-31T20:39:11.672 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:38:20 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d16f9ed2288cc604-006477b06f-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:39:11 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:39:11.673 DEBUG:boto:path=/pklxkd-58/ 2023-05-31T20:39:11.674 DEBUG:boto:auth_path=/pklxkd-58/ 2023-05-31T20:39:11.674 DEBUG:boto:path=/pklxkd-58/?prefix=obj00000000000000000021 2023-05-31T20:39:11.674 DEBUG:boto:auth_path=/pklxkd-58/?prefix=obj00000000000000000021 2023-05-31T20:39:11.675 DEBUG:boto:Method: GET 2023-05-31T20:39:11.675 DEBUG:boto:Path: /pklxkd-58/?prefix=obj00000000000000000021 2023-05-31T20:39:11.675 DEBUG:boto:Data: 2023-05-31T20:39:11.676 DEBUG:boto:Headers: {} 2023-05-31T20:39:11.676 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:39:11.677 DEBUG:boto:Port: 8001 2023-05-31T20:39:11.677 DEBUG:boto:Params: {} 2023-05-31T20:39:11.678 DEBUG:boto:Token: None 2023-05-31T20:39:11.678 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:39:11 GMT /pklxkd-58/ 2023-05-31T20:39:11.678 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:GHNSu1S8h4v2Qp+durSNxgCJG9E= 2023-05-31T20:39:11.679 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:39:11 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:GHNSu1S8h4v2Qp+durSNxgCJG9E=', 'Content-Length': '0'} 2023-05-31T20:39:11.681 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000086b8dbc209e621cf-006477b06f-1c17-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:39:11 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:39:11.682 DEBUG:boto:b'pklxkd-58obj000000000000000000211000falseobj000000000000000000212023-05-31T20:38:20.802Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:39:11.683 DEBUG:boto:path=/pklxkd-58/ 2023-05-31T20:39:11.683 DEBUG:boto:auth_path=/pklxkd-58/ 2023-05-31T20:39:11.684 DEBUG:boto:path=/pklxkd-58/?prefix=obj00000000000000000021 2023-05-31T20:39:11.684 DEBUG:boto:auth_path=/pklxkd-58/?prefix=obj00000000000000000021 2023-05-31T20:39:11.684 DEBUG:boto:Method: GET 2023-05-31T20:39:11.685 DEBUG:boto:Path: /pklxkd-58/?prefix=obj00000000000000000021 2023-05-31T20:39:11.685 DEBUG:boto:Data: 2023-05-31T20:39:11.685 DEBUG:boto:Headers: {} 2023-05-31T20:39:11.686 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:39:11.686 DEBUG:boto:Port: 8000 2023-05-31T20:39:11.686 DEBUG:boto:Params: {} 2023-05-31T20:39:11.687 DEBUG:boto:Token: None 2023-05-31T20:39:11.687 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:39:11 GMT /pklxkd-58/ 2023-05-31T20:39:11.688 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:GHNSu1S8h4v2Qp+durSNxgCJG9E= 2023-05-31T20:39:11.688 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:39:11 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:GHNSu1S8h4v2Qp+durSNxgCJG9E=', 'Content-Length': '0'} 2023-05-31T20:39:11.692 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000059dc372fa9e6e6a5-006477b06f-1630-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:39:11 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:39:11.693 DEBUG:boto:b'pklxkd-58obj000000000000000000211000falseobj000000000000000000212023-05-31T20:38:20.802Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:39:11.694 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000022 2023-05-31T20:39:11.694 DEBUG:boto:path=/pklxkd-58/obj00000000000000000022 2023-05-31T20:39:11.694 DEBUG:boto:auth_path=/pklxkd-58/obj00000000000000000022 2023-05-31T20:39:11.695 DEBUG:boto:path=/pklxkd-58/obj00000000000000000022?versionId=null 2023-05-31T20:39:11.695 DEBUG:boto:auth_path=/pklxkd-58/obj00000000000000000022?versionId=null 2023-05-31T20:39:11.695 DEBUG:boto:Method: GET 2023-05-31T20:39:11.696 DEBUG:boto:Path: /pklxkd-58/obj00000000000000000022?versionId=null 2023-05-31T20:39:11.696 DEBUG:boto:Data: 2023-05-31T20:39:11.697 DEBUG:boto:Headers: {} 2023-05-31T20:39:11.697 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:39:11.697 DEBUG:boto:Port: 8001 2023-05-31T20:39:11.698 DEBUG:boto:Params: {} 2023-05-31T20:39:11.698 DEBUG:boto:Token: None 2023-05-31T20:39:11.698 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:39:11 GMT /pklxkd-58/obj00000000000000000022?versionId=null 2023-05-31T20:39:11.699 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:jTZ90JNkZJyWIqg0nFarJrrUvRE= 2023-05-31T20:39:11.699 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:39:11 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:jTZ90JNkZJyWIqg0nFarJrrUvRE=', 'Content-Length': '0'} 2023-05-31T20:39:11.701 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:38:20 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007643e442ed7005eb-006477b06f-1c17-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:39:11 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:39:11.702 DEBUG:boto:path=/pklxkd-58/obj00000000000000000022 2023-05-31T20:39:11.702 DEBUG:boto:auth_path=/pklxkd-58/obj00000000000000000022 2023-05-31T20:39:11.703 DEBUG:boto:path=/pklxkd-58/obj00000000000000000022?versionId=null 2023-05-31T20:39:11.703 DEBUG:boto:auth_path=/pklxkd-58/obj00000000000000000022?versionId=null 2023-05-31T20:39:11.703 DEBUG:boto:Method: GET 2023-05-31T20:39:11.704 DEBUG:boto:Path: /pklxkd-58/obj00000000000000000022?versionId=null 2023-05-31T20:39:11.704 DEBUG:boto:Data: 2023-05-31T20:39:11.705 DEBUG:boto:Headers: {} 2023-05-31T20:39:11.705 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:39:11.705 DEBUG:boto:Port: 8000 2023-05-31T20:39:11.706 DEBUG:boto:Params: {} 2023-05-31T20:39:11.706 DEBUG:boto:Token: None 2023-05-31T20:39:11.706 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:39:11 GMT /pklxkd-58/obj00000000000000000022?versionId=null 2023-05-31T20:39:11.707 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:jTZ90JNkZJyWIqg0nFarJrrUvRE= 2023-05-31T20:39:11.707 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:39:11 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:jTZ90JNkZJyWIqg0nFarJrrUvRE=', 'Content-Length': '0'} 2023-05-31T20:39:11.709 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:38:20 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008b8a7786b132bb0d-006477b06f-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:39:11 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:39:11.752 DEBUG:boto:path=/pklxkd-58/obj00000000000000000022 2023-05-31T20:39:11.753 DEBUG:boto:auth_path=/pklxkd-58/obj00000000000000000022 2023-05-31T20:39:11.753 DEBUG:boto:path=/pklxkd-58/obj00000000000000000022?versionId=null 2023-05-31T20:39:11.754 DEBUG:boto:auth_path=/pklxkd-58/obj00000000000000000022?versionId=null 2023-05-31T20:39:11.754 DEBUG:boto:Method: HEAD 2023-05-31T20:39:11.755 DEBUG:boto:Path: /pklxkd-58/obj00000000000000000022?versionId=null 2023-05-31T20:39:11.755 DEBUG:boto:Data: 2023-05-31T20:39:11.755 DEBUG:boto:Headers: {} 2023-05-31T20:39:11.756 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:39:11.756 DEBUG:boto:Port: 8001 2023-05-31T20:39:11.756 DEBUG:boto:Params: {} 2023-05-31T20:39:11.757 DEBUG:boto:Token: None 2023-05-31T20:39:11.757 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:39:11 GMT /pklxkd-58/obj00000000000000000022?versionId=null 2023-05-31T20:39:11.758 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:wgFqIcBh2k/ERM2oAyTNegJ0w8U= 2023-05-31T20:39:11.758 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:39:11 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:wgFqIcBh2k/ERM2oAyTNegJ0w8U=', 'Content-Length': '0'} 2023-05-31T20:39:11.760 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:38:20 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000028ea153c1c88849b-006477b06f-1c17-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:39:11 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:39:11.761 DEBUG:boto:path=/pklxkd-58/obj00000000000000000022 2023-05-31T20:39:11.762 DEBUG:boto:auth_path=/pklxkd-58/obj00000000000000000022 2023-05-31T20:39:11.762 DEBUG:boto:path=/pklxkd-58/obj00000000000000000022?versionId=null 2023-05-31T20:39:11.762 DEBUG:boto:auth_path=/pklxkd-58/obj00000000000000000022?versionId=null 2023-05-31T20:39:11.763 DEBUG:boto:Method: HEAD 2023-05-31T20:39:11.763 DEBUG:boto:Path: /pklxkd-58/obj00000000000000000022?versionId=null 2023-05-31T20:39:11.763 DEBUG:boto:Data: 2023-05-31T20:39:11.764 DEBUG:boto:Headers: {} 2023-05-31T20:39:11.764 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:39:11.764 DEBUG:boto:Port: 8000 2023-05-31T20:39:11.765 DEBUG:boto:Params: {} 2023-05-31T20:39:11.765 DEBUG:boto:Token: None 2023-05-31T20:39:11.766 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:39:11 GMT /pklxkd-58/obj00000000000000000022?versionId=null 2023-05-31T20:39:11.766 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:wgFqIcBh2k/ERM2oAyTNegJ0w8U= 2023-05-31T20:39:11.767 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:39:11 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:wgFqIcBh2k/ERM2oAyTNegJ0w8U=', 'Content-Length': '0'} 2023-05-31T20:39:11.769 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:38:20 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005ff526112f595e95-006477b06f-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:39:11 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:39:11.770 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000022 2023-05-31T20:39:11.770 DEBUG:boto:path=/pklxkd-58/obj00000000000000000022 2023-05-31T20:39:11.771 DEBUG:boto:auth_path=/pklxkd-58/obj00000000000000000022 2023-05-31T20:39:11.771 DEBUG:boto:path=/pklxkd-58/obj00000000000000000022?versionId=null 2023-05-31T20:39:11.771 DEBUG:boto:auth_path=/pklxkd-58/obj00000000000000000022?versionId=null 2023-05-31T20:39:11.772 DEBUG:boto:Method: GET 2023-05-31T20:39:11.772 DEBUG:boto:Path: /pklxkd-58/obj00000000000000000022?versionId=null 2023-05-31T20:39:11.772 DEBUG:boto:Data: 2023-05-31T20:39:11.773 DEBUG:boto:Headers: {} 2023-05-31T20:39:11.773 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:39:11.774 DEBUG:boto:Port: 8001 2023-05-31T20:39:11.774 DEBUG:boto:Params: {} 2023-05-31T20:39:11.774 DEBUG:boto:Token: None 2023-05-31T20:39:11.775 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:39:11 GMT /pklxkd-58/obj00000000000000000022?versionId=null 2023-05-31T20:39:11.775 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:jTZ90JNkZJyWIqg0nFarJrrUvRE= 2023-05-31T20:39:11.776 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:39:11 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:jTZ90JNkZJyWIqg0nFarJrrUvRE=', 'Content-Length': '0'} 2023-05-31T20:39:11.778 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:38:20 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a68a76577e97b3cf-006477b06f-1c17-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:39:11 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:39:11.779 DEBUG:boto:path=/pklxkd-58/obj00000000000000000022 2023-05-31T20:39:11.779 DEBUG:boto:auth_path=/pklxkd-58/obj00000000000000000022 2023-05-31T20:39:11.779 DEBUG:boto:path=/pklxkd-58/obj00000000000000000022?versionId=null 2023-05-31T20:39:11.780 DEBUG:boto:auth_path=/pklxkd-58/obj00000000000000000022?versionId=null 2023-05-31T20:39:11.780 DEBUG:boto:Method: GET 2023-05-31T20:39:11.781 DEBUG:boto:Path: /pklxkd-58/obj00000000000000000022?versionId=null 2023-05-31T20:39:11.781 DEBUG:boto:Data: 2023-05-31T20:39:11.781 DEBUG:boto:Headers: {} 2023-05-31T20:39:11.782 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:39:11.782 DEBUG:boto:Port: 8000 2023-05-31T20:39:11.782 DEBUG:boto:Params: {} 2023-05-31T20:39:11.783 DEBUG:boto:Token: None 2023-05-31T20:39:11.783 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:39:11 GMT /pklxkd-58/obj00000000000000000022?versionId=null 2023-05-31T20:39:11.784 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:jTZ90JNkZJyWIqg0nFarJrrUvRE= 2023-05-31T20:39:11.784 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:39:11 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:jTZ90JNkZJyWIqg0nFarJrrUvRE=', 'Content-Length': '0'} 2023-05-31T20:39:11.786 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:38:20 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000ecd9eadffa33921-006477b06f-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:39:11 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:39:11.787 DEBUG:boto:path=/pklxkd-58/ 2023-05-31T20:39:11.788 DEBUG:boto:auth_path=/pklxkd-58/ 2023-05-31T20:39:11.788 DEBUG:boto:path=/pklxkd-58/?prefix=obj00000000000000000022 2023-05-31T20:39:11.789 DEBUG:boto:auth_path=/pklxkd-58/?prefix=obj00000000000000000022 2023-05-31T20:39:11.789 DEBUG:boto:Method: GET 2023-05-31T20:39:11.789 DEBUG:boto:Path: /pklxkd-58/?prefix=obj00000000000000000022 2023-05-31T20:39:11.790 DEBUG:boto:Data: 2023-05-31T20:39:11.790 DEBUG:boto:Headers: {} 2023-05-31T20:39:11.790 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:39:11.791 DEBUG:boto:Port: 8001 2023-05-31T20:39:11.791 DEBUG:boto:Params: {} 2023-05-31T20:39:11.791 DEBUG:boto:Token: None 2023-05-31T20:39:11.792 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:39:11 GMT /pklxkd-58/ 2023-05-31T20:39:11.792 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:GHNSu1S8h4v2Qp+durSNxgCJG9E= 2023-05-31T20:39:11.793 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:39:11 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:GHNSu1S8h4v2Qp+durSNxgCJG9E=', 'Content-Length': '0'} 2023-05-31T20:39:11.795 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000599f474e69892ab7-006477b06f-1c17-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:39:11 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:39:11.796 DEBUG:boto:b'pklxkd-58obj000000000000000000221000falseobj000000000000000000222023-05-31T20:38:20.598Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:39:11.797 DEBUG:boto:path=/pklxkd-58/ 2023-05-31T20:39:11.797 DEBUG:boto:auth_path=/pklxkd-58/ 2023-05-31T20:39:11.798 DEBUG:boto:path=/pklxkd-58/?prefix=obj00000000000000000022 2023-05-31T20:39:11.798 DEBUG:boto:auth_path=/pklxkd-58/?prefix=obj00000000000000000022 2023-05-31T20:39:11.798 DEBUG:boto:Method: GET 2023-05-31T20:39:11.799 DEBUG:boto:Path: /pklxkd-58/?prefix=obj00000000000000000022 2023-05-31T20:39:11.799 DEBUG:boto:Data: 2023-05-31T20:39:11.799 DEBUG:boto:Headers: {} 2023-05-31T20:39:11.800 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:39:11.800 DEBUG:boto:Port: 8000 2023-05-31T20:39:11.801 DEBUG:boto:Params: {} 2023-05-31T20:39:11.801 DEBUG:boto:Token: None 2023-05-31T20:39:11.802 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:39:11 GMT /pklxkd-58/ 2023-05-31T20:39:11.802 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:GHNSu1S8h4v2Qp+durSNxgCJG9E= 2023-05-31T20:39:11.802 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:39:11 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:GHNSu1S8h4v2Qp+durSNxgCJG9E=', 'Content-Length': '0'} 2023-05-31T20:39:11.806 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000052298bd1828ff9ee-006477b06f-1630-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:39:11 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:39:11.807 DEBUG:boto:b'pklxkd-58obj000000000000000000221000falseobj000000000000000000222023-05-31T20:38:20.598Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:39:11.808 DEBUG:boto:path=/pklxkd-58/obj00000000000000000022 2023-05-31T20:39:11.808 DEBUG:boto:auth_path=/pklxkd-58/obj00000000000000000022 2023-05-31T20:39:11.809 DEBUG:boto:Method: HEAD 2023-05-31T20:39:11.809 DEBUG:boto:Path: /pklxkd-58/obj00000000000000000022 2023-05-31T20:39:11.809 DEBUG:boto:Data: 2023-05-31T20:39:11.810 DEBUG:boto:Headers: {} 2023-05-31T20:39:11.810 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:39:11.811 DEBUG:boto:Port: 8001 2023-05-31T20:39:11.811 DEBUG:boto:Params: {} 2023-05-31T20:39:11.811 DEBUG:boto:Token: None 2023-05-31T20:39:11.812 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:39:11 GMT /pklxkd-58/obj00000000000000000022 2023-05-31T20:39:11.812 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:LgerLI0YOUKV82CUjC+1aWKHElE= 2023-05-31T20:39:11.813 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:39:11 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:LgerLI0YOUKV82CUjC+1aWKHElE=', 'Content-Length': '0'} 2023-05-31T20:39:11.815 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:38:20 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001705ee0eb088d69a-006477b06f-1c17-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:39:11 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:39:11.815 DEBUG:boto:path=/pklxkd-58/obj00000000000000000022 2023-05-31T20:39:11.816 DEBUG:boto:auth_path=/pklxkd-58/obj00000000000000000022 2023-05-31T20:39:11.816 DEBUG:boto:Method: HEAD 2023-05-31T20:39:11.816 DEBUG:boto:Path: /pklxkd-58/obj00000000000000000022 2023-05-31T20:39:11.817 DEBUG:boto:Data: 2023-05-31T20:39:11.817 DEBUG:boto:Headers: {} 2023-05-31T20:39:11.817 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:39:11.818 DEBUG:boto:Port: 8000 2023-05-31T20:39:11.818 DEBUG:boto:Params: {} 2023-05-31T20:39:11.819 DEBUG:boto:Token: None 2023-05-31T20:39:11.819 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:39:11 GMT /pklxkd-58/obj00000000000000000022 2023-05-31T20:39:11.819 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:LgerLI0YOUKV82CUjC+1aWKHElE= 2023-05-31T20:39:11.820 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:39:11 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:LgerLI0YOUKV82CUjC+1aWKHElE=', 'Content-Length': '0'} 2023-05-31T20:39:11.822 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:38:20 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f7019388b4f7901a-006477b06f-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:39:11 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:39:11.822 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000022 2023-05-31T20:39:11.823 DEBUG:boto:path=/pklxkd-58/obj00000000000000000022 2023-05-31T20:39:11.823 DEBUG:boto:auth_path=/pklxkd-58/obj00000000000000000022 2023-05-31T20:39:11.823 DEBUG:boto:Method: GET 2023-05-31T20:39:11.824 DEBUG:boto:Path: /pklxkd-58/obj00000000000000000022 2023-05-31T20:39:11.824 DEBUG:boto:Data: 2023-05-31T20:39:11.824 DEBUG:boto:Headers: {} 2023-05-31T20:39:11.825 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:39:11.825 DEBUG:boto:Port: 8001 2023-05-31T20:39:11.826 DEBUG:boto:Params: {} 2023-05-31T20:39:11.826 DEBUG:boto:Token: None 2023-05-31T20:39:11.826 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:39:11 GMT /pklxkd-58/obj00000000000000000022 2023-05-31T20:39:11.827 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:KF2AhY3ACYz82kAFdSsQU4uGjGY= 2023-05-31T20:39:11.827 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:39:11 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:KF2AhY3ACYz82kAFdSsQU4uGjGY=', 'Content-Length': '0'} 2023-05-31T20:39:11.829 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:38:20 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006a039b9ffdaf913c-006477b06f-1c17-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:39:11 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:39:11.830 DEBUG:boto:path=/pklxkd-58/obj00000000000000000022 2023-05-31T20:39:11.830 DEBUG:boto:auth_path=/pklxkd-58/obj00000000000000000022 2023-05-31T20:39:11.831 DEBUG:boto:Method: GET 2023-05-31T20:39:11.831 DEBUG:boto:Path: /pklxkd-58/obj00000000000000000022 2023-05-31T20:39:11.831 DEBUG:boto:Data: 2023-05-31T20:39:11.832 DEBUG:boto:Headers: {} 2023-05-31T20:39:11.832 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:39:11.833 DEBUG:boto:Port: 8000 2023-05-31T20:39:11.833 DEBUG:boto:Params: {} 2023-05-31T20:39:11.833 DEBUG:boto:Token: None 2023-05-31T20:39:11.834 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:39:11 GMT /pklxkd-58/obj00000000000000000022 2023-05-31T20:39:11.834 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:KF2AhY3ACYz82kAFdSsQU4uGjGY= 2023-05-31T20:39:11.835 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:39:11 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:KF2AhY3ACYz82kAFdSsQU4uGjGY=', 'Content-Length': '0'} 2023-05-31T20:39:11.837 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:38:20 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ce29db3a034b70b9-006477b06f-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:39:11 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:39:11.885 DEBUG:boto:path=/pklxkd-58/ 2023-05-31T20:39:11.885 DEBUG:boto:auth_path=/pklxkd-58/ 2023-05-31T20:39:11.886 DEBUG:boto:path=/pklxkd-58/?prefix=obj00000000000000000022 2023-05-31T20:39:11.886 DEBUG:boto:auth_path=/pklxkd-58/?prefix=obj00000000000000000022 2023-05-31T20:39:11.886 DEBUG:boto:Method: GET 2023-05-31T20:39:11.887 DEBUG:boto:Path: /pklxkd-58/?prefix=obj00000000000000000022 2023-05-31T20:39:11.887 DEBUG:boto:Data: 2023-05-31T20:39:11.887 DEBUG:boto:Headers: {} 2023-05-31T20:39:11.888 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:39:11.888 DEBUG:boto:Port: 8001 2023-05-31T20:39:11.889 DEBUG:boto:Params: {} 2023-05-31T20:39:11.889 DEBUG:boto:Token: None 2023-05-31T20:39:11.890 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:39:11 GMT /pklxkd-58/ 2023-05-31T20:39:11.890 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:GHNSu1S8h4v2Qp+durSNxgCJG9E= 2023-05-31T20:39:11.891 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:39:11 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:GHNSu1S8h4v2Qp+durSNxgCJG9E=', 'Content-Length': '0'} 2023-05-31T20:39:11.917 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000fcdc6bdfe2eea441-006477b06f-1c17-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:39:11 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:39:11.918 DEBUG:boto:b'pklxkd-58obj000000000000000000221000falseobj000000000000000000222023-05-31T20:38:20.598Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:39:11.919 DEBUG:boto:path=/pklxkd-58/ 2023-05-31T20:39:11.919 DEBUG:boto:auth_path=/pklxkd-58/ 2023-05-31T20:39:11.920 DEBUG:boto:path=/pklxkd-58/?prefix=obj00000000000000000022 2023-05-31T20:39:11.920 DEBUG:boto:auth_path=/pklxkd-58/?prefix=obj00000000000000000022 2023-05-31T20:39:11.920 DEBUG:boto:Method: GET 2023-05-31T20:39:11.921 DEBUG:boto:Path: /pklxkd-58/?prefix=obj00000000000000000022 2023-05-31T20:39:11.921 DEBUG:boto:Data: 2023-05-31T20:39:11.921 DEBUG:boto:Headers: {} 2023-05-31T20:39:11.922 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:39:11.922 DEBUG:boto:Port: 8000 2023-05-31T20:39:11.922 DEBUG:boto:Params: {} 2023-05-31T20:39:11.923 DEBUG:boto:Token: None 2023-05-31T20:39:11.923 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:39:11 GMT /pklxkd-58/ 2023-05-31T20:39:11.924 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:GHNSu1S8h4v2Qp+durSNxgCJG9E= 2023-05-31T20:39:11.924 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:39:11 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:GHNSu1S8h4v2Qp+durSNxgCJG9E=', 'Content-Length': '0'} 2023-05-31T20:39:11.928 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c1ca64f0162b54a4-006477b06f-1630-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:39:11 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:39:11.929 DEBUG:boto:b'pklxkd-58obj000000000000000000221000falseobj000000000000000000222023-05-31T20:38:20.598Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:39:11.930 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000023 2023-05-31T20:39:11.931 DEBUG:boto:path=/pklxkd-58/obj00000000000000000023 2023-05-31T20:39:11.931 DEBUG:boto:auth_path=/pklxkd-58/obj00000000000000000023 2023-05-31T20:39:11.932 DEBUG:boto:path=/pklxkd-58/obj00000000000000000023?versionId=null 2023-05-31T20:39:11.932 DEBUG:boto:auth_path=/pklxkd-58/obj00000000000000000023?versionId=null 2023-05-31T20:39:11.932 DEBUG:boto:Method: GET 2023-05-31T20:39:11.933 DEBUG:boto:Path: /pklxkd-58/obj00000000000000000023?versionId=null 2023-05-31T20:39:11.933 DEBUG:boto:Data: 2023-05-31T20:39:11.934 DEBUG:boto:Headers: {} 2023-05-31T20:39:11.934 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:39:11.934 DEBUG:boto:Port: 8001 2023-05-31T20:39:11.935 DEBUG:boto:Params: {} 2023-05-31T20:39:11.935 DEBUG:boto:Token: None 2023-05-31T20:39:11.936 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:39:11 GMT /pklxkd-58/obj00000000000000000023?versionId=null 2023-05-31T20:39:11.936 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:RT7NbPEwesO0B12YwNQIliH0Xl8= 2023-05-31T20:39:11.936 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:39:11 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:RT7NbPEwesO0B12YwNQIliH0Xl8=', 'Content-Length': '0'} 2023-05-31T20:39:11.940 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:38:20 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007a03bada2db3980f-006477b06f-1c17-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:39:11 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:39:11.980 DEBUG:boto:path=/pklxkd-58/obj00000000000000000023 2023-05-31T20:39:11.981 DEBUG:boto:auth_path=/pklxkd-58/obj00000000000000000023 2023-05-31T20:39:11.981 DEBUG:boto:path=/pklxkd-58/obj00000000000000000023?versionId=null 2023-05-31T20:39:11.982 DEBUG:boto:auth_path=/pklxkd-58/obj00000000000000000023?versionId=null 2023-05-31T20:39:11.982 DEBUG:boto:Method: GET 2023-05-31T20:39:11.982 DEBUG:boto:Path: /pklxkd-58/obj00000000000000000023?versionId=null 2023-05-31T20:39:11.983 DEBUG:boto:Data: 2023-05-31T20:39:11.983 DEBUG:boto:Headers: {} 2023-05-31T20:39:11.983 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:39:11.984 DEBUG:boto:Port: 8000 2023-05-31T20:39:11.984 DEBUG:boto:Params: {} 2023-05-31T20:39:11.985 DEBUG:boto:Token: None 2023-05-31T20:39:11.985 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:39:11 GMT /pklxkd-58/obj00000000000000000023?versionId=null 2023-05-31T20:39:11.985 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:RT7NbPEwesO0B12YwNQIliH0Xl8= 2023-05-31T20:39:11.986 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:39:11 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:RT7NbPEwesO0B12YwNQIliH0Xl8=', 'Content-Length': '0'} 2023-05-31T20:39:11.988 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:38:20 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006e161855a0d88260-006477b06f-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:39:11 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:39:11.989 DEBUG:boto:path=/pklxkd-58/obj00000000000000000023 2023-05-31T20:39:11.989 DEBUG:boto:auth_path=/pklxkd-58/obj00000000000000000023 2023-05-31T20:39:11.989 DEBUG:boto:path=/pklxkd-58/obj00000000000000000023?versionId=null 2023-05-31T20:39:11.990 DEBUG:boto:auth_path=/pklxkd-58/obj00000000000000000023?versionId=null 2023-05-31T20:39:11.990 DEBUG:boto:Method: HEAD 2023-05-31T20:39:11.990 DEBUG:boto:Path: /pklxkd-58/obj00000000000000000023?versionId=null 2023-05-31T20:39:11.991 DEBUG:boto:Data: 2023-05-31T20:39:11.991 DEBUG:boto:Headers: {} 2023-05-31T20:39:11.991 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:39:11.992 DEBUG:boto:Port: 8001 2023-05-31T20:39:11.992 DEBUG:boto:Params: {} 2023-05-31T20:39:11.992 DEBUG:boto:Token: None 2023-05-31T20:39:11.993 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:39:11 GMT /pklxkd-58/obj00000000000000000023?versionId=null 2023-05-31T20:39:11.993 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:gGxghrri1BVHAlIdujqqcm5FT+c= 2023-05-31T20:39:11.994 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:39:11 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:gGxghrri1BVHAlIdujqqcm5FT+c=', 'Content-Length': '0'} 2023-05-31T20:39:11.996 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:38:20 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009a6bfc9fe57f5030-006477b06f-1c17-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:39:11 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:39:11.996 DEBUG:boto:path=/pklxkd-58/obj00000000000000000023 2023-05-31T20:39:11.997 DEBUG:boto:auth_path=/pklxkd-58/obj00000000000000000023 2023-05-31T20:39:11.997 DEBUG:boto:path=/pklxkd-58/obj00000000000000000023?versionId=null 2023-05-31T20:39:11.997 DEBUG:boto:auth_path=/pklxkd-58/obj00000000000000000023?versionId=null 2023-05-31T20:39:11.998 DEBUG:boto:Method: HEAD 2023-05-31T20:39:11.998 DEBUG:boto:Path: /pklxkd-58/obj00000000000000000023?versionId=null 2023-05-31T20:39:11.998 DEBUG:boto:Data: 2023-05-31T20:39:11.999 DEBUG:boto:Headers: {} 2023-05-31T20:39:11.999 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:39:11.999 DEBUG:boto:Port: 8000 2023-05-31T20:39:12.000 DEBUG:boto:Params: {} 2023-05-31T20:39:12.000 DEBUG:boto:Token: None 2023-05-31T20:39:12.000 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:39:12 GMT /pklxkd-58/obj00000000000000000023?versionId=null 2023-05-31T20:39:12.001 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:L6RIBDZ2ri2sY5zt5uiNxX5itzc= 2023-05-31T20:39:12.001 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:39:12 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:L6RIBDZ2ri2sY5zt5uiNxX5itzc=', 'Content-Length': '0'} 2023-05-31T20:39:12.003 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:38:20 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000efd5fa8fd315e5c5-006477b06f-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:39:11 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:39:12.004 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000023 2023-05-31T20:39:12.004 DEBUG:boto:path=/pklxkd-58/obj00000000000000000023 2023-05-31T20:39:12.005 DEBUG:boto:auth_path=/pklxkd-58/obj00000000000000000023 2023-05-31T20:39:12.005 DEBUG:boto:path=/pklxkd-58/obj00000000000000000023?versionId=null 2023-05-31T20:39:12.006 DEBUG:boto:auth_path=/pklxkd-58/obj00000000000000000023?versionId=null 2023-05-31T20:39:12.006 DEBUG:boto:Method: GET 2023-05-31T20:39:12.006 DEBUG:boto:Path: /pklxkd-58/obj00000000000000000023?versionId=null 2023-05-31T20:39:12.007 DEBUG:boto:Data: 2023-05-31T20:39:12.007 DEBUG:boto:Headers: {} 2023-05-31T20:39:12.007 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:39:12.008 DEBUG:boto:Port: 8001 2023-05-31T20:39:12.008 DEBUG:boto:Params: {} 2023-05-31T20:39:12.008 DEBUG:boto:Token: None 2023-05-31T20:39:12.009 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:39:12 GMT /pklxkd-58/obj00000000000000000023?versionId=null 2023-05-31T20:39:12.009 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:5+piLKtU2WDxE/RSvt06xwZCU0c= 2023-05-31T20:39:12.010 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:39:12 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:5+piLKtU2WDxE/RSvt06xwZCU0c=', 'Content-Length': '0'} 2023-05-31T20:39:12.012 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:38:20 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a0b518009fa3e3ad-006477b070-1c17-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:39:12 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:39:12.012 DEBUG:boto:path=/pklxkd-58/obj00000000000000000023 2023-05-31T20:39:12.013 DEBUG:boto:auth_path=/pklxkd-58/obj00000000000000000023 2023-05-31T20:39:12.013 DEBUG:boto:path=/pklxkd-58/obj00000000000000000023?versionId=null 2023-05-31T20:39:12.014 DEBUG:boto:auth_path=/pklxkd-58/obj00000000000000000023?versionId=null 2023-05-31T20:39:12.014 DEBUG:boto:Method: GET 2023-05-31T20:39:12.014 DEBUG:boto:Path: /pklxkd-58/obj00000000000000000023?versionId=null 2023-05-31T20:39:12.015 DEBUG:boto:Data: 2023-05-31T20:39:12.015 DEBUG:boto:Headers: {} 2023-05-31T20:39:12.015 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:39:12.016 DEBUG:boto:Port: 8000 2023-05-31T20:39:12.016 DEBUG:boto:Params: {} 2023-05-31T20:39:12.016 DEBUG:boto:Token: None 2023-05-31T20:39:12.017 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:39:12 GMT /pklxkd-58/obj00000000000000000023?versionId=null 2023-05-31T20:39:12.017 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:5+piLKtU2WDxE/RSvt06xwZCU0c= 2023-05-31T20:39:12.018 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:39:12 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:5+piLKtU2WDxE/RSvt06xwZCU0c=', 'Content-Length': '0'} 2023-05-31T20:39:12.020 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:38:20 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009c5687e9789231e6-006477b070-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:39:12 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:39:12.021 DEBUG:boto:path=/pklxkd-58/ 2023-05-31T20:39:12.021 DEBUG:boto:auth_path=/pklxkd-58/ 2023-05-31T20:39:12.021 DEBUG:boto:path=/pklxkd-58/?prefix=obj00000000000000000023 2023-05-31T20:39:12.022 DEBUG:boto:auth_path=/pklxkd-58/?prefix=obj00000000000000000023 2023-05-31T20:39:12.022 DEBUG:boto:Method: GET 2023-05-31T20:39:12.022 DEBUG:boto:Path: /pklxkd-58/?prefix=obj00000000000000000023 2023-05-31T20:39:12.023 DEBUG:boto:Data: 2023-05-31T20:39:12.023 DEBUG:boto:Headers: {} 2023-05-31T20:39:12.023 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:39:12.024 DEBUG:boto:Port: 8001 2023-05-31T20:39:12.024 DEBUG:boto:Params: {} 2023-05-31T20:39:12.025 DEBUG:boto:Token: None 2023-05-31T20:39:12.025 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:39:12 GMT /pklxkd-58/ 2023-05-31T20:39:12.025 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:ryNlxDRGgK16+XFs6oQP3KSCLXY= 2023-05-31T20:39:12.026 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:39:12 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:ryNlxDRGgK16+XFs6oQP3KSCLXY=', 'Content-Length': '0'} 2023-05-31T20:39:12.028 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000057dd703db6738ca5-006477b070-1c17-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:39:12 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:39:12.029 DEBUG:boto:b'pklxkd-58obj000000000000000000231000falseobj000000000000000000232023-05-31T20:38:20.347Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:39:12.030 DEBUG:boto:path=/pklxkd-58/ 2023-05-31T20:39:12.030 DEBUG:boto:auth_path=/pklxkd-58/ 2023-05-31T20:39:12.031 DEBUG:boto:path=/pklxkd-58/?prefix=obj00000000000000000023 2023-05-31T20:39:12.031 DEBUG:boto:auth_path=/pklxkd-58/?prefix=obj00000000000000000023 2023-05-31T20:39:12.031 DEBUG:boto:Method: GET 2023-05-31T20:39:12.032 DEBUG:boto:Path: /pklxkd-58/?prefix=obj00000000000000000023 2023-05-31T20:39:12.032 DEBUG:boto:Data: 2023-05-31T20:39:12.032 DEBUG:boto:Headers: {} 2023-05-31T20:39:12.033 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:39:12.033 DEBUG:boto:Port: 8000 2023-05-31T20:39:12.033 DEBUG:boto:Params: {} 2023-05-31T20:39:12.034 DEBUG:boto:Token: None 2023-05-31T20:39:12.034 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:39:12 GMT /pklxkd-58/ 2023-05-31T20:39:12.035 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:ryNlxDRGgK16+XFs6oQP3KSCLXY= 2023-05-31T20:39:12.035 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:39:12 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:ryNlxDRGgK16+XFs6oQP3KSCLXY=', 'Content-Length': '0'} 2023-05-31T20:39:12.039 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000076b0d4854bdf8114-006477b070-1630-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:39:12 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:39:12.039 DEBUG:boto:b'pklxkd-58obj000000000000000000231000falseobj000000000000000000232023-05-31T20:38:20.347Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:39:12.040 DEBUG:boto:path=/pklxkd-58/obj00000000000000000023 2023-05-31T20:39:12.041 DEBUG:boto:auth_path=/pklxkd-58/obj00000000000000000023 2023-05-31T20:39:12.041 DEBUG:boto:Method: HEAD 2023-05-31T20:39:12.041 DEBUG:boto:Path: /pklxkd-58/obj00000000000000000023 2023-05-31T20:39:12.042 DEBUG:boto:Data: 2023-05-31T20:39:12.042 DEBUG:boto:Headers: {} 2023-05-31T20:39:12.042 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:39:12.043 DEBUG:boto:Port: 8001 2023-05-31T20:39:12.043 DEBUG:boto:Params: {} 2023-05-31T20:39:12.043 DEBUG:boto:Token: None 2023-05-31T20:39:12.044 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:39:12 GMT /pklxkd-58/obj00000000000000000023 2023-05-31T20:39:12.044 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:+MmuA6IzvDHldfx0W8a+4WADBHg= 2023-05-31T20:39:12.045 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:39:12 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:+MmuA6IzvDHldfx0W8a+4WADBHg=', 'Content-Length': '0'} 2023-05-31T20:39:12.046 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:38:20 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002d65ff19d106bd61-006477b070-1c17-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:39:12 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:39:12.047 DEBUG:boto:path=/pklxkd-58/obj00000000000000000023 2023-05-31T20:39:12.047 DEBUG:boto:auth_path=/pklxkd-58/obj00000000000000000023 2023-05-31T20:39:12.048 DEBUG:boto:Method: HEAD 2023-05-31T20:39:12.048 DEBUG:boto:Path: /pklxkd-58/obj00000000000000000023 2023-05-31T20:39:12.049 DEBUG:boto:Data: 2023-05-31T20:39:12.049 DEBUG:boto:Headers: {} 2023-05-31T20:39:12.049 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:39:12.050 DEBUG:boto:Port: 8000 2023-05-31T20:39:12.050 DEBUG:boto:Params: {} 2023-05-31T20:39:12.050 DEBUG:boto:Token: None 2023-05-31T20:39:12.051 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:39:12 GMT /pklxkd-58/obj00000000000000000023 2023-05-31T20:39:12.051 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:+MmuA6IzvDHldfx0W8a+4WADBHg= 2023-05-31T20:39:12.051 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:39:12 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:+MmuA6IzvDHldfx0W8a+4WADBHg=', 'Content-Length': '0'} 2023-05-31T20:39:12.056 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:38:20 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000029060c52740e74b9-006477b070-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:39:12 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:39:12.057 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000023 2023-05-31T20:39:12.057 DEBUG:boto:path=/pklxkd-58/obj00000000000000000023 2023-05-31T20:39:12.057 DEBUG:boto:auth_path=/pklxkd-58/obj00000000000000000023 2023-05-31T20:39:12.058 DEBUG:boto:Method: GET 2023-05-31T20:39:12.058 DEBUG:boto:Path: /pklxkd-58/obj00000000000000000023 2023-05-31T20:39:12.058 DEBUG:boto:Data: 2023-05-31T20:39:12.059 DEBUG:boto:Headers: {} 2023-05-31T20:39:12.059 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:39:12.059 DEBUG:boto:Port: 8001 2023-05-31T20:39:12.060 DEBUG:boto:Params: {} 2023-05-31T20:39:12.060 DEBUG:boto:Token: None 2023-05-31T20:39:12.061 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:39:12 GMT /pklxkd-58/obj00000000000000000023 2023-05-31T20:39:12.061 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:FVKw7Ml6hyZf4hmKkrrX8kRDb44= 2023-05-31T20:39:12.062 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:39:12 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:FVKw7Ml6hyZf4hmKkrrX8kRDb44=', 'Content-Length': '0'} 2023-05-31T20:39:12.063 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:38:20 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000240dd9b28267ed60-006477b070-1c17-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:39:12 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:39:12.064 DEBUG:boto:path=/pklxkd-58/obj00000000000000000023 2023-05-31T20:39:12.064 DEBUG:boto:auth_path=/pklxkd-58/obj00000000000000000023 2023-05-31T20:39:12.065 DEBUG:boto:Method: GET 2023-05-31T20:39:12.065 DEBUG:boto:Path: /pklxkd-58/obj00000000000000000023 2023-05-31T20:39:12.066 DEBUG:boto:Data: 2023-05-31T20:39:12.066 DEBUG:boto:Headers: {} 2023-05-31T20:39:12.066 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:39:12.067 DEBUG:boto:Port: 8000 2023-05-31T20:39:12.067 DEBUG:boto:Params: {} 2023-05-31T20:39:12.067 DEBUG:boto:Token: None 2023-05-31T20:39:12.068 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:39:12 GMT /pklxkd-58/obj00000000000000000023 2023-05-31T20:39:12.068 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:FVKw7Ml6hyZf4hmKkrrX8kRDb44= 2023-05-31T20:39:12.069 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:39:12 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:FVKw7Ml6hyZf4hmKkrrX8kRDb44=', 'Content-Length': '0'} 2023-05-31T20:39:12.070 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:38:20 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e4928fad11e6cb99-006477b070-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:39:12 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:39:12.071 DEBUG:boto:path=/pklxkd-58/ 2023-05-31T20:39:12.072 DEBUG:boto:auth_path=/pklxkd-58/ 2023-05-31T20:39:12.072 DEBUG:boto:path=/pklxkd-58/?prefix=obj00000000000000000023 2023-05-31T20:39:12.072 DEBUG:boto:auth_path=/pklxkd-58/?prefix=obj00000000000000000023 2023-05-31T20:39:12.073 DEBUG:boto:Method: GET 2023-05-31T20:39:12.073 DEBUG:boto:Path: /pklxkd-58/?prefix=obj00000000000000000023 2023-05-31T20:39:12.073 DEBUG:boto:Data: 2023-05-31T20:39:12.074 DEBUG:boto:Headers: {} 2023-05-31T20:39:12.074 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:39:12.075 DEBUG:boto:Port: 8001 2023-05-31T20:39:12.075 DEBUG:boto:Params: {} 2023-05-31T20:39:12.075 DEBUG:boto:Token: None 2023-05-31T20:39:12.076 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:39:12 GMT /pklxkd-58/ 2023-05-31T20:39:12.076 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:ryNlxDRGgK16+XFs6oQP3KSCLXY= 2023-05-31T20:39:12.077 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:39:12 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:ryNlxDRGgK16+XFs6oQP3KSCLXY=', 'Content-Length': '0'} 2023-05-31T20:39:12.079 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000067605c3d310f92e9-006477b070-1c17-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:39:12 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:39:12.079 DEBUG:boto:b'pklxkd-58obj000000000000000000231000falseobj000000000000000000232023-05-31T20:38:20.347Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:39:12.080 DEBUG:boto:path=/pklxkd-58/ 2023-05-31T20:39:12.081 DEBUG:boto:auth_path=/pklxkd-58/ 2023-05-31T20:39:12.081 DEBUG:boto:path=/pklxkd-58/?prefix=obj00000000000000000023 2023-05-31T20:39:12.081 DEBUG:boto:auth_path=/pklxkd-58/?prefix=obj00000000000000000023 2023-05-31T20:39:12.082 DEBUG:boto:Method: GET 2023-05-31T20:39:12.082 DEBUG:boto:Path: /pklxkd-58/?prefix=obj00000000000000000023 2023-05-31T20:39:12.082 DEBUG:boto:Data: 2023-05-31T20:39:12.083 DEBUG:boto:Headers: {} 2023-05-31T20:39:12.083 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:39:12.083 DEBUG:boto:Port: 8000 2023-05-31T20:39:12.084 DEBUG:boto:Params: {} 2023-05-31T20:39:12.084 DEBUG:boto:Token: None 2023-05-31T20:39:12.085 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:39:12 GMT /pklxkd-58/ 2023-05-31T20:39:12.085 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:ryNlxDRGgK16+XFs6oQP3KSCLXY= 2023-05-31T20:39:12.085 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:39:12 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:ryNlxDRGgK16+XFs6oQP3KSCLXY=', 'Content-Length': '0'} 2023-05-31T20:39:12.089 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a2f9c981ee54384a-006477b070-1630-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:39:12 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:39:12.090 DEBUG:boto:b'pklxkd-58obj000000000000000000231000falseobj000000000000000000232023-05-31T20:38:20.347Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:39:12.091 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000024 2023-05-31T20:39:12.092 DEBUG:boto:path=/pklxkd-58/obj00000000000000000024 2023-05-31T20:39:12.092 DEBUG:boto:auth_path=/pklxkd-58/obj00000000000000000024 2023-05-31T20:39:12.093 DEBUG:boto:path=/pklxkd-58/obj00000000000000000024?versionId=null 2023-05-31T20:39:12.093 DEBUG:boto:auth_path=/pklxkd-58/obj00000000000000000024?versionId=null 2023-05-31T20:39:12.093 DEBUG:boto:Method: GET 2023-05-31T20:39:12.094 DEBUG:boto:Path: /pklxkd-58/obj00000000000000000024?versionId=null 2023-05-31T20:39:12.094 DEBUG:boto:Data: 2023-05-31T20:39:12.095 DEBUG:boto:Headers: {} 2023-05-31T20:39:12.095 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:39:12.096 DEBUG:boto:Port: 8001 2023-05-31T20:39:12.096 DEBUG:boto:Params: {} 2023-05-31T20:39:12.098 DEBUG:boto:Token: None 2023-05-31T20:39:12.098 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:39:12 GMT /pklxkd-58/obj00000000000000000024?versionId=null 2023-05-31T20:39:12.099 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:KdWkqMzHz7mYiDWovBhi5lwEZ8s= 2023-05-31T20:39:12.100 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:39:12 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:KdWkqMzHz7mYiDWovBhi5lwEZ8s=', 'Content-Length': '0'} 2023-05-31T20:39:12.102 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:38:22 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009ae7c3aee0238805-006477b070-1c17-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:39:12 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:39:12.102 DEBUG:boto:path=/pklxkd-58/obj00000000000000000024 2023-05-31T20:39:12.103 DEBUG:boto:auth_path=/pklxkd-58/obj00000000000000000024 2023-05-31T20:39:12.103 DEBUG:boto:path=/pklxkd-58/obj00000000000000000024?versionId=null 2023-05-31T20:39:12.104 DEBUG:boto:auth_path=/pklxkd-58/obj00000000000000000024?versionId=null 2023-05-31T20:39:12.104 DEBUG:boto:Method: GET 2023-05-31T20:39:12.104 DEBUG:boto:Path: /pklxkd-58/obj00000000000000000024?versionId=null 2023-05-31T20:39:12.105 DEBUG:boto:Data: 2023-05-31T20:39:12.105 DEBUG:boto:Headers: {} 2023-05-31T20:39:12.106 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:39:12.107 DEBUG:boto:Port: 8000 2023-05-31T20:39:12.107 DEBUG:boto:Params: {} 2023-05-31T20:39:12.108 DEBUG:boto:Token: None 2023-05-31T20:39:12.109 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:39:12 GMT /pklxkd-58/obj00000000000000000024?versionId=null 2023-05-31T20:39:12.109 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:KdWkqMzHz7mYiDWovBhi5lwEZ8s= 2023-05-31T20:39:12.110 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:39:12 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:KdWkqMzHz7mYiDWovBhi5lwEZ8s=', 'Content-Length': '0'} 2023-05-31T20:39:12.113 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:38:22 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000324c6393a109fa06-006477b070-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:39:12 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:39:12.114 DEBUG:boto:path=/pklxkd-58/obj00000000000000000024 2023-05-31T20:39:12.115 DEBUG:boto:auth_path=/pklxkd-58/obj00000000000000000024 2023-05-31T20:39:12.115 DEBUG:boto:path=/pklxkd-58/obj00000000000000000024?versionId=null 2023-05-31T20:39:12.115 DEBUG:boto:auth_path=/pklxkd-58/obj00000000000000000024?versionId=null 2023-05-31T20:39:12.116 DEBUG:boto:Method: HEAD 2023-05-31T20:39:12.116 DEBUG:boto:Path: /pklxkd-58/obj00000000000000000024?versionId=null 2023-05-31T20:39:12.117 DEBUG:boto:Data: 2023-05-31T20:39:12.117 DEBUG:boto:Headers: {} 2023-05-31T20:39:12.117 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:39:12.118 DEBUG:boto:Port: 8001 2023-05-31T20:39:12.118 DEBUG:boto:Params: {} 2023-05-31T20:39:12.119 DEBUG:boto:Token: None 2023-05-31T20:39:12.119 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:39:12 GMT /pklxkd-58/obj00000000000000000024?versionId=null 2023-05-31T20:39:12.120 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:1Ll+2qw0OUuV5LL8YxkLwwzhXo4= 2023-05-31T20:39:12.121 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:39:12 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:1Ll+2qw0OUuV5LL8YxkLwwzhXo4=', 'Content-Length': '0'} 2023-05-31T20:39:12.122 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:38:22 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000064fd9abaf77f632-006477b070-1c17-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:39:12 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:39:12.123 DEBUG:boto:path=/pklxkd-58/obj00000000000000000024 2023-05-31T20:39:12.123 DEBUG:boto:auth_path=/pklxkd-58/obj00000000000000000024 2023-05-31T20:39:12.124 DEBUG:boto:path=/pklxkd-58/obj00000000000000000024?versionId=null 2023-05-31T20:39:12.124 DEBUG:boto:auth_path=/pklxkd-58/obj00000000000000000024?versionId=null 2023-05-31T20:39:12.125 DEBUG:boto:Method: HEAD 2023-05-31T20:39:12.125 DEBUG:boto:Path: /pklxkd-58/obj00000000000000000024?versionId=null 2023-05-31T20:39:12.125 DEBUG:boto:Data: 2023-05-31T20:39:12.126 DEBUG:boto:Headers: {} 2023-05-31T20:39:12.126 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:39:12.126 DEBUG:boto:Port: 8000 2023-05-31T20:39:12.126 DEBUG:boto:Params: {} 2023-05-31T20:39:12.127 DEBUG:boto:Token: None 2023-05-31T20:39:12.127 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:39:12 GMT /pklxkd-58/obj00000000000000000024?versionId=null 2023-05-31T20:39:12.128 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:1Ll+2qw0OUuV5LL8YxkLwwzhXo4= 2023-05-31T20:39:12.128 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:39:12 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:1Ll+2qw0OUuV5LL8YxkLwwzhXo4=', 'Content-Length': '0'} 2023-05-31T20:39:12.130 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:38:22 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ecb2af9e5b88a43e-006477b070-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:39:12 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:39:12.130 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000024 2023-05-31T20:39:12.131 DEBUG:boto:path=/pklxkd-58/obj00000000000000000024 2023-05-31T20:39:12.131 DEBUG:boto:auth_path=/pklxkd-58/obj00000000000000000024 2023-05-31T20:39:12.132 DEBUG:boto:path=/pklxkd-58/obj00000000000000000024?versionId=null 2023-05-31T20:39:12.132 DEBUG:boto:auth_path=/pklxkd-58/obj00000000000000000024?versionId=null 2023-05-31T20:39:12.132 DEBUG:boto:Method: GET 2023-05-31T20:39:12.133 DEBUG:boto:Path: /pklxkd-58/obj00000000000000000024?versionId=null 2023-05-31T20:39:12.133 DEBUG:boto:Data: 2023-05-31T20:39:12.133 DEBUG:boto:Headers: {} 2023-05-31T20:39:12.134 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:39:12.134 DEBUG:boto:Port: 8001 2023-05-31T20:39:12.134 DEBUG:boto:Params: {} 2023-05-31T20:39:12.135 DEBUG:boto:Token: None 2023-05-31T20:39:12.135 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:39:12 GMT /pklxkd-58/obj00000000000000000024?versionId=null 2023-05-31T20:39:12.136 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:KdWkqMzHz7mYiDWovBhi5lwEZ8s= 2023-05-31T20:39:12.136 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:39:12 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:KdWkqMzHz7mYiDWovBhi5lwEZ8s=', 'Content-Length': '0'} 2023-05-31T20:39:12.138 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:38:22 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000037ba7bbf093b3eab-006477b070-1c17-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:39:12 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:39:12.139 DEBUG:boto:path=/pklxkd-58/obj00000000000000000024 2023-05-31T20:39:12.139 DEBUG:boto:auth_path=/pklxkd-58/obj00000000000000000024 2023-05-31T20:39:12.140 DEBUG:boto:path=/pklxkd-58/obj00000000000000000024?versionId=null 2023-05-31T20:39:12.140 DEBUG:boto:auth_path=/pklxkd-58/obj00000000000000000024?versionId=null 2023-05-31T20:39:12.140 DEBUG:boto:Method: GET 2023-05-31T20:39:12.141 DEBUG:boto:Path: /pklxkd-58/obj00000000000000000024?versionId=null 2023-05-31T20:39:12.141 DEBUG:boto:Data: 2023-05-31T20:39:12.142 DEBUG:boto:Headers: {} 2023-05-31T20:39:12.142 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:39:12.142 DEBUG:boto:Port: 8000 2023-05-31T20:39:12.143 DEBUG:boto:Params: {} 2023-05-31T20:39:12.143 DEBUG:boto:Token: None 2023-05-31T20:39:12.143 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:39:12 GMT /pklxkd-58/obj00000000000000000024?versionId=null 2023-05-31T20:39:12.144 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:KdWkqMzHz7mYiDWovBhi5lwEZ8s= 2023-05-31T20:39:12.144 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:39:12 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:KdWkqMzHz7mYiDWovBhi5lwEZ8s=', 'Content-Length': '0'} 2023-05-31T20:39:12.146 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:38:22 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008312e6b68f88663e-006477b070-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:39:12 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:39:12.192 DEBUG:boto:path=/pklxkd-58/ 2023-05-31T20:39:12.193 DEBUG:boto:auth_path=/pklxkd-58/ 2023-05-31T20:39:12.193 DEBUG:boto:path=/pklxkd-58/?prefix=obj00000000000000000024 2023-05-31T20:39:12.194 DEBUG:boto:auth_path=/pklxkd-58/?prefix=obj00000000000000000024 2023-05-31T20:39:12.194 DEBUG:boto:Method: GET 2023-05-31T20:39:12.194 DEBUG:boto:Path: /pklxkd-58/?prefix=obj00000000000000000024 2023-05-31T20:39:12.195 DEBUG:boto:Data: 2023-05-31T20:39:12.195 DEBUG:boto:Headers: {} 2023-05-31T20:39:12.195 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:39:12.196 DEBUG:boto:Port: 8001 2023-05-31T20:39:12.196 DEBUG:boto:Params: {} 2023-05-31T20:39:12.197 DEBUG:boto:Token: None 2023-05-31T20:39:12.197 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:39:12 GMT /pklxkd-58/ 2023-05-31T20:39:12.197 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:ryNlxDRGgK16+XFs6oQP3KSCLXY= 2023-05-31T20:39:12.198 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:39:12 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:ryNlxDRGgK16+XFs6oQP3KSCLXY=', 'Content-Length': '0'} 2023-05-31T20:39:12.201 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000dd7ca59c4eca686a-006477b070-1c17-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:39:12 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:39:12.201 DEBUG:boto:b'pklxkd-58obj000000000000000000241000falseobj000000000000000000242023-05-31T20:38:22.813Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:39:12.202 DEBUG:boto:path=/pklxkd-58/ 2023-05-31T20:39:12.202 DEBUG:boto:auth_path=/pklxkd-58/ 2023-05-31T20:39:12.203 DEBUG:boto:path=/pklxkd-58/?prefix=obj00000000000000000024 2023-05-31T20:39:12.203 DEBUG:boto:auth_path=/pklxkd-58/?prefix=obj00000000000000000024 2023-05-31T20:39:12.203 DEBUG:boto:Method: GET 2023-05-31T20:39:12.204 DEBUG:boto:Path: /pklxkd-58/?prefix=obj00000000000000000024 2023-05-31T20:39:12.204 DEBUG:boto:Data: 2023-05-31T20:39:12.204 DEBUG:boto:Headers: {} 2023-05-31T20:39:12.205 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:39:12.205 DEBUG:boto:Port: 8000 2023-05-31T20:39:12.206 DEBUG:boto:Params: {} 2023-05-31T20:39:12.206 DEBUG:boto:Token: None 2023-05-31T20:39:12.206 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:39:12 GMT /pklxkd-58/ 2023-05-31T20:39:12.207 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:ryNlxDRGgK16+XFs6oQP3KSCLXY= 2023-05-31T20:39:12.207 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:39:12 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:ryNlxDRGgK16+XFs6oQP3KSCLXY=', 'Content-Length': '0'} 2023-05-31T20:39:12.214 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000001063b7204388e35b-006477b070-1630-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:39:12 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:39:12.214 DEBUG:boto:b'pklxkd-58obj000000000000000000241000falseobj000000000000000000242023-05-31T20:38:22.813Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:39:12.215 DEBUG:boto:path=/pklxkd-58/obj00000000000000000024 2023-05-31T20:39:12.216 DEBUG:boto:auth_path=/pklxkd-58/obj00000000000000000024 2023-05-31T20:39:12.216 DEBUG:boto:Method: HEAD 2023-05-31T20:39:12.217 DEBUG:boto:Path: /pklxkd-58/obj00000000000000000024 2023-05-31T20:39:12.217 DEBUG:boto:Data: 2023-05-31T20:39:12.217 DEBUG:boto:Headers: {} 2023-05-31T20:39:12.218 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:39:12.218 DEBUG:boto:Port: 8001 2023-05-31T20:39:12.218 DEBUG:boto:Params: {} 2023-05-31T20:39:12.219 DEBUG:boto:Token: None 2023-05-31T20:39:12.219 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:39:12 GMT /pklxkd-58/obj00000000000000000024 2023-05-31T20:39:12.220 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:jqBHrl81cCBFk+mF1jG7uHvqAEw= 2023-05-31T20:39:12.220 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:39:12 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:jqBHrl81cCBFk+mF1jG7uHvqAEw=', 'Content-Length': '0'} 2023-05-31T20:39:12.222 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:38:22 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003a9015242878896b-006477b070-1c17-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:39:12 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:39:12.223 DEBUG:boto:path=/pklxkd-58/obj00000000000000000024 2023-05-31T20:39:12.223 DEBUG:boto:auth_path=/pklxkd-58/obj00000000000000000024 2023-05-31T20:39:12.224 DEBUG:boto:Method: HEAD 2023-05-31T20:39:12.224 DEBUG:boto:Path: /pklxkd-58/obj00000000000000000024 2023-05-31T20:39:12.224 DEBUG:boto:Data: 2023-05-31T20:39:12.225 DEBUG:boto:Headers: {} 2023-05-31T20:39:12.225 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:39:12.225 DEBUG:boto:Port: 8000 2023-05-31T20:39:12.226 DEBUG:boto:Params: {} 2023-05-31T20:39:12.226 DEBUG:boto:Token: None 2023-05-31T20:39:12.226 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:39:12 GMT /pklxkd-58/obj00000000000000000024 2023-05-31T20:39:12.227 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:jqBHrl81cCBFk+mF1jG7uHvqAEw= 2023-05-31T20:39:12.227 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:39:12 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:jqBHrl81cCBFk+mF1jG7uHvqAEw=', 'Content-Length': '0'} 2023-05-31T20:39:12.229 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:38:22 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000050774ffcd06cef44-006477b070-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:39:12 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:39:12.230 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000024 2023-05-31T20:39:12.230 DEBUG:boto:path=/pklxkd-58/obj00000000000000000024 2023-05-31T20:39:12.231 DEBUG:boto:auth_path=/pklxkd-58/obj00000000000000000024 2023-05-31T20:39:12.231 DEBUG:boto:Method: GET 2023-05-31T20:39:12.231 DEBUG:boto:Path: /pklxkd-58/obj00000000000000000024 2023-05-31T20:39:12.232 DEBUG:boto:Data: 2023-05-31T20:39:12.232 DEBUG:boto:Headers: {} 2023-05-31T20:39:12.232 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:39:12.233 DEBUG:boto:Port: 8001 2023-05-31T20:39:12.233 DEBUG:boto:Params: {} 2023-05-31T20:39:12.234 DEBUG:boto:Token: None 2023-05-31T20:39:12.234 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:39:12 GMT /pklxkd-58/obj00000000000000000024 2023-05-31T20:39:12.234 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:L+xPXZbVo6A8l1HyWBeZue4RE90= 2023-05-31T20:39:12.235 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:39:12 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:L+xPXZbVo6A8l1HyWBeZue4RE90=', 'Content-Length': '0'} 2023-05-31T20:39:12.237 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:38:22 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ecddd37e7b5ca32a-006477b070-1c17-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:39:12 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:39:12.281 DEBUG:boto:path=/pklxkd-58/obj00000000000000000024 2023-05-31T20:39:12.282 DEBUG:boto:auth_path=/pklxkd-58/obj00000000000000000024 2023-05-31T20:39:12.282 DEBUG:boto:Method: GET 2023-05-31T20:39:12.283 DEBUG:boto:Path: /pklxkd-58/obj00000000000000000024 2023-05-31T20:39:12.283 DEBUG:boto:Data: 2023-05-31T20:39:12.284 DEBUG:boto:Headers: {} 2023-05-31T20:39:12.284 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:39:12.284 DEBUG:boto:Port: 8000 2023-05-31T20:39:12.285 DEBUG:boto:Params: {} 2023-05-31T20:39:12.285 DEBUG:boto:Token: None 2023-05-31T20:39:12.286 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:39:12 GMT /pklxkd-58/obj00000000000000000024 2023-05-31T20:39:12.286 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:L+xPXZbVo6A8l1HyWBeZue4RE90= 2023-05-31T20:39:12.286 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:39:12 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:L+xPXZbVo6A8l1HyWBeZue4RE90=', 'Content-Length': '0'} 2023-05-31T20:39:12.289 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:38:22 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000022173621bb697072-006477b070-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:39:12 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:39:12.290 DEBUG:boto:path=/pklxkd-58/ 2023-05-31T20:39:12.290 DEBUG:boto:auth_path=/pklxkd-58/ 2023-05-31T20:39:12.291 DEBUG:boto:path=/pklxkd-58/?prefix=obj00000000000000000024 2023-05-31T20:39:12.291 DEBUG:boto:auth_path=/pklxkd-58/?prefix=obj00000000000000000024 2023-05-31T20:39:12.291 DEBUG:boto:Method: GET 2023-05-31T20:39:12.292 DEBUG:boto:Path: /pklxkd-58/?prefix=obj00000000000000000024 2023-05-31T20:39:12.292 DEBUG:boto:Data: 2023-05-31T20:39:12.292 DEBUG:boto:Headers: {} 2023-05-31T20:39:12.293 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:39:12.293 DEBUG:boto:Port: 8001 2023-05-31T20:39:12.293 DEBUG:boto:Params: {} 2023-05-31T20:39:12.294 DEBUG:boto:Token: None 2023-05-31T20:39:12.294 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:39:12 GMT /pklxkd-58/ 2023-05-31T20:39:12.295 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:ryNlxDRGgK16+XFs6oQP3KSCLXY= 2023-05-31T20:39:12.295 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:39:12 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:ryNlxDRGgK16+XFs6oQP3KSCLXY=', 'Content-Length': '0'} 2023-05-31T20:39:12.298 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000004889db1f82fbc69b-006477b070-1c17-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:39:12 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:39:12.299 DEBUG:boto:b'pklxkd-58obj000000000000000000241000falseobj000000000000000000242023-05-31T20:38:22.813Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:39:12.301 DEBUG:boto:path=/pklxkd-58/ 2023-05-31T20:39:12.301 DEBUG:boto:auth_path=/pklxkd-58/ 2023-05-31T20:39:12.301 DEBUG:boto:path=/pklxkd-58/?prefix=obj00000000000000000024 2023-05-31T20:39:12.302 DEBUG:boto:auth_path=/pklxkd-58/?prefix=obj00000000000000000024 2023-05-31T20:39:12.302 DEBUG:boto:Method: GET 2023-05-31T20:39:12.303 DEBUG:boto:Path: /pklxkd-58/?prefix=obj00000000000000000024 2023-05-31T20:39:12.303 DEBUG:boto:Data: 2023-05-31T20:39:12.303 DEBUG:boto:Headers: {} 2023-05-31T20:39:12.304 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:39:12.304 DEBUG:boto:Port: 8000 2023-05-31T20:39:12.305 DEBUG:boto:Params: {} 2023-05-31T20:39:12.305 DEBUG:boto:Token: None 2023-05-31T20:39:12.305 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:39:12 GMT /pklxkd-58/ 2023-05-31T20:39:12.306 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:ryNlxDRGgK16+XFs6oQP3KSCLXY= 2023-05-31T20:39:12.306 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:39:12 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:ryNlxDRGgK16+XFs6oQP3KSCLXY=', 'Content-Length': '0'} 2023-05-31T20:39:12.310 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ee098bd331532a4a-006477b070-1630-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:39:12 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:39:12.356 DEBUG:boto:b'pklxkd-58obj000000000000000000241000falseobj000000000000000000242023-05-31T20:38:22.813Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:39:12.357 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000025 2023-05-31T20:39:12.358 DEBUG:boto:path=/pklxkd-58/obj00000000000000000025 2023-05-31T20:39:12.359 DEBUG:boto:auth_path=/pklxkd-58/obj00000000000000000025 2023-05-31T20:39:12.359 DEBUG:boto:path=/pklxkd-58/obj00000000000000000025?versionId=null 2023-05-31T20:39:12.359 DEBUG:boto:auth_path=/pklxkd-58/obj00000000000000000025?versionId=null 2023-05-31T20:39:12.360 DEBUG:boto:Method: GET 2023-05-31T20:39:12.360 DEBUG:boto:Path: /pklxkd-58/obj00000000000000000025?versionId=null 2023-05-31T20:39:12.360 DEBUG:boto:Data: 2023-05-31T20:39:12.362 DEBUG:boto:Headers: {} 2023-05-31T20:39:12.362 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:39:12.362 DEBUG:boto:Port: 8001 2023-05-31T20:39:12.363 DEBUG:boto:Params: {} 2023-05-31T20:39:12.363 DEBUG:boto:Token: None 2023-05-31T20:39:12.364 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:39:12 GMT /pklxkd-58/obj00000000000000000025?versionId=null 2023-05-31T20:39:12.364 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:y4aK6NP/oWvHAduIG4VWJTx4LVw= 2023-05-31T20:39:12.365 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:39:12 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:y4aK6NP/oWvHAduIG4VWJTx4LVw=', 'Content-Length': '0'} 2023-05-31T20:39:12.367 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:38:19 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000deef393ddb74bb15-006477b070-1c17-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:39:12 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:39:12.368 DEBUG:boto:path=/pklxkd-58/obj00000000000000000025 2023-05-31T20:39:12.372 DEBUG:boto:auth_path=/pklxkd-58/obj00000000000000000025 2023-05-31T20:39:12.372 DEBUG:boto:path=/pklxkd-58/obj00000000000000000025?versionId=null 2023-05-31T20:39:12.373 DEBUG:boto:auth_path=/pklxkd-58/obj00000000000000000025?versionId=null 2023-05-31T20:39:12.373 DEBUG:boto:Method: GET 2023-05-31T20:39:12.373 DEBUG:boto:Path: /pklxkd-58/obj00000000000000000025?versionId=null 2023-05-31T20:39:12.374 DEBUG:boto:Data: 2023-05-31T20:39:12.374 DEBUG:boto:Headers: {} 2023-05-31T20:39:12.375 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:39:12.375 DEBUG:boto:Port: 8000 2023-05-31T20:39:12.375 DEBUG:boto:Params: {} 2023-05-31T20:39:12.376 DEBUG:boto:Token: None 2023-05-31T20:39:12.376 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:39:12 GMT /pklxkd-58/obj00000000000000000025?versionId=null 2023-05-31T20:39:12.377 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:y4aK6NP/oWvHAduIG4VWJTx4LVw= 2023-05-31T20:39:12.377 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:39:12 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:y4aK6NP/oWvHAduIG4VWJTx4LVw=', 'Content-Length': '0'} 2023-05-31T20:39:12.379 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:38:19 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000fb85fe5d45e933a9-006477b070-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:39:12 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:39:12.380 DEBUG:boto:path=/pklxkd-58/obj00000000000000000025 2023-05-31T20:39:12.380 DEBUG:boto:auth_path=/pklxkd-58/obj00000000000000000025 2023-05-31T20:39:12.381 DEBUG:boto:path=/pklxkd-58/obj00000000000000000025?versionId=null 2023-05-31T20:39:12.381 DEBUG:boto:auth_path=/pklxkd-58/obj00000000000000000025?versionId=null 2023-05-31T20:39:12.381 DEBUG:boto:Method: HEAD 2023-05-31T20:39:12.382 DEBUG:boto:Path: /pklxkd-58/obj00000000000000000025?versionId=null 2023-05-31T20:39:12.382 DEBUG:boto:Data: 2023-05-31T20:39:12.382 DEBUG:boto:Headers: {} 2023-05-31T20:39:12.383 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:39:12.383 DEBUG:boto:Port: 8001 2023-05-31T20:39:12.383 DEBUG:boto:Params: {} 2023-05-31T20:39:12.384 DEBUG:boto:Token: None 2023-05-31T20:39:12.384 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:39:12 GMT /pklxkd-58/obj00000000000000000025?versionId=null 2023-05-31T20:39:12.385 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:DbdGGhyR2G8hg9yNAfFjjM3M1RU= 2023-05-31T20:39:12.385 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:39:12 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:DbdGGhyR2G8hg9yNAfFjjM3M1RU=', 'Content-Length': '0'} 2023-05-31T20:39:12.387 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:38:19 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ccf3fe0d198b7979-006477b070-1c17-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:39:12 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:39:12.388 DEBUG:boto:path=/pklxkd-58/obj00000000000000000025 2023-05-31T20:39:12.389 DEBUG:boto:auth_path=/pklxkd-58/obj00000000000000000025 2023-05-31T20:39:12.389 DEBUG:boto:path=/pklxkd-58/obj00000000000000000025?versionId=null 2023-05-31T20:39:12.390 DEBUG:boto:auth_path=/pklxkd-58/obj00000000000000000025?versionId=null 2023-05-31T20:39:12.390 DEBUG:boto:Method: HEAD 2023-05-31T20:39:12.391 DEBUG:boto:Path: /pklxkd-58/obj00000000000000000025?versionId=null 2023-05-31T20:39:12.391 DEBUG:boto:Data: 2023-05-31T20:39:12.391 DEBUG:boto:Headers: {} 2023-05-31T20:39:12.392 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:39:12.392 DEBUG:boto:Port: 8000 2023-05-31T20:39:12.393 DEBUG:boto:Params: {} 2023-05-31T20:39:12.393 DEBUG:boto:Token: None 2023-05-31T20:39:12.394 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:39:12 GMT /pklxkd-58/obj00000000000000000025?versionId=null 2023-05-31T20:39:12.394 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:DbdGGhyR2G8hg9yNAfFjjM3M1RU= 2023-05-31T20:39:12.395 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:39:12 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:DbdGGhyR2G8hg9yNAfFjjM3M1RU=', 'Content-Length': '0'} 2023-05-31T20:39:12.397 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:38:19 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f16d3796ad07dec7-006477b070-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:39:12 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:39:12.397 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000025 2023-05-31T20:39:12.398 DEBUG:boto:path=/pklxkd-58/obj00000000000000000025 2023-05-31T20:39:12.398 DEBUG:boto:auth_path=/pklxkd-58/obj00000000000000000025 2023-05-31T20:39:12.399 DEBUG:boto:path=/pklxkd-58/obj00000000000000000025?versionId=null 2023-05-31T20:39:12.399 DEBUG:boto:auth_path=/pklxkd-58/obj00000000000000000025?versionId=null 2023-05-31T20:39:12.399 DEBUG:boto:Method: GET 2023-05-31T20:39:12.400 DEBUG:boto:Path: /pklxkd-58/obj00000000000000000025?versionId=null 2023-05-31T20:39:12.400 DEBUG:boto:Data: 2023-05-31T20:39:12.401 DEBUG:boto:Headers: {} 2023-05-31T20:39:12.401 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:39:12.402 DEBUG:boto:Port: 8001 2023-05-31T20:39:12.402 DEBUG:boto:Params: {} 2023-05-31T20:39:12.403 DEBUG:boto:Token: None 2023-05-31T20:39:12.403 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:39:12 GMT /pklxkd-58/obj00000000000000000025?versionId=null 2023-05-31T20:39:12.404 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:y4aK6NP/oWvHAduIG4VWJTx4LVw= 2023-05-31T20:39:12.404 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:39:12 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:y4aK6NP/oWvHAduIG4VWJTx4LVw=', 'Content-Length': '0'} 2023-05-31T20:39:12.406 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:38:19 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007c30359b115fd914-006477b070-1c17-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:39:12 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:39:12.407 DEBUG:boto:path=/pklxkd-58/obj00000000000000000025 2023-05-31T20:39:12.407 DEBUG:boto:auth_path=/pklxkd-58/obj00000000000000000025 2023-05-31T20:39:12.408 DEBUG:boto:path=/pklxkd-58/obj00000000000000000025?versionId=null 2023-05-31T20:39:12.408 DEBUG:boto:auth_path=/pklxkd-58/obj00000000000000000025?versionId=null 2023-05-31T20:39:12.409 DEBUG:boto:Method: GET 2023-05-31T20:39:12.409 DEBUG:boto:Path: /pklxkd-58/obj00000000000000000025?versionId=null 2023-05-31T20:39:12.409 DEBUG:boto:Data: 2023-05-31T20:39:12.412 DEBUG:boto:Headers: {} 2023-05-31T20:39:12.413 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:39:12.413 DEBUG:boto:Port: 8000 2023-05-31T20:39:12.413 DEBUG:boto:Params: {} 2023-05-31T20:39:12.414 DEBUG:boto:Token: None 2023-05-31T20:39:12.414 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:39:12 GMT /pklxkd-58/obj00000000000000000025?versionId=null 2023-05-31T20:39:12.415 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:y4aK6NP/oWvHAduIG4VWJTx4LVw= 2023-05-31T20:39:12.415 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:39:12 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:y4aK6NP/oWvHAduIG4VWJTx4LVw=', 'Content-Length': '0'} 2023-05-31T20:39:12.417 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:38:19 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c0c1e2a6ef0a0dd5-006477b070-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:39:12 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:39:12.418 DEBUG:boto:path=/pklxkd-58/ 2023-05-31T20:39:12.418 DEBUG:boto:auth_path=/pklxkd-58/ 2023-05-31T20:39:12.419 DEBUG:boto:path=/pklxkd-58/?prefix=obj00000000000000000025 2023-05-31T20:39:12.419 DEBUG:boto:auth_path=/pklxkd-58/?prefix=obj00000000000000000025 2023-05-31T20:39:12.419 DEBUG:boto:Method: GET 2023-05-31T20:39:12.420 DEBUG:boto:Path: /pklxkd-58/?prefix=obj00000000000000000025 2023-05-31T20:39:12.420 DEBUG:boto:Data: 2023-05-31T20:39:12.420 DEBUG:boto:Headers: {} 2023-05-31T20:39:12.421 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:39:12.421 DEBUG:boto:Port: 8001 2023-05-31T20:39:12.422 DEBUG:boto:Params: {} 2023-05-31T20:39:12.422 DEBUG:boto:Token: None 2023-05-31T20:39:12.423 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:39:12 GMT /pklxkd-58/ 2023-05-31T20:39:12.423 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:ryNlxDRGgK16+XFs6oQP3KSCLXY= 2023-05-31T20:39:12.424 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:39:12 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:ryNlxDRGgK16+XFs6oQP3KSCLXY=', 'Content-Length': '0'} 2023-05-31T20:39:12.426 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000047283264a3694e22-006477b070-1c17-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:39:12 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:39:12.427 DEBUG:boto:b'pklxkd-58obj000000000000000000251000falseobj000000000000000000252023-05-31T20:38:19.001Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:39:12.428 DEBUG:boto:path=/pklxkd-58/ 2023-05-31T20:39:12.428 DEBUG:boto:auth_path=/pklxkd-58/ 2023-05-31T20:39:12.428 DEBUG:boto:path=/pklxkd-58/?prefix=obj00000000000000000025 2023-05-31T20:39:12.429 DEBUG:boto:auth_path=/pklxkd-58/?prefix=obj00000000000000000025 2023-05-31T20:39:12.429 DEBUG:boto:Method: GET 2023-05-31T20:39:12.429 DEBUG:boto:Path: /pklxkd-58/?prefix=obj00000000000000000025 2023-05-31T20:39:12.430 DEBUG:boto:Data: 2023-05-31T20:39:12.430 DEBUG:boto:Headers: {} 2023-05-31T20:39:12.431 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:39:12.431 DEBUG:boto:Port: 8000 2023-05-31T20:39:12.431 DEBUG:boto:Params: {} 2023-05-31T20:39:12.432 DEBUG:boto:Token: None 2023-05-31T20:39:12.432 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:39:12 GMT /pklxkd-58/ 2023-05-31T20:39:12.434 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:ryNlxDRGgK16+XFs6oQP3KSCLXY= 2023-05-31T20:39:12.435 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:39:12 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:ryNlxDRGgK16+XFs6oQP3KSCLXY=', 'Content-Length': '0'} 2023-05-31T20:39:12.439 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000040d24bc8d992ca8a-006477b070-1630-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:39:12 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:39:12.480 DEBUG:boto:b'pklxkd-58obj000000000000000000251000falseobj000000000000000000252023-05-31T20:38:19.001Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:39:12.481 DEBUG:boto:path=/pklxkd-58/obj00000000000000000025 2023-05-31T20:39:12.482 DEBUG:boto:auth_path=/pklxkd-58/obj00000000000000000025 2023-05-31T20:39:12.482 DEBUG:boto:Method: HEAD 2023-05-31T20:39:12.482 DEBUG:boto:Path: /pklxkd-58/obj00000000000000000025 2023-05-31T20:39:12.483 DEBUG:boto:Data: 2023-05-31T20:39:12.483 DEBUG:boto:Headers: {} 2023-05-31T20:39:12.483 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:39:12.484 DEBUG:boto:Port: 8001 2023-05-31T20:39:12.484 DEBUG:boto:Params: {} 2023-05-31T20:39:12.485 DEBUG:boto:Token: None 2023-05-31T20:39:12.485 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:39:12 GMT /pklxkd-58/obj00000000000000000025 2023-05-31T20:39:12.485 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:tE/q/ImsHHu+4AFMKDiJKHglSP4= 2023-05-31T20:39:12.486 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:39:12 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:tE/q/ImsHHu+4AFMKDiJKHglSP4=', 'Content-Length': '0'} 2023-05-31T20:39:12.489 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:38:19 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000bac6268162ff7af-006477b070-1c17-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:39:12 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:39:12.490 DEBUG:boto:path=/pklxkd-58/obj00000000000000000025 2023-05-31T20:39:12.491 DEBUG:boto:auth_path=/pklxkd-58/obj00000000000000000025 2023-05-31T20:39:12.492 DEBUG:boto:Method: HEAD 2023-05-31T20:39:12.493 DEBUG:boto:Path: /pklxkd-58/obj00000000000000000025 2023-05-31T20:39:12.493 DEBUG:boto:Data: 2023-05-31T20:39:12.494 DEBUG:boto:Headers: {} 2023-05-31T20:39:12.494 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:39:12.495 DEBUG:boto:Port: 8000 2023-05-31T20:39:12.496 DEBUG:boto:Params: {} 2023-05-31T20:39:12.497 DEBUG:boto:Token: None 2023-05-31T20:39:12.497 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:39:12 GMT /pklxkd-58/obj00000000000000000025 2023-05-31T20:39:12.498 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:tE/q/ImsHHu+4AFMKDiJKHglSP4= 2023-05-31T20:39:12.498 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:39:12 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:tE/q/ImsHHu+4AFMKDiJKHglSP4=', 'Content-Length': '0'} 2023-05-31T20:39:12.500 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:38:19 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000559944962c67f3c-006477b070-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:39:12 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:39:12.501 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000025 2023-05-31T20:39:12.501 DEBUG:boto:path=/pklxkd-58/obj00000000000000000025 2023-05-31T20:39:12.502 DEBUG:boto:auth_path=/pklxkd-58/obj00000000000000000025 2023-05-31T20:39:12.502 DEBUG:boto:Method: GET 2023-05-31T20:39:12.503 DEBUG:boto:Path: /pklxkd-58/obj00000000000000000025 2023-05-31T20:39:12.503 DEBUG:boto:Data: 2023-05-31T20:39:12.503 DEBUG:boto:Headers: {} 2023-05-31T20:39:12.504 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:39:12.504 DEBUG:boto:Port: 8001 2023-05-31T20:39:12.504 DEBUG:boto:Params: {} 2023-05-31T20:39:12.505 DEBUG:boto:Token: None 2023-05-31T20:39:12.505 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:39:12 GMT /pklxkd-58/obj00000000000000000025 2023-05-31T20:39:12.506 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:BB/5YCelCEYFVJy/Fp1rtfQkJb0= 2023-05-31T20:39:12.506 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:39:12 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:BB/5YCelCEYFVJy/Fp1rtfQkJb0=', 'Content-Length': '0'} 2023-05-31T20:39:12.508 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:38:19 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000da1fd2ab5604184d-006477b070-1c17-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:39:12 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:39:12.509 DEBUG:boto:path=/pklxkd-58/obj00000000000000000025 2023-05-31T20:39:12.510 DEBUG:boto:auth_path=/pklxkd-58/obj00000000000000000025 2023-05-31T20:39:12.510 DEBUG:boto:Method: GET 2023-05-31T20:39:12.510 DEBUG:boto:Path: /pklxkd-58/obj00000000000000000025 2023-05-31T20:39:12.511 DEBUG:boto:Data: 2023-05-31T20:39:12.512 DEBUG:boto:Headers: {} 2023-05-31T20:39:12.512 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:39:12.513 DEBUG:boto:Port: 8000 2023-05-31T20:39:12.513 DEBUG:boto:Params: {} 2023-05-31T20:39:12.513 DEBUG:boto:Token: None 2023-05-31T20:39:12.514 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:39:12 GMT /pklxkd-58/obj00000000000000000025 2023-05-31T20:39:12.514 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:BB/5YCelCEYFVJy/Fp1rtfQkJb0= 2023-05-31T20:39:12.515 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:39:12 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:BB/5YCelCEYFVJy/Fp1rtfQkJb0=', 'Content-Length': '0'} 2023-05-31T20:39:12.517 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:38:19 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f4b9f41765ac6274-006477b070-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:39:12 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:39:12.517 DEBUG:boto:path=/pklxkd-58/ 2023-05-31T20:39:12.518 DEBUG:boto:auth_path=/pklxkd-58/ 2023-05-31T20:39:12.518 DEBUG:boto:path=/pklxkd-58/?prefix=obj00000000000000000025 2023-05-31T20:39:12.518 DEBUG:boto:auth_path=/pklxkd-58/?prefix=obj00000000000000000025 2023-05-31T20:39:12.519 DEBUG:boto:Method: GET 2023-05-31T20:39:12.519 DEBUG:boto:Path: /pklxkd-58/?prefix=obj00000000000000000025 2023-05-31T20:39:12.520 DEBUG:boto:Data: 2023-05-31T20:39:12.520 DEBUG:boto:Headers: {} 2023-05-31T20:39:12.520 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:39:12.521 DEBUG:boto:Port: 8001 2023-05-31T20:39:12.521 DEBUG:boto:Params: {} 2023-05-31T20:39:12.522 DEBUG:boto:Token: None 2023-05-31T20:39:12.522 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:39:12 GMT /pklxkd-58/ 2023-05-31T20:39:12.523 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:ryNlxDRGgK16+XFs6oQP3KSCLXY= 2023-05-31T20:39:12.523 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:39:12 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:ryNlxDRGgK16+XFs6oQP3KSCLXY=', 'Content-Length': '0'} 2023-05-31T20:39:12.526 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c322a65623256f81-006477b070-1c17-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:39:12 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:39:12.568 DEBUG:boto:b'pklxkd-58obj000000000000000000251000falseobj000000000000000000252023-05-31T20:38:19.001Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:39:12.570 DEBUG:boto:path=/pklxkd-58/ 2023-05-31T20:39:12.570 DEBUG:boto:auth_path=/pklxkd-58/ 2023-05-31T20:39:12.570 DEBUG:boto:path=/pklxkd-58/?prefix=obj00000000000000000025 2023-05-31T20:39:12.571 DEBUG:boto:auth_path=/pklxkd-58/?prefix=obj00000000000000000025 2023-05-31T20:39:12.571 DEBUG:boto:Method: GET 2023-05-31T20:39:12.572 DEBUG:boto:Path: /pklxkd-58/?prefix=obj00000000000000000025 2023-05-31T20:39:12.572 DEBUG:boto:Data: 2023-05-31T20:39:12.573 DEBUG:boto:Headers: {} 2023-05-31T20:39:12.573 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:39:12.573 DEBUG:boto:Port: 8000 2023-05-31T20:39:12.574 DEBUG:boto:Params: {} 2023-05-31T20:39:12.574 DEBUG:boto:Token: None 2023-05-31T20:39:12.575 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:39:12 GMT /pklxkd-58/ 2023-05-31T20:39:12.575 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:ryNlxDRGgK16+XFs6oQP3KSCLXY= 2023-05-31T20:39:12.575 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:39:12 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:ryNlxDRGgK16+XFs6oQP3KSCLXY=', 'Content-Length': '0'} 2023-05-31T20:39:12.580 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000cfb24949450d3147-006477b070-1630-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:39:12 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:39:12.580 DEBUG:boto:b'pklxkd-58obj000000000000000000251000falseobj000000000000000000252023-05-31T20:38:19.001Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:39:12.581 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000026 2023-05-31T20:39:12.581 DEBUG:boto:path=/pklxkd-58/obj00000000000000000026 2023-05-31T20:39:12.582 DEBUG:boto:auth_path=/pklxkd-58/obj00000000000000000026 2023-05-31T20:39:12.582 DEBUG:boto:path=/pklxkd-58/obj00000000000000000026?versionId=null 2023-05-31T20:39:12.582 DEBUG:boto:auth_path=/pklxkd-58/obj00000000000000000026?versionId=null 2023-05-31T20:39:12.583 DEBUG:boto:Method: GET 2023-05-31T20:39:12.583 DEBUG:boto:Path: /pklxkd-58/obj00000000000000000026?versionId=null 2023-05-31T20:39:12.587 DEBUG:boto:Data: 2023-05-31T20:39:12.588 DEBUG:boto:Headers: {} 2023-05-31T20:39:12.588 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:39:12.589 DEBUG:boto:Port: 8001 2023-05-31T20:39:12.589 DEBUG:boto:Params: {} 2023-05-31T20:39:12.590 DEBUG:boto:Token: None 2023-05-31T20:39:12.590 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:39:12 GMT /pklxkd-58/obj00000000000000000026?versionId=null 2023-05-31T20:39:12.591 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:6Rm5f3XMdIsCl/bzx4Q6tA3d6Kk= 2023-05-31T20:39:12.591 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:39:12 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:6Rm5f3XMdIsCl/bzx4Q6tA3d6Kk=', 'Content-Length': '0'} 2023-05-31T20:39:12.594 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:38:22 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ae285e3c7fe6f2d1-006477b070-1c17-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:39:12 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:39:12.595 DEBUG:boto:path=/pklxkd-58/obj00000000000000000026 2023-05-31T20:39:12.595 DEBUG:boto:auth_path=/pklxkd-58/obj00000000000000000026 2023-05-31T20:39:12.596 DEBUG:boto:path=/pklxkd-58/obj00000000000000000026?versionId=null 2023-05-31T20:39:12.596 DEBUG:boto:auth_path=/pklxkd-58/obj00000000000000000026?versionId=null 2023-05-31T20:39:12.596 DEBUG:boto:Method: GET 2023-05-31T20:39:12.597 DEBUG:boto:Path: /pklxkd-58/obj00000000000000000026?versionId=null 2023-05-31T20:39:12.598 DEBUG:boto:Data: 2023-05-31T20:39:12.598 DEBUG:boto:Headers: {} 2023-05-31T20:39:12.598 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:39:12.599 DEBUG:boto:Port: 8000 2023-05-31T20:39:12.599 DEBUG:boto:Params: {} 2023-05-31T20:39:12.599 DEBUG:boto:Token: None 2023-05-31T20:39:12.600 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:39:12 GMT /pklxkd-58/obj00000000000000000026?versionId=null 2023-05-31T20:39:12.600 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:6Rm5f3XMdIsCl/bzx4Q6tA3d6Kk= 2023-05-31T20:39:12.601 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:39:12 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:6Rm5f3XMdIsCl/bzx4Q6tA3d6Kk=', 'Content-Length': '0'} 2023-05-31T20:39:12.603 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:38:22 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006a8cfc8071f8742e-006477b070-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:39:12 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:39:12.603 DEBUG:boto:path=/pklxkd-58/obj00000000000000000026 2023-05-31T20:39:12.604 DEBUG:boto:auth_path=/pklxkd-58/obj00000000000000000026 2023-05-31T20:39:12.604 DEBUG:boto:path=/pklxkd-58/obj00000000000000000026?versionId=null 2023-05-31T20:39:12.605 DEBUG:boto:auth_path=/pklxkd-58/obj00000000000000000026?versionId=null 2023-05-31T20:39:12.605 DEBUG:boto:Method: HEAD 2023-05-31T20:39:12.605 DEBUG:boto:Path: /pklxkd-58/obj00000000000000000026?versionId=null 2023-05-31T20:39:12.606 DEBUG:boto:Data: 2023-05-31T20:39:12.606 DEBUG:boto:Headers: {} 2023-05-31T20:39:12.606 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:39:12.607 DEBUG:boto:Port: 8001 2023-05-31T20:39:12.607 DEBUG:boto:Params: {} 2023-05-31T20:39:12.608 DEBUG:boto:Token: None 2023-05-31T20:39:12.608 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:39:12 GMT /pklxkd-58/obj00000000000000000026?versionId=null 2023-05-31T20:39:12.609 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:9o89g00n2AFsLH5OkQ1qbNi2/4A= 2023-05-31T20:39:12.609 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:39:12 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:9o89g00n2AFsLH5OkQ1qbNi2/4A=', 'Content-Length': '0'} 2023-05-31T20:39:12.611 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:38:22 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000be9d68bda61c51a2-006477b070-1c17-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:39:12 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:39:12.612 DEBUG:boto:path=/pklxkd-58/obj00000000000000000026 2023-05-31T20:39:12.612 DEBUG:boto:auth_path=/pklxkd-58/obj00000000000000000026 2023-05-31T20:39:12.612 DEBUG:boto:path=/pklxkd-58/obj00000000000000000026?versionId=null 2023-05-31T20:39:12.613 DEBUG:boto:auth_path=/pklxkd-58/obj00000000000000000026?versionId=null 2023-05-31T20:39:12.613 DEBUG:boto:Method: HEAD 2023-05-31T20:39:12.614 DEBUG:boto:Path: /pklxkd-58/obj00000000000000000026?versionId=null 2023-05-31T20:39:12.614 DEBUG:boto:Data: 2023-05-31T20:39:12.614 DEBUG:boto:Headers: {} 2023-05-31T20:39:12.615 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:39:12.615 DEBUG:boto:Port: 8000 2023-05-31T20:39:12.615 DEBUG:boto:Params: {} 2023-05-31T20:39:12.616 DEBUG:boto:Token: None 2023-05-31T20:39:12.616 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:39:12 GMT /pklxkd-58/obj00000000000000000026?versionId=null 2023-05-31T20:39:12.616 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:9o89g00n2AFsLH5OkQ1qbNi2/4A= 2023-05-31T20:39:12.617 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:39:12 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:9o89g00n2AFsLH5OkQ1qbNi2/4A=', 'Content-Length': '0'} 2023-05-31T20:39:12.619 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:38:22 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004d1d07db256aabff-006477b070-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:39:12 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:39:12.620 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000026 2023-05-31T20:39:12.620 DEBUG:boto:path=/pklxkd-58/obj00000000000000000026 2023-05-31T20:39:12.621 DEBUG:boto:auth_path=/pklxkd-58/obj00000000000000000026 2023-05-31T20:39:12.621 DEBUG:boto:path=/pklxkd-58/obj00000000000000000026?versionId=null 2023-05-31T20:39:12.621 DEBUG:boto:auth_path=/pklxkd-58/obj00000000000000000026?versionId=null 2023-05-31T20:39:12.622 DEBUG:boto:Method: GET 2023-05-31T20:39:12.622 DEBUG:boto:Path: /pklxkd-58/obj00000000000000000026?versionId=null 2023-05-31T20:39:12.623 DEBUG:boto:Data: 2023-05-31T20:39:12.623 DEBUG:boto:Headers: {} 2023-05-31T20:39:12.623 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:39:12.624 DEBUG:boto:Port: 8001 2023-05-31T20:39:12.624 DEBUG:boto:Params: {} 2023-05-31T20:39:12.624 DEBUG:boto:Token: None 2023-05-31T20:39:12.625 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:39:12 GMT /pklxkd-58/obj00000000000000000026?versionId=null 2023-05-31T20:39:12.625 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:6Rm5f3XMdIsCl/bzx4Q6tA3d6Kk= 2023-05-31T20:39:12.626 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:39:12 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:6Rm5f3XMdIsCl/bzx4Q6tA3d6Kk=', 'Content-Length': '0'} 2023-05-31T20:39:12.628 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:38:22 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a6fee15ab3012c24-006477b070-1c17-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:39:12 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:39:12.628 DEBUG:boto:path=/pklxkd-58/obj00000000000000000026 2023-05-31T20:39:12.629 DEBUG:boto:auth_path=/pklxkd-58/obj00000000000000000026 2023-05-31T20:39:12.629 DEBUG:boto:path=/pklxkd-58/obj00000000000000000026?versionId=null 2023-05-31T20:39:12.630 DEBUG:boto:auth_path=/pklxkd-58/obj00000000000000000026?versionId=null 2023-05-31T20:39:12.630 DEBUG:boto:Method: GET 2023-05-31T20:39:12.630 DEBUG:boto:Path: /pklxkd-58/obj00000000000000000026?versionId=null 2023-05-31T20:39:12.631 DEBUG:boto:Data: 2023-05-31T20:39:12.631 DEBUG:boto:Headers: {} 2023-05-31T20:39:12.631 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:39:12.632 DEBUG:boto:Port: 8000 2023-05-31T20:39:12.632 DEBUG:boto:Params: {} 2023-05-31T20:39:12.632 DEBUG:boto:Token: None 2023-05-31T20:39:12.633 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:39:12 GMT /pklxkd-58/obj00000000000000000026?versionId=null 2023-05-31T20:39:12.633 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:6Rm5f3XMdIsCl/bzx4Q6tA3d6Kk= 2023-05-31T20:39:12.634 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:39:12 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:6Rm5f3XMdIsCl/bzx4Q6tA3d6Kk=', 'Content-Length': '0'} 2023-05-31T20:39:12.636 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:38:22 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e6e1a267b357a760-006477b070-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:39:12 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:39:12.636 DEBUG:boto:path=/pklxkd-58/ 2023-05-31T20:39:12.637 DEBUG:boto:auth_path=/pklxkd-58/ 2023-05-31T20:39:12.637 DEBUG:boto:path=/pklxkd-58/?prefix=obj00000000000000000026 2023-05-31T20:39:12.638 DEBUG:boto:auth_path=/pklxkd-58/?prefix=obj00000000000000000026 2023-05-31T20:39:12.638 DEBUG:boto:Method: GET 2023-05-31T20:39:12.639 DEBUG:boto:Path: /pklxkd-58/?prefix=obj00000000000000000026 2023-05-31T20:39:12.639 DEBUG:boto:Data: 2023-05-31T20:39:12.639 DEBUG:boto:Headers: {} 2023-05-31T20:39:12.640 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:39:12.640 DEBUG:boto:Port: 8001 2023-05-31T20:39:12.641 DEBUG:boto:Params: {} 2023-05-31T20:39:12.641 DEBUG:boto:Token: None 2023-05-31T20:39:12.642 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:39:12 GMT /pklxkd-58/ 2023-05-31T20:39:12.642 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:ryNlxDRGgK16+XFs6oQP3KSCLXY= 2023-05-31T20:39:12.643 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:39:12 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:ryNlxDRGgK16+XFs6oQP3KSCLXY=', 'Content-Length': '0'} 2023-05-31T20:39:12.645 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000007d1933568e9024b6-006477b070-1c17-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:39:12 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:39:12.646 DEBUG:boto:b'pklxkd-58obj000000000000000000261000falseobj000000000000000000262023-05-31T20:38:22.407Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:39:12.646 DEBUG:boto:path=/pklxkd-58/ 2023-05-31T20:39:12.647 DEBUG:boto:auth_path=/pklxkd-58/ 2023-05-31T20:39:12.647 DEBUG:boto:path=/pklxkd-58/?prefix=obj00000000000000000026 2023-05-31T20:39:12.648 DEBUG:boto:auth_path=/pklxkd-58/?prefix=obj00000000000000000026 2023-05-31T20:39:12.648 DEBUG:boto:Method: GET 2023-05-31T20:39:12.648 DEBUG:boto:Path: /pklxkd-58/?prefix=obj00000000000000000026 2023-05-31T20:39:12.649 DEBUG:boto:Data: 2023-05-31T20:39:12.649 DEBUG:boto:Headers: {} 2023-05-31T20:39:12.650 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:39:12.650 DEBUG:boto:Port: 8000 2023-05-31T20:39:12.650 DEBUG:boto:Params: {} 2023-05-31T20:39:12.651 DEBUG:boto:Token: None 2023-05-31T20:39:12.651 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:39:12 GMT /pklxkd-58/ 2023-05-31T20:39:12.652 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:ryNlxDRGgK16+XFs6oQP3KSCLXY= 2023-05-31T20:39:12.652 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:39:12 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:ryNlxDRGgK16+XFs6oQP3KSCLXY=', 'Content-Length': '0'} 2023-05-31T20:39:12.656 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f0834e2bb6f8a3e9-006477b070-1630-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:39:12 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:39:12.657 DEBUG:boto:b'pklxkd-58obj000000000000000000261000falseobj000000000000000000262023-05-31T20:38:22.407Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:39:12.657 DEBUG:boto:path=/pklxkd-58/obj00000000000000000026 2023-05-31T20:39:12.658 DEBUG:boto:auth_path=/pklxkd-58/obj00000000000000000026 2023-05-31T20:39:12.658 DEBUG:boto:Method: HEAD 2023-05-31T20:39:12.658 DEBUG:boto:Path: /pklxkd-58/obj00000000000000000026 2023-05-31T20:39:12.659 DEBUG:boto:Data: 2023-05-31T20:39:12.659 DEBUG:boto:Headers: {} 2023-05-31T20:39:12.659 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:39:12.660 DEBUG:boto:Port: 8001 2023-05-31T20:39:12.660 DEBUG:boto:Params: {} 2023-05-31T20:39:12.661 DEBUG:boto:Token: None 2023-05-31T20:39:12.661 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:39:12 GMT /pklxkd-58/obj00000000000000000026 2023-05-31T20:39:12.661 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:uj5hJ4R5TEjUFTmgI1s9I77eTuo= 2023-05-31T20:39:12.662 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:39:12 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:uj5hJ4R5TEjUFTmgI1s9I77eTuo=', 'Content-Length': '0'} 2023-05-31T20:39:12.663 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:38:22 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000173b4d3f0fabeab0-006477b070-1c17-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:39:12 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:39:12.664 DEBUG:boto:path=/pklxkd-58/obj00000000000000000026 2023-05-31T20:39:12.664 DEBUG:boto:auth_path=/pklxkd-58/obj00000000000000000026 2023-05-31T20:39:12.665 DEBUG:boto:Method: HEAD 2023-05-31T20:39:12.665 DEBUG:boto:Path: /pklxkd-58/obj00000000000000000026 2023-05-31T20:39:12.666 DEBUG:boto:Data: 2023-05-31T20:39:12.666 DEBUG:boto:Headers: {} 2023-05-31T20:39:12.666 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:39:12.667 DEBUG:boto:Port: 8000 2023-05-31T20:39:12.667 DEBUG:boto:Params: {} 2023-05-31T20:39:12.667 DEBUG:boto:Token: None 2023-05-31T20:39:12.668 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:39:12 GMT /pklxkd-58/obj00000000000000000026 2023-05-31T20:39:12.668 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:uj5hJ4R5TEjUFTmgI1s9I77eTuo= 2023-05-31T20:39:12.669 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:39:12 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:uj5hJ4R5TEjUFTmgI1s9I77eTuo=', 'Content-Length': '0'} 2023-05-31T20:39:12.670 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:38:22 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000fcc71da675be9b0c-006477b070-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:39:12 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:39:12.671 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000026 2023-05-31T20:39:12.672 DEBUG:boto:path=/pklxkd-58/obj00000000000000000026 2023-05-31T20:39:12.672 DEBUG:boto:auth_path=/pklxkd-58/obj00000000000000000026 2023-05-31T20:39:12.673 DEBUG:boto:Method: GET 2023-05-31T20:39:12.673 DEBUG:boto:Path: /pklxkd-58/obj00000000000000000026 2023-05-31T20:39:12.673 DEBUG:boto:Data: 2023-05-31T20:39:12.674 DEBUG:boto:Headers: {} 2023-05-31T20:39:12.674 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:39:12.675 DEBUG:boto:Port: 8001 2023-05-31T20:39:12.676 DEBUG:boto:Params: {} 2023-05-31T20:39:12.677 DEBUG:boto:Token: None 2023-05-31T20:39:12.677 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:39:12 GMT /pklxkd-58/obj00000000000000000026 2023-05-31T20:39:12.678 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:OEO1LQrQvCydpgSj5Qm9aTvc3h4= 2023-05-31T20:39:12.678 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:39:12 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:OEO1LQrQvCydpgSj5Qm9aTvc3h4=', 'Content-Length': '0'} 2023-05-31T20:39:12.680 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:38:22 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006281e532f1d6c561-006477b070-1c17-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:39:12 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:39:12.681 DEBUG:boto:path=/pklxkd-58/obj00000000000000000026 2023-05-31T20:39:12.681 DEBUG:boto:auth_path=/pklxkd-58/obj00000000000000000026 2023-05-31T20:39:12.682 DEBUG:boto:Method: GET 2023-05-31T20:39:12.682 DEBUG:boto:Path: /pklxkd-58/obj00000000000000000026 2023-05-31T20:39:12.683 DEBUG:boto:Data: 2023-05-31T20:39:12.683 DEBUG:boto:Headers: {} 2023-05-31T20:39:12.683 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:39:12.684 DEBUG:boto:Port: 8000 2023-05-31T20:39:12.684 DEBUG:boto:Params: {} 2023-05-31T20:39:12.685 DEBUG:boto:Token: None 2023-05-31T20:39:12.685 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:39:12 GMT /pklxkd-58/obj00000000000000000026 2023-05-31T20:39:12.686 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:OEO1LQrQvCydpgSj5Qm9aTvc3h4= 2023-05-31T20:39:12.686 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:39:12 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:OEO1LQrQvCydpgSj5Qm9aTvc3h4=', 'Content-Length': '0'} 2023-05-31T20:39:12.688 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:38:22 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b7929a6a1567aa20-006477b070-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:39:12 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:39:12.733 DEBUG:boto:path=/pklxkd-58/ 2023-05-31T20:39:12.733 DEBUG:boto:auth_path=/pklxkd-58/ 2023-05-31T20:39:12.734 DEBUG:boto:path=/pklxkd-58/?prefix=obj00000000000000000026 2023-05-31T20:39:12.734 DEBUG:boto:auth_path=/pklxkd-58/?prefix=obj00000000000000000026 2023-05-31T20:39:12.734 DEBUG:boto:Method: GET 2023-05-31T20:39:12.735 DEBUG:boto:Path: /pklxkd-58/?prefix=obj00000000000000000026 2023-05-31T20:39:12.735 DEBUG:boto:Data: 2023-05-31T20:39:12.736 DEBUG:boto:Headers: {} 2023-05-31T20:39:12.736 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:39:12.737 DEBUG:boto:Port: 8001 2023-05-31T20:39:12.737 DEBUG:boto:Params: {} 2023-05-31T20:39:12.737 DEBUG:boto:Token: None 2023-05-31T20:39:12.738 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:39:12 GMT /pklxkd-58/ 2023-05-31T20:39:12.738 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:ryNlxDRGgK16+XFs6oQP3KSCLXY= 2023-05-31T20:39:12.739 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:39:12 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:ryNlxDRGgK16+XFs6oQP3KSCLXY=', 'Content-Length': '0'} 2023-05-31T20:39:12.742 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000043566199aaeb0e89-006477b070-1c17-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:39:12 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:39:12.742 DEBUG:boto:b'pklxkd-58obj000000000000000000261000falseobj000000000000000000262023-05-31T20:38:22.407Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:39:12.743 DEBUG:boto:path=/pklxkd-58/ 2023-05-31T20:39:12.743 DEBUG:boto:auth_path=/pklxkd-58/ 2023-05-31T20:39:12.744 DEBUG:boto:path=/pklxkd-58/?prefix=obj00000000000000000026 2023-05-31T20:39:12.744 DEBUG:boto:auth_path=/pklxkd-58/?prefix=obj00000000000000000026 2023-05-31T20:39:12.745 DEBUG:boto:Method: GET 2023-05-31T20:39:12.745 DEBUG:boto:Path: /pklxkd-58/?prefix=obj00000000000000000026 2023-05-31T20:39:12.745 DEBUG:boto:Data: 2023-05-31T20:39:12.746 DEBUG:boto:Headers: {} 2023-05-31T20:39:12.747 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:39:12.748 DEBUG:boto:Port: 8000 2023-05-31T20:39:12.749 DEBUG:boto:Params: {} 2023-05-31T20:39:12.749 DEBUG:boto:Token: None 2023-05-31T20:39:12.750 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:39:12 GMT /pklxkd-58/ 2023-05-31T20:39:12.750 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:ryNlxDRGgK16+XFs6oQP3KSCLXY= 2023-05-31T20:39:12.751 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:39:12 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:ryNlxDRGgK16+XFs6oQP3KSCLXY=', 'Content-Length': '0'} 2023-05-31T20:39:12.755 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b0b89ddb0e2774c3-006477b070-1630-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:39:12 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:39:12.756 DEBUG:boto:b'pklxkd-58obj000000000000000000261000falseobj000000000000000000262023-05-31T20:38:22.407Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:39:12.757 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000027 2023-05-31T20:39:12.757 DEBUG:boto:path=/pklxkd-58/obj00000000000000000027 2023-05-31T20:39:12.758 DEBUG:boto:auth_path=/pklxkd-58/obj00000000000000000027 2023-05-31T20:39:12.758 DEBUG:boto:path=/pklxkd-58/obj00000000000000000027?versionId=null 2023-05-31T20:39:12.758 DEBUG:boto:auth_path=/pklxkd-58/obj00000000000000000027?versionId=null 2023-05-31T20:39:12.759 DEBUG:boto:Method: GET 2023-05-31T20:39:12.759 DEBUG:boto:Path: /pklxkd-58/obj00000000000000000027?versionId=null 2023-05-31T20:39:12.760 DEBUG:boto:Data: 2023-05-31T20:39:12.760 DEBUG:boto:Headers: {} 2023-05-31T20:39:12.761 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:39:12.761 DEBUG:boto:Port: 8001 2023-05-31T20:39:12.761 DEBUG:boto:Params: {} 2023-05-31T20:39:12.762 DEBUG:boto:Token: None 2023-05-31T20:39:12.762 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:39:12 GMT /pklxkd-58/obj00000000000000000027?versionId=null 2023-05-31T20:39:12.763 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:c1Zt9c4xcdTwLscaZb07rzLRgDM= 2023-05-31T20:39:12.763 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:39:12 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:c1Zt9c4xcdTwLscaZb07rzLRgDM=', 'Content-Length': '0'} 2023-05-31T20:39:12.766 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:38:21 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005cc8c11a0cd98548-006477b070-1c17-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:39:12 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:39:12.812 DEBUG:boto:path=/pklxkd-58/obj00000000000000000027 2023-05-31T20:39:12.813 DEBUG:boto:auth_path=/pklxkd-58/obj00000000000000000027 2023-05-31T20:39:12.814 DEBUG:boto:path=/pklxkd-58/obj00000000000000000027?versionId=null 2023-05-31T20:39:12.814 DEBUG:boto:auth_path=/pklxkd-58/obj00000000000000000027?versionId=null 2023-05-31T20:39:12.814 DEBUG:boto:Method: GET 2023-05-31T20:39:12.815 DEBUG:boto:Path: /pklxkd-58/obj00000000000000000027?versionId=null 2023-05-31T20:39:12.815 DEBUG:boto:Data: 2023-05-31T20:39:12.816 DEBUG:boto:Headers: {} 2023-05-31T20:39:12.816 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:39:12.816 DEBUG:boto:Port: 8000 2023-05-31T20:39:12.817 DEBUG:boto:Params: {} 2023-05-31T20:39:12.817 DEBUG:boto:Token: None 2023-05-31T20:39:12.818 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:39:12 GMT /pklxkd-58/obj00000000000000000027?versionId=null 2023-05-31T20:39:12.818 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:c1Zt9c4xcdTwLscaZb07rzLRgDM= 2023-05-31T20:39:12.819 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:39:12 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:c1Zt9c4xcdTwLscaZb07rzLRgDM=', 'Content-Length': '0'} 2023-05-31T20:39:12.821 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:38:21 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000094994c72e6715635-006477b070-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:39:12 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:39:12.822 DEBUG:boto:path=/pklxkd-58/obj00000000000000000027 2023-05-31T20:39:12.822 DEBUG:boto:auth_path=/pklxkd-58/obj00000000000000000027 2023-05-31T20:39:12.822 DEBUG:boto:path=/pklxkd-58/obj00000000000000000027?versionId=null 2023-05-31T20:39:12.823 DEBUG:boto:auth_path=/pklxkd-58/obj00000000000000000027?versionId=null 2023-05-31T20:39:12.823 DEBUG:boto:Method: HEAD 2023-05-31T20:39:12.823 DEBUG:boto:Path: /pklxkd-58/obj00000000000000000027?versionId=null 2023-05-31T20:39:12.824 DEBUG:boto:Data: 2023-05-31T20:39:12.824 DEBUG:boto:Headers: {} 2023-05-31T20:39:12.824 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:39:12.825 DEBUG:boto:Port: 8001 2023-05-31T20:39:12.825 DEBUG:boto:Params: {} 2023-05-31T20:39:12.826 DEBUG:boto:Token: None 2023-05-31T20:39:12.826 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:39:12 GMT /pklxkd-58/obj00000000000000000027?versionId=null 2023-05-31T20:39:12.826 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:0D2/29eE7oWfEwaxrvWmtDK5Zis= 2023-05-31T20:39:12.827 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:39:12 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:0D2/29eE7oWfEwaxrvWmtDK5Zis=', 'Content-Length': '0'} 2023-05-31T20:39:12.829 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:38:21 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c0a19608de7f0e08-006477b070-1c17-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:39:12 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:39:12.830 DEBUG:boto:path=/pklxkd-58/obj00000000000000000027 2023-05-31T20:39:12.830 DEBUG:boto:auth_path=/pklxkd-58/obj00000000000000000027 2023-05-31T20:39:12.830 DEBUG:boto:path=/pklxkd-58/obj00000000000000000027?versionId=null 2023-05-31T20:39:12.831 DEBUG:boto:auth_path=/pklxkd-58/obj00000000000000000027?versionId=null 2023-05-31T20:39:12.831 DEBUG:boto:Method: HEAD 2023-05-31T20:39:12.831 DEBUG:boto:Path: /pklxkd-58/obj00000000000000000027?versionId=null 2023-05-31T20:39:12.832 DEBUG:boto:Data: 2023-05-31T20:39:12.832 DEBUG:boto:Headers: {} 2023-05-31T20:39:12.832 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:39:12.833 DEBUG:boto:Port: 8000 2023-05-31T20:39:12.833 DEBUG:boto:Params: {} 2023-05-31T20:39:12.833 DEBUG:boto:Token: None 2023-05-31T20:39:12.834 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:39:12 GMT /pklxkd-58/obj00000000000000000027?versionId=null 2023-05-31T20:39:12.834 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:0D2/29eE7oWfEwaxrvWmtDK5Zis= 2023-05-31T20:39:12.835 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:39:12 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:0D2/29eE7oWfEwaxrvWmtDK5Zis=', 'Content-Length': '0'} 2023-05-31T20:39:12.836 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:38:21 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d0ae570f08f53de0-006477b070-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:39:12 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:39:12.837 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000027 2023-05-31T20:39:12.838 DEBUG:boto:path=/pklxkd-58/obj00000000000000000027 2023-05-31T20:39:12.838 DEBUG:boto:auth_path=/pklxkd-58/obj00000000000000000027 2023-05-31T20:39:12.838 DEBUG:boto:path=/pklxkd-58/obj00000000000000000027?versionId=null 2023-05-31T20:39:12.839 DEBUG:boto:auth_path=/pklxkd-58/obj00000000000000000027?versionId=null 2023-05-31T20:39:12.839 DEBUG:boto:Method: GET 2023-05-31T20:39:12.839 DEBUG:boto:Path: /pklxkd-58/obj00000000000000000027?versionId=null 2023-05-31T20:39:12.840 DEBUG:boto:Data: 2023-05-31T20:39:12.840 DEBUG:boto:Headers: {} 2023-05-31T20:39:12.841 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:39:12.841 DEBUG:boto:Port: 8001 2023-05-31T20:39:12.842 DEBUG:boto:Params: {} 2023-05-31T20:39:12.842 DEBUG:boto:Token: None 2023-05-31T20:39:12.843 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:39:12 GMT /pklxkd-58/obj00000000000000000027?versionId=null 2023-05-31T20:39:12.843 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:c1Zt9c4xcdTwLscaZb07rzLRgDM= 2023-05-31T20:39:12.843 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:39:12 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:c1Zt9c4xcdTwLscaZb07rzLRgDM=', 'Content-Length': '0'} 2023-05-31T20:39:12.845 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:38:21 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004a6c39dfe4e7adaf-006477b070-1c17-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:39:12 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:39:12.846 DEBUG:boto:path=/pklxkd-58/obj00000000000000000027 2023-05-31T20:39:12.847 DEBUG:boto:auth_path=/pklxkd-58/obj00000000000000000027 2023-05-31T20:39:12.847 DEBUG:boto:path=/pklxkd-58/obj00000000000000000027?versionId=null 2023-05-31T20:39:12.847 DEBUG:boto:auth_path=/pklxkd-58/obj00000000000000000027?versionId=null 2023-05-31T20:39:12.848 DEBUG:boto:Method: GET 2023-05-31T20:39:12.848 DEBUG:boto:Path: /pklxkd-58/obj00000000000000000027?versionId=null 2023-05-31T20:39:12.849 DEBUG:boto:Data: 2023-05-31T20:39:12.850 DEBUG:boto:Headers: {} 2023-05-31T20:39:12.851 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:39:12.852 DEBUG:boto:Port: 8000 2023-05-31T20:39:12.852 DEBUG:boto:Params: {} 2023-05-31T20:39:12.853 DEBUG:boto:Token: None 2023-05-31T20:39:12.854 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:39:12 GMT /pklxkd-58/obj00000000000000000027?versionId=null 2023-05-31T20:39:12.854 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:c1Zt9c4xcdTwLscaZb07rzLRgDM= 2023-05-31T20:39:12.855 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:39:12 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:c1Zt9c4xcdTwLscaZb07rzLRgDM=', 'Content-Length': '0'} 2023-05-31T20:39:12.857 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:38:21 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000a6273d361bb01f0-006477b070-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:39:12 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:39:12.900 DEBUG:boto:path=/pklxkd-58/ 2023-05-31T20:39:12.901 DEBUG:boto:auth_path=/pklxkd-58/ 2023-05-31T20:39:12.901 DEBUG:boto:path=/pklxkd-58/?prefix=obj00000000000000000027 2023-05-31T20:39:12.902 DEBUG:boto:auth_path=/pklxkd-58/?prefix=obj00000000000000000027 2023-05-31T20:39:12.902 DEBUG:boto:Method: GET 2023-05-31T20:39:12.903 DEBUG:boto:Path: /pklxkd-58/?prefix=obj00000000000000000027 2023-05-31T20:39:12.903 DEBUG:boto:Data: 2023-05-31T20:39:12.904 DEBUG:boto:Headers: {} 2023-05-31T20:39:12.905 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:39:12.905 DEBUG:boto:Port: 8001 2023-05-31T20:39:12.905 DEBUG:boto:Params: {} 2023-05-31T20:39:12.906 DEBUG:boto:Token: None 2023-05-31T20:39:12.906 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:39:12 GMT /pklxkd-58/ 2023-05-31T20:39:12.907 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:ryNlxDRGgK16+XFs6oQP3KSCLXY= 2023-05-31T20:39:12.907 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:39:12 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:ryNlxDRGgK16+XFs6oQP3KSCLXY=', 'Content-Length': '0'} 2023-05-31T20:39:12.910 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000658d6fb9bb0fe887-006477b070-1c17-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:39:12 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:39:12.911 DEBUG:boto:b'pklxkd-58obj000000000000000000271000falseobj000000000000000000272023-05-31T20:38:21.615Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:39:12.912 DEBUG:boto:path=/pklxkd-58/ 2023-05-31T20:39:12.912 DEBUG:boto:auth_path=/pklxkd-58/ 2023-05-31T20:39:12.913 DEBUG:boto:path=/pklxkd-58/?prefix=obj00000000000000000027 2023-05-31T20:39:12.913 DEBUG:boto:auth_path=/pklxkd-58/?prefix=obj00000000000000000027 2023-05-31T20:39:12.913 DEBUG:boto:Method: GET 2023-05-31T20:39:12.914 DEBUG:boto:Path: /pklxkd-58/?prefix=obj00000000000000000027 2023-05-31T20:39:12.914 DEBUG:boto:Data: 2023-05-31T20:39:12.915 DEBUG:boto:Headers: {} 2023-05-31T20:39:12.915 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:39:12.916 DEBUG:boto:Port: 8000 2023-05-31T20:39:12.916 DEBUG:boto:Params: {} 2023-05-31T20:39:12.917 DEBUG:boto:Token: None 2023-05-31T20:39:12.917 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:39:12 GMT /pklxkd-58/ 2023-05-31T20:39:12.917 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:ryNlxDRGgK16+XFs6oQP3KSCLXY= 2023-05-31T20:39:12.918 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:39:12 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:ryNlxDRGgK16+XFs6oQP3KSCLXY=', 'Content-Length': '0'} 2023-05-31T20:39:12.923 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000000d5a9e8c3b53c7d0-006477b070-1630-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:39:12 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:39:12.924 DEBUG:boto:b'pklxkd-58obj000000000000000000271000falseobj000000000000000000272023-05-31T20:38:21.615Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:39:12.925 DEBUG:boto:path=/pklxkd-58/obj00000000000000000027 2023-05-31T20:39:12.926 DEBUG:boto:auth_path=/pklxkd-58/obj00000000000000000027 2023-05-31T20:39:12.926 DEBUG:boto:Method: HEAD 2023-05-31T20:39:12.927 DEBUG:boto:Path: /pklxkd-58/obj00000000000000000027 2023-05-31T20:39:12.928 DEBUG:boto:Data: 2023-05-31T20:39:12.928 DEBUG:boto:Headers: {} 2023-05-31T20:39:12.929 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:39:12.930 DEBUG:boto:Port: 8001 2023-05-31T20:39:12.930 DEBUG:boto:Params: {} 2023-05-31T20:39:12.930 DEBUG:boto:Token: None 2023-05-31T20:39:12.931 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:39:12 GMT /pklxkd-58/obj00000000000000000027 2023-05-31T20:39:12.931 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:xZKHtoDDz2oBlVaHWfPykzi1VJw= 2023-05-31T20:39:12.932 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:39:12 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:xZKHtoDDz2oBlVaHWfPykzi1VJw=', 'Content-Length': '0'} 2023-05-31T20:39:12.934 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:38:21 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000789b9d227e65d76d-006477b070-1c17-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:39:12 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:39:12.934 DEBUG:boto:path=/pklxkd-58/obj00000000000000000027 2023-05-31T20:39:12.935 DEBUG:boto:auth_path=/pklxkd-58/obj00000000000000000027 2023-05-31T20:39:12.935 DEBUG:boto:Method: HEAD 2023-05-31T20:39:12.936 DEBUG:boto:Path: /pklxkd-58/obj00000000000000000027 2023-05-31T20:39:12.936 DEBUG:boto:Data: 2023-05-31T20:39:12.937 DEBUG:boto:Headers: {} 2023-05-31T20:39:12.937 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:39:12.937 DEBUG:boto:Port: 8000 2023-05-31T20:39:12.938 DEBUG:boto:Params: {} 2023-05-31T20:39:12.938 DEBUG:boto:Token: None 2023-05-31T20:39:12.939 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:39:12 GMT /pklxkd-58/obj00000000000000000027 2023-05-31T20:39:12.939 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:xZKHtoDDz2oBlVaHWfPykzi1VJw= 2023-05-31T20:39:12.940 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:39:12 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:xZKHtoDDz2oBlVaHWfPykzi1VJw=', 'Content-Length': '0'} 2023-05-31T20:39:12.942 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:38:21 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d2f146b670f13d1d-006477b070-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:39:12 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:39:12.942 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000027 2023-05-31T20:39:12.943 DEBUG:boto:path=/pklxkd-58/obj00000000000000000027 2023-05-31T20:39:12.943 DEBUG:boto:auth_path=/pklxkd-58/obj00000000000000000027 2023-05-31T20:39:12.944 DEBUG:boto:Method: GET 2023-05-31T20:39:12.944 DEBUG:boto:Path: /pklxkd-58/obj00000000000000000027 2023-05-31T20:39:12.944 DEBUG:boto:Data: 2023-05-31T20:39:12.945 DEBUG:boto:Headers: {} 2023-05-31T20:39:12.945 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:39:12.946 DEBUG:boto:Port: 8001 2023-05-31T20:39:12.946 DEBUG:boto:Params: {} 2023-05-31T20:39:12.947 DEBUG:boto:Token: None 2023-05-31T20:39:12.947 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:39:12 GMT /pklxkd-58/obj00000000000000000027 2023-05-31T20:39:12.948 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:Vn5wBl6+uehq84bXdxJ7P97KbO8= 2023-05-31T20:39:12.948 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:39:12 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:Vn5wBl6+uehq84bXdxJ7P97KbO8=', 'Content-Length': '0'} 2023-05-31T20:39:12.950 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:38:21 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000deb632a61c994c90-006477b070-1c17-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:39:12 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:39:12.992 DEBUG:boto:path=/pklxkd-58/obj00000000000000000027 2023-05-31T20:39:12.993 DEBUG:boto:auth_path=/pklxkd-58/obj00000000000000000027 2023-05-31T20:39:12.993 DEBUG:boto:Method: GET 2023-05-31T20:39:12.994 DEBUG:boto:Path: /pklxkd-58/obj00000000000000000027 2023-05-31T20:39:12.994 DEBUG:boto:Data: 2023-05-31T20:39:12.995 DEBUG:boto:Headers: {} 2023-05-31T20:39:12.995 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:39:12.995 DEBUG:boto:Port: 8000 2023-05-31T20:39:12.996 DEBUG:boto:Params: {} 2023-05-31T20:39:12.996 DEBUG:boto:Token: None 2023-05-31T20:39:12.997 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:39:12 GMT /pklxkd-58/obj00000000000000000027 2023-05-31T20:39:12.997 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:Vn5wBl6+uehq84bXdxJ7P97KbO8= 2023-05-31T20:39:12.998 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:39:12 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:Vn5wBl6+uehq84bXdxJ7P97KbO8=', 'Content-Length': '0'} 2023-05-31T20:39:13.000 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:38:21 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000030514afe10e05c47-006477b070-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:39:12 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:39:13.001 DEBUG:boto:path=/pklxkd-58/ 2023-05-31T20:39:13.001 DEBUG:boto:auth_path=/pklxkd-58/ 2023-05-31T20:39:13.001 DEBUG:boto:path=/pklxkd-58/?prefix=obj00000000000000000027 2023-05-31T20:39:13.002 DEBUG:boto:auth_path=/pklxkd-58/?prefix=obj00000000000000000027 2023-05-31T20:39:13.002 DEBUG:boto:Method: GET 2023-05-31T20:39:13.003 DEBUG:boto:Path: /pklxkd-58/?prefix=obj00000000000000000027 2023-05-31T20:39:13.003 DEBUG:boto:Data: 2023-05-31T20:39:13.003 DEBUG:boto:Headers: {} 2023-05-31T20:39:13.004 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:39:13.004 DEBUG:boto:Port: 8001 2023-05-31T20:39:13.005 DEBUG:boto:Params: {} 2023-05-31T20:39:13.005 DEBUG:boto:Token: None 2023-05-31T20:39:13.006 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:39:13 GMT /pklxkd-58/ 2023-05-31T20:39:13.006 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:GhwzgzgAYYFLDcfdK1Agett0HKk= 2023-05-31T20:39:13.007 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:39:13 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:GhwzgzgAYYFLDcfdK1Agett0HKk=', 'Content-Length': '0'} 2023-05-31T20:39:13.009 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000760b7be2ffb8b9f1-006477b071-1c17-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:39:13 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:39:13.010 DEBUG:boto:b'pklxkd-58obj000000000000000000271000falseobj000000000000000000272023-05-31T20:38:21.615Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:39:13.012 DEBUG:boto:path=/pklxkd-58/ 2023-05-31T20:39:13.013 DEBUG:boto:auth_path=/pklxkd-58/ 2023-05-31T20:39:13.014 DEBUG:boto:path=/pklxkd-58/?prefix=obj00000000000000000027 2023-05-31T20:39:13.014 DEBUG:boto:auth_path=/pklxkd-58/?prefix=obj00000000000000000027 2023-05-31T20:39:13.014 DEBUG:boto:Method: GET 2023-05-31T20:39:13.015 DEBUG:boto:Path: /pklxkd-58/?prefix=obj00000000000000000027 2023-05-31T20:39:13.015 DEBUG:boto:Data: 2023-05-31T20:39:13.016 DEBUG:boto:Headers: {} 2023-05-31T20:39:13.016 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:39:13.017 DEBUG:boto:Port: 8000 2023-05-31T20:39:13.017 DEBUG:boto:Params: {} 2023-05-31T20:39:13.018 DEBUG:boto:Token: None 2023-05-31T20:39:13.019 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:39:13 GMT /pklxkd-58/ 2023-05-31T20:39:13.020 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:GhwzgzgAYYFLDcfdK1Agett0HKk= 2023-05-31T20:39:13.020 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:39:13 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:GhwzgzgAYYFLDcfdK1Agett0HKk=', 'Content-Length': '0'} 2023-05-31T20:39:13.024 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000755815f4feb4786d-006477b071-1630-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:39:13 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:39:13.025 DEBUG:boto:b'pklxkd-58obj000000000000000000271000falseobj000000000000000000272023-05-31T20:38:21.615Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:39:13.026 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000028 2023-05-31T20:39:13.026 DEBUG:boto:path=/pklxkd-58/obj00000000000000000028 2023-05-31T20:39:13.027 DEBUG:boto:auth_path=/pklxkd-58/obj00000000000000000028 2023-05-31T20:39:13.027 DEBUG:boto:path=/pklxkd-58/obj00000000000000000028?versionId=null 2023-05-31T20:39:13.027 DEBUG:boto:auth_path=/pklxkd-58/obj00000000000000000028?versionId=null 2023-05-31T20:39:13.028 DEBUG:boto:Method: GET 2023-05-31T20:39:13.028 DEBUG:boto:Path: /pklxkd-58/obj00000000000000000028?versionId=null 2023-05-31T20:39:13.028 DEBUG:boto:Data: 2023-05-31T20:39:13.029 DEBUG:boto:Headers: {} 2023-05-31T20:39:13.029 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:39:13.030 DEBUG:boto:Port: 8001 2023-05-31T20:39:13.031 DEBUG:boto:Params: {} 2023-05-31T20:39:13.032 DEBUG:boto:Token: None 2023-05-31T20:39:13.032 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:39:13 GMT /pklxkd-58/obj00000000000000000028?versionId=null 2023-05-31T20:39:13.033 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:FmKi1ojEpvzi09GnMGWQqudaiLo= 2023-05-31T20:39:13.033 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:39:13 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:FmKi1ojEpvzi09GnMGWQqudaiLo=', 'Content-Length': '0'} 2023-05-31T20:39:13.035 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:38:15 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000af0dce782aa7d76f-006477b071-1c17-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:39:13 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:39:13.036 DEBUG:boto:path=/pklxkd-58/obj00000000000000000028 2023-05-31T20:39:13.037 DEBUG:boto:auth_path=/pklxkd-58/obj00000000000000000028 2023-05-31T20:39:13.037 DEBUG:boto:path=/pklxkd-58/obj00000000000000000028?versionId=null 2023-05-31T20:39:13.038 DEBUG:boto:auth_path=/pklxkd-58/obj00000000000000000028?versionId=null 2023-05-31T20:39:13.038 DEBUG:boto:Method: GET 2023-05-31T20:39:13.038 DEBUG:boto:Path: /pklxkd-58/obj00000000000000000028?versionId=null 2023-05-31T20:39:13.039 DEBUG:boto:Data: 2023-05-31T20:39:13.039 DEBUG:boto:Headers: {} 2023-05-31T20:39:13.040 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:39:13.040 DEBUG:boto:Port: 8000 2023-05-31T20:39:13.041 DEBUG:boto:Params: {} 2023-05-31T20:39:13.041 DEBUG:boto:Token: None 2023-05-31T20:39:13.042 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:39:13 GMT /pklxkd-58/obj00000000000000000028?versionId=null 2023-05-31T20:39:13.042 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:FmKi1ojEpvzi09GnMGWQqudaiLo= 2023-05-31T20:39:13.043 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:39:13 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:FmKi1ojEpvzi09GnMGWQqudaiLo=', 'Content-Length': '0'} 2023-05-31T20:39:13.044 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:38:15 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000af0d1f1c50b9547c-006477b071-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:39:13 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:39:13.088 DEBUG:boto:path=/pklxkd-58/obj00000000000000000028 2023-05-31T20:39:13.089 DEBUG:boto:auth_path=/pklxkd-58/obj00000000000000000028 2023-05-31T20:39:13.089 DEBUG:boto:path=/pklxkd-58/obj00000000000000000028?versionId=null 2023-05-31T20:39:13.090 DEBUG:boto:auth_path=/pklxkd-58/obj00000000000000000028?versionId=null 2023-05-31T20:39:13.090 DEBUG:boto:Method: HEAD 2023-05-31T20:39:13.091 DEBUG:boto:Path: /pklxkd-58/obj00000000000000000028?versionId=null 2023-05-31T20:39:13.092 DEBUG:boto:Data: 2023-05-31T20:39:13.092 DEBUG:boto:Headers: {} 2023-05-31T20:39:13.093 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:39:13.094 DEBUG:boto:Port: 8001 2023-05-31T20:39:13.094 DEBUG:boto:Params: {} 2023-05-31T20:39:13.095 DEBUG:boto:Token: None 2023-05-31T20:39:13.096 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:39:13 GMT /pklxkd-58/obj00000000000000000028?versionId=null 2023-05-31T20:39:13.097 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:SfxdAKMG7Jq+o0bfZgB4GtQTlF0= 2023-05-31T20:39:13.097 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:39:13 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:SfxdAKMG7Jq+o0bfZgB4GtQTlF0=', 'Content-Length': '0'} 2023-05-31T20:39:13.100 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:38:15 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000464998f97a8f3350-006477b071-1c17-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:39:13 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:39:13.101 DEBUG:boto:path=/pklxkd-58/obj00000000000000000028 2023-05-31T20:39:13.102 DEBUG:boto:auth_path=/pklxkd-58/obj00000000000000000028 2023-05-31T20:39:13.103 DEBUG:boto:path=/pklxkd-58/obj00000000000000000028?versionId=null 2023-05-31T20:39:13.104 DEBUG:boto:auth_path=/pklxkd-58/obj00000000000000000028?versionId=null 2023-05-31T20:39:13.104 DEBUG:boto:Method: HEAD 2023-05-31T20:39:13.105 DEBUG:boto:Path: /pklxkd-58/obj00000000000000000028?versionId=null 2023-05-31T20:39:13.106 DEBUG:boto:Data: 2023-05-31T20:39:13.106 DEBUG:boto:Headers: {} 2023-05-31T20:39:13.107 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:39:13.108 DEBUG:boto:Port: 8000 2023-05-31T20:39:13.108 DEBUG:boto:Params: {} 2023-05-31T20:39:13.109 DEBUG:boto:Token: None 2023-05-31T20:39:13.109 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:39:13 GMT /pklxkd-58/obj00000000000000000028?versionId=null 2023-05-31T20:39:13.110 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:SfxdAKMG7Jq+o0bfZgB4GtQTlF0= 2023-05-31T20:39:13.110 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:39:13 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:SfxdAKMG7Jq+o0bfZgB4GtQTlF0=', 'Content-Length': '0'} 2023-05-31T20:39:13.112 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:38:15 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000671aa907f235a650-006477b071-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:39:13 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:39:13.113 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000028 2023-05-31T20:39:13.113 DEBUG:boto:path=/pklxkd-58/obj00000000000000000028 2023-05-31T20:39:13.114 DEBUG:boto:auth_path=/pklxkd-58/obj00000000000000000028 2023-05-31T20:39:13.114 DEBUG:boto:path=/pklxkd-58/obj00000000000000000028?versionId=null 2023-05-31T20:39:13.114 DEBUG:boto:auth_path=/pklxkd-58/obj00000000000000000028?versionId=null 2023-05-31T20:39:13.115 DEBUG:boto:Method: GET 2023-05-31T20:39:13.115 DEBUG:boto:Path: /pklxkd-58/obj00000000000000000028?versionId=null 2023-05-31T20:39:13.116 DEBUG:boto:Data: 2023-05-31T20:39:13.116 DEBUG:boto:Headers: {} 2023-05-31T20:39:13.117 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:39:13.117 DEBUG:boto:Port: 8001 2023-05-31T20:39:13.117 DEBUG:boto:Params: {} 2023-05-31T20:39:13.118 DEBUG:boto:Token: None 2023-05-31T20:39:13.118 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:39:13 GMT /pklxkd-58/obj00000000000000000028?versionId=null 2023-05-31T20:39:13.119 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:FmKi1ojEpvzi09GnMGWQqudaiLo= 2023-05-31T20:39:13.119 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:39:13 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:FmKi1ojEpvzi09GnMGWQqudaiLo=', 'Content-Length': '0'} 2023-05-31T20:39:13.121 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:38:15 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c589c92b9bc8c42f-006477b071-1c17-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:39:13 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:39:13.123 DEBUG:boto:path=/pklxkd-58/obj00000000000000000028 2023-05-31T20:39:13.124 DEBUG:boto:auth_path=/pklxkd-58/obj00000000000000000028 2023-05-31T20:39:13.124 DEBUG:boto:path=/pklxkd-58/obj00000000000000000028?versionId=null 2023-05-31T20:39:13.125 DEBUG:boto:auth_path=/pklxkd-58/obj00000000000000000028?versionId=null 2023-05-31T20:39:13.126 DEBUG:boto:Method: GET 2023-05-31T20:39:13.127 DEBUG:boto:Path: /pklxkd-58/obj00000000000000000028?versionId=null 2023-05-31T20:39:13.128 DEBUG:boto:Data: 2023-05-31T20:39:13.128 DEBUG:boto:Headers: {} 2023-05-31T20:39:13.129 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:39:13.130 DEBUG:boto:Port: 8000 2023-05-31T20:39:13.130 DEBUG:boto:Params: {} 2023-05-31T20:39:13.131 DEBUG:boto:Token: None 2023-05-31T20:39:13.132 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:39:13 GMT /pklxkd-58/obj00000000000000000028?versionId=null 2023-05-31T20:39:13.132 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:FmKi1ojEpvzi09GnMGWQqudaiLo= 2023-05-31T20:39:13.133 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:39:13 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:FmKi1ojEpvzi09GnMGWQqudaiLo=', 'Content-Length': '0'} 2023-05-31T20:39:13.136 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:38:15 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000032e880cd311ed9c5-006477b071-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:39:13 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:39:13.137 DEBUG:boto:path=/pklxkd-58/ 2023-05-31T20:39:13.137 DEBUG:boto:auth_path=/pklxkd-58/ 2023-05-31T20:39:13.138 DEBUG:boto:path=/pklxkd-58/?prefix=obj00000000000000000028 2023-05-31T20:39:13.139 DEBUG:boto:auth_path=/pklxkd-58/?prefix=obj00000000000000000028 2023-05-31T20:39:13.140 DEBUG:boto:Method: GET 2023-05-31T20:39:13.141 DEBUG:boto:Path: /pklxkd-58/?prefix=obj00000000000000000028 2023-05-31T20:39:13.142 DEBUG:boto:Data: 2023-05-31T20:39:13.143 DEBUG:boto:Headers: {} 2023-05-31T20:39:13.143 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:39:13.144 DEBUG:boto:Port: 8001 2023-05-31T20:39:13.145 DEBUG:boto:Params: {} 2023-05-31T20:39:13.146 DEBUG:boto:Token: None 2023-05-31T20:39:13.147 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:39:13 GMT /pklxkd-58/ 2023-05-31T20:39:13.148 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:GhwzgzgAYYFLDcfdK1Agett0HKk= 2023-05-31T20:39:13.149 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:39:13 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:GhwzgzgAYYFLDcfdK1Agett0HKk=', 'Content-Length': '0'} 2023-05-31T20:39:13.151 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000003c8f36784a4629bf-006477b071-1c17-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:39:13 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:39:13.196 DEBUG:boto:b'pklxkd-58obj000000000000000000281000falseobj000000000000000000282023-05-31T20:38:15.316Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:39:13.197 DEBUG:boto:path=/pklxkd-58/ 2023-05-31T20:39:13.198 DEBUG:boto:auth_path=/pklxkd-58/ 2023-05-31T20:39:13.198 DEBUG:boto:path=/pklxkd-58/?prefix=obj00000000000000000028 2023-05-31T20:39:13.199 DEBUG:boto:auth_path=/pklxkd-58/?prefix=obj00000000000000000028 2023-05-31T20:39:13.200 DEBUG:boto:Method: GET 2023-05-31T20:39:13.201 DEBUG:boto:Path: /pklxkd-58/?prefix=obj00000000000000000028 2023-05-31T20:39:13.201 DEBUG:boto:Data: 2023-05-31T20:39:13.202 DEBUG:boto:Headers: {} 2023-05-31T20:39:13.203 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:39:13.204 DEBUG:boto:Port: 8000 2023-05-31T20:39:13.205 DEBUG:boto:Params: {} 2023-05-31T20:39:13.206 DEBUG:boto:Token: None 2023-05-31T20:39:13.207 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:39:13 GMT /pklxkd-58/ 2023-05-31T20:39:13.208 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:GhwzgzgAYYFLDcfdK1Agett0HKk= 2023-05-31T20:39:13.209 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:39:13 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:GhwzgzgAYYFLDcfdK1Agett0HKk=', 'Content-Length': '0'} 2023-05-31T20:39:13.213 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d6a52cbf11a818de-006477b071-1630-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:39:13 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:39:13.215 DEBUG:boto:b'pklxkd-58obj000000000000000000281000falseobj000000000000000000282023-05-31T20:38:15.316Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:39:13.217 DEBUG:boto:path=/pklxkd-58/obj00000000000000000028 2023-05-31T20:39:13.217 DEBUG:boto:auth_path=/pklxkd-58/obj00000000000000000028 2023-05-31T20:39:13.218 DEBUG:boto:Method: HEAD 2023-05-31T20:39:13.219 DEBUG:boto:Path: /pklxkd-58/obj00000000000000000028 2023-05-31T20:39:13.220 DEBUG:boto:Data: 2023-05-31T20:39:13.221 DEBUG:boto:Headers: {} 2023-05-31T20:39:13.221 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:39:13.222 DEBUG:boto:Port: 8001 2023-05-31T20:39:13.223 DEBUG:boto:Params: {} 2023-05-31T20:39:13.223 DEBUG:boto:Token: None 2023-05-31T20:39:13.224 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:39:13 GMT /pklxkd-58/obj00000000000000000028 2023-05-31T20:39:13.224 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:3use8QDl7GOP9QY3H/L/rWdr0nw= 2023-05-31T20:39:13.225 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:39:13 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:3use8QDl7GOP9QY3H/L/rWdr0nw=', 'Content-Length': '0'} 2023-05-31T20:39:13.227 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:38:15 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000fffe43b79c2a93e8-006477b071-1c17-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:39:13 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:39:13.228 DEBUG:boto:path=/pklxkd-58/obj00000000000000000028 2023-05-31T20:39:13.228 DEBUG:boto:auth_path=/pklxkd-58/obj00000000000000000028 2023-05-31T20:39:13.229 DEBUG:boto:Method: HEAD 2023-05-31T20:39:13.229 DEBUG:boto:Path: /pklxkd-58/obj00000000000000000028 2023-05-31T20:39:13.229 DEBUG:boto:Data: 2023-05-31T20:39:13.230 DEBUG:boto:Headers: {} 2023-05-31T20:39:13.230 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:39:13.230 DEBUG:boto:Port: 8000 2023-05-31T20:39:13.231 DEBUG:boto:Params: {} 2023-05-31T20:39:13.231 DEBUG:boto:Token: None 2023-05-31T20:39:13.231 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:39:13 GMT /pklxkd-58/obj00000000000000000028 2023-05-31T20:39:13.232 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:3use8QDl7GOP9QY3H/L/rWdr0nw= 2023-05-31T20:39:13.232 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:39:13 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:3use8QDl7GOP9QY3H/L/rWdr0nw=', 'Content-Length': '0'} 2023-05-31T20:39:13.234 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:38:15 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000bfab8de9313853ca-006477b071-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:39:13 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:39:13.235 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000028 2023-05-31T20:39:13.235 DEBUG:boto:path=/pklxkd-58/obj00000000000000000028 2023-05-31T20:39:13.235 DEBUG:boto:auth_path=/pklxkd-58/obj00000000000000000028 2023-05-31T20:39:13.236 DEBUG:boto:Method: GET 2023-05-31T20:39:13.236 DEBUG:boto:Path: /pklxkd-58/obj00000000000000000028 2023-05-31T20:39:13.237 DEBUG:boto:Data: 2023-05-31T20:39:13.237 DEBUG:boto:Headers: {} 2023-05-31T20:39:13.237 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:39:13.238 DEBUG:boto:Port: 8001 2023-05-31T20:39:13.239 DEBUG:boto:Params: {} 2023-05-31T20:39:13.239 DEBUG:boto:Token: None 2023-05-31T20:39:13.239 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:39:13 GMT /pklxkd-58/obj00000000000000000028 2023-05-31T20:39:13.240 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:8angCnKbIzOFr2vkZ4YJEqpOhAA= 2023-05-31T20:39:13.240 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:39:13 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:8angCnKbIzOFr2vkZ4YJEqpOhAA=', 'Content-Length': '0'} 2023-05-31T20:39:13.242 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:38:15 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000007c783bad814cd09-006477b071-1c17-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:39:13 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:39:13.244 DEBUG:boto:path=/pklxkd-58/obj00000000000000000028 2023-05-31T20:39:13.245 DEBUG:boto:auth_path=/pklxkd-58/obj00000000000000000028 2023-05-31T20:39:13.246 DEBUG:boto:Method: GET 2023-05-31T20:39:13.247 DEBUG:boto:Path: /pklxkd-58/obj00000000000000000028 2023-05-31T20:39:13.248 DEBUG:boto:Data: 2023-05-31T20:39:13.248 DEBUG:boto:Headers: {} 2023-05-31T20:39:13.249 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:39:13.250 DEBUG:boto:Port: 8000 2023-05-31T20:39:13.250 DEBUG:boto:Params: {} 2023-05-31T20:39:13.251 DEBUG:boto:Token: None 2023-05-31T20:39:13.252 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:39:13 GMT /pklxkd-58/obj00000000000000000028 2023-05-31T20:39:13.253 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:8angCnKbIzOFr2vkZ4YJEqpOhAA= 2023-05-31T20:39:13.253 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:39:13 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:8angCnKbIzOFr2vkZ4YJEqpOhAA=', 'Content-Length': '0'} 2023-05-31T20:39:13.255 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:38:15 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000fa6f34264bd899a5-006477b071-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:39:13 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:39:13.255 DEBUG:boto:path=/pklxkd-58/ 2023-05-31T20:39:13.256 DEBUG:boto:auth_path=/pklxkd-58/ 2023-05-31T20:39:13.256 DEBUG:boto:path=/pklxkd-58/?prefix=obj00000000000000000028 2023-05-31T20:39:13.257 DEBUG:boto:auth_path=/pklxkd-58/?prefix=obj00000000000000000028 2023-05-31T20:39:13.257 DEBUG:boto:Method: GET 2023-05-31T20:39:13.257 DEBUG:boto:Path: /pklxkd-58/?prefix=obj00000000000000000028 2023-05-31T20:39:13.258 DEBUG:boto:Data: 2023-05-31T20:39:13.258 DEBUG:boto:Headers: {} 2023-05-31T20:39:13.258 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:39:13.259 DEBUG:boto:Port: 8001 2023-05-31T20:39:13.259 DEBUG:boto:Params: {} 2023-05-31T20:39:13.260 DEBUG:boto:Token: None 2023-05-31T20:39:13.260 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:39:13 GMT /pklxkd-58/ 2023-05-31T20:39:13.261 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:GhwzgzgAYYFLDcfdK1Agett0HKk= 2023-05-31T20:39:13.261 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:39:13 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:GhwzgzgAYYFLDcfdK1Agett0HKk=', 'Content-Length': '0'} 2023-05-31T20:39:13.263 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000eda2cde700707fa9-006477b071-1c17-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:39:13 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:39:13.264 DEBUG:boto:b'pklxkd-58obj000000000000000000281000falseobj000000000000000000282023-05-31T20:38:15.316Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:39:13.265 DEBUG:boto:path=/pklxkd-58/ 2023-05-31T20:39:13.265 DEBUG:boto:auth_path=/pklxkd-58/ 2023-05-31T20:39:13.265 DEBUG:boto:path=/pklxkd-58/?prefix=obj00000000000000000028 2023-05-31T20:39:13.266 DEBUG:boto:auth_path=/pklxkd-58/?prefix=obj00000000000000000028 2023-05-31T20:39:13.266 DEBUG:boto:Method: GET 2023-05-31T20:39:13.266 DEBUG:boto:Path: /pklxkd-58/?prefix=obj00000000000000000028 2023-05-31T20:39:13.267 DEBUG:boto:Data: 2023-05-31T20:39:13.267 DEBUG:boto:Headers: {} 2023-05-31T20:39:13.267 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:39:13.268 DEBUG:boto:Port: 8000 2023-05-31T20:39:13.268 DEBUG:boto:Params: {} 2023-05-31T20:39:13.269 DEBUG:boto:Token: None 2023-05-31T20:39:13.269 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:39:13 GMT /pklxkd-58/ 2023-05-31T20:39:13.270 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:GhwzgzgAYYFLDcfdK1Agett0HKk= 2023-05-31T20:39:13.270 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:39:13 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:GhwzgzgAYYFLDcfdK1Agett0HKk=', 'Content-Length': '0'} 2023-05-31T20:39:13.274 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000562c6562a0b814f5-006477b071-1630-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:39:13 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:39:13.275 DEBUG:boto:b'pklxkd-58obj000000000000000000281000falseobj000000000000000000282023-05-31T20:38:15.316Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:39:13.277 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000029 2023-05-31T20:39:13.277 DEBUG:boto:path=/pklxkd-58/obj00000000000000000029 2023-05-31T20:39:13.278 DEBUG:boto:auth_path=/pklxkd-58/obj00000000000000000029 2023-05-31T20:39:13.279 DEBUG:boto:path=/pklxkd-58/obj00000000000000000029?versionId=null 2023-05-31T20:39:13.279 DEBUG:boto:auth_path=/pklxkd-58/obj00000000000000000029?versionId=null 2023-05-31T20:39:13.279 DEBUG:boto:Method: GET 2023-05-31T20:39:13.280 DEBUG:boto:Path: /pklxkd-58/obj00000000000000000029?versionId=null 2023-05-31T20:39:13.280 DEBUG:boto:Data: 2023-05-31T20:39:13.281 DEBUG:boto:Headers: {} 2023-05-31T20:39:13.281 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:39:13.282 DEBUG:boto:Port: 8001 2023-05-31T20:39:13.283 DEBUG:boto:Params: {} 2023-05-31T20:39:13.284 DEBUG:boto:Token: None 2023-05-31T20:39:13.285 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:39:13 GMT /pklxkd-58/obj00000000000000000029?versionId=null 2023-05-31T20:39:13.286 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:LBWuFcOXGF5qZKBigM0z8uC/4Uk= 2023-05-31T20:39:13.287 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:39:13 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:LBWuFcOXGF5qZKBigM0z8uC/4Uk=', 'Content-Length': '0'} 2023-05-31T20:39:13.290 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:38:20 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000bfe6222bbc0536bf-006477b071-1c17-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:39:13 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:39:13.291 DEBUG:boto:path=/pklxkd-58/obj00000000000000000029 2023-05-31T20:39:13.292 DEBUG:boto:auth_path=/pklxkd-58/obj00000000000000000029 2023-05-31T20:39:13.293 DEBUG:boto:path=/pklxkd-58/obj00000000000000000029?versionId=null 2023-05-31T20:39:13.293 DEBUG:boto:auth_path=/pklxkd-58/obj00000000000000000029?versionId=null 2023-05-31T20:39:13.293 DEBUG:boto:Method: GET 2023-05-31T20:39:13.294 DEBUG:boto:Path: /pklxkd-58/obj00000000000000000029?versionId=null 2023-05-31T20:39:13.294 DEBUG:boto:Data: 2023-05-31T20:39:13.295 DEBUG:boto:Headers: {} 2023-05-31T20:39:13.295 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:39:13.295 DEBUG:boto:Port: 8000 2023-05-31T20:39:13.296 DEBUG:boto:Params: {} 2023-05-31T20:39:13.296 DEBUG:boto:Token: None 2023-05-31T20:39:13.297 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:39:13 GMT /pklxkd-58/obj00000000000000000029?versionId=null 2023-05-31T20:39:13.297 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:LBWuFcOXGF5qZKBigM0z8uC/4Uk= 2023-05-31T20:39:13.297 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:39:13 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:LBWuFcOXGF5qZKBigM0z8uC/4Uk=', 'Content-Length': '0'} 2023-05-31T20:39:13.300 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:38:20 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009a3e19c0dd239e8b-006477b071-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:39:13 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:39:13.340 DEBUG:boto:path=/pklxkd-58/obj00000000000000000029 2023-05-31T20:39:13.341 DEBUG:boto:auth_path=/pklxkd-58/obj00000000000000000029 2023-05-31T20:39:13.342 DEBUG:boto:path=/pklxkd-58/obj00000000000000000029?versionId=null 2023-05-31T20:39:13.342 DEBUG:boto:auth_path=/pklxkd-58/obj00000000000000000029?versionId=null 2023-05-31T20:39:13.343 DEBUG:boto:Method: HEAD 2023-05-31T20:39:13.344 DEBUG:boto:Path: /pklxkd-58/obj00000000000000000029?versionId=null 2023-05-31T20:39:13.344 DEBUG:boto:Data: 2023-05-31T20:39:13.345 DEBUG:boto:Headers: {} 2023-05-31T20:39:13.346 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:39:13.346 DEBUG:boto:Port: 8001 2023-05-31T20:39:13.347 DEBUG:boto:Params: {} 2023-05-31T20:39:13.347 DEBUG:boto:Token: None 2023-05-31T20:39:13.348 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:39:13 GMT /pklxkd-58/obj00000000000000000029?versionId=null 2023-05-31T20:39:13.349 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:llABaOxb0D2nzuOSQ920gsqiVpo= 2023-05-31T20:39:13.350 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:39:13 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:llABaOxb0D2nzuOSQ920gsqiVpo=', 'Content-Length': '0'} 2023-05-31T20:39:13.353 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:38:20 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005335ade985841df1-006477b071-1c17-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:39:13 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:39:13.354 DEBUG:boto:path=/pklxkd-58/obj00000000000000000029 2023-05-31T20:39:13.355 DEBUG:boto:auth_path=/pklxkd-58/obj00000000000000000029 2023-05-31T20:39:13.356 DEBUG:boto:path=/pklxkd-58/obj00000000000000000029?versionId=null 2023-05-31T20:39:13.356 DEBUG:boto:auth_path=/pklxkd-58/obj00000000000000000029?versionId=null 2023-05-31T20:39:13.356 DEBUG:boto:Method: HEAD 2023-05-31T20:39:13.357 DEBUG:boto:Path: /pklxkd-58/obj00000000000000000029?versionId=null 2023-05-31T20:39:13.357 DEBUG:boto:Data: 2023-05-31T20:39:13.358 DEBUG:boto:Headers: {} 2023-05-31T20:39:13.358 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:39:13.358 DEBUG:boto:Port: 8000 2023-05-31T20:39:13.359 DEBUG:boto:Params: {} 2023-05-31T20:39:13.359 DEBUG:boto:Token: None 2023-05-31T20:39:13.359 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:39:13 GMT /pklxkd-58/obj00000000000000000029?versionId=null 2023-05-31T20:39:13.360 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:llABaOxb0D2nzuOSQ920gsqiVpo= 2023-05-31T20:39:13.360 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:39:13 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:llABaOxb0D2nzuOSQ920gsqiVpo=', 'Content-Length': '0'} 2023-05-31T20:39:13.362 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:38:20 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001c014d34c2e2ceb2-006477b071-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:39:13 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:39:13.363 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000029 2023-05-31T20:39:13.363 DEBUG:boto:path=/pklxkd-58/obj00000000000000000029 2023-05-31T20:39:13.364 DEBUG:boto:auth_path=/pklxkd-58/obj00000000000000000029 2023-05-31T20:39:13.364 DEBUG:boto:path=/pklxkd-58/obj00000000000000000029?versionId=null 2023-05-31T20:39:13.364 DEBUG:boto:auth_path=/pklxkd-58/obj00000000000000000029?versionId=null 2023-05-31T20:39:13.365 DEBUG:boto:Method: GET 2023-05-31T20:39:13.365 DEBUG:boto:Path: /pklxkd-58/obj00000000000000000029?versionId=null 2023-05-31T20:39:13.366 DEBUG:boto:Data: 2023-05-31T20:39:13.366 DEBUG:boto:Headers: {} 2023-05-31T20:39:13.367 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:39:13.367 DEBUG:boto:Port: 8001 2023-05-31T20:39:13.367 DEBUG:boto:Params: {} 2023-05-31T20:39:13.368 DEBUG:boto:Token: None 2023-05-31T20:39:13.368 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:39:13 GMT /pklxkd-58/obj00000000000000000029?versionId=null 2023-05-31T20:39:13.369 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:LBWuFcOXGF5qZKBigM0z8uC/4Uk= 2023-05-31T20:39:13.369 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:39:13 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:LBWuFcOXGF5qZKBigM0z8uC/4Uk=', 'Content-Length': '0'} 2023-05-31T20:39:13.371 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:38:20 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000930c22a2b73dce7e-006477b071-1c17-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:39:13 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:39:13.373 DEBUG:boto:path=/pklxkd-58/obj00000000000000000029 2023-05-31T20:39:13.374 DEBUG:boto:auth_path=/pklxkd-58/obj00000000000000000029 2023-05-31T20:39:13.374 DEBUG:boto:path=/pklxkd-58/obj00000000000000000029?versionId=null 2023-05-31T20:39:13.375 DEBUG:boto:auth_path=/pklxkd-58/obj00000000000000000029?versionId=null 2023-05-31T20:39:13.376 DEBUG:boto:Method: GET 2023-05-31T20:39:13.377 DEBUG:boto:Path: /pklxkd-58/obj00000000000000000029?versionId=null 2023-05-31T20:39:13.377 DEBUG:boto:Data: 2023-05-31T20:39:13.377 DEBUG:boto:Headers: {} 2023-05-31T20:39:13.378 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:39:13.378 DEBUG:boto:Port: 8000 2023-05-31T20:39:13.378 DEBUG:boto:Params: {} 2023-05-31T20:39:13.379 DEBUG:boto:Token: None 2023-05-31T20:39:13.379 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:39:13 GMT /pklxkd-58/obj00000000000000000029?versionId=null 2023-05-31T20:39:13.380 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:LBWuFcOXGF5qZKBigM0z8uC/4Uk= 2023-05-31T20:39:13.380 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:39:13 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:LBWuFcOXGF5qZKBigM0z8uC/4Uk=', 'Content-Length': '0'} 2023-05-31T20:39:13.382 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:38:20 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000fefaa2b575b9c0a4-006477b071-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:39:13 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:39:13.383 DEBUG:boto:path=/pklxkd-58/ 2023-05-31T20:39:13.384 DEBUG:boto:auth_path=/pklxkd-58/ 2023-05-31T20:39:13.384 DEBUG:boto:path=/pklxkd-58/?prefix=obj00000000000000000029 2023-05-31T20:39:13.384 DEBUG:boto:auth_path=/pklxkd-58/?prefix=obj00000000000000000029 2023-05-31T20:39:13.385 DEBUG:boto:Method: GET 2023-05-31T20:39:13.386 DEBUG:boto:Path: /pklxkd-58/?prefix=obj00000000000000000029 2023-05-31T20:39:13.386 DEBUG:boto:Data: 2023-05-31T20:39:13.386 DEBUG:boto:Headers: {} 2023-05-31T20:39:13.387 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:39:13.388 DEBUG:boto:Port: 8001 2023-05-31T20:39:13.388 DEBUG:boto:Params: {} 2023-05-31T20:39:13.389 DEBUG:boto:Token: None 2023-05-31T20:39:13.389 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:39:13 GMT /pklxkd-58/ 2023-05-31T20:39:13.390 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:GhwzgzgAYYFLDcfdK1Agett0HKk= 2023-05-31T20:39:13.391 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:39:13 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:GhwzgzgAYYFLDcfdK1Agett0HKk=', 'Content-Length': '0'} 2023-05-31T20:39:13.394 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000003fc236abd01e966a-006477b071-1c17-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:39:13 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:39:13.395 DEBUG:boto:b'pklxkd-58obj000000000000000000291000falseobj000000000000000000292023-05-31T20:38:20.265Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:39:13.397 DEBUG:boto:path=/pklxkd-58/ 2023-05-31T20:39:13.398 DEBUG:boto:auth_path=/pklxkd-58/ 2023-05-31T20:39:13.399 DEBUG:boto:path=/pklxkd-58/?prefix=obj00000000000000000029 2023-05-31T20:39:13.399 DEBUG:boto:auth_path=/pklxkd-58/?prefix=obj00000000000000000029 2023-05-31T20:39:13.400 DEBUG:boto:Method: GET 2023-05-31T20:39:13.401 DEBUG:boto:Path: /pklxkd-58/?prefix=obj00000000000000000029 2023-05-31T20:39:13.401 DEBUG:boto:Data: 2023-05-31T20:39:13.401 DEBUG:boto:Headers: {} 2023-05-31T20:39:13.402 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:39:13.402 DEBUG:boto:Port: 8000 2023-05-31T20:39:13.403 DEBUG:boto:Params: {} 2023-05-31T20:39:13.403 DEBUG:boto:Token: None 2023-05-31T20:39:13.403 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:39:13 GMT /pklxkd-58/ 2023-05-31T20:39:13.404 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:GhwzgzgAYYFLDcfdK1Agett0HKk= 2023-05-31T20:39:13.404 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:39:13 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:GhwzgzgAYYFLDcfdK1Agett0HKk=', 'Content-Length': '0'} 2023-05-31T20:39:13.408 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000adebb0d1619f10a5-006477b071-1630-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:39:13 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:39:13.409 DEBUG:boto:b'pklxkd-58obj000000000000000000291000falseobj000000000000000000292023-05-31T20:38:20.265Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:39:13.410 DEBUG:boto:path=/pklxkd-58/obj00000000000000000029 2023-05-31T20:39:13.410 DEBUG:boto:auth_path=/pklxkd-58/obj00000000000000000029 2023-05-31T20:39:13.411 DEBUG:boto:Method: HEAD 2023-05-31T20:39:13.411 DEBUG:boto:Path: /pklxkd-58/obj00000000000000000029 2023-05-31T20:39:13.411 DEBUG:boto:Data: 2023-05-31T20:39:13.412 DEBUG:boto:Headers: {} 2023-05-31T20:39:13.412 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:39:13.412 DEBUG:boto:Port: 8001 2023-05-31T20:39:13.413 DEBUG:boto:Params: {} 2023-05-31T20:39:13.413 DEBUG:boto:Token: None 2023-05-31T20:39:13.413 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:39:13 GMT /pklxkd-58/obj00000000000000000029 2023-05-31T20:39:13.414 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:TidNMTz5WwXluHaQNrVPYG+Wd9g= 2023-05-31T20:39:13.414 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:39:13 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:TidNMTz5WwXluHaQNrVPYG+Wd9g=', 'Content-Length': '0'} 2023-05-31T20:39:13.416 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:38:20 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007fda90d54d4ce99a-006477b071-1c17-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:39:13 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:39:13.417 DEBUG:boto:path=/pklxkd-58/obj00000000000000000029 2023-05-31T20:39:13.417 DEBUG:boto:auth_path=/pklxkd-58/obj00000000000000000029 2023-05-31T20:39:13.418 DEBUG:boto:Method: HEAD 2023-05-31T20:39:13.418 DEBUG:boto:Path: /pklxkd-58/obj00000000000000000029 2023-05-31T20:39:13.418 DEBUG:boto:Data: 2023-05-31T20:39:13.419 DEBUG:boto:Headers: {} 2023-05-31T20:39:13.419 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:39:13.419 DEBUG:boto:Port: 8000 2023-05-31T20:39:13.420 DEBUG:boto:Params: {} 2023-05-31T20:39:13.420 DEBUG:boto:Token: None 2023-05-31T20:39:13.420 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:39:13 GMT /pklxkd-58/obj00000000000000000029 2023-05-31T20:39:13.421 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:TidNMTz5WwXluHaQNrVPYG+Wd9g= 2023-05-31T20:39:13.421 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:39:13 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:TidNMTz5WwXluHaQNrVPYG+Wd9g=', 'Content-Length': '0'} 2023-05-31T20:39:13.423 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:38:20 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000319677c3d4c8a0d1-006477b071-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:39:13 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:39:13.424 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000029 2023-05-31T20:39:13.424 DEBUG:boto:path=/pklxkd-58/obj00000000000000000029 2023-05-31T20:39:13.425 DEBUG:boto:auth_path=/pklxkd-58/obj00000000000000000029 2023-05-31T20:39:13.425 DEBUG:boto:Method: GET 2023-05-31T20:39:13.426 DEBUG:boto:Path: /pklxkd-58/obj00000000000000000029 2023-05-31T20:39:13.426 DEBUG:boto:Data: 2023-05-31T20:39:13.426 DEBUG:boto:Headers: {} 2023-05-31T20:39:13.427 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:39:13.427 DEBUG:boto:Port: 8001 2023-05-31T20:39:13.427 DEBUG:boto:Params: {} 2023-05-31T20:39:13.428 DEBUG:boto:Token: None 2023-05-31T20:39:13.428 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:39:13 GMT /pklxkd-58/obj00000000000000000029 2023-05-31T20:39:13.429 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:2VyrIZusxTRXRmtJ6bVpuZ6GVmU= 2023-05-31T20:39:13.429 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:39:13 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:2VyrIZusxTRXRmtJ6bVpuZ6GVmU=', 'Content-Length': '0'} 2023-05-31T20:39:13.433 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:38:20 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003a4a4fbf604acaf1-006477b071-1c17-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:39:13 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:39:13.472 DEBUG:boto:path=/pklxkd-58/obj00000000000000000029 2023-05-31T20:39:13.473 DEBUG:boto:auth_path=/pklxkd-58/obj00000000000000000029 2023-05-31T20:39:13.473 DEBUG:boto:Method: GET 2023-05-31T20:39:13.474 DEBUG:boto:Path: /pklxkd-58/obj00000000000000000029 2023-05-31T20:39:13.474 DEBUG:boto:Data: 2023-05-31T20:39:13.475 DEBUG:boto:Headers: {} 2023-05-31T20:39:13.475 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:39:13.475 DEBUG:boto:Port: 8000 2023-05-31T20:39:13.476 DEBUG:boto:Params: {} 2023-05-31T20:39:13.476 DEBUG:boto:Token: None 2023-05-31T20:39:13.477 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:39:13 GMT /pklxkd-58/obj00000000000000000029 2023-05-31T20:39:13.477 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:2VyrIZusxTRXRmtJ6bVpuZ6GVmU= 2023-05-31T20:39:13.478 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:39:13 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:2VyrIZusxTRXRmtJ6bVpuZ6GVmU=', 'Content-Length': '0'} 2023-05-31T20:39:13.480 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:38:20 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a843339d86d93dcf-006477b071-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:39:13 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:39:13.480 DEBUG:boto:path=/pklxkd-58/ 2023-05-31T20:39:13.481 DEBUG:boto:auth_path=/pklxkd-58/ 2023-05-31T20:39:13.481 DEBUG:boto:path=/pklxkd-58/?prefix=obj00000000000000000029 2023-05-31T20:39:13.481 DEBUG:boto:auth_path=/pklxkd-58/?prefix=obj00000000000000000029 2023-05-31T20:39:13.482 DEBUG:boto:Method: GET 2023-05-31T20:39:13.482 DEBUG:boto:Path: /pklxkd-58/?prefix=obj00000000000000000029 2023-05-31T20:39:13.483 DEBUG:boto:Data: 2023-05-31T20:39:13.483 DEBUG:boto:Headers: {} 2023-05-31T20:39:13.483 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:39:13.484 DEBUG:boto:Port: 8001 2023-05-31T20:39:13.484 DEBUG:boto:Params: {} 2023-05-31T20:39:13.484 DEBUG:boto:Token: None 2023-05-31T20:39:13.485 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:39:13 GMT /pklxkd-58/ 2023-05-31T20:39:13.485 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:GhwzgzgAYYFLDcfdK1Agett0HKk= 2023-05-31T20:39:13.486 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:39:13 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:GhwzgzgAYYFLDcfdK1Agett0HKk=', 'Content-Length': '0'} 2023-05-31T20:39:13.488 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a94640c561768469-006477b071-1c17-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:39:13 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:39:13.489 DEBUG:boto:b'pklxkd-58obj000000000000000000291000falseobj000000000000000000292023-05-31T20:38:20.265Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:39:13.490 DEBUG:boto:path=/pklxkd-58/ 2023-05-31T20:39:13.490 DEBUG:boto:auth_path=/pklxkd-58/ 2023-05-31T20:39:13.490 DEBUG:boto:path=/pklxkd-58/?prefix=obj00000000000000000029 2023-05-31T20:39:13.491 DEBUG:boto:auth_path=/pklxkd-58/?prefix=obj00000000000000000029 2023-05-31T20:39:13.491 DEBUG:boto:Method: GET 2023-05-31T20:39:13.492 DEBUG:boto:Path: /pklxkd-58/?prefix=obj00000000000000000029 2023-05-31T20:39:13.492 DEBUG:boto:Data: 2023-05-31T20:39:13.492 DEBUG:boto:Headers: {} 2023-05-31T20:39:13.493 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:39:13.493 DEBUG:boto:Port: 8000 2023-05-31T20:39:13.493 DEBUG:boto:Params: {} 2023-05-31T20:39:13.494 DEBUG:boto:Token: None 2023-05-31T20:39:13.494 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:39:13 GMT /pklxkd-58/ 2023-05-31T20:39:13.495 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:GhwzgzgAYYFLDcfdK1Agett0HKk= 2023-05-31T20:39:13.495 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:39:13 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:GhwzgzgAYYFLDcfdK1Agett0HKk=', 'Content-Length': '0'} 2023-05-31T20:39:13.499 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000000eee84385e9f09e-006477b071-1630-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:39:13 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:39:13.500 DEBUG:boto:b'pklxkd-58obj000000000000000000291000falseobj000000000000000000292023-05-31T20:38:20.265Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:39:13.500 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000030 2023-05-31T20:39:13.501 DEBUG:boto:path=/pklxkd-58/obj00000000000000000030 2023-05-31T20:39:13.501 DEBUG:boto:auth_path=/pklxkd-58/obj00000000000000000030 2023-05-31T20:39:13.502 DEBUG:boto:path=/pklxkd-58/obj00000000000000000030?versionId=null 2023-05-31T20:39:13.502 DEBUG:boto:auth_path=/pklxkd-58/obj00000000000000000030?versionId=null 2023-05-31T20:39:13.502 DEBUG:boto:Method: GET 2023-05-31T20:39:13.503 DEBUG:boto:Path: /pklxkd-58/obj00000000000000000030?versionId=null 2023-05-31T20:39:13.503 DEBUG:boto:Data: 2023-05-31T20:39:13.503 DEBUG:boto:Headers: {} 2023-05-31T20:39:13.504 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:39:13.504 DEBUG:boto:Port: 8001 2023-05-31T20:39:13.504 DEBUG:boto:Params: {} 2023-05-31T20:39:13.505 DEBUG:boto:Token: None 2023-05-31T20:39:13.505 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:39:13 GMT /pklxkd-58/obj00000000000000000030?versionId=null 2023-05-31T20:39:13.506 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:DZRxyMukPe5/qC5ACVKhqMqxEhw= 2023-05-31T20:39:13.506 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:39:13 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:DZRxyMukPe5/qC5ACVKhqMqxEhw=', 'Content-Length': '0'} 2023-05-31T20:39:13.509 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:38:21 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f7f1d507ebee7cc6-006477b071-1c17-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:39:13 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:39:13.510 DEBUG:boto:path=/pklxkd-58/obj00000000000000000030 2023-05-31T20:39:13.510 DEBUG:boto:auth_path=/pklxkd-58/obj00000000000000000030 2023-05-31T20:39:13.511 DEBUG:boto:path=/pklxkd-58/obj00000000000000000030?versionId=null 2023-05-31T20:39:13.511 DEBUG:boto:auth_path=/pklxkd-58/obj00000000000000000030?versionId=null 2023-05-31T20:39:13.512 DEBUG:boto:Method: GET 2023-05-31T20:39:13.512 DEBUG:boto:Path: /pklxkd-58/obj00000000000000000030?versionId=null 2023-05-31T20:39:13.512 DEBUG:boto:Data: 2023-05-31T20:39:13.513 DEBUG:boto:Headers: {} 2023-05-31T20:39:13.513 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:39:13.514 DEBUG:boto:Port: 8000 2023-05-31T20:39:13.514 DEBUG:boto:Params: {} 2023-05-31T20:39:13.514 DEBUG:boto:Token: None 2023-05-31T20:39:13.515 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:39:13 GMT /pklxkd-58/obj00000000000000000030?versionId=null 2023-05-31T20:39:13.515 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:DZRxyMukPe5/qC5ACVKhqMqxEhw= 2023-05-31T20:39:13.516 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:39:13 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:DZRxyMukPe5/qC5ACVKhqMqxEhw=', 'Content-Length': '0'} 2023-05-31T20:39:13.518 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:38:21 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ee3b22cf9fec04fa-006477b071-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:39:13 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:39:13.560 DEBUG:boto:path=/pklxkd-58/obj00000000000000000030 2023-05-31T20:39:13.561 DEBUG:boto:auth_path=/pklxkd-58/obj00000000000000000030 2023-05-31T20:39:13.561 DEBUG:boto:path=/pklxkd-58/obj00000000000000000030?versionId=null 2023-05-31T20:39:13.562 DEBUG:boto:auth_path=/pklxkd-58/obj00000000000000000030?versionId=null 2023-05-31T20:39:13.562 DEBUG:boto:Method: HEAD 2023-05-31T20:39:13.562 DEBUG:boto:Path: /pklxkd-58/obj00000000000000000030?versionId=null 2023-05-31T20:39:13.563 DEBUG:boto:Data: 2023-05-31T20:39:13.563 DEBUG:boto:Headers: {} 2023-05-31T20:39:13.563 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:39:13.564 DEBUG:boto:Port: 8001 2023-05-31T20:39:13.564 DEBUG:boto:Params: {} 2023-05-31T20:39:13.565 DEBUG:boto:Token: None 2023-05-31T20:39:13.565 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:39:13 GMT /pklxkd-58/obj00000000000000000030?versionId=null 2023-05-31T20:39:13.565 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:YPeBKAHbVWxYbtQ8lVeyZA+y+9c= 2023-05-31T20:39:13.566 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:39:13 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:YPeBKAHbVWxYbtQ8lVeyZA+y+9c=', 'Content-Length': '0'} 2023-05-31T20:39:13.568 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:38:21 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006890c2e84d96c28f-006477b071-1c17-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:39:13 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:39:13.569 DEBUG:boto:path=/pklxkd-58/obj00000000000000000030 2023-05-31T20:39:13.569 DEBUG:boto:auth_path=/pklxkd-58/obj00000000000000000030 2023-05-31T20:39:13.569 DEBUG:boto:path=/pklxkd-58/obj00000000000000000030?versionId=null 2023-05-31T20:39:13.570 DEBUG:boto:auth_path=/pklxkd-58/obj00000000000000000030?versionId=null 2023-05-31T20:39:13.570 DEBUG:boto:Method: HEAD 2023-05-31T20:39:13.571 DEBUG:boto:Path: /pklxkd-58/obj00000000000000000030?versionId=null 2023-05-31T20:39:13.571 DEBUG:boto:Data: 2023-05-31T20:39:13.571 DEBUG:boto:Headers: {} 2023-05-31T20:39:13.572 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:39:13.572 DEBUG:boto:Port: 8000 2023-05-31T20:39:13.572 DEBUG:boto:Params: {} 2023-05-31T20:39:13.573 DEBUG:boto:Token: None 2023-05-31T20:39:13.573 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:39:13 GMT /pklxkd-58/obj00000000000000000030?versionId=null 2023-05-31T20:39:13.574 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:YPeBKAHbVWxYbtQ8lVeyZA+y+9c= 2023-05-31T20:39:13.574 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:39:13 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:YPeBKAHbVWxYbtQ8lVeyZA+y+9c=', 'Content-Length': '0'} 2023-05-31T20:39:13.576 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:38:21 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a626f522f6b67cbc-006477b071-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:39:13 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:39:13.576 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000030 2023-05-31T20:39:13.577 DEBUG:boto:path=/pklxkd-58/obj00000000000000000030 2023-05-31T20:39:13.577 DEBUG:boto:auth_path=/pklxkd-58/obj00000000000000000030 2023-05-31T20:39:13.578 DEBUG:boto:path=/pklxkd-58/obj00000000000000000030?versionId=null 2023-05-31T20:39:13.578 DEBUG:boto:auth_path=/pklxkd-58/obj00000000000000000030?versionId=null 2023-05-31T20:39:13.578 DEBUG:boto:Method: GET 2023-05-31T20:39:13.579 DEBUG:boto:Path: /pklxkd-58/obj00000000000000000030?versionId=null 2023-05-31T20:39:13.579 DEBUG:boto:Data: 2023-05-31T20:39:13.579 DEBUG:boto:Headers: {} 2023-05-31T20:39:13.580 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:39:13.580 DEBUG:boto:Port: 8001 2023-05-31T20:39:13.581 DEBUG:boto:Params: {} 2023-05-31T20:39:13.581 DEBUG:boto:Token: None 2023-05-31T20:39:13.581 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:39:13 GMT /pklxkd-58/obj00000000000000000030?versionId=null 2023-05-31T20:39:13.582 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:DZRxyMukPe5/qC5ACVKhqMqxEhw= 2023-05-31T20:39:13.582 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:39:13 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:DZRxyMukPe5/qC5ACVKhqMqxEhw=', 'Content-Length': '0'} 2023-05-31T20:39:13.584 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:38:21 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000553439795a83279d-006477b071-1c17-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:39:13 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:39:13.585 DEBUG:boto:path=/pklxkd-58/obj00000000000000000030 2023-05-31T20:39:13.586 DEBUG:boto:auth_path=/pklxkd-58/obj00000000000000000030 2023-05-31T20:39:13.586 DEBUG:boto:path=/pklxkd-58/obj00000000000000000030?versionId=null 2023-05-31T20:39:13.586 DEBUG:boto:auth_path=/pklxkd-58/obj00000000000000000030?versionId=null 2023-05-31T20:39:13.587 DEBUG:boto:Method: GET 2023-05-31T20:39:13.587 DEBUG:boto:Path: /pklxkd-58/obj00000000000000000030?versionId=null 2023-05-31T20:39:13.587 DEBUG:boto:Data: 2023-05-31T20:39:13.588 DEBUG:boto:Headers: {} 2023-05-31T20:39:13.588 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:39:13.589 DEBUG:boto:Port: 8000 2023-05-31T20:39:13.589 DEBUG:boto:Params: {} 2023-05-31T20:39:13.589 DEBUG:boto:Token: None 2023-05-31T20:39:13.590 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:39:13 GMT /pklxkd-58/obj00000000000000000030?versionId=null 2023-05-31T20:39:13.590 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:DZRxyMukPe5/qC5ACVKhqMqxEhw= 2023-05-31T20:39:13.591 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:39:13 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:DZRxyMukPe5/qC5ACVKhqMqxEhw=', 'Content-Length': '0'} 2023-05-31T20:39:13.593 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:38:21 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000047556e72563792b7-006477b071-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:39:13 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:39:13.593 DEBUG:boto:path=/pklxkd-58/ 2023-05-31T20:39:13.594 DEBUG:boto:auth_path=/pklxkd-58/ 2023-05-31T20:39:13.594 DEBUG:boto:path=/pklxkd-58/?prefix=obj00000000000000000030 2023-05-31T20:39:13.595 DEBUG:boto:auth_path=/pklxkd-58/?prefix=obj00000000000000000030 2023-05-31T20:39:13.595 DEBUG:boto:Method: GET 2023-05-31T20:39:13.595 DEBUG:boto:Path: /pklxkd-58/?prefix=obj00000000000000000030 2023-05-31T20:39:13.596 DEBUG:boto:Data: 2023-05-31T20:39:13.596 DEBUG:boto:Headers: {} 2023-05-31T20:39:13.596 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:39:13.597 DEBUG:boto:Port: 8001 2023-05-31T20:39:13.597 DEBUG:boto:Params: {} 2023-05-31T20:39:13.598 DEBUG:boto:Token: None 2023-05-31T20:39:13.598 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:39:13 GMT /pklxkd-58/ 2023-05-31T20:39:13.598 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:GhwzgzgAYYFLDcfdK1Agett0HKk= 2023-05-31T20:39:13.599 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:39:13 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:GhwzgzgAYYFLDcfdK1Agett0HKk=', 'Content-Length': '0'} 2023-05-31T20:39:13.601 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000014c0b7841d9def17-006477b071-1c17-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:39:13 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:39:13.644 DEBUG:boto:b'pklxkd-58obj000000000000000000301000falseobj000000000000000000302023-05-31T20:38:21.641Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:39:13.645 DEBUG:boto:path=/pklxkd-58/ 2023-05-31T20:39:13.646 DEBUG:boto:auth_path=/pklxkd-58/ 2023-05-31T20:39:13.646 DEBUG:boto:path=/pklxkd-58/?prefix=obj00000000000000000030 2023-05-31T20:39:13.646 DEBUG:boto:auth_path=/pklxkd-58/?prefix=obj00000000000000000030 2023-05-31T20:39:13.647 DEBUG:boto:Method: GET 2023-05-31T20:39:13.647 DEBUG:boto:Path: /pklxkd-58/?prefix=obj00000000000000000030 2023-05-31T20:39:13.647 DEBUG:boto:Data: 2023-05-31T20:39:13.648 DEBUG:boto:Headers: {} 2023-05-31T20:39:13.648 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:39:13.649 DEBUG:boto:Port: 8000 2023-05-31T20:39:13.649 DEBUG:boto:Params: {} 2023-05-31T20:39:13.649 DEBUG:boto:Token: None 2023-05-31T20:39:13.650 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:39:13 GMT /pklxkd-58/ 2023-05-31T20:39:13.650 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:GhwzgzgAYYFLDcfdK1Agett0HKk= 2023-05-31T20:39:13.651 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:39:13 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:GhwzgzgAYYFLDcfdK1Agett0HKk=', 'Content-Length': '0'} 2023-05-31T20:39:13.655 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f7c9b302d6ec182b-006477b071-1630-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:39:13 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:39:13.656 DEBUG:boto:b'pklxkd-58obj000000000000000000301000falseobj000000000000000000302023-05-31T20:38:21.641Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:39:13.656 DEBUG:boto:path=/pklxkd-58/obj00000000000000000030 2023-05-31T20:39:13.657 DEBUG:boto:auth_path=/pklxkd-58/obj00000000000000000030 2023-05-31T20:39:13.657 DEBUG:boto:Method: HEAD 2023-05-31T20:39:13.657 DEBUG:boto:Path: /pklxkd-58/obj00000000000000000030 2023-05-31T20:39:13.658 DEBUG:boto:Data: 2023-05-31T20:39:13.658 DEBUG:boto:Headers: {} 2023-05-31T20:39:13.658 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:39:13.659 DEBUG:boto:Port: 8001 2023-05-31T20:39:13.659 DEBUG:boto:Params: {} 2023-05-31T20:39:13.659 DEBUG:boto:Token: None 2023-05-31T20:39:13.661 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:39:13 GMT /pklxkd-58/obj00000000000000000030 2023-05-31T20:39:13.661 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:+km8zAYdAF2GSXSSDBK8DFrst1U= 2023-05-31T20:39:13.662 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:39:13 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:+km8zAYdAF2GSXSSDBK8DFrst1U=', 'Content-Length': '0'} 2023-05-31T20:39:13.663 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:38:21 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a53d3bf3196363ec-006477b071-1c17-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:39:13 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:39:13.664 DEBUG:boto:path=/pklxkd-58/obj00000000000000000030 2023-05-31T20:39:13.664 DEBUG:boto:auth_path=/pklxkd-58/obj00000000000000000030 2023-05-31T20:39:13.665 DEBUG:boto:Method: HEAD 2023-05-31T20:39:13.666 DEBUG:boto:Path: /pklxkd-58/obj00000000000000000030 2023-05-31T20:39:13.666 DEBUG:boto:Data: 2023-05-31T20:39:13.666 DEBUG:boto:Headers: {} 2023-05-31T20:39:13.667 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:39:13.667 DEBUG:boto:Port: 8000 2023-05-31T20:39:13.667 DEBUG:boto:Params: {} 2023-05-31T20:39:13.668 DEBUG:boto:Token: None 2023-05-31T20:39:13.668 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:39:13 GMT /pklxkd-58/obj00000000000000000030 2023-05-31T20:39:13.669 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:+km8zAYdAF2GSXSSDBK8DFrst1U= 2023-05-31T20:39:13.669 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:39:13 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:+km8zAYdAF2GSXSSDBK8DFrst1U=', 'Content-Length': '0'} 2023-05-31T20:39:13.671 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:38:21 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000eb72a7213aadf62e-006477b071-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:39:13 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:39:13.672 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000030 2023-05-31T20:39:13.672 DEBUG:boto:path=/pklxkd-58/obj00000000000000000030 2023-05-31T20:39:13.672 DEBUG:boto:auth_path=/pklxkd-58/obj00000000000000000030 2023-05-31T20:39:13.673 DEBUG:boto:Method: GET 2023-05-31T20:39:13.674 DEBUG:boto:Path: /pklxkd-58/obj00000000000000000030 2023-05-31T20:39:13.674 DEBUG:boto:Data: 2023-05-31T20:39:13.675 DEBUG:boto:Headers: {} 2023-05-31T20:39:13.675 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:39:13.676 DEBUG:boto:Port: 8001 2023-05-31T20:39:13.676 DEBUG:boto:Params: {} 2023-05-31T20:39:13.677 DEBUG:boto:Token: None 2023-05-31T20:39:13.678 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:39:13 GMT /pklxkd-58/obj00000000000000000030 2023-05-31T20:39:13.678 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:+mppvtXDOD2w29KbOWWuhWZWAIc= 2023-05-31T20:39:13.679 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:39:13 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:+mppvtXDOD2w29KbOWWuhWZWAIc=', 'Content-Length': '0'} 2023-05-31T20:39:13.681 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:38:21 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000cf291c1beb47d913-006477b071-1c17-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:39:13 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:39:13.682 DEBUG:boto:path=/pklxkd-58/obj00000000000000000030 2023-05-31T20:39:13.682 DEBUG:boto:auth_path=/pklxkd-58/obj00000000000000000030 2023-05-31T20:39:13.683 DEBUG:boto:Method: GET 2023-05-31T20:39:13.683 DEBUG:boto:Path: /pklxkd-58/obj00000000000000000030 2023-05-31T20:39:13.684 DEBUG:boto:Data: 2023-05-31T20:39:13.684 DEBUG:boto:Headers: {} 2023-05-31T20:39:13.684 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:39:13.685 DEBUG:boto:Port: 8000 2023-05-31T20:39:13.685 DEBUG:boto:Params: {} 2023-05-31T20:39:13.686 DEBUG:boto:Token: None 2023-05-31T20:39:13.686 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:39:13 GMT /pklxkd-58/obj00000000000000000030 2023-05-31T20:39:13.686 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:+mppvtXDOD2w29KbOWWuhWZWAIc= 2023-05-31T20:39:13.687 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:39:13 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:+mppvtXDOD2w29KbOWWuhWZWAIc=', 'Content-Length': '0'} 2023-05-31T20:39:13.689 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:38:21 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a503cf202619f5b3-006477b071-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:39:13 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:39:13.690 DEBUG:boto:path=/pklxkd-58/ 2023-05-31T20:39:13.691 DEBUG:boto:auth_path=/pklxkd-58/ 2023-05-31T20:39:13.691 DEBUG:boto:path=/pklxkd-58/?prefix=obj00000000000000000030 2023-05-31T20:39:13.691 DEBUG:boto:auth_path=/pklxkd-58/?prefix=obj00000000000000000030 2023-05-31T20:39:13.692 DEBUG:boto:Method: GET 2023-05-31T20:39:13.692 DEBUG:boto:Path: /pklxkd-58/?prefix=obj00000000000000000030 2023-05-31T20:39:13.692 DEBUG:boto:Data: 2023-05-31T20:39:13.693 DEBUG:boto:Headers: {} 2023-05-31T20:39:13.693 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:39:13.694 DEBUG:boto:Port: 8001 2023-05-31T20:39:13.694 DEBUG:boto:Params: {} 2023-05-31T20:39:13.694 DEBUG:boto:Token: None 2023-05-31T20:39:13.695 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:39:13 GMT /pklxkd-58/ 2023-05-31T20:39:13.695 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:GhwzgzgAYYFLDcfdK1Agett0HKk= 2023-05-31T20:39:13.696 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:39:13 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:GhwzgzgAYYFLDcfdK1Agett0HKk=', 'Content-Length': '0'} 2023-05-31T20:39:13.705 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000434d34e0f661b7ab-006477b071-1c17-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:39:13 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:39:13.712 DEBUG:boto:b'pklxkd-58obj000000000000000000301000falseobj000000000000000000302023-05-31T20:38:21.641Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:39:13.714 DEBUG:boto:path=/pklxkd-58/ 2023-05-31T20:39:13.714 DEBUG:boto:auth_path=/pklxkd-58/ 2023-05-31T20:39:13.715 DEBUG:boto:path=/pklxkd-58/?prefix=obj00000000000000000030 2023-05-31T20:39:13.716 DEBUG:boto:auth_path=/pklxkd-58/?prefix=obj00000000000000000030 2023-05-31T20:39:13.717 DEBUG:boto:Method: GET 2023-05-31T20:39:13.717 DEBUG:boto:Path: /pklxkd-58/?prefix=obj00000000000000000030 2023-05-31T20:39:13.718 DEBUG:boto:Data: 2023-05-31T20:39:13.719 DEBUG:boto:Headers: {} 2023-05-31T20:39:13.719 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:39:13.720 DEBUG:boto:Port: 8000 2023-05-31T20:39:13.721 DEBUG:boto:Params: {} 2023-05-31T20:39:13.721 DEBUG:boto:Token: None 2023-05-31T20:39:13.722 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:39:13 GMT /pklxkd-58/ 2023-05-31T20:39:13.723 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:GhwzgzgAYYFLDcfdK1Agett0HKk= 2023-05-31T20:39:13.724 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:39:13 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:GhwzgzgAYYFLDcfdK1Agett0HKk=', 'Content-Length': '0'} 2023-05-31T20:39:13.731 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e13dd9b0c249f7b1-006477b071-1630-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:39:13 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:39:13.772 DEBUG:boto:b'pklxkd-58obj000000000000000000301000falseobj000000000000000000302023-05-31T20:38:21.641Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:39:13.773 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000031 2023-05-31T20:39:13.774 DEBUG:boto:path=/pklxkd-58/obj00000000000000000031 2023-05-31T20:39:13.774 DEBUG:boto:auth_path=/pklxkd-58/obj00000000000000000031 2023-05-31T20:39:13.775 DEBUG:boto:path=/pklxkd-58/obj00000000000000000031?versionId=null 2023-05-31T20:39:13.775 DEBUG:boto:auth_path=/pklxkd-58/obj00000000000000000031?versionId=null 2023-05-31T20:39:13.775 DEBUG:boto:Method: GET 2023-05-31T20:39:13.776 DEBUG:boto:Path: /pklxkd-58/obj00000000000000000031?versionId=null 2023-05-31T20:39:13.776 DEBUG:boto:Data: 2023-05-31T20:39:13.777 DEBUG:boto:Headers: {} 2023-05-31T20:39:13.777 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:39:13.777 DEBUG:boto:Port: 8001 2023-05-31T20:39:13.778 DEBUG:boto:Params: {} 2023-05-31T20:39:13.778 DEBUG:boto:Token: None 2023-05-31T20:39:13.779 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:39:13 GMT /pklxkd-58/obj00000000000000000031?versionId=null 2023-05-31T20:39:13.779 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:lGyPBaxEOpyRp9LTvL+j1Mnjn1U= 2023-05-31T20:39:13.780 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:39:13 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:lGyPBaxEOpyRp9LTvL+j1Mnjn1U=', 'Content-Length': '0'} 2023-05-31T20:39:13.784 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:38:18 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008fc1b289ac5c573e-006477b071-1c17-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:39:13 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:39:13.785 DEBUG:boto:path=/pklxkd-58/obj00000000000000000031 2023-05-31T20:39:13.786 DEBUG:boto:auth_path=/pklxkd-58/obj00000000000000000031 2023-05-31T20:39:13.787 DEBUG:boto:path=/pklxkd-58/obj00000000000000000031?versionId=null 2023-05-31T20:39:13.787 DEBUG:boto:auth_path=/pklxkd-58/obj00000000000000000031?versionId=null 2023-05-31T20:39:13.787 DEBUG:boto:Method: GET 2023-05-31T20:39:13.788 DEBUG:boto:Path: /pklxkd-58/obj00000000000000000031?versionId=null 2023-05-31T20:39:13.788 DEBUG:boto:Data: 2023-05-31T20:39:13.789 DEBUG:boto:Headers: {} 2023-05-31T20:39:13.789 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:39:13.789 DEBUG:boto:Port: 8000 2023-05-31T20:39:13.791 DEBUG:boto:Params: {} 2023-05-31T20:39:13.791 DEBUG:boto:Token: None 2023-05-31T20:39:13.792 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:39:13 GMT /pklxkd-58/obj00000000000000000031?versionId=null 2023-05-31T20:39:13.792 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:lGyPBaxEOpyRp9LTvL+j1Mnjn1U= 2023-05-31T20:39:13.793 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:39:13 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:lGyPBaxEOpyRp9LTvL+j1Mnjn1U=', 'Content-Length': '0'} 2023-05-31T20:39:13.795 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:38:18 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003a77e50f40dfe440-006477b071-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:39:13 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:39:13.797 DEBUG:boto:path=/pklxkd-58/obj00000000000000000031 2023-05-31T20:39:13.797 DEBUG:boto:auth_path=/pklxkd-58/obj00000000000000000031 2023-05-31T20:39:13.798 DEBUG:boto:path=/pklxkd-58/obj00000000000000000031?versionId=null 2023-05-31T20:39:13.798 DEBUG:boto:auth_path=/pklxkd-58/obj00000000000000000031?versionId=null 2023-05-31T20:39:13.799 DEBUG:boto:Method: HEAD 2023-05-31T20:39:13.799 DEBUG:boto:Path: /pklxkd-58/obj00000000000000000031?versionId=null 2023-05-31T20:39:13.799 DEBUG:boto:Data: 2023-05-31T20:39:13.800 DEBUG:boto:Headers: {} 2023-05-31T20:39:13.800 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:39:13.801 DEBUG:boto:Port: 8001 2023-05-31T20:39:13.801 DEBUG:boto:Params: {} 2023-05-31T20:39:13.801 DEBUG:boto:Token: None 2023-05-31T20:39:13.802 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:39:13 GMT /pklxkd-58/obj00000000000000000031?versionId=null 2023-05-31T20:39:13.802 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:gJ3PTGfFckv9a8riM/5UR+w92Bg= 2023-05-31T20:39:13.802 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:39:13 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:gJ3PTGfFckv9a8riM/5UR+w92Bg=', 'Content-Length': '0'} 2023-05-31T20:39:13.805 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:38:18 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d7daa0c9457d04a5-006477b071-1c17-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:39:13 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:39:13.806 DEBUG:boto:path=/pklxkd-58/obj00000000000000000031 2023-05-31T20:39:13.806 DEBUG:boto:auth_path=/pklxkd-58/obj00000000000000000031 2023-05-31T20:39:13.806 DEBUG:boto:path=/pklxkd-58/obj00000000000000000031?versionId=null 2023-05-31T20:39:13.807 DEBUG:boto:auth_path=/pklxkd-58/obj00000000000000000031?versionId=null 2023-05-31T20:39:13.807 DEBUG:boto:Method: HEAD 2023-05-31T20:39:13.808 DEBUG:boto:Path: /pklxkd-58/obj00000000000000000031?versionId=null 2023-05-31T20:39:13.808 DEBUG:boto:Data: 2023-05-31T20:39:13.809 DEBUG:boto:Headers: {} 2023-05-31T20:39:13.809 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:39:13.809 DEBUG:boto:Port: 8000 2023-05-31T20:39:13.810 DEBUG:boto:Params: {} 2023-05-31T20:39:13.810 DEBUG:boto:Token: None 2023-05-31T20:39:13.810 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:39:13 GMT /pklxkd-58/obj00000000000000000031?versionId=null 2023-05-31T20:39:13.811 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:gJ3PTGfFckv9a8riM/5UR+w92Bg= 2023-05-31T20:39:13.811 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:39:13 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:gJ3PTGfFckv9a8riM/5UR+w92Bg=', 'Content-Length': '0'} 2023-05-31T20:39:13.814 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:38:18 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004bf4cd40b9ccaaa6-006477b071-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:39:13 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:39:13.815 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000031 2023-05-31T20:39:13.815 DEBUG:boto:path=/pklxkd-58/obj00000000000000000031 2023-05-31T20:39:13.816 DEBUG:boto:auth_path=/pklxkd-58/obj00000000000000000031 2023-05-31T20:39:13.816 DEBUG:boto:path=/pklxkd-58/obj00000000000000000031?versionId=null 2023-05-31T20:39:13.817 DEBUG:boto:auth_path=/pklxkd-58/obj00000000000000000031?versionId=null 2023-05-31T20:39:13.817 DEBUG:boto:Method: GET 2023-05-31T20:39:13.818 DEBUG:boto:Path: /pklxkd-58/obj00000000000000000031?versionId=null 2023-05-31T20:39:13.818 DEBUG:boto:Data: 2023-05-31T20:39:13.818 DEBUG:boto:Headers: {} 2023-05-31T20:39:13.819 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:39:13.819 DEBUG:boto:Port: 8001 2023-05-31T20:39:13.820 DEBUG:boto:Params: {} 2023-05-31T20:39:13.820 DEBUG:boto:Token: None 2023-05-31T20:39:13.821 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:39:13 GMT /pklxkd-58/obj00000000000000000031?versionId=null 2023-05-31T20:39:13.821 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:lGyPBaxEOpyRp9LTvL+j1Mnjn1U= 2023-05-31T20:39:13.823 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:39:13 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:lGyPBaxEOpyRp9LTvL+j1Mnjn1U=', 'Content-Length': '0'} 2023-05-31T20:39:13.825 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:38:18 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008af0e81bf06bc627-006477b071-1c17-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:39:13 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:39:13.868 DEBUG:boto:path=/pklxkd-58/obj00000000000000000031 2023-05-31T20:39:13.869 DEBUG:boto:auth_path=/pklxkd-58/obj00000000000000000031 2023-05-31T20:39:13.869 DEBUG:boto:path=/pklxkd-58/obj00000000000000000031?versionId=null 2023-05-31T20:39:13.870 DEBUG:boto:auth_path=/pklxkd-58/obj00000000000000000031?versionId=null 2023-05-31T20:39:13.870 DEBUG:boto:Method: GET 2023-05-31T20:39:13.871 DEBUG:boto:Path: /pklxkd-58/obj00000000000000000031?versionId=null 2023-05-31T20:39:13.871 DEBUG:boto:Data: 2023-05-31T20:39:13.871 DEBUG:boto:Headers: {} 2023-05-31T20:39:13.872 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:39:13.872 DEBUG:boto:Port: 8000 2023-05-31T20:39:13.872 DEBUG:boto:Params: {} 2023-05-31T20:39:13.873 DEBUG:boto:Token: None 2023-05-31T20:39:13.873 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:39:13 GMT /pklxkd-58/obj00000000000000000031?versionId=null 2023-05-31T20:39:13.874 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:lGyPBaxEOpyRp9LTvL+j1Mnjn1U= 2023-05-31T20:39:13.875 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:39:13 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:lGyPBaxEOpyRp9LTvL+j1Mnjn1U=', 'Content-Length': '0'} 2023-05-31T20:39:13.877 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:38:18 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000273ca43cdf931a87-006477b071-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:39:13 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:39:13.878 DEBUG:boto:path=/pklxkd-58/ 2023-05-31T20:39:13.879 DEBUG:boto:auth_path=/pklxkd-58/ 2023-05-31T20:39:13.879 DEBUG:boto:path=/pklxkd-58/?prefix=obj00000000000000000031 2023-05-31T20:39:13.880 DEBUG:boto:auth_path=/pklxkd-58/?prefix=obj00000000000000000031 2023-05-31T20:39:13.880 DEBUG:boto:Method: GET 2023-05-31T20:39:13.880 DEBUG:boto:Path: /pklxkd-58/?prefix=obj00000000000000000031 2023-05-31T20:39:13.881 DEBUG:boto:Data: 2023-05-31T20:39:13.881 DEBUG:boto:Headers: {} 2023-05-31T20:39:13.882 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:39:13.882 DEBUG:boto:Port: 8001 2023-05-31T20:39:13.882 DEBUG:boto:Params: {} 2023-05-31T20:39:13.883 DEBUG:boto:Token: None 2023-05-31T20:39:13.883 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:39:13 GMT /pklxkd-58/ 2023-05-31T20:39:13.884 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:GhwzgzgAYYFLDcfdK1Agett0HKk= 2023-05-31T20:39:13.884 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:39:13 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:GhwzgzgAYYFLDcfdK1Agett0HKk=', 'Content-Length': '0'} 2023-05-31T20:39:13.887 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e40a7bb448589a13-006477b071-1c17-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:39:13 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:39:13.888 DEBUG:boto:b'pklxkd-58obj000000000000000000311000falseobj000000000000000000312023-05-31T20:38:18.440Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:39:13.889 DEBUG:boto:path=/pklxkd-58/ 2023-05-31T20:39:13.890 DEBUG:boto:auth_path=/pklxkd-58/ 2023-05-31T20:39:13.890 DEBUG:boto:path=/pklxkd-58/?prefix=obj00000000000000000031 2023-05-31T20:39:13.891 DEBUG:boto:auth_path=/pklxkd-58/?prefix=obj00000000000000000031 2023-05-31T20:39:13.891 DEBUG:boto:Method: GET 2023-05-31T20:39:13.892 DEBUG:boto:Path: /pklxkd-58/?prefix=obj00000000000000000031 2023-05-31T20:39:13.892 DEBUG:boto:Data: 2023-05-31T20:39:13.893 DEBUG:boto:Headers: {} 2023-05-31T20:39:13.893 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:39:13.894 DEBUG:boto:Port: 8000 2023-05-31T20:39:13.894 DEBUG:boto:Params: {} 2023-05-31T20:39:13.894 DEBUG:boto:Token: None 2023-05-31T20:39:13.895 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:39:13 GMT /pklxkd-58/ 2023-05-31T20:39:13.896 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:GhwzgzgAYYFLDcfdK1Agett0HKk= 2023-05-31T20:39:13.896 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:39:13 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:GhwzgzgAYYFLDcfdK1Agett0HKk=', 'Content-Length': '0'} 2023-05-31T20:39:13.900 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000009fffdf3e2f39070-006477b071-1630-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:39:13 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:39:13.901 DEBUG:boto:b'pklxkd-58obj000000000000000000311000falseobj000000000000000000312023-05-31T20:38:18.440Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:39:13.902 DEBUG:boto:path=/pklxkd-58/obj00000000000000000031 2023-05-31T20:39:13.903 DEBUG:boto:auth_path=/pklxkd-58/obj00000000000000000031 2023-05-31T20:39:13.904 DEBUG:boto:Method: HEAD 2023-05-31T20:39:13.905 DEBUG:boto:Path: /pklxkd-58/obj00000000000000000031 2023-05-31T20:39:13.906 DEBUG:boto:Data: 2023-05-31T20:39:13.907 DEBUG:boto:Headers: {} 2023-05-31T20:39:13.907 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:39:13.908 DEBUG:boto:Port: 8001 2023-05-31T20:39:13.909 DEBUG:boto:Params: {} 2023-05-31T20:39:13.910 DEBUG:boto:Token: None 2023-05-31T20:39:13.911 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:39:13 GMT /pklxkd-58/obj00000000000000000031 2023-05-31T20:39:13.911 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:86BfnORtrgvAoog/VJKyUQWK6ec= 2023-05-31T20:39:13.912 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:39:13 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:86BfnORtrgvAoog/VJKyUQWK6ec=', 'Content-Length': '0'} 2023-05-31T20:39:13.913 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:38:18 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008945d4ad6a069b00-006477b071-1c17-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:39:13 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:39:13.914 DEBUG:boto:path=/pklxkd-58/obj00000000000000000031 2023-05-31T20:39:13.914 DEBUG:boto:auth_path=/pklxkd-58/obj00000000000000000031 2023-05-31T20:39:13.915 DEBUG:boto:Method: HEAD 2023-05-31T20:39:13.915 DEBUG:boto:Path: /pklxkd-58/obj00000000000000000031 2023-05-31T20:39:13.915 DEBUG:boto:Data: 2023-05-31T20:39:13.916 DEBUG:boto:Headers: {} 2023-05-31T20:39:13.916 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:39:13.917 DEBUG:boto:Port: 8000 2023-05-31T20:39:13.917 DEBUG:boto:Params: {} 2023-05-31T20:39:13.917 DEBUG:boto:Token: None 2023-05-31T20:39:13.918 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:39:13 GMT /pklxkd-58/obj00000000000000000031 2023-05-31T20:39:13.918 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:86BfnORtrgvAoog/VJKyUQWK6ec= 2023-05-31T20:39:13.919 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:39:13 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:86BfnORtrgvAoog/VJKyUQWK6ec=', 'Content-Length': '0'} 2023-05-31T20:39:13.921 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:38:18 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000136b0e7f655dc4ec-006477b071-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:39:13 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:39:13.921 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000031 2023-05-31T20:39:13.922 DEBUG:boto:path=/pklxkd-58/obj00000000000000000031 2023-05-31T20:39:13.922 DEBUG:boto:auth_path=/pklxkd-58/obj00000000000000000031 2023-05-31T20:39:13.922 DEBUG:boto:Method: GET 2023-05-31T20:39:13.923 DEBUG:boto:Path: /pklxkd-58/obj00000000000000000031 2023-05-31T20:39:13.923 DEBUG:boto:Data: 2023-05-31T20:39:13.923 DEBUG:boto:Headers: {} 2023-05-31T20:39:13.924 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:39:13.924 DEBUG:boto:Port: 8001 2023-05-31T20:39:13.925 DEBUG:boto:Params: {} 2023-05-31T20:39:13.925 DEBUG:boto:Token: None 2023-05-31T20:39:13.926 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:39:13 GMT /pklxkd-58/obj00000000000000000031 2023-05-31T20:39:13.926 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:2clB9tPgLEBSgVfZo/H6eeLwvQs= 2023-05-31T20:39:13.926 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:39:13 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:2clB9tPgLEBSgVfZo/H6eeLwvQs=', 'Content-Length': '0'} 2023-05-31T20:39:13.929 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:38:18 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000058d2384c3f94cd51-006477b071-1c17-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:39:13 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:39:13.929 DEBUG:boto:path=/pklxkd-58/obj00000000000000000031 2023-05-31T20:39:13.930 DEBUG:boto:auth_path=/pklxkd-58/obj00000000000000000031 2023-05-31T20:39:13.930 DEBUG:boto:Method: GET 2023-05-31T20:39:13.930 DEBUG:boto:Path: /pklxkd-58/obj00000000000000000031 2023-05-31T20:39:13.931 DEBUG:boto:Data: 2023-05-31T20:39:13.931 DEBUG:boto:Headers: {} 2023-05-31T20:39:13.932 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:39:13.932 DEBUG:boto:Port: 8000 2023-05-31T20:39:13.932 DEBUG:boto:Params: {} 2023-05-31T20:39:13.933 DEBUG:boto:Token: None 2023-05-31T20:39:13.934 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:39:13 GMT /pklxkd-58/obj00000000000000000031 2023-05-31T20:39:13.934 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:2clB9tPgLEBSgVfZo/H6eeLwvQs= 2023-05-31T20:39:13.935 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:39:13 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:2clB9tPgLEBSgVfZo/H6eeLwvQs=', 'Content-Length': '0'} 2023-05-31T20:39:13.937 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:38:18 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000806806615957ca4a-006477b071-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:39:13 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:39:13.984 DEBUG:boto:path=/pklxkd-58/ 2023-05-31T20:39:13.985 DEBUG:boto:auth_path=/pklxkd-58/ 2023-05-31T20:39:13.986 DEBUG:boto:path=/pklxkd-58/?prefix=obj00000000000000000031 2023-05-31T20:39:13.987 DEBUG:boto:auth_path=/pklxkd-58/?prefix=obj00000000000000000031 2023-05-31T20:39:13.988 DEBUG:boto:Method: GET 2023-05-31T20:39:13.989 DEBUG:boto:Path: /pklxkd-58/?prefix=obj00000000000000000031 2023-05-31T20:39:13.989 DEBUG:boto:Data: 2023-05-31T20:39:13.989 DEBUG:boto:Headers: {} 2023-05-31T20:39:13.990 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:39:13.990 DEBUG:boto:Port: 8001 2023-05-31T20:39:13.991 DEBUG:boto:Params: {} 2023-05-31T20:39:13.991 DEBUG:boto:Token: None 2023-05-31T20:39:13.991 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:39:13 GMT /pklxkd-58/ 2023-05-31T20:39:13.992 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:GhwzgzgAYYFLDcfdK1Agett0HKk= 2023-05-31T20:39:13.992 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:39:13 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:GhwzgzgAYYFLDcfdK1Agett0HKk=', 'Content-Length': '0'} 2023-05-31T20:39:13.995 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000002fc637c73dd7a8d2-006477b071-1c17-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:39:13 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:39:13.996 DEBUG:boto:b'pklxkd-58obj000000000000000000311000falseobj000000000000000000312023-05-31T20:38:18.440Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:39:13.997 DEBUG:boto:path=/pklxkd-58/ 2023-05-31T20:39:13.997 DEBUG:boto:auth_path=/pklxkd-58/ 2023-05-31T20:39:13.998 DEBUG:boto:path=/pklxkd-58/?prefix=obj00000000000000000031 2023-05-31T20:39:13.998 DEBUG:boto:auth_path=/pklxkd-58/?prefix=obj00000000000000000031 2023-05-31T20:39:13.999 DEBUG:boto:Method: GET 2023-05-31T20:39:13.999 DEBUG:boto:Path: /pklxkd-58/?prefix=obj00000000000000000031 2023-05-31T20:39:14.000 DEBUG:boto:Data: 2023-05-31T20:39:14.000 DEBUG:boto:Headers: {} 2023-05-31T20:39:14.000 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:39:14.001 DEBUG:boto:Port: 8000 2023-05-31T20:39:14.001 DEBUG:boto:Params: {} 2023-05-31T20:39:14.002 DEBUG:boto:Token: None 2023-05-31T20:39:14.002 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:39:14 GMT /pklxkd-58/ 2023-05-31T20:39:14.002 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:WG6sc4Xwd7Q+ft8669zIyLEPh2A= 2023-05-31T20:39:14.003 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:39:14 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:WG6sc4Xwd7Q+ft8669zIyLEPh2A=', 'Content-Length': '0'} 2023-05-31T20:39:14.008 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d6569f7a48a93541-006477b072-1630-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:39:14 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:39:14.011 DEBUG:boto:b'pklxkd-58obj000000000000000000311000falseobj000000000000000000312023-05-31T20:38:18.440Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:39:14.012 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000032 2023-05-31T20:39:14.012 DEBUG:boto:path=/pklxkd-58/obj00000000000000000032 2023-05-31T20:39:14.013 DEBUG:boto:auth_path=/pklxkd-58/obj00000000000000000032 2023-05-31T20:39:14.013 DEBUG:boto:path=/pklxkd-58/obj00000000000000000032?versionId=null 2023-05-31T20:39:14.013 DEBUG:boto:auth_path=/pklxkd-58/obj00000000000000000032?versionId=null 2023-05-31T20:39:14.014 DEBUG:boto:Method: GET 2023-05-31T20:39:14.014 DEBUG:boto:Path: /pklxkd-58/obj00000000000000000032?versionId=null 2023-05-31T20:39:14.015 DEBUG:boto:Data: 2023-05-31T20:39:14.015 DEBUG:boto:Headers: {} 2023-05-31T20:39:14.015 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:39:14.016 DEBUG:boto:Port: 8001 2023-05-31T20:39:14.018 DEBUG:boto:Params: {} 2023-05-31T20:39:14.018 DEBUG:boto:Token: None 2023-05-31T20:39:14.019 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:39:14 GMT /pklxkd-58/obj00000000000000000032?versionId=null 2023-05-31T20:39:14.020 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:iiuG4On7VCAypuo3691VWyxkg8Y= 2023-05-31T20:39:14.020 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:39:14 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:iiuG4On7VCAypuo3691VWyxkg8Y=', 'Content-Length': '0'} 2023-05-31T20:39:14.023 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:38:15 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009de5d2b7bf457eb1-006477b072-1c17-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:39:14 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:39:14.024 DEBUG:boto:path=/pklxkd-58/obj00000000000000000032 2023-05-31T20:39:14.024 DEBUG:boto:auth_path=/pklxkd-58/obj00000000000000000032 2023-05-31T20:39:14.025 DEBUG:boto:path=/pklxkd-58/obj00000000000000000032?versionId=null 2023-05-31T20:39:14.025 DEBUG:boto:auth_path=/pklxkd-58/obj00000000000000000032?versionId=null 2023-05-31T20:39:14.025 DEBUG:boto:Method: GET 2023-05-31T20:39:14.026 DEBUG:boto:Path: /pklxkd-58/obj00000000000000000032?versionId=null 2023-05-31T20:39:14.026 DEBUG:boto:Data: 2023-05-31T20:39:14.027 DEBUG:boto:Headers: {} 2023-05-31T20:39:14.027 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:39:14.027 DEBUG:boto:Port: 8000 2023-05-31T20:39:14.028 DEBUG:boto:Params: {} 2023-05-31T20:39:14.028 DEBUG:boto:Token: None 2023-05-31T20:39:14.029 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:39:14 GMT /pklxkd-58/obj00000000000000000032?versionId=null 2023-05-31T20:39:14.029 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:iiuG4On7VCAypuo3691VWyxkg8Y= 2023-05-31T20:39:14.030 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:39:14 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:iiuG4On7VCAypuo3691VWyxkg8Y=', 'Content-Length': '0'} 2023-05-31T20:39:14.032 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:38:15 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000040e2cb824fb3a2ba-006477b072-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:39:14 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:39:14.033 DEBUG:boto:path=/pklxkd-58/obj00000000000000000032 2023-05-31T20:39:14.033 DEBUG:boto:auth_path=/pklxkd-58/obj00000000000000000032 2023-05-31T20:39:14.034 DEBUG:boto:path=/pklxkd-58/obj00000000000000000032?versionId=null 2023-05-31T20:39:14.034 DEBUG:boto:auth_path=/pklxkd-58/obj00000000000000000032?versionId=null 2023-05-31T20:39:14.034 DEBUG:boto:Method: HEAD 2023-05-31T20:39:14.035 DEBUG:boto:Path: /pklxkd-58/obj00000000000000000032?versionId=null 2023-05-31T20:39:14.035 DEBUG:boto:Data: 2023-05-31T20:39:14.035 DEBUG:boto:Headers: {} 2023-05-31T20:39:14.036 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:39:14.037 DEBUG:boto:Port: 8001 2023-05-31T20:39:14.037 DEBUG:boto:Params: {} 2023-05-31T20:39:14.037 DEBUG:boto:Token: None 2023-05-31T20:39:14.038 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:39:14 GMT /pklxkd-58/obj00000000000000000032?versionId=null 2023-05-31T20:39:14.038 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:unT+K+JCPWAxcF/Jh8CU7mpSCRE= 2023-05-31T20:39:14.039 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:39:14 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:unT+K+JCPWAxcF/Jh8CU7mpSCRE=', 'Content-Length': '0'} 2023-05-31T20:39:14.041 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:38:15 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003151130bbe2f5b5a-006477b072-1c17-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:39:14 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:39:14.041 DEBUG:boto:path=/pklxkd-58/obj00000000000000000032 2023-05-31T20:39:14.042 DEBUG:boto:auth_path=/pklxkd-58/obj00000000000000000032 2023-05-31T20:39:14.042 DEBUG:boto:path=/pklxkd-58/obj00000000000000000032?versionId=null 2023-05-31T20:39:14.042 DEBUG:boto:auth_path=/pklxkd-58/obj00000000000000000032?versionId=null 2023-05-31T20:39:14.043 DEBUG:boto:Method: HEAD 2023-05-31T20:39:14.043 DEBUG:boto:Path: /pklxkd-58/obj00000000000000000032?versionId=null 2023-05-31T20:39:14.043 DEBUG:boto:Data: 2023-05-31T20:39:14.044 DEBUG:boto:Headers: {} 2023-05-31T20:39:14.044 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:39:14.045 DEBUG:boto:Port: 8000 2023-05-31T20:39:14.045 DEBUG:boto:Params: {} 2023-05-31T20:39:14.045 DEBUG:boto:Token: None 2023-05-31T20:39:14.046 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:39:14 GMT /pklxkd-58/obj00000000000000000032?versionId=null 2023-05-31T20:39:14.046 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:unT+K+JCPWAxcF/Jh8CU7mpSCRE= 2023-05-31T20:39:14.047 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:39:14 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:unT+K+JCPWAxcF/Jh8CU7mpSCRE=', 'Content-Length': '0'} 2023-05-31T20:39:14.048 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:38:15 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000803301dfa8b63825-006477b072-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:39:14 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:39:14.049 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000032 2023-05-31T20:39:14.049 DEBUG:boto:path=/pklxkd-58/obj00000000000000000032 2023-05-31T20:39:14.050 DEBUG:boto:auth_path=/pklxkd-58/obj00000000000000000032 2023-05-31T20:39:14.050 DEBUG:boto:path=/pklxkd-58/obj00000000000000000032?versionId=null 2023-05-31T20:39:14.050 DEBUG:boto:auth_path=/pklxkd-58/obj00000000000000000032?versionId=null 2023-05-31T20:39:14.052 DEBUG:boto:Method: GET 2023-05-31T20:39:14.052 DEBUG:boto:Path: /pklxkd-58/obj00000000000000000032?versionId=null 2023-05-31T20:39:14.053 DEBUG:boto:Data: 2023-05-31T20:39:14.053 DEBUG:boto:Headers: {} 2023-05-31T20:39:14.053 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:39:14.054 DEBUG:boto:Port: 8001 2023-05-31T20:39:14.054 DEBUG:boto:Params: {} 2023-05-31T20:39:14.054 DEBUG:boto:Token: None 2023-05-31T20:39:14.055 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:39:14 GMT /pklxkd-58/obj00000000000000000032?versionId=null 2023-05-31T20:39:14.055 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:iiuG4On7VCAypuo3691VWyxkg8Y= 2023-05-31T20:39:14.056 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:39:14 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:iiuG4On7VCAypuo3691VWyxkg8Y=', 'Content-Length': '0'} 2023-05-31T20:39:14.057 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:38:15 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000003818d2c4eb097f6-006477b072-1c17-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:39:14 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:39:14.058 DEBUG:boto:path=/pklxkd-58/obj00000000000000000032 2023-05-31T20:39:14.059 DEBUG:boto:auth_path=/pklxkd-58/obj00000000000000000032 2023-05-31T20:39:14.059 DEBUG:boto:path=/pklxkd-58/obj00000000000000000032?versionId=null 2023-05-31T20:39:14.060 DEBUG:boto:auth_path=/pklxkd-58/obj00000000000000000032?versionId=null 2023-05-31T20:39:14.060 DEBUG:boto:Method: GET 2023-05-31T20:39:14.060 DEBUG:boto:Path: /pklxkd-58/obj00000000000000000032?versionId=null 2023-05-31T20:39:14.061 DEBUG:boto:Data: 2023-05-31T20:39:14.061 DEBUG:boto:Headers: {} 2023-05-31T20:39:14.062 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:39:14.062 DEBUG:boto:Port: 8000 2023-05-31T20:39:14.063 DEBUG:boto:Params: {} 2023-05-31T20:39:14.063 DEBUG:boto:Token: None 2023-05-31T20:39:14.064 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:39:14 GMT /pklxkd-58/obj00000000000000000032?versionId=null 2023-05-31T20:39:14.064 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:iiuG4On7VCAypuo3691VWyxkg8Y= 2023-05-31T20:39:14.065 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:39:14 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:iiuG4On7VCAypuo3691VWyxkg8Y=', 'Content-Length': '0'} 2023-05-31T20:39:14.066 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:38:15 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000feab01f79f74510c-006477b072-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:39:14 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:39:14.108 DEBUG:boto:path=/pklxkd-58/ 2023-05-31T20:39:14.109 DEBUG:boto:auth_path=/pklxkd-58/ 2023-05-31T20:39:14.110 DEBUG:boto:path=/pklxkd-58/?prefix=obj00000000000000000032 2023-05-31T20:39:14.111 DEBUG:boto:auth_path=/pklxkd-58/?prefix=obj00000000000000000032 2023-05-31T20:39:14.112 DEBUG:boto:Method: GET 2023-05-31T20:39:14.113 DEBUG:boto:Path: /pklxkd-58/?prefix=obj00000000000000000032 2023-05-31T20:39:14.113 DEBUG:boto:Data: 2023-05-31T20:39:14.114 DEBUG:boto:Headers: {} 2023-05-31T20:39:14.114 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:39:14.115 DEBUG:boto:Port: 8001 2023-05-31T20:39:14.115 DEBUG:boto:Params: {} 2023-05-31T20:39:14.116 DEBUG:boto:Token: None 2023-05-31T20:39:14.116 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:39:14 GMT /pklxkd-58/ 2023-05-31T20:39:14.117 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:WG6sc4Xwd7Q+ft8669zIyLEPh2A= 2023-05-31T20:39:14.117 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:39:14 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:WG6sc4Xwd7Q+ft8669zIyLEPh2A=', 'Content-Length': '0'} 2023-05-31T20:39:14.119 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000050080286bb9ce710-006477b072-1c17-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:39:14 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:39:14.120 DEBUG:boto:b'pklxkd-58obj000000000000000000321000falseobj000000000000000000322023-05-31T20:38:15.861Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:39:14.121 DEBUG:boto:path=/pklxkd-58/ 2023-05-31T20:39:14.121 DEBUG:boto:auth_path=/pklxkd-58/ 2023-05-31T20:39:14.122 DEBUG:boto:path=/pklxkd-58/?prefix=obj00000000000000000032 2023-05-31T20:39:14.122 DEBUG:boto:auth_path=/pklxkd-58/?prefix=obj00000000000000000032 2023-05-31T20:39:14.123 DEBUG:boto:Method: GET 2023-05-31T20:39:14.124 DEBUG:boto:Path: /pklxkd-58/?prefix=obj00000000000000000032 2023-05-31T20:39:14.124 DEBUG:boto:Data: 2023-05-31T20:39:14.124 DEBUG:boto:Headers: {} 2023-05-31T20:39:14.125 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:39:14.125 DEBUG:boto:Port: 8000 2023-05-31T20:39:14.125 DEBUG:boto:Params: {} 2023-05-31T20:39:14.127 DEBUG:boto:Token: None 2023-05-31T20:39:14.127 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:39:14 GMT /pklxkd-58/ 2023-05-31T20:39:14.128 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:WG6sc4Xwd7Q+ft8669zIyLEPh2A= 2023-05-31T20:39:14.129 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:39:14 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:WG6sc4Xwd7Q+ft8669zIyLEPh2A=', 'Content-Length': '0'} 2023-05-31T20:39:14.133 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000582393db2ef370f4-006477b072-1630-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:39:14 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:39:14.136 DEBUG:boto:b'pklxkd-58obj000000000000000000321000falseobj000000000000000000322023-05-31T20:38:15.861Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:39:14.137 DEBUG:boto:path=/pklxkd-58/obj00000000000000000032 2023-05-31T20:39:14.138 DEBUG:boto:auth_path=/pklxkd-58/obj00000000000000000032 2023-05-31T20:39:14.138 DEBUG:boto:Method: HEAD 2023-05-31T20:39:14.139 DEBUG:boto:Path: /pklxkd-58/obj00000000000000000032 2023-05-31T20:39:14.140 DEBUG:boto:Data: 2023-05-31T20:39:14.140 DEBUG:boto:Headers: {} 2023-05-31T20:39:14.141 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:39:14.141 DEBUG:boto:Port: 8001 2023-05-31T20:39:14.141 DEBUG:boto:Params: {} 2023-05-31T20:39:14.142 DEBUG:boto:Token: None 2023-05-31T20:39:14.142 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:39:14 GMT /pklxkd-58/obj00000000000000000032 2023-05-31T20:39:14.143 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:xXLWhKHMEv5Y2FNEwZqoHhn2c4k= 2023-05-31T20:39:14.143 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:39:14 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:xXLWhKHMEv5Y2FNEwZqoHhn2c4k=', 'Content-Length': '0'} 2023-05-31T20:39:14.145 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:38:15 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c9bdb40663f8bccd-006477b072-1c17-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:39:14 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:39:14.146 DEBUG:boto:path=/pklxkd-58/obj00000000000000000032 2023-05-31T20:39:14.146 DEBUG:boto:auth_path=/pklxkd-58/obj00000000000000000032 2023-05-31T20:39:14.147 DEBUG:boto:Method: HEAD 2023-05-31T20:39:14.147 DEBUG:boto:Path: /pklxkd-58/obj00000000000000000032 2023-05-31T20:39:14.147 DEBUG:boto:Data: 2023-05-31T20:39:14.148 DEBUG:boto:Headers: {} 2023-05-31T20:39:14.148 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:39:14.148 DEBUG:boto:Port: 8000 2023-05-31T20:39:14.149 DEBUG:boto:Params: {} 2023-05-31T20:39:14.149 DEBUG:boto:Token: None 2023-05-31T20:39:14.149 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:39:14 GMT /pklxkd-58/obj00000000000000000032 2023-05-31T20:39:14.150 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:xXLWhKHMEv5Y2FNEwZqoHhn2c4k= 2023-05-31T20:39:14.150 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:39:14 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:xXLWhKHMEv5Y2FNEwZqoHhn2c4k=', 'Content-Length': '0'} 2023-05-31T20:39:14.152 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:38:15 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002b6d6f048d7214a4-006477b072-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:39:14 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:39:14.153 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000032 2023-05-31T20:39:14.153 DEBUG:boto:path=/pklxkd-58/obj00000000000000000032 2023-05-31T20:39:14.154 DEBUG:boto:auth_path=/pklxkd-58/obj00000000000000000032 2023-05-31T20:39:14.154 DEBUG:boto:Method: GET 2023-05-31T20:39:14.155 DEBUG:boto:Path: /pklxkd-58/obj00000000000000000032 2023-05-31T20:39:14.155 DEBUG:boto:Data: 2023-05-31T20:39:14.155 DEBUG:boto:Headers: {} 2023-05-31T20:39:14.156 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:39:14.156 DEBUG:boto:Port: 8001 2023-05-31T20:39:14.157 DEBUG:boto:Params: {} 2023-05-31T20:39:14.157 DEBUG:boto:Token: None 2023-05-31T20:39:14.158 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:39:14 GMT /pklxkd-58/obj00000000000000000032 2023-05-31T20:39:14.158 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:rt+06alqZ2zys6a5ZOTlxJFWa/E= 2023-05-31T20:39:14.158 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:39:14 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:rt+06alqZ2zys6a5ZOTlxJFWa/E=', 'Content-Length': '0'} 2023-05-31T20:39:14.161 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:38:15 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005c5d925394eead9f-006477b072-1c17-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:39:14 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:39:14.209 DEBUG:boto:path=/pklxkd-58/obj00000000000000000032 2023-05-31T20:39:14.210 DEBUG:boto:auth_path=/pklxkd-58/obj00000000000000000032 2023-05-31T20:39:14.210 DEBUG:boto:Method: GET 2023-05-31T20:39:14.211 DEBUG:boto:Path: /pklxkd-58/obj00000000000000000032 2023-05-31T20:39:14.211 DEBUG:boto:Data: 2023-05-31T20:39:14.211 DEBUG:boto:Headers: {} 2023-05-31T20:39:14.212 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:39:14.213 DEBUG:boto:Port: 8000 2023-05-31T20:39:14.214 DEBUG:boto:Params: {} 2023-05-31T20:39:14.215 DEBUG:boto:Token: None 2023-05-31T20:39:14.215 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:39:14 GMT /pklxkd-58/obj00000000000000000032 2023-05-31T20:39:14.225 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:rt+06alqZ2zys6a5ZOTlxJFWa/E= 2023-05-31T20:39:14.225 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:39:14 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:rt+06alqZ2zys6a5ZOTlxJFWa/E=', 'Content-Length': '0'} 2023-05-31T20:39:14.227 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:38:15 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000c63e8e9b5111e5a-006477b072-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:39:14 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:39:14.228 DEBUG:boto:path=/pklxkd-58/ 2023-05-31T20:39:14.228 DEBUG:boto:auth_path=/pklxkd-58/ 2023-05-31T20:39:14.229 DEBUG:boto:path=/pklxkd-58/?prefix=obj00000000000000000032 2023-05-31T20:39:14.234 DEBUG:boto:auth_path=/pklxkd-58/?prefix=obj00000000000000000032 2023-05-31T20:39:14.234 DEBUG:boto:Method: GET 2023-05-31T20:39:14.235 DEBUG:boto:Path: /pklxkd-58/?prefix=obj00000000000000000032 2023-05-31T20:39:14.236 DEBUG:boto:Data: 2023-05-31T20:39:14.236 DEBUG:boto:Headers: {} 2023-05-31T20:39:14.237 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:39:14.237 DEBUG:boto:Port: 8001 2023-05-31T20:39:14.237 DEBUG:boto:Params: {} 2023-05-31T20:39:14.238 DEBUG:boto:Token: None 2023-05-31T20:39:14.238 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:39:14 GMT /pklxkd-58/ 2023-05-31T20:39:14.239 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:WG6sc4Xwd7Q+ft8669zIyLEPh2A= 2023-05-31T20:39:14.239 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:39:14 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:WG6sc4Xwd7Q+ft8669zIyLEPh2A=', 'Content-Length': '0'} 2023-05-31T20:39:14.243 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000394b0c1bf6e49228-006477b072-1c17-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:39:14 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:39:14.243 DEBUG:boto:b'pklxkd-58obj000000000000000000321000falseobj000000000000000000322023-05-31T20:38:15.861Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:39:14.244 DEBUG:boto:path=/pklxkd-58/ 2023-05-31T20:39:14.245 DEBUG:boto:auth_path=/pklxkd-58/ 2023-05-31T20:39:14.245 DEBUG:boto:path=/pklxkd-58/?prefix=obj00000000000000000032 2023-05-31T20:39:14.246 DEBUG:boto:auth_path=/pklxkd-58/?prefix=obj00000000000000000032 2023-05-31T20:39:14.247 DEBUG:boto:Method: GET 2023-05-31T20:39:14.247 DEBUG:boto:Path: /pklxkd-58/?prefix=obj00000000000000000032 2023-05-31T20:39:14.248 DEBUG:boto:Data: 2023-05-31T20:39:14.248 DEBUG:boto:Headers: {} 2023-05-31T20:39:14.248 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:39:14.249 DEBUG:boto:Port: 8000 2023-05-31T20:39:14.249 DEBUG:boto:Params: {} 2023-05-31T20:39:14.250 DEBUG:boto:Token: None 2023-05-31T20:39:14.250 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:39:14 GMT /pklxkd-58/ 2023-05-31T20:39:14.251 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:WG6sc4Xwd7Q+ft8669zIyLEPh2A= 2023-05-31T20:39:14.251 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:39:14 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:WG6sc4Xwd7Q+ft8669zIyLEPh2A=', 'Content-Length': '0'} 2023-05-31T20:39:14.255 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c35ce66902c5ae3b-006477b072-1630-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:39:14 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:39:14.256 DEBUG:boto:b'pklxkd-58obj000000000000000000321000falseobj000000000000000000322023-05-31T20:38:15.861Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:39:14.257 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000033 2023-05-31T20:39:14.257 DEBUG:boto:path=/pklxkd-58/obj00000000000000000033 2023-05-31T20:39:14.258 DEBUG:boto:auth_path=/pklxkd-58/obj00000000000000000033 2023-05-31T20:39:14.258 DEBUG:boto:path=/pklxkd-58/obj00000000000000000033?versionId=null 2023-05-31T20:39:14.259 DEBUG:boto:auth_path=/pklxkd-58/obj00000000000000000033?versionId=null 2023-05-31T20:39:14.259 DEBUG:boto:Method: GET 2023-05-31T20:39:14.260 DEBUG:boto:Path: /pklxkd-58/obj00000000000000000033?versionId=null 2023-05-31T20:39:14.260 DEBUG:boto:Data: 2023-05-31T20:39:14.260 DEBUG:boto:Headers: {} 2023-05-31T20:39:14.261 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:39:14.261 DEBUG:boto:Port: 8001 2023-05-31T20:39:14.261 DEBUG:boto:Params: {} 2023-05-31T20:39:14.262 DEBUG:boto:Token: None 2023-05-31T20:39:14.262 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:39:14 GMT /pklxkd-58/obj00000000000000000033?versionId=null 2023-05-31T20:39:14.263 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:mNSB3P0QGfndACd2CTGm8Dmdhik= 2023-05-31T20:39:14.263 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:39:14 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:mNSB3P0QGfndACd2CTGm8Dmdhik=', 'Content-Length': '0'} 2023-05-31T20:39:14.266 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:38:22 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009c8b5b6d1013e245-006477b072-1c17-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:39:14 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:39:14.266 DEBUG:boto:path=/pklxkd-58/obj00000000000000000033 2023-05-31T20:39:14.267 DEBUG:boto:auth_path=/pklxkd-58/obj00000000000000000033 2023-05-31T20:39:14.267 DEBUG:boto:path=/pklxkd-58/obj00000000000000000033?versionId=null 2023-05-31T20:39:14.267 DEBUG:boto:auth_path=/pklxkd-58/obj00000000000000000033?versionId=null 2023-05-31T20:39:14.268 DEBUG:boto:Method: GET 2023-05-31T20:39:14.268 DEBUG:boto:Path: /pklxkd-58/obj00000000000000000033?versionId=null 2023-05-31T20:39:14.269 DEBUG:boto:Data: 2023-05-31T20:39:14.269 DEBUG:boto:Headers: {} 2023-05-31T20:39:14.270 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:39:14.270 DEBUG:boto:Port: 8000 2023-05-31T20:39:14.270 DEBUG:boto:Params: {} 2023-05-31T20:39:14.271 DEBUG:boto:Token: None 2023-05-31T20:39:14.271 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:39:14 GMT /pklxkd-58/obj00000000000000000033?versionId=null 2023-05-31T20:39:14.272 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:mNSB3P0QGfndACd2CTGm8Dmdhik= 2023-05-31T20:39:14.272 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:39:14 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:mNSB3P0QGfndACd2CTGm8Dmdhik=', 'Content-Length': '0'} 2023-05-31T20:39:14.274 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:38:22 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c400624b5a6c2436-006477b072-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:39:14 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:39:14.320 DEBUG:boto:path=/pklxkd-58/obj00000000000000000033 2023-05-31T20:39:14.321 DEBUG:boto:auth_path=/pklxkd-58/obj00000000000000000033 2023-05-31T20:39:14.321 DEBUG:boto:path=/pklxkd-58/obj00000000000000000033?versionId=null 2023-05-31T20:39:14.322 DEBUG:boto:auth_path=/pklxkd-58/obj00000000000000000033?versionId=null 2023-05-31T20:39:14.322 DEBUG:boto:Method: HEAD 2023-05-31T20:39:14.322 DEBUG:boto:Path: /pklxkd-58/obj00000000000000000033?versionId=null 2023-05-31T20:39:14.323 DEBUG:boto:Data: 2023-05-31T20:39:14.323 DEBUG:boto:Headers: {} 2023-05-31T20:39:14.323 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:39:14.324 DEBUG:boto:Port: 8001 2023-05-31T20:39:14.324 DEBUG:boto:Params: {} 2023-05-31T20:39:14.324 DEBUG:boto:Token: None 2023-05-31T20:39:14.325 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:39:14 GMT /pklxkd-58/obj00000000000000000033?versionId=null 2023-05-31T20:39:14.325 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:tXhQgp8RPHckiaVlKYCTd1mFGsg= 2023-05-31T20:39:14.326 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:39:14 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:tXhQgp8RPHckiaVlKYCTd1mFGsg=', 'Content-Length': '0'} 2023-05-31T20:39:14.328 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:38:22 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000027570040e4480170-006477b072-1c17-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:39:14 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:39:14.329 DEBUG:boto:path=/pklxkd-58/obj00000000000000000033 2023-05-31T20:39:14.330 DEBUG:boto:auth_path=/pklxkd-58/obj00000000000000000033 2023-05-31T20:39:14.330 DEBUG:boto:path=/pklxkd-58/obj00000000000000000033?versionId=null 2023-05-31T20:39:14.330 DEBUG:boto:auth_path=/pklxkd-58/obj00000000000000000033?versionId=null 2023-05-31T20:39:14.331 DEBUG:boto:Method: HEAD 2023-05-31T20:39:14.331 DEBUG:boto:Path: /pklxkd-58/obj00000000000000000033?versionId=null 2023-05-31T20:39:14.332 DEBUG:boto:Data: 2023-05-31T20:39:14.332 DEBUG:boto:Headers: {} 2023-05-31T20:39:14.333 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:39:14.333 DEBUG:boto:Port: 8000 2023-05-31T20:39:14.333 DEBUG:boto:Params: {} 2023-05-31T20:39:14.334 DEBUG:boto:Token: None 2023-05-31T20:39:14.334 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:39:14 GMT /pklxkd-58/obj00000000000000000033?versionId=null 2023-05-31T20:39:14.335 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:tXhQgp8RPHckiaVlKYCTd1mFGsg= 2023-05-31T20:39:14.335 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:39:14 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:tXhQgp8RPHckiaVlKYCTd1mFGsg=', 'Content-Length': '0'} 2023-05-31T20:39:14.337 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:38:22 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000cd76c843acd3d140-006477b072-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:39:14 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:39:14.338 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000033 2023-05-31T20:39:14.338 DEBUG:boto:path=/pklxkd-58/obj00000000000000000033 2023-05-31T20:39:14.338 DEBUG:boto:auth_path=/pklxkd-58/obj00000000000000000033 2023-05-31T20:39:14.339 DEBUG:boto:path=/pklxkd-58/obj00000000000000000033?versionId=null 2023-05-31T20:39:14.339 DEBUG:boto:auth_path=/pklxkd-58/obj00000000000000000033?versionId=null 2023-05-31T20:39:14.340 DEBUG:boto:Method: GET 2023-05-31T20:39:14.340 DEBUG:boto:Path: /pklxkd-58/obj00000000000000000033?versionId=null 2023-05-31T20:39:14.340 DEBUG:boto:Data: 2023-05-31T20:39:14.341 DEBUG:boto:Headers: {} 2023-05-31T20:39:14.342 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:39:14.342 DEBUG:boto:Port: 8001 2023-05-31T20:39:14.342 DEBUG:boto:Params: {} 2023-05-31T20:39:14.343 DEBUG:boto:Token: None 2023-05-31T20:39:14.343 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:39:14 GMT /pklxkd-58/obj00000000000000000033?versionId=null 2023-05-31T20:39:14.344 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:mNSB3P0QGfndACd2CTGm8Dmdhik= 2023-05-31T20:39:14.344 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:39:14 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:mNSB3P0QGfndACd2CTGm8Dmdhik=', 'Content-Length': '0'} 2023-05-31T20:39:14.346 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:38:22 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000066d7206f4fc9ded0-006477b072-1c17-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:39:14 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:39:14.347 DEBUG:boto:path=/pklxkd-58/obj00000000000000000033 2023-05-31T20:39:14.348 DEBUG:boto:auth_path=/pklxkd-58/obj00000000000000000033 2023-05-31T20:39:14.348 DEBUG:boto:path=/pklxkd-58/obj00000000000000000033?versionId=null 2023-05-31T20:39:14.349 DEBUG:boto:auth_path=/pklxkd-58/obj00000000000000000033?versionId=null 2023-05-31T20:39:14.349 DEBUG:boto:Method: GET 2023-05-31T20:39:14.349 DEBUG:boto:Path: /pklxkd-58/obj00000000000000000033?versionId=null 2023-05-31T20:39:14.350 DEBUG:boto:Data: 2023-05-31T20:39:14.350 DEBUG:boto:Headers: {} 2023-05-31T20:39:14.350 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:39:14.351 DEBUG:boto:Port: 8000 2023-05-31T20:39:14.351 DEBUG:boto:Params: {} 2023-05-31T20:39:14.351 DEBUG:boto:Token: None 2023-05-31T20:39:14.352 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:39:14 GMT /pklxkd-58/obj00000000000000000033?versionId=null 2023-05-31T20:39:14.352 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:mNSB3P0QGfndACd2CTGm8Dmdhik= 2023-05-31T20:39:14.353 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:39:14 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:mNSB3P0QGfndACd2CTGm8Dmdhik=', 'Content-Length': '0'} 2023-05-31T20:39:14.355 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:38:22 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000070d436a73550fd3f-006477b072-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:39:14 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:39:14.355 DEBUG:boto:path=/pklxkd-58/ 2023-05-31T20:39:14.356 DEBUG:boto:auth_path=/pklxkd-58/ 2023-05-31T20:39:14.356 DEBUG:boto:path=/pklxkd-58/?prefix=obj00000000000000000033 2023-05-31T20:39:14.357 DEBUG:boto:auth_path=/pklxkd-58/?prefix=obj00000000000000000033 2023-05-31T20:39:14.357 DEBUG:boto:Method: GET 2023-05-31T20:39:14.357 DEBUG:boto:Path: /pklxkd-58/?prefix=obj00000000000000000033 2023-05-31T20:39:14.358 DEBUG:boto:Data: 2023-05-31T20:39:14.358 DEBUG:boto:Headers: {} 2023-05-31T20:39:14.358 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:39:14.359 DEBUG:boto:Port: 8001 2023-05-31T20:39:14.359 DEBUG:boto:Params: {} 2023-05-31T20:39:14.360 DEBUG:boto:Token: None 2023-05-31T20:39:14.360 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:39:14 GMT /pklxkd-58/ 2023-05-31T20:39:14.361 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:WG6sc4Xwd7Q+ft8669zIyLEPh2A= 2023-05-31T20:39:14.361 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:39:14 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:WG6sc4Xwd7Q+ft8669zIyLEPh2A=', 'Content-Length': '0'} 2023-05-31T20:39:14.364 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b63857e98148ae18-006477b072-1c17-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:39:14 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:39:14.365 DEBUG:boto:b'pklxkd-58obj000000000000000000331000falseobj000000000000000000332023-05-31T20:38:22.084Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:39:14.366 DEBUG:boto:path=/pklxkd-58/ 2023-05-31T20:39:14.366 DEBUG:boto:auth_path=/pklxkd-58/ 2023-05-31T20:39:14.366 DEBUG:boto:path=/pklxkd-58/?prefix=obj00000000000000000033 2023-05-31T20:39:14.367 DEBUG:boto:auth_path=/pklxkd-58/?prefix=obj00000000000000000033 2023-05-31T20:39:14.368 DEBUG:boto:Method: GET 2023-05-31T20:39:14.368 DEBUG:boto:Path: /pklxkd-58/?prefix=obj00000000000000000033 2023-05-31T20:39:14.368 DEBUG:boto:Data: 2023-05-31T20:39:14.369 DEBUG:boto:Headers: {} 2023-05-31T20:39:14.370 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:39:14.371 DEBUG:boto:Port: 8000 2023-05-31T20:39:14.372 DEBUG:boto:Params: {} 2023-05-31T20:39:14.373 DEBUG:boto:Token: None 2023-05-31T20:39:14.374 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:39:14 GMT /pklxkd-58/ 2023-05-31T20:39:14.374 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:WG6sc4Xwd7Q+ft8669zIyLEPh2A= 2023-05-31T20:39:14.375 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:39:14 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:WG6sc4Xwd7Q+ft8669zIyLEPh2A=', 'Content-Length': '0'} 2023-05-31T20:39:14.379 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000214e8780500a5075-006477b072-1630-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:39:14 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:39:14.381 DEBUG:boto:b'pklxkd-58obj000000000000000000331000falseobj000000000000000000332023-05-31T20:38:22.084Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:39:14.382 DEBUG:boto:path=/pklxkd-58/obj00000000000000000033 2023-05-31T20:39:14.382 DEBUG:boto:auth_path=/pklxkd-58/obj00000000000000000033 2023-05-31T20:39:14.383 DEBUG:boto:Method: HEAD 2023-05-31T20:39:14.383 DEBUG:boto:Path: /pklxkd-58/obj00000000000000000033 2023-05-31T20:39:14.384 DEBUG:boto:Data: 2023-05-31T20:39:14.384 DEBUG:boto:Headers: {} 2023-05-31T20:39:14.385 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:39:14.385 DEBUG:boto:Port: 8001 2023-05-31T20:39:14.387 DEBUG:boto:Params: {} 2023-05-31T20:39:14.388 DEBUG:boto:Token: None 2023-05-31T20:39:14.389 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:39:14 GMT /pklxkd-58/obj00000000000000000033 2023-05-31T20:39:14.389 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:xdjRcFSQ60lkFIrI6RHjwxVj0l0= 2023-05-31T20:39:14.390 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:39:14 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:xdjRcFSQ60lkFIrI6RHjwxVj0l0=', 'Content-Length': '0'} 2023-05-31T20:39:14.394 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:38:22 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000085c49321421ebf4a-006477b072-1c17-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:39:14 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:39:14.394 DEBUG:boto:path=/pklxkd-58/obj00000000000000000033 2023-05-31T20:39:14.395 DEBUG:boto:auth_path=/pklxkd-58/obj00000000000000000033 2023-05-31T20:39:14.395 DEBUG:boto:Method: HEAD 2023-05-31T20:39:14.396 DEBUG:boto:Path: /pklxkd-58/obj00000000000000000033 2023-05-31T20:39:14.396 DEBUG:boto:Data: 2023-05-31T20:39:14.397 DEBUG:boto:Headers: {} 2023-05-31T20:39:14.397 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:39:14.398 DEBUG:boto:Port: 8000 2023-05-31T20:39:14.398 DEBUG:boto:Params: {} 2023-05-31T20:39:14.398 DEBUG:boto:Token: None 2023-05-31T20:39:14.399 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:39:14 GMT /pklxkd-58/obj00000000000000000033 2023-05-31T20:39:14.399 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:xdjRcFSQ60lkFIrI6RHjwxVj0l0= 2023-05-31T20:39:14.400 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:39:14 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:xdjRcFSQ60lkFIrI6RHjwxVj0l0=', 'Content-Length': '0'} 2023-05-31T20:39:14.402 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:38:22 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c7ab5c53e7cb0994-006477b072-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:39:14 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:39:14.402 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000033 2023-05-31T20:39:14.403 DEBUG:boto:path=/pklxkd-58/obj00000000000000000033 2023-05-31T20:39:14.403 DEBUG:boto:auth_path=/pklxkd-58/obj00000000000000000033 2023-05-31T20:39:14.404 DEBUG:boto:Method: GET 2023-05-31T20:39:14.404 DEBUG:boto:Path: /pklxkd-58/obj00000000000000000033 2023-05-31T20:39:14.405 DEBUG:boto:Data: 2023-05-31T20:39:14.405 DEBUG:boto:Headers: {} 2023-05-31T20:39:14.406 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:39:14.406 DEBUG:boto:Port: 8001 2023-05-31T20:39:14.407 DEBUG:boto:Params: {} 2023-05-31T20:39:14.407 DEBUG:boto:Token: None 2023-05-31T20:39:14.409 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:39:14 GMT /pklxkd-58/obj00000000000000000033 2023-05-31T20:39:14.410 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:ZcRzoeyxwFTRZbmEoYMPq6K0ptw= 2023-05-31T20:39:14.410 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:39:14 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:ZcRzoeyxwFTRZbmEoYMPq6K0ptw=', 'Content-Length': '0'} 2023-05-31T20:39:14.412 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:38:22 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000087df113b6c4ec49c-006477b072-1c17-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:39:14 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:39:14.413 DEBUG:boto:path=/pklxkd-58/obj00000000000000000033 2023-05-31T20:39:14.414 DEBUG:boto:auth_path=/pklxkd-58/obj00000000000000000033 2023-05-31T20:39:14.414 DEBUG:boto:Method: GET 2023-05-31T20:39:14.414 DEBUG:boto:Path: /pklxkd-58/obj00000000000000000033 2023-05-31T20:39:14.415 DEBUG:boto:Data: 2023-05-31T20:39:14.416 DEBUG:boto:Headers: {} 2023-05-31T20:39:14.416 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:39:14.417 DEBUG:boto:Port: 8000 2023-05-31T20:39:14.418 DEBUG:boto:Params: {} 2023-05-31T20:39:14.419 DEBUG:boto:Token: None 2023-05-31T20:39:14.420 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:39:14 GMT /pklxkd-58/obj00000000000000000033 2023-05-31T20:39:14.421 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:ZcRzoeyxwFTRZbmEoYMPq6K0ptw= 2023-05-31T20:39:14.421 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:39:14 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:ZcRzoeyxwFTRZbmEoYMPq6K0ptw=', 'Content-Length': '0'} 2023-05-31T20:39:14.424 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:38:22 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c4dd6b92b6106a44-006477b072-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:39:14 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:39:14.468 DEBUG:boto:path=/pklxkd-58/ 2023-05-31T20:39:14.469 DEBUG:boto:auth_path=/pklxkd-58/ 2023-05-31T20:39:14.469 DEBUG:boto:path=/pklxkd-58/?prefix=obj00000000000000000033 2023-05-31T20:39:14.470 DEBUG:boto:auth_path=/pklxkd-58/?prefix=obj00000000000000000033 2023-05-31T20:39:14.470 DEBUG:boto:Method: GET 2023-05-31T20:39:14.470 DEBUG:boto:Path: /pklxkd-58/?prefix=obj00000000000000000033 2023-05-31T20:39:14.471 DEBUG:boto:Data: 2023-05-31T20:39:14.471 DEBUG:boto:Headers: {} 2023-05-31T20:39:14.471 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:39:14.472 DEBUG:boto:Port: 8001 2023-05-31T20:39:14.472 DEBUG:boto:Params: {} 2023-05-31T20:39:14.473 DEBUG:boto:Token: None 2023-05-31T20:39:14.473 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:39:14 GMT /pklxkd-58/ 2023-05-31T20:39:14.474 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:WG6sc4Xwd7Q+ft8669zIyLEPh2A= 2023-05-31T20:39:14.474 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:39:14 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:WG6sc4Xwd7Q+ft8669zIyLEPh2A=', 'Content-Length': '0'} 2023-05-31T20:39:14.477 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000005d55850abdce7c83-006477b072-1c17-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:39:14 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:39:14.478 DEBUG:boto:b'pklxkd-58obj000000000000000000331000falseobj000000000000000000332023-05-31T20:38:22.084Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:39:14.479 DEBUG:boto:path=/pklxkd-58/ 2023-05-31T20:39:14.479 DEBUG:boto:auth_path=/pklxkd-58/ 2023-05-31T20:39:14.480 DEBUG:boto:path=/pklxkd-58/?prefix=obj00000000000000000033 2023-05-31T20:39:14.480 DEBUG:boto:auth_path=/pklxkd-58/?prefix=obj00000000000000000033 2023-05-31T20:39:14.480 DEBUG:boto:Method: GET 2023-05-31T20:39:14.481 DEBUG:boto:Path: /pklxkd-58/?prefix=obj00000000000000000033 2023-05-31T20:39:14.481 DEBUG:boto:Data: 2023-05-31T20:39:14.481 DEBUG:boto:Headers: {} 2023-05-31T20:39:14.482 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:39:14.482 DEBUG:boto:Port: 8000 2023-05-31T20:39:14.482 DEBUG:boto:Params: {} 2023-05-31T20:39:14.483 DEBUG:boto:Token: None 2023-05-31T20:39:14.483 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:39:14 GMT /pklxkd-58/ 2023-05-31T20:39:14.484 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:WG6sc4Xwd7Q+ft8669zIyLEPh2A= 2023-05-31T20:39:14.484 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:39:14 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:WG6sc4Xwd7Q+ft8669zIyLEPh2A=', 'Content-Length': '0'} 2023-05-31T20:39:14.488 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000000b6c1bd79f7fa028-006477b072-1630-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:39:14 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:39:14.489 DEBUG:boto:b'pklxkd-58obj000000000000000000331000falseobj000000000000000000332023-05-31T20:38:22.084Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:39:14.490 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000036 2023-05-31T20:39:14.491 DEBUG:boto:path=/pklxkd-58/obj00000000000000000036 2023-05-31T20:39:14.491 DEBUG:boto:auth_path=/pklxkd-58/obj00000000000000000036 2023-05-31T20:39:14.492 DEBUG:boto:path=/pklxkd-58/obj00000000000000000036?versionId=null 2023-05-31T20:39:14.492 DEBUG:boto:auth_path=/pklxkd-58/obj00000000000000000036?versionId=null 2023-05-31T20:39:14.492 DEBUG:boto:Method: GET 2023-05-31T20:39:14.493 DEBUG:boto:Path: /pklxkd-58/obj00000000000000000036?versionId=null 2023-05-31T20:39:14.493 DEBUG:boto:Data: 2023-05-31T20:39:14.494 DEBUG:boto:Headers: {} 2023-05-31T20:39:14.494 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:39:14.495 DEBUG:boto:Port: 8001 2023-05-31T20:39:14.495 DEBUG:boto:Params: {} 2023-05-31T20:39:14.495 DEBUG:boto:Token: None 2023-05-31T20:39:14.496 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:39:14 GMT /pklxkd-58/obj00000000000000000036?versionId=null 2023-05-31T20:39:14.497 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:1m5Og3jJizMmS5Y+nb+mhw0t4HU= 2023-05-31T20:39:14.497 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:39:14 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:1m5Og3jJizMmS5Y+nb+mhw0t4HU=', 'Content-Length': '0'} 2023-05-31T20:39:14.500 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:38:22 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006e1d6e6ee4067247-006477b072-1c17-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:39:14 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:39:14.544 DEBUG:boto:path=/pklxkd-58/obj00000000000000000036 2023-05-31T20:39:14.545 DEBUG:boto:auth_path=/pklxkd-58/obj00000000000000000036 2023-05-31T20:39:14.545 DEBUG:boto:path=/pklxkd-58/obj00000000000000000036?versionId=null 2023-05-31T20:39:14.546 DEBUG:boto:auth_path=/pklxkd-58/obj00000000000000000036?versionId=null 2023-05-31T20:39:14.546 DEBUG:boto:Method: GET 2023-05-31T20:39:14.546 DEBUG:boto:Path: /pklxkd-58/obj00000000000000000036?versionId=null 2023-05-31T20:39:14.547 DEBUG:boto:Data: 2023-05-31T20:39:14.547 DEBUG:boto:Headers: {} 2023-05-31T20:39:14.547 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:39:14.548 DEBUG:boto:Port: 8000 2023-05-31T20:39:14.548 DEBUG:boto:Params: {} 2023-05-31T20:39:14.549 DEBUG:boto:Token: None 2023-05-31T20:39:14.549 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:39:14 GMT /pklxkd-58/obj00000000000000000036?versionId=null 2023-05-31T20:39:14.550 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:1m5Og3jJizMmS5Y+nb+mhw0t4HU= 2023-05-31T20:39:14.550 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:39:14 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:1m5Og3jJizMmS5Y+nb+mhw0t4HU=', 'Content-Length': '0'} 2023-05-31T20:39:14.552 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:38:22 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000974e857deb04772a-006477b072-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:39:14 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:39:14.553 DEBUG:boto:path=/pklxkd-58/obj00000000000000000036 2023-05-31T20:39:14.553 DEBUG:boto:auth_path=/pklxkd-58/obj00000000000000000036 2023-05-31T20:39:14.554 DEBUG:boto:path=/pklxkd-58/obj00000000000000000036?versionId=null 2023-05-31T20:39:14.554 DEBUG:boto:auth_path=/pklxkd-58/obj00000000000000000036?versionId=null 2023-05-31T20:39:14.555 DEBUG:boto:Method: HEAD 2023-05-31T20:39:14.555 DEBUG:boto:Path: /pklxkd-58/obj00000000000000000036?versionId=null 2023-05-31T20:39:14.555 DEBUG:boto:Data: 2023-05-31T20:39:14.556 DEBUG:boto:Headers: {} 2023-05-31T20:39:14.556 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:39:14.557 DEBUG:boto:Port: 8001 2023-05-31T20:39:14.557 DEBUG:boto:Params: {} 2023-05-31T20:39:14.558 DEBUG:boto:Token: None 2023-05-31T20:39:14.558 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:39:14 GMT /pklxkd-58/obj00000000000000000036?versionId=null 2023-05-31T20:39:14.559 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:vtkJdHMyorWDNB6p9GMILwybwEc= 2023-05-31T20:39:14.559 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:39:14 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:vtkJdHMyorWDNB6p9GMILwybwEc=', 'Content-Length': '0'} 2023-05-31T20:39:14.561 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:38:22 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004c7a01ee0a42add6-006477b072-1c17-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:39:14 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:39:14.562 DEBUG:boto:path=/pklxkd-58/obj00000000000000000036 2023-05-31T20:39:14.562 DEBUG:boto:auth_path=/pklxkd-58/obj00000000000000000036 2023-05-31T20:39:14.563 DEBUG:boto:path=/pklxkd-58/obj00000000000000000036?versionId=null 2023-05-31T20:39:14.563 DEBUG:boto:auth_path=/pklxkd-58/obj00000000000000000036?versionId=null 2023-05-31T20:39:14.563 DEBUG:boto:Method: HEAD 2023-05-31T20:39:14.564 DEBUG:boto:Path: /pklxkd-58/obj00000000000000000036?versionId=null 2023-05-31T20:39:14.564 DEBUG:boto:Data: 2023-05-31T20:39:14.564 DEBUG:boto:Headers: {} 2023-05-31T20:39:14.565 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:39:14.565 DEBUG:boto:Port: 8000 2023-05-31T20:39:14.565 DEBUG:boto:Params: {} 2023-05-31T20:39:14.566 DEBUG:boto:Token: None 2023-05-31T20:39:14.566 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:39:14 GMT /pklxkd-58/obj00000000000000000036?versionId=null 2023-05-31T20:39:14.567 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:vtkJdHMyorWDNB6p9GMILwybwEc= 2023-05-31T20:39:14.567 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:39:14 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:vtkJdHMyorWDNB6p9GMILwybwEc=', 'Content-Length': '0'} 2023-05-31T20:39:14.569 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:38:22 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c8a301d8e913f204-006477b072-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:39:14 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:39:14.570 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000036 2023-05-31T20:39:14.570 DEBUG:boto:path=/pklxkd-58/obj00000000000000000036 2023-05-31T20:39:14.570 DEBUG:boto:auth_path=/pklxkd-58/obj00000000000000000036 2023-05-31T20:39:14.571 DEBUG:boto:path=/pklxkd-58/obj00000000000000000036?versionId=null 2023-05-31T20:39:14.571 DEBUG:boto:auth_path=/pklxkd-58/obj00000000000000000036?versionId=null 2023-05-31T20:39:14.571 DEBUG:boto:Method: GET 2023-05-31T20:39:14.572 DEBUG:boto:Path: /pklxkd-58/obj00000000000000000036?versionId=null 2023-05-31T20:39:14.572 DEBUG:boto:Data: 2023-05-31T20:39:14.573 DEBUG:boto:Headers: {} 2023-05-31T20:39:14.573 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:39:14.573 DEBUG:boto:Port: 8001 2023-05-31T20:39:14.574 DEBUG:boto:Params: {} 2023-05-31T20:39:14.574 DEBUG:boto:Token: None 2023-05-31T20:39:14.574 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:39:14 GMT /pklxkd-58/obj00000000000000000036?versionId=null 2023-05-31T20:39:14.575 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:1m5Og3jJizMmS5Y+nb+mhw0t4HU= 2023-05-31T20:39:14.575 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:39:14 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:1m5Og3jJizMmS5Y+nb+mhw0t4HU=', 'Content-Length': '0'} 2023-05-31T20:39:14.577 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:38:22 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000052f193a918658586-006477b072-1c17-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:39:14 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:39:14.578 DEBUG:boto:path=/pklxkd-58/obj00000000000000000036 2023-05-31T20:39:14.578 DEBUG:boto:auth_path=/pklxkd-58/obj00000000000000000036 2023-05-31T20:39:14.579 DEBUG:boto:path=/pklxkd-58/obj00000000000000000036?versionId=null 2023-05-31T20:39:14.579 DEBUG:boto:auth_path=/pklxkd-58/obj00000000000000000036?versionId=null 2023-05-31T20:39:14.579 DEBUG:boto:Method: GET 2023-05-31T20:39:14.580 DEBUG:boto:Path: /pklxkd-58/obj00000000000000000036?versionId=null 2023-05-31T20:39:14.580 DEBUG:boto:Data: 2023-05-31T20:39:14.581 DEBUG:boto:Headers: {} 2023-05-31T20:39:14.581 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:39:14.581 DEBUG:boto:Port: 8000 2023-05-31T20:39:14.582 DEBUG:boto:Params: {} 2023-05-31T20:39:14.582 DEBUG:boto:Token: None 2023-05-31T20:39:14.583 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:39:14 GMT /pklxkd-58/obj00000000000000000036?versionId=null 2023-05-31T20:39:14.583 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:1m5Og3jJizMmS5Y+nb+mhw0t4HU= 2023-05-31T20:39:14.584 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:39:14 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:1m5Og3jJizMmS5Y+nb+mhw0t4HU=', 'Content-Length': '0'} 2023-05-31T20:39:14.585 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:38:22 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004e3d0508573824ce-006477b072-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:39:14 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:39:14.628 DEBUG:boto:path=/pklxkd-58/ 2023-05-31T20:39:14.629 DEBUG:boto:auth_path=/pklxkd-58/ 2023-05-31T20:39:14.629 DEBUG:boto:path=/pklxkd-58/?prefix=obj00000000000000000036 2023-05-31T20:39:14.630 DEBUG:boto:auth_path=/pklxkd-58/?prefix=obj00000000000000000036 2023-05-31T20:39:14.630 DEBUG:boto:Method: GET 2023-05-31T20:39:14.631 DEBUG:boto:Path: /pklxkd-58/?prefix=obj00000000000000000036 2023-05-31T20:39:14.631 DEBUG:boto:Data: 2023-05-31T20:39:14.631 DEBUG:boto:Headers: {} 2023-05-31T20:39:14.632 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:39:14.632 DEBUG:boto:Port: 8001 2023-05-31T20:39:14.632 DEBUG:boto:Params: {} 2023-05-31T20:39:14.633 DEBUG:boto:Token: None 2023-05-31T20:39:14.633 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:39:14 GMT /pklxkd-58/ 2023-05-31T20:39:14.634 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:WG6sc4Xwd7Q+ft8669zIyLEPh2A= 2023-05-31T20:39:14.634 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:39:14 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:WG6sc4Xwd7Q+ft8669zIyLEPh2A=', 'Content-Length': '0'} 2023-05-31T20:39:14.637 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000007cc1a56e27a325a7-006477b072-1c17-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:39:14 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:39:14.637 DEBUG:boto:b'pklxkd-58obj000000000000000000361000falseobj000000000000000000362023-05-31T20:38:22.636Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:39:14.638 DEBUG:boto:path=/pklxkd-58/ 2023-05-31T20:39:14.639 DEBUG:boto:auth_path=/pklxkd-58/ 2023-05-31T20:39:14.639 DEBUG:boto:path=/pklxkd-58/?prefix=obj00000000000000000036 2023-05-31T20:39:14.639 DEBUG:boto:auth_path=/pklxkd-58/?prefix=obj00000000000000000036 2023-05-31T20:39:14.640 DEBUG:boto:Method: GET 2023-05-31T20:39:14.640 DEBUG:boto:Path: /pklxkd-58/?prefix=obj00000000000000000036 2023-05-31T20:39:14.640 DEBUG:boto:Data: 2023-05-31T20:39:14.641 DEBUG:boto:Headers: {} 2023-05-31T20:39:14.641 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:39:14.641 DEBUG:boto:Port: 8000 2023-05-31T20:39:14.642 DEBUG:boto:Params: {} 2023-05-31T20:39:14.642 DEBUG:boto:Token: None 2023-05-31T20:39:14.643 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:39:14 GMT /pklxkd-58/ 2023-05-31T20:39:14.644 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:WG6sc4Xwd7Q+ft8669zIyLEPh2A= 2023-05-31T20:39:14.644 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:39:14 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:WG6sc4Xwd7Q+ft8669zIyLEPh2A=', 'Content-Length': '0'} 2023-05-31T20:39:14.650 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000095be4abca942b6c6-006477b072-1630-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:39:14 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:39:14.651 DEBUG:boto:b'pklxkd-58obj000000000000000000361000falseobj000000000000000000362023-05-31T20:38:22.636Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:39:14.652 DEBUG:boto:path=/pklxkd-58/obj00000000000000000036 2023-05-31T20:39:14.653 DEBUG:boto:auth_path=/pklxkd-58/obj00000000000000000036 2023-05-31T20:39:14.653 DEBUG:boto:Method: HEAD 2023-05-31T20:39:14.654 DEBUG:boto:Path: /pklxkd-58/obj00000000000000000036 2023-05-31T20:39:14.654 DEBUG:boto:Data: 2023-05-31T20:39:14.655 DEBUG:boto:Headers: {} 2023-05-31T20:39:14.655 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:39:14.656 DEBUG:boto:Port: 8001 2023-05-31T20:39:14.656 DEBUG:boto:Params: {} 2023-05-31T20:39:14.657 DEBUG:boto:Token: None 2023-05-31T20:39:14.658 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:39:14 GMT /pklxkd-58/obj00000000000000000036 2023-05-31T20:39:14.658 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:JM/cXsGWZg//tfCBTmhQNpJXFfM= 2023-05-31T20:39:14.659 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:39:14 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:JM/cXsGWZg//tfCBTmhQNpJXFfM=', 'Content-Length': '0'} 2023-05-31T20:39:14.661 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:38:22 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000097352a76ea1b920a-006477b072-1c17-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:39:14 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:39:14.662 DEBUG:boto:path=/pklxkd-58/obj00000000000000000036 2023-05-31T20:39:14.663 DEBUG:boto:auth_path=/pklxkd-58/obj00000000000000000036 2023-05-31T20:39:14.663 DEBUG:boto:Method: HEAD 2023-05-31T20:39:14.664 DEBUG:boto:Path: /pklxkd-58/obj00000000000000000036 2023-05-31T20:39:14.665 DEBUG:boto:Data: 2023-05-31T20:39:14.665 DEBUG:boto:Headers: {} 2023-05-31T20:39:14.666 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:39:14.667 DEBUG:boto:Port: 8000 2023-05-31T20:39:14.668 DEBUG:boto:Params: {} 2023-05-31T20:39:14.668 DEBUG:boto:Token: None 2023-05-31T20:39:14.669 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:39:14 GMT /pklxkd-58/obj00000000000000000036 2023-05-31T20:39:14.670 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:JM/cXsGWZg//tfCBTmhQNpJXFfM= 2023-05-31T20:39:14.671 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:39:14 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:JM/cXsGWZg//tfCBTmhQNpJXFfM=', 'Content-Length': '0'} 2023-05-31T20:39:14.673 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:38:22 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a9777d2021944644-006477b072-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:39:14 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:39:14.674 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000036 2023-05-31T20:39:14.675 DEBUG:boto:path=/pklxkd-58/obj00000000000000000036 2023-05-31T20:39:14.675 DEBUG:boto:auth_path=/pklxkd-58/obj00000000000000000036 2023-05-31T20:39:14.676 DEBUG:boto:Method: GET 2023-05-31T20:39:14.677 DEBUG:boto:Path: /pklxkd-58/obj00000000000000000036 2023-05-31T20:39:14.678 DEBUG:boto:Data: 2023-05-31T20:39:14.678 DEBUG:boto:Headers: {} 2023-05-31T20:39:14.679 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:39:14.680 DEBUG:boto:Port: 8001 2023-05-31T20:39:14.681 DEBUG:boto:Params: {} 2023-05-31T20:39:14.683 DEBUG:boto:Token: None 2023-05-31T20:39:14.684 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:39:14 GMT /pklxkd-58/obj00000000000000000036 2023-05-31T20:39:14.685 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:a33ys37gJk7wWMeXhQbnNhRNuCs= 2023-05-31T20:39:14.685 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:39:14 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:a33ys37gJk7wWMeXhQbnNhRNuCs=', 'Content-Length': '0'} 2023-05-31T20:39:14.688 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:38:22 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000076557371eec87693-006477b072-1c17-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:39:14 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:39:14.733 DEBUG:boto:path=/pklxkd-58/obj00000000000000000036 2023-05-31T20:39:14.734 DEBUG:boto:auth_path=/pklxkd-58/obj00000000000000000036 2023-05-31T20:39:14.734 DEBUG:boto:Method: GET 2023-05-31T20:39:14.735 DEBUG:boto:Path: /pklxkd-58/obj00000000000000000036 2023-05-31T20:39:14.735 DEBUG:boto:Data: 2023-05-31T20:39:14.736 DEBUG:boto:Headers: {} 2023-05-31T20:39:14.736 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:39:14.737 DEBUG:boto:Port: 8000 2023-05-31T20:39:14.737 DEBUG:boto:Params: {} 2023-05-31T20:39:14.738 DEBUG:boto:Token: None 2023-05-31T20:39:14.738 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:39:14 GMT /pklxkd-58/obj00000000000000000036 2023-05-31T20:39:14.739 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:a33ys37gJk7wWMeXhQbnNhRNuCs= 2023-05-31T20:39:14.739 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:39:14 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:a33ys37gJk7wWMeXhQbnNhRNuCs=', 'Content-Length': '0'} 2023-05-31T20:39:14.742 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:38:22 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000017e4a460e838d6a-006477b072-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:39:14 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:39:14.743 DEBUG:boto:path=/pklxkd-58/ 2023-05-31T20:39:14.744 DEBUG:boto:auth_path=/pklxkd-58/ 2023-05-31T20:39:14.744 DEBUG:boto:path=/pklxkd-58/?prefix=obj00000000000000000036 2023-05-31T20:39:14.745 DEBUG:boto:auth_path=/pklxkd-58/?prefix=obj00000000000000000036 2023-05-31T20:39:14.745 DEBUG:boto:Method: GET 2023-05-31T20:39:14.746 DEBUG:boto:Path: /pklxkd-58/?prefix=obj00000000000000000036 2023-05-31T20:39:14.746 DEBUG:boto:Data: 2023-05-31T20:39:14.746 DEBUG:boto:Headers: {} 2023-05-31T20:39:14.747 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:39:14.747 DEBUG:boto:Port: 8001 2023-05-31T20:39:14.747 DEBUG:boto:Params: {} 2023-05-31T20:39:14.748 DEBUG:boto:Token: None 2023-05-31T20:39:14.749 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:39:14 GMT /pklxkd-58/ 2023-05-31T20:39:14.750 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:WG6sc4Xwd7Q+ft8669zIyLEPh2A= 2023-05-31T20:39:14.750 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:39:14 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:WG6sc4Xwd7Q+ft8669zIyLEPh2A=', 'Content-Length': '0'} 2023-05-31T20:39:14.753 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c7e03db48668912f-006477b072-1c17-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:39:14 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:39:14.754 DEBUG:boto:b'pklxkd-58obj000000000000000000361000falseobj000000000000000000362023-05-31T20:38:22.636Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:39:14.756 DEBUG:boto:path=/pklxkd-58/ 2023-05-31T20:39:14.757 DEBUG:boto:auth_path=/pklxkd-58/ 2023-05-31T20:39:14.758 DEBUG:boto:path=/pklxkd-58/?prefix=obj00000000000000000036 2023-05-31T20:39:14.758 DEBUG:boto:auth_path=/pklxkd-58/?prefix=obj00000000000000000036 2023-05-31T20:39:14.759 DEBUG:boto:Method: GET 2023-05-31T20:39:14.759 DEBUG:boto:Path: /pklxkd-58/?prefix=obj00000000000000000036 2023-05-31T20:39:14.760 DEBUG:boto:Data: 2023-05-31T20:39:14.760 DEBUG:boto:Headers: {} 2023-05-31T20:39:14.760 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:39:14.761 DEBUG:boto:Port: 8000 2023-05-31T20:39:14.761 DEBUG:boto:Params: {} 2023-05-31T20:39:14.761 DEBUG:boto:Token: None 2023-05-31T20:39:14.762 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:39:14 GMT /pklxkd-58/ 2023-05-31T20:39:14.763 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:WG6sc4Xwd7Q+ft8669zIyLEPh2A= 2023-05-31T20:39:14.763 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:39:14 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:WG6sc4Xwd7Q+ft8669zIyLEPh2A=', 'Content-Length': '0'} 2023-05-31T20:39:14.767 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000af104c2571aff419-006477b072-1630-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:39:14 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:39:14.808 DEBUG:boto:b'pklxkd-58obj000000000000000000361000falseobj000000000000000000362023-05-31T20:38:22.636Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:39:14.810 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000037 2023-05-31T20:39:14.810 DEBUG:boto:path=/pklxkd-58/obj00000000000000000037 2023-05-31T20:39:14.811 DEBUG:boto:auth_path=/pklxkd-58/obj00000000000000000037 2023-05-31T20:39:14.813 DEBUG:boto:path=/pklxkd-58/obj00000000000000000037?versionId=null 2023-05-31T20:39:14.813 DEBUG:boto:auth_path=/pklxkd-58/obj00000000000000000037?versionId=null 2023-05-31T20:39:14.815 DEBUG:boto:Method: GET 2023-05-31T20:39:14.815 DEBUG:boto:Path: /pklxkd-58/obj00000000000000000037?versionId=null 2023-05-31T20:39:14.815 DEBUG:boto:Data: 2023-05-31T20:39:14.816 DEBUG:boto:Headers: {} 2023-05-31T20:39:14.817 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:39:14.817 DEBUG:boto:Port: 8001 2023-05-31T20:39:14.818 DEBUG:boto:Params: {} 2023-05-31T20:39:14.818 DEBUG:boto:Token: None 2023-05-31T20:39:14.818 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:39:14 GMT /pklxkd-58/obj00000000000000000037?versionId=null 2023-05-31T20:39:14.819 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:czDCsinCSy3tlyRvkhJZ/edG2Fg= 2023-05-31T20:39:14.820 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:39:14 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:czDCsinCSy3tlyRvkhJZ/edG2Fg=', 'Content-Length': '0'} 2023-05-31T20:39:14.825 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:38:19 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000edbaa9c9bcdaf6ba-006477b072-1c17-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:39:14 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:39:14.826 DEBUG:boto:path=/pklxkd-58/obj00000000000000000037 2023-05-31T20:39:14.826 DEBUG:boto:auth_path=/pklxkd-58/obj00000000000000000037 2023-05-31T20:39:14.827 DEBUG:boto:path=/pklxkd-58/obj00000000000000000037?versionId=null 2023-05-31T20:39:14.827 DEBUG:boto:auth_path=/pklxkd-58/obj00000000000000000037?versionId=null 2023-05-31T20:39:14.828 DEBUG:boto:Method: GET 2023-05-31T20:39:14.828 DEBUG:boto:Path: /pklxkd-58/obj00000000000000000037?versionId=null 2023-05-31T20:39:14.829 DEBUG:boto:Data: 2023-05-31T20:39:14.829 DEBUG:boto:Headers: {} 2023-05-31T20:39:14.830 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:39:14.830 DEBUG:boto:Port: 8000 2023-05-31T20:39:14.830 DEBUG:boto:Params: {} 2023-05-31T20:39:14.831 DEBUG:boto:Token: None 2023-05-31T20:39:14.831 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:39:14 GMT /pklxkd-58/obj00000000000000000037?versionId=null 2023-05-31T20:39:14.832 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:czDCsinCSy3tlyRvkhJZ/edG2Fg= 2023-05-31T20:39:14.832 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:39:14 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:czDCsinCSy3tlyRvkhJZ/edG2Fg=', 'Content-Length': '0'} 2023-05-31T20:39:14.834 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:38:19 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005df5106f90417759-006477b072-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:39:14 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:39:14.837 DEBUG:boto:path=/pklxkd-58/obj00000000000000000037 2023-05-31T20:39:14.838 DEBUG:boto:auth_path=/pklxkd-58/obj00000000000000000037 2023-05-31T20:39:14.839 DEBUG:boto:path=/pklxkd-58/obj00000000000000000037?versionId=null 2023-05-31T20:39:14.841 DEBUG:boto:auth_path=/pklxkd-58/obj00000000000000000037?versionId=null 2023-05-31T20:39:14.841 DEBUG:boto:Method: HEAD 2023-05-31T20:39:14.842 DEBUG:boto:Path: /pklxkd-58/obj00000000000000000037?versionId=null 2023-05-31T20:39:14.843 DEBUG:boto:Data: 2023-05-31T20:39:14.844 DEBUG:boto:Headers: {} 2023-05-31T20:39:14.844 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:39:14.845 DEBUG:boto:Port: 8001 2023-05-31T20:39:14.846 DEBUG:boto:Params: {} 2023-05-31T20:39:14.847 DEBUG:boto:Token: None 2023-05-31T20:39:14.847 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:39:14 GMT /pklxkd-58/obj00000000000000000037?versionId=null 2023-05-31T20:39:14.848 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:PsGHwoNJpYuvmYSTkLL5vlc6jBY= 2023-05-31T20:39:14.848 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:39:14 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:PsGHwoNJpYuvmYSTkLL5vlc6jBY=', 'Content-Length': '0'} 2023-05-31T20:39:14.850 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:38:19 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009b1bc7258cf22bfb-006477b072-1c17-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:39:14 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:39:14.851 DEBUG:boto:path=/pklxkd-58/obj00000000000000000037 2023-05-31T20:39:14.851 DEBUG:boto:auth_path=/pklxkd-58/obj00000000000000000037 2023-05-31T20:39:14.852 DEBUG:boto:path=/pklxkd-58/obj00000000000000000037?versionId=null 2023-05-31T20:39:14.852 DEBUG:boto:auth_path=/pklxkd-58/obj00000000000000000037?versionId=null 2023-05-31T20:39:14.853 DEBUG:boto:Method: HEAD 2023-05-31T20:39:14.853 DEBUG:boto:Path: /pklxkd-58/obj00000000000000000037?versionId=null 2023-05-31T20:39:14.853 DEBUG:boto:Data: 2023-05-31T20:39:14.854 DEBUG:boto:Headers: {} 2023-05-31T20:39:14.854 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:39:14.854 DEBUG:boto:Port: 8000 2023-05-31T20:39:14.855 DEBUG:boto:Params: {} 2023-05-31T20:39:14.855 DEBUG:boto:Token: None 2023-05-31T20:39:14.856 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:39:14 GMT /pklxkd-58/obj00000000000000000037?versionId=null 2023-05-31T20:39:14.856 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:PsGHwoNJpYuvmYSTkLL5vlc6jBY= 2023-05-31T20:39:14.857 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:39:14 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:PsGHwoNJpYuvmYSTkLL5vlc6jBY=', 'Content-Length': '0'} 2023-05-31T20:39:14.859 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:38:19 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000916114d8b3bd30be-006477b072-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:39:14 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:39:14.860 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000037 2023-05-31T20:39:14.860 DEBUG:boto:path=/pklxkd-58/obj00000000000000000037 2023-05-31T20:39:14.861 DEBUG:boto:auth_path=/pklxkd-58/obj00000000000000000037 2023-05-31T20:39:14.861 DEBUG:boto:path=/pklxkd-58/obj00000000000000000037?versionId=null 2023-05-31T20:39:14.862 DEBUG:boto:auth_path=/pklxkd-58/obj00000000000000000037?versionId=null 2023-05-31T20:39:14.862 DEBUG:boto:Method: GET 2023-05-31T20:39:14.863 DEBUG:boto:Path: /pklxkd-58/obj00000000000000000037?versionId=null 2023-05-31T20:39:14.863 DEBUG:boto:Data: 2023-05-31T20:39:14.864 DEBUG:boto:Headers: {} 2023-05-31T20:39:14.864 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:39:14.865 DEBUG:boto:Port: 8001 2023-05-31T20:39:14.865 DEBUG:boto:Params: {} 2023-05-31T20:39:14.866 DEBUG:boto:Token: None 2023-05-31T20:39:14.868 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:39:14 GMT /pklxkd-58/obj00000000000000000037?versionId=null 2023-05-31T20:39:14.868 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:czDCsinCSy3tlyRvkhJZ/edG2Fg= 2023-05-31T20:39:14.869 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:39:14 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:czDCsinCSy3tlyRvkhJZ/edG2Fg=', 'Content-Length': '0'} 2023-05-31T20:39:14.871 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:38:19 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000df002c998ce327bf-006477b072-1c17-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:39:14 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:39:14.872 DEBUG:boto:path=/pklxkd-58/obj00000000000000000037 2023-05-31T20:39:14.872 DEBUG:boto:auth_path=/pklxkd-58/obj00000000000000000037 2023-05-31T20:39:14.872 DEBUG:boto:path=/pklxkd-58/obj00000000000000000037?versionId=null 2023-05-31T20:39:14.873 DEBUG:boto:auth_path=/pklxkd-58/obj00000000000000000037?versionId=null 2023-05-31T20:39:14.873 DEBUG:boto:Method: GET 2023-05-31T20:39:14.873 DEBUG:boto:Path: /pklxkd-58/obj00000000000000000037?versionId=null 2023-05-31T20:39:14.874 DEBUG:boto:Data: 2023-05-31T20:39:14.874 DEBUG:boto:Headers: {} 2023-05-31T20:39:14.875 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:39:14.875 DEBUG:boto:Port: 8000 2023-05-31T20:39:14.875 DEBUG:boto:Params: {} 2023-05-31T20:39:14.876 DEBUG:boto:Token: None 2023-05-31T20:39:14.876 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:39:14 GMT /pklxkd-58/obj00000000000000000037?versionId=null 2023-05-31T20:39:14.877 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:czDCsinCSy3tlyRvkhJZ/edG2Fg= 2023-05-31T20:39:14.877 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:39:14 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:czDCsinCSy3tlyRvkhJZ/edG2Fg=', 'Content-Length': '0'} 2023-05-31T20:39:14.879 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:38:19 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000077b4936e81a5ec78-006477b072-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:39:14 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:39:14.880 DEBUG:boto:path=/pklxkd-58/ 2023-05-31T20:39:14.880 DEBUG:boto:auth_path=/pklxkd-58/ 2023-05-31T20:39:14.881 DEBUG:boto:path=/pklxkd-58/?prefix=obj00000000000000000037 2023-05-31T20:39:14.881 DEBUG:boto:auth_path=/pklxkd-58/?prefix=obj00000000000000000037 2023-05-31T20:39:14.881 DEBUG:boto:Method: GET 2023-05-31T20:39:14.882 DEBUG:boto:Path: /pklxkd-58/?prefix=obj00000000000000000037 2023-05-31T20:39:14.882 DEBUG:boto:Data: 2023-05-31T20:39:14.882 DEBUG:boto:Headers: {} 2023-05-31T20:39:14.883 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:39:14.883 DEBUG:boto:Port: 8001 2023-05-31T20:39:14.884 DEBUG:boto:Params: {} 2023-05-31T20:39:14.884 DEBUG:boto:Token: None 2023-05-31T20:39:14.885 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:39:14 GMT /pklxkd-58/ 2023-05-31T20:39:14.885 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:WG6sc4Xwd7Q+ft8669zIyLEPh2A= 2023-05-31T20:39:14.886 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:39:14 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:WG6sc4Xwd7Q+ft8669zIyLEPh2A=', 'Content-Length': '0'} 2023-05-31T20:39:14.888 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a8fa0d8e4d3a4f8f-006477b072-1c17-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:39:14 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:39:14.889 DEBUG:boto:b'pklxkd-58obj000000000000000000371000falseobj000000000000000000372023-05-31T20:38:19.358Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:39:14.890 DEBUG:boto:path=/pklxkd-58/ 2023-05-31T20:39:14.890 DEBUG:boto:auth_path=/pklxkd-58/ 2023-05-31T20:39:14.891 DEBUG:boto:path=/pklxkd-58/?prefix=obj00000000000000000037 2023-05-31T20:39:14.891 DEBUG:boto:auth_path=/pklxkd-58/?prefix=obj00000000000000000037 2023-05-31T20:39:14.891 DEBUG:boto:Method: GET 2023-05-31T20:39:14.892 DEBUG:boto:Path: /pklxkd-58/?prefix=obj00000000000000000037 2023-05-31T20:39:14.892 DEBUG:boto:Data: 2023-05-31T20:39:14.893 DEBUG:boto:Headers: {} 2023-05-31T20:39:14.893 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:39:14.893 DEBUG:boto:Port: 8000 2023-05-31T20:39:14.894 DEBUG:boto:Params: {} 2023-05-31T20:39:14.894 DEBUG:boto:Token: None 2023-05-31T20:39:14.895 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:39:14 GMT /pklxkd-58/ 2023-05-31T20:39:14.895 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:WG6sc4Xwd7Q+ft8669zIyLEPh2A= 2023-05-31T20:39:14.896 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:39:14 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:WG6sc4Xwd7Q+ft8669zIyLEPh2A=', 'Content-Length': '0'} 2023-05-31T20:39:14.900 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000073620dec077988d0-006477b072-1630-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:39:14 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:39:14.900 DEBUG:boto:b'pklxkd-58obj000000000000000000371000falseobj000000000000000000372023-05-31T20:38:19.358Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:39:14.901 DEBUG:boto:path=/pklxkd-58/obj00000000000000000037 2023-05-31T20:39:14.901 DEBUG:boto:auth_path=/pklxkd-58/obj00000000000000000037 2023-05-31T20:39:14.902 DEBUG:boto:Method: HEAD 2023-05-31T20:39:14.902 DEBUG:boto:Path: /pklxkd-58/obj00000000000000000037 2023-05-31T20:39:14.903 DEBUG:boto:Data: 2023-05-31T20:39:14.903 DEBUG:boto:Headers: {} 2023-05-31T20:39:14.903 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:39:14.904 DEBUG:boto:Port: 8001 2023-05-31T20:39:14.904 DEBUG:boto:Params: {} 2023-05-31T20:39:14.905 DEBUG:boto:Token: None 2023-05-31T20:39:14.905 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:39:14 GMT /pklxkd-58/obj00000000000000000037 2023-05-31T20:39:14.906 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:3QD1D06ygmMVEXO4xU65PnRu/hY= 2023-05-31T20:39:14.906 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:39:14 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:3QD1D06ygmMVEXO4xU65PnRu/hY=', 'Content-Length': '0'} 2023-05-31T20:39:14.908 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:38:19 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e6ba77ba1d2e8891-006477b072-1c17-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:39:14 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:39:14.909 DEBUG:boto:path=/pklxkd-58/obj00000000000000000037 2023-05-31T20:39:14.909 DEBUG:boto:auth_path=/pklxkd-58/obj00000000000000000037 2023-05-31T20:39:14.910 DEBUG:boto:Method: HEAD 2023-05-31T20:39:14.911 DEBUG:boto:Path: /pklxkd-58/obj00000000000000000037 2023-05-31T20:39:14.911 DEBUG:boto:Data: 2023-05-31T20:39:14.911 DEBUG:boto:Headers: {} 2023-05-31T20:39:14.912 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:39:14.912 DEBUG:boto:Port: 8000 2023-05-31T20:39:14.913 DEBUG:boto:Params: {} 2023-05-31T20:39:14.913 DEBUG:boto:Token: None 2023-05-31T20:39:14.914 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:39:14 GMT /pklxkd-58/obj00000000000000000037 2023-05-31T20:39:14.914 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:3QD1D06ygmMVEXO4xU65PnRu/hY= 2023-05-31T20:39:14.915 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:39:14 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:3QD1D06ygmMVEXO4xU65PnRu/hY=', 'Content-Length': '0'} 2023-05-31T20:39:14.917 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:38:19 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b2771afb893f4372-006477b072-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:39:14 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:39:14.917 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000037 2023-05-31T20:39:14.918 DEBUG:boto:path=/pklxkd-58/obj00000000000000000037 2023-05-31T20:39:14.918 DEBUG:boto:auth_path=/pklxkd-58/obj00000000000000000037 2023-05-31T20:39:14.919 DEBUG:boto:Method: GET 2023-05-31T20:39:14.919 DEBUG:boto:Path: /pklxkd-58/obj00000000000000000037 2023-05-31T20:39:14.919 DEBUG:boto:Data: 2023-05-31T20:39:14.920 DEBUG:boto:Headers: {} 2023-05-31T20:39:14.920 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:39:14.921 DEBUG:boto:Port: 8001 2023-05-31T20:39:14.921 DEBUG:boto:Params: {} 2023-05-31T20:39:14.922 DEBUG:boto:Token: None 2023-05-31T20:39:14.922 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:39:14 GMT /pklxkd-58/obj00000000000000000037 2023-05-31T20:39:14.923 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:XiuAcAVlRbrFak4HeBa8mk5j/78= 2023-05-31T20:39:14.923 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:39:14 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:XiuAcAVlRbrFak4HeBa8mk5j/78=', 'Content-Length': '0'} 2023-05-31T20:39:14.925 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:38:19 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000003f29bbc49d27e0a-006477b072-1c17-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:39:14 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:39:14.968 DEBUG:boto:path=/pklxkd-58/obj00000000000000000037 2023-05-31T20:39:14.969 DEBUG:boto:auth_path=/pklxkd-58/obj00000000000000000037 2023-05-31T20:39:14.969 DEBUG:boto:Method: GET 2023-05-31T20:39:14.970 DEBUG:boto:Path: /pklxkd-58/obj00000000000000000037 2023-05-31T20:39:14.970 DEBUG:boto:Data: 2023-05-31T20:39:14.971 DEBUG:boto:Headers: {} 2023-05-31T20:39:14.971 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:39:14.971 DEBUG:boto:Port: 8000 2023-05-31T20:39:14.972 DEBUG:boto:Params: {} 2023-05-31T20:39:14.973 DEBUG:boto:Token: None 2023-05-31T20:39:14.973 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:39:14 GMT /pklxkd-58/obj00000000000000000037 2023-05-31T20:39:14.974 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:XiuAcAVlRbrFak4HeBa8mk5j/78= 2023-05-31T20:39:14.974 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:39:14 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:XiuAcAVlRbrFak4HeBa8mk5j/78=', 'Content-Length': '0'} 2023-05-31T20:39:14.976 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:38:19 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009716f884edd24bf7-006477b072-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:39:14 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:39:14.977 DEBUG:boto:path=/pklxkd-58/ 2023-05-31T20:39:14.978 DEBUG:boto:auth_path=/pklxkd-58/ 2023-05-31T20:39:14.978 DEBUG:boto:path=/pklxkd-58/?prefix=obj00000000000000000037 2023-05-31T20:39:14.978 DEBUG:boto:auth_path=/pklxkd-58/?prefix=obj00000000000000000037 2023-05-31T20:39:14.979 DEBUG:boto:Method: GET 2023-05-31T20:39:14.979 DEBUG:boto:Path: /pklxkd-58/?prefix=obj00000000000000000037 2023-05-31T20:39:14.980 DEBUG:boto:Data: 2023-05-31T20:39:14.980 DEBUG:boto:Headers: {} 2023-05-31T20:39:14.981 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:39:14.981 DEBUG:boto:Port: 8001 2023-05-31T20:39:14.982 DEBUG:boto:Params: {} 2023-05-31T20:39:14.983 DEBUG:boto:Token: None 2023-05-31T20:39:14.983 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:39:14 GMT /pklxkd-58/ 2023-05-31T20:39:14.984 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:WG6sc4Xwd7Q+ft8669zIyLEPh2A= 2023-05-31T20:39:14.985 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:39:14 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:WG6sc4Xwd7Q+ft8669zIyLEPh2A=', 'Content-Length': '0'} 2023-05-31T20:39:14.988 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ff4c8d0c7852031e-006477b072-1c17-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:39:14 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:39:14.989 DEBUG:boto:b'pklxkd-58obj000000000000000000371000falseobj000000000000000000372023-05-31T20:38:19.358Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:39:14.990 DEBUG:boto:path=/pklxkd-58/ 2023-05-31T20:39:14.990 DEBUG:boto:auth_path=/pklxkd-58/ 2023-05-31T20:39:14.991 DEBUG:boto:path=/pklxkd-58/?prefix=obj00000000000000000037 2023-05-31T20:39:14.992 DEBUG:boto:auth_path=/pklxkd-58/?prefix=obj00000000000000000037 2023-05-31T20:39:14.993 DEBUG:boto:Method: GET 2023-05-31T20:39:14.993 DEBUG:boto:Path: /pklxkd-58/?prefix=obj00000000000000000037 2023-05-31T20:39:14.994 DEBUG:boto:Data: 2023-05-31T20:39:14.994 DEBUG:boto:Headers: {} 2023-05-31T20:39:14.995 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:39:14.995 DEBUG:boto:Port: 8000 2023-05-31T20:39:14.996 DEBUG:boto:Params: {} 2023-05-31T20:39:14.996 DEBUG:boto:Token: None 2023-05-31T20:39:14.997 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:39:14 GMT /pklxkd-58/ 2023-05-31T20:39:14.998 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:WG6sc4Xwd7Q+ft8669zIyLEPh2A= 2023-05-31T20:39:14.998 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:39:14 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:WG6sc4Xwd7Q+ft8669zIyLEPh2A=', 'Content-Length': '0'} 2023-05-31T20:39:15.002 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f0896705d773a299-006477b072-1630-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:39:14 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:39:15.004 DEBUG:boto:b'pklxkd-58obj000000000000000000371000falseobj000000000000000000372023-05-31T20:38:19.358Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:39:15.005 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000039 2023-05-31T20:39:15.008 DEBUG:boto:path=/pklxkd-58/obj00000000000000000039 2023-05-31T20:39:15.009 DEBUG:boto:auth_path=/pklxkd-58/obj00000000000000000039 2023-05-31T20:39:15.010 DEBUG:boto:path=/pklxkd-58/obj00000000000000000039?versionId=null 2023-05-31T20:39:15.010 DEBUG:boto:auth_path=/pklxkd-58/obj00000000000000000039?versionId=null 2023-05-31T20:39:15.011 DEBUG:boto:Method: GET 2023-05-31T20:39:15.012 DEBUG:boto:Path: /pklxkd-58/obj00000000000000000039?versionId=null 2023-05-31T20:39:15.012 DEBUG:boto:Data: 2023-05-31T20:39:15.012 DEBUG:boto:Headers: {} 2023-05-31T20:39:15.013 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:39:15.013 DEBUG:boto:Port: 8001 2023-05-31T20:39:15.014 DEBUG:boto:Params: {} 2023-05-31T20:39:15.014 DEBUG:boto:Token: None 2023-05-31T20:39:15.015 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:39:15 GMT /pklxkd-58/obj00000000000000000039?versionId=null 2023-05-31T20:39:15.015 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:E9XJdvA+9+hcCamgTBKVstMFoOM= 2023-05-31T20:39:15.016 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:39:15 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:E9XJdvA+9+hcCamgTBKVstMFoOM=', 'Content-Length': '0'} 2023-05-31T20:39:15.018 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:38:22 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009bfbdd14f8f2129c-006477b073-1c17-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:39:15 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:39:15.018 DEBUG:boto:path=/pklxkd-58/obj00000000000000000039 2023-05-31T20:39:15.019 DEBUG:boto:auth_path=/pklxkd-58/obj00000000000000000039 2023-05-31T20:39:15.020 DEBUG:boto:path=/pklxkd-58/obj00000000000000000039?versionId=null 2023-05-31T20:39:15.021 DEBUG:boto:auth_path=/pklxkd-58/obj00000000000000000039?versionId=null 2023-05-31T20:39:15.022 DEBUG:boto:Method: GET 2023-05-31T20:39:15.023 DEBUG:boto:Path: /pklxkd-58/obj00000000000000000039?versionId=null 2023-05-31T20:39:15.024 DEBUG:boto:Data: 2023-05-31T20:39:15.025 DEBUG:boto:Headers: {} 2023-05-31T20:39:15.026 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:39:15.026 DEBUG:boto:Port: 8000 2023-05-31T20:39:15.027 DEBUG:boto:Params: {} 2023-05-31T20:39:15.028 DEBUG:boto:Token: None 2023-05-31T20:39:15.029 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:39:15 GMT /pklxkd-58/obj00000000000000000039?versionId=null 2023-05-31T20:39:15.030 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:E9XJdvA+9+hcCamgTBKVstMFoOM= 2023-05-31T20:39:15.031 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:39:15 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:E9XJdvA+9+hcCamgTBKVstMFoOM=', 'Content-Length': '0'} 2023-05-31T20:39:15.033 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:38:22 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000151874111c769733-006477b073-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:39:15 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:39:15.077 DEBUG:boto:path=/pklxkd-58/obj00000000000000000039 2023-05-31T20:39:15.078 DEBUG:boto:auth_path=/pklxkd-58/obj00000000000000000039 2023-05-31T20:39:15.079 DEBUG:boto:path=/pklxkd-58/obj00000000000000000039?versionId=null 2023-05-31T20:39:15.080 DEBUG:boto:auth_path=/pklxkd-58/obj00000000000000000039?versionId=null 2023-05-31T20:39:15.080 DEBUG:boto:Method: HEAD 2023-05-31T20:39:15.081 DEBUG:boto:Path: /pklxkd-58/obj00000000000000000039?versionId=null 2023-05-31T20:39:15.082 DEBUG:boto:Data: 2023-05-31T20:39:15.082 DEBUG:boto:Headers: {} 2023-05-31T20:39:15.083 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:39:15.084 DEBUG:boto:Port: 8001 2023-05-31T20:39:15.084 DEBUG:boto:Params: {} 2023-05-31T20:39:15.085 DEBUG:boto:Token: None 2023-05-31T20:39:15.086 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:39:15 GMT /pklxkd-58/obj00000000000000000039?versionId=null 2023-05-31T20:39:15.086 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:ufzGSjWi9cvqZJipEkHqHKWgvyw= 2023-05-31T20:39:15.087 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:39:15 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:ufzGSjWi9cvqZJipEkHqHKWgvyw=', 'Content-Length': '0'} 2023-05-31T20:39:15.089 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:38:22 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006ea948789d065615-006477b073-1c17-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:39:15 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:39:15.089 DEBUG:boto:path=/pklxkd-58/obj00000000000000000039 2023-05-31T20:39:15.090 DEBUG:boto:auth_path=/pklxkd-58/obj00000000000000000039 2023-05-31T20:39:15.090 DEBUG:boto:path=/pklxkd-58/obj00000000000000000039?versionId=null 2023-05-31T20:39:15.091 DEBUG:boto:auth_path=/pklxkd-58/obj00000000000000000039?versionId=null 2023-05-31T20:39:15.091 DEBUG:boto:Method: HEAD 2023-05-31T20:39:15.092 DEBUG:boto:Path: /pklxkd-58/obj00000000000000000039?versionId=null 2023-05-31T20:39:15.093 DEBUG:boto:Data: 2023-05-31T20:39:15.093 DEBUG:boto:Headers: {} 2023-05-31T20:39:15.093 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:39:15.094 DEBUG:boto:Port: 8000 2023-05-31T20:39:15.094 DEBUG:boto:Params: {} 2023-05-31T20:39:15.095 DEBUG:boto:Token: None 2023-05-31T20:39:15.095 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:39:15 GMT /pklxkd-58/obj00000000000000000039?versionId=null 2023-05-31T20:39:15.096 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:ufzGSjWi9cvqZJipEkHqHKWgvyw= 2023-05-31T20:39:15.096 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:39:15 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:ufzGSjWi9cvqZJipEkHqHKWgvyw=', 'Content-Length': '0'} 2023-05-31T20:39:15.098 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:38:22 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b0a208c97958691d-006477b073-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:39:15 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:39:15.099 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000039 2023-05-31T20:39:15.100 DEBUG:boto:path=/pklxkd-58/obj00000000000000000039 2023-05-31T20:39:15.101 DEBUG:boto:auth_path=/pklxkd-58/obj00000000000000000039 2023-05-31T20:39:15.101 DEBUG:boto:path=/pklxkd-58/obj00000000000000000039?versionId=null 2023-05-31T20:39:15.102 DEBUG:boto:auth_path=/pklxkd-58/obj00000000000000000039?versionId=null 2023-05-31T20:39:15.103 DEBUG:boto:Method: GET 2023-05-31T20:39:15.104 DEBUG:boto:Path: /pklxkd-58/obj00000000000000000039?versionId=null 2023-05-31T20:39:15.104 DEBUG:boto:Data: 2023-05-31T20:39:15.105 DEBUG:boto:Headers: {} 2023-05-31T20:39:15.106 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:39:15.107 DEBUG:boto:Port: 8001 2023-05-31T20:39:15.108 DEBUG:boto:Params: {} 2023-05-31T20:39:15.109 DEBUG:boto:Token: None 2023-05-31T20:39:15.109 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:39:15 GMT /pklxkd-58/obj00000000000000000039?versionId=null 2023-05-31T20:39:15.110 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:E9XJdvA+9+hcCamgTBKVstMFoOM= 2023-05-31T20:39:15.110 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:39:15 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:E9XJdvA+9+hcCamgTBKVstMFoOM=', 'Content-Length': '0'} 2023-05-31T20:39:15.113 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:38:22 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a82f2868bf51e168-006477b073-1c17-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:39:15 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:39:15.114 DEBUG:boto:path=/pklxkd-58/obj00000000000000000039 2023-05-31T20:39:15.115 DEBUG:boto:auth_path=/pklxkd-58/obj00000000000000000039 2023-05-31T20:39:15.116 DEBUG:boto:path=/pklxkd-58/obj00000000000000000039?versionId=null 2023-05-31T20:39:15.117 DEBUG:boto:auth_path=/pklxkd-58/obj00000000000000000039?versionId=null 2023-05-31T20:39:15.118 DEBUG:boto:Method: GET 2023-05-31T20:39:15.119 DEBUG:boto:Path: /pklxkd-58/obj00000000000000000039?versionId=null 2023-05-31T20:39:15.119 DEBUG:boto:Data: 2023-05-31T20:39:15.120 DEBUG:boto:Headers: {} 2023-05-31T20:39:15.121 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:39:15.121 DEBUG:boto:Port: 8000 2023-05-31T20:39:15.122 DEBUG:boto:Params: {} 2023-05-31T20:39:15.122 DEBUG:boto:Token: None 2023-05-31T20:39:15.123 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:39:15 GMT /pklxkd-58/obj00000000000000000039?versionId=null 2023-05-31T20:39:15.124 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:E9XJdvA+9+hcCamgTBKVstMFoOM= 2023-05-31T20:39:15.125 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:39:15 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:E9XJdvA+9+hcCamgTBKVstMFoOM=', 'Content-Length': '0'} 2023-05-31T20:39:15.128 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:38:22 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d45c540821b3cca0-006477b073-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:39:15 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:39:15.129 DEBUG:boto:path=/pklxkd-58/ 2023-05-31T20:39:15.130 DEBUG:boto:auth_path=/pklxkd-58/ 2023-05-31T20:39:15.131 DEBUG:boto:path=/pklxkd-58/?prefix=obj00000000000000000039 2023-05-31T20:39:15.132 DEBUG:boto:auth_path=/pklxkd-58/?prefix=obj00000000000000000039 2023-05-31T20:39:15.132 DEBUG:boto:Method: GET 2023-05-31T20:39:15.133 DEBUG:boto:Path: /pklxkd-58/?prefix=obj00000000000000000039 2023-05-31T20:39:15.134 DEBUG:boto:Data: 2023-05-31T20:39:15.134 DEBUG:boto:Headers: {} 2023-05-31T20:39:15.135 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:39:15.136 DEBUG:boto:Port: 8001 2023-05-31T20:39:15.137 DEBUG:boto:Params: {} 2023-05-31T20:39:15.138 DEBUG:boto:Token: None 2023-05-31T20:39:15.139 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:39:15 GMT /pklxkd-58/ 2023-05-31T20:39:15.139 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:kp0WjK8lKKWPr4lFyLKkBJk2eCo= 2023-05-31T20:39:15.140 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:39:15 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:kp0WjK8lKKWPr4lFyLKkBJk2eCo=', 'Content-Length': '0'} 2023-05-31T20:39:15.143 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000004e349ffd89164e97-006477b073-1c17-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:39:15 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:39:15.184 DEBUG:boto:b'pklxkd-58obj000000000000000000391000falseobj000000000000000000392023-05-31T20:38:22.383Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:39:15.185 DEBUG:boto:path=/pklxkd-58/ 2023-05-31T20:39:15.186 DEBUG:boto:auth_path=/pklxkd-58/ 2023-05-31T20:39:15.186 DEBUG:boto:path=/pklxkd-58/?prefix=obj00000000000000000039 2023-05-31T20:39:15.187 DEBUG:boto:auth_path=/pklxkd-58/?prefix=obj00000000000000000039 2023-05-31T20:39:15.187 DEBUG:boto:Method: GET 2023-05-31T20:39:15.188 DEBUG:boto:Path: /pklxkd-58/?prefix=obj00000000000000000039 2023-05-31T20:39:15.189 DEBUG:boto:Data: 2023-05-31T20:39:15.189 DEBUG:boto:Headers: {} 2023-05-31T20:39:15.189 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:39:15.190 DEBUG:boto:Port: 8000 2023-05-31T20:39:15.190 DEBUG:boto:Params: {} 2023-05-31T20:39:15.191 DEBUG:boto:Token: None 2023-05-31T20:39:15.191 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:39:15 GMT /pklxkd-58/ 2023-05-31T20:39:15.192 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:kp0WjK8lKKWPr4lFyLKkBJk2eCo= 2023-05-31T20:39:15.192 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:39:15 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:kp0WjK8lKKWPr4lFyLKkBJk2eCo=', 'Content-Length': '0'} 2023-05-31T20:39:15.196 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000628e927ff22f44b0-006477b073-1630-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:39:15 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:39:15.197 DEBUG:boto:b'pklxkd-58obj000000000000000000391000falseobj000000000000000000392023-05-31T20:38:22.383Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:39:15.199 DEBUG:boto:path=/pklxkd-58/obj00000000000000000039 2023-05-31T20:39:15.200 DEBUG:boto:auth_path=/pklxkd-58/obj00000000000000000039 2023-05-31T20:39:15.201 DEBUG:boto:Method: HEAD 2023-05-31T20:39:15.202 DEBUG:boto:Path: /pklxkd-58/obj00000000000000000039 2023-05-31T20:39:15.203 DEBUG:boto:Data: 2023-05-31T20:39:15.203 DEBUG:boto:Headers: {} 2023-05-31T20:39:15.204 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:39:15.205 DEBUG:boto:Port: 8001 2023-05-31T20:39:15.205 DEBUG:boto:Params: {} 2023-05-31T20:39:15.206 DEBUG:boto:Token: None 2023-05-31T20:39:15.207 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:39:15 GMT /pklxkd-58/obj00000000000000000039 2023-05-31T20:39:15.208 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:EzHLIGau3OYpHMNmpw6YwxqOWTI= 2023-05-31T20:39:15.208 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:39:15 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:EzHLIGau3OYpHMNmpw6YwxqOWTI=', 'Content-Length': '0'} 2023-05-31T20:39:15.211 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:38:22 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003760ad027a7b0652-006477b073-1c17-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:39:15 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:39:15.212 DEBUG:boto:path=/pklxkd-58/obj00000000000000000039 2023-05-31T20:39:15.212 DEBUG:boto:auth_path=/pklxkd-58/obj00000000000000000039 2023-05-31T20:39:15.214 DEBUG:boto:Method: HEAD 2023-05-31T20:39:15.214 DEBUG:boto:Path: /pklxkd-58/obj00000000000000000039 2023-05-31T20:39:15.215 DEBUG:boto:Data: 2023-05-31T20:39:15.215 DEBUG:boto:Headers: {} 2023-05-31T20:39:15.216 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:39:15.216 DEBUG:boto:Port: 8000 2023-05-31T20:39:15.217 DEBUG:boto:Params: {} 2023-05-31T20:39:15.218 DEBUG:boto:Token: None 2023-05-31T20:39:15.218 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:39:15 GMT /pklxkd-58/obj00000000000000000039 2023-05-31T20:39:15.219 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:EzHLIGau3OYpHMNmpw6YwxqOWTI= 2023-05-31T20:39:15.219 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:39:15 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:EzHLIGau3OYpHMNmpw6YwxqOWTI=', 'Content-Length': '0'} 2023-05-31T20:39:15.221 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:38:22 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000066905cdf0542a6e8-006477b073-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:39:15 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:39:15.222 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000039 2023-05-31T20:39:15.222 DEBUG:boto:path=/pklxkd-58/obj00000000000000000039 2023-05-31T20:39:15.223 DEBUG:boto:auth_path=/pklxkd-58/obj00000000000000000039 2023-05-31T20:39:15.223 DEBUG:boto:Method: GET 2023-05-31T20:39:15.224 DEBUG:boto:Path: /pklxkd-58/obj00000000000000000039 2023-05-31T20:39:15.224 DEBUG:boto:Data: 2023-05-31T20:39:15.225 DEBUG:boto:Headers: {} 2023-05-31T20:39:15.225 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:39:15.226 DEBUG:boto:Port: 8001 2023-05-31T20:39:15.226 DEBUG:boto:Params: {} 2023-05-31T20:39:15.227 DEBUG:boto:Token: None 2023-05-31T20:39:15.228 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:39:15 GMT /pklxkd-58/obj00000000000000000039 2023-05-31T20:39:15.229 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:u1Ps4MXy2QgdQcVyfi2aRWaZ19M= 2023-05-31T20:39:15.230 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:39:15 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:u1Ps4MXy2QgdQcVyfi2aRWaZ19M=', 'Content-Length': '0'} 2023-05-31T20:39:15.232 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:38:22 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b0684db95def2642-006477b073-1c17-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:39:15 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:39:15.234 DEBUG:boto:path=/pklxkd-58/obj00000000000000000039 2023-05-31T20:39:15.235 DEBUG:boto:auth_path=/pklxkd-58/obj00000000000000000039 2023-05-31T20:39:15.235 DEBUG:boto:Method: GET 2023-05-31T20:39:15.236 DEBUG:boto:Path: /pklxkd-58/obj00000000000000000039 2023-05-31T20:39:15.237 DEBUG:boto:Data: 2023-05-31T20:39:15.238 DEBUG:boto:Headers: {} 2023-05-31T20:39:15.239 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:39:15.240 DEBUG:boto:Port: 8000 2023-05-31T20:39:15.241 DEBUG:boto:Params: {} 2023-05-31T20:39:15.241 DEBUG:boto:Token: None 2023-05-31T20:39:15.242 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:39:15 GMT /pklxkd-58/obj00000000000000000039 2023-05-31T20:39:15.242 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:u1Ps4MXy2QgdQcVyfi2aRWaZ19M= 2023-05-31T20:39:15.243 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:39:15 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:u1Ps4MXy2QgdQcVyfi2aRWaZ19M=', 'Content-Length': '0'} 2023-05-31T20:39:15.245 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:38:22 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f645980f4cf1b1ba-006477b073-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:39:15 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:39:15.246 DEBUG:boto:path=/pklxkd-58/ 2023-05-31T20:39:15.246 DEBUG:boto:auth_path=/pklxkd-58/ 2023-05-31T20:39:15.246 DEBUG:boto:path=/pklxkd-58/?prefix=obj00000000000000000039 2023-05-31T20:39:15.247 DEBUG:boto:auth_path=/pklxkd-58/?prefix=obj00000000000000000039 2023-05-31T20:39:15.247 DEBUG:boto:Method: GET 2023-05-31T20:39:15.247 DEBUG:boto:Path: /pklxkd-58/?prefix=obj00000000000000000039 2023-05-31T20:39:15.248 DEBUG:boto:Data: 2023-05-31T20:39:15.248 DEBUG:boto:Headers: {} 2023-05-31T20:39:15.249 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:39:15.249 DEBUG:boto:Port: 8001 2023-05-31T20:39:15.250 DEBUG:boto:Params: {} 2023-05-31T20:39:15.251 DEBUG:boto:Token: None 2023-05-31T20:39:15.251 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:39:15 GMT /pklxkd-58/ 2023-05-31T20:39:15.252 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:kp0WjK8lKKWPr4lFyLKkBJk2eCo= 2023-05-31T20:39:15.253 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:39:15 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:kp0WjK8lKKWPr4lFyLKkBJk2eCo=', 'Content-Length': '0'} 2023-05-31T20:39:15.256 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000acfc8d573d571ec6-006477b073-1c17-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:39:15 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:39:15.257 DEBUG:boto:b'pklxkd-58obj000000000000000000391000falseobj000000000000000000392023-05-31T20:38:22.383Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:39:15.258 DEBUG:boto:path=/pklxkd-58/ 2023-05-31T20:39:15.258 DEBUG:boto:auth_path=/pklxkd-58/ 2023-05-31T20:39:15.258 DEBUG:boto:path=/pklxkd-58/?prefix=obj00000000000000000039 2023-05-31T20:39:15.259 DEBUG:boto:auth_path=/pklxkd-58/?prefix=obj00000000000000000039 2023-05-31T20:39:15.259 DEBUG:boto:Method: GET 2023-05-31T20:39:15.260 DEBUG:boto:Path: /pklxkd-58/?prefix=obj00000000000000000039 2023-05-31T20:39:15.260 DEBUG:boto:Data: 2023-05-31T20:39:15.261 DEBUG:boto:Headers: {} 2023-05-31T20:39:15.261 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:39:15.262 DEBUG:boto:Port: 8000 2023-05-31T20:39:15.262 DEBUG:boto:Params: {} 2023-05-31T20:39:15.263 DEBUG:boto:Token: None 2023-05-31T20:39:15.263 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:39:15 GMT /pklxkd-58/ 2023-05-31T20:39:15.264 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:kp0WjK8lKKWPr4lFyLKkBJk2eCo= 2023-05-31T20:39:15.264 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:39:15 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:kp0WjK8lKKWPr4lFyLKkBJk2eCo=', 'Content-Length': '0'} 2023-05-31T20:39:15.268 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000000cd166cb5c22281f-006477b073-1630-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:39:15 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:39:15.312 DEBUG:boto:b'pklxkd-58obj000000000000000000391000falseobj000000000000000000392023-05-31T20:38:22.383Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:39:15.313 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000040 2023-05-31T20:39:15.314 DEBUG:boto:path=/pklxkd-58/obj00000000000000000040 2023-05-31T20:39:15.314 DEBUG:boto:auth_path=/pklxkd-58/obj00000000000000000040 2023-05-31T20:39:15.315 DEBUG:boto:path=/pklxkd-58/obj00000000000000000040?versionId=null 2023-05-31T20:39:15.315 DEBUG:boto:auth_path=/pklxkd-58/obj00000000000000000040?versionId=null 2023-05-31T20:39:15.316 DEBUG:boto:Method: GET 2023-05-31T20:39:15.316 DEBUG:boto:Path: /pklxkd-58/obj00000000000000000040?versionId=null 2023-05-31T20:39:15.316 DEBUG:boto:Data: 2023-05-31T20:39:15.317 DEBUG:boto:Headers: {} 2023-05-31T20:39:15.317 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:39:15.318 DEBUG:boto:Port: 8001 2023-05-31T20:39:15.318 DEBUG:boto:Params: {} 2023-05-31T20:39:15.318 DEBUG:boto:Token: None 2023-05-31T20:39:15.319 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:39:15 GMT /pklxkd-58/obj00000000000000000040?versionId=null 2023-05-31T20:39:15.319 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:oq7ucCai5AxYrGF/W70twr5YssY= 2023-05-31T20:39:15.320 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:39:15 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:oq7ucCai5AxYrGF/W70twr5YssY=', 'Content-Length': '0'} 2023-05-31T20:39:15.322 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:38:22 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c20e667a769d64cd-006477b073-1c17-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:39:15 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:39:15.323 DEBUG:boto:path=/pklxkd-58/obj00000000000000000040 2023-05-31T20:39:15.324 DEBUG:boto:auth_path=/pklxkd-58/obj00000000000000000040 2023-05-31T20:39:15.325 DEBUG:boto:path=/pklxkd-58/obj00000000000000000040?versionId=null 2023-05-31T20:39:15.326 DEBUG:boto:auth_path=/pklxkd-58/obj00000000000000000040?versionId=null 2023-05-31T20:39:15.327 DEBUG:boto:Method: GET 2023-05-31T20:39:15.328 DEBUG:boto:Path: /pklxkd-58/obj00000000000000000040?versionId=null 2023-05-31T20:39:15.328 DEBUG:boto:Data: 2023-05-31T20:39:15.329 DEBUG:boto:Headers: {} 2023-05-31T20:39:15.330 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:39:15.330 DEBUG:boto:Port: 8000 2023-05-31T20:39:15.331 DEBUG:boto:Params: {} 2023-05-31T20:39:15.331 DEBUG:boto:Token: None 2023-05-31T20:39:15.332 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:39:15 GMT /pklxkd-58/obj00000000000000000040?versionId=null 2023-05-31T20:39:15.333 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:oq7ucCai5AxYrGF/W70twr5YssY= 2023-05-31T20:39:15.334 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:39:15 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:oq7ucCai5AxYrGF/W70twr5YssY=', 'Content-Length': '0'} 2023-05-31T20:39:15.337 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:38:22 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001e35c2d4be3db135-006477b073-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:39:15 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:39:15.337 DEBUG:boto:path=/pklxkd-58/obj00000000000000000040 2023-05-31T20:39:15.338 DEBUG:boto:auth_path=/pklxkd-58/obj00000000000000000040 2023-05-31T20:39:15.338 DEBUG:boto:path=/pklxkd-58/obj00000000000000000040?versionId=null 2023-05-31T20:39:15.338 DEBUG:boto:auth_path=/pklxkd-58/obj00000000000000000040?versionId=null 2023-05-31T20:39:15.339 DEBUG:boto:Method: HEAD 2023-05-31T20:39:15.339 DEBUG:boto:Path: /pklxkd-58/obj00000000000000000040?versionId=null 2023-05-31T20:39:15.339 DEBUG:boto:Data: 2023-05-31T20:39:15.340 DEBUG:boto:Headers: {} 2023-05-31T20:39:15.340 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:39:15.340 DEBUG:boto:Port: 8001 2023-05-31T20:39:15.341 DEBUG:boto:Params: {} 2023-05-31T20:39:15.341 DEBUG:boto:Token: None 2023-05-31T20:39:15.342 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:39:15 GMT /pklxkd-58/obj00000000000000000040?versionId=null 2023-05-31T20:39:15.342 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:If9mtwIj1Xk++2Zitf1qTfu5Hs4= 2023-05-31T20:39:15.343 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:39:15 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:If9mtwIj1Xk++2Zitf1qTfu5Hs4=', 'Content-Length': '0'} 2023-05-31T20:39:15.345 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:38:22 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007c918739fa458cc8-006477b073-1c17-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:39:15 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:39:15.345 DEBUG:boto:path=/pklxkd-58/obj00000000000000000040 2023-05-31T20:39:15.346 DEBUG:boto:auth_path=/pklxkd-58/obj00000000000000000040 2023-05-31T20:39:15.346 DEBUG:boto:path=/pklxkd-58/obj00000000000000000040?versionId=null 2023-05-31T20:39:15.346 DEBUG:boto:auth_path=/pklxkd-58/obj00000000000000000040?versionId=null 2023-05-31T20:39:15.347 DEBUG:boto:Method: HEAD 2023-05-31T20:39:15.347 DEBUG:boto:Path: /pklxkd-58/obj00000000000000000040?versionId=null 2023-05-31T20:39:15.347 DEBUG:boto:Data: 2023-05-31T20:39:15.348 DEBUG:boto:Headers: {} 2023-05-31T20:39:15.348 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:39:15.348 DEBUG:boto:Port: 8000 2023-05-31T20:39:15.349 DEBUG:boto:Params: {} 2023-05-31T20:39:15.349 DEBUG:boto:Token: None 2023-05-31T20:39:15.350 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:39:15 GMT /pklxkd-58/obj00000000000000000040?versionId=null 2023-05-31T20:39:15.350 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:If9mtwIj1Xk++2Zitf1qTfu5Hs4= 2023-05-31T20:39:15.350 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:39:15 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:If9mtwIj1Xk++2Zitf1qTfu5Hs4=', 'Content-Length': '0'} 2023-05-31T20:39:15.352 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:38:22 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000aaf8165e5f2b8f60-006477b073-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:39:15 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:39:15.353 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000040 2023-05-31T20:39:15.353 DEBUG:boto:path=/pklxkd-58/obj00000000000000000040 2023-05-31T20:39:15.354 DEBUG:boto:auth_path=/pklxkd-58/obj00000000000000000040 2023-05-31T20:39:15.354 DEBUG:boto:path=/pklxkd-58/obj00000000000000000040?versionId=null 2023-05-31T20:39:15.354 DEBUG:boto:auth_path=/pklxkd-58/obj00000000000000000040?versionId=null 2023-05-31T20:39:15.355 DEBUG:boto:Method: GET 2023-05-31T20:39:15.355 DEBUG:boto:Path: /pklxkd-58/obj00000000000000000040?versionId=null 2023-05-31T20:39:15.355 DEBUG:boto:Data: 2023-05-31T20:39:15.356 DEBUG:boto:Headers: {} 2023-05-31T20:39:15.356 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:39:15.356 DEBUG:boto:Port: 8001 2023-05-31T20:39:15.357 DEBUG:boto:Params: {} 2023-05-31T20:39:15.357 DEBUG:boto:Token: None 2023-05-31T20:39:15.358 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:39:15 GMT /pklxkd-58/obj00000000000000000040?versionId=null 2023-05-31T20:39:15.358 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:oq7ucCai5AxYrGF/W70twr5YssY= 2023-05-31T20:39:15.359 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:39:15 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:oq7ucCai5AxYrGF/W70twr5YssY=', 'Content-Length': '0'} 2023-05-31T20:39:15.360 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:38:22 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000df950838644cbfa8-006477b073-1c17-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:39:15 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:39:15.404 DEBUG:boto:path=/pklxkd-58/obj00000000000000000040 2023-05-31T20:39:15.405 DEBUG:boto:auth_path=/pklxkd-58/obj00000000000000000040 2023-05-31T20:39:15.406 DEBUG:boto:path=/pklxkd-58/obj00000000000000000040?versionId=null 2023-05-31T20:39:15.407 DEBUG:boto:auth_path=/pklxkd-58/obj00000000000000000040?versionId=null 2023-05-31T20:39:15.408 DEBUG:boto:Method: GET 2023-05-31T20:39:15.409 DEBUG:boto:Path: /pklxkd-58/obj00000000000000000040?versionId=null 2023-05-31T20:39:15.409 DEBUG:boto:Data: 2023-05-31T20:39:15.411 DEBUG:boto:Headers: {} 2023-05-31T20:39:15.411 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:39:15.412 DEBUG:boto:Port: 8000 2023-05-31T20:39:15.413 DEBUG:boto:Params: {} 2023-05-31T20:39:15.414 DEBUG:boto:Token: None 2023-05-31T20:39:15.416 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:39:15 GMT /pklxkd-58/obj00000000000000000040?versionId=null 2023-05-31T20:39:15.417 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:oq7ucCai5AxYrGF/W70twr5YssY= 2023-05-31T20:39:15.418 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:39:15 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:oq7ucCai5AxYrGF/W70twr5YssY=', 'Content-Length': '0'} 2023-05-31T20:39:15.421 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:38:22 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003881932523c17699-006477b073-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:39:15 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:39:15.422 DEBUG:boto:path=/pklxkd-58/ 2023-05-31T20:39:15.423 DEBUG:boto:auth_path=/pklxkd-58/ 2023-05-31T20:39:15.424 DEBUG:boto:path=/pklxkd-58/?prefix=obj00000000000000000040 2023-05-31T20:39:15.425 DEBUG:boto:auth_path=/pklxkd-58/?prefix=obj00000000000000000040 2023-05-31T20:39:15.426 DEBUG:boto:Method: GET 2023-05-31T20:39:15.427 DEBUG:boto:Path: /pklxkd-58/?prefix=obj00000000000000000040 2023-05-31T20:39:15.428 DEBUG:boto:Data: 2023-05-31T20:39:15.428 DEBUG:boto:Headers: {} 2023-05-31T20:39:15.429 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:39:15.430 DEBUG:boto:Port: 8001 2023-05-31T20:39:15.430 DEBUG:boto:Params: {} 2023-05-31T20:39:15.431 DEBUG:boto:Token: None 2023-05-31T20:39:15.432 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:39:15 GMT /pklxkd-58/ 2023-05-31T20:39:15.433 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:kp0WjK8lKKWPr4lFyLKkBJk2eCo= 2023-05-31T20:39:15.434 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:39:15 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:kp0WjK8lKKWPr4lFyLKkBJk2eCo=', 'Content-Length': '0'} 2023-05-31T20:39:15.437 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000094e999ea954be192-006477b073-1c17-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:39:15 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:39:15.438 DEBUG:boto:b'pklxkd-58obj000000000000000000401000falseobj000000000000000000402023-05-31T20:38:22.327Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:39:15.439 DEBUG:boto:path=/pklxkd-58/ 2023-05-31T20:39:15.440 DEBUG:boto:auth_path=/pklxkd-58/ 2023-05-31T20:39:15.440 DEBUG:boto:path=/pklxkd-58/?prefix=obj00000000000000000040 2023-05-31T20:39:15.441 DEBUG:boto:auth_path=/pklxkd-58/?prefix=obj00000000000000000040 2023-05-31T20:39:15.442 DEBUG:boto:Method: GET 2023-05-31T20:39:15.442 DEBUG:boto:Path: /pklxkd-58/?prefix=obj00000000000000000040 2023-05-31T20:39:15.443 DEBUG:boto:Data: 2023-05-31T20:39:15.443 DEBUG:boto:Headers: {} 2023-05-31T20:39:15.443 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:39:15.444 DEBUG:boto:Port: 8000 2023-05-31T20:39:15.444 DEBUG:boto:Params: {} 2023-05-31T20:39:15.445 DEBUG:boto:Token: None 2023-05-31T20:39:15.446 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:39:15 GMT /pklxkd-58/ 2023-05-31T20:39:15.447 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:kp0WjK8lKKWPr4lFyLKkBJk2eCo= 2023-05-31T20:39:15.447 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:39:15 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:kp0WjK8lKKWPr4lFyLKkBJk2eCo=', 'Content-Length': '0'} 2023-05-31T20:39:15.451 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ab82f8de19384749-006477b073-1630-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:39:15 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:39:15.496 DEBUG:boto:b'pklxkd-58obj000000000000000000401000falseobj000000000000000000402023-05-31T20:38:22.327Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:39:15.498 DEBUG:boto:path=/pklxkd-58/obj00000000000000000040 2023-05-31T20:39:15.499 DEBUG:boto:auth_path=/pklxkd-58/obj00000000000000000040 2023-05-31T20:39:15.499 DEBUG:boto:Method: HEAD 2023-05-31T20:39:15.500 DEBUG:boto:Path: /pklxkd-58/obj00000000000000000040 2023-05-31T20:39:15.500 DEBUG:boto:Data: 2023-05-31T20:39:15.501 DEBUG:boto:Headers: {} 2023-05-31T20:39:15.501 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:39:15.502 DEBUG:boto:Port: 8001 2023-05-31T20:39:15.503 DEBUG:boto:Params: {} 2023-05-31T20:39:15.504 DEBUG:boto:Token: None 2023-05-31T20:39:15.505 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:39:15 GMT /pklxkd-58/obj00000000000000000040 2023-05-31T20:39:15.506 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:MtcN3A+Rtl5O2Fn6z0/c+6rzQ2w= 2023-05-31T20:39:15.507 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:39:15 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:MtcN3A+Rtl5O2Fn6z0/c+6rzQ2w=', 'Content-Length': '0'} 2023-05-31T20:39:15.510 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:38:22 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000229e31f2f54a2e66-006477b073-1c17-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:39:15 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:39:15.511 DEBUG:boto:path=/pklxkd-58/obj00000000000000000040 2023-05-31T20:39:15.512 DEBUG:boto:auth_path=/pklxkd-58/obj00000000000000000040 2023-05-31T20:39:15.512 DEBUG:boto:Method: HEAD 2023-05-31T20:39:15.513 DEBUG:boto:Path: /pklxkd-58/obj00000000000000000040 2023-05-31T20:39:15.514 DEBUG:boto:Data: 2023-05-31T20:39:15.515 DEBUG:boto:Headers: {} 2023-05-31T20:39:15.516 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:39:15.517 DEBUG:boto:Port: 8000 2023-05-31T20:39:15.518 DEBUG:boto:Params: {} 2023-05-31T20:39:15.519 DEBUG:boto:Token: None 2023-05-31T20:39:15.520 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:39:15 GMT /pklxkd-58/obj00000000000000000040 2023-05-31T20:39:15.521 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:MtcN3A+Rtl5O2Fn6z0/c+6rzQ2w= 2023-05-31T20:39:15.521 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:39:15 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:MtcN3A+Rtl5O2Fn6z0/c+6rzQ2w=', 'Content-Length': '0'} 2023-05-31T20:39:15.523 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:38:22 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000dc53df389b7c23f7-006477b073-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:39:15 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:39:15.524 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000040 2023-05-31T20:39:15.525 DEBUG:boto:path=/pklxkd-58/obj00000000000000000040 2023-05-31T20:39:15.526 DEBUG:boto:auth_path=/pklxkd-58/obj00000000000000000040 2023-05-31T20:39:15.527 DEBUG:boto:Method: GET 2023-05-31T20:39:15.528 DEBUG:boto:Path: /pklxkd-58/obj00000000000000000040 2023-05-31T20:39:15.529 DEBUG:boto:Data: 2023-05-31T20:39:15.529 DEBUG:boto:Headers: {} 2023-05-31T20:39:15.529 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:39:15.530 DEBUG:boto:Port: 8001 2023-05-31T20:39:15.530 DEBUG:boto:Params: {} 2023-05-31T20:39:15.531 DEBUG:boto:Token: None 2023-05-31T20:39:15.531 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:39:15 GMT /pklxkd-58/obj00000000000000000040 2023-05-31T20:39:15.532 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:GtUflTlvwLV6RmoCGSII+akrH8U= 2023-05-31T20:39:15.533 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:39:15 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:GtUflTlvwLV6RmoCGSII+akrH8U=', 'Content-Length': '0'} 2023-05-31T20:39:15.536 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:38:22 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000be4466a9805139d9-006477b073-1c17-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:39:15 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:39:15.536 DEBUG:boto:path=/pklxkd-58/obj00000000000000000040 2023-05-31T20:39:15.537 DEBUG:boto:auth_path=/pklxkd-58/obj00000000000000000040 2023-05-31T20:39:15.537 DEBUG:boto:Method: GET 2023-05-31T20:39:15.538 DEBUG:boto:Path: /pklxkd-58/obj00000000000000000040 2023-05-31T20:39:15.538 DEBUG:boto:Data: 2023-05-31T20:39:15.538 DEBUG:boto:Headers: {} 2023-05-31T20:39:15.539 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:39:15.539 DEBUG:boto:Port: 8000 2023-05-31T20:39:15.540 DEBUG:boto:Params: {} 2023-05-31T20:39:15.541 DEBUG:boto:Token: None 2023-05-31T20:39:15.541 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:39:15 GMT /pklxkd-58/obj00000000000000000040 2023-05-31T20:39:15.542 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:GtUflTlvwLV6RmoCGSII+akrH8U= 2023-05-31T20:39:15.543 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:39:15 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:GtUflTlvwLV6RmoCGSII+akrH8U=', 'Content-Length': '0'} 2023-05-31T20:39:15.545 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:38:22 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000de6f93a664d25647-006477b073-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:39:15 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:39:15.547 DEBUG:boto:path=/pklxkd-58/ 2023-05-31T20:39:15.547 DEBUG:boto:auth_path=/pklxkd-58/ 2023-05-31T20:39:15.548 DEBUG:boto:path=/pklxkd-58/?prefix=obj00000000000000000040 2023-05-31T20:39:15.548 DEBUG:boto:auth_path=/pklxkd-58/?prefix=obj00000000000000000040 2023-05-31T20:39:15.549 DEBUG:boto:Method: GET 2023-05-31T20:39:15.550 DEBUG:boto:Path: /pklxkd-58/?prefix=obj00000000000000000040 2023-05-31T20:39:15.550 DEBUG:boto:Data: 2023-05-31T20:39:15.551 DEBUG:boto:Headers: {} 2023-05-31T20:39:15.552 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:39:15.552 DEBUG:boto:Port: 8001 2023-05-31T20:39:15.553 DEBUG:boto:Params: {} 2023-05-31T20:39:15.554 DEBUG:boto:Token: None 2023-05-31T20:39:15.554 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:39:15 GMT /pklxkd-58/ 2023-05-31T20:39:15.555 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:kp0WjK8lKKWPr4lFyLKkBJk2eCo= 2023-05-31T20:39:15.556 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:39:15 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:kp0WjK8lKKWPr4lFyLKkBJk2eCo=', 'Content-Length': '0'} 2023-05-31T20:39:15.559 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b624837d55da02cc-006477b073-1c17-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:39:15 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:39:15.561 DEBUG:boto:b'pklxkd-58obj000000000000000000401000falseobj000000000000000000402023-05-31T20:38:22.327Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:39:15.562 DEBUG:boto:path=/pklxkd-58/ 2023-05-31T20:39:15.563 DEBUG:boto:auth_path=/pklxkd-58/ 2023-05-31T20:39:15.563 DEBUG:boto:path=/pklxkd-58/?prefix=obj00000000000000000040 2023-05-31T20:39:15.564 DEBUG:boto:auth_path=/pklxkd-58/?prefix=obj00000000000000000040 2023-05-31T20:39:15.564 DEBUG:boto:Method: GET 2023-05-31T20:39:15.565 DEBUG:boto:Path: /pklxkd-58/?prefix=obj00000000000000000040 2023-05-31T20:39:15.566 DEBUG:boto:Data: 2023-05-31T20:39:15.566 DEBUG:boto:Headers: {} 2023-05-31T20:39:15.567 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:39:15.567 DEBUG:boto:Port: 8000 2023-05-31T20:39:15.568 DEBUG:boto:Params: {} 2023-05-31T20:39:15.568 DEBUG:boto:Token: None 2023-05-31T20:39:15.569 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:39:15 GMT /pklxkd-58/ 2023-05-31T20:39:15.569 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:kp0WjK8lKKWPr4lFyLKkBJk2eCo= 2023-05-31T20:39:15.569 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:39:15 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:kp0WjK8lKKWPr4lFyLKkBJk2eCo=', 'Content-Length': '0'} 2023-05-31T20:39:15.573 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000021c16ff0be3312f0-006477b073-1630-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:39:15 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:39:15.574 DEBUG:boto:b'pklxkd-58obj000000000000000000401000falseobj000000000000000000402023-05-31T20:38:22.327Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:39:15.575 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000042 2023-05-31T20:39:15.576 DEBUG:boto:path=/pklxkd-58/obj00000000000000000042 2023-05-31T20:39:15.576 DEBUG:boto:auth_path=/pklxkd-58/obj00000000000000000042 2023-05-31T20:39:15.577 DEBUG:boto:path=/pklxkd-58/obj00000000000000000042?versionId=null 2023-05-31T20:39:15.577 DEBUG:boto:auth_path=/pklxkd-58/obj00000000000000000042?versionId=null 2023-05-31T20:39:15.578 DEBUG:boto:Method: GET 2023-05-31T20:39:15.578 DEBUG:boto:Path: /pklxkd-58/obj00000000000000000042?versionId=null 2023-05-31T20:39:15.579 DEBUG:boto:Data: 2023-05-31T20:39:15.579 DEBUG:boto:Headers: {} 2023-05-31T20:39:15.579 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:39:15.580 DEBUG:boto:Port: 8001 2023-05-31T20:39:15.581 DEBUG:boto:Params: {} 2023-05-31T20:39:15.582 DEBUG:boto:Token: None 2023-05-31T20:39:15.583 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:39:15 GMT /pklxkd-58/obj00000000000000000042?versionId=null 2023-05-31T20:39:15.583 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:ZY2S6cU3PF8SePE0I1NphprZPsg= 2023-05-31T20:39:15.584 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:39:15 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:ZY2S6cU3PF8SePE0I1NphprZPsg=', 'Content-Length': '0'} 2023-05-31T20:39:15.587 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:38:18 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005a8aee53441f021b-006477b073-1c17-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:39:15 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:39:15.588 DEBUG:boto:path=/pklxkd-58/obj00000000000000000042 2023-05-31T20:39:15.589 DEBUG:boto:auth_path=/pklxkd-58/obj00000000000000000042 2023-05-31T20:39:15.590 DEBUG:boto:path=/pklxkd-58/obj00000000000000000042?versionId=null 2023-05-31T20:39:15.590 DEBUG:boto:auth_path=/pklxkd-58/obj00000000000000000042?versionId=null 2023-05-31T20:39:15.591 DEBUG:boto:Method: GET 2023-05-31T20:39:15.592 DEBUG:boto:Path: /pklxkd-58/obj00000000000000000042?versionId=null 2023-05-31T20:39:15.593 DEBUG:boto:Data: 2023-05-31T20:39:15.594 DEBUG:boto:Headers: {} 2023-05-31T20:39:15.594 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:39:15.595 DEBUG:boto:Port: 8000 2023-05-31T20:39:15.596 DEBUG:boto:Params: {} 2023-05-31T20:39:15.597 DEBUG:boto:Token: None 2023-05-31T20:39:15.597 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:39:15 GMT /pklxkd-58/obj00000000000000000042?versionId=null 2023-05-31T20:39:15.598 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:ZY2S6cU3PF8SePE0I1NphprZPsg= 2023-05-31T20:39:15.599 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:39:15 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:ZY2S6cU3PF8SePE0I1NphprZPsg=', 'Content-Length': '0'} 2023-05-31T20:39:15.601 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:38:18 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d37dbc2ef9a7dc02-006477b073-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:39:15 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:39:15.602 DEBUG:boto:path=/pklxkd-58/obj00000000000000000042 2023-05-31T20:39:15.603 DEBUG:boto:auth_path=/pklxkd-58/obj00000000000000000042 2023-05-31T20:39:15.604 DEBUG:boto:path=/pklxkd-58/obj00000000000000000042?versionId=null 2023-05-31T20:39:15.604 DEBUG:boto:auth_path=/pklxkd-58/obj00000000000000000042?versionId=null 2023-05-31T20:39:15.605 DEBUG:boto:Method: HEAD 2023-05-31T20:39:15.605 DEBUG:boto:Path: /pklxkd-58/obj00000000000000000042?versionId=null 2023-05-31T20:39:15.605 DEBUG:boto:Data: 2023-05-31T20:39:15.606 DEBUG:boto:Headers: {} 2023-05-31T20:39:15.606 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:39:15.606 DEBUG:boto:Port: 8001 2023-05-31T20:39:15.607 DEBUG:boto:Params: {} 2023-05-31T20:39:15.607 DEBUG:boto:Token: None 2023-05-31T20:39:15.607 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:39:15 GMT /pklxkd-58/obj00000000000000000042?versionId=null 2023-05-31T20:39:15.608 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:lzfglGPAt5IDve7RfdaH4ei2mc0= 2023-05-31T20:39:15.608 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:39:15 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:lzfglGPAt5IDve7RfdaH4ei2mc0=', 'Content-Length': '0'} 2023-05-31T20:39:15.610 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:38:18 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003493d7e85a391bef-006477b073-1c17-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:39:15 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:39:15.611 DEBUG:boto:path=/pklxkd-58/obj00000000000000000042 2023-05-31T20:39:15.611 DEBUG:boto:auth_path=/pklxkd-58/obj00000000000000000042 2023-05-31T20:39:15.611 DEBUG:boto:path=/pklxkd-58/obj00000000000000000042?versionId=null 2023-05-31T20:39:15.612 DEBUG:boto:auth_path=/pklxkd-58/obj00000000000000000042?versionId=null 2023-05-31T20:39:15.612 DEBUG:boto:Method: HEAD 2023-05-31T20:39:15.613 DEBUG:boto:Path: /pklxkd-58/obj00000000000000000042?versionId=null 2023-05-31T20:39:15.613 DEBUG:boto:Data: 2023-05-31T20:39:15.613 DEBUG:boto:Headers: {} 2023-05-31T20:39:15.614 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:39:15.614 DEBUG:boto:Port: 8000 2023-05-31T20:39:15.614 DEBUG:boto:Params: {} 2023-05-31T20:39:15.615 DEBUG:boto:Token: None 2023-05-31T20:39:15.615 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:39:15 GMT /pklxkd-58/obj00000000000000000042?versionId=null 2023-05-31T20:39:15.616 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:lzfglGPAt5IDve7RfdaH4ei2mc0= 2023-05-31T20:39:15.616 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:39:15 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:lzfglGPAt5IDve7RfdaH4ei2mc0=', 'Content-Length': '0'} 2023-05-31T20:39:15.618 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:38:18 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004cbe06166c0c9b51-006477b073-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:39:15 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:39:15.618 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000042 2023-05-31T20:39:15.619 DEBUG:boto:path=/pklxkd-58/obj00000000000000000042 2023-05-31T20:39:15.619 DEBUG:boto:auth_path=/pklxkd-58/obj00000000000000000042 2023-05-31T20:39:15.620 DEBUG:boto:path=/pklxkd-58/obj00000000000000000042?versionId=null 2023-05-31T20:39:15.620 DEBUG:boto:auth_path=/pklxkd-58/obj00000000000000000042?versionId=null 2023-05-31T20:39:15.621 DEBUG:boto:Method: GET 2023-05-31T20:39:15.621 DEBUG:boto:Path: /pklxkd-58/obj00000000000000000042?versionId=null 2023-05-31T20:39:15.621 DEBUG:boto:Data: 2023-05-31T20:39:15.622 DEBUG:boto:Headers: {} 2023-05-31T20:39:15.622 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:39:15.623 DEBUG:boto:Port: 8001 2023-05-31T20:39:15.623 DEBUG:boto:Params: {} 2023-05-31T20:39:15.623 DEBUG:boto:Token: None 2023-05-31T20:39:15.624 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:39:15 GMT /pklxkd-58/obj00000000000000000042?versionId=null 2023-05-31T20:39:15.624 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:ZY2S6cU3PF8SePE0I1NphprZPsg= 2023-05-31T20:39:15.625 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:39:15 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:ZY2S6cU3PF8SePE0I1NphprZPsg=', 'Content-Length': '0'} 2023-05-31T20:39:15.627 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:38:18 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e0966ebc6629eed8-006477b073-1c17-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:39:15 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:39:15.627 DEBUG:boto:path=/pklxkd-58/obj00000000000000000042 2023-05-31T20:39:15.628 DEBUG:boto:auth_path=/pklxkd-58/obj00000000000000000042 2023-05-31T20:39:15.628 DEBUG:boto:path=/pklxkd-58/obj00000000000000000042?versionId=null 2023-05-31T20:39:15.629 DEBUG:boto:auth_path=/pklxkd-58/obj00000000000000000042?versionId=null 2023-05-31T20:39:15.629 DEBUG:boto:Method: GET 2023-05-31T20:39:15.629 DEBUG:boto:Path: /pklxkd-58/obj00000000000000000042?versionId=null 2023-05-31T20:39:15.630 DEBUG:boto:Data: 2023-05-31T20:39:15.630 DEBUG:boto:Headers: {} 2023-05-31T20:39:15.630 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:39:15.631 DEBUG:boto:Port: 8000 2023-05-31T20:39:15.631 DEBUG:boto:Params: {} 2023-05-31T20:39:15.632 DEBUG:boto:Token: None 2023-05-31T20:39:15.632 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:39:15 GMT /pklxkd-58/obj00000000000000000042?versionId=null 2023-05-31T20:39:15.633 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:ZY2S6cU3PF8SePE0I1NphprZPsg= 2023-05-31T20:39:15.633 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:39:15 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:ZY2S6cU3PF8SePE0I1NphprZPsg=', 'Content-Length': '0'} 2023-05-31T20:39:15.635 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:38:18 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ac8d18199420c20d-006477b073-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:39:15 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:39:15.676 DEBUG:boto:path=/pklxkd-58/ 2023-05-31T20:39:15.678 DEBUG:boto:auth_path=/pklxkd-58/ 2023-05-31T20:39:15.679 DEBUG:boto:path=/pklxkd-58/?prefix=obj00000000000000000042 2023-05-31T20:39:15.679 DEBUG:boto:auth_path=/pklxkd-58/?prefix=obj00000000000000000042 2023-05-31T20:39:15.680 DEBUG:boto:Method: GET 2023-05-31T20:39:15.681 DEBUG:boto:Path: /pklxkd-58/?prefix=obj00000000000000000042 2023-05-31T20:39:15.682 DEBUG:boto:Data: 2023-05-31T20:39:15.683 DEBUG:boto:Headers: {} 2023-05-31T20:39:15.683 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:39:15.684 DEBUG:boto:Port: 8001 2023-05-31T20:39:15.685 DEBUG:boto:Params: {} 2023-05-31T20:39:15.686 DEBUG:boto:Token: None 2023-05-31T20:39:15.686 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:39:15 GMT /pklxkd-58/ 2023-05-31T20:39:15.687 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:kp0WjK8lKKWPr4lFyLKkBJk2eCo= 2023-05-31T20:39:15.687 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:39:15 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:kp0WjK8lKKWPr4lFyLKkBJk2eCo=', 'Content-Length': '0'} 2023-05-31T20:39:15.690 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000009fdef792cf263905-006477b073-1c17-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:39:15 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:39:15.690 DEBUG:boto:b'pklxkd-58obj000000000000000000421000falseobj000000000000000000422023-05-31T20:38:18.977Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:39:15.691 DEBUG:boto:path=/pklxkd-58/ 2023-05-31T20:39:15.691 DEBUG:boto:auth_path=/pklxkd-58/ 2023-05-31T20:39:15.692 DEBUG:boto:path=/pklxkd-58/?prefix=obj00000000000000000042 2023-05-31T20:39:15.692 DEBUG:boto:auth_path=/pklxkd-58/?prefix=obj00000000000000000042 2023-05-31T20:39:15.693 DEBUG:boto:Method: GET 2023-05-31T20:39:15.693 DEBUG:boto:Path: /pklxkd-58/?prefix=obj00000000000000000042 2023-05-31T20:39:15.694 DEBUG:boto:Data: 2023-05-31T20:39:15.694 DEBUG:boto:Headers: {} 2023-05-31T20:39:15.694 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:39:15.695 DEBUG:boto:Port: 8000 2023-05-31T20:39:15.696 DEBUG:boto:Params: {} 2023-05-31T20:39:15.696 DEBUG:boto:Token: None 2023-05-31T20:39:15.697 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:39:15 GMT /pklxkd-58/ 2023-05-31T20:39:15.697 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:kp0WjK8lKKWPr4lFyLKkBJk2eCo= 2023-05-31T20:39:15.698 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:39:15 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:kp0WjK8lKKWPr4lFyLKkBJk2eCo=', 'Content-Length': '0'} 2023-05-31T20:39:15.702 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000004aa454de2aef7554-006477b073-1630-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:39:15 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:39:15.703 DEBUG:boto:b'pklxkd-58obj000000000000000000421000falseobj000000000000000000422023-05-31T20:38:18.977Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:39:15.704 DEBUG:boto:path=/pklxkd-58/obj00000000000000000042 2023-05-31T20:39:15.704 DEBUG:boto:auth_path=/pklxkd-58/obj00000000000000000042 2023-05-31T20:39:15.705 DEBUG:boto:Method: HEAD 2023-05-31T20:39:15.706 DEBUG:boto:Path: /pklxkd-58/obj00000000000000000042 2023-05-31T20:39:15.707 DEBUG:boto:Data: 2023-05-31T20:39:15.707 DEBUG:boto:Headers: {} 2023-05-31T20:39:15.708 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:39:15.709 DEBUG:boto:Port: 8001 2023-05-31T20:39:15.709 DEBUG:boto:Params: {} 2023-05-31T20:39:15.711 DEBUG:boto:Token: None 2023-05-31T20:39:15.712 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:39:15 GMT /pklxkd-58/obj00000000000000000042 2023-05-31T20:39:15.712 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:9J1LJrHd++PB5t625M4dFx2PMPo= 2023-05-31T20:39:15.713 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:39:15 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:9J1LJrHd++PB5t625M4dFx2PMPo=', 'Content-Length': '0'} 2023-05-31T20:39:15.715 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:38:18 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ba9a85e8033339a8-006477b073-1c17-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:39:15 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:39:15.715 DEBUG:boto:path=/pklxkd-58/obj00000000000000000042 2023-05-31T20:39:15.716 DEBUG:boto:auth_path=/pklxkd-58/obj00000000000000000042 2023-05-31T20:39:15.716 DEBUG:boto:Method: HEAD 2023-05-31T20:39:15.717 DEBUG:boto:Path: /pklxkd-58/obj00000000000000000042 2023-05-31T20:39:15.717 DEBUG:boto:Data: 2023-05-31T20:39:15.717 DEBUG:boto:Headers: {} 2023-05-31T20:39:15.718 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:39:15.718 DEBUG:boto:Port: 8000 2023-05-31T20:39:15.718 DEBUG:boto:Params: {} 2023-05-31T20:39:15.719 DEBUG:boto:Token: None 2023-05-31T20:39:15.719 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:39:15 GMT /pklxkd-58/obj00000000000000000042 2023-05-31T20:39:15.720 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:9J1LJrHd++PB5t625M4dFx2PMPo= 2023-05-31T20:39:15.720 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:39:15 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:9J1LJrHd++PB5t625M4dFx2PMPo=', 'Content-Length': '0'} 2023-05-31T20:39:15.722 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:38:18 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a293dbe7284779b9-006477b073-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:39:15 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:39:15.723 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000042 2023-05-31T20:39:15.724 DEBUG:boto:path=/pklxkd-58/obj00000000000000000042 2023-05-31T20:39:15.725 DEBUG:boto:auth_path=/pklxkd-58/obj00000000000000000042 2023-05-31T20:39:15.726 DEBUG:boto:Method: GET 2023-05-31T20:39:15.727 DEBUG:boto:Path: /pklxkd-58/obj00000000000000000042 2023-05-31T20:39:15.728 DEBUG:boto:Data: 2023-05-31T20:39:15.729 DEBUG:boto:Headers: {} 2023-05-31T20:39:15.730 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:39:15.731 DEBUG:boto:Port: 8001 2023-05-31T20:39:15.731 DEBUG:boto:Params: {} 2023-05-31T20:39:15.732 DEBUG:boto:Token: None 2023-05-31T20:39:15.733 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:39:15 GMT /pklxkd-58/obj00000000000000000042 2023-05-31T20:39:15.734 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:DPi3dKKNhCQX/BMvKFcu43wNSZ4= 2023-05-31T20:39:15.734 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:39:15 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:DPi3dKKNhCQX/BMvKFcu43wNSZ4=', 'Content-Length': '0'} 2023-05-31T20:39:15.736 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:38:18 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000013b9688094124c86-006477b073-1c17-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:39:15 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:39:15.737 DEBUG:boto:path=/pklxkd-58/obj00000000000000000042 2023-05-31T20:39:15.737 DEBUG:boto:auth_path=/pklxkd-58/obj00000000000000000042 2023-05-31T20:39:15.738 DEBUG:boto:Method: GET 2023-05-31T20:39:15.738 DEBUG:boto:Path: /pklxkd-58/obj00000000000000000042 2023-05-31T20:39:15.739 DEBUG:boto:Data: 2023-05-31T20:39:15.739 DEBUG:boto:Headers: {} 2023-05-31T20:39:15.739 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:39:15.740 DEBUG:boto:Port: 8000 2023-05-31T20:39:15.741 DEBUG:boto:Params: {} 2023-05-31T20:39:15.741 DEBUG:boto:Token: None 2023-05-31T20:39:15.742 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:39:15 GMT /pklxkd-58/obj00000000000000000042 2023-05-31T20:39:15.742 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:DPi3dKKNhCQX/BMvKFcu43wNSZ4= 2023-05-31T20:39:15.743 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:39:15 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:DPi3dKKNhCQX/BMvKFcu43wNSZ4=', 'Content-Length': '0'} 2023-05-31T20:39:15.745 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:38:18 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008b5ad103c1378ffe-006477b073-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:39:15 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:39:15.746 DEBUG:boto:path=/pklxkd-58/ 2023-05-31T20:39:15.746 DEBUG:boto:auth_path=/pklxkd-58/ 2023-05-31T20:39:15.747 DEBUG:boto:path=/pklxkd-58/?prefix=obj00000000000000000042 2023-05-31T20:39:15.748 DEBUG:boto:auth_path=/pklxkd-58/?prefix=obj00000000000000000042 2023-05-31T20:39:15.748 DEBUG:boto:Method: GET 2023-05-31T20:39:15.749 DEBUG:boto:Path: /pklxkd-58/?prefix=obj00000000000000000042 2023-05-31T20:39:15.750 DEBUG:boto:Data: 2023-05-31T20:39:15.750 DEBUG:boto:Headers: {} 2023-05-31T20:39:15.751 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:39:15.751 DEBUG:boto:Port: 8001 2023-05-31T20:39:15.752 DEBUG:boto:Params: {} 2023-05-31T20:39:15.753 DEBUG:boto:Token: None 2023-05-31T20:39:15.754 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:39:15 GMT /pklxkd-58/ 2023-05-31T20:39:15.754 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:kp0WjK8lKKWPr4lFyLKkBJk2eCo= 2023-05-31T20:39:15.755 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:39:15 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:kp0WjK8lKKWPr4lFyLKkBJk2eCo=', 'Content-Length': '0'} 2023-05-31T20:39:15.759 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000db5ce5eaf285286d-006477b073-1c17-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:39:15 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:39:15.804 DEBUG:boto:b'pklxkd-58obj000000000000000000421000falseobj000000000000000000422023-05-31T20:38:18.977Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:39:15.805 DEBUG:boto:path=/pklxkd-58/ 2023-05-31T20:39:15.806 DEBUG:boto:auth_path=/pklxkd-58/ 2023-05-31T20:39:15.806 DEBUG:boto:path=/pklxkd-58/?prefix=obj00000000000000000042 2023-05-31T20:39:15.807 DEBUG:boto:auth_path=/pklxkd-58/?prefix=obj00000000000000000042 2023-05-31T20:39:15.808 DEBUG:boto:Method: GET 2023-05-31T20:39:15.809 DEBUG:boto:Path: /pklxkd-58/?prefix=obj00000000000000000042 2023-05-31T20:39:15.809 DEBUG:boto:Data: 2023-05-31T20:39:15.809 DEBUG:boto:Headers: {} 2023-05-31T20:39:15.810 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:39:15.810 DEBUG:boto:Port: 8000 2023-05-31T20:39:15.810 DEBUG:boto:Params: {} 2023-05-31T20:39:15.811 DEBUG:boto:Token: None 2023-05-31T20:39:15.811 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:39:15 GMT /pklxkd-58/ 2023-05-31T20:39:15.812 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:kp0WjK8lKKWPr4lFyLKkBJk2eCo= 2023-05-31T20:39:15.813 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:39:15 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:kp0WjK8lKKWPr4lFyLKkBJk2eCo=', 'Content-Length': '0'} 2023-05-31T20:39:15.816 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000032a663c1492591af-006477b073-1630-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:39:15 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:39:15.817 DEBUG:boto:b'pklxkd-58obj000000000000000000421000falseobj000000000000000000422023-05-31T20:38:18.977Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:39:15.818 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000045 2023-05-31T20:39:15.818 DEBUG:boto:path=/pklxkd-58/obj00000000000000000045 2023-05-31T20:39:15.818 DEBUG:boto:auth_path=/pklxkd-58/obj00000000000000000045 2023-05-31T20:39:15.819 DEBUG:boto:path=/pklxkd-58/obj00000000000000000045?versionId=null 2023-05-31T20:39:15.819 DEBUG:boto:auth_path=/pklxkd-58/obj00000000000000000045?versionId=null 2023-05-31T20:39:15.820 DEBUG:boto:Method: GET 2023-05-31T20:39:15.820 DEBUG:boto:Path: /pklxkd-58/obj00000000000000000045?versionId=null 2023-05-31T20:39:15.820 DEBUG:boto:Data: 2023-05-31T20:39:15.821 DEBUG:boto:Headers: {} 2023-05-31T20:39:15.821 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:39:15.821 DEBUG:boto:Port: 8001 2023-05-31T20:39:15.822 DEBUG:boto:Params: {} 2023-05-31T20:39:15.822 DEBUG:boto:Token: None 2023-05-31T20:39:15.823 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:39:15 GMT /pklxkd-58/obj00000000000000000045?versionId=null 2023-05-31T20:39:15.823 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:4bQzfzP1nrhXxVPliAuS3rLV8hg= 2023-05-31T20:39:15.823 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:39:15 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:4bQzfzP1nrhXxVPliAuS3rLV8hg=', 'Content-Length': '0'} 2023-05-31T20:39:15.826 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:38:22 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000034e86e3ebb9ad4c7-006477b073-1c17-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:39:15 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:39:15.827 DEBUG:boto:path=/pklxkd-58/obj00000000000000000045 2023-05-31T20:39:15.827 DEBUG:boto:auth_path=/pklxkd-58/obj00000000000000000045 2023-05-31T20:39:15.828 DEBUG:boto:path=/pklxkd-58/obj00000000000000000045?versionId=null 2023-05-31T20:39:15.828 DEBUG:boto:auth_path=/pklxkd-58/obj00000000000000000045?versionId=null 2023-05-31T20:39:15.828 DEBUG:boto:Method: GET 2023-05-31T20:39:15.829 DEBUG:boto:Path: /pklxkd-58/obj00000000000000000045?versionId=null 2023-05-31T20:39:15.829 DEBUG:boto:Data: 2023-05-31T20:39:15.829 DEBUG:boto:Headers: {} 2023-05-31T20:39:15.830 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:39:15.830 DEBUG:boto:Port: 8000 2023-05-31T20:39:15.830 DEBUG:boto:Params: {} 2023-05-31T20:39:15.831 DEBUG:boto:Token: None 2023-05-31T20:39:15.831 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:39:15 GMT /pklxkd-58/obj00000000000000000045?versionId=null 2023-05-31T20:39:15.832 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:4bQzfzP1nrhXxVPliAuS3rLV8hg= 2023-05-31T20:39:15.832 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:39:15 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:4bQzfzP1nrhXxVPliAuS3rLV8hg=', 'Content-Length': '0'} 2023-05-31T20:39:15.834 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:38:22 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006419b20415873013-006477b073-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:39:15 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:39:15.877 DEBUG:boto:path=/pklxkd-58/obj00000000000000000045 2023-05-31T20:39:15.878 DEBUG:boto:auth_path=/pklxkd-58/obj00000000000000000045 2023-05-31T20:39:15.879 DEBUG:boto:path=/pklxkd-58/obj00000000000000000045?versionId=null 2023-05-31T20:39:15.879 DEBUG:boto:auth_path=/pklxkd-58/obj00000000000000000045?versionId=null 2023-05-31T20:39:15.880 DEBUG:boto:Method: HEAD 2023-05-31T20:39:15.881 DEBUG:boto:Path: /pklxkd-58/obj00000000000000000045?versionId=null 2023-05-31T20:39:15.882 DEBUG:boto:Data: 2023-05-31T20:39:15.882 DEBUG:boto:Headers: {} 2023-05-31T20:39:15.883 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:39:15.883 DEBUG:boto:Port: 8001 2023-05-31T20:39:15.884 DEBUG:boto:Params: {} 2023-05-31T20:39:15.884 DEBUG:boto:Token: None 2023-05-31T20:39:15.885 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:39:15 GMT /pklxkd-58/obj00000000000000000045?versionId=null 2023-05-31T20:39:15.885 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:NQA+K3EuWfda5qdrgKaxkDDt2lw= 2023-05-31T20:39:15.886 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:39:15 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:NQA+K3EuWfda5qdrgKaxkDDt2lw=', 'Content-Length': '0'} 2023-05-31T20:39:15.888 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:38:22 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000239046e89bfe207b-006477b073-1c17-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:39:15 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:39:15.888 DEBUG:boto:path=/pklxkd-58/obj00000000000000000045 2023-05-31T20:39:15.889 DEBUG:boto:auth_path=/pklxkd-58/obj00000000000000000045 2023-05-31T20:39:15.890 DEBUG:boto:path=/pklxkd-58/obj00000000000000000045?versionId=null 2023-05-31T20:39:15.890 DEBUG:boto:auth_path=/pklxkd-58/obj00000000000000000045?versionId=null 2023-05-31T20:39:15.891 DEBUG:boto:Method: HEAD 2023-05-31T20:39:15.892 DEBUG:boto:Path: /pklxkd-58/obj00000000000000000045?versionId=null 2023-05-31T20:39:15.892 DEBUG:boto:Data: 2023-05-31T20:39:15.893 DEBUG:boto:Headers: {} 2023-05-31T20:39:15.893 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:39:15.894 DEBUG:boto:Port: 8000 2023-05-31T20:39:15.894 DEBUG:boto:Params: {} 2023-05-31T20:39:15.895 DEBUG:boto:Token: None 2023-05-31T20:39:15.895 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:39:15 GMT /pklxkd-58/obj00000000000000000045?versionId=null 2023-05-31T20:39:15.896 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:NQA+K3EuWfda5qdrgKaxkDDt2lw= 2023-05-31T20:39:15.897 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:39:15 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:NQA+K3EuWfda5qdrgKaxkDDt2lw=', 'Content-Length': '0'} 2023-05-31T20:39:15.899 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:38:22 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a119d5e1c9ecea1e-006477b073-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:39:15 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:39:15.899 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000045 2023-05-31T20:39:15.900 DEBUG:boto:path=/pklxkd-58/obj00000000000000000045 2023-05-31T20:39:15.901 DEBUG:boto:auth_path=/pklxkd-58/obj00000000000000000045 2023-05-31T20:39:15.901 DEBUG:boto:path=/pklxkd-58/obj00000000000000000045?versionId=null 2023-05-31T20:39:15.902 DEBUG:boto:auth_path=/pklxkd-58/obj00000000000000000045?versionId=null 2023-05-31T20:39:15.903 DEBUG:boto:Method: GET 2023-05-31T20:39:15.903 DEBUG:boto:Path: /pklxkd-58/obj00000000000000000045?versionId=null 2023-05-31T20:39:15.904 DEBUG:boto:Data: 2023-05-31T20:39:15.904 DEBUG:boto:Headers: {} 2023-05-31T20:39:15.905 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:39:15.905 DEBUG:boto:Port: 8001 2023-05-31T20:39:15.906 DEBUG:boto:Params: {} 2023-05-31T20:39:15.906 DEBUG:boto:Token: None 2023-05-31T20:39:15.907 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:39:15 GMT /pklxkd-58/obj00000000000000000045?versionId=null 2023-05-31T20:39:15.907 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:4bQzfzP1nrhXxVPliAuS3rLV8hg= 2023-05-31T20:39:15.908 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:39:15 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:4bQzfzP1nrhXxVPliAuS3rLV8hg=', 'Content-Length': '0'} 2023-05-31T20:39:15.910 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:38:22 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004ef5221ac350c2be-006477b073-1c17-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:39:15 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:39:15.911 DEBUG:boto:path=/pklxkd-58/obj00000000000000000045 2023-05-31T20:39:15.911 DEBUG:boto:auth_path=/pklxkd-58/obj00000000000000000045 2023-05-31T20:39:15.911 DEBUG:boto:path=/pklxkd-58/obj00000000000000000045?versionId=null 2023-05-31T20:39:15.912 DEBUG:boto:auth_path=/pklxkd-58/obj00000000000000000045?versionId=null 2023-05-31T20:39:15.912 DEBUG:boto:Method: GET 2023-05-31T20:39:15.913 DEBUG:boto:Path: /pklxkd-58/obj00000000000000000045?versionId=null 2023-05-31T20:39:15.913 DEBUG:boto:Data: 2023-05-31T20:39:15.914 DEBUG:boto:Headers: {} 2023-05-31T20:39:15.914 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:39:15.915 DEBUG:boto:Port: 8000 2023-05-31T20:39:15.915 DEBUG:boto:Params: {} 2023-05-31T20:39:15.915 DEBUG:boto:Token: None 2023-05-31T20:39:15.916 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:39:15 GMT /pklxkd-58/obj00000000000000000045?versionId=null 2023-05-31T20:39:15.916 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:4bQzfzP1nrhXxVPliAuS3rLV8hg= 2023-05-31T20:39:15.917 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:39:15 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:4bQzfzP1nrhXxVPliAuS3rLV8hg=', 'Content-Length': '0'} 2023-05-31T20:39:15.919 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:38:22 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000060e5d1781ca0a052-006477b073-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:39:15 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:39:15.920 DEBUG:boto:path=/pklxkd-58/ 2023-05-31T20:39:15.921 DEBUG:boto:auth_path=/pklxkd-58/ 2023-05-31T20:39:15.922 DEBUG:boto:path=/pklxkd-58/?prefix=obj00000000000000000045 2023-05-31T20:39:15.923 DEBUG:boto:auth_path=/pklxkd-58/?prefix=obj00000000000000000045 2023-05-31T20:39:15.923 DEBUG:boto:Method: GET 2023-05-31T20:39:15.924 DEBUG:boto:Path: /pklxkd-58/?prefix=obj00000000000000000045 2023-05-31T20:39:15.924 DEBUG:boto:Data: 2023-05-31T20:39:15.924 DEBUG:boto:Headers: {} 2023-05-31T20:39:15.925 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:39:15.925 DEBUG:boto:Port: 8001 2023-05-31T20:39:15.925 DEBUG:boto:Params: {} 2023-05-31T20:39:15.926 DEBUG:boto:Token: None 2023-05-31T20:39:15.926 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:39:15 GMT /pklxkd-58/ 2023-05-31T20:39:15.927 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:kp0WjK8lKKWPr4lFyLKkBJk2eCo= 2023-05-31T20:39:15.927 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:39:15 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:kp0WjK8lKKWPr4lFyLKkBJk2eCo=', 'Content-Length': '0'} 2023-05-31T20:39:15.930 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000009783e08048030d9a-006477b073-1c17-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:39:15 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:39:15.931 DEBUG:boto:b'pklxkd-58obj000000000000000000451000falseobj000000000000000000452023-05-31T20:38:22.250Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:39:15.931 DEBUG:boto:path=/pklxkd-58/ 2023-05-31T20:39:15.932 DEBUG:boto:auth_path=/pklxkd-58/ 2023-05-31T20:39:15.932 DEBUG:boto:path=/pklxkd-58/?prefix=obj00000000000000000045 2023-05-31T20:39:15.933 DEBUG:boto:auth_path=/pklxkd-58/?prefix=obj00000000000000000045 2023-05-31T20:39:15.933 DEBUG:boto:Method: GET 2023-05-31T20:39:15.933 DEBUG:boto:Path: /pklxkd-58/?prefix=obj00000000000000000045 2023-05-31T20:39:15.934 DEBUG:boto:Data: 2023-05-31T20:39:15.934 DEBUG:boto:Headers: {} 2023-05-31T20:39:15.934 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:39:15.935 DEBUG:boto:Port: 8000 2023-05-31T20:39:15.935 DEBUG:boto:Params: {} 2023-05-31T20:39:15.936 DEBUG:boto:Token: None 2023-05-31T20:39:15.936 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:39:15 GMT /pklxkd-58/ 2023-05-31T20:39:15.937 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:kp0WjK8lKKWPr4lFyLKkBJk2eCo= 2023-05-31T20:39:15.937 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:39:15 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:kp0WjK8lKKWPr4lFyLKkBJk2eCo=', 'Content-Length': '0'} 2023-05-31T20:39:15.941 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000070b9cf772ae19944-006477b073-1630-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:39:15 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:39:15.942 DEBUG:boto:b'pklxkd-58obj000000000000000000451000falseobj000000000000000000452023-05-31T20:38:22.250Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:39:15.943 DEBUG:boto:path=/pklxkd-58/obj00000000000000000045 2023-05-31T20:39:15.943 DEBUG:boto:auth_path=/pklxkd-58/obj00000000000000000045 2023-05-31T20:39:15.943 DEBUG:boto:Method: HEAD 2023-05-31T20:39:15.944 DEBUG:boto:Path: /pklxkd-58/obj00000000000000000045 2023-05-31T20:39:15.944 DEBUG:boto:Data: 2023-05-31T20:39:15.944 DEBUG:boto:Headers: {} 2023-05-31T20:39:15.945 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:39:15.945 DEBUG:boto:Port: 8001 2023-05-31T20:39:15.945 DEBUG:boto:Params: {} 2023-05-31T20:39:15.946 DEBUG:boto:Token: None 2023-05-31T20:39:15.946 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:39:15 GMT /pklxkd-58/obj00000000000000000045 2023-05-31T20:39:15.947 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:UqvVDlNWMpTQVM2CKS3ZSra0j6M= 2023-05-31T20:39:15.947 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:39:15 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:UqvVDlNWMpTQVM2CKS3ZSra0j6M=', 'Content-Length': '0'} 2023-05-31T20:39:15.950 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:38:22 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002347ec5e80bf564d-006477b073-1c17-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:39:15 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:39:15.951 DEBUG:boto:path=/pklxkd-58/obj00000000000000000045 2023-05-31T20:39:15.951 DEBUG:boto:auth_path=/pklxkd-58/obj00000000000000000045 2023-05-31T20:39:15.951 DEBUG:boto:Method: HEAD 2023-05-31T20:39:15.952 DEBUG:boto:Path: /pklxkd-58/obj00000000000000000045 2023-05-31T20:39:15.952 DEBUG:boto:Data: 2023-05-31T20:39:15.952 DEBUG:boto:Headers: {} 2023-05-31T20:39:15.953 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:39:15.953 DEBUG:boto:Port: 8000 2023-05-31T20:39:15.953 DEBUG:boto:Params: {} 2023-05-31T20:39:15.954 DEBUG:boto:Token: None 2023-05-31T20:39:15.954 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:39:15 GMT /pklxkd-58/obj00000000000000000045 2023-05-31T20:39:15.955 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:UqvVDlNWMpTQVM2CKS3ZSra0j6M= 2023-05-31T20:39:15.955 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:39:15 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:UqvVDlNWMpTQVM2CKS3ZSra0j6M=', 'Content-Length': '0'} 2023-05-31T20:39:15.957 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:38:22 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000edb4011c529dadb2-006477b073-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:39:15 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:39:15.958 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000045 2023-05-31T20:39:15.959 DEBUG:boto:path=/pklxkd-58/obj00000000000000000045 2023-05-31T20:39:15.959 DEBUG:boto:auth_path=/pklxkd-58/obj00000000000000000045 2023-05-31T20:39:15.960 DEBUG:boto:Method: GET 2023-05-31T20:39:15.960 DEBUG:boto:Path: /pklxkd-58/obj00000000000000000045 2023-05-31T20:39:15.961 DEBUG:boto:Data: 2023-05-31T20:39:15.961 DEBUG:boto:Headers: {} 2023-05-31T20:39:15.961 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:39:15.962 DEBUG:boto:Port: 8001 2023-05-31T20:39:15.962 DEBUG:boto:Params: {} 2023-05-31T20:39:15.962 DEBUG:boto:Token: None 2023-05-31T20:39:15.963 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:39:15 GMT /pklxkd-58/obj00000000000000000045 2023-05-31T20:39:15.963 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:SO3JKJLxTuqd21v2NJkAVlPFjE4= 2023-05-31T20:39:15.964 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:39:15 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:SO3JKJLxTuqd21v2NJkAVlPFjE4=', 'Content-Length': '0'} 2023-05-31T20:39:15.966 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:38:22 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000838670c8d09585e8-006477b073-1c17-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:39:15 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:39:15.967 DEBUG:boto:path=/pklxkd-58/obj00000000000000000045 2023-05-31T20:39:15.967 DEBUG:boto:auth_path=/pklxkd-58/obj00000000000000000045 2023-05-31T20:39:15.968 DEBUG:boto:Method: GET 2023-05-31T20:39:15.968 DEBUG:boto:Path: /pklxkd-58/obj00000000000000000045 2023-05-31T20:39:15.969 DEBUG:boto:Data: 2023-05-31T20:39:15.969 DEBUG:boto:Headers: {} 2023-05-31T20:39:15.970 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:39:15.970 DEBUG:boto:Port: 8000 2023-05-31T20:39:15.970 DEBUG:boto:Params: {} 2023-05-31T20:39:15.971 DEBUG:boto:Token: None 2023-05-31T20:39:15.971 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:39:15 GMT /pklxkd-58/obj00000000000000000045 2023-05-31T20:39:15.972 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:SO3JKJLxTuqd21v2NJkAVlPFjE4= 2023-05-31T20:39:15.972 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:39:15 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:SO3JKJLxTuqd21v2NJkAVlPFjE4=', 'Content-Length': '0'} 2023-05-31T20:39:15.974 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:38:22 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000399ead295a7b08e5-006477b073-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:39:15 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:39:16.016 DEBUG:boto:path=/pklxkd-58/ 2023-05-31T20:39:16.017 DEBUG:boto:auth_path=/pklxkd-58/ 2023-05-31T20:39:16.017 DEBUG:boto:path=/pklxkd-58/?prefix=obj00000000000000000045 2023-05-31T20:39:16.018 DEBUG:boto:auth_path=/pklxkd-58/?prefix=obj00000000000000000045 2023-05-31T20:39:16.018 DEBUG:boto:Method: GET 2023-05-31T20:39:16.018 DEBUG:boto:Path: /pklxkd-58/?prefix=obj00000000000000000045 2023-05-31T20:39:16.019 DEBUG:boto:Data: 2023-05-31T20:39:16.019 DEBUG:boto:Headers: {} 2023-05-31T20:39:16.019 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:39:16.020 DEBUG:boto:Port: 8001 2023-05-31T20:39:16.020 DEBUG:boto:Params: {} 2023-05-31T20:39:16.021 DEBUG:boto:Token: None 2023-05-31T20:39:16.021 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:39:16 GMT /pklxkd-58/ 2023-05-31T20:39:16.022 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:LuRM/phPvIS2vQFGx8h56FjMq7Y= 2023-05-31T20:39:16.022 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:39:16 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:LuRM/phPvIS2vQFGx8h56FjMq7Y=', 'Content-Length': '0'} 2023-05-31T20:39:16.024 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000027fed744f7d33544-006477b074-1c17-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:39:16 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:39:16.025 DEBUG:boto:b'pklxkd-58obj000000000000000000451000falseobj000000000000000000452023-05-31T20:38:22.250Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:39:16.026 DEBUG:boto:path=/pklxkd-58/ 2023-05-31T20:39:16.027 DEBUG:boto:auth_path=/pklxkd-58/ 2023-05-31T20:39:16.027 DEBUG:boto:path=/pklxkd-58/?prefix=obj00000000000000000045 2023-05-31T20:39:16.027 DEBUG:boto:auth_path=/pklxkd-58/?prefix=obj00000000000000000045 2023-05-31T20:39:16.028 DEBUG:boto:Method: GET 2023-05-31T20:39:16.028 DEBUG:boto:Path: /pklxkd-58/?prefix=obj00000000000000000045 2023-05-31T20:39:16.029 DEBUG:boto:Data: 2023-05-31T20:39:16.029 DEBUG:boto:Headers: {} 2023-05-31T20:39:16.029 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:39:16.030 DEBUG:boto:Port: 8000 2023-05-31T20:39:16.030 DEBUG:boto:Params: {} 2023-05-31T20:39:16.030 DEBUG:boto:Token: None 2023-05-31T20:39:16.031 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:39:16 GMT /pklxkd-58/ 2023-05-31T20:39:16.032 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:LuRM/phPvIS2vQFGx8h56FjMq7Y= 2023-05-31T20:39:16.032 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:39:16 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:LuRM/phPvIS2vQFGx8h56FjMq7Y=', 'Content-Length': '0'} 2023-05-31T20:39:16.037 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000154f0e9a1d535e05-006477b074-1630-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:39:16 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:39:16.037 DEBUG:boto:b'pklxkd-58obj000000000000000000451000falseobj000000000000000000452023-05-31T20:38:22.250Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:39:16.038 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000046 2023-05-31T20:39:16.039 DEBUG:boto:path=/pklxkd-58/obj00000000000000000046 2023-05-31T20:39:16.039 DEBUG:boto:auth_path=/pklxkd-58/obj00000000000000000046 2023-05-31T20:39:16.040 DEBUG:boto:path=/pklxkd-58/obj00000000000000000046?versionId=null 2023-05-31T20:39:16.040 DEBUG:boto:auth_path=/pklxkd-58/obj00000000000000000046?versionId=null 2023-05-31T20:39:16.040 DEBUG:boto:Method: GET 2023-05-31T20:39:16.041 DEBUG:boto:Path: /pklxkd-58/obj00000000000000000046?versionId=null 2023-05-31T20:39:16.041 DEBUG:boto:Data: 2023-05-31T20:39:16.041 DEBUG:boto:Headers: {} 2023-05-31T20:39:16.042 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:39:16.042 DEBUG:boto:Port: 8001 2023-05-31T20:39:16.042 DEBUG:boto:Params: {} 2023-05-31T20:39:16.043 DEBUG:boto:Token: None 2023-05-31T20:39:16.043 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:39:16 GMT /pklxkd-58/obj00000000000000000046?versionId=null 2023-05-31T20:39:16.044 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:TH6mpqdixulFLA+UFfs14YLBI0k= 2023-05-31T20:39:16.044 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:39:16 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:TH6mpqdixulFLA+UFfs14YLBI0k=', 'Content-Length': '0'} 2023-05-31T20:39:16.047 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:38:17 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000af3f018be603d76a-006477b074-1c17-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:39:16 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:39:16.088 DEBUG:boto:path=/pklxkd-58/obj00000000000000000046 2023-05-31T20:39:16.089 DEBUG:boto:auth_path=/pklxkd-58/obj00000000000000000046 2023-05-31T20:39:16.090 DEBUG:boto:path=/pklxkd-58/obj00000000000000000046?versionId=null 2023-05-31T20:39:16.090 DEBUG:boto:auth_path=/pklxkd-58/obj00000000000000000046?versionId=null 2023-05-31T20:39:16.091 DEBUG:boto:Method: GET 2023-05-31T20:39:16.092 DEBUG:boto:Path: /pklxkd-58/obj00000000000000000046?versionId=null 2023-05-31T20:39:16.092 DEBUG:boto:Data: 2023-05-31T20:39:16.092 DEBUG:boto:Headers: {} 2023-05-31T20:39:16.093 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:39:16.093 DEBUG:boto:Port: 8000 2023-05-31T20:39:16.094 DEBUG:boto:Params: {} 2023-05-31T20:39:16.094 DEBUG:boto:Token: None 2023-05-31T20:39:16.095 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:39:16 GMT /pklxkd-58/obj00000000000000000046?versionId=null 2023-05-31T20:39:16.095 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:TH6mpqdixulFLA+UFfs14YLBI0k= 2023-05-31T20:39:16.096 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:39:16 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:TH6mpqdixulFLA+UFfs14YLBI0k=', 'Content-Length': '0'} 2023-05-31T20:39:16.098 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:38:17 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000069c3ded04da48cda-006477b074-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:39:16 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:39:16.099 DEBUG:boto:path=/pklxkd-58/obj00000000000000000046 2023-05-31T20:39:16.099 DEBUG:boto:auth_path=/pklxkd-58/obj00000000000000000046 2023-05-31T20:39:16.100 DEBUG:boto:path=/pklxkd-58/obj00000000000000000046?versionId=null 2023-05-31T20:39:16.100 DEBUG:boto:auth_path=/pklxkd-58/obj00000000000000000046?versionId=null 2023-05-31T20:39:16.101 DEBUG:boto:Method: HEAD 2023-05-31T20:39:16.101 DEBUG:boto:Path: /pklxkd-58/obj00000000000000000046?versionId=null 2023-05-31T20:39:16.102 DEBUG:boto:Data: 2023-05-31T20:39:16.102 DEBUG:boto:Headers: {} 2023-05-31T20:39:16.102 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:39:16.103 DEBUG:boto:Port: 8001 2023-05-31T20:39:16.103 DEBUG:boto:Params: {} 2023-05-31T20:39:16.104 DEBUG:boto:Token: None 2023-05-31T20:39:16.104 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:39:16 GMT /pklxkd-58/obj00000000000000000046?versionId=null 2023-05-31T20:39:16.105 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:7zbA+XDpsKQlhVwCNM2zLMJdX0o= 2023-05-31T20:39:16.106 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:39:16 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:7zbA+XDpsKQlhVwCNM2zLMJdX0o=', 'Content-Length': '0'} 2023-05-31T20:39:16.107 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:38:17 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000005dbf9d6c26f4329-006477b074-1c17-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:39:16 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:39:16.108 DEBUG:boto:path=/pklxkd-58/obj00000000000000000046 2023-05-31T20:39:16.108 DEBUG:boto:auth_path=/pklxkd-58/obj00000000000000000046 2023-05-31T20:39:16.109 DEBUG:boto:path=/pklxkd-58/obj00000000000000000046?versionId=null 2023-05-31T20:39:16.109 DEBUG:boto:auth_path=/pklxkd-58/obj00000000000000000046?versionId=null 2023-05-31T20:39:16.110 DEBUG:boto:Method: HEAD 2023-05-31T20:39:16.110 DEBUG:boto:Path: /pklxkd-58/obj00000000000000000046?versionId=null 2023-05-31T20:39:16.110 DEBUG:boto:Data: 2023-05-31T20:39:16.111 DEBUG:boto:Headers: {} 2023-05-31T20:39:16.111 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:39:16.112 DEBUG:boto:Port: 8000 2023-05-31T20:39:16.112 DEBUG:boto:Params: {} 2023-05-31T20:39:16.114 DEBUG:boto:Token: None 2023-05-31T20:39:16.115 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:39:16 GMT /pklxkd-58/obj00000000000000000046?versionId=null 2023-05-31T20:39:16.115 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:7zbA+XDpsKQlhVwCNM2zLMJdX0o= 2023-05-31T20:39:16.116 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:39:16 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:7zbA+XDpsKQlhVwCNM2zLMJdX0o=', 'Content-Length': '0'} 2023-05-31T20:39:16.119 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:38:17 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003af9914141794139-006477b074-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:39:16 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:39:16.119 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000046 2023-05-31T20:39:16.120 DEBUG:boto:path=/pklxkd-58/obj00000000000000000046 2023-05-31T20:39:16.120 DEBUG:boto:auth_path=/pklxkd-58/obj00000000000000000046 2023-05-31T20:39:16.121 DEBUG:boto:path=/pklxkd-58/obj00000000000000000046?versionId=null 2023-05-31T20:39:16.121 DEBUG:boto:auth_path=/pklxkd-58/obj00000000000000000046?versionId=null 2023-05-31T20:39:16.122 DEBUG:boto:Method: GET 2023-05-31T20:39:16.122 DEBUG:boto:Path: /pklxkd-58/obj00000000000000000046?versionId=null 2023-05-31T20:39:16.123 DEBUG:boto:Data: 2023-05-31T20:39:16.124 DEBUG:boto:Headers: {} 2023-05-31T20:39:16.124 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:39:16.125 DEBUG:boto:Port: 8001 2023-05-31T20:39:16.125 DEBUG:boto:Params: {} 2023-05-31T20:39:16.126 DEBUG:boto:Token: None 2023-05-31T20:39:16.127 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:39:16 GMT /pklxkd-58/obj00000000000000000046?versionId=null 2023-05-31T20:39:16.127 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:TH6mpqdixulFLA+UFfs14YLBI0k= 2023-05-31T20:39:16.128 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:39:16 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:TH6mpqdixulFLA+UFfs14YLBI0k=', 'Content-Length': '0'} 2023-05-31T20:39:16.130 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:38:17 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007f443e6235a83ac2-006477b074-1c17-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:39:16 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:39:16.132 DEBUG:boto:path=/pklxkd-58/obj00000000000000000046 2023-05-31T20:39:16.132 DEBUG:boto:auth_path=/pklxkd-58/obj00000000000000000046 2023-05-31T20:39:16.133 DEBUG:boto:path=/pklxkd-58/obj00000000000000000046?versionId=null 2023-05-31T20:39:16.134 DEBUG:boto:auth_path=/pklxkd-58/obj00000000000000000046?versionId=null 2023-05-31T20:39:16.135 DEBUG:boto:Method: GET 2023-05-31T20:39:16.135 DEBUG:boto:Path: /pklxkd-58/obj00000000000000000046?versionId=null 2023-05-31T20:39:16.136 DEBUG:boto:Data: 2023-05-31T20:39:16.136 DEBUG:boto:Headers: {} 2023-05-31T20:39:16.136 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:39:16.137 DEBUG:boto:Port: 8000 2023-05-31T20:39:16.137 DEBUG:boto:Params: {} 2023-05-31T20:39:16.137 DEBUG:boto:Token: None 2023-05-31T20:39:16.138 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:39:16 GMT /pklxkd-58/obj00000000000000000046?versionId=null 2023-05-31T20:39:16.138 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:TH6mpqdixulFLA+UFfs14YLBI0k= 2023-05-31T20:39:16.139 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:39:16 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:TH6mpqdixulFLA+UFfs14YLBI0k=', 'Content-Length': '0'} 2023-05-31T20:39:16.141 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:38:17 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000fe387036ddc06bcc-006477b074-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:39:16 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:39:16.141 DEBUG:boto:path=/pklxkd-58/ 2023-05-31T20:39:16.142 DEBUG:boto:auth_path=/pklxkd-58/ 2023-05-31T20:39:16.142 DEBUG:boto:path=/pklxkd-58/?prefix=obj00000000000000000046 2023-05-31T20:39:16.142 DEBUG:boto:auth_path=/pklxkd-58/?prefix=obj00000000000000000046 2023-05-31T20:39:16.143 DEBUG:boto:Method: GET 2023-05-31T20:39:16.143 DEBUG:boto:Path: /pklxkd-58/?prefix=obj00000000000000000046 2023-05-31T20:39:16.143 DEBUG:boto:Data: 2023-05-31T20:39:16.144 DEBUG:boto:Headers: {} 2023-05-31T20:39:16.144 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:39:16.145 DEBUG:boto:Port: 8001 2023-05-31T20:39:16.145 DEBUG:boto:Params: {} 2023-05-31T20:39:16.145 DEBUG:boto:Token: None 2023-05-31T20:39:16.146 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:39:16 GMT /pklxkd-58/ 2023-05-31T20:39:16.146 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:LuRM/phPvIS2vQFGx8h56FjMq7Y= 2023-05-31T20:39:16.147 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:39:16 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:LuRM/phPvIS2vQFGx8h56FjMq7Y=', 'Content-Length': '0'} 2023-05-31T20:39:16.150 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000003012e35b0fcdfde9-006477b074-1c17-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:39:16 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:39:16.151 DEBUG:boto:b'pklxkd-58obj000000000000000000461000falseobj000000000000000000462023-05-31T20:38:17.357Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:39:16.152 DEBUG:boto:path=/pklxkd-58/ 2023-05-31T20:39:16.152 DEBUG:boto:auth_path=/pklxkd-58/ 2023-05-31T20:39:16.153 DEBUG:boto:path=/pklxkd-58/?prefix=obj00000000000000000046 2023-05-31T20:39:16.153 DEBUG:boto:auth_path=/pklxkd-58/?prefix=obj00000000000000000046 2023-05-31T20:39:16.153 DEBUG:boto:Method: GET 2023-05-31T20:39:16.154 DEBUG:boto:Path: /pklxkd-58/?prefix=obj00000000000000000046 2023-05-31T20:39:16.154 DEBUG:boto:Data: 2023-05-31T20:39:16.155 DEBUG:boto:Headers: {} 2023-05-31T20:39:16.155 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:39:16.155 DEBUG:boto:Port: 8000 2023-05-31T20:39:16.156 DEBUG:boto:Params: {} 2023-05-31T20:39:16.157 DEBUG:boto:Token: None 2023-05-31T20:39:16.157 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:39:16 GMT /pklxkd-58/ 2023-05-31T20:39:16.158 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:LuRM/phPvIS2vQFGx8h56FjMq7Y= 2023-05-31T20:39:16.158 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:39:16 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:LuRM/phPvIS2vQFGx8h56FjMq7Y=', 'Content-Length': '0'} 2023-05-31T20:39:16.162 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000001504a9f0764f883a-006477b074-1630-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:39:16 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:39:16.163 DEBUG:boto:b'pklxkd-58obj000000000000000000461000falseobj000000000000000000462023-05-31T20:38:17.357Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:39:16.164 DEBUG:boto:path=/pklxkd-58/obj00000000000000000046 2023-05-31T20:39:16.164 DEBUG:boto:auth_path=/pklxkd-58/obj00000000000000000046 2023-05-31T20:39:16.165 DEBUG:boto:Method: HEAD 2023-05-31T20:39:16.165 DEBUG:boto:Path: /pklxkd-58/obj00000000000000000046 2023-05-31T20:39:16.166 DEBUG:boto:Data: 2023-05-31T20:39:16.166 DEBUG:boto:Headers: {} 2023-05-31T20:39:16.167 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:39:16.167 DEBUG:boto:Port: 8001 2023-05-31T20:39:16.168 DEBUG:boto:Params: {} 2023-05-31T20:39:16.168 DEBUG:boto:Token: None 2023-05-31T20:39:16.169 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:39:16 GMT /pklxkd-58/obj00000000000000000046 2023-05-31T20:39:16.169 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:6qAcl6AzRBoMCMuun+QF5+fcCM8= 2023-05-31T20:39:16.170 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:39:16 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:6qAcl6AzRBoMCMuun+QF5+fcCM8=', 'Content-Length': '0'} 2023-05-31T20:39:16.172 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:38:17 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003eebff31f3cf1e4a-006477b074-1c17-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:39:16 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:39:16.173 DEBUG:boto:path=/pklxkd-58/obj00000000000000000046 2023-05-31T20:39:16.173 DEBUG:boto:auth_path=/pklxkd-58/obj00000000000000000046 2023-05-31T20:39:16.174 DEBUG:boto:Method: HEAD 2023-05-31T20:39:16.174 DEBUG:boto:Path: /pklxkd-58/obj00000000000000000046 2023-05-31T20:39:16.174 DEBUG:boto:Data: 2023-05-31T20:39:16.175 DEBUG:boto:Headers: {} 2023-05-31T20:39:16.175 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:39:16.176 DEBUG:boto:Port: 8000 2023-05-31T20:39:16.176 DEBUG:boto:Params: {} 2023-05-31T20:39:16.177 DEBUG:boto:Token: None 2023-05-31T20:39:16.177 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:39:16 GMT /pklxkd-58/obj00000000000000000046 2023-05-31T20:39:16.178 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:6qAcl6AzRBoMCMuun+QF5+fcCM8= 2023-05-31T20:39:16.179 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:39:16 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:6qAcl6AzRBoMCMuun+QF5+fcCM8=', 'Content-Length': '0'} 2023-05-31T20:39:16.182 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:38:17 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e7acf8e1fb5f571a-006477b074-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:39:16 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:39:16.183 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000046 2023-05-31T20:39:16.183 DEBUG:boto:path=/pklxkd-58/obj00000000000000000046 2023-05-31T20:39:16.184 DEBUG:boto:auth_path=/pklxkd-58/obj00000000000000000046 2023-05-31T20:39:16.185 DEBUG:boto:Method: GET 2023-05-31T20:39:16.185 DEBUG:boto:Path: /pklxkd-58/obj00000000000000000046 2023-05-31T20:39:16.186 DEBUG:boto:Data: 2023-05-31T20:39:16.187 DEBUG:boto:Headers: {} 2023-05-31T20:39:16.187 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:39:16.188 DEBUG:boto:Port: 8001 2023-05-31T20:39:16.189 DEBUG:boto:Params: {} 2023-05-31T20:39:16.190 DEBUG:boto:Token: None 2023-05-31T20:39:16.190 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:39:16 GMT /pklxkd-58/obj00000000000000000046 2023-05-31T20:39:16.191 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:ZXk51/MZbBRyYdeaTuHZQKQcLGU= 2023-05-31T20:39:16.192 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:39:16 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:ZXk51/MZbBRyYdeaTuHZQKQcLGU=', 'Content-Length': '0'} 2023-05-31T20:39:16.194 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:38:17 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000023fb5f328861044d-006477b074-1c17-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:39:16 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:39:16.195 DEBUG:boto:path=/pklxkd-58/obj00000000000000000046 2023-05-31T20:39:16.196 DEBUG:boto:auth_path=/pklxkd-58/obj00000000000000000046 2023-05-31T20:39:16.197 DEBUG:boto:Method: GET 2023-05-31T20:39:16.198 DEBUG:boto:Path: /pklxkd-58/obj00000000000000000046 2023-05-31T20:39:16.199 DEBUG:boto:Data: 2023-05-31T20:39:16.200 DEBUG:boto:Headers: {} 2023-05-31T20:39:16.201 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:39:16.202 DEBUG:boto:Port: 8000 2023-05-31T20:39:16.202 DEBUG:boto:Params: {} 2023-05-31T20:39:16.202 DEBUG:boto:Token: None 2023-05-31T20:39:16.203 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:39:16 GMT /pklxkd-58/obj00000000000000000046 2023-05-31T20:39:16.203 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:ZXk51/MZbBRyYdeaTuHZQKQcLGU= 2023-05-31T20:39:16.204 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:39:16 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:ZXk51/MZbBRyYdeaTuHZQKQcLGU=', 'Content-Length': '0'} 2023-05-31T20:39:16.206 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:38:17 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008594e5dc4eac5a27-006477b074-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:39:16 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:39:16.250 DEBUG:boto:path=/pklxkd-58/ 2023-05-31T20:39:16.251 DEBUG:boto:auth_path=/pklxkd-58/ 2023-05-31T20:39:16.251 DEBUG:boto:path=/pklxkd-58/?prefix=obj00000000000000000046 2023-05-31T20:39:16.252 DEBUG:boto:auth_path=/pklxkd-58/?prefix=obj00000000000000000046 2023-05-31T20:39:16.252 DEBUG:boto:Method: GET 2023-05-31T20:39:16.252 DEBUG:boto:Path: /pklxkd-58/?prefix=obj00000000000000000046 2023-05-31T20:39:16.253 DEBUG:boto:Data: 2023-05-31T20:39:16.253 DEBUG:boto:Headers: {} 2023-05-31T20:39:16.253 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:39:16.254 DEBUG:boto:Port: 8001 2023-05-31T20:39:16.254 DEBUG:boto:Params: {} 2023-05-31T20:39:16.255 DEBUG:boto:Token: None 2023-05-31T20:39:16.256 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:39:16 GMT /pklxkd-58/ 2023-05-31T20:39:16.256 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:LuRM/phPvIS2vQFGx8h56FjMq7Y= 2023-05-31T20:39:16.257 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:39:16 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:LuRM/phPvIS2vQFGx8h56FjMq7Y=', 'Content-Length': '0'} 2023-05-31T20:39:16.259 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000004c42b592ae33c8aa-006477b074-1c17-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:39:16 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:39:16.260 DEBUG:boto:b'pklxkd-58obj000000000000000000461000falseobj000000000000000000462023-05-31T20:38:17.357Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:39:16.261 DEBUG:boto:path=/pklxkd-58/ 2023-05-31T20:39:16.261 DEBUG:boto:auth_path=/pklxkd-58/ 2023-05-31T20:39:16.262 DEBUG:boto:path=/pklxkd-58/?prefix=obj00000000000000000046 2023-05-31T20:39:16.262 DEBUG:boto:auth_path=/pklxkd-58/?prefix=obj00000000000000000046 2023-05-31T20:39:16.263 DEBUG:boto:Method: GET 2023-05-31T20:39:16.264 DEBUG:boto:Path: /pklxkd-58/?prefix=obj00000000000000000046 2023-05-31T20:39:16.264 DEBUG:boto:Data: 2023-05-31T20:39:16.265 DEBUG:boto:Headers: {} 2023-05-31T20:39:16.265 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:39:16.266 DEBUG:boto:Port: 8000 2023-05-31T20:39:16.266 DEBUG:boto:Params: {} 2023-05-31T20:39:16.266 DEBUG:boto:Token: None 2023-05-31T20:39:16.267 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:39:16 GMT /pklxkd-58/ 2023-05-31T20:39:16.267 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:LuRM/phPvIS2vQFGx8h56FjMq7Y= 2023-05-31T20:39:16.268 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:39:16 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:LuRM/phPvIS2vQFGx8h56FjMq7Y=', 'Content-Length': '0'} 2023-05-31T20:39:16.273 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000002f4f2c8cb0af79ef-006477b074-1630-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:39:16 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:39:16.274 DEBUG:boto:b'pklxkd-58obj000000000000000000461000falseobj000000000000000000462023-05-31T20:38:17.357Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:39:16.275 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000048 2023-05-31T20:39:16.276 DEBUG:boto:path=/pklxkd-58/obj00000000000000000048 2023-05-31T20:39:16.277 DEBUG:boto:auth_path=/pklxkd-58/obj00000000000000000048 2023-05-31T20:39:16.278 DEBUG:boto:path=/pklxkd-58/obj00000000000000000048?versionId=null 2023-05-31T20:39:16.278 DEBUG:boto:auth_path=/pklxkd-58/obj00000000000000000048?versionId=null 2023-05-31T20:39:16.279 DEBUG:boto:Method: GET 2023-05-31T20:39:16.281 DEBUG:boto:Path: /pklxkd-58/obj00000000000000000048?versionId=null 2023-05-31T20:39:16.281 DEBUG:boto:Data: 2023-05-31T20:39:16.282 DEBUG:boto:Headers: {} 2023-05-31T20:39:16.282 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:39:16.283 DEBUG:boto:Port: 8001 2023-05-31T20:39:16.284 DEBUG:boto:Params: {} 2023-05-31T20:39:16.284 DEBUG:boto:Token: None 2023-05-31T20:39:16.285 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:39:16 GMT /pklxkd-58/obj00000000000000000048?versionId=null 2023-05-31T20:39:16.286 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:5ocNMnLSOn2RB4vOsqZ4KpB32O0= 2023-05-31T20:39:16.287 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:39:16 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:5ocNMnLSOn2RB4vOsqZ4KpB32O0=', 'Content-Length': '0'} 2023-05-31T20:39:16.289 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:38:22 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000058b4e2823433a2cf-006477b074-1c17-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:39:16 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:39:16.289 DEBUG:boto:path=/pklxkd-58/obj00000000000000000048 2023-05-31T20:39:16.290 DEBUG:boto:auth_path=/pklxkd-58/obj00000000000000000048 2023-05-31T20:39:16.290 DEBUG:boto:path=/pklxkd-58/obj00000000000000000048?versionId=null 2023-05-31T20:39:16.291 DEBUG:boto:auth_path=/pklxkd-58/obj00000000000000000048?versionId=null 2023-05-31T20:39:16.291 DEBUG:boto:Method: GET 2023-05-31T20:39:16.291 DEBUG:boto:Path: /pklxkd-58/obj00000000000000000048?versionId=null 2023-05-31T20:39:16.292 DEBUG:boto:Data: 2023-05-31T20:39:16.292 DEBUG:boto:Headers: {} 2023-05-31T20:39:16.293 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:39:16.293 DEBUG:boto:Port: 8000 2023-05-31T20:39:16.293 DEBUG:boto:Params: {} 2023-05-31T20:39:16.294 DEBUG:boto:Token: None 2023-05-31T20:39:16.294 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:39:16 GMT /pklxkd-58/obj00000000000000000048?versionId=null 2023-05-31T20:39:16.296 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:5ocNMnLSOn2RB4vOsqZ4KpB32O0= 2023-05-31T20:39:16.296 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:39:16 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:5ocNMnLSOn2RB4vOsqZ4KpB32O0=', 'Content-Length': '0'} 2023-05-31T20:39:16.299 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:38:22 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008da81b7af8aa3f66-006477b074-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:39:16 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:39:16.299 DEBUG:boto:path=/pklxkd-58/obj00000000000000000048 2023-05-31T20:39:16.300 DEBUG:boto:auth_path=/pklxkd-58/obj00000000000000000048 2023-05-31T20:39:16.300 DEBUG:boto:path=/pklxkd-58/obj00000000000000000048?versionId=null 2023-05-31T20:39:16.300 DEBUG:boto:auth_path=/pklxkd-58/obj00000000000000000048?versionId=null 2023-05-31T20:39:16.301 DEBUG:boto:Method: HEAD 2023-05-31T20:39:16.301 DEBUG:boto:Path: /pklxkd-58/obj00000000000000000048?versionId=null 2023-05-31T20:39:16.301 DEBUG:boto:Data: 2023-05-31T20:39:16.302 DEBUG:boto:Headers: {} 2023-05-31T20:39:16.302 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:39:16.303 DEBUG:boto:Port: 8001 2023-05-31T20:39:16.303 DEBUG:boto:Params: {} 2023-05-31T20:39:16.304 DEBUG:boto:Token: None 2023-05-31T20:39:16.304 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:39:16 GMT /pklxkd-58/obj00000000000000000048?versionId=null 2023-05-31T20:39:16.305 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:Ngoy9yeFEVzO7T8RzZZuJjFmPio= 2023-05-31T20:39:16.305 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:39:16 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:Ngoy9yeFEVzO7T8RzZZuJjFmPio=', 'Content-Length': '0'} 2023-05-31T20:39:16.307 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:38:22 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006e78907e20cbf7fa-006477b074-1c17-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:39:16 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:39:16.308 DEBUG:boto:path=/pklxkd-58/obj00000000000000000048 2023-05-31T20:39:16.308 DEBUG:boto:auth_path=/pklxkd-58/obj00000000000000000048 2023-05-31T20:39:16.309 DEBUG:boto:path=/pklxkd-58/obj00000000000000000048?versionId=null 2023-05-31T20:39:16.309 DEBUG:boto:auth_path=/pklxkd-58/obj00000000000000000048?versionId=null 2023-05-31T20:39:16.309 DEBUG:boto:Method: HEAD 2023-05-31T20:39:16.310 DEBUG:boto:Path: /pklxkd-58/obj00000000000000000048?versionId=null 2023-05-31T20:39:16.310 DEBUG:boto:Data: 2023-05-31T20:39:16.310 DEBUG:boto:Headers: {} 2023-05-31T20:39:16.311 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:39:16.311 DEBUG:boto:Port: 8000 2023-05-31T20:39:16.312 DEBUG:boto:Params: {} 2023-05-31T20:39:16.312 DEBUG:boto:Token: None 2023-05-31T20:39:16.313 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:39:16 GMT /pklxkd-58/obj00000000000000000048?versionId=null 2023-05-31T20:39:16.313 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:Ngoy9yeFEVzO7T8RzZZuJjFmPio= 2023-05-31T20:39:16.313 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:39:16 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:Ngoy9yeFEVzO7T8RzZZuJjFmPio=', 'Content-Length': '0'} 2023-05-31T20:39:16.315 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:38:22 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002c483ff9459f99b1-006477b074-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:39:16 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:39:16.316 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000048 2023-05-31T20:39:16.316 DEBUG:boto:path=/pklxkd-58/obj00000000000000000048 2023-05-31T20:39:16.317 DEBUG:boto:auth_path=/pklxkd-58/obj00000000000000000048 2023-05-31T20:39:16.317 DEBUG:boto:path=/pklxkd-58/obj00000000000000000048?versionId=null 2023-05-31T20:39:16.317 DEBUG:boto:auth_path=/pklxkd-58/obj00000000000000000048?versionId=null 2023-05-31T20:39:16.318 DEBUG:boto:Method: GET 2023-05-31T20:39:16.318 DEBUG:boto:Path: /pklxkd-58/obj00000000000000000048?versionId=null 2023-05-31T20:39:16.319 DEBUG:boto:Data: 2023-05-31T20:39:16.319 DEBUG:boto:Headers: {} 2023-05-31T20:39:16.319 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:39:16.320 DEBUG:boto:Port: 8001 2023-05-31T20:39:16.320 DEBUG:boto:Params: {} 2023-05-31T20:39:16.320 DEBUG:boto:Token: None 2023-05-31T20:39:16.321 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:39:16 GMT /pklxkd-58/obj00000000000000000048?versionId=null 2023-05-31T20:39:16.321 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:5ocNMnLSOn2RB4vOsqZ4KpB32O0= 2023-05-31T20:39:16.322 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:39:16 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:5ocNMnLSOn2RB4vOsqZ4KpB32O0=', 'Content-Length': '0'} 2023-05-31T20:39:16.324 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:38:22 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009c2321299c227fd7-006477b074-1c17-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:39:16 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:39:16.364 DEBUG:boto:path=/pklxkd-58/obj00000000000000000048 2023-05-31T20:39:16.365 DEBUG:boto:auth_path=/pklxkd-58/obj00000000000000000048 2023-05-31T20:39:16.365 DEBUG:boto:path=/pklxkd-58/obj00000000000000000048?versionId=null 2023-05-31T20:39:16.366 DEBUG:boto:auth_path=/pklxkd-58/obj00000000000000000048?versionId=null 2023-05-31T20:39:16.366 DEBUG:boto:Method: GET 2023-05-31T20:39:16.366 DEBUG:boto:Path: /pklxkd-58/obj00000000000000000048?versionId=null 2023-05-31T20:39:16.367 DEBUG:boto:Data: 2023-05-31T20:39:16.367 DEBUG:boto:Headers: {} 2023-05-31T20:39:16.367 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:39:16.368 DEBUG:boto:Port: 8000 2023-05-31T20:39:16.368 DEBUG:boto:Params: {} 2023-05-31T20:39:16.369 DEBUG:boto:Token: None 2023-05-31T20:39:16.369 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:39:16 GMT /pklxkd-58/obj00000000000000000048?versionId=null 2023-05-31T20:39:16.370 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:5ocNMnLSOn2RB4vOsqZ4KpB32O0= 2023-05-31T20:39:16.370 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:39:16 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:5ocNMnLSOn2RB4vOsqZ4KpB32O0=', 'Content-Length': '0'} 2023-05-31T20:39:16.372 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:38:22 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000418a99c85ec315a2-006477b074-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:39:16 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:39:16.373 DEBUG:boto:path=/pklxkd-58/ 2023-05-31T20:39:16.373 DEBUG:boto:auth_path=/pklxkd-58/ 2023-05-31T20:39:16.374 DEBUG:boto:path=/pklxkd-58/?prefix=obj00000000000000000048 2023-05-31T20:39:16.374 DEBUG:boto:auth_path=/pklxkd-58/?prefix=obj00000000000000000048 2023-05-31T20:39:16.374 DEBUG:boto:Method: GET 2023-05-31T20:39:16.375 DEBUG:boto:Path: /pklxkd-58/?prefix=obj00000000000000000048 2023-05-31T20:39:16.375 DEBUG:boto:Data: 2023-05-31T20:39:16.375 DEBUG:boto:Headers: {} 2023-05-31T20:39:16.376 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:39:16.376 DEBUG:boto:Port: 8001 2023-05-31T20:39:16.377 DEBUG:boto:Params: {} 2023-05-31T20:39:16.377 DEBUG:boto:Token: None 2023-05-31T20:39:16.377 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:39:16 GMT /pklxkd-58/ 2023-05-31T20:39:16.378 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:LuRM/phPvIS2vQFGx8h56FjMq7Y= 2023-05-31T20:39:16.378 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:39:16 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:LuRM/phPvIS2vQFGx8h56FjMq7Y=', 'Content-Length': '0'} 2023-05-31T20:39:16.383 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000003e08f3650323cd1f-006477b074-1c17-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:39:16 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:39:16.384 DEBUG:boto:b'pklxkd-58obj000000000000000000481000falseobj000000000000000000482023-05-31T20:38:22.489Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:39:16.385 DEBUG:boto:path=/pklxkd-58/ 2023-05-31T20:39:16.385 DEBUG:boto:auth_path=/pklxkd-58/ 2023-05-31T20:39:16.385 DEBUG:boto:path=/pklxkd-58/?prefix=obj00000000000000000048 2023-05-31T20:39:16.386 DEBUG:boto:auth_path=/pklxkd-58/?prefix=obj00000000000000000048 2023-05-31T20:39:16.386 DEBUG:boto:Method: GET 2023-05-31T20:39:16.386 DEBUG:boto:Path: /pklxkd-58/?prefix=obj00000000000000000048 2023-05-31T20:39:16.387 DEBUG:boto:Data: 2023-05-31T20:39:16.387 DEBUG:boto:Headers: {} 2023-05-31T20:39:16.388 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:39:16.388 DEBUG:boto:Port: 8000 2023-05-31T20:39:16.389 DEBUG:boto:Params: {} 2023-05-31T20:39:16.389 DEBUG:boto:Token: None 2023-05-31T20:39:16.390 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:39:16 GMT /pklxkd-58/ 2023-05-31T20:39:16.390 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:LuRM/phPvIS2vQFGx8h56FjMq7Y= 2023-05-31T20:39:16.390 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:39:16 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:LuRM/phPvIS2vQFGx8h56FjMq7Y=', 'Content-Length': '0'} 2023-05-31T20:39:16.396 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000046a32925e939a609-006477b074-1630-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:39:16 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:39:16.440 DEBUG:boto:b'pklxkd-58obj000000000000000000481000falseobj000000000000000000482023-05-31T20:38:22.489Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:39:16.442 DEBUG:boto:path=/pklxkd-58/obj00000000000000000048 2023-05-31T20:39:16.442 DEBUG:boto:auth_path=/pklxkd-58/obj00000000000000000048 2023-05-31T20:39:16.443 DEBUG:boto:Method: HEAD 2023-05-31T20:39:16.443 DEBUG:boto:Path: /pklxkd-58/obj00000000000000000048 2023-05-31T20:39:16.444 DEBUG:boto:Data: 2023-05-31T20:39:16.444 DEBUG:boto:Headers: {} 2023-05-31T20:39:16.445 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:39:16.445 DEBUG:boto:Port: 8001 2023-05-31T20:39:16.445 DEBUG:boto:Params: {} 2023-05-31T20:39:16.446 DEBUG:boto:Token: None 2023-05-31T20:39:16.446 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:39:16 GMT /pklxkd-58/obj00000000000000000048 2023-05-31T20:39:16.447 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:FMaFhRaFgyjC5aOkEoLEg5Cq+xE= 2023-05-31T20:39:16.448 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:39:16 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:FMaFhRaFgyjC5aOkEoLEg5Cq+xE=', 'Content-Length': '0'} 2023-05-31T20:39:16.450 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:38:22 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000e7331401f804cf8-006477b074-1c17-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:39:16 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:39:16.450 DEBUG:boto:path=/pklxkd-58/obj00000000000000000048 2023-05-31T20:39:16.451 DEBUG:boto:auth_path=/pklxkd-58/obj00000000000000000048 2023-05-31T20:39:16.451 DEBUG:boto:Method: HEAD 2023-05-31T20:39:16.452 DEBUG:boto:Path: /pklxkd-58/obj00000000000000000048 2023-05-31T20:39:16.452 DEBUG:boto:Data: 2023-05-31T20:39:16.453 DEBUG:boto:Headers: {} 2023-05-31T20:39:16.454 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:39:16.454 DEBUG:boto:Port: 8000 2023-05-31T20:39:16.455 DEBUG:boto:Params: {} 2023-05-31T20:39:16.456 DEBUG:boto:Token: None 2023-05-31T20:39:16.457 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:39:16 GMT /pklxkd-58/obj00000000000000000048 2023-05-31T20:39:16.458 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:FMaFhRaFgyjC5aOkEoLEg5Cq+xE= 2023-05-31T20:39:16.459 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:39:16 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:FMaFhRaFgyjC5aOkEoLEg5Cq+xE=', 'Content-Length': '0'} 2023-05-31T20:39:16.461 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:38:22 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000058299e0847e768fc-006477b074-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:39:16 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:39:16.461 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000048 2023-05-31T20:39:16.462 DEBUG:boto:path=/pklxkd-58/obj00000000000000000048 2023-05-31T20:39:16.462 DEBUG:boto:auth_path=/pklxkd-58/obj00000000000000000048 2023-05-31T20:39:16.463 DEBUG:boto:Method: GET 2023-05-31T20:39:16.464 DEBUG:boto:Path: /pklxkd-58/obj00000000000000000048 2023-05-31T20:39:16.464 DEBUG:boto:Data: 2023-05-31T20:39:16.464 DEBUG:boto:Headers: {} 2023-05-31T20:39:16.465 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:39:16.465 DEBUG:boto:Port: 8001 2023-05-31T20:39:16.466 DEBUG:boto:Params: {} 2023-05-31T20:39:16.466 DEBUG:boto:Token: None 2023-05-31T20:39:16.467 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:39:16 GMT /pklxkd-58/obj00000000000000000048 2023-05-31T20:39:16.467 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:EG9dsMxtn3q6GecLjDNwNpM+W0I= 2023-05-31T20:39:16.468 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:39:16 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:EG9dsMxtn3q6GecLjDNwNpM+W0I=', 'Content-Length': '0'} 2023-05-31T20:39:16.470 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:38:22 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c5955b2c8ad77b85-006477b074-1c17-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:39:16 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:39:16.471 DEBUG:boto:path=/pklxkd-58/obj00000000000000000048 2023-05-31T20:39:16.471 DEBUG:boto:auth_path=/pklxkd-58/obj00000000000000000048 2023-05-31T20:39:16.471 DEBUG:boto:Method: GET 2023-05-31T20:39:16.472 DEBUG:boto:Path: /pklxkd-58/obj00000000000000000048 2023-05-31T20:39:16.473 DEBUG:boto:Data: 2023-05-31T20:39:16.473 DEBUG:boto:Headers: {} 2023-05-31T20:39:16.473 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:39:16.474 DEBUG:boto:Port: 8000 2023-05-31T20:39:16.474 DEBUG:boto:Params: {} 2023-05-31T20:39:16.474 DEBUG:boto:Token: None 2023-05-31T20:39:16.475 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:39:16 GMT /pklxkd-58/obj00000000000000000048 2023-05-31T20:39:16.475 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:EG9dsMxtn3q6GecLjDNwNpM+W0I= 2023-05-31T20:39:16.476 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:39:16 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:EG9dsMxtn3q6GecLjDNwNpM+W0I=', 'Content-Length': '0'} 2023-05-31T20:39:16.478 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:38:22 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002b6384eb77c99e60-006477b074-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:39:16 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:39:16.478 DEBUG:boto:path=/pklxkd-58/ 2023-05-31T20:39:16.479 DEBUG:boto:auth_path=/pklxkd-58/ 2023-05-31T20:39:16.479 DEBUG:boto:path=/pklxkd-58/?prefix=obj00000000000000000048 2023-05-31T20:39:16.480 DEBUG:boto:auth_path=/pklxkd-58/?prefix=obj00000000000000000048 2023-05-31T20:39:16.480 DEBUG:boto:Method: GET 2023-05-31T20:39:16.480 DEBUG:boto:Path: /pklxkd-58/?prefix=obj00000000000000000048 2023-05-31T20:39:16.481 DEBUG:boto:Data: 2023-05-31T20:39:16.481 DEBUG:boto:Headers: {} 2023-05-31T20:39:16.482 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:39:16.482 DEBUG:boto:Port: 8001 2023-05-31T20:39:16.483 DEBUG:boto:Params: {} 2023-05-31T20:39:16.483 DEBUG:boto:Token: None 2023-05-31T20:39:16.483 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:39:16 GMT /pklxkd-58/ 2023-05-31T20:39:16.484 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:LuRM/phPvIS2vQFGx8h56FjMq7Y= 2023-05-31T20:39:16.484 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:39:16 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:LuRM/phPvIS2vQFGx8h56FjMq7Y=', 'Content-Length': '0'} 2023-05-31T20:39:16.487 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b4bf9b47cbdb8167-006477b074-1c17-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:39:16 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:39:16.488 DEBUG:boto:b'pklxkd-58obj000000000000000000481000falseobj000000000000000000482023-05-31T20:38:22.489Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:39:16.488 DEBUG:boto:path=/pklxkd-58/ 2023-05-31T20:39:16.489 DEBUG:boto:auth_path=/pklxkd-58/ 2023-05-31T20:39:16.489 DEBUG:boto:path=/pklxkd-58/?prefix=obj00000000000000000048 2023-05-31T20:39:16.490 DEBUG:boto:auth_path=/pklxkd-58/?prefix=obj00000000000000000048 2023-05-31T20:39:16.490 DEBUG:boto:Method: GET 2023-05-31T20:39:16.490 DEBUG:boto:Path: /pklxkd-58/?prefix=obj00000000000000000048 2023-05-31T20:39:16.491 DEBUG:boto:Data: 2023-05-31T20:39:16.491 DEBUG:boto:Headers: {} 2023-05-31T20:39:16.491 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:39:16.492 DEBUG:boto:Port: 8000 2023-05-31T20:39:16.492 DEBUG:boto:Params: {} 2023-05-31T20:39:16.493 DEBUG:boto:Token: None 2023-05-31T20:39:16.493 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:39:16 GMT /pklxkd-58/ 2023-05-31T20:39:16.494 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:LuRM/phPvIS2vQFGx8h56FjMq7Y= 2023-05-31T20:39:16.494 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:39:16 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:LuRM/phPvIS2vQFGx8h56FjMq7Y=', 'Content-Length': '0'} 2023-05-31T20:39:16.498 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000024b0f881aa52c2dd-006477b074-1630-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:39:16 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:39:16.499 DEBUG:boto:b'pklxkd-58obj000000000000000000481000falseobj000000000000000000482023-05-31T20:38:22.489Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:39:16.500 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000049 2023-05-31T20:39:16.500 DEBUG:boto:path=/pklxkd-58/obj00000000000000000049 2023-05-31T20:39:16.501 DEBUG:boto:auth_path=/pklxkd-58/obj00000000000000000049 2023-05-31T20:39:16.501 DEBUG:boto:path=/pklxkd-58/obj00000000000000000049?versionId=null 2023-05-31T20:39:16.502 DEBUG:boto:auth_path=/pklxkd-58/obj00000000000000000049?versionId=null 2023-05-31T20:39:16.502 DEBUG:boto:Method: GET 2023-05-31T20:39:16.502 DEBUG:boto:Path: /pklxkd-58/obj00000000000000000049?versionId=null 2023-05-31T20:39:16.503 DEBUG:boto:Data: 2023-05-31T20:39:16.503 DEBUG:boto:Headers: {} 2023-05-31T20:39:16.504 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:39:16.504 DEBUG:boto:Port: 8001 2023-05-31T20:39:16.504 DEBUG:boto:Params: {} 2023-05-31T20:39:16.505 DEBUG:boto:Token: None 2023-05-31T20:39:16.505 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:39:16 GMT /pklxkd-58/obj00000000000000000049?versionId=null 2023-05-31T20:39:16.506 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:VnuKW6dTROMt8XsKuoo7hWenEDE= 2023-05-31T20:39:16.506 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:39:16 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:VnuKW6dTROMt8XsKuoo7hWenEDE=', 'Content-Length': '0'} 2023-05-31T20:39:16.508 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:38:18 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e944574fe672ee88-006477b074-1c17-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:39:16 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:39:16.509 DEBUG:boto:path=/pklxkd-58/obj00000000000000000049 2023-05-31T20:39:16.510 DEBUG:boto:auth_path=/pklxkd-58/obj00000000000000000049 2023-05-31T20:39:16.510 DEBUG:boto:path=/pklxkd-58/obj00000000000000000049?versionId=null 2023-05-31T20:39:16.510 DEBUG:boto:auth_path=/pklxkd-58/obj00000000000000000049?versionId=null 2023-05-31T20:39:16.511 DEBUG:boto:Method: GET 2023-05-31T20:39:16.511 DEBUG:boto:Path: /pklxkd-58/obj00000000000000000049?versionId=null 2023-05-31T20:39:16.512 DEBUG:boto:Data: 2023-05-31T20:39:16.512 DEBUG:boto:Headers: {} 2023-05-31T20:39:16.513 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:39:16.513 DEBUG:boto:Port: 8000 2023-05-31T20:39:16.513 DEBUG:boto:Params: {} 2023-05-31T20:39:16.514 DEBUG:boto:Token: None 2023-05-31T20:39:16.514 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:39:16 GMT /pklxkd-58/obj00000000000000000049?versionId=null 2023-05-31T20:39:16.515 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:VnuKW6dTROMt8XsKuoo7hWenEDE= 2023-05-31T20:39:16.515 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:39:16 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:VnuKW6dTROMt8XsKuoo7hWenEDE=', 'Content-Length': '0'} 2023-05-31T20:39:16.517 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:38:18 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000268cf9f9c8ab9975-006477b074-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:39:16 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:39:16.560 DEBUG:boto:path=/pklxkd-58/obj00000000000000000049 2023-05-31T20:39:16.561 DEBUG:boto:auth_path=/pklxkd-58/obj00000000000000000049 2023-05-31T20:39:16.561 DEBUG:boto:path=/pklxkd-58/obj00000000000000000049?versionId=null 2023-05-31T20:39:16.562 DEBUG:boto:auth_path=/pklxkd-58/obj00000000000000000049?versionId=null 2023-05-31T20:39:16.563 DEBUG:boto:Method: HEAD 2023-05-31T20:39:16.564 DEBUG:boto:Path: /pklxkd-58/obj00000000000000000049?versionId=null 2023-05-31T20:39:16.565 DEBUG:boto:Data: 2023-05-31T20:39:16.566 DEBUG:boto:Headers: {} 2023-05-31T20:39:16.567 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:39:16.567 DEBUG:boto:Port: 8001 2023-05-31T20:39:16.569 DEBUG:boto:Params: {} 2023-05-31T20:39:16.570 DEBUG:boto:Token: None 2023-05-31T20:39:16.570 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:39:16 GMT /pklxkd-58/obj00000000000000000049?versionId=null 2023-05-31T20:39:16.571 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:BsuH1WiK1MEXZjteFrkLA2TsNlo= 2023-05-31T20:39:16.572 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:39:16 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:BsuH1WiK1MEXZjteFrkLA2TsNlo=', 'Content-Length': '0'} 2023-05-31T20:39:16.574 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:38:18 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b8953aad382c892d-006477b074-1c17-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:39:16 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:39:16.575 DEBUG:boto:path=/pklxkd-58/obj00000000000000000049 2023-05-31T20:39:16.575 DEBUG:boto:auth_path=/pklxkd-58/obj00000000000000000049 2023-05-31T20:39:16.576 DEBUG:boto:path=/pklxkd-58/obj00000000000000000049?versionId=null 2023-05-31T20:39:16.576 DEBUG:boto:auth_path=/pklxkd-58/obj00000000000000000049?versionId=null 2023-05-31T20:39:16.576 DEBUG:boto:Method: HEAD 2023-05-31T20:39:16.577 DEBUG:boto:Path: /pklxkd-58/obj00000000000000000049?versionId=null 2023-05-31T20:39:16.578 DEBUG:boto:Data: 2023-05-31T20:39:16.578 DEBUG:boto:Headers: {} 2023-05-31T20:39:16.578 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:39:16.579 DEBUG:boto:Port: 8000 2023-05-31T20:39:16.579 DEBUG:boto:Params: {} 2023-05-31T20:39:16.579 DEBUG:boto:Token: None 2023-05-31T20:39:16.580 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:39:16 GMT /pklxkd-58/obj00000000000000000049?versionId=null 2023-05-31T20:39:16.580 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:BsuH1WiK1MEXZjteFrkLA2TsNlo= 2023-05-31T20:39:16.581 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:39:16 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:BsuH1WiK1MEXZjteFrkLA2TsNlo=', 'Content-Length': '0'} 2023-05-31T20:39:16.582 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:38:18 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000084a8c34d34bb2edf-006477b074-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:39:16 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:39:16.583 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000049 2023-05-31T20:39:16.584 DEBUG:boto:path=/pklxkd-58/obj00000000000000000049 2023-05-31T20:39:16.585 DEBUG:boto:auth_path=/pklxkd-58/obj00000000000000000049 2023-05-31T20:39:16.586 DEBUG:boto:path=/pklxkd-58/obj00000000000000000049?versionId=null 2023-05-31T20:39:16.587 DEBUG:boto:auth_path=/pklxkd-58/obj00000000000000000049?versionId=null 2023-05-31T20:39:16.588 DEBUG:boto:Method: GET 2023-05-31T20:39:16.589 DEBUG:boto:Path: /pklxkd-58/obj00000000000000000049?versionId=null 2023-05-31T20:39:16.590 DEBUG:boto:Data: 2023-05-31T20:39:16.591 DEBUG:boto:Headers: {} 2023-05-31T20:39:16.591 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:39:16.592 DEBUG:boto:Port: 8001 2023-05-31T20:39:16.593 DEBUG:boto:Params: {} 2023-05-31T20:39:16.594 DEBUG:boto:Token: None 2023-05-31T20:39:16.594 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:39:16 GMT /pklxkd-58/obj00000000000000000049?versionId=null 2023-05-31T20:39:16.595 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:VnuKW6dTROMt8XsKuoo7hWenEDE= 2023-05-31T20:39:16.595 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:39:16 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:VnuKW6dTROMt8XsKuoo7hWenEDE=', 'Content-Length': '0'} 2023-05-31T20:39:16.598 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:38:18 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f539d0ea78a230f5-006477b074-1c17-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:39:16 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:39:16.598 DEBUG:boto:path=/pklxkd-58/obj00000000000000000049 2023-05-31T20:39:16.599 DEBUG:boto:auth_path=/pklxkd-58/obj00000000000000000049 2023-05-31T20:39:16.599 DEBUG:boto:path=/pklxkd-58/obj00000000000000000049?versionId=null 2023-05-31T20:39:16.599 DEBUG:boto:auth_path=/pklxkd-58/obj00000000000000000049?versionId=null 2023-05-31T20:39:16.600 DEBUG:boto:Method: GET 2023-05-31T20:39:16.600 DEBUG:boto:Path: /pklxkd-58/obj00000000000000000049?versionId=null 2023-05-31T20:39:16.601 DEBUG:boto:Data: 2023-05-31T20:39:16.601 DEBUG:boto:Headers: {} 2023-05-31T20:39:16.601 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:39:16.602 DEBUG:boto:Port: 8000 2023-05-31T20:39:16.602 DEBUG:boto:Params: {} 2023-05-31T20:39:16.603 DEBUG:boto:Token: None 2023-05-31T20:39:16.603 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:39:16 GMT /pklxkd-58/obj00000000000000000049?versionId=null 2023-05-31T20:39:16.604 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:VnuKW6dTROMt8XsKuoo7hWenEDE= 2023-05-31T20:39:16.604 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:39:16 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:VnuKW6dTROMt8XsKuoo7hWenEDE=', 'Content-Length': '0'} 2023-05-31T20:39:16.606 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:38:18 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000072f12f32a4e03a3d-006477b074-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:39:16 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:39:16.607 DEBUG:boto:path=/pklxkd-58/ 2023-05-31T20:39:16.608 DEBUG:boto:auth_path=/pklxkd-58/ 2023-05-31T20:39:16.608 DEBUG:boto:path=/pklxkd-58/?prefix=obj00000000000000000049 2023-05-31T20:39:16.609 DEBUG:boto:auth_path=/pklxkd-58/?prefix=obj00000000000000000049 2023-05-31T20:39:16.609 DEBUG:boto:Method: GET 2023-05-31T20:39:16.609 DEBUG:boto:Path: /pklxkd-58/?prefix=obj00000000000000000049 2023-05-31T20:39:16.610 DEBUG:boto:Data: 2023-05-31T20:39:16.611 DEBUG:boto:Headers: {} 2023-05-31T20:39:16.611 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:39:16.612 DEBUG:boto:Port: 8001 2023-05-31T20:39:16.612 DEBUG:boto:Params: {} 2023-05-31T20:39:16.613 DEBUG:boto:Token: None 2023-05-31T20:39:16.613 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:39:16 GMT /pklxkd-58/ 2023-05-31T20:39:16.614 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:LuRM/phPvIS2vQFGx8h56FjMq7Y= 2023-05-31T20:39:16.614 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:39:16 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:LuRM/phPvIS2vQFGx8h56FjMq7Y=', 'Content-Length': '0'} 2023-05-31T20:39:16.617 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b7c60444f36f54a3-006477b074-1c17-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:39:16 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:39:16.618 DEBUG:boto:b'pklxkd-58obj000000000000000000491000falseobj000000000000000000492023-05-31T20:38:18.527Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:39:16.619 DEBUG:boto:path=/pklxkd-58/ 2023-05-31T20:39:16.619 DEBUG:boto:auth_path=/pklxkd-58/ 2023-05-31T20:39:16.620 DEBUG:boto:path=/pklxkd-58/?prefix=obj00000000000000000049 2023-05-31T20:39:16.620 DEBUG:boto:auth_path=/pklxkd-58/?prefix=obj00000000000000000049 2023-05-31T20:39:16.621 DEBUG:boto:Method: GET 2023-05-31T20:39:16.621 DEBUG:boto:Path: /pklxkd-58/?prefix=obj00000000000000000049 2023-05-31T20:39:16.621 DEBUG:boto:Data: 2023-05-31T20:39:16.622 DEBUG:boto:Headers: {} 2023-05-31T20:39:16.622 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:39:16.623 DEBUG:boto:Port: 8000 2023-05-31T20:39:16.624 DEBUG:boto:Params: {} 2023-05-31T20:39:16.624 DEBUG:boto:Token: None 2023-05-31T20:39:16.625 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:39:16 GMT /pklxkd-58/ 2023-05-31T20:39:16.627 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:LuRM/phPvIS2vQFGx8h56FjMq7Y= 2023-05-31T20:39:16.628 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:39:16 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:LuRM/phPvIS2vQFGx8h56FjMq7Y=', 'Content-Length': '0'} 2023-05-31T20:39:16.633 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a143b18ec16ff14b-006477b074-1630-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:39:16 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:39:16.633 DEBUG:boto:b'pklxkd-58obj000000000000000000491000falseobj000000000000000000492023-05-31T20:38:18.527Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:39:16.634 DEBUG:boto:path=/pklxkd-58/obj00000000000000000049 2023-05-31T20:39:16.634 DEBUG:boto:auth_path=/pklxkd-58/obj00000000000000000049 2023-05-31T20:39:16.635 DEBUG:boto:Method: HEAD 2023-05-31T20:39:16.635 DEBUG:boto:Path: /pklxkd-58/obj00000000000000000049 2023-05-31T20:39:16.635 DEBUG:boto:Data: 2023-05-31T20:39:16.636 DEBUG:boto:Headers: {} 2023-05-31T20:39:16.636 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:39:16.637 DEBUG:boto:Port: 8001 2023-05-31T20:39:16.637 DEBUG:boto:Params: {} 2023-05-31T20:39:16.638 DEBUG:boto:Token: None 2023-05-31T20:39:16.638 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:39:16 GMT /pklxkd-58/obj00000000000000000049 2023-05-31T20:39:16.639 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:GQtTbARASvxwyA7U0PMEELUBuHk= 2023-05-31T20:39:16.639 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:39:16 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:GQtTbARASvxwyA7U0PMEELUBuHk=', 'Content-Length': '0'} 2023-05-31T20:39:16.641 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:38:18 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007f7ac6a52d25e758-006477b074-1c17-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:39:16 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:39:16.643 DEBUG:boto:path=/pklxkd-58/obj00000000000000000049 2023-05-31T20:39:16.644 DEBUG:boto:auth_path=/pklxkd-58/obj00000000000000000049 2023-05-31T20:39:16.645 DEBUG:boto:Method: HEAD 2023-05-31T20:39:16.645 DEBUG:boto:Path: /pklxkd-58/obj00000000000000000049 2023-05-31T20:39:16.646 DEBUG:boto:Data: 2023-05-31T20:39:16.647 DEBUG:boto:Headers: {} 2023-05-31T20:39:16.648 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:39:16.649 DEBUG:boto:Port: 8000 2023-05-31T20:39:16.649 DEBUG:boto:Params: {} 2023-05-31T20:39:16.650 DEBUG:boto:Token: None 2023-05-31T20:39:16.651 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:39:16 GMT /pklxkd-58/obj00000000000000000049 2023-05-31T20:39:16.652 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:GQtTbARASvxwyA7U0PMEELUBuHk= 2023-05-31T20:39:16.653 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:39:16 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:GQtTbARASvxwyA7U0PMEELUBuHk=', 'Content-Length': '0'} 2023-05-31T20:39:16.655 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:38:18 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e4035f27a2ae73ef-006477b074-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:39:16 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:39:16.655 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000049 2023-05-31T20:39:16.656 DEBUG:boto:path=/pklxkd-58/obj00000000000000000049 2023-05-31T20:39:16.656 DEBUG:boto:auth_path=/pklxkd-58/obj00000000000000000049 2023-05-31T20:39:16.657 DEBUG:boto:Method: GET 2023-05-31T20:39:16.657 DEBUG:boto:Path: /pklxkd-58/obj00000000000000000049 2023-05-31T20:39:16.658 DEBUG:boto:Data: 2023-05-31T20:39:16.658 DEBUG:boto:Headers: {} 2023-05-31T20:39:16.658 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:39:16.659 DEBUG:boto:Port: 8001 2023-05-31T20:39:16.659 DEBUG:boto:Params: {} 2023-05-31T20:39:16.660 DEBUG:boto:Token: None 2023-05-31T20:39:16.660 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:39:16 GMT /pklxkd-58/obj00000000000000000049 2023-05-31T20:39:16.661 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:F1Lg8/BNFpN9SF5qT/4glZ4WOKw= 2023-05-31T20:39:16.661 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:39:16 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:F1Lg8/BNFpN9SF5qT/4glZ4WOKw=', 'Content-Length': '0'} 2023-05-31T20:39:16.663 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:38:18 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003f2254cb51516961-006477b074-1c17-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:39:16 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:39:16.664 DEBUG:boto:path=/pklxkd-58/obj00000000000000000049 2023-05-31T20:39:16.664 DEBUG:boto:auth_path=/pklxkd-58/obj00000000000000000049 2023-05-31T20:39:16.665 DEBUG:boto:Method: GET 2023-05-31T20:39:16.665 DEBUG:boto:Path: /pklxkd-58/obj00000000000000000049 2023-05-31T20:39:16.666 DEBUG:boto:Data: 2023-05-31T20:39:16.666 DEBUG:boto:Headers: {} 2023-05-31T20:39:16.667 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:39:16.667 DEBUG:boto:Port: 8000 2023-05-31T20:39:16.668 DEBUG:boto:Params: {} 2023-05-31T20:39:16.668 DEBUG:boto:Token: None 2023-05-31T20:39:16.669 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:39:16 GMT /pklxkd-58/obj00000000000000000049 2023-05-31T20:39:16.669 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:F1Lg8/BNFpN9SF5qT/4glZ4WOKw= 2023-05-31T20:39:16.670 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:39:16 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:F1Lg8/BNFpN9SF5qT/4glZ4WOKw=', 'Content-Length': '0'} 2023-05-31T20:39:16.672 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:38:18 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000059d188b42f874467-006477b074-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:39:16 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:39:16.673 DEBUG:boto:path=/pklxkd-58/ 2023-05-31T20:39:16.674 DEBUG:boto:auth_path=/pklxkd-58/ 2023-05-31T20:39:16.675 DEBUG:boto:path=/pklxkd-58/?prefix=obj00000000000000000049 2023-05-31T20:39:16.676 DEBUG:boto:auth_path=/pklxkd-58/?prefix=obj00000000000000000049 2023-05-31T20:39:16.677 DEBUG:boto:Method: GET 2023-05-31T20:39:16.678 DEBUG:boto:Path: /pklxkd-58/?prefix=obj00000000000000000049 2023-05-31T20:39:16.678 DEBUG:boto:Data: 2023-05-31T20:39:16.679 DEBUG:boto:Headers: {} 2023-05-31T20:39:16.679 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:39:16.680 DEBUG:boto:Port: 8001 2023-05-31T20:39:16.680 DEBUG:boto:Params: {} 2023-05-31T20:39:16.681 DEBUG:boto:Token: None 2023-05-31T20:39:16.681 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:39:16 GMT /pklxkd-58/ 2023-05-31T20:39:16.682 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:LuRM/phPvIS2vQFGx8h56FjMq7Y= 2023-05-31T20:39:16.682 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:39:16 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:LuRM/phPvIS2vQFGx8h56FjMq7Y=', 'Content-Length': '0'} 2023-05-31T20:39:16.685 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000050d77a6921ee76de-006477b074-1c17-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:39:16 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:39:16.685 DEBUG:boto:b'pklxkd-58obj000000000000000000491000falseobj000000000000000000492023-05-31T20:38:18.527Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:39:16.686 DEBUG:boto:path=/pklxkd-58/ 2023-05-31T20:39:16.687 DEBUG:boto:auth_path=/pklxkd-58/ 2023-05-31T20:39:16.687 DEBUG:boto:path=/pklxkd-58/?prefix=obj00000000000000000049 2023-05-31T20:39:16.687 DEBUG:boto:auth_path=/pklxkd-58/?prefix=obj00000000000000000049 2023-05-31T20:39:16.688 DEBUG:boto:Method: GET 2023-05-31T20:39:16.688 DEBUG:boto:Path: /pklxkd-58/?prefix=obj00000000000000000049 2023-05-31T20:39:16.689 DEBUG:boto:Data: 2023-05-31T20:39:16.689 DEBUG:boto:Headers: {} 2023-05-31T20:39:16.689 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:39:16.690 DEBUG:boto:Port: 8000 2023-05-31T20:39:16.691 DEBUG:boto:Params: {} 2023-05-31T20:39:16.692 DEBUG:boto:Token: None 2023-05-31T20:39:16.693 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:39:16 GMT /pklxkd-58/ 2023-05-31T20:39:16.694 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:LuRM/phPvIS2vQFGx8h56FjMq7Y= 2023-05-31T20:39:16.695 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:39:16 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:LuRM/phPvIS2vQFGx8h56FjMq7Y=', 'Content-Length': '0'} 2023-05-31T20:39:16.700 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000006706790a97d10e74-006477b074-1630-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:39:16 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:39:16.740 DEBUG:boto:b'pklxkd-58obj000000000000000000491000falseobj000000000000000000492023-05-31T20:38:18.527Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:39:16.741 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000050 2023-05-31T20:39:16.742 DEBUG:boto:path=/pklxkd-58/obj00000000000000000050 2023-05-31T20:39:16.742 DEBUG:boto:auth_path=/pklxkd-58/obj00000000000000000050 2023-05-31T20:39:16.744 DEBUG:boto:path=/pklxkd-58/obj00000000000000000050?versionId=null 2023-05-31T20:39:16.744 DEBUG:boto:auth_path=/pklxkd-58/obj00000000000000000050?versionId=null 2023-05-31T20:39:16.745 DEBUG:boto:Method: GET 2023-05-31T20:39:16.746 DEBUG:boto:Path: /pklxkd-58/obj00000000000000000050?versionId=null 2023-05-31T20:39:16.746 DEBUG:boto:Data: 2023-05-31T20:39:16.747 DEBUG:boto:Headers: {} 2023-05-31T20:39:16.747 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:39:16.748 DEBUG:boto:Port: 8001 2023-05-31T20:39:16.748 DEBUG:boto:Params: {} 2023-05-31T20:39:16.748 DEBUG:boto:Token: None 2023-05-31T20:39:16.749 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:39:16 GMT /pklxkd-58/obj00000000000000000050?versionId=null 2023-05-31T20:39:16.749 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:lIgnasJbeIdCHKotWJg4cZ0XSQ4= 2023-05-31T20:39:16.750 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:39:16 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:lIgnasJbeIdCHKotWJg4cZ0XSQ4=', 'Content-Length': '0'} 2023-05-31T20:39:16.752 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:38:20 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a17be64594a8d36b-006477b074-1c17-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:39:16 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:39:16.753 DEBUG:boto:path=/pklxkd-58/obj00000000000000000050 2023-05-31T20:39:16.753 DEBUG:boto:auth_path=/pklxkd-58/obj00000000000000000050 2023-05-31T20:39:16.754 DEBUG:boto:path=/pklxkd-58/obj00000000000000000050?versionId=null 2023-05-31T20:39:16.754 DEBUG:boto:auth_path=/pklxkd-58/obj00000000000000000050?versionId=null 2023-05-31T20:39:16.754 DEBUG:boto:Method: GET 2023-05-31T20:39:16.755 DEBUG:boto:Path: /pklxkd-58/obj00000000000000000050?versionId=null 2023-05-31T20:39:16.756 DEBUG:boto:Data: 2023-05-31T20:39:16.756 DEBUG:boto:Headers: {} 2023-05-31T20:39:16.756 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:39:16.757 DEBUG:boto:Port: 8000 2023-05-31T20:39:16.757 DEBUG:boto:Params: {} 2023-05-31T20:39:16.757 DEBUG:boto:Token: None 2023-05-31T20:39:16.758 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:39:16 GMT /pklxkd-58/obj00000000000000000050?versionId=null 2023-05-31T20:39:16.758 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:lIgnasJbeIdCHKotWJg4cZ0XSQ4= 2023-05-31T20:39:16.759 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:39:16 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:lIgnasJbeIdCHKotWJg4cZ0XSQ4=', 'Content-Length': '0'} 2023-05-31T20:39:16.761 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:38:20 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002f5cfb2945678a60-006477b074-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:39:16 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:39:16.761 DEBUG:boto:path=/pklxkd-58/obj00000000000000000050 2023-05-31T20:39:16.762 DEBUG:boto:auth_path=/pklxkd-58/obj00000000000000000050 2023-05-31T20:39:16.762 DEBUG:boto:path=/pklxkd-58/obj00000000000000000050?versionId=null 2023-05-31T20:39:16.762 DEBUG:boto:auth_path=/pklxkd-58/obj00000000000000000050?versionId=null 2023-05-31T20:39:16.763 DEBUG:boto:Method: HEAD 2023-05-31T20:39:16.763 DEBUG:boto:Path: /pklxkd-58/obj00000000000000000050?versionId=null 2023-05-31T20:39:16.763 DEBUG:boto:Data: 2023-05-31T20:39:16.764 DEBUG:boto:Headers: {} 2023-05-31T20:39:16.764 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:39:16.764 DEBUG:boto:Port: 8001 2023-05-31T20:39:16.765 DEBUG:boto:Params: {} 2023-05-31T20:39:16.765 DEBUG:boto:Token: None 2023-05-31T20:39:16.766 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:39:16 GMT /pklxkd-58/obj00000000000000000050?versionId=null 2023-05-31T20:39:16.766 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:SetgygQe5uqLrIhJi8rS8VPcHS8= 2023-05-31T20:39:16.767 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:39:16 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:SetgygQe5uqLrIhJi8rS8VPcHS8=', 'Content-Length': '0'} 2023-05-31T20:39:16.769 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:38:20 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000090089244055572f0-006477b074-1c17-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:39:16 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:39:16.769 DEBUG:boto:path=/pklxkd-58/obj00000000000000000050 2023-05-31T20:39:16.770 DEBUG:boto:auth_path=/pklxkd-58/obj00000000000000000050 2023-05-31T20:39:16.770 DEBUG:boto:path=/pklxkd-58/obj00000000000000000050?versionId=null 2023-05-31T20:39:16.770 DEBUG:boto:auth_path=/pklxkd-58/obj00000000000000000050?versionId=null 2023-05-31T20:39:16.771 DEBUG:boto:Method: HEAD 2023-05-31T20:39:16.771 DEBUG:boto:Path: /pklxkd-58/obj00000000000000000050?versionId=null 2023-05-31T20:39:16.771 DEBUG:boto:Data: 2023-05-31T20:39:16.772 DEBUG:boto:Headers: {} 2023-05-31T20:39:16.772 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:39:16.772 DEBUG:boto:Port: 8000 2023-05-31T20:39:16.773 DEBUG:boto:Params: {} 2023-05-31T20:39:16.773 DEBUG:boto:Token: None 2023-05-31T20:39:16.773 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:39:16 GMT /pklxkd-58/obj00000000000000000050?versionId=null 2023-05-31T20:39:16.774 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:SetgygQe5uqLrIhJi8rS8VPcHS8= 2023-05-31T20:39:16.774 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:39:16 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:SetgygQe5uqLrIhJi8rS8VPcHS8=', 'Content-Length': '0'} 2023-05-31T20:39:16.776 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:38:20 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d2d00a55657025f1-006477b074-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:39:16 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:39:16.776 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000050 2023-05-31T20:39:16.777 DEBUG:boto:path=/pklxkd-58/obj00000000000000000050 2023-05-31T20:39:16.777 DEBUG:boto:auth_path=/pklxkd-58/obj00000000000000000050 2023-05-31T20:39:16.778 DEBUG:boto:path=/pklxkd-58/obj00000000000000000050?versionId=null 2023-05-31T20:39:16.778 DEBUG:boto:auth_path=/pklxkd-58/obj00000000000000000050?versionId=null 2023-05-31T20:39:16.778 DEBUG:boto:Method: GET 2023-05-31T20:39:16.779 DEBUG:boto:Path: /pklxkd-58/obj00000000000000000050?versionId=null 2023-05-31T20:39:16.779 DEBUG:boto:Data: 2023-05-31T20:39:16.779 DEBUG:boto:Headers: {} 2023-05-31T20:39:16.780 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:39:16.780 DEBUG:boto:Port: 8001 2023-05-31T20:39:16.781 DEBUG:boto:Params: {} 2023-05-31T20:39:16.781 DEBUG:boto:Token: None 2023-05-31T20:39:16.782 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:39:16 GMT /pklxkd-58/obj00000000000000000050?versionId=null 2023-05-31T20:39:16.782 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:lIgnasJbeIdCHKotWJg4cZ0XSQ4= 2023-05-31T20:39:16.782 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:39:16 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:lIgnasJbeIdCHKotWJg4cZ0XSQ4=', 'Content-Length': '0'} 2023-05-31T20:39:16.784 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:38:20 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006f9dd20406187be3-006477b074-1c17-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:39:16 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:39:16.832 DEBUG:boto:path=/pklxkd-58/obj00000000000000000050 2023-05-31T20:39:16.833 DEBUG:boto:auth_path=/pklxkd-58/obj00000000000000000050 2023-05-31T20:39:16.834 DEBUG:boto:path=/pklxkd-58/obj00000000000000000050?versionId=null 2023-05-31T20:39:16.834 DEBUG:boto:auth_path=/pklxkd-58/obj00000000000000000050?versionId=null 2023-05-31T20:39:16.835 DEBUG:boto:Method: GET 2023-05-31T20:39:16.835 DEBUG:boto:Path: /pklxkd-58/obj00000000000000000050?versionId=null 2023-05-31T20:39:16.835 DEBUG:boto:Data: 2023-05-31T20:39:16.836 DEBUG:boto:Headers: {} 2023-05-31T20:39:16.836 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:39:16.837 DEBUG:boto:Port: 8000 2023-05-31T20:39:16.837 DEBUG:boto:Params: {} 2023-05-31T20:39:16.838 DEBUG:boto:Token: None 2023-05-31T20:39:16.838 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:39:16 GMT /pklxkd-58/obj00000000000000000050?versionId=null 2023-05-31T20:39:16.838 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:lIgnasJbeIdCHKotWJg4cZ0XSQ4= 2023-05-31T20:39:16.839 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:39:16 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:lIgnasJbeIdCHKotWJg4cZ0XSQ4=', 'Content-Length': '0'} 2023-05-31T20:39:16.841 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:38:20 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000006ff6d25bab0062a-006477b074-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:39:16 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:39:16.842 DEBUG:boto:path=/pklxkd-58/ 2023-05-31T20:39:16.842 DEBUG:boto:auth_path=/pklxkd-58/ 2023-05-31T20:39:16.843 DEBUG:boto:path=/pklxkd-58/?prefix=obj00000000000000000050 2023-05-31T20:39:16.843 DEBUG:boto:auth_path=/pklxkd-58/?prefix=obj00000000000000000050 2023-05-31T20:39:16.843 DEBUG:boto:Method: GET 2023-05-31T20:39:16.844 DEBUG:boto:Path: /pklxkd-58/?prefix=obj00000000000000000050 2023-05-31T20:39:16.844 DEBUG:boto:Data: 2023-05-31T20:39:16.844 DEBUG:boto:Headers: {} 2023-05-31T20:39:16.845 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:39:16.845 DEBUG:boto:Port: 8001 2023-05-31T20:39:16.846 DEBUG:boto:Params: {} 2023-05-31T20:39:16.846 DEBUG:boto:Token: None 2023-05-31T20:39:16.846 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:39:16 GMT /pklxkd-58/ 2023-05-31T20:39:16.847 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:LuRM/phPvIS2vQFGx8h56FjMq7Y= 2023-05-31T20:39:16.847 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:39:16 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:LuRM/phPvIS2vQFGx8h56FjMq7Y=', 'Content-Length': '0'} 2023-05-31T20:39:16.850 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000073775a11f398fde2-006477b074-1c17-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:39:16 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:39:16.881 DEBUG:boto:b'pklxkd-58obj000000000000000000501000falseobj000000000000000000502023-05-31T20:38:20.183Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:39:16.882 DEBUG:boto:path=/pklxkd-58/ 2023-05-31T20:39:16.882 DEBUG:boto:auth_path=/pklxkd-58/ 2023-05-31T20:39:16.883 DEBUG:boto:path=/pklxkd-58/?prefix=obj00000000000000000050 2023-05-31T20:39:16.883 DEBUG:boto:auth_path=/pklxkd-58/?prefix=obj00000000000000000050 2023-05-31T20:39:16.884 DEBUG:boto:Method: GET 2023-05-31T20:39:16.884 DEBUG:boto:Path: /pklxkd-58/?prefix=obj00000000000000000050 2023-05-31T20:39:16.884 DEBUG:boto:Data: 2023-05-31T20:39:16.885 DEBUG:boto:Headers: {} 2023-05-31T20:39:16.885 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:39:16.885 DEBUG:boto:Port: 8000 2023-05-31T20:39:16.886 DEBUG:boto:Params: {} 2023-05-31T20:39:16.886 DEBUG:boto:Token: None 2023-05-31T20:39:16.887 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:39:16 GMT /pklxkd-58/ 2023-05-31T20:39:16.888 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:LuRM/phPvIS2vQFGx8h56FjMq7Y= 2023-05-31T20:39:16.889 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:39:16 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:LuRM/phPvIS2vQFGx8h56FjMq7Y=', 'Content-Length': '0'} 2023-05-31T20:39:16.893 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f7999921abc679f0-006477b074-1630-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:39:16 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:39:16.893 DEBUG:boto:b'pklxkd-58obj000000000000000000501000falseobj000000000000000000502023-05-31T20:38:20.183Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:39:16.894 DEBUG:boto:path=/pklxkd-58/obj00000000000000000050 2023-05-31T20:39:16.894 DEBUG:boto:auth_path=/pklxkd-58/obj00000000000000000050 2023-05-31T20:39:16.895 DEBUG:boto:Method: HEAD 2023-05-31T20:39:16.895 DEBUG:boto:Path: /pklxkd-58/obj00000000000000000050 2023-05-31T20:39:16.895 DEBUG:boto:Data: 2023-05-31T20:39:16.896 DEBUG:boto:Headers: {} 2023-05-31T20:39:16.896 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:39:16.896 DEBUG:boto:Port: 8001 2023-05-31T20:39:16.897 DEBUG:boto:Params: {} 2023-05-31T20:39:16.897 DEBUG:boto:Token: None 2023-05-31T20:39:16.897 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:39:16 GMT /pklxkd-58/obj00000000000000000050 2023-05-31T20:39:16.898 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:Yaz2jZADVTdJm02t5RMP1605IOs= 2023-05-31T20:39:16.899 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:39:16 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:Yaz2jZADVTdJm02t5RMP1605IOs=', 'Content-Length': '0'} 2023-05-31T20:39:16.901 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:38:20 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000072f1148a17344c08-006477b074-1c17-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:39:16 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:39:16.902 DEBUG:boto:path=/pklxkd-58/obj00000000000000000050 2023-05-31T20:39:16.902 DEBUG:boto:auth_path=/pklxkd-58/obj00000000000000000050 2023-05-31T20:39:16.903 DEBUG:boto:Method: HEAD 2023-05-31T20:39:16.903 DEBUG:boto:Path: /pklxkd-58/obj00000000000000000050 2023-05-31T20:39:16.903 DEBUG:boto:Data: 2023-05-31T20:39:16.904 DEBUG:boto:Headers: {} 2023-05-31T20:39:16.904 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:39:16.904 DEBUG:boto:Port: 8000 2023-05-31T20:39:16.905 DEBUG:boto:Params: {} 2023-05-31T20:39:16.905 DEBUG:boto:Token: None 2023-05-31T20:39:16.905 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:39:16 GMT /pklxkd-58/obj00000000000000000050 2023-05-31T20:39:16.906 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:Yaz2jZADVTdJm02t5RMP1605IOs= 2023-05-31T20:39:16.906 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:39:16 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:Yaz2jZADVTdJm02t5RMP1605IOs=', 'Content-Length': '0'} 2023-05-31T20:39:16.909 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:38:20 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000077903e04df703f54-006477b074-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:39:16 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:39:16.910 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000050 2023-05-31T20:39:16.910 DEBUG:boto:path=/pklxkd-58/obj00000000000000000050 2023-05-31T20:39:16.910 DEBUG:boto:auth_path=/pklxkd-58/obj00000000000000000050 2023-05-31T20:39:16.911 DEBUG:boto:Method: GET 2023-05-31T20:39:16.912 DEBUG:boto:Path: /pklxkd-58/obj00000000000000000050 2023-05-31T20:39:16.912 DEBUG:boto:Data: 2023-05-31T20:39:16.912 DEBUG:boto:Headers: {} 2023-05-31T20:39:16.913 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:39:16.913 DEBUG:boto:Port: 8001 2023-05-31T20:39:16.913 DEBUG:boto:Params: {} 2023-05-31T20:39:16.914 DEBUG:boto:Token: None 2023-05-31T20:39:16.914 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:39:16 GMT /pklxkd-58/obj00000000000000000050 2023-05-31T20:39:16.915 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:PCxxUtE8z+tPA3Uyg5TX6SC32n8= 2023-05-31T20:39:16.915 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:39:16 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:PCxxUtE8z+tPA3Uyg5TX6SC32n8=', 'Content-Length': '0'} 2023-05-31T20:39:16.917 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:38:20 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003b285ef8c2459ef4-006477b074-1c17-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:39:16 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:39:16.918 DEBUG:boto:path=/pklxkd-58/obj00000000000000000050 2023-05-31T20:39:16.918 DEBUG:boto:auth_path=/pklxkd-58/obj00000000000000000050 2023-05-31T20:39:16.919 DEBUG:boto:Method: GET 2023-05-31T20:39:16.919 DEBUG:boto:Path: /pklxkd-58/obj00000000000000000050 2023-05-31T20:39:16.920 DEBUG:boto:Data: 2023-05-31T20:39:16.920 DEBUG:boto:Headers: {} 2023-05-31T20:39:16.921 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:39:16.921 DEBUG:boto:Port: 8000 2023-05-31T20:39:16.921 DEBUG:boto:Params: {} 2023-05-31T20:39:16.922 DEBUG:boto:Token: None 2023-05-31T20:39:16.923 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:39:16 GMT /pklxkd-58/obj00000000000000000050 2023-05-31T20:39:16.923 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:PCxxUtE8z+tPA3Uyg5TX6SC32n8= 2023-05-31T20:39:16.923 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:39:16 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:PCxxUtE8z+tPA3Uyg5TX6SC32n8=', 'Content-Length': '0'} 2023-05-31T20:39:16.925 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:38:20 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b9c99c8adf652cf0-006477b074-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:39:16 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:39:16.968 DEBUG:boto:path=/pklxkd-58/ 2023-05-31T20:39:16.969 DEBUG:boto:auth_path=/pklxkd-58/ 2023-05-31T20:39:16.969 DEBUG:boto:path=/pklxkd-58/?prefix=obj00000000000000000050 2023-05-31T20:39:16.970 DEBUG:boto:auth_path=/pklxkd-58/?prefix=obj00000000000000000050 2023-05-31T20:39:16.970 DEBUG:boto:Method: GET 2023-05-31T20:39:16.970 DEBUG:boto:Path: /pklxkd-58/?prefix=obj00000000000000000050 2023-05-31T20:39:16.971 DEBUG:boto:Data: 2023-05-31T20:39:16.971 DEBUG:boto:Headers: {} 2023-05-31T20:39:16.971 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:39:16.972 DEBUG:boto:Port: 8001 2023-05-31T20:39:16.972 DEBUG:boto:Params: {} 2023-05-31T20:39:16.973 DEBUG:boto:Token: None 2023-05-31T20:39:16.973 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:39:16 GMT /pklxkd-58/ 2023-05-31T20:39:16.974 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:LuRM/phPvIS2vQFGx8h56FjMq7Y= 2023-05-31T20:39:16.974 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:39:16 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:LuRM/phPvIS2vQFGx8h56FjMq7Y=', 'Content-Length': '0'} 2023-05-31T20:39:16.977 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000009e2fe105b9ff8b89-006477b074-1c17-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:39:16 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:39:16.977 DEBUG:boto:b'pklxkd-58obj000000000000000000501000falseobj000000000000000000502023-05-31T20:38:20.183Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:39:16.978 DEBUG:boto:path=/pklxkd-58/ 2023-05-31T20:39:16.978 DEBUG:boto:auth_path=/pklxkd-58/ 2023-05-31T20:39:16.979 DEBUG:boto:path=/pklxkd-58/?prefix=obj00000000000000000050 2023-05-31T20:39:16.979 DEBUG:boto:auth_path=/pklxkd-58/?prefix=obj00000000000000000050 2023-05-31T20:39:16.980 DEBUG:boto:Method: GET 2023-05-31T20:39:16.980 DEBUG:boto:Path: /pklxkd-58/?prefix=obj00000000000000000050 2023-05-31T20:39:16.980 DEBUG:boto:Data: 2023-05-31T20:39:16.981 DEBUG:boto:Headers: {} 2023-05-31T20:39:16.981 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:39:16.981 DEBUG:boto:Port: 8000 2023-05-31T20:39:16.982 DEBUG:boto:Params: {} 2023-05-31T20:39:16.982 DEBUG:boto:Token: None 2023-05-31T20:39:16.983 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:39:16 GMT /pklxkd-58/ 2023-05-31T20:39:16.983 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:LuRM/phPvIS2vQFGx8h56FjMq7Y= 2023-05-31T20:39:16.984 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:39:16 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:LuRM/phPvIS2vQFGx8h56FjMq7Y=', 'Content-Length': '0'} 2023-05-31T20:39:16.987 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000004a9586dc933d2456-006477b074-1630-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:39:16 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:39:16.988 DEBUG:boto:b'pklxkd-58obj000000000000000000501000falseobj000000000000000000502023-05-31T20:38:20.183Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:39:16.989 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000053 2023-05-31T20:39:16.989 DEBUG:boto:path=/pklxkd-58/obj00000000000000000053 2023-05-31T20:39:16.989 DEBUG:boto:auth_path=/pklxkd-58/obj00000000000000000053 2023-05-31T20:39:16.990 DEBUG:boto:path=/pklxkd-58/obj00000000000000000053?versionId=null 2023-05-31T20:39:16.990 DEBUG:boto:auth_path=/pklxkd-58/obj00000000000000000053?versionId=null 2023-05-31T20:39:16.991 DEBUG:boto:Method: GET 2023-05-31T20:39:16.991 DEBUG:boto:Path: /pklxkd-58/obj00000000000000000053?versionId=null 2023-05-31T20:39:16.991 DEBUG:boto:Data: 2023-05-31T20:39:16.992 DEBUG:boto:Headers: {} 2023-05-31T20:39:16.992 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:39:16.993 DEBUG:boto:Port: 8001 2023-05-31T20:39:16.993 DEBUG:boto:Params: {} 2023-05-31T20:39:16.993 DEBUG:boto:Token: None 2023-05-31T20:39:16.994 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:39:16 GMT /pklxkd-58/obj00000000000000000053?versionId=null 2023-05-31T20:39:16.995 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:uUsFWDArh4iEwPESmDyGABC/Ko4= 2023-05-31T20:39:16.995 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:39:16 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:uUsFWDArh4iEwPESmDyGABC/Ko4=', 'Content-Length': '0'} 2023-05-31T20:39:16.998 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:38:22 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005cf9a73ba0bf714f-006477b074-1c17-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:39:16 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:39:17.044 DEBUG:boto:path=/pklxkd-58/obj00000000000000000053 2023-05-31T20:39:17.046 DEBUG:boto:auth_path=/pklxkd-58/obj00000000000000000053 2023-05-31T20:39:17.047 DEBUG:boto:path=/pklxkd-58/obj00000000000000000053?versionId=null 2023-05-31T20:39:17.048 DEBUG:boto:auth_path=/pklxkd-58/obj00000000000000000053?versionId=null 2023-05-31T20:39:17.049 DEBUG:boto:Method: GET 2023-05-31T20:39:17.050 DEBUG:boto:Path: /pklxkd-58/obj00000000000000000053?versionId=null 2023-05-31T20:39:17.051 DEBUG:boto:Data: 2023-05-31T20:39:17.052 DEBUG:boto:Headers: {} 2023-05-31T20:39:17.053 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:39:17.053 DEBUG:boto:Port: 8000 2023-05-31T20:39:17.053 DEBUG:boto:Params: {} 2023-05-31T20:39:17.054 DEBUG:boto:Token: None 2023-05-31T20:39:17.055 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:39:17 GMT /pklxkd-58/obj00000000000000000053?versionId=null 2023-05-31T20:39:17.055 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:H+LykaL4sJt8DMAlaAjapb3+yV4= 2023-05-31T20:39:17.056 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:39:17 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:H+LykaL4sJt8DMAlaAjapb3+yV4=', 'Content-Length': '0'} 2023-05-31T20:39:17.058 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:38:22 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000067ae2eef9a5fd252-006477b075-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:39:17 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:39:17.059 DEBUG:boto:path=/pklxkd-58/obj00000000000000000053 2023-05-31T20:39:17.059 DEBUG:boto:auth_path=/pklxkd-58/obj00000000000000000053 2023-05-31T20:39:17.060 DEBUG:boto:path=/pklxkd-58/obj00000000000000000053?versionId=null 2023-05-31T20:39:17.060 DEBUG:boto:auth_path=/pklxkd-58/obj00000000000000000053?versionId=null 2023-05-31T20:39:17.061 DEBUG:boto:Method: HEAD 2023-05-31T20:39:17.061 DEBUG:boto:Path: /pklxkd-58/obj00000000000000000053?versionId=null 2023-05-31T20:39:17.061 DEBUG:boto:Data: 2023-05-31T20:39:17.062 DEBUG:boto:Headers: {} 2023-05-31T20:39:17.062 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:39:17.063 DEBUG:boto:Port: 8001 2023-05-31T20:39:17.063 DEBUG:boto:Params: {} 2023-05-31T20:39:17.064 DEBUG:boto:Token: None 2023-05-31T20:39:17.065 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:39:17 GMT /pklxkd-58/obj00000000000000000053?versionId=null 2023-05-31T20:39:17.065 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:fZCnJsrTnAYZvwKPV928tReYeJs= 2023-05-31T20:39:17.066 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:39:17 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:fZCnJsrTnAYZvwKPV928tReYeJs=', 'Content-Length': '0'} 2023-05-31T20:39:17.068 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:38:22 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004d7a30a86c39a046-006477b075-1c17-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:39:17 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:39:17.069 DEBUG:boto:path=/pklxkd-58/obj00000000000000000053 2023-05-31T20:39:17.069 DEBUG:boto:auth_path=/pklxkd-58/obj00000000000000000053 2023-05-31T20:39:17.070 DEBUG:boto:path=/pklxkd-58/obj00000000000000000053?versionId=null 2023-05-31T20:39:17.070 DEBUG:boto:auth_path=/pklxkd-58/obj00000000000000000053?versionId=null 2023-05-31T20:39:17.071 DEBUG:boto:Method: HEAD 2023-05-31T20:39:17.071 DEBUG:boto:Path: /pklxkd-58/obj00000000000000000053?versionId=null 2023-05-31T20:39:17.071 DEBUG:boto:Data: 2023-05-31T20:39:17.072 DEBUG:boto:Headers: {} 2023-05-31T20:39:17.072 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:39:17.073 DEBUG:boto:Port: 8000 2023-05-31T20:39:17.073 DEBUG:boto:Params: {} 2023-05-31T20:39:17.073 DEBUG:boto:Token: None 2023-05-31T20:39:17.074 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:39:17 GMT /pklxkd-58/obj00000000000000000053?versionId=null 2023-05-31T20:39:17.074 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:fZCnJsrTnAYZvwKPV928tReYeJs= 2023-05-31T20:39:17.075 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:39:17 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:fZCnJsrTnAYZvwKPV928tReYeJs=', 'Content-Length': '0'} 2023-05-31T20:39:17.077 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:38:22 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ee6af3d825b5cb20-006477b075-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:39:17 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:39:17.078 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000053 2023-05-31T20:39:17.078 DEBUG:boto:path=/pklxkd-58/obj00000000000000000053 2023-05-31T20:39:17.079 DEBUG:boto:auth_path=/pklxkd-58/obj00000000000000000053 2023-05-31T20:39:17.079 DEBUG:boto:path=/pklxkd-58/obj00000000000000000053?versionId=null 2023-05-31T20:39:17.079 DEBUG:boto:auth_path=/pklxkd-58/obj00000000000000000053?versionId=null 2023-05-31T20:39:17.080 DEBUG:boto:Method: GET 2023-05-31T20:39:17.080 DEBUG:boto:Path: /pklxkd-58/obj00000000000000000053?versionId=null 2023-05-31T20:39:17.081 DEBUG:boto:Data: 2023-05-31T20:39:17.081 DEBUG:boto:Headers: {} 2023-05-31T20:39:17.081 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:39:17.082 DEBUG:boto:Port: 8001 2023-05-31T20:39:17.082 DEBUG:boto:Params: {} 2023-05-31T20:39:17.082 DEBUG:boto:Token: None 2023-05-31T20:39:17.083 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:39:17 GMT /pklxkd-58/obj00000000000000000053?versionId=null 2023-05-31T20:39:17.083 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:H+LykaL4sJt8DMAlaAjapb3+yV4= 2023-05-31T20:39:17.084 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:39:17 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:H+LykaL4sJt8DMAlaAjapb3+yV4=', 'Content-Length': '0'} 2023-05-31T20:39:17.086 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:38:22 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008881bb8d587bb551-006477b075-1c17-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:39:17 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:39:17.087 DEBUG:boto:path=/pklxkd-58/obj00000000000000000053 2023-05-31T20:39:17.087 DEBUG:boto:auth_path=/pklxkd-58/obj00000000000000000053 2023-05-31T20:39:17.088 DEBUG:boto:path=/pklxkd-58/obj00000000000000000053?versionId=null 2023-05-31T20:39:17.088 DEBUG:boto:auth_path=/pklxkd-58/obj00000000000000000053?versionId=null 2023-05-31T20:39:17.089 DEBUG:boto:Method: GET 2023-05-31T20:39:17.089 DEBUG:boto:Path: /pklxkd-58/obj00000000000000000053?versionId=null 2023-05-31T20:39:17.090 DEBUG:boto:Data: 2023-05-31T20:39:17.090 DEBUG:boto:Headers: {} 2023-05-31T20:39:17.090 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:39:17.091 DEBUG:boto:Port: 8000 2023-05-31T20:39:17.091 DEBUG:boto:Params: {} 2023-05-31T20:39:17.092 DEBUG:boto:Token: None 2023-05-31T20:39:17.093 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:39:17 GMT /pklxkd-58/obj00000000000000000053?versionId=null 2023-05-31T20:39:17.093 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:H+LykaL4sJt8DMAlaAjapb3+yV4= 2023-05-31T20:39:17.094 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:39:17 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:H+LykaL4sJt8DMAlaAjapb3+yV4=', 'Content-Length': '0'} 2023-05-31T20:39:17.096 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:38:22 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002d93100c6db9bb28-006477b075-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:39:17 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:39:17.098 DEBUG:boto:path=/pklxkd-58/ 2023-05-31T20:39:17.099 DEBUG:boto:auth_path=/pklxkd-58/ 2023-05-31T20:39:17.100 DEBUG:boto:path=/pklxkd-58/?prefix=obj00000000000000000053 2023-05-31T20:39:17.100 DEBUG:boto:auth_path=/pklxkd-58/?prefix=obj00000000000000000053 2023-05-31T20:39:17.100 DEBUG:boto:Method: GET 2023-05-31T20:39:17.101 DEBUG:boto:Path: /pklxkd-58/?prefix=obj00000000000000000053 2023-05-31T20:39:17.101 DEBUG:boto:Data: 2023-05-31T20:39:17.102 DEBUG:boto:Headers: {} 2023-05-31T20:39:17.102 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:39:17.103 DEBUG:boto:Port: 8001 2023-05-31T20:39:17.103 DEBUG:boto:Params: {} 2023-05-31T20:39:17.104 DEBUG:boto:Token: None 2023-05-31T20:39:17.104 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:39:17 GMT /pklxkd-58/ 2023-05-31T20:39:17.105 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:kahcpIxmAQ2hT4M2tN+pL/3o20Q= 2023-05-31T20:39:17.105 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:39:17 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:kahcpIxmAQ2hT4M2tN+pL/3o20Q=', 'Content-Length': '0'} 2023-05-31T20:39:17.108 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000043b9cd7727da9576-006477b075-1c17-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:39:17 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:39:17.108 DEBUG:boto:b'pklxkd-58obj000000000000000000531000falseobj000000000000000000532023-05-31T20:38:22.873Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:39:17.109 DEBUG:boto:path=/pklxkd-58/ 2023-05-31T20:39:17.110 DEBUG:boto:auth_path=/pklxkd-58/ 2023-05-31T20:39:17.110 DEBUG:boto:path=/pklxkd-58/?prefix=obj00000000000000000053 2023-05-31T20:39:17.111 DEBUG:boto:auth_path=/pklxkd-58/?prefix=obj00000000000000000053 2023-05-31T20:39:17.111 DEBUG:boto:Method: GET 2023-05-31T20:39:17.112 DEBUG:boto:Path: /pklxkd-58/?prefix=obj00000000000000000053 2023-05-31T20:39:17.112 DEBUG:boto:Data: 2023-05-31T20:39:17.113 DEBUG:boto:Headers: {} 2023-05-31T20:39:17.113 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:39:17.114 DEBUG:boto:Port: 8000 2023-05-31T20:39:17.114 DEBUG:boto:Params: {} 2023-05-31T20:39:17.115 DEBUG:boto:Token: None 2023-05-31T20:39:17.116 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:39:17 GMT /pklxkd-58/ 2023-05-31T20:39:17.116 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:kahcpIxmAQ2hT4M2tN+pL/3o20Q= 2023-05-31T20:39:17.117 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:39:17 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:kahcpIxmAQ2hT4M2tN+pL/3o20Q=', 'Content-Length': '0'} 2023-05-31T20:39:17.121 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a6ec68a239e3dd3c-006477b075-1630-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:39:17 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:39:17.121 DEBUG:boto:b'pklxkd-58obj000000000000000000531000falseobj000000000000000000532023-05-31T20:38:22.873Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:39:17.122 DEBUG:boto:path=/pklxkd-58/obj00000000000000000053 2023-05-31T20:39:17.123 DEBUG:boto:auth_path=/pklxkd-58/obj00000000000000000053 2023-05-31T20:39:17.124 DEBUG:boto:Method: HEAD 2023-05-31T20:39:17.125 DEBUG:boto:Path: /pklxkd-58/obj00000000000000000053 2023-05-31T20:39:17.126 DEBUG:boto:Data: 2023-05-31T20:39:17.127 DEBUG:boto:Headers: {} 2023-05-31T20:39:17.128 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:39:17.128 DEBUG:boto:Port: 8001 2023-05-31T20:39:17.129 DEBUG:boto:Params: {} 2023-05-31T20:39:17.130 DEBUG:boto:Token: None 2023-05-31T20:39:17.131 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:39:17 GMT /pklxkd-58/obj00000000000000000053 2023-05-31T20:39:17.131 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:tPtM4WK0RHwlnf3svfl6nr/oN5Q= 2023-05-31T20:39:17.132 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:39:17 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:tPtM4WK0RHwlnf3svfl6nr/oN5Q=', 'Content-Length': '0'} 2023-05-31T20:39:17.134 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:38:22 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000dfa9fdeea41829ff-006477b075-1c17-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:39:17 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:39:17.135 DEBUG:boto:path=/pklxkd-58/obj00000000000000000053 2023-05-31T20:39:17.136 DEBUG:boto:auth_path=/pklxkd-58/obj00000000000000000053 2023-05-31T20:39:17.136 DEBUG:boto:Method: HEAD 2023-05-31T20:39:17.137 DEBUG:boto:Path: /pklxkd-58/obj00000000000000000053 2023-05-31T20:39:17.138 DEBUG:boto:Data: 2023-05-31T20:39:17.139 DEBUG:boto:Headers: {} 2023-05-31T20:39:17.139 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:39:17.140 DEBUG:boto:Port: 8000 2023-05-31T20:39:17.141 DEBUG:boto:Params: {} 2023-05-31T20:39:17.142 DEBUG:boto:Token: None 2023-05-31T20:39:17.143 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:39:17 GMT /pklxkd-58/obj00000000000000000053 2023-05-31T20:39:17.144 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:tPtM4WK0RHwlnf3svfl6nr/oN5Q= 2023-05-31T20:39:17.145 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:39:17 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:tPtM4WK0RHwlnf3svfl6nr/oN5Q=', 'Content-Length': '0'} 2023-05-31T20:39:17.147 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:38:22 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002039d394e7fccac6-006477b075-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:39:17 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:39:17.148 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000053 2023-05-31T20:39:17.148 DEBUG:boto:path=/pklxkd-58/obj00000000000000000053 2023-05-31T20:39:17.149 DEBUG:boto:auth_path=/pklxkd-58/obj00000000000000000053 2023-05-31T20:39:17.149 DEBUG:boto:Method: GET 2023-05-31T20:39:17.149 DEBUG:boto:Path: /pklxkd-58/obj00000000000000000053 2023-05-31T20:39:17.150 DEBUG:boto:Data: 2023-05-31T20:39:17.150 DEBUG:boto:Headers: {} 2023-05-31T20:39:17.150 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:39:17.151 DEBUG:boto:Port: 8001 2023-05-31T20:39:17.151 DEBUG:boto:Params: {} 2023-05-31T20:39:17.152 DEBUG:boto:Token: None 2023-05-31T20:39:17.152 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:39:17 GMT /pklxkd-58/obj00000000000000000053 2023-05-31T20:39:17.153 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:+IeG+S/sC0ixW8Tl8RJ2zaFD0gI= 2023-05-31T20:39:17.153 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:39:17 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:+IeG+S/sC0ixW8Tl8RJ2zaFD0gI=', 'Content-Length': '0'} 2023-05-31T20:39:17.155 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:38:22 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000748e00ecdb874b4e-006477b075-1c17-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:39:17 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:39:17.156 DEBUG:boto:path=/pklxkd-58/obj00000000000000000053 2023-05-31T20:39:17.156 DEBUG:boto:auth_path=/pklxkd-58/obj00000000000000000053 2023-05-31T20:39:17.157 DEBUG:boto:Method: GET 2023-05-31T20:39:17.157 DEBUG:boto:Path: /pklxkd-58/obj00000000000000000053 2023-05-31T20:39:17.157 DEBUG:boto:Data: 2023-05-31T20:39:17.158 DEBUG:boto:Headers: {} 2023-05-31T20:39:17.158 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:39:17.159 DEBUG:boto:Port: 8000 2023-05-31T20:39:17.159 DEBUG:boto:Params: {} 2023-05-31T20:39:17.159 DEBUG:boto:Token: None 2023-05-31T20:39:17.160 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:39:17 GMT /pklxkd-58/obj00000000000000000053 2023-05-31T20:39:17.160 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:+IeG+S/sC0ixW8Tl8RJ2zaFD0gI= 2023-05-31T20:39:17.161 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:39:17 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:+IeG+S/sC0ixW8Tl8RJ2zaFD0gI=', 'Content-Length': '0'} 2023-05-31T20:39:17.163 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:38:22 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001a727dbebb7172fd-006477b075-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:39:17 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:39:17.204 DEBUG:boto:path=/pklxkd-58/ 2023-05-31T20:39:17.205 DEBUG:boto:auth_path=/pklxkd-58/ 2023-05-31T20:39:17.206 DEBUG:boto:path=/pklxkd-58/?prefix=obj00000000000000000053 2023-05-31T20:39:17.206 DEBUG:boto:auth_path=/pklxkd-58/?prefix=obj00000000000000000053 2023-05-31T20:39:17.206 DEBUG:boto:Method: GET 2023-05-31T20:39:17.207 DEBUG:boto:Path: /pklxkd-58/?prefix=obj00000000000000000053 2023-05-31T20:39:17.208 DEBUG:boto:Data: 2023-05-31T20:39:17.208 DEBUG:boto:Headers: {} 2023-05-31T20:39:17.209 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:39:17.210 DEBUG:boto:Port: 8001 2023-05-31T20:39:17.211 DEBUG:boto:Params: {} 2023-05-31T20:39:17.211 DEBUG:boto:Token: None 2023-05-31T20:39:17.212 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:39:17 GMT /pklxkd-58/ 2023-05-31T20:39:17.212 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:kahcpIxmAQ2hT4M2tN+pL/3o20Q= 2023-05-31T20:39:17.213 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:39:17 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:kahcpIxmAQ2hT4M2tN+pL/3o20Q=', 'Content-Length': '0'} 2023-05-31T20:39:17.216 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000086d0da28d8041f08-006477b075-1c17-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:39:17 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:39:17.217 DEBUG:boto:b'pklxkd-58obj000000000000000000531000falseobj000000000000000000532023-05-31T20:38:22.873Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:39:17.218 DEBUG:boto:path=/pklxkd-58/ 2023-05-31T20:39:17.218 DEBUG:boto:auth_path=/pklxkd-58/ 2023-05-31T20:39:17.218 DEBUG:boto:path=/pklxkd-58/?prefix=obj00000000000000000053 2023-05-31T20:39:17.219 DEBUG:boto:auth_path=/pklxkd-58/?prefix=obj00000000000000000053 2023-05-31T20:39:17.219 DEBUG:boto:Method: GET 2023-05-31T20:39:17.220 DEBUG:boto:Path: /pklxkd-58/?prefix=obj00000000000000000053 2023-05-31T20:39:17.220 DEBUG:boto:Data: 2023-05-31T20:39:17.220 DEBUG:boto:Headers: {} 2023-05-31T20:39:17.221 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:39:17.221 DEBUG:boto:Port: 8000 2023-05-31T20:39:17.221 DEBUG:boto:Params: {} 2023-05-31T20:39:17.222 DEBUG:boto:Token: None 2023-05-31T20:39:17.222 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:39:17 GMT /pklxkd-58/ 2023-05-31T20:39:17.223 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:kahcpIxmAQ2hT4M2tN+pL/3o20Q= 2023-05-31T20:39:17.223 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:39:17 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:kahcpIxmAQ2hT4M2tN+pL/3o20Q=', 'Content-Length': '0'} 2023-05-31T20:39:17.228 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000037e10939cec61561-006477b075-1630-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:39:17 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:39:17.229 DEBUG:boto:b'pklxkd-58obj000000000000000000531000falseobj000000000000000000532023-05-31T20:38:22.873Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:39:17.230 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000055 2023-05-31T20:39:17.230 DEBUG:boto:path=/pklxkd-58/obj00000000000000000055 2023-05-31T20:39:17.231 DEBUG:boto:auth_path=/pklxkd-58/obj00000000000000000055 2023-05-31T20:39:17.231 DEBUG:boto:path=/pklxkd-58/obj00000000000000000055?versionId=null 2023-05-31T20:39:17.232 DEBUG:boto:auth_path=/pklxkd-58/obj00000000000000000055?versionId=null 2023-05-31T20:39:17.232 DEBUG:boto:Method: GET 2023-05-31T20:39:17.232 DEBUG:boto:Path: /pklxkd-58/obj00000000000000000055?versionId=null 2023-05-31T20:39:17.233 DEBUG:boto:Data: 2023-05-31T20:39:17.233 DEBUG:boto:Headers: {} 2023-05-31T20:39:17.234 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:39:17.234 DEBUG:boto:Port: 8001 2023-05-31T20:39:17.234 DEBUG:boto:Params: {} 2023-05-31T20:39:17.235 DEBUG:boto:Token: None 2023-05-31T20:39:17.235 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:39:17 GMT /pklxkd-58/obj00000000000000000055?versionId=null 2023-05-31T20:39:17.236 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:VMGAULvoQmIfI/lmTE+vQXPKdLw= 2023-05-31T20:39:17.236 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:39:17 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:VMGAULvoQmIfI/lmTE+vQXPKdLw=', 'Content-Length': '0'} 2023-05-31T20:39:17.238 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:38:17 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006a1474cda1b0a0f4-006477b075-1c17-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:39:17 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:39:17.239 DEBUG:boto:path=/pklxkd-58/obj00000000000000000055 2023-05-31T20:39:17.239 DEBUG:boto:auth_path=/pklxkd-58/obj00000000000000000055 2023-05-31T20:39:17.240 DEBUG:boto:path=/pklxkd-58/obj00000000000000000055?versionId=null 2023-05-31T20:39:17.241 DEBUG:boto:auth_path=/pklxkd-58/obj00000000000000000055?versionId=null 2023-05-31T20:39:17.241 DEBUG:boto:Method: GET 2023-05-31T20:39:17.241 DEBUG:boto:Path: /pklxkd-58/obj00000000000000000055?versionId=null 2023-05-31T20:39:17.242 DEBUG:boto:Data: 2023-05-31T20:39:17.242 DEBUG:boto:Headers: {} 2023-05-31T20:39:17.245 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:39:17.246 DEBUG:boto:Port: 8000 2023-05-31T20:39:17.246 DEBUG:boto:Params: {} 2023-05-31T20:39:17.247 DEBUG:boto:Token: None 2023-05-31T20:39:17.248 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:39:17 GMT /pklxkd-58/obj00000000000000000055?versionId=null 2023-05-31T20:39:17.249 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:VMGAULvoQmIfI/lmTE+vQXPKdLw= 2023-05-31T20:39:17.250 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:39:17 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:VMGAULvoQmIfI/lmTE+vQXPKdLw=', 'Content-Length': '0'} 2023-05-31T20:39:17.252 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:38:17 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002f8a29fe716e1320-006477b075-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:39:17 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:39:17.253 DEBUG:boto:path=/pklxkd-58/obj00000000000000000055 2023-05-31T20:39:17.253 DEBUG:boto:auth_path=/pklxkd-58/obj00000000000000000055 2023-05-31T20:39:17.254 DEBUG:boto:path=/pklxkd-58/obj00000000000000000055?versionId=null 2023-05-31T20:39:17.254 DEBUG:boto:auth_path=/pklxkd-58/obj00000000000000000055?versionId=null 2023-05-31T20:39:17.254 DEBUG:boto:Method: HEAD 2023-05-31T20:39:17.255 DEBUG:boto:Path: /pklxkd-58/obj00000000000000000055?versionId=null 2023-05-31T20:39:17.255 DEBUG:boto:Data: 2023-05-31T20:39:17.256 DEBUG:boto:Headers: {} 2023-05-31T20:39:17.257 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:39:17.257 DEBUG:boto:Port: 8001 2023-05-31T20:39:17.257 DEBUG:boto:Params: {} 2023-05-31T20:39:17.258 DEBUG:boto:Token: None 2023-05-31T20:39:17.258 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:39:17 GMT /pklxkd-58/obj00000000000000000055?versionId=null 2023-05-31T20:39:17.259 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:POL/WFbY93N/0CIhK4GT8aEr9jk= 2023-05-31T20:39:17.259 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:39:17 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:POL/WFbY93N/0CIhK4GT8aEr9jk=', 'Content-Length': '0'} 2023-05-31T20:39:17.262 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:38:17 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d44ce832b910a1ff-006477b075-1c17-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:39:17 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:39:17.263 DEBUG:boto:path=/pklxkd-58/obj00000000000000000055 2023-05-31T20:39:17.264 DEBUG:boto:auth_path=/pklxkd-58/obj00000000000000000055 2023-05-31T20:39:17.265 DEBUG:boto:path=/pklxkd-58/obj00000000000000000055?versionId=null 2023-05-31T20:39:17.266 DEBUG:boto:auth_path=/pklxkd-58/obj00000000000000000055?versionId=null 2023-05-31T20:39:17.266 DEBUG:boto:Method: HEAD 2023-05-31T20:39:17.267 DEBUG:boto:Path: /pklxkd-58/obj00000000000000000055?versionId=null 2023-05-31T20:39:17.268 DEBUG:boto:Data: 2023-05-31T20:39:17.268 DEBUG:boto:Headers: {} 2023-05-31T20:39:17.269 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:39:17.270 DEBUG:boto:Port: 8000 2023-05-31T20:39:17.270 DEBUG:boto:Params: {} 2023-05-31T20:39:17.271 DEBUG:boto:Token: None 2023-05-31T20:39:17.271 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:39:17 GMT /pklxkd-58/obj00000000000000000055?versionId=null 2023-05-31T20:39:17.272 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:POL/WFbY93N/0CIhK4GT8aEr9jk= 2023-05-31T20:39:17.272 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:39:17 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:POL/WFbY93N/0CIhK4GT8aEr9jk=', 'Content-Length': '0'} 2023-05-31T20:39:17.274 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:38:17 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000af538e32efedf4c5-006477b075-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:39:17 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:39:17.275 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000055 2023-05-31T20:39:17.275 DEBUG:boto:path=/pklxkd-58/obj00000000000000000055 2023-05-31T20:39:17.276 DEBUG:boto:auth_path=/pklxkd-58/obj00000000000000000055 2023-05-31T20:39:17.276 DEBUG:boto:path=/pklxkd-58/obj00000000000000000055?versionId=null 2023-05-31T20:39:17.276 DEBUG:boto:auth_path=/pklxkd-58/obj00000000000000000055?versionId=null 2023-05-31T20:39:17.277 DEBUG:boto:Method: GET 2023-05-31T20:39:17.277 DEBUG:boto:Path: /pklxkd-58/obj00000000000000000055?versionId=null 2023-05-31T20:39:17.278 DEBUG:boto:Data: 2023-05-31T20:39:17.278 DEBUG:boto:Headers: {} 2023-05-31T20:39:17.278 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:39:17.279 DEBUG:boto:Port: 8001 2023-05-31T20:39:17.280 DEBUG:boto:Params: {} 2023-05-31T20:39:17.281 DEBUG:boto:Token: None 2023-05-31T20:39:17.282 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:39:17 GMT /pklxkd-58/obj00000000000000000055?versionId=null 2023-05-31T20:39:17.283 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:VMGAULvoQmIfI/lmTE+vQXPKdLw= 2023-05-31T20:39:17.284 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:39:17 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:VMGAULvoQmIfI/lmTE+vQXPKdLw=', 'Content-Length': '0'} 2023-05-31T20:39:17.286 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:38:17 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f10b0435d05aa2fe-006477b075-1c17-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:39:17 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:39:17.287 DEBUG:boto:path=/pklxkd-58/obj00000000000000000055 2023-05-31T20:39:17.288 DEBUG:boto:auth_path=/pklxkd-58/obj00000000000000000055 2023-05-31T20:39:17.288 DEBUG:boto:path=/pklxkd-58/obj00000000000000000055?versionId=null 2023-05-31T20:39:17.288 DEBUG:boto:auth_path=/pklxkd-58/obj00000000000000000055?versionId=null 2023-05-31T20:39:17.289 DEBUG:boto:Method: GET 2023-05-31T20:39:17.289 DEBUG:boto:Path: /pklxkd-58/obj00000000000000000055?versionId=null 2023-05-31T20:39:17.290 DEBUG:boto:Data: 2023-05-31T20:39:17.290 DEBUG:boto:Headers: {} 2023-05-31T20:39:17.290 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:39:17.291 DEBUG:boto:Port: 8000 2023-05-31T20:39:17.291 DEBUG:boto:Params: {} 2023-05-31T20:39:17.291 DEBUG:boto:Token: None 2023-05-31T20:39:17.292 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:39:17 GMT /pklxkd-58/obj00000000000000000055?versionId=null 2023-05-31T20:39:17.293 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:VMGAULvoQmIfI/lmTE+vQXPKdLw= 2023-05-31T20:39:17.293 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:39:17 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:VMGAULvoQmIfI/lmTE+vQXPKdLw=', 'Content-Length': '0'} 2023-05-31T20:39:17.295 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:38:17 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000050d13eb06f91ca03-006477b075-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:39:17 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:39:17.297 DEBUG:boto:path=/pklxkd-58/ 2023-05-31T20:39:17.298 DEBUG:boto:auth_path=/pklxkd-58/ 2023-05-31T20:39:17.299 DEBUG:boto:path=/pklxkd-58/?prefix=obj00000000000000000055 2023-05-31T20:39:17.299 DEBUG:boto:auth_path=/pklxkd-58/?prefix=obj00000000000000000055 2023-05-31T20:39:17.300 DEBUG:boto:Method: GET 2023-05-31T20:39:17.301 DEBUG:boto:Path: /pklxkd-58/?prefix=obj00000000000000000055 2023-05-31T20:39:17.302 DEBUG:boto:Data: 2023-05-31T20:39:17.303 DEBUG:boto:Headers: {} 2023-05-31T20:39:17.303 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:39:17.304 DEBUG:boto:Port: 8001 2023-05-31T20:39:17.305 DEBUG:boto:Params: {} 2023-05-31T20:39:17.306 DEBUG:boto:Token: None 2023-05-31T20:39:17.306 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:39:17 GMT /pklxkd-58/ 2023-05-31T20:39:17.307 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:kahcpIxmAQ2hT4M2tN+pL/3o20Q= 2023-05-31T20:39:17.307 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:39:17 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:kahcpIxmAQ2hT4M2tN+pL/3o20Q=', 'Content-Length': '0'} 2023-05-31T20:39:17.310 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000405c18773b254002-006477b075-1c17-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:39:17 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:39:17.311 DEBUG:boto:b'pklxkd-58obj000000000000000000551000falseobj000000000000000000552023-05-31T20:38:17.862Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:39:17.311 DEBUG:boto:path=/pklxkd-58/ 2023-05-31T20:39:17.312 DEBUG:boto:auth_path=/pklxkd-58/ 2023-05-31T20:39:17.312 DEBUG:boto:path=/pklxkd-58/?prefix=obj00000000000000000055 2023-05-31T20:39:17.312 DEBUG:boto:auth_path=/pklxkd-58/?prefix=obj00000000000000000055 2023-05-31T20:39:17.313 DEBUG:boto:Method: GET 2023-05-31T20:39:17.314 DEBUG:boto:Path: /pklxkd-58/?prefix=obj00000000000000000055 2023-05-31T20:39:17.315 DEBUG:boto:Data: 2023-05-31T20:39:17.316 DEBUG:boto:Headers: {} 2023-05-31T20:39:17.317 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:39:17.318 DEBUG:boto:Port: 8000 2023-05-31T20:39:17.319 DEBUG:boto:Params: {} 2023-05-31T20:39:17.320 DEBUG:boto:Token: None 2023-05-31T20:39:17.321 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:39:17 GMT /pklxkd-58/ 2023-05-31T20:39:17.321 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:kahcpIxmAQ2hT4M2tN+pL/3o20Q= 2023-05-31T20:39:17.322 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:39:17 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:kahcpIxmAQ2hT4M2tN+pL/3o20Q=', 'Content-Length': '0'} 2023-05-31T20:39:17.326 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000fdde1419867eb268-006477b075-1630-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:39:17 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:39:17.327 DEBUG:boto:b'pklxkd-58obj000000000000000000551000falseobj000000000000000000552023-05-31T20:38:17.862Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:39:17.328 DEBUG:boto:path=/pklxkd-58/obj00000000000000000055 2023-05-31T20:39:17.328 DEBUG:boto:auth_path=/pklxkd-58/obj00000000000000000055 2023-05-31T20:39:17.328 DEBUG:boto:Method: HEAD 2023-05-31T20:39:17.329 DEBUG:boto:Path: /pklxkd-58/obj00000000000000000055 2023-05-31T20:39:17.329 DEBUG:boto:Data: 2023-05-31T20:39:17.329 DEBUG:boto:Headers: {} 2023-05-31T20:39:17.330 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:39:17.331 DEBUG:boto:Port: 8001 2023-05-31T20:39:17.331 DEBUG:boto:Params: {} 2023-05-31T20:39:17.331 DEBUG:boto:Token: None 2023-05-31T20:39:17.332 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:39:17 GMT /pklxkd-58/obj00000000000000000055 2023-05-31T20:39:17.333 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:cX3YYA51eqEobDM+3TkDImiBWzw= 2023-05-31T20:39:17.334 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:39:17 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:cX3YYA51eqEobDM+3TkDImiBWzw=', 'Content-Length': '0'} 2023-05-31T20:39:17.337 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:38:17 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000001aafb1cf9ecb0be-006477b075-1c17-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:39:17 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:39:17.338 DEBUG:boto:path=/pklxkd-58/obj00000000000000000055 2023-05-31T20:39:17.339 DEBUG:boto:auth_path=/pklxkd-58/obj00000000000000000055 2023-05-31T20:39:17.340 DEBUG:boto:Method: HEAD 2023-05-31T20:39:17.340 DEBUG:boto:Path: /pklxkd-58/obj00000000000000000055 2023-05-31T20:39:17.341 DEBUG:boto:Data: 2023-05-31T20:39:17.342 DEBUG:boto:Headers: {} 2023-05-31T20:39:17.343 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:39:17.343 DEBUG:boto:Port: 8000 2023-05-31T20:39:17.344 DEBUG:boto:Params: {} 2023-05-31T20:39:17.344 DEBUG:boto:Token: None 2023-05-31T20:39:17.345 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:39:17 GMT /pklxkd-58/obj00000000000000000055 2023-05-31T20:39:17.345 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:cX3YYA51eqEobDM+3TkDImiBWzw= 2023-05-31T20:39:17.345 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:39:17 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:cX3YYA51eqEobDM+3TkDImiBWzw=', 'Content-Length': '0'} 2023-05-31T20:39:17.348 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:38:17 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000444904c19a8bb112-006477b075-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:39:17 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:39:17.348 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000055 2023-05-31T20:39:17.349 DEBUG:boto:path=/pklxkd-58/obj00000000000000000055 2023-05-31T20:39:17.350 DEBUG:boto:auth_path=/pklxkd-58/obj00000000000000000055 2023-05-31T20:39:17.351 DEBUG:boto:Method: GET 2023-05-31T20:39:17.352 DEBUG:boto:Path: /pklxkd-58/obj00000000000000000055 2023-05-31T20:39:17.353 DEBUG:boto:Data: 2023-05-31T20:39:17.354 DEBUG:boto:Headers: {} 2023-05-31T20:39:17.354 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:39:17.355 DEBUG:boto:Port: 8001 2023-05-31T20:39:17.356 DEBUG:boto:Params: {} 2023-05-31T20:39:17.357 DEBUG:boto:Token: None 2023-05-31T20:39:17.358 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:39:17 GMT /pklxkd-58/obj00000000000000000055 2023-05-31T20:39:17.358 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:+kQXit/KQJGEHgWOXpMqKDWbCsU= 2023-05-31T20:39:17.359 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:39:17 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:+kQXit/KQJGEHgWOXpMqKDWbCsU=', 'Content-Length': '0'} 2023-05-31T20:39:17.362 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:38:17 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000226ca6558c867bd2-006477b075-1c17-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:39:17 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:39:17.408 DEBUG:boto:path=/pklxkd-58/obj00000000000000000055 2023-05-31T20:39:17.409 DEBUG:boto:auth_path=/pklxkd-58/obj00000000000000000055 2023-05-31T20:39:17.410 DEBUG:boto:Method: GET 2023-05-31T20:39:17.411 DEBUG:boto:Path: /pklxkd-58/obj00000000000000000055 2023-05-31T20:39:17.412 DEBUG:boto:Data: 2023-05-31T20:39:17.412 DEBUG:boto:Headers: {} 2023-05-31T20:39:17.413 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:39:17.414 DEBUG:boto:Port: 8000 2023-05-31T20:39:17.415 DEBUG:boto:Params: {} 2023-05-31T20:39:17.415 DEBUG:boto:Token: None 2023-05-31T20:39:17.416 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:39:17 GMT /pklxkd-58/obj00000000000000000055 2023-05-31T20:39:17.416 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:+kQXit/KQJGEHgWOXpMqKDWbCsU= 2023-05-31T20:39:17.417 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:39:17 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:+kQXit/KQJGEHgWOXpMqKDWbCsU=', 'Content-Length': '0'} 2023-05-31T20:39:17.419 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:38:17 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000015c816142f23376b-006477b075-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:39:17 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:39:17.420 DEBUG:boto:path=/pklxkd-58/ 2023-05-31T20:39:17.420 DEBUG:boto:auth_path=/pklxkd-58/ 2023-05-31T20:39:17.421 DEBUG:boto:path=/pklxkd-58/?prefix=obj00000000000000000055 2023-05-31T20:39:17.421 DEBUG:boto:auth_path=/pklxkd-58/?prefix=obj00000000000000000055 2023-05-31T20:39:17.422 DEBUG:boto:Method: GET 2023-05-31T20:39:17.422 DEBUG:boto:Path: /pklxkd-58/?prefix=obj00000000000000000055 2023-05-31T20:39:17.422 DEBUG:boto:Data: 2023-05-31T20:39:17.423 DEBUG:boto:Headers: {} 2023-05-31T20:39:17.423 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:39:17.424 DEBUG:boto:Port: 8001 2023-05-31T20:39:17.424 DEBUG:boto:Params: {} 2023-05-31T20:39:17.425 DEBUG:boto:Token: None 2023-05-31T20:39:17.425 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:39:17 GMT /pklxkd-58/ 2023-05-31T20:39:17.425 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:kahcpIxmAQ2hT4M2tN+pL/3o20Q= 2023-05-31T20:39:17.426 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:39:17 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:kahcpIxmAQ2hT4M2tN+pL/3o20Q=', 'Content-Length': '0'} 2023-05-31T20:39:17.428 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ddfeab127d2b5f0b-006477b075-1c17-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:39:17 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:39:17.430 DEBUG:boto:b'pklxkd-58obj000000000000000000551000falseobj000000000000000000552023-05-31T20:38:17.862Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:39:17.431 DEBUG:boto:path=/pklxkd-58/ 2023-05-31T20:39:17.432 DEBUG:boto:auth_path=/pklxkd-58/ 2023-05-31T20:39:17.433 DEBUG:boto:path=/pklxkd-58/?prefix=obj00000000000000000055 2023-05-31T20:39:17.434 DEBUG:boto:auth_path=/pklxkd-58/?prefix=obj00000000000000000055 2023-05-31T20:39:17.434 DEBUG:boto:Method: GET 2023-05-31T20:39:17.435 DEBUG:boto:Path: /pklxkd-58/?prefix=obj00000000000000000055 2023-05-31T20:39:17.436 DEBUG:boto:Data: 2023-05-31T20:39:17.437 DEBUG:boto:Headers: {} 2023-05-31T20:39:17.437 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:39:17.438 DEBUG:boto:Port: 8000 2023-05-31T20:39:17.439 DEBUG:boto:Params: {} 2023-05-31T20:39:17.439 DEBUG:boto:Token: None 2023-05-31T20:39:17.440 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:39:17 GMT /pklxkd-58/ 2023-05-31T20:39:17.440 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:kahcpIxmAQ2hT4M2tN+pL/3o20Q= 2023-05-31T20:39:17.441 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:39:17 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:kahcpIxmAQ2hT4M2tN+pL/3o20Q=', 'Content-Length': '0'} 2023-05-31T20:39:17.445 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a343744bb604f4fa-006477b075-1630-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:39:17 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:39:17.447 DEBUG:boto:b'pklxkd-58obj000000000000000000551000falseobj000000000000000000552023-05-31T20:38:17.862Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:39:17.448 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000056 2023-05-31T20:39:17.448 DEBUG:boto:path=/pklxkd-58/obj00000000000000000056 2023-05-31T20:39:17.449 DEBUG:boto:auth_path=/pklxkd-58/obj00000000000000000056 2023-05-31T20:39:17.450 DEBUG:boto:path=/pklxkd-58/obj00000000000000000056?versionId=null 2023-05-31T20:39:17.451 DEBUG:boto:auth_path=/pklxkd-58/obj00000000000000000056?versionId=null 2023-05-31T20:39:17.451 DEBUG:boto:Method: GET 2023-05-31T20:39:17.451 DEBUG:boto:Path: /pklxkd-58/obj00000000000000000056?versionId=null 2023-05-31T20:39:17.452 DEBUG:boto:Data: 2023-05-31T20:39:17.452 DEBUG:boto:Headers: {} 2023-05-31T20:39:17.453 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:39:17.453 DEBUG:boto:Port: 8001 2023-05-31T20:39:17.454 DEBUG:boto:Params: {} 2023-05-31T20:39:17.455 DEBUG:boto:Token: None 2023-05-31T20:39:17.455 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:39:17 GMT /pklxkd-58/obj00000000000000000056?versionId=null 2023-05-31T20:39:17.457 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:Z59GbZcYZblSgTSmv5hMknWWVnA= 2023-05-31T20:39:17.457 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:39:17 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:Z59GbZcYZblSgTSmv5hMknWWVnA=', 'Content-Length': '0'} 2023-05-31T20:39:17.461 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:38:20 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001cabd45a375610b8-006477b075-1c17-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:39:17 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:39:17.462 DEBUG:boto:path=/pklxkd-58/obj00000000000000000056 2023-05-31T20:39:17.463 DEBUG:boto:auth_path=/pklxkd-58/obj00000000000000000056 2023-05-31T20:39:17.464 DEBUG:boto:path=/pklxkd-58/obj00000000000000000056?versionId=null 2023-05-31T20:39:17.465 DEBUG:boto:auth_path=/pklxkd-58/obj00000000000000000056?versionId=null 2023-05-31T20:39:17.466 DEBUG:boto:Method: GET 2023-05-31T20:39:17.467 DEBUG:boto:Path: /pklxkd-58/obj00000000000000000056?versionId=null 2023-05-31T20:39:17.468 DEBUG:boto:Data: 2023-05-31T20:39:17.469 DEBUG:boto:Headers: {} 2023-05-31T20:39:17.470 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:39:17.471 DEBUG:boto:Port: 8000 2023-05-31T20:39:17.471 DEBUG:boto:Params: {} 2023-05-31T20:39:17.472 DEBUG:boto:Token: None 2023-05-31T20:39:17.473 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:39:17 GMT /pklxkd-58/obj00000000000000000056?versionId=null 2023-05-31T20:39:17.474 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:Z59GbZcYZblSgTSmv5hMknWWVnA= 2023-05-31T20:39:17.475 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:39:17 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:Z59GbZcYZblSgTSmv5hMknWWVnA=', 'Content-Length': '0'} 2023-05-31T20:39:17.477 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:38:20 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000053efb202eb625a8f-006477b075-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:39:17 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:39:17.478 DEBUG:boto:path=/pklxkd-58/obj00000000000000000056 2023-05-31T20:39:17.478 DEBUG:boto:auth_path=/pklxkd-58/obj00000000000000000056 2023-05-31T20:39:17.479 DEBUG:boto:path=/pklxkd-58/obj00000000000000000056?versionId=null 2023-05-31T20:39:17.479 DEBUG:boto:auth_path=/pklxkd-58/obj00000000000000000056?versionId=null 2023-05-31T20:39:17.479 DEBUG:boto:Method: HEAD 2023-05-31T20:39:17.480 DEBUG:boto:Path: /pklxkd-58/obj00000000000000000056?versionId=null 2023-05-31T20:39:17.480 DEBUG:boto:Data: 2023-05-31T20:39:17.480 DEBUG:boto:Headers: {} 2023-05-31T20:39:17.481 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:39:17.481 DEBUG:boto:Port: 8001 2023-05-31T20:39:17.482 DEBUG:boto:Params: {} 2023-05-31T20:39:17.482 DEBUG:boto:Token: None 2023-05-31T20:39:17.483 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:39:17 GMT /pklxkd-58/obj00000000000000000056?versionId=null 2023-05-31T20:39:17.483 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:pCXqjLBLr2aZNAyDpMVOBQaxFqA= 2023-05-31T20:39:17.484 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:39:17 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:pCXqjLBLr2aZNAyDpMVOBQaxFqA=', 'Content-Length': '0'} 2023-05-31T20:39:17.486 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:38:20 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f3251d034696f90d-006477b075-1c17-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:39:17 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:39:17.487 DEBUG:boto:path=/pklxkd-58/obj00000000000000000056 2023-05-31T20:39:17.488 DEBUG:boto:auth_path=/pklxkd-58/obj00000000000000000056 2023-05-31T20:39:17.489 DEBUG:boto:path=/pklxkd-58/obj00000000000000000056?versionId=null 2023-05-31T20:39:17.489 DEBUG:boto:auth_path=/pklxkd-58/obj00000000000000000056?versionId=null 2023-05-31T20:39:17.490 DEBUG:boto:Method: HEAD 2023-05-31T20:39:17.491 DEBUG:boto:Path: /pklxkd-58/obj00000000000000000056?versionId=null 2023-05-31T20:39:17.491 DEBUG:boto:Data: 2023-05-31T20:39:17.492 DEBUG:boto:Headers: {} 2023-05-31T20:39:17.492 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:39:17.493 DEBUG:boto:Port: 8000 2023-05-31T20:39:17.493 DEBUG:boto:Params: {} 2023-05-31T20:39:17.494 DEBUG:boto:Token: None 2023-05-31T20:39:17.495 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:39:17 GMT /pklxkd-58/obj00000000000000000056?versionId=null 2023-05-31T20:39:17.496 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:pCXqjLBLr2aZNAyDpMVOBQaxFqA= 2023-05-31T20:39:17.496 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:39:17 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:pCXqjLBLr2aZNAyDpMVOBQaxFqA=', 'Content-Length': '0'} 2023-05-31T20:39:17.498 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:38:20 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b6ef1974bb2703a7-006477b075-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:39:17 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:39:17.499 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000056 2023-05-31T20:39:17.500 DEBUG:boto:path=/pklxkd-58/obj00000000000000000056 2023-05-31T20:39:17.500 DEBUG:boto:auth_path=/pklxkd-58/obj00000000000000000056 2023-05-31T20:39:17.501 DEBUG:boto:path=/pklxkd-58/obj00000000000000000056?versionId=null 2023-05-31T20:39:17.501 DEBUG:boto:auth_path=/pklxkd-58/obj00000000000000000056?versionId=null 2023-05-31T20:39:17.501 DEBUG:boto:Method: GET 2023-05-31T20:39:17.502 DEBUG:boto:Path: /pklxkd-58/obj00000000000000000056?versionId=null 2023-05-31T20:39:17.502 DEBUG:boto:Data: 2023-05-31T20:39:17.502 DEBUG:boto:Headers: {} 2023-05-31T20:39:17.503 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:39:17.503 DEBUG:boto:Port: 8001 2023-05-31T20:39:17.503 DEBUG:boto:Params: {} 2023-05-31T20:39:17.504 DEBUG:boto:Token: None 2023-05-31T20:39:17.504 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:39:17 GMT /pklxkd-58/obj00000000000000000056?versionId=null 2023-05-31T20:39:17.505 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:Z59GbZcYZblSgTSmv5hMknWWVnA= 2023-05-31T20:39:17.505 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:39:17 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:Z59GbZcYZblSgTSmv5hMknWWVnA=', 'Content-Length': '0'} 2023-05-31T20:39:17.507 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:38:20 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f55d1a1e114ab47c-006477b075-1c17-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:39:17 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:39:17.552 DEBUG:boto:path=/pklxkd-58/obj00000000000000000056 2023-05-31T20:39:17.553 DEBUG:boto:auth_path=/pklxkd-58/obj00000000000000000056 2023-05-31T20:39:17.553 DEBUG:boto:path=/pklxkd-58/obj00000000000000000056?versionId=null 2023-05-31T20:39:17.554 DEBUG:boto:auth_path=/pklxkd-58/obj00000000000000000056?versionId=null 2023-05-31T20:39:17.554 DEBUG:boto:Method: GET 2023-05-31T20:39:17.554 DEBUG:boto:Path: /pklxkd-58/obj00000000000000000056?versionId=null 2023-05-31T20:39:17.555 DEBUG:boto:Data: 2023-05-31T20:39:17.555 DEBUG:boto:Headers: {} 2023-05-31T20:39:17.555 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:39:17.556 DEBUG:boto:Port: 8000 2023-05-31T20:39:17.556 DEBUG:boto:Params: {} 2023-05-31T20:39:17.557 DEBUG:boto:Token: None 2023-05-31T20:39:17.558 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:39:17 GMT /pklxkd-58/obj00000000000000000056?versionId=null 2023-05-31T20:39:17.559 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:Z59GbZcYZblSgTSmv5hMknWWVnA= 2023-05-31T20:39:17.560 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:39:17 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:Z59GbZcYZblSgTSmv5hMknWWVnA=', 'Content-Length': '0'} 2023-05-31T20:39:17.563 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:38:20 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007883686014670f9b-006477b075-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:39:17 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:39:17.565 DEBUG:boto:path=/pklxkd-58/ 2023-05-31T20:39:17.565 DEBUG:boto:auth_path=/pklxkd-58/ 2023-05-31T20:39:17.566 DEBUG:boto:path=/pklxkd-58/?prefix=obj00000000000000000056 2023-05-31T20:39:17.567 DEBUG:boto:auth_path=/pklxkd-58/?prefix=obj00000000000000000056 2023-05-31T20:39:17.567 DEBUG:boto:Method: GET 2023-05-31T20:39:17.568 DEBUG:boto:Path: /pklxkd-58/?prefix=obj00000000000000000056 2023-05-31T20:39:17.568 DEBUG:boto:Data: 2023-05-31T20:39:17.569 DEBUG:boto:Headers: {} 2023-05-31T20:39:17.569 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:39:17.570 DEBUG:boto:Port: 8001 2023-05-31T20:39:17.570 DEBUG:boto:Params: {} 2023-05-31T20:39:17.571 DEBUG:boto:Token: None 2023-05-31T20:39:17.571 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:39:17 GMT /pklxkd-58/ 2023-05-31T20:39:17.572 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:kahcpIxmAQ2hT4M2tN+pL/3o20Q= 2023-05-31T20:39:17.573 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:39:17 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:kahcpIxmAQ2hT4M2tN+pL/3o20Q=', 'Content-Length': '0'} 2023-05-31T20:39:17.575 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000000afb4d29c523eb3-006477b075-1c17-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:39:17 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:39:17.576 DEBUG:boto:b'pklxkd-58obj000000000000000000561000falseobj000000000000000000562023-05-31T20:38:20.370Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:39:17.578 DEBUG:boto:path=/pklxkd-58/ 2023-05-31T20:39:17.579 DEBUG:boto:auth_path=/pklxkd-58/ 2023-05-31T20:39:17.579 DEBUG:boto:path=/pklxkd-58/?prefix=obj00000000000000000056 2023-05-31T20:39:17.580 DEBUG:boto:auth_path=/pklxkd-58/?prefix=obj00000000000000000056 2023-05-31T20:39:17.581 DEBUG:boto:Method: GET 2023-05-31T20:39:17.583 DEBUG:boto:Path: /pklxkd-58/?prefix=obj00000000000000000056 2023-05-31T20:39:17.583 DEBUG:boto:Data: 2023-05-31T20:39:17.584 DEBUG:boto:Headers: {} 2023-05-31T20:39:17.585 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:39:17.586 DEBUG:boto:Port: 8000 2023-05-31T20:39:17.587 DEBUG:boto:Params: {} 2023-05-31T20:39:17.587 DEBUG:boto:Token: None 2023-05-31T20:39:17.588 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:39:17 GMT /pklxkd-58/ 2023-05-31T20:39:17.589 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:kahcpIxmAQ2hT4M2tN+pL/3o20Q= 2023-05-31T20:39:17.590 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:39:17 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:kahcpIxmAQ2hT4M2tN+pL/3o20Q=', 'Content-Length': '0'} 2023-05-31T20:39:17.595 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000bec601dd7833b366-006477b075-1630-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:39:17 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:39:17.596 DEBUG:boto:b'pklxkd-58obj000000000000000000561000falseobj000000000000000000562023-05-31T20:38:20.370Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:39:17.597 DEBUG:boto:path=/pklxkd-58/obj00000000000000000056 2023-05-31T20:39:17.597 DEBUG:boto:auth_path=/pklxkd-58/obj00000000000000000056 2023-05-31T20:39:17.598 DEBUG:boto:Method: HEAD 2023-05-31T20:39:17.599 DEBUG:boto:Path: /pklxkd-58/obj00000000000000000056 2023-05-31T20:39:17.599 DEBUG:boto:Data: 2023-05-31T20:39:17.600 DEBUG:boto:Headers: {} 2023-05-31T20:39:17.601 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:39:17.602 DEBUG:boto:Port: 8001 2023-05-31T20:39:17.602 DEBUG:boto:Params: {} 2023-05-31T20:39:17.603 DEBUG:boto:Token: None 2023-05-31T20:39:17.603 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:39:17 GMT /pklxkd-58/obj00000000000000000056 2023-05-31T20:39:17.604 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:TKYgUFGXIvUsn4jSgdPCdXzeAGM= 2023-05-31T20:39:17.604 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:39:17 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:TKYgUFGXIvUsn4jSgdPCdXzeAGM=', 'Content-Length': '0'} 2023-05-31T20:39:17.606 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:38:20 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008815d37d018a7043-006477b075-1c17-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:39:17 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:39:17.607 DEBUG:boto:path=/pklxkd-58/obj00000000000000000056 2023-05-31T20:39:17.607 DEBUG:boto:auth_path=/pklxkd-58/obj00000000000000000056 2023-05-31T20:39:17.608 DEBUG:boto:Method: HEAD 2023-05-31T20:39:17.608 DEBUG:boto:Path: /pklxkd-58/obj00000000000000000056 2023-05-31T20:39:17.609 DEBUG:boto:Data: 2023-05-31T20:39:17.609 DEBUG:boto:Headers: {} 2023-05-31T20:39:17.609 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:39:17.610 DEBUG:boto:Port: 8000 2023-05-31T20:39:17.610 DEBUG:boto:Params: {} 2023-05-31T20:39:17.610 DEBUG:boto:Token: None 2023-05-31T20:39:17.611 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:39:17 GMT /pklxkd-58/obj00000000000000000056 2023-05-31T20:39:17.612 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:TKYgUFGXIvUsn4jSgdPCdXzeAGM= 2023-05-31T20:39:17.612 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:39:17 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:TKYgUFGXIvUsn4jSgdPCdXzeAGM=', 'Content-Length': '0'} 2023-05-31T20:39:17.614 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:38:20 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000096bbdc8db42cd134-006477b075-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:39:17 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:39:17.614 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000056 2023-05-31T20:39:17.615 DEBUG:boto:path=/pklxkd-58/obj00000000000000000056 2023-05-31T20:39:17.615 DEBUG:boto:auth_path=/pklxkd-58/obj00000000000000000056 2023-05-31T20:39:17.616 DEBUG:boto:Method: GET 2023-05-31T20:39:17.616 DEBUG:boto:Path: /pklxkd-58/obj00000000000000000056 2023-05-31T20:39:17.616 DEBUG:boto:Data: 2023-05-31T20:39:17.617 DEBUG:boto:Headers: {} 2023-05-31T20:39:17.617 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:39:17.618 DEBUG:boto:Port: 8001 2023-05-31T20:39:17.618 DEBUG:boto:Params: {} 2023-05-31T20:39:17.618 DEBUG:boto:Token: None 2023-05-31T20:39:17.619 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:39:17 GMT /pklxkd-58/obj00000000000000000056 2023-05-31T20:39:17.619 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:RiaW/8TsHqGRPXBarSOxBmpzD18= 2023-05-31T20:39:17.620 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:39:17 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:RiaW/8TsHqGRPXBarSOxBmpzD18=', 'Content-Length': '0'} 2023-05-31T20:39:17.622 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:38:20 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002c24b09108f5ed7d-006477b075-1c17-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:39:17 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:39:17.622 DEBUG:boto:path=/pklxkd-58/obj00000000000000000056 2023-05-31T20:39:17.623 DEBUG:boto:auth_path=/pklxkd-58/obj00000000000000000056 2023-05-31T20:39:17.623 DEBUG:boto:Method: GET 2023-05-31T20:39:17.624 DEBUG:boto:Path: /pklxkd-58/obj00000000000000000056 2023-05-31T20:39:17.624 DEBUG:boto:Data: 2023-05-31T20:39:17.624 DEBUG:boto:Headers: {} 2023-05-31T20:39:17.625 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:39:17.625 DEBUG:boto:Port: 8000 2023-05-31T20:39:17.625 DEBUG:boto:Params: {} 2023-05-31T20:39:17.626 DEBUG:boto:Token: None 2023-05-31T20:39:17.626 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:39:17 GMT /pklxkd-58/obj00000000000000000056 2023-05-31T20:39:17.627 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:RiaW/8TsHqGRPXBarSOxBmpzD18= 2023-05-31T20:39:17.627 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:39:17 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:RiaW/8TsHqGRPXBarSOxBmpzD18=', 'Content-Length': '0'} 2023-05-31T20:39:17.629 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:38:20 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000816acdd590361081-006477b075-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:39:17 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:39:17.672 DEBUG:boto:path=/pklxkd-58/ 2023-05-31T20:39:17.673 DEBUG:boto:auth_path=/pklxkd-58/ 2023-05-31T20:39:17.673 DEBUG:boto:path=/pklxkd-58/?prefix=obj00000000000000000056 2023-05-31T20:39:17.674 DEBUG:boto:auth_path=/pklxkd-58/?prefix=obj00000000000000000056 2023-05-31T20:39:17.674 DEBUG:boto:Method: GET 2023-05-31T20:39:17.674 DEBUG:boto:Path: /pklxkd-58/?prefix=obj00000000000000000056 2023-05-31T20:39:17.675 DEBUG:boto:Data: 2023-05-31T20:39:17.675 DEBUG:boto:Headers: {} 2023-05-31T20:39:17.675 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:39:17.676 DEBUG:boto:Port: 8001 2023-05-31T20:39:17.676 DEBUG:boto:Params: {} 2023-05-31T20:39:17.677 DEBUG:boto:Token: None 2023-05-31T20:39:17.677 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:39:17 GMT /pklxkd-58/ 2023-05-31T20:39:17.678 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:kahcpIxmAQ2hT4M2tN+pL/3o20Q= 2023-05-31T20:39:17.678 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:39:17 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:kahcpIxmAQ2hT4M2tN+pL/3o20Q=', 'Content-Length': '0'} 2023-05-31T20:39:17.680 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000008a15c3b90d7497a8-006477b075-1c17-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:39:17 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:39:17.681 DEBUG:boto:b'pklxkd-58obj000000000000000000561000falseobj000000000000000000562023-05-31T20:38:20.370Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:39:17.682 DEBUG:boto:path=/pklxkd-58/ 2023-05-31T20:39:17.682 DEBUG:boto:auth_path=/pklxkd-58/ 2023-05-31T20:39:17.682 DEBUG:boto:path=/pklxkd-58/?prefix=obj00000000000000000056 2023-05-31T20:39:17.683 DEBUG:boto:auth_path=/pklxkd-58/?prefix=obj00000000000000000056 2023-05-31T20:39:17.683 DEBUG:boto:Method: GET 2023-05-31T20:39:17.684 DEBUG:boto:Path: /pklxkd-58/?prefix=obj00000000000000000056 2023-05-31T20:39:17.684 DEBUG:boto:Data: 2023-05-31T20:39:17.684 DEBUG:boto:Headers: {} 2023-05-31T20:39:17.685 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:39:17.685 DEBUG:boto:Port: 8000 2023-05-31T20:39:17.685 DEBUG:boto:Params: {} 2023-05-31T20:39:17.686 DEBUG:boto:Token: None 2023-05-31T20:39:17.686 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:39:17 GMT /pklxkd-58/ 2023-05-31T20:39:17.687 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:kahcpIxmAQ2hT4M2tN+pL/3o20Q= 2023-05-31T20:39:17.687 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:39:17 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:kahcpIxmAQ2hT4M2tN+pL/3o20Q=', 'Content-Length': '0'} 2023-05-31T20:39:17.691 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000002baae892146aa04a-006477b075-1630-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:39:17 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:39:17.692 DEBUG:boto:b'pklxkd-58obj000000000000000000561000falseobj000000000000000000562023-05-31T20:38:20.370Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:39:17.693 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000057 2023-05-31T20:39:17.693 DEBUG:boto:path=/pklxkd-58/obj00000000000000000057 2023-05-31T20:39:17.693 DEBUG:boto:auth_path=/pklxkd-58/obj00000000000000000057 2023-05-31T20:39:17.694 DEBUG:boto:path=/pklxkd-58/obj00000000000000000057?versionId=null 2023-05-31T20:39:17.694 DEBUG:boto:auth_path=/pklxkd-58/obj00000000000000000057?versionId=null 2023-05-31T20:39:17.694 DEBUG:boto:Method: GET 2023-05-31T20:39:17.695 DEBUG:boto:Path: /pklxkd-58/obj00000000000000000057?versionId=null 2023-05-31T20:39:17.695 DEBUG:boto:Data: 2023-05-31T20:39:17.696 DEBUG:boto:Headers: {} 2023-05-31T20:39:17.696 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:39:17.697 DEBUG:boto:Port: 8001 2023-05-31T20:39:17.697 DEBUG:boto:Params: {} 2023-05-31T20:39:17.697 DEBUG:boto:Token: None 2023-05-31T20:39:17.698 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:39:17 GMT /pklxkd-58/obj00000000000000000057?versionId=null 2023-05-31T20:39:17.698 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:Xefs+zE0rD22dvqHAv56SWc0wvo= 2023-05-31T20:39:17.699 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:39:17 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:Xefs+zE0rD22dvqHAv56SWc0wvo=', 'Content-Length': '0'} 2023-05-31T20:39:17.701 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:38:21 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000020300a7de341e516-006477b075-1c17-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:39:17 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:39:17.701 DEBUG:boto:path=/pklxkd-58/obj00000000000000000057 2023-05-31T20:39:17.702 DEBUG:boto:auth_path=/pklxkd-58/obj00000000000000000057 2023-05-31T20:39:17.702 DEBUG:boto:path=/pklxkd-58/obj00000000000000000057?versionId=null 2023-05-31T20:39:17.703 DEBUG:boto:auth_path=/pklxkd-58/obj00000000000000000057?versionId=null 2023-05-31T20:39:17.703 DEBUG:boto:Method: GET 2023-05-31T20:39:17.703 DEBUG:boto:Path: /pklxkd-58/obj00000000000000000057?versionId=null 2023-05-31T20:39:17.704 DEBUG:boto:Data: 2023-05-31T20:39:17.704 DEBUG:boto:Headers: {} 2023-05-31T20:39:17.704 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:39:17.705 DEBUG:boto:Port: 8000 2023-05-31T20:39:17.705 DEBUG:boto:Params: {} 2023-05-31T20:39:17.706 DEBUG:boto:Token: None 2023-05-31T20:39:17.706 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:39:17 GMT /pklxkd-58/obj00000000000000000057?versionId=null 2023-05-31T20:39:17.706 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:Xefs+zE0rD22dvqHAv56SWc0wvo= 2023-05-31T20:39:17.707 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:39:17 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:Xefs+zE0rD22dvqHAv56SWc0wvo=', 'Content-Length': '0'} 2023-05-31T20:39:17.709 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:38:21 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b958605fccdfc70e-006477b075-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:39:17 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:39:17.710 DEBUG:boto:path=/pklxkd-58/obj00000000000000000057 2023-05-31T20:39:17.710 DEBUG:boto:auth_path=/pklxkd-58/obj00000000000000000057 2023-05-31T20:39:17.711 DEBUG:boto:path=/pklxkd-58/obj00000000000000000057?versionId=null 2023-05-31T20:39:17.712 DEBUG:boto:auth_path=/pklxkd-58/obj00000000000000000057?versionId=null 2023-05-31T20:39:17.713 DEBUG:boto:Method: HEAD 2023-05-31T20:39:17.714 DEBUG:boto:Path: /pklxkd-58/obj00000000000000000057?versionId=null 2023-05-31T20:39:17.715 DEBUG:boto:Data: 2023-05-31T20:39:17.716 DEBUG:boto:Headers: {} 2023-05-31T20:39:17.718 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:39:17.718 DEBUG:boto:Port: 8001 2023-05-31T20:39:17.719 DEBUG:boto:Params: {} 2023-05-31T20:39:17.720 DEBUG:boto:Token: None 2023-05-31T20:39:17.722 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:39:17 GMT /pklxkd-58/obj00000000000000000057?versionId=null 2023-05-31T20:39:17.723 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:AaBthq9wOid+/+2NJh2mAUZ8jK8= 2023-05-31T20:39:17.723 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:39:17 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:AaBthq9wOid+/+2NJh2mAUZ8jK8=', 'Content-Length': '0'} 2023-05-31T20:39:17.725 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:38:21 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009d0af6028110d0d9-006477b075-1c17-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:39:17 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:39:17.726 DEBUG:boto:path=/pklxkd-58/obj00000000000000000057 2023-05-31T20:39:17.726 DEBUG:boto:auth_path=/pklxkd-58/obj00000000000000000057 2023-05-31T20:39:17.726 DEBUG:boto:path=/pklxkd-58/obj00000000000000000057?versionId=null 2023-05-31T20:39:17.727 DEBUG:boto:auth_path=/pklxkd-58/obj00000000000000000057?versionId=null 2023-05-31T20:39:17.727 DEBUG:boto:Method: HEAD 2023-05-31T20:39:17.727 DEBUG:boto:Path: /pklxkd-58/obj00000000000000000057?versionId=null 2023-05-31T20:39:17.728 DEBUG:boto:Data: 2023-05-31T20:39:17.728 DEBUG:boto:Headers: {} 2023-05-31T20:39:17.729 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:39:17.729 DEBUG:boto:Port: 8000 2023-05-31T20:39:17.729 DEBUG:boto:Params: {} 2023-05-31T20:39:17.730 DEBUG:boto:Token: None 2023-05-31T20:39:17.730 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:39:17 GMT /pklxkd-58/obj00000000000000000057?versionId=null 2023-05-31T20:39:17.730 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:AaBthq9wOid+/+2NJh2mAUZ8jK8= 2023-05-31T20:39:17.731 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:39:17 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:AaBthq9wOid+/+2NJh2mAUZ8jK8=', 'Content-Length': '0'} 2023-05-31T20:39:17.733 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:38:21 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000096996710081760a4-006477b075-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:39:17 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:39:17.733 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000057 2023-05-31T20:39:17.734 DEBUG:boto:path=/pklxkd-58/obj00000000000000000057 2023-05-31T20:39:17.734 DEBUG:boto:auth_path=/pklxkd-58/obj00000000000000000057 2023-05-31T20:39:17.734 DEBUG:boto:path=/pklxkd-58/obj00000000000000000057?versionId=null 2023-05-31T20:39:17.735 DEBUG:boto:auth_path=/pklxkd-58/obj00000000000000000057?versionId=null 2023-05-31T20:39:17.735 DEBUG:boto:Method: GET 2023-05-31T20:39:17.735 DEBUG:boto:Path: /pklxkd-58/obj00000000000000000057?versionId=null 2023-05-31T20:39:17.736 DEBUG:boto:Data: 2023-05-31T20:39:17.736 DEBUG:boto:Headers: {} 2023-05-31T20:39:17.737 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:39:17.737 DEBUG:boto:Port: 8001 2023-05-31T20:39:17.737 DEBUG:boto:Params: {} 2023-05-31T20:39:17.738 DEBUG:boto:Token: None 2023-05-31T20:39:17.738 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:39:17 GMT /pklxkd-58/obj00000000000000000057?versionId=null 2023-05-31T20:39:17.739 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:Xefs+zE0rD22dvqHAv56SWc0wvo= 2023-05-31T20:39:17.739 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:39:17 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:Xefs+zE0rD22dvqHAv56SWc0wvo=', 'Content-Length': '0'} 2023-05-31T20:39:17.741 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:38:21 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001c4000536aab3143-006477b075-1c17-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:39:17 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:39:17.784 DEBUG:boto:path=/pklxkd-58/obj00000000000000000057 2023-05-31T20:39:17.785 DEBUG:boto:auth_path=/pklxkd-58/obj00000000000000000057 2023-05-31T20:39:17.785 DEBUG:boto:path=/pklxkd-58/obj00000000000000000057?versionId=null 2023-05-31T20:39:17.786 DEBUG:boto:auth_path=/pklxkd-58/obj00000000000000000057?versionId=null 2023-05-31T20:39:17.786 DEBUG:boto:Method: GET 2023-05-31T20:39:17.786 DEBUG:boto:Path: /pklxkd-58/obj00000000000000000057?versionId=null 2023-05-31T20:39:17.787 DEBUG:boto:Data: 2023-05-31T20:39:17.787 DEBUG:boto:Headers: {} 2023-05-31T20:39:17.788 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:39:17.788 DEBUG:boto:Port: 8000 2023-05-31T20:39:17.788 DEBUG:boto:Params: {} 2023-05-31T20:39:17.789 DEBUG:boto:Token: None 2023-05-31T20:39:17.789 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:39:17 GMT /pklxkd-58/obj00000000000000000057?versionId=null 2023-05-31T20:39:17.790 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:Xefs+zE0rD22dvqHAv56SWc0wvo= 2023-05-31T20:39:17.790 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:39:17 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:Xefs+zE0rD22dvqHAv56SWc0wvo=', 'Content-Length': '0'} 2023-05-31T20:39:17.792 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:38:21 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000014faf243fa832ad3-006477b075-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:39:17 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:39:17.793 DEBUG:boto:path=/pklxkd-58/ 2023-05-31T20:39:17.793 DEBUG:boto:auth_path=/pklxkd-58/ 2023-05-31T20:39:17.794 DEBUG:boto:path=/pklxkd-58/?prefix=obj00000000000000000057 2023-05-31T20:39:17.794 DEBUG:boto:auth_path=/pklxkd-58/?prefix=obj00000000000000000057 2023-05-31T20:39:17.795 DEBUG:boto:Method: GET 2023-05-31T20:39:17.795 DEBUG:boto:Path: /pklxkd-58/?prefix=obj00000000000000000057 2023-05-31T20:39:17.796 DEBUG:boto:Data: 2023-05-31T20:39:17.796 DEBUG:boto:Headers: {} 2023-05-31T20:39:17.797 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:39:17.797 DEBUG:boto:Port: 8001 2023-05-31T20:39:17.797 DEBUG:boto:Params: {} 2023-05-31T20:39:17.798 DEBUG:boto:Token: None 2023-05-31T20:39:17.798 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:39:17 GMT /pklxkd-58/ 2023-05-31T20:39:17.799 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:kahcpIxmAQ2hT4M2tN+pL/3o20Q= 2023-05-31T20:39:17.799 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:39:17 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:kahcpIxmAQ2hT4M2tN+pL/3o20Q=', 'Content-Length': '0'} 2023-05-31T20:39:17.802 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000009a9448f790362824-006477b075-1c17-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:39:17 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:39:17.802 DEBUG:boto:b'pklxkd-58obj000000000000000000571000falseobj000000000000000000572023-05-31T20:38:21.818Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:39:17.803 DEBUG:boto:path=/pklxkd-58/ 2023-05-31T20:39:17.803 DEBUG:boto:auth_path=/pklxkd-58/ 2023-05-31T20:39:17.804 DEBUG:boto:path=/pklxkd-58/?prefix=obj00000000000000000057 2023-05-31T20:39:17.804 DEBUG:boto:auth_path=/pklxkd-58/?prefix=obj00000000000000000057 2023-05-31T20:39:17.805 DEBUG:boto:Method: GET 2023-05-31T20:39:17.806 DEBUG:boto:Path: /pklxkd-58/?prefix=obj00000000000000000057 2023-05-31T20:39:17.806 DEBUG:boto:Data: 2023-05-31T20:39:17.806 DEBUG:boto:Headers: {} 2023-05-31T20:39:17.807 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:39:17.807 DEBUG:boto:Port: 8000 2023-05-31T20:39:17.807 DEBUG:boto:Params: {} 2023-05-31T20:39:17.808 DEBUG:boto:Token: None 2023-05-31T20:39:17.808 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:39:17 GMT /pklxkd-58/ 2023-05-31T20:39:17.809 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:kahcpIxmAQ2hT4M2tN+pL/3o20Q= 2023-05-31T20:39:17.810 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:39:17 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:kahcpIxmAQ2hT4M2tN+pL/3o20Q=', 'Content-Length': '0'} 2023-05-31T20:39:17.815 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000691765e659e9da85-006477b075-1630-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:39:17 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:39:17.816 DEBUG:boto:b'pklxkd-58obj000000000000000000571000falseobj000000000000000000572023-05-31T20:38:21.818Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:39:17.817 DEBUG:boto:path=/pklxkd-58/obj00000000000000000057 2023-05-31T20:39:17.818 DEBUG:boto:auth_path=/pklxkd-58/obj00000000000000000057 2023-05-31T20:39:17.819 DEBUG:boto:Method: HEAD 2023-05-31T20:39:17.819 DEBUG:boto:Path: /pklxkd-58/obj00000000000000000057 2023-05-31T20:39:17.820 DEBUG:boto:Data: 2023-05-31T20:39:17.821 DEBUG:boto:Headers: {} 2023-05-31T20:39:17.821 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:39:17.822 DEBUG:boto:Port: 8001 2023-05-31T20:39:17.822 DEBUG:boto:Params: {} 2023-05-31T20:39:17.822 DEBUG:boto:Token: None 2023-05-31T20:39:17.823 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:39:17 GMT /pklxkd-58/obj00000000000000000057 2023-05-31T20:39:17.823 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:HQpWJRrA2LT8EOMo8VDWSGpzpbg= 2023-05-31T20:39:17.824 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:39:17 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:HQpWJRrA2LT8EOMo8VDWSGpzpbg=', 'Content-Length': '0'} 2023-05-31T20:39:17.826 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:38:21 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000fca7072180f20979-006477b075-1c17-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:39:17 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:39:17.826 DEBUG:boto:path=/pklxkd-58/obj00000000000000000057 2023-05-31T20:39:17.827 DEBUG:boto:auth_path=/pklxkd-58/obj00000000000000000057 2023-05-31T20:39:17.827 DEBUG:boto:Method: HEAD 2023-05-31T20:39:17.827 DEBUG:boto:Path: /pklxkd-58/obj00000000000000000057 2023-05-31T20:39:17.828 DEBUG:boto:Data: 2023-05-31T20:39:17.828 DEBUG:boto:Headers: {} 2023-05-31T20:39:17.828 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:39:17.829 DEBUG:boto:Port: 8000 2023-05-31T20:39:17.829 DEBUG:boto:Params: {} 2023-05-31T20:39:17.830 DEBUG:boto:Token: None 2023-05-31T20:39:17.830 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:39:17 GMT /pklxkd-58/obj00000000000000000057 2023-05-31T20:39:17.831 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:HQpWJRrA2LT8EOMo8VDWSGpzpbg= 2023-05-31T20:39:17.831 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:39:17 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:HQpWJRrA2LT8EOMo8VDWSGpzpbg=', 'Content-Length': '0'} 2023-05-31T20:39:17.833 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:38:21 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c7f745f0564764f9-006477b075-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:39:17 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:39:17.834 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000057 2023-05-31T20:39:17.834 DEBUG:boto:path=/pklxkd-58/obj00000000000000000057 2023-05-31T20:39:17.835 DEBUG:boto:auth_path=/pklxkd-58/obj00000000000000000057 2023-05-31T20:39:17.835 DEBUG:boto:Method: GET 2023-05-31T20:39:17.835 DEBUG:boto:Path: /pklxkd-58/obj00000000000000000057 2023-05-31T20:39:17.836 DEBUG:boto:Data: 2023-05-31T20:39:17.836 DEBUG:boto:Headers: {} 2023-05-31T20:39:17.836 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:39:17.837 DEBUG:boto:Port: 8001 2023-05-31T20:39:17.837 DEBUG:boto:Params: {} 2023-05-31T20:39:17.838 DEBUG:boto:Token: None 2023-05-31T20:39:17.838 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:39:17 GMT /pklxkd-58/obj00000000000000000057 2023-05-31T20:39:17.838 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:NLZxydMewJVqxaOv+22am601r3Q= 2023-05-31T20:39:17.839 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:39:17 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:NLZxydMewJVqxaOv+22am601r3Q=', 'Content-Length': '0'} 2023-05-31T20:39:17.841 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:38:21 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006a1dff91867509a6-006477b075-1c17-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:39:17 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:39:17.842 DEBUG:boto:path=/pklxkd-58/obj00000000000000000057 2023-05-31T20:39:17.842 DEBUG:boto:auth_path=/pklxkd-58/obj00000000000000000057 2023-05-31T20:39:17.843 DEBUG:boto:Method: GET 2023-05-31T20:39:17.843 DEBUG:boto:Path: /pklxkd-58/obj00000000000000000057 2023-05-31T20:39:17.843 DEBUG:boto:Data: 2023-05-31T20:39:17.844 DEBUG:boto:Headers: {} 2023-05-31T20:39:17.844 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:39:17.844 DEBUG:boto:Port: 8000 2023-05-31T20:39:17.845 DEBUG:boto:Params: {} 2023-05-31T20:39:17.845 DEBUG:boto:Token: None 2023-05-31T20:39:17.846 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:39:17 GMT /pklxkd-58/obj00000000000000000057 2023-05-31T20:39:17.846 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:NLZxydMewJVqxaOv+22am601r3Q= 2023-05-31T20:39:17.846 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:39:17 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:NLZxydMewJVqxaOv+22am601r3Q=', 'Content-Length': '0'} 2023-05-31T20:39:17.848 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:38:21 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000038dc7a6f4263305a-006477b075-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:39:17 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:39:17.892 DEBUG:boto:path=/pklxkd-58/ 2023-05-31T20:39:17.893 DEBUG:boto:auth_path=/pklxkd-58/ 2023-05-31T20:39:17.894 DEBUG:boto:path=/pklxkd-58/?prefix=obj00000000000000000057 2023-05-31T20:39:17.894 DEBUG:boto:auth_path=/pklxkd-58/?prefix=obj00000000000000000057 2023-05-31T20:39:17.894 DEBUG:boto:Method: GET 2023-05-31T20:39:17.895 DEBUG:boto:Path: /pklxkd-58/?prefix=obj00000000000000000057 2023-05-31T20:39:17.895 DEBUG:boto:Data: 2023-05-31T20:39:17.896 DEBUG:boto:Headers: {} 2023-05-31T20:39:17.897 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:39:17.897 DEBUG:boto:Port: 8001 2023-05-31T20:39:17.898 DEBUG:boto:Params: {} 2023-05-31T20:39:17.898 DEBUG:boto:Token: None 2023-05-31T20:39:17.898 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:39:17 GMT /pklxkd-58/ 2023-05-31T20:39:17.899 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:kahcpIxmAQ2hT4M2tN+pL/3o20Q= 2023-05-31T20:39:17.899 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:39:17 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:kahcpIxmAQ2hT4M2tN+pL/3o20Q=', 'Content-Length': '0'} 2023-05-31T20:39:17.902 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000000a3a7d31e2f91298-006477b075-1c17-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:39:17 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:39:17.902 DEBUG:boto:b'pklxkd-58obj000000000000000000571000falseobj000000000000000000572023-05-31T20:38:21.818Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:39:17.903 DEBUG:boto:path=/pklxkd-58/ 2023-05-31T20:39:17.903 DEBUG:boto:auth_path=/pklxkd-58/ 2023-05-31T20:39:17.904 DEBUG:boto:path=/pklxkd-58/?prefix=obj00000000000000000057 2023-05-31T20:39:17.904 DEBUG:boto:auth_path=/pklxkd-58/?prefix=obj00000000000000000057 2023-05-31T20:39:17.905 DEBUG:boto:Method: GET 2023-05-31T20:39:17.905 DEBUG:boto:Path: /pklxkd-58/?prefix=obj00000000000000000057 2023-05-31T20:39:17.905 DEBUG:boto:Data: 2023-05-31T20:39:17.906 DEBUG:boto:Headers: {} 2023-05-31T20:39:17.906 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:39:17.907 DEBUG:boto:Port: 8000 2023-05-31T20:39:17.907 DEBUG:boto:Params: {} 2023-05-31T20:39:17.908 DEBUG:boto:Token: None 2023-05-31T20:39:17.908 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:39:17 GMT /pklxkd-58/ 2023-05-31T20:39:17.909 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:kahcpIxmAQ2hT4M2tN+pL/3o20Q= 2023-05-31T20:39:17.909 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:39:17 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:kahcpIxmAQ2hT4M2tN+pL/3o20Q=', 'Content-Length': '0'} 2023-05-31T20:39:17.913 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d9bdca7e8f7eb3e6-006477b075-1630-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:39:17 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:39:17.914 DEBUG:boto:b'pklxkd-58obj000000000000000000571000falseobj000000000000000000572023-05-31T20:38:21.818Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:39:17.914 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000058 2023-05-31T20:39:17.915 DEBUG:boto:path=/pklxkd-58/obj00000000000000000058 2023-05-31T20:39:17.915 DEBUG:boto:auth_path=/pklxkd-58/obj00000000000000000058 2023-05-31T20:39:17.916 DEBUG:boto:path=/pklxkd-58/obj00000000000000000058?versionId=null 2023-05-31T20:39:17.916 DEBUG:boto:auth_path=/pklxkd-58/obj00000000000000000058?versionId=null 2023-05-31T20:39:17.916 DEBUG:boto:Method: GET 2023-05-31T20:39:17.917 DEBUG:boto:Path: /pklxkd-58/obj00000000000000000058?versionId=null 2023-05-31T20:39:17.917 DEBUG:boto:Data: 2023-05-31T20:39:17.917 DEBUG:boto:Headers: {} 2023-05-31T20:39:17.918 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:39:17.918 DEBUG:boto:Port: 8001 2023-05-31T20:39:17.918 DEBUG:boto:Params: {} 2023-05-31T20:39:17.919 DEBUG:boto:Token: None 2023-05-31T20:39:17.919 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:39:17 GMT /pklxkd-58/obj00000000000000000058?versionId=null 2023-05-31T20:39:17.920 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:HLNMuU9Z40alcCusyCe5ruwnYe8= 2023-05-31T20:39:17.920 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:39:17 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:HLNMuU9Z40alcCusyCe5ruwnYe8=', 'Content-Length': '0'} 2023-05-31T20:39:17.922 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:38:17 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000301f9c9d1648510a-006477b075-1c17-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:39:17 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:39:17.923 DEBUG:boto:path=/pklxkd-58/obj00000000000000000058 2023-05-31T20:39:17.924 DEBUG:boto:auth_path=/pklxkd-58/obj00000000000000000058 2023-05-31T20:39:17.924 DEBUG:boto:path=/pklxkd-58/obj00000000000000000058?versionId=null 2023-05-31T20:39:17.924 DEBUG:boto:auth_path=/pklxkd-58/obj00000000000000000058?versionId=null 2023-05-31T20:39:17.925 DEBUG:boto:Method: GET 2023-05-31T20:39:17.925 DEBUG:boto:Path: /pklxkd-58/obj00000000000000000058?versionId=null 2023-05-31T20:39:17.926 DEBUG:boto:Data: 2023-05-31T20:39:17.926 DEBUG:boto:Headers: {} 2023-05-31T20:39:17.926 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:39:17.927 DEBUG:boto:Port: 8000 2023-05-31T20:39:17.927 DEBUG:boto:Params: {} 2023-05-31T20:39:17.928 DEBUG:boto:Token: None 2023-05-31T20:39:17.929 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:39:17 GMT /pklxkd-58/obj00000000000000000058?versionId=null 2023-05-31T20:39:17.929 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:HLNMuU9Z40alcCusyCe5ruwnYe8= 2023-05-31T20:39:17.930 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:39:17 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:HLNMuU9Z40alcCusyCe5ruwnYe8=', 'Content-Length': '0'} 2023-05-31T20:39:17.932 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:38:17 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a8cf1433c334ebba-006477b075-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:39:17 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:39:17.932 DEBUG:boto:path=/pklxkd-58/obj00000000000000000058 2023-05-31T20:39:17.933 DEBUG:boto:auth_path=/pklxkd-58/obj00000000000000000058 2023-05-31T20:39:17.933 DEBUG:boto:path=/pklxkd-58/obj00000000000000000058?versionId=null 2023-05-31T20:39:17.933 DEBUG:boto:auth_path=/pklxkd-58/obj00000000000000000058?versionId=null 2023-05-31T20:39:17.934 DEBUG:boto:Method: HEAD 2023-05-31T20:39:17.934 DEBUG:boto:Path: /pklxkd-58/obj00000000000000000058?versionId=null 2023-05-31T20:39:17.934 DEBUG:boto:Data: 2023-05-31T20:39:17.935 DEBUG:boto:Headers: {} 2023-05-31T20:39:17.935 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:39:17.935 DEBUG:boto:Port: 8001 2023-05-31T20:39:17.936 DEBUG:boto:Params: {} 2023-05-31T20:39:17.936 DEBUG:boto:Token: None 2023-05-31T20:39:17.937 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:39:17 GMT /pklxkd-58/obj00000000000000000058?versionId=null 2023-05-31T20:39:17.937 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:efHTL5TcBUC3P45JEPGTiRUbATU= 2023-05-31T20:39:17.937 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:39:17 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:efHTL5TcBUC3P45JEPGTiRUbATU=', 'Content-Length': '0'} 2023-05-31T20:39:17.940 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:38:17 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b2bf18f075030b8e-006477b075-1c17-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:39:17 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:39:17.941 DEBUG:boto:path=/pklxkd-58/obj00000000000000000058 2023-05-31T20:39:17.941 DEBUG:boto:auth_path=/pklxkd-58/obj00000000000000000058 2023-05-31T20:39:17.941 DEBUG:boto:path=/pklxkd-58/obj00000000000000000058?versionId=null 2023-05-31T20:39:17.942 DEBUG:boto:auth_path=/pklxkd-58/obj00000000000000000058?versionId=null 2023-05-31T20:39:17.942 DEBUG:boto:Method: HEAD 2023-05-31T20:39:17.942 DEBUG:boto:Path: /pklxkd-58/obj00000000000000000058?versionId=null 2023-05-31T20:39:17.943 DEBUG:boto:Data: 2023-05-31T20:39:17.943 DEBUG:boto:Headers: {} 2023-05-31T20:39:17.943 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:39:17.944 DEBUG:boto:Port: 8000 2023-05-31T20:39:17.944 DEBUG:boto:Params: {} 2023-05-31T20:39:17.945 DEBUG:boto:Token: None 2023-05-31T20:39:17.945 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:39:17 GMT /pklxkd-58/obj00000000000000000058?versionId=null 2023-05-31T20:39:17.946 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:efHTL5TcBUC3P45JEPGTiRUbATU= 2023-05-31T20:39:17.946 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:39:17 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:efHTL5TcBUC3P45JEPGTiRUbATU=', 'Content-Length': '0'} 2023-05-31T20:39:17.948 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:38:17 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005926d759256e6e6a-006477b075-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:39:17 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:39:17.949 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000058 2023-05-31T20:39:17.950 DEBUG:boto:path=/pklxkd-58/obj00000000000000000058 2023-05-31T20:39:17.950 DEBUG:boto:auth_path=/pklxkd-58/obj00000000000000000058 2023-05-31T20:39:17.950 DEBUG:boto:path=/pklxkd-58/obj00000000000000000058?versionId=null 2023-05-31T20:39:17.951 DEBUG:boto:auth_path=/pklxkd-58/obj00000000000000000058?versionId=null 2023-05-31T20:39:17.951 DEBUG:boto:Method: GET 2023-05-31T20:39:17.951 DEBUG:boto:Path: /pklxkd-58/obj00000000000000000058?versionId=null 2023-05-31T20:39:17.952 DEBUG:boto:Data: 2023-05-31T20:39:17.952 DEBUG:boto:Headers: {} 2023-05-31T20:39:17.952 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:39:17.953 DEBUG:boto:Port: 8001 2023-05-31T20:39:17.953 DEBUG:boto:Params: {} 2023-05-31T20:39:17.954 DEBUG:boto:Token: None 2023-05-31T20:39:17.954 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:39:17 GMT /pklxkd-58/obj00000000000000000058?versionId=null 2023-05-31T20:39:17.954 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:HLNMuU9Z40alcCusyCe5ruwnYe8= 2023-05-31T20:39:17.955 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:39:17 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:HLNMuU9Z40alcCusyCe5ruwnYe8=', 'Content-Length': '0'} 2023-05-31T20:39:17.958 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:38:17 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000011f093384ae0cbe2-006477b075-1c17-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:39:17 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:39:17.959 DEBUG:boto:path=/pklxkd-58/obj00000000000000000058 2023-05-31T20:39:17.959 DEBUG:boto:auth_path=/pklxkd-58/obj00000000000000000058 2023-05-31T20:39:17.959 DEBUG:boto:path=/pklxkd-58/obj00000000000000000058?versionId=null 2023-05-31T20:39:17.960 DEBUG:boto:auth_path=/pklxkd-58/obj00000000000000000058?versionId=null 2023-05-31T20:39:17.960 DEBUG:boto:Method: GET 2023-05-31T20:39:17.960 DEBUG:boto:Path: /pklxkd-58/obj00000000000000000058?versionId=null 2023-05-31T20:39:17.961 DEBUG:boto:Data: 2023-05-31T20:39:17.961 DEBUG:boto:Headers: {} 2023-05-31T20:39:17.962 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:39:17.962 DEBUG:boto:Port: 8000 2023-05-31T20:39:17.962 DEBUG:boto:Params: {} 2023-05-31T20:39:17.963 DEBUG:boto:Token: None 2023-05-31T20:39:17.963 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:39:17 GMT /pklxkd-58/obj00000000000000000058?versionId=null 2023-05-31T20:39:17.964 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:HLNMuU9Z40alcCusyCe5ruwnYe8= 2023-05-31T20:39:17.964 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:39:17 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:HLNMuU9Z40alcCusyCe5ruwnYe8=', 'Content-Length': '0'} 2023-05-31T20:39:17.966 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:38:17 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006f71a0b75c181619-006477b075-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:39:17 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:39:17.967 DEBUG:boto:path=/pklxkd-58/ 2023-05-31T20:39:17.967 DEBUG:boto:auth_path=/pklxkd-58/ 2023-05-31T20:39:17.968 DEBUG:boto:path=/pklxkd-58/?prefix=obj00000000000000000058 2023-05-31T20:39:17.968 DEBUG:boto:auth_path=/pklxkd-58/?prefix=obj00000000000000000058 2023-05-31T20:39:17.968 DEBUG:boto:Method: GET 2023-05-31T20:39:17.969 DEBUG:boto:Path: /pklxkd-58/?prefix=obj00000000000000000058 2023-05-31T20:39:17.969 DEBUG:boto:Data: 2023-05-31T20:39:17.969 DEBUG:boto:Headers: {} 2023-05-31T20:39:17.970 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:39:17.970 DEBUG:boto:Port: 8001 2023-05-31T20:39:17.970 DEBUG:boto:Params: {} 2023-05-31T20:39:17.971 DEBUG:boto:Token: None 2023-05-31T20:39:17.971 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:39:17 GMT /pklxkd-58/ 2023-05-31T20:39:17.972 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:kahcpIxmAQ2hT4M2tN+pL/3o20Q= 2023-05-31T20:39:17.972 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:39:17 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:kahcpIxmAQ2hT4M2tN+pL/3o20Q=', 'Content-Length': '0'} 2023-05-31T20:39:17.975 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000de8b1eef050eb2d8-006477b075-1c17-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:39:17 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:39:18.016 DEBUG:boto:b'pklxkd-58obj000000000000000000581000falseobj000000000000000000582023-05-31T20:38:17.631Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:39:18.017 DEBUG:boto:path=/pklxkd-58/ 2023-05-31T20:39:18.018 DEBUG:boto:auth_path=/pklxkd-58/ 2023-05-31T20:39:18.018 DEBUG:boto:path=/pklxkd-58/?prefix=obj00000000000000000058 2023-05-31T20:39:18.018 DEBUG:boto:auth_path=/pklxkd-58/?prefix=obj00000000000000000058 2023-05-31T20:39:18.019 DEBUG:boto:Method: GET 2023-05-31T20:39:18.019 DEBUG:boto:Path: /pklxkd-58/?prefix=obj00000000000000000058 2023-05-31T20:39:18.020 DEBUG:boto:Data: 2023-05-31T20:39:18.020 DEBUG:boto:Headers: {} 2023-05-31T20:39:18.020 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:39:18.021 DEBUG:boto:Port: 8000 2023-05-31T20:39:18.021 DEBUG:boto:Params: {} 2023-05-31T20:39:18.021 DEBUG:boto:Token: None 2023-05-31T20:39:18.022 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:39:18 GMT /pklxkd-58/ 2023-05-31T20:39:18.022 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:3DG8wpCqcsv/yPsAzk0+okqkXZU= 2023-05-31T20:39:18.024 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:39:18 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:3DG8wpCqcsv/yPsAzk0+okqkXZU=', 'Content-Length': '0'} 2023-05-31T20:39:18.028 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d7e116a315a9620a-006477b076-1630-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:39:18 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:39:18.029 DEBUG:boto:b'pklxkd-58obj000000000000000000581000falseobj000000000000000000582023-05-31T20:38:17.631Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:39:18.030 DEBUG:boto:path=/pklxkd-58/obj00000000000000000058 2023-05-31T20:39:18.030 DEBUG:boto:auth_path=/pklxkd-58/obj00000000000000000058 2023-05-31T20:39:18.031 DEBUG:boto:Method: HEAD 2023-05-31T20:39:18.031 DEBUG:boto:Path: /pklxkd-58/obj00000000000000000058 2023-05-31T20:39:18.031 DEBUG:boto:Data: 2023-05-31T20:39:18.032 DEBUG:boto:Headers: {} 2023-05-31T20:39:18.032 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:39:18.033 DEBUG:boto:Port: 8001 2023-05-31T20:39:18.033 DEBUG:boto:Params: {} 2023-05-31T20:39:18.033 DEBUG:boto:Token: None 2023-05-31T20:39:18.034 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:39:18 GMT /pklxkd-58/obj00000000000000000058 2023-05-31T20:39:18.034 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:9m2MUsgfXc3wS2wSck67WPQDoQ4= 2023-05-31T20:39:18.035 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:39:18 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:9m2MUsgfXc3wS2wSck67WPQDoQ4=', 'Content-Length': '0'} 2023-05-31T20:39:18.037 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:38:17 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a2a2a1093b6cdb9c-006477b076-1c17-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:39:18 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:39:18.038 DEBUG:boto:path=/pklxkd-58/obj00000000000000000058 2023-05-31T20:39:18.038 DEBUG:boto:auth_path=/pklxkd-58/obj00000000000000000058 2023-05-31T20:39:18.039 DEBUG:boto:Method: HEAD 2023-05-31T20:39:18.039 DEBUG:boto:Path: /pklxkd-58/obj00000000000000000058 2023-05-31T20:39:18.039 DEBUG:boto:Data: 2023-05-31T20:39:18.040 DEBUG:boto:Headers: {} 2023-05-31T20:39:18.041 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:39:18.041 DEBUG:boto:Port: 8000 2023-05-31T20:39:18.041 DEBUG:boto:Params: {} 2023-05-31T20:39:18.042 DEBUG:boto:Token: None 2023-05-31T20:39:18.042 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:39:18 GMT /pklxkd-58/obj00000000000000000058 2023-05-31T20:39:18.043 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:9m2MUsgfXc3wS2wSck67WPQDoQ4= 2023-05-31T20:39:18.043 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:39:18 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:9m2MUsgfXc3wS2wSck67WPQDoQ4=', 'Content-Length': '0'} 2023-05-31T20:39:18.045 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:38:17 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000015421fd5fb680d50-006477b076-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:39:18 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:39:18.045 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000058 2023-05-31T20:39:18.046 DEBUG:boto:path=/pklxkd-58/obj00000000000000000058 2023-05-31T20:39:18.046 DEBUG:boto:auth_path=/pklxkd-58/obj00000000000000000058 2023-05-31T20:39:18.047 DEBUG:boto:Method: GET 2023-05-31T20:39:18.047 DEBUG:boto:Path: /pklxkd-58/obj00000000000000000058 2023-05-31T20:39:18.047 DEBUG:boto:Data: 2023-05-31T20:39:18.048 DEBUG:boto:Headers: {} 2023-05-31T20:39:18.048 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:39:18.049 DEBUG:boto:Port: 8001 2023-05-31T20:39:18.049 DEBUG:boto:Params: {} 2023-05-31T20:39:18.049 DEBUG:boto:Token: None 2023-05-31T20:39:18.050 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:39:18 GMT /pklxkd-58/obj00000000000000000058 2023-05-31T20:39:18.050 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:S0QBPl24IGc+1MxoRcGeS3pAjA0= 2023-05-31T20:39:18.051 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:39:18 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:S0QBPl24IGc+1MxoRcGeS3pAjA0=', 'Content-Length': '0'} 2023-05-31T20:39:18.053 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:38:17 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b7230d68600122bf-006477b076-1c17-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:39:18 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:39:18.054 DEBUG:boto:path=/pklxkd-58/obj00000000000000000058 2023-05-31T20:39:18.054 DEBUG:boto:auth_path=/pklxkd-58/obj00000000000000000058 2023-05-31T20:39:18.055 DEBUG:boto:Method: GET 2023-05-31T20:39:18.056 DEBUG:boto:Path: /pklxkd-58/obj00000000000000000058 2023-05-31T20:39:18.057 DEBUG:boto:Data: 2023-05-31T20:39:18.058 DEBUG:boto:Headers: {} 2023-05-31T20:39:18.059 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:39:18.059 DEBUG:boto:Port: 8000 2023-05-31T20:39:18.060 DEBUG:boto:Params: {} 2023-05-31T20:39:18.061 DEBUG:boto:Token: None 2023-05-31T20:39:18.061 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:39:18 GMT /pklxkd-58/obj00000000000000000058 2023-05-31T20:39:18.062 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:S0QBPl24IGc+1MxoRcGeS3pAjA0= 2023-05-31T20:39:18.063 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:39:18 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:S0QBPl24IGc+1MxoRcGeS3pAjA0=', 'Content-Length': '0'} 2023-05-31T20:39:18.065 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:38:17 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ffbbb3d9f94e6377-006477b076-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:39:18 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:39:18.113 DEBUG:boto:path=/pklxkd-58/ 2023-05-31T20:39:18.113 DEBUG:boto:auth_path=/pklxkd-58/ 2023-05-31T20:39:18.114 DEBUG:boto:path=/pklxkd-58/?prefix=obj00000000000000000058 2023-05-31T20:39:18.114 DEBUG:boto:auth_path=/pklxkd-58/?prefix=obj00000000000000000058 2023-05-31T20:39:18.114 DEBUG:boto:Method: GET 2023-05-31T20:39:18.115 DEBUG:boto:Path: /pklxkd-58/?prefix=obj00000000000000000058 2023-05-31T20:39:18.115 DEBUG:boto:Data: 2023-05-31T20:39:18.115 DEBUG:boto:Headers: {} 2023-05-31T20:39:18.116 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:39:18.116 DEBUG:boto:Port: 8001 2023-05-31T20:39:18.117 DEBUG:boto:Params: {} 2023-05-31T20:39:18.117 DEBUG:boto:Token: None 2023-05-31T20:39:18.117 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:39:18 GMT /pklxkd-58/ 2023-05-31T20:39:18.118 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:3DG8wpCqcsv/yPsAzk0+okqkXZU= 2023-05-31T20:39:18.118 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:39:18 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:3DG8wpCqcsv/yPsAzk0+okqkXZU=', 'Content-Length': '0'} 2023-05-31T20:39:18.121 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000090066466e73c1a0c-006477b076-1c17-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:39:18 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:39:18.121 DEBUG:boto:b'pklxkd-58obj000000000000000000581000falseobj000000000000000000582023-05-31T20:38:17.631Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:39:18.123 DEBUG:boto:path=/pklxkd-58/ 2023-05-31T20:39:18.124 DEBUG:boto:auth_path=/pklxkd-58/ 2023-05-31T20:39:18.124 DEBUG:boto:path=/pklxkd-58/?prefix=obj00000000000000000058 2023-05-31T20:39:18.125 DEBUG:boto:auth_path=/pklxkd-58/?prefix=obj00000000000000000058 2023-05-31T20:39:18.125 DEBUG:boto:Method: GET 2023-05-31T20:39:18.126 DEBUG:boto:Path: /pklxkd-58/?prefix=obj00000000000000000058 2023-05-31T20:39:18.126 DEBUG:boto:Data: 2023-05-31T20:39:18.126 DEBUG:boto:Headers: {} 2023-05-31T20:39:18.127 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:39:18.127 DEBUG:boto:Port: 8000 2023-05-31T20:39:18.127 DEBUG:boto:Params: {} 2023-05-31T20:39:18.128 DEBUG:boto:Token: None 2023-05-31T20:39:18.128 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:39:18 GMT /pklxkd-58/ 2023-05-31T20:39:18.129 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:3DG8wpCqcsv/yPsAzk0+okqkXZU= 2023-05-31T20:39:18.129 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:39:18 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:3DG8wpCqcsv/yPsAzk0+okqkXZU=', 'Content-Length': '0'} 2023-05-31T20:39:18.134 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e028a5f698b0d4d1-006477b076-1630-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:39:18 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:39:18.134 DEBUG:boto:b'pklxkd-58obj000000000000000000581000falseobj000000000000000000582023-05-31T20:38:17.631Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:39:18.135 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000059 2023-05-31T20:39:18.135 DEBUG:boto:path=/pklxkd-58/obj00000000000000000059 2023-05-31T20:39:18.136 DEBUG:boto:auth_path=/pklxkd-58/obj00000000000000000059 2023-05-31T20:39:18.136 DEBUG:boto:path=/pklxkd-58/obj00000000000000000059?versionId=null 2023-05-31T20:39:18.137 DEBUG:boto:auth_path=/pklxkd-58/obj00000000000000000059?versionId=null 2023-05-31T20:39:18.137 DEBUG:boto:Method: GET 2023-05-31T20:39:18.138 DEBUG:boto:Path: /pklxkd-58/obj00000000000000000059?versionId=null 2023-05-31T20:39:18.138 DEBUG:boto:Data: 2023-05-31T20:39:18.138 DEBUG:boto:Headers: {} 2023-05-31T20:39:18.139 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:39:18.140 DEBUG:boto:Port: 8001 2023-05-31T20:39:18.140 DEBUG:boto:Params: {} 2023-05-31T20:39:18.141 DEBUG:boto:Token: None 2023-05-31T20:39:18.142 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:39:18 GMT /pklxkd-58/obj00000000000000000059?versionId=null 2023-05-31T20:39:18.143 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:gYztoXat1obpTZGscJjEZN+5gNA= 2023-05-31T20:39:18.144 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:39:18 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:gYztoXat1obpTZGscJjEZN+5gNA=', 'Content-Length': '0'} 2023-05-31T20:39:18.147 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:38:18 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002cebe35cb188bf9c-006477b076-1c17-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:39:18 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:39:18.188 DEBUG:boto:path=/pklxkd-58/obj00000000000000000059 2023-05-31T20:39:18.190 DEBUG:boto:auth_path=/pklxkd-58/obj00000000000000000059 2023-05-31T20:39:18.190 DEBUG:boto:path=/pklxkd-58/obj00000000000000000059?versionId=null 2023-05-31T20:39:18.190 DEBUG:boto:auth_path=/pklxkd-58/obj00000000000000000059?versionId=null 2023-05-31T20:39:18.191 DEBUG:boto:Method: GET 2023-05-31T20:39:18.191 DEBUG:boto:Path: /pklxkd-58/obj00000000000000000059?versionId=null 2023-05-31T20:39:18.191 DEBUG:boto:Data: 2023-05-31T20:39:18.192 DEBUG:boto:Headers: {} 2023-05-31T20:39:18.192 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:39:18.193 DEBUG:boto:Port: 8000 2023-05-31T20:39:18.193 DEBUG:boto:Params: {} 2023-05-31T20:39:18.193 DEBUG:boto:Token: None 2023-05-31T20:39:18.194 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:39:18 GMT /pklxkd-58/obj00000000000000000059?versionId=null 2023-05-31T20:39:18.194 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:gYztoXat1obpTZGscJjEZN+5gNA= 2023-05-31T20:39:18.195 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:39:18 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:gYztoXat1obpTZGscJjEZN+5gNA=', 'Content-Length': '0'} 2023-05-31T20:39:18.197 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:38:18 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000401a73fa10ffbc9d-006477b076-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:39:18 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:39:18.198 DEBUG:boto:path=/pklxkd-58/obj00000000000000000059 2023-05-31T20:39:18.198 DEBUG:boto:auth_path=/pklxkd-58/obj00000000000000000059 2023-05-31T20:39:18.199 DEBUG:boto:path=/pklxkd-58/obj00000000000000000059?versionId=null 2023-05-31T20:39:18.199 DEBUG:boto:auth_path=/pklxkd-58/obj00000000000000000059?versionId=null 2023-05-31T20:39:18.199 DEBUG:boto:Method: HEAD 2023-05-31T20:39:18.200 DEBUG:boto:Path: /pklxkd-58/obj00000000000000000059?versionId=null 2023-05-31T20:39:18.200 DEBUG:boto:Data: 2023-05-31T20:39:18.200 DEBUG:boto:Headers: {} 2023-05-31T20:39:18.201 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:39:18.201 DEBUG:boto:Port: 8001 2023-05-31T20:39:18.202 DEBUG:boto:Params: {} 2023-05-31T20:39:18.203 DEBUG:boto:Token: None 2023-05-31T20:39:18.204 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:39:18 GMT /pklxkd-58/obj00000000000000000059?versionId=null 2023-05-31T20:39:18.205 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:laZlgvQR1eNtdfHOXgHpf127KXk= 2023-05-31T20:39:18.206 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:39:18 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:laZlgvQR1eNtdfHOXgHpf127KXk=', 'Content-Length': '0'} 2023-05-31T20:39:18.209 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:38:18 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007d5869497970d48f-006477b076-1c17-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:39:18 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:39:18.210 DEBUG:boto:path=/pklxkd-58/obj00000000000000000059 2023-05-31T20:39:18.210 DEBUG:boto:auth_path=/pklxkd-58/obj00000000000000000059 2023-05-31T20:39:18.210 DEBUG:boto:path=/pklxkd-58/obj00000000000000000059?versionId=null 2023-05-31T20:39:18.211 DEBUG:boto:auth_path=/pklxkd-58/obj00000000000000000059?versionId=null 2023-05-31T20:39:18.211 DEBUG:boto:Method: HEAD 2023-05-31T20:39:18.212 DEBUG:boto:Path: /pklxkd-58/obj00000000000000000059?versionId=null 2023-05-31T20:39:18.212 DEBUG:boto:Data: 2023-05-31T20:39:18.213 DEBUG:boto:Headers: {} 2023-05-31T20:39:18.214 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:39:18.215 DEBUG:boto:Port: 8000 2023-05-31T20:39:18.216 DEBUG:boto:Params: {} 2023-05-31T20:39:18.216 DEBUG:boto:Token: None 2023-05-31T20:39:18.217 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:39:18 GMT /pklxkd-58/obj00000000000000000059?versionId=null 2023-05-31T20:39:18.218 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:laZlgvQR1eNtdfHOXgHpf127KXk= 2023-05-31T20:39:18.218 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:39:18 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:laZlgvQR1eNtdfHOXgHpf127KXk=', 'Content-Length': '0'} 2023-05-31T20:39:18.220 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:38:18 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f889b4fbcf6f15ea-006477b076-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:39:18 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:39:18.221 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000059 2023-05-31T20:39:18.221 DEBUG:boto:path=/pklxkd-58/obj00000000000000000059 2023-05-31T20:39:18.222 DEBUG:boto:auth_path=/pklxkd-58/obj00000000000000000059 2023-05-31T20:39:18.222 DEBUG:boto:path=/pklxkd-58/obj00000000000000000059?versionId=null 2023-05-31T20:39:18.223 DEBUG:boto:auth_path=/pklxkd-58/obj00000000000000000059?versionId=null 2023-05-31T20:39:18.223 DEBUG:boto:Method: GET 2023-05-31T20:39:18.224 DEBUG:boto:Path: /pklxkd-58/obj00000000000000000059?versionId=null 2023-05-31T20:39:18.224 DEBUG:boto:Data: 2023-05-31T20:39:18.225 DEBUG:boto:Headers: {} 2023-05-31T20:39:18.226 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:39:18.226 DEBUG:boto:Port: 8001 2023-05-31T20:39:18.227 DEBUG:boto:Params: {} 2023-05-31T20:39:18.227 DEBUG:boto:Token: None 2023-05-31T20:39:18.228 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:39:18 GMT /pklxkd-58/obj00000000000000000059?versionId=null 2023-05-31T20:39:18.228 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:gYztoXat1obpTZGscJjEZN+5gNA= 2023-05-31T20:39:18.229 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:39:18 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:gYztoXat1obpTZGscJjEZN+5gNA=', 'Content-Length': '0'} 2023-05-31T20:39:18.230 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:38:18 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006d7c1249233a5ca5-006477b076-1c17-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:39:18 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:39:18.231 DEBUG:boto:path=/pklxkd-58/obj00000000000000000059 2023-05-31T20:39:18.231 DEBUG:boto:auth_path=/pklxkd-58/obj00000000000000000059 2023-05-31T20:39:18.232 DEBUG:boto:path=/pklxkd-58/obj00000000000000000059?versionId=null 2023-05-31T20:39:18.232 DEBUG:boto:auth_path=/pklxkd-58/obj00000000000000000059?versionId=null 2023-05-31T20:39:18.233 DEBUG:boto:Method: GET 2023-05-31T20:39:18.233 DEBUG:boto:Path: /pklxkd-58/obj00000000000000000059?versionId=null 2023-05-31T20:39:18.233 DEBUG:boto:Data: 2023-05-31T20:39:18.234 DEBUG:boto:Headers: {} 2023-05-31T20:39:18.234 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:39:18.234 DEBUG:boto:Port: 8000 2023-05-31T20:39:18.235 DEBUG:boto:Params: {} 2023-05-31T20:39:18.235 DEBUG:boto:Token: None 2023-05-31T20:39:18.236 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:39:18 GMT /pklxkd-58/obj00000000000000000059?versionId=null 2023-05-31T20:39:18.236 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:gYztoXat1obpTZGscJjEZN+5gNA= 2023-05-31T20:39:18.237 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:39:18 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:gYztoXat1obpTZGscJjEZN+5gNA=', 'Content-Length': '0'} 2023-05-31T20:39:18.239 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:38:18 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d414a4ae5e404ff8-006477b076-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:39:18 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:39:18.284 DEBUG:boto:path=/pklxkd-58/ 2023-05-31T20:39:18.285 DEBUG:boto:auth_path=/pklxkd-58/ 2023-05-31T20:39:18.286 DEBUG:boto:path=/pklxkd-58/?prefix=obj00000000000000000059 2023-05-31T20:39:18.286 DEBUG:boto:auth_path=/pklxkd-58/?prefix=obj00000000000000000059 2023-05-31T20:39:18.286 DEBUG:boto:Method: GET 2023-05-31T20:39:18.287 DEBUG:boto:Path: /pklxkd-58/?prefix=obj00000000000000000059 2023-05-31T20:39:18.287 DEBUG:boto:Data: 2023-05-31T20:39:18.287 DEBUG:boto:Headers: {} 2023-05-31T20:39:18.288 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:39:18.288 DEBUG:boto:Port: 8001 2023-05-31T20:39:18.289 DEBUG:boto:Params: {} 2023-05-31T20:39:18.289 DEBUG:boto:Token: None 2023-05-31T20:39:18.289 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:39:18 GMT /pklxkd-58/ 2023-05-31T20:39:18.290 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:3DG8wpCqcsv/yPsAzk0+okqkXZU= 2023-05-31T20:39:18.290 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:39:18 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:3DG8wpCqcsv/yPsAzk0+okqkXZU=', 'Content-Length': '0'} 2023-05-31T20:39:18.293 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000cf966b967a8c826c-006477b076-1c17-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:39:18 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:39:18.294 DEBUG:boto:b'pklxkd-58obj000000000000000000591000falseobj000000000000000000592023-05-31T20:38:18.560Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:39:18.294 DEBUG:boto:path=/pklxkd-58/ 2023-05-31T20:39:18.295 DEBUG:boto:auth_path=/pklxkd-58/ 2023-05-31T20:39:18.295 DEBUG:boto:path=/pklxkd-58/?prefix=obj00000000000000000059 2023-05-31T20:39:18.295 DEBUG:boto:auth_path=/pklxkd-58/?prefix=obj00000000000000000059 2023-05-31T20:39:18.296 DEBUG:boto:Method: GET 2023-05-31T20:39:18.296 DEBUG:boto:Path: /pklxkd-58/?prefix=obj00000000000000000059 2023-05-31T20:39:18.297 DEBUG:boto:Data: 2023-05-31T20:39:18.297 DEBUG:boto:Headers: {} 2023-05-31T20:39:18.297 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:39:18.298 DEBUG:boto:Port: 8000 2023-05-31T20:39:18.298 DEBUG:boto:Params: {} 2023-05-31T20:39:18.298 DEBUG:boto:Token: None 2023-05-31T20:39:18.299 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:39:18 GMT /pklxkd-58/ 2023-05-31T20:39:18.299 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:3DG8wpCqcsv/yPsAzk0+okqkXZU= 2023-05-31T20:39:18.300 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:39:18 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:3DG8wpCqcsv/yPsAzk0+okqkXZU=', 'Content-Length': '0'} 2023-05-31T20:39:18.304 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000574ca9b0f700004e-006477b076-1630-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:39:18 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:39:18.305 DEBUG:boto:b'pklxkd-58obj000000000000000000591000falseobj000000000000000000592023-05-31T20:38:18.560Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:39:18.306 DEBUG:boto:path=/pklxkd-58/obj00000000000000000059 2023-05-31T20:39:18.307 DEBUG:boto:auth_path=/pklxkd-58/obj00000000000000000059 2023-05-31T20:39:18.308 DEBUG:boto:Method: HEAD 2023-05-31T20:39:18.308 DEBUG:boto:Path: /pklxkd-58/obj00000000000000000059 2023-05-31T20:39:18.309 DEBUG:boto:Data: 2023-05-31T20:39:18.310 DEBUG:boto:Headers: {} 2023-05-31T20:39:18.310 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:39:18.311 DEBUG:boto:Port: 8001 2023-05-31T20:39:18.311 DEBUG:boto:Params: {} 2023-05-31T20:39:18.311 DEBUG:boto:Token: None 2023-05-31T20:39:18.312 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:39:18 GMT /pklxkd-58/obj00000000000000000059 2023-05-31T20:39:18.313 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:2o3t9j/CImFuwLSczdBUI54/lvw= 2023-05-31T20:39:18.314 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:39:18 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:2o3t9j/CImFuwLSczdBUI54/lvw=', 'Content-Length': '0'} 2023-05-31T20:39:18.316 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:38:18 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000dcf104e4332fcaab-006477b076-1c17-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:39:18 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:39:18.317 DEBUG:boto:path=/pklxkd-58/obj00000000000000000059 2023-05-31T20:39:18.318 DEBUG:boto:auth_path=/pklxkd-58/obj00000000000000000059 2023-05-31T20:39:18.318 DEBUG:boto:Method: HEAD 2023-05-31T20:39:18.319 DEBUG:boto:Path: /pklxkd-58/obj00000000000000000059 2023-05-31T20:39:18.319 DEBUG:boto:Data: 2023-05-31T20:39:18.320 DEBUG:boto:Headers: {} 2023-05-31T20:39:18.320 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:39:18.321 DEBUG:boto:Port: 8000 2023-05-31T20:39:18.321 DEBUG:boto:Params: {} 2023-05-31T20:39:18.322 DEBUG:boto:Token: None 2023-05-31T20:39:18.322 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:39:18 GMT /pklxkd-58/obj00000000000000000059 2023-05-31T20:39:18.323 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:2o3t9j/CImFuwLSczdBUI54/lvw= 2023-05-31T20:39:18.323 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:39:18 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:2o3t9j/CImFuwLSczdBUI54/lvw=', 'Content-Length': '0'} 2023-05-31T20:39:18.326 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:38:18 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c64608bc26ba1635-006477b076-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:39:18 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:39:18.327 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000059 2023-05-31T20:39:18.327 DEBUG:boto:path=/pklxkd-58/obj00000000000000000059 2023-05-31T20:39:18.328 DEBUG:boto:auth_path=/pklxkd-58/obj00000000000000000059 2023-05-31T20:39:18.328 DEBUG:boto:Method: GET 2023-05-31T20:39:18.329 DEBUG:boto:Path: /pklxkd-58/obj00000000000000000059 2023-05-31T20:39:18.329 DEBUG:boto:Data: 2023-05-31T20:39:18.330 DEBUG:boto:Headers: {} 2023-05-31T20:39:18.330 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:39:18.330 DEBUG:boto:Port: 8001 2023-05-31T20:39:18.332 DEBUG:boto:Params: {} 2023-05-31T20:39:18.332 DEBUG:boto:Token: None 2023-05-31T20:39:18.333 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:39:18 GMT /pklxkd-58/obj00000000000000000059 2023-05-31T20:39:18.333 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:6aDZUbWRAeDfgotBYQDn7XSA0hM= 2023-05-31T20:39:18.333 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:39:18 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:6aDZUbWRAeDfgotBYQDn7XSA0hM=', 'Content-Length': '0'} 2023-05-31T20:39:18.335 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:38:18 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000d19d358809db02b-006477b076-1c17-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:39:18 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:39:18.381 DEBUG:boto:path=/pklxkd-58/obj00000000000000000059 2023-05-31T20:39:18.382 DEBUG:boto:auth_path=/pklxkd-58/obj00000000000000000059 2023-05-31T20:39:18.383 DEBUG:boto:Method: GET 2023-05-31T20:39:18.384 DEBUG:boto:Path: /pklxkd-58/obj00000000000000000059 2023-05-31T20:39:18.385 DEBUG:boto:Data: 2023-05-31T20:39:18.386 DEBUG:boto:Headers: {} 2023-05-31T20:39:18.386 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:39:18.387 DEBUG:boto:Port: 8000 2023-05-31T20:39:18.387 DEBUG:boto:Params: {} 2023-05-31T20:39:18.388 DEBUG:boto:Token: None 2023-05-31T20:39:18.388 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:39:18 GMT /pklxkd-58/obj00000000000000000059 2023-05-31T20:39:18.388 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:6aDZUbWRAeDfgotBYQDn7XSA0hM= 2023-05-31T20:39:18.389 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:39:18 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:6aDZUbWRAeDfgotBYQDn7XSA0hM=', 'Content-Length': '0'} 2023-05-31T20:39:18.391 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:38:18 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a9817e9543379120-006477b076-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:39:18 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:39:18.392 DEBUG:boto:path=/pklxkd-58/ 2023-05-31T20:39:18.392 DEBUG:boto:auth_path=/pklxkd-58/ 2023-05-31T20:39:18.393 DEBUG:boto:path=/pklxkd-58/?prefix=obj00000000000000000059 2023-05-31T20:39:18.393 DEBUG:boto:auth_path=/pklxkd-58/?prefix=obj00000000000000000059 2023-05-31T20:39:18.394 DEBUG:boto:Method: GET 2023-05-31T20:39:18.394 DEBUG:boto:Path: /pklxkd-58/?prefix=obj00000000000000000059 2023-05-31T20:39:18.394 DEBUG:boto:Data: 2023-05-31T20:39:18.395 DEBUG:boto:Headers: {} 2023-05-31T20:39:18.395 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:39:18.395 DEBUG:boto:Port: 8001 2023-05-31T20:39:18.396 DEBUG:boto:Params: {} 2023-05-31T20:39:18.396 DEBUG:boto:Token: None 2023-05-31T20:39:18.397 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:39:18 GMT /pklxkd-58/ 2023-05-31T20:39:18.397 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:3DG8wpCqcsv/yPsAzk0+okqkXZU= 2023-05-31T20:39:18.397 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:39:18 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:3DG8wpCqcsv/yPsAzk0+okqkXZU=', 'Content-Length': '0'} 2023-05-31T20:39:18.400 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000006c8177009d535f2f-006477b076-1c17-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:39:18 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:39:18.401 DEBUG:boto:b'pklxkd-58obj000000000000000000591000falseobj000000000000000000592023-05-31T20:38:18.560Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:39:18.401 DEBUG:boto:path=/pklxkd-58/ 2023-05-31T20:39:18.402 DEBUG:boto:auth_path=/pklxkd-58/ 2023-05-31T20:39:18.402 DEBUG:boto:path=/pklxkd-58/?prefix=obj00000000000000000059 2023-05-31T20:39:18.402 DEBUG:boto:auth_path=/pklxkd-58/?prefix=obj00000000000000000059 2023-05-31T20:39:18.403 DEBUG:boto:Method: GET 2023-05-31T20:39:18.403 DEBUG:boto:Path: /pklxkd-58/?prefix=obj00000000000000000059 2023-05-31T20:39:18.403 DEBUG:boto:Data: 2023-05-31T20:39:18.404 DEBUG:boto:Headers: {} 2023-05-31T20:39:18.404 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:39:18.405 DEBUG:boto:Port: 8000 2023-05-31T20:39:18.405 DEBUG:boto:Params: {} 2023-05-31T20:39:18.406 DEBUG:boto:Token: None 2023-05-31T20:39:18.406 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:39:18 GMT /pklxkd-58/ 2023-05-31T20:39:18.406 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:3DG8wpCqcsv/yPsAzk0+okqkXZU= 2023-05-31T20:39:18.407 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:39:18 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:3DG8wpCqcsv/yPsAzk0+okqkXZU=', 'Content-Length': '0'} 2023-05-31T20:39:18.411 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000008da5fe4ee3204cac-006477b076-1630-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:39:18 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:39:18.412 DEBUG:boto:b'pklxkd-58obj000000000000000000591000falseobj000000000000000000592023-05-31T20:38:18.560Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:39:18.413 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000060 2023-05-31T20:39:18.414 DEBUG:boto:path=/pklxkd-58/obj00000000000000000060 2023-05-31T20:39:18.415 DEBUG:boto:auth_path=/pklxkd-58/obj00000000000000000060 2023-05-31T20:39:18.416 DEBUG:boto:path=/pklxkd-58/obj00000000000000000060?versionId=null 2023-05-31T20:39:18.416 DEBUG:boto:auth_path=/pklxkd-58/obj00000000000000000060?versionId=null 2023-05-31T20:39:18.417 DEBUG:boto:Method: GET 2023-05-31T20:39:18.418 DEBUG:boto:Path: /pklxkd-58/obj00000000000000000060?versionId=null 2023-05-31T20:39:18.419 DEBUG:boto:Data: 2023-05-31T20:39:18.419 DEBUG:boto:Headers: {} 2023-05-31T20:39:18.420 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:39:18.421 DEBUG:boto:Port: 8001 2023-05-31T20:39:18.422 DEBUG:boto:Params: {} 2023-05-31T20:39:18.423 DEBUG:boto:Token: None 2023-05-31T20:39:18.423 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:39:18 GMT /pklxkd-58/obj00000000000000000060?versionId=null 2023-05-31T20:39:18.424 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:kiqP1uoJGzW7ldYNp3jp9RUNNaM= 2023-05-31T20:39:18.425 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:39:18 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:kiqP1uoJGzW7ldYNp3jp9RUNNaM=', 'Content-Length': '0'} 2023-05-31T20:39:18.428 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:38:22 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002a5b371beb91f0d6-006477b076-1c17-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:39:18 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:39:18.428 DEBUG:boto:path=/pklxkd-58/obj00000000000000000060 2023-05-31T20:39:18.429 DEBUG:boto:auth_path=/pklxkd-58/obj00000000000000000060 2023-05-31T20:39:18.429 DEBUG:boto:path=/pklxkd-58/obj00000000000000000060?versionId=null 2023-05-31T20:39:18.429 DEBUG:boto:auth_path=/pklxkd-58/obj00000000000000000060?versionId=null 2023-05-31T20:39:18.430 DEBUG:boto:Method: GET 2023-05-31T20:39:18.430 DEBUG:boto:Path: /pklxkd-58/obj00000000000000000060?versionId=null 2023-05-31T20:39:18.430 DEBUG:boto:Data: 2023-05-31T20:39:18.431 DEBUG:boto:Headers: {} 2023-05-31T20:39:18.431 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:39:18.432 DEBUG:boto:Port: 8000 2023-05-31T20:39:18.432 DEBUG:boto:Params: {} 2023-05-31T20:39:18.432 DEBUG:boto:Token: None 2023-05-31T20:39:18.433 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:39:18 GMT /pklxkd-58/obj00000000000000000060?versionId=null 2023-05-31T20:39:18.433 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:kiqP1uoJGzW7ldYNp3jp9RUNNaM= 2023-05-31T20:39:18.434 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:39:18 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:kiqP1uoJGzW7ldYNp3jp9RUNNaM=', 'Content-Length': '0'} 2023-05-31T20:39:18.436 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:38:22 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f825967076dcdeb0-006477b076-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:39:18 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:39:18.480 DEBUG:boto:path=/pklxkd-58/obj00000000000000000060 2023-05-31T20:39:18.481 DEBUG:boto:auth_path=/pklxkd-58/obj00000000000000000060 2023-05-31T20:39:18.481 DEBUG:boto:path=/pklxkd-58/obj00000000000000000060?versionId=null 2023-05-31T20:39:18.482 DEBUG:boto:auth_path=/pklxkd-58/obj00000000000000000060?versionId=null 2023-05-31T20:39:18.482 DEBUG:boto:Method: HEAD 2023-05-31T20:39:18.482 DEBUG:boto:Path: /pklxkd-58/obj00000000000000000060?versionId=null 2023-05-31T20:39:18.483 DEBUG:boto:Data: 2023-05-31T20:39:18.483 DEBUG:boto:Headers: {} 2023-05-31T20:39:18.483 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:39:18.484 DEBUG:boto:Port: 8001 2023-05-31T20:39:18.484 DEBUG:boto:Params: {} 2023-05-31T20:39:18.485 DEBUG:boto:Token: None 2023-05-31T20:39:18.485 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:39:18 GMT /pklxkd-58/obj00000000000000000060?versionId=null 2023-05-31T20:39:18.485 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:y+/zo/BSVuqhMvbvH1LLrHOfRPg= 2023-05-31T20:39:18.486 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:39:18 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:y+/zo/BSVuqhMvbvH1LLrHOfRPg=', 'Content-Length': '0'} 2023-05-31T20:39:18.488 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:38:22 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009dc96d4d2f2aae4b-006477b076-1c17-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:39:18 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:39:18.488 DEBUG:boto:path=/pklxkd-58/obj00000000000000000060 2023-05-31T20:39:18.489 DEBUG:boto:auth_path=/pklxkd-58/obj00000000000000000060 2023-05-31T20:39:18.489 DEBUG:boto:path=/pklxkd-58/obj00000000000000000060?versionId=null 2023-05-31T20:39:18.489 DEBUG:boto:auth_path=/pklxkd-58/obj00000000000000000060?versionId=null 2023-05-31T20:39:18.490 DEBUG:boto:Method: HEAD 2023-05-31T20:39:18.490 DEBUG:boto:Path: /pklxkd-58/obj00000000000000000060?versionId=null 2023-05-31T20:39:18.490 DEBUG:boto:Data: 2023-05-31T20:39:18.491 DEBUG:boto:Headers: {} 2023-05-31T20:39:18.491 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:39:18.491 DEBUG:boto:Port: 8000 2023-05-31T20:39:18.492 DEBUG:boto:Params: {} 2023-05-31T20:39:18.492 DEBUG:boto:Token: None 2023-05-31T20:39:18.493 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:39:18 GMT /pklxkd-58/obj00000000000000000060?versionId=null 2023-05-31T20:39:18.493 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:y+/zo/BSVuqhMvbvH1LLrHOfRPg= 2023-05-31T20:39:18.493 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:39:18 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:y+/zo/BSVuqhMvbvH1LLrHOfRPg=', 'Content-Length': '0'} 2023-05-31T20:39:18.495 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:38:22 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d2e2e0d80ea6ef3d-006477b076-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:39:18 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:39:18.496 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000060 2023-05-31T20:39:18.496 DEBUG:boto:path=/pklxkd-58/obj00000000000000000060 2023-05-31T20:39:18.497 DEBUG:boto:auth_path=/pklxkd-58/obj00000000000000000060 2023-05-31T20:39:18.497 DEBUG:boto:path=/pklxkd-58/obj00000000000000000060?versionId=null 2023-05-31T20:39:18.498 DEBUG:boto:auth_path=/pklxkd-58/obj00000000000000000060?versionId=null 2023-05-31T20:39:18.498 DEBUG:boto:Method: GET 2023-05-31T20:39:18.498 DEBUG:boto:Path: /pklxkd-58/obj00000000000000000060?versionId=null 2023-05-31T20:39:18.499 DEBUG:boto:Data: 2023-05-31T20:39:18.499 DEBUG:boto:Headers: {} 2023-05-31T20:39:18.499 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:39:18.500 DEBUG:boto:Port: 8001 2023-05-31T20:39:18.500 DEBUG:boto:Params: {} 2023-05-31T20:39:18.501 DEBUG:boto:Token: None 2023-05-31T20:39:18.502 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:39:18 GMT /pklxkd-58/obj00000000000000000060?versionId=null 2023-05-31T20:39:18.502 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:kiqP1uoJGzW7ldYNp3jp9RUNNaM= 2023-05-31T20:39:18.502 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:39:18 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:kiqP1uoJGzW7ldYNp3jp9RUNNaM=', 'Content-Length': '0'} 2023-05-31T20:39:18.504 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:38:22 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002e5db2fb59ad220e-006477b076-1c17-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:39:18 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:39:18.505 DEBUG:boto:path=/pklxkd-58/obj00000000000000000060 2023-05-31T20:39:18.505 DEBUG:boto:auth_path=/pklxkd-58/obj00000000000000000060 2023-05-31T20:39:18.506 DEBUG:boto:path=/pklxkd-58/obj00000000000000000060?versionId=null 2023-05-31T20:39:18.506 DEBUG:boto:auth_path=/pklxkd-58/obj00000000000000000060?versionId=null 2023-05-31T20:39:18.506 DEBUG:boto:Method: GET 2023-05-31T20:39:18.507 DEBUG:boto:Path: /pklxkd-58/obj00000000000000000060?versionId=null 2023-05-31T20:39:18.507 DEBUG:boto:Data: 2023-05-31T20:39:18.508 DEBUG:boto:Headers: {} 2023-05-31T20:39:18.508 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:39:18.508 DEBUG:boto:Port: 8000 2023-05-31T20:39:18.509 DEBUG:boto:Params: {} 2023-05-31T20:39:18.509 DEBUG:boto:Token: None 2023-05-31T20:39:18.509 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:39:18 GMT /pklxkd-58/obj00000000000000000060?versionId=null 2023-05-31T20:39:18.510 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:kiqP1uoJGzW7ldYNp3jp9RUNNaM= 2023-05-31T20:39:18.510 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:39:18 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:kiqP1uoJGzW7ldYNp3jp9RUNNaM=', 'Content-Length': '0'} 2023-05-31T20:39:18.512 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:38:22 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009f56b8166f4728fc-006477b076-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:39:18 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:39:18.513 DEBUG:boto:path=/pklxkd-58/ 2023-05-31T20:39:18.514 DEBUG:boto:auth_path=/pklxkd-58/ 2023-05-31T20:39:18.514 DEBUG:boto:path=/pklxkd-58/?prefix=obj00000000000000000060 2023-05-31T20:39:18.514 DEBUG:boto:auth_path=/pklxkd-58/?prefix=obj00000000000000000060 2023-05-31T20:39:18.515 DEBUG:boto:Method: GET 2023-05-31T20:39:18.515 DEBUG:boto:Path: /pklxkd-58/?prefix=obj00000000000000000060 2023-05-31T20:39:18.515 DEBUG:boto:Data: 2023-05-31T20:39:18.516 DEBUG:boto:Headers: {} 2023-05-31T20:39:18.516 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:39:18.517 DEBUG:boto:Port: 8001 2023-05-31T20:39:18.517 DEBUG:boto:Params: {} 2023-05-31T20:39:18.517 DEBUG:boto:Token: None 2023-05-31T20:39:18.518 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:39:18 GMT /pklxkd-58/ 2023-05-31T20:39:18.518 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:3DG8wpCqcsv/yPsAzk0+okqkXZU= 2023-05-31T20:39:18.519 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:39:18 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:3DG8wpCqcsv/yPsAzk0+okqkXZU=', 'Content-Length': '0'} 2023-05-31T20:39:18.521 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ce88faa64bdf0bf8-006477b076-1c17-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:39:18 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:39:18.521 DEBUG:boto:b'pklxkd-58obj000000000000000000601000falseobj000000000000000000602023-05-31T20:38:22.575Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:39:18.522 DEBUG:boto:path=/pklxkd-58/ 2023-05-31T20:39:18.523 DEBUG:boto:auth_path=/pklxkd-58/ 2023-05-31T20:39:18.523 DEBUG:boto:path=/pklxkd-58/?prefix=obj00000000000000000060 2023-05-31T20:39:18.523 DEBUG:boto:auth_path=/pklxkd-58/?prefix=obj00000000000000000060 2023-05-31T20:39:18.524 DEBUG:boto:Method: GET 2023-05-31T20:39:18.524 DEBUG:boto:Path: /pklxkd-58/?prefix=obj00000000000000000060 2023-05-31T20:39:18.525 DEBUG:boto:Data: 2023-05-31T20:39:18.525 DEBUG:boto:Headers: {} 2023-05-31T20:39:18.525 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:39:18.526 DEBUG:boto:Port: 8000 2023-05-31T20:39:18.526 DEBUG:boto:Params: {} 2023-05-31T20:39:18.526 DEBUG:boto:Token: None 2023-05-31T20:39:18.527 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:39:18 GMT /pklxkd-58/ 2023-05-31T20:39:18.528 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:3DG8wpCqcsv/yPsAzk0+okqkXZU= 2023-05-31T20:39:18.528 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:39:18 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:3DG8wpCqcsv/yPsAzk0+okqkXZU=', 'Content-Length': '0'} 2023-05-31T20:39:18.533 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000001c87df46c6883e70-006477b076-1630-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:39:18 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:39:18.534 DEBUG:boto:b'pklxkd-58obj000000000000000000601000falseobj000000000000000000602023-05-31T20:38:22.575Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:39:18.536 DEBUG:boto:path=/pklxkd-58/obj00000000000000000060 2023-05-31T20:39:18.536 DEBUG:boto:auth_path=/pklxkd-58/obj00000000000000000060 2023-05-31T20:39:18.537 DEBUG:boto:Method: HEAD 2023-05-31T20:39:18.538 DEBUG:boto:Path: /pklxkd-58/obj00000000000000000060 2023-05-31T20:39:18.538 DEBUG:boto:Data: 2023-05-31T20:39:18.539 DEBUG:boto:Headers: {} 2023-05-31T20:39:18.540 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:39:18.540 DEBUG:boto:Port: 8001 2023-05-31T20:39:18.541 DEBUG:boto:Params: {} 2023-05-31T20:39:18.542 DEBUG:boto:Token: None 2023-05-31T20:39:18.543 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:39:18 GMT /pklxkd-58/obj00000000000000000060 2023-05-31T20:39:18.544 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:yrY+9DH20+yndpYrz+Ziy8fNrHs= 2023-05-31T20:39:18.545 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:39:18 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:yrY+9DH20+yndpYrz+Ziy8fNrHs=', 'Content-Length': '0'} 2023-05-31T20:39:18.547 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:38:22 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005aabf7c2357f8953-006477b076-1c17-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:39:18 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:39:18.548 DEBUG:boto:path=/pklxkd-58/obj00000000000000000060 2023-05-31T20:39:18.549 DEBUG:boto:auth_path=/pklxkd-58/obj00000000000000000060 2023-05-31T20:39:18.550 DEBUG:boto:Method: HEAD 2023-05-31T20:39:18.550 DEBUG:boto:Path: /pklxkd-58/obj00000000000000000060 2023-05-31T20:39:18.551 DEBUG:boto:Data: 2023-05-31T20:39:18.552 DEBUG:boto:Headers: {} 2023-05-31T20:39:18.553 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:39:18.554 DEBUG:boto:Port: 8000 2023-05-31T20:39:18.555 DEBUG:boto:Params: {} 2023-05-31T20:39:18.556 DEBUG:boto:Token: None 2023-05-31T20:39:18.557 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:39:18 GMT /pklxkd-58/obj00000000000000000060 2023-05-31T20:39:18.557 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:yrY+9DH20+yndpYrz+Ziy8fNrHs= 2023-05-31T20:39:18.558 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:39:18 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:yrY+9DH20+yndpYrz+Ziy8fNrHs=', 'Content-Length': '0'} 2023-05-31T20:39:18.560 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:38:22 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008d2570cb758d19a5-006477b076-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:39:18 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:39:18.560 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000060 2023-05-31T20:39:18.561 DEBUG:boto:path=/pklxkd-58/obj00000000000000000060 2023-05-31T20:39:18.561 DEBUG:boto:auth_path=/pklxkd-58/obj00000000000000000060 2023-05-31T20:39:18.562 DEBUG:boto:Method: GET 2023-05-31T20:39:18.562 DEBUG:boto:Path: /pklxkd-58/obj00000000000000000060 2023-05-31T20:39:18.563 DEBUG:boto:Data: 2023-05-31T20:39:18.563 DEBUG:boto:Headers: {} 2023-05-31T20:39:18.563 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:39:18.564 DEBUG:boto:Port: 8001 2023-05-31T20:39:18.564 DEBUG:boto:Params: {} 2023-05-31T20:39:18.565 DEBUG:boto:Token: None 2023-05-31T20:39:18.565 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:39:18 GMT /pklxkd-58/obj00000000000000000060 2023-05-31T20:39:18.565 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:GBPMWgwUoMx+wqHTMEIZ0D6IamM= 2023-05-31T20:39:18.566 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:39:18 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:GBPMWgwUoMx+wqHTMEIZ0D6IamM=', 'Content-Length': '0'} 2023-05-31T20:39:18.568 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:38:22 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000024517f6bb3c3801f-006477b076-1c17-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:39:18 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:39:18.608 DEBUG:boto:path=/pklxkd-58/obj00000000000000000060 2023-05-31T20:39:18.609 DEBUG:boto:auth_path=/pklxkd-58/obj00000000000000000060 2023-05-31T20:39:18.610 DEBUG:boto:Method: GET 2023-05-31T20:39:18.610 DEBUG:boto:Path: /pklxkd-58/obj00000000000000000060 2023-05-31T20:39:18.611 DEBUG:boto:Data: 2023-05-31T20:39:18.612 DEBUG:boto:Headers: {} 2023-05-31T20:39:18.613 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:39:18.613 DEBUG:boto:Port: 8000 2023-05-31T20:39:18.614 DEBUG:boto:Params: {} 2023-05-31T20:39:18.615 DEBUG:boto:Token: None 2023-05-31T20:39:18.616 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:39:18 GMT /pklxkd-58/obj00000000000000000060 2023-05-31T20:39:18.617 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:GBPMWgwUoMx+wqHTMEIZ0D6IamM= 2023-05-31T20:39:18.618 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:39:18 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:GBPMWgwUoMx+wqHTMEIZ0D6IamM=', 'Content-Length': '0'} 2023-05-31T20:39:18.621 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:38:22 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c5648f90048b8a96-006477b076-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:39:18 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:39:18.622 DEBUG:boto:path=/pklxkd-58/ 2023-05-31T20:39:18.622 DEBUG:boto:auth_path=/pklxkd-58/ 2023-05-31T20:39:18.622 DEBUG:boto:path=/pklxkd-58/?prefix=obj00000000000000000060 2023-05-31T20:39:18.623 DEBUG:boto:auth_path=/pklxkd-58/?prefix=obj00000000000000000060 2023-05-31T20:39:18.623 DEBUG:boto:Method: GET 2023-05-31T20:39:18.623 DEBUG:boto:Path: /pklxkd-58/?prefix=obj00000000000000000060 2023-05-31T20:39:18.624 DEBUG:boto:Data: 2023-05-31T20:39:18.624 DEBUG:boto:Headers: {} 2023-05-31T20:39:18.625 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:39:18.625 DEBUG:boto:Port: 8001 2023-05-31T20:39:18.625 DEBUG:boto:Params: {} 2023-05-31T20:39:18.626 DEBUG:boto:Token: None 2023-05-31T20:39:18.626 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:39:18 GMT /pklxkd-58/ 2023-05-31T20:39:18.627 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:3DG8wpCqcsv/yPsAzk0+okqkXZU= 2023-05-31T20:39:18.627 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:39:18 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:3DG8wpCqcsv/yPsAzk0+okqkXZU=', 'Content-Length': '0'} 2023-05-31T20:39:18.630 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000003b411741efb83f27-006477b076-1c17-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:39:18 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:39:18.631 DEBUG:boto:b'pklxkd-58obj000000000000000000601000falseobj000000000000000000602023-05-31T20:38:22.575Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:39:18.632 DEBUG:boto:path=/pklxkd-58/ 2023-05-31T20:39:18.632 DEBUG:boto:auth_path=/pklxkd-58/ 2023-05-31T20:39:18.633 DEBUG:boto:path=/pklxkd-58/?prefix=obj00000000000000000060 2023-05-31T20:39:18.633 DEBUG:boto:auth_path=/pklxkd-58/?prefix=obj00000000000000000060 2023-05-31T20:39:18.633 DEBUG:boto:Method: GET 2023-05-31T20:39:18.634 DEBUG:boto:Path: /pklxkd-58/?prefix=obj00000000000000000060 2023-05-31T20:39:18.635 DEBUG:boto:Data: 2023-05-31T20:39:18.635 DEBUG:boto:Headers: {} 2023-05-31T20:39:18.636 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:39:18.636 DEBUG:boto:Port: 8000 2023-05-31T20:39:18.636 DEBUG:boto:Params: {} 2023-05-31T20:39:18.637 DEBUG:boto:Token: None 2023-05-31T20:39:18.637 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:39:18 GMT /pklxkd-58/ 2023-05-31T20:39:18.638 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:3DG8wpCqcsv/yPsAzk0+okqkXZU= 2023-05-31T20:39:18.638 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:39:18 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:3DG8wpCqcsv/yPsAzk0+okqkXZU=', 'Content-Length': '0'} 2023-05-31T20:39:18.642 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000043d5bc867b429e8e-006477b076-1630-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:39:18 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:39:18.643 DEBUG:boto:b'pklxkd-58obj000000000000000000601000falseobj000000000000000000602023-05-31T20:38:22.575Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:39:18.644 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000061 2023-05-31T20:39:18.644 DEBUG:boto:path=/pklxkd-58/obj00000000000000000061 2023-05-31T20:39:18.645 DEBUG:boto:auth_path=/pklxkd-58/obj00000000000000000061 2023-05-31T20:39:18.645 DEBUG:boto:path=/pklxkd-58/obj00000000000000000061?versionId=null 2023-05-31T20:39:18.646 DEBUG:boto:auth_path=/pklxkd-58/obj00000000000000000061?versionId=null 2023-05-31T20:39:18.646 DEBUG:boto:Method: GET 2023-05-31T20:39:18.647 DEBUG:boto:Path: /pklxkd-58/obj00000000000000000061?versionId=null 2023-05-31T20:39:18.647 DEBUG:boto:Data: 2023-05-31T20:39:18.648 DEBUG:boto:Headers: {} 2023-05-31T20:39:18.649 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:39:18.649 DEBUG:boto:Port: 8001 2023-05-31T20:39:18.650 DEBUG:boto:Params: {} 2023-05-31T20:39:18.651 DEBUG:boto:Token: None 2023-05-31T20:39:18.652 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:39:18 GMT /pklxkd-58/obj00000000000000000061?versionId=null 2023-05-31T20:39:18.653 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:YQDqR/WGNPXNAqrXBHZao31BjLY= 2023-05-31T20:39:18.654 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:39:18 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:YQDqR/WGNPXNAqrXBHZao31BjLY=', 'Content-Length': '0'} 2023-05-31T20:39:18.656 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:38:21 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b607bb36130b8783-006477b076-1c17-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:39:18 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:39:18.657 DEBUG:boto:path=/pklxkd-58/obj00000000000000000061 2023-05-31T20:39:18.658 DEBUG:boto:auth_path=/pklxkd-58/obj00000000000000000061 2023-05-31T20:39:18.659 DEBUG:boto:path=/pklxkd-58/obj00000000000000000061?versionId=null 2023-05-31T20:39:18.659 DEBUG:boto:auth_path=/pklxkd-58/obj00000000000000000061?versionId=null 2023-05-31T20:39:18.659 DEBUG:boto:Method: GET 2023-05-31T20:39:18.660 DEBUG:boto:Path: /pklxkd-58/obj00000000000000000061?versionId=null 2023-05-31T20:39:18.661 DEBUG:boto:Data: 2023-05-31T20:39:18.662 DEBUG:boto:Headers: {} 2023-05-31T20:39:18.662 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:39:18.662 DEBUG:boto:Port: 8000 2023-05-31T20:39:18.663 DEBUG:boto:Params: {} 2023-05-31T20:39:18.663 DEBUG:boto:Token: None 2023-05-31T20:39:18.664 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:39:18 GMT /pklxkd-58/obj00000000000000000061?versionId=null 2023-05-31T20:39:18.664 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:YQDqR/WGNPXNAqrXBHZao31BjLY= 2023-05-31T20:39:18.665 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:39:18 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:YQDqR/WGNPXNAqrXBHZao31BjLY=', 'Content-Length': '0'} 2023-05-31T20:39:18.667 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:38:21 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f8c3728d6c2156b4-006477b076-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:39:18 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:39:18.668 DEBUG:boto:path=/pklxkd-58/obj00000000000000000061 2023-05-31T20:39:18.668 DEBUG:boto:auth_path=/pklxkd-58/obj00000000000000000061 2023-05-31T20:39:18.668 DEBUG:boto:path=/pklxkd-58/obj00000000000000000061?versionId=null 2023-05-31T20:39:18.669 DEBUG:boto:auth_path=/pklxkd-58/obj00000000000000000061?versionId=null 2023-05-31T20:39:18.669 DEBUG:boto:Method: HEAD 2023-05-31T20:39:18.669 DEBUG:boto:Path: /pklxkd-58/obj00000000000000000061?versionId=null 2023-05-31T20:39:18.670 DEBUG:boto:Data: 2023-05-31T20:39:18.670 DEBUG:boto:Headers: {} 2023-05-31T20:39:18.670 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:39:18.671 DEBUG:boto:Port: 8001 2023-05-31T20:39:18.671 DEBUG:boto:Params: {} 2023-05-31T20:39:18.671 DEBUG:boto:Token: None 2023-05-31T20:39:18.672 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:39:18 GMT /pklxkd-58/obj00000000000000000061?versionId=null 2023-05-31T20:39:18.672 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:tWx8NzC3MsI+nZmzIXqkZT2GITc= 2023-05-31T20:39:18.673 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:39:18 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:tWx8NzC3MsI+nZmzIXqkZT2GITc=', 'Content-Length': '0'} 2023-05-31T20:39:18.675 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:38:21 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000965b5123b0ed462d-006477b076-1c17-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:39:18 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:39:18.675 DEBUG:boto:path=/pklxkd-58/obj00000000000000000061 2023-05-31T20:39:18.676 DEBUG:boto:auth_path=/pklxkd-58/obj00000000000000000061 2023-05-31T20:39:18.676 DEBUG:boto:path=/pklxkd-58/obj00000000000000000061?versionId=null 2023-05-31T20:39:18.676 DEBUG:boto:auth_path=/pklxkd-58/obj00000000000000000061?versionId=null 2023-05-31T20:39:18.677 DEBUG:boto:Method: HEAD 2023-05-31T20:39:18.677 DEBUG:boto:Path: /pklxkd-58/obj00000000000000000061?versionId=null 2023-05-31T20:39:18.677 DEBUG:boto:Data: 2023-05-31T20:39:18.678 DEBUG:boto:Headers: {} 2023-05-31T20:39:18.678 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:39:18.678 DEBUG:boto:Port: 8000 2023-05-31T20:39:18.679 DEBUG:boto:Params: {} 2023-05-31T20:39:18.679 DEBUG:boto:Token: None 2023-05-31T20:39:18.680 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:39:18 GMT /pklxkd-58/obj00000000000000000061?versionId=null 2023-05-31T20:39:18.680 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:tWx8NzC3MsI+nZmzIXqkZT2GITc= 2023-05-31T20:39:18.680 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:39:18 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:tWx8NzC3MsI+nZmzIXqkZT2GITc=', 'Content-Length': '0'} 2023-05-31T20:39:18.682 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:38:21 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000fb16e8794d99334f-006477b076-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:39:18 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:39:18.683 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000061 2023-05-31T20:39:18.683 DEBUG:boto:path=/pklxkd-58/obj00000000000000000061 2023-05-31T20:39:18.684 DEBUG:boto:auth_path=/pklxkd-58/obj00000000000000000061 2023-05-31T20:39:18.684 DEBUG:boto:path=/pklxkd-58/obj00000000000000000061?versionId=null 2023-05-31T20:39:18.684 DEBUG:boto:auth_path=/pklxkd-58/obj00000000000000000061?versionId=null 2023-05-31T20:39:18.685 DEBUG:boto:Method: GET 2023-05-31T20:39:18.685 DEBUG:boto:Path: /pklxkd-58/obj00000000000000000061?versionId=null 2023-05-31T20:39:18.685 DEBUG:boto:Data: 2023-05-31T20:39:18.686 DEBUG:boto:Headers: {} 2023-05-31T20:39:18.686 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:39:18.686 DEBUG:boto:Port: 8001 2023-05-31T20:39:18.687 DEBUG:boto:Params: {} 2023-05-31T20:39:18.687 DEBUG:boto:Token: None 2023-05-31T20:39:18.688 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:39:18 GMT /pklxkd-58/obj00000000000000000061?versionId=null 2023-05-31T20:39:18.688 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:YQDqR/WGNPXNAqrXBHZao31BjLY= 2023-05-31T20:39:18.689 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:39:18 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:YQDqR/WGNPXNAqrXBHZao31BjLY=', 'Content-Length': '0'} 2023-05-31T20:39:18.691 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:38:21 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002375995eed7a587b-006477b076-1c17-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:39:18 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:39:18.691 DEBUG:boto:path=/pklxkd-58/obj00000000000000000061 2023-05-31T20:39:18.692 DEBUG:boto:auth_path=/pklxkd-58/obj00000000000000000061 2023-05-31T20:39:18.692 DEBUG:boto:path=/pklxkd-58/obj00000000000000000061?versionId=null 2023-05-31T20:39:18.693 DEBUG:boto:auth_path=/pklxkd-58/obj00000000000000000061?versionId=null 2023-05-31T20:39:18.693 DEBUG:boto:Method: GET 2023-05-31T20:39:18.693 DEBUG:boto:Path: /pklxkd-58/obj00000000000000000061?versionId=null 2023-05-31T20:39:18.694 DEBUG:boto:Data: 2023-05-31T20:39:18.694 DEBUG:boto:Headers: {} 2023-05-31T20:39:18.695 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:39:18.695 DEBUG:boto:Port: 8000 2023-05-31T20:39:18.695 DEBUG:boto:Params: {} 2023-05-31T20:39:18.696 DEBUG:boto:Token: None 2023-05-31T20:39:18.696 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:39:18 GMT /pklxkd-58/obj00000000000000000061?versionId=null 2023-05-31T20:39:18.697 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:YQDqR/WGNPXNAqrXBHZao31BjLY= 2023-05-31T20:39:18.697 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:39:18 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:YQDqR/WGNPXNAqrXBHZao31BjLY=', 'Content-Length': '0'} 2023-05-31T20:39:18.699 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:38:21 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000082a34e25d43418e1-006477b076-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:39:18 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:39:18.700 DEBUG:boto:path=/pklxkd-58/ 2023-05-31T20:39:18.700 DEBUG:boto:auth_path=/pklxkd-58/ 2023-05-31T20:39:18.700 DEBUG:boto:path=/pklxkd-58/?prefix=obj00000000000000000061 2023-05-31T20:39:18.701 DEBUG:boto:auth_path=/pklxkd-58/?prefix=obj00000000000000000061 2023-05-31T20:39:18.701 DEBUG:boto:Method: GET 2023-05-31T20:39:18.702 DEBUG:boto:Path: /pklxkd-58/?prefix=obj00000000000000000061 2023-05-31T20:39:18.702 DEBUG:boto:Data: 2023-05-31T20:39:18.702 DEBUG:boto:Headers: {} 2023-05-31T20:39:18.703 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:39:18.703 DEBUG:boto:Port: 8001 2023-05-31T20:39:18.703 DEBUG:boto:Params: {} 2023-05-31T20:39:18.704 DEBUG:boto:Token: None 2023-05-31T20:39:18.704 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:39:18 GMT /pklxkd-58/ 2023-05-31T20:39:18.705 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:3DG8wpCqcsv/yPsAzk0+okqkXZU= 2023-05-31T20:39:18.705 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:39:18 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:3DG8wpCqcsv/yPsAzk0+okqkXZU=', 'Content-Length': '0'} 2023-05-31T20:39:18.707 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000896175859551a18c-006477b076-1c17-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:39:18 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:39:18.708 DEBUG:boto:b'pklxkd-58obj000000000000000000611000falseobj000000000000000000612023-05-31T20:38:21.154Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:39:18.709 DEBUG:boto:path=/pklxkd-58/ 2023-05-31T20:39:18.709 DEBUG:boto:auth_path=/pklxkd-58/ 2023-05-31T20:39:18.709 DEBUG:boto:path=/pklxkd-58/?prefix=obj00000000000000000061 2023-05-31T20:39:18.710 DEBUG:boto:auth_path=/pklxkd-58/?prefix=obj00000000000000000061 2023-05-31T20:39:18.710 DEBUG:boto:Method: GET 2023-05-31T20:39:18.710 DEBUG:boto:Path: /pklxkd-58/?prefix=obj00000000000000000061 2023-05-31T20:39:18.711 DEBUG:boto:Data: 2023-05-31T20:39:18.711 DEBUG:boto:Headers: {} 2023-05-31T20:39:18.711 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:39:18.712 DEBUG:boto:Port: 8000 2023-05-31T20:39:18.712 DEBUG:boto:Params: {} 2023-05-31T20:39:18.713 DEBUG:boto:Token: None 2023-05-31T20:39:18.713 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:39:18 GMT /pklxkd-58/ 2023-05-31T20:39:18.713 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:3DG8wpCqcsv/yPsAzk0+okqkXZU= 2023-05-31T20:39:18.714 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:39:18 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:3DG8wpCqcsv/yPsAzk0+okqkXZU=', 'Content-Length': '0'} 2023-05-31T20:39:18.718 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000cc9326876f43dbff-006477b076-1630-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:39:18 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:39:18.718 DEBUG:boto:b'pklxkd-58obj000000000000000000611000falseobj000000000000000000612023-05-31T20:38:21.154Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:39:18.719 DEBUG:boto:path=/pklxkd-58/obj00000000000000000061 2023-05-31T20:39:18.719 DEBUG:boto:auth_path=/pklxkd-58/obj00000000000000000061 2023-05-31T20:39:18.720 DEBUG:boto:Method: HEAD 2023-05-31T20:39:18.720 DEBUG:boto:Path: /pklxkd-58/obj00000000000000000061 2023-05-31T20:39:18.721 DEBUG:boto:Data: 2023-05-31T20:39:18.721 DEBUG:boto:Headers: {} 2023-05-31T20:39:18.721 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:39:18.722 DEBUG:boto:Port: 8001 2023-05-31T20:39:18.722 DEBUG:boto:Params: {} 2023-05-31T20:39:18.722 DEBUG:boto:Token: None 2023-05-31T20:39:18.723 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:39:18 GMT /pklxkd-58/obj00000000000000000061 2023-05-31T20:39:18.723 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:tktRx2MtYUkhrMSoi/7MaA+hFpg= 2023-05-31T20:39:18.723 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:39:18 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:tktRx2MtYUkhrMSoi/7MaA+hFpg=', 'Content-Length': '0'} 2023-05-31T20:39:18.725 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:38:21 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ccc79706d3fd9f90-006477b076-1c17-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:39:18 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:39:18.726 DEBUG:boto:path=/pklxkd-58/obj00000000000000000061 2023-05-31T20:39:18.726 DEBUG:boto:auth_path=/pklxkd-58/obj00000000000000000061 2023-05-31T20:39:18.727 DEBUG:boto:Method: HEAD 2023-05-31T20:39:18.727 DEBUG:boto:Path: /pklxkd-58/obj00000000000000000061 2023-05-31T20:39:18.727 DEBUG:boto:Data: 2023-05-31T20:39:18.728 DEBUG:boto:Headers: {} 2023-05-31T20:39:18.728 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:39:18.728 DEBUG:boto:Port: 8000 2023-05-31T20:39:18.729 DEBUG:boto:Params: {} 2023-05-31T20:39:18.729 DEBUG:boto:Token: None 2023-05-31T20:39:18.729 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:39:18 GMT /pklxkd-58/obj00000000000000000061 2023-05-31T20:39:18.730 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:tktRx2MtYUkhrMSoi/7MaA+hFpg= 2023-05-31T20:39:18.730 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:39:18 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:tktRx2MtYUkhrMSoi/7MaA+hFpg=', 'Content-Length': '0'} 2023-05-31T20:39:18.732 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:38:21 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004dc3190350335bdf-006477b076-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:39:18 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:39:18.733 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000061 2023-05-31T20:39:18.733 DEBUG:boto:path=/pklxkd-58/obj00000000000000000061 2023-05-31T20:39:18.734 DEBUG:boto:auth_path=/pklxkd-58/obj00000000000000000061 2023-05-31T20:39:18.734 DEBUG:boto:Method: GET 2023-05-31T20:39:18.735 DEBUG:boto:Path: /pklxkd-58/obj00000000000000000061 2023-05-31T20:39:18.735 DEBUG:boto:Data: 2023-05-31T20:39:18.735 DEBUG:boto:Headers: {} 2023-05-31T20:39:18.736 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:39:18.736 DEBUG:boto:Port: 8001 2023-05-31T20:39:18.736 DEBUG:boto:Params: {} 2023-05-31T20:39:18.737 DEBUG:boto:Token: None 2023-05-31T20:39:18.737 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:39:18 GMT /pklxkd-58/obj00000000000000000061 2023-05-31T20:39:18.738 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:THMqRbbmb4ZnAAVMevK/Lbh5TIU= 2023-05-31T20:39:18.738 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:39:18 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:THMqRbbmb4ZnAAVMevK/Lbh5TIU=', 'Content-Length': '0'} 2023-05-31T20:39:18.740 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:38:21 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000096ad20e225e1cf12-006477b076-1c17-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:39:18 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:39:18.741 DEBUG:boto:path=/pklxkd-58/obj00000000000000000061 2023-05-31T20:39:18.741 DEBUG:boto:auth_path=/pklxkd-58/obj00000000000000000061 2023-05-31T20:39:18.742 DEBUG:boto:Method: GET 2023-05-31T20:39:18.742 DEBUG:boto:Path: /pklxkd-58/obj00000000000000000061 2023-05-31T20:39:18.742 DEBUG:boto:Data: 2023-05-31T20:39:18.743 DEBUG:boto:Headers: {} 2023-05-31T20:39:18.743 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:39:18.743 DEBUG:boto:Port: 8000 2023-05-31T20:39:18.744 DEBUG:boto:Params: {} 2023-05-31T20:39:18.744 DEBUG:boto:Token: None 2023-05-31T20:39:18.745 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:39:18 GMT /pklxkd-58/obj00000000000000000061 2023-05-31T20:39:18.745 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:THMqRbbmb4ZnAAVMevK/Lbh5TIU= 2023-05-31T20:39:18.746 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:39:18 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:THMqRbbmb4ZnAAVMevK/Lbh5TIU=', 'Content-Length': '0'} 2023-05-31T20:39:18.748 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:38:21 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008355c7595bd07cfb-006477b076-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:39:18 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:39:18.789 DEBUG:boto:path=/pklxkd-58/ 2023-05-31T20:39:18.790 DEBUG:boto:auth_path=/pklxkd-58/ 2023-05-31T20:39:18.790 DEBUG:boto:path=/pklxkd-58/?prefix=obj00000000000000000061 2023-05-31T20:39:18.790 DEBUG:boto:auth_path=/pklxkd-58/?prefix=obj00000000000000000061 2023-05-31T20:39:18.791 DEBUG:boto:Method: GET 2023-05-31T20:39:18.791 DEBUG:boto:Path: /pklxkd-58/?prefix=obj00000000000000000061 2023-05-31T20:39:18.791 DEBUG:boto:Data: 2023-05-31T20:39:18.792 DEBUG:boto:Headers: {} 2023-05-31T20:39:18.792 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:39:18.793 DEBUG:boto:Port: 8001 2023-05-31T20:39:18.793 DEBUG:boto:Params: {} 2023-05-31T20:39:18.794 DEBUG:boto:Token: None 2023-05-31T20:39:18.794 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:39:18 GMT /pklxkd-58/ 2023-05-31T20:39:18.795 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:3DG8wpCqcsv/yPsAzk0+okqkXZU= 2023-05-31T20:39:18.795 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:39:18 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:3DG8wpCqcsv/yPsAzk0+okqkXZU=', 'Content-Length': '0'} 2023-05-31T20:39:18.798 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000099bf13ca385ec148-006477b076-1c17-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:39:18 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:39:18.799 DEBUG:boto:b'pklxkd-58obj000000000000000000611000falseobj000000000000000000612023-05-31T20:38:21.154Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:39:18.799 DEBUG:boto:path=/pklxkd-58/ 2023-05-31T20:39:18.800 DEBUG:boto:auth_path=/pklxkd-58/ 2023-05-31T20:39:18.800 DEBUG:boto:path=/pklxkd-58/?prefix=obj00000000000000000061 2023-05-31T20:39:18.801 DEBUG:boto:auth_path=/pklxkd-58/?prefix=obj00000000000000000061 2023-05-31T20:39:18.801 DEBUG:boto:Method: GET 2023-05-31T20:39:18.801 DEBUG:boto:Path: /pklxkd-58/?prefix=obj00000000000000000061 2023-05-31T20:39:18.802 DEBUG:boto:Data: 2023-05-31T20:39:18.802 DEBUG:boto:Headers: {} 2023-05-31T20:39:18.802 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:39:18.803 DEBUG:boto:Port: 8000 2023-05-31T20:39:18.803 DEBUG:boto:Params: {} 2023-05-31T20:39:18.803 DEBUG:boto:Token: None 2023-05-31T20:39:18.804 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:39:18 GMT /pklxkd-58/ 2023-05-31T20:39:18.804 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:3DG8wpCqcsv/yPsAzk0+okqkXZU= 2023-05-31T20:39:18.805 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:39:18 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:3DG8wpCqcsv/yPsAzk0+okqkXZU=', 'Content-Length': '0'} 2023-05-31T20:39:18.809 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000000ffcdbd90397e9c6-006477b076-1630-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:39:18 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:39:18.810 DEBUG:boto:b'pklxkd-58obj000000000000000000611000falseobj000000000000000000612023-05-31T20:38:21.154Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:39:18.810 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000062 2023-05-31T20:39:18.811 DEBUG:boto:path=/pklxkd-58/obj00000000000000000062 2023-05-31T20:39:18.811 DEBUG:boto:auth_path=/pklxkd-58/obj00000000000000000062 2023-05-31T20:39:18.811 DEBUG:boto:path=/pklxkd-58/obj00000000000000000062?versionId=null 2023-05-31T20:39:18.812 DEBUG:boto:auth_path=/pklxkd-58/obj00000000000000000062?versionId=null 2023-05-31T20:39:18.812 DEBUG:boto:Method: GET 2023-05-31T20:39:18.813 DEBUG:boto:Path: /pklxkd-58/obj00000000000000000062?versionId=null 2023-05-31T20:39:18.813 DEBUG:boto:Data: 2023-05-31T20:39:18.813 DEBUG:boto:Headers: {} 2023-05-31T20:39:18.814 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:39:18.814 DEBUG:boto:Port: 8001 2023-05-31T20:39:18.814 DEBUG:boto:Params: {} 2023-05-31T20:39:18.815 DEBUG:boto:Token: None 2023-05-31T20:39:18.815 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:39:18 GMT /pklxkd-58/obj00000000000000000062?versionId=null 2023-05-31T20:39:18.816 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:lUjg8xTlojX73QilvoDgwzqTxuQ= 2023-05-31T20:39:18.816 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:39:18 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:lUjg8xTlojX73QilvoDgwzqTxuQ=', 'Content-Length': '0'} 2023-05-31T20:39:18.819 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:38:21 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001b5a6b73ad1fc93d-006477b076-1c17-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:39:18 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:39:18.864 DEBUG:boto:path=/pklxkd-58/obj00000000000000000062 2023-05-31T20:39:18.865 DEBUG:boto:auth_path=/pklxkd-58/obj00000000000000000062 2023-05-31T20:39:18.865 DEBUG:boto:path=/pklxkd-58/obj00000000000000000062?versionId=null 2023-05-31T20:39:18.866 DEBUG:boto:auth_path=/pklxkd-58/obj00000000000000000062?versionId=null 2023-05-31T20:39:18.866 DEBUG:boto:Method: GET 2023-05-31T20:39:18.866 DEBUG:boto:Path: /pklxkd-58/obj00000000000000000062?versionId=null 2023-05-31T20:39:18.867 DEBUG:boto:Data: 2023-05-31T20:39:18.867 DEBUG:boto:Headers: {} 2023-05-31T20:39:18.867 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:39:18.868 DEBUG:boto:Port: 8000 2023-05-31T20:39:18.868 DEBUG:boto:Params: {} 2023-05-31T20:39:18.869 DEBUG:boto:Token: None 2023-05-31T20:39:18.869 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:39:18 GMT /pklxkd-58/obj00000000000000000062?versionId=null 2023-05-31T20:39:18.870 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:lUjg8xTlojX73QilvoDgwzqTxuQ= 2023-05-31T20:39:18.871 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:39:18 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:lUjg8xTlojX73QilvoDgwzqTxuQ=', 'Content-Length': '0'} 2023-05-31T20:39:18.873 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:38:21 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000fc05c17ecbb1c05b-006477b076-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:39:18 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:39:18.875 DEBUG:boto:path=/pklxkd-58/obj00000000000000000062 2023-05-31T20:39:18.875 DEBUG:boto:auth_path=/pklxkd-58/obj00000000000000000062 2023-05-31T20:39:18.876 DEBUG:boto:path=/pklxkd-58/obj00000000000000000062?versionId=null 2023-05-31T20:39:18.876 DEBUG:boto:auth_path=/pklxkd-58/obj00000000000000000062?versionId=null 2023-05-31T20:39:18.877 DEBUG:boto:Method: HEAD 2023-05-31T20:39:18.877 DEBUG:boto:Path: /pklxkd-58/obj00000000000000000062?versionId=null 2023-05-31T20:39:18.878 DEBUG:boto:Data: 2023-05-31T20:39:18.878 DEBUG:boto:Headers: {} 2023-05-31T20:39:18.878 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:39:18.879 DEBUG:boto:Port: 8001 2023-05-31T20:39:18.879 DEBUG:boto:Params: {} 2023-05-31T20:39:18.879 DEBUG:boto:Token: None 2023-05-31T20:39:18.880 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:39:18 GMT /pklxkd-58/obj00000000000000000062?versionId=null 2023-05-31T20:39:18.880 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:J7XRSwXbZfRcnqVW2ACTPW52TCE= 2023-05-31T20:39:18.881 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:39:18 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:J7XRSwXbZfRcnqVW2ACTPW52TCE=', 'Content-Length': '0'} 2023-05-31T20:39:18.882 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:38:21 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000012077c1e8aadd075-006477b076-1c17-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:39:18 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:39:18.883 DEBUG:boto:path=/pklxkd-58/obj00000000000000000062 2023-05-31T20:39:18.883 DEBUG:boto:auth_path=/pklxkd-58/obj00000000000000000062 2023-05-31T20:39:18.884 DEBUG:boto:path=/pklxkd-58/obj00000000000000000062?versionId=null 2023-05-31T20:39:18.884 DEBUG:boto:auth_path=/pklxkd-58/obj00000000000000000062?versionId=null 2023-05-31T20:39:18.885 DEBUG:boto:Method: HEAD 2023-05-31T20:39:18.885 DEBUG:boto:Path: /pklxkd-58/obj00000000000000000062?versionId=null 2023-05-31T20:39:18.885 DEBUG:boto:Data: 2023-05-31T20:39:18.886 DEBUG:boto:Headers: {} 2023-05-31T20:39:18.886 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:39:18.886 DEBUG:boto:Port: 8000 2023-05-31T20:39:18.887 DEBUG:boto:Params: {} 2023-05-31T20:39:18.887 DEBUG:boto:Token: None 2023-05-31T20:39:18.888 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:39:18 GMT /pklxkd-58/obj00000000000000000062?versionId=null 2023-05-31T20:39:18.888 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:J7XRSwXbZfRcnqVW2ACTPW52TCE= 2023-05-31T20:39:18.888 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:39:18 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:J7XRSwXbZfRcnqVW2ACTPW52TCE=', 'Content-Length': '0'} 2023-05-31T20:39:18.891 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:38:21 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000032070c841171e0c4-006477b076-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:39:18 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:39:18.892 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000062 2023-05-31T20:39:18.892 DEBUG:boto:path=/pklxkd-58/obj00000000000000000062 2023-05-31T20:39:18.893 DEBUG:boto:auth_path=/pklxkd-58/obj00000000000000000062 2023-05-31T20:39:18.893 DEBUG:boto:path=/pklxkd-58/obj00000000000000000062?versionId=null 2023-05-31T20:39:18.894 DEBUG:boto:auth_path=/pklxkd-58/obj00000000000000000062?versionId=null 2023-05-31T20:39:18.896 DEBUG:boto:Method: GET 2023-05-31T20:39:18.897 DEBUG:boto:Path: /pklxkd-58/obj00000000000000000062?versionId=null 2023-05-31T20:39:18.897 DEBUG:boto:Data: 2023-05-31T20:39:18.898 DEBUG:boto:Headers: {} 2023-05-31T20:39:18.898 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:39:18.899 DEBUG:boto:Port: 8001 2023-05-31T20:39:18.899 DEBUG:boto:Params: {} 2023-05-31T20:39:18.900 DEBUG:boto:Token: None 2023-05-31T20:39:18.900 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:39:18 GMT /pklxkd-58/obj00000000000000000062?versionId=null 2023-05-31T20:39:18.901 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:lUjg8xTlojX73QilvoDgwzqTxuQ= 2023-05-31T20:39:18.902 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:39:18 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:lUjg8xTlojX73QilvoDgwzqTxuQ=', 'Content-Length': '0'} 2023-05-31T20:39:18.904 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:38:21 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ae547e8e2131ff0d-006477b076-1c17-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:39:18 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:39:18.905 DEBUG:boto:path=/pklxkd-58/obj00000000000000000062 2023-05-31T20:39:18.906 DEBUG:boto:auth_path=/pklxkd-58/obj00000000000000000062 2023-05-31T20:39:18.906 DEBUG:boto:path=/pklxkd-58/obj00000000000000000062?versionId=null 2023-05-31T20:39:18.907 DEBUG:boto:auth_path=/pklxkd-58/obj00000000000000000062?versionId=null 2023-05-31T20:39:18.907 DEBUG:boto:Method: GET 2023-05-31T20:39:18.908 DEBUG:boto:Path: /pklxkd-58/obj00000000000000000062?versionId=null 2023-05-31T20:39:18.908 DEBUG:boto:Data: 2023-05-31T20:39:18.908 DEBUG:boto:Headers: {} 2023-05-31T20:39:18.909 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:39:18.909 DEBUG:boto:Port: 8000 2023-05-31T20:39:18.910 DEBUG:boto:Params: {} 2023-05-31T20:39:18.910 DEBUG:boto:Token: None 2023-05-31T20:39:18.911 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:39:18 GMT /pklxkd-58/obj00000000000000000062?versionId=null 2023-05-31T20:39:18.911 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:lUjg8xTlojX73QilvoDgwzqTxuQ= 2023-05-31T20:39:18.912 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:39:18 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:lUjg8xTlojX73QilvoDgwzqTxuQ=', 'Content-Length': '0'} 2023-05-31T20:39:18.915 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:38:21 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003451aea56bfbe54d-006477b076-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:39:18 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:39:18.917 DEBUG:boto:path=/pklxkd-58/ 2023-05-31T20:39:18.918 DEBUG:boto:auth_path=/pklxkd-58/ 2023-05-31T20:39:18.919 DEBUG:boto:path=/pklxkd-58/?prefix=obj00000000000000000062 2023-05-31T20:39:18.920 DEBUG:boto:auth_path=/pklxkd-58/?prefix=obj00000000000000000062 2023-05-31T20:39:18.921 DEBUG:boto:Method: GET 2023-05-31T20:39:18.922 DEBUG:boto:Path: /pklxkd-58/?prefix=obj00000000000000000062 2023-05-31T20:39:18.922 DEBUG:boto:Data: 2023-05-31T20:39:18.923 DEBUG:boto:Headers: {} 2023-05-31T20:39:18.923 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:39:18.923 DEBUG:boto:Port: 8001 2023-05-31T20:39:18.924 DEBUG:boto:Params: {} 2023-05-31T20:39:18.924 DEBUG:boto:Token: None 2023-05-31T20:39:18.925 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:39:18 GMT /pklxkd-58/ 2023-05-31T20:39:18.926 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:3DG8wpCqcsv/yPsAzk0+okqkXZU= 2023-05-31T20:39:18.926 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:39:18 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:3DG8wpCqcsv/yPsAzk0+okqkXZU=', 'Content-Length': '0'} 2023-05-31T20:39:18.929 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d9a33bf070ded4bf-006477b076-1c17-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:39:18 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:39:18.929 DEBUG:boto:b'pklxkd-58obj000000000000000000621000falseobj000000000000000000622023-05-31T20:38:21.794Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:39:18.930 DEBUG:boto:path=/pklxkd-58/ 2023-05-31T20:39:18.931 DEBUG:boto:auth_path=/pklxkd-58/ 2023-05-31T20:39:18.932 DEBUG:boto:path=/pklxkd-58/?prefix=obj00000000000000000062 2023-05-31T20:39:18.932 DEBUG:boto:auth_path=/pklxkd-58/?prefix=obj00000000000000000062 2023-05-31T20:39:18.933 DEBUG:boto:Method: GET 2023-05-31T20:39:18.933 DEBUG:boto:Path: /pklxkd-58/?prefix=obj00000000000000000062 2023-05-31T20:39:18.933 DEBUG:boto:Data: 2023-05-31T20:39:18.934 DEBUG:boto:Headers: {} 2023-05-31T20:39:18.935 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:39:18.936 DEBUG:boto:Port: 8000 2023-05-31T20:39:18.937 DEBUG:boto:Params: {} 2023-05-31T20:39:18.938 DEBUG:boto:Token: None 2023-05-31T20:39:18.939 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:39:18 GMT /pklxkd-58/ 2023-05-31T20:39:18.940 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:3DG8wpCqcsv/yPsAzk0+okqkXZU= 2023-05-31T20:39:18.941 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:39:18 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:3DG8wpCqcsv/yPsAzk0+okqkXZU=', 'Content-Length': '0'} 2023-05-31T20:39:18.945 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000be53b339d8eb4d2e-006477b076-1630-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:39:18 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:39:18.946 DEBUG:boto:b'pklxkd-58obj000000000000000000621000falseobj000000000000000000622023-05-31T20:38:21.794Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:39:18.947 DEBUG:boto:path=/pklxkd-58/obj00000000000000000062 2023-05-31T20:39:18.948 DEBUG:boto:auth_path=/pklxkd-58/obj00000000000000000062 2023-05-31T20:39:18.948 DEBUG:boto:Method: HEAD 2023-05-31T20:39:18.949 DEBUG:boto:Path: /pklxkd-58/obj00000000000000000062 2023-05-31T20:39:18.949 DEBUG:boto:Data: 2023-05-31T20:39:18.949 DEBUG:boto:Headers: {} 2023-05-31T20:39:18.950 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:39:18.950 DEBUG:boto:Port: 8001 2023-05-31T20:39:18.951 DEBUG:boto:Params: {} 2023-05-31T20:39:18.951 DEBUG:boto:Token: None 2023-05-31T20:39:18.952 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:39:18 GMT /pklxkd-58/obj00000000000000000062 2023-05-31T20:39:18.953 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:3wxzshmxGqpRj0leN0s9lfqdq2M= 2023-05-31T20:39:18.953 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:39:18 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:3wxzshmxGqpRj0leN0s9lfqdq2M=', 'Content-Length': '0'} 2023-05-31T20:39:18.955 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:38:21 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002f4f6239cc36c697-006477b076-1c17-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:39:18 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:39:18.956 DEBUG:boto:path=/pklxkd-58/obj00000000000000000062 2023-05-31T20:39:18.957 DEBUG:boto:auth_path=/pklxkd-58/obj00000000000000000062 2023-05-31T20:39:18.957 DEBUG:boto:Method: HEAD 2023-05-31T20:39:18.958 DEBUG:boto:Path: /pklxkd-58/obj00000000000000000062 2023-05-31T20:39:18.958 DEBUG:boto:Data: 2023-05-31T20:39:18.958 DEBUG:boto:Headers: {} 2023-05-31T20:39:18.959 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:39:18.959 DEBUG:boto:Port: 8000 2023-05-31T20:39:18.960 DEBUG:boto:Params: {} 2023-05-31T20:39:18.960 DEBUG:boto:Token: None 2023-05-31T20:39:18.961 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:39:18 GMT /pklxkd-58/obj00000000000000000062 2023-05-31T20:39:18.961 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:3wxzshmxGqpRj0leN0s9lfqdq2M= 2023-05-31T20:39:18.961 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:39:18 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:3wxzshmxGqpRj0leN0s9lfqdq2M=', 'Content-Length': '0'} 2023-05-31T20:39:18.964 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:38:21 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005582b7881bb5039e-006477b076-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:39:18 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:39:18.965 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000062 2023-05-31T20:39:18.965 DEBUG:boto:path=/pklxkd-58/obj00000000000000000062 2023-05-31T20:39:18.966 DEBUG:boto:auth_path=/pklxkd-58/obj00000000000000000062 2023-05-31T20:39:18.966 DEBUG:boto:Method: GET 2023-05-31T20:39:18.967 DEBUG:boto:Path: /pklxkd-58/obj00000000000000000062 2023-05-31T20:39:18.967 DEBUG:boto:Data: 2023-05-31T20:39:18.968 DEBUG:boto:Headers: {} 2023-05-31T20:39:18.968 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:39:18.969 DEBUG:boto:Port: 8001 2023-05-31T20:39:18.969 DEBUG:boto:Params: {} 2023-05-31T20:39:18.970 DEBUG:boto:Token: None 2023-05-31T20:39:18.970 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:39:18 GMT /pklxkd-58/obj00000000000000000062 2023-05-31T20:39:18.971 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:HkWzoDbu33/1uEMZ7OZaXCYstCw= 2023-05-31T20:39:18.972 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:39:18 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:HkWzoDbu33/1uEMZ7OZaXCYstCw=', 'Content-Length': '0'} 2023-05-31T20:39:18.975 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:38:21 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007a61e6eff97b5726-006477b076-1c17-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:39:18 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:39:19.021 DEBUG:boto:path=/pklxkd-58/obj00000000000000000062 2023-05-31T20:39:19.022 DEBUG:boto:auth_path=/pklxkd-58/obj00000000000000000062 2023-05-31T20:39:19.022 DEBUG:boto:Method: GET 2023-05-31T20:39:19.023 DEBUG:boto:Path: /pklxkd-58/obj00000000000000000062 2023-05-31T20:39:19.024 DEBUG:boto:Data: 2023-05-31T20:39:19.025 DEBUG:boto:Headers: {} 2023-05-31T20:39:19.025 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:39:19.026 DEBUG:boto:Port: 8000 2023-05-31T20:39:19.026 DEBUG:boto:Params: {} 2023-05-31T20:39:19.027 DEBUG:boto:Token: None 2023-05-31T20:39:19.028 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:39:19 GMT /pklxkd-58/obj00000000000000000062 2023-05-31T20:39:19.028 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:Xpp+Ru1Oq1RqqaWyRHDUY26Y6vw= 2023-05-31T20:39:19.029 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:39:19 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:Xpp+Ru1Oq1RqqaWyRHDUY26Y6vw=', 'Content-Length': '0'} 2023-05-31T20:39:19.031 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:38:21 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000084b9e68cfeb91f71-006477b077-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:39:19 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:39:19.032 DEBUG:boto:path=/pklxkd-58/ 2023-05-31T20:39:19.032 DEBUG:boto:auth_path=/pklxkd-58/ 2023-05-31T20:39:19.032 DEBUG:boto:path=/pklxkd-58/?prefix=obj00000000000000000062 2023-05-31T20:39:19.033 DEBUG:boto:auth_path=/pklxkd-58/?prefix=obj00000000000000000062 2023-05-31T20:39:19.033 DEBUG:boto:Method: GET 2023-05-31T20:39:19.034 DEBUG:boto:Path: /pklxkd-58/?prefix=obj00000000000000000062 2023-05-31T20:39:19.034 DEBUG:boto:Data: 2023-05-31T20:39:19.034 DEBUG:boto:Headers: {} 2023-05-31T20:39:19.035 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:39:19.035 DEBUG:boto:Port: 8001 2023-05-31T20:39:19.035 DEBUG:boto:Params: {} 2023-05-31T20:39:19.036 DEBUG:boto:Token: None 2023-05-31T20:39:19.036 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:39:19 GMT /pklxkd-58/ 2023-05-31T20:39:19.037 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:WOF0ykFTgI6McLQyUlzyKPU4swY= 2023-05-31T20:39:19.037 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:39:19 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:WOF0ykFTgI6McLQyUlzyKPU4swY=', 'Content-Length': '0'} 2023-05-31T20:39:19.040 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000006b566e7b418696e5-006477b077-1c17-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:39:19 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:39:19.040 DEBUG:boto:b'pklxkd-58obj000000000000000000621000falseobj000000000000000000622023-05-31T20:38:21.794Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:39:19.041 DEBUG:boto:path=/pklxkd-58/ 2023-05-31T20:39:19.042 DEBUG:boto:auth_path=/pklxkd-58/ 2023-05-31T20:39:19.042 DEBUG:boto:path=/pklxkd-58/?prefix=obj00000000000000000062 2023-05-31T20:39:19.042 DEBUG:boto:auth_path=/pklxkd-58/?prefix=obj00000000000000000062 2023-05-31T20:39:19.043 DEBUG:boto:Method: GET 2023-05-31T20:39:19.043 DEBUG:boto:Path: /pklxkd-58/?prefix=obj00000000000000000062 2023-05-31T20:39:19.043 DEBUG:boto:Data: 2023-05-31T20:39:19.044 DEBUG:boto:Headers: {} 2023-05-31T20:39:19.044 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:39:19.044 DEBUG:boto:Port: 8000 2023-05-31T20:39:19.045 DEBUG:boto:Params: {} 2023-05-31T20:39:19.045 DEBUG:boto:Token: None 2023-05-31T20:39:19.046 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:39:19 GMT /pklxkd-58/ 2023-05-31T20:39:19.046 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:WOF0ykFTgI6McLQyUlzyKPU4swY= 2023-05-31T20:39:19.046 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:39:19 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:WOF0ykFTgI6McLQyUlzyKPU4swY=', 'Content-Length': '0'} 2023-05-31T20:39:19.051 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000013b4b6142b6933a4-006477b077-1630-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:39:19 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:39:19.051 DEBUG:boto:b'pklxkd-58obj000000000000000000621000falseobj000000000000000000622023-05-31T20:38:21.794Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:39:19.052 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000065 2023-05-31T20:39:19.052 DEBUG:boto:path=/pklxkd-58/obj00000000000000000065 2023-05-31T20:39:19.053 DEBUG:boto:auth_path=/pklxkd-58/obj00000000000000000065 2023-05-31T20:39:19.053 DEBUG:boto:path=/pklxkd-58/obj00000000000000000065?versionId=null 2023-05-31T20:39:19.054 DEBUG:boto:auth_path=/pklxkd-58/obj00000000000000000065?versionId=null 2023-05-31T20:39:19.054 DEBUG:boto:Method: GET 2023-05-31T20:39:19.054 DEBUG:boto:Path: /pklxkd-58/obj00000000000000000065?versionId=null 2023-05-31T20:39:19.055 DEBUG:boto:Data: 2023-05-31T20:39:19.055 DEBUG:boto:Headers: {} 2023-05-31T20:39:19.055 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:39:19.056 DEBUG:boto:Port: 8001 2023-05-31T20:39:19.056 DEBUG:boto:Params: {} 2023-05-31T20:39:19.057 DEBUG:boto:Token: None 2023-05-31T20:39:19.057 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:39:19 GMT /pklxkd-58/obj00000000000000000065?versionId=null 2023-05-31T20:39:19.057 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:PlRmugVyQ0ksUisOb3AA7zb1TLI= 2023-05-31T20:39:19.058 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:39:19 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:PlRmugVyQ0ksUisOb3AA7zb1TLI=', 'Content-Length': '0'} 2023-05-31T20:39:19.060 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:38:22 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005230d81bc0b626ff-006477b077-1c17-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:39:19 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:39:19.061 DEBUG:boto:path=/pklxkd-58/obj00000000000000000065 2023-05-31T20:39:19.061 DEBUG:boto:auth_path=/pklxkd-58/obj00000000000000000065 2023-05-31T20:39:19.062 DEBUG:boto:path=/pklxkd-58/obj00000000000000000065?versionId=null 2023-05-31T20:39:19.062 DEBUG:boto:auth_path=/pklxkd-58/obj00000000000000000065?versionId=null 2023-05-31T20:39:19.063 DEBUG:boto:Method: GET 2023-05-31T20:39:19.063 DEBUG:boto:Path: /pklxkd-58/obj00000000000000000065?versionId=null 2023-05-31T20:39:19.063 DEBUG:boto:Data: 2023-05-31T20:39:19.064 DEBUG:boto:Headers: {} 2023-05-31T20:39:19.064 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:39:19.064 DEBUG:boto:Port: 8000 2023-05-31T20:39:19.065 DEBUG:boto:Params: {} 2023-05-31T20:39:19.065 DEBUG:boto:Token: None 2023-05-31T20:39:19.066 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:39:19 GMT /pklxkd-58/obj00000000000000000065?versionId=null 2023-05-31T20:39:19.066 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:PlRmugVyQ0ksUisOb3AA7zb1TLI= 2023-05-31T20:39:19.066 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:39:19 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:PlRmugVyQ0ksUisOb3AA7zb1TLI=', 'Content-Length': '0'} 2023-05-31T20:39:19.068 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:38:22 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004303dba85885c1e5-006477b077-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:39:19 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:39:19.069 DEBUG:boto:path=/pklxkd-58/obj00000000000000000065 2023-05-31T20:39:19.070 DEBUG:boto:auth_path=/pklxkd-58/obj00000000000000000065 2023-05-31T20:39:19.070 DEBUG:boto:path=/pklxkd-58/obj00000000000000000065?versionId=null 2023-05-31T20:39:19.070 DEBUG:boto:auth_path=/pklxkd-58/obj00000000000000000065?versionId=null 2023-05-31T20:39:19.071 DEBUG:boto:Method: HEAD 2023-05-31T20:39:19.071 DEBUG:boto:Path: /pklxkd-58/obj00000000000000000065?versionId=null 2023-05-31T20:39:19.071 DEBUG:boto:Data: 2023-05-31T20:39:19.072 DEBUG:boto:Headers: {} 2023-05-31T20:39:19.072 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:39:19.073 DEBUG:boto:Port: 8001 2023-05-31T20:39:19.073 DEBUG:boto:Params: {} 2023-05-31T20:39:19.073 DEBUG:boto:Token: None 2023-05-31T20:39:19.074 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:39:19 GMT /pklxkd-58/obj00000000000000000065?versionId=null 2023-05-31T20:39:19.074 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:E4nH3uViFjv21p5EqCMX4RQBuV0= 2023-05-31T20:39:19.075 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:39:19 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:E4nH3uViFjv21p5EqCMX4RQBuV0=', 'Content-Length': '0'} 2023-05-31T20:39:19.077 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:38:22 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000029122397a7757db7-006477b077-1c17-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:39:19 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:39:19.077 DEBUG:boto:path=/pklxkd-58/obj00000000000000000065 2023-05-31T20:39:19.078 DEBUG:boto:auth_path=/pklxkd-58/obj00000000000000000065 2023-05-31T20:39:19.078 DEBUG:boto:path=/pklxkd-58/obj00000000000000000065?versionId=null 2023-05-31T20:39:19.079 DEBUG:boto:auth_path=/pklxkd-58/obj00000000000000000065?versionId=null 2023-05-31T20:39:19.079 DEBUG:boto:Method: HEAD 2023-05-31T20:39:19.079 DEBUG:boto:Path: /pklxkd-58/obj00000000000000000065?versionId=null 2023-05-31T20:39:19.080 DEBUG:boto:Data: 2023-05-31T20:39:19.080 DEBUG:boto:Headers: {} 2023-05-31T20:39:19.080 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:39:19.081 DEBUG:boto:Port: 8000 2023-05-31T20:39:19.081 DEBUG:boto:Params: {} 2023-05-31T20:39:19.081 DEBUG:boto:Token: None 2023-05-31T20:39:19.082 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:39:19 GMT /pklxkd-58/obj00000000000000000065?versionId=null 2023-05-31T20:39:19.082 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:E4nH3uViFjv21p5EqCMX4RQBuV0= 2023-05-31T20:39:19.083 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:39:19 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:E4nH3uViFjv21p5EqCMX4RQBuV0=', 'Content-Length': '0'} 2023-05-31T20:39:19.085 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:38:22 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003a4548d34c4c06b4-006477b077-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:39:19 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:39:19.085 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000065 2023-05-31T20:39:19.086 DEBUG:boto:path=/pklxkd-58/obj00000000000000000065 2023-05-31T20:39:19.086 DEBUG:boto:auth_path=/pklxkd-58/obj00000000000000000065 2023-05-31T20:39:19.086 DEBUG:boto:path=/pklxkd-58/obj00000000000000000065?versionId=null 2023-05-31T20:39:19.087 DEBUG:boto:auth_path=/pklxkd-58/obj00000000000000000065?versionId=null 2023-05-31T20:39:19.087 DEBUG:boto:Method: GET 2023-05-31T20:39:19.087 DEBUG:boto:Path: /pklxkd-58/obj00000000000000000065?versionId=null 2023-05-31T20:39:19.088 DEBUG:boto:Data: 2023-05-31T20:39:19.088 DEBUG:boto:Headers: {} 2023-05-31T20:39:19.089 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:39:19.089 DEBUG:boto:Port: 8001 2023-05-31T20:39:19.089 DEBUG:boto:Params: {} 2023-05-31T20:39:19.090 DEBUG:boto:Token: None 2023-05-31T20:39:19.090 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:39:19 GMT /pklxkd-58/obj00000000000000000065?versionId=null 2023-05-31T20:39:19.091 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:PlRmugVyQ0ksUisOb3AA7zb1TLI= 2023-05-31T20:39:19.091 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:39:19 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:PlRmugVyQ0ksUisOb3AA7zb1TLI=', 'Content-Length': '0'} 2023-05-31T20:39:19.093 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:38:22 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ec529ffb17c37418-006477b077-1c17-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:39:19 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:39:19.094 DEBUG:boto:path=/pklxkd-58/obj00000000000000000065 2023-05-31T20:39:19.094 DEBUG:boto:auth_path=/pklxkd-58/obj00000000000000000065 2023-05-31T20:39:19.095 DEBUG:boto:path=/pklxkd-58/obj00000000000000000065?versionId=null 2023-05-31T20:39:19.095 DEBUG:boto:auth_path=/pklxkd-58/obj00000000000000000065?versionId=null 2023-05-31T20:39:19.095 DEBUG:boto:Method: GET 2023-05-31T20:39:19.096 DEBUG:boto:Path: /pklxkd-58/obj00000000000000000065?versionId=null 2023-05-31T20:39:19.096 DEBUG:boto:Data: 2023-05-31T20:39:19.097 DEBUG:boto:Headers: {} 2023-05-31T20:39:19.097 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:39:19.097 DEBUG:boto:Port: 8000 2023-05-31T20:39:19.098 DEBUG:boto:Params: {} 2023-05-31T20:39:19.098 DEBUG:boto:Token: None 2023-05-31T20:39:19.098 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:39:19 GMT /pklxkd-58/obj00000000000000000065?versionId=null 2023-05-31T20:39:19.099 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:PlRmugVyQ0ksUisOb3AA7zb1TLI= 2023-05-31T20:39:19.099 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:39:19 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:PlRmugVyQ0ksUisOb3AA7zb1TLI=', 'Content-Length': '0'} 2023-05-31T20:39:19.101 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:38:22 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d00403a72f0ae921-006477b077-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:39:19 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:39:19.148 DEBUG:boto:path=/pklxkd-58/ 2023-05-31T20:39:19.149 DEBUG:boto:auth_path=/pklxkd-58/ 2023-05-31T20:39:19.149 DEBUG:boto:path=/pklxkd-58/?prefix=obj00000000000000000065 2023-05-31T20:39:19.150 DEBUG:boto:auth_path=/pklxkd-58/?prefix=obj00000000000000000065 2023-05-31T20:39:19.150 DEBUG:boto:Method: GET 2023-05-31T20:39:19.151 DEBUG:boto:Path: /pklxkd-58/?prefix=obj00000000000000000065 2023-05-31T20:39:19.151 DEBUG:boto:Data: 2023-05-31T20:39:19.152 DEBUG:boto:Headers: {} 2023-05-31T20:39:19.152 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:39:19.153 DEBUG:boto:Port: 8001 2023-05-31T20:39:19.156 DEBUG:boto:Params: {} 2023-05-31T20:39:19.157 DEBUG:boto:Token: None 2023-05-31T20:39:19.157 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:39:19 GMT /pklxkd-58/ 2023-05-31T20:39:19.158 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:WOF0ykFTgI6McLQyUlzyKPU4swY= 2023-05-31T20:39:19.158 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:39:19 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:WOF0ykFTgI6McLQyUlzyKPU4swY=', 'Content-Length': '0'} 2023-05-31T20:39:19.162 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a7a058b181bf30f0-006477b077-1c17-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:39:19 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:39:19.162 DEBUG:boto:b'pklxkd-58obj000000000000000000651000falseobj000000000000000000652023-05-31T20:38:22.003Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:39:19.163 DEBUG:boto:path=/pklxkd-58/ 2023-05-31T20:39:19.163 DEBUG:boto:auth_path=/pklxkd-58/ 2023-05-31T20:39:19.164 DEBUG:boto:path=/pklxkd-58/?prefix=obj00000000000000000065 2023-05-31T20:39:19.164 DEBUG:boto:auth_path=/pklxkd-58/?prefix=obj00000000000000000065 2023-05-31T20:39:19.165 DEBUG:boto:Method: GET 2023-05-31T20:39:19.165 DEBUG:boto:Path: /pklxkd-58/?prefix=obj00000000000000000065 2023-05-31T20:39:19.166 DEBUG:boto:Data: 2023-05-31T20:39:19.166 DEBUG:boto:Headers: {} 2023-05-31T20:39:19.167 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:39:19.167 DEBUG:boto:Port: 8000 2023-05-31T20:39:19.167 DEBUG:boto:Params: {} 2023-05-31T20:39:19.168 DEBUG:boto:Token: None 2023-05-31T20:39:19.169 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:39:19 GMT /pklxkd-58/ 2023-05-31T20:39:19.169 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:WOF0ykFTgI6McLQyUlzyKPU4swY= 2023-05-31T20:39:19.170 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:39:19 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:WOF0ykFTgI6McLQyUlzyKPU4swY=', 'Content-Length': '0'} 2023-05-31T20:39:19.175 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000004a130ef82a2c8e43-006477b077-1630-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:39:19 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:39:19.176 DEBUG:boto:b'pklxkd-58obj000000000000000000651000falseobj000000000000000000652023-05-31T20:38:22.003Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:39:19.176 DEBUG:boto:path=/pklxkd-58/obj00000000000000000065 2023-05-31T20:39:19.177 DEBUG:boto:auth_path=/pklxkd-58/obj00000000000000000065 2023-05-31T20:39:19.177 DEBUG:boto:Method: HEAD 2023-05-31T20:39:19.178 DEBUG:boto:Path: /pklxkd-58/obj00000000000000000065 2023-05-31T20:39:19.178 DEBUG:boto:Data: 2023-05-31T20:39:19.179 DEBUG:boto:Headers: {} 2023-05-31T20:39:19.179 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:39:19.179 DEBUG:boto:Port: 8001 2023-05-31T20:39:19.180 DEBUG:boto:Params: {} 2023-05-31T20:39:19.180 DEBUG:boto:Token: None 2023-05-31T20:39:19.181 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:39:19 GMT /pklxkd-58/obj00000000000000000065 2023-05-31T20:39:19.182 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:CAu4MkxOFQSpCpqk9MFAahVg8Dw= 2023-05-31T20:39:19.182 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:39:19 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:CAu4MkxOFQSpCpqk9MFAahVg8Dw=', 'Content-Length': '0'} 2023-05-31T20:39:19.185 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:38:22 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a236428827bd86e3-006477b077-1c17-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:39:19 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:39:19.186 DEBUG:boto:path=/pklxkd-58/obj00000000000000000065 2023-05-31T20:39:19.186 DEBUG:boto:auth_path=/pklxkd-58/obj00000000000000000065 2023-05-31T20:39:19.187 DEBUG:boto:Method: HEAD 2023-05-31T20:39:19.187 DEBUG:boto:Path: /pklxkd-58/obj00000000000000000065 2023-05-31T20:39:19.188 DEBUG:boto:Data: 2023-05-31T20:39:19.189 DEBUG:boto:Headers: {} 2023-05-31T20:39:19.189 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:39:19.190 DEBUG:boto:Port: 8000 2023-05-31T20:39:19.191 DEBUG:boto:Params: {} 2023-05-31T20:39:19.192 DEBUG:boto:Token: None 2023-05-31T20:39:19.193 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:39:19 GMT /pklxkd-58/obj00000000000000000065 2023-05-31T20:39:19.194 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:CAu4MkxOFQSpCpqk9MFAahVg8Dw= 2023-05-31T20:39:19.194 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:39:19 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:CAu4MkxOFQSpCpqk9MFAahVg8Dw=', 'Content-Length': '0'} 2023-05-31T20:39:19.197 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:38:22 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000024988fa2259a13a9-006477b077-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:39:19 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:39:19.197 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000065 2023-05-31T20:39:19.198 DEBUG:boto:path=/pklxkd-58/obj00000000000000000065 2023-05-31T20:39:19.199 DEBUG:boto:auth_path=/pklxkd-58/obj00000000000000000065 2023-05-31T20:39:19.199 DEBUG:boto:Method: GET 2023-05-31T20:39:19.200 DEBUG:boto:Path: /pklxkd-58/obj00000000000000000065 2023-05-31T20:39:19.201 DEBUG:boto:Data: 2023-05-31T20:39:19.201 DEBUG:boto:Headers: {} 2023-05-31T20:39:19.202 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:39:19.203 DEBUG:boto:Port: 8001 2023-05-31T20:39:19.204 DEBUG:boto:Params: {} 2023-05-31T20:39:19.204 DEBUG:boto:Token: None 2023-05-31T20:39:19.205 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:39:19 GMT /pklxkd-58/obj00000000000000000065 2023-05-31T20:39:19.206 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:12UzVxyfZcdE9XbWhSWwTnFvFq4= 2023-05-31T20:39:19.207 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:39:19 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:12UzVxyfZcdE9XbWhSWwTnFvFq4=', 'Content-Length': '0'} 2023-05-31T20:39:19.209 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:38:22 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000bf9e9caf0e09eaf9-006477b077-1c17-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:39:19 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:39:19.256 DEBUG:boto:path=/pklxkd-58/obj00000000000000000065 2023-05-31T20:39:19.258 DEBUG:boto:auth_path=/pklxkd-58/obj00000000000000000065 2023-05-31T20:39:19.258 DEBUG:boto:Method: GET 2023-05-31T20:39:19.259 DEBUG:boto:Path: /pklxkd-58/obj00000000000000000065 2023-05-31T20:39:19.259 DEBUG:boto:Data: 2023-05-31T20:39:19.259 DEBUG:boto:Headers: {} 2023-05-31T20:39:19.260 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:39:19.260 DEBUG:boto:Port: 8000 2023-05-31T20:39:19.260 DEBUG:boto:Params: {} 2023-05-31T20:39:19.261 DEBUG:boto:Token: None 2023-05-31T20:39:19.261 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:39:19 GMT /pklxkd-58/obj00000000000000000065 2023-05-31T20:39:19.262 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:12UzVxyfZcdE9XbWhSWwTnFvFq4= 2023-05-31T20:39:19.262 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:39:19 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:12UzVxyfZcdE9XbWhSWwTnFvFq4=', 'Content-Length': '0'} 2023-05-31T20:39:19.264 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:38:22 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000db80136dfcfa596f-006477b077-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:39:19 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:39:19.265 DEBUG:boto:path=/pklxkd-58/ 2023-05-31T20:39:19.266 DEBUG:boto:auth_path=/pklxkd-58/ 2023-05-31T20:39:19.266 DEBUG:boto:path=/pklxkd-58/?prefix=obj00000000000000000065 2023-05-31T20:39:19.266 DEBUG:boto:auth_path=/pklxkd-58/?prefix=obj00000000000000000065 2023-05-31T20:39:19.267 DEBUG:boto:Method: GET 2023-05-31T20:39:19.267 DEBUG:boto:Path: /pklxkd-58/?prefix=obj00000000000000000065 2023-05-31T20:39:19.268 DEBUG:boto:Data: 2023-05-31T20:39:19.268 DEBUG:boto:Headers: {} 2023-05-31T20:39:19.269 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:39:19.269 DEBUG:boto:Port: 8001 2023-05-31T20:39:19.270 DEBUG:boto:Params: {} 2023-05-31T20:39:19.270 DEBUG:boto:Token: None 2023-05-31T20:39:19.270 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:39:19 GMT /pklxkd-58/ 2023-05-31T20:39:19.271 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:WOF0ykFTgI6McLQyUlzyKPU4swY= 2023-05-31T20:39:19.271 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:39:19 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:WOF0ykFTgI6McLQyUlzyKPU4swY=', 'Content-Length': '0'} 2023-05-31T20:39:19.274 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000007150f1bdad1b95c0-006477b077-1c17-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:39:19 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:39:19.275 DEBUG:boto:b'pklxkd-58obj000000000000000000651000falseobj000000000000000000652023-05-31T20:38:22.003Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:39:19.275 DEBUG:boto:path=/pklxkd-58/ 2023-05-31T20:39:19.276 DEBUG:boto:auth_path=/pklxkd-58/ 2023-05-31T20:39:19.276 DEBUG:boto:path=/pklxkd-58/?prefix=obj00000000000000000065 2023-05-31T20:39:19.277 DEBUG:boto:auth_path=/pklxkd-58/?prefix=obj00000000000000000065 2023-05-31T20:39:19.277 DEBUG:boto:Method: GET 2023-05-31T20:39:19.278 DEBUG:boto:Path: /pklxkd-58/?prefix=obj00000000000000000065 2023-05-31T20:39:19.278 DEBUG:boto:Data: 2023-05-31T20:39:19.278 DEBUG:boto:Headers: {} 2023-05-31T20:39:19.279 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:39:19.279 DEBUG:boto:Port: 8000 2023-05-31T20:39:19.280 DEBUG:boto:Params: {} 2023-05-31T20:39:19.280 DEBUG:boto:Token: None 2023-05-31T20:39:19.280 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:39:19 GMT /pklxkd-58/ 2023-05-31T20:39:19.281 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:WOF0ykFTgI6McLQyUlzyKPU4swY= 2023-05-31T20:39:19.281 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:39:19 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:WOF0ykFTgI6McLQyUlzyKPU4swY=', 'Content-Length': '0'} 2023-05-31T20:39:19.286 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000002579a5e995111bbb-006477b077-1630-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:39:19 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:39:19.332 DEBUG:boto:b'pklxkd-58obj000000000000000000651000falseobj000000000000000000652023-05-31T20:38:22.003Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:39:19.333 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000133 2023-05-31T20:39:19.334 DEBUG:boto:path=/pklxkd-58/obj00000000000000000133 2023-05-31T20:39:19.334 DEBUG:boto:auth_path=/pklxkd-58/obj00000000000000000133 2023-05-31T20:39:19.334 DEBUG:boto:path=/pklxkd-58/obj00000000000000000133?versionId=null 2023-05-31T20:39:19.335 DEBUG:boto:auth_path=/pklxkd-58/obj00000000000000000133?versionId=null 2023-05-31T20:39:19.335 DEBUG:boto:Method: GET 2023-05-31T20:39:19.335 DEBUG:boto:Path: /pklxkd-58/obj00000000000000000133?versionId=null 2023-05-31T20:39:19.336 DEBUG:boto:Data: 2023-05-31T20:39:19.336 DEBUG:boto:Headers: {} 2023-05-31T20:39:19.336 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:39:19.337 DEBUG:boto:Port: 8001 2023-05-31T20:39:19.337 DEBUG:boto:Params: {} 2023-05-31T20:39:19.338 DEBUG:boto:Token: None 2023-05-31T20:39:19.338 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:39:19 GMT /pklxkd-58/obj00000000000000000133?versionId=null 2023-05-31T20:39:19.338 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:xyUcOYYEiJZ4aW4GB3CzSuapdR4= 2023-05-31T20:39:19.339 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:39:19 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:xyUcOYYEiJZ4aW4GB3CzSuapdR4=', 'Content-Length': '0'} 2023-05-31T20:39:19.341 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:38:19 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009e1fda4e81edb3ed-006477b077-1c17-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:39:19 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:39:19.342 DEBUG:boto:path=/pklxkd-58/obj00000000000000000133 2023-05-31T20:39:19.343 DEBUG:boto:auth_path=/pklxkd-58/obj00000000000000000133 2023-05-31T20:39:19.343 DEBUG:boto:path=/pklxkd-58/obj00000000000000000133?versionId=null 2023-05-31T20:39:19.344 DEBUG:boto:auth_path=/pklxkd-58/obj00000000000000000133?versionId=null 2023-05-31T20:39:19.345 DEBUG:boto:Method: GET 2023-05-31T20:39:19.345 DEBUG:boto:Path: /pklxkd-58/obj00000000000000000133?versionId=null 2023-05-31T20:39:19.346 DEBUG:boto:Data: 2023-05-31T20:39:19.346 DEBUG:boto:Headers: {} 2023-05-31T20:39:19.347 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:39:19.347 DEBUG:boto:Port: 8000 2023-05-31T20:39:19.347 DEBUG:boto:Params: {} 2023-05-31T20:39:19.348 DEBUG:boto:Token: None 2023-05-31T20:39:19.348 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:39:19 GMT /pklxkd-58/obj00000000000000000133?versionId=null 2023-05-31T20:39:19.349 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:xyUcOYYEiJZ4aW4GB3CzSuapdR4= 2023-05-31T20:39:19.349 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:39:19 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:xyUcOYYEiJZ4aW4GB3CzSuapdR4=', 'Content-Length': '0'} 2023-05-31T20:39:19.352 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:38:19 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b689d32b4ae91580-006477b077-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:39:19 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:39:19.352 DEBUG:boto:path=/pklxkd-58/obj00000000000000000133 2023-05-31T20:39:19.353 DEBUG:boto:auth_path=/pklxkd-58/obj00000000000000000133 2023-05-31T20:39:19.354 DEBUG:boto:path=/pklxkd-58/obj00000000000000000133?versionId=null 2023-05-31T20:39:19.355 DEBUG:boto:auth_path=/pklxkd-58/obj00000000000000000133?versionId=null 2023-05-31T20:39:19.356 DEBUG:boto:Method: HEAD 2023-05-31T20:39:19.357 DEBUG:boto:Path: /pklxkd-58/obj00000000000000000133?versionId=null 2023-05-31T20:39:19.358 DEBUG:boto:Data: 2023-05-31T20:39:19.359 DEBUG:boto:Headers: {} 2023-05-31T20:39:19.359 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:39:19.360 DEBUG:boto:Port: 8001 2023-05-31T20:39:19.360 DEBUG:boto:Params: {} 2023-05-31T20:39:19.361 DEBUG:boto:Token: None 2023-05-31T20:39:19.361 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:39:19 GMT /pklxkd-58/obj00000000000000000133?versionId=null 2023-05-31T20:39:19.362 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:XNUDgPEDGT+b9LCMpnRcyUPBG2k= 2023-05-31T20:39:19.363 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:39:19 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:XNUDgPEDGT+b9LCMpnRcyUPBG2k=', 'Content-Length': '0'} 2023-05-31T20:39:19.365 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:38:19 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000064ebc8b653ded65e-006477b077-1c17-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:39:19 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:39:19.365 DEBUG:boto:path=/pklxkd-58/obj00000000000000000133 2023-05-31T20:39:19.366 DEBUG:boto:auth_path=/pklxkd-58/obj00000000000000000133 2023-05-31T20:39:19.366 DEBUG:boto:path=/pklxkd-58/obj00000000000000000133?versionId=null 2023-05-31T20:39:19.367 DEBUG:boto:auth_path=/pklxkd-58/obj00000000000000000133?versionId=null 2023-05-31T20:39:19.367 DEBUG:boto:Method: HEAD 2023-05-31T20:39:19.368 DEBUG:boto:Path: /pklxkd-58/obj00000000000000000133?versionId=null 2023-05-31T20:39:19.368 DEBUG:boto:Data: 2023-05-31T20:39:19.369 DEBUG:boto:Headers: {} 2023-05-31T20:39:19.369 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:39:19.370 DEBUG:boto:Port: 8000 2023-05-31T20:39:19.370 DEBUG:boto:Params: {} 2023-05-31T20:39:19.371 DEBUG:boto:Token: None 2023-05-31T20:39:19.371 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:39:19 GMT /pklxkd-58/obj00000000000000000133?versionId=null 2023-05-31T20:39:19.372 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:XNUDgPEDGT+b9LCMpnRcyUPBG2k= 2023-05-31T20:39:19.372 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:39:19 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:XNUDgPEDGT+b9LCMpnRcyUPBG2k=', 'Content-Length': '0'} 2023-05-31T20:39:19.375 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:38:19 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000083a12d6886d5ec44-006477b077-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:39:19 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:39:19.376 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000133 2023-05-31T20:39:19.377 DEBUG:boto:path=/pklxkd-58/obj00000000000000000133 2023-05-31T20:39:19.377 DEBUG:boto:auth_path=/pklxkd-58/obj00000000000000000133 2023-05-31T20:39:19.377 DEBUG:boto:path=/pklxkd-58/obj00000000000000000133?versionId=null 2023-05-31T20:39:19.378 DEBUG:boto:auth_path=/pklxkd-58/obj00000000000000000133?versionId=null 2023-05-31T20:39:19.380 DEBUG:boto:Method: GET 2023-05-31T20:39:19.380 DEBUG:boto:Path: /pklxkd-58/obj00000000000000000133?versionId=null 2023-05-31T20:39:19.383 DEBUG:boto:Data: 2023-05-31T20:39:19.383 DEBUG:boto:Headers: {} 2023-05-31T20:39:19.384 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:39:19.385 DEBUG:boto:Port: 8001 2023-05-31T20:39:19.385 DEBUG:boto:Params: {} 2023-05-31T20:39:19.385 DEBUG:boto:Token: None 2023-05-31T20:39:19.386 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:39:19 GMT /pklxkd-58/obj00000000000000000133?versionId=null 2023-05-31T20:39:19.387 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:xyUcOYYEiJZ4aW4GB3CzSuapdR4= 2023-05-31T20:39:19.388 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:39:19 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:xyUcOYYEiJZ4aW4GB3CzSuapdR4=', 'Content-Length': '0'} 2023-05-31T20:39:19.390 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:38:19 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002154ad91f74c460a-006477b077-1c17-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:39:19 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:39:19.432 DEBUG:boto:path=/pklxkd-58/obj00000000000000000133 2023-05-31T20:39:19.433 DEBUG:boto:auth_path=/pklxkd-58/obj00000000000000000133 2023-05-31T20:39:19.433 DEBUG:boto:path=/pklxkd-58/obj00000000000000000133?versionId=null 2023-05-31T20:39:19.434 DEBUG:boto:auth_path=/pklxkd-58/obj00000000000000000133?versionId=null 2023-05-31T20:39:19.434 DEBUG:boto:Method: GET 2023-05-31T20:39:19.434 DEBUG:boto:Path: /pklxkd-58/obj00000000000000000133?versionId=null 2023-05-31T20:39:19.435 DEBUG:boto:Data: 2023-05-31T20:39:19.435 DEBUG:boto:Headers: {} 2023-05-31T20:39:19.435 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:39:19.436 DEBUG:boto:Port: 8000 2023-05-31T20:39:19.436 DEBUG:boto:Params: {} 2023-05-31T20:39:19.437 DEBUG:boto:Token: None 2023-05-31T20:39:19.437 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:39:19 GMT /pklxkd-58/obj00000000000000000133?versionId=null 2023-05-31T20:39:19.438 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:xyUcOYYEiJZ4aW4GB3CzSuapdR4= 2023-05-31T20:39:19.438 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:39:19 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:xyUcOYYEiJZ4aW4GB3CzSuapdR4=', 'Content-Length': '0'} 2023-05-31T20:39:19.440 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:38:19 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e0353a230c266f3b-006477b077-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:39:19 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:39:19.441 DEBUG:boto:path=/pklxkd-58/ 2023-05-31T20:39:19.442 DEBUG:boto:auth_path=/pklxkd-58/ 2023-05-31T20:39:19.442 DEBUG:boto:path=/pklxkd-58/?prefix=obj00000000000000000133 2023-05-31T20:39:19.442 DEBUG:boto:auth_path=/pklxkd-58/?prefix=obj00000000000000000133 2023-05-31T20:39:19.443 DEBUG:boto:Method: GET 2023-05-31T20:39:19.443 DEBUG:boto:Path: /pklxkd-58/?prefix=obj00000000000000000133 2023-05-31T20:39:19.443 DEBUG:boto:Data: 2023-05-31T20:39:19.444 DEBUG:boto:Headers: {} 2023-05-31T20:39:19.444 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:39:19.445 DEBUG:boto:Port: 8001 2023-05-31T20:39:19.445 DEBUG:boto:Params: {} 2023-05-31T20:39:19.445 DEBUG:boto:Token: None 2023-05-31T20:39:19.446 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:39:19 GMT /pklxkd-58/ 2023-05-31T20:39:19.446 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:WOF0ykFTgI6McLQyUlzyKPU4swY= 2023-05-31T20:39:19.447 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:39:19 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:WOF0ykFTgI6McLQyUlzyKPU4swY=', 'Content-Length': '0'} 2023-05-31T20:39:19.449 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000df7451071d19b85d-006477b077-1c17-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:39:19 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:39:19.450 DEBUG:boto:b'pklxkd-58obj000000000000000001331000falseobj000000000000000001332023-05-31T20:38:19.997Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:39:19.451 DEBUG:boto:path=/pklxkd-58/ 2023-05-31T20:39:19.451 DEBUG:boto:auth_path=/pklxkd-58/ 2023-05-31T20:39:19.452 DEBUG:boto:path=/pklxkd-58/?prefix=obj00000000000000000133 2023-05-31T20:39:19.452 DEBUG:boto:auth_path=/pklxkd-58/?prefix=obj00000000000000000133 2023-05-31T20:39:19.453 DEBUG:boto:Method: GET 2023-05-31T20:39:19.453 DEBUG:boto:Path: /pklxkd-58/?prefix=obj00000000000000000133 2023-05-31T20:39:19.454 DEBUG:boto:Data: 2023-05-31T20:39:19.454 DEBUG:boto:Headers: {} 2023-05-31T20:39:19.455 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:39:19.455 DEBUG:boto:Port: 8000 2023-05-31T20:39:19.455 DEBUG:boto:Params: {} 2023-05-31T20:39:19.456 DEBUG:boto:Token: None 2023-05-31T20:39:19.456 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:39:19 GMT /pklxkd-58/ 2023-05-31T20:39:19.457 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:WOF0ykFTgI6McLQyUlzyKPU4swY= 2023-05-31T20:39:19.457 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:39:19 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:WOF0ykFTgI6McLQyUlzyKPU4swY=', 'Content-Length': '0'} 2023-05-31T20:39:19.461 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000004bfa79cf22c10e7-006477b077-1630-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:39:19 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:39:19.462 DEBUG:boto:b'pklxkd-58obj000000000000000001331000falseobj000000000000000001332023-05-31T20:38:19.997Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:39:19.463 DEBUG:boto:path=/pklxkd-58/obj00000000000000000133 2023-05-31T20:39:19.463 DEBUG:boto:auth_path=/pklxkd-58/obj00000000000000000133 2023-05-31T20:39:19.463 DEBUG:boto:Method: HEAD 2023-05-31T20:39:19.464 DEBUG:boto:Path: /pklxkd-58/obj00000000000000000133 2023-05-31T20:39:19.464 DEBUG:boto:Data: 2023-05-31T20:39:19.464 DEBUG:boto:Headers: {} 2023-05-31T20:39:19.465 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:39:19.465 DEBUG:boto:Port: 8001 2023-05-31T20:39:19.465 DEBUG:boto:Params: {} 2023-05-31T20:39:19.466 DEBUG:boto:Token: None 2023-05-31T20:39:19.466 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:39:19 GMT /pklxkd-58/obj00000000000000000133 2023-05-31T20:39:19.467 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:T0c0rp7UQf779RLzPG5XXypEFNI= 2023-05-31T20:39:19.467 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:39:19 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:T0c0rp7UQf779RLzPG5XXypEFNI=', 'Content-Length': '0'} 2023-05-31T20:39:19.469 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:38:19 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005f83c6d80ad9aca7-006477b077-1c17-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:39:19 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:39:19.470 DEBUG:boto:path=/pklxkd-58/obj00000000000000000133 2023-05-31T20:39:19.470 DEBUG:boto:auth_path=/pklxkd-58/obj00000000000000000133 2023-05-31T20:39:19.471 DEBUG:boto:Method: HEAD 2023-05-31T20:39:19.471 DEBUG:boto:Path: /pklxkd-58/obj00000000000000000133 2023-05-31T20:39:19.471 DEBUG:boto:Data: 2023-05-31T20:39:19.472 DEBUG:boto:Headers: {} 2023-05-31T20:39:19.472 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:39:19.472 DEBUG:boto:Port: 8000 2023-05-31T20:39:19.473 DEBUG:boto:Params: {} 2023-05-31T20:39:19.473 DEBUG:boto:Token: None 2023-05-31T20:39:19.473 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:39:19 GMT /pklxkd-58/obj00000000000000000133 2023-05-31T20:39:19.474 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:T0c0rp7UQf779RLzPG5XXypEFNI= 2023-05-31T20:39:19.474 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:39:19 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:T0c0rp7UQf779RLzPG5XXypEFNI=', 'Content-Length': '0'} 2023-05-31T20:39:19.476 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:38:19 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002148eb88ef9b9464-006477b077-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:39:19 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:39:19.477 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000133 2023-05-31T20:39:19.477 DEBUG:boto:path=/pklxkd-58/obj00000000000000000133 2023-05-31T20:39:19.478 DEBUG:boto:auth_path=/pklxkd-58/obj00000000000000000133 2023-05-31T20:39:19.478 DEBUG:boto:Method: GET 2023-05-31T20:39:19.478 DEBUG:boto:Path: /pklxkd-58/obj00000000000000000133 2023-05-31T20:39:19.479 DEBUG:boto:Data: 2023-05-31T20:39:19.479 DEBUG:boto:Headers: {} 2023-05-31T20:39:19.480 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:39:19.480 DEBUG:boto:Port: 8001 2023-05-31T20:39:19.481 DEBUG:boto:Params: {} 2023-05-31T20:39:19.481 DEBUG:boto:Token: None 2023-05-31T20:39:19.482 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:39:19 GMT /pklxkd-58/obj00000000000000000133 2023-05-31T20:39:19.483 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:1rAhc8LPt2GMmb2YHpQcdTMSsM4= 2023-05-31T20:39:19.483 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:39:19 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:1rAhc8LPt2GMmb2YHpQcdTMSsM4=', 'Content-Length': '0'} 2023-05-31T20:39:19.485 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:38:19 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000ea6308bc6e1fd25-006477b077-1c17-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:39:19 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:39:19.528 DEBUG:boto:path=/pklxkd-58/obj00000000000000000133 2023-05-31T20:39:19.529 DEBUG:boto:auth_path=/pklxkd-58/obj00000000000000000133 2023-05-31T20:39:19.529 DEBUG:boto:Method: GET 2023-05-31T20:39:19.530 DEBUG:boto:Path: /pklxkd-58/obj00000000000000000133 2023-05-31T20:39:19.530 DEBUG:boto:Data: 2023-05-31T20:39:19.530 DEBUG:boto:Headers: {} 2023-05-31T20:39:19.531 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:39:19.531 DEBUG:boto:Port: 8000 2023-05-31T20:39:19.531 DEBUG:boto:Params: {} 2023-05-31T20:39:19.532 DEBUG:boto:Token: None 2023-05-31T20:39:19.532 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:39:19 GMT /pklxkd-58/obj00000000000000000133 2023-05-31T20:39:19.533 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:1rAhc8LPt2GMmb2YHpQcdTMSsM4= 2023-05-31T20:39:19.533 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:39:19 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:1rAhc8LPt2GMmb2YHpQcdTMSsM4=', 'Content-Length': '0'} 2023-05-31T20:39:19.535 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:38:19 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005b9f9304245d3161-006477b077-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:39:19 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:39:19.536 DEBUG:boto:path=/pklxkd-58/ 2023-05-31T20:39:19.536 DEBUG:boto:auth_path=/pklxkd-58/ 2023-05-31T20:39:19.537 DEBUG:boto:path=/pklxkd-58/?prefix=obj00000000000000000133 2023-05-31T20:39:19.537 DEBUG:boto:auth_path=/pklxkd-58/?prefix=obj00000000000000000133 2023-05-31T20:39:19.538 DEBUG:boto:Method: GET 2023-05-31T20:39:19.538 DEBUG:boto:Path: /pklxkd-58/?prefix=obj00000000000000000133 2023-05-31T20:39:19.538 DEBUG:boto:Data: 2023-05-31T20:39:19.539 DEBUG:boto:Headers: {} 2023-05-31T20:39:19.539 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:39:19.539 DEBUG:boto:Port: 8001 2023-05-31T20:39:19.540 DEBUG:boto:Params: {} 2023-05-31T20:39:19.540 DEBUG:boto:Token: None 2023-05-31T20:39:19.541 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:39:19 GMT /pklxkd-58/ 2023-05-31T20:39:19.541 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:WOF0ykFTgI6McLQyUlzyKPU4swY= 2023-05-31T20:39:19.542 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:39:19 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:WOF0ykFTgI6McLQyUlzyKPU4swY=', 'Content-Length': '0'} 2023-05-31T20:39:19.544 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000061760655115cffc8-006477b077-1c17-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:39:19 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:39:19.545 DEBUG:boto:b'pklxkd-58obj000000000000000001331000falseobj000000000000000001332023-05-31T20:38:19.997Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:39:19.545 DEBUG:boto:path=/pklxkd-58/ 2023-05-31T20:39:19.546 DEBUG:boto:auth_path=/pklxkd-58/ 2023-05-31T20:39:19.546 DEBUG:boto:path=/pklxkd-58/?prefix=obj00000000000000000133 2023-05-31T20:39:19.547 DEBUG:boto:auth_path=/pklxkd-58/?prefix=obj00000000000000000133 2023-05-31T20:39:19.547 DEBUG:boto:Method: GET 2023-05-31T20:39:19.547 DEBUG:boto:Path: /pklxkd-58/?prefix=obj00000000000000000133 2023-05-31T20:39:19.548 DEBUG:boto:Data: 2023-05-31T20:39:19.548 DEBUG:boto:Headers: {} 2023-05-31T20:39:19.548 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:39:19.549 DEBUG:boto:Port: 8000 2023-05-31T20:39:19.549 DEBUG:boto:Params: {} 2023-05-31T20:39:19.550 DEBUG:boto:Token: None 2023-05-31T20:39:19.550 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:39:19 GMT /pklxkd-58/ 2023-05-31T20:39:19.550 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:WOF0ykFTgI6McLQyUlzyKPU4swY= 2023-05-31T20:39:19.551 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:39:19 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:WOF0ykFTgI6McLQyUlzyKPU4swY=', 'Content-Length': '0'} 2023-05-31T20:39:19.556 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000525ef2f465c6b343-006477b077-1630-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:39:19 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:39:19.558 DEBUG:boto:b'pklxkd-58obj000000000000000001331000falseobj000000000000000001332023-05-31T20:38:19.997Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:39:19.559 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000162 2023-05-31T20:39:19.560 DEBUG:boto:path=/pklxkd-58/obj00000000000000000162 2023-05-31T20:39:19.561 DEBUG:boto:auth_path=/pklxkd-58/obj00000000000000000162 2023-05-31T20:39:19.562 DEBUG:boto:path=/pklxkd-58/obj00000000000000000162?versionId=null 2023-05-31T20:39:19.562 DEBUG:boto:auth_path=/pklxkd-58/obj00000000000000000162?versionId=null 2023-05-31T20:39:19.563 DEBUG:boto:Method: GET 2023-05-31T20:39:19.564 DEBUG:boto:Path: /pklxkd-58/obj00000000000000000162?versionId=null 2023-05-31T20:39:19.564 DEBUG:boto:Data: 2023-05-31T20:39:19.565 DEBUG:boto:Headers: {} 2023-05-31T20:39:19.566 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:39:19.566 DEBUG:boto:Port: 8001 2023-05-31T20:39:19.567 DEBUG:boto:Params: {} 2023-05-31T20:39:19.567 DEBUG:boto:Token: None 2023-05-31T20:39:19.568 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:39:19 GMT /pklxkd-58/obj00000000000000000162?versionId=null 2023-05-31T20:39:19.569 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:Y3cjQmOUYgbLfqhXJQLdIPa59D8= 2023-05-31T20:39:19.570 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:39:19 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:Y3cjQmOUYgbLfqhXJQLdIPa59D8=', 'Content-Length': '0'} 2023-05-31T20:39:19.573 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:38:20 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008b82ac3eb1ec2035-006477b077-1c17-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:39:19 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:39:19.574 DEBUG:boto:path=/pklxkd-58/obj00000000000000000162 2023-05-31T20:39:19.574 DEBUG:boto:auth_path=/pklxkd-58/obj00000000000000000162 2023-05-31T20:39:19.575 DEBUG:boto:path=/pklxkd-58/obj00000000000000000162?versionId=null 2023-05-31T20:39:19.576 DEBUG:boto:auth_path=/pklxkd-58/obj00000000000000000162?versionId=null 2023-05-31T20:39:19.577 DEBUG:boto:Method: GET 2023-05-31T20:39:19.577 DEBUG:boto:Path: /pklxkd-58/obj00000000000000000162?versionId=null 2023-05-31T20:39:19.579 DEBUG:boto:Data: 2023-05-31T20:39:19.580 DEBUG:boto:Headers: {} 2023-05-31T20:39:19.581 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:39:19.581 DEBUG:boto:Port: 8000 2023-05-31T20:39:19.582 DEBUG:boto:Params: {} 2023-05-31T20:39:19.582 DEBUG:boto:Token: None 2023-05-31T20:39:19.583 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:39:19 GMT /pklxkd-58/obj00000000000000000162?versionId=null 2023-05-31T20:39:19.583 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:Y3cjQmOUYgbLfqhXJQLdIPa59D8= 2023-05-31T20:39:19.585 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:39:19 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:Y3cjQmOUYgbLfqhXJQLdIPa59D8=', 'Content-Length': '0'} 2023-05-31T20:39:19.587 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:38:20 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001eeca80a8a66958d-006477b077-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:39:19 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:39:19.628 DEBUG:boto:path=/pklxkd-58/obj00000000000000000162 2023-05-31T20:39:19.629 DEBUG:boto:auth_path=/pklxkd-58/obj00000000000000000162 2023-05-31T20:39:19.629 DEBUG:boto:path=/pklxkd-58/obj00000000000000000162?versionId=null 2023-05-31T20:39:19.630 DEBUG:boto:auth_path=/pklxkd-58/obj00000000000000000162?versionId=null 2023-05-31T20:39:19.630 DEBUG:boto:Method: HEAD 2023-05-31T20:39:19.631 DEBUG:boto:Path: /pklxkd-58/obj00000000000000000162?versionId=null 2023-05-31T20:39:19.631 DEBUG:boto:Data: 2023-05-31T20:39:19.631 DEBUG:boto:Headers: {} 2023-05-31T20:39:19.632 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:39:19.632 DEBUG:boto:Port: 8001 2023-05-31T20:39:19.633 DEBUG:boto:Params: {} 2023-05-31T20:39:19.633 DEBUG:boto:Token: None 2023-05-31T20:39:19.633 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:39:19 GMT /pklxkd-58/obj00000000000000000162?versionId=null 2023-05-31T20:39:19.634 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:fyPXwME8sB/Lq6KrFh1G32iSCDg= 2023-05-31T20:39:19.634 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:39:19 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:fyPXwME8sB/Lq6KrFh1G32iSCDg=', 'Content-Length': '0'} 2023-05-31T20:39:19.637 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:38:20 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000f571a45819f79f9-006477b077-1c17-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:39:19 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:39:19.638 DEBUG:boto:path=/pklxkd-58/obj00000000000000000162 2023-05-31T20:39:19.639 DEBUG:boto:auth_path=/pklxkd-58/obj00000000000000000162 2023-05-31T20:39:19.640 DEBUG:boto:path=/pklxkd-58/obj00000000000000000162?versionId=null 2023-05-31T20:39:19.641 DEBUG:boto:auth_path=/pklxkd-58/obj00000000000000000162?versionId=null 2023-05-31T20:39:19.642 DEBUG:boto:Method: HEAD 2023-05-31T20:39:19.643 DEBUG:boto:Path: /pklxkd-58/obj00000000000000000162?versionId=null 2023-05-31T20:39:19.644 DEBUG:boto:Data: 2023-05-31T20:39:19.644 DEBUG:boto:Headers: {} 2023-05-31T20:39:19.645 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:39:19.646 DEBUG:boto:Port: 8000 2023-05-31T20:39:19.646 DEBUG:boto:Params: {} 2023-05-31T20:39:19.647 DEBUG:boto:Token: None 2023-05-31T20:39:19.648 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:39:19 GMT /pklxkd-58/obj00000000000000000162?versionId=null 2023-05-31T20:39:19.648 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:fyPXwME8sB/Lq6KrFh1G32iSCDg= 2023-05-31T20:39:19.649 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:39:19 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:fyPXwME8sB/Lq6KrFh1G32iSCDg=', 'Content-Length': '0'} 2023-05-31T20:39:19.651 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:38:20 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a27e27e506e46b0f-006477b077-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:39:19 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:39:19.652 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000162 2023-05-31T20:39:19.653 DEBUG:boto:path=/pklxkd-58/obj00000000000000000162 2023-05-31T20:39:19.654 DEBUG:boto:auth_path=/pklxkd-58/obj00000000000000000162 2023-05-31T20:39:19.654 DEBUG:boto:path=/pklxkd-58/obj00000000000000000162?versionId=null 2023-05-31T20:39:19.655 DEBUG:boto:auth_path=/pklxkd-58/obj00000000000000000162?versionId=null 2023-05-31T20:39:19.656 DEBUG:boto:Method: GET 2023-05-31T20:39:19.656 DEBUG:boto:Path: /pklxkd-58/obj00000000000000000162?versionId=null 2023-05-31T20:39:19.657 DEBUG:boto:Data: 2023-05-31T20:39:19.658 DEBUG:boto:Headers: {} 2023-05-31T20:39:19.659 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:39:19.659 DEBUG:boto:Port: 8001 2023-05-31T20:39:19.660 DEBUG:boto:Params: {} 2023-05-31T20:39:19.660 DEBUG:boto:Token: None 2023-05-31T20:39:19.661 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:39:19 GMT /pklxkd-58/obj00000000000000000162?versionId=null 2023-05-31T20:39:19.662 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:Y3cjQmOUYgbLfqhXJQLdIPa59D8= 2023-05-31T20:39:19.663 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:39:19 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:Y3cjQmOUYgbLfqhXJQLdIPa59D8=', 'Content-Length': '0'} 2023-05-31T20:39:19.666 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:38:20 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f6252a1dab4f554a-006477b077-1c17-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:39:19 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:39:19.709 DEBUG:boto:path=/pklxkd-58/obj00000000000000000162 2023-05-31T20:39:19.710 DEBUG:boto:auth_path=/pklxkd-58/obj00000000000000000162 2023-05-31T20:39:19.711 DEBUG:boto:path=/pklxkd-58/obj00000000000000000162?versionId=null 2023-05-31T20:39:19.711 DEBUG:boto:auth_path=/pklxkd-58/obj00000000000000000162?versionId=null 2023-05-31T20:39:19.712 DEBUG:boto:Method: GET 2023-05-31T20:39:19.712 DEBUG:boto:Path: /pklxkd-58/obj00000000000000000162?versionId=null 2023-05-31T20:39:19.713 DEBUG:boto:Data: 2023-05-31T20:39:19.713 DEBUG:boto:Headers: {} 2023-05-31T20:39:19.713 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:39:19.714 DEBUG:boto:Port: 8000 2023-05-31T20:39:19.714 DEBUG:boto:Params: {} 2023-05-31T20:39:19.717 DEBUG:boto:Token: None 2023-05-31T20:39:19.718 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:39:19 GMT /pklxkd-58/obj00000000000000000162?versionId=null 2023-05-31T20:39:19.719 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:Y3cjQmOUYgbLfqhXJQLdIPa59D8= 2023-05-31T20:39:19.719 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:39:19 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:Y3cjQmOUYgbLfqhXJQLdIPa59D8=', 'Content-Length': '0'} 2023-05-31T20:39:19.721 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:38:20 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000075ed73f818fd2ee4-006477b077-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:39:19 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:39:19.722 DEBUG:boto:path=/pklxkd-58/ 2023-05-31T20:39:19.722 DEBUG:boto:auth_path=/pklxkd-58/ 2023-05-31T20:39:19.723 DEBUG:boto:path=/pklxkd-58/?prefix=obj00000000000000000162 2023-05-31T20:39:19.723 DEBUG:boto:auth_path=/pklxkd-58/?prefix=obj00000000000000000162 2023-05-31T20:39:19.723 DEBUG:boto:Method: GET 2023-05-31T20:39:19.724 DEBUG:boto:Path: /pklxkd-58/?prefix=obj00000000000000000162 2023-05-31T20:39:19.724 DEBUG:boto:Data: 2023-05-31T20:39:19.725 DEBUG:boto:Headers: {} 2023-05-31T20:39:19.725 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:39:19.726 DEBUG:boto:Port: 8001 2023-05-31T20:39:19.726 DEBUG:boto:Params: {} 2023-05-31T20:39:19.726 DEBUG:boto:Token: None 2023-05-31T20:39:19.727 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:39:19 GMT /pklxkd-58/ 2023-05-31T20:39:19.728 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:WOF0ykFTgI6McLQyUlzyKPU4swY= 2023-05-31T20:39:19.728 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:39:19 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:WOF0ykFTgI6McLQyUlzyKPU4swY=', 'Content-Length': '0'} 2023-05-31T20:39:19.731 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000003de6c736fca77db0-006477b077-1c17-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:39:19 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:39:19.732 DEBUG:boto:b'pklxkd-58obj000000000000000001621000falseobj000000000000000001622023-05-31T20:38:20.538Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:39:19.733 DEBUG:boto:path=/pklxkd-58/ 2023-05-31T20:39:19.733 DEBUG:boto:auth_path=/pklxkd-58/ 2023-05-31T20:39:19.734 DEBUG:boto:path=/pklxkd-58/?prefix=obj00000000000000000162 2023-05-31T20:39:19.734 DEBUG:boto:auth_path=/pklxkd-58/?prefix=obj00000000000000000162 2023-05-31T20:39:19.734 DEBUG:boto:Method: GET 2023-05-31T20:39:19.735 DEBUG:boto:Path: /pklxkd-58/?prefix=obj00000000000000000162 2023-05-31T20:39:19.735 DEBUG:boto:Data: 2023-05-31T20:39:19.736 DEBUG:boto:Headers: {} 2023-05-31T20:39:19.736 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:39:19.737 DEBUG:boto:Port: 8000 2023-05-31T20:39:19.737 DEBUG:boto:Params: {} 2023-05-31T20:39:19.738 DEBUG:boto:Token: None 2023-05-31T20:39:19.739 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:39:19 GMT /pklxkd-58/ 2023-05-31T20:39:19.740 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:WOF0ykFTgI6McLQyUlzyKPU4swY= 2023-05-31T20:39:19.740 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:39:19 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:WOF0ykFTgI6McLQyUlzyKPU4swY=', 'Content-Length': '0'} 2023-05-31T20:39:19.744 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000003dbca84962a7045e-006477b077-1630-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:39:19 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:39:19.745 DEBUG:boto:b'pklxkd-58obj000000000000000001621000falseobj000000000000000001622023-05-31T20:38:20.538Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:39:19.746 DEBUG:boto:path=/pklxkd-58/obj00000000000000000162 2023-05-31T20:39:19.746 DEBUG:boto:auth_path=/pklxkd-58/obj00000000000000000162 2023-05-31T20:39:19.746 DEBUG:boto:Method: HEAD 2023-05-31T20:39:19.748 DEBUG:boto:Path: /pklxkd-58/obj00000000000000000162 2023-05-31T20:39:19.750 DEBUG:boto:Data: 2023-05-31T20:39:19.750 DEBUG:boto:Headers: {} 2023-05-31T20:39:19.750 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:39:19.751 DEBUG:boto:Port: 8001 2023-05-31T20:39:19.751 DEBUG:boto:Params: {} 2023-05-31T20:39:19.751 DEBUG:boto:Token: None 2023-05-31T20:39:19.752 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:39:19 GMT /pklxkd-58/obj00000000000000000162 2023-05-31T20:39:19.752 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:k96rkMTAwjE0k+ZVYmhzpg5o4bE= 2023-05-31T20:39:19.753 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:39:19 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:k96rkMTAwjE0k+ZVYmhzpg5o4bE=', 'Content-Length': '0'} 2023-05-31T20:39:19.755 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:38:20 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000003398b6f60fc600a-006477b077-1c17-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:39:19 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:39:19.755 DEBUG:boto:path=/pklxkd-58/obj00000000000000000162 2023-05-31T20:39:19.756 DEBUG:boto:auth_path=/pklxkd-58/obj00000000000000000162 2023-05-31T20:39:19.756 DEBUG:boto:Method: HEAD 2023-05-31T20:39:19.756 DEBUG:boto:Path: /pklxkd-58/obj00000000000000000162 2023-05-31T20:39:19.757 DEBUG:boto:Data: 2023-05-31T20:39:19.757 DEBUG:boto:Headers: {} 2023-05-31T20:39:19.757 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:39:19.758 DEBUG:boto:Port: 8000 2023-05-31T20:39:19.758 DEBUG:boto:Params: {} 2023-05-31T20:39:19.759 DEBUG:boto:Token: None 2023-05-31T20:39:19.759 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:39:19 GMT /pklxkd-58/obj00000000000000000162 2023-05-31T20:39:19.759 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:k96rkMTAwjE0k+ZVYmhzpg5o4bE= 2023-05-31T20:39:19.760 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:39:19 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:k96rkMTAwjE0k+ZVYmhzpg5o4bE=', 'Content-Length': '0'} 2023-05-31T20:39:19.764 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:38:20 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000047265e683367d1a3-006477b077-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:39:19 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:39:19.765 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000162 2023-05-31T20:39:19.765 DEBUG:boto:path=/pklxkd-58/obj00000000000000000162 2023-05-31T20:39:19.766 DEBUG:boto:auth_path=/pklxkd-58/obj00000000000000000162 2023-05-31T20:39:19.766 DEBUG:boto:Method: GET 2023-05-31T20:39:19.767 DEBUG:boto:Path: /pklxkd-58/obj00000000000000000162 2023-05-31T20:39:19.767 DEBUG:boto:Data: 2023-05-31T20:39:19.767 DEBUG:boto:Headers: {} 2023-05-31T20:39:19.768 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:39:19.768 DEBUG:boto:Port: 8001 2023-05-31T20:39:19.768 DEBUG:boto:Params: {} 2023-05-31T20:39:19.769 DEBUG:boto:Token: None 2023-05-31T20:39:19.769 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:39:19 GMT /pklxkd-58/obj00000000000000000162 2023-05-31T20:39:19.770 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:Y061wN9vAOjkonjIn63mt6Nug3Q= 2023-05-31T20:39:19.770 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:39:19 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:Y061wN9vAOjkonjIn63mt6Nug3Q=', 'Content-Length': '0'} 2023-05-31T20:39:19.772 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:38:20 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000debf6869d5c5e19c-006477b077-1c17-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:39:19 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:39:19.773 DEBUG:boto:path=/pklxkd-58/obj00000000000000000162 2023-05-31T20:39:19.773 DEBUG:boto:auth_path=/pklxkd-58/obj00000000000000000162 2023-05-31T20:39:19.773 DEBUG:boto:Method: GET 2023-05-31T20:39:19.774 DEBUG:boto:Path: /pklxkd-58/obj00000000000000000162 2023-05-31T20:39:19.775 DEBUG:boto:Data: 2023-05-31T20:39:19.775 DEBUG:boto:Headers: {} 2023-05-31T20:39:19.775 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:39:19.776 DEBUG:boto:Port: 8000 2023-05-31T20:39:19.776 DEBUG:boto:Params: {} 2023-05-31T20:39:19.776 DEBUG:boto:Token: None 2023-05-31T20:39:19.777 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:39:19 GMT /pklxkd-58/obj00000000000000000162 2023-05-31T20:39:19.777 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:Y061wN9vAOjkonjIn63mt6Nug3Q= 2023-05-31T20:39:19.778 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:39:19 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:Y061wN9vAOjkonjIn63mt6Nug3Q=', 'Content-Length': '0'} 2023-05-31T20:39:19.781 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:38:20 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000a7fb2e16173661a-006477b077-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:39:19 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:39:19.825 DEBUG:boto:path=/pklxkd-58/ 2023-05-31T20:39:19.825 DEBUG:boto:auth_path=/pklxkd-58/ 2023-05-31T20:39:19.826 DEBUG:boto:path=/pklxkd-58/?prefix=obj00000000000000000162 2023-05-31T20:39:19.826 DEBUG:boto:auth_path=/pklxkd-58/?prefix=obj00000000000000000162 2023-05-31T20:39:19.826 DEBUG:boto:Method: GET 2023-05-31T20:39:19.827 DEBUG:boto:Path: /pklxkd-58/?prefix=obj00000000000000000162 2023-05-31T20:39:19.827 DEBUG:boto:Data: 2023-05-31T20:39:19.827 DEBUG:boto:Headers: {} 2023-05-31T20:39:19.828 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:39:19.828 DEBUG:boto:Port: 8001 2023-05-31T20:39:19.828 DEBUG:boto:Params: {} 2023-05-31T20:39:19.829 DEBUG:boto:Token: None 2023-05-31T20:39:19.829 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:39:19 GMT /pklxkd-58/ 2023-05-31T20:39:19.830 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:WOF0ykFTgI6McLQyUlzyKPU4swY= 2023-05-31T20:39:19.830 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:39:19 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:WOF0ykFTgI6McLQyUlzyKPU4swY=', 'Content-Length': '0'} 2023-05-31T20:39:19.833 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b69208018051dd64-006477b077-1c17-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:39:19 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:39:19.834 DEBUG:boto:b'pklxkd-58obj000000000000000001621000falseobj000000000000000001622023-05-31T20:38:20.538Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:39:19.835 DEBUG:boto:path=/pklxkd-58/ 2023-05-31T20:39:19.835 DEBUG:boto:auth_path=/pklxkd-58/ 2023-05-31T20:39:19.836 DEBUG:boto:path=/pklxkd-58/?prefix=obj00000000000000000162 2023-05-31T20:39:19.836 DEBUG:boto:auth_path=/pklxkd-58/?prefix=obj00000000000000000162 2023-05-31T20:39:19.836 DEBUG:boto:Method: GET 2023-05-31T20:39:19.837 DEBUG:boto:Path: /pklxkd-58/?prefix=obj00000000000000000162 2023-05-31T20:39:19.837 DEBUG:boto:Data: 2023-05-31T20:39:19.837 DEBUG:boto:Headers: {} 2023-05-31T20:39:19.838 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:39:19.838 DEBUG:boto:Port: 8000 2023-05-31T20:39:19.838 DEBUG:boto:Params: {} 2023-05-31T20:39:19.839 DEBUG:boto:Token: None 2023-05-31T20:39:19.839 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:39:19 GMT /pklxkd-58/ 2023-05-31T20:39:19.840 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:WOF0ykFTgI6McLQyUlzyKPU4swY= 2023-05-31T20:39:19.841 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:39:19 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:WOF0ykFTgI6McLQyUlzyKPU4swY=', 'Content-Length': '0'} 2023-05-31T20:39:19.847 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000008f4676a87918e4b6-006477b077-1630-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:39:19 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:39:19.848 DEBUG:boto:b'pklxkd-58obj000000000000000001621000falseobj000000000000000001622023-05-31T20:38:20.538Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:39:19.848 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000271 2023-05-31T20:39:19.849 DEBUG:boto:path=/pklxkd-58/obj00000000000000000271 2023-05-31T20:39:19.849 DEBUG:boto:auth_path=/pklxkd-58/obj00000000000000000271 2023-05-31T20:39:19.850 DEBUG:boto:path=/pklxkd-58/obj00000000000000000271?versionId=null 2023-05-31T20:39:19.850 DEBUG:boto:auth_path=/pklxkd-58/obj00000000000000000271?versionId=null 2023-05-31T20:39:19.850 DEBUG:boto:Method: GET 2023-05-31T20:39:19.851 DEBUG:boto:Path: /pklxkd-58/obj00000000000000000271?versionId=null 2023-05-31T20:39:19.851 DEBUG:boto:Data: 2023-05-31T20:39:19.851 DEBUG:boto:Headers: {} 2023-05-31T20:39:19.852 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:39:19.852 DEBUG:boto:Port: 8001 2023-05-31T20:39:19.852 DEBUG:boto:Params: {} 2023-05-31T20:39:19.853 DEBUG:boto:Token: None 2023-05-31T20:39:19.854 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:39:19 GMT /pklxkd-58/obj00000000000000000271?versionId=null 2023-05-31T20:39:19.854 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:r/10ce/1u9IKKbYA5jx4xEcxJi0= 2023-05-31T20:39:19.855 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:39:19 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:r/10ce/1u9IKKbYA5jx4xEcxJi0=', 'Content-Length': '0'} 2023-05-31T20:39:19.858 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:38:20 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000043c4694b98a95e6e-006477b077-1c17-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:39:19 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:39:19.860 DEBUG:boto:path=/pklxkd-58/obj00000000000000000271 2023-05-31T20:39:19.861 DEBUG:boto:auth_path=/pklxkd-58/obj00000000000000000271 2023-05-31T20:39:19.862 DEBUG:boto:path=/pklxkd-58/obj00000000000000000271?versionId=null 2023-05-31T20:39:19.863 DEBUG:boto:auth_path=/pklxkd-58/obj00000000000000000271?versionId=null 2023-05-31T20:39:19.864 DEBUG:boto:Method: GET 2023-05-31T20:39:19.865 DEBUG:boto:Path: /pklxkd-58/obj00000000000000000271?versionId=null 2023-05-31T20:39:19.866 DEBUG:boto:Data: 2023-05-31T20:39:19.866 DEBUG:boto:Headers: {} 2023-05-31T20:39:19.867 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:39:19.867 DEBUG:boto:Port: 8000 2023-05-31T20:39:19.868 DEBUG:boto:Params: {} 2023-05-31T20:39:19.868 DEBUG:boto:Token: None 2023-05-31T20:39:19.869 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:39:19 GMT /pklxkd-58/obj00000000000000000271?versionId=null 2023-05-31T20:39:19.869 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:r/10ce/1u9IKKbYA5jx4xEcxJi0= 2023-05-31T20:39:19.869 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:39:19 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:r/10ce/1u9IKKbYA5jx4xEcxJi0=', 'Content-Length': '0'} 2023-05-31T20:39:19.872 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:38:20 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c90fbaf1ab38fd08-006477b077-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:39:19 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:39:19.872 DEBUG:boto:path=/pklxkd-58/obj00000000000000000271 2023-05-31T20:39:19.873 DEBUG:boto:auth_path=/pklxkd-58/obj00000000000000000271 2023-05-31T20:39:19.873 DEBUG:boto:path=/pklxkd-58/obj00000000000000000271?versionId=null 2023-05-31T20:39:19.875 DEBUG:boto:auth_path=/pklxkd-58/obj00000000000000000271?versionId=null 2023-05-31T20:39:19.875 DEBUG:boto:Method: HEAD 2023-05-31T20:39:19.875 DEBUG:boto:Path: /pklxkd-58/obj00000000000000000271?versionId=null 2023-05-31T20:39:19.876 DEBUG:boto:Data: 2023-05-31T20:39:19.876 DEBUG:boto:Headers: {} 2023-05-31T20:39:19.877 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:39:19.877 DEBUG:boto:Port: 8001 2023-05-31T20:39:19.877 DEBUG:boto:Params: {} 2023-05-31T20:39:19.878 DEBUG:boto:Token: None 2023-05-31T20:39:19.879 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:39:19 GMT /pklxkd-58/obj00000000000000000271?versionId=null 2023-05-31T20:39:19.879 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:WHwWM+8jKkxe64PXiIIgkNMzLrs= 2023-05-31T20:39:19.880 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:39:19 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:WHwWM+8jKkxe64PXiIIgkNMzLrs=', 'Content-Length': '0'} 2023-05-31T20:39:19.882 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:38:20 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f1765a85dc8c868c-006477b077-1c17-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:39:19 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:39:19.883 DEBUG:boto:path=/pklxkd-58/obj00000000000000000271 2023-05-31T20:39:19.884 DEBUG:boto:auth_path=/pklxkd-58/obj00000000000000000271 2023-05-31T20:39:19.885 DEBUG:boto:path=/pklxkd-58/obj00000000000000000271?versionId=null 2023-05-31T20:39:19.885 DEBUG:boto:auth_path=/pklxkd-58/obj00000000000000000271?versionId=null 2023-05-31T20:39:19.886 DEBUG:boto:Method: HEAD 2023-05-31T20:39:19.886 DEBUG:boto:Path: /pklxkd-58/obj00000000000000000271?versionId=null 2023-05-31T20:39:19.887 DEBUG:boto:Data: 2023-05-31T20:39:19.887 DEBUG:boto:Headers: {} 2023-05-31T20:39:19.887 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:39:19.888 DEBUG:boto:Port: 8000 2023-05-31T20:39:19.888 DEBUG:boto:Params: {} 2023-05-31T20:39:19.889 DEBUG:boto:Token: None 2023-05-31T20:39:19.889 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:39:19 GMT /pklxkd-58/obj00000000000000000271?versionId=null 2023-05-31T20:39:19.890 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:WHwWM+8jKkxe64PXiIIgkNMzLrs= 2023-05-31T20:39:19.890 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:39:19 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:WHwWM+8jKkxe64PXiIIgkNMzLrs=', 'Content-Length': '0'} 2023-05-31T20:39:19.893 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:38:20 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000018e2f46c54ae880a-006477b077-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:39:19 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:39:19.894 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000271 2023-05-31T20:39:19.894 DEBUG:boto:path=/pklxkd-58/obj00000000000000000271 2023-05-31T20:39:19.895 DEBUG:boto:auth_path=/pklxkd-58/obj00000000000000000271 2023-05-31T20:39:19.895 DEBUG:boto:path=/pklxkd-58/obj00000000000000000271?versionId=null 2023-05-31T20:39:19.895 DEBUG:boto:auth_path=/pklxkd-58/obj00000000000000000271?versionId=null 2023-05-31T20:39:19.896 DEBUG:boto:Method: GET 2023-05-31T20:39:19.896 DEBUG:boto:Path: /pklxkd-58/obj00000000000000000271?versionId=null 2023-05-31T20:39:19.897 DEBUG:boto:Data: 2023-05-31T20:39:19.897 DEBUG:boto:Headers: {} 2023-05-31T20:39:19.897 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:39:19.898 DEBUG:boto:Port: 8001 2023-05-31T20:39:19.898 DEBUG:boto:Params: {} 2023-05-31T20:39:19.898 DEBUG:boto:Token: None 2023-05-31T20:39:19.899 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:39:19 GMT /pklxkd-58/obj00000000000000000271?versionId=null 2023-05-31T20:39:19.899 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:r/10ce/1u9IKKbYA5jx4xEcxJi0= 2023-05-31T20:39:19.900 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:39:19 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:r/10ce/1u9IKKbYA5jx4xEcxJi0=', 'Content-Length': '0'} 2023-05-31T20:39:19.902 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:38:20 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009671a4bba427f68e-006477b077-1c17-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:39:19 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:39:19.948 DEBUG:boto:path=/pklxkd-58/obj00000000000000000271 2023-05-31T20:39:19.949 DEBUG:boto:auth_path=/pklxkd-58/obj00000000000000000271 2023-05-31T20:39:19.949 DEBUG:boto:path=/pklxkd-58/obj00000000000000000271?versionId=null 2023-05-31T20:39:19.950 DEBUG:boto:auth_path=/pklxkd-58/obj00000000000000000271?versionId=null 2023-05-31T20:39:19.950 DEBUG:boto:Method: GET 2023-05-31T20:39:19.951 DEBUG:boto:Path: /pklxkd-58/obj00000000000000000271?versionId=null 2023-05-31T20:39:19.951 DEBUG:boto:Data: 2023-05-31T20:39:19.951 DEBUG:boto:Headers: {} 2023-05-31T20:39:19.952 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:39:19.952 DEBUG:boto:Port: 8000 2023-05-31T20:39:19.952 DEBUG:boto:Params: {} 2023-05-31T20:39:19.953 DEBUG:boto:Token: None 2023-05-31T20:39:19.953 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:39:19 GMT /pklxkd-58/obj00000000000000000271?versionId=null 2023-05-31T20:39:19.954 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:r/10ce/1u9IKKbYA5jx4xEcxJi0= 2023-05-31T20:39:19.954 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:39:19 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:r/10ce/1u9IKKbYA5jx4xEcxJi0=', 'Content-Length': '0'} 2023-05-31T20:39:19.956 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:38:20 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a591a322629644d8-006477b077-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:39:19 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:39:19.957 DEBUG:boto:path=/pklxkd-58/ 2023-05-31T20:39:19.957 DEBUG:boto:auth_path=/pklxkd-58/ 2023-05-31T20:39:19.958 DEBUG:boto:path=/pklxkd-58/?prefix=obj00000000000000000271 2023-05-31T20:39:19.958 DEBUG:boto:auth_path=/pklxkd-58/?prefix=obj00000000000000000271 2023-05-31T20:39:19.958 DEBUG:boto:Method: GET 2023-05-31T20:39:19.959 DEBUG:boto:Path: /pklxkd-58/?prefix=obj00000000000000000271 2023-05-31T20:39:19.959 DEBUG:boto:Data: 2023-05-31T20:39:19.959 DEBUG:boto:Headers: {} 2023-05-31T20:39:19.960 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:39:19.960 DEBUG:boto:Port: 8001 2023-05-31T20:39:19.960 DEBUG:boto:Params: {} 2023-05-31T20:39:19.961 DEBUG:boto:Token: None 2023-05-31T20:39:19.961 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:39:19 GMT /pklxkd-58/ 2023-05-31T20:39:19.962 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:WOF0ykFTgI6McLQyUlzyKPU4swY= 2023-05-31T20:39:19.962 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:39:19 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:WOF0ykFTgI6McLQyUlzyKPU4swY=', 'Content-Length': '0'} 2023-05-31T20:39:19.965 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000003082ce09f53f61f2-006477b077-1c17-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:39:19 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:39:19.965 DEBUG:boto:b'pklxkd-58obj000000000000000002711000falseobj000000000000000002712023-05-31T20:38:20.240Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:39:19.966 DEBUG:boto:path=/pklxkd-58/ 2023-05-31T20:39:19.966 DEBUG:boto:auth_path=/pklxkd-58/ 2023-05-31T20:39:19.967 DEBUG:boto:path=/pklxkd-58/?prefix=obj00000000000000000271 2023-05-31T20:39:19.967 DEBUG:boto:auth_path=/pklxkd-58/?prefix=obj00000000000000000271 2023-05-31T20:39:19.967 DEBUG:boto:Method: GET 2023-05-31T20:39:19.968 DEBUG:boto:Path: /pklxkd-58/?prefix=obj00000000000000000271 2023-05-31T20:39:19.968 DEBUG:boto:Data: 2023-05-31T20:39:19.968 DEBUG:boto:Headers: {} 2023-05-31T20:39:19.969 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:39:19.969 DEBUG:boto:Port: 8000 2023-05-31T20:39:19.969 DEBUG:boto:Params: {} 2023-05-31T20:39:19.970 DEBUG:boto:Token: None 2023-05-31T20:39:19.970 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:39:19 GMT /pklxkd-58/ 2023-05-31T20:39:19.971 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:WOF0ykFTgI6McLQyUlzyKPU4swY= 2023-05-31T20:39:19.971 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:39:19 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:WOF0ykFTgI6McLQyUlzyKPU4swY=', 'Content-Length': '0'} 2023-05-31T20:39:19.975 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000695df0e7076f8c18-006477b077-1630-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:39:19 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:39:20.020 DEBUG:boto:b'pklxkd-58obj000000000000000002711000falseobj000000000000000002712023-05-31T20:38:20.240Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:39:20.021 DEBUG:boto:path=/pklxkd-58/obj00000000000000000271 2023-05-31T20:39:20.022 DEBUG:boto:auth_path=/pklxkd-58/obj00000000000000000271 2023-05-31T20:39:20.022 DEBUG:boto:Method: HEAD 2023-05-31T20:39:20.023 DEBUG:boto:Path: /pklxkd-58/obj00000000000000000271 2023-05-31T20:39:20.023 DEBUG:boto:Data: 2023-05-31T20:39:20.023 DEBUG:boto:Headers: {} 2023-05-31T20:39:20.024 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:39:20.024 DEBUG:boto:Port: 8001 2023-05-31T20:39:20.024 DEBUG:boto:Params: {} 2023-05-31T20:39:20.025 DEBUG:boto:Token: None 2023-05-31T20:39:20.025 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:39:20 GMT /pklxkd-58/obj00000000000000000271 2023-05-31T20:39:20.026 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:KDkJmvcSgtlLCDzypuNYzwnwsZE= 2023-05-31T20:39:20.026 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:39:20 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:KDkJmvcSgtlLCDzypuNYzwnwsZE=', 'Content-Length': '0'} 2023-05-31T20:39:20.028 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:38:20 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d4c8b123b4ac0b28-006477b078-1c17-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:39:20 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:39:20.029 DEBUG:boto:path=/pklxkd-58/obj00000000000000000271 2023-05-31T20:39:20.029 DEBUG:boto:auth_path=/pklxkd-58/obj00000000000000000271 2023-05-31T20:39:20.029 DEBUG:boto:Method: HEAD 2023-05-31T20:39:20.030 DEBUG:boto:Path: /pklxkd-58/obj00000000000000000271 2023-05-31T20:39:20.030 DEBUG:boto:Data: 2023-05-31T20:39:20.030 DEBUG:boto:Headers: {} 2023-05-31T20:39:20.031 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:39:20.031 DEBUG:boto:Port: 8000 2023-05-31T20:39:20.031 DEBUG:boto:Params: {} 2023-05-31T20:39:20.032 DEBUG:boto:Token: None 2023-05-31T20:39:20.032 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:39:20 GMT /pklxkd-58/obj00000000000000000271 2023-05-31T20:39:20.033 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:KDkJmvcSgtlLCDzypuNYzwnwsZE= 2023-05-31T20:39:20.034 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:39:20 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:KDkJmvcSgtlLCDzypuNYzwnwsZE=', 'Content-Length': '0'} 2023-05-31T20:39:20.036 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:38:20 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a5a4681d02cfa67a-006477b078-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:39:20 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:39:20.036 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000271 2023-05-31T20:39:20.037 DEBUG:boto:path=/pklxkd-58/obj00000000000000000271 2023-05-31T20:39:20.037 DEBUG:boto:auth_path=/pklxkd-58/obj00000000000000000271 2023-05-31T20:39:20.037 DEBUG:boto:Method: GET 2023-05-31T20:39:20.038 DEBUG:boto:Path: /pklxkd-58/obj00000000000000000271 2023-05-31T20:39:20.038 DEBUG:boto:Data: 2023-05-31T20:39:20.039 DEBUG:boto:Headers: {} 2023-05-31T20:39:20.039 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:39:20.039 DEBUG:boto:Port: 8001 2023-05-31T20:39:20.040 DEBUG:boto:Params: {} 2023-05-31T20:39:20.040 DEBUG:boto:Token: None 2023-05-31T20:39:20.040 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:39:20 GMT /pklxkd-58/obj00000000000000000271 2023-05-31T20:39:20.041 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:j/dzK8XrfDFcmxfOo15NSJI7rmg= 2023-05-31T20:39:20.041 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:39:20 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:j/dzK8XrfDFcmxfOo15NSJI7rmg=', 'Content-Length': '0'} 2023-05-31T20:39:20.043 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:38:20 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f075e124d44ccc7b-006477b078-1c17-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:39:20 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:39:20.044 DEBUG:boto:path=/pklxkd-58/obj00000000000000000271 2023-05-31T20:39:20.044 DEBUG:boto:auth_path=/pklxkd-58/obj00000000000000000271 2023-05-31T20:39:20.045 DEBUG:boto:Method: GET 2023-05-31T20:39:20.046 DEBUG:boto:Path: /pklxkd-58/obj00000000000000000271 2023-05-31T20:39:20.046 DEBUG:boto:Data: 2023-05-31T20:39:20.046 DEBUG:boto:Headers: {} 2023-05-31T20:39:20.047 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:39:20.047 DEBUG:boto:Port: 8000 2023-05-31T20:39:20.047 DEBUG:boto:Params: {} 2023-05-31T20:39:20.048 DEBUG:boto:Token: None 2023-05-31T20:39:20.048 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:39:20 GMT /pklxkd-58/obj00000000000000000271 2023-05-31T20:39:20.049 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:j/dzK8XrfDFcmxfOo15NSJI7rmg= 2023-05-31T20:39:20.049 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:39:20 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:j/dzK8XrfDFcmxfOo15NSJI7rmg=', 'Content-Length': '0'} 2023-05-31T20:39:20.052 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:38:20 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000082864955f25a44a3-006477b078-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:39:20 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:39:20.052 DEBUG:boto:path=/pklxkd-58/ 2023-05-31T20:39:20.053 DEBUG:boto:auth_path=/pklxkd-58/ 2023-05-31T20:39:20.053 DEBUG:boto:path=/pklxkd-58/?prefix=obj00000000000000000271 2023-05-31T20:39:20.053 DEBUG:boto:auth_path=/pklxkd-58/?prefix=obj00000000000000000271 2023-05-31T20:39:20.054 DEBUG:boto:Method: GET 2023-05-31T20:39:20.054 DEBUG:boto:Path: /pklxkd-58/?prefix=obj00000000000000000271 2023-05-31T20:39:20.055 DEBUG:boto:Data: 2023-05-31T20:39:20.055 DEBUG:boto:Headers: {} 2023-05-31T20:39:20.055 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:39:20.056 DEBUG:boto:Port: 8001 2023-05-31T20:39:20.056 DEBUG:boto:Params: {} 2023-05-31T20:39:20.057 DEBUG:boto:Token: None 2023-05-31T20:39:20.058 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:39:20 GMT /pklxkd-58/ 2023-05-31T20:39:20.058 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:g4vKvspu1xsYG4eXUMtZgdknNMM= 2023-05-31T20:39:20.059 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:39:20 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:g4vKvspu1xsYG4eXUMtZgdknNMM=', 'Content-Length': '0'} 2023-05-31T20:39:20.061 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000082c26ec9cae77469-006477b078-1c17-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:39:20 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:39:20.062 DEBUG:boto:b'pklxkd-58obj000000000000000002711000falseobj000000000000000002712023-05-31T20:38:20.240Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:39:20.063 DEBUG:boto:path=/pklxkd-58/ 2023-05-31T20:39:20.063 DEBUG:boto:auth_path=/pklxkd-58/ 2023-05-31T20:39:20.063 DEBUG:boto:path=/pklxkd-58/?prefix=obj00000000000000000271 2023-05-31T20:39:20.064 DEBUG:boto:auth_path=/pklxkd-58/?prefix=obj00000000000000000271 2023-05-31T20:39:20.064 DEBUG:boto:Method: GET 2023-05-31T20:39:20.065 DEBUG:boto:Path: /pklxkd-58/?prefix=obj00000000000000000271 2023-05-31T20:39:20.065 DEBUG:boto:Data: 2023-05-31T20:39:20.065 DEBUG:boto:Headers: {} 2023-05-31T20:39:20.066 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:39:20.066 DEBUG:boto:Port: 8000 2023-05-31T20:39:20.066 DEBUG:boto:Params: {} 2023-05-31T20:39:20.067 DEBUG:boto:Token: None 2023-05-31T20:39:20.068 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:39:20 GMT /pklxkd-58/ 2023-05-31T20:39:20.069 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:g4vKvspu1xsYG4eXUMtZgdknNMM= 2023-05-31T20:39:20.070 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:39:20 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:g4vKvspu1xsYG4eXUMtZgdknNMM=', 'Content-Length': '0'} 2023-05-31T20:39:20.075 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000004e11a9e19b0ad65b-006477b078-1630-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:39:20 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:39:20.076 DEBUG:boto:b'pklxkd-58obj000000000000000002711000falseobj000000000000000002712023-05-31T20:38:20.240Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:39:20.077 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000283 2023-05-31T20:39:20.078 DEBUG:boto:path=/pklxkd-58/obj00000000000000000283 2023-05-31T20:39:20.079 DEBUG:boto:auth_path=/pklxkd-58/obj00000000000000000283 2023-05-31T20:39:20.080 DEBUG:boto:path=/pklxkd-58/obj00000000000000000283?versionId=null 2023-05-31T20:39:20.080 DEBUG:boto:auth_path=/pklxkd-58/obj00000000000000000283?versionId=null 2023-05-31T20:39:20.081 DEBUG:boto:Method: GET 2023-05-31T20:39:20.082 DEBUG:boto:Path: /pklxkd-58/obj00000000000000000283?versionId=null 2023-05-31T20:39:20.082 DEBUG:boto:Data: 2023-05-31T20:39:20.083 DEBUG:boto:Headers: {} 2023-05-31T20:39:20.083 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:39:20.084 DEBUG:boto:Port: 8001 2023-05-31T20:39:20.084 DEBUG:boto:Params: {} 2023-05-31T20:39:20.085 DEBUG:boto:Token: None 2023-05-31T20:39:20.086 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:39:20 GMT /pklxkd-58/obj00000000000000000283?versionId=null 2023-05-31T20:39:20.086 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:PQJOKyyXD5Kn8i1UaGANCpuNGw8= 2023-05-31T20:39:20.087 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:39:20 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:PQJOKyyXD5Kn8i1UaGANCpuNGw8=', 'Content-Length': '0'} 2023-05-31T20:39:20.090 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:38:20 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006dccaf557ae52815-006477b078-1c17-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:39:20 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:39:20.091 DEBUG:boto:path=/pklxkd-58/obj00000000000000000283 2023-05-31T20:39:20.092 DEBUG:boto:auth_path=/pklxkd-58/obj00000000000000000283 2023-05-31T20:39:20.092 DEBUG:boto:path=/pklxkd-58/obj00000000000000000283?versionId=null 2023-05-31T20:39:20.093 DEBUG:boto:auth_path=/pklxkd-58/obj00000000000000000283?versionId=null 2023-05-31T20:39:20.093 DEBUG:boto:Method: GET 2023-05-31T20:39:20.094 DEBUG:boto:Path: /pklxkd-58/obj00000000000000000283?versionId=null 2023-05-31T20:39:20.094 DEBUG:boto:Data: 2023-05-31T20:39:20.094 DEBUG:boto:Headers: {} 2023-05-31T20:39:20.095 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:39:20.095 DEBUG:boto:Port: 8000 2023-05-31T20:39:20.095 DEBUG:boto:Params: {} 2023-05-31T20:39:20.096 DEBUG:boto:Token: None 2023-05-31T20:39:20.096 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:39:20 GMT /pklxkd-58/obj00000000000000000283?versionId=null 2023-05-31T20:39:20.097 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:PQJOKyyXD5Kn8i1UaGANCpuNGw8= 2023-05-31T20:39:20.097 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:39:20 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:PQJOKyyXD5Kn8i1UaGANCpuNGw8=', 'Content-Length': '0'} 2023-05-31T20:39:20.099 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:38:20 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000136df7656362c96b-006477b078-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:39:20 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:39:20.100 DEBUG:boto:path=/pklxkd-58/obj00000000000000000283 2023-05-31T20:39:20.100 DEBUG:boto:auth_path=/pklxkd-58/obj00000000000000000283 2023-05-31T20:39:20.100 DEBUG:boto:path=/pklxkd-58/obj00000000000000000283?versionId=null 2023-05-31T20:39:20.101 DEBUG:boto:auth_path=/pklxkd-58/obj00000000000000000283?versionId=null 2023-05-31T20:39:20.101 DEBUG:boto:Method: HEAD 2023-05-31T20:39:20.101 DEBUG:boto:Path: /pklxkd-58/obj00000000000000000283?versionId=null 2023-05-31T20:39:20.102 DEBUG:boto:Data: 2023-05-31T20:39:20.102 DEBUG:boto:Headers: {} 2023-05-31T20:39:20.102 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:39:20.103 DEBUG:boto:Port: 8001 2023-05-31T20:39:20.103 DEBUG:boto:Params: {} 2023-05-31T20:39:20.103 DEBUG:boto:Token: None 2023-05-31T20:39:20.104 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:39:20 GMT /pklxkd-58/obj00000000000000000283?versionId=null 2023-05-31T20:39:20.104 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:xLAkRhy0sAgITdBW8QL3yc0gybs= 2023-05-31T20:39:20.105 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:39:20 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:xLAkRhy0sAgITdBW8QL3yc0gybs=', 'Content-Length': '0'} 2023-05-31T20:39:20.106 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:38:20 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c47a8e1f822a8da1-006477b078-1c17-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:39:20 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:39:20.107 DEBUG:boto:path=/pklxkd-58/obj00000000000000000283 2023-05-31T20:39:20.107 DEBUG:boto:auth_path=/pklxkd-58/obj00000000000000000283 2023-05-31T20:39:20.108 DEBUG:boto:path=/pklxkd-58/obj00000000000000000283?versionId=null 2023-05-31T20:39:20.108 DEBUG:boto:auth_path=/pklxkd-58/obj00000000000000000283?versionId=null 2023-05-31T20:39:20.109 DEBUG:boto:Method: HEAD 2023-05-31T20:39:20.109 DEBUG:boto:Path: /pklxkd-58/obj00000000000000000283?versionId=null 2023-05-31T20:39:20.109 DEBUG:boto:Data: 2023-05-31T20:39:20.110 DEBUG:boto:Headers: {} 2023-05-31T20:39:20.110 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:39:20.110 DEBUG:boto:Port: 8000 2023-05-31T20:39:20.111 DEBUG:boto:Params: {} 2023-05-31T20:39:20.111 DEBUG:boto:Token: None 2023-05-31T20:39:20.112 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:39:20 GMT /pklxkd-58/obj00000000000000000283?versionId=null 2023-05-31T20:39:20.112 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:xLAkRhy0sAgITdBW8QL3yc0gybs= 2023-05-31T20:39:20.112 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:39:20 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:xLAkRhy0sAgITdBW8QL3yc0gybs=', 'Content-Length': '0'} 2023-05-31T20:39:20.114 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:38:20 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000056287a9c1e72a410-006477b078-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:39:20 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:39:20.115 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000283 2023-05-31T20:39:20.115 DEBUG:boto:path=/pklxkd-58/obj00000000000000000283 2023-05-31T20:39:20.116 DEBUG:boto:auth_path=/pklxkd-58/obj00000000000000000283 2023-05-31T20:39:20.116 DEBUG:boto:path=/pklxkd-58/obj00000000000000000283?versionId=null 2023-05-31T20:39:20.117 DEBUG:boto:auth_path=/pklxkd-58/obj00000000000000000283?versionId=null 2023-05-31T20:39:20.117 DEBUG:boto:Method: GET 2023-05-31T20:39:20.117 DEBUG:boto:Path: /pklxkd-58/obj00000000000000000283?versionId=null 2023-05-31T20:39:20.118 DEBUG:boto:Data: 2023-05-31T20:39:20.118 DEBUG:boto:Headers: {} 2023-05-31T20:39:20.118 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:39:20.119 DEBUG:boto:Port: 8001 2023-05-31T20:39:20.119 DEBUG:boto:Params: {} 2023-05-31T20:39:20.119 DEBUG:boto:Token: None 2023-05-31T20:39:20.120 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:39:20 GMT /pklxkd-58/obj00000000000000000283?versionId=null 2023-05-31T20:39:20.120 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:PQJOKyyXD5Kn8i1UaGANCpuNGw8= 2023-05-31T20:39:20.121 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:39:20 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:PQJOKyyXD5Kn8i1UaGANCpuNGw8=', 'Content-Length': '0'} 2023-05-31T20:39:20.123 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:38:20 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000dc5529a644f1b01f-006477b078-1c17-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:39:20 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:39:20.123 DEBUG:boto:path=/pklxkd-58/obj00000000000000000283 2023-05-31T20:39:20.124 DEBUG:boto:auth_path=/pklxkd-58/obj00000000000000000283 2023-05-31T20:39:20.125 DEBUG:boto:path=/pklxkd-58/obj00000000000000000283?versionId=null 2023-05-31T20:39:20.125 DEBUG:boto:auth_path=/pklxkd-58/obj00000000000000000283?versionId=null 2023-05-31T20:39:20.126 DEBUG:boto:Method: GET 2023-05-31T20:39:20.127 DEBUG:boto:Path: /pklxkd-58/obj00000000000000000283?versionId=null 2023-05-31T20:39:20.128 DEBUG:boto:Data: 2023-05-31T20:39:20.129 DEBUG:boto:Headers: {} 2023-05-31T20:39:20.129 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:39:20.130 DEBUG:boto:Port: 8000 2023-05-31T20:39:20.131 DEBUG:boto:Params: {} 2023-05-31T20:39:20.132 DEBUG:boto:Token: None 2023-05-31T20:39:20.138 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:39:20 GMT /pklxkd-58/obj00000000000000000283?versionId=null 2023-05-31T20:39:20.141 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:PQJOKyyXD5Kn8i1UaGANCpuNGw8= 2023-05-31T20:39:20.141 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:39:20 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:PQJOKyyXD5Kn8i1UaGANCpuNGw8=', 'Content-Length': '0'} 2023-05-31T20:39:20.143 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:38:20 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000dbddff9127f6c693-006477b078-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:39:20 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:39:20.144 DEBUG:boto:path=/pklxkd-58/ 2023-05-31T20:39:20.144 DEBUG:boto:auth_path=/pklxkd-58/ 2023-05-31T20:39:20.145 DEBUG:boto:path=/pklxkd-58/?prefix=obj00000000000000000283 2023-05-31T20:39:20.145 DEBUG:boto:auth_path=/pklxkd-58/?prefix=obj00000000000000000283 2023-05-31T20:39:20.145 DEBUG:boto:Method: GET 2023-05-31T20:39:20.146 DEBUG:boto:Path: /pklxkd-58/?prefix=obj00000000000000000283 2023-05-31T20:39:20.146 DEBUG:boto:Data: 2023-05-31T20:39:20.146 DEBUG:boto:Headers: {} 2023-05-31T20:39:20.147 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:39:20.147 DEBUG:boto:Port: 8001 2023-05-31T20:39:20.147 DEBUG:boto:Params: {} 2023-05-31T20:39:20.148 DEBUG:boto:Token: None 2023-05-31T20:39:20.148 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:39:20 GMT /pklxkd-58/ 2023-05-31T20:39:20.149 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:g4vKvspu1xsYG4eXUMtZgdknNMM= 2023-05-31T20:39:20.149 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:39:20 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:g4vKvspu1xsYG4eXUMtZgdknNMM=', 'Content-Length': '0'} 2023-05-31T20:39:20.152 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000008f48f26ebf26fdf4-006477b078-1c17-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:39:20 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:39:20.152 DEBUG:boto:b'pklxkd-58obj000000000000000002831000falseobj000000000000000002832023-05-31T20:38:20.953Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:39:20.153 DEBUG:boto:path=/pklxkd-58/ 2023-05-31T20:39:20.153 DEBUG:boto:auth_path=/pklxkd-58/ 2023-05-31T20:39:20.154 DEBUG:boto:path=/pklxkd-58/?prefix=obj00000000000000000283 2023-05-31T20:39:20.154 DEBUG:boto:auth_path=/pklxkd-58/?prefix=obj00000000000000000283 2023-05-31T20:39:20.155 DEBUG:boto:Method: GET 2023-05-31T20:39:20.155 DEBUG:boto:Path: /pklxkd-58/?prefix=obj00000000000000000283 2023-05-31T20:39:20.155 DEBUG:boto:Data: 2023-05-31T20:39:20.156 DEBUG:boto:Headers: {} 2023-05-31T20:39:20.156 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:39:20.156 DEBUG:boto:Port: 8000 2023-05-31T20:39:20.157 DEBUG:boto:Params: {} 2023-05-31T20:39:20.157 DEBUG:boto:Token: None 2023-05-31T20:39:20.158 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:39:20 GMT /pklxkd-58/ 2023-05-31T20:39:20.158 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:g4vKvspu1xsYG4eXUMtZgdknNMM= 2023-05-31T20:39:20.158 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:39:20 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:g4vKvspu1xsYG4eXUMtZgdknNMM=', 'Content-Length': '0'} 2023-05-31T20:39:20.162 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000008981106ccab89d7-006477b078-1630-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:39:20 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:39:20.163 DEBUG:boto:b'pklxkd-58obj000000000000000002831000falseobj000000000000000002832023-05-31T20:38:20.953Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:39:20.164 DEBUG:boto:path=/pklxkd-58/obj00000000000000000283 2023-05-31T20:39:20.164 DEBUG:boto:auth_path=/pklxkd-58/obj00000000000000000283 2023-05-31T20:39:20.165 DEBUG:boto:Method: HEAD 2023-05-31T20:39:20.165 DEBUG:boto:Path: /pklxkd-58/obj00000000000000000283 2023-05-31T20:39:20.165 DEBUG:boto:Data: 2023-05-31T20:39:20.166 DEBUG:boto:Headers: {} 2023-05-31T20:39:20.166 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:39:20.166 DEBUG:boto:Port: 8001 2023-05-31T20:39:20.167 DEBUG:boto:Params: {} 2023-05-31T20:39:20.167 DEBUG:boto:Token: None 2023-05-31T20:39:20.167 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:39:20 GMT /pklxkd-58/obj00000000000000000283 2023-05-31T20:39:20.168 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:lHWHanj7LyinBLubhpcEKbobJdU= 2023-05-31T20:39:20.168 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:39:20 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:lHWHanj7LyinBLubhpcEKbobJdU=', 'Content-Length': '0'} 2023-05-31T20:39:20.171 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:38:20 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007b76d206fdd411e8-006477b078-1c17-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:39:20 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:39:20.172 DEBUG:boto:path=/pklxkd-58/obj00000000000000000283 2023-05-31T20:39:20.173 DEBUG:boto:auth_path=/pklxkd-58/obj00000000000000000283 2023-05-31T20:39:20.175 DEBUG:boto:Method: HEAD 2023-05-31T20:39:20.175 DEBUG:boto:Path: /pklxkd-58/obj00000000000000000283 2023-05-31T20:39:20.176 DEBUG:boto:Data: 2023-05-31T20:39:20.176 DEBUG:boto:Headers: {} 2023-05-31T20:39:20.177 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:39:20.177 DEBUG:boto:Port: 8000 2023-05-31T20:39:20.177 DEBUG:boto:Params: {} 2023-05-31T20:39:20.178 DEBUG:boto:Token: None 2023-05-31T20:39:20.178 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:39:20 GMT /pklxkd-58/obj00000000000000000283 2023-05-31T20:39:20.178 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:lHWHanj7LyinBLubhpcEKbobJdU= 2023-05-31T20:39:20.179 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:39:20 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:lHWHanj7LyinBLubhpcEKbobJdU=', 'Content-Length': '0'} 2023-05-31T20:39:20.181 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:38:20 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000dae36f15a799cf5f-006477b078-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:39:20 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:39:20.182 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000283 2023-05-31T20:39:20.183 DEBUG:boto:path=/pklxkd-58/obj00000000000000000283 2023-05-31T20:39:20.184 DEBUG:boto:auth_path=/pklxkd-58/obj00000000000000000283 2023-05-31T20:39:20.185 DEBUG:boto:Method: GET 2023-05-31T20:39:20.186 DEBUG:boto:Path: /pklxkd-58/obj00000000000000000283 2023-05-31T20:39:20.187 DEBUG:boto:Data: 2023-05-31T20:39:20.188 DEBUG:boto:Headers: {} 2023-05-31T20:39:20.188 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:39:20.189 DEBUG:boto:Port: 8001 2023-05-31T20:39:20.190 DEBUG:boto:Params: {} 2023-05-31T20:39:20.190 DEBUG:boto:Token: None 2023-05-31T20:39:20.191 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:39:20 GMT /pklxkd-58/obj00000000000000000283 2023-05-31T20:39:20.192 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:2FxFUJqPSvgJXsVR4rXikMiKQkQ= 2023-05-31T20:39:20.193 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:39:20 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:2FxFUJqPSvgJXsVR4rXikMiKQkQ=', 'Content-Length': '0'} 2023-05-31T20:39:20.195 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:38:20 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003db849d5250e5ad2-006477b078-1c17-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:39:20 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:39:20.196 DEBUG:boto:path=/pklxkd-58/obj00000000000000000283 2023-05-31T20:39:20.196 DEBUG:boto:auth_path=/pklxkd-58/obj00000000000000000283 2023-05-31T20:39:20.197 DEBUG:boto:Method: GET 2023-05-31T20:39:20.198 DEBUG:boto:Path: /pklxkd-58/obj00000000000000000283 2023-05-31T20:39:20.198 DEBUG:boto:Data: 2023-05-31T20:39:20.199 DEBUG:boto:Headers: {} 2023-05-31T20:39:20.199 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:39:20.200 DEBUG:boto:Port: 8000 2023-05-31T20:39:20.201 DEBUG:boto:Params: {} 2023-05-31T20:39:20.202 DEBUG:boto:Token: None 2023-05-31T20:39:20.202 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:39:20 GMT /pklxkd-58/obj00000000000000000283 2023-05-31T20:39:20.203 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:2FxFUJqPSvgJXsVR4rXikMiKQkQ= 2023-05-31T20:39:20.204 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:39:20 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:2FxFUJqPSvgJXsVR4rXikMiKQkQ=', 'Content-Length': '0'} 2023-05-31T20:39:20.207 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:38:20 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003135585985a20c24-006477b078-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:39:20 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:39:20.248 DEBUG:boto:path=/pklxkd-58/ 2023-05-31T20:39:20.250 DEBUG:boto:auth_path=/pklxkd-58/ 2023-05-31T20:39:20.251 DEBUG:boto:path=/pklxkd-58/?prefix=obj00000000000000000283 2023-05-31T20:39:20.251 DEBUG:boto:auth_path=/pklxkd-58/?prefix=obj00000000000000000283 2023-05-31T20:39:20.252 DEBUG:boto:Method: GET 2023-05-31T20:39:20.253 DEBUG:boto:Path: /pklxkd-58/?prefix=obj00000000000000000283 2023-05-31T20:39:20.254 DEBUG:boto:Data: 2023-05-31T20:39:20.257 DEBUG:boto:Headers: {} 2023-05-31T20:39:20.258 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:39:20.258 DEBUG:boto:Port: 8001 2023-05-31T20:39:20.259 DEBUG:boto:Params: {} 2023-05-31T20:39:20.260 DEBUG:boto:Token: None 2023-05-31T20:39:20.261 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:39:20 GMT /pklxkd-58/ 2023-05-31T20:39:20.262 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:g4vKvspu1xsYG4eXUMtZgdknNMM= 2023-05-31T20:39:20.264 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:39:20 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:g4vKvspu1xsYG4eXUMtZgdknNMM=', 'Content-Length': '0'} 2023-05-31T20:39:20.267 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a5be51cecce30863-006477b078-1c17-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:39:20 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:39:20.268 DEBUG:boto:b'pklxkd-58obj000000000000000002831000falseobj000000000000000002832023-05-31T20:38:20.953Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:39:20.269 DEBUG:boto:path=/pklxkd-58/ 2023-05-31T20:39:20.271 DEBUG:boto:auth_path=/pklxkd-58/ 2023-05-31T20:39:20.272 DEBUG:boto:path=/pklxkd-58/?prefix=obj00000000000000000283 2023-05-31T20:39:20.276 DEBUG:boto:auth_path=/pklxkd-58/?prefix=obj00000000000000000283 2023-05-31T20:39:20.277 DEBUG:boto:Method: GET 2023-05-31T20:39:20.278 DEBUG:boto:Path: /pklxkd-58/?prefix=obj00000000000000000283 2023-05-31T20:39:20.279 DEBUG:boto:Data: 2023-05-31T20:39:20.279 DEBUG:boto:Headers: {} 2023-05-31T20:39:20.280 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:39:20.281 DEBUG:boto:Port: 8000 2023-05-31T20:39:20.282 DEBUG:boto:Params: {} 2023-05-31T20:39:20.284 DEBUG:boto:Token: None 2023-05-31T20:39:20.285 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:39:20 GMT /pklxkd-58/ 2023-05-31T20:39:20.286 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:g4vKvspu1xsYG4eXUMtZgdknNMM= 2023-05-31T20:39:20.287 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:39:20 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:g4vKvspu1xsYG4eXUMtZgdknNMM=', 'Content-Length': '0'} 2023-05-31T20:39:20.292 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000dccb1d4cc2b85239-006477b078-1630-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:39:20 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:39:20.292 DEBUG:boto:b'pklxkd-58obj000000000000000002831000falseobj000000000000000002832023-05-31T20:38:20.953Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:39:20.293 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000317 2023-05-31T20:39:20.294 DEBUG:boto:path=/pklxkd-58/obj00000000000000000317 2023-05-31T20:39:20.294 DEBUG:boto:auth_path=/pklxkd-58/obj00000000000000000317 2023-05-31T20:39:20.294 DEBUG:boto:path=/pklxkd-58/obj00000000000000000317?versionId=null 2023-05-31T20:39:20.295 DEBUG:boto:auth_path=/pklxkd-58/obj00000000000000000317?versionId=null 2023-05-31T20:39:20.295 DEBUG:boto:Method: GET 2023-05-31T20:39:20.296 DEBUG:boto:Path: /pklxkd-58/obj00000000000000000317?versionId=null 2023-05-31T20:39:20.296 DEBUG:boto:Data: 2023-05-31T20:39:20.296 DEBUG:boto:Headers: {} 2023-05-31T20:39:20.297 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:39:20.297 DEBUG:boto:Port: 8001 2023-05-31T20:39:20.297 DEBUG:boto:Params: {} 2023-05-31T20:39:20.298 DEBUG:boto:Token: None 2023-05-31T20:39:20.298 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:39:20 GMT /pklxkd-58/obj00000000000000000317?versionId=null 2023-05-31T20:39:20.299 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:CjpfdIjoDjypZNJ+IFlbbS0FVyU= 2023-05-31T20:39:20.300 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:39:20 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:CjpfdIjoDjypZNJ+IFlbbS0FVyU=', 'Content-Length': '0'} 2023-05-31T20:39:20.302 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:38:20 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000043da7b456a3d3d23-006477b078-1c17-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:39:20 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:39:20.302 DEBUG:boto:path=/pklxkd-58/obj00000000000000000317 2023-05-31T20:39:20.303 DEBUG:boto:auth_path=/pklxkd-58/obj00000000000000000317 2023-05-31T20:39:20.303 DEBUG:boto:path=/pklxkd-58/obj00000000000000000317?versionId=null 2023-05-31T20:39:20.304 DEBUG:boto:auth_path=/pklxkd-58/obj00000000000000000317?versionId=null 2023-05-31T20:39:20.304 DEBUG:boto:Method: GET 2023-05-31T20:39:20.304 DEBUG:boto:Path: /pklxkd-58/obj00000000000000000317?versionId=null 2023-05-31T20:39:20.305 DEBUG:boto:Data: 2023-05-31T20:39:20.305 DEBUG:boto:Headers: {} 2023-05-31T20:39:20.305 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:39:20.306 DEBUG:boto:Port: 8000 2023-05-31T20:39:20.306 DEBUG:boto:Params: {} 2023-05-31T20:39:20.306 DEBUG:boto:Token: None 2023-05-31T20:39:20.307 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:39:20 GMT /pklxkd-58/obj00000000000000000317?versionId=null 2023-05-31T20:39:20.307 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:CjpfdIjoDjypZNJ+IFlbbS0FVyU= 2023-05-31T20:39:20.308 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:39:20 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:CjpfdIjoDjypZNJ+IFlbbS0FVyU=', 'Content-Length': '0'} 2023-05-31T20:39:20.310 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:38:20 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000eb22bf86c010f3b3-006477b078-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:39:20 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:39:20.311 DEBUG:boto:path=/pklxkd-58/obj00000000000000000317 2023-05-31T20:39:20.311 DEBUG:boto:auth_path=/pklxkd-58/obj00000000000000000317 2023-05-31T20:39:20.311 DEBUG:boto:path=/pklxkd-58/obj00000000000000000317?versionId=null 2023-05-31T20:39:20.312 DEBUG:boto:auth_path=/pklxkd-58/obj00000000000000000317?versionId=null 2023-05-31T20:39:20.312 DEBUG:boto:Method: HEAD 2023-05-31T20:39:20.312 DEBUG:boto:Path: /pklxkd-58/obj00000000000000000317?versionId=null 2023-05-31T20:39:20.313 DEBUG:boto:Data: 2023-05-31T20:39:20.313 DEBUG:boto:Headers: {} 2023-05-31T20:39:20.313 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:39:20.314 DEBUG:boto:Port: 8001 2023-05-31T20:39:20.314 DEBUG:boto:Params: {} 2023-05-31T20:39:20.314 DEBUG:boto:Token: None 2023-05-31T20:39:20.315 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:39:20 GMT /pklxkd-58/obj00000000000000000317?versionId=null 2023-05-31T20:39:20.315 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:Z+UBsnW4JyOCBN9lngRL6wtlXQw= 2023-05-31T20:39:20.316 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:39:20 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:Z+UBsnW4JyOCBN9lngRL6wtlXQw=', 'Content-Length': '0'} 2023-05-31T20:39:20.317 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:38:20 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000bef3024a8c6ddbbc-006477b078-1c17-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:39:20 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:39:20.318 DEBUG:boto:path=/pklxkd-58/obj00000000000000000317 2023-05-31T20:39:20.318 DEBUG:boto:auth_path=/pklxkd-58/obj00000000000000000317 2023-05-31T20:39:20.319 DEBUG:boto:path=/pklxkd-58/obj00000000000000000317?versionId=null 2023-05-31T20:39:20.319 DEBUG:boto:auth_path=/pklxkd-58/obj00000000000000000317?versionId=null 2023-05-31T20:39:20.319 DEBUG:boto:Method: HEAD 2023-05-31T20:39:20.320 DEBUG:boto:Path: /pklxkd-58/obj00000000000000000317?versionId=null 2023-05-31T20:39:20.320 DEBUG:boto:Data: 2023-05-31T20:39:20.321 DEBUG:boto:Headers: {} 2023-05-31T20:39:20.321 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:39:20.321 DEBUG:boto:Port: 8000 2023-05-31T20:39:20.322 DEBUG:boto:Params: {} 2023-05-31T20:39:20.322 DEBUG:boto:Token: None 2023-05-31T20:39:20.322 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:39:20 GMT /pklxkd-58/obj00000000000000000317?versionId=null 2023-05-31T20:39:20.323 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:Z+UBsnW4JyOCBN9lngRL6wtlXQw= 2023-05-31T20:39:20.323 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:39:20 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:Z+UBsnW4JyOCBN9lngRL6wtlXQw=', 'Content-Length': '0'} 2023-05-31T20:39:20.325 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:38:20 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ede07f38a40275f7-006477b078-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:39:20 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:39:20.325 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000317 2023-05-31T20:39:20.326 DEBUG:boto:path=/pklxkd-58/obj00000000000000000317 2023-05-31T20:39:20.326 DEBUG:boto:auth_path=/pklxkd-58/obj00000000000000000317 2023-05-31T20:39:20.327 DEBUG:boto:path=/pklxkd-58/obj00000000000000000317?versionId=null 2023-05-31T20:39:20.327 DEBUG:boto:auth_path=/pklxkd-58/obj00000000000000000317?versionId=null 2023-05-31T20:39:20.327 DEBUG:boto:Method: GET 2023-05-31T20:39:20.328 DEBUG:boto:Path: /pklxkd-58/obj00000000000000000317?versionId=null 2023-05-31T20:39:20.328 DEBUG:boto:Data: 2023-05-31T20:39:20.328 DEBUG:boto:Headers: {} 2023-05-31T20:39:20.329 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:39:20.329 DEBUG:boto:Port: 8001 2023-05-31T20:39:20.329 DEBUG:boto:Params: {} 2023-05-31T20:39:20.330 DEBUG:boto:Token: None 2023-05-31T20:39:20.330 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:39:20 GMT /pklxkd-58/obj00000000000000000317?versionId=null 2023-05-31T20:39:20.331 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:CjpfdIjoDjypZNJ+IFlbbS0FVyU= 2023-05-31T20:39:20.331 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:39:20 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:CjpfdIjoDjypZNJ+IFlbbS0FVyU=', 'Content-Length': '0'} 2023-05-31T20:39:20.333 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:38:20 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005dffcf0812e498f9-006477b078-1c17-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:39:20 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:39:20.334 DEBUG:boto:path=/pklxkd-58/obj00000000000000000317 2023-05-31T20:39:20.334 DEBUG:boto:auth_path=/pklxkd-58/obj00000000000000000317 2023-05-31T20:39:20.334 DEBUG:boto:path=/pklxkd-58/obj00000000000000000317?versionId=null 2023-05-31T20:39:20.335 DEBUG:boto:auth_path=/pklxkd-58/obj00000000000000000317?versionId=null 2023-05-31T20:39:20.335 DEBUG:boto:Method: GET 2023-05-31T20:39:20.336 DEBUG:boto:Path: /pklxkd-58/obj00000000000000000317?versionId=null 2023-05-31T20:39:20.336 DEBUG:boto:Data: 2023-05-31T20:39:20.336 DEBUG:boto:Headers: {} 2023-05-31T20:39:20.337 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:39:20.337 DEBUG:boto:Port: 8000 2023-05-31T20:39:20.337 DEBUG:boto:Params: {} 2023-05-31T20:39:20.338 DEBUG:boto:Token: None 2023-05-31T20:39:20.338 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:39:20 GMT /pklxkd-58/obj00000000000000000317?versionId=null 2023-05-31T20:39:20.339 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:CjpfdIjoDjypZNJ+IFlbbS0FVyU= 2023-05-31T20:39:20.339 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:39:20 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:CjpfdIjoDjypZNJ+IFlbbS0FVyU=', 'Content-Length': '0'} 2023-05-31T20:39:20.341 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:38:20 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004c18a6d39c7fc849-006477b078-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:39:20 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:39:20.341 DEBUG:boto:path=/pklxkd-58/ 2023-05-31T20:39:20.342 DEBUG:boto:auth_path=/pklxkd-58/ 2023-05-31T20:39:20.342 DEBUG:boto:path=/pklxkd-58/?prefix=obj00000000000000000317 2023-05-31T20:39:20.343 DEBUG:boto:auth_path=/pklxkd-58/?prefix=obj00000000000000000317 2023-05-31T20:39:20.343 DEBUG:boto:Method: GET 2023-05-31T20:39:20.343 DEBUG:boto:Path: /pklxkd-58/?prefix=obj00000000000000000317 2023-05-31T20:39:20.344 DEBUG:boto:Data: 2023-05-31T20:39:20.344 DEBUG:boto:Headers: {} 2023-05-31T20:39:20.344 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:39:20.345 DEBUG:boto:Port: 8001 2023-05-31T20:39:20.345 DEBUG:boto:Params: {} 2023-05-31T20:39:20.346 DEBUG:boto:Token: None 2023-05-31T20:39:20.346 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:39:20 GMT /pklxkd-58/ 2023-05-31T20:39:20.346 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:g4vKvspu1xsYG4eXUMtZgdknNMM= 2023-05-31T20:39:20.347 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:39:20 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:g4vKvspu1xsYG4eXUMtZgdknNMM=', 'Content-Length': '0'} 2023-05-31T20:39:20.349 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000000d5cdf1ce304a4bd-006477b078-1c17-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:39:20 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:39:20.397 DEBUG:boto:b'pklxkd-58obj000000000000000003171000falseobj000000000000000003172023-05-31T20:38:20.065Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:39:20.398 DEBUG:boto:path=/pklxkd-58/ 2023-05-31T20:39:20.398 DEBUG:boto:auth_path=/pklxkd-58/ 2023-05-31T20:39:20.399 DEBUG:boto:path=/pklxkd-58/?prefix=obj00000000000000000317 2023-05-31T20:39:20.399 DEBUG:boto:auth_path=/pklxkd-58/?prefix=obj00000000000000000317 2023-05-31T20:39:20.400 DEBUG:boto:Method: GET 2023-05-31T20:39:20.400 DEBUG:boto:Path: /pklxkd-58/?prefix=obj00000000000000000317 2023-05-31T20:39:20.400 DEBUG:boto:Data: 2023-05-31T20:39:20.401 DEBUG:boto:Headers: {} 2023-05-31T20:39:20.401 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:39:20.402 DEBUG:boto:Port: 8000 2023-05-31T20:39:20.402 DEBUG:boto:Params: {} 2023-05-31T20:39:20.402 DEBUG:boto:Token: None 2023-05-31T20:39:20.403 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:39:20 GMT /pklxkd-58/ 2023-05-31T20:39:20.403 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:g4vKvspu1xsYG4eXUMtZgdknNMM= 2023-05-31T20:39:20.404 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:39:20 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:g4vKvspu1xsYG4eXUMtZgdknNMM=', 'Content-Length': '0'} 2023-05-31T20:39:20.408 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000600b7a9b23106c90-006477b078-1630-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:39:20 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:39:20.408 DEBUG:boto:b'pklxkd-58obj000000000000000003171000falseobj000000000000000003172023-05-31T20:38:20.065Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:39:20.409 DEBUG:boto:path=/pklxkd-58/obj00000000000000000317 2023-05-31T20:39:20.409 DEBUG:boto:auth_path=/pklxkd-58/obj00000000000000000317 2023-05-31T20:39:20.410 DEBUG:boto:Method: HEAD 2023-05-31T20:39:20.410 DEBUG:boto:Path: /pklxkd-58/obj00000000000000000317 2023-05-31T20:39:20.411 DEBUG:boto:Data: 2023-05-31T20:39:20.411 DEBUG:boto:Headers: {} 2023-05-31T20:39:20.411 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:39:20.412 DEBUG:boto:Port: 8001 2023-05-31T20:39:20.412 DEBUG:boto:Params: {} 2023-05-31T20:39:20.412 DEBUG:boto:Token: None 2023-05-31T20:39:20.413 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:39:20 GMT /pklxkd-58/obj00000000000000000317 2023-05-31T20:39:20.413 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:Ll803ZU3P5YdEbfjNwmaYKcGir8= 2023-05-31T20:39:20.414 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:39:20 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:Ll803ZU3P5YdEbfjNwmaYKcGir8=', 'Content-Length': '0'} 2023-05-31T20:39:20.415 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:38:20 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005b01a29e24b18143-006477b078-1c17-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:39:20 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:39:20.416 DEBUG:boto:path=/pklxkd-58/obj00000000000000000317 2023-05-31T20:39:20.416 DEBUG:boto:auth_path=/pklxkd-58/obj00000000000000000317 2023-05-31T20:39:20.417 DEBUG:boto:Method: HEAD 2023-05-31T20:39:20.417 DEBUG:boto:Path: /pklxkd-58/obj00000000000000000317 2023-05-31T20:39:20.417 DEBUG:boto:Data: 2023-05-31T20:39:20.418 DEBUG:boto:Headers: {} 2023-05-31T20:39:20.418 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:39:20.419 DEBUG:boto:Port: 8000 2023-05-31T20:39:20.419 DEBUG:boto:Params: {} 2023-05-31T20:39:20.419 DEBUG:boto:Token: None 2023-05-31T20:39:20.420 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:39:20 GMT /pklxkd-58/obj00000000000000000317 2023-05-31T20:39:20.420 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:Ll803ZU3P5YdEbfjNwmaYKcGir8= 2023-05-31T20:39:20.420 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:39:20 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:Ll803ZU3P5YdEbfjNwmaYKcGir8=', 'Content-Length': '0'} 2023-05-31T20:39:20.423 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:38:20 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000071000befb020bde1-006477b078-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:39:20 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:39:20.424 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000317 2023-05-31T20:39:20.424 DEBUG:boto:path=/pklxkd-58/obj00000000000000000317 2023-05-31T20:39:20.425 DEBUG:boto:auth_path=/pklxkd-58/obj00000000000000000317 2023-05-31T20:39:20.425 DEBUG:boto:Method: GET 2023-05-31T20:39:20.426 DEBUG:boto:Path: /pklxkd-58/obj00000000000000000317 2023-05-31T20:39:20.426 DEBUG:boto:Data: 2023-05-31T20:39:20.427 DEBUG:boto:Headers: {} 2023-05-31T20:39:20.427 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:39:20.428 DEBUG:boto:Port: 8001 2023-05-31T20:39:20.428 DEBUG:boto:Params: {} 2023-05-31T20:39:20.428 DEBUG:boto:Token: None 2023-05-31T20:39:20.429 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:39:20 GMT /pklxkd-58/obj00000000000000000317 2023-05-31T20:39:20.429 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:U8ejVfaeRiX1Jq3ZNqEN+ZEqoUs= 2023-05-31T20:39:20.430 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:39:20 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:U8ejVfaeRiX1Jq3ZNqEN+ZEqoUs=', 'Content-Length': '0'} 2023-05-31T20:39:20.431 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:38:20 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ac6f94f2f17c254e-006477b078-1c17-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:39:20 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:39:20.432 DEBUG:boto:path=/pklxkd-58/obj00000000000000000317 2023-05-31T20:39:20.433 DEBUG:boto:auth_path=/pklxkd-58/obj00000000000000000317 2023-05-31T20:39:20.433 DEBUG:boto:Method: GET 2023-05-31T20:39:20.433 DEBUG:boto:Path: /pklxkd-58/obj00000000000000000317 2023-05-31T20:39:20.434 DEBUG:boto:Data: 2023-05-31T20:39:20.434 DEBUG:boto:Headers: {} 2023-05-31T20:39:20.434 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:39:20.435 DEBUG:boto:Port: 8000 2023-05-31T20:39:20.435 DEBUG:boto:Params: {} 2023-05-31T20:39:20.436 DEBUG:boto:Token: None 2023-05-31T20:39:20.436 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:39:20 GMT /pklxkd-58/obj00000000000000000317 2023-05-31T20:39:20.437 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:U8ejVfaeRiX1Jq3ZNqEN+ZEqoUs= 2023-05-31T20:39:20.437 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:39:20 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:U8ejVfaeRiX1Jq3ZNqEN+ZEqoUs=', 'Content-Length': '0'} 2023-05-31T20:39:20.439 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:38:20 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000062f7c0c0e341e6e6-006477b078-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:39:20 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:39:20.440 DEBUG:boto:path=/pklxkd-58/ 2023-05-31T20:39:20.440 DEBUG:boto:auth_path=/pklxkd-58/ 2023-05-31T20:39:20.441 DEBUG:boto:path=/pklxkd-58/?prefix=obj00000000000000000317 2023-05-31T20:39:20.441 DEBUG:boto:auth_path=/pklxkd-58/?prefix=obj00000000000000000317 2023-05-31T20:39:20.441 DEBUG:boto:Method: GET 2023-05-31T20:39:20.442 DEBUG:boto:Path: /pklxkd-58/?prefix=obj00000000000000000317 2023-05-31T20:39:20.442 DEBUG:boto:Data: 2023-05-31T20:39:20.442 DEBUG:boto:Headers: {} 2023-05-31T20:39:20.443 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:39:20.443 DEBUG:boto:Port: 8001 2023-05-31T20:39:20.443 DEBUG:boto:Params: {} 2023-05-31T20:39:20.444 DEBUG:boto:Token: None 2023-05-31T20:39:20.444 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:39:20 GMT /pklxkd-58/ 2023-05-31T20:39:20.445 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:g4vKvspu1xsYG4eXUMtZgdknNMM= 2023-05-31T20:39:20.446 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:39:20 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:g4vKvspu1xsYG4eXUMtZgdknNMM=', 'Content-Length': '0'} 2023-05-31T20:39:20.448 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000006c36550d953c35a4-006477b078-1c17-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:39:20 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:39:20.448 DEBUG:boto:b'pklxkd-58obj000000000000000003171000falseobj000000000000000003172023-05-31T20:38:20.065Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:39:20.449 DEBUG:boto:path=/pklxkd-58/ 2023-05-31T20:39:20.450 DEBUG:boto:auth_path=/pklxkd-58/ 2023-05-31T20:39:20.450 DEBUG:boto:path=/pklxkd-58/?prefix=obj00000000000000000317 2023-05-31T20:39:20.450 DEBUG:boto:auth_path=/pklxkd-58/?prefix=obj00000000000000000317 2023-05-31T20:39:20.451 DEBUG:boto:Method: GET 2023-05-31T20:39:20.451 DEBUG:boto:Path: /pklxkd-58/?prefix=obj00000000000000000317 2023-05-31T20:39:20.451 DEBUG:boto:Data: 2023-05-31T20:39:20.452 DEBUG:boto:Headers: {} 2023-05-31T20:39:20.452 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:39:20.453 DEBUG:boto:Port: 8000 2023-05-31T20:39:20.453 DEBUG:boto:Params: {} 2023-05-31T20:39:20.453 DEBUG:boto:Token: None 2023-05-31T20:39:20.454 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:39:20 GMT /pklxkd-58/ 2023-05-31T20:39:20.454 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:g4vKvspu1xsYG4eXUMtZgdknNMM= 2023-05-31T20:39:20.455 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:39:20 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:g4vKvspu1xsYG4eXUMtZgdknNMM=', 'Content-Length': '0'} 2023-05-31T20:39:20.459 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000002cceadb536a28693-006477b078-1630-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:39:20 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:39:20.459 DEBUG:boto:b'pklxkd-58obj000000000000000003171000falseobj000000000000000003172023-05-31T20:38:20.065Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:39:20.460 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000423 2023-05-31T20:39:20.460 DEBUG:boto:path=/pklxkd-58/obj00000000000000000423 2023-05-31T20:39:20.461 DEBUG:boto:auth_path=/pklxkd-58/obj00000000000000000423 2023-05-31T20:39:20.461 DEBUG:boto:path=/pklxkd-58/obj00000000000000000423?versionId=null 2023-05-31T20:39:20.462 DEBUG:boto:auth_path=/pklxkd-58/obj00000000000000000423?versionId=null 2023-05-31T20:39:20.462 DEBUG:boto:Method: GET 2023-05-31T20:39:20.462 DEBUG:boto:Path: /pklxkd-58/obj00000000000000000423?versionId=null 2023-05-31T20:39:20.463 DEBUG:boto:Data: 2023-05-31T20:39:20.463 DEBUG:boto:Headers: {} 2023-05-31T20:39:20.463 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:39:20.464 DEBUG:boto:Port: 8001 2023-05-31T20:39:20.464 DEBUG:boto:Params: {} 2023-05-31T20:39:20.465 DEBUG:boto:Token: None 2023-05-31T20:39:20.465 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:39:20 GMT /pklxkd-58/obj00000000000000000423?versionId=null 2023-05-31T20:39:20.465 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:WrCe1/wvDB+HLV6IjdzAlylQPyQ= 2023-05-31T20:39:20.466 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:39:20 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:WrCe1/wvDB+HLV6IjdzAlylQPyQ=', 'Content-Length': '0'} 2023-05-31T20:39:20.469 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:38:20 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000006e9c0ff1d548f73-006477b078-1c17-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:39:20 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:39:20.470 DEBUG:boto:path=/pklxkd-58/obj00000000000000000423 2023-05-31T20:39:20.471 DEBUG:boto:auth_path=/pklxkd-58/obj00000000000000000423 2023-05-31T20:39:20.472 DEBUG:boto:path=/pklxkd-58/obj00000000000000000423?versionId=null 2023-05-31T20:39:20.474 DEBUG:boto:auth_path=/pklxkd-58/obj00000000000000000423?versionId=null 2023-05-31T20:39:20.475 DEBUG:boto:Method: GET 2023-05-31T20:39:20.475 DEBUG:boto:Path: /pklxkd-58/obj00000000000000000423?versionId=null 2023-05-31T20:39:20.476 DEBUG:boto:Data: 2023-05-31T20:39:20.476 DEBUG:boto:Headers: {} 2023-05-31T20:39:20.476 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:39:20.477 DEBUG:boto:Port: 8000 2023-05-31T20:39:20.477 DEBUG:boto:Params: {} 2023-05-31T20:39:20.478 DEBUG:boto:Token: None 2023-05-31T20:39:20.478 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:39:20 GMT /pklxkd-58/obj00000000000000000423?versionId=null 2023-05-31T20:39:20.479 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:WrCe1/wvDB+HLV6IjdzAlylQPyQ= 2023-05-31T20:39:20.480 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:39:20 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:WrCe1/wvDB+HLV6IjdzAlylQPyQ=', 'Content-Length': '0'} 2023-05-31T20:39:20.485 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:38:20 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000007c501770eb29b04-006477b078-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:39:20 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:39:20.486 DEBUG:boto:path=/pklxkd-58/obj00000000000000000423 2023-05-31T20:39:20.486 DEBUG:boto:auth_path=/pklxkd-58/obj00000000000000000423 2023-05-31T20:39:20.487 DEBUG:boto:path=/pklxkd-58/obj00000000000000000423?versionId=null 2023-05-31T20:39:20.490 DEBUG:boto:auth_path=/pklxkd-58/obj00000000000000000423?versionId=null 2023-05-31T20:39:20.491 DEBUG:boto:Method: HEAD 2023-05-31T20:39:20.492 DEBUG:boto:Path: /pklxkd-58/obj00000000000000000423?versionId=null 2023-05-31T20:39:20.493 DEBUG:boto:Data: 2023-05-31T20:39:20.493 DEBUG:boto:Headers: {} 2023-05-31T20:39:20.493 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:39:20.494 DEBUG:boto:Port: 8001 2023-05-31T20:39:20.494 DEBUG:boto:Params: {} 2023-05-31T20:39:20.494 DEBUG:boto:Token: None 2023-05-31T20:39:20.495 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:39:20 GMT /pklxkd-58/obj00000000000000000423?versionId=null 2023-05-31T20:39:20.495 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:xcNuz7SO3hVgsBk7fCskwqvnt54= 2023-05-31T20:39:20.496 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:39:20 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:xcNuz7SO3hVgsBk7fCskwqvnt54=', 'Content-Length': '0'} 2023-05-31T20:39:20.501 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:38:20 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000002416c23a13fb0b9-006477b078-1c17-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:39:20 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:39:20.502 DEBUG:boto:path=/pklxkd-58/obj00000000000000000423 2023-05-31T20:39:20.502 DEBUG:boto:auth_path=/pklxkd-58/obj00000000000000000423 2023-05-31T20:39:20.503 DEBUG:boto:path=/pklxkd-58/obj00000000000000000423?versionId=null 2023-05-31T20:39:20.503 DEBUG:boto:auth_path=/pklxkd-58/obj00000000000000000423?versionId=null 2023-05-31T20:39:20.503 DEBUG:boto:Method: HEAD 2023-05-31T20:39:20.504 DEBUG:boto:Path: /pklxkd-58/obj00000000000000000423?versionId=null 2023-05-31T20:39:20.504 DEBUG:boto:Data: 2023-05-31T20:39:20.505 DEBUG:boto:Headers: {} 2023-05-31T20:39:20.505 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:39:20.505 DEBUG:boto:Port: 8000 2023-05-31T20:39:20.506 DEBUG:boto:Params: {} 2023-05-31T20:39:20.506 DEBUG:boto:Token: None 2023-05-31T20:39:20.507 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:39:20 GMT /pklxkd-58/obj00000000000000000423?versionId=null 2023-05-31T20:39:20.508 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:xcNuz7SO3hVgsBk7fCskwqvnt54= 2023-05-31T20:39:20.509 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:39:20 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:xcNuz7SO3hVgsBk7fCskwqvnt54=', 'Content-Length': '0'} 2023-05-31T20:39:20.511 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:38:20 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008c6beb39d0930858-006477b078-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:39:20 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:39:20.512 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000423 2023-05-31T20:39:20.512 DEBUG:boto:path=/pklxkd-58/obj00000000000000000423 2023-05-31T20:39:20.513 DEBUG:boto:auth_path=/pklxkd-58/obj00000000000000000423 2023-05-31T20:39:20.513 DEBUG:boto:path=/pklxkd-58/obj00000000000000000423?versionId=null 2023-05-31T20:39:20.513 DEBUG:boto:auth_path=/pklxkd-58/obj00000000000000000423?versionId=null 2023-05-31T20:39:20.514 DEBUG:boto:Method: GET 2023-05-31T20:39:20.514 DEBUG:boto:Path: /pklxkd-58/obj00000000000000000423?versionId=null 2023-05-31T20:39:20.515 DEBUG:boto:Data: 2023-05-31T20:39:20.515 DEBUG:boto:Headers: {} 2023-05-31T20:39:20.516 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:39:20.517 DEBUG:boto:Port: 8001 2023-05-31T20:39:20.517 DEBUG:boto:Params: {} 2023-05-31T20:39:20.518 DEBUG:boto:Token: None 2023-05-31T20:39:20.518 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:39:20 GMT /pklxkd-58/obj00000000000000000423?versionId=null 2023-05-31T20:39:20.518 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:WrCe1/wvDB+HLV6IjdzAlylQPyQ= 2023-05-31T20:39:20.519 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:39:20 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:WrCe1/wvDB+HLV6IjdzAlylQPyQ=', 'Content-Length': '0'} 2023-05-31T20:39:20.521 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:38:20 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e6b50fb2d7629fe0-006477b078-1c17-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:39:20 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:39:20.521 DEBUG:boto:path=/pklxkd-58/obj00000000000000000423 2023-05-31T20:39:20.522 DEBUG:boto:auth_path=/pklxkd-58/obj00000000000000000423 2023-05-31T20:39:20.522 DEBUG:boto:path=/pklxkd-58/obj00000000000000000423?versionId=null 2023-05-31T20:39:20.523 DEBUG:boto:auth_path=/pklxkd-58/obj00000000000000000423?versionId=null 2023-05-31T20:39:20.523 DEBUG:boto:Method: GET 2023-05-31T20:39:20.523 DEBUG:boto:Path: /pklxkd-58/obj00000000000000000423?versionId=null 2023-05-31T20:39:20.524 DEBUG:boto:Data: 2023-05-31T20:39:20.524 DEBUG:boto:Headers: {} 2023-05-31T20:39:20.525 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:39:20.525 DEBUG:boto:Port: 8000 2023-05-31T20:39:20.526 DEBUG:boto:Params: {} 2023-05-31T20:39:20.526 DEBUG:boto:Token: None 2023-05-31T20:39:20.527 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:39:20 GMT /pklxkd-58/obj00000000000000000423?versionId=null 2023-05-31T20:39:20.527 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:WrCe1/wvDB+HLV6IjdzAlylQPyQ= 2023-05-31T20:39:20.528 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:39:20 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:WrCe1/wvDB+HLV6IjdzAlylQPyQ=', 'Content-Length': '0'} 2023-05-31T20:39:20.530 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:38:20 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002376fedd24d16a8e-006477b078-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:39:20 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:39:20.578 DEBUG:boto:path=/pklxkd-58/ 2023-05-31T20:39:20.578 DEBUG:boto:auth_path=/pklxkd-58/ 2023-05-31T20:39:20.579 DEBUG:boto:path=/pklxkd-58/?prefix=obj00000000000000000423 2023-05-31T20:39:20.579 DEBUG:boto:auth_path=/pklxkd-58/?prefix=obj00000000000000000423 2023-05-31T20:39:20.580 DEBUG:boto:Method: GET 2023-05-31T20:39:20.580 DEBUG:boto:Path: /pklxkd-58/?prefix=obj00000000000000000423 2023-05-31T20:39:20.580 DEBUG:boto:Data: 2023-05-31T20:39:20.581 DEBUG:boto:Headers: {} 2023-05-31T20:39:20.581 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:39:20.582 DEBUG:boto:Port: 8001 2023-05-31T20:39:20.582 DEBUG:boto:Params: {} 2023-05-31T20:39:20.582 DEBUG:boto:Token: None 2023-05-31T20:39:20.583 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:39:20 GMT /pklxkd-58/ 2023-05-31T20:39:20.584 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:g4vKvspu1xsYG4eXUMtZgdknNMM= 2023-05-31T20:39:20.584 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:39:20 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:g4vKvspu1xsYG4eXUMtZgdknNMM=', 'Content-Length': '0'} 2023-05-31T20:39:20.587 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f00b89244a871a69-006477b078-1c17-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:39:20 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:39:20.587 DEBUG:boto:b'pklxkd-58obj000000000000000004231000falseobj000000000000000004232023-05-31T20:38:20.434Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:39:20.588 DEBUG:boto:path=/pklxkd-58/ 2023-05-31T20:39:20.589 DEBUG:boto:auth_path=/pklxkd-58/ 2023-05-31T20:39:20.589 DEBUG:boto:path=/pklxkd-58/?prefix=obj00000000000000000423 2023-05-31T20:39:20.589 DEBUG:boto:auth_path=/pklxkd-58/?prefix=obj00000000000000000423 2023-05-31T20:39:20.590 DEBUG:boto:Method: GET 2023-05-31T20:39:20.590 DEBUG:boto:Path: /pklxkd-58/?prefix=obj00000000000000000423 2023-05-31T20:39:20.590 DEBUG:boto:Data: 2023-05-31T20:39:20.591 DEBUG:boto:Headers: {} 2023-05-31T20:39:20.591 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:39:20.591 DEBUG:boto:Port: 8000 2023-05-31T20:39:20.592 DEBUG:boto:Params: {} 2023-05-31T20:39:20.592 DEBUG:boto:Token: None 2023-05-31T20:39:20.593 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:39:20 GMT /pklxkd-58/ 2023-05-31T20:39:20.593 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:g4vKvspu1xsYG4eXUMtZgdknNMM= 2023-05-31T20:39:20.594 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:39:20 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:g4vKvspu1xsYG4eXUMtZgdknNMM=', 'Content-Length': '0'} 2023-05-31T20:39:20.598 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000232c4d18d64279f4-006477b078-1630-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:39:20 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:39:20.599 DEBUG:boto:b'pklxkd-58obj000000000000000004231000falseobj000000000000000004232023-05-31T20:38:20.434Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:39:20.600 DEBUG:boto:path=/pklxkd-58/obj00000000000000000423 2023-05-31T20:39:20.601 DEBUG:boto:auth_path=/pklxkd-58/obj00000000000000000423 2023-05-31T20:39:20.601 DEBUG:boto:Method: HEAD 2023-05-31T20:39:20.602 DEBUG:boto:Path: /pklxkd-58/obj00000000000000000423 2023-05-31T20:39:20.602 DEBUG:boto:Data: 2023-05-31T20:39:20.603 DEBUG:boto:Headers: {} 2023-05-31T20:39:20.603 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:39:20.603 DEBUG:boto:Port: 8001 2023-05-31T20:39:20.604 DEBUG:boto:Params: {} 2023-05-31T20:39:20.604 DEBUG:boto:Token: None 2023-05-31T20:39:20.605 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:39:20 GMT /pklxkd-58/obj00000000000000000423 2023-05-31T20:39:20.606 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:RUwsDoEYqQ8OoVN6wSEodlKy4oQ= 2023-05-31T20:39:20.606 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:39:20 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:RUwsDoEYqQ8OoVN6wSEodlKy4oQ=', 'Content-Length': '0'} 2023-05-31T20:39:20.608 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:38:20 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000078b21f40097c8c8b-006477b078-1c17-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:39:20 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:39:20.609 DEBUG:boto:path=/pklxkd-58/obj00000000000000000423 2023-05-31T20:39:20.609 DEBUG:boto:auth_path=/pklxkd-58/obj00000000000000000423 2023-05-31T20:39:20.609 DEBUG:boto:Method: HEAD 2023-05-31T20:39:20.610 DEBUG:boto:Path: /pklxkd-58/obj00000000000000000423 2023-05-31T20:39:20.610 DEBUG:boto:Data: 2023-05-31T20:39:20.610 DEBUG:boto:Headers: {} 2023-05-31T20:39:20.611 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:39:20.611 DEBUG:boto:Port: 8000 2023-05-31T20:39:20.611 DEBUG:boto:Params: {} 2023-05-31T20:39:20.612 DEBUG:boto:Token: None 2023-05-31T20:39:20.612 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:39:20 GMT /pklxkd-58/obj00000000000000000423 2023-05-31T20:39:20.613 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:RUwsDoEYqQ8OoVN6wSEodlKy4oQ= 2023-05-31T20:39:20.613 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:39:20 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:RUwsDoEYqQ8OoVN6wSEodlKy4oQ=', 'Content-Length': '0'} 2023-05-31T20:39:20.615 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:38:20 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000dcd64eac8038765f-006477b078-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:39:20 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:39:20.615 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000423 2023-05-31T20:39:20.616 DEBUG:boto:path=/pklxkd-58/obj00000000000000000423 2023-05-31T20:39:20.616 DEBUG:boto:auth_path=/pklxkd-58/obj00000000000000000423 2023-05-31T20:39:20.617 DEBUG:boto:Method: GET 2023-05-31T20:39:20.617 DEBUG:boto:Path: /pklxkd-58/obj00000000000000000423 2023-05-31T20:39:20.618 DEBUG:boto:Data: 2023-05-31T20:39:20.618 DEBUG:boto:Headers: {} 2023-05-31T20:39:20.618 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:39:20.619 DEBUG:boto:Port: 8001 2023-05-31T20:39:20.619 DEBUG:boto:Params: {} 2023-05-31T20:39:20.619 DEBUG:boto:Token: None 2023-05-31T20:39:20.620 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:39:20 GMT /pklxkd-58/obj00000000000000000423 2023-05-31T20:39:20.620 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:E+yZ4/j/d3wP8b+0eoRyk540sbU= 2023-05-31T20:39:20.621 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:39:20 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:E+yZ4/j/d3wP8b+0eoRyk540sbU=', 'Content-Length': '0'} 2023-05-31T20:39:20.623 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:38:20 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000042e8bb9e6f4960e5-006477b078-1c17-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:39:20 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:39:20.623 DEBUG:boto:path=/pklxkd-58/obj00000000000000000423 2023-05-31T20:39:20.624 DEBUG:boto:auth_path=/pklxkd-58/obj00000000000000000423 2023-05-31T20:39:20.624 DEBUG:boto:Method: GET 2023-05-31T20:39:20.625 DEBUG:boto:Path: /pklxkd-58/obj00000000000000000423 2023-05-31T20:39:20.625 DEBUG:boto:Data: 2023-05-31T20:39:20.625 DEBUG:boto:Headers: {} 2023-05-31T20:39:20.626 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:39:20.626 DEBUG:boto:Port: 8000 2023-05-31T20:39:20.626 DEBUG:boto:Params: {} 2023-05-31T20:39:20.627 DEBUG:boto:Token: None 2023-05-31T20:39:20.627 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:39:20 GMT /pklxkd-58/obj00000000000000000423 2023-05-31T20:39:20.628 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:E+yZ4/j/d3wP8b+0eoRyk540sbU= 2023-05-31T20:39:20.628 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:39:20 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:E+yZ4/j/d3wP8b+0eoRyk540sbU=', 'Content-Length': '0'} 2023-05-31T20:39:20.630 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:38:20 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007a82f905ef4bd3a4-006477b078-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:39:20 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:39:20.631 DEBUG:boto:path=/pklxkd-58/ 2023-05-31T20:39:20.631 DEBUG:boto:auth_path=/pklxkd-58/ 2023-05-31T20:39:20.631 DEBUG:boto:path=/pklxkd-58/?prefix=obj00000000000000000423 2023-05-31T20:39:20.632 DEBUG:boto:auth_path=/pklxkd-58/?prefix=obj00000000000000000423 2023-05-31T20:39:20.632 DEBUG:boto:Method: GET 2023-05-31T20:39:20.633 DEBUG:boto:Path: /pklxkd-58/?prefix=obj00000000000000000423 2023-05-31T20:39:20.633 DEBUG:boto:Data: 2023-05-31T20:39:20.633 DEBUG:boto:Headers: {} 2023-05-31T20:39:20.634 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:39:20.634 DEBUG:boto:Port: 8001 2023-05-31T20:39:20.634 DEBUG:boto:Params: {} 2023-05-31T20:39:20.635 DEBUG:boto:Token: None 2023-05-31T20:39:20.635 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:39:20 GMT /pklxkd-58/ 2023-05-31T20:39:20.636 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:g4vKvspu1xsYG4eXUMtZgdknNMM= 2023-05-31T20:39:20.636 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:39:20 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:g4vKvspu1xsYG4eXUMtZgdknNMM=', 'Content-Length': '0'} 2023-05-31T20:39:20.638 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000447bb3a918b2b6b1-006477b078-1c17-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:39:20 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:39:20.639 DEBUG:boto:b'pklxkd-58obj000000000000000004231000falseobj000000000000000004232023-05-31T20:38:20.434Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:39:20.640 DEBUG:boto:path=/pklxkd-58/ 2023-05-31T20:39:20.640 DEBUG:boto:auth_path=/pklxkd-58/ 2023-05-31T20:39:20.640 DEBUG:boto:path=/pklxkd-58/?prefix=obj00000000000000000423 2023-05-31T20:39:20.641 DEBUG:boto:auth_path=/pklxkd-58/?prefix=obj00000000000000000423 2023-05-31T20:39:20.641 DEBUG:boto:Method: GET 2023-05-31T20:39:20.641 DEBUG:boto:Path: /pklxkd-58/?prefix=obj00000000000000000423 2023-05-31T20:39:20.642 DEBUG:boto:Data: 2023-05-31T20:39:20.642 DEBUG:boto:Headers: {} 2023-05-31T20:39:20.642 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:39:20.643 DEBUG:boto:Port: 8000 2023-05-31T20:39:20.643 DEBUG:boto:Params: {} 2023-05-31T20:39:20.644 DEBUG:boto:Token: None 2023-05-31T20:39:20.644 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:39:20 GMT /pklxkd-58/ 2023-05-31T20:39:20.644 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:g4vKvspu1xsYG4eXUMtZgdknNMM= 2023-05-31T20:39:20.645 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:39:20 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:g4vKvspu1xsYG4eXUMtZgdknNMM=', 'Content-Length': '0'} 2023-05-31T20:39:20.649 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e379eed12677b6e1-006477b078-1630-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:39:20 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:39:20.649 DEBUG:boto:b'pklxkd-58obj000000000000000004231000falseobj000000000000000004232023-05-31T20:38:20.434Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:39:20.650 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000467 2023-05-31T20:39:20.650 DEBUG:boto:path=/pklxkd-58/obj00000000000000000467 2023-05-31T20:39:20.651 DEBUG:boto:auth_path=/pklxkd-58/obj00000000000000000467 2023-05-31T20:39:20.651 DEBUG:boto:path=/pklxkd-58/obj00000000000000000467?versionId=null 2023-05-31T20:39:20.651 DEBUG:boto:auth_path=/pklxkd-58/obj00000000000000000467?versionId=null 2023-05-31T20:39:20.652 DEBUG:boto:Method: GET 2023-05-31T20:39:20.652 DEBUG:boto:Path: /pklxkd-58/obj00000000000000000467?versionId=null 2023-05-31T20:39:20.653 DEBUG:boto:Data: 2023-05-31T20:39:20.653 DEBUG:boto:Headers: {} 2023-05-31T20:39:20.653 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:39:20.654 DEBUG:boto:Port: 8001 2023-05-31T20:39:20.654 DEBUG:boto:Params: {} 2023-05-31T20:39:20.654 DEBUG:boto:Token: None 2023-05-31T20:39:20.655 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:39:20 GMT /pklxkd-58/obj00000000000000000467?versionId=null 2023-05-31T20:39:20.655 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:+J/uju7PNIqVX1LzuI1ApmI8WHk= 2023-05-31T20:39:20.656 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:39:20 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:+J/uju7PNIqVX1LzuI1ApmI8WHk=', 'Content-Length': '0'} 2023-05-31T20:39:20.658 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:38:20 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004d1e38b841c3e2e2-006477b078-1c17-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:39:20 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:39:20.659 DEBUG:boto:path=/pklxkd-58/obj00000000000000000467 2023-05-31T20:39:20.659 DEBUG:boto:auth_path=/pklxkd-58/obj00000000000000000467 2023-05-31T20:39:20.659 DEBUG:boto:path=/pklxkd-58/obj00000000000000000467?versionId=null 2023-05-31T20:39:20.660 DEBUG:boto:auth_path=/pklxkd-58/obj00000000000000000467?versionId=null 2023-05-31T20:39:20.660 DEBUG:boto:Method: GET 2023-05-31T20:39:20.661 DEBUG:boto:Path: /pklxkd-58/obj00000000000000000467?versionId=null 2023-05-31T20:39:20.661 DEBUG:boto:Data: 2023-05-31T20:39:20.661 DEBUG:boto:Headers: {} 2023-05-31T20:39:20.662 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:39:20.662 DEBUG:boto:Port: 8000 2023-05-31T20:39:20.662 DEBUG:boto:Params: {} 2023-05-31T20:39:20.663 DEBUG:boto:Token: None 2023-05-31T20:39:20.663 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:39:20 GMT /pklxkd-58/obj00000000000000000467?versionId=null 2023-05-31T20:39:20.664 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:+J/uju7PNIqVX1LzuI1ApmI8WHk= 2023-05-31T20:39:20.664 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:39:20 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:+J/uju7PNIqVX1LzuI1ApmI8WHk=', 'Content-Length': '0'} 2023-05-31T20:39:20.666 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:38:20 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000448b9eb4429348ef-006477b078-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:39:20 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:39:20.708 DEBUG:boto:path=/pklxkd-58/obj00000000000000000467 2023-05-31T20:39:20.709 DEBUG:boto:auth_path=/pklxkd-58/obj00000000000000000467 2023-05-31T20:39:20.709 DEBUG:boto:path=/pklxkd-58/obj00000000000000000467?versionId=null 2023-05-31T20:39:20.710 DEBUG:boto:auth_path=/pklxkd-58/obj00000000000000000467?versionId=null 2023-05-31T20:39:20.710 DEBUG:boto:Method: HEAD 2023-05-31T20:39:20.710 DEBUG:boto:Path: /pklxkd-58/obj00000000000000000467?versionId=null 2023-05-31T20:39:20.711 DEBUG:boto:Data: 2023-05-31T20:39:20.711 DEBUG:boto:Headers: {} 2023-05-31T20:39:20.711 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:39:20.712 DEBUG:boto:Port: 8001 2023-05-31T20:39:20.712 DEBUG:boto:Params: {} 2023-05-31T20:39:20.712 DEBUG:boto:Token: None 2023-05-31T20:39:20.713 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:39:20 GMT /pklxkd-58/obj00000000000000000467?versionId=null 2023-05-31T20:39:20.713 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:n2IO36p0vjvDzk1hDjvwVVq2AZM= 2023-05-31T20:39:20.714 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:39:20 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:n2IO36p0vjvDzk1hDjvwVVq2AZM=', 'Content-Length': '0'} 2023-05-31T20:39:20.716 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:38:20 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b40060be39a03864-006477b078-1c17-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:39:20 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:39:20.717 DEBUG:boto:path=/pklxkd-58/obj00000000000000000467 2023-05-31T20:39:20.717 DEBUG:boto:auth_path=/pklxkd-58/obj00000000000000000467 2023-05-31T20:39:20.718 DEBUG:boto:path=/pklxkd-58/obj00000000000000000467?versionId=null 2023-05-31T20:39:20.718 DEBUG:boto:auth_path=/pklxkd-58/obj00000000000000000467?versionId=null 2023-05-31T20:39:20.718 DEBUG:boto:Method: HEAD 2023-05-31T20:39:20.719 DEBUG:boto:Path: /pklxkd-58/obj00000000000000000467?versionId=null 2023-05-31T20:39:20.719 DEBUG:boto:Data: 2023-05-31T20:39:20.719 DEBUG:boto:Headers: {} 2023-05-31T20:39:20.720 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:39:20.721 DEBUG:boto:Port: 8000 2023-05-31T20:39:20.721 DEBUG:boto:Params: {} 2023-05-31T20:39:20.721 DEBUG:boto:Token: None 2023-05-31T20:39:20.722 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:39:20 GMT /pklxkd-58/obj00000000000000000467?versionId=null 2023-05-31T20:39:20.722 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:n2IO36p0vjvDzk1hDjvwVVq2AZM= 2023-05-31T20:39:20.722 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:39:20 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:n2IO36p0vjvDzk1hDjvwVVq2AZM=', 'Content-Length': '0'} 2023-05-31T20:39:20.724 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:38:20 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008a130d41294c70f5-006477b078-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:39:20 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:39:20.725 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000467 2023-05-31T20:39:20.725 DEBUG:boto:path=/pklxkd-58/obj00000000000000000467 2023-05-31T20:39:20.726 DEBUG:boto:auth_path=/pklxkd-58/obj00000000000000000467 2023-05-31T20:39:20.726 DEBUG:boto:path=/pklxkd-58/obj00000000000000000467?versionId=null 2023-05-31T20:39:20.726 DEBUG:boto:auth_path=/pklxkd-58/obj00000000000000000467?versionId=null 2023-05-31T20:39:20.727 DEBUG:boto:Method: GET 2023-05-31T20:39:20.727 DEBUG:boto:Path: /pklxkd-58/obj00000000000000000467?versionId=null 2023-05-31T20:39:20.728 DEBUG:boto:Data: 2023-05-31T20:39:20.728 DEBUG:boto:Headers: {} 2023-05-31T20:39:20.728 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:39:20.729 DEBUG:boto:Port: 8001 2023-05-31T20:39:20.729 DEBUG:boto:Params: {} 2023-05-31T20:39:20.730 DEBUG:boto:Token: None 2023-05-31T20:39:20.730 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:39:20 GMT /pklxkd-58/obj00000000000000000467?versionId=null 2023-05-31T20:39:20.731 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:+J/uju7PNIqVX1LzuI1ApmI8WHk= 2023-05-31T20:39:20.731 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:39:20 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:+J/uju7PNIqVX1LzuI1ApmI8WHk=', 'Content-Length': '0'} 2023-05-31T20:39:20.733 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:38:20 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000011a7b0cd69f297a9-006477b078-1c17-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:39:20 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:39:20.734 DEBUG:boto:path=/pklxkd-58/obj00000000000000000467 2023-05-31T20:39:20.734 DEBUG:boto:auth_path=/pklxkd-58/obj00000000000000000467 2023-05-31T20:39:20.734 DEBUG:boto:path=/pklxkd-58/obj00000000000000000467?versionId=null 2023-05-31T20:39:20.735 DEBUG:boto:auth_path=/pklxkd-58/obj00000000000000000467?versionId=null 2023-05-31T20:39:20.735 DEBUG:boto:Method: GET 2023-05-31T20:39:20.735 DEBUG:boto:Path: /pklxkd-58/obj00000000000000000467?versionId=null 2023-05-31T20:39:20.736 DEBUG:boto:Data: 2023-05-31T20:39:20.736 DEBUG:boto:Headers: {} 2023-05-31T20:39:20.737 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:39:20.737 DEBUG:boto:Port: 8000 2023-05-31T20:39:20.737 DEBUG:boto:Params: {} 2023-05-31T20:39:20.738 DEBUG:boto:Token: None 2023-05-31T20:39:20.738 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:39:20 GMT /pklxkd-58/obj00000000000000000467?versionId=null 2023-05-31T20:39:20.739 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:+J/uju7PNIqVX1LzuI1ApmI8WHk= 2023-05-31T20:39:20.739 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:39:20 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:+J/uju7PNIqVX1LzuI1ApmI8WHk=', 'Content-Length': '0'} 2023-05-31T20:39:20.741 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:38:20 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004d31eaf75babe7d2-006477b078-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:39:20 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:39:20.741 DEBUG:boto:path=/pklxkd-58/ 2023-05-31T20:39:20.742 DEBUG:boto:auth_path=/pklxkd-58/ 2023-05-31T20:39:20.742 DEBUG:boto:path=/pklxkd-58/?prefix=obj00000000000000000467 2023-05-31T20:39:20.742 DEBUG:boto:auth_path=/pklxkd-58/?prefix=obj00000000000000000467 2023-05-31T20:39:20.743 DEBUG:boto:Method: GET 2023-05-31T20:39:20.743 DEBUG:boto:Path: /pklxkd-58/?prefix=obj00000000000000000467 2023-05-31T20:39:20.744 DEBUG:boto:Data: 2023-05-31T20:39:20.744 DEBUG:boto:Headers: {} 2023-05-31T20:39:20.744 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:39:20.745 DEBUG:boto:Port: 8001 2023-05-31T20:39:20.745 DEBUG:boto:Params: {} 2023-05-31T20:39:20.745 DEBUG:boto:Token: None 2023-05-31T20:39:20.746 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:39:20 GMT /pklxkd-58/ 2023-05-31T20:39:20.746 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:g4vKvspu1xsYG4eXUMtZgdknNMM= 2023-05-31T20:39:20.747 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:39:20 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:g4vKvspu1xsYG4eXUMtZgdknNMM=', 'Content-Length': '0'} 2023-05-31T20:39:20.749 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000002a8d99b2c295740a-006477b078-1c17-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:39:20 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:39:20.792 DEBUG:boto:b'pklxkd-58obj000000000000000004671000falseobj000000000000000004672023-05-31T20:38:20.327Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:39:20.793 DEBUG:boto:path=/pklxkd-58/ 2023-05-31T20:39:20.794 DEBUG:boto:auth_path=/pklxkd-58/ 2023-05-31T20:39:20.794 DEBUG:boto:path=/pklxkd-58/?prefix=obj00000000000000000467 2023-05-31T20:39:20.795 DEBUG:boto:auth_path=/pklxkd-58/?prefix=obj00000000000000000467 2023-05-31T20:39:20.795 DEBUG:boto:Method: GET 2023-05-31T20:39:20.796 DEBUG:boto:Path: /pklxkd-58/?prefix=obj00000000000000000467 2023-05-31T20:39:20.797 DEBUG:boto:Data: 2023-05-31T20:39:20.797 DEBUG:boto:Headers: {} 2023-05-31T20:39:20.798 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:39:20.799 DEBUG:boto:Port: 8000 2023-05-31T20:39:20.799 DEBUG:boto:Params: {} 2023-05-31T20:39:20.800 DEBUG:boto:Token: None 2023-05-31T20:39:20.800 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:39:20 GMT /pklxkd-58/ 2023-05-31T20:39:20.801 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:g4vKvspu1xsYG4eXUMtZgdknNMM= 2023-05-31T20:39:20.801 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:39:20 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:g4vKvspu1xsYG4eXUMtZgdknNMM=', 'Content-Length': '0'} 2023-05-31T20:39:20.806 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000012a98cb2a80257e3-006477b078-1630-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:39:20 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:39:20.807 DEBUG:boto:b'pklxkd-58obj000000000000000004671000falseobj000000000000000004672023-05-31T20:38:20.327Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:39:20.808 DEBUG:boto:path=/pklxkd-58/obj00000000000000000467 2023-05-31T20:39:20.808 DEBUG:boto:auth_path=/pklxkd-58/obj00000000000000000467 2023-05-31T20:39:20.809 DEBUG:boto:Method: HEAD 2023-05-31T20:39:20.810 DEBUG:boto:Path: /pklxkd-58/obj00000000000000000467 2023-05-31T20:39:20.810 DEBUG:boto:Data: 2023-05-31T20:39:20.811 DEBUG:boto:Headers: {} 2023-05-31T20:39:20.811 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:39:20.811 DEBUG:boto:Port: 8001 2023-05-31T20:39:20.812 DEBUG:boto:Params: {} 2023-05-31T20:39:20.812 DEBUG:boto:Token: None 2023-05-31T20:39:20.813 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:39:20 GMT /pklxkd-58/obj00000000000000000467 2023-05-31T20:39:20.813 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:7gK0kc+4V78MShmpBlAdJnh2hOk= 2023-05-31T20:39:20.813 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:39:20 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:7gK0kc+4V78MShmpBlAdJnh2hOk=', 'Content-Length': '0'} 2023-05-31T20:39:20.816 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:38:20 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000130182b54452fe1c-006477b078-1c17-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:39:20 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:39:20.817 DEBUG:boto:path=/pklxkd-58/obj00000000000000000467 2023-05-31T20:39:20.817 DEBUG:boto:auth_path=/pklxkd-58/obj00000000000000000467 2023-05-31T20:39:20.818 DEBUG:boto:Method: HEAD 2023-05-31T20:39:20.818 DEBUG:boto:Path: /pklxkd-58/obj00000000000000000467 2023-05-31T20:39:20.818 DEBUG:boto:Data: 2023-05-31T20:39:20.819 DEBUG:boto:Headers: {} 2023-05-31T20:39:20.820 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:39:20.820 DEBUG:boto:Port: 8000 2023-05-31T20:39:20.820 DEBUG:boto:Params: {} 2023-05-31T20:39:20.821 DEBUG:boto:Token: None 2023-05-31T20:39:20.821 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:39:20 GMT /pklxkd-58/obj00000000000000000467 2023-05-31T20:39:20.822 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:7gK0kc+4V78MShmpBlAdJnh2hOk= 2023-05-31T20:39:20.822 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:39:20 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:7gK0kc+4V78MShmpBlAdJnh2hOk=', 'Content-Length': '0'} 2023-05-31T20:39:20.824 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:38:20 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e2e7499f718b5fd6-006477b078-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:39:20 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:39:20.825 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000467 2023-05-31T20:39:20.826 DEBUG:boto:path=/pklxkd-58/obj00000000000000000467 2023-05-31T20:39:20.828 DEBUG:boto:auth_path=/pklxkd-58/obj00000000000000000467 2023-05-31T20:39:20.829 DEBUG:boto:Method: GET 2023-05-31T20:39:20.830 DEBUG:boto:Path: /pklxkd-58/obj00000000000000000467 2023-05-31T20:39:20.831 DEBUG:boto:Data: 2023-05-31T20:39:20.832 DEBUG:boto:Headers: {} 2023-05-31T20:39:20.833 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:39:20.834 DEBUG:boto:Port: 8001 2023-05-31T20:39:20.835 DEBUG:boto:Params: {} 2023-05-31T20:39:20.835 DEBUG:boto:Token: None 2023-05-31T20:39:20.836 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:39:20 GMT /pklxkd-58/obj00000000000000000467 2023-05-31T20:39:20.837 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:wX+FTsGwh+aCFtWmqhdfQH3Uo7E= 2023-05-31T20:39:20.839 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:39:20 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:wX+FTsGwh+aCFtWmqhdfQH3Uo7E=', 'Content-Length': '0'} 2023-05-31T20:39:20.841 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:38:20 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e003c7be3051aed6-006477b078-1c17-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:39:20 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:39:20.842 DEBUG:boto:path=/pklxkd-58/obj00000000000000000467 2023-05-31T20:39:20.844 DEBUG:boto:auth_path=/pklxkd-58/obj00000000000000000467 2023-05-31T20:39:20.845 DEBUG:boto:Method: GET 2023-05-31T20:39:20.846 DEBUG:boto:Path: /pklxkd-58/obj00000000000000000467 2023-05-31T20:39:20.846 DEBUG:boto:Data: 2023-05-31T20:39:20.847 DEBUG:boto:Headers: {} 2023-05-31T20:39:20.847 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:39:20.848 DEBUG:boto:Port: 8000 2023-05-31T20:39:20.848 DEBUG:boto:Params: {} 2023-05-31T20:39:20.849 DEBUG:boto:Token: None 2023-05-31T20:39:20.849 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:39:20 GMT /pklxkd-58/obj00000000000000000467 2023-05-31T20:39:20.850 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:wX+FTsGwh+aCFtWmqhdfQH3Uo7E= 2023-05-31T20:39:20.850 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:39:20 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:wX+FTsGwh+aCFtWmqhdfQH3Uo7E=', 'Content-Length': '0'} 2023-05-31T20:39:20.852 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:38:20 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000033ab8eadb99b0098-006477b078-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:39:20 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:39:20.896 DEBUG:boto:path=/pklxkd-58/ 2023-05-31T20:39:20.897 DEBUG:boto:auth_path=/pklxkd-58/ 2023-05-31T20:39:20.897 DEBUG:boto:path=/pklxkd-58/?prefix=obj00000000000000000467 2023-05-31T20:39:20.898 DEBUG:boto:auth_path=/pklxkd-58/?prefix=obj00000000000000000467 2023-05-31T20:39:20.898 DEBUG:boto:Method: GET 2023-05-31T20:39:20.899 DEBUG:boto:Path: /pklxkd-58/?prefix=obj00000000000000000467 2023-05-31T20:39:20.899 DEBUG:boto:Data: 2023-05-31T20:39:20.899 DEBUG:boto:Headers: {} 2023-05-31T20:39:20.900 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:39:20.900 DEBUG:boto:Port: 8001 2023-05-31T20:39:20.900 DEBUG:boto:Params: {} 2023-05-31T20:39:20.901 DEBUG:boto:Token: None 2023-05-31T20:39:20.901 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:39:20 GMT /pklxkd-58/ 2023-05-31T20:39:20.902 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:g4vKvspu1xsYG4eXUMtZgdknNMM= 2023-05-31T20:39:20.903 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:39:20 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:g4vKvspu1xsYG4eXUMtZgdknNMM=', 'Content-Length': '0'} 2023-05-31T20:39:20.905 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000006e34bb66d7259a0f-006477b078-1c17-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:39:20 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:39:20.906 DEBUG:boto:b'pklxkd-58obj000000000000000004671000falseobj000000000000000004672023-05-31T20:38:20.327Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:39:20.908 DEBUG:boto:path=/pklxkd-58/ 2023-05-31T20:39:20.909 DEBUG:boto:auth_path=/pklxkd-58/ 2023-05-31T20:39:20.910 DEBUG:boto:path=/pklxkd-58/?prefix=obj00000000000000000467 2023-05-31T20:39:20.911 DEBUG:boto:auth_path=/pklxkd-58/?prefix=obj00000000000000000467 2023-05-31T20:39:20.912 DEBUG:boto:Method: GET 2023-05-31T20:39:20.913 DEBUG:boto:Path: /pklxkd-58/?prefix=obj00000000000000000467 2023-05-31T20:39:20.914 DEBUG:boto:Data: 2023-05-31T20:39:20.915 DEBUG:boto:Headers: {} 2023-05-31T20:39:20.916 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:39:20.916 DEBUG:boto:Port: 8000 2023-05-31T20:39:20.917 DEBUG:boto:Params: {} 2023-05-31T20:39:20.919 DEBUG:boto:Token: None 2023-05-31T20:39:20.919 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:39:20 GMT /pklxkd-58/ 2023-05-31T20:39:20.920 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:g4vKvspu1xsYG4eXUMtZgdknNMM= 2023-05-31T20:39:20.921 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:39:20 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:g4vKvspu1xsYG4eXUMtZgdknNMM=', 'Content-Length': '0'} 2023-05-31T20:39:20.925 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000006360e791d08412bc-006477b078-1630-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:39:20 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:39:20.926 DEBUG:boto:b'pklxkd-58obj000000000000000004671000falseobj000000000000000004672023-05-31T20:38:20.327Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:39:20.927 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=pklxkd-58 zones={a2, a1} 2023-05-31T20:39:20.928 INFO:tasks.rgw_multisite_tests:Create several generations of objects, then run bucket sync ... ok 2023-05-31T20:39:20.932 DEBUG:boto:path=/pklxkd-59/ 2023-05-31T20:39:20.933 DEBUG:boto:auth_path=/pklxkd-59/ 2023-05-31T20:39:20.933 DEBUG:boto:Method: PUT 2023-05-31T20:39:20.934 DEBUG:boto:Path: /pklxkd-59/ 2023-05-31T20:39:20.934 DEBUG:boto:Data: 2023-05-31T20:39:20.934 DEBUG:boto:Headers: {} 2023-05-31T20:39:20.935 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:39:20.935 DEBUG:boto:Port: 8000 2023-05-31T20:39:20.935 DEBUG:boto:Params: {} 2023-05-31T20:39:20.936 DEBUG:boto:Token: None 2023-05-31T20:39:20.936 DEBUG:boto:StringToSign: PUT Wed, 31 May 2023 20:39:20 GMT /pklxkd-59/ 2023-05-31T20:39:20.937 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:6Ewpfb0ebsjHXWK8JIj/SFZMD2g= 2023-05-31T20:39:20.937 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:39:20 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:6Ewpfb0ebsjHXWK8JIj/SFZMD2g=', 'Content-Length': '0'} 2023-05-31T20:39:20.963 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000c8dba790f641016e-006477b078-1630-a1'), ('Content-Length', '0'), ('Date', 'Wed, 31 May 2023 20:39:20 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:39:20.964 DEBUG:rgw_multi.tests:created bucket=pklxkd-59 2023-05-31T20:39:20.965 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-31T20:39:20.966 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-31T20:39:20.967 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-05-31T20:39:21.022 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2023-05-31T20:39:21.022 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2023-05-31T20:39:21.085 INFO:teuthology.orchestra.run.smithi099.stderr:No --period given, using current period=2e6123d2-d695-4350-9a01-61a455bc782b 2023-05-31T20:39:21.099 INFO:teuthology.orchestra.run.smithi099.stdout:[ 2023-05-31T20:39:21.100 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T20:39:21.100 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1685562631.516089_59.1", 2023-05-31T20:39:21.100 INFO:teuthology.orchestra.run.smithi099.stdout: "last_update": "2023-05-31T19:50:31.516089Z" 2023-05-31T20:39:21.100 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:39:21.100 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T20:39:21.101 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "", 2023-05-31T20:39:21.101 INFO:teuthology.orchestra.run.smithi099.stdout: "last_update": "0.000000" 2023-05-31T20:39:21.101 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:39:21.101 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T20:39:21.101 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1685562970.274572_93.1", 2023-05-31T20:39:21.101 INFO:teuthology.orchestra.run.smithi099.stdout: "last_update": "2023-05-31T19:56:10.274572Z" 2023-05-31T20:39:21.101 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:39:21.102 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T20:39:21.102 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1685565560.956172_429.1", 2023-05-31T20:39:21.102 INFO:teuthology.orchestra.run.smithi099.stdout: "last_update": "2023-05-31T20:39:20.956172Z" 2023-05-31T20:39:21.102 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T20:39:21.102 INFO:teuthology.orchestra.run.smithi099.stdout:] 2023-05-31T20:39:21.103 DEBUG:rgw_multi.tests:master meta markers={0: '1_1685562631.516089_59.1', 1: '', 2: '1_1685562970.274572_93.1', 3: '1_1685565560.956172_429.1'} 2023-05-31T20:39:21.104 INFO:rgw_multi.tests:starting meta checkpoint for zone=a2 2023-05-31T20:39:21.104 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-31T20:39:21.104 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-31T20:39:21.105 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-05-31T20:39:21.157 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2023-05-31T20:39:21.157 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2023-05-31T20:39:21.215 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:39:21.207+0000 7f96b8adb780 20 RGW-SYNC:meta: read sync status 2023-05-31T20:39:21.215 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:39:21.208+0000 7f96b8adb780 20 RGW-SYNC:meta: read sync status 2023-05-31T20:39:21.237 INFO:teuthology.orchestra.run.smithi099.stdout:{ 2023-05-31T20:39:21.237 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_status": { 2023-05-31T20:39:21.237 INFO:teuthology.orchestra.run.smithi099.stdout: "info": { 2023-05-31T20:39:21.238 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "sync", 2023-05-31T20:39:21.238 INFO:teuthology.orchestra.run.smithi099.stdout: "num_shards": 4, 2023-05-31T20:39:21.238 INFO:teuthology.orchestra.run.smithi099.stdout: "period": "2e6123d2-d695-4350-9a01-61a455bc782b", 2023-05-31T20:39:21.238 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2023-05-31T20:39:21.238 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:39:21.238 INFO:teuthology.orchestra.run.smithi099.stdout: "markers": [ 2023-05-31T20:39:21.239 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T20:39:21.239 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 0, 2023-05-31T20:39:21.239 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T20:39:21.239 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2023-05-31T20:39:21.239 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1685562631.516089_59.1", 2023-05-31T20:39:21.239 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T20:39:21.240 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 1, 2023-05-31T20:39:21.240 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T20:39:21.240 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2023-05-31T19:50:31.516089Z", 2023-05-31T20:39:21.240 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2023-05-31T20:39:21.240 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T20:39:21.240 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:39:21.240 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T20:39:21.241 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 1, 2023-05-31T20:39:21.241 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T20:39:21.241 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2023-05-31T20:39:21.241 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "", 2023-05-31T20:39:21.241 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T20:39:21.241 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2023-05-31T20:39:21.242 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T20:39:21.242 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "0.000000", 2023-05-31T20:39:21.242 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2023-05-31T20:39:21.242 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T20:39:21.242 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:39:21.243 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T20:39:21.243 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 2, 2023-05-31T20:39:21.243 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T20:39:21.243 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2023-05-31T20:39:21.243 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1685562970.274572_93.1", 2023-05-31T20:39:21.243 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T20:39:21.244 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2023-05-31T20:39:21.244 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T20:39:21.244 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2023-05-31T19:56:10.274572Z", 2023-05-31T20:39:21.244 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2023-05-31T20:39:21.244 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T20:39:21.245 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:39:21.245 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T20:39:21.245 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 3, 2023-05-31T20:39:21.246 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T20:39:21.246 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2023-05-31T20:39:21.246 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1685565495.106573_402.1", 2023-05-31T20:39:21.246 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T20:39:21.246 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2023-05-31T20:39:21.246 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T20:39:21.248 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2023-05-31T20:38:15.106573Z", 2023-05-31T20:39:21.248 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2023-05-31T20:39:21.248 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T20:39:21.249 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T20:39:21.249 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2023-05-31T20:39:21.249 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:39:21.249 INFO:teuthology.orchestra.run.smithi099.stdout: "full_sync": { 2023-05-31T20:39:21.249 INFO:teuthology.orchestra.run.smithi099.stdout: "total": 1, 2023-05-31T20:39:21.249 INFO:teuthology.orchestra.run.smithi099.stdout: "complete": 1 2023-05-31T20:39:21.250 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:39:21.250 INFO:teuthology.orchestra.run.smithi099.stdout: "current_time": "2023-05-31T20:39:21Z" 2023-05-31T20:39:21.250 INFO:teuthology.orchestra.run.smithi099.stdout:} 2023-05-31T20:39:21.251 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "2e6123d2-d695-4350-9a01-61a455bc782b", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1685562631.516089_59.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-05-31T19:50:31.516089Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1685562970.274572_93.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-05-31T19:56:10.274572Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1685565495.106573_402.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-05-31T20:38:15.106573Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-05-31T20:39:21Z" } 2023-05-31T20:39:21.252 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1685562631.516089_59.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-05-31T19:50:31.516089Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1685562970.274572_93.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-05-31T19:56:10.274572Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1685565495.106573_402.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-05-31T20:38:15.106573Z', 'realm_epoch': 2}}] 2023-05-31T20:39:21.252 DEBUG:rgw_multi.tests:log_status={0: '1_1685562631.516089_59.1', 1: '', 2: '1_1685562970.274572_93.1', 3: '1_1685565560.956172_429.1'} 2023-05-31T20:39:21.252 DEBUG:rgw_multi.tests:sync_status={0: '1_1685562631.516089_59.1', 1: '', 2: '1_1685562970.274572_93.1', 3: '1_1685565495.106573_402.1'} 2023-05-31T20:39:21.253 WARNING:rgw_multi.tests:zone a2 behind master: shard=3 master=1_1685565560.956172_429.1 target=1_1685565495.106573_402.1 2023-05-31T20:39:26.254 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-31T20:39:26.255 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-31T20:39:26.257 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-05-31T20:39:26.320 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2023-05-31T20:39:26.321 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2023-05-31T20:39:26.379 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:39:26.374+0000 7f5d4e163780 20 RGW-SYNC:meta: read sync status 2023-05-31T20:39:26.380 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:39:26.375+0000 7f5d4e163780 20 RGW-SYNC:meta: read sync status 2023-05-31T20:39:26.397 INFO:teuthology.orchestra.run.smithi099.stdout:{ 2023-05-31T20:39:26.398 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_status": { 2023-05-31T20:39:26.398 INFO:teuthology.orchestra.run.smithi099.stdout: "info": { 2023-05-31T20:39:26.398 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "sync", 2023-05-31T20:39:26.398 INFO:teuthology.orchestra.run.smithi099.stdout: "num_shards": 4, 2023-05-31T20:39:26.398 INFO:teuthology.orchestra.run.smithi099.stdout: "period": "2e6123d2-d695-4350-9a01-61a455bc782b", 2023-05-31T20:39:26.399 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2023-05-31T20:39:26.399 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:39:26.399 INFO:teuthology.orchestra.run.smithi099.stdout: "markers": [ 2023-05-31T20:39:26.399 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T20:39:26.399 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 0, 2023-05-31T20:39:26.399 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T20:39:26.400 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2023-05-31T20:39:26.400 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1685562631.516089_59.1", 2023-05-31T20:39:26.400 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T20:39:26.400 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 1, 2023-05-31T20:39:26.400 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T20:39:26.400 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2023-05-31T19:50:31.516089Z", 2023-05-31T20:39:26.400 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2023-05-31T20:39:26.401 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T20:39:26.401 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:39:26.401 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T20:39:26.401 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 1, 2023-05-31T20:39:26.401 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T20:39:26.401 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2023-05-31T20:39:26.402 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "", 2023-05-31T20:39:26.402 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T20:39:26.402 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2023-05-31T20:39:26.402 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T20:39:26.402 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "0.000000", 2023-05-31T20:39:26.402 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2023-05-31T20:39:26.402 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T20:39:26.403 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:39:26.403 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T20:39:26.403 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 2, 2023-05-31T20:39:26.403 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T20:39:26.403 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2023-05-31T20:39:26.403 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1685562970.274572_93.1", 2023-05-31T20:39:26.403 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T20:39:26.404 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2023-05-31T20:39:26.404 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T20:39:26.404 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2023-05-31T19:56:10.274572Z", 2023-05-31T20:39:26.404 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2023-05-31T20:39:26.404 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T20:39:26.404 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:39:26.404 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T20:39:26.405 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 3, 2023-05-31T20:39:26.405 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T20:39:26.405 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2023-05-31T20:39:26.405 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1685565560.956172_429.1", 2023-05-31T20:39:26.406 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T20:39:26.406 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2023-05-31T20:39:26.406 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T20:39:26.406 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2023-05-31T20:39:20.956172Z", 2023-05-31T20:39:26.406 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2023-05-31T20:39:26.406 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T20:39:26.407 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T20:39:26.407 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2023-05-31T20:39:26.407 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:39:26.407 INFO:teuthology.orchestra.run.smithi099.stdout: "full_sync": { 2023-05-31T20:39:26.407 INFO:teuthology.orchestra.run.smithi099.stdout: "total": 1, 2023-05-31T20:39:26.407 INFO:teuthology.orchestra.run.smithi099.stdout: "complete": 1 2023-05-31T20:39:26.407 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:39:26.408 INFO:teuthology.orchestra.run.smithi099.stdout: "current_time": "2023-05-31T20:39:26Z" 2023-05-31T20:39:26.408 INFO:teuthology.orchestra.run.smithi099.stdout:} 2023-05-31T20:39:26.409 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "2e6123d2-d695-4350-9a01-61a455bc782b", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1685562631.516089_59.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-05-31T19:50:31.516089Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1685562970.274572_93.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-05-31T19:56:10.274572Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1685565560.956172_429.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-05-31T20:39:20.956172Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-05-31T20:39:26Z" } 2023-05-31T20:39:26.409 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1685562631.516089_59.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-05-31T19:50:31.516089Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1685562970.274572_93.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-05-31T19:56:10.274572Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1685565560.956172_429.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-05-31T20:39:20.956172Z', 'realm_epoch': 2}}] 2023-05-31T20:39:26.410 DEBUG:rgw_multi.tests:log_status={0: '1_1685562631.516089_59.1', 1: '', 2: '1_1685562970.274572_93.1', 3: '1_1685565560.956172_429.1'} 2023-05-31T20:39:26.410 DEBUG:rgw_multi.tests:sync_status={0: '1_1685562631.516089_59.1', 1: '', 2: '1_1685562970.274572_93.1', 3: '1_1685565560.956172_429.1'} 2023-05-31T20:39:26.410 INFO:rgw_multi.tests:finish meta checkpoint for zone=a2 2023-05-31T20:39:26.411 DEBUG:boto:path=/pklxkd-59/ 2023-05-31T20:39:26.411 DEBUG:boto:auth_path=/pklxkd-59/ 2023-05-31T20:39:26.411 DEBUG:boto:Method: HEAD 2023-05-31T20:39:26.412 DEBUG:boto:Path: /pklxkd-59/ 2023-05-31T20:39:26.413 DEBUG:boto:Data: 2023-05-31T20:39:26.414 DEBUG:boto:Headers: {} 2023-05-31T20:39:26.415 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:39:26.415 DEBUG:boto:Port: 8000 2023-05-31T20:39:26.416 DEBUG:boto:Params: {} 2023-05-31T20:39:26.417 DEBUG:boto:Token: None 2023-05-31T20:39:26.418 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:39:26 GMT /pklxkd-59/ 2023-05-31T20:39:26.419 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:tXBynH0OtEm6OhAHklETCiwFGjI= 2023-05-31T20:39:26.420 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:39:26 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:tXBynH0OtEm6OhAHklETCiwFGjI=', 'Content-Length': '0'} 2023-05-31T20:39:26.424 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000028294154385b8ac5-006477b07e-1630-a1'), ('Content-Length', '0'), ('Date', 'Wed, 31 May 2023 20:39:26 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:39:26.424 DEBUG:boto:path=/pklxkd-59/obj11 2023-05-31T20:39:26.425 DEBUG:boto:auth_path=/pklxkd-59/obj11 2023-05-31T20:39:26.425 DEBUG:boto:Method: PUT 2023-05-31T20:39:26.426 DEBUG:boto:Path: /pklxkd-59/obj11 2023-05-31T20:39:26.426 DEBUG:boto:Data: 2023-05-31T20:39:26.426 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-05-31T20:39:26.427 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:39:26.427 DEBUG:boto:Port: 8000 2023-05-31T20:39:26.427 DEBUG:boto:Params: {} 2023-05-31T20:39:26.428 DEBUG:boto:Token: None 2023-05-31T20:39:26.428 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Wed, 31 May 2023 20:39:26 GMT /pklxkd-59/obj11 2023-05-31T20:39:26.429 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:kgVE/vHA/S++y0SEiBnXFUUWq7k= 2023-05-31T20:39:26.429 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Wed, 31 May 2023 20:39:26 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:kgVE/vHA/S++y0SEiBnXFUUWq7k='} 2023-05-31T20:39:26.450 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000007e8559b5cf75daa1-006477b07e-1630-a1'), ('Date', 'Wed, 31 May 2023 20:39:26 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:39:26.451 DEBUG:boto:path=/pklxkd-59/ 2023-05-31T20:39:26.452 DEBUG:boto:auth_path=/pklxkd-59/ 2023-05-31T20:39:26.453 DEBUG:boto:Method: HEAD 2023-05-31T20:39:26.454 DEBUG:boto:Path: /pklxkd-59/ 2023-05-31T20:39:26.455 DEBUG:boto:Data: 2023-05-31T20:39:26.455 DEBUG:boto:Headers: {} 2023-05-31T20:39:26.457 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:39:26.458 DEBUG:boto:Port: 8000 2023-05-31T20:39:26.459 DEBUG:boto:Params: {} 2023-05-31T20:39:26.460 DEBUG:boto:Token: None 2023-05-31T20:39:26.460 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:39:26 GMT /pklxkd-59/ 2023-05-31T20:39:26.461 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:tXBynH0OtEm6OhAHklETCiwFGjI= 2023-05-31T20:39:26.462 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:39:26 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:tXBynH0OtEm6OhAHklETCiwFGjI=', 'Content-Length': '0'} 2023-05-31T20:39:26.465 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '3'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000001a2ca263680037c2-006477b07e-1630-a1'), ('Content-Length', '0'), ('Date', 'Wed, 31 May 2023 20:39:26 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:39:26.467 DEBUG:boto:path=/pklxkd-59/obj22 2023-05-31T20:39:26.468 DEBUG:boto:auth_path=/pklxkd-59/obj22 2023-05-31T20:39:26.469 DEBUG:boto:Method: PUT 2023-05-31T20:39:26.469 DEBUG:boto:Path: /pklxkd-59/obj22 2023-05-31T20:39:26.469 DEBUG:boto:Data: 2023-05-31T20:39:26.470 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-05-31T20:39:26.470 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:39:26.471 DEBUG:boto:Port: 8000 2023-05-31T20:39:26.471 DEBUG:boto:Params: {} 2023-05-31T20:39:26.471 DEBUG:boto:Token: None 2023-05-31T20:39:26.472 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Wed, 31 May 2023 20:39:26 GMT /pklxkd-59/obj22 2023-05-31T20:39:26.473 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:lpwtnG8iWp1AM5MRyTc2ofIjs6Y= 2023-05-31T20:39:26.474 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Wed, 31 May 2023 20:39:26 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:lpwtnG8iWp1AM5MRyTc2ofIjs6Y='} 2023-05-31T20:39:26.516 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000d10dafd86ef40501-006477b07e-1630-a1'), ('Date', 'Wed, 31 May 2023 20:39:26 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:39:26.517 DEBUG:boto:path=/pklxkd-59/ 2023-05-31T20:39:26.518 DEBUG:boto:auth_path=/pklxkd-59/ 2023-05-31T20:39:26.519 DEBUG:boto:Method: HEAD 2023-05-31T20:39:26.519 DEBUG:boto:Path: /pklxkd-59/ 2023-05-31T20:39:26.520 DEBUG:boto:Data: 2023-05-31T20:39:26.520 DEBUG:boto:Headers: {} 2023-05-31T20:39:26.521 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:39:26.521 DEBUG:boto:Port: 8000 2023-05-31T20:39:26.522 DEBUG:boto:Params: {} 2023-05-31T20:39:26.522 DEBUG:boto:Token: None 2023-05-31T20:39:26.523 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:39:26 GMT /pklxkd-59/ 2023-05-31T20:39:26.523 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:tXBynH0OtEm6OhAHklETCiwFGjI= 2023-05-31T20:39:26.524 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:39:26 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:tXBynH0OtEm6OhAHklETCiwFGjI=', 'Content-Length': '0'} 2023-05-31T20:39:26.528 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '2'), ('X-RGW-Bytes-Used', '6'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000007a4a88f77da7178-006477b07e-1630-a1'), ('Content-Length', '0'), ('Date', 'Wed, 31 May 2023 20:39:26 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:39:26.529 DEBUG:boto:path=/pklxkd-59/obj33 2023-05-31T20:39:26.529 DEBUG:boto:auth_path=/pklxkd-59/obj33 2023-05-31T20:39:26.530 DEBUG:boto:Method: PUT 2023-05-31T20:39:26.530 DEBUG:boto:Path: /pklxkd-59/obj33 2023-05-31T20:39:26.531 DEBUG:boto:Data: 2023-05-31T20:39:26.531 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-05-31T20:39:26.532 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:39:26.532 DEBUG:boto:Port: 8000 2023-05-31T20:39:26.533 DEBUG:boto:Params: {} 2023-05-31T20:39:26.534 DEBUG:boto:Token: None 2023-05-31T20:39:26.535 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Wed, 31 May 2023 20:39:26 GMT /pklxkd-59/obj33 2023-05-31T20:39:26.535 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:PZytG+HHJOdKCC/1YITfpsUlzt4= 2023-05-31T20:39:26.536 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Wed, 31 May 2023 20:39:26 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:PZytG+HHJOdKCC/1YITfpsUlzt4='} 2023-05-31T20:39:26.548 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000d16f2f78ef720b4a-006477b07e-1630-a1'), ('Date', 'Wed, 31 May 2023 20:39:26 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:39:26.549 DEBUG:boto:path=/pklxkd-59/ 2023-05-31T20:39:26.550 DEBUG:boto:auth_path=/pklxkd-59/ 2023-05-31T20:39:26.551 DEBUG:boto:Method: HEAD 2023-05-31T20:39:26.552 DEBUG:boto:Path: /pklxkd-59/ 2023-05-31T20:39:26.553 DEBUG:boto:Data: 2023-05-31T20:39:26.553 DEBUG:boto:Headers: {} 2023-05-31T20:39:26.554 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:39:26.555 DEBUG:boto:Port: 8000 2023-05-31T20:39:26.555 DEBUG:boto:Params: {} 2023-05-31T20:39:26.556 DEBUG:boto:Token: None 2023-05-31T20:39:26.557 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:39:26 GMT /pklxkd-59/ 2023-05-31T20:39:26.558 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:tXBynH0OtEm6OhAHklETCiwFGjI= 2023-05-31T20:39:26.558 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:39:26 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:tXBynH0OtEm6OhAHklETCiwFGjI=', 'Content-Length': '0'} 2023-05-31T20:39:26.561 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '3'), ('X-RGW-Bytes-Used', '9'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000019b01e6e0c7b71cb-006477b07e-1630-a1'), ('Content-Length', '0'), ('Date', 'Wed, 31 May 2023 20:39:26 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:39:26.562 DEBUG:boto:path=/pklxkd-59/obj44 2023-05-31T20:39:26.562 DEBUG:boto:auth_path=/pklxkd-59/obj44 2023-05-31T20:39:26.563 DEBUG:boto:Method: PUT 2023-05-31T20:39:26.563 DEBUG:boto:Path: /pklxkd-59/obj44 2023-05-31T20:39:26.563 DEBUG:boto:Data: 2023-05-31T20:39:26.564 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-05-31T20:39:26.564 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:39:26.565 DEBUG:boto:Port: 8000 2023-05-31T20:39:26.565 DEBUG:boto:Params: {} 2023-05-31T20:39:26.565 DEBUG:boto:Token: None 2023-05-31T20:39:26.566 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Wed, 31 May 2023 20:39:26 GMT /pklxkd-59/obj44 2023-05-31T20:39:26.567 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:hFl+tkJY28SyhIxcmYNzkPGnltY= 2023-05-31T20:39:26.567 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Wed, 31 May 2023 20:39:26 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:hFl+tkJY28SyhIxcmYNzkPGnltY='} 2023-05-31T20:39:26.612 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000015565f9edde6587f-006477b07e-1630-a1'), ('Date', 'Wed, 31 May 2023 20:39:26 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:39:26.613 DEBUG:boto:path=/pklxkd-59/ 2023-05-31T20:39:26.613 DEBUG:boto:auth_path=/pklxkd-59/ 2023-05-31T20:39:26.614 DEBUG:boto:Method: HEAD 2023-05-31T20:39:26.614 DEBUG:boto:Path: /pklxkd-59/ 2023-05-31T20:39:26.614 DEBUG:boto:Data: 2023-05-31T20:39:26.615 DEBUG:boto:Headers: {} 2023-05-31T20:39:26.615 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:39:26.615 DEBUG:boto:Port: 8000 2023-05-31T20:39:26.616 DEBUG:boto:Params: {} 2023-05-31T20:39:26.616 DEBUG:boto:Token: None 2023-05-31T20:39:26.617 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:39:26 GMT /pklxkd-59/ 2023-05-31T20:39:26.617 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:tXBynH0OtEm6OhAHklETCiwFGjI= 2023-05-31T20:39:26.618 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:39:26 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:tXBynH0OtEm6OhAHklETCiwFGjI=', 'Content-Length': '0'} 2023-05-31T20:39:26.620 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '4'), ('X-RGW-Bytes-Used', '12'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000018cb0a1690dac915-006477b07e-1630-a1'), ('Content-Length', '0'), ('Date', 'Wed, 31 May 2023 20:39:26 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:39:26.621 DEBUG:boto:path=/pklxkd-59/obj55 2023-05-31T20:39:26.622 DEBUG:boto:auth_path=/pklxkd-59/obj55 2023-05-31T20:39:26.623 DEBUG:boto:Method: PUT 2023-05-31T20:39:26.624 DEBUG:boto:Path: /pklxkd-59/obj55 2023-05-31T20:39:26.624 DEBUG:boto:Data: 2023-05-31T20:39:26.625 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-05-31T20:39:26.626 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:39:26.626 DEBUG:boto:Port: 8000 2023-05-31T20:39:26.627 DEBUG:boto:Params: {} 2023-05-31T20:39:26.628 DEBUG:boto:Token: None 2023-05-31T20:39:26.629 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Wed, 31 May 2023 20:39:26 GMT /pklxkd-59/obj55 2023-05-31T20:39:26.630 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:XzPFLB/FxsrXiYlpjJLvp4dCruM= 2023-05-31T20:39:26.631 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Wed, 31 May 2023 20:39:26 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:XzPFLB/FxsrXiYlpjJLvp4dCruM='} 2023-05-31T20:39:26.638 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000c5e692e8b1d6cc30-006477b07e-1630-a1'), ('Date', 'Wed, 31 May 2023 20:39:26 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:39:26.639 DEBUG:rgw_multi.tests:bucket checkpoint: source=a1 target=a2 bucket=pklxkd-59 2023-05-31T20:39:26.639 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'pklxkd-59', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-05-31T20:39:26.639 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'pklxkd-59', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-05-31T20:39:26.640 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket pklxkd-59 --source-zone a1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2023-05-31T20:39:26.697 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2023-05-31T20:39:26.697 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2023-05-31T20:39:26.744 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:39:26.737+0000 7fed78406780 1 waiting to reach incremental sync.. 2023-05-31T20:39:31.743 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:39:31.739+0000 7fed78406780 1 waiting to reach incremental sync.. 2023-05-31T20:39:36.744 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:39:36.740+0000 7fed78406780 1 waiting to reach incremental sync.. 2023-05-31T20:39:41.744 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:39:41.740+0000 7fed78406780 1 waiting to reach incremental sync.. 2023-05-31T20:39:46.745 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:39:46.741+0000 7fed78406780 1 waiting to reach incremental sync.. 2023-05-31T20:39:51.746 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:39:51.740+0000 7fed78406780 1 waiting to reach incremental sync.. 2023-05-31T20:39:56.746 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:39:56.741+0000 7fed78406780 1 waiting to reach incremental sync.. 2023-05-31T20:40:01.747 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:40:01.742+0000 7fed78406780 1 waiting to reach incremental sync.. 2023-05-31T20:40:06.748 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:40:06.743+0000 7fed78406780 1 waiting to reach incremental sync.. 2023-05-31T20:40:11.748 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:40:11.743+0000 7fed78406780 1 waiting to reach incremental sync.. 2023-05-31T20:40:16.749 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:40:16.745+0000 7fed78406780 1 waiting to reach incremental sync.. 2023-05-31T20:40:21.750 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:40:21.745+0000 7fed78406780 1 waiting to reach incremental sync.. 2023-05-31T20:40:26.750 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:40:26.746+0000 7fed78406780 1 waiting to reach incremental sync.. 2023-05-31T20:40:31.751 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:40:31.747+0000 7fed78406780 1 waiting to reach incremental sync.. 2023-05-31T20:40:36.751 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:40:36.746+0000 7fed78406780 1 waiting to reach incremental sync.. 2023-05-31T20:40:41.752 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:40:41.747+0000 7fed78406780 1 waiting to reach incremental sync.. 2023-05-31T20:40:46.753 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:40:46.748+0000 7fed78406780 1 waiting to reach incremental sync.. 2023-05-31T20:40:51.754 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:40:51.749+0000 7fed78406780 1 waiting to reach incremental sync.. 2023-05-31T20:40:56.754 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:40:56.750+0000 7fed78406780 1 waiting to reach incremental sync.. 2023-05-31T20:41:01.755 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:41:01.751+0000 7fed78406780 1 waiting to reach incremental sync.. 2023-05-31T20:41:06.756 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:41:06.751+0000 7fed78406780 1 waiting to reach incremental sync.. 2023-05-31T20:41:11.756 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:41:11.752+0000 7fed78406780 1 waiting to reach incremental sync.. 2023-05-31T20:41:16.757 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:41:16.752+0000 7fed78406780 1 waiting to reach incremental sync.. 2023-05-31T20:41:21.758 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:41:21.752+0000 7fed78406780 1 waiting to reach incremental sync.. 2023-05-31T20:41:26.758 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:41:26.753+0000 7fed78406780 1 waiting to reach incremental sync.. 2023-05-31T20:41:31.759 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:41:31.754+0000 7fed78406780 1 waiting to reach incremental sync.. 2023-05-31T20:41:36.760 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:41:36.755+0000 7fed78406780 1 waiting to reach incremental sync.. 2023-05-31T20:41:41.760 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:41:41.756+0000 7fed78406780 1 waiting to reach incremental sync.. 2023-05-31T20:41:46.761 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:41:46.757+0000 7fed78406780 1 waiting to reach incremental sync.. 2023-05-31T20:41:51.762 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:41:51.757+0000 7fed78406780 1 waiting to reach incremental sync.. 2023-05-31T20:41:56.762 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:41:56.758+0000 7fed78406780 1 waiting to reach incremental sync.. 2023-05-31T20:42:01.763 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:42:01.758+0000 7fed78406780 1 waiting to reach incremental sync.. 2023-05-31T20:42:06.763 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:42:06.758+0000 7fed78406780 1 waiting to reach incremental sync.. 2023-05-31T20:42:11.764 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:42:11.759+0000 7fed78406780 1 waiting to reach incremental sync.. 2023-05-31T20:42:16.765 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:42:16.760+0000 7fed78406780 1 waiting to reach incremental sync.. 2023-05-31T20:42:21.765 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:42:21.761+0000 7fed78406780 1 waiting to reach incremental sync.. 2023-05-31T20:42:26.766 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:42:26.762+0000 7fed78406780 1 waiting to reach incremental sync.. 2023-05-31T20:42:31.767 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:42:31.763+0000 7fed78406780 1 waiting to reach incremental sync.. 2023-05-31T20:42:36.768 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:42:36.763+0000 7fed78406780 1 waiting to reach incremental sync.. 2023-05-31T20:42:41.768 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:42:41.763+0000 7fed78406780 1 waiting to reach incremental sync.. 2023-05-31T20:42:46.769 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:42:46.763+0000 7fed78406780 1 waiting to reach incremental sync.. 2023-05-31T20:42:51.769 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:42:51.764+0000 7fed78406780 1 waiting to reach incremental sync.. 2023-05-31T20:42:56.770 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:42:56.765+0000 7fed78406780 1 waiting to reach incremental sync.. 2023-05-31T20:43:01.771 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:43:01.766+0000 7fed78406780 1 waiting to reach incremental sync.. 2023-05-31T20:43:06.773 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:43:06.768+0000 7fed78406780 1 bucket sync caught up with source: 2023-05-31T20:43:06.773 INFO:teuthology.orchestra.run.smithi099.stderr: local status: [, , , , , , , , 00000000001.66.6, 00000000002.63.6, 00000000002.53.6] 2023-05-31T20:43:06.773 INFO:teuthology.orchestra.run.smithi099.stderr: remote markers: [, , , , , , , , 00000000001.66.6, 00000000002.63.6, 00000000002.53.6] 2023-05-31T20:43:06.774 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:43:06.768+0000 7fed78406780 0 bucket checkpoint complete 2023-05-31T20:43:06.792 DEBUG:rgw_multi.tests:bucket checkpoint: source=a2 target=a1 bucket=pklxkd-59 2023-05-31T20:43:06.793 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'pklxkd-59', '--source-zone', 'a2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-05-31T20:43:06.793 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'pklxkd-59', '--source-zone', 'a2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-05-31T20:43:06.794 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket pklxkd-59 --source-zone a2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2023-05-31T20:43:06.856 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2023-05-31T20:43:06.856 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2023-05-31T20:43:06.897 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:43:06.893+0000 7f51d7c68780 1 bucket sync caught up with source: 2023-05-31T20:43:06.898 INFO:teuthology.orchestra.run.smithi099.stderr: local status: [, , , , , , , , 00000000001.1995.6, 00000000002.2283.6, 00000000002.2004.6] 2023-05-31T20:43:06.898 INFO:teuthology.orchestra.run.smithi099.stderr: remote markers: [, , , , , , , , 00000000001.1995.6, 00000000002.2283.6, 00000000002.2004.6] 2023-05-31T20:43:06.898 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:43:06.893+0000 7f51d7c68780 0 bucket checkpoint complete 2023-05-31T20:43:06.911 INFO:tasks.rgw_multi.zone_rados:comparing bucket=pklxkd-59 zones={a2, a1} 2023-05-31T20:43:06.912 DEBUG:boto:path=/pklxkd-59/ 2023-05-31T20:43:06.912 DEBUG:boto:auth_path=/pklxkd-59/ 2023-05-31T20:43:06.913 DEBUG:boto:Method: HEAD 2023-05-31T20:43:06.913 DEBUG:boto:Path: /pklxkd-59/ 2023-05-31T20:43:06.913 DEBUG:boto:Data: 2023-05-31T20:43:06.914 DEBUG:boto:Headers: {} 2023-05-31T20:43:06.914 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:43:06.914 DEBUG:boto:Port: 8001 2023-05-31T20:43:06.915 DEBUG:boto:Params: {} 2023-05-31T20:43:06.915 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8001} 2023-05-31T20:43:06.916 DEBUG:boto:Token: None 2023-05-31T20:43:06.916 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:43:06 GMT /pklxkd-59/ 2023-05-31T20:43:06.917 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:RlJXiy2xqwnf2z41/yhrn4zNaGA= 2023-05-31T20:43:06.917 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:43:06 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:RlJXiy2xqwnf2z41/yhrn4zNaGA=', 'Content-Length': '0'} 2023-05-31T20:43:06.921 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '5'), ('X-RGW-Bytes-Used', '15'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000b33380d9a7d5505f-006477b15a-1c17-a2'), ('Content-Length', '0'), ('Date', 'Wed, 31 May 2023 20:43:06 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:43:06.922 DEBUG:boto:path=/pklxkd-59/ 2023-05-31T20:43:06.923 DEBUG:boto:auth_path=/pklxkd-59/ 2023-05-31T20:43:06.923 DEBUG:boto:Method: HEAD 2023-05-31T20:43:06.924 DEBUG:boto:Path: /pklxkd-59/ 2023-05-31T20:43:06.925 DEBUG:boto:Data: 2023-05-31T20:43:06.926 DEBUG:boto:Headers: {} 2023-05-31T20:43:06.926 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:43:06.927 DEBUG:boto:Port: 8000 2023-05-31T20:43:06.928 DEBUG:boto:Params: {} 2023-05-31T20:43:06.929 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2023-05-31T20:43:06.930 DEBUG:boto:Token: None 2023-05-31T20:43:06.931 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:43:06 GMT /pklxkd-59/ 2023-05-31T20:43:06.932 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:RlJXiy2xqwnf2z41/yhrn4zNaGA= 2023-05-31T20:43:06.933 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:43:06 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:RlJXiy2xqwnf2z41/yhrn4zNaGA=', 'Content-Length': '0'} 2023-05-31T20:43:06.937 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '5'), ('X-RGW-Bytes-Used', '15'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000bc229b3d17742672-006477b15a-1630-a1'), ('Content-Length', '0'), ('Date', 'Wed, 31 May 2023 20:43:06 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:43:06.938 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2023-05-31T20:43:06.938 DEBUG:boto:path=/pklxkd-59/ 2023-05-31T20:43:06.939 DEBUG:boto:auth_path=/pklxkd-59/ 2023-05-31T20:43:06.939 DEBUG:boto:path=/pklxkd-59/?versions&max-keys=999 2023-05-31T20:43:06.939 DEBUG:boto:auth_path=/pklxkd-59/?versions&max-keys=999 2023-05-31T20:43:06.940 DEBUG:boto:Method: GET 2023-05-31T20:43:06.940 DEBUG:boto:Path: /pklxkd-59/?versions&max-keys=999 2023-05-31T20:43:06.941 DEBUG:boto:Data: 2023-05-31T20:43:06.941 DEBUG:boto:Headers: {} 2023-05-31T20:43:06.941 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:43:06.942 DEBUG:boto:Port: 8001 2023-05-31T20:43:06.942 DEBUG:boto:Params: {} 2023-05-31T20:43:06.942 DEBUG:boto:Token: None 2023-05-31T20:43:06.943 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:43:06 GMT /pklxkd-59/?versions 2023-05-31T20:43:06.943 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:P2R5Egj7dnQhlfZE0nD8sdc+oiI= 2023-05-31T20:43:06.944 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:43:06 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:P2R5Egj7dnQhlfZE0nD8sdc+oiI=', 'Content-Length': '0'} 2023-05-31T20:43:06.946 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000047d73cc7e8533a19-006477b15a-1c17-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:43:06 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:43:06.946 DEBUG:boto:b'pklxkd-59999falseobj11nulltrue2023-05-31T20:39:26.427Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj22nulltrue2023-05-31T20:39:26.473Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj33nulltrue2023-05-31T20:39:26.534Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj44nulltrue2023-05-31T20:39:26.565Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj55nulltrue2023-05-31T20:39:26.630Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:43:06.947 DEBUG:tasks.rgw_multi.zone_rados:o=obj11 2023-05-31T20:43:06.948 DEBUG:tasks.rgw_multi.zone_rados:o=obj22 2023-05-31T20:43:06.948 DEBUG:tasks.rgw_multi.zone_rados:o=obj33 2023-05-31T20:43:06.949 DEBUG:tasks.rgw_multi.zone_rados:o=obj44 2023-05-31T20:43:06.949 DEBUG:tasks.rgw_multi.zone_rados:o=obj55 2023-05-31T20:43:06.949 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2023-05-31T20:43:06.950 DEBUG:boto:path=/pklxkd-59/ 2023-05-31T20:43:06.950 DEBUG:boto:auth_path=/pklxkd-59/ 2023-05-31T20:43:06.950 DEBUG:boto:path=/pklxkd-59/?versions&max-keys=999 2023-05-31T20:43:06.951 DEBUG:boto:auth_path=/pklxkd-59/?versions&max-keys=999 2023-05-31T20:43:06.951 DEBUG:boto:Method: GET 2023-05-31T20:43:06.952 DEBUG:boto:Path: /pklxkd-59/?versions&max-keys=999 2023-05-31T20:43:06.952 DEBUG:boto:Data: 2023-05-31T20:43:06.952 DEBUG:boto:Headers: {} 2023-05-31T20:43:06.953 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:43:06.953 DEBUG:boto:Port: 8000 2023-05-31T20:43:06.953 DEBUG:boto:Params: {} 2023-05-31T20:43:06.954 DEBUG:boto:Token: None 2023-05-31T20:43:06.954 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:43:06 GMT /pklxkd-59/?versions 2023-05-31T20:43:06.955 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:P2R5Egj7dnQhlfZE0nD8sdc+oiI= 2023-05-31T20:43:06.955 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:43:06 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:P2R5Egj7dnQhlfZE0nD8sdc+oiI=', 'Content-Length': '0'} 2023-05-31T20:43:06.958 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000335a6d1e9dfade49-006477b15a-1630-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:43:06 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:43:06.958 DEBUG:boto:b'pklxkd-59999falseobj11nulltrue2023-05-31T20:39:26.427Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj22nulltrue2023-05-31T20:39:26.473Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj33nulltrue2023-05-31T20:39:26.534Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj44nulltrue2023-05-31T20:39:26.565Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj55nulltrue2023-05-31T20:39:26.630Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:43:06.959 DEBUG:tasks.rgw_multi.zone_rados:o=obj11 2023-05-31T20:43:06.959 DEBUG:tasks.rgw_multi.zone_rados:o=obj22 2023-05-31T20:43:06.960 DEBUG:tasks.rgw_multi.zone_rados:o=obj33 2023-05-31T20:43:06.960 DEBUG:tasks.rgw_multi.zone_rados:o=obj44 2023-05-31T20:43:06.960 DEBUG:tasks.rgw_multi.zone_rados:o=obj55 2023-05-31T20:43:06.961 DEBUG:boto:path=/pklxkd-59/ 2023-05-31T20:43:06.961 DEBUG:boto:auth_path=/pklxkd-59/ 2023-05-31T20:43:06.961 DEBUG:boto:path=/pklxkd-59/?versions&max-keys=999 2023-05-31T20:43:06.962 DEBUG:boto:auth_path=/pklxkd-59/?versions&max-keys=999 2023-05-31T20:43:06.962 DEBUG:boto:Method: GET 2023-05-31T20:43:06.963 DEBUG:boto:Path: /pklxkd-59/?versions&max-keys=999 2023-05-31T20:43:06.963 DEBUG:boto:Data: 2023-05-31T20:43:06.964 DEBUG:boto:Headers: {} 2023-05-31T20:43:06.965 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:43:06.965 DEBUG:boto:Port: 8001 2023-05-31T20:43:06.966 DEBUG:boto:Params: {} 2023-05-31T20:43:06.966 DEBUG:boto:Token: None 2023-05-31T20:43:06.967 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:43:06 GMT /pklxkd-59/?versions 2023-05-31T20:43:06.968 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:P2R5Egj7dnQhlfZE0nD8sdc+oiI= 2023-05-31T20:43:06.968 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:43:06 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:P2R5Egj7dnQhlfZE0nD8sdc+oiI=', 'Content-Length': '0'} 2023-05-31T20:43:06.970 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000005a51d737fccb416d-006477b15a-1c17-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:43:06 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:43:06.971 DEBUG:boto:b'pklxkd-59999falseobj11nulltrue2023-05-31T20:39:26.427Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj22nulltrue2023-05-31T20:39:26.473Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj33nulltrue2023-05-31T20:39:26.534Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj44nulltrue2023-05-31T20:39:26.565Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj55nulltrue2023-05-31T20:39:26.630Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:43:06.972 DEBUG:boto:path=/pklxkd-59/ 2023-05-31T20:43:06.972 DEBUG:boto:auth_path=/pklxkd-59/ 2023-05-31T20:43:06.973 DEBUG:boto:path=/pklxkd-59/?versions&max-keys=999 2023-05-31T20:43:06.973 DEBUG:boto:auth_path=/pklxkd-59/?versions&max-keys=999 2023-05-31T20:43:06.974 DEBUG:boto:Method: GET 2023-05-31T20:43:06.974 DEBUG:boto:Path: /pklxkd-59/?versions&max-keys=999 2023-05-31T20:43:06.975 DEBUG:boto:Data: 2023-05-31T20:43:06.975 DEBUG:boto:Headers: {} 2023-05-31T20:43:06.976 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:43:06.976 DEBUG:boto:Port: 8000 2023-05-31T20:43:06.977 DEBUG:boto:Params: {} 2023-05-31T20:43:06.977 DEBUG:boto:Token: None 2023-05-31T20:43:06.978 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:43:06 GMT /pklxkd-59/?versions 2023-05-31T20:43:06.978 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:P2R5Egj7dnQhlfZE0nD8sdc+oiI= 2023-05-31T20:43:06.979 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:43:06 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:P2R5Egj7dnQhlfZE0nD8sdc+oiI=', 'Content-Length': '0'} 2023-05-31T20:43:06.981 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000004434be3f349a87c4-006477b15a-1630-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:43:06 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:43:06.981 DEBUG:boto:b'pklxkd-59999falseobj11nulltrue2023-05-31T20:39:26.427Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj22nulltrue2023-05-31T20:39:26.473Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj33nulltrue2023-05-31T20:39:26.534Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj44nulltrue2023-05-31T20:39:26.565Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj55nulltrue2023-05-31T20:39:26.630Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:43:06.982 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj11 2023-05-31T20:43:06.982 DEBUG:boto:path=/pklxkd-59/obj11 2023-05-31T20:43:06.983 DEBUG:boto:auth_path=/pklxkd-59/obj11 2023-05-31T20:43:06.983 DEBUG:boto:path=/pklxkd-59/obj11?versionId=null 2023-05-31T20:43:06.984 DEBUG:boto:auth_path=/pklxkd-59/obj11?versionId=null 2023-05-31T20:43:06.984 DEBUG:boto:Method: GET 2023-05-31T20:43:06.984 DEBUG:boto:Path: /pklxkd-59/obj11?versionId=null 2023-05-31T20:43:06.985 DEBUG:boto:Data: 2023-05-31T20:43:06.985 DEBUG:boto:Headers: {} 2023-05-31T20:43:06.985 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:43:06.986 DEBUG:boto:Port: 8001 2023-05-31T20:43:06.986 DEBUG:boto:Params: {} 2023-05-31T20:43:06.987 DEBUG:boto:Token: None 2023-05-31T20:43:06.987 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:43:06 GMT /pklxkd-59/obj11?versionId=null 2023-05-31T20:43:06.987 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:2TL7JGpr8YlJ8A1EgHb5F7KNvBE= 2023-05-31T20:43:06.988 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:43:06 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:2TL7JGpr8YlJ8A1EgHb5F7KNvBE=', 'Content-Length': '0'} 2023-05-31T20:43:06.990 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:39:26 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e31443ad0280d211-006477b15a-1c17-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:43:06 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:43:06.990 DEBUG:boto:path=/pklxkd-59/obj11 2023-05-31T20:43:06.991 DEBUG:boto:auth_path=/pklxkd-59/obj11 2023-05-31T20:43:06.991 DEBUG:boto:path=/pklxkd-59/obj11?versionId=null 2023-05-31T20:43:06.991 DEBUG:boto:auth_path=/pklxkd-59/obj11?versionId=null 2023-05-31T20:43:06.992 DEBUG:boto:Method: GET 2023-05-31T20:43:06.992 DEBUG:boto:Path: /pklxkd-59/obj11?versionId=null 2023-05-31T20:43:06.993 DEBUG:boto:Data: 2023-05-31T20:43:06.993 DEBUG:boto:Headers: {} 2023-05-31T20:43:06.993 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:43:06.994 DEBUG:boto:Port: 8000 2023-05-31T20:43:06.994 DEBUG:boto:Params: {} 2023-05-31T20:43:06.994 DEBUG:boto:Token: None 2023-05-31T20:43:06.995 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:43:06 GMT /pklxkd-59/obj11?versionId=null 2023-05-31T20:43:06.995 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:2TL7JGpr8YlJ8A1EgHb5F7KNvBE= 2023-05-31T20:43:06.996 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:43:06 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:2TL7JGpr8YlJ8A1EgHb5F7KNvBE=', 'Content-Length': '0'} 2023-05-31T20:43:06.997 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:39:26 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ebd9c0701a6eb46e-006477b15a-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:43:06 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:43:06.998 DEBUG:boto:path=/pklxkd-59/obj11 2023-05-31T20:43:06.998 DEBUG:boto:auth_path=/pklxkd-59/obj11 2023-05-31T20:43:06.999 DEBUG:boto:path=/pklxkd-59/obj11?versionId=null 2023-05-31T20:43:06.999 DEBUG:boto:auth_path=/pklxkd-59/obj11?versionId=null 2023-05-31T20:43:06.999 DEBUG:boto:Method: HEAD 2023-05-31T20:43:07.000 DEBUG:boto:Path: /pklxkd-59/obj11?versionId=null 2023-05-31T20:43:07.000 DEBUG:boto:Data: 2023-05-31T20:43:07.001 DEBUG:boto:Headers: {} 2023-05-31T20:43:07.001 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:43:07.001 DEBUG:boto:Port: 8001 2023-05-31T20:43:07.002 DEBUG:boto:Params: {} 2023-05-31T20:43:07.002 DEBUG:boto:Token: None 2023-05-31T20:43:07.002 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:43:07 GMT /pklxkd-59/obj11?versionId=null 2023-05-31T20:43:07.003 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:r2WiW3/PP6/siEfF7j2QGxzMaoE= 2023-05-31T20:43:07.003 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:43:07 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:r2WiW3/PP6/siEfF7j2QGxzMaoE=', 'Content-Length': '0'} 2023-05-31T20:43:07.004 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:39:26 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000cdaecec2e858f0d5-006477b15b-1c17-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:43:07 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:43:07.005 DEBUG:boto:path=/pklxkd-59/obj11 2023-05-31T20:43:07.005 DEBUG:boto:auth_path=/pklxkd-59/obj11 2023-05-31T20:43:07.006 DEBUG:boto:path=/pklxkd-59/obj11?versionId=null 2023-05-31T20:43:07.006 DEBUG:boto:auth_path=/pklxkd-59/obj11?versionId=null 2023-05-31T20:43:07.006 DEBUG:boto:Method: HEAD 2023-05-31T20:43:07.007 DEBUG:boto:Path: /pklxkd-59/obj11?versionId=null 2023-05-31T20:43:07.007 DEBUG:boto:Data: 2023-05-31T20:43:07.007 DEBUG:boto:Headers: {} 2023-05-31T20:43:07.008 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:43:07.008 DEBUG:boto:Port: 8000 2023-05-31T20:43:07.008 DEBUG:boto:Params: {} 2023-05-31T20:43:07.009 DEBUG:boto:Token: None 2023-05-31T20:43:07.009 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:43:07 GMT /pklxkd-59/obj11?versionId=null 2023-05-31T20:43:07.010 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:r2WiW3/PP6/siEfF7j2QGxzMaoE= 2023-05-31T20:43:07.010 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:43:07 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:r2WiW3/PP6/siEfF7j2QGxzMaoE=', 'Content-Length': '0'} 2023-05-31T20:43:07.012 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:39:26 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000014c8bfb9bc23f0bc-006477b15b-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:43:07 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:43:07.012 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj11 2023-05-31T20:43:07.013 DEBUG:boto:path=/pklxkd-59/obj11 2023-05-31T20:43:07.013 DEBUG:boto:auth_path=/pklxkd-59/obj11 2023-05-31T20:43:07.013 DEBUG:boto:path=/pklxkd-59/obj11?versionId=null 2023-05-31T20:43:07.014 DEBUG:boto:auth_path=/pklxkd-59/obj11?versionId=null 2023-05-31T20:43:07.014 DEBUG:boto:Method: GET 2023-05-31T20:43:07.014 DEBUG:boto:Path: /pklxkd-59/obj11?versionId=null 2023-05-31T20:43:07.015 DEBUG:boto:Data: 2023-05-31T20:43:07.015 DEBUG:boto:Headers: {} 2023-05-31T20:43:07.015 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:43:07.016 DEBUG:boto:Port: 8001 2023-05-31T20:43:07.016 DEBUG:boto:Params: {} 2023-05-31T20:43:07.017 DEBUG:boto:Token: None 2023-05-31T20:43:07.017 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:43:07 GMT /pklxkd-59/obj11?versionId=null 2023-05-31T20:43:07.017 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:y+qGZAnKaPGu4ZhkrPvj2DAhzK8= 2023-05-31T20:43:07.018 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:43:07 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:y+qGZAnKaPGu4ZhkrPvj2DAhzK8=', 'Content-Length': '0'} 2023-05-31T20:43:07.019 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:39:26 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000864e43985f38a73d-006477b15b-1c17-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:43:07 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:43:07.020 DEBUG:boto:path=/pklxkd-59/obj11 2023-05-31T20:43:07.021 DEBUG:boto:auth_path=/pklxkd-59/obj11 2023-05-31T20:43:07.021 DEBUG:boto:path=/pklxkd-59/obj11?versionId=null 2023-05-31T20:43:07.021 DEBUG:boto:auth_path=/pklxkd-59/obj11?versionId=null 2023-05-31T20:43:07.022 DEBUG:boto:Method: GET 2023-05-31T20:43:07.022 DEBUG:boto:Path: /pklxkd-59/obj11?versionId=null 2023-05-31T20:43:07.022 DEBUG:boto:Data: 2023-05-31T20:43:07.023 DEBUG:boto:Headers: {} 2023-05-31T20:43:07.023 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:43:07.023 DEBUG:boto:Port: 8000 2023-05-31T20:43:07.024 DEBUG:boto:Params: {} 2023-05-31T20:43:07.024 DEBUG:boto:Token: None 2023-05-31T20:43:07.025 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:43:07 GMT /pklxkd-59/obj11?versionId=null 2023-05-31T20:43:07.025 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:y+qGZAnKaPGu4ZhkrPvj2DAhzK8= 2023-05-31T20:43:07.025 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:43:07 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:y+qGZAnKaPGu4ZhkrPvj2DAhzK8=', 'Content-Length': '0'} 2023-05-31T20:43:07.027 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:39:26 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003141e701f6438266-006477b15b-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:43:07 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:43:07.027 DEBUG:boto:path=/pklxkd-59/ 2023-05-31T20:43:07.028 DEBUG:boto:auth_path=/pklxkd-59/ 2023-05-31T20:43:07.028 DEBUG:boto:path=/pklxkd-59/?prefix=obj11 2023-05-31T20:43:07.029 DEBUG:boto:auth_path=/pklxkd-59/?prefix=obj11 2023-05-31T20:43:07.029 DEBUG:boto:Method: GET 2023-05-31T20:43:07.029 DEBUG:boto:Path: /pklxkd-59/?prefix=obj11 2023-05-31T20:43:07.030 DEBUG:boto:Data: 2023-05-31T20:43:07.030 DEBUG:boto:Headers: {} 2023-05-31T20:43:07.030 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:43:07.031 DEBUG:boto:Port: 8001 2023-05-31T20:43:07.031 DEBUG:boto:Params: {} 2023-05-31T20:43:07.032 DEBUG:boto:Token: None 2023-05-31T20:43:07.032 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:43:07 GMT /pklxkd-59/ 2023-05-31T20:43:07.032 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:mHjWQcyrqf5WjOocPTRbXTtj8YQ= 2023-05-31T20:43:07.033 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:43:07 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:mHjWQcyrqf5WjOocPTRbXTtj8YQ=', 'Content-Length': '0'} 2023-05-31T20:43:07.035 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000059cc52d5837494eb-006477b15b-1c17-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:43:07 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:43:07.035 DEBUG:boto:b'pklxkd-59obj111000falseobj112023-05-31T20:39:26.427Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:43:07.036 DEBUG:boto:path=/pklxkd-59/ 2023-05-31T20:43:07.037 DEBUG:boto:auth_path=/pklxkd-59/ 2023-05-31T20:43:07.037 DEBUG:boto:path=/pklxkd-59/?prefix=obj11 2023-05-31T20:43:07.037 DEBUG:boto:auth_path=/pklxkd-59/?prefix=obj11 2023-05-31T20:43:07.038 DEBUG:boto:Method: GET 2023-05-31T20:43:07.038 DEBUG:boto:Path: /pklxkd-59/?prefix=obj11 2023-05-31T20:43:07.038 DEBUG:boto:Data: 2023-05-31T20:43:07.039 DEBUG:boto:Headers: {} 2023-05-31T20:43:07.039 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:43:07.039 DEBUG:boto:Port: 8000 2023-05-31T20:43:07.040 DEBUG:boto:Params: {} 2023-05-31T20:43:07.040 DEBUG:boto:Token: None 2023-05-31T20:43:07.041 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:43:07 GMT /pklxkd-59/ 2023-05-31T20:43:07.041 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:mHjWQcyrqf5WjOocPTRbXTtj8YQ= 2023-05-31T20:43:07.041 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:43:07 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:mHjWQcyrqf5WjOocPTRbXTtj8YQ=', 'Content-Length': '0'} 2023-05-31T20:43:07.043 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d59437411521a75a-006477b15b-1630-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:43:07 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:43:07.044 DEBUG:boto:b'pklxkd-59obj111000falseobj112023-05-31T20:39:26.427Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:43:07.045 DEBUG:boto:path=/pklxkd-59/obj11 2023-05-31T20:43:07.045 DEBUG:boto:auth_path=/pklxkd-59/obj11 2023-05-31T20:43:07.045 DEBUG:boto:Method: HEAD 2023-05-31T20:43:07.046 DEBUG:boto:Path: /pklxkd-59/obj11 2023-05-31T20:43:07.046 DEBUG:boto:Data: 2023-05-31T20:43:07.046 DEBUG:boto:Headers: {} 2023-05-31T20:43:07.047 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:43:07.047 DEBUG:boto:Port: 8001 2023-05-31T20:43:07.047 DEBUG:boto:Params: {} 2023-05-31T20:43:07.048 DEBUG:boto:Token: None 2023-05-31T20:43:07.048 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:43:07 GMT /pklxkd-59/obj11 2023-05-31T20:43:07.049 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:HT/BPqYvbhZvR74bT8qfGI0RIp0= 2023-05-31T20:43:07.049 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:43:07 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:HT/BPqYvbhZvR74bT8qfGI0RIp0=', 'Content-Length': '0'} 2023-05-31T20:43:07.051 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:39:26 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000832ed3fe62069385-006477b15b-1c17-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:43:07 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:43:07.051 DEBUG:boto:path=/pklxkd-59/obj11 2023-05-31T20:43:07.052 DEBUG:boto:auth_path=/pklxkd-59/obj11 2023-05-31T20:43:07.052 DEBUG:boto:Method: HEAD 2023-05-31T20:43:07.052 DEBUG:boto:Path: /pklxkd-59/obj11 2023-05-31T20:43:07.053 DEBUG:boto:Data: 2023-05-31T20:43:07.053 DEBUG:boto:Headers: {} 2023-05-31T20:43:07.053 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:43:07.054 DEBUG:boto:Port: 8000 2023-05-31T20:43:07.054 DEBUG:boto:Params: {} 2023-05-31T20:43:07.054 DEBUG:boto:Token: None 2023-05-31T20:43:07.055 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:43:07 GMT /pklxkd-59/obj11 2023-05-31T20:43:07.055 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:HT/BPqYvbhZvR74bT8qfGI0RIp0= 2023-05-31T20:43:07.056 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:43:07 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:HT/BPqYvbhZvR74bT8qfGI0RIp0=', 'Content-Length': '0'} 2023-05-31T20:43:07.057 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:39:26 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000073064631f5bd9eae-006477b15b-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:43:07 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:43:07.058 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj11 2023-05-31T20:43:07.058 DEBUG:boto:path=/pklxkd-59/obj11 2023-05-31T20:43:07.059 DEBUG:boto:auth_path=/pklxkd-59/obj11 2023-05-31T20:43:07.059 DEBUG:boto:Method: GET 2023-05-31T20:43:07.059 DEBUG:boto:Path: /pklxkd-59/obj11 2023-05-31T20:43:07.060 DEBUG:boto:Data: 2023-05-31T20:43:07.060 DEBUG:boto:Headers: {} 2023-05-31T20:43:07.061 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:43:07.061 DEBUG:boto:Port: 8001 2023-05-31T20:43:07.061 DEBUG:boto:Params: {} 2023-05-31T20:43:07.062 DEBUG:boto:Token: None 2023-05-31T20:43:07.062 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:43:07 GMT /pklxkd-59/obj11 2023-05-31T20:43:07.063 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:9Sacv3nckion2BM9YUXueGN0Dco= 2023-05-31T20:43:07.063 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:43:07 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:9Sacv3nckion2BM9YUXueGN0Dco=', 'Content-Length': '0'} 2023-05-31T20:43:07.065 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:39:26 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009279d9cf31b0449f-006477b15b-1c17-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:43:07 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:43:07.065 DEBUG:boto:path=/pklxkd-59/obj11 2023-05-31T20:43:07.066 DEBUG:boto:auth_path=/pklxkd-59/obj11 2023-05-31T20:43:07.066 DEBUG:boto:Method: GET 2023-05-31T20:43:07.067 DEBUG:boto:Path: /pklxkd-59/obj11 2023-05-31T20:43:07.067 DEBUG:boto:Data: 2023-05-31T20:43:07.067 DEBUG:boto:Headers: {} 2023-05-31T20:43:07.068 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:43:07.068 DEBUG:boto:Port: 8000 2023-05-31T20:43:07.068 DEBUG:boto:Params: {} 2023-05-31T20:43:07.069 DEBUG:boto:Token: None 2023-05-31T20:43:07.069 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:43:07 GMT /pklxkd-59/obj11 2023-05-31T20:43:07.070 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:9Sacv3nckion2BM9YUXueGN0Dco= 2023-05-31T20:43:07.070 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:43:07 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:9Sacv3nckion2BM9YUXueGN0Dco=', 'Content-Length': '0'} 2023-05-31T20:43:07.072 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:39:26 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002c9e0b78c58eab64-006477b15b-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:43:07 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:43:07.072 DEBUG:boto:path=/pklxkd-59/ 2023-05-31T20:43:07.073 DEBUG:boto:auth_path=/pklxkd-59/ 2023-05-31T20:43:07.073 DEBUG:boto:path=/pklxkd-59/?prefix=obj11 2023-05-31T20:43:07.074 DEBUG:boto:auth_path=/pklxkd-59/?prefix=obj11 2023-05-31T20:43:07.074 DEBUG:boto:Method: GET 2023-05-31T20:43:07.074 DEBUG:boto:Path: /pklxkd-59/?prefix=obj11 2023-05-31T20:43:07.075 DEBUG:boto:Data: 2023-05-31T20:43:07.075 DEBUG:boto:Headers: {} 2023-05-31T20:43:07.075 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:43:07.076 DEBUG:boto:Port: 8001 2023-05-31T20:43:07.076 DEBUG:boto:Params: {} 2023-05-31T20:43:07.077 DEBUG:boto:Token: None 2023-05-31T20:43:07.077 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:43:07 GMT /pklxkd-59/ 2023-05-31T20:43:07.077 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:mHjWQcyrqf5WjOocPTRbXTtj8YQ= 2023-05-31T20:43:07.078 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:43:07 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:mHjWQcyrqf5WjOocPTRbXTtj8YQ=', 'Content-Length': '0'} 2023-05-31T20:43:07.080 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c3f05e0fa3a07873-006477b15b-1c17-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:43:07 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:43:07.120 DEBUG:boto:b'pklxkd-59obj111000falseobj112023-05-31T20:39:26.427Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:43:07.121 DEBUG:boto:path=/pklxkd-59/ 2023-05-31T20:43:07.121 DEBUG:boto:auth_path=/pklxkd-59/ 2023-05-31T20:43:07.122 DEBUG:boto:path=/pklxkd-59/?prefix=obj11 2023-05-31T20:43:07.122 DEBUG:boto:auth_path=/pklxkd-59/?prefix=obj11 2023-05-31T20:43:07.122 DEBUG:boto:Method: GET 2023-05-31T20:43:07.123 DEBUG:boto:Path: /pklxkd-59/?prefix=obj11 2023-05-31T20:43:07.123 DEBUG:boto:Data: 2023-05-31T20:43:07.123 DEBUG:boto:Headers: {} 2023-05-31T20:43:07.124 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:43:07.124 DEBUG:boto:Port: 8000 2023-05-31T20:43:07.125 DEBUG:boto:Params: {} 2023-05-31T20:43:07.125 DEBUG:boto:Token: None 2023-05-31T20:43:07.125 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:43:07 GMT /pklxkd-59/ 2023-05-31T20:43:07.126 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:mHjWQcyrqf5WjOocPTRbXTtj8YQ= 2023-05-31T20:43:07.126 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:43:07 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:mHjWQcyrqf5WjOocPTRbXTtj8YQ=', 'Content-Length': '0'} 2023-05-31T20:43:07.128 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000cba6ea2ec818de52-006477b15b-1630-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:43:07 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:43:07.129 DEBUG:boto:b'pklxkd-59obj111000falseobj112023-05-31T20:39:26.427Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:43:07.129 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj22 2023-05-31T20:43:07.130 DEBUG:boto:path=/pklxkd-59/obj22 2023-05-31T20:43:07.130 DEBUG:boto:auth_path=/pklxkd-59/obj22 2023-05-31T20:43:07.131 DEBUG:boto:path=/pklxkd-59/obj22?versionId=null 2023-05-31T20:43:07.131 DEBUG:boto:auth_path=/pklxkd-59/obj22?versionId=null 2023-05-31T20:43:07.131 DEBUG:boto:Method: GET 2023-05-31T20:43:07.132 DEBUG:boto:Path: /pklxkd-59/obj22?versionId=null 2023-05-31T20:43:07.132 DEBUG:boto:Data: 2023-05-31T20:43:07.132 DEBUG:boto:Headers: {} 2023-05-31T20:43:07.133 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:43:07.133 DEBUG:boto:Port: 8001 2023-05-31T20:43:07.134 DEBUG:boto:Params: {} 2023-05-31T20:43:07.134 DEBUG:boto:Token: None 2023-05-31T20:43:07.134 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:43:07 GMT /pklxkd-59/obj22?versionId=null 2023-05-31T20:43:07.135 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:yNm0adLHW37mrHB7x0xwm3yOutc= 2023-05-31T20:43:07.135 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:43:07 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:yNm0adLHW37mrHB7x0xwm3yOutc=', 'Content-Length': '0'} 2023-05-31T20:43:07.137 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:39:26 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000389c997affcf0beb-006477b15b-1c17-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:43:07 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:43:07.138 DEBUG:boto:path=/pklxkd-59/obj22 2023-05-31T20:43:07.138 DEBUG:boto:auth_path=/pklxkd-59/obj22 2023-05-31T20:43:07.138 DEBUG:boto:path=/pklxkd-59/obj22?versionId=null 2023-05-31T20:43:07.139 DEBUG:boto:auth_path=/pklxkd-59/obj22?versionId=null 2023-05-31T20:43:07.139 DEBUG:boto:Method: GET 2023-05-31T20:43:07.139 DEBUG:boto:Path: /pklxkd-59/obj22?versionId=null 2023-05-31T20:43:07.140 DEBUG:boto:Data: 2023-05-31T20:43:07.140 DEBUG:boto:Headers: {} 2023-05-31T20:43:07.141 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:43:07.141 DEBUG:boto:Port: 8000 2023-05-31T20:43:07.141 DEBUG:boto:Params: {} 2023-05-31T20:43:07.142 DEBUG:boto:Token: None 2023-05-31T20:43:07.142 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:43:07 GMT /pklxkd-59/obj22?versionId=null 2023-05-31T20:43:07.142 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:yNm0adLHW37mrHB7x0xwm3yOutc= 2023-05-31T20:43:07.143 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:43:07 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:yNm0adLHW37mrHB7x0xwm3yOutc=', 'Content-Length': '0'} 2023-05-31T20:43:07.144 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:39:26 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000286009cad0c620be-006477b15b-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:43:07 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:43:07.188 DEBUG:boto:path=/pklxkd-59/obj22 2023-05-31T20:43:07.189 DEBUG:boto:auth_path=/pklxkd-59/obj22 2023-05-31T20:43:07.189 DEBUG:boto:path=/pklxkd-59/obj22?versionId=null 2023-05-31T20:43:07.189 DEBUG:boto:auth_path=/pklxkd-59/obj22?versionId=null 2023-05-31T20:43:07.190 DEBUG:boto:Method: HEAD 2023-05-31T20:43:07.190 DEBUG:boto:Path: /pklxkd-59/obj22?versionId=null 2023-05-31T20:43:07.191 DEBUG:boto:Data: 2023-05-31T20:43:07.191 DEBUG:boto:Headers: {} 2023-05-31T20:43:07.191 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:43:07.191 DEBUG:boto:Port: 8001 2023-05-31T20:43:07.192 DEBUG:boto:Params: {} 2023-05-31T20:43:07.192 DEBUG:boto:Token: None 2023-05-31T20:43:07.193 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:43:07 GMT /pklxkd-59/obj22?versionId=null 2023-05-31T20:43:07.193 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:C6MM/Y/PAs65q+qYxDOlnBjs1c4= 2023-05-31T20:43:07.194 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:43:07 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:C6MM/Y/PAs65q+qYxDOlnBjs1c4=', 'Content-Length': '0'} 2023-05-31T20:43:07.195 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:39:26 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000092dffda4138c9f9a-006477b15b-1c17-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:43:07 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:43:07.196 DEBUG:boto:path=/pklxkd-59/obj22 2023-05-31T20:43:07.196 DEBUG:boto:auth_path=/pklxkd-59/obj22 2023-05-31T20:43:07.196 DEBUG:boto:path=/pklxkd-59/obj22?versionId=null 2023-05-31T20:43:07.197 DEBUG:boto:auth_path=/pklxkd-59/obj22?versionId=null 2023-05-31T20:43:07.197 DEBUG:boto:Method: HEAD 2023-05-31T20:43:07.197 DEBUG:boto:Path: /pklxkd-59/obj22?versionId=null 2023-05-31T20:43:07.198 DEBUG:boto:Data: 2023-05-31T20:43:07.198 DEBUG:boto:Headers: {} 2023-05-31T20:43:07.198 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:43:07.199 DEBUG:boto:Port: 8000 2023-05-31T20:43:07.199 DEBUG:boto:Params: {} 2023-05-31T20:43:07.199 DEBUG:boto:Token: None 2023-05-31T20:43:07.200 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:43:07 GMT /pklxkd-59/obj22?versionId=null 2023-05-31T20:43:07.200 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:C6MM/Y/PAs65q+qYxDOlnBjs1c4= 2023-05-31T20:43:07.201 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:43:07 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:C6MM/Y/PAs65q+qYxDOlnBjs1c4=', 'Content-Length': '0'} 2023-05-31T20:43:07.202 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:39:26 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ce12e865be6681b4-006477b15b-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:43:07 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:43:07.203 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj22 2023-05-31T20:43:07.203 DEBUG:boto:path=/pklxkd-59/obj22 2023-05-31T20:43:07.203 DEBUG:boto:auth_path=/pklxkd-59/obj22 2023-05-31T20:43:07.204 DEBUG:boto:path=/pklxkd-59/obj22?versionId=null 2023-05-31T20:43:07.204 DEBUG:boto:auth_path=/pklxkd-59/obj22?versionId=null 2023-05-31T20:43:07.205 DEBUG:boto:Method: GET 2023-05-31T20:43:07.205 DEBUG:boto:Path: /pklxkd-59/obj22?versionId=null 2023-05-31T20:43:07.205 DEBUG:boto:Data: 2023-05-31T20:43:07.206 DEBUG:boto:Headers: {} 2023-05-31T20:43:07.206 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:43:07.206 DEBUG:boto:Port: 8001 2023-05-31T20:43:07.207 DEBUG:boto:Params: {} 2023-05-31T20:43:07.207 DEBUG:boto:Token: None 2023-05-31T20:43:07.207 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:43:07 GMT /pklxkd-59/obj22?versionId=null 2023-05-31T20:43:07.208 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:yNm0adLHW37mrHB7x0xwm3yOutc= 2023-05-31T20:43:07.208 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:43:07 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:yNm0adLHW37mrHB7x0xwm3yOutc=', 'Content-Length': '0'} 2023-05-31T20:43:07.210 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:39:26 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001e359e47ab7ef973-006477b15b-1c17-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:43:07 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:43:07.211 DEBUG:boto:path=/pklxkd-59/obj22 2023-05-31T20:43:07.211 DEBUG:boto:auth_path=/pklxkd-59/obj22 2023-05-31T20:43:07.212 DEBUG:boto:path=/pklxkd-59/obj22?versionId=null 2023-05-31T20:43:07.212 DEBUG:boto:auth_path=/pklxkd-59/obj22?versionId=null 2023-05-31T20:43:07.212 DEBUG:boto:Method: GET 2023-05-31T20:43:07.213 DEBUG:boto:Path: /pklxkd-59/obj22?versionId=null 2023-05-31T20:43:07.213 DEBUG:boto:Data: 2023-05-31T20:43:07.213 DEBUG:boto:Headers: {} 2023-05-31T20:43:07.214 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:43:07.214 DEBUG:boto:Port: 8000 2023-05-31T20:43:07.214 DEBUG:boto:Params: {} 2023-05-31T20:43:07.215 DEBUG:boto:Token: None 2023-05-31T20:43:07.215 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:43:07 GMT /pklxkd-59/obj22?versionId=null 2023-05-31T20:43:07.216 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:yNm0adLHW37mrHB7x0xwm3yOutc= 2023-05-31T20:43:07.216 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:43:07 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:yNm0adLHW37mrHB7x0xwm3yOutc=', 'Content-Length': '0'} 2023-05-31T20:43:07.218 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:39:26 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005496696feb7027bc-006477b15b-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:43:07 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:43:07.218 DEBUG:boto:path=/pklxkd-59/ 2023-05-31T20:43:07.219 DEBUG:boto:auth_path=/pklxkd-59/ 2023-05-31T20:43:07.219 DEBUG:boto:path=/pklxkd-59/?prefix=obj22 2023-05-31T20:43:07.219 DEBUG:boto:auth_path=/pklxkd-59/?prefix=obj22 2023-05-31T20:43:07.220 DEBUG:boto:Method: GET 2023-05-31T20:43:07.220 DEBUG:boto:Path: /pklxkd-59/?prefix=obj22 2023-05-31T20:43:07.221 DEBUG:boto:Data: 2023-05-31T20:43:07.221 DEBUG:boto:Headers: {} 2023-05-31T20:43:07.221 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:43:07.222 DEBUG:boto:Port: 8001 2023-05-31T20:43:07.222 DEBUG:boto:Params: {} 2023-05-31T20:43:07.222 DEBUG:boto:Token: None 2023-05-31T20:43:07.223 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:43:07 GMT /pklxkd-59/ 2023-05-31T20:43:07.223 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:mHjWQcyrqf5WjOocPTRbXTtj8YQ= 2023-05-31T20:43:07.224 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:43:07 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:mHjWQcyrqf5WjOocPTRbXTtj8YQ=', 'Content-Length': '0'} 2023-05-31T20:43:07.225 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d98a81f5428da118-006477b15b-1c17-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:43:07 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:43:07.226 DEBUG:boto:b'pklxkd-59obj221000falseobj222023-05-31T20:39:26.473Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:43:07.227 DEBUG:boto:path=/pklxkd-59/ 2023-05-31T20:43:07.227 DEBUG:boto:auth_path=/pklxkd-59/ 2023-05-31T20:43:07.227 DEBUG:boto:path=/pklxkd-59/?prefix=obj22 2023-05-31T20:43:07.228 DEBUG:boto:auth_path=/pklxkd-59/?prefix=obj22 2023-05-31T20:43:07.228 DEBUG:boto:Method: GET 2023-05-31T20:43:07.229 DEBUG:boto:Path: /pklxkd-59/?prefix=obj22 2023-05-31T20:43:07.229 DEBUG:boto:Data: 2023-05-31T20:43:07.229 DEBUG:boto:Headers: {} 2023-05-31T20:43:07.230 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:43:07.230 DEBUG:boto:Port: 8000 2023-05-31T20:43:07.230 DEBUG:boto:Params: {} 2023-05-31T20:43:07.231 DEBUG:boto:Token: None 2023-05-31T20:43:07.231 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:43:07 GMT /pklxkd-59/ 2023-05-31T20:43:07.232 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:mHjWQcyrqf5WjOocPTRbXTtj8YQ= 2023-05-31T20:43:07.232 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:43:07 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:mHjWQcyrqf5WjOocPTRbXTtj8YQ=', 'Content-Length': '0'} 2023-05-31T20:43:07.234 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000033995ba7643bed24-006477b15b-1630-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:43:07 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:43:07.234 DEBUG:boto:b'pklxkd-59obj221000falseobj222023-05-31T20:39:26.473Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:43:07.235 DEBUG:boto:path=/pklxkd-59/obj22 2023-05-31T20:43:07.235 DEBUG:boto:auth_path=/pklxkd-59/obj22 2023-05-31T20:43:07.236 DEBUG:boto:Method: HEAD 2023-05-31T20:43:07.236 DEBUG:boto:Path: /pklxkd-59/obj22 2023-05-31T20:43:07.236 DEBUG:boto:Data: 2023-05-31T20:43:07.237 DEBUG:boto:Headers: {} 2023-05-31T20:43:07.237 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:43:07.237 DEBUG:boto:Port: 8001 2023-05-31T20:43:07.238 DEBUG:boto:Params: {} 2023-05-31T20:43:07.238 DEBUG:boto:Token: None 2023-05-31T20:43:07.238 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:43:07 GMT /pklxkd-59/obj22 2023-05-31T20:43:07.239 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:6ouNDYPvwJwsYNjH9ndyZcXBORU= 2023-05-31T20:43:07.239 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:43:07 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:6ouNDYPvwJwsYNjH9ndyZcXBORU=', 'Content-Length': '0'} 2023-05-31T20:43:07.241 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:39:26 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b0d232ed2cd4c211-006477b15b-1c17-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:43:07 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:43:07.241 DEBUG:boto:path=/pklxkd-59/obj22 2023-05-31T20:43:07.242 DEBUG:boto:auth_path=/pklxkd-59/obj22 2023-05-31T20:43:07.242 DEBUG:boto:Method: HEAD 2023-05-31T20:43:07.242 DEBUG:boto:Path: /pklxkd-59/obj22 2023-05-31T20:43:07.243 DEBUG:boto:Data: 2023-05-31T20:43:07.243 DEBUG:boto:Headers: {} 2023-05-31T20:43:07.243 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:43:07.244 DEBUG:boto:Port: 8000 2023-05-31T20:43:07.244 DEBUG:boto:Params: {} 2023-05-31T20:43:07.244 DEBUG:boto:Token: None 2023-05-31T20:43:07.245 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:43:07 GMT /pklxkd-59/obj22 2023-05-31T20:43:07.245 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:6ouNDYPvwJwsYNjH9ndyZcXBORU= 2023-05-31T20:43:07.245 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:43:07 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:6ouNDYPvwJwsYNjH9ndyZcXBORU=', 'Content-Length': '0'} 2023-05-31T20:43:07.247 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:39:26 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007d91c8f3e5fabb93-006477b15b-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:43:07 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:43:07.247 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj22 2023-05-31T20:43:07.248 DEBUG:boto:path=/pklxkd-59/obj22 2023-05-31T20:43:07.248 DEBUG:boto:auth_path=/pklxkd-59/obj22 2023-05-31T20:43:07.249 DEBUG:boto:Method: GET 2023-05-31T20:43:07.249 DEBUG:boto:Path: /pklxkd-59/obj22 2023-05-31T20:43:07.249 DEBUG:boto:Data: 2023-05-31T20:43:07.250 DEBUG:boto:Headers: {} 2023-05-31T20:43:07.250 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:43:07.250 DEBUG:boto:Port: 8001 2023-05-31T20:43:07.251 DEBUG:boto:Params: {} 2023-05-31T20:43:07.252 DEBUG:boto:Token: None 2023-05-31T20:43:07.252 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:43:07 GMT /pklxkd-59/obj22 2023-05-31T20:43:07.252 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:dqFPZa5vntDFHPH2ZcSk8BFEzkU= 2023-05-31T20:43:07.253 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:43:07 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:dqFPZa5vntDFHPH2ZcSk8BFEzkU=', 'Content-Length': '0'} 2023-05-31T20:43:07.254 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:39:26 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002d32b44382e757ab-006477b15b-1c17-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:43:07 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:43:07.255 DEBUG:boto:path=/pklxkd-59/obj22 2023-05-31T20:43:07.255 DEBUG:boto:auth_path=/pklxkd-59/obj22 2023-05-31T20:43:07.256 DEBUG:boto:Method: GET 2023-05-31T20:43:07.256 DEBUG:boto:Path: /pklxkd-59/obj22 2023-05-31T20:43:07.257 DEBUG:boto:Data: 2023-05-31T20:43:07.257 DEBUG:boto:Headers: {} 2023-05-31T20:43:07.258 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:43:07.258 DEBUG:boto:Port: 8000 2023-05-31T20:43:07.258 DEBUG:boto:Params: {} 2023-05-31T20:43:07.259 DEBUG:boto:Token: None 2023-05-31T20:43:07.259 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:43:07 GMT /pklxkd-59/obj22 2023-05-31T20:43:07.260 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:dqFPZa5vntDFHPH2ZcSk8BFEzkU= 2023-05-31T20:43:07.260 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:43:07 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:dqFPZa5vntDFHPH2ZcSk8BFEzkU=', 'Content-Length': '0'} 2023-05-31T20:43:07.262 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:39:26 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004a2d33c89291e423-006477b15b-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:43:07 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:43:07.262 DEBUG:boto:path=/pklxkd-59/ 2023-05-31T20:43:07.263 DEBUG:boto:auth_path=/pklxkd-59/ 2023-05-31T20:43:07.263 DEBUG:boto:path=/pklxkd-59/?prefix=obj22 2023-05-31T20:43:07.264 DEBUG:boto:auth_path=/pklxkd-59/?prefix=obj22 2023-05-31T20:43:07.264 DEBUG:boto:Method: GET 2023-05-31T20:43:07.264 DEBUG:boto:Path: /pklxkd-59/?prefix=obj22 2023-05-31T20:43:07.265 DEBUG:boto:Data: 2023-05-31T20:43:07.265 DEBUG:boto:Headers: {} 2023-05-31T20:43:07.266 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:43:07.266 DEBUG:boto:Port: 8001 2023-05-31T20:43:07.266 DEBUG:boto:Params: {} 2023-05-31T20:43:07.267 DEBUG:boto:Token: None 2023-05-31T20:43:07.267 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:43:07 GMT /pklxkd-59/ 2023-05-31T20:43:07.268 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:mHjWQcyrqf5WjOocPTRbXTtj8YQ= 2023-05-31T20:43:07.268 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:43:07 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:mHjWQcyrqf5WjOocPTRbXTtj8YQ=', 'Content-Length': '0'} 2023-05-31T20:43:07.270 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000384b4bf94da55fa9-006477b15b-1c17-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:43:07 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:43:07.271 DEBUG:boto:b'pklxkd-59obj221000falseobj222023-05-31T20:39:26.473Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:43:07.271 DEBUG:boto:path=/pklxkd-59/ 2023-05-31T20:43:07.272 DEBUG:boto:auth_path=/pklxkd-59/ 2023-05-31T20:43:07.272 DEBUG:boto:path=/pklxkd-59/?prefix=obj22 2023-05-31T20:43:07.273 DEBUG:boto:auth_path=/pklxkd-59/?prefix=obj22 2023-05-31T20:43:07.273 DEBUG:boto:Method: GET 2023-05-31T20:43:07.273 DEBUG:boto:Path: /pklxkd-59/?prefix=obj22 2023-05-31T20:43:07.274 DEBUG:boto:Data: 2023-05-31T20:43:07.274 DEBUG:boto:Headers: {} 2023-05-31T20:43:07.274 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:43:07.275 DEBUG:boto:Port: 8000 2023-05-31T20:43:07.275 DEBUG:boto:Params: {} 2023-05-31T20:43:07.275 DEBUG:boto:Token: None 2023-05-31T20:43:07.276 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:43:07 GMT /pklxkd-59/ 2023-05-31T20:43:07.276 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:mHjWQcyrqf5WjOocPTRbXTtj8YQ= 2023-05-31T20:43:07.277 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:43:07 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:mHjWQcyrqf5WjOocPTRbXTtj8YQ=', 'Content-Length': '0'} 2023-05-31T20:43:07.279 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000001ee4c81180832ae5-006477b15b-1630-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:43:07 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:43:07.279 DEBUG:boto:b'pklxkd-59obj221000falseobj222023-05-31T20:39:26.473Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:43:07.280 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj33 2023-05-31T20:43:07.280 DEBUG:boto:path=/pklxkd-59/obj33 2023-05-31T20:43:07.281 DEBUG:boto:auth_path=/pklxkd-59/obj33 2023-05-31T20:43:07.281 DEBUG:boto:path=/pklxkd-59/obj33?versionId=null 2023-05-31T20:43:07.281 DEBUG:boto:auth_path=/pklxkd-59/obj33?versionId=null 2023-05-31T20:43:07.282 DEBUG:boto:Method: GET 2023-05-31T20:43:07.282 DEBUG:boto:Path: /pklxkd-59/obj33?versionId=null 2023-05-31T20:43:07.282 DEBUG:boto:Data: 2023-05-31T20:43:07.283 DEBUG:boto:Headers: {} 2023-05-31T20:43:07.283 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:43:07.284 DEBUG:boto:Port: 8001 2023-05-31T20:43:07.284 DEBUG:boto:Params: {} 2023-05-31T20:43:07.284 DEBUG:boto:Token: None 2023-05-31T20:43:07.285 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:43:07 GMT /pklxkd-59/obj33?versionId=null 2023-05-31T20:43:07.285 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:/7SwH3kb91zXWC430dl0a/bSuAc= 2023-05-31T20:43:07.286 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:43:07 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:/7SwH3kb91zXWC430dl0a/bSuAc=', 'Content-Length': '0'} 2023-05-31T20:43:07.287 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:39:26 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b7858664e06151a1-006477b15b-1c17-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:43:07 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:43:07.288 DEBUG:boto:path=/pklxkd-59/obj33 2023-05-31T20:43:07.288 DEBUG:boto:auth_path=/pklxkd-59/obj33 2023-05-31T20:43:07.289 DEBUG:boto:path=/pklxkd-59/obj33?versionId=null 2023-05-31T20:43:07.289 DEBUG:boto:auth_path=/pklxkd-59/obj33?versionId=null 2023-05-31T20:43:07.289 DEBUG:boto:Method: GET 2023-05-31T20:43:07.290 DEBUG:boto:Path: /pklxkd-59/obj33?versionId=null 2023-05-31T20:43:07.290 DEBUG:boto:Data: 2023-05-31T20:43:07.290 DEBUG:boto:Headers: {} 2023-05-31T20:43:07.291 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:43:07.291 DEBUG:boto:Port: 8000 2023-05-31T20:43:07.291 DEBUG:boto:Params: {} 2023-05-31T20:43:07.292 DEBUG:boto:Token: None 2023-05-31T20:43:07.292 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:43:07 GMT /pklxkd-59/obj33?versionId=null 2023-05-31T20:43:07.293 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:/7SwH3kb91zXWC430dl0a/bSuAc= 2023-05-31T20:43:07.293 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:43:07 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:/7SwH3kb91zXWC430dl0a/bSuAc=', 'Content-Length': '0'} 2023-05-31T20:43:07.295 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:39:26 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a2582e500d3306d6-006477b15b-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:43:07 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:43:07.295 DEBUG:boto:path=/pklxkd-59/obj33 2023-05-31T20:43:07.296 DEBUG:boto:auth_path=/pklxkd-59/obj33 2023-05-31T20:43:07.296 DEBUG:boto:path=/pklxkd-59/obj33?versionId=null 2023-05-31T20:43:07.296 DEBUG:boto:auth_path=/pklxkd-59/obj33?versionId=null 2023-05-31T20:43:07.297 DEBUG:boto:Method: HEAD 2023-05-31T20:43:07.297 DEBUG:boto:Path: /pklxkd-59/obj33?versionId=null 2023-05-31T20:43:07.297 DEBUG:boto:Data: 2023-05-31T20:43:07.298 DEBUG:boto:Headers: {} 2023-05-31T20:43:07.298 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:43:07.298 DEBUG:boto:Port: 8001 2023-05-31T20:43:07.299 DEBUG:boto:Params: {} 2023-05-31T20:43:07.299 DEBUG:boto:Token: None 2023-05-31T20:43:07.299 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:43:07 GMT /pklxkd-59/obj33?versionId=null 2023-05-31T20:43:07.300 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:ITjNk+JuEhqWDqkHwyiri79p6lI= 2023-05-31T20:43:07.300 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:43:07 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:ITjNk+JuEhqWDqkHwyiri79p6lI=', 'Content-Length': '0'} 2023-05-31T20:43:07.302 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:39:26 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000008c0e8e63ef14f1c-006477b15b-1c17-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:43:07 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:43:07.302 DEBUG:boto:path=/pklxkd-59/obj33 2023-05-31T20:43:07.302 DEBUG:boto:auth_path=/pklxkd-59/obj33 2023-05-31T20:43:07.303 DEBUG:boto:path=/pklxkd-59/obj33?versionId=null 2023-05-31T20:43:07.303 DEBUG:boto:auth_path=/pklxkd-59/obj33?versionId=null 2023-05-31T20:43:07.303 DEBUG:boto:Method: HEAD 2023-05-31T20:43:07.304 DEBUG:boto:Path: /pklxkd-59/obj33?versionId=null 2023-05-31T20:43:07.304 DEBUG:boto:Data: 2023-05-31T20:43:07.305 DEBUG:boto:Headers: {} 2023-05-31T20:43:07.305 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:43:07.305 DEBUG:boto:Port: 8000 2023-05-31T20:43:07.306 DEBUG:boto:Params: {} 2023-05-31T20:43:07.306 DEBUG:boto:Token: None 2023-05-31T20:43:07.306 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:43:07 GMT /pklxkd-59/obj33?versionId=null 2023-05-31T20:43:07.307 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:ITjNk+JuEhqWDqkHwyiri79p6lI= 2023-05-31T20:43:07.307 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:43:07 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:ITjNk+JuEhqWDqkHwyiri79p6lI=', 'Content-Length': '0'} 2023-05-31T20:43:07.309 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:39:26 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004ebc143880b81930-006477b15b-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:43:07 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:43:07.309 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj33 2023-05-31T20:43:07.310 DEBUG:boto:path=/pklxkd-59/obj33 2023-05-31T20:43:07.310 DEBUG:boto:auth_path=/pklxkd-59/obj33 2023-05-31T20:43:07.310 DEBUG:boto:path=/pklxkd-59/obj33?versionId=null 2023-05-31T20:43:07.311 DEBUG:boto:auth_path=/pklxkd-59/obj33?versionId=null 2023-05-31T20:43:07.311 DEBUG:boto:Method: GET 2023-05-31T20:43:07.311 DEBUG:boto:Path: /pklxkd-59/obj33?versionId=null 2023-05-31T20:43:07.312 DEBUG:boto:Data: 2023-05-31T20:43:07.312 DEBUG:boto:Headers: {} 2023-05-31T20:43:07.312 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:43:07.313 DEBUG:boto:Port: 8001 2023-05-31T20:43:07.313 DEBUG:boto:Params: {} 2023-05-31T20:43:07.314 DEBUG:boto:Token: None 2023-05-31T20:43:07.314 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:43:07 GMT /pklxkd-59/obj33?versionId=null 2023-05-31T20:43:07.314 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:/7SwH3kb91zXWC430dl0a/bSuAc= 2023-05-31T20:43:07.315 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:43:07 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:/7SwH3kb91zXWC430dl0a/bSuAc=', 'Content-Length': '0'} 2023-05-31T20:43:07.316 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:39:26 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009d8dd2aa1dce91f0-006477b15b-1c17-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:43:07 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:43:07.317 DEBUG:boto:path=/pklxkd-59/obj33 2023-05-31T20:43:07.317 DEBUG:boto:auth_path=/pklxkd-59/obj33 2023-05-31T20:43:07.318 DEBUG:boto:path=/pklxkd-59/obj33?versionId=null 2023-05-31T20:43:07.318 DEBUG:boto:auth_path=/pklxkd-59/obj33?versionId=null 2023-05-31T20:43:07.318 DEBUG:boto:Method: GET 2023-05-31T20:43:07.319 DEBUG:boto:Path: /pklxkd-59/obj33?versionId=null 2023-05-31T20:43:07.319 DEBUG:boto:Data: 2023-05-31T20:43:07.319 DEBUG:boto:Headers: {} 2023-05-31T20:43:07.320 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:43:07.320 DEBUG:boto:Port: 8000 2023-05-31T20:43:07.320 DEBUG:boto:Params: {} 2023-05-31T20:43:07.321 DEBUG:boto:Token: None 2023-05-31T20:43:07.321 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:43:07 GMT /pklxkd-59/obj33?versionId=null 2023-05-31T20:43:07.322 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:/7SwH3kb91zXWC430dl0a/bSuAc= 2023-05-31T20:43:07.322 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:43:07 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:/7SwH3kb91zXWC430dl0a/bSuAc=', 'Content-Length': '0'} 2023-05-31T20:43:07.324 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:39:26 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006d2e7349546da7fb-006477b15b-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:43:07 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:43:07.324 DEBUG:boto:path=/pklxkd-59/ 2023-05-31T20:43:07.325 DEBUG:boto:auth_path=/pklxkd-59/ 2023-05-31T20:43:07.325 DEBUG:boto:path=/pklxkd-59/?prefix=obj33 2023-05-31T20:43:07.325 DEBUG:boto:auth_path=/pklxkd-59/?prefix=obj33 2023-05-31T20:43:07.326 DEBUG:boto:Method: GET 2023-05-31T20:43:07.326 DEBUG:boto:Path: /pklxkd-59/?prefix=obj33 2023-05-31T20:43:07.326 DEBUG:boto:Data: 2023-05-31T20:43:07.327 DEBUG:boto:Headers: {} 2023-05-31T20:43:07.327 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:43:07.327 DEBUG:boto:Port: 8001 2023-05-31T20:43:07.328 DEBUG:boto:Params: {} 2023-05-31T20:43:07.328 DEBUG:boto:Token: None 2023-05-31T20:43:07.329 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:43:07 GMT /pklxkd-59/ 2023-05-31T20:43:07.329 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:mHjWQcyrqf5WjOocPTRbXTtj8YQ= 2023-05-31T20:43:07.330 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:43:07 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:mHjWQcyrqf5WjOocPTRbXTtj8YQ=', 'Content-Length': '0'} 2023-05-31T20:43:07.332 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000055b39619abba1005-006477b15b-1c17-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:43:07 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:43:07.372 DEBUG:boto:b'pklxkd-59obj331000falseobj332023-05-31T20:39:26.534Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:43:07.373 DEBUG:boto:path=/pklxkd-59/ 2023-05-31T20:43:07.373 DEBUG:boto:auth_path=/pklxkd-59/ 2023-05-31T20:43:07.374 DEBUG:boto:path=/pklxkd-59/?prefix=obj33 2023-05-31T20:43:07.374 DEBUG:boto:auth_path=/pklxkd-59/?prefix=obj33 2023-05-31T20:43:07.374 DEBUG:boto:Method: GET 2023-05-31T20:43:07.375 DEBUG:boto:Path: /pklxkd-59/?prefix=obj33 2023-05-31T20:43:07.375 DEBUG:boto:Data: 2023-05-31T20:43:07.375 DEBUG:boto:Headers: {} 2023-05-31T20:43:07.376 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:43:07.376 DEBUG:boto:Port: 8000 2023-05-31T20:43:07.377 DEBUG:boto:Params: {} 2023-05-31T20:43:07.377 DEBUG:boto:Token: None 2023-05-31T20:43:07.377 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:43:07 GMT /pklxkd-59/ 2023-05-31T20:43:07.378 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:mHjWQcyrqf5WjOocPTRbXTtj8YQ= 2023-05-31T20:43:07.378 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:43:07 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:mHjWQcyrqf5WjOocPTRbXTtj8YQ=', 'Content-Length': '0'} 2023-05-31T20:43:07.380 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d9c0c83f18358f0e-006477b15b-1630-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:43:07 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:43:07.381 DEBUG:boto:b'pklxkd-59obj331000falseobj332023-05-31T20:39:26.534Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:43:07.382 DEBUG:boto:path=/pklxkd-59/obj33 2023-05-31T20:43:07.382 DEBUG:boto:auth_path=/pklxkd-59/obj33 2023-05-31T20:43:07.382 DEBUG:boto:Method: HEAD 2023-05-31T20:43:07.383 DEBUG:boto:Path: /pklxkd-59/obj33 2023-05-31T20:43:07.383 DEBUG:boto:Data: 2023-05-31T20:43:07.383 DEBUG:boto:Headers: {} 2023-05-31T20:43:07.384 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:43:07.384 DEBUG:boto:Port: 8001 2023-05-31T20:43:07.384 DEBUG:boto:Params: {} 2023-05-31T20:43:07.385 DEBUG:boto:Token: None 2023-05-31T20:43:07.385 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:43:07 GMT /pklxkd-59/obj33 2023-05-31T20:43:07.385 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:6TSJ8GY2fXgxhodTxpEL1/WY4nM= 2023-05-31T20:43:07.386 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:43:07 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:6TSJ8GY2fXgxhodTxpEL1/WY4nM=', 'Content-Length': '0'} 2023-05-31T20:43:07.387 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:39:26 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000083cfccf2eff3a857-006477b15b-1c17-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:43:07 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:43:07.388 DEBUG:boto:path=/pklxkd-59/obj33 2023-05-31T20:43:07.388 DEBUG:boto:auth_path=/pklxkd-59/obj33 2023-05-31T20:43:07.388 DEBUG:boto:Method: HEAD 2023-05-31T20:43:07.389 DEBUG:boto:Path: /pklxkd-59/obj33 2023-05-31T20:43:07.389 DEBUG:boto:Data: 2023-05-31T20:43:07.389 DEBUG:boto:Headers: {} 2023-05-31T20:43:07.390 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:43:07.390 DEBUG:boto:Port: 8000 2023-05-31T20:43:07.390 DEBUG:boto:Params: {} 2023-05-31T20:43:07.391 DEBUG:boto:Token: None 2023-05-31T20:43:07.391 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:43:07 GMT /pklxkd-59/obj33 2023-05-31T20:43:07.392 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:6TSJ8GY2fXgxhodTxpEL1/WY4nM= 2023-05-31T20:43:07.392 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:43:07 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:6TSJ8GY2fXgxhodTxpEL1/WY4nM=', 'Content-Length': '0'} 2023-05-31T20:43:07.393 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:39:26 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000cb68c1ec678774ff-006477b15b-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:43:07 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:43:07.394 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj33 2023-05-31T20:43:07.394 DEBUG:boto:path=/pklxkd-59/obj33 2023-05-31T20:43:07.395 DEBUG:boto:auth_path=/pklxkd-59/obj33 2023-05-31T20:43:07.395 DEBUG:boto:Method: GET 2023-05-31T20:43:07.395 DEBUG:boto:Path: /pklxkd-59/obj33 2023-05-31T20:43:07.396 DEBUG:boto:Data: 2023-05-31T20:43:07.396 DEBUG:boto:Headers: {} 2023-05-31T20:43:07.397 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:43:07.397 DEBUG:boto:Port: 8001 2023-05-31T20:43:07.397 DEBUG:boto:Params: {} 2023-05-31T20:43:07.398 DEBUG:boto:Token: None 2023-05-31T20:43:07.398 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:43:07 GMT /pklxkd-59/obj33 2023-05-31T20:43:07.398 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:0lXxnl3I0RhKJ8I/ZjEDFYYUBBw= 2023-05-31T20:43:07.399 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:43:07 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:0lXxnl3I0RhKJ8I/ZjEDFYYUBBw=', 'Content-Length': '0'} 2023-05-31T20:43:07.400 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:39:26 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006d8d282fe65fe67d-006477b15b-1c17-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:43:07 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:43:07.401 DEBUG:boto:path=/pklxkd-59/obj33 2023-05-31T20:43:07.402 DEBUG:boto:auth_path=/pklxkd-59/obj33 2023-05-31T20:43:07.402 DEBUG:boto:Method: GET 2023-05-31T20:43:07.402 DEBUG:boto:Path: /pklxkd-59/obj33 2023-05-31T20:43:07.403 DEBUG:boto:Data: 2023-05-31T20:43:07.403 DEBUG:boto:Headers: {} 2023-05-31T20:43:07.403 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:43:07.404 DEBUG:boto:Port: 8000 2023-05-31T20:43:07.404 DEBUG:boto:Params: {} 2023-05-31T20:43:07.404 DEBUG:boto:Token: None 2023-05-31T20:43:07.405 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:43:07 GMT /pklxkd-59/obj33 2023-05-31T20:43:07.405 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:0lXxnl3I0RhKJ8I/ZjEDFYYUBBw= 2023-05-31T20:43:07.406 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:43:07 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:0lXxnl3I0RhKJ8I/ZjEDFYYUBBw=', 'Content-Length': '0'} 2023-05-31T20:43:07.407 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:39:26 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000687d806e6d015edd-006477b15b-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:43:07 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:43:07.452 DEBUG:boto:path=/pklxkd-59/ 2023-05-31T20:43:07.453 DEBUG:boto:auth_path=/pklxkd-59/ 2023-05-31T20:43:07.453 DEBUG:boto:path=/pklxkd-59/?prefix=obj33 2023-05-31T20:43:07.453 DEBUG:boto:auth_path=/pklxkd-59/?prefix=obj33 2023-05-31T20:43:07.454 DEBUG:boto:Method: GET 2023-05-31T20:43:07.454 DEBUG:boto:Path: /pklxkd-59/?prefix=obj33 2023-05-31T20:43:07.454 DEBUG:boto:Data: 2023-05-31T20:43:07.455 DEBUG:boto:Headers: {} 2023-05-31T20:43:07.455 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:43:07.456 DEBUG:boto:Port: 8001 2023-05-31T20:43:07.456 DEBUG:boto:Params: {} 2023-05-31T20:43:07.456 DEBUG:boto:Token: None 2023-05-31T20:43:07.457 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:43:07 GMT /pklxkd-59/ 2023-05-31T20:43:07.457 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:mHjWQcyrqf5WjOocPTRbXTtj8YQ= 2023-05-31T20:43:07.458 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:43:07 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:mHjWQcyrqf5WjOocPTRbXTtj8YQ=', 'Content-Length': '0'} 2023-05-31T20:43:07.460 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000003827ec55297ace1d-006477b15b-1c17-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:43:07 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:43:07.460 DEBUG:boto:b'pklxkd-59obj331000falseobj332023-05-31T20:39:26.534Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:43:07.461 DEBUG:boto:path=/pklxkd-59/ 2023-05-31T20:43:07.461 DEBUG:boto:auth_path=/pklxkd-59/ 2023-05-31T20:43:07.462 DEBUG:boto:path=/pklxkd-59/?prefix=obj33 2023-05-31T20:43:07.462 DEBUG:boto:auth_path=/pklxkd-59/?prefix=obj33 2023-05-31T20:43:07.462 DEBUG:boto:Method: GET 2023-05-31T20:43:07.463 DEBUG:boto:Path: /pklxkd-59/?prefix=obj33 2023-05-31T20:43:07.463 DEBUG:boto:Data: 2023-05-31T20:43:07.463 DEBUG:boto:Headers: {} 2023-05-31T20:43:07.464 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:43:07.464 DEBUG:boto:Port: 8000 2023-05-31T20:43:07.465 DEBUG:boto:Params: {} 2023-05-31T20:43:07.465 DEBUG:boto:Token: None 2023-05-31T20:43:07.465 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:43:07 GMT /pklxkd-59/ 2023-05-31T20:43:07.466 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:mHjWQcyrqf5WjOocPTRbXTtj8YQ= 2023-05-31T20:43:07.466 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:43:07 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:mHjWQcyrqf5WjOocPTRbXTtj8YQ=', 'Content-Length': '0'} 2023-05-31T20:43:07.468 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000601a9e0fe69549cc-006477b15b-1630-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:43:07 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:43:07.469 DEBUG:boto:b'pklxkd-59obj331000falseobj332023-05-31T20:39:26.534Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:43:07.469 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj44 2023-05-31T20:43:07.470 DEBUG:boto:path=/pklxkd-59/obj44 2023-05-31T20:43:07.470 DEBUG:boto:auth_path=/pklxkd-59/obj44 2023-05-31T20:43:07.470 DEBUG:boto:path=/pklxkd-59/obj44?versionId=null 2023-05-31T20:43:07.471 DEBUG:boto:auth_path=/pklxkd-59/obj44?versionId=null 2023-05-31T20:43:07.471 DEBUG:boto:Method: GET 2023-05-31T20:43:07.471 DEBUG:boto:Path: /pklxkd-59/obj44?versionId=null 2023-05-31T20:43:07.472 DEBUG:boto:Data: 2023-05-31T20:43:07.472 DEBUG:boto:Headers: {} 2023-05-31T20:43:07.472 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:43:07.473 DEBUG:boto:Port: 8001 2023-05-31T20:43:07.473 DEBUG:boto:Params: {} 2023-05-31T20:43:07.474 DEBUG:boto:Token: None 2023-05-31T20:43:07.474 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:43:07 GMT /pklxkd-59/obj44?versionId=null 2023-05-31T20:43:07.474 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:3AcsMujlp3tmYoYeBSX9vkvxbQg= 2023-05-31T20:43:07.475 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:43:07 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:3AcsMujlp3tmYoYeBSX9vkvxbQg=', 'Content-Length': '0'} 2023-05-31T20:43:07.476 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:39:26 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002d52a769659012c8-006477b15b-1c17-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:43:07 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:43:07.477 DEBUG:boto:path=/pklxkd-59/obj44 2023-05-31T20:43:07.477 DEBUG:boto:auth_path=/pklxkd-59/obj44 2023-05-31T20:43:07.478 DEBUG:boto:path=/pklxkd-59/obj44?versionId=null 2023-05-31T20:43:07.478 DEBUG:boto:auth_path=/pklxkd-59/obj44?versionId=null 2023-05-31T20:43:07.478 DEBUG:boto:Method: GET 2023-05-31T20:43:07.479 DEBUG:boto:Path: /pklxkd-59/obj44?versionId=null 2023-05-31T20:43:07.479 DEBUG:boto:Data: 2023-05-31T20:43:07.480 DEBUG:boto:Headers: {} 2023-05-31T20:43:07.480 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:43:07.480 DEBUG:boto:Port: 8000 2023-05-31T20:43:07.481 DEBUG:boto:Params: {} 2023-05-31T20:43:07.481 DEBUG:boto:Token: None 2023-05-31T20:43:07.481 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:43:07 GMT /pklxkd-59/obj44?versionId=null 2023-05-31T20:43:07.482 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:3AcsMujlp3tmYoYeBSX9vkvxbQg= 2023-05-31T20:43:07.482 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:43:07 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:3AcsMujlp3tmYoYeBSX9vkvxbQg=', 'Content-Length': '0'} 2023-05-31T20:43:07.484 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:39:26 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006575c489179e9aea-006477b15b-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:43:07 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:43:07.484 DEBUG:boto:path=/pklxkd-59/obj44 2023-05-31T20:43:07.485 DEBUG:boto:auth_path=/pklxkd-59/obj44 2023-05-31T20:43:07.485 DEBUG:boto:path=/pklxkd-59/obj44?versionId=null 2023-05-31T20:43:07.486 DEBUG:boto:auth_path=/pklxkd-59/obj44?versionId=null 2023-05-31T20:43:07.486 DEBUG:boto:Method: HEAD 2023-05-31T20:43:07.486 DEBUG:boto:Path: /pklxkd-59/obj44?versionId=null 2023-05-31T20:43:07.487 DEBUG:boto:Data: 2023-05-31T20:43:07.487 DEBUG:boto:Headers: {} 2023-05-31T20:43:07.487 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:43:07.487 DEBUG:boto:Port: 8001 2023-05-31T20:43:07.488 DEBUG:boto:Params: {} 2023-05-31T20:43:07.488 DEBUG:boto:Token: None 2023-05-31T20:43:07.489 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:43:07 GMT /pklxkd-59/obj44?versionId=null 2023-05-31T20:43:07.489 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:0UjVjd2JXUII+S8ic23kiZyTkH0= 2023-05-31T20:43:07.489 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:43:07 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:0UjVjd2JXUII+S8ic23kiZyTkH0=', 'Content-Length': '0'} 2023-05-31T20:43:07.491 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:39:26 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f3f650db10973cae-006477b15b-1c17-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:43:07 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:43:07.491 DEBUG:boto:path=/pklxkd-59/obj44 2023-05-31T20:43:07.492 DEBUG:boto:auth_path=/pklxkd-59/obj44 2023-05-31T20:43:07.492 DEBUG:boto:path=/pklxkd-59/obj44?versionId=null 2023-05-31T20:43:07.493 DEBUG:boto:auth_path=/pklxkd-59/obj44?versionId=null 2023-05-31T20:43:07.493 DEBUG:boto:Method: HEAD 2023-05-31T20:43:07.493 DEBUG:boto:Path: /pklxkd-59/obj44?versionId=null 2023-05-31T20:43:07.494 DEBUG:boto:Data: 2023-05-31T20:43:07.494 DEBUG:boto:Headers: {} 2023-05-31T20:43:07.494 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:43:07.494 DEBUG:boto:Port: 8000 2023-05-31T20:43:07.495 DEBUG:boto:Params: {} 2023-05-31T20:43:07.495 DEBUG:boto:Token: None 2023-05-31T20:43:07.496 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:43:07 GMT /pklxkd-59/obj44?versionId=null 2023-05-31T20:43:07.496 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:0UjVjd2JXUII+S8ic23kiZyTkH0= 2023-05-31T20:43:07.496 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:43:07 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:0UjVjd2JXUII+S8ic23kiZyTkH0=', 'Content-Length': '0'} 2023-05-31T20:43:07.498 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:39:26 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000bab57e6f582c676f-006477b15b-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:43:07 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:43:07.498 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj44 2023-05-31T20:43:07.499 DEBUG:boto:path=/pklxkd-59/obj44 2023-05-31T20:43:07.499 DEBUG:boto:auth_path=/pklxkd-59/obj44 2023-05-31T20:43:07.499 DEBUG:boto:path=/pklxkd-59/obj44?versionId=null 2023-05-31T20:43:07.500 DEBUG:boto:auth_path=/pklxkd-59/obj44?versionId=null 2023-05-31T20:43:07.500 DEBUG:boto:Method: GET 2023-05-31T20:43:07.501 DEBUG:boto:Path: /pklxkd-59/obj44?versionId=null 2023-05-31T20:43:07.501 DEBUG:boto:Data: 2023-05-31T20:43:07.501 DEBUG:boto:Headers: {} 2023-05-31T20:43:07.502 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:43:07.502 DEBUG:boto:Port: 8001 2023-05-31T20:43:07.502 DEBUG:boto:Params: {} 2023-05-31T20:43:07.503 DEBUG:boto:Token: None 2023-05-31T20:43:07.503 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:43:07 GMT /pklxkd-59/obj44?versionId=null 2023-05-31T20:43:07.503 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:3AcsMujlp3tmYoYeBSX9vkvxbQg= 2023-05-31T20:43:07.504 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:43:07 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:3AcsMujlp3tmYoYeBSX9vkvxbQg=', 'Content-Length': '0'} 2023-05-31T20:43:07.506 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:39:26 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000089b2884aa95bddd1-006477b15b-1c17-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:43:07 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:43:07.506 DEBUG:boto:path=/pklxkd-59/obj44 2023-05-31T20:43:07.507 DEBUG:boto:auth_path=/pklxkd-59/obj44 2023-05-31T20:43:07.507 DEBUG:boto:path=/pklxkd-59/obj44?versionId=null 2023-05-31T20:43:07.507 DEBUG:boto:auth_path=/pklxkd-59/obj44?versionId=null 2023-05-31T20:43:07.508 DEBUG:boto:Method: GET 2023-05-31T20:43:07.508 DEBUG:boto:Path: /pklxkd-59/obj44?versionId=null 2023-05-31T20:43:07.508 DEBUG:boto:Data: 2023-05-31T20:43:07.509 DEBUG:boto:Headers: {} 2023-05-31T20:43:07.509 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:43:07.509 DEBUG:boto:Port: 8000 2023-05-31T20:43:07.510 DEBUG:boto:Params: {} 2023-05-31T20:43:07.510 DEBUG:boto:Token: None 2023-05-31T20:43:07.511 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:43:07 GMT /pklxkd-59/obj44?versionId=null 2023-05-31T20:43:07.511 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:3AcsMujlp3tmYoYeBSX9vkvxbQg= 2023-05-31T20:43:07.511 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:43:07 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:3AcsMujlp3tmYoYeBSX9vkvxbQg=', 'Content-Length': '0'} 2023-05-31T20:43:07.513 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:39:26 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000df227cf225ebaaf7-006477b15b-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:43:07 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:43:07.513 DEBUG:boto:path=/pklxkd-59/ 2023-05-31T20:43:07.514 DEBUG:boto:auth_path=/pklxkd-59/ 2023-05-31T20:43:07.514 DEBUG:boto:path=/pklxkd-59/?prefix=obj44 2023-05-31T20:43:07.515 DEBUG:boto:auth_path=/pklxkd-59/?prefix=obj44 2023-05-31T20:43:07.515 DEBUG:boto:Method: GET 2023-05-31T20:43:07.515 DEBUG:boto:Path: /pklxkd-59/?prefix=obj44 2023-05-31T20:43:07.516 DEBUG:boto:Data: 2023-05-31T20:43:07.516 DEBUG:boto:Headers: {} 2023-05-31T20:43:07.516 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:43:07.517 DEBUG:boto:Port: 8001 2023-05-31T20:43:07.517 DEBUG:boto:Params: {} 2023-05-31T20:43:07.517 DEBUG:boto:Token: None 2023-05-31T20:43:07.518 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:43:07 GMT /pklxkd-59/ 2023-05-31T20:43:07.518 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:mHjWQcyrqf5WjOocPTRbXTtj8YQ= 2023-05-31T20:43:07.519 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:43:07 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:mHjWQcyrqf5WjOocPTRbXTtj8YQ=', 'Content-Length': '0'} 2023-05-31T20:43:07.521 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b192978057e94ffa-006477b15b-1c17-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:43:07 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:43:07.521 DEBUG:boto:b'pklxkd-59obj441000falseobj442023-05-31T20:39:26.565Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:43:07.522 DEBUG:boto:path=/pklxkd-59/ 2023-05-31T20:43:07.522 DEBUG:boto:auth_path=/pklxkd-59/ 2023-05-31T20:43:07.523 DEBUG:boto:path=/pklxkd-59/?prefix=obj44 2023-05-31T20:43:07.523 DEBUG:boto:auth_path=/pklxkd-59/?prefix=obj44 2023-05-31T20:43:07.523 DEBUG:boto:Method: GET 2023-05-31T20:43:07.524 DEBUG:boto:Path: /pklxkd-59/?prefix=obj44 2023-05-31T20:43:07.524 DEBUG:boto:Data: 2023-05-31T20:43:07.524 DEBUG:boto:Headers: {} 2023-05-31T20:43:07.525 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:43:07.525 DEBUG:boto:Port: 8000 2023-05-31T20:43:07.525 DEBUG:boto:Params: {} 2023-05-31T20:43:07.526 DEBUG:boto:Token: None 2023-05-31T20:43:07.526 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:43:07 GMT /pklxkd-59/ 2023-05-31T20:43:07.527 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:mHjWQcyrqf5WjOocPTRbXTtj8YQ= 2023-05-31T20:43:07.527 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:43:07 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:mHjWQcyrqf5WjOocPTRbXTtj8YQ=', 'Content-Length': '0'} 2023-05-31T20:43:07.529 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c609a138ac4bdf74-006477b15b-1630-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:43:07 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:43:07.530 DEBUG:boto:b'pklxkd-59obj441000falseobj442023-05-31T20:39:26.565Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:43:07.530 DEBUG:boto:path=/pklxkd-59/obj44 2023-05-31T20:43:07.531 DEBUG:boto:auth_path=/pklxkd-59/obj44 2023-05-31T20:43:07.531 DEBUG:boto:Method: HEAD 2023-05-31T20:43:07.531 DEBUG:boto:Path: /pklxkd-59/obj44 2023-05-31T20:43:07.532 DEBUG:boto:Data: 2023-05-31T20:43:07.532 DEBUG:boto:Headers: {} 2023-05-31T20:43:07.532 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:43:07.533 DEBUG:boto:Port: 8001 2023-05-31T20:43:07.533 DEBUG:boto:Params: {} 2023-05-31T20:43:07.533 DEBUG:boto:Token: None 2023-05-31T20:43:07.534 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:43:07 GMT /pklxkd-59/obj44 2023-05-31T20:43:07.534 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:PBtlVdgeBNxj2bxj/wTeGMEvbKo= 2023-05-31T20:43:07.534 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:43:07 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:PBtlVdgeBNxj2bxj/wTeGMEvbKo=', 'Content-Length': '0'} 2023-05-31T20:43:07.536 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:39:26 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c8e2957903ff7e85-006477b15b-1c17-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:43:07 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:43:07.537 DEBUG:boto:path=/pklxkd-59/obj44 2023-05-31T20:43:07.537 DEBUG:boto:auth_path=/pklxkd-59/obj44 2023-05-31T20:43:07.537 DEBUG:boto:Method: HEAD 2023-05-31T20:43:07.538 DEBUG:boto:Path: /pklxkd-59/obj44 2023-05-31T20:43:07.538 DEBUG:boto:Data: 2023-05-31T20:43:07.538 DEBUG:boto:Headers: {} 2023-05-31T20:43:07.539 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:43:07.539 DEBUG:boto:Port: 8000 2023-05-31T20:43:07.539 DEBUG:boto:Params: {} 2023-05-31T20:43:07.540 DEBUG:boto:Token: None 2023-05-31T20:43:07.540 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:43:07 GMT /pklxkd-59/obj44 2023-05-31T20:43:07.540 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:PBtlVdgeBNxj2bxj/wTeGMEvbKo= 2023-05-31T20:43:07.541 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:43:07 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:PBtlVdgeBNxj2bxj/wTeGMEvbKo=', 'Content-Length': '0'} 2023-05-31T20:43:07.542 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:39:26 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e47422f2f7782fb6-006477b15b-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:43:07 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:43:07.543 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj44 2023-05-31T20:43:07.543 DEBUG:boto:path=/pklxkd-59/obj44 2023-05-31T20:43:07.543 DEBUG:boto:auth_path=/pklxkd-59/obj44 2023-05-31T20:43:07.544 DEBUG:boto:Method: GET 2023-05-31T20:43:07.544 DEBUG:boto:Path: /pklxkd-59/obj44 2023-05-31T20:43:07.545 DEBUG:boto:Data: 2023-05-31T20:43:07.545 DEBUG:boto:Headers: {} 2023-05-31T20:43:07.545 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:43:07.546 DEBUG:boto:Port: 8001 2023-05-31T20:43:07.546 DEBUG:boto:Params: {} 2023-05-31T20:43:07.546 DEBUG:boto:Token: None 2023-05-31T20:43:07.547 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:43:07 GMT /pklxkd-59/obj44 2023-05-31T20:43:07.547 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:WUyAYyaCPDtfc9R1VEpJlLl9Pe4= 2023-05-31T20:43:07.548 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:43:07 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:WUyAYyaCPDtfc9R1VEpJlLl9Pe4=', 'Content-Length': '0'} 2023-05-31T20:43:07.549 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:39:26 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000bd011ec635181a1-006477b15b-1c17-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:43:07 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:43:07.592 DEBUG:boto:path=/pklxkd-59/obj44 2023-05-31T20:43:07.593 DEBUG:boto:auth_path=/pklxkd-59/obj44 2023-05-31T20:43:07.594 DEBUG:boto:Method: GET 2023-05-31T20:43:07.594 DEBUG:boto:Path: /pklxkd-59/obj44 2023-05-31T20:43:07.595 DEBUG:boto:Data: 2023-05-31T20:43:07.596 DEBUG:boto:Headers: {} 2023-05-31T20:43:07.596 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:43:07.597 DEBUG:boto:Port: 8000 2023-05-31T20:43:07.597 DEBUG:boto:Params: {} 2023-05-31T20:43:07.597 DEBUG:boto:Token: None 2023-05-31T20:43:07.598 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:43:07 GMT /pklxkd-59/obj44 2023-05-31T20:43:07.598 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:WUyAYyaCPDtfc9R1VEpJlLl9Pe4= 2023-05-31T20:43:07.599 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:43:07 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:WUyAYyaCPDtfc9R1VEpJlLl9Pe4=', 'Content-Length': '0'} 2023-05-31T20:43:07.601 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:39:26 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000018fe9c22037ee31d-006477b15b-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:43:07 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:43:07.601 DEBUG:boto:path=/pklxkd-59/ 2023-05-31T20:43:07.602 DEBUG:boto:auth_path=/pklxkd-59/ 2023-05-31T20:43:07.602 DEBUG:boto:path=/pklxkd-59/?prefix=obj44 2023-05-31T20:43:07.602 DEBUG:boto:auth_path=/pklxkd-59/?prefix=obj44 2023-05-31T20:43:07.603 DEBUG:boto:Method: GET 2023-05-31T20:43:07.603 DEBUG:boto:Path: /pklxkd-59/?prefix=obj44 2023-05-31T20:43:07.603 DEBUG:boto:Data: 2023-05-31T20:43:07.604 DEBUG:boto:Headers: {} 2023-05-31T20:43:07.604 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:43:07.605 DEBUG:boto:Port: 8001 2023-05-31T20:43:07.605 DEBUG:boto:Params: {} 2023-05-31T20:43:07.605 DEBUG:boto:Token: None 2023-05-31T20:43:07.606 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:43:07 GMT /pklxkd-59/ 2023-05-31T20:43:07.606 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:mHjWQcyrqf5WjOocPTRbXTtj8YQ= 2023-05-31T20:43:07.607 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:43:07 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:mHjWQcyrqf5WjOocPTRbXTtj8YQ=', 'Content-Length': '0'} 2023-05-31T20:43:07.609 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000005537929592143db8-006477b15b-1c17-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:43:07 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:43:07.609 DEBUG:boto:b'pklxkd-59obj441000falseobj442023-05-31T20:39:26.565Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:43:07.610 DEBUG:boto:path=/pklxkd-59/ 2023-05-31T20:43:07.610 DEBUG:boto:auth_path=/pklxkd-59/ 2023-05-31T20:43:07.611 DEBUG:boto:path=/pklxkd-59/?prefix=obj44 2023-05-31T20:43:07.611 DEBUG:boto:auth_path=/pklxkd-59/?prefix=obj44 2023-05-31T20:43:07.611 DEBUG:boto:Method: GET 2023-05-31T20:43:07.612 DEBUG:boto:Path: /pklxkd-59/?prefix=obj44 2023-05-31T20:43:07.612 DEBUG:boto:Data: 2023-05-31T20:43:07.612 DEBUG:boto:Headers: {} 2023-05-31T20:43:07.613 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:43:07.613 DEBUG:boto:Port: 8000 2023-05-31T20:43:07.614 DEBUG:boto:Params: {} 2023-05-31T20:43:07.614 DEBUG:boto:Token: None 2023-05-31T20:43:07.614 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:43:07 GMT /pklxkd-59/ 2023-05-31T20:43:07.615 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:mHjWQcyrqf5WjOocPTRbXTtj8YQ= 2023-05-31T20:43:07.615 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:43:07 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:mHjWQcyrqf5WjOocPTRbXTtj8YQ=', 'Content-Length': '0'} 2023-05-31T20:43:07.617 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000005ea699439c978157-006477b15b-1630-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:43:07 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:43:07.618 DEBUG:boto:b'pklxkd-59obj441000falseobj442023-05-31T20:39:26.565Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:43:07.619 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj55 2023-05-31T20:43:07.619 DEBUG:boto:path=/pklxkd-59/obj55 2023-05-31T20:43:07.619 DEBUG:boto:auth_path=/pklxkd-59/obj55 2023-05-31T20:43:07.620 DEBUG:boto:path=/pklxkd-59/obj55?versionId=null 2023-05-31T20:43:07.620 DEBUG:boto:auth_path=/pklxkd-59/obj55?versionId=null 2023-05-31T20:43:07.620 DEBUG:boto:Method: GET 2023-05-31T20:43:07.621 DEBUG:boto:Path: /pklxkd-59/obj55?versionId=null 2023-05-31T20:43:07.621 DEBUG:boto:Data: 2023-05-31T20:43:07.622 DEBUG:boto:Headers: {} 2023-05-31T20:43:07.622 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:43:07.622 DEBUG:boto:Port: 8001 2023-05-31T20:43:07.623 DEBUG:boto:Params: {} 2023-05-31T20:43:07.623 DEBUG:boto:Token: None 2023-05-31T20:43:07.623 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:43:07 GMT /pklxkd-59/obj55?versionId=null 2023-05-31T20:43:07.624 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:VhqDN+aFOG2iomUWKNt63sb41KI= 2023-05-31T20:43:07.624 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:43:07 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:VhqDN+aFOG2iomUWKNt63sb41KI=', 'Content-Length': '0'} 2023-05-31T20:43:07.626 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:39:26 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000037db9bc0062f0217-006477b15b-1c17-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:43:07 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:43:07.627 DEBUG:boto:path=/pklxkd-59/obj55 2023-05-31T20:43:07.627 DEBUG:boto:auth_path=/pklxkd-59/obj55 2023-05-31T20:43:07.627 DEBUG:boto:path=/pklxkd-59/obj55?versionId=null 2023-05-31T20:43:07.628 DEBUG:boto:auth_path=/pklxkd-59/obj55?versionId=null 2023-05-31T20:43:07.628 DEBUG:boto:Method: GET 2023-05-31T20:43:07.628 DEBUG:boto:Path: /pklxkd-59/obj55?versionId=null 2023-05-31T20:43:07.629 DEBUG:boto:Data: 2023-05-31T20:43:07.629 DEBUG:boto:Headers: {} 2023-05-31T20:43:07.630 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:43:07.630 DEBUG:boto:Port: 8000 2023-05-31T20:43:07.630 DEBUG:boto:Params: {} 2023-05-31T20:43:07.631 DEBUG:boto:Token: None 2023-05-31T20:43:07.631 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:43:07 GMT /pklxkd-59/obj55?versionId=null 2023-05-31T20:43:07.631 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:VhqDN+aFOG2iomUWKNt63sb41KI= 2023-05-31T20:43:07.632 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:43:07 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:VhqDN+aFOG2iomUWKNt63sb41KI=', 'Content-Length': '0'} 2023-05-31T20:43:07.634 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:39:26 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000370b765cc6e289a4-006477b15b-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:43:07 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:43:07.634 DEBUG:boto:path=/pklxkd-59/obj55 2023-05-31T20:43:07.635 DEBUG:boto:auth_path=/pklxkd-59/obj55 2023-05-31T20:43:07.635 DEBUG:boto:path=/pklxkd-59/obj55?versionId=null 2023-05-31T20:43:07.635 DEBUG:boto:auth_path=/pklxkd-59/obj55?versionId=null 2023-05-31T20:43:07.636 DEBUG:boto:Method: HEAD 2023-05-31T20:43:07.636 DEBUG:boto:Path: /pklxkd-59/obj55?versionId=null 2023-05-31T20:43:07.636 DEBUG:boto:Data: 2023-05-31T20:43:07.637 DEBUG:boto:Headers: {} 2023-05-31T20:43:07.637 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:43:07.637 DEBUG:boto:Port: 8001 2023-05-31T20:43:07.638 DEBUG:boto:Params: {} 2023-05-31T20:43:07.638 DEBUG:boto:Token: None 2023-05-31T20:43:07.638 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:43:07 GMT /pklxkd-59/obj55?versionId=null 2023-05-31T20:43:07.639 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:GJ4KPDoso3fVmME+4RQR1I3q7jc= 2023-05-31T20:43:07.639 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:43:07 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:GJ4KPDoso3fVmME+4RQR1I3q7jc=', 'Content-Length': '0'} 2023-05-31T20:43:07.640 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:39:26 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000096112919a90094fd-006477b15b-1c17-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:43:07 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:43:07.641 DEBUG:boto:path=/pklxkd-59/obj55 2023-05-31T20:43:07.641 DEBUG:boto:auth_path=/pklxkd-59/obj55 2023-05-31T20:43:07.642 DEBUG:boto:path=/pklxkd-59/obj55?versionId=null 2023-05-31T20:43:07.642 DEBUG:boto:auth_path=/pklxkd-59/obj55?versionId=null 2023-05-31T20:43:07.642 DEBUG:boto:Method: HEAD 2023-05-31T20:43:07.643 DEBUG:boto:Path: /pklxkd-59/obj55?versionId=null 2023-05-31T20:43:07.643 DEBUG:boto:Data: 2023-05-31T20:43:07.643 DEBUG:boto:Headers: {} 2023-05-31T20:43:07.644 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:43:07.644 DEBUG:boto:Port: 8000 2023-05-31T20:43:07.645 DEBUG:boto:Params: {} 2023-05-31T20:43:07.645 DEBUG:boto:Token: None 2023-05-31T20:43:07.645 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:43:07 GMT /pklxkd-59/obj55?versionId=null 2023-05-31T20:43:07.646 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:GJ4KPDoso3fVmME+4RQR1I3q7jc= 2023-05-31T20:43:07.646 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:43:07 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:GJ4KPDoso3fVmME+4RQR1I3q7jc=', 'Content-Length': '0'} 2023-05-31T20:43:07.647 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:39:26 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000518130a1ab3c0d61-006477b15b-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:43:07 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:43:07.648 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj55 2023-05-31T20:43:07.648 DEBUG:boto:path=/pklxkd-59/obj55 2023-05-31T20:43:07.649 DEBUG:boto:auth_path=/pklxkd-59/obj55 2023-05-31T20:43:07.649 DEBUG:boto:path=/pklxkd-59/obj55?versionId=null 2023-05-31T20:43:07.649 DEBUG:boto:auth_path=/pklxkd-59/obj55?versionId=null 2023-05-31T20:43:07.650 DEBUG:boto:Method: GET 2023-05-31T20:43:07.650 DEBUG:boto:Path: /pklxkd-59/obj55?versionId=null 2023-05-31T20:43:07.651 DEBUG:boto:Data: 2023-05-31T20:43:07.651 DEBUG:boto:Headers: {} 2023-05-31T20:43:07.651 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:43:07.652 DEBUG:boto:Port: 8001 2023-05-31T20:43:07.652 DEBUG:boto:Params: {} 2023-05-31T20:43:07.652 DEBUG:boto:Token: None 2023-05-31T20:43:07.653 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:43:07 GMT /pklxkd-59/obj55?versionId=null 2023-05-31T20:43:07.653 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:VhqDN+aFOG2iomUWKNt63sb41KI= 2023-05-31T20:43:07.654 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:43:07 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:VhqDN+aFOG2iomUWKNt63sb41KI=', 'Content-Length': '0'} 2023-05-31T20:43:07.655 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:39:26 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000050d614fa6a9f996c-006477b15b-1c17-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:43:07 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:43:07.696 DEBUG:boto:path=/pklxkd-59/obj55 2023-05-31T20:43:07.697 DEBUG:boto:auth_path=/pklxkd-59/obj55 2023-05-31T20:43:07.697 DEBUG:boto:path=/pklxkd-59/obj55?versionId=null 2023-05-31T20:43:07.697 DEBUG:boto:auth_path=/pklxkd-59/obj55?versionId=null 2023-05-31T20:43:07.698 DEBUG:boto:Method: GET 2023-05-31T20:43:07.698 DEBUG:boto:Path: /pklxkd-59/obj55?versionId=null 2023-05-31T20:43:07.699 DEBUG:boto:Data: 2023-05-31T20:43:07.699 DEBUG:boto:Headers: {} 2023-05-31T20:43:07.699 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:43:07.700 DEBUG:boto:Port: 8000 2023-05-31T20:43:07.700 DEBUG:boto:Params: {} 2023-05-31T20:43:07.700 DEBUG:boto:Token: None 2023-05-31T20:43:07.701 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:43:07 GMT /pklxkd-59/obj55?versionId=null 2023-05-31T20:43:07.701 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:VhqDN+aFOG2iomUWKNt63sb41KI= 2023-05-31T20:43:07.702 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:43:07 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:VhqDN+aFOG2iomUWKNt63sb41KI=', 'Content-Length': '0'} 2023-05-31T20:43:07.703 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:39:26 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001a6c5b4d160b2802-006477b15b-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:43:07 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:43:07.704 DEBUG:boto:path=/pklxkd-59/ 2023-05-31T20:43:07.705 DEBUG:boto:auth_path=/pklxkd-59/ 2023-05-31T20:43:07.705 DEBUG:boto:path=/pklxkd-59/?prefix=obj55 2023-05-31T20:43:07.705 DEBUG:boto:auth_path=/pklxkd-59/?prefix=obj55 2023-05-31T20:43:07.706 DEBUG:boto:Method: GET 2023-05-31T20:43:07.706 DEBUG:boto:Path: /pklxkd-59/?prefix=obj55 2023-05-31T20:43:07.706 DEBUG:boto:Data: 2023-05-31T20:43:07.707 DEBUG:boto:Headers: {} 2023-05-31T20:43:07.707 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:43:07.707 DEBUG:boto:Port: 8001 2023-05-31T20:43:07.708 DEBUG:boto:Params: {} 2023-05-31T20:43:07.708 DEBUG:boto:Token: None 2023-05-31T20:43:07.709 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:43:07 GMT /pklxkd-59/ 2023-05-31T20:43:07.709 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:mHjWQcyrqf5WjOocPTRbXTtj8YQ= 2023-05-31T20:43:07.709 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:43:07 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:mHjWQcyrqf5WjOocPTRbXTtj8YQ=', 'Content-Length': '0'} 2023-05-31T20:43:07.712 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000636b1a4d8b6cf11f-006477b15b-1c17-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:43:07 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:43:07.712 DEBUG:boto:b'pklxkd-59obj551000falseobj552023-05-31T20:39:26.630Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:43:07.713 DEBUG:boto:path=/pklxkd-59/ 2023-05-31T20:43:07.713 DEBUG:boto:auth_path=/pklxkd-59/ 2023-05-31T20:43:07.714 DEBUG:boto:path=/pklxkd-59/?prefix=obj55 2023-05-31T20:43:07.714 DEBUG:boto:auth_path=/pklxkd-59/?prefix=obj55 2023-05-31T20:43:07.714 DEBUG:boto:Method: GET 2023-05-31T20:43:07.715 DEBUG:boto:Path: /pklxkd-59/?prefix=obj55 2023-05-31T20:43:07.715 DEBUG:boto:Data: 2023-05-31T20:43:07.716 DEBUG:boto:Headers: {} 2023-05-31T20:43:07.716 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:43:07.716 DEBUG:boto:Port: 8000 2023-05-31T20:43:07.717 DEBUG:boto:Params: {} 2023-05-31T20:43:07.717 DEBUG:boto:Token: None 2023-05-31T20:43:07.717 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:43:07 GMT /pklxkd-59/ 2023-05-31T20:43:07.718 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:mHjWQcyrqf5WjOocPTRbXTtj8YQ= 2023-05-31T20:43:07.718 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:43:07 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:mHjWQcyrqf5WjOocPTRbXTtj8YQ=', 'Content-Length': '0'} 2023-05-31T20:43:07.721 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000008cee0dd40de5d761-006477b15b-1630-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:43:07 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:43:07.721 DEBUG:boto:b'pklxkd-59obj551000falseobj552023-05-31T20:39:26.630Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:43:07.722 DEBUG:boto:path=/pklxkd-59/obj55 2023-05-31T20:43:07.722 DEBUG:boto:auth_path=/pklxkd-59/obj55 2023-05-31T20:43:07.722 DEBUG:boto:Method: HEAD 2023-05-31T20:43:07.723 DEBUG:boto:Path: /pklxkd-59/obj55 2023-05-31T20:43:07.723 DEBUG:boto:Data: 2023-05-31T20:43:07.723 DEBUG:boto:Headers: {} 2023-05-31T20:43:07.724 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:43:07.724 DEBUG:boto:Port: 8001 2023-05-31T20:43:07.724 DEBUG:boto:Params: {} 2023-05-31T20:43:07.725 DEBUG:boto:Token: None 2023-05-31T20:43:07.725 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:43:07 GMT /pklxkd-59/obj55 2023-05-31T20:43:07.726 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:IEOyp8X6miJVv5/vRuOz9KyBzTA= 2023-05-31T20:43:07.726 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:43:07 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:IEOyp8X6miJVv5/vRuOz9KyBzTA=', 'Content-Length': '0'} 2023-05-31T20:43:07.728 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:39:26 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000344def317fee31c1-006477b15b-1c17-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:43:07 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:43:07.728 DEBUG:boto:path=/pklxkd-59/obj55 2023-05-31T20:43:07.729 DEBUG:boto:auth_path=/pklxkd-59/obj55 2023-05-31T20:43:07.729 DEBUG:boto:Method: HEAD 2023-05-31T20:43:07.729 DEBUG:boto:Path: /pklxkd-59/obj55 2023-05-31T20:43:07.730 DEBUG:boto:Data: 2023-05-31T20:43:07.730 DEBUG:boto:Headers: {} 2023-05-31T20:43:07.730 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:43:07.731 DEBUG:boto:Port: 8000 2023-05-31T20:43:07.731 DEBUG:boto:Params: {} 2023-05-31T20:43:07.731 DEBUG:boto:Token: None 2023-05-31T20:43:07.732 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:43:07 GMT /pklxkd-59/obj55 2023-05-31T20:43:07.732 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:IEOyp8X6miJVv5/vRuOz9KyBzTA= 2023-05-31T20:43:07.733 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:43:07 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:IEOyp8X6miJVv5/vRuOz9KyBzTA=', 'Content-Length': '0'} 2023-05-31T20:43:07.734 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:39:26 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000029b7ad130ca96e94-006477b15b-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:43:07 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:43:07.735 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj55 2023-05-31T20:43:07.735 DEBUG:boto:path=/pklxkd-59/obj55 2023-05-31T20:43:07.736 DEBUG:boto:auth_path=/pklxkd-59/obj55 2023-05-31T20:43:07.736 DEBUG:boto:Method: GET 2023-05-31T20:43:07.736 DEBUG:boto:Path: /pklxkd-59/obj55 2023-05-31T20:43:07.737 DEBUG:boto:Data: 2023-05-31T20:43:07.737 DEBUG:boto:Headers: {} 2023-05-31T20:43:07.737 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:43:07.738 DEBUG:boto:Port: 8001 2023-05-31T20:43:07.738 DEBUG:boto:Params: {} 2023-05-31T20:43:07.738 DEBUG:boto:Token: None 2023-05-31T20:43:07.739 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:43:07 GMT /pklxkd-59/obj55 2023-05-31T20:43:07.739 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:gXOxpqHSDonA2v4r7sDTxcGYyg4= 2023-05-31T20:43:07.740 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:43:07 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:gXOxpqHSDonA2v4r7sDTxcGYyg4=', 'Content-Length': '0'} 2023-05-31T20:43:07.741 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:39:26 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000002434fad193ad2fa-006477b15b-1c17-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:43:07 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:43:07.742 DEBUG:boto:path=/pklxkd-59/obj55 2023-05-31T20:43:07.742 DEBUG:boto:auth_path=/pklxkd-59/obj55 2023-05-31T20:43:07.743 DEBUG:boto:Method: GET 2023-05-31T20:43:07.743 DEBUG:boto:Path: /pklxkd-59/obj55 2023-05-31T20:43:07.743 DEBUG:boto:Data: 2023-05-31T20:43:07.744 DEBUG:boto:Headers: {} 2023-05-31T20:43:07.744 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:43:07.744 DEBUG:boto:Port: 8000 2023-05-31T20:43:07.745 DEBUG:boto:Params: {} 2023-05-31T20:43:07.745 DEBUG:boto:Token: None 2023-05-31T20:43:07.746 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:43:07 GMT /pklxkd-59/obj55 2023-05-31T20:43:07.746 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:gXOxpqHSDonA2v4r7sDTxcGYyg4= 2023-05-31T20:43:07.746 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:43:07 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:gXOxpqHSDonA2v4r7sDTxcGYyg4=', 'Content-Length': '0'} 2023-05-31T20:43:07.748 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:39:26 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d386b57613cf3f91-006477b15b-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:43:07 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:43:07.792 DEBUG:boto:path=/pklxkd-59/ 2023-05-31T20:43:07.793 DEBUG:boto:auth_path=/pklxkd-59/ 2023-05-31T20:43:07.793 DEBUG:boto:path=/pklxkd-59/?prefix=obj55 2023-05-31T20:43:07.794 DEBUG:boto:auth_path=/pklxkd-59/?prefix=obj55 2023-05-31T20:43:07.794 DEBUG:boto:Method: GET 2023-05-31T20:43:07.794 DEBUG:boto:Path: /pklxkd-59/?prefix=obj55 2023-05-31T20:43:07.795 DEBUG:boto:Data: 2023-05-31T20:43:07.795 DEBUG:boto:Headers: {} 2023-05-31T20:43:07.795 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:43:07.796 DEBUG:boto:Port: 8001 2023-05-31T20:43:07.796 DEBUG:boto:Params: {} 2023-05-31T20:43:07.797 DEBUG:boto:Token: None 2023-05-31T20:43:07.797 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:43:07 GMT /pklxkd-59/ 2023-05-31T20:43:07.797 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:mHjWQcyrqf5WjOocPTRbXTtj8YQ= 2023-05-31T20:43:07.798 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:43:07 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:mHjWQcyrqf5WjOocPTRbXTtj8YQ=', 'Content-Length': '0'} 2023-05-31T20:43:07.800 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000008beb487f581e8e6b-006477b15b-1c17-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:43:07 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:43:07.801 DEBUG:boto:b'pklxkd-59obj551000falseobj552023-05-31T20:39:26.630Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:43:07.801 DEBUG:boto:path=/pklxkd-59/ 2023-05-31T20:43:07.802 DEBUG:boto:auth_path=/pklxkd-59/ 2023-05-31T20:43:07.802 DEBUG:boto:path=/pklxkd-59/?prefix=obj55 2023-05-31T20:43:07.802 DEBUG:boto:auth_path=/pklxkd-59/?prefix=obj55 2023-05-31T20:43:07.803 DEBUG:boto:Method: GET 2023-05-31T20:43:07.803 DEBUG:boto:Path: /pklxkd-59/?prefix=obj55 2023-05-31T20:43:07.803 DEBUG:boto:Data: 2023-05-31T20:43:07.804 DEBUG:boto:Headers: {} 2023-05-31T20:43:07.804 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:43:07.805 DEBUG:boto:Port: 8000 2023-05-31T20:43:07.805 DEBUG:boto:Params: {} 2023-05-31T20:43:07.805 DEBUG:boto:Token: None 2023-05-31T20:43:07.806 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:43:07 GMT /pklxkd-59/ 2023-05-31T20:43:07.806 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:mHjWQcyrqf5WjOocPTRbXTtj8YQ= 2023-05-31T20:43:07.807 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:43:07 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:mHjWQcyrqf5WjOocPTRbXTtj8YQ=', 'Content-Length': '0'} 2023-05-31T20:43:07.809 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b17a55f7e75d84ec-006477b15b-1630-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:43:07 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:43:07.810 DEBUG:boto:b'pklxkd-59obj551000falseobj552023-05-31T20:39:26.630Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:43:07.810 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=pklxkd-59 zones={a2, a1} 2023-05-31T20:43:07.811 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'reshard', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'pklxkd-59', '--num-shards', '17', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-05-31T20:43:07.811 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'reshard', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'pklxkd-59', '--num-shards', '17', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-05-31T20:43:07.812 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket reshard --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --bucket pklxkd-59 --num-shards 17 --yes-i-really-mean-it --cluster c1 --debug-rgw 0 --debug-ms 0 2023-05-31T20:43:07.864 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2023-05-31T20:43:07.864 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2023-05-31T20:43:07.905 INFO:teuthology.orchestra.run.smithi099.stdout:tenant: 2023-05-31T20:43:07.905 INFO:teuthology.orchestra.run.smithi099.stdout:bucket name: pklxkd-59 2023-05-31T20:43:07.908 INFO:teuthology.orchestra.run.smithi099.stdout:total entries: 5 2023-05-31T20:43:07.967 DEBUG:boto:path=/pklxkd-59/ 2023-05-31T20:43:07.968 DEBUG:boto:auth_path=/pklxkd-59/ 2023-05-31T20:43:07.968 DEBUG:boto:Method: HEAD 2023-05-31T20:43:07.968 DEBUG:boto:Path: /pklxkd-59/ 2023-05-31T20:43:07.969 DEBUG:boto:Data: 2023-05-31T20:43:07.969 DEBUG:boto:Headers: {} 2023-05-31T20:43:07.969 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:43:07.970 DEBUG:boto:Port: 8000 2023-05-31T20:43:07.970 DEBUG:boto:Params: {} 2023-05-31T20:43:07.970 DEBUG:boto:Token: None 2023-05-31T20:43:07.971 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:43:07 GMT /pklxkd-59/ 2023-05-31T20:43:07.971 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:tDJaoXy5avQP1d+boJjnV4LmbrU= 2023-05-31T20:43:07.972 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:43:07 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:tDJaoXy5avQP1d+boJjnV4LmbrU=', 'Content-Length': '0'} 2023-05-31T20:43:07.974 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '5'), ('X-RGW-Bytes-Used', '15'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000b9cff7fd22073390-006477b15b-1630-a1'), ('Content-Length', '0'), ('Date', 'Wed, 31 May 2023 20:43:07 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:43:07.975 DEBUG:boto:path=/pklxkd-59/obj17 2023-05-31T20:43:07.975 DEBUG:boto:auth_path=/pklxkd-59/obj17 2023-05-31T20:43:07.975 DEBUG:boto:Method: PUT 2023-05-31T20:43:07.976 DEBUG:boto:Path: /pklxkd-59/obj17 2023-05-31T20:43:07.976 DEBUG:boto:Data: 2023-05-31T20:43:07.976 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-05-31T20:43:07.977 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:43:07.977 DEBUG:boto:Port: 8000 2023-05-31T20:43:07.977 DEBUG:boto:Params: {} 2023-05-31T20:43:07.978 DEBUG:boto:Token: None 2023-05-31T20:43:07.978 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Wed, 31 May 2023 20:43:07 GMT /pklxkd-59/obj17 2023-05-31T20:43:07.978 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:mM88VjmWSLyDdA3UpXPeVz++rbM= 2023-05-31T20:43:07.979 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Wed, 31 May 2023 20:43:07 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:mM88VjmWSLyDdA3UpXPeVz++rbM='} 2023-05-31T20:43:07.984 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000007151af6e0e63bc46-006477b15b-1630-a1'), ('Date', 'Wed, 31 May 2023 20:43:07 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:43:07.985 DEBUG:boto:path=/pklxkd-59/ 2023-05-31T20:43:07.985 DEBUG:boto:auth_path=/pklxkd-59/ 2023-05-31T20:43:07.985 DEBUG:boto:Method: HEAD 2023-05-31T20:43:07.986 DEBUG:boto:Path: /pklxkd-59/ 2023-05-31T20:43:07.986 DEBUG:boto:Data: 2023-05-31T20:43:07.986 DEBUG:boto:Headers: {} 2023-05-31T20:43:07.987 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:43:07.987 DEBUG:boto:Port: 8000 2023-05-31T20:43:07.987 DEBUG:boto:Params: {} 2023-05-31T20:43:07.988 DEBUG:boto:Token: None 2023-05-31T20:43:07.988 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:43:07 GMT /pklxkd-59/ 2023-05-31T20:43:07.988 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:tDJaoXy5avQP1d+boJjnV4LmbrU= 2023-05-31T20:43:07.989 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:43:07 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:tDJaoXy5avQP1d+boJjnV4LmbrU=', 'Content-Length': '0'} 2023-05-31T20:43:07.991 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '6'), ('X-RGW-Bytes-Used', '18'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000004bc8502d68f00ba-006477b15b-1630-a1'), ('Content-Length', '0'), ('Date', 'Wed, 31 May 2023 20:43:07 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:43:07.991 DEBUG:boto:path=/pklxkd-59/obj34 2023-05-31T20:43:07.992 DEBUG:boto:auth_path=/pklxkd-59/obj34 2023-05-31T20:43:07.992 DEBUG:boto:Method: PUT 2023-05-31T20:43:07.992 DEBUG:boto:Path: /pklxkd-59/obj34 2023-05-31T20:43:07.993 DEBUG:boto:Data: 2023-05-31T20:43:07.993 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-05-31T20:43:07.993 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:43:07.994 DEBUG:boto:Port: 8000 2023-05-31T20:43:07.994 DEBUG:boto:Params: {} 2023-05-31T20:43:07.994 DEBUG:boto:Token: None 2023-05-31T20:43:07.995 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Wed, 31 May 2023 20:43:07 GMT /pklxkd-59/obj34 2023-05-31T20:43:07.995 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:VPor5/hgh6mnucVLxLWJK1NjXqA= 2023-05-31T20:43:07.996 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Wed, 31 May 2023 20:43:07 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:VPor5/hgh6mnucVLxLWJK1NjXqA='} 2023-05-31T20:43:08.002 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000007f0297620a38f47f-006477b15b-1630-a1'), ('Date', 'Wed, 31 May 2023 20:43:07 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:43:08.002 DEBUG:boto:path=/pklxkd-59/ 2023-05-31T20:43:08.003 DEBUG:boto:auth_path=/pklxkd-59/ 2023-05-31T20:43:08.003 DEBUG:boto:Method: HEAD 2023-05-31T20:43:08.003 DEBUG:boto:Path: /pklxkd-59/ 2023-05-31T20:43:08.004 DEBUG:boto:Data: 2023-05-31T20:43:08.004 DEBUG:boto:Headers: {} 2023-05-31T20:43:08.004 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:43:08.005 DEBUG:boto:Port: 8000 2023-05-31T20:43:08.005 DEBUG:boto:Params: {} 2023-05-31T20:43:08.005 DEBUG:boto:Token: None 2023-05-31T20:43:08.006 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:43:08 GMT /pklxkd-59/ 2023-05-31T20:43:08.006 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:ewBXLkpP4/TEK8zKc5cCyymNZPE= 2023-05-31T20:43:08.006 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:43:08 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:ewBXLkpP4/TEK8zKc5cCyymNZPE=', 'Content-Length': '0'} 2023-05-31T20:43:08.009 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '7'), ('X-RGW-Bytes-Used', '21'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000d364e893be32601b-006477b15c-1630-a1'), ('Content-Length', '0'), ('Date', 'Wed, 31 May 2023 20:43:08 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:43:08.009 DEBUG:boto:path=/pklxkd-59/obj51 2023-05-31T20:43:08.010 DEBUG:boto:auth_path=/pklxkd-59/obj51 2023-05-31T20:43:08.010 DEBUG:boto:Method: PUT 2023-05-31T20:43:08.010 DEBUG:boto:Path: /pklxkd-59/obj51 2023-05-31T20:43:08.011 DEBUG:boto:Data: 2023-05-31T20:43:08.011 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-05-31T20:43:08.011 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:43:08.012 DEBUG:boto:Port: 8000 2023-05-31T20:43:08.012 DEBUG:boto:Params: {} 2023-05-31T20:43:08.012 DEBUG:boto:Token: None 2023-05-31T20:43:08.013 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Wed, 31 May 2023 20:43:08 GMT /pklxkd-59/obj51 2023-05-31T20:43:08.013 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:RR8cn4b4Y8R5pInmErzpPxh1TgM= 2023-05-31T20:43:08.013 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Wed, 31 May 2023 20:43:08 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:RR8cn4b4Y8R5pInmErzpPxh1TgM='} 2023-05-31T20:43:08.056 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000040adae8a12865ad4-006477b15c-1630-a1'), ('Date', 'Wed, 31 May 2023 20:43:08 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:43:08.057 DEBUG:boto:path=/pklxkd-59/ 2023-05-31T20:43:08.057 DEBUG:boto:auth_path=/pklxkd-59/ 2023-05-31T20:43:08.058 DEBUG:boto:Method: HEAD 2023-05-31T20:43:08.058 DEBUG:boto:Path: /pklxkd-59/ 2023-05-31T20:43:08.058 DEBUG:boto:Data: 2023-05-31T20:43:08.059 DEBUG:boto:Headers: {} 2023-05-31T20:43:08.059 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:43:08.059 DEBUG:boto:Port: 8000 2023-05-31T20:43:08.060 DEBUG:boto:Params: {} 2023-05-31T20:43:08.060 DEBUG:boto:Token: None 2023-05-31T20:43:08.060 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:43:08 GMT /pklxkd-59/ 2023-05-31T20:43:08.061 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:ewBXLkpP4/TEK8zKc5cCyymNZPE= 2023-05-31T20:43:08.061 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:43:08 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:ewBXLkpP4/TEK8zKc5cCyymNZPE=', 'Content-Length': '0'} 2023-05-31T20:43:08.064 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '8'), ('X-RGW-Bytes-Used', '24'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000edc2c63e65c30f95-006477b15c-1630-a1'), ('Content-Length', '0'), ('Date', 'Wed, 31 May 2023 20:43:08 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:43:08.064 DEBUG:boto:path=/pklxkd-59/obj68 2023-05-31T20:43:08.065 DEBUG:boto:auth_path=/pklxkd-59/obj68 2023-05-31T20:43:08.065 DEBUG:boto:Method: PUT 2023-05-31T20:43:08.065 DEBUG:boto:Path: /pklxkd-59/obj68 2023-05-31T20:43:08.066 DEBUG:boto:Data: 2023-05-31T20:43:08.066 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-05-31T20:43:08.066 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:43:08.067 DEBUG:boto:Port: 8000 2023-05-31T20:43:08.067 DEBUG:boto:Params: {} 2023-05-31T20:43:08.067 DEBUG:boto:Token: None 2023-05-31T20:43:08.068 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Wed, 31 May 2023 20:43:08 GMT /pklxkd-59/obj68 2023-05-31T20:43:08.068 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:o+5wewD6d4suE5e0SdHiR/Tdip0= 2023-05-31T20:43:08.068 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Wed, 31 May 2023 20:43:08 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:o+5wewD6d4suE5e0SdHiR/Tdip0='} 2023-05-31T20:43:08.073 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000004e687c2021b99845-006477b15c-1630-a1'), ('Date', 'Wed, 31 May 2023 20:43:08 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:43:08.074 DEBUG:boto:path=/pklxkd-59/ 2023-05-31T20:43:08.074 DEBUG:boto:auth_path=/pklxkd-59/ 2023-05-31T20:43:08.074 DEBUG:boto:Method: HEAD 2023-05-31T20:43:08.075 DEBUG:boto:Path: /pklxkd-59/ 2023-05-31T20:43:08.075 DEBUG:boto:Data: 2023-05-31T20:43:08.075 DEBUG:boto:Headers: {} 2023-05-31T20:43:08.076 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:43:08.076 DEBUG:boto:Port: 8000 2023-05-31T20:43:08.076 DEBUG:boto:Params: {} 2023-05-31T20:43:08.077 DEBUG:boto:Token: None 2023-05-31T20:43:08.077 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:43:08 GMT /pklxkd-59/ 2023-05-31T20:43:08.078 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:ewBXLkpP4/TEK8zKc5cCyymNZPE= 2023-05-31T20:43:08.078 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:43:08 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:ewBXLkpP4/TEK8zKc5cCyymNZPE=', 'Content-Length': '0'} 2023-05-31T20:43:08.080 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '9'), ('X-RGW-Bytes-Used', '27'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000007054393b91a4bfc-006477b15c-1630-a1'), ('Content-Length', '0'), ('Date', 'Wed, 31 May 2023 20:43:08 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:43:08.080 DEBUG:boto:path=/pklxkd-59/obj85 2023-05-31T20:43:08.081 DEBUG:boto:auth_path=/pklxkd-59/obj85 2023-05-31T20:43:08.081 DEBUG:boto:Method: PUT 2023-05-31T20:43:08.082 DEBUG:boto:Path: /pklxkd-59/obj85 2023-05-31T20:43:08.082 DEBUG:boto:Data: 2023-05-31T20:43:08.082 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-05-31T20:43:08.082 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:43:08.083 DEBUG:boto:Port: 8000 2023-05-31T20:43:08.083 DEBUG:boto:Params: {} 2023-05-31T20:43:08.084 DEBUG:boto:Token: None 2023-05-31T20:43:08.084 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Wed, 31 May 2023 20:43:08 GMT /pklxkd-59/obj85 2023-05-31T20:43:08.084 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:snVkEns58fdIQS6Uys91tw8YF3o= 2023-05-31T20:43:08.085 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Wed, 31 May 2023 20:43:08 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:snVkEns58fdIQS6Uys91tw8YF3o='} 2023-05-31T20:43:08.094 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000bcf6d3c8edbbe21e-006477b15c-1630-a1'), ('Date', 'Wed, 31 May 2023 20:43:08 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:43:08.094 DEBUG:rgw_multi.tests:bucket checkpoint: source=a1 target=a2 bucket=pklxkd-59 2023-05-31T20:43:08.095 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'pklxkd-59', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-05-31T20:43:08.095 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'pklxkd-59', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-05-31T20:43:08.096 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket pklxkd-59 --source-zone a1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2023-05-31T20:43:08.152 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2023-05-31T20:43:08.152 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2023-05-31T20:43:08.189 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:43:08.185+0000 7fa1f7910780 1 waiting to reach latest gen 1, on 0.. 2023-05-31T20:43:13.190 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:43:13.186+0000 7fa1f7910780 1 waiting to reach latest gen 1, on 0.. 2023-05-31T20:43:18.190 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:43:18.186+0000 7fa1f7910780 1 waiting to reach latest gen 1, on 0.. 2023-05-31T20:43:23.191 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:43:23.186+0000 7fa1f7910780 1 waiting to reach latest gen 1, on 0.. 2023-05-31T20:43:28.193 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:43:28.188+0000 7fa1f7910780 1 waiting for incremental sync to catch up: 2023-05-31T20:43:28.194 INFO:teuthology.orchestra.run.smithi099.stderr: local status: [, , , , , , , , , , , , , , , , ] 2023-05-31T20:43:28.194 INFO:teuthology.orchestra.run.smithi099.stderr: remote markers: [00000000001.34.6, , , , , , 00000000001.57.6, , 00000000001.138.6, , , , 00000000001.65.6, 00000000001.98.6, , , ] 2023-05-31T20:43:33.195 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:43:33.190+0000 7fa1f7910780 1 waiting for incremental sync to catch up: 2023-05-31T20:43:33.196 INFO:teuthology.orchestra.run.smithi099.stderr: local status: [, , , , , , , , , , , , , , , , ] 2023-05-31T20:43:33.196 INFO:teuthology.orchestra.run.smithi099.stderr: remote markers: [00000000001.34.6, , , , , , 00000000001.57.6, , 00000000001.138.6, , , , 00000000001.65.6, 00000000001.98.6, , , ] 2023-05-31T20:43:38.196 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:43:38.191+0000 7fa1f7910780 1 waiting for incremental sync to catch up: 2023-05-31T20:43:38.197 INFO:teuthology.orchestra.run.smithi099.stderr: local status: [, , , , , , , , , , , , , , , , ] 2023-05-31T20:43:38.197 INFO:teuthology.orchestra.run.smithi099.stderr: remote markers: [00000000001.34.6, , , , , , 00000000001.57.6, , 00000000001.138.6, , , , 00000000001.65.6, 00000000001.98.6, , , ] 2023-05-31T20:43:43.198 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:43:43.193+0000 7fa1f7910780 1 waiting for incremental sync to catch up: 2023-05-31T20:43:43.198 INFO:teuthology.orchestra.run.smithi099.stderr: local status: [, , , , , , , , , , , , , , , , ] 2023-05-31T20:43:43.198 INFO:teuthology.orchestra.run.smithi099.stderr: remote markers: [00000000001.34.6, , , , , , 00000000001.57.6, , 00000000001.138.6, , , , 00000000001.65.6, 00000000001.98.6, , , ] 2023-05-31T20:43:48.199 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:43:48.195+0000 7fa1f7910780 1 waiting for incremental sync to catch up: 2023-05-31T20:43:48.200 INFO:teuthology.orchestra.run.smithi099.stderr: local status: [, , , , , , , , , , , , , , , , ] 2023-05-31T20:43:48.200 INFO:teuthology.orchestra.run.smithi099.stderr: remote markers: [00000000001.34.6, , , , , , 00000000001.57.6, , 00000000001.138.6, , , , 00000000001.65.6, 00000000001.98.6, , , ] 2023-05-31T20:43:53.201 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:43:53.196+0000 7fa1f7910780 1 waiting for incremental sync to catch up: 2023-05-31T20:43:53.201 INFO:teuthology.orchestra.run.smithi099.stderr: local status: [, , , , , , , , , , , , , , , , ] 2023-05-31T20:43:53.202 INFO:teuthology.orchestra.run.smithi099.stderr: remote markers: [00000000001.34.6, , , , , , 00000000001.57.6, , 00000000001.138.6, , , , 00000000001.65.6, 00000000001.98.6, , , ] 2023-05-31T20:43:58.202 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:43:58.198+0000 7fa1f7910780 1 waiting for incremental sync to catch up: 2023-05-31T20:43:58.203 INFO:teuthology.orchestra.run.smithi099.stderr: local status: [, , , , , , , , , , , , , , , , ] 2023-05-31T20:43:58.203 INFO:teuthology.orchestra.run.smithi099.stderr: remote markers: [00000000001.34.6, , , , , , 00000000001.57.6, , 00000000001.138.6, , , , 00000000001.65.6, 00000000001.98.6, , , ] 2023-05-31T20:44:03.204 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:44:03.199+0000 7fa1f7910780 1 waiting for incremental sync to catch up: 2023-05-31T20:44:03.204 INFO:teuthology.orchestra.run.smithi099.stderr: local status: [, , , , , , , , , , , , , , , , ] 2023-05-31T20:44:03.204 INFO:teuthology.orchestra.run.smithi099.stderr: remote markers: [00000000001.34.6, , , , , , 00000000001.57.6, , 00000000001.138.6, , , , 00000000001.65.6, 00000000001.98.6, , , ] 2023-05-31T20:44:08.205 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:44:08.200+0000 7fa1f7910780 1 waiting for incremental sync to catch up: 2023-05-31T20:44:08.205 INFO:teuthology.orchestra.run.smithi099.stderr: local status: [, , , , , , 00000000001.57.6, , , , , , , , , , ] 2023-05-31T20:44:08.206 INFO:teuthology.orchestra.run.smithi099.stderr: remote markers: [00000000001.34.6, , , , , , 00000000001.57.6, , 00000000001.138.6, , , , 00000000001.65.6, 00000000001.98.6, , , ] 2023-05-31T20:44:13.206 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:44:13.201+0000 7fa1f7910780 1 waiting for incremental sync to catch up: 2023-05-31T20:44:13.206 INFO:teuthology.orchestra.run.smithi099.stderr: local status: [, , , , , , 00000000001.57.6, , , , , , , , , , ] 2023-05-31T20:44:13.207 INFO:teuthology.orchestra.run.smithi099.stderr: remote markers: [00000000001.34.6, , , , , , 00000000001.57.6, , 00000000001.138.6, , , , 00000000001.65.6, 00000000001.98.6, , , ] 2023-05-31T20:44:18.208 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:44:18.202+0000 7fa1f7910780 1 waiting for incremental sync to catch up: 2023-05-31T20:44:18.208 INFO:teuthology.orchestra.run.smithi099.stderr: local status: [, , , , , , 00000000001.57.6, , , , , , , , , , ] 2023-05-31T20:44:18.208 INFO:teuthology.orchestra.run.smithi099.stderr: remote markers: [00000000001.34.6, , , , , , 00000000001.57.6, , 00000000001.138.6, , , , 00000000001.65.6, 00000000001.98.6, , , ] 2023-05-31T20:44:23.209 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:44:23.204+0000 7fa1f7910780 1 waiting for incremental sync to catch up: 2023-05-31T20:44:23.209 INFO:teuthology.orchestra.run.smithi099.stderr: local status: [, , , , , , 00000000001.57.6, , , , , , , , , , ] 2023-05-31T20:44:23.210 INFO:teuthology.orchestra.run.smithi099.stderr: remote markers: [00000000001.34.6, , , , , , 00000000001.57.6, , 00000000001.138.6, , , , 00000000001.65.6, 00000000001.98.6, , , ] 2023-05-31T20:44:28.211 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:44:28.207+0000 7fa1f7910780 1 bucket sync caught up with source: 2023-05-31T20:44:28.211 INFO:teuthology.orchestra.run.smithi099.stderr: local status: [00000000001.34.6, , , , , , 00000000001.57.6, , 00000000001.138.6, , , , 00000000001.65.6, 00000000001.98.6, , , ] 2023-05-31T20:44:28.211 INFO:teuthology.orchestra.run.smithi099.stderr: remote markers: [00000000001.34.6, , , , , , 00000000001.57.6, , 00000000001.138.6, , , , 00000000001.65.6, 00000000001.98.6, , , ] 2023-05-31T20:44:28.211 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:44:28.207+0000 7fa1f7910780 0 bucket checkpoint complete 2023-05-31T20:44:28.234 DEBUG:rgw_multi.tests:bucket checkpoint: source=a2 target=a1 bucket=pklxkd-59 2023-05-31T20:44:28.235 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'pklxkd-59', '--source-zone', 'a2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-05-31T20:44:28.235 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'pklxkd-59', '--source-zone', 'a2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-05-31T20:44:28.236 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket pklxkd-59 --source-zone a2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2023-05-31T20:44:28.298 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2023-05-31T20:44:28.298 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2023-05-31T20:44:28.349 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:44:28.345+0000 7fa7785e6780 1 waiting for incremental sync to catch up: 2023-05-31T20:44:28.349 INFO:teuthology.orchestra.run.smithi099.stderr: local status: [, , , , , , , , 00000000001.1995.6, 00000000002.2283.6, 00000000003.2008.6] 2023-05-31T20:44:28.349 INFO:teuthology.orchestra.run.smithi099.stderr: remote markers: [, 00000000001.3577.6, 00000000001.3888.6, , , 00000000001.2001.6, , , 00000000001.1995.6, 00000000003.2287.6, 00000000003.2008.6] 2023-05-31T20:44:33.350 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:44:33.346+0000 7fa7785e6780 1 waiting for incremental sync to catch up: 2023-05-31T20:44:33.350 INFO:teuthology.orchestra.run.smithi099.stderr: local status: [, , , , , , , , 00000000001.1995.6, 00000000002.2283.6, 00000000003.2008.6] 2023-05-31T20:44:33.350 INFO:teuthology.orchestra.run.smithi099.stderr: remote markers: [, 00000000001.3577.6, 00000000001.3888.6, , , 00000000001.2001.6, , , 00000000001.1995.6, 00000000003.2287.6, 00000000003.2008.6] 2023-05-31T20:44:38.351 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:44:38.347+0000 7fa7785e6780 1 waiting for incremental sync to catch up: 2023-05-31T20:44:38.352 INFO:teuthology.orchestra.run.smithi099.stderr: local status: [, , , , , , , , 00000000001.1995.6, 00000000002.2283.6, 00000000003.2008.6] 2023-05-31T20:44:38.352 INFO:teuthology.orchestra.run.smithi099.stderr: remote markers: [, 00000000001.3577.6, 00000000001.3888.6, , , 00000000001.2001.6, , , 00000000001.1995.6, 00000000003.2287.6, 00000000003.2008.6] 2023-05-31T20:44:43.353 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:44:43.349+0000 7fa7785e6780 1 waiting for incremental sync to catch up: 2023-05-31T20:44:43.353 INFO:teuthology.orchestra.run.smithi099.stderr: local status: [, , , , , , , , 00000000001.1995.6, 00000000002.2283.6, 00000000003.2008.6] 2023-05-31T20:44:43.353 INFO:teuthology.orchestra.run.smithi099.stderr: remote markers: [, 00000000001.3577.6, 00000000001.3888.6, , , 00000000001.2001.6, , , 00000000001.1995.6, 00000000003.2287.6, 00000000003.2008.6] 2023-05-31T20:44:48.354 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:44:48.349+0000 7fa7785e6780 1 bucket sync caught up with source: 2023-05-31T20:44:48.355 INFO:teuthology.orchestra.run.smithi099.stderr: local status: [, 00000000001.3577.6, 00000000001.3888.6, , , 00000000001.2001.6, , , 00000000001.1995.6, 00000000003.2287.6, 00000000003.2008.6] 2023-05-31T20:44:48.355 INFO:teuthology.orchestra.run.smithi099.stderr: remote markers: [, 00000000001.3577.6, 00000000001.3888.6, , , 00000000001.2001.6, , , 00000000001.1995.6, 00000000003.2287.6, 00000000003.2008.6] 2023-05-31T20:44:48.355 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:44:48.349+0000 7fa7785e6780 0 bucket checkpoint complete 2023-05-31T20:44:48.374 INFO:tasks.rgw_multi.zone_rados:comparing bucket=pklxkd-59 zones={a2, a1} 2023-05-31T20:44:48.375 DEBUG:boto:path=/pklxkd-59/ 2023-05-31T20:44:48.375 DEBUG:boto:auth_path=/pklxkd-59/ 2023-05-31T20:44:48.375 DEBUG:boto:Method: HEAD 2023-05-31T20:44:48.376 DEBUG:boto:Path: /pklxkd-59/ 2023-05-31T20:44:48.376 DEBUG:boto:Data: 2023-05-31T20:44:48.376 DEBUG:boto:Headers: {} 2023-05-31T20:44:48.377 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:44:48.377 DEBUG:boto:Port: 8001 2023-05-31T20:44:48.377 DEBUG:boto:Params: {} 2023-05-31T20:44:48.378 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8001} 2023-05-31T20:44:48.378 DEBUG:boto:Token: None 2023-05-31T20:44:48.379 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:44:48 GMT /pklxkd-59/ 2023-05-31T20:44:48.379 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:DLwLe1yvtHAYXQJfJExFb3hLctU= 2023-05-31T20:44:48.380 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:44:48 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:DLwLe1yvtHAYXQJfJExFb3hLctU=', 'Content-Length': '0'} 2023-05-31T20:44:48.383 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '10'), ('X-RGW-Bytes-Used', '30'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000e7fa3bf91b09810f-006477b1c0-1c17-a2'), ('Content-Length', '0'), ('Date', 'Wed, 31 May 2023 20:44:48 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:44:48.384 DEBUG:boto:path=/pklxkd-59/ 2023-05-31T20:44:48.384 DEBUG:boto:auth_path=/pklxkd-59/ 2023-05-31T20:44:48.385 DEBUG:boto:Method: HEAD 2023-05-31T20:44:48.385 DEBUG:boto:Path: /pklxkd-59/ 2023-05-31T20:44:48.385 DEBUG:boto:Data: 2023-05-31T20:44:48.386 DEBUG:boto:Headers: {} 2023-05-31T20:44:48.386 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:44:48.386 DEBUG:boto:Port: 8000 2023-05-31T20:44:48.387 DEBUG:boto:Params: {} 2023-05-31T20:44:48.387 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2023-05-31T20:44:48.388 DEBUG:boto:Token: None 2023-05-31T20:44:48.388 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:44:48 GMT /pklxkd-59/ 2023-05-31T20:44:48.388 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:DLwLe1yvtHAYXQJfJExFb3hLctU= 2023-05-31T20:44:48.389 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:44:48 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:DLwLe1yvtHAYXQJfJExFb3hLctU=', 'Content-Length': '0'} 2023-05-31T20:44:48.392 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '10'), ('X-RGW-Bytes-Used', '30'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000cc9480cda0d9c63c-006477b1c0-1630-a1'), ('Content-Length', '0'), ('Date', 'Wed, 31 May 2023 20:44:48 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:44:48.392 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2023-05-31T20:44:48.393 DEBUG:boto:path=/pklxkd-59/ 2023-05-31T20:44:48.393 DEBUG:boto:auth_path=/pklxkd-59/ 2023-05-31T20:44:48.393 DEBUG:boto:path=/pklxkd-59/?versions&max-keys=999 2023-05-31T20:44:48.394 DEBUG:boto:auth_path=/pklxkd-59/?versions&max-keys=999 2023-05-31T20:44:48.394 DEBUG:boto:Method: GET 2023-05-31T20:44:48.395 DEBUG:boto:Path: /pklxkd-59/?versions&max-keys=999 2023-05-31T20:44:48.395 DEBUG:boto:Data: 2023-05-31T20:44:48.395 DEBUG:boto:Headers: {} 2023-05-31T20:44:48.396 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:44:48.396 DEBUG:boto:Port: 8001 2023-05-31T20:44:48.396 DEBUG:boto:Params: {} 2023-05-31T20:44:48.397 DEBUG:boto:Token: None 2023-05-31T20:44:48.397 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:44:48 GMT /pklxkd-59/?versions 2023-05-31T20:44:48.397 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:w7D7AcmqfYnM9RzHQZrn2KygD/Q= 2023-05-31T20:44:48.398 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:44:48 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:w7D7AcmqfYnM9RzHQZrn2KygD/Q=', 'Content-Length': '0'} 2023-05-31T20:44:48.400 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a35ce89743e33567-006477b1c0-1c17-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:44:48 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:44:48.401 DEBUG:boto:b'pklxkd-59999falseobj11nulltrue2023-05-31T20:39:26.427Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj17nulltrue2023-05-31T20:43:07.977Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj22nulltrue2023-05-31T20:39:26.473Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj33nulltrue2023-05-31T20:39:26.534Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj34nulltrue2023-05-31T20:43:07.994Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj44nulltrue2023-05-31T20:39:26.565Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj51nulltrue2023-05-31T20:43:08.011Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj55nulltrue2023-05-31T20:39:26.630Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj68nulltrue2023-05-31T20:43:08.066Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj85nulltrue2023-05-31T20:43:08.086Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:44:48.402 DEBUG:tasks.rgw_multi.zone_rados:o=obj11 2023-05-31T20:44:48.402 DEBUG:tasks.rgw_multi.zone_rados:o=obj17 2023-05-31T20:44:48.403 DEBUG:tasks.rgw_multi.zone_rados:o=obj22 2023-05-31T20:44:48.403 DEBUG:tasks.rgw_multi.zone_rados:o=obj33 2023-05-31T20:44:48.403 DEBUG:tasks.rgw_multi.zone_rados:o=obj34 2023-05-31T20:44:48.404 DEBUG:tasks.rgw_multi.zone_rados:o=obj44 2023-05-31T20:44:48.404 DEBUG:tasks.rgw_multi.zone_rados:o=obj51 2023-05-31T20:44:48.404 DEBUG:tasks.rgw_multi.zone_rados:o=obj55 2023-05-31T20:44:48.404 DEBUG:tasks.rgw_multi.zone_rados:o=obj68 2023-05-31T20:44:48.405 DEBUG:tasks.rgw_multi.zone_rados:o=obj85 2023-05-31T20:44:48.405 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2023-05-31T20:44:48.405 DEBUG:boto:path=/pklxkd-59/ 2023-05-31T20:44:48.406 DEBUG:boto:auth_path=/pklxkd-59/ 2023-05-31T20:44:48.406 DEBUG:boto:path=/pklxkd-59/?versions&max-keys=999 2023-05-31T20:44:48.407 DEBUG:boto:auth_path=/pklxkd-59/?versions&max-keys=999 2023-05-31T20:44:48.407 DEBUG:boto:Method: GET 2023-05-31T20:44:48.407 DEBUG:boto:Path: /pklxkd-59/?versions&max-keys=999 2023-05-31T20:44:48.408 DEBUG:boto:Data: 2023-05-31T20:44:48.408 DEBUG:boto:Headers: {} 2023-05-31T20:44:48.408 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:44:48.409 DEBUG:boto:Port: 8000 2023-05-31T20:44:48.409 DEBUG:boto:Params: {} 2023-05-31T20:44:48.409 DEBUG:boto:Token: None 2023-05-31T20:44:48.410 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:44:48 GMT /pklxkd-59/?versions 2023-05-31T20:44:48.410 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:w7D7AcmqfYnM9RzHQZrn2KygD/Q= 2023-05-31T20:44:48.411 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:44:48 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:w7D7AcmqfYnM9RzHQZrn2KygD/Q=', 'Content-Length': '0'} 2023-05-31T20:44:48.413 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000091a0a64c4d8b9888-006477b1c0-1630-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:44:48 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:44:48.414 DEBUG:boto:b'pklxkd-59999falseobj11nulltrue2023-05-31T20:39:26.427Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj17nulltrue2023-05-31T20:43:07.977Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj22nulltrue2023-05-31T20:39:26.473Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj33nulltrue2023-05-31T20:39:26.534Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj34nulltrue2023-05-31T20:43:07.994Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj44nulltrue2023-05-31T20:39:26.565Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj51nulltrue2023-05-31T20:43:08.011Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj55nulltrue2023-05-31T20:39:26.630Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj68nulltrue2023-05-31T20:43:08.066Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj85nulltrue2023-05-31T20:43:08.086Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:44:48.415 DEBUG:tasks.rgw_multi.zone_rados:o=obj11 2023-05-31T20:44:48.415 DEBUG:tasks.rgw_multi.zone_rados:o=obj17 2023-05-31T20:44:48.415 DEBUG:tasks.rgw_multi.zone_rados:o=obj22 2023-05-31T20:44:48.416 DEBUG:tasks.rgw_multi.zone_rados:o=obj33 2023-05-31T20:44:48.416 DEBUG:tasks.rgw_multi.zone_rados:o=obj34 2023-05-31T20:44:48.416 DEBUG:tasks.rgw_multi.zone_rados:o=obj44 2023-05-31T20:44:48.417 DEBUG:tasks.rgw_multi.zone_rados:o=obj51 2023-05-31T20:44:48.417 DEBUG:tasks.rgw_multi.zone_rados:o=obj55 2023-05-31T20:44:48.417 DEBUG:tasks.rgw_multi.zone_rados:o=obj68 2023-05-31T20:44:48.418 DEBUG:tasks.rgw_multi.zone_rados:o=obj85 2023-05-31T20:44:48.418 DEBUG:boto:path=/pklxkd-59/ 2023-05-31T20:44:48.418 DEBUG:boto:auth_path=/pklxkd-59/ 2023-05-31T20:44:48.419 DEBUG:boto:path=/pklxkd-59/?versions&max-keys=999 2023-05-31T20:44:48.419 DEBUG:boto:auth_path=/pklxkd-59/?versions&max-keys=999 2023-05-31T20:44:48.419 DEBUG:boto:Method: GET 2023-05-31T20:44:48.420 DEBUG:boto:Path: /pklxkd-59/?versions&max-keys=999 2023-05-31T20:44:48.420 DEBUG:boto:Data: 2023-05-31T20:44:48.420 DEBUG:boto:Headers: {} 2023-05-31T20:44:48.421 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:44:48.421 DEBUG:boto:Port: 8001 2023-05-31T20:44:48.421 DEBUG:boto:Params: {} 2023-05-31T20:44:48.422 DEBUG:boto:Token: None 2023-05-31T20:44:48.422 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:44:48 GMT /pklxkd-59/?versions 2023-05-31T20:44:48.423 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:w7D7AcmqfYnM9RzHQZrn2KygD/Q= 2023-05-31T20:44:48.423 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:44:48 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:w7D7AcmqfYnM9RzHQZrn2KygD/Q=', 'Content-Length': '0'} 2023-05-31T20:44:48.425 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000072b8315e6a835ebb-006477b1c0-1c17-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:44:48 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:44:48.426 DEBUG:boto:b'pklxkd-59999falseobj11nulltrue2023-05-31T20:39:26.427Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj17nulltrue2023-05-31T20:43:07.977Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj22nulltrue2023-05-31T20:39:26.473Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj33nulltrue2023-05-31T20:39:26.534Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj34nulltrue2023-05-31T20:43:07.994Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj44nulltrue2023-05-31T20:39:26.565Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj51nulltrue2023-05-31T20:43:08.011Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj55nulltrue2023-05-31T20:39:26.630Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj68nulltrue2023-05-31T20:43:08.066Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj85nulltrue2023-05-31T20:43:08.086Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:44:48.427 DEBUG:boto:path=/pklxkd-59/ 2023-05-31T20:44:48.427 DEBUG:boto:auth_path=/pklxkd-59/ 2023-05-31T20:44:48.428 DEBUG:boto:path=/pklxkd-59/?versions&max-keys=999 2023-05-31T20:44:48.428 DEBUG:boto:auth_path=/pklxkd-59/?versions&max-keys=999 2023-05-31T20:44:48.428 DEBUG:boto:Method: GET 2023-05-31T20:44:48.429 DEBUG:boto:Path: /pklxkd-59/?versions&max-keys=999 2023-05-31T20:44:48.429 DEBUG:boto:Data: 2023-05-31T20:44:48.429 DEBUG:boto:Headers: {} 2023-05-31T20:44:48.430 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:44:48.430 DEBUG:boto:Port: 8000 2023-05-31T20:44:48.430 DEBUG:boto:Params: {} 2023-05-31T20:44:48.431 DEBUG:boto:Token: None 2023-05-31T20:44:48.431 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:44:48 GMT /pklxkd-59/?versions 2023-05-31T20:44:48.431 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:w7D7AcmqfYnM9RzHQZrn2KygD/Q= 2023-05-31T20:44:48.432 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:44:48 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:w7D7AcmqfYnM9RzHQZrn2KygD/Q=', 'Content-Length': '0'} 2023-05-31T20:44:48.434 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c86f267550d25ae6-006477b1c0-1630-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:44:48 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:44:48.435 DEBUG:boto:b'pklxkd-59999falseobj11nulltrue2023-05-31T20:39:26.427Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj17nulltrue2023-05-31T20:43:07.977Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj22nulltrue2023-05-31T20:39:26.473Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj33nulltrue2023-05-31T20:39:26.534Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj34nulltrue2023-05-31T20:43:07.994Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj44nulltrue2023-05-31T20:39:26.565Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj51nulltrue2023-05-31T20:43:08.011Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj55nulltrue2023-05-31T20:39:26.630Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj68nulltrue2023-05-31T20:43:08.066Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj85nulltrue2023-05-31T20:43:08.086Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:44:48.436 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj11 2023-05-31T20:44:48.437 DEBUG:boto:path=/pklxkd-59/obj11 2023-05-31T20:44:48.437 DEBUG:boto:auth_path=/pklxkd-59/obj11 2023-05-31T20:44:48.437 DEBUG:boto:path=/pklxkd-59/obj11?versionId=null 2023-05-31T20:44:48.438 DEBUG:boto:auth_path=/pklxkd-59/obj11?versionId=null 2023-05-31T20:44:48.438 DEBUG:boto:Method: GET 2023-05-31T20:44:48.438 DEBUG:boto:Path: /pklxkd-59/obj11?versionId=null 2023-05-31T20:44:48.439 DEBUG:boto:Data: 2023-05-31T20:44:48.439 DEBUG:boto:Headers: {} 2023-05-31T20:44:48.439 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:44:48.440 DEBUG:boto:Port: 8001 2023-05-31T20:44:48.440 DEBUG:boto:Params: {} 2023-05-31T20:44:48.441 DEBUG:boto:Token: None 2023-05-31T20:44:48.441 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:44:48 GMT /pklxkd-59/obj11?versionId=null 2023-05-31T20:44:48.442 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:4m+40qB7zWnOsd+bMzsi0CQX+Wc= 2023-05-31T20:44:48.442 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:44:48 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:4m+40qB7zWnOsd+bMzsi0CQX+Wc=', 'Content-Length': '0'} 2023-05-31T20:44:48.444 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:39:26 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ab74a03b3db73f1c-006477b1c0-1c17-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:44:48 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:44:48.445 DEBUG:boto:path=/pklxkd-59/obj11 2023-05-31T20:44:48.445 DEBUG:boto:auth_path=/pklxkd-59/obj11 2023-05-31T20:44:48.445 DEBUG:boto:path=/pklxkd-59/obj11?versionId=null 2023-05-31T20:44:48.446 DEBUG:boto:auth_path=/pklxkd-59/obj11?versionId=null 2023-05-31T20:44:48.446 DEBUG:boto:Method: GET 2023-05-31T20:44:48.446 DEBUG:boto:Path: /pklxkd-59/obj11?versionId=null 2023-05-31T20:44:48.447 DEBUG:boto:Data: 2023-05-31T20:44:48.447 DEBUG:boto:Headers: {} 2023-05-31T20:44:48.447 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:44:48.448 DEBUG:boto:Port: 8000 2023-05-31T20:44:48.448 DEBUG:boto:Params: {} 2023-05-31T20:44:48.449 DEBUG:boto:Token: None 2023-05-31T20:44:48.449 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:44:48 GMT /pklxkd-59/obj11?versionId=null 2023-05-31T20:44:48.449 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:4m+40qB7zWnOsd+bMzsi0CQX+Wc= 2023-05-31T20:44:48.450 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:44:48 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:4m+40qB7zWnOsd+bMzsi0CQX+Wc=', 'Content-Length': '0'} 2023-05-31T20:44:48.452 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:39:26 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000fe098d0b51f37e89-006477b1c0-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:44:48 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:44:48.453 DEBUG:boto:path=/pklxkd-59/obj11 2023-05-31T20:44:48.454 DEBUG:boto:auth_path=/pklxkd-59/obj11 2023-05-31T20:44:48.454 DEBUG:boto:path=/pklxkd-59/obj11?versionId=null 2023-05-31T20:44:48.455 DEBUG:boto:auth_path=/pklxkd-59/obj11?versionId=null 2023-05-31T20:44:48.455 DEBUG:boto:Method: HEAD 2023-05-31T20:44:48.456 DEBUG:boto:Path: /pklxkd-59/obj11?versionId=null 2023-05-31T20:44:48.456 DEBUG:boto:Data: 2023-05-31T20:44:48.456 DEBUG:boto:Headers: {} 2023-05-31T20:44:48.457 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:44:48.457 DEBUG:boto:Port: 8001 2023-05-31T20:44:48.457 DEBUG:boto:Params: {} 2023-05-31T20:44:48.458 DEBUG:boto:Token: None 2023-05-31T20:44:48.458 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:44:48 GMT /pklxkd-59/obj11?versionId=null 2023-05-31T20:44:48.459 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:G4NMFQFHBXvqNPnYVeZYtkaKZ+M= 2023-05-31T20:44:48.459 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:44:48 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:G4NMFQFHBXvqNPnYVeZYtkaKZ+M=', 'Content-Length': '0'} 2023-05-31T20:44:48.460 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:39:26 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f8e57974df7fcc42-006477b1c0-1c17-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:44:48 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:44:48.461 DEBUG:boto:path=/pklxkd-59/obj11 2023-05-31T20:44:48.461 DEBUG:boto:auth_path=/pklxkd-59/obj11 2023-05-31T20:44:48.462 DEBUG:boto:path=/pklxkd-59/obj11?versionId=null 2023-05-31T20:44:48.462 DEBUG:boto:auth_path=/pklxkd-59/obj11?versionId=null 2023-05-31T20:44:48.462 DEBUG:boto:Method: HEAD 2023-05-31T20:44:48.463 DEBUG:boto:Path: /pklxkd-59/obj11?versionId=null 2023-05-31T20:44:48.463 DEBUG:boto:Data: 2023-05-31T20:44:48.463 DEBUG:boto:Headers: {} 2023-05-31T20:44:48.464 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:44:48.464 DEBUG:boto:Port: 8000 2023-05-31T20:44:48.464 DEBUG:boto:Params: {} 2023-05-31T20:44:48.465 DEBUG:boto:Token: None 2023-05-31T20:44:48.465 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:44:48 GMT /pklxkd-59/obj11?versionId=null 2023-05-31T20:44:48.466 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:G4NMFQFHBXvqNPnYVeZYtkaKZ+M= 2023-05-31T20:44:48.466 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:44:48 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:G4NMFQFHBXvqNPnYVeZYtkaKZ+M=', 'Content-Length': '0'} 2023-05-31T20:44:48.468 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:39:26 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006a4da15cf404f12a-006477b1c0-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:44:48 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:44:48.468 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj11 2023-05-31T20:44:48.469 DEBUG:boto:path=/pklxkd-59/obj11 2023-05-31T20:44:48.469 DEBUG:boto:auth_path=/pklxkd-59/obj11 2023-05-31T20:44:48.469 DEBUG:boto:path=/pklxkd-59/obj11?versionId=null 2023-05-31T20:44:48.470 DEBUG:boto:auth_path=/pklxkd-59/obj11?versionId=null 2023-05-31T20:44:48.470 DEBUG:boto:Method: GET 2023-05-31T20:44:48.470 DEBUG:boto:Path: /pklxkd-59/obj11?versionId=null 2023-05-31T20:44:48.471 DEBUG:boto:Data: 2023-05-31T20:44:48.471 DEBUG:boto:Headers: {} 2023-05-31T20:44:48.471 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:44:48.472 DEBUG:boto:Port: 8001 2023-05-31T20:44:48.472 DEBUG:boto:Params: {} 2023-05-31T20:44:48.473 DEBUG:boto:Token: None 2023-05-31T20:44:48.473 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:44:48 GMT /pklxkd-59/obj11?versionId=null 2023-05-31T20:44:48.473 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:4m+40qB7zWnOsd+bMzsi0CQX+Wc= 2023-05-31T20:44:48.474 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:44:48 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:4m+40qB7zWnOsd+bMzsi0CQX+Wc=', 'Content-Length': '0'} 2023-05-31T20:44:48.476 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:39:26 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000454c71efa667cb68-006477b1c0-1c17-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:44:48 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:44:48.516 DEBUG:boto:path=/pklxkd-59/obj11 2023-05-31T20:44:48.517 DEBUG:boto:auth_path=/pklxkd-59/obj11 2023-05-31T20:44:48.517 DEBUG:boto:path=/pklxkd-59/obj11?versionId=null 2023-05-31T20:44:48.517 DEBUG:boto:auth_path=/pklxkd-59/obj11?versionId=null 2023-05-31T20:44:48.518 DEBUG:boto:Method: GET 2023-05-31T20:44:48.518 DEBUG:boto:Path: /pklxkd-59/obj11?versionId=null 2023-05-31T20:44:48.519 DEBUG:boto:Data: 2023-05-31T20:44:48.519 DEBUG:boto:Headers: {} 2023-05-31T20:44:48.519 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:44:48.520 DEBUG:boto:Port: 8000 2023-05-31T20:44:48.520 DEBUG:boto:Params: {} 2023-05-31T20:44:48.520 DEBUG:boto:Token: None 2023-05-31T20:44:48.521 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:44:48 GMT /pklxkd-59/obj11?versionId=null 2023-05-31T20:44:48.521 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:4m+40qB7zWnOsd+bMzsi0CQX+Wc= 2023-05-31T20:44:48.522 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:44:48 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:4m+40qB7zWnOsd+bMzsi0CQX+Wc=', 'Content-Length': '0'} 2023-05-31T20:44:48.523 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:39:26 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f38820cb0e25279d-006477b1c0-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:44:48 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:44:48.524 DEBUG:boto:path=/pklxkd-59/ 2023-05-31T20:44:48.524 DEBUG:boto:auth_path=/pklxkd-59/ 2023-05-31T20:44:48.525 DEBUG:boto:path=/pklxkd-59/?prefix=obj11 2023-05-31T20:44:48.525 DEBUG:boto:auth_path=/pklxkd-59/?prefix=obj11 2023-05-31T20:44:48.526 DEBUG:boto:Method: GET 2023-05-31T20:44:48.526 DEBUG:boto:Path: /pklxkd-59/?prefix=obj11 2023-05-31T20:44:48.526 DEBUG:boto:Data: 2023-05-31T20:44:48.527 DEBUG:boto:Headers: {} 2023-05-31T20:44:48.527 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:44:48.527 DEBUG:boto:Port: 8001 2023-05-31T20:44:48.528 DEBUG:boto:Params: {} 2023-05-31T20:44:48.528 DEBUG:boto:Token: None 2023-05-31T20:44:48.529 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:44:48 GMT /pklxkd-59/ 2023-05-31T20:44:48.529 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:717u7lUCeWRZyRakKJN50YyqEZ8= 2023-05-31T20:44:48.529 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:44:48 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:717u7lUCeWRZyRakKJN50YyqEZ8=', 'Content-Length': '0'} 2023-05-31T20:44:48.532 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000008ecb1809b4042d41-006477b1c0-1c17-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:44:48 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:44:48.532 DEBUG:boto:b'pklxkd-59obj111000falseobj112023-05-31T20:39:26.427Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:44:48.533 DEBUG:boto:path=/pklxkd-59/ 2023-05-31T20:44:48.533 DEBUG:boto:auth_path=/pklxkd-59/ 2023-05-31T20:44:48.534 DEBUG:boto:path=/pklxkd-59/?prefix=obj11 2023-05-31T20:44:48.534 DEBUG:boto:auth_path=/pklxkd-59/?prefix=obj11 2023-05-31T20:44:48.535 DEBUG:boto:Method: GET 2023-05-31T20:44:48.535 DEBUG:boto:Path: /pklxkd-59/?prefix=obj11 2023-05-31T20:44:48.535 DEBUG:boto:Data: 2023-05-31T20:44:48.536 DEBUG:boto:Headers: {} 2023-05-31T20:44:48.536 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:44:48.536 DEBUG:boto:Port: 8000 2023-05-31T20:44:48.537 DEBUG:boto:Params: {} 2023-05-31T20:44:48.537 DEBUG:boto:Token: None 2023-05-31T20:44:48.538 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:44:48 GMT /pklxkd-59/ 2023-05-31T20:44:48.538 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:717u7lUCeWRZyRakKJN50YyqEZ8= 2023-05-31T20:44:48.539 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:44:48 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:717u7lUCeWRZyRakKJN50YyqEZ8=', 'Content-Length': '0'} 2023-05-31T20:44:48.541 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e0ec7e67090ef79f-006477b1c0-1630-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:44:48 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:44:48.542 DEBUG:boto:b'pklxkd-59obj111000falseobj112023-05-31T20:39:26.427Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:44:48.543 DEBUG:boto:path=/pklxkd-59/obj11 2023-05-31T20:44:48.543 DEBUG:boto:auth_path=/pklxkd-59/obj11 2023-05-31T20:44:48.543 DEBUG:boto:Method: HEAD 2023-05-31T20:44:48.544 DEBUG:boto:Path: /pklxkd-59/obj11 2023-05-31T20:44:48.544 DEBUG:boto:Data: 2023-05-31T20:44:48.544 DEBUG:boto:Headers: {} 2023-05-31T20:44:48.545 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:44:48.545 DEBUG:boto:Port: 8001 2023-05-31T20:44:48.546 DEBUG:boto:Params: {} 2023-05-31T20:44:48.546 DEBUG:boto:Token: None 2023-05-31T20:44:48.546 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:44:48 GMT /pklxkd-59/obj11 2023-05-31T20:44:48.547 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:qFRTo9B/9g/XwpfA0ol/tB0BLHE= 2023-05-31T20:44:48.547 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:44:48 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:qFRTo9B/9g/XwpfA0ol/tB0BLHE=', 'Content-Length': '0'} 2023-05-31T20:44:48.549 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:39:26 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f625f2c412364746-006477b1c0-1c17-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:44:48 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:44:48.549 DEBUG:boto:path=/pklxkd-59/obj11 2023-05-31T20:44:48.550 DEBUG:boto:auth_path=/pklxkd-59/obj11 2023-05-31T20:44:48.550 DEBUG:boto:Method: HEAD 2023-05-31T20:44:48.550 DEBUG:boto:Path: /pklxkd-59/obj11 2023-05-31T20:44:48.551 DEBUG:boto:Data: 2023-05-31T20:44:48.551 DEBUG:boto:Headers: {} 2023-05-31T20:44:48.551 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:44:48.552 DEBUG:boto:Port: 8000 2023-05-31T20:44:48.552 DEBUG:boto:Params: {} 2023-05-31T20:44:48.552 DEBUG:boto:Token: None 2023-05-31T20:44:48.553 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:44:48 GMT /pklxkd-59/obj11 2023-05-31T20:44:48.553 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:qFRTo9B/9g/XwpfA0ol/tB0BLHE= 2023-05-31T20:44:48.554 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:44:48 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:qFRTo9B/9g/XwpfA0ol/tB0BLHE=', 'Content-Length': '0'} 2023-05-31T20:44:48.555 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:39:26 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d20375fd7a643baf-006477b1c0-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:44:48 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:44:48.556 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj11 2023-05-31T20:44:48.556 DEBUG:boto:path=/pklxkd-59/obj11 2023-05-31T20:44:48.557 DEBUG:boto:auth_path=/pklxkd-59/obj11 2023-05-31T20:44:48.557 DEBUG:boto:Method: GET 2023-05-31T20:44:48.557 DEBUG:boto:Path: /pklxkd-59/obj11 2023-05-31T20:44:48.558 DEBUG:boto:Data: 2023-05-31T20:44:48.558 DEBUG:boto:Headers: {} 2023-05-31T20:44:48.558 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:44:48.559 DEBUG:boto:Port: 8001 2023-05-31T20:44:48.559 DEBUG:boto:Params: {} 2023-05-31T20:44:48.560 DEBUG:boto:Token: None 2023-05-31T20:44:48.560 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:44:48 GMT /pklxkd-59/obj11 2023-05-31T20:44:48.560 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:5G41qTtW0icUwqtFvPp7Y2M/T3M= 2023-05-31T20:44:48.561 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:44:48 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:5G41qTtW0icUwqtFvPp7Y2M/T3M=', 'Content-Length': '0'} 2023-05-31T20:44:48.562 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:39:26 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000752e1a0b76a92a43-006477b1c0-1c17-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:44:48 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:44:48.563 DEBUG:boto:path=/pklxkd-59/obj11 2023-05-31T20:44:48.564 DEBUG:boto:auth_path=/pklxkd-59/obj11 2023-05-31T20:44:48.564 DEBUG:boto:Method: GET 2023-05-31T20:44:48.564 DEBUG:boto:Path: /pklxkd-59/obj11 2023-05-31T20:44:48.565 DEBUG:boto:Data: 2023-05-31T20:44:48.566 DEBUG:boto:Headers: {} 2023-05-31T20:44:48.566 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:44:48.566 DEBUG:boto:Port: 8000 2023-05-31T20:44:48.567 DEBUG:boto:Params: {} 2023-05-31T20:44:48.568 DEBUG:boto:Token: None 2023-05-31T20:44:48.568 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:44:48 GMT /pklxkd-59/obj11 2023-05-31T20:44:48.569 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:5G41qTtW0icUwqtFvPp7Y2M/T3M= 2023-05-31T20:44:48.570 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:44:48 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:5G41qTtW0icUwqtFvPp7Y2M/T3M=', 'Content-Length': '0'} 2023-05-31T20:44:48.572 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:39:26 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000093b4b42aaa7036d5-006477b1c0-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:44:48 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:44:48.612 DEBUG:boto:path=/pklxkd-59/ 2023-05-31T20:44:48.613 DEBUG:boto:auth_path=/pklxkd-59/ 2023-05-31T20:44:48.613 DEBUG:boto:path=/pklxkd-59/?prefix=obj11 2023-05-31T20:44:48.614 DEBUG:boto:auth_path=/pklxkd-59/?prefix=obj11 2023-05-31T20:44:48.615 DEBUG:boto:Method: GET 2023-05-31T20:44:48.615 DEBUG:boto:Path: /pklxkd-59/?prefix=obj11 2023-05-31T20:44:48.615 DEBUG:boto:Data: 2023-05-31T20:44:48.616 DEBUG:boto:Headers: {} 2023-05-31T20:44:48.616 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:44:48.616 DEBUG:boto:Port: 8001 2023-05-31T20:44:48.617 DEBUG:boto:Params: {} 2023-05-31T20:44:48.617 DEBUG:boto:Token: None 2023-05-31T20:44:48.618 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:44:48 GMT /pklxkd-59/ 2023-05-31T20:44:48.618 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:717u7lUCeWRZyRakKJN50YyqEZ8= 2023-05-31T20:44:48.618 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:44:48 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:717u7lUCeWRZyRakKJN50YyqEZ8=', 'Content-Length': '0'} 2023-05-31T20:44:48.621 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000063eb25b53646340d-006477b1c0-1c17-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:44:48 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:44:48.622 DEBUG:boto:b'pklxkd-59obj111000falseobj112023-05-31T20:39:26.427Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:44:48.622 DEBUG:boto:path=/pklxkd-59/ 2023-05-31T20:44:48.623 DEBUG:boto:auth_path=/pklxkd-59/ 2023-05-31T20:44:48.623 DEBUG:boto:path=/pklxkd-59/?prefix=obj11 2023-05-31T20:44:48.623 DEBUG:boto:auth_path=/pklxkd-59/?prefix=obj11 2023-05-31T20:44:48.624 DEBUG:boto:Method: GET 2023-05-31T20:44:48.624 DEBUG:boto:Path: /pklxkd-59/?prefix=obj11 2023-05-31T20:44:48.625 DEBUG:boto:Data: 2023-05-31T20:44:48.625 DEBUG:boto:Headers: {} 2023-05-31T20:44:48.625 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:44:48.626 DEBUG:boto:Port: 8000 2023-05-31T20:44:48.626 DEBUG:boto:Params: {} 2023-05-31T20:44:48.627 DEBUG:boto:Token: None 2023-05-31T20:44:48.627 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:44:48 GMT /pklxkd-59/ 2023-05-31T20:44:48.628 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:717u7lUCeWRZyRakKJN50YyqEZ8= 2023-05-31T20:44:48.628 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:44:48 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:717u7lUCeWRZyRakKJN50YyqEZ8=', 'Content-Length': '0'} 2023-05-31T20:44:48.631 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000038b134a1112856fc-006477b1c0-1630-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:44:48 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:44:48.632 DEBUG:boto:b'pklxkd-59obj111000falseobj112023-05-31T20:39:26.427Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:44:48.632 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj17 2023-05-31T20:44:48.633 DEBUG:boto:path=/pklxkd-59/obj17 2023-05-31T20:44:48.633 DEBUG:boto:auth_path=/pklxkd-59/obj17 2023-05-31T20:44:48.633 DEBUG:boto:path=/pklxkd-59/obj17?versionId=null 2023-05-31T20:44:48.634 DEBUG:boto:auth_path=/pklxkd-59/obj17?versionId=null 2023-05-31T20:44:48.634 DEBUG:boto:Method: GET 2023-05-31T20:44:48.634 DEBUG:boto:Path: /pklxkd-59/obj17?versionId=null 2023-05-31T20:44:48.635 DEBUG:boto:Data: 2023-05-31T20:44:48.635 DEBUG:boto:Headers: {} 2023-05-31T20:44:48.636 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:44:48.636 DEBUG:boto:Port: 8001 2023-05-31T20:44:48.636 DEBUG:boto:Params: {} 2023-05-31T20:44:48.637 DEBUG:boto:Token: None 2023-05-31T20:44:48.637 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:44:48 GMT /pklxkd-59/obj17?versionId=null 2023-05-31T20:44:48.638 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:8Rb8zS3gfBo+4EAXQQDuffTDZq8= 2023-05-31T20:44:48.639 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:44:48 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:8Rb8zS3gfBo+4EAXQQDuffTDZq8=', 'Content-Length': '0'} 2023-05-31T20:44:48.641 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:43:07 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ee86191c11a05ce4-006477b1c0-1c17-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:44:48 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:44:48.684 DEBUG:boto:path=/pklxkd-59/obj17 2023-05-31T20:44:48.685 DEBUG:boto:auth_path=/pklxkd-59/obj17 2023-05-31T20:44:48.685 DEBUG:boto:path=/pklxkd-59/obj17?versionId=null 2023-05-31T20:44:48.686 DEBUG:boto:auth_path=/pklxkd-59/obj17?versionId=null 2023-05-31T20:44:48.686 DEBUG:boto:Method: GET 2023-05-31T20:44:48.686 DEBUG:boto:Path: /pklxkd-59/obj17?versionId=null 2023-05-31T20:44:48.687 DEBUG:boto:Data: 2023-05-31T20:44:48.687 DEBUG:boto:Headers: {} 2023-05-31T20:44:48.688 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:44:48.688 DEBUG:boto:Port: 8000 2023-05-31T20:44:48.688 DEBUG:boto:Params: {} 2023-05-31T20:44:48.689 DEBUG:boto:Token: None 2023-05-31T20:44:48.689 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:44:48 GMT /pklxkd-59/obj17?versionId=null 2023-05-31T20:44:48.690 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:8Rb8zS3gfBo+4EAXQQDuffTDZq8= 2023-05-31T20:44:48.690 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:44:48 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:8Rb8zS3gfBo+4EAXQQDuffTDZq8=', 'Content-Length': '0'} 2023-05-31T20:44:48.692 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:43:07 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000850f0df9aa830aa5-006477b1c0-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:44:48 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:44:48.693 DEBUG:boto:path=/pklxkd-59/obj17 2023-05-31T20:44:48.693 DEBUG:boto:auth_path=/pklxkd-59/obj17 2023-05-31T20:44:48.694 DEBUG:boto:path=/pklxkd-59/obj17?versionId=null 2023-05-31T20:44:48.694 DEBUG:boto:auth_path=/pklxkd-59/obj17?versionId=null 2023-05-31T20:44:48.694 DEBUG:boto:Method: HEAD 2023-05-31T20:44:48.695 DEBUG:boto:Path: /pklxkd-59/obj17?versionId=null 2023-05-31T20:44:48.695 DEBUG:boto:Data: 2023-05-31T20:44:48.695 DEBUG:boto:Headers: {} 2023-05-31T20:44:48.696 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:44:48.696 DEBUG:boto:Port: 8001 2023-05-31T20:44:48.696 DEBUG:boto:Params: {} 2023-05-31T20:44:48.697 DEBUG:boto:Token: None 2023-05-31T20:44:48.697 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:44:48 GMT /pklxkd-59/obj17?versionId=null 2023-05-31T20:44:48.698 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:lKTPZxlCT4GB5wufvufDq0pxfKc= 2023-05-31T20:44:48.698 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:44:48 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:lKTPZxlCT4GB5wufvufDq0pxfKc=', 'Content-Length': '0'} 2023-05-31T20:44:48.700 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:43:07 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000054247360f99559f1-006477b1c0-1c17-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:44:48 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:44:48.701 DEBUG:boto:path=/pklxkd-59/obj17 2023-05-31T20:44:48.701 DEBUG:boto:auth_path=/pklxkd-59/obj17 2023-05-31T20:44:48.701 DEBUG:boto:path=/pklxkd-59/obj17?versionId=null 2023-05-31T20:44:48.702 DEBUG:boto:auth_path=/pklxkd-59/obj17?versionId=null 2023-05-31T20:44:48.702 DEBUG:boto:Method: HEAD 2023-05-31T20:44:48.702 DEBUG:boto:Path: /pklxkd-59/obj17?versionId=null 2023-05-31T20:44:48.703 DEBUG:boto:Data: 2023-05-31T20:44:48.703 DEBUG:boto:Headers: {} 2023-05-31T20:44:48.703 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:44:48.704 DEBUG:boto:Port: 8000 2023-05-31T20:44:48.704 DEBUG:boto:Params: {} 2023-05-31T20:44:48.704 DEBUG:boto:Token: None 2023-05-31T20:44:48.705 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:44:48 GMT /pklxkd-59/obj17?versionId=null 2023-05-31T20:44:48.705 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:lKTPZxlCT4GB5wufvufDq0pxfKc= 2023-05-31T20:44:48.705 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:44:48 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:lKTPZxlCT4GB5wufvufDq0pxfKc=', 'Content-Length': '0'} 2023-05-31T20:44:48.707 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:43:07 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c19739a6d83a34b0-006477b1c0-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:44:48 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:44:48.708 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj17 2023-05-31T20:44:48.709 DEBUG:boto:path=/pklxkd-59/obj17 2023-05-31T20:44:48.709 DEBUG:boto:auth_path=/pklxkd-59/obj17 2023-05-31T20:44:48.709 DEBUG:boto:path=/pklxkd-59/obj17?versionId=null 2023-05-31T20:44:48.710 DEBUG:boto:auth_path=/pklxkd-59/obj17?versionId=null 2023-05-31T20:44:48.710 DEBUG:boto:Method: GET 2023-05-31T20:44:48.711 DEBUG:boto:Path: /pklxkd-59/obj17?versionId=null 2023-05-31T20:44:48.711 DEBUG:boto:Data: 2023-05-31T20:44:48.712 DEBUG:boto:Headers: {} 2023-05-31T20:44:48.712 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:44:48.712 DEBUG:boto:Port: 8001 2023-05-31T20:44:48.713 DEBUG:boto:Params: {} 2023-05-31T20:44:48.713 DEBUG:boto:Token: None 2023-05-31T20:44:48.714 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:44:48 GMT /pklxkd-59/obj17?versionId=null 2023-05-31T20:44:48.714 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:8Rb8zS3gfBo+4EAXQQDuffTDZq8= 2023-05-31T20:44:48.714 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:44:48 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:8Rb8zS3gfBo+4EAXQQDuffTDZq8=', 'Content-Length': '0'} 2023-05-31T20:44:48.716 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:43:07 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000cae46fc38b3cb99d-006477b1c0-1c17-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:44:48 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:44:48.717 DEBUG:boto:path=/pklxkd-59/obj17 2023-05-31T20:44:48.717 DEBUG:boto:auth_path=/pklxkd-59/obj17 2023-05-31T20:44:48.718 DEBUG:boto:path=/pklxkd-59/obj17?versionId=null 2023-05-31T20:44:48.718 DEBUG:boto:auth_path=/pklxkd-59/obj17?versionId=null 2023-05-31T20:44:48.718 DEBUG:boto:Method: GET 2023-05-31T20:44:48.719 DEBUG:boto:Path: /pklxkd-59/obj17?versionId=null 2023-05-31T20:44:48.719 DEBUG:boto:Data: 2023-05-31T20:44:48.719 DEBUG:boto:Headers: {} 2023-05-31T20:44:48.720 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:44:48.720 DEBUG:boto:Port: 8000 2023-05-31T20:44:48.721 DEBUG:boto:Params: {} 2023-05-31T20:44:48.721 DEBUG:boto:Token: None 2023-05-31T20:44:48.722 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:44:48 GMT /pklxkd-59/obj17?versionId=null 2023-05-31T20:44:48.722 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:8Rb8zS3gfBo+4EAXQQDuffTDZq8= 2023-05-31T20:44:48.722 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:44:48 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:8Rb8zS3gfBo+4EAXQQDuffTDZq8=', 'Content-Length': '0'} 2023-05-31T20:44:48.724 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:43:07 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d51b471c8f498ed1-006477b1c0-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:44:48 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:44:48.725 DEBUG:boto:path=/pklxkd-59/ 2023-05-31T20:44:48.725 DEBUG:boto:auth_path=/pklxkd-59/ 2023-05-31T20:44:48.726 DEBUG:boto:path=/pklxkd-59/?prefix=obj17 2023-05-31T20:44:48.726 DEBUG:boto:auth_path=/pklxkd-59/?prefix=obj17 2023-05-31T20:44:48.726 DEBUG:boto:Method: GET 2023-05-31T20:44:48.727 DEBUG:boto:Path: /pklxkd-59/?prefix=obj17 2023-05-31T20:44:48.727 DEBUG:boto:Data: 2023-05-31T20:44:48.727 DEBUG:boto:Headers: {} 2023-05-31T20:44:48.728 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:44:48.728 DEBUG:boto:Port: 8001 2023-05-31T20:44:48.729 DEBUG:boto:Params: {} 2023-05-31T20:44:48.729 DEBUG:boto:Token: None 2023-05-31T20:44:48.729 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:44:48 GMT /pklxkd-59/ 2023-05-31T20:44:48.730 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:717u7lUCeWRZyRakKJN50YyqEZ8= 2023-05-31T20:44:48.730 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:44:48 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:717u7lUCeWRZyRakKJN50YyqEZ8=', 'Content-Length': '0'} 2023-05-31T20:44:48.733 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000086f88ecc34ee00f4-006477b1c0-1c17-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:44:48 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:44:48.733 DEBUG:boto:b'pklxkd-59obj171000falseobj172023-05-31T20:43:07.977Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:44:48.734 DEBUG:boto:path=/pklxkd-59/ 2023-05-31T20:44:48.734 DEBUG:boto:auth_path=/pklxkd-59/ 2023-05-31T20:44:48.735 DEBUG:boto:path=/pklxkd-59/?prefix=obj17 2023-05-31T20:44:48.735 DEBUG:boto:auth_path=/pklxkd-59/?prefix=obj17 2023-05-31T20:44:48.735 DEBUG:boto:Method: GET 2023-05-31T20:44:48.736 DEBUG:boto:Path: /pklxkd-59/?prefix=obj17 2023-05-31T20:44:48.736 DEBUG:boto:Data: 2023-05-31T20:44:48.736 DEBUG:boto:Headers: {} 2023-05-31T20:44:48.737 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:44:48.737 DEBUG:boto:Port: 8000 2023-05-31T20:44:48.738 DEBUG:boto:Params: {} 2023-05-31T20:44:48.738 DEBUG:boto:Token: None 2023-05-31T20:44:48.739 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:44:48 GMT /pklxkd-59/ 2023-05-31T20:44:48.739 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:717u7lUCeWRZyRakKJN50YyqEZ8= 2023-05-31T20:44:48.740 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:44:48 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:717u7lUCeWRZyRakKJN50YyqEZ8=', 'Content-Length': '0'} 2023-05-31T20:44:48.742 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000009218a114d58e6ed-006477b1c0-1630-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:44:48 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:44:48.784 DEBUG:boto:b'pklxkd-59obj171000falseobj172023-05-31T20:43:07.977Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:44:48.785 DEBUG:boto:path=/pklxkd-59/obj17 2023-05-31T20:44:48.785 DEBUG:boto:auth_path=/pklxkd-59/obj17 2023-05-31T20:44:48.786 DEBUG:boto:Method: HEAD 2023-05-31T20:44:48.786 DEBUG:boto:Path: /pklxkd-59/obj17 2023-05-31T20:44:48.786 DEBUG:boto:Data: 2023-05-31T20:44:48.787 DEBUG:boto:Headers: {} 2023-05-31T20:44:48.787 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:44:48.787 DEBUG:boto:Port: 8001 2023-05-31T20:44:48.788 DEBUG:boto:Params: {} 2023-05-31T20:44:48.788 DEBUG:boto:Token: None 2023-05-31T20:44:48.789 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:44:48 GMT /pklxkd-59/obj17 2023-05-31T20:44:48.789 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:VMGUBjW9497WnJKlMC+qk+khtTM= 2023-05-31T20:44:48.789 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:44:48 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:VMGUBjW9497WnJKlMC+qk+khtTM=', 'Content-Length': '0'} 2023-05-31T20:44:48.791 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:43:07 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c4b1aa564588991a-006477b1c0-1c17-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:44:48 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:44:48.792 DEBUG:boto:path=/pklxkd-59/obj17 2023-05-31T20:44:48.792 DEBUG:boto:auth_path=/pklxkd-59/obj17 2023-05-31T20:44:48.793 DEBUG:boto:Method: HEAD 2023-05-31T20:44:48.793 DEBUG:boto:Path: /pklxkd-59/obj17 2023-05-31T20:44:48.794 DEBUG:boto:Data: 2023-05-31T20:44:48.794 DEBUG:boto:Headers: {} 2023-05-31T20:44:48.795 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:44:48.795 DEBUG:boto:Port: 8000 2023-05-31T20:44:48.795 DEBUG:boto:Params: {} 2023-05-31T20:44:48.796 DEBUG:boto:Token: None 2023-05-31T20:44:48.796 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:44:48 GMT /pklxkd-59/obj17 2023-05-31T20:44:48.797 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:VMGUBjW9497WnJKlMC+qk+khtTM= 2023-05-31T20:44:48.797 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:44:48 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:VMGUBjW9497WnJKlMC+qk+khtTM=', 'Content-Length': '0'} 2023-05-31T20:44:48.799 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:43:07 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000062172a37b54e4766-006477b1c0-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:44:48 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:44:48.800 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj17 2023-05-31T20:44:48.800 DEBUG:boto:path=/pklxkd-59/obj17 2023-05-31T20:44:48.801 DEBUG:boto:auth_path=/pklxkd-59/obj17 2023-05-31T20:44:48.801 DEBUG:boto:Method: GET 2023-05-31T20:44:48.801 DEBUG:boto:Path: /pklxkd-59/obj17 2023-05-31T20:44:48.802 DEBUG:boto:Data: 2023-05-31T20:44:48.802 DEBUG:boto:Headers: {} 2023-05-31T20:44:48.802 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:44:48.803 DEBUG:boto:Port: 8001 2023-05-31T20:44:48.803 DEBUG:boto:Params: {} 2023-05-31T20:44:48.804 DEBUG:boto:Token: None 2023-05-31T20:44:48.804 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:44:48 GMT /pklxkd-59/obj17 2023-05-31T20:44:48.805 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:C9i8MMPPXjC+Gkg/aA7D6o+DQc8= 2023-05-31T20:44:48.805 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:44:48 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:C9i8MMPPXjC+Gkg/aA7D6o+DQc8=', 'Content-Length': '0'} 2023-05-31T20:44:48.807 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:43:07 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000044587e490a3528cd-006477b1c0-1c17-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:44:48 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:44:48.807 DEBUG:boto:path=/pklxkd-59/obj17 2023-05-31T20:44:48.808 DEBUG:boto:auth_path=/pklxkd-59/obj17 2023-05-31T20:44:48.808 DEBUG:boto:Method: GET 2023-05-31T20:44:48.809 DEBUG:boto:Path: /pklxkd-59/obj17 2023-05-31T20:44:48.809 DEBUG:boto:Data: 2023-05-31T20:44:48.809 DEBUG:boto:Headers: {} 2023-05-31T20:44:48.810 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:44:48.810 DEBUG:boto:Port: 8000 2023-05-31T20:44:48.810 DEBUG:boto:Params: {} 2023-05-31T20:44:48.811 DEBUG:boto:Token: None 2023-05-31T20:44:48.811 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:44:48 GMT /pklxkd-59/obj17 2023-05-31T20:44:48.812 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:C9i8MMPPXjC+Gkg/aA7D6o+DQc8= 2023-05-31T20:44:48.812 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:44:48 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:C9i8MMPPXjC+Gkg/aA7D6o+DQc8=', 'Content-Length': '0'} 2023-05-31T20:44:48.814 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:43:07 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008723c3675d5686ee-006477b1c0-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:44:48 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:44:48.815 DEBUG:boto:path=/pklxkd-59/ 2023-05-31T20:44:48.815 DEBUG:boto:auth_path=/pklxkd-59/ 2023-05-31T20:44:48.816 DEBUG:boto:path=/pklxkd-59/?prefix=obj17 2023-05-31T20:44:48.816 DEBUG:boto:auth_path=/pklxkd-59/?prefix=obj17 2023-05-31T20:44:48.817 DEBUG:boto:Method: GET 2023-05-31T20:44:48.817 DEBUG:boto:Path: /pklxkd-59/?prefix=obj17 2023-05-31T20:44:48.817 DEBUG:boto:Data: 2023-05-31T20:44:48.818 DEBUG:boto:Headers: {} 2023-05-31T20:44:48.818 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:44:48.818 DEBUG:boto:Port: 8001 2023-05-31T20:44:48.819 DEBUG:boto:Params: {} 2023-05-31T20:44:48.819 DEBUG:boto:Token: None 2023-05-31T20:44:48.820 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:44:48 GMT /pklxkd-59/ 2023-05-31T20:44:48.820 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:717u7lUCeWRZyRakKJN50YyqEZ8= 2023-05-31T20:44:48.820 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:44:48 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:717u7lUCeWRZyRakKJN50YyqEZ8=', 'Content-Length': '0'} 2023-05-31T20:44:48.823 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000007807e86c3e3a783a-006477b1c0-1c17-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:44:48 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:44:48.823 DEBUG:boto:b'pklxkd-59obj171000falseobj172023-05-31T20:43:07.977Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:44:48.824 DEBUG:boto:path=/pklxkd-59/ 2023-05-31T20:44:48.825 DEBUG:boto:auth_path=/pklxkd-59/ 2023-05-31T20:44:48.825 DEBUG:boto:path=/pklxkd-59/?prefix=obj17 2023-05-31T20:44:48.825 DEBUG:boto:auth_path=/pklxkd-59/?prefix=obj17 2023-05-31T20:44:48.826 DEBUG:boto:Method: GET 2023-05-31T20:44:48.826 DEBUG:boto:Path: /pklxkd-59/?prefix=obj17 2023-05-31T20:44:48.827 DEBUG:boto:Data: 2023-05-31T20:44:48.827 DEBUG:boto:Headers: {} 2023-05-31T20:44:48.827 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:44:48.828 DEBUG:boto:Port: 8000 2023-05-31T20:44:48.828 DEBUG:boto:Params: {} 2023-05-31T20:44:48.828 DEBUG:boto:Token: None 2023-05-31T20:44:48.829 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:44:48 GMT /pklxkd-59/ 2023-05-31T20:44:48.829 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:717u7lUCeWRZyRakKJN50YyqEZ8= 2023-05-31T20:44:48.830 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:44:48 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:717u7lUCeWRZyRakKJN50YyqEZ8=', 'Content-Length': '0'} 2023-05-31T20:44:48.832 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000000c5c4dbb4e757761-006477b1c0-1630-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:44:48 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:44:48.833 DEBUG:boto:b'pklxkd-59obj171000falseobj172023-05-31T20:43:07.977Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:44:48.833 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj22 2023-05-31T20:44:48.834 DEBUG:boto:path=/pklxkd-59/obj22 2023-05-31T20:44:48.834 DEBUG:boto:auth_path=/pklxkd-59/obj22 2023-05-31T20:44:48.835 DEBUG:boto:path=/pklxkd-59/obj22?versionId=null 2023-05-31T20:44:48.835 DEBUG:boto:auth_path=/pklxkd-59/obj22?versionId=null 2023-05-31T20:44:48.836 DEBUG:boto:Method: GET 2023-05-31T20:44:48.836 DEBUG:boto:Path: /pklxkd-59/obj22?versionId=null 2023-05-31T20:44:48.836 DEBUG:boto:Data: 2023-05-31T20:44:48.837 DEBUG:boto:Headers: {} 2023-05-31T20:44:48.837 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:44:48.837 DEBUG:boto:Port: 8001 2023-05-31T20:44:48.838 DEBUG:boto:Params: {} 2023-05-31T20:44:48.838 DEBUG:boto:Token: None 2023-05-31T20:44:48.839 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:44:48 GMT /pklxkd-59/obj22?versionId=null 2023-05-31T20:44:48.839 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:5qbT9rFHhtsiuxiBuvysXOq4W1c= 2023-05-31T20:44:48.840 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:44:48 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:5qbT9rFHhtsiuxiBuvysXOq4W1c=', 'Content-Length': '0'} 2023-05-31T20:44:48.841 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:39:26 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002945ddf42ec6b229-006477b1c0-1c17-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:44:48 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:44:48.842 DEBUG:boto:path=/pklxkd-59/obj22 2023-05-31T20:44:48.843 DEBUG:boto:auth_path=/pklxkd-59/obj22 2023-05-31T20:44:48.843 DEBUG:boto:path=/pklxkd-59/obj22?versionId=null 2023-05-31T20:44:48.843 DEBUG:boto:auth_path=/pklxkd-59/obj22?versionId=null 2023-05-31T20:44:48.844 DEBUG:boto:Method: GET 2023-05-31T20:44:48.844 DEBUG:boto:Path: /pklxkd-59/obj22?versionId=null 2023-05-31T20:44:48.845 DEBUG:boto:Data: 2023-05-31T20:44:48.845 DEBUG:boto:Headers: {} 2023-05-31T20:44:48.845 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:44:48.846 DEBUG:boto:Port: 8000 2023-05-31T20:44:48.846 DEBUG:boto:Params: {} 2023-05-31T20:44:48.846 DEBUG:boto:Token: None 2023-05-31T20:44:48.847 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:44:48 GMT /pklxkd-59/obj22?versionId=null 2023-05-31T20:44:48.847 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:5qbT9rFHhtsiuxiBuvysXOq4W1c= 2023-05-31T20:44:48.848 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:44:48 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:5qbT9rFHhtsiuxiBuvysXOq4W1c=', 'Content-Length': '0'} 2023-05-31T20:44:48.850 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:39:26 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005388668ff1eab1e7-006477b1c0-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:44:48 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:44:48.850 DEBUG:boto:path=/pklxkd-59/obj22 2023-05-31T20:44:48.851 DEBUG:boto:auth_path=/pklxkd-59/obj22 2023-05-31T20:44:48.851 DEBUG:boto:path=/pklxkd-59/obj22?versionId=null 2023-05-31T20:44:48.852 DEBUG:boto:auth_path=/pklxkd-59/obj22?versionId=null 2023-05-31T20:44:48.852 DEBUG:boto:Method: HEAD 2023-05-31T20:44:48.853 DEBUG:boto:Path: /pklxkd-59/obj22?versionId=null 2023-05-31T20:44:48.853 DEBUG:boto:Data: 2023-05-31T20:44:48.853 DEBUG:boto:Headers: {} 2023-05-31T20:44:48.854 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:44:48.854 DEBUG:boto:Port: 8001 2023-05-31T20:44:48.854 DEBUG:boto:Params: {} 2023-05-31T20:44:48.855 DEBUG:boto:Token: None 2023-05-31T20:44:48.855 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:44:48 GMT /pklxkd-59/obj22?versionId=null 2023-05-31T20:44:48.855 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:RqeeoIIX2UPucjnAssJB7l/rx6U= 2023-05-31T20:44:48.856 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:44:48 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:RqeeoIIX2UPucjnAssJB7l/rx6U=', 'Content-Length': '0'} 2023-05-31T20:44:48.857 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:39:26 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000382a01d2bb143ce6-006477b1c0-1c17-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:44:48 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:44:48.858 DEBUG:boto:path=/pklxkd-59/obj22 2023-05-31T20:44:48.858 DEBUG:boto:auth_path=/pklxkd-59/obj22 2023-05-31T20:44:48.859 DEBUG:boto:path=/pklxkd-59/obj22?versionId=null 2023-05-31T20:44:48.859 DEBUG:boto:auth_path=/pklxkd-59/obj22?versionId=null 2023-05-31T20:44:48.859 DEBUG:boto:Method: HEAD 2023-05-31T20:44:48.860 DEBUG:boto:Path: /pklxkd-59/obj22?versionId=null 2023-05-31T20:44:48.860 DEBUG:boto:Data: 2023-05-31T20:44:48.861 DEBUG:boto:Headers: {} 2023-05-31T20:44:48.861 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:44:48.861 DEBUG:boto:Port: 8000 2023-05-31T20:44:48.862 DEBUG:boto:Params: {} 2023-05-31T20:44:48.862 DEBUG:boto:Token: None 2023-05-31T20:44:48.862 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:44:48 GMT /pklxkd-59/obj22?versionId=null 2023-05-31T20:44:48.863 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:RqeeoIIX2UPucjnAssJB7l/rx6U= 2023-05-31T20:44:48.863 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:44:48 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:RqeeoIIX2UPucjnAssJB7l/rx6U=', 'Content-Length': '0'} 2023-05-31T20:44:48.865 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:39:26 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003c465567ea09a92f-006477b1c0-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:44:48 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:44:48.866 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj22 2023-05-31T20:44:48.866 DEBUG:boto:path=/pklxkd-59/obj22 2023-05-31T20:44:48.866 DEBUG:boto:auth_path=/pklxkd-59/obj22 2023-05-31T20:44:48.867 DEBUG:boto:path=/pklxkd-59/obj22?versionId=null 2023-05-31T20:44:48.867 DEBUG:boto:auth_path=/pklxkd-59/obj22?versionId=null 2023-05-31T20:44:48.868 DEBUG:boto:Method: GET 2023-05-31T20:44:48.868 DEBUG:boto:Path: /pklxkd-59/obj22?versionId=null 2023-05-31T20:44:48.868 DEBUG:boto:Data: 2023-05-31T20:44:48.869 DEBUG:boto:Headers: {} 2023-05-31T20:44:48.869 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:44:48.870 DEBUG:boto:Port: 8001 2023-05-31T20:44:48.870 DEBUG:boto:Params: {} 2023-05-31T20:44:48.871 DEBUG:boto:Token: None 2023-05-31T20:44:48.871 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:44:48 GMT /pklxkd-59/obj22?versionId=null 2023-05-31T20:44:48.871 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:5qbT9rFHhtsiuxiBuvysXOq4W1c= 2023-05-31T20:44:48.872 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:44:48 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:5qbT9rFHhtsiuxiBuvysXOq4W1c=', 'Content-Length': '0'} 2023-05-31T20:44:48.874 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:39:26 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000066cb22dea991a17f-006477b1c0-1c17-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:44:48 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:44:48.875 DEBUG:boto:path=/pklxkd-59/obj22 2023-05-31T20:44:48.875 DEBUG:boto:auth_path=/pklxkd-59/obj22 2023-05-31T20:44:48.875 DEBUG:boto:path=/pklxkd-59/obj22?versionId=null 2023-05-31T20:44:48.876 DEBUG:boto:auth_path=/pklxkd-59/obj22?versionId=null 2023-05-31T20:44:48.876 DEBUG:boto:Method: GET 2023-05-31T20:44:48.877 DEBUG:boto:Path: /pklxkd-59/obj22?versionId=null 2023-05-31T20:44:48.877 DEBUG:boto:Data: 2023-05-31T20:44:48.877 DEBUG:boto:Headers: {} 2023-05-31T20:44:48.878 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:44:48.878 DEBUG:boto:Port: 8000 2023-05-31T20:44:48.879 DEBUG:boto:Params: {} 2023-05-31T20:44:48.880 DEBUG:boto:Token: None 2023-05-31T20:44:48.884 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:44:48 GMT /pklxkd-59/obj22?versionId=null 2023-05-31T20:44:48.884 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:5qbT9rFHhtsiuxiBuvysXOq4W1c= 2023-05-31T20:44:48.885 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:44:48 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:5qbT9rFHhtsiuxiBuvysXOq4W1c=', 'Content-Length': '0'} 2023-05-31T20:44:48.887 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:39:26 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005ef64a49fb36bf64-006477b1c0-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:44:48 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:44:48.887 DEBUG:boto:path=/pklxkd-59/ 2023-05-31T20:44:48.888 DEBUG:boto:auth_path=/pklxkd-59/ 2023-05-31T20:44:48.888 DEBUG:boto:path=/pklxkd-59/?prefix=obj22 2023-05-31T20:44:48.888 DEBUG:boto:auth_path=/pklxkd-59/?prefix=obj22 2023-05-31T20:44:48.889 DEBUG:boto:Method: GET 2023-05-31T20:44:48.889 DEBUG:boto:Path: /pklxkd-59/?prefix=obj22 2023-05-31T20:44:48.889 DEBUG:boto:Data: 2023-05-31T20:44:48.890 DEBUG:boto:Headers: {} 2023-05-31T20:44:48.890 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:44:48.890 DEBUG:boto:Port: 8001 2023-05-31T20:44:48.891 DEBUG:boto:Params: {} 2023-05-31T20:44:48.891 DEBUG:boto:Token: None 2023-05-31T20:44:48.892 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:44:48 GMT /pklxkd-59/ 2023-05-31T20:44:48.892 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:717u7lUCeWRZyRakKJN50YyqEZ8= 2023-05-31T20:44:48.893 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:44:48 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:717u7lUCeWRZyRakKJN50YyqEZ8=', 'Content-Length': '0'} 2023-05-31T20:44:48.895 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b3b1a73559ed7b79-006477b1c0-1c17-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:44:48 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:44:48.895 DEBUG:boto:b'pklxkd-59obj221000falseobj222023-05-31T20:39:26.473Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:44:48.896 DEBUG:boto:path=/pklxkd-59/ 2023-05-31T20:44:48.896 DEBUG:boto:auth_path=/pklxkd-59/ 2023-05-31T20:44:48.897 DEBUG:boto:path=/pklxkd-59/?prefix=obj22 2023-05-31T20:44:48.898 DEBUG:boto:auth_path=/pklxkd-59/?prefix=obj22 2023-05-31T20:44:48.898 DEBUG:boto:Method: GET 2023-05-31T20:44:48.898 DEBUG:boto:Path: /pklxkd-59/?prefix=obj22 2023-05-31T20:44:48.899 DEBUG:boto:Data: 2023-05-31T20:44:48.899 DEBUG:boto:Headers: {} 2023-05-31T20:44:48.900 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:44:48.900 DEBUG:boto:Port: 8000 2023-05-31T20:44:48.900 DEBUG:boto:Params: {} 2023-05-31T20:44:48.901 DEBUG:boto:Token: None 2023-05-31T20:44:48.901 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:44:48 GMT /pklxkd-59/ 2023-05-31T20:44:48.901 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:717u7lUCeWRZyRakKJN50YyqEZ8= 2023-05-31T20:44:48.902 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:44:48 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:717u7lUCeWRZyRakKJN50YyqEZ8=', 'Content-Length': '0'} 2023-05-31T20:44:48.905 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000037b6414f5e1b4bec-006477b1c0-1630-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:44:48 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:44:48.905 DEBUG:boto:b'pklxkd-59obj221000falseobj222023-05-31T20:39:26.473Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:44:48.906 DEBUG:boto:path=/pklxkd-59/obj22 2023-05-31T20:44:48.906 DEBUG:boto:auth_path=/pklxkd-59/obj22 2023-05-31T20:44:48.907 DEBUG:boto:Method: HEAD 2023-05-31T20:44:48.907 DEBUG:boto:Path: /pklxkd-59/obj22 2023-05-31T20:44:48.907 DEBUG:boto:Data: 2023-05-31T20:44:48.908 DEBUG:boto:Headers: {} 2023-05-31T20:44:48.908 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:44:48.908 DEBUG:boto:Port: 8001 2023-05-31T20:44:48.909 DEBUG:boto:Params: {} 2023-05-31T20:44:48.909 DEBUG:boto:Token: None 2023-05-31T20:44:48.909 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:44:48 GMT /pklxkd-59/obj22 2023-05-31T20:44:48.910 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:FxBmQ2U4p+bvaF1Qmes4HG3vzVI= 2023-05-31T20:44:48.910 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:44:48 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:FxBmQ2U4p+bvaF1Qmes4HG3vzVI=', 'Content-Length': '0'} 2023-05-31T20:44:48.912 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:39:26 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009142cfaa2d9665ae-006477b1c0-1c17-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:44:48 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:44:48.913 DEBUG:boto:path=/pklxkd-59/obj22 2023-05-31T20:44:48.913 DEBUG:boto:auth_path=/pklxkd-59/obj22 2023-05-31T20:44:48.913 DEBUG:boto:Method: HEAD 2023-05-31T20:44:48.914 DEBUG:boto:Path: /pklxkd-59/obj22 2023-05-31T20:44:48.914 DEBUG:boto:Data: 2023-05-31T20:44:48.914 DEBUG:boto:Headers: {} 2023-05-31T20:44:48.915 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:44:48.915 DEBUG:boto:Port: 8000 2023-05-31T20:44:48.915 DEBUG:boto:Params: {} 2023-05-31T20:44:48.916 DEBUG:boto:Token: None 2023-05-31T20:44:48.916 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:44:48 GMT /pklxkd-59/obj22 2023-05-31T20:44:48.917 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:FxBmQ2U4p+bvaF1Qmes4HG3vzVI= 2023-05-31T20:44:48.917 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:44:48 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:FxBmQ2U4p+bvaF1Qmes4HG3vzVI=', 'Content-Length': '0'} 2023-05-31T20:44:48.919 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:39:26 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c9e00994ffad63a1-006477b1c0-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:44:48 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:44:48.919 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj22 2023-05-31T20:44:48.920 DEBUG:boto:path=/pklxkd-59/obj22 2023-05-31T20:44:48.920 DEBUG:boto:auth_path=/pklxkd-59/obj22 2023-05-31T20:44:48.920 DEBUG:boto:Method: GET 2023-05-31T20:44:48.921 DEBUG:boto:Path: /pklxkd-59/obj22 2023-05-31T20:44:48.921 DEBUG:boto:Data: 2023-05-31T20:44:48.921 DEBUG:boto:Headers: {} 2023-05-31T20:44:48.922 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:44:48.922 DEBUG:boto:Port: 8001 2023-05-31T20:44:48.922 DEBUG:boto:Params: {} 2023-05-31T20:44:48.923 DEBUG:boto:Token: None 2023-05-31T20:44:48.923 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:44:48 GMT /pklxkd-59/obj22 2023-05-31T20:44:48.924 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:Nj81JBH3Ea2GO4D56ndodoOzoZQ= 2023-05-31T20:44:48.924 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:44:48 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:Nj81JBH3Ea2GO4D56ndodoOzoZQ=', 'Content-Length': '0'} 2023-05-31T20:44:48.926 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:39:26 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c9c5f50a1fa05258-006477b1c0-1c17-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:44:48 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:44:48.927 DEBUG:boto:path=/pklxkd-59/obj22 2023-05-31T20:44:48.927 DEBUG:boto:auth_path=/pklxkd-59/obj22 2023-05-31T20:44:48.928 DEBUG:boto:Method: GET 2023-05-31T20:44:48.928 DEBUG:boto:Path: /pklxkd-59/obj22 2023-05-31T20:44:48.929 DEBUG:boto:Data: 2023-05-31T20:44:48.929 DEBUG:boto:Headers: {} 2023-05-31T20:44:48.929 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:44:48.930 DEBUG:boto:Port: 8000 2023-05-31T20:44:48.930 DEBUG:boto:Params: {} 2023-05-31T20:44:48.930 DEBUG:boto:Token: None 2023-05-31T20:44:48.931 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:44:48 GMT /pklxkd-59/obj22 2023-05-31T20:44:48.931 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:Nj81JBH3Ea2GO4D56ndodoOzoZQ= 2023-05-31T20:44:48.932 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:44:48 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:Nj81JBH3Ea2GO4D56ndodoOzoZQ=', 'Content-Length': '0'} 2023-05-31T20:44:48.933 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:39:26 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000eebc2ffe4ef8e215-006477b1c0-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:44:48 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:44:48.976 DEBUG:boto:path=/pklxkd-59/ 2023-05-31T20:44:48.977 DEBUG:boto:auth_path=/pklxkd-59/ 2023-05-31T20:44:48.977 DEBUG:boto:path=/pklxkd-59/?prefix=obj22 2023-05-31T20:44:48.978 DEBUG:boto:auth_path=/pklxkd-59/?prefix=obj22 2023-05-31T20:44:48.978 DEBUG:boto:Method: GET 2023-05-31T20:44:48.978 DEBUG:boto:Path: /pklxkd-59/?prefix=obj22 2023-05-31T20:44:48.979 DEBUG:boto:Data: 2023-05-31T20:44:48.979 DEBUG:boto:Headers: {} 2023-05-31T20:44:48.979 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:44:48.980 DEBUG:boto:Port: 8001 2023-05-31T20:44:48.980 DEBUG:boto:Params: {} 2023-05-31T20:44:48.981 DEBUG:boto:Token: None 2023-05-31T20:44:48.981 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:44:48 GMT /pklxkd-59/ 2023-05-31T20:44:48.981 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:717u7lUCeWRZyRakKJN50YyqEZ8= 2023-05-31T20:44:48.982 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:44:48 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:717u7lUCeWRZyRakKJN50YyqEZ8=', 'Content-Length': '0'} 2023-05-31T20:44:48.984 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000004c65b7b6a26ad336-006477b1c0-1c17-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:44:48 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:44:48.985 DEBUG:boto:b'pklxkd-59obj221000falseobj222023-05-31T20:39:26.473Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:44:48.985 DEBUG:boto:path=/pklxkd-59/ 2023-05-31T20:44:48.986 DEBUG:boto:auth_path=/pklxkd-59/ 2023-05-31T20:44:48.986 DEBUG:boto:path=/pklxkd-59/?prefix=obj22 2023-05-31T20:44:48.986 DEBUG:boto:auth_path=/pklxkd-59/?prefix=obj22 2023-05-31T20:44:48.987 DEBUG:boto:Method: GET 2023-05-31T20:44:48.987 DEBUG:boto:Path: /pklxkd-59/?prefix=obj22 2023-05-31T20:44:48.988 DEBUG:boto:Data: 2023-05-31T20:44:48.988 DEBUG:boto:Headers: {} 2023-05-31T20:44:48.988 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:44:48.989 DEBUG:boto:Port: 8000 2023-05-31T20:44:48.989 DEBUG:boto:Params: {} 2023-05-31T20:44:48.990 DEBUG:boto:Token: None 2023-05-31T20:44:48.990 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:44:48 GMT /pklxkd-59/ 2023-05-31T20:44:48.990 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:717u7lUCeWRZyRakKJN50YyqEZ8= 2023-05-31T20:44:48.991 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:44:48 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:717u7lUCeWRZyRakKJN50YyqEZ8=', 'Content-Length': '0'} 2023-05-31T20:44:48.994 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e2ce17ad05b419c9-006477b1c0-1630-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:44:48 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:44:48.994 DEBUG:boto:b'pklxkd-59obj221000falseobj222023-05-31T20:39:26.473Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:44:48.995 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj33 2023-05-31T20:44:48.995 DEBUG:boto:path=/pklxkd-59/obj33 2023-05-31T20:44:48.995 DEBUG:boto:auth_path=/pklxkd-59/obj33 2023-05-31T20:44:48.996 DEBUG:boto:path=/pklxkd-59/obj33?versionId=null 2023-05-31T20:44:48.996 DEBUG:boto:auth_path=/pklxkd-59/obj33?versionId=null 2023-05-31T20:44:48.997 DEBUG:boto:Method: GET 2023-05-31T20:44:48.997 DEBUG:boto:Path: /pklxkd-59/obj33?versionId=null 2023-05-31T20:44:48.997 DEBUG:boto:Data: 2023-05-31T20:44:48.998 DEBUG:boto:Headers: {} 2023-05-31T20:44:48.998 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:44:48.998 DEBUG:boto:Port: 8001 2023-05-31T20:44:48.999 DEBUG:boto:Params: {} 2023-05-31T20:44:48.999 DEBUG:boto:Token: None 2023-05-31T20:44:49.000 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:44:49 GMT /pklxkd-59/obj33?versionId=null 2023-05-31T20:44:49.000 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:Up0bq8HbbR3PlyqqObBOd04iWrs= 2023-05-31T20:44:49.001 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:44:49 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:Up0bq8HbbR3PlyqqObBOd04iWrs=', 'Content-Length': '0'} 2023-05-31T20:44:49.002 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:39:26 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003c3ddf879345f667-006477b1c0-1c17-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:44:48 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:44:49.044 DEBUG:boto:path=/pklxkd-59/obj33 2023-05-31T20:44:49.045 DEBUG:boto:auth_path=/pklxkd-59/obj33 2023-05-31T20:44:49.046 DEBUG:boto:path=/pklxkd-59/obj33?versionId=null 2023-05-31T20:44:49.046 DEBUG:boto:auth_path=/pklxkd-59/obj33?versionId=null 2023-05-31T20:44:49.047 DEBUG:boto:Method: GET 2023-05-31T20:44:49.048 DEBUG:boto:Path: /pklxkd-59/obj33?versionId=null 2023-05-31T20:44:49.048 DEBUG:boto:Data: 2023-05-31T20:44:49.049 DEBUG:boto:Headers: {} 2023-05-31T20:44:49.050 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:44:49.050 DEBUG:boto:Port: 8000 2023-05-31T20:44:49.051 DEBUG:boto:Params: {} 2023-05-31T20:44:49.052 DEBUG:boto:Token: None 2023-05-31T20:44:49.053 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:44:49 GMT /pklxkd-59/obj33?versionId=null 2023-05-31T20:44:49.053 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:Up0bq8HbbR3PlyqqObBOd04iWrs= 2023-05-31T20:44:49.054 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:44:49 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:Up0bq8HbbR3PlyqqObBOd04iWrs=', 'Content-Length': '0'} 2023-05-31T20:44:49.057 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:39:26 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ae698049672771d6-006477b1c1-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:44:49 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:44:49.058 DEBUG:boto:path=/pklxkd-59/obj33 2023-05-31T20:44:49.058 DEBUG:boto:auth_path=/pklxkd-59/obj33 2023-05-31T20:44:49.059 DEBUG:boto:path=/pklxkd-59/obj33?versionId=null 2023-05-31T20:44:49.060 DEBUG:boto:auth_path=/pklxkd-59/obj33?versionId=null 2023-05-31T20:44:49.060 DEBUG:boto:Method: HEAD 2023-05-31T20:44:49.061 DEBUG:boto:Path: /pklxkd-59/obj33?versionId=null 2023-05-31T20:44:49.061 DEBUG:boto:Data: 2023-05-31T20:44:49.062 DEBUG:boto:Headers: {} 2023-05-31T20:44:49.063 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:44:49.064 DEBUG:boto:Port: 8001 2023-05-31T20:44:49.064 DEBUG:boto:Params: {} 2023-05-31T20:44:49.065 DEBUG:boto:Token: None 2023-05-31T20:44:49.066 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:44:49 GMT /pklxkd-59/obj33?versionId=null 2023-05-31T20:44:49.066 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:vEe+oUWz1ETEjxMmt5nA23RUKgo= 2023-05-31T20:44:49.067 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:44:49 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:vEe+oUWz1ETEjxMmt5nA23RUKgo=', 'Content-Length': '0'} 2023-05-31T20:44:49.069 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:39:26 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f9c2dabde846473f-006477b1c1-1c17-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:44:49 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:44:49.070 DEBUG:boto:path=/pklxkd-59/obj33 2023-05-31T20:44:49.071 DEBUG:boto:auth_path=/pklxkd-59/obj33 2023-05-31T20:44:49.072 DEBUG:boto:path=/pklxkd-59/obj33?versionId=null 2023-05-31T20:44:49.072 DEBUG:boto:auth_path=/pklxkd-59/obj33?versionId=null 2023-05-31T20:44:49.073 DEBUG:boto:Method: HEAD 2023-05-31T20:44:49.074 DEBUG:boto:Path: /pklxkd-59/obj33?versionId=null 2023-05-31T20:44:49.074 DEBUG:boto:Data: 2023-05-31T20:44:49.075 DEBUG:boto:Headers: {} 2023-05-31T20:44:49.076 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:44:49.076 DEBUG:boto:Port: 8000 2023-05-31T20:44:49.077 DEBUG:boto:Params: {} 2023-05-31T20:44:49.078 DEBUG:boto:Token: None 2023-05-31T20:44:49.078 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:44:49 GMT /pklxkd-59/obj33?versionId=null 2023-05-31T20:44:49.079 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:vEe+oUWz1ETEjxMmt5nA23RUKgo= 2023-05-31T20:44:49.080 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:44:49 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:vEe+oUWz1ETEjxMmt5nA23RUKgo=', 'Content-Length': '0'} 2023-05-31T20:44:49.082 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:39:26 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ecef32c5642382c2-006477b1c1-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:44:49 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:44:49.082 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj33 2023-05-31T20:44:49.083 DEBUG:boto:path=/pklxkd-59/obj33 2023-05-31T20:44:49.083 DEBUG:boto:auth_path=/pklxkd-59/obj33 2023-05-31T20:44:49.083 DEBUG:boto:path=/pklxkd-59/obj33?versionId=null 2023-05-31T20:44:49.084 DEBUG:boto:auth_path=/pklxkd-59/obj33?versionId=null 2023-05-31T20:44:49.084 DEBUG:boto:Method: GET 2023-05-31T20:44:49.084 DEBUG:boto:Path: /pklxkd-59/obj33?versionId=null 2023-05-31T20:44:49.085 DEBUG:boto:Data: 2023-05-31T20:44:49.085 DEBUG:boto:Headers: {} 2023-05-31T20:44:49.086 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:44:49.086 DEBUG:boto:Port: 8001 2023-05-31T20:44:49.087 DEBUG:boto:Params: {} 2023-05-31T20:44:49.087 DEBUG:boto:Token: None 2023-05-31T20:44:49.087 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:44:49 GMT /pklxkd-59/obj33?versionId=null 2023-05-31T20:44:49.088 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:Up0bq8HbbR3PlyqqObBOd04iWrs= 2023-05-31T20:44:49.088 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:44:49 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:Up0bq8HbbR3PlyqqObBOd04iWrs=', 'Content-Length': '0'} 2023-05-31T20:44:49.090 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:39:26 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005f721061f55c4ad3-006477b1c1-1c17-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:44:49 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:44:49.091 DEBUG:boto:path=/pklxkd-59/obj33 2023-05-31T20:44:49.091 DEBUG:boto:auth_path=/pklxkd-59/obj33 2023-05-31T20:44:49.092 DEBUG:boto:path=/pklxkd-59/obj33?versionId=null 2023-05-31T20:44:49.092 DEBUG:boto:auth_path=/pklxkd-59/obj33?versionId=null 2023-05-31T20:44:49.092 DEBUG:boto:Method: GET 2023-05-31T20:44:49.093 DEBUG:boto:Path: /pklxkd-59/obj33?versionId=null 2023-05-31T20:44:49.093 DEBUG:boto:Data: 2023-05-31T20:44:49.094 DEBUG:boto:Headers: {} 2023-05-31T20:44:49.094 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:44:49.094 DEBUG:boto:Port: 8000 2023-05-31T20:44:49.095 DEBUG:boto:Params: {} 2023-05-31T20:44:49.095 DEBUG:boto:Token: None 2023-05-31T20:44:49.095 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:44:49 GMT /pklxkd-59/obj33?versionId=null 2023-05-31T20:44:49.096 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:Up0bq8HbbR3PlyqqObBOd04iWrs= 2023-05-31T20:44:49.096 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:44:49 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:Up0bq8HbbR3PlyqqObBOd04iWrs=', 'Content-Length': '0'} 2023-05-31T20:44:49.098 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:39:26 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d1b48cf20253fb8b-006477b1c1-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:44:49 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:44:49.099 DEBUG:boto:path=/pklxkd-59/ 2023-05-31T20:44:49.100 DEBUG:boto:auth_path=/pklxkd-59/ 2023-05-31T20:44:49.100 DEBUG:boto:path=/pklxkd-59/?prefix=obj33 2023-05-31T20:44:49.100 DEBUG:boto:auth_path=/pklxkd-59/?prefix=obj33 2023-05-31T20:44:49.101 DEBUG:boto:Method: GET 2023-05-31T20:44:49.101 DEBUG:boto:Path: /pklxkd-59/?prefix=obj33 2023-05-31T20:44:49.101 DEBUG:boto:Data: 2023-05-31T20:44:49.102 DEBUG:boto:Headers: {} 2023-05-31T20:44:49.102 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:44:49.102 DEBUG:boto:Port: 8001 2023-05-31T20:44:49.103 DEBUG:boto:Params: {} 2023-05-31T20:44:49.103 DEBUG:boto:Token: None 2023-05-31T20:44:49.103 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:44:49 GMT /pklxkd-59/ 2023-05-31T20:44:49.104 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:VK2eJyJ+BXpP/6492MS9jzEZ3kw= 2023-05-31T20:44:49.104 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:44:49 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:VK2eJyJ+BXpP/6492MS9jzEZ3kw=', 'Content-Length': '0'} 2023-05-31T20:44:49.107 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000001d65129c37bdd68a-006477b1c1-1c17-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:44:49 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:44:49.107 DEBUG:boto:b'pklxkd-59obj331000falseobj332023-05-31T20:39:26.534Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:44:49.108 DEBUG:boto:path=/pklxkd-59/ 2023-05-31T20:44:49.108 DEBUG:boto:auth_path=/pklxkd-59/ 2023-05-31T20:44:49.109 DEBUG:boto:path=/pklxkd-59/?prefix=obj33 2023-05-31T20:44:49.109 DEBUG:boto:auth_path=/pklxkd-59/?prefix=obj33 2023-05-31T20:44:49.110 DEBUG:boto:Method: GET 2023-05-31T20:44:49.110 DEBUG:boto:Path: /pklxkd-59/?prefix=obj33 2023-05-31T20:44:49.110 DEBUG:boto:Data: 2023-05-31T20:44:49.111 DEBUG:boto:Headers: {} 2023-05-31T20:44:49.111 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:44:49.111 DEBUG:boto:Port: 8000 2023-05-31T20:44:49.112 DEBUG:boto:Params: {} 2023-05-31T20:44:49.112 DEBUG:boto:Token: None 2023-05-31T20:44:49.112 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:44:49 GMT /pklxkd-59/ 2023-05-31T20:44:49.113 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:VK2eJyJ+BXpP/6492MS9jzEZ3kw= 2023-05-31T20:44:49.113 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:44:49 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:VK2eJyJ+BXpP/6492MS9jzEZ3kw=', 'Content-Length': '0'} 2023-05-31T20:44:49.116 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000efa7f65839a00565-006477b1c1-1630-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:44:49 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:44:49.116 DEBUG:boto:b'pklxkd-59obj331000falseobj332023-05-31T20:39:26.534Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:44:49.117 DEBUG:boto:path=/pklxkd-59/obj33 2023-05-31T20:44:49.117 DEBUG:boto:auth_path=/pklxkd-59/obj33 2023-05-31T20:44:49.118 DEBUG:boto:Method: HEAD 2023-05-31T20:44:49.118 DEBUG:boto:Path: /pklxkd-59/obj33 2023-05-31T20:44:49.118 DEBUG:boto:Data: 2023-05-31T20:44:49.119 DEBUG:boto:Headers: {} 2023-05-31T20:44:49.119 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:44:49.119 DEBUG:boto:Port: 8001 2023-05-31T20:44:49.120 DEBUG:boto:Params: {} 2023-05-31T20:44:49.120 DEBUG:boto:Token: None 2023-05-31T20:44:49.121 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:44:49 GMT /pklxkd-59/obj33 2023-05-31T20:44:49.121 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:Mo0wS38cNmCwkH9X5L5911MPZX0= 2023-05-31T20:44:49.121 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:44:49 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:Mo0wS38cNmCwkH9X5L5911MPZX0=', 'Content-Length': '0'} 2023-05-31T20:44:49.123 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:39:26 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000007279b5b55f2c24a-006477b1c1-1c17-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:44:49 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:44:49.124 DEBUG:boto:path=/pklxkd-59/obj33 2023-05-31T20:44:49.124 DEBUG:boto:auth_path=/pklxkd-59/obj33 2023-05-31T20:44:49.124 DEBUG:boto:Method: HEAD 2023-05-31T20:44:49.125 DEBUG:boto:Path: /pklxkd-59/obj33 2023-05-31T20:44:49.125 DEBUG:boto:Data: 2023-05-31T20:44:49.125 DEBUG:boto:Headers: {} 2023-05-31T20:44:49.126 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:44:49.126 DEBUG:boto:Port: 8000 2023-05-31T20:44:49.126 DEBUG:boto:Params: {} 2023-05-31T20:44:49.127 DEBUG:boto:Token: None 2023-05-31T20:44:49.127 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:44:49 GMT /pklxkd-59/obj33 2023-05-31T20:44:49.128 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:Mo0wS38cNmCwkH9X5L5911MPZX0= 2023-05-31T20:44:49.128 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:44:49 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:Mo0wS38cNmCwkH9X5L5911MPZX0=', 'Content-Length': '0'} 2023-05-31T20:44:49.130 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:39:26 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f6ec7e0b64e0213d-006477b1c1-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:44:49 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:44:49.130 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj33 2023-05-31T20:44:49.131 DEBUG:boto:path=/pklxkd-59/obj33 2023-05-31T20:44:49.131 DEBUG:boto:auth_path=/pklxkd-59/obj33 2023-05-31T20:44:49.131 DEBUG:boto:Method: GET 2023-05-31T20:44:49.132 DEBUG:boto:Path: /pklxkd-59/obj33 2023-05-31T20:44:49.132 DEBUG:boto:Data: 2023-05-31T20:44:49.133 DEBUG:boto:Headers: {} 2023-05-31T20:44:49.133 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:44:49.133 DEBUG:boto:Port: 8001 2023-05-31T20:44:49.134 DEBUG:boto:Params: {} 2023-05-31T20:44:49.134 DEBUG:boto:Token: None 2023-05-31T20:44:49.134 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:44:49 GMT /pklxkd-59/obj33 2023-05-31T20:44:49.135 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:vgyzLS3YeIOLcgBGt7Y/vIrrAzw= 2023-05-31T20:44:49.135 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:44:49 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:vgyzLS3YeIOLcgBGt7Y/vIrrAzw=', 'Content-Length': '0'} 2023-05-31T20:44:49.137 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:39:26 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000039f94bbb53d3610-006477b1c1-1c17-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:44:49 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:44:49.138 DEBUG:boto:path=/pklxkd-59/obj33 2023-05-31T20:44:49.138 DEBUG:boto:auth_path=/pklxkd-59/obj33 2023-05-31T20:44:49.139 DEBUG:boto:Method: GET 2023-05-31T20:44:49.139 DEBUG:boto:Path: /pklxkd-59/obj33 2023-05-31T20:44:49.140 DEBUG:boto:Data: 2023-05-31T20:44:49.140 DEBUG:boto:Headers: {} 2023-05-31T20:44:49.140 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:44:49.141 DEBUG:boto:Port: 8000 2023-05-31T20:44:49.141 DEBUG:boto:Params: {} 2023-05-31T20:44:49.141 DEBUG:boto:Token: None 2023-05-31T20:44:49.142 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:44:49 GMT /pklxkd-59/obj33 2023-05-31T20:44:49.142 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:vgyzLS3YeIOLcgBGt7Y/vIrrAzw= 2023-05-31T20:44:49.143 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:44:49 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:vgyzLS3YeIOLcgBGt7Y/vIrrAzw=', 'Content-Length': '0'} 2023-05-31T20:44:49.145 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:39:26 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f3a80d5eed6f4686-006477b1c1-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:44:49 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:44:49.145 DEBUG:boto:path=/pklxkd-59/ 2023-05-31T20:44:49.146 DEBUG:boto:auth_path=/pklxkd-59/ 2023-05-31T20:44:49.146 DEBUG:boto:path=/pklxkd-59/?prefix=obj33 2023-05-31T20:44:49.147 DEBUG:boto:auth_path=/pklxkd-59/?prefix=obj33 2023-05-31T20:44:49.147 DEBUG:boto:Method: GET 2023-05-31T20:44:49.147 DEBUG:boto:Path: /pklxkd-59/?prefix=obj33 2023-05-31T20:44:49.148 DEBUG:boto:Data: 2023-05-31T20:44:49.148 DEBUG:boto:Headers: {} 2023-05-31T20:44:49.148 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:44:49.149 DEBUG:boto:Port: 8001 2023-05-31T20:44:49.149 DEBUG:boto:Params: {} 2023-05-31T20:44:49.149 DEBUG:boto:Token: None 2023-05-31T20:44:49.150 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:44:49 GMT /pklxkd-59/ 2023-05-31T20:44:49.150 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:VK2eJyJ+BXpP/6492MS9jzEZ3kw= 2023-05-31T20:44:49.151 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:44:49 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:VK2eJyJ+BXpP/6492MS9jzEZ3kw=', 'Content-Length': '0'} 2023-05-31T20:44:49.153 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000002649d9c65bc9efcd-006477b1c1-1c17-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:44:49 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:44:49.196 DEBUG:boto:b'pklxkd-59obj331000falseobj332023-05-31T20:39:26.534Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:44:49.197 DEBUG:boto:path=/pklxkd-59/ 2023-05-31T20:44:49.197 DEBUG:boto:auth_path=/pklxkd-59/ 2023-05-31T20:44:49.198 DEBUG:boto:path=/pklxkd-59/?prefix=obj33 2023-05-31T20:44:49.198 DEBUG:boto:auth_path=/pklxkd-59/?prefix=obj33 2023-05-31T20:44:49.198 DEBUG:boto:Method: GET 2023-05-31T20:44:49.199 DEBUG:boto:Path: /pklxkd-59/?prefix=obj33 2023-05-31T20:44:49.199 DEBUG:boto:Data: 2023-05-31T20:44:49.200 DEBUG:boto:Headers: {} 2023-05-31T20:44:49.200 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:44:49.200 DEBUG:boto:Port: 8000 2023-05-31T20:44:49.201 DEBUG:boto:Params: {} 2023-05-31T20:44:49.201 DEBUG:boto:Token: None 2023-05-31T20:44:49.201 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:44:49 GMT /pklxkd-59/ 2023-05-31T20:44:49.202 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:VK2eJyJ+BXpP/6492MS9jzEZ3kw= 2023-05-31T20:44:49.202 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:44:49 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:VK2eJyJ+BXpP/6492MS9jzEZ3kw=', 'Content-Length': '0'} 2023-05-31T20:44:49.205 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000062b43874c0fd9ca9-006477b1c1-1630-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:44:49 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:44:49.205 DEBUG:boto:b'pklxkd-59obj331000falseobj332023-05-31T20:39:26.534Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:44:49.206 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj34 2023-05-31T20:44:49.206 DEBUG:boto:path=/pklxkd-59/obj34 2023-05-31T20:44:49.207 DEBUG:boto:auth_path=/pklxkd-59/obj34 2023-05-31T20:44:49.207 DEBUG:boto:path=/pklxkd-59/obj34?versionId=null 2023-05-31T20:44:49.208 DEBUG:boto:auth_path=/pklxkd-59/obj34?versionId=null 2023-05-31T20:44:49.208 DEBUG:boto:Method: GET 2023-05-31T20:44:49.208 DEBUG:boto:Path: /pklxkd-59/obj34?versionId=null 2023-05-31T20:44:49.209 DEBUG:boto:Data: 2023-05-31T20:44:49.209 DEBUG:boto:Headers: {} 2023-05-31T20:44:49.209 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:44:49.210 DEBUG:boto:Port: 8001 2023-05-31T20:44:49.210 DEBUG:boto:Params: {} 2023-05-31T20:44:49.210 DEBUG:boto:Token: None 2023-05-31T20:44:49.211 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:44:49 GMT /pklxkd-59/obj34?versionId=null 2023-05-31T20:44:49.211 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:3Ch4kevzsDU0JnakZQcUN+9hTAc= 2023-05-31T20:44:49.212 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:44:49 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:3Ch4kevzsDU0JnakZQcUN+9hTAc=', 'Content-Length': '0'} 2023-05-31T20:44:49.214 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:43:07 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003c4ecf6cc2ed7daa-006477b1c1-1c17-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:44:49 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:44:49.214 DEBUG:boto:path=/pklxkd-59/obj34 2023-05-31T20:44:49.215 DEBUG:boto:auth_path=/pklxkd-59/obj34 2023-05-31T20:44:49.215 DEBUG:boto:path=/pklxkd-59/obj34?versionId=null 2023-05-31T20:44:49.215 DEBUG:boto:auth_path=/pklxkd-59/obj34?versionId=null 2023-05-31T20:44:49.216 DEBUG:boto:Method: GET 2023-05-31T20:44:49.216 DEBUG:boto:Path: /pklxkd-59/obj34?versionId=null 2023-05-31T20:44:49.217 DEBUG:boto:Data: 2023-05-31T20:44:49.217 DEBUG:boto:Headers: {} 2023-05-31T20:44:49.217 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:44:49.218 DEBUG:boto:Port: 8000 2023-05-31T20:44:49.218 DEBUG:boto:Params: {} 2023-05-31T20:44:49.218 DEBUG:boto:Token: None 2023-05-31T20:44:49.219 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:44:49 GMT /pklxkd-59/obj34?versionId=null 2023-05-31T20:44:49.219 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:3Ch4kevzsDU0JnakZQcUN+9hTAc= 2023-05-31T20:44:49.220 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:44:49 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:3Ch4kevzsDU0JnakZQcUN+9hTAc=', 'Content-Length': '0'} 2023-05-31T20:44:49.222 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:43:07 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008b05eab136d0c44e-006477b1c1-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:44:49 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:44:49.264 DEBUG:boto:path=/pklxkd-59/obj34 2023-05-31T20:44:49.265 DEBUG:boto:auth_path=/pklxkd-59/obj34 2023-05-31T20:44:49.265 DEBUG:boto:path=/pklxkd-59/obj34?versionId=null 2023-05-31T20:44:49.265 DEBUG:boto:auth_path=/pklxkd-59/obj34?versionId=null 2023-05-31T20:44:49.266 DEBUG:boto:Method: HEAD 2023-05-31T20:44:49.266 DEBUG:boto:Path: /pklxkd-59/obj34?versionId=null 2023-05-31T20:44:49.266 DEBUG:boto:Data: 2023-05-31T20:44:49.267 DEBUG:boto:Headers: {} 2023-05-31T20:44:49.267 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:44:49.267 DEBUG:boto:Port: 8001 2023-05-31T20:44:49.268 DEBUG:boto:Params: {} 2023-05-31T20:44:49.268 DEBUG:boto:Token: None 2023-05-31T20:44:49.268 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:44:49 GMT /pklxkd-59/obj34?versionId=null 2023-05-31T20:44:49.269 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:a2LZAS5c61R5am8Z8TZQgStr9Fg= 2023-05-31T20:44:49.269 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:44:49 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:a2LZAS5c61R5am8Z8TZQgStr9Fg=', 'Content-Length': '0'} 2023-05-31T20:44:49.271 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:43:07 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f221e8c9d7f072e4-006477b1c1-1c17-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:44:49 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:44:49.272 DEBUG:boto:path=/pklxkd-59/obj34 2023-05-31T20:44:49.272 DEBUG:boto:auth_path=/pklxkd-59/obj34 2023-05-31T20:44:49.272 DEBUG:boto:path=/pklxkd-59/obj34?versionId=null 2023-05-31T20:44:49.273 DEBUG:boto:auth_path=/pklxkd-59/obj34?versionId=null 2023-05-31T20:44:49.273 DEBUG:boto:Method: HEAD 2023-05-31T20:44:49.273 DEBUG:boto:Path: /pklxkd-59/obj34?versionId=null 2023-05-31T20:44:49.274 DEBUG:boto:Data: 2023-05-31T20:44:49.274 DEBUG:boto:Headers: {} 2023-05-31T20:44:49.274 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:44:49.275 DEBUG:boto:Port: 8000 2023-05-31T20:44:49.275 DEBUG:boto:Params: {} 2023-05-31T20:44:49.275 DEBUG:boto:Token: None 2023-05-31T20:44:49.276 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:44:49 GMT /pklxkd-59/obj34?versionId=null 2023-05-31T20:44:49.276 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:a2LZAS5c61R5am8Z8TZQgStr9Fg= 2023-05-31T20:44:49.277 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:44:49 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:a2LZAS5c61R5am8Z8TZQgStr9Fg=', 'Content-Length': '0'} 2023-05-31T20:44:49.278 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:43:07 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f18d8c5fc1c37da8-006477b1c1-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:44:49 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:44:49.279 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj34 2023-05-31T20:44:49.279 DEBUG:boto:path=/pklxkd-59/obj34 2023-05-31T20:44:49.280 DEBUG:boto:auth_path=/pklxkd-59/obj34 2023-05-31T20:44:49.280 DEBUG:boto:path=/pklxkd-59/obj34?versionId=null 2023-05-31T20:44:49.281 DEBUG:boto:auth_path=/pklxkd-59/obj34?versionId=null 2023-05-31T20:44:49.281 DEBUG:boto:Method: GET 2023-05-31T20:44:49.281 DEBUG:boto:Path: /pklxkd-59/obj34?versionId=null 2023-05-31T20:44:49.282 DEBUG:boto:Data: 2023-05-31T20:44:49.282 DEBUG:boto:Headers: {} 2023-05-31T20:44:49.282 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:44:49.283 DEBUG:boto:Port: 8001 2023-05-31T20:44:49.283 DEBUG:boto:Params: {} 2023-05-31T20:44:49.283 DEBUG:boto:Token: None 2023-05-31T20:44:49.284 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:44:49 GMT /pklxkd-59/obj34?versionId=null 2023-05-31T20:44:49.284 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:3Ch4kevzsDU0JnakZQcUN+9hTAc= 2023-05-31T20:44:49.285 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:44:49 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:3Ch4kevzsDU0JnakZQcUN+9hTAc=', 'Content-Length': '0'} 2023-05-31T20:44:49.287 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:43:07 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003717d6299f7be04d-006477b1c1-1c17-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:44:49 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:44:49.287 DEBUG:boto:path=/pklxkd-59/obj34 2023-05-31T20:44:49.288 DEBUG:boto:auth_path=/pklxkd-59/obj34 2023-05-31T20:44:49.288 DEBUG:boto:path=/pklxkd-59/obj34?versionId=null 2023-05-31T20:44:49.288 DEBUG:boto:auth_path=/pklxkd-59/obj34?versionId=null 2023-05-31T20:44:49.289 DEBUG:boto:Method: GET 2023-05-31T20:44:49.289 DEBUG:boto:Path: /pklxkd-59/obj34?versionId=null 2023-05-31T20:44:49.289 DEBUG:boto:Data: 2023-05-31T20:44:49.290 DEBUG:boto:Headers: {} 2023-05-31T20:44:49.290 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:44:49.290 DEBUG:boto:Port: 8000 2023-05-31T20:44:49.291 DEBUG:boto:Params: {} 2023-05-31T20:44:49.291 DEBUG:boto:Token: None 2023-05-31T20:44:49.292 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:44:49 GMT /pklxkd-59/obj34?versionId=null 2023-05-31T20:44:49.292 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:3Ch4kevzsDU0JnakZQcUN+9hTAc= 2023-05-31T20:44:49.292 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:44:49 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:3Ch4kevzsDU0JnakZQcUN+9hTAc=', 'Content-Length': '0'} 2023-05-31T20:44:49.294 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:43:07 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c289e6c64cf75f44-006477b1c1-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:44:49 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:44:49.295 DEBUG:boto:path=/pklxkd-59/ 2023-05-31T20:44:49.295 DEBUG:boto:auth_path=/pklxkd-59/ 2023-05-31T20:44:49.296 DEBUG:boto:path=/pklxkd-59/?prefix=obj34 2023-05-31T20:44:49.296 DEBUG:boto:auth_path=/pklxkd-59/?prefix=obj34 2023-05-31T20:44:49.296 DEBUG:boto:Method: GET 2023-05-31T20:44:49.297 DEBUG:boto:Path: /pklxkd-59/?prefix=obj34 2023-05-31T20:44:49.297 DEBUG:boto:Data: 2023-05-31T20:44:49.297 DEBUG:boto:Headers: {} 2023-05-31T20:44:49.298 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:44:49.298 DEBUG:boto:Port: 8001 2023-05-31T20:44:49.298 DEBUG:boto:Params: {} 2023-05-31T20:44:49.299 DEBUG:boto:Token: None 2023-05-31T20:44:49.299 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:44:49 GMT /pklxkd-59/ 2023-05-31T20:44:49.300 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:VK2eJyJ+BXpP/6492MS9jzEZ3kw= 2023-05-31T20:44:49.300 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:44:49 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:VK2eJyJ+BXpP/6492MS9jzEZ3kw=', 'Content-Length': '0'} 2023-05-31T20:44:49.302 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000009b65203e2abdb881-006477b1c1-1c17-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:44:49 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:44:49.303 DEBUG:boto:b'pklxkd-59obj341000falseobj342023-05-31T20:43:07.994Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:44:49.303 DEBUG:boto:path=/pklxkd-59/ 2023-05-31T20:44:49.304 DEBUG:boto:auth_path=/pklxkd-59/ 2023-05-31T20:44:49.304 DEBUG:boto:path=/pklxkd-59/?prefix=obj34 2023-05-31T20:44:49.305 DEBUG:boto:auth_path=/pklxkd-59/?prefix=obj34 2023-05-31T20:44:49.305 DEBUG:boto:Method: GET 2023-05-31T20:44:49.305 DEBUG:boto:Path: /pklxkd-59/?prefix=obj34 2023-05-31T20:44:49.306 DEBUG:boto:Data: 2023-05-31T20:44:49.306 DEBUG:boto:Headers: {} 2023-05-31T20:44:49.306 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:44:49.307 DEBUG:boto:Port: 8000 2023-05-31T20:44:49.307 DEBUG:boto:Params: {} 2023-05-31T20:44:49.307 DEBUG:boto:Token: None 2023-05-31T20:44:49.308 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:44:49 GMT /pklxkd-59/ 2023-05-31T20:44:49.308 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:VK2eJyJ+BXpP/6492MS9jzEZ3kw= 2023-05-31T20:44:49.309 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:44:49 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:VK2eJyJ+BXpP/6492MS9jzEZ3kw=', 'Content-Length': '0'} 2023-05-31T20:44:49.311 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000059ee1d75d7e948f3-006477b1c1-1630-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:44:49 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:44:49.312 DEBUG:boto:b'pklxkd-59obj341000falseobj342023-05-31T20:43:07.994Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:44:49.313 DEBUG:boto:path=/pklxkd-59/obj34 2023-05-31T20:44:49.313 DEBUG:boto:auth_path=/pklxkd-59/obj34 2023-05-31T20:44:49.313 DEBUG:boto:Method: HEAD 2023-05-31T20:44:49.314 DEBUG:boto:Path: /pklxkd-59/obj34 2023-05-31T20:44:49.314 DEBUG:boto:Data: 2023-05-31T20:44:49.314 DEBUG:boto:Headers: {} 2023-05-31T20:44:49.315 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:44:49.315 DEBUG:boto:Port: 8001 2023-05-31T20:44:49.315 DEBUG:boto:Params: {} 2023-05-31T20:44:49.316 DEBUG:boto:Token: None 2023-05-31T20:44:49.316 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:44:49 GMT /pklxkd-59/obj34 2023-05-31T20:44:49.316 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:y8UCI05rcDQzLXcSpLBmJ6j9Eb4= 2023-05-31T20:44:49.317 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:44:49 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:y8UCI05rcDQzLXcSpLBmJ6j9Eb4=', 'Content-Length': '0'} 2023-05-31T20:44:49.319 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:43:07 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ef4831581c221303-006477b1c1-1c17-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:44:49 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:44:49.319 DEBUG:boto:path=/pklxkd-59/obj34 2023-05-31T20:44:49.319 DEBUG:boto:auth_path=/pklxkd-59/obj34 2023-05-31T20:44:49.320 DEBUG:boto:Method: HEAD 2023-05-31T20:44:49.320 DEBUG:boto:Path: /pklxkd-59/obj34 2023-05-31T20:44:49.321 DEBUG:boto:Data: 2023-05-31T20:44:49.321 DEBUG:boto:Headers: {} 2023-05-31T20:44:49.321 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:44:49.322 DEBUG:boto:Port: 8000 2023-05-31T20:44:49.322 DEBUG:boto:Params: {} 2023-05-31T20:44:49.322 DEBUG:boto:Token: None 2023-05-31T20:44:49.323 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:44:49 GMT /pklxkd-59/obj34 2023-05-31T20:44:49.323 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:y8UCI05rcDQzLXcSpLBmJ6j9Eb4= 2023-05-31T20:44:49.323 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:44:49 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:y8UCI05rcDQzLXcSpLBmJ6j9Eb4=', 'Content-Length': '0'} 2023-05-31T20:44:49.325 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:43:07 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001fdc3d8e692a1905-006477b1c1-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:44:49 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:44:49.326 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj34 2023-05-31T20:44:49.326 DEBUG:boto:path=/pklxkd-59/obj34 2023-05-31T20:44:49.326 DEBUG:boto:auth_path=/pklxkd-59/obj34 2023-05-31T20:44:49.327 DEBUG:boto:Method: GET 2023-05-31T20:44:49.327 DEBUG:boto:Path: /pklxkd-59/obj34 2023-05-31T20:44:49.327 DEBUG:boto:Data: 2023-05-31T20:44:49.328 DEBUG:boto:Headers: {} 2023-05-31T20:44:49.328 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:44:49.329 DEBUG:boto:Port: 8001 2023-05-31T20:44:49.329 DEBUG:boto:Params: {} 2023-05-31T20:44:49.329 DEBUG:boto:Token: None 2023-05-31T20:44:49.330 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:44:49 GMT /pklxkd-59/obj34 2023-05-31T20:44:49.330 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:L7LOHVNNxsjtcVSAvWYN6HWjMgQ= 2023-05-31T20:44:49.331 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:44:49 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:L7LOHVNNxsjtcVSAvWYN6HWjMgQ=', 'Content-Length': '0'} 2023-05-31T20:44:49.332 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:43:07 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008fc446d77eff1fcd-006477b1c1-1c17-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:44:49 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:44:49.333 DEBUG:boto:path=/pklxkd-59/obj34 2023-05-31T20:44:49.333 DEBUG:boto:auth_path=/pklxkd-59/obj34 2023-05-31T20:44:49.334 DEBUG:boto:Method: GET 2023-05-31T20:44:49.334 DEBUG:boto:Path: /pklxkd-59/obj34 2023-05-31T20:44:49.334 DEBUG:boto:Data: 2023-05-31T20:44:49.335 DEBUG:boto:Headers: {} 2023-05-31T20:44:49.335 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:44:49.336 DEBUG:boto:Port: 8000 2023-05-31T20:44:49.336 DEBUG:boto:Params: {} 2023-05-31T20:44:49.336 DEBUG:boto:Token: None 2023-05-31T20:44:49.337 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:44:49 GMT /pklxkd-59/obj34 2023-05-31T20:44:49.337 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:L7LOHVNNxsjtcVSAvWYN6HWjMgQ= 2023-05-31T20:44:49.338 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:44:49 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:L7LOHVNNxsjtcVSAvWYN6HWjMgQ=', 'Content-Length': '0'} 2023-05-31T20:44:49.339 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:43:07 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009656fc6ff854ae20-006477b1c1-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:44:49 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:44:49.340 DEBUG:boto:path=/pklxkd-59/ 2023-05-31T20:44:49.340 DEBUG:boto:auth_path=/pklxkd-59/ 2023-05-31T20:44:49.341 DEBUG:boto:path=/pklxkd-59/?prefix=obj34 2023-05-31T20:44:49.341 DEBUG:boto:auth_path=/pklxkd-59/?prefix=obj34 2023-05-31T20:44:49.341 DEBUG:boto:Method: GET 2023-05-31T20:44:49.342 DEBUG:boto:Path: /pklxkd-59/?prefix=obj34 2023-05-31T20:44:49.342 DEBUG:boto:Data: 2023-05-31T20:44:49.342 DEBUG:boto:Headers: {} 2023-05-31T20:44:49.343 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:44:49.343 DEBUG:boto:Port: 8001 2023-05-31T20:44:49.343 DEBUG:boto:Params: {} 2023-05-31T20:44:49.344 DEBUG:boto:Token: None 2023-05-31T20:44:49.344 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:44:49 GMT /pklxkd-59/ 2023-05-31T20:44:49.345 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:VK2eJyJ+BXpP/6492MS9jzEZ3kw= 2023-05-31T20:44:49.345 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:44:49 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:VK2eJyJ+BXpP/6492MS9jzEZ3kw=', 'Content-Length': '0'} 2023-05-31T20:44:49.347 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000082faa4531e8c4ae2-006477b1c1-1c17-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:44:49 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:44:49.348 DEBUG:boto:b'pklxkd-59obj341000falseobj342023-05-31T20:43:07.994Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:44:49.348 DEBUG:boto:path=/pklxkd-59/ 2023-05-31T20:44:49.349 DEBUG:boto:auth_path=/pklxkd-59/ 2023-05-31T20:44:49.349 DEBUG:boto:path=/pklxkd-59/?prefix=obj34 2023-05-31T20:44:49.350 DEBUG:boto:auth_path=/pklxkd-59/?prefix=obj34 2023-05-31T20:44:49.350 DEBUG:boto:Method: GET 2023-05-31T20:44:49.350 DEBUG:boto:Path: /pklxkd-59/?prefix=obj34 2023-05-31T20:44:49.351 DEBUG:boto:Data: 2023-05-31T20:44:49.351 DEBUG:boto:Headers: {} 2023-05-31T20:44:49.351 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:44:49.352 DEBUG:boto:Port: 8000 2023-05-31T20:44:49.352 DEBUG:boto:Params: {} 2023-05-31T20:44:49.352 DEBUG:boto:Token: None 2023-05-31T20:44:49.353 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:44:49 GMT /pklxkd-59/ 2023-05-31T20:44:49.353 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:VK2eJyJ+BXpP/6492MS9jzEZ3kw= 2023-05-31T20:44:49.354 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:44:49 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:VK2eJyJ+BXpP/6492MS9jzEZ3kw=', 'Content-Length': '0'} 2023-05-31T20:44:49.356 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000380a32336aa39a2a-006477b1c1-1630-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:44:49 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:44:49.357 DEBUG:boto:b'pklxkd-59obj341000falseobj342023-05-31T20:43:07.994Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:44:49.358 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj44 2023-05-31T20:44:49.358 DEBUG:boto:path=/pklxkd-59/obj44 2023-05-31T20:44:49.358 DEBUG:boto:auth_path=/pklxkd-59/obj44 2023-05-31T20:44:49.359 DEBUG:boto:path=/pklxkd-59/obj44?versionId=null 2023-05-31T20:44:49.359 DEBUG:boto:auth_path=/pklxkd-59/obj44?versionId=null 2023-05-31T20:44:49.359 DEBUG:boto:Method: GET 2023-05-31T20:44:49.360 DEBUG:boto:Path: /pklxkd-59/obj44?versionId=null 2023-05-31T20:44:49.360 DEBUG:boto:Data: 2023-05-31T20:44:49.360 DEBUG:boto:Headers: {} 2023-05-31T20:44:49.361 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:44:49.361 DEBUG:boto:Port: 8001 2023-05-31T20:44:49.361 DEBUG:boto:Params: {} 2023-05-31T20:44:49.362 DEBUG:boto:Token: None 2023-05-31T20:44:49.362 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:44:49 GMT /pklxkd-59/obj44?versionId=null 2023-05-31T20:44:49.363 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:Cv5+E5v52T6A5Eir8O3sKSSqDHA= 2023-05-31T20:44:49.363 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:44:49 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:Cv5+E5v52T6A5Eir8O3sKSSqDHA=', 'Content-Length': '0'} 2023-05-31T20:44:49.365 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:39:26 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000007a095ffe12d63d3-006477b1c1-1c17-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:44:49 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:44:49.366 DEBUG:boto:path=/pklxkd-59/obj44 2023-05-31T20:44:49.366 DEBUG:boto:auth_path=/pklxkd-59/obj44 2023-05-31T20:44:49.366 DEBUG:boto:path=/pklxkd-59/obj44?versionId=null 2023-05-31T20:44:49.367 DEBUG:boto:auth_path=/pklxkd-59/obj44?versionId=null 2023-05-31T20:44:49.367 DEBUG:boto:Method: GET 2023-05-31T20:44:49.367 DEBUG:boto:Path: /pklxkd-59/obj44?versionId=null 2023-05-31T20:44:49.368 DEBUG:boto:Data: 2023-05-31T20:44:49.368 DEBUG:boto:Headers: {} 2023-05-31T20:44:49.369 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:44:49.369 DEBUG:boto:Port: 8000 2023-05-31T20:44:49.369 DEBUG:boto:Params: {} 2023-05-31T20:44:49.370 DEBUG:boto:Token: None 2023-05-31T20:44:49.370 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:44:49 GMT /pklxkd-59/obj44?versionId=null 2023-05-31T20:44:49.370 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:Cv5+E5v52T6A5Eir8O3sKSSqDHA= 2023-05-31T20:44:49.371 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:44:49 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:Cv5+E5v52T6A5Eir8O3sKSSqDHA=', 'Content-Length': '0'} 2023-05-31T20:44:49.373 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:39:26 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003b6219cd5d905e1c-006477b1c1-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:44:49 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:44:49.416 DEBUG:boto:path=/pklxkd-59/obj44 2023-05-31T20:44:49.417 DEBUG:boto:auth_path=/pklxkd-59/obj44 2023-05-31T20:44:49.417 DEBUG:boto:path=/pklxkd-59/obj44?versionId=null 2023-05-31T20:44:49.417 DEBUG:boto:auth_path=/pklxkd-59/obj44?versionId=null 2023-05-31T20:44:49.418 DEBUG:boto:Method: HEAD 2023-05-31T20:44:49.418 DEBUG:boto:Path: /pklxkd-59/obj44?versionId=null 2023-05-31T20:44:49.418 DEBUG:boto:Data: 2023-05-31T20:44:49.419 DEBUG:boto:Headers: {} 2023-05-31T20:44:49.419 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:44:49.419 DEBUG:boto:Port: 8001 2023-05-31T20:44:49.420 DEBUG:boto:Params: {} 2023-05-31T20:44:49.420 DEBUG:boto:Token: None 2023-05-31T20:44:49.420 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:44:49 GMT /pklxkd-59/obj44?versionId=null 2023-05-31T20:44:49.421 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:7+O2Wu1v9YMcu3iQRS8nILhCHNo= 2023-05-31T20:44:49.421 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:44:49 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:7+O2Wu1v9YMcu3iQRS8nILhCHNo=', 'Content-Length': '0'} 2023-05-31T20:44:49.423 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:39:26 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000099d343a112af58dd-006477b1c1-1c17-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:44:49 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:44:49.424 DEBUG:boto:path=/pklxkd-59/obj44 2023-05-31T20:44:49.424 DEBUG:boto:auth_path=/pklxkd-59/obj44 2023-05-31T20:44:49.424 DEBUG:boto:path=/pklxkd-59/obj44?versionId=null 2023-05-31T20:44:49.425 DEBUG:boto:auth_path=/pklxkd-59/obj44?versionId=null 2023-05-31T20:44:49.425 DEBUG:boto:Method: HEAD 2023-05-31T20:44:49.425 DEBUG:boto:Path: /pklxkd-59/obj44?versionId=null 2023-05-31T20:44:49.426 DEBUG:boto:Data: 2023-05-31T20:44:49.426 DEBUG:boto:Headers: {} 2023-05-31T20:44:49.426 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:44:49.427 DEBUG:boto:Port: 8000 2023-05-31T20:44:49.427 DEBUG:boto:Params: {} 2023-05-31T20:44:49.427 DEBUG:boto:Token: None 2023-05-31T20:44:49.428 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:44:49 GMT /pklxkd-59/obj44?versionId=null 2023-05-31T20:44:49.428 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:7+O2Wu1v9YMcu3iQRS8nILhCHNo= 2023-05-31T20:44:49.428 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:44:49 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:7+O2Wu1v9YMcu3iQRS8nILhCHNo=', 'Content-Length': '0'} 2023-05-31T20:44:49.430 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:39:26 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c16220699cf6b1d8-006477b1c1-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:44:49 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:44:49.431 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj44 2023-05-31T20:44:49.431 DEBUG:boto:path=/pklxkd-59/obj44 2023-05-31T20:44:49.431 DEBUG:boto:auth_path=/pklxkd-59/obj44 2023-05-31T20:44:49.432 DEBUG:boto:path=/pklxkd-59/obj44?versionId=null 2023-05-31T20:44:49.432 DEBUG:boto:auth_path=/pklxkd-59/obj44?versionId=null 2023-05-31T20:44:49.432 DEBUG:boto:Method: GET 2023-05-31T20:44:49.433 DEBUG:boto:Path: /pklxkd-59/obj44?versionId=null 2023-05-31T20:44:49.433 DEBUG:boto:Data: 2023-05-31T20:44:49.434 DEBUG:boto:Headers: {} 2023-05-31T20:44:49.434 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:44:49.434 DEBUG:boto:Port: 8001 2023-05-31T20:44:49.435 DEBUG:boto:Params: {} 2023-05-31T20:44:49.435 DEBUG:boto:Token: None 2023-05-31T20:44:49.435 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:44:49 GMT /pklxkd-59/obj44?versionId=null 2023-05-31T20:44:49.436 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:Cv5+E5v52T6A5Eir8O3sKSSqDHA= 2023-05-31T20:44:49.436 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:44:49 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:Cv5+E5v52T6A5Eir8O3sKSSqDHA=', 'Content-Length': '0'} 2023-05-31T20:44:49.438 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:39:26 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000bac8fee7588cf233-006477b1c1-1c17-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:44:49 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:44:49.439 DEBUG:boto:path=/pklxkd-59/obj44 2023-05-31T20:44:49.439 DEBUG:boto:auth_path=/pklxkd-59/obj44 2023-05-31T20:44:49.440 DEBUG:boto:path=/pklxkd-59/obj44?versionId=null 2023-05-31T20:44:49.440 DEBUG:boto:auth_path=/pklxkd-59/obj44?versionId=null 2023-05-31T20:44:49.440 DEBUG:boto:Method: GET 2023-05-31T20:44:49.441 DEBUG:boto:Path: /pklxkd-59/obj44?versionId=null 2023-05-31T20:44:49.441 DEBUG:boto:Data: 2023-05-31T20:44:49.441 DEBUG:boto:Headers: {} 2023-05-31T20:44:49.442 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:44:49.442 DEBUG:boto:Port: 8000 2023-05-31T20:44:49.442 DEBUG:boto:Params: {} 2023-05-31T20:44:49.443 DEBUG:boto:Token: None 2023-05-31T20:44:49.443 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:44:49 GMT /pklxkd-59/obj44?versionId=null 2023-05-31T20:44:49.444 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:Cv5+E5v52T6A5Eir8O3sKSSqDHA= 2023-05-31T20:44:49.444 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:44:49 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:Cv5+E5v52T6A5Eir8O3sKSSqDHA=', 'Content-Length': '0'} 2023-05-31T20:44:49.446 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:39:26 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000ae532f1ef52dcc7-006477b1c1-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:44:49 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:44:49.446 DEBUG:boto:path=/pklxkd-59/ 2023-05-31T20:44:49.447 DEBUG:boto:auth_path=/pklxkd-59/ 2023-05-31T20:44:49.447 DEBUG:boto:path=/pklxkd-59/?prefix=obj44 2023-05-31T20:44:49.448 DEBUG:boto:auth_path=/pklxkd-59/?prefix=obj44 2023-05-31T20:44:49.448 DEBUG:boto:Method: GET 2023-05-31T20:44:49.448 DEBUG:boto:Path: /pklxkd-59/?prefix=obj44 2023-05-31T20:44:49.449 DEBUG:boto:Data: 2023-05-31T20:44:49.449 DEBUG:boto:Headers: {} 2023-05-31T20:44:49.449 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:44:49.450 DEBUG:boto:Port: 8001 2023-05-31T20:44:49.450 DEBUG:boto:Params: {} 2023-05-31T20:44:49.450 DEBUG:boto:Token: None 2023-05-31T20:44:49.451 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:44:49 GMT /pklxkd-59/ 2023-05-31T20:44:49.451 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:VK2eJyJ+BXpP/6492MS9jzEZ3kw= 2023-05-31T20:44:49.452 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:44:49 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:VK2eJyJ+BXpP/6492MS9jzEZ3kw=', 'Content-Length': '0'} 2023-05-31T20:44:49.454 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000049e3cf22eeda1b81-006477b1c1-1c17-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:44:49 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:44:49.454 DEBUG:boto:b'pklxkd-59obj441000falseobj442023-05-31T20:39:26.565Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:44:49.455 DEBUG:boto:path=/pklxkd-59/ 2023-05-31T20:44:49.456 DEBUG:boto:auth_path=/pklxkd-59/ 2023-05-31T20:44:49.456 DEBUG:boto:path=/pklxkd-59/?prefix=obj44 2023-05-31T20:44:49.456 DEBUG:boto:auth_path=/pklxkd-59/?prefix=obj44 2023-05-31T20:44:49.457 DEBUG:boto:Method: GET 2023-05-31T20:44:49.457 DEBUG:boto:Path: /pklxkd-59/?prefix=obj44 2023-05-31T20:44:49.458 DEBUG:boto:Data: 2023-05-31T20:44:49.458 DEBUG:boto:Headers: {} 2023-05-31T20:44:49.458 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:44:49.459 DEBUG:boto:Port: 8000 2023-05-31T20:44:49.459 DEBUG:boto:Params: {} 2023-05-31T20:44:49.459 DEBUG:boto:Token: None 2023-05-31T20:44:49.460 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:44:49 GMT /pklxkd-59/ 2023-05-31T20:44:49.460 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:VK2eJyJ+BXpP/6492MS9jzEZ3kw= 2023-05-31T20:44:49.461 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:44:49 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:VK2eJyJ+BXpP/6492MS9jzEZ3kw=', 'Content-Length': '0'} 2023-05-31T20:44:49.463 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000007b480d5e81acff83-006477b1c1-1630-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:44:49 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:44:49.464 DEBUG:boto:b'pklxkd-59obj441000falseobj442023-05-31T20:39:26.565Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:44:49.464 DEBUG:boto:path=/pklxkd-59/obj44 2023-05-31T20:44:49.465 DEBUG:boto:auth_path=/pklxkd-59/obj44 2023-05-31T20:44:49.465 DEBUG:boto:Method: HEAD 2023-05-31T20:44:49.466 DEBUG:boto:Path: /pklxkd-59/obj44 2023-05-31T20:44:49.466 DEBUG:boto:Data: 2023-05-31T20:44:49.466 DEBUG:boto:Headers: {} 2023-05-31T20:44:49.467 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:44:49.467 DEBUG:boto:Port: 8001 2023-05-31T20:44:49.467 DEBUG:boto:Params: {} 2023-05-31T20:44:49.468 DEBUG:boto:Token: None 2023-05-31T20:44:49.468 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:44:49 GMT /pklxkd-59/obj44 2023-05-31T20:44:49.468 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:8KKqyfqq8QYFdO7FwPSTAEAF448= 2023-05-31T20:44:49.469 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:44:49 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:8KKqyfqq8QYFdO7FwPSTAEAF448=', 'Content-Length': '0'} 2023-05-31T20:44:49.470 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:39:26 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000066161aa75208da3a-006477b1c1-1c17-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:44:49 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:44:49.471 DEBUG:boto:path=/pklxkd-59/obj44 2023-05-31T20:44:49.471 DEBUG:boto:auth_path=/pklxkd-59/obj44 2023-05-31T20:44:49.472 DEBUG:boto:Method: HEAD 2023-05-31T20:44:49.472 DEBUG:boto:Path: /pklxkd-59/obj44 2023-05-31T20:44:49.472 DEBUG:boto:Data: 2023-05-31T20:44:49.473 DEBUG:boto:Headers: {} 2023-05-31T20:44:49.473 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:44:49.473 DEBUG:boto:Port: 8000 2023-05-31T20:44:49.474 DEBUG:boto:Params: {} 2023-05-31T20:44:49.474 DEBUG:boto:Token: None 2023-05-31T20:44:49.475 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:44:49 GMT /pklxkd-59/obj44 2023-05-31T20:44:49.475 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:8KKqyfqq8QYFdO7FwPSTAEAF448= 2023-05-31T20:44:49.475 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:44:49 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:8KKqyfqq8QYFdO7FwPSTAEAF448=', 'Content-Length': '0'} 2023-05-31T20:44:49.477 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:39:26 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004a54b84dd6ce8363-006477b1c1-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:44:49 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:44:49.478 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj44 2023-05-31T20:44:49.478 DEBUG:boto:path=/pklxkd-59/obj44 2023-05-31T20:44:49.478 DEBUG:boto:auth_path=/pklxkd-59/obj44 2023-05-31T20:44:49.479 DEBUG:boto:Method: GET 2023-05-31T20:44:49.479 DEBUG:boto:Path: /pklxkd-59/obj44 2023-05-31T20:44:49.480 DEBUG:boto:Data: 2023-05-31T20:44:49.480 DEBUG:boto:Headers: {} 2023-05-31T20:44:49.480 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:44:49.481 DEBUG:boto:Port: 8001 2023-05-31T20:44:49.481 DEBUG:boto:Params: {} 2023-05-31T20:44:49.481 DEBUG:boto:Token: None 2023-05-31T20:44:49.482 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:44:49 GMT /pklxkd-59/obj44 2023-05-31T20:44:49.482 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:HEzBZHeztkG9xMcHBpdvumgr5Rk= 2023-05-31T20:44:49.483 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:44:49 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:HEzBZHeztkG9xMcHBpdvumgr5Rk=', 'Content-Length': '0'} 2023-05-31T20:44:49.484 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:39:26 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d1f4b4f3695850bc-006477b1c1-1c17-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:44:49 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:44:49.485 DEBUG:boto:path=/pklxkd-59/obj44 2023-05-31T20:44:49.486 DEBUG:boto:auth_path=/pklxkd-59/obj44 2023-05-31T20:44:49.486 DEBUG:boto:Method: GET 2023-05-31T20:44:49.486 DEBUG:boto:Path: /pklxkd-59/obj44 2023-05-31T20:44:49.487 DEBUG:boto:Data: 2023-05-31T20:44:49.487 DEBUG:boto:Headers: {} 2023-05-31T20:44:49.487 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:44:49.488 DEBUG:boto:Port: 8000 2023-05-31T20:44:49.488 DEBUG:boto:Params: {} 2023-05-31T20:44:49.488 DEBUG:boto:Token: None 2023-05-31T20:44:49.489 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:44:49 GMT /pklxkd-59/obj44 2023-05-31T20:44:49.489 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:HEzBZHeztkG9xMcHBpdvumgr5Rk= 2023-05-31T20:44:49.490 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:44:49 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:HEzBZHeztkG9xMcHBpdvumgr5Rk=', 'Content-Length': '0'} 2023-05-31T20:44:49.492 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:39:26 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008659460d84a97dc0-006477b1c1-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:44:49 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:44:49.492 DEBUG:boto:path=/pklxkd-59/ 2023-05-31T20:44:49.493 DEBUG:boto:auth_path=/pklxkd-59/ 2023-05-31T20:44:49.493 DEBUG:boto:path=/pklxkd-59/?prefix=obj44 2023-05-31T20:44:49.493 DEBUG:boto:auth_path=/pklxkd-59/?prefix=obj44 2023-05-31T20:44:49.494 DEBUG:boto:Method: GET 2023-05-31T20:44:49.494 DEBUG:boto:Path: /pklxkd-59/?prefix=obj44 2023-05-31T20:44:49.494 DEBUG:boto:Data: 2023-05-31T20:44:49.495 DEBUG:boto:Headers: {} 2023-05-31T20:44:49.495 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:44:49.496 DEBUG:boto:Port: 8001 2023-05-31T20:44:49.496 DEBUG:boto:Params: {} 2023-05-31T20:44:49.497 DEBUG:boto:Token: None 2023-05-31T20:44:49.497 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:44:49 GMT /pklxkd-59/ 2023-05-31T20:44:49.497 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:VK2eJyJ+BXpP/6492MS9jzEZ3kw= 2023-05-31T20:44:49.498 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:44:49 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:VK2eJyJ+BXpP/6492MS9jzEZ3kw=', 'Content-Length': '0'} 2023-05-31T20:44:49.500 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a65f37b6d10b3618-006477b1c1-1c17-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:44:49 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:44:49.544 DEBUG:boto:b'pklxkd-59obj441000falseobj442023-05-31T20:39:26.565Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:44:49.545 DEBUG:boto:path=/pklxkd-59/ 2023-05-31T20:44:49.545 DEBUG:boto:auth_path=/pklxkd-59/ 2023-05-31T20:44:49.546 DEBUG:boto:path=/pklxkd-59/?prefix=obj44 2023-05-31T20:44:49.546 DEBUG:boto:auth_path=/pklxkd-59/?prefix=obj44 2023-05-31T20:44:49.546 DEBUG:boto:Method: GET 2023-05-31T20:44:49.547 DEBUG:boto:Path: /pklxkd-59/?prefix=obj44 2023-05-31T20:44:49.547 DEBUG:boto:Data: 2023-05-31T20:44:49.547 DEBUG:boto:Headers: {} 2023-05-31T20:44:49.548 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:44:49.548 DEBUG:boto:Port: 8000 2023-05-31T20:44:49.549 DEBUG:boto:Params: {} 2023-05-31T20:44:49.549 DEBUG:boto:Token: None 2023-05-31T20:44:49.549 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:44:49 GMT /pklxkd-59/ 2023-05-31T20:44:49.550 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:VK2eJyJ+BXpP/6492MS9jzEZ3kw= 2023-05-31T20:44:49.550 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:44:49 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:VK2eJyJ+BXpP/6492MS9jzEZ3kw=', 'Content-Length': '0'} 2023-05-31T20:44:49.553 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f03523aba0a0da99-006477b1c1-1630-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:44:49 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:44:49.553 DEBUG:boto:b'pklxkd-59obj441000falseobj442023-05-31T20:39:26.565Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:44:49.554 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj51 2023-05-31T20:44:49.554 DEBUG:boto:path=/pklxkd-59/obj51 2023-05-31T20:44:49.555 DEBUG:boto:auth_path=/pklxkd-59/obj51 2023-05-31T20:44:49.555 DEBUG:boto:path=/pklxkd-59/obj51?versionId=null 2023-05-31T20:44:49.555 DEBUG:boto:auth_path=/pklxkd-59/obj51?versionId=null 2023-05-31T20:44:49.556 DEBUG:boto:Method: GET 2023-05-31T20:44:49.556 DEBUG:boto:Path: /pklxkd-59/obj51?versionId=null 2023-05-31T20:44:49.557 DEBUG:boto:Data: 2023-05-31T20:44:49.557 DEBUG:boto:Headers: {} 2023-05-31T20:44:49.557 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:44:49.558 DEBUG:boto:Port: 8001 2023-05-31T20:44:49.558 DEBUG:boto:Params: {} 2023-05-31T20:44:49.559 DEBUG:boto:Token: None 2023-05-31T20:44:49.559 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:44:49 GMT /pklxkd-59/obj51?versionId=null 2023-05-31T20:44:49.559 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:sSAKckmzQe4B28ZjkGDnPDqRnW4= 2023-05-31T20:44:49.560 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:44:49 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:sSAKckmzQe4B28ZjkGDnPDqRnW4=', 'Content-Length': '0'} 2023-05-31T20:44:49.562 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:43:08 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000055218dd34747978e-006477b1c1-1c17-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:44:49 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:44:49.563 DEBUG:boto:path=/pklxkd-59/obj51 2023-05-31T20:44:49.563 DEBUG:boto:auth_path=/pklxkd-59/obj51 2023-05-31T20:44:49.563 DEBUG:boto:path=/pklxkd-59/obj51?versionId=null 2023-05-31T20:44:49.564 DEBUG:boto:auth_path=/pklxkd-59/obj51?versionId=null 2023-05-31T20:44:49.564 DEBUG:boto:Method: GET 2023-05-31T20:44:49.565 DEBUG:boto:Path: /pklxkd-59/obj51?versionId=null 2023-05-31T20:44:49.565 DEBUG:boto:Data: 2023-05-31T20:44:49.565 DEBUG:boto:Headers: {} 2023-05-31T20:44:49.566 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:44:49.566 DEBUG:boto:Port: 8000 2023-05-31T20:44:49.566 DEBUG:boto:Params: {} 2023-05-31T20:44:49.567 DEBUG:boto:Token: None 2023-05-31T20:44:49.567 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:44:49 GMT /pklxkd-59/obj51?versionId=null 2023-05-31T20:44:49.568 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:sSAKckmzQe4B28ZjkGDnPDqRnW4= 2023-05-31T20:44:49.568 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:44:49 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:sSAKckmzQe4B28ZjkGDnPDqRnW4=', 'Content-Length': '0'} 2023-05-31T20:44:49.570 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:43:08 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000eaa1dd75a720e745-006477b1c1-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:44:49 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:44:49.612 DEBUG:boto:path=/pklxkd-59/obj51 2023-05-31T20:44:49.613 DEBUG:boto:auth_path=/pklxkd-59/obj51 2023-05-31T20:44:49.613 DEBUG:boto:path=/pklxkd-59/obj51?versionId=null 2023-05-31T20:44:49.613 DEBUG:boto:auth_path=/pklxkd-59/obj51?versionId=null 2023-05-31T20:44:49.614 DEBUG:boto:Method: HEAD 2023-05-31T20:44:49.614 DEBUG:boto:Path: /pklxkd-59/obj51?versionId=null 2023-05-31T20:44:49.614 DEBUG:boto:Data: 2023-05-31T20:44:49.615 DEBUG:boto:Headers: {} 2023-05-31T20:44:49.615 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:44:49.615 DEBUG:boto:Port: 8001 2023-05-31T20:44:49.616 DEBUG:boto:Params: {} 2023-05-31T20:44:49.616 DEBUG:boto:Token: None 2023-05-31T20:44:49.616 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:44:49 GMT /pklxkd-59/obj51?versionId=null 2023-05-31T20:44:49.617 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:HJ6iwu7gLfCAY4JtttNPErR3wrs= 2023-05-31T20:44:49.617 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:44:49 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:HJ6iwu7gLfCAY4JtttNPErR3wrs=', 'Content-Length': '0'} 2023-05-31T20:44:49.619 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:43:08 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000035adfed98db5b1ba-006477b1c1-1c17-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:44:49 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:44:49.620 DEBUG:boto:path=/pklxkd-59/obj51 2023-05-31T20:44:49.620 DEBUG:boto:auth_path=/pklxkd-59/obj51 2023-05-31T20:44:49.620 DEBUG:boto:path=/pklxkd-59/obj51?versionId=null 2023-05-31T20:44:49.621 DEBUG:boto:auth_path=/pklxkd-59/obj51?versionId=null 2023-05-31T20:44:49.621 DEBUG:boto:Method: HEAD 2023-05-31T20:44:49.621 DEBUG:boto:Path: /pklxkd-59/obj51?versionId=null 2023-05-31T20:44:49.622 DEBUG:boto:Data: 2023-05-31T20:44:49.622 DEBUG:boto:Headers: {} 2023-05-31T20:44:49.622 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:44:49.623 DEBUG:boto:Port: 8000 2023-05-31T20:44:49.623 DEBUG:boto:Params: {} 2023-05-31T20:44:49.623 DEBUG:boto:Token: None 2023-05-31T20:44:49.624 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:44:49 GMT /pklxkd-59/obj51?versionId=null 2023-05-31T20:44:49.624 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:HJ6iwu7gLfCAY4JtttNPErR3wrs= 2023-05-31T20:44:49.624 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:44:49 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:HJ6iwu7gLfCAY4JtttNPErR3wrs=', 'Content-Length': '0'} 2023-05-31T20:44:49.626 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:43:08 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000068d31bd65446419f-006477b1c1-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:44:49 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:44:49.627 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj51 2023-05-31T20:44:49.627 DEBUG:boto:path=/pklxkd-59/obj51 2023-05-31T20:44:49.628 DEBUG:boto:auth_path=/pklxkd-59/obj51 2023-05-31T20:44:49.628 DEBUG:boto:path=/pklxkd-59/obj51?versionId=null 2023-05-31T20:44:49.628 DEBUG:boto:auth_path=/pklxkd-59/obj51?versionId=null 2023-05-31T20:44:49.629 DEBUG:boto:Method: GET 2023-05-31T20:44:49.629 DEBUG:boto:Path: /pklxkd-59/obj51?versionId=null 2023-05-31T20:44:49.629 DEBUG:boto:Data: 2023-05-31T20:44:49.630 DEBUG:boto:Headers: {} 2023-05-31T20:44:49.630 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:44:49.630 DEBUG:boto:Port: 8001 2023-05-31T20:44:49.631 DEBUG:boto:Params: {} 2023-05-31T20:44:49.631 DEBUG:boto:Token: None 2023-05-31T20:44:49.632 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:44:49 GMT /pklxkd-59/obj51?versionId=null 2023-05-31T20:44:49.632 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:sSAKckmzQe4B28ZjkGDnPDqRnW4= 2023-05-31T20:44:49.632 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:44:49 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:sSAKckmzQe4B28ZjkGDnPDqRnW4=', 'Content-Length': '0'} 2023-05-31T20:44:49.634 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:43:08 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e3f08f46fcde7b77-006477b1c1-1c17-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:44:49 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:44:49.635 DEBUG:boto:path=/pklxkd-59/obj51 2023-05-31T20:44:49.635 DEBUG:boto:auth_path=/pklxkd-59/obj51 2023-05-31T20:44:49.635 DEBUG:boto:path=/pklxkd-59/obj51?versionId=null 2023-05-31T20:44:49.636 DEBUG:boto:auth_path=/pklxkd-59/obj51?versionId=null 2023-05-31T20:44:49.636 DEBUG:boto:Method: GET 2023-05-31T20:44:49.637 DEBUG:boto:Path: /pklxkd-59/obj51?versionId=null 2023-05-31T20:44:49.637 DEBUG:boto:Data: 2023-05-31T20:44:49.637 DEBUG:boto:Headers: {} 2023-05-31T20:44:49.638 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:44:49.638 DEBUG:boto:Port: 8000 2023-05-31T20:44:49.638 DEBUG:boto:Params: {} 2023-05-31T20:44:49.639 DEBUG:boto:Token: None 2023-05-31T20:44:49.639 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:44:49 GMT /pklxkd-59/obj51?versionId=null 2023-05-31T20:44:49.639 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:sSAKckmzQe4B28ZjkGDnPDqRnW4= 2023-05-31T20:44:49.640 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:44:49 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:sSAKckmzQe4B28ZjkGDnPDqRnW4=', 'Content-Length': '0'} 2023-05-31T20:44:49.642 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:43:08 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007c53fdf66b3e5486-006477b1c1-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:44:49 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:44:49.642 DEBUG:boto:path=/pklxkd-59/ 2023-05-31T20:44:49.643 DEBUG:boto:auth_path=/pklxkd-59/ 2023-05-31T20:44:49.643 DEBUG:boto:path=/pklxkd-59/?prefix=obj51 2023-05-31T20:44:49.643 DEBUG:boto:auth_path=/pklxkd-59/?prefix=obj51 2023-05-31T20:44:49.644 DEBUG:boto:Method: GET 2023-05-31T20:44:49.644 DEBUG:boto:Path: /pklxkd-59/?prefix=obj51 2023-05-31T20:44:49.645 DEBUG:boto:Data: 2023-05-31T20:44:49.645 DEBUG:boto:Headers: {} 2023-05-31T20:44:49.645 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:44:49.646 DEBUG:boto:Port: 8001 2023-05-31T20:44:49.646 DEBUG:boto:Params: {} 2023-05-31T20:44:49.646 DEBUG:boto:Token: None 2023-05-31T20:44:49.647 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:44:49 GMT /pklxkd-59/ 2023-05-31T20:44:49.647 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:VK2eJyJ+BXpP/6492MS9jzEZ3kw= 2023-05-31T20:44:49.648 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:44:49 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:VK2eJyJ+BXpP/6492MS9jzEZ3kw=', 'Content-Length': '0'} 2023-05-31T20:44:49.650 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000601ca9d8f3348ce8-006477b1c1-1c17-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:44:49 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:44:49.651 DEBUG:boto:b'pklxkd-59obj511000falseobj512023-05-31T20:43:08.011Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:44:49.651 DEBUG:boto:path=/pklxkd-59/ 2023-05-31T20:44:49.652 DEBUG:boto:auth_path=/pklxkd-59/ 2023-05-31T20:44:49.652 DEBUG:boto:path=/pklxkd-59/?prefix=obj51 2023-05-31T20:44:49.653 DEBUG:boto:auth_path=/pklxkd-59/?prefix=obj51 2023-05-31T20:44:49.653 DEBUG:boto:Method: GET 2023-05-31T20:44:49.654 DEBUG:boto:Path: /pklxkd-59/?prefix=obj51 2023-05-31T20:44:49.654 DEBUG:boto:Data: 2023-05-31T20:44:49.655 DEBUG:boto:Headers: {} 2023-05-31T20:44:49.655 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:44:49.655 DEBUG:boto:Port: 8000 2023-05-31T20:44:49.656 DEBUG:boto:Params: {} 2023-05-31T20:44:49.656 DEBUG:boto:Token: None 2023-05-31T20:44:49.656 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:44:49 GMT /pklxkd-59/ 2023-05-31T20:44:49.657 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:VK2eJyJ+BXpP/6492MS9jzEZ3kw= 2023-05-31T20:44:49.657 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:44:49 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:VK2eJyJ+BXpP/6492MS9jzEZ3kw=', 'Content-Length': '0'} 2023-05-31T20:44:49.660 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000038145e8d22467a39-006477b1c1-1630-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:44:49 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:44:49.661 DEBUG:boto:b'pklxkd-59obj511000falseobj512023-05-31T20:43:08.011Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:44:49.661 DEBUG:boto:path=/pklxkd-59/obj51 2023-05-31T20:44:49.662 DEBUG:boto:auth_path=/pklxkd-59/obj51 2023-05-31T20:44:49.662 DEBUG:boto:Method: HEAD 2023-05-31T20:44:49.663 DEBUG:boto:Path: /pklxkd-59/obj51 2023-05-31T20:44:49.663 DEBUG:boto:Data: 2023-05-31T20:44:49.663 DEBUG:boto:Headers: {} 2023-05-31T20:44:49.664 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:44:49.664 DEBUG:boto:Port: 8001 2023-05-31T20:44:49.664 DEBUG:boto:Params: {} 2023-05-31T20:44:49.665 DEBUG:boto:Token: None 2023-05-31T20:44:49.665 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:44:49 GMT /pklxkd-59/obj51 2023-05-31T20:44:49.665 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:JcAG43K0HG8rDJr17WPA4q00DEc= 2023-05-31T20:44:49.666 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:44:49 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:JcAG43K0HG8rDJr17WPA4q00DEc=', 'Content-Length': '0'} 2023-05-31T20:44:49.668 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:43:08 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a1c4084512e86610-006477b1c1-1c17-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:44:49 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:44:49.668 DEBUG:boto:path=/pklxkd-59/obj51 2023-05-31T20:44:49.669 DEBUG:boto:auth_path=/pklxkd-59/obj51 2023-05-31T20:44:49.669 DEBUG:boto:Method: HEAD 2023-05-31T20:44:49.669 DEBUG:boto:Path: /pklxkd-59/obj51 2023-05-31T20:44:49.670 DEBUG:boto:Data: 2023-05-31T20:44:49.670 DEBUG:boto:Headers: {} 2023-05-31T20:44:49.670 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:44:49.671 DEBUG:boto:Port: 8000 2023-05-31T20:44:49.671 DEBUG:boto:Params: {} 2023-05-31T20:44:49.671 DEBUG:boto:Token: None 2023-05-31T20:44:49.672 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:44:49 GMT /pklxkd-59/obj51 2023-05-31T20:44:49.672 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:JcAG43K0HG8rDJr17WPA4q00DEc= 2023-05-31T20:44:49.672 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:44:49 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:JcAG43K0HG8rDJr17WPA4q00DEc=', 'Content-Length': '0'} 2023-05-31T20:44:49.674 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:43:08 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000008e18c4dfbfec34e-006477b1c1-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:44:49 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:44:49.675 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj51 2023-05-31T20:44:49.675 DEBUG:boto:path=/pklxkd-59/obj51 2023-05-31T20:44:49.676 DEBUG:boto:auth_path=/pklxkd-59/obj51 2023-05-31T20:44:49.676 DEBUG:boto:Method: GET 2023-05-31T20:44:49.676 DEBUG:boto:Path: /pklxkd-59/obj51 2023-05-31T20:44:49.677 DEBUG:boto:Data: 2023-05-31T20:44:49.677 DEBUG:boto:Headers: {} 2023-05-31T20:44:49.677 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:44:49.678 DEBUG:boto:Port: 8001 2023-05-31T20:44:49.678 DEBUG:boto:Params: {} 2023-05-31T20:44:49.679 DEBUG:boto:Token: None 2023-05-31T20:44:49.679 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:44:49 GMT /pklxkd-59/obj51 2023-05-31T20:44:49.680 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:1ZZ0m9iXjx2fOGSR9x7y8THMxPc= 2023-05-31T20:44:49.680 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:44:49 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:1ZZ0m9iXjx2fOGSR9x7y8THMxPc=', 'Content-Length': '0'} 2023-05-31T20:44:49.682 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:43:08 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d9437c92d644aad6-006477b1c1-1c17-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:44:49 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:44:49.683 DEBUG:boto:path=/pklxkd-59/obj51 2023-05-31T20:44:49.683 DEBUG:boto:auth_path=/pklxkd-59/obj51 2023-05-31T20:44:49.683 DEBUG:boto:Method: GET 2023-05-31T20:44:49.684 DEBUG:boto:Path: /pklxkd-59/obj51 2023-05-31T20:44:49.684 DEBUG:boto:Data: 2023-05-31T20:44:49.685 DEBUG:boto:Headers: {} 2023-05-31T20:44:49.685 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:44:49.685 DEBUG:boto:Port: 8000 2023-05-31T20:44:49.686 DEBUG:boto:Params: {} 2023-05-31T20:44:49.686 DEBUG:boto:Token: None 2023-05-31T20:44:49.686 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:44:49 GMT /pklxkd-59/obj51 2023-05-31T20:44:49.687 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:1ZZ0m9iXjx2fOGSR9x7y8THMxPc= 2023-05-31T20:44:49.687 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:44:49 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:1ZZ0m9iXjx2fOGSR9x7y8THMxPc=', 'Content-Length': '0'} 2023-05-31T20:44:49.689 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:43:08 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b1cf2c95e9b668ef-006477b1c1-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:44:49 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:44:49.690 DEBUG:boto:path=/pklxkd-59/ 2023-05-31T20:44:49.690 DEBUG:boto:auth_path=/pklxkd-59/ 2023-05-31T20:44:49.691 DEBUG:boto:path=/pklxkd-59/?prefix=obj51 2023-05-31T20:44:49.691 DEBUG:boto:auth_path=/pklxkd-59/?prefix=obj51 2023-05-31T20:44:49.691 DEBUG:boto:Method: GET 2023-05-31T20:44:49.692 DEBUG:boto:Path: /pklxkd-59/?prefix=obj51 2023-05-31T20:44:49.692 DEBUG:boto:Data: 2023-05-31T20:44:49.692 DEBUG:boto:Headers: {} 2023-05-31T20:44:49.693 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:44:49.693 DEBUG:boto:Port: 8001 2023-05-31T20:44:49.693 DEBUG:boto:Params: {} 2023-05-31T20:44:49.694 DEBUG:boto:Token: None 2023-05-31T20:44:49.694 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:44:49 GMT /pklxkd-59/ 2023-05-31T20:44:49.695 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:VK2eJyJ+BXpP/6492MS9jzEZ3kw= 2023-05-31T20:44:49.695 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:44:49 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:VK2eJyJ+BXpP/6492MS9jzEZ3kw=', 'Content-Length': '0'} 2023-05-31T20:44:49.697 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000067451a6bb68057be-006477b1c1-1c17-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:44:49 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:44:49.698 DEBUG:boto:b'pklxkd-59obj511000falseobj512023-05-31T20:43:08.011Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:44:49.698 DEBUG:boto:path=/pklxkd-59/ 2023-05-31T20:44:49.699 DEBUG:boto:auth_path=/pklxkd-59/ 2023-05-31T20:44:49.699 DEBUG:boto:path=/pklxkd-59/?prefix=obj51 2023-05-31T20:44:49.700 DEBUG:boto:auth_path=/pklxkd-59/?prefix=obj51 2023-05-31T20:44:49.700 DEBUG:boto:Method: GET 2023-05-31T20:44:49.700 DEBUG:boto:Path: /pklxkd-59/?prefix=obj51 2023-05-31T20:44:49.701 DEBUG:boto:Data: 2023-05-31T20:44:49.701 DEBUG:boto:Headers: {} 2023-05-31T20:44:49.702 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:44:49.702 DEBUG:boto:Port: 8000 2023-05-31T20:44:49.702 DEBUG:boto:Params: {} 2023-05-31T20:44:49.703 DEBUG:boto:Token: None 2023-05-31T20:44:49.703 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:44:49 GMT /pklxkd-59/ 2023-05-31T20:44:49.703 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:VK2eJyJ+BXpP/6492MS9jzEZ3kw= 2023-05-31T20:44:49.704 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:44:49 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:VK2eJyJ+BXpP/6492MS9jzEZ3kw=', 'Content-Length': '0'} 2023-05-31T20:44:49.707 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000000c2fb3c46ec7d273-006477b1c1-1630-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:44:49 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:44:49.748 DEBUG:boto:b'pklxkd-59obj511000falseobj512023-05-31T20:43:08.011Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:44:49.749 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj55 2023-05-31T20:44:49.749 DEBUG:boto:path=/pklxkd-59/obj55 2023-05-31T20:44:49.750 DEBUG:boto:auth_path=/pklxkd-59/obj55 2023-05-31T20:44:49.750 DEBUG:boto:path=/pklxkd-59/obj55?versionId=null 2023-05-31T20:44:49.750 DEBUG:boto:auth_path=/pklxkd-59/obj55?versionId=null 2023-05-31T20:44:49.751 DEBUG:boto:Method: GET 2023-05-31T20:44:49.751 DEBUG:boto:Path: /pklxkd-59/obj55?versionId=null 2023-05-31T20:44:49.752 DEBUG:boto:Data: 2023-05-31T20:44:49.752 DEBUG:boto:Headers: {} 2023-05-31T20:44:49.752 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:44:49.753 DEBUG:boto:Port: 8001 2023-05-31T20:44:49.753 DEBUG:boto:Params: {} 2023-05-31T20:44:49.754 DEBUG:boto:Token: None 2023-05-31T20:44:49.754 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:44:49 GMT /pklxkd-59/obj55?versionId=null 2023-05-31T20:44:49.755 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:eCwC4aEZipyxIK21/jY8v8lTlVo= 2023-05-31T20:44:49.755 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:44:49 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:eCwC4aEZipyxIK21/jY8v8lTlVo=', 'Content-Length': '0'} 2023-05-31T20:44:49.757 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:39:26 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000fb2609270fb16e05-006477b1c1-1c17-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:44:49 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:44:49.758 DEBUG:boto:path=/pklxkd-59/obj55 2023-05-31T20:44:49.758 DEBUG:boto:auth_path=/pklxkd-59/obj55 2023-05-31T20:44:49.759 DEBUG:boto:path=/pklxkd-59/obj55?versionId=null 2023-05-31T20:44:49.759 DEBUG:boto:auth_path=/pklxkd-59/obj55?versionId=null 2023-05-31T20:44:49.759 DEBUG:boto:Method: GET 2023-05-31T20:44:49.760 DEBUG:boto:Path: /pklxkd-59/obj55?versionId=null 2023-05-31T20:44:49.760 DEBUG:boto:Data: 2023-05-31T20:44:49.760 DEBUG:boto:Headers: {} 2023-05-31T20:44:49.761 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:44:49.761 DEBUG:boto:Port: 8000 2023-05-31T20:44:49.762 DEBUG:boto:Params: {} 2023-05-31T20:44:49.762 DEBUG:boto:Token: None 2023-05-31T20:44:49.762 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:44:49 GMT /pklxkd-59/obj55?versionId=null 2023-05-31T20:44:49.763 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:eCwC4aEZipyxIK21/jY8v8lTlVo= 2023-05-31T20:44:49.763 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:44:49 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:eCwC4aEZipyxIK21/jY8v8lTlVo=', 'Content-Length': '0'} 2023-05-31T20:44:49.765 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:39:26 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002485b11af846ab90-006477b1c1-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:44:49 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:44:49.766 DEBUG:boto:path=/pklxkd-59/obj55 2023-05-31T20:44:49.766 DEBUG:boto:auth_path=/pklxkd-59/obj55 2023-05-31T20:44:49.766 DEBUG:boto:path=/pklxkd-59/obj55?versionId=null 2023-05-31T20:44:49.767 DEBUG:boto:auth_path=/pklxkd-59/obj55?versionId=null 2023-05-31T20:44:49.767 DEBUG:boto:Method: HEAD 2023-05-31T20:44:49.768 DEBUG:boto:Path: /pklxkd-59/obj55?versionId=null 2023-05-31T20:44:49.768 DEBUG:boto:Data: 2023-05-31T20:44:49.768 DEBUG:boto:Headers: {} 2023-05-31T20:44:49.769 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:44:49.769 DEBUG:boto:Port: 8001 2023-05-31T20:44:49.769 DEBUG:boto:Params: {} 2023-05-31T20:44:49.770 DEBUG:boto:Token: None 2023-05-31T20:44:49.770 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:44:49 GMT /pklxkd-59/obj55?versionId=null 2023-05-31T20:44:49.770 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:VQlkH/c09PPs28u2VpKDL6SioUY= 2023-05-31T20:44:49.771 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:44:49 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:VQlkH/c09PPs28u2VpKDL6SioUY=', 'Content-Length': '0'} 2023-05-31T20:44:49.773 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:39:26 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b742c89025f2b470-006477b1c1-1c17-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:44:49 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:44:49.774 DEBUG:boto:path=/pklxkd-59/obj55 2023-05-31T20:44:49.774 DEBUG:boto:auth_path=/pklxkd-59/obj55 2023-05-31T20:44:49.774 DEBUG:boto:path=/pklxkd-59/obj55?versionId=null 2023-05-31T20:44:49.775 DEBUG:boto:auth_path=/pklxkd-59/obj55?versionId=null 2023-05-31T20:44:49.775 DEBUG:boto:Method: HEAD 2023-05-31T20:44:49.775 DEBUG:boto:Path: /pklxkd-59/obj55?versionId=null 2023-05-31T20:44:49.776 DEBUG:boto:Data: 2023-05-31T20:44:49.776 DEBUG:boto:Headers: {} 2023-05-31T20:44:49.776 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:44:49.777 DEBUG:boto:Port: 8000 2023-05-31T20:44:49.777 DEBUG:boto:Params: {} 2023-05-31T20:44:49.777 DEBUG:boto:Token: None 2023-05-31T20:44:49.778 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:44:49 GMT /pklxkd-59/obj55?versionId=null 2023-05-31T20:44:49.778 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:VQlkH/c09PPs28u2VpKDL6SioUY= 2023-05-31T20:44:49.778 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:44:49 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:VQlkH/c09PPs28u2VpKDL6SioUY=', 'Content-Length': '0'} 2023-05-31T20:44:49.780 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:39:26 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000cd4fd22eb1a3f006-006477b1c1-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:44:49 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:44:49.781 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj55 2023-05-31T20:44:49.781 DEBUG:boto:path=/pklxkd-59/obj55 2023-05-31T20:44:49.782 DEBUG:boto:auth_path=/pklxkd-59/obj55 2023-05-31T20:44:49.782 DEBUG:boto:path=/pklxkd-59/obj55?versionId=null 2023-05-31T20:44:49.782 DEBUG:boto:auth_path=/pklxkd-59/obj55?versionId=null 2023-05-31T20:44:49.783 DEBUG:boto:Method: GET 2023-05-31T20:44:49.783 DEBUG:boto:Path: /pklxkd-59/obj55?versionId=null 2023-05-31T20:44:49.783 DEBUG:boto:Data: 2023-05-31T20:44:49.784 DEBUG:boto:Headers: {} 2023-05-31T20:44:49.784 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:44:49.784 DEBUG:boto:Port: 8001 2023-05-31T20:44:49.785 DEBUG:boto:Params: {} 2023-05-31T20:44:49.785 DEBUG:boto:Token: None 2023-05-31T20:44:49.786 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:44:49 GMT /pklxkd-59/obj55?versionId=null 2023-05-31T20:44:49.786 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:eCwC4aEZipyxIK21/jY8v8lTlVo= 2023-05-31T20:44:49.786 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:44:49 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:eCwC4aEZipyxIK21/jY8v8lTlVo=', 'Content-Length': '0'} 2023-05-31T20:44:49.788 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:39:26 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ae682ed4ba170702-006477b1c1-1c17-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:44:49 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:44:49.832 DEBUG:boto:path=/pklxkd-59/obj55 2023-05-31T20:44:49.833 DEBUG:boto:auth_path=/pklxkd-59/obj55 2023-05-31T20:44:49.833 DEBUG:boto:path=/pklxkd-59/obj55?versionId=null 2023-05-31T20:44:49.834 DEBUG:boto:auth_path=/pklxkd-59/obj55?versionId=null 2023-05-31T20:44:49.834 DEBUG:boto:Method: GET 2023-05-31T20:44:49.835 DEBUG:boto:Path: /pklxkd-59/obj55?versionId=null 2023-05-31T20:44:49.835 DEBUG:boto:Data: 2023-05-31T20:44:49.835 DEBUG:boto:Headers: {} 2023-05-31T20:44:49.836 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:44:49.836 DEBUG:boto:Port: 8000 2023-05-31T20:44:49.836 DEBUG:boto:Params: {} 2023-05-31T20:44:49.837 DEBUG:boto:Token: None 2023-05-31T20:44:49.838 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:44:49 GMT /pklxkd-59/obj55?versionId=null 2023-05-31T20:44:49.838 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:eCwC4aEZipyxIK21/jY8v8lTlVo= 2023-05-31T20:44:49.838 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:44:49 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:eCwC4aEZipyxIK21/jY8v8lTlVo=', 'Content-Length': '0'} 2023-05-31T20:44:49.841 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:39:26 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000dced441c4673b8d7-006477b1c1-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:44:49 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:44:49.842 DEBUG:boto:path=/pklxkd-59/ 2023-05-31T20:44:49.842 DEBUG:boto:auth_path=/pklxkd-59/ 2023-05-31T20:44:49.843 DEBUG:boto:path=/pklxkd-59/?prefix=obj55 2023-05-31T20:44:49.843 DEBUG:boto:auth_path=/pklxkd-59/?prefix=obj55 2023-05-31T20:44:49.844 DEBUG:boto:Method: GET 2023-05-31T20:44:49.844 DEBUG:boto:Path: /pklxkd-59/?prefix=obj55 2023-05-31T20:44:49.844 DEBUG:boto:Data: 2023-05-31T20:44:49.845 DEBUG:boto:Headers: {} 2023-05-31T20:44:49.845 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:44:49.845 DEBUG:boto:Port: 8001 2023-05-31T20:44:49.846 DEBUG:boto:Params: {} 2023-05-31T20:44:49.846 DEBUG:boto:Token: None 2023-05-31T20:44:49.847 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:44:49 GMT /pklxkd-59/ 2023-05-31T20:44:49.847 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:VK2eJyJ+BXpP/6492MS9jzEZ3kw= 2023-05-31T20:44:49.848 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:44:49 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:VK2eJyJ+BXpP/6492MS9jzEZ3kw=', 'Content-Length': '0'} 2023-05-31T20:44:49.850 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000bc731031da8f12f1-006477b1c1-1c17-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:44:49 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:44:49.850 DEBUG:boto:b'pklxkd-59obj551000falseobj552023-05-31T20:39:26.630Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:44:49.851 DEBUG:boto:path=/pklxkd-59/ 2023-05-31T20:44:49.851 DEBUG:boto:auth_path=/pklxkd-59/ 2023-05-31T20:44:49.852 DEBUG:boto:path=/pklxkd-59/?prefix=obj55 2023-05-31T20:44:49.852 DEBUG:boto:auth_path=/pklxkd-59/?prefix=obj55 2023-05-31T20:44:49.853 DEBUG:boto:Method: GET 2023-05-31T20:44:49.853 DEBUG:boto:Path: /pklxkd-59/?prefix=obj55 2023-05-31T20:44:49.853 DEBUG:boto:Data: 2023-05-31T20:44:49.854 DEBUG:boto:Headers: {} 2023-05-31T20:44:49.854 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:44:49.855 DEBUG:boto:Port: 8000 2023-05-31T20:44:49.855 DEBUG:boto:Params: {} 2023-05-31T20:44:49.856 DEBUG:boto:Token: None 2023-05-31T20:44:49.856 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:44:49 GMT /pklxkd-59/ 2023-05-31T20:44:49.857 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:VK2eJyJ+BXpP/6492MS9jzEZ3kw= 2023-05-31T20:44:49.857 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:44:49 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:VK2eJyJ+BXpP/6492MS9jzEZ3kw=', 'Content-Length': '0'} 2023-05-31T20:44:49.860 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000000fced34067e36402-006477b1c1-1630-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:44:49 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:44:49.900 DEBUG:boto:b'pklxkd-59obj551000falseobj552023-05-31T20:39:26.630Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:44:49.901 DEBUG:boto:path=/pklxkd-59/obj55 2023-05-31T20:44:49.901 DEBUG:boto:auth_path=/pklxkd-59/obj55 2023-05-31T20:44:49.902 DEBUG:boto:Method: HEAD 2023-05-31T20:44:49.902 DEBUG:boto:Path: /pklxkd-59/obj55 2023-05-31T20:44:49.902 DEBUG:boto:Data: 2023-05-31T20:44:49.903 DEBUG:boto:Headers: {} 2023-05-31T20:44:49.903 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:44:49.903 DEBUG:boto:Port: 8001 2023-05-31T20:44:49.904 DEBUG:boto:Params: {} 2023-05-31T20:44:49.904 DEBUG:boto:Token: None 2023-05-31T20:44:49.904 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:44:49 GMT /pklxkd-59/obj55 2023-05-31T20:44:49.905 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:g2v7czRgHzvc0ovvLbMx/xM660o= 2023-05-31T20:44:49.905 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:44:49 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:g2v7czRgHzvc0ovvLbMx/xM660o=', 'Content-Length': '0'} 2023-05-31T20:44:49.907 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:39:26 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000f76f1ac560d72e0-006477b1c1-1c17-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:44:49 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:44:49.908 DEBUG:boto:path=/pklxkd-59/obj55 2023-05-31T20:44:49.908 DEBUG:boto:auth_path=/pklxkd-59/obj55 2023-05-31T20:44:49.908 DEBUG:boto:Method: HEAD 2023-05-31T20:44:49.909 DEBUG:boto:Path: /pklxkd-59/obj55 2023-05-31T20:44:49.909 DEBUG:boto:Data: 2023-05-31T20:44:49.909 DEBUG:boto:Headers: {} 2023-05-31T20:44:49.910 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:44:49.910 DEBUG:boto:Port: 8000 2023-05-31T20:44:49.910 DEBUG:boto:Params: {} 2023-05-31T20:44:49.911 DEBUG:boto:Token: None 2023-05-31T20:44:49.911 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:44:49 GMT /pklxkd-59/obj55 2023-05-31T20:44:49.911 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:g2v7czRgHzvc0ovvLbMx/xM660o= 2023-05-31T20:44:49.912 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:44:49 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:g2v7czRgHzvc0ovvLbMx/xM660o=', 'Content-Length': '0'} 2023-05-31T20:44:49.914 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:39:26 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000a3307690f8b2b76-006477b1c1-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:44:49 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:44:49.914 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj55 2023-05-31T20:44:49.915 DEBUG:boto:path=/pklxkd-59/obj55 2023-05-31T20:44:49.915 DEBUG:boto:auth_path=/pklxkd-59/obj55 2023-05-31T20:44:49.915 DEBUG:boto:Method: GET 2023-05-31T20:44:49.916 DEBUG:boto:Path: /pklxkd-59/obj55 2023-05-31T20:44:49.916 DEBUG:boto:Data: 2023-05-31T20:44:49.917 DEBUG:boto:Headers: {} 2023-05-31T20:44:49.917 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:44:49.917 DEBUG:boto:Port: 8001 2023-05-31T20:44:49.918 DEBUG:boto:Params: {} 2023-05-31T20:44:49.918 DEBUG:boto:Token: None 2023-05-31T20:44:49.919 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:44:49 GMT /pklxkd-59/obj55 2023-05-31T20:44:49.919 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:2+ieb/KUYvaRoe/2cv+PnQy0Di8= 2023-05-31T20:44:49.920 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:44:49 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:2+ieb/KUYvaRoe/2cv+PnQy0Di8=', 'Content-Length': '0'} 2023-05-31T20:44:49.921 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:39:26 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f3c57a10848f8673-006477b1c1-1c17-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:44:49 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:44:49.922 DEBUG:boto:path=/pklxkd-59/obj55 2023-05-31T20:44:49.922 DEBUG:boto:auth_path=/pklxkd-59/obj55 2023-05-31T20:44:49.923 DEBUG:boto:Method: GET 2023-05-31T20:44:49.923 DEBUG:boto:Path: /pklxkd-59/obj55 2023-05-31T20:44:49.923 DEBUG:boto:Data: 2023-05-31T20:44:49.924 DEBUG:boto:Headers: {} 2023-05-31T20:44:49.924 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:44:49.924 DEBUG:boto:Port: 8000 2023-05-31T20:44:49.925 DEBUG:boto:Params: {} 2023-05-31T20:44:49.925 DEBUG:boto:Token: None 2023-05-31T20:44:49.926 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:44:49 GMT /pklxkd-59/obj55 2023-05-31T20:44:49.926 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:2+ieb/KUYvaRoe/2cv+PnQy0Di8= 2023-05-31T20:44:49.927 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:44:49 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:2+ieb/KUYvaRoe/2cv+PnQy0Di8=', 'Content-Length': '0'} 2023-05-31T20:44:49.928 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:39:26 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004b6ca818676e748b-006477b1c1-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:44:49 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:44:49.929 DEBUG:boto:path=/pklxkd-59/ 2023-05-31T20:44:49.930 DEBUG:boto:auth_path=/pklxkd-59/ 2023-05-31T20:44:49.930 DEBUG:boto:path=/pklxkd-59/?prefix=obj55 2023-05-31T20:44:49.930 DEBUG:boto:auth_path=/pklxkd-59/?prefix=obj55 2023-05-31T20:44:49.931 DEBUG:boto:Method: GET 2023-05-31T20:44:49.931 DEBUG:boto:Path: /pklxkd-59/?prefix=obj55 2023-05-31T20:44:49.931 DEBUG:boto:Data: 2023-05-31T20:44:49.932 DEBUG:boto:Headers: {} 2023-05-31T20:44:49.932 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:44:49.933 DEBUG:boto:Port: 8001 2023-05-31T20:44:49.933 DEBUG:boto:Params: {} 2023-05-31T20:44:49.934 DEBUG:boto:Token: None 2023-05-31T20:44:49.934 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:44:49 GMT /pklxkd-59/ 2023-05-31T20:44:49.934 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:VK2eJyJ+BXpP/6492MS9jzEZ3kw= 2023-05-31T20:44:49.935 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:44:49 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:VK2eJyJ+BXpP/6492MS9jzEZ3kw=', 'Content-Length': '0'} 2023-05-31T20:44:49.937 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000af716e3d4235d5c7-006477b1c1-1c17-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:44:49 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:44:49.937 DEBUG:boto:b'pklxkd-59obj551000falseobj552023-05-31T20:39:26.630Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:44:49.938 DEBUG:boto:path=/pklxkd-59/ 2023-05-31T20:44:49.938 DEBUG:boto:auth_path=/pklxkd-59/ 2023-05-31T20:44:49.939 DEBUG:boto:path=/pklxkd-59/?prefix=obj55 2023-05-31T20:44:49.939 DEBUG:boto:auth_path=/pklxkd-59/?prefix=obj55 2023-05-31T20:44:49.940 DEBUG:boto:Method: GET 2023-05-31T20:44:49.940 DEBUG:boto:Path: /pklxkd-59/?prefix=obj55 2023-05-31T20:44:49.940 DEBUG:boto:Data: 2023-05-31T20:44:49.941 DEBUG:boto:Headers: {} 2023-05-31T20:44:49.941 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:44:49.941 DEBUG:boto:Port: 8000 2023-05-31T20:44:49.942 DEBUG:boto:Params: {} 2023-05-31T20:44:49.942 DEBUG:boto:Token: None 2023-05-31T20:44:49.943 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:44:49 GMT /pklxkd-59/ 2023-05-31T20:44:49.943 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:VK2eJyJ+BXpP/6492MS9jzEZ3kw= 2023-05-31T20:44:49.943 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:44:49 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:VK2eJyJ+BXpP/6492MS9jzEZ3kw=', 'Content-Length': '0'} 2023-05-31T20:44:49.946 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000005aa99fda01d6269b-006477b1c1-1630-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:44:49 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:44:49.946 DEBUG:boto:b'pklxkd-59obj551000falseobj552023-05-31T20:39:26.630Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:44:49.947 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj68 2023-05-31T20:44:49.947 DEBUG:boto:path=/pklxkd-59/obj68 2023-05-31T20:44:49.948 DEBUG:boto:auth_path=/pklxkd-59/obj68 2023-05-31T20:44:49.948 DEBUG:boto:path=/pklxkd-59/obj68?versionId=null 2023-05-31T20:44:49.949 DEBUG:boto:auth_path=/pklxkd-59/obj68?versionId=null 2023-05-31T20:44:49.949 DEBUG:boto:Method: GET 2023-05-31T20:44:49.949 DEBUG:boto:Path: /pklxkd-59/obj68?versionId=null 2023-05-31T20:44:49.950 DEBUG:boto:Data: 2023-05-31T20:44:49.950 DEBUG:boto:Headers: {} 2023-05-31T20:44:49.950 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:44:49.951 DEBUG:boto:Port: 8001 2023-05-31T20:44:49.951 DEBUG:boto:Params: {} 2023-05-31T20:44:49.952 DEBUG:boto:Token: None 2023-05-31T20:44:49.952 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:44:49 GMT /pklxkd-59/obj68?versionId=null 2023-05-31T20:44:49.953 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:oO3I3jbmWIoQWRJe10qr3SsgAmk= 2023-05-31T20:44:49.954 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:44:49 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:oO3I3jbmWIoQWRJe10qr3SsgAmk=', 'Content-Length': '0'} 2023-05-31T20:44:49.956 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:43:08 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000034f5109d36581fa5-006477b1c1-1c17-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:44:49 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:44:49.957 DEBUG:boto:path=/pklxkd-59/obj68 2023-05-31T20:44:49.958 DEBUG:boto:auth_path=/pklxkd-59/obj68 2023-05-31T20:44:49.958 DEBUG:boto:path=/pklxkd-59/obj68?versionId=null 2023-05-31T20:44:49.958 DEBUG:boto:auth_path=/pklxkd-59/obj68?versionId=null 2023-05-31T20:44:49.959 DEBUG:boto:Method: GET 2023-05-31T20:44:49.959 DEBUG:boto:Path: /pklxkd-59/obj68?versionId=null 2023-05-31T20:44:49.959 DEBUG:boto:Data: 2023-05-31T20:44:49.960 DEBUG:boto:Headers: {} 2023-05-31T20:44:49.960 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:44:49.960 DEBUG:boto:Port: 8000 2023-05-31T20:44:49.961 DEBUG:boto:Params: {} 2023-05-31T20:44:49.961 DEBUG:boto:Token: None 2023-05-31T20:44:49.962 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:44:49 GMT /pklxkd-59/obj68?versionId=null 2023-05-31T20:44:49.962 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:oO3I3jbmWIoQWRJe10qr3SsgAmk= 2023-05-31T20:44:49.962 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:44:49 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:oO3I3jbmWIoQWRJe10qr3SsgAmk=', 'Content-Length': '0'} 2023-05-31T20:44:49.964 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:43:08 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ef4c47a034b75bd7-006477b1c1-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:44:49 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:44:49.965 DEBUG:boto:path=/pklxkd-59/obj68 2023-05-31T20:44:49.965 DEBUG:boto:auth_path=/pklxkd-59/obj68 2023-05-31T20:44:49.966 DEBUG:boto:path=/pklxkd-59/obj68?versionId=null 2023-05-31T20:44:49.966 DEBUG:boto:auth_path=/pklxkd-59/obj68?versionId=null 2023-05-31T20:44:49.966 DEBUG:boto:Method: HEAD 2023-05-31T20:44:49.967 DEBUG:boto:Path: /pklxkd-59/obj68?versionId=null 2023-05-31T20:44:49.967 DEBUG:boto:Data: 2023-05-31T20:44:49.967 DEBUG:boto:Headers: {} 2023-05-31T20:44:49.968 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:44:49.968 DEBUG:boto:Port: 8001 2023-05-31T20:44:49.969 DEBUG:boto:Params: {} 2023-05-31T20:44:49.969 DEBUG:boto:Token: None 2023-05-31T20:44:49.969 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:44:49 GMT /pklxkd-59/obj68?versionId=null 2023-05-31T20:44:49.970 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:M0RXLWiWteEwqOhI27er4erJ/Yk= 2023-05-31T20:44:49.970 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:44:49 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:M0RXLWiWteEwqOhI27er4erJ/Yk=', 'Content-Length': '0'} 2023-05-31T20:44:49.972 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:43:08 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009c06d0f216e6e92f-006477b1c1-1c17-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:44:49 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:44:49.973 DEBUG:boto:path=/pklxkd-59/obj68 2023-05-31T20:44:49.973 DEBUG:boto:auth_path=/pklxkd-59/obj68 2023-05-31T20:44:49.973 DEBUG:boto:path=/pklxkd-59/obj68?versionId=null 2023-05-31T20:44:49.974 DEBUG:boto:auth_path=/pklxkd-59/obj68?versionId=null 2023-05-31T20:44:49.974 DEBUG:boto:Method: HEAD 2023-05-31T20:44:49.974 DEBUG:boto:Path: /pklxkd-59/obj68?versionId=null 2023-05-31T20:44:49.975 DEBUG:boto:Data: 2023-05-31T20:44:49.975 DEBUG:boto:Headers: {} 2023-05-31T20:44:49.975 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:44:49.976 DEBUG:boto:Port: 8000 2023-05-31T20:44:49.976 DEBUG:boto:Params: {} 2023-05-31T20:44:49.976 DEBUG:boto:Token: None 2023-05-31T20:44:49.977 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:44:49 GMT /pklxkd-59/obj68?versionId=null 2023-05-31T20:44:49.977 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:M0RXLWiWteEwqOhI27er4erJ/Yk= 2023-05-31T20:44:49.977 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:44:49 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:M0RXLWiWteEwqOhI27er4erJ/Yk=', 'Content-Length': '0'} 2023-05-31T20:44:49.979 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:43:08 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d355e34d0a278485-006477b1c1-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:44:49 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:44:49.980 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj68 2023-05-31T20:44:49.980 DEBUG:boto:path=/pklxkd-59/obj68 2023-05-31T20:44:49.980 DEBUG:boto:auth_path=/pklxkd-59/obj68 2023-05-31T20:44:49.981 DEBUG:boto:path=/pklxkd-59/obj68?versionId=null 2023-05-31T20:44:49.981 DEBUG:boto:auth_path=/pklxkd-59/obj68?versionId=null 2023-05-31T20:44:49.981 DEBUG:boto:Method: GET 2023-05-31T20:44:49.982 DEBUG:boto:Path: /pklxkd-59/obj68?versionId=null 2023-05-31T20:44:49.982 DEBUG:boto:Data: 2023-05-31T20:44:49.982 DEBUG:boto:Headers: {} 2023-05-31T20:44:49.983 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:44:49.983 DEBUG:boto:Port: 8001 2023-05-31T20:44:49.984 DEBUG:boto:Params: {} 2023-05-31T20:44:49.984 DEBUG:boto:Token: None 2023-05-31T20:44:49.984 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:44:49 GMT /pklxkd-59/obj68?versionId=null 2023-05-31T20:44:49.985 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:oO3I3jbmWIoQWRJe10qr3SsgAmk= 2023-05-31T20:44:49.985 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:44:49 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:oO3I3jbmWIoQWRJe10qr3SsgAmk=', 'Content-Length': '0'} 2023-05-31T20:44:49.987 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:43:08 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000dcabdc6ebb7df7ed-006477b1c1-1c17-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:44:49 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:44:50.028 DEBUG:boto:path=/pklxkd-59/obj68 2023-05-31T20:44:50.029 DEBUG:boto:auth_path=/pklxkd-59/obj68 2023-05-31T20:44:50.029 DEBUG:boto:path=/pklxkd-59/obj68?versionId=null 2023-05-31T20:44:50.029 DEBUG:boto:auth_path=/pklxkd-59/obj68?versionId=null 2023-05-31T20:44:50.030 DEBUG:boto:Method: GET 2023-05-31T20:44:50.030 DEBUG:boto:Path: /pklxkd-59/obj68?versionId=null 2023-05-31T20:44:50.030 DEBUG:boto:Data: 2023-05-31T20:44:50.031 DEBUG:boto:Headers: {} 2023-05-31T20:44:50.031 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:44:50.032 DEBUG:boto:Port: 8000 2023-05-31T20:44:50.032 DEBUG:boto:Params: {} 2023-05-31T20:44:50.033 DEBUG:boto:Token: None 2023-05-31T20:44:50.033 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:44:50 GMT /pklxkd-59/obj68?versionId=null 2023-05-31T20:44:50.033 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:cnsQV2QZqH/LZW5FcjesiB2I6vk= 2023-05-31T20:44:50.034 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:44:50 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:cnsQV2QZqH/LZW5FcjesiB2I6vk=', 'Content-Length': '0'} 2023-05-31T20:44:50.036 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:43:08 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000bf44b61a730f048a-006477b1c2-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:44:50 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:44:50.037 DEBUG:boto:path=/pklxkd-59/ 2023-05-31T20:44:50.037 DEBUG:boto:auth_path=/pklxkd-59/ 2023-05-31T20:44:50.037 DEBUG:boto:path=/pklxkd-59/?prefix=obj68 2023-05-31T20:44:50.038 DEBUG:boto:auth_path=/pklxkd-59/?prefix=obj68 2023-05-31T20:44:50.038 DEBUG:boto:Method: GET 2023-05-31T20:44:50.038 DEBUG:boto:Path: /pklxkd-59/?prefix=obj68 2023-05-31T20:44:50.039 DEBUG:boto:Data: 2023-05-31T20:44:50.039 DEBUG:boto:Headers: {} 2023-05-31T20:44:50.040 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:44:50.040 DEBUG:boto:Port: 8001 2023-05-31T20:44:50.040 DEBUG:boto:Params: {} 2023-05-31T20:44:50.041 DEBUG:boto:Token: None 2023-05-31T20:44:50.041 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:44:50 GMT /pklxkd-59/ 2023-05-31T20:44:50.041 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:yNGiUWAW4Yg2l1bOAUM5KexBO6Q= 2023-05-31T20:44:50.042 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:44:50 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:yNGiUWAW4Yg2l1bOAUM5KexBO6Q=', 'Content-Length': '0'} 2023-05-31T20:44:50.044 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000008fee419a86736556-006477b1c2-1c17-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:44:50 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:44:50.045 DEBUG:boto:b'pklxkd-59obj681000falseobj682023-05-31T20:43:08.066Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:44:50.045 DEBUG:boto:path=/pklxkd-59/ 2023-05-31T20:44:50.046 DEBUG:boto:auth_path=/pklxkd-59/ 2023-05-31T20:44:50.046 DEBUG:boto:path=/pklxkd-59/?prefix=obj68 2023-05-31T20:44:50.046 DEBUG:boto:auth_path=/pklxkd-59/?prefix=obj68 2023-05-31T20:44:50.047 DEBUG:boto:Method: GET 2023-05-31T20:44:50.047 DEBUG:boto:Path: /pklxkd-59/?prefix=obj68 2023-05-31T20:44:50.047 DEBUG:boto:Data: 2023-05-31T20:44:50.048 DEBUG:boto:Headers: {} 2023-05-31T20:44:50.048 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:44:50.048 DEBUG:boto:Port: 8000 2023-05-31T20:44:50.049 DEBUG:boto:Params: {} 2023-05-31T20:44:50.049 DEBUG:boto:Token: None 2023-05-31T20:44:50.050 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:44:50 GMT /pklxkd-59/ 2023-05-31T20:44:50.050 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:yNGiUWAW4Yg2l1bOAUM5KexBO6Q= 2023-05-31T20:44:50.050 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:44:50 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:yNGiUWAW4Yg2l1bOAUM5KexBO6Q=', 'Content-Length': '0'} 2023-05-31T20:44:50.053 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000003d4d2ede374c29fe-006477b1c2-1630-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:44:50 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:44:50.053 DEBUG:boto:b'pklxkd-59obj681000falseobj682023-05-31T20:43:08.066Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:44:50.054 DEBUG:boto:path=/pklxkd-59/obj68 2023-05-31T20:44:50.054 DEBUG:boto:auth_path=/pklxkd-59/obj68 2023-05-31T20:44:50.055 DEBUG:boto:Method: HEAD 2023-05-31T20:44:50.055 DEBUG:boto:Path: /pklxkd-59/obj68 2023-05-31T20:44:50.055 DEBUG:boto:Data: 2023-05-31T20:44:50.056 DEBUG:boto:Headers: {} 2023-05-31T20:44:50.056 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:44:50.057 DEBUG:boto:Port: 8001 2023-05-31T20:44:50.057 DEBUG:boto:Params: {} 2023-05-31T20:44:50.058 DEBUG:boto:Token: None 2023-05-31T20:44:50.058 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:44:50 GMT /pklxkd-59/obj68 2023-05-31T20:44:50.058 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:u5NI4KLXU291ZC1D2vb1kzvRjU8= 2023-05-31T20:44:50.059 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:44:50 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:u5NI4KLXU291ZC1D2vb1kzvRjU8=', 'Content-Length': '0'} 2023-05-31T20:44:50.060 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:43:08 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d2f71f41f07137f6-006477b1c2-1c17-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:44:50 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:44:50.061 DEBUG:boto:path=/pklxkd-59/obj68 2023-05-31T20:44:50.061 DEBUG:boto:auth_path=/pklxkd-59/obj68 2023-05-31T20:44:50.062 DEBUG:boto:Method: HEAD 2023-05-31T20:44:50.062 DEBUG:boto:Path: /pklxkd-59/obj68 2023-05-31T20:44:50.062 DEBUG:boto:Data: 2023-05-31T20:44:50.063 DEBUG:boto:Headers: {} 2023-05-31T20:44:50.063 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:44:50.063 DEBUG:boto:Port: 8000 2023-05-31T20:44:50.064 DEBUG:boto:Params: {} 2023-05-31T20:44:50.064 DEBUG:boto:Token: None 2023-05-31T20:44:50.065 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:44:50 GMT /pklxkd-59/obj68 2023-05-31T20:44:50.065 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:u5NI4KLXU291ZC1D2vb1kzvRjU8= 2023-05-31T20:44:50.065 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:44:50 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:u5NI4KLXU291ZC1D2vb1kzvRjU8=', 'Content-Length': '0'} 2023-05-31T20:44:50.067 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:43:08 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000077cd559eb42ce017-006477b1c2-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:44:50 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:44:50.068 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj68 2023-05-31T20:44:50.068 DEBUG:boto:path=/pklxkd-59/obj68 2023-05-31T20:44:50.069 DEBUG:boto:auth_path=/pklxkd-59/obj68 2023-05-31T20:44:50.069 DEBUG:boto:Method: GET 2023-05-31T20:44:50.069 DEBUG:boto:Path: /pklxkd-59/obj68 2023-05-31T20:44:50.070 DEBUG:boto:Data: 2023-05-31T20:44:50.070 DEBUG:boto:Headers: {} 2023-05-31T20:44:50.071 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:44:50.071 DEBUG:boto:Port: 8001 2023-05-31T20:44:50.071 DEBUG:boto:Params: {} 2023-05-31T20:44:50.072 DEBUG:boto:Token: None 2023-05-31T20:44:50.072 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:44:50 GMT /pklxkd-59/obj68 2023-05-31T20:44:50.072 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:t5acb/uKV8loqMiIRcsQGXCtEFU= 2023-05-31T20:44:50.073 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:44:50 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:t5acb/uKV8loqMiIRcsQGXCtEFU=', 'Content-Length': '0'} 2023-05-31T20:44:50.075 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:43:08 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000012dd3304a50c61cc-006477b1c2-1c17-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:44:50 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:44:50.120 DEBUG:boto:path=/pklxkd-59/obj68 2023-05-31T20:44:50.121 DEBUG:boto:auth_path=/pklxkd-59/obj68 2023-05-31T20:44:50.121 DEBUG:boto:Method: GET 2023-05-31T20:44:50.121 DEBUG:boto:Path: /pklxkd-59/obj68 2023-05-31T20:44:50.122 DEBUG:boto:Data: 2023-05-31T20:44:50.122 DEBUG:boto:Headers: {} 2023-05-31T20:44:50.123 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:44:50.123 DEBUG:boto:Port: 8000 2023-05-31T20:44:50.123 DEBUG:boto:Params: {} 2023-05-31T20:44:50.124 DEBUG:boto:Token: None 2023-05-31T20:44:50.124 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:44:50 GMT /pklxkd-59/obj68 2023-05-31T20:44:50.125 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:t5acb/uKV8loqMiIRcsQGXCtEFU= 2023-05-31T20:44:50.125 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:44:50 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:t5acb/uKV8loqMiIRcsQGXCtEFU=', 'Content-Length': '0'} 2023-05-31T20:44:50.127 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:43:08 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005afb51a4e76e6c9d-006477b1c2-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:44:50 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:44:50.128 DEBUG:boto:path=/pklxkd-59/ 2023-05-31T20:44:50.128 DEBUG:boto:auth_path=/pklxkd-59/ 2023-05-31T20:44:50.129 DEBUG:boto:path=/pklxkd-59/?prefix=obj68 2023-05-31T20:44:50.129 DEBUG:boto:auth_path=/pklxkd-59/?prefix=obj68 2023-05-31T20:44:50.130 DEBUG:boto:Method: GET 2023-05-31T20:44:50.130 DEBUG:boto:Path: /pklxkd-59/?prefix=obj68 2023-05-31T20:44:50.131 DEBUG:boto:Data: 2023-05-31T20:44:50.131 DEBUG:boto:Headers: {} 2023-05-31T20:44:50.132 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:44:50.132 DEBUG:boto:Port: 8001 2023-05-31T20:44:50.132 DEBUG:boto:Params: {} 2023-05-31T20:44:50.133 DEBUG:boto:Token: None 2023-05-31T20:44:50.133 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:44:50 GMT /pklxkd-59/ 2023-05-31T20:44:50.134 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:yNGiUWAW4Yg2l1bOAUM5KexBO6Q= 2023-05-31T20:44:50.134 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:44:50 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:yNGiUWAW4Yg2l1bOAUM5KexBO6Q=', 'Content-Length': '0'} 2023-05-31T20:44:50.136 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000094ebc387b139d07-006477b1c2-1c17-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:44:50 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:44:50.137 DEBUG:boto:b'pklxkd-59obj681000falseobj682023-05-31T20:43:08.066Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:44:50.138 DEBUG:boto:path=/pklxkd-59/ 2023-05-31T20:44:50.138 DEBUG:boto:auth_path=/pklxkd-59/ 2023-05-31T20:44:50.138 DEBUG:boto:path=/pklxkd-59/?prefix=obj68 2023-05-31T20:44:50.139 DEBUG:boto:auth_path=/pklxkd-59/?prefix=obj68 2023-05-31T20:44:50.139 DEBUG:boto:Method: GET 2023-05-31T20:44:50.140 DEBUG:boto:Path: /pklxkd-59/?prefix=obj68 2023-05-31T20:44:50.140 DEBUG:boto:Data: 2023-05-31T20:44:50.140 DEBUG:boto:Headers: {} 2023-05-31T20:44:50.141 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:44:50.141 DEBUG:boto:Port: 8000 2023-05-31T20:44:50.141 DEBUG:boto:Params: {} 2023-05-31T20:44:50.142 DEBUG:boto:Token: None 2023-05-31T20:44:50.142 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:44:50 GMT /pklxkd-59/ 2023-05-31T20:44:50.143 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:yNGiUWAW4Yg2l1bOAUM5KexBO6Q= 2023-05-31T20:44:50.143 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:44:50 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:yNGiUWAW4Yg2l1bOAUM5KexBO6Q=', 'Content-Length': '0'} 2023-05-31T20:44:50.145 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000095071ad1b6c99b1c-006477b1c2-1630-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:44:50 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:44:50.146 DEBUG:boto:b'pklxkd-59obj681000falseobj682023-05-31T20:43:08.066Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:44:50.147 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj85 2023-05-31T20:44:50.147 DEBUG:boto:path=/pklxkd-59/obj85 2023-05-31T20:44:50.147 DEBUG:boto:auth_path=/pklxkd-59/obj85 2023-05-31T20:44:50.148 DEBUG:boto:path=/pklxkd-59/obj85?versionId=null 2023-05-31T20:44:50.148 DEBUG:boto:auth_path=/pklxkd-59/obj85?versionId=null 2023-05-31T20:44:50.149 DEBUG:boto:Method: GET 2023-05-31T20:44:50.149 DEBUG:boto:Path: /pklxkd-59/obj85?versionId=null 2023-05-31T20:44:50.149 DEBUG:boto:Data: 2023-05-31T20:44:50.150 DEBUG:boto:Headers: {} 2023-05-31T20:44:50.150 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:44:50.150 DEBUG:boto:Port: 8001 2023-05-31T20:44:50.151 DEBUG:boto:Params: {} 2023-05-31T20:44:50.151 DEBUG:boto:Token: None 2023-05-31T20:44:50.151 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:44:50 GMT /pklxkd-59/obj85?versionId=null 2023-05-31T20:44:50.152 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:fn/noqC4b/PcAAy0MTqn1E8aDug= 2023-05-31T20:44:50.152 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:44:50 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:fn/noqC4b/PcAAy0MTqn1E8aDug=', 'Content-Length': '0'} 2023-05-31T20:44:50.154 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:43:08 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000010de1d9709592599-006477b1c2-1c17-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:44:50 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:44:50.155 DEBUG:boto:path=/pklxkd-59/obj85 2023-05-31T20:44:50.156 DEBUG:boto:auth_path=/pklxkd-59/obj85 2023-05-31T20:44:50.156 DEBUG:boto:path=/pklxkd-59/obj85?versionId=null 2023-05-31T20:44:50.156 DEBUG:boto:auth_path=/pklxkd-59/obj85?versionId=null 2023-05-31T20:44:50.157 DEBUG:boto:Method: GET 2023-05-31T20:44:50.157 DEBUG:boto:Path: /pklxkd-59/obj85?versionId=null 2023-05-31T20:44:50.157 DEBUG:boto:Data: 2023-05-31T20:44:50.158 DEBUG:boto:Headers: {} 2023-05-31T20:44:50.158 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:44:50.158 DEBUG:boto:Port: 8000 2023-05-31T20:44:50.159 DEBUG:boto:Params: {} 2023-05-31T20:44:50.159 DEBUG:boto:Token: None 2023-05-31T20:44:50.160 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:44:50 GMT /pklxkd-59/obj85?versionId=null 2023-05-31T20:44:50.160 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:fn/noqC4b/PcAAy0MTqn1E8aDug= 2023-05-31T20:44:50.161 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:44:50 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:fn/noqC4b/PcAAy0MTqn1E8aDug=', 'Content-Length': '0'} 2023-05-31T20:44:50.163 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:43:08 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004079250a20fa1b9e-006477b1c2-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:44:50 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:44:50.204 DEBUG:boto:path=/pklxkd-59/obj85 2023-05-31T20:44:50.205 DEBUG:boto:auth_path=/pklxkd-59/obj85 2023-05-31T20:44:50.205 DEBUG:boto:path=/pklxkd-59/obj85?versionId=null 2023-05-31T20:44:50.205 DEBUG:boto:auth_path=/pklxkd-59/obj85?versionId=null 2023-05-31T20:44:50.206 DEBUG:boto:Method: HEAD 2023-05-31T20:44:50.206 DEBUG:boto:Path: /pklxkd-59/obj85?versionId=null 2023-05-31T20:44:50.206 DEBUG:boto:Data: 2023-05-31T20:44:50.207 DEBUG:boto:Headers: {} 2023-05-31T20:44:50.207 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:44:50.207 DEBUG:boto:Port: 8001 2023-05-31T20:44:50.208 DEBUG:boto:Params: {} 2023-05-31T20:44:50.208 DEBUG:boto:Token: None 2023-05-31T20:44:50.209 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:44:50 GMT /pklxkd-59/obj85?versionId=null 2023-05-31T20:44:50.209 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:ynZNxbLWyjT/TmHwR0KpI5SIgJc= 2023-05-31T20:44:50.209 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:44:50 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:ynZNxbLWyjT/TmHwR0KpI5SIgJc=', 'Content-Length': '0'} 2023-05-31T20:44:50.211 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:43:08 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000da124ad6e5991752-006477b1c2-1c17-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:44:50 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:44:50.212 DEBUG:boto:path=/pklxkd-59/obj85 2023-05-31T20:44:50.212 DEBUG:boto:auth_path=/pklxkd-59/obj85 2023-05-31T20:44:50.213 DEBUG:boto:path=/pklxkd-59/obj85?versionId=null 2023-05-31T20:44:50.213 DEBUG:boto:auth_path=/pklxkd-59/obj85?versionId=null 2023-05-31T20:44:50.213 DEBUG:boto:Method: HEAD 2023-05-31T20:44:50.214 DEBUG:boto:Path: /pklxkd-59/obj85?versionId=null 2023-05-31T20:44:50.214 DEBUG:boto:Data: 2023-05-31T20:44:50.214 DEBUG:boto:Headers: {} 2023-05-31T20:44:50.215 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:44:50.215 DEBUG:boto:Port: 8000 2023-05-31T20:44:50.215 DEBUG:boto:Params: {} 2023-05-31T20:44:50.216 DEBUG:boto:Token: None 2023-05-31T20:44:50.216 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:44:50 GMT /pklxkd-59/obj85?versionId=null 2023-05-31T20:44:50.217 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:ynZNxbLWyjT/TmHwR0KpI5SIgJc= 2023-05-31T20:44:50.217 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:44:50 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:ynZNxbLWyjT/TmHwR0KpI5SIgJc=', 'Content-Length': '0'} 2023-05-31T20:44:50.219 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:43:08 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002aba1d5416cf2567-006477b1c2-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:44:50 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:44:50.219 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj85 2023-05-31T20:44:50.220 DEBUG:boto:path=/pklxkd-59/obj85 2023-05-31T20:44:50.220 DEBUG:boto:auth_path=/pklxkd-59/obj85 2023-05-31T20:44:50.221 DEBUG:boto:path=/pklxkd-59/obj85?versionId=null 2023-05-31T20:44:50.221 DEBUG:boto:auth_path=/pklxkd-59/obj85?versionId=null 2023-05-31T20:44:50.221 DEBUG:boto:Method: GET 2023-05-31T20:44:50.222 DEBUG:boto:Path: /pklxkd-59/obj85?versionId=null 2023-05-31T20:44:50.222 DEBUG:boto:Data: 2023-05-31T20:44:50.222 DEBUG:boto:Headers: {} 2023-05-31T20:44:50.223 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:44:50.223 DEBUG:boto:Port: 8001 2023-05-31T20:44:50.223 DEBUG:boto:Params: {} 2023-05-31T20:44:50.224 DEBUG:boto:Token: None 2023-05-31T20:44:50.224 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:44:50 GMT /pklxkd-59/obj85?versionId=null 2023-05-31T20:44:50.225 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:fn/noqC4b/PcAAy0MTqn1E8aDug= 2023-05-31T20:44:50.225 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:44:50 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:fn/noqC4b/PcAAy0MTqn1E8aDug=', 'Content-Length': '0'} 2023-05-31T20:44:50.227 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:43:08 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000bc97d579b623f1e7-006477b1c2-1c17-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:44:50 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:44:50.227 DEBUG:boto:path=/pklxkd-59/obj85 2023-05-31T20:44:50.228 DEBUG:boto:auth_path=/pklxkd-59/obj85 2023-05-31T20:44:50.228 DEBUG:boto:path=/pklxkd-59/obj85?versionId=null 2023-05-31T20:44:50.229 DEBUG:boto:auth_path=/pklxkd-59/obj85?versionId=null 2023-05-31T20:44:50.229 DEBUG:boto:Method: GET 2023-05-31T20:44:50.229 DEBUG:boto:Path: /pklxkd-59/obj85?versionId=null 2023-05-31T20:44:50.230 DEBUG:boto:Data: 2023-05-31T20:44:50.230 DEBUG:boto:Headers: {} 2023-05-31T20:44:50.230 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:44:50.231 DEBUG:boto:Port: 8000 2023-05-31T20:44:50.231 DEBUG:boto:Params: {} 2023-05-31T20:44:50.231 DEBUG:boto:Token: None 2023-05-31T20:44:50.232 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:44:50 GMT /pklxkd-59/obj85?versionId=null 2023-05-31T20:44:50.232 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:fn/noqC4b/PcAAy0MTqn1E8aDug= 2023-05-31T20:44:50.233 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:44:50 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:fn/noqC4b/PcAAy0MTqn1E8aDug=', 'Content-Length': '0'} 2023-05-31T20:44:50.234 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:43:08 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000040387fa1f9a5d1a6-006477b1c2-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:44:50 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:44:50.235 DEBUG:boto:path=/pklxkd-59/ 2023-05-31T20:44:50.235 DEBUG:boto:auth_path=/pklxkd-59/ 2023-05-31T20:44:50.236 DEBUG:boto:path=/pklxkd-59/?prefix=obj85 2023-05-31T20:44:50.236 DEBUG:boto:auth_path=/pklxkd-59/?prefix=obj85 2023-05-31T20:44:50.237 DEBUG:boto:Method: GET 2023-05-31T20:44:50.237 DEBUG:boto:Path: /pklxkd-59/?prefix=obj85 2023-05-31T20:44:50.237 DEBUG:boto:Data: 2023-05-31T20:44:50.238 DEBUG:boto:Headers: {} 2023-05-31T20:44:50.238 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:44:50.238 DEBUG:boto:Port: 8001 2023-05-31T20:44:50.239 DEBUG:boto:Params: {} 2023-05-31T20:44:50.239 DEBUG:boto:Token: None 2023-05-31T20:44:50.240 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:44:50 GMT /pklxkd-59/ 2023-05-31T20:44:50.240 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:yNGiUWAW4Yg2l1bOAUM5KexBO6Q= 2023-05-31T20:44:50.240 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:44:50 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:yNGiUWAW4Yg2l1bOAUM5KexBO6Q=', 'Content-Length': '0'} 2023-05-31T20:44:50.243 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000412d811ba039ade6-006477b1c2-1c17-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:44:50 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:44:50.243 DEBUG:boto:b'pklxkd-59obj851000falseobj852023-05-31T20:43:08.086Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:44:50.244 DEBUG:boto:path=/pklxkd-59/ 2023-05-31T20:44:50.244 DEBUG:boto:auth_path=/pklxkd-59/ 2023-05-31T20:44:50.245 DEBUG:boto:path=/pklxkd-59/?prefix=obj85 2023-05-31T20:44:50.245 DEBUG:boto:auth_path=/pklxkd-59/?prefix=obj85 2023-05-31T20:44:50.245 DEBUG:boto:Method: GET 2023-05-31T20:44:50.246 DEBUG:boto:Path: /pklxkd-59/?prefix=obj85 2023-05-31T20:44:50.246 DEBUG:boto:Data: 2023-05-31T20:44:50.246 DEBUG:boto:Headers: {} 2023-05-31T20:44:50.247 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:44:50.247 DEBUG:boto:Port: 8000 2023-05-31T20:44:50.248 DEBUG:boto:Params: {} 2023-05-31T20:44:50.248 DEBUG:boto:Token: None 2023-05-31T20:44:50.249 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:44:50 GMT /pklxkd-59/ 2023-05-31T20:44:50.249 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:yNGiUWAW4Yg2l1bOAUM5KexBO6Q= 2023-05-31T20:44:50.249 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:44:50 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:yNGiUWAW4Yg2l1bOAUM5KexBO6Q=', 'Content-Length': '0'} 2023-05-31T20:44:50.252 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c4f7e6234a6fc9a4-006477b1c2-1630-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:44:50 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:44:50.253 DEBUG:boto:b'pklxkd-59obj851000falseobj852023-05-31T20:43:08.086Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:44:50.253 DEBUG:boto:path=/pklxkd-59/obj85 2023-05-31T20:44:50.254 DEBUG:boto:auth_path=/pklxkd-59/obj85 2023-05-31T20:44:50.254 DEBUG:boto:Method: HEAD 2023-05-31T20:44:50.254 DEBUG:boto:Path: /pklxkd-59/obj85 2023-05-31T20:44:50.255 DEBUG:boto:Data: 2023-05-31T20:44:50.255 DEBUG:boto:Headers: {} 2023-05-31T20:44:50.255 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:44:50.256 DEBUG:boto:Port: 8001 2023-05-31T20:44:50.256 DEBUG:boto:Params: {} 2023-05-31T20:44:50.256 DEBUG:boto:Token: None 2023-05-31T20:44:50.257 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:44:50 GMT /pklxkd-59/obj85 2023-05-31T20:44:50.257 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:Jut0+LIOhO/3DSQQIJOHhvAlxT4= 2023-05-31T20:44:50.258 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:44:50 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:Jut0+LIOhO/3DSQQIJOHhvAlxT4=', 'Content-Length': '0'} 2023-05-31T20:44:50.259 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:43:08 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ecfe4f2fa40bf775-006477b1c2-1c17-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:44:50 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:44:50.260 DEBUG:boto:path=/pklxkd-59/obj85 2023-05-31T20:44:50.260 DEBUG:boto:auth_path=/pklxkd-59/obj85 2023-05-31T20:44:50.261 DEBUG:boto:Method: HEAD 2023-05-31T20:44:50.261 DEBUG:boto:Path: /pklxkd-59/obj85 2023-05-31T20:44:50.261 DEBUG:boto:Data: 2023-05-31T20:44:50.262 DEBUG:boto:Headers: {} 2023-05-31T20:44:50.262 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:44:50.262 DEBUG:boto:Port: 8000 2023-05-31T20:44:50.263 DEBUG:boto:Params: {} 2023-05-31T20:44:50.263 DEBUG:boto:Token: None 2023-05-31T20:44:50.263 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:44:50 GMT /pklxkd-59/obj85 2023-05-31T20:44:50.264 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:Jut0+LIOhO/3DSQQIJOHhvAlxT4= 2023-05-31T20:44:50.264 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:44:50 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:Jut0+LIOhO/3DSQQIJOHhvAlxT4=', 'Content-Length': '0'} 2023-05-31T20:44:50.266 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:43:08 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000010ab381def35e56b-006477b1c2-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:44:50 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:44:50.266 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj85 2023-05-31T20:44:50.267 DEBUG:boto:path=/pklxkd-59/obj85 2023-05-31T20:44:50.267 DEBUG:boto:auth_path=/pklxkd-59/obj85 2023-05-31T20:44:50.267 DEBUG:boto:Method: GET 2023-05-31T20:44:50.268 DEBUG:boto:Path: /pklxkd-59/obj85 2023-05-31T20:44:50.268 DEBUG:boto:Data: 2023-05-31T20:44:50.269 DEBUG:boto:Headers: {} 2023-05-31T20:44:50.269 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:44:50.269 DEBUG:boto:Port: 8001 2023-05-31T20:44:50.270 DEBUG:boto:Params: {} 2023-05-31T20:44:50.270 DEBUG:boto:Token: None 2023-05-31T20:44:50.270 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:44:50 GMT /pklxkd-59/obj85 2023-05-31T20:44:50.271 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:MsdxVC1v+XKdzN5FRWSuB0yMfoY= 2023-05-31T20:44:50.271 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:44:50 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:MsdxVC1v+XKdzN5FRWSuB0yMfoY=', 'Content-Length': '0'} 2023-05-31T20:44:50.273 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:43:08 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002383f36849b33bc9-006477b1c2-1c17-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:44:50 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:44:50.273 DEBUG:boto:path=/pklxkd-59/obj85 2023-05-31T20:44:50.274 DEBUG:boto:auth_path=/pklxkd-59/obj85 2023-05-31T20:44:50.274 DEBUG:boto:Method: GET 2023-05-31T20:44:50.275 DEBUG:boto:Path: /pklxkd-59/obj85 2023-05-31T20:44:50.275 DEBUG:boto:Data: 2023-05-31T20:44:50.275 DEBUG:boto:Headers: {} 2023-05-31T20:44:50.276 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:44:50.276 DEBUG:boto:Port: 8000 2023-05-31T20:44:50.276 DEBUG:boto:Params: {} 2023-05-31T20:44:50.277 DEBUG:boto:Token: None 2023-05-31T20:44:50.277 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:44:50 GMT /pklxkd-59/obj85 2023-05-31T20:44:50.278 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:MsdxVC1v+XKdzN5FRWSuB0yMfoY= 2023-05-31T20:44:50.278 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:44:50 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:MsdxVC1v+XKdzN5FRWSuB0yMfoY=', 'Content-Length': '0'} 2023-05-31T20:44:50.280 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:43:08 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f34d453d5ec847fa-006477b1c2-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:44:50 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:44:50.281 DEBUG:boto:path=/pklxkd-59/ 2023-05-31T20:44:50.281 DEBUG:boto:auth_path=/pklxkd-59/ 2023-05-31T20:44:50.281 DEBUG:boto:path=/pklxkd-59/?prefix=obj85 2023-05-31T20:44:50.282 DEBUG:boto:auth_path=/pklxkd-59/?prefix=obj85 2023-05-31T20:44:50.282 DEBUG:boto:Method: GET 2023-05-31T20:44:50.282 DEBUG:boto:Path: /pklxkd-59/?prefix=obj85 2023-05-31T20:44:50.283 DEBUG:boto:Data: 2023-05-31T20:44:50.283 DEBUG:boto:Headers: {} 2023-05-31T20:44:50.283 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:44:50.284 DEBUG:boto:Port: 8001 2023-05-31T20:44:50.284 DEBUG:boto:Params: {} 2023-05-31T20:44:50.285 DEBUG:boto:Token: None 2023-05-31T20:44:50.285 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:44:50 GMT /pklxkd-59/ 2023-05-31T20:44:50.285 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:yNGiUWAW4Yg2l1bOAUM5KexBO6Q= 2023-05-31T20:44:50.286 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:44:50 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:yNGiUWAW4Yg2l1bOAUM5KexBO6Q=', 'Content-Length': '0'} 2023-05-31T20:44:50.288 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f731654dbb22023d-006477b1c2-1c17-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:44:50 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:44:50.328 DEBUG:boto:b'pklxkd-59obj851000falseobj852023-05-31T20:43:08.086Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:44:50.329 DEBUG:boto:path=/pklxkd-59/ 2023-05-31T20:44:50.329 DEBUG:boto:auth_path=/pklxkd-59/ 2023-05-31T20:44:50.330 DEBUG:boto:path=/pklxkd-59/?prefix=obj85 2023-05-31T20:44:50.330 DEBUG:boto:auth_path=/pklxkd-59/?prefix=obj85 2023-05-31T20:44:50.330 DEBUG:boto:Method: GET 2023-05-31T20:44:50.331 DEBUG:boto:Path: /pklxkd-59/?prefix=obj85 2023-05-31T20:44:50.331 DEBUG:boto:Data: 2023-05-31T20:44:50.331 DEBUG:boto:Headers: {} 2023-05-31T20:44:50.332 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:44:50.332 DEBUG:boto:Port: 8000 2023-05-31T20:44:50.332 DEBUG:boto:Params: {} 2023-05-31T20:44:50.333 DEBUG:boto:Token: None 2023-05-31T20:44:50.333 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:44:50 GMT /pklxkd-59/ 2023-05-31T20:44:50.334 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:yNGiUWAW4Yg2l1bOAUM5KexBO6Q= 2023-05-31T20:44:50.334 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:44:50 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:yNGiUWAW4Yg2l1bOAUM5KexBO6Q=', 'Content-Length': '0'} 2023-05-31T20:44:50.337 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d23592cfada00696-006477b1c2-1630-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:44:50 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:44:50.338 DEBUG:boto:b'pklxkd-59obj851000falseobj852023-05-31T20:43:08.086Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:44:50.338 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=pklxkd-59 zones={a2, a1} 2023-05-31T20:44:50.339 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'reshard', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'pklxkd-59', '--num-shards', '19', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-05-31T20:44:50.339 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'reshard', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'pklxkd-59', '--num-shards', '19', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-05-31T20:44:50.339 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket reshard --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --bucket pklxkd-59 --num-shards 19 --yes-i-really-mean-it --cluster c1 --debug-rgw 0 --debug-ms 0 2023-05-31T20:44:50.399 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2023-05-31T20:44:50.399 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2023-05-31T20:44:50.464 INFO:teuthology.orchestra.run.smithi099.stdout:tenant: 2023-05-31T20:44:50.464 INFO:teuthology.orchestra.run.smithi099.stdout:bucket name: pklxkd-59 2023-05-31T20:44:50.470 INFO:teuthology.orchestra.run.smithi099.stdout:total entries: 10 2023-05-31T20:44:50.546 DEBUG:boto:path=/pklxkd-59/ 2023-05-31T20:44:50.547 DEBUG:boto:auth_path=/pklxkd-59/ 2023-05-31T20:44:50.547 DEBUG:boto:Method: HEAD 2023-05-31T20:44:50.547 DEBUG:boto:Path: /pklxkd-59/ 2023-05-31T20:44:50.548 DEBUG:boto:Data: 2023-05-31T20:44:50.548 DEBUG:boto:Headers: {} 2023-05-31T20:44:50.548 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:44:50.549 DEBUG:boto:Port: 8000 2023-05-31T20:44:50.549 DEBUG:boto:Params: {} 2023-05-31T20:44:50.549 DEBUG:boto:Token: None 2023-05-31T20:44:50.550 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:44:50 GMT /pklxkd-59/ 2023-05-31T20:44:50.550 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:xu0ec7T8VfzGH1EqCKGyjtTrFU4= 2023-05-31T20:44:50.551 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:44:50 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:xu0ec7T8VfzGH1EqCKGyjtTrFU4=', 'Content-Length': '0'} 2023-05-31T20:44:50.554 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '10'), ('X-RGW-Bytes-Used', '30'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000007da097a07e04beff-006477b1c2-1630-a1'), ('Content-Length', '0'), ('Date', 'Wed, 31 May 2023 20:44:50 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:44:50.554 DEBUG:boto:path=/pklxkd-59/obj19 2023-05-31T20:44:50.555 DEBUG:boto:auth_path=/pklxkd-59/obj19 2023-05-31T20:44:50.555 DEBUG:boto:Method: PUT 2023-05-31T20:44:50.555 DEBUG:boto:Path: /pklxkd-59/obj19 2023-05-31T20:44:50.556 DEBUG:boto:Data: 2023-05-31T20:44:50.556 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-05-31T20:44:50.556 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:44:50.557 DEBUG:boto:Port: 8000 2023-05-31T20:44:50.557 DEBUG:boto:Params: {} 2023-05-31T20:44:50.558 DEBUG:boto:Token: None 2023-05-31T20:44:50.558 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Wed, 31 May 2023 20:44:50 GMT /pklxkd-59/obj19 2023-05-31T20:44:50.558 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:7ehbs5ngMftPIEIZjjSez+zHMLE= 2023-05-31T20:44:50.559 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Wed, 31 May 2023 20:44:50 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:7ehbs5ngMftPIEIZjjSez+zHMLE='} 2023-05-31T20:44:50.600 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000009075e8e31843da56-006477b1c2-1630-a1'), ('Date', 'Wed, 31 May 2023 20:44:50 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:44:50.601 DEBUG:boto:path=/pklxkd-59/ 2023-05-31T20:44:50.601 DEBUG:boto:auth_path=/pklxkd-59/ 2023-05-31T20:44:50.601 DEBUG:boto:Method: HEAD 2023-05-31T20:44:50.602 DEBUG:boto:Path: /pklxkd-59/ 2023-05-31T20:44:50.602 DEBUG:boto:Data: 2023-05-31T20:44:50.602 DEBUG:boto:Headers: {} 2023-05-31T20:44:50.603 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:44:50.603 DEBUG:boto:Port: 8000 2023-05-31T20:44:50.603 DEBUG:boto:Params: {} 2023-05-31T20:44:50.604 DEBUG:boto:Token: None 2023-05-31T20:44:50.604 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:44:50 GMT /pklxkd-59/ 2023-05-31T20:44:50.605 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:xu0ec7T8VfzGH1EqCKGyjtTrFU4= 2023-05-31T20:44:50.605 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:44:50 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:xu0ec7T8VfzGH1EqCKGyjtTrFU4=', 'Content-Length': '0'} 2023-05-31T20:44:50.607 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '11'), ('X-RGW-Bytes-Used', '33'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000911332b74282efe5-006477b1c2-1630-a1'), ('Content-Length', '0'), ('Date', 'Wed, 31 May 2023 20:44:50 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:44:50.608 DEBUG:boto:path=/pklxkd-59/obj38 2023-05-31T20:44:50.608 DEBUG:boto:auth_path=/pklxkd-59/obj38 2023-05-31T20:44:50.609 DEBUG:boto:Method: PUT 2023-05-31T20:44:50.609 DEBUG:boto:Path: /pklxkd-59/obj38 2023-05-31T20:44:50.609 DEBUG:boto:Data: 2023-05-31T20:44:50.610 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-05-31T20:44:50.610 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:44:50.610 DEBUG:boto:Port: 8000 2023-05-31T20:44:50.611 DEBUG:boto:Params: {} 2023-05-31T20:44:50.611 DEBUG:boto:Token: None 2023-05-31T20:44:50.611 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Wed, 31 May 2023 20:44:50 GMT /pklxkd-59/obj38 2023-05-31T20:44:50.612 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:hcQtRdzmBkESxMJA7nTyRnx6scs= 2023-05-31T20:44:50.612 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Wed, 31 May 2023 20:44:50 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:hcQtRdzmBkESxMJA7nTyRnx6scs='} 2023-05-31T20:44:50.618 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000031c8a7389192f631-006477b1c2-1630-a1'), ('Date', 'Wed, 31 May 2023 20:44:50 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:44:50.618 DEBUG:boto:path=/pklxkd-59/ 2023-05-31T20:44:50.619 DEBUG:boto:auth_path=/pklxkd-59/ 2023-05-31T20:44:50.619 DEBUG:boto:Method: HEAD 2023-05-31T20:44:50.619 DEBUG:boto:Path: /pklxkd-59/ 2023-05-31T20:44:50.620 DEBUG:boto:Data: 2023-05-31T20:44:50.620 DEBUG:boto:Headers: {} 2023-05-31T20:44:50.620 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:44:50.621 DEBUG:boto:Port: 8000 2023-05-31T20:44:50.621 DEBUG:boto:Params: {} 2023-05-31T20:44:50.621 DEBUG:boto:Token: None 2023-05-31T20:44:50.622 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:44:50 GMT /pklxkd-59/ 2023-05-31T20:44:50.622 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:xu0ec7T8VfzGH1EqCKGyjtTrFU4= 2023-05-31T20:44:50.622 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:44:50 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:xu0ec7T8VfzGH1EqCKGyjtTrFU4=', 'Content-Length': '0'} 2023-05-31T20:44:50.625 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '12'), ('X-RGW-Bytes-Used', '36'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000007c109b03dc21a2a8-006477b1c2-1630-a1'), ('Content-Length', '0'), ('Date', 'Wed, 31 May 2023 20:44:50 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:44:50.625 DEBUG:boto:path=/pklxkd-59/obj57 2023-05-31T20:44:50.626 DEBUG:boto:auth_path=/pklxkd-59/obj57 2023-05-31T20:44:50.626 DEBUG:boto:Method: PUT 2023-05-31T20:44:50.626 DEBUG:boto:Path: /pklxkd-59/obj57 2023-05-31T20:44:50.627 DEBUG:boto:Data: 2023-05-31T20:44:50.627 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-05-31T20:44:50.627 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:44:50.627 DEBUG:boto:Port: 8000 2023-05-31T20:44:50.628 DEBUG:boto:Params: {} 2023-05-31T20:44:50.628 DEBUG:boto:Token: None 2023-05-31T20:44:50.629 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Wed, 31 May 2023 20:44:50 GMT /pklxkd-59/obj57 2023-05-31T20:44:50.629 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:V6XddlmawX2SEYPbu8d3W9YRwSw= 2023-05-31T20:44:50.629 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Wed, 31 May 2023 20:44:50 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:V6XddlmawX2SEYPbu8d3W9YRwSw='} 2023-05-31T20:44:50.672 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000e667bcb92f92f6f2-006477b1c2-1630-a1'), ('Date', 'Wed, 31 May 2023 20:44:50 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:44:50.673 DEBUG:boto:path=/pklxkd-59/ 2023-05-31T20:44:50.673 DEBUG:boto:auth_path=/pklxkd-59/ 2023-05-31T20:44:50.673 DEBUG:boto:Method: HEAD 2023-05-31T20:44:50.674 DEBUG:boto:Path: /pklxkd-59/ 2023-05-31T20:44:50.674 DEBUG:boto:Data: 2023-05-31T20:44:50.674 DEBUG:boto:Headers: {} 2023-05-31T20:44:50.675 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:44:50.675 DEBUG:boto:Port: 8000 2023-05-31T20:44:50.675 DEBUG:boto:Params: {} 2023-05-31T20:44:50.676 DEBUG:boto:Token: None 2023-05-31T20:44:50.676 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:44:50 GMT /pklxkd-59/ 2023-05-31T20:44:50.677 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:xu0ec7T8VfzGH1EqCKGyjtTrFU4= 2023-05-31T20:44:50.677 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:44:50 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:xu0ec7T8VfzGH1EqCKGyjtTrFU4=', 'Content-Length': '0'} 2023-05-31T20:44:50.679 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '13'), ('X-RGW-Bytes-Used', '39'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000f44fc61af172e18a-006477b1c2-1630-a1'), ('Content-Length', '0'), ('Date', 'Wed, 31 May 2023 20:44:50 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:44:50.680 DEBUG:boto:path=/pklxkd-59/obj76 2023-05-31T20:44:50.680 DEBUG:boto:auth_path=/pklxkd-59/obj76 2023-05-31T20:44:50.681 DEBUG:boto:Method: PUT 2023-05-31T20:44:50.681 DEBUG:boto:Path: /pklxkd-59/obj76 2023-05-31T20:44:50.681 DEBUG:boto:Data: 2023-05-31T20:44:50.682 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-05-31T20:44:50.682 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:44:50.682 DEBUG:boto:Port: 8000 2023-05-31T20:44:50.683 DEBUG:boto:Params: {} 2023-05-31T20:44:50.683 DEBUG:boto:Token: None 2023-05-31T20:44:50.684 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Wed, 31 May 2023 20:44:50 GMT /pklxkd-59/obj76 2023-05-31T20:44:50.684 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:ZSQcKoCxewB4R/XOrltfU8ewOnA= 2023-05-31T20:44:50.684 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Wed, 31 May 2023 20:44:50 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:ZSQcKoCxewB4R/XOrltfU8ewOnA='} 2023-05-31T20:44:50.695 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000552793d25e16d04a-006477b1c2-1630-a1'), ('Date', 'Wed, 31 May 2023 20:44:50 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:44:50.696 DEBUG:boto:path=/pklxkd-59/ 2023-05-31T20:44:50.696 DEBUG:boto:auth_path=/pklxkd-59/ 2023-05-31T20:44:50.697 DEBUG:boto:Method: HEAD 2023-05-31T20:44:50.697 DEBUG:boto:Path: /pklxkd-59/ 2023-05-31T20:44:50.697 DEBUG:boto:Data: 2023-05-31T20:44:50.698 DEBUG:boto:Headers: {} 2023-05-31T20:44:50.698 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:44:50.698 DEBUG:boto:Port: 8000 2023-05-31T20:44:50.699 DEBUG:boto:Params: {} 2023-05-31T20:44:50.699 DEBUG:boto:Token: None 2023-05-31T20:44:50.699 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:44:50 GMT /pklxkd-59/ 2023-05-31T20:44:50.700 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:xu0ec7T8VfzGH1EqCKGyjtTrFU4= 2023-05-31T20:44:50.700 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:44:50 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:xu0ec7T8VfzGH1EqCKGyjtTrFU4=', 'Content-Length': '0'} 2023-05-31T20:44:50.703 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '14'), ('X-RGW-Bytes-Used', '42'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000db64a00268faf9c0-006477b1c2-1630-a1'), ('Content-Length', '0'), ('Date', 'Wed, 31 May 2023 20:44:50 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:44:50.703 DEBUG:boto:path=/pklxkd-59/obj95 2023-05-31T20:44:50.704 DEBUG:boto:auth_path=/pklxkd-59/obj95 2023-05-31T20:44:50.704 DEBUG:boto:Method: PUT 2023-05-31T20:44:50.704 DEBUG:boto:Path: /pklxkd-59/obj95 2023-05-31T20:44:50.705 DEBUG:boto:Data: 2023-05-31T20:44:50.705 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-05-31T20:44:50.705 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:44:50.706 DEBUG:boto:Port: 8000 2023-05-31T20:44:50.706 DEBUG:boto:Params: {} 2023-05-31T20:44:50.706 DEBUG:boto:Token: None 2023-05-31T20:44:50.707 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Wed, 31 May 2023 20:44:50 GMT /pklxkd-59/obj95 2023-05-31T20:44:50.707 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:PM22VJJ7xsZYDlAGgWsoNQoDnEg= 2023-05-31T20:44:50.707 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Wed, 31 May 2023 20:44:50 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:PM22VJJ7xsZYDlAGgWsoNQoDnEg='} 2023-05-31T20:44:50.713 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000000f72f31b49a1a9f2-006477b1c2-1630-a1'), ('Date', 'Wed, 31 May 2023 20:44:50 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:44:50.713 DEBUG:rgw_multi.tests:bucket checkpoint: source=a1 target=a2 bucket=pklxkd-59 2023-05-31T20:44:50.714 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'pklxkd-59', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-05-31T20:44:50.714 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'pklxkd-59', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-05-31T20:44:50.714 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket pklxkd-59 --source-zone a1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2023-05-31T20:44:50.770 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2023-05-31T20:44:50.771 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2023-05-31T20:44:50.827 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:44:50.822+0000 7f330d3ba780 1 waiting to reach latest gen 2, on 1.. 2023-05-31T20:44:55.828 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:44:55.823+0000 7f330d3ba780 1 waiting to reach latest gen 2, on 1.. 2023-05-31T20:45:00.829 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:45:00.823+0000 7f330d3ba780 1 waiting to reach latest gen 2, on 1.. 2023-05-31T20:45:05.829 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:45:05.824+0000 7f330d3ba780 1 waiting to reach latest gen 2, on 1.. 2023-05-31T20:45:10.831 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:45:10.827+0000 7f330d3ba780 1 waiting for incremental sync to catch up: 2023-05-31T20:45:10.832 INFO:teuthology.orchestra.run.smithi099.stderr: local status: [, , , , , , , , , , , , , , , , , , ] 2023-05-31T20:45:10.832 INFO:teuthology.orchestra.run.smithi099.stderr: remote markers: [, , , , , , 00000000002.47.6, 00000000004.74.6, , , , , , , , , , , ] 2023-05-31T20:45:15.833 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:45:15.829+0000 7f330d3ba780 1 waiting for incremental sync to catch up: 2023-05-31T20:45:15.833 INFO:teuthology.orchestra.run.smithi099.stderr: local status: [, , , , , , , , , , , , , , , , , , ] 2023-05-31T20:45:15.834 INFO:teuthology.orchestra.run.smithi099.stderr: remote markers: [, , , , , , 00000000002.47.6, 00000000004.74.6, , , , , , , , , , , ] 2023-05-31T20:45:20.835 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:45:20.830+0000 7f330d3ba780 1 waiting for incremental sync to catch up: 2023-05-31T20:45:20.835 INFO:teuthology.orchestra.run.smithi099.stderr: local status: [, , , , , , , , , , , , , , , , , , ] 2023-05-31T20:45:20.835 INFO:teuthology.orchestra.run.smithi099.stderr: remote markers: [, , , , , , 00000000002.47.6, 00000000004.74.6, , , , , , , , , , , ] 2023-05-31T20:45:25.836 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:45:25.832+0000 7f330d3ba780 1 waiting for incremental sync to catch up: 2023-05-31T20:45:25.837 INFO:teuthology.orchestra.run.smithi099.stderr: local status: [, , , , , , , , , , , , , , , , , , ] 2023-05-31T20:45:25.837 INFO:teuthology.orchestra.run.smithi099.stderr: remote markers: [, , , , , , 00000000002.47.6, 00000000004.74.6, , , , , , , , , , , ] 2023-05-31T20:45:30.838 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:45:30.832+0000 7f330d3ba780 1 waiting for incremental sync to catch up: 2023-05-31T20:45:30.838 INFO:teuthology.orchestra.run.smithi099.stderr: local status: [, , , , , , , 00000000004.74.6, , , , , , , , , , , ] 2023-05-31T20:45:30.838 INFO:teuthology.orchestra.run.smithi099.stderr: remote markers: [, , , , , , 00000000002.47.6, 00000000004.74.6, , , , , , , , , , , ] 2023-05-31T20:45:35.839 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:45:35.834+0000 7f330d3ba780 1 waiting for incremental sync to catch up: 2023-05-31T20:45:35.839 INFO:teuthology.orchestra.run.smithi099.stderr: local status: [, , , , , , , 00000000004.74.6, , , , , , , , , , , ] 2023-05-31T20:45:35.840 INFO:teuthology.orchestra.run.smithi099.stderr: remote markers: [, , , , , , 00000000002.47.6, 00000000004.74.6, , , , , , , , , , , ] 2023-05-31T20:45:40.841 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:45:40.836+0000 7f330d3ba780 1 waiting for incremental sync to catch up: 2023-05-31T20:45:40.841 INFO:teuthology.orchestra.run.smithi099.stderr: local status: [, , , , , , , 00000000004.74.6, , , , , , , , , , , ] 2023-05-31T20:45:40.842 INFO:teuthology.orchestra.run.smithi099.stderr: remote markers: [, , , , , , 00000000002.47.6, 00000000004.74.6, , , , , , , , , , , ] 2023-05-31T20:45:45.842 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:45:45.837+0000 7f330d3ba780 1 waiting for incremental sync to catch up: 2023-05-31T20:45:45.843 INFO:teuthology.orchestra.run.smithi099.stderr: local status: [, , , , , , , 00000000004.74.6, , , , , , , , , , , ] 2023-05-31T20:45:45.843 INFO:teuthology.orchestra.run.smithi099.stderr: remote markers: [, , , , , , 00000000002.47.6, 00000000004.74.6, , , , , , , , , , , ] 2023-05-31T20:45:50.844 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:45:50.839+0000 7f330d3ba780 1 waiting for incremental sync to catch up: 2023-05-31T20:45:50.844 INFO:teuthology.orchestra.run.smithi099.stderr: local status: [, , , , , , , 00000000004.74.6, , , , , , , , , , , ] 2023-05-31T20:45:50.845 INFO:teuthology.orchestra.run.smithi099.stderr: remote markers: [, , , , , , 00000000002.47.6, 00000000004.74.6, , , , , , , , , , , ] 2023-05-31T20:45:55.846 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:45:55.842+0000 7f330d3ba780 1 waiting for incremental sync to catch up: 2023-05-31T20:45:55.846 INFO:teuthology.orchestra.run.smithi099.stderr: local status: [, , , , , , , 00000000004.74.6, , , , , , , , , , , ] 2023-05-31T20:45:55.846 INFO:teuthology.orchestra.run.smithi099.stderr: remote markers: [, , , , , , 00000000002.47.6, 00000000004.74.6, , , , , , , , , , , ] 2023-05-31T20:46:00.847 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:46:00.843+0000 7f330d3ba780 1 waiting for incremental sync to catch up: 2023-05-31T20:46:00.848 INFO:teuthology.orchestra.run.smithi099.stderr: local status: [, , , , , , , 00000000004.74.6, , , , , , , , , , , ] 2023-05-31T20:46:00.848 INFO:teuthology.orchestra.run.smithi099.stderr: remote markers: [, , , , , , 00000000002.47.6, 00000000004.74.6, , , , , , , , , , , ] 2023-05-31T20:46:05.849 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:46:05.845+0000 7f330d3ba780 1 waiting for incremental sync to catch up: 2023-05-31T20:46:05.849 INFO:teuthology.orchestra.run.smithi099.stderr: local status: [, , , , , , , 00000000004.74.6, , , , , , , , , , , ] 2023-05-31T20:46:05.849 INFO:teuthology.orchestra.run.smithi099.stderr: remote markers: [, , , , , , 00000000002.47.6, 00000000004.74.6, , , , , , , , , , , ] 2023-05-31T20:46:10.851 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:46:10.846+0000 7f330d3ba780 1 bucket sync caught up with source: 2023-05-31T20:46:10.851 INFO:teuthology.orchestra.run.smithi099.stderr: local status: [, , , , , , 00000000002.47.6, 00000000004.74.6, , , , , , , , , , , ] 2023-05-31T20:46:10.851 INFO:teuthology.orchestra.run.smithi099.stderr: remote markers: [, , , , , , 00000000002.47.6, 00000000004.74.6, , , , , , , , , , , ] 2023-05-31T20:46:10.851 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:46:10.846+0000 7f330d3ba780 0 bucket checkpoint complete 2023-05-31T20:46:10.869 DEBUG:rgw_multi.tests:bucket checkpoint: source=a2 target=a1 bucket=pklxkd-59 2023-05-31T20:46:10.870 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'pklxkd-59', '--source-zone', 'a2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-05-31T20:46:10.870 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'pklxkd-59', '--source-zone', 'a2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-05-31T20:46:10.871 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket pklxkd-59 --source-zone a2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2023-05-31T20:46:10.934 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2023-05-31T20:46:10.935 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2023-05-31T20:46:10.980 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:46:10.976+0000 7f4dd404b780 1 waiting for incremental sync to catch up: 2023-05-31T20:46:10.980 INFO:teuthology.orchestra.run.smithi099.stderr: local status: [, 00000000002.3584.6, 00000000001.3888.6, 00000000001.2027.6, , 00000000001.2001.6, , , 00000000002.2013.6, 00000000004.2294.6, 00000000003.2008.6] 2023-05-31T20:46:10.980 INFO:teuthology.orchestra.run.smithi099.stderr: remote markers: [, 00000000002.3584.6, 00000000001.3888.6, 00000000001.2027.6, 00000000001.50.6, 00000000001.2001.6, , , 00000000002.2013.6, 00000000004.2294.6, 00000000003.2008.6] 2023-05-31T20:46:15.981 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:46:15.976+0000 7f4dd404b780 1 waiting for incremental sync to catch up: 2023-05-31T20:46:15.982 INFO:teuthology.orchestra.run.smithi099.stderr: local status: [, 00000000002.3584.6, 00000000001.3888.6, 00000000001.2027.6, , 00000000001.2001.6, , , 00000000002.2013.6, 00000000004.2294.6, 00000000003.2008.6] 2023-05-31T20:46:15.982 INFO:teuthology.orchestra.run.smithi099.stderr: remote markers: [, 00000000002.3584.6, 00000000001.3888.6, 00000000001.2027.6, 00000000001.50.6, 00000000001.2001.6, , , 00000000002.2013.6, 00000000004.2294.6, 00000000003.2008.6] 2023-05-31T20:46:20.982 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:46:20.977+0000 7f4dd404b780 1 waiting for incremental sync to catch up: 2023-05-31T20:46:20.983 INFO:teuthology.orchestra.run.smithi099.stderr: local status: [, 00000000002.3584.6, 00000000001.3888.6, 00000000001.2027.6, , 00000000001.2001.6, , , 00000000002.2013.6, 00000000004.2294.6, 00000000003.2008.6] 2023-05-31T20:46:20.983 INFO:teuthology.orchestra.run.smithi099.stderr: remote markers: [, 00000000002.3584.6, 00000000001.3888.6, 00000000001.2027.6, 00000000001.50.6, 00000000001.2001.6, , , 00000000002.2013.6, 00000000004.2294.6, 00000000003.2008.6] 2023-05-31T20:46:25.984 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:46:25.979+0000 7f4dd404b780 1 waiting for incremental sync to catch up: 2023-05-31T20:46:25.984 INFO:teuthology.orchestra.run.smithi099.stderr: local status: [, 00000000002.3584.6, 00000000001.3888.6, 00000000001.2027.6, , 00000000001.2001.6, , , 00000000002.2013.6, 00000000004.2294.6, 00000000003.2008.6] 2023-05-31T20:46:25.984 INFO:teuthology.orchestra.run.smithi099.stderr: remote markers: [, 00000000002.3584.6, 00000000001.3888.6, 00000000001.2027.6, 00000000001.50.6, 00000000001.2001.6, , , 00000000002.2013.6, 00000000004.2294.6, 00000000003.2008.6] 2023-05-31T20:46:30.985 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:46:30.980+0000 7f4dd404b780 1 bucket sync caught up with source: 2023-05-31T20:46:30.985 INFO:teuthology.orchestra.run.smithi099.stderr: local status: [, 00000000002.3584.6, 00000000001.3888.6, 00000000001.2027.6, 00000000001.50.6, 00000000001.2001.6, , , 00000000002.2013.6, 00000000004.2294.6, 00000000003.2008.6] 2023-05-31T20:46:30.986 INFO:teuthology.orchestra.run.smithi099.stderr: remote markers: [, 00000000002.3584.6, 00000000001.3888.6, 00000000001.2027.6, 00000000001.50.6, 00000000001.2001.6, , , 00000000002.2013.6, 00000000004.2294.6, 00000000003.2008.6] 2023-05-31T20:46:30.986 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:46:30.980+0000 7f4dd404b780 0 bucket checkpoint complete 2023-05-31T20:46:31.003 INFO:tasks.rgw_multi.zone_rados:comparing bucket=pklxkd-59 zones={a2, a1} 2023-05-31T20:46:31.003 DEBUG:boto:path=/pklxkd-59/ 2023-05-31T20:46:31.004 DEBUG:boto:auth_path=/pklxkd-59/ 2023-05-31T20:46:31.004 DEBUG:boto:Method: HEAD 2023-05-31T20:46:31.004 DEBUG:boto:Path: /pklxkd-59/ 2023-05-31T20:46:31.005 DEBUG:boto:Data: 2023-05-31T20:46:31.005 DEBUG:boto:Headers: {} 2023-05-31T20:46:31.005 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:46:31.006 DEBUG:boto:Port: 8001 2023-05-31T20:46:31.006 DEBUG:boto:Params: {} 2023-05-31T20:46:31.007 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8001} 2023-05-31T20:46:31.007 DEBUG:boto:Token: None 2023-05-31T20:46:31.008 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:46:31 GMT /pklxkd-59/ 2023-05-31T20:46:31.008 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:lxdZKmgiy5oFd7avGwrDbO0D72U= 2023-05-31T20:46:31.008 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:46:31 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:lxdZKmgiy5oFd7avGwrDbO0D72U=', 'Content-Length': '0'} 2023-05-31T20:46:31.012 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '15'), ('X-RGW-Bytes-Used', '45'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000737078706c2620ac-006477b227-1c17-a2'), ('Content-Length', '0'), ('Date', 'Wed, 31 May 2023 20:46:31 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:46:31.012 DEBUG:boto:path=/pklxkd-59/ 2023-05-31T20:46:31.013 DEBUG:boto:auth_path=/pklxkd-59/ 2023-05-31T20:46:31.013 DEBUG:boto:Method: HEAD 2023-05-31T20:46:31.014 DEBUG:boto:Path: /pklxkd-59/ 2023-05-31T20:46:31.014 DEBUG:boto:Data: 2023-05-31T20:46:31.014 DEBUG:boto:Headers: {} 2023-05-31T20:46:31.015 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:46:31.015 DEBUG:boto:Port: 8000 2023-05-31T20:46:31.015 DEBUG:boto:Params: {} 2023-05-31T20:46:31.016 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2023-05-31T20:46:31.016 DEBUG:boto:Token: None 2023-05-31T20:46:31.017 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:46:31 GMT /pklxkd-59/ 2023-05-31T20:46:31.017 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:lxdZKmgiy5oFd7avGwrDbO0D72U= 2023-05-31T20:46:31.017 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:46:31 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:lxdZKmgiy5oFd7avGwrDbO0D72U=', 'Content-Length': '0'} 2023-05-31T20:46:31.021 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '15'), ('X-RGW-Bytes-Used', '45'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000040bd273201bd3374-006477b227-1630-a1'), ('Content-Length', '0'), ('Date', 'Wed, 31 May 2023 20:46:31 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:46:31.021 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2023-05-31T20:46:31.022 DEBUG:boto:path=/pklxkd-59/ 2023-05-31T20:46:31.022 DEBUG:boto:auth_path=/pklxkd-59/ 2023-05-31T20:46:31.022 DEBUG:boto:path=/pklxkd-59/?versions&max-keys=999 2023-05-31T20:46:31.023 DEBUG:boto:auth_path=/pklxkd-59/?versions&max-keys=999 2023-05-31T20:46:31.023 DEBUG:boto:Method: GET 2023-05-31T20:46:31.023 DEBUG:boto:Path: /pklxkd-59/?versions&max-keys=999 2023-05-31T20:46:31.024 DEBUG:boto:Data: 2023-05-31T20:46:31.024 DEBUG:boto:Headers: {} 2023-05-31T20:46:31.025 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:46:31.025 DEBUG:boto:Port: 8001 2023-05-31T20:46:31.025 DEBUG:boto:Params: {} 2023-05-31T20:46:31.026 DEBUG:boto:Token: None 2023-05-31T20:46:31.026 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:46:31 GMT /pklxkd-59/?versions 2023-05-31T20:46:31.027 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:Sdk5zjgfzVKhMFr+HGIMjvtv7Aw= 2023-05-31T20:46:31.027 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:46:31 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:Sdk5zjgfzVKhMFr+HGIMjvtv7Aw=', 'Content-Length': '0'} 2023-05-31T20:46:31.029 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000685f6f5b8193b640-006477b227-1c17-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:46:31 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:46:31.030 DEBUG:boto:b'pklxkd-59999falseobj11nulltrue2023-05-31T20:39:26.427Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj17nulltrue2023-05-31T20:43:07.977Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj19nulltrue2023-05-31T20:44:50.557Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj22nulltrue2023-05-31T20:39:26.473Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj33nulltrue2023-05-31T20:39:26.534Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj34nulltrue2023-05-31T20:43:07.994Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj38nulltrue2023-05-31T20:44:50.610Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj44nulltrue2023-05-31T20:39:26.565Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj51nulltrue2023-05-31T20:43:08.011Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj55nulltrue2023-05-31T20:39:26.630Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj57nulltrue2023-05-31T20:44:50.627Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj68nulltrue2023-05-31T20:43:08.066Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj76nulltrue2023-05-31T20:44:50.682Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj85nulltrue2023-05-31T20:43:08.086Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj95nulltrue2023-05-31T20:44:50.705Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:46:31.031 DEBUG:tasks.rgw_multi.zone_rados:o=obj11 2023-05-31T20:46:31.032 DEBUG:tasks.rgw_multi.zone_rados:o=obj17 2023-05-31T20:46:31.032 DEBUG:tasks.rgw_multi.zone_rados:o=obj19 2023-05-31T20:46:31.032 DEBUG:tasks.rgw_multi.zone_rados:o=obj22 2023-05-31T20:46:31.033 DEBUG:tasks.rgw_multi.zone_rados:o=obj33 2023-05-31T20:46:31.033 DEBUG:tasks.rgw_multi.zone_rados:o=obj34 2023-05-31T20:46:31.033 DEBUG:tasks.rgw_multi.zone_rados:o=obj38 2023-05-31T20:46:31.033 DEBUG:tasks.rgw_multi.zone_rados:o=obj44 2023-05-31T20:46:31.034 DEBUG:tasks.rgw_multi.zone_rados:o=obj51 2023-05-31T20:46:31.034 DEBUG:tasks.rgw_multi.zone_rados:o=obj55 2023-05-31T20:46:31.034 DEBUG:tasks.rgw_multi.zone_rados:o=obj57 2023-05-31T20:46:31.035 DEBUG:tasks.rgw_multi.zone_rados:o=obj68 2023-05-31T20:46:31.035 DEBUG:tasks.rgw_multi.zone_rados:o=obj76 2023-05-31T20:46:31.035 DEBUG:tasks.rgw_multi.zone_rados:o=obj85 2023-05-31T20:46:31.035 DEBUG:tasks.rgw_multi.zone_rados:o=obj95 2023-05-31T20:46:31.036 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2023-05-31T20:46:31.036 DEBUG:boto:path=/pklxkd-59/ 2023-05-31T20:46:31.037 DEBUG:boto:auth_path=/pklxkd-59/ 2023-05-31T20:46:31.037 DEBUG:boto:path=/pklxkd-59/?versions&max-keys=999 2023-05-31T20:46:31.037 DEBUG:boto:auth_path=/pklxkd-59/?versions&max-keys=999 2023-05-31T20:46:31.038 DEBUG:boto:Method: GET 2023-05-31T20:46:31.038 DEBUG:boto:Path: /pklxkd-59/?versions&max-keys=999 2023-05-31T20:46:31.038 DEBUG:boto:Data: 2023-05-31T20:46:31.039 DEBUG:boto:Headers: {} 2023-05-31T20:46:31.039 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:46:31.039 DEBUG:boto:Port: 8000 2023-05-31T20:46:31.040 DEBUG:boto:Params: {} 2023-05-31T20:46:31.040 DEBUG:boto:Token: None 2023-05-31T20:46:31.040 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:46:31 GMT /pklxkd-59/?versions 2023-05-31T20:46:31.041 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:Sdk5zjgfzVKhMFr+HGIMjvtv7Aw= 2023-05-31T20:46:31.041 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:46:31 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:Sdk5zjgfzVKhMFr+HGIMjvtv7Aw=', 'Content-Length': '0'} 2023-05-31T20:46:31.044 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000aad9530518504ae1-006477b227-1630-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:46:31 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:46:31.045 DEBUG:boto:b'pklxkd-59999falseobj11nulltrue2023-05-31T20:39:26.427Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj17nulltrue2023-05-31T20:43:07.977Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj19nulltrue2023-05-31T20:44:50.557Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj22nulltrue2023-05-31T20:39:26.473Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj33nulltrue2023-05-31T20:39:26.534Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj34nulltrue2023-05-31T20:43:07.994Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj38nulltrue2023-05-31T20:44:50.610Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj44nulltrue2023-05-31T20:39:26.565Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj51nulltrue2023-05-31T20:43:08.011Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj55nulltrue2023-05-31T20:39:26.630Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj57nulltrue2023-05-31T20:44:50.627Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj68nulltrue2023-05-31T20:43:08.066Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj76nulltrue2023-05-31T20:44:50.682Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj85nulltrue2023-05-31T20:43:08.086Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj95nulltrue2023-05-31T20:44:50.705Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:46:31.046 DEBUG:tasks.rgw_multi.zone_rados:o=obj11 2023-05-31T20:46:31.046 DEBUG:tasks.rgw_multi.zone_rados:o=obj17 2023-05-31T20:46:31.046 DEBUG:tasks.rgw_multi.zone_rados:o=obj19 2023-05-31T20:46:31.047 DEBUG:tasks.rgw_multi.zone_rados:o=obj22 2023-05-31T20:46:31.047 DEBUG:tasks.rgw_multi.zone_rados:o=obj33 2023-05-31T20:46:31.047 DEBUG:tasks.rgw_multi.zone_rados:o=obj34 2023-05-31T20:46:31.048 DEBUG:tasks.rgw_multi.zone_rados:o=obj38 2023-05-31T20:46:31.048 DEBUG:tasks.rgw_multi.zone_rados:o=obj44 2023-05-31T20:46:31.048 DEBUG:tasks.rgw_multi.zone_rados:o=obj51 2023-05-31T20:46:31.049 DEBUG:tasks.rgw_multi.zone_rados:o=obj55 2023-05-31T20:46:31.049 DEBUG:tasks.rgw_multi.zone_rados:o=obj57 2023-05-31T20:46:31.049 DEBUG:tasks.rgw_multi.zone_rados:o=obj68 2023-05-31T20:46:31.049 DEBUG:tasks.rgw_multi.zone_rados:o=obj76 2023-05-31T20:46:31.050 DEBUG:tasks.rgw_multi.zone_rados:o=obj85 2023-05-31T20:46:31.050 DEBUG:tasks.rgw_multi.zone_rados:o=obj95 2023-05-31T20:46:31.050 DEBUG:boto:path=/pklxkd-59/ 2023-05-31T20:46:31.051 DEBUG:boto:auth_path=/pklxkd-59/ 2023-05-31T20:46:31.051 DEBUG:boto:path=/pklxkd-59/?versions&max-keys=999 2023-05-31T20:46:31.051 DEBUG:boto:auth_path=/pklxkd-59/?versions&max-keys=999 2023-05-31T20:46:31.052 DEBUG:boto:Method: GET 2023-05-31T20:46:31.052 DEBUG:boto:Path: /pklxkd-59/?versions&max-keys=999 2023-05-31T20:46:31.052 DEBUG:boto:Data: 2023-05-31T20:46:31.053 DEBUG:boto:Headers: {} 2023-05-31T20:46:31.053 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:46:31.053 DEBUG:boto:Port: 8001 2023-05-31T20:46:31.054 DEBUG:boto:Params: {} 2023-05-31T20:46:31.054 DEBUG:boto:Token: None 2023-05-31T20:46:31.055 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:46:31 GMT /pklxkd-59/?versions 2023-05-31T20:46:31.055 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:Sdk5zjgfzVKhMFr+HGIMjvtv7Aw= 2023-05-31T20:46:31.055 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:46:31 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:Sdk5zjgfzVKhMFr+HGIMjvtv7Aw=', 'Content-Length': '0'} 2023-05-31T20:46:31.058 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000fdb3df0261209497-006477b227-1c17-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:46:31 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:46:31.058 DEBUG:boto:b'pklxkd-59999falseobj11nulltrue2023-05-31T20:39:26.427Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj17nulltrue2023-05-31T20:43:07.977Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj19nulltrue2023-05-31T20:44:50.557Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj22nulltrue2023-05-31T20:39:26.473Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj33nulltrue2023-05-31T20:39:26.534Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj34nulltrue2023-05-31T20:43:07.994Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj38nulltrue2023-05-31T20:44:50.610Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj44nulltrue2023-05-31T20:39:26.565Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj51nulltrue2023-05-31T20:43:08.011Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj55nulltrue2023-05-31T20:39:26.630Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj57nulltrue2023-05-31T20:44:50.627Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj68nulltrue2023-05-31T20:43:08.066Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj76nulltrue2023-05-31T20:44:50.682Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj85nulltrue2023-05-31T20:43:08.086Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj95nulltrue2023-05-31T20:44:50.705Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:46:31.059 DEBUG:boto:path=/pklxkd-59/ 2023-05-31T20:46:31.060 DEBUG:boto:auth_path=/pklxkd-59/ 2023-05-31T20:46:31.060 DEBUG:boto:path=/pklxkd-59/?versions&max-keys=999 2023-05-31T20:46:31.060 DEBUG:boto:auth_path=/pklxkd-59/?versions&max-keys=999 2023-05-31T20:46:31.061 DEBUG:boto:Method: GET 2023-05-31T20:46:31.061 DEBUG:boto:Path: /pklxkd-59/?versions&max-keys=999 2023-05-31T20:46:31.062 DEBUG:boto:Data: 2023-05-31T20:46:31.062 DEBUG:boto:Headers: {} 2023-05-31T20:46:31.062 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:46:31.063 DEBUG:boto:Port: 8000 2023-05-31T20:46:31.063 DEBUG:boto:Params: {} 2023-05-31T20:46:31.063 DEBUG:boto:Token: None 2023-05-31T20:46:31.064 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:46:31 GMT /pklxkd-59/?versions 2023-05-31T20:46:31.064 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:Sdk5zjgfzVKhMFr+HGIMjvtv7Aw= 2023-05-31T20:46:31.065 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:46:31 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:Sdk5zjgfzVKhMFr+HGIMjvtv7Aw=', 'Content-Length': '0'} 2023-05-31T20:46:31.067 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d5774645bd738daf-006477b227-1630-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:46:31 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:46:31.068 DEBUG:boto:b'pklxkd-59999falseobj11nulltrue2023-05-31T20:39:26.427Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj17nulltrue2023-05-31T20:43:07.977Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj19nulltrue2023-05-31T20:44:50.557Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj22nulltrue2023-05-31T20:39:26.473Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj33nulltrue2023-05-31T20:39:26.534Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj34nulltrue2023-05-31T20:43:07.994Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj38nulltrue2023-05-31T20:44:50.610Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj44nulltrue2023-05-31T20:39:26.565Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj51nulltrue2023-05-31T20:43:08.011Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj55nulltrue2023-05-31T20:39:26.630Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj57nulltrue2023-05-31T20:44:50.627Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj68nulltrue2023-05-31T20:43:08.066Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj76nulltrue2023-05-31T20:44:50.682Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj85nulltrue2023-05-31T20:43:08.086Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj95nulltrue2023-05-31T20:44:50.705Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:46:31.069 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj11 2023-05-31T20:46:31.069 DEBUG:boto:path=/pklxkd-59/obj11 2023-05-31T20:46:31.070 DEBUG:boto:auth_path=/pklxkd-59/obj11 2023-05-31T20:46:31.070 DEBUG:boto:path=/pklxkd-59/obj11?versionId=null 2023-05-31T20:46:31.070 DEBUG:boto:auth_path=/pklxkd-59/obj11?versionId=null 2023-05-31T20:46:31.071 DEBUG:boto:Method: GET 2023-05-31T20:46:31.071 DEBUG:boto:Path: /pklxkd-59/obj11?versionId=null 2023-05-31T20:46:31.071 DEBUG:boto:Data: 2023-05-31T20:46:31.072 DEBUG:boto:Headers: {} 2023-05-31T20:46:31.072 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:46:31.073 DEBUG:boto:Port: 8001 2023-05-31T20:46:31.073 DEBUG:boto:Params: {} 2023-05-31T20:46:31.073 DEBUG:boto:Token: None 2023-05-31T20:46:31.074 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:46:31 GMT /pklxkd-59/obj11?versionId=null 2023-05-31T20:46:31.074 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:vyWAm5U9CPJbWrbf6PDT0L9K4W8= 2023-05-31T20:46:31.075 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:46:31 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:vyWAm5U9CPJbWrbf6PDT0L9K4W8=', 'Content-Length': '0'} 2023-05-31T20:46:31.076 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:39:26 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003550d21ac4e146ca-006477b227-1c17-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:46:31 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:46:31.077 DEBUG:boto:path=/pklxkd-59/obj11 2023-05-31T20:46:31.077 DEBUG:boto:auth_path=/pklxkd-59/obj11 2023-05-31T20:46:31.077 DEBUG:boto:path=/pklxkd-59/obj11?versionId=null 2023-05-31T20:46:31.078 DEBUG:boto:auth_path=/pklxkd-59/obj11?versionId=null 2023-05-31T20:46:31.078 DEBUG:boto:Method: GET 2023-05-31T20:46:31.079 DEBUG:boto:Path: /pklxkd-59/obj11?versionId=null 2023-05-31T20:46:31.079 DEBUG:boto:Data: 2023-05-31T20:46:31.079 DEBUG:boto:Headers: {} 2023-05-31T20:46:31.080 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:46:31.080 DEBUG:boto:Port: 8000 2023-05-31T20:46:31.080 DEBUG:boto:Params: {} 2023-05-31T20:46:31.081 DEBUG:boto:Token: None 2023-05-31T20:46:31.081 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:46:31 GMT /pklxkd-59/obj11?versionId=null 2023-05-31T20:46:31.082 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:vyWAm5U9CPJbWrbf6PDT0L9K4W8= 2023-05-31T20:46:31.082 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:46:31 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:vyWAm5U9CPJbWrbf6PDT0L9K4W8=', 'Content-Length': '0'} 2023-05-31T20:46:31.084 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:39:26 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000043fb85badc2463bc-006477b227-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:46:31 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:46:31.084 DEBUG:boto:path=/pklxkd-59/obj11 2023-05-31T20:46:31.085 DEBUG:boto:auth_path=/pklxkd-59/obj11 2023-05-31T20:46:31.085 DEBUG:boto:path=/pklxkd-59/obj11?versionId=null 2023-05-31T20:46:31.085 DEBUG:boto:auth_path=/pklxkd-59/obj11?versionId=null 2023-05-31T20:46:31.086 DEBUG:boto:Method: HEAD 2023-05-31T20:46:31.086 DEBUG:boto:Path: /pklxkd-59/obj11?versionId=null 2023-05-31T20:46:31.086 DEBUG:boto:Data: 2023-05-31T20:46:31.087 DEBUG:boto:Headers: {} 2023-05-31T20:46:31.087 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:46:31.087 DEBUG:boto:Port: 8001 2023-05-31T20:46:31.088 DEBUG:boto:Params: {} 2023-05-31T20:46:31.088 DEBUG:boto:Token: None 2023-05-31T20:46:31.088 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:46:31 GMT /pklxkd-59/obj11?versionId=null 2023-05-31T20:46:31.089 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:QtrFBXAXhbZcDU/kRAxCua/jinA= 2023-05-31T20:46:31.089 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:46:31 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:QtrFBXAXhbZcDU/kRAxCua/jinA=', 'Content-Length': '0'} 2023-05-31T20:46:31.091 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:39:26 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000007d3b82b1a71871b-006477b227-1c17-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:46:31 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:46:31.091 DEBUG:boto:path=/pklxkd-59/obj11 2023-05-31T20:46:31.092 DEBUG:boto:auth_path=/pklxkd-59/obj11 2023-05-31T20:46:31.092 DEBUG:boto:path=/pklxkd-59/obj11?versionId=null 2023-05-31T20:46:31.092 DEBUG:boto:auth_path=/pklxkd-59/obj11?versionId=null 2023-05-31T20:46:31.093 DEBUG:boto:Method: HEAD 2023-05-31T20:46:31.093 DEBUG:boto:Path: /pklxkd-59/obj11?versionId=null 2023-05-31T20:46:31.093 DEBUG:boto:Data: 2023-05-31T20:46:31.094 DEBUG:boto:Headers: {} 2023-05-31T20:46:31.094 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:46:31.094 DEBUG:boto:Port: 8000 2023-05-31T20:46:31.095 DEBUG:boto:Params: {} 2023-05-31T20:46:31.095 DEBUG:boto:Token: None 2023-05-31T20:46:31.095 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:46:31 GMT /pklxkd-59/obj11?versionId=null 2023-05-31T20:46:31.096 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:QtrFBXAXhbZcDU/kRAxCua/jinA= 2023-05-31T20:46:31.096 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:46:31 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:QtrFBXAXhbZcDU/kRAxCua/jinA=', 'Content-Length': '0'} 2023-05-31T20:46:31.098 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:39:26 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000faed58867cec3de6-006477b227-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:46:31 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:46:31.098 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj11 2023-05-31T20:46:31.099 DEBUG:boto:path=/pklxkd-59/obj11 2023-05-31T20:46:31.099 DEBUG:boto:auth_path=/pklxkd-59/obj11 2023-05-31T20:46:31.099 DEBUG:boto:path=/pklxkd-59/obj11?versionId=null 2023-05-31T20:46:31.100 DEBUG:boto:auth_path=/pklxkd-59/obj11?versionId=null 2023-05-31T20:46:31.100 DEBUG:boto:Method: GET 2023-05-31T20:46:31.100 DEBUG:boto:Path: /pklxkd-59/obj11?versionId=null 2023-05-31T20:46:31.101 DEBUG:boto:Data: 2023-05-31T20:46:31.101 DEBUG:boto:Headers: {} 2023-05-31T20:46:31.102 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:46:31.102 DEBUG:boto:Port: 8001 2023-05-31T20:46:31.102 DEBUG:boto:Params: {} 2023-05-31T20:46:31.103 DEBUG:boto:Token: None 2023-05-31T20:46:31.103 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:46:31 GMT /pklxkd-59/obj11?versionId=null 2023-05-31T20:46:31.103 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:vyWAm5U9CPJbWrbf6PDT0L9K4W8= 2023-05-31T20:46:31.104 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:46:31 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:vyWAm5U9CPJbWrbf6PDT0L9K4W8=', 'Content-Length': '0'} 2023-05-31T20:46:31.106 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:39:26 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004424f5d99c7ea9dd-006477b227-1c17-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:46:31 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:46:31.106 DEBUG:boto:path=/pklxkd-59/obj11 2023-05-31T20:46:31.107 DEBUG:boto:auth_path=/pklxkd-59/obj11 2023-05-31T20:46:31.107 DEBUG:boto:path=/pklxkd-59/obj11?versionId=null 2023-05-31T20:46:31.107 DEBUG:boto:auth_path=/pklxkd-59/obj11?versionId=null 2023-05-31T20:46:31.108 DEBUG:boto:Method: GET 2023-05-31T20:46:31.108 DEBUG:boto:Path: /pklxkd-59/obj11?versionId=null 2023-05-31T20:46:31.108 DEBUG:boto:Data: 2023-05-31T20:46:31.109 DEBUG:boto:Headers: {} 2023-05-31T20:46:31.109 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:46:31.109 DEBUG:boto:Port: 8000 2023-05-31T20:46:31.110 DEBUG:boto:Params: {} 2023-05-31T20:46:31.110 DEBUG:boto:Token: None 2023-05-31T20:46:31.111 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:46:31 GMT /pklxkd-59/obj11?versionId=null 2023-05-31T20:46:31.111 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:vyWAm5U9CPJbWrbf6PDT0L9K4W8= 2023-05-31T20:46:31.111 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:46:31 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:vyWAm5U9CPJbWrbf6PDT0L9K4W8=', 'Content-Length': '0'} 2023-05-31T20:46:31.113 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:39:26 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000057a20d4041a136a1-006477b227-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:46:31 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:46:31.114 DEBUG:boto:path=/pklxkd-59/ 2023-05-31T20:46:31.114 DEBUG:boto:auth_path=/pklxkd-59/ 2023-05-31T20:46:31.115 DEBUG:boto:path=/pklxkd-59/?prefix=obj11 2023-05-31T20:46:31.115 DEBUG:boto:auth_path=/pklxkd-59/?prefix=obj11 2023-05-31T20:46:31.115 DEBUG:boto:Method: GET 2023-05-31T20:46:31.116 DEBUG:boto:Path: /pklxkd-59/?prefix=obj11 2023-05-31T20:46:31.116 DEBUG:boto:Data: 2023-05-31T20:46:31.116 DEBUG:boto:Headers: {} 2023-05-31T20:46:31.117 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:46:31.117 DEBUG:boto:Port: 8001 2023-05-31T20:46:31.117 DEBUG:boto:Params: {} 2023-05-31T20:46:31.118 DEBUG:boto:Token: None 2023-05-31T20:46:31.118 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:46:31 GMT /pklxkd-59/ 2023-05-31T20:46:31.119 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:gTuLAyT7WsrIPapdTlEudfl02iI= 2023-05-31T20:46:31.119 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:46:31 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:gTuLAyT7WsrIPapdTlEudfl02iI=', 'Content-Length': '0'} 2023-05-31T20:46:31.121 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000070e9c5cbc3cc4d08-006477b227-1c17-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:46:31 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:46:31.122 DEBUG:boto:b'pklxkd-59obj111000falseobj112023-05-31T20:39:26.427Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:46:31.123 DEBUG:boto:path=/pklxkd-59/ 2023-05-31T20:46:31.123 DEBUG:boto:auth_path=/pklxkd-59/ 2023-05-31T20:46:31.123 DEBUG:boto:path=/pklxkd-59/?prefix=obj11 2023-05-31T20:46:31.124 DEBUG:boto:auth_path=/pklxkd-59/?prefix=obj11 2023-05-31T20:46:31.124 DEBUG:boto:Method: GET 2023-05-31T20:46:31.124 DEBUG:boto:Path: /pklxkd-59/?prefix=obj11 2023-05-31T20:46:31.125 DEBUG:boto:Data: 2023-05-31T20:46:31.125 DEBUG:boto:Headers: {} 2023-05-31T20:46:31.125 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:46:31.126 DEBUG:boto:Port: 8000 2023-05-31T20:46:31.126 DEBUG:boto:Params: {} 2023-05-31T20:46:31.127 DEBUG:boto:Token: None 2023-05-31T20:46:31.127 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:46:31 GMT /pklxkd-59/ 2023-05-31T20:46:31.127 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:gTuLAyT7WsrIPapdTlEudfl02iI= 2023-05-31T20:46:31.128 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:46:31 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:gTuLAyT7WsrIPapdTlEudfl02iI=', 'Content-Length': '0'} 2023-05-31T20:46:31.130 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000082adb466a988eab6-006477b227-1630-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:46:31 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:46:31.131 DEBUG:boto:b'pklxkd-59obj111000falseobj112023-05-31T20:39:26.427Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:46:31.131 DEBUG:boto:path=/pklxkd-59/obj11 2023-05-31T20:46:31.132 DEBUG:boto:auth_path=/pklxkd-59/obj11 2023-05-31T20:46:31.132 DEBUG:boto:Method: HEAD 2023-05-31T20:46:31.132 DEBUG:boto:Path: /pklxkd-59/obj11 2023-05-31T20:46:31.133 DEBUG:boto:Data: 2023-05-31T20:46:31.133 DEBUG:boto:Headers: {} 2023-05-31T20:46:31.133 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:46:31.134 DEBUG:boto:Port: 8001 2023-05-31T20:46:31.134 DEBUG:boto:Params: {} 2023-05-31T20:46:31.135 DEBUG:boto:Token: None 2023-05-31T20:46:31.135 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:46:31 GMT /pklxkd-59/obj11 2023-05-31T20:46:31.135 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:oHYhq682WYiiktQCvSdZY/9EJcg= 2023-05-31T20:46:31.136 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:46:31 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:oHYhq682WYiiktQCvSdZY/9EJcg=', 'Content-Length': '0'} 2023-05-31T20:46:31.137 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:39:26 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b78da280091b4277-006477b227-1c17-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:46:31 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:46:31.138 DEBUG:boto:path=/pklxkd-59/obj11 2023-05-31T20:46:31.138 DEBUG:boto:auth_path=/pklxkd-59/obj11 2023-05-31T20:46:31.139 DEBUG:boto:Method: HEAD 2023-05-31T20:46:31.139 DEBUG:boto:Path: /pklxkd-59/obj11 2023-05-31T20:46:31.139 DEBUG:boto:Data: 2023-05-31T20:46:31.140 DEBUG:boto:Headers: {} 2023-05-31T20:46:31.140 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:46:31.140 DEBUG:boto:Port: 8000 2023-05-31T20:46:31.141 DEBUG:boto:Params: {} 2023-05-31T20:46:31.141 DEBUG:boto:Token: None 2023-05-31T20:46:31.141 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:46:31 GMT /pklxkd-59/obj11 2023-05-31T20:46:31.142 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:oHYhq682WYiiktQCvSdZY/9EJcg= 2023-05-31T20:46:31.142 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:46:31 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:oHYhq682WYiiktQCvSdZY/9EJcg=', 'Content-Length': '0'} 2023-05-31T20:46:31.144 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:39:26 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000074504d79b6dd92d1-006477b227-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:46:31 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:46:31.144 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj11 2023-05-31T20:46:31.145 DEBUG:boto:path=/pklxkd-59/obj11 2023-05-31T20:46:31.145 DEBUG:boto:auth_path=/pklxkd-59/obj11 2023-05-31T20:46:31.145 DEBUG:boto:Method: GET 2023-05-31T20:46:31.146 DEBUG:boto:Path: /pklxkd-59/obj11 2023-05-31T20:46:31.146 DEBUG:boto:Data: 2023-05-31T20:46:31.146 DEBUG:boto:Headers: {} 2023-05-31T20:46:31.147 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:46:31.147 DEBUG:boto:Port: 8001 2023-05-31T20:46:31.147 DEBUG:boto:Params: {} 2023-05-31T20:46:31.148 DEBUG:boto:Token: None 2023-05-31T20:46:31.148 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:46:31 GMT /pklxkd-59/obj11 2023-05-31T20:46:31.149 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:ra/ZDWCyHzmGuefkaOas81+9Ip4= 2023-05-31T20:46:31.149 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:46:31 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:ra/ZDWCyHzmGuefkaOas81+9Ip4=', 'Content-Length': '0'} 2023-05-31T20:46:31.151 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:39:26 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006da0ca8f7823b4ac-006477b227-1c17-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:46:31 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:46:31.192 DEBUG:boto:path=/pklxkd-59/obj11 2023-05-31T20:46:31.193 DEBUG:boto:auth_path=/pklxkd-59/obj11 2023-05-31T20:46:31.193 DEBUG:boto:Method: GET 2023-05-31T20:46:31.194 DEBUG:boto:Path: /pklxkd-59/obj11 2023-05-31T20:46:31.194 DEBUG:boto:Data: 2023-05-31T20:46:31.194 DEBUG:boto:Headers: {} 2023-05-31T20:46:31.195 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:46:31.195 DEBUG:boto:Port: 8000 2023-05-31T20:46:31.195 DEBUG:boto:Params: {} 2023-05-31T20:46:31.196 DEBUG:boto:Token: None 2023-05-31T20:46:31.196 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:46:31 GMT /pklxkd-59/obj11 2023-05-31T20:46:31.197 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:ra/ZDWCyHzmGuefkaOas81+9Ip4= 2023-05-31T20:46:31.197 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:46:31 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:ra/ZDWCyHzmGuefkaOas81+9Ip4=', 'Content-Length': '0'} 2023-05-31T20:46:31.199 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:39:26 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e0703ad94e91c613-006477b227-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:46:31 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:46:31.199 DEBUG:boto:path=/pklxkd-59/ 2023-05-31T20:46:31.200 DEBUG:boto:auth_path=/pklxkd-59/ 2023-05-31T20:46:31.200 DEBUG:boto:path=/pklxkd-59/?prefix=obj11 2023-05-31T20:46:31.201 DEBUG:boto:auth_path=/pklxkd-59/?prefix=obj11 2023-05-31T20:46:31.201 DEBUG:boto:Method: GET 2023-05-31T20:46:31.201 DEBUG:boto:Path: /pklxkd-59/?prefix=obj11 2023-05-31T20:46:31.202 DEBUG:boto:Data: 2023-05-31T20:46:31.202 DEBUG:boto:Headers: {} 2023-05-31T20:46:31.202 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:46:31.203 DEBUG:boto:Port: 8001 2023-05-31T20:46:31.203 DEBUG:boto:Params: {} 2023-05-31T20:46:31.203 DEBUG:boto:Token: None 2023-05-31T20:46:31.204 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:46:31 GMT /pklxkd-59/ 2023-05-31T20:46:31.204 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:gTuLAyT7WsrIPapdTlEudfl02iI= 2023-05-31T20:46:31.205 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:46:31 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:gTuLAyT7WsrIPapdTlEudfl02iI=', 'Content-Length': '0'} 2023-05-31T20:46:31.207 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000069a44be01c53435a-006477b227-1c17-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:46:31 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:46:31.207 DEBUG:boto:b'pklxkd-59obj111000falseobj112023-05-31T20:39:26.427Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:46:31.208 DEBUG:boto:path=/pklxkd-59/ 2023-05-31T20:46:31.208 DEBUG:boto:auth_path=/pklxkd-59/ 2023-05-31T20:46:31.209 DEBUG:boto:path=/pklxkd-59/?prefix=obj11 2023-05-31T20:46:31.209 DEBUG:boto:auth_path=/pklxkd-59/?prefix=obj11 2023-05-31T20:46:31.209 DEBUG:boto:Method: GET 2023-05-31T20:46:31.210 DEBUG:boto:Path: /pklxkd-59/?prefix=obj11 2023-05-31T20:46:31.210 DEBUG:boto:Data: 2023-05-31T20:46:31.211 DEBUG:boto:Headers: {} 2023-05-31T20:46:31.211 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:46:31.211 DEBUG:boto:Port: 8000 2023-05-31T20:46:31.212 DEBUG:boto:Params: {} 2023-05-31T20:46:31.212 DEBUG:boto:Token: None 2023-05-31T20:46:31.212 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:46:31 GMT /pklxkd-59/ 2023-05-31T20:46:31.213 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:gTuLAyT7WsrIPapdTlEudfl02iI= 2023-05-31T20:46:31.213 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:46:31 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:gTuLAyT7WsrIPapdTlEudfl02iI=', 'Content-Length': '0'} 2023-05-31T20:46:31.216 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d3bfad4ccb19e8f2-006477b227-1630-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:46:31 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:46:31.217 DEBUG:boto:b'pklxkd-59obj111000falseobj112023-05-31T20:39:26.427Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:46:31.217 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj17 2023-05-31T20:46:31.218 DEBUG:boto:path=/pklxkd-59/obj17 2023-05-31T20:46:31.218 DEBUG:boto:auth_path=/pklxkd-59/obj17 2023-05-31T20:46:31.219 DEBUG:boto:path=/pklxkd-59/obj17?versionId=null 2023-05-31T20:46:31.219 DEBUG:boto:auth_path=/pklxkd-59/obj17?versionId=null 2023-05-31T20:46:31.219 DEBUG:boto:Method: GET 2023-05-31T20:46:31.220 DEBUG:boto:Path: /pklxkd-59/obj17?versionId=null 2023-05-31T20:46:31.220 DEBUG:boto:Data: 2023-05-31T20:46:31.220 DEBUG:boto:Headers: {} 2023-05-31T20:46:31.221 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:46:31.221 DEBUG:boto:Port: 8001 2023-05-31T20:46:31.221 DEBUG:boto:Params: {} 2023-05-31T20:46:31.222 DEBUG:boto:Token: None 2023-05-31T20:46:31.222 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:46:31 GMT /pklxkd-59/obj17?versionId=null 2023-05-31T20:46:31.223 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:/55SmAPyCbZBYh68rrYrK1wcTS0= 2023-05-31T20:46:31.223 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:46:31 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:/55SmAPyCbZBYh68rrYrK1wcTS0=', 'Content-Length': '0'} 2023-05-31T20:46:31.225 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:43:07 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003f6d345a375c3604-006477b227-1c17-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:46:31 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:46:31.225 DEBUG:boto:path=/pklxkd-59/obj17 2023-05-31T20:46:31.226 DEBUG:boto:auth_path=/pklxkd-59/obj17 2023-05-31T20:46:31.226 DEBUG:boto:path=/pklxkd-59/obj17?versionId=null 2023-05-31T20:46:31.226 DEBUG:boto:auth_path=/pklxkd-59/obj17?versionId=null 2023-05-31T20:46:31.227 DEBUG:boto:Method: GET 2023-05-31T20:46:31.227 DEBUG:boto:Path: /pklxkd-59/obj17?versionId=null 2023-05-31T20:46:31.227 DEBUG:boto:Data: 2023-05-31T20:46:31.228 DEBUG:boto:Headers: {} 2023-05-31T20:46:31.228 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:46:31.228 DEBUG:boto:Port: 8000 2023-05-31T20:46:31.229 DEBUG:boto:Params: {} 2023-05-31T20:46:31.229 DEBUG:boto:Token: None 2023-05-31T20:46:31.230 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:46:31 GMT /pklxkd-59/obj17?versionId=null 2023-05-31T20:46:31.230 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:/55SmAPyCbZBYh68rrYrK1wcTS0= 2023-05-31T20:46:31.230 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:46:31 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:/55SmAPyCbZBYh68rrYrK1wcTS0=', 'Content-Length': '0'} 2023-05-31T20:46:31.232 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:43:07 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002335ed66c612ba8a-006477b227-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:46:31 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:46:31.276 DEBUG:boto:path=/pklxkd-59/obj17 2023-05-31T20:46:31.277 DEBUG:boto:auth_path=/pklxkd-59/obj17 2023-05-31T20:46:31.277 DEBUG:boto:path=/pklxkd-59/obj17?versionId=null 2023-05-31T20:46:31.277 DEBUG:boto:auth_path=/pklxkd-59/obj17?versionId=null 2023-05-31T20:46:31.278 DEBUG:boto:Method: HEAD 2023-05-31T20:46:31.278 DEBUG:boto:Path: /pklxkd-59/obj17?versionId=null 2023-05-31T20:46:31.278 DEBUG:boto:Data: 2023-05-31T20:46:31.279 DEBUG:boto:Headers: {} 2023-05-31T20:46:31.279 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:46:31.279 DEBUG:boto:Port: 8001 2023-05-31T20:46:31.280 DEBUG:boto:Params: {} 2023-05-31T20:46:31.280 DEBUG:boto:Token: None 2023-05-31T20:46:31.280 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:46:31 GMT /pklxkd-59/obj17?versionId=null 2023-05-31T20:46:31.281 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:1zV22+yjQ/25XkFg8njrRTMagxw= 2023-05-31T20:46:31.281 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:46:31 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:1zV22+yjQ/25XkFg8njrRTMagxw=', 'Content-Length': '0'} 2023-05-31T20:46:31.283 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:43:07 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b2d789f87deacff7-006477b227-1c17-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:46:31 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:46:31.283 DEBUG:boto:path=/pklxkd-59/obj17 2023-05-31T20:46:31.284 DEBUG:boto:auth_path=/pklxkd-59/obj17 2023-05-31T20:46:31.284 DEBUG:boto:path=/pklxkd-59/obj17?versionId=null 2023-05-31T20:46:31.284 DEBUG:boto:auth_path=/pklxkd-59/obj17?versionId=null 2023-05-31T20:46:31.285 DEBUG:boto:Method: HEAD 2023-05-31T20:46:31.285 DEBUG:boto:Path: /pklxkd-59/obj17?versionId=null 2023-05-31T20:46:31.285 DEBUG:boto:Data: 2023-05-31T20:46:31.286 DEBUG:boto:Headers: {} 2023-05-31T20:46:31.286 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:46:31.286 DEBUG:boto:Port: 8000 2023-05-31T20:46:31.287 DEBUG:boto:Params: {} 2023-05-31T20:46:31.287 DEBUG:boto:Token: None 2023-05-31T20:46:31.287 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:46:31 GMT /pklxkd-59/obj17?versionId=null 2023-05-31T20:46:31.288 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:1zV22+yjQ/25XkFg8njrRTMagxw= 2023-05-31T20:46:31.288 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:46:31 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:1zV22+yjQ/25XkFg8njrRTMagxw=', 'Content-Length': '0'} 2023-05-31T20:46:31.290 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:43:07 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000cced8904c9ca4b7d-006477b227-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:46:31 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:46:31.291 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj17 2023-05-31T20:46:31.291 DEBUG:boto:path=/pklxkd-59/obj17 2023-05-31T20:46:31.292 DEBUG:boto:auth_path=/pklxkd-59/obj17 2023-05-31T20:46:31.292 DEBUG:boto:path=/pklxkd-59/obj17?versionId=null 2023-05-31T20:46:31.292 DEBUG:boto:auth_path=/pklxkd-59/obj17?versionId=null 2023-05-31T20:46:31.293 DEBUG:boto:Method: GET 2023-05-31T20:46:31.293 DEBUG:boto:Path: /pklxkd-59/obj17?versionId=null 2023-05-31T20:46:31.293 DEBUG:boto:Data: 2023-05-31T20:46:31.294 DEBUG:boto:Headers: {} 2023-05-31T20:46:31.294 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:46:31.294 DEBUG:boto:Port: 8001 2023-05-31T20:46:31.295 DEBUG:boto:Params: {} 2023-05-31T20:46:31.295 DEBUG:boto:Token: None 2023-05-31T20:46:31.296 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:46:31 GMT /pklxkd-59/obj17?versionId=null 2023-05-31T20:46:31.296 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:/55SmAPyCbZBYh68rrYrK1wcTS0= 2023-05-31T20:46:31.296 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:46:31 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:/55SmAPyCbZBYh68rrYrK1wcTS0=', 'Content-Length': '0'} 2023-05-31T20:46:31.298 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:43:07 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000098461bc1647a722b-006477b227-1c17-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:46:31 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:46:31.299 DEBUG:boto:path=/pklxkd-59/obj17 2023-05-31T20:46:31.299 DEBUG:boto:auth_path=/pklxkd-59/obj17 2023-05-31T20:46:31.299 DEBUG:boto:path=/pklxkd-59/obj17?versionId=null 2023-05-31T20:46:31.300 DEBUG:boto:auth_path=/pklxkd-59/obj17?versionId=null 2023-05-31T20:46:31.300 DEBUG:boto:Method: GET 2023-05-31T20:46:31.301 DEBUG:boto:Path: /pklxkd-59/obj17?versionId=null 2023-05-31T20:46:31.301 DEBUG:boto:Data: 2023-05-31T20:46:31.301 DEBUG:boto:Headers: {} 2023-05-31T20:46:31.302 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:46:31.302 DEBUG:boto:Port: 8000 2023-05-31T20:46:31.302 DEBUG:boto:Params: {} 2023-05-31T20:46:31.303 DEBUG:boto:Token: None 2023-05-31T20:46:31.303 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:46:31 GMT /pklxkd-59/obj17?versionId=null 2023-05-31T20:46:31.304 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:/55SmAPyCbZBYh68rrYrK1wcTS0= 2023-05-31T20:46:31.304 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:46:31 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:/55SmAPyCbZBYh68rrYrK1wcTS0=', 'Content-Length': '0'} 2023-05-31T20:46:31.306 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:43:07 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000238e8175b371343b-006477b227-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:46:31 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:46:31.306 DEBUG:boto:path=/pklxkd-59/ 2023-05-31T20:46:31.307 DEBUG:boto:auth_path=/pklxkd-59/ 2023-05-31T20:46:31.307 DEBUG:boto:path=/pklxkd-59/?prefix=obj17 2023-05-31T20:46:31.308 DEBUG:boto:auth_path=/pklxkd-59/?prefix=obj17 2023-05-31T20:46:31.308 DEBUG:boto:Method: GET 2023-05-31T20:46:31.308 DEBUG:boto:Path: /pklxkd-59/?prefix=obj17 2023-05-31T20:46:31.309 DEBUG:boto:Data: 2023-05-31T20:46:31.309 DEBUG:boto:Headers: {} 2023-05-31T20:46:31.309 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:46:31.310 DEBUG:boto:Port: 8001 2023-05-31T20:46:31.310 DEBUG:boto:Params: {} 2023-05-31T20:46:31.310 DEBUG:boto:Token: None 2023-05-31T20:46:31.311 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:46:31 GMT /pklxkd-59/ 2023-05-31T20:46:31.311 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:gTuLAyT7WsrIPapdTlEudfl02iI= 2023-05-31T20:46:31.312 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:46:31 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:gTuLAyT7WsrIPapdTlEudfl02iI=', 'Content-Length': '0'} 2023-05-31T20:46:31.314 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000008bd7e93ee8c70432-006477b227-1c17-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:46:31 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:46:31.314 DEBUG:boto:b'pklxkd-59obj171000falseobj172023-05-31T20:43:07.977Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:46:31.315 DEBUG:boto:path=/pklxkd-59/ 2023-05-31T20:46:31.315 DEBUG:boto:auth_path=/pklxkd-59/ 2023-05-31T20:46:31.316 DEBUG:boto:path=/pklxkd-59/?prefix=obj17 2023-05-31T20:46:31.316 DEBUG:boto:auth_path=/pklxkd-59/?prefix=obj17 2023-05-31T20:46:31.316 DEBUG:boto:Method: GET 2023-05-31T20:46:31.317 DEBUG:boto:Path: /pklxkd-59/?prefix=obj17 2023-05-31T20:46:31.317 DEBUG:boto:Data: 2023-05-31T20:46:31.317 DEBUG:boto:Headers: {} 2023-05-31T20:46:31.318 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:46:31.318 DEBUG:boto:Port: 8000 2023-05-31T20:46:31.319 DEBUG:boto:Params: {} 2023-05-31T20:46:31.319 DEBUG:boto:Token: None 2023-05-31T20:46:31.319 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:46:31 GMT /pklxkd-59/ 2023-05-31T20:46:31.320 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:gTuLAyT7WsrIPapdTlEudfl02iI= 2023-05-31T20:46:31.320 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:46:31 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:gTuLAyT7WsrIPapdTlEudfl02iI=', 'Content-Length': '0'} 2023-05-31T20:46:31.323 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000007447d3ea5e378ab-006477b227-1630-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:46:31 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:46:31.324 DEBUG:boto:b'pklxkd-59obj171000falseobj172023-05-31T20:43:07.977Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:46:31.324 DEBUG:boto:path=/pklxkd-59/obj17 2023-05-31T20:46:31.325 DEBUG:boto:auth_path=/pklxkd-59/obj17 2023-05-31T20:46:31.325 DEBUG:boto:Method: HEAD 2023-05-31T20:46:31.325 DEBUG:boto:Path: /pklxkd-59/obj17 2023-05-31T20:46:31.326 DEBUG:boto:Data: 2023-05-31T20:46:31.326 DEBUG:boto:Headers: {} 2023-05-31T20:46:31.326 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:46:31.327 DEBUG:boto:Port: 8001 2023-05-31T20:46:31.327 DEBUG:boto:Params: {} 2023-05-31T20:46:31.327 DEBUG:boto:Token: None 2023-05-31T20:46:31.328 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:46:31 GMT /pklxkd-59/obj17 2023-05-31T20:46:31.328 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:zaJ6rEeTOVu8JjWiRmLOJDm+FAg= 2023-05-31T20:46:31.328 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:46:31 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:zaJ6rEeTOVu8JjWiRmLOJDm+FAg=', 'Content-Length': '0'} 2023-05-31T20:46:31.330 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:43:07 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000bc27a1fc2b5bd6ad-006477b227-1c17-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:46:31 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:46:31.331 DEBUG:boto:path=/pklxkd-59/obj17 2023-05-31T20:46:31.331 DEBUG:boto:auth_path=/pklxkd-59/obj17 2023-05-31T20:46:31.331 DEBUG:boto:Method: HEAD 2023-05-31T20:46:31.332 DEBUG:boto:Path: /pklxkd-59/obj17 2023-05-31T20:46:31.332 DEBUG:boto:Data: 2023-05-31T20:46:31.332 DEBUG:boto:Headers: {} 2023-05-31T20:46:31.333 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:46:31.333 DEBUG:boto:Port: 8000 2023-05-31T20:46:31.333 DEBUG:boto:Params: {} 2023-05-31T20:46:31.334 DEBUG:boto:Token: None 2023-05-31T20:46:31.334 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:46:31 GMT /pklxkd-59/obj17 2023-05-31T20:46:31.334 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:zaJ6rEeTOVu8JjWiRmLOJDm+FAg= 2023-05-31T20:46:31.335 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:46:31 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:zaJ6rEeTOVu8JjWiRmLOJDm+FAg=', 'Content-Length': '0'} 2023-05-31T20:46:31.337 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:43:07 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000254277f6489a389f-006477b227-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:46:31 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:46:31.337 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj17 2023-05-31T20:46:31.337 DEBUG:boto:path=/pklxkd-59/obj17 2023-05-31T20:46:31.338 DEBUG:boto:auth_path=/pklxkd-59/obj17 2023-05-31T20:46:31.338 DEBUG:boto:Method: GET 2023-05-31T20:46:31.339 DEBUG:boto:Path: /pklxkd-59/obj17 2023-05-31T20:46:31.339 DEBUG:boto:Data: 2023-05-31T20:46:31.339 DEBUG:boto:Headers: {} 2023-05-31T20:46:31.340 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:46:31.340 DEBUG:boto:Port: 8001 2023-05-31T20:46:31.340 DEBUG:boto:Params: {} 2023-05-31T20:46:31.341 DEBUG:boto:Token: None 2023-05-31T20:46:31.341 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:46:31 GMT /pklxkd-59/obj17 2023-05-31T20:46:31.342 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:sFd4NmEZNbr4QkomKk3t6/q8Zo8= 2023-05-31T20:46:31.342 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:46:31 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:sFd4NmEZNbr4QkomKk3t6/q8Zo8=', 'Content-Length': '0'} 2023-05-31T20:46:31.344 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:43:07 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ee5eedcd7b24ac8b-006477b227-1c17-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:46:31 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:46:31.344 DEBUG:boto:path=/pklxkd-59/obj17 2023-05-31T20:46:31.345 DEBUG:boto:auth_path=/pklxkd-59/obj17 2023-05-31T20:46:31.345 DEBUG:boto:Method: GET 2023-05-31T20:46:31.345 DEBUG:boto:Path: /pklxkd-59/obj17 2023-05-31T20:46:31.346 DEBUG:boto:Data: 2023-05-31T20:46:31.346 DEBUG:boto:Headers: {} 2023-05-31T20:46:31.346 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:46:31.347 DEBUG:boto:Port: 8000 2023-05-31T20:46:31.347 DEBUG:boto:Params: {} 2023-05-31T20:46:31.348 DEBUG:boto:Token: None 2023-05-31T20:46:31.348 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:46:31 GMT /pklxkd-59/obj17 2023-05-31T20:46:31.348 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:sFd4NmEZNbr4QkomKk3t6/q8Zo8= 2023-05-31T20:46:31.349 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:46:31 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:sFd4NmEZNbr4QkomKk3t6/q8Zo8=', 'Content-Length': '0'} 2023-05-31T20:46:31.350 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:43:07 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000ef8d06526903d33-006477b227-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:46:31 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:46:31.351 DEBUG:boto:path=/pklxkd-59/ 2023-05-31T20:46:31.352 DEBUG:boto:auth_path=/pklxkd-59/ 2023-05-31T20:46:31.352 DEBUG:boto:path=/pklxkd-59/?prefix=obj17 2023-05-31T20:46:31.352 DEBUG:boto:auth_path=/pklxkd-59/?prefix=obj17 2023-05-31T20:46:31.353 DEBUG:boto:Method: GET 2023-05-31T20:46:31.353 DEBUG:boto:Path: /pklxkd-59/?prefix=obj17 2023-05-31T20:46:31.353 DEBUG:boto:Data: 2023-05-31T20:46:31.354 DEBUG:boto:Headers: {} 2023-05-31T20:46:31.354 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:46:31.354 DEBUG:boto:Port: 8001 2023-05-31T20:46:31.355 DEBUG:boto:Params: {} 2023-05-31T20:46:31.355 DEBUG:boto:Token: None 2023-05-31T20:46:31.356 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:46:31 GMT /pklxkd-59/ 2023-05-31T20:46:31.356 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:gTuLAyT7WsrIPapdTlEudfl02iI= 2023-05-31T20:46:31.356 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:46:31 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:gTuLAyT7WsrIPapdTlEudfl02iI=', 'Content-Length': '0'} 2023-05-31T20:46:31.358 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b2be8eedae3b9daf-006477b227-1c17-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:46:31 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:46:31.359 DEBUG:boto:b'pklxkd-59obj171000falseobj172023-05-31T20:43:07.977Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:46:31.360 DEBUG:boto:path=/pklxkd-59/ 2023-05-31T20:46:31.360 DEBUG:boto:auth_path=/pklxkd-59/ 2023-05-31T20:46:31.360 DEBUG:boto:path=/pklxkd-59/?prefix=obj17 2023-05-31T20:46:31.361 DEBUG:boto:auth_path=/pklxkd-59/?prefix=obj17 2023-05-31T20:46:31.361 DEBUG:boto:Method: GET 2023-05-31T20:46:31.361 DEBUG:boto:Path: /pklxkd-59/?prefix=obj17 2023-05-31T20:46:31.362 DEBUG:boto:Data: 2023-05-31T20:46:31.362 DEBUG:boto:Headers: {} 2023-05-31T20:46:31.363 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:46:31.363 DEBUG:boto:Port: 8000 2023-05-31T20:46:31.363 DEBUG:boto:Params: {} 2023-05-31T20:46:31.364 DEBUG:boto:Token: None 2023-05-31T20:46:31.364 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:46:31 GMT /pklxkd-59/ 2023-05-31T20:46:31.364 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:gTuLAyT7WsrIPapdTlEudfl02iI= 2023-05-31T20:46:31.365 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:46:31 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:gTuLAyT7WsrIPapdTlEudfl02iI=', 'Content-Length': '0'} 2023-05-31T20:46:31.368 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000006ec14cc16e28ee48-006477b227-1630-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:46:31 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:46:31.368 DEBUG:boto:b'pklxkd-59obj171000falseobj172023-05-31T20:43:07.977Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:46:31.369 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj19 2023-05-31T20:46:31.369 DEBUG:boto:path=/pklxkd-59/obj19 2023-05-31T20:46:31.370 DEBUG:boto:auth_path=/pklxkd-59/obj19 2023-05-31T20:46:31.370 DEBUG:boto:path=/pklxkd-59/obj19?versionId=null 2023-05-31T20:46:31.370 DEBUG:boto:auth_path=/pklxkd-59/obj19?versionId=null 2023-05-31T20:46:31.371 DEBUG:boto:Method: GET 2023-05-31T20:46:31.371 DEBUG:boto:Path: /pklxkd-59/obj19?versionId=null 2023-05-31T20:46:31.372 DEBUG:boto:Data: 2023-05-31T20:46:31.372 DEBUG:boto:Headers: {} 2023-05-31T20:46:31.372 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:46:31.373 DEBUG:boto:Port: 8001 2023-05-31T20:46:31.373 DEBUG:boto:Params: {} 2023-05-31T20:46:31.373 DEBUG:boto:Token: None 2023-05-31T20:46:31.374 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:46:31 GMT /pklxkd-59/obj19?versionId=null 2023-05-31T20:46:31.374 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:sXi3/m9Wpk3UKj9YfIhQZVjFj08= 2023-05-31T20:46:31.375 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:46:31 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:sXi3/m9Wpk3UKj9YfIhQZVjFj08=', 'Content-Length': '0'} 2023-05-31T20:46:31.377 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:44:50 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000bf517cb2005ff2e2-006477b227-1c17-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:46:31 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:46:31.377 DEBUG:boto:path=/pklxkd-59/obj19 2023-05-31T20:46:31.378 DEBUG:boto:auth_path=/pklxkd-59/obj19 2023-05-31T20:46:31.378 DEBUG:boto:path=/pklxkd-59/obj19?versionId=null 2023-05-31T20:46:31.378 DEBUG:boto:auth_path=/pklxkd-59/obj19?versionId=null 2023-05-31T20:46:31.379 DEBUG:boto:Method: GET 2023-05-31T20:46:31.379 DEBUG:boto:Path: /pklxkd-59/obj19?versionId=null 2023-05-31T20:46:31.379 DEBUG:boto:Data: 2023-05-31T20:46:31.380 DEBUG:boto:Headers: {} 2023-05-31T20:46:31.380 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:46:31.380 DEBUG:boto:Port: 8000 2023-05-31T20:46:31.381 DEBUG:boto:Params: {} 2023-05-31T20:46:31.381 DEBUG:boto:Token: None 2023-05-31T20:46:31.382 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:46:31 GMT /pklxkd-59/obj19?versionId=null 2023-05-31T20:46:31.382 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:sXi3/m9Wpk3UKj9YfIhQZVjFj08= 2023-05-31T20:46:31.383 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:46:31 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:sXi3/m9Wpk3UKj9YfIhQZVjFj08=', 'Content-Length': '0'} 2023-05-31T20:46:31.384 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:44:50 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004d54925121531454-006477b227-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:46:31 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:46:31.428 DEBUG:boto:path=/pklxkd-59/obj19 2023-05-31T20:46:31.429 DEBUG:boto:auth_path=/pklxkd-59/obj19 2023-05-31T20:46:31.429 DEBUG:boto:path=/pklxkd-59/obj19?versionId=null 2023-05-31T20:46:31.430 DEBUG:boto:auth_path=/pklxkd-59/obj19?versionId=null 2023-05-31T20:46:31.430 DEBUG:boto:Method: HEAD 2023-05-31T20:46:31.430 DEBUG:boto:Path: /pklxkd-59/obj19?versionId=null 2023-05-31T20:46:31.431 DEBUG:boto:Data: 2023-05-31T20:46:31.431 DEBUG:boto:Headers: {} 2023-05-31T20:46:31.431 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:46:31.432 DEBUG:boto:Port: 8001 2023-05-31T20:46:31.432 DEBUG:boto:Params: {} 2023-05-31T20:46:31.432 DEBUG:boto:Token: None 2023-05-31T20:46:31.433 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:46:31 GMT /pklxkd-59/obj19?versionId=null 2023-05-31T20:46:31.433 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:jULdpbbr5CumogxnCxYeko+MpHE= 2023-05-31T20:46:31.434 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:46:31 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:jULdpbbr5CumogxnCxYeko+MpHE=', 'Content-Length': '0'} 2023-05-31T20:46:31.435 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:44:50 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004f8148df23aa859b-006477b227-1c17-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:46:31 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:46:31.436 DEBUG:boto:path=/pklxkd-59/obj19 2023-05-31T20:46:31.436 DEBUG:boto:auth_path=/pklxkd-59/obj19 2023-05-31T20:46:31.437 DEBUG:boto:path=/pklxkd-59/obj19?versionId=null 2023-05-31T20:46:31.437 DEBUG:boto:auth_path=/pklxkd-59/obj19?versionId=null 2023-05-31T20:46:31.437 DEBUG:boto:Method: HEAD 2023-05-31T20:46:31.438 DEBUG:boto:Path: /pklxkd-59/obj19?versionId=null 2023-05-31T20:46:31.438 DEBUG:boto:Data: 2023-05-31T20:46:31.438 DEBUG:boto:Headers: {} 2023-05-31T20:46:31.439 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:46:31.439 DEBUG:boto:Port: 8000 2023-05-31T20:46:31.439 DEBUG:boto:Params: {} 2023-05-31T20:46:31.440 DEBUG:boto:Token: None 2023-05-31T20:46:31.440 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:46:31 GMT /pklxkd-59/obj19?versionId=null 2023-05-31T20:46:31.441 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:jULdpbbr5CumogxnCxYeko+MpHE= 2023-05-31T20:46:31.441 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:46:31 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:jULdpbbr5CumogxnCxYeko+MpHE=', 'Content-Length': '0'} 2023-05-31T20:46:31.443 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:44:50 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006bbaa5dea6d6848f-006477b227-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:46:31 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:46:31.443 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj19 2023-05-31T20:46:31.444 DEBUG:boto:path=/pklxkd-59/obj19 2023-05-31T20:46:31.444 DEBUG:boto:auth_path=/pklxkd-59/obj19 2023-05-31T20:46:31.445 DEBUG:boto:path=/pklxkd-59/obj19?versionId=null 2023-05-31T20:46:31.445 DEBUG:boto:auth_path=/pklxkd-59/obj19?versionId=null 2023-05-31T20:46:31.445 DEBUG:boto:Method: GET 2023-05-31T20:46:31.446 DEBUG:boto:Path: /pklxkd-59/obj19?versionId=null 2023-05-31T20:46:31.446 DEBUG:boto:Data: 2023-05-31T20:46:31.446 DEBUG:boto:Headers: {} 2023-05-31T20:46:31.447 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:46:31.447 DEBUG:boto:Port: 8001 2023-05-31T20:46:31.447 DEBUG:boto:Params: {} 2023-05-31T20:46:31.448 DEBUG:boto:Token: None 2023-05-31T20:46:31.448 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:46:31 GMT /pklxkd-59/obj19?versionId=null 2023-05-31T20:46:31.449 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:sXi3/m9Wpk3UKj9YfIhQZVjFj08= 2023-05-31T20:46:31.449 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:46:31 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:sXi3/m9Wpk3UKj9YfIhQZVjFj08=', 'Content-Length': '0'} 2023-05-31T20:46:31.451 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:44:50 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f09a1d65699b38c5-006477b227-1c17-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:46:31 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:46:31.452 DEBUG:boto:path=/pklxkd-59/obj19 2023-05-31T20:46:31.452 DEBUG:boto:auth_path=/pklxkd-59/obj19 2023-05-31T20:46:31.453 DEBUG:boto:path=/pklxkd-59/obj19?versionId=null 2023-05-31T20:46:31.453 DEBUG:boto:auth_path=/pklxkd-59/obj19?versionId=null 2023-05-31T20:46:31.454 DEBUG:boto:Method: GET 2023-05-31T20:46:31.454 DEBUG:boto:Path: /pklxkd-59/obj19?versionId=null 2023-05-31T20:46:31.454 DEBUG:boto:Data: 2023-05-31T20:46:31.455 DEBUG:boto:Headers: {} 2023-05-31T20:46:31.455 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:46:31.455 DEBUG:boto:Port: 8000 2023-05-31T20:46:31.456 DEBUG:boto:Params: {} 2023-05-31T20:46:31.456 DEBUG:boto:Token: None 2023-05-31T20:46:31.457 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:46:31 GMT /pklxkd-59/obj19?versionId=null 2023-05-31T20:46:31.457 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:sXi3/m9Wpk3UKj9YfIhQZVjFj08= 2023-05-31T20:46:31.458 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:46:31 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:sXi3/m9Wpk3UKj9YfIhQZVjFj08=', 'Content-Length': '0'} 2023-05-31T20:46:31.460 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:44:50 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006d660d83d86c2c9d-006477b227-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:46:31 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:46:31.460 DEBUG:boto:path=/pklxkd-59/ 2023-05-31T20:46:31.461 DEBUG:boto:auth_path=/pklxkd-59/ 2023-05-31T20:46:31.461 DEBUG:boto:path=/pklxkd-59/?prefix=obj19 2023-05-31T20:46:31.461 DEBUG:boto:auth_path=/pklxkd-59/?prefix=obj19 2023-05-31T20:46:31.462 DEBUG:boto:Method: GET 2023-05-31T20:46:31.462 DEBUG:boto:Path: /pklxkd-59/?prefix=obj19 2023-05-31T20:46:31.463 DEBUG:boto:Data: 2023-05-31T20:46:31.463 DEBUG:boto:Headers: {} 2023-05-31T20:46:31.464 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:46:31.464 DEBUG:boto:Port: 8001 2023-05-31T20:46:31.464 DEBUG:boto:Params: {} 2023-05-31T20:46:31.465 DEBUG:boto:Token: None 2023-05-31T20:46:31.465 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:46:31 GMT /pklxkd-59/ 2023-05-31T20:46:31.465 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:gTuLAyT7WsrIPapdTlEudfl02iI= 2023-05-31T20:46:31.466 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:46:31 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:gTuLAyT7WsrIPapdTlEudfl02iI=', 'Content-Length': '0'} 2023-05-31T20:46:31.468 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000fcb7e40ca6de681e-006477b227-1c17-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:46:31 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:46:31.469 DEBUG:boto:b'pklxkd-59obj191000falseobj192023-05-31T20:44:50.557Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:46:31.470 DEBUG:boto:path=/pklxkd-59/ 2023-05-31T20:46:31.470 DEBUG:boto:auth_path=/pklxkd-59/ 2023-05-31T20:46:31.470 DEBUG:boto:path=/pklxkd-59/?prefix=obj19 2023-05-31T20:46:31.471 DEBUG:boto:auth_path=/pklxkd-59/?prefix=obj19 2023-05-31T20:46:31.471 DEBUG:boto:Method: GET 2023-05-31T20:46:31.472 DEBUG:boto:Path: /pklxkd-59/?prefix=obj19 2023-05-31T20:46:31.472 DEBUG:boto:Data: 2023-05-31T20:46:31.473 DEBUG:boto:Headers: {} 2023-05-31T20:46:31.473 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:46:31.473 DEBUG:boto:Port: 8000 2023-05-31T20:46:31.474 DEBUG:boto:Params: {} 2023-05-31T20:46:31.474 DEBUG:boto:Token: None 2023-05-31T20:46:31.474 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:46:31 GMT /pklxkd-59/ 2023-05-31T20:46:31.475 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:gTuLAyT7WsrIPapdTlEudfl02iI= 2023-05-31T20:46:31.475 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:46:31 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:gTuLAyT7WsrIPapdTlEudfl02iI=', 'Content-Length': '0'} 2023-05-31T20:46:31.478 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000007bba1ef401a65aa-006477b227-1630-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:46:31 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:46:31.479 DEBUG:boto:b'pklxkd-59obj191000falseobj192023-05-31T20:44:50.557Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:46:31.479 DEBUG:boto:path=/pklxkd-59/obj19 2023-05-31T20:46:31.480 DEBUG:boto:auth_path=/pklxkd-59/obj19 2023-05-31T20:46:31.480 DEBUG:boto:Method: HEAD 2023-05-31T20:46:31.480 DEBUG:boto:Path: /pklxkd-59/obj19 2023-05-31T20:46:31.481 DEBUG:boto:Data: 2023-05-31T20:46:31.481 DEBUG:boto:Headers: {} 2023-05-31T20:46:31.481 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:46:31.482 DEBUG:boto:Port: 8001 2023-05-31T20:46:31.482 DEBUG:boto:Params: {} 2023-05-31T20:46:31.483 DEBUG:boto:Token: None 2023-05-31T20:46:31.483 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:46:31 GMT /pklxkd-59/obj19 2023-05-31T20:46:31.483 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:QN84XRenhSNfz07f/WhHF6YK7Es= 2023-05-31T20:46:31.484 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:46:31 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:QN84XRenhSNfz07f/WhHF6YK7Es=', 'Content-Length': '0'} 2023-05-31T20:46:31.485 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:44:50 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005e59ad7029ae1b65-006477b227-1c17-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:46:31 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:46:31.486 DEBUG:boto:path=/pklxkd-59/obj19 2023-05-31T20:46:31.486 DEBUG:boto:auth_path=/pklxkd-59/obj19 2023-05-31T20:46:31.487 DEBUG:boto:Method: HEAD 2023-05-31T20:46:31.487 DEBUG:boto:Path: /pklxkd-59/obj19 2023-05-31T20:46:31.487 DEBUG:boto:Data: 2023-05-31T20:46:31.488 DEBUG:boto:Headers: {} 2023-05-31T20:46:31.488 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:46:31.489 DEBUG:boto:Port: 8000 2023-05-31T20:46:31.489 DEBUG:boto:Params: {} 2023-05-31T20:46:31.489 DEBUG:boto:Token: None 2023-05-31T20:46:31.490 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:46:31 GMT /pklxkd-59/obj19 2023-05-31T20:46:31.490 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:QN84XRenhSNfz07f/WhHF6YK7Es= 2023-05-31T20:46:31.491 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:46:31 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:QN84XRenhSNfz07f/WhHF6YK7Es=', 'Content-Length': '0'} 2023-05-31T20:46:31.492 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:44:50 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004c046ee93e0a9eeb-006477b227-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:46:31 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:46:31.493 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj19 2023-05-31T20:46:31.493 DEBUG:boto:path=/pklxkd-59/obj19 2023-05-31T20:46:31.494 DEBUG:boto:auth_path=/pklxkd-59/obj19 2023-05-31T20:46:31.494 DEBUG:boto:Method: GET 2023-05-31T20:46:31.495 DEBUG:boto:Path: /pklxkd-59/obj19 2023-05-31T20:46:31.495 DEBUG:boto:Data: 2023-05-31T20:46:31.495 DEBUG:boto:Headers: {} 2023-05-31T20:46:31.496 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:46:31.496 DEBUG:boto:Port: 8001 2023-05-31T20:46:31.496 DEBUG:boto:Params: {} 2023-05-31T20:46:31.497 DEBUG:boto:Token: None 2023-05-31T20:46:31.497 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:46:31 GMT /pklxkd-59/obj19 2023-05-31T20:46:31.498 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:m4aCgMBsvBH1CG7B30L3tCTMtoc= 2023-05-31T20:46:31.498 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:46:31 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:m4aCgMBsvBH1CG7B30L3tCTMtoc=', 'Content-Length': '0'} 2023-05-31T20:46:31.500 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:44:50 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000064720bdb650a2d93-006477b227-1c17-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:46:31 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:46:31.501 DEBUG:boto:path=/pklxkd-59/obj19 2023-05-31T20:46:31.502 DEBUG:boto:auth_path=/pklxkd-59/obj19 2023-05-31T20:46:31.502 DEBUG:boto:Method: GET 2023-05-31T20:46:31.502 DEBUG:boto:Path: /pklxkd-59/obj19 2023-05-31T20:46:31.503 DEBUG:boto:Data: 2023-05-31T20:46:31.503 DEBUG:boto:Headers: {} 2023-05-31T20:46:31.503 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:46:31.504 DEBUG:boto:Port: 8000 2023-05-31T20:46:31.505 DEBUG:boto:Params: {} 2023-05-31T20:46:31.505 DEBUG:boto:Token: None 2023-05-31T20:46:31.506 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:46:31 GMT /pklxkd-59/obj19 2023-05-31T20:46:31.506 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:m4aCgMBsvBH1CG7B30L3tCTMtoc= 2023-05-31T20:46:31.506 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:46:31 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:m4aCgMBsvBH1CG7B30L3tCTMtoc=', 'Content-Length': '0'} 2023-05-31T20:46:31.509 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:44:50 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002ccf83082409a992-006477b227-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:46:31 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:46:31.552 DEBUG:boto:path=/pklxkd-59/ 2023-05-31T20:46:31.553 DEBUG:boto:auth_path=/pklxkd-59/ 2023-05-31T20:46:31.553 DEBUG:boto:path=/pklxkd-59/?prefix=obj19 2023-05-31T20:46:31.554 DEBUG:boto:auth_path=/pklxkd-59/?prefix=obj19 2023-05-31T20:46:31.554 DEBUG:boto:Method: GET 2023-05-31T20:46:31.554 DEBUG:boto:Path: /pklxkd-59/?prefix=obj19 2023-05-31T20:46:31.555 DEBUG:boto:Data: 2023-05-31T20:46:31.555 DEBUG:boto:Headers: {} 2023-05-31T20:46:31.555 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:46:31.556 DEBUG:boto:Port: 8001 2023-05-31T20:46:31.556 DEBUG:boto:Params: {} 2023-05-31T20:46:31.557 DEBUG:boto:Token: None 2023-05-31T20:46:31.557 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:46:31 GMT /pklxkd-59/ 2023-05-31T20:46:31.558 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:gTuLAyT7WsrIPapdTlEudfl02iI= 2023-05-31T20:46:31.558 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:46:31 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:gTuLAyT7WsrIPapdTlEudfl02iI=', 'Content-Length': '0'} 2023-05-31T20:46:31.561 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000748a351de9237474-006477b227-1c17-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:46:31 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:46:31.561 DEBUG:boto:b'pklxkd-59obj191000falseobj192023-05-31T20:44:50.557Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:46:31.562 DEBUG:boto:path=/pklxkd-59/ 2023-05-31T20:46:31.562 DEBUG:boto:auth_path=/pklxkd-59/ 2023-05-31T20:46:31.563 DEBUG:boto:path=/pklxkd-59/?prefix=obj19 2023-05-31T20:46:31.563 DEBUG:boto:auth_path=/pklxkd-59/?prefix=obj19 2023-05-31T20:46:31.563 DEBUG:boto:Method: GET 2023-05-31T20:46:31.564 DEBUG:boto:Path: /pklxkd-59/?prefix=obj19 2023-05-31T20:46:31.564 DEBUG:boto:Data: 2023-05-31T20:46:31.565 DEBUG:boto:Headers: {} 2023-05-31T20:46:31.565 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:46:31.565 DEBUG:boto:Port: 8000 2023-05-31T20:46:31.566 DEBUG:boto:Params: {} 2023-05-31T20:46:31.566 DEBUG:boto:Token: None 2023-05-31T20:46:31.567 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:46:31 GMT /pklxkd-59/ 2023-05-31T20:46:31.567 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:gTuLAyT7WsrIPapdTlEudfl02iI= 2023-05-31T20:46:31.567 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:46:31 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:gTuLAyT7WsrIPapdTlEudfl02iI=', 'Content-Length': '0'} 2023-05-31T20:46:31.570 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000029f02e4b071e70fe-006477b227-1630-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:46:31 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:46:31.571 DEBUG:boto:b'pklxkd-59obj191000falseobj192023-05-31T20:44:50.557Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:46:31.572 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj22 2023-05-31T20:46:31.572 DEBUG:boto:path=/pklxkd-59/obj22 2023-05-31T20:46:31.573 DEBUG:boto:auth_path=/pklxkd-59/obj22 2023-05-31T20:46:31.573 DEBUG:boto:path=/pklxkd-59/obj22?versionId=null 2023-05-31T20:46:31.573 DEBUG:boto:auth_path=/pklxkd-59/obj22?versionId=null 2023-05-31T20:46:31.574 DEBUG:boto:Method: GET 2023-05-31T20:46:31.574 DEBUG:boto:Path: /pklxkd-59/obj22?versionId=null 2023-05-31T20:46:31.574 DEBUG:boto:Data: 2023-05-31T20:46:31.575 DEBUG:boto:Headers: {} 2023-05-31T20:46:31.575 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:46:31.575 DEBUG:boto:Port: 8001 2023-05-31T20:46:31.576 DEBUG:boto:Params: {} 2023-05-31T20:46:31.576 DEBUG:boto:Token: None 2023-05-31T20:46:31.577 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:46:31 GMT /pklxkd-59/obj22?versionId=null 2023-05-31T20:46:31.577 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:nSM9cALescr8tF6Bef8xBSs5R0Q= 2023-05-31T20:46:31.577 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:46:31 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:nSM9cALescr8tF6Bef8xBSs5R0Q=', 'Content-Length': '0'} 2023-05-31T20:46:31.579 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:39:26 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000012381397f63cc646-006477b227-1c17-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:46:31 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:46:31.620 DEBUG:boto:path=/pklxkd-59/obj22 2023-05-31T20:46:31.621 DEBUG:boto:auth_path=/pklxkd-59/obj22 2023-05-31T20:46:31.621 DEBUG:boto:path=/pklxkd-59/obj22?versionId=null 2023-05-31T20:46:31.622 DEBUG:boto:auth_path=/pklxkd-59/obj22?versionId=null 2023-05-31T20:46:31.622 DEBUG:boto:Method: GET 2023-05-31T20:46:31.622 DEBUG:boto:Path: /pklxkd-59/obj22?versionId=null 2023-05-31T20:46:31.623 DEBUG:boto:Data: 2023-05-31T20:46:31.623 DEBUG:boto:Headers: {} 2023-05-31T20:46:31.623 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:46:31.624 DEBUG:boto:Port: 8000 2023-05-31T20:46:31.624 DEBUG:boto:Params: {} 2023-05-31T20:46:31.625 DEBUG:boto:Token: None 2023-05-31T20:46:31.625 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:46:31 GMT /pklxkd-59/obj22?versionId=null 2023-05-31T20:46:31.625 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:nSM9cALescr8tF6Bef8xBSs5R0Q= 2023-05-31T20:46:31.626 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:46:31 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:nSM9cALescr8tF6Bef8xBSs5R0Q=', 'Content-Length': '0'} 2023-05-31T20:46:31.628 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:39:26 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000020dfab5d7558c4e5-006477b227-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:46:31 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:46:31.628 DEBUG:boto:path=/pklxkd-59/obj22 2023-05-31T20:46:31.629 DEBUG:boto:auth_path=/pklxkd-59/obj22 2023-05-31T20:46:31.629 DEBUG:boto:path=/pklxkd-59/obj22?versionId=null 2023-05-31T20:46:31.630 DEBUG:boto:auth_path=/pklxkd-59/obj22?versionId=null 2023-05-31T20:46:31.630 DEBUG:boto:Method: HEAD 2023-05-31T20:46:31.630 DEBUG:boto:Path: /pklxkd-59/obj22?versionId=null 2023-05-31T20:46:31.631 DEBUG:boto:Data: 2023-05-31T20:46:31.631 DEBUG:boto:Headers: {} 2023-05-31T20:46:31.631 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:46:31.631 DEBUG:boto:Port: 8001 2023-05-31T20:46:31.632 DEBUG:boto:Params: {} 2023-05-31T20:46:31.632 DEBUG:boto:Token: None 2023-05-31T20:46:31.633 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:46:31 GMT /pklxkd-59/obj22?versionId=null 2023-05-31T20:46:31.633 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:AsMMTdqN+rfLfMa7pRSVu7vYRlU= 2023-05-31T20:46:31.633 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:46:31 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:AsMMTdqN+rfLfMa7pRSVu7vYRlU=', 'Content-Length': '0'} 2023-05-31T20:46:31.635 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:39:26 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000c9086729693c322-006477b227-1c17-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:46:31 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:46:31.636 DEBUG:boto:path=/pklxkd-59/obj22 2023-05-31T20:46:31.636 DEBUG:boto:auth_path=/pklxkd-59/obj22 2023-05-31T20:46:31.637 DEBUG:boto:path=/pklxkd-59/obj22?versionId=null 2023-05-31T20:46:31.637 DEBUG:boto:auth_path=/pklxkd-59/obj22?versionId=null 2023-05-31T20:46:31.637 DEBUG:boto:Method: HEAD 2023-05-31T20:46:31.638 DEBUG:boto:Path: /pklxkd-59/obj22?versionId=null 2023-05-31T20:46:31.638 DEBUG:boto:Data: 2023-05-31T20:46:31.638 DEBUG:boto:Headers: {} 2023-05-31T20:46:31.639 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:46:31.639 DEBUG:boto:Port: 8000 2023-05-31T20:46:31.639 DEBUG:boto:Params: {} 2023-05-31T20:46:31.640 DEBUG:boto:Token: None 2023-05-31T20:46:31.640 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:46:31 GMT /pklxkd-59/obj22?versionId=null 2023-05-31T20:46:31.641 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:AsMMTdqN+rfLfMa7pRSVu7vYRlU= 2023-05-31T20:46:31.641 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:46:31 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:AsMMTdqN+rfLfMa7pRSVu7vYRlU=', 'Content-Length': '0'} 2023-05-31T20:46:31.643 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:39:26 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008753f8439652a45d-006477b227-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:46:31 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:46:31.643 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj22 2023-05-31T20:46:31.644 DEBUG:boto:path=/pklxkd-59/obj22 2023-05-31T20:46:31.644 DEBUG:boto:auth_path=/pklxkd-59/obj22 2023-05-31T20:46:31.644 DEBUG:boto:path=/pklxkd-59/obj22?versionId=null 2023-05-31T20:46:31.645 DEBUG:boto:auth_path=/pklxkd-59/obj22?versionId=null 2023-05-31T20:46:31.645 DEBUG:boto:Method: GET 2023-05-31T20:46:31.645 DEBUG:boto:Path: /pklxkd-59/obj22?versionId=null 2023-05-31T20:46:31.646 DEBUG:boto:Data: 2023-05-31T20:46:31.646 DEBUG:boto:Headers: {} 2023-05-31T20:46:31.647 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:46:31.647 DEBUG:boto:Port: 8001 2023-05-31T20:46:31.647 DEBUG:boto:Params: {} 2023-05-31T20:46:31.648 DEBUG:boto:Token: None 2023-05-31T20:46:31.648 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:46:31 GMT /pklxkd-59/obj22?versionId=null 2023-05-31T20:46:31.648 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:nSM9cALescr8tF6Bef8xBSs5R0Q= 2023-05-31T20:46:31.649 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:46:31 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:nSM9cALescr8tF6Bef8xBSs5R0Q=', 'Content-Length': '0'} 2023-05-31T20:46:31.651 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:39:26 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000010b41f4480116a15-006477b227-1c17-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:46:31 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:46:31.651 DEBUG:boto:path=/pklxkd-59/obj22 2023-05-31T20:46:31.652 DEBUG:boto:auth_path=/pklxkd-59/obj22 2023-05-31T20:46:31.652 DEBUG:boto:path=/pklxkd-59/obj22?versionId=null 2023-05-31T20:46:31.652 DEBUG:boto:auth_path=/pklxkd-59/obj22?versionId=null 2023-05-31T20:46:31.653 DEBUG:boto:Method: GET 2023-05-31T20:46:31.653 DEBUG:boto:Path: /pklxkd-59/obj22?versionId=null 2023-05-31T20:46:31.653 DEBUG:boto:Data: 2023-05-31T20:46:31.654 DEBUG:boto:Headers: {} 2023-05-31T20:46:31.654 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:46:31.654 DEBUG:boto:Port: 8000 2023-05-31T20:46:31.655 DEBUG:boto:Params: {} 2023-05-31T20:46:31.655 DEBUG:boto:Token: None 2023-05-31T20:46:31.656 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:46:31 GMT /pklxkd-59/obj22?versionId=null 2023-05-31T20:46:31.656 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:nSM9cALescr8tF6Bef8xBSs5R0Q= 2023-05-31T20:46:31.656 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:46:31 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:nSM9cALescr8tF6Bef8xBSs5R0Q=', 'Content-Length': '0'} 2023-05-31T20:46:31.658 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:39:26 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000eef37a7b2a121fca-006477b227-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:46:31 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:46:31.659 DEBUG:boto:path=/pklxkd-59/ 2023-05-31T20:46:31.659 DEBUG:boto:auth_path=/pklxkd-59/ 2023-05-31T20:46:31.660 DEBUG:boto:path=/pklxkd-59/?prefix=obj22 2023-05-31T20:46:31.660 DEBUG:boto:auth_path=/pklxkd-59/?prefix=obj22 2023-05-31T20:46:31.660 DEBUG:boto:Method: GET 2023-05-31T20:46:31.661 DEBUG:boto:Path: /pklxkd-59/?prefix=obj22 2023-05-31T20:46:31.661 DEBUG:boto:Data: 2023-05-31T20:46:31.662 DEBUG:boto:Headers: {} 2023-05-31T20:46:31.662 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:46:31.662 DEBUG:boto:Port: 8001 2023-05-31T20:46:31.663 DEBUG:boto:Params: {} 2023-05-31T20:46:31.663 DEBUG:boto:Token: None 2023-05-31T20:46:31.663 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:46:31 GMT /pklxkd-59/ 2023-05-31T20:46:31.664 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:gTuLAyT7WsrIPapdTlEudfl02iI= 2023-05-31T20:46:31.664 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:46:31 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:gTuLAyT7WsrIPapdTlEudfl02iI=', 'Content-Length': '0'} 2023-05-31T20:46:31.666 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ad852a1acd0cbbfb-006477b227-1c17-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:46:31 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:46:31.667 DEBUG:boto:b'pklxkd-59obj221000falseobj222023-05-31T20:39:26.473Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:46:31.668 DEBUG:boto:path=/pklxkd-59/ 2023-05-31T20:46:31.668 DEBUG:boto:auth_path=/pklxkd-59/ 2023-05-31T20:46:31.668 DEBUG:boto:path=/pklxkd-59/?prefix=obj22 2023-05-31T20:46:31.669 DEBUG:boto:auth_path=/pklxkd-59/?prefix=obj22 2023-05-31T20:46:31.669 DEBUG:boto:Method: GET 2023-05-31T20:46:31.669 DEBUG:boto:Path: /pklxkd-59/?prefix=obj22 2023-05-31T20:46:31.670 DEBUG:boto:Data: 2023-05-31T20:46:31.670 DEBUG:boto:Headers: {} 2023-05-31T20:46:31.670 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:46:31.671 DEBUG:boto:Port: 8000 2023-05-31T20:46:31.671 DEBUG:boto:Params: {} 2023-05-31T20:46:31.672 DEBUG:boto:Token: None 2023-05-31T20:46:31.672 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:46:31 GMT /pklxkd-59/ 2023-05-31T20:46:31.672 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:gTuLAyT7WsrIPapdTlEudfl02iI= 2023-05-31T20:46:31.673 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:46:31 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:gTuLAyT7WsrIPapdTlEudfl02iI=', 'Content-Length': '0'} 2023-05-31T20:46:31.676 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d6be09f052085dc7-006477b227-1630-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:46:31 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:46:31.676 DEBUG:boto:b'pklxkd-59obj221000falseobj222023-05-31T20:39:26.473Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:46:31.677 DEBUG:boto:path=/pklxkd-59/obj22 2023-05-31T20:46:31.677 DEBUG:boto:auth_path=/pklxkd-59/obj22 2023-05-31T20:46:31.678 DEBUG:boto:Method: HEAD 2023-05-31T20:46:31.678 DEBUG:boto:Path: /pklxkd-59/obj22 2023-05-31T20:46:31.678 DEBUG:boto:Data: 2023-05-31T20:46:31.679 DEBUG:boto:Headers: {} 2023-05-31T20:46:31.679 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:46:31.679 DEBUG:boto:Port: 8001 2023-05-31T20:46:31.680 DEBUG:boto:Params: {} 2023-05-31T20:46:31.680 DEBUG:boto:Token: None 2023-05-31T20:46:31.680 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:46:31 GMT /pklxkd-59/obj22 2023-05-31T20:46:31.681 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:fxClb9FabaJf1Aipz8UgcEgiEBY= 2023-05-31T20:46:31.681 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:46:31 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:fxClb9FabaJf1Aipz8UgcEgiEBY=', 'Content-Length': '0'} 2023-05-31T20:46:31.683 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:39:26 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000689e8c71dfb4a282-006477b227-1c17-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:46:31 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:46:31.683 DEBUG:boto:path=/pklxkd-59/obj22 2023-05-31T20:46:31.684 DEBUG:boto:auth_path=/pklxkd-59/obj22 2023-05-31T20:46:31.684 DEBUG:boto:Method: HEAD 2023-05-31T20:46:31.684 DEBUG:boto:Path: /pklxkd-59/obj22 2023-05-31T20:46:31.685 DEBUG:boto:Data: 2023-05-31T20:46:31.685 DEBUG:boto:Headers: {} 2023-05-31T20:46:31.685 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:46:31.686 DEBUG:boto:Port: 8000 2023-05-31T20:46:31.686 DEBUG:boto:Params: {} 2023-05-31T20:46:31.686 DEBUG:boto:Token: None 2023-05-31T20:46:31.687 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:46:31 GMT /pklxkd-59/obj22 2023-05-31T20:46:31.687 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:fxClb9FabaJf1Aipz8UgcEgiEBY= 2023-05-31T20:46:31.688 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:46:31 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:fxClb9FabaJf1Aipz8UgcEgiEBY=', 'Content-Length': '0'} 2023-05-31T20:46:31.689 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:39:26 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009c3247ace138aac4-006477b227-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:46:31 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:46:31.690 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj22 2023-05-31T20:46:31.690 DEBUG:boto:path=/pklxkd-59/obj22 2023-05-31T20:46:31.690 DEBUG:boto:auth_path=/pklxkd-59/obj22 2023-05-31T20:46:31.691 DEBUG:boto:Method: GET 2023-05-31T20:46:31.691 DEBUG:boto:Path: /pklxkd-59/obj22 2023-05-31T20:46:31.692 DEBUG:boto:Data: 2023-05-31T20:46:31.692 DEBUG:boto:Headers: {} 2023-05-31T20:46:31.692 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:46:31.693 DEBUG:boto:Port: 8001 2023-05-31T20:46:31.693 DEBUG:boto:Params: {} 2023-05-31T20:46:31.693 DEBUG:boto:Token: None 2023-05-31T20:46:31.694 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:46:31 GMT /pklxkd-59/obj22 2023-05-31T20:46:31.694 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:+195992QX0SueivhRcjqAb5GY90= 2023-05-31T20:46:31.695 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:46:31 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:+195992QX0SueivhRcjqAb5GY90=', 'Content-Length': '0'} 2023-05-31T20:46:31.696 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:39:26 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000397a75fc47c70ebe-006477b227-1c17-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:46:31 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:46:31.697 DEBUG:boto:path=/pklxkd-59/obj22 2023-05-31T20:46:31.697 DEBUG:boto:auth_path=/pklxkd-59/obj22 2023-05-31T20:46:31.698 DEBUG:boto:Method: GET 2023-05-31T20:46:31.698 DEBUG:boto:Path: /pklxkd-59/obj22 2023-05-31T20:46:31.699 DEBUG:boto:Data: 2023-05-31T20:46:31.699 DEBUG:boto:Headers: {} 2023-05-31T20:46:31.699 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:46:31.700 DEBUG:boto:Port: 8000 2023-05-31T20:46:31.700 DEBUG:boto:Params: {} 2023-05-31T20:46:31.704 DEBUG:boto:Token: None 2023-05-31T20:46:31.705 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:46:31 GMT /pklxkd-59/obj22 2023-05-31T20:46:31.705 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:+195992QX0SueivhRcjqAb5GY90= 2023-05-31T20:46:31.706 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:46:31 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:+195992QX0SueivhRcjqAb5GY90=', 'Content-Length': '0'} 2023-05-31T20:46:31.708 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:39:26 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f6307a91f8bff2dd-006477b227-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:46:31 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:46:31.708 DEBUG:boto:path=/pklxkd-59/ 2023-05-31T20:46:31.709 DEBUG:boto:auth_path=/pklxkd-59/ 2023-05-31T20:46:31.709 DEBUG:boto:path=/pklxkd-59/?prefix=obj22 2023-05-31T20:46:31.710 DEBUG:boto:auth_path=/pklxkd-59/?prefix=obj22 2023-05-31T20:46:31.710 DEBUG:boto:Method: GET 2023-05-31T20:46:31.710 DEBUG:boto:Path: /pklxkd-59/?prefix=obj22 2023-05-31T20:46:31.711 DEBUG:boto:Data: 2023-05-31T20:46:31.711 DEBUG:boto:Headers: {} 2023-05-31T20:46:31.711 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:46:31.712 DEBUG:boto:Port: 8001 2023-05-31T20:46:31.712 DEBUG:boto:Params: {} 2023-05-31T20:46:31.713 DEBUG:boto:Token: None 2023-05-31T20:46:31.713 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:46:31 GMT /pklxkd-59/ 2023-05-31T20:46:31.713 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:gTuLAyT7WsrIPapdTlEudfl02iI= 2023-05-31T20:46:31.714 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:46:31 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:gTuLAyT7WsrIPapdTlEudfl02iI=', 'Content-Length': '0'} 2023-05-31T20:46:31.716 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000dc21891ccadeebab-006477b227-1c17-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:46:31 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:46:31.717 DEBUG:boto:b'pklxkd-59obj221000falseobj222023-05-31T20:39:26.473Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:46:31.717 DEBUG:boto:path=/pklxkd-59/ 2023-05-31T20:46:31.718 DEBUG:boto:auth_path=/pklxkd-59/ 2023-05-31T20:46:31.718 DEBUG:boto:path=/pklxkd-59/?prefix=obj22 2023-05-31T20:46:31.719 DEBUG:boto:auth_path=/pklxkd-59/?prefix=obj22 2023-05-31T20:46:31.719 DEBUG:boto:Method: GET 2023-05-31T20:46:31.719 DEBUG:boto:Path: /pklxkd-59/?prefix=obj22 2023-05-31T20:46:31.720 DEBUG:boto:Data: 2023-05-31T20:46:31.720 DEBUG:boto:Headers: {} 2023-05-31T20:46:31.720 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:46:31.721 DEBUG:boto:Port: 8000 2023-05-31T20:46:31.721 DEBUG:boto:Params: {} 2023-05-31T20:46:31.721 DEBUG:boto:Token: None 2023-05-31T20:46:31.722 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:46:31 GMT /pklxkd-59/ 2023-05-31T20:46:31.722 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:gTuLAyT7WsrIPapdTlEudfl02iI= 2023-05-31T20:46:31.723 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:46:31 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:gTuLAyT7WsrIPapdTlEudfl02iI=', 'Content-Length': '0'} 2023-05-31T20:46:31.726 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000000a0015e66f3b9409-006477b227-1630-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:46:31 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:46:31.726 DEBUG:boto:b'pklxkd-59obj221000falseobj222023-05-31T20:39:26.473Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:46:31.727 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj33 2023-05-31T20:46:31.727 DEBUG:boto:path=/pklxkd-59/obj33 2023-05-31T20:46:31.728 DEBUG:boto:auth_path=/pklxkd-59/obj33 2023-05-31T20:46:31.728 DEBUG:boto:path=/pklxkd-59/obj33?versionId=null 2023-05-31T20:46:31.728 DEBUG:boto:auth_path=/pklxkd-59/obj33?versionId=null 2023-05-31T20:46:31.729 DEBUG:boto:Method: GET 2023-05-31T20:46:31.729 DEBUG:boto:Path: /pklxkd-59/obj33?versionId=null 2023-05-31T20:46:31.729 DEBUG:boto:Data: 2023-05-31T20:46:31.730 DEBUG:boto:Headers: {} 2023-05-31T20:46:31.730 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:46:31.730 DEBUG:boto:Port: 8001 2023-05-31T20:46:31.731 DEBUG:boto:Params: {} 2023-05-31T20:46:31.731 DEBUG:boto:Token: None 2023-05-31T20:46:31.732 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:46:31 GMT /pklxkd-59/obj33?versionId=null 2023-05-31T20:46:31.732 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:Shxu0fo4ODU2lfD6D5ZcGkF0SQo= 2023-05-31T20:46:31.732 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:46:31 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:Shxu0fo4ODU2lfD6D5ZcGkF0SQo=', 'Content-Length': '0'} 2023-05-31T20:46:31.734 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:39:26 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000cee1b3624c0ca6ab-006477b227-1c17-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:46:31 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:46:31.776 DEBUG:boto:path=/pklxkd-59/obj33 2023-05-31T20:46:31.777 DEBUG:boto:auth_path=/pklxkd-59/obj33 2023-05-31T20:46:31.777 DEBUG:boto:path=/pklxkd-59/obj33?versionId=null 2023-05-31T20:46:31.778 DEBUG:boto:auth_path=/pklxkd-59/obj33?versionId=null 2023-05-31T20:46:31.778 DEBUG:boto:Method: GET 2023-05-31T20:46:31.778 DEBUG:boto:Path: /pklxkd-59/obj33?versionId=null 2023-05-31T20:46:31.779 DEBUG:boto:Data: 2023-05-31T20:46:31.779 DEBUG:boto:Headers: {} 2023-05-31T20:46:31.779 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:46:31.780 DEBUG:boto:Port: 8000 2023-05-31T20:46:31.780 DEBUG:boto:Params: {} 2023-05-31T20:46:31.781 DEBUG:boto:Token: None 2023-05-31T20:46:31.781 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:46:31 GMT /pklxkd-59/obj33?versionId=null 2023-05-31T20:46:31.782 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:Shxu0fo4ODU2lfD6D5ZcGkF0SQo= 2023-05-31T20:46:31.782 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:46:31 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:Shxu0fo4ODU2lfD6D5ZcGkF0SQo=', 'Content-Length': '0'} 2023-05-31T20:46:31.784 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:39:26 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000013e62f4c8abe27ac-006477b227-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:46:31 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:46:31.785 DEBUG:boto:path=/pklxkd-59/obj33 2023-05-31T20:46:31.785 DEBUG:boto:auth_path=/pklxkd-59/obj33 2023-05-31T20:46:31.785 DEBUG:boto:path=/pklxkd-59/obj33?versionId=null 2023-05-31T20:46:31.786 DEBUG:boto:auth_path=/pklxkd-59/obj33?versionId=null 2023-05-31T20:46:31.786 DEBUG:boto:Method: HEAD 2023-05-31T20:46:31.786 DEBUG:boto:Path: /pklxkd-59/obj33?versionId=null 2023-05-31T20:46:31.787 DEBUG:boto:Data: 2023-05-31T20:46:31.787 DEBUG:boto:Headers: {} 2023-05-31T20:46:31.787 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:46:31.788 DEBUG:boto:Port: 8001 2023-05-31T20:46:31.788 DEBUG:boto:Params: {} 2023-05-31T20:46:31.788 DEBUG:boto:Token: None 2023-05-31T20:46:31.789 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:46:31 GMT /pklxkd-59/obj33?versionId=null 2023-05-31T20:46:31.789 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:VZbaORiQY86QuWGuF1kwG4pZhho= 2023-05-31T20:46:31.790 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:46:31 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:VZbaORiQY86QuWGuF1kwG4pZhho=', 'Content-Length': '0'} 2023-05-31T20:46:31.791 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:39:26 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000031e5678450497047-006477b227-1c17-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:46:31 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:46:31.792 DEBUG:boto:path=/pklxkd-59/obj33 2023-05-31T20:46:31.792 DEBUG:boto:auth_path=/pklxkd-59/obj33 2023-05-31T20:46:31.793 DEBUG:boto:path=/pklxkd-59/obj33?versionId=null 2023-05-31T20:46:31.793 DEBUG:boto:auth_path=/pklxkd-59/obj33?versionId=null 2023-05-31T20:46:31.793 DEBUG:boto:Method: HEAD 2023-05-31T20:46:31.794 DEBUG:boto:Path: /pklxkd-59/obj33?versionId=null 2023-05-31T20:46:31.794 DEBUG:boto:Data: 2023-05-31T20:46:31.794 DEBUG:boto:Headers: {} 2023-05-31T20:46:31.795 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:46:31.795 DEBUG:boto:Port: 8000 2023-05-31T20:46:31.795 DEBUG:boto:Params: {} 2023-05-31T20:46:31.796 DEBUG:boto:Token: None 2023-05-31T20:46:31.796 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:46:31 GMT /pklxkd-59/obj33?versionId=null 2023-05-31T20:46:31.796 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:VZbaORiQY86QuWGuF1kwG4pZhho= 2023-05-31T20:46:31.797 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:46:31 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:VZbaORiQY86QuWGuF1kwG4pZhho=', 'Content-Length': '0'} 2023-05-31T20:46:31.799 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:39:26 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000035232f8add8098fb-006477b227-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:46:31 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:46:31.799 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj33 2023-05-31T20:46:31.800 DEBUG:boto:path=/pklxkd-59/obj33 2023-05-31T20:46:31.800 DEBUG:boto:auth_path=/pklxkd-59/obj33 2023-05-31T20:46:31.800 DEBUG:boto:path=/pklxkd-59/obj33?versionId=null 2023-05-31T20:46:31.801 DEBUG:boto:auth_path=/pklxkd-59/obj33?versionId=null 2023-05-31T20:46:31.801 DEBUG:boto:Method: GET 2023-05-31T20:46:31.801 DEBUG:boto:Path: /pklxkd-59/obj33?versionId=null 2023-05-31T20:46:31.802 DEBUG:boto:Data: 2023-05-31T20:46:31.802 DEBUG:boto:Headers: {} 2023-05-31T20:46:31.802 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:46:31.803 DEBUG:boto:Port: 8001 2023-05-31T20:46:31.803 DEBUG:boto:Params: {} 2023-05-31T20:46:31.803 DEBUG:boto:Token: None 2023-05-31T20:46:31.804 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:46:31 GMT /pklxkd-59/obj33?versionId=null 2023-05-31T20:46:31.804 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:Shxu0fo4ODU2lfD6D5ZcGkF0SQo= 2023-05-31T20:46:31.805 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:46:31 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:Shxu0fo4ODU2lfD6D5ZcGkF0SQo=', 'Content-Length': '0'} 2023-05-31T20:46:31.806 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:39:26 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002857a245815f2544-006477b227-1c17-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:46:31 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:46:31.807 DEBUG:boto:path=/pklxkd-59/obj33 2023-05-31T20:46:31.807 DEBUG:boto:auth_path=/pklxkd-59/obj33 2023-05-31T20:46:31.808 DEBUG:boto:path=/pklxkd-59/obj33?versionId=null 2023-05-31T20:46:31.808 DEBUG:boto:auth_path=/pklxkd-59/obj33?versionId=null 2023-05-31T20:46:31.809 DEBUG:boto:Method: GET 2023-05-31T20:46:31.809 DEBUG:boto:Path: /pklxkd-59/obj33?versionId=null 2023-05-31T20:46:31.809 DEBUG:boto:Data: 2023-05-31T20:46:31.810 DEBUG:boto:Headers: {} 2023-05-31T20:46:31.810 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:46:31.810 DEBUG:boto:Port: 8000 2023-05-31T20:46:31.811 DEBUG:boto:Params: {} 2023-05-31T20:46:31.811 DEBUG:boto:Token: None 2023-05-31T20:46:31.811 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:46:31 GMT /pklxkd-59/obj33?versionId=null 2023-05-31T20:46:31.812 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:Shxu0fo4ODU2lfD6D5ZcGkF0SQo= 2023-05-31T20:46:31.812 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:46:31 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:Shxu0fo4ODU2lfD6D5ZcGkF0SQo=', 'Content-Length': '0'} 2023-05-31T20:46:31.814 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:39:26 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f2f36ac5ca03fa93-006477b227-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:46:31 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:46:31.815 DEBUG:boto:path=/pklxkd-59/ 2023-05-31T20:46:31.815 DEBUG:boto:auth_path=/pklxkd-59/ 2023-05-31T20:46:31.816 DEBUG:boto:path=/pklxkd-59/?prefix=obj33 2023-05-31T20:46:31.816 DEBUG:boto:auth_path=/pklxkd-59/?prefix=obj33 2023-05-31T20:46:31.816 DEBUG:boto:Method: GET 2023-05-31T20:46:31.817 DEBUG:boto:Path: /pklxkd-59/?prefix=obj33 2023-05-31T20:46:31.817 DEBUG:boto:Data: 2023-05-31T20:46:31.817 DEBUG:boto:Headers: {} 2023-05-31T20:46:31.818 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:46:31.818 DEBUG:boto:Port: 8001 2023-05-31T20:46:31.818 DEBUG:boto:Params: {} 2023-05-31T20:46:31.819 DEBUG:boto:Token: None 2023-05-31T20:46:31.819 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:46:31 GMT /pklxkd-59/ 2023-05-31T20:46:31.820 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:gTuLAyT7WsrIPapdTlEudfl02iI= 2023-05-31T20:46:31.820 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:46:31 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:gTuLAyT7WsrIPapdTlEudfl02iI=', 'Content-Length': '0'} 2023-05-31T20:46:31.822 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a8a517b4badcf1d9-006477b227-1c17-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:46:31 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:46:31.823 DEBUG:boto:b'pklxkd-59obj331000falseobj332023-05-31T20:39:26.534Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:46:31.824 DEBUG:boto:path=/pklxkd-59/ 2023-05-31T20:46:31.824 DEBUG:boto:auth_path=/pklxkd-59/ 2023-05-31T20:46:31.824 DEBUG:boto:path=/pklxkd-59/?prefix=obj33 2023-05-31T20:46:31.825 DEBUG:boto:auth_path=/pklxkd-59/?prefix=obj33 2023-05-31T20:46:31.825 DEBUG:boto:Method: GET 2023-05-31T20:46:31.825 DEBUG:boto:Path: /pklxkd-59/?prefix=obj33 2023-05-31T20:46:31.826 DEBUG:boto:Data: 2023-05-31T20:46:31.826 DEBUG:boto:Headers: {} 2023-05-31T20:46:31.827 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:46:31.827 DEBUG:boto:Port: 8000 2023-05-31T20:46:31.827 DEBUG:boto:Params: {} 2023-05-31T20:46:31.828 DEBUG:boto:Token: None 2023-05-31T20:46:31.828 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:46:31 GMT /pklxkd-59/ 2023-05-31T20:46:31.828 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:gTuLAyT7WsrIPapdTlEudfl02iI= 2023-05-31T20:46:31.829 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:46:31 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:gTuLAyT7WsrIPapdTlEudfl02iI=', 'Content-Length': '0'} 2023-05-31T20:46:31.832 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000083a77a24c1ec4ea-006477b227-1630-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:46:31 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:46:31.832 DEBUG:boto:b'pklxkd-59obj331000falseobj332023-05-31T20:39:26.534Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:46:31.833 DEBUG:boto:path=/pklxkd-59/obj33 2023-05-31T20:46:31.833 DEBUG:boto:auth_path=/pklxkd-59/obj33 2023-05-31T20:46:31.834 DEBUG:boto:Method: HEAD 2023-05-31T20:46:31.834 DEBUG:boto:Path: /pklxkd-59/obj33 2023-05-31T20:46:31.834 DEBUG:boto:Data: 2023-05-31T20:46:31.835 DEBUG:boto:Headers: {} 2023-05-31T20:46:31.835 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:46:31.835 DEBUG:boto:Port: 8001 2023-05-31T20:46:31.836 DEBUG:boto:Params: {} 2023-05-31T20:46:31.836 DEBUG:boto:Token: None 2023-05-31T20:46:31.836 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:46:31 GMT /pklxkd-59/obj33 2023-05-31T20:46:31.837 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:XDStwC1WBz5jqfn4rl7BoP7H7U8= 2023-05-31T20:46:31.837 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:46:31 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:XDStwC1WBz5jqfn4rl7BoP7H7U8=', 'Content-Length': '0'} 2023-05-31T20:46:31.839 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:39:26 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005b68e227980d3bec-006477b227-1c17-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:46:31 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:46:31.840 DEBUG:boto:path=/pklxkd-59/obj33 2023-05-31T20:46:31.840 DEBUG:boto:auth_path=/pklxkd-59/obj33 2023-05-31T20:46:31.840 DEBUG:boto:Method: HEAD 2023-05-31T20:46:31.841 DEBUG:boto:Path: /pklxkd-59/obj33 2023-05-31T20:46:31.841 DEBUG:boto:Data: 2023-05-31T20:46:31.841 DEBUG:boto:Headers: {} 2023-05-31T20:46:31.842 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:46:31.842 DEBUG:boto:Port: 8000 2023-05-31T20:46:31.842 DEBUG:boto:Params: {} 2023-05-31T20:46:31.843 DEBUG:boto:Token: None 2023-05-31T20:46:31.843 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:46:31 GMT /pklxkd-59/obj33 2023-05-31T20:46:31.843 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:XDStwC1WBz5jqfn4rl7BoP7H7U8= 2023-05-31T20:46:31.844 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:46:31 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:XDStwC1WBz5jqfn4rl7BoP7H7U8=', 'Content-Length': '0'} 2023-05-31T20:46:31.846 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:39:26 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000087dfe60f5974843f-006477b227-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:46:31 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:46:31.846 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj33 2023-05-31T20:46:31.847 DEBUG:boto:path=/pklxkd-59/obj33 2023-05-31T20:46:31.847 DEBUG:boto:auth_path=/pklxkd-59/obj33 2023-05-31T20:46:31.847 DEBUG:boto:Method: GET 2023-05-31T20:46:31.848 DEBUG:boto:Path: /pklxkd-59/obj33 2023-05-31T20:46:31.848 DEBUG:boto:Data: 2023-05-31T20:46:31.848 DEBUG:boto:Headers: {} 2023-05-31T20:46:31.849 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:46:31.849 DEBUG:boto:Port: 8001 2023-05-31T20:46:31.849 DEBUG:boto:Params: {} 2023-05-31T20:46:31.850 DEBUG:boto:Token: None 2023-05-31T20:46:31.850 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:46:31 GMT /pklxkd-59/obj33 2023-05-31T20:46:31.851 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:Ap4ZInq17F12f/byBBrR+VL43cw= 2023-05-31T20:46:31.851 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:46:31 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:Ap4ZInq17F12f/byBBrR+VL43cw=', 'Content-Length': '0'} 2023-05-31T20:46:31.853 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:39:26 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ce9788c5c2cdfdfc-006477b227-1c17-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:46:31 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:46:31.854 DEBUG:boto:path=/pklxkd-59/obj33 2023-05-31T20:46:31.854 DEBUG:boto:auth_path=/pklxkd-59/obj33 2023-05-31T20:46:31.855 DEBUG:boto:Method: GET 2023-05-31T20:46:31.855 DEBUG:boto:Path: /pklxkd-59/obj33 2023-05-31T20:46:31.855 DEBUG:boto:Data: 2023-05-31T20:46:31.856 DEBUG:boto:Headers: {} 2023-05-31T20:46:31.856 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:46:31.856 DEBUG:boto:Port: 8000 2023-05-31T20:46:31.857 DEBUG:boto:Params: {} 2023-05-31T20:46:31.857 DEBUG:boto:Token: None 2023-05-31T20:46:31.858 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:46:31 GMT /pklxkd-59/obj33 2023-05-31T20:46:31.858 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:Ap4ZInq17F12f/byBBrR+VL43cw= 2023-05-31T20:46:31.858 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:46:31 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:Ap4ZInq17F12f/byBBrR+VL43cw=', 'Content-Length': '0'} 2023-05-31T20:46:31.860 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:39:26 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f0f06f572e7d7d6c-006477b227-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:46:31 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:46:31.861 DEBUG:boto:path=/pklxkd-59/ 2023-05-31T20:46:31.861 DEBUG:boto:auth_path=/pklxkd-59/ 2023-05-31T20:46:31.862 DEBUG:boto:path=/pklxkd-59/?prefix=obj33 2023-05-31T20:46:31.862 DEBUG:boto:auth_path=/pklxkd-59/?prefix=obj33 2023-05-31T20:46:31.862 DEBUG:boto:Method: GET 2023-05-31T20:46:31.863 DEBUG:boto:Path: /pklxkd-59/?prefix=obj33 2023-05-31T20:46:31.863 DEBUG:boto:Data: 2023-05-31T20:46:31.864 DEBUG:boto:Headers: {} 2023-05-31T20:46:31.864 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:46:31.864 DEBUG:boto:Port: 8001 2023-05-31T20:46:31.865 DEBUG:boto:Params: {} 2023-05-31T20:46:31.865 DEBUG:boto:Token: None 2023-05-31T20:46:31.865 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:46:31 GMT /pklxkd-59/ 2023-05-31T20:46:31.866 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:gTuLAyT7WsrIPapdTlEudfl02iI= 2023-05-31T20:46:31.866 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:46:31 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:gTuLAyT7WsrIPapdTlEudfl02iI=', 'Content-Length': '0'} 2023-05-31T20:46:31.868 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000098859b3af510780b-006477b227-1c17-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:46:31 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:46:31.869 DEBUG:boto:b'pklxkd-59obj331000falseobj332023-05-31T20:39:26.534Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:46:31.870 DEBUG:boto:path=/pklxkd-59/ 2023-05-31T20:46:31.870 DEBUG:boto:auth_path=/pklxkd-59/ 2023-05-31T20:46:31.870 DEBUG:boto:path=/pklxkd-59/?prefix=obj33 2023-05-31T20:46:31.871 DEBUG:boto:auth_path=/pklxkd-59/?prefix=obj33 2023-05-31T20:46:31.871 DEBUG:boto:Method: GET 2023-05-31T20:46:31.871 DEBUG:boto:Path: /pklxkd-59/?prefix=obj33 2023-05-31T20:46:31.872 DEBUG:boto:Data: 2023-05-31T20:46:31.872 DEBUG:boto:Headers: {} 2023-05-31T20:46:31.873 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:46:31.873 DEBUG:boto:Port: 8000 2023-05-31T20:46:31.873 DEBUG:boto:Params: {} 2023-05-31T20:46:31.874 DEBUG:boto:Token: None 2023-05-31T20:46:31.874 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:46:31 GMT /pklxkd-59/ 2023-05-31T20:46:31.874 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:gTuLAyT7WsrIPapdTlEudfl02iI= 2023-05-31T20:46:31.875 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:46:31 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:gTuLAyT7WsrIPapdTlEudfl02iI=', 'Content-Length': '0'} 2023-05-31T20:46:31.878 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000098a050c1a7ad062d-006477b227-1630-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:46:31 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:46:31.920 DEBUG:boto:b'pklxkd-59obj331000falseobj332023-05-31T20:39:26.534Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:46:31.921 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj34 2023-05-31T20:46:31.921 DEBUG:boto:path=/pklxkd-59/obj34 2023-05-31T20:46:31.922 DEBUG:boto:auth_path=/pklxkd-59/obj34 2023-05-31T20:46:31.922 DEBUG:boto:path=/pklxkd-59/obj34?versionId=null 2023-05-31T20:46:31.922 DEBUG:boto:auth_path=/pklxkd-59/obj34?versionId=null 2023-05-31T20:46:31.923 DEBUG:boto:Method: GET 2023-05-31T20:46:31.923 DEBUG:boto:Path: /pklxkd-59/obj34?versionId=null 2023-05-31T20:46:31.924 DEBUG:boto:Data: 2023-05-31T20:46:31.924 DEBUG:boto:Headers: {} 2023-05-31T20:46:31.924 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:46:31.925 DEBUG:boto:Port: 8001 2023-05-31T20:46:31.925 DEBUG:boto:Params: {} 2023-05-31T20:46:31.925 DEBUG:boto:Token: None 2023-05-31T20:46:31.926 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:46:31 GMT /pklxkd-59/obj34?versionId=null 2023-05-31T20:46:31.926 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:Xw/rSg112QR3XayrDZrsaAxfOpA= 2023-05-31T20:46:31.927 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:46:31 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:Xw/rSg112QR3XayrDZrsaAxfOpA=', 'Content-Length': '0'} 2023-05-31T20:46:31.929 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:43:07 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004109253a7ab2ba11-006477b227-1c17-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:46:31 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:46:31.929 DEBUG:boto:path=/pklxkd-59/obj34 2023-05-31T20:46:31.930 DEBUG:boto:auth_path=/pklxkd-59/obj34 2023-05-31T20:46:31.930 DEBUG:boto:path=/pklxkd-59/obj34?versionId=null 2023-05-31T20:46:31.931 DEBUG:boto:auth_path=/pklxkd-59/obj34?versionId=null 2023-05-31T20:46:31.931 DEBUG:boto:Method: GET 2023-05-31T20:46:31.931 DEBUG:boto:Path: /pklxkd-59/obj34?versionId=null 2023-05-31T20:46:31.932 DEBUG:boto:Data: 2023-05-31T20:46:31.932 DEBUG:boto:Headers: {} 2023-05-31T20:46:31.932 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:46:31.933 DEBUG:boto:Port: 8000 2023-05-31T20:46:31.933 DEBUG:boto:Params: {} 2023-05-31T20:46:31.933 DEBUG:boto:Token: None 2023-05-31T20:46:31.934 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:46:31 GMT /pklxkd-59/obj34?versionId=null 2023-05-31T20:46:31.934 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:Xw/rSg112QR3XayrDZrsaAxfOpA= 2023-05-31T20:46:31.935 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:46:31 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:Xw/rSg112QR3XayrDZrsaAxfOpA=', 'Content-Length': '0'} 2023-05-31T20:46:31.936 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:43:07 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001a084f02c6d5bc38-006477b227-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:46:31 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:46:31.937 DEBUG:boto:path=/pklxkd-59/obj34 2023-05-31T20:46:31.937 DEBUG:boto:auth_path=/pklxkd-59/obj34 2023-05-31T20:46:31.938 DEBUG:boto:path=/pklxkd-59/obj34?versionId=null 2023-05-31T20:46:31.938 DEBUG:boto:auth_path=/pklxkd-59/obj34?versionId=null 2023-05-31T20:46:31.938 DEBUG:boto:Method: HEAD 2023-05-31T20:46:31.939 DEBUG:boto:Path: /pklxkd-59/obj34?versionId=null 2023-05-31T20:46:31.939 DEBUG:boto:Data: 2023-05-31T20:46:31.939 DEBUG:boto:Headers: {} 2023-05-31T20:46:31.940 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:46:31.940 DEBUG:boto:Port: 8001 2023-05-31T20:46:31.940 DEBUG:boto:Params: {} 2023-05-31T20:46:31.941 DEBUG:boto:Token: None 2023-05-31T20:46:31.941 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:46:31 GMT /pklxkd-59/obj34?versionId=null 2023-05-31T20:46:31.942 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:sJ1KMqYU72OU49GG+11d68OI300= 2023-05-31T20:46:31.942 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:46:31 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:sJ1KMqYU72OU49GG+11d68OI300=', 'Content-Length': '0'} 2023-05-31T20:46:31.944 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:43:07 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001899613f9919f5bc-006477b227-1c17-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:46:31 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:46:31.944 DEBUG:boto:path=/pklxkd-59/obj34 2023-05-31T20:46:31.945 DEBUG:boto:auth_path=/pklxkd-59/obj34 2023-05-31T20:46:31.945 DEBUG:boto:path=/pklxkd-59/obj34?versionId=null 2023-05-31T20:46:31.945 DEBUG:boto:auth_path=/pklxkd-59/obj34?versionId=null 2023-05-31T20:46:31.946 DEBUG:boto:Method: HEAD 2023-05-31T20:46:31.946 DEBUG:boto:Path: /pklxkd-59/obj34?versionId=null 2023-05-31T20:46:31.946 DEBUG:boto:Data: 2023-05-31T20:46:31.947 DEBUG:boto:Headers: {} 2023-05-31T20:46:31.947 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:46:31.947 DEBUG:boto:Port: 8000 2023-05-31T20:46:31.948 DEBUG:boto:Params: {} 2023-05-31T20:46:31.948 DEBUG:boto:Token: None 2023-05-31T20:46:31.948 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:46:31 GMT /pklxkd-59/obj34?versionId=null 2023-05-31T20:46:31.949 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:sJ1KMqYU72OU49GG+11d68OI300= 2023-05-31T20:46:31.949 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:46:31 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:sJ1KMqYU72OU49GG+11d68OI300=', 'Content-Length': '0'} 2023-05-31T20:46:31.951 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:43:07 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000086985cc05f836519-006477b227-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:46:31 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:46:31.951 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj34 2023-05-31T20:46:31.952 DEBUG:boto:path=/pklxkd-59/obj34 2023-05-31T20:46:31.952 DEBUG:boto:auth_path=/pklxkd-59/obj34 2023-05-31T20:46:31.953 DEBUG:boto:path=/pklxkd-59/obj34?versionId=null 2023-05-31T20:46:31.953 DEBUG:boto:auth_path=/pklxkd-59/obj34?versionId=null 2023-05-31T20:46:31.953 DEBUG:boto:Method: GET 2023-05-31T20:46:31.954 DEBUG:boto:Path: /pklxkd-59/obj34?versionId=null 2023-05-31T20:46:31.954 DEBUG:boto:Data: 2023-05-31T20:46:31.954 DEBUG:boto:Headers: {} 2023-05-31T20:46:31.955 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:46:31.955 DEBUG:boto:Port: 8001 2023-05-31T20:46:31.955 DEBUG:boto:Params: {} 2023-05-31T20:46:31.956 DEBUG:boto:Token: None 2023-05-31T20:46:31.956 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:46:31 GMT /pklxkd-59/obj34?versionId=null 2023-05-31T20:46:31.957 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:Xw/rSg112QR3XayrDZrsaAxfOpA= 2023-05-31T20:46:31.957 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:46:31 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:Xw/rSg112QR3XayrDZrsaAxfOpA=', 'Content-Length': '0'} 2023-05-31T20:46:31.959 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:43:07 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000076c14967bc71321e-006477b227-1c17-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:46:31 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:46:32.000 DEBUG:boto:path=/pklxkd-59/obj34 2023-05-31T20:46:32.001 DEBUG:boto:auth_path=/pklxkd-59/obj34 2023-05-31T20:46:32.001 DEBUG:boto:path=/pklxkd-59/obj34?versionId=null 2023-05-31T20:46:32.001 DEBUG:boto:auth_path=/pklxkd-59/obj34?versionId=null 2023-05-31T20:46:32.002 DEBUG:boto:Method: GET 2023-05-31T20:46:32.002 DEBUG:boto:Path: /pklxkd-59/obj34?versionId=null 2023-05-31T20:46:32.002 DEBUG:boto:Data: 2023-05-31T20:46:32.003 DEBUG:boto:Headers: {} 2023-05-31T20:46:32.003 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:46:32.004 DEBUG:boto:Port: 8000 2023-05-31T20:46:32.004 DEBUG:boto:Params: {} 2023-05-31T20:46:32.004 DEBUG:boto:Token: None 2023-05-31T20:46:32.005 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:46:32 GMT /pklxkd-59/obj34?versionId=null 2023-05-31T20:46:32.005 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:RIOtWNh/tHq0b/Dyy9mAqghlajw= 2023-05-31T20:46:32.006 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:46:32 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:RIOtWNh/tHq0b/Dyy9mAqghlajw=', 'Content-Length': '0'} 2023-05-31T20:46:32.007 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:43:07 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000027ef6d23900f6305-006477b228-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:46:32 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:46:32.008 DEBUG:boto:path=/pklxkd-59/ 2023-05-31T20:46:32.008 DEBUG:boto:auth_path=/pklxkd-59/ 2023-05-31T20:46:32.009 DEBUG:boto:path=/pklxkd-59/?prefix=obj34 2023-05-31T20:46:32.009 DEBUG:boto:auth_path=/pklxkd-59/?prefix=obj34 2023-05-31T20:46:32.009 DEBUG:boto:Method: GET 2023-05-31T20:46:32.010 DEBUG:boto:Path: /pklxkd-59/?prefix=obj34 2023-05-31T20:46:32.010 DEBUG:boto:Data: 2023-05-31T20:46:32.011 DEBUG:boto:Headers: {} 2023-05-31T20:46:32.011 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:46:32.011 DEBUG:boto:Port: 8001 2023-05-31T20:46:32.012 DEBUG:boto:Params: {} 2023-05-31T20:46:32.012 DEBUG:boto:Token: None 2023-05-31T20:46:32.012 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:46:32 GMT /pklxkd-59/ 2023-05-31T20:46:32.013 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:3Lv9IKKSTFx3S8X7r8veHGM3xUY= 2023-05-31T20:46:32.013 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:46:32 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:3Lv9IKKSTFx3S8X7r8veHGM3xUY=', 'Content-Length': '0'} 2023-05-31T20:46:32.017 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d4c2b6ca23d59f1d-006477b228-1c17-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:46:32 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:46:32.017 DEBUG:boto:b'pklxkd-59obj341000falseobj342023-05-31T20:43:07.994Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:46:32.018 DEBUG:boto:path=/pklxkd-59/ 2023-05-31T20:46:32.018 DEBUG:boto:auth_path=/pklxkd-59/ 2023-05-31T20:46:32.019 DEBUG:boto:path=/pklxkd-59/?prefix=obj34 2023-05-31T20:46:32.019 DEBUG:boto:auth_path=/pklxkd-59/?prefix=obj34 2023-05-31T20:46:32.019 DEBUG:boto:Method: GET 2023-05-31T20:46:32.020 DEBUG:boto:Path: /pklxkd-59/?prefix=obj34 2023-05-31T20:46:32.020 DEBUG:boto:Data: 2023-05-31T20:46:32.021 DEBUG:boto:Headers: {} 2023-05-31T20:46:32.021 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:46:32.021 DEBUG:boto:Port: 8000 2023-05-31T20:46:32.022 DEBUG:boto:Params: {} 2023-05-31T20:46:32.022 DEBUG:boto:Token: None 2023-05-31T20:46:32.022 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:46:32 GMT /pklxkd-59/ 2023-05-31T20:46:32.023 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:3Lv9IKKSTFx3S8X7r8veHGM3xUY= 2023-05-31T20:46:32.023 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:46:32 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:3Lv9IKKSTFx3S8X7r8veHGM3xUY=', 'Content-Length': '0'} 2023-05-31T20:46:32.026 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000008c4001744b4122ee-006477b228-1630-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:46:32 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:46:32.027 DEBUG:boto:b'pklxkd-59obj341000falseobj342023-05-31T20:43:07.994Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:46:32.027 DEBUG:boto:path=/pklxkd-59/obj34 2023-05-31T20:46:32.028 DEBUG:boto:auth_path=/pklxkd-59/obj34 2023-05-31T20:46:32.028 DEBUG:boto:Method: HEAD 2023-05-31T20:46:32.029 DEBUG:boto:Path: /pklxkd-59/obj34 2023-05-31T20:46:32.029 DEBUG:boto:Data: 2023-05-31T20:46:32.029 DEBUG:boto:Headers: {} 2023-05-31T20:46:32.030 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:46:32.030 DEBUG:boto:Port: 8001 2023-05-31T20:46:32.030 DEBUG:boto:Params: {} 2023-05-31T20:46:32.031 DEBUG:boto:Token: None 2023-05-31T20:46:32.031 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:46:32 GMT /pklxkd-59/obj34 2023-05-31T20:46:32.031 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:f1pTdfDkFPps5A7PBIz8jCAkC7E= 2023-05-31T20:46:32.032 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:46:32 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:f1pTdfDkFPps5A7PBIz8jCAkC7E=', 'Content-Length': '0'} 2023-05-31T20:46:32.034 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:43:07 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002997cbbc3322c64d-006477b228-1c17-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:46:32 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:46:32.034 DEBUG:boto:path=/pklxkd-59/obj34 2023-05-31T20:46:32.034 DEBUG:boto:auth_path=/pklxkd-59/obj34 2023-05-31T20:46:32.035 DEBUG:boto:Method: HEAD 2023-05-31T20:46:32.035 DEBUG:boto:Path: /pklxkd-59/obj34 2023-05-31T20:46:32.035 DEBUG:boto:Data: 2023-05-31T20:46:32.036 DEBUG:boto:Headers: {} 2023-05-31T20:46:32.036 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:46:32.037 DEBUG:boto:Port: 8000 2023-05-31T20:46:32.037 DEBUG:boto:Params: {} 2023-05-31T20:46:32.037 DEBUG:boto:Token: None 2023-05-31T20:46:32.038 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:46:32 GMT /pklxkd-59/obj34 2023-05-31T20:46:32.038 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:f1pTdfDkFPps5A7PBIz8jCAkC7E= 2023-05-31T20:46:32.038 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:46:32 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:f1pTdfDkFPps5A7PBIz8jCAkC7E=', 'Content-Length': '0'} 2023-05-31T20:46:32.040 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:43:07 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000722fa3c7e2966d2e-006477b228-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:46:32 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:46:32.041 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj34 2023-05-31T20:46:32.041 DEBUG:boto:path=/pklxkd-59/obj34 2023-05-31T20:46:32.041 DEBUG:boto:auth_path=/pklxkd-59/obj34 2023-05-31T20:46:32.042 DEBUG:boto:Method: GET 2023-05-31T20:46:32.042 DEBUG:boto:Path: /pklxkd-59/obj34 2023-05-31T20:46:32.042 DEBUG:boto:Data: 2023-05-31T20:46:32.043 DEBUG:boto:Headers: {} 2023-05-31T20:46:32.043 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:46:32.044 DEBUG:boto:Port: 8001 2023-05-31T20:46:32.044 DEBUG:boto:Params: {} 2023-05-31T20:46:32.044 DEBUG:boto:Token: None 2023-05-31T20:46:32.045 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:46:32 GMT /pklxkd-59/obj34 2023-05-31T20:46:32.045 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:nnW2HKrZYe08GrDf5SuaU8D4bFA= 2023-05-31T20:46:32.046 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:46:32 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:nnW2HKrZYe08GrDf5SuaU8D4bFA=', 'Content-Length': '0'} 2023-05-31T20:46:32.047 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:43:07 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c9b37711a3d1bef0-006477b228-1c17-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:46:32 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:46:32.048 DEBUG:boto:path=/pklxkd-59/obj34 2023-05-31T20:46:32.048 DEBUG:boto:auth_path=/pklxkd-59/obj34 2023-05-31T20:46:32.049 DEBUG:boto:Method: GET 2023-05-31T20:46:32.049 DEBUG:boto:Path: /pklxkd-59/obj34 2023-05-31T20:46:32.050 DEBUG:boto:Data: 2023-05-31T20:46:32.050 DEBUG:boto:Headers: {} 2023-05-31T20:46:32.050 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:46:32.051 DEBUG:boto:Port: 8000 2023-05-31T20:46:32.051 DEBUG:boto:Params: {} 2023-05-31T20:46:32.051 DEBUG:boto:Token: None 2023-05-31T20:46:32.052 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:46:32 GMT /pklxkd-59/obj34 2023-05-31T20:46:32.052 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:nnW2HKrZYe08GrDf5SuaU8D4bFA= 2023-05-31T20:46:32.053 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:46:32 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:nnW2HKrZYe08GrDf5SuaU8D4bFA=', 'Content-Length': '0'} 2023-05-31T20:46:32.054 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:43:07 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e1d494a80f88aa57-006477b228-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:46:32 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:46:32.096 DEBUG:boto:path=/pklxkd-59/ 2023-05-31T20:46:32.097 DEBUG:boto:auth_path=/pklxkd-59/ 2023-05-31T20:46:32.097 DEBUG:boto:path=/pklxkd-59/?prefix=obj34 2023-05-31T20:46:32.097 DEBUG:boto:auth_path=/pklxkd-59/?prefix=obj34 2023-05-31T20:46:32.098 DEBUG:boto:Method: GET 2023-05-31T20:46:32.098 DEBUG:boto:Path: /pklxkd-59/?prefix=obj34 2023-05-31T20:46:32.098 DEBUG:boto:Data: 2023-05-31T20:46:32.099 DEBUG:boto:Headers: {} 2023-05-31T20:46:32.099 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:46:32.100 DEBUG:boto:Port: 8001 2023-05-31T20:46:32.100 DEBUG:boto:Params: {} 2023-05-31T20:46:32.100 DEBUG:boto:Token: None 2023-05-31T20:46:32.101 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:46:32 GMT /pklxkd-59/ 2023-05-31T20:46:32.101 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:3Lv9IKKSTFx3S8X7r8veHGM3xUY= 2023-05-31T20:46:32.102 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:46:32 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:3Lv9IKKSTFx3S8X7r8veHGM3xUY=', 'Content-Length': '0'} 2023-05-31T20:46:32.104 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000287c0845ac16a169-006477b228-1c17-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:46:32 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:46:32.105 DEBUG:boto:b'pklxkd-59obj341000falseobj342023-05-31T20:43:07.994Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:46:32.105 DEBUG:boto:path=/pklxkd-59/ 2023-05-31T20:46:32.106 DEBUG:boto:auth_path=/pklxkd-59/ 2023-05-31T20:46:32.106 DEBUG:boto:path=/pklxkd-59/?prefix=obj34 2023-05-31T20:46:32.106 DEBUG:boto:auth_path=/pklxkd-59/?prefix=obj34 2023-05-31T20:46:32.107 DEBUG:boto:Method: GET 2023-05-31T20:46:32.107 DEBUG:boto:Path: /pklxkd-59/?prefix=obj34 2023-05-31T20:46:32.107 DEBUG:boto:Data: 2023-05-31T20:46:32.108 DEBUG:boto:Headers: {} 2023-05-31T20:46:32.108 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:46:32.109 DEBUG:boto:Port: 8000 2023-05-31T20:46:32.109 DEBUG:boto:Params: {} 2023-05-31T20:46:32.109 DEBUG:boto:Token: None 2023-05-31T20:46:32.110 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:46:32 GMT /pklxkd-59/ 2023-05-31T20:46:32.110 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:3Lv9IKKSTFx3S8X7r8veHGM3xUY= 2023-05-31T20:46:32.111 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:46:32 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:3Lv9IKKSTFx3S8X7r8veHGM3xUY=', 'Content-Length': '0'} 2023-05-31T20:46:32.113 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000003bde85532c8d0558-006477b228-1630-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:46:32 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:46:32.114 DEBUG:boto:b'pklxkd-59obj341000falseobj342023-05-31T20:43:07.994Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:46:32.114 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj38 2023-05-31T20:46:32.115 DEBUG:boto:path=/pklxkd-59/obj38 2023-05-31T20:46:32.115 DEBUG:boto:auth_path=/pklxkd-59/obj38 2023-05-31T20:46:32.115 DEBUG:boto:path=/pklxkd-59/obj38?versionId=null 2023-05-31T20:46:32.116 DEBUG:boto:auth_path=/pklxkd-59/obj38?versionId=null 2023-05-31T20:46:32.116 DEBUG:boto:Method: GET 2023-05-31T20:46:32.117 DEBUG:boto:Path: /pklxkd-59/obj38?versionId=null 2023-05-31T20:46:32.117 DEBUG:boto:Data: 2023-05-31T20:46:32.117 DEBUG:boto:Headers: {} 2023-05-31T20:46:32.118 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:46:32.118 DEBUG:boto:Port: 8001 2023-05-31T20:46:32.118 DEBUG:boto:Params: {} 2023-05-31T20:46:32.119 DEBUG:boto:Token: None 2023-05-31T20:46:32.119 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:46:32 GMT /pklxkd-59/obj38?versionId=null 2023-05-31T20:46:32.119 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:urQh4/k39oOVbipfC3p13N5ZBU0= 2023-05-31T20:46:32.120 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:46:32 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:urQh4/k39oOVbipfC3p13N5ZBU0=', 'Content-Length': '0'} 2023-05-31T20:46:32.122 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:44:50 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000032443e7773b5c4a-006477b228-1c17-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:46:32 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:46:32.164 DEBUG:boto:path=/pklxkd-59/obj38 2023-05-31T20:46:32.165 DEBUG:boto:auth_path=/pklxkd-59/obj38 2023-05-31T20:46:32.165 DEBUG:boto:path=/pklxkd-59/obj38?versionId=null 2023-05-31T20:46:32.165 DEBUG:boto:auth_path=/pklxkd-59/obj38?versionId=null 2023-05-31T20:46:32.166 DEBUG:boto:Method: GET 2023-05-31T20:46:32.166 DEBUG:boto:Path: /pklxkd-59/obj38?versionId=null 2023-05-31T20:46:32.166 DEBUG:boto:Data: 2023-05-31T20:46:32.167 DEBUG:boto:Headers: {} 2023-05-31T20:46:32.167 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:46:32.168 DEBUG:boto:Port: 8000 2023-05-31T20:46:32.168 DEBUG:boto:Params: {} 2023-05-31T20:46:32.168 DEBUG:boto:Token: None 2023-05-31T20:46:32.169 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:46:32 GMT /pklxkd-59/obj38?versionId=null 2023-05-31T20:46:32.169 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:urQh4/k39oOVbipfC3p13N5ZBU0= 2023-05-31T20:46:32.170 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:46:32 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:urQh4/k39oOVbipfC3p13N5ZBU0=', 'Content-Length': '0'} 2023-05-31T20:46:32.171 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:44:50 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ff1889db4ba703bf-006477b228-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:46:32 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:46:32.172 DEBUG:boto:path=/pklxkd-59/obj38 2023-05-31T20:46:32.172 DEBUG:boto:auth_path=/pklxkd-59/obj38 2023-05-31T20:46:32.173 DEBUG:boto:path=/pklxkd-59/obj38?versionId=null 2023-05-31T20:46:32.173 DEBUG:boto:auth_path=/pklxkd-59/obj38?versionId=null 2023-05-31T20:46:32.173 DEBUG:boto:Method: HEAD 2023-05-31T20:46:32.174 DEBUG:boto:Path: /pklxkd-59/obj38?versionId=null 2023-05-31T20:46:32.174 DEBUG:boto:Data: 2023-05-31T20:46:32.174 DEBUG:boto:Headers: {} 2023-05-31T20:46:32.175 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:46:32.175 DEBUG:boto:Port: 8001 2023-05-31T20:46:32.175 DEBUG:boto:Params: {} 2023-05-31T20:46:32.176 DEBUG:boto:Token: None 2023-05-31T20:46:32.176 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:46:32 GMT /pklxkd-59/obj38?versionId=null 2023-05-31T20:46:32.177 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:IJyuu/PrIrWKG4F5kz6E0rZ7Aa8= 2023-05-31T20:46:32.177 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:46:32 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:IJyuu/PrIrWKG4F5kz6E0rZ7Aa8=', 'Content-Length': '0'} 2023-05-31T20:46:32.179 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:44:50 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004a494d147ba31838-006477b228-1c17-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:46:32 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:46:32.179 DEBUG:boto:path=/pklxkd-59/obj38 2023-05-31T20:46:32.180 DEBUG:boto:auth_path=/pklxkd-59/obj38 2023-05-31T20:46:32.180 DEBUG:boto:path=/pklxkd-59/obj38?versionId=null 2023-05-31T20:46:32.180 DEBUG:boto:auth_path=/pklxkd-59/obj38?versionId=null 2023-05-31T20:46:32.181 DEBUG:boto:Method: HEAD 2023-05-31T20:46:32.181 DEBUG:boto:Path: /pklxkd-59/obj38?versionId=null 2023-05-31T20:46:32.181 DEBUG:boto:Data: 2023-05-31T20:46:32.182 DEBUG:boto:Headers: {} 2023-05-31T20:46:32.182 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:46:32.182 DEBUG:boto:Port: 8000 2023-05-31T20:46:32.183 DEBUG:boto:Params: {} 2023-05-31T20:46:32.183 DEBUG:boto:Token: None 2023-05-31T20:46:32.183 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:46:32 GMT /pklxkd-59/obj38?versionId=null 2023-05-31T20:46:32.184 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:IJyuu/PrIrWKG4F5kz6E0rZ7Aa8= 2023-05-31T20:46:32.184 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:46:32 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:IJyuu/PrIrWKG4F5kz6E0rZ7Aa8=', 'Content-Length': '0'} 2023-05-31T20:46:32.186 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:44:50 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000050b8859341b27088-006477b228-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:46:32 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:46:32.186 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj38 2023-05-31T20:46:32.187 DEBUG:boto:path=/pklxkd-59/obj38 2023-05-31T20:46:32.187 DEBUG:boto:auth_path=/pklxkd-59/obj38 2023-05-31T20:46:32.187 DEBUG:boto:path=/pklxkd-59/obj38?versionId=null 2023-05-31T20:46:32.188 DEBUG:boto:auth_path=/pklxkd-59/obj38?versionId=null 2023-05-31T20:46:32.188 DEBUG:boto:Method: GET 2023-05-31T20:46:32.188 DEBUG:boto:Path: /pklxkd-59/obj38?versionId=null 2023-05-31T20:46:32.189 DEBUG:boto:Data: 2023-05-31T20:46:32.189 DEBUG:boto:Headers: {} 2023-05-31T20:46:32.189 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:46:32.190 DEBUG:boto:Port: 8001 2023-05-31T20:46:32.190 DEBUG:boto:Params: {} 2023-05-31T20:46:32.190 DEBUG:boto:Token: None 2023-05-31T20:46:32.191 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:46:32 GMT /pklxkd-59/obj38?versionId=null 2023-05-31T20:46:32.191 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:urQh4/k39oOVbipfC3p13N5ZBU0= 2023-05-31T20:46:32.192 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:46:32 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:urQh4/k39oOVbipfC3p13N5ZBU0=', 'Content-Length': '0'} 2023-05-31T20:46:32.194 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:44:50 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ff1ee3130d2475f6-006477b228-1c17-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:46:32 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:46:32.194 DEBUG:boto:path=/pklxkd-59/obj38 2023-05-31T20:46:32.195 DEBUG:boto:auth_path=/pklxkd-59/obj38 2023-05-31T20:46:32.195 DEBUG:boto:path=/pklxkd-59/obj38?versionId=null 2023-05-31T20:46:32.195 DEBUG:boto:auth_path=/pklxkd-59/obj38?versionId=null 2023-05-31T20:46:32.196 DEBUG:boto:Method: GET 2023-05-31T20:46:32.196 DEBUG:boto:Path: /pklxkd-59/obj38?versionId=null 2023-05-31T20:46:32.196 DEBUG:boto:Data: 2023-05-31T20:46:32.197 DEBUG:boto:Headers: {} 2023-05-31T20:46:32.197 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:46:32.197 DEBUG:boto:Port: 8000 2023-05-31T20:46:32.198 DEBUG:boto:Params: {} 2023-05-31T20:46:32.198 DEBUG:boto:Token: None 2023-05-31T20:46:32.199 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:46:32 GMT /pklxkd-59/obj38?versionId=null 2023-05-31T20:46:32.199 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:urQh4/k39oOVbipfC3p13N5ZBU0= 2023-05-31T20:46:32.199 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:46:32 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:urQh4/k39oOVbipfC3p13N5ZBU0=', 'Content-Length': '0'} 2023-05-31T20:46:32.201 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:44:50 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b2552ac621847358-006477b228-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:46:32 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:46:32.202 DEBUG:boto:path=/pklxkd-59/ 2023-05-31T20:46:32.202 DEBUG:boto:auth_path=/pklxkd-59/ 2023-05-31T20:46:32.203 DEBUG:boto:path=/pklxkd-59/?prefix=obj38 2023-05-31T20:46:32.203 DEBUG:boto:auth_path=/pklxkd-59/?prefix=obj38 2023-05-31T20:46:32.203 DEBUG:boto:Method: GET 2023-05-31T20:46:32.204 DEBUG:boto:Path: /pklxkd-59/?prefix=obj38 2023-05-31T20:46:32.204 DEBUG:boto:Data: 2023-05-31T20:46:32.204 DEBUG:boto:Headers: {} 2023-05-31T20:46:32.205 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:46:32.205 DEBUG:boto:Port: 8001 2023-05-31T20:46:32.205 DEBUG:boto:Params: {} 2023-05-31T20:46:32.206 DEBUG:boto:Token: None 2023-05-31T20:46:32.206 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:46:32 GMT /pklxkd-59/ 2023-05-31T20:46:32.207 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:3Lv9IKKSTFx3S8X7r8veHGM3xUY= 2023-05-31T20:46:32.207 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:46:32 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:3Lv9IKKSTFx3S8X7r8veHGM3xUY=', 'Content-Length': '0'} 2023-05-31T20:46:32.209 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000493f691d9f1fccc1-006477b228-1c17-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:46:32 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:46:32.210 DEBUG:boto:b'pklxkd-59obj381000falseobj382023-05-31T20:44:50.610Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:46:32.210 DEBUG:boto:path=/pklxkd-59/ 2023-05-31T20:46:32.211 DEBUG:boto:auth_path=/pklxkd-59/ 2023-05-31T20:46:32.211 DEBUG:boto:path=/pklxkd-59/?prefix=obj38 2023-05-31T20:46:32.211 DEBUG:boto:auth_path=/pklxkd-59/?prefix=obj38 2023-05-31T20:46:32.212 DEBUG:boto:Method: GET 2023-05-31T20:46:32.212 DEBUG:boto:Path: /pklxkd-59/?prefix=obj38 2023-05-31T20:46:32.213 DEBUG:boto:Data: 2023-05-31T20:46:32.213 DEBUG:boto:Headers: {} 2023-05-31T20:46:32.213 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:46:32.214 DEBUG:boto:Port: 8000 2023-05-31T20:46:32.214 DEBUG:boto:Params: {} 2023-05-31T20:46:32.214 DEBUG:boto:Token: None 2023-05-31T20:46:32.215 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:46:32 GMT /pklxkd-59/ 2023-05-31T20:46:32.215 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:3Lv9IKKSTFx3S8X7r8veHGM3xUY= 2023-05-31T20:46:32.216 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:46:32 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:3Lv9IKKSTFx3S8X7r8veHGM3xUY=', 'Content-Length': '0'} 2023-05-31T20:46:32.218 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000627fdfc7e87b7371-006477b228-1630-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:46:32 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:46:32.219 DEBUG:boto:b'pklxkd-59obj381000falseobj382023-05-31T20:44:50.610Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:46:32.220 DEBUG:boto:path=/pklxkd-59/obj38 2023-05-31T20:46:32.220 DEBUG:boto:auth_path=/pklxkd-59/obj38 2023-05-31T20:46:32.220 DEBUG:boto:Method: HEAD 2023-05-31T20:46:32.221 DEBUG:boto:Path: /pklxkd-59/obj38 2023-05-31T20:46:32.221 DEBUG:boto:Data: 2023-05-31T20:46:32.221 DEBUG:boto:Headers: {} 2023-05-31T20:46:32.222 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:46:32.222 DEBUG:boto:Port: 8001 2023-05-31T20:46:32.222 DEBUG:boto:Params: {} 2023-05-31T20:46:32.223 DEBUG:boto:Token: None 2023-05-31T20:46:32.223 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:46:32 GMT /pklxkd-59/obj38 2023-05-31T20:46:32.224 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:XrveCqTkPQDof5LmSwefdxJEjsM= 2023-05-31T20:46:32.224 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:46:32 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:XrveCqTkPQDof5LmSwefdxJEjsM=', 'Content-Length': '0'} 2023-05-31T20:46:32.226 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:44:50 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f772ec55562be12b-006477b228-1c17-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:46:32 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:46:32.226 DEBUG:boto:path=/pklxkd-59/obj38 2023-05-31T20:46:32.227 DEBUG:boto:auth_path=/pklxkd-59/obj38 2023-05-31T20:46:32.227 DEBUG:boto:Method: HEAD 2023-05-31T20:46:32.227 DEBUG:boto:Path: /pklxkd-59/obj38 2023-05-31T20:46:32.228 DEBUG:boto:Data: 2023-05-31T20:46:32.228 DEBUG:boto:Headers: {} 2023-05-31T20:46:32.228 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:46:32.229 DEBUG:boto:Port: 8000 2023-05-31T20:46:32.229 DEBUG:boto:Params: {} 2023-05-31T20:46:32.229 DEBUG:boto:Token: None 2023-05-31T20:46:32.230 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:46:32 GMT /pklxkd-59/obj38 2023-05-31T20:46:32.230 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:XrveCqTkPQDof5LmSwefdxJEjsM= 2023-05-31T20:46:32.231 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:46:32 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:XrveCqTkPQDof5LmSwefdxJEjsM=', 'Content-Length': '0'} 2023-05-31T20:46:32.232 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:44:50 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f77bb10ad55d9e2b-006477b228-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:46:32 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:46:32.233 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj38 2023-05-31T20:46:32.233 DEBUG:boto:path=/pklxkd-59/obj38 2023-05-31T20:46:32.233 DEBUG:boto:auth_path=/pklxkd-59/obj38 2023-05-31T20:46:32.234 DEBUG:boto:Method: GET 2023-05-31T20:46:32.234 DEBUG:boto:Path: /pklxkd-59/obj38 2023-05-31T20:46:32.235 DEBUG:boto:Data: 2023-05-31T20:46:32.235 DEBUG:boto:Headers: {} 2023-05-31T20:46:32.235 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:46:32.236 DEBUG:boto:Port: 8001 2023-05-31T20:46:32.236 DEBUG:boto:Params: {} 2023-05-31T20:46:32.236 DEBUG:boto:Token: None 2023-05-31T20:46:32.237 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:46:32 GMT /pklxkd-59/obj38 2023-05-31T20:46:32.237 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:0wCDvxdfAtNvohO/St1Lqydye1E= 2023-05-31T20:46:32.238 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:46:32 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:0wCDvxdfAtNvohO/St1Lqydye1E=', 'Content-Length': '0'} 2023-05-31T20:46:32.239 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:44:50 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009e466765474541bb-006477b228-1c17-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:46:32 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:46:32.240 DEBUG:boto:path=/pklxkd-59/obj38 2023-05-31T20:46:32.240 DEBUG:boto:auth_path=/pklxkd-59/obj38 2023-05-31T20:46:32.241 DEBUG:boto:Method: GET 2023-05-31T20:46:32.241 DEBUG:boto:Path: /pklxkd-59/obj38 2023-05-31T20:46:32.242 DEBUG:boto:Data: 2023-05-31T20:46:32.242 DEBUG:boto:Headers: {} 2023-05-31T20:46:32.242 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:46:32.243 DEBUG:boto:Port: 8000 2023-05-31T20:46:32.243 DEBUG:boto:Params: {} 2023-05-31T20:46:32.243 DEBUG:boto:Token: None 2023-05-31T20:46:32.244 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:46:32 GMT /pklxkd-59/obj38 2023-05-31T20:46:32.244 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:0wCDvxdfAtNvohO/St1Lqydye1E= 2023-05-31T20:46:32.245 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:46:32 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:0wCDvxdfAtNvohO/St1Lqydye1E=', 'Content-Length': '0'} 2023-05-31T20:46:32.246 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:44:50 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c1cc301281a2b41f-006477b228-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:46:32 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:46:32.247 DEBUG:boto:path=/pklxkd-59/ 2023-05-31T20:46:32.247 DEBUG:boto:auth_path=/pklxkd-59/ 2023-05-31T20:46:32.248 DEBUG:boto:path=/pklxkd-59/?prefix=obj38 2023-05-31T20:46:32.248 DEBUG:boto:auth_path=/pklxkd-59/?prefix=obj38 2023-05-31T20:46:32.248 DEBUG:boto:Method: GET 2023-05-31T20:46:32.249 DEBUG:boto:Path: /pklxkd-59/?prefix=obj38 2023-05-31T20:46:32.249 DEBUG:boto:Data: 2023-05-31T20:46:32.249 DEBUG:boto:Headers: {} 2023-05-31T20:46:32.250 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:46:32.250 DEBUG:boto:Port: 8001 2023-05-31T20:46:32.251 DEBUG:boto:Params: {} 2023-05-31T20:46:32.251 DEBUG:boto:Token: None 2023-05-31T20:46:32.251 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:46:32 GMT /pklxkd-59/ 2023-05-31T20:46:32.252 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:3Lv9IKKSTFx3S8X7r8veHGM3xUY= 2023-05-31T20:46:32.252 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:46:32 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:3Lv9IKKSTFx3S8X7r8veHGM3xUY=', 'Content-Length': '0'} 2023-05-31T20:46:32.255 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b6163ed68498b577-006477b228-1c17-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:46:32 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:46:32.255 DEBUG:boto:b'pklxkd-59obj381000falseobj382023-05-31T20:44:50.610Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:46:32.256 DEBUG:boto:path=/pklxkd-59/ 2023-05-31T20:46:32.256 DEBUG:boto:auth_path=/pklxkd-59/ 2023-05-31T20:46:32.256 DEBUG:boto:path=/pklxkd-59/?prefix=obj38 2023-05-31T20:46:32.257 DEBUG:boto:auth_path=/pklxkd-59/?prefix=obj38 2023-05-31T20:46:32.257 DEBUG:boto:Method: GET 2023-05-31T20:46:32.258 DEBUG:boto:Path: /pklxkd-59/?prefix=obj38 2023-05-31T20:46:32.258 DEBUG:boto:Data: 2023-05-31T20:46:32.258 DEBUG:boto:Headers: {} 2023-05-31T20:46:32.259 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:46:32.259 DEBUG:boto:Port: 8000 2023-05-31T20:46:32.259 DEBUG:boto:Params: {} 2023-05-31T20:46:32.260 DEBUG:boto:Token: None 2023-05-31T20:46:32.260 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:46:32 GMT /pklxkd-59/ 2023-05-31T20:46:32.261 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:3Lv9IKKSTFx3S8X7r8veHGM3xUY= 2023-05-31T20:46:32.261 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:46:32 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:3Lv9IKKSTFx3S8X7r8veHGM3xUY=', 'Content-Length': '0'} 2023-05-31T20:46:32.264 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000322cc16367e68806-006477b228-1630-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:46:32 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:46:32.265 DEBUG:boto:b'pklxkd-59obj381000falseobj382023-05-31T20:44:50.610Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:46:32.265 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj44 2023-05-31T20:46:32.265 DEBUG:boto:path=/pklxkd-59/obj44 2023-05-31T20:46:32.266 DEBUG:boto:auth_path=/pklxkd-59/obj44 2023-05-31T20:46:32.266 DEBUG:boto:path=/pklxkd-59/obj44?versionId=null 2023-05-31T20:46:32.267 DEBUG:boto:auth_path=/pklxkd-59/obj44?versionId=null 2023-05-31T20:46:32.267 DEBUG:boto:Method: GET 2023-05-31T20:46:32.267 DEBUG:boto:Path: /pklxkd-59/obj44?versionId=null 2023-05-31T20:46:32.268 DEBUG:boto:Data: 2023-05-31T20:46:32.268 DEBUG:boto:Headers: {} 2023-05-31T20:46:32.268 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:46:32.269 DEBUG:boto:Port: 8001 2023-05-31T20:46:32.269 DEBUG:boto:Params: {} 2023-05-31T20:46:32.269 DEBUG:boto:Token: None 2023-05-31T20:46:32.270 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:46:32 GMT /pklxkd-59/obj44?versionId=null 2023-05-31T20:46:32.270 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:O7UsXxonVPLjuWVel3HYGgb6UNU= 2023-05-31T20:46:32.271 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:46:32 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:O7UsXxonVPLjuWVel3HYGgb6UNU=', 'Content-Length': '0'} 2023-05-31T20:46:32.272 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:39:26 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000072fe4514dadb7544-006477b228-1c17-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:46:32 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:46:32.273 DEBUG:boto:path=/pklxkd-59/obj44 2023-05-31T20:46:32.273 DEBUG:boto:auth_path=/pklxkd-59/obj44 2023-05-31T20:46:32.274 DEBUG:boto:path=/pklxkd-59/obj44?versionId=null 2023-05-31T20:46:32.274 DEBUG:boto:auth_path=/pklxkd-59/obj44?versionId=null 2023-05-31T20:46:32.274 DEBUG:boto:Method: GET 2023-05-31T20:46:32.275 DEBUG:boto:Path: /pklxkd-59/obj44?versionId=null 2023-05-31T20:46:32.275 DEBUG:boto:Data: 2023-05-31T20:46:32.275 DEBUG:boto:Headers: {} 2023-05-31T20:46:32.276 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:46:32.276 DEBUG:boto:Port: 8000 2023-05-31T20:46:32.277 DEBUG:boto:Params: {} 2023-05-31T20:46:32.277 DEBUG:boto:Token: None 2023-05-31T20:46:32.277 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:46:32 GMT /pklxkd-59/obj44?versionId=null 2023-05-31T20:46:32.278 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:O7UsXxonVPLjuWVel3HYGgb6UNU= 2023-05-31T20:46:32.278 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:46:32 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:O7UsXxonVPLjuWVel3HYGgb6UNU=', 'Content-Length': '0'} 2023-05-31T20:46:32.280 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:39:26 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005c4de5508adde00b-006477b228-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:46:32 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:46:32.320 DEBUG:boto:path=/pklxkd-59/obj44 2023-05-31T20:46:32.321 DEBUG:boto:auth_path=/pklxkd-59/obj44 2023-05-31T20:46:32.321 DEBUG:boto:path=/pklxkd-59/obj44?versionId=null 2023-05-31T20:46:32.321 DEBUG:boto:auth_path=/pklxkd-59/obj44?versionId=null 2023-05-31T20:46:32.322 DEBUG:boto:Method: HEAD 2023-05-31T20:46:32.322 DEBUG:boto:Path: /pklxkd-59/obj44?versionId=null 2023-05-31T20:46:32.322 DEBUG:boto:Data: 2023-05-31T20:46:32.323 DEBUG:boto:Headers: {} 2023-05-31T20:46:32.323 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:46:32.323 DEBUG:boto:Port: 8001 2023-05-31T20:46:32.324 DEBUG:boto:Params: {} 2023-05-31T20:46:32.324 DEBUG:boto:Token: None 2023-05-31T20:46:32.324 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:46:32 GMT /pklxkd-59/obj44?versionId=null 2023-05-31T20:46:32.325 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:IZA1aBHSdRUFBTK/Q8DcvoQs8sw= 2023-05-31T20:46:32.325 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:46:32 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:IZA1aBHSdRUFBTK/Q8DcvoQs8sw=', 'Content-Length': '0'} 2023-05-31T20:46:32.327 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:39:26 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007756006b0114a294-006477b228-1c17-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:46:32 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:46:32.328 DEBUG:boto:path=/pklxkd-59/obj44 2023-05-31T20:46:32.328 DEBUG:boto:auth_path=/pklxkd-59/obj44 2023-05-31T20:46:32.328 DEBUG:boto:path=/pklxkd-59/obj44?versionId=null 2023-05-31T20:46:32.329 DEBUG:boto:auth_path=/pklxkd-59/obj44?versionId=null 2023-05-31T20:46:32.329 DEBUG:boto:Method: HEAD 2023-05-31T20:46:32.329 DEBUG:boto:Path: /pklxkd-59/obj44?versionId=null 2023-05-31T20:46:32.330 DEBUG:boto:Data: 2023-05-31T20:46:32.330 DEBUG:boto:Headers: {} 2023-05-31T20:46:32.330 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:46:32.331 DEBUG:boto:Port: 8000 2023-05-31T20:46:32.331 DEBUG:boto:Params: {} 2023-05-31T20:46:32.331 DEBUG:boto:Token: None 2023-05-31T20:46:32.332 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:46:32 GMT /pklxkd-59/obj44?versionId=null 2023-05-31T20:46:32.332 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:IZA1aBHSdRUFBTK/Q8DcvoQs8sw= 2023-05-31T20:46:32.332 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:46:32 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:IZA1aBHSdRUFBTK/Q8DcvoQs8sw=', 'Content-Length': '0'} 2023-05-31T20:46:32.334 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:39:26 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000035229cd812482f5-006477b228-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:46:32 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:46:32.335 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj44 2023-05-31T20:46:32.335 DEBUG:boto:path=/pklxkd-59/obj44 2023-05-31T20:46:32.336 DEBUG:boto:auth_path=/pklxkd-59/obj44 2023-05-31T20:46:32.336 DEBUG:boto:path=/pklxkd-59/obj44?versionId=null 2023-05-31T20:46:32.336 DEBUG:boto:auth_path=/pklxkd-59/obj44?versionId=null 2023-05-31T20:46:32.337 DEBUG:boto:Method: GET 2023-05-31T20:46:32.337 DEBUG:boto:Path: /pklxkd-59/obj44?versionId=null 2023-05-31T20:46:32.337 DEBUG:boto:Data: 2023-05-31T20:46:32.338 DEBUG:boto:Headers: {} 2023-05-31T20:46:32.338 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:46:32.338 DEBUG:boto:Port: 8001 2023-05-31T20:46:32.339 DEBUG:boto:Params: {} 2023-05-31T20:46:32.339 DEBUG:boto:Token: None 2023-05-31T20:46:32.340 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:46:32 GMT /pklxkd-59/obj44?versionId=null 2023-05-31T20:46:32.340 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:O7UsXxonVPLjuWVel3HYGgb6UNU= 2023-05-31T20:46:32.340 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:46:32 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:O7UsXxonVPLjuWVel3HYGgb6UNU=', 'Content-Length': '0'} 2023-05-31T20:46:32.342 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:39:26 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000cc3429ec388b442a-006477b228-1c17-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:46:32 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:46:32.343 DEBUG:boto:path=/pklxkd-59/obj44 2023-05-31T20:46:32.343 DEBUG:boto:auth_path=/pklxkd-59/obj44 2023-05-31T20:46:32.344 DEBUG:boto:path=/pklxkd-59/obj44?versionId=null 2023-05-31T20:46:32.344 DEBUG:boto:auth_path=/pklxkd-59/obj44?versionId=null 2023-05-31T20:46:32.344 DEBUG:boto:Method: GET 2023-05-31T20:46:32.345 DEBUG:boto:Path: /pklxkd-59/obj44?versionId=null 2023-05-31T20:46:32.345 DEBUG:boto:Data: 2023-05-31T20:46:32.345 DEBUG:boto:Headers: {} 2023-05-31T20:46:32.346 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:46:32.346 DEBUG:boto:Port: 8000 2023-05-31T20:46:32.346 DEBUG:boto:Params: {} 2023-05-31T20:46:32.347 DEBUG:boto:Token: None 2023-05-31T20:46:32.347 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:46:32 GMT /pklxkd-59/obj44?versionId=null 2023-05-31T20:46:32.348 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:O7UsXxonVPLjuWVel3HYGgb6UNU= 2023-05-31T20:46:32.348 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:46:32 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:O7UsXxonVPLjuWVel3HYGgb6UNU=', 'Content-Length': '0'} 2023-05-31T20:46:32.350 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:39:26 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e2dce2361b9ec072-006477b228-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:46:32 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:46:32.351 DEBUG:boto:path=/pklxkd-59/ 2023-05-31T20:46:32.351 DEBUG:boto:auth_path=/pklxkd-59/ 2023-05-31T20:46:32.351 DEBUG:boto:path=/pklxkd-59/?prefix=obj44 2023-05-31T20:46:32.352 DEBUG:boto:auth_path=/pklxkd-59/?prefix=obj44 2023-05-31T20:46:32.352 DEBUG:boto:Method: GET 2023-05-31T20:46:32.353 DEBUG:boto:Path: /pklxkd-59/?prefix=obj44 2023-05-31T20:46:32.353 DEBUG:boto:Data: 2023-05-31T20:46:32.353 DEBUG:boto:Headers: {} 2023-05-31T20:46:32.354 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:46:32.354 DEBUG:boto:Port: 8001 2023-05-31T20:46:32.354 DEBUG:boto:Params: {} 2023-05-31T20:46:32.355 DEBUG:boto:Token: None 2023-05-31T20:46:32.355 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:46:32 GMT /pklxkd-59/ 2023-05-31T20:46:32.356 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:3Lv9IKKSTFx3S8X7r8veHGM3xUY= 2023-05-31T20:46:32.356 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:46:32 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:3Lv9IKKSTFx3S8X7r8veHGM3xUY=', 'Content-Length': '0'} 2023-05-31T20:46:32.358 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c68c02d689eff1a6-006477b228-1c17-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:46:32 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:46:32.359 DEBUG:boto:b'pklxkd-59obj441000falseobj442023-05-31T20:39:26.565Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:46:32.359 DEBUG:boto:path=/pklxkd-59/ 2023-05-31T20:46:32.360 DEBUG:boto:auth_path=/pklxkd-59/ 2023-05-31T20:46:32.360 DEBUG:boto:path=/pklxkd-59/?prefix=obj44 2023-05-31T20:46:32.360 DEBUG:boto:auth_path=/pklxkd-59/?prefix=obj44 2023-05-31T20:46:32.361 DEBUG:boto:Method: GET 2023-05-31T20:46:32.361 DEBUG:boto:Path: /pklxkd-59/?prefix=obj44 2023-05-31T20:46:32.361 DEBUG:boto:Data: 2023-05-31T20:46:32.362 DEBUG:boto:Headers: {} 2023-05-31T20:46:32.362 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:46:32.363 DEBUG:boto:Port: 8000 2023-05-31T20:46:32.363 DEBUG:boto:Params: {} 2023-05-31T20:46:32.363 DEBUG:boto:Token: None 2023-05-31T20:46:32.364 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:46:32 GMT /pklxkd-59/ 2023-05-31T20:46:32.364 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:3Lv9IKKSTFx3S8X7r8veHGM3xUY= 2023-05-31T20:46:32.365 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:46:32 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:3Lv9IKKSTFx3S8X7r8veHGM3xUY=', 'Content-Length': '0'} 2023-05-31T20:46:32.367 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c096553319110fc6-006477b228-1630-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:46:32 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:46:32.368 DEBUG:boto:b'pklxkd-59obj441000falseobj442023-05-31T20:39:26.565Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:46:32.368 DEBUG:boto:path=/pklxkd-59/obj44 2023-05-31T20:46:32.369 DEBUG:boto:auth_path=/pklxkd-59/obj44 2023-05-31T20:46:32.369 DEBUG:boto:Method: HEAD 2023-05-31T20:46:32.370 DEBUG:boto:Path: /pklxkd-59/obj44 2023-05-31T20:46:32.370 DEBUG:boto:Data: 2023-05-31T20:46:32.370 DEBUG:boto:Headers: {} 2023-05-31T20:46:32.371 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:46:32.371 DEBUG:boto:Port: 8001 2023-05-31T20:46:32.371 DEBUG:boto:Params: {} 2023-05-31T20:46:32.372 DEBUG:boto:Token: None 2023-05-31T20:46:32.372 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:46:32 GMT /pklxkd-59/obj44 2023-05-31T20:46:32.372 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:8HS4ZWIQYkPKN3Vp+8MeW84MVIY= 2023-05-31T20:46:32.373 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:46:32 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:8HS4ZWIQYkPKN3Vp+8MeW84MVIY=', 'Content-Length': '0'} 2023-05-31T20:46:32.374 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:39:26 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000032f181d64f41030e-006477b228-1c17-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:46:32 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:46:32.375 DEBUG:boto:path=/pklxkd-59/obj44 2023-05-31T20:46:32.375 DEBUG:boto:auth_path=/pklxkd-59/obj44 2023-05-31T20:46:32.376 DEBUG:boto:Method: HEAD 2023-05-31T20:46:32.376 DEBUG:boto:Path: /pklxkd-59/obj44 2023-05-31T20:46:32.376 DEBUG:boto:Data: 2023-05-31T20:46:32.377 DEBUG:boto:Headers: {} 2023-05-31T20:46:32.377 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:46:32.377 DEBUG:boto:Port: 8000 2023-05-31T20:46:32.378 DEBUG:boto:Params: {} 2023-05-31T20:46:32.378 DEBUG:boto:Token: None 2023-05-31T20:46:32.379 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:46:32 GMT /pklxkd-59/obj44 2023-05-31T20:46:32.379 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:8HS4ZWIQYkPKN3Vp+8MeW84MVIY= 2023-05-31T20:46:32.379 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:46:32 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:8HS4ZWIQYkPKN3Vp+8MeW84MVIY=', 'Content-Length': '0'} 2023-05-31T20:46:32.381 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:39:26 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000095db96e02fb0e3a7-006477b228-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:46:32 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:46:32.381 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj44 2023-05-31T20:46:32.382 DEBUG:boto:path=/pklxkd-59/obj44 2023-05-31T20:46:32.382 DEBUG:boto:auth_path=/pklxkd-59/obj44 2023-05-31T20:46:32.382 DEBUG:boto:Method: GET 2023-05-31T20:46:32.383 DEBUG:boto:Path: /pklxkd-59/obj44 2023-05-31T20:46:32.383 DEBUG:boto:Data: 2023-05-31T20:46:32.383 DEBUG:boto:Headers: {} 2023-05-31T20:46:32.384 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:46:32.384 DEBUG:boto:Port: 8001 2023-05-31T20:46:32.385 DEBUG:boto:Params: {} 2023-05-31T20:46:32.385 DEBUG:boto:Token: None 2023-05-31T20:46:32.385 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:46:32 GMT /pklxkd-59/obj44 2023-05-31T20:46:32.386 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:7fhzMySsZSu/siwAMGR5W1PsUU0= 2023-05-31T20:46:32.386 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:46:32 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:7fhzMySsZSu/siwAMGR5W1PsUU0=', 'Content-Length': '0'} 2023-05-31T20:46:32.388 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:39:26 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b002a69dbc0983c1-006477b228-1c17-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:46:32 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:46:32.388 DEBUG:boto:path=/pklxkd-59/obj44 2023-05-31T20:46:32.389 DEBUG:boto:auth_path=/pklxkd-59/obj44 2023-05-31T20:46:32.389 DEBUG:boto:Method: GET 2023-05-31T20:46:32.389 DEBUG:boto:Path: /pklxkd-59/obj44 2023-05-31T20:46:32.390 DEBUG:boto:Data: 2023-05-31T20:46:32.390 DEBUG:boto:Headers: {} 2023-05-31T20:46:32.391 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:46:32.391 DEBUG:boto:Port: 8000 2023-05-31T20:46:32.391 DEBUG:boto:Params: {} 2023-05-31T20:46:32.392 DEBUG:boto:Token: None 2023-05-31T20:46:32.392 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:46:32 GMT /pklxkd-59/obj44 2023-05-31T20:46:32.392 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:7fhzMySsZSu/siwAMGR5W1PsUU0= 2023-05-31T20:46:32.393 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:46:32 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:7fhzMySsZSu/siwAMGR5W1PsUU0=', 'Content-Length': '0'} 2023-05-31T20:46:32.394 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:39:26 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002f06d619178961c2-006477b228-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:46:32 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:46:32.395 DEBUG:boto:path=/pklxkd-59/ 2023-05-31T20:46:32.395 DEBUG:boto:auth_path=/pklxkd-59/ 2023-05-31T20:46:32.396 DEBUG:boto:path=/pklxkd-59/?prefix=obj44 2023-05-31T20:46:32.396 DEBUG:boto:auth_path=/pklxkd-59/?prefix=obj44 2023-05-31T20:46:32.397 DEBUG:boto:Method: GET 2023-05-31T20:46:32.397 DEBUG:boto:Path: /pklxkd-59/?prefix=obj44 2023-05-31T20:46:32.397 DEBUG:boto:Data: 2023-05-31T20:46:32.398 DEBUG:boto:Headers: {} 2023-05-31T20:46:32.398 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:46:32.398 DEBUG:boto:Port: 8001 2023-05-31T20:46:32.399 DEBUG:boto:Params: {} 2023-05-31T20:46:32.399 DEBUG:boto:Token: None 2023-05-31T20:46:32.400 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:46:32 GMT /pklxkd-59/ 2023-05-31T20:46:32.400 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:3Lv9IKKSTFx3S8X7r8veHGM3xUY= 2023-05-31T20:46:32.400 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:46:32 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:3Lv9IKKSTFx3S8X7r8veHGM3xUY=', 'Content-Length': '0'} 2023-05-31T20:46:32.403 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000031d0dea848514c9a-006477b228-1c17-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:46:32 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:46:32.403 DEBUG:boto:b'pklxkd-59obj441000falseobj442023-05-31T20:39:26.565Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:46:32.404 DEBUG:boto:path=/pklxkd-59/ 2023-05-31T20:46:32.404 DEBUG:boto:auth_path=/pklxkd-59/ 2023-05-31T20:46:32.404 DEBUG:boto:path=/pklxkd-59/?prefix=obj44 2023-05-31T20:46:32.405 DEBUG:boto:auth_path=/pklxkd-59/?prefix=obj44 2023-05-31T20:46:32.405 DEBUG:boto:Method: GET 2023-05-31T20:46:32.406 DEBUG:boto:Path: /pklxkd-59/?prefix=obj44 2023-05-31T20:46:32.406 DEBUG:boto:Data: 2023-05-31T20:46:32.406 DEBUG:boto:Headers: {} 2023-05-31T20:46:32.407 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:46:32.407 DEBUG:boto:Port: 8000 2023-05-31T20:46:32.407 DEBUG:boto:Params: {} 2023-05-31T20:46:32.408 DEBUG:boto:Token: None 2023-05-31T20:46:32.408 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:46:32 GMT /pklxkd-59/ 2023-05-31T20:46:32.409 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:3Lv9IKKSTFx3S8X7r8veHGM3xUY= 2023-05-31T20:46:32.409 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:46:32 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:3Lv9IKKSTFx3S8X7r8veHGM3xUY=', 'Content-Length': '0'} 2023-05-31T20:46:32.412 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000bba9f5aab7da521d-006477b228-1630-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:46:32 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:46:32.412 DEBUG:boto:b'pklxkd-59obj441000falseobj442023-05-31T20:39:26.565Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:46:32.413 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj51 2023-05-31T20:46:32.413 DEBUG:boto:path=/pklxkd-59/obj51 2023-05-31T20:46:32.414 DEBUG:boto:auth_path=/pklxkd-59/obj51 2023-05-31T20:46:32.414 DEBUG:boto:path=/pklxkd-59/obj51?versionId=null 2023-05-31T20:46:32.414 DEBUG:boto:auth_path=/pklxkd-59/obj51?versionId=null 2023-05-31T20:46:32.415 DEBUG:boto:Method: GET 2023-05-31T20:46:32.415 DEBUG:boto:Path: /pklxkd-59/obj51?versionId=null 2023-05-31T20:46:32.416 DEBUG:boto:Data: 2023-05-31T20:46:32.416 DEBUG:boto:Headers: {} 2023-05-31T20:46:32.416 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:46:32.417 DEBUG:boto:Port: 8001 2023-05-31T20:46:32.417 DEBUG:boto:Params: {} 2023-05-31T20:46:32.417 DEBUG:boto:Token: None 2023-05-31T20:46:32.418 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:46:32 GMT /pklxkd-59/obj51?versionId=null 2023-05-31T20:46:32.418 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:NIf3eIdoNdsnEZal4tJZw+jGepY= 2023-05-31T20:46:32.419 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:46:32 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:NIf3eIdoNdsnEZal4tJZw+jGepY=', 'Content-Length': '0'} 2023-05-31T20:46:32.421 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:43:08 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000397edf3c0ae8f99d-006477b228-1c17-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:46:32 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:46:32.422 DEBUG:boto:path=/pklxkd-59/obj51 2023-05-31T20:46:32.422 DEBUG:boto:auth_path=/pklxkd-59/obj51 2023-05-31T20:46:32.423 DEBUG:boto:path=/pklxkd-59/obj51?versionId=null 2023-05-31T20:46:32.423 DEBUG:boto:auth_path=/pklxkd-59/obj51?versionId=null 2023-05-31T20:46:32.424 DEBUG:boto:Method: GET 2023-05-31T20:46:32.424 DEBUG:boto:Path: /pklxkd-59/obj51?versionId=null 2023-05-31T20:46:32.424 DEBUG:boto:Data: 2023-05-31T20:46:32.425 DEBUG:boto:Headers: {} 2023-05-31T20:46:32.425 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:46:32.426 DEBUG:boto:Port: 8000 2023-05-31T20:46:32.426 DEBUG:boto:Params: {} 2023-05-31T20:46:32.427 DEBUG:boto:Token: None 2023-05-31T20:46:32.427 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:46:32 GMT /pklxkd-59/obj51?versionId=null 2023-05-31T20:46:32.427 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:NIf3eIdoNdsnEZal4tJZw+jGepY= 2023-05-31T20:46:32.428 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:46:32 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:NIf3eIdoNdsnEZal4tJZw+jGepY=', 'Content-Length': '0'} 2023-05-31T20:46:32.430 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:43:08 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d9bd7894e57c89e7-006477b228-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:46:32 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:46:32.430 DEBUG:boto:path=/pklxkd-59/obj51 2023-05-31T20:46:32.431 DEBUG:boto:auth_path=/pklxkd-59/obj51 2023-05-31T20:46:32.431 DEBUG:boto:path=/pklxkd-59/obj51?versionId=null 2023-05-31T20:46:32.431 DEBUG:boto:auth_path=/pklxkd-59/obj51?versionId=null 2023-05-31T20:46:32.432 DEBUG:boto:Method: HEAD 2023-05-31T20:46:32.432 DEBUG:boto:Path: /pklxkd-59/obj51?versionId=null 2023-05-31T20:46:32.433 DEBUG:boto:Data: 2023-05-31T20:46:32.433 DEBUG:boto:Headers: {} 2023-05-31T20:46:32.434 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:46:32.434 DEBUG:boto:Port: 8001 2023-05-31T20:46:32.434 DEBUG:boto:Params: {} 2023-05-31T20:46:32.435 DEBUG:boto:Token: None 2023-05-31T20:46:32.435 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:46:32 GMT /pklxkd-59/obj51?versionId=null 2023-05-31T20:46:32.436 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:tBg3VV/4ecAQbd76k0aZDebzpsE= 2023-05-31T20:46:32.436 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:46:32 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:tBg3VV/4ecAQbd76k0aZDebzpsE=', 'Content-Length': '0'} 2023-05-31T20:46:32.438 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:43:08 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003837a1afb7d289d6-006477b228-1c17-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:46:32 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:46:32.439 DEBUG:boto:path=/pklxkd-59/obj51 2023-05-31T20:46:32.439 DEBUG:boto:auth_path=/pklxkd-59/obj51 2023-05-31T20:46:32.440 DEBUG:boto:path=/pklxkd-59/obj51?versionId=null 2023-05-31T20:46:32.440 DEBUG:boto:auth_path=/pklxkd-59/obj51?versionId=null 2023-05-31T20:46:32.440 DEBUG:boto:Method: HEAD 2023-05-31T20:46:32.441 DEBUG:boto:Path: /pklxkd-59/obj51?versionId=null 2023-05-31T20:46:32.441 DEBUG:boto:Data: 2023-05-31T20:46:32.441 DEBUG:boto:Headers: {} 2023-05-31T20:46:32.442 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:46:32.442 DEBUG:boto:Port: 8000 2023-05-31T20:46:32.442 DEBUG:boto:Params: {} 2023-05-31T20:46:32.443 DEBUG:boto:Token: None 2023-05-31T20:46:32.443 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:46:32 GMT /pklxkd-59/obj51?versionId=null 2023-05-31T20:46:32.444 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:tBg3VV/4ecAQbd76k0aZDebzpsE= 2023-05-31T20:46:32.444 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:46:32 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:tBg3VV/4ecAQbd76k0aZDebzpsE=', 'Content-Length': '0'} 2023-05-31T20:46:32.446 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:43:08 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000060d305c878f6af5e-006477b228-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:46:32 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:46:32.446 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj51 2023-05-31T20:46:32.447 DEBUG:boto:path=/pklxkd-59/obj51 2023-05-31T20:46:32.447 DEBUG:boto:auth_path=/pklxkd-59/obj51 2023-05-31T20:46:32.448 DEBUG:boto:path=/pklxkd-59/obj51?versionId=null 2023-05-31T20:46:32.448 DEBUG:boto:auth_path=/pklxkd-59/obj51?versionId=null 2023-05-31T20:46:32.448 DEBUG:boto:Method: GET 2023-05-31T20:46:32.449 DEBUG:boto:Path: /pklxkd-59/obj51?versionId=null 2023-05-31T20:46:32.449 DEBUG:boto:Data: 2023-05-31T20:46:32.450 DEBUG:boto:Headers: {} 2023-05-31T20:46:32.450 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:46:32.450 DEBUG:boto:Port: 8001 2023-05-31T20:46:32.451 DEBUG:boto:Params: {} 2023-05-31T20:46:32.451 DEBUG:boto:Token: None 2023-05-31T20:46:32.452 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:46:32 GMT /pklxkd-59/obj51?versionId=null 2023-05-31T20:46:32.452 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:NIf3eIdoNdsnEZal4tJZw+jGepY= 2023-05-31T20:46:32.453 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:46:32 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:NIf3eIdoNdsnEZal4tJZw+jGepY=', 'Content-Length': '0'} 2023-05-31T20:46:32.454 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:43:08 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f677ff8d2679165a-006477b228-1c17-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:46:32 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:46:32.496 DEBUG:boto:path=/pklxkd-59/obj51 2023-05-31T20:46:32.497 DEBUG:boto:auth_path=/pklxkd-59/obj51 2023-05-31T20:46:32.497 DEBUG:boto:path=/pklxkd-59/obj51?versionId=null 2023-05-31T20:46:32.497 DEBUG:boto:auth_path=/pklxkd-59/obj51?versionId=null 2023-05-31T20:46:32.498 DEBUG:boto:Method: GET 2023-05-31T20:46:32.498 DEBUG:boto:Path: /pklxkd-59/obj51?versionId=null 2023-05-31T20:46:32.499 DEBUG:boto:Data: 2023-05-31T20:46:32.499 DEBUG:boto:Headers: {} 2023-05-31T20:46:32.499 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:46:32.500 DEBUG:boto:Port: 8000 2023-05-31T20:46:32.500 DEBUG:boto:Params: {} 2023-05-31T20:46:32.500 DEBUG:boto:Token: None 2023-05-31T20:46:32.501 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:46:32 GMT /pklxkd-59/obj51?versionId=null 2023-05-31T20:46:32.502 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:NIf3eIdoNdsnEZal4tJZw+jGepY= 2023-05-31T20:46:32.502 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:46:32 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:NIf3eIdoNdsnEZal4tJZw+jGepY=', 'Content-Length': '0'} 2023-05-31T20:46:32.504 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:43:08 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d3a00878909b85e4-006477b228-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:46:32 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:46:32.505 DEBUG:boto:path=/pklxkd-59/ 2023-05-31T20:46:32.505 DEBUG:boto:auth_path=/pklxkd-59/ 2023-05-31T20:46:32.505 DEBUG:boto:path=/pklxkd-59/?prefix=obj51 2023-05-31T20:46:32.506 DEBUG:boto:auth_path=/pklxkd-59/?prefix=obj51 2023-05-31T20:46:32.506 DEBUG:boto:Method: GET 2023-05-31T20:46:32.506 DEBUG:boto:Path: /pklxkd-59/?prefix=obj51 2023-05-31T20:46:32.507 DEBUG:boto:Data: 2023-05-31T20:46:32.507 DEBUG:boto:Headers: {} 2023-05-31T20:46:32.507 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:46:32.508 DEBUG:boto:Port: 8001 2023-05-31T20:46:32.508 DEBUG:boto:Params: {} 2023-05-31T20:46:32.509 DEBUG:boto:Token: None 2023-05-31T20:46:32.509 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:46:32 GMT /pklxkd-59/ 2023-05-31T20:46:32.509 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:3Lv9IKKSTFx3S8X7r8veHGM3xUY= 2023-05-31T20:46:32.510 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:46:32 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:3Lv9IKKSTFx3S8X7r8veHGM3xUY=', 'Content-Length': '0'} 2023-05-31T20:46:32.512 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000038f3568c7ba1a7d3-006477b228-1c17-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:46:32 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:46:32.513 DEBUG:boto:b'pklxkd-59obj511000falseobj512023-05-31T20:43:08.011Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:46:32.513 DEBUG:boto:path=/pklxkd-59/ 2023-05-31T20:46:32.514 DEBUG:boto:auth_path=/pklxkd-59/ 2023-05-31T20:46:32.514 DEBUG:boto:path=/pklxkd-59/?prefix=obj51 2023-05-31T20:46:32.514 DEBUG:boto:auth_path=/pklxkd-59/?prefix=obj51 2023-05-31T20:46:32.515 DEBUG:boto:Method: GET 2023-05-31T20:46:32.515 DEBUG:boto:Path: /pklxkd-59/?prefix=obj51 2023-05-31T20:46:32.516 DEBUG:boto:Data: 2023-05-31T20:46:32.516 DEBUG:boto:Headers: {} 2023-05-31T20:46:32.516 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:46:32.517 DEBUG:boto:Port: 8000 2023-05-31T20:46:32.517 DEBUG:boto:Params: {} 2023-05-31T20:46:32.518 DEBUG:boto:Token: None 2023-05-31T20:46:32.518 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:46:32 GMT /pklxkd-59/ 2023-05-31T20:46:32.518 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:3Lv9IKKSTFx3S8X7r8veHGM3xUY= 2023-05-31T20:46:32.519 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:46:32 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:3Lv9IKKSTFx3S8X7r8veHGM3xUY=', 'Content-Length': '0'} 2023-05-31T20:46:32.522 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ee7d0758eaea717d-006477b228-1630-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:46:32 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:46:32.522 DEBUG:boto:b'pklxkd-59obj511000falseobj512023-05-31T20:43:08.011Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:46:32.523 DEBUG:boto:path=/pklxkd-59/obj51 2023-05-31T20:46:32.523 DEBUG:boto:auth_path=/pklxkd-59/obj51 2023-05-31T20:46:32.524 DEBUG:boto:Method: HEAD 2023-05-31T20:46:32.524 DEBUG:boto:Path: /pklxkd-59/obj51 2023-05-31T20:46:32.524 DEBUG:boto:Data: 2023-05-31T20:46:32.525 DEBUG:boto:Headers: {} 2023-05-31T20:46:32.525 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:46:32.526 DEBUG:boto:Port: 8001 2023-05-31T20:46:32.526 DEBUG:boto:Params: {} 2023-05-31T20:46:32.526 DEBUG:boto:Token: None 2023-05-31T20:46:32.527 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:46:32 GMT /pklxkd-59/obj51 2023-05-31T20:46:32.527 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:cY1oyyy1540sIUvYRZrwSYhjsYs= 2023-05-31T20:46:32.528 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:46:32 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:cY1oyyy1540sIUvYRZrwSYhjsYs=', 'Content-Length': '0'} 2023-05-31T20:46:32.529 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:43:08 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c6bf232dc243d000-006477b228-1c17-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:46:32 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:46:32.530 DEBUG:boto:path=/pklxkd-59/obj51 2023-05-31T20:46:32.530 DEBUG:boto:auth_path=/pklxkd-59/obj51 2023-05-31T20:46:32.530 DEBUG:boto:Method: HEAD 2023-05-31T20:46:32.531 DEBUG:boto:Path: /pklxkd-59/obj51 2023-05-31T20:46:32.531 DEBUG:boto:Data: 2023-05-31T20:46:32.531 DEBUG:boto:Headers: {} 2023-05-31T20:46:32.532 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:46:32.532 DEBUG:boto:Port: 8000 2023-05-31T20:46:32.533 DEBUG:boto:Params: {} 2023-05-31T20:46:32.533 DEBUG:boto:Token: None 2023-05-31T20:46:32.533 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:46:32 GMT /pklxkd-59/obj51 2023-05-31T20:46:32.534 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:cY1oyyy1540sIUvYRZrwSYhjsYs= 2023-05-31T20:46:32.534 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:46:32 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:cY1oyyy1540sIUvYRZrwSYhjsYs=', 'Content-Length': '0'} 2023-05-31T20:46:32.536 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:43:08 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000093a331662f75dc76-006477b228-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:46:32 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:46:32.536 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj51 2023-05-31T20:46:32.537 DEBUG:boto:path=/pklxkd-59/obj51 2023-05-31T20:46:32.537 DEBUG:boto:auth_path=/pklxkd-59/obj51 2023-05-31T20:46:32.538 DEBUG:boto:Method: GET 2023-05-31T20:46:32.538 DEBUG:boto:Path: /pklxkd-59/obj51 2023-05-31T20:46:32.538 DEBUG:boto:Data: 2023-05-31T20:46:32.539 DEBUG:boto:Headers: {} 2023-05-31T20:46:32.539 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:46:32.540 DEBUG:boto:Port: 8001 2023-05-31T20:46:32.540 DEBUG:boto:Params: {} 2023-05-31T20:46:32.540 DEBUG:boto:Token: None 2023-05-31T20:46:32.541 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:46:32 GMT /pklxkd-59/obj51 2023-05-31T20:46:32.541 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:uB0RiuF3hYCY9kIFKHr6dzlfPIQ= 2023-05-31T20:46:32.542 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:46:32 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:uB0RiuF3hYCY9kIFKHr6dzlfPIQ=', 'Content-Length': '0'} 2023-05-31T20:46:32.543 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:43:08 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003d4ad0e80087dd3a-006477b228-1c17-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:46:32 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:46:32.544 DEBUG:boto:path=/pklxkd-59/obj51 2023-05-31T20:46:32.544 DEBUG:boto:auth_path=/pklxkd-59/obj51 2023-05-31T20:46:32.545 DEBUG:boto:Method: GET 2023-05-31T20:46:32.545 DEBUG:boto:Path: /pklxkd-59/obj51 2023-05-31T20:46:32.546 DEBUG:boto:Data: 2023-05-31T20:46:32.546 DEBUG:boto:Headers: {} 2023-05-31T20:46:32.546 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:46:32.547 DEBUG:boto:Port: 8000 2023-05-31T20:46:32.547 DEBUG:boto:Params: {} 2023-05-31T20:46:32.548 DEBUG:boto:Token: None 2023-05-31T20:46:32.548 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:46:32 GMT /pklxkd-59/obj51 2023-05-31T20:46:32.548 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:uB0RiuF3hYCY9kIFKHr6dzlfPIQ= 2023-05-31T20:46:32.549 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:46:32 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:uB0RiuF3hYCY9kIFKHr6dzlfPIQ=', 'Content-Length': '0'} 2023-05-31T20:46:32.551 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:43:08 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000933cce4d2d276bd3-006477b228-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:46:32 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:46:32.592 DEBUG:boto:path=/pklxkd-59/ 2023-05-31T20:46:32.593 DEBUG:boto:auth_path=/pklxkd-59/ 2023-05-31T20:46:32.593 DEBUG:boto:path=/pklxkd-59/?prefix=obj51 2023-05-31T20:46:32.593 DEBUG:boto:auth_path=/pklxkd-59/?prefix=obj51 2023-05-31T20:46:32.594 DEBUG:boto:Method: GET 2023-05-31T20:46:32.594 DEBUG:boto:Path: /pklxkd-59/?prefix=obj51 2023-05-31T20:46:32.595 DEBUG:boto:Data: 2023-05-31T20:46:32.595 DEBUG:boto:Headers: {} 2023-05-31T20:46:32.595 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:46:32.596 DEBUG:boto:Port: 8001 2023-05-31T20:46:32.596 DEBUG:boto:Params: {} 2023-05-31T20:46:32.597 DEBUG:boto:Token: None 2023-05-31T20:46:32.597 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:46:32 GMT /pklxkd-59/ 2023-05-31T20:46:32.598 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:3Lv9IKKSTFx3S8X7r8veHGM3xUY= 2023-05-31T20:46:32.598 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:46:32 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:3Lv9IKKSTFx3S8X7r8veHGM3xUY=', 'Content-Length': '0'} 2023-05-31T20:46:32.600 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c885b970648f8a31-006477b228-1c17-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:46:32 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:46:32.601 DEBUG:boto:b'pklxkd-59obj511000falseobj512023-05-31T20:43:08.011Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:46:32.601 DEBUG:boto:path=/pklxkd-59/ 2023-05-31T20:46:32.602 DEBUG:boto:auth_path=/pklxkd-59/ 2023-05-31T20:46:32.602 DEBUG:boto:path=/pklxkd-59/?prefix=obj51 2023-05-31T20:46:32.602 DEBUG:boto:auth_path=/pklxkd-59/?prefix=obj51 2023-05-31T20:46:32.603 DEBUG:boto:Method: GET 2023-05-31T20:46:32.603 DEBUG:boto:Path: /pklxkd-59/?prefix=obj51 2023-05-31T20:46:32.603 DEBUG:boto:Data: 2023-05-31T20:46:32.604 DEBUG:boto:Headers: {} 2023-05-31T20:46:32.604 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:46:32.605 DEBUG:boto:Port: 8000 2023-05-31T20:46:32.605 DEBUG:boto:Params: {} 2023-05-31T20:46:32.605 DEBUG:boto:Token: None 2023-05-31T20:46:32.606 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:46:32 GMT /pklxkd-59/ 2023-05-31T20:46:32.606 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:3Lv9IKKSTFx3S8X7r8veHGM3xUY= 2023-05-31T20:46:32.607 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:46:32 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:3Lv9IKKSTFx3S8X7r8veHGM3xUY=', 'Content-Length': '0'} 2023-05-31T20:46:32.609 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000643f3b125582f845-006477b228-1630-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:46:32 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:46:32.610 DEBUG:boto:b'pklxkd-59obj511000falseobj512023-05-31T20:43:08.011Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:46:32.610 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj55 2023-05-31T20:46:32.611 DEBUG:boto:path=/pklxkd-59/obj55 2023-05-31T20:46:32.611 DEBUG:boto:auth_path=/pklxkd-59/obj55 2023-05-31T20:46:32.612 DEBUG:boto:path=/pklxkd-59/obj55?versionId=null 2023-05-31T20:46:32.612 DEBUG:boto:auth_path=/pklxkd-59/obj55?versionId=null 2023-05-31T20:46:32.612 DEBUG:boto:Method: GET 2023-05-31T20:46:32.613 DEBUG:boto:Path: /pklxkd-59/obj55?versionId=null 2023-05-31T20:46:32.613 DEBUG:boto:Data: 2023-05-31T20:46:32.613 DEBUG:boto:Headers: {} 2023-05-31T20:46:32.614 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:46:32.614 DEBUG:boto:Port: 8001 2023-05-31T20:46:32.615 DEBUG:boto:Params: {} 2023-05-31T20:46:32.615 DEBUG:boto:Token: None 2023-05-31T20:46:32.616 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:46:32 GMT /pklxkd-59/obj55?versionId=null 2023-05-31T20:46:32.616 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:z13Jm+c/UqDdxsaTiXKuWtfTDRs= 2023-05-31T20:46:32.617 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:46:32 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:z13Jm+c/UqDdxsaTiXKuWtfTDRs=', 'Content-Length': '0'} 2023-05-31T20:46:32.618 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:39:26 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000957e1e67c00666f2-006477b228-1c17-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:46:32 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:46:32.660 DEBUG:boto:path=/pklxkd-59/obj55 2023-05-31T20:46:32.661 DEBUG:boto:auth_path=/pklxkd-59/obj55 2023-05-31T20:46:32.661 DEBUG:boto:path=/pklxkd-59/obj55?versionId=null 2023-05-31T20:46:32.662 DEBUG:boto:auth_path=/pklxkd-59/obj55?versionId=null 2023-05-31T20:46:32.662 DEBUG:boto:Method: GET 2023-05-31T20:46:32.662 DEBUG:boto:Path: /pklxkd-59/obj55?versionId=null 2023-05-31T20:46:32.663 DEBUG:boto:Data: 2023-05-31T20:46:32.663 DEBUG:boto:Headers: {} 2023-05-31T20:46:32.663 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:46:32.664 DEBUG:boto:Port: 8000 2023-05-31T20:46:32.664 DEBUG:boto:Params: {} 2023-05-31T20:46:32.665 DEBUG:boto:Token: None 2023-05-31T20:46:32.665 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:46:32 GMT /pklxkd-59/obj55?versionId=null 2023-05-31T20:46:32.666 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:z13Jm+c/UqDdxsaTiXKuWtfTDRs= 2023-05-31T20:46:32.666 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:46:32 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:z13Jm+c/UqDdxsaTiXKuWtfTDRs=', 'Content-Length': '0'} 2023-05-31T20:46:32.668 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:39:26 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c345174ccce83a65-006477b228-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:46:32 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:46:32.668 DEBUG:boto:path=/pklxkd-59/obj55 2023-05-31T20:46:32.669 DEBUG:boto:auth_path=/pklxkd-59/obj55 2023-05-31T20:46:32.669 DEBUG:boto:path=/pklxkd-59/obj55?versionId=null 2023-05-31T20:46:32.670 DEBUG:boto:auth_path=/pklxkd-59/obj55?versionId=null 2023-05-31T20:46:32.670 DEBUG:boto:Method: HEAD 2023-05-31T20:46:32.670 DEBUG:boto:Path: /pklxkd-59/obj55?versionId=null 2023-05-31T20:46:32.671 DEBUG:boto:Data: 2023-05-31T20:46:32.671 DEBUG:boto:Headers: {} 2023-05-31T20:46:32.671 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:46:32.672 DEBUG:boto:Port: 8001 2023-05-31T20:46:32.672 DEBUG:boto:Params: {} 2023-05-31T20:46:32.672 DEBUG:boto:Token: None 2023-05-31T20:46:32.673 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:46:32 GMT /pklxkd-59/obj55?versionId=null 2023-05-31T20:46:32.673 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:ILdo1ljqMDdbnoA/6APwYcM6q/0= 2023-05-31T20:46:32.674 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:46:32 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:ILdo1ljqMDdbnoA/6APwYcM6q/0=', 'Content-Length': '0'} 2023-05-31T20:46:32.675 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:39:26 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d763171b777b2dde-006477b228-1c17-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:46:32 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:46:32.676 DEBUG:boto:path=/pklxkd-59/obj55 2023-05-31T20:46:32.676 DEBUG:boto:auth_path=/pklxkd-59/obj55 2023-05-31T20:46:32.677 DEBUG:boto:path=/pklxkd-59/obj55?versionId=null 2023-05-31T20:46:32.677 DEBUG:boto:auth_path=/pklxkd-59/obj55?versionId=null 2023-05-31T20:46:32.678 DEBUG:boto:Method: HEAD 2023-05-31T20:46:32.678 DEBUG:boto:Path: /pklxkd-59/obj55?versionId=null 2023-05-31T20:46:32.678 DEBUG:boto:Data: 2023-05-31T20:46:32.679 DEBUG:boto:Headers: {} 2023-05-31T20:46:32.679 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:46:32.679 DEBUG:boto:Port: 8000 2023-05-31T20:46:32.680 DEBUG:boto:Params: {} 2023-05-31T20:46:32.680 DEBUG:boto:Token: None 2023-05-31T20:46:32.680 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:46:32 GMT /pklxkd-59/obj55?versionId=null 2023-05-31T20:46:32.681 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:ILdo1ljqMDdbnoA/6APwYcM6q/0= 2023-05-31T20:46:32.681 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:46:32 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:ILdo1ljqMDdbnoA/6APwYcM6q/0=', 'Content-Length': '0'} 2023-05-31T20:46:32.683 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:39:26 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000021cc89c4efa00643-006477b228-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:46:32 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:46:32.684 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj55 2023-05-31T20:46:32.684 DEBUG:boto:path=/pklxkd-59/obj55 2023-05-31T20:46:32.684 DEBUG:boto:auth_path=/pklxkd-59/obj55 2023-05-31T20:46:32.685 DEBUG:boto:path=/pklxkd-59/obj55?versionId=null 2023-05-31T20:46:32.685 DEBUG:boto:auth_path=/pklxkd-59/obj55?versionId=null 2023-05-31T20:46:32.686 DEBUG:boto:Method: GET 2023-05-31T20:46:32.686 DEBUG:boto:Path: /pklxkd-59/obj55?versionId=null 2023-05-31T20:46:32.686 DEBUG:boto:Data: 2023-05-31T20:46:32.687 DEBUG:boto:Headers: {} 2023-05-31T20:46:32.687 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:46:32.687 DEBUG:boto:Port: 8001 2023-05-31T20:46:32.688 DEBUG:boto:Params: {} 2023-05-31T20:46:32.688 DEBUG:boto:Token: None 2023-05-31T20:46:32.689 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:46:32 GMT /pklxkd-59/obj55?versionId=null 2023-05-31T20:46:32.689 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:z13Jm+c/UqDdxsaTiXKuWtfTDRs= 2023-05-31T20:46:32.689 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:46:32 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:z13Jm+c/UqDdxsaTiXKuWtfTDRs=', 'Content-Length': '0'} 2023-05-31T20:46:32.691 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:39:26 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ce72f197fede0230-006477b228-1c17-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:46:32 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:46:32.692 DEBUG:boto:path=/pklxkd-59/obj55 2023-05-31T20:46:32.692 DEBUG:boto:auth_path=/pklxkd-59/obj55 2023-05-31T20:46:32.692 DEBUG:boto:path=/pklxkd-59/obj55?versionId=null 2023-05-31T20:46:32.693 DEBUG:boto:auth_path=/pklxkd-59/obj55?versionId=null 2023-05-31T20:46:32.693 DEBUG:boto:Method: GET 2023-05-31T20:46:32.694 DEBUG:boto:Path: /pklxkd-59/obj55?versionId=null 2023-05-31T20:46:32.694 DEBUG:boto:Data: 2023-05-31T20:46:32.694 DEBUG:boto:Headers: {} 2023-05-31T20:46:32.695 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:46:32.695 DEBUG:boto:Port: 8000 2023-05-31T20:46:32.695 DEBUG:boto:Params: {} 2023-05-31T20:46:32.696 DEBUG:boto:Token: None 2023-05-31T20:46:32.696 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:46:32 GMT /pklxkd-59/obj55?versionId=null 2023-05-31T20:46:32.697 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:z13Jm+c/UqDdxsaTiXKuWtfTDRs= 2023-05-31T20:46:32.697 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:46:32 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:z13Jm+c/UqDdxsaTiXKuWtfTDRs=', 'Content-Length': '0'} 2023-05-31T20:46:32.699 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:39:26 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005983109cf16486cb-006477b228-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:46:32 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:46:32.699 DEBUG:boto:path=/pklxkd-59/ 2023-05-31T20:46:32.700 DEBUG:boto:auth_path=/pklxkd-59/ 2023-05-31T20:46:32.700 DEBUG:boto:path=/pklxkd-59/?prefix=obj55 2023-05-31T20:46:32.701 DEBUG:boto:auth_path=/pklxkd-59/?prefix=obj55 2023-05-31T20:46:32.701 DEBUG:boto:Method: GET 2023-05-31T20:46:32.701 DEBUG:boto:Path: /pklxkd-59/?prefix=obj55 2023-05-31T20:46:32.702 DEBUG:boto:Data: 2023-05-31T20:46:32.702 DEBUG:boto:Headers: {} 2023-05-31T20:46:32.702 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:46:32.703 DEBUG:boto:Port: 8001 2023-05-31T20:46:32.703 DEBUG:boto:Params: {} 2023-05-31T20:46:32.703 DEBUG:boto:Token: None 2023-05-31T20:46:32.704 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:46:32 GMT /pklxkd-59/ 2023-05-31T20:46:32.704 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:3Lv9IKKSTFx3S8X7r8veHGM3xUY= 2023-05-31T20:46:32.705 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:46:32 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:3Lv9IKKSTFx3S8X7r8veHGM3xUY=', 'Content-Length': '0'} 2023-05-31T20:46:32.707 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a396af8c65661774-006477b228-1c17-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:46:32 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:46:32.707 DEBUG:boto:b'pklxkd-59obj551000falseobj552023-05-31T20:39:26.630Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:46:32.708 DEBUG:boto:path=/pklxkd-59/ 2023-05-31T20:46:32.708 DEBUG:boto:auth_path=/pklxkd-59/ 2023-05-31T20:46:32.709 DEBUG:boto:path=/pklxkd-59/?prefix=obj55 2023-05-31T20:46:32.709 DEBUG:boto:auth_path=/pklxkd-59/?prefix=obj55 2023-05-31T20:46:32.709 DEBUG:boto:Method: GET 2023-05-31T20:46:32.710 DEBUG:boto:Path: /pklxkd-59/?prefix=obj55 2023-05-31T20:46:32.710 DEBUG:boto:Data: 2023-05-31T20:46:32.711 DEBUG:boto:Headers: {} 2023-05-31T20:46:32.711 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:46:32.711 DEBUG:boto:Port: 8000 2023-05-31T20:46:32.712 DEBUG:boto:Params: {} 2023-05-31T20:46:32.712 DEBUG:boto:Token: None 2023-05-31T20:46:32.712 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:46:32 GMT /pklxkd-59/ 2023-05-31T20:46:32.713 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:3Lv9IKKSTFx3S8X7r8veHGM3xUY= 2023-05-31T20:46:32.713 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:46:32 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:3Lv9IKKSTFx3S8X7r8veHGM3xUY=', 'Content-Length': '0'} 2023-05-31T20:46:32.716 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000003dc4575e53775f78-006477b228-1630-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:46:32 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:46:32.717 DEBUG:boto:b'pklxkd-59obj551000falseobj552023-05-31T20:39:26.630Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:46:32.717 DEBUG:boto:path=/pklxkd-59/obj55 2023-05-31T20:46:32.717 DEBUG:boto:auth_path=/pklxkd-59/obj55 2023-05-31T20:46:32.718 DEBUG:boto:Method: HEAD 2023-05-31T20:46:32.718 DEBUG:boto:Path: /pklxkd-59/obj55 2023-05-31T20:46:32.718 DEBUG:boto:Data: 2023-05-31T20:46:32.719 DEBUG:boto:Headers: {} 2023-05-31T20:46:32.719 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:46:32.719 DEBUG:boto:Port: 8001 2023-05-31T20:46:32.720 DEBUG:boto:Params: {} 2023-05-31T20:46:32.720 DEBUG:boto:Token: None 2023-05-31T20:46:32.721 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:46:32 GMT /pklxkd-59/obj55 2023-05-31T20:46:32.721 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:ZhWrQvtEvdYyCGp0zvujUss0tSc= 2023-05-31T20:46:32.721 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:46:32 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:ZhWrQvtEvdYyCGp0zvujUss0tSc=', 'Content-Length': '0'} 2023-05-31T20:46:32.723 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:39:26 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008865cf708696df78-006477b228-1c17-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:46:32 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:46:32.724 DEBUG:boto:path=/pklxkd-59/obj55 2023-05-31T20:46:32.724 DEBUG:boto:auth_path=/pklxkd-59/obj55 2023-05-31T20:46:32.724 DEBUG:boto:Method: HEAD 2023-05-31T20:46:32.725 DEBUG:boto:Path: /pklxkd-59/obj55 2023-05-31T20:46:32.725 DEBUG:boto:Data: 2023-05-31T20:46:32.725 DEBUG:boto:Headers: {} 2023-05-31T20:46:32.726 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:46:32.726 DEBUG:boto:Port: 8000 2023-05-31T20:46:32.726 DEBUG:boto:Params: {} 2023-05-31T20:46:32.727 DEBUG:boto:Token: None 2023-05-31T20:46:32.727 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:46:32 GMT /pklxkd-59/obj55 2023-05-31T20:46:32.728 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:ZhWrQvtEvdYyCGp0zvujUss0tSc= 2023-05-31T20:46:32.728 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:46:32 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:ZhWrQvtEvdYyCGp0zvujUss0tSc=', 'Content-Length': '0'} 2023-05-31T20:46:32.730 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:39:26 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ac19f6bb88c6026b-006477b228-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:46:32 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:46:32.730 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj55 2023-05-31T20:46:32.731 DEBUG:boto:path=/pklxkd-59/obj55 2023-05-31T20:46:32.731 DEBUG:boto:auth_path=/pklxkd-59/obj55 2023-05-31T20:46:32.731 DEBUG:boto:Method: GET 2023-05-31T20:46:32.732 DEBUG:boto:Path: /pklxkd-59/obj55 2023-05-31T20:46:32.732 DEBUG:boto:Data: 2023-05-31T20:46:32.732 DEBUG:boto:Headers: {} 2023-05-31T20:46:32.733 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:46:32.733 DEBUG:boto:Port: 8001 2023-05-31T20:46:32.733 DEBUG:boto:Params: {} 2023-05-31T20:46:32.734 DEBUG:boto:Token: None 2023-05-31T20:46:32.734 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:46:32 GMT /pklxkd-59/obj55 2023-05-31T20:46:32.735 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:MkI52XpvuI1mEOu7f/dDmQAOHa0= 2023-05-31T20:46:32.735 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:46:32 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:MkI52XpvuI1mEOu7f/dDmQAOHa0=', 'Content-Length': '0'} 2023-05-31T20:46:32.737 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:39:26 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a56adf31011d4e07-006477b228-1c17-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:46:32 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:46:32.737 DEBUG:boto:path=/pklxkd-59/obj55 2023-05-31T20:46:32.738 DEBUG:boto:auth_path=/pklxkd-59/obj55 2023-05-31T20:46:32.738 DEBUG:boto:Method: GET 2023-05-31T20:46:32.738 DEBUG:boto:Path: /pklxkd-59/obj55 2023-05-31T20:46:32.739 DEBUG:boto:Data: 2023-05-31T20:46:32.739 DEBUG:boto:Headers: {} 2023-05-31T20:46:32.739 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:46:32.740 DEBUG:boto:Port: 8000 2023-05-31T20:46:32.740 DEBUG:boto:Params: {} 2023-05-31T20:46:32.741 DEBUG:boto:Token: None 2023-05-31T20:46:32.741 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:46:32 GMT /pklxkd-59/obj55 2023-05-31T20:46:32.741 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:MkI52XpvuI1mEOu7f/dDmQAOHa0= 2023-05-31T20:46:32.742 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:46:32 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:MkI52XpvuI1mEOu7f/dDmQAOHa0=', 'Content-Length': '0'} 2023-05-31T20:46:32.743 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:39:26 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000925d519c8fbf0e69-006477b228-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:46:32 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:46:32.744 DEBUG:boto:path=/pklxkd-59/ 2023-05-31T20:46:32.744 DEBUG:boto:auth_path=/pklxkd-59/ 2023-05-31T20:46:32.745 DEBUG:boto:path=/pklxkd-59/?prefix=obj55 2023-05-31T20:46:32.745 DEBUG:boto:auth_path=/pklxkd-59/?prefix=obj55 2023-05-31T20:46:32.746 DEBUG:boto:Method: GET 2023-05-31T20:46:32.746 DEBUG:boto:Path: /pklxkd-59/?prefix=obj55 2023-05-31T20:46:32.746 DEBUG:boto:Data: 2023-05-31T20:46:32.747 DEBUG:boto:Headers: {} 2023-05-31T20:46:32.747 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:46:32.747 DEBUG:boto:Port: 8001 2023-05-31T20:46:32.748 DEBUG:boto:Params: {} 2023-05-31T20:46:32.748 DEBUG:boto:Token: None 2023-05-31T20:46:32.748 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:46:32 GMT /pklxkd-59/ 2023-05-31T20:46:32.749 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:3Lv9IKKSTFx3S8X7r8veHGM3xUY= 2023-05-31T20:46:32.749 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:46:32 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:3Lv9IKKSTFx3S8X7r8veHGM3xUY=', 'Content-Length': '0'} 2023-05-31T20:46:32.751 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ab7e578db955747f-006477b228-1c17-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:46:32 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:46:32.752 DEBUG:boto:b'pklxkd-59obj551000falseobj552023-05-31T20:39:26.630Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:46:32.753 DEBUG:boto:path=/pklxkd-59/ 2023-05-31T20:46:32.753 DEBUG:boto:auth_path=/pklxkd-59/ 2023-05-31T20:46:32.753 DEBUG:boto:path=/pklxkd-59/?prefix=obj55 2023-05-31T20:46:32.754 DEBUG:boto:auth_path=/pklxkd-59/?prefix=obj55 2023-05-31T20:46:32.754 DEBUG:boto:Method: GET 2023-05-31T20:46:32.754 DEBUG:boto:Path: /pklxkd-59/?prefix=obj55 2023-05-31T20:46:32.755 DEBUG:boto:Data: 2023-05-31T20:46:32.755 DEBUG:boto:Headers: {} 2023-05-31T20:46:32.755 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:46:32.756 DEBUG:boto:Port: 8000 2023-05-31T20:46:32.756 DEBUG:boto:Params: {} 2023-05-31T20:46:32.757 DEBUG:boto:Token: None 2023-05-31T20:46:32.757 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:46:32 GMT /pklxkd-59/ 2023-05-31T20:46:32.757 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:3Lv9IKKSTFx3S8X7r8veHGM3xUY= 2023-05-31T20:46:32.758 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:46:32 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:3Lv9IKKSTFx3S8X7r8veHGM3xUY=', 'Content-Length': '0'} 2023-05-31T20:46:32.760 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f6056b0c635c259a-006477b228-1630-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:46:32 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:46:32.761 DEBUG:boto:b'pklxkd-59obj551000falseobj552023-05-31T20:39:26.630Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:46:32.762 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj57 2023-05-31T20:46:32.762 DEBUG:boto:path=/pklxkd-59/obj57 2023-05-31T20:46:32.762 DEBUG:boto:auth_path=/pklxkd-59/obj57 2023-05-31T20:46:32.763 DEBUG:boto:path=/pklxkd-59/obj57?versionId=null 2023-05-31T20:46:32.763 DEBUG:boto:auth_path=/pklxkd-59/obj57?versionId=null 2023-05-31T20:46:32.763 DEBUG:boto:Method: GET 2023-05-31T20:46:32.764 DEBUG:boto:Path: /pklxkd-59/obj57?versionId=null 2023-05-31T20:46:32.764 DEBUG:boto:Data: 2023-05-31T20:46:32.764 DEBUG:boto:Headers: {} 2023-05-31T20:46:32.765 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:46:32.765 DEBUG:boto:Port: 8001 2023-05-31T20:46:32.766 DEBUG:boto:Params: {} 2023-05-31T20:46:32.766 DEBUG:boto:Token: None 2023-05-31T20:46:32.766 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:46:32 GMT /pklxkd-59/obj57?versionId=null 2023-05-31T20:46:32.767 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:8f7exNPUGyVc1LjMTaIw7+fPSew= 2023-05-31T20:46:32.767 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:46:32 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:8f7exNPUGyVc1LjMTaIw7+fPSew=', 'Content-Length': '0'} 2023-05-31T20:46:32.769 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:44:50 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007efd5f1d5ceb6187-006477b228-1c17-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:46:32 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:46:32.769 DEBUG:boto:path=/pklxkd-59/obj57 2023-05-31T20:46:32.770 DEBUG:boto:auth_path=/pklxkd-59/obj57 2023-05-31T20:46:32.770 DEBUG:boto:path=/pklxkd-59/obj57?versionId=null 2023-05-31T20:46:32.771 DEBUG:boto:auth_path=/pklxkd-59/obj57?versionId=null 2023-05-31T20:46:32.771 DEBUG:boto:Method: GET 2023-05-31T20:46:32.771 DEBUG:boto:Path: /pklxkd-59/obj57?versionId=null 2023-05-31T20:46:32.772 DEBUG:boto:Data: 2023-05-31T20:46:32.772 DEBUG:boto:Headers: {} 2023-05-31T20:46:32.772 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:46:32.773 DEBUG:boto:Port: 8000 2023-05-31T20:46:32.773 DEBUG:boto:Params: {} 2023-05-31T20:46:32.773 DEBUG:boto:Token: None 2023-05-31T20:46:32.774 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:46:32 GMT /pklxkd-59/obj57?versionId=null 2023-05-31T20:46:32.774 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:8f7exNPUGyVc1LjMTaIw7+fPSew= 2023-05-31T20:46:32.775 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:46:32 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:8f7exNPUGyVc1LjMTaIw7+fPSew=', 'Content-Length': '0'} 2023-05-31T20:46:32.776 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:44:50 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000602a967d975363a9-006477b228-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:46:32 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:46:32.820 DEBUG:boto:path=/pklxkd-59/obj57 2023-05-31T20:46:32.821 DEBUG:boto:auth_path=/pklxkd-59/obj57 2023-05-31T20:46:32.821 DEBUG:boto:path=/pklxkd-59/obj57?versionId=null 2023-05-31T20:46:32.821 DEBUG:boto:auth_path=/pklxkd-59/obj57?versionId=null 2023-05-31T20:46:32.822 DEBUG:boto:Method: HEAD 2023-05-31T20:46:32.822 DEBUG:boto:Path: /pklxkd-59/obj57?versionId=null 2023-05-31T20:46:32.822 DEBUG:boto:Data: 2023-05-31T20:46:32.823 DEBUG:boto:Headers: {} 2023-05-31T20:46:32.823 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:46:32.823 DEBUG:boto:Port: 8001 2023-05-31T20:46:32.824 DEBUG:boto:Params: {} 2023-05-31T20:46:32.824 DEBUG:boto:Token: None 2023-05-31T20:46:32.824 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:46:32 GMT /pklxkd-59/obj57?versionId=null 2023-05-31T20:46:32.825 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:ulA7aDWZ/1JVFK8WAPrhlAdbY+U= 2023-05-31T20:46:32.825 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:46:32 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:ulA7aDWZ/1JVFK8WAPrhlAdbY+U=', 'Content-Length': '0'} 2023-05-31T20:46:32.827 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:44:50 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000048466943ec398a63-006477b228-1c17-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:46:32 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:46:32.828 DEBUG:boto:path=/pklxkd-59/obj57 2023-05-31T20:46:32.828 DEBUG:boto:auth_path=/pklxkd-59/obj57 2023-05-31T20:46:32.828 DEBUG:boto:path=/pklxkd-59/obj57?versionId=null 2023-05-31T20:46:32.829 DEBUG:boto:auth_path=/pklxkd-59/obj57?versionId=null 2023-05-31T20:46:32.829 DEBUG:boto:Method: HEAD 2023-05-31T20:46:32.829 DEBUG:boto:Path: /pklxkd-59/obj57?versionId=null 2023-05-31T20:46:32.830 DEBUG:boto:Data: 2023-05-31T20:46:32.830 DEBUG:boto:Headers: {} 2023-05-31T20:46:32.830 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:46:32.831 DEBUG:boto:Port: 8000 2023-05-31T20:46:32.831 DEBUG:boto:Params: {} 2023-05-31T20:46:32.831 DEBUG:boto:Token: None 2023-05-31T20:46:32.832 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:46:32 GMT /pklxkd-59/obj57?versionId=null 2023-05-31T20:46:32.832 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:ulA7aDWZ/1JVFK8WAPrhlAdbY+U= 2023-05-31T20:46:32.833 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:46:32 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:ulA7aDWZ/1JVFK8WAPrhlAdbY+U=', 'Content-Length': '0'} 2023-05-31T20:46:32.834 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:44:50 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000dba0d7e04af9ba2a-006477b228-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:46:32 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:46:32.835 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj57 2023-05-31T20:46:32.835 DEBUG:boto:path=/pklxkd-59/obj57 2023-05-31T20:46:32.836 DEBUG:boto:auth_path=/pklxkd-59/obj57 2023-05-31T20:46:32.836 DEBUG:boto:path=/pklxkd-59/obj57?versionId=null 2023-05-31T20:46:32.836 DEBUG:boto:auth_path=/pklxkd-59/obj57?versionId=null 2023-05-31T20:46:32.837 DEBUG:boto:Method: GET 2023-05-31T20:46:32.837 DEBUG:boto:Path: /pklxkd-59/obj57?versionId=null 2023-05-31T20:46:32.838 DEBUG:boto:Data: 2023-05-31T20:46:32.838 DEBUG:boto:Headers: {} 2023-05-31T20:46:32.838 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:46:32.839 DEBUG:boto:Port: 8001 2023-05-31T20:46:32.839 DEBUG:boto:Params: {} 2023-05-31T20:46:32.839 DEBUG:boto:Token: None 2023-05-31T20:46:32.840 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:46:32 GMT /pklxkd-59/obj57?versionId=null 2023-05-31T20:46:32.840 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:8f7exNPUGyVc1LjMTaIw7+fPSew= 2023-05-31T20:46:32.841 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:46:32 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:8f7exNPUGyVc1LjMTaIw7+fPSew=', 'Content-Length': '0'} 2023-05-31T20:46:32.842 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:44:50 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c04d034e7ef775ee-006477b228-1c17-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:46:32 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:46:32.843 DEBUG:boto:path=/pklxkd-59/obj57 2023-05-31T20:46:32.843 DEBUG:boto:auth_path=/pklxkd-59/obj57 2023-05-31T20:46:32.844 DEBUG:boto:path=/pklxkd-59/obj57?versionId=null 2023-05-31T20:46:32.844 DEBUG:boto:auth_path=/pklxkd-59/obj57?versionId=null 2023-05-31T20:46:32.844 DEBUG:boto:Method: GET 2023-05-31T20:46:32.845 DEBUG:boto:Path: /pklxkd-59/obj57?versionId=null 2023-05-31T20:46:32.845 DEBUG:boto:Data: 2023-05-31T20:46:32.845 DEBUG:boto:Headers: {} 2023-05-31T20:46:32.846 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:46:32.846 DEBUG:boto:Port: 8000 2023-05-31T20:46:32.846 DEBUG:boto:Params: {} 2023-05-31T20:46:32.847 DEBUG:boto:Token: None 2023-05-31T20:46:32.847 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:46:32 GMT /pklxkd-59/obj57?versionId=null 2023-05-31T20:46:32.848 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:8f7exNPUGyVc1LjMTaIw7+fPSew= 2023-05-31T20:46:32.848 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:46:32 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:8f7exNPUGyVc1LjMTaIw7+fPSew=', 'Content-Length': '0'} 2023-05-31T20:46:32.850 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:44:50 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001f7532b710d4f176-006477b228-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:46:32 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:46:32.850 DEBUG:boto:path=/pklxkd-59/ 2023-05-31T20:46:32.851 DEBUG:boto:auth_path=/pklxkd-59/ 2023-05-31T20:46:32.851 DEBUG:boto:path=/pklxkd-59/?prefix=obj57 2023-05-31T20:46:32.851 DEBUG:boto:auth_path=/pklxkd-59/?prefix=obj57 2023-05-31T20:46:32.852 DEBUG:boto:Method: GET 2023-05-31T20:46:32.852 DEBUG:boto:Path: /pklxkd-59/?prefix=obj57 2023-05-31T20:46:32.853 DEBUG:boto:Data: 2023-05-31T20:46:32.853 DEBUG:boto:Headers: {} 2023-05-31T20:46:32.853 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:46:32.854 DEBUG:boto:Port: 8001 2023-05-31T20:46:32.854 DEBUG:boto:Params: {} 2023-05-31T20:46:32.854 DEBUG:boto:Token: None 2023-05-31T20:46:32.855 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:46:32 GMT /pklxkd-59/ 2023-05-31T20:46:32.855 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:3Lv9IKKSTFx3S8X7r8veHGM3xUY= 2023-05-31T20:46:32.856 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:46:32 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:3Lv9IKKSTFx3S8X7r8veHGM3xUY=', 'Content-Length': '0'} 2023-05-31T20:46:32.858 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000003b6307acd78e7f38-006477b228-1c17-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:46:32 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:46:32.859 DEBUG:boto:b'pklxkd-59obj571000falseobj572023-05-31T20:44:50.627Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:46:32.859 DEBUG:boto:path=/pklxkd-59/ 2023-05-31T20:46:32.860 DEBUG:boto:auth_path=/pklxkd-59/ 2023-05-31T20:46:32.860 DEBUG:boto:path=/pklxkd-59/?prefix=obj57 2023-05-31T20:46:32.860 DEBUG:boto:auth_path=/pklxkd-59/?prefix=obj57 2023-05-31T20:46:32.861 DEBUG:boto:Method: GET 2023-05-31T20:46:32.861 DEBUG:boto:Path: /pklxkd-59/?prefix=obj57 2023-05-31T20:46:32.861 DEBUG:boto:Data: 2023-05-31T20:46:32.862 DEBUG:boto:Headers: {} 2023-05-31T20:46:32.862 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:46:32.862 DEBUG:boto:Port: 8000 2023-05-31T20:46:32.863 DEBUG:boto:Params: {} 2023-05-31T20:46:32.863 DEBUG:boto:Token: None 2023-05-31T20:46:32.864 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:46:32 GMT /pklxkd-59/ 2023-05-31T20:46:32.864 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:3Lv9IKKSTFx3S8X7r8veHGM3xUY= 2023-05-31T20:46:32.864 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:46:32 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:3Lv9IKKSTFx3S8X7r8veHGM3xUY=', 'Content-Length': '0'} 2023-05-31T20:46:32.867 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000002cba8fd765898bf8-006477b228-1630-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:46:32 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:46:32.868 DEBUG:boto:b'pklxkd-59obj571000falseobj572023-05-31T20:44:50.627Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:46:32.868 DEBUG:boto:path=/pklxkd-59/obj57 2023-05-31T20:46:32.869 DEBUG:boto:auth_path=/pklxkd-59/obj57 2023-05-31T20:46:32.869 DEBUG:boto:Method: HEAD 2023-05-31T20:46:32.869 DEBUG:boto:Path: /pklxkd-59/obj57 2023-05-31T20:46:32.870 DEBUG:boto:Data: 2023-05-31T20:46:32.870 DEBUG:boto:Headers: {} 2023-05-31T20:46:32.870 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:46:32.871 DEBUG:boto:Port: 8001 2023-05-31T20:46:32.871 DEBUG:boto:Params: {} 2023-05-31T20:46:32.871 DEBUG:boto:Token: None 2023-05-31T20:46:32.872 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:46:32 GMT /pklxkd-59/obj57 2023-05-31T20:46:32.872 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:bU0JxY1MSF7yUNm82t5+99iYeX0= 2023-05-31T20:46:32.873 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:46:32 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:bU0JxY1MSF7yUNm82t5+99iYeX0=', 'Content-Length': '0'} 2023-05-31T20:46:32.874 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:44:50 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000aac50a1cb1243e17-006477b228-1c17-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:46:32 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:46:32.875 DEBUG:boto:path=/pklxkd-59/obj57 2023-05-31T20:46:32.875 DEBUG:boto:auth_path=/pklxkd-59/obj57 2023-05-31T20:46:32.876 DEBUG:boto:Method: HEAD 2023-05-31T20:46:32.876 DEBUG:boto:Path: /pklxkd-59/obj57 2023-05-31T20:46:32.876 DEBUG:boto:Data: 2023-05-31T20:46:32.877 DEBUG:boto:Headers: {} 2023-05-31T20:46:32.877 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:46:32.877 DEBUG:boto:Port: 8000 2023-05-31T20:46:32.878 DEBUG:boto:Params: {} 2023-05-31T20:46:32.878 DEBUG:boto:Token: None 2023-05-31T20:46:32.878 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:46:32 GMT /pklxkd-59/obj57 2023-05-31T20:46:32.879 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:bU0JxY1MSF7yUNm82t5+99iYeX0= 2023-05-31T20:46:32.879 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:46:32 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:bU0JxY1MSF7yUNm82t5+99iYeX0=', 'Content-Length': '0'} 2023-05-31T20:46:32.881 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:44:50 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000014000163c8b6e912-006477b228-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:46:32 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:46:32.881 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj57 2023-05-31T20:46:32.882 DEBUG:boto:path=/pklxkd-59/obj57 2023-05-31T20:46:32.882 DEBUG:boto:auth_path=/pklxkd-59/obj57 2023-05-31T20:46:32.883 DEBUG:boto:Method: GET 2023-05-31T20:46:32.883 DEBUG:boto:Path: /pklxkd-59/obj57 2023-05-31T20:46:32.883 DEBUG:boto:Data: 2023-05-31T20:46:32.884 DEBUG:boto:Headers: {} 2023-05-31T20:46:32.884 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:46:32.884 DEBUG:boto:Port: 8001 2023-05-31T20:46:32.885 DEBUG:boto:Params: {} 2023-05-31T20:46:32.885 DEBUG:boto:Token: None 2023-05-31T20:46:32.885 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:46:32 GMT /pklxkd-59/obj57 2023-05-31T20:46:32.886 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:kY5iSWlOrU3wSs7iaQ8kwkgx0Xc= 2023-05-31T20:46:32.886 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:46:32 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:kY5iSWlOrU3wSs7iaQ8kwkgx0Xc=', 'Content-Length': '0'} 2023-05-31T20:46:32.888 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:44:50 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008254ca7e9023954b-006477b228-1c17-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:46:32 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:46:32.889 DEBUG:boto:path=/pklxkd-59/obj57 2023-05-31T20:46:32.889 DEBUG:boto:auth_path=/pklxkd-59/obj57 2023-05-31T20:46:32.890 DEBUG:boto:Method: GET 2023-05-31T20:46:32.890 DEBUG:boto:Path: /pklxkd-59/obj57 2023-05-31T20:46:32.890 DEBUG:boto:Data: 2023-05-31T20:46:32.891 DEBUG:boto:Headers: {} 2023-05-31T20:46:32.891 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:46:32.891 DEBUG:boto:Port: 8000 2023-05-31T20:46:32.892 DEBUG:boto:Params: {} 2023-05-31T20:46:32.892 DEBUG:boto:Token: None 2023-05-31T20:46:32.892 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:46:32 GMT /pklxkd-59/obj57 2023-05-31T20:46:32.893 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:kY5iSWlOrU3wSs7iaQ8kwkgx0Xc= 2023-05-31T20:46:32.893 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:46:32 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:kY5iSWlOrU3wSs7iaQ8kwkgx0Xc=', 'Content-Length': '0'} 2023-05-31T20:46:32.895 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:44:50 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d86df253d2cc76c5-006477b228-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:46:32 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:46:32.896 DEBUG:boto:path=/pklxkd-59/ 2023-05-31T20:46:32.896 DEBUG:boto:auth_path=/pklxkd-59/ 2023-05-31T20:46:32.897 DEBUG:boto:path=/pklxkd-59/?prefix=obj57 2023-05-31T20:46:32.897 DEBUG:boto:auth_path=/pklxkd-59/?prefix=obj57 2023-05-31T20:46:32.897 DEBUG:boto:Method: GET 2023-05-31T20:46:32.898 DEBUG:boto:Path: /pklxkd-59/?prefix=obj57 2023-05-31T20:46:32.898 DEBUG:boto:Data: 2023-05-31T20:46:32.898 DEBUG:boto:Headers: {} 2023-05-31T20:46:32.899 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:46:32.899 DEBUG:boto:Port: 8001 2023-05-31T20:46:32.899 DEBUG:boto:Params: {} 2023-05-31T20:46:32.900 DEBUG:boto:Token: None 2023-05-31T20:46:32.900 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:46:32 GMT /pklxkd-59/ 2023-05-31T20:46:32.901 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:3Lv9IKKSTFx3S8X7r8veHGM3xUY= 2023-05-31T20:46:32.901 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:46:32 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:3Lv9IKKSTFx3S8X7r8veHGM3xUY=', 'Content-Length': '0'} 2023-05-31T20:46:32.903 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000054981f6e4c0ffecf-006477b228-1c17-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:46:32 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:46:32.944 DEBUG:boto:b'pklxkd-59obj571000falseobj572023-05-31T20:44:50.627Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:46:32.945 DEBUG:boto:path=/pklxkd-59/ 2023-05-31T20:46:32.945 DEBUG:boto:auth_path=/pklxkd-59/ 2023-05-31T20:46:32.946 DEBUG:boto:path=/pklxkd-59/?prefix=obj57 2023-05-31T20:46:32.946 DEBUG:boto:auth_path=/pklxkd-59/?prefix=obj57 2023-05-31T20:46:32.946 DEBUG:boto:Method: GET 2023-05-31T20:46:32.947 DEBUG:boto:Path: /pklxkd-59/?prefix=obj57 2023-05-31T20:46:32.947 DEBUG:boto:Data: 2023-05-31T20:46:32.947 DEBUG:boto:Headers: {} 2023-05-31T20:46:32.948 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:46:32.948 DEBUG:boto:Port: 8000 2023-05-31T20:46:32.948 DEBUG:boto:Params: {} 2023-05-31T20:46:32.949 DEBUG:boto:Token: None 2023-05-31T20:46:32.949 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:46:32 GMT /pklxkd-59/ 2023-05-31T20:46:32.950 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:3Lv9IKKSTFx3S8X7r8veHGM3xUY= 2023-05-31T20:46:32.950 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:46:32 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:3Lv9IKKSTFx3S8X7r8veHGM3xUY=', 'Content-Length': '0'} 2023-05-31T20:46:32.953 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a0219723fa0f8c33-006477b228-1630-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:46:32 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:46:32.953 DEBUG:boto:b'pklxkd-59obj571000falseobj572023-05-31T20:44:50.627Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:46:32.954 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj68 2023-05-31T20:46:32.954 DEBUG:boto:path=/pklxkd-59/obj68 2023-05-31T20:46:32.955 DEBUG:boto:auth_path=/pklxkd-59/obj68 2023-05-31T20:46:32.955 DEBUG:boto:path=/pklxkd-59/obj68?versionId=null 2023-05-31T20:46:32.955 DEBUG:boto:auth_path=/pklxkd-59/obj68?versionId=null 2023-05-31T20:46:32.956 DEBUG:boto:Method: GET 2023-05-31T20:46:32.956 DEBUG:boto:Path: /pklxkd-59/obj68?versionId=null 2023-05-31T20:46:32.956 DEBUG:boto:Data: 2023-05-31T20:46:32.957 DEBUG:boto:Headers: {} 2023-05-31T20:46:32.957 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:46:32.957 DEBUG:boto:Port: 8001 2023-05-31T20:46:32.958 DEBUG:boto:Params: {} 2023-05-31T20:46:32.958 DEBUG:boto:Token: None 2023-05-31T20:46:32.959 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:46:32 GMT /pklxkd-59/obj68?versionId=null 2023-05-31T20:46:32.959 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:HK/wgzjk8ci7A8qcmcsgZ2ASAtk= 2023-05-31T20:46:32.959 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:46:32 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:HK/wgzjk8ci7A8qcmcsgZ2ASAtk=', 'Content-Length': '0'} 2023-05-31T20:46:32.961 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:43:08 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008ba37073ffcc384e-006477b228-1c17-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:46:32 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:46:32.962 DEBUG:boto:path=/pklxkd-59/obj68 2023-05-31T20:46:32.962 DEBUG:boto:auth_path=/pklxkd-59/obj68 2023-05-31T20:46:32.962 DEBUG:boto:path=/pklxkd-59/obj68?versionId=null 2023-05-31T20:46:32.963 DEBUG:boto:auth_path=/pklxkd-59/obj68?versionId=null 2023-05-31T20:46:32.963 DEBUG:boto:Method: GET 2023-05-31T20:46:32.964 DEBUG:boto:Path: /pklxkd-59/obj68?versionId=null 2023-05-31T20:46:32.964 DEBUG:boto:Data: 2023-05-31T20:46:32.964 DEBUG:boto:Headers: {} 2023-05-31T20:46:32.965 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:46:32.965 DEBUG:boto:Port: 8000 2023-05-31T20:46:32.965 DEBUG:boto:Params: {} 2023-05-31T20:46:32.966 DEBUG:boto:Token: None 2023-05-31T20:46:32.966 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:46:32 GMT /pklxkd-59/obj68?versionId=null 2023-05-31T20:46:32.967 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:HK/wgzjk8ci7A8qcmcsgZ2ASAtk= 2023-05-31T20:46:32.967 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:46:32 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:HK/wgzjk8ci7A8qcmcsgZ2ASAtk=', 'Content-Length': '0'} 2023-05-31T20:46:32.969 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:43:08 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000fe169d1704f00469-006477b228-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:46:32 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:46:33.012 DEBUG:boto:path=/pklxkd-59/obj68 2023-05-31T20:46:33.013 DEBUG:boto:auth_path=/pklxkd-59/obj68 2023-05-31T20:46:33.013 DEBUG:boto:path=/pklxkd-59/obj68?versionId=null 2023-05-31T20:46:33.013 DEBUG:boto:auth_path=/pklxkd-59/obj68?versionId=null 2023-05-31T20:46:33.014 DEBUG:boto:Method: HEAD 2023-05-31T20:46:33.014 DEBUG:boto:Path: /pklxkd-59/obj68?versionId=null 2023-05-31T20:46:33.014 DEBUG:boto:Data: 2023-05-31T20:46:33.015 DEBUG:boto:Headers: {} 2023-05-31T20:46:33.015 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:46:33.015 DEBUG:boto:Port: 8001 2023-05-31T20:46:33.016 DEBUG:boto:Params: {} 2023-05-31T20:46:33.016 DEBUG:boto:Token: None 2023-05-31T20:46:33.016 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:46:33 GMT /pklxkd-59/obj68?versionId=null 2023-05-31T20:46:33.017 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:/S41YVC0uw7Wa9DrBbaqOweVe5Q= 2023-05-31T20:46:33.017 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:46:33 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:/S41YVC0uw7Wa9DrBbaqOweVe5Q=', 'Content-Length': '0'} 2023-05-31T20:46:33.019 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:43:08 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006692b7c1ba72d3e3-006477b229-1c17-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:46:33 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:46:33.019 DEBUG:boto:path=/pklxkd-59/obj68 2023-05-31T20:46:33.020 DEBUG:boto:auth_path=/pklxkd-59/obj68 2023-05-31T20:46:33.020 DEBUG:boto:path=/pklxkd-59/obj68?versionId=null 2023-05-31T20:46:33.020 DEBUG:boto:auth_path=/pklxkd-59/obj68?versionId=null 2023-05-31T20:46:33.021 DEBUG:boto:Method: HEAD 2023-05-31T20:46:33.021 DEBUG:boto:Path: /pklxkd-59/obj68?versionId=null 2023-05-31T20:46:33.021 DEBUG:boto:Data: 2023-05-31T20:46:33.022 DEBUG:boto:Headers: {} 2023-05-31T20:46:33.022 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:46:33.022 DEBUG:boto:Port: 8000 2023-05-31T20:46:33.023 DEBUG:boto:Params: {} 2023-05-31T20:46:33.023 DEBUG:boto:Token: None 2023-05-31T20:46:33.023 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:46:33 GMT /pklxkd-59/obj68?versionId=null 2023-05-31T20:46:33.024 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:/S41YVC0uw7Wa9DrBbaqOweVe5Q= 2023-05-31T20:46:33.024 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:46:33 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:/S41YVC0uw7Wa9DrBbaqOweVe5Q=', 'Content-Length': '0'} 2023-05-31T20:46:33.026 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:43:08 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a4a6c069ecac5112-006477b229-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:46:33 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:46:33.027 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj68 2023-05-31T20:46:33.027 DEBUG:boto:path=/pklxkd-59/obj68 2023-05-31T20:46:33.028 DEBUG:boto:auth_path=/pklxkd-59/obj68 2023-05-31T20:46:33.028 DEBUG:boto:path=/pklxkd-59/obj68?versionId=null 2023-05-31T20:46:33.028 DEBUG:boto:auth_path=/pklxkd-59/obj68?versionId=null 2023-05-31T20:46:33.029 DEBUG:boto:Method: GET 2023-05-31T20:46:33.029 DEBUG:boto:Path: /pklxkd-59/obj68?versionId=null 2023-05-31T20:46:33.029 DEBUG:boto:Data: 2023-05-31T20:46:33.030 DEBUG:boto:Headers: {} 2023-05-31T20:46:33.030 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:46:33.030 DEBUG:boto:Port: 8001 2023-05-31T20:46:33.031 DEBUG:boto:Params: {} 2023-05-31T20:46:33.031 DEBUG:boto:Token: None 2023-05-31T20:46:33.032 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:46:33 GMT /pklxkd-59/obj68?versionId=null 2023-05-31T20:46:33.032 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:faU2l5UykPqYFjpBKJCX5ZVoOz8= 2023-05-31T20:46:33.032 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:46:33 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:faU2l5UykPqYFjpBKJCX5ZVoOz8=', 'Content-Length': '0'} 2023-05-31T20:46:33.034 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:43:08 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b95d931ef7582f49-006477b229-1c17-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:46:33 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:46:33.035 DEBUG:boto:path=/pklxkd-59/obj68 2023-05-31T20:46:33.035 DEBUG:boto:auth_path=/pklxkd-59/obj68 2023-05-31T20:46:33.036 DEBUG:boto:path=/pklxkd-59/obj68?versionId=null 2023-05-31T20:46:33.036 DEBUG:boto:auth_path=/pklxkd-59/obj68?versionId=null 2023-05-31T20:46:33.036 DEBUG:boto:Method: GET 2023-05-31T20:46:33.037 DEBUG:boto:Path: /pklxkd-59/obj68?versionId=null 2023-05-31T20:46:33.037 DEBUG:boto:Data: 2023-05-31T20:46:33.038 DEBUG:boto:Headers: {} 2023-05-31T20:46:33.038 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:46:33.038 DEBUG:boto:Port: 8000 2023-05-31T20:46:33.039 DEBUG:boto:Params: {} 2023-05-31T20:46:33.039 DEBUG:boto:Token: None 2023-05-31T20:46:33.039 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:46:33 GMT /pklxkd-59/obj68?versionId=null 2023-05-31T20:46:33.040 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:faU2l5UykPqYFjpBKJCX5ZVoOz8= 2023-05-31T20:46:33.040 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:46:33 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:faU2l5UykPqYFjpBKJCX5ZVoOz8=', 'Content-Length': '0'} 2023-05-31T20:46:33.042 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:43:08 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000fef8f44b18fd17ff-006477b229-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:46:33 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:46:33.043 DEBUG:boto:path=/pklxkd-59/ 2023-05-31T20:46:33.043 DEBUG:boto:auth_path=/pklxkd-59/ 2023-05-31T20:46:33.044 DEBUG:boto:path=/pklxkd-59/?prefix=obj68 2023-05-31T20:46:33.044 DEBUG:boto:auth_path=/pklxkd-59/?prefix=obj68 2023-05-31T20:46:33.044 DEBUG:boto:Method: GET 2023-05-31T20:46:33.045 DEBUG:boto:Path: /pklxkd-59/?prefix=obj68 2023-05-31T20:46:33.045 DEBUG:boto:Data: 2023-05-31T20:46:33.045 DEBUG:boto:Headers: {} 2023-05-31T20:46:33.046 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:46:33.046 DEBUG:boto:Port: 8001 2023-05-31T20:46:33.046 DEBUG:boto:Params: {} 2023-05-31T20:46:33.047 DEBUG:boto:Token: None 2023-05-31T20:46:33.047 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:46:33 GMT /pklxkd-59/ 2023-05-31T20:46:33.048 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:Mj6a1GBX0GcdAr9LvxXBxqhmE7E= 2023-05-31T20:46:33.048 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:46:33 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:Mj6a1GBX0GcdAr9LvxXBxqhmE7E=', 'Content-Length': '0'} 2023-05-31T20:46:33.050 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000042719b76d765de80-006477b229-1c17-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:46:33 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:46:33.051 DEBUG:boto:b'pklxkd-59obj681000falseobj682023-05-31T20:43:08.066Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:46:33.051 DEBUG:boto:path=/pklxkd-59/ 2023-05-31T20:46:33.052 DEBUG:boto:auth_path=/pklxkd-59/ 2023-05-31T20:46:33.052 DEBUG:boto:path=/pklxkd-59/?prefix=obj68 2023-05-31T20:46:33.053 DEBUG:boto:auth_path=/pklxkd-59/?prefix=obj68 2023-05-31T20:46:33.053 DEBUG:boto:Method: GET 2023-05-31T20:46:33.053 DEBUG:boto:Path: /pklxkd-59/?prefix=obj68 2023-05-31T20:46:33.054 DEBUG:boto:Data: 2023-05-31T20:46:33.054 DEBUG:boto:Headers: {} 2023-05-31T20:46:33.054 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:46:33.055 DEBUG:boto:Port: 8000 2023-05-31T20:46:33.055 DEBUG:boto:Params: {} 2023-05-31T20:46:33.055 DEBUG:boto:Token: None 2023-05-31T20:46:33.056 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:46:33 GMT /pklxkd-59/ 2023-05-31T20:46:33.056 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:Mj6a1GBX0GcdAr9LvxXBxqhmE7E= 2023-05-31T20:46:33.057 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:46:33 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:Mj6a1GBX0GcdAr9LvxXBxqhmE7E=', 'Content-Length': '0'} 2023-05-31T20:46:33.059 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000008b7c5f15769dd755-006477b229-1630-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:46:33 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:46:33.060 DEBUG:boto:b'pklxkd-59obj681000falseobj682023-05-31T20:43:08.066Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:46:33.061 DEBUG:boto:path=/pklxkd-59/obj68 2023-05-31T20:46:33.061 DEBUG:boto:auth_path=/pklxkd-59/obj68 2023-05-31T20:46:33.061 DEBUG:boto:Method: HEAD 2023-05-31T20:46:33.062 DEBUG:boto:Path: /pklxkd-59/obj68 2023-05-31T20:46:33.062 DEBUG:boto:Data: 2023-05-31T20:46:33.062 DEBUG:boto:Headers: {} 2023-05-31T20:46:33.063 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:46:33.063 DEBUG:boto:Port: 8001 2023-05-31T20:46:33.063 DEBUG:boto:Params: {} 2023-05-31T20:46:33.064 DEBUG:boto:Token: None 2023-05-31T20:46:33.064 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:46:33 GMT /pklxkd-59/obj68 2023-05-31T20:46:33.064 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:9BRsMlg7c6MT65+RzEk/8b24u/M= 2023-05-31T20:46:33.065 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:46:33 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:9BRsMlg7c6MT65+RzEk/8b24u/M=', 'Content-Length': '0'} 2023-05-31T20:46:33.067 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:43:08 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000083f5f9010387d030-006477b229-1c17-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:46:33 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:46:33.067 DEBUG:boto:path=/pklxkd-59/obj68 2023-05-31T20:46:33.067 DEBUG:boto:auth_path=/pklxkd-59/obj68 2023-05-31T20:46:33.068 DEBUG:boto:Method: HEAD 2023-05-31T20:46:33.068 DEBUG:boto:Path: /pklxkd-59/obj68 2023-05-31T20:46:33.068 DEBUG:boto:Data: 2023-05-31T20:46:33.069 DEBUG:boto:Headers: {} 2023-05-31T20:46:33.069 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:46:33.069 DEBUG:boto:Port: 8000 2023-05-31T20:46:33.070 DEBUG:boto:Params: {} 2023-05-31T20:46:33.070 DEBUG:boto:Token: None 2023-05-31T20:46:33.071 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:46:33 GMT /pklxkd-59/obj68 2023-05-31T20:46:33.071 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:9BRsMlg7c6MT65+RzEk/8b24u/M= 2023-05-31T20:46:33.071 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:46:33 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:9BRsMlg7c6MT65+RzEk/8b24u/M=', 'Content-Length': '0'} 2023-05-31T20:46:33.073 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:43:08 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000011b9e0b43db7a582-006477b229-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:46:33 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:46:33.073 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj68 2023-05-31T20:46:33.074 DEBUG:boto:path=/pklxkd-59/obj68 2023-05-31T20:46:33.074 DEBUG:boto:auth_path=/pklxkd-59/obj68 2023-05-31T20:46:33.075 DEBUG:boto:Method: GET 2023-05-31T20:46:33.075 DEBUG:boto:Path: /pklxkd-59/obj68 2023-05-31T20:46:33.075 DEBUG:boto:Data: 2023-05-31T20:46:33.076 DEBUG:boto:Headers: {} 2023-05-31T20:46:33.076 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:46:33.076 DEBUG:boto:Port: 8001 2023-05-31T20:46:33.077 DEBUG:boto:Params: {} 2023-05-31T20:46:33.077 DEBUG:boto:Token: None 2023-05-31T20:46:33.078 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:46:33 GMT /pklxkd-59/obj68 2023-05-31T20:46:33.078 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:dzZTfjduM2rjy4oZXetej1xbe1I= 2023-05-31T20:46:33.078 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:46:33 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:dzZTfjduM2rjy4oZXetej1xbe1I=', 'Content-Length': '0'} 2023-05-31T20:46:33.080 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:43:08 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004655131c5d6028d4-006477b229-1c17-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:46:33 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:46:33.081 DEBUG:boto:path=/pklxkd-59/obj68 2023-05-31T20:46:33.081 DEBUG:boto:auth_path=/pklxkd-59/obj68 2023-05-31T20:46:33.081 DEBUG:boto:Method: GET 2023-05-31T20:46:33.082 DEBUG:boto:Path: /pklxkd-59/obj68 2023-05-31T20:46:33.082 DEBUG:boto:Data: 2023-05-31T20:46:33.082 DEBUG:boto:Headers: {} 2023-05-31T20:46:33.083 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:46:33.083 DEBUG:boto:Port: 8000 2023-05-31T20:46:33.084 DEBUG:boto:Params: {} 2023-05-31T20:46:33.084 DEBUG:boto:Token: None 2023-05-31T20:46:33.084 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:46:33 GMT /pklxkd-59/obj68 2023-05-31T20:46:33.085 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:dzZTfjduM2rjy4oZXetej1xbe1I= 2023-05-31T20:46:33.085 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:46:33 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:dzZTfjduM2rjy4oZXetej1xbe1I=', 'Content-Length': '0'} 2023-05-31T20:46:33.087 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:43:08 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000010745f7a4e7f9b70-006477b229-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:46:33 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:46:33.088 DEBUG:boto:path=/pklxkd-59/ 2023-05-31T20:46:33.088 DEBUG:boto:auth_path=/pklxkd-59/ 2023-05-31T20:46:33.089 DEBUG:boto:path=/pklxkd-59/?prefix=obj68 2023-05-31T20:46:33.089 DEBUG:boto:auth_path=/pklxkd-59/?prefix=obj68 2023-05-31T20:46:33.089 DEBUG:boto:Method: GET 2023-05-31T20:46:33.090 DEBUG:boto:Path: /pklxkd-59/?prefix=obj68 2023-05-31T20:46:33.090 DEBUG:boto:Data: 2023-05-31T20:46:33.090 DEBUG:boto:Headers: {} 2023-05-31T20:46:33.091 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:46:33.091 DEBUG:boto:Port: 8001 2023-05-31T20:46:33.091 DEBUG:boto:Params: {} 2023-05-31T20:46:33.092 DEBUG:boto:Token: None 2023-05-31T20:46:33.092 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:46:33 GMT /pklxkd-59/ 2023-05-31T20:46:33.093 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:Mj6a1GBX0GcdAr9LvxXBxqhmE7E= 2023-05-31T20:46:33.093 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:46:33 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:Mj6a1GBX0GcdAr9LvxXBxqhmE7E=', 'Content-Length': '0'} 2023-05-31T20:46:33.095 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000007697be8aee5ff711-006477b229-1c17-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:46:33 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:46:33.096 DEBUG:boto:b'pklxkd-59obj681000falseobj682023-05-31T20:43:08.066Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:46:33.096 DEBUG:boto:path=/pklxkd-59/ 2023-05-31T20:46:33.097 DEBUG:boto:auth_path=/pklxkd-59/ 2023-05-31T20:46:33.097 DEBUG:boto:path=/pklxkd-59/?prefix=obj68 2023-05-31T20:46:33.097 DEBUG:boto:auth_path=/pklxkd-59/?prefix=obj68 2023-05-31T20:46:33.098 DEBUG:boto:Method: GET 2023-05-31T20:46:33.098 DEBUG:boto:Path: /pklxkd-59/?prefix=obj68 2023-05-31T20:46:33.098 DEBUG:boto:Data: 2023-05-31T20:46:33.099 DEBUG:boto:Headers: {} 2023-05-31T20:46:33.099 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:46:33.099 DEBUG:boto:Port: 8000 2023-05-31T20:46:33.100 DEBUG:boto:Params: {} 2023-05-31T20:46:33.100 DEBUG:boto:Token: None 2023-05-31T20:46:33.101 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:46:33 GMT /pklxkd-59/ 2023-05-31T20:46:33.101 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:Mj6a1GBX0GcdAr9LvxXBxqhmE7E= 2023-05-31T20:46:33.101 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:46:33 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:Mj6a1GBX0GcdAr9LvxXBxqhmE7E=', 'Content-Length': '0'} 2023-05-31T20:46:33.104 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000069a34a489838470e-006477b229-1630-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:46:33 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:46:33.105 DEBUG:boto:b'pklxkd-59obj681000falseobj682023-05-31T20:43:08.066Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:46:33.105 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj76 2023-05-31T20:46:33.106 DEBUG:boto:path=/pklxkd-59/obj76 2023-05-31T20:46:33.106 DEBUG:boto:auth_path=/pklxkd-59/obj76 2023-05-31T20:46:33.106 DEBUG:boto:path=/pklxkd-59/obj76?versionId=null 2023-05-31T20:46:33.107 DEBUG:boto:auth_path=/pklxkd-59/obj76?versionId=null 2023-05-31T20:46:33.107 DEBUG:boto:Method: GET 2023-05-31T20:46:33.107 DEBUG:boto:Path: /pklxkd-59/obj76?versionId=null 2023-05-31T20:46:33.108 DEBUG:boto:Data: 2023-05-31T20:46:33.108 DEBUG:boto:Headers: {} 2023-05-31T20:46:33.109 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:46:33.109 DEBUG:boto:Port: 8001 2023-05-31T20:46:33.109 DEBUG:boto:Params: {} 2023-05-31T20:46:33.110 DEBUG:boto:Token: None 2023-05-31T20:46:33.110 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:46:33 GMT /pklxkd-59/obj76?versionId=null 2023-05-31T20:46:33.110 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:5okfnVxmYzUXi8yRdakA80PqQs4= 2023-05-31T20:46:33.111 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:46:33 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:5okfnVxmYzUXi8yRdakA80PqQs4=', 'Content-Length': '0'} 2023-05-31T20:46:33.113 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:44:50 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e217372bbbf4ef1e-006477b229-1c17-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:46:33 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:46:33.113 DEBUG:boto:path=/pklxkd-59/obj76 2023-05-31T20:46:33.114 DEBUG:boto:auth_path=/pklxkd-59/obj76 2023-05-31T20:46:33.114 DEBUG:boto:path=/pklxkd-59/obj76?versionId=null 2023-05-31T20:46:33.114 DEBUG:boto:auth_path=/pklxkd-59/obj76?versionId=null 2023-05-31T20:46:33.115 DEBUG:boto:Method: GET 2023-05-31T20:46:33.115 DEBUG:boto:Path: /pklxkd-59/obj76?versionId=null 2023-05-31T20:46:33.115 DEBUG:boto:Data: 2023-05-31T20:46:33.116 DEBUG:boto:Headers: {} 2023-05-31T20:46:33.116 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:46:33.117 DEBUG:boto:Port: 8000 2023-05-31T20:46:33.117 DEBUG:boto:Params: {} 2023-05-31T20:46:33.117 DEBUG:boto:Token: None 2023-05-31T20:46:33.118 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:46:33 GMT /pklxkd-59/obj76?versionId=null 2023-05-31T20:46:33.118 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:5okfnVxmYzUXi8yRdakA80PqQs4= 2023-05-31T20:46:33.119 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:46:33 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:5okfnVxmYzUXi8yRdakA80PqQs4=', 'Content-Length': '0'} 2023-05-31T20:46:33.120 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:44:50 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000042962b0b17705d13-006477b229-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:46:33 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:46:33.164 DEBUG:boto:path=/pklxkd-59/obj76 2023-05-31T20:46:33.165 DEBUG:boto:auth_path=/pklxkd-59/obj76 2023-05-31T20:46:33.165 DEBUG:boto:path=/pklxkd-59/obj76?versionId=null 2023-05-31T20:46:33.165 DEBUG:boto:auth_path=/pklxkd-59/obj76?versionId=null 2023-05-31T20:46:33.166 DEBUG:boto:Method: HEAD 2023-05-31T20:46:33.166 DEBUG:boto:Path: /pklxkd-59/obj76?versionId=null 2023-05-31T20:46:33.166 DEBUG:boto:Data: 2023-05-31T20:46:33.167 DEBUG:boto:Headers: {} 2023-05-31T20:46:33.167 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:46:33.167 DEBUG:boto:Port: 8001 2023-05-31T20:46:33.168 DEBUG:boto:Params: {} 2023-05-31T20:46:33.168 DEBUG:boto:Token: None 2023-05-31T20:46:33.168 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:46:33 GMT /pklxkd-59/obj76?versionId=null 2023-05-31T20:46:33.169 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:jJ3rS3YvckAiKWSUnJoIHU8LJcM= 2023-05-31T20:46:33.169 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:46:33 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:jJ3rS3YvckAiKWSUnJoIHU8LJcM=', 'Content-Length': '0'} 2023-05-31T20:46:33.171 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:44:50 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c2c7e791194b7b32-006477b229-1c17-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:46:33 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:46:33.172 DEBUG:boto:path=/pklxkd-59/obj76 2023-05-31T20:46:33.172 DEBUG:boto:auth_path=/pklxkd-59/obj76 2023-05-31T20:46:33.172 DEBUG:boto:path=/pklxkd-59/obj76?versionId=null 2023-05-31T20:46:33.173 DEBUG:boto:auth_path=/pklxkd-59/obj76?versionId=null 2023-05-31T20:46:33.173 DEBUG:boto:Method: HEAD 2023-05-31T20:46:33.173 DEBUG:boto:Path: /pklxkd-59/obj76?versionId=null 2023-05-31T20:46:33.174 DEBUG:boto:Data: 2023-05-31T20:46:33.174 DEBUG:boto:Headers: {} 2023-05-31T20:46:33.174 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:46:33.175 DEBUG:boto:Port: 8000 2023-05-31T20:46:33.175 DEBUG:boto:Params: {} 2023-05-31T20:46:33.175 DEBUG:boto:Token: None 2023-05-31T20:46:33.176 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:46:33 GMT /pklxkd-59/obj76?versionId=null 2023-05-31T20:46:33.176 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:jJ3rS3YvckAiKWSUnJoIHU8LJcM= 2023-05-31T20:46:33.177 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:46:33 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:jJ3rS3YvckAiKWSUnJoIHU8LJcM=', 'Content-Length': '0'} 2023-05-31T20:46:33.178 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:44:50 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000087b51bb8678bc393-006477b229-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:46:33 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:46:33.179 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj76 2023-05-31T20:46:33.179 DEBUG:boto:path=/pklxkd-59/obj76 2023-05-31T20:46:33.180 DEBUG:boto:auth_path=/pklxkd-59/obj76 2023-05-31T20:46:33.180 DEBUG:boto:path=/pklxkd-59/obj76?versionId=null 2023-05-31T20:46:33.180 DEBUG:boto:auth_path=/pklxkd-59/obj76?versionId=null 2023-05-31T20:46:33.181 DEBUG:boto:Method: GET 2023-05-31T20:46:33.181 DEBUG:boto:Path: /pklxkd-59/obj76?versionId=null 2023-05-31T20:46:33.181 DEBUG:boto:Data: 2023-05-31T20:46:33.182 DEBUG:boto:Headers: {} 2023-05-31T20:46:33.182 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:46:33.182 DEBUG:boto:Port: 8001 2023-05-31T20:46:33.183 DEBUG:boto:Params: {} 2023-05-31T20:46:33.183 DEBUG:boto:Token: None 2023-05-31T20:46:33.184 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:46:33 GMT /pklxkd-59/obj76?versionId=null 2023-05-31T20:46:33.184 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:5okfnVxmYzUXi8yRdakA80PqQs4= 2023-05-31T20:46:33.184 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:46:33 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:5okfnVxmYzUXi8yRdakA80PqQs4=', 'Content-Length': '0'} 2023-05-31T20:46:33.186 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:44:50 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000088c15544f6874cc3-006477b229-1c17-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:46:33 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:46:33.187 DEBUG:boto:path=/pklxkd-59/obj76 2023-05-31T20:46:33.187 DEBUG:boto:auth_path=/pklxkd-59/obj76 2023-05-31T20:46:33.188 DEBUG:boto:path=/pklxkd-59/obj76?versionId=null 2023-05-31T20:46:33.188 DEBUG:boto:auth_path=/pklxkd-59/obj76?versionId=null 2023-05-31T20:46:33.188 DEBUG:boto:Method: GET 2023-05-31T20:46:33.189 DEBUG:boto:Path: /pklxkd-59/obj76?versionId=null 2023-05-31T20:46:33.189 DEBUG:boto:Data: 2023-05-31T20:46:33.189 DEBUG:boto:Headers: {} 2023-05-31T20:46:33.190 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:46:33.190 DEBUG:boto:Port: 8000 2023-05-31T20:46:33.190 DEBUG:boto:Params: {} 2023-05-31T20:46:33.191 DEBUG:boto:Token: None 2023-05-31T20:46:33.191 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:46:33 GMT /pklxkd-59/obj76?versionId=null 2023-05-31T20:46:33.192 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:5okfnVxmYzUXi8yRdakA80PqQs4= 2023-05-31T20:46:33.192 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:46:33 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:5okfnVxmYzUXi8yRdakA80PqQs4=', 'Content-Length': '0'} 2023-05-31T20:46:33.194 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:44:50 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000096d6a379efb51587-006477b229-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:46:33 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:46:33.194 DEBUG:boto:path=/pklxkd-59/ 2023-05-31T20:46:33.195 DEBUG:boto:auth_path=/pklxkd-59/ 2023-05-31T20:46:33.195 DEBUG:boto:path=/pklxkd-59/?prefix=obj76 2023-05-31T20:46:33.196 DEBUG:boto:auth_path=/pklxkd-59/?prefix=obj76 2023-05-31T20:46:33.196 DEBUG:boto:Method: GET 2023-05-31T20:46:33.196 DEBUG:boto:Path: /pklxkd-59/?prefix=obj76 2023-05-31T20:46:33.197 DEBUG:boto:Data: 2023-05-31T20:46:33.197 DEBUG:boto:Headers: {} 2023-05-31T20:46:33.197 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:46:33.198 DEBUG:boto:Port: 8001 2023-05-31T20:46:33.198 DEBUG:boto:Params: {} 2023-05-31T20:46:33.199 DEBUG:boto:Token: None 2023-05-31T20:46:33.199 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:46:33 GMT /pklxkd-59/ 2023-05-31T20:46:33.199 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:Mj6a1GBX0GcdAr9LvxXBxqhmE7E= 2023-05-31T20:46:33.200 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:46:33 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:Mj6a1GBX0GcdAr9LvxXBxqhmE7E=', 'Content-Length': '0'} 2023-05-31T20:46:33.202 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d5c60ae622d6443f-006477b229-1c17-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:46:33 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:46:33.202 DEBUG:boto:b'pklxkd-59obj761000falseobj762023-05-31T20:44:50.682Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:46:33.203 DEBUG:boto:path=/pklxkd-59/ 2023-05-31T20:46:33.204 DEBUG:boto:auth_path=/pklxkd-59/ 2023-05-31T20:46:33.204 DEBUG:boto:path=/pklxkd-59/?prefix=obj76 2023-05-31T20:46:33.204 DEBUG:boto:auth_path=/pklxkd-59/?prefix=obj76 2023-05-31T20:46:33.205 DEBUG:boto:Method: GET 2023-05-31T20:46:33.205 DEBUG:boto:Path: /pklxkd-59/?prefix=obj76 2023-05-31T20:46:33.205 DEBUG:boto:Data: 2023-05-31T20:46:33.206 DEBUG:boto:Headers: {} 2023-05-31T20:46:33.206 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:46:33.206 DEBUG:boto:Port: 8000 2023-05-31T20:46:33.207 DEBUG:boto:Params: {} 2023-05-31T20:46:33.207 DEBUG:boto:Token: None 2023-05-31T20:46:33.207 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:46:33 GMT /pklxkd-59/ 2023-05-31T20:46:33.208 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:Mj6a1GBX0GcdAr9LvxXBxqhmE7E= 2023-05-31T20:46:33.208 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:46:33 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:Mj6a1GBX0GcdAr9LvxXBxqhmE7E=', 'Content-Length': '0'} 2023-05-31T20:46:33.211 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000633c177b6f646895-006477b229-1630-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:46:33 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:46:33.212 DEBUG:boto:b'pklxkd-59obj761000falseobj762023-05-31T20:44:50.682Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:46:33.212 DEBUG:boto:path=/pklxkd-59/obj76 2023-05-31T20:46:33.213 DEBUG:boto:auth_path=/pklxkd-59/obj76 2023-05-31T20:46:33.213 DEBUG:boto:Method: HEAD 2023-05-31T20:46:33.214 DEBUG:boto:Path: /pklxkd-59/obj76 2023-05-31T20:46:33.214 DEBUG:boto:Data: 2023-05-31T20:46:33.214 DEBUG:boto:Headers: {} 2023-05-31T20:46:33.214 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:46:33.215 DEBUG:boto:Port: 8001 2023-05-31T20:46:33.215 DEBUG:boto:Params: {} 2023-05-31T20:46:33.216 DEBUG:boto:Token: None 2023-05-31T20:46:33.216 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:46:33 GMT /pklxkd-59/obj76 2023-05-31T20:46:33.216 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:iZbC2Y3CGh07DVOtVZuCpcOCBTE= 2023-05-31T20:46:33.217 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:46:33 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:iZbC2Y3CGh07DVOtVZuCpcOCBTE=', 'Content-Length': '0'} 2023-05-31T20:46:33.218 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:44:50 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005b18b36c78f5c6e6-006477b229-1c17-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:46:33 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:46:33.219 DEBUG:boto:path=/pklxkd-59/obj76 2023-05-31T20:46:33.219 DEBUG:boto:auth_path=/pklxkd-59/obj76 2023-05-31T20:46:33.220 DEBUG:boto:Method: HEAD 2023-05-31T20:46:33.220 DEBUG:boto:Path: /pklxkd-59/obj76 2023-05-31T20:46:33.220 DEBUG:boto:Data: 2023-05-31T20:46:33.221 DEBUG:boto:Headers: {} 2023-05-31T20:46:33.221 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:46:33.221 DEBUG:boto:Port: 8000 2023-05-31T20:46:33.222 DEBUG:boto:Params: {} 2023-05-31T20:46:33.222 DEBUG:boto:Token: None 2023-05-31T20:46:33.223 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:46:33 GMT /pklxkd-59/obj76 2023-05-31T20:46:33.223 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:iZbC2Y3CGh07DVOtVZuCpcOCBTE= 2023-05-31T20:46:33.224 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:46:33 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:iZbC2Y3CGh07DVOtVZuCpcOCBTE=', 'Content-Length': '0'} 2023-05-31T20:46:33.226 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:44:50 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c016c3e02cd39445-006477b229-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:46:33 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:46:33.226 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj76 2023-05-31T20:46:33.227 DEBUG:boto:path=/pklxkd-59/obj76 2023-05-31T20:46:33.227 DEBUG:boto:auth_path=/pklxkd-59/obj76 2023-05-31T20:46:33.227 DEBUG:boto:Method: GET 2023-05-31T20:46:33.228 DEBUG:boto:Path: /pklxkd-59/obj76 2023-05-31T20:46:33.228 DEBUG:boto:Data: 2023-05-31T20:46:33.228 DEBUG:boto:Headers: {} 2023-05-31T20:46:33.229 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:46:33.229 DEBUG:boto:Port: 8001 2023-05-31T20:46:33.230 DEBUG:boto:Params: {} 2023-05-31T20:46:33.230 DEBUG:boto:Token: None 2023-05-31T20:46:33.230 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:46:33 GMT /pklxkd-59/obj76 2023-05-31T20:46:33.231 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:L8NYiopQJgKaye7glq3Ajf5VZvk= 2023-05-31T20:46:33.231 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:46:33 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:L8NYiopQJgKaye7glq3Ajf5VZvk=', 'Content-Length': '0'} 2023-05-31T20:46:33.233 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:44:50 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000cdd215f2fff914d9-006477b229-1c17-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:46:33 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:46:33.234 DEBUG:boto:path=/pklxkd-59/obj76 2023-05-31T20:46:33.234 DEBUG:boto:auth_path=/pklxkd-59/obj76 2023-05-31T20:46:33.235 DEBUG:boto:Method: GET 2023-05-31T20:46:33.235 DEBUG:boto:Path: /pklxkd-59/obj76 2023-05-31T20:46:33.235 DEBUG:boto:Data: 2023-05-31T20:46:33.236 DEBUG:boto:Headers: {} 2023-05-31T20:46:33.236 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:46:33.237 DEBUG:boto:Port: 8000 2023-05-31T20:46:33.237 DEBUG:boto:Params: {} 2023-05-31T20:46:33.237 DEBUG:boto:Token: None 2023-05-31T20:46:33.238 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:46:33 GMT /pklxkd-59/obj76 2023-05-31T20:46:33.238 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:L8NYiopQJgKaye7glq3Ajf5VZvk= 2023-05-31T20:46:33.239 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:46:33 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:L8NYiopQJgKaye7glq3Ajf5VZvk=', 'Content-Length': '0'} 2023-05-31T20:46:33.240 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:44:50 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000055b15cdd87cb8f05-006477b229-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:46:33 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:46:33.241 DEBUG:boto:path=/pklxkd-59/ 2023-05-31T20:46:33.241 DEBUG:boto:auth_path=/pklxkd-59/ 2023-05-31T20:46:33.242 DEBUG:boto:path=/pklxkd-59/?prefix=obj76 2023-05-31T20:46:33.242 DEBUG:boto:auth_path=/pklxkd-59/?prefix=obj76 2023-05-31T20:46:33.242 DEBUG:boto:Method: GET 2023-05-31T20:46:33.243 DEBUG:boto:Path: /pklxkd-59/?prefix=obj76 2023-05-31T20:46:33.243 DEBUG:boto:Data: 2023-05-31T20:46:33.243 DEBUG:boto:Headers: {} 2023-05-31T20:46:33.244 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:46:33.244 DEBUG:boto:Port: 8001 2023-05-31T20:46:33.244 DEBUG:boto:Params: {} 2023-05-31T20:46:33.245 DEBUG:boto:Token: None 2023-05-31T20:46:33.245 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:46:33 GMT /pklxkd-59/ 2023-05-31T20:46:33.246 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:Mj6a1GBX0GcdAr9LvxXBxqhmE7E= 2023-05-31T20:46:33.246 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:46:33 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:Mj6a1GBX0GcdAr9LvxXBxqhmE7E=', 'Content-Length': '0'} 2023-05-31T20:46:33.248 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000037a648d42d56874e-006477b229-1c17-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:46:33 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:46:33.249 DEBUG:boto:b'pklxkd-59obj761000falseobj762023-05-31T20:44:50.682Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:46:33.250 DEBUG:boto:path=/pklxkd-59/ 2023-05-31T20:46:33.250 DEBUG:boto:auth_path=/pklxkd-59/ 2023-05-31T20:46:33.250 DEBUG:boto:path=/pklxkd-59/?prefix=obj76 2023-05-31T20:46:33.251 DEBUG:boto:auth_path=/pklxkd-59/?prefix=obj76 2023-05-31T20:46:33.251 DEBUG:boto:Method: GET 2023-05-31T20:46:33.251 DEBUG:boto:Path: /pklxkd-59/?prefix=obj76 2023-05-31T20:46:33.252 DEBUG:boto:Data: 2023-05-31T20:46:33.252 DEBUG:boto:Headers: {} 2023-05-31T20:46:33.253 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:46:33.253 DEBUG:boto:Port: 8000 2023-05-31T20:46:33.253 DEBUG:boto:Params: {} 2023-05-31T20:46:33.254 DEBUG:boto:Token: None 2023-05-31T20:46:33.254 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:46:33 GMT /pklxkd-59/ 2023-05-31T20:46:33.254 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:Mj6a1GBX0GcdAr9LvxXBxqhmE7E= 2023-05-31T20:46:33.255 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:46:33 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:Mj6a1GBX0GcdAr9LvxXBxqhmE7E=', 'Content-Length': '0'} 2023-05-31T20:46:33.258 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000005db673c373fbe7bb-006477b229-1630-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:46:33 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:46:33.258 DEBUG:boto:b'pklxkd-59obj761000falseobj762023-05-31T20:44:50.682Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:46:33.259 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj85 2023-05-31T20:46:33.259 DEBUG:boto:path=/pklxkd-59/obj85 2023-05-31T20:46:33.260 DEBUG:boto:auth_path=/pklxkd-59/obj85 2023-05-31T20:46:33.260 DEBUG:boto:path=/pklxkd-59/obj85?versionId=null 2023-05-31T20:46:33.261 DEBUG:boto:auth_path=/pklxkd-59/obj85?versionId=null 2023-05-31T20:46:33.261 DEBUG:boto:Method: GET 2023-05-31T20:46:33.261 DEBUG:boto:Path: /pklxkd-59/obj85?versionId=null 2023-05-31T20:46:33.262 DEBUG:boto:Data: 2023-05-31T20:46:33.262 DEBUG:boto:Headers: {} 2023-05-31T20:46:33.262 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:46:33.263 DEBUG:boto:Port: 8001 2023-05-31T20:46:33.263 DEBUG:boto:Params: {} 2023-05-31T20:46:33.263 DEBUG:boto:Token: None 2023-05-31T20:46:33.264 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:46:33 GMT /pklxkd-59/obj85?versionId=null 2023-05-31T20:46:33.264 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:OXlsaJbExDiFpZBZYCauofch/Y0= 2023-05-31T20:46:33.265 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:46:33 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:OXlsaJbExDiFpZBZYCauofch/Y0=', 'Content-Length': '0'} 2023-05-31T20:46:33.266 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:43:08 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003acf1ff1fd38d192-006477b229-1c17-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:46:33 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:46:33.308 DEBUG:boto:path=/pklxkd-59/obj85 2023-05-31T20:46:33.309 DEBUG:boto:auth_path=/pklxkd-59/obj85 2023-05-31T20:46:33.309 DEBUG:boto:path=/pklxkd-59/obj85?versionId=null 2023-05-31T20:46:33.309 DEBUG:boto:auth_path=/pklxkd-59/obj85?versionId=null 2023-05-31T20:46:33.310 DEBUG:boto:Method: GET 2023-05-31T20:46:33.310 DEBUG:boto:Path: /pklxkd-59/obj85?versionId=null 2023-05-31T20:46:33.310 DEBUG:boto:Data: 2023-05-31T20:46:33.311 DEBUG:boto:Headers: {} 2023-05-31T20:46:33.311 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:46:33.311 DEBUG:boto:Port: 8000 2023-05-31T20:46:33.312 DEBUG:boto:Params: {} 2023-05-31T20:46:33.312 DEBUG:boto:Token: None 2023-05-31T20:46:33.313 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:46:33 GMT /pklxkd-59/obj85?versionId=null 2023-05-31T20:46:33.313 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:OXlsaJbExDiFpZBZYCauofch/Y0= 2023-05-31T20:46:33.314 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:46:33 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:OXlsaJbExDiFpZBZYCauofch/Y0=', 'Content-Length': '0'} 2023-05-31T20:46:33.315 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:43:08 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000092013fb757f9b2d9-006477b229-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:46:33 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:46:33.316 DEBUG:boto:path=/pklxkd-59/obj85 2023-05-31T20:46:33.316 DEBUG:boto:auth_path=/pklxkd-59/obj85 2023-05-31T20:46:33.317 DEBUG:boto:path=/pklxkd-59/obj85?versionId=null 2023-05-31T20:46:33.317 DEBUG:boto:auth_path=/pklxkd-59/obj85?versionId=null 2023-05-31T20:46:33.317 DEBUG:boto:Method: HEAD 2023-05-31T20:46:33.318 DEBUG:boto:Path: /pklxkd-59/obj85?versionId=null 2023-05-31T20:46:33.318 DEBUG:boto:Data: 2023-05-31T20:46:33.318 DEBUG:boto:Headers: {} 2023-05-31T20:46:33.319 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:46:33.319 DEBUG:boto:Port: 8001 2023-05-31T20:46:33.319 DEBUG:boto:Params: {} 2023-05-31T20:46:33.320 DEBUG:boto:Token: None 2023-05-31T20:46:33.320 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:46:33 GMT /pklxkd-59/obj85?versionId=null 2023-05-31T20:46:33.321 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:LGLssZ4rRcs7fJ03EyrtnNva9VQ= 2023-05-31T20:46:33.321 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:46:33 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:LGLssZ4rRcs7fJ03EyrtnNva9VQ=', 'Content-Length': '0'} 2023-05-31T20:46:33.323 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:43:08 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000dbbeb7649cfd2416-006477b229-1c17-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:46:33 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:46:33.323 DEBUG:boto:path=/pklxkd-59/obj85 2023-05-31T20:46:33.324 DEBUG:boto:auth_path=/pklxkd-59/obj85 2023-05-31T20:46:33.324 DEBUG:boto:path=/pklxkd-59/obj85?versionId=null 2023-05-31T20:46:33.324 DEBUG:boto:auth_path=/pklxkd-59/obj85?versionId=null 2023-05-31T20:46:33.325 DEBUG:boto:Method: HEAD 2023-05-31T20:46:33.325 DEBUG:boto:Path: /pklxkd-59/obj85?versionId=null 2023-05-31T20:46:33.325 DEBUG:boto:Data: 2023-05-31T20:46:33.326 DEBUG:boto:Headers: {} 2023-05-31T20:46:33.326 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:46:33.326 DEBUG:boto:Port: 8000 2023-05-31T20:46:33.327 DEBUG:boto:Params: {} 2023-05-31T20:46:33.327 DEBUG:boto:Token: None 2023-05-31T20:46:33.327 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:46:33 GMT /pklxkd-59/obj85?versionId=null 2023-05-31T20:46:33.328 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:LGLssZ4rRcs7fJ03EyrtnNva9VQ= 2023-05-31T20:46:33.328 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:46:33 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:LGLssZ4rRcs7fJ03EyrtnNva9VQ=', 'Content-Length': '0'} 2023-05-31T20:46:33.330 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:43:08 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006440bb4a1a8eb077-006477b229-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:46:33 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:46:33.330 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj85 2023-05-31T20:46:33.331 DEBUG:boto:path=/pklxkd-59/obj85 2023-05-31T20:46:33.331 DEBUG:boto:auth_path=/pklxkd-59/obj85 2023-05-31T20:46:33.331 DEBUG:boto:path=/pklxkd-59/obj85?versionId=null 2023-05-31T20:46:33.332 DEBUG:boto:auth_path=/pklxkd-59/obj85?versionId=null 2023-05-31T20:46:33.332 DEBUG:boto:Method: GET 2023-05-31T20:46:33.333 DEBUG:boto:Path: /pklxkd-59/obj85?versionId=null 2023-05-31T20:46:33.333 DEBUG:boto:Data: 2023-05-31T20:46:33.333 DEBUG:boto:Headers: {} 2023-05-31T20:46:33.334 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:46:33.334 DEBUG:boto:Port: 8001 2023-05-31T20:46:33.334 DEBUG:boto:Params: {} 2023-05-31T20:46:33.335 DEBUG:boto:Token: None 2023-05-31T20:46:33.335 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:46:33 GMT /pklxkd-59/obj85?versionId=null 2023-05-31T20:46:33.336 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:OXlsaJbExDiFpZBZYCauofch/Y0= 2023-05-31T20:46:33.336 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:46:33 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:OXlsaJbExDiFpZBZYCauofch/Y0=', 'Content-Length': '0'} 2023-05-31T20:46:33.338 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:43:08 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000efa1cef891b2c544-006477b229-1c17-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:46:33 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:46:33.339 DEBUG:boto:path=/pklxkd-59/obj85 2023-05-31T20:46:33.339 DEBUG:boto:auth_path=/pklxkd-59/obj85 2023-05-31T20:46:33.339 DEBUG:boto:path=/pklxkd-59/obj85?versionId=null 2023-05-31T20:46:33.340 DEBUG:boto:auth_path=/pklxkd-59/obj85?versionId=null 2023-05-31T20:46:33.340 DEBUG:boto:Method: GET 2023-05-31T20:46:33.340 DEBUG:boto:Path: /pklxkd-59/obj85?versionId=null 2023-05-31T20:46:33.341 DEBUG:boto:Data: 2023-05-31T20:46:33.341 DEBUG:boto:Headers: {} 2023-05-31T20:46:33.341 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:46:33.342 DEBUG:boto:Port: 8000 2023-05-31T20:46:33.342 DEBUG:boto:Params: {} 2023-05-31T20:46:33.343 DEBUG:boto:Token: None 2023-05-31T20:46:33.343 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:46:33 GMT /pklxkd-59/obj85?versionId=null 2023-05-31T20:46:33.343 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:OXlsaJbExDiFpZBZYCauofch/Y0= 2023-05-31T20:46:33.344 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:46:33 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:OXlsaJbExDiFpZBZYCauofch/Y0=', 'Content-Length': '0'} 2023-05-31T20:46:33.345 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:43:08 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000fc35c3efead24836-006477b229-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:46:33 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:46:33.346 DEBUG:boto:path=/pklxkd-59/ 2023-05-31T20:46:33.346 DEBUG:boto:auth_path=/pklxkd-59/ 2023-05-31T20:46:33.347 DEBUG:boto:path=/pklxkd-59/?prefix=obj85 2023-05-31T20:46:33.347 DEBUG:boto:auth_path=/pklxkd-59/?prefix=obj85 2023-05-31T20:46:33.348 DEBUG:boto:Method: GET 2023-05-31T20:46:33.348 DEBUG:boto:Path: /pklxkd-59/?prefix=obj85 2023-05-31T20:46:33.348 DEBUG:boto:Data: 2023-05-31T20:46:33.349 DEBUG:boto:Headers: {} 2023-05-31T20:46:33.349 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:46:33.349 DEBUG:boto:Port: 8001 2023-05-31T20:46:33.350 DEBUG:boto:Params: {} 2023-05-31T20:46:33.350 DEBUG:boto:Token: None 2023-05-31T20:46:33.351 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:46:33 GMT /pklxkd-59/ 2023-05-31T20:46:33.351 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:Mj6a1GBX0GcdAr9LvxXBxqhmE7E= 2023-05-31T20:46:33.351 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:46:33 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:Mj6a1GBX0GcdAr9LvxXBxqhmE7E=', 'Content-Length': '0'} 2023-05-31T20:46:33.354 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000be13188b37174afb-006477b229-1c17-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:46:33 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:46:33.354 DEBUG:boto:b'pklxkd-59obj851000falseobj852023-05-31T20:43:08.086Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:46:33.355 DEBUG:boto:path=/pklxkd-59/ 2023-05-31T20:46:33.356 DEBUG:boto:auth_path=/pklxkd-59/ 2023-05-31T20:46:33.356 DEBUG:boto:path=/pklxkd-59/?prefix=obj85 2023-05-31T20:46:33.356 DEBUG:boto:auth_path=/pklxkd-59/?prefix=obj85 2023-05-31T20:46:33.357 DEBUG:boto:Method: GET 2023-05-31T20:46:33.357 DEBUG:boto:Path: /pklxkd-59/?prefix=obj85 2023-05-31T20:46:33.357 DEBUG:boto:Data: 2023-05-31T20:46:33.358 DEBUG:boto:Headers: {} 2023-05-31T20:46:33.358 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:46:33.358 DEBUG:boto:Port: 8000 2023-05-31T20:46:33.359 DEBUG:boto:Params: {} 2023-05-31T20:46:33.359 DEBUG:boto:Token: None 2023-05-31T20:46:33.360 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:46:33 GMT /pklxkd-59/ 2023-05-31T20:46:33.360 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:Mj6a1GBX0GcdAr9LvxXBxqhmE7E= 2023-05-31T20:46:33.360 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:46:33 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:Mj6a1GBX0GcdAr9LvxXBxqhmE7E=', 'Content-Length': '0'} 2023-05-31T20:46:33.364 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000cb94aad46c67803c-006477b229-1630-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:46:33 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:46:33.364 DEBUG:boto:b'pklxkd-59obj851000falseobj852023-05-31T20:43:08.086Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:46:33.365 DEBUG:boto:path=/pklxkd-59/obj85 2023-05-31T20:46:33.365 DEBUG:boto:auth_path=/pklxkd-59/obj85 2023-05-31T20:46:33.366 DEBUG:boto:Method: HEAD 2023-05-31T20:46:33.366 DEBUG:boto:Path: /pklxkd-59/obj85 2023-05-31T20:46:33.366 DEBUG:boto:Data: 2023-05-31T20:46:33.367 DEBUG:boto:Headers: {} 2023-05-31T20:46:33.367 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:46:33.367 DEBUG:boto:Port: 8001 2023-05-31T20:46:33.368 DEBUG:boto:Params: {} 2023-05-31T20:46:33.368 DEBUG:boto:Token: None 2023-05-31T20:46:33.368 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:46:33 GMT /pklxkd-59/obj85 2023-05-31T20:46:33.369 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:eTZ9lIAF4Kx+q57pKZ8qkjE2Trk= 2023-05-31T20:46:33.369 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:46:33 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:eTZ9lIAF4Kx+q57pKZ8qkjE2Trk=', 'Content-Length': '0'} 2023-05-31T20:46:33.371 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:43:08 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000fb16608f87e52baa-006477b229-1c17-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:46:33 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:46:33.372 DEBUG:boto:path=/pklxkd-59/obj85 2023-05-31T20:46:33.372 DEBUG:boto:auth_path=/pklxkd-59/obj85 2023-05-31T20:46:33.373 DEBUG:boto:Method: HEAD 2023-05-31T20:46:33.373 DEBUG:boto:Path: /pklxkd-59/obj85 2023-05-31T20:46:33.373 DEBUG:boto:Data: 2023-05-31T20:46:33.374 DEBUG:boto:Headers: {} 2023-05-31T20:46:33.374 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:46:33.374 DEBUG:boto:Port: 8000 2023-05-31T20:46:33.375 DEBUG:boto:Params: {} 2023-05-31T20:46:33.375 DEBUG:boto:Token: None 2023-05-31T20:46:33.375 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:46:33 GMT /pklxkd-59/obj85 2023-05-31T20:46:33.376 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:eTZ9lIAF4Kx+q57pKZ8qkjE2Trk= 2023-05-31T20:46:33.376 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:46:33 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:eTZ9lIAF4Kx+q57pKZ8qkjE2Trk=', 'Content-Length': '0'} 2023-05-31T20:46:33.378 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:43:08 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000dccd78fef852e9af-006477b229-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:46:33 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:46:33.378 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj85 2023-05-31T20:46:33.379 DEBUG:boto:path=/pklxkd-59/obj85 2023-05-31T20:46:33.379 DEBUG:boto:auth_path=/pklxkd-59/obj85 2023-05-31T20:46:33.379 DEBUG:boto:Method: GET 2023-05-31T20:46:33.380 DEBUG:boto:Path: /pklxkd-59/obj85 2023-05-31T20:46:33.380 DEBUG:boto:Data: 2023-05-31T20:46:33.381 DEBUG:boto:Headers: {} 2023-05-31T20:46:33.381 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:46:33.381 DEBUG:boto:Port: 8001 2023-05-31T20:46:33.382 DEBUG:boto:Params: {} 2023-05-31T20:46:33.382 DEBUG:boto:Token: None 2023-05-31T20:46:33.382 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:46:33 GMT /pklxkd-59/obj85 2023-05-31T20:46:33.383 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:j3PPcgG8cSPT7nF+MzGI3paZR7I= 2023-05-31T20:46:33.383 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:46:33 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:j3PPcgG8cSPT7nF+MzGI3paZR7I=', 'Content-Length': '0'} 2023-05-31T20:46:33.385 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:43:08 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009c8914d9e54e011f-006477b229-1c17-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:46:33 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:46:33.386 DEBUG:boto:path=/pklxkd-59/obj85 2023-05-31T20:46:33.386 DEBUG:boto:auth_path=/pklxkd-59/obj85 2023-05-31T20:46:33.386 DEBUG:boto:Method: GET 2023-05-31T20:46:33.387 DEBUG:boto:Path: /pklxkd-59/obj85 2023-05-31T20:46:33.387 DEBUG:boto:Data: 2023-05-31T20:46:33.388 DEBUG:boto:Headers: {} 2023-05-31T20:46:33.388 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:46:33.388 DEBUG:boto:Port: 8000 2023-05-31T20:46:33.389 DEBUG:boto:Params: {} 2023-05-31T20:46:33.389 DEBUG:boto:Token: None 2023-05-31T20:46:33.389 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:46:33 GMT /pklxkd-59/obj85 2023-05-31T20:46:33.390 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:j3PPcgG8cSPT7nF+MzGI3paZR7I= 2023-05-31T20:46:33.390 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:46:33 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:j3PPcgG8cSPT7nF+MzGI3paZR7I=', 'Content-Length': '0'} 2023-05-31T20:46:33.392 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:43:08 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004bd525cd7df66a48-006477b229-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:46:33 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:46:33.393 DEBUG:boto:path=/pklxkd-59/ 2023-05-31T20:46:33.393 DEBUG:boto:auth_path=/pklxkd-59/ 2023-05-31T20:46:33.394 DEBUG:boto:path=/pklxkd-59/?prefix=obj85 2023-05-31T20:46:33.394 DEBUG:boto:auth_path=/pklxkd-59/?prefix=obj85 2023-05-31T20:46:33.394 DEBUG:boto:Method: GET 2023-05-31T20:46:33.395 DEBUG:boto:Path: /pklxkd-59/?prefix=obj85 2023-05-31T20:46:33.395 DEBUG:boto:Data: 2023-05-31T20:46:33.395 DEBUG:boto:Headers: {} 2023-05-31T20:46:33.396 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:46:33.396 DEBUG:boto:Port: 8001 2023-05-31T20:46:33.397 DEBUG:boto:Params: {} 2023-05-31T20:46:33.397 DEBUG:boto:Token: None 2023-05-31T20:46:33.397 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:46:33 GMT /pklxkd-59/ 2023-05-31T20:46:33.398 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:Mj6a1GBX0GcdAr9LvxXBxqhmE7E= 2023-05-31T20:46:33.398 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:46:33 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:Mj6a1GBX0GcdAr9LvxXBxqhmE7E=', 'Content-Length': '0'} 2023-05-31T20:46:33.401 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000063418d663c4fbf89-006477b229-1c17-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:46:33 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:46:33.444 DEBUG:boto:b'pklxkd-59obj851000falseobj852023-05-31T20:43:08.086Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:46:33.445 DEBUG:boto:path=/pklxkd-59/ 2023-05-31T20:46:33.446 DEBUG:boto:auth_path=/pklxkd-59/ 2023-05-31T20:46:33.446 DEBUG:boto:path=/pklxkd-59/?prefix=obj85 2023-05-31T20:46:33.446 DEBUG:boto:auth_path=/pklxkd-59/?prefix=obj85 2023-05-31T20:46:33.447 DEBUG:boto:Method: GET 2023-05-31T20:46:33.447 DEBUG:boto:Path: /pklxkd-59/?prefix=obj85 2023-05-31T20:46:33.448 DEBUG:boto:Data: 2023-05-31T20:46:33.448 DEBUG:boto:Headers: {} 2023-05-31T20:46:33.448 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:46:33.449 DEBUG:boto:Port: 8000 2023-05-31T20:46:33.449 DEBUG:boto:Params: {} 2023-05-31T20:46:33.449 DEBUG:boto:Token: None 2023-05-31T20:46:33.450 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:46:33 GMT /pklxkd-59/ 2023-05-31T20:46:33.450 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:Mj6a1GBX0GcdAr9LvxXBxqhmE7E= 2023-05-31T20:46:33.451 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:46:33 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:Mj6a1GBX0GcdAr9LvxXBxqhmE7E=', 'Content-Length': '0'} 2023-05-31T20:46:33.454 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000fd3cf759e63663e8-006477b229-1630-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:46:33 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:46:33.454 DEBUG:boto:b'pklxkd-59obj851000falseobj852023-05-31T20:43:08.086Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:46:33.455 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj95 2023-05-31T20:46:33.455 DEBUG:boto:path=/pklxkd-59/obj95 2023-05-31T20:46:33.456 DEBUG:boto:auth_path=/pklxkd-59/obj95 2023-05-31T20:46:33.456 DEBUG:boto:path=/pklxkd-59/obj95?versionId=null 2023-05-31T20:46:33.457 DEBUG:boto:auth_path=/pklxkd-59/obj95?versionId=null 2023-05-31T20:46:33.457 DEBUG:boto:Method: GET 2023-05-31T20:46:33.457 DEBUG:boto:Path: /pklxkd-59/obj95?versionId=null 2023-05-31T20:46:33.458 DEBUG:boto:Data: 2023-05-31T20:46:33.458 DEBUG:boto:Headers: {} 2023-05-31T20:46:33.458 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:46:33.459 DEBUG:boto:Port: 8001 2023-05-31T20:46:33.459 DEBUG:boto:Params: {} 2023-05-31T20:46:33.459 DEBUG:boto:Token: None 2023-05-31T20:46:33.460 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:46:33 GMT /pklxkd-59/obj95?versionId=null 2023-05-31T20:46:33.460 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:3JmEQ2kjM/6/IwZ0HjrZEj6W9io= 2023-05-31T20:46:33.461 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:46:33 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:3JmEQ2kjM/6/IwZ0HjrZEj6W9io=', 'Content-Length': '0'} 2023-05-31T20:46:33.463 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:44:50 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b4c019c48cb9c6e4-006477b229-1c17-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:46:33 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:46:33.463 DEBUG:boto:path=/pklxkd-59/obj95 2023-05-31T20:46:33.464 DEBUG:boto:auth_path=/pklxkd-59/obj95 2023-05-31T20:46:33.464 DEBUG:boto:path=/pklxkd-59/obj95?versionId=null 2023-05-31T20:46:33.464 DEBUG:boto:auth_path=/pklxkd-59/obj95?versionId=null 2023-05-31T20:46:33.465 DEBUG:boto:Method: GET 2023-05-31T20:46:33.465 DEBUG:boto:Path: /pklxkd-59/obj95?versionId=null 2023-05-31T20:46:33.466 DEBUG:boto:Data: 2023-05-31T20:46:33.466 DEBUG:boto:Headers: {} 2023-05-31T20:46:33.466 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:46:33.467 DEBUG:boto:Port: 8000 2023-05-31T20:46:33.467 DEBUG:boto:Params: {} 2023-05-31T20:46:33.467 DEBUG:boto:Token: None 2023-05-31T20:46:33.468 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:46:33 GMT /pklxkd-59/obj95?versionId=null 2023-05-31T20:46:33.468 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:3JmEQ2kjM/6/IwZ0HjrZEj6W9io= 2023-05-31T20:46:33.469 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:46:33 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:3JmEQ2kjM/6/IwZ0HjrZEj6W9io=', 'Content-Length': '0'} 2023-05-31T20:46:33.471 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:44:50 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b8c30f96b95f915b-006477b229-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:46:33 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:46:33.512 DEBUG:boto:path=/pklxkd-59/obj95 2023-05-31T20:46:33.513 DEBUG:boto:auth_path=/pklxkd-59/obj95 2023-05-31T20:46:33.513 DEBUG:boto:path=/pklxkd-59/obj95?versionId=null 2023-05-31T20:46:33.513 DEBUG:boto:auth_path=/pklxkd-59/obj95?versionId=null 2023-05-31T20:46:33.514 DEBUG:boto:Method: HEAD 2023-05-31T20:46:33.514 DEBUG:boto:Path: /pklxkd-59/obj95?versionId=null 2023-05-31T20:46:33.514 DEBUG:boto:Data: 2023-05-31T20:46:33.515 DEBUG:boto:Headers: {} 2023-05-31T20:46:33.515 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:46:33.515 DEBUG:boto:Port: 8001 2023-05-31T20:46:33.516 DEBUG:boto:Params: {} 2023-05-31T20:46:33.516 DEBUG:boto:Token: None 2023-05-31T20:46:33.517 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:46:33 GMT /pklxkd-59/obj95?versionId=null 2023-05-31T20:46:33.517 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:gqFp9yE2R1OM/D0O/xrzZ1vS78c= 2023-05-31T20:46:33.517 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:46:33 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:gqFp9yE2R1OM/D0O/xrzZ1vS78c=', 'Content-Length': '0'} 2023-05-31T20:46:33.519 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:44:50 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000914355b672558acc-006477b229-1c17-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:46:33 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:46:33.520 DEBUG:boto:path=/pklxkd-59/obj95 2023-05-31T20:46:33.520 DEBUG:boto:auth_path=/pklxkd-59/obj95 2023-05-31T20:46:33.520 DEBUG:boto:path=/pklxkd-59/obj95?versionId=null 2023-05-31T20:46:33.521 DEBUG:boto:auth_path=/pklxkd-59/obj95?versionId=null 2023-05-31T20:46:33.521 DEBUG:boto:Method: HEAD 2023-05-31T20:46:33.521 DEBUG:boto:Path: /pklxkd-59/obj95?versionId=null 2023-05-31T20:46:33.522 DEBUG:boto:Data: 2023-05-31T20:46:33.522 DEBUG:boto:Headers: {} 2023-05-31T20:46:33.522 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:46:33.523 DEBUG:boto:Port: 8000 2023-05-31T20:46:33.523 DEBUG:boto:Params: {} 2023-05-31T20:46:33.523 DEBUG:boto:Token: None 2023-05-31T20:46:33.524 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:46:33 GMT /pklxkd-59/obj95?versionId=null 2023-05-31T20:46:33.524 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:gqFp9yE2R1OM/D0O/xrzZ1vS78c= 2023-05-31T20:46:33.525 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:46:33 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:gqFp9yE2R1OM/D0O/xrzZ1vS78c=', 'Content-Length': '0'} 2023-05-31T20:46:33.526 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:44:50 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006f81265c51f826db-006477b229-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:46:33 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:46:33.527 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj95 2023-05-31T20:46:33.527 DEBUG:boto:path=/pklxkd-59/obj95 2023-05-31T20:46:33.528 DEBUG:boto:auth_path=/pklxkd-59/obj95 2023-05-31T20:46:33.528 DEBUG:boto:path=/pklxkd-59/obj95?versionId=null 2023-05-31T20:46:33.528 DEBUG:boto:auth_path=/pklxkd-59/obj95?versionId=null 2023-05-31T20:46:33.529 DEBUG:boto:Method: GET 2023-05-31T20:46:33.529 DEBUG:boto:Path: /pklxkd-59/obj95?versionId=null 2023-05-31T20:46:33.530 DEBUG:boto:Data: 2023-05-31T20:46:33.530 DEBUG:boto:Headers: {} 2023-05-31T20:46:33.530 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:46:33.531 DEBUG:boto:Port: 8001 2023-05-31T20:46:33.531 DEBUG:boto:Params: {} 2023-05-31T20:46:33.531 DEBUG:boto:Token: None 2023-05-31T20:46:33.532 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:46:33 GMT /pklxkd-59/obj95?versionId=null 2023-05-31T20:46:33.532 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:3JmEQ2kjM/6/IwZ0HjrZEj6W9io= 2023-05-31T20:46:33.533 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:46:33 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:3JmEQ2kjM/6/IwZ0HjrZEj6W9io=', 'Content-Length': '0'} 2023-05-31T20:46:33.534 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:44:50 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a0efb7b4304ee84c-006477b229-1c17-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:46:33 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:46:33.535 DEBUG:boto:path=/pklxkd-59/obj95 2023-05-31T20:46:33.535 DEBUG:boto:auth_path=/pklxkd-59/obj95 2023-05-31T20:46:33.536 DEBUG:boto:path=/pklxkd-59/obj95?versionId=null 2023-05-31T20:46:33.536 DEBUG:boto:auth_path=/pklxkd-59/obj95?versionId=null 2023-05-31T20:46:33.537 DEBUG:boto:Method: GET 2023-05-31T20:46:33.537 DEBUG:boto:Path: /pklxkd-59/obj95?versionId=null 2023-05-31T20:46:33.537 DEBUG:boto:Data: 2023-05-31T20:46:33.538 DEBUG:boto:Headers: {} 2023-05-31T20:46:33.538 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:46:33.538 DEBUG:boto:Port: 8000 2023-05-31T20:46:33.539 DEBUG:boto:Params: {} 2023-05-31T20:46:33.539 DEBUG:boto:Token: None 2023-05-31T20:46:33.539 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:46:33 GMT /pklxkd-59/obj95?versionId=null 2023-05-31T20:46:33.540 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:3JmEQ2kjM/6/IwZ0HjrZEj6W9io= 2023-05-31T20:46:33.540 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:46:33 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:3JmEQ2kjM/6/IwZ0HjrZEj6W9io=', 'Content-Length': '0'} 2023-05-31T20:46:33.542 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:44:50 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c3e3bfebab0e55fe-006477b229-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:46:33 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:46:33.543 DEBUG:boto:path=/pklxkd-59/ 2023-05-31T20:46:33.543 DEBUG:boto:auth_path=/pklxkd-59/ 2023-05-31T20:46:33.543 DEBUG:boto:path=/pklxkd-59/?prefix=obj95 2023-05-31T20:46:33.544 DEBUG:boto:auth_path=/pklxkd-59/?prefix=obj95 2023-05-31T20:46:33.544 DEBUG:boto:Method: GET 2023-05-31T20:46:33.544 DEBUG:boto:Path: /pklxkd-59/?prefix=obj95 2023-05-31T20:46:33.545 DEBUG:boto:Data: 2023-05-31T20:46:33.545 DEBUG:boto:Headers: {} 2023-05-31T20:46:33.546 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:46:33.546 DEBUG:boto:Port: 8001 2023-05-31T20:46:33.546 DEBUG:boto:Params: {} 2023-05-31T20:46:33.547 DEBUG:boto:Token: None 2023-05-31T20:46:33.547 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:46:33 GMT /pklxkd-59/ 2023-05-31T20:46:33.547 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:Mj6a1GBX0GcdAr9LvxXBxqhmE7E= 2023-05-31T20:46:33.548 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:46:33 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:Mj6a1GBX0GcdAr9LvxXBxqhmE7E=', 'Content-Length': '0'} 2023-05-31T20:46:33.550 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000090f0e223a95be8f9-006477b229-1c17-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:46:33 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:46:33.551 DEBUG:boto:b'pklxkd-59obj951000falseobj952023-05-31T20:44:50.705Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:46:33.551 DEBUG:boto:path=/pklxkd-59/ 2023-05-31T20:46:33.552 DEBUG:boto:auth_path=/pklxkd-59/ 2023-05-31T20:46:33.552 DEBUG:boto:path=/pklxkd-59/?prefix=obj95 2023-05-31T20:46:33.553 DEBUG:boto:auth_path=/pklxkd-59/?prefix=obj95 2023-05-31T20:46:33.553 DEBUG:boto:Method: GET 2023-05-31T20:46:33.553 DEBUG:boto:Path: /pklxkd-59/?prefix=obj95 2023-05-31T20:46:33.554 DEBUG:boto:Data: 2023-05-31T20:46:33.554 DEBUG:boto:Headers: {} 2023-05-31T20:46:33.554 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:46:33.555 DEBUG:boto:Port: 8000 2023-05-31T20:46:33.555 DEBUG:boto:Params: {} 2023-05-31T20:46:33.555 DEBUG:boto:Token: None 2023-05-31T20:46:33.556 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:46:33 GMT /pklxkd-59/ 2023-05-31T20:46:33.556 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:Mj6a1GBX0GcdAr9LvxXBxqhmE7E= 2023-05-31T20:46:33.557 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:46:33 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:Mj6a1GBX0GcdAr9LvxXBxqhmE7E=', 'Content-Length': '0'} 2023-05-31T20:46:33.560 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000bd6d7319c38c6302-006477b229-1630-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:46:33 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:46:33.560 DEBUG:boto:b'pklxkd-59obj951000falseobj952023-05-31T20:44:50.705Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:46:33.561 DEBUG:boto:path=/pklxkd-59/obj95 2023-05-31T20:46:33.561 DEBUG:boto:auth_path=/pklxkd-59/obj95 2023-05-31T20:46:33.562 DEBUG:boto:Method: HEAD 2023-05-31T20:46:33.562 DEBUG:boto:Path: /pklxkd-59/obj95 2023-05-31T20:46:33.562 DEBUG:boto:Data: 2023-05-31T20:46:33.563 DEBUG:boto:Headers: {} 2023-05-31T20:46:33.563 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:46:33.563 DEBUG:boto:Port: 8001 2023-05-31T20:46:33.564 DEBUG:boto:Params: {} 2023-05-31T20:46:33.564 DEBUG:boto:Token: None 2023-05-31T20:46:33.564 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:46:33 GMT /pklxkd-59/obj95 2023-05-31T20:46:33.565 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:50LDmLDgdPIqDlW34HKSTO+Lrgk= 2023-05-31T20:46:33.565 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:46:33 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:50LDmLDgdPIqDlW34HKSTO+Lrgk=', 'Content-Length': '0'} 2023-05-31T20:46:33.567 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:44:50 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d7bf13a6244d4f6b-006477b229-1c17-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:46:33 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:46:33.567 DEBUG:boto:path=/pklxkd-59/obj95 2023-05-31T20:46:33.568 DEBUG:boto:auth_path=/pklxkd-59/obj95 2023-05-31T20:46:33.568 DEBUG:boto:Method: HEAD 2023-05-31T20:46:33.568 DEBUG:boto:Path: /pklxkd-59/obj95 2023-05-31T20:46:33.569 DEBUG:boto:Data: 2023-05-31T20:46:33.569 DEBUG:boto:Headers: {} 2023-05-31T20:46:33.569 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:46:33.570 DEBUG:boto:Port: 8000 2023-05-31T20:46:33.570 DEBUG:boto:Params: {} 2023-05-31T20:46:33.571 DEBUG:boto:Token: None 2023-05-31T20:46:33.571 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:46:33 GMT /pklxkd-59/obj95 2023-05-31T20:46:33.571 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:50LDmLDgdPIqDlW34HKSTO+Lrgk= 2023-05-31T20:46:33.572 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:46:33 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:50LDmLDgdPIqDlW34HKSTO+Lrgk=', 'Content-Length': '0'} 2023-05-31T20:46:33.573 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:44:50 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000011b03b6ecb5b4ee1-006477b229-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:46:33 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:46:33.574 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj95 2023-05-31T20:46:33.574 DEBUG:boto:path=/pklxkd-59/obj95 2023-05-31T20:46:33.575 DEBUG:boto:auth_path=/pklxkd-59/obj95 2023-05-31T20:46:33.575 DEBUG:boto:Method: GET 2023-05-31T20:46:33.576 DEBUG:boto:Path: /pklxkd-59/obj95 2023-05-31T20:46:33.576 DEBUG:boto:Data: 2023-05-31T20:46:33.576 DEBUG:boto:Headers: {} 2023-05-31T20:46:33.577 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:46:33.577 DEBUG:boto:Port: 8001 2023-05-31T20:46:33.577 DEBUG:boto:Params: {} 2023-05-31T20:46:33.578 DEBUG:boto:Token: None 2023-05-31T20:46:33.578 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:46:33 GMT /pklxkd-59/obj95 2023-05-31T20:46:33.579 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:VAHoTWPHUeTnWYQt5GM9hu1SYzM= 2023-05-31T20:46:33.579 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:46:33 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:VAHoTWPHUeTnWYQt5GM9hu1SYzM=', 'Content-Length': '0'} 2023-05-31T20:46:33.581 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:44:50 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002d7e9d6e7129bf67-006477b229-1c17-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:46:33 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:46:33.581 DEBUG:boto:path=/pklxkd-59/obj95 2023-05-31T20:46:33.582 DEBUG:boto:auth_path=/pklxkd-59/obj95 2023-05-31T20:46:33.582 DEBUG:boto:Method: GET 2023-05-31T20:46:33.582 DEBUG:boto:Path: /pklxkd-59/obj95 2023-05-31T20:46:33.583 DEBUG:boto:Data: 2023-05-31T20:46:33.583 DEBUG:boto:Headers: {} 2023-05-31T20:46:33.584 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:46:33.584 DEBUG:boto:Port: 8000 2023-05-31T20:46:33.584 DEBUG:boto:Params: {} 2023-05-31T20:46:33.585 DEBUG:boto:Token: None 2023-05-31T20:46:33.585 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:46:33 GMT /pklxkd-59/obj95 2023-05-31T20:46:33.585 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:VAHoTWPHUeTnWYQt5GM9hu1SYzM= 2023-05-31T20:46:33.586 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:46:33 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:VAHoTWPHUeTnWYQt5GM9hu1SYzM=', 'Content-Length': '0'} 2023-05-31T20:46:33.588 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:44:50 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006641b6afaf7c4e5c-006477b229-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:46:33 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:46:33.588 DEBUG:boto:path=/pklxkd-59/ 2023-05-31T20:46:33.589 DEBUG:boto:auth_path=/pklxkd-59/ 2023-05-31T20:46:33.589 DEBUG:boto:path=/pklxkd-59/?prefix=obj95 2023-05-31T20:46:33.589 DEBUG:boto:auth_path=/pklxkd-59/?prefix=obj95 2023-05-31T20:46:33.590 DEBUG:boto:Method: GET 2023-05-31T20:46:33.590 DEBUG:boto:Path: /pklxkd-59/?prefix=obj95 2023-05-31T20:46:33.591 DEBUG:boto:Data: 2023-05-31T20:46:33.591 DEBUG:boto:Headers: {} 2023-05-31T20:46:33.591 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:46:33.592 DEBUG:boto:Port: 8001 2023-05-31T20:46:33.592 DEBUG:boto:Params: {} 2023-05-31T20:46:33.592 DEBUG:boto:Token: None 2023-05-31T20:46:33.593 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:46:33 GMT /pklxkd-59/ 2023-05-31T20:46:33.593 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:Mj6a1GBX0GcdAr9LvxXBxqhmE7E= 2023-05-31T20:46:33.594 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:46:33 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:Mj6a1GBX0GcdAr9LvxXBxqhmE7E=', 'Content-Length': '0'} 2023-05-31T20:46:33.596 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000086b073893a22d64c-006477b229-1c17-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:46:33 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:46:33.596 DEBUG:boto:b'pklxkd-59obj951000falseobj952023-05-31T20:44:50.705Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:46:33.597 DEBUG:boto:path=/pklxkd-59/ 2023-05-31T20:46:33.597 DEBUG:boto:auth_path=/pklxkd-59/ 2023-05-31T20:46:33.598 DEBUG:boto:path=/pklxkd-59/?prefix=obj95 2023-05-31T20:46:33.598 DEBUG:boto:auth_path=/pklxkd-59/?prefix=obj95 2023-05-31T20:46:33.598 DEBUG:boto:Method: GET 2023-05-31T20:46:33.599 DEBUG:boto:Path: /pklxkd-59/?prefix=obj95 2023-05-31T20:46:33.599 DEBUG:boto:Data: 2023-05-31T20:46:33.599 DEBUG:boto:Headers: {} 2023-05-31T20:46:33.600 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:46:33.600 DEBUG:boto:Port: 8000 2023-05-31T20:46:33.601 DEBUG:boto:Params: {} 2023-05-31T20:46:33.601 DEBUG:boto:Token: None 2023-05-31T20:46:33.601 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:46:33 GMT /pklxkd-59/ 2023-05-31T20:46:33.602 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:Mj6a1GBX0GcdAr9LvxXBxqhmE7E= 2023-05-31T20:46:33.602 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:46:33 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:Mj6a1GBX0GcdAr9LvxXBxqhmE7E=', 'Content-Length': '0'} 2023-05-31T20:46:33.605 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d3dfe7bc5251dbce-006477b229-1630-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:46:33 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:46:33.606 DEBUG:boto:b'pklxkd-59obj951000falseobj952023-05-31T20:44:50.705Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:46:33.606 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=pklxkd-59 zones={a2, a1} 2023-05-31T20:46:33.607 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'reshard', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'pklxkd-59', '--num-shards', '23', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-05-31T20:46:33.607 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'reshard', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'pklxkd-59', '--num-shards', '23', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-05-31T20:46:33.607 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket reshard --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --bucket pklxkd-59 --num-shards 23 --yes-i-really-mean-it --cluster c1 --debug-rgw 0 --debug-ms 0 2023-05-31T20:46:33.667 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2023-05-31T20:46:33.667 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2023-05-31T20:46:33.736 INFO:teuthology.orchestra.run.smithi099.stdout:tenant: 2023-05-31T20:46:33.737 INFO:teuthology.orchestra.run.smithi099.stdout:bucket name: pklxkd-59 2023-05-31T20:46:33.743 INFO:teuthology.orchestra.run.smithi099.stdout:total entries: 15 2023-05-31T20:46:33.803 DEBUG:boto:path=/pklxkd-59/ 2023-05-31T20:46:33.803 DEBUG:boto:auth_path=/pklxkd-59/ 2023-05-31T20:46:33.804 DEBUG:boto:Method: HEAD 2023-05-31T20:46:33.804 DEBUG:boto:Path: /pklxkd-59/ 2023-05-31T20:46:33.804 DEBUG:boto:Data: 2023-05-31T20:46:33.805 DEBUG:boto:Headers: {} 2023-05-31T20:46:33.805 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:46:33.805 DEBUG:boto:Port: 8000 2023-05-31T20:46:33.806 DEBUG:boto:Params: {} 2023-05-31T20:46:33.806 DEBUG:boto:Token: None 2023-05-31T20:46:33.806 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:46:33 GMT /pklxkd-59/ 2023-05-31T20:46:33.807 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:jBJ096NtfqMFhsgsu66e18r2tWA= 2023-05-31T20:46:33.807 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:46:33 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:jBJ096NtfqMFhsgsu66e18r2tWA=', 'Content-Length': '0'} 2023-05-31T20:46:33.810 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '15'), ('X-RGW-Bytes-Used', '45'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000dfc5968ccb2b21c7-006477b229-1630-a1'), ('Content-Length', '0'), ('Date', 'Wed, 31 May 2023 20:46:33 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:46:33.810 DEBUG:boto:path=/pklxkd-59/obj23 2023-05-31T20:46:33.811 DEBUG:boto:auth_path=/pklxkd-59/obj23 2023-05-31T20:46:33.811 DEBUG:boto:Method: PUT 2023-05-31T20:46:33.812 DEBUG:boto:Path: /pklxkd-59/obj23 2023-05-31T20:46:33.812 DEBUG:boto:Data: 2023-05-31T20:46:33.812 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-05-31T20:46:33.813 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:46:33.813 DEBUG:boto:Port: 8000 2023-05-31T20:46:33.813 DEBUG:boto:Params: {} 2023-05-31T20:46:33.814 DEBUG:boto:Token: None 2023-05-31T20:46:33.814 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Wed, 31 May 2023 20:46:33 GMT /pklxkd-59/obj23 2023-05-31T20:46:33.814 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:i7NXWEWiXwDyzJqKn5T1OjYR0ug= 2023-05-31T20:46:33.815 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Wed, 31 May 2023 20:46:33 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:i7NXWEWiXwDyzJqKn5T1OjYR0ug='} 2023-05-31T20:46:33.820 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000000b5f3cb0859629d5-006477b229-1630-a1'), ('Date', 'Wed, 31 May 2023 20:46:33 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:46:33.821 DEBUG:boto:path=/pklxkd-59/ 2023-05-31T20:46:33.821 DEBUG:boto:auth_path=/pklxkd-59/ 2023-05-31T20:46:33.822 DEBUG:boto:Method: HEAD 2023-05-31T20:46:33.822 DEBUG:boto:Path: /pklxkd-59/ 2023-05-31T20:46:33.822 DEBUG:boto:Data: 2023-05-31T20:46:33.823 DEBUG:boto:Headers: {} 2023-05-31T20:46:33.823 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:46:33.823 DEBUG:boto:Port: 8000 2023-05-31T20:46:33.824 DEBUG:boto:Params: {} 2023-05-31T20:46:33.824 DEBUG:boto:Token: None 2023-05-31T20:46:33.824 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:46:33 GMT /pklxkd-59/ 2023-05-31T20:46:33.825 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:jBJ096NtfqMFhsgsu66e18r2tWA= 2023-05-31T20:46:33.825 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:46:33 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:jBJ096NtfqMFhsgsu66e18r2tWA=', 'Content-Length': '0'} 2023-05-31T20:46:33.827 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '16'), ('X-RGW-Bytes-Used', '48'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000dc1a6c3925082761-006477b229-1630-a1'), ('Content-Length', '0'), ('Date', 'Wed, 31 May 2023 20:46:33 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:46:33.828 DEBUG:boto:path=/pklxkd-59/obj46 2023-05-31T20:46:33.828 DEBUG:boto:auth_path=/pklxkd-59/obj46 2023-05-31T20:46:33.829 DEBUG:boto:Method: PUT 2023-05-31T20:46:33.829 DEBUG:boto:Path: /pklxkd-59/obj46 2023-05-31T20:46:33.829 DEBUG:boto:Data: 2023-05-31T20:46:33.830 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-05-31T20:46:33.830 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:46:33.830 DEBUG:boto:Port: 8000 2023-05-31T20:46:33.831 DEBUG:boto:Params: {} 2023-05-31T20:46:33.831 DEBUG:boto:Token: None 2023-05-31T20:46:33.831 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Wed, 31 May 2023 20:46:33 GMT /pklxkd-59/obj46 2023-05-31T20:46:33.832 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:XDCm0ln7KDry+hZHrfzRSqg8Vsc= 2023-05-31T20:46:33.832 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Wed, 31 May 2023 20:46:33 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:XDCm0ln7KDry+hZHrfzRSqg8Vsc='} 2023-05-31T20:46:33.876 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000a3b69f24463c2185-006477b229-1630-a1'), ('Date', 'Wed, 31 May 2023 20:46:33 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:46:33.877 DEBUG:boto:path=/pklxkd-59/ 2023-05-31T20:46:33.877 DEBUG:boto:auth_path=/pklxkd-59/ 2023-05-31T20:46:33.877 DEBUG:boto:Method: HEAD 2023-05-31T20:46:33.878 DEBUG:boto:Path: /pklxkd-59/ 2023-05-31T20:46:33.878 DEBUG:boto:Data: 2023-05-31T20:46:33.878 DEBUG:boto:Headers: {} 2023-05-31T20:46:33.879 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:46:33.879 DEBUG:boto:Port: 8000 2023-05-31T20:46:33.879 DEBUG:boto:Params: {} 2023-05-31T20:46:33.880 DEBUG:boto:Token: None 2023-05-31T20:46:33.880 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:46:33 GMT /pklxkd-59/ 2023-05-31T20:46:33.881 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:jBJ096NtfqMFhsgsu66e18r2tWA= 2023-05-31T20:46:33.881 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:46:33 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:jBJ096NtfqMFhsgsu66e18r2tWA=', 'Content-Length': '0'} 2023-05-31T20:46:33.884 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '17'), ('X-RGW-Bytes-Used', '51'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000002265951eecac5ce4-006477b229-1630-a1'), ('Content-Length', '0'), ('Date', 'Wed, 31 May 2023 20:46:33 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:46:33.884 DEBUG:boto:path=/pklxkd-59/obj69 2023-05-31T20:46:33.884 DEBUG:boto:auth_path=/pklxkd-59/obj69 2023-05-31T20:46:33.885 DEBUG:boto:Method: PUT 2023-05-31T20:46:33.885 DEBUG:boto:Path: /pklxkd-59/obj69 2023-05-31T20:46:33.886 DEBUG:boto:Data: 2023-05-31T20:46:33.886 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-05-31T20:46:33.886 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:46:33.887 DEBUG:boto:Port: 8000 2023-05-31T20:46:33.887 DEBUG:boto:Params: {} 2023-05-31T20:46:33.887 DEBUG:boto:Token: None 2023-05-31T20:46:33.888 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Wed, 31 May 2023 20:46:33 GMT /pklxkd-59/obj69 2023-05-31T20:46:33.888 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:R6J+Meon6LODitdmDHDMbNpl5c8= 2023-05-31T20:46:33.888 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Wed, 31 May 2023 20:46:33 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:R6J+Meon6LODitdmDHDMbNpl5c8='} 2023-05-31T20:46:33.895 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000009854db7374d5f1ae-006477b229-1630-a1'), ('Date', 'Wed, 31 May 2023 20:46:33 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:46:33.895 DEBUG:boto:path=/pklxkd-59/ 2023-05-31T20:46:33.895 DEBUG:boto:auth_path=/pklxkd-59/ 2023-05-31T20:46:33.896 DEBUG:boto:Method: HEAD 2023-05-31T20:46:33.896 DEBUG:boto:Path: /pklxkd-59/ 2023-05-31T20:46:33.896 DEBUG:boto:Data: 2023-05-31T20:46:33.897 DEBUG:boto:Headers: {} 2023-05-31T20:46:33.897 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:46:33.897 DEBUG:boto:Port: 8000 2023-05-31T20:46:33.898 DEBUG:boto:Params: {} 2023-05-31T20:46:33.898 DEBUG:boto:Token: None 2023-05-31T20:46:33.899 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:46:33 GMT /pklxkd-59/ 2023-05-31T20:46:33.899 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:jBJ096NtfqMFhsgsu66e18r2tWA= 2023-05-31T20:46:33.899 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:46:33 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:jBJ096NtfqMFhsgsu66e18r2tWA=', 'Content-Length': '0'} 2023-05-31T20:46:33.902 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '18'), ('X-RGW-Bytes-Used', '54'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000c2827aa47e7cf90e-006477b229-1630-a1'), ('Content-Length', '0'), ('Date', 'Wed, 31 May 2023 20:46:33 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:46:33.902 DEBUG:boto:path=/pklxkd-59/obj92 2023-05-31T20:46:33.903 DEBUG:boto:auth_path=/pklxkd-59/obj92 2023-05-31T20:46:33.903 DEBUG:boto:Method: PUT 2023-05-31T20:46:33.903 DEBUG:boto:Path: /pklxkd-59/obj92 2023-05-31T20:46:33.904 DEBUG:boto:Data: 2023-05-31T20:46:33.904 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-05-31T20:46:33.904 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:46:33.905 DEBUG:boto:Port: 8000 2023-05-31T20:46:33.905 DEBUG:boto:Params: {} 2023-05-31T20:46:33.905 DEBUG:boto:Token: None 2023-05-31T20:46:33.906 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Wed, 31 May 2023 20:46:33 GMT /pklxkd-59/obj92 2023-05-31T20:46:33.906 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:XuMLLGqsk7a/ljAgkzA+Jm6i/Eo= 2023-05-31T20:46:33.907 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Wed, 31 May 2023 20:46:33 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:XuMLLGqsk7a/ljAgkzA+Jm6i/Eo='} 2023-05-31T20:46:33.948 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000004a5cad476b77000a-006477b229-1630-a1'), ('Date', 'Wed, 31 May 2023 20:46:33 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:46:33.949 DEBUG:boto:path=/pklxkd-59/ 2023-05-31T20:46:33.949 DEBUG:boto:auth_path=/pklxkd-59/ 2023-05-31T20:46:33.949 DEBUG:boto:Method: HEAD 2023-05-31T20:46:33.950 DEBUG:boto:Path: /pklxkd-59/ 2023-05-31T20:46:33.950 DEBUG:boto:Data: 2023-05-31T20:46:33.950 DEBUG:boto:Headers: {} 2023-05-31T20:46:33.951 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:46:33.951 DEBUG:boto:Port: 8000 2023-05-31T20:46:33.951 DEBUG:boto:Params: {} 2023-05-31T20:46:33.952 DEBUG:boto:Token: None 2023-05-31T20:46:33.952 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:46:33 GMT /pklxkd-59/ 2023-05-31T20:46:33.953 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:jBJ096NtfqMFhsgsu66e18r2tWA= 2023-05-31T20:46:33.953 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:46:33 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:jBJ096NtfqMFhsgsu66e18r2tWA=', 'Content-Length': '0'} 2023-05-31T20:46:33.955 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '19'), ('X-RGW-Bytes-Used', '57'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000adc622029eafdba8-006477b229-1630-a1'), ('Content-Length', '0'), ('Date', 'Wed, 31 May 2023 20:46:33 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:46:33.956 DEBUG:boto:path=/pklxkd-59/obj115 2023-05-31T20:46:33.956 DEBUG:boto:auth_path=/pklxkd-59/obj115 2023-05-31T20:46:33.957 DEBUG:boto:Method: PUT 2023-05-31T20:46:33.957 DEBUG:boto:Path: /pklxkd-59/obj115 2023-05-31T20:46:33.957 DEBUG:boto:Data: 2023-05-31T20:46:33.958 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-05-31T20:46:33.958 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:46:33.958 DEBUG:boto:Port: 8000 2023-05-31T20:46:33.959 DEBUG:boto:Params: {} 2023-05-31T20:46:33.959 DEBUG:boto:Token: None 2023-05-31T20:46:33.960 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Wed, 31 May 2023 20:46:33 GMT /pklxkd-59/obj115 2023-05-31T20:46:33.960 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:Fwxi2voLDxSYjDfhTDxu2Wwcf/s= 2023-05-31T20:46:33.960 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Wed, 31 May 2023 20:46:33 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:Fwxi2voLDxSYjDfhTDxu2Wwcf/s='} 2023-05-31T20:46:33.966 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000052ab4485099338d6-006477b229-1630-a1'), ('Date', 'Wed, 31 May 2023 20:46:33 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:46:33.966 DEBUG:rgw_multi.tests:bucket checkpoint: source=a1 target=a2 bucket=pklxkd-59 2023-05-31T20:46:33.967 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'pklxkd-59', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-05-31T20:46:33.967 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'pklxkd-59', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-05-31T20:46:33.967 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket pklxkd-59 --source-zone a1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2023-05-31T20:46:34.024 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2023-05-31T20:46:34.024 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2023-05-31T20:46:34.062 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:46:34.057+0000 7f954a6d2780 1 waiting to reach latest gen 3, on 2.. 2023-05-31T20:46:39.063 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:46:39.059+0000 7f954a6d2780 1 waiting to reach latest gen 3, on 2.. 2023-05-31T20:46:44.063 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:46:44.059+0000 7f954a6d2780 1 waiting to reach latest gen 3, on 2.. 2023-05-31T20:46:49.066 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:46:49.061+0000 7f954a6d2780 1 waiting for incremental sync to catch up: 2023-05-31T20:46:49.066 INFO:teuthology.orchestra.run.smithi099.stderr: local status: [, , , , , , , , 00000000002.76.6, , 00000000002.63.6, , , , , , 00000000001.72.6, , 00000000001.65.6, , , , ] 2023-05-31T20:46:49.066 INFO:teuthology.orchestra.run.smithi099.stderr: remote markers: [, , , , , 00000000001.57.6, , , 00000000002.76.6, , 00000000002.63.6, , , , , , 00000000001.72.6, , 00000000001.65.6, , , , ] 2023-05-31T20:46:54.067 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:46:54.063+0000 7f954a6d2780 1 waiting for incremental sync to catch up: 2023-05-31T20:46:54.068 INFO:teuthology.orchestra.run.smithi099.stderr: local status: [, , , , , , , , 00000000002.76.6, , 00000000002.63.6, , , , , , 00000000001.72.6, , 00000000001.65.6, , , , ] 2023-05-31T20:46:54.068 INFO:teuthology.orchestra.run.smithi099.stderr: remote markers: [, , , , , 00000000001.57.6, , , 00000000002.76.6, , 00000000002.63.6, , , , , , 00000000001.72.6, , 00000000001.65.6, , , , ] 2023-05-31T20:46:59.069 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:46:59.064+0000 7f954a6d2780 1 waiting for incremental sync to catch up: 2023-05-31T20:46:59.069 INFO:teuthology.orchestra.run.smithi099.stderr: local status: [, , , , , , , , 00000000002.76.6, , 00000000002.63.6, , , , , , 00000000001.72.6, , 00000000001.65.6, , , , ] 2023-05-31T20:46:59.070 INFO:teuthology.orchestra.run.smithi099.stderr: remote markers: [, , , , , 00000000001.57.6, , , 00000000002.76.6, , 00000000002.63.6, , , , , , 00000000001.72.6, , 00000000001.65.6, , , , ] 2023-05-31T20:47:04.071 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:47:04.066+0000 7f954a6d2780 1 waiting for incremental sync to catch up: 2023-05-31T20:47:04.071 INFO:teuthology.orchestra.run.smithi099.stderr: local status: [, , , , , , , , 00000000002.76.6, , 00000000002.63.6, , , , , , 00000000001.72.6, , 00000000001.65.6, , , , ] 2023-05-31T20:47:04.071 INFO:teuthology.orchestra.run.smithi099.stderr: remote markers: [, , , , , 00000000001.57.6, , , 00000000002.76.6, , 00000000002.63.6, , , , , , 00000000001.72.6, , 00000000001.65.6, , , , ] 2023-05-31T20:47:09.073 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:47:09.067+0000 7f954a6d2780 1 waiting for incremental sync to catch up: 2023-05-31T20:47:09.073 INFO:teuthology.orchestra.run.smithi099.stderr: local status: [, , , , , , , , 00000000002.76.6, , 00000000002.63.6, , , , , , 00000000001.72.6, , 00000000001.65.6, , , , ] 2023-05-31T20:47:09.073 INFO:teuthology.orchestra.run.smithi099.stderr: remote markers: [, , , , , 00000000001.57.6, , , 00000000002.76.6, , 00000000002.63.6, , , , , , 00000000001.72.6, , 00000000001.65.6, , , , ] 2023-05-31T20:47:14.074 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:47:14.069+0000 7f954a6d2780 1 waiting for incremental sync to catch up: 2023-05-31T20:47:14.075 INFO:teuthology.orchestra.run.smithi099.stderr: local status: [, , , , , , , , 00000000002.76.6, , 00000000002.63.6, , , , , , 00000000001.72.6, , 00000000001.65.6, , , , ] 2023-05-31T20:47:14.075 INFO:teuthology.orchestra.run.smithi099.stderr: remote markers: [, , , , , 00000000001.57.6, , , 00000000002.76.6, , 00000000002.63.6, , , , , , 00000000001.72.6, , 00000000001.65.6, , , , ] 2023-05-31T20:47:19.076 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:47:19.072+0000 7f954a6d2780 1 waiting for incremental sync to catch up: 2023-05-31T20:47:19.077 INFO:teuthology.orchestra.run.smithi099.stderr: local status: [, , , , , , , , 00000000002.76.6, , 00000000002.63.6, , , , , , 00000000001.72.6, , 00000000001.65.6, , , , ] 2023-05-31T20:47:19.077 INFO:teuthology.orchestra.run.smithi099.stderr: remote markers: [, , , , , 00000000001.57.6, , , 00000000002.76.6, , 00000000002.63.6, , , , , , 00000000001.72.6, , 00000000001.65.6, , , , ] 2023-05-31T20:47:24.079 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:47:24.074+0000 7f954a6d2780 1 waiting for incremental sync to catch up: 2023-05-31T20:47:24.079 INFO:teuthology.orchestra.run.smithi099.stderr: local status: [, , , , , , , , 00000000002.76.6, , 00000000002.63.6, , , , , , 00000000001.72.6, , 00000000001.65.6, , , , ] 2023-05-31T20:47:24.079 INFO:teuthology.orchestra.run.smithi099.stderr: remote markers: [, , , , , 00000000001.57.6, , , 00000000002.76.6, , 00000000002.63.6, , , , , , 00000000001.72.6, , 00000000001.65.6, , , , ] 2023-05-31T20:47:29.080 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:47:29.076+0000 7f954a6d2780 1 waiting for incremental sync to catch up: 2023-05-31T20:47:29.081 INFO:teuthology.orchestra.run.smithi099.stderr: local status: [, , , , , , , , 00000000002.76.6, , 00000000002.63.6, , , , , , 00000000001.72.6, , 00000000001.65.6, , , , ] 2023-05-31T20:47:29.081 INFO:teuthology.orchestra.run.smithi099.stderr: remote markers: [, , , , , 00000000001.57.6, , , 00000000002.76.6, , 00000000002.63.6, , , , , , 00000000001.72.6, , 00000000001.65.6, , , , ] 2023-05-31T20:47:34.082 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:47:34.078+0000 7f954a6d2780 1 waiting for incremental sync to catch up: 2023-05-31T20:47:34.082 INFO:teuthology.orchestra.run.smithi099.stderr: local status: [, , , , , , , , 00000000002.76.6, , 00000000002.63.6, , , , , , 00000000001.72.6, , 00000000001.65.6, , , , ] 2023-05-31T20:47:34.083 INFO:teuthology.orchestra.run.smithi099.stderr: remote markers: [, , , , , 00000000001.57.6, , , 00000000002.76.6, , 00000000002.63.6, , , , , , 00000000001.72.6, , 00000000001.65.6, , , , ] 2023-05-31T20:47:39.084 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:47:39.079+0000 7f954a6d2780 1 waiting for incremental sync to catch up: 2023-05-31T20:47:39.084 INFO:teuthology.orchestra.run.smithi099.stderr: local status: [, , , , , , , , 00000000002.76.6, , 00000000002.63.6, , , , , , 00000000001.72.6, , 00000000001.65.6, , , , ] 2023-05-31T20:47:39.084 INFO:teuthology.orchestra.run.smithi099.stderr: remote markers: [, , , , , 00000000001.57.6, , , 00000000002.76.6, , 00000000002.63.6, , , , , , 00000000001.72.6, , 00000000001.65.6, , , , ] 2023-05-31T20:47:44.086 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:47:44.081+0000 7f954a6d2780 1 waiting for incremental sync to catch up: 2023-05-31T20:47:44.086 INFO:teuthology.orchestra.run.smithi099.stderr: local status: [, , , , , , , , 00000000002.76.6, , 00000000002.63.6, , , , , , 00000000001.72.6, , 00000000001.65.6, , , , ] 2023-05-31T20:47:44.086 INFO:teuthology.orchestra.run.smithi099.stderr: remote markers: [, , , , , 00000000001.57.6, , , 00000000002.76.6, , 00000000002.63.6, , , , , , 00000000001.72.6, , 00000000001.65.6, , , , ] 2023-05-31T20:47:49.088 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:47:49.082+0000 7f954a6d2780 1 waiting for incremental sync to catch up: 2023-05-31T20:47:49.088 INFO:teuthology.orchestra.run.smithi099.stderr: local status: [, , , , , , , , 00000000002.76.6, , 00000000002.63.6, , , , , , 00000000001.72.6, , 00000000001.65.6, , , , ] 2023-05-31T20:47:49.088 INFO:teuthology.orchestra.run.smithi099.stderr: remote markers: [, , , , , 00000000001.57.6, , , 00000000002.76.6, , 00000000002.63.6, , , , , , 00000000001.72.6, , 00000000001.65.6, , , , ] 2023-05-31T20:47:54.089 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:47:54.084+0000 7f954a6d2780 1 bucket sync caught up with source: 2023-05-31T20:47:54.090 INFO:teuthology.orchestra.run.smithi099.stderr: local status: [, , , , , 00000000001.57.6, , , 00000000002.76.6, , 00000000002.63.6, , , , , , 00000000001.72.6, , 00000000001.65.6, , , , ] 2023-05-31T20:47:54.090 INFO:teuthology.orchestra.run.smithi099.stderr: remote markers: [, , , , , 00000000001.57.6, , , 00000000002.76.6, , 00000000002.63.6, , , , , , 00000000001.72.6, , 00000000001.65.6, , , , ] 2023-05-31T20:47:54.090 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:47:54.084+0000 7f954a6d2780 0 bucket checkpoint complete 2023-05-31T20:47:54.108 DEBUG:rgw_multi.tests:bucket checkpoint: source=a2 target=a1 bucket=pklxkd-59 2023-05-31T20:47:54.108 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'pklxkd-59', '--source-zone', 'a2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-05-31T20:47:54.109 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'pklxkd-59', '--source-zone', 'a2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-05-31T20:47:54.109 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket pklxkd-59 --source-zone a2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2023-05-31T20:47:54.171 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2023-05-31T20:47:54.171 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2023-05-31T20:47:54.215 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:47:54.210+0000 7f6685c72780 1 waiting for incremental sync to catch up: 2023-05-31T20:47:54.216 INFO:teuthology.orchestra.run.smithi099.stderr: local status: [, 00000000002.3584.6, 00000000002.3906.6, 00000000001.2027.6, 00000000002.54.6, 00000000002.2021.6, , , 00000000002.2013.6, 00000000004.2294.6, 00000000004.2018.6] 2023-05-31T20:47:54.216 INFO:teuthology.orchestra.run.smithi099.stderr: remote markers: [, 00000000002.3584.6, 00000000002.3906.6, 00000000001.2027.6, 00000000002.54.6, 00000000002.2021.6, , , 00000000002.2013.6, 00000000005.2299.6, 00000000004.2018.6] 2023-05-31T20:47:59.217 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:47:59.213+0000 7f6685c72780 1 waiting for incremental sync to catch up: 2023-05-31T20:47:59.217 INFO:teuthology.orchestra.run.smithi099.stderr: local status: [, 00000000002.3584.6, 00000000002.3906.6, 00000000001.2027.6, 00000000002.54.6, 00000000002.2021.6, , , 00000000002.2013.6, 00000000004.2294.6, 00000000004.2018.6] 2023-05-31T20:47:59.217 INFO:teuthology.orchestra.run.smithi099.stderr: remote markers: [, 00000000002.3584.6, 00000000002.3906.6, 00000000001.2027.6, 00000000002.54.6, 00000000002.2021.6, , , 00000000002.2013.6, 00000000005.2299.6, 00000000004.2018.6] 2023-05-31T20:48:04.218 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:48:04.214+0000 7f6685c72780 1 waiting for incremental sync to catch up: 2023-05-31T20:48:04.228 INFO:teuthology.orchestra.run.smithi099.stderr: local status: [, 00000000002.3584.6, 00000000002.3906.6, 00000000001.2027.6, 00000000002.54.6, 00000000002.2021.6, , , 00000000002.2013.6, 00000000004.2294.6, 00000000004.2018.6] 2023-05-31T20:48:04.229 INFO:teuthology.orchestra.run.smithi099.stderr: remote markers: [, 00000000002.3584.6, 00000000002.3906.6, 00000000001.2027.6, 00000000002.54.6, 00000000002.2021.6, , , 00000000002.2013.6, 00000000005.2299.6, 00000000004.2018.6] 2023-05-31T20:48:09.219 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:48:09.215+0000 7f6685c72780 1 bucket sync caught up with source: 2023-05-31T20:48:09.220 INFO:teuthology.orchestra.run.smithi099.stderr: local status: [, 00000000002.3584.6, 00000000002.3906.6, 00000000001.2027.6, 00000000002.54.6, 00000000002.2021.6, , , 00000000002.2013.6, 00000000005.2299.6, 00000000004.2018.6] 2023-05-31T20:48:09.220 INFO:teuthology.orchestra.run.smithi099.stderr: remote markers: [, 00000000002.3584.6, 00000000002.3906.6, 00000000001.2027.6, 00000000002.54.6, 00000000002.2021.6, , , 00000000002.2013.6, 00000000005.2299.6, 00000000004.2018.6] 2023-05-31T20:48:09.220 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:48:09.215+0000 7f6685c72780 0 bucket checkpoint complete 2023-05-31T20:48:09.238 INFO:tasks.rgw_multi.zone_rados:comparing bucket=pklxkd-59 zones={a2, a1} 2023-05-31T20:48:09.238 DEBUG:boto:path=/pklxkd-59/ 2023-05-31T20:48:09.239 DEBUG:boto:auth_path=/pklxkd-59/ 2023-05-31T20:48:09.240 DEBUG:boto:Method: HEAD 2023-05-31T20:48:09.241 DEBUG:boto:Path: /pklxkd-59/ 2023-05-31T20:48:09.241 DEBUG:boto:Data: 2023-05-31T20:48:09.242 DEBUG:boto:Headers: {} 2023-05-31T20:48:09.242 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:48:09.243 DEBUG:boto:Port: 8001 2023-05-31T20:48:09.244 DEBUG:boto:Params: {} 2023-05-31T20:48:09.245 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8001} 2023-05-31T20:48:09.245 DEBUG:boto:Token: None 2023-05-31T20:48:09.246 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:48:09 GMT /pklxkd-59/ 2023-05-31T20:48:09.247 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:G2nq4/HNTac1y4Qu7+YgQThZoFs= 2023-05-31T20:48:09.248 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:48:09 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:G2nq4/HNTac1y4Qu7+YgQThZoFs=', 'Content-Length': '0'} 2023-05-31T20:48:09.252 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '20'), ('X-RGW-Bytes-Used', '60'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000dab4c947bb4577a3-006477b289-1c17-a2'), ('Content-Length', '0'), ('Date', 'Wed, 31 May 2023 20:48:09 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:48:09.253 DEBUG:boto:path=/pklxkd-59/ 2023-05-31T20:48:09.253 DEBUG:boto:auth_path=/pklxkd-59/ 2023-05-31T20:48:09.254 DEBUG:boto:Method: HEAD 2023-05-31T20:48:09.255 DEBUG:boto:Path: /pklxkd-59/ 2023-05-31T20:48:09.255 DEBUG:boto:Data: 2023-05-31T20:48:09.256 DEBUG:boto:Headers: {} 2023-05-31T20:48:09.257 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:48:09.257 DEBUG:boto:Port: 8000 2023-05-31T20:48:09.258 DEBUG:boto:Params: {} 2023-05-31T20:48:09.259 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2023-05-31T20:48:09.259 DEBUG:boto:Token: None 2023-05-31T20:48:09.260 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:48:09 GMT /pklxkd-59/ 2023-05-31T20:48:09.261 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:G2nq4/HNTac1y4Qu7+YgQThZoFs= 2023-05-31T20:48:09.262 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:48:09 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:G2nq4/HNTac1y4Qu7+YgQThZoFs=', 'Content-Length': '0'} 2023-05-31T20:48:09.266 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '20'), ('X-RGW-Bytes-Used', '60'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000f7d330e83516ed0c-006477b289-1630-a1'), ('Content-Length', '0'), ('Date', 'Wed, 31 May 2023 20:48:09 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:48:09.266 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2023-05-31T20:48:09.267 DEBUG:boto:path=/pklxkd-59/ 2023-05-31T20:48:09.268 DEBUG:boto:auth_path=/pklxkd-59/ 2023-05-31T20:48:09.268 DEBUG:boto:path=/pklxkd-59/?versions&max-keys=999 2023-05-31T20:48:09.269 DEBUG:boto:auth_path=/pklxkd-59/?versions&max-keys=999 2023-05-31T20:48:09.270 DEBUG:boto:Method: GET 2023-05-31T20:48:09.270 DEBUG:boto:Path: /pklxkd-59/?versions&max-keys=999 2023-05-31T20:48:09.271 DEBUG:boto:Data: 2023-05-31T20:48:09.272 DEBUG:boto:Headers: {} 2023-05-31T20:48:09.272 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:48:09.273 DEBUG:boto:Port: 8001 2023-05-31T20:48:09.274 DEBUG:boto:Params: {} 2023-05-31T20:48:09.274 DEBUG:boto:Token: None 2023-05-31T20:48:09.275 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:48:09 GMT /pklxkd-59/?versions 2023-05-31T20:48:09.276 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:YbJNSMPTdkxpfXGXk8jQWRUsIBI= 2023-05-31T20:48:09.276 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:48:09 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:YbJNSMPTdkxpfXGXk8jQWRUsIBI=', 'Content-Length': '0'} 2023-05-31T20:48:09.279 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000005be9e24e406afb11-006477b289-1c17-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:48:09 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:48:09.280 DEBUG:boto:b'pklxkd-59999falseobj11nulltrue2023-05-31T20:39:26.427Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj115nulltrue2023-05-31T20:46:33.958Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj17nulltrue2023-05-31T20:43:07.977Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj19nulltrue2023-05-31T20:44:50.557Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj22nulltrue2023-05-31T20:39:26.473Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj23nulltrue2023-05-31T20:46:33.813Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj33nulltrue2023-05-31T20:39:26.534Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj34nulltrue2023-05-31T20:43:07.994Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj38nulltrue2023-05-31T20:44:50.610Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj44nulltrue2023-05-31T20:39:26.565Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj46nulltrue2023-05-31T20:46:33.830Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj51nulltrue2023-05-31T20:43:08.011Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj55nulltrue2023-05-31T20:39:26.630Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj57nulltrue2023-05-31T20:44:50.627Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj68nulltrue2023-05-31T20:43:08.066Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj69nulltrue2023-05-31T20:46:33.886Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj76nulltrue2023-05-31T20:44:50.682Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj85nulltrue2023-05-31T20:43:08.086Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj92nulltrue2023-05-31T20:46:33.904Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj95nulltrue2023-05-31T20:44:50.705Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:48:09.283 DEBUG:tasks.rgw_multi.zone_rados:o=obj11 2023-05-31T20:48:09.283 DEBUG:tasks.rgw_multi.zone_rados:o=obj115 2023-05-31T20:48:09.284 DEBUG:tasks.rgw_multi.zone_rados:o=obj17 2023-05-31T20:48:09.285 DEBUG:tasks.rgw_multi.zone_rados:o=obj19 2023-05-31T20:48:09.285 DEBUG:tasks.rgw_multi.zone_rados:o=obj22 2023-05-31T20:48:09.286 DEBUG:tasks.rgw_multi.zone_rados:o=obj23 2023-05-31T20:48:09.286 DEBUG:tasks.rgw_multi.zone_rados:o=obj33 2023-05-31T20:48:09.287 DEBUG:tasks.rgw_multi.zone_rados:o=obj34 2023-05-31T20:48:09.287 DEBUG:tasks.rgw_multi.zone_rados:o=obj38 2023-05-31T20:48:09.288 DEBUG:tasks.rgw_multi.zone_rados:o=obj44 2023-05-31T20:48:09.289 DEBUG:tasks.rgw_multi.zone_rados:o=obj46 2023-05-31T20:48:09.289 DEBUG:tasks.rgw_multi.zone_rados:o=obj51 2023-05-31T20:48:09.290 DEBUG:tasks.rgw_multi.zone_rados:o=obj55 2023-05-31T20:48:09.290 DEBUG:tasks.rgw_multi.zone_rados:o=obj57 2023-05-31T20:48:09.291 DEBUG:tasks.rgw_multi.zone_rados:o=obj68 2023-05-31T20:48:09.291 DEBUG:tasks.rgw_multi.zone_rados:o=obj69 2023-05-31T20:48:09.292 DEBUG:tasks.rgw_multi.zone_rados:o=obj76 2023-05-31T20:48:09.293 DEBUG:tasks.rgw_multi.zone_rados:o=obj85 2023-05-31T20:48:09.293 DEBUG:tasks.rgw_multi.zone_rados:o=obj92 2023-05-31T20:48:09.294 DEBUG:tasks.rgw_multi.zone_rados:o=obj95 2023-05-31T20:48:09.294 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2023-05-31T20:48:09.295 DEBUG:boto:path=/pklxkd-59/ 2023-05-31T20:48:09.296 DEBUG:boto:auth_path=/pklxkd-59/ 2023-05-31T20:48:09.296 DEBUG:boto:path=/pklxkd-59/?versions&max-keys=999 2023-05-31T20:48:09.297 DEBUG:boto:auth_path=/pklxkd-59/?versions&max-keys=999 2023-05-31T20:48:09.297 DEBUG:boto:Method: GET 2023-05-31T20:48:09.298 DEBUG:boto:Path: /pklxkd-59/?versions&max-keys=999 2023-05-31T20:48:09.299 DEBUG:boto:Data: 2023-05-31T20:48:09.299 DEBUG:boto:Headers: {} 2023-05-31T20:48:09.300 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:48:09.301 DEBUG:boto:Port: 8000 2023-05-31T20:48:09.301 DEBUG:boto:Params: {} 2023-05-31T20:48:09.301 DEBUG:boto:Token: None 2023-05-31T20:48:09.302 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:48:09 GMT /pklxkd-59/?versions 2023-05-31T20:48:09.302 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:YbJNSMPTdkxpfXGXk8jQWRUsIBI= 2023-05-31T20:48:09.303 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:48:09 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:YbJNSMPTdkxpfXGXk8jQWRUsIBI=', 'Content-Length': '0'} 2023-05-31T20:48:09.306 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000698c4d3b4e15388f-006477b289-1630-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:48:09 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:48:09.306 DEBUG:boto:b'pklxkd-59999falseobj11nulltrue2023-05-31T20:39:26.427Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj115nulltrue2023-05-31T20:46:33.958Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj17nulltrue2023-05-31T20:43:07.977Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj19nulltrue2023-05-31T20:44:50.557Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj22nulltrue2023-05-31T20:39:26.473Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj23nulltrue2023-05-31T20:46:33.813Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj33nulltrue2023-05-31T20:39:26.534Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj34nulltrue2023-05-31T20:43:07.994Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj38nulltrue2023-05-31T20:44:50.610Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj44nulltrue2023-05-31T20:39:26.565Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj46nulltrue2023-05-31T20:46:33.830Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj51nulltrue2023-05-31T20:43:08.011Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj55nulltrue2023-05-31T20:39:26.630Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj57nulltrue2023-05-31T20:44:50.627Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj68nulltrue2023-05-31T20:43:08.066Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj69nulltrue2023-05-31T20:46:33.886Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj76nulltrue2023-05-31T20:44:50.682Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj85nulltrue2023-05-31T20:43:08.086Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj92nulltrue2023-05-31T20:46:33.904Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj95nulltrue2023-05-31T20:44:50.705Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:48:09.308 DEBUG:tasks.rgw_multi.zone_rados:o=obj11 2023-05-31T20:48:09.308 DEBUG:tasks.rgw_multi.zone_rados:o=obj115 2023-05-31T20:48:09.308 DEBUG:tasks.rgw_multi.zone_rados:o=obj17 2023-05-31T20:48:09.309 DEBUG:tasks.rgw_multi.zone_rados:o=obj19 2023-05-31T20:48:09.309 DEBUG:tasks.rgw_multi.zone_rados:o=obj22 2023-05-31T20:48:09.309 DEBUG:tasks.rgw_multi.zone_rados:o=obj23 2023-05-31T20:48:09.310 DEBUG:tasks.rgw_multi.zone_rados:o=obj33 2023-05-31T20:48:09.310 DEBUG:tasks.rgw_multi.zone_rados:o=obj34 2023-05-31T20:48:09.310 DEBUG:tasks.rgw_multi.zone_rados:o=obj38 2023-05-31T20:48:09.311 DEBUG:tasks.rgw_multi.zone_rados:o=obj44 2023-05-31T20:48:09.311 DEBUG:tasks.rgw_multi.zone_rados:o=obj46 2023-05-31T20:48:09.311 DEBUG:tasks.rgw_multi.zone_rados:o=obj51 2023-05-31T20:48:09.311 DEBUG:tasks.rgw_multi.zone_rados:o=obj55 2023-05-31T20:48:09.312 DEBUG:tasks.rgw_multi.zone_rados:o=obj57 2023-05-31T20:48:09.312 DEBUG:tasks.rgw_multi.zone_rados:o=obj68 2023-05-31T20:48:09.312 DEBUG:tasks.rgw_multi.zone_rados:o=obj69 2023-05-31T20:48:09.313 DEBUG:tasks.rgw_multi.zone_rados:o=obj76 2023-05-31T20:48:09.313 DEBUG:tasks.rgw_multi.zone_rados:o=obj85 2023-05-31T20:48:09.313 DEBUG:tasks.rgw_multi.zone_rados:o=obj92 2023-05-31T20:48:09.313 DEBUG:tasks.rgw_multi.zone_rados:o=obj95 2023-05-31T20:48:09.314 DEBUG:boto:path=/pklxkd-59/ 2023-05-31T20:48:09.314 DEBUG:boto:auth_path=/pklxkd-59/ 2023-05-31T20:48:09.314 DEBUG:boto:path=/pklxkd-59/?versions&max-keys=999 2023-05-31T20:48:09.315 DEBUG:boto:auth_path=/pklxkd-59/?versions&max-keys=999 2023-05-31T20:48:09.315 DEBUG:boto:Method: GET 2023-05-31T20:48:09.316 DEBUG:boto:Path: /pklxkd-59/?versions&max-keys=999 2023-05-31T20:48:09.316 DEBUG:boto:Data: 2023-05-31T20:48:09.316 DEBUG:boto:Headers: {} 2023-05-31T20:48:09.317 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:48:09.317 DEBUG:boto:Port: 8001 2023-05-31T20:48:09.317 DEBUG:boto:Params: {} 2023-05-31T20:48:09.318 DEBUG:boto:Token: None 2023-05-31T20:48:09.318 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:48:09 GMT /pklxkd-59/?versions 2023-05-31T20:48:09.318 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:YbJNSMPTdkxpfXGXk8jQWRUsIBI= 2023-05-31T20:48:09.319 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:48:09 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:YbJNSMPTdkxpfXGXk8jQWRUsIBI=', 'Content-Length': '0'} 2023-05-31T20:48:09.321 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000008467154285af8013-006477b289-1c17-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:48:09 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:48:09.322 DEBUG:boto:b'pklxkd-59999falseobj11nulltrue2023-05-31T20:39:26.427Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj115nulltrue2023-05-31T20:46:33.958Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj17nulltrue2023-05-31T20:43:07.977Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj19nulltrue2023-05-31T20:44:50.557Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj22nulltrue2023-05-31T20:39:26.473Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj23nulltrue2023-05-31T20:46:33.813Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj33nulltrue2023-05-31T20:39:26.534Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj34nulltrue2023-05-31T20:43:07.994Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj38nulltrue2023-05-31T20:44:50.610Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj44nulltrue2023-05-31T20:39:26.565Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj46nulltrue2023-05-31T20:46:33.830Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj51nulltrue2023-05-31T20:43:08.011Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj55nulltrue2023-05-31T20:39:26.630Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj57nulltrue2023-05-31T20:44:50.627Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj68nulltrue2023-05-31T20:43:08.066Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj69nulltrue2023-05-31T20:46:33.886Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj76nulltrue2023-05-31T20:44:50.682Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj85nulltrue2023-05-31T20:43:08.086Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj92nulltrue2023-05-31T20:46:33.904Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj95nulltrue2023-05-31T20:44:50.705Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:48:09.323 DEBUG:boto:path=/pklxkd-59/ 2023-05-31T20:48:09.324 DEBUG:boto:auth_path=/pklxkd-59/ 2023-05-31T20:48:09.324 DEBUG:boto:path=/pklxkd-59/?versions&max-keys=999 2023-05-31T20:48:09.324 DEBUG:boto:auth_path=/pklxkd-59/?versions&max-keys=999 2023-05-31T20:48:09.325 DEBUG:boto:Method: GET 2023-05-31T20:48:09.325 DEBUG:boto:Path: /pklxkd-59/?versions&max-keys=999 2023-05-31T20:48:09.325 DEBUG:boto:Data: 2023-05-31T20:48:09.326 DEBUG:boto:Headers: {} 2023-05-31T20:48:09.326 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:48:09.326 DEBUG:boto:Port: 8000 2023-05-31T20:48:09.327 DEBUG:boto:Params: {} 2023-05-31T20:48:09.327 DEBUG:boto:Token: None 2023-05-31T20:48:09.327 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:48:09 GMT /pklxkd-59/?versions 2023-05-31T20:48:09.328 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:YbJNSMPTdkxpfXGXk8jQWRUsIBI= 2023-05-31T20:48:09.328 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:48:09 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:YbJNSMPTdkxpfXGXk8jQWRUsIBI=', 'Content-Length': '0'} 2023-05-31T20:48:09.332 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000008c4cc98059515c19-006477b289-1630-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:48:09 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:48:09.332 DEBUG:boto:b'pklxkd-59999falseobj11nulltrue2023-05-31T20:39:26.427Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj115nulltrue2023-05-31T20:46:33.958Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj17nulltrue2023-05-31T20:43:07.977Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj19nulltrue2023-05-31T20:44:50.557Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj22nulltrue2023-05-31T20:39:26.473Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj23nulltrue2023-05-31T20:46:33.813Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj33nulltrue2023-05-31T20:39:26.534Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj34nulltrue2023-05-31T20:43:07.994Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj38nulltrue2023-05-31T20:44:50.610Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj44nulltrue2023-05-31T20:39:26.565Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj46nulltrue2023-05-31T20:46:33.830Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj51nulltrue2023-05-31T20:43:08.011Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj55nulltrue2023-05-31T20:39:26.630Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj57nulltrue2023-05-31T20:44:50.627Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj68nulltrue2023-05-31T20:43:08.066Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj69nulltrue2023-05-31T20:46:33.886Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj76nulltrue2023-05-31T20:44:50.682Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj85nulltrue2023-05-31T20:43:08.086Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj92nulltrue2023-05-31T20:46:33.904Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj95nulltrue2023-05-31T20:44:50.705Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:48:09.334 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj11 2023-05-31T20:48:09.334 DEBUG:boto:path=/pklxkd-59/obj11 2023-05-31T20:48:09.335 DEBUG:boto:auth_path=/pklxkd-59/obj11 2023-05-31T20:48:09.335 DEBUG:boto:path=/pklxkd-59/obj11?versionId=null 2023-05-31T20:48:09.335 DEBUG:boto:auth_path=/pklxkd-59/obj11?versionId=null 2023-05-31T20:48:09.336 DEBUG:boto:Method: GET 2023-05-31T20:48:09.336 DEBUG:boto:Path: /pklxkd-59/obj11?versionId=null 2023-05-31T20:48:09.336 DEBUG:boto:Data: 2023-05-31T20:48:09.337 DEBUG:boto:Headers: {} 2023-05-31T20:48:09.337 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:48:09.337 DEBUG:boto:Port: 8001 2023-05-31T20:48:09.338 DEBUG:boto:Params: {} 2023-05-31T20:48:09.338 DEBUG:boto:Token: None 2023-05-31T20:48:09.339 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:48:09 GMT /pklxkd-59/obj11?versionId=null 2023-05-31T20:48:09.339 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:ssxjhfcXmDshToIXIgNAO6z9BXU= 2023-05-31T20:48:09.339 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:48:09 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:ssxjhfcXmDshToIXIgNAO6z9BXU=', 'Content-Length': '0'} 2023-05-31T20:48:09.342 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:39:26 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003fcf9ad12e3cb9cf-006477b289-1c17-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:48:09 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:48:09.342 DEBUG:boto:path=/pklxkd-59/obj11 2023-05-31T20:48:09.343 DEBUG:boto:auth_path=/pklxkd-59/obj11 2023-05-31T20:48:09.343 DEBUG:boto:path=/pklxkd-59/obj11?versionId=null 2023-05-31T20:48:09.343 DEBUG:boto:auth_path=/pklxkd-59/obj11?versionId=null 2023-05-31T20:48:09.344 DEBUG:boto:Method: GET 2023-05-31T20:48:09.344 DEBUG:boto:Path: /pklxkd-59/obj11?versionId=null 2023-05-31T20:48:09.344 DEBUG:boto:Data: 2023-05-31T20:48:09.345 DEBUG:boto:Headers: {} 2023-05-31T20:48:09.345 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:48:09.345 DEBUG:boto:Port: 8000 2023-05-31T20:48:09.346 DEBUG:boto:Params: {} 2023-05-31T20:48:09.346 DEBUG:boto:Token: None 2023-05-31T20:48:09.347 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:48:09 GMT /pklxkd-59/obj11?versionId=null 2023-05-31T20:48:09.347 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:ssxjhfcXmDshToIXIgNAO6z9BXU= 2023-05-31T20:48:09.347 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:48:09 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:ssxjhfcXmDshToIXIgNAO6z9BXU=', 'Content-Length': '0'} 2023-05-31T20:48:09.350 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:39:26 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f95ac24505a948da-006477b289-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:48:09 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:48:09.392 DEBUG:boto:path=/pklxkd-59/obj11 2023-05-31T20:48:09.393 DEBUG:boto:auth_path=/pklxkd-59/obj11 2023-05-31T20:48:09.393 DEBUG:boto:path=/pklxkd-59/obj11?versionId=null 2023-05-31T20:48:09.394 DEBUG:boto:auth_path=/pklxkd-59/obj11?versionId=null 2023-05-31T20:48:09.394 DEBUG:boto:Method: HEAD 2023-05-31T20:48:09.394 DEBUG:boto:Path: /pklxkd-59/obj11?versionId=null 2023-05-31T20:48:09.395 DEBUG:boto:Data: 2023-05-31T20:48:09.395 DEBUG:boto:Headers: {} 2023-05-31T20:48:09.395 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:48:09.396 DEBUG:boto:Port: 8001 2023-05-31T20:48:09.396 DEBUG:boto:Params: {} 2023-05-31T20:48:09.396 DEBUG:boto:Token: None 2023-05-31T20:48:09.397 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:48:09 GMT /pklxkd-59/obj11?versionId=null 2023-05-31T20:48:09.397 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:6dNtn802WM+s4IO5QiWV7Dpa5CI= 2023-05-31T20:48:09.398 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:48:09 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:6dNtn802WM+s4IO5QiWV7Dpa5CI=', 'Content-Length': '0'} 2023-05-31T20:48:09.400 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:39:26 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001a1da44837a2f2e5-006477b289-1c17-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:48:09 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:48:09.400 DEBUG:boto:path=/pklxkd-59/obj11 2023-05-31T20:48:09.401 DEBUG:boto:auth_path=/pklxkd-59/obj11 2023-05-31T20:48:09.401 DEBUG:boto:path=/pklxkd-59/obj11?versionId=null 2023-05-31T20:48:09.401 DEBUG:boto:auth_path=/pklxkd-59/obj11?versionId=null 2023-05-31T20:48:09.402 DEBUG:boto:Method: HEAD 2023-05-31T20:48:09.402 DEBUG:boto:Path: /pklxkd-59/obj11?versionId=null 2023-05-31T20:48:09.402 DEBUG:boto:Data: 2023-05-31T20:48:09.403 DEBUG:boto:Headers: {} 2023-05-31T20:48:09.403 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:48:09.403 DEBUG:boto:Port: 8000 2023-05-31T20:48:09.404 DEBUG:boto:Params: {} 2023-05-31T20:48:09.404 DEBUG:boto:Token: None 2023-05-31T20:48:09.405 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:48:09 GMT /pklxkd-59/obj11?versionId=null 2023-05-31T20:48:09.405 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:6dNtn802WM+s4IO5QiWV7Dpa5CI= 2023-05-31T20:48:09.405 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:48:09 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:6dNtn802WM+s4IO5QiWV7Dpa5CI=', 'Content-Length': '0'} 2023-05-31T20:48:09.408 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:39:26 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000173d54a19a0974ab-006477b289-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:48:09 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:48:09.409 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj11 2023-05-31T20:48:09.409 DEBUG:boto:path=/pklxkd-59/obj11 2023-05-31T20:48:09.409 DEBUG:boto:auth_path=/pklxkd-59/obj11 2023-05-31T20:48:09.410 DEBUG:boto:path=/pklxkd-59/obj11?versionId=null 2023-05-31T20:48:09.410 DEBUG:boto:auth_path=/pklxkd-59/obj11?versionId=null 2023-05-31T20:48:09.410 DEBUG:boto:Method: GET 2023-05-31T20:48:09.411 DEBUG:boto:Path: /pklxkd-59/obj11?versionId=null 2023-05-31T20:48:09.411 DEBUG:boto:Data: 2023-05-31T20:48:09.411 DEBUG:boto:Headers: {} 2023-05-31T20:48:09.412 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:48:09.412 DEBUG:boto:Port: 8001 2023-05-31T20:48:09.413 DEBUG:boto:Params: {} 2023-05-31T20:48:09.413 DEBUG:boto:Token: None 2023-05-31T20:48:09.413 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:48:09 GMT /pklxkd-59/obj11?versionId=null 2023-05-31T20:48:09.414 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:ssxjhfcXmDshToIXIgNAO6z9BXU= 2023-05-31T20:48:09.414 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:48:09 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:ssxjhfcXmDshToIXIgNAO6z9BXU=', 'Content-Length': '0'} 2023-05-31T20:48:09.416 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:39:26 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f972d7b15f95ffd0-006477b289-1c17-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:48:09 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:48:09.417 DEBUG:boto:path=/pklxkd-59/obj11 2023-05-31T20:48:09.417 DEBUG:boto:auth_path=/pklxkd-59/obj11 2023-05-31T20:48:09.418 DEBUG:boto:path=/pklxkd-59/obj11?versionId=null 2023-05-31T20:48:09.418 DEBUG:boto:auth_path=/pklxkd-59/obj11?versionId=null 2023-05-31T20:48:09.418 DEBUG:boto:Method: GET 2023-05-31T20:48:09.419 DEBUG:boto:Path: /pklxkd-59/obj11?versionId=null 2023-05-31T20:48:09.419 DEBUG:boto:Data: 2023-05-31T20:48:09.419 DEBUG:boto:Headers: {} 2023-05-31T20:48:09.420 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:48:09.420 DEBUG:boto:Port: 8000 2023-05-31T20:48:09.421 DEBUG:boto:Params: {} 2023-05-31T20:48:09.421 DEBUG:boto:Token: None 2023-05-31T20:48:09.421 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:48:09 GMT /pklxkd-59/obj11?versionId=null 2023-05-31T20:48:09.422 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:ssxjhfcXmDshToIXIgNAO6z9BXU= 2023-05-31T20:48:09.422 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:48:09 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:ssxjhfcXmDshToIXIgNAO6z9BXU=', 'Content-Length': '0'} 2023-05-31T20:48:09.424 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:39:26 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001ebbd2bdff228618-006477b289-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:48:09 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:48:09.425 DEBUG:boto:path=/pklxkd-59/ 2023-05-31T20:48:09.426 DEBUG:boto:auth_path=/pklxkd-59/ 2023-05-31T20:48:09.426 DEBUG:boto:path=/pklxkd-59/?prefix=obj11 2023-05-31T20:48:09.426 DEBUG:boto:auth_path=/pklxkd-59/?prefix=obj11 2023-05-31T20:48:09.427 DEBUG:boto:Method: GET 2023-05-31T20:48:09.427 DEBUG:boto:Path: /pklxkd-59/?prefix=obj11 2023-05-31T20:48:09.427 DEBUG:boto:Data: 2023-05-31T20:48:09.428 DEBUG:boto:Headers: {} 2023-05-31T20:48:09.428 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:48:09.428 DEBUG:boto:Port: 8001 2023-05-31T20:48:09.429 DEBUG:boto:Params: {} 2023-05-31T20:48:09.429 DEBUG:boto:Token: None 2023-05-31T20:48:09.430 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:48:09 GMT /pklxkd-59/ 2023-05-31T20:48:09.430 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:9STbDo2uP4g1xSEvz1y1wz540ZQ= 2023-05-31T20:48:09.430 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:48:09 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:9STbDo2uP4g1xSEvz1y1wz540ZQ=', 'Content-Length': '0'} 2023-05-31T20:48:09.433 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000008cdf6d5f0d8f9680-006477b289-1c17-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:48:09 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:48:09.433 DEBUG:boto:b'pklxkd-59obj111000falseobj112023-05-31T20:39:26.427Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj1152023-05-31T20:46:33.958Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:48:09.434 DEBUG:boto:path=/pklxkd-59/ 2023-05-31T20:48:09.435 DEBUG:boto:auth_path=/pklxkd-59/ 2023-05-31T20:48:09.435 DEBUG:boto:path=/pklxkd-59/?prefix=obj11 2023-05-31T20:48:09.435 DEBUG:boto:auth_path=/pklxkd-59/?prefix=obj11 2023-05-31T20:48:09.436 DEBUG:boto:Method: GET 2023-05-31T20:48:09.436 DEBUG:boto:Path: /pklxkd-59/?prefix=obj11 2023-05-31T20:48:09.436 DEBUG:boto:Data: 2023-05-31T20:48:09.437 DEBUG:boto:Headers: {} 2023-05-31T20:48:09.437 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:48:09.437 DEBUG:boto:Port: 8000 2023-05-31T20:48:09.438 DEBUG:boto:Params: {} 2023-05-31T20:48:09.438 DEBUG:boto:Token: None 2023-05-31T20:48:09.439 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:48:09 GMT /pklxkd-59/ 2023-05-31T20:48:09.439 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:9STbDo2uP4g1xSEvz1y1wz540ZQ= 2023-05-31T20:48:09.439 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:48:09 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:9STbDo2uP4g1xSEvz1y1wz540ZQ=', 'Content-Length': '0'} 2023-05-31T20:48:09.443 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a6912a6ceeaec28d-006477b289-1630-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:48:09 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:48:09.443 DEBUG:boto:b'pklxkd-59obj111000falseobj112023-05-31T20:39:26.427Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj1152023-05-31T20:46:33.958Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:48:09.444 DEBUG:boto:path=/pklxkd-59/obj11 2023-05-31T20:48:09.444 DEBUG:boto:auth_path=/pklxkd-59/obj11 2023-05-31T20:48:09.445 DEBUG:boto:Method: HEAD 2023-05-31T20:48:09.445 DEBUG:boto:Path: /pklxkd-59/obj11 2023-05-31T20:48:09.445 DEBUG:boto:Data: 2023-05-31T20:48:09.446 DEBUG:boto:Headers: {} 2023-05-31T20:48:09.446 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:48:09.446 DEBUG:boto:Port: 8001 2023-05-31T20:48:09.447 DEBUG:boto:Params: {} 2023-05-31T20:48:09.447 DEBUG:boto:Token: None 2023-05-31T20:48:09.447 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:48:09 GMT /pklxkd-59/obj11 2023-05-31T20:48:09.448 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:P4AsbmLQIPvS7iUudCFtA7vcKOM= 2023-05-31T20:48:09.448 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:48:09 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:P4AsbmLQIPvS7iUudCFtA7vcKOM=', 'Content-Length': '0'} 2023-05-31T20:48:09.451 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:39:26 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007279c16ee52f74c0-006477b289-1c17-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:48:09 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:48:09.451 DEBUG:boto:path=/pklxkd-59/obj11 2023-05-31T20:48:09.452 DEBUG:boto:auth_path=/pklxkd-59/obj11 2023-05-31T20:48:09.452 DEBUG:boto:Method: HEAD 2023-05-31T20:48:09.453 DEBUG:boto:Path: /pklxkd-59/obj11 2023-05-31T20:48:09.453 DEBUG:boto:Data: 2023-05-31T20:48:09.453 DEBUG:boto:Headers: {} 2023-05-31T20:48:09.454 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:48:09.454 DEBUG:boto:Port: 8000 2023-05-31T20:48:09.454 DEBUG:boto:Params: {} 2023-05-31T20:48:09.455 DEBUG:boto:Token: None 2023-05-31T20:48:09.455 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:48:09 GMT /pklxkd-59/obj11 2023-05-31T20:48:09.455 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:P4AsbmLQIPvS7iUudCFtA7vcKOM= 2023-05-31T20:48:09.456 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:48:09 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:P4AsbmLQIPvS7iUudCFtA7vcKOM=', 'Content-Length': '0'} 2023-05-31T20:48:09.458 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:39:26 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000044da3528104822c8-006477b289-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:48:09 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:48:09.458 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj11 2023-05-31T20:48:09.459 DEBUG:boto:path=/pklxkd-59/obj11 2023-05-31T20:48:09.459 DEBUG:boto:auth_path=/pklxkd-59/obj11 2023-05-31T20:48:09.459 DEBUG:boto:Method: GET 2023-05-31T20:48:09.460 DEBUG:boto:Path: /pklxkd-59/obj11 2023-05-31T20:48:09.460 DEBUG:boto:Data: 2023-05-31T20:48:09.460 DEBUG:boto:Headers: {} 2023-05-31T20:48:09.461 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:48:09.461 DEBUG:boto:Port: 8001 2023-05-31T20:48:09.461 DEBUG:boto:Params: {} 2023-05-31T20:48:09.462 DEBUG:boto:Token: None 2023-05-31T20:48:09.462 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:48:09 GMT /pklxkd-59/obj11 2023-05-31T20:48:09.463 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:4ZGMpPBemu205AJuzX0PF6zakd8= 2023-05-31T20:48:09.463 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:48:09 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:4ZGMpPBemu205AJuzX0PF6zakd8=', 'Content-Length': '0'} 2023-05-31T20:48:09.465 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:39:26 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d4f8a4b74c2cee7c-006477b289-1c17-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:48:09 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:48:09.508 DEBUG:boto:path=/pklxkd-59/obj11 2023-05-31T20:48:09.509 DEBUG:boto:auth_path=/pklxkd-59/obj11 2023-05-31T20:48:09.509 DEBUG:boto:Method: GET 2023-05-31T20:48:09.509 DEBUG:boto:Path: /pklxkd-59/obj11 2023-05-31T20:48:09.510 DEBUG:boto:Data: 2023-05-31T20:48:09.510 DEBUG:boto:Headers: {} 2023-05-31T20:48:09.511 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:48:09.511 DEBUG:boto:Port: 8000 2023-05-31T20:48:09.511 DEBUG:boto:Params: {} 2023-05-31T20:48:09.512 DEBUG:boto:Token: None 2023-05-31T20:48:09.512 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:48:09 GMT /pklxkd-59/obj11 2023-05-31T20:48:09.513 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:4ZGMpPBemu205AJuzX0PF6zakd8= 2023-05-31T20:48:09.513 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:48:09 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:4ZGMpPBemu205AJuzX0PF6zakd8=', 'Content-Length': '0'} 2023-05-31T20:48:09.515 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:39:26 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006c6b388334eb1651-006477b289-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:48:09 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:48:09.516 DEBUG:boto:path=/pklxkd-59/ 2023-05-31T20:48:09.516 DEBUG:boto:auth_path=/pklxkd-59/ 2023-05-31T20:48:09.516 DEBUG:boto:path=/pklxkd-59/?prefix=obj11 2023-05-31T20:48:09.517 DEBUG:boto:auth_path=/pklxkd-59/?prefix=obj11 2023-05-31T20:48:09.517 DEBUG:boto:Method: GET 2023-05-31T20:48:09.517 DEBUG:boto:Path: /pklxkd-59/?prefix=obj11 2023-05-31T20:48:09.518 DEBUG:boto:Data: 2023-05-31T20:48:09.518 DEBUG:boto:Headers: {} 2023-05-31T20:48:09.519 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:48:09.519 DEBUG:boto:Port: 8001 2023-05-31T20:48:09.519 DEBUG:boto:Params: {} 2023-05-31T20:48:09.520 DEBUG:boto:Token: None 2023-05-31T20:48:09.520 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:48:09 GMT /pklxkd-59/ 2023-05-31T20:48:09.520 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:9STbDo2uP4g1xSEvz1y1wz540ZQ= 2023-05-31T20:48:09.521 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:48:09 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:9STbDo2uP4g1xSEvz1y1wz540ZQ=', 'Content-Length': '0'} 2023-05-31T20:48:09.523 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000021ffb586adcef28c-006477b289-1c17-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:48:09 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:48:09.524 DEBUG:boto:b'pklxkd-59obj111000falseobj112023-05-31T20:39:26.427Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj1152023-05-31T20:46:33.958Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:48:09.524 DEBUG:boto:path=/pklxkd-59/ 2023-05-31T20:48:09.525 DEBUG:boto:auth_path=/pklxkd-59/ 2023-05-31T20:48:09.525 DEBUG:boto:path=/pklxkd-59/?prefix=obj11 2023-05-31T20:48:09.526 DEBUG:boto:auth_path=/pklxkd-59/?prefix=obj11 2023-05-31T20:48:09.526 DEBUG:boto:Method: GET 2023-05-31T20:48:09.526 DEBUG:boto:Path: /pklxkd-59/?prefix=obj11 2023-05-31T20:48:09.527 DEBUG:boto:Data: 2023-05-31T20:48:09.527 DEBUG:boto:Headers: {} 2023-05-31T20:48:09.527 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:48:09.528 DEBUG:boto:Port: 8000 2023-05-31T20:48:09.528 DEBUG:boto:Params: {} 2023-05-31T20:48:09.528 DEBUG:boto:Token: None 2023-05-31T20:48:09.529 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:48:09 GMT /pklxkd-59/ 2023-05-31T20:48:09.529 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:9STbDo2uP4g1xSEvz1y1wz540ZQ= 2023-05-31T20:48:09.530 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:48:09 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:9STbDo2uP4g1xSEvz1y1wz540ZQ=', 'Content-Length': '0'} 2023-05-31T20:48:09.533 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000abe4fa7e5c61e193-006477b289-1630-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:48:09 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:48:09.533 DEBUG:boto:b'pklxkd-59obj111000falseobj112023-05-31T20:39:26.427Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj1152023-05-31T20:46:33.958Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:48:09.534 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj115 2023-05-31T20:48:09.534 DEBUG:boto:path=/pklxkd-59/obj115 2023-05-31T20:48:09.535 DEBUG:boto:auth_path=/pklxkd-59/obj115 2023-05-31T20:48:09.535 DEBUG:boto:path=/pklxkd-59/obj115?versionId=null 2023-05-31T20:48:09.535 DEBUG:boto:auth_path=/pklxkd-59/obj115?versionId=null 2023-05-31T20:48:09.536 DEBUG:boto:Method: GET 2023-05-31T20:48:09.536 DEBUG:boto:Path: /pklxkd-59/obj115?versionId=null 2023-05-31T20:48:09.536 DEBUG:boto:Data: 2023-05-31T20:48:09.537 DEBUG:boto:Headers: {} 2023-05-31T20:48:09.537 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:48:09.537 DEBUG:boto:Port: 8001 2023-05-31T20:48:09.538 DEBUG:boto:Params: {} 2023-05-31T20:48:09.538 DEBUG:boto:Token: None 2023-05-31T20:48:09.539 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:48:09 GMT /pklxkd-59/obj115?versionId=null 2023-05-31T20:48:09.539 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:nSjWWODuJcSCWRcHxD6rodJMcGc= 2023-05-31T20:48:09.539 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:48:09 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:nSjWWODuJcSCWRcHxD6rodJMcGc=', 'Content-Length': '0'} 2023-05-31T20:48:09.541 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:46:33 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a0e3ce0414484b60-006477b289-1c17-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:48:09 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:48:09.542 DEBUG:boto:path=/pklxkd-59/obj115 2023-05-31T20:48:09.542 DEBUG:boto:auth_path=/pklxkd-59/obj115 2023-05-31T20:48:09.543 DEBUG:boto:path=/pklxkd-59/obj115?versionId=null 2023-05-31T20:48:09.543 DEBUG:boto:auth_path=/pklxkd-59/obj115?versionId=null 2023-05-31T20:48:09.544 DEBUG:boto:Method: GET 2023-05-31T20:48:09.544 DEBUG:boto:Path: /pklxkd-59/obj115?versionId=null 2023-05-31T20:48:09.544 DEBUG:boto:Data: 2023-05-31T20:48:09.545 DEBUG:boto:Headers: {} 2023-05-31T20:48:09.545 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:48:09.545 DEBUG:boto:Port: 8000 2023-05-31T20:48:09.546 DEBUG:boto:Params: {} 2023-05-31T20:48:09.546 DEBUG:boto:Token: None 2023-05-31T20:48:09.546 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:48:09 GMT /pklxkd-59/obj115?versionId=null 2023-05-31T20:48:09.547 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:nSjWWODuJcSCWRcHxD6rodJMcGc= 2023-05-31T20:48:09.547 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:48:09 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:nSjWWODuJcSCWRcHxD6rodJMcGc=', 'Content-Length': '0'} 2023-05-31T20:48:09.549 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:46:33 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001c07d6b79d249797-006477b289-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:48:09 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:48:09.550 DEBUG:boto:path=/pklxkd-59/obj115 2023-05-31T20:48:09.550 DEBUG:boto:auth_path=/pklxkd-59/obj115 2023-05-31T20:48:09.550 DEBUG:boto:path=/pklxkd-59/obj115?versionId=null 2023-05-31T20:48:09.551 DEBUG:boto:auth_path=/pklxkd-59/obj115?versionId=null 2023-05-31T20:48:09.551 DEBUG:boto:Method: HEAD 2023-05-31T20:48:09.551 DEBUG:boto:Path: /pklxkd-59/obj115?versionId=null 2023-05-31T20:48:09.552 DEBUG:boto:Data: 2023-05-31T20:48:09.552 DEBUG:boto:Headers: {} 2023-05-31T20:48:09.552 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:48:09.553 DEBUG:boto:Port: 8001 2023-05-31T20:48:09.553 DEBUG:boto:Params: {} 2023-05-31T20:48:09.553 DEBUG:boto:Token: None 2023-05-31T20:48:09.554 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:48:09 GMT /pklxkd-59/obj115?versionId=null 2023-05-31T20:48:09.554 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:aJi7Yvhtmk/1Hx+JBp1k1tzbGd8= 2023-05-31T20:48:09.555 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:48:09 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:aJi7Yvhtmk/1Hx+JBp1k1tzbGd8=', 'Content-Length': '0'} 2023-05-31T20:48:09.556 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:46:33 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f26b402c74db1d28-006477b289-1c17-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:48:09 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:48:09.557 DEBUG:boto:path=/pklxkd-59/obj115 2023-05-31T20:48:09.557 DEBUG:boto:auth_path=/pklxkd-59/obj115 2023-05-31T20:48:09.558 DEBUG:boto:path=/pklxkd-59/obj115?versionId=null 2023-05-31T20:48:09.558 DEBUG:boto:auth_path=/pklxkd-59/obj115?versionId=null 2023-05-31T20:48:09.558 DEBUG:boto:Method: HEAD 2023-05-31T20:48:09.559 DEBUG:boto:Path: /pklxkd-59/obj115?versionId=null 2023-05-31T20:48:09.559 DEBUG:boto:Data: 2023-05-31T20:48:09.559 DEBUG:boto:Headers: {} 2023-05-31T20:48:09.560 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:48:09.560 DEBUG:boto:Port: 8000 2023-05-31T20:48:09.560 DEBUG:boto:Params: {} 2023-05-31T20:48:09.561 DEBUG:boto:Token: None 2023-05-31T20:48:09.561 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:48:09 GMT /pklxkd-59/obj115?versionId=null 2023-05-31T20:48:09.561 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:aJi7Yvhtmk/1Hx+JBp1k1tzbGd8= 2023-05-31T20:48:09.562 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:48:09 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:aJi7Yvhtmk/1Hx+JBp1k1tzbGd8=', 'Content-Length': '0'} 2023-05-31T20:48:09.563 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:46:33 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000045864b53515e8ce6-006477b289-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:48:09 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:48:09.564 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj115 2023-05-31T20:48:09.564 DEBUG:boto:path=/pklxkd-59/obj115 2023-05-31T20:48:09.565 DEBUG:boto:auth_path=/pklxkd-59/obj115 2023-05-31T20:48:09.565 DEBUG:boto:path=/pklxkd-59/obj115?versionId=null 2023-05-31T20:48:09.565 DEBUG:boto:auth_path=/pklxkd-59/obj115?versionId=null 2023-05-31T20:48:09.566 DEBUG:boto:Method: GET 2023-05-31T20:48:09.566 DEBUG:boto:Path: /pklxkd-59/obj115?versionId=null 2023-05-31T20:48:09.567 DEBUG:boto:Data: 2023-05-31T20:48:09.567 DEBUG:boto:Headers: {} 2023-05-31T20:48:09.567 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:48:09.568 DEBUG:boto:Port: 8001 2023-05-31T20:48:09.568 DEBUG:boto:Params: {} 2023-05-31T20:48:09.568 DEBUG:boto:Token: None 2023-05-31T20:48:09.569 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:48:09 GMT /pklxkd-59/obj115?versionId=null 2023-05-31T20:48:09.569 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:nSjWWODuJcSCWRcHxD6rodJMcGc= 2023-05-31T20:48:09.570 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:48:09 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:nSjWWODuJcSCWRcHxD6rodJMcGc=', 'Content-Length': '0'} 2023-05-31T20:48:09.571 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:46:33 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009a380c38c2771d35-006477b289-1c17-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:48:09 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:48:09.572 DEBUG:boto:path=/pklxkd-59/obj115 2023-05-31T20:48:09.572 DEBUG:boto:auth_path=/pklxkd-59/obj115 2023-05-31T20:48:09.573 DEBUG:boto:path=/pklxkd-59/obj115?versionId=null 2023-05-31T20:48:09.573 DEBUG:boto:auth_path=/pklxkd-59/obj115?versionId=null 2023-05-31T20:48:09.574 DEBUG:boto:Method: GET 2023-05-31T20:48:09.574 DEBUG:boto:Path: /pklxkd-59/obj115?versionId=null 2023-05-31T20:48:09.574 DEBUG:boto:Data: 2023-05-31T20:48:09.575 DEBUG:boto:Headers: {} 2023-05-31T20:48:09.575 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:48:09.575 DEBUG:boto:Port: 8000 2023-05-31T20:48:09.576 DEBUG:boto:Params: {} 2023-05-31T20:48:09.576 DEBUG:boto:Token: None 2023-05-31T20:48:09.576 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:48:09 GMT /pklxkd-59/obj115?versionId=null 2023-05-31T20:48:09.577 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:nSjWWODuJcSCWRcHxD6rodJMcGc= 2023-05-31T20:48:09.577 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:48:09 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:nSjWWODuJcSCWRcHxD6rodJMcGc=', 'Content-Length': '0'} 2023-05-31T20:48:09.579 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:46:33 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b90408740b80987f-006477b289-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:48:09 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:48:09.580 DEBUG:boto:path=/pklxkd-59/ 2023-05-31T20:48:09.580 DEBUG:boto:auth_path=/pklxkd-59/ 2023-05-31T20:48:09.581 DEBUG:boto:path=/pklxkd-59/?prefix=obj115 2023-05-31T20:48:09.581 DEBUG:boto:auth_path=/pklxkd-59/?prefix=obj115 2023-05-31T20:48:09.581 DEBUG:boto:Method: GET 2023-05-31T20:48:09.582 DEBUG:boto:Path: /pklxkd-59/?prefix=obj115 2023-05-31T20:48:09.582 DEBUG:boto:Data: 2023-05-31T20:48:09.582 DEBUG:boto:Headers: {} 2023-05-31T20:48:09.583 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:48:09.583 DEBUG:boto:Port: 8001 2023-05-31T20:48:09.583 DEBUG:boto:Params: {} 2023-05-31T20:48:09.584 DEBUG:boto:Token: None 2023-05-31T20:48:09.584 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:48:09 GMT /pklxkd-59/ 2023-05-31T20:48:09.585 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:9STbDo2uP4g1xSEvz1y1wz540ZQ= 2023-05-31T20:48:09.585 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:48:09 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:9STbDo2uP4g1xSEvz1y1wz540ZQ=', 'Content-Length': '0'} 2023-05-31T20:48:09.587 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000077848baa5fca619f-006477b289-1c17-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:48:09 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:48:09.588 DEBUG:boto:b'pklxkd-59obj1151000falseobj1152023-05-31T20:46:33.958Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:48:09.588 DEBUG:boto:path=/pklxkd-59/ 2023-05-31T20:48:09.589 DEBUG:boto:auth_path=/pklxkd-59/ 2023-05-31T20:48:09.589 DEBUG:boto:path=/pklxkd-59/?prefix=obj115 2023-05-31T20:48:09.590 DEBUG:boto:auth_path=/pklxkd-59/?prefix=obj115 2023-05-31T20:48:09.590 DEBUG:boto:Method: GET 2023-05-31T20:48:09.590 DEBUG:boto:Path: /pklxkd-59/?prefix=obj115 2023-05-31T20:48:09.591 DEBUG:boto:Data: 2023-05-31T20:48:09.591 DEBUG:boto:Headers: {} 2023-05-31T20:48:09.591 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:48:09.592 DEBUG:boto:Port: 8000 2023-05-31T20:48:09.592 DEBUG:boto:Params: {} 2023-05-31T20:48:09.592 DEBUG:boto:Token: None 2023-05-31T20:48:09.593 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:48:09 GMT /pklxkd-59/ 2023-05-31T20:48:09.593 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:9STbDo2uP4g1xSEvz1y1wz540ZQ= 2023-05-31T20:48:09.594 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:48:09 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:9STbDo2uP4g1xSEvz1y1wz540ZQ=', 'Content-Length': '0'} 2023-05-31T20:48:09.597 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000004c3c5cc46cecb318-006477b289-1630-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:48:09 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:48:09.597 DEBUG:boto:b'pklxkd-59obj1151000falseobj1152023-05-31T20:46:33.958Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:48:09.598 DEBUG:boto:path=/pklxkd-59/obj115 2023-05-31T20:48:09.598 DEBUG:boto:auth_path=/pklxkd-59/obj115 2023-05-31T20:48:09.599 DEBUG:boto:Method: HEAD 2023-05-31T20:48:09.599 DEBUG:boto:Path: /pklxkd-59/obj115 2023-05-31T20:48:09.600 DEBUG:boto:Data: 2023-05-31T20:48:09.600 DEBUG:boto:Headers: {} 2023-05-31T20:48:09.600 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:48:09.601 DEBUG:boto:Port: 8001 2023-05-31T20:48:09.601 DEBUG:boto:Params: {} 2023-05-31T20:48:09.601 DEBUG:boto:Token: None 2023-05-31T20:48:09.602 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:48:09 GMT /pklxkd-59/obj115 2023-05-31T20:48:09.602 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:7bD5KMZEbmDJJvJz1ih5s7ZvFuU= 2023-05-31T20:48:09.602 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:48:09 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:7bD5KMZEbmDJJvJz1ih5s7ZvFuU=', 'Content-Length': '0'} 2023-05-31T20:48:09.604 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:46:33 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003090e0442ca454c4-006477b289-1c17-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:48:09 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:48:09.605 DEBUG:boto:path=/pklxkd-59/obj115 2023-05-31T20:48:09.605 DEBUG:boto:auth_path=/pklxkd-59/obj115 2023-05-31T20:48:09.606 DEBUG:boto:Method: HEAD 2023-05-31T20:48:09.606 DEBUG:boto:Path: /pklxkd-59/obj115 2023-05-31T20:48:09.606 DEBUG:boto:Data: 2023-05-31T20:48:09.607 DEBUG:boto:Headers: {} 2023-05-31T20:48:09.607 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:48:09.607 DEBUG:boto:Port: 8000 2023-05-31T20:48:09.608 DEBUG:boto:Params: {} 2023-05-31T20:48:09.608 DEBUG:boto:Token: None 2023-05-31T20:48:09.608 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:48:09 GMT /pklxkd-59/obj115 2023-05-31T20:48:09.609 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:7bD5KMZEbmDJJvJz1ih5s7ZvFuU= 2023-05-31T20:48:09.609 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:48:09 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:7bD5KMZEbmDJJvJz1ih5s7ZvFuU=', 'Content-Length': '0'} 2023-05-31T20:48:09.611 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:46:33 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000388a62592854f7aa-006477b289-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:48:09 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:48:09.611 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj115 2023-05-31T20:48:09.612 DEBUG:boto:path=/pklxkd-59/obj115 2023-05-31T20:48:09.612 DEBUG:boto:auth_path=/pklxkd-59/obj115 2023-05-31T20:48:09.612 DEBUG:boto:Method: GET 2023-05-31T20:48:09.613 DEBUG:boto:Path: /pklxkd-59/obj115 2023-05-31T20:48:09.613 DEBUG:boto:Data: 2023-05-31T20:48:09.614 DEBUG:boto:Headers: {} 2023-05-31T20:48:09.614 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:48:09.614 DEBUG:boto:Port: 8001 2023-05-31T20:48:09.615 DEBUG:boto:Params: {} 2023-05-31T20:48:09.615 DEBUG:boto:Token: None 2023-05-31T20:48:09.615 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:48:09 GMT /pklxkd-59/obj115 2023-05-31T20:48:09.616 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:yEmpFhCFj6CiK/llqFkeXK6c/l8= 2023-05-31T20:48:09.616 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:48:09 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:yEmpFhCFj6CiK/llqFkeXK6c/l8=', 'Content-Length': '0'} 2023-05-31T20:48:09.618 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:46:33 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000acd508c2f9130fd6-006477b289-1c17-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:48:09 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:48:09.660 DEBUG:boto:path=/pklxkd-59/obj115 2023-05-31T20:48:09.661 DEBUG:boto:auth_path=/pklxkd-59/obj115 2023-05-31T20:48:09.661 DEBUG:boto:Method: GET 2023-05-31T20:48:09.661 DEBUG:boto:Path: /pklxkd-59/obj115 2023-05-31T20:48:09.662 DEBUG:boto:Data: 2023-05-31T20:48:09.662 DEBUG:boto:Headers: {} 2023-05-31T20:48:09.663 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:48:09.663 DEBUG:boto:Port: 8000 2023-05-31T20:48:09.663 DEBUG:boto:Params: {} 2023-05-31T20:48:09.664 DEBUG:boto:Token: None 2023-05-31T20:48:09.664 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:48:09 GMT /pklxkd-59/obj115 2023-05-31T20:48:09.665 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:yEmpFhCFj6CiK/llqFkeXK6c/l8= 2023-05-31T20:48:09.665 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:48:09 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:yEmpFhCFj6CiK/llqFkeXK6c/l8=', 'Content-Length': '0'} 2023-05-31T20:48:09.667 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:46:33 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009ee925d13dc1919a-006477b289-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:48:09 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:48:09.667 DEBUG:boto:path=/pklxkd-59/ 2023-05-31T20:48:09.668 DEBUG:boto:auth_path=/pklxkd-59/ 2023-05-31T20:48:09.668 DEBUG:boto:path=/pklxkd-59/?prefix=obj115 2023-05-31T20:48:09.669 DEBUG:boto:auth_path=/pklxkd-59/?prefix=obj115 2023-05-31T20:48:09.669 DEBUG:boto:Method: GET 2023-05-31T20:48:09.669 DEBUG:boto:Path: /pklxkd-59/?prefix=obj115 2023-05-31T20:48:09.670 DEBUG:boto:Data: 2023-05-31T20:48:09.670 DEBUG:boto:Headers: {} 2023-05-31T20:48:09.670 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:48:09.671 DEBUG:boto:Port: 8001 2023-05-31T20:48:09.671 DEBUG:boto:Params: {} 2023-05-31T20:48:09.671 DEBUG:boto:Token: None 2023-05-31T20:48:09.672 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:48:09 GMT /pklxkd-59/ 2023-05-31T20:48:09.672 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:9STbDo2uP4g1xSEvz1y1wz540ZQ= 2023-05-31T20:48:09.673 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:48:09 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:9STbDo2uP4g1xSEvz1y1wz540ZQ=', 'Content-Length': '0'} 2023-05-31T20:48:09.675 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c4cc4b9f6dc3d4f5-006477b289-1c17-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:48:09 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:48:09.676 DEBUG:boto:b'pklxkd-59obj1151000falseobj1152023-05-31T20:46:33.958Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:48:09.676 DEBUG:boto:path=/pklxkd-59/ 2023-05-31T20:48:09.677 DEBUG:boto:auth_path=/pklxkd-59/ 2023-05-31T20:48:09.677 DEBUG:boto:path=/pklxkd-59/?prefix=obj115 2023-05-31T20:48:09.677 DEBUG:boto:auth_path=/pklxkd-59/?prefix=obj115 2023-05-31T20:48:09.678 DEBUG:boto:Method: GET 2023-05-31T20:48:09.678 DEBUG:boto:Path: /pklxkd-59/?prefix=obj115 2023-05-31T20:48:09.678 DEBUG:boto:Data: 2023-05-31T20:48:09.679 DEBUG:boto:Headers: {} 2023-05-31T20:48:09.679 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:48:09.679 DEBUG:boto:Port: 8000 2023-05-31T20:48:09.680 DEBUG:boto:Params: {} 2023-05-31T20:48:09.680 DEBUG:boto:Token: None 2023-05-31T20:48:09.681 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:48:09 GMT /pklxkd-59/ 2023-05-31T20:48:09.681 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:9STbDo2uP4g1xSEvz1y1wz540ZQ= 2023-05-31T20:48:09.682 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:48:09 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:9STbDo2uP4g1xSEvz1y1wz540ZQ=', 'Content-Length': '0'} 2023-05-31T20:48:09.684 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000553b0e155b4870bc-006477b289-1630-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:48:09 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:48:09.685 DEBUG:boto:b'pklxkd-59obj1151000falseobj1152023-05-31T20:46:33.958Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:48:09.686 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj17 2023-05-31T20:48:09.686 DEBUG:boto:path=/pklxkd-59/obj17 2023-05-31T20:48:09.687 DEBUG:boto:auth_path=/pklxkd-59/obj17 2023-05-31T20:48:09.687 DEBUG:boto:path=/pklxkd-59/obj17?versionId=null 2023-05-31T20:48:09.687 DEBUG:boto:auth_path=/pklxkd-59/obj17?versionId=null 2023-05-31T20:48:09.688 DEBUG:boto:Method: GET 2023-05-31T20:48:09.688 DEBUG:boto:Path: /pklxkd-59/obj17?versionId=null 2023-05-31T20:48:09.688 DEBUG:boto:Data: 2023-05-31T20:48:09.689 DEBUG:boto:Headers: {} 2023-05-31T20:48:09.689 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:48:09.689 DEBUG:boto:Port: 8001 2023-05-31T20:48:09.690 DEBUG:boto:Params: {} 2023-05-31T20:48:09.690 DEBUG:boto:Token: None 2023-05-31T20:48:09.691 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:48:09 GMT /pklxkd-59/obj17?versionId=null 2023-05-31T20:48:09.691 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:UOXXYQUc/LErCEosTzQCx5qPENs= 2023-05-31T20:48:09.691 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:48:09 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:UOXXYQUc/LErCEosTzQCx5qPENs=', 'Content-Length': '0'} 2023-05-31T20:48:09.693 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:43:07 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005af1c2508052c0a6-006477b289-1c17-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:48:09 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:48:09.694 DEBUG:boto:path=/pklxkd-59/obj17 2023-05-31T20:48:09.694 DEBUG:boto:auth_path=/pklxkd-59/obj17 2023-05-31T20:48:09.695 DEBUG:boto:path=/pklxkd-59/obj17?versionId=null 2023-05-31T20:48:09.695 DEBUG:boto:auth_path=/pklxkd-59/obj17?versionId=null 2023-05-31T20:48:09.695 DEBUG:boto:Method: GET 2023-05-31T20:48:09.696 DEBUG:boto:Path: /pklxkd-59/obj17?versionId=null 2023-05-31T20:48:09.696 DEBUG:boto:Data: 2023-05-31T20:48:09.696 DEBUG:boto:Headers: {} 2023-05-31T20:48:09.697 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:48:09.697 DEBUG:boto:Port: 8000 2023-05-31T20:48:09.697 DEBUG:boto:Params: {} 2023-05-31T20:48:09.698 DEBUG:boto:Token: None 2023-05-31T20:48:09.698 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:48:09 GMT /pklxkd-59/obj17?versionId=null 2023-05-31T20:48:09.699 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:UOXXYQUc/LErCEosTzQCx5qPENs= 2023-05-31T20:48:09.699 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:48:09 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:UOXXYQUc/LErCEosTzQCx5qPENs=', 'Content-Length': '0'} 2023-05-31T20:48:09.701 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:43:07 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000073d2c5fdd63ff478-006477b289-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:48:09 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:48:09.701 DEBUG:boto:path=/pklxkd-59/obj17 2023-05-31T20:48:09.702 DEBUG:boto:auth_path=/pklxkd-59/obj17 2023-05-31T20:48:09.702 DEBUG:boto:path=/pklxkd-59/obj17?versionId=null 2023-05-31T20:48:09.703 DEBUG:boto:auth_path=/pklxkd-59/obj17?versionId=null 2023-05-31T20:48:09.703 DEBUG:boto:Method: HEAD 2023-05-31T20:48:09.703 DEBUG:boto:Path: /pklxkd-59/obj17?versionId=null 2023-05-31T20:48:09.704 DEBUG:boto:Data: 2023-05-31T20:48:09.704 DEBUG:boto:Headers: {} 2023-05-31T20:48:09.704 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:48:09.705 DEBUG:boto:Port: 8001 2023-05-31T20:48:09.705 DEBUG:boto:Params: {} 2023-05-31T20:48:09.705 DEBUG:boto:Token: None 2023-05-31T20:48:09.706 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:48:09 GMT /pklxkd-59/obj17?versionId=null 2023-05-31T20:48:09.706 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:v+6kPmTOhHUR0FJ6kbyVnzKJsFI= 2023-05-31T20:48:09.706 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:48:09 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:v+6kPmTOhHUR0FJ6kbyVnzKJsFI=', 'Content-Length': '0'} 2023-05-31T20:48:09.708 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:43:07 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000aefd848a501df038-006477b289-1c17-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:48:09 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:48:09.709 DEBUG:boto:path=/pklxkd-59/obj17 2023-05-31T20:48:09.709 DEBUG:boto:auth_path=/pklxkd-59/obj17 2023-05-31T20:48:09.709 DEBUG:boto:path=/pklxkd-59/obj17?versionId=null 2023-05-31T20:48:09.710 DEBUG:boto:auth_path=/pklxkd-59/obj17?versionId=null 2023-05-31T20:48:09.710 DEBUG:boto:Method: HEAD 2023-05-31T20:48:09.710 DEBUG:boto:Path: /pklxkd-59/obj17?versionId=null 2023-05-31T20:48:09.711 DEBUG:boto:Data: 2023-05-31T20:48:09.711 DEBUG:boto:Headers: {} 2023-05-31T20:48:09.711 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:48:09.712 DEBUG:boto:Port: 8000 2023-05-31T20:48:09.712 DEBUG:boto:Params: {} 2023-05-31T20:48:09.712 DEBUG:boto:Token: None 2023-05-31T20:48:09.713 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:48:09 GMT /pklxkd-59/obj17?versionId=null 2023-05-31T20:48:09.713 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:v+6kPmTOhHUR0FJ6kbyVnzKJsFI= 2023-05-31T20:48:09.714 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:48:09 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:v+6kPmTOhHUR0FJ6kbyVnzKJsFI=', 'Content-Length': '0'} 2023-05-31T20:48:09.715 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:43:07 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001c6a5e57a506ac3c-006477b289-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:48:09 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:48:09.716 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj17 2023-05-31T20:48:09.716 DEBUG:boto:path=/pklxkd-59/obj17 2023-05-31T20:48:09.717 DEBUG:boto:auth_path=/pklxkd-59/obj17 2023-05-31T20:48:09.717 DEBUG:boto:path=/pklxkd-59/obj17?versionId=null 2023-05-31T20:48:09.717 DEBUG:boto:auth_path=/pklxkd-59/obj17?versionId=null 2023-05-31T20:48:09.718 DEBUG:boto:Method: GET 2023-05-31T20:48:09.718 DEBUG:boto:Path: /pklxkd-59/obj17?versionId=null 2023-05-31T20:48:09.718 DEBUG:boto:Data: 2023-05-31T20:48:09.719 DEBUG:boto:Headers: {} 2023-05-31T20:48:09.719 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:48:09.720 DEBUG:boto:Port: 8001 2023-05-31T20:48:09.720 DEBUG:boto:Params: {} 2023-05-31T20:48:09.720 DEBUG:boto:Token: None 2023-05-31T20:48:09.721 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:48:09 GMT /pklxkd-59/obj17?versionId=null 2023-05-31T20:48:09.721 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:UOXXYQUc/LErCEosTzQCx5qPENs= 2023-05-31T20:48:09.722 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:48:09 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:UOXXYQUc/LErCEosTzQCx5qPENs=', 'Content-Length': '0'} 2023-05-31T20:48:09.723 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:43:07 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a31d74e5b957ef57-006477b289-1c17-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:48:09 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:48:09.724 DEBUG:boto:path=/pklxkd-59/obj17 2023-05-31T20:48:09.724 DEBUG:boto:auth_path=/pklxkd-59/obj17 2023-05-31T20:48:09.725 DEBUG:boto:path=/pklxkd-59/obj17?versionId=null 2023-05-31T20:48:09.725 DEBUG:boto:auth_path=/pklxkd-59/obj17?versionId=null 2023-05-31T20:48:09.725 DEBUG:boto:Method: GET 2023-05-31T20:48:09.726 DEBUG:boto:Path: /pklxkd-59/obj17?versionId=null 2023-05-31T20:48:09.726 DEBUG:boto:Data: 2023-05-31T20:48:09.726 DEBUG:boto:Headers: {} 2023-05-31T20:48:09.727 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:48:09.727 DEBUG:boto:Port: 8000 2023-05-31T20:48:09.727 DEBUG:boto:Params: {} 2023-05-31T20:48:09.728 DEBUG:boto:Token: None 2023-05-31T20:48:09.728 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:48:09 GMT /pklxkd-59/obj17?versionId=null 2023-05-31T20:48:09.729 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:UOXXYQUc/LErCEosTzQCx5qPENs= 2023-05-31T20:48:09.729 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:48:09 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:UOXXYQUc/LErCEosTzQCx5qPENs=', 'Content-Length': '0'} 2023-05-31T20:48:09.731 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:43:07 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000575ccfe93f39f634-006477b289-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:48:09 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:48:09.731 DEBUG:boto:path=/pklxkd-59/ 2023-05-31T20:48:09.732 DEBUG:boto:auth_path=/pklxkd-59/ 2023-05-31T20:48:09.732 DEBUG:boto:path=/pklxkd-59/?prefix=obj17 2023-05-31T20:48:09.732 DEBUG:boto:auth_path=/pklxkd-59/?prefix=obj17 2023-05-31T20:48:09.733 DEBUG:boto:Method: GET 2023-05-31T20:48:09.733 DEBUG:boto:Path: /pklxkd-59/?prefix=obj17 2023-05-31T20:48:09.734 DEBUG:boto:Data: 2023-05-31T20:48:09.734 DEBUG:boto:Headers: {} 2023-05-31T20:48:09.734 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:48:09.735 DEBUG:boto:Port: 8001 2023-05-31T20:48:09.735 DEBUG:boto:Params: {} 2023-05-31T20:48:09.735 DEBUG:boto:Token: None 2023-05-31T20:48:09.736 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:48:09 GMT /pklxkd-59/ 2023-05-31T20:48:09.736 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:9STbDo2uP4g1xSEvz1y1wz540ZQ= 2023-05-31T20:48:09.737 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:48:09 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:9STbDo2uP4g1xSEvz1y1wz540ZQ=', 'Content-Length': '0'} 2023-05-31T20:48:09.739 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000003da2777e741d0895-006477b289-1c17-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:48:09 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:48:09.739 DEBUG:boto:b'pklxkd-59obj171000falseobj172023-05-31T20:43:07.977Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:48:09.740 DEBUG:boto:path=/pklxkd-59/ 2023-05-31T20:48:09.740 DEBUG:boto:auth_path=/pklxkd-59/ 2023-05-31T20:48:09.741 DEBUG:boto:path=/pklxkd-59/?prefix=obj17 2023-05-31T20:48:09.741 DEBUG:boto:auth_path=/pklxkd-59/?prefix=obj17 2023-05-31T20:48:09.741 DEBUG:boto:Method: GET 2023-05-31T20:48:09.742 DEBUG:boto:Path: /pklxkd-59/?prefix=obj17 2023-05-31T20:48:09.742 DEBUG:boto:Data: 2023-05-31T20:48:09.742 DEBUG:boto:Headers: {} 2023-05-31T20:48:09.743 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:48:09.743 DEBUG:boto:Port: 8000 2023-05-31T20:48:09.743 DEBUG:boto:Params: {} 2023-05-31T20:48:09.744 DEBUG:boto:Token: None 2023-05-31T20:48:09.744 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:48:09 GMT /pklxkd-59/ 2023-05-31T20:48:09.745 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:9STbDo2uP4g1xSEvz1y1wz540ZQ= 2023-05-31T20:48:09.745 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:48:09 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:9STbDo2uP4g1xSEvz1y1wz540ZQ=', 'Content-Length': '0'} 2023-05-31T20:48:09.748 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000009e4e7023e6114454-006477b289-1630-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:48:09 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:48:09.749 DEBUG:boto:b'pklxkd-59obj171000falseobj172023-05-31T20:43:07.977Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:48:09.749 DEBUG:boto:path=/pklxkd-59/obj17 2023-05-31T20:48:09.750 DEBUG:boto:auth_path=/pklxkd-59/obj17 2023-05-31T20:48:09.750 DEBUG:boto:Method: HEAD 2023-05-31T20:48:09.750 DEBUG:boto:Path: /pklxkd-59/obj17 2023-05-31T20:48:09.751 DEBUG:boto:Data: 2023-05-31T20:48:09.751 DEBUG:boto:Headers: {} 2023-05-31T20:48:09.751 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:48:09.752 DEBUG:boto:Port: 8001 2023-05-31T20:48:09.752 DEBUG:boto:Params: {} 2023-05-31T20:48:09.752 DEBUG:boto:Token: None 2023-05-31T20:48:09.753 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:48:09 GMT /pklxkd-59/obj17 2023-05-31T20:48:09.753 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:zjzjOusPcNKHrnrGswQ9YPdj5xI= 2023-05-31T20:48:09.753 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:48:09 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:zjzjOusPcNKHrnrGswQ9YPdj5xI=', 'Content-Length': '0'} 2023-05-31T20:48:09.755 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:43:07 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000012cc168dde8983fa-006477b289-1c17-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:48:09 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:48:09.756 DEBUG:boto:path=/pklxkd-59/obj17 2023-05-31T20:48:09.756 DEBUG:boto:auth_path=/pklxkd-59/obj17 2023-05-31T20:48:09.756 DEBUG:boto:Method: HEAD 2023-05-31T20:48:09.757 DEBUG:boto:Path: /pklxkd-59/obj17 2023-05-31T20:48:09.757 DEBUG:boto:Data: 2023-05-31T20:48:09.757 DEBUG:boto:Headers: {} 2023-05-31T20:48:09.758 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:48:09.758 DEBUG:boto:Port: 8000 2023-05-31T20:48:09.758 DEBUG:boto:Params: {} 2023-05-31T20:48:09.759 DEBUG:boto:Token: None 2023-05-31T20:48:09.759 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:48:09 GMT /pklxkd-59/obj17 2023-05-31T20:48:09.760 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:zjzjOusPcNKHrnrGswQ9YPdj5xI= 2023-05-31T20:48:09.760 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:48:09 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:zjzjOusPcNKHrnrGswQ9YPdj5xI=', 'Content-Length': '0'} 2023-05-31T20:48:09.762 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:43:07 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d8792ed616027a64-006477b289-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:48:09 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:48:09.762 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj17 2023-05-31T20:48:09.763 DEBUG:boto:path=/pklxkd-59/obj17 2023-05-31T20:48:09.763 DEBUG:boto:auth_path=/pklxkd-59/obj17 2023-05-31T20:48:09.763 DEBUG:boto:Method: GET 2023-05-31T20:48:09.764 DEBUG:boto:Path: /pklxkd-59/obj17 2023-05-31T20:48:09.764 DEBUG:boto:Data: 2023-05-31T20:48:09.765 DEBUG:boto:Headers: {} 2023-05-31T20:48:09.765 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:48:09.765 DEBUG:boto:Port: 8001 2023-05-31T20:48:09.766 DEBUG:boto:Params: {} 2023-05-31T20:48:09.766 DEBUG:boto:Token: None 2023-05-31T20:48:09.766 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:48:09 GMT /pklxkd-59/obj17 2023-05-31T20:48:09.767 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:hqxhPEDhkV//WuA0E0RgEObmZZU= 2023-05-31T20:48:09.767 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:48:09 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:hqxhPEDhkV//WuA0E0RgEObmZZU=', 'Content-Length': '0'} 2023-05-31T20:48:09.769 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:43:07 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000044fe2cf8827d76b2-006477b289-1c17-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:48:09 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:48:09.770 DEBUG:boto:path=/pklxkd-59/obj17 2023-05-31T20:48:09.770 DEBUG:boto:auth_path=/pklxkd-59/obj17 2023-05-31T20:48:09.771 DEBUG:boto:Method: GET 2023-05-31T20:48:09.771 DEBUG:boto:Path: /pklxkd-59/obj17 2023-05-31T20:48:09.771 DEBUG:boto:Data: 2023-05-31T20:48:09.772 DEBUG:boto:Headers: {} 2023-05-31T20:48:09.772 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:48:09.772 DEBUG:boto:Port: 8000 2023-05-31T20:48:09.773 DEBUG:boto:Params: {} 2023-05-31T20:48:09.773 DEBUG:boto:Token: None 2023-05-31T20:48:09.774 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:48:09 GMT /pklxkd-59/obj17 2023-05-31T20:48:09.774 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:hqxhPEDhkV//WuA0E0RgEObmZZU= 2023-05-31T20:48:09.774 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:48:09 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:hqxhPEDhkV//WuA0E0RgEObmZZU=', 'Content-Length': '0'} 2023-05-31T20:48:09.776 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:43:07 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008ecf366450bd8494-006477b289-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:48:09 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:48:09.820 DEBUG:boto:path=/pklxkd-59/ 2023-05-31T20:48:09.821 DEBUG:boto:auth_path=/pklxkd-59/ 2023-05-31T20:48:09.821 DEBUG:boto:path=/pklxkd-59/?prefix=obj17 2023-05-31T20:48:09.821 DEBUG:boto:auth_path=/pklxkd-59/?prefix=obj17 2023-05-31T20:48:09.822 DEBUG:boto:Method: GET 2023-05-31T20:48:09.822 DEBUG:boto:Path: /pklxkd-59/?prefix=obj17 2023-05-31T20:48:09.822 DEBUG:boto:Data: 2023-05-31T20:48:09.823 DEBUG:boto:Headers: {} 2023-05-31T20:48:09.823 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:48:09.823 DEBUG:boto:Port: 8001 2023-05-31T20:48:09.824 DEBUG:boto:Params: {} 2023-05-31T20:48:09.824 DEBUG:boto:Token: None 2023-05-31T20:48:09.825 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:48:09 GMT /pklxkd-59/ 2023-05-31T20:48:09.825 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:9STbDo2uP4g1xSEvz1y1wz540ZQ= 2023-05-31T20:48:09.825 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:48:09 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:9STbDo2uP4g1xSEvz1y1wz540ZQ=', 'Content-Length': '0'} 2023-05-31T20:48:09.828 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000375d5c9bc761552e-006477b289-1c17-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:48:09 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:48:09.828 DEBUG:boto:b'pklxkd-59obj171000falseobj172023-05-31T20:43:07.977Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:48:09.829 DEBUG:boto:path=/pklxkd-59/ 2023-05-31T20:48:09.829 DEBUG:boto:auth_path=/pklxkd-59/ 2023-05-31T20:48:09.830 DEBUG:boto:path=/pklxkd-59/?prefix=obj17 2023-05-31T20:48:09.830 DEBUG:boto:auth_path=/pklxkd-59/?prefix=obj17 2023-05-31T20:48:09.830 DEBUG:boto:Method: GET 2023-05-31T20:48:09.831 DEBUG:boto:Path: /pklxkd-59/?prefix=obj17 2023-05-31T20:48:09.831 DEBUG:boto:Data: 2023-05-31T20:48:09.831 DEBUG:boto:Headers: {} 2023-05-31T20:48:09.832 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:48:09.832 DEBUG:boto:Port: 8000 2023-05-31T20:48:09.832 DEBUG:boto:Params: {} 2023-05-31T20:48:09.833 DEBUG:boto:Token: None 2023-05-31T20:48:09.833 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:48:09 GMT /pklxkd-59/ 2023-05-31T20:48:09.834 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:9STbDo2uP4g1xSEvz1y1wz540ZQ= 2023-05-31T20:48:09.834 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:48:09 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:9STbDo2uP4g1xSEvz1y1wz540ZQ=', 'Content-Length': '0'} 2023-05-31T20:48:09.837 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c5784e6fa4ed15c5-006477b289-1630-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:48:09 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:48:09.838 DEBUG:boto:b'pklxkd-59obj171000falseobj172023-05-31T20:43:07.977Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:48:09.838 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj19 2023-05-31T20:48:09.839 DEBUG:boto:path=/pklxkd-59/obj19 2023-05-31T20:48:09.839 DEBUG:boto:auth_path=/pklxkd-59/obj19 2023-05-31T20:48:09.839 DEBUG:boto:path=/pklxkd-59/obj19?versionId=null 2023-05-31T20:48:09.840 DEBUG:boto:auth_path=/pklxkd-59/obj19?versionId=null 2023-05-31T20:48:09.840 DEBUG:boto:Method: GET 2023-05-31T20:48:09.840 DEBUG:boto:Path: /pklxkd-59/obj19?versionId=null 2023-05-31T20:48:09.841 DEBUG:boto:Data: 2023-05-31T20:48:09.841 DEBUG:boto:Headers: {} 2023-05-31T20:48:09.841 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:48:09.842 DEBUG:boto:Port: 8001 2023-05-31T20:48:09.842 DEBUG:boto:Params: {} 2023-05-31T20:48:09.843 DEBUG:boto:Token: None 2023-05-31T20:48:09.843 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:48:09 GMT /pklxkd-59/obj19?versionId=null 2023-05-31T20:48:09.843 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:vdTmfiRZQFLgRMVJo20XI4Is7QE= 2023-05-31T20:48:09.844 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:48:09 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:vdTmfiRZQFLgRMVJo20XI4Is7QE=', 'Content-Length': '0'} 2023-05-31T20:48:09.846 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:44:50 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002d7c8499ddd9d478-006477b289-1c17-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:48:09 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:48:09.846 DEBUG:boto:path=/pklxkd-59/obj19 2023-05-31T20:48:09.847 DEBUG:boto:auth_path=/pklxkd-59/obj19 2023-05-31T20:48:09.847 DEBUG:boto:path=/pklxkd-59/obj19?versionId=null 2023-05-31T20:48:09.847 DEBUG:boto:auth_path=/pklxkd-59/obj19?versionId=null 2023-05-31T20:48:09.848 DEBUG:boto:Method: GET 2023-05-31T20:48:09.848 DEBUG:boto:Path: /pklxkd-59/obj19?versionId=null 2023-05-31T20:48:09.848 DEBUG:boto:Data: 2023-05-31T20:48:09.849 DEBUG:boto:Headers: {} 2023-05-31T20:48:09.849 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:48:09.849 DEBUG:boto:Port: 8000 2023-05-31T20:48:09.850 DEBUG:boto:Params: {} 2023-05-31T20:48:09.850 DEBUG:boto:Token: None 2023-05-31T20:48:09.851 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:48:09 GMT /pklxkd-59/obj19?versionId=null 2023-05-31T20:48:09.851 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:vdTmfiRZQFLgRMVJo20XI4Is7QE= 2023-05-31T20:48:09.851 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:48:09 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:vdTmfiRZQFLgRMVJo20XI4Is7QE=', 'Content-Length': '0'} 2023-05-31T20:48:09.853 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:44:50 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ec8f9c8ddea3afd5-006477b289-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:48:09 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:48:09.854 DEBUG:boto:path=/pklxkd-59/obj19 2023-05-31T20:48:09.854 DEBUG:boto:auth_path=/pklxkd-59/obj19 2023-05-31T20:48:09.855 DEBUG:boto:path=/pklxkd-59/obj19?versionId=null 2023-05-31T20:48:09.855 DEBUG:boto:auth_path=/pklxkd-59/obj19?versionId=null 2023-05-31T20:48:09.855 DEBUG:boto:Method: HEAD 2023-05-31T20:48:09.856 DEBUG:boto:Path: /pklxkd-59/obj19?versionId=null 2023-05-31T20:48:09.856 DEBUG:boto:Data: 2023-05-31T20:48:09.856 DEBUG:boto:Headers: {} 2023-05-31T20:48:09.857 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:48:09.857 DEBUG:boto:Port: 8001 2023-05-31T20:48:09.857 DEBUG:boto:Params: {} 2023-05-31T20:48:09.858 DEBUG:boto:Token: None 2023-05-31T20:48:09.858 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:48:09 GMT /pklxkd-59/obj19?versionId=null 2023-05-31T20:48:09.858 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:JFYFxnKhnveUsN1jAC6bA0wbQU0= 2023-05-31T20:48:09.859 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:48:09 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:JFYFxnKhnveUsN1jAC6bA0wbQU0=', 'Content-Length': '0'} 2023-05-31T20:48:09.860 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:44:50 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f16f557942efab68-006477b289-1c17-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:48:09 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:48:09.861 DEBUG:boto:path=/pklxkd-59/obj19 2023-05-31T20:48:09.861 DEBUG:boto:auth_path=/pklxkd-59/obj19 2023-05-31T20:48:09.862 DEBUG:boto:path=/pklxkd-59/obj19?versionId=null 2023-05-31T20:48:09.862 DEBUG:boto:auth_path=/pklxkd-59/obj19?versionId=null 2023-05-31T20:48:09.862 DEBUG:boto:Method: HEAD 2023-05-31T20:48:09.863 DEBUG:boto:Path: /pklxkd-59/obj19?versionId=null 2023-05-31T20:48:09.863 DEBUG:boto:Data: 2023-05-31T20:48:09.863 DEBUG:boto:Headers: {} 2023-05-31T20:48:09.864 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:48:09.864 DEBUG:boto:Port: 8000 2023-05-31T20:48:09.864 DEBUG:boto:Params: {} 2023-05-31T20:48:09.865 DEBUG:boto:Token: None 2023-05-31T20:48:09.865 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:48:09 GMT /pklxkd-59/obj19?versionId=null 2023-05-31T20:48:09.866 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:JFYFxnKhnveUsN1jAC6bA0wbQU0= 2023-05-31T20:48:09.866 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:48:09 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:JFYFxnKhnveUsN1jAC6bA0wbQU0=', 'Content-Length': '0'} 2023-05-31T20:48:09.868 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:44:50 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ff3cb36d5760be18-006477b289-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:48:09 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:48:09.868 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj19 2023-05-31T20:48:09.869 DEBUG:boto:path=/pklxkd-59/obj19 2023-05-31T20:48:09.869 DEBUG:boto:auth_path=/pklxkd-59/obj19 2023-05-31T20:48:09.870 DEBUG:boto:path=/pklxkd-59/obj19?versionId=null 2023-05-31T20:48:09.870 DEBUG:boto:auth_path=/pklxkd-59/obj19?versionId=null 2023-05-31T20:48:09.870 DEBUG:boto:Method: GET 2023-05-31T20:48:09.871 DEBUG:boto:Path: /pklxkd-59/obj19?versionId=null 2023-05-31T20:48:09.871 DEBUG:boto:Data: 2023-05-31T20:48:09.871 DEBUG:boto:Headers: {} 2023-05-31T20:48:09.872 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:48:09.872 DEBUG:boto:Port: 8001 2023-05-31T20:48:09.872 DEBUG:boto:Params: {} 2023-05-31T20:48:09.873 DEBUG:boto:Token: None 2023-05-31T20:48:09.873 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:48:09 GMT /pklxkd-59/obj19?versionId=null 2023-05-31T20:48:09.874 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:vdTmfiRZQFLgRMVJo20XI4Is7QE= 2023-05-31T20:48:09.874 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:48:09 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:vdTmfiRZQFLgRMVJo20XI4Is7QE=', 'Content-Length': '0'} 2023-05-31T20:48:09.876 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:44:50 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d48312da808cad21-006477b289-1c17-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:48:09 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:48:09.916 DEBUG:boto:path=/pklxkd-59/obj19 2023-05-31T20:48:09.917 DEBUG:boto:auth_path=/pklxkd-59/obj19 2023-05-31T20:48:09.917 DEBUG:boto:path=/pklxkd-59/obj19?versionId=null 2023-05-31T20:48:09.917 DEBUG:boto:auth_path=/pklxkd-59/obj19?versionId=null 2023-05-31T20:48:09.918 DEBUG:boto:Method: GET 2023-05-31T20:48:09.918 DEBUG:boto:Path: /pklxkd-59/obj19?versionId=null 2023-05-31T20:48:09.918 DEBUG:boto:Data: 2023-05-31T20:48:09.919 DEBUG:boto:Headers: {} 2023-05-31T20:48:09.919 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:48:09.919 DEBUG:boto:Port: 8000 2023-05-31T20:48:09.920 DEBUG:boto:Params: {} 2023-05-31T20:48:09.920 DEBUG:boto:Token: None 2023-05-31T20:48:09.921 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:48:09 GMT /pklxkd-59/obj19?versionId=null 2023-05-31T20:48:09.921 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:vdTmfiRZQFLgRMVJo20XI4Is7QE= 2023-05-31T20:48:09.922 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:48:09 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:vdTmfiRZQFLgRMVJo20XI4Is7QE=', 'Content-Length': '0'} 2023-05-31T20:48:09.923 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:44:50 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001cc5fa84206c96f7-006477b289-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:48:09 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:48:09.924 DEBUG:boto:path=/pklxkd-59/ 2023-05-31T20:48:09.925 DEBUG:boto:auth_path=/pklxkd-59/ 2023-05-31T20:48:09.925 DEBUG:boto:path=/pklxkd-59/?prefix=obj19 2023-05-31T20:48:09.925 DEBUG:boto:auth_path=/pklxkd-59/?prefix=obj19 2023-05-31T20:48:09.926 DEBUG:boto:Method: GET 2023-05-31T20:48:09.926 DEBUG:boto:Path: /pklxkd-59/?prefix=obj19 2023-05-31T20:48:09.926 DEBUG:boto:Data: 2023-05-31T20:48:09.927 DEBUG:boto:Headers: {} 2023-05-31T20:48:09.927 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:48:09.927 DEBUG:boto:Port: 8001 2023-05-31T20:48:09.928 DEBUG:boto:Params: {} 2023-05-31T20:48:09.928 DEBUG:boto:Token: None 2023-05-31T20:48:09.929 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:48:09 GMT /pklxkd-59/ 2023-05-31T20:48:09.929 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:9STbDo2uP4g1xSEvz1y1wz540ZQ= 2023-05-31T20:48:09.929 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:48:09 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:9STbDo2uP4g1xSEvz1y1wz540ZQ=', 'Content-Length': '0'} 2023-05-31T20:48:09.932 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ff8aa01e4e05bc92-006477b289-1c17-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:48:09 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:48:09.932 DEBUG:boto:b'pklxkd-59obj191000falseobj192023-05-31T20:44:50.557Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:48:09.933 DEBUG:boto:path=/pklxkd-59/ 2023-05-31T20:48:09.933 DEBUG:boto:auth_path=/pklxkd-59/ 2023-05-31T20:48:09.934 DEBUG:boto:path=/pklxkd-59/?prefix=obj19 2023-05-31T20:48:09.934 DEBUG:boto:auth_path=/pklxkd-59/?prefix=obj19 2023-05-31T20:48:09.934 DEBUG:boto:Method: GET 2023-05-31T20:48:09.935 DEBUG:boto:Path: /pklxkd-59/?prefix=obj19 2023-05-31T20:48:09.935 DEBUG:boto:Data: 2023-05-31T20:48:09.936 DEBUG:boto:Headers: {} 2023-05-31T20:48:09.936 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:48:09.936 DEBUG:boto:Port: 8000 2023-05-31T20:48:09.937 DEBUG:boto:Params: {} 2023-05-31T20:48:09.937 DEBUG:boto:Token: None 2023-05-31T20:48:09.937 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:48:09 GMT /pklxkd-59/ 2023-05-31T20:48:09.938 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:9STbDo2uP4g1xSEvz1y1wz540ZQ= 2023-05-31T20:48:09.938 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:48:09 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:9STbDo2uP4g1xSEvz1y1wz540ZQ=', 'Content-Length': '0'} 2023-05-31T20:48:09.941 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b54ec671b8547a1e-006477b289-1630-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:48:09 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:48:09.942 DEBUG:boto:b'pklxkd-59obj191000falseobj192023-05-31T20:44:50.557Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:48:09.942 DEBUG:boto:path=/pklxkd-59/obj19 2023-05-31T20:48:09.943 DEBUG:boto:auth_path=/pklxkd-59/obj19 2023-05-31T20:48:09.943 DEBUG:boto:Method: HEAD 2023-05-31T20:48:09.944 DEBUG:boto:Path: /pklxkd-59/obj19 2023-05-31T20:48:09.944 DEBUG:boto:Data: 2023-05-31T20:48:09.944 DEBUG:boto:Headers: {} 2023-05-31T20:48:09.945 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:48:09.945 DEBUG:boto:Port: 8001 2023-05-31T20:48:09.945 DEBUG:boto:Params: {} 2023-05-31T20:48:09.946 DEBUG:boto:Token: None 2023-05-31T20:48:09.946 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:48:09 GMT /pklxkd-59/obj19 2023-05-31T20:48:09.946 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:1N6pg3r4XqmoxGFhHYGQb1QpbCU= 2023-05-31T20:48:09.947 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:48:09 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:1N6pg3r4XqmoxGFhHYGQb1QpbCU=', 'Content-Length': '0'} 2023-05-31T20:48:09.948 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:44:50 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000073504fd86e149c9b-006477b289-1c17-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:48:09 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:48:09.949 DEBUG:boto:path=/pklxkd-59/obj19 2023-05-31T20:48:09.949 DEBUG:boto:auth_path=/pklxkd-59/obj19 2023-05-31T20:48:09.950 DEBUG:boto:Method: HEAD 2023-05-31T20:48:09.950 DEBUG:boto:Path: /pklxkd-59/obj19 2023-05-31T20:48:09.950 DEBUG:boto:Data: 2023-05-31T20:48:09.951 DEBUG:boto:Headers: {} 2023-05-31T20:48:09.951 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:48:09.951 DEBUG:boto:Port: 8000 2023-05-31T20:48:09.952 DEBUG:boto:Params: {} 2023-05-31T20:48:09.952 DEBUG:boto:Token: None 2023-05-31T20:48:09.952 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:48:09 GMT /pklxkd-59/obj19 2023-05-31T20:48:09.953 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:1N6pg3r4XqmoxGFhHYGQb1QpbCU= 2023-05-31T20:48:09.953 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:48:09 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:1N6pg3r4XqmoxGFhHYGQb1QpbCU=', 'Content-Length': '0'} 2023-05-31T20:48:09.955 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:44:50 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c3ddf900b498950a-006477b289-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:48:09 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:48:09.956 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj19 2023-05-31T20:48:09.956 DEBUG:boto:path=/pklxkd-59/obj19 2023-05-31T20:48:09.956 DEBUG:boto:auth_path=/pklxkd-59/obj19 2023-05-31T20:48:09.957 DEBUG:boto:Method: GET 2023-05-31T20:48:09.957 DEBUG:boto:Path: /pklxkd-59/obj19 2023-05-31T20:48:09.958 DEBUG:boto:Data: 2023-05-31T20:48:09.958 DEBUG:boto:Headers: {} 2023-05-31T20:48:09.958 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:48:09.959 DEBUG:boto:Port: 8001 2023-05-31T20:48:09.959 DEBUG:boto:Params: {} 2023-05-31T20:48:09.959 DEBUG:boto:Token: None 2023-05-31T20:48:09.960 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:48:09 GMT /pklxkd-59/obj19 2023-05-31T20:48:09.960 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:cPPztrG/mIPhBaSWeXnCSi5jUug= 2023-05-31T20:48:09.961 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:48:09 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:cPPztrG/mIPhBaSWeXnCSi5jUug=', 'Content-Length': '0'} 2023-05-31T20:48:09.962 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:44:50 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009ac312548ec6f436-006477b289-1c17-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:48:09 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:48:09.963 DEBUG:boto:path=/pklxkd-59/obj19 2023-05-31T20:48:09.963 DEBUG:boto:auth_path=/pklxkd-59/obj19 2023-05-31T20:48:09.964 DEBUG:boto:Method: GET 2023-05-31T20:48:09.964 DEBUG:boto:Path: /pklxkd-59/obj19 2023-05-31T20:48:09.964 DEBUG:boto:Data: 2023-05-31T20:48:09.965 DEBUG:boto:Headers: {} 2023-05-31T20:48:09.965 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:48:09.966 DEBUG:boto:Port: 8000 2023-05-31T20:48:09.966 DEBUG:boto:Params: {} 2023-05-31T20:48:09.966 DEBUG:boto:Token: None 2023-05-31T20:48:09.967 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:48:09 GMT /pklxkd-59/obj19 2023-05-31T20:48:09.967 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:cPPztrG/mIPhBaSWeXnCSi5jUug= 2023-05-31T20:48:09.968 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:48:09 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:cPPztrG/mIPhBaSWeXnCSi5jUug=', 'Content-Length': '0'} 2023-05-31T20:48:09.969 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:44:50 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000055319f5a84e4bda4-006477b289-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:48:09 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:48:10.012 DEBUG:boto:path=/pklxkd-59/ 2023-05-31T20:48:10.013 DEBUG:boto:auth_path=/pklxkd-59/ 2023-05-31T20:48:10.013 DEBUG:boto:path=/pklxkd-59/?prefix=obj19 2023-05-31T20:48:10.013 DEBUG:boto:auth_path=/pklxkd-59/?prefix=obj19 2023-05-31T20:48:10.014 DEBUG:boto:Method: GET 2023-05-31T20:48:10.014 DEBUG:boto:Path: /pklxkd-59/?prefix=obj19 2023-05-31T20:48:10.014 DEBUG:boto:Data: 2023-05-31T20:48:10.015 DEBUG:boto:Headers: {} 2023-05-31T20:48:10.015 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:48:10.015 DEBUG:boto:Port: 8001 2023-05-31T20:48:10.016 DEBUG:boto:Params: {} 2023-05-31T20:48:10.016 DEBUG:boto:Token: None 2023-05-31T20:48:10.017 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:48:10 GMT /pklxkd-59/ 2023-05-31T20:48:10.017 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:Bd5LOmi2aGEdCv7mvkUPjKIz99o= 2023-05-31T20:48:10.017 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:48:10 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:Bd5LOmi2aGEdCv7mvkUPjKIz99o=', 'Content-Length': '0'} 2023-05-31T20:48:10.020 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c557d00e7adcea9a-006477b28a-1c17-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:48:10 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:48:10.020 DEBUG:boto:b'pklxkd-59obj191000falseobj192023-05-31T20:44:50.557Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:48:10.021 DEBUG:boto:path=/pklxkd-59/ 2023-05-31T20:48:10.021 DEBUG:boto:auth_path=/pklxkd-59/ 2023-05-31T20:48:10.022 DEBUG:boto:path=/pklxkd-59/?prefix=obj19 2023-05-31T20:48:10.022 DEBUG:boto:auth_path=/pklxkd-59/?prefix=obj19 2023-05-31T20:48:10.022 DEBUG:boto:Method: GET 2023-05-31T20:48:10.023 DEBUG:boto:Path: /pklxkd-59/?prefix=obj19 2023-05-31T20:48:10.023 DEBUG:boto:Data: 2023-05-31T20:48:10.023 DEBUG:boto:Headers: {} 2023-05-31T20:48:10.024 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:48:10.024 DEBUG:boto:Port: 8000 2023-05-31T20:48:10.025 DEBUG:boto:Params: {} 2023-05-31T20:48:10.025 DEBUG:boto:Token: None 2023-05-31T20:48:10.025 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:48:10 GMT /pklxkd-59/ 2023-05-31T20:48:10.026 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:Bd5LOmi2aGEdCv7mvkUPjKIz99o= 2023-05-31T20:48:10.026 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:48:10 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:Bd5LOmi2aGEdCv7mvkUPjKIz99o=', 'Content-Length': '0'} 2023-05-31T20:48:10.029 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000005304ce4974f4997f-006477b28a-1630-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:48:10 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:48:10.030 DEBUG:boto:b'pklxkd-59obj191000falseobj192023-05-31T20:44:50.557Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:48:10.030 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj22 2023-05-31T20:48:10.031 DEBUG:boto:path=/pklxkd-59/obj22 2023-05-31T20:48:10.031 DEBUG:boto:auth_path=/pklxkd-59/obj22 2023-05-31T20:48:10.031 DEBUG:boto:path=/pklxkd-59/obj22?versionId=null 2023-05-31T20:48:10.032 DEBUG:boto:auth_path=/pklxkd-59/obj22?versionId=null 2023-05-31T20:48:10.032 DEBUG:boto:Method: GET 2023-05-31T20:48:10.033 DEBUG:boto:Path: /pklxkd-59/obj22?versionId=null 2023-05-31T20:48:10.033 DEBUG:boto:Data: 2023-05-31T20:48:10.033 DEBUG:boto:Headers: {} 2023-05-31T20:48:10.034 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:48:10.034 DEBUG:boto:Port: 8001 2023-05-31T20:48:10.034 DEBUG:boto:Params: {} 2023-05-31T20:48:10.035 DEBUG:boto:Token: None 2023-05-31T20:48:10.035 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:48:10 GMT /pklxkd-59/obj22?versionId=null 2023-05-31T20:48:10.036 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:qEAycBSCEnNYPe2jsU95G+q+TQ0= 2023-05-31T20:48:10.036 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:48:10 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:qEAycBSCEnNYPe2jsU95G+q+TQ0=', 'Content-Length': '0'} 2023-05-31T20:48:10.038 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:39:26 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000fb8058376041baa7-006477b28a-1c17-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:48:10 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:48:10.038 DEBUG:boto:path=/pklxkd-59/obj22 2023-05-31T20:48:10.039 DEBUG:boto:auth_path=/pklxkd-59/obj22 2023-05-31T20:48:10.039 DEBUG:boto:path=/pklxkd-59/obj22?versionId=null 2023-05-31T20:48:10.039 DEBUG:boto:auth_path=/pklxkd-59/obj22?versionId=null 2023-05-31T20:48:10.040 DEBUG:boto:Method: GET 2023-05-31T20:48:10.040 DEBUG:boto:Path: /pklxkd-59/obj22?versionId=null 2023-05-31T20:48:10.040 DEBUG:boto:Data: 2023-05-31T20:48:10.041 DEBUG:boto:Headers: {} 2023-05-31T20:48:10.041 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:48:10.042 DEBUG:boto:Port: 8000 2023-05-31T20:48:10.042 DEBUG:boto:Params: {} 2023-05-31T20:48:10.042 DEBUG:boto:Token: None 2023-05-31T20:48:10.043 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:48:10 GMT /pklxkd-59/obj22?versionId=null 2023-05-31T20:48:10.043 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:qEAycBSCEnNYPe2jsU95G+q+TQ0= 2023-05-31T20:48:10.044 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:48:10 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:qEAycBSCEnNYPe2jsU95G+q+TQ0=', 'Content-Length': '0'} 2023-05-31T20:48:10.045 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:39:26 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003fbeda1016c23ef4-006477b28a-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:48:10 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:48:10.046 DEBUG:boto:path=/pklxkd-59/obj22 2023-05-31T20:48:10.046 DEBUG:boto:auth_path=/pklxkd-59/obj22 2023-05-31T20:48:10.047 DEBUG:boto:path=/pklxkd-59/obj22?versionId=null 2023-05-31T20:48:10.047 DEBUG:boto:auth_path=/pklxkd-59/obj22?versionId=null 2023-05-31T20:48:10.047 DEBUG:boto:Method: HEAD 2023-05-31T20:48:10.048 DEBUG:boto:Path: /pklxkd-59/obj22?versionId=null 2023-05-31T20:48:10.048 DEBUG:boto:Data: 2023-05-31T20:48:10.048 DEBUG:boto:Headers: {} 2023-05-31T20:48:10.049 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:48:10.049 DEBUG:boto:Port: 8001 2023-05-31T20:48:10.049 DEBUG:boto:Params: {} 2023-05-31T20:48:10.050 DEBUG:boto:Token: None 2023-05-31T20:48:10.050 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:48:10 GMT /pklxkd-59/obj22?versionId=null 2023-05-31T20:48:10.051 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:gRkpafMm4auEywBdUKw436S/rtc= 2023-05-31T20:48:10.051 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:48:10 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:gRkpafMm4auEywBdUKw436S/rtc=', 'Content-Length': '0'} 2023-05-31T20:48:10.053 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:39:26 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000076bd1e27191a6e4f-006477b28a-1c17-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:48:10 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:48:10.053 DEBUG:boto:path=/pklxkd-59/obj22 2023-05-31T20:48:10.054 DEBUG:boto:auth_path=/pklxkd-59/obj22 2023-05-31T20:48:10.054 DEBUG:boto:path=/pklxkd-59/obj22?versionId=null 2023-05-31T20:48:10.054 DEBUG:boto:auth_path=/pklxkd-59/obj22?versionId=null 2023-05-31T20:48:10.055 DEBUG:boto:Method: HEAD 2023-05-31T20:48:10.055 DEBUG:boto:Path: /pklxkd-59/obj22?versionId=null 2023-05-31T20:48:10.055 DEBUG:boto:Data: 2023-05-31T20:48:10.056 DEBUG:boto:Headers: {} 2023-05-31T20:48:10.056 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:48:10.056 DEBUG:boto:Port: 8000 2023-05-31T20:48:10.057 DEBUG:boto:Params: {} 2023-05-31T20:48:10.057 DEBUG:boto:Token: None 2023-05-31T20:48:10.057 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:48:10 GMT /pklxkd-59/obj22?versionId=null 2023-05-31T20:48:10.058 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:gRkpafMm4auEywBdUKw436S/rtc= 2023-05-31T20:48:10.058 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:48:10 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:gRkpafMm4auEywBdUKw436S/rtc=', 'Content-Length': '0'} 2023-05-31T20:48:10.060 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:39:26 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000099d681ad3c5f2ab-006477b28a-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:48:10 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:48:10.061 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj22 2023-05-31T20:48:10.061 DEBUG:boto:path=/pklxkd-59/obj22 2023-05-31T20:48:10.061 DEBUG:boto:auth_path=/pklxkd-59/obj22 2023-05-31T20:48:10.062 DEBUG:boto:path=/pklxkd-59/obj22?versionId=null 2023-05-31T20:48:10.062 DEBUG:boto:auth_path=/pklxkd-59/obj22?versionId=null 2023-05-31T20:48:10.062 DEBUG:boto:Method: GET 2023-05-31T20:48:10.063 DEBUG:boto:Path: /pklxkd-59/obj22?versionId=null 2023-05-31T20:48:10.063 DEBUG:boto:Data: 2023-05-31T20:48:10.063 DEBUG:boto:Headers: {} 2023-05-31T20:48:10.064 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:48:10.064 DEBUG:boto:Port: 8001 2023-05-31T20:48:10.065 DEBUG:boto:Params: {} 2023-05-31T20:48:10.065 DEBUG:boto:Token: None 2023-05-31T20:48:10.065 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:48:10 GMT /pklxkd-59/obj22?versionId=null 2023-05-31T20:48:10.066 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:qEAycBSCEnNYPe2jsU95G+q+TQ0= 2023-05-31T20:48:10.066 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:48:10 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:qEAycBSCEnNYPe2jsU95G+q+TQ0=', 'Content-Length': '0'} 2023-05-31T20:48:10.068 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:39:26 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000033f6618d1924125a-006477b28a-1c17-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:48:10 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:48:10.108 DEBUG:boto:path=/pklxkd-59/obj22 2023-05-31T20:48:10.109 DEBUG:boto:auth_path=/pklxkd-59/obj22 2023-05-31T20:48:10.109 DEBUG:boto:path=/pklxkd-59/obj22?versionId=null 2023-05-31T20:48:10.109 DEBUG:boto:auth_path=/pklxkd-59/obj22?versionId=null 2023-05-31T20:48:10.110 DEBUG:boto:Method: GET 2023-05-31T20:48:10.110 DEBUG:boto:Path: /pklxkd-59/obj22?versionId=null 2023-05-31T20:48:10.110 DEBUG:boto:Data: 2023-05-31T20:48:10.111 DEBUG:boto:Headers: {} 2023-05-31T20:48:10.111 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:48:10.111 DEBUG:boto:Port: 8000 2023-05-31T20:48:10.112 DEBUG:boto:Params: {} 2023-05-31T20:48:10.112 DEBUG:boto:Token: None 2023-05-31T20:48:10.113 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:48:10 GMT /pklxkd-59/obj22?versionId=null 2023-05-31T20:48:10.113 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:qEAycBSCEnNYPe2jsU95G+q+TQ0= 2023-05-31T20:48:10.113 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:48:10 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:qEAycBSCEnNYPe2jsU95G+q+TQ0=', 'Content-Length': '0'} 2023-05-31T20:48:10.115 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:39:26 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000902777735d8b014b-006477b28a-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:48:10 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:48:10.116 DEBUG:boto:path=/pklxkd-59/ 2023-05-31T20:48:10.116 DEBUG:boto:auth_path=/pklxkd-59/ 2023-05-31T20:48:10.117 DEBUG:boto:path=/pklxkd-59/?prefix=obj22 2023-05-31T20:48:10.117 DEBUG:boto:auth_path=/pklxkd-59/?prefix=obj22 2023-05-31T20:48:10.117 DEBUG:boto:Method: GET 2023-05-31T20:48:10.118 DEBUG:boto:Path: /pklxkd-59/?prefix=obj22 2023-05-31T20:48:10.118 DEBUG:boto:Data: 2023-05-31T20:48:10.119 DEBUG:boto:Headers: {} 2023-05-31T20:48:10.119 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:48:10.119 DEBUG:boto:Port: 8001 2023-05-31T20:48:10.120 DEBUG:boto:Params: {} 2023-05-31T20:48:10.120 DEBUG:boto:Token: None 2023-05-31T20:48:10.120 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:48:10 GMT /pklxkd-59/ 2023-05-31T20:48:10.121 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:Bd5LOmi2aGEdCv7mvkUPjKIz99o= 2023-05-31T20:48:10.121 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:48:10 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:Bd5LOmi2aGEdCv7mvkUPjKIz99o=', 'Content-Length': '0'} 2023-05-31T20:48:10.124 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000063b5bf595375893e-006477b28a-1c17-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:48:10 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:48:10.124 DEBUG:boto:b'pklxkd-59obj221000falseobj222023-05-31T20:39:26.473Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:48:10.125 DEBUG:boto:path=/pklxkd-59/ 2023-05-31T20:48:10.125 DEBUG:boto:auth_path=/pklxkd-59/ 2023-05-31T20:48:10.125 DEBUG:boto:path=/pklxkd-59/?prefix=obj22 2023-05-31T20:48:10.126 DEBUG:boto:auth_path=/pklxkd-59/?prefix=obj22 2023-05-31T20:48:10.126 DEBUG:boto:Method: GET 2023-05-31T20:48:10.127 DEBUG:boto:Path: /pklxkd-59/?prefix=obj22 2023-05-31T20:48:10.127 DEBUG:boto:Data: 2023-05-31T20:48:10.127 DEBUG:boto:Headers: {} 2023-05-31T20:48:10.128 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:48:10.128 DEBUG:boto:Port: 8000 2023-05-31T20:48:10.128 DEBUG:boto:Params: {} 2023-05-31T20:48:10.129 DEBUG:boto:Token: None 2023-05-31T20:48:10.129 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:48:10 GMT /pklxkd-59/ 2023-05-31T20:48:10.130 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:Bd5LOmi2aGEdCv7mvkUPjKIz99o= 2023-05-31T20:48:10.130 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:48:10 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:Bd5LOmi2aGEdCv7mvkUPjKIz99o=', 'Content-Length': '0'} 2023-05-31T20:48:10.133 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000034d7ed1a44bfd2a9-006477b28a-1630-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:48:10 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:48:10.134 DEBUG:boto:b'pklxkd-59obj221000falseobj222023-05-31T20:39:26.473Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:48:10.134 DEBUG:boto:path=/pklxkd-59/obj22 2023-05-31T20:48:10.135 DEBUG:boto:auth_path=/pklxkd-59/obj22 2023-05-31T20:48:10.135 DEBUG:boto:Method: HEAD 2023-05-31T20:48:10.135 DEBUG:boto:Path: /pklxkd-59/obj22 2023-05-31T20:48:10.136 DEBUG:boto:Data: 2023-05-31T20:48:10.136 DEBUG:boto:Headers: {} 2023-05-31T20:48:10.136 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:48:10.137 DEBUG:boto:Port: 8001 2023-05-31T20:48:10.137 DEBUG:boto:Params: {} 2023-05-31T20:48:10.137 DEBUG:boto:Token: None 2023-05-31T20:48:10.138 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:48:10 GMT /pklxkd-59/obj22 2023-05-31T20:48:10.138 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:RHj4CpDbnjoJB2wP6rKfxmFpZS8= 2023-05-31T20:48:10.138 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:48:10 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:RHj4CpDbnjoJB2wP6rKfxmFpZS8=', 'Content-Length': '0'} 2023-05-31T20:48:10.140 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:39:26 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d5e93bc82f064d5d-006477b28a-1c17-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:48:10 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:48:10.141 DEBUG:boto:path=/pklxkd-59/obj22 2023-05-31T20:48:10.141 DEBUG:boto:auth_path=/pklxkd-59/obj22 2023-05-31T20:48:10.141 DEBUG:boto:Method: HEAD 2023-05-31T20:48:10.142 DEBUG:boto:Path: /pklxkd-59/obj22 2023-05-31T20:48:10.142 DEBUG:boto:Data: 2023-05-31T20:48:10.142 DEBUG:boto:Headers: {} 2023-05-31T20:48:10.143 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:48:10.143 DEBUG:boto:Port: 8000 2023-05-31T20:48:10.143 DEBUG:boto:Params: {} 2023-05-31T20:48:10.144 DEBUG:boto:Token: None 2023-05-31T20:48:10.144 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:48:10 GMT /pklxkd-59/obj22 2023-05-31T20:48:10.145 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:RHj4CpDbnjoJB2wP6rKfxmFpZS8= 2023-05-31T20:48:10.145 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:48:10 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:RHj4CpDbnjoJB2wP6rKfxmFpZS8=', 'Content-Length': '0'} 2023-05-31T20:48:10.147 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:39:26 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ff917d9f535ac6e3-006477b28a-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:48:10 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:48:10.147 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj22 2023-05-31T20:48:10.148 DEBUG:boto:path=/pklxkd-59/obj22 2023-05-31T20:48:10.148 DEBUG:boto:auth_path=/pklxkd-59/obj22 2023-05-31T20:48:10.148 DEBUG:boto:Method: GET 2023-05-31T20:48:10.149 DEBUG:boto:Path: /pklxkd-59/obj22 2023-05-31T20:48:10.149 DEBUG:boto:Data: 2023-05-31T20:48:10.149 DEBUG:boto:Headers: {} 2023-05-31T20:48:10.150 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:48:10.150 DEBUG:boto:Port: 8001 2023-05-31T20:48:10.150 DEBUG:boto:Params: {} 2023-05-31T20:48:10.151 DEBUG:boto:Token: None 2023-05-31T20:48:10.151 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:48:10 GMT /pklxkd-59/obj22 2023-05-31T20:48:10.152 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:r+6lzBu61sQOcDBeE/kBhu6xxtQ= 2023-05-31T20:48:10.152 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:48:10 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:r+6lzBu61sQOcDBeE/kBhu6xxtQ=', 'Content-Length': '0'} 2023-05-31T20:48:10.154 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:39:26 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000093c4c214e6333c2f-006477b28a-1c17-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:48:10 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:48:10.154 DEBUG:boto:path=/pklxkd-59/obj22 2023-05-31T20:48:10.155 DEBUG:boto:auth_path=/pklxkd-59/obj22 2023-05-31T20:48:10.155 DEBUG:boto:Method: GET 2023-05-31T20:48:10.155 DEBUG:boto:Path: /pklxkd-59/obj22 2023-05-31T20:48:10.156 DEBUG:boto:Data: 2023-05-31T20:48:10.156 DEBUG:boto:Headers: {} 2023-05-31T20:48:10.157 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:48:10.157 DEBUG:boto:Port: 8000 2023-05-31T20:48:10.157 DEBUG:boto:Params: {} 2023-05-31T20:48:10.158 DEBUG:boto:Token: None 2023-05-31T20:48:10.158 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:48:10 GMT /pklxkd-59/obj22 2023-05-31T20:48:10.158 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:r+6lzBu61sQOcDBeE/kBhu6xxtQ= 2023-05-31T20:48:10.159 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:48:10 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:r+6lzBu61sQOcDBeE/kBhu6xxtQ=', 'Content-Length': '0'} 2023-05-31T20:48:10.161 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:39:26 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e90780aab30631f2-006477b28a-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:48:10 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:48:10.204 DEBUG:boto:path=/pklxkd-59/ 2023-05-31T20:48:10.205 DEBUG:boto:auth_path=/pklxkd-59/ 2023-05-31T20:48:10.205 DEBUG:boto:path=/pklxkd-59/?prefix=obj22 2023-05-31T20:48:10.205 DEBUG:boto:auth_path=/pklxkd-59/?prefix=obj22 2023-05-31T20:48:10.206 DEBUG:boto:Method: GET 2023-05-31T20:48:10.206 DEBUG:boto:Path: /pklxkd-59/?prefix=obj22 2023-05-31T20:48:10.206 DEBUG:boto:Data: 2023-05-31T20:48:10.207 DEBUG:boto:Headers: {} 2023-05-31T20:48:10.207 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:48:10.207 DEBUG:boto:Port: 8001 2023-05-31T20:48:10.208 DEBUG:boto:Params: {} 2023-05-31T20:48:10.208 DEBUG:boto:Token: None 2023-05-31T20:48:10.209 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:48:10 GMT /pklxkd-59/ 2023-05-31T20:48:10.209 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:Bd5LOmi2aGEdCv7mvkUPjKIz99o= 2023-05-31T20:48:10.210 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:48:10 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:Bd5LOmi2aGEdCv7mvkUPjKIz99o=', 'Content-Length': '0'} 2023-05-31T20:48:10.212 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ad4982df353ab065-006477b28a-1c17-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:48:10 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:48:10.213 DEBUG:boto:b'pklxkd-59obj221000falseobj222023-05-31T20:39:26.473Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:48:10.213 DEBUG:boto:path=/pklxkd-59/ 2023-05-31T20:48:10.214 DEBUG:boto:auth_path=/pklxkd-59/ 2023-05-31T20:48:10.214 DEBUG:boto:path=/pklxkd-59/?prefix=obj22 2023-05-31T20:48:10.214 DEBUG:boto:auth_path=/pklxkd-59/?prefix=obj22 2023-05-31T20:48:10.215 DEBUG:boto:Method: GET 2023-05-31T20:48:10.215 DEBUG:boto:Path: /pklxkd-59/?prefix=obj22 2023-05-31T20:48:10.215 DEBUG:boto:Data: 2023-05-31T20:48:10.216 DEBUG:boto:Headers: {} 2023-05-31T20:48:10.216 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:48:10.217 DEBUG:boto:Port: 8000 2023-05-31T20:48:10.217 DEBUG:boto:Params: {} 2023-05-31T20:48:10.217 DEBUG:boto:Token: None 2023-05-31T20:48:10.218 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:48:10 GMT /pklxkd-59/ 2023-05-31T20:48:10.218 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:Bd5LOmi2aGEdCv7mvkUPjKIz99o= 2023-05-31T20:48:10.218 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:48:10 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:Bd5LOmi2aGEdCv7mvkUPjKIz99o=', 'Content-Length': '0'} 2023-05-31T20:48:10.222 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000084829ac2a2f4972f-006477b28a-1630-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:48:10 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:48:10.222 DEBUG:boto:b'pklxkd-59obj221000falseobj222023-05-31T20:39:26.473Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:48:10.223 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj23 2023-05-31T20:48:10.223 DEBUG:boto:path=/pklxkd-59/obj23 2023-05-31T20:48:10.223 DEBUG:boto:auth_path=/pklxkd-59/obj23 2023-05-31T20:48:10.224 DEBUG:boto:path=/pklxkd-59/obj23?versionId=null 2023-05-31T20:48:10.224 DEBUG:boto:auth_path=/pklxkd-59/obj23?versionId=null 2023-05-31T20:48:10.224 DEBUG:boto:Method: GET 2023-05-31T20:48:10.225 DEBUG:boto:Path: /pklxkd-59/obj23?versionId=null 2023-05-31T20:48:10.225 DEBUG:boto:Data: 2023-05-31T20:48:10.225 DEBUG:boto:Headers: {} 2023-05-31T20:48:10.226 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:48:10.226 DEBUG:boto:Port: 8001 2023-05-31T20:48:10.227 DEBUG:boto:Params: {} 2023-05-31T20:48:10.227 DEBUG:boto:Token: None 2023-05-31T20:48:10.227 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:48:10 GMT /pklxkd-59/obj23?versionId=null 2023-05-31T20:48:10.228 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:AfOt+ZhhV4I+XH8IF3ZcQsL75Pw= 2023-05-31T20:48:10.228 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:48:10 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:AfOt+ZhhV4I+XH8IF3ZcQsL75Pw=', 'Content-Length': '0'} 2023-05-31T20:48:10.230 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:46:33 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000084c3a651feac5d6d-006477b28a-1c17-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:48:10 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:48:10.272 DEBUG:boto:path=/pklxkd-59/obj23 2023-05-31T20:48:10.273 DEBUG:boto:auth_path=/pklxkd-59/obj23 2023-05-31T20:48:10.273 DEBUG:boto:path=/pklxkd-59/obj23?versionId=null 2023-05-31T20:48:10.273 DEBUG:boto:auth_path=/pklxkd-59/obj23?versionId=null 2023-05-31T20:48:10.274 DEBUG:boto:Method: GET 2023-05-31T20:48:10.274 DEBUG:boto:Path: /pklxkd-59/obj23?versionId=null 2023-05-31T20:48:10.274 DEBUG:boto:Data: 2023-05-31T20:48:10.275 DEBUG:boto:Headers: {} 2023-05-31T20:48:10.275 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:48:10.275 DEBUG:boto:Port: 8000 2023-05-31T20:48:10.276 DEBUG:boto:Params: {} 2023-05-31T20:48:10.276 DEBUG:boto:Token: None 2023-05-31T20:48:10.277 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:48:10 GMT /pklxkd-59/obj23?versionId=null 2023-05-31T20:48:10.277 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:AfOt+ZhhV4I+XH8IF3ZcQsL75Pw= 2023-05-31T20:48:10.278 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:48:10 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:AfOt+ZhhV4I+XH8IF3ZcQsL75Pw=', 'Content-Length': '0'} 2023-05-31T20:48:10.279 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:46:33 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f960c9719d1362e3-006477b28a-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:48:10 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:48:10.280 DEBUG:boto:path=/pklxkd-59/obj23 2023-05-31T20:48:10.280 DEBUG:boto:auth_path=/pklxkd-59/obj23 2023-05-31T20:48:10.281 DEBUG:boto:path=/pklxkd-59/obj23?versionId=null 2023-05-31T20:48:10.281 DEBUG:boto:auth_path=/pklxkd-59/obj23?versionId=null 2023-05-31T20:48:10.281 DEBUG:boto:Method: HEAD 2023-05-31T20:48:10.282 DEBUG:boto:Path: /pklxkd-59/obj23?versionId=null 2023-05-31T20:48:10.282 DEBUG:boto:Data: 2023-05-31T20:48:10.282 DEBUG:boto:Headers: {} 2023-05-31T20:48:10.283 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:48:10.283 DEBUG:boto:Port: 8001 2023-05-31T20:48:10.283 DEBUG:boto:Params: {} 2023-05-31T20:48:10.284 DEBUG:boto:Token: None 2023-05-31T20:48:10.284 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:48:10 GMT /pklxkd-59/obj23?versionId=null 2023-05-31T20:48:10.284 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:Miw5/cOC7azEXaeDlqOcJRZvph0= 2023-05-31T20:48:10.285 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:48:10 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:Miw5/cOC7azEXaeDlqOcJRZvph0=', 'Content-Length': '0'} 2023-05-31T20:48:10.287 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:46:33 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b0b2d90618c3902b-006477b28a-1c17-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:48:10 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:48:10.287 DEBUG:boto:path=/pklxkd-59/obj23 2023-05-31T20:48:10.288 DEBUG:boto:auth_path=/pklxkd-59/obj23 2023-05-31T20:48:10.288 DEBUG:boto:path=/pklxkd-59/obj23?versionId=null 2023-05-31T20:48:10.288 DEBUG:boto:auth_path=/pklxkd-59/obj23?versionId=null 2023-05-31T20:48:10.289 DEBUG:boto:Method: HEAD 2023-05-31T20:48:10.289 DEBUG:boto:Path: /pklxkd-59/obj23?versionId=null 2023-05-31T20:48:10.289 DEBUG:boto:Data: 2023-05-31T20:48:10.290 DEBUG:boto:Headers: {} 2023-05-31T20:48:10.290 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:48:10.290 DEBUG:boto:Port: 8000 2023-05-31T20:48:10.291 DEBUG:boto:Params: {} 2023-05-31T20:48:10.291 DEBUG:boto:Token: None 2023-05-31T20:48:10.291 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:48:10 GMT /pklxkd-59/obj23?versionId=null 2023-05-31T20:48:10.292 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:Miw5/cOC7azEXaeDlqOcJRZvph0= 2023-05-31T20:48:10.292 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:48:10 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:Miw5/cOC7azEXaeDlqOcJRZvph0=', 'Content-Length': '0'} 2023-05-31T20:48:10.294 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:46:33 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004a6b600af530b71c-006477b28a-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:48:10 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:48:10.295 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj23 2023-05-31T20:48:10.295 DEBUG:boto:path=/pklxkd-59/obj23 2023-05-31T20:48:10.295 DEBUG:boto:auth_path=/pklxkd-59/obj23 2023-05-31T20:48:10.296 DEBUG:boto:path=/pklxkd-59/obj23?versionId=null 2023-05-31T20:48:10.296 DEBUG:boto:auth_path=/pklxkd-59/obj23?versionId=null 2023-05-31T20:48:10.296 DEBUG:boto:Method: GET 2023-05-31T20:48:10.297 DEBUG:boto:Path: /pklxkd-59/obj23?versionId=null 2023-05-31T20:48:10.297 DEBUG:boto:Data: 2023-05-31T20:48:10.297 DEBUG:boto:Headers: {} 2023-05-31T20:48:10.298 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:48:10.298 DEBUG:boto:Port: 8001 2023-05-31T20:48:10.299 DEBUG:boto:Params: {} 2023-05-31T20:48:10.299 DEBUG:boto:Token: None 2023-05-31T20:48:10.299 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:48:10 GMT /pklxkd-59/obj23?versionId=null 2023-05-31T20:48:10.300 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:AfOt+ZhhV4I+XH8IF3ZcQsL75Pw= 2023-05-31T20:48:10.300 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:48:10 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:AfOt+ZhhV4I+XH8IF3ZcQsL75Pw=', 'Content-Length': '0'} 2023-05-31T20:48:10.302 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:46:33 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d49524a644e1362c-006477b28a-1c17-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:48:10 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:48:10.303 DEBUG:boto:path=/pklxkd-59/obj23 2023-05-31T20:48:10.303 DEBUG:boto:auth_path=/pklxkd-59/obj23 2023-05-31T20:48:10.303 DEBUG:boto:path=/pklxkd-59/obj23?versionId=null 2023-05-31T20:48:10.304 DEBUG:boto:auth_path=/pklxkd-59/obj23?versionId=null 2023-05-31T20:48:10.304 DEBUG:boto:Method: GET 2023-05-31T20:48:10.304 DEBUG:boto:Path: /pklxkd-59/obj23?versionId=null 2023-05-31T20:48:10.305 DEBUG:boto:Data: 2023-05-31T20:48:10.305 DEBUG:boto:Headers: {} 2023-05-31T20:48:10.305 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:48:10.306 DEBUG:boto:Port: 8000 2023-05-31T20:48:10.306 DEBUG:boto:Params: {} 2023-05-31T20:48:10.307 DEBUG:boto:Token: None 2023-05-31T20:48:10.307 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:48:10 GMT /pklxkd-59/obj23?versionId=null 2023-05-31T20:48:10.307 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:AfOt+ZhhV4I+XH8IF3ZcQsL75Pw= 2023-05-31T20:48:10.308 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:48:10 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:AfOt+ZhhV4I+XH8IF3ZcQsL75Pw=', 'Content-Length': '0'} 2023-05-31T20:48:10.310 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:46:33 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b93b3074d4eaac02-006477b28a-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:48:10 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:48:10.310 DEBUG:boto:path=/pklxkd-59/ 2023-05-31T20:48:10.311 DEBUG:boto:auth_path=/pklxkd-59/ 2023-05-31T20:48:10.311 DEBUG:boto:path=/pklxkd-59/?prefix=obj23 2023-05-31T20:48:10.311 DEBUG:boto:auth_path=/pklxkd-59/?prefix=obj23 2023-05-31T20:48:10.312 DEBUG:boto:Method: GET 2023-05-31T20:48:10.312 DEBUG:boto:Path: /pklxkd-59/?prefix=obj23 2023-05-31T20:48:10.312 DEBUG:boto:Data: 2023-05-31T20:48:10.313 DEBUG:boto:Headers: {} 2023-05-31T20:48:10.313 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:48:10.314 DEBUG:boto:Port: 8001 2023-05-31T20:48:10.314 DEBUG:boto:Params: {} 2023-05-31T20:48:10.314 DEBUG:boto:Token: None 2023-05-31T20:48:10.315 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:48:10 GMT /pklxkd-59/ 2023-05-31T20:48:10.315 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:Bd5LOmi2aGEdCv7mvkUPjKIz99o= 2023-05-31T20:48:10.315 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:48:10 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:Bd5LOmi2aGEdCv7mvkUPjKIz99o=', 'Content-Length': '0'} 2023-05-31T20:48:10.318 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a38baef5c0eae868-006477b28a-1c17-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:48:10 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:48:10.318 DEBUG:boto:b'pklxkd-59obj231000falseobj232023-05-31T20:46:33.813Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:48:10.319 DEBUG:boto:path=/pklxkd-59/ 2023-05-31T20:48:10.320 DEBUG:boto:auth_path=/pklxkd-59/ 2023-05-31T20:48:10.320 DEBUG:boto:path=/pklxkd-59/?prefix=obj23 2023-05-31T20:48:10.320 DEBUG:boto:auth_path=/pklxkd-59/?prefix=obj23 2023-05-31T20:48:10.321 DEBUG:boto:Method: GET 2023-05-31T20:48:10.321 DEBUG:boto:Path: /pklxkd-59/?prefix=obj23 2023-05-31T20:48:10.321 DEBUG:boto:Data: 2023-05-31T20:48:10.322 DEBUG:boto:Headers: {} 2023-05-31T20:48:10.322 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:48:10.322 DEBUG:boto:Port: 8000 2023-05-31T20:48:10.323 DEBUG:boto:Params: {} 2023-05-31T20:48:10.323 DEBUG:boto:Token: None 2023-05-31T20:48:10.323 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:48:10 GMT /pklxkd-59/ 2023-05-31T20:48:10.324 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:Bd5LOmi2aGEdCv7mvkUPjKIz99o= 2023-05-31T20:48:10.324 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:48:10 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:Bd5LOmi2aGEdCv7mvkUPjKIz99o=', 'Content-Length': '0'} 2023-05-31T20:48:10.328 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f8df8701d2b6ddfe-006477b28a-1630-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:48:10 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:48:10.328 DEBUG:boto:b'pklxkd-59obj231000falseobj232023-05-31T20:46:33.813Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:48:10.329 DEBUG:boto:path=/pklxkd-59/obj23 2023-05-31T20:48:10.329 DEBUG:boto:auth_path=/pklxkd-59/obj23 2023-05-31T20:48:10.329 DEBUG:boto:Method: HEAD 2023-05-31T20:48:10.330 DEBUG:boto:Path: /pklxkd-59/obj23 2023-05-31T20:48:10.330 DEBUG:boto:Data: 2023-05-31T20:48:10.330 DEBUG:boto:Headers: {} 2023-05-31T20:48:10.331 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:48:10.331 DEBUG:boto:Port: 8001 2023-05-31T20:48:10.331 DEBUG:boto:Params: {} 2023-05-31T20:48:10.332 DEBUG:boto:Token: None 2023-05-31T20:48:10.332 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:48:10 GMT /pklxkd-59/obj23 2023-05-31T20:48:10.333 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:ci5/eAymYpQzq8bN/ccnzRehmjM= 2023-05-31T20:48:10.333 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:48:10 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:ci5/eAymYpQzq8bN/ccnzRehmjM=', 'Content-Length': '0'} 2023-05-31T20:48:10.335 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:46:33 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b2be149bef808da0-006477b28a-1c17-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:48:10 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:48:10.335 DEBUG:boto:path=/pklxkd-59/obj23 2023-05-31T20:48:10.336 DEBUG:boto:auth_path=/pklxkd-59/obj23 2023-05-31T20:48:10.336 DEBUG:boto:Method: HEAD 2023-05-31T20:48:10.336 DEBUG:boto:Path: /pklxkd-59/obj23 2023-05-31T20:48:10.337 DEBUG:boto:Data: 2023-05-31T20:48:10.337 DEBUG:boto:Headers: {} 2023-05-31T20:48:10.337 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:48:10.338 DEBUG:boto:Port: 8000 2023-05-31T20:48:10.338 DEBUG:boto:Params: {} 2023-05-31T20:48:10.338 DEBUG:boto:Token: None 2023-05-31T20:48:10.339 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:48:10 GMT /pklxkd-59/obj23 2023-05-31T20:48:10.339 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:ci5/eAymYpQzq8bN/ccnzRehmjM= 2023-05-31T20:48:10.340 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:48:10 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:ci5/eAymYpQzq8bN/ccnzRehmjM=', 'Content-Length': '0'} 2023-05-31T20:48:10.341 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:46:33 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a166f7db589b690c-006477b28a-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:48:10 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:48:10.342 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj23 2023-05-31T20:48:10.342 DEBUG:boto:path=/pklxkd-59/obj23 2023-05-31T20:48:10.343 DEBUG:boto:auth_path=/pklxkd-59/obj23 2023-05-31T20:48:10.343 DEBUG:boto:Method: GET 2023-05-31T20:48:10.343 DEBUG:boto:Path: /pklxkd-59/obj23 2023-05-31T20:48:10.344 DEBUG:boto:Data: 2023-05-31T20:48:10.344 DEBUG:boto:Headers: {} 2023-05-31T20:48:10.344 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:48:10.345 DEBUG:boto:Port: 8001 2023-05-31T20:48:10.345 DEBUG:boto:Params: {} 2023-05-31T20:48:10.346 DEBUG:boto:Token: None 2023-05-31T20:48:10.346 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:48:10 GMT /pklxkd-59/obj23 2023-05-31T20:48:10.346 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:mt3SSBi4QuZfj+8pA6Ed0RS+yUA= 2023-05-31T20:48:10.347 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:48:10 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:mt3SSBi4QuZfj+8pA6Ed0RS+yUA=', 'Content-Length': '0'} 2023-05-31T20:48:10.349 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:46:33 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000013636610f7f68973-006477b28a-1c17-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:48:10 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:48:10.349 DEBUG:boto:path=/pklxkd-59/obj23 2023-05-31T20:48:10.350 DEBUG:boto:auth_path=/pklxkd-59/obj23 2023-05-31T20:48:10.350 DEBUG:boto:Method: GET 2023-05-31T20:48:10.350 DEBUG:boto:Path: /pklxkd-59/obj23 2023-05-31T20:48:10.351 DEBUG:boto:Data: 2023-05-31T20:48:10.351 DEBUG:boto:Headers: {} 2023-05-31T20:48:10.351 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:48:10.352 DEBUG:boto:Port: 8000 2023-05-31T20:48:10.352 DEBUG:boto:Params: {} 2023-05-31T20:48:10.352 DEBUG:boto:Token: None 2023-05-31T20:48:10.353 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:48:10 GMT /pklxkd-59/obj23 2023-05-31T20:48:10.353 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:mt3SSBi4QuZfj+8pA6Ed0RS+yUA= 2023-05-31T20:48:10.354 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:48:10 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:mt3SSBi4QuZfj+8pA6Ed0RS+yUA=', 'Content-Length': '0'} 2023-05-31T20:48:10.355 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:46:33 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002b787cb538a55b34-006477b28a-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:48:10 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:48:10.356 DEBUG:boto:path=/pklxkd-59/ 2023-05-31T20:48:10.356 DEBUG:boto:auth_path=/pklxkd-59/ 2023-05-31T20:48:10.357 DEBUG:boto:path=/pklxkd-59/?prefix=obj23 2023-05-31T20:48:10.357 DEBUG:boto:auth_path=/pklxkd-59/?prefix=obj23 2023-05-31T20:48:10.357 DEBUG:boto:Method: GET 2023-05-31T20:48:10.358 DEBUG:boto:Path: /pklxkd-59/?prefix=obj23 2023-05-31T20:48:10.358 DEBUG:boto:Data: 2023-05-31T20:48:10.359 DEBUG:boto:Headers: {} 2023-05-31T20:48:10.359 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:48:10.359 DEBUG:boto:Port: 8001 2023-05-31T20:48:10.360 DEBUG:boto:Params: {} 2023-05-31T20:48:10.360 DEBUG:boto:Token: None 2023-05-31T20:48:10.360 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:48:10 GMT /pklxkd-59/ 2023-05-31T20:48:10.361 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:Bd5LOmi2aGEdCv7mvkUPjKIz99o= 2023-05-31T20:48:10.361 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:48:10 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:Bd5LOmi2aGEdCv7mvkUPjKIz99o=', 'Content-Length': '0'} 2023-05-31T20:48:10.363 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000004951158faf79fbc7-006477b28a-1c17-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:48:10 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:48:10.364 DEBUG:boto:b'pklxkd-59obj231000falseobj232023-05-31T20:46:33.813Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:48:10.365 DEBUG:boto:path=/pklxkd-59/ 2023-05-31T20:48:10.365 DEBUG:boto:auth_path=/pklxkd-59/ 2023-05-31T20:48:10.365 DEBUG:boto:path=/pklxkd-59/?prefix=obj23 2023-05-31T20:48:10.366 DEBUG:boto:auth_path=/pklxkd-59/?prefix=obj23 2023-05-31T20:48:10.366 DEBUG:boto:Method: GET 2023-05-31T20:48:10.366 DEBUG:boto:Path: /pklxkd-59/?prefix=obj23 2023-05-31T20:48:10.367 DEBUG:boto:Data: 2023-05-31T20:48:10.367 DEBUG:boto:Headers: {} 2023-05-31T20:48:10.368 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:48:10.368 DEBUG:boto:Port: 8000 2023-05-31T20:48:10.368 DEBUG:boto:Params: {} 2023-05-31T20:48:10.369 DEBUG:boto:Token: None 2023-05-31T20:48:10.369 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:48:10 GMT /pklxkd-59/ 2023-05-31T20:48:10.369 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:Bd5LOmi2aGEdCv7mvkUPjKIz99o= 2023-05-31T20:48:10.370 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:48:10 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:Bd5LOmi2aGEdCv7mvkUPjKIz99o=', 'Content-Length': '0'} 2023-05-31T20:48:10.373 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a00578fa0e7f9798-006477b28a-1630-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:48:10 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:48:10.374 DEBUG:boto:b'pklxkd-59obj231000falseobj232023-05-31T20:46:33.813Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:48:10.374 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj33 2023-05-31T20:48:10.375 DEBUG:boto:path=/pklxkd-59/obj33 2023-05-31T20:48:10.375 DEBUG:boto:auth_path=/pklxkd-59/obj33 2023-05-31T20:48:10.375 DEBUG:boto:path=/pklxkd-59/obj33?versionId=null 2023-05-31T20:48:10.376 DEBUG:boto:auth_path=/pklxkd-59/obj33?versionId=null 2023-05-31T20:48:10.376 DEBUG:boto:Method: GET 2023-05-31T20:48:10.377 DEBUG:boto:Path: /pklxkd-59/obj33?versionId=null 2023-05-31T20:48:10.377 DEBUG:boto:Data: 2023-05-31T20:48:10.377 DEBUG:boto:Headers: {} 2023-05-31T20:48:10.378 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:48:10.378 DEBUG:boto:Port: 8001 2023-05-31T20:48:10.378 DEBUG:boto:Params: {} 2023-05-31T20:48:10.379 DEBUG:boto:Token: None 2023-05-31T20:48:10.379 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:48:10 GMT /pklxkd-59/obj33?versionId=null 2023-05-31T20:48:10.379 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:HkKROS4EJa54a8u3nOLR+gqnh6o= 2023-05-31T20:48:10.380 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:48:10 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:HkKROS4EJa54a8u3nOLR+gqnh6o=', 'Content-Length': '0'} 2023-05-31T20:48:10.382 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:39:26 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000cc897ab2f98a7308-006477b28a-1c17-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:48:10 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:48:10.382 DEBUG:boto:path=/pklxkd-59/obj33 2023-05-31T20:48:10.383 DEBUG:boto:auth_path=/pklxkd-59/obj33 2023-05-31T20:48:10.383 DEBUG:boto:path=/pklxkd-59/obj33?versionId=null 2023-05-31T20:48:10.383 DEBUG:boto:auth_path=/pklxkd-59/obj33?versionId=null 2023-05-31T20:48:10.384 DEBUG:boto:Method: GET 2023-05-31T20:48:10.384 DEBUG:boto:Path: /pklxkd-59/obj33?versionId=null 2023-05-31T20:48:10.385 DEBUG:boto:Data: 2023-05-31T20:48:10.385 DEBUG:boto:Headers: {} 2023-05-31T20:48:10.385 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:48:10.386 DEBUG:boto:Port: 8000 2023-05-31T20:48:10.386 DEBUG:boto:Params: {} 2023-05-31T20:48:10.386 DEBUG:boto:Token: None 2023-05-31T20:48:10.387 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:48:10 GMT /pklxkd-59/obj33?versionId=null 2023-05-31T20:48:10.387 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:HkKROS4EJa54a8u3nOLR+gqnh6o= 2023-05-31T20:48:10.388 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:48:10 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:HkKROS4EJa54a8u3nOLR+gqnh6o=', 'Content-Length': '0'} 2023-05-31T20:48:10.390 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:39:26 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c6515199f046c06d-006477b28a-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:48:10 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:48:10.390 DEBUG:boto:path=/pklxkd-59/obj33 2023-05-31T20:48:10.391 DEBUG:boto:auth_path=/pklxkd-59/obj33 2023-05-31T20:48:10.391 DEBUG:boto:path=/pklxkd-59/obj33?versionId=null 2023-05-31T20:48:10.391 DEBUG:boto:auth_path=/pklxkd-59/obj33?versionId=null 2023-05-31T20:48:10.392 DEBUG:boto:Method: HEAD 2023-05-31T20:48:10.392 DEBUG:boto:Path: /pklxkd-59/obj33?versionId=null 2023-05-31T20:48:10.392 DEBUG:boto:Data: 2023-05-31T20:48:10.393 DEBUG:boto:Headers: {} 2023-05-31T20:48:10.393 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:48:10.393 DEBUG:boto:Port: 8001 2023-05-31T20:48:10.394 DEBUG:boto:Params: {} 2023-05-31T20:48:10.394 DEBUG:boto:Token: None 2023-05-31T20:48:10.395 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:48:10 GMT /pklxkd-59/obj33?versionId=null 2023-05-31T20:48:10.395 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:fnVLbXTjsUE/ugc5MWmsRlKaQ00= 2023-05-31T20:48:10.395 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:48:10 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:fnVLbXTjsUE/ugc5MWmsRlKaQ00=', 'Content-Length': '0'} 2023-05-31T20:48:10.397 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:39:26 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004a20b443a9b3f8a4-006477b28a-1c17-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:48:10 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:48:10.398 DEBUG:boto:path=/pklxkd-59/obj33 2023-05-31T20:48:10.398 DEBUG:boto:auth_path=/pklxkd-59/obj33 2023-05-31T20:48:10.398 DEBUG:boto:path=/pklxkd-59/obj33?versionId=null 2023-05-31T20:48:10.399 DEBUG:boto:auth_path=/pklxkd-59/obj33?versionId=null 2023-05-31T20:48:10.399 DEBUG:boto:Method: HEAD 2023-05-31T20:48:10.399 DEBUG:boto:Path: /pklxkd-59/obj33?versionId=null 2023-05-31T20:48:10.400 DEBUG:boto:Data: 2023-05-31T20:48:10.400 DEBUG:boto:Headers: {} 2023-05-31T20:48:10.400 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:48:10.401 DEBUG:boto:Port: 8000 2023-05-31T20:48:10.401 DEBUG:boto:Params: {} 2023-05-31T20:48:10.401 DEBUG:boto:Token: None 2023-05-31T20:48:10.402 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:48:10 GMT /pklxkd-59/obj33?versionId=null 2023-05-31T20:48:10.402 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:fnVLbXTjsUE/ugc5MWmsRlKaQ00= 2023-05-31T20:48:10.403 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:48:10 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:fnVLbXTjsUE/ugc5MWmsRlKaQ00=', 'Content-Length': '0'} 2023-05-31T20:48:10.404 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:39:26 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000af2bd48c907216fb-006477b28a-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:48:10 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:48:10.405 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj33 2023-05-31T20:48:10.405 DEBUG:boto:path=/pklxkd-59/obj33 2023-05-31T20:48:10.406 DEBUG:boto:auth_path=/pklxkd-59/obj33 2023-05-31T20:48:10.406 DEBUG:boto:path=/pklxkd-59/obj33?versionId=null 2023-05-31T20:48:10.406 DEBUG:boto:auth_path=/pklxkd-59/obj33?versionId=null 2023-05-31T20:48:10.407 DEBUG:boto:Method: GET 2023-05-31T20:48:10.407 DEBUG:boto:Path: /pklxkd-59/obj33?versionId=null 2023-05-31T20:48:10.407 DEBUG:boto:Data: 2023-05-31T20:48:10.408 DEBUG:boto:Headers: {} 2023-05-31T20:48:10.408 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:48:10.409 DEBUG:boto:Port: 8001 2023-05-31T20:48:10.409 DEBUG:boto:Params: {} 2023-05-31T20:48:10.409 DEBUG:boto:Token: None 2023-05-31T20:48:10.410 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:48:10 GMT /pklxkd-59/obj33?versionId=null 2023-05-31T20:48:10.410 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:HkKROS4EJa54a8u3nOLR+gqnh6o= 2023-05-31T20:48:10.410 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:48:10 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:HkKROS4EJa54a8u3nOLR+gqnh6o=', 'Content-Length': '0'} 2023-05-31T20:48:10.412 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:39:26 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007e8a62b7e4867b3a-006477b28a-1c17-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:48:10 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:48:10.413 DEBUG:boto:path=/pklxkd-59/obj33 2023-05-31T20:48:10.413 DEBUG:boto:auth_path=/pklxkd-59/obj33 2023-05-31T20:48:10.414 DEBUG:boto:path=/pklxkd-59/obj33?versionId=null 2023-05-31T20:48:10.414 DEBUG:boto:auth_path=/pklxkd-59/obj33?versionId=null 2023-05-31T20:48:10.415 DEBUG:boto:Method: GET 2023-05-31T20:48:10.415 DEBUG:boto:Path: /pklxkd-59/obj33?versionId=null 2023-05-31T20:48:10.415 DEBUG:boto:Data: 2023-05-31T20:48:10.416 DEBUG:boto:Headers: {} 2023-05-31T20:48:10.416 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:48:10.416 DEBUG:boto:Port: 8000 2023-05-31T20:48:10.417 DEBUG:boto:Params: {} 2023-05-31T20:48:10.417 DEBUG:boto:Token: None 2023-05-31T20:48:10.418 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:48:10 GMT /pklxkd-59/obj33?versionId=null 2023-05-31T20:48:10.418 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:HkKROS4EJa54a8u3nOLR+gqnh6o= 2023-05-31T20:48:10.418 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:48:10 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:HkKROS4EJa54a8u3nOLR+gqnh6o=', 'Content-Length': '0'} 2023-05-31T20:48:10.420 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:39:26 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000007968f7dbcf2fc35-006477b28a-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:48:10 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:48:10.464 DEBUG:boto:path=/pklxkd-59/ 2023-05-31T20:48:10.465 DEBUG:boto:auth_path=/pklxkd-59/ 2023-05-31T20:48:10.466 DEBUG:boto:path=/pklxkd-59/?prefix=obj33 2023-05-31T20:48:10.466 DEBUG:boto:auth_path=/pklxkd-59/?prefix=obj33 2023-05-31T20:48:10.467 DEBUG:boto:Method: GET 2023-05-31T20:48:10.467 DEBUG:boto:Path: /pklxkd-59/?prefix=obj33 2023-05-31T20:48:10.467 DEBUG:boto:Data: 2023-05-31T20:48:10.468 DEBUG:boto:Headers: {} 2023-05-31T20:48:10.468 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:48:10.469 DEBUG:boto:Port: 8001 2023-05-31T20:48:10.469 DEBUG:boto:Params: {} 2023-05-31T20:48:10.469 DEBUG:boto:Token: None 2023-05-31T20:48:10.470 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:48:10 GMT /pklxkd-59/ 2023-05-31T20:48:10.470 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:Bd5LOmi2aGEdCv7mvkUPjKIz99o= 2023-05-31T20:48:10.471 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:48:10 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:Bd5LOmi2aGEdCv7mvkUPjKIz99o=', 'Content-Length': '0'} 2023-05-31T20:48:10.473 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000588fb62f23c659ab-006477b28a-1c17-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:48:10 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:48:10.473 DEBUG:boto:b'pklxkd-59obj331000falseobj332023-05-31T20:39:26.534Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:48:10.474 DEBUG:boto:path=/pklxkd-59/ 2023-05-31T20:48:10.474 DEBUG:boto:auth_path=/pklxkd-59/ 2023-05-31T20:48:10.475 DEBUG:boto:path=/pklxkd-59/?prefix=obj33 2023-05-31T20:48:10.475 DEBUG:boto:auth_path=/pklxkd-59/?prefix=obj33 2023-05-31T20:48:10.476 DEBUG:boto:Method: GET 2023-05-31T20:48:10.476 DEBUG:boto:Path: /pklxkd-59/?prefix=obj33 2023-05-31T20:48:10.476 DEBUG:boto:Data: 2023-05-31T20:48:10.477 DEBUG:boto:Headers: {} 2023-05-31T20:48:10.477 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:48:10.477 DEBUG:boto:Port: 8000 2023-05-31T20:48:10.478 DEBUG:boto:Params: {} 2023-05-31T20:48:10.478 DEBUG:boto:Token: None 2023-05-31T20:48:10.479 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:48:10 GMT /pklxkd-59/ 2023-05-31T20:48:10.479 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:Bd5LOmi2aGEdCv7mvkUPjKIz99o= 2023-05-31T20:48:10.479 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:48:10 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:Bd5LOmi2aGEdCv7mvkUPjKIz99o=', 'Content-Length': '0'} 2023-05-31T20:48:10.483 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000009b52775518e9aa14-006477b28a-1630-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:48:10 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:48:10.484 DEBUG:boto:b'pklxkd-59obj331000falseobj332023-05-31T20:39:26.534Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:48:10.485 DEBUG:boto:path=/pklxkd-59/obj33 2023-05-31T20:48:10.485 DEBUG:boto:auth_path=/pklxkd-59/obj33 2023-05-31T20:48:10.485 DEBUG:boto:Method: HEAD 2023-05-31T20:48:10.486 DEBUG:boto:Path: /pklxkd-59/obj33 2023-05-31T20:48:10.486 DEBUG:boto:Data: 2023-05-31T20:48:10.486 DEBUG:boto:Headers: {} 2023-05-31T20:48:10.487 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:48:10.487 DEBUG:boto:Port: 8001 2023-05-31T20:48:10.487 DEBUG:boto:Params: {} 2023-05-31T20:48:10.488 DEBUG:boto:Token: None 2023-05-31T20:48:10.488 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:48:10 GMT /pklxkd-59/obj33 2023-05-31T20:48:10.489 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:sWVO9v0nDY9GdA7U7Jv481lu5sU= 2023-05-31T20:48:10.489 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:48:10 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:sWVO9v0nDY9GdA7U7Jv481lu5sU=', 'Content-Length': '0'} 2023-05-31T20:48:10.491 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:39:26 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b1f354aa74ac7c81-006477b28a-1c17-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:48:10 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:48:10.491 DEBUG:boto:path=/pklxkd-59/obj33 2023-05-31T20:48:10.492 DEBUG:boto:auth_path=/pklxkd-59/obj33 2023-05-31T20:48:10.492 DEBUG:boto:Method: HEAD 2023-05-31T20:48:10.492 DEBUG:boto:Path: /pklxkd-59/obj33 2023-05-31T20:48:10.493 DEBUG:boto:Data: 2023-05-31T20:48:10.493 DEBUG:boto:Headers: {} 2023-05-31T20:48:10.493 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:48:10.494 DEBUG:boto:Port: 8000 2023-05-31T20:48:10.494 DEBUG:boto:Params: {} 2023-05-31T20:48:10.494 DEBUG:boto:Token: None 2023-05-31T20:48:10.495 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:48:10 GMT /pklxkd-59/obj33 2023-05-31T20:48:10.495 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:sWVO9v0nDY9GdA7U7Jv481lu5sU= 2023-05-31T20:48:10.496 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:48:10 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:sWVO9v0nDY9GdA7U7Jv481lu5sU=', 'Content-Length': '0'} 2023-05-31T20:48:10.498 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:39:26 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000520898009c458c62-006477b28a-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:48:10 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:48:10.498 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj33 2023-05-31T20:48:10.499 DEBUG:boto:path=/pklxkd-59/obj33 2023-05-31T20:48:10.499 DEBUG:boto:auth_path=/pklxkd-59/obj33 2023-05-31T20:48:10.499 DEBUG:boto:Method: GET 2023-05-31T20:48:10.500 DEBUG:boto:Path: /pklxkd-59/obj33 2023-05-31T20:48:10.500 DEBUG:boto:Data: 2023-05-31T20:48:10.500 DEBUG:boto:Headers: {} 2023-05-31T20:48:10.501 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:48:10.501 DEBUG:boto:Port: 8001 2023-05-31T20:48:10.502 DEBUG:boto:Params: {} 2023-05-31T20:48:10.502 DEBUG:boto:Token: None 2023-05-31T20:48:10.502 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:48:10 GMT /pklxkd-59/obj33 2023-05-31T20:48:10.503 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:m9CSRGpgakAJCS6VsjXIYp5I8Fg= 2023-05-31T20:48:10.503 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:48:10 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:m9CSRGpgakAJCS6VsjXIYp5I8Fg=', 'Content-Length': '0'} 2023-05-31T20:48:10.505 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:39:26 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e91de062ac790539-006477b28a-1c17-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:48:10 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:48:10.548 DEBUG:boto:path=/pklxkd-59/obj33 2023-05-31T20:48:10.549 DEBUG:boto:auth_path=/pklxkd-59/obj33 2023-05-31T20:48:10.549 DEBUG:boto:Method: GET 2023-05-31T20:48:10.549 DEBUG:boto:Path: /pklxkd-59/obj33 2023-05-31T20:48:10.550 DEBUG:boto:Data: 2023-05-31T20:48:10.550 DEBUG:boto:Headers: {} 2023-05-31T20:48:10.550 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:48:10.551 DEBUG:boto:Port: 8000 2023-05-31T20:48:10.551 DEBUG:boto:Params: {} 2023-05-31T20:48:10.552 DEBUG:boto:Token: None 2023-05-31T20:48:10.552 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:48:10 GMT /pklxkd-59/obj33 2023-05-31T20:48:10.552 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:m9CSRGpgakAJCS6VsjXIYp5I8Fg= 2023-05-31T20:48:10.553 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:48:10 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:m9CSRGpgakAJCS6VsjXIYp5I8Fg=', 'Content-Length': '0'} 2023-05-31T20:48:10.555 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:39:26 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002be0e4cf4418c123-006477b28a-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:48:10 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:48:10.555 DEBUG:boto:path=/pklxkd-59/ 2023-05-31T20:48:10.556 DEBUG:boto:auth_path=/pklxkd-59/ 2023-05-31T20:48:10.556 DEBUG:boto:path=/pklxkd-59/?prefix=obj33 2023-05-31T20:48:10.556 DEBUG:boto:auth_path=/pklxkd-59/?prefix=obj33 2023-05-31T20:48:10.557 DEBUG:boto:Method: GET 2023-05-31T20:48:10.557 DEBUG:boto:Path: /pklxkd-59/?prefix=obj33 2023-05-31T20:48:10.558 DEBUG:boto:Data: 2023-05-31T20:48:10.558 DEBUG:boto:Headers: {} 2023-05-31T20:48:10.558 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:48:10.559 DEBUG:boto:Port: 8001 2023-05-31T20:48:10.559 DEBUG:boto:Params: {} 2023-05-31T20:48:10.559 DEBUG:boto:Token: None 2023-05-31T20:48:10.560 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:48:10 GMT /pklxkd-59/ 2023-05-31T20:48:10.560 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:Bd5LOmi2aGEdCv7mvkUPjKIz99o= 2023-05-31T20:48:10.561 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:48:10 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:Bd5LOmi2aGEdCv7mvkUPjKIz99o=', 'Content-Length': '0'} 2023-05-31T20:48:10.563 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000003ad570c43a67e5f2-006477b28a-1c17-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:48:10 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:48:10.564 DEBUG:boto:b'pklxkd-59obj331000falseobj332023-05-31T20:39:26.534Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:48:10.564 DEBUG:boto:path=/pklxkd-59/ 2023-05-31T20:48:10.565 DEBUG:boto:auth_path=/pklxkd-59/ 2023-05-31T20:48:10.565 DEBUG:boto:path=/pklxkd-59/?prefix=obj33 2023-05-31T20:48:10.566 DEBUG:boto:auth_path=/pklxkd-59/?prefix=obj33 2023-05-31T20:48:10.566 DEBUG:boto:Method: GET 2023-05-31T20:48:10.566 DEBUG:boto:Path: /pklxkd-59/?prefix=obj33 2023-05-31T20:48:10.567 DEBUG:boto:Data: 2023-05-31T20:48:10.567 DEBUG:boto:Headers: {} 2023-05-31T20:48:10.567 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:48:10.568 DEBUG:boto:Port: 8000 2023-05-31T20:48:10.568 DEBUG:boto:Params: {} 2023-05-31T20:48:10.568 DEBUG:boto:Token: None 2023-05-31T20:48:10.569 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:48:10 GMT /pklxkd-59/ 2023-05-31T20:48:10.569 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:Bd5LOmi2aGEdCv7mvkUPjKIz99o= 2023-05-31T20:48:10.570 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:48:10 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:Bd5LOmi2aGEdCv7mvkUPjKIz99o=', 'Content-Length': '0'} 2023-05-31T20:48:10.572 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f2fb56c5da12329b-006477b28a-1630-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:48:10 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:48:10.573 DEBUG:boto:b'pklxkd-59obj331000falseobj332023-05-31T20:39:26.534Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:48:10.574 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj34 2023-05-31T20:48:10.574 DEBUG:boto:path=/pklxkd-59/obj34 2023-05-31T20:48:10.574 DEBUG:boto:auth_path=/pklxkd-59/obj34 2023-05-31T20:48:10.575 DEBUG:boto:path=/pklxkd-59/obj34?versionId=null 2023-05-31T20:48:10.575 DEBUG:boto:auth_path=/pklxkd-59/obj34?versionId=null 2023-05-31T20:48:10.575 DEBUG:boto:Method: GET 2023-05-31T20:48:10.576 DEBUG:boto:Path: /pklxkd-59/obj34?versionId=null 2023-05-31T20:48:10.576 DEBUG:boto:Data: 2023-05-31T20:48:10.576 DEBUG:boto:Headers: {} 2023-05-31T20:48:10.577 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:48:10.577 DEBUG:boto:Port: 8001 2023-05-31T20:48:10.578 DEBUG:boto:Params: {} 2023-05-31T20:48:10.578 DEBUG:boto:Token: None 2023-05-31T20:48:10.578 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:48:10 GMT /pklxkd-59/obj34?versionId=null 2023-05-31T20:48:10.579 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:+mMTM6qVGzpM2V2vPUpZ4x8W5wk= 2023-05-31T20:48:10.579 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:48:10 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:+mMTM6qVGzpM2V2vPUpZ4x8W5wk=', 'Content-Length': '0'} 2023-05-31T20:48:10.581 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:43:07 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008bb71fa81824796a-006477b28a-1c17-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:48:10 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:48:10.582 DEBUG:boto:path=/pklxkd-59/obj34 2023-05-31T20:48:10.582 DEBUG:boto:auth_path=/pklxkd-59/obj34 2023-05-31T20:48:10.582 DEBUG:boto:path=/pklxkd-59/obj34?versionId=null 2023-05-31T20:48:10.583 DEBUG:boto:auth_path=/pklxkd-59/obj34?versionId=null 2023-05-31T20:48:10.583 DEBUG:boto:Method: GET 2023-05-31T20:48:10.583 DEBUG:boto:Path: /pklxkd-59/obj34?versionId=null 2023-05-31T20:48:10.584 DEBUG:boto:Data: 2023-05-31T20:48:10.584 DEBUG:boto:Headers: {} 2023-05-31T20:48:10.585 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:48:10.585 DEBUG:boto:Port: 8000 2023-05-31T20:48:10.585 DEBUG:boto:Params: {} 2023-05-31T20:48:10.586 DEBUG:boto:Token: None 2023-05-31T20:48:10.586 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:48:10 GMT /pklxkd-59/obj34?versionId=null 2023-05-31T20:48:10.586 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:+mMTM6qVGzpM2V2vPUpZ4x8W5wk= 2023-05-31T20:48:10.587 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:48:10 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:+mMTM6qVGzpM2V2vPUpZ4x8W5wk=', 'Content-Length': '0'} 2023-05-31T20:48:10.589 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:43:07 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000fbd7c3fbb5ac6f4e-006477b28a-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:48:10 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:48:10.632 DEBUG:boto:path=/pklxkd-59/obj34 2023-05-31T20:48:10.633 DEBUG:boto:auth_path=/pklxkd-59/obj34 2023-05-31T20:48:10.633 DEBUG:boto:path=/pklxkd-59/obj34?versionId=null 2023-05-31T20:48:10.633 DEBUG:boto:auth_path=/pklxkd-59/obj34?versionId=null 2023-05-31T20:48:10.634 DEBUG:boto:Method: HEAD 2023-05-31T20:48:10.634 DEBUG:boto:Path: /pklxkd-59/obj34?versionId=null 2023-05-31T20:48:10.634 DEBUG:boto:Data: 2023-05-31T20:48:10.635 DEBUG:boto:Headers: {} 2023-05-31T20:48:10.635 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:48:10.635 DEBUG:boto:Port: 8001 2023-05-31T20:48:10.636 DEBUG:boto:Params: {} 2023-05-31T20:48:10.636 DEBUG:boto:Token: None 2023-05-31T20:48:10.636 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:48:10 GMT /pklxkd-59/obj34?versionId=null 2023-05-31T20:48:10.637 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:bf+jjrJl0qyxNOKdespHlryhiWA= 2023-05-31T20:48:10.637 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:48:10 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:bf+jjrJl0qyxNOKdespHlryhiWA=', 'Content-Length': '0'} 2023-05-31T20:48:10.639 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:43:07 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f87de842c2433d64-006477b28a-1c17-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:48:10 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:48:10.640 DEBUG:boto:path=/pklxkd-59/obj34 2023-05-31T20:48:10.640 DEBUG:boto:auth_path=/pklxkd-59/obj34 2023-05-31T20:48:10.640 DEBUG:boto:path=/pklxkd-59/obj34?versionId=null 2023-05-31T20:48:10.641 DEBUG:boto:auth_path=/pklxkd-59/obj34?versionId=null 2023-05-31T20:48:10.641 DEBUG:boto:Method: HEAD 2023-05-31T20:48:10.641 DEBUG:boto:Path: /pklxkd-59/obj34?versionId=null 2023-05-31T20:48:10.642 DEBUG:boto:Data: 2023-05-31T20:48:10.642 DEBUG:boto:Headers: {} 2023-05-31T20:48:10.642 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:48:10.643 DEBUG:boto:Port: 8000 2023-05-31T20:48:10.643 DEBUG:boto:Params: {} 2023-05-31T20:48:10.643 DEBUG:boto:Token: None 2023-05-31T20:48:10.644 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:48:10 GMT /pklxkd-59/obj34?versionId=null 2023-05-31T20:48:10.644 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:bf+jjrJl0qyxNOKdespHlryhiWA= 2023-05-31T20:48:10.645 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:48:10 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:bf+jjrJl0qyxNOKdespHlryhiWA=', 'Content-Length': '0'} 2023-05-31T20:48:10.646 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:43:07 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c85b72f236446df4-006477b28a-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:48:10 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:48:10.647 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj34 2023-05-31T20:48:10.647 DEBUG:boto:path=/pklxkd-59/obj34 2023-05-31T20:48:10.647 DEBUG:boto:auth_path=/pklxkd-59/obj34 2023-05-31T20:48:10.648 DEBUG:boto:path=/pklxkd-59/obj34?versionId=null 2023-05-31T20:48:10.648 DEBUG:boto:auth_path=/pklxkd-59/obj34?versionId=null 2023-05-31T20:48:10.649 DEBUG:boto:Method: GET 2023-05-31T20:48:10.649 DEBUG:boto:Path: /pklxkd-59/obj34?versionId=null 2023-05-31T20:48:10.649 DEBUG:boto:Data: 2023-05-31T20:48:10.650 DEBUG:boto:Headers: {} 2023-05-31T20:48:10.650 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:48:10.650 DEBUG:boto:Port: 8001 2023-05-31T20:48:10.651 DEBUG:boto:Params: {} 2023-05-31T20:48:10.651 DEBUG:boto:Token: None 2023-05-31T20:48:10.651 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:48:10 GMT /pklxkd-59/obj34?versionId=null 2023-05-31T20:48:10.652 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:+mMTM6qVGzpM2V2vPUpZ4x8W5wk= 2023-05-31T20:48:10.652 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:48:10 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:+mMTM6qVGzpM2V2vPUpZ4x8W5wk=', 'Content-Length': '0'} 2023-05-31T20:48:10.654 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:43:07 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007168d6f96e518d31-006477b28a-1c17-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:48:10 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:48:10.655 DEBUG:boto:path=/pklxkd-59/obj34 2023-05-31T20:48:10.655 DEBUG:boto:auth_path=/pklxkd-59/obj34 2023-05-31T20:48:10.655 DEBUG:boto:path=/pklxkd-59/obj34?versionId=null 2023-05-31T20:48:10.656 DEBUG:boto:auth_path=/pklxkd-59/obj34?versionId=null 2023-05-31T20:48:10.656 DEBUG:boto:Method: GET 2023-05-31T20:48:10.657 DEBUG:boto:Path: /pklxkd-59/obj34?versionId=null 2023-05-31T20:48:10.657 DEBUG:boto:Data: 2023-05-31T20:48:10.657 DEBUG:boto:Headers: {} 2023-05-31T20:48:10.658 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:48:10.658 DEBUG:boto:Port: 8000 2023-05-31T20:48:10.658 DEBUG:boto:Params: {} 2023-05-31T20:48:10.659 DEBUG:boto:Token: None 2023-05-31T20:48:10.659 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:48:10 GMT /pklxkd-59/obj34?versionId=null 2023-05-31T20:48:10.660 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:+mMTM6qVGzpM2V2vPUpZ4x8W5wk= 2023-05-31T20:48:10.660 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:48:10 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:+mMTM6qVGzpM2V2vPUpZ4x8W5wk=', 'Content-Length': '0'} 2023-05-31T20:48:10.662 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:43:07 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000023b051ad0ac3cae9-006477b28a-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:48:10 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:48:10.663 DEBUG:boto:path=/pklxkd-59/ 2023-05-31T20:48:10.663 DEBUG:boto:auth_path=/pklxkd-59/ 2023-05-31T20:48:10.663 DEBUG:boto:path=/pklxkd-59/?prefix=obj34 2023-05-31T20:48:10.664 DEBUG:boto:auth_path=/pklxkd-59/?prefix=obj34 2023-05-31T20:48:10.664 DEBUG:boto:Method: GET 2023-05-31T20:48:10.664 DEBUG:boto:Path: /pklxkd-59/?prefix=obj34 2023-05-31T20:48:10.665 DEBUG:boto:Data: 2023-05-31T20:48:10.665 DEBUG:boto:Headers: {} 2023-05-31T20:48:10.665 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:48:10.666 DEBUG:boto:Port: 8001 2023-05-31T20:48:10.666 DEBUG:boto:Params: {} 2023-05-31T20:48:10.667 DEBUG:boto:Token: None 2023-05-31T20:48:10.667 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:48:10 GMT /pklxkd-59/ 2023-05-31T20:48:10.667 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:Bd5LOmi2aGEdCv7mvkUPjKIz99o= 2023-05-31T20:48:10.668 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:48:10 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:Bd5LOmi2aGEdCv7mvkUPjKIz99o=', 'Content-Length': '0'} 2023-05-31T20:48:10.670 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000089801f9ecf3997e5-006477b28a-1c17-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:48:10 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:48:10.671 DEBUG:boto:b'pklxkd-59obj341000falseobj342023-05-31T20:43:07.994Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:48:10.672 DEBUG:boto:path=/pklxkd-59/ 2023-05-31T20:48:10.672 DEBUG:boto:auth_path=/pklxkd-59/ 2023-05-31T20:48:10.672 DEBUG:boto:path=/pklxkd-59/?prefix=obj34 2023-05-31T20:48:10.673 DEBUG:boto:auth_path=/pklxkd-59/?prefix=obj34 2023-05-31T20:48:10.673 DEBUG:boto:Method: GET 2023-05-31T20:48:10.673 DEBUG:boto:Path: /pklxkd-59/?prefix=obj34 2023-05-31T20:48:10.674 DEBUG:boto:Data: 2023-05-31T20:48:10.674 DEBUG:boto:Headers: {} 2023-05-31T20:48:10.675 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:48:10.675 DEBUG:boto:Port: 8000 2023-05-31T20:48:10.675 DEBUG:boto:Params: {} 2023-05-31T20:48:10.676 DEBUG:boto:Token: None 2023-05-31T20:48:10.676 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:48:10 GMT /pklxkd-59/ 2023-05-31T20:48:10.677 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:Bd5LOmi2aGEdCv7mvkUPjKIz99o= 2023-05-31T20:48:10.677 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:48:10 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:Bd5LOmi2aGEdCv7mvkUPjKIz99o=', 'Content-Length': '0'} 2023-05-31T20:48:10.680 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000002d860ebec24c708e-006477b28a-1630-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:48:10 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:48:10.681 DEBUG:boto:b'pklxkd-59obj341000falseobj342023-05-31T20:43:07.994Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:48:10.681 DEBUG:boto:path=/pklxkd-59/obj34 2023-05-31T20:48:10.682 DEBUG:boto:auth_path=/pklxkd-59/obj34 2023-05-31T20:48:10.682 DEBUG:boto:Method: HEAD 2023-05-31T20:48:10.683 DEBUG:boto:Path: /pklxkd-59/obj34 2023-05-31T20:48:10.683 DEBUG:boto:Data: 2023-05-31T20:48:10.683 DEBUG:boto:Headers: {} 2023-05-31T20:48:10.684 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:48:10.684 DEBUG:boto:Port: 8001 2023-05-31T20:48:10.684 DEBUG:boto:Params: {} 2023-05-31T20:48:10.685 DEBUG:boto:Token: None 2023-05-31T20:48:10.685 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:48:10 GMT /pklxkd-59/obj34 2023-05-31T20:48:10.685 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:x7m8PjBh6m67lpKuAcsSGvquxKM= 2023-05-31T20:48:10.686 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:48:10 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:x7m8PjBh6m67lpKuAcsSGvquxKM=', 'Content-Length': '0'} 2023-05-31T20:48:10.688 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:43:07 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000bbb99088389c6eb4-006477b28a-1c17-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:48:10 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:48:10.688 DEBUG:boto:path=/pklxkd-59/obj34 2023-05-31T20:48:10.689 DEBUG:boto:auth_path=/pklxkd-59/obj34 2023-05-31T20:48:10.689 DEBUG:boto:Method: HEAD 2023-05-31T20:48:10.689 DEBUG:boto:Path: /pklxkd-59/obj34 2023-05-31T20:48:10.690 DEBUG:boto:Data: 2023-05-31T20:48:10.690 DEBUG:boto:Headers: {} 2023-05-31T20:48:10.690 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:48:10.691 DEBUG:boto:Port: 8000 2023-05-31T20:48:10.691 DEBUG:boto:Params: {} 2023-05-31T20:48:10.691 DEBUG:boto:Token: None 2023-05-31T20:48:10.692 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:48:10 GMT /pklxkd-59/obj34 2023-05-31T20:48:10.692 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:x7m8PjBh6m67lpKuAcsSGvquxKM= 2023-05-31T20:48:10.693 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:48:10 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:x7m8PjBh6m67lpKuAcsSGvquxKM=', 'Content-Length': '0'} 2023-05-31T20:48:10.695 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:43:07 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000fc6211d0a37aee6e-006477b28a-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:48:10 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:48:10.695 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj34 2023-05-31T20:48:10.695 DEBUG:boto:path=/pklxkd-59/obj34 2023-05-31T20:48:10.696 DEBUG:boto:auth_path=/pklxkd-59/obj34 2023-05-31T20:48:10.696 DEBUG:boto:Method: GET 2023-05-31T20:48:10.697 DEBUG:boto:Path: /pklxkd-59/obj34 2023-05-31T20:48:10.697 DEBUG:boto:Data: 2023-05-31T20:48:10.697 DEBUG:boto:Headers: {} 2023-05-31T20:48:10.698 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:48:10.698 DEBUG:boto:Port: 8001 2023-05-31T20:48:10.698 DEBUG:boto:Params: {} 2023-05-31T20:48:10.699 DEBUG:boto:Token: None 2023-05-31T20:48:10.699 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:48:10 GMT /pklxkd-59/obj34 2023-05-31T20:48:10.700 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:CmS6NjynhwglbYO2mf+JnpmD7jw= 2023-05-31T20:48:10.700 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:48:10 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:CmS6NjynhwglbYO2mf+JnpmD7jw=', 'Content-Length': '0'} 2023-05-31T20:48:10.702 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:43:07 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000df67d6b7b6e7aed5-006477b28a-1c17-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:48:10 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:48:10.703 DEBUG:boto:path=/pklxkd-59/obj34 2023-05-31T20:48:10.703 DEBUG:boto:auth_path=/pklxkd-59/obj34 2023-05-31T20:48:10.703 DEBUG:boto:Method: GET 2023-05-31T20:48:10.704 DEBUG:boto:Path: /pklxkd-59/obj34 2023-05-31T20:48:10.704 DEBUG:boto:Data: 2023-05-31T20:48:10.705 DEBUG:boto:Headers: {} 2023-05-31T20:48:10.705 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:48:10.705 DEBUG:boto:Port: 8000 2023-05-31T20:48:10.706 DEBUG:boto:Params: {} 2023-05-31T20:48:10.706 DEBUG:boto:Token: None 2023-05-31T20:48:10.706 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:48:10 GMT /pklxkd-59/obj34 2023-05-31T20:48:10.707 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:CmS6NjynhwglbYO2mf+JnpmD7jw= 2023-05-31T20:48:10.707 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:48:10 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:CmS6NjynhwglbYO2mf+JnpmD7jw=', 'Content-Length': '0'} 2023-05-31T20:48:10.709 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:43:07 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007a92b0197e85b9d7-006477b28a-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:48:10 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:48:10.710 DEBUG:boto:path=/pklxkd-59/ 2023-05-31T20:48:10.710 DEBUG:boto:auth_path=/pklxkd-59/ 2023-05-31T20:48:10.710 DEBUG:boto:path=/pklxkd-59/?prefix=obj34 2023-05-31T20:48:10.711 DEBUG:boto:auth_path=/pklxkd-59/?prefix=obj34 2023-05-31T20:48:10.711 DEBUG:boto:Method: GET 2023-05-31T20:48:10.711 DEBUG:boto:Path: /pklxkd-59/?prefix=obj34 2023-05-31T20:48:10.712 DEBUG:boto:Data: 2023-05-31T20:48:10.712 DEBUG:boto:Headers: {} 2023-05-31T20:48:10.713 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:48:10.713 DEBUG:boto:Port: 8001 2023-05-31T20:48:10.713 DEBUG:boto:Params: {} 2023-05-31T20:48:10.714 DEBUG:boto:Token: None 2023-05-31T20:48:10.714 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:48:10 GMT /pklxkd-59/ 2023-05-31T20:48:10.714 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:Bd5LOmi2aGEdCv7mvkUPjKIz99o= 2023-05-31T20:48:10.715 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:48:10 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:Bd5LOmi2aGEdCv7mvkUPjKIz99o=', 'Content-Length': '0'} 2023-05-31T20:48:10.717 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000009781cf61b8a77f47-006477b28a-1c17-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:48:10 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:48:10.718 DEBUG:boto:b'pklxkd-59obj341000falseobj342023-05-31T20:43:07.994Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:48:10.718 DEBUG:boto:path=/pklxkd-59/ 2023-05-31T20:48:10.719 DEBUG:boto:auth_path=/pklxkd-59/ 2023-05-31T20:48:10.719 DEBUG:boto:path=/pklxkd-59/?prefix=obj34 2023-05-31T20:48:10.719 DEBUG:boto:auth_path=/pklxkd-59/?prefix=obj34 2023-05-31T20:48:10.720 DEBUG:boto:Method: GET 2023-05-31T20:48:10.720 DEBUG:boto:Path: /pklxkd-59/?prefix=obj34 2023-05-31T20:48:10.721 DEBUG:boto:Data: 2023-05-31T20:48:10.721 DEBUG:boto:Headers: {} 2023-05-31T20:48:10.721 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:48:10.722 DEBUG:boto:Port: 8000 2023-05-31T20:48:10.722 DEBUG:boto:Params: {} 2023-05-31T20:48:10.722 DEBUG:boto:Token: None 2023-05-31T20:48:10.723 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:48:10 GMT /pklxkd-59/ 2023-05-31T20:48:10.723 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:Bd5LOmi2aGEdCv7mvkUPjKIz99o= 2023-05-31T20:48:10.724 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:48:10 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:Bd5LOmi2aGEdCv7mvkUPjKIz99o=', 'Content-Length': '0'} 2023-05-31T20:48:10.726 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000af4121ac7d11c251-006477b28a-1630-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:48:10 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:48:10.768 DEBUG:boto:b'pklxkd-59obj341000falseobj342023-05-31T20:43:07.994Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:48:10.769 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj38 2023-05-31T20:48:10.769 DEBUG:boto:path=/pklxkd-59/obj38 2023-05-31T20:48:10.770 DEBUG:boto:auth_path=/pklxkd-59/obj38 2023-05-31T20:48:10.770 DEBUG:boto:path=/pklxkd-59/obj38?versionId=null 2023-05-31T20:48:10.770 DEBUG:boto:auth_path=/pklxkd-59/obj38?versionId=null 2023-05-31T20:48:10.771 DEBUG:boto:Method: GET 2023-05-31T20:48:10.771 DEBUG:boto:Path: /pklxkd-59/obj38?versionId=null 2023-05-31T20:48:10.771 DEBUG:boto:Data: 2023-05-31T20:48:10.772 DEBUG:boto:Headers: {} 2023-05-31T20:48:10.772 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:48:10.773 DEBUG:boto:Port: 8001 2023-05-31T20:48:10.773 DEBUG:boto:Params: {} 2023-05-31T20:48:10.773 DEBUG:boto:Token: None 2023-05-31T20:48:10.774 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:48:10 GMT /pklxkd-59/obj38?versionId=null 2023-05-31T20:48:10.774 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:ygZIeoU3IPF+tJBCVEhShFYZH1g= 2023-05-31T20:48:10.775 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:48:10 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:ygZIeoU3IPF+tJBCVEhShFYZH1g=', 'Content-Length': '0'} 2023-05-31T20:48:10.776 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:44:50 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f002940732822758-006477b28a-1c17-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:48:10 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:48:10.777 DEBUG:boto:path=/pklxkd-59/obj38 2023-05-31T20:48:10.777 DEBUG:boto:auth_path=/pklxkd-59/obj38 2023-05-31T20:48:10.778 DEBUG:boto:path=/pklxkd-59/obj38?versionId=null 2023-05-31T20:48:10.778 DEBUG:boto:auth_path=/pklxkd-59/obj38?versionId=null 2023-05-31T20:48:10.778 DEBUG:boto:Method: GET 2023-05-31T20:48:10.779 DEBUG:boto:Path: /pklxkd-59/obj38?versionId=null 2023-05-31T20:48:10.779 DEBUG:boto:Data: 2023-05-31T20:48:10.779 DEBUG:boto:Headers: {} 2023-05-31T20:48:10.780 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:48:10.780 DEBUG:boto:Port: 8000 2023-05-31T20:48:10.781 DEBUG:boto:Params: {} 2023-05-31T20:48:10.781 DEBUG:boto:Token: None 2023-05-31T20:48:10.781 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:48:10 GMT /pklxkd-59/obj38?versionId=null 2023-05-31T20:48:10.782 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:ygZIeoU3IPF+tJBCVEhShFYZH1g= 2023-05-31T20:48:10.782 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:48:10 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:ygZIeoU3IPF+tJBCVEhShFYZH1g=', 'Content-Length': '0'} 2023-05-31T20:48:10.784 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:44:50 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000060ebb458216cd518-006477b28a-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:48:10 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:48:10.785 DEBUG:boto:path=/pklxkd-59/obj38 2023-05-31T20:48:10.785 DEBUG:boto:auth_path=/pklxkd-59/obj38 2023-05-31T20:48:10.786 DEBUG:boto:path=/pklxkd-59/obj38?versionId=null 2023-05-31T20:48:10.786 DEBUG:boto:auth_path=/pklxkd-59/obj38?versionId=null 2023-05-31T20:48:10.786 DEBUG:boto:Method: HEAD 2023-05-31T20:48:10.787 DEBUG:boto:Path: /pklxkd-59/obj38?versionId=null 2023-05-31T20:48:10.787 DEBUG:boto:Data: 2023-05-31T20:48:10.787 DEBUG:boto:Headers: {} 2023-05-31T20:48:10.788 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:48:10.788 DEBUG:boto:Port: 8001 2023-05-31T20:48:10.788 DEBUG:boto:Params: {} 2023-05-31T20:48:10.789 DEBUG:boto:Token: None 2023-05-31T20:48:10.789 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:48:10 GMT /pklxkd-59/obj38?versionId=null 2023-05-31T20:48:10.789 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:ZL7ni8+ZQju/ScEKBFfZVbhNO/c= 2023-05-31T20:48:10.790 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:48:10 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:ZL7ni8+ZQju/ScEKBFfZVbhNO/c=', 'Content-Length': '0'} 2023-05-31T20:48:10.792 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:44:50 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007029b58543f38462-006477b28a-1c17-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:48:10 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:48:10.792 DEBUG:boto:path=/pklxkd-59/obj38 2023-05-31T20:48:10.792 DEBUG:boto:auth_path=/pklxkd-59/obj38 2023-05-31T20:48:10.793 DEBUG:boto:path=/pklxkd-59/obj38?versionId=null 2023-05-31T20:48:10.793 DEBUG:boto:auth_path=/pklxkd-59/obj38?versionId=null 2023-05-31T20:48:10.793 DEBUG:boto:Method: HEAD 2023-05-31T20:48:10.794 DEBUG:boto:Path: /pklxkd-59/obj38?versionId=null 2023-05-31T20:48:10.794 DEBUG:boto:Data: 2023-05-31T20:48:10.794 DEBUG:boto:Headers: {} 2023-05-31T20:48:10.795 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:48:10.795 DEBUG:boto:Port: 8000 2023-05-31T20:48:10.795 DEBUG:boto:Params: {} 2023-05-31T20:48:10.796 DEBUG:boto:Token: None 2023-05-31T20:48:10.796 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:48:10 GMT /pklxkd-59/obj38?versionId=null 2023-05-31T20:48:10.797 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:ZL7ni8+ZQju/ScEKBFfZVbhNO/c= 2023-05-31T20:48:10.797 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:48:10 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:ZL7ni8+ZQju/ScEKBFfZVbhNO/c=', 'Content-Length': '0'} 2023-05-31T20:48:10.799 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:44:50 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000802a0c159058cdb8-006477b28a-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:48:10 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:48:10.799 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj38 2023-05-31T20:48:10.800 DEBUG:boto:path=/pklxkd-59/obj38 2023-05-31T20:48:10.800 DEBUG:boto:auth_path=/pklxkd-59/obj38 2023-05-31T20:48:10.800 DEBUG:boto:path=/pklxkd-59/obj38?versionId=null 2023-05-31T20:48:10.801 DEBUG:boto:auth_path=/pklxkd-59/obj38?versionId=null 2023-05-31T20:48:10.801 DEBUG:boto:Method: GET 2023-05-31T20:48:10.802 DEBUG:boto:Path: /pklxkd-59/obj38?versionId=null 2023-05-31T20:48:10.802 DEBUG:boto:Data: 2023-05-31T20:48:10.802 DEBUG:boto:Headers: {} 2023-05-31T20:48:10.803 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:48:10.803 DEBUG:boto:Port: 8001 2023-05-31T20:48:10.803 DEBUG:boto:Params: {} 2023-05-31T20:48:10.804 DEBUG:boto:Token: None 2023-05-31T20:48:10.804 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:48:10 GMT /pklxkd-59/obj38?versionId=null 2023-05-31T20:48:10.805 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:ygZIeoU3IPF+tJBCVEhShFYZH1g= 2023-05-31T20:48:10.805 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:48:10 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:ygZIeoU3IPF+tJBCVEhShFYZH1g=', 'Content-Length': '0'} 2023-05-31T20:48:10.807 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:44:50 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000070caf41bf596a008-006477b28a-1c17-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:48:10 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:48:10.807 DEBUG:boto:path=/pklxkd-59/obj38 2023-05-31T20:48:10.808 DEBUG:boto:auth_path=/pklxkd-59/obj38 2023-05-31T20:48:10.808 DEBUG:boto:path=/pklxkd-59/obj38?versionId=null 2023-05-31T20:48:10.809 DEBUG:boto:auth_path=/pklxkd-59/obj38?versionId=null 2023-05-31T20:48:10.809 DEBUG:boto:Method: GET 2023-05-31T20:48:10.809 DEBUG:boto:Path: /pklxkd-59/obj38?versionId=null 2023-05-31T20:48:10.810 DEBUG:boto:Data: 2023-05-31T20:48:10.810 DEBUG:boto:Headers: {} 2023-05-31T20:48:10.810 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:48:10.811 DEBUG:boto:Port: 8000 2023-05-31T20:48:10.811 DEBUG:boto:Params: {} 2023-05-31T20:48:10.811 DEBUG:boto:Token: None 2023-05-31T20:48:10.812 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:48:10 GMT /pklxkd-59/obj38?versionId=null 2023-05-31T20:48:10.812 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:ygZIeoU3IPF+tJBCVEhShFYZH1g= 2023-05-31T20:48:10.813 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:48:10 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:ygZIeoU3IPF+tJBCVEhShFYZH1g=', 'Content-Length': '0'} 2023-05-31T20:48:10.815 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:44:50 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008270e0b1f79e5acb-006477b28a-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:48:10 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:48:10.815 DEBUG:boto:path=/pklxkd-59/ 2023-05-31T20:48:10.816 DEBUG:boto:auth_path=/pklxkd-59/ 2023-05-31T20:48:10.816 DEBUG:boto:path=/pklxkd-59/?prefix=obj38 2023-05-31T20:48:10.817 DEBUG:boto:auth_path=/pklxkd-59/?prefix=obj38 2023-05-31T20:48:10.817 DEBUG:boto:Method: GET 2023-05-31T20:48:10.817 DEBUG:boto:Path: /pklxkd-59/?prefix=obj38 2023-05-31T20:48:10.818 DEBUG:boto:Data: 2023-05-31T20:48:10.818 DEBUG:boto:Headers: {} 2023-05-31T20:48:10.818 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:48:10.819 DEBUG:boto:Port: 8001 2023-05-31T20:48:10.819 DEBUG:boto:Params: {} 2023-05-31T20:48:10.819 DEBUG:boto:Token: None 2023-05-31T20:48:10.820 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:48:10 GMT /pklxkd-59/ 2023-05-31T20:48:10.820 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:Bd5LOmi2aGEdCv7mvkUPjKIz99o= 2023-05-31T20:48:10.821 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:48:10 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:Bd5LOmi2aGEdCv7mvkUPjKIz99o=', 'Content-Length': '0'} 2023-05-31T20:48:10.823 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000034259e1dd2c9c775-006477b28a-1c17-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:48:10 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:48:10.824 DEBUG:boto:b'pklxkd-59obj381000falseobj382023-05-31T20:44:50.610Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:48:10.824 DEBUG:boto:path=/pklxkd-59/ 2023-05-31T20:48:10.825 DEBUG:boto:auth_path=/pklxkd-59/ 2023-05-31T20:48:10.825 DEBUG:boto:path=/pklxkd-59/?prefix=obj38 2023-05-31T20:48:10.825 DEBUG:boto:auth_path=/pklxkd-59/?prefix=obj38 2023-05-31T20:48:10.826 DEBUG:boto:Method: GET 2023-05-31T20:48:10.826 DEBUG:boto:Path: /pklxkd-59/?prefix=obj38 2023-05-31T20:48:10.827 DEBUG:boto:Data: 2023-05-31T20:48:10.827 DEBUG:boto:Headers: {} 2023-05-31T20:48:10.827 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:48:10.828 DEBUG:boto:Port: 8000 2023-05-31T20:48:10.828 DEBUG:boto:Params: {} 2023-05-31T20:48:10.828 DEBUG:boto:Token: None 2023-05-31T20:48:10.829 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:48:10 GMT /pklxkd-59/ 2023-05-31T20:48:10.829 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:Bd5LOmi2aGEdCv7mvkUPjKIz99o= 2023-05-31T20:48:10.830 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:48:10 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:Bd5LOmi2aGEdCv7mvkUPjKIz99o=', 'Content-Length': '0'} 2023-05-31T20:48:10.833 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000017e84f67c9c9863c-006477b28a-1630-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:48:10 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:48:10.833 DEBUG:boto:b'pklxkd-59obj381000falseobj382023-05-31T20:44:50.610Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:48:10.834 DEBUG:boto:path=/pklxkd-59/obj38 2023-05-31T20:48:10.834 DEBUG:boto:auth_path=/pklxkd-59/obj38 2023-05-31T20:48:10.834 DEBUG:boto:Method: HEAD 2023-05-31T20:48:10.835 DEBUG:boto:Path: /pklxkd-59/obj38 2023-05-31T20:48:10.835 DEBUG:boto:Data: 2023-05-31T20:48:10.835 DEBUG:boto:Headers: {} 2023-05-31T20:48:10.836 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:48:10.836 DEBUG:boto:Port: 8001 2023-05-31T20:48:10.836 DEBUG:boto:Params: {} 2023-05-31T20:48:10.837 DEBUG:boto:Token: None 2023-05-31T20:48:10.837 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:48:10 GMT /pklxkd-59/obj38 2023-05-31T20:48:10.838 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:h5TFsJxNbUTQHP6F5hT+KOGGRHI= 2023-05-31T20:48:10.838 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:48:10 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:h5TFsJxNbUTQHP6F5hT+KOGGRHI=', 'Content-Length': '0'} 2023-05-31T20:48:10.840 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:44:50 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000019d0e4eadb86c90f-006477b28a-1c17-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:48:10 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:48:10.840 DEBUG:boto:path=/pklxkd-59/obj38 2023-05-31T20:48:10.841 DEBUG:boto:auth_path=/pklxkd-59/obj38 2023-05-31T20:48:10.841 DEBUG:boto:Method: HEAD 2023-05-31T20:48:10.841 DEBUG:boto:Path: /pklxkd-59/obj38 2023-05-31T20:48:10.842 DEBUG:boto:Data: 2023-05-31T20:48:10.842 DEBUG:boto:Headers: {} 2023-05-31T20:48:10.842 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:48:10.843 DEBUG:boto:Port: 8000 2023-05-31T20:48:10.843 DEBUG:boto:Params: {} 2023-05-31T20:48:10.843 DEBUG:boto:Token: None 2023-05-31T20:48:10.844 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:48:10 GMT /pklxkd-59/obj38 2023-05-31T20:48:10.844 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:h5TFsJxNbUTQHP6F5hT+KOGGRHI= 2023-05-31T20:48:10.844 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:48:10 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:h5TFsJxNbUTQHP6F5hT+KOGGRHI=', 'Content-Length': '0'} 2023-05-31T20:48:10.846 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:44:50 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007ea42de89ef54a7f-006477b28a-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:48:10 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:48:10.847 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj38 2023-05-31T20:48:10.847 DEBUG:boto:path=/pklxkd-59/obj38 2023-05-31T20:48:10.848 DEBUG:boto:auth_path=/pklxkd-59/obj38 2023-05-31T20:48:10.848 DEBUG:boto:Method: GET 2023-05-31T20:48:10.848 DEBUG:boto:Path: /pklxkd-59/obj38 2023-05-31T20:48:10.849 DEBUG:boto:Data: 2023-05-31T20:48:10.849 DEBUG:boto:Headers: {} 2023-05-31T20:48:10.849 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:48:10.850 DEBUG:boto:Port: 8001 2023-05-31T20:48:10.850 DEBUG:boto:Params: {} 2023-05-31T20:48:10.850 DEBUG:boto:Token: None 2023-05-31T20:48:10.851 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:48:10 GMT /pklxkd-59/obj38 2023-05-31T20:48:10.851 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:wzgafWP9SQXYi+PUBm+7rYjSMDs= 2023-05-31T20:48:10.852 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:48:10 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:wzgafWP9SQXYi+PUBm+7rYjSMDs=', 'Content-Length': '0'} 2023-05-31T20:48:10.853 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:44:50 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ed3bfeb7c6b34d04-006477b28a-1c17-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:48:10 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:48:10.854 DEBUG:boto:path=/pklxkd-59/obj38 2023-05-31T20:48:10.854 DEBUG:boto:auth_path=/pklxkd-59/obj38 2023-05-31T20:48:10.855 DEBUG:boto:Method: GET 2023-05-31T20:48:10.855 DEBUG:boto:Path: /pklxkd-59/obj38 2023-05-31T20:48:10.856 DEBUG:boto:Data: 2023-05-31T20:48:10.856 DEBUG:boto:Headers: {} 2023-05-31T20:48:10.856 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:48:10.857 DEBUG:boto:Port: 8000 2023-05-31T20:48:10.857 DEBUG:boto:Params: {} 2023-05-31T20:48:10.857 DEBUG:boto:Token: None 2023-05-31T20:48:10.858 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:48:10 GMT /pklxkd-59/obj38 2023-05-31T20:48:10.858 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:wzgafWP9SQXYi+PUBm+7rYjSMDs= 2023-05-31T20:48:10.859 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:48:10 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:wzgafWP9SQXYi+PUBm+7rYjSMDs=', 'Content-Length': '0'} 2023-05-31T20:48:10.860 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:44:50 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003f4873ab5e22f648-006477b28a-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:48:10 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:48:10.908 DEBUG:boto:path=/pklxkd-59/ 2023-05-31T20:48:10.909 DEBUG:boto:auth_path=/pklxkd-59/ 2023-05-31T20:48:10.909 DEBUG:boto:path=/pklxkd-59/?prefix=obj38 2023-05-31T20:48:10.909 DEBUG:boto:auth_path=/pklxkd-59/?prefix=obj38 2023-05-31T20:48:10.910 DEBUG:boto:Method: GET 2023-05-31T20:48:10.910 DEBUG:boto:Path: /pklxkd-59/?prefix=obj38 2023-05-31T20:48:10.911 DEBUG:boto:Data: 2023-05-31T20:48:10.911 DEBUG:boto:Headers: {} 2023-05-31T20:48:10.911 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:48:10.912 DEBUG:boto:Port: 8001 2023-05-31T20:48:10.912 DEBUG:boto:Params: {} 2023-05-31T20:48:10.912 DEBUG:boto:Token: None 2023-05-31T20:48:10.913 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:48:10 GMT /pklxkd-59/ 2023-05-31T20:48:10.913 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:Bd5LOmi2aGEdCv7mvkUPjKIz99o= 2023-05-31T20:48:10.914 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:48:10 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:Bd5LOmi2aGEdCv7mvkUPjKIz99o=', 'Content-Length': '0'} 2023-05-31T20:48:10.916 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e50d8ae524011247-006477b28a-1c17-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:48:10 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:48:10.917 DEBUG:boto:b'pklxkd-59obj381000falseobj382023-05-31T20:44:50.610Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:48:10.917 DEBUG:boto:path=/pklxkd-59/ 2023-05-31T20:48:10.918 DEBUG:boto:auth_path=/pklxkd-59/ 2023-05-31T20:48:10.918 DEBUG:boto:path=/pklxkd-59/?prefix=obj38 2023-05-31T20:48:10.918 DEBUG:boto:auth_path=/pklxkd-59/?prefix=obj38 2023-05-31T20:48:10.919 DEBUG:boto:Method: GET 2023-05-31T20:48:10.919 DEBUG:boto:Path: /pklxkd-59/?prefix=obj38 2023-05-31T20:48:10.920 DEBUG:boto:Data: 2023-05-31T20:48:10.920 DEBUG:boto:Headers: {} 2023-05-31T20:48:10.920 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:48:10.921 DEBUG:boto:Port: 8000 2023-05-31T20:48:10.921 DEBUG:boto:Params: {} 2023-05-31T20:48:10.921 DEBUG:boto:Token: None 2023-05-31T20:48:10.922 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:48:10 GMT /pklxkd-59/ 2023-05-31T20:48:10.922 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:Bd5LOmi2aGEdCv7mvkUPjKIz99o= 2023-05-31T20:48:10.923 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:48:10 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:Bd5LOmi2aGEdCv7mvkUPjKIz99o=', 'Content-Length': '0'} 2023-05-31T20:48:10.925 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000eab336600ca49aa0-006477b28a-1630-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:48:10 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:48:10.926 DEBUG:boto:b'pklxkd-59obj381000falseobj382023-05-31T20:44:50.610Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:48:10.927 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj44 2023-05-31T20:48:10.927 DEBUG:boto:path=/pklxkd-59/obj44 2023-05-31T20:48:10.928 DEBUG:boto:auth_path=/pklxkd-59/obj44 2023-05-31T20:48:10.928 DEBUG:boto:path=/pklxkd-59/obj44?versionId=null 2023-05-31T20:48:10.928 DEBUG:boto:auth_path=/pklxkd-59/obj44?versionId=null 2023-05-31T20:48:10.929 DEBUG:boto:Method: GET 2023-05-31T20:48:10.929 DEBUG:boto:Path: /pklxkd-59/obj44?versionId=null 2023-05-31T20:48:10.929 DEBUG:boto:Data: 2023-05-31T20:48:10.930 DEBUG:boto:Headers: {} 2023-05-31T20:48:10.930 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:48:10.930 DEBUG:boto:Port: 8001 2023-05-31T20:48:10.931 DEBUG:boto:Params: {} 2023-05-31T20:48:10.931 DEBUG:boto:Token: None 2023-05-31T20:48:10.932 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:48:10 GMT /pklxkd-59/obj44?versionId=null 2023-05-31T20:48:10.932 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:q3eSo3E6WngirbeH+J9UKStGSeM= 2023-05-31T20:48:10.932 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:48:10 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:q3eSo3E6WngirbeH+J9UKStGSeM=', 'Content-Length': '0'} 2023-05-31T20:48:10.934 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:39:26 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000098de559ef617854e-006477b28a-1c17-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:48:10 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:48:10.976 DEBUG:boto:path=/pklxkd-59/obj44 2023-05-31T20:48:10.977 DEBUG:boto:auth_path=/pklxkd-59/obj44 2023-05-31T20:48:10.977 DEBUG:boto:path=/pklxkd-59/obj44?versionId=null 2023-05-31T20:48:10.977 DEBUG:boto:auth_path=/pklxkd-59/obj44?versionId=null 2023-05-31T20:48:10.978 DEBUG:boto:Method: GET 2023-05-31T20:48:10.978 DEBUG:boto:Path: /pklxkd-59/obj44?versionId=null 2023-05-31T20:48:10.978 DEBUG:boto:Data: 2023-05-31T20:48:10.979 DEBUG:boto:Headers: {} 2023-05-31T20:48:10.979 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:48:10.980 DEBUG:boto:Port: 8000 2023-05-31T20:48:10.980 DEBUG:boto:Params: {} 2023-05-31T20:48:10.980 DEBUG:boto:Token: None 2023-05-31T20:48:10.981 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:48:10 GMT /pklxkd-59/obj44?versionId=null 2023-05-31T20:48:10.981 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:q3eSo3E6WngirbeH+J9UKStGSeM= 2023-05-31T20:48:10.982 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:48:10 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:q3eSo3E6WngirbeH+J9UKStGSeM=', 'Content-Length': '0'} 2023-05-31T20:48:10.984 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:39:26 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009b024f2d7d319631-006477b28a-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:48:10 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:48:10.984 DEBUG:boto:path=/pklxkd-59/obj44 2023-05-31T20:48:10.985 DEBUG:boto:auth_path=/pklxkd-59/obj44 2023-05-31T20:48:10.985 DEBUG:boto:path=/pklxkd-59/obj44?versionId=null 2023-05-31T20:48:10.985 DEBUG:boto:auth_path=/pklxkd-59/obj44?versionId=null 2023-05-31T20:48:10.986 DEBUG:boto:Method: HEAD 2023-05-31T20:48:10.986 DEBUG:boto:Path: /pklxkd-59/obj44?versionId=null 2023-05-31T20:48:10.986 DEBUG:boto:Data: 2023-05-31T20:48:10.987 DEBUG:boto:Headers: {} 2023-05-31T20:48:10.987 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:48:10.987 DEBUG:boto:Port: 8001 2023-05-31T20:48:10.988 DEBUG:boto:Params: {} 2023-05-31T20:48:10.988 DEBUG:boto:Token: None 2023-05-31T20:48:10.988 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:48:10 GMT /pklxkd-59/obj44?versionId=null 2023-05-31T20:48:10.989 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:H0LM6EnN7n44PLf2KKXRS2PGyg4= 2023-05-31T20:48:10.989 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:48:10 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:H0LM6EnN7n44PLf2KKXRS2PGyg4=', 'Content-Length': '0'} 2023-05-31T20:48:10.991 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:39:26 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000af79fc3425544436-006477b28a-1c17-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:48:10 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:48:10.991 DEBUG:boto:path=/pklxkd-59/obj44 2023-05-31T20:48:10.992 DEBUG:boto:auth_path=/pklxkd-59/obj44 2023-05-31T20:48:10.992 DEBUG:boto:path=/pklxkd-59/obj44?versionId=null 2023-05-31T20:48:10.992 DEBUG:boto:auth_path=/pklxkd-59/obj44?versionId=null 2023-05-31T20:48:10.993 DEBUG:boto:Method: HEAD 2023-05-31T20:48:10.993 DEBUG:boto:Path: /pklxkd-59/obj44?versionId=null 2023-05-31T20:48:10.993 DEBUG:boto:Data: 2023-05-31T20:48:10.994 DEBUG:boto:Headers: {} 2023-05-31T20:48:10.994 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:48:10.994 DEBUG:boto:Port: 8000 2023-05-31T20:48:10.995 DEBUG:boto:Params: {} 2023-05-31T20:48:10.995 DEBUG:boto:Token: None 2023-05-31T20:48:10.995 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:48:10 GMT /pklxkd-59/obj44?versionId=null 2023-05-31T20:48:10.996 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:H0LM6EnN7n44PLf2KKXRS2PGyg4= 2023-05-31T20:48:10.996 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:48:10 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:H0LM6EnN7n44PLf2KKXRS2PGyg4=', 'Content-Length': '0'} 2023-05-31T20:48:10.998 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:39:26 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000403d9d29993cf2fc-006477b28a-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:48:10 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:48:10.999 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj44 2023-05-31T20:48:10.999 DEBUG:boto:path=/pklxkd-59/obj44 2023-05-31T20:48:10.999 DEBUG:boto:auth_path=/pklxkd-59/obj44 2023-05-31T20:48:11.000 DEBUG:boto:path=/pklxkd-59/obj44?versionId=null 2023-05-31T20:48:11.000 DEBUG:boto:auth_path=/pklxkd-59/obj44?versionId=null 2023-05-31T20:48:11.000 DEBUG:boto:Method: GET 2023-05-31T20:48:11.001 DEBUG:boto:Path: /pklxkd-59/obj44?versionId=null 2023-05-31T20:48:11.001 DEBUG:boto:Data: 2023-05-31T20:48:11.002 DEBUG:boto:Headers: {} 2023-05-31T20:48:11.002 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:48:11.002 DEBUG:boto:Port: 8001 2023-05-31T20:48:11.003 DEBUG:boto:Params: {} 2023-05-31T20:48:11.003 DEBUG:boto:Token: None 2023-05-31T20:48:11.003 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:48:11 GMT /pklxkd-59/obj44?versionId=null 2023-05-31T20:48:11.004 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:A0JCHNVKOK+JvGaDmlpaLOccQww= 2023-05-31T20:48:11.004 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:48:11 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:A0JCHNVKOK+JvGaDmlpaLOccQww=', 'Content-Length': '0'} 2023-05-31T20:48:11.006 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:39:26 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002670724b26fe1567-006477b28b-1c17-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:48:11 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:48:11.007 DEBUG:boto:path=/pklxkd-59/obj44 2023-05-31T20:48:11.007 DEBUG:boto:auth_path=/pklxkd-59/obj44 2023-05-31T20:48:11.008 DEBUG:boto:path=/pklxkd-59/obj44?versionId=null 2023-05-31T20:48:11.008 DEBUG:boto:auth_path=/pklxkd-59/obj44?versionId=null 2023-05-31T20:48:11.008 DEBUG:boto:Method: GET 2023-05-31T20:48:11.009 DEBUG:boto:Path: /pklxkd-59/obj44?versionId=null 2023-05-31T20:48:11.009 DEBUG:boto:Data: 2023-05-31T20:48:11.009 DEBUG:boto:Headers: {} 2023-05-31T20:48:11.010 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:48:11.010 DEBUG:boto:Port: 8000 2023-05-31T20:48:11.010 DEBUG:boto:Params: {} 2023-05-31T20:48:11.011 DEBUG:boto:Token: None 2023-05-31T20:48:11.011 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:48:11 GMT /pklxkd-59/obj44?versionId=null 2023-05-31T20:48:11.012 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:A0JCHNVKOK+JvGaDmlpaLOccQww= 2023-05-31T20:48:11.012 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:48:11 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:A0JCHNVKOK+JvGaDmlpaLOccQww=', 'Content-Length': '0'} 2023-05-31T20:48:11.014 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:39:26 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000071a96b2e8cd383c5-006477b28b-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:48:11 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:48:11.015 DEBUG:boto:path=/pklxkd-59/ 2023-05-31T20:48:11.015 DEBUG:boto:auth_path=/pklxkd-59/ 2023-05-31T20:48:11.015 DEBUG:boto:path=/pklxkd-59/?prefix=obj44 2023-05-31T20:48:11.016 DEBUG:boto:auth_path=/pklxkd-59/?prefix=obj44 2023-05-31T20:48:11.016 DEBUG:boto:Method: GET 2023-05-31T20:48:11.016 DEBUG:boto:Path: /pklxkd-59/?prefix=obj44 2023-05-31T20:48:11.017 DEBUG:boto:Data: 2023-05-31T20:48:11.017 DEBUG:boto:Headers: {} 2023-05-31T20:48:11.017 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:48:11.018 DEBUG:boto:Port: 8001 2023-05-31T20:48:11.018 DEBUG:boto:Params: {} 2023-05-31T20:48:11.019 DEBUG:boto:Token: None 2023-05-31T20:48:11.019 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:48:11 GMT /pklxkd-59/ 2023-05-31T20:48:11.019 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:l166NKUsN4SBNL48MAZina5NQhQ= 2023-05-31T20:48:11.020 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:48:11 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:l166NKUsN4SBNL48MAZina5NQhQ=', 'Content-Length': '0'} 2023-05-31T20:48:11.022 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d887892af2621605-006477b28b-1c17-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:48:11 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:48:11.023 DEBUG:boto:b'pklxkd-59obj441000falseobj442023-05-31T20:39:26.565Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:48:11.023 DEBUG:boto:path=/pklxkd-59/ 2023-05-31T20:48:11.024 DEBUG:boto:auth_path=/pklxkd-59/ 2023-05-31T20:48:11.024 DEBUG:boto:path=/pklxkd-59/?prefix=obj44 2023-05-31T20:48:11.025 DEBUG:boto:auth_path=/pklxkd-59/?prefix=obj44 2023-05-31T20:48:11.025 DEBUG:boto:Method: GET 2023-05-31T20:48:11.025 DEBUG:boto:Path: /pklxkd-59/?prefix=obj44 2023-05-31T20:48:11.026 DEBUG:boto:Data: 2023-05-31T20:48:11.026 DEBUG:boto:Headers: {} 2023-05-31T20:48:11.026 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:48:11.027 DEBUG:boto:Port: 8000 2023-05-31T20:48:11.027 DEBUG:boto:Params: {} 2023-05-31T20:48:11.027 DEBUG:boto:Token: None 2023-05-31T20:48:11.028 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:48:11 GMT /pklxkd-59/ 2023-05-31T20:48:11.028 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:l166NKUsN4SBNL48MAZina5NQhQ= 2023-05-31T20:48:11.029 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:48:11 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:l166NKUsN4SBNL48MAZina5NQhQ=', 'Content-Length': '0'} 2023-05-31T20:48:11.032 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000007ce1c09b9ed40e6-006477b28b-1630-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:48:11 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:48:11.032 DEBUG:boto:b'pklxkd-59obj441000falseobj442023-05-31T20:39:26.565Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:48:11.033 DEBUG:boto:path=/pklxkd-59/obj44 2023-05-31T20:48:11.033 DEBUG:boto:auth_path=/pklxkd-59/obj44 2023-05-31T20:48:11.033 DEBUG:boto:Method: HEAD 2023-05-31T20:48:11.034 DEBUG:boto:Path: /pklxkd-59/obj44 2023-05-31T20:48:11.034 DEBUG:boto:Data: 2023-05-31T20:48:11.034 DEBUG:boto:Headers: {} 2023-05-31T20:48:11.035 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:48:11.035 DEBUG:boto:Port: 8001 2023-05-31T20:48:11.035 DEBUG:boto:Params: {} 2023-05-31T20:48:11.036 DEBUG:boto:Token: None 2023-05-31T20:48:11.036 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:48:11 GMT /pklxkd-59/obj44 2023-05-31T20:48:11.037 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:Cp99QiCFnDaTdfsaM0TXgOfinOs= 2023-05-31T20:48:11.037 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:48:11 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:Cp99QiCFnDaTdfsaM0TXgOfinOs=', 'Content-Length': '0'} 2023-05-31T20:48:11.038 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:39:26 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000cdb1c4536cdcde1d-006477b28b-1c17-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:48:11 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:48:11.039 DEBUG:boto:path=/pklxkd-59/obj44 2023-05-31T20:48:11.039 DEBUG:boto:auth_path=/pklxkd-59/obj44 2023-05-31T20:48:11.040 DEBUG:boto:Method: HEAD 2023-05-31T20:48:11.040 DEBUG:boto:Path: /pklxkd-59/obj44 2023-05-31T20:48:11.040 DEBUG:boto:Data: 2023-05-31T20:48:11.041 DEBUG:boto:Headers: {} 2023-05-31T20:48:11.041 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:48:11.041 DEBUG:boto:Port: 8000 2023-05-31T20:48:11.042 DEBUG:boto:Params: {} 2023-05-31T20:48:11.042 DEBUG:boto:Token: None 2023-05-31T20:48:11.042 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:48:11 GMT /pklxkd-59/obj44 2023-05-31T20:48:11.043 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:Cp99QiCFnDaTdfsaM0TXgOfinOs= 2023-05-31T20:48:11.043 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:48:11 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:Cp99QiCFnDaTdfsaM0TXgOfinOs=', 'Content-Length': '0'} 2023-05-31T20:48:11.045 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:39:26 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f3015f4df22e41f1-006477b28b-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:48:11 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:48:11.046 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj44 2023-05-31T20:48:11.046 DEBUG:boto:path=/pklxkd-59/obj44 2023-05-31T20:48:11.046 DEBUG:boto:auth_path=/pklxkd-59/obj44 2023-05-31T20:48:11.047 DEBUG:boto:Method: GET 2023-05-31T20:48:11.047 DEBUG:boto:Path: /pklxkd-59/obj44 2023-05-31T20:48:11.047 DEBUG:boto:Data: 2023-05-31T20:48:11.048 DEBUG:boto:Headers: {} 2023-05-31T20:48:11.048 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:48:11.048 DEBUG:boto:Port: 8001 2023-05-31T20:48:11.049 DEBUG:boto:Params: {} 2023-05-31T20:48:11.049 DEBUG:boto:Token: None 2023-05-31T20:48:11.050 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:48:11 GMT /pklxkd-59/obj44 2023-05-31T20:48:11.050 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:R37PBJG67mUqscwqP9kJWcquUnU= 2023-05-31T20:48:11.050 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:48:11 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:R37PBJG67mUqscwqP9kJWcquUnU=', 'Content-Length': '0'} 2023-05-31T20:48:11.052 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:39:26 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000843262eeff33393a-006477b28b-1c17-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:48:11 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:48:11.053 DEBUG:boto:path=/pklxkd-59/obj44 2023-05-31T20:48:11.053 DEBUG:boto:auth_path=/pklxkd-59/obj44 2023-05-31T20:48:11.053 DEBUG:boto:Method: GET 2023-05-31T20:48:11.054 DEBUG:boto:Path: /pklxkd-59/obj44 2023-05-31T20:48:11.054 DEBUG:boto:Data: 2023-05-31T20:48:11.055 DEBUG:boto:Headers: {} 2023-05-31T20:48:11.055 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:48:11.055 DEBUG:boto:Port: 8000 2023-05-31T20:48:11.056 DEBUG:boto:Params: {} 2023-05-31T20:48:11.056 DEBUG:boto:Token: None 2023-05-31T20:48:11.056 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:48:11 GMT /pklxkd-59/obj44 2023-05-31T20:48:11.057 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:R37PBJG67mUqscwqP9kJWcquUnU= 2023-05-31T20:48:11.057 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:48:11 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:R37PBJG67mUqscwqP9kJWcquUnU=', 'Content-Length': '0'} 2023-05-31T20:48:11.059 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:39:26 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000475beb17a9e04b15-006477b28b-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:48:11 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:48:11.059 DEBUG:boto:path=/pklxkd-59/ 2023-05-31T20:48:11.060 DEBUG:boto:auth_path=/pklxkd-59/ 2023-05-31T20:48:11.060 DEBUG:boto:path=/pklxkd-59/?prefix=obj44 2023-05-31T20:48:11.061 DEBUG:boto:auth_path=/pklxkd-59/?prefix=obj44 2023-05-31T20:48:11.061 DEBUG:boto:Method: GET 2023-05-31T20:48:11.061 DEBUG:boto:Path: /pklxkd-59/?prefix=obj44 2023-05-31T20:48:11.062 DEBUG:boto:Data: 2023-05-31T20:48:11.062 DEBUG:boto:Headers: {} 2023-05-31T20:48:11.062 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:48:11.063 DEBUG:boto:Port: 8001 2023-05-31T20:48:11.063 DEBUG:boto:Params: {} 2023-05-31T20:48:11.063 DEBUG:boto:Token: None 2023-05-31T20:48:11.064 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:48:11 GMT /pklxkd-59/ 2023-05-31T20:48:11.064 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:l166NKUsN4SBNL48MAZina5NQhQ= 2023-05-31T20:48:11.065 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:48:11 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:l166NKUsN4SBNL48MAZina5NQhQ=', 'Content-Length': '0'} 2023-05-31T20:48:11.067 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000001326d0bb66089dfe-006477b28b-1c17-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:48:11 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:48:11.067 DEBUG:boto:b'pklxkd-59obj441000falseobj442023-05-31T20:39:26.565Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:48:11.068 DEBUG:boto:path=/pklxkd-59/ 2023-05-31T20:48:11.068 DEBUG:boto:auth_path=/pklxkd-59/ 2023-05-31T20:48:11.069 DEBUG:boto:path=/pklxkd-59/?prefix=obj44 2023-05-31T20:48:11.069 DEBUG:boto:auth_path=/pklxkd-59/?prefix=obj44 2023-05-31T20:48:11.069 DEBUG:boto:Method: GET 2023-05-31T20:48:11.070 DEBUG:boto:Path: /pklxkd-59/?prefix=obj44 2023-05-31T20:48:11.070 DEBUG:boto:Data: 2023-05-31T20:48:11.070 DEBUG:boto:Headers: {} 2023-05-31T20:48:11.071 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:48:11.071 DEBUG:boto:Port: 8000 2023-05-31T20:48:11.071 DEBUG:boto:Params: {} 2023-05-31T20:48:11.072 DEBUG:boto:Token: None 2023-05-31T20:48:11.072 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:48:11 GMT /pklxkd-59/ 2023-05-31T20:48:11.073 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:l166NKUsN4SBNL48MAZina5NQhQ= 2023-05-31T20:48:11.073 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:48:11 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:l166NKUsN4SBNL48MAZina5NQhQ=', 'Content-Length': '0'} 2023-05-31T20:48:11.076 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000008a8c6a02e6d88786-006477b28b-1630-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:48:11 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:48:11.076 DEBUG:boto:b'pklxkd-59obj441000falseobj442023-05-31T20:39:26.565Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:48:11.077 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj46 2023-05-31T20:48:11.077 DEBUG:boto:path=/pklxkd-59/obj46 2023-05-31T20:48:11.078 DEBUG:boto:auth_path=/pklxkd-59/obj46 2023-05-31T20:48:11.078 DEBUG:boto:path=/pklxkd-59/obj46?versionId=null 2023-05-31T20:48:11.078 DEBUG:boto:auth_path=/pklxkd-59/obj46?versionId=null 2023-05-31T20:48:11.079 DEBUG:boto:Method: GET 2023-05-31T20:48:11.079 DEBUG:boto:Path: /pklxkd-59/obj46?versionId=null 2023-05-31T20:48:11.079 DEBUG:boto:Data: 2023-05-31T20:48:11.080 DEBUG:boto:Headers: {} 2023-05-31T20:48:11.080 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:48:11.081 DEBUG:boto:Port: 8001 2023-05-31T20:48:11.081 DEBUG:boto:Params: {} 2023-05-31T20:48:11.081 DEBUG:boto:Token: None 2023-05-31T20:48:11.082 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:48:11 GMT /pklxkd-59/obj46?versionId=null 2023-05-31T20:48:11.082 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:OthuDSnYNmqHtdBlXxYe/JVh3s8= 2023-05-31T20:48:11.083 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:48:11 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:OthuDSnYNmqHtdBlXxYe/JVh3s8=', 'Content-Length': '0'} 2023-05-31T20:48:11.084 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:46:33 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007f7fe1a04b6e24eb-006477b28b-1c17-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:48:11 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:48:11.085 DEBUG:boto:path=/pklxkd-59/obj46 2023-05-31T20:48:11.085 DEBUG:boto:auth_path=/pklxkd-59/obj46 2023-05-31T20:48:11.086 DEBUG:boto:path=/pklxkd-59/obj46?versionId=null 2023-05-31T20:48:11.086 DEBUG:boto:auth_path=/pklxkd-59/obj46?versionId=null 2023-05-31T20:48:11.086 DEBUG:boto:Method: GET 2023-05-31T20:48:11.087 DEBUG:boto:Path: /pklxkd-59/obj46?versionId=null 2023-05-31T20:48:11.087 DEBUG:boto:Data: 2023-05-31T20:48:11.088 DEBUG:boto:Headers: {} 2023-05-31T20:48:11.088 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:48:11.088 DEBUG:boto:Port: 8000 2023-05-31T20:48:11.089 DEBUG:boto:Params: {} 2023-05-31T20:48:11.089 DEBUG:boto:Token: None 2023-05-31T20:48:11.089 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:48:11 GMT /pklxkd-59/obj46?versionId=null 2023-05-31T20:48:11.090 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:OthuDSnYNmqHtdBlXxYe/JVh3s8= 2023-05-31T20:48:11.090 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:48:11 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:OthuDSnYNmqHtdBlXxYe/JVh3s8=', 'Content-Length': '0'} 2023-05-31T20:48:11.092 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:46:33 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000046200a062a654642-006477b28b-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:48:11 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:48:11.092 DEBUG:boto:path=/pklxkd-59/obj46 2023-05-31T20:48:11.093 DEBUG:boto:auth_path=/pklxkd-59/obj46 2023-05-31T20:48:11.093 DEBUG:boto:path=/pklxkd-59/obj46?versionId=null 2023-05-31T20:48:11.094 DEBUG:boto:auth_path=/pklxkd-59/obj46?versionId=null 2023-05-31T20:48:11.094 DEBUG:boto:Method: HEAD 2023-05-31T20:48:11.094 DEBUG:boto:Path: /pklxkd-59/obj46?versionId=null 2023-05-31T20:48:11.095 DEBUG:boto:Data: 2023-05-31T20:48:11.095 DEBUG:boto:Headers: {} 2023-05-31T20:48:11.095 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:48:11.096 DEBUG:boto:Port: 8001 2023-05-31T20:48:11.096 DEBUG:boto:Params: {} 2023-05-31T20:48:11.096 DEBUG:boto:Token: None 2023-05-31T20:48:11.097 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:48:11 GMT /pklxkd-59/obj46?versionId=null 2023-05-31T20:48:11.097 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:oeFqWapUZhZC78XFi5Zz/q+VxNs= 2023-05-31T20:48:11.097 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:48:11 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:oeFqWapUZhZC78XFi5Zz/q+VxNs=', 'Content-Length': '0'} 2023-05-31T20:48:11.099 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:46:33 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004cde06b4d59de4c3-006477b28b-1c17-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:48:11 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:48:11.099 DEBUG:boto:path=/pklxkd-59/obj46 2023-05-31T20:48:11.100 DEBUG:boto:auth_path=/pklxkd-59/obj46 2023-05-31T20:48:11.100 DEBUG:boto:path=/pklxkd-59/obj46?versionId=null 2023-05-31T20:48:11.101 DEBUG:boto:auth_path=/pklxkd-59/obj46?versionId=null 2023-05-31T20:48:11.101 DEBUG:boto:Method: HEAD 2023-05-31T20:48:11.101 DEBUG:boto:Path: /pklxkd-59/obj46?versionId=null 2023-05-31T20:48:11.102 DEBUG:boto:Data: 2023-05-31T20:48:11.102 DEBUG:boto:Headers: {} 2023-05-31T20:48:11.102 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:48:11.103 DEBUG:boto:Port: 8000 2023-05-31T20:48:11.103 DEBUG:boto:Params: {} 2023-05-31T20:48:11.103 DEBUG:boto:Token: None 2023-05-31T20:48:11.104 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:48:11 GMT /pklxkd-59/obj46?versionId=null 2023-05-31T20:48:11.104 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:oeFqWapUZhZC78XFi5Zz/q+VxNs= 2023-05-31T20:48:11.104 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:48:11 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:oeFqWapUZhZC78XFi5Zz/q+VxNs=', 'Content-Length': '0'} 2023-05-31T20:48:11.106 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:46:33 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000012f4d82845638a4d-006477b28b-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:48:11 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:48:11.107 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj46 2023-05-31T20:48:11.107 DEBUG:boto:path=/pklxkd-59/obj46 2023-05-31T20:48:11.107 DEBUG:boto:auth_path=/pklxkd-59/obj46 2023-05-31T20:48:11.108 DEBUG:boto:path=/pklxkd-59/obj46?versionId=null 2023-05-31T20:48:11.108 DEBUG:boto:auth_path=/pklxkd-59/obj46?versionId=null 2023-05-31T20:48:11.109 DEBUG:boto:Method: GET 2023-05-31T20:48:11.109 DEBUG:boto:Path: /pklxkd-59/obj46?versionId=null 2023-05-31T20:48:11.109 DEBUG:boto:Data: 2023-05-31T20:48:11.110 DEBUG:boto:Headers: {} 2023-05-31T20:48:11.110 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:48:11.110 DEBUG:boto:Port: 8001 2023-05-31T20:48:11.111 DEBUG:boto:Params: {} 2023-05-31T20:48:11.111 DEBUG:boto:Token: None 2023-05-31T20:48:11.111 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:48:11 GMT /pklxkd-59/obj46?versionId=null 2023-05-31T20:48:11.112 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:OthuDSnYNmqHtdBlXxYe/JVh3s8= 2023-05-31T20:48:11.112 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:48:11 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:OthuDSnYNmqHtdBlXxYe/JVh3s8=', 'Content-Length': '0'} 2023-05-31T20:48:11.114 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:46:33 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000932c340ca96bd39d-006477b28b-1c17-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:48:11 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:48:11.156 DEBUG:boto:path=/pklxkd-59/obj46 2023-05-31T20:48:11.157 DEBUG:boto:auth_path=/pklxkd-59/obj46 2023-05-31T20:48:11.157 DEBUG:boto:path=/pklxkd-59/obj46?versionId=null 2023-05-31T20:48:11.157 DEBUG:boto:auth_path=/pklxkd-59/obj46?versionId=null 2023-05-31T20:48:11.158 DEBUG:boto:Method: GET 2023-05-31T20:48:11.158 DEBUG:boto:Path: /pklxkd-59/obj46?versionId=null 2023-05-31T20:48:11.158 DEBUG:boto:Data: 2023-05-31T20:48:11.159 DEBUG:boto:Headers: {} 2023-05-31T20:48:11.159 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:48:11.159 DEBUG:boto:Port: 8000 2023-05-31T20:48:11.160 DEBUG:boto:Params: {} 2023-05-31T20:48:11.160 DEBUG:boto:Token: None 2023-05-31T20:48:11.161 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:48:11 GMT /pklxkd-59/obj46?versionId=null 2023-05-31T20:48:11.161 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:OthuDSnYNmqHtdBlXxYe/JVh3s8= 2023-05-31T20:48:11.161 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:48:11 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:OthuDSnYNmqHtdBlXxYe/JVh3s8=', 'Content-Length': '0'} 2023-05-31T20:48:11.163 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:46:33 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000bd80fd6248f5758b-006477b28b-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:48:11 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:48:11.164 DEBUG:boto:path=/pklxkd-59/ 2023-05-31T20:48:11.164 DEBUG:boto:auth_path=/pklxkd-59/ 2023-05-31T20:48:11.165 DEBUG:boto:path=/pklxkd-59/?prefix=obj46 2023-05-31T20:48:11.165 DEBUG:boto:auth_path=/pklxkd-59/?prefix=obj46 2023-05-31T20:48:11.166 DEBUG:boto:Method: GET 2023-05-31T20:48:11.166 DEBUG:boto:Path: /pklxkd-59/?prefix=obj46 2023-05-31T20:48:11.166 DEBUG:boto:Data: 2023-05-31T20:48:11.167 DEBUG:boto:Headers: {} 2023-05-31T20:48:11.167 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:48:11.167 DEBUG:boto:Port: 8001 2023-05-31T20:48:11.168 DEBUG:boto:Params: {} 2023-05-31T20:48:11.168 DEBUG:boto:Token: None 2023-05-31T20:48:11.168 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:48:11 GMT /pklxkd-59/ 2023-05-31T20:48:11.169 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:l166NKUsN4SBNL48MAZina5NQhQ= 2023-05-31T20:48:11.169 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:48:11 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:l166NKUsN4SBNL48MAZina5NQhQ=', 'Content-Length': '0'} 2023-05-31T20:48:11.171 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000000cc32c251002006f-006477b28b-1c17-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:48:11 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:48:11.172 DEBUG:boto:b'pklxkd-59obj461000falseobj462023-05-31T20:46:33.830Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:48:11.173 DEBUG:boto:path=/pklxkd-59/ 2023-05-31T20:48:11.173 DEBUG:boto:auth_path=/pklxkd-59/ 2023-05-31T20:48:11.173 DEBUG:boto:path=/pklxkd-59/?prefix=obj46 2023-05-31T20:48:11.174 DEBUG:boto:auth_path=/pklxkd-59/?prefix=obj46 2023-05-31T20:48:11.174 DEBUG:boto:Method: GET 2023-05-31T20:48:11.174 DEBUG:boto:Path: /pklxkd-59/?prefix=obj46 2023-05-31T20:48:11.175 DEBUG:boto:Data: 2023-05-31T20:48:11.175 DEBUG:boto:Headers: {} 2023-05-31T20:48:11.175 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:48:11.176 DEBUG:boto:Port: 8000 2023-05-31T20:48:11.176 DEBUG:boto:Params: {} 2023-05-31T20:48:11.177 DEBUG:boto:Token: None 2023-05-31T20:48:11.177 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:48:11 GMT /pklxkd-59/ 2023-05-31T20:48:11.177 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:l166NKUsN4SBNL48MAZina5NQhQ= 2023-05-31T20:48:11.178 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:48:11 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:l166NKUsN4SBNL48MAZina5NQhQ=', 'Content-Length': '0'} 2023-05-31T20:48:11.181 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000008d52ee5649f1435c-006477b28b-1630-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:48:11 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:48:11.181 DEBUG:boto:b'pklxkd-59obj461000falseobj462023-05-31T20:46:33.830Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:48:11.182 DEBUG:boto:path=/pklxkd-59/obj46 2023-05-31T20:48:11.182 DEBUG:boto:auth_path=/pklxkd-59/obj46 2023-05-31T20:48:11.182 DEBUG:boto:Method: HEAD 2023-05-31T20:48:11.183 DEBUG:boto:Path: /pklxkd-59/obj46 2023-05-31T20:48:11.183 DEBUG:boto:Data: 2023-05-31T20:48:11.183 DEBUG:boto:Headers: {} 2023-05-31T20:48:11.184 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:48:11.184 DEBUG:boto:Port: 8001 2023-05-31T20:48:11.184 DEBUG:boto:Params: {} 2023-05-31T20:48:11.185 DEBUG:boto:Token: None 2023-05-31T20:48:11.185 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:48:11 GMT /pklxkd-59/obj46 2023-05-31T20:48:11.186 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:QrlRrT1JH20K/qDWv4bC/WMUabs= 2023-05-31T20:48:11.186 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:48:11 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:QrlRrT1JH20K/qDWv4bC/WMUabs=', 'Content-Length': '0'} 2023-05-31T20:48:11.188 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:46:33 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006650b7c3615f988e-006477b28b-1c17-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:48:11 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:48:11.188 DEBUG:boto:path=/pklxkd-59/obj46 2023-05-31T20:48:11.189 DEBUG:boto:auth_path=/pklxkd-59/obj46 2023-05-31T20:48:11.189 DEBUG:boto:Method: HEAD 2023-05-31T20:48:11.189 DEBUG:boto:Path: /pklxkd-59/obj46 2023-05-31T20:48:11.190 DEBUG:boto:Data: 2023-05-31T20:48:11.190 DEBUG:boto:Headers: {} 2023-05-31T20:48:11.190 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:48:11.191 DEBUG:boto:Port: 8000 2023-05-31T20:48:11.191 DEBUG:boto:Params: {} 2023-05-31T20:48:11.191 DEBUG:boto:Token: None 2023-05-31T20:48:11.192 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:48:11 GMT /pklxkd-59/obj46 2023-05-31T20:48:11.192 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:QrlRrT1JH20K/qDWv4bC/WMUabs= 2023-05-31T20:48:11.193 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:48:11 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:QrlRrT1JH20K/qDWv4bC/WMUabs=', 'Content-Length': '0'} 2023-05-31T20:48:11.194 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:46:33 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f415364be8ee6278-006477b28b-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:48:11 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:48:11.195 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj46 2023-05-31T20:48:11.195 DEBUG:boto:path=/pklxkd-59/obj46 2023-05-31T20:48:11.196 DEBUG:boto:auth_path=/pklxkd-59/obj46 2023-05-31T20:48:11.196 DEBUG:boto:Method: GET 2023-05-31T20:48:11.196 DEBUG:boto:Path: /pklxkd-59/obj46 2023-05-31T20:48:11.197 DEBUG:boto:Data: 2023-05-31T20:48:11.197 DEBUG:boto:Headers: {} 2023-05-31T20:48:11.197 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:48:11.198 DEBUG:boto:Port: 8001 2023-05-31T20:48:11.198 DEBUG:boto:Params: {} 2023-05-31T20:48:11.198 DEBUG:boto:Token: None 2023-05-31T20:48:11.199 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:48:11 GMT /pklxkd-59/obj46 2023-05-31T20:48:11.199 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:ljzmmzqcm3x6uc7o2pDIhsiZbqU= 2023-05-31T20:48:11.200 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:48:11 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:ljzmmzqcm3x6uc7o2pDIhsiZbqU=', 'Content-Length': '0'} 2023-05-31T20:48:11.202 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:46:33 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000365d0512507ca9e8-006477b28b-1c17-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:48:11 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:48:11.202 DEBUG:boto:path=/pklxkd-59/obj46 2023-05-31T20:48:11.203 DEBUG:boto:auth_path=/pklxkd-59/obj46 2023-05-31T20:48:11.203 DEBUG:boto:Method: GET 2023-05-31T20:48:11.203 DEBUG:boto:Path: /pklxkd-59/obj46 2023-05-31T20:48:11.204 DEBUG:boto:Data: 2023-05-31T20:48:11.204 DEBUG:boto:Headers: {} 2023-05-31T20:48:11.204 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:48:11.205 DEBUG:boto:Port: 8000 2023-05-31T20:48:11.205 DEBUG:boto:Params: {} 2023-05-31T20:48:11.206 DEBUG:boto:Token: None 2023-05-31T20:48:11.206 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:48:11 GMT /pklxkd-59/obj46 2023-05-31T20:48:11.206 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:ljzmmzqcm3x6uc7o2pDIhsiZbqU= 2023-05-31T20:48:11.207 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:48:11 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:ljzmmzqcm3x6uc7o2pDIhsiZbqU=', 'Content-Length': '0'} 2023-05-31T20:48:11.208 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:46:33 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000dbda716ed5340916-006477b28b-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:48:11 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:48:11.252 DEBUG:boto:path=/pklxkd-59/ 2023-05-31T20:48:11.253 DEBUG:boto:auth_path=/pklxkd-59/ 2023-05-31T20:48:11.253 DEBUG:boto:path=/pklxkd-59/?prefix=obj46 2023-05-31T20:48:11.254 DEBUG:boto:auth_path=/pklxkd-59/?prefix=obj46 2023-05-31T20:48:11.254 DEBUG:boto:Method: GET 2023-05-31T20:48:11.254 DEBUG:boto:Path: /pklxkd-59/?prefix=obj46 2023-05-31T20:48:11.255 DEBUG:boto:Data: 2023-05-31T20:48:11.255 DEBUG:boto:Headers: {} 2023-05-31T20:48:11.255 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:48:11.256 DEBUG:boto:Port: 8001 2023-05-31T20:48:11.256 DEBUG:boto:Params: {} 2023-05-31T20:48:11.257 DEBUG:boto:Token: None 2023-05-31T20:48:11.257 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:48:11 GMT /pklxkd-59/ 2023-05-31T20:48:11.257 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:l166NKUsN4SBNL48MAZina5NQhQ= 2023-05-31T20:48:11.258 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:48:11 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:l166NKUsN4SBNL48MAZina5NQhQ=', 'Content-Length': '0'} 2023-05-31T20:48:11.260 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000003ee2c392b7a1854b-006477b28b-1c17-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:48:11 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:48:11.261 DEBUG:boto:b'pklxkd-59obj461000falseobj462023-05-31T20:46:33.830Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:48:11.261 DEBUG:boto:path=/pklxkd-59/ 2023-05-31T20:48:11.262 DEBUG:boto:auth_path=/pklxkd-59/ 2023-05-31T20:48:11.262 DEBUG:boto:path=/pklxkd-59/?prefix=obj46 2023-05-31T20:48:11.263 DEBUG:boto:auth_path=/pklxkd-59/?prefix=obj46 2023-05-31T20:48:11.263 DEBUG:boto:Method: GET 2023-05-31T20:48:11.263 DEBUG:boto:Path: /pklxkd-59/?prefix=obj46 2023-05-31T20:48:11.264 DEBUG:boto:Data: 2023-05-31T20:48:11.264 DEBUG:boto:Headers: {} 2023-05-31T20:48:11.264 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:48:11.265 DEBUG:boto:Port: 8000 2023-05-31T20:48:11.265 DEBUG:boto:Params: {} 2023-05-31T20:48:11.266 DEBUG:boto:Token: None 2023-05-31T20:48:11.266 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:48:11 GMT /pklxkd-59/ 2023-05-31T20:48:11.266 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:l166NKUsN4SBNL48MAZina5NQhQ= 2023-05-31T20:48:11.267 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:48:11 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:l166NKUsN4SBNL48MAZina5NQhQ=', 'Content-Length': '0'} 2023-05-31T20:48:11.270 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b45c333dd0e2501d-006477b28b-1630-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:48:11 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:48:11.271 DEBUG:boto:b'pklxkd-59obj461000falseobj462023-05-31T20:46:33.830Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:48:11.271 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj51 2023-05-31T20:48:11.272 DEBUG:boto:path=/pklxkd-59/obj51 2023-05-31T20:48:11.272 DEBUG:boto:auth_path=/pklxkd-59/obj51 2023-05-31T20:48:11.273 DEBUG:boto:path=/pklxkd-59/obj51?versionId=null 2023-05-31T20:48:11.273 DEBUG:boto:auth_path=/pklxkd-59/obj51?versionId=null 2023-05-31T20:48:11.273 DEBUG:boto:Method: GET 2023-05-31T20:48:11.274 DEBUG:boto:Path: /pklxkd-59/obj51?versionId=null 2023-05-31T20:48:11.274 DEBUG:boto:Data: 2023-05-31T20:48:11.274 DEBUG:boto:Headers: {} 2023-05-31T20:48:11.275 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:48:11.275 DEBUG:boto:Port: 8001 2023-05-31T20:48:11.275 DEBUG:boto:Params: {} 2023-05-31T20:48:11.276 DEBUG:boto:Token: None 2023-05-31T20:48:11.276 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:48:11 GMT /pklxkd-59/obj51?versionId=null 2023-05-31T20:48:11.277 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:DrWLnYj2WClGk4nt/Ob6Bnfmpfc= 2023-05-31T20:48:11.277 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:48:11 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:DrWLnYj2WClGk4nt/Ob6Bnfmpfc=', 'Content-Length': '0'} 2023-05-31T20:48:11.279 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:43:08 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000135c192f3982ebbd-006477b28b-1c17-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:48:11 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:48:11.280 DEBUG:boto:path=/pklxkd-59/obj51 2023-05-31T20:48:11.280 DEBUG:boto:auth_path=/pklxkd-59/obj51 2023-05-31T20:48:11.281 DEBUG:boto:path=/pklxkd-59/obj51?versionId=null 2023-05-31T20:48:11.281 DEBUG:boto:auth_path=/pklxkd-59/obj51?versionId=null 2023-05-31T20:48:11.281 DEBUG:boto:Method: GET 2023-05-31T20:48:11.282 DEBUG:boto:Path: /pklxkd-59/obj51?versionId=null 2023-05-31T20:48:11.282 DEBUG:boto:Data: 2023-05-31T20:48:11.282 DEBUG:boto:Headers: {} 2023-05-31T20:48:11.283 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:48:11.283 DEBUG:boto:Port: 8000 2023-05-31T20:48:11.283 DEBUG:boto:Params: {} 2023-05-31T20:48:11.284 DEBUG:boto:Token: None 2023-05-31T20:48:11.284 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:48:11 GMT /pklxkd-59/obj51?versionId=null 2023-05-31T20:48:11.285 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:DrWLnYj2WClGk4nt/Ob6Bnfmpfc= 2023-05-31T20:48:11.285 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:48:11 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:DrWLnYj2WClGk4nt/Ob6Bnfmpfc=', 'Content-Length': '0'} 2023-05-31T20:48:11.287 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:43:08 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000003e2a1c3984a12a5-006477b28b-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:48:11 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:48:11.288 DEBUG:boto:path=/pklxkd-59/obj51 2023-05-31T20:48:11.288 DEBUG:boto:auth_path=/pklxkd-59/obj51 2023-05-31T20:48:11.289 DEBUG:boto:path=/pklxkd-59/obj51?versionId=null 2023-05-31T20:48:11.289 DEBUG:boto:auth_path=/pklxkd-59/obj51?versionId=null 2023-05-31T20:48:11.289 DEBUG:boto:Method: HEAD 2023-05-31T20:48:11.290 DEBUG:boto:Path: /pklxkd-59/obj51?versionId=null 2023-05-31T20:48:11.290 DEBUG:boto:Data: 2023-05-31T20:48:11.290 DEBUG:boto:Headers: {} 2023-05-31T20:48:11.291 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:48:11.291 DEBUG:boto:Port: 8001 2023-05-31T20:48:11.291 DEBUG:boto:Params: {} 2023-05-31T20:48:11.292 DEBUG:boto:Token: None 2023-05-31T20:48:11.292 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:48:11 GMT /pklxkd-59/obj51?versionId=null 2023-05-31T20:48:11.292 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:EtkJj2ZHjV1BARfwVwDJWinjRIk= 2023-05-31T20:48:11.293 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:48:11 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:EtkJj2ZHjV1BARfwVwDJWinjRIk=', 'Content-Length': '0'} 2023-05-31T20:48:11.295 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:43:08 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000037c7f24ff0cbb7a3-006477b28b-1c17-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:48:11 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:48:11.295 DEBUG:boto:path=/pklxkd-59/obj51 2023-05-31T20:48:11.296 DEBUG:boto:auth_path=/pklxkd-59/obj51 2023-05-31T20:48:11.296 DEBUG:boto:path=/pklxkd-59/obj51?versionId=null 2023-05-31T20:48:11.296 DEBUG:boto:auth_path=/pklxkd-59/obj51?versionId=null 2023-05-31T20:48:11.297 DEBUG:boto:Method: HEAD 2023-05-31T20:48:11.297 DEBUG:boto:Path: /pklxkd-59/obj51?versionId=null 2023-05-31T20:48:11.297 DEBUG:boto:Data: 2023-05-31T20:48:11.298 DEBUG:boto:Headers: {} 2023-05-31T20:48:11.298 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:48:11.298 DEBUG:boto:Port: 8000 2023-05-31T20:48:11.299 DEBUG:boto:Params: {} 2023-05-31T20:48:11.299 DEBUG:boto:Token: None 2023-05-31T20:48:11.300 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:48:11 GMT /pklxkd-59/obj51?versionId=null 2023-05-31T20:48:11.300 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:EtkJj2ZHjV1BARfwVwDJWinjRIk= 2023-05-31T20:48:11.300 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:48:11 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:EtkJj2ZHjV1BARfwVwDJWinjRIk=', 'Content-Length': '0'} 2023-05-31T20:48:11.302 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:43:08 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000444f1ba03a0228e9-006477b28b-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:48:11 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:48:11.303 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj51 2023-05-31T20:48:11.303 DEBUG:boto:path=/pklxkd-59/obj51 2023-05-31T20:48:11.303 DEBUG:boto:auth_path=/pklxkd-59/obj51 2023-05-31T20:48:11.304 DEBUG:boto:path=/pklxkd-59/obj51?versionId=null 2023-05-31T20:48:11.304 DEBUG:boto:auth_path=/pklxkd-59/obj51?versionId=null 2023-05-31T20:48:11.305 DEBUG:boto:Method: GET 2023-05-31T20:48:11.305 DEBUG:boto:Path: /pklxkd-59/obj51?versionId=null 2023-05-31T20:48:11.305 DEBUG:boto:Data: 2023-05-31T20:48:11.306 DEBUG:boto:Headers: {} 2023-05-31T20:48:11.306 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:48:11.306 DEBUG:boto:Port: 8001 2023-05-31T20:48:11.307 DEBUG:boto:Params: {} 2023-05-31T20:48:11.307 DEBUG:boto:Token: None 2023-05-31T20:48:11.307 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:48:11 GMT /pklxkd-59/obj51?versionId=null 2023-05-31T20:48:11.308 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:DrWLnYj2WClGk4nt/Ob6Bnfmpfc= 2023-05-31T20:48:11.308 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:48:11 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:DrWLnYj2WClGk4nt/Ob6Bnfmpfc=', 'Content-Length': '0'} 2023-05-31T20:48:11.310 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:43:08 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003e451d895e18a96a-006477b28b-1c17-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:48:11 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:48:11.311 DEBUG:boto:path=/pklxkd-59/obj51 2023-05-31T20:48:11.311 DEBUG:boto:auth_path=/pklxkd-59/obj51 2023-05-31T20:48:11.312 DEBUG:boto:path=/pklxkd-59/obj51?versionId=null 2023-05-31T20:48:11.312 DEBUG:boto:auth_path=/pklxkd-59/obj51?versionId=null 2023-05-31T20:48:11.312 DEBUG:boto:Method: GET 2023-05-31T20:48:11.313 DEBUG:boto:Path: /pklxkd-59/obj51?versionId=null 2023-05-31T20:48:11.313 DEBUG:boto:Data: 2023-05-31T20:48:11.314 DEBUG:boto:Headers: {} 2023-05-31T20:48:11.314 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:48:11.314 DEBUG:boto:Port: 8000 2023-05-31T20:48:11.315 DEBUG:boto:Params: {} 2023-05-31T20:48:11.315 DEBUG:boto:Token: None 2023-05-31T20:48:11.315 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:48:11 GMT /pklxkd-59/obj51?versionId=null 2023-05-31T20:48:11.316 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:DrWLnYj2WClGk4nt/Ob6Bnfmpfc= 2023-05-31T20:48:11.316 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:48:11 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:DrWLnYj2WClGk4nt/Ob6Bnfmpfc=', 'Content-Length': '0'} 2023-05-31T20:48:11.318 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:43:08 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000017072208ce7acadf-006477b28b-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:48:11 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:48:11.319 DEBUG:boto:path=/pklxkd-59/ 2023-05-31T20:48:11.319 DEBUG:boto:auth_path=/pklxkd-59/ 2023-05-31T20:48:11.319 DEBUG:boto:path=/pklxkd-59/?prefix=obj51 2023-05-31T20:48:11.320 DEBUG:boto:auth_path=/pklxkd-59/?prefix=obj51 2023-05-31T20:48:11.320 DEBUG:boto:Method: GET 2023-05-31T20:48:11.320 DEBUG:boto:Path: /pklxkd-59/?prefix=obj51 2023-05-31T20:48:11.321 DEBUG:boto:Data: 2023-05-31T20:48:11.321 DEBUG:boto:Headers: {} 2023-05-31T20:48:11.321 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:48:11.322 DEBUG:boto:Port: 8001 2023-05-31T20:48:11.322 DEBUG:boto:Params: {} 2023-05-31T20:48:11.323 DEBUG:boto:Token: None 2023-05-31T20:48:11.323 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:48:11 GMT /pklxkd-59/ 2023-05-31T20:48:11.323 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:l166NKUsN4SBNL48MAZina5NQhQ= 2023-05-31T20:48:11.324 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:48:11 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:l166NKUsN4SBNL48MAZina5NQhQ=', 'Content-Length': '0'} 2023-05-31T20:48:11.326 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000fd9f8c66f4fb210f-006477b28b-1c17-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:48:11 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:48:11.327 DEBUG:boto:b'pklxkd-59obj511000falseobj512023-05-31T20:43:08.011Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:48:11.327 DEBUG:boto:path=/pklxkd-59/ 2023-05-31T20:48:11.328 DEBUG:boto:auth_path=/pklxkd-59/ 2023-05-31T20:48:11.328 DEBUG:boto:path=/pklxkd-59/?prefix=obj51 2023-05-31T20:48:11.328 DEBUG:boto:auth_path=/pklxkd-59/?prefix=obj51 2023-05-31T20:48:11.329 DEBUG:boto:Method: GET 2023-05-31T20:48:11.329 DEBUG:boto:Path: /pklxkd-59/?prefix=obj51 2023-05-31T20:48:11.329 DEBUG:boto:Data: 2023-05-31T20:48:11.330 DEBUG:boto:Headers: {} 2023-05-31T20:48:11.330 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:48:11.331 DEBUG:boto:Port: 8000 2023-05-31T20:48:11.331 DEBUG:boto:Params: {} 2023-05-31T20:48:11.331 DEBUG:boto:Token: None 2023-05-31T20:48:11.332 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:48:11 GMT /pklxkd-59/ 2023-05-31T20:48:11.332 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:l166NKUsN4SBNL48MAZina5NQhQ= 2023-05-31T20:48:11.333 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:48:11 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:l166NKUsN4SBNL48MAZina5NQhQ=', 'Content-Length': '0'} 2023-05-31T20:48:11.336 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d7cc823907c43d51-006477b28b-1630-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:48:11 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:48:11.336 DEBUG:boto:b'pklxkd-59obj511000falseobj512023-05-31T20:43:08.011Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:48:11.337 DEBUG:boto:path=/pklxkd-59/obj51 2023-05-31T20:48:11.337 DEBUG:boto:auth_path=/pklxkd-59/obj51 2023-05-31T20:48:11.337 DEBUG:boto:Method: HEAD 2023-05-31T20:48:11.338 DEBUG:boto:Path: /pklxkd-59/obj51 2023-05-31T20:48:11.338 DEBUG:boto:Data: 2023-05-31T20:48:11.338 DEBUG:boto:Headers: {} 2023-05-31T20:48:11.339 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:48:11.339 DEBUG:boto:Port: 8001 2023-05-31T20:48:11.339 DEBUG:boto:Params: {} 2023-05-31T20:48:11.340 DEBUG:boto:Token: None 2023-05-31T20:48:11.340 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:48:11 GMT /pklxkd-59/obj51 2023-05-31T20:48:11.341 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:n8hAMQwVL22TI2tL7twDC529GJM= 2023-05-31T20:48:11.341 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:48:11 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:n8hAMQwVL22TI2tL7twDC529GJM=', 'Content-Length': '0'} 2023-05-31T20:48:11.343 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:43:08 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000fb2488f8caaad6b8-006477b28b-1c17-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:48:11 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:48:11.343 DEBUG:boto:path=/pklxkd-59/obj51 2023-05-31T20:48:11.343 DEBUG:boto:auth_path=/pklxkd-59/obj51 2023-05-31T20:48:11.344 DEBUG:boto:Method: HEAD 2023-05-31T20:48:11.344 DEBUG:boto:Path: /pklxkd-59/obj51 2023-05-31T20:48:11.345 DEBUG:boto:Data: 2023-05-31T20:48:11.345 DEBUG:boto:Headers: {} 2023-05-31T20:48:11.345 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:48:11.346 DEBUG:boto:Port: 8000 2023-05-31T20:48:11.346 DEBUG:boto:Params: {} 2023-05-31T20:48:11.346 DEBUG:boto:Token: None 2023-05-31T20:48:11.347 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:48:11 GMT /pklxkd-59/obj51 2023-05-31T20:48:11.347 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:n8hAMQwVL22TI2tL7twDC529GJM= 2023-05-31T20:48:11.347 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:48:11 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:n8hAMQwVL22TI2tL7twDC529GJM=', 'Content-Length': '0'} 2023-05-31T20:48:11.349 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:43:08 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001ea751d37b3d42ba-006477b28b-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:48:11 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:48:11.349 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj51 2023-05-31T20:48:11.350 DEBUG:boto:path=/pklxkd-59/obj51 2023-05-31T20:48:11.350 DEBUG:boto:auth_path=/pklxkd-59/obj51 2023-05-31T20:48:11.351 DEBUG:boto:Method: GET 2023-05-31T20:48:11.351 DEBUG:boto:Path: /pklxkd-59/obj51 2023-05-31T20:48:11.351 DEBUG:boto:Data: 2023-05-31T20:48:11.352 DEBUG:boto:Headers: {} 2023-05-31T20:48:11.352 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:48:11.352 DEBUG:boto:Port: 8001 2023-05-31T20:48:11.353 DEBUG:boto:Params: {} 2023-05-31T20:48:11.353 DEBUG:boto:Token: None 2023-05-31T20:48:11.354 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:48:11 GMT /pklxkd-59/obj51 2023-05-31T20:48:11.354 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:ze1u/Qki/oxZfylorxb6fF8RjhU= 2023-05-31T20:48:11.354 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:48:11 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:ze1u/Qki/oxZfylorxb6fF8RjhU=', 'Content-Length': '0'} 2023-05-31T20:48:11.356 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:43:08 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006b53571aabab737d-006477b28b-1c17-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:48:11 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:48:11.357 DEBUG:boto:path=/pklxkd-59/obj51 2023-05-31T20:48:11.357 DEBUG:boto:auth_path=/pklxkd-59/obj51 2023-05-31T20:48:11.357 DEBUG:boto:Method: GET 2023-05-31T20:48:11.358 DEBUG:boto:Path: /pklxkd-59/obj51 2023-05-31T20:48:11.358 DEBUG:boto:Data: 2023-05-31T20:48:11.359 DEBUG:boto:Headers: {} 2023-05-31T20:48:11.359 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:48:11.359 DEBUG:boto:Port: 8000 2023-05-31T20:48:11.360 DEBUG:boto:Params: {} 2023-05-31T20:48:11.360 DEBUG:boto:Token: None 2023-05-31T20:48:11.360 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:48:11 GMT /pklxkd-59/obj51 2023-05-31T20:48:11.361 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:ze1u/Qki/oxZfylorxb6fF8RjhU= 2023-05-31T20:48:11.361 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:48:11 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:ze1u/Qki/oxZfylorxb6fF8RjhU=', 'Content-Length': '0'} 2023-05-31T20:48:11.363 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:43:08 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000cfa972a13ca6eb08-006477b28b-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:48:11 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:48:11.404 DEBUG:boto:path=/pklxkd-59/ 2023-05-31T20:48:11.405 DEBUG:boto:auth_path=/pklxkd-59/ 2023-05-31T20:48:11.405 DEBUG:boto:path=/pklxkd-59/?prefix=obj51 2023-05-31T20:48:11.405 DEBUG:boto:auth_path=/pklxkd-59/?prefix=obj51 2023-05-31T20:48:11.406 DEBUG:boto:Method: GET 2023-05-31T20:48:11.406 DEBUG:boto:Path: /pklxkd-59/?prefix=obj51 2023-05-31T20:48:11.406 DEBUG:boto:Data: 2023-05-31T20:48:11.407 DEBUG:boto:Headers: {} 2023-05-31T20:48:11.407 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:48:11.407 DEBUG:boto:Port: 8001 2023-05-31T20:48:11.408 DEBUG:boto:Params: {} 2023-05-31T20:48:11.408 DEBUG:boto:Token: None 2023-05-31T20:48:11.409 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:48:11 GMT /pklxkd-59/ 2023-05-31T20:48:11.409 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:l166NKUsN4SBNL48MAZina5NQhQ= 2023-05-31T20:48:11.409 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:48:11 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:l166NKUsN4SBNL48MAZina5NQhQ=', 'Content-Length': '0'} 2023-05-31T20:48:11.412 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ce86b8baa37c39bd-006477b28b-1c17-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:48:11 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:48:11.412 DEBUG:boto:b'pklxkd-59obj511000falseobj512023-05-31T20:43:08.011Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:48:11.413 DEBUG:boto:path=/pklxkd-59/ 2023-05-31T20:48:11.413 DEBUG:boto:auth_path=/pklxkd-59/ 2023-05-31T20:48:11.414 DEBUG:boto:path=/pklxkd-59/?prefix=obj51 2023-05-31T20:48:11.414 DEBUG:boto:auth_path=/pklxkd-59/?prefix=obj51 2023-05-31T20:48:11.414 DEBUG:boto:Method: GET 2023-05-31T20:48:11.415 DEBUG:boto:Path: /pklxkd-59/?prefix=obj51 2023-05-31T20:48:11.415 DEBUG:boto:Data: 2023-05-31T20:48:11.415 DEBUG:boto:Headers: {} 2023-05-31T20:48:11.416 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:48:11.416 DEBUG:boto:Port: 8000 2023-05-31T20:48:11.416 DEBUG:boto:Params: {} 2023-05-31T20:48:11.417 DEBUG:boto:Token: None 2023-05-31T20:48:11.417 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:48:11 GMT /pklxkd-59/ 2023-05-31T20:48:11.418 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:l166NKUsN4SBNL48MAZina5NQhQ= 2023-05-31T20:48:11.418 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:48:11 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:l166NKUsN4SBNL48MAZina5NQhQ=', 'Content-Length': '0'} 2023-05-31T20:48:11.421 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000073cf3f6b875e169b-006477b28b-1630-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:48:11 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:48:11.421 DEBUG:boto:b'pklxkd-59obj511000falseobj512023-05-31T20:43:08.011Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:48:11.422 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj55 2023-05-31T20:48:11.422 DEBUG:boto:path=/pklxkd-59/obj55 2023-05-31T20:48:11.423 DEBUG:boto:auth_path=/pklxkd-59/obj55 2023-05-31T20:48:11.423 DEBUG:boto:path=/pklxkd-59/obj55?versionId=null 2023-05-31T20:48:11.423 DEBUG:boto:auth_path=/pklxkd-59/obj55?versionId=null 2023-05-31T20:48:11.424 DEBUG:boto:Method: GET 2023-05-31T20:48:11.424 DEBUG:boto:Path: /pklxkd-59/obj55?versionId=null 2023-05-31T20:48:11.424 DEBUG:boto:Data: 2023-05-31T20:48:11.425 DEBUG:boto:Headers: {} 2023-05-31T20:48:11.425 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:48:11.425 DEBUG:boto:Port: 8001 2023-05-31T20:48:11.426 DEBUG:boto:Params: {} 2023-05-31T20:48:11.426 DEBUG:boto:Token: None 2023-05-31T20:48:11.427 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:48:11 GMT /pklxkd-59/obj55?versionId=null 2023-05-31T20:48:11.427 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:ipuVu3SXqzG9mrZXpfHBdbKfPEY= 2023-05-31T20:48:11.427 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:48:11 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:ipuVu3SXqzG9mrZXpfHBdbKfPEY=', 'Content-Length': '0'} 2023-05-31T20:48:11.429 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:39:26 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000040831385b8e27f8-006477b28b-1c17-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:48:11 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:48:11.430 DEBUG:boto:path=/pklxkd-59/obj55 2023-05-31T20:48:11.430 DEBUG:boto:auth_path=/pklxkd-59/obj55 2023-05-31T20:48:11.430 DEBUG:boto:path=/pklxkd-59/obj55?versionId=null 2023-05-31T20:48:11.431 DEBUG:boto:auth_path=/pklxkd-59/obj55?versionId=null 2023-05-31T20:48:11.431 DEBUG:boto:Method: GET 2023-05-31T20:48:11.432 DEBUG:boto:Path: /pklxkd-59/obj55?versionId=null 2023-05-31T20:48:11.432 DEBUG:boto:Data: 2023-05-31T20:48:11.432 DEBUG:boto:Headers: {} 2023-05-31T20:48:11.433 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:48:11.433 DEBUG:boto:Port: 8000 2023-05-31T20:48:11.433 DEBUG:boto:Params: {} 2023-05-31T20:48:11.434 DEBUG:boto:Token: None 2023-05-31T20:48:11.434 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:48:11 GMT /pklxkd-59/obj55?versionId=null 2023-05-31T20:48:11.435 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:ipuVu3SXqzG9mrZXpfHBdbKfPEY= 2023-05-31T20:48:11.435 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:48:11 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:ipuVu3SXqzG9mrZXpfHBdbKfPEY=', 'Content-Length': '0'} 2023-05-31T20:48:11.437 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:39:26 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000692f723f2f2ef758-006477b28b-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:48:11 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:48:11.437 DEBUG:boto:path=/pklxkd-59/obj55 2023-05-31T20:48:11.438 DEBUG:boto:auth_path=/pklxkd-59/obj55 2023-05-31T20:48:11.438 DEBUG:boto:path=/pklxkd-59/obj55?versionId=null 2023-05-31T20:48:11.438 DEBUG:boto:auth_path=/pklxkd-59/obj55?versionId=null 2023-05-31T20:48:11.439 DEBUG:boto:Method: HEAD 2023-05-31T20:48:11.439 DEBUG:boto:Path: /pklxkd-59/obj55?versionId=null 2023-05-31T20:48:11.439 DEBUG:boto:Data: 2023-05-31T20:48:11.440 DEBUG:boto:Headers: {} 2023-05-31T20:48:11.440 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:48:11.440 DEBUG:boto:Port: 8001 2023-05-31T20:48:11.441 DEBUG:boto:Params: {} 2023-05-31T20:48:11.441 DEBUG:boto:Token: None 2023-05-31T20:48:11.442 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:48:11 GMT /pklxkd-59/obj55?versionId=null 2023-05-31T20:48:11.442 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:LALudZ2WGBuAV32PFTeVmwj48jo= 2023-05-31T20:48:11.442 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:48:11 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:LALudZ2WGBuAV32PFTeVmwj48jo=', 'Content-Length': '0'} 2023-05-31T20:48:11.444 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:39:26 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000dff5ed70159aaa3c-006477b28b-1c17-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:48:11 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:48:11.444 DEBUG:boto:path=/pklxkd-59/obj55 2023-05-31T20:48:11.445 DEBUG:boto:auth_path=/pklxkd-59/obj55 2023-05-31T20:48:11.445 DEBUG:boto:path=/pklxkd-59/obj55?versionId=null 2023-05-31T20:48:11.446 DEBUG:boto:auth_path=/pklxkd-59/obj55?versionId=null 2023-05-31T20:48:11.446 DEBUG:boto:Method: HEAD 2023-05-31T20:48:11.446 DEBUG:boto:Path: /pklxkd-59/obj55?versionId=null 2023-05-31T20:48:11.447 DEBUG:boto:Data: 2023-05-31T20:48:11.447 DEBUG:boto:Headers: {} 2023-05-31T20:48:11.447 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:48:11.448 DEBUG:boto:Port: 8000 2023-05-31T20:48:11.448 DEBUG:boto:Params: {} 2023-05-31T20:48:11.448 DEBUG:boto:Token: None 2023-05-31T20:48:11.449 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:48:11 GMT /pklxkd-59/obj55?versionId=null 2023-05-31T20:48:11.449 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:LALudZ2WGBuAV32PFTeVmwj48jo= 2023-05-31T20:48:11.449 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:48:11 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:LALudZ2WGBuAV32PFTeVmwj48jo=', 'Content-Length': '0'} 2023-05-31T20:48:11.451 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:39:26 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000216357857c8142a2-006477b28b-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:48:11 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:48:11.452 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj55 2023-05-31T20:48:11.452 DEBUG:boto:path=/pklxkd-59/obj55 2023-05-31T20:48:11.452 DEBUG:boto:auth_path=/pklxkd-59/obj55 2023-05-31T20:48:11.453 DEBUG:boto:path=/pklxkd-59/obj55?versionId=null 2023-05-31T20:48:11.453 DEBUG:boto:auth_path=/pklxkd-59/obj55?versionId=null 2023-05-31T20:48:11.453 DEBUG:boto:Method: GET 2023-05-31T20:48:11.454 DEBUG:boto:Path: /pklxkd-59/obj55?versionId=null 2023-05-31T20:48:11.454 DEBUG:boto:Data: 2023-05-31T20:48:11.455 DEBUG:boto:Headers: {} 2023-05-31T20:48:11.455 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:48:11.455 DEBUG:boto:Port: 8001 2023-05-31T20:48:11.456 DEBUG:boto:Params: {} 2023-05-31T20:48:11.456 DEBUG:boto:Token: None 2023-05-31T20:48:11.456 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:48:11 GMT /pklxkd-59/obj55?versionId=null 2023-05-31T20:48:11.457 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:ipuVu3SXqzG9mrZXpfHBdbKfPEY= 2023-05-31T20:48:11.457 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:48:11 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:ipuVu3SXqzG9mrZXpfHBdbKfPEY=', 'Content-Length': '0'} 2023-05-31T20:48:11.459 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:39:26 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000052a18f6e8f1a5ccb-006477b28b-1c17-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:48:11 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:48:11.500 DEBUG:boto:path=/pklxkd-59/obj55 2023-05-31T20:48:11.501 DEBUG:boto:auth_path=/pklxkd-59/obj55 2023-05-31T20:48:11.501 DEBUG:boto:path=/pklxkd-59/obj55?versionId=null 2023-05-31T20:48:11.501 DEBUG:boto:auth_path=/pklxkd-59/obj55?versionId=null 2023-05-31T20:48:11.502 DEBUG:boto:Method: GET 2023-05-31T20:48:11.502 DEBUG:boto:Path: /pklxkd-59/obj55?versionId=null 2023-05-31T20:48:11.502 DEBUG:boto:Data: 2023-05-31T20:48:11.503 DEBUG:boto:Headers: {} 2023-05-31T20:48:11.503 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:48:11.503 DEBUG:boto:Port: 8000 2023-05-31T20:48:11.504 DEBUG:boto:Params: {} 2023-05-31T20:48:11.504 DEBUG:boto:Token: None 2023-05-31T20:48:11.505 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:48:11 GMT /pklxkd-59/obj55?versionId=null 2023-05-31T20:48:11.505 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:ipuVu3SXqzG9mrZXpfHBdbKfPEY= 2023-05-31T20:48:11.505 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:48:11 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:ipuVu3SXqzG9mrZXpfHBdbKfPEY=', 'Content-Length': '0'} 2023-05-31T20:48:11.507 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:39:26 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000035ccac67c963c88b-006477b28b-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:48:11 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:48:11.508 DEBUG:boto:path=/pklxkd-59/ 2023-05-31T20:48:11.508 DEBUG:boto:auth_path=/pklxkd-59/ 2023-05-31T20:48:11.509 DEBUG:boto:path=/pklxkd-59/?prefix=obj55 2023-05-31T20:48:11.509 DEBUG:boto:auth_path=/pklxkd-59/?prefix=obj55 2023-05-31T20:48:11.509 DEBUG:boto:Method: GET 2023-05-31T20:48:11.510 DEBUG:boto:Path: /pklxkd-59/?prefix=obj55 2023-05-31T20:48:11.510 DEBUG:boto:Data: 2023-05-31T20:48:11.510 DEBUG:boto:Headers: {} 2023-05-31T20:48:11.511 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:48:11.511 DEBUG:boto:Port: 8001 2023-05-31T20:48:11.512 DEBUG:boto:Params: {} 2023-05-31T20:48:11.512 DEBUG:boto:Token: None 2023-05-31T20:48:11.512 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:48:11 GMT /pklxkd-59/ 2023-05-31T20:48:11.513 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:l166NKUsN4SBNL48MAZina5NQhQ= 2023-05-31T20:48:11.513 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:48:11 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:l166NKUsN4SBNL48MAZina5NQhQ=', 'Content-Length': '0'} 2023-05-31T20:48:11.516 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000593723bb813aa8ce-006477b28b-1c17-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:48:11 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:48:11.516 DEBUG:boto:b'pklxkd-59obj551000falseobj552023-05-31T20:39:26.630Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:48:11.517 DEBUG:boto:path=/pklxkd-59/ 2023-05-31T20:48:11.517 DEBUG:boto:auth_path=/pklxkd-59/ 2023-05-31T20:48:11.518 DEBUG:boto:path=/pklxkd-59/?prefix=obj55 2023-05-31T20:48:11.518 DEBUG:boto:auth_path=/pklxkd-59/?prefix=obj55 2023-05-31T20:48:11.518 DEBUG:boto:Method: GET 2023-05-31T20:48:11.519 DEBUG:boto:Path: /pklxkd-59/?prefix=obj55 2023-05-31T20:48:11.519 DEBUG:boto:Data: 2023-05-31T20:48:11.519 DEBUG:boto:Headers: {} 2023-05-31T20:48:11.520 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:48:11.520 DEBUG:boto:Port: 8000 2023-05-31T20:48:11.520 DEBUG:boto:Params: {} 2023-05-31T20:48:11.521 DEBUG:boto:Token: None 2023-05-31T20:48:11.521 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:48:11 GMT /pklxkd-59/ 2023-05-31T20:48:11.522 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:l166NKUsN4SBNL48MAZina5NQhQ= 2023-05-31T20:48:11.522 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:48:11 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:l166NKUsN4SBNL48MAZina5NQhQ=', 'Content-Length': '0'} 2023-05-31T20:48:11.525 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000005bf1e80fddaea988-006477b28b-1630-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:48:11 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:48:11.525 DEBUG:boto:b'pklxkd-59obj551000falseobj552023-05-31T20:39:26.630Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:48:11.526 DEBUG:boto:path=/pklxkd-59/obj55 2023-05-31T20:48:11.526 DEBUG:boto:auth_path=/pklxkd-59/obj55 2023-05-31T20:48:11.527 DEBUG:boto:Method: HEAD 2023-05-31T20:48:11.527 DEBUG:boto:Path: /pklxkd-59/obj55 2023-05-31T20:48:11.527 DEBUG:boto:Data: 2023-05-31T20:48:11.528 DEBUG:boto:Headers: {} 2023-05-31T20:48:11.528 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:48:11.528 DEBUG:boto:Port: 8001 2023-05-31T20:48:11.529 DEBUG:boto:Params: {} 2023-05-31T20:48:11.529 DEBUG:boto:Token: None 2023-05-31T20:48:11.530 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:48:11 GMT /pklxkd-59/obj55 2023-05-31T20:48:11.530 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:227GoOV3NTs3HE7XA9NykNahWQU= 2023-05-31T20:48:11.530 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:48:11 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:227GoOV3NTs3HE7XA9NykNahWQU=', 'Content-Length': '0'} 2023-05-31T20:48:11.532 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:39:26 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008c60dca2a4e94a01-006477b28b-1c17-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:48:11 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:48:11.533 DEBUG:boto:path=/pklxkd-59/obj55 2023-05-31T20:48:11.533 DEBUG:boto:auth_path=/pklxkd-59/obj55 2023-05-31T20:48:11.533 DEBUG:boto:Method: HEAD 2023-05-31T20:48:11.534 DEBUG:boto:Path: /pklxkd-59/obj55 2023-05-31T20:48:11.534 DEBUG:boto:Data: 2023-05-31T20:48:11.534 DEBUG:boto:Headers: {} 2023-05-31T20:48:11.535 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:48:11.535 DEBUG:boto:Port: 8000 2023-05-31T20:48:11.535 DEBUG:boto:Params: {} 2023-05-31T20:48:11.536 DEBUG:boto:Token: None 2023-05-31T20:48:11.536 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:48:11 GMT /pklxkd-59/obj55 2023-05-31T20:48:11.536 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:227GoOV3NTs3HE7XA9NykNahWQU= 2023-05-31T20:48:11.537 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:48:11 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:227GoOV3NTs3HE7XA9NykNahWQU=', 'Content-Length': '0'} 2023-05-31T20:48:11.538 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:39:26 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004f9008c2b41f67ed-006477b28b-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:48:11 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:48:11.539 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj55 2023-05-31T20:48:11.539 DEBUG:boto:path=/pklxkd-59/obj55 2023-05-31T20:48:11.540 DEBUG:boto:auth_path=/pklxkd-59/obj55 2023-05-31T20:48:11.540 DEBUG:boto:Method: GET 2023-05-31T20:48:11.540 DEBUG:boto:Path: /pklxkd-59/obj55 2023-05-31T20:48:11.541 DEBUG:boto:Data: 2023-05-31T20:48:11.541 DEBUG:boto:Headers: {} 2023-05-31T20:48:11.542 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:48:11.542 DEBUG:boto:Port: 8001 2023-05-31T20:48:11.542 DEBUG:boto:Params: {} 2023-05-31T20:48:11.543 DEBUG:boto:Token: None 2023-05-31T20:48:11.543 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:48:11 GMT /pklxkd-59/obj55 2023-05-31T20:48:11.543 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:KG4GIFvUc9VLFt47sJF4PZCbm9s= 2023-05-31T20:48:11.544 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:48:11 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:KG4GIFvUc9VLFt47sJF4PZCbm9s=', 'Content-Length': '0'} 2023-05-31T20:48:11.546 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:39:26 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008ff91d711815b4e2-006477b28b-1c17-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:48:11 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:48:11.546 DEBUG:boto:path=/pklxkd-59/obj55 2023-05-31T20:48:11.547 DEBUG:boto:auth_path=/pklxkd-59/obj55 2023-05-31T20:48:11.547 DEBUG:boto:Method: GET 2023-05-31T20:48:11.547 DEBUG:boto:Path: /pklxkd-59/obj55 2023-05-31T20:48:11.548 DEBUG:boto:Data: 2023-05-31T20:48:11.548 DEBUG:boto:Headers: {} 2023-05-31T20:48:11.548 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:48:11.549 DEBUG:boto:Port: 8000 2023-05-31T20:48:11.549 DEBUG:boto:Params: {} 2023-05-31T20:48:11.550 DEBUG:boto:Token: None 2023-05-31T20:48:11.550 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:48:11 GMT /pklxkd-59/obj55 2023-05-31T20:48:11.550 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:KG4GIFvUc9VLFt47sJF4PZCbm9s= 2023-05-31T20:48:11.551 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:48:11 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:KG4GIFvUc9VLFt47sJF4PZCbm9s=', 'Content-Length': '0'} 2023-05-31T20:48:11.553 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:39:26 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007712b195fc48d9bd-006477b28b-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:48:11 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:48:11.553 DEBUG:boto:path=/pklxkd-59/ 2023-05-31T20:48:11.554 DEBUG:boto:auth_path=/pklxkd-59/ 2023-05-31T20:48:11.554 DEBUG:boto:path=/pklxkd-59/?prefix=obj55 2023-05-31T20:48:11.554 DEBUG:boto:auth_path=/pklxkd-59/?prefix=obj55 2023-05-31T20:48:11.555 DEBUG:boto:Method: GET 2023-05-31T20:48:11.555 DEBUG:boto:Path: /pklxkd-59/?prefix=obj55 2023-05-31T20:48:11.556 DEBUG:boto:Data: 2023-05-31T20:48:11.556 DEBUG:boto:Headers: {} 2023-05-31T20:48:11.556 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:48:11.557 DEBUG:boto:Port: 8001 2023-05-31T20:48:11.557 DEBUG:boto:Params: {} 2023-05-31T20:48:11.557 DEBUG:boto:Token: None 2023-05-31T20:48:11.558 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:48:11 GMT /pklxkd-59/ 2023-05-31T20:48:11.558 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:l166NKUsN4SBNL48MAZina5NQhQ= 2023-05-31T20:48:11.559 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:48:11 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:l166NKUsN4SBNL48MAZina5NQhQ=', 'Content-Length': '0'} 2023-05-31T20:48:11.561 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b5d89260e384bac1-006477b28b-1c17-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:48:11 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:48:11.562 DEBUG:boto:b'pklxkd-59obj551000falseobj552023-05-31T20:39:26.630Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:48:11.562 DEBUG:boto:path=/pklxkd-59/ 2023-05-31T20:48:11.563 DEBUG:boto:auth_path=/pklxkd-59/ 2023-05-31T20:48:11.563 DEBUG:boto:path=/pklxkd-59/?prefix=obj55 2023-05-31T20:48:11.563 DEBUG:boto:auth_path=/pklxkd-59/?prefix=obj55 2023-05-31T20:48:11.564 DEBUG:boto:Method: GET 2023-05-31T20:48:11.564 DEBUG:boto:Path: /pklxkd-59/?prefix=obj55 2023-05-31T20:48:11.564 DEBUG:boto:Data: 2023-05-31T20:48:11.565 DEBUG:boto:Headers: {} 2023-05-31T20:48:11.565 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:48:11.565 DEBUG:boto:Port: 8000 2023-05-31T20:48:11.566 DEBUG:boto:Params: {} 2023-05-31T20:48:11.566 DEBUG:boto:Token: None 2023-05-31T20:48:11.567 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:48:11 GMT /pklxkd-59/ 2023-05-31T20:48:11.567 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:l166NKUsN4SBNL48MAZina5NQhQ= 2023-05-31T20:48:11.567 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:48:11 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:l166NKUsN4SBNL48MAZina5NQhQ=', 'Content-Length': '0'} 2023-05-31T20:48:11.571 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a9b67a126f397c1e-006477b28b-1630-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:48:11 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:48:11.571 DEBUG:boto:b'pklxkd-59obj551000falseobj552023-05-31T20:39:26.630Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:48:11.572 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj57 2023-05-31T20:48:11.572 DEBUG:boto:path=/pklxkd-59/obj57 2023-05-31T20:48:11.573 DEBUG:boto:auth_path=/pklxkd-59/obj57 2023-05-31T20:48:11.573 DEBUG:boto:path=/pklxkd-59/obj57?versionId=null 2023-05-31T20:48:11.573 DEBUG:boto:auth_path=/pklxkd-59/obj57?versionId=null 2023-05-31T20:48:11.574 DEBUG:boto:Method: GET 2023-05-31T20:48:11.574 DEBUG:boto:Path: /pklxkd-59/obj57?versionId=null 2023-05-31T20:48:11.574 DEBUG:boto:Data: 2023-05-31T20:48:11.575 DEBUG:boto:Headers: {} 2023-05-31T20:48:11.575 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:48:11.575 DEBUG:boto:Port: 8001 2023-05-31T20:48:11.576 DEBUG:boto:Params: {} 2023-05-31T20:48:11.576 DEBUG:boto:Token: None 2023-05-31T20:48:11.577 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:48:11 GMT /pklxkd-59/obj57?versionId=null 2023-05-31T20:48:11.577 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:GmnAM8QKceOFJwjm+QYg6DhmU3A= 2023-05-31T20:48:11.577 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:48:11 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:GmnAM8QKceOFJwjm+QYg6DhmU3A=', 'Content-Length': '0'} 2023-05-31T20:48:11.579 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:44:50 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000671989e2def274f7-006477b28b-1c17-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:48:11 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:48:11.580 DEBUG:boto:path=/pklxkd-59/obj57 2023-05-31T20:48:11.580 DEBUG:boto:auth_path=/pklxkd-59/obj57 2023-05-31T20:48:11.581 DEBUG:boto:path=/pklxkd-59/obj57?versionId=null 2023-05-31T20:48:11.581 DEBUG:boto:auth_path=/pklxkd-59/obj57?versionId=null 2023-05-31T20:48:11.582 DEBUG:boto:Method: GET 2023-05-31T20:48:11.582 DEBUG:boto:Path: /pklxkd-59/obj57?versionId=null 2023-05-31T20:48:11.582 DEBUG:boto:Data: 2023-05-31T20:48:11.583 DEBUG:boto:Headers: {} 2023-05-31T20:48:11.583 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:48:11.583 DEBUG:boto:Port: 8000 2023-05-31T20:48:11.584 DEBUG:boto:Params: {} 2023-05-31T20:48:11.584 DEBUG:boto:Token: None 2023-05-31T20:48:11.584 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:48:11 GMT /pklxkd-59/obj57?versionId=null 2023-05-31T20:48:11.585 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:GmnAM8QKceOFJwjm+QYg6DhmU3A= 2023-05-31T20:48:11.585 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:48:11 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:GmnAM8QKceOFJwjm+QYg6DhmU3A=', 'Content-Length': '0'} 2023-05-31T20:48:11.587 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:44:50 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e73031297d1a31c5-006477b28b-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:48:11 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:48:11.628 DEBUG:boto:path=/pklxkd-59/obj57 2023-05-31T20:48:11.629 DEBUG:boto:auth_path=/pklxkd-59/obj57 2023-05-31T20:48:11.629 DEBUG:boto:path=/pklxkd-59/obj57?versionId=null 2023-05-31T20:48:11.629 DEBUG:boto:auth_path=/pklxkd-59/obj57?versionId=null 2023-05-31T20:48:11.630 DEBUG:boto:Method: HEAD 2023-05-31T20:48:11.630 DEBUG:boto:Path: /pklxkd-59/obj57?versionId=null 2023-05-31T20:48:11.630 DEBUG:boto:Data: 2023-05-31T20:48:11.631 DEBUG:boto:Headers: {} 2023-05-31T20:48:11.631 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:48:11.631 DEBUG:boto:Port: 8001 2023-05-31T20:48:11.631 DEBUG:boto:Params: {} 2023-05-31T20:48:11.632 DEBUG:boto:Token: None 2023-05-31T20:48:11.632 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:48:11 GMT /pklxkd-59/obj57?versionId=null 2023-05-31T20:48:11.633 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:0jy7aibXv0BQzyfICbef8gn4+eY= 2023-05-31T20:48:11.633 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:48:11 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:0jy7aibXv0BQzyfICbef8gn4+eY=', 'Content-Length': '0'} 2023-05-31T20:48:11.635 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:44:50 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e4d471655b73330c-006477b28b-1c17-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:48:11 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:48:11.635 DEBUG:boto:path=/pklxkd-59/obj57 2023-05-31T20:48:11.636 DEBUG:boto:auth_path=/pklxkd-59/obj57 2023-05-31T20:48:11.636 DEBUG:boto:path=/pklxkd-59/obj57?versionId=null 2023-05-31T20:48:11.636 DEBUG:boto:auth_path=/pklxkd-59/obj57?versionId=null 2023-05-31T20:48:11.637 DEBUG:boto:Method: HEAD 2023-05-31T20:48:11.637 DEBUG:boto:Path: /pklxkd-59/obj57?versionId=null 2023-05-31T20:48:11.637 DEBUG:boto:Data: 2023-05-31T20:48:11.638 DEBUG:boto:Headers: {} 2023-05-31T20:48:11.638 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:48:11.638 DEBUG:boto:Port: 8000 2023-05-31T20:48:11.639 DEBUG:boto:Params: {} 2023-05-31T20:48:11.639 DEBUG:boto:Token: None 2023-05-31T20:48:11.639 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:48:11 GMT /pklxkd-59/obj57?versionId=null 2023-05-31T20:48:11.640 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:0jy7aibXv0BQzyfICbef8gn4+eY= 2023-05-31T20:48:11.640 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:48:11 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:0jy7aibXv0BQzyfICbef8gn4+eY=', 'Content-Length': '0'} 2023-05-31T20:48:11.642 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:44:50 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009293ebfa10858628-006477b28b-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:48:11 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:48:11.642 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj57 2023-05-31T20:48:11.643 DEBUG:boto:path=/pklxkd-59/obj57 2023-05-31T20:48:11.643 DEBUG:boto:auth_path=/pklxkd-59/obj57 2023-05-31T20:48:11.643 DEBUG:boto:path=/pklxkd-59/obj57?versionId=null 2023-05-31T20:48:11.644 DEBUG:boto:auth_path=/pklxkd-59/obj57?versionId=null 2023-05-31T20:48:11.644 DEBUG:boto:Method: GET 2023-05-31T20:48:11.645 DEBUG:boto:Path: /pklxkd-59/obj57?versionId=null 2023-05-31T20:48:11.645 DEBUG:boto:Data: 2023-05-31T20:48:11.645 DEBUG:boto:Headers: {} 2023-05-31T20:48:11.646 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:48:11.646 DEBUG:boto:Port: 8001 2023-05-31T20:48:11.646 DEBUG:boto:Params: {} 2023-05-31T20:48:11.647 DEBUG:boto:Token: None 2023-05-31T20:48:11.647 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:48:11 GMT /pklxkd-59/obj57?versionId=null 2023-05-31T20:48:11.647 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:GmnAM8QKceOFJwjm+QYg6DhmU3A= 2023-05-31T20:48:11.648 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:48:11 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:GmnAM8QKceOFJwjm+QYg6DhmU3A=', 'Content-Length': '0'} 2023-05-31T20:48:11.650 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:44:50 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005a94df275d3b1b6d-006477b28b-1c17-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:48:11 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:48:11.651 DEBUG:boto:path=/pklxkd-59/obj57 2023-05-31T20:48:11.651 DEBUG:boto:auth_path=/pklxkd-59/obj57 2023-05-31T20:48:11.651 DEBUG:boto:path=/pklxkd-59/obj57?versionId=null 2023-05-31T20:48:11.652 DEBUG:boto:auth_path=/pklxkd-59/obj57?versionId=null 2023-05-31T20:48:11.652 DEBUG:boto:Method: GET 2023-05-31T20:48:11.652 DEBUG:boto:Path: /pklxkd-59/obj57?versionId=null 2023-05-31T20:48:11.653 DEBUG:boto:Data: 2023-05-31T20:48:11.653 DEBUG:boto:Headers: {} 2023-05-31T20:48:11.654 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:48:11.654 DEBUG:boto:Port: 8000 2023-05-31T20:48:11.654 DEBUG:boto:Params: {} 2023-05-31T20:48:11.655 DEBUG:boto:Token: None 2023-05-31T20:48:11.655 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:48:11 GMT /pklxkd-59/obj57?versionId=null 2023-05-31T20:48:11.655 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:GmnAM8QKceOFJwjm+QYg6DhmU3A= 2023-05-31T20:48:11.656 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:48:11 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:GmnAM8QKceOFJwjm+QYg6DhmU3A=', 'Content-Length': '0'} 2023-05-31T20:48:11.658 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:44:50 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b6e580244eec80d8-006477b28b-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:48:11 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:48:11.658 DEBUG:boto:path=/pklxkd-59/ 2023-05-31T20:48:11.659 DEBUG:boto:auth_path=/pklxkd-59/ 2023-05-31T20:48:11.659 DEBUG:boto:path=/pklxkd-59/?prefix=obj57 2023-05-31T20:48:11.660 DEBUG:boto:auth_path=/pklxkd-59/?prefix=obj57 2023-05-31T20:48:11.660 DEBUG:boto:Method: GET 2023-05-31T20:48:11.660 DEBUG:boto:Path: /pklxkd-59/?prefix=obj57 2023-05-31T20:48:11.661 DEBUG:boto:Data: 2023-05-31T20:48:11.661 DEBUG:boto:Headers: {} 2023-05-31T20:48:11.661 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:48:11.662 DEBUG:boto:Port: 8001 2023-05-31T20:48:11.662 DEBUG:boto:Params: {} 2023-05-31T20:48:11.662 DEBUG:boto:Token: None 2023-05-31T20:48:11.663 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:48:11 GMT /pklxkd-59/ 2023-05-31T20:48:11.663 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:l166NKUsN4SBNL48MAZina5NQhQ= 2023-05-31T20:48:11.664 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:48:11 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:l166NKUsN4SBNL48MAZina5NQhQ=', 'Content-Length': '0'} 2023-05-31T20:48:11.666 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000057f37d6e85932e40-006477b28b-1c17-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:48:11 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:48:11.667 DEBUG:boto:b'pklxkd-59obj571000falseobj572023-05-31T20:44:50.627Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:48:11.667 DEBUG:boto:path=/pklxkd-59/ 2023-05-31T20:48:11.668 DEBUG:boto:auth_path=/pklxkd-59/ 2023-05-31T20:48:11.668 DEBUG:boto:path=/pklxkd-59/?prefix=obj57 2023-05-31T20:48:11.668 DEBUG:boto:auth_path=/pklxkd-59/?prefix=obj57 2023-05-31T20:48:11.669 DEBUG:boto:Method: GET 2023-05-31T20:48:11.669 DEBUG:boto:Path: /pklxkd-59/?prefix=obj57 2023-05-31T20:48:11.669 DEBUG:boto:Data: 2023-05-31T20:48:11.670 DEBUG:boto:Headers: {} 2023-05-31T20:48:11.670 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:48:11.671 DEBUG:boto:Port: 8000 2023-05-31T20:48:11.671 DEBUG:boto:Params: {} 2023-05-31T20:48:11.671 DEBUG:boto:Token: None 2023-05-31T20:48:11.672 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:48:11 GMT /pklxkd-59/ 2023-05-31T20:48:11.672 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:l166NKUsN4SBNL48MAZina5NQhQ= 2023-05-31T20:48:11.673 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:48:11 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:l166NKUsN4SBNL48MAZina5NQhQ=', 'Content-Length': '0'} 2023-05-31T20:48:11.675 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000001cf3dcd6287ff8da-006477b28b-1630-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:48:11 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:48:11.676 DEBUG:boto:b'pklxkd-59obj571000falseobj572023-05-31T20:44:50.627Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:48:11.677 DEBUG:boto:path=/pklxkd-59/obj57 2023-05-31T20:48:11.677 DEBUG:boto:auth_path=/pklxkd-59/obj57 2023-05-31T20:48:11.677 DEBUG:boto:Method: HEAD 2023-05-31T20:48:11.678 DEBUG:boto:Path: /pklxkd-59/obj57 2023-05-31T20:48:11.678 DEBUG:boto:Data: 2023-05-31T20:48:11.678 DEBUG:boto:Headers: {} 2023-05-31T20:48:11.679 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:48:11.679 DEBUG:boto:Port: 8001 2023-05-31T20:48:11.679 DEBUG:boto:Params: {} 2023-05-31T20:48:11.680 DEBUG:boto:Token: None 2023-05-31T20:48:11.680 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:48:11 GMT /pklxkd-59/obj57 2023-05-31T20:48:11.681 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:8vxjKOb7tFvyFhYD/aSj+4isErA= 2023-05-31T20:48:11.681 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:48:11 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:8vxjKOb7tFvyFhYD/aSj+4isErA=', 'Content-Length': '0'} 2023-05-31T20:48:11.682 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:44:50 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005cf3a3448afaa53d-006477b28b-1c17-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:48:11 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:48:11.683 DEBUG:boto:path=/pklxkd-59/obj57 2023-05-31T20:48:11.683 DEBUG:boto:auth_path=/pklxkd-59/obj57 2023-05-31T20:48:11.684 DEBUG:boto:Method: HEAD 2023-05-31T20:48:11.684 DEBUG:boto:Path: /pklxkd-59/obj57 2023-05-31T20:48:11.684 DEBUG:boto:Data: 2023-05-31T20:48:11.685 DEBUG:boto:Headers: {} 2023-05-31T20:48:11.685 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:48:11.685 DEBUG:boto:Port: 8000 2023-05-31T20:48:11.686 DEBUG:boto:Params: {} 2023-05-31T20:48:11.686 DEBUG:boto:Token: None 2023-05-31T20:48:11.687 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:48:11 GMT /pklxkd-59/obj57 2023-05-31T20:48:11.687 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:8vxjKOb7tFvyFhYD/aSj+4isErA= 2023-05-31T20:48:11.687 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:48:11 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:8vxjKOb7tFvyFhYD/aSj+4isErA=', 'Content-Length': '0'} 2023-05-31T20:48:11.689 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:44:50 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000034db32c9f25ac5e7-006477b28b-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:48:11 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:48:11.689 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj57 2023-05-31T20:48:11.690 DEBUG:boto:path=/pklxkd-59/obj57 2023-05-31T20:48:11.690 DEBUG:boto:auth_path=/pklxkd-59/obj57 2023-05-31T20:48:11.691 DEBUG:boto:Method: GET 2023-05-31T20:48:11.691 DEBUG:boto:Path: /pklxkd-59/obj57 2023-05-31T20:48:11.691 DEBUG:boto:Data: 2023-05-31T20:48:11.692 DEBUG:boto:Headers: {} 2023-05-31T20:48:11.692 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:48:11.692 DEBUG:boto:Port: 8001 2023-05-31T20:48:11.693 DEBUG:boto:Params: {} 2023-05-31T20:48:11.693 DEBUG:boto:Token: None 2023-05-31T20:48:11.694 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:48:11 GMT /pklxkd-59/obj57 2023-05-31T20:48:11.694 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:ie+656Il75HhUnWJ42CKmV0mdDw= 2023-05-31T20:48:11.694 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:48:11 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:ie+656Il75HhUnWJ42CKmV0mdDw=', 'Content-Length': '0'} 2023-05-31T20:48:11.696 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:44:50 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008aca4d0023e39be7-006477b28b-1c17-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:48:11 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:48:11.697 DEBUG:boto:path=/pklxkd-59/obj57 2023-05-31T20:48:11.697 DEBUG:boto:auth_path=/pklxkd-59/obj57 2023-05-31T20:48:11.698 DEBUG:boto:Method: GET 2023-05-31T20:48:11.698 DEBUG:boto:Path: /pklxkd-59/obj57 2023-05-31T20:48:11.698 DEBUG:boto:Data: 2023-05-31T20:48:11.699 DEBUG:boto:Headers: {} 2023-05-31T20:48:11.699 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:48:11.699 DEBUG:boto:Port: 8000 2023-05-31T20:48:11.700 DEBUG:boto:Params: {} 2023-05-31T20:48:11.700 DEBUG:boto:Token: None 2023-05-31T20:48:11.701 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:48:11 GMT /pklxkd-59/obj57 2023-05-31T20:48:11.701 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:ie+656Il75HhUnWJ42CKmV0mdDw= 2023-05-31T20:48:11.701 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:48:11 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:ie+656Il75HhUnWJ42CKmV0mdDw=', 'Content-Length': '0'} 2023-05-31T20:48:11.703 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:44:50 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000eed173e836df0029-006477b28b-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:48:11 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:48:11.704 DEBUG:boto:path=/pklxkd-59/ 2023-05-31T20:48:11.704 DEBUG:boto:auth_path=/pklxkd-59/ 2023-05-31T20:48:11.705 DEBUG:boto:path=/pklxkd-59/?prefix=obj57 2023-05-31T20:48:11.705 DEBUG:boto:auth_path=/pklxkd-59/?prefix=obj57 2023-05-31T20:48:11.705 DEBUG:boto:Method: GET 2023-05-31T20:48:11.706 DEBUG:boto:Path: /pklxkd-59/?prefix=obj57 2023-05-31T20:48:11.706 DEBUG:boto:Data: 2023-05-31T20:48:11.706 DEBUG:boto:Headers: {} 2023-05-31T20:48:11.707 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:48:11.707 DEBUG:boto:Port: 8001 2023-05-31T20:48:11.707 DEBUG:boto:Params: {} 2023-05-31T20:48:11.708 DEBUG:boto:Token: None 2023-05-31T20:48:11.708 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:48:11 GMT /pklxkd-59/ 2023-05-31T20:48:11.709 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:l166NKUsN4SBNL48MAZina5NQhQ= 2023-05-31T20:48:11.709 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:48:11 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:l166NKUsN4SBNL48MAZina5NQhQ=', 'Content-Length': '0'} 2023-05-31T20:48:11.711 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f01739afc93f8548-006477b28b-1c17-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:48:11 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:48:11.752 DEBUG:boto:b'pklxkd-59obj571000falseobj572023-05-31T20:44:50.627Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:48:11.753 DEBUG:boto:path=/pklxkd-59/ 2023-05-31T20:48:11.753 DEBUG:boto:auth_path=/pklxkd-59/ 2023-05-31T20:48:11.754 DEBUG:boto:path=/pklxkd-59/?prefix=obj57 2023-05-31T20:48:11.754 DEBUG:boto:auth_path=/pklxkd-59/?prefix=obj57 2023-05-31T20:48:11.754 DEBUG:boto:Method: GET 2023-05-31T20:48:11.755 DEBUG:boto:Path: /pklxkd-59/?prefix=obj57 2023-05-31T20:48:11.755 DEBUG:boto:Data: 2023-05-31T20:48:11.755 DEBUG:boto:Headers: {} 2023-05-31T20:48:11.756 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:48:11.756 DEBUG:boto:Port: 8000 2023-05-31T20:48:11.756 DEBUG:boto:Params: {} 2023-05-31T20:48:11.757 DEBUG:boto:Token: None 2023-05-31T20:48:11.757 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:48:11 GMT /pklxkd-59/ 2023-05-31T20:48:11.758 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:l166NKUsN4SBNL48MAZina5NQhQ= 2023-05-31T20:48:11.758 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:48:11 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:l166NKUsN4SBNL48MAZina5NQhQ=', 'Content-Length': '0'} 2023-05-31T20:48:11.761 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000051bc3852f13da55-006477b28b-1630-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:48:11 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:48:11.762 DEBUG:boto:b'pklxkd-59obj571000falseobj572023-05-31T20:44:50.627Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:48:11.762 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj68 2023-05-31T20:48:11.763 DEBUG:boto:path=/pklxkd-59/obj68 2023-05-31T20:48:11.763 DEBUG:boto:auth_path=/pklxkd-59/obj68 2023-05-31T20:48:11.764 DEBUG:boto:path=/pklxkd-59/obj68?versionId=null 2023-05-31T20:48:11.764 DEBUG:boto:auth_path=/pklxkd-59/obj68?versionId=null 2023-05-31T20:48:11.764 DEBUG:boto:Method: GET 2023-05-31T20:48:11.765 DEBUG:boto:Path: /pklxkd-59/obj68?versionId=null 2023-05-31T20:48:11.765 DEBUG:boto:Data: 2023-05-31T20:48:11.765 DEBUG:boto:Headers: {} 2023-05-31T20:48:11.766 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:48:11.766 DEBUG:boto:Port: 8001 2023-05-31T20:48:11.766 DEBUG:boto:Params: {} 2023-05-31T20:48:11.767 DEBUG:boto:Token: None 2023-05-31T20:48:11.767 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:48:11 GMT /pklxkd-59/obj68?versionId=null 2023-05-31T20:48:11.768 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:iXIgUcWsuMMajY9rK4O2KBB3szk= 2023-05-31T20:48:11.768 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:48:11 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:iXIgUcWsuMMajY9rK4O2KBB3szk=', 'Content-Length': '0'} 2023-05-31T20:48:11.770 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:43:08 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000bc4416e54368dd24-006477b28b-1c17-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:48:11 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:48:11.771 DEBUG:boto:path=/pklxkd-59/obj68 2023-05-31T20:48:11.771 DEBUG:boto:auth_path=/pklxkd-59/obj68 2023-05-31T20:48:11.772 DEBUG:boto:path=/pklxkd-59/obj68?versionId=null 2023-05-31T20:48:11.772 DEBUG:boto:auth_path=/pklxkd-59/obj68?versionId=null 2023-05-31T20:48:11.772 DEBUG:boto:Method: GET 2023-05-31T20:48:11.773 DEBUG:boto:Path: /pklxkd-59/obj68?versionId=null 2023-05-31T20:48:11.773 DEBUG:boto:Data: 2023-05-31T20:48:11.773 DEBUG:boto:Headers: {} 2023-05-31T20:48:11.774 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:48:11.774 DEBUG:boto:Port: 8000 2023-05-31T20:48:11.774 DEBUG:boto:Params: {} 2023-05-31T20:48:11.775 DEBUG:boto:Token: None 2023-05-31T20:48:11.775 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:48:11 GMT /pklxkd-59/obj68?versionId=null 2023-05-31T20:48:11.776 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:iXIgUcWsuMMajY9rK4O2KBB3szk= 2023-05-31T20:48:11.776 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:48:11 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:iXIgUcWsuMMajY9rK4O2KBB3szk=', 'Content-Length': '0'} 2023-05-31T20:48:11.778 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:43:08 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000683a99fb1dbda6b5-006477b28b-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:48:11 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:48:11.820 DEBUG:boto:path=/pklxkd-59/obj68 2023-05-31T20:48:11.821 DEBUG:boto:auth_path=/pklxkd-59/obj68 2023-05-31T20:48:11.821 DEBUG:boto:path=/pklxkd-59/obj68?versionId=null 2023-05-31T20:48:11.821 DEBUG:boto:auth_path=/pklxkd-59/obj68?versionId=null 2023-05-31T20:48:11.822 DEBUG:boto:Method: HEAD 2023-05-31T20:48:11.822 DEBUG:boto:Path: /pklxkd-59/obj68?versionId=null 2023-05-31T20:48:11.822 DEBUG:boto:Data: 2023-05-31T20:48:11.823 DEBUG:boto:Headers: {} 2023-05-31T20:48:11.823 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:48:11.823 DEBUG:boto:Port: 8001 2023-05-31T20:48:11.823 DEBUG:boto:Params: {} 2023-05-31T20:48:11.824 DEBUG:boto:Token: None 2023-05-31T20:48:11.824 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:48:11 GMT /pklxkd-59/obj68?versionId=null 2023-05-31T20:48:11.825 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:GoyG92rZIE18VoIMZQmbNCtEx3M= 2023-05-31T20:48:11.825 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:48:11 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:GoyG92rZIE18VoIMZQmbNCtEx3M=', 'Content-Length': '0'} 2023-05-31T20:48:11.827 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:43:08 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e7f1bb663ec91d88-006477b28b-1c17-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:48:11 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:48:11.827 DEBUG:boto:path=/pklxkd-59/obj68 2023-05-31T20:48:11.828 DEBUG:boto:auth_path=/pklxkd-59/obj68 2023-05-31T20:48:11.828 DEBUG:boto:path=/pklxkd-59/obj68?versionId=null 2023-05-31T20:48:11.828 DEBUG:boto:auth_path=/pklxkd-59/obj68?versionId=null 2023-05-31T20:48:11.829 DEBUG:boto:Method: HEAD 2023-05-31T20:48:11.829 DEBUG:boto:Path: /pklxkd-59/obj68?versionId=null 2023-05-31T20:48:11.829 DEBUG:boto:Data: 2023-05-31T20:48:11.830 DEBUG:boto:Headers: {} 2023-05-31T20:48:11.830 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:48:11.830 DEBUG:boto:Port: 8000 2023-05-31T20:48:11.831 DEBUG:boto:Params: {} 2023-05-31T20:48:11.831 DEBUG:boto:Token: None 2023-05-31T20:48:11.831 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:48:11 GMT /pklxkd-59/obj68?versionId=null 2023-05-31T20:48:11.832 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:GoyG92rZIE18VoIMZQmbNCtEx3M= 2023-05-31T20:48:11.832 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:48:11 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:GoyG92rZIE18VoIMZQmbNCtEx3M=', 'Content-Length': '0'} 2023-05-31T20:48:11.834 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:43:08 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000031255ec9544f3874-006477b28b-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:48:11 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:48:11.835 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj68 2023-05-31T20:48:11.835 DEBUG:boto:path=/pklxkd-59/obj68 2023-05-31T20:48:11.835 DEBUG:boto:auth_path=/pklxkd-59/obj68 2023-05-31T20:48:11.836 DEBUG:boto:path=/pklxkd-59/obj68?versionId=null 2023-05-31T20:48:11.836 DEBUG:boto:auth_path=/pklxkd-59/obj68?versionId=null 2023-05-31T20:48:11.836 DEBUG:boto:Method: GET 2023-05-31T20:48:11.837 DEBUG:boto:Path: /pklxkd-59/obj68?versionId=null 2023-05-31T20:48:11.837 DEBUG:boto:Data: 2023-05-31T20:48:11.837 DEBUG:boto:Headers: {} 2023-05-31T20:48:11.838 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:48:11.838 DEBUG:boto:Port: 8001 2023-05-31T20:48:11.839 DEBUG:boto:Params: {} 2023-05-31T20:48:11.839 DEBUG:boto:Token: None 2023-05-31T20:48:11.839 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:48:11 GMT /pklxkd-59/obj68?versionId=null 2023-05-31T20:48:11.840 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:iXIgUcWsuMMajY9rK4O2KBB3szk= 2023-05-31T20:48:11.840 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:48:11 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:iXIgUcWsuMMajY9rK4O2KBB3szk=', 'Content-Length': '0'} 2023-05-31T20:48:11.842 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:43:08 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002d41573e8488e548-006477b28b-1c17-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:48:11 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:48:11.842 DEBUG:boto:path=/pklxkd-59/obj68 2023-05-31T20:48:11.843 DEBUG:boto:auth_path=/pklxkd-59/obj68 2023-05-31T20:48:11.843 DEBUG:boto:path=/pklxkd-59/obj68?versionId=null 2023-05-31T20:48:11.844 DEBUG:boto:auth_path=/pklxkd-59/obj68?versionId=null 2023-05-31T20:48:11.844 DEBUG:boto:Method: GET 2023-05-31T20:48:11.844 DEBUG:boto:Path: /pklxkd-59/obj68?versionId=null 2023-05-31T20:48:11.845 DEBUG:boto:Data: 2023-05-31T20:48:11.845 DEBUG:boto:Headers: {} 2023-05-31T20:48:11.845 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:48:11.846 DEBUG:boto:Port: 8000 2023-05-31T20:48:11.846 DEBUG:boto:Params: {} 2023-05-31T20:48:11.846 DEBUG:boto:Token: None 2023-05-31T20:48:11.847 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:48:11 GMT /pklxkd-59/obj68?versionId=null 2023-05-31T20:48:11.847 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:iXIgUcWsuMMajY9rK4O2KBB3szk= 2023-05-31T20:48:11.848 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:48:11 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:iXIgUcWsuMMajY9rK4O2KBB3szk=', 'Content-Length': '0'} 2023-05-31T20:48:11.850 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:43:08 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000097528df7fc855872-006477b28b-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:48:11 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:48:11.850 DEBUG:boto:path=/pklxkd-59/ 2023-05-31T20:48:11.851 DEBUG:boto:auth_path=/pklxkd-59/ 2023-05-31T20:48:11.851 DEBUG:boto:path=/pklxkd-59/?prefix=obj68 2023-05-31T20:48:11.851 DEBUG:boto:auth_path=/pklxkd-59/?prefix=obj68 2023-05-31T20:48:11.852 DEBUG:boto:Method: GET 2023-05-31T20:48:11.852 DEBUG:boto:Path: /pklxkd-59/?prefix=obj68 2023-05-31T20:48:11.853 DEBUG:boto:Data: 2023-05-31T20:48:11.853 DEBUG:boto:Headers: {} 2023-05-31T20:48:11.853 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:48:11.854 DEBUG:boto:Port: 8001 2023-05-31T20:48:11.854 DEBUG:boto:Params: {} 2023-05-31T20:48:11.854 DEBUG:boto:Token: None 2023-05-31T20:48:11.855 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:48:11 GMT /pklxkd-59/ 2023-05-31T20:48:11.855 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:l166NKUsN4SBNL48MAZina5NQhQ= 2023-05-31T20:48:11.856 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:48:11 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:l166NKUsN4SBNL48MAZina5NQhQ=', 'Content-Length': '0'} 2023-05-31T20:48:11.858 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000002723415bdaa36263-006477b28b-1c17-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:48:11 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:48:11.858 DEBUG:boto:b'pklxkd-59obj681000falseobj682023-05-31T20:43:08.066Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:48:11.859 DEBUG:boto:path=/pklxkd-59/ 2023-05-31T20:48:11.859 DEBUG:boto:auth_path=/pklxkd-59/ 2023-05-31T20:48:11.860 DEBUG:boto:path=/pklxkd-59/?prefix=obj68 2023-05-31T20:48:11.860 DEBUG:boto:auth_path=/pklxkd-59/?prefix=obj68 2023-05-31T20:48:11.860 DEBUG:boto:Method: GET 2023-05-31T20:48:11.861 DEBUG:boto:Path: /pklxkd-59/?prefix=obj68 2023-05-31T20:48:11.861 DEBUG:boto:Data: 2023-05-31T20:48:11.861 DEBUG:boto:Headers: {} 2023-05-31T20:48:11.862 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:48:11.862 DEBUG:boto:Port: 8000 2023-05-31T20:48:11.863 DEBUG:boto:Params: {} 2023-05-31T20:48:11.863 DEBUG:boto:Token: None 2023-05-31T20:48:11.863 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:48:11 GMT /pklxkd-59/ 2023-05-31T20:48:11.864 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:l166NKUsN4SBNL48MAZina5NQhQ= 2023-05-31T20:48:11.864 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:48:11 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:l166NKUsN4SBNL48MAZina5NQhQ=', 'Content-Length': '0'} 2023-05-31T20:48:11.868 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000003302635c7789c5db-006477b28b-1630-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:48:11 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:48:11.868 DEBUG:boto:b'pklxkd-59obj681000falseobj682023-05-31T20:43:08.066Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:48:11.869 DEBUG:boto:path=/pklxkd-59/obj68 2023-05-31T20:48:11.869 DEBUG:boto:auth_path=/pklxkd-59/obj68 2023-05-31T20:48:11.870 DEBUG:boto:Method: HEAD 2023-05-31T20:48:11.870 DEBUG:boto:Path: /pklxkd-59/obj68 2023-05-31T20:48:11.870 DEBUG:boto:Data: 2023-05-31T20:48:11.871 DEBUG:boto:Headers: {} 2023-05-31T20:48:11.871 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:48:11.871 DEBUG:boto:Port: 8001 2023-05-31T20:48:11.872 DEBUG:boto:Params: {} 2023-05-31T20:48:11.872 DEBUG:boto:Token: None 2023-05-31T20:48:11.872 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:48:11 GMT /pklxkd-59/obj68 2023-05-31T20:48:11.873 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:demKHTrd0BQN1yIptvQTVoBi7yY= 2023-05-31T20:48:11.873 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:48:11 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:demKHTrd0BQN1yIptvQTVoBi7yY=', 'Content-Length': '0'} 2023-05-31T20:48:11.875 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:43:08 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000fa6a5104ca34cd71-006477b28b-1c17-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:48:11 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:48:11.875 DEBUG:boto:path=/pklxkd-59/obj68 2023-05-31T20:48:11.876 DEBUG:boto:auth_path=/pklxkd-59/obj68 2023-05-31T20:48:11.876 DEBUG:boto:Method: HEAD 2023-05-31T20:48:11.876 DEBUG:boto:Path: /pklxkd-59/obj68 2023-05-31T20:48:11.877 DEBUG:boto:Data: 2023-05-31T20:48:11.877 DEBUG:boto:Headers: {} 2023-05-31T20:48:11.877 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:48:11.878 DEBUG:boto:Port: 8000 2023-05-31T20:48:11.878 DEBUG:boto:Params: {} 2023-05-31T20:48:11.878 DEBUG:boto:Token: None 2023-05-31T20:48:11.879 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:48:11 GMT /pklxkd-59/obj68 2023-05-31T20:48:11.879 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:demKHTrd0BQN1yIptvQTVoBi7yY= 2023-05-31T20:48:11.880 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:48:11 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:demKHTrd0BQN1yIptvQTVoBi7yY=', 'Content-Length': '0'} 2023-05-31T20:48:11.881 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:43:08 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007da0ccdb1951530e-006477b28b-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:48:11 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:48:11.882 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj68 2023-05-31T20:48:11.882 DEBUG:boto:path=/pklxkd-59/obj68 2023-05-31T20:48:11.883 DEBUG:boto:auth_path=/pklxkd-59/obj68 2023-05-31T20:48:11.883 DEBUG:boto:Method: GET 2023-05-31T20:48:11.883 DEBUG:boto:Path: /pklxkd-59/obj68 2023-05-31T20:48:11.884 DEBUG:boto:Data: 2023-05-31T20:48:11.884 DEBUG:boto:Headers: {} 2023-05-31T20:48:11.885 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:48:11.885 DEBUG:boto:Port: 8001 2023-05-31T20:48:11.885 DEBUG:boto:Params: {} 2023-05-31T20:48:11.886 DEBUG:boto:Token: None 2023-05-31T20:48:11.886 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:48:11 GMT /pklxkd-59/obj68 2023-05-31T20:48:11.886 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:U+vdDHqwXeY6mjnGdubytTRRz3w= 2023-05-31T20:48:11.887 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:48:11 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:U+vdDHqwXeY6mjnGdubytTRRz3w=', 'Content-Length': '0'} 2023-05-31T20:48:11.889 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:43:08 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000009afe7115016c97f-006477b28b-1c17-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:48:11 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:48:11.889 DEBUG:boto:path=/pklxkd-59/obj68 2023-05-31T20:48:11.890 DEBUG:boto:auth_path=/pklxkd-59/obj68 2023-05-31T20:48:11.890 DEBUG:boto:Method: GET 2023-05-31T20:48:11.890 DEBUG:boto:Path: /pklxkd-59/obj68 2023-05-31T20:48:11.891 DEBUG:boto:Data: 2023-05-31T20:48:11.891 DEBUG:boto:Headers: {} 2023-05-31T20:48:11.891 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:48:11.892 DEBUG:boto:Port: 8000 2023-05-31T20:48:11.892 DEBUG:boto:Params: {} 2023-05-31T20:48:11.893 DEBUG:boto:Token: None 2023-05-31T20:48:11.893 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:48:11 GMT /pklxkd-59/obj68 2023-05-31T20:48:11.893 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:U+vdDHqwXeY6mjnGdubytTRRz3w= 2023-05-31T20:48:11.894 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:48:11 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:U+vdDHqwXeY6mjnGdubytTRRz3w=', 'Content-Length': '0'} 2023-05-31T20:48:11.896 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:43:08 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d2893d1215ef45a5-006477b28b-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:48:11 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:48:11.896 DEBUG:boto:path=/pklxkd-59/ 2023-05-31T20:48:11.897 DEBUG:boto:auth_path=/pklxkd-59/ 2023-05-31T20:48:11.897 DEBUG:boto:path=/pklxkd-59/?prefix=obj68 2023-05-31T20:48:11.897 DEBUG:boto:auth_path=/pklxkd-59/?prefix=obj68 2023-05-31T20:48:11.898 DEBUG:boto:Method: GET 2023-05-31T20:48:11.898 DEBUG:boto:Path: /pklxkd-59/?prefix=obj68 2023-05-31T20:48:11.898 DEBUG:boto:Data: 2023-05-31T20:48:11.899 DEBUG:boto:Headers: {} 2023-05-31T20:48:11.899 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:48:11.899 DEBUG:boto:Port: 8001 2023-05-31T20:48:11.900 DEBUG:boto:Params: {} 2023-05-31T20:48:11.900 DEBUG:boto:Token: None 2023-05-31T20:48:11.901 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:48:11 GMT /pklxkd-59/ 2023-05-31T20:48:11.901 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:l166NKUsN4SBNL48MAZina5NQhQ= 2023-05-31T20:48:11.901 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:48:11 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:l166NKUsN4SBNL48MAZina5NQhQ=', 'Content-Length': '0'} 2023-05-31T20:48:11.904 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000003a7a536b800cdeda-006477b28b-1c17-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:48:11 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:48:11.904 DEBUG:boto:b'pklxkd-59obj681000falseobj682023-05-31T20:43:08.066Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:48:11.905 DEBUG:boto:path=/pklxkd-59/ 2023-05-31T20:48:11.905 DEBUG:boto:auth_path=/pklxkd-59/ 2023-05-31T20:48:11.906 DEBUG:boto:path=/pklxkd-59/?prefix=obj68 2023-05-31T20:48:11.906 DEBUG:boto:auth_path=/pklxkd-59/?prefix=obj68 2023-05-31T20:48:11.906 DEBUG:boto:Method: GET 2023-05-31T20:48:11.907 DEBUG:boto:Path: /pklxkd-59/?prefix=obj68 2023-05-31T20:48:11.907 DEBUG:boto:Data: 2023-05-31T20:48:11.907 DEBUG:boto:Headers: {} 2023-05-31T20:48:11.908 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:48:11.908 DEBUG:boto:Port: 8000 2023-05-31T20:48:11.909 DEBUG:boto:Params: {} 2023-05-31T20:48:11.909 DEBUG:boto:Token: None 2023-05-31T20:48:11.909 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:48:11 GMT /pklxkd-59/ 2023-05-31T20:48:11.910 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:l166NKUsN4SBNL48MAZina5NQhQ= 2023-05-31T20:48:11.910 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:48:11 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:l166NKUsN4SBNL48MAZina5NQhQ=', 'Content-Length': '0'} 2023-05-31T20:48:11.913 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000001725644ed32406e-006477b28b-1630-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:48:11 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:48:11.914 DEBUG:boto:b'pklxkd-59obj681000falseobj682023-05-31T20:43:08.066Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:48:11.914 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj69 2023-05-31T20:48:11.915 DEBUG:boto:path=/pklxkd-59/obj69 2023-05-31T20:48:11.915 DEBUG:boto:auth_path=/pklxkd-59/obj69 2023-05-31T20:48:11.915 DEBUG:boto:path=/pklxkd-59/obj69?versionId=null 2023-05-31T20:48:11.916 DEBUG:boto:auth_path=/pklxkd-59/obj69?versionId=null 2023-05-31T20:48:11.916 DEBUG:boto:Method: GET 2023-05-31T20:48:11.916 DEBUG:boto:Path: /pklxkd-59/obj69?versionId=null 2023-05-31T20:48:11.917 DEBUG:boto:Data: 2023-05-31T20:48:11.917 DEBUG:boto:Headers: {} 2023-05-31T20:48:11.918 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:48:11.918 DEBUG:boto:Port: 8001 2023-05-31T20:48:11.918 DEBUG:boto:Params: {} 2023-05-31T20:48:11.919 DEBUG:boto:Token: None 2023-05-31T20:48:11.919 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:48:11 GMT /pklxkd-59/obj69?versionId=null 2023-05-31T20:48:11.919 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:eCNyqUeuP0nUZtZayGKtHABsK8s= 2023-05-31T20:48:11.920 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:48:11 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:eCNyqUeuP0nUZtZayGKtHABsK8s=', 'Content-Length': '0'} 2023-05-31T20:48:11.922 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:46:33 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000372196671facea9e-006477b28b-1c17-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:48:11 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:48:11.923 DEBUG:boto:path=/pklxkd-59/obj69 2023-05-31T20:48:11.923 DEBUG:boto:auth_path=/pklxkd-59/obj69 2023-05-31T20:48:11.923 DEBUG:boto:path=/pklxkd-59/obj69?versionId=null 2023-05-31T20:48:11.924 DEBUG:boto:auth_path=/pklxkd-59/obj69?versionId=null 2023-05-31T20:48:11.924 DEBUG:boto:Method: GET 2023-05-31T20:48:11.924 DEBUG:boto:Path: /pklxkd-59/obj69?versionId=null 2023-05-31T20:48:11.925 DEBUG:boto:Data: 2023-05-31T20:48:11.925 DEBUG:boto:Headers: {} 2023-05-31T20:48:11.926 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:48:11.926 DEBUG:boto:Port: 8000 2023-05-31T20:48:11.926 DEBUG:boto:Params: {} 2023-05-31T20:48:11.927 DEBUG:boto:Token: None 2023-05-31T20:48:11.927 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:48:11 GMT /pklxkd-59/obj69?versionId=null 2023-05-31T20:48:11.927 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:eCNyqUeuP0nUZtZayGKtHABsK8s= 2023-05-31T20:48:11.928 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:48:11 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:eCNyqUeuP0nUZtZayGKtHABsK8s=', 'Content-Length': '0'} 2023-05-31T20:48:11.930 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:46:33 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008e6f235b8b3a6897-006477b28b-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:48:11 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:48:11.930 DEBUG:boto:path=/pklxkd-59/obj69 2023-05-31T20:48:11.931 DEBUG:boto:auth_path=/pklxkd-59/obj69 2023-05-31T20:48:11.931 DEBUG:boto:path=/pklxkd-59/obj69?versionId=null 2023-05-31T20:48:11.931 DEBUG:boto:auth_path=/pklxkd-59/obj69?versionId=null 2023-05-31T20:48:11.932 DEBUG:boto:Method: HEAD 2023-05-31T20:48:11.932 DEBUG:boto:Path: /pklxkd-59/obj69?versionId=null 2023-05-31T20:48:11.932 DEBUG:boto:Data: 2023-05-31T20:48:11.933 DEBUG:boto:Headers: {} 2023-05-31T20:48:11.933 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:48:11.933 DEBUG:boto:Port: 8001 2023-05-31T20:48:11.934 DEBUG:boto:Params: {} 2023-05-31T20:48:11.934 DEBUG:boto:Token: None 2023-05-31T20:48:11.935 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:48:11 GMT /pklxkd-59/obj69?versionId=null 2023-05-31T20:48:11.935 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:XbXU9wNwD612oASvDPKiN/SHGHQ= 2023-05-31T20:48:11.935 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:48:11 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:XbXU9wNwD612oASvDPKiN/SHGHQ=', 'Content-Length': '0'} 2023-05-31T20:48:11.937 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:46:33 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000090ab61410450bfb9-006477b28b-1c17-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:48:11 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:48:11.938 DEBUG:boto:path=/pklxkd-59/obj69 2023-05-31T20:48:11.938 DEBUG:boto:auth_path=/pklxkd-59/obj69 2023-05-31T20:48:11.938 DEBUG:boto:path=/pklxkd-59/obj69?versionId=null 2023-05-31T20:48:11.939 DEBUG:boto:auth_path=/pklxkd-59/obj69?versionId=null 2023-05-31T20:48:11.939 DEBUG:boto:Method: HEAD 2023-05-31T20:48:11.939 DEBUG:boto:Path: /pklxkd-59/obj69?versionId=null 2023-05-31T20:48:11.940 DEBUG:boto:Data: 2023-05-31T20:48:11.940 DEBUG:boto:Headers: {} 2023-05-31T20:48:11.940 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:48:11.941 DEBUG:boto:Port: 8000 2023-05-31T20:48:11.941 DEBUG:boto:Params: {} 2023-05-31T20:48:11.941 DEBUG:boto:Token: None 2023-05-31T20:48:11.942 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:48:11 GMT /pklxkd-59/obj69?versionId=null 2023-05-31T20:48:11.942 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:XbXU9wNwD612oASvDPKiN/SHGHQ= 2023-05-31T20:48:11.942 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:48:11 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:XbXU9wNwD612oASvDPKiN/SHGHQ=', 'Content-Length': '0'} 2023-05-31T20:48:11.944 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:46:33 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000003791c6ffa93a65-006477b28b-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:48:11 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:48:11.945 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj69 2023-05-31T20:48:11.945 DEBUG:boto:path=/pklxkd-59/obj69 2023-05-31T20:48:11.946 DEBUG:boto:auth_path=/pklxkd-59/obj69 2023-05-31T20:48:11.946 DEBUG:boto:path=/pklxkd-59/obj69?versionId=null 2023-05-31T20:48:11.946 DEBUG:boto:auth_path=/pklxkd-59/obj69?versionId=null 2023-05-31T20:48:11.947 DEBUG:boto:Method: GET 2023-05-31T20:48:11.947 DEBUG:boto:Path: /pklxkd-59/obj69?versionId=null 2023-05-31T20:48:11.947 DEBUG:boto:Data: 2023-05-31T20:48:11.948 DEBUG:boto:Headers: {} 2023-05-31T20:48:11.948 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:48:11.948 DEBUG:boto:Port: 8001 2023-05-31T20:48:11.949 DEBUG:boto:Params: {} 2023-05-31T20:48:11.949 DEBUG:boto:Token: None 2023-05-31T20:48:11.950 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:48:11 GMT /pklxkd-59/obj69?versionId=null 2023-05-31T20:48:11.950 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:eCNyqUeuP0nUZtZayGKtHABsK8s= 2023-05-31T20:48:11.950 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:48:11 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:eCNyqUeuP0nUZtZayGKtHABsK8s=', 'Content-Length': '0'} 2023-05-31T20:48:11.952 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:46:33 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000467fa672b3140381-006477b28b-1c17-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:48:11 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:48:11.996 DEBUG:boto:path=/pklxkd-59/obj69 2023-05-31T20:48:11.997 DEBUG:boto:auth_path=/pklxkd-59/obj69 2023-05-31T20:48:11.997 DEBUG:boto:path=/pklxkd-59/obj69?versionId=null 2023-05-31T20:48:11.997 DEBUG:boto:auth_path=/pklxkd-59/obj69?versionId=null 2023-05-31T20:48:11.998 DEBUG:boto:Method: GET 2023-05-31T20:48:11.998 DEBUG:boto:Path: /pklxkd-59/obj69?versionId=null 2023-05-31T20:48:11.998 DEBUG:boto:Data: 2023-05-31T20:48:11.999 DEBUG:boto:Headers: {} 2023-05-31T20:48:11.999 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:48:11.999 DEBUG:boto:Port: 8000 2023-05-31T20:48:12.000 DEBUG:boto:Params: {} 2023-05-31T20:48:12.000 DEBUG:boto:Token: None 2023-05-31T20:48:12.001 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:48:12 GMT /pklxkd-59/obj69?versionId=null 2023-05-31T20:48:12.001 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:EWqMwMOk77bE0yi3zvIhsIKkyaI= 2023-05-31T20:48:12.001 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:48:12 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:EWqMwMOk77bE0yi3zvIhsIKkyaI=', 'Content-Length': '0'} 2023-05-31T20:48:12.003 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:46:33 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e34f54334d082b09-006477b28b-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:48:12 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:48:12.004 DEBUG:boto:path=/pklxkd-59/ 2023-05-31T20:48:12.004 DEBUG:boto:auth_path=/pklxkd-59/ 2023-05-31T20:48:12.005 DEBUG:boto:path=/pklxkd-59/?prefix=obj69 2023-05-31T20:48:12.005 DEBUG:boto:auth_path=/pklxkd-59/?prefix=obj69 2023-05-31T20:48:12.005 DEBUG:boto:Method: GET 2023-05-31T20:48:12.006 DEBUG:boto:Path: /pklxkd-59/?prefix=obj69 2023-05-31T20:48:12.006 DEBUG:boto:Data: 2023-05-31T20:48:12.006 DEBUG:boto:Headers: {} 2023-05-31T20:48:12.007 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:48:12.007 DEBUG:boto:Port: 8001 2023-05-31T20:48:12.008 DEBUG:boto:Params: {} 2023-05-31T20:48:12.008 DEBUG:boto:Token: None 2023-05-31T20:48:12.008 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:48:12 GMT /pklxkd-59/ 2023-05-31T20:48:12.009 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:7yviw5uwOHHeIaSmQ01+rDvoyyQ= 2023-05-31T20:48:12.009 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:48:12 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:7yviw5uwOHHeIaSmQ01+rDvoyyQ=', 'Content-Length': '0'} 2023-05-31T20:48:12.012 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000006f73f5948e31b4fd-006477b28c-1c17-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:48:12 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:48:12.012 DEBUG:boto:b'pklxkd-59obj691000falseobj692023-05-31T20:46:33.886Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:48:12.013 DEBUG:boto:path=/pklxkd-59/ 2023-05-31T20:48:12.013 DEBUG:boto:auth_path=/pklxkd-59/ 2023-05-31T20:48:12.013 DEBUG:boto:path=/pklxkd-59/?prefix=obj69 2023-05-31T20:48:12.014 DEBUG:boto:auth_path=/pklxkd-59/?prefix=obj69 2023-05-31T20:48:12.014 DEBUG:boto:Method: GET 2023-05-31T20:48:12.015 DEBUG:boto:Path: /pklxkd-59/?prefix=obj69 2023-05-31T20:48:12.015 DEBUG:boto:Data: 2023-05-31T20:48:12.015 DEBUG:boto:Headers: {} 2023-05-31T20:48:12.016 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:48:12.016 DEBUG:boto:Port: 8000 2023-05-31T20:48:12.016 DEBUG:boto:Params: {} 2023-05-31T20:48:12.017 DEBUG:boto:Token: None 2023-05-31T20:48:12.017 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:48:12 GMT /pklxkd-59/ 2023-05-31T20:48:12.018 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:7yviw5uwOHHeIaSmQ01+rDvoyyQ= 2023-05-31T20:48:12.018 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:48:12 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:7yviw5uwOHHeIaSmQ01+rDvoyyQ=', 'Content-Length': '0'} 2023-05-31T20:48:12.021 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000008b1628810b24127f-006477b28c-1630-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:48:12 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:48:12.022 DEBUG:boto:b'pklxkd-59obj691000falseobj692023-05-31T20:46:33.886Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:48:12.022 DEBUG:boto:path=/pklxkd-59/obj69 2023-05-31T20:48:12.023 DEBUG:boto:auth_path=/pklxkd-59/obj69 2023-05-31T20:48:12.023 DEBUG:boto:Method: HEAD 2023-05-31T20:48:12.023 DEBUG:boto:Path: /pklxkd-59/obj69 2023-05-31T20:48:12.024 DEBUG:boto:Data: 2023-05-31T20:48:12.024 DEBUG:boto:Headers: {} 2023-05-31T20:48:12.024 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:48:12.025 DEBUG:boto:Port: 8001 2023-05-31T20:48:12.025 DEBUG:boto:Params: {} 2023-05-31T20:48:12.025 DEBUG:boto:Token: None 2023-05-31T20:48:12.026 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:48:12 GMT /pklxkd-59/obj69 2023-05-31T20:48:12.026 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:1lDo7UboRLIbs6WE3Se6pWbPU1Y= 2023-05-31T20:48:12.027 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:48:12 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:1lDo7UboRLIbs6WE3Se6pWbPU1Y=', 'Content-Length': '0'} 2023-05-31T20:48:12.028 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:46:33 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ed29414faa5c54e1-006477b28c-1c17-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:48:12 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:48:12.029 DEBUG:boto:path=/pklxkd-59/obj69 2023-05-31T20:48:12.029 DEBUG:boto:auth_path=/pklxkd-59/obj69 2023-05-31T20:48:12.030 DEBUG:boto:Method: HEAD 2023-05-31T20:48:12.030 DEBUG:boto:Path: /pklxkd-59/obj69 2023-05-31T20:48:12.030 DEBUG:boto:Data: 2023-05-31T20:48:12.031 DEBUG:boto:Headers: {} 2023-05-31T20:48:12.031 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:48:12.031 DEBUG:boto:Port: 8000 2023-05-31T20:48:12.032 DEBUG:boto:Params: {} 2023-05-31T20:48:12.032 DEBUG:boto:Token: None 2023-05-31T20:48:12.032 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:48:12 GMT /pklxkd-59/obj69 2023-05-31T20:48:12.033 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:1lDo7UboRLIbs6WE3Se6pWbPU1Y= 2023-05-31T20:48:12.033 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:48:12 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:1lDo7UboRLIbs6WE3Se6pWbPU1Y=', 'Content-Length': '0'} 2023-05-31T20:48:12.035 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:46:33 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000092f8175defae3d77-006477b28c-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:48:12 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:48:12.036 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj69 2023-05-31T20:48:12.036 DEBUG:boto:path=/pklxkd-59/obj69 2023-05-31T20:48:12.036 DEBUG:boto:auth_path=/pklxkd-59/obj69 2023-05-31T20:48:12.037 DEBUG:boto:Method: GET 2023-05-31T20:48:12.037 DEBUG:boto:Path: /pklxkd-59/obj69 2023-05-31T20:48:12.038 DEBUG:boto:Data: 2023-05-31T20:48:12.038 DEBUG:boto:Headers: {} 2023-05-31T20:48:12.038 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:48:12.039 DEBUG:boto:Port: 8001 2023-05-31T20:48:12.039 DEBUG:boto:Params: {} 2023-05-31T20:48:12.039 DEBUG:boto:Token: None 2023-05-31T20:48:12.040 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:48:12 GMT /pklxkd-59/obj69 2023-05-31T20:48:12.040 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:Ws1wrgn96XYgZm1nr6Yzv/fjFwc= 2023-05-31T20:48:12.041 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:48:12 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:Ws1wrgn96XYgZm1nr6Yzv/fjFwc=', 'Content-Length': '0'} 2023-05-31T20:48:12.042 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:46:33 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ab5153bf235faf20-006477b28c-1c17-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:48:12 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:48:12.043 DEBUG:boto:path=/pklxkd-59/obj69 2023-05-31T20:48:12.043 DEBUG:boto:auth_path=/pklxkd-59/obj69 2023-05-31T20:48:12.044 DEBUG:boto:Method: GET 2023-05-31T20:48:12.044 DEBUG:boto:Path: /pklxkd-59/obj69 2023-05-31T20:48:12.045 DEBUG:boto:Data: 2023-05-31T20:48:12.045 DEBUG:boto:Headers: {} 2023-05-31T20:48:12.045 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:48:12.046 DEBUG:boto:Port: 8000 2023-05-31T20:48:12.046 DEBUG:boto:Params: {} 2023-05-31T20:48:12.046 DEBUG:boto:Token: None 2023-05-31T20:48:12.047 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:48:12 GMT /pklxkd-59/obj69 2023-05-31T20:48:12.047 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:Ws1wrgn96XYgZm1nr6Yzv/fjFwc= 2023-05-31T20:48:12.048 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:48:12 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:Ws1wrgn96XYgZm1nr6Yzv/fjFwc=', 'Content-Length': '0'} 2023-05-31T20:48:12.050 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:46:33 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000bef221f5196497f-006477b28c-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:48:12 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:48:12.092 DEBUG:boto:path=/pklxkd-59/ 2023-05-31T20:48:12.092 DEBUG:boto:auth_path=/pklxkd-59/ 2023-05-31T20:48:12.093 DEBUG:boto:path=/pklxkd-59/?prefix=obj69 2023-05-31T20:48:12.093 DEBUG:boto:auth_path=/pklxkd-59/?prefix=obj69 2023-05-31T20:48:12.094 DEBUG:boto:Method: GET 2023-05-31T20:48:12.094 DEBUG:boto:Path: /pklxkd-59/?prefix=obj69 2023-05-31T20:48:12.094 DEBUG:boto:Data: 2023-05-31T20:48:12.095 DEBUG:boto:Headers: {} 2023-05-31T20:48:12.095 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:48:12.095 DEBUG:boto:Port: 8001 2023-05-31T20:48:12.096 DEBUG:boto:Params: {} 2023-05-31T20:48:12.096 DEBUG:boto:Token: None 2023-05-31T20:48:12.096 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:48:12 GMT /pklxkd-59/ 2023-05-31T20:48:12.097 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:7yviw5uwOHHeIaSmQ01+rDvoyyQ= 2023-05-31T20:48:12.097 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:48:12 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:7yviw5uwOHHeIaSmQ01+rDvoyyQ=', 'Content-Length': '0'} 2023-05-31T20:48:12.100 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000004182200d0605348c-006477b28c-1c17-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:48:12 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:48:12.100 DEBUG:boto:b'pklxkd-59obj691000falseobj692023-05-31T20:46:33.886Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:48:12.101 DEBUG:boto:path=/pklxkd-59/ 2023-05-31T20:48:12.101 DEBUG:boto:auth_path=/pklxkd-59/ 2023-05-31T20:48:12.102 DEBUG:boto:path=/pklxkd-59/?prefix=obj69 2023-05-31T20:48:12.102 DEBUG:boto:auth_path=/pklxkd-59/?prefix=obj69 2023-05-31T20:48:12.102 DEBUG:boto:Method: GET 2023-05-31T20:48:12.103 DEBUG:boto:Path: /pklxkd-59/?prefix=obj69 2023-05-31T20:48:12.103 DEBUG:boto:Data: 2023-05-31T20:48:12.103 DEBUG:boto:Headers: {} 2023-05-31T20:48:12.104 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:48:12.104 DEBUG:boto:Port: 8000 2023-05-31T20:48:12.104 DEBUG:boto:Params: {} 2023-05-31T20:48:12.105 DEBUG:boto:Token: None 2023-05-31T20:48:12.105 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:48:12 GMT /pklxkd-59/ 2023-05-31T20:48:12.106 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:7yviw5uwOHHeIaSmQ01+rDvoyyQ= 2023-05-31T20:48:12.106 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:48:12 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:7yviw5uwOHHeIaSmQ01+rDvoyyQ=', 'Content-Length': '0'} 2023-05-31T20:48:12.109 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000abbd1c1dea103805-006477b28c-1630-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:48:12 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:48:12.110 DEBUG:boto:b'pklxkd-59obj691000falseobj692023-05-31T20:46:33.886Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:48:12.110 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj76 2023-05-31T20:48:12.110 DEBUG:boto:path=/pklxkd-59/obj76 2023-05-31T20:48:12.111 DEBUG:boto:auth_path=/pklxkd-59/obj76 2023-05-31T20:48:12.111 DEBUG:boto:path=/pklxkd-59/obj76?versionId=null 2023-05-31T20:48:12.112 DEBUG:boto:auth_path=/pklxkd-59/obj76?versionId=null 2023-05-31T20:48:12.112 DEBUG:boto:Method: GET 2023-05-31T20:48:12.112 DEBUG:boto:Path: /pklxkd-59/obj76?versionId=null 2023-05-31T20:48:12.113 DEBUG:boto:Data: 2023-05-31T20:48:12.113 DEBUG:boto:Headers: {} 2023-05-31T20:48:12.113 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:48:12.114 DEBUG:boto:Port: 8001 2023-05-31T20:48:12.114 DEBUG:boto:Params: {} 2023-05-31T20:48:12.114 DEBUG:boto:Token: None 2023-05-31T20:48:12.115 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:48:12 GMT /pklxkd-59/obj76?versionId=null 2023-05-31T20:48:12.115 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:+hSJ7c+lWNNEvSwPP523y3dYptc= 2023-05-31T20:48:12.116 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:48:12 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:+hSJ7c+lWNNEvSwPP523y3dYptc=', 'Content-Length': '0'} 2023-05-31T20:48:12.117 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:44:50 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000786dca4efe44ab9b-006477b28c-1c17-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:48:12 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:48:12.118 DEBUG:boto:path=/pklxkd-59/obj76 2023-05-31T20:48:12.118 DEBUG:boto:auth_path=/pklxkd-59/obj76 2023-05-31T20:48:12.119 DEBUG:boto:path=/pklxkd-59/obj76?versionId=null 2023-05-31T20:48:12.119 DEBUG:boto:auth_path=/pklxkd-59/obj76?versionId=null 2023-05-31T20:48:12.119 DEBUG:boto:Method: GET 2023-05-31T20:48:12.120 DEBUG:boto:Path: /pklxkd-59/obj76?versionId=null 2023-05-31T20:48:12.120 DEBUG:boto:Data: 2023-05-31T20:48:12.121 DEBUG:boto:Headers: {} 2023-05-31T20:48:12.121 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:48:12.121 DEBUG:boto:Port: 8000 2023-05-31T20:48:12.122 DEBUG:boto:Params: {} 2023-05-31T20:48:12.122 DEBUG:boto:Token: None 2023-05-31T20:48:12.122 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:48:12 GMT /pklxkd-59/obj76?versionId=null 2023-05-31T20:48:12.123 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:+hSJ7c+lWNNEvSwPP523y3dYptc= 2023-05-31T20:48:12.123 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:48:12 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:+hSJ7c+lWNNEvSwPP523y3dYptc=', 'Content-Length': '0'} 2023-05-31T20:48:12.125 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:44:50 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000bc8ea8cb819fbbec-006477b28c-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:48:12 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:48:12.126 DEBUG:boto:path=/pklxkd-59/obj76 2023-05-31T20:48:12.126 DEBUG:boto:auth_path=/pklxkd-59/obj76 2023-05-31T20:48:12.126 DEBUG:boto:path=/pklxkd-59/obj76?versionId=null 2023-05-31T20:48:12.127 DEBUG:boto:auth_path=/pklxkd-59/obj76?versionId=null 2023-05-31T20:48:12.127 DEBUG:boto:Method: HEAD 2023-05-31T20:48:12.127 DEBUG:boto:Path: /pklxkd-59/obj76?versionId=null 2023-05-31T20:48:12.128 DEBUG:boto:Data: 2023-05-31T20:48:12.128 DEBUG:boto:Headers: {} 2023-05-31T20:48:12.128 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:48:12.129 DEBUG:boto:Port: 8001 2023-05-31T20:48:12.129 DEBUG:boto:Params: {} 2023-05-31T20:48:12.129 DEBUG:boto:Token: None 2023-05-31T20:48:12.130 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:48:12 GMT /pklxkd-59/obj76?versionId=null 2023-05-31T20:48:12.130 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:LWbacJbW4G6ktoR8adrDOvV+RXo= 2023-05-31T20:48:12.130 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:48:12 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:LWbacJbW4G6ktoR8adrDOvV+RXo=', 'Content-Length': '0'} 2023-05-31T20:48:12.132 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:44:50 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000a6f4cfaf890fde8-006477b28c-1c17-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:48:12 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:48:12.133 DEBUG:boto:path=/pklxkd-59/obj76 2023-05-31T20:48:12.133 DEBUG:boto:auth_path=/pklxkd-59/obj76 2023-05-31T20:48:12.133 DEBUG:boto:path=/pklxkd-59/obj76?versionId=null 2023-05-31T20:48:12.134 DEBUG:boto:auth_path=/pklxkd-59/obj76?versionId=null 2023-05-31T20:48:12.134 DEBUG:boto:Method: HEAD 2023-05-31T20:48:12.134 DEBUG:boto:Path: /pklxkd-59/obj76?versionId=null 2023-05-31T20:48:12.135 DEBUG:boto:Data: 2023-05-31T20:48:12.135 DEBUG:boto:Headers: {} 2023-05-31T20:48:12.135 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:48:12.136 DEBUG:boto:Port: 8000 2023-05-31T20:48:12.136 DEBUG:boto:Params: {} 2023-05-31T20:48:12.136 DEBUG:boto:Token: None 2023-05-31T20:48:12.137 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:48:12 GMT /pklxkd-59/obj76?versionId=null 2023-05-31T20:48:12.137 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:LWbacJbW4G6ktoR8adrDOvV+RXo= 2023-05-31T20:48:12.138 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:48:12 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:LWbacJbW4G6ktoR8adrDOvV+RXo=', 'Content-Length': '0'} 2023-05-31T20:48:12.139 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:44:50 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000003cae04169665b53-006477b28c-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:48:12 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:48:12.140 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj76 2023-05-31T20:48:12.140 DEBUG:boto:path=/pklxkd-59/obj76 2023-05-31T20:48:12.141 DEBUG:boto:auth_path=/pklxkd-59/obj76 2023-05-31T20:48:12.141 DEBUG:boto:path=/pklxkd-59/obj76?versionId=null 2023-05-31T20:48:12.141 DEBUG:boto:auth_path=/pklxkd-59/obj76?versionId=null 2023-05-31T20:48:12.142 DEBUG:boto:Method: GET 2023-05-31T20:48:12.142 DEBUG:boto:Path: /pklxkd-59/obj76?versionId=null 2023-05-31T20:48:12.142 DEBUG:boto:Data: 2023-05-31T20:48:12.143 DEBUG:boto:Headers: {} 2023-05-31T20:48:12.143 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:48:12.143 DEBUG:boto:Port: 8001 2023-05-31T20:48:12.144 DEBUG:boto:Params: {} 2023-05-31T20:48:12.144 DEBUG:boto:Token: None 2023-05-31T20:48:12.145 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:48:12 GMT /pklxkd-59/obj76?versionId=null 2023-05-31T20:48:12.145 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:+hSJ7c+lWNNEvSwPP523y3dYptc= 2023-05-31T20:48:12.145 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:48:12 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:+hSJ7c+lWNNEvSwPP523y3dYptc=', 'Content-Length': '0'} 2023-05-31T20:48:12.147 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:44:50 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005501d74183b01b39-006477b28c-1c17-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:48:12 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:48:12.188 DEBUG:boto:path=/pklxkd-59/obj76 2023-05-31T20:48:12.189 DEBUG:boto:auth_path=/pklxkd-59/obj76 2023-05-31T20:48:12.189 DEBUG:boto:path=/pklxkd-59/obj76?versionId=null 2023-05-31T20:48:12.189 DEBUG:boto:auth_path=/pklxkd-59/obj76?versionId=null 2023-05-31T20:48:12.190 DEBUG:boto:Method: GET 2023-05-31T20:48:12.190 DEBUG:boto:Path: /pklxkd-59/obj76?versionId=null 2023-05-31T20:48:12.190 DEBUG:boto:Data: 2023-05-31T20:48:12.191 DEBUG:boto:Headers: {} 2023-05-31T20:48:12.191 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:48:12.191 DEBUG:boto:Port: 8000 2023-05-31T20:48:12.192 DEBUG:boto:Params: {} 2023-05-31T20:48:12.192 DEBUG:boto:Token: None 2023-05-31T20:48:12.193 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:48:12 GMT /pklxkd-59/obj76?versionId=null 2023-05-31T20:48:12.193 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:+hSJ7c+lWNNEvSwPP523y3dYptc= 2023-05-31T20:48:12.193 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:48:12 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:+hSJ7c+lWNNEvSwPP523y3dYptc=', 'Content-Length': '0'} 2023-05-31T20:48:12.195 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:44:50 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000cf93c6f31f8940a6-006477b28c-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:48:12 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:48:12.196 DEBUG:boto:path=/pklxkd-59/ 2023-05-31T20:48:12.196 DEBUG:boto:auth_path=/pklxkd-59/ 2023-05-31T20:48:12.196 DEBUG:boto:path=/pklxkd-59/?prefix=obj76 2023-05-31T20:48:12.197 DEBUG:boto:auth_path=/pklxkd-59/?prefix=obj76 2023-05-31T20:48:12.197 DEBUG:boto:Method: GET 2023-05-31T20:48:12.197 DEBUG:boto:Path: /pklxkd-59/?prefix=obj76 2023-05-31T20:48:12.198 DEBUG:boto:Data: 2023-05-31T20:48:12.198 DEBUG:boto:Headers: {} 2023-05-31T20:48:12.199 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:48:12.199 DEBUG:boto:Port: 8001 2023-05-31T20:48:12.199 DEBUG:boto:Params: {} 2023-05-31T20:48:12.200 DEBUG:boto:Token: None 2023-05-31T20:48:12.200 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:48:12 GMT /pklxkd-59/ 2023-05-31T20:48:12.200 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:7yviw5uwOHHeIaSmQ01+rDvoyyQ= 2023-05-31T20:48:12.201 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:48:12 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:7yviw5uwOHHeIaSmQ01+rDvoyyQ=', 'Content-Length': '0'} 2023-05-31T20:48:12.203 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000001b2638cd681df166-006477b28c-1c17-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:48:12 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:48:12.203 DEBUG:boto:b'pklxkd-59obj761000falseobj762023-05-31T20:44:50.682Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:48:12.204 DEBUG:boto:path=/pklxkd-59/ 2023-05-31T20:48:12.205 DEBUG:boto:auth_path=/pklxkd-59/ 2023-05-31T20:48:12.205 DEBUG:boto:path=/pklxkd-59/?prefix=obj76 2023-05-31T20:48:12.205 DEBUG:boto:auth_path=/pklxkd-59/?prefix=obj76 2023-05-31T20:48:12.206 DEBUG:boto:Method: GET 2023-05-31T20:48:12.206 DEBUG:boto:Path: /pklxkd-59/?prefix=obj76 2023-05-31T20:48:12.206 DEBUG:boto:Data: 2023-05-31T20:48:12.207 DEBUG:boto:Headers: {} 2023-05-31T20:48:12.207 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:48:12.207 DEBUG:boto:Port: 8000 2023-05-31T20:48:12.208 DEBUG:boto:Params: {} 2023-05-31T20:48:12.208 DEBUG:boto:Token: None 2023-05-31T20:48:12.209 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:48:12 GMT /pklxkd-59/ 2023-05-31T20:48:12.209 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:7yviw5uwOHHeIaSmQ01+rDvoyyQ= 2023-05-31T20:48:12.209 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:48:12 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:7yviw5uwOHHeIaSmQ01+rDvoyyQ=', 'Content-Length': '0'} 2023-05-31T20:48:12.212 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000046f3151de613c40b-006477b28c-1630-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:48:12 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:48:12.213 DEBUG:boto:b'pklxkd-59obj761000falseobj762023-05-31T20:44:50.682Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:48:12.213 DEBUG:boto:path=/pklxkd-59/obj76 2023-05-31T20:48:12.214 DEBUG:boto:auth_path=/pklxkd-59/obj76 2023-05-31T20:48:12.214 DEBUG:boto:Method: HEAD 2023-05-31T20:48:12.215 DEBUG:boto:Path: /pklxkd-59/obj76 2023-05-31T20:48:12.215 DEBUG:boto:Data: 2023-05-31T20:48:12.215 DEBUG:boto:Headers: {} 2023-05-31T20:48:12.216 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:48:12.216 DEBUG:boto:Port: 8001 2023-05-31T20:48:12.216 DEBUG:boto:Params: {} 2023-05-31T20:48:12.217 DEBUG:boto:Token: None 2023-05-31T20:48:12.217 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:48:12 GMT /pklxkd-59/obj76 2023-05-31T20:48:12.217 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:5OIUTlQCe/mfRo8sGZbHSKbXC/w= 2023-05-31T20:48:12.218 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:48:12 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:5OIUTlQCe/mfRo8sGZbHSKbXC/w=', 'Content-Length': '0'} 2023-05-31T20:48:12.220 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:44:50 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002d9252150c4d2b1c-006477b28c-1c17-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:48:12 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:48:12.220 DEBUG:boto:path=/pklxkd-59/obj76 2023-05-31T20:48:12.221 DEBUG:boto:auth_path=/pklxkd-59/obj76 2023-05-31T20:48:12.221 DEBUG:boto:Method: HEAD 2023-05-31T20:48:12.221 DEBUG:boto:Path: /pklxkd-59/obj76 2023-05-31T20:48:12.222 DEBUG:boto:Data: 2023-05-31T20:48:12.222 DEBUG:boto:Headers: {} 2023-05-31T20:48:12.222 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:48:12.223 DEBUG:boto:Port: 8000 2023-05-31T20:48:12.223 DEBUG:boto:Params: {} 2023-05-31T20:48:12.223 DEBUG:boto:Token: None 2023-05-31T20:48:12.224 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:48:12 GMT /pklxkd-59/obj76 2023-05-31T20:48:12.224 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:5OIUTlQCe/mfRo8sGZbHSKbXC/w= 2023-05-31T20:48:12.224 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:48:12 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:5OIUTlQCe/mfRo8sGZbHSKbXC/w=', 'Content-Length': '0'} 2023-05-31T20:48:12.226 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:44:50 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000050e8a326eebd77dc-006477b28c-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:48:12 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:48:12.226 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj76 2023-05-31T20:48:12.227 DEBUG:boto:path=/pklxkd-59/obj76 2023-05-31T20:48:12.227 DEBUG:boto:auth_path=/pklxkd-59/obj76 2023-05-31T20:48:12.228 DEBUG:boto:Method: GET 2023-05-31T20:48:12.228 DEBUG:boto:Path: /pklxkd-59/obj76 2023-05-31T20:48:12.228 DEBUG:boto:Data: 2023-05-31T20:48:12.229 DEBUG:boto:Headers: {} 2023-05-31T20:48:12.229 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:48:12.229 DEBUG:boto:Port: 8001 2023-05-31T20:48:12.230 DEBUG:boto:Params: {} 2023-05-31T20:48:12.230 DEBUG:boto:Token: None 2023-05-31T20:48:12.231 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:48:12 GMT /pklxkd-59/obj76 2023-05-31T20:48:12.231 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:EjDDFkBYMlToyjXZH8lAG64Wj5o= 2023-05-31T20:48:12.231 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:48:12 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:EjDDFkBYMlToyjXZH8lAG64Wj5o=', 'Content-Length': '0'} 2023-05-31T20:48:12.233 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:44:50 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000457f409134d42acb-006477b28c-1c17-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:48:12 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:48:12.234 DEBUG:boto:path=/pklxkd-59/obj76 2023-05-31T20:48:12.234 DEBUG:boto:auth_path=/pklxkd-59/obj76 2023-05-31T20:48:12.235 DEBUG:boto:Method: GET 2023-05-31T20:48:12.235 DEBUG:boto:Path: /pklxkd-59/obj76 2023-05-31T20:48:12.235 DEBUG:boto:Data: 2023-05-31T20:48:12.236 DEBUG:boto:Headers: {} 2023-05-31T20:48:12.236 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:48:12.236 DEBUG:boto:Port: 8000 2023-05-31T20:48:12.237 DEBUG:boto:Params: {} 2023-05-31T20:48:12.237 DEBUG:boto:Token: None 2023-05-31T20:48:12.237 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:48:12 GMT /pklxkd-59/obj76 2023-05-31T20:48:12.238 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:EjDDFkBYMlToyjXZH8lAG64Wj5o= 2023-05-31T20:48:12.238 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:48:12 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:EjDDFkBYMlToyjXZH8lAG64Wj5o=', 'Content-Length': '0'} 2023-05-31T20:48:12.240 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:44:50 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000094ecbc3030bd9bd2-006477b28c-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:48:12 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:48:12.241 DEBUG:boto:path=/pklxkd-59/ 2023-05-31T20:48:12.241 DEBUG:boto:auth_path=/pklxkd-59/ 2023-05-31T20:48:12.241 DEBUG:boto:path=/pklxkd-59/?prefix=obj76 2023-05-31T20:48:12.242 DEBUG:boto:auth_path=/pklxkd-59/?prefix=obj76 2023-05-31T20:48:12.242 DEBUG:boto:Method: GET 2023-05-31T20:48:12.242 DEBUG:boto:Path: /pklxkd-59/?prefix=obj76 2023-05-31T20:48:12.243 DEBUG:boto:Data: 2023-05-31T20:48:12.243 DEBUG:boto:Headers: {} 2023-05-31T20:48:12.243 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:48:12.244 DEBUG:boto:Port: 8001 2023-05-31T20:48:12.244 DEBUG:boto:Params: {} 2023-05-31T20:48:12.244 DEBUG:boto:Token: None 2023-05-31T20:48:12.245 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:48:12 GMT /pklxkd-59/ 2023-05-31T20:48:12.245 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:7yviw5uwOHHeIaSmQ01+rDvoyyQ= 2023-05-31T20:48:12.246 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:48:12 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:7yviw5uwOHHeIaSmQ01+rDvoyyQ=', 'Content-Length': '0'} 2023-05-31T20:48:12.248 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000076092910ceada21-006477b28c-1c17-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:48:12 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:48:12.249 DEBUG:boto:b'pklxkd-59obj761000falseobj762023-05-31T20:44:50.682Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:48:12.249 DEBUG:boto:path=/pklxkd-59/ 2023-05-31T20:48:12.250 DEBUG:boto:auth_path=/pklxkd-59/ 2023-05-31T20:48:12.250 DEBUG:boto:path=/pklxkd-59/?prefix=obj76 2023-05-31T20:48:12.250 DEBUG:boto:auth_path=/pklxkd-59/?prefix=obj76 2023-05-31T20:48:12.251 DEBUG:boto:Method: GET 2023-05-31T20:48:12.251 DEBUG:boto:Path: /pklxkd-59/?prefix=obj76 2023-05-31T20:48:12.251 DEBUG:boto:Data: 2023-05-31T20:48:12.252 DEBUG:boto:Headers: {} 2023-05-31T20:48:12.252 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:48:12.252 DEBUG:boto:Port: 8000 2023-05-31T20:48:12.253 DEBUG:boto:Params: {} 2023-05-31T20:48:12.253 DEBUG:boto:Token: None 2023-05-31T20:48:12.254 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:48:12 GMT /pklxkd-59/ 2023-05-31T20:48:12.254 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:7yviw5uwOHHeIaSmQ01+rDvoyyQ= 2023-05-31T20:48:12.254 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:48:12 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:7yviw5uwOHHeIaSmQ01+rDvoyyQ=', 'Content-Length': '0'} 2023-05-31T20:48:12.257 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000084df21d616d29db1-006477b28c-1630-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:48:12 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:48:12.258 DEBUG:boto:b'pklxkd-59obj761000falseobj762023-05-31T20:44:50.682Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:48:12.259 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj85 2023-05-31T20:48:12.259 DEBUG:boto:path=/pklxkd-59/obj85 2023-05-31T20:48:12.259 DEBUG:boto:auth_path=/pklxkd-59/obj85 2023-05-31T20:48:12.260 DEBUG:boto:path=/pklxkd-59/obj85?versionId=null 2023-05-31T20:48:12.260 DEBUG:boto:auth_path=/pklxkd-59/obj85?versionId=null 2023-05-31T20:48:12.260 DEBUG:boto:Method: GET 2023-05-31T20:48:12.261 DEBUG:boto:Path: /pklxkd-59/obj85?versionId=null 2023-05-31T20:48:12.261 DEBUG:boto:Data: 2023-05-31T20:48:12.261 DEBUG:boto:Headers: {} 2023-05-31T20:48:12.262 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:48:12.262 DEBUG:boto:Port: 8001 2023-05-31T20:48:12.263 DEBUG:boto:Params: {} 2023-05-31T20:48:12.263 DEBUG:boto:Token: None 2023-05-31T20:48:12.263 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:48:12 GMT /pklxkd-59/obj85?versionId=null 2023-05-31T20:48:12.264 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:nRGB1Ze7X/GYkXhAvnkcnUXdVGA= 2023-05-31T20:48:12.264 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:48:12 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:nRGB1Ze7X/GYkXhAvnkcnUXdVGA=', 'Content-Length': '0'} 2023-05-31T20:48:12.266 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:43:08 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000fa3fd94f02addc3-006477b28c-1c17-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:48:12 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:48:12.308 DEBUG:boto:path=/pklxkd-59/obj85 2023-05-31T20:48:12.309 DEBUG:boto:auth_path=/pklxkd-59/obj85 2023-05-31T20:48:12.309 DEBUG:boto:path=/pklxkd-59/obj85?versionId=null 2023-05-31T20:48:12.309 DEBUG:boto:auth_path=/pklxkd-59/obj85?versionId=null 2023-05-31T20:48:12.310 DEBUG:boto:Method: GET 2023-05-31T20:48:12.310 DEBUG:boto:Path: /pklxkd-59/obj85?versionId=null 2023-05-31T20:48:12.310 DEBUG:boto:Data: 2023-05-31T20:48:12.311 DEBUG:boto:Headers: {} 2023-05-31T20:48:12.311 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:48:12.311 DEBUG:boto:Port: 8000 2023-05-31T20:48:12.312 DEBUG:boto:Params: {} 2023-05-31T20:48:12.312 DEBUG:boto:Token: None 2023-05-31T20:48:12.313 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:48:12 GMT /pklxkd-59/obj85?versionId=null 2023-05-31T20:48:12.313 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:nRGB1Ze7X/GYkXhAvnkcnUXdVGA= 2023-05-31T20:48:12.313 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:48:12 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:nRGB1Ze7X/GYkXhAvnkcnUXdVGA=', 'Content-Length': '0'} 2023-05-31T20:48:12.315 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:43:08 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003fb6591aba4f5774-006477b28c-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:48:12 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:48:12.316 DEBUG:boto:path=/pklxkd-59/obj85 2023-05-31T20:48:12.316 DEBUG:boto:auth_path=/pklxkd-59/obj85 2023-05-31T20:48:12.317 DEBUG:boto:path=/pklxkd-59/obj85?versionId=null 2023-05-31T20:48:12.317 DEBUG:boto:auth_path=/pklxkd-59/obj85?versionId=null 2023-05-31T20:48:12.317 DEBUG:boto:Method: HEAD 2023-05-31T20:48:12.318 DEBUG:boto:Path: /pklxkd-59/obj85?versionId=null 2023-05-31T20:48:12.318 DEBUG:boto:Data: 2023-05-31T20:48:12.318 DEBUG:boto:Headers: {} 2023-05-31T20:48:12.319 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:48:12.319 DEBUG:boto:Port: 8001 2023-05-31T20:48:12.319 DEBUG:boto:Params: {} 2023-05-31T20:48:12.320 DEBUG:boto:Token: None 2023-05-31T20:48:12.320 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:48:12 GMT /pklxkd-59/obj85?versionId=null 2023-05-31T20:48:12.320 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:9sQS7hZTsPUxUYBdlIjw5QINmsU= 2023-05-31T20:48:12.321 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:48:12 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:9sQS7hZTsPUxUYBdlIjw5QINmsU=', 'Content-Length': '0'} 2023-05-31T20:48:12.322 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:43:08 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f86a24549f49d084-006477b28c-1c17-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:48:12 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:48:12.323 DEBUG:boto:path=/pklxkd-59/obj85 2023-05-31T20:48:12.323 DEBUG:boto:auth_path=/pklxkd-59/obj85 2023-05-31T20:48:12.324 DEBUG:boto:path=/pklxkd-59/obj85?versionId=null 2023-05-31T20:48:12.324 DEBUG:boto:auth_path=/pklxkd-59/obj85?versionId=null 2023-05-31T20:48:12.324 DEBUG:boto:Method: HEAD 2023-05-31T20:48:12.325 DEBUG:boto:Path: /pklxkd-59/obj85?versionId=null 2023-05-31T20:48:12.325 DEBUG:boto:Data: 2023-05-31T20:48:12.325 DEBUG:boto:Headers: {} 2023-05-31T20:48:12.326 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:48:12.326 DEBUG:boto:Port: 8000 2023-05-31T20:48:12.326 DEBUG:boto:Params: {} 2023-05-31T20:48:12.327 DEBUG:boto:Token: None 2023-05-31T20:48:12.327 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:48:12 GMT /pklxkd-59/obj85?versionId=null 2023-05-31T20:48:12.327 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:9sQS7hZTsPUxUYBdlIjw5QINmsU= 2023-05-31T20:48:12.328 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:48:12 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:9sQS7hZTsPUxUYBdlIjw5QINmsU=', 'Content-Length': '0'} 2023-05-31T20:48:12.330 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:43:08 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000004b239a70f0565bd-006477b28c-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:48:12 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:48:12.330 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj85 2023-05-31T20:48:12.330 DEBUG:boto:path=/pklxkd-59/obj85 2023-05-31T20:48:12.331 DEBUG:boto:auth_path=/pklxkd-59/obj85 2023-05-31T20:48:12.331 DEBUG:boto:path=/pklxkd-59/obj85?versionId=null 2023-05-31T20:48:12.332 DEBUG:boto:auth_path=/pklxkd-59/obj85?versionId=null 2023-05-31T20:48:12.332 DEBUG:boto:Method: GET 2023-05-31T20:48:12.332 DEBUG:boto:Path: /pklxkd-59/obj85?versionId=null 2023-05-31T20:48:12.333 DEBUG:boto:Data: 2023-05-31T20:48:12.333 DEBUG:boto:Headers: {} 2023-05-31T20:48:12.333 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:48:12.334 DEBUG:boto:Port: 8001 2023-05-31T20:48:12.334 DEBUG:boto:Params: {} 2023-05-31T20:48:12.334 DEBUG:boto:Token: None 2023-05-31T20:48:12.335 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:48:12 GMT /pklxkd-59/obj85?versionId=null 2023-05-31T20:48:12.335 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:nRGB1Ze7X/GYkXhAvnkcnUXdVGA= 2023-05-31T20:48:12.336 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:48:12 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:nRGB1Ze7X/GYkXhAvnkcnUXdVGA=', 'Content-Length': '0'} 2023-05-31T20:48:12.338 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:43:08 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006c030b22c7959681-006477b28c-1c17-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:48:12 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:48:12.338 DEBUG:boto:path=/pklxkd-59/obj85 2023-05-31T20:48:12.339 DEBUG:boto:auth_path=/pklxkd-59/obj85 2023-05-31T20:48:12.339 DEBUG:boto:path=/pklxkd-59/obj85?versionId=null 2023-05-31T20:48:12.339 DEBUG:boto:auth_path=/pklxkd-59/obj85?versionId=null 2023-05-31T20:48:12.340 DEBUG:boto:Method: GET 2023-05-31T20:48:12.340 DEBUG:boto:Path: /pklxkd-59/obj85?versionId=null 2023-05-31T20:48:12.341 DEBUG:boto:Data: 2023-05-31T20:48:12.341 DEBUG:boto:Headers: {} 2023-05-31T20:48:12.341 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:48:12.342 DEBUG:boto:Port: 8000 2023-05-31T20:48:12.342 DEBUG:boto:Params: {} 2023-05-31T20:48:12.342 DEBUG:boto:Token: None 2023-05-31T20:48:12.343 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:48:12 GMT /pklxkd-59/obj85?versionId=null 2023-05-31T20:48:12.343 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:nRGB1Ze7X/GYkXhAvnkcnUXdVGA= 2023-05-31T20:48:12.344 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:48:12 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:nRGB1Ze7X/GYkXhAvnkcnUXdVGA=', 'Content-Length': '0'} 2023-05-31T20:48:12.345 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:43:08 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000da5c538ddd754184-006477b28c-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:48:12 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:48:12.346 DEBUG:boto:path=/pklxkd-59/ 2023-05-31T20:48:12.346 DEBUG:boto:auth_path=/pklxkd-59/ 2023-05-31T20:48:12.347 DEBUG:boto:path=/pklxkd-59/?prefix=obj85 2023-05-31T20:48:12.347 DEBUG:boto:auth_path=/pklxkd-59/?prefix=obj85 2023-05-31T20:48:12.347 DEBUG:boto:Method: GET 2023-05-31T20:48:12.348 DEBUG:boto:Path: /pklxkd-59/?prefix=obj85 2023-05-31T20:48:12.348 DEBUG:boto:Data: 2023-05-31T20:48:12.349 DEBUG:boto:Headers: {} 2023-05-31T20:48:12.349 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:48:12.349 DEBUG:boto:Port: 8001 2023-05-31T20:48:12.350 DEBUG:boto:Params: {} 2023-05-31T20:48:12.350 DEBUG:boto:Token: None 2023-05-31T20:48:12.350 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:48:12 GMT /pklxkd-59/ 2023-05-31T20:48:12.351 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:7yviw5uwOHHeIaSmQ01+rDvoyyQ= 2023-05-31T20:48:12.351 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:48:12 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:7yviw5uwOHHeIaSmQ01+rDvoyyQ=', 'Content-Length': '0'} 2023-05-31T20:48:12.353 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c1aa185c05a575bb-006477b28c-1c17-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:48:12 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:48:12.354 DEBUG:boto:b'pklxkd-59obj851000falseobj852023-05-31T20:43:08.086Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:48:12.355 DEBUG:boto:path=/pklxkd-59/ 2023-05-31T20:48:12.355 DEBUG:boto:auth_path=/pklxkd-59/ 2023-05-31T20:48:12.355 DEBUG:boto:path=/pklxkd-59/?prefix=obj85 2023-05-31T20:48:12.356 DEBUG:boto:auth_path=/pklxkd-59/?prefix=obj85 2023-05-31T20:48:12.356 DEBUG:boto:Method: GET 2023-05-31T20:48:12.356 DEBUG:boto:Path: /pklxkd-59/?prefix=obj85 2023-05-31T20:48:12.357 DEBUG:boto:Data: 2023-05-31T20:48:12.357 DEBUG:boto:Headers: {} 2023-05-31T20:48:12.357 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:48:12.358 DEBUG:boto:Port: 8000 2023-05-31T20:48:12.358 DEBUG:boto:Params: {} 2023-05-31T20:48:12.359 DEBUG:boto:Token: None 2023-05-31T20:48:12.359 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:48:12 GMT /pklxkd-59/ 2023-05-31T20:48:12.359 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:7yviw5uwOHHeIaSmQ01+rDvoyyQ= 2023-05-31T20:48:12.360 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:48:12 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:7yviw5uwOHHeIaSmQ01+rDvoyyQ=', 'Content-Length': '0'} 2023-05-31T20:48:12.363 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000084a5de8ab8d58723-006477b28c-1630-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:48:12 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:48:12.363 DEBUG:boto:b'pklxkd-59obj851000falseobj852023-05-31T20:43:08.086Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:48:12.364 DEBUG:boto:path=/pklxkd-59/obj85 2023-05-31T20:48:12.364 DEBUG:boto:auth_path=/pklxkd-59/obj85 2023-05-31T20:48:12.365 DEBUG:boto:Method: HEAD 2023-05-31T20:48:12.365 DEBUG:boto:Path: /pklxkd-59/obj85 2023-05-31T20:48:12.365 DEBUG:boto:Data: 2023-05-31T20:48:12.365 DEBUG:boto:Headers: {} 2023-05-31T20:48:12.366 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:48:12.366 DEBUG:boto:Port: 8001 2023-05-31T20:48:12.366 DEBUG:boto:Params: {} 2023-05-31T20:48:12.367 DEBUG:boto:Token: None 2023-05-31T20:48:12.367 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:48:12 GMT /pklxkd-59/obj85 2023-05-31T20:48:12.368 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:9armA8Gy3zatTVk9+cRReyKNLVs= 2023-05-31T20:48:12.368 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:48:12 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:9armA8Gy3zatTVk9+cRReyKNLVs=', 'Content-Length': '0'} 2023-05-31T20:48:12.370 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:43:08 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002957995246c7f567-006477b28c-1c17-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:48:12 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:48:12.370 DEBUG:boto:path=/pklxkd-59/obj85 2023-05-31T20:48:12.371 DEBUG:boto:auth_path=/pklxkd-59/obj85 2023-05-31T20:48:12.371 DEBUG:boto:Method: HEAD 2023-05-31T20:48:12.371 DEBUG:boto:Path: /pklxkd-59/obj85 2023-05-31T20:48:12.372 DEBUG:boto:Data: 2023-05-31T20:48:12.372 DEBUG:boto:Headers: {} 2023-05-31T20:48:12.372 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:48:12.373 DEBUG:boto:Port: 8000 2023-05-31T20:48:12.373 DEBUG:boto:Params: {} 2023-05-31T20:48:12.373 DEBUG:boto:Token: None 2023-05-31T20:48:12.374 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:48:12 GMT /pklxkd-59/obj85 2023-05-31T20:48:12.374 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:9armA8Gy3zatTVk9+cRReyKNLVs= 2023-05-31T20:48:12.375 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:48:12 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:9armA8Gy3zatTVk9+cRReyKNLVs=', 'Content-Length': '0'} 2023-05-31T20:48:12.376 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:43:08 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ada744afca82aa30-006477b28c-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:48:12 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:48:12.377 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj85 2023-05-31T20:48:12.377 DEBUG:boto:path=/pklxkd-59/obj85 2023-05-31T20:48:12.377 DEBUG:boto:auth_path=/pklxkd-59/obj85 2023-05-31T20:48:12.378 DEBUG:boto:Method: GET 2023-05-31T20:48:12.378 DEBUG:boto:Path: /pklxkd-59/obj85 2023-05-31T20:48:12.379 DEBUG:boto:Data: 2023-05-31T20:48:12.379 DEBUG:boto:Headers: {} 2023-05-31T20:48:12.379 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:48:12.380 DEBUG:boto:Port: 8001 2023-05-31T20:48:12.380 DEBUG:boto:Params: {} 2023-05-31T20:48:12.380 DEBUG:boto:Token: None 2023-05-31T20:48:12.381 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:48:12 GMT /pklxkd-59/obj85 2023-05-31T20:48:12.381 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:DxQZPNwDsB7Sk7/grs8yzd5jLK4= 2023-05-31T20:48:12.382 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:48:12 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:DxQZPNwDsB7Sk7/grs8yzd5jLK4=', 'Content-Length': '0'} 2023-05-31T20:48:12.384 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:43:08 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d3c65ecc39dd7654-006477b28c-1c17-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:48:12 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:48:12.384 DEBUG:boto:path=/pklxkd-59/obj85 2023-05-31T20:48:12.385 DEBUG:boto:auth_path=/pklxkd-59/obj85 2023-05-31T20:48:12.385 DEBUG:boto:Method: GET 2023-05-31T20:48:12.385 DEBUG:boto:Path: /pklxkd-59/obj85 2023-05-31T20:48:12.386 DEBUG:boto:Data: 2023-05-31T20:48:12.386 DEBUG:boto:Headers: {} 2023-05-31T20:48:12.386 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:48:12.387 DEBUG:boto:Port: 8000 2023-05-31T20:48:12.387 DEBUG:boto:Params: {} 2023-05-31T20:48:12.387 DEBUG:boto:Token: None 2023-05-31T20:48:12.388 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:48:12 GMT /pklxkd-59/obj85 2023-05-31T20:48:12.388 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:DxQZPNwDsB7Sk7/grs8yzd5jLK4= 2023-05-31T20:48:12.389 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:48:12 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:DxQZPNwDsB7Sk7/grs8yzd5jLK4=', 'Content-Length': '0'} 2023-05-31T20:48:12.391 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:43:08 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000031d478f1a14f40b7-006477b28c-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:48:12 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:48:12.391 DEBUG:boto:path=/pklxkd-59/ 2023-05-31T20:48:12.392 DEBUG:boto:auth_path=/pklxkd-59/ 2023-05-31T20:48:12.392 DEBUG:boto:path=/pklxkd-59/?prefix=obj85 2023-05-31T20:48:12.392 DEBUG:boto:auth_path=/pklxkd-59/?prefix=obj85 2023-05-31T20:48:12.393 DEBUG:boto:Method: GET 2023-05-31T20:48:12.393 DEBUG:boto:Path: /pklxkd-59/?prefix=obj85 2023-05-31T20:48:12.393 DEBUG:boto:Data: 2023-05-31T20:48:12.394 DEBUG:boto:Headers: {} 2023-05-31T20:48:12.394 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:48:12.394 DEBUG:boto:Port: 8001 2023-05-31T20:48:12.395 DEBUG:boto:Params: {} 2023-05-31T20:48:12.395 DEBUG:boto:Token: None 2023-05-31T20:48:12.396 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:48:12 GMT /pklxkd-59/ 2023-05-31T20:48:12.396 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:7yviw5uwOHHeIaSmQ01+rDvoyyQ= 2023-05-31T20:48:12.396 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:48:12 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:7yviw5uwOHHeIaSmQ01+rDvoyyQ=', 'Content-Length': '0'} 2023-05-31T20:48:12.399 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000000764b141a9d00aaa-006477b28c-1c17-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:48:12 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:48:12.399 DEBUG:boto:b'pklxkd-59obj851000falseobj852023-05-31T20:43:08.086Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:48:12.400 DEBUG:boto:path=/pklxkd-59/ 2023-05-31T20:48:12.400 DEBUG:boto:auth_path=/pklxkd-59/ 2023-05-31T20:48:12.400 DEBUG:boto:path=/pklxkd-59/?prefix=obj85 2023-05-31T20:48:12.401 DEBUG:boto:auth_path=/pklxkd-59/?prefix=obj85 2023-05-31T20:48:12.401 DEBUG:boto:Method: GET 2023-05-31T20:48:12.402 DEBUG:boto:Path: /pklxkd-59/?prefix=obj85 2023-05-31T20:48:12.402 DEBUG:boto:Data: 2023-05-31T20:48:12.402 DEBUG:boto:Headers: {} 2023-05-31T20:48:12.403 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:48:12.403 DEBUG:boto:Port: 8000 2023-05-31T20:48:12.403 DEBUG:boto:Params: {} 2023-05-31T20:48:12.404 DEBUG:boto:Token: None 2023-05-31T20:48:12.404 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:48:12 GMT /pklxkd-59/ 2023-05-31T20:48:12.404 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:7yviw5uwOHHeIaSmQ01+rDvoyyQ= 2023-05-31T20:48:12.405 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:48:12 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:7yviw5uwOHHeIaSmQ01+rDvoyyQ=', 'Content-Length': '0'} 2023-05-31T20:48:12.408 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a02c1ba0dee5cf60-006477b28c-1630-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:48:12 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:48:12.408 DEBUG:boto:b'pklxkd-59obj851000falseobj852023-05-31T20:43:08.086Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:48:12.409 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj92 2023-05-31T20:48:12.409 DEBUG:boto:path=/pklxkd-59/obj92 2023-05-31T20:48:12.410 DEBUG:boto:auth_path=/pklxkd-59/obj92 2023-05-31T20:48:12.410 DEBUG:boto:path=/pklxkd-59/obj92?versionId=null 2023-05-31T20:48:12.410 DEBUG:boto:auth_path=/pklxkd-59/obj92?versionId=null 2023-05-31T20:48:12.411 DEBUG:boto:Method: GET 2023-05-31T20:48:12.411 DEBUG:boto:Path: /pklxkd-59/obj92?versionId=null 2023-05-31T20:48:12.412 DEBUG:boto:Data: 2023-05-31T20:48:12.412 DEBUG:boto:Headers: {} 2023-05-31T20:48:12.412 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:48:12.413 DEBUG:boto:Port: 8001 2023-05-31T20:48:12.413 DEBUG:boto:Params: {} 2023-05-31T20:48:12.413 DEBUG:boto:Token: None 2023-05-31T20:48:12.414 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:48:12 GMT /pklxkd-59/obj92?versionId=null 2023-05-31T20:48:12.414 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:SXtbCHSIsi2Sxxthi0B0SaQqGwo= 2023-05-31T20:48:12.415 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:48:12 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:SXtbCHSIsi2Sxxthi0B0SaQqGwo=', 'Content-Length': '0'} 2023-05-31T20:48:12.417 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:46:33 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000070e74a65fdef6f89-006477b28c-1c17-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:48:12 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:48:12.417 DEBUG:boto:path=/pklxkd-59/obj92 2023-05-31T20:48:12.418 DEBUG:boto:auth_path=/pklxkd-59/obj92 2023-05-31T20:48:12.418 DEBUG:boto:path=/pklxkd-59/obj92?versionId=null 2023-05-31T20:48:12.418 DEBUG:boto:auth_path=/pklxkd-59/obj92?versionId=null 2023-05-31T20:48:12.419 DEBUG:boto:Method: GET 2023-05-31T20:48:12.419 DEBUG:boto:Path: /pklxkd-59/obj92?versionId=null 2023-05-31T20:48:12.420 DEBUG:boto:Data: 2023-05-31T20:48:12.420 DEBUG:boto:Headers: {} 2023-05-31T20:48:12.420 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:48:12.421 DEBUG:boto:Port: 8000 2023-05-31T20:48:12.421 DEBUG:boto:Params: {} 2023-05-31T20:48:12.421 DEBUG:boto:Token: None 2023-05-31T20:48:12.422 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:48:12 GMT /pklxkd-59/obj92?versionId=null 2023-05-31T20:48:12.422 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:SXtbCHSIsi2Sxxthi0B0SaQqGwo= 2023-05-31T20:48:12.423 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:48:12 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:SXtbCHSIsi2Sxxthi0B0SaQqGwo=', 'Content-Length': '0'} 2023-05-31T20:48:12.424 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:46:33 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000036e01c410b3e295f-006477b28c-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:48:12 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:48:12.468 DEBUG:boto:path=/pklxkd-59/obj92 2023-05-31T20:48:12.469 DEBUG:boto:auth_path=/pklxkd-59/obj92 2023-05-31T20:48:12.469 DEBUG:boto:path=/pklxkd-59/obj92?versionId=null 2023-05-31T20:48:12.469 DEBUG:boto:auth_path=/pklxkd-59/obj92?versionId=null 2023-05-31T20:48:12.470 DEBUG:boto:Method: HEAD 2023-05-31T20:48:12.470 DEBUG:boto:Path: /pklxkd-59/obj92?versionId=null 2023-05-31T20:48:12.470 DEBUG:boto:Data: 2023-05-31T20:48:12.471 DEBUG:boto:Headers: {} 2023-05-31T20:48:12.471 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:48:12.471 DEBUG:boto:Port: 8001 2023-05-31T20:48:12.472 DEBUG:boto:Params: {} 2023-05-31T20:48:12.472 DEBUG:boto:Token: None 2023-05-31T20:48:12.472 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:48:12 GMT /pklxkd-59/obj92?versionId=null 2023-05-31T20:48:12.473 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:Ootufa8A4zzjJF2Q1ShRE357OEM= 2023-05-31T20:48:12.473 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:48:12 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:Ootufa8A4zzjJF2Q1ShRE357OEM=', 'Content-Length': '0'} 2023-05-31T20:48:12.475 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:46:33 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ccfe25a64ebe7863-006477b28c-1c17-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:48:12 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:48:12.476 DEBUG:boto:path=/pklxkd-59/obj92 2023-05-31T20:48:12.476 DEBUG:boto:auth_path=/pklxkd-59/obj92 2023-05-31T20:48:12.476 DEBUG:boto:path=/pklxkd-59/obj92?versionId=null 2023-05-31T20:48:12.477 DEBUG:boto:auth_path=/pklxkd-59/obj92?versionId=null 2023-05-31T20:48:12.477 DEBUG:boto:Method: HEAD 2023-05-31T20:48:12.477 DEBUG:boto:Path: /pklxkd-59/obj92?versionId=null 2023-05-31T20:48:12.478 DEBUG:boto:Data: 2023-05-31T20:48:12.478 DEBUG:boto:Headers: {} 2023-05-31T20:48:12.478 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:48:12.479 DEBUG:boto:Port: 8000 2023-05-31T20:48:12.479 DEBUG:boto:Params: {} 2023-05-31T20:48:12.479 DEBUG:boto:Token: None 2023-05-31T20:48:12.480 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:48:12 GMT /pklxkd-59/obj92?versionId=null 2023-05-31T20:48:12.480 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:Ootufa8A4zzjJF2Q1ShRE357OEM= 2023-05-31T20:48:12.481 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:48:12 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:Ootufa8A4zzjJF2Q1ShRE357OEM=', 'Content-Length': '0'} 2023-05-31T20:48:12.482 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:46:33 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008b0534055922343b-006477b28c-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:48:12 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:48:12.483 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj92 2023-05-31T20:48:12.483 DEBUG:boto:path=/pklxkd-59/obj92 2023-05-31T20:48:12.484 DEBUG:boto:auth_path=/pklxkd-59/obj92 2023-05-31T20:48:12.484 DEBUG:boto:path=/pklxkd-59/obj92?versionId=null 2023-05-31T20:48:12.484 DEBUG:boto:auth_path=/pklxkd-59/obj92?versionId=null 2023-05-31T20:48:12.485 DEBUG:boto:Method: GET 2023-05-31T20:48:12.485 DEBUG:boto:Path: /pklxkd-59/obj92?versionId=null 2023-05-31T20:48:12.485 DEBUG:boto:Data: 2023-05-31T20:48:12.486 DEBUG:boto:Headers: {} 2023-05-31T20:48:12.486 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:48:12.486 DEBUG:boto:Port: 8001 2023-05-31T20:48:12.487 DEBUG:boto:Params: {} 2023-05-31T20:48:12.487 DEBUG:boto:Token: None 2023-05-31T20:48:12.488 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:48:12 GMT /pklxkd-59/obj92?versionId=null 2023-05-31T20:48:12.488 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:SXtbCHSIsi2Sxxthi0B0SaQqGwo= 2023-05-31T20:48:12.488 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:48:12 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:SXtbCHSIsi2Sxxthi0B0SaQqGwo=', 'Content-Length': '0'} 2023-05-31T20:48:12.490 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:46:33 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000051e74908873bd090-006477b28c-1c17-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:48:12 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:48:12.491 DEBUG:boto:path=/pklxkd-59/obj92 2023-05-31T20:48:12.491 DEBUG:boto:auth_path=/pklxkd-59/obj92 2023-05-31T20:48:12.492 DEBUG:boto:path=/pklxkd-59/obj92?versionId=null 2023-05-31T20:48:12.492 DEBUG:boto:auth_path=/pklxkd-59/obj92?versionId=null 2023-05-31T20:48:12.492 DEBUG:boto:Method: GET 2023-05-31T20:48:12.493 DEBUG:boto:Path: /pklxkd-59/obj92?versionId=null 2023-05-31T20:48:12.493 DEBUG:boto:Data: 2023-05-31T20:48:12.493 DEBUG:boto:Headers: {} 2023-05-31T20:48:12.494 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:48:12.494 DEBUG:boto:Port: 8000 2023-05-31T20:48:12.494 DEBUG:boto:Params: {} 2023-05-31T20:48:12.495 DEBUG:boto:Token: None 2023-05-31T20:48:12.495 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:48:12 GMT /pklxkd-59/obj92?versionId=null 2023-05-31T20:48:12.496 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:SXtbCHSIsi2Sxxthi0B0SaQqGwo= 2023-05-31T20:48:12.496 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:48:12 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:SXtbCHSIsi2Sxxthi0B0SaQqGwo=', 'Content-Length': '0'} 2023-05-31T20:48:12.501 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:46:33 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000093132b31d2a774db-006477b28c-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:48:12 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:48:12.501 DEBUG:boto:path=/pklxkd-59/ 2023-05-31T20:48:12.502 DEBUG:boto:auth_path=/pklxkd-59/ 2023-05-31T20:48:12.502 DEBUG:boto:path=/pklxkd-59/?prefix=obj92 2023-05-31T20:48:12.502 DEBUG:boto:auth_path=/pklxkd-59/?prefix=obj92 2023-05-31T20:48:12.503 DEBUG:boto:Method: GET 2023-05-31T20:48:12.503 DEBUG:boto:Path: /pklxkd-59/?prefix=obj92 2023-05-31T20:48:12.503 DEBUG:boto:Data: 2023-05-31T20:48:12.504 DEBUG:boto:Headers: {} 2023-05-31T20:48:12.504 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:48:12.505 DEBUG:boto:Port: 8001 2023-05-31T20:48:12.505 DEBUG:boto:Params: {} 2023-05-31T20:48:12.505 DEBUG:boto:Token: None 2023-05-31T20:48:12.506 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:48:12 GMT /pklxkd-59/ 2023-05-31T20:48:12.506 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:7yviw5uwOHHeIaSmQ01+rDvoyyQ= 2023-05-31T20:48:12.507 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:48:12 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:7yviw5uwOHHeIaSmQ01+rDvoyyQ=', 'Content-Length': '0'} 2023-05-31T20:48:12.509 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000000b398cba31a628ed-006477b28c-1c17-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:48:12 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:48:12.509 DEBUG:boto:b'pklxkd-59obj921000falseobj922023-05-31T20:46:33.904Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:48:12.510 DEBUG:boto:path=/pklxkd-59/ 2023-05-31T20:48:12.510 DEBUG:boto:auth_path=/pklxkd-59/ 2023-05-31T20:48:12.511 DEBUG:boto:path=/pklxkd-59/?prefix=obj92 2023-05-31T20:48:12.511 DEBUG:boto:auth_path=/pklxkd-59/?prefix=obj92 2023-05-31T20:48:12.512 DEBUG:boto:Method: GET 2023-05-31T20:48:12.512 DEBUG:boto:Path: /pklxkd-59/?prefix=obj92 2023-05-31T20:48:12.512 DEBUG:boto:Data: 2023-05-31T20:48:12.513 DEBUG:boto:Headers: {} 2023-05-31T20:48:12.513 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:48:12.513 DEBUG:boto:Port: 8000 2023-05-31T20:48:12.514 DEBUG:boto:Params: {} 2023-05-31T20:48:12.514 DEBUG:boto:Token: None 2023-05-31T20:48:12.514 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:48:12 GMT /pklxkd-59/ 2023-05-31T20:48:12.515 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:7yviw5uwOHHeIaSmQ01+rDvoyyQ= 2023-05-31T20:48:12.515 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:48:12 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:7yviw5uwOHHeIaSmQ01+rDvoyyQ=', 'Content-Length': '0'} 2023-05-31T20:48:12.519 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000282421591782db32-006477b28c-1630-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:48:12 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:48:12.519 DEBUG:boto:b'pklxkd-59obj921000falseobj922023-05-31T20:46:33.904Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:48:12.520 DEBUG:boto:path=/pklxkd-59/obj92 2023-05-31T20:48:12.520 DEBUG:boto:auth_path=/pklxkd-59/obj92 2023-05-31T20:48:12.520 DEBUG:boto:Method: HEAD 2023-05-31T20:48:12.521 DEBUG:boto:Path: /pklxkd-59/obj92 2023-05-31T20:48:12.521 DEBUG:boto:Data: 2023-05-31T20:48:12.521 DEBUG:boto:Headers: {} 2023-05-31T20:48:12.522 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:48:12.522 DEBUG:boto:Port: 8001 2023-05-31T20:48:12.522 DEBUG:boto:Params: {} 2023-05-31T20:48:12.523 DEBUG:boto:Token: None 2023-05-31T20:48:12.523 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:48:12 GMT /pklxkd-59/obj92 2023-05-31T20:48:12.524 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:TLOE7gfq4bt5G6SlIErJkuFXigg= 2023-05-31T20:48:12.524 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:48:12 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:TLOE7gfq4bt5G6SlIErJkuFXigg=', 'Content-Length': '0'} 2023-05-31T20:48:12.526 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:46:33 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000dd7e081b04979c7c-006477b28c-1c17-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:48:12 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:48:12.526 DEBUG:boto:path=/pklxkd-59/obj92 2023-05-31T20:48:12.527 DEBUG:boto:auth_path=/pklxkd-59/obj92 2023-05-31T20:48:12.527 DEBUG:boto:Method: HEAD 2023-05-31T20:48:12.527 DEBUG:boto:Path: /pklxkd-59/obj92 2023-05-31T20:48:12.528 DEBUG:boto:Data: 2023-05-31T20:48:12.528 DEBUG:boto:Headers: {} 2023-05-31T20:48:12.528 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:48:12.529 DEBUG:boto:Port: 8000 2023-05-31T20:48:12.529 DEBUG:boto:Params: {} 2023-05-31T20:48:12.529 DEBUG:boto:Token: None 2023-05-31T20:48:12.530 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:48:12 GMT /pklxkd-59/obj92 2023-05-31T20:48:12.530 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:TLOE7gfq4bt5G6SlIErJkuFXigg= 2023-05-31T20:48:12.530 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:48:12 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:TLOE7gfq4bt5G6SlIErJkuFXigg=', 'Content-Length': '0'} 2023-05-31T20:48:12.532 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:46:33 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000004971d5b27b453a9-006477b28c-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:48:12 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:48:12.533 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj92 2023-05-31T20:48:12.533 DEBUG:boto:path=/pklxkd-59/obj92 2023-05-31T20:48:12.533 DEBUG:boto:auth_path=/pklxkd-59/obj92 2023-05-31T20:48:12.534 DEBUG:boto:Method: GET 2023-05-31T20:48:12.534 DEBUG:boto:Path: /pklxkd-59/obj92 2023-05-31T20:48:12.535 DEBUG:boto:Data: 2023-05-31T20:48:12.535 DEBUG:boto:Headers: {} 2023-05-31T20:48:12.535 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:48:12.536 DEBUG:boto:Port: 8001 2023-05-31T20:48:12.536 DEBUG:boto:Params: {} 2023-05-31T20:48:12.536 DEBUG:boto:Token: None 2023-05-31T20:48:12.537 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:48:12 GMT /pklxkd-59/obj92 2023-05-31T20:48:12.537 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:0Vj/PqLpYVf/zyC/if3tb9NeMw8= 2023-05-31T20:48:12.538 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:48:12 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:0Vj/PqLpYVf/zyC/if3tb9NeMw8=', 'Content-Length': '0'} 2023-05-31T20:48:12.539 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:46:33 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000084d122f16957c070-006477b28c-1c17-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:48:12 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:48:12.580 DEBUG:boto:path=/pklxkd-59/obj92 2023-05-31T20:48:12.581 DEBUG:boto:auth_path=/pklxkd-59/obj92 2023-05-31T20:48:12.581 DEBUG:boto:Method: GET 2023-05-31T20:48:12.581 DEBUG:boto:Path: /pklxkd-59/obj92 2023-05-31T20:48:12.582 DEBUG:boto:Data: 2023-05-31T20:48:12.582 DEBUG:boto:Headers: {} 2023-05-31T20:48:12.582 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:48:12.583 DEBUG:boto:Port: 8000 2023-05-31T20:48:12.583 DEBUG:boto:Params: {} 2023-05-31T20:48:12.583 DEBUG:boto:Token: None 2023-05-31T20:48:12.584 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:48:12 GMT /pklxkd-59/obj92 2023-05-31T20:48:12.584 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:0Vj/PqLpYVf/zyC/if3tb9NeMw8= 2023-05-31T20:48:12.585 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:48:12 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:0Vj/PqLpYVf/zyC/if3tb9NeMw8=', 'Content-Length': '0'} 2023-05-31T20:48:12.587 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:46:33 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ce501d89224908b6-006477b28c-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:48:12 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:48:12.587 DEBUG:boto:path=/pklxkd-59/ 2023-05-31T20:48:12.588 DEBUG:boto:auth_path=/pklxkd-59/ 2023-05-31T20:48:12.588 DEBUG:boto:path=/pklxkd-59/?prefix=obj92 2023-05-31T20:48:12.588 DEBUG:boto:auth_path=/pklxkd-59/?prefix=obj92 2023-05-31T20:48:12.589 DEBUG:boto:Method: GET 2023-05-31T20:48:12.589 DEBUG:boto:Path: /pklxkd-59/?prefix=obj92 2023-05-31T20:48:12.589 DEBUG:boto:Data: 2023-05-31T20:48:12.590 DEBUG:boto:Headers: {} 2023-05-31T20:48:12.590 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:48:12.590 DEBUG:boto:Port: 8001 2023-05-31T20:48:12.591 DEBUG:boto:Params: {} 2023-05-31T20:48:12.591 DEBUG:boto:Token: None 2023-05-31T20:48:12.592 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:48:12 GMT /pklxkd-59/ 2023-05-31T20:48:12.592 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:7yviw5uwOHHeIaSmQ01+rDvoyyQ= 2023-05-31T20:48:12.592 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:48:12 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:7yviw5uwOHHeIaSmQ01+rDvoyyQ=', 'Content-Length': '0'} 2023-05-31T20:48:12.595 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000bb30257ebf1da0e8-006477b28c-1c17-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:48:12 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:48:12.595 DEBUG:boto:b'pklxkd-59obj921000falseobj922023-05-31T20:46:33.904Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:48:12.596 DEBUG:boto:path=/pklxkd-59/ 2023-05-31T20:48:12.596 DEBUG:boto:auth_path=/pklxkd-59/ 2023-05-31T20:48:12.597 DEBUG:boto:path=/pklxkd-59/?prefix=obj92 2023-05-31T20:48:12.597 DEBUG:boto:auth_path=/pklxkd-59/?prefix=obj92 2023-05-31T20:48:12.597 DEBUG:boto:Method: GET 2023-05-31T20:48:12.598 DEBUG:boto:Path: /pklxkd-59/?prefix=obj92 2023-05-31T20:48:12.598 DEBUG:boto:Data: 2023-05-31T20:48:12.598 DEBUG:boto:Headers: {} 2023-05-31T20:48:12.599 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:48:12.599 DEBUG:boto:Port: 8000 2023-05-31T20:48:12.599 DEBUG:boto:Params: {} 2023-05-31T20:48:12.600 DEBUG:boto:Token: None 2023-05-31T20:48:12.600 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:48:12 GMT /pklxkd-59/ 2023-05-31T20:48:12.601 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:7yviw5uwOHHeIaSmQ01+rDvoyyQ= 2023-05-31T20:48:12.601 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:48:12 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:7yviw5uwOHHeIaSmQ01+rDvoyyQ=', 'Content-Length': '0'} 2023-05-31T20:48:12.604 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c108e51c59d2e495-006477b28c-1630-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:48:12 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:48:12.605 DEBUG:boto:b'pklxkd-59obj921000falseobj922023-05-31T20:46:33.904Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:48:12.605 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj95 2023-05-31T20:48:12.605 DEBUG:boto:path=/pklxkd-59/obj95 2023-05-31T20:48:12.606 DEBUG:boto:auth_path=/pklxkd-59/obj95 2023-05-31T20:48:12.606 DEBUG:boto:path=/pklxkd-59/obj95?versionId=null 2023-05-31T20:48:12.607 DEBUG:boto:auth_path=/pklxkd-59/obj95?versionId=null 2023-05-31T20:48:12.607 DEBUG:boto:Method: GET 2023-05-31T20:48:12.607 DEBUG:boto:Path: /pklxkd-59/obj95?versionId=null 2023-05-31T20:48:12.608 DEBUG:boto:Data: 2023-05-31T20:48:12.608 DEBUG:boto:Headers: {} 2023-05-31T20:48:12.608 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:48:12.609 DEBUG:boto:Port: 8001 2023-05-31T20:48:12.609 DEBUG:boto:Params: {} 2023-05-31T20:48:12.609 DEBUG:boto:Token: None 2023-05-31T20:48:12.610 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:48:12 GMT /pklxkd-59/obj95?versionId=null 2023-05-31T20:48:12.610 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:HmbZls8TGTGRG90RmMBxgwYDucQ= 2023-05-31T20:48:12.611 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:48:12 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:HmbZls8TGTGRG90RmMBxgwYDucQ=', 'Content-Length': '0'} 2023-05-31T20:48:12.612 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:44:50 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d1e4ee3852172d9e-006477b28c-1c17-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:48:12 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:48:12.613 DEBUG:boto:path=/pklxkd-59/obj95 2023-05-31T20:48:12.613 DEBUG:boto:auth_path=/pklxkd-59/obj95 2023-05-31T20:48:12.614 DEBUG:boto:path=/pklxkd-59/obj95?versionId=null 2023-05-31T20:48:12.614 DEBUG:boto:auth_path=/pklxkd-59/obj95?versionId=null 2023-05-31T20:48:12.614 DEBUG:boto:Method: GET 2023-05-31T20:48:12.615 DEBUG:boto:Path: /pklxkd-59/obj95?versionId=null 2023-05-31T20:48:12.615 DEBUG:boto:Data: 2023-05-31T20:48:12.615 DEBUG:boto:Headers: {} 2023-05-31T20:48:12.616 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:48:12.616 DEBUG:boto:Port: 8000 2023-05-31T20:48:12.616 DEBUG:boto:Params: {} 2023-05-31T20:48:12.617 DEBUG:boto:Token: None 2023-05-31T20:48:12.617 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:48:12 GMT /pklxkd-59/obj95?versionId=null 2023-05-31T20:48:12.618 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:HmbZls8TGTGRG90RmMBxgwYDucQ= 2023-05-31T20:48:12.618 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:48:12 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:HmbZls8TGTGRG90RmMBxgwYDucQ=', 'Content-Length': '0'} 2023-05-31T20:48:12.620 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:44:50 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000014895f4bbb79c6af-006477b28c-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:48:12 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:48:12.620 DEBUG:boto:path=/pklxkd-59/obj95 2023-05-31T20:48:12.621 DEBUG:boto:auth_path=/pklxkd-59/obj95 2023-05-31T20:48:12.621 DEBUG:boto:path=/pklxkd-59/obj95?versionId=null 2023-05-31T20:48:12.622 DEBUG:boto:auth_path=/pklxkd-59/obj95?versionId=null 2023-05-31T20:48:12.622 DEBUG:boto:Method: HEAD 2023-05-31T20:48:12.622 DEBUG:boto:Path: /pklxkd-59/obj95?versionId=null 2023-05-31T20:48:12.622 DEBUG:boto:Data: 2023-05-31T20:48:12.623 DEBUG:boto:Headers: {} 2023-05-31T20:48:12.623 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:48:12.623 DEBUG:boto:Port: 8001 2023-05-31T20:48:12.624 DEBUG:boto:Params: {} 2023-05-31T20:48:12.624 DEBUG:boto:Token: None 2023-05-31T20:48:12.625 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:48:12 GMT /pklxkd-59/obj95?versionId=null 2023-05-31T20:48:12.625 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:oVYtoPuInKNLOFVcMKTo5aIlQys= 2023-05-31T20:48:12.625 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:48:12 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:oVYtoPuInKNLOFVcMKTo5aIlQys=', 'Content-Length': '0'} 2023-05-31T20:48:12.627 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:44:50 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000acf8c388cbfa1d91-006477b28c-1c17-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:48:12 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:48:12.628 DEBUG:boto:path=/pklxkd-59/obj95 2023-05-31T20:48:12.628 DEBUG:boto:auth_path=/pklxkd-59/obj95 2023-05-31T20:48:12.628 DEBUG:boto:path=/pklxkd-59/obj95?versionId=null 2023-05-31T20:48:12.629 DEBUG:boto:auth_path=/pklxkd-59/obj95?versionId=null 2023-05-31T20:48:12.629 DEBUG:boto:Method: HEAD 2023-05-31T20:48:12.629 DEBUG:boto:Path: /pklxkd-59/obj95?versionId=null 2023-05-31T20:48:12.630 DEBUG:boto:Data: 2023-05-31T20:48:12.630 DEBUG:boto:Headers: {} 2023-05-31T20:48:12.630 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:48:12.631 DEBUG:boto:Port: 8000 2023-05-31T20:48:12.631 DEBUG:boto:Params: {} 2023-05-31T20:48:12.631 DEBUG:boto:Token: None 2023-05-31T20:48:12.632 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:48:12 GMT /pklxkd-59/obj95?versionId=null 2023-05-31T20:48:12.632 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:oVYtoPuInKNLOFVcMKTo5aIlQys= 2023-05-31T20:48:12.632 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:48:12 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:oVYtoPuInKNLOFVcMKTo5aIlQys=', 'Content-Length': '0'} 2023-05-31T20:48:12.634 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:44:50 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000da58d5f6932e6d35-006477b28c-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:48:12 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:48:12.635 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj95 2023-05-31T20:48:12.635 DEBUG:boto:path=/pklxkd-59/obj95 2023-05-31T20:48:12.635 DEBUG:boto:auth_path=/pklxkd-59/obj95 2023-05-31T20:48:12.636 DEBUG:boto:path=/pklxkd-59/obj95?versionId=null 2023-05-31T20:48:12.636 DEBUG:boto:auth_path=/pklxkd-59/obj95?versionId=null 2023-05-31T20:48:12.637 DEBUG:boto:Method: GET 2023-05-31T20:48:12.637 DEBUG:boto:Path: /pklxkd-59/obj95?versionId=null 2023-05-31T20:48:12.637 DEBUG:boto:Data: 2023-05-31T20:48:12.638 DEBUG:boto:Headers: {} 2023-05-31T20:48:12.638 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:48:12.638 DEBUG:boto:Port: 8001 2023-05-31T20:48:12.639 DEBUG:boto:Params: {} 2023-05-31T20:48:12.639 DEBUG:boto:Token: None 2023-05-31T20:48:12.639 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:48:12 GMT /pklxkd-59/obj95?versionId=null 2023-05-31T20:48:12.640 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:HmbZls8TGTGRG90RmMBxgwYDucQ= 2023-05-31T20:48:12.640 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:48:12 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:HmbZls8TGTGRG90RmMBxgwYDucQ=', 'Content-Length': '0'} 2023-05-31T20:48:12.642 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:44:50 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000cd44fb79a2a340a9-006477b28c-1c17-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:48:12 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:48:12.643 DEBUG:boto:path=/pklxkd-59/obj95 2023-05-31T20:48:12.643 DEBUG:boto:auth_path=/pklxkd-59/obj95 2023-05-31T20:48:12.643 DEBUG:boto:path=/pklxkd-59/obj95?versionId=null 2023-05-31T20:48:12.644 DEBUG:boto:auth_path=/pklxkd-59/obj95?versionId=null 2023-05-31T20:48:12.644 DEBUG:boto:Method: GET 2023-05-31T20:48:12.644 DEBUG:boto:Path: /pklxkd-59/obj95?versionId=null 2023-05-31T20:48:12.645 DEBUG:boto:Data: 2023-05-31T20:48:12.645 DEBUG:boto:Headers: {} 2023-05-31T20:48:12.645 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:48:12.646 DEBUG:boto:Port: 8000 2023-05-31T20:48:12.646 DEBUG:boto:Params: {} 2023-05-31T20:48:12.647 DEBUG:boto:Token: None 2023-05-31T20:48:12.647 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:48:12 GMT /pklxkd-59/obj95?versionId=null 2023-05-31T20:48:12.647 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:HmbZls8TGTGRG90RmMBxgwYDucQ= 2023-05-31T20:48:12.648 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:48:12 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:HmbZls8TGTGRG90RmMBxgwYDucQ=', 'Content-Length': '0'} 2023-05-31T20:48:12.650 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:44:50 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009880d2af010edb8e-006477b28c-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:48:12 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:48:12.650 DEBUG:boto:path=/pklxkd-59/ 2023-05-31T20:48:12.651 DEBUG:boto:auth_path=/pklxkd-59/ 2023-05-31T20:48:12.651 DEBUG:boto:path=/pklxkd-59/?prefix=obj95 2023-05-31T20:48:12.651 DEBUG:boto:auth_path=/pklxkd-59/?prefix=obj95 2023-05-31T20:48:12.652 DEBUG:boto:Method: GET 2023-05-31T20:48:12.652 DEBUG:boto:Path: /pklxkd-59/?prefix=obj95 2023-05-31T20:48:12.652 DEBUG:boto:Data: 2023-05-31T20:48:12.653 DEBUG:boto:Headers: {} 2023-05-31T20:48:12.653 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:48:12.653 DEBUG:boto:Port: 8001 2023-05-31T20:48:12.654 DEBUG:boto:Params: {} 2023-05-31T20:48:12.654 DEBUG:boto:Token: None 2023-05-31T20:48:12.655 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:48:12 GMT /pklxkd-59/ 2023-05-31T20:48:12.655 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:7yviw5uwOHHeIaSmQ01+rDvoyyQ= 2023-05-31T20:48:12.655 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:48:12 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:7yviw5uwOHHeIaSmQ01+rDvoyyQ=', 'Content-Length': '0'} 2023-05-31T20:48:12.658 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000007e462b60687cb529-006477b28c-1c17-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:48:12 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:48:12.658 DEBUG:boto:b'pklxkd-59obj951000falseobj952023-05-31T20:44:50.705Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:48:12.659 DEBUG:boto:path=/pklxkd-59/ 2023-05-31T20:48:12.659 DEBUG:boto:auth_path=/pklxkd-59/ 2023-05-31T20:48:12.660 DEBUG:boto:path=/pklxkd-59/?prefix=obj95 2023-05-31T20:48:12.660 DEBUG:boto:auth_path=/pklxkd-59/?prefix=obj95 2023-05-31T20:48:12.660 DEBUG:boto:Method: GET 2023-05-31T20:48:12.661 DEBUG:boto:Path: /pklxkd-59/?prefix=obj95 2023-05-31T20:48:12.661 DEBUG:boto:Data: 2023-05-31T20:48:12.661 DEBUG:boto:Headers: {} 2023-05-31T20:48:12.662 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:48:12.662 DEBUG:boto:Port: 8000 2023-05-31T20:48:12.662 DEBUG:boto:Params: {} 2023-05-31T20:48:12.663 DEBUG:boto:Token: None 2023-05-31T20:48:12.663 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:48:12 GMT /pklxkd-59/ 2023-05-31T20:48:12.664 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:7yviw5uwOHHeIaSmQ01+rDvoyyQ= 2023-05-31T20:48:12.664 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:48:12 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:7yviw5uwOHHeIaSmQ01+rDvoyyQ=', 'Content-Length': '0'} 2023-05-31T20:48:12.667 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000cb58989e9db89a62-006477b28c-1630-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:48:12 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:48:12.668 DEBUG:boto:b'pklxkd-59obj951000falseobj952023-05-31T20:44:50.705Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:48:12.668 DEBUG:boto:path=/pklxkd-59/obj95 2023-05-31T20:48:12.669 DEBUG:boto:auth_path=/pklxkd-59/obj95 2023-05-31T20:48:12.669 DEBUG:boto:Method: HEAD 2023-05-31T20:48:12.669 DEBUG:boto:Path: /pklxkd-59/obj95 2023-05-31T20:48:12.670 DEBUG:boto:Data: 2023-05-31T20:48:12.670 DEBUG:boto:Headers: {} 2023-05-31T20:48:12.670 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:48:12.671 DEBUG:boto:Port: 8001 2023-05-31T20:48:12.671 DEBUG:boto:Params: {} 2023-05-31T20:48:12.671 DEBUG:boto:Token: None 2023-05-31T20:48:12.672 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:48:12 GMT /pklxkd-59/obj95 2023-05-31T20:48:12.672 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:A3luaPQm/lhLQYFd7lc5V/5edY4= 2023-05-31T20:48:12.673 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:48:12 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:A3luaPQm/lhLQYFd7lc5V/5edY4=', 'Content-Length': '0'} 2023-05-31T20:48:12.674 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:44:50 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005a3c92e214ad7e7b-006477b28c-1c17-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:48:12 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:48:12.675 DEBUG:boto:path=/pklxkd-59/obj95 2023-05-31T20:48:12.675 DEBUG:boto:auth_path=/pklxkd-59/obj95 2023-05-31T20:48:12.676 DEBUG:boto:Method: HEAD 2023-05-31T20:48:12.676 DEBUG:boto:Path: /pklxkd-59/obj95 2023-05-31T20:48:12.676 DEBUG:boto:Data: 2023-05-31T20:48:12.677 DEBUG:boto:Headers: {} 2023-05-31T20:48:12.677 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:48:12.677 DEBUG:boto:Port: 8000 2023-05-31T20:48:12.678 DEBUG:boto:Params: {} 2023-05-31T20:48:12.678 DEBUG:boto:Token: None 2023-05-31T20:48:12.678 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:48:12 GMT /pklxkd-59/obj95 2023-05-31T20:48:12.679 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:A3luaPQm/lhLQYFd7lc5V/5edY4= 2023-05-31T20:48:12.679 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:48:12 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:A3luaPQm/lhLQYFd7lc5V/5edY4=', 'Content-Length': '0'} 2023-05-31T20:48:12.681 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:44:50 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000739cc820a5bc39b9-006477b28c-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:48:12 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:48:12.681 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj95 2023-05-31T20:48:12.682 DEBUG:boto:path=/pklxkd-59/obj95 2023-05-31T20:48:12.682 DEBUG:boto:auth_path=/pklxkd-59/obj95 2023-05-31T20:48:12.683 DEBUG:boto:Method: GET 2023-05-31T20:48:12.683 DEBUG:boto:Path: /pklxkd-59/obj95 2023-05-31T20:48:12.683 DEBUG:boto:Data: 2023-05-31T20:48:12.684 DEBUG:boto:Headers: {} 2023-05-31T20:48:12.684 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:48:12.684 DEBUG:boto:Port: 8001 2023-05-31T20:48:12.685 DEBUG:boto:Params: {} 2023-05-31T20:48:12.685 DEBUG:boto:Token: None 2023-05-31T20:48:12.686 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:48:12 GMT /pklxkd-59/obj95 2023-05-31T20:48:12.686 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:/hchSayiKZYZAkev1ihyDWgoJeM= 2023-05-31T20:48:12.686 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:48:12 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:/hchSayiKZYZAkev1ihyDWgoJeM=', 'Content-Length': '0'} 2023-05-31T20:48:12.688 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:44:50 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003c972b8d5efcc0d9-006477b28c-1c17-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:48:12 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:48:12.689 DEBUG:boto:path=/pklxkd-59/obj95 2023-05-31T20:48:12.689 DEBUG:boto:auth_path=/pklxkd-59/obj95 2023-05-31T20:48:12.690 DEBUG:boto:Method: GET 2023-05-31T20:48:12.690 DEBUG:boto:Path: /pklxkd-59/obj95 2023-05-31T20:48:12.690 DEBUG:boto:Data: 2023-05-31T20:48:12.691 DEBUG:boto:Headers: {} 2023-05-31T20:48:12.691 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:48:12.692 DEBUG:boto:Port: 8000 2023-05-31T20:48:12.692 DEBUG:boto:Params: {} 2023-05-31T20:48:12.692 DEBUG:boto:Token: None 2023-05-31T20:48:12.693 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:48:12 GMT /pklxkd-59/obj95 2023-05-31T20:48:12.693 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:/hchSayiKZYZAkev1ihyDWgoJeM= 2023-05-31T20:48:12.694 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:48:12 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:/hchSayiKZYZAkev1ihyDWgoJeM=', 'Content-Length': '0'} 2023-05-31T20:48:12.695 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:44:50 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000de069db710a1f1c6-006477b28c-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:48:12 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:48:12.696 DEBUG:boto:path=/pklxkd-59/ 2023-05-31T20:48:12.696 DEBUG:boto:auth_path=/pklxkd-59/ 2023-05-31T20:48:12.697 DEBUG:boto:path=/pklxkd-59/?prefix=obj95 2023-05-31T20:48:12.697 DEBUG:boto:auth_path=/pklxkd-59/?prefix=obj95 2023-05-31T20:48:12.697 DEBUG:boto:Method: GET 2023-05-31T20:48:12.698 DEBUG:boto:Path: /pklxkd-59/?prefix=obj95 2023-05-31T20:48:12.698 DEBUG:boto:Data: 2023-05-31T20:48:12.698 DEBUG:boto:Headers: {} 2023-05-31T20:48:12.699 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:48:12.699 DEBUG:boto:Port: 8001 2023-05-31T20:48:12.699 DEBUG:boto:Params: {} 2023-05-31T20:48:12.700 DEBUG:boto:Token: None 2023-05-31T20:48:12.700 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:48:12 GMT /pklxkd-59/ 2023-05-31T20:48:12.701 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:7yviw5uwOHHeIaSmQ01+rDvoyyQ= 2023-05-31T20:48:12.701 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:48:12 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:7yviw5uwOHHeIaSmQ01+rDvoyyQ=', 'Content-Length': '0'} 2023-05-31T20:48:12.704 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000009565955eaacf2f3f-006477b28c-1c17-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:48:12 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:48:12.744 DEBUG:boto:b'pklxkd-59obj951000falseobj952023-05-31T20:44:50.705Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:48:12.745 DEBUG:boto:path=/pklxkd-59/ 2023-05-31T20:48:12.745 DEBUG:boto:auth_path=/pklxkd-59/ 2023-05-31T20:48:12.746 DEBUG:boto:path=/pklxkd-59/?prefix=obj95 2023-05-31T20:48:12.746 DEBUG:boto:auth_path=/pklxkd-59/?prefix=obj95 2023-05-31T20:48:12.746 DEBUG:boto:Method: GET 2023-05-31T20:48:12.747 DEBUG:boto:Path: /pklxkd-59/?prefix=obj95 2023-05-31T20:48:12.747 DEBUG:boto:Data: 2023-05-31T20:48:12.747 DEBUG:boto:Headers: {} 2023-05-31T20:48:12.748 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:48:12.748 DEBUG:boto:Port: 8000 2023-05-31T20:48:12.749 DEBUG:boto:Params: {} 2023-05-31T20:48:12.749 DEBUG:boto:Token: None 2023-05-31T20:48:12.750 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:48:12 GMT /pklxkd-59/ 2023-05-31T20:48:12.750 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:7yviw5uwOHHeIaSmQ01+rDvoyyQ= 2023-05-31T20:48:12.750 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:48:12 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:7yviw5uwOHHeIaSmQ01+rDvoyyQ=', 'Content-Length': '0'} 2023-05-31T20:48:12.753 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000005e5e8db4a9d9fcb7-006477b28c-1630-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:48:12 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:48:12.754 DEBUG:boto:b'pklxkd-59obj951000falseobj952023-05-31T20:44:50.705Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:48:12.754 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=pklxkd-59 zones={a2, a1} 2023-05-31T20:48:12.755 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'reshard', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'pklxkd-59', '--num-shards', '29', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-05-31T20:48:12.755 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'reshard', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'pklxkd-59', '--num-shards', '29', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-05-31T20:48:12.756 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket reshard --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --bucket pklxkd-59 --num-shards 29 --yes-i-really-mean-it --cluster c1 --debug-rgw 0 --debug-ms 0 2023-05-31T20:48:12.814 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2023-05-31T20:48:12.814 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2023-05-31T20:48:12.872 INFO:teuthology.orchestra.run.smithi099.stdout:tenant: 2023-05-31T20:48:12.872 INFO:teuthology.orchestra.run.smithi099.stdout:bucket name: pklxkd-59 2023-05-31T20:48:12.880 INFO:teuthology.orchestra.run.smithi099.stdout:total entries: 20 2023-05-31T20:48:12.944 DEBUG:boto:path=/pklxkd-59/ 2023-05-31T20:48:12.945 DEBUG:boto:auth_path=/pklxkd-59/ 2023-05-31T20:48:12.945 DEBUG:boto:Method: HEAD 2023-05-31T20:48:12.946 DEBUG:boto:Path: /pklxkd-59/ 2023-05-31T20:48:12.946 DEBUG:boto:Data: 2023-05-31T20:48:12.946 DEBUG:boto:Headers: {} 2023-05-31T20:48:12.947 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:48:12.947 DEBUG:boto:Port: 8000 2023-05-31T20:48:12.947 DEBUG:boto:Params: {} 2023-05-31T20:48:12.948 DEBUG:boto:Token: None 2023-05-31T20:48:12.948 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:48:12 GMT /pklxkd-59/ 2023-05-31T20:48:12.949 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:7IyZEd1V3hXsFJWtriSA8OSIHPc= 2023-05-31T20:48:12.949 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:48:12 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:7IyZEd1V3hXsFJWtriSA8OSIHPc=', 'Content-Length': '0'} 2023-05-31T20:48:12.952 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '20'), ('X-RGW-Bytes-Used', '60'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000074582844be18b287-006477b28c-1630-a1'), ('Content-Length', '0'), ('Date', 'Wed, 31 May 2023 20:48:12 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:48:12.953 DEBUG:boto:path=/pklxkd-59/obj29 2023-05-31T20:48:12.953 DEBUG:boto:auth_path=/pklxkd-59/obj29 2023-05-31T20:48:12.954 DEBUG:boto:Method: PUT 2023-05-31T20:48:12.954 DEBUG:boto:Path: /pklxkd-59/obj29 2023-05-31T20:48:12.955 DEBUG:boto:Data: 2023-05-31T20:48:12.955 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-05-31T20:48:12.955 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:48:12.956 DEBUG:boto:Port: 8000 2023-05-31T20:48:12.956 DEBUG:boto:Params: {} 2023-05-31T20:48:12.956 DEBUG:boto:Token: None 2023-05-31T20:48:12.957 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Wed, 31 May 2023 20:48:12 GMT /pklxkd-59/obj29 2023-05-31T20:48:12.957 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:+vE96gp/ndcGNg7gxn3fLamsWWQ= 2023-05-31T20:48:12.957 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Wed, 31 May 2023 20:48:12 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:+vE96gp/ndcGNg7gxn3fLamsWWQ='} 2023-05-31T20:48:12.965 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000064c6bf8b55792eca-006477b28c-1630-a1'), ('Date', 'Wed, 31 May 2023 20:48:12 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:48:12.965 DEBUG:boto:path=/pklxkd-59/ 2023-05-31T20:48:12.966 DEBUG:boto:auth_path=/pklxkd-59/ 2023-05-31T20:48:12.966 DEBUG:boto:Method: HEAD 2023-05-31T20:48:12.967 DEBUG:boto:Path: /pklxkd-59/ 2023-05-31T20:48:12.967 DEBUG:boto:Data: 2023-05-31T20:48:12.967 DEBUG:boto:Headers: {} 2023-05-31T20:48:12.967 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:48:12.968 DEBUG:boto:Port: 8000 2023-05-31T20:48:12.968 DEBUG:boto:Params: {} 2023-05-31T20:48:12.969 DEBUG:boto:Token: None 2023-05-31T20:48:12.969 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:48:12 GMT /pklxkd-59/ 2023-05-31T20:48:12.969 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:7IyZEd1V3hXsFJWtriSA8OSIHPc= 2023-05-31T20:48:12.970 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:48:12 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:7IyZEd1V3hXsFJWtriSA8OSIHPc=', 'Content-Length': '0'} 2023-05-31T20:48:12.972 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '21'), ('X-RGW-Bytes-Used', '63'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000dc417482a98e2948-006477b28c-1630-a1'), ('Content-Length', '0'), ('Date', 'Wed, 31 May 2023 20:48:12 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:48:12.973 DEBUG:boto:path=/pklxkd-59/obj58 2023-05-31T20:48:12.973 DEBUG:boto:auth_path=/pklxkd-59/obj58 2023-05-31T20:48:12.974 DEBUG:boto:Method: PUT 2023-05-31T20:48:12.974 DEBUG:boto:Path: /pklxkd-59/obj58 2023-05-31T20:48:12.974 DEBUG:boto:Data: 2023-05-31T20:48:12.975 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-05-31T20:48:12.975 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:48:12.975 DEBUG:boto:Port: 8000 2023-05-31T20:48:12.976 DEBUG:boto:Params: {} 2023-05-31T20:48:12.976 DEBUG:boto:Token: None 2023-05-31T20:48:12.977 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Wed, 31 May 2023 20:48:12 GMT /pklxkd-59/obj58 2023-05-31T20:48:12.977 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:bnzwXXcGcZZFddsn0kaZ90sCXfo= 2023-05-31T20:48:12.977 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Wed, 31 May 2023 20:48:12 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:bnzwXXcGcZZFddsn0kaZ90sCXfo='} 2023-05-31T20:48:13.020 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000005cb9f137f430c7e5-006477b28c-1630-a1'), ('Date', 'Wed, 31 May 2023 20:48:12 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:48:13.021 DEBUG:boto:path=/pklxkd-59/ 2023-05-31T20:48:13.021 DEBUG:boto:auth_path=/pklxkd-59/ 2023-05-31T20:48:13.022 DEBUG:boto:Method: HEAD 2023-05-31T20:48:13.022 DEBUG:boto:Path: /pklxkd-59/ 2023-05-31T20:48:13.022 DEBUG:boto:Data: 2023-05-31T20:48:13.023 DEBUG:boto:Headers: {} 2023-05-31T20:48:13.023 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:48:13.023 DEBUG:boto:Port: 8000 2023-05-31T20:48:13.024 DEBUG:boto:Params: {} 2023-05-31T20:48:13.024 DEBUG:boto:Token: None 2023-05-31T20:48:13.025 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:48:13 GMT /pklxkd-59/ 2023-05-31T20:48:13.025 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:T/GIqoH36VLydQZgccl1N0NPSrw= 2023-05-31T20:48:13.025 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:48:13 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:T/GIqoH36VLydQZgccl1N0NPSrw=', 'Content-Length': '0'} 2023-05-31T20:48:13.028 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '22'), ('X-RGW-Bytes-Used', '66'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000b65740275b0f8a24-006477b28d-1630-a1'), ('Content-Length', '0'), ('Date', 'Wed, 31 May 2023 20:48:13 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:48:13.029 DEBUG:boto:path=/pklxkd-59/obj87 2023-05-31T20:48:13.029 DEBUG:boto:auth_path=/pklxkd-59/obj87 2023-05-31T20:48:13.029 DEBUG:boto:Method: PUT 2023-05-31T20:48:13.030 DEBUG:boto:Path: /pklxkd-59/obj87 2023-05-31T20:48:13.030 DEBUG:boto:Data: 2023-05-31T20:48:13.030 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-05-31T20:48:13.031 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:48:13.031 DEBUG:boto:Port: 8000 2023-05-31T20:48:13.031 DEBUG:boto:Params: {} 2023-05-31T20:48:13.032 DEBUG:boto:Token: None 2023-05-31T20:48:13.032 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Wed, 31 May 2023 20:48:13 GMT /pklxkd-59/obj87 2023-05-31T20:48:13.032 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:8ipzQwNtLBejzA0sk24bflDEoQs= 2023-05-31T20:48:13.033 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Wed, 31 May 2023 20:48:13 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:8ipzQwNtLBejzA0sk24bflDEoQs='} 2023-05-31T20:48:13.042 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000005e747b1b68ac0fee-006477b28d-1630-a1'), ('Date', 'Wed, 31 May 2023 20:48:13 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:48:13.042 DEBUG:boto:path=/pklxkd-59/ 2023-05-31T20:48:13.042 DEBUG:boto:auth_path=/pklxkd-59/ 2023-05-31T20:48:13.043 DEBUG:boto:Method: HEAD 2023-05-31T20:48:13.043 DEBUG:boto:Path: /pklxkd-59/ 2023-05-31T20:48:13.043 DEBUG:boto:Data: 2023-05-31T20:48:13.044 DEBUG:boto:Headers: {} 2023-05-31T20:48:13.044 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:48:13.044 DEBUG:boto:Port: 8000 2023-05-31T20:48:13.045 DEBUG:boto:Params: {} 2023-05-31T20:48:13.045 DEBUG:boto:Token: None 2023-05-31T20:48:13.046 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:48:13 GMT /pklxkd-59/ 2023-05-31T20:48:13.046 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:T/GIqoH36VLydQZgccl1N0NPSrw= 2023-05-31T20:48:13.046 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:48:13 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:T/GIqoH36VLydQZgccl1N0NPSrw=', 'Content-Length': '0'} 2023-05-31T20:48:13.049 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '23'), ('X-RGW-Bytes-Used', '69'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000006c999bcfff2431c3-006477b28d-1630-a1'), ('Content-Length', '0'), ('Date', 'Wed, 31 May 2023 20:48:13 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:48:13.050 DEBUG:boto:path=/pklxkd-59/obj116 2023-05-31T20:48:13.050 DEBUG:boto:auth_path=/pklxkd-59/obj116 2023-05-31T20:48:13.050 DEBUG:boto:Method: PUT 2023-05-31T20:48:13.051 DEBUG:boto:Path: /pklxkd-59/obj116 2023-05-31T20:48:13.051 DEBUG:boto:Data: 2023-05-31T20:48:13.051 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-05-31T20:48:13.052 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:48:13.052 DEBUG:boto:Port: 8000 2023-05-31T20:48:13.052 DEBUG:boto:Params: {} 2023-05-31T20:48:13.053 DEBUG:boto:Token: None 2023-05-31T20:48:13.053 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Wed, 31 May 2023 20:48:13 GMT /pklxkd-59/obj116 2023-05-31T20:48:13.054 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:Z5KZGQBY59IwL5DA0K+Mc/GlTsc= 2023-05-31T20:48:13.054 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Wed, 31 May 2023 20:48:13 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:Z5KZGQBY59IwL5DA0K+Mc/GlTsc='} 2023-05-31T20:48:13.096 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000096eaefbed89a708e-006477b28d-1630-a1'), ('Date', 'Wed, 31 May 2023 20:48:13 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:48:13.097 DEBUG:boto:path=/pklxkd-59/ 2023-05-31T20:48:13.097 DEBUG:boto:auth_path=/pklxkd-59/ 2023-05-31T20:48:13.097 DEBUG:boto:Method: HEAD 2023-05-31T20:48:13.098 DEBUG:boto:Path: /pklxkd-59/ 2023-05-31T20:48:13.098 DEBUG:boto:Data: 2023-05-31T20:48:13.098 DEBUG:boto:Headers: {} 2023-05-31T20:48:13.099 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:48:13.099 DEBUG:boto:Port: 8000 2023-05-31T20:48:13.099 DEBUG:boto:Params: {} 2023-05-31T20:48:13.100 DEBUG:boto:Token: None 2023-05-31T20:48:13.100 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:48:13 GMT /pklxkd-59/ 2023-05-31T20:48:13.101 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:T/GIqoH36VLydQZgccl1N0NPSrw= 2023-05-31T20:48:13.101 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:48:13 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:T/GIqoH36VLydQZgccl1N0NPSrw=', 'Content-Length': '0'} 2023-05-31T20:48:13.104 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '24'), ('X-RGW-Bytes-Used', '72'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000040753fb5217f5208-006477b28d-1630-a1'), ('Content-Length', '0'), ('Date', 'Wed, 31 May 2023 20:48:13 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:48:13.105 DEBUG:boto:path=/pklxkd-59/obj145 2023-05-31T20:48:13.105 DEBUG:boto:auth_path=/pklxkd-59/obj145 2023-05-31T20:48:13.105 DEBUG:boto:Method: PUT 2023-05-31T20:48:13.106 DEBUG:boto:Path: /pklxkd-59/obj145 2023-05-31T20:48:13.106 DEBUG:boto:Data: 2023-05-31T20:48:13.107 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-05-31T20:48:13.107 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:48:13.107 DEBUG:boto:Port: 8000 2023-05-31T20:48:13.108 DEBUG:boto:Params: {} 2023-05-31T20:48:13.108 DEBUG:boto:Token: None 2023-05-31T20:48:13.109 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Wed, 31 May 2023 20:48:13 GMT /pklxkd-59/obj145 2023-05-31T20:48:13.109 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:osX9FBaKbmtErolj523uEOH/QwI= 2023-05-31T20:48:13.109 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Wed, 31 May 2023 20:48:13 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:osX9FBaKbmtErolj523uEOH/QwI='} 2023-05-31T20:48:13.115 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000d84c5095934f3a11-006477b28d-1630-a1'), ('Date', 'Wed, 31 May 2023 20:48:13 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:48:13.115 DEBUG:rgw_multi.tests:bucket checkpoint: source=a1 target=a2 bucket=pklxkd-59 2023-05-31T20:48:13.116 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'pklxkd-59', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-05-31T20:48:13.116 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'pklxkd-59', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-05-31T20:48:13.116 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket pklxkd-59 --source-zone a1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2023-05-31T20:48:13.173 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2023-05-31T20:48:13.173 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2023-05-31T20:48:13.214 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:48:13.210+0000 7ff2db554780 1 waiting to reach latest gen 4, on 3.. 2023-05-31T20:48:18.215 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:48:18.211+0000 7ff2db554780 1 waiting to reach latest gen 4, on 3.. 2023-05-31T20:48:23.215 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:48:23.210+0000 7ff2db554780 1 waiting to reach latest gen 4, on 3.. 2023-05-31T20:48:28.216 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:48:28.211+0000 7ff2db554780 1 waiting to reach latest gen 4, on 3.. 2023-05-31T20:48:33.218 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:48:33.213+0000 7ff2db554780 1 waiting for incremental sync to catch up: 2023-05-31T20:48:33.219 INFO:teuthology.orchestra.run.smithi099.stderr: local status: [, , , , , , , , , , , , , , , , , , , , , , , , , , , , ] 2023-05-31T20:48:33.219 INFO:teuthology.orchestra.run.smithi099.stderr: remote markers: [00000000002.83.6, 00000000001.68.6, , , , , 00000000001.60.6, , , , , , , , , , , 00000000002.50.6, 00000000001.102.6, , , , , , , , , , ] 2023-05-31T20:48:38.220 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:48:38.215+0000 7ff2db554780 1 waiting for incremental sync to catch up: 2023-05-31T20:48:38.221 INFO:teuthology.orchestra.run.smithi099.stderr: local status: [, , , , , , , , , , , , , , , , , , , , , , , , , , , , ] 2023-05-31T20:48:38.221 INFO:teuthology.orchestra.run.smithi099.stderr: remote markers: [00000000002.83.6, 00000000001.68.6, , , , , 00000000001.60.6, , , , , , , , , , , 00000000002.50.6, 00000000001.102.6, , , , , , , , , , ] 2023-05-31T20:48:43.222 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:48:43.218+0000 7ff2db554780 1 waiting for incremental sync to catch up: 2023-05-31T20:48:43.223 INFO:teuthology.orchestra.run.smithi099.stderr: local status: [, , , , , , , , , , , , , , , , , , , , , , , , , , , , ] 2023-05-31T20:48:43.223 INFO:teuthology.orchestra.run.smithi099.stderr: remote markers: [00000000002.83.6, 00000000001.68.6, , , , , 00000000001.60.6, , , , , , , , , , , 00000000002.50.6, 00000000001.102.6, , , , , , , , , , ] 2023-05-31T20:48:48.224 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:48:48.220+0000 7ff2db554780 1 waiting for incremental sync to catch up: 2023-05-31T20:48:48.225 INFO:teuthology.orchestra.run.smithi099.stderr: local status: [, , , , , , , , , , , , , , , , , , , , , , , , , , , , ] 2023-05-31T20:48:48.225 INFO:teuthology.orchestra.run.smithi099.stderr: remote markers: [00000000002.83.6, 00000000001.68.6, , , , , 00000000001.60.6, , , , , , , , , , , 00000000002.50.6, 00000000001.102.6, , , , , , , , , , ] 2023-05-31T20:48:53.226 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:48:53.222+0000 7ff2db554780 1 waiting for incremental sync to catch up: 2023-05-31T20:48:53.227 INFO:teuthology.orchestra.run.smithi099.stderr: local status: [, 00000000001.68.6, , , , , , , , , , , , , , , , 00000000002.50.6, , , , , , , , , , , ] 2023-05-31T20:48:53.227 INFO:teuthology.orchestra.run.smithi099.stderr: remote markers: [00000000002.83.6, 00000000001.68.6, , , , , 00000000001.60.6, , , , , , , , , , , 00000000002.50.6, 00000000001.102.6, , , , , , , , , , ] 2023-05-31T20:48:58.228 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:48:58.224+0000 7ff2db554780 1 waiting for incremental sync to catch up: 2023-05-31T20:48:58.229 INFO:teuthology.orchestra.run.smithi099.stderr: local status: [, 00000000001.68.6, , , , , , , , , , , , , , , , 00000000002.50.6, , , , , , , , , , , ] 2023-05-31T20:48:58.229 INFO:teuthology.orchestra.run.smithi099.stderr: remote markers: [00000000002.83.6, 00000000001.68.6, , , , , 00000000001.60.6, , , , , , , , , , , 00000000002.50.6, 00000000001.102.6, , , , , , , , , , ] 2023-05-31T20:49:03.230 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:49:03.225+0000 7ff2db554780 1 waiting for incremental sync to catch up: 2023-05-31T20:49:03.231 INFO:teuthology.orchestra.run.smithi099.stderr: local status: [, 00000000001.68.6, , , , , , , , , , , , , , , , 00000000002.50.6, , , , , , , , , , , ] 2023-05-31T20:49:03.231 INFO:teuthology.orchestra.run.smithi099.stderr: remote markers: [00000000002.83.6, 00000000001.68.6, , , , , 00000000001.60.6, , , , , , , , , , , 00000000002.50.6, 00000000001.102.6, , , , , , , , , , ] 2023-05-31T20:49:08.233 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:49:08.228+0000 7ff2db554780 1 waiting for incremental sync to catch up: 2023-05-31T20:49:08.233 INFO:teuthology.orchestra.run.smithi099.stderr: local status: [00000000002.83.6, 00000000001.68.6, , , , , 00000000001.60.6, , , , , , , , , , , 00000000002.50.6, , , , , , , , , , , ] 2023-05-31T20:49:08.233 INFO:teuthology.orchestra.run.smithi099.stderr: remote markers: [00000000002.83.6, 00000000001.68.6, , , , , 00000000001.60.6, , , , , , , , , , , 00000000002.50.6, 00000000001.102.6, , , , , , , , , , ] 2023-05-31T20:49:13.235 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:49:13.230+0000 7ff2db554780 1 waiting for incremental sync to catch up: 2023-05-31T20:49:13.235 INFO:teuthology.orchestra.run.smithi099.stderr: local status: [00000000002.83.6, 00000000001.68.6, , , , , 00000000001.60.6, , , , , , , , , , , 00000000002.50.6, , , , , , , , , , , ] 2023-05-31T20:49:13.235 INFO:teuthology.orchestra.run.smithi099.stderr: remote markers: [00000000002.83.6, 00000000001.68.6, , , , , 00000000001.60.6, , , , , , , , , , , 00000000002.50.6, 00000000001.102.6, , , , , , , , , , ] 2023-05-31T20:49:18.237 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:49:18.232+0000 7ff2db554780 1 waiting for incremental sync to catch up: 2023-05-31T20:49:18.237 INFO:teuthology.orchestra.run.smithi099.stderr: local status: [00000000002.83.6, 00000000001.68.6, , , , , 00000000001.60.6, , , , , , , , , , , 00000000002.50.6, , , , , , , , , , , ] 2023-05-31T20:49:18.237 INFO:teuthology.orchestra.run.smithi099.stderr: remote markers: [00000000002.83.6, 00000000001.68.6, , , , , 00000000001.60.6, , , , , , , , , , , 00000000002.50.6, 00000000001.102.6, , , , , , , , , , ] 2023-05-31T20:49:23.239 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:49:23.234+0000 7ff2db554780 1 waiting for incremental sync to catch up: 2023-05-31T20:49:23.239 INFO:teuthology.orchestra.run.smithi099.stderr: local status: [00000000002.83.6, 00000000001.68.6, , , , , 00000000001.60.6, , , , , , , , , , , 00000000002.50.6, , , , , , , , , , , ] 2023-05-31T20:49:23.239 INFO:teuthology.orchestra.run.smithi099.stderr: remote markers: [00000000002.83.6, 00000000001.68.6, , , , , 00000000001.60.6, , , , , , , , , , , 00000000002.50.6, 00000000001.102.6, , , , , , , , , , ] 2023-05-31T20:49:28.241 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:49:28.237+0000 7ff2db554780 1 waiting for incremental sync to catch up: 2023-05-31T20:49:28.241 INFO:teuthology.orchestra.run.smithi099.stderr: local status: [00000000002.83.6, 00000000001.68.6, , , , , 00000000001.60.6, , , , , , , , , , , 00000000002.50.6, , , , , , , , , , , ] 2023-05-31T20:49:28.241 INFO:teuthology.orchestra.run.smithi099.stderr: remote markers: [00000000002.83.6, 00000000001.68.6, , , , , 00000000001.60.6, , , , , , , , , , , 00000000002.50.6, 00000000001.102.6, , , , , , , , , , ] 2023-05-31T20:49:33.243 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:49:33.239+0000 7ff2db554780 1 waiting for incremental sync to catch up: 2023-05-31T20:49:33.243 INFO:teuthology.orchestra.run.smithi099.stderr: local status: [00000000002.83.6, 00000000001.68.6, , , , , 00000000001.60.6, , , , , , , , , , , 00000000002.50.6, , , , , , , , , , , ] 2023-05-31T20:49:33.243 INFO:teuthology.orchestra.run.smithi099.stderr: remote markers: [00000000002.83.6, 00000000001.68.6, , , , , 00000000001.60.6, , , , , , , , , , , 00000000002.50.6, 00000000001.102.6, , , , , , , , , , ] 2023-05-31T20:49:38.245 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:49:38.241+0000 7ff2db554780 1 waiting for incremental sync to catch up: 2023-05-31T20:49:38.246 INFO:teuthology.orchestra.run.smithi099.stderr: local status: [00000000002.83.6, 00000000001.68.6, , , , , 00000000001.60.6, , , , , , , , , , , 00000000002.50.6, , , , , , , , , , , ] 2023-05-31T20:49:38.246 INFO:teuthology.orchestra.run.smithi099.stderr: remote markers: [00000000002.83.6, 00000000001.68.6, , , , , 00000000001.60.6, , , , , , , , , , , 00000000002.50.6, 00000000001.102.6, , , , , , , , , , ] 2023-05-31T20:49:43.247 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:49:43.243+0000 7ff2db554780 1 waiting for incremental sync to catch up: 2023-05-31T20:49:43.247 INFO:teuthology.orchestra.run.smithi099.stderr: local status: [00000000002.83.6, 00000000001.68.6, , , , , 00000000001.60.6, , , , , , , , , , , 00000000002.50.6, , , , , , , , , , , ] 2023-05-31T20:49:43.248 INFO:teuthology.orchestra.run.smithi099.stderr: remote markers: [00000000002.83.6, 00000000001.68.6, , , , , 00000000001.60.6, , , , , , , , , , , 00000000002.50.6, 00000000001.102.6, , , , , , , , , , ] 2023-05-31T20:49:48.249 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:49:48.244+0000 7ff2db554780 1 waiting for incremental sync to catch up: 2023-05-31T20:49:48.249 INFO:teuthology.orchestra.run.smithi099.stderr: local status: [00000000002.83.6, 00000000001.68.6, , , , , 00000000001.60.6, , , , , , , , , , , 00000000002.50.6, , , , , , , , , , , ] 2023-05-31T20:49:48.250 INFO:teuthology.orchestra.run.smithi099.stderr: remote markers: [00000000002.83.6, 00000000001.68.6, , , , , 00000000001.60.6, , , , , , , , , , , 00000000002.50.6, 00000000001.102.6, , , , , , , , , , ] 2023-05-31T20:49:53.251 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:49:53.246+0000 7ff2db554780 1 waiting for incremental sync to catch up: 2023-05-31T20:49:53.251 INFO:teuthology.orchestra.run.smithi099.stderr: local status: [00000000002.83.6, 00000000001.68.6, , , , , 00000000001.60.6, , , , , , , , , , , 00000000002.50.6, , , , , , , , , , , ] 2023-05-31T20:49:53.251 INFO:teuthology.orchestra.run.smithi099.stderr: remote markers: [00000000002.83.6, 00000000001.68.6, , , , , 00000000001.60.6, , , , , , , , , , , 00000000002.50.6, 00000000001.102.6, , , , , , , , , , ] 2023-05-31T20:49:58.253 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:49:58.248+0000 7ff2db554780 1 waiting for incremental sync to catch up: 2023-05-31T20:49:58.253 INFO:teuthology.orchestra.run.smithi099.stderr: local status: [00000000002.83.6, 00000000001.68.6, , , , , 00000000001.60.6, , , , , , , , , , , 00000000002.50.6, , , , , , , , , , , ] 2023-05-31T20:49:58.253 INFO:teuthology.orchestra.run.smithi099.stderr: remote markers: [00000000002.83.6, 00000000001.68.6, , , , , 00000000001.60.6, , , , , , , , , , , 00000000002.50.6, 00000000001.102.6, , , , , , , , , , ] 2023-05-31T20:50:03.255 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:50:03.250+0000 7ff2db554780 1 waiting for incremental sync to catch up: 2023-05-31T20:50:03.256 INFO:teuthology.orchestra.run.smithi099.stderr: local status: [00000000002.83.6, 00000000001.68.6, , , , , 00000000001.60.6, , , , , , , , , , , 00000000002.50.6, , , , , , , , , , , ] 2023-05-31T20:50:03.257 INFO:teuthology.orchestra.run.smithi099.stderr: remote markers: [00000000002.83.6, 00000000001.68.6, , , , , 00000000001.60.6, , , , , , , , , , , 00000000002.50.6, 00000000001.102.6, , , , , , , , , , ] 2023-05-31T20:50:08.257 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:50:08.253+0000 7ff2db554780 1 bucket sync caught up with source: 2023-05-31T20:50:08.258 INFO:teuthology.orchestra.run.smithi099.stderr: local status: [00000000002.83.6, 00000000001.68.6, , , , , 00000000001.60.6, , , , , , , , , , , 00000000002.50.6, 00000000001.102.6, , , , , , , , , , ] 2023-05-31T20:50:08.258 INFO:teuthology.orchestra.run.smithi099.stderr: remote markers: [00000000002.83.6, 00000000001.68.6, , , , , 00000000001.60.6, , , , , , , , , , , 00000000002.50.6, 00000000001.102.6, , , , , , , , , , ] 2023-05-31T20:50:08.258 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:50:08.253+0000 7ff2db554780 0 bucket checkpoint complete 2023-05-31T20:50:08.282 DEBUG:rgw_multi.tests:bucket checkpoint: source=a2 target=a1 bucket=pklxkd-59 2023-05-31T20:50:08.283 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'pklxkd-59', '--source-zone', 'a2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-05-31T20:50:08.283 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'pklxkd-59', '--source-zone', 'a2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-05-31T20:50:08.284 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket pklxkd-59 --source-zone a2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2023-05-31T20:50:08.345 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2023-05-31T20:50:08.345 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2023-05-31T20:50:08.409 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:50:08.405+0000 7f8d83a0e780 1 waiting for incremental sync to catch up: 2023-05-31T20:50:08.410 INFO:teuthology.orchestra.run.smithi099.stderr: local status: [, 00000000002.3584.6, 00000000002.3906.6, 00000000001.2027.6, 00000000003.60.6, 00000000004.2031.6, , 00000000001.3914.6, 00000000002.2013.6, 00000000005.2299.6, 00000000004.2018.6] 2023-05-31T20:50:08.410 INFO:teuthology.orchestra.run.smithi099.stderr: remote markers: [, 00000000002.3584.6, 00000000002.3906.6, 00000000001.2027.6, 00000000003.60.6, 00000000004.2031.6, 00000000001.377.6, 00000000001.3914.6, 00000000002.2013.6, 00000000005.2299.6, 00000000004.2018.6] 2023-05-31T20:50:13.411 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:50:13.407+0000 7f8d83a0e780 1 waiting for incremental sync to catch up: 2023-05-31T20:50:13.411 INFO:teuthology.orchestra.run.smithi099.stderr: local status: [, 00000000002.3584.6, 00000000002.3906.6, 00000000001.2027.6, 00000000003.60.6, 00000000004.2031.6, , 00000000001.3914.6, 00000000002.2013.6, 00000000005.2299.6, 00000000004.2018.6] 2023-05-31T20:50:13.412 INFO:teuthology.orchestra.run.smithi099.stderr: remote markers: [, 00000000002.3584.6, 00000000002.3906.6, 00000000001.2027.6, 00000000003.60.6, 00000000004.2031.6, 00000000001.377.6, 00000000001.3914.6, 00000000002.2013.6, 00000000005.2299.6, 00000000004.2018.6] 2023-05-31T20:50:18.413 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:50:18.408+0000 7f8d83a0e780 1 waiting for incremental sync to catch up: 2023-05-31T20:50:18.413 INFO:teuthology.orchestra.run.smithi099.stderr: local status: [, 00000000002.3584.6, 00000000002.3906.6, 00000000001.2027.6, 00000000003.60.6, 00000000004.2031.6, , 00000000001.3914.6, 00000000002.2013.6, 00000000005.2299.6, 00000000004.2018.6] 2023-05-31T20:50:18.413 INFO:teuthology.orchestra.run.smithi099.stderr: remote markers: [, 00000000002.3584.6, 00000000002.3906.6, 00000000001.2027.6, 00000000003.60.6, 00000000004.2031.6, 00000000001.377.6, 00000000001.3914.6, 00000000002.2013.6, 00000000005.2299.6, 00000000004.2018.6] 2023-05-31T20:50:23.414 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:50:23.410+0000 7f8d83a0e780 1 waiting for incremental sync to catch up: 2023-05-31T20:50:23.414 INFO:teuthology.orchestra.run.smithi099.stderr: local status: [, 00000000002.3584.6, 00000000002.3906.6, 00000000001.2027.6, 00000000003.60.6, 00000000004.2031.6, , 00000000001.3914.6, 00000000002.2013.6, 00000000005.2299.6, 00000000004.2018.6] 2023-05-31T20:50:23.414 INFO:teuthology.orchestra.run.smithi099.stderr: remote markers: [, 00000000002.3584.6, 00000000002.3906.6, 00000000001.2027.6, 00000000003.60.6, 00000000004.2031.6, 00000000001.377.6, 00000000001.3914.6, 00000000002.2013.6, 00000000005.2299.6, 00000000004.2018.6] 2023-05-31T20:50:28.415 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:50:28.410+0000 7f8d83a0e780 1 bucket sync caught up with source: 2023-05-31T20:50:28.415 INFO:teuthology.orchestra.run.smithi099.stderr: local status: [, 00000000002.3584.6, 00000000002.3906.6, 00000000001.2027.6, 00000000003.60.6, 00000000004.2031.6, 00000000001.377.6, 00000000001.3914.6, 00000000002.2013.6, 00000000005.2299.6, 00000000004.2018.6] 2023-05-31T20:50:28.416 INFO:teuthology.orchestra.run.smithi099.stderr: remote markers: [, 00000000002.3584.6, 00000000002.3906.6, 00000000001.2027.6, 00000000003.60.6, 00000000004.2031.6, 00000000001.377.6, 00000000001.3914.6, 00000000002.2013.6, 00000000005.2299.6, 00000000004.2018.6] 2023-05-31T20:50:28.416 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:50:28.410+0000 7f8d83a0e780 0 bucket checkpoint complete 2023-05-31T20:50:28.433 INFO:tasks.rgw_multi.zone_rados:comparing bucket=pklxkd-59 zones={a2, a1} 2023-05-31T20:50:28.433 DEBUG:boto:path=/pklxkd-59/ 2023-05-31T20:50:28.434 DEBUG:boto:auth_path=/pklxkd-59/ 2023-05-31T20:50:28.434 DEBUG:boto:Method: HEAD 2023-05-31T20:50:28.434 DEBUG:boto:Path: /pklxkd-59/ 2023-05-31T20:50:28.435 DEBUG:boto:Data: 2023-05-31T20:50:28.435 DEBUG:boto:Headers: {} 2023-05-31T20:50:28.435 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:50:28.436 DEBUG:boto:Port: 8001 2023-05-31T20:50:28.436 DEBUG:boto:Params: {} 2023-05-31T20:50:28.437 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8001} 2023-05-31T20:50:28.437 DEBUG:boto:Token: None 2023-05-31T20:50:28.438 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:50:28 GMT /pklxkd-59/ 2023-05-31T20:50:28.438 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:rxslnWAeSDdl7LrXS+Ex905Ol0Q= 2023-05-31T20:50:28.439 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:50:28 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:rxslnWAeSDdl7LrXS+Ex905Ol0Q=', 'Content-Length': '0'} 2023-05-31T20:50:28.442 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '25'), ('X-RGW-Bytes-Used', '75'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000368475f18aa56786-006477b314-1c17-a2'), ('Content-Length', '0'), ('Date', 'Wed, 31 May 2023 20:50:28 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:50:28.443 DEBUG:boto:path=/pklxkd-59/ 2023-05-31T20:50:28.443 DEBUG:boto:auth_path=/pklxkd-59/ 2023-05-31T20:50:28.443 DEBUG:boto:Method: HEAD 2023-05-31T20:50:28.444 DEBUG:boto:Path: /pklxkd-59/ 2023-05-31T20:50:28.444 DEBUG:boto:Data: 2023-05-31T20:50:28.444 DEBUG:boto:Headers: {} 2023-05-31T20:50:28.445 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:50:28.445 DEBUG:boto:Port: 8000 2023-05-31T20:50:28.445 DEBUG:boto:Params: {} 2023-05-31T20:50:28.446 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2023-05-31T20:50:28.446 DEBUG:boto:Token: None 2023-05-31T20:50:28.447 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:50:28 GMT /pklxkd-59/ 2023-05-31T20:50:28.447 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:rxslnWAeSDdl7LrXS+Ex905Ol0Q= 2023-05-31T20:50:28.448 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:50:28 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:rxslnWAeSDdl7LrXS+Ex905Ol0Q=', 'Content-Length': '0'} 2023-05-31T20:50:28.452 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '25'), ('X-RGW-Bytes-Used', '75'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000b13e05b5f8dc8b9a-006477b314-1630-a1'), ('Content-Length', '0'), ('Date', 'Wed, 31 May 2023 20:50:28 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:50:28.452 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2023-05-31T20:50:28.453 DEBUG:boto:path=/pklxkd-59/ 2023-05-31T20:50:28.453 DEBUG:boto:auth_path=/pklxkd-59/ 2023-05-31T20:50:28.453 DEBUG:boto:path=/pklxkd-59/?versions&max-keys=999 2023-05-31T20:50:28.454 DEBUG:boto:auth_path=/pklxkd-59/?versions&max-keys=999 2023-05-31T20:50:28.454 DEBUG:boto:Method: GET 2023-05-31T20:50:28.455 DEBUG:boto:Path: /pklxkd-59/?versions&max-keys=999 2023-05-31T20:50:28.455 DEBUG:boto:Data: 2023-05-31T20:50:28.455 DEBUG:boto:Headers: {} 2023-05-31T20:50:28.456 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:50:28.456 DEBUG:boto:Port: 8001 2023-05-31T20:50:28.457 DEBUG:boto:Params: {} 2023-05-31T20:50:28.457 DEBUG:boto:Token: None 2023-05-31T20:50:28.458 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:50:28 GMT /pklxkd-59/?versions 2023-05-31T20:50:28.458 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:n4eONMqxTQaJZg/c3Xl4SURVQA8= 2023-05-31T20:50:28.459 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:50:28 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:n4eONMqxTQaJZg/c3Xl4SURVQA8=', 'Content-Length': '0'} 2023-05-31T20:50:28.461 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000931142bf5827ac40-006477b314-1c17-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:50:28 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:50:28.462 DEBUG:boto:b'pklxkd-59999falseobj11nulltrue2023-05-31T20:39:26.427Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj115nulltrue2023-05-31T20:46:33.958Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj116nulltrue2023-05-31T20:48:13.052Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj145nulltrue2023-05-31T20:48:13.107Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj17nulltrue2023-05-31T20:43:07.977Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj19nulltrue2023-05-31T20:44:50.557Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj22nulltrue2023-05-31T20:39:26.473Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj23nulltrue2023-05-31T20:46:33.813Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj29nulltrue2023-05-31T20:48:12.956Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj33nulltrue2023-05-31T20:39:26.534Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj34nulltrue2023-05-31T20:43:07.994Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj38nulltrue2023-05-31T20:44:50.610Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj44nulltrue2023-05-31T20:39:26.565Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj46nulltrue2023-05-31T20:46:33.830Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj51nulltrue2023-05-31T20:43:08.011Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj55nulltrue2023-05-31T20:39:26.630Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj57nulltrue2023-05-31T20:44:50.627Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj58nulltrue2023-05-31T20:48:12.976Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj68nulltrue2023-05-31T20:43:08.066Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj69nulltrue2023-05-31T20:46:33.886Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj76nulltrue2023-05-31T20:44:50.682Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj85nulltrue2023-05-31T20:43:08.086Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj87nulltrue2023-05-31T20:48:13.031Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj92nulltrue2023-05-31T20:46:33.904Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj95nulltrue2023-05-31T20:44:50.705Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:50:28.464 DEBUG:tasks.rgw_multi.zone_rados:o=obj11 2023-05-31T20:50:28.464 DEBUG:tasks.rgw_multi.zone_rados:o=obj115 2023-05-31T20:50:28.464 DEBUG:tasks.rgw_multi.zone_rados:o=obj116 2023-05-31T20:50:28.465 DEBUG:tasks.rgw_multi.zone_rados:o=obj145 2023-05-31T20:50:28.465 DEBUG:tasks.rgw_multi.zone_rados:o=obj17 2023-05-31T20:50:28.465 DEBUG:tasks.rgw_multi.zone_rados:o=obj19 2023-05-31T20:50:28.466 DEBUG:tasks.rgw_multi.zone_rados:o=obj22 2023-05-31T20:50:28.466 DEBUG:tasks.rgw_multi.zone_rados:o=obj23 2023-05-31T20:50:28.466 DEBUG:tasks.rgw_multi.zone_rados:o=obj29 2023-05-31T20:50:28.467 DEBUG:tasks.rgw_multi.zone_rados:o=obj33 2023-05-31T20:50:28.467 DEBUG:tasks.rgw_multi.zone_rados:o=obj34 2023-05-31T20:50:28.467 DEBUG:tasks.rgw_multi.zone_rados:o=obj38 2023-05-31T20:50:28.468 DEBUG:tasks.rgw_multi.zone_rados:o=obj44 2023-05-31T20:50:28.468 DEBUG:tasks.rgw_multi.zone_rados:o=obj46 2023-05-31T20:50:28.469 DEBUG:tasks.rgw_multi.zone_rados:o=obj51 2023-05-31T20:50:28.469 DEBUG:tasks.rgw_multi.zone_rados:o=obj55 2023-05-31T20:50:28.469 DEBUG:tasks.rgw_multi.zone_rados:o=obj57 2023-05-31T20:50:28.469 DEBUG:tasks.rgw_multi.zone_rados:o=obj58 2023-05-31T20:50:28.470 DEBUG:tasks.rgw_multi.zone_rados:o=obj68 2023-05-31T20:50:28.470 DEBUG:tasks.rgw_multi.zone_rados:o=obj69 2023-05-31T20:50:28.471 DEBUG:tasks.rgw_multi.zone_rados:o=obj76 2023-05-31T20:50:28.471 DEBUG:tasks.rgw_multi.zone_rados:o=obj85 2023-05-31T20:50:28.471 DEBUG:tasks.rgw_multi.zone_rados:o=obj87 2023-05-31T20:50:28.471 DEBUG:tasks.rgw_multi.zone_rados:o=obj92 2023-05-31T20:50:28.472 DEBUG:tasks.rgw_multi.zone_rados:o=obj95 2023-05-31T20:50:28.472 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2023-05-31T20:50:28.473 DEBUG:boto:path=/pklxkd-59/ 2023-05-31T20:50:28.473 DEBUG:boto:auth_path=/pklxkd-59/ 2023-05-31T20:50:28.473 DEBUG:boto:path=/pklxkd-59/?versions&max-keys=999 2023-05-31T20:50:28.474 DEBUG:boto:auth_path=/pklxkd-59/?versions&max-keys=999 2023-05-31T20:50:28.474 DEBUG:boto:Method: GET 2023-05-31T20:50:28.474 DEBUG:boto:Path: /pklxkd-59/?versions&max-keys=999 2023-05-31T20:50:28.475 DEBUG:boto:Data: 2023-05-31T20:50:28.475 DEBUG:boto:Headers: {} 2023-05-31T20:50:28.476 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:50:28.476 DEBUG:boto:Port: 8000 2023-05-31T20:50:28.477 DEBUG:boto:Params: {} 2023-05-31T20:50:28.477 DEBUG:boto:Token: None 2023-05-31T20:50:28.478 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:50:28 GMT /pklxkd-59/?versions 2023-05-31T20:50:28.478 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:n4eONMqxTQaJZg/c3Xl4SURVQA8= 2023-05-31T20:50:28.479 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:50:28 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:n4eONMqxTQaJZg/c3Xl4SURVQA8=', 'Content-Length': '0'} 2023-05-31T20:50:28.482 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000007c36004189bf5f15-006477b314-1630-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:50:28 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:50:28.483 DEBUG:boto:b'pklxkd-59999falseobj11nulltrue2023-05-31T20:39:26.427Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj115nulltrue2023-05-31T20:46:33.958Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj116nulltrue2023-05-31T20:48:13.052Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj145nulltrue2023-05-31T20:48:13.107Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj17nulltrue2023-05-31T20:43:07.977Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj19nulltrue2023-05-31T20:44:50.557Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj22nulltrue2023-05-31T20:39:26.473Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj23nulltrue2023-05-31T20:46:33.813Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj29nulltrue2023-05-31T20:48:12.956Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj33nulltrue2023-05-31T20:39:26.534Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj34nulltrue2023-05-31T20:43:07.994Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj38nulltrue2023-05-31T20:44:50.610Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj44nulltrue2023-05-31T20:39:26.565Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj46nulltrue2023-05-31T20:46:33.830Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj51nulltrue2023-05-31T20:43:08.011Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj55nulltrue2023-05-31T20:39:26.630Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj57nulltrue2023-05-31T20:44:50.627Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj58nulltrue2023-05-31T20:48:12.976Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj68nulltrue2023-05-31T20:43:08.066Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj69nulltrue2023-05-31T20:46:33.886Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj76nulltrue2023-05-31T20:44:50.682Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj85nulltrue2023-05-31T20:43:08.086Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj87nulltrue2023-05-31T20:48:13.031Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj92nulltrue2023-05-31T20:46:33.904Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj95nulltrue2023-05-31T20:44:50.705Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:50:28.484 DEBUG:tasks.rgw_multi.zone_rados:o=obj11 2023-05-31T20:50:28.485 DEBUG:tasks.rgw_multi.zone_rados:o=obj115 2023-05-31T20:50:28.485 DEBUG:tasks.rgw_multi.zone_rados:o=obj116 2023-05-31T20:50:28.485 DEBUG:tasks.rgw_multi.zone_rados:o=obj145 2023-05-31T20:50:28.486 DEBUG:tasks.rgw_multi.zone_rados:o=obj17 2023-05-31T20:50:28.486 DEBUG:tasks.rgw_multi.zone_rados:o=obj19 2023-05-31T20:50:28.487 DEBUG:tasks.rgw_multi.zone_rados:o=obj22 2023-05-31T20:50:28.487 DEBUG:tasks.rgw_multi.zone_rados:o=obj23 2023-05-31T20:50:28.487 DEBUG:tasks.rgw_multi.zone_rados:o=obj29 2023-05-31T20:50:28.488 DEBUG:tasks.rgw_multi.zone_rados:o=obj33 2023-05-31T20:50:28.488 DEBUG:tasks.rgw_multi.zone_rados:o=obj34 2023-05-31T20:50:28.488 DEBUG:tasks.rgw_multi.zone_rados:o=obj38 2023-05-31T20:50:28.489 DEBUG:tasks.rgw_multi.zone_rados:o=obj44 2023-05-31T20:50:28.489 DEBUG:tasks.rgw_multi.zone_rados:o=obj46 2023-05-31T20:50:28.489 DEBUG:tasks.rgw_multi.zone_rados:o=obj51 2023-05-31T20:50:28.490 DEBUG:tasks.rgw_multi.zone_rados:o=obj55 2023-05-31T20:50:28.490 DEBUG:tasks.rgw_multi.zone_rados:o=obj57 2023-05-31T20:50:28.490 DEBUG:tasks.rgw_multi.zone_rados:o=obj58 2023-05-31T20:50:28.491 DEBUG:tasks.rgw_multi.zone_rados:o=obj68 2023-05-31T20:50:28.491 DEBUG:tasks.rgw_multi.zone_rados:o=obj69 2023-05-31T20:50:28.491 DEBUG:tasks.rgw_multi.zone_rados:o=obj76 2023-05-31T20:50:28.491 DEBUG:tasks.rgw_multi.zone_rados:o=obj85 2023-05-31T20:50:28.492 DEBUG:tasks.rgw_multi.zone_rados:o=obj87 2023-05-31T20:50:28.492 DEBUG:tasks.rgw_multi.zone_rados:o=obj92 2023-05-31T20:50:28.492 DEBUG:tasks.rgw_multi.zone_rados:o=obj95 2023-05-31T20:50:28.493 DEBUG:boto:path=/pklxkd-59/ 2023-05-31T20:50:28.493 DEBUG:boto:auth_path=/pklxkd-59/ 2023-05-31T20:50:28.494 DEBUG:boto:path=/pklxkd-59/?versions&max-keys=999 2023-05-31T20:50:28.494 DEBUG:boto:auth_path=/pklxkd-59/?versions&max-keys=999 2023-05-31T20:50:28.494 DEBUG:boto:Method: GET 2023-05-31T20:50:28.495 DEBUG:boto:Path: /pklxkd-59/?versions&max-keys=999 2023-05-31T20:50:28.495 DEBUG:boto:Data: 2023-05-31T20:50:28.496 DEBUG:boto:Headers: {} 2023-05-31T20:50:28.496 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:50:28.496 DEBUG:boto:Port: 8001 2023-05-31T20:50:28.497 DEBUG:boto:Params: {} 2023-05-31T20:50:28.497 DEBUG:boto:Token: None 2023-05-31T20:50:28.498 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:50:28 GMT /pklxkd-59/?versions 2023-05-31T20:50:28.498 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:n4eONMqxTQaJZg/c3Xl4SURVQA8= 2023-05-31T20:50:28.499 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:50:28 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:n4eONMqxTQaJZg/c3Xl4SURVQA8=', 'Content-Length': '0'} 2023-05-31T20:50:28.501 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f7f3a979dc2c3bbc-006477b314-1c17-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:50:28 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:50:28.502 DEBUG:boto:b'pklxkd-59999falseobj11nulltrue2023-05-31T20:39:26.427Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj115nulltrue2023-05-31T20:46:33.958Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj116nulltrue2023-05-31T20:48:13.052Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj145nulltrue2023-05-31T20:48:13.107Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj17nulltrue2023-05-31T20:43:07.977Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj19nulltrue2023-05-31T20:44:50.557Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj22nulltrue2023-05-31T20:39:26.473Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj23nulltrue2023-05-31T20:46:33.813Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj29nulltrue2023-05-31T20:48:12.956Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj33nulltrue2023-05-31T20:39:26.534Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj34nulltrue2023-05-31T20:43:07.994Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj38nulltrue2023-05-31T20:44:50.610Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj44nulltrue2023-05-31T20:39:26.565Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj46nulltrue2023-05-31T20:46:33.830Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj51nulltrue2023-05-31T20:43:08.011Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj55nulltrue2023-05-31T20:39:26.630Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj57nulltrue2023-05-31T20:44:50.627Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj58nulltrue2023-05-31T20:48:12.976Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj68nulltrue2023-05-31T20:43:08.066Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj69nulltrue2023-05-31T20:46:33.886Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj76nulltrue2023-05-31T20:44:50.682Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj85nulltrue2023-05-31T20:43:08.086Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj87nulltrue2023-05-31T20:48:13.031Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj92nulltrue2023-05-31T20:46:33.904Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj95nulltrue2023-05-31T20:44:50.705Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:50:28.504 DEBUG:boto:path=/pklxkd-59/ 2023-05-31T20:50:28.504 DEBUG:boto:auth_path=/pklxkd-59/ 2023-05-31T20:50:28.505 DEBUG:boto:path=/pklxkd-59/?versions&max-keys=999 2023-05-31T20:50:28.505 DEBUG:boto:auth_path=/pklxkd-59/?versions&max-keys=999 2023-05-31T20:50:28.505 DEBUG:boto:Method: GET 2023-05-31T20:50:28.506 DEBUG:boto:Path: /pklxkd-59/?versions&max-keys=999 2023-05-31T20:50:28.506 DEBUG:boto:Data: 2023-05-31T20:50:28.507 DEBUG:boto:Headers: {} 2023-05-31T20:50:28.507 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:50:28.508 DEBUG:boto:Port: 8000 2023-05-31T20:50:28.508 DEBUG:boto:Params: {} 2023-05-31T20:50:28.509 DEBUG:boto:Token: None 2023-05-31T20:50:28.509 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:50:28 GMT /pklxkd-59/?versions 2023-05-31T20:50:28.509 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:n4eONMqxTQaJZg/c3Xl4SURVQA8= 2023-05-31T20:50:28.510 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:50:28 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:n4eONMqxTQaJZg/c3Xl4SURVQA8=', 'Content-Length': '0'} 2023-05-31T20:50:28.514 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000875029d4a2962b43-006477b314-1630-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:50:28 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:50:28.514 DEBUG:boto:b'pklxkd-59999falseobj11nulltrue2023-05-31T20:39:26.427Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj115nulltrue2023-05-31T20:46:33.958Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj116nulltrue2023-05-31T20:48:13.052Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj145nulltrue2023-05-31T20:48:13.107Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj17nulltrue2023-05-31T20:43:07.977Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj19nulltrue2023-05-31T20:44:50.557Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj22nulltrue2023-05-31T20:39:26.473Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj23nulltrue2023-05-31T20:46:33.813Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj29nulltrue2023-05-31T20:48:12.956Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj33nulltrue2023-05-31T20:39:26.534Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj34nulltrue2023-05-31T20:43:07.994Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj38nulltrue2023-05-31T20:44:50.610Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj44nulltrue2023-05-31T20:39:26.565Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj46nulltrue2023-05-31T20:46:33.830Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj51nulltrue2023-05-31T20:43:08.011Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj55nulltrue2023-05-31T20:39:26.630Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj57nulltrue2023-05-31T20:44:50.627Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj58nulltrue2023-05-31T20:48:12.976Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj68nulltrue2023-05-31T20:43:08.066Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj69nulltrue2023-05-31T20:46:33.886Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj76nulltrue2023-05-31T20:44:50.682Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj85nulltrue2023-05-31T20:43:08.086Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj87nulltrue2023-05-31T20:48:13.031Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj92nulltrue2023-05-31T20:46:33.904Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj95nulltrue2023-05-31T20:44:50.705Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:50:28.516 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj11 2023-05-31T20:50:28.516 DEBUG:boto:path=/pklxkd-59/obj11 2023-05-31T20:50:28.517 DEBUG:boto:auth_path=/pklxkd-59/obj11 2023-05-31T20:50:28.517 DEBUG:boto:path=/pklxkd-59/obj11?versionId=null 2023-05-31T20:50:28.517 DEBUG:boto:auth_path=/pklxkd-59/obj11?versionId=null 2023-05-31T20:50:28.518 DEBUG:boto:Method: GET 2023-05-31T20:50:28.518 DEBUG:boto:Path: /pklxkd-59/obj11?versionId=null 2023-05-31T20:50:28.519 DEBUG:boto:Data: 2023-05-31T20:50:28.519 DEBUG:boto:Headers: {} 2023-05-31T20:50:28.519 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:50:28.520 DEBUG:boto:Port: 8001 2023-05-31T20:50:28.520 DEBUG:boto:Params: {} 2023-05-31T20:50:28.520 DEBUG:boto:Token: None 2023-05-31T20:50:28.521 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:50:28 GMT /pklxkd-59/obj11?versionId=null 2023-05-31T20:50:28.521 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:cdXPeTakfokKLi25YeYzvZ0zVak= 2023-05-31T20:50:28.522 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:50:28 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:cdXPeTakfokKLi25YeYzvZ0zVak=', 'Content-Length': '0'} 2023-05-31T20:50:28.523 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:39:26 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000095effc57a5821d21-006477b314-1c17-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:50:28 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:50:28.524 DEBUG:boto:path=/pklxkd-59/obj11 2023-05-31T20:50:28.525 DEBUG:boto:auth_path=/pklxkd-59/obj11 2023-05-31T20:50:28.525 DEBUG:boto:path=/pklxkd-59/obj11?versionId=null 2023-05-31T20:50:28.525 DEBUG:boto:auth_path=/pklxkd-59/obj11?versionId=null 2023-05-31T20:50:28.526 DEBUG:boto:Method: GET 2023-05-31T20:50:28.526 DEBUG:boto:Path: /pklxkd-59/obj11?versionId=null 2023-05-31T20:50:28.526 DEBUG:boto:Data: 2023-05-31T20:50:28.527 DEBUG:boto:Headers: {} 2023-05-31T20:50:28.527 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:50:28.527 DEBUG:boto:Port: 8000 2023-05-31T20:50:28.528 DEBUG:boto:Params: {} 2023-05-31T20:50:28.528 DEBUG:boto:Token: None 2023-05-31T20:50:28.529 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:50:28 GMT /pklxkd-59/obj11?versionId=null 2023-05-31T20:50:28.529 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:cdXPeTakfokKLi25YeYzvZ0zVak= 2023-05-31T20:50:28.529 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:50:28 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:cdXPeTakfokKLi25YeYzvZ0zVak=', 'Content-Length': '0'} 2023-05-31T20:50:28.532 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:39:26 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000060758ba00871b99-006477b314-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:50:28 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:50:28.532 DEBUG:boto:path=/pklxkd-59/obj11 2023-05-31T20:50:28.533 DEBUG:boto:auth_path=/pklxkd-59/obj11 2023-05-31T20:50:28.533 DEBUG:boto:path=/pklxkd-59/obj11?versionId=null 2023-05-31T20:50:28.533 DEBUG:boto:auth_path=/pklxkd-59/obj11?versionId=null 2023-05-31T20:50:28.534 DEBUG:boto:Method: HEAD 2023-05-31T20:50:28.534 DEBUG:boto:Path: /pklxkd-59/obj11?versionId=null 2023-05-31T20:50:28.534 DEBUG:boto:Data: 2023-05-31T20:50:28.535 DEBUG:boto:Headers: {} 2023-05-31T20:50:28.535 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:50:28.535 DEBUG:boto:Port: 8001 2023-05-31T20:50:28.536 DEBUG:boto:Params: {} 2023-05-31T20:50:28.536 DEBUG:boto:Token: None 2023-05-31T20:50:28.537 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:50:28 GMT /pklxkd-59/obj11?versionId=null 2023-05-31T20:50:28.537 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:PrOvCQeONr5ohq/5fhdq+jhBUSc= 2023-05-31T20:50:28.537 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:50:28 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:PrOvCQeONr5ohq/5fhdq+jhBUSc=', 'Content-Length': '0'} 2023-05-31T20:50:28.539 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:39:26 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000087180fc7dd09fc47-006477b314-1c17-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:50:28 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:50:28.540 DEBUG:boto:path=/pklxkd-59/obj11 2023-05-31T20:50:28.540 DEBUG:boto:auth_path=/pklxkd-59/obj11 2023-05-31T20:50:28.540 DEBUG:boto:path=/pklxkd-59/obj11?versionId=null 2023-05-31T20:50:28.541 DEBUG:boto:auth_path=/pklxkd-59/obj11?versionId=null 2023-05-31T20:50:28.541 DEBUG:boto:Method: HEAD 2023-05-31T20:50:28.541 DEBUG:boto:Path: /pklxkd-59/obj11?versionId=null 2023-05-31T20:50:28.542 DEBUG:boto:Data: 2023-05-31T20:50:28.542 DEBUG:boto:Headers: {} 2023-05-31T20:50:28.543 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:50:28.543 DEBUG:boto:Port: 8000 2023-05-31T20:50:28.543 DEBUG:boto:Params: {} 2023-05-31T20:50:28.544 DEBUG:boto:Token: None 2023-05-31T20:50:28.544 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:50:28 GMT /pklxkd-59/obj11?versionId=null 2023-05-31T20:50:28.544 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:PrOvCQeONr5ohq/5fhdq+jhBUSc= 2023-05-31T20:50:28.545 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:50:28 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:PrOvCQeONr5ohq/5fhdq+jhBUSc=', 'Content-Length': '0'} 2023-05-31T20:50:28.547 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:39:26 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000a9c2744f93bf57a-006477b314-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:50:28 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:50:28.547 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj11 2023-05-31T20:50:28.548 DEBUG:boto:path=/pklxkd-59/obj11 2023-05-31T20:50:28.548 DEBUG:boto:auth_path=/pklxkd-59/obj11 2023-05-31T20:50:28.548 DEBUG:boto:path=/pklxkd-59/obj11?versionId=null 2023-05-31T20:50:28.549 DEBUG:boto:auth_path=/pklxkd-59/obj11?versionId=null 2023-05-31T20:50:28.549 DEBUG:boto:Method: GET 2023-05-31T20:50:28.549 DEBUG:boto:Path: /pklxkd-59/obj11?versionId=null 2023-05-31T20:50:28.550 DEBUG:boto:Data: 2023-05-31T20:50:28.550 DEBUG:boto:Headers: {} 2023-05-31T20:50:28.550 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:50:28.551 DEBUG:boto:Port: 8001 2023-05-31T20:50:28.551 DEBUG:boto:Params: {} 2023-05-31T20:50:28.552 DEBUG:boto:Token: None 2023-05-31T20:50:28.552 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:50:28 GMT /pklxkd-59/obj11?versionId=null 2023-05-31T20:50:28.552 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:cdXPeTakfokKLi25YeYzvZ0zVak= 2023-05-31T20:50:28.553 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:50:28 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:cdXPeTakfokKLi25YeYzvZ0zVak=', 'Content-Length': '0'} 2023-05-31T20:50:28.555 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:39:26 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d615647f549fb770-006477b314-1c17-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:50:28 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:50:28.555 DEBUG:boto:path=/pklxkd-59/obj11 2023-05-31T20:50:28.556 DEBUG:boto:auth_path=/pklxkd-59/obj11 2023-05-31T20:50:28.556 DEBUG:boto:path=/pklxkd-59/obj11?versionId=null 2023-05-31T20:50:28.556 DEBUG:boto:auth_path=/pklxkd-59/obj11?versionId=null 2023-05-31T20:50:28.557 DEBUG:boto:Method: GET 2023-05-31T20:50:28.557 DEBUG:boto:Path: /pklxkd-59/obj11?versionId=null 2023-05-31T20:50:28.557 DEBUG:boto:Data: 2023-05-31T20:50:28.558 DEBUG:boto:Headers: {} 2023-05-31T20:50:28.558 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:50:28.559 DEBUG:boto:Port: 8000 2023-05-31T20:50:28.559 DEBUG:boto:Params: {} 2023-05-31T20:50:28.559 DEBUG:boto:Token: None 2023-05-31T20:50:28.560 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:50:28 GMT /pklxkd-59/obj11?versionId=null 2023-05-31T20:50:28.560 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:cdXPeTakfokKLi25YeYzvZ0zVak= 2023-05-31T20:50:28.561 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:50:28 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:cdXPeTakfokKLi25YeYzvZ0zVak=', 'Content-Length': '0'} 2023-05-31T20:50:28.563 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:39:26 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000046689c83e255d753-006477b314-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:50:28 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:50:28.563 DEBUG:boto:path=/pklxkd-59/ 2023-05-31T20:50:28.564 DEBUG:boto:auth_path=/pklxkd-59/ 2023-05-31T20:50:28.564 DEBUG:boto:path=/pklxkd-59/?prefix=obj11 2023-05-31T20:50:28.565 DEBUG:boto:auth_path=/pklxkd-59/?prefix=obj11 2023-05-31T20:50:28.565 DEBUG:boto:Method: GET 2023-05-31T20:50:28.565 DEBUG:boto:Path: /pklxkd-59/?prefix=obj11 2023-05-31T20:50:28.566 DEBUG:boto:Data: 2023-05-31T20:50:28.566 DEBUG:boto:Headers: {} 2023-05-31T20:50:28.566 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:50:28.567 DEBUG:boto:Port: 8001 2023-05-31T20:50:28.567 DEBUG:boto:Params: {} 2023-05-31T20:50:28.567 DEBUG:boto:Token: None 2023-05-31T20:50:28.568 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:50:28 GMT /pklxkd-59/ 2023-05-31T20:50:28.568 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:lLtg7I9o4pC7jMarhXaP1H+yh9o= 2023-05-31T20:50:28.569 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:50:28 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:lLtg7I9o4pC7jMarhXaP1H+yh9o=', 'Content-Length': '0'} 2023-05-31T20:50:28.571 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000368a52990c40e677-006477b314-1c17-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:50:28 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:50:28.571 DEBUG:boto:b'pklxkd-59obj111000falseobj112023-05-31T20:39:26.427Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj1152023-05-31T20:46:33.958Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj1162023-05-31T20:48:13.052Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:50:28.572 DEBUG:boto:path=/pklxkd-59/ 2023-05-31T20:50:28.573 DEBUG:boto:auth_path=/pklxkd-59/ 2023-05-31T20:50:28.573 DEBUG:boto:path=/pklxkd-59/?prefix=obj11 2023-05-31T20:50:28.573 DEBUG:boto:auth_path=/pklxkd-59/?prefix=obj11 2023-05-31T20:50:28.574 DEBUG:boto:Method: GET 2023-05-31T20:50:28.574 DEBUG:boto:Path: /pklxkd-59/?prefix=obj11 2023-05-31T20:50:28.574 DEBUG:boto:Data: 2023-05-31T20:50:28.575 DEBUG:boto:Headers: {} 2023-05-31T20:50:28.575 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:50:28.575 DEBUG:boto:Port: 8000 2023-05-31T20:50:28.576 DEBUG:boto:Params: {} 2023-05-31T20:50:28.576 DEBUG:boto:Token: None 2023-05-31T20:50:28.577 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:50:28 GMT /pklxkd-59/ 2023-05-31T20:50:28.577 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:lLtg7I9o4pC7jMarhXaP1H+yh9o= 2023-05-31T20:50:28.577 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:50:28 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:lLtg7I9o4pC7jMarhXaP1H+yh9o=', 'Content-Length': '0'} 2023-05-31T20:50:28.581 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000004041c2b0b9a3e106-006477b314-1630-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:50:28 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:50:28.581 DEBUG:boto:b'pklxkd-59obj111000falseobj112023-05-31T20:39:26.427Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj1152023-05-31T20:46:33.958Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj1162023-05-31T20:48:13.052Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:50:28.582 DEBUG:boto:path=/pklxkd-59/obj11 2023-05-31T20:50:28.582 DEBUG:boto:auth_path=/pklxkd-59/obj11 2023-05-31T20:50:28.583 DEBUG:boto:Method: HEAD 2023-05-31T20:50:28.583 DEBUG:boto:Path: /pklxkd-59/obj11 2023-05-31T20:50:28.583 DEBUG:boto:Data: 2023-05-31T20:50:28.584 DEBUG:boto:Headers: {} 2023-05-31T20:50:28.584 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:50:28.584 DEBUG:boto:Port: 8001 2023-05-31T20:50:28.585 DEBUG:boto:Params: {} 2023-05-31T20:50:28.585 DEBUG:boto:Token: None 2023-05-31T20:50:28.585 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:50:28 GMT /pklxkd-59/obj11 2023-05-31T20:50:28.586 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:3VP7bNniNkFdnNoymJu3D8Avl18= 2023-05-31T20:50:28.586 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:50:28 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:3VP7bNniNkFdnNoymJu3D8Avl18=', 'Content-Length': '0'} 2023-05-31T20:50:28.588 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:39:26 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004f88507bbe97ce5f-006477b314-1c17-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:50:28 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:50:28.588 DEBUG:boto:path=/pklxkd-59/obj11 2023-05-31T20:50:28.589 DEBUG:boto:auth_path=/pklxkd-59/obj11 2023-05-31T20:50:28.589 DEBUG:boto:Method: HEAD 2023-05-31T20:50:28.590 DEBUG:boto:Path: /pklxkd-59/obj11 2023-05-31T20:50:28.590 DEBUG:boto:Data: 2023-05-31T20:50:28.590 DEBUG:boto:Headers: {} 2023-05-31T20:50:28.590 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:50:28.591 DEBUG:boto:Port: 8000 2023-05-31T20:50:28.591 DEBUG:boto:Params: {} 2023-05-31T20:50:28.592 DEBUG:boto:Token: None 2023-05-31T20:50:28.592 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:50:28 GMT /pklxkd-59/obj11 2023-05-31T20:50:28.592 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:3VP7bNniNkFdnNoymJu3D8Avl18= 2023-05-31T20:50:28.593 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:50:28 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:3VP7bNniNkFdnNoymJu3D8Avl18=', 'Content-Length': '0'} 2023-05-31T20:50:28.595 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:39:26 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c47c2e0357742fe5-006477b314-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:50:28 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:50:28.595 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj11 2023-05-31T20:50:28.595 DEBUG:boto:path=/pklxkd-59/obj11 2023-05-31T20:50:28.596 DEBUG:boto:auth_path=/pklxkd-59/obj11 2023-05-31T20:50:28.596 DEBUG:boto:Method: GET 2023-05-31T20:50:28.597 DEBUG:boto:Path: /pklxkd-59/obj11 2023-05-31T20:50:28.597 DEBUG:boto:Data: 2023-05-31T20:50:28.597 DEBUG:boto:Headers: {} 2023-05-31T20:50:28.598 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:50:28.598 DEBUG:boto:Port: 8001 2023-05-31T20:50:28.598 DEBUG:boto:Params: {} 2023-05-31T20:50:28.599 DEBUG:boto:Token: None 2023-05-31T20:50:28.599 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:50:28 GMT /pklxkd-59/obj11 2023-05-31T20:50:28.600 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:H09kscasweKTXwDHRPprpCcc5Ss= 2023-05-31T20:50:28.600 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:50:28 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:H09kscasweKTXwDHRPprpCcc5Ss=', 'Content-Length': '0'} 2023-05-31T20:50:28.602 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:39:26 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000df5445ed2e2834b8-006477b314-1c17-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:50:28 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:50:28.602 DEBUG:boto:path=/pklxkd-59/obj11 2023-05-31T20:50:28.603 DEBUG:boto:auth_path=/pklxkd-59/obj11 2023-05-31T20:50:28.603 DEBUG:boto:Method: GET 2023-05-31T20:50:28.603 DEBUG:boto:Path: /pklxkd-59/obj11 2023-05-31T20:50:28.604 DEBUG:boto:Data: 2023-05-31T20:50:28.604 DEBUG:boto:Headers: {} 2023-05-31T20:50:28.605 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:50:28.605 DEBUG:boto:Port: 8000 2023-05-31T20:50:28.605 DEBUG:boto:Params: {} 2023-05-31T20:50:28.606 DEBUG:boto:Token: None 2023-05-31T20:50:28.606 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:50:28 GMT /pklxkd-59/obj11 2023-05-31T20:50:28.606 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:H09kscasweKTXwDHRPprpCcc5Ss= 2023-05-31T20:50:28.607 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:50:28 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:H09kscasweKTXwDHRPprpCcc5Ss=', 'Content-Length': '0'} 2023-05-31T20:50:28.609 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:39:26 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007f175b9fb0a526db-006477b314-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:50:28 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:50:28.652 DEBUG:boto:path=/pklxkd-59/ 2023-05-31T20:50:28.653 DEBUG:boto:auth_path=/pklxkd-59/ 2023-05-31T20:50:28.653 DEBUG:boto:path=/pklxkd-59/?prefix=obj11 2023-05-31T20:50:28.654 DEBUG:boto:auth_path=/pklxkd-59/?prefix=obj11 2023-05-31T20:50:28.654 DEBUG:boto:Method: GET 2023-05-31T20:50:28.654 DEBUG:boto:Path: /pklxkd-59/?prefix=obj11 2023-05-31T20:50:28.655 DEBUG:boto:Data: 2023-05-31T20:50:28.655 DEBUG:boto:Headers: {} 2023-05-31T20:50:28.655 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:50:28.656 DEBUG:boto:Port: 8001 2023-05-31T20:50:28.656 DEBUG:boto:Params: {} 2023-05-31T20:50:28.656 DEBUG:boto:Token: None 2023-05-31T20:50:28.657 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:50:28 GMT /pklxkd-59/ 2023-05-31T20:50:28.657 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:lLtg7I9o4pC7jMarhXaP1H+yh9o= 2023-05-31T20:50:28.658 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:50:28 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:lLtg7I9o4pC7jMarhXaP1H+yh9o=', 'Content-Length': '0'} 2023-05-31T20:50:28.660 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000348360c2e87e7a87-006477b314-1c17-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:50:28 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:50:28.661 DEBUG:boto:b'pklxkd-59obj111000falseobj112023-05-31T20:39:26.427Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj1152023-05-31T20:46:33.958Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj1162023-05-31T20:48:13.052Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:50:28.662 DEBUG:boto:path=/pklxkd-59/ 2023-05-31T20:50:28.662 DEBUG:boto:auth_path=/pklxkd-59/ 2023-05-31T20:50:28.662 DEBUG:boto:path=/pklxkd-59/?prefix=obj11 2023-05-31T20:50:28.663 DEBUG:boto:auth_path=/pklxkd-59/?prefix=obj11 2023-05-31T20:50:28.663 DEBUG:boto:Method: GET 2023-05-31T20:50:28.663 DEBUG:boto:Path: /pklxkd-59/?prefix=obj11 2023-05-31T20:50:28.664 DEBUG:boto:Data: 2023-05-31T20:50:28.664 DEBUG:boto:Headers: {} 2023-05-31T20:50:28.665 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:50:28.665 DEBUG:boto:Port: 8000 2023-05-31T20:50:28.665 DEBUG:boto:Params: {} 2023-05-31T20:50:28.666 DEBUG:boto:Token: None 2023-05-31T20:50:28.666 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:50:28 GMT /pklxkd-59/ 2023-05-31T20:50:28.666 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:lLtg7I9o4pC7jMarhXaP1H+yh9o= 2023-05-31T20:50:28.667 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:50:28 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:lLtg7I9o4pC7jMarhXaP1H+yh9o=', 'Content-Length': '0'} 2023-05-31T20:50:28.671 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000035695e4c5bb869f-006477b314-1630-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:50:28 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:50:28.671 DEBUG:boto:b'pklxkd-59obj111000falseobj112023-05-31T20:39:26.427Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj1152023-05-31T20:46:33.958Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj1162023-05-31T20:48:13.052Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:50:28.672 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj115 2023-05-31T20:50:28.672 DEBUG:boto:path=/pklxkd-59/obj115 2023-05-31T20:50:28.673 DEBUG:boto:auth_path=/pklxkd-59/obj115 2023-05-31T20:50:28.673 DEBUG:boto:path=/pklxkd-59/obj115?versionId=null 2023-05-31T20:50:28.673 DEBUG:boto:auth_path=/pklxkd-59/obj115?versionId=null 2023-05-31T20:50:28.674 DEBUG:boto:Method: GET 2023-05-31T20:50:28.674 DEBUG:boto:Path: /pklxkd-59/obj115?versionId=null 2023-05-31T20:50:28.674 DEBUG:boto:Data: 2023-05-31T20:50:28.675 DEBUG:boto:Headers: {} 2023-05-31T20:50:28.675 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:50:28.675 DEBUG:boto:Port: 8001 2023-05-31T20:50:28.676 DEBUG:boto:Params: {} 2023-05-31T20:50:28.676 DEBUG:boto:Token: None 2023-05-31T20:50:28.677 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:50:28 GMT /pklxkd-59/obj115?versionId=null 2023-05-31T20:50:28.677 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:8plEhAp2iLhlXV2SURc93GsuEjY= 2023-05-31T20:50:28.677 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:50:28 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:8plEhAp2iLhlXV2SURc93GsuEjY=', 'Content-Length': '0'} 2023-05-31T20:50:28.679 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:46:33 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e93d2eaeba968b51-006477b314-1c17-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:50:28 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:50:28.720 DEBUG:boto:path=/pklxkd-59/obj115 2023-05-31T20:50:28.721 DEBUG:boto:auth_path=/pklxkd-59/obj115 2023-05-31T20:50:28.721 DEBUG:boto:path=/pklxkd-59/obj115?versionId=null 2023-05-31T20:50:28.721 DEBUG:boto:auth_path=/pklxkd-59/obj115?versionId=null 2023-05-31T20:50:28.722 DEBUG:boto:Method: GET 2023-05-31T20:50:28.722 DEBUG:boto:Path: /pklxkd-59/obj115?versionId=null 2023-05-31T20:50:28.723 DEBUG:boto:Data: 2023-05-31T20:50:28.723 DEBUG:boto:Headers: {} 2023-05-31T20:50:28.723 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:50:28.724 DEBUG:boto:Port: 8000 2023-05-31T20:50:28.724 DEBUG:boto:Params: {} 2023-05-31T20:50:28.724 DEBUG:boto:Token: None 2023-05-31T20:50:28.725 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:50:28 GMT /pklxkd-59/obj115?versionId=null 2023-05-31T20:50:28.725 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:8plEhAp2iLhlXV2SURc93GsuEjY= 2023-05-31T20:50:28.726 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:50:28 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:8plEhAp2iLhlXV2SURc93GsuEjY=', 'Content-Length': '0'} 2023-05-31T20:50:28.728 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:46:33 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000abd2798a5f0bbef8-006477b314-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:50:28 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:50:28.728 DEBUG:boto:path=/pklxkd-59/obj115 2023-05-31T20:50:28.729 DEBUG:boto:auth_path=/pklxkd-59/obj115 2023-05-31T20:50:28.729 DEBUG:boto:path=/pklxkd-59/obj115?versionId=null 2023-05-31T20:50:28.729 DEBUG:boto:auth_path=/pklxkd-59/obj115?versionId=null 2023-05-31T20:50:28.730 DEBUG:boto:Method: HEAD 2023-05-31T20:50:28.730 DEBUG:boto:Path: /pklxkd-59/obj115?versionId=null 2023-05-31T20:50:28.730 DEBUG:boto:Data: 2023-05-31T20:50:28.731 DEBUG:boto:Headers: {} 2023-05-31T20:50:28.731 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:50:28.731 DEBUG:boto:Port: 8001 2023-05-31T20:50:28.732 DEBUG:boto:Params: {} 2023-05-31T20:50:28.732 DEBUG:boto:Token: None 2023-05-31T20:50:28.732 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:50:28 GMT /pklxkd-59/obj115?versionId=null 2023-05-31T20:50:28.733 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:FX+pikIHaom1fOdFFNSrncDWXCo= 2023-05-31T20:50:28.733 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:50:28 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:FX+pikIHaom1fOdFFNSrncDWXCo=', 'Content-Length': '0'} 2023-05-31T20:50:28.735 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:46:33 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d6be1a52da6f81c0-006477b314-1c17-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:50:28 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:50:28.736 DEBUG:boto:path=/pklxkd-59/obj115 2023-05-31T20:50:28.736 DEBUG:boto:auth_path=/pklxkd-59/obj115 2023-05-31T20:50:28.737 DEBUG:boto:path=/pklxkd-59/obj115?versionId=null 2023-05-31T20:50:28.737 DEBUG:boto:auth_path=/pklxkd-59/obj115?versionId=null 2023-05-31T20:50:28.737 DEBUG:boto:Method: HEAD 2023-05-31T20:50:28.738 DEBUG:boto:Path: /pklxkd-59/obj115?versionId=null 2023-05-31T20:50:28.738 DEBUG:boto:Data: 2023-05-31T20:50:28.738 DEBUG:boto:Headers: {} 2023-05-31T20:50:28.738 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:50:28.739 DEBUG:boto:Port: 8000 2023-05-31T20:50:28.739 DEBUG:boto:Params: {} 2023-05-31T20:50:28.740 DEBUG:boto:Token: None 2023-05-31T20:50:28.740 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:50:28 GMT /pklxkd-59/obj115?versionId=null 2023-05-31T20:50:28.740 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:FX+pikIHaom1fOdFFNSrncDWXCo= 2023-05-31T20:50:28.741 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:50:28 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:FX+pikIHaom1fOdFFNSrncDWXCo=', 'Content-Length': '0'} 2023-05-31T20:50:28.742 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:46:33 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ddddcd86224cb072-006477b314-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:50:28 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:50:28.743 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj115 2023-05-31T20:50:28.743 DEBUG:boto:path=/pklxkd-59/obj115 2023-05-31T20:50:28.744 DEBUG:boto:auth_path=/pklxkd-59/obj115 2023-05-31T20:50:28.744 DEBUG:boto:path=/pklxkd-59/obj115?versionId=null 2023-05-31T20:50:28.744 DEBUG:boto:auth_path=/pklxkd-59/obj115?versionId=null 2023-05-31T20:50:28.745 DEBUG:boto:Method: GET 2023-05-31T20:50:28.745 DEBUG:boto:Path: /pklxkd-59/obj115?versionId=null 2023-05-31T20:50:28.746 DEBUG:boto:Data: 2023-05-31T20:50:28.746 DEBUG:boto:Headers: {} 2023-05-31T20:50:28.746 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:50:28.747 DEBUG:boto:Port: 8001 2023-05-31T20:50:28.747 DEBUG:boto:Params: {} 2023-05-31T20:50:28.747 DEBUG:boto:Token: None 2023-05-31T20:50:28.748 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:50:28 GMT /pklxkd-59/obj115?versionId=null 2023-05-31T20:50:28.748 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:8plEhAp2iLhlXV2SURc93GsuEjY= 2023-05-31T20:50:28.749 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:50:28 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:8plEhAp2iLhlXV2SURc93GsuEjY=', 'Content-Length': '0'} 2023-05-31T20:50:28.751 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:46:33 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000092c712eccc3d219b-006477b314-1c17-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:50:28 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:50:28.751 DEBUG:boto:path=/pklxkd-59/obj115 2023-05-31T20:50:28.752 DEBUG:boto:auth_path=/pklxkd-59/obj115 2023-05-31T20:50:28.752 DEBUG:boto:path=/pklxkd-59/obj115?versionId=null 2023-05-31T20:50:28.752 DEBUG:boto:auth_path=/pklxkd-59/obj115?versionId=null 2023-05-31T20:50:28.753 DEBUG:boto:Method: GET 2023-05-31T20:50:28.753 DEBUG:boto:Path: /pklxkd-59/obj115?versionId=null 2023-05-31T20:50:28.753 DEBUG:boto:Data: 2023-05-31T20:50:28.754 DEBUG:boto:Headers: {} 2023-05-31T20:50:28.754 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:50:28.755 DEBUG:boto:Port: 8000 2023-05-31T20:50:28.755 DEBUG:boto:Params: {} 2023-05-31T20:50:28.755 DEBUG:boto:Token: None 2023-05-31T20:50:28.756 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:50:28 GMT /pklxkd-59/obj115?versionId=null 2023-05-31T20:50:28.756 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:8plEhAp2iLhlXV2SURc93GsuEjY= 2023-05-31T20:50:28.757 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:50:28 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:8plEhAp2iLhlXV2SURc93GsuEjY=', 'Content-Length': '0'} 2023-05-31T20:50:28.758 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:46:33 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000009c8de211dbb6439-006477b314-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:50:28 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:50:28.759 DEBUG:boto:path=/pklxkd-59/ 2023-05-31T20:50:28.759 DEBUG:boto:auth_path=/pklxkd-59/ 2023-05-31T20:50:28.760 DEBUG:boto:path=/pklxkd-59/?prefix=obj115 2023-05-31T20:50:28.760 DEBUG:boto:auth_path=/pklxkd-59/?prefix=obj115 2023-05-31T20:50:28.761 DEBUG:boto:Method: GET 2023-05-31T20:50:28.761 DEBUG:boto:Path: /pklxkd-59/?prefix=obj115 2023-05-31T20:50:28.761 DEBUG:boto:Data: 2023-05-31T20:50:28.762 DEBUG:boto:Headers: {} 2023-05-31T20:50:28.762 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:50:28.762 DEBUG:boto:Port: 8001 2023-05-31T20:50:28.763 DEBUG:boto:Params: {} 2023-05-31T20:50:28.763 DEBUG:boto:Token: None 2023-05-31T20:50:28.763 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:50:28 GMT /pklxkd-59/ 2023-05-31T20:50:28.764 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:lLtg7I9o4pC7jMarhXaP1H+yh9o= 2023-05-31T20:50:28.764 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:50:28 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:lLtg7I9o4pC7jMarhXaP1H+yh9o=', 'Content-Length': '0'} 2023-05-31T20:50:28.766 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000005634d20772bfebb0-006477b314-1c17-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:50:28 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:50:28.767 DEBUG:boto:b'pklxkd-59obj1151000falseobj1152023-05-31T20:46:33.958Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:50:28.768 DEBUG:boto:path=/pklxkd-59/ 2023-05-31T20:50:28.768 DEBUG:boto:auth_path=/pklxkd-59/ 2023-05-31T20:50:28.768 DEBUG:boto:path=/pklxkd-59/?prefix=obj115 2023-05-31T20:50:28.769 DEBUG:boto:auth_path=/pklxkd-59/?prefix=obj115 2023-05-31T20:50:28.769 DEBUG:boto:Method: GET 2023-05-31T20:50:28.769 DEBUG:boto:Path: /pklxkd-59/?prefix=obj115 2023-05-31T20:50:28.770 DEBUG:boto:Data: 2023-05-31T20:50:28.770 DEBUG:boto:Headers: {} 2023-05-31T20:50:28.770 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:50:28.771 DEBUG:boto:Port: 8000 2023-05-31T20:50:28.771 DEBUG:boto:Params: {} 2023-05-31T20:50:28.772 DEBUG:boto:Token: None 2023-05-31T20:50:28.772 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:50:28 GMT /pklxkd-59/ 2023-05-31T20:50:28.772 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:lLtg7I9o4pC7jMarhXaP1H+yh9o= 2023-05-31T20:50:28.773 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:50:28 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:lLtg7I9o4pC7jMarhXaP1H+yh9o=', 'Content-Length': '0'} 2023-05-31T20:50:28.776 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000069803ae91d37f6cf-006477b314-1630-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:50:28 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:50:28.777 DEBUG:boto:b'pklxkd-59obj1151000falseobj1152023-05-31T20:46:33.958Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:50:28.778 DEBUG:boto:path=/pklxkd-59/obj115 2023-05-31T20:50:28.778 DEBUG:boto:auth_path=/pklxkd-59/obj115 2023-05-31T20:50:28.778 DEBUG:boto:Method: HEAD 2023-05-31T20:50:28.779 DEBUG:boto:Path: /pklxkd-59/obj115 2023-05-31T20:50:28.779 DEBUG:boto:Data: 2023-05-31T20:50:28.779 DEBUG:boto:Headers: {} 2023-05-31T20:50:28.780 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:50:28.780 DEBUG:boto:Port: 8001 2023-05-31T20:50:28.780 DEBUG:boto:Params: {} 2023-05-31T20:50:28.781 DEBUG:boto:Token: None 2023-05-31T20:50:28.781 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:50:28 GMT /pklxkd-59/obj115 2023-05-31T20:50:28.781 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:paUb70DzKA4pSdk5xKVr5r+Atp0= 2023-05-31T20:50:28.782 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:50:28 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:paUb70DzKA4pSdk5xKVr5r+Atp0=', 'Content-Length': '0'} 2023-05-31T20:50:28.784 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:46:33 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006c28e919b31d250e-006477b314-1c17-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:50:28 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:50:28.784 DEBUG:boto:path=/pklxkd-59/obj115 2023-05-31T20:50:28.785 DEBUG:boto:auth_path=/pklxkd-59/obj115 2023-05-31T20:50:28.785 DEBUG:boto:Method: HEAD 2023-05-31T20:50:28.785 DEBUG:boto:Path: /pklxkd-59/obj115 2023-05-31T20:50:28.786 DEBUG:boto:Data: 2023-05-31T20:50:28.786 DEBUG:boto:Headers: {} 2023-05-31T20:50:28.786 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:50:28.787 DEBUG:boto:Port: 8000 2023-05-31T20:50:28.787 DEBUG:boto:Params: {} 2023-05-31T20:50:28.787 DEBUG:boto:Token: None 2023-05-31T20:50:28.788 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:50:28 GMT /pklxkd-59/obj115 2023-05-31T20:50:28.788 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:paUb70DzKA4pSdk5xKVr5r+Atp0= 2023-05-31T20:50:28.789 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:50:28 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:paUb70DzKA4pSdk5xKVr5r+Atp0=', 'Content-Length': '0'} 2023-05-31T20:50:28.790 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:46:33 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003a491a404191573a-006477b314-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:50:28 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:50:28.791 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj115 2023-05-31T20:50:28.791 DEBUG:boto:path=/pklxkd-59/obj115 2023-05-31T20:50:28.792 DEBUG:boto:auth_path=/pklxkd-59/obj115 2023-05-31T20:50:28.792 DEBUG:boto:Method: GET 2023-05-31T20:50:28.792 DEBUG:boto:Path: /pklxkd-59/obj115 2023-05-31T20:50:28.793 DEBUG:boto:Data: 2023-05-31T20:50:28.793 DEBUG:boto:Headers: {} 2023-05-31T20:50:28.793 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:50:28.794 DEBUG:boto:Port: 8001 2023-05-31T20:50:28.794 DEBUG:boto:Params: {} 2023-05-31T20:50:28.794 DEBUG:boto:Token: None 2023-05-31T20:50:28.795 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:50:28 GMT /pklxkd-59/obj115 2023-05-31T20:50:28.795 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:SoGxKlkJ6HNb/WlYQSvO6V/MrSo= 2023-05-31T20:50:28.796 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:50:28 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:SoGxKlkJ6HNb/WlYQSvO6V/MrSo=', 'Content-Length': '0'} 2023-05-31T20:50:28.798 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:46:33 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000cfe0eb752f845489-006477b314-1c17-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:50:28 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:50:28.798 DEBUG:boto:path=/pklxkd-59/obj115 2023-05-31T20:50:28.799 DEBUG:boto:auth_path=/pklxkd-59/obj115 2023-05-31T20:50:28.799 DEBUG:boto:Method: GET 2023-05-31T20:50:28.799 DEBUG:boto:Path: /pklxkd-59/obj115 2023-05-31T20:50:28.800 DEBUG:boto:Data: 2023-05-31T20:50:28.800 DEBUG:boto:Headers: {} 2023-05-31T20:50:28.801 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:50:28.801 DEBUG:boto:Port: 8000 2023-05-31T20:50:28.801 DEBUG:boto:Params: {} 2023-05-31T20:50:28.802 DEBUG:boto:Token: None 2023-05-31T20:50:28.802 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:50:28 GMT /pklxkd-59/obj115 2023-05-31T20:50:28.803 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:SoGxKlkJ6HNb/WlYQSvO6V/MrSo= 2023-05-31T20:50:28.803 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:50:28 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:SoGxKlkJ6HNb/WlYQSvO6V/MrSo=', 'Content-Length': '0'} 2023-05-31T20:50:28.805 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:46:33 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009c7c16d53b904645-006477b314-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:50:28 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:50:28.806 DEBUG:boto:path=/pklxkd-59/ 2023-05-31T20:50:28.806 DEBUG:boto:auth_path=/pklxkd-59/ 2023-05-31T20:50:28.806 DEBUG:boto:path=/pklxkd-59/?prefix=obj115 2023-05-31T20:50:28.807 DEBUG:boto:auth_path=/pklxkd-59/?prefix=obj115 2023-05-31T20:50:28.807 DEBUG:boto:Method: GET 2023-05-31T20:50:28.807 DEBUG:boto:Path: /pklxkd-59/?prefix=obj115 2023-05-31T20:50:28.808 DEBUG:boto:Data: 2023-05-31T20:50:28.808 DEBUG:boto:Headers: {} 2023-05-31T20:50:28.808 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:50:28.809 DEBUG:boto:Port: 8001 2023-05-31T20:50:28.809 DEBUG:boto:Params: {} 2023-05-31T20:50:28.810 DEBUG:boto:Token: None 2023-05-31T20:50:28.810 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:50:28 GMT /pklxkd-59/ 2023-05-31T20:50:28.810 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:lLtg7I9o4pC7jMarhXaP1H+yh9o= 2023-05-31T20:50:28.811 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:50:28 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:lLtg7I9o4pC7jMarhXaP1H+yh9o=', 'Content-Length': '0'} 2023-05-31T20:50:28.813 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000acafa3f9cbb0d0f0-006477b314-1c17-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:50:28 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:50:28.813 DEBUG:boto:b'pklxkd-59obj1151000falseobj1152023-05-31T20:46:33.958Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:50:28.814 DEBUG:boto:path=/pklxkd-59/ 2023-05-31T20:50:28.815 DEBUG:boto:auth_path=/pklxkd-59/ 2023-05-31T20:50:28.815 DEBUG:boto:path=/pklxkd-59/?prefix=obj115 2023-05-31T20:50:28.815 DEBUG:boto:auth_path=/pklxkd-59/?prefix=obj115 2023-05-31T20:50:28.816 DEBUG:boto:Method: GET 2023-05-31T20:50:28.816 DEBUG:boto:Path: /pklxkd-59/?prefix=obj115 2023-05-31T20:50:28.816 DEBUG:boto:Data: 2023-05-31T20:50:28.817 DEBUG:boto:Headers: {} 2023-05-31T20:50:28.817 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:50:28.817 DEBUG:boto:Port: 8000 2023-05-31T20:50:28.818 DEBUG:boto:Params: {} 2023-05-31T20:50:28.818 DEBUG:boto:Token: None 2023-05-31T20:50:28.819 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:50:28 GMT /pklxkd-59/ 2023-05-31T20:50:28.819 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:lLtg7I9o4pC7jMarhXaP1H+yh9o= 2023-05-31T20:50:28.819 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:50:28 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:lLtg7I9o4pC7jMarhXaP1H+yh9o=', 'Content-Length': '0'} 2023-05-31T20:50:28.823 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000011e0d443b79c39ff-006477b314-1630-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:50:28 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:50:28.864 DEBUG:boto:b'pklxkd-59obj1151000falseobj1152023-05-31T20:46:33.958Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:50:28.865 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj116 2023-05-31T20:50:28.865 DEBUG:boto:path=/pklxkd-59/obj116 2023-05-31T20:50:28.866 DEBUG:boto:auth_path=/pklxkd-59/obj116 2023-05-31T20:50:28.866 DEBUG:boto:path=/pklxkd-59/obj116?versionId=null 2023-05-31T20:50:28.867 DEBUG:boto:auth_path=/pklxkd-59/obj116?versionId=null 2023-05-31T20:50:28.867 DEBUG:boto:Method: GET 2023-05-31T20:50:28.867 DEBUG:boto:Path: /pklxkd-59/obj116?versionId=null 2023-05-31T20:50:28.868 DEBUG:boto:Data: 2023-05-31T20:50:28.868 DEBUG:boto:Headers: {} 2023-05-31T20:50:28.868 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:50:28.869 DEBUG:boto:Port: 8001 2023-05-31T20:50:28.869 DEBUG:boto:Params: {} 2023-05-31T20:50:28.869 DEBUG:boto:Token: None 2023-05-31T20:50:28.870 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:50:28 GMT /pklxkd-59/obj116?versionId=null 2023-05-31T20:50:28.870 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:SRWMaIe0je2ilU4tDBGhLSamYkg= 2023-05-31T20:50:28.871 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:50:28 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:SRWMaIe0je2ilU4tDBGhLSamYkg=', 'Content-Length': '0'} 2023-05-31T20:50:28.873 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:48:13 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001e7ac5bc22acfee7-006477b314-1c17-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:50:28 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:50:28.873 DEBUG:boto:path=/pklxkd-59/obj116 2023-05-31T20:50:28.874 DEBUG:boto:auth_path=/pklxkd-59/obj116 2023-05-31T20:50:28.874 DEBUG:boto:path=/pklxkd-59/obj116?versionId=null 2023-05-31T20:50:28.875 DEBUG:boto:auth_path=/pklxkd-59/obj116?versionId=null 2023-05-31T20:50:28.875 DEBUG:boto:Method: GET 2023-05-31T20:50:28.875 DEBUG:boto:Path: /pklxkd-59/obj116?versionId=null 2023-05-31T20:50:28.876 DEBUG:boto:Data: 2023-05-31T20:50:28.876 DEBUG:boto:Headers: {} 2023-05-31T20:50:28.876 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:50:28.877 DEBUG:boto:Port: 8000 2023-05-31T20:50:28.877 DEBUG:boto:Params: {} 2023-05-31T20:50:28.878 DEBUG:boto:Token: None 2023-05-31T20:50:28.878 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:50:28 GMT /pklxkd-59/obj116?versionId=null 2023-05-31T20:50:28.878 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:SRWMaIe0je2ilU4tDBGhLSamYkg= 2023-05-31T20:50:28.879 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:50:28 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:SRWMaIe0je2ilU4tDBGhLSamYkg=', 'Content-Length': '0'} 2023-05-31T20:50:28.881 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:48:13 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f7f87b64d4fc2aaf-006477b314-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:50:28 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:50:28.881 DEBUG:boto:path=/pklxkd-59/obj116 2023-05-31T20:50:28.882 DEBUG:boto:auth_path=/pklxkd-59/obj116 2023-05-31T20:50:28.882 DEBUG:boto:path=/pklxkd-59/obj116?versionId=null 2023-05-31T20:50:28.882 DEBUG:boto:auth_path=/pklxkd-59/obj116?versionId=null 2023-05-31T20:50:28.883 DEBUG:boto:Method: HEAD 2023-05-31T20:50:28.883 DEBUG:boto:Path: /pklxkd-59/obj116?versionId=null 2023-05-31T20:50:28.883 DEBUG:boto:Data: 2023-05-31T20:50:28.884 DEBUG:boto:Headers: {} 2023-05-31T20:50:28.884 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:50:28.884 DEBUG:boto:Port: 8001 2023-05-31T20:50:28.885 DEBUG:boto:Params: {} 2023-05-31T20:50:28.885 DEBUG:boto:Token: None 2023-05-31T20:50:28.885 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:50:28 GMT /pklxkd-59/obj116?versionId=null 2023-05-31T20:50:28.886 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:7mPeAH5A0xLQgOp4cIlCvWiP4SY= 2023-05-31T20:50:28.886 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:50:28 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:7mPeAH5A0xLQgOp4cIlCvWiP4SY=', 'Content-Length': '0'} 2023-05-31T20:50:28.888 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:48:13 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000090c484a268afde82-006477b314-1c17-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:50:28 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:50:28.889 DEBUG:boto:path=/pklxkd-59/obj116 2023-05-31T20:50:28.889 DEBUG:boto:auth_path=/pklxkd-59/obj116 2023-05-31T20:50:28.889 DEBUG:boto:path=/pklxkd-59/obj116?versionId=null 2023-05-31T20:50:28.890 DEBUG:boto:auth_path=/pklxkd-59/obj116?versionId=null 2023-05-31T20:50:28.890 DEBUG:boto:Method: HEAD 2023-05-31T20:50:28.890 DEBUG:boto:Path: /pklxkd-59/obj116?versionId=null 2023-05-31T20:50:28.891 DEBUG:boto:Data: 2023-05-31T20:50:28.891 DEBUG:boto:Headers: {} 2023-05-31T20:50:28.891 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:50:28.892 DEBUG:boto:Port: 8000 2023-05-31T20:50:28.892 DEBUG:boto:Params: {} 2023-05-31T20:50:28.892 DEBUG:boto:Token: None 2023-05-31T20:50:28.893 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:50:28 GMT /pklxkd-59/obj116?versionId=null 2023-05-31T20:50:28.893 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:7mPeAH5A0xLQgOp4cIlCvWiP4SY= 2023-05-31T20:50:28.894 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:50:28 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:7mPeAH5A0xLQgOp4cIlCvWiP4SY=', 'Content-Length': '0'} 2023-05-31T20:50:28.895 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:48:13 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006e33ca135b621d6b-006477b314-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:50:28 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:50:28.896 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj116 2023-05-31T20:50:28.896 DEBUG:boto:path=/pklxkd-59/obj116 2023-05-31T20:50:28.896 DEBUG:boto:auth_path=/pklxkd-59/obj116 2023-05-31T20:50:28.897 DEBUG:boto:path=/pklxkd-59/obj116?versionId=null 2023-05-31T20:50:28.897 DEBUG:boto:auth_path=/pklxkd-59/obj116?versionId=null 2023-05-31T20:50:28.898 DEBUG:boto:Method: GET 2023-05-31T20:50:28.898 DEBUG:boto:Path: /pklxkd-59/obj116?versionId=null 2023-05-31T20:50:28.898 DEBUG:boto:Data: 2023-05-31T20:50:28.899 DEBUG:boto:Headers: {} 2023-05-31T20:50:28.899 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:50:28.899 DEBUG:boto:Port: 8001 2023-05-31T20:50:28.900 DEBUG:boto:Params: {} 2023-05-31T20:50:28.900 DEBUG:boto:Token: None 2023-05-31T20:50:28.900 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:50:28 GMT /pklxkd-59/obj116?versionId=null 2023-05-31T20:50:28.901 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:SRWMaIe0je2ilU4tDBGhLSamYkg= 2023-05-31T20:50:28.901 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:50:28 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:SRWMaIe0je2ilU4tDBGhLSamYkg=', 'Content-Length': '0'} 2023-05-31T20:50:28.903 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:48:13 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000094b2c40f7602ff0c-006477b314-1c17-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:50:28 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:50:28.904 DEBUG:boto:path=/pklxkd-59/obj116 2023-05-31T20:50:28.904 DEBUG:boto:auth_path=/pklxkd-59/obj116 2023-05-31T20:50:28.904 DEBUG:boto:path=/pklxkd-59/obj116?versionId=null 2023-05-31T20:50:28.905 DEBUG:boto:auth_path=/pklxkd-59/obj116?versionId=null 2023-05-31T20:50:28.905 DEBUG:boto:Method: GET 2023-05-31T20:50:28.905 DEBUG:boto:Path: /pklxkd-59/obj116?versionId=null 2023-05-31T20:50:28.906 DEBUG:boto:Data: 2023-05-31T20:50:28.906 DEBUG:boto:Headers: {} 2023-05-31T20:50:28.906 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:50:28.907 DEBUG:boto:Port: 8000 2023-05-31T20:50:28.907 DEBUG:boto:Params: {} 2023-05-31T20:50:28.908 DEBUG:boto:Token: None 2023-05-31T20:50:28.908 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:50:28 GMT /pklxkd-59/obj116?versionId=null 2023-05-31T20:50:28.908 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:SRWMaIe0je2ilU4tDBGhLSamYkg= 2023-05-31T20:50:28.909 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:50:28 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:SRWMaIe0je2ilU4tDBGhLSamYkg=', 'Content-Length': '0'} 2023-05-31T20:50:28.911 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:48:13 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d95de60fe1e22013-006477b314-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:50:28 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:50:28.911 DEBUG:boto:path=/pklxkd-59/ 2023-05-31T20:50:28.912 DEBUG:boto:auth_path=/pklxkd-59/ 2023-05-31T20:50:28.912 DEBUG:boto:path=/pklxkd-59/?prefix=obj116 2023-05-31T20:50:28.912 DEBUG:boto:auth_path=/pklxkd-59/?prefix=obj116 2023-05-31T20:50:28.913 DEBUG:boto:Method: GET 2023-05-31T20:50:28.913 DEBUG:boto:Path: /pklxkd-59/?prefix=obj116 2023-05-31T20:50:28.914 DEBUG:boto:Data: 2023-05-31T20:50:28.914 DEBUG:boto:Headers: {} 2023-05-31T20:50:28.914 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:50:28.915 DEBUG:boto:Port: 8001 2023-05-31T20:50:28.915 DEBUG:boto:Params: {} 2023-05-31T20:50:28.915 DEBUG:boto:Token: None 2023-05-31T20:50:28.916 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:50:28 GMT /pklxkd-59/ 2023-05-31T20:50:28.916 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:lLtg7I9o4pC7jMarhXaP1H+yh9o= 2023-05-31T20:50:28.917 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:50:28 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:lLtg7I9o4pC7jMarhXaP1H+yh9o=', 'Content-Length': '0'} 2023-05-31T20:50:28.919 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000001379e40100e8c0ea-006477b314-1c17-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:50:28 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:50:28.920 DEBUG:boto:b'pklxkd-59obj1161000falseobj1162023-05-31T20:48:13.052Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:50:28.920 DEBUG:boto:path=/pklxkd-59/ 2023-05-31T20:50:28.921 DEBUG:boto:auth_path=/pklxkd-59/ 2023-05-31T20:50:28.921 DEBUG:boto:path=/pklxkd-59/?prefix=obj116 2023-05-31T20:50:28.921 DEBUG:boto:auth_path=/pklxkd-59/?prefix=obj116 2023-05-31T20:50:28.922 DEBUG:boto:Method: GET 2023-05-31T20:50:28.922 DEBUG:boto:Path: /pklxkd-59/?prefix=obj116 2023-05-31T20:50:28.922 DEBUG:boto:Data: 2023-05-31T20:50:28.923 DEBUG:boto:Headers: {} 2023-05-31T20:50:28.923 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:50:28.923 DEBUG:boto:Port: 8000 2023-05-31T20:50:28.924 DEBUG:boto:Params: {} 2023-05-31T20:50:28.924 DEBUG:boto:Token: None 2023-05-31T20:50:28.925 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:50:28 GMT /pklxkd-59/ 2023-05-31T20:50:28.925 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:lLtg7I9o4pC7jMarhXaP1H+yh9o= 2023-05-31T20:50:28.925 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:50:28 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:lLtg7I9o4pC7jMarhXaP1H+yh9o=', 'Content-Length': '0'} 2023-05-31T20:50:28.929 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000639372f43c73c4e9-006477b314-1630-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:50:28 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:50:28.929 DEBUG:boto:b'pklxkd-59obj1161000falseobj1162023-05-31T20:48:13.052Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:50:28.930 DEBUG:boto:path=/pklxkd-59/obj116 2023-05-31T20:50:28.930 DEBUG:boto:auth_path=/pklxkd-59/obj116 2023-05-31T20:50:28.931 DEBUG:boto:Method: HEAD 2023-05-31T20:50:28.931 DEBUG:boto:Path: /pklxkd-59/obj116 2023-05-31T20:50:28.931 DEBUG:boto:Data: 2023-05-31T20:50:28.932 DEBUG:boto:Headers: {} 2023-05-31T20:50:28.932 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:50:28.932 DEBUG:boto:Port: 8001 2023-05-31T20:50:28.933 DEBUG:boto:Params: {} 2023-05-31T20:50:28.933 DEBUG:boto:Token: None 2023-05-31T20:50:28.934 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:50:28 GMT /pklxkd-59/obj116 2023-05-31T20:50:28.934 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:y7kvxW109OcAt1mxLY7arD1owl4= 2023-05-31T20:50:28.934 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:50:28 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:y7kvxW109OcAt1mxLY7arD1owl4=', 'Content-Length': '0'} 2023-05-31T20:50:28.936 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:48:13 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ceaa4b21e1af824d-006477b314-1c17-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:50:28 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:50:28.937 DEBUG:boto:path=/pklxkd-59/obj116 2023-05-31T20:50:28.937 DEBUG:boto:auth_path=/pklxkd-59/obj116 2023-05-31T20:50:28.937 DEBUG:boto:Method: HEAD 2023-05-31T20:50:28.938 DEBUG:boto:Path: /pklxkd-59/obj116 2023-05-31T20:50:28.938 DEBUG:boto:Data: 2023-05-31T20:50:28.938 DEBUG:boto:Headers: {} 2023-05-31T20:50:28.939 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:50:28.939 DEBUG:boto:Port: 8000 2023-05-31T20:50:28.939 DEBUG:boto:Params: {} 2023-05-31T20:50:28.940 DEBUG:boto:Token: None 2023-05-31T20:50:28.940 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:50:28 GMT /pklxkd-59/obj116 2023-05-31T20:50:28.941 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:y7kvxW109OcAt1mxLY7arD1owl4= 2023-05-31T20:50:28.941 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:50:28 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:y7kvxW109OcAt1mxLY7arD1owl4=', 'Content-Length': '0'} 2023-05-31T20:50:28.943 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:48:13 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003acbf8e2df52da39-006477b314-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:50:28 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:50:28.943 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj116 2023-05-31T20:50:28.944 DEBUG:boto:path=/pklxkd-59/obj116 2023-05-31T20:50:28.944 DEBUG:boto:auth_path=/pklxkd-59/obj116 2023-05-31T20:50:28.944 DEBUG:boto:Method: GET 2023-05-31T20:50:28.945 DEBUG:boto:Path: /pklxkd-59/obj116 2023-05-31T20:50:28.945 DEBUG:boto:Data: 2023-05-31T20:50:28.945 DEBUG:boto:Headers: {} 2023-05-31T20:50:28.946 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:50:28.946 DEBUG:boto:Port: 8001 2023-05-31T20:50:28.947 DEBUG:boto:Params: {} 2023-05-31T20:50:28.947 DEBUG:boto:Token: None 2023-05-31T20:50:28.947 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:50:28 GMT /pklxkd-59/obj116 2023-05-31T20:50:28.948 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:oM6EwQN04zEkgVMWh0ckwJbbcL4= 2023-05-31T20:50:28.948 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:50:28 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:oM6EwQN04zEkgVMWh0ckwJbbcL4=', 'Content-Length': '0'} 2023-05-31T20:50:28.950 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:48:13 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000266d74c79e6682e1-006477b314-1c17-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:50:28 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:50:28.951 DEBUG:boto:path=/pklxkd-59/obj116 2023-05-31T20:50:28.951 DEBUG:boto:auth_path=/pklxkd-59/obj116 2023-05-31T20:50:28.951 DEBUG:boto:Method: GET 2023-05-31T20:50:28.952 DEBUG:boto:Path: /pklxkd-59/obj116 2023-05-31T20:50:28.952 DEBUG:boto:Data: 2023-05-31T20:50:28.952 DEBUG:boto:Headers: {} 2023-05-31T20:50:28.953 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:50:28.953 DEBUG:boto:Port: 8000 2023-05-31T20:50:28.953 DEBUG:boto:Params: {} 2023-05-31T20:50:28.954 DEBUG:boto:Token: None 2023-05-31T20:50:28.954 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:50:28 GMT /pklxkd-59/obj116 2023-05-31T20:50:28.955 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:oM6EwQN04zEkgVMWh0ckwJbbcL4= 2023-05-31T20:50:28.955 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:50:28 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:oM6EwQN04zEkgVMWh0ckwJbbcL4=', 'Content-Length': '0'} 2023-05-31T20:50:28.957 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:48:13 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c0084a5b407564c4-006477b314-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:50:28 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:50:29.000 DEBUG:boto:path=/pklxkd-59/ 2023-05-31T20:50:29.001 DEBUG:boto:auth_path=/pklxkd-59/ 2023-05-31T20:50:29.001 DEBUG:boto:path=/pklxkd-59/?prefix=obj116 2023-05-31T20:50:29.002 DEBUG:boto:auth_path=/pklxkd-59/?prefix=obj116 2023-05-31T20:50:29.002 DEBUG:boto:Method: GET 2023-05-31T20:50:29.002 DEBUG:boto:Path: /pklxkd-59/?prefix=obj116 2023-05-31T20:50:29.003 DEBUG:boto:Data: 2023-05-31T20:50:29.003 DEBUG:boto:Headers: {} 2023-05-31T20:50:29.003 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:50:29.004 DEBUG:boto:Port: 8001 2023-05-31T20:50:29.004 DEBUG:boto:Params: {} 2023-05-31T20:50:29.004 DEBUG:boto:Token: None 2023-05-31T20:50:29.005 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:50:29 GMT /pklxkd-59/ 2023-05-31T20:50:29.005 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:Oq/vWXu48It/gsNY1dtGdMC0LjA= 2023-05-31T20:50:29.006 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:50:29 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:Oq/vWXu48It/gsNY1dtGdMC0LjA=', 'Content-Length': '0'} 2023-05-31T20:50:29.008 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000582652881750d453-006477b315-1c17-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:50:29 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:50:29.009 DEBUG:boto:b'pklxkd-59obj1161000falseobj1162023-05-31T20:48:13.052Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:50:29.009 DEBUG:boto:path=/pklxkd-59/ 2023-05-31T20:50:29.010 DEBUG:boto:auth_path=/pklxkd-59/ 2023-05-31T20:50:29.010 DEBUG:boto:path=/pklxkd-59/?prefix=obj116 2023-05-31T20:50:29.011 DEBUG:boto:auth_path=/pklxkd-59/?prefix=obj116 2023-05-31T20:50:29.011 DEBUG:boto:Method: GET 2023-05-31T20:50:29.011 DEBUG:boto:Path: /pklxkd-59/?prefix=obj116 2023-05-31T20:50:29.012 DEBUG:boto:Data: 2023-05-31T20:50:29.012 DEBUG:boto:Headers: {} 2023-05-31T20:50:29.012 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:50:29.013 DEBUG:boto:Port: 8000 2023-05-31T20:50:29.013 DEBUG:boto:Params: {} 2023-05-31T20:50:29.013 DEBUG:boto:Token: None 2023-05-31T20:50:29.014 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:50:29 GMT /pklxkd-59/ 2023-05-31T20:50:29.014 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:Oq/vWXu48It/gsNY1dtGdMC0LjA= 2023-05-31T20:50:29.015 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:50:29 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:Oq/vWXu48It/gsNY1dtGdMC0LjA=', 'Content-Length': '0'} 2023-05-31T20:50:29.018 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d2e9224443f05e21-006477b315-1630-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:50:29 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:50:29.019 DEBUG:boto:b'pklxkd-59obj1161000falseobj1162023-05-31T20:48:13.052Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:50:29.020 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj145 2023-05-31T20:50:29.020 DEBUG:boto:path=/pklxkd-59/obj145 2023-05-31T20:50:29.020 DEBUG:boto:auth_path=/pklxkd-59/obj145 2023-05-31T20:50:29.021 DEBUG:boto:path=/pklxkd-59/obj145?versionId=null 2023-05-31T20:50:29.021 DEBUG:boto:auth_path=/pklxkd-59/obj145?versionId=null 2023-05-31T20:50:29.021 DEBUG:boto:Method: GET 2023-05-31T20:50:29.022 DEBUG:boto:Path: /pklxkd-59/obj145?versionId=null 2023-05-31T20:50:29.022 DEBUG:boto:Data: 2023-05-31T20:50:29.023 DEBUG:boto:Headers: {} 2023-05-31T20:50:29.023 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:50:29.023 DEBUG:boto:Port: 8001 2023-05-31T20:50:29.024 DEBUG:boto:Params: {} 2023-05-31T20:50:29.024 DEBUG:boto:Token: None 2023-05-31T20:50:29.024 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:50:29 GMT /pklxkd-59/obj145?versionId=null 2023-05-31T20:50:29.025 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:BBKOgSg9NQrZOjmbnYy+P89Ubdw= 2023-05-31T20:50:29.025 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:50:29 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:BBKOgSg9NQrZOjmbnYy+P89Ubdw=', 'Content-Length': '0'} 2023-05-31T20:50:29.028 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:48:13 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003a86ab6c54a6d231-006477b315-1c17-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:50:29 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:50:29.068 DEBUG:boto:path=/pklxkd-59/obj145 2023-05-31T20:50:29.069 DEBUG:boto:auth_path=/pklxkd-59/obj145 2023-05-31T20:50:29.069 DEBUG:boto:path=/pklxkd-59/obj145?versionId=null 2023-05-31T20:50:29.070 DEBUG:boto:auth_path=/pklxkd-59/obj145?versionId=null 2023-05-31T20:50:29.070 DEBUG:boto:Method: GET 2023-05-31T20:50:29.071 DEBUG:boto:Path: /pklxkd-59/obj145?versionId=null 2023-05-31T20:50:29.071 DEBUG:boto:Data: 2023-05-31T20:50:29.072 DEBUG:boto:Headers: {} 2023-05-31T20:50:29.072 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:50:29.072 DEBUG:boto:Port: 8000 2023-05-31T20:50:29.073 DEBUG:boto:Params: {} 2023-05-31T20:50:29.073 DEBUG:boto:Token: None 2023-05-31T20:50:29.074 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:50:29 GMT /pklxkd-59/obj145?versionId=null 2023-05-31T20:50:29.074 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:BBKOgSg9NQrZOjmbnYy+P89Ubdw= 2023-05-31T20:50:29.075 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:50:29 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:BBKOgSg9NQrZOjmbnYy+P89Ubdw=', 'Content-Length': '0'} 2023-05-31T20:50:29.077 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:48:13 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b06ff18448f55ff8-006477b315-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:50:29 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:50:29.077 DEBUG:boto:path=/pklxkd-59/obj145 2023-05-31T20:50:29.078 DEBUG:boto:auth_path=/pklxkd-59/obj145 2023-05-31T20:50:29.078 DEBUG:boto:path=/pklxkd-59/obj145?versionId=null 2023-05-31T20:50:29.079 DEBUG:boto:auth_path=/pklxkd-59/obj145?versionId=null 2023-05-31T20:50:29.079 DEBUG:boto:Method: HEAD 2023-05-31T20:50:29.079 DEBUG:boto:Path: /pklxkd-59/obj145?versionId=null 2023-05-31T20:50:29.080 DEBUG:boto:Data: 2023-05-31T20:50:29.080 DEBUG:boto:Headers: {} 2023-05-31T20:50:29.080 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:50:29.081 DEBUG:boto:Port: 8001 2023-05-31T20:50:29.081 DEBUG:boto:Params: {} 2023-05-31T20:50:29.081 DEBUG:boto:Token: None 2023-05-31T20:50:29.082 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:50:29 GMT /pklxkd-59/obj145?versionId=null 2023-05-31T20:50:29.082 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:LP4fPH9dy5HGqOHgnDttWeSIofM= 2023-05-31T20:50:29.083 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:50:29 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:LP4fPH9dy5HGqOHgnDttWeSIofM=', 'Content-Length': '0'} 2023-05-31T20:50:29.085 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:48:13 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000045c7ce986a7b0fc1-006477b315-1c17-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:50:29 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:50:29.085 DEBUG:boto:path=/pklxkd-59/obj145 2023-05-31T20:50:29.086 DEBUG:boto:auth_path=/pklxkd-59/obj145 2023-05-31T20:50:29.086 DEBUG:boto:path=/pklxkd-59/obj145?versionId=null 2023-05-31T20:50:29.086 DEBUG:boto:auth_path=/pklxkd-59/obj145?versionId=null 2023-05-31T20:50:29.087 DEBUG:boto:Method: HEAD 2023-05-31T20:50:29.087 DEBUG:boto:Path: /pklxkd-59/obj145?versionId=null 2023-05-31T20:50:29.087 DEBUG:boto:Data: 2023-05-31T20:50:29.088 DEBUG:boto:Headers: {} 2023-05-31T20:50:29.088 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:50:29.088 DEBUG:boto:Port: 8000 2023-05-31T20:50:29.089 DEBUG:boto:Params: {} 2023-05-31T20:50:29.089 DEBUG:boto:Token: None 2023-05-31T20:50:29.090 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:50:29 GMT /pklxkd-59/obj145?versionId=null 2023-05-31T20:50:29.090 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:LP4fPH9dy5HGqOHgnDttWeSIofM= 2023-05-31T20:50:29.091 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:50:29 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:LP4fPH9dy5HGqOHgnDttWeSIofM=', 'Content-Length': '0'} 2023-05-31T20:50:29.092 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:48:13 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e9002d0f711fb61d-006477b315-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:50:29 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:50:29.093 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj145 2023-05-31T20:50:29.093 DEBUG:boto:path=/pklxkd-59/obj145 2023-05-31T20:50:29.094 DEBUG:boto:auth_path=/pklxkd-59/obj145 2023-05-31T20:50:29.094 DEBUG:boto:path=/pklxkd-59/obj145?versionId=null 2023-05-31T20:50:29.095 DEBUG:boto:auth_path=/pklxkd-59/obj145?versionId=null 2023-05-31T20:50:29.095 DEBUG:boto:Method: GET 2023-05-31T20:50:29.095 DEBUG:boto:Path: /pklxkd-59/obj145?versionId=null 2023-05-31T20:50:29.096 DEBUG:boto:Data: 2023-05-31T20:50:29.096 DEBUG:boto:Headers: {} 2023-05-31T20:50:29.096 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:50:29.097 DEBUG:boto:Port: 8001 2023-05-31T20:50:29.097 DEBUG:boto:Params: {} 2023-05-31T20:50:29.098 DEBUG:boto:Token: None 2023-05-31T20:50:29.098 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:50:29 GMT /pklxkd-59/obj145?versionId=null 2023-05-31T20:50:29.099 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:BBKOgSg9NQrZOjmbnYy+P89Ubdw= 2023-05-31T20:50:29.099 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:50:29 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:BBKOgSg9NQrZOjmbnYy+P89Ubdw=', 'Content-Length': '0'} 2023-05-31T20:50:29.101 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:48:13 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ca5be801064cf358-006477b315-1c17-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:50:29 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:50:29.101 DEBUG:boto:path=/pklxkd-59/obj145 2023-05-31T20:50:29.102 DEBUG:boto:auth_path=/pklxkd-59/obj145 2023-05-31T20:50:29.102 DEBUG:boto:path=/pklxkd-59/obj145?versionId=null 2023-05-31T20:50:29.103 DEBUG:boto:auth_path=/pklxkd-59/obj145?versionId=null 2023-05-31T20:50:29.103 DEBUG:boto:Method: GET 2023-05-31T20:50:29.103 DEBUG:boto:Path: /pklxkd-59/obj145?versionId=null 2023-05-31T20:50:29.104 DEBUG:boto:Data: 2023-05-31T20:50:29.104 DEBUG:boto:Headers: {} 2023-05-31T20:50:29.105 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:50:29.105 DEBUG:boto:Port: 8000 2023-05-31T20:50:29.105 DEBUG:boto:Params: {} 2023-05-31T20:50:29.106 DEBUG:boto:Token: None 2023-05-31T20:50:29.106 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:50:29 GMT /pklxkd-59/obj145?versionId=null 2023-05-31T20:50:29.107 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:BBKOgSg9NQrZOjmbnYy+P89Ubdw= 2023-05-31T20:50:29.107 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:50:29 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:BBKOgSg9NQrZOjmbnYy+P89Ubdw=', 'Content-Length': '0'} 2023-05-31T20:50:29.109 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:48:13 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000094a2d4e54a055ed8-006477b315-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:50:29 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:50:29.152 DEBUG:boto:path=/pklxkd-59/ 2023-05-31T20:50:29.153 DEBUG:boto:auth_path=/pklxkd-59/ 2023-05-31T20:50:29.153 DEBUG:boto:path=/pklxkd-59/?prefix=obj145 2023-05-31T20:50:29.154 DEBUG:boto:auth_path=/pklxkd-59/?prefix=obj145 2023-05-31T20:50:29.154 DEBUG:boto:Method: GET 2023-05-31T20:50:29.154 DEBUG:boto:Path: /pklxkd-59/?prefix=obj145 2023-05-31T20:50:29.155 DEBUG:boto:Data: 2023-05-31T20:50:29.155 DEBUG:boto:Headers: {} 2023-05-31T20:50:29.155 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:50:29.156 DEBUG:boto:Port: 8001 2023-05-31T20:50:29.156 DEBUG:boto:Params: {} 2023-05-31T20:50:29.157 DEBUG:boto:Token: None 2023-05-31T20:50:29.157 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:50:29 GMT /pklxkd-59/ 2023-05-31T20:50:29.157 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:Oq/vWXu48It/gsNY1dtGdMC0LjA= 2023-05-31T20:50:29.158 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:50:29 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:Oq/vWXu48It/gsNY1dtGdMC0LjA=', 'Content-Length': '0'} 2023-05-31T20:50:29.160 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000002b3fc9e4a4c0f935-006477b315-1c17-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:50:29 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:50:29.161 DEBUG:boto:b'pklxkd-59obj1451000falseobj1452023-05-31T20:48:13.107Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:50:29.162 DEBUG:boto:path=/pklxkd-59/ 2023-05-31T20:50:29.162 DEBUG:boto:auth_path=/pklxkd-59/ 2023-05-31T20:50:29.162 DEBUG:boto:path=/pklxkd-59/?prefix=obj145 2023-05-31T20:50:29.163 DEBUG:boto:auth_path=/pklxkd-59/?prefix=obj145 2023-05-31T20:50:29.163 DEBUG:boto:Method: GET 2023-05-31T20:50:29.164 DEBUG:boto:Path: /pklxkd-59/?prefix=obj145 2023-05-31T20:50:29.164 DEBUG:boto:Data: 2023-05-31T20:50:29.164 DEBUG:boto:Headers: {} 2023-05-31T20:50:29.165 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:50:29.165 DEBUG:boto:Port: 8000 2023-05-31T20:50:29.165 DEBUG:boto:Params: {} 2023-05-31T20:50:29.166 DEBUG:boto:Token: None 2023-05-31T20:50:29.166 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:50:29 GMT /pklxkd-59/ 2023-05-31T20:50:29.167 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:Oq/vWXu48It/gsNY1dtGdMC0LjA= 2023-05-31T20:50:29.167 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:50:29 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:Oq/vWXu48It/gsNY1dtGdMC0LjA=', 'Content-Length': '0'} 2023-05-31T20:50:29.171 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000377e959bd126cf50-006477b315-1630-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:50:29 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:50:29.171 DEBUG:boto:b'pklxkd-59obj1451000falseobj1452023-05-31T20:48:13.107Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:50:29.172 DEBUG:boto:path=/pklxkd-59/obj145 2023-05-31T20:50:29.172 DEBUG:boto:auth_path=/pklxkd-59/obj145 2023-05-31T20:50:29.173 DEBUG:boto:Method: HEAD 2023-05-31T20:50:29.173 DEBUG:boto:Path: /pklxkd-59/obj145 2023-05-31T20:50:29.173 DEBUG:boto:Data: 2023-05-31T20:50:29.174 DEBUG:boto:Headers: {} 2023-05-31T20:50:29.174 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:50:29.174 DEBUG:boto:Port: 8001 2023-05-31T20:50:29.174 DEBUG:boto:Params: {} 2023-05-31T20:50:29.175 DEBUG:boto:Token: None 2023-05-31T20:50:29.175 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:50:29 GMT /pklxkd-59/obj145 2023-05-31T20:50:29.176 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:3bgDTZ3cieoUewq6P3gepCHWLTY= 2023-05-31T20:50:29.176 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:50:29 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:3bgDTZ3cieoUewq6P3gepCHWLTY=', 'Content-Length': '0'} 2023-05-31T20:50:29.178 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:48:13 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000191758f9b34801a7-006477b315-1c17-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:50:29 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:50:29.179 DEBUG:boto:path=/pklxkd-59/obj145 2023-05-31T20:50:29.179 DEBUG:boto:auth_path=/pklxkd-59/obj145 2023-05-31T20:50:29.179 DEBUG:boto:Method: HEAD 2023-05-31T20:50:29.180 DEBUG:boto:Path: /pklxkd-59/obj145 2023-05-31T20:50:29.180 DEBUG:boto:Data: 2023-05-31T20:50:29.180 DEBUG:boto:Headers: {} 2023-05-31T20:50:29.181 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:50:29.181 DEBUG:boto:Port: 8000 2023-05-31T20:50:29.181 DEBUG:boto:Params: {} 2023-05-31T20:50:29.182 DEBUG:boto:Token: None 2023-05-31T20:50:29.182 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:50:29 GMT /pklxkd-59/obj145 2023-05-31T20:50:29.183 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:3bgDTZ3cieoUewq6P3gepCHWLTY= 2023-05-31T20:50:29.183 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:50:29 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:3bgDTZ3cieoUewq6P3gepCHWLTY=', 'Content-Length': '0'} 2023-05-31T20:50:29.185 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:48:13 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000fcc7598c14769d89-006477b315-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:50:29 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:50:29.185 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj145 2023-05-31T20:50:29.185 DEBUG:boto:path=/pklxkd-59/obj145 2023-05-31T20:50:29.186 DEBUG:boto:auth_path=/pklxkd-59/obj145 2023-05-31T20:50:29.186 DEBUG:boto:Method: GET 2023-05-31T20:50:29.187 DEBUG:boto:Path: /pklxkd-59/obj145 2023-05-31T20:50:29.187 DEBUG:boto:Data: 2023-05-31T20:50:29.187 DEBUG:boto:Headers: {} 2023-05-31T20:50:29.188 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:50:29.188 DEBUG:boto:Port: 8001 2023-05-31T20:50:29.189 DEBUG:boto:Params: {} 2023-05-31T20:50:29.189 DEBUG:boto:Token: None 2023-05-31T20:50:29.189 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:50:29 GMT /pklxkd-59/obj145 2023-05-31T20:50:29.190 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:GYywz5ZnDEOcd4JjhvMtIOeeL7Q= 2023-05-31T20:50:29.190 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:50:29 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:GYywz5ZnDEOcd4JjhvMtIOeeL7Q=', 'Content-Length': '0'} 2023-05-31T20:50:29.192 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:48:13 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005c53081cad2894ae-006477b315-1c17-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:50:29 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:50:29.232 DEBUG:boto:path=/pklxkd-59/obj145 2023-05-31T20:50:29.233 DEBUG:boto:auth_path=/pklxkd-59/obj145 2023-05-31T20:50:29.233 DEBUG:boto:Method: GET 2023-05-31T20:50:29.234 DEBUG:boto:Path: /pklxkd-59/obj145 2023-05-31T20:50:29.234 DEBUG:boto:Data: 2023-05-31T20:50:29.234 DEBUG:boto:Headers: {} 2023-05-31T20:50:29.235 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:50:29.235 DEBUG:boto:Port: 8000 2023-05-31T20:50:29.235 DEBUG:boto:Params: {} 2023-05-31T20:50:29.236 DEBUG:boto:Token: None 2023-05-31T20:50:29.236 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:50:29 GMT /pklxkd-59/obj145 2023-05-31T20:50:29.237 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:GYywz5ZnDEOcd4JjhvMtIOeeL7Q= 2023-05-31T20:50:29.237 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:50:29 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:GYywz5ZnDEOcd4JjhvMtIOeeL7Q=', 'Content-Length': '0'} 2023-05-31T20:50:29.239 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:48:13 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d52f0aac3212c686-006477b315-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:50:29 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:50:29.239 DEBUG:boto:path=/pklxkd-59/ 2023-05-31T20:50:29.240 DEBUG:boto:auth_path=/pklxkd-59/ 2023-05-31T20:50:29.240 DEBUG:boto:path=/pklxkd-59/?prefix=obj145 2023-05-31T20:50:29.241 DEBUG:boto:auth_path=/pklxkd-59/?prefix=obj145 2023-05-31T20:50:29.241 DEBUG:boto:Method: GET 2023-05-31T20:50:29.242 DEBUG:boto:Path: /pklxkd-59/?prefix=obj145 2023-05-31T20:50:29.242 DEBUG:boto:Data: 2023-05-31T20:50:29.242 DEBUG:boto:Headers: {} 2023-05-31T20:50:29.243 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:50:29.243 DEBUG:boto:Port: 8001 2023-05-31T20:50:29.243 DEBUG:boto:Params: {} 2023-05-31T20:50:29.244 DEBUG:boto:Token: None 2023-05-31T20:50:29.244 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:50:29 GMT /pklxkd-59/ 2023-05-31T20:50:29.245 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:Oq/vWXu48It/gsNY1dtGdMC0LjA= 2023-05-31T20:50:29.245 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:50:29 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:Oq/vWXu48It/gsNY1dtGdMC0LjA=', 'Content-Length': '0'} 2023-05-31T20:50:29.247 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a8f4883878670453-006477b315-1c17-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:50:29 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:50:29.248 DEBUG:boto:b'pklxkd-59obj1451000falseobj1452023-05-31T20:48:13.107Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:50:29.248 DEBUG:boto:path=/pklxkd-59/ 2023-05-31T20:50:29.249 DEBUG:boto:auth_path=/pklxkd-59/ 2023-05-31T20:50:29.249 DEBUG:boto:path=/pklxkd-59/?prefix=obj145 2023-05-31T20:50:29.250 DEBUG:boto:auth_path=/pklxkd-59/?prefix=obj145 2023-05-31T20:50:29.250 DEBUG:boto:Method: GET 2023-05-31T20:50:29.250 DEBUG:boto:Path: /pklxkd-59/?prefix=obj145 2023-05-31T20:50:29.251 DEBUG:boto:Data: 2023-05-31T20:50:29.251 DEBUG:boto:Headers: {} 2023-05-31T20:50:29.252 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:50:29.252 DEBUG:boto:Port: 8000 2023-05-31T20:50:29.252 DEBUG:boto:Params: {} 2023-05-31T20:50:29.253 DEBUG:boto:Token: None 2023-05-31T20:50:29.253 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:50:29 GMT /pklxkd-59/ 2023-05-31T20:50:29.254 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:Oq/vWXu48It/gsNY1dtGdMC0LjA= 2023-05-31T20:50:29.254 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:50:29 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:Oq/vWXu48It/gsNY1dtGdMC0LjA=', 'Content-Length': '0'} 2023-05-31T20:50:29.257 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000009c91f4a77264db0f-006477b315-1630-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:50:29 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:50:29.258 DEBUG:boto:b'pklxkd-59obj1451000falseobj1452023-05-31T20:48:13.107Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:50:29.259 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj17 2023-05-31T20:50:29.259 DEBUG:boto:path=/pklxkd-59/obj17 2023-05-31T20:50:29.260 DEBUG:boto:auth_path=/pklxkd-59/obj17 2023-05-31T20:50:29.260 DEBUG:boto:path=/pklxkd-59/obj17?versionId=null 2023-05-31T20:50:29.260 DEBUG:boto:auth_path=/pklxkd-59/obj17?versionId=null 2023-05-31T20:50:29.261 DEBUG:boto:Method: GET 2023-05-31T20:50:29.261 DEBUG:boto:Path: /pklxkd-59/obj17?versionId=null 2023-05-31T20:50:29.262 DEBUG:boto:Data: 2023-05-31T20:50:29.262 DEBUG:boto:Headers: {} 2023-05-31T20:50:29.262 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:50:29.263 DEBUG:boto:Port: 8001 2023-05-31T20:50:29.263 DEBUG:boto:Params: {} 2023-05-31T20:50:29.264 DEBUG:boto:Token: None 2023-05-31T20:50:29.264 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:50:29 GMT /pklxkd-59/obj17?versionId=null 2023-05-31T20:50:29.265 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:r2sW4jFTy2LlNe2TdmNvFOWPufs= 2023-05-31T20:50:29.265 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:50:29 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:r2sW4jFTy2LlNe2TdmNvFOWPufs=', 'Content-Length': '0'} 2023-05-31T20:50:29.267 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:43:07 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ec8e3c01a9d5e1b6-006477b315-1c17-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:50:29 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:50:29.268 DEBUG:boto:path=/pklxkd-59/obj17 2023-05-31T20:50:29.268 DEBUG:boto:auth_path=/pklxkd-59/obj17 2023-05-31T20:50:29.269 DEBUG:boto:path=/pklxkd-59/obj17?versionId=null 2023-05-31T20:50:29.269 DEBUG:boto:auth_path=/pklxkd-59/obj17?versionId=null 2023-05-31T20:50:29.269 DEBUG:boto:Method: GET 2023-05-31T20:50:29.270 DEBUG:boto:Path: /pklxkd-59/obj17?versionId=null 2023-05-31T20:50:29.270 DEBUG:boto:Data: 2023-05-31T20:50:29.270 DEBUG:boto:Headers: {} 2023-05-31T20:50:29.271 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:50:29.271 DEBUG:boto:Port: 8000 2023-05-31T20:50:29.271 DEBUG:boto:Params: {} 2023-05-31T20:50:29.272 DEBUG:boto:Token: None 2023-05-31T20:50:29.272 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:50:29 GMT /pklxkd-59/obj17?versionId=null 2023-05-31T20:50:29.273 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:r2sW4jFTy2LlNe2TdmNvFOWPufs= 2023-05-31T20:50:29.273 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:50:29 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:r2sW4jFTy2LlNe2TdmNvFOWPufs=', 'Content-Length': '0'} 2023-05-31T20:50:29.275 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:43:07 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000729bb3b8cdfb5c47-006477b315-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:50:29 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:50:29.316 DEBUG:boto:path=/pklxkd-59/obj17 2023-05-31T20:50:29.317 DEBUG:boto:auth_path=/pklxkd-59/obj17 2023-05-31T20:50:29.317 DEBUG:boto:path=/pklxkd-59/obj17?versionId=null 2023-05-31T20:50:29.318 DEBUG:boto:auth_path=/pklxkd-59/obj17?versionId=null 2023-05-31T20:50:29.318 DEBUG:boto:Method: HEAD 2023-05-31T20:50:29.318 DEBUG:boto:Path: /pklxkd-59/obj17?versionId=null 2023-05-31T20:50:29.319 DEBUG:boto:Data: 2023-05-31T20:50:29.319 DEBUG:boto:Headers: {} 2023-05-31T20:50:29.319 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:50:29.320 DEBUG:boto:Port: 8001 2023-05-31T20:50:29.320 DEBUG:boto:Params: {} 2023-05-31T20:50:29.320 DEBUG:boto:Token: None 2023-05-31T20:50:29.321 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:50:29 GMT /pklxkd-59/obj17?versionId=null 2023-05-31T20:50:29.321 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:iqKeR2eTTZc9grxmPUTHLW9fcCc= 2023-05-31T20:50:29.322 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:50:29 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:iqKeR2eTTZc9grxmPUTHLW9fcCc=', 'Content-Length': '0'} 2023-05-31T20:50:29.324 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:43:07 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a180a819853856b8-006477b315-1c17-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:50:29 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:50:29.324 DEBUG:boto:path=/pklxkd-59/obj17 2023-05-31T20:50:29.325 DEBUG:boto:auth_path=/pklxkd-59/obj17 2023-05-31T20:50:29.325 DEBUG:boto:path=/pklxkd-59/obj17?versionId=null 2023-05-31T20:50:29.326 DEBUG:boto:auth_path=/pklxkd-59/obj17?versionId=null 2023-05-31T20:50:29.326 DEBUG:boto:Method: HEAD 2023-05-31T20:50:29.326 DEBUG:boto:Path: /pklxkd-59/obj17?versionId=null 2023-05-31T20:50:29.327 DEBUG:boto:Data: 2023-05-31T20:50:29.327 DEBUG:boto:Headers: {} 2023-05-31T20:50:29.327 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:50:29.328 DEBUG:boto:Port: 8000 2023-05-31T20:50:29.328 DEBUG:boto:Params: {} 2023-05-31T20:50:29.328 DEBUG:boto:Token: None 2023-05-31T20:50:29.329 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:50:29 GMT /pklxkd-59/obj17?versionId=null 2023-05-31T20:50:29.329 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:iqKeR2eTTZc9grxmPUTHLW9fcCc= 2023-05-31T20:50:29.330 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:50:29 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:iqKeR2eTTZc9grxmPUTHLW9fcCc=', 'Content-Length': '0'} 2023-05-31T20:50:29.331 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:43:07 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000088f456a01ef74245-006477b315-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:50:29 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:50:29.332 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj17 2023-05-31T20:50:29.333 DEBUG:boto:path=/pklxkd-59/obj17 2023-05-31T20:50:29.333 DEBUG:boto:auth_path=/pklxkd-59/obj17 2023-05-31T20:50:29.333 DEBUG:boto:path=/pklxkd-59/obj17?versionId=null 2023-05-31T20:50:29.334 DEBUG:boto:auth_path=/pklxkd-59/obj17?versionId=null 2023-05-31T20:50:29.334 DEBUG:boto:Method: GET 2023-05-31T20:50:29.334 DEBUG:boto:Path: /pklxkd-59/obj17?versionId=null 2023-05-31T20:50:29.335 DEBUG:boto:Data: 2023-05-31T20:50:29.335 DEBUG:boto:Headers: {} 2023-05-31T20:50:29.335 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:50:29.336 DEBUG:boto:Port: 8001 2023-05-31T20:50:29.336 DEBUG:boto:Params: {} 2023-05-31T20:50:29.337 DEBUG:boto:Token: None 2023-05-31T20:50:29.337 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:50:29 GMT /pklxkd-59/obj17?versionId=null 2023-05-31T20:50:29.337 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:r2sW4jFTy2LlNe2TdmNvFOWPufs= 2023-05-31T20:50:29.338 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:50:29 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:r2sW4jFTy2LlNe2TdmNvFOWPufs=', 'Content-Length': '0'} 2023-05-31T20:50:29.340 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:43:07 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000715401923f182856-006477b315-1c17-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:50:29 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:50:29.340 DEBUG:boto:path=/pklxkd-59/obj17 2023-05-31T20:50:29.341 DEBUG:boto:auth_path=/pklxkd-59/obj17 2023-05-31T20:50:29.341 DEBUG:boto:path=/pklxkd-59/obj17?versionId=null 2023-05-31T20:50:29.341 DEBUG:boto:auth_path=/pklxkd-59/obj17?versionId=null 2023-05-31T20:50:29.342 DEBUG:boto:Method: GET 2023-05-31T20:50:29.342 DEBUG:boto:Path: /pklxkd-59/obj17?versionId=null 2023-05-31T20:50:29.343 DEBUG:boto:Data: 2023-05-31T20:50:29.343 DEBUG:boto:Headers: {} 2023-05-31T20:50:29.343 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:50:29.344 DEBUG:boto:Port: 8000 2023-05-31T20:50:29.344 DEBUG:boto:Params: {} 2023-05-31T20:50:29.344 DEBUG:boto:Token: None 2023-05-31T20:50:29.345 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:50:29 GMT /pklxkd-59/obj17?versionId=null 2023-05-31T20:50:29.345 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:r2sW4jFTy2LlNe2TdmNvFOWPufs= 2023-05-31T20:50:29.346 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:50:29 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:r2sW4jFTy2LlNe2TdmNvFOWPufs=', 'Content-Length': '0'} 2023-05-31T20:50:29.348 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:43:07 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003a21020731952b97-006477b315-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:50:29 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:50:29.349 DEBUG:boto:path=/pklxkd-59/ 2023-05-31T20:50:29.349 DEBUG:boto:auth_path=/pklxkd-59/ 2023-05-31T20:50:29.349 DEBUG:boto:path=/pklxkd-59/?prefix=obj17 2023-05-31T20:50:29.350 DEBUG:boto:auth_path=/pklxkd-59/?prefix=obj17 2023-05-31T20:50:29.350 DEBUG:boto:Method: GET 2023-05-31T20:50:29.351 DEBUG:boto:Path: /pklxkd-59/?prefix=obj17 2023-05-31T20:50:29.351 DEBUG:boto:Data: 2023-05-31T20:50:29.351 DEBUG:boto:Headers: {} 2023-05-31T20:50:29.352 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:50:29.352 DEBUG:boto:Port: 8001 2023-05-31T20:50:29.352 DEBUG:boto:Params: {} 2023-05-31T20:50:29.353 DEBUG:boto:Token: None 2023-05-31T20:50:29.353 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:50:29 GMT /pklxkd-59/ 2023-05-31T20:50:29.354 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:Oq/vWXu48It/gsNY1dtGdMC0LjA= 2023-05-31T20:50:29.354 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:50:29 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:Oq/vWXu48It/gsNY1dtGdMC0LjA=', 'Content-Length': '0'} 2023-05-31T20:50:29.357 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000009d8f8ca1007776b2-006477b315-1c17-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:50:29 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:50:29.357 DEBUG:boto:b'pklxkd-59obj171000falseobj172023-05-31T20:43:07.977Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:50:29.358 DEBUG:boto:path=/pklxkd-59/ 2023-05-31T20:50:29.358 DEBUG:boto:auth_path=/pklxkd-59/ 2023-05-31T20:50:29.359 DEBUG:boto:path=/pklxkd-59/?prefix=obj17 2023-05-31T20:50:29.359 DEBUG:boto:auth_path=/pklxkd-59/?prefix=obj17 2023-05-31T20:50:29.360 DEBUG:boto:Method: GET 2023-05-31T20:50:29.360 DEBUG:boto:Path: /pklxkd-59/?prefix=obj17 2023-05-31T20:50:29.360 DEBUG:boto:Data: 2023-05-31T20:50:29.361 DEBUG:boto:Headers: {} 2023-05-31T20:50:29.361 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:50:29.361 DEBUG:boto:Port: 8000 2023-05-31T20:50:29.362 DEBUG:boto:Params: {} 2023-05-31T20:50:29.362 DEBUG:boto:Token: None 2023-05-31T20:50:29.363 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:50:29 GMT /pklxkd-59/ 2023-05-31T20:50:29.363 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:Oq/vWXu48It/gsNY1dtGdMC0LjA= 2023-05-31T20:50:29.364 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:50:29 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:Oq/vWXu48It/gsNY1dtGdMC0LjA=', 'Content-Length': '0'} 2023-05-31T20:50:29.367 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000007070e7bd473ab0db-006477b315-1630-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:50:29 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:50:29.368 DEBUG:boto:b'pklxkd-59obj171000falseobj172023-05-31T20:43:07.977Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:50:29.369 DEBUG:boto:path=/pklxkd-59/obj17 2023-05-31T20:50:29.369 DEBUG:boto:auth_path=/pklxkd-59/obj17 2023-05-31T20:50:29.369 DEBUG:boto:Method: HEAD 2023-05-31T20:50:29.370 DEBUG:boto:Path: /pklxkd-59/obj17 2023-05-31T20:50:29.370 DEBUG:boto:Data: 2023-05-31T20:50:29.370 DEBUG:boto:Headers: {} 2023-05-31T20:50:29.371 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:50:29.371 DEBUG:boto:Port: 8001 2023-05-31T20:50:29.371 DEBUG:boto:Params: {} 2023-05-31T20:50:29.372 DEBUG:boto:Token: None 2023-05-31T20:50:29.372 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:50:29 GMT /pklxkd-59/obj17 2023-05-31T20:50:29.373 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:6SBoEyrLaLMw0p2c03UMX9xC2KQ= 2023-05-31T20:50:29.373 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:50:29 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:6SBoEyrLaLMw0p2c03UMX9xC2KQ=', 'Content-Length': '0'} 2023-05-31T20:50:29.375 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:43:07 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007cb25e7d51eb50ed-006477b315-1c17-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:50:29 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:50:29.375 DEBUG:boto:path=/pklxkd-59/obj17 2023-05-31T20:50:29.376 DEBUG:boto:auth_path=/pklxkd-59/obj17 2023-05-31T20:50:29.376 DEBUG:boto:Method: HEAD 2023-05-31T20:50:29.377 DEBUG:boto:Path: /pklxkd-59/obj17 2023-05-31T20:50:29.377 DEBUG:boto:Data: 2023-05-31T20:50:29.378 DEBUG:boto:Headers: {} 2023-05-31T20:50:29.378 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:50:29.378 DEBUG:boto:Port: 8000 2023-05-31T20:50:29.379 DEBUG:boto:Params: {} 2023-05-31T20:50:29.379 DEBUG:boto:Token: None 2023-05-31T20:50:29.379 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:50:29 GMT /pklxkd-59/obj17 2023-05-31T20:50:29.380 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:6SBoEyrLaLMw0p2c03UMX9xC2KQ= 2023-05-31T20:50:29.380 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:50:29 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:6SBoEyrLaLMw0p2c03UMX9xC2KQ=', 'Content-Length': '0'} 2023-05-31T20:50:29.382 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:43:07 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000005369f0275288538-006477b315-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:50:29 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:50:29.383 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj17 2023-05-31T20:50:29.383 DEBUG:boto:path=/pklxkd-59/obj17 2023-05-31T20:50:29.383 DEBUG:boto:auth_path=/pklxkd-59/obj17 2023-05-31T20:50:29.384 DEBUG:boto:Method: GET 2023-05-31T20:50:29.384 DEBUG:boto:Path: /pklxkd-59/obj17 2023-05-31T20:50:29.385 DEBUG:boto:Data: 2023-05-31T20:50:29.385 DEBUG:boto:Headers: {} 2023-05-31T20:50:29.385 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:50:29.386 DEBUG:boto:Port: 8001 2023-05-31T20:50:29.386 DEBUG:boto:Params: {} 2023-05-31T20:50:29.386 DEBUG:boto:Token: None 2023-05-31T20:50:29.387 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:50:29 GMT /pklxkd-59/obj17 2023-05-31T20:50:29.387 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:AOtC26SQvfDZNEcUSAtfjO57G9s= 2023-05-31T20:50:29.388 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:50:29 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:AOtC26SQvfDZNEcUSAtfjO57G9s=', 'Content-Length': '0'} 2023-05-31T20:50:29.390 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:43:07 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d98de892810ba17d-006477b315-1c17-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:50:29 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:50:29.390 DEBUG:boto:path=/pklxkd-59/obj17 2023-05-31T20:50:29.391 DEBUG:boto:auth_path=/pklxkd-59/obj17 2023-05-31T20:50:29.391 DEBUG:boto:Method: GET 2023-05-31T20:50:29.392 DEBUG:boto:Path: /pklxkd-59/obj17 2023-05-31T20:50:29.392 DEBUG:boto:Data: 2023-05-31T20:50:29.392 DEBUG:boto:Headers: {} 2023-05-31T20:50:29.393 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:50:29.393 DEBUG:boto:Port: 8000 2023-05-31T20:50:29.393 DEBUG:boto:Params: {} 2023-05-31T20:50:29.394 DEBUG:boto:Token: None 2023-05-31T20:50:29.394 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:50:29 GMT /pklxkd-59/obj17 2023-05-31T20:50:29.395 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:AOtC26SQvfDZNEcUSAtfjO57G9s= 2023-05-31T20:50:29.395 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:50:29 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:AOtC26SQvfDZNEcUSAtfjO57G9s=', 'Content-Length': '0'} 2023-05-31T20:50:29.397 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:43:07 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f7fab1e2f782651d-006477b315-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:50:29 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:50:29.398 DEBUG:boto:path=/pklxkd-59/ 2023-05-31T20:50:29.398 DEBUG:boto:auth_path=/pklxkd-59/ 2023-05-31T20:50:29.399 DEBUG:boto:path=/pklxkd-59/?prefix=obj17 2023-05-31T20:50:29.399 DEBUG:boto:auth_path=/pklxkd-59/?prefix=obj17 2023-05-31T20:50:29.399 DEBUG:boto:Method: GET 2023-05-31T20:50:29.400 DEBUG:boto:Path: /pklxkd-59/?prefix=obj17 2023-05-31T20:50:29.400 DEBUG:boto:Data: 2023-05-31T20:50:29.400 DEBUG:boto:Headers: {} 2023-05-31T20:50:29.401 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:50:29.401 DEBUG:boto:Port: 8001 2023-05-31T20:50:29.402 DEBUG:boto:Params: {} 2023-05-31T20:50:29.402 DEBUG:boto:Token: None 2023-05-31T20:50:29.402 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:50:29 GMT /pklxkd-59/ 2023-05-31T20:50:29.403 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:Oq/vWXu48It/gsNY1dtGdMC0LjA= 2023-05-31T20:50:29.403 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:50:29 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:Oq/vWXu48It/gsNY1dtGdMC0LjA=', 'Content-Length': '0'} 2023-05-31T20:50:29.406 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000177a6b51ee4e556f-006477b315-1c17-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:50:29 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:50:29.406 DEBUG:boto:b'pklxkd-59obj171000falseobj172023-05-31T20:43:07.977Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:50:29.407 DEBUG:boto:path=/pklxkd-59/ 2023-05-31T20:50:29.407 DEBUG:boto:auth_path=/pklxkd-59/ 2023-05-31T20:50:29.408 DEBUG:boto:path=/pklxkd-59/?prefix=obj17 2023-05-31T20:50:29.408 DEBUG:boto:auth_path=/pklxkd-59/?prefix=obj17 2023-05-31T20:50:29.408 DEBUG:boto:Method: GET 2023-05-31T20:50:29.409 DEBUG:boto:Path: /pklxkd-59/?prefix=obj17 2023-05-31T20:50:29.409 DEBUG:boto:Data: 2023-05-31T20:50:29.410 DEBUG:boto:Headers: {} 2023-05-31T20:50:29.410 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:50:29.410 DEBUG:boto:Port: 8000 2023-05-31T20:50:29.411 DEBUG:boto:Params: {} 2023-05-31T20:50:29.411 DEBUG:boto:Token: None 2023-05-31T20:50:29.412 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:50:29 GMT /pklxkd-59/ 2023-05-31T20:50:29.412 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:Oq/vWXu48It/gsNY1dtGdMC0LjA= 2023-05-31T20:50:29.412 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:50:29 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:Oq/vWXu48It/gsNY1dtGdMC0LjA=', 'Content-Length': '0'} 2023-05-31T20:50:29.416 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000044ed82d6769ce2cd-006477b315-1630-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:50:29 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:50:29.417 DEBUG:boto:b'pklxkd-59obj171000falseobj172023-05-31T20:43:07.977Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:50:29.417 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj19 2023-05-31T20:50:29.418 DEBUG:boto:path=/pklxkd-59/obj19 2023-05-31T20:50:29.418 DEBUG:boto:auth_path=/pklxkd-59/obj19 2023-05-31T20:50:29.419 DEBUG:boto:path=/pklxkd-59/obj19?versionId=null 2023-05-31T20:50:29.419 DEBUG:boto:auth_path=/pklxkd-59/obj19?versionId=null 2023-05-31T20:50:29.419 DEBUG:boto:Method: GET 2023-05-31T20:50:29.420 DEBUG:boto:Path: /pklxkd-59/obj19?versionId=null 2023-05-31T20:50:29.420 DEBUG:boto:Data: 2023-05-31T20:50:29.420 DEBUG:boto:Headers: {} 2023-05-31T20:50:29.421 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:50:29.421 DEBUG:boto:Port: 8001 2023-05-31T20:50:29.422 DEBUG:boto:Params: {} 2023-05-31T20:50:29.422 DEBUG:boto:Token: None 2023-05-31T20:50:29.423 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:50:29 GMT /pklxkd-59/obj19?versionId=null 2023-05-31T20:50:29.423 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:+emTA4l1zvLRDxCvC/7Mn2wdxpQ= 2023-05-31T20:50:29.423 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:50:29 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:+emTA4l1zvLRDxCvC/7Mn2wdxpQ=', 'Content-Length': '0'} 2023-05-31T20:50:29.425 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:44:50 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000254cdb759c853a05-006477b315-1c17-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:50:29 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:50:29.468 DEBUG:boto:path=/pklxkd-59/obj19 2023-05-31T20:50:29.469 DEBUG:boto:auth_path=/pklxkd-59/obj19 2023-05-31T20:50:29.469 DEBUG:boto:path=/pklxkd-59/obj19?versionId=null 2023-05-31T20:50:29.470 DEBUG:boto:auth_path=/pklxkd-59/obj19?versionId=null 2023-05-31T20:50:29.470 DEBUG:boto:Method: GET 2023-05-31T20:50:29.470 DEBUG:boto:Path: /pklxkd-59/obj19?versionId=null 2023-05-31T20:50:29.471 DEBUG:boto:Data: 2023-05-31T20:50:29.471 DEBUG:boto:Headers: {} 2023-05-31T20:50:29.472 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:50:29.472 DEBUG:boto:Port: 8000 2023-05-31T20:50:29.472 DEBUG:boto:Params: {} 2023-05-31T20:50:29.473 DEBUG:boto:Token: None 2023-05-31T20:50:29.473 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:50:29 GMT /pklxkd-59/obj19?versionId=null 2023-05-31T20:50:29.474 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:+emTA4l1zvLRDxCvC/7Mn2wdxpQ= 2023-05-31T20:50:29.474 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:50:29 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:+emTA4l1zvLRDxCvC/7Mn2wdxpQ=', 'Content-Length': '0'} 2023-05-31T20:50:29.476 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:44:50 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007b13083d50b9a0c3-006477b315-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:50:29 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:50:29.477 DEBUG:boto:path=/pklxkd-59/obj19 2023-05-31T20:50:29.477 DEBUG:boto:auth_path=/pklxkd-59/obj19 2023-05-31T20:50:29.478 DEBUG:boto:path=/pklxkd-59/obj19?versionId=null 2023-05-31T20:50:29.478 DEBUG:boto:auth_path=/pklxkd-59/obj19?versionId=null 2023-05-31T20:50:29.479 DEBUG:boto:Method: HEAD 2023-05-31T20:50:29.479 DEBUG:boto:Path: /pklxkd-59/obj19?versionId=null 2023-05-31T20:50:29.479 DEBUG:boto:Data: 2023-05-31T20:50:29.480 DEBUG:boto:Headers: {} 2023-05-31T20:50:29.480 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:50:29.480 DEBUG:boto:Port: 8001 2023-05-31T20:50:29.481 DEBUG:boto:Params: {} 2023-05-31T20:50:29.481 DEBUG:boto:Token: None 2023-05-31T20:50:29.482 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:50:29 GMT /pklxkd-59/obj19?versionId=null 2023-05-31T20:50:29.482 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:s8c/rFuWV3xig90ec/DCrGtSYO0= 2023-05-31T20:50:29.482 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:50:29 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:s8c/rFuWV3xig90ec/DCrGtSYO0=', 'Content-Length': '0'} 2023-05-31T20:50:29.484 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:44:50 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b191aa41de3dc068-006477b315-1c17-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:50:29 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:50:29.485 DEBUG:boto:path=/pklxkd-59/obj19 2023-05-31T20:50:29.485 DEBUG:boto:auth_path=/pklxkd-59/obj19 2023-05-31T20:50:29.485 DEBUG:boto:path=/pklxkd-59/obj19?versionId=null 2023-05-31T20:50:29.486 DEBUG:boto:auth_path=/pklxkd-59/obj19?versionId=null 2023-05-31T20:50:29.486 DEBUG:boto:Method: HEAD 2023-05-31T20:50:29.486 DEBUG:boto:Path: /pklxkd-59/obj19?versionId=null 2023-05-31T20:50:29.487 DEBUG:boto:Data: 2023-05-31T20:50:29.487 DEBUG:boto:Headers: {} 2023-05-31T20:50:29.487 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:50:29.488 DEBUG:boto:Port: 8000 2023-05-31T20:50:29.488 DEBUG:boto:Params: {} 2023-05-31T20:50:29.489 DEBUG:boto:Token: None 2023-05-31T20:50:29.489 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:50:29 GMT /pklxkd-59/obj19?versionId=null 2023-05-31T20:50:29.489 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:s8c/rFuWV3xig90ec/DCrGtSYO0= 2023-05-31T20:50:29.490 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:50:29 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:s8c/rFuWV3xig90ec/DCrGtSYO0=', 'Content-Length': '0'} 2023-05-31T20:50:29.492 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:44:50 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000cbd969adadefb63c-006477b315-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:50:29 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:50:29.492 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj19 2023-05-31T20:50:29.493 DEBUG:boto:path=/pklxkd-59/obj19 2023-05-31T20:50:29.493 DEBUG:boto:auth_path=/pklxkd-59/obj19 2023-05-31T20:50:29.494 DEBUG:boto:path=/pklxkd-59/obj19?versionId=null 2023-05-31T20:50:29.494 DEBUG:boto:auth_path=/pklxkd-59/obj19?versionId=null 2023-05-31T20:50:29.494 DEBUG:boto:Method: GET 2023-05-31T20:50:29.495 DEBUG:boto:Path: /pklxkd-59/obj19?versionId=null 2023-05-31T20:50:29.495 DEBUG:boto:Data: 2023-05-31T20:50:29.495 DEBUG:boto:Headers: {} 2023-05-31T20:50:29.496 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:50:29.496 DEBUG:boto:Port: 8001 2023-05-31T20:50:29.497 DEBUG:boto:Params: {} 2023-05-31T20:50:29.497 DEBUG:boto:Token: None 2023-05-31T20:50:29.497 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:50:29 GMT /pklxkd-59/obj19?versionId=null 2023-05-31T20:50:29.498 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:+emTA4l1zvLRDxCvC/7Mn2wdxpQ= 2023-05-31T20:50:29.498 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:50:29 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:+emTA4l1zvLRDxCvC/7Mn2wdxpQ=', 'Content-Length': '0'} 2023-05-31T20:50:29.500 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:44:50 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000002a24c48cff36ace-006477b315-1c17-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:50:29 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:50:29.501 DEBUG:boto:path=/pklxkd-59/obj19 2023-05-31T20:50:29.501 DEBUG:boto:auth_path=/pklxkd-59/obj19 2023-05-31T20:50:29.502 DEBUG:boto:path=/pklxkd-59/obj19?versionId=null 2023-05-31T20:50:29.502 DEBUG:boto:auth_path=/pklxkd-59/obj19?versionId=null 2023-05-31T20:50:29.503 DEBUG:boto:Method: GET 2023-05-31T20:50:29.503 DEBUG:boto:Path: /pklxkd-59/obj19?versionId=null 2023-05-31T20:50:29.503 DEBUG:boto:Data: 2023-05-31T20:50:29.504 DEBUG:boto:Headers: {} 2023-05-31T20:50:29.504 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:50:29.505 DEBUG:boto:Port: 8000 2023-05-31T20:50:29.505 DEBUG:boto:Params: {} 2023-05-31T20:50:29.505 DEBUG:boto:Token: None 2023-05-31T20:50:29.506 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:50:29 GMT /pklxkd-59/obj19?versionId=null 2023-05-31T20:50:29.506 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:+emTA4l1zvLRDxCvC/7Mn2wdxpQ= 2023-05-31T20:50:29.507 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:50:29 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:+emTA4l1zvLRDxCvC/7Mn2wdxpQ=', 'Content-Length': '0'} 2023-05-31T20:50:29.509 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:44:50 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000087c290f94e6fa9c5-006477b315-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:50:29 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:50:29.552 DEBUG:boto:path=/pklxkd-59/ 2023-05-31T20:50:29.553 DEBUG:boto:auth_path=/pklxkd-59/ 2023-05-31T20:50:29.553 DEBUG:boto:path=/pklxkd-59/?prefix=obj19 2023-05-31T20:50:29.554 DEBUG:boto:auth_path=/pklxkd-59/?prefix=obj19 2023-05-31T20:50:29.554 DEBUG:boto:Method: GET 2023-05-31T20:50:29.554 DEBUG:boto:Path: /pklxkd-59/?prefix=obj19 2023-05-31T20:50:29.555 DEBUG:boto:Data: 2023-05-31T20:50:29.555 DEBUG:boto:Headers: {} 2023-05-31T20:50:29.555 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:50:29.556 DEBUG:boto:Port: 8001 2023-05-31T20:50:29.556 DEBUG:boto:Params: {} 2023-05-31T20:50:29.557 DEBUG:boto:Token: None 2023-05-31T20:50:29.557 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:50:29 GMT /pklxkd-59/ 2023-05-31T20:50:29.558 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:Oq/vWXu48It/gsNY1dtGdMC0LjA= 2023-05-31T20:50:29.558 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:50:29 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:Oq/vWXu48It/gsNY1dtGdMC0LjA=', 'Content-Length': '0'} 2023-05-31T20:50:29.560 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000005ae4e543e77d757c-006477b315-1c17-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:50:29 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:50:29.561 DEBUG:boto:b'pklxkd-59obj191000falseobj192023-05-31T20:44:50.557Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:50:29.562 DEBUG:boto:path=/pklxkd-59/ 2023-05-31T20:50:29.562 DEBUG:boto:auth_path=/pklxkd-59/ 2023-05-31T20:50:29.563 DEBUG:boto:path=/pklxkd-59/?prefix=obj19 2023-05-31T20:50:29.563 DEBUG:boto:auth_path=/pklxkd-59/?prefix=obj19 2023-05-31T20:50:29.563 DEBUG:boto:Method: GET 2023-05-31T20:50:29.564 DEBUG:boto:Path: /pklxkd-59/?prefix=obj19 2023-05-31T20:50:29.564 DEBUG:boto:Data: 2023-05-31T20:50:29.564 DEBUG:boto:Headers: {} 2023-05-31T20:50:29.565 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:50:29.565 DEBUG:boto:Port: 8000 2023-05-31T20:50:29.566 DEBUG:boto:Params: {} 2023-05-31T20:50:29.566 DEBUG:boto:Token: None 2023-05-31T20:50:29.566 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:50:29 GMT /pklxkd-59/ 2023-05-31T20:50:29.567 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:Oq/vWXu48It/gsNY1dtGdMC0LjA= 2023-05-31T20:50:29.567 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:50:29 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:Oq/vWXu48It/gsNY1dtGdMC0LjA=', 'Content-Length': '0'} 2023-05-31T20:50:29.571 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000007102792d6ebb0bd0-006477b315-1630-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:50:29 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:50:29.571 DEBUG:boto:b'pklxkd-59obj191000falseobj192023-05-31T20:44:50.557Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:50:29.572 DEBUG:boto:path=/pklxkd-59/obj19 2023-05-31T20:50:29.573 DEBUG:boto:auth_path=/pklxkd-59/obj19 2023-05-31T20:50:29.573 DEBUG:boto:Method: HEAD 2023-05-31T20:50:29.573 DEBUG:boto:Path: /pklxkd-59/obj19 2023-05-31T20:50:29.574 DEBUG:boto:Data: 2023-05-31T20:50:29.574 DEBUG:boto:Headers: {} 2023-05-31T20:50:29.574 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:50:29.575 DEBUG:boto:Port: 8001 2023-05-31T20:50:29.575 DEBUG:boto:Params: {} 2023-05-31T20:50:29.575 DEBUG:boto:Token: None 2023-05-31T20:50:29.576 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:50:29 GMT /pklxkd-59/obj19 2023-05-31T20:50:29.576 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:QJO3qn9AjMM/uXk8dhn10fmaDLU= 2023-05-31T20:50:29.577 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:50:29 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:QJO3qn9AjMM/uXk8dhn10fmaDLU=', 'Content-Length': '0'} 2023-05-31T20:50:29.578 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:44:50 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000873dc2d8329130b3-006477b315-1c17-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:50:29 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:50:29.579 DEBUG:boto:path=/pklxkd-59/obj19 2023-05-31T20:50:29.579 DEBUG:boto:auth_path=/pklxkd-59/obj19 2023-05-31T20:50:29.580 DEBUG:boto:Method: HEAD 2023-05-31T20:50:29.580 DEBUG:boto:Path: /pklxkd-59/obj19 2023-05-31T20:50:29.580 DEBUG:boto:Data: 2023-05-31T20:50:29.581 DEBUG:boto:Headers: {} 2023-05-31T20:50:29.581 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:50:29.581 DEBUG:boto:Port: 8000 2023-05-31T20:50:29.582 DEBUG:boto:Params: {} 2023-05-31T20:50:29.582 DEBUG:boto:Token: None 2023-05-31T20:50:29.582 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:50:29 GMT /pklxkd-59/obj19 2023-05-31T20:50:29.583 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:QJO3qn9AjMM/uXk8dhn10fmaDLU= 2023-05-31T20:50:29.583 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:50:29 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:QJO3qn9AjMM/uXk8dhn10fmaDLU=', 'Content-Length': '0'} 2023-05-31T20:50:29.585 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:44:50 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000025e7a281fd1e29c1-006477b315-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:50:29 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:50:29.588 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj19 2023-05-31T20:50:29.589 DEBUG:boto:path=/pklxkd-59/obj19 2023-05-31T20:50:29.589 DEBUG:boto:auth_path=/pklxkd-59/obj19 2023-05-31T20:50:29.590 DEBUG:boto:Method: GET 2023-05-31T20:50:29.590 DEBUG:boto:Path: /pklxkd-59/obj19 2023-05-31T20:50:29.590 DEBUG:boto:Data: 2023-05-31T20:50:29.591 DEBUG:boto:Headers: {} 2023-05-31T20:50:29.591 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:50:29.591 DEBUG:boto:Port: 8001 2023-05-31T20:50:29.592 DEBUG:boto:Params: {} 2023-05-31T20:50:29.592 DEBUG:boto:Token: None 2023-05-31T20:50:29.593 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:50:29 GMT /pklxkd-59/obj19 2023-05-31T20:50:29.593 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:53Np6Yrh9X6vfh1mLkb5CckLtys= 2023-05-31T20:50:29.594 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:50:29 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:53Np6Yrh9X6vfh1mLkb5CckLtys=', 'Content-Length': '0'} 2023-05-31T20:50:29.596 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:44:50 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000429b9336dd4ebc2b-006477b315-1c17-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:50:29 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:50:29.596 DEBUG:boto:path=/pklxkd-59/obj19 2023-05-31T20:50:29.597 DEBUG:boto:auth_path=/pklxkd-59/obj19 2023-05-31T20:50:29.597 DEBUG:boto:Method: GET 2023-05-31T20:50:29.598 DEBUG:boto:Path: /pklxkd-59/obj19 2023-05-31T20:50:29.598 DEBUG:boto:Data: 2023-05-31T20:50:29.598 DEBUG:boto:Headers: {} 2023-05-31T20:50:29.599 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:50:29.599 DEBUG:boto:Port: 8000 2023-05-31T20:50:29.600 DEBUG:boto:Params: {} 2023-05-31T20:50:29.600 DEBUG:boto:Token: None 2023-05-31T20:50:29.601 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:50:29 GMT /pklxkd-59/obj19 2023-05-31T20:50:29.601 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:53Np6Yrh9X6vfh1mLkb5CckLtys= 2023-05-31T20:50:29.601 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:50:29 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:53Np6Yrh9X6vfh1mLkb5CckLtys=', 'Content-Length': '0'} 2023-05-31T20:50:29.603 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:44:50 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000060e1de1995f2eb7d-006477b315-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:50:29 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:50:29.604 DEBUG:boto:path=/pklxkd-59/ 2023-05-31T20:50:29.604 DEBUG:boto:auth_path=/pklxkd-59/ 2023-05-31T20:50:29.605 DEBUG:boto:path=/pklxkd-59/?prefix=obj19 2023-05-31T20:50:29.605 DEBUG:boto:auth_path=/pklxkd-59/?prefix=obj19 2023-05-31T20:50:29.605 DEBUG:boto:Method: GET 2023-05-31T20:50:29.606 DEBUG:boto:Path: /pklxkd-59/?prefix=obj19 2023-05-31T20:50:29.606 DEBUG:boto:Data: 2023-05-31T20:50:29.607 DEBUG:boto:Headers: {} 2023-05-31T20:50:29.607 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:50:29.607 DEBUG:boto:Port: 8001 2023-05-31T20:50:29.608 DEBUG:boto:Params: {} 2023-05-31T20:50:29.608 DEBUG:boto:Token: None 2023-05-31T20:50:29.609 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:50:29 GMT /pklxkd-59/ 2023-05-31T20:50:29.609 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:Oq/vWXu48It/gsNY1dtGdMC0LjA= 2023-05-31T20:50:29.609 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:50:29 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:Oq/vWXu48It/gsNY1dtGdMC0LjA=', 'Content-Length': '0'} 2023-05-31T20:50:29.612 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a49a64b2162ae77b-006477b315-1c17-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:50:29 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:50:29.613 DEBUG:boto:b'pklxkd-59obj191000falseobj192023-05-31T20:44:50.557Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:50:29.613 DEBUG:boto:path=/pklxkd-59/ 2023-05-31T20:50:29.614 DEBUG:boto:auth_path=/pklxkd-59/ 2023-05-31T20:50:29.614 DEBUG:boto:path=/pklxkd-59/?prefix=obj19 2023-05-31T20:50:29.615 DEBUG:boto:auth_path=/pklxkd-59/?prefix=obj19 2023-05-31T20:50:29.615 DEBUG:boto:Method: GET 2023-05-31T20:50:29.615 DEBUG:boto:Path: /pklxkd-59/?prefix=obj19 2023-05-31T20:50:29.616 DEBUG:boto:Data: 2023-05-31T20:50:29.616 DEBUG:boto:Headers: {} 2023-05-31T20:50:29.616 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:50:29.617 DEBUG:boto:Port: 8000 2023-05-31T20:50:29.617 DEBUG:boto:Params: {} 2023-05-31T20:50:29.618 DEBUG:boto:Token: None 2023-05-31T20:50:29.618 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:50:29 GMT /pklxkd-59/ 2023-05-31T20:50:29.618 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:Oq/vWXu48It/gsNY1dtGdMC0LjA= 2023-05-31T20:50:29.619 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:50:29 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:Oq/vWXu48It/gsNY1dtGdMC0LjA=', 'Content-Length': '0'} 2023-05-31T20:50:29.622 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d89a88c7b0ad507b-006477b315-1630-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:50:29 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:50:29.623 DEBUG:boto:b'pklxkd-59obj191000falseobj192023-05-31T20:44:50.557Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:50:29.624 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj22 2023-05-31T20:50:29.624 DEBUG:boto:path=/pklxkd-59/obj22 2023-05-31T20:50:29.624 DEBUG:boto:auth_path=/pklxkd-59/obj22 2023-05-31T20:50:29.625 DEBUG:boto:path=/pklxkd-59/obj22?versionId=null 2023-05-31T20:50:29.625 DEBUG:boto:auth_path=/pklxkd-59/obj22?versionId=null 2023-05-31T20:50:29.626 DEBUG:boto:Method: GET 2023-05-31T20:50:29.626 DEBUG:boto:Path: /pklxkd-59/obj22?versionId=null 2023-05-31T20:50:29.626 DEBUG:boto:Data: 2023-05-31T20:50:29.627 DEBUG:boto:Headers: {} 2023-05-31T20:50:29.627 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:50:29.627 DEBUG:boto:Port: 8001 2023-05-31T20:50:29.628 DEBUG:boto:Params: {} 2023-05-31T20:50:29.628 DEBUG:boto:Token: None 2023-05-31T20:50:29.629 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:50:29 GMT /pklxkd-59/obj22?versionId=null 2023-05-31T20:50:29.629 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:PImRSFAxtGjp4A2ofCbPaEiYNLw= 2023-05-31T20:50:29.630 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:50:29 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:PImRSFAxtGjp4A2ofCbPaEiYNLw=', 'Content-Length': '0'} 2023-05-31T20:50:29.632 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:39:26 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009f11fbfeaf28c36b-006477b315-1c17-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:50:29 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:50:29.672 DEBUG:boto:path=/pklxkd-59/obj22 2023-05-31T20:50:29.673 DEBUG:boto:auth_path=/pklxkd-59/obj22 2023-05-31T20:50:29.673 DEBUG:boto:path=/pklxkd-59/obj22?versionId=null 2023-05-31T20:50:29.673 DEBUG:boto:auth_path=/pklxkd-59/obj22?versionId=null 2023-05-31T20:50:29.674 DEBUG:boto:Method: GET 2023-05-31T20:50:29.674 DEBUG:boto:Path: /pklxkd-59/obj22?versionId=null 2023-05-31T20:50:29.675 DEBUG:boto:Data: 2023-05-31T20:50:29.675 DEBUG:boto:Headers: {} 2023-05-31T20:50:29.675 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:50:29.676 DEBUG:boto:Port: 8000 2023-05-31T20:50:29.676 DEBUG:boto:Params: {} 2023-05-31T20:50:29.677 DEBUG:boto:Token: None 2023-05-31T20:50:29.677 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:50:29 GMT /pklxkd-59/obj22?versionId=null 2023-05-31T20:50:29.677 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:PImRSFAxtGjp4A2ofCbPaEiYNLw= 2023-05-31T20:50:29.678 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:50:29 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:PImRSFAxtGjp4A2ofCbPaEiYNLw=', 'Content-Length': '0'} 2023-05-31T20:50:29.680 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:39:26 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002809264ec13fea8d-006477b315-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:50:29 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:50:29.681 DEBUG:boto:path=/pklxkd-59/obj22 2023-05-31T20:50:29.681 DEBUG:boto:auth_path=/pklxkd-59/obj22 2023-05-31T20:50:29.681 DEBUG:boto:path=/pklxkd-59/obj22?versionId=null 2023-05-31T20:50:29.682 DEBUG:boto:auth_path=/pklxkd-59/obj22?versionId=null 2023-05-31T20:50:29.682 DEBUG:boto:Method: HEAD 2023-05-31T20:50:29.682 DEBUG:boto:Path: /pklxkd-59/obj22?versionId=null 2023-05-31T20:50:29.683 DEBUG:boto:Data: 2023-05-31T20:50:29.683 DEBUG:boto:Headers: {} 2023-05-31T20:50:29.683 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:50:29.684 DEBUG:boto:Port: 8001 2023-05-31T20:50:29.684 DEBUG:boto:Params: {} 2023-05-31T20:50:29.685 DEBUG:boto:Token: None 2023-05-31T20:50:29.685 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:50:29 GMT /pklxkd-59/obj22?versionId=null 2023-05-31T20:50:29.685 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:fJGTt7We4k8iRaFrgtZfjX3ONq4= 2023-05-31T20:50:29.686 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:50:29 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:fJGTt7We4k8iRaFrgtZfjX3ONq4=', 'Content-Length': '0'} 2023-05-31T20:50:29.688 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:39:26 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b3feff950481e5e3-006477b315-1c17-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:50:29 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:50:29.688 DEBUG:boto:path=/pklxkd-59/obj22 2023-05-31T20:50:29.689 DEBUG:boto:auth_path=/pklxkd-59/obj22 2023-05-31T20:50:29.689 DEBUG:boto:path=/pklxkd-59/obj22?versionId=null 2023-05-31T20:50:29.689 DEBUG:boto:auth_path=/pklxkd-59/obj22?versionId=null 2023-05-31T20:50:29.690 DEBUG:boto:Method: HEAD 2023-05-31T20:50:29.690 DEBUG:boto:Path: /pklxkd-59/obj22?versionId=null 2023-05-31T20:50:29.690 DEBUG:boto:Data: 2023-05-31T20:50:29.691 DEBUG:boto:Headers: {} 2023-05-31T20:50:29.691 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:50:29.692 DEBUG:boto:Port: 8000 2023-05-31T20:50:29.692 DEBUG:boto:Params: {} 2023-05-31T20:50:29.692 DEBUG:boto:Token: None 2023-05-31T20:50:29.693 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:50:29 GMT /pklxkd-59/obj22?versionId=null 2023-05-31T20:50:29.693 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:fJGTt7We4k8iRaFrgtZfjX3ONq4= 2023-05-31T20:50:29.694 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:50:29 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:fJGTt7We4k8iRaFrgtZfjX3ONq4=', 'Content-Length': '0'} 2023-05-31T20:50:29.695 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:39:26 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000920b7a7816ff3008-006477b315-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:50:29 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:50:29.696 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj22 2023-05-31T20:50:29.696 DEBUG:boto:path=/pklxkd-59/obj22 2023-05-31T20:50:29.697 DEBUG:boto:auth_path=/pklxkd-59/obj22 2023-05-31T20:50:29.697 DEBUG:boto:path=/pklxkd-59/obj22?versionId=null 2023-05-31T20:50:29.698 DEBUG:boto:auth_path=/pklxkd-59/obj22?versionId=null 2023-05-31T20:50:29.698 DEBUG:boto:Method: GET 2023-05-31T20:50:29.698 DEBUG:boto:Path: /pklxkd-59/obj22?versionId=null 2023-05-31T20:50:29.699 DEBUG:boto:Data: 2023-05-31T20:50:29.699 DEBUG:boto:Headers: {} 2023-05-31T20:50:29.699 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:50:29.700 DEBUG:boto:Port: 8001 2023-05-31T20:50:29.700 DEBUG:boto:Params: {} 2023-05-31T20:50:29.701 DEBUG:boto:Token: None 2023-05-31T20:50:29.701 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:50:29 GMT /pklxkd-59/obj22?versionId=null 2023-05-31T20:50:29.701 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:PImRSFAxtGjp4A2ofCbPaEiYNLw= 2023-05-31T20:50:29.702 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:50:29 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:PImRSFAxtGjp4A2ofCbPaEiYNLw=', 'Content-Length': '0'} 2023-05-31T20:50:29.704 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:39:26 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d760801ba376c981-006477b315-1c17-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:50:29 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:50:29.704 DEBUG:boto:path=/pklxkd-59/obj22 2023-05-31T20:50:29.705 DEBUG:boto:auth_path=/pklxkd-59/obj22 2023-05-31T20:50:29.705 DEBUG:boto:path=/pklxkd-59/obj22?versionId=null 2023-05-31T20:50:29.705 DEBUG:boto:auth_path=/pklxkd-59/obj22?versionId=null 2023-05-31T20:50:29.706 DEBUG:boto:Method: GET 2023-05-31T20:50:29.706 DEBUG:boto:Path: /pklxkd-59/obj22?versionId=null 2023-05-31T20:50:29.707 DEBUG:boto:Data: 2023-05-31T20:50:29.707 DEBUG:boto:Headers: {} 2023-05-31T20:50:29.707 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:50:29.708 DEBUG:boto:Port: 8000 2023-05-31T20:50:29.708 DEBUG:boto:Params: {} 2023-05-31T20:50:29.709 DEBUG:boto:Token: None 2023-05-31T20:50:29.709 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:50:29 GMT /pklxkd-59/obj22?versionId=null 2023-05-31T20:50:29.710 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:PImRSFAxtGjp4A2ofCbPaEiYNLw= 2023-05-31T20:50:29.710 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:50:29 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:PImRSFAxtGjp4A2ofCbPaEiYNLw=', 'Content-Length': '0'} 2023-05-31T20:50:29.712 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:39:26 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ed31b4fafffe86fd-006477b315-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:50:29 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:50:29.712 DEBUG:boto:path=/pklxkd-59/ 2023-05-31T20:50:29.713 DEBUG:boto:auth_path=/pklxkd-59/ 2023-05-31T20:50:29.713 DEBUG:boto:path=/pklxkd-59/?prefix=obj22 2023-05-31T20:50:29.714 DEBUG:boto:auth_path=/pklxkd-59/?prefix=obj22 2023-05-31T20:50:29.714 DEBUG:boto:Method: GET 2023-05-31T20:50:29.714 DEBUG:boto:Path: /pklxkd-59/?prefix=obj22 2023-05-31T20:50:29.715 DEBUG:boto:Data: 2023-05-31T20:50:29.715 DEBUG:boto:Headers: {} 2023-05-31T20:50:29.716 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:50:29.716 DEBUG:boto:Port: 8001 2023-05-31T20:50:29.716 DEBUG:boto:Params: {} 2023-05-31T20:50:29.717 DEBUG:boto:Token: None 2023-05-31T20:50:29.717 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:50:29 GMT /pklxkd-59/ 2023-05-31T20:50:29.718 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:Oq/vWXu48It/gsNY1dtGdMC0LjA= 2023-05-31T20:50:29.718 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:50:29 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:Oq/vWXu48It/gsNY1dtGdMC0LjA=', 'Content-Length': '0'} 2023-05-31T20:50:29.721 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d4d3039b7fdb083d-006477b315-1c17-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:50:29 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:50:29.721 DEBUG:boto:b'pklxkd-59obj221000falseobj222023-05-31T20:39:26.473Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:50:29.722 DEBUG:boto:path=/pklxkd-59/ 2023-05-31T20:50:29.722 DEBUG:boto:auth_path=/pklxkd-59/ 2023-05-31T20:50:29.723 DEBUG:boto:path=/pklxkd-59/?prefix=obj22 2023-05-31T20:50:29.723 DEBUG:boto:auth_path=/pklxkd-59/?prefix=obj22 2023-05-31T20:50:29.723 DEBUG:boto:Method: GET 2023-05-31T20:50:29.724 DEBUG:boto:Path: /pklxkd-59/?prefix=obj22 2023-05-31T20:50:29.724 DEBUG:boto:Data: 2023-05-31T20:50:29.724 DEBUG:boto:Headers: {} 2023-05-31T20:50:29.725 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:50:29.725 DEBUG:boto:Port: 8000 2023-05-31T20:50:29.726 DEBUG:boto:Params: {} 2023-05-31T20:50:29.726 DEBUG:boto:Token: None 2023-05-31T20:50:29.726 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:50:29 GMT /pklxkd-59/ 2023-05-31T20:50:29.727 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:Oq/vWXu48It/gsNY1dtGdMC0LjA= 2023-05-31T20:50:29.727 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:50:29 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:Oq/vWXu48It/gsNY1dtGdMC0LjA=', 'Content-Length': '0'} 2023-05-31T20:50:29.730 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000069e431d5e1789eb8-006477b315-1630-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:50:29 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:50:29.731 DEBUG:boto:b'pklxkd-59obj221000falseobj222023-05-31T20:39:26.473Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:50:29.732 DEBUG:boto:path=/pklxkd-59/obj22 2023-05-31T20:50:29.732 DEBUG:boto:auth_path=/pklxkd-59/obj22 2023-05-31T20:50:29.732 DEBUG:boto:Method: HEAD 2023-05-31T20:50:29.733 DEBUG:boto:Path: /pklxkd-59/obj22 2023-05-31T20:50:29.733 DEBUG:boto:Data: 2023-05-31T20:50:29.733 DEBUG:boto:Headers: {} 2023-05-31T20:50:29.734 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:50:29.734 DEBUG:boto:Port: 8001 2023-05-31T20:50:29.735 DEBUG:boto:Params: {} 2023-05-31T20:50:29.735 DEBUG:boto:Token: None 2023-05-31T20:50:29.735 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:50:29 GMT /pklxkd-59/obj22 2023-05-31T20:50:29.736 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:I4MUZyBUcgAWCDywo2kzZzh8LrE= 2023-05-31T20:50:29.736 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:50:29 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:I4MUZyBUcgAWCDywo2kzZzh8LrE=', 'Content-Length': '0'} 2023-05-31T20:50:29.738 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:39:26 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000025d00bf664430130-006477b315-1c17-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:50:29 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:50:29.739 DEBUG:boto:path=/pklxkd-59/obj22 2023-05-31T20:50:29.739 DEBUG:boto:auth_path=/pklxkd-59/obj22 2023-05-31T20:50:29.740 DEBUG:boto:Method: HEAD 2023-05-31T20:50:29.740 DEBUG:boto:Path: /pklxkd-59/obj22 2023-05-31T20:50:29.740 DEBUG:boto:Data: 2023-05-31T20:50:29.741 DEBUG:boto:Headers: {} 2023-05-31T20:50:29.741 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:50:29.741 DEBUG:boto:Port: 8000 2023-05-31T20:50:29.742 DEBUG:boto:Params: {} 2023-05-31T20:50:29.742 DEBUG:boto:Token: None 2023-05-31T20:50:29.742 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:50:29 GMT /pklxkd-59/obj22 2023-05-31T20:50:29.743 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:I4MUZyBUcgAWCDywo2kzZzh8LrE= 2023-05-31T20:50:29.743 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:50:29 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:I4MUZyBUcgAWCDywo2kzZzh8LrE=', 'Content-Length': '0'} 2023-05-31T20:50:29.745 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:39:26 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d0a188f877cc27d2-006477b315-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:50:29 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:50:29.746 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj22 2023-05-31T20:50:29.746 DEBUG:boto:path=/pklxkd-59/obj22 2023-05-31T20:50:29.746 DEBUG:boto:auth_path=/pklxkd-59/obj22 2023-05-31T20:50:29.747 DEBUG:boto:Method: GET 2023-05-31T20:50:29.747 DEBUG:boto:Path: /pklxkd-59/obj22 2023-05-31T20:50:29.748 DEBUG:boto:Data: 2023-05-31T20:50:29.748 DEBUG:boto:Headers: {} 2023-05-31T20:50:29.748 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:50:29.749 DEBUG:boto:Port: 8001 2023-05-31T20:50:29.749 DEBUG:boto:Params: {} 2023-05-31T20:50:29.750 DEBUG:boto:Token: None 2023-05-31T20:50:29.750 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:50:29 GMT /pklxkd-59/obj22 2023-05-31T20:50:29.750 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:re1AXdj9QcyTNkVRL1Rb9lXuYJc= 2023-05-31T20:50:29.751 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:50:29 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:re1AXdj9QcyTNkVRL1Rb9lXuYJc=', 'Content-Length': '0'} 2023-05-31T20:50:29.753 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:39:26 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000cfc6a4e935d24bf6-006477b315-1c17-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:50:29 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:50:29.753 DEBUG:boto:path=/pklxkd-59/obj22 2023-05-31T20:50:29.754 DEBUG:boto:auth_path=/pklxkd-59/obj22 2023-05-31T20:50:29.754 DEBUG:boto:Method: GET 2023-05-31T20:50:29.755 DEBUG:boto:Path: /pklxkd-59/obj22 2023-05-31T20:50:29.755 DEBUG:boto:Data: 2023-05-31T20:50:29.755 DEBUG:boto:Headers: {} 2023-05-31T20:50:29.756 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:50:29.756 DEBUG:boto:Port: 8000 2023-05-31T20:50:29.757 DEBUG:boto:Params: {} 2023-05-31T20:50:29.757 DEBUG:boto:Token: None 2023-05-31T20:50:29.757 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:50:29 GMT /pklxkd-59/obj22 2023-05-31T20:50:29.758 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:re1AXdj9QcyTNkVRL1Rb9lXuYJc= 2023-05-31T20:50:29.758 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:50:29 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:re1AXdj9QcyTNkVRL1Rb9lXuYJc=', 'Content-Length': '0'} 2023-05-31T20:50:29.760 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:39:26 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000fc6e15af4ceee84b-006477b315-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:50:29 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:50:29.804 DEBUG:boto:path=/pklxkd-59/ 2023-05-31T20:50:29.806 DEBUG:boto:auth_path=/pklxkd-59/ 2023-05-31T20:50:29.806 DEBUG:boto:path=/pklxkd-59/?prefix=obj22 2023-05-31T20:50:29.807 DEBUG:boto:auth_path=/pklxkd-59/?prefix=obj22 2023-05-31T20:50:29.808 DEBUG:boto:Method: GET 2023-05-31T20:50:29.808 DEBUG:boto:Path: /pklxkd-59/?prefix=obj22 2023-05-31T20:50:29.809 DEBUG:boto:Data: 2023-05-31T20:50:29.810 DEBUG:boto:Headers: {} 2023-05-31T20:50:29.811 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:50:29.811 DEBUG:boto:Port: 8001 2023-05-31T20:50:29.812 DEBUG:boto:Params: {} 2023-05-31T20:50:29.813 DEBUG:boto:Token: None 2023-05-31T20:50:29.814 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:50:29 GMT /pklxkd-59/ 2023-05-31T20:50:29.815 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:Oq/vWXu48It/gsNY1dtGdMC0LjA= 2023-05-31T20:50:29.815 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:50:29 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:Oq/vWXu48It/gsNY1dtGdMC0LjA=', 'Content-Length': '0'} 2023-05-31T20:50:29.818 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000009d16da751a67b324-006477b315-1c17-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:50:29 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:50:29.819 DEBUG:boto:b'pklxkd-59obj221000falseobj222023-05-31T20:39:26.473Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:50:29.820 DEBUG:boto:path=/pklxkd-59/ 2023-05-31T20:50:29.821 DEBUG:boto:auth_path=/pklxkd-59/ 2023-05-31T20:50:29.822 DEBUG:boto:path=/pklxkd-59/?prefix=obj22 2023-05-31T20:50:29.822 DEBUG:boto:auth_path=/pklxkd-59/?prefix=obj22 2023-05-31T20:50:29.823 DEBUG:boto:Method: GET 2023-05-31T20:50:29.824 DEBUG:boto:Path: /pklxkd-59/?prefix=obj22 2023-05-31T20:50:29.825 DEBUG:boto:Data: 2023-05-31T20:50:29.826 DEBUG:boto:Headers: {} 2023-05-31T20:50:29.828 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:50:29.829 DEBUG:boto:Port: 8000 2023-05-31T20:50:29.830 DEBUG:boto:Params: {} 2023-05-31T20:50:29.831 DEBUG:boto:Token: None 2023-05-31T20:50:29.832 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:50:29 GMT /pklxkd-59/ 2023-05-31T20:50:29.833 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:Oq/vWXu48It/gsNY1dtGdMC0LjA= 2023-05-31T20:50:29.834 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:50:29 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:Oq/vWXu48It/gsNY1dtGdMC0LjA=', 'Content-Length': '0'} 2023-05-31T20:50:29.838 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000003353652802df6d81-006477b315-1630-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:50:29 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:50:29.839 DEBUG:boto:b'pklxkd-59obj221000falseobj222023-05-31T20:39:26.473Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:50:29.840 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj23 2023-05-31T20:50:29.841 DEBUG:boto:path=/pklxkd-59/obj23 2023-05-31T20:50:29.842 DEBUG:boto:auth_path=/pklxkd-59/obj23 2023-05-31T20:50:29.842 DEBUG:boto:path=/pklxkd-59/obj23?versionId=null 2023-05-31T20:50:29.843 DEBUG:boto:auth_path=/pklxkd-59/obj23?versionId=null 2023-05-31T20:50:29.844 DEBUG:boto:Method: GET 2023-05-31T20:50:29.844 DEBUG:boto:Path: /pklxkd-59/obj23?versionId=null 2023-05-31T20:50:29.845 DEBUG:boto:Data: 2023-05-31T20:50:29.846 DEBUG:boto:Headers: {} 2023-05-31T20:50:29.846 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:50:29.847 DEBUG:boto:Port: 8001 2023-05-31T20:50:29.848 DEBUG:boto:Params: {} 2023-05-31T20:50:29.849 DEBUG:boto:Token: None 2023-05-31T20:50:29.849 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:50:29 GMT /pklxkd-59/obj23?versionId=null 2023-05-31T20:50:29.850 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:IL1DAhJ62ScAt2k/ZdFWcUTTzbk= 2023-05-31T20:50:29.851 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:50:29 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:IL1DAhJ62ScAt2k/ZdFWcUTTzbk=', 'Content-Length': '0'} 2023-05-31T20:50:29.853 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:46:33 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d66f69027ef40eaa-006477b315-1c17-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:50:29 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:50:29.854 DEBUG:boto:path=/pklxkd-59/obj23 2023-05-31T20:50:29.855 DEBUG:boto:auth_path=/pklxkd-59/obj23 2023-05-31T20:50:29.856 DEBUG:boto:path=/pklxkd-59/obj23?versionId=null 2023-05-31T20:50:29.856 DEBUG:boto:auth_path=/pklxkd-59/obj23?versionId=null 2023-05-31T20:50:29.857 DEBUG:boto:Method: GET 2023-05-31T20:50:29.858 DEBUG:boto:Path: /pklxkd-59/obj23?versionId=null 2023-05-31T20:50:29.858 DEBUG:boto:Data: 2023-05-31T20:50:29.859 DEBUG:boto:Headers: {} 2023-05-31T20:50:29.859 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:50:29.860 DEBUG:boto:Port: 8000 2023-05-31T20:50:29.860 DEBUG:boto:Params: {} 2023-05-31T20:50:29.860 DEBUG:boto:Token: None 2023-05-31T20:50:29.861 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:50:29 GMT /pklxkd-59/obj23?versionId=null 2023-05-31T20:50:29.861 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:IL1DAhJ62ScAt2k/ZdFWcUTTzbk= 2023-05-31T20:50:29.862 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:50:29 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:IL1DAhJ62ScAt2k/ZdFWcUTTzbk=', 'Content-Length': '0'} 2023-05-31T20:50:29.863 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:46:33 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001ca18b1970e0a0e3-006477b315-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:50:29 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:50:29.864 DEBUG:boto:path=/pklxkd-59/obj23 2023-05-31T20:50:29.864 DEBUG:boto:auth_path=/pklxkd-59/obj23 2023-05-31T20:50:29.865 DEBUG:boto:path=/pklxkd-59/obj23?versionId=null 2023-05-31T20:50:29.865 DEBUG:boto:auth_path=/pklxkd-59/obj23?versionId=null 2023-05-31T20:50:29.865 DEBUG:boto:Method: HEAD 2023-05-31T20:50:29.866 DEBUG:boto:Path: /pklxkd-59/obj23?versionId=null 2023-05-31T20:50:29.866 DEBUG:boto:Data: 2023-05-31T20:50:29.866 DEBUG:boto:Headers: {} 2023-05-31T20:50:29.867 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:50:29.867 DEBUG:boto:Port: 8001 2023-05-31T20:50:29.868 DEBUG:boto:Params: {} 2023-05-31T20:50:29.868 DEBUG:boto:Token: None 2023-05-31T20:50:29.868 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:50:29 GMT /pklxkd-59/obj23?versionId=null 2023-05-31T20:50:29.869 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:BfjStXPEFnYnvLcQ9UOILEVLYxk= 2023-05-31T20:50:29.869 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:50:29 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:BfjStXPEFnYnvLcQ9UOILEVLYxk=', 'Content-Length': '0'} 2023-05-31T20:50:29.871 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:46:33 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003d1aa24c4ac0d196-006477b315-1c17-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:50:29 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:50:29.871 DEBUG:boto:path=/pklxkd-59/obj23 2023-05-31T20:50:29.872 DEBUG:boto:auth_path=/pklxkd-59/obj23 2023-05-31T20:50:29.872 DEBUG:boto:path=/pklxkd-59/obj23?versionId=null 2023-05-31T20:50:29.872 DEBUG:boto:auth_path=/pklxkd-59/obj23?versionId=null 2023-05-31T20:50:29.873 DEBUG:boto:Method: HEAD 2023-05-31T20:50:29.873 DEBUG:boto:Path: /pklxkd-59/obj23?versionId=null 2023-05-31T20:50:29.873 DEBUG:boto:Data: 2023-05-31T20:50:29.874 DEBUG:boto:Headers: {} 2023-05-31T20:50:29.874 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:50:29.875 DEBUG:boto:Port: 8000 2023-05-31T20:50:29.875 DEBUG:boto:Params: {} 2023-05-31T20:50:29.875 DEBUG:boto:Token: None 2023-05-31T20:50:29.876 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:50:29 GMT /pklxkd-59/obj23?versionId=null 2023-05-31T20:50:29.876 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:BfjStXPEFnYnvLcQ9UOILEVLYxk= 2023-05-31T20:50:29.877 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:50:29 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:BfjStXPEFnYnvLcQ9UOILEVLYxk=', 'Content-Length': '0'} 2023-05-31T20:50:29.878 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:46:33 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000129aa514032d4883-006477b315-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:50:29 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:50:29.879 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj23 2023-05-31T20:50:29.879 DEBUG:boto:path=/pklxkd-59/obj23 2023-05-31T20:50:29.880 DEBUG:boto:auth_path=/pklxkd-59/obj23 2023-05-31T20:50:29.880 DEBUG:boto:path=/pklxkd-59/obj23?versionId=null 2023-05-31T20:50:29.881 DEBUG:boto:auth_path=/pklxkd-59/obj23?versionId=null 2023-05-31T20:50:29.881 DEBUG:boto:Method: GET 2023-05-31T20:50:29.881 DEBUG:boto:Path: /pklxkd-59/obj23?versionId=null 2023-05-31T20:50:29.882 DEBUG:boto:Data: 2023-05-31T20:50:29.882 DEBUG:boto:Headers: {} 2023-05-31T20:50:29.883 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:50:29.883 DEBUG:boto:Port: 8001 2023-05-31T20:50:29.883 DEBUG:boto:Params: {} 2023-05-31T20:50:29.884 DEBUG:boto:Token: None 2023-05-31T20:50:29.884 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:50:29 GMT /pklxkd-59/obj23?versionId=null 2023-05-31T20:50:29.885 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:IL1DAhJ62ScAt2k/ZdFWcUTTzbk= 2023-05-31T20:50:29.885 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:50:29 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:IL1DAhJ62ScAt2k/ZdFWcUTTzbk=', 'Content-Length': '0'} 2023-05-31T20:50:29.887 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:46:33 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000cac0504d84b23f5c-006477b315-1c17-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:50:29 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:50:29.928 DEBUG:boto:path=/pklxkd-59/obj23 2023-05-31T20:50:29.929 DEBUG:boto:auth_path=/pklxkd-59/obj23 2023-05-31T20:50:29.929 DEBUG:boto:path=/pklxkd-59/obj23?versionId=null 2023-05-31T20:50:29.930 DEBUG:boto:auth_path=/pklxkd-59/obj23?versionId=null 2023-05-31T20:50:29.930 DEBUG:boto:Method: GET 2023-05-31T20:50:29.931 DEBUG:boto:Path: /pklxkd-59/obj23?versionId=null 2023-05-31T20:50:29.931 DEBUG:boto:Data: 2023-05-31T20:50:29.931 DEBUG:boto:Headers: {} 2023-05-31T20:50:29.932 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:50:29.932 DEBUG:boto:Port: 8000 2023-05-31T20:50:29.932 DEBUG:boto:Params: {} 2023-05-31T20:50:29.933 DEBUG:boto:Token: None 2023-05-31T20:50:29.933 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:50:29 GMT /pklxkd-59/obj23?versionId=null 2023-05-31T20:50:29.934 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:IL1DAhJ62ScAt2k/ZdFWcUTTzbk= 2023-05-31T20:50:29.934 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:50:29 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:IL1DAhJ62ScAt2k/ZdFWcUTTzbk=', 'Content-Length': '0'} 2023-05-31T20:50:29.936 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:46:33 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000041dcaa02f4c9b65f-006477b315-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:50:29 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:50:29.937 DEBUG:boto:path=/pklxkd-59/ 2023-05-31T20:50:29.938 DEBUG:boto:auth_path=/pklxkd-59/ 2023-05-31T20:50:29.938 DEBUG:boto:path=/pklxkd-59/?prefix=obj23 2023-05-31T20:50:29.938 DEBUG:boto:auth_path=/pklxkd-59/?prefix=obj23 2023-05-31T20:50:29.939 DEBUG:boto:Method: GET 2023-05-31T20:50:29.939 DEBUG:boto:Path: /pklxkd-59/?prefix=obj23 2023-05-31T20:50:29.939 DEBUG:boto:Data: 2023-05-31T20:50:29.940 DEBUG:boto:Headers: {} 2023-05-31T20:50:29.940 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:50:29.940 DEBUG:boto:Port: 8001 2023-05-31T20:50:29.941 DEBUG:boto:Params: {} 2023-05-31T20:50:29.941 DEBUG:boto:Token: None 2023-05-31T20:50:29.942 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:50:29 GMT /pklxkd-59/ 2023-05-31T20:50:29.942 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:Oq/vWXu48It/gsNY1dtGdMC0LjA= 2023-05-31T20:50:29.943 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:50:29 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:Oq/vWXu48It/gsNY1dtGdMC0LjA=', 'Content-Length': '0'} 2023-05-31T20:50:29.945 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000064dcf8703b02d013-006477b315-1c17-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:50:29 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:50:29.946 DEBUG:boto:b'pklxkd-59obj231000falseobj232023-05-31T20:46:33.813Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:50:29.946 DEBUG:boto:path=/pklxkd-59/ 2023-05-31T20:50:29.947 DEBUG:boto:auth_path=/pklxkd-59/ 2023-05-31T20:50:29.947 DEBUG:boto:path=/pklxkd-59/?prefix=obj23 2023-05-31T20:50:29.948 DEBUG:boto:auth_path=/pklxkd-59/?prefix=obj23 2023-05-31T20:50:29.948 DEBUG:boto:Method: GET 2023-05-31T20:50:29.948 DEBUG:boto:Path: /pklxkd-59/?prefix=obj23 2023-05-31T20:50:29.949 DEBUG:boto:Data: 2023-05-31T20:50:29.949 DEBUG:boto:Headers: {} 2023-05-31T20:50:29.950 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:50:29.950 DEBUG:boto:Port: 8000 2023-05-31T20:50:29.950 DEBUG:boto:Params: {} 2023-05-31T20:50:29.951 DEBUG:boto:Token: None 2023-05-31T20:50:29.951 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:50:29 GMT /pklxkd-59/ 2023-05-31T20:50:29.952 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:Oq/vWXu48It/gsNY1dtGdMC0LjA= 2023-05-31T20:50:29.952 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:50:29 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:Oq/vWXu48It/gsNY1dtGdMC0LjA=', 'Content-Length': '0'} 2023-05-31T20:50:29.955 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000009a96f6106f8ffc61-006477b315-1630-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:50:29 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:50:29.996 DEBUG:boto:b'pklxkd-59obj231000falseobj232023-05-31T20:46:33.813Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:50:29.997 DEBUG:boto:path=/pklxkd-59/obj23 2023-05-31T20:50:29.998 DEBUG:boto:auth_path=/pklxkd-59/obj23 2023-05-31T20:50:29.998 DEBUG:boto:Method: HEAD 2023-05-31T20:50:29.998 DEBUG:boto:Path: /pklxkd-59/obj23 2023-05-31T20:50:29.999 DEBUG:boto:Data: 2023-05-31T20:50:29.999 DEBUG:boto:Headers: {} 2023-05-31T20:50:29.999 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:50:30.000 DEBUG:boto:Port: 8001 2023-05-31T20:50:30.000 DEBUG:boto:Params: {} 2023-05-31T20:50:30.001 DEBUG:boto:Token: None 2023-05-31T20:50:30.001 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:50:30 GMT /pklxkd-59/obj23 2023-05-31T20:50:30.001 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:X3kWP4t4QBrZZlyPrxYzB2OXkgQ= 2023-05-31T20:50:30.002 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:50:30 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:X3kWP4t4QBrZZlyPrxYzB2OXkgQ=', 'Content-Length': '0'} 2023-05-31T20:50:30.004 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:46:33 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000133eea41420f90d3-006477b315-1c17-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:50:29 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:50:30.004 DEBUG:boto:path=/pklxkd-59/obj23 2023-05-31T20:50:30.005 DEBUG:boto:auth_path=/pklxkd-59/obj23 2023-05-31T20:50:30.005 DEBUG:boto:Method: HEAD 2023-05-31T20:50:30.006 DEBUG:boto:Path: /pklxkd-59/obj23 2023-05-31T20:50:30.006 DEBUG:boto:Data: 2023-05-31T20:50:30.006 DEBUG:boto:Headers: {} 2023-05-31T20:50:30.007 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:50:30.007 DEBUG:boto:Port: 8000 2023-05-31T20:50:30.007 DEBUG:boto:Params: {} 2023-05-31T20:50:30.008 DEBUG:boto:Token: None 2023-05-31T20:50:30.008 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:50:30 GMT /pklxkd-59/obj23 2023-05-31T20:50:30.009 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:X3kWP4t4QBrZZlyPrxYzB2OXkgQ= 2023-05-31T20:50:30.009 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:50:30 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:X3kWP4t4QBrZZlyPrxYzB2OXkgQ=', 'Content-Length': '0'} 2023-05-31T20:50:30.011 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:46:33 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009220b84a7bc5c677-006477b316-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:50:30 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:50:30.011 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj23 2023-05-31T20:50:30.012 DEBUG:boto:path=/pklxkd-59/obj23 2023-05-31T20:50:30.012 DEBUG:boto:auth_path=/pklxkd-59/obj23 2023-05-31T20:50:30.013 DEBUG:boto:Method: GET 2023-05-31T20:50:30.013 DEBUG:boto:Path: /pklxkd-59/obj23 2023-05-31T20:50:30.013 DEBUG:boto:Data: 2023-05-31T20:50:30.014 DEBUG:boto:Headers: {} 2023-05-31T20:50:30.014 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:50:30.014 DEBUG:boto:Port: 8001 2023-05-31T20:50:30.015 DEBUG:boto:Params: {} 2023-05-31T20:50:30.015 DEBUG:boto:Token: None 2023-05-31T20:50:30.016 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:50:30 GMT /pklxkd-59/obj23 2023-05-31T20:50:30.016 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:Sq62huapxpiUaiJfaaA1TLKKZDA= 2023-05-31T20:50:30.017 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:50:30 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:Sq62huapxpiUaiJfaaA1TLKKZDA=', 'Content-Length': '0'} 2023-05-31T20:50:30.018 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:46:33 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000594e76eff78c9393-006477b316-1c17-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:50:30 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:50:30.019 DEBUG:boto:path=/pklxkd-59/obj23 2023-05-31T20:50:30.019 DEBUG:boto:auth_path=/pklxkd-59/obj23 2023-05-31T20:50:30.020 DEBUG:boto:Method: GET 2023-05-31T20:50:30.020 DEBUG:boto:Path: /pklxkd-59/obj23 2023-05-31T20:50:30.020 DEBUG:boto:Data: 2023-05-31T20:50:30.021 DEBUG:boto:Headers: {} 2023-05-31T20:50:30.021 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:50:30.022 DEBUG:boto:Port: 8000 2023-05-31T20:50:30.022 DEBUG:boto:Params: {} 2023-05-31T20:50:30.023 DEBUG:boto:Token: None 2023-05-31T20:50:30.023 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:50:30 GMT /pklxkd-59/obj23 2023-05-31T20:50:30.024 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:Sq62huapxpiUaiJfaaA1TLKKZDA= 2023-05-31T20:50:30.024 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:50:30 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:Sq62huapxpiUaiJfaaA1TLKKZDA=', 'Content-Length': '0'} 2023-05-31T20:50:30.026 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:46:33 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000090d791342d9ff524-006477b316-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:50:30 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:50:30.027 DEBUG:boto:path=/pklxkd-59/ 2023-05-31T20:50:30.028 DEBUG:boto:auth_path=/pklxkd-59/ 2023-05-31T20:50:30.028 DEBUG:boto:path=/pklxkd-59/?prefix=obj23 2023-05-31T20:50:30.029 DEBUG:boto:auth_path=/pklxkd-59/?prefix=obj23 2023-05-31T20:50:30.029 DEBUG:boto:Method: GET 2023-05-31T20:50:30.030 DEBUG:boto:Path: /pklxkd-59/?prefix=obj23 2023-05-31T20:50:30.030 DEBUG:boto:Data: 2023-05-31T20:50:30.030 DEBUG:boto:Headers: {} 2023-05-31T20:50:30.031 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:50:30.031 DEBUG:boto:Port: 8001 2023-05-31T20:50:30.032 DEBUG:boto:Params: {} 2023-05-31T20:50:30.032 DEBUG:boto:Token: None 2023-05-31T20:50:30.032 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:50:30 GMT /pklxkd-59/ 2023-05-31T20:50:30.033 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:3Yz4bfuBElJRjudql5UcebaTIW4= 2023-05-31T20:50:30.033 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:50:30 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:3Yz4bfuBElJRjudql5UcebaTIW4=', 'Content-Length': '0'} 2023-05-31T20:50:30.036 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000200a4c1c07ecf841-006477b316-1c17-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:50:30 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:50:30.036 DEBUG:boto:b'pklxkd-59obj231000falseobj232023-05-31T20:46:33.813Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:50:30.037 DEBUG:boto:path=/pklxkd-59/ 2023-05-31T20:50:30.037 DEBUG:boto:auth_path=/pklxkd-59/ 2023-05-31T20:50:30.038 DEBUG:boto:path=/pklxkd-59/?prefix=obj23 2023-05-31T20:50:30.038 DEBUG:boto:auth_path=/pklxkd-59/?prefix=obj23 2023-05-31T20:50:30.039 DEBUG:boto:Method: GET 2023-05-31T20:50:30.039 DEBUG:boto:Path: /pklxkd-59/?prefix=obj23 2023-05-31T20:50:30.039 DEBUG:boto:Data: 2023-05-31T20:50:30.040 DEBUG:boto:Headers: {} 2023-05-31T20:50:30.040 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:50:30.040 DEBUG:boto:Port: 8000 2023-05-31T20:50:30.041 DEBUG:boto:Params: {} 2023-05-31T20:50:30.041 DEBUG:boto:Token: None 2023-05-31T20:50:30.042 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:50:30 GMT /pklxkd-59/ 2023-05-31T20:50:30.042 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:3Yz4bfuBElJRjudql5UcebaTIW4= 2023-05-31T20:50:30.042 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:50:30 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:3Yz4bfuBElJRjudql5UcebaTIW4=', 'Content-Length': '0'} 2023-05-31T20:50:30.046 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000efe5c7c5eda9d971-006477b316-1630-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:50:30 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:50:30.047 DEBUG:boto:b'pklxkd-59obj231000falseobj232023-05-31T20:46:33.813Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:50:30.048 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj29 2023-05-31T20:50:30.048 DEBUG:boto:path=/pklxkd-59/obj29 2023-05-31T20:50:30.049 DEBUG:boto:auth_path=/pklxkd-59/obj29 2023-05-31T20:50:30.049 DEBUG:boto:path=/pklxkd-59/obj29?versionId=null 2023-05-31T20:50:30.049 DEBUG:boto:auth_path=/pklxkd-59/obj29?versionId=null 2023-05-31T20:50:30.050 DEBUG:boto:Method: GET 2023-05-31T20:50:30.050 DEBUG:boto:Path: /pklxkd-59/obj29?versionId=null 2023-05-31T20:50:30.051 DEBUG:boto:Data: 2023-05-31T20:50:30.051 DEBUG:boto:Headers: {} 2023-05-31T20:50:30.051 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:50:30.052 DEBUG:boto:Port: 8001 2023-05-31T20:50:30.052 DEBUG:boto:Params: {} 2023-05-31T20:50:30.052 DEBUG:boto:Token: None 2023-05-31T20:50:30.053 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:50:30 GMT /pklxkd-59/obj29?versionId=null 2023-05-31T20:50:30.053 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:M3xDYH3LpxNMn+52oozoUpzMXcU= 2023-05-31T20:50:30.054 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:50:30 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:M3xDYH3LpxNMn+52oozoUpzMXcU=', 'Content-Length': '0'} 2023-05-31T20:50:30.056 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:48:12 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000232e781ef2e6fbb2-006477b316-1c17-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:50:30 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:50:30.056 DEBUG:boto:path=/pklxkd-59/obj29 2023-05-31T20:50:30.057 DEBUG:boto:auth_path=/pklxkd-59/obj29 2023-05-31T20:50:30.057 DEBUG:boto:path=/pklxkd-59/obj29?versionId=null 2023-05-31T20:50:30.057 DEBUG:boto:auth_path=/pklxkd-59/obj29?versionId=null 2023-05-31T20:50:30.058 DEBUG:boto:Method: GET 2023-05-31T20:50:30.058 DEBUG:boto:Path: /pklxkd-59/obj29?versionId=null 2023-05-31T20:50:30.059 DEBUG:boto:Data: 2023-05-31T20:50:30.059 DEBUG:boto:Headers: {} 2023-05-31T20:50:30.059 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:50:30.060 DEBUG:boto:Port: 8000 2023-05-31T20:50:30.060 DEBUG:boto:Params: {} 2023-05-31T20:50:30.060 DEBUG:boto:Token: None 2023-05-31T20:50:30.061 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:50:30 GMT /pklxkd-59/obj29?versionId=null 2023-05-31T20:50:30.061 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:M3xDYH3LpxNMn+52oozoUpzMXcU= 2023-05-31T20:50:30.062 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:50:30 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:M3xDYH3LpxNMn+52oozoUpzMXcU=', 'Content-Length': '0'} 2023-05-31T20:50:30.064 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:48:12 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003ae42ead49d9065b-006477b316-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:50:30 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:50:30.065 DEBUG:boto:path=/pklxkd-59/obj29 2023-05-31T20:50:30.065 DEBUG:boto:auth_path=/pklxkd-59/obj29 2023-05-31T20:50:30.065 DEBUG:boto:path=/pklxkd-59/obj29?versionId=null 2023-05-31T20:50:30.066 DEBUG:boto:auth_path=/pklxkd-59/obj29?versionId=null 2023-05-31T20:50:30.066 DEBUG:boto:Method: HEAD 2023-05-31T20:50:30.066 DEBUG:boto:Path: /pklxkd-59/obj29?versionId=null 2023-05-31T20:50:30.067 DEBUG:boto:Data: 2023-05-31T20:50:30.067 DEBUG:boto:Headers: {} 2023-05-31T20:50:30.067 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:50:30.068 DEBUG:boto:Port: 8001 2023-05-31T20:50:30.068 DEBUG:boto:Params: {} 2023-05-31T20:50:30.069 DEBUG:boto:Token: None 2023-05-31T20:50:30.069 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:50:30 GMT /pklxkd-59/obj29?versionId=null 2023-05-31T20:50:30.069 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:8mtEqmL1LXFLfCxoQDh0pWj+wHw= 2023-05-31T20:50:30.070 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:50:30 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:8mtEqmL1LXFLfCxoQDh0pWj+wHw=', 'Content-Length': '0'} 2023-05-31T20:50:30.071 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:48:12 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f66c42462138c922-006477b316-1c17-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:50:30 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:50:30.072 DEBUG:boto:path=/pklxkd-59/obj29 2023-05-31T20:50:30.072 DEBUG:boto:auth_path=/pklxkd-59/obj29 2023-05-31T20:50:30.073 DEBUG:boto:path=/pklxkd-59/obj29?versionId=null 2023-05-31T20:50:30.073 DEBUG:boto:auth_path=/pklxkd-59/obj29?versionId=null 2023-05-31T20:50:30.073 DEBUG:boto:Method: HEAD 2023-05-31T20:50:30.074 DEBUG:boto:Path: /pklxkd-59/obj29?versionId=null 2023-05-31T20:50:30.074 DEBUG:boto:Data: 2023-05-31T20:50:30.074 DEBUG:boto:Headers: {} 2023-05-31T20:50:30.075 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:50:30.075 DEBUG:boto:Port: 8000 2023-05-31T20:50:30.075 DEBUG:boto:Params: {} 2023-05-31T20:50:30.076 DEBUG:boto:Token: None 2023-05-31T20:50:30.076 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:50:30 GMT /pklxkd-59/obj29?versionId=null 2023-05-31T20:50:30.077 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:8mtEqmL1LXFLfCxoQDh0pWj+wHw= 2023-05-31T20:50:30.077 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:50:30 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:8mtEqmL1LXFLfCxoQDh0pWj+wHw=', 'Content-Length': '0'} 2023-05-31T20:50:30.079 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:48:12 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a62fecf7901fddb2-006477b316-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:50:30 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:50:30.079 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj29 2023-05-31T20:50:30.080 DEBUG:boto:path=/pklxkd-59/obj29 2023-05-31T20:50:30.080 DEBUG:boto:auth_path=/pklxkd-59/obj29 2023-05-31T20:50:30.080 DEBUG:boto:path=/pklxkd-59/obj29?versionId=null 2023-05-31T20:50:30.081 DEBUG:boto:auth_path=/pklxkd-59/obj29?versionId=null 2023-05-31T20:50:30.081 DEBUG:boto:Method: GET 2023-05-31T20:50:30.082 DEBUG:boto:Path: /pklxkd-59/obj29?versionId=null 2023-05-31T20:50:30.082 DEBUG:boto:Data: 2023-05-31T20:50:30.082 DEBUG:boto:Headers: {} 2023-05-31T20:50:30.083 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:50:30.083 DEBUG:boto:Port: 8001 2023-05-31T20:50:30.083 DEBUG:boto:Params: {} 2023-05-31T20:50:30.084 DEBUG:boto:Token: None 2023-05-31T20:50:30.084 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:50:30 GMT /pklxkd-59/obj29?versionId=null 2023-05-31T20:50:30.085 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:M3xDYH3LpxNMn+52oozoUpzMXcU= 2023-05-31T20:50:30.085 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:50:30 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:M3xDYH3LpxNMn+52oozoUpzMXcU=', 'Content-Length': '0'} 2023-05-31T20:50:30.087 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:48:12 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006b652d23d159ed29-006477b316-1c17-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:50:30 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:50:30.088 DEBUG:boto:path=/pklxkd-59/obj29 2023-05-31T20:50:30.088 DEBUG:boto:auth_path=/pklxkd-59/obj29 2023-05-31T20:50:30.089 DEBUG:boto:path=/pklxkd-59/obj29?versionId=null 2023-05-31T20:50:30.089 DEBUG:boto:auth_path=/pklxkd-59/obj29?versionId=null 2023-05-31T20:50:30.089 DEBUG:boto:Method: GET 2023-05-31T20:50:30.090 DEBUG:boto:Path: /pklxkd-59/obj29?versionId=null 2023-05-31T20:50:30.090 DEBUG:boto:Data: 2023-05-31T20:50:30.091 DEBUG:boto:Headers: {} 2023-05-31T20:50:30.091 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:50:30.091 DEBUG:boto:Port: 8000 2023-05-31T20:50:30.092 DEBUG:boto:Params: {} 2023-05-31T20:50:30.092 DEBUG:boto:Token: None 2023-05-31T20:50:30.092 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:50:30 GMT /pklxkd-59/obj29?versionId=null 2023-05-31T20:50:30.093 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:M3xDYH3LpxNMn+52oozoUpzMXcU= 2023-05-31T20:50:30.093 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:50:30 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:M3xDYH3LpxNMn+52oozoUpzMXcU=', 'Content-Length': '0'} 2023-05-31T20:50:30.095 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:48:12 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e49045b1656174be-006477b316-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:50:30 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:50:30.136 DEBUG:boto:path=/pklxkd-59/ 2023-05-31T20:50:30.137 DEBUG:boto:auth_path=/pklxkd-59/ 2023-05-31T20:50:30.137 DEBUG:boto:path=/pklxkd-59/?prefix=obj29 2023-05-31T20:50:30.138 DEBUG:boto:auth_path=/pklxkd-59/?prefix=obj29 2023-05-31T20:50:30.138 DEBUG:boto:Method: GET 2023-05-31T20:50:30.138 DEBUG:boto:Path: /pklxkd-59/?prefix=obj29 2023-05-31T20:50:30.139 DEBUG:boto:Data: 2023-05-31T20:50:30.139 DEBUG:boto:Headers: {} 2023-05-31T20:50:30.139 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:50:30.140 DEBUG:boto:Port: 8001 2023-05-31T20:50:30.140 DEBUG:boto:Params: {} 2023-05-31T20:50:30.141 DEBUG:boto:Token: None 2023-05-31T20:50:30.141 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:50:30 GMT /pklxkd-59/ 2023-05-31T20:50:30.142 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:3Yz4bfuBElJRjudql5UcebaTIW4= 2023-05-31T20:50:30.142 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:50:30 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:3Yz4bfuBElJRjudql5UcebaTIW4=', 'Content-Length': '0'} 2023-05-31T20:50:30.145 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000005db498b100581ce0-006477b316-1c17-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:50:30 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:50:30.145 DEBUG:boto:b'pklxkd-59obj291000falseobj292023-05-31T20:48:12.956Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:50:30.146 DEBUG:boto:path=/pklxkd-59/ 2023-05-31T20:50:30.147 DEBUG:boto:auth_path=/pklxkd-59/ 2023-05-31T20:50:30.147 DEBUG:boto:path=/pklxkd-59/?prefix=obj29 2023-05-31T20:50:30.147 DEBUG:boto:auth_path=/pklxkd-59/?prefix=obj29 2023-05-31T20:50:30.148 DEBUG:boto:Method: GET 2023-05-31T20:50:30.148 DEBUG:boto:Path: /pklxkd-59/?prefix=obj29 2023-05-31T20:50:30.149 DEBUG:boto:Data: 2023-05-31T20:50:30.149 DEBUG:boto:Headers: {} 2023-05-31T20:50:30.149 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:50:30.150 DEBUG:boto:Port: 8000 2023-05-31T20:50:30.150 DEBUG:boto:Params: {} 2023-05-31T20:50:30.150 DEBUG:boto:Token: None 2023-05-31T20:50:30.151 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:50:30 GMT /pklxkd-59/ 2023-05-31T20:50:30.151 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:3Yz4bfuBElJRjudql5UcebaTIW4= 2023-05-31T20:50:30.152 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:50:30 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:3Yz4bfuBElJRjudql5UcebaTIW4=', 'Content-Length': '0'} 2023-05-31T20:50:30.155 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000064cdfdca8c6babad-006477b316-1630-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:50:30 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:50:30.155 DEBUG:boto:b'pklxkd-59obj291000falseobj292023-05-31T20:48:12.956Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:50:30.156 DEBUG:boto:path=/pklxkd-59/obj29 2023-05-31T20:50:30.156 DEBUG:boto:auth_path=/pklxkd-59/obj29 2023-05-31T20:50:30.157 DEBUG:boto:Method: HEAD 2023-05-31T20:50:30.157 DEBUG:boto:Path: /pklxkd-59/obj29 2023-05-31T20:50:30.158 DEBUG:boto:Data: 2023-05-31T20:50:30.158 DEBUG:boto:Headers: {} 2023-05-31T20:50:30.158 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:50:30.159 DEBUG:boto:Port: 8001 2023-05-31T20:50:30.159 DEBUG:boto:Params: {} 2023-05-31T20:50:30.159 DEBUG:boto:Token: None 2023-05-31T20:50:30.160 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:50:30 GMT /pklxkd-59/obj29 2023-05-31T20:50:30.160 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:bVX9Z7YYHj/kln/gJGgoRQe0x2A= 2023-05-31T20:50:30.161 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:50:30 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:bVX9Z7YYHj/kln/gJGgoRQe0x2A=', 'Content-Length': '0'} 2023-05-31T20:50:30.162 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:48:12 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000fbbcae9451e448c7-006477b316-1c17-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:50:30 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:50:30.163 DEBUG:boto:path=/pklxkd-59/obj29 2023-05-31T20:50:30.163 DEBUG:boto:auth_path=/pklxkd-59/obj29 2023-05-31T20:50:30.163 DEBUG:boto:Method: HEAD 2023-05-31T20:50:30.164 DEBUG:boto:Path: /pklxkd-59/obj29 2023-05-31T20:50:30.164 DEBUG:boto:Data: 2023-05-31T20:50:30.165 DEBUG:boto:Headers: {} 2023-05-31T20:50:30.165 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:50:30.165 DEBUG:boto:Port: 8000 2023-05-31T20:50:30.166 DEBUG:boto:Params: {} 2023-05-31T20:50:30.166 DEBUG:boto:Token: None 2023-05-31T20:50:30.166 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:50:30 GMT /pklxkd-59/obj29 2023-05-31T20:50:30.167 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:bVX9Z7YYHj/kln/gJGgoRQe0x2A= 2023-05-31T20:50:30.167 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:50:30 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:bVX9Z7YYHj/kln/gJGgoRQe0x2A=', 'Content-Length': '0'} 2023-05-31T20:50:30.169 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:48:12 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000074b2a50926190f8b-006477b316-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:50:30 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:50:30.169 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj29 2023-05-31T20:50:30.170 DEBUG:boto:path=/pklxkd-59/obj29 2023-05-31T20:50:30.170 DEBUG:boto:auth_path=/pklxkd-59/obj29 2023-05-31T20:50:30.171 DEBUG:boto:Method: GET 2023-05-31T20:50:30.171 DEBUG:boto:Path: /pklxkd-59/obj29 2023-05-31T20:50:30.171 DEBUG:boto:Data: 2023-05-31T20:50:30.172 DEBUG:boto:Headers: {} 2023-05-31T20:50:30.172 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:50:30.173 DEBUG:boto:Port: 8001 2023-05-31T20:50:30.173 DEBUG:boto:Params: {} 2023-05-31T20:50:30.173 DEBUG:boto:Token: None 2023-05-31T20:50:30.174 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:50:30 GMT /pklxkd-59/obj29 2023-05-31T20:50:30.174 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:T7DjVwGSwSlhEimJU1jSCkv5qKI= 2023-05-31T20:50:30.175 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:50:30 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:T7DjVwGSwSlhEimJU1jSCkv5qKI=', 'Content-Length': '0'} 2023-05-31T20:50:30.176 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:48:12 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d64170b96b9cee61-006477b316-1c17-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:50:30 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:50:30.177 DEBUG:boto:path=/pklxkd-59/obj29 2023-05-31T20:50:30.178 DEBUG:boto:auth_path=/pklxkd-59/obj29 2023-05-31T20:50:30.178 DEBUG:boto:Method: GET 2023-05-31T20:50:30.178 DEBUG:boto:Path: /pklxkd-59/obj29 2023-05-31T20:50:30.179 DEBUG:boto:Data: 2023-05-31T20:50:30.179 DEBUG:boto:Headers: {} 2023-05-31T20:50:30.179 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:50:30.180 DEBUG:boto:Port: 8000 2023-05-31T20:50:30.180 DEBUG:boto:Params: {} 2023-05-31T20:50:30.181 DEBUG:boto:Token: None 2023-05-31T20:50:30.181 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:50:30 GMT /pklxkd-59/obj29 2023-05-31T20:50:30.181 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:T7DjVwGSwSlhEimJU1jSCkv5qKI= 2023-05-31T20:50:30.182 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:50:30 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:T7DjVwGSwSlhEimJU1jSCkv5qKI=', 'Content-Length': '0'} 2023-05-31T20:50:30.184 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:48:12 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e3eb20bdb3bc2b32-006477b316-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:50:30 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:50:30.184 DEBUG:boto:path=/pklxkd-59/ 2023-05-31T20:50:30.185 DEBUG:boto:auth_path=/pklxkd-59/ 2023-05-31T20:50:30.185 DEBUG:boto:path=/pklxkd-59/?prefix=obj29 2023-05-31T20:50:30.186 DEBUG:boto:auth_path=/pklxkd-59/?prefix=obj29 2023-05-31T20:50:30.186 DEBUG:boto:Method: GET 2023-05-31T20:50:30.186 DEBUG:boto:Path: /pklxkd-59/?prefix=obj29 2023-05-31T20:50:30.187 DEBUG:boto:Data: 2023-05-31T20:50:30.187 DEBUG:boto:Headers: {} 2023-05-31T20:50:30.187 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:50:30.188 DEBUG:boto:Port: 8001 2023-05-31T20:50:30.188 DEBUG:boto:Params: {} 2023-05-31T20:50:30.189 DEBUG:boto:Token: None 2023-05-31T20:50:30.189 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:50:30 GMT /pklxkd-59/ 2023-05-31T20:50:30.190 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:3Yz4bfuBElJRjudql5UcebaTIW4= 2023-05-31T20:50:30.190 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:50:30 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:3Yz4bfuBElJRjudql5UcebaTIW4=', 'Content-Length': '0'} 2023-05-31T20:50:30.192 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000231f65cdb82c3d39-006477b316-1c17-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:50:30 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:50:30.236 DEBUG:boto:b'pklxkd-59obj291000falseobj292023-05-31T20:48:12.956Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:50:30.237 DEBUG:boto:path=/pklxkd-59/ 2023-05-31T20:50:30.237 DEBUG:boto:auth_path=/pklxkd-59/ 2023-05-31T20:50:30.238 DEBUG:boto:path=/pklxkd-59/?prefix=obj29 2023-05-31T20:50:30.238 DEBUG:boto:auth_path=/pklxkd-59/?prefix=obj29 2023-05-31T20:50:30.239 DEBUG:boto:Method: GET 2023-05-31T20:50:30.239 DEBUG:boto:Path: /pklxkd-59/?prefix=obj29 2023-05-31T20:50:30.239 DEBUG:boto:Data: 2023-05-31T20:50:30.240 DEBUG:boto:Headers: {} 2023-05-31T20:50:30.240 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:50:30.240 DEBUG:boto:Port: 8000 2023-05-31T20:50:30.241 DEBUG:boto:Params: {} 2023-05-31T20:50:30.241 DEBUG:boto:Token: None 2023-05-31T20:50:30.242 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:50:30 GMT /pklxkd-59/ 2023-05-31T20:50:30.242 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:3Yz4bfuBElJRjudql5UcebaTIW4= 2023-05-31T20:50:30.243 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:50:30 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:3Yz4bfuBElJRjudql5UcebaTIW4=', 'Content-Length': '0'} 2023-05-31T20:50:30.246 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000010f7be242de9b575-006477b316-1630-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:50:30 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:50:30.247 DEBUG:boto:b'pklxkd-59obj291000falseobj292023-05-31T20:48:12.956Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:50:30.247 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj33 2023-05-31T20:50:30.248 DEBUG:boto:path=/pklxkd-59/obj33 2023-05-31T20:50:30.248 DEBUG:boto:auth_path=/pklxkd-59/obj33 2023-05-31T20:50:30.248 DEBUG:boto:path=/pklxkd-59/obj33?versionId=null 2023-05-31T20:50:30.249 DEBUG:boto:auth_path=/pklxkd-59/obj33?versionId=null 2023-05-31T20:50:30.249 DEBUG:boto:Method: GET 2023-05-31T20:50:30.249 DEBUG:boto:Path: /pklxkd-59/obj33?versionId=null 2023-05-31T20:50:30.250 DEBUG:boto:Data: 2023-05-31T20:50:30.250 DEBUG:boto:Headers: {} 2023-05-31T20:50:30.250 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:50:30.251 DEBUG:boto:Port: 8001 2023-05-31T20:50:30.251 DEBUG:boto:Params: {} 2023-05-31T20:50:30.252 DEBUG:boto:Token: None 2023-05-31T20:50:30.252 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:50:30 GMT /pklxkd-59/obj33?versionId=null 2023-05-31T20:50:30.252 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:+Qrh2b8fqHD9k276ymNzBLlzjH0= 2023-05-31T20:50:30.253 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:50:30 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:+Qrh2b8fqHD9k276ymNzBLlzjH0=', 'Content-Length': '0'} 2023-05-31T20:50:30.255 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:39:26 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000044d03446eef0beb4-006477b316-1c17-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:50:30 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:50:30.255 DEBUG:boto:path=/pklxkd-59/obj33 2023-05-31T20:50:30.256 DEBUG:boto:auth_path=/pklxkd-59/obj33 2023-05-31T20:50:30.256 DEBUG:boto:path=/pklxkd-59/obj33?versionId=null 2023-05-31T20:50:30.257 DEBUG:boto:auth_path=/pklxkd-59/obj33?versionId=null 2023-05-31T20:50:30.257 DEBUG:boto:Method: GET 2023-05-31T20:50:30.257 DEBUG:boto:Path: /pklxkd-59/obj33?versionId=null 2023-05-31T20:50:30.258 DEBUG:boto:Data: 2023-05-31T20:50:30.258 DEBUG:boto:Headers: {} 2023-05-31T20:50:30.258 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:50:30.259 DEBUG:boto:Port: 8000 2023-05-31T20:50:30.259 DEBUG:boto:Params: {} 2023-05-31T20:50:30.259 DEBUG:boto:Token: None 2023-05-31T20:50:30.260 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:50:30 GMT /pklxkd-59/obj33?versionId=null 2023-05-31T20:50:30.260 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:+Qrh2b8fqHD9k276ymNzBLlzjH0= 2023-05-31T20:50:30.261 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:50:30 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:+Qrh2b8fqHD9k276ymNzBLlzjH0=', 'Content-Length': '0'} 2023-05-31T20:50:30.263 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:39:26 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000025a459c0fb705eb2-006477b316-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:50:30 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:50:30.304 DEBUG:boto:path=/pklxkd-59/obj33 2023-05-31T20:50:30.305 DEBUG:boto:auth_path=/pklxkd-59/obj33 2023-05-31T20:50:30.305 DEBUG:boto:path=/pklxkd-59/obj33?versionId=null 2023-05-31T20:50:30.305 DEBUG:boto:auth_path=/pklxkd-59/obj33?versionId=null 2023-05-31T20:50:30.306 DEBUG:boto:Method: HEAD 2023-05-31T20:50:30.306 DEBUG:boto:Path: /pklxkd-59/obj33?versionId=null 2023-05-31T20:50:30.306 DEBUG:boto:Data: 2023-05-31T20:50:30.307 DEBUG:boto:Headers: {} 2023-05-31T20:50:30.307 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:50:30.307 DEBUG:boto:Port: 8001 2023-05-31T20:50:30.308 DEBUG:boto:Params: {} 2023-05-31T20:50:30.308 DEBUG:boto:Token: None 2023-05-31T20:50:30.309 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:50:30 GMT /pklxkd-59/obj33?versionId=null 2023-05-31T20:50:30.309 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:G3l8KkaUBRJb0BfNBeqLVZWgCzU= 2023-05-31T20:50:30.309 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:50:30 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:G3l8KkaUBRJb0BfNBeqLVZWgCzU=', 'Content-Length': '0'} 2023-05-31T20:50:30.311 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:39:26 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f67e626feceb9359-006477b316-1c17-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:50:30 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:50:30.312 DEBUG:boto:path=/pklxkd-59/obj33 2023-05-31T20:50:30.312 DEBUG:boto:auth_path=/pklxkd-59/obj33 2023-05-31T20:50:30.312 DEBUG:boto:path=/pklxkd-59/obj33?versionId=null 2023-05-31T20:50:30.313 DEBUG:boto:auth_path=/pklxkd-59/obj33?versionId=null 2023-05-31T20:50:30.313 DEBUG:boto:Method: HEAD 2023-05-31T20:50:30.313 DEBUG:boto:Path: /pklxkd-59/obj33?versionId=null 2023-05-31T20:50:30.314 DEBUG:boto:Data: 2023-05-31T20:50:30.314 DEBUG:boto:Headers: {} 2023-05-31T20:50:30.314 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:50:30.315 DEBUG:boto:Port: 8000 2023-05-31T20:50:30.315 DEBUG:boto:Params: {} 2023-05-31T20:50:30.316 DEBUG:boto:Token: None 2023-05-31T20:50:30.316 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:50:30 GMT /pklxkd-59/obj33?versionId=null 2023-05-31T20:50:30.317 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:G3l8KkaUBRJb0BfNBeqLVZWgCzU= 2023-05-31T20:50:30.317 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:50:30 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:G3l8KkaUBRJb0BfNBeqLVZWgCzU=', 'Content-Length': '0'} 2023-05-31T20:50:30.319 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:39:26 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000053b5a46e25a72e41-006477b316-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:50:30 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:50:30.319 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj33 2023-05-31T20:50:30.320 DEBUG:boto:path=/pklxkd-59/obj33 2023-05-31T20:50:30.320 DEBUG:boto:auth_path=/pklxkd-59/obj33 2023-05-31T20:50:30.320 DEBUG:boto:path=/pklxkd-59/obj33?versionId=null 2023-05-31T20:50:30.321 DEBUG:boto:auth_path=/pklxkd-59/obj33?versionId=null 2023-05-31T20:50:30.321 DEBUG:boto:Method: GET 2023-05-31T20:50:30.321 DEBUG:boto:Path: /pklxkd-59/obj33?versionId=null 2023-05-31T20:50:30.322 DEBUG:boto:Data: 2023-05-31T20:50:30.322 DEBUG:boto:Headers: {} 2023-05-31T20:50:30.322 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:50:30.323 DEBUG:boto:Port: 8001 2023-05-31T20:50:30.323 DEBUG:boto:Params: {} 2023-05-31T20:50:30.324 DEBUG:boto:Token: None 2023-05-31T20:50:30.324 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:50:30 GMT /pklxkd-59/obj33?versionId=null 2023-05-31T20:50:30.324 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:+Qrh2b8fqHD9k276ymNzBLlzjH0= 2023-05-31T20:50:30.325 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:50:30 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:+Qrh2b8fqHD9k276ymNzBLlzjH0=', 'Content-Length': '0'} 2023-05-31T20:50:30.327 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:39:26 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001be7f4c8434fb9bb-006477b316-1c17-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:50:30 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:50:30.327 DEBUG:boto:path=/pklxkd-59/obj33 2023-05-31T20:50:30.328 DEBUG:boto:auth_path=/pklxkd-59/obj33 2023-05-31T20:50:30.328 DEBUG:boto:path=/pklxkd-59/obj33?versionId=null 2023-05-31T20:50:30.328 DEBUG:boto:auth_path=/pklxkd-59/obj33?versionId=null 2023-05-31T20:50:30.329 DEBUG:boto:Method: GET 2023-05-31T20:50:30.329 DEBUG:boto:Path: /pklxkd-59/obj33?versionId=null 2023-05-31T20:50:30.330 DEBUG:boto:Data: 2023-05-31T20:50:30.330 DEBUG:boto:Headers: {} 2023-05-31T20:50:30.331 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:50:30.331 DEBUG:boto:Port: 8000 2023-05-31T20:50:30.331 DEBUG:boto:Params: {} 2023-05-31T20:50:30.332 DEBUG:boto:Token: None 2023-05-31T20:50:30.332 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:50:30 GMT /pklxkd-59/obj33?versionId=null 2023-05-31T20:50:30.333 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:+Qrh2b8fqHD9k276ymNzBLlzjH0= 2023-05-31T20:50:30.333 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:50:30 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:+Qrh2b8fqHD9k276ymNzBLlzjH0=', 'Content-Length': '0'} 2023-05-31T20:50:30.336 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:39:26 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000020358994b4e2af0-006477b316-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:50:30 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:50:30.336 DEBUG:boto:path=/pklxkd-59/ 2023-05-31T20:50:30.337 DEBUG:boto:auth_path=/pklxkd-59/ 2023-05-31T20:50:30.337 DEBUG:boto:path=/pklxkd-59/?prefix=obj33 2023-05-31T20:50:30.338 DEBUG:boto:auth_path=/pklxkd-59/?prefix=obj33 2023-05-31T20:50:30.338 DEBUG:boto:Method: GET 2023-05-31T20:50:30.338 DEBUG:boto:Path: /pklxkd-59/?prefix=obj33 2023-05-31T20:50:30.339 DEBUG:boto:Data: 2023-05-31T20:50:30.339 DEBUG:boto:Headers: {} 2023-05-31T20:50:30.339 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:50:30.340 DEBUG:boto:Port: 8001 2023-05-31T20:50:30.340 DEBUG:boto:Params: {} 2023-05-31T20:50:30.341 DEBUG:boto:Token: None 2023-05-31T20:50:30.341 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:50:30 GMT /pklxkd-59/ 2023-05-31T20:50:30.341 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:3Yz4bfuBElJRjudql5UcebaTIW4= 2023-05-31T20:50:30.342 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:50:30 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:3Yz4bfuBElJRjudql5UcebaTIW4=', 'Content-Length': '0'} 2023-05-31T20:50:30.344 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e231093479e640d3-006477b316-1c17-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:50:30 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:50:30.345 DEBUG:boto:b'pklxkd-59obj331000falseobj332023-05-31T20:39:26.534Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:50:30.346 DEBUG:boto:path=/pklxkd-59/ 2023-05-31T20:50:30.346 DEBUG:boto:auth_path=/pklxkd-59/ 2023-05-31T20:50:30.347 DEBUG:boto:path=/pklxkd-59/?prefix=obj33 2023-05-31T20:50:30.347 DEBUG:boto:auth_path=/pklxkd-59/?prefix=obj33 2023-05-31T20:50:30.347 DEBUG:boto:Method: GET 2023-05-31T20:50:30.348 DEBUG:boto:Path: /pklxkd-59/?prefix=obj33 2023-05-31T20:50:30.348 DEBUG:boto:Data: 2023-05-31T20:50:30.348 DEBUG:boto:Headers: {} 2023-05-31T20:50:30.349 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:50:30.349 DEBUG:boto:Port: 8000 2023-05-31T20:50:30.349 DEBUG:boto:Params: {} 2023-05-31T20:50:30.350 DEBUG:boto:Token: None 2023-05-31T20:50:30.350 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:50:30 GMT /pklxkd-59/ 2023-05-31T20:50:30.351 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:3Yz4bfuBElJRjudql5UcebaTIW4= 2023-05-31T20:50:30.351 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:50:30 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:3Yz4bfuBElJRjudql5UcebaTIW4=', 'Content-Length': '0'} 2023-05-31T20:50:30.354 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d96e23f6b64139ea-006477b316-1630-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:50:30 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:50:30.355 DEBUG:boto:b'pklxkd-59obj331000falseobj332023-05-31T20:39:26.534Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:50:30.356 DEBUG:boto:path=/pklxkd-59/obj33 2023-05-31T20:50:30.356 DEBUG:boto:auth_path=/pklxkd-59/obj33 2023-05-31T20:50:30.356 DEBUG:boto:Method: HEAD 2023-05-31T20:50:30.357 DEBUG:boto:Path: /pklxkd-59/obj33 2023-05-31T20:50:30.357 DEBUG:boto:Data: 2023-05-31T20:50:30.357 DEBUG:boto:Headers: {} 2023-05-31T20:50:30.358 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:50:30.358 DEBUG:boto:Port: 8001 2023-05-31T20:50:30.358 DEBUG:boto:Params: {} 2023-05-31T20:50:30.359 DEBUG:boto:Token: None 2023-05-31T20:50:30.359 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:50:30 GMT /pklxkd-59/obj33 2023-05-31T20:50:30.359 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:lid9wG5ZU0lTPVxe/uBd6w0NxGY= 2023-05-31T20:50:30.360 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:50:30 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:lid9wG5ZU0lTPVxe/uBd6w0NxGY=', 'Content-Length': '0'} 2023-05-31T20:50:30.362 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:39:26 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ba596bc920041115-006477b316-1c17-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:50:30 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:50:30.362 DEBUG:boto:path=/pklxkd-59/obj33 2023-05-31T20:50:30.363 DEBUG:boto:auth_path=/pklxkd-59/obj33 2023-05-31T20:50:30.363 DEBUG:boto:Method: HEAD 2023-05-31T20:50:30.364 DEBUG:boto:Path: /pklxkd-59/obj33 2023-05-31T20:50:30.364 DEBUG:boto:Data: 2023-05-31T20:50:30.364 DEBUG:boto:Headers: {} 2023-05-31T20:50:30.365 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:50:30.365 DEBUG:boto:Port: 8000 2023-05-31T20:50:30.366 DEBUG:boto:Params: {} 2023-05-31T20:50:30.366 DEBUG:boto:Token: None 2023-05-31T20:50:30.367 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:50:30 GMT /pklxkd-59/obj33 2023-05-31T20:50:30.367 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:lid9wG5ZU0lTPVxe/uBd6w0NxGY= 2023-05-31T20:50:30.368 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:50:30 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:lid9wG5ZU0lTPVxe/uBd6w0NxGY=', 'Content-Length': '0'} 2023-05-31T20:50:30.370 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:39:26 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c00a0c04a7111c45-006477b316-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:50:30 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:50:30.370 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj33 2023-05-31T20:50:30.371 DEBUG:boto:path=/pklxkd-59/obj33 2023-05-31T20:50:30.371 DEBUG:boto:auth_path=/pklxkd-59/obj33 2023-05-31T20:50:30.371 DEBUG:boto:Method: GET 2023-05-31T20:50:30.372 DEBUG:boto:Path: /pklxkd-59/obj33 2023-05-31T20:50:30.372 DEBUG:boto:Data: 2023-05-31T20:50:30.372 DEBUG:boto:Headers: {} 2023-05-31T20:50:30.373 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:50:30.373 DEBUG:boto:Port: 8001 2023-05-31T20:50:30.374 DEBUG:boto:Params: {} 2023-05-31T20:50:30.374 DEBUG:boto:Token: None 2023-05-31T20:50:30.374 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:50:30 GMT /pklxkd-59/obj33 2023-05-31T20:50:30.375 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:Owd/e9vQJttH2s+MfFwYSkGwjh4= 2023-05-31T20:50:30.375 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:50:30 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:Owd/e9vQJttH2s+MfFwYSkGwjh4=', 'Content-Length': '0'} 2023-05-31T20:50:30.377 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:39:26 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000765b92021230fc39-006477b316-1c17-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:50:30 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:50:30.377 DEBUG:boto:path=/pklxkd-59/obj33 2023-05-31T20:50:30.378 DEBUG:boto:auth_path=/pklxkd-59/obj33 2023-05-31T20:50:30.378 DEBUG:boto:Method: GET 2023-05-31T20:50:30.379 DEBUG:boto:Path: /pklxkd-59/obj33 2023-05-31T20:50:30.379 DEBUG:boto:Data: 2023-05-31T20:50:30.379 DEBUG:boto:Headers: {} 2023-05-31T20:50:30.380 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:50:30.380 DEBUG:boto:Port: 8000 2023-05-31T20:50:30.380 DEBUG:boto:Params: {} 2023-05-31T20:50:30.381 DEBUG:boto:Token: None 2023-05-31T20:50:30.381 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:50:30 GMT /pklxkd-59/obj33 2023-05-31T20:50:30.382 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:Owd/e9vQJttH2s+MfFwYSkGwjh4= 2023-05-31T20:50:30.382 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:50:30 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:Owd/e9vQJttH2s+MfFwYSkGwjh4=', 'Content-Length': '0'} 2023-05-31T20:50:30.384 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:39:26 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000053b1f14e3fec48a3-006477b316-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:50:30 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:50:30.385 DEBUG:boto:path=/pklxkd-59/ 2023-05-31T20:50:30.385 DEBUG:boto:auth_path=/pklxkd-59/ 2023-05-31T20:50:30.386 DEBUG:boto:path=/pklxkd-59/?prefix=obj33 2023-05-31T20:50:30.386 DEBUG:boto:auth_path=/pklxkd-59/?prefix=obj33 2023-05-31T20:50:30.386 DEBUG:boto:Method: GET 2023-05-31T20:50:30.387 DEBUG:boto:Path: /pklxkd-59/?prefix=obj33 2023-05-31T20:50:30.387 DEBUG:boto:Data: 2023-05-31T20:50:30.387 DEBUG:boto:Headers: {} 2023-05-31T20:50:30.388 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:50:30.388 DEBUG:boto:Port: 8001 2023-05-31T20:50:30.389 DEBUG:boto:Params: {} 2023-05-31T20:50:30.389 DEBUG:boto:Token: None 2023-05-31T20:50:30.389 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:50:30 GMT /pklxkd-59/ 2023-05-31T20:50:30.390 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:3Yz4bfuBElJRjudql5UcebaTIW4= 2023-05-31T20:50:30.390 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:50:30 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:3Yz4bfuBElJRjudql5UcebaTIW4=', 'Content-Length': '0'} 2023-05-31T20:50:30.393 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ee38611235fc0bb0-006477b316-1c17-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:50:30 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:50:30.440 DEBUG:boto:b'pklxkd-59obj331000falseobj332023-05-31T20:39:26.534Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:50:30.441 DEBUG:boto:path=/pklxkd-59/ 2023-05-31T20:50:30.442 DEBUG:boto:auth_path=/pklxkd-59/ 2023-05-31T20:50:30.442 DEBUG:boto:path=/pklxkd-59/?prefix=obj33 2023-05-31T20:50:30.442 DEBUG:boto:auth_path=/pklxkd-59/?prefix=obj33 2023-05-31T20:50:30.443 DEBUG:boto:Method: GET 2023-05-31T20:50:30.443 DEBUG:boto:Path: /pklxkd-59/?prefix=obj33 2023-05-31T20:50:30.444 DEBUG:boto:Data: 2023-05-31T20:50:30.444 DEBUG:boto:Headers: {} 2023-05-31T20:50:30.444 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:50:30.445 DEBUG:boto:Port: 8000 2023-05-31T20:50:30.445 DEBUG:boto:Params: {} 2023-05-31T20:50:30.445 DEBUG:boto:Token: None 2023-05-31T20:50:30.446 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:50:30 GMT /pklxkd-59/ 2023-05-31T20:50:30.446 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:3Yz4bfuBElJRjudql5UcebaTIW4= 2023-05-31T20:50:30.447 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:50:30 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:3Yz4bfuBElJRjudql5UcebaTIW4=', 'Content-Length': '0'} 2023-05-31T20:50:30.450 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000fc6e4107632977bb-006477b316-1630-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:50:30 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:50:30.451 DEBUG:boto:b'pklxkd-59obj331000falseobj332023-05-31T20:39:26.534Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:50:30.452 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj34 2023-05-31T20:50:30.452 DEBUG:boto:path=/pklxkd-59/obj34 2023-05-31T20:50:30.453 DEBUG:boto:auth_path=/pklxkd-59/obj34 2023-05-31T20:50:30.453 DEBUG:boto:path=/pklxkd-59/obj34?versionId=null 2023-05-31T20:50:30.453 DEBUG:boto:auth_path=/pklxkd-59/obj34?versionId=null 2023-05-31T20:50:30.454 DEBUG:boto:Method: GET 2023-05-31T20:50:30.454 DEBUG:boto:Path: /pklxkd-59/obj34?versionId=null 2023-05-31T20:50:30.454 DEBUG:boto:Data: 2023-05-31T20:50:30.455 DEBUG:boto:Headers: {} 2023-05-31T20:50:30.455 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:50:30.456 DEBUG:boto:Port: 8001 2023-05-31T20:50:30.456 DEBUG:boto:Params: {} 2023-05-31T20:50:30.456 DEBUG:boto:Token: None 2023-05-31T20:50:30.457 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:50:30 GMT /pklxkd-59/obj34?versionId=null 2023-05-31T20:50:30.457 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:FBOFbgS/sGUlIY0npmTqkBB8/sc= 2023-05-31T20:50:30.458 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:50:30 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:FBOFbgS/sGUlIY0npmTqkBB8/sc=', 'Content-Length': '0'} 2023-05-31T20:50:30.459 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:43:07 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d30f645c62d1c138-006477b316-1c17-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:50:30 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:50:30.460 DEBUG:boto:path=/pklxkd-59/obj34 2023-05-31T20:50:30.460 DEBUG:boto:auth_path=/pklxkd-59/obj34 2023-05-31T20:50:30.461 DEBUG:boto:path=/pklxkd-59/obj34?versionId=null 2023-05-31T20:50:30.461 DEBUG:boto:auth_path=/pklxkd-59/obj34?versionId=null 2023-05-31T20:50:30.461 DEBUG:boto:Method: GET 2023-05-31T20:50:30.462 DEBUG:boto:Path: /pklxkd-59/obj34?versionId=null 2023-05-31T20:50:30.462 DEBUG:boto:Data: 2023-05-31T20:50:30.463 DEBUG:boto:Headers: {} 2023-05-31T20:50:30.463 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:50:30.463 DEBUG:boto:Port: 8000 2023-05-31T20:50:30.464 DEBUG:boto:Params: {} 2023-05-31T20:50:30.464 DEBUG:boto:Token: None 2023-05-31T20:50:30.464 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:50:30 GMT /pklxkd-59/obj34?versionId=null 2023-05-31T20:50:30.465 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:FBOFbgS/sGUlIY0npmTqkBB8/sc= 2023-05-31T20:50:30.465 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:50:30 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:FBOFbgS/sGUlIY0npmTqkBB8/sc=', 'Content-Length': '0'} 2023-05-31T20:50:30.467 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:43:07 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000870bb88a88d48d99-006477b316-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:50:30 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:50:30.512 DEBUG:boto:path=/pklxkd-59/obj34 2023-05-31T20:50:30.513 DEBUG:boto:auth_path=/pklxkd-59/obj34 2023-05-31T20:50:30.513 DEBUG:boto:path=/pklxkd-59/obj34?versionId=null 2023-05-31T20:50:30.514 DEBUG:boto:auth_path=/pklxkd-59/obj34?versionId=null 2023-05-31T20:50:30.514 DEBUG:boto:Method: HEAD 2023-05-31T20:50:30.514 DEBUG:boto:Path: /pklxkd-59/obj34?versionId=null 2023-05-31T20:50:30.515 DEBUG:boto:Data: 2023-05-31T20:50:30.515 DEBUG:boto:Headers: {} 2023-05-31T20:50:30.515 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:50:30.516 DEBUG:boto:Port: 8001 2023-05-31T20:50:30.516 DEBUG:boto:Params: {} 2023-05-31T20:50:30.516 DEBUG:boto:Token: None 2023-05-31T20:50:30.517 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:50:30 GMT /pklxkd-59/obj34?versionId=null 2023-05-31T20:50:30.517 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:OmOaqM/nrrqietfcNCtlwaTyKpY= 2023-05-31T20:50:30.518 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:50:30 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:OmOaqM/nrrqietfcNCtlwaTyKpY=', 'Content-Length': '0'} 2023-05-31T20:50:30.519 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:43:07 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ddb7ba271b6179d5-006477b316-1c17-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:50:30 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:50:30.520 DEBUG:boto:path=/pklxkd-59/obj34 2023-05-31T20:50:30.521 DEBUG:boto:auth_path=/pklxkd-59/obj34 2023-05-31T20:50:30.521 DEBUG:boto:path=/pklxkd-59/obj34?versionId=null 2023-05-31T20:50:30.521 DEBUG:boto:auth_path=/pklxkd-59/obj34?versionId=null 2023-05-31T20:50:30.522 DEBUG:boto:Method: HEAD 2023-05-31T20:50:30.522 DEBUG:boto:Path: /pklxkd-59/obj34?versionId=null 2023-05-31T20:50:30.522 DEBUG:boto:Data: 2023-05-31T20:50:30.522 DEBUG:boto:Headers: {} 2023-05-31T20:50:30.523 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:50:30.523 DEBUG:boto:Port: 8000 2023-05-31T20:50:30.524 DEBUG:boto:Params: {} 2023-05-31T20:50:30.524 DEBUG:boto:Token: None 2023-05-31T20:50:30.524 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:50:30 GMT /pklxkd-59/obj34?versionId=null 2023-05-31T20:50:30.525 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:OmOaqM/nrrqietfcNCtlwaTyKpY= 2023-05-31T20:50:30.525 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:50:30 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:OmOaqM/nrrqietfcNCtlwaTyKpY=', 'Content-Length': '0'} 2023-05-31T20:50:30.527 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:43:07 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006736418c193b4894-006477b316-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:50:30 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:50:30.527 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj34 2023-05-31T20:50:30.528 DEBUG:boto:path=/pklxkd-59/obj34 2023-05-31T20:50:30.528 DEBUG:boto:auth_path=/pklxkd-59/obj34 2023-05-31T20:50:30.529 DEBUG:boto:path=/pklxkd-59/obj34?versionId=null 2023-05-31T20:50:30.529 DEBUG:boto:auth_path=/pklxkd-59/obj34?versionId=null 2023-05-31T20:50:30.529 DEBUG:boto:Method: GET 2023-05-31T20:50:30.530 DEBUG:boto:Path: /pklxkd-59/obj34?versionId=null 2023-05-31T20:50:30.530 DEBUG:boto:Data: 2023-05-31T20:50:30.530 DEBUG:boto:Headers: {} 2023-05-31T20:50:30.531 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:50:30.531 DEBUG:boto:Port: 8001 2023-05-31T20:50:30.531 DEBUG:boto:Params: {} 2023-05-31T20:50:30.532 DEBUG:boto:Token: None 2023-05-31T20:50:30.532 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:50:30 GMT /pklxkd-59/obj34?versionId=null 2023-05-31T20:50:30.533 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:FBOFbgS/sGUlIY0npmTqkBB8/sc= 2023-05-31T20:50:30.533 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:50:30 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:FBOFbgS/sGUlIY0npmTqkBB8/sc=', 'Content-Length': '0'} 2023-05-31T20:50:30.535 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:43:07 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002314b07bba4add7a-006477b316-1c17-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:50:30 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:50:30.536 DEBUG:boto:path=/pklxkd-59/obj34 2023-05-31T20:50:30.536 DEBUG:boto:auth_path=/pklxkd-59/obj34 2023-05-31T20:50:30.536 DEBUG:boto:path=/pklxkd-59/obj34?versionId=null 2023-05-31T20:50:30.537 DEBUG:boto:auth_path=/pklxkd-59/obj34?versionId=null 2023-05-31T20:50:30.537 DEBUG:boto:Method: GET 2023-05-31T20:50:30.537 DEBUG:boto:Path: /pklxkd-59/obj34?versionId=null 2023-05-31T20:50:30.538 DEBUG:boto:Data: 2023-05-31T20:50:30.538 DEBUG:boto:Headers: {} 2023-05-31T20:50:30.538 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:50:30.539 DEBUG:boto:Port: 8000 2023-05-31T20:50:30.539 DEBUG:boto:Params: {} 2023-05-31T20:50:30.540 DEBUG:boto:Token: None 2023-05-31T20:50:30.540 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:50:30 GMT /pklxkd-59/obj34?versionId=null 2023-05-31T20:50:30.540 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:FBOFbgS/sGUlIY0npmTqkBB8/sc= 2023-05-31T20:50:30.541 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:50:30 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:FBOFbgS/sGUlIY0npmTqkBB8/sc=', 'Content-Length': '0'} 2023-05-31T20:50:30.543 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:43:07 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000027af91d6506155dc-006477b316-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:50:30 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:50:30.543 DEBUG:boto:path=/pklxkd-59/ 2023-05-31T20:50:30.544 DEBUG:boto:auth_path=/pklxkd-59/ 2023-05-31T20:50:30.544 DEBUG:boto:path=/pklxkd-59/?prefix=obj34 2023-05-31T20:50:30.545 DEBUG:boto:auth_path=/pklxkd-59/?prefix=obj34 2023-05-31T20:50:30.545 DEBUG:boto:Method: GET 2023-05-31T20:50:30.545 DEBUG:boto:Path: /pklxkd-59/?prefix=obj34 2023-05-31T20:50:30.546 DEBUG:boto:Data: 2023-05-31T20:50:30.546 DEBUG:boto:Headers: {} 2023-05-31T20:50:30.546 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:50:30.547 DEBUG:boto:Port: 8001 2023-05-31T20:50:30.547 DEBUG:boto:Params: {} 2023-05-31T20:50:30.547 DEBUG:boto:Token: None 2023-05-31T20:50:30.548 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:50:30 GMT /pklxkd-59/ 2023-05-31T20:50:30.548 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:3Yz4bfuBElJRjudql5UcebaTIW4= 2023-05-31T20:50:30.549 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:50:30 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:3Yz4bfuBElJRjudql5UcebaTIW4=', 'Content-Length': '0'} 2023-05-31T20:50:30.551 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000040577eefdf408852-006477b316-1c17-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:50:30 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:50:30.552 DEBUG:boto:b'pklxkd-59obj341000falseobj342023-05-31T20:43:07.994Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:50:30.553 DEBUG:boto:path=/pklxkd-59/ 2023-05-31T20:50:30.553 DEBUG:boto:auth_path=/pklxkd-59/ 2023-05-31T20:50:30.553 DEBUG:boto:path=/pklxkd-59/?prefix=obj34 2023-05-31T20:50:30.554 DEBUG:boto:auth_path=/pklxkd-59/?prefix=obj34 2023-05-31T20:50:30.554 DEBUG:boto:Method: GET 2023-05-31T20:50:30.554 DEBUG:boto:Path: /pklxkd-59/?prefix=obj34 2023-05-31T20:50:30.555 DEBUG:boto:Data: 2023-05-31T20:50:30.555 DEBUG:boto:Headers: {} 2023-05-31T20:50:30.555 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:50:30.556 DEBUG:boto:Port: 8000 2023-05-31T20:50:30.556 DEBUG:boto:Params: {} 2023-05-31T20:50:30.557 DEBUG:boto:Token: None 2023-05-31T20:50:30.557 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:50:30 GMT /pklxkd-59/ 2023-05-31T20:50:30.557 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:3Yz4bfuBElJRjudql5UcebaTIW4= 2023-05-31T20:50:30.558 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:50:30 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:3Yz4bfuBElJRjudql5UcebaTIW4=', 'Content-Length': '0'} 2023-05-31T20:50:30.561 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000da90a32b3151ddc1-006477b316-1630-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:50:30 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:50:30.562 DEBUG:boto:b'pklxkd-59obj341000falseobj342023-05-31T20:43:07.994Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:50:30.563 DEBUG:boto:path=/pklxkd-59/obj34 2023-05-31T20:50:30.563 DEBUG:boto:auth_path=/pklxkd-59/obj34 2023-05-31T20:50:30.563 DEBUG:boto:Method: HEAD 2023-05-31T20:50:30.564 DEBUG:boto:Path: /pklxkd-59/obj34 2023-05-31T20:50:30.564 DEBUG:boto:Data: 2023-05-31T20:50:30.564 DEBUG:boto:Headers: {} 2023-05-31T20:50:30.565 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:50:30.565 DEBUG:boto:Port: 8001 2023-05-31T20:50:30.565 DEBUG:boto:Params: {} 2023-05-31T20:50:30.566 DEBUG:boto:Token: None 2023-05-31T20:50:30.566 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:50:30 GMT /pklxkd-59/obj34 2023-05-31T20:50:30.567 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:VAuRdGzWdBrzJP5CguB/PMdviQk= 2023-05-31T20:50:30.567 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:50:30 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:VAuRdGzWdBrzJP5CguB/PMdviQk=', 'Content-Length': '0'} 2023-05-31T20:50:30.569 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:43:07 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000feeb8342b44311fc-006477b316-1c17-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:50:30 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:50:30.570 DEBUG:boto:path=/pklxkd-59/obj34 2023-05-31T20:50:30.570 DEBUG:boto:auth_path=/pklxkd-59/obj34 2023-05-31T20:50:30.570 DEBUG:boto:Method: HEAD 2023-05-31T20:50:30.571 DEBUG:boto:Path: /pklxkd-59/obj34 2023-05-31T20:50:30.571 DEBUG:boto:Data: 2023-05-31T20:50:30.571 DEBUG:boto:Headers: {} 2023-05-31T20:50:30.572 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:50:30.572 DEBUG:boto:Port: 8000 2023-05-31T20:50:30.572 DEBUG:boto:Params: {} 2023-05-31T20:50:30.573 DEBUG:boto:Token: None 2023-05-31T20:50:30.573 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:50:30 GMT /pklxkd-59/obj34 2023-05-31T20:50:30.573 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:VAuRdGzWdBrzJP5CguB/PMdviQk= 2023-05-31T20:50:30.574 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:50:30 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:VAuRdGzWdBrzJP5CguB/PMdviQk=', 'Content-Length': '0'} 2023-05-31T20:50:30.575 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:43:07 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006ceacc4622689632-006477b316-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:50:30 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:50:30.576 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj34 2023-05-31T20:50:30.576 DEBUG:boto:path=/pklxkd-59/obj34 2023-05-31T20:50:30.577 DEBUG:boto:auth_path=/pklxkd-59/obj34 2023-05-31T20:50:30.578 DEBUG:boto:Method: GET 2023-05-31T20:50:30.578 DEBUG:boto:Path: /pklxkd-59/obj34 2023-05-31T20:50:30.579 DEBUG:boto:Data: 2023-05-31T20:50:30.579 DEBUG:boto:Headers: {} 2023-05-31T20:50:30.579 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:50:30.580 DEBUG:boto:Port: 8001 2023-05-31T20:50:30.580 DEBUG:boto:Params: {} 2023-05-31T20:50:30.581 DEBUG:boto:Token: None 2023-05-31T20:50:30.581 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:50:30 GMT /pklxkd-59/obj34 2023-05-31T20:50:30.582 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:UWhzd+WKCahMd2KWS0avDx/CH+U= 2023-05-31T20:50:30.582 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:50:30 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:UWhzd+WKCahMd2KWS0avDx/CH+U=', 'Content-Length': '0'} 2023-05-31T20:50:30.584 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:43:07 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000053fbf5ac358da61b-006477b316-1c17-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:50:30 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:50:30.585 DEBUG:boto:path=/pklxkd-59/obj34 2023-05-31T20:50:30.585 DEBUG:boto:auth_path=/pklxkd-59/obj34 2023-05-31T20:50:30.585 DEBUG:boto:Method: GET 2023-05-31T20:50:30.586 DEBUG:boto:Path: /pklxkd-59/obj34 2023-05-31T20:50:30.586 DEBUG:boto:Data: 2023-05-31T20:50:30.587 DEBUG:boto:Headers: {} 2023-05-31T20:50:30.587 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:50:30.587 DEBUG:boto:Port: 8000 2023-05-31T20:50:30.588 DEBUG:boto:Params: {} 2023-05-31T20:50:30.588 DEBUG:boto:Token: None 2023-05-31T20:50:30.589 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:50:30 GMT /pklxkd-59/obj34 2023-05-31T20:50:30.589 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:UWhzd+WKCahMd2KWS0avDx/CH+U= 2023-05-31T20:50:30.589 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:50:30 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:UWhzd+WKCahMd2KWS0avDx/CH+U=', 'Content-Length': '0'} 2023-05-31T20:50:30.591 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:43:07 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ab1411d738f0e9d1-006477b316-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:50:30 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:50:30.592 DEBUG:boto:path=/pklxkd-59/ 2023-05-31T20:50:30.593 DEBUG:boto:auth_path=/pklxkd-59/ 2023-05-31T20:50:30.593 DEBUG:boto:path=/pklxkd-59/?prefix=obj34 2023-05-31T20:50:30.593 DEBUG:boto:auth_path=/pklxkd-59/?prefix=obj34 2023-05-31T20:50:30.594 DEBUG:boto:Method: GET 2023-05-31T20:50:30.594 DEBUG:boto:Path: /pklxkd-59/?prefix=obj34 2023-05-31T20:50:30.594 DEBUG:boto:Data: 2023-05-31T20:50:30.595 DEBUG:boto:Headers: {} 2023-05-31T20:50:30.595 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:50:30.595 DEBUG:boto:Port: 8001 2023-05-31T20:50:30.596 DEBUG:boto:Params: {} 2023-05-31T20:50:30.596 DEBUG:boto:Token: None 2023-05-31T20:50:30.597 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:50:30 GMT /pklxkd-59/ 2023-05-31T20:50:30.597 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:3Yz4bfuBElJRjudql5UcebaTIW4= 2023-05-31T20:50:30.598 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:50:30 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:3Yz4bfuBElJRjudql5UcebaTIW4=', 'Content-Length': '0'} 2023-05-31T20:50:30.600 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000005805b04adadcb790-006477b316-1c17-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:50:30 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:50:30.600 DEBUG:boto:b'pklxkd-59obj341000falseobj342023-05-31T20:43:07.994Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:50:30.601 DEBUG:boto:path=/pklxkd-59/ 2023-05-31T20:50:30.601 DEBUG:boto:auth_path=/pklxkd-59/ 2023-05-31T20:50:30.602 DEBUG:boto:path=/pklxkd-59/?prefix=obj34 2023-05-31T20:50:30.602 DEBUG:boto:auth_path=/pklxkd-59/?prefix=obj34 2023-05-31T20:50:30.603 DEBUG:boto:Method: GET 2023-05-31T20:50:30.603 DEBUG:boto:Path: /pklxkd-59/?prefix=obj34 2023-05-31T20:50:30.603 DEBUG:boto:Data: 2023-05-31T20:50:30.604 DEBUG:boto:Headers: {} 2023-05-31T20:50:30.604 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:50:30.605 DEBUG:boto:Port: 8000 2023-05-31T20:50:30.605 DEBUG:boto:Params: {} 2023-05-31T20:50:30.605 DEBUG:boto:Token: None 2023-05-31T20:50:30.606 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:50:30 GMT /pklxkd-59/ 2023-05-31T20:50:30.606 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:3Yz4bfuBElJRjudql5UcebaTIW4= 2023-05-31T20:50:30.607 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:50:30 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:3Yz4bfuBElJRjudql5UcebaTIW4=', 'Content-Length': '0'} 2023-05-31T20:50:30.610 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d52e40a14a74be57-006477b316-1630-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:50:30 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:50:30.611 DEBUG:boto:b'pklxkd-59obj341000falseobj342023-05-31T20:43:07.994Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:50:30.611 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj38 2023-05-31T20:50:30.612 DEBUG:boto:path=/pklxkd-59/obj38 2023-05-31T20:50:30.612 DEBUG:boto:auth_path=/pklxkd-59/obj38 2023-05-31T20:50:30.612 DEBUG:boto:path=/pklxkd-59/obj38?versionId=null 2023-05-31T20:50:30.613 DEBUG:boto:auth_path=/pklxkd-59/obj38?versionId=null 2023-05-31T20:50:30.613 DEBUG:boto:Method: GET 2023-05-31T20:50:30.614 DEBUG:boto:Path: /pklxkd-59/obj38?versionId=null 2023-05-31T20:50:30.614 DEBUG:boto:Data: 2023-05-31T20:50:30.614 DEBUG:boto:Headers: {} 2023-05-31T20:50:30.615 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:50:30.615 DEBUG:boto:Port: 8001 2023-05-31T20:50:30.615 DEBUG:boto:Params: {} 2023-05-31T20:50:30.616 DEBUG:boto:Token: None 2023-05-31T20:50:30.616 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:50:30 GMT /pklxkd-59/obj38?versionId=null 2023-05-31T20:50:30.617 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:CgieiEmTr+o+KGiMLxtzrjSlgOk= 2023-05-31T20:50:30.617 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:50:30 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:CgieiEmTr+o+KGiMLxtzrjSlgOk=', 'Content-Length': '0'} 2023-05-31T20:50:30.619 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:44:50 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000082c1a55d639ab58b-006477b316-1c17-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:50:30 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:50:30.620 DEBUG:boto:path=/pklxkd-59/obj38 2023-05-31T20:50:30.620 DEBUG:boto:auth_path=/pklxkd-59/obj38 2023-05-31T20:50:30.620 DEBUG:boto:path=/pklxkd-59/obj38?versionId=null 2023-05-31T20:50:30.621 DEBUG:boto:auth_path=/pklxkd-59/obj38?versionId=null 2023-05-31T20:50:30.621 DEBUG:boto:Method: GET 2023-05-31T20:50:30.622 DEBUG:boto:Path: /pklxkd-59/obj38?versionId=null 2023-05-31T20:50:30.622 DEBUG:boto:Data: 2023-05-31T20:50:30.622 DEBUG:boto:Headers: {} 2023-05-31T20:50:30.623 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:50:30.623 DEBUG:boto:Port: 8000 2023-05-31T20:50:30.623 DEBUG:boto:Params: {} 2023-05-31T20:50:30.624 DEBUG:boto:Token: None 2023-05-31T20:50:30.624 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:50:30 GMT /pklxkd-59/obj38?versionId=null 2023-05-31T20:50:30.625 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:CgieiEmTr+o+KGiMLxtzrjSlgOk= 2023-05-31T20:50:30.625 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:50:30 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:CgieiEmTr+o+KGiMLxtzrjSlgOk=', 'Content-Length': '0'} 2023-05-31T20:50:30.627 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:44:50 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000032e4e83e5903d511-006477b316-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:50:30 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:50:30.668 DEBUG:boto:path=/pklxkd-59/obj38 2023-05-31T20:50:30.669 DEBUG:boto:auth_path=/pklxkd-59/obj38 2023-05-31T20:50:30.669 DEBUG:boto:path=/pklxkd-59/obj38?versionId=null 2023-05-31T20:50:30.670 DEBUG:boto:auth_path=/pklxkd-59/obj38?versionId=null 2023-05-31T20:50:30.670 DEBUG:boto:Method: HEAD 2023-05-31T20:50:30.670 DEBUG:boto:Path: /pklxkd-59/obj38?versionId=null 2023-05-31T20:50:30.671 DEBUG:boto:Data: 2023-05-31T20:50:30.671 DEBUG:boto:Headers: {} 2023-05-31T20:50:30.671 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:50:30.672 DEBUG:boto:Port: 8001 2023-05-31T20:50:30.672 DEBUG:boto:Params: {} 2023-05-31T20:50:30.672 DEBUG:boto:Token: None 2023-05-31T20:50:30.673 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:50:30 GMT /pklxkd-59/obj38?versionId=null 2023-05-31T20:50:30.673 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:/oQ5egrKklIGN0vS+PXCsofXIIw= 2023-05-31T20:50:30.674 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:50:30 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:/oQ5egrKklIGN0vS+PXCsofXIIw=', 'Content-Length': '0'} 2023-05-31T20:50:30.676 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:44:50 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001d16628fdce88d1c-006477b316-1c17-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:50:30 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:50:30.676 DEBUG:boto:path=/pklxkd-59/obj38 2023-05-31T20:50:30.677 DEBUG:boto:auth_path=/pklxkd-59/obj38 2023-05-31T20:50:30.677 DEBUG:boto:path=/pklxkd-59/obj38?versionId=null 2023-05-31T20:50:30.677 DEBUG:boto:auth_path=/pklxkd-59/obj38?versionId=null 2023-05-31T20:50:30.678 DEBUG:boto:Method: HEAD 2023-05-31T20:50:30.678 DEBUG:boto:Path: /pklxkd-59/obj38?versionId=null 2023-05-31T20:50:30.678 DEBUG:boto:Data: 2023-05-31T20:50:30.679 DEBUG:boto:Headers: {} 2023-05-31T20:50:30.679 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:50:30.680 DEBUG:boto:Port: 8000 2023-05-31T20:50:30.680 DEBUG:boto:Params: {} 2023-05-31T20:50:30.680 DEBUG:boto:Token: None 2023-05-31T20:50:30.681 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:50:30 GMT /pklxkd-59/obj38?versionId=null 2023-05-31T20:50:30.681 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:/oQ5egrKklIGN0vS+PXCsofXIIw= 2023-05-31T20:50:30.682 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:50:30 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:/oQ5egrKklIGN0vS+PXCsofXIIw=', 'Content-Length': '0'} 2023-05-31T20:50:30.683 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:44:50 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a99844b2712dee3f-006477b316-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:50:30 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:50:30.684 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj38 2023-05-31T20:50:30.684 DEBUG:boto:path=/pklxkd-59/obj38 2023-05-31T20:50:30.685 DEBUG:boto:auth_path=/pklxkd-59/obj38 2023-05-31T20:50:30.685 DEBUG:boto:path=/pklxkd-59/obj38?versionId=null 2023-05-31T20:50:30.685 DEBUG:boto:auth_path=/pklxkd-59/obj38?versionId=null 2023-05-31T20:50:30.686 DEBUG:boto:Method: GET 2023-05-31T20:50:30.686 DEBUG:boto:Path: /pklxkd-59/obj38?versionId=null 2023-05-31T20:50:30.687 DEBUG:boto:Data: 2023-05-31T20:50:30.687 DEBUG:boto:Headers: {} 2023-05-31T20:50:30.687 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:50:30.688 DEBUG:boto:Port: 8001 2023-05-31T20:50:30.688 DEBUG:boto:Params: {} 2023-05-31T20:50:30.688 DEBUG:boto:Token: None 2023-05-31T20:50:30.689 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:50:30 GMT /pklxkd-59/obj38?versionId=null 2023-05-31T20:50:30.689 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:CgieiEmTr+o+KGiMLxtzrjSlgOk= 2023-05-31T20:50:30.690 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:50:30 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:CgieiEmTr+o+KGiMLxtzrjSlgOk=', 'Content-Length': '0'} 2023-05-31T20:50:30.691 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:44:50 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d24aa3dc7b709b6e-006477b316-1c17-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:50:30 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:50:30.692 DEBUG:boto:path=/pklxkd-59/obj38 2023-05-31T20:50:30.692 DEBUG:boto:auth_path=/pklxkd-59/obj38 2023-05-31T20:50:30.693 DEBUG:boto:path=/pklxkd-59/obj38?versionId=null 2023-05-31T20:50:30.693 DEBUG:boto:auth_path=/pklxkd-59/obj38?versionId=null 2023-05-31T20:50:30.694 DEBUG:boto:Method: GET 2023-05-31T20:50:30.694 DEBUG:boto:Path: /pklxkd-59/obj38?versionId=null 2023-05-31T20:50:30.694 DEBUG:boto:Data: 2023-05-31T20:50:30.695 DEBUG:boto:Headers: {} 2023-05-31T20:50:30.695 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:50:30.695 DEBUG:boto:Port: 8000 2023-05-31T20:50:30.696 DEBUG:boto:Params: {} 2023-05-31T20:50:30.696 DEBUG:boto:Token: None 2023-05-31T20:50:30.697 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:50:30 GMT /pklxkd-59/obj38?versionId=null 2023-05-31T20:50:30.697 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:CgieiEmTr+o+KGiMLxtzrjSlgOk= 2023-05-31T20:50:30.698 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:50:30 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:CgieiEmTr+o+KGiMLxtzrjSlgOk=', 'Content-Length': '0'} 2023-05-31T20:50:30.700 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:44:50 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c2b22b26f39884d2-006477b316-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:50:30 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:50:30.700 DEBUG:boto:path=/pklxkd-59/ 2023-05-31T20:50:30.701 DEBUG:boto:auth_path=/pklxkd-59/ 2023-05-31T20:50:30.701 DEBUG:boto:path=/pklxkd-59/?prefix=obj38 2023-05-31T20:50:30.701 DEBUG:boto:auth_path=/pklxkd-59/?prefix=obj38 2023-05-31T20:50:30.702 DEBUG:boto:Method: GET 2023-05-31T20:50:30.702 DEBUG:boto:Path: /pklxkd-59/?prefix=obj38 2023-05-31T20:50:30.703 DEBUG:boto:Data: 2023-05-31T20:50:30.703 DEBUG:boto:Headers: {} 2023-05-31T20:50:30.703 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:50:30.704 DEBUG:boto:Port: 8001 2023-05-31T20:50:30.704 DEBUG:boto:Params: {} 2023-05-31T20:50:30.704 DEBUG:boto:Token: None 2023-05-31T20:50:30.705 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:50:30 GMT /pklxkd-59/ 2023-05-31T20:50:30.705 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:3Yz4bfuBElJRjudql5UcebaTIW4= 2023-05-31T20:50:30.706 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:50:30 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:3Yz4bfuBElJRjudql5UcebaTIW4=', 'Content-Length': '0'} 2023-05-31T20:50:30.708 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f57dc96dd47351be-006477b316-1c17-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:50:30 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:50:30.708 DEBUG:boto:b'pklxkd-59obj381000falseobj382023-05-31T20:44:50.610Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:50:30.709 DEBUG:boto:path=/pklxkd-59/ 2023-05-31T20:50:30.710 DEBUG:boto:auth_path=/pklxkd-59/ 2023-05-31T20:50:30.710 DEBUG:boto:path=/pklxkd-59/?prefix=obj38 2023-05-31T20:50:30.710 DEBUG:boto:auth_path=/pklxkd-59/?prefix=obj38 2023-05-31T20:50:30.711 DEBUG:boto:Method: GET 2023-05-31T20:50:30.711 DEBUG:boto:Path: /pklxkd-59/?prefix=obj38 2023-05-31T20:50:30.711 DEBUG:boto:Data: 2023-05-31T20:50:30.712 DEBUG:boto:Headers: {} 2023-05-31T20:50:30.712 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:50:30.713 DEBUG:boto:Port: 8000 2023-05-31T20:50:30.713 DEBUG:boto:Params: {} 2023-05-31T20:50:30.713 DEBUG:boto:Token: None 2023-05-31T20:50:30.714 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:50:30 GMT /pklxkd-59/ 2023-05-31T20:50:30.714 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:3Yz4bfuBElJRjudql5UcebaTIW4= 2023-05-31T20:50:30.715 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:50:30 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:3Yz4bfuBElJRjudql5UcebaTIW4=', 'Content-Length': '0'} 2023-05-31T20:50:30.718 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000649993292ab43448-006477b316-1630-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:50:30 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:50:30.718 DEBUG:boto:b'pklxkd-59obj381000falseobj382023-05-31T20:44:50.610Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:50:30.719 DEBUG:boto:path=/pklxkd-59/obj38 2023-05-31T20:50:30.719 DEBUG:boto:auth_path=/pklxkd-59/obj38 2023-05-31T20:50:30.720 DEBUG:boto:Method: HEAD 2023-05-31T20:50:30.720 DEBUG:boto:Path: /pklxkd-59/obj38 2023-05-31T20:50:30.721 DEBUG:boto:Data: 2023-05-31T20:50:30.721 DEBUG:boto:Headers: {} 2023-05-31T20:50:30.721 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:50:30.722 DEBUG:boto:Port: 8001 2023-05-31T20:50:30.722 DEBUG:boto:Params: {} 2023-05-31T20:50:30.722 DEBUG:boto:Token: None 2023-05-31T20:50:30.723 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:50:30 GMT /pklxkd-59/obj38 2023-05-31T20:50:30.723 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:S8/XRI50kokKMyP3g6VxSsxW/Ig= 2023-05-31T20:50:30.724 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:50:30 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:S8/XRI50kokKMyP3g6VxSsxW/Ig=', 'Content-Length': '0'} 2023-05-31T20:50:30.725 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:44:50 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006736b9d1d6e99d2b-006477b316-1c17-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:50:30 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:50:30.726 DEBUG:boto:path=/pklxkd-59/obj38 2023-05-31T20:50:30.726 DEBUG:boto:auth_path=/pklxkd-59/obj38 2023-05-31T20:50:30.727 DEBUG:boto:Method: HEAD 2023-05-31T20:50:30.727 DEBUG:boto:Path: /pklxkd-59/obj38 2023-05-31T20:50:30.727 DEBUG:boto:Data: 2023-05-31T20:50:30.728 DEBUG:boto:Headers: {} 2023-05-31T20:50:30.728 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:50:30.729 DEBUG:boto:Port: 8000 2023-05-31T20:50:30.729 DEBUG:boto:Params: {} 2023-05-31T20:50:30.729 DEBUG:boto:Token: None 2023-05-31T20:50:30.730 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:50:30 GMT /pklxkd-59/obj38 2023-05-31T20:50:30.730 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:S8/XRI50kokKMyP3g6VxSsxW/Ig= 2023-05-31T20:50:30.730 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:50:30 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:S8/XRI50kokKMyP3g6VxSsxW/Ig=', 'Content-Length': '0'} 2023-05-31T20:50:30.732 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:44:50 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004d7b1b56e494c238-006477b316-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:50:30 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:50:30.733 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj38 2023-05-31T20:50:30.733 DEBUG:boto:path=/pklxkd-59/obj38 2023-05-31T20:50:30.733 DEBUG:boto:auth_path=/pklxkd-59/obj38 2023-05-31T20:50:30.734 DEBUG:boto:Method: GET 2023-05-31T20:50:30.734 DEBUG:boto:Path: /pklxkd-59/obj38 2023-05-31T20:50:30.735 DEBUG:boto:Data: 2023-05-31T20:50:30.735 DEBUG:boto:Headers: {} 2023-05-31T20:50:30.735 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:50:30.736 DEBUG:boto:Port: 8001 2023-05-31T20:50:30.736 DEBUG:boto:Params: {} 2023-05-31T20:50:30.737 DEBUG:boto:Token: None 2023-05-31T20:50:30.737 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:50:30 GMT /pklxkd-59/obj38 2023-05-31T20:50:30.737 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:bUJXyBrAy7Lzamvt3S0tOGm6Ef4= 2023-05-31T20:50:30.738 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:50:30 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:bUJXyBrAy7Lzamvt3S0tOGm6Ef4=', 'Content-Length': '0'} 2023-05-31T20:50:30.739 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:44:50 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000056ac60c143585bef-006477b316-1c17-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:50:30 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:50:30.780 DEBUG:boto:path=/pklxkd-59/obj38 2023-05-31T20:50:30.781 DEBUG:boto:auth_path=/pklxkd-59/obj38 2023-05-31T20:50:30.781 DEBUG:boto:Method: GET 2023-05-31T20:50:30.782 DEBUG:boto:Path: /pklxkd-59/obj38 2023-05-31T20:50:30.782 DEBUG:boto:Data: 2023-05-31T20:50:30.782 DEBUG:boto:Headers: {} 2023-05-31T20:50:30.783 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:50:30.783 DEBUG:boto:Port: 8000 2023-05-31T20:50:30.783 DEBUG:boto:Params: {} 2023-05-31T20:50:30.784 DEBUG:boto:Token: None 2023-05-31T20:50:30.784 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:50:30 GMT /pklxkd-59/obj38 2023-05-31T20:50:30.785 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:bUJXyBrAy7Lzamvt3S0tOGm6Ef4= 2023-05-31T20:50:30.785 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:50:30 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:bUJXyBrAy7Lzamvt3S0tOGm6Ef4=', 'Content-Length': '0'} 2023-05-31T20:50:30.787 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:44:50 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000037ed497e17337515-006477b316-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:50:30 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:50:30.788 DEBUG:boto:path=/pklxkd-59/ 2023-05-31T20:50:30.788 DEBUG:boto:auth_path=/pklxkd-59/ 2023-05-31T20:50:30.789 DEBUG:boto:path=/pklxkd-59/?prefix=obj38 2023-05-31T20:50:30.789 DEBUG:boto:auth_path=/pklxkd-59/?prefix=obj38 2023-05-31T20:50:30.789 DEBUG:boto:Method: GET 2023-05-31T20:50:30.790 DEBUG:boto:Path: /pklxkd-59/?prefix=obj38 2023-05-31T20:50:30.790 DEBUG:boto:Data: 2023-05-31T20:50:30.791 DEBUG:boto:Headers: {} 2023-05-31T20:50:30.791 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:50:30.791 DEBUG:boto:Port: 8001 2023-05-31T20:50:30.792 DEBUG:boto:Params: {} 2023-05-31T20:50:30.792 DEBUG:boto:Token: None 2023-05-31T20:50:30.793 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:50:30 GMT /pklxkd-59/ 2023-05-31T20:50:30.793 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:3Yz4bfuBElJRjudql5UcebaTIW4= 2023-05-31T20:50:30.793 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:50:30 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:3Yz4bfuBElJRjudql5UcebaTIW4=', 'Content-Length': '0'} 2023-05-31T20:50:30.796 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000085783c561ba3c74e-006477b316-1c17-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:50:30 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:50:30.797 DEBUG:boto:b'pklxkd-59obj381000falseobj382023-05-31T20:44:50.610Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:50:30.797 DEBUG:boto:path=/pklxkd-59/ 2023-05-31T20:50:30.798 DEBUG:boto:auth_path=/pklxkd-59/ 2023-05-31T20:50:30.798 DEBUG:boto:path=/pklxkd-59/?prefix=obj38 2023-05-31T20:50:30.798 DEBUG:boto:auth_path=/pklxkd-59/?prefix=obj38 2023-05-31T20:50:30.799 DEBUG:boto:Method: GET 2023-05-31T20:50:30.799 DEBUG:boto:Path: /pklxkd-59/?prefix=obj38 2023-05-31T20:50:30.799 DEBUG:boto:Data: 2023-05-31T20:50:30.800 DEBUG:boto:Headers: {} 2023-05-31T20:50:30.800 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:50:30.801 DEBUG:boto:Port: 8000 2023-05-31T20:50:30.801 DEBUG:boto:Params: {} 2023-05-31T20:50:30.801 DEBUG:boto:Token: None 2023-05-31T20:50:30.802 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:50:30 GMT /pklxkd-59/ 2023-05-31T20:50:30.802 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:3Yz4bfuBElJRjudql5UcebaTIW4= 2023-05-31T20:50:30.803 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:50:30 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:3Yz4bfuBElJRjudql5UcebaTIW4=', 'Content-Length': '0'} 2023-05-31T20:50:30.806 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b49429b3842daecd-006477b316-1630-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:50:30 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:50:30.807 DEBUG:boto:b'pklxkd-59obj381000falseobj382023-05-31T20:44:50.610Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:50:30.807 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj44 2023-05-31T20:50:30.808 DEBUG:boto:path=/pklxkd-59/obj44 2023-05-31T20:50:30.808 DEBUG:boto:auth_path=/pklxkd-59/obj44 2023-05-31T20:50:30.808 DEBUG:boto:path=/pklxkd-59/obj44?versionId=null 2023-05-31T20:50:30.809 DEBUG:boto:auth_path=/pklxkd-59/obj44?versionId=null 2023-05-31T20:50:30.809 DEBUG:boto:Method: GET 2023-05-31T20:50:30.810 DEBUG:boto:Path: /pklxkd-59/obj44?versionId=null 2023-05-31T20:50:30.810 DEBUG:boto:Data: 2023-05-31T20:50:30.810 DEBUG:boto:Headers: {} 2023-05-31T20:50:30.811 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:50:30.811 DEBUG:boto:Port: 8001 2023-05-31T20:50:30.811 DEBUG:boto:Params: {} 2023-05-31T20:50:30.812 DEBUG:boto:Token: None 2023-05-31T20:50:30.812 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:50:30 GMT /pklxkd-59/obj44?versionId=null 2023-05-31T20:50:30.813 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:BSuI+63gelR5nvTiomofJW9uzP0= 2023-05-31T20:50:30.813 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:50:30 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:BSuI+63gelR5nvTiomofJW9uzP0=', 'Content-Length': '0'} 2023-05-31T20:50:30.815 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:39:26 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ee5f8dee4d43d916-006477b316-1c17-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:50:30 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:50:30.816 DEBUG:boto:path=/pklxkd-59/obj44 2023-05-31T20:50:30.816 DEBUG:boto:auth_path=/pklxkd-59/obj44 2023-05-31T20:50:30.816 DEBUG:boto:path=/pklxkd-59/obj44?versionId=null 2023-05-31T20:50:30.817 DEBUG:boto:auth_path=/pklxkd-59/obj44?versionId=null 2023-05-31T20:50:30.817 DEBUG:boto:Method: GET 2023-05-31T20:50:30.818 DEBUG:boto:Path: /pklxkd-59/obj44?versionId=null 2023-05-31T20:50:30.818 DEBUG:boto:Data: 2023-05-31T20:50:30.818 DEBUG:boto:Headers: {} 2023-05-31T20:50:30.819 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:50:30.819 DEBUG:boto:Port: 8000 2023-05-31T20:50:30.819 DEBUG:boto:Params: {} 2023-05-31T20:50:30.820 DEBUG:boto:Token: None 2023-05-31T20:50:30.820 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:50:30 GMT /pklxkd-59/obj44?versionId=null 2023-05-31T20:50:30.821 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:BSuI+63gelR5nvTiomofJW9uzP0= 2023-05-31T20:50:30.821 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:50:30 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:BSuI+63gelR5nvTiomofJW9uzP0=', 'Content-Length': '0'} 2023-05-31T20:50:30.823 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:39:26 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000045ba874d1b28d160-006477b316-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:50:30 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:50:30.864 DEBUG:boto:path=/pklxkd-59/obj44 2023-05-31T20:50:30.865 DEBUG:boto:auth_path=/pklxkd-59/obj44 2023-05-31T20:50:30.865 DEBUG:boto:path=/pklxkd-59/obj44?versionId=null 2023-05-31T20:50:30.865 DEBUG:boto:auth_path=/pklxkd-59/obj44?versionId=null 2023-05-31T20:50:30.866 DEBUG:boto:Method: HEAD 2023-05-31T20:50:30.866 DEBUG:boto:Path: /pklxkd-59/obj44?versionId=null 2023-05-31T20:50:30.866 DEBUG:boto:Data: 2023-05-31T20:50:30.867 DEBUG:boto:Headers: {} 2023-05-31T20:50:30.867 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:50:30.868 DEBUG:boto:Port: 8001 2023-05-31T20:50:30.868 DEBUG:boto:Params: {} 2023-05-31T20:50:30.868 DEBUG:boto:Token: None 2023-05-31T20:50:30.869 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:50:30 GMT /pklxkd-59/obj44?versionId=null 2023-05-31T20:50:30.869 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:Y6VSWHSbvjZap1gM2K+U6pMC5vQ= 2023-05-31T20:50:30.870 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:50:30 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:Y6VSWHSbvjZap1gM2K+U6pMC5vQ=', 'Content-Length': '0'} 2023-05-31T20:50:30.872 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:39:26 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000021f69f0b84ad81f5-006477b316-1c17-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:50:30 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:50:30.872 DEBUG:boto:path=/pklxkd-59/obj44 2023-05-31T20:50:30.873 DEBUG:boto:auth_path=/pklxkd-59/obj44 2023-05-31T20:50:30.873 DEBUG:boto:path=/pklxkd-59/obj44?versionId=null 2023-05-31T20:50:30.873 DEBUG:boto:auth_path=/pklxkd-59/obj44?versionId=null 2023-05-31T20:50:30.874 DEBUG:boto:Method: HEAD 2023-05-31T20:50:30.874 DEBUG:boto:Path: /pklxkd-59/obj44?versionId=null 2023-05-31T20:50:30.874 DEBUG:boto:Data: 2023-05-31T20:50:30.875 DEBUG:boto:Headers: {} 2023-05-31T20:50:30.875 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:50:30.875 DEBUG:boto:Port: 8000 2023-05-31T20:50:30.876 DEBUG:boto:Params: {} 2023-05-31T20:50:30.876 DEBUG:boto:Token: None 2023-05-31T20:50:30.876 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:50:30 GMT /pklxkd-59/obj44?versionId=null 2023-05-31T20:50:30.877 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:Y6VSWHSbvjZap1gM2K+U6pMC5vQ= 2023-05-31T20:50:30.877 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:50:30 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:Y6VSWHSbvjZap1gM2K+U6pMC5vQ=', 'Content-Length': '0'} 2023-05-31T20:50:30.879 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:39:26 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e464a1e583f2f93f-006477b316-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:50:30 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:50:30.880 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj44 2023-05-31T20:50:30.880 DEBUG:boto:path=/pklxkd-59/obj44 2023-05-31T20:50:30.881 DEBUG:boto:auth_path=/pklxkd-59/obj44 2023-05-31T20:50:30.881 DEBUG:boto:path=/pklxkd-59/obj44?versionId=null 2023-05-31T20:50:30.881 DEBUG:boto:auth_path=/pklxkd-59/obj44?versionId=null 2023-05-31T20:50:30.882 DEBUG:boto:Method: GET 2023-05-31T20:50:30.882 DEBUG:boto:Path: /pklxkd-59/obj44?versionId=null 2023-05-31T20:50:30.883 DEBUG:boto:Data: 2023-05-31T20:50:30.883 DEBUG:boto:Headers: {} 2023-05-31T20:50:30.883 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:50:30.884 DEBUG:boto:Port: 8001 2023-05-31T20:50:30.884 DEBUG:boto:Params: {} 2023-05-31T20:50:30.885 DEBUG:boto:Token: None 2023-05-31T20:50:30.885 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:50:30 GMT /pklxkd-59/obj44?versionId=null 2023-05-31T20:50:30.885 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:BSuI+63gelR5nvTiomofJW9uzP0= 2023-05-31T20:50:30.886 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:50:30 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:BSuI+63gelR5nvTiomofJW9uzP0=', 'Content-Length': '0'} 2023-05-31T20:50:30.888 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:39:26 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006dfcd1722a439a03-006477b316-1c17-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:50:30 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:50:30.889 DEBUG:boto:path=/pklxkd-59/obj44 2023-05-31T20:50:30.889 DEBUG:boto:auth_path=/pklxkd-59/obj44 2023-05-31T20:50:30.889 DEBUG:boto:path=/pklxkd-59/obj44?versionId=null 2023-05-31T20:50:30.890 DEBUG:boto:auth_path=/pklxkd-59/obj44?versionId=null 2023-05-31T20:50:30.890 DEBUG:boto:Method: GET 2023-05-31T20:50:30.890 DEBUG:boto:Path: /pklxkd-59/obj44?versionId=null 2023-05-31T20:50:30.891 DEBUG:boto:Data: 2023-05-31T20:50:30.891 DEBUG:boto:Headers: {} 2023-05-31T20:50:30.892 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:50:30.892 DEBUG:boto:Port: 8000 2023-05-31T20:50:30.892 DEBUG:boto:Params: {} 2023-05-31T20:50:30.893 DEBUG:boto:Token: None 2023-05-31T20:50:30.893 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:50:30 GMT /pklxkd-59/obj44?versionId=null 2023-05-31T20:50:30.894 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:BSuI+63gelR5nvTiomofJW9uzP0= 2023-05-31T20:50:30.894 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:50:30 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:BSuI+63gelR5nvTiomofJW9uzP0=', 'Content-Length': '0'} 2023-05-31T20:50:30.896 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:39:26 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006f95e466753633d4-006477b316-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:50:30 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:50:30.897 DEBUG:boto:path=/pklxkd-59/ 2023-05-31T20:50:30.897 DEBUG:boto:auth_path=/pklxkd-59/ 2023-05-31T20:50:30.898 DEBUG:boto:path=/pklxkd-59/?prefix=obj44 2023-05-31T20:50:30.898 DEBUG:boto:auth_path=/pklxkd-59/?prefix=obj44 2023-05-31T20:50:30.898 DEBUG:boto:Method: GET 2023-05-31T20:50:30.899 DEBUG:boto:Path: /pklxkd-59/?prefix=obj44 2023-05-31T20:50:30.899 DEBUG:boto:Data: 2023-05-31T20:50:30.900 DEBUG:boto:Headers: {} 2023-05-31T20:50:30.900 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:50:30.900 DEBUG:boto:Port: 8001 2023-05-31T20:50:30.901 DEBUG:boto:Params: {} 2023-05-31T20:50:30.901 DEBUG:boto:Token: None 2023-05-31T20:50:30.902 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:50:30 GMT /pklxkd-59/ 2023-05-31T20:50:30.902 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:3Yz4bfuBElJRjudql5UcebaTIW4= 2023-05-31T20:50:30.902 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:50:30 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:3Yz4bfuBElJRjudql5UcebaTIW4=', 'Content-Length': '0'} 2023-05-31T20:50:30.905 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000019f89f123b44cf23-006477b316-1c17-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:50:30 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:50:30.905 DEBUG:boto:b'pklxkd-59obj441000falseobj442023-05-31T20:39:26.565Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:50:30.906 DEBUG:boto:path=/pklxkd-59/ 2023-05-31T20:50:30.906 DEBUG:boto:auth_path=/pklxkd-59/ 2023-05-31T20:50:30.907 DEBUG:boto:path=/pklxkd-59/?prefix=obj44 2023-05-31T20:50:30.907 DEBUG:boto:auth_path=/pklxkd-59/?prefix=obj44 2023-05-31T20:50:30.907 DEBUG:boto:Method: GET 2023-05-31T20:50:30.908 DEBUG:boto:Path: /pklxkd-59/?prefix=obj44 2023-05-31T20:50:30.908 DEBUG:boto:Data: 2023-05-31T20:50:30.909 DEBUG:boto:Headers: {} 2023-05-31T20:50:30.909 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:50:30.909 DEBUG:boto:Port: 8000 2023-05-31T20:50:30.910 DEBUG:boto:Params: {} 2023-05-31T20:50:30.910 DEBUG:boto:Token: None 2023-05-31T20:50:30.910 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:50:30 GMT /pklxkd-59/ 2023-05-31T20:50:30.911 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:3Yz4bfuBElJRjudql5UcebaTIW4= 2023-05-31T20:50:30.911 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:50:30 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:3Yz4bfuBElJRjudql5UcebaTIW4=', 'Content-Length': '0'} 2023-05-31T20:50:30.915 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000000ea87a2c7338670c-006477b316-1630-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:50:30 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:50:30.915 DEBUG:boto:b'pklxkd-59obj441000falseobj442023-05-31T20:39:26.565Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:50:30.916 DEBUG:boto:path=/pklxkd-59/obj44 2023-05-31T20:50:30.917 DEBUG:boto:auth_path=/pklxkd-59/obj44 2023-05-31T20:50:30.917 DEBUG:boto:Method: HEAD 2023-05-31T20:50:30.917 DEBUG:boto:Path: /pklxkd-59/obj44 2023-05-31T20:50:30.918 DEBUG:boto:Data: 2023-05-31T20:50:30.918 DEBUG:boto:Headers: {} 2023-05-31T20:50:30.918 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:50:30.919 DEBUG:boto:Port: 8001 2023-05-31T20:50:30.919 DEBUG:boto:Params: {} 2023-05-31T20:50:30.919 DEBUG:boto:Token: None 2023-05-31T20:50:30.920 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:50:30 GMT /pklxkd-59/obj44 2023-05-31T20:50:30.920 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:c8BIEjOSJVKa3TPixMImXYF9SkA= 2023-05-31T20:50:30.921 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:50:30 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:c8BIEjOSJVKa3TPixMImXYF9SkA=', 'Content-Length': '0'} 2023-05-31T20:50:30.923 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:39:26 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c0e0c3c2c528740d-006477b316-1c17-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:50:30 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:50:30.923 DEBUG:boto:path=/pklxkd-59/obj44 2023-05-31T20:50:30.923 DEBUG:boto:auth_path=/pklxkd-59/obj44 2023-05-31T20:50:30.924 DEBUG:boto:Method: HEAD 2023-05-31T20:50:30.924 DEBUG:boto:Path: /pklxkd-59/obj44 2023-05-31T20:50:30.925 DEBUG:boto:Data: 2023-05-31T20:50:30.925 DEBUG:boto:Headers: {} 2023-05-31T20:50:30.925 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:50:30.926 DEBUG:boto:Port: 8000 2023-05-31T20:50:30.926 DEBUG:boto:Params: {} 2023-05-31T20:50:30.926 DEBUG:boto:Token: None 2023-05-31T20:50:30.927 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:50:30 GMT /pklxkd-59/obj44 2023-05-31T20:50:30.927 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:c8BIEjOSJVKa3TPixMImXYF9SkA= 2023-05-31T20:50:30.928 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:50:30 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:c8BIEjOSJVKa3TPixMImXYF9SkA=', 'Content-Length': '0'} 2023-05-31T20:50:30.930 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:39:26 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b9cf776f60196083-006477b316-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:50:30 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:50:30.930 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj44 2023-05-31T20:50:30.931 DEBUG:boto:path=/pklxkd-59/obj44 2023-05-31T20:50:30.931 DEBUG:boto:auth_path=/pklxkd-59/obj44 2023-05-31T20:50:30.931 DEBUG:boto:Method: GET 2023-05-31T20:50:30.932 DEBUG:boto:Path: /pklxkd-59/obj44 2023-05-31T20:50:30.932 DEBUG:boto:Data: 2023-05-31T20:50:30.932 DEBUG:boto:Headers: {} 2023-05-31T20:50:30.933 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:50:30.933 DEBUG:boto:Port: 8001 2023-05-31T20:50:30.934 DEBUG:boto:Params: {} 2023-05-31T20:50:30.934 DEBUG:boto:Token: None 2023-05-31T20:50:30.934 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:50:30 GMT /pklxkd-59/obj44 2023-05-31T20:50:30.935 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:ABRtho5D73NTSnQ1jvvCgM06VU8= 2023-05-31T20:50:30.935 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:50:30 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:ABRtho5D73NTSnQ1jvvCgM06VU8=', 'Content-Length': '0'} 2023-05-31T20:50:30.937 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:39:26 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007e32adbc698f77ad-006477b316-1c17-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:50:30 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:50:30.984 DEBUG:boto:path=/pklxkd-59/obj44 2023-05-31T20:50:30.985 DEBUG:boto:auth_path=/pklxkd-59/obj44 2023-05-31T20:50:30.985 DEBUG:boto:Method: GET 2023-05-31T20:50:30.986 DEBUG:boto:Path: /pklxkd-59/obj44 2023-05-31T20:50:30.986 DEBUG:boto:Data: 2023-05-31T20:50:30.986 DEBUG:boto:Headers: {} 2023-05-31T20:50:30.987 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:50:30.987 DEBUG:boto:Port: 8000 2023-05-31T20:50:30.987 DEBUG:boto:Params: {} 2023-05-31T20:50:30.988 DEBUG:boto:Token: None 2023-05-31T20:50:30.988 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:50:30 GMT /pklxkd-59/obj44 2023-05-31T20:50:30.989 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:ABRtho5D73NTSnQ1jvvCgM06VU8= 2023-05-31T20:50:30.989 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:50:30 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:ABRtho5D73NTSnQ1jvvCgM06VU8=', 'Content-Length': '0'} 2023-05-31T20:50:30.991 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:39:26 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007240044fc8b7ec8f-006477b316-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:50:30 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:50:30.992 DEBUG:boto:path=/pklxkd-59/ 2023-05-31T20:50:30.992 DEBUG:boto:auth_path=/pklxkd-59/ 2023-05-31T20:50:30.993 DEBUG:boto:path=/pklxkd-59/?prefix=obj44 2023-05-31T20:50:30.993 DEBUG:boto:auth_path=/pklxkd-59/?prefix=obj44 2023-05-31T20:50:30.994 DEBUG:boto:Method: GET 2023-05-31T20:50:30.994 DEBUG:boto:Path: /pklxkd-59/?prefix=obj44 2023-05-31T20:50:30.994 DEBUG:boto:Data: 2023-05-31T20:50:30.995 DEBUG:boto:Headers: {} 2023-05-31T20:50:30.995 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:50:30.995 DEBUG:boto:Port: 8001 2023-05-31T20:50:30.996 DEBUG:boto:Params: {} 2023-05-31T20:50:30.996 DEBUG:boto:Token: None 2023-05-31T20:50:30.997 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:50:30 GMT /pklxkd-59/ 2023-05-31T20:50:30.997 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:3Yz4bfuBElJRjudql5UcebaTIW4= 2023-05-31T20:50:30.998 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:50:30 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:3Yz4bfuBElJRjudql5UcebaTIW4=', 'Content-Length': '0'} 2023-05-31T20:50:31.000 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000067f9c7cd5f735b26-006477b316-1c17-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:50:30 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:50:31.001 DEBUG:boto:b'pklxkd-59obj441000falseobj442023-05-31T20:39:26.565Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:50:31.001 DEBUG:boto:path=/pklxkd-59/ 2023-05-31T20:50:31.002 DEBUG:boto:auth_path=/pklxkd-59/ 2023-05-31T20:50:31.002 DEBUG:boto:path=/pklxkd-59/?prefix=obj44 2023-05-31T20:50:31.002 DEBUG:boto:auth_path=/pklxkd-59/?prefix=obj44 2023-05-31T20:50:31.003 DEBUG:boto:Method: GET 2023-05-31T20:50:31.003 DEBUG:boto:Path: /pklxkd-59/?prefix=obj44 2023-05-31T20:50:31.003 DEBUG:boto:Data: 2023-05-31T20:50:31.004 DEBUG:boto:Headers: {} 2023-05-31T20:50:31.004 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:50:31.005 DEBUG:boto:Port: 8000 2023-05-31T20:50:31.005 DEBUG:boto:Params: {} 2023-05-31T20:50:31.005 DEBUG:boto:Token: None 2023-05-31T20:50:31.006 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:50:31 GMT /pklxkd-59/ 2023-05-31T20:50:31.006 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:apPqLCZHOWwza3uFvs+KzJ/Flyg= 2023-05-31T20:50:31.007 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:50:31 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:apPqLCZHOWwza3uFvs+KzJ/Flyg=', 'Content-Length': '0'} 2023-05-31T20:50:31.010 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b8cbbcc0ac7e1d88-006477b317-1630-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:50:31 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:50:31.011 DEBUG:boto:b'pklxkd-59obj441000falseobj442023-05-31T20:39:26.565Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:50:31.011 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj46 2023-05-31T20:50:31.012 DEBUG:boto:path=/pklxkd-59/obj46 2023-05-31T20:50:31.012 DEBUG:boto:auth_path=/pklxkd-59/obj46 2023-05-31T20:50:31.013 DEBUG:boto:path=/pklxkd-59/obj46?versionId=null 2023-05-31T20:50:31.013 DEBUG:boto:auth_path=/pklxkd-59/obj46?versionId=null 2023-05-31T20:50:31.013 DEBUG:boto:Method: GET 2023-05-31T20:50:31.014 DEBUG:boto:Path: /pklxkd-59/obj46?versionId=null 2023-05-31T20:50:31.014 DEBUG:boto:Data: 2023-05-31T20:50:31.014 DEBUG:boto:Headers: {} 2023-05-31T20:50:31.015 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:50:31.015 DEBUG:boto:Port: 8001 2023-05-31T20:50:31.016 DEBUG:boto:Params: {} 2023-05-31T20:50:31.016 DEBUG:boto:Token: None 2023-05-31T20:50:31.016 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:50:31 GMT /pklxkd-59/obj46?versionId=null 2023-05-31T20:50:31.017 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:C4aC04pyG8393bhHCv6KwAjSkNU= 2023-05-31T20:50:31.017 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:50:31 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:C4aC04pyG8393bhHCv6KwAjSkNU=', 'Content-Length': '0'} 2023-05-31T20:50:31.019 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:46:33 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000bd48cd25f8088e54-006477b317-1c17-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:50:31 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:50:31.020 DEBUG:boto:path=/pklxkd-59/obj46 2023-05-31T20:50:31.020 DEBUG:boto:auth_path=/pklxkd-59/obj46 2023-05-31T20:50:31.021 DEBUG:boto:path=/pklxkd-59/obj46?versionId=null 2023-05-31T20:50:31.021 DEBUG:boto:auth_path=/pklxkd-59/obj46?versionId=null 2023-05-31T20:50:31.021 DEBUG:boto:Method: GET 2023-05-31T20:50:31.022 DEBUG:boto:Path: /pklxkd-59/obj46?versionId=null 2023-05-31T20:50:31.022 DEBUG:boto:Data: 2023-05-31T20:50:31.022 DEBUG:boto:Headers: {} 2023-05-31T20:50:31.023 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:50:31.023 DEBUG:boto:Port: 8000 2023-05-31T20:50:31.024 DEBUG:boto:Params: {} 2023-05-31T20:50:31.024 DEBUG:boto:Token: None 2023-05-31T20:50:31.024 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:50:31 GMT /pklxkd-59/obj46?versionId=null 2023-05-31T20:50:31.025 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:C4aC04pyG8393bhHCv6KwAjSkNU= 2023-05-31T20:50:31.025 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:50:31 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:C4aC04pyG8393bhHCv6KwAjSkNU=', 'Content-Length': '0'} 2023-05-31T20:50:31.027 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:46:33 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000053480522ed73c123-006477b317-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:50:31 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:50:31.068 DEBUG:boto:path=/pklxkd-59/obj46 2023-05-31T20:50:31.069 DEBUG:boto:auth_path=/pklxkd-59/obj46 2023-05-31T20:50:31.069 DEBUG:boto:path=/pklxkd-59/obj46?versionId=null 2023-05-31T20:50:31.070 DEBUG:boto:auth_path=/pklxkd-59/obj46?versionId=null 2023-05-31T20:50:31.070 DEBUG:boto:Method: HEAD 2023-05-31T20:50:31.070 DEBUG:boto:Path: /pklxkd-59/obj46?versionId=null 2023-05-31T20:50:31.071 DEBUG:boto:Data: 2023-05-31T20:50:31.071 DEBUG:boto:Headers: {} 2023-05-31T20:50:31.071 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:50:31.072 DEBUG:boto:Port: 8001 2023-05-31T20:50:31.072 DEBUG:boto:Params: {} 2023-05-31T20:50:31.072 DEBUG:boto:Token: None 2023-05-31T20:50:31.073 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:50:31 GMT /pklxkd-59/obj46?versionId=null 2023-05-31T20:50:31.073 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:miXyQgF6wYSfcNpYsG06HcK69vY= 2023-05-31T20:50:31.074 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:50:31 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:miXyQgF6wYSfcNpYsG06HcK69vY=', 'Content-Length': '0'} 2023-05-31T20:50:31.075 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:46:33 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000040f4d2bb8e9c25ec-006477b317-1c17-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:50:31 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:50:31.076 DEBUG:boto:path=/pklxkd-59/obj46 2023-05-31T20:50:31.076 DEBUG:boto:auth_path=/pklxkd-59/obj46 2023-05-31T20:50:31.077 DEBUG:boto:path=/pklxkd-59/obj46?versionId=null 2023-05-31T20:50:31.077 DEBUG:boto:auth_path=/pklxkd-59/obj46?versionId=null 2023-05-31T20:50:31.077 DEBUG:boto:Method: HEAD 2023-05-31T20:50:31.078 DEBUG:boto:Path: /pklxkd-59/obj46?versionId=null 2023-05-31T20:50:31.078 DEBUG:boto:Data: 2023-05-31T20:50:31.078 DEBUG:boto:Headers: {} 2023-05-31T20:50:31.079 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:50:31.079 DEBUG:boto:Port: 8000 2023-05-31T20:50:31.079 DEBUG:boto:Params: {} 2023-05-31T20:50:31.080 DEBUG:boto:Token: None 2023-05-31T20:50:31.080 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:50:31 GMT /pklxkd-59/obj46?versionId=null 2023-05-31T20:50:31.080 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:miXyQgF6wYSfcNpYsG06HcK69vY= 2023-05-31T20:50:31.081 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:50:31 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:miXyQgF6wYSfcNpYsG06HcK69vY=', 'Content-Length': '0'} 2023-05-31T20:50:31.083 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:46:33 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009cc5067aa3473ce7-006477b317-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:50:31 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:50:31.083 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj46 2023-05-31T20:50:31.084 DEBUG:boto:path=/pklxkd-59/obj46 2023-05-31T20:50:31.084 DEBUG:boto:auth_path=/pklxkd-59/obj46 2023-05-31T20:50:31.084 DEBUG:boto:path=/pklxkd-59/obj46?versionId=null 2023-05-31T20:50:31.085 DEBUG:boto:auth_path=/pklxkd-59/obj46?versionId=null 2023-05-31T20:50:31.085 DEBUG:boto:Method: GET 2023-05-31T20:50:31.085 DEBUG:boto:Path: /pklxkd-59/obj46?versionId=null 2023-05-31T20:50:31.086 DEBUG:boto:Data: 2023-05-31T20:50:31.086 DEBUG:boto:Headers: {} 2023-05-31T20:50:31.086 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:50:31.087 DEBUG:boto:Port: 8001 2023-05-31T20:50:31.087 DEBUG:boto:Params: {} 2023-05-31T20:50:31.088 DEBUG:boto:Token: None 2023-05-31T20:50:31.088 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:50:31 GMT /pklxkd-59/obj46?versionId=null 2023-05-31T20:50:31.088 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:C4aC04pyG8393bhHCv6KwAjSkNU= 2023-05-31T20:50:31.089 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:50:31 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:C4aC04pyG8393bhHCv6KwAjSkNU=', 'Content-Length': '0'} 2023-05-31T20:50:31.091 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:46:33 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ecbd0cc1ac307966-006477b317-1c17-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:50:31 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:50:31.091 DEBUG:boto:path=/pklxkd-59/obj46 2023-05-31T20:50:31.092 DEBUG:boto:auth_path=/pklxkd-59/obj46 2023-05-31T20:50:31.092 DEBUG:boto:path=/pklxkd-59/obj46?versionId=null 2023-05-31T20:50:31.092 DEBUG:boto:auth_path=/pklxkd-59/obj46?versionId=null 2023-05-31T20:50:31.093 DEBUG:boto:Method: GET 2023-05-31T20:50:31.093 DEBUG:boto:Path: /pklxkd-59/obj46?versionId=null 2023-05-31T20:50:31.093 DEBUG:boto:Data: 2023-05-31T20:50:31.094 DEBUG:boto:Headers: {} 2023-05-31T20:50:31.094 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:50:31.095 DEBUG:boto:Port: 8000 2023-05-31T20:50:31.095 DEBUG:boto:Params: {} 2023-05-31T20:50:31.095 DEBUG:boto:Token: None 2023-05-31T20:50:31.096 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:50:31 GMT /pklxkd-59/obj46?versionId=null 2023-05-31T20:50:31.096 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:C4aC04pyG8393bhHCv6KwAjSkNU= 2023-05-31T20:50:31.097 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:50:31 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:C4aC04pyG8393bhHCv6KwAjSkNU=', 'Content-Length': '0'} 2023-05-31T20:50:31.098 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:46:33 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000399398d82be1b670-006477b317-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:50:31 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:50:31.099 DEBUG:boto:path=/pklxkd-59/ 2023-05-31T20:50:31.099 DEBUG:boto:auth_path=/pklxkd-59/ 2023-05-31T20:50:31.100 DEBUG:boto:path=/pklxkd-59/?prefix=obj46 2023-05-31T20:50:31.100 DEBUG:boto:auth_path=/pklxkd-59/?prefix=obj46 2023-05-31T20:50:31.101 DEBUG:boto:Method: GET 2023-05-31T20:50:31.101 DEBUG:boto:Path: /pklxkd-59/?prefix=obj46 2023-05-31T20:50:31.101 DEBUG:boto:Data: 2023-05-31T20:50:31.102 DEBUG:boto:Headers: {} 2023-05-31T20:50:31.102 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:50:31.102 DEBUG:boto:Port: 8001 2023-05-31T20:50:31.103 DEBUG:boto:Params: {} 2023-05-31T20:50:31.103 DEBUG:boto:Token: None 2023-05-31T20:50:31.103 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:50:31 GMT /pklxkd-59/ 2023-05-31T20:50:31.104 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:apPqLCZHOWwza3uFvs+KzJ/Flyg= 2023-05-31T20:50:31.104 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:50:31 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:apPqLCZHOWwza3uFvs+KzJ/Flyg=', 'Content-Length': '0'} 2023-05-31T20:50:31.107 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000005ef4591c7c3cbfbb-006477b317-1c17-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:50:31 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:50:31.107 DEBUG:boto:b'pklxkd-59obj461000falseobj462023-05-31T20:46:33.830Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:50:31.108 DEBUG:boto:path=/pklxkd-59/ 2023-05-31T20:50:31.108 DEBUG:boto:auth_path=/pklxkd-59/ 2023-05-31T20:50:31.109 DEBUG:boto:path=/pklxkd-59/?prefix=obj46 2023-05-31T20:50:31.109 DEBUG:boto:auth_path=/pklxkd-59/?prefix=obj46 2023-05-31T20:50:31.109 DEBUG:boto:Method: GET 2023-05-31T20:50:31.110 DEBUG:boto:Path: /pklxkd-59/?prefix=obj46 2023-05-31T20:50:31.110 DEBUG:boto:Data: 2023-05-31T20:50:31.110 DEBUG:boto:Headers: {} 2023-05-31T20:50:31.111 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:50:31.111 DEBUG:boto:Port: 8000 2023-05-31T20:50:31.111 DEBUG:boto:Params: {} 2023-05-31T20:50:31.112 DEBUG:boto:Token: None 2023-05-31T20:50:31.112 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:50:31 GMT /pklxkd-59/ 2023-05-31T20:50:31.113 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:apPqLCZHOWwza3uFvs+KzJ/Flyg= 2023-05-31T20:50:31.113 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:50:31 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:apPqLCZHOWwza3uFvs+KzJ/Flyg=', 'Content-Length': '0'} 2023-05-31T20:50:31.116 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b4d4cfba2bd30070-006477b317-1630-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:50:31 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:50:31.117 DEBUG:boto:b'pklxkd-59obj461000falseobj462023-05-31T20:46:33.830Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:50:31.117 DEBUG:boto:path=/pklxkd-59/obj46 2023-05-31T20:50:31.118 DEBUG:boto:auth_path=/pklxkd-59/obj46 2023-05-31T20:50:31.118 DEBUG:boto:Method: HEAD 2023-05-31T20:50:31.119 DEBUG:boto:Path: /pklxkd-59/obj46 2023-05-31T20:50:31.119 DEBUG:boto:Data: 2023-05-31T20:50:31.119 DEBUG:boto:Headers: {} 2023-05-31T20:50:31.120 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:50:31.120 DEBUG:boto:Port: 8001 2023-05-31T20:50:31.120 DEBUG:boto:Params: {} 2023-05-31T20:50:31.121 DEBUG:boto:Token: None 2023-05-31T20:50:31.121 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:50:31 GMT /pklxkd-59/obj46 2023-05-31T20:50:31.121 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:EU4P3WNj+VRdJBJ4UrX6PIEEbf0= 2023-05-31T20:50:31.122 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:50:31 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:EU4P3WNj+VRdJBJ4UrX6PIEEbf0=', 'Content-Length': '0'} 2023-05-31T20:50:31.124 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:46:33 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000079c1fea077618107-006477b317-1c17-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:50:31 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:50:31.124 DEBUG:boto:path=/pklxkd-59/obj46 2023-05-31T20:50:31.125 DEBUG:boto:auth_path=/pklxkd-59/obj46 2023-05-31T20:50:31.125 DEBUG:boto:Method: HEAD 2023-05-31T20:50:31.125 DEBUG:boto:Path: /pklxkd-59/obj46 2023-05-31T20:50:31.126 DEBUG:boto:Data: 2023-05-31T20:50:31.126 DEBUG:boto:Headers: {} 2023-05-31T20:50:31.126 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:50:31.127 DEBUG:boto:Port: 8000 2023-05-31T20:50:31.127 DEBUG:boto:Params: {} 2023-05-31T20:50:31.127 DEBUG:boto:Token: None 2023-05-31T20:50:31.128 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:50:31 GMT /pklxkd-59/obj46 2023-05-31T20:50:31.128 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:EU4P3WNj+VRdJBJ4UrX6PIEEbf0= 2023-05-31T20:50:31.128 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:50:31 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:EU4P3WNj+VRdJBJ4UrX6PIEEbf0=', 'Content-Length': '0'} 2023-05-31T20:50:31.130 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:46:33 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007477199f1c8b84aa-006477b317-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:50:31 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:50:31.131 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj46 2023-05-31T20:50:31.131 DEBUG:boto:path=/pklxkd-59/obj46 2023-05-31T20:50:31.131 DEBUG:boto:auth_path=/pklxkd-59/obj46 2023-05-31T20:50:31.132 DEBUG:boto:Method: GET 2023-05-31T20:50:31.132 DEBUG:boto:Path: /pklxkd-59/obj46 2023-05-31T20:50:31.132 DEBUG:boto:Data: 2023-05-31T20:50:31.133 DEBUG:boto:Headers: {} 2023-05-31T20:50:31.133 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:50:31.133 DEBUG:boto:Port: 8001 2023-05-31T20:50:31.134 DEBUG:boto:Params: {} 2023-05-31T20:50:31.134 DEBUG:boto:Token: None 2023-05-31T20:50:31.135 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:50:31 GMT /pklxkd-59/obj46 2023-05-31T20:50:31.135 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:xkxLOskFdQByfODwg/JAXJqcrLg= 2023-05-31T20:50:31.135 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:50:31 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:xkxLOskFdQByfODwg/JAXJqcrLg=', 'Content-Length': '0'} 2023-05-31T20:50:31.137 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:46:33 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e3d4eee7d1d1413f-006477b317-1c17-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:50:31 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:50:31.138 DEBUG:boto:path=/pklxkd-59/obj46 2023-05-31T20:50:31.138 DEBUG:boto:auth_path=/pklxkd-59/obj46 2023-05-31T20:50:31.139 DEBUG:boto:Method: GET 2023-05-31T20:50:31.139 DEBUG:boto:Path: /pklxkd-59/obj46 2023-05-31T20:50:31.139 DEBUG:boto:Data: 2023-05-31T20:50:31.140 DEBUG:boto:Headers: {} 2023-05-31T20:50:31.140 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:50:31.140 DEBUG:boto:Port: 8000 2023-05-31T20:50:31.141 DEBUG:boto:Params: {} 2023-05-31T20:50:31.141 DEBUG:boto:Token: None 2023-05-31T20:50:31.142 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:50:31 GMT /pklxkd-59/obj46 2023-05-31T20:50:31.142 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:xkxLOskFdQByfODwg/JAXJqcrLg= 2023-05-31T20:50:31.142 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:50:31 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:xkxLOskFdQByfODwg/JAXJqcrLg=', 'Content-Length': '0'} 2023-05-31T20:50:31.144 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:46:33 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e02d370ef0420dc5-006477b317-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:50:31 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:50:31.145 DEBUG:boto:path=/pklxkd-59/ 2023-05-31T20:50:31.145 DEBUG:boto:auth_path=/pklxkd-59/ 2023-05-31T20:50:31.145 DEBUG:boto:path=/pklxkd-59/?prefix=obj46 2023-05-31T20:50:31.146 DEBUG:boto:auth_path=/pklxkd-59/?prefix=obj46 2023-05-31T20:50:31.146 DEBUG:boto:Method: GET 2023-05-31T20:50:31.147 DEBUG:boto:Path: /pklxkd-59/?prefix=obj46 2023-05-31T20:50:31.147 DEBUG:boto:Data: 2023-05-31T20:50:31.147 DEBUG:boto:Headers: {} 2023-05-31T20:50:31.148 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:50:31.148 DEBUG:boto:Port: 8001 2023-05-31T20:50:31.148 DEBUG:boto:Params: {} 2023-05-31T20:50:31.149 DEBUG:boto:Token: None 2023-05-31T20:50:31.149 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:50:31 GMT /pklxkd-59/ 2023-05-31T20:50:31.150 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:apPqLCZHOWwza3uFvs+KzJ/Flyg= 2023-05-31T20:50:31.150 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:50:31 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:apPqLCZHOWwza3uFvs+KzJ/Flyg=', 'Content-Length': '0'} 2023-05-31T20:50:31.152 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ecc5c420248ad640-006477b317-1c17-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:50:31 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:50:31.153 DEBUG:boto:b'pklxkd-59obj461000falseobj462023-05-31T20:46:33.830Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:50:31.153 DEBUG:boto:path=/pklxkd-59/ 2023-05-31T20:50:31.154 DEBUG:boto:auth_path=/pklxkd-59/ 2023-05-31T20:50:31.154 DEBUG:boto:path=/pklxkd-59/?prefix=obj46 2023-05-31T20:50:31.155 DEBUG:boto:auth_path=/pklxkd-59/?prefix=obj46 2023-05-31T20:50:31.155 DEBUG:boto:Method: GET 2023-05-31T20:50:31.155 DEBUG:boto:Path: /pklxkd-59/?prefix=obj46 2023-05-31T20:50:31.156 DEBUG:boto:Data: 2023-05-31T20:50:31.156 DEBUG:boto:Headers: {} 2023-05-31T20:50:31.156 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:50:31.157 DEBUG:boto:Port: 8000 2023-05-31T20:50:31.157 DEBUG:boto:Params: {} 2023-05-31T20:50:31.157 DEBUG:boto:Token: None 2023-05-31T20:50:31.158 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:50:31 GMT /pklxkd-59/ 2023-05-31T20:50:31.158 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:apPqLCZHOWwza3uFvs+KzJ/Flyg= 2023-05-31T20:50:31.159 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:50:31 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:apPqLCZHOWwza3uFvs+KzJ/Flyg=', 'Content-Length': '0'} 2023-05-31T20:50:31.162 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000008c1e51138442c567-006477b317-1630-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:50:31 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:50:31.204 DEBUG:boto:b'pklxkd-59obj461000falseobj462023-05-31T20:46:33.830Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:50:31.205 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj51 2023-05-31T20:50:31.205 DEBUG:boto:path=/pklxkd-59/obj51 2023-05-31T20:50:31.206 DEBUG:boto:auth_path=/pklxkd-59/obj51 2023-05-31T20:50:31.206 DEBUG:boto:path=/pklxkd-59/obj51?versionId=null 2023-05-31T20:50:31.206 DEBUG:boto:auth_path=/pklxkd-59/obj51?versionId=null 2023-05-31T20:50:31.207 DEBUG:boto:Method: GET 2023-05-31T20:50:31.207 DEBUG:boto:Path: /pklxkd-59/obj51?versionId=null 2023-05-31T20:50:31.207 DEBUG:boto:Data: 2023-05-31T20:50:31.208 DEBUG:boto:Headers: {} 2023-05-31T20:50:31.208 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:50:31.209 DEBUG:boto:Port: 8001 2023-05-31T20:50:31.209 DEBUG:boto:Params: {} 2023-05-31T20:50:31.209 DEBUG:boto:Token: None 2023-05-31T20:50:31.210 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:50:31 GMT /pklxkd-59/obj51?versionId=null 2023-05-31T20:50:31.210 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:QFvuhsxbVTIO987H8CuTAlsv+EY= 2023-05-31T20:50:31.211 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:50:31 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:QFvuhsxbVTIO987H8CuTAlsv+EY=', 'Content-Length': '0'} 2023-05-31T20:50:31.213 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:43:08 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a08c9be9dc703fdd-006477b317-1c17-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:50:31 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:50:31.213 DEBUG:boto:path=/pklxkd-59/obj51 2023-05-31T20:50:31.214 DEBUG:boto:auth_path=/pklxkd-59/obj51 2023-05-31T20:50:31.214 DEBUG:boto:path=/pklxkd-59/obj51?versionId=null 2023-05-31T20:50:31.214 DEBUG:boto:auth_path=/pklxkd-59/obj51?versionId=null 2023-05-31T20:50:31.215 DEBUG:boto:Method: GET 2023-05-31T20:50:31.215 DEBUG:boto:Path: /pklxkd-59/obj51?versionId=null 2023-05-31T20:50:31.215 DEBUG:boto:Data: 2023-05-31T20:50:31.216 DEBUG:boto:Headers: {} 2023-05-31T20:50:31.216 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:50:31.216 DEBUG:boto:Port: 8000 2023-05-31T20:50:31.217 DEBUG:boto:Params: {} 2023-05-31T20:50:31.217 DEBUG:boto:Token: None 2023-05-31T20:50:31.218 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:50:31 GMT /pklxkd-59/obj51?versionId=null 2023-05-31T20:50:31.218 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:QFvuhsxbVTIO987H8CuTAlsv+EY= 2023-05-31T20:50:31.218 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:50:31 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:QFvuhsxbVTIO987H8CuTAlsv+EY=', 'Content-Length': '0'} 2023-05-31T20:50:31.220 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:43:08 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a22412c15408e24a-006477b317-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:50:31 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:50:31.221 DEBUG:boto:path=/pklxkd-59/obj51 2023-05-31T20:50:31.221 DEBUG:boto:auth_path=/pklxkd-59/obj51 2023-05-31T20:50:31.222 DEBUG:boto:path=/pklxkd-59/obj51?versionId=null 2023-05-31T20:50:31.222 DEBUG:boto:auth_path=/pklxkd-59/obj51?versionId=null 2023-05-31T20:50:31.222 DEBUG:boto:Method: HEAD 2023-05-31T20:50:31.223 DEBUG:boto:Path: /pklxkd-59/obj51?versionId=null 2023-05-31T20:50:31.223 DEBUG:boto:Data: 2023-05-31T20:50:31.223 DEBUG:boto:Headers: {} 2023-05-31T20:50:31.224 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:50:31.224 DEBUG:boto:Port: 8001 2023-05-31T20:50:31.224 DEBUG:boto:Params: {} 2023-05-31T20:50:31.225 DEBUG:boto:Token: None 2023-05-31T20:50:31.225 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:50:31 GMT /pklxkd-59/obj51?versionId=null 2023-05-31T20:50:31.226 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:3AXQJu9RvhNHWn3s6Ccq7Rkye4M= 2023-05-31T20:50:31.226 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:50:31 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:3AXQJu9RvhNHWn3s6Ccq7Rkye4M=', 'Content-Length': '0'} 2023-05-31T20:50:31.228 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:43:08 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000afc25b2842d6d4ae-006477b317-1c17-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:50:31 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:50:31.228 DEBUG:boto:path=/pklxkd-59/obj51 2023-05-31T20:50:31.229 DEBUG:boto:auth_path=/pklxkd-59/obj51 2023-05-31T20:50:31.229 DEBUG:boto:path=/pklxkd-59/obj51?versionId=null 2023-05-31T20:50:31.229 DEBUG:boto:auth_path=/pklxkd-59/obj51?versionId=null 2023-05-31T20:50:31.230 DEBUG:boto:Method: HEAD 2023-05-31T20:50:31.230 DEBUG:boto:Path: /pklxkd-59/obj51?versionId=null 2023-05-31T20:50:31.230 DEBUG:boto:Data: 2023-05-31T20:50:31.231 DEBUG:boto:Headers: {} 2023-05-31T20:50:31.231 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:50:31.231 DEBUG:boto:Port: 8000 2023-05-31T20:50:31.232 DEBUG:boto:Params: {} 2023-05-31T20:50:31.232 DEBUG:boto:Token: None 2023-05-31T20:50:31.233 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:50:31 GMT /pklxkd-59/obj51?versionId=null 2023-05-31T20:50:31.233 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:3AXQJu9RvhNHWn3s6Ccq7Rkye4M= 2023-05-31T20:50:31.233 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:50:31 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:3AXQJu9RvhNHWn3s6Ccq7Rkye4M=', 'Content-Length': '0'} 2023-05-31T20:50:31.235 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:43:08 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000899c220aac8f8f3a-006477b317-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:50:31 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:50:31.236 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj51 2023-05-31T20:50:31.236 DEBUG:boto:path=/pklxkd-59/obj51 2023-05-31T20:50:31.236 DEBUG:boto:auth_path=/pklxkd-59/obj51 2023-05-31T20:50:31.237 DEBUG:boto:path=/pklxkd-59/obj51?versionId=null 2023-05-31T20:50:31.237 DEBUG:boto:auth_path=/pklxkd-59/obj51?versionId=null 2023-05-31T20:50:31.237 DEBUG:boto:Method: GET 2023-05-31T20:50:31.238 DEBUG:boto:Path: /pklxkd-59/obj51?versionId=null 2023-05-31T20:50:31.238 DEBUG:boto:Data: 2023-05-31T20:50:31.238 DEBUG:boto:Headers: {} 2023-05-31T20:50:31.239 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:50:31.239 DEBUG:boto:Port: 8001 2023-05-31T20:50:31.239 DEBUG:boto:Params: {} 2023-05-31T20:50:31.240 DEBUG:boto:Token: None 2023-05-31T20:50:31.240 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:50:31 GMT /pklxkd-59/obj51?versionId=null 2023-05-31T20:50:31.241 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:QFvuhsxbVTIO987H8CuTAlsv+EY= 2023-05-31T20:50:31.241 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:50:31 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:QFvuhsxbVTIO987H8CuTAlsv+EY=', 'Content-Length': '0'} 2023-05-31T20:50:31.243 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:43:08 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c20c6d9b73b15755-006477b317-1c17-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:50:31 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:50:31.284 DEBUG:boto:path=/pklxkd-59/obj51 2023-05-31T20:50:31.285 DEBUG:boto:auth_path=/pklxkd-59/obj51 2023-05-31T20:50:31.285 DEBUG:boto:path=/pklxkd-59/obj51?versionId=null 2023-05-31T20:50:31.285 DEBUG:boto:auth_path=/pklxkd-59/obj51?versionId=null 2023-05-31T20:50:31.286 DEBUG:boto:Method: GET 2023-05-31T20:50:31.286 DEBUG:boto:Path: /pklxkd-59/obj51?versionId=null 2023-05-31T20:50:31.286 DEBUG:boto:Data: 2023-05-31T20:50:31.287 DEBUG:boto:Headers: {} 2023-05-31T20:50:31.287 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:50:31.288 DEBUG:boto:Port: 8000 2023-05-31T20:50:31.288 DEBUG:boto:Params: {} 2023-05-31T20:50:31.288 DEBUG:boto:Token: None 2023-05-31T20:50:31.289 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:50:31 GMT /pklxkd-59/obj51?versionId=null 2023-05-31T20:50:31.289 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:QFvuhsxbVTIO987H8CuTAlsv+EY= 2023-05-31T20:50:31.290 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:50:31 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:QFvuhsxbVTIO987H8CuTAlsv+EY=', 'Content-Length': '0'} 2023-05-31T20:50:31.292 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:43:08 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ca6ebdd42f29bb69-006477b317-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:50:31 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:50:31.292 DEBUG:boto:path=/pklxkd-59/ 2023-05-31T20:50:31.293 DEBUG:boto:auth_path=/pklxkd-59/ 2023-05-31T20:50:31.293 DEBUG:boto:path=/pklxkd-59/?prefix=obj51 2023-05-31T20:50:31.294 DEBUG:boto:auth_path=/pklxkd-59/?prefix=obj51 2023-05-31T20:50:31.294 DEBUG:boto:Method: GET 2023-05-31T20:50:31.294 DEBUG:boto:Path: /pklxkd-59/?prefix=obj51 2023-05-31T20:50:31.295 DEBUG:boto:Data: 2023-05-31T20:50:31.295 DEBUG:boto:Headers: {} 2023-05-31T20:50:31.295 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:50:31.296 DEBUG:boto:Port: 8001 2023-05-31T20:50:31.296 DEBUG:boto:Params: {} 2023-05-31T20:50:31.296 DEBUG:boto:Token: None 2023-05-31T20:50:31.297 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:50:31 GMT /pklxkd-59/ 2023-05-31T20:50:31.297 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:apPqLCZHOWwza3uFvs+KzJ/Flyg= 2023-05-31T20:50:31.298 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:50:31 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:apPqLCZHOWwza3uFvs+KzJ/Flyg=', 'Content-Length': '0'} 2023-05-31T20:50:31.300 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000724a7b8326bd7a57-006477b317-1c17-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:50:31 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:50:31.301 DEBUG:boto:b'pklxkd-59obj511000falseobj512023-05-31T20:43:08.011Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:50:31.301 DEBUG:boto:path=/pklxkd-59/ 2023-05-31T20:50:31.302 DEBUG:boto:auth_path=/pklxkd-59/ 2023-05-31T20:50:31.302 DEBUG:boto:path=/pklxkd-59/?prefix=obj51 2023-05-31T20:50:31.302 DEBUG:boto:auth_path=/pklxkd-59/?prefix=obj51 2023-05-31T20:50:31.303 DEBUG:boto:Method: GET 2023-05-31T20:50:31.303 DEBUG:boto:Path: /pklxkd-59/?prefix=obj51 2023-05-31T20:50:31.303 DEBUG:boto:Data: 2023-05-31T20:50:31.304 DEBUG:boto:Headers: {} 2023-05-31T20:50:31.304 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:50:31.304 DEBUG:boto:Port: 8000 2023-05-31T20:50:31.305 DEBUG:boto:Params: {} 2023-05-31T20:50:31.305 DEBUG:boto:Token: None 2023-05-31T20:50:31.306 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:50:31 GMT /pklxkd-59/ 2023-05-31T20:50:31.306 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:apPqLCZHOWwza3uFvs+KzJ/Flyg= 2023-05-31T20:50:31.306 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:50:31 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:apPqLCZHOWwza3uFvs+KzJ/Flyg=', 'Content-Length': '0'} 2023-05-31T20:50:31.310 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000be7eeadf02e2d20c-006477b317-1630-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:50:31 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:50:31.310 DEBUG:boto:b'pklxkd-59obj511000falseobj512023-05-31T20:43:08.011Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:50:31.311 DEBUG:boto:path=/pklxkd-59/obj51 2023-05-31T20:50:31.311 DEBUG:boto:auth_path=/pklxkd-59/obj51 2023-05-31T20:50:31.312 DEBUG:boto:Method: HEAD 2023-05-31T20:50:31.312 DEBUG:boto:Path: /pklxkd-59/obj51 2023-05-31T20:50:31.312 DEBUG:boto:Data: 2023-05-31T20:50:31.313 DEBUG:boto:Headers: {} 2023-05-31T20:50:31.313 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:50:31.313 DEBUG:boto:Port: 8001 2023-05-31T20:50:31.314 DEBUG:boto:Params: {} 2023-05-31T20:50:31.314 DEBUG:boto:Token: None 2023-05-31T20:50:31.315 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:50:31 GMT /pklxkd-59/obj51 2023-05-31T20:50:31.315 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:ywktc7bQlPXaB2Es8LAX1AVUVWc= 2023-05-31T20:50:31.315 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:50:31 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:ywktc7bQlPXaB2Es8LAX1AVUVWc=', 'Content-Length': '0'} 2023-05-31T20:50:31.317 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:43:08 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000d736691f0d8aba6-006477b317-1c17-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:50:31 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:50:31.318 DEBUG:boto:path=/pklxkd-59/obj51 2023-05-31T20:50:31.318 DEBUG:boto:auth_path=/pklxkd-59/obj51 2023-05-31T20:50:31.318 DEBUG:boto:Method: HEAD 2023-05-31T20:50:31.319 DEBUG:boto:Path: /pklxkd-59/obj51 2023-05-31T20:50:31.319 DEBUG:boto:Data: 2023-05-31T20:50:31.319 DEBUG:boto:Headers: {} 2023-05-31T20:50:31.320 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:50:31.320 DEBUG:boto:Port: 8000 2023-05-31T20:50:31.320 DEBUG:boto:Params: {} 2023-05-31T20:50:31.321 DEBUG:boto:Token: None 2023-05-31T20:50:31.321 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:50:31 GMT /pklxkd-59/obj51 2023-05-31T20:50:31.322 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:ywktc7bQlPXaB2Es8LAX1AVUVWc= 2023-05-31T20:50:31.322 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:50:31 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:ywktc7bQlPXaB2Es8LAX1AVUVWc=', 'Content-Length': '0'} 2023-05-31T20:50:31.324 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:43:08 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f5fa2c661f0cd5ed-006477b317-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:50:31 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:50:31.324 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj51 2023-05-31T20:50:31.325 DEBUG:boto:path=/pklxkd-59/obj51 2023-05-31T20:50:31.325 DEBUG:boto:auth_path=/pklxkd-59/obj51 2023-05-31T20:50:31.325 DEBUG:boto:Method: GET 2023-05-31T20:50:31.326 DEBUG:boto:Path: /pklxkd-59/obj51 2023-05-31T20:50:31.326 DEBUG:boto:Data: 2023-05-31T20:50:31.326 DEBUG:boto:Headers: {} 2023-05-31T20:50:31.327 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:50:31.327 DEBUG:boto:Port: 8001 2023-05-31T20:50:31.327 DEBUG:boto:Params: {} 2023-05-31T20:50:31.328 DEBUG:boto:Token: None 2023-05-31T20:50:31.328 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:50:31 GMT /pklxkd-59/obj51 2023-05-31T20:50:31.329 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:3jXZWtXw3fk7n9Q0SC3XyGWSihc= 2023-05-31T20:50:31.329 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:50:31 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:3jXZWtXw3fk7n9Q0SC3XyGWSihc=', 'Content-Length': '0'} 2023-05-31T20:50:31.331 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:43:08 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007b8db5953ae3c4b1-006477b317-1c17-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:50:31 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:50:31.376 DEBUG:boto:path=/pklxkd-59/obj51 2023-05-31T20:50:31.377 DEBUG:boto:auth_path=/pklxkd-59/obj51 2023-05-31T20:50:31.377 DEBUG:boto:Method: GET 2023-05-31T20:50:31.377 DEBUG:boto:Path: /pklxkd-59/obj51 2023-05-31T20:50:31.378 DEBUG:boto:Data: 2023-05-31T20:50:31.378 DEBUG:boto:Headers: {} 2023-05-31T20:50:31.378 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:50:31.379 DEBUG:boto:Port: 8000 2023-05-31T20:50:31.379 DEBUG:boto:Params: {} 2023-05-31T20:50:31.380 DEBUG:boto:Token: None 2023-05-31T20:50:31.380 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:50:31 GMT /pklxkd-59/obj51 2023-05-31T20:50:31.380 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:3jXZWtXw3fk7n9Q0SC3XyGWSihc= 2023-05-31T20:50:31.381 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:50:31 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:3jXZWtXw3fk7n9Q0SC3XyGWSihc=', 'Content-Length': '0'} 2023-05-31T20:50:31.383 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:43:08 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000044f8aa362efc22f3-006477b317-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:50:31 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:50:31.384 DEBUG:boto:path=/pklxkd-59/ 2023-05-31T20:50:31.384 DEBUG:boto:auth_path=/pklxkd-59/ 2023-05-31T20:50:31.384 DEBUG:boto:path=/pklxkd-59/?prefix=obj51 2023-05-31T20:50:31.385 DEBUG:boto:auth_path=/pklxkd-59/?prefix=obj51 2023-05-31T20:50:31.385 DEBUG:boto:Method: GET 2023-05-31T20:50:31.385 DEBUG:boto:Path: /pklxkd-59/?prefix=obj51 2023-05-31T20:50:31.386 DEBUG:boto:Data: 2023-05-31T20:50:31.386 DEBUG:boto:Headers: {} 2023-05-31T20:50:31.386 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:50:31.387 DEBUG:boto:Port: 8001 2023-05-31T20:50:31.387 DEBUG:boto:Params: {} 2023-05-31T20:50:31.388 DEBUG:boto:Token: None 2023-05-31T20:50:31.388 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:50:31 GMT /pklxkd-59/ 2023-05-31T20:50:31.388 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:apPqLCZHOWwza3uFvs+KzJ/Flyg= 2023-05-31T20:50:31.389 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:50:31 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:apPqLCZHOWwza3uFvs+KzJ/Flyg=', 'Content-Length': '0'} 2023-05-31T20:50:31.391 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000061af5e71fa074f65-006477b317-1c17-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:50:31 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:50:31.392 DEBUG:boto:b'pklxkd-59obj511000falseobj512023-05-31T20:43:08.011Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:50:31.393 DEBUG:boto:path=/pklxkd-59/ 2023-05-31T20:50:31.393 DEBUG:boto:auth_path=/pklxkd-59/ 2023-05-31T20:50:31.393 DEBUG:boto:path=/pklxkd-59/?prefix=obj51 2023-05-31T20:50:31.394 DEBUG:boto:auth_path=/pklxkd-59/?prefix=obj51 2023-05-31T20:50:31.394 DEBUG:boto:Method: GET 2023-05-31T20:50:31.394 DEBUG:boto:Path: /pklxkd-59/?prefix=obj51 2023-05-31T20:50:31.395 DEBUG:boto:Data: 2023-05-31T20:50:31.395 DEBUG:boto:Headers: {} 2023-05-31T20:50:31.395 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:50:31.396 DEBUG:boto:Port: 8000 2023-05-31T20:50:31.396 DEBUG:boto:Params: {} 2023-05-31T20:50:31.397 DEBUG:boto:Token: None 2023-05-31T20:50:31.397 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:50:31 GMT /pklxkd-59/ 2023-05-31T20:50:31.397 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:apPqLCZHOWwza3uFvs+KzJ/Flyg= 2023-05-31T20:50:31.398 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:50:31 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:apPqLCZHOWwza3uFvs+KzJ/Flyg=', 'Content-Length': '0'} 2023-05-31T20:50:31.401 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000055b79183b687faad-006477b317-1630-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:50:31 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:50:31.401 DEBUG:boto:b'pklxkd-59obj511000falseobj512023-05-31T20:43:08.011Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:50:31.402 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj55 2023-05-31T20:50:31.402 DEBUG:boto:path=/pklxkd-59/obj55 2023-05-31T20:50:31.403 DEBUG:boto:auth_path=/pklxkd-59/obj55 2023-05-31T20:50:31.403 DEBUG:boto:path=/pklxkd-59/obj55?versionId=null 2023-05-31T20:50:31.403 DEBUG:boto:auth_path=/pklxkd-59/obj55?versionId=null 2023-05-31T20:50:31.404 DEBUG:boto:Method: GET 2023-05-31T20:50:31.404 DEBUG:boto:Path: /pklxkd-59/obj55?versionId=null 2023-05-31T20:50:31.405 DEBUG:boto:Data: 2023-05-31T20:50:31.405 DEBUG:boto:Headers: {} 2023-05-31T20:50:31.405 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:50:31.406 DEBUG:boto:Port: 8001 2023-05-31T20:50:31.406 DEBUG:boto:Params: {} 2023-05-31T20:50:31.406 DEBUG:boto:Token: None 2023-05-31T20:50:31.407 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:50:31 GMT /pklxkd-59/obj55?versionId=null 2023-05-31T20:50:31.407 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:CL2/VWuo2yJSPc7M/FfjDid+Fpg= 2023-05-31T20:50:31.408 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:50:31 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:CL2/VWuo2yJSPc7M/FfjDid+Fpg=', 'Content-Length': '0'} 2023-05-31T20:50:31.409 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:39:26 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ee1576284daea78e-006477b317-1c17-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:50:31 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:50:31.410 DEBUG:boto:path=/pklxkd-59/obj55 2023-05-31T20:50:31.411 DEBUG:boto:auth_path=/pklxkd-59/obj55 2023-05-31T20:50:31.411 DEBUG:boto:path=/pklxkd-59/obj55?versionId=null 2023-05-31T20:50:31.411 DEBUG:boto:auth_path=/pklxkd-59/obj55?versionId=null 2023-05-31T20:50:31.412 DEBUG:boto:Method: GET 2023-05-31T20:50:31.412 DEBUG:boto:Path: /pklxkd-59/obj55?versionId=null 2023-05-31T20:50:31.412 DEBUG:boto:Data: 2023-05-31T20:50:31.413 DEBUG:boto:Headers: {} 2023-05-31T20:50:31.413 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:50:31.413 DEBUG:boto:Port: 8000 2023-05-31T20:50:31.414 DEBUG:boto:Params: {} 2023-05-31T20:50:31.414 DEBUG:boto:Token: None 2023-05-31T20:50:31.415 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:50:31 GMT /pklxkd-59/obj55?versionId=null 2023-05-31T20:50:31.415 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:CL2/VWuo2yJSPc7M/FfjDid+Fpg= 2023-05-31T20:50:31.415 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:50:31 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:CL2/VWuo2yJSPc7M/FfjDid+Fpg=', 'Content-Length': '0'} 2023-05-31T20:50:31.417 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:39:26 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e25f0c3312bb5a13-006477b317-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:50:31 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:50:31.418 DEBUG:boto:path=/pklxkd-59/obj55 2023-05-31T20:50:31.418 DEBUG:boto:auth_path=/pklxkd-59/obj55 2023-05-31T20:50:31.419 DEBUG:boto:path=/pklxkd-59/obj55?versionId=null 2023-05-31T20:50:31.419 DEBUG:boto:auth_path=/pklxkd-59/obj55?versionId=null 2023-05-31T20:50:31.419 DEBUG:boto:Method: HEAD 2023-05-31T20:50:31.420 DEBUG:boto:Path: /pklxkd-59/obj55?versionId=null 2023-05-31T20:50:31.420 DEBUG:boto:Data: 2023-05-31T20:50:31.420 DEBUG:boto:Headers: {} 2023-05-31T20:50:31.421 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:50:31.421 DEBUG:boto:Port: 8001 2023-05-31T20:50:31.421 DEBUG:boto:Params: {} 2023-05-31T20:50:31.422 DEBUG:boto:Token: None 2023-05-31T20:50:31.422 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:50:31 GMT /pklxkd-59/obj55?versionId=null 2023-05-31T20:50:31.422 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:sCFpzObT3EixsPqTC4uYWldNFtA= 2023-05-31T20:50:31.423 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:50:31 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:sCFpzObT3EixsPqTC4uYWldNFtA=', 'Content-Length': '0'} 2023-05-31T20:50:31.424 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:39:26 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002896c549e28a9252-006477b317-1c17-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:50:31 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:50:31.425 DEBUG:boto:path=/pklxkd-59/obj55 2023-05-31T20:50:31.425 DEBUG:boto:auth_path=/pklxkd-59/obj55 2023-05-31T20:50:31.426 DEBUG:boto:path=/pklxkd-59/obj55?versionId=null 2023-05-31T20:50:31.426 DEBUG:boto:auth_path=/pklxkd-59/obj55?versionId=null 2023-05-31T20:50:31.426 DEBUG:boto:Method: HEAD 2023-05-31T20:50:31.427 DEBUG:boto:Path: /pklxkd-59/obj55?versionId=null 2023-05-31T20:50:31.427 DEBUG:boto:Data: 2023-05-31T20:50:31.427 DEBUG:boto:Headers: {} 2023-05-31T20:50:31.428 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:50:31.428 DEBUG:boto:Port: 8000 2023-05-31T20:50:31.428 DEBUG:boto:Params: {} 2023-05-31T20:50:31.429 DEBUG:boto:Token: None 2023-05-31T20:50:31.429 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:50:31 GMT /pklxkd-59/obj55?versionId=null 2023-05-31T20:50:31.429 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:sCFpzObT3EixsPqTC4uYWldNFtA= 2023-05-31T20:50:31.430 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:50:31 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:sCFpzObT3EixsPqTC4uYWldNFtA=', 'Content-Length': '0'} 2023-05-31T20:50:31.431 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:39:26 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d969047d15aab2a0-006477b317-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:50:31 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:50:31.432 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj55 2023-05-31T20:50:31.432 DEBUG:boto:path=/pklxkd-59/obj55 2023-05-31T20:50:31.433 DEBUG:boto:auth_path=/pklxkd-59/obj55 2023-05-31T20:50:31.433 DEBUG:boto:path=/pklxkd-59/obj55?versionId=null 2023-05-31T20:50:31.434 DEBUG:boto:auth_path=/pklxkd-59/obj55?versionId=null 2023-05-31T20:50:31.434 DEBUG:boto:Method: GET 2023-05-31T20:50:31.434 DEBUG:boto:Path: /pklxkd-59/obj55?versionId=null 2023-05-31T20:50:31.435 DEBUG:boto:Data: 2023-05-31T20:50:31.435 DEBUG:boto:Headers: {} 2023-05-31T20:50:31.435 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:50:31.436 DEBUG:boto:Port: 8001 2023-05-31T20:50:31.436 DEBUG:boto:Params: {} 2023-05-31T20:50:31.436 DEBUG:boto:Token: None 2023-05-31T20:50:31.437 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:50:31 GMT /pklxkd-59/obj55?versionId=null 2023-05-31T20:50:31.437 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:CL2/VWuo2yJSPc7M/FfjDid+Fpg= 2023-05-31T20:50:31.438 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:50:31 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:CL2/VWuo2yJSPc7M/FfjDid+Fpg=', 'Content-Length': '0'} 2023-05-31T20:50:31.439 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:39:26 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007b31eab12788726b-006477b317-1c17-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:50:31 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:50:31.440 DEBUG:boto:path=/pklxkd-59/obj55 2023-05-31T20:50:31.440 DEBUG:boto:auth_path=/pklxkd-59/obj55 2023-05-31T20:50:31.441 DEBUG:boto:path=/pklxkd-59/obj55?versionId=null 2023-05-31T20:50:31.441 DEBUG:boto:auth_path=/pklxkd-59/obj55?versionId=null 2023-05-31T20:50:31.441 DEBUG:boto:Method: GET 2023-05-31T20:50:31.442 DEBUG:boto:Path: /pklxkd-59/obj55?versionId=null 2023-05-31T20:50:31.442 DEBUG:boto:Data: 2023-05-31T20:50:31.442 DEBUG:boto:Headers: {} 2023-05-31T20:50:31.443 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:50:31.443 DEBUG:boto:Port: 8000 2023-05-31T20:50:31.443 DEBUG:boto:Params: {} 2023-05-31T20:50:31.444 DEBUG:boto:Token: None 2023-05-31T20:50:31.444 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:50:31 GMT /pklxkd-59/obj55?versionId=null 2023-05-31T20:50:31.445 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:CL2/VWuo2yJSPc7M/FfjDid+Fpg= 2023-05-31T20:50:31.445 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:50:31 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:CL2/VWuo2yJSPc7M/FfjDid+Fpg=', 'Content-Length': '0'} 2023-05-31T20:50:31.447 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:39:26 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002466fb5438614aea-006477b317-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:50:31 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:50:31.447 DEBUG:boto:path=/pklxkd-59/ 2023-05-31T20:50:31.448 DEBUG:boto:auth_path=/pklxkd-59/ 2023-05-31T20:50:31.448 DEBUG:boto:path=/pklxkd-59/?prefix=obj55 2023-05-31T20:50:31.449 DEBUG:boto:auth_path=/pklxkd-59/?prefix=obj55 2023-05-31T20:50:31.449 DEBUG:boto:Method: GET 2023-05-31T20:50:31.449 DEBUG:boto:Path: /pklxkd-59/?prefix=obj55 2023-05-31T20:50:31.450 DEBUG:boto:Data: 2023-05-31T20:50:31.450 DEBUG:boto:Headers: {} 2023-05-31T20:50:31.450 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:50:31.451 DEBUG:boto:Port: 8001 2023-05-31T20:50:31.451 DEBUG:boto:Params: {} 2023-05-31T20:50:31.451 DEBUG:boto:Token: None 2023-05-31T20:50:31.452 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:50:31 GMT /pklxkd-59/ 2023-05-31T20:50:31.452 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:apPqLCZHOWwza3uFvs+KzJ/Flyg= 2023-05-31T20:50:31.453 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:50:31 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:apPqLCZHOWwza3uFvs+KzJ/Flyg=', 'Content-Length': '0'} 2023-05-31T20:50:31.455 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000001ed45df41577a773-006477b317-1c17-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:50:31 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:50:31.455 DEBUG:boto:b'pklxkd-59obj551000falseobj552023-05-31T20:39:26.630Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:50:31.456 DEBUG:boto:path=/pklxkd-59/ 2023-05-31T20:50:31.457 DEBUG:boto:auth_path=/pklxkd-59/ 2023-05-31T20:50:31.457 DEBUG:boto:path=/pklxkd-59/?prefix=obj55 2023-05-31T20:50:31.457 DEBUG:boto:auth_path=/pklxkd-59/?prefix=obj55 2023-05-31T20:50:31.458 DEBUG:boto:Method: GET 2023-05-31T20:50:31.458 DEBUG:boto:Path: /pklxkd-59/?prefix=obj55 2023-05-31T20:50:31.458 DEBUG:boto:Data: 2023-05-31T20:50:31.459 DEBUG:boto:Headers: {} 2023-05-31T20:50:31.459 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:50:31.459 DEBUG:boto:Port: 8000 2023-05-31T20:50:31.460 DEBUG:boto:Params: {} 2023-05-31T20:50:31.460 DEBUG:boto:Token: None 2023-05-31T20:50:31.461 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:50:31 GMT /pklxkd-59/ 2023-05-31T20:50:31.461 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:apPqLCZHOWwza3uFvs+KzJ/Flyg= 2023-05-31T20:50:31.461 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:50:31 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:apPqLCZHOWwza3uFvs+KzJ/Flyg=', 'Content-Length': '0'} 2023-05-31T20:50:31.465 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000fb9167751ffcdca5-006477b317-1630-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:50:31 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:50:31.465 DEBUG:boto:b'pklxkd-59obj551000falseobj552023-05-31T20:39:26.630Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:50:31.466 DEBUG:boto:path=/pklxkd-59/obj55 2023-05-31T20:50:31.466 DEBUG:boto:auth_path=/pklxkd-59/obj55 2023-05-31T20:50:31.466 DEBUG:boto:Method: HEAD 2023-05-31T20:50:31.467 DEBUG:boto:Path: /pklxkd-59/obj55 2023-05-31T20:50:31.467 DEBUG:boto:Data: 2023-05-31T20:50:31.467 DEBUG:boto:Headers: {} 2023-05-31T20:50:31.468 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:50:31.468 DEBUG:boto:Port: 8001 2023-05-31T20:50:31.468 DEBUG:boto:Params: {} 2023-05-31T20:50:31.469 DEBUG:boto:Token: None 2023-05-31T20:50:31.469 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:50:31 GMT /pklxkd-59/obj55 2023-05-31T20:50:31.470 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:Ir4hAqTI2+JVkC3cZ5JaDUhXgRg= 2023-05-31T20:50:31.470 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:50:31 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:Ir4hAqTI2+JVkC3cZ5JaDUhXgRg=', 'Content-Length': '0'} 2023-05-31T20:50:31.472 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:39:26 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002fa7cacba3477580-006477b317-1c17-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:50:31 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:50:31.472 DEBUG:boto:path=/pklxkd-59/obj55 2023-05-31T20:50:31.473 DEBUG:boto:auth_path=/pklxkd-59/obj55 2023-05-31T20:50:31.473 DEBUG:boto:Method: HEAD 2023-05-31T20:50:31.474 DEBUG:boto:Path: /pklxkd-59/obj55 2023-05-31T20:50:31.474 DEBUG:boto:Data: 2023-05-31T20:50:31.474 DEBUG:boto:Headers: {} 2023-05-31T20:50:31.474 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:50:31.475 DEBUG:boto:Port: 8000 2023-05-31T20:50:31.475 DEBUG:boto:Params: {} 2023-05-31T20:50:31.476 DEBUG:boto:Token: None 2023-05-31T20:50:31.476 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:50:31 GMT /pklxkd-59/obj55 2023-05-31T20:50:31.476 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:Ir4hAqTI2+JVkC3cZ5JaDUhXgRg= 2023-05-31T20:50:31.477 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:50:31 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:Ir4hAqTI2+JVkC3cZ5JaDUhXgRg=', 'Content-Length': '0'} 2023-05-31T20:50:31.478 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:39:26 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000cca91f30b3dc37ef-006477b317-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:50:31 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:50:31.479 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj55 2023-05-31T20:50:31.479 DEBUG:boto:path=/pklxkd-59/obj55 2023-05-31T20:50:31.480 DEBUG:boto:auth_path=/pklxkd-59/obj55 2023-05-31T20:50:31.480 DEBUG:boto:Method: GET 2023-05-31T20:50:31.480 DEBUG:boto:Path: /pklxkd-59/obj55 2023-05-31T20:50:31.481 DEBUG:boto:Data: 2023-05-31T20:50:31.481 DEBUG:boto:Headers: {} 2023-05-31T20:50:31.482 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:50:31.482 DEBUG:boto:Port: 8001 2023-05-31T20:50:31.482 DEBUG:boto:Params: {} 2023-05-31T20:50:31.483 DEBUG:boto:Token: None 2023-05-31T20:50:31.483 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:50:31 GMT /pklxkd-59/obj55 2023-05-31T20:50:31.483 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:8d00LauL1VBprzKiC3dbXNFS9D4= 2023-05-31T20:50:31.484 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:50:31 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:8d00LauL1VBprzKiC3dbXNFS9D4=', 'Content-Length': '0'} 2023-05-31T20:50:31.486 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:39:26 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002ea9c29c5ba909e6-006477b317-1c17-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:50:31 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:50:31.486 DEBUG:boto:path=/pklxkd-59/obj55 2023-05-31T20:50:31.487 DEBUG:boto:auth_path=/pklxkd-59/obj55 2023-05-31T20:50:31.487 DEBUG:boto:Method: GET 2023-05-31T20:50:31.487 DEBUG:boto:Path: /pklxkd-59/obj55 2023-05-31T20:50:31.488 DEBUG:boto:Data: 2023-05-31T20:50:31.488 DEBUG:boto:Headers: {} 2023-05-31T20:50:31.488 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:50:31.489 DEBUG:boto:Port: 8000 2023-05-31T20:50:31.489 DEBUG:boto:Params: {} 2023-05-31T20:50:31.490 DEBUG:boto:Token: None 2023-05-31T20:50:31.490 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:50:31 GMT /pklxkd-59/obj55 2023-05-31T20:50:31.490 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:8d00LauL1VBprzKiC3dbXNFS9D4= 2023-05-31T20:50:31.491 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:50:31 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:8d00LauL1VBprzKiC3dbXNFS9D4=', 'Content-Length': '0'} 2023-05-31T20:50:31.493 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:39:26 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000019647cf642fe9e73-006477b317-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:50:31 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:50:31.536 DEBUG:boto:path=/pklxkd-59/ 2023-05-31T20:50:31.537 DEBUG:boto:auth_path=/pklxkd-59/ 2023-05-31T20:50:31.537 DEBUG:boto:path=/pklxkd-59/?prefix=obj55 2023-05-31T20:50:31.537 DEBUG:boto:auth_path=/pklxkd-59/?prefix=obj55 2023-05-31T20:50:31.538 DEBUG:boto:Method: GET 2023-05-31T20:50:31.538 DEBUG:boto:Path: /pklxkd-59/?prefix=obj55 2023-05-31T20:50:31.538 DEBUG:boto:Data: 2023-05-31T20:50:31.539 DEBUG:boto:Headers: {} 2023-05-31T20:50:31.539 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:50:31.540 DEBUG:boto:Port: 8001 2023-05-31T20:50:31.540 DEBUG:boto:Params: {} 2023-05-31T20:50:31.540 DEBUG:boto:Token: None 2023-05-31T20:50:31.541 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:50:31 GMT /pklxkd-59/ 2023-05-31T20:50:31.541 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:apPqLCZHOWwza3uFvs+KzJ/Flyg= 2023-05-31T20:50:31.542 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:50:31 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:apPqLCZHOWwza3uFvs+KzJ/Flyg=', 'Content-Length': '0'} 2023-05-31T20:50:31.544 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000055c5420ebbd423e9-006477b317-1c17-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:50:31 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:50:31.545 DEBUG:boto:b'pklxkd-59obj551000falseobj552023-05-31T20:39:26.630Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:50:31.545 DEBUG:boto:path=/pklxkd-59/ 2023-05-31T20:50:31.546 DEBUG:boto:auth_path=/pklxkd-59/ 2023-05-31T20:50:31.546 DEBUG:boto:path=/pklxkd-59/?prefix=obj55 2023-05-31T20:50:31.546 DEBUG:boto:auth_path=/pklxkd-59/?prefix=obj55 2023-05-31T20:50:31.547 DEBUG:boto:Method: GET 2023-05-31T20:50:31.547 DEBUG:boto:Path: /pklxkd-59/?prefix=obj55 2023-05-31T20:50:31.548 DEBUG:boto:Data: 2023-05-31T20:50:31.548 DEBUG:boto:Headers: {} 2023-05-31T20:50:31.548 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:50:31.549 DEBUG:boto:Port: 8000 2023-05-31T20:50:31.549 DEBUG:boto:Params: {} 2023-05-31T20:50:31.549 DEBUG:boto:Token: None 2023-05-31T20:50:31.550 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:50:31 GMT /pklxkd-59/ 2023-05-31T20:50:31.550 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:apPqLCZHOWwza3uFvs+KzJ/Flyg= 2023-05-31T20:50:31.551 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:50:31 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:apPqLCZHOWwza3uFvs+KzJ/Flyg=', 'Content-Length': '0'} 2023-05-31T20:50:31.554 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ee38293b9959242f-006477b317-1630-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:50:31 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:50:31.555 DEBUG:boto:b'pklxkd-59obj551000falseobj552023-05-31T20:39:26.630Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:50:31.555 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj57 2023-05-31T20:50:31.556 DEBUG:boto:path=/pklxkd-59/obj57 2023-05-31T20:50:31.556 DEBUG:boto:auth_path=/pklxkd-59/obj57 2023-05-31T20:50:31.556 DEBUG:boto:path=/pklxkd-59/obj57?versionId=null 2023-05-31T20:50:31.557 DEBUG:boto:auth_path=/pklxkd-59/obj57?versionId=null 2023-05-31T20:50:31.557 DEBUG:boto:Method: GET 2023-05-31T20:50:31.557 DEBUG:boto:Path: /pklxkd-59/obj57?versionId=null 2023-05-31T20:50:31.558 DEBUG:boto:Data: 2023-05-31T20:50:31.558 DEBUG:boto:Headers: {} 2023-05-31T20:50:31.559 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:50:31.559 DEBUG:boto:Port: 8001 2023-05-31T20:50:31.559 DEBUG:boto:Params: {} 2023-05-31T20:50:31.560 DEBUG:boto:Token: None 2023-05-31T20:50:31.560 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:50:31 GMT /pklxkd-59/obj57?versionId=null 2023-05-31T20:50:31.560 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:6DSPYxwl0g21OeMQf+CoeeLjDes= 2023-05-31T20:50:31.561 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:50:31 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:6DSPYxwl0g21OeMQf+CoeeLjDes=', 'Content-Length': '0'} 2023-05-31T20:50:31.563 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:44:50 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000048f50d47fca3003e-006477b317-1c17-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:50:31 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:50:31.608 DEBUG:boto:path=/pklxkd-59/obj57 2023-05-31T20:50:31.609 DEBUG:boto:auth_path=/pklxkd-59/obj57 2023-05-31T20:50:31.609 DEBUG:boto:path=/pklxkd-59/obj57?versionId=null 2023-05-31T20:50:31.610 DEBUG:boto:auth_path=/pklxkd-59/obj57?versionId=null 2023-05-31T20:50:31.610 DEBUG:boto:Method: GET 2023-05-31T20:50:31.610 DEBUG:boto:Path: /pklxkd-59/obj57?versionId=null 2023-05-31T20:50:31.611 DEBUG:boto:Data: 2023-05-31T20:50:31.611 DEBUG:boto:Headers: {} 2023-05-31T20:50:31.611 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:50:31.612 DEBUG:boto:Port: 8000 2023-05-31T20:50:31.612 DEBUG:boto:Params: {} 2023-05-31T20:50:31.612 DEBUG:boto:Token: None 2023-05-31T20:50:31.613 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:50:31 GMT /pklxkd-59/obj57?versionId=null 2023-05-31T20:50:31.613 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:6DSPYxwl0g21OeMQf+CoeeLjDes= 2023-05-31T20:50:31.614 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:50:31 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:6DSPYxwl0g21OeMQf+CoeeLjDes=', 'Content-Length': '0'} 2023-05-31T20:50:31.616 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:44:50 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002c1adde67f51c8aa-006477b317-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:50:31 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:50:31.616 DEBUG:boto:path=/pklxkd-59/obj57 2023-05-31T20:50:31.617 DEBUG:boto:auth_path=/pklxkd-59/obj57 2023-05-31T20:50:31.617 DEBUG:boto:path=/pklxkd-59/obj57?versionId=null 2023-05-31T20:50:31.617 DEBUG:boto:auth_path=/pklxkd-59/obj57?versionId=null 2023-05-31T20:50:31.618 DEBUG:boto:Method: HEAD 2023-05-31T20:50:31.618 DEBUG:boto:Path: /pklxkd-59/obj57?versionId=null 2023-05-31T20:50:31.618 DEBUG:boto:Data: 2023-05-31T20:50:31.619 DEBUG:boto:Headers: {} 2023-05-31T20:50:31.619 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:50:31.619 DEBUG:boto:Port: 8001 2023-05-31T20:50:31.620 DEBUG:boto:Params: {} 2023-05-31T20:50:31.620 DEBUG:boto:Token: None 2023-05-31T20:50:31.621 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:50:31 GMT /pklxkd-59/obj57?versionId=null 2023-05-31T20:50:31.621 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:Fd53oM0Huw+zm22WoNJPVfBMVq0= 2023-05-31T20:50:31.621 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:50:31 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:Fd53oM0Huw+zm22WoNJPVfBMVq0=', 'Content-Length': '0'} 2023-05-31T20:50:31.623 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:44:50 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e1611a295678442a-006477b317-1c17-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:50:31 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:50:31.624 DEBUG:boto:path=/pklxkd-59/obj57 2023-05-31T20:50:31.624 DEBUG:boto:auth_path=/pklxkd-59/obj57 2023-05-31T20:50:31.625 DEBUG:boto:path=/pklxkd-59/obj57?versionId=null 2023-05-31T20:50:31.625 DEBUG:boto:auth_path=/pklxkd-59/obj57?versionId=null 2023-05-31T20:50:31.625 DEBUG:boto:Method: HEAD 2023-05-31T20:50:31.626 DEBUG:boto:Path: /pklxkd-59/obj57?versionId=null 2023-05-31T20:50:31.626 DEBUG:boto:Data: 2023-05-31T20:50:31.626 DEBUG:boto:Headers: {} 2023-05-31T20:50:31.627 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:50:31.627 DEBUG:boto:Port: 8000 2023-05-31T20:50:31.627 DEBUG:boto:Params: {} 2023-05-31T20:50:31.628 DEBUG:boto:Token: None 2023-05-31T20:50:31.628 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:50:31 GMT /pklxkd-59/obj57?versionId=null 2023-05-31T20:50:31.628 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:Fd53oM0Huw+zm22WoNJPVfBMVq0= 2023-05-31T20:50:31.629 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:50:31 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:Fd53oM0Huw+zm22WoNJPVfBMVq0=', 'Content-Length': '0'} 2023-05-31T20:50:31.630 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:44:50 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e1eb09a593e43435-006477b317-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:50:31 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:50:31.631 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj57 2023-05-31T20:50:31.631 DEBUG:boto:path=/pklxkd-59/obj57 2023-05-31T20:50:31.632 DEBUG:boto:auth_path=/pklxkd-59/obj57 2023-05-31T20:50:31.632 DEBUG:boto:path=/pklxkd-59/obj57?versionId=null 2023-05-31T20:50:31.632 DEBUG:boto:auth_path=/pklxkd-59/obj57?versionId=null 2023-05-31T20:50:31.633 DEBUG:boto:Method: GET 2023-05-31T20:50:31.633 DEBUG:boto:Path: /pklxkd-59/obj57?versionId=null 2023-05-31T20:50:31.634 DEBUG:boto:Data: 2023-05-31T20:50:31.634 DEBUG:boto:Headers: {} 2023-05-31T20:50:31.634 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:50:31.635 DEBUG:boto:Port: 8001 2023-05-31T20:50:31.635 DEBUG:boto:Params: {} 2023-05-31T20:50:31.635 DEBUG:boto:Token: None 2023-05-31T20:50:31.636 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:50:31 GMT /pklxkd-59/obj57?versionId=null 2023-05-31T20:50:31.636 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:6DSPYxwl0g21OeMQf+CoeeLjDes= 2023-05-31T20:50:31.637 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:50:31 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:6DSPYxwl0g21OeMQf+CoeeLjDes=', 'Content-Length': '0'} 2023-05-31T20:50:31.638 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:44:50 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003391ad3f73c44b0c-006477b317-1c17-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:50:31 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:50:31.639 DEBUG:boto:path=/pklxkd-59/obj57 2023-05-31T20:50:31.639 DEBUG:boto:auth_path=/pklxkd-59/obj57 2023-05-31T20:50:31.640 DEBUG:boto:path=/pklxkd-59/obj57?versionId=null 2023-05-31T20:50:31.640 DEBUG:boto:auth_path=/pklxkd-59/obj57?versionId=null 2023-05-31T20:50:31.641 DEBUG:boto:Method: GET 2023-05-31T20:50:31.641 DEBUG:boto:Path: /pklxkd-59/obj57?versionId=null 2023-05-31T20:50:31.641 DEBUG:boto:Data: 2023-05-31T20:50:31.642 DEBUG:boto:Headers: {} 2023-05-31T20:50:31.642 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:50:31.642 DEBUG:boto:Port: 8000 2023-05-31T20:50:31.643 DEBUG:boto:Params: {} 2023-05-31T20:50:31.643 DEBUG:boto:Token: None 2023-05-31T20:50:31.643 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:50:31 GMT /pklxkd-59/obj57?versionId=null 2023-05-31T20:50:31.644 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:6DSPYxwl0g21OeMQf+CoeeLjDes= 2023-05-31T20:50:31.644 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:50:31 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:6DSPYxwl0g21OeMQf+CoeeLjDes=', 'Content-Length': '0'} 2023-05-31T20:50:31.646 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:44:50 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000202141484e055b10-006477b317-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:50:31 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:50:31.647 DEBUG:boto:path=/pklxkd-59/ 2023-05-31T20:50:31.647 DEBUG:boto:auth_path=/pklxkd-59/ 2023-05-31T20:50:31.648 DEBUG:boto:path=/pklxkd-59/?prefix=obj57 2023-05-31T20:50:31.648 DEBUG:boto:auth_path=/pklxkd-59/?prefix=obj57 2023-05-31T20:50:31.648 DEBUG:boto:Method: GET 2023-05-31T20:50:31.649 DEBUG:boto:Path: /pklxkd-59/?prefix=obj57 2023-05-31T20:50:31.649 DEBUG:boto:Data: 2023-05-31T20:50:31.649 DEBUG:boto:Headers: {} 2023-05-31T20:50:31.650 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:50:31.650 DEBUG:boto:Port: 8001 2023-05-31T20:50:31.650 DEBUG:boto:Params: {} 2023-05-31T20:50:31.651 DEBUG:boto:Token: None 2023-05-31T20:50:31.651 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:50:31 GMT /pklxkd-59/ 2023-05-31T20:50:31.652 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:apPqLCZHOWwza3uFvs+KzJ/Flyg= 2023-05-31T20:50:31.652 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:50:31 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:apPqLCZHOWwza3uFvs+KzJ/Flyg=', 'Content-Length': '0'} 2023-05-31T20:50:31.655 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000005349c79ed47b6cfd-006477b317-1c17-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:50:31 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:50:31.655 DEBUG:boto:b'pklxkd-59obj571000falseobj572023-05-31T20:44:50.627Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:50:31.656 DEBUG:boto:path=/pklxkd-59/ 2023-05-31T20:50:31.656 DEBUG:boto:auth_path=/pklxkd-59/ 2023-05-31T20:50:31.657 DEBUG:boto:path=/pklxkd-59/?prefix=obj57 2023-05-31T20:50:31.657 DEBUG:boto:auth_path=/pklxkd-59/?prefix=obj57 2023-05-31T20:50:31.657 DEBUG:boto:Method: GET 2023-05-31T20:50:31.658 DEBUG:boto:Path: /pklxkd-59/?prefix=obj57 2023-05-31T20:50:31.658 DEBUG:boto:Data: 2023-05-31T20:50:31.658 DEBUG:boto:Headers: {} 2023-05-31T20:50:31.659 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:50:31.659 DEBUG:boto:Port: 8000 2023-05-31T20:50:31.659 DEBUG:boto:Params: {} 2023-05-31T20:50:31.660 DEBUG:boto:Token: None 2023-05-31T20:50:31.660 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:50:31 GMT /pklxkd-59/ 2023-05-31T20:50:31.661 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:apPqLCZHOWwza3uFvs+KzJ/Flyg= 2023-05-31T20:50:31.661 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:50:31 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:apPqLCZHOWwza3uFvs+KzJ/Flyg=', 'Content-Length': '0'} 2023-05-31T20:50:31.665 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000039e09b9d2a8f52b1-006477b317-1630-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:50:31 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:50:31.665 DEBUG:boto:b'pklxkd-59obj571000falseobj572023-05-31T20:44:50.627Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:50:31.666 DEBUG:boto:path=/pklxkd-59/obj57 2023-05-31T20:50:31.666 DEBUG:boto:auth_path=/pklxkd-59/obj57 2023-05-31T20:50:31.666 DEBUG:boto:Method: HEAD 2023-05-31T20:50:31.667 DEBUG:boto:Path: /pklxkd-59/obj57 2023-05-31T20:50:31.667 DEBUG:boto:Data: 2023-05-31T20:50:31.667 DEBUG:boto:Headers: {} 2023-05-31T20:50:31.668 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:50:31.668 DEBUG:boto:Port: 8001 2023-05-31T20:50:31.668 DEBUG:boto:Params: {} 2023-05-31T20:50:31.669 DEBUG:boto:Token: None 2023-05-31T20:50:31.669 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:50:31 GMT /pklxkd-59/obj57 2023-05-31T20:50:31.669 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:1IuilwPgZMUABU2mkShn8te4uoQ= 2023-05-31T20:50:31.670 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:50:31 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:1IuilwPgZMUABU2mkShn8te4uoQ=', 'Content-Length': '0'} 2023-05-31T20:50:31.672 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:44:50 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000128860ce24f5ba34-006477b317-1c17-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:50:31 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:50:31.672 DEBUG:boto:path=/pklxkd-59/obj57 2023-05-31T20:50:31.673 DEBUG:boto:auth_path=/pklxkd-59/obj57 2023-05-31T20:50:31.673 DEBUG:boto:Method: HEAD 2023-05-31T20:50:31.673 DEBUG:boto:Path: /pklxkd-59/obj57 2023-05-31T20:50:31.674 DEBUG:boto:Data: 2023-05-31T20:50:31.674 DEBUG:boto:Headers: {} 2023-05-31T20:50:31.674 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:50:31.675 DEBUG:boto:Port: 8000 2023-05-31T20:50:31.675 DEBUG:boto:Params: {} 2023-05-31T20:50:31.675 DEBUG:boto:Token: None 2023-05-31T20:50:31.676 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:50:31 GMT /pklxkd-59/obj57 2023-05-31T20:50:31.676 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:1IuilwPgZMUABU2mkShn8te4uoQ= 2023-05-31T20:50:31.676 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:50:31 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:1IuilwPgZMUABU2mkShn8te4uoQ=', 'Content-Length': '0'} 2023-05-31T20:50:31.678 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:44:50 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000bcf6c1e2e48e03a2-006477b317-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:50:31 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:50:31.679 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj57 2023-05-31T20:50:31.679 DEBUG:boto:path=/pklxkd-59/obj57 2023-05-31T20:50:31.679 DEBUG:boto:auth_path=/pklxkd-59/obj57 2023-05-31T20:50:31.680 DEBUG:boto:Method: GET 2023-05-31T20:50:31.680 DEBUG:boto:Path: /pklxkd-59/obj57 2023-05-31T20:50:31.680 DEBUG:boto:Data: 2023-05-31T20:50:31.681 DEBUG:boto:Headers: {} 2023-05-31T20:50:31.681 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:50:31.681 DEBUG:boto:Port: 8001 2023-05-31T20:50:31.682 DEBUG:boto:Params: {} 2023-05-31T20:50:31.682 DEBUG:boto:Token: None 2023-05-31T20:50:31.683 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:50:31 GMT /pklxkd-59/obj57 2023-05-31T20:50:31.683 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:wIW5mivzbmYSEuNro9hW6WTHCGg= 2023-05-31T20:50:31.683 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:50:31 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:wIW5mivzbmYSEuNro9hW6WTHCGg=', 'Content-Length': '0'} 2023-05-31T20:50:31.685 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:44:50 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a3442eaf159bc03e-006477b317-1c17-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:50:31 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:50:31.686 DEBUG:boto:path=/pklxkd-59/obj57 2023-05-31T20:50:31.686 DEBUG:boto:auth_path=/pklxkd-59/obj57 2023-05-31T20:50:31.687 DEBUG:boto:Method: GET 2023-05-31T20:50:31.687 DEBUG:boto:Path: /pklxkd-59/obj57 2023-05-31T20:50:31.687 DEBUG:boto:Data: 2023-05-31T20:50:31.688 DEBUG:boto:Headers: {} 2023-05-31T20:50:31.688 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:50:31.688 DEBUG:boto:Port: 8000 2023-05-31T20:50:31.689 DEBUG:boto:Params: {} 2023-05-31T20:50:31.689 DEBUG:boto:Token: None 2023-05-31T20:50:31.689 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:50:31 GMT /pklxkd-59/obj57 2023-05-31T20:50:31.690 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:wIW5mivzbmYSEuNro9hW6WTHCGg= 2023-05-31T20:50:31.690 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:50:31 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:wIW5mivzbmYSEuNro9hW6WTHCGg=', 'Content-Length': '0'} 2023-05-31T20:50:31.692 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:44:50 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b45a29b8456c3f04-006477b317-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:50:31 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:50:31.693 DEBUG:boto:path=/pklxkd-59/ 2023-05-31T20:50:31.693 DEBUG:boto:auth_path=/pklxkd-59/ 2023-05-31T20:50:31.694 DEBUG:boto:path=/pklxkd-59/?prefix=obj57 2023-05-31T20:50:31.694 DEBUG:boto:auth_path=/pklxkd-59/?prefix=obj57 2023-05-31T20:50:31.694 DEBUG:boto:Method: GET 2023-05-31T20:50:31.695 DEBUG:boto:Path: /pklxkd-59/?prefix=obj57 2023-05-31T20:50:31.695 DEBUG:boto:Data: 2023-05-31T20:50:31.695 DEBUG:boto:Headers: {} 2023-05-31T20:50:31.696 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:50:31.696 DEBUG:boto:Port: 8001 2023-05-31T20:50:31.696 DEBUG:boto:Params: {} 2023-05-31T20:50:31.697 DEBUG:boto:Token: None 2023-05-31T20:50:31.697 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:50:31 GMT /pklxkd-59/ 2023-05-31T20:50:31.698 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:apPqLCZHOWwza3uFvs+KzJ/Flyg= 2023-05-31T20:50:31.698 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:50:31 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:apPqLCZHOWwza3uFvs+KzJ/Flyg=', 'Content-Length': '0'} 2023-05-31T20:50:31.700 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f37f564e8bfcea17-006477b317-1c17-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:50:31 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:50:31.701 DEBUG:boto:b'pklxkd-59obj571000falseobj572023-05-31T20:44:50.627Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:50:31.701 DEBUG:boto:path=/pklxkd-59/ 2023-05-31T20:50:31.702 DEBUG:boto:auth_path=/pklxkd-59/ 2023-05-31T20:50:31.702 DEBUG:boto:path=/pklxkd-59/?prefix=obj57 2023-05-31T20:50:31.703 DEBUG:boto:auth_path=/pklxkd-59/?prefix=obj57 2023-05-31T20:50:31.703 DEBUG:boto:Method: GET 2023-05-31T20:50:31.703 DEBUG:boto:Path: /pklxkd-59/?prefix=obj57 2023-05-31T20:50:31.704 DEBUG:boto:Data: 2023-05-31T20:50:31.704 DEBUG:boto:Headers: {} 2023-05-31T20:50:31.704 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:50:31.705 DEBUG:boto:Port: 8000 2023-05-31T20:50:31.705 DEBUG:boto:Params: {} 2023-05-31T20:50:31.705 DEBUG:boto:Token: None 2023-05-31T20:50:31.706 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:50:31 GMT /pklxkd-59/ 2023-05-31T20:50:31.706 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:apPqLCZHOWwza3uFvs+KzJ/Flyg= 2023-05-31T20:50:31.707 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:50:31 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:apPqLCZHOWwza3uFvs+KzJ/Flyg=', 'Content-Length': '0'} 2023-05-31T20:50:31.710 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000005e67c36ce300abf0-006477b317-1630-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:50:31 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:50:31.752 DEBUG:boto:b'pklxkd-59obj571000falseobj572023-05-31T20:44:50.627Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:50:31.753 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj58 2023-05-31T20:50:31.753 DEBUG:boto:path=/pklxkd-59/obj58 2023-05-31T20:50:31.754 DEBUG:boto:auth_path=/pklxkd-59/obj58 2023-05-31T20:50:31.754 DEBUG:boto:path=/pklxkd-59/obj58?versionId=null 2023-05-31T20:50:31.754 DEBUG:boto:auth_path=/pklxkd-59/obj58?versionId=null 2023-05-31T20:50:31.755 DEBUG:boto:Method: GET 2023-05-31T20:50:31.755 DEBUG:boto:Path: /pklxkd-59/obj58?versionId=null 2023-05-31T20:50:31.755 DEBUG:boto:Data: 2023-05-31T20:50:31.756 DEBUG:boto:Headers: {} 2023-05-31T20:50:31.756 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:50:31.756 DEBUG:boto:Port: 8001 2023-05-31T20:50:31.757 DEBUG:boto:Params: {} 2023-05-31T20:50:31.757 DEBUG:boto:Token: None 2023-05-31T20:50:31.758 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:50:31 GMT /pklxkd-59/obj58?versionId=null 2023-05-31T20:50:31.758 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:dzYZ5puaZu+sNUMaRflcEHs9nyo= 2023-05-31T20:50:31.758 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:50:31 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:dzYZ5puaZu+sNUMaRflcEHs9nyo=', 'Content-Length': '0'} 2023-05-31T20:50:31.761 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:48:12 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000098d47c6969185b4d-006477b317-1c17-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:50:31 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:50:31.761 DEBUG:boto:path=/pklxkd-59/obj58 2023-05-31T20:50:31.762 DEBUG:boto:auth_path=/pklxkd-59/obj58 2023-05-31T20:50:31.762 DEBUG:boto:path=/pklxkd-59/obj58?versionId=null 2023-05-31T20:50:31.762 DEBUG:boto:auth_path=/pklxkd-59/obj58?versionId=null 2023-05-31T20:50:31.763 DEBUG:boto:Method: GET 2023-05-31T20:50:31.763 DEBUG:boto:Path: /pklxkd-59/obj58?versionId=null 2023-05-31T20:50:31.763 DEBUG:boto:Data: 2023-05-31T20:50:31.764 DEBUG:boto:Headers: {} 2023-05-31T20:50:31.764 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:50:31.765 DEBUG:boto:Port: 8000 2023-05-31T20:50:31.765 DEBUG:boto:Params: {} 2023-05-31T20:50:31.765 DEBUG:boto:Token: None 2023-05-31T20:50:31.766 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:50:31 GMT /pklxkd-59/obj58?versionId=null 2023-05-31T20:50:31.766 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:dzYZ5puaZu+sNUMaRflcEHs9nyo= 2023-05-31T20:50:31.766 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:50:31 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:dzYZ5puaZu+sNUMaRflcEHs9nyo=', 'Content-Length': '0'} 2023-05-31T20:50:31.768 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:48:12 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c54c82c62f0c7a19-006477b317-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:50:31 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:50:31.769 DEBUG:boto:path=/pklxkd-59/obj58 2023-05-31T20:50:31.769 DEBUG:boto:auth_path=/pklxkd-59/obj58 2023-05-31T20:50:31.770 DEBUG:boto:path=/pklxkd-59/obj58?versionId=null 2023-05-31T20:50:31.770 DEBUG:boto:auth_path=/pklxkd-59/obj58?versionId=null 2023-05-31T20:50:31.770 DEBUG:boto:Method: HEAD 2023-05-31T20:50:31.771 DEBUG:boto:Path: /pklxkd-59/obj58?versionId=null 2023-05-31T20:50:31.771 DEBUG:boto:Data: 2023-05-31T20:50:31.771 DEBUG:boto:Headers: {} 2023-05-31T20:50:31.772 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:50:31.772 DEBUG:boto:Port: 8001 2023-05-31T20:50:31.772 DEBUG:boto:Params: {} 2023-05-31T20:50:31.773 DEBUG:boto:Token: None 2023-05-31T20:50:31.773 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:50:31 GMT /pklxkd-59/obj58?versionId=null 2023-05-31T20:50:31.774 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:C1t9cb61PqKMxqKct9iqATryRzs= 2023-05-31T20:50:31.774 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:50:31 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:C1t9cb61PqKMxqKct9iqATryRzs=', 'Content-Length': '0'} 2023-05-31T20:50:31.776 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:48:12 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d5bc62c92f051679-006477b317-1c17-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:50:31 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:50:31.776 DEBUG:boto:path=/pklxkd-59/obj58 2023-05-31T20:50:31.777 DEBUG:boto:auth_path=/pklxkd-59/obj58 2023-05-31T20:50:31.777 DEBUG:boto:path=/pklxkd-59/obj58?versionId=null 2023-05-31T20:50:31.777 DEBUG:boto:auth_path=/pklxkd-59/obj58?versionId=null 2023-05-31T20:50:31.778 DEBUG:boto:Method: HEAD 2023-05-31T20:50:31.778 DEBUG:boto:Path: /pklxkd-59/obj58?versionId=null 2023-05-31T20:50:31.778 DEBUG:boto:Data: 2023-05-31T20:50:31.779 DEBUG:boto:Headers: {} 2023-05-31T20:50:31.779 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:50:31.779 DEBUG:boto:Port: 8000 2023-05-31T20:50:31.780 DEBUG:boto:Params: {} 2023-05-31T20:50:31.780 DEBUG:boto:Token: None 2023-05-31T20:50:31.781 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:50:31 GMT /pklxkd-59/obj58?versionId=null 2023-05-31T20:50:31.781 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:C1t9cb61PqKMxqKct9iqATryRzs= 2023-05-31T20:50:31.781 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:50:31 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:C1t9cb61PqKMxqKct9iqATryRzs=', 'Content-Length': '0'} 2023-05-31T20:50:31.783 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:48:12 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000af9876ddc6ebcb4-006477b317-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:50:31 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:50:31.784 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj58 2023-05-31T20:50:31.784 DEBUG:boto:path=/pklxkd-59/obj58 2023-05-31T20:50:31.784 DEBUG:boto:auth_path=/pklxkd-59/obj58 2023-05-31T20:50:31.785 DEBUG:boto:path=/pklxkd-59/obj58?versionId=null 2023-05-31T20:50:31.785 DEBUG:boto:auth_path=/pklxkd-59/obj58?versionId=null 2023-05-31T20:50:31.785 DEBUG:boto:Method: GET 2023-05-31T20:50:31.786 DEBUG:boto:Path: /pklxkd-59/obj58?versionId=null 2023-05-31T20:50:31.786 DEBUG:boto:Data: 2023-05-31T20:50:31.786 DEBUG:boto:Headers: {} 2023-05-31T20:50:31.787 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:50:31.787 DEBUG:boto:Port: 8001 2023-05-31T20:50:31.788 DEBUG:boto:Params: {} 2023-05-31T20:50:31.788 DEBUG:boto:Token: None 2023-05-31T20:50:31.788 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:50:31 GMT /pklxkd-59/obj58?versionId=null 2023-05-31T20:50:31.789 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:dzYZ5puaZu+sNUMaRflcEHs9nyo= 2023-05-31T20:50:31.789 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:50:31 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:dzYZ5puaZu+sNUMaRflcEHs9nyo=', 'Content-Length': '0'} 2023-05-31T20:50:31.791 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:48:12 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ef7e50445d9a0918-006477b317-1c17-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:50:31 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:50:31.792 DEBUG:boto:path=/pklxkd-59/obj58 2023-05-31T20:50:31.792 DEBUG:boto:auth_path=/pklxkd-59/obj58 2023-05-31T20:50:31.793 DEBUG:boto:path=/pklxkd-59/obj58?versionId=null 2023-05-31T20:50:31.793 DEBUG:boto:auth_path=/pklxkd-59/obj58?versionId=null 2023-05-31T20:50:31.793 DEBUG:boto:Method: GET 2023-05-31T20:50:31.794 DEBUG:boto:Path: /pklxkd-59/obj58?versionId=null 2023-05-31T20:50:31.794 DEBUG:boto:Data: 2023-05-31T20:50:31.794 DEBUG:boto:Headers: {} 2023-05-31T20:50:31.795 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:50:31.795 DEBUG:boto:Port: 8000 2023-05-31T20:50:31.795 DEBUG:boto:Params: {} 2023-05-31T20:50:31.796 DEBUG:boto:Token: None 2023-05-31T20:50:31.796 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:50:31 GMT /pklxkd-59/obj58?versionId=null 2023-05-31T20:50:31.797 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:dzYZ5puaZu+sNUMaRflcEHs9nyo= 2023-05-31T20:50:31.797 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:50:31 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:dzYZ5puaZu+sNUMaRflcEHs9nyo=', 'Content-Length': '0'} 2023-05-31T20:50:31.799 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:48:12 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d86f2404ef9266f0-006477b317-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:50:31 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:50:31.800 DEBUG:boto:path=/pklxkd-59/ 2023-05-31T20:50:31.800 DEBUG:boto:auth_path=/pklxkd-59/ 2023-05-31T20:50:31.801 DEBUG:boto:path=/pklxkd-59/?prefix=obj58 2023-05-31T20:50:31.801 DEBUG:boto:auth_path=/pklxkd-59/?prefix=obj58 2023-05-31T20:50:31.801 DEBUG:boto:Method: GET 2023-05-31T20:50:31.802 DEBUG:boto:Path: /pklxkd-59/?prefix=obj58 2023-05-31T20:50:31.802 DEBUG:boto:Data: 2023-05-31T20:50:31.802 DEBUG:boto:Headers: {} 2023-05-31T20:50:31.803 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:50:31.803 DEBUG:boto:Port: 8001 2023-05-31T20:50:31.803 DEBUG:boto:Params: {} 2023-05-31T20:50:31.804 DEBUG:boto:Token: None 2023-05-31T20:50:31.804 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:50:31 GMT /pklxkd-59/ 2023-05-31T20:50:31.805 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:apPqLCZHOWwza3uFvs+KzJ/Flyg= 2023-05-31T20:50:31.805 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:50:31 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:apPqLCZHOWwza3uFvs+KzJ/Flyg=', 'Content-Length': '0'} 2023-05-31T20:50:31.808 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000001f36dc4218921464-006477b317-1c17-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:50:31 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:50:31.808 DEBUG:boto:b'pklxkd-59obj581000falseobj582023-05-31T20:48:12.976Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:50:31.809 DEBUG:boto:path=/pklxkd-59/ 2023-05-31T20:50:31.809 DEBUG:boto:auth_path=/pklxkd-59/ 2023-05-31T20:50:31.810 DEBUG:boto:path=/pklxkd-59/?prefix=obj58 2023-05-31T20:50:31.810 DEBUG:boto:auth_path=/pklxkd-59/?prefix=obj58 2023-05-31T20:50:31.810 DEBUG:boto:Method: GET 2023-05-31T20:50:31.811 DEBUG:boto:Path: /pklxkd-59/?prefix=obj58 2023-05-31T20:50:31.811 DEBUG:boto:Data: 2023-05-31T20:50:31.811 DEBUG:boto:Headers: {} 2023-05-31T20:50:31.812 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:50:31.812 DEBUG:boto:Port: 8000 2023-05-31T20:50:31.812 DEBUG:boto:Params: {} 2023-05-31T20:50:31.813 DEBUG:boto:Token: None 2023-05-31T20:50:31.813 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:50:31 GMT /pklxkd-59/ 2023-05-31T20:50:31.814 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:apPqLCZHOWwza3uFvs+KzJ/Flyg= 2023-05-31T20:50:31.814 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:50:31 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:apPqLCZHOWwza3uFvs+KzJ/Flyg=', 'Content-Length': '0'} 2023-05-31T20:50:31.817 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000005ed7ecefe8aab541-006477b317-1630-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:50:31 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:50:31.818 DEBUG:boto:b'pklxkd-59obj581000falseobj582023-05-31T20:48:12.976Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:50:31.819 DEBUG:boto:path=/pklxkd-59/obj58 2023-05-31T20:50:31.819 DEBUG:boto:auth_path=/pklxkd-59/obj58 2023-05-31T20:50:31.819 DEBUG:boto:Method: HEAD 2023-05-31T20:50:31.820 DEBUG:boto:Path: /pklxkd-59/obj58 2023-05-31T20:50:31.820 DEBUG:boto:Data: 2023-05-31T20:50:31.820 DEBUG:boto:Headers: {} 2023-05-31T20:50:31.821 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:50:31.821 DEBUG:boto:Port: 8001 2023-05-31T20:50:31.821 DEBUG:boto:Params: {} 2023-05-31T20:50:31.822 DEBUG:boto:Token: None 2023-05-31T20:50:31.822 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:50:31 GMT /pklxkd-59/obj58 2023-05-31T20:50:31.822 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:OylcOcWNktpMG7WKUbX6q4fT0nY= 2023-05-31T20:50:31.823 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:50:31 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:OylcOcWNktpMG7WKUbX6q4fT0nY=', 'Content-Length': '0'} 2023-05-31T20:50:31.825 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:48:12 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000fae6c62f9e23ad5f-006477b317-1c17-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:50:31 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:50:31.825 DEBUG:boto:path=/pklxkd-59/obj58 2023-05-31T20:50:31.826 DEBUG:boto:auth_path=/pklxkd-59/obj58 2023-05-31T20:50:31.826 DEBUG:boto:Method: HEAD 2023-05-31T20:50:31.826 DEBUG:boto:Path: /pklxkd-59/obj58 2023-05-31T20:50:31.827 DEBUG:boto:Data: 2023-05-31T20:50:31.827 DEBUG:boto:Headers: {} 2023-05-31T20:50:31.827 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:50:31.828 DEBUG:boto:Port: 8000 2023-05-31T20:50:31.828 DEBUG:boto:Params: {} 2023-05-31T20:50:31.828 DEBUG:boto:Token: None 2023-05-31T20:50:31.829 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:50:31 GMT /pklxkd-59/obj58 2023-05-31T20:50:31.829 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:OylcOcWNktpMG7WKUbX6q4fT0nY= 2023-05-31T20:50:31.830 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:50:31 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:OylcOcWNktpMG7WKUbX6q4fT0nY=', 'Content-Length': '0'} 2023-05-31T20:50:31.831 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:48:12 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000fc197f17eb6bd9a9-006477b317-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:50:31 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:50:31.832 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj58 2023-05-31T20:50:31.832 DEBUG:boto:path=/pklxkd-59/obj58 2023-05-31T20:50:31.832 DEBUG:boto:auth_path=/pklxkd-59/obj58 2023-05-31T20:50:31.833 DEBUG:boto:Method: GET 2023-05-31T20:50:31.833 DEBUG:boto:Path: /pklxkd-59/obj58 2023-05-31T20:50:31.834 DEBUG:boto:Data: 2023-05-31T20:50:31.834 DEBUG:boto:Headers: {} 2023-05-31T20:50:31.834 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:50:31.835 DEBUG:boto:Port: 8001 2023-05-31T20:50:31.835 DEBUG:boto:Params: {} 2023-05-31T20:50:31.835 DEBUG:boto:Token: None 2023-05-31T20:50:31.836 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:50:31 GMT /pklxkd-59/obj58 2023-05-31T20:50:31.836 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:oeuwWChHTTGhoj0MbI3ohzLJdBM= 2023-05-31T20:50:31.837 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:50:31 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:oeuwWChHTTGhoj0MbI3ohzLJdBM=', 'Content-Length': '0'} 2023-05-31T20:50:31.838 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:48:12 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000258d8fb74b76d9b6-006477b317-1c17-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:50:31 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:50:31.880 DEBUG:boto:path=/pklxkd-59/obj58 2023-05-31T20:50:31.881 DEBUG:boto:auth_path=/pklxkd-59/obj58 2023-05-31T20:50:31.881 DEBUG:boto:Method: GET 2023-05-31T20:50:31.881 DEBUG:boto:Path: /pklxkd-59/obj58 2023-05-31T20:50:31.882 DEBUG:boto:Data: 2023-05-31T20:50:31.882 DEBUG:boto:Headers: {} 2023-05-31T20:50:31.883 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:50:31.883 DEBUG:boto:Port: 8000 2023-05-31T20:50:31.883 DEBUG:boto:Params: {} 2023-05-31T20:50:31.884 DEBUG:boto:Token: None 2023-05-31T20:50:31.884 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:50:31 GMT /pklxkd-59/obj58 2023-05-31T20:50:31.885 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:oeuwWChHTTGhoj0MbI3ohzLJdBM= 2023-05-31T20:50:31.885 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:50:31 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:oeuwWChHTTGhoj0MbI3ohzLJdBM=', 'Content-Length': '0'} 2023-05-31T20:50:31.887 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:48:12 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000021b8f627656dc228-006477b317-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:50:31 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:50:31.887 DEBUG:boto:path=/pklxkd-59/ 2023-05-31T20:50:31.888 DEBUG:boto:auth_path=/pklxkd-59/ 2023-05-31T20:50:31.888 DEBUG:boto:path=/pklxkd-59/?prefix=obj58 2023-05-31T20:50:31.889 DEBUG:boto:auth_path=/pklxkd-59/?prefix=obj58 2023-05-31T20:50:31.889 DEBUG:boto:Method: GET 2023-05-31T20:50:31.889 DEBUG:boto:Path: /pklxkd-59/?prefix=obj58 2023-05-31T20:50:31.890 DEBUG:boto:Data: 2023-05-31T20:50:31.890 DEBUG:boto:Headers: {} 2023-05-31T20:50:31.890 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:50:31.891 DEBUG:boto:Port: 8001 2023-05-31T20:50:31.891 DEBUG:boto:Params: {} 2023-05-31T20:50:31.891 DEBUG:boto:Token: None 2023-05-31T20:50:31.892 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:50:31 GMT /pklxkd-59/ 2023-05-31T20:50:31.892 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:apPqLCZHOWwza3uFvs+KzJ/Flyg= 2023-05-31T20:50:31.893 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:50:31 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:apPqLCZHOWwza3uFvs+KzJ/Flyg=', 'Content-Length': '0'} 2023-05-31T20:50:31.895 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000062417e976ab789ac-006477b317-1c17-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:50:31 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:50:31.896 DEBUG:boto:b'pklxkd-59obj581000falseobj582023-05-31T20:48:12.976Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:50:31.896 DEBUG:boto:path=/pklxkd-59/ 2023-05-31T20:50:31.897 DEBUG:boto:auth_path=/pklxkd-59/ 2023-05-31T20:50:31.897 DEBUG:boto:path=/pklxkd-59/?prefix=obj58 2023-05-31T20:50:31.898 DEBUG:boto:auth_path=/pklxkd-59/?prefix=obj58 2023-05-31T20:50:31.898 DEBUG:boto:Method: GET 2023-05-31T20:50:31.898 DEBUG:boto:Path: /pklxkd-59/?prefix=obj58 2023-05-31T20:50:31.899 DEBUG:boto:Data: 2023-05-31T20:50:31.899 DEBUG:boto:Headers: {} 2023-05-31T20:50:31.899 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:50:31.900 DEBUG:boto:Port: 8000 2023-05-31T20:50:31.900 DEBUG:boto:Params: {} 2023-05-31T20:50:31.900 DEBUG:boto:Token: None 2023-05-31T20:50:31.901 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:50:31 GMT /pklxkd-59/ 2023-05-31T20:50:31.901 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:apPqLCZHOWwza3uFvs+KzJ/Flyg= 2023-05-31T20:50:31.902 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:50:31 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:apPqLCZHOWwza3uFvs+KzJ/Flyg=', 'Content-Length': '0'} 2023-05-31T20:50:31.905 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000046bf01893100dbc3-006477b317-1630-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:50:31 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:50:31.905 DEBUG:boto:b'pklxkd-59obj581000falseobj582023-05-31T20:48:12.976Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:50:31.906 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj68 2023-05-31T20:50:31.906 DEBUG:boto:path=/pklxkd-59/obj68 2023-05-31T20:50:31.907 DEBUG:boto:auth_path=/pklxkd-59/obj68 2023-05-31T20:50:31.907 DEBUG:boto:path=/pklxkd-59/obj68?versionId=null 2023-05-31T20:50:31.908 DEBUG:boto:auth_path=/pklxkd-59/obj68?versionId=null 2023-05-31T20:50:31.908 DEBUG:boto:Method: GET 2023-05-31T20:50:31.908 DEBUG:boto:Path: /pklxkd-59/obj68?versionId=null 2023-05-31T20:50:31.909 DEBUG:boto:Data: 2023-05-31T20:50:31.909 DEBUG:boto:Headers: {} 2023-05-31T20:50:31.909 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:50:31.910 DEBUG:boto:Port: 8001 2023-05-31T20:50:31.910 DEBUG:boto:Params: {} 2023-05-31T20:50:31.911 DEBUG:boto:Token: None 2023-05-31T20:50:31.911 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:50:31 GMT /pklxkd-59/obj68?versionId=null 2023-05-31T20:50:31.911 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:oI+aQ6VDOZs7FNlDoVz0OrB4AY0= 2023-05-31T20:50:31.912 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:50:31 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:oI+aQ6VDOZs7FNlDoVz0OrB4AY0=', 'Content-Length': '0'} 2023-05-31T20:50:31.914 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:43:08 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000047601cff50a02e84-006477b317-1c17-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:50:31 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:50:31.914 DEBUG:boto:path=/pklxkd-59/obj68 2023-05-31T20:50:31.915 DEBUG:boto:auth_path=/pklxkd-59/obj68 2023-05-31T20:50:31.915 DEBUG:boto:path=/pklxkd-59/obj68?versionId=null 2023-05-31T20:50:31.915 DEBUG:boto:auth_path=/pklxkd-59/obj68?versionId=null 2023-05-31T20:50:31.916 DEBUG:boto:Method: GET 2023-05-31T20:50:31.916 DEBUG:boto:Path: /pklxkd-59/obj68?versionId=null 2023-05-31T20:50:31.917 DEBUG:boto:Data: 2023-05-31T20:50:31.917 DEBUG:boto:Headers: {} 2023-05-31T20:50:31.917 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:50:31.918 DEBUG:boto:Port: 8000 2023-05-31T20:50:31.918 DEBUG:boto:Params: {} 2023-05-31T20:50:31.918 DEBUG:boto:Token: None 2023-05-31T20:50:31.919 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:50:31 GMT /pklxkd-59/obj68?versionId=null 2023-05-31T20:50:31.919 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:oI+aQ6VDOZs7FNlDoVz0OrB4AY0= 2023-05-31T20:50:31.920 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:50:31 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:oI+aQ6VDOZs7FNlDoVz0OrB4AY0=', 'Content-Length': '0'} 2023-05-31T20:50:31.922 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:43:08 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000003c09775a32d2df6-006477b317-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:50:31 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:50:31.922 DEBUG:boto:path=/pklxkd-59/obj68 2023-05-31T20:50:31.923 DEBUG:boto:auth_path=/pklxkd-59/obj68 2023-05-31T20:50:31.923 DEBUG:boto:path=/pklxkd-59/obj68?versionId=null 2023-05-31T20:50:31.923 DEBUG:boto:auth_path=/pklxkd-59/obj68?versionId=null 2023-05-31T20:50:31.924 DEBUG:boto:Method: HEAD 2023-05-31T20:50:31.924 DEBUG:boto:Path: /pklxkd-59/obj68?versionId=null 2023-05-31T20:50:31.924 DEBUG:boto:Data: 2023-05-31T20:50:31.925 DEBUG:boto:Headers: {} 2023-05-31T20:50:31.925 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:50:31.925 DEBUG:boto:Port: 8001 2023-05-31T20:50:31.926 DEBUG:boto:Params: {} 2023-05-31T20:50:31.926 DEBUG:boto:Token: None 2023-05-31T20:50:31.926 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:50:31 GMT /pklxkd-59/obj68?versionId=null 2023-05-31T20:50:31.927 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:QNoq0sG80lWRPmDzgbFqfAeveuo= 2023-05-31T20:50:31.927 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:50:31 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:QNoq0sG80lWRPmDzgbFqfAeveuo=', 'Content-Length': '0'} 2023-05-31T20:50:31.929 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:43:08 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d699b594b05d28af-006477b317-1c17-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:50:31 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:50:31.929 DEBUG:boto:path=/pklxkd-59/obj68 2023-05-31T20:50:31.930 DEBUG:boto:auth_path=/pklxkd-59/obj68 2023-05-31T20:50:31.930 DEBUG:boto:path=/pklxkd-59/obj68?versionId=null 2023-05-31T20:50:31.931 DEBUG:boto:auth_path=/pklxkd-59/obj68?versionId=null 2023-05-31T20:50:31.931 DEBUG:boto:Method: HEAD 2023-05-31T20:50:31.931 DEBUG:boto:Path: /pklxkd-59/obj68?versionId=null 2023-05-31T20:50:31.932 DEBUG:boto:Data: 2023-05-31T20:50:31.932 DEBUG:boto:Headers: {} 2023-05-31T20:50:31.932 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:50:31.933 DEBUG:boto:Port: 8000 2023-05-31T20:50:31.933 DEBUG:boto:Params: {} 2023-05-31T20:50:31.933 DEBUG:boto:Token: None 2023-05-31T20:50:31.934 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:50:31 GMT /pklxkd-59/obj68?versionId=null 2023-05-31T20:50:31.934 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:QNoq0sG80lWRPmDzgbFqfAeveuo= 2023-05-31T20:50:31.934 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:50:31 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:QNoq0sG80lWRPmDzgbFqfAeveuo=', 'Content-Length': '0'} 2023-05-31T20:50:31.936 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:43:08 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000683888140dfb44ed-006477b317-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:50:31 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:50:31.937 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj68 2023-05-31T20:50:31.937 DEBUG:boto:path=/pklxkd-59/obj68 2023-05-31T20:50:31.938 DEBUG:boto:auth_path=/pklxkd-59/obj68 2023-05-31T20:50:31.938 DEBUG:boto:path=/pklxkd-59/obj68?versionId=null 2023-05-31T20:50:31.938 DEBUG:boto:auth_path=/pklxkd-59/obj68?versionId=null 2023-05-31T20:50:31.939 DEBUG:boto:Method: GET 2023-05-31T20:50:31.939 DEBUG:boto:Path: /pklxkd-59/obj68?versionId=null 2023-05-31T20:50:31.939 DEBUG:boto:Data: 2023-05-31T20:50:31.940 DEBUG:boto:Headers: {} 2023-05-31T20:50:31.940 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:50:31.940 DEBUG:boto:Port: 8001 2023-05-31T20:50:31.941 DEBUG:boto:Params: {} 2023-05-31T20:50:31.941 DEBUG:boto:Token: None 2023-05-31T20:50:31.942 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:50:31 GMT /pklxkd-59/obj68?versionId=null 2023-05-31T20:50:31.942 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:oI+aQ6VDOZs7FNlDoVz0OrB4AY0= 2023-05-31T20:50:31.942 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:50:31 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:oI+aQ6VDOZs7FNlDoVz0OrB4AY0=', 'Content-Length': '0'} 2023-05-31T20:50:31.944 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:43:08 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f8f58cc79e08db51-006477b317-1c17-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:50:31 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:50:31.945 DEBUG:boto:path=/pklxkd-59/obj68 2023-05-31T20:50:31.945 DEBUG:boto:auth_path=/pklxkd-59/obj68 2023-05-31T20:50:31.945 DEBUG:boto:path=/pklxkd-59/obj68?versionId=null 2023-05-31T20:50:31.946 DEBUG:boto:auth_path=/pklxkd-59/obj68?versionId=null 2023-05-31T20:50:31.946 DEBUG:boto:Method: GET 2023-05-31T20:50:31.947 DEBUG:boto:Path: /pklxkd-59/obj68?versionId=null 2023-05-31T20:50:31.947 DEBUG:boto:Data: 2023-05-31T20:50:31.947 DEBUG:boto:Headers: {} 2023-05-31T20:50:31.948 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:50:31.948 DEBUG:boto:Port: 8000 2023-05-31T20:50:31.948 DEBUG:boto:Params: {} 2023-05-31T20:50:31.949 DEBUG:boto:Token: None 2023-05-31T20:50:31.949 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:50:31 GMT /pklxkd-59/obj68?versionId=null 2023-05-31T20:50:31.950 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:oI+aQ6VDOZs7FNlDoVz0OrB4AY0= 2023-05-31T20:50:31.950 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:50:31 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:oI+aQ6VDOZs7FNlDoVz0OrB4AY0=', 'Content-Length': '0'} 2023-05-31T20:50:31.952 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:43:08 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009679bf15cbfbe440-006477b317-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:50:31 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:50:31.952 DEBUG:boto:path=/pklxkd-59/ 2023-05-31T20:50:31.953 DEBUG:boto:auth_path=/pklxkd-59/ 2023-05-31T20:50:31.953 DEBUG:boto:path=/pklxkd-59/?prefix=obj68 2023-05-31T20:50:31.953 DEBUG:boto:auth_path=/pklxkd-59/?prefix=obj68 2023-05-31T20:50:31.954 DEBUG:boto:Method: GET 2023-05-31T20:50:31.954 DEBUG:boto:Path: /pklxkd-59/?prefix=obj68 2023-05-31T20:50:31.955 DEBUG:boto:Data: 2023-05-31T20:50:31.955 DEBUG:boto:Headers: {} 2023-05-31T20:50:31.955 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:50:31.956 DEBUG:boto:Port: 8001 2023-05-31T20:50:31.956 DEBUG:boto:Params: {} 2023-05-31T20:50:31.956 DEBUG:boto:Token: None 2023-05-31T20:50:31.957 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:50:31 GMT /pklxkd-59/ 2023-05-31T20:50:31.957 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:apPqLCZHOWwza3uFvs+KzJ/Flyg= 2023-05-31T20:50:31.958 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:50:31 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:apPqLCZHOWwza3uFvs+KzJ/Flyg=', 'Content-Length': '0'} 2023-05-31T20:50:31.960 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f37e451cffe19d47-006477b317-1c17-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:50:31 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:50:32.000 DEBUG:boto:b'pklxkd-59obj681000falseobj682023-05-31T20:43:08.066Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:50:32.001 DEBUG:boto:path=/pklxkd-59/ 2023-05-31T20:50:32.001 DEBUG:boto:auth_path=/pklxkd-59/ 2023-05-31T20:50:32.002 DEBUG:boto:path=/pklxkd-59/?prefix=obj68 2023-05-31T20:50:32.002 DEBUG:boto:auth_path=/pklxkd-59/?prefix=obj68 2023-05-31T20:50:32.002 DEBUG:boto:Method: GET 2023-05-31T20:50:32.003 DEBUG:boto:Path: /pklxkd-59/?prefix=obj68 2023-05-31T20:50:32.003 DEBUG:boto:Data: 2023-05-31T20:50:32.003 DEBUG:boto:Headers: {} 2023-05-31T20:50:32.004 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:50:32.004 DEBUG:boto:Port: 8000 2023-05-31T20:50:32.005 DEBUG:boto:Params: {} 2023-05-31T20:50:32.005 DEBUG:boto:Token: None 2023-05-31T20:50:32.005 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:50:32 GMT /pklxkd-59/ 2023-05-31T20:50:32.006 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:s+yZS3HyomxcW/vxaV/hqlSq7Vg= 2023-05-31T20:50:32.006 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:50:32 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:s+yZS3HyomxcW/vxaV/hqlSq7Vg=', 'Content-Length': '0'} 2023-05-31T20:50:32.010 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a65736a9b631414e-006477b318-1630-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:50:32 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:50:32.010 DEBUG:boto:b'pklxkd-59obj681000falseobj682023-05-31T20:43:08.066Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:50:32.011 DEBUG:boto:path=/pklxkd-59/obj68 2023-05-31T20:50:32.011 DEBUG:boto:auth_path=/pklxkd-59/obj68 2023-05-31T20:50:32.012 DEBUG:boto:Method: HEAD 2023-05-31T20:50:32.012 DEBUG:boto:Path: /pklxkd-59/obj68 2023-05-31T20:50:32.012 DEBUG:boto:Data: 2023-05-31T20:50:32.013 DEBUG:boto:Headers: {} 2023-05-31T20:50:32.013 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:50:32.013 DEBUG:boto:Port: 8001 2023-05-31T20:50:32.014 DEBUG:boto:Params: {} 2023-05-31T20:50:32.014 DEBUG:boto:Token: None 2023-05-31T20:50:32.014 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:50:32 GMT /pklxkd-59/obj68 2023-05-31T20:50:32.015 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:tixLHJ6eDRf5vpV9y9xcGQY1f8M= 2023-05-31T20:50:32.015 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:50:32 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:tixLHJ6eDRf5vpV9y9xcGQY1f8M=', 'Content-Length': '0'} 2023-05-31T20:50:32.017 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:43:08 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f2543b6826c32ec3-006477b318-1c17-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:50:32 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:50:32.017 DEBUG:boto:path=/pklxkd-59/obj68 2023-05-31T20:50:32.018 DEBUG:boto:auth_path=/pklxkd-59/obj68 2023-05-31T20:50:32.018 DEBUG:boto:Method: HEAD 2023-05-31T20:50:32.018 DEBUG:boto:Path: /pklxkd-59/obj68 2023-05-31T20:50:32.019 DEBUG:boto:Data: 2023-05-31T20:50:32.019 DEBUG:boto:Headers: {} 2023-05-31T20:50:32.019 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:50:32.020 DEBUG:boto:Port: 8000 2023-05-31T20:50:32.020 DEBUG:boto:Params: {} 2023-05-31T20:50:32.020 DEBUG:boto:Token: None 2023-05-31T20:50:32.021 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:50:32 GMT /pklxkd-59/obj68 2023-05-31T20:50:32.021 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:tixLHJ6eDRf5vpV9y9xcGQY1f8M= 2023-05-31T20:50:32.022 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:50:32 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:tixLHJ6eDRf5vpV9y9xcGQY1f8M=', 'Content-Length': '0'} 2023-05-31T20:50:32.023 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:43:08 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000435a376ba2715341-006477b318-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:50:32 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:50:32.024 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj68 2023-05-31T20:50:32.024 DEBUG:boto:path=/pklxkd-59/obj68 2023-05-31T20:50:32.025 DEBUG:boto:auth_path=/pklxkd-59/obj68 2023-05-31T20:50:32.025 DEBUG:boto:Method: GET 2023-05-31T20:50:32.025 DEBUG:boto:Path: /pklxkd-59/obj68 2023-05-31T20:50:32.026 DEBUG:boto:Data: 2023-05-31T20:50:32.026 DEBUG:boto:Headers: {} 2023-05-31T20:50:32.026 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:50:32.027 DEBUG:boto:Port: 8001 2023-05-31T20:50:32.027 DEBUG:boto:Params: {} 2023-05-31T20:50:32.027 DEBUG:boto:Token: None 2023-05-31T20:50:32.028 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:50:32 GMT /pklxkd-59/obj68 2023-05-31T20:50:32.028 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:G3Rby2ZOO3QnyCEHkYE1aB3P1O0= 2023-05-31T20:50:32.029 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:50:32 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:G3Rby2ZOO3QnyCEHkYE1aB3P1O0=', 'Content-Length': '0'} 2023-05-31T20:50:32.030 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:43:08 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004736acd61040c780-006477b318-1c17-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:50:32 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:50:32.031 DEBUG:boto:path=/pklxkd-59/obj68 2023-05-31T20:50:32.031 DEBUG:boto:auth_path=/pklxkd-59/obj68 2023-05-31T20:50:32.032 DEBUG:boto:Method: GET 2023-05-31T20:50:32.032 DEBUG:boto:Path: /pklxkd-59/obj68 2023-05-31T20:50:32.033 DEBUG:boto:Data: 2023-05-31T20:50:32.033 DEBUG:boto:Headers: {} 2023-05-31T20:50:32.033 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:50:32.034 DEBUG:boto:Port: 8000 2023-05-31T20:50:32.034 DEBUG:boto:Params: {} 2023-05-31T20:50:32.034 DEBUG:boto:Token: None 2023-05-31T20:50:32.035 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:50:32 GMT /pklxkd-59/obj68 2023-05-31T20:50:32.035 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:G3Rby2ZOO3QnyCEHkYE1aB3P1O0= 2023-05-31T20:50:32.036 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:50:32 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:G3Rby2ZOO3QnyCEHkYE1aB3P1O0=', 'Content-Length': '0'} 2023-05-31T20:50:32.038 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:43:08 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006898884ac9d73890-006477b318-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:50:32 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:50:32.080 DEBUG:boto:path=/pklxkd-59/ 2023-05-31T20:50:32.081 DEBUG:boto:auth_path=/pklxkd-59/ 2023-05-31T20:50:32.081 DEBUG:boto:path=/pklxkd-59/?prefix=obj68 2023-05-31T20:50:32.082 DEBUG:boto:auth_path=/pklxkd-59/?prefix=obj68 2023-05-31T20:50:32.082 DEBUG:boto:Method: GET 2023-05-31T20:50:32.082 DEBUG:boto:Path: /pklxkd-59/?prefix=obj68 2023-05-31T20:50:32.083 DEBUG:boto:Data: 2023-05-31T20:50:32.083 DEBUG:boto:Headers: {} 2023-05-31T20:50:32.084 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:50:32.084 DEBUG:boto:Port: 8001 2023-05-31T20:50:32.084 DEBUG:boto:Params: {} 2023-05-31T20:50:32.085 DEBUG:boto:Token: None 2023-05-31T20:50:32.085 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:50:32 GMT /pklxkd-59/ 2023-05-31T20:50:32.086 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:s+yZS3HyomxcW/vxaV/hqlSq7Vg= 2023-05-31T20:50:32.086 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:50:32 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:s+yZS3HyomxcW/vxaV/hqlSq7Vg=', 'Content-Length': '0'} 2023-05-31T20:50:32.089 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000077aea78978398d40-006477b318-1c17-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:50:32 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:50:32.089 DEBUG:boto:b'pklxkd-59obj681000falseobj682023-05-31T20:43:08.066Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:50:32.090 DEBUG:boto:path=/pklxkd-59/ 2023-05-31T20:50:32.090 DEBUG:boto:auth_path=/pklxkd-59/ 2023-05-31T20:50:32.091 DEBUG:boto:path=/pklxkd-59/?prefix=obj68 2023-05-31T20:50:32.091 DEBUG:boto:auth_path=/pklxkd-59/?prefix=obj68 2023-05-31T20:50:32.091 DEBUG:boto:Method: GET 2023-05-31T20:50:32.092 DEBUG:boto:Path: /pklxkd-59/?prefix=obj68 2023-05-31T20:50:32.092 DEBUG:boto:Data: 2023-05-31T20:50:32.093 DEBUG:boto:Headers: {} 2023-05-31T20:50:32.093 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:50:32.093 DEBUG:boto:Port: 8000 2023-05-31T20:50:32.094 DEBUG:boto:Params: {} 2023-05-31T20:50:32.094 DEBUG:boto:Token: None 2023-05-31T20:50:32.095 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:50:32 GMT /pklxkd-59/ 2023-05-31T20:50:32.095 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:s+yZS3HyomxcW/vxaV/hqlSq7Vg= 2023-05-31T20:50:32.096 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:50:32 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:s+yZS3HyomxcW/vxaV/hqlSq7Vg=', 'Content-Length': '0'} 2023-05-31T20:50:32.099 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000960c33679dd737de-006477b318-1630-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:50:32 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:50:32.100 DEBUG:boto:b'pklxkd-59obj681000falseobj682023-05-31T20:43:08.066Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:50:32.101 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj69 2023-05-31T20:50:32.101 DEBUG:boto:path=/pklxkd-59/obj69 2023-05-31T20:50:32.102 DEBUG:boto:auth_path=/pklxkd-59/obj69 2023-05-31T20:50:32.102 DEBUG:boto:path=/pklxkd-59/obj69?versionId=null 2023-05-31T20:50:32.102 DEBUG:boto:auth_path=/pklxkd-59/obj69?versionId=null 2023-05-31T20:50:32.103 DEBUG:boto:Method: GET 2023-05-31T20:50:32.103 DEBUG:boto:Path: /pklxkd-59/obj69?versionId=null 2023-05-31T20:50:32.104 DEBUG:boto:Data: 2023-05-31T20:50:32.104 DEBUG:boto:Headers: {} 2023-05-31T20:50:32.104 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:50:32.105 DEBUG:boto:Port: 8001 2023-05-31T20:50:32.105 DEBUG:boto:Params: {} 2023-05-31T20:50:32.105 DEBUG:boto:Token: None 2023-05-31T20:50:32.106 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:50:32 GMT /pklxkd-59/obj69?versionId=null 2023-05-31T20:50:32.106 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:NfoODUAiIHundi+VhY2wkXkWW2k= 2023-05-31T20:50:32.107 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:50:32 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:NfoODUAiIHundi+VhY2wkXkWW2k=', 'Content-Length': '0'} 2023-05-31T20:50:32.109 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:46:33 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003e626ffc67c357c6-006477b318-1c17-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:50:32 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:50:32.156 DEBUG:boto:path=/pklxkd-59/obj69 2023-05-31T20:50:32.157 DEBUG:boto:auth_path=/pklxkd-59/obj69 2023-05-31T20:50:32.157 DEBUG:boto:path=/pklxkd-59/obj69?versionId=null 2023-05-31T20:50:32.158 DEBUG:boto:auth_path=/pklxkd-59/obj69?versionId=null 2023-05-31T20:50:32.158 DEBUG:boto:Method: GET 2023-05-31T20:50:32.158 DEBUG:boto:Path: /pklxkd-59/obj69?versionId=null 2023-05-31T20:50:32.159 DEBUG:boto:Data: 2023-05-31T20:50:32.159 DEBUG:boto:Headers: {} 2023-05-31T20:50:32.160 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:50:32.160 DEBUG:boto:Port: 8000 2023-05-31T20:50:32.160 DEBUG:boto:Params: {} 2023-05-31T20:50:32.161 DEBUG:boto:Token: None 2023-05-31T20:50:32.161 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:50:32 GMT /pklxkd-59/obj69?versionId=null 2023-05-31T20:50:32.162 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:NfoODUAiIHundi+VhY2wkXkWW2k= 2023-05-31T20:50:32.162 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:50:32 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:NfoODUAiIHundi+VhY2wkXkWW2k=', 'Content-Length': '0'} 2023-05-31T20:50:32.164 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:46:33 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002eb2d8fe84bda843-006477b318-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:50:32 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:50:32.165 DEBUG:boto:path=/pklxkd-59/obj69 2023-05-31T20:50:32.165 DEBUG:boto:auth_path=/pklxkd-59/obj69 2023-05-31T20:50:32.166 DEBUG:boto:path=/pklxkd-59/obj69?versionId=null 2023-05-31T20:50:32.166 DEBUG:boto:auth_path=/pklxkd-59/obj69?versionId=null 2023-05-31T20:50:32.166 DEBUG:boto:Method: HEAD 2023-05-31T20:50:32.167 DEBUG:boto:Path: /pklxkd-59/obj69?versionId=null 2023-05-31T20:50:32.167 DEBUG:boto:Data: 2023-05-31T20:50:32.168 DEBUG:boto:Headers: {} 2023-05-31T20:50:32.168 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:50:32.168 DEBUG:boto:Port: 8001 2023-05-31T20:50:32.169 DEBUG:boto:Params: {} 2023-05-31T20:50:32.169 DEBUG:boto:Token: None 2023-05-31T20:50:32.169 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:50:32 GMT /pklxkd-59/obj69?versionId=null 2023-05-31T20:50:32.170 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:eNaPeNQ2G07m14Be5WD0FRPzR+U= 2023-05-31T20:50:32.170 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:50:32 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:eNaPeNQ2G07m14Be5WD0FRPzR+U=', 'Content-Length': '0'} 2023-05-31T20:50:32.172 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:46:33 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d49a5823efbb50f8-006477b318-1c17-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:50:32 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:50:32.173 DEBUG:boto:path=/pklxkd-59/obj69 2023-05-31T20:50:32.173 DEBUG:boto:auth_path=/pklxkd-59/obj69 2023-05-31T20:50:32.173 DEBUG:boto:path=/pklxkd-59/obj69?versionId=null 2023-05-31T20:50:32.174 DEBUG:boto:auth_path=/pklxkd-59/obj69?versionId=null 2023-05-31T20:50:32.174 DEBUG:boto:Method: HEAD 2023-05-31T20:50:32.174 DEBUG:boto:Path: /pklxkd-59/obj69?versionId=null 2023-05-31T20:50:32.175 DEBUG:boto:Data: 2023-05-31T20:50:32.175 DEBUG:boto:Headers: {} 2023-05-31T20:50:32.175 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:50:32.176 DEBUG:boto:Port: 8000 2023-05-31T20:50:32.176 DEBUG:boto:Params: {} 2023-05-31T20:50:32.177 DEBUG:boto:Token: None 2023-05-31T20:50:32.177 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:50:32 GMT /pklxkd-59/obj69?versionId=null 2023-05-31T20:50:32.177 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:eNaPeNQ2G07m14Be5WD0FRPzR+U= 2023-05-31T20:50:32.178 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:50:32 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:eNaPeNQ2G07m14Be5WD0FRPzR+U=', 'Content-Length': '0'} 2023-05-31T20:50:32.180 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:46:33 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c7be4a2b1cf8cecd-006477b318-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:50:32 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:50:32.181 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj69 2023-05-31T20:50:32.181 DEBUG:boto:path=/pklxkd-59/obj69 2023-05-31T20:50:32.182 DEBUG:boto:auth_path=/pklxkd-59/obj69 2023-05-31T20:50:32.183 DEBUG:boto:path=/pklxkd-59/obj69?versionId=null 2023-05-31T20:50:32.183 DEBUG:boto:auth_path=/pklxkd-59/obj69?versionId=null 2023-05-31T20:50:32.184 DEBUG:boto:Method: GET 2023-05-31T20:50:32.185 DEBUG:boto:Path: /pklxkd-59/obj69?versionId=null 2023-05-31T20:50:32.185 DEBUG:boto:Data: 2023-05-31T20:50:32.186 DEBUG:boto:Headers: {} 2023-05-31T20:50:32.186 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:50:32.187 DEBUG:boto:Port: 8001 2023-05-31T20:50:32.188 DEBUG:boto:Params: {} 2023-05-31T20:50:32.188 DEBUG:boto:Token: None 2023-05-31T20:50:32.189 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:50:32 GMT /pklxkd-59/obj69?versionId=null 2023-05-31T20:50:32.190 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:NfoODUAiIHundi+VhY2wkXkWW2k= 2023-05-31T20:50:32.190 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:50:32 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:NfoODUAiIHundi+VhY2wkXkWW2k=', 'Content-Length': '0'} 2023-05-31T20:50:32.193 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:46:33 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000027cc5fc01613999e-006477b318-1c17-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:50:32 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:50:32.193 DEBUG:boto:path=/pklxkd-59/obj69 2023-05-31T20:50:32.194 DEBUG:boto:auth_path=/pklxkd-59/obj69 2023-05-31T20:50:32.194 DEBUG:boto:path=/pklxkd-59/obj69?versionId=null 2023-05-31T20:50:32.194 DEBUG:boto:auth_path=/pklxkd-59/obj69?versionId=null 2023-05-31T20:50:32.195 DEBUG:boto:Method: GET 2023-05-31T20:50:32.195 DEBUG:boto:Path: /pklxkd-59/obj69?versionId=null 2023-05-31T20:50:32.195 DEBUG:boto:Data: 2023-05-31T20:50:32.196 DEBUG:boto:Headers: {} 2023-05-31T20:50:32.196 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:50:32.197 DEBUG:boto:Port: 8000 2023-05-31T20:50:32.197 DEBUG:boto:Params: {} 2023-05-31T20:50:32.197 DEBUG:boto:Token: None 2023-05-31T20:50:32.198 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:50:32 GMT /pklxkd-59/obj69?versionId=null 2023-05-31T20:50:32.198 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:NfoODUAiIHundi+VhY2wkXkWW2k= 2023-05-31T20:50:32.199 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:50:32 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:NfoODUAiIHundi+VhY2wkXkWW2k=', 'Content-Length': '0'} 2023-05-31T20:50:32.200 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:46:33 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006ea084f69e724c78-006477b318-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:50:32 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:50:32.201 DEBUG:boto:path=/pklxkd-59/ 2023-05-31T20:50:32.202 DEBUG:boto:auth_path=/pklxkd-59/ 2023-05-31T20:50:32.202 DEBUG:boto:path=/pklxkd-59/?prefix=obj69 2023-05-31T20:50:32.202 DEBUG:boto:auth_path=/pklxkd-59/?prefix=obj69 2023-05-31T20:50:32.203 DEBUG:boto:Method: GET 2023-05-31T20:50:32.203 DEBUG:boto:Path: /pklxkd-59/?prefix=obj69 2023-05-31T20:50:32.203 DEBUG:boto:Data: 2023-05-31T20:50:32.204 DEBUG:boto:Headers: {} 2023-05-31T20:50:32.204 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:50:32.205 DEBUG:boto:Port: 8001 2023-05-31T20:50:32.205 DEBUG:boto:Params: {} 2023-05-31T20:50:32.205 DEBUG:boto:Token: None 2023-05-31T20:50:32.206 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:50:32 GMT /pklxkd-59/ 2023-05-31T20:50:32.206 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:s+yZS3HyomxcW/vxaV/hqlSq7Vg= 2023-05-31T20:50:32.207 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:50:32 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:s+yZS3HyomxcW/vxaV/hqlSq7Vg=', 'Content-Length': '0'} 2023-05-31T20:50:32.209 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000465a0b3769c5f34d-006477b318-1c17-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:50:32 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:50:32.210 DEBUG:boto:b'pklxkd-59obj691000falseobj692023-05-31T20:46:33.886Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:50:32.210 DEBUG:boto:path=/pklxkd-59/ 2023-05-31T20:50:32.211 DEBUG:boto:auth_path=/pklxkd-59/ 2023-05-31T20:50:32.211 DEBUG:boto:path=/pklxkd-59/?prefix=obj69 2023-05-31T20:50:32.211 DEBUG:boto:auth_path=/pklxkd-59/?prefix=obj69 2023-05-31T20:50:32.212 DEBUG:boto:Method: GET 2023-05-31T20:50:32.212 DEBUG:boto:Path: /pklxkd-59/?prefix=obj69 2023-05-31T20:50:32.213 DEBUG:boto:Data: 2023-05-31T20:50:32.213 DEBUG:boto:Headers: {} 2023-05-31T20:50:32.214 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:50:32.215 DEBUG:boto:Port: 8000 2023-05-31T20:50:32.215 DEBUG:boto:Params: {} 2023-05-31T20:50:32.216 DEBUG:boto:Token: None 2023-05-31T20:50:32.216 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:50:32 GMT /pklxkd-59/ 2023-05-31T20:50:32.217 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:s+yZS3HyomxcW/vxaV/hqlSq7Vg= 2023-05-31T20:50:32.217 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:50:32 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:s+yZS3HyomxcW/vxaV/hqlSq7Vg=', 'Content-Length': '0'} 2023-05-31T20:50:32.220 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000004d6cd84c942701a0-006477b318-1630-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:50:32 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:50:32.264 DEBUG:boto:b'pklxkd-59obj691000falseobj692023-05-31T20:46:33.886Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:50:32.265 DEBUG:boto:path=/pklxkd-59/obj69 2023-05-31T20:50:32.265 DEBUG:boto:auth_path=/pklxkd-59/obj69 2023-05-31T20:50:32.266 DEBUG:boto:Method: HEAD 2023-05-31T20:50:32.266 DEBUG:boto:Path: /pklxkd-59/obj69 2023-05-31T20:50:32.267 DEBUG:boto:Data: 2023-05-31T20:50:32.267 DEBUG:boto:Headers: {} 2023-05-31T20:50:32.267 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:50:32.268 DEBUG:boto:Port: 8001 2023-05-31T20:50:32.268 DEBUG:boto:Params: {} 2023-05-31T20:50:32.269 DEBUG:boto:Token: None 2023-05-31T20:50:32.269 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:50:32 GMT /pklxkd-59/obj69 2023-05-31T20:50:32.270 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:he593T2U1NDbS9DSxenqW2sAils= 2023-05-31T20:50:32.270 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:50:32 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:he593T2U1NDbS9DSxenqW2sAils=', 'Content-Length': '0'} 2023-05-31T20:50:32.272 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:46:33 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000fce2bc83a005dfb0-006477b318-1c17-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:50:32 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:50:32.272 DEBUG:boto:path=/pklxkd-59/obj69 2023-05-31T20:50:32.273 DEBUG:boto:auth_path=/pklxkd-59/obj69 2023-05-31T20:50:32.273 DEBUG:boto:Method: HEAD 2023-05-31T20:50:32.274 DEBUG:boto:Path: /pklxkd-59/obj69 2023-05-31T20:50:32.274 DEBUG:boto:Data: 2023-05-31T20:50:32.274 DEBUG:boto:Headers: {} 2023-05-31T20:50:32.275 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:50:32.275 DEBUG:boto:Port: 8000 2023-05-31T20:50:32.275 DEBUG:boto:Params: {} 2023-05-31T20:50:32.276 DEBUG:boto:Token: None 2023-05-31T20:50:32.276 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:50:32 GMT /pklxkd-59/obj69 2023-05-31T20:50:32.277 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:he593T2U1NDbS9DSxenqW2sAils= 2023-05-31T20:50:32.277 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:50:32 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:he593T2U1NDbS9DSxenqW2sAils=', 'Content-Length': '0'} 2023-05-31T20:50:32.279 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:46:33 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000021d9fd61d5acc8b3-006477b318-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:50:32 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:50:32.279 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj69 2023-05-31T20:50:32.280 DEBUG:boto:path=/pklxkd-59/obj69 2023-05-31T20:50:32.280 DEBUG:boto:auth_path=/pklxkd-59/obj69 2023-05-31T20:50:32.281 DEBUG:boto:Method: GET 2023-05-31T20:50:32.281 DEBUG:boto:Path: /pklxkd-59/obj69 2023-05-31T20:50:32.281 DEBUG:boto:Data: 2023-05-31T20:50:32.282 DEBUG:boto:Headers: {} 2023-05-31T20:50:32.282 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:50:32.282 DEBUG:boto:Port: 8001 2023-05-31T20:50:32.283 DEBUG:boto:Params: {} 2023-05-31T20:50:32.283 DEBUG:boto:Token: None 2023-05-31T20:50:32.284 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:50:32 GMT /pklxkd-59/obj69 2023-05-31T20:50:32.284 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:eyo5UXsyFDpI3bNzs+ht3XoD2MI= 2023-05-31T20:50:32.284 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:50:32 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:eyo5UXsyFDpI3bNzs+ht3XoD2MI=', 'Content-Length': '0'} 2023-05-31T20:50:32.286 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:46:33 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000039977ca22a91ecd8-006477b318-1c17-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:50:32 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:50:32.287 DEBUG:boto:path=/pklxkd-59/obj69 2023-05-31T20:50:32.287 DEBUG:boto:auth_path=/pklxkd-59/obj69 2023-05-31T20:50:32.288 DEBUG:boto:Method: GET 2023-05-31T20:50:32.288 DEBUG:boto:Path: /pklxkd-59/obj69 2023-05-31T20:50:32.289 DEBUG:boto:Data: 2023-05-31T20:50:32.289 DEBUG:boto:Headers: {} 2023-05-31T20:50:32.289 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:50:32.290 DEBUG:boto:Port: 8000 2023-05-31T20:50:32.290 DEBUG:boto:Params: {} 2023-05-31T20:50:32.290 DEBUG:boto:Token: None 2023-05-31T20:50:32.291 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:50:32 GMT /pklxkd-59/obj69 2023-05-31T20:50:32.291 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:eyo5UXsyFDpI3bNzs+ht3XoD2MI= 2023-05-31T20:50:32.292 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:50:32 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:eyo5UXsyFDpI3bNzs+ht3XoD2MI=', 'Content-Length': '0'} 2023-05-31T20:50:32.294 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:46:33 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000126aabfbe8d7ff62-006477b318-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:50:32 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:50:32.294 DEBUG:boto:path=/pklxkd-59/ 2023-05-31T20:50:32.295 DEBUG:boto:auth_path=/pklxkd-59/ 2023-05-31T20:50:32.295 DEBUG:boto:path=/pklxkd-59/?prefix=obj69 2023-05-31T20:50:32.295 DEBUG:boto:auth_path=/pklxkd-59/?prefix=obj69 2023-05-31T20:50:32.296 DEBUG:boto:Method: GET 2023-05-31T20:50:32.296 DEBUG:boto:Path: /pklxkd-59/?prefix=obj69 2023-05-31T20:50:32.297 DEBUG:boto:Data: 2023-05-31T20:50:32.297 DEBUG:boto:Headers: {} 2023-05-31T20:50:32.297 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:50:32.298 DEBUG:boto:Port: 8001 2023-05-31T20:50:32.298 DEBUG:boto:Params: {} 2023-05-31T20:50:32.298 DEBUG:boto:Token: None 2023-05-31T20:50:32.299 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:50:32 GMT /pklxkd-59/ 2023-05-31T20:50:32.299 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:s+yZS3HyomxcW/vxaV/hqlSq7Vg= 2023-05-31T20:50:32.300 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:50:32 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:s+yZS3HyomxcW/vxaV/hqlSq7Vg=', 'Content-Length': '0'} 2023-05-31T20:50:32.302 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d79904599f787c01-006477b318-1c17-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:50:32 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:50:32.303 DEBUG:boto:b'pklxkd-59obj691000falseobj692023-05-31T20:46:33.886Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:50:32.303 DEBUG:boto:path=/pklxkd-59/ 2023-05-31T20:50:32.304 DEBUG:boto:auth_path=/pklxkd-59/ 2023-05-31T20:50:32.304 DEBUG:boto:path=/pklxkd-59/?prefix=obj69 2023-05-31T20:50:32.304 DEBUG:boto:auth_path=/pklxkd-59/?prefix=obj69 2023-05-31T20:50:32.305 DEBUG:boto:Method: GET 2023-05-31T20:50:32.305 DEBUG:boto:Path: /pklxkd-59/?prefix=obj69 2023-05-31T20:50:32.305 DEBUG:boto:Data: 2023-05-31T20:50:32.306 DEBUG:boto:Headers: {} 2023-05-31T20:50:32.306 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:50:32.306 DEBUG:boto:Port: 8000 2023-05-31T20:50:32.307 DEBUG:boto:Params: {} 2023-05-31T20:50:32.307 DEBUG:boto:Token: None 2023-05-31T20:50:32.308 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:50:32 GMT /pklxkd-59/ 2023-05-31T20:50:32.308 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:s+yZS3HyomxcW/vxaV/hqlSq7Vg= 2023-05-31T20:50:32.308 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:50:32 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:s+yZS3HyomxcW/vxaV/hqlSq7Vg=', 'Content-Length': '0'} 2023-05-31T20:50:32.312 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000006d2a168497b20804-006477b318-1630-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:50:32 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:50:32.312 DEBUG:boto:b'pklxkd-59obj691000falseobj692023-05-31T20:46:33.886Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:50:32.313 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj76 2023-05-31T20:50:32.313 DEBUG:boto:path=/pklxkd-59/obj76 2023-05-31T20:50:32.314 DEBUG:boto:auth_path=/pklxkd-59/obj76 2023-05-31T20:50:32.314 DEBUG:boto:path=/pklxkd-59/obj76?versionId=null 2023-05-31T20:50:32.314 DEBUG:boto:auth_path=/pklxkd-59/obj76?versionId=null 2023-05-31T20:50:32.315 DEBUG:boto:Method: GET 2023-05-31T20:50:32.315 DEBUG:boto:Path: /pklxkd-59/obj76?versionId=null 2023-05-31T20:50:32.315 DEBUG:boto:Data: 2023-05-31T20:50:32.316 DEBUG:boto:Headers: {} 2023-05-31T20:50:32.316 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:50:32.317 DEBUG:boto:Port: 8001 2023-05-31T20:50:32.317 DEBUG:boto:Params: {} 2023-05-31T20:50:32.317 DEBUG:boto:Token: None 2023-05-31T20:50:32.318 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:50:32 GMT /pklxkd-59/obj76?versionId=null 2023-05-31T20:50:32.318 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:7VQZu3CGTTlOdxS9pqB138+dIPk= 2023-05-31T20:50:32.319 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:50:32 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:7VQZu3CGTTlOdxS9pqB138+dIPk=', 'Content-Length': '0'} 2023-05-31T20:50:32.320 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:44:50 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000052f0cfd933f97b65-006477b318-1c17-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:50:32 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:50:32.321 DEBUG:boto:path=/pklxkd-59/obj76 2023-05-31T20:50:32.321 DEBUG:boto:auth_path=/pklxkd-59/obj76 2023-05-31T20:50:32.322 DEBUG:boto:path=/pklxkd-59/obj76?versionId=null 2023-05-31T20:50:32.322 DEBUG:boto:auth_path=/pklxkd-59/obj76?versionId=null 2023-05-31T20:50:32.322 DEBUG:boto:Method: GET 2023-05-31T20:50:32.323 DEBUG:boto:Path: /pklxkd-59/obj76?versionId=null 2023-05-31T20:50:32.323 DEBUG:boto:Data: 2023-05-31T20:50:32.324 DEBUG:boto:Headers: {} 2023-05-31T20:50:32.324 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:50:32.324 DEBUG:boto:Port: 8000 2023-05-31T20:50:32.325 DEBUG:boto:Params: {} 2023-05-31T20:50:32.325 DEBUG:boto:Token: None 2023-05-31T20:50:32.326 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:50:32 GMT /pklxkd-59/obj76?versionId=null 2023-05-31T20:50:32.326 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:7VQZu3CGTTlOdxS9pqB138+dIPk= 2023-05-31T20:50:32.327 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:50:32 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:7VQZu3CGTTlOdxS9pqB138+dIPk=', 'Content-Length': '0'} 2023-05-31T20:50:32.328 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:44:50 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000559301ddf674828b-006477b318-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:50:32 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:50:32.329 DEBUG:boto:path=/pklxkd-59/obj76 2023-05-31T20:50:32.329 DEBUG:boto:auth_path=/pklxkd-59/obj76 2023-05-31T20:50:32.330 DEBUG:boto:path=/pklxkd-59/obj76?versionId=null 2023-05-31T20:50:32.330 DEBUG:boto:auth_path=/pklxkd-59/obj76?versionId=null 2023-05-31T20:50:32.330 DEBUG:boto:Method: HEAD 2023-05-31T20:50:32.331 DEBUG:boto:Path: /pklxkd-59/obj76?versionId=null 2023-05-31T20:50:32.331 DEBUG:boto:Data: 2023-05-31T20:50:32.331 DEBUG:boto:Headers: {} 2023-05-31T20:50:32.332 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:50:32.332 DEBUG:boto:Port: 8001 2023-05-31T20:50:32.332 DEBUG:boto:Params: {} 2023-05-31T20:50:32.333 DEBUG:boto:Token: None 2023-05-31T20:50:32.333 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:50:32 GMT /pklxkd-59/obj76?versionId=null 2023-05-31T20:50:32.333 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:Al1CWD1sV2OEKaOF8tJ6ZICpd8Q= 2023-05-31T20:50:32.334 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:50:32 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:Al1CWD1sV2OEKaOF8tJ6ZICpd8Q=', 'Content-Length': '0'} 2023-05-31T20:50:32.336 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:44:50 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008eeb342a2bb0f047-006477b318-1c17-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:50:32 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:50:32.336 DEBUG:boto:path=/pklxkd-59/obj76 2023-05-31T20:50:32.336 DEBUG:boto:auth_path=/pklxkd-59/obj76 2023-05-31T20:50:32.337 DEBUG:boto:path=/pklxkd-59/obj76?versionId=null 2023-05-31T20:50:32.337 DEBUG:boto:auth_path=/pklxkd-59/obj76?versionId=null 2023-05-31T20:50:32.338 DEBUG:boto:Method: HEAD 2023-05-31T20:50:32.338 DEBUG:boto:Path: /pklxkd-59/obj76?versionId=null 2023-05-31T20:50:32.338 DEBUG:boto:Data: 2023-05-31T20:50:32.339 DEBUG:boto:Headers: {} 2023-05-31T20:50:32.339 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:50:32.339 DEBUG:boto:Port: 8000 2023-05-31T20:50:32.340 DEBUG:boto:Params: {} 2023-05-31T20:50:32.340 DEBUG:boto:Token: None 2023-05-31T20:50:32.340 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:50:32 GMT /pklxkd-59/obj76?versionId=null 2023-05-31T20:50:32.341 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:Al1CWD1sV2OEKaOF8tJ6ZICpd8Q= 2023-05-31T20:50:32.341 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:50:32 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:Al1CWD1sV2OEKaOF8tJ6ZICpd8Q=', 'Content-Length': '0'} 2023-05-31T20:50:32.343 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:44:50 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000022512d51f5a8748d-006477b318-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:50:32 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:50:32.344 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj76 2023-05-31T20:50:32.344 DEBUG:boto:path=/pklxkd-59/obj76 2023-05-31T20:50:32.344 DEBUG:boto:auth_path=/pklxkd-59/obj76 2023-05-31T20:50:32.345 DEBUG:boto:path=/pklxkd-59/obj76?versionId=null 2023-05-31T20:50:32.345 DEBUG:boto:auth_path=/pklxkd-59/obj76?versionId=null 2023-05-31T20:50:32.345 DEBUG:boto:Method: GET 2023-05-31T20:50:32.346 DEBUG:boto:Path: /pklxkd-59/obj76?versionId=null 2023-05-31T20:50:32.346 DEBUG:boto:Data: 2023-05-31T20:50:32.347 DEBUG:boto:Headers: {} 2023-05-31T20:50:32.347 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:50:32.347 DEBUG:boto:Port: 8001 2023-05-31T20:50:32.348 DEBUG:boto:Params: {} 2023-05-31T20:50:32.348 DEBUG:boto:Token: None 2023-05-31T20:50:32.348 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:50:32 GMT /pklxkd-59/obj76?versionId=null 2023-05-31T20:50:32.349 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:7VQZu3CGTTlOdxS9pqB138+dIPk= 2023-05-31T20:50:32.349 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:50:32 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:7VQZu3CGTTlOdxS9pqB138+dIPk=', 'Content-Length': '0'} 2023-05-31T20:50:32.351 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:44:50 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000beeeb97bd9eee2e1-006477b318-1c17-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:50:32 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:50:32.392 DEBUG:boto:path=/pklxkd-59/obj76 2023-05-31T20:50:32.393 DEBUG:boto:auth_path=/pklxkd-59/obj76 2023-05-31T20:50:32.393 DEBUG:boto:path=/pklxkd-59/obj76?versionId=null 2023-05-31T20:50:32.393 DEBUG:boto:auth_path=/pklxkd-59/obj76?versionId=null 2023-05-31T20:50:32.394 DEBUG:boto:Method: GET 2023-05-31T20:50:32.394 DEBUG:boto:Path: /pklxkd-59/obj76?versionId=null 2023-05-31T20:50:32.395 DEBUG:boto:Data: 2023-05-31T20:50:32.395 DEBUG:boto:Headers: {} 2023-05-31T20:50:32.395 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:50:32.396 DEBUG:boto:Port: 8000 2023-05-31T20:50:32.396 DEBUG:boto:Params: {} 2023-05-31T20:50:32.397 DEBUG:boto:Token: None 2023-05-31T20:50:32.397 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:50:32 GMT /pklxkd-59/obj76?versionId=null 2023-05-31T20:50:32.397 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:7VQZu3CGTTlOdxS9pqB138+dIPk= 2023-05-31T20:50:32.398 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:50:32 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:7VQZu3CGTTlOdxS9pqB138+dIPk=', 'Content-Length': '0'} 2023-05-31T20:50:32.400 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:44:50 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000096a0011aead388d9-006477b318-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:50:32 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:50:32.400 DEBUG:boto:path=/pklxkd-59/ 2023-05-31T20:50:32.401 DEBUG:boto:auth_path=/pklxkd-59/ 2023-05-31T20:50:32.401 DEBUG:boto:path=/pklxkd-59/?prefix=obj76 2023-05-31T20:50:32.401 DEBUG:boto:auth_path=/pklxkd-59/?prefix=obj76 2023-05-31T20:50:32.402 DEBUG:boto:Method: GET 2023-05-31T20:50:32.402 DEBUG:boto:Path: /pklxkd-59/?prefix=obj76 2023-05-31T20:50:32.402 DEBUG:boto:Data: 2023-05-31T20:50:32.403 DEBUG:boto:Headers: {} 2023-05-31T20:50:32.403 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:50:32.404 DEBUG:boto:Port: 8001 2023-05-31T20:50:32.404 DEBUG:boto:Params: {} 2023-05-31T20:50:32.404 DEBUG:boto:Token: None 2023-05-31T20:50:32.405 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:50:32 GMT /pklxkd-59/ 2023-05-31T20:50:32.405 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:s+yZS3HyomxcW/vxaV/hqlSq7Vg= 2023-05-31T20:50:32.406 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:50:32 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:s+yZS3HyomxcW/vxaV/hqlSq7Vg=', 'Content-Length': '0'} 2023-05-31T20:50:32.408 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000100a1517cbb5b30a-006477b318-1c17-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:50:32 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:50:32.408 DEBUG:boto:b'pklxkd-59obj761000falseobj762023-05-31T20:44:50.682Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:50:32.409 DEBUG:boto:path=/pklxkd-59/ 2023-05-31T20:50:32.409 DEBUG:boto:auth_path=/pklxkd-59/ 2023-05-31T20:50:32.410 DEBUG:boto:path=/pklxkd-59/?prefix=obj76 2023-05-31T20:50:32.410 DEBUG:boto:auth_path=/pklxkd-59/?prefix=obj76 2023-05-31T20:50:32.410 DEBUG:boto:Method: GET 2023-05-31T20:50:32.411 DEBUG:boto:Path: /pklxkd-59/?prefix=obj76 2023-05-31T20:50:32.411 DEBUG:boto:Data: 2023-05-31T20:50:32.412 DEBUG:boto:Headers: {} 2023-05-31T20:50:32.412 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:50:32.412 DEBUG:boto:Port: 8000 2023-05-31T20:50:32.413 DEBUG:boto:Params: {} 2023-05-31T20:50:32.413 DEBUG:boto:Token: None 2023-05-31T20:50:32.413 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:50:32 GMT /pklxkd-59/ 2023-05-31T20:50:32.414 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:s+yZS3HyomxcW/vxaV/hqlSq7Vg= 2023-05-31T20:50:32.414 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:50:32 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:s+yZS3HyomxcW/vxaV/hqlSq7Vg=', 'Content-Length': '0'} 2023-05-31T20:50:32.417 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000008e71e24d73bb708e-006477b318-1630-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:50:32 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:50:32.418 DEBUG:boto:b'pklxkd-59obj761000falseobj762023-05-31T20:44:50.682Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:50:32.418 DEBUG:boto:path=/pklxkd-59/obj76 2023-05-31T20:50:32.419 DEBUG:boto:auth_path=/pklxkd-59/obj76 2023-05-31T20:50:32.419 DEBUG:boto:Method: HEAD 2023-05-31T20:50:32.420 DEBUG:boto:Path: /pklxkd-59/obj76 2023-05-31T20:50:32.420 DEBUG:boto:Data: 2023-05-31T20:50:32.420 DEBUG:boto:Headers: {} 2023-05-31T20:50:32.421 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:50:32.421 DEBUG:boto:Port: 8001 2023-05-31T20:50:32.421 DEBUG:boto:Params: {} 2023-05-31T20:50:32.422 DEBUG:boto:Token: None 2023-05-31T20:50:32.422 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:50:32 GMT /pklxkd-59/obj76 2023-05-31T20:50:32.423 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:00feZ0O8Ors3jFUuTgtJXv8/VOc= 2023-05-31T20:50:32.423 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:50:32 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:00feZ0O8Ors3jFUuTgtJXv8/VOc=', 'Content-Length': '0'} 2023-05-31T20:50:32.425 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:44:50 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c183546524296b2d-006477b318-1c17-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:50:32 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:50:32.425 DEBUG:boto:path=/pklxkd-59/obj76 2023-05-31T20:50:32.426 DEBUG:boto:auth_path=/pklxkd-59/obj76 2023-05-31T20:50:32.426 DEBUG:boto:Method: HEAD 2023-05-31T20:50:32.426 DEBUG:boto:Path: /pklxkd-59/obj76 2023-05-31T20:50:32.427 DEBUG:boto:Data: 2023-05-31T20:50:32.427 DEBUG:boto:Headers: {} 2023-05-31T20:50:32.427 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:50:32.428 DEBUG:boto:Port: 8000 2023-05-31T20:50:32.428 DEBUG:boto:Params: {} 2023-05-31T20:50:32.429 DEBUG:boto:Token: None 2023-05-31T20:50:32.429 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:50:32 GMT /pklxkd-59/obj76 2023-05-31T20:50:32.429 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:00feZ0O8Ors3jFUuTgtJXv8/VOc= 2023-05-31T20:50:32.430 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:50:32 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:00feZ0O8Ors3jFUuTgtJXv8/VOc=', 'Content-Length': '0'} 2023-05-31T20:50:32.431 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:44:50 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000bc2762f70b09ce95-006477b318-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:50:32 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:50:32.432 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj76 2023-05-31T20:50:32.432 DEBUG:boto:path=/pklxkd-59/obj76 2023-05-31T20:50:32.433 DEBUG:boto:auth_path=/pklxkd-59/obj76 2023-05-31T20:50:32.433 DEBUG:boto:Method: GET 2023-05-31T20:50:32.433 DEBUG:boto:Path: /pklxkd-59/obj76 2023-05-31T20:50:32.434 DEBUG:boto:Data: 2023-05-31T20:50:32.434 DEBUG:boto:Headers: {} 2023-05-31T20:50:32.435 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:50:32.435 DEBUG:boto:Port: 8001 2023-05-31T20:50:32.435 DEBUG:boto:Params: {} 2023-05-31T20:50:32.436 DEBUG:boto:Token: None 2023-05-31T20:50:32.436 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:50:32 GMT /pklxkd-59/obj76 2023-05-31T20:50:32.437 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:/NZQG7KF4OgAwSEs9Tr2ElmwRyM= 2023-05-31T20:50:32.437 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:50:32 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:/NZQG7KF4OgAwSEs9Tr2ElmwRyM=', 'Content-Length': '0'} 2023-05-31T20:50:32.439 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:44:50 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a40f6a2d4fcc6d7a-006477b318-1c17-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:50:32 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:50:32.439 DEBUG:boto:path=/pklxkd-59/obj76 2023-05-31T20:50:32.440 DEBUG:boto:auth_path=/pklxkd-59/obj76 2023-05-31T20:50:32.440 DEBUG:boto:Method: GET 2023-05-31T20:50:32.441 DEBUG:boto:Path: /pklxkd-59/obj76 2023-05-31T20:50:32.441 DEBUG:boto:Data: 2023-05-31T20:50:32.441 DEBUG:boto:Headers: {} 2023-05-31T20:50:32.442 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:50:32.442 DEBUG:boto:Port: 8000 2023-05-31T20:50:32.442 DEBUG:boto:Params: {} 2023-05-31T20:50:32.443 DEBUG:boto:Token: None 2023-05-31T20:50:32.443 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:50:32 GMT /pklxkd-59/obj76 2023-05-31T20:50:32.444 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:/NZQG7KF4OgAwSEs9Tr2ElmwRyM= 2023-05-31T20:50:32.444 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:50:32 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:/NZQG7KF4OgAwSEs9Tr2ElmwRyM=', 'Content-Length': '0'} 2023-05-31T20:50:32.446 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:44:50 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000004971576a3a81e1c-006477b318-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:50:32 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:50:32.488 DEBUG:boto:path=/pklxkd-59/ 2023-05-31T20:50:32.489 DEBUG:boto:auth_path=/pklxkd-59/ 2023-05-31T20:50:32.489 DEBUG:boto:path=/pklxkd-59/?prefix=obj76 2023-05-31T20:50:32.489 DEBUG:boto:auth_path=/pklxkd-59/?prefix=obj76 2023-05-31T20:50:32.490 DEBUG:boto:Method: GET 2023-05-31T20:50:32.490 DEBUG:boto:Path: /pklxkd-59/?prefix=obj76 2023-05-31T20:50:32.490 DEBUG:boto:Data: 2023-05-31T20:50:32.491 DEBUG:boto:Headers: {} 2023-05-31T20:50:32.491 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:50:32.492 DEBUG:boto:Port: 8001 2023-05-31T20:50:32.492 DEBUG:boto:Params: {} 2023-05-31T20:50:32.492 DEBUG:boto:Token: None 2023-05-31T20:50:32.493 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:50:32 GMT /pklxkd-59/ 2023-05-31T20:50:32.493 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:s+yZS3HyomxcW/vxaV/hqlSq7Vg= 2023-05-31T20:50:32.494 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:50:32 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:s+yZS3HyomxcW/vxaV/hqlSq7Vg=', 'Content-Length': '0'} 2023-05-31T20:50:32.496 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000535e55625ef77215-006477b318-1c17-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:50:32 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:50:32.497 DEBUG:boto:b'pklxkd-59obj761000falseobj762023-05-31T20:44:50.682Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:50:32.497 DEBUG:boto:path=/pklxkd-59/ 2023-05-31T20:50:32.498 DEBUG:boto:auth_path=/pklxkd-59/ 2023-05-31T20:50:32.498 DEBUG:boto:path=/pklxkd-59/?prefix=obj76 2023-05-31T20:50:32.498 DEBUG:boto:auth_path=/pklxkd-59/?prefix=obj76 2023-05-31T20:50:32.499 DEBUG:boto:Method: GET 2023-05-31T20:50:32.499 DEBUG:boto:Path: /pklxkd-59/?prefix=obj76 2023-05-31T20:50:32.499 DEBUG:boto:Data: 2023-05-31T20:50:32.500 DEBUG:boto:Headers: {} 2023-05-31T20:50:32.500 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:50:32.500 DEBUG:boto:Port: 8000 2023-05-31T20:50:32.501 DEBUG:boto:Params: {} 2023-05-31T20:50:32.501 DEBUG:boto:Token: None 2023-05-31T20:50:32.502 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:50:32 GMT /pklxkd-59/ 2023-05-31T20:50:32.502 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:s+yZS3HyomxcW/vxaV/hqlSq7Vg= 2023-05-31T20:50:32.503 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:50:32 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:s+yZS3HyomxcW/vxaV/hqlSq7Vg=', 'Content-Length': '0'} 2023-05-31T20:50:32.506 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b5bd4fd5e8965d77-006477b318-1630-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:50:32 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:50:32.506 DEBUG:boto:b'pklxkd-59obj761000falseobj762023-05-31T20:44:50.682Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:50:32.507 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj85 2023-05-31T20:50:32.507 DEBUG:boto:path=/pklxkd-59/obj85 2023-05-31T20:50:32.508 DEBUG:boto:auth_path=/pklxkd-59/obj85 2023-05-31T20:50:32.508 DEBUG:boto:path=/pklxkd-59/obj85?versionId=null 2023-05-31T20:50:32.509 DEBUG:boto:auth_path=/pklxkd-59/obj85?versionId=null 2023-05-31T20:50:32.509 DEBUG:boto:Method: GET 2023-05-31T20:50:32.509 DEBUG:boto:Path: /pklxkd-59/obj85?versionId=null 2023-05-31T20:50:32.510 DEBUG:boto:Data: 2023-05-31T20:50:32.510 DEBUG:boto:Headers: {} 2023-05-31T20:50:32.511 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:50:32.511 DEBUG:boto:Port: 8001 2023-05-31T20:50:32.511 DEBUG:boto:Params: {} 2023-05-31T20:50:32.512 DEBUG:boto:Token: None 2023-05-31T20:50:32.512 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:50:32 GMT /pklxkd-59/obj85?versionId=null 2023-05-31T20:50:32.513 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:EDlrqNYHiExlw9ndjlo+O81u9q8= 2023-05-31T20:50:32.513 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:50:32 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:EDlrqNYHiExlw9ndjlo+O81u9q8=', 'Content-Length': '0'} 2023-05-31T20:50:32.515 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:43:08 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009a2fc91c85bc216f-006477b318-1c17-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:50:32 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:50:32.556 DEBUG:boto:path=/pklxkd-59/obj85 2023-05-31T20:50:32.557 DEBUG:boto:auth_path=/pklxkd-59/obj85 2023-05-31T20:50:32.557 DEBUG:boto:path=/pklxkd-59/obj85?versionId=null 2023-05-31T20:50:32.557 DEBUG:boto:auth_path=/pklxkd-59/obj85?versionId=null 2023-05-31T20:50:32.558 DEBUG:boto:Method: GET 2023-05-31T20:50:32.558 DEBUG:boto:Path: /pklxkd-59/obj85?versionId=null 2023-05-31T20:50:32.559 DEBUG:boto:Data: 2023-05-31T20:50:32.559 DEBUG:boto:Headers: {} 2023-05-31T20:50:32.559 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:50:32.560 DEBUG:boto:Port: 8000 2023-05-31T20:50:32.560 DEBUG:boto:Params: {} 2023-05-31T20:50:32.561 DEBUG:boto:Token: None 2023-05-31T20:50:32.561 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:50:32 GMT /pklxkd-59/obj85?versionId=null 2023-05-31T20:50:32.561 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:EDlrqNYHiExlw9ndjlo+O81u9q8= 2023-05-31T20:50:32.562 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:50:32 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:EDlrqNYHiExlw9ndjlo+O81u9q8=', 'Content-Length': '0'} 2023-05-31T20:50:32.564 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:43:08 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a3ebeb0c1b7bcacc-006477b318-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:50:32 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:50:32.565 DEBUG:boto:path=/pklxkd-59/obj85 2023-05-31T20:50:32.565 DEBUG:boto:auth_path=/pklxkd-59/obj85 2023-05-31T20:50:32.565 DEBUG:boto:path=/pklxkd-59/obj85?versionId=null 2023-05-31T20:50:32.566 DEBUG:boto:auth_path=/pklxkd-59/obj85?versionId=null 2023-05-31T20:50:32.566 DEBUG:boto:Method: HEAD 2023-05-31T20:50:32.566 DEBUG:boto:Path: /pklxkd-59/obj85?versionId=null 2023-05-31T20:50:32.567 DEBUG:boto:Data: 2023-05-31T20:50:32.567 DEBUG:boto:Headers: {} 2023-05-31T20:50:32.567 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:50:32.568 DEBUG:boto:Port: 8001 2023-05-31T20:50:32.568 DEBUG:boto:Params: {} 2023-05-31T20:50:32.568 DEBUG:boto:Token: None 2023-05-31T20:50:32.569 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:50:32 GMT /pklxkd-59/obj85?versionId=null 2023-05-31T20:50:32.569 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:tylwfreERWvevK3OeIWFIJhtfm4= 2023-05-31T20:50:32.570 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:50:32 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:tylwfreERWvevK3OeIWFIJhtfm4=', 'Content-Length': '0'} 2023-05-31T20:50:32.571 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:43:08 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000025a0cac7a83f5e0f-006477b318-1c17-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:50:32 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:50:32.572 DEBUG:boto:path=/pklxkd-59/obj85 2023-05-31T20:50:32.572 DEBUG:boto:auth_path=/pklxkd-59/obj85 2023-05-31T20:50:32.573 DEBUG:boto:path=/pklxkd-59/obj85?versionId=null 2023-05-31T20:50:32.573 DEBUG:boto:auth_path=/pklxkd-59/obj85?versionId=null 2023-05-31T20:50:32.573 DEBUG:boto:Method: HEAD 2023-05-31T20:50:32.574 DEBUG:boto:Path: /pklxkd-59/obj85?versionId=null 2023-05-31T20:50:32.574 DEBUG:boto:Data: 2023-05-31T20:50:32.574 DEBUG:boto:Headers: {} 2023-05-31T20:50:32.575 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:50:32.575 DEBUG:boto:Port: 8000 2023-05-31T20:50:32.575 DEBUG:boto:Params: {} 2023-05-31T20:50:32.576 DEBUG:boto:Token: None 2023-05-31T20:50:32.576 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:50:32 GMT /pklxkd-59/obj85?versionId=null 2023-05-31T20:50:32.577 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:tylwfreERWvevK3OeIWFIJhtfm4= 2023-05-31T20:50:32.577 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:50:32 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:tylwfreERWvevK3OeIWFIJhtfm4=', 'Content-Length': '0'} 2023-05-31T20:50:32.579 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:43:08 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000485539a705247010-006477b318-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:50:32 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:50:32.579 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj85 2023-05-31T20:50:32.580 DEBUG:boto:path=/pklxkd-59/obj85 2023-05-31T20:50:32.580 DEBUG:boto:auth_path=/pklxkd-59/obj85 2023-05-31T20:50:32.581 DEBUG:boto:path=/pklxkd-59/obj85?versionId=null 2023-05-31T20:50:32.581 DEBUG:boto:auth_path=/pklxkd-59/obj85?versionId=null 2023-05-31T20:50:32.581 DEBUG:boto:Method: GET 2023-05-31T20:50:32.582 DEBUG:boto:Path: /pklxkd-59/obj85?versionId=null 2023-05-31T20:50:32.582 DEBUG:boto:Data: 2023-05-31T20:50:32.582 DEBUG:boto:Headers: {} 2023-05-31T20:50:32.583 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:50:32.583 DEBUG:boto:Port: 8001 2023-05-31T20:50:32.583 DEBUG:boto:Params: {} 2023-05-31T20:50:32.584 DEBUG:boto:Token: None 2023-05-31T20:50:32.584 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:50:32 GMT /pklxkd-59/obj85?versionId=null 2023-05-31T20:50:32.585 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:EDlrqNYHiExlw9ndjlo+O81u9q8= 2023-05-31T20:50:32.585 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:50:32 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:EDlrqNYHiExlw9ndjlo+O81u9q8=', 'Content-Length': '0'} 2023-05-31T20:50:32.587 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:43:08 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000070f522cf17adb8f1-006477b318-1c17-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:50:32 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:50:32.587 DEBUG:boto:path=/pklxkd-59/obj85 2023-05-31T20:50:32.588 DEBUG:boto:auth_path=/pklxkd-59/obj85 2023-05-31T20:50:32.588 DEBUG:boto:path=/pklxkd-59/obj85?versionId=null 2023-05-31T20:50:32.589 DEBUG:boto:auth_path=/pklxkd-59/obj85?versionId=null 2023-05-31T20:50:32.589 DEBUG:boto:Method: GET 2023-05-31T20:50:32.589 DEBUG:boto:Path: /pklxkd-59/obj85?versionId=null 2023-05-31T20:50:32.590 DEBUG:boto:Data: 2023-05-31T20:50:32.590 DEBUG:boto:Headers: {} 2023-05-31T20:50:32.590 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:50:32.591 DEBUG:boto:Port: 8000 2023-05-31T20:50:32.591 DEBUG:boto:Params: {} 2023-05-31T20:50:32.592 DEBUG:boto:Token: None 2023-05-31T20:50:32.592 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:50:32 GMT /pklxkd-59/obj85?versionId=null 2023-05-31T20:50:32.592 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:EDlrqNYHiExlw9ndjlo+O81u9q8= 2023-05-31T20:50:32.593 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:50:32 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:EDlrqNYHiExlw9ndjlo+O81u9q8=', 'Content-Length': '0'} 2023-05-31T20:50:32.595 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:43:08 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006afbb0eb384b1ae8-006477b318-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:50:32 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:50:32.595 DEBUG:boto:path=/pklxkd-59/ 2023-05-31T20:50:32.596 DEBUG:boto:auth_path=/pklxkd-59/ 2023-05-31T20:50:32.596 DEBUG:boto:path=/pklxkd-59/?prefix=obj85 2023-05-31T20:50:32.596 DEBUG:boto:auth_path=/pklxkd-59/?prefix=obj85 2023-05-31T20:50:32.597 DEBUG:boto:Method: GET 2023-05-31T20:50:32.597 DEBUG:boto:Path: /pklxkd-59/?prefix=obj85 2023-05-31T20:50:32.597 DEBUG:boto:Data: 2023-05-31T20:50:32.598 DEBUG:boto:Headers: {} 2023-05-31T20:50:32.598 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:50:32.599 DEBUG:boto:Port: 8001 2023-05-31T20:50:32.599 DEBUG:boto:Params: {} 2023-05-31T20:50:32.599 DEBUG:boto:Token: None 2023-05-31T20:50:32.600 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:50:32 GMT /pklxkd-59/ 2023-05-31T20:50:32.600 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:s+yZS3HyomxcW/vxaV/hqlSq7Vg= 2023-05-31T20:50:32.601 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:50:32 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:s+yZS3HyomxcW/vxaV/hqlSq7Vg=', 'Content-Length': '0'} 2023-05-31T20:50:32.603 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ee4331a0c1d5406c-006477b318-1c17-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:50:32 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:50:32.603 DEBUG:boto:b'pklxkd-59obj851000falseobj852023-05-31T20:43:08.086Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:50:32.604 DEBUG:boto:path=/pklxkd-59/ 2023-05-31T20:50:32.604 DEBUG:boto:auth_path=/pklxkd-59/ 2023-05-31T20:50:32.605 DEBUG:boto:path=/pklxkd-59/?prefix=obj85 2023-05-31T20:50:32.605 DEBUG:boto:auth_path=/pklxkd-59/?prefix=obj85 2023-05-31T20:50:32.606 DEBUG:boto:Method: GET 2023-05-31T20:50:32.606 DEBUG:boto:Path: /pklxkd-59/?prefix=obj85 2023-05-31T20:50:32.606 DEBUG:boto:Data: 2023-05-31T20:50:32.607 DEBUG:boto:Headers: {} 2023-05-31T20:50:32.607 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:50:32.607 DEBUG:boto:Port: 8000 2023-05-31T20:50:32.608 DEBUG:boto:Params: {} 2023-05-31T20:50:32.608 DEBUG:boto:Token: None 2023-05-31T20:50:32.608 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:50:32 GMT /pklxkd-59/ 2023-05-31T20:50:32.609 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:s+yZS3HyomxcW/vxaV/hqlSq7Vg= 2023-05-31T20:50:32.609 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:50:32 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:s+yZS3HyomxcW/vxaV/hqlSq7Vg=', 'Content-Length': '0'} 2023-05-31T20:50:32.613 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000002d2967afbe0778d9-006477b318-1630-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:50:32 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:50:32.613 DEBUG:boto:b'pklxkd-59obj851000falseobj852023-05-31T20:43:08.086Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:50:32.614 DEBUG:boto:path=/pklxkd-59/obj85 2023-05-31T20:50:32.614 DEBUG:boto:auth_path=/pklxkd-59/obj85 2023-05-31T20:50:32.615 DEBUG:boto:Method: HEAD 2023-05-31T20:50:32.615 DEBUG:boto:Path: /pklxkd-59/obj85 2023-05-31T20:50:32.615 DEBUG:boto:Data: 2023-05-31T20:50:32.616 DEBUG:boto:Headers: {} 2023-05-31T20:50:32.616 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:50:32.616 DEBUG:boto:Port: 8001 2023-05-31T20:50:32.617 DEBUG:boto:Params: {} 2023-05-31T20:50:32.617 DEBUG:boto:Token: None 2023-05-31T20:50:32.618 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:50:32 GMT /pklxkd-59/obj85 2023-05-31T20:50:32.618 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:ZFqC49xHVLcbUyI+dsuy3XCjh+g= 2023-05-31T20:50:32.619 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:50:32 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:ZFqC49xHVLcbUyI+dsuy3XCjh+g=', 'Content-Length': '0'} 2023-05-31T20:50:32.621 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:43:08 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f24ae22adb652c18-006477b318-1c17-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:50:32 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:50:32.621 DEBUG:boto:path=/pklxkd-59/obj85 2023-05-31T20:50:32.622 DEBUG:boto:auth_path=/pklxkd-59/obj85 2023-05-31T20:50:32.622 DEBUG:boto:Method: HEAD 2023-05-31T20:50:32.623 DEBUG:boto:Path: /pklxkd-59/obj85 2023-05-31T20:50:32.623 DEBUG:boto:Data: 2023-05-31T20:50:32.623 DEBUG:boto:Headers: {} 2023-05-31T20:50:32.624 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:50:32.624 DEBUG:boto:Port: 8000 2023-05-31T20:50:32.625 DEBUG:boto:Params: {} 2023-05-31T20:50:32.625 DEBUG:boto:Token: None 2023-05-31T20:50:32.626 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:50:32 GMT /pklxkd-59/obj85 2023-05-31T20:50:32.626 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:ZFqC49xHVLcbUyI+dsuy3XCjh+g= 2023-05-31T20:50:32.627 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:50:32 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:ZFqC49xHVLcbUyI+dsuy3XCjh+g=', 'Content-Length': '0'} 2023-05-31T20:50:32.629 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:43:08 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ca1ce723b6a55a90-006477b318-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:50:32 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:50:32.629 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj85 2023-05-31T20:50:32.629 DEBUG:boto:path=/pklxkd-59/obj85 2023-05-31T20:50:32.630 DEBUG:boto:auth_path=/pklxkd-59/obj85 2023-05-31T20:50:32.630 DEBUG:boto:Method: GET 2023-05-31T20:50:32.631 DEBUG:boto:Path: /pklxkd-59/obj85 2023-05-31T20:50:32.631 DEBUG:boto:Data: 2023-05-31T20:50:32.631 DEBUG:boto:Headers: {} 2023-05-31T20:50:32.632 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:50:32.632 DEBUG:boto:Port: 8001 2023-05-31T20:50:32.632 DEBUG:boto:Params: {} 2023-05-31T20:50:32.633 DEBUG:boto:Token: None 2023-05-31T20:50:32.633 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:50:32 GMT /pklxkd-59/obj85 2023-05-31T20:50:32.634 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:0wjF6c86GhmdpqRlGt6Di14uIjQ= 2023-05-31T20:50:32.634 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:50:32 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:0wjF6c86GhmdpqRlGt6Di14uIjQ=', 'Content-Length': '0'} 2023-05-31T20:50:32.636 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:43:08 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000010e9c03180eb35ce-006477b318-1c17-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:50:32 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:50:32.636 DEBUG:boto:path=/pklxkd-59/obj85 2023-05-31T20:50:32.637 DEBUG:boto:auth_path=/pklxkd-59/obj85 2023-05-31T20:50:32.637 DEBUG:boto:Method: GET 2023-05-31T20:50:32.638 DEBUG:boto:Path: /pklxkd-59/obj85 2023-05-31T20:50:32.638 DEBUG:boto:Data: 2023-05-31T20:50:32.638 DEBUG:boto:Headers: {} 2023-05-31T20:50:32.639 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:50:32.639 DEBUG:boto:Port: 8000 2023-05-31T20:50:32.639 DEBUG:boto:Params: {} 2023-05-31T20:50:32.640 DEBUG:boto:Token: None 2023-05-31T20:50:32.640 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:50:32 GMT /pklxkd-59/obj85 2023-05-31T20:50:32.641 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:0wjF6c86GhmdpqRlGt6Di14uIjQ= 2023-05-31T20:50:32.641 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:50:32 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:0wjF6c86GhmdpqRlGt6Di14uIjQ=', 'Content-Length': '0'} 2023-05-31T20:50:32.643 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:43:08 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000635a8a630cfe57c5-006477b318-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:50:32 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:50:32.684 DEBUG:boto:path=/pklxkd-59/ 2023-05-31T20:50:32.685 DEBUG:boto:auth_path=/pklxkd-59/ 2023-05-31T20:50:32.685 DEBUG:boto:path=/pklxkd-59/?prefix=obj85 2023-05-31T20:50:32.685 DEBUG:boto:auth_path=/pklxkd-59/?prefix=obj85 2023-05-31T20:50:32.686 DEBUG:boto:Method: GET 2023-05-31T20:50:32.686 DEBUG:boto:Path: /pklxkd-59/?prefix=obj85 2023-05-31T20:50:32.686 DEBUG:boto:Data: 2023-05-31T20:50:32.687 DEBUG:boto:Headers: {} 2023-05-31T20:50:32.687 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:50:32.687 DEBUG:boto:Port: 8001 2023-05-31T20:50:32.688 DEBUG:boto:Params: {} 2023-05-31T20:50:32.688 DEBUG:boto:Token: None 2023-05-31T20:50:32.689 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:50:32 GMT /pklxkd-59/ 2023-05-31T20:50:32.689 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:s+yZS3HyomxcW/vxaV/hqlSq7Vg= 2023-05-31T20:50:32.690 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:50:32 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:s+yZS3HyomxcW/vxaV/hqlSq7Vg=', 'Content-Length': '0'} 2023-05-31T20:50:32.692 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ed2b6492c305cf41-006477b318-1c17-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:50:32 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:50:32.693 DEBUG:boto:b'pklxkd-59obj851000falseobj852023-05-31T20:43:08.086Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:50:32.693 DEBUG:boto:path=/pklxkd-59/ 2023-05-31T20:50:32.694 DEBUG:boto:auth_path=/pklxkd-59/ 2023-05-31T20:50:32.694 DEBUG:boto:path=/pklxkd-59/?prefix=obj85 2023-05-31T20:50:32.695 DEBUG:boto:auth_path=/pklxkd-59/?prefix=obj85 2023-05-31T20:50:32.695 DEBUG:boto:Method: GET 2023-05-31T20:50:32.695 DEBUG:boto:Path: /pklxkd-59/?prefix=obj85 2023-05-31T20:50:32.696 DEBUG:boto:Data: 2023-05-31T20:50:32.696 DEBUG:boto:Headers: {} 2023-05-31T20:50:32.697 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:50:32.697 DEBUG:boto:Port: 8000 2023-05-31T20:50:32.697 DEBUG:boto:Params: {} 2023-05-31T20:50:32.698 DEBUG:boto:Token: None 2023-05-31T20:50:32.698 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:50:32 GMT /pklxkd-59/ 2023-05-31T20:50:32.699 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:s+yZS3HyomxcW/vxaV/hqlSq7Vg= 2023-05-31T20:50:32.699 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:50:32 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:s+yZS3HyomxcW/vxaV/hqlSq7Vg=', 'Content-Length': '0'} 2023-05-31T20:50:32.702 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ec75a60b0dfd64dc-006477b318-1630-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:50:32 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:50:32.703 DEBUG:boto:b'pklxkd-59obj851000falseobj852023-05-31T20:43:08.086Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:50:32.704 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj87 2023-05-31T20:50:32.704 DEBUG:boto:path=/pklxkd-59/obj87 2023-05-31T20:50:32.704 DEBUG:boto:auth_path=/pklxkd-59/obj87 2023-05-31T20:50:32.705 DEBUG:boto:path=/pklxkd-59/obj87?versionId=null 2023-05-31T20:50:32.705 DEBUG:boto:auth_path=/pklxkd-59/obj87?versionId=null 2023-05-31T20:50:32.706 DEBUG:boto:Method: GET 2023-05-31T20:50:32.706 DEBUG:boto:Path: /pklxkd-59/obj87?versionId=null 2023-05-31T20:50:32.706 DEBUG:boto:Data: 2023-05-31T20:50:32.707 DEBUG:boto:Headers: {} 2023-05-31T20:50:32.707 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:50:32.707 DEBUG:boto:Port: 8001 2023-05-31T20:50:32.708 DEBUG:boto:Params: {} 2023-05-31T20:50:32.708 DEBUG:boto:Token: None 2023-05-31T20:50:32.709 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:50:32 GMT /pklxkd-59/obj87?versionId=null 2023-05-31T20:50:32.709 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:lOP97DqXyhdzb+/URURlRS5O8ew= 2023-05-31T20:50:32.710 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:50:32 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:lOP97DqXyhdzb+/URURlRS5O8ew=', 'Content-Length': '0'} 2023-05-31T20:50:32.711 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:48:13 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000033884b702400d687-006477b318-1c17-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:50:32 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:50:32.752 DEBUG:boto:path=/pklxkd-59/obj87 2023-05-31T20:50:32.753 DEBUG:boto:auth_path=/pklxkd-59/obj87 2023-05-31T20:50:32.753 DEBUG:boto:path=/pklxkd-59/obj87?versionId=null 2023-05-31T20:50:32.754 DEBUG:boto:auth_path=/pklxkd-59/obj87?versionId=null 2023-05-31T20:50:32.754 DEBUG:boto:Method: GET 2023-05-31T20:50:32.754 DEBUG:boto:Path: /pklxkd-59/obj87?versionId=null 2023-05-31T20:50:32.755 DEBUG:boto:Data: 2023-05-31T20:50:32.755 DEBUG:boto:Headers: {} 2023-05-31T20:50:32.756 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:50:32.756 DEBUG:boto:Port: 8000 2023-05-31T20:50:32.756 DEBUG:boto:Params: {} 2023-05-31T20:50:32.757 DEBUG:boto:Token: None 2023-05-31T20:50:32.757 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:50:32 GMT /pklxkd-59/obj87?versionId=null 2023-05-31T20:50:32.758 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:lOP97DqXyhdzb+/URURlRS5O8ew= 2023-05-31T20:50:32.758 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:50:32 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:lOP97DqXyhdzb+/URURlRS5O8ew=', 'Content-Length': '0'} 2023-05-31T20:50:32.760 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:48:13 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000137c9b99e5a51e27-006477b318-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:50:32 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:50:32.761 DEBUG:boto:path=/pklxkd-59/obj87 2023-05-31T20:50:32.761 DEBUG:boto:auth_path=/pklxkd-59/obj87 2023-05-31T20:50:32.761 DEBUG:boto:path=/pklxkd-59/obj87?versionId=null 2023-05-31T20:50:32.762 DEBUG:boto:auth_path=/pklxkd-59/obj87?versionId=null 2023-05-31T20:50:32.762 DEBUG:boto:Method: HEAD 2023-05-31T20:50:32.762 DEBUG:boto:Path: /pklxkd-59/obj87?versionId=null 2023-05-31T20:50:32.763 DEBUG:boto:Data: 2023-05-31T20:50:32.763 DEBUG:boto:Headers: {} 2023-05-31T20:50:32.764 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:50:32.764 DEBUG:boto:Port: 8001 2023-05-31T20:50:32.764 DEBUG:boto:Params: {} 2023-05-31T20:50:32.765 DEBUG:boto:Token: None 2023-05-31T20:50:32.765 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:50:32 GMT /pklxkd-59/obj87?versionId=null 2023-05-31T20:50:32.766 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:JqUHYjKz/QZTP4eujjDDhlJWKhw= 2023-05-31T20:50:32.766 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:50:32 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:JqUHYjKz/QZTP4eujjDDhlJWKhw=', 'Content-Length': '0'} 2023-05-31T20:50:32.768 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:48:13 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000038cb7dce8c33c6be-006477b318-1c17-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:50:32 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:50:32.768 DEBUG:boto:path=/pklxkd-59/obj87 2023-05-31T20:50:32.769 DEBUG:boto:auth_path=/pklxkd-59/obj87 2023-05-31T20:50:32.769 DEBUG:boto:path=/pklxkd-59/obj87?versionId=null 2023-05-31T20:50:32.769 DEBUG:boto:auth_path=/pklxkd-59/obj87?versionId=null 2023-05-31T20:50:32.770 DEBUG:boto:Method: HEAD 2023-05-31T20:50:32.770 DEBUG:boto:Path: /pklxkd-59/obj87?versionId=null 2023-05-31T20:50:32.770 DEBUG:boto:Data: 2023-05-31T20:50:32.771 DEBUG:boto:Headers: {} 2023-05-31T20:50:32.771 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:50:32.771 DEBUG:boto:Port: 8000 2023-05-31T20:50:32.772 DEBUG:boto:Params: {} 2023-05-31T20:50:32.772 DEBUG:boto:Token: None 2023-05-31T20:50:32.773 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:50:32 GMT /pklxkd-59/obj87?versionId=null 2023-05-31T20:50:32.773 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:JqUHYjKz/QZTP4eujjDDhlJWKhw= 2023-05-31T20:50:32.773 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:50:32 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:JqUHYjKz/QZTP4eujjDDhlJWKhw=', 'Content-Length': '0'} 2023-05-31T20:50:32.775 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:48:13 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a7f235f3fda22dfd-006477b318-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:50:32 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:50:32.776 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj87 2023-05-31T20:50:32.776 DEBUG:boto:path=/pklxkd-59/obj87 2023-05-31T20:50:32.777 DEBUG:boto:auth_path=/pklxkd-59/obj87 2023-05-31T20:50:32.777 DEBUG:boto:path=/pklxkd-59/obj87?versionId=null 2023-05-31T20:50:32.777 DEBUG:boto:auth_path=/pklxkd-59/obj87?versionId=null 2023-05-31T20:50:32.778 DEBUG:boto:Method: GET 2023-05-31T20:50:32.778 DEBUG:boto:Path: /pklxkd-59/obj87?versionId=null 2023-05-31T20:50:32.778 DEBUG:boto:Data: 2023-05-31T20:50:32.779 DEBUG:boto:Headers: {} 2023-05-31T20:50:32.779 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:50:32.779 DEBUG:boto:Port: 8001 2023-05-31T20:50:32.780 DEBUG:boto:Params: {} 2023-05-31T20:50:32.780 DEBUG:boto:Token: None 2023-05-31T20:50:32.781 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:50:32 GMT /pklxkd-59/obj87?versionId=null 2023-05-31T20:50:32.781 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:lOP97DqXyhdzb+/URURlRS5O8ew= 2023-05-31T20:50:32.781 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:50:32 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:lOP97DqXyhdzb+/URURlRS5O8ew=', 'Content-Length': '0'} 2023-05-31T20:50:32.783 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:48:13 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003fc277bef17e1ee8-006477b318-1c17-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:50:32 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:50:32.784 DEBUG:boto:path=/pklxkd-59/obj87 2023-05-31T20:50:32.784 DEBUG:boto:auth_path=/pklxkd-59/obj87 2023-05-31T20:50:32.785 DEBUG:boto:path=/pklxkd-59/obj87?versionId=null 2023-05-31T20:50:32.785 DEBUG:boto:auth_path=/pklxkd-59/obj87?versionId=null 2023-05-31T20:50:32.786 DEBUG:boto:Method: GET 2023-05-31T20:50:32.786 DEBUG:boto:Path: /pklxkd-59/obj87?versionId=null 2023-05-31T20:50:32.786 DEBUG:boto:Data: 2023-05-31T20:50:32.787 DEBUG:boto:Headers: {} 2023-05-31T20:50:32.787 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:50:32.787 DEBUG:boto:Port: 8000 2023-05-31T20:50:32.788 DEBUG:boto:Params: {} 2023-05-31T20:50:32.788 DEBUG:boto:Token: None 2023-05-31T20:50:32.788 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:50:32 GMT /pklxkd-59/obj87?versionId=null 2023-05-31T20:50:32.789 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:lOP97DqXyhdzb+/URURlRS5O8ew= 2023-05-31T20:50:32.789 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:50:32 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:lOP97DqXyhdzb+/URURlRS5O8ew=', 'Content-Length': '0'} 2023-05-31T20:50:32.791 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:48:13 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006547cdd505172eb3-006477b318-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:50:32 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:50:32.792 DEBUG:boto:path=/pklxkd-59/ 2023-05-31T20:50:32.792 DEBUG:boto:auth_path=/pklxkd-59/ 2023-05-31T20:50:32.793 DEBUG:boto:path=/pklxkd-59/?prefix=obj87 2023-05-31T20:50:32.793 DEBUG:boto:auth_path=/pklxkd-59/?prefix=obj87 2023-05-31T20:50:32.793 DEBUG:boto:Method: GET 2023-05-31T20:50:32.794 DEBUG:boto:Path: /pklxkd-59/?prefix=obj87 2023-05-31T20:50:32.794 DEBUG:boto:Data: 2023-05-31T20:50:32.794 DEBUG:boto:Headers: {} 2023-05-31T20:50:32.795 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:50:32.795 DEBUG:boto:Port: 8001 2023-05-31T20:50:32.796 DEBUG:boto:Params: {} 2023-05-31T20:50:32.796 DEBUG:boto:Token: None 2023-05-31T20:50:32.796 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:50:32 GMT /pklxkd-59/ 2023-05-31T20:50:32.797 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:s+yZS3HyomxcW/vxaV/hqlSq7Vg= 2023-05-31T20:50:32.797 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:50:32 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:s+yZS3HyomxcW/vxaV/hqlSq7Vg=', 'Content-Length': '0'} 2023-05-31T20:50:32.799 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000030aa80f2514612d0-006477b318-1c17-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:50:32 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:50:32.800 DEBUG:boto:b'pklxkd-59obj871000falseobj872023-05-31T20:48:13.031Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:50:32.801 DEBUG:boto:path=/pklxkd-59/ 2023-05-31T20:50:32.801 DEBUG:boto:auth_path=/pklxkd-59/ 2023-05-31T20:50:32.801 DEBUG:boto:path=/pklxkd-59/?prefix=obj87 2023-05-31T20:50:32.802 DEBUG:boto:auth_path=/pklxkd-59/?prefix=obj87 2023-05-31T20:50:32.802 DEBUG:boto:Method: GET 2023-05-31T20:50:32.802 DEBUG:boto:Path: /pklxkd-59/?prefix=obj87 2023-05-31T20:50:32.803 DEBUG:boto:Data: 2023-05-31T20:50:32.803 DEBUG:boto:Headers: {} 2023-05-31T20:50:32.804 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:50:32.804 DEBUG:boto:Port: 8000 2023-05-31T20:50:32.804 DEBUG:boto:Params: {} 2023-05-31T20:50:32.805 DEBUG:boto:Token: None 2023-05-31T20:50:32.805 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:50:32 GMT /pklxkd-59/ 2023-05-31T20:50:32.805 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:s+yZS3HyomxcW/vxaV/hqlSq7Vg= 2023-05-31T20:50:32.806 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:50:32 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:s+yZS3HyomxcW/vxaV/hqlSq7Vg=', 'Content-Length': '0'} 2023-05-31T20:50:32.809 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000000fad916dd997a587-006477b318-1630-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:50:32 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:50:32.810 DEBUG:boto:b'pklxkd-59obj871000falseobj872023-05-31T20:48:13.031Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:50:32.810 DEBUG:boto:path=/pklxkd-59/obj87 2023-05-31T20:50:32.811 DEBUG:boto:auth_path=/pklxkd-59/obj87 2023-05-31T20:50:32.811 DEBUG:boto:Method: HEAD 2023-05-31T20:50:32.812 DEBUG:boto:Path: /pklxkd-59/obj87 2023-05-31T20:50:32.812 DEBUG:boto:Data: 2023-05-31T20:50:32.812 DEBUG:boto:Headers: {} 2023-05-31T20:50:32.813 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:50:32.813 DEBUG:boto:Port: 8001 2023-05-31T20:50:32.813 DEBUG:boto:Params: {} 2023-05-31T20:50:32.814 DEBUG:boto:Token: None 2023-05-31T20:50:32.814 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:50:32 GMT /pklxkd-59/obj87 2023-05-31T20:50:32.814 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:KknwEBlmeXZzURFYgrpcpuKYDCg= 2023-05-31T20:50:32.815 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:50:32 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:KknwEBlmeXZzURFYgrpcpuKYDCg=', 'Content-Length': '0'} 2023-05-31T20:50:32.816 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:48:13 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a533d47146b3f24e-006477b318-1c17-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:50:32 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:50:32.817 DEBUG:boto:path=/pklxkd-59/obj87 2023-05-31T20:50:32.817 DEBUG:boto:auth_path=/pklxkd-59/obj87 2023-05-31T20:50:32.818 DEBUG:boto:Method: HEAD 2023-05-31T20:50:32.818 DEBUG:boto:Path: /pklxkd-59/obj87 2023-05-31T20:50:32.818 DEBUG:boto:Data: 2023-05-31T20:50:32.819 DEBUG:boto:Headers: {} 2023-05-31T20:50:32.819 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:50:32.820 DEBUG:boto:Port: 8000 2023-05-31T20:50:32.820 DEBUG:boto:Params: {} 2023-05-31T20:50:32.820 DEBUG:boto:Token: None 2023-05-31T20:50:32.821 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:50:32 GMT /pklxkd-59/obj87 2023-05-31T20:50:32.821 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:KknwEBlmeXZzURFYgrpcpuKYDCg= 2023-05-31T20:50:32.822 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:50:32 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:KknwEBlmeXZzURFYgrpcpuKYDCg=', 'Content-Length': '0'} 2023-05-31T20:50:32.823 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:48:13 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000017d0b49eaec22eb-006477b318-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:50:32 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:50:32.824 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj87 2023-05-31T20:50:32.824 DEBUG:boto:path=/pklxkd-59/obj87 2023-05-31T20:50:32.825 DEBUG:boto:auth_path=/pklxkd-59/obj87 2023-05-31T20:50:32.825 DEBUG:boto:Method: GET 2023-05-31T20:50:32.825 DEBUG:boto:Path: /pklxkd-59/obj87 2023-05-31T20:50:32.826 DEBUG:boto:Data: 2023-05-31T20:50:32.826 DEBUG:boto:Headers: {} 2023-05-31T20:50:32.826 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:50:32.827 DEBUG:boto:Port: 8001 2023-05-31T20:50:32.827 DEBUG:boto:Params: {} 2023-05-31T20:50:32.827 DEBUG:boto:Token: None 2023-05-31T20:50:32.828 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:50:32 GMT /pklxkd-59/obj87 2023-05-31T20:50:32.828 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:oSv2gszQeVjD9vJNVZf28D2jVeM= 2023-05-31T20:50:32.829 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:50:32 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:oSv2gszQeVjD9vJNVZf28D2jVeM=', 'Content-Length': '0'} 2023-05-31T20:50:32.831 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:48:13 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b16253c9974f17e2-006477b318-1c17-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:50:32 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:50:32.831 DEBUG:boto:path=/pklxkd-59/obj87 2023-05-31T20:50:32.832 DEBUG:boto:auth_path=/pklxkd-59/obj87 2023-05-31T20:50:32.832 DEBUG:boto:Method: GET 2023-05-31T20:50:32.832 DEBUG:boto:Path: /pklxkd-59/obj87 2023-05-31T20:50:32.833 DEBUG:boto:Data: 2023-05-31T20:50:32.833 DEBUG:boto:Headers: {} 2023-05-31T20:50:32.833 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:50:32.834 DEBUG:boto:Port: 8000 2023-05-31T20:50:32.834 DEBUG:boto:Params: {} 2023-05-31T20:50:32.834 DEBUG:boto:Token: None 2023-05-31T20:50:32.835 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:50:32 GMT /pklxkd-59/obj87 2023-05-31T20:50:32.835 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:oSv2gszQeVjD9vJNVZf28D2jVeM= 2023-05-31T20:50:32.836 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:50:32 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:oSv2gszQeVjD9vJNVZf28D2jVeM=', 'Content-Length': '0'} 2023-05-31T20:50:32.838 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:48:13 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004fd171ae3fb83ec3-006477b318-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:50:32 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:50:32.838 DEBUG:boto:path=/pklxkd-59/ 2023-05-31T20:50:32.839 DEBUG:boto:auth_path=/pklxkd-59/ 2023-05-31T20:50:32.839 DEBUG:boto:path=/pklxkd-59/?prefix=obj87 2023-05-31T20:50:32.839 DEBUG:boto:auth_path=/pklxkd-59/?prefix=obj87 2023-05-31T20:50:32.840 DEBUG:boto:Method: GET 2023-05-31T20:50:32.840 DEBUG:boto:Path: /pklxkd-59/?prefix=obj87 2023-05-31T20:50:32.840 DEBUG:boto:Data: 2023-05-31T20:50:32.841 DEBUG:boto:Headers: {} 2023-05-31T20:50:32.841 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:50:32.841 DEBUG:boto:Port: 8001 2023-05-31T20:50:32.842 DEBUG:boto:Params: {} 2023-05-31T20:50:32.842 DEBUG:boto:Token: None 2023-05-31T20:50:32.843 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:50:32 GMT /pklxkd-59/ 2023-05-31T20:50:32.843 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:s+yZS3HyomxcW/vxaV/hqlSq7Vg= 2023-05-31T20:50:32.844 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:50:32 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:s+yZS3HyomxcW/vxaV/hqlSq7Vg=', 'Content-Length': '0'} 2023-05-31T20:50:32.846 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000031d1feea5c2665af-006477b318-1c17-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:50:32 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:50:32.846 DEBUG:boto:b'pklxkd-59obj871000falseobj872023-05-31T20:48:13.031Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:50:32.847 DEBUG:boto:path=/pklxkd-59/ 2023-05-31T20:50:32.847 DEBUG:boto:auth_path=/pklxkd-59/ 2023-05-31T20:50:32.848 DEBUG:boto:path=/pklxkd-59/?prefix=obj87 2023-05-31T20:50:32.848 DEBUG:boto:auth_path=/pklxkd-59/?prefix=obj87 2023-05-31T20:50:32.848 DEBUG:boto:Method: GET 2023-05-31T20:50:32.849 DEBUG:boto:Path: /pklxkd-59/?prefix=obj87 2023-05-31T20:50:32.849 DEBUG:boto:Data: 2023-05-31T20:50:32.850 DEBUG:boto:Headers: {} 2023-05-31T20:50:32.850 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:50:32.850 DEBUG:boto:Port: 8000 2023-05-31T20:50:32.851 DEBUG:boto:Params: {} 2023-05-31T20:50:32.851 DEBUG:boto:Token: None 2023-05-31T20:50:32.852 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:50:32 GMT /pklxkd-59/ 2023-05-31T20:50:32.852 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:s+yZS3HyomxcW/vxaV/hqlSq7Vg= 2023-05-31T20:50:32.853 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:50:32 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:s+yZS3HyomxcW/vxaV/hqlSq7Vg=', 'Content-Length': '0'} 2023-05-31T20:50:32.856 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000003ae8214ad023c066-006477b318-1630-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:50:32 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:50:32.856 DEBUG:boto:b'pklxkd-59obj871000falseobj872023-05-31T20:48:13.031Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:50:32.857 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj92 2023-05-31T20:50:32.857 DEBUG:boto:path=/pklxkd-59/obj92 2023-05-31T20:50:32.858 DEBUG:boto:auth_path=/pklxkd-59/obj92 2023-05-31T20:50:32.858 DEBUG:boto:path=/pklxkd-59/obj92?versionId=null 2023-05-31T20:50:32.858 DEBUG:boto:auth_path=/pklxkd-59/obj92?versionId=null 2023-05-31T20:50:32.859 DEBUG:boto:Method: GET 2023-05-31T20:50:32.859 DEBUG:boto:Path: /pklxkd-59/obj92?versionId=null 2023-05-31T20:50:32.859 DEBUG:boto:Data: 2023-05-31T20:50:32.860 DEBUG:boto:Headers: {} 2023-05-31T20:50:32.860 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:50:32.860 DEBUG:boto:Port: 8001 2023-05-31T20:50:32.861 DEBUG:boto:Params: {} 2023-05-31T20:50:32.861 DEBUG:boto:Token: None 2023-05-31T20:50:32.862 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:50:32 GMT /pklxkd-59/obj92?versionId=null 2023-05-31T20:50:32.862 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:EsYSSHq911WlPFKpcxFbEhwDJFg= 2023-05-31T20:50:32.862 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:50:32 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:EsYSSHq911WlPFKpcxFbEhwDJFg=', 'Content-Length': '0'} 2023-05-31T20:50:32.864 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:46:33 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b19456d23d5193b0-006477b318-1c17-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:50:32 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:50:32.865 DEBUG:boto:path=/pklxkd-59/obj92 2023-05-31T20:50:32.865 DEBUG:boto:auth_path=/pklxkd-59/obj92 2023-05-31T20:50:32.866 DEBUG:boto:path=/pklxkd-59/obj92?versionId=null 2023-05-31T20:50:32.866 DEBUG:boto:auth_path=/pklxkd-59/obj92?versionId=null 2023-05-31T20:50:32.867 DEBUG:boto:Method: GET 2023-05-31T20:50:32.867 DEBUG:boto:Path: /pklxkd-59/obj92?versionId=null 2023-05-31T20:50:32.867 DEBUG:boto:Data: 2023-05-31T20:50:32.868 DEBUG:boto:Headers: {} 2023-05-31T20:50:32.868 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:50:32.869 DEBUG:boto:Port: 8000 2023-05-31T20:50:32.869 DEBUG:boto:Params: {} 2023-05-31T20:50:32.869 DEBUG:boto:Token: None 2023-05-31T20:50:32.870 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:50:32 GMT /pklxkd-59/obj92?versionId=null 2023-05-31T20:50:32.870 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:EsYSSHq911WlPFKpcxFbEhwDJFg= 2023-05-31T20:50:32.871 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:50:32 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:EsYSSHq911WlPFKpcxFbEhwDJFg=', 'Content-Length': '0'} 2023-05-31T20:50:32.873 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:46:33 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c17ef4cee5fbc627-006477b318-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:50:32 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:50:32.873 DEBUG:boto:path=/pklxkd-59/obj92 2023-05-31T20:50:32.874 DEBUG:boto:auth_path=/pklxkd-59/obj92 2023-05-31T20:50:32.874 DEBUG:boto:path=/pklxkd-59/obj92?versionId=null 2023-05-31T20:50:32.875 DEBUG:boto:auth_path=/pklxkd-59/obj92?versionId=null 2023-05-31T20:50:32.875 DEBUG:boto:Method: HEAD 2023-05-31T20:50:32.875 DEBUG:boto:Path: /pklxkd-59/obj92?versionId=null 2023-05-31T20:50:32.876 DEBUG:boto:Data: 2023-05-31T20:50:32.876 DEBUG:boto:Headers: {} 2023-05-31T20:50:32.876 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:50:32.877 DEBUG:boto:Port: 8001 2023-05-31T20:50:32.877 DEBUG:boto:Params: {} 2023-05-31T20:50:32.877 DEBUG:boto:Token: None 2023-05-31T20:50:32.878 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:50:32 GMT /pklxkd-59/obj92?versionId=null 2023-05-31T20:50:32.878 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:hK2ZwEhKCvdzTgRegDvLTxi6bcA= 2023-05-31T20:50:32.878 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:50:32 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:hK2ZwEhKCvdzTgRegDvLTxi6bcA=', 'Content-Length': '0'} 2023-05-31T20:50:32.880 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:46:33 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000432194ec16eb1bac-006477b318-1c17-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:50:32 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:50:32.881 DEBUG:boto:path=/pklxkd-59/obj92 2023-05-31T20:50:32.881 DEBUG:boto:auth_path=/pklxkd-59/obj92 2023-05-31T20:50:32.881 DEBUG:boto:path=/pklxkd-59/obj92?versionId=null 2023-05-31T20:50:32.882 DEBUG:boto:auth_path=/pklxkd-59/obj92?versionId=null 2023-05-31T20:50:32.882 DEBUG:boto:Method: HEAD 2023-05-31T20:50:32.882 DEBUG:boto:Path: /pklxkd-59/obj92?versionId=null 2023-05-31T20:50:32.883 DEBUG:boto:Data: 2023-05-31T20:50:32.883 DEBUG:boto:Headers: {} 2023-05-31T20:50:32.884 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:50:32.884 DEBUG:boto:Port: 8000 2023-05-31T20:50:32.884 DEBUG:boto:Params: {} 2023-05-31T20:50:32.885 DEBUG:boto:Token: None 2023-05-31T20:50:32.885 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:50:32 GMT /pklxkd-59/obj92?versionId=null 2023-05-31T20:50:32.885 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:hK2ZwEhKCvdzTgRegDvLTxi6bcA= 2023-05-31T20:50:32.886 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:50:32 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:hK2ZwEhKCvdzTgRegDvLTxi6bcA=', 'Content-Length': '0'} 2023-05-31T20:50:32.888 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:46:33 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000cd8d7fea535f5c9e-006477b318-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:50:32 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:50:32.888 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj92 2023-05-31T20:50:32.889 DEBUG:boto:path=/pklxkd-59/obj92 2023-05-31T20:50:32.889 DEBUG:boto:auth_path=/pklxkd-59/obj92 2023-05-31T20:50:32.889 DEBUG:boto:path=/pklxkd-59/obj92?versionId=null 2023-05-31T20:50:32.890 DEBUG:boto:auth_path=/pklxkd-59/obj92?versionId=null 2023-05-31T20:50:32.890 DEBUG:boto:Method: GET 2023-05-31T20:50:32.890 DEBUG:boto:Path: /pklxkd-59/obj92?versionId=null 2023-05-31T20:50:32.891 DEBUG:boto:Data: 2023-05-31T20:50:32.891 DEBUG:boto:Headers: {} 2023-05-31T20:50:32.892 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:50:32.892 DEBUG:boto:Port: 8001 2023-05-31T20:50:32.892 DEBUG:boto:Params: {} 2023-05-31T20:50:32.893 DEBUG:boto:Token: None 2023-05-31T20:50:32.893 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:50:32 GMT /pklxkd-59/obj92?versionId=null 2023-05-31T20:50:32.894 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:EsYSSHq911WlPFKpcxFbEhwDJFg= 2023-05-31T20:50:32.894 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:50:32 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:EsYSSHq911WlPFKpcxFbEhwDJFg=', 'Content-Length': '0'} 2023-05-31T20:50:32.896 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:46:33 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004acb312d18e79152-006477b318-1c17-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:50:32 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:50:32.936 DEBUG:boto:path=/pklxkd-59/obj92 2023-05-31T20:50:32.937 DEBUG:boto:auth_path=/pklxkd-59/obj92 2023-05-31T20:50:32.937 DEBUG:boto:path=/pklxkd-59/obj92?versionId=null 2023-05-31T20:50:32.938 DEBUG:boto:auth_path=/pklxkd-59/obj92?versionId=null 2023-05-31T20:50:32.938 DEBUG:boto:Method: GET 2023-05-31T20:50:32.938 DEBUG:boto:Path: /pklxkd-59/obj92?versionId=null 2023-05-31T20:50:32.939 DEBUG:boto:Data: 2023-05-31T20:50:32.939 DEBUG:boto:Headers: {} 2023-05-31T20:50:32.939 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:50:32.940 DEBUG:boto:Port: 8000 2023-05-31T20:50:32.940 DEBUG:boto:Params: {} 2023-05-31T20:50:32.941 DEBUG:boto:Token: None 2023-05-31T20:50:32.941 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:50:32 GMT /pklxkd-59/obj92?versionId=null 2023-05-31T20:50:32.941 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:EsYSSHq911WlPFKpcxFbEhwDJFg= 2023-05-31T20:50:32.942 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:50:32 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:EsYSSHq911WlPFKpcxFbEhwDJFg=', 'Content-Length': '0'} 2023-05-31T20:50:32.944 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:46:33 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009f6bdfd3c8ba9e59-006477b318-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:50:32 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:50:32.944 DEBUG:boto:path=/pklxkd-59/ 2023-05-31T20:50:32.945 DEBUG:boto:auth_path=/pklxkd-59/ 2023-05-31T20:50:32.945 DEBUG:boto:path=/pklxkd-59/?prefix=obj92 2023-05-31T20:50:32.945 DEBUG:boto:auth_path=/pklxkd-59/?prefix=obj92 2023-05-31T20:50:32.946 DEBUG:boto:Method: GET 2023-05-31T20:50:32.946 DEBUG:boto:Path: /pklxkd-59/?prefix=obj92 2023-05-31T20:50:32.946 DEBUG:boto:Data: 2023-05-31T20:50:32.947 DEBUG:boto:Headers: {} 2023-05-31T20:50:32.947 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:50:32.947 DEBUG:boto:Port: 8001 2023-05-31T20:50:32.948 DEBUG:boto:Params: {} 2023-05-31T20:50:32.948 DEBUG:boto:Token: None 2023-05-31T20:50:32.949 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:50:32 GMT /pklxkd-59/ 2023-05-31T20:50:32.949 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:s+yZS3HyomxcW/vxaV/hqlSq7Vg= 2023-05-31T20:50:32.949 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:50:32 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:s+yZS3HyomxcW/vxaV/hqlSq7Vg=', 'Content-Length': '0'} 2023-05-31T20:50:32.952 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000028c3bc6971ff201b-006477b318-1c17-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:50:32 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:50:32.952 DEBUG:boto:b'pklxkd-59obj921000falseobj922023-05-31T20:46:33.904Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:50:32.953 DEBUG:boto:path=/pklxkd-59/ 2023-05-31T20:50:32.953 DEBUG:boto:auth_path=/pklxkd-59/ 2023-05-31T20:50:32.954 DEBUG:boto:path=/pklxkd-59/?prefix=obj92 2023-05-31T20:50:32.954 DEBUG:boto:auth_path=/pklxkd-59/?prefix=obj92 2023-05-31T20:50:32.954 DEBUG:boto:Method: GET 2023-05-31T20:50:32.955 DEBUG:boto:Path: /pklxkd-59/?prefix=obj92 2023-05-31T20:50:32.955 DEBUG:boto:Data: 2023-05-31T20:50:32.955 DEBUG:boto:Headers: {} 2023-05-31T20:50:32.956 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:50:32.956 DEBUG:boto:Port: 8000 2023-05-31T20:50:32.956 DEBUG:boto:Params: {} 2023-05-31T20:50:32.957 DEBUG:boto:Token: None 2023-05-31T20:50:32.957 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:50:32 GMT /pklxkd-59/ 2023-05-31T20:50:32.958 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:s+yZS3HyomxcW/vxaV/hqlSq7Vg= 2023-05-31T20:50:32.958 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:50:32 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:s+yZS3HyomxcW/vxaV/hqlSq7Vg=', 'Content-Length': '0'} 2023-05-31T20:50:32.962 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c86ac83f13597d46-006477b318-1630-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:50:32 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:50:32.963 DEBUG:boto:b'pklxkd-59obj921000falseobj922023-05-31T20:46:33.904Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:50:32.963 DEBUG:boto:path=/pklxkd-59/obj92 2023-05-31T20:50:32.964 DEBUG:boto:auth_path=/pklxkd-59/obj92 2023-05-31T20:50:32.964 DEBUG:boto:Method: HEAD 2023-05-31T20:50:32.965 DEBUG:boto:Path: /pklxkd-59/obj92 2023-05-31T20:50:32.965 DEBUG:boto:Data: 2023-05-31T20:50:32.965 DEBUG:boto:Headers: {} 2023-05-31T20:50:32.966 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:50:32.966 DEBUG:boto:Port: 8001 2023-05-31T20:50:32.966 DEBUG:boto:Params: {} 2023-05-31T20:50:32.967 DEBUG:boto:Token: None 2023-05-31T20:50:32.967 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:50:32 GMT /pklxkd-59/obj92 2023-05-31T20:50:32.967 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:o9nw+cIHSQFTkeyQQ2LXEpqamms= 2023-05-31T20:50:32.968 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:50:32 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:o9nw+cIHSQFTkeyQQ2LXEpqamms=', 'Content-Length': '0'} 2023-05-31T20:50:32.970 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:46:33 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008e48b727d6c0c70d-006477b318-1c17-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:50:32 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:50:32.970 DEBUG:boto:path=/pklxkd-59/obj92 2023-05-31T20:50:32.971 DEBUG:boto:auth_path=/pklxkd-59/obj92 2023-05-31T20:50:32.971 DEBUG:boto:Method: HEAD 2023-05-31T20:50:32.971 DEBUG:boto:Path: /pklxkd-59/obj92 2023-05-31T20:50:32.972 DEBUG:boto:Data: 2023-05-31T20:50:32.972 DEBUG:boto:Headers: {} 2023-05-31T20:50:32.972 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:50:32.973 DEBUG:boto:Port: 8000 2023-05-31T20:50:32.973 DEBUG:boto:Params: {} 2023-05-31T20:50:32.973 DEBUG:boto:Token: None 2023-05-31T20:50:32.974 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:50:32 GMT /pklxkd-59/obj92 2023-05-31T20:50:32.974 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:o9nw+cIHSQFTkeyQQ2LXEpqamms= 2023-05-31T20:50:32.975 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:50:32 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:o9nw+cIHSQFTkeyQQ2LXEpqamms=', 'Content-Length': '0'} 2023-05-31T20:50:32.976 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:46:33 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000224122a13b8d0bf8-006477b318-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:50:32 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:50:32.977 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj92 2023-05-31T20:50:32.977 DEBUG:boto:path=/pklxkd-59/obj92 2023-05-31T20:50:32.977 DEBUG:boto:auth_path=/pklxkd-59/obj92 2023-05-31T20:50:32.978 DEBUG:boto:Method: GET 2023-05-31T20:50:32.978 DEBUG:boto:Path: /pklxkd-59/obj92 2023-05-31T20:50:32.978 DEBUG:boto:Data: 2023-05-31T20:50:32.979 DEBUG:boto:Headers: {} 2023-05-31T20:50:32.979 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:50:32.979 DEBUG:boto:Port: 8001 2023-05-31T20:50:32.980 DEBUG:boto:Params: {} 2023-05-31T20:50:32.980 DEBUG:boto:Token: None 2023-05-31T20:50:32.981 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:50:32 GMT /pklxkd-59/obj92 2023-05-31T20:50:32.981 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:zHDVow5K1SXcWwemDCdb5u2nHHc= 2023-05-31T20:50:32.981 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:50:32 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:zHDVow5K1SXcWwemDCdb5u2nHHc=', 'Content-Length': '0'} 2023-05-31T20:50:32.983 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:46:33 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000054efb1a73536c41-006477b318-1c17-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:50:32 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:50:32.984 DEBUG:boto:path=/pklxkd-59/obj92 2023-05-31T20:50:32.984 DEBUG:boto:auth_path=/pklxkd-59/obj92 2023-05-31T20:50:32.984 DEBUG:boto:Method: GET 2023-05-31T20:50:32.985 DEBUG:boto:Path: /pklxkd-59/obj92 2023-05-31T20:50:32.985 DEBUG:boto:Data: 2023-05-31T20:50:32.986 DEBUG:boto:Headers: {} 2023-05-31T20:50:32.986 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:50:32.986 DEBUG:boto:Port: 8000 2023-05-31T20:50:32.987 DEBUG:boto:Params: {} 2023-05-31T20:50:32.987 DEBUG:boto:Token: None 2023-05-31T20:50:32.987 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:50:32 GMT /pklxkd-59/obj92 2023-05-31T20:50:32.988 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:zHDVow5K1SXcWwemDCdb5u2nHHc= 2023-05-31T20:50:32.988 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:50:32 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:zHDVow5K1SXcWwemDCdb5u2nHHc=', 'Content-Length': '0'} 2023-05-31T20:50:32.990 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:46:33 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ac60b2f84f8ba356-006477b318-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:50:32 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:50:32.991 DEBUG:boto:path=/pklxkd-59/ 2023-05-31T20:50:32.991 DEBUG:boto:auth_path=/pklxkd-59/ 2023-05-31T20:50:32.992 DEBUG:boto:path=/pklxkd-59/?prefix=obj92 2023-05-31T20:50:32.992 DEBUG:boto:auth_path=/pklxkd-59/?prefix=obj92 2023-05-31T20:50:32.993 DEBUG:boto:Method: GET 2023-05-31T20:50:32.993 DEBUG:boto:Path: /pklxkd-59/?prefix=obj92 2023-05-31T20:50:32.993 DEBUG:boto:Data: 2023-05-31T20:50:32.994 DEBUG:boto:Headers: {} 2023-05-31T20:50:32.994 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:50:32.994 DEBUG:boto:Port: 8001 2023-05-31T20:50:32.995 DEBUG:boto:Params: {} 2023-05-31T20:50:32.995 DEBUG:boto:Token: None 2023-05-31T20:50:32.996 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:50:32 GMT /pklxkd-59/ 2023-05-31T20:50:32.996 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:s+yZS3HyomxcW/vxaV/hqlSq7Vg= 2023-05-31T20:50:32.996 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:50:32 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:s+yZS3HyomxcW/vxaV/hqlSq7Vg=', 'Content-Length': '0'} 2023-05-31T20:50:32.999 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000009b465d91e43c754-006477b318-1c17-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:50:32 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:50:32.999 DEBUG:boto:b'pklxkd-59obj921000falseobj922023-05-31T20:46:33.904Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:50:33.000 DEBUG:boto:path=/pklxkd-59/ 2023-05-31T20:50:33.000 DEBUG:boto:auth_path=/pklxkd-59/ 2023-05-31T20:50:33.001 DEBUG:boto:path=/pklxkd-59/?prefix=obj92 2023-05-31T20:50:33.001 DEBUG:boto:auth_path=/pklxkd-59/?prefix=obj92 2023-05-31T20:50:33.001 DEBUG:boto:Method: GET 2023-05-31T20:50:33.002 DEBUG:boto:Path: /pklxkd-59/?prefix=obj92 2023-05-31T20:50:33.002 DEBUG:boto:Data: 2023-05-31T20:50:33.002 DEBUG:boto:Headers: {} 2023-05-31T20:50:33.003 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:50:33.003 DEBUG:boto:Port: 8000 2023-05-31T20:50:33.003 DEBUG:boto:Params: {} 2023-05-31T20:50:33.004 DEBUG:boto:Token: None 2023-05-31T20:50:33.004 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:50:33 GMT /pklxkd-59/ 2023-05-31T20:50:33.005 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:65lcolxnX6hKgchTUmAkGy88XzI= 2023-05-31T20:50:33.005 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:50:33 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:65lcolxnX6hKgchTUmAkGy88XzI=', 'Content-Length': '0'} 2023-05-31T20:50:33.008 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000dc71f06906f8abf5-006477b319-1630-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:50:33 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:50:33.009 DEBUG:boto:b'pklxkd-59obj921000falseobj922023-05-31T20:46:33.904Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:50:33.010 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj95 2023-05-31T20:50:33.010 DEBUG:boto:path=/pklxkd-59/obj95 2023-05-31T20:50:33.010 DEBUG:boto:auth_path=/pklxkd-59/obj95 2023-05-31T20:50:33.011 DEBUG:boto:path=/pklxkd-59/obj95?versionId=null 2023-05-31T20:50:33.011 DEBUG:boto:auth_path=/pklxkd-59/obj95?versionId=null 2023-05-31T20:50:33.012 DEBUG:boto:Method: GET 2023-05-31T20:50:33.012 DEBUG:boto:Path: /pklxkd-59/obj95?versionId=null 2023-05-31T20:50:33.012 DEBUG:boto:Data: 2023-05-31T20:50:33.013 DEBUG:boto:Headers: {} 2023-05-31T20:50:33.013 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:50:33.013 DEBUG:boto:Port: 8001 2023-05-31T20:50:33.014 DEBUG:boto:Params: {} 2023-05-31T20:50:33.014 DEBUG:boto:Token: None 2023-05-31T20:50:33.015 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:50:33 GMT /pklxkd-59/obj95?versionId=null 2023-05-31T20:50:33.015 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:zomAhM7ODSnQSTq6eTFMj7+aTA4= 2023-05-31T20:50:33.015 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:50:33 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:zomAhM7ODSnQSTq6eTFMj7+aTA4=', 'Content-Length': '0'} 2023-05-31T20:50:33.017 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:44:50 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ea6db69568290a50-006477b319-1c17-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:50:33 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:50:33.018 DEBUG:boto:path=/pklxkd-59/obj95 2023-05-31T20:50:33.018 DEBUG:boto:auth_path=/pklxkd-59/obj95 2023-05-31T20:50:33.019 DEBUG:boto:path=/pklxkd-59/obj95?versionId=null 2023-05-31T20:50:33.019 DEBUG:boto:auth_path=/pklxkd-59/obj95?versionId=null 2023-05-31T20:50:33.019 DEBUG:boto:Method: GET 2023-05-31T20:50:33.020 DEBUG:boto:Path: /pklxkd-59/obj95?versionId=null 2023-05-31T20:50:33.020 DEBUG:boto:Data: 2023-05-31T20:50:33.020 DEBUG:boto:Headers: {} 2023-05-31T20:50:33.021 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:50:33.021 DEBUG:boto:Port: 8000 2023-05-31T20:50:33.022 DEBUG:boto:Params: {} 2023-05-31T20:50:33.022 DEBUG:boto:Token: None 2023-05-31T20:50:33.022 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:50:33 GMT /pklxkd-59/obj95?versionId=null 2023-05-31T20:50:33.023 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:zomAhM7ODSnQSTq6eTFMj7+aTA4= 2023-05-31T20:50:33.023 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:50:33 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:zomAhM7ODSnQSTq6eTFMj7+aTA4=', 'Content-Length': '0'} 2023-05-31T20:50:33.025 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:44:50 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b79f0a54f645451b-006477b319-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:50:33 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:50:33.026 DEBUG:boto:path=/pklxkd-59/obj95 2023-05-31T20:50:33.026 DEBUG:boto:auth_path=/pklxkd-59/obj95 2023-05-31T20:50:33.026 DEBUG:boto:path=/pklxkd-59/obj95?versionId=null 2023-05-31T20:50:33.027 DEBUG:boto:auth_path=/pklxkd-59/obj95?versionId=null 2023-05-31T20:50:33.027 DEBUG:boto:Method: HEAD 2023-05-31T20:50:33.027 DEBUG:boto:Path: /pklxkd-59/obj95?versionId=null 2023-05-31T20:50:33.028 DEBUG:boto:Data: 2023-05-31T20:50:33.028 DEBUG:boto:Headers: {} 2023-05-31T20:50:33.028 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:50:33.029 DEBUG:boto:Port: 8001 2023-05-31T20:50:33.029 DEBUG:boto:Params: {} 2023-05-31T20:50:33.029 DEBUG:boto:Token: None 2023-05-31T20:50:33.030 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:50:33 GMT /pklxkd-59/obj95?versionId=null 2023-05-31T20:50:33.030 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:tKws/LYVfGJ78SCK1KyEi4tE3D0= 2023-05-31T20:50:33.031 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:50:33 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:tKws/LYVfGJ78SCK1KyEi4tE3D0=', 'Content-Length': '0'} 2023-05-31T20:50:33.032 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:44:50 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000cf037e4420a2b07c-006477b319-1c17-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:50:33 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:50:33.033 DEBUG:boto:path=/pklxkd-59/obj95 2023-05-31T20:50:33.033 DEBUG:boto:auth_path=/pklxkd-59/obj95 2023-05-31T20:50:33.033 DEBUG:boto:path=/pklxkd-59/obj95?versionId=null 2023-05-31T20:50:33.034 DEBUG:boto:auth_path=/pklxkd-59/obj95?versionId=null 2023-05-31T20:50:33.034 DEBUG:boto:Method: HEAD 2023-05-31T20:50:33.034 DEBUG:boto:Path: /pklxkd-59/obj95?versionId=null 2023-05-31T20:50:33.035 DEBUG:boto:Data: 2023-05-31T20:50:33.035 DEBUG:boto:Headers: {} 2023-05-31T20:50:33.035 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:50:33.036 DEBUG:boto:Port: 8000 2023-05-31T20:50:33.036 DEBUG:boto:Params: {} 2023-05-31T20:50:33.037 DEBUG:boto:Token: None 2023-05-31T20:50:33.037 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:50:33 GMT /pklxkd-59/obj95?versionId=null 2023-05-31T20:50:33.037 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:tKws/LYVfGJ78SCK1KyEi4tE3D0= 2023-05-31T20:50:33.038 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:50:33 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:tKws/LYVfGJ78SCK1KyEi4tE3D0=', 'Content-Length': '0'} 2023-05-31T20:50:33.039 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:44:50 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000675432def46083b1-006477b319-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:50:33 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:50:33.040 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj95 2023-05-31T20:50:33.040 DEBUG:boto:path=/pklxkd-59/obj95 2023-05-31T20:50:33.041 DEBUG:boto:auth_path=/pklxkd-59/obj95 2023-05-31T20:50:33.041 DEBUG:boto:path=/pklxkd-59/obj95?versionId=null 2023-05-31T20:50:33.041 DEBUG:boto:auth_path=/pklxkd-59/obj95?versionId=null 2023-05-31T20:50:33.042 DEBUG:boto:Method: GET 2023-05-31T20:50:33.042 DEBUG:boto:Path: /pklxkd-59/obj95?versionId=null 2023-05-31T20:50:33.042 DEBUG:boto:Data: 2023-05-31T20:50:33.043 DEBUG:boto:Headers: {} 2023-05-31T20:50:33.043 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:50:33.044 DEBUG:boto:Port: 8001 2023-05-31T20:50:33.044 DEBUG:boto:Params: {} 2023-05-31T20:50:33.044 DEBUG:boto:Token: None 2023-05-31T20:50:33.045 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:50:33 GMT /pklxkd-59/obj95?versionId=null 2023-05-31T20:50:33.045 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:zomAhM7ODSnQSTq6eTFMj7+aTA4= 2023-05-31T20:50:33.046 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:50:33 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:zomAhM7ODSnQSTq6eTFMj7+aTA4=', 'Content-Length': '0'} 2023-05-31T20:50:33.047 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:44:50 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002a542f0bd0692a89-006477b319-1c17-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:50:33 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:50:33.048 DEBUG:boto:path=/pklxkd-59/obj95 2023-05-31T20:50:33.049 DEBUG:boto:auth_path=/pklxkd-59/obj95 2023-05-31T20:50:33.049 DEBUG:boto:path=/pklxkd-59/obj95?versionId=null 2023-05-31T20:50:33.049 DEBUG:boto:auth_path=/pklxkd-59/obj95?versionId=null 2023-05-31T20:50:33.050 DEBUG:boto:Method: GET 2023-05-31T20:50:33.050 DEBUG:boto:Path: /pklxkd-59/obj95?versionId=null 2023-05-31T20:50:33.050 DEBUG:boto:Data: 2023-05-31T20:50:33.051 DEBUG:boto:Headers: {} 2023-05-31T20:50:33.051 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:50:33.051 DEBUG:boto:Port: 8000 2023-05-31T20:50:33.052 DEBUG:boto:Params: {} 2023-05-31T20:50:33.052 DEBUG:boto:Token: None 2023-05-31T20:50:33.053 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:50:33 GMT /pklxkd-59/obj95?versionId=null 2023-05-31T20:50:33.053 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:zomAhM7ODSnQSTq6eTFMj7+aTA4= 2023-05-31T20:50:33.053 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:50:33 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:zomAhM7ODSnQSTq6eTFMj7+aTA4=', 'Content-Length': '0'} 2023-05-31T20:50:33.055 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:44:50 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001372d2f0e122a243-006477b319-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:50:33 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:50:33.096 DEBUG:boto:path=/pklxkd-59/ 2023-05-31T20:50:33.097 DEBUG:boto:auth_path=/pklxkd-59/ 2023-05-31T20:50:33.098 DEBUG:boto:path=/pklxkd-59/?prefix=obj95 2023-05-31T20:50:33.098 DEBUG:boto:auth_path=/pklxkd-59/?prefix=obj95 2023-05-31T20:50:33.098 DEBUG:boto:Method: GET 2023-05-31T20:50:33.099 DEBUG:boto:Path: /pklxkd-59/?prefix=obj95 2023-05-31T20:50:33.099 DEBUG:boto:Data: 2023-05-31T20:50:33.099 DEBUG:boto:Headers: {} 2023-05-31T20:50:33.100 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:50:33.100 DEBUG:boto:Port: 8001 2023-05-31T20:50:33.100 DEBUG:boto:Params: {} 2023-05-31T20:50:33.101 DEBUG:boto:Token: None 2023-05-31T20:50:33.101 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:50:33 GMT /pklxkd-59/ 2023-05-31T20:50:33.102 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:65lcolxnX6hKgchTUmAkGy88XzI= 2023-05-31T20:50:33.102 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:50:33 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:65lcolxnX6hKgchTUmAkGy88XzI=', 'Content-Length': '0'} 2023-05-31T20:50:33.105 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000006fe8fcca92b66d11-006477b319-1c17-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:50:33 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:50:33.105 DEBUG:boto:b'pklxkd-59obj951000falseobj952023-05-31T20:44:50.705Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:50:33.106 DEBUG:boto:path=/pklxkd-59/ 2023-05-31T20:50:33.106 DEBUG:boto:auth_path=/pklxkd-59/ 2023-05-31T20:50:33.107 DEBUG:boto:path=/pklxkd-59/?prefix=obj95 2023-05-31T20:50:33.107 DEBUG:boto:auth_path=/pklxkd-59/?prefix=obj95 2023-05-31T20:50:33.108 DEBUG:boto:Method: GET 2023-05-31T20:50:33.108 DEBUG:boto:Path: /pklxkd-59/?prefix=obj95 2023-05-31T20:50:33.108 DEBUG:boto:Data: 2023-05-31T20:50:33.109 DEBUG:boto:Headers: {} 2023-05-31T20:50:33.109 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:50:33.109 DEBUG:boto:Port: 8000 2023-05-31T20:50:33.110 DEBUG:boto:Params: {} 2023-05-31T20:50:33.110 DEBUG:boto:Token: None 2023-05-31T20:50:33.110 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:50:33 GMT /pklxkd-59/ 2023-05-31T20:50:33.111 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:65lcolxnX6hKgchTUmAkGy88XzI= 2023-05-31T20:50:33.111 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:50:33 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:65lcolxnX6hKgchTUmAkGy88XzI=', 'Content-Length': '0'} 2023-05-31T20:50:33.115 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000031156357a205336f-006477b319-1630-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:50:33 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:50:33.115 DEBUG:boto:b'pklxkd-59obj951000falseobj952023-05-31T20:44:50.705Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:50:33.116 DEBUG:boto:path=/pklxkd-59/obj95 2023-05-31T20:50:33.116 DEBUG:boto:auth_path=/pklxkd-59/obj95 2023-05-31T20:50:33.117 DEBUG:boto:Method: HEAD 2023-05-31T20:50:33.117 DEBUG:boto:Path: /pklxkd-59/obj95 2023-05-31T20:50:33.117 DEBUG:boto:Data: 2023-05-31T20:50:33.118 DEBUG:boto:Headers: {} 2023-05-31T20:50:33.118 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:50:33.118 DEBUG:boto:Port: 8001 2023-05-31T20:50:33.119 DEBUG:boto:Params: {} 2023-05-31T20:50:33.119 DEBUG:boto:Token: None 2023-05-31T20:50:33.119 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:50:33 GMT /pklxkd-59/obj95 2023-05-31T20:50:33.120 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:ZWn/8LMrFH3kflV1YY92BLNJaQ0= 2023-05-31T20:50:33.120 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:50:33 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:ZWn/8LMrFH3kflV1YY92BLNJaQ0=', 'Content-Length': '0'} 2023-05-31T20:50:33.122 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:44:50 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009aa491886a0d551f-006477b319-1c17-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:50:33 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:50:33.122 DEBUG:boto:path=/pklxkd-59/obj95 2023-05-31T20:50:33.123 DEBUG:boto:auth_path=/pklxkd-59/obj95 2023-05-31T20:50:33.123 DEBUG:boto:Method: HEAD 2023-05-31T20:50:33.124 DEBUG:boto:Path: /pklxkd-59/obj95 2023-05-31T20:50:33.124 DEBUG:boto:Data: 2023-05-31T20:50:33.124 DEBUG:boto:Headers: {} 2023-05-31T20:50:33.125 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:50:33.125 DEBUG:boto:Port: 8000 2023-05-31T20:50:33.125 DEBUG:boto:Params: {} 2023-05-31T20:50:33.126 DEBUG:boto:Token: None 2023-05-31T20:50:33.126 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:50:33 GMT /pklxkd-59/obj95 2023-05-31T20:50:33.126 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:ZWn/8LMrFH3kflV1YY92BLNJaQ0= 2023-05-31T20:50:33.127 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:50:33 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:ZWn/8LMrFH3kflV1YY92BLNJaQ0=', 'Content-Length': '0'} 2023-05-31T20:50:33.129 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:44:50 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000015e38925c2f46f16-006477b319-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:50:33 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:50:33.130 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj95 2023-05-31T20:50:33.130 DEBUG:boto:path=/pklxkd-59/obj95 2023-05-31T20:50:33.131 DEBUG:boto:auth_path=/pklxkd-59/obj95 2023-05-31T20:50:33.131 DEBUG:boto:Method: GET 2023-05-31T20:50:33.131 DEBUG:boto:Path: /pklxkd-59/obj95 2023-05-31T20:50:33.132 DEBUG:boto:Data: 2023-05-31T20:50:33.132 DEBUG:boto:Headers: {} 2023-05-31T20:50:33.132 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:50:33.133 DEBUG:boto:Port: 8001 2023-05-31T20:50:33.133 DEBUG:boto:Params: {} 2023-05-31T20:50:33.134 DEBUG:boto:Token: None 2023-05-31T20:50:33.134 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:50:33 GMT /pklxkd-59/obj95 2023-05-31T20:50:33.134 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:aj2rix7Afle6YsMRcioVmeu1n2k= 2023-05-31T20:50:33.135 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:50:33 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:aj2rix7Afle6YsMRcioVmeu1n2k=', 'Content-Length': '0'} 2023-05-31T20:50:33.137 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:44:50 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000dbbfb8ada0efde31-006477b319-1c17-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:50:33 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:50:33.180 DEBUG:boto:path=/pklxkd-59/obj95 2023-05-31T20:50:33.181 DEBUG:boto:auth_path=/pklxkd-59/obj95 2023-05-31T20:50:33.181 DEBUG:boto:Method: GET 2023-05-31T20:50:33.182 DEBUG:boto:Path: /pklxkd-59/obj95 2023-05-31T20:50:33.182 DEBUG:boto:Data: 2023-05-31T20:50:33.182 DEBUG:boto:Headers: {} 2023-05-31T20:50:33.183 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:50:33.183 DEBUG:boto:Port: 8000 2023-05-31T20:50:33.183 DEBUG:boto:Params: {} 2023-05-31T20:50:33.184 DEBUG:boto:Token: None 2023-05-31T20:50:33.184 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:50:33 GMT /pklxkd-59/obj95 2023-05-31T20:50:33.185 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:aj2rix7Afle6YsMRcioVmeu1n2k= 2023-05-31T20:50:33.185 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:50:33 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:aj2rix7Afle6YsMRcioVmeu1n2k=', 'Content-Length': '0'} 2023-05-31T20:50:33.187 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:44:50 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000d75cc0bab772d47-006477b319-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:50:33 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:50:33.187 DEBUG:boto:path=/pklxkd-59/ 2023-05-31T20:50:33.188 DEBUG:boto:auth_path=/pklxkd-59/ 2023-05-31T20:50:33.188 DEBUG:boto:path=/pklxkd-59/?prefix=obj95 2023-05-31T20:50:33.188 DEBUG:boto:auth_path=/pklxkd-59/?prefix=obj95 2023-05-31T20:50:33.189 DEBUG:boto:Method: GET 2023-05-31T20:50:33.189 DEBUG:boto:Path: /pklxkd-59/?prefix=obj95 2023-05-31T20:50:33.190 DEBUG:boto:Data: 2023-05-31T20:50:33.190 DEBUG:boto:Headers: {} 2023-05-31T20:50:33.190 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:50:33.191 DEBUG:boto:Port: 8001 2023-05-31T20:50:33.191 DEBUG:boto:Params: {} 2023-05-31T20:50:33.191 DEBUG:boto:Token: None 2023-05-31T20:50:33.192 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:50:33 GMT /pklxkd-59/ 2023-05-31T20:50:33.192 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:65lcolxnX6hKgchTUmAkGy88XzI= 2023-05-31T20:50:33.193 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:50:33 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:65lcolxnX6hKgchTUmAkGy88XzI=', 'Content-Length': '0'} 2023-05-31T20:50:33.195 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000bc79299a7b52e4ac-006477b319-1c17-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:50:33 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:50:33.195 DEBUG:boto:b'pklxkd-59obj951000falseobj952023-05-31T20:44:50.705Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:50:33.196 DEBUG:boto:path=/pklxkd-59/ 2023-05-31T20:50:33.197 DEBUG:boto:auth_path=/pklxkd-59/ 2023-05-31T20:50:33.197 DEBUG:boto:path=/pklxkd-59/?prefix=obj95 2023-05-31T20:50:33.197 DEBUG:boto:auth_path=/pklxkd-59/?prefix=obj95 2023-05-31T20:50:33.198 DEBUG:boto:Method: GET 2023-05-31T20:50:33.198 DEBUG:boto:Path: /pklxkd-59/?prefix=obj95 2023-05-31T20:50:33.198 DEBUG:boto:Data: 2023-05-31T20:50:33.199 DEBUG:boto:Headers: {} 2023-05-31T20:50:33.199 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:50:33.199 DEBUG:boto:Port: 8000 2023-05-31T20:50:33.200 DEBUG:boto:Params: {} 2023-05-31T20:50:33.200 DEBUG:boto:Token: None 2023-05-31T20:50:33.201 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:50:33 GMT /pklxkd-59/ 2023-05-31T20:50:33.201 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:65lcolxnX6hKgchTUmAkGy88XzI= 2023-05-31T20:50:33.201 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:50:33 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:65lcolxnX6hKgchTUmAkGy88XzI=', 'Content-Length': '0'} 2023-05-31T20:50:33.205 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000edc59f4a5ba6f110-006477b319-1630-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:50:33 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:50:33.206 DEBUG:boto:b'pklxkd-59obj951000falseobj952023-05-31T20:44:50.705Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:50:33.206 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=pklxkd-59 zones={a2, a1} 2023-05-31T20:50:33.207 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'reshard', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'pklxkd-59', '--num-shards', '31', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-05-31T20:50:33.207 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'reshard', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'pklxkd-59', '--num-shards', '31', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-05-31T20:50:33.207 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket reshard --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --bucket pklxkd-59 --num-shards 31 --yes-i-really-mean-it --cluster c1 --debug-rgw 0 --debug-ms 0 2023-05-31T20:50:33.268 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2023-05-31T20:50:33.268 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2023-05-31T20:50:33.340 INFO:teuthology.orchestra.run.smithi099.stdout:tenant: 2023-05-31T20:50:33.341 INFO:teuthology.orchestra.run.smithi099.stdout:bucket name: pklxkd-59 2023-05-31T20:50:33.352 INFO:teuthology.orchestra.run.smithi099.stdout:total entries: 25 2023-05-31T20:50:33.448 DEBUG:boto:path=/pklxkd-59/ 2023-05-31T20:50:33.448 DEBUG:boto:auth_path=/pklxkd-59/ 2023-05-31T20:50:33.449 DEBUG:boto:Method: HEAD 2023-05-31T20:50:33.449 DEBUG:boto:Path: /pklxkd-59/ 2023-05-31T20:50:33.450 DEBUG:boto:Data: 2023-05-31T20:50:33.450 DEBUG:boto:Headers: {} 2023-05-31T20:50:33.450 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:50:33.451 DEBUG:boto:Port: 8000 2023-05-31T20:50:33.451 DEBUG:boto:Params: {} 2023-05-31T20:50:33.451 DEBUG:boto:Token: None 2023-05-31T20:50:33.452 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:50:33 GMT /pklxkd-59/ 2023-05-31T20:50:33.452 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:i27ZhanjX6um7diVFc/7GbiZjHw= 2023-05-31T20:50:33.453 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:50:33 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:i27ZhanjX6um7diVFc/7GbiZjHw=', 'Content-Length': '0'} 2023-05-31T20:50:33.456 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '25'), ('X-RGW-Bytes-Used', '75'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000ca0c546c36c70af4-006477b319-1630-a1'), ('Content-Length', '0'), ('Date', 'Wed, 31 May 2023 20:50:33 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:50:33.457 DEBUG:boto:path=/pklxkd-59/obj31 2023-05-31T20:50:33.457 DEBUG:boto:auth_path=/pklxkd-59/obj31 2023-05-31T20:50:33.458 DEBUG:boto:Method: PUT 2023-05-31T20:50:33.458 DEBUG:boto:Path: /pklxkd-59/obj31 2023-05-31T20:50:33.458 DEBUG:boto:Data: 2023-05-31T20:50:33.459 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-05-31T20:50:33.459 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:50:33.459 DEBUG:boto:Port: 8000 2023-05-31T20:50:33.460 DEBUG:boto:Params: {} 2023-05-31T20:50:33.460 DEBUG:boto:Token: None 2023-05-31T20:50:33.461 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Wed, 31 May 2023 20:50:33 GMT /pklxkd-59/obj31 2023-05-31T20:50:33.461 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:4TEmgk1ODqMDkrJyNJgiK2M+hKg= 2023-05-31T20:50:33.461 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Wed, 31 May 2023 20:50:33 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:4TEmgk1ODqMDkrJyNJgiK2M+hKg='} 2023-05-31T20:50:33.504 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000006b54928dcd727bbc-006477b319-1630-a1'), ('Date', 'Wed, 31 May 2023 20:50:33 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:50:33.505 DEBUG:boto:path=/pklxkd-59/ 2023-05-31T20:50:33.505 DEBUG:boto:auth_path=/pklxkd-59/ 2023-05-31T20:50:33.506 DEBUG:boto:Method: HEAD 2023-05-31T20:50:33.506 DEBUG:boto:Path: /pklxkd-59/ 2023-05-31T20:50:33.506 DEBUG:boto:Data: 2023-05-31T20:50:33.507 DEBUG:boto:Headers: {} 2023-05-31T20:50:33.507 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:50:33.507 DEBUG:boto:Port: 8000 2023-05-31T20:50:33.508 DEBUG:boto:Params: {} 2023-05-31T20:50:33.508 DEBUG:boto:Token: None 2023-05-31T20:50:33.508 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:50:33 GMT /pklxkd-59/ 2023-05-31T20:50:33.509 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:i27ZhanjX6um7diVFc/7GbiZjHw= 2023-05-31T20:50:33.509 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:50:33 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:i27ZhanjX6um7diVFc/7GbiZjHw=', 'Content-Length': '0'} 2023-05-31T20:50:33.512 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '26'), ('X-RGW-Bytes-Used', '78'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000fc8446a72e67f812-006477b319-1630-a1'), ('Content-Length', '0'), ('Date', 'Wed, 31 May 2023 20:50:33 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:50:33.513 DEBUG:boto:path=/pklxkd-59/obj62 2023-05-31T20:50:33.513 DEBUG:boto:auth_path=/pklxkd-59/obj62 2023-05-31T20:50:33.514 DEBUG:boto:Method: PUT 2023-05-31T20:50:33.514 DEBUG:boto:Path: /pklxkd-59/obj62 2023-05-31T20:50:33.514 DEBUG:boto:Data: 2023-05-31T20:50:33.515 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-05-31T20:50:33.515 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:50:33.515 DEBUG:boto:Port: 8000 2023-05-31T20:50:33.516 DEBUG:boto:Params: {} 2023-05-31T20:50:33.516 DEBUG:boto:Token: None 2023-05-31T20:50:33.517 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Wed, 31 May 2023 20:50:33 GMT /pklxkd-59/obj62 2023-05-31T20:50:33.517 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:Y/fdOOOkbSsYLJCxQcaVM51mPvc= 2023-05-31T20:50:33.517 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Wed, 31 May 2023 20:50:33 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:Y/fdOOOkbSsYLJCxQcaVM51mPvc='} 2023-05-31T20:50:33.522 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000003d7a2e1dcaeb7107-006477b319-1630-a1'), ('Date', 'Wed, 31 May 2023 20:50:33 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:50:33.523 DEBUG:boto:path=/pklxkd-59/ 2023-05-31T20:50:33.523 DEBUG:boto:auth_path=/pklxkd-59/ 2023-05-31T20:50:33.524 DEBUG:boto:Method: HEAD 2023-05-31T20:50:33.524 DEBUG:boto:Path: /pklxkd-59/ 2023-05-31T20:50:33.524 DEBUG:boto:Data: 2023-05-31T20:50:33.525 DEBUG:boto:Headers: {} 2023-05-31T20:50:33.525 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:50:33.525 DEBUG:boto:Port: 8000 2023-05-31T20:50:33.525 DEBUG:boto:Params: {} 2023-05-31T20:50:33.526 DEBUG:boto:Token: None 2023-05-31T20:50:33.526 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:50:33 GMT /pklxkd-59/ 2023-05-31T20:50:33.527 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:i27ZhanjX6um7diVFc/7GbiZjHw= 2023-05-31T20:50:33.527 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:50:33 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:i27ZhanjX6um7diVFc/7GbiZjHw=', 'Content-Length': '0'} 2023-05-31T20:50:33.530 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '27'), ('X-RGW-Bytes-Used', '81'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000004d3654ee1f2bac97-006477b319-1630-a1'), ('Content-Length', '0'), ('Date', 'Wed, 31 May 2023 20:50:33 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:50:33.530 DEBUG:boto:path=/pklxkd-59/obj93 2023-05-31T20:50:33.531 DEBUG:boto:auth_path=/pklxkd-59/obj93 2023-05-31T20:50:33.531 DEBUG:boto:Method: PUT 2023-05-31T20:50:33.532 DEBUG:boto:Path: /pklxkd-59/obj93 2023-05-31T20:50:33.532 DEBUG:boto:Data: 2023-05-31T20:50:33.532 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-05-31T20:50:33.533 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:50:33.533 DEBUG:boto:Port: 8000 2023-05-31T20:50:33.533 DEBUG:boto:Params: {} 2023-05-31T20:50:33.534 DEBUG:boto:Token: None 2023-05-31T20:50:33.534 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Wed, 31 May 2023 20:50:33 GMT /pklxkd-59/obj93 2023-05-31T20:50:33.534 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:PFvJ/qGELkO1KwrLnU1cgN96G+0= 2023-05-31T20:50:33.535 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Wed, 31 May 2023 20:50:33 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:PFvJ/qGELkO1KwrLnU1cgN96G+0='} 2023-05-31T20:50:33.576 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000cd6294f350bb0c0c-006477b319-1630-a1'), ('Date', 'Wed, 31 May 2023 20:50:33 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:50:33.577 DEBUG:boto:path=/pklxkd-59/ 2023-05-31T20:50:33.577 DEBUG:boto:auth_path=/pklxkd-59/ 2023-05-31T20:50:33.578 DEBUG:boto:Method: HEAD 2023-05-31T20:50:33.578 DEBUG:boto:Path: /pklxkd-59/ 2023-05-31T20:50:33.578 DEBUG:boto:Data: 2023-05-31T20:50:33.579 DEBUG:boto:Headers: {} 2023-05-31T20:50:33.579 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:50:33.579 DEBUG:boto:Port: 8000 2023-05-31T20:50:33.580 DEBUG:boto:Params: {} 2023-05-31T20:50:33.580 DEBUG:boto:Token: None 2023-05-31T20:50:33.580 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:50:33 GMT /pklxkd-59/ 2023-05-31T20:50:33.581 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:i27ZhanjX6um7diVFc/7GbiZjHw= 2023-05-31T20:50:33.581 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:50:33 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:i27ZhanjX6um7diVFc/7GbiZjHw=', 'Content-Length': '0'} 2023-05-31T20:50:33.584 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '28'), ('X-RGW-Bytes-Used', '84'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000000846de80a8585cc8-006477b319-1630-a1'), ('Content-Length', '0'), ('Date', 'Wed, 31 May 2023 20:50:33 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:50:33.585 DEBUG:boto:path=/pklxkd-59/obj124 2023-05-31T20:50:33.585 DEBUG:boto:auth_path=/pklxkd-59/obj124 2023-05-31T20:50:33.585 DEBUG:boto:Method: PUT 2023-05-31T20:50:33.586 DEBUG:boto:Path: /pklxkd-59/obj124 2023-05-31T20:50:33.586 DEBUG:boto:Data: 2023-05-31T20:50:33.586 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-05-31T20:50:33.587 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:50:33.587 DEBUG:boto:Port: 8000 2023-05-31T20:50:33.587 DEBUG:boto:Params: {} 2023-05-31T20:50:33.588 DEBUG:boto:Token: None 2023-05-31T20:50:33.588 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Wed, 31 May 2023 20:50:33 GMT /pklxkd-59/obj124 2023-05-31T20:50:33.589 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:Pn5ul7XxXJ2KCvh0KHS9PvZWOfk= 2023-05-31T20:50:33.589 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Wed, 31 May 2023 20:50:33 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:Pn5ul7XxXJ2KCvh0KHS9PvZWOfk='} 2023-05-31T20:50:33.594 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000080ef33bf2ca20201-006477b319-1630-a1'), ('Date', 'Wed, 31 May 2023 20:50:33 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:50:33.595 DEBUG:boto:path=/pklxkd-59/ 2023-05-31T20:50:33.595 DEBUG:boto:auth_path=/pklxkd-59/ 2023-05-31T20:50:33.596 DEBUG:boto:Method: HEAD 2023-05-31T20:50:33.596 DEBUG:boto:Path: /pklxkd-59/ 2023-05-31T20:50:33.596 DEBUG:boto:Data: 2023-05-31T20:50:33.597 DEBUG:boto:Headers: {} 2023-05-31T20:50:33.597 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:50:33.597 DEBUG:boto:Port: 8000 2023-05-31T20:50:33.598 DEBUG:boto:Params: {} 2023-05-31T20:50:33.598 DEBUG:boto:Token: None 2023-05-31T20:50:33.599 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:50:33 GMT /pklxkd-59/ 2023-05-31T20:50:33.599 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:i27ZhanjX6um7diVFc/7GbiZjHw= 2023-05-31T20:50:33.599 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:50:33 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:i27ZhanjX6um7diVFc/7GbiZjHw=', 'Content-Length': '0'} 2023-05-31T20:50:33.602 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '29'), ('X-RGW-Bytes-Used', '87'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000f004b89a4bd8347e-006477b319-1630-a1'), ('Content-Length', '0'), ('Date', 'Wed, 31 May 2023 20:50:33 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:50:33.603 DEBUG:boto:path=/pklxkd-59/obj155 2023-05-31T20:50:33.603 DEBUG:boto:auth_path=/pklxkd-59/obj155 2023-05-31T20:50:33.603 DEBUG:boto:Method: PUT 2023-05-31T20:50:33.604 DEBUG:boto:Path: /pklxkd-59/obj155 2023-05-31T20:50:33.604 DEBUG:boto:Data: 2023-05-31T20:50:33.605 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-05-31T20:50:33.605 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:50:33.605 DEBUG:boto:Port: 8000 2023-05-31T20:50:33.606 DEBUG:boto:Params: {} 2023-05-31T20:50:33.606 DEBUG:boto:Token: None 2023-05-31T20:50:33.606 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Wed, 31 May 2023 20:50:33 GMT /pklxkd-59/obj155 2023-05-31T20:50:33.607 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:42oIBGEnF2meAUSSgc2qIB2S0/Q= 2023-05-31T20:50:33.607 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Wed, 31 May 2023 20:50:33 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:42oIBGEnF2meAUSSgc2qIB2S0/Q='} 2023-05-31T20:50:33.652 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000f73a6280d74c4c04-006477b319-1630-a1'), ('Date', 'Wed, 31 May 2023 20:50:33 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:50:33.653 DEBUG:rgw_multi.tests:bucket checkpoint: source=a1 target=a2 bucket=pklxkd-59 2023-05-31T20:50:33.653 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'pklxkd-59', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-05-31T20:50:33.654 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'pklxkd-59', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-05-31T20:50:33.654 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket pklxkd-59 --source-zone a1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2023-05-31T20:50:33.710 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2023-05-31T20:50:33.710 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2023-05-31T20:50:33.751 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:50:33.747+0000 7ff56cadc780 1 waiting to reach latest gen 5, on 4.. 2023-05-31T20:50:38.752 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:50:38.747+0000 7ff56cadc780 1 waiting to reach latest gen 5, on 4.. 2023-05-31T20:50:43.753 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:50:43.748+0000 7ff56cadc780 1 waiting to reach latest gen 5, on 4.. 2023-05-31T20:50:48.753 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:50:48.749+0000 7ff56cadc780 1 waiting to reach latest gen 5, on 4.. 2023-05-31T20:50:53.756 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:50:53.752+0000 7ff56cadc780 1 waiting for incremental sync to catch up: 2023-05-31T20:50:53.757 INFO:teuthology.orchestra.run.smithi099.stderr: local status: [, , , , , , , , , , , , , , , , , , , , , , , , , , , , , , ] 2023-05-31T20:50:53.757 INFO:teuthology.orchestra.run.smithi099.stderr: remote markers: [, , 00000000002.105.6, , 00000000001.85.6, , 00000000001.72.6, , , , , , , , , , 00000000001.74.6, , , , , , , , , , , , , , 00000000001.86.6] 2023-05-31T20:50:58.758 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:50:58.754+0000 7ff56cadc780 1 waiting for incremental sync to catch up: 2023-05-31T20:50:58.759 INFO:teuthology.orchestra.run.smithi099.stderr: local status: [, , , , , , , , , , , , , , , , , , , , , , , , , , , , , , ] 2023-05-31T20:50:58.759 INFO:teuthology.orchestra.run.smithi099.stderr: remote markers: [, , 00000000002.105.6, , 00000000001.85.6, , 00000000001.72.6, , , , , , , , , , 00000000001.74.6, , , , , , , , , , , , , , 00000000001.86.6] 2023-05-31T20:51:03.761 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:51:03.757+0000 7ff56cadc780 1 waiting for incremental sync to catch up: 2023-05-31T20:51:03.761 INFO:teuthology.orchestra.run.smithi099.stderr: local status: [, , , , , , , , , , , , , , , , , , , , , , , , , , , , , , ] 2023-05-31T20:51:03.761 INFO:teuthology.orchestra.run.smithi099.stderr: remote markers: [, , 00000000002.105.6, , 00000000001.85.6, , 00000000001.72.6, , , , , , , , , , 00000000001.74.6, , , , , , , , , , , , , , 00000000001.86.6] 2023-05-31T20:51:08.763 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:51:08.759+0000 7ff56cadc780 1 waiting for incremental sync to catch up: 2023-05-31T20:51:08.763 INFO:teuthology.orchestra.run.smithi099.stderr: local status: [, , , , 00000000001.85.6, , , , , , , , , , , , 00000000001.74.6, , , , , , , , , , , , , , ] 2023-05-31T20:51:08.763 INFO:teuthology.orchestra.run.smithi099.stderr: remote markers: [, , 00000000002.105.6, , 00000000001.85.6, , 00000000001.72.6, , , , , , , , , , 00000000001.74.6, , , , , , , , , , , , , , 00000000001.86.6] 2023-05-31T20:51:13.765 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:51:13.760+0000 7ff56cadc780 1 waiting for incremental sync to catch up: 2023-05-31T20:51:13.765 INFO:teuthology.orchestra.run.smithi099.stderr: local status: [, , , , 00000000001.85.6, , , , , , , , , , , , 00000000001.74.6, , , , , , , , , , , , , , ] 2023-05-31T20:51:13.765 INFO:teuthology.orchestra.run.smithi099.stderr: remote markers: [, , 00000000002.105.6, , 00000000001.85.6, , 00000000001.72.6, , , , , , , , , , 00000000001.74.6, , , , , , , , , , , , , , 00000000001.86.6] 2023-05-31T20:51:18.767 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:51:18.762+0000 7ff56cadc780 1 waiting for incremental sync to catch up: 2023-05-31T20:51:18.768 INFO:teuthology.orchestra.run.smithi099.stderr: local status: [, , , , 00000000001.85.6, , , , , , , , , , , , 00000000001.74.6, , , , , , , , , , , , , , ] 2023-05-31T20:51:18.768 INFO:teuthology.orchestra.run.smithi099.stderr: remote markers: [, , 00000000002.105.6, , 00000000001.85.6, , 00000000001.72.6, , , , , , , , , , 00000000001.74.6, , , , , , , , , , , , , , 00000000001.86.6] 2023-05-31T20:51:23.769 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:51:23.764+0000 7ff56cadc780 1 waiting for incremental sync to catch up: 2023-05-31T20:51:23.770 INFO:teuthology.orchestra.run.smithi099.stderr: local status: [, , , , 00000000001.85.6, , , , , , , , , , , , 00000000001.74.6, , , , , , , , , , , , , , ] 2023-05-31T20:51:23.770 INFO:teuthology.orchestra.run.smithi099.stderr: remote markers: [, , 00000000002.105.6, , 00000000001.85.6, , 00000000001.72.6, , , , , , , , , , 00000000001.74.6, , , , , , , , , , , , , , 00000000001.86.6] 2023-05-31T20:51:28.771 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:51:28.766+0000 7ff56cadc780 1 waiting for incremental sync to catch up: 2023-05-31T20:51:28.771 INFO:teuthology.orchestra.run.smithi099.stderr: local status: [, , , , 00000000001.85.6, , , , , , , , , , , , 00000000001.74.6, , , , , , , , , , , , , , ] 2023-05-31T20:51:28.772 INFO:teuthology.orchestra.run.smithi099.stderr: remote markers: [, , 00000000002.105.6, , 00000000001.85.6, , 00000000001.72.6, , , , , , , , , , 00000000001.74.6, , , , , , , , , , , , , , 00000000001.86.6] 2023-05-31T20:51:33.773 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:51:33.769+0000 7ff56cadc780 1 waiting for incremental sync to catch up: 2023-05-31T20:51:33.774 INFO:teuthology.orchestra.run.smithi099.stderr: local status: [, , , , 00000000001.85.6, , , , , , , , , , , , 00000000001.74.6, , , , , , , , , , , , , , ] 2023-05-31T20:51:33.774 INFO:teuthology.orchestra.run.smithi099.stderr: remote markers: [, , 00000000002.105.6, , 00000000001.85.6, , 00000000001.72.6, , , , , , , , , , 00000000001.74.6, , , , , , , , , , , , , , 00000000001.86.6] 2023-05-31T20:51:38.776 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:51:38.772+0000 7ff56cadc780 1 waiting for incremental sync to catch up: 2023-05-31T20:51:38.776 INFO:teuthology.orchestra.run.smithi099.stderr: local status: [, , , , 00000000001.85.6, , , , , , , , , , , , 00000000001.74.6, , , , , , , , , , , , , , ] 2023-05-31T20:51:38.776 INFO:teuthology.orchestra.run.smithi099.stderr: remote markers: [, , 00000000002.105.6, , 00000000001.85.6, , 00000000001.72.6, , , , , , , , , , 00000000001.74.6, , , , , , , , , , , , , , 00000000001.86.6] 2023-05-31T20:51:43.778 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:51:43.773+0000 7ff56cadc780 1 waiting for incremental sync to catch up: 2023-05-31T20:51:43.778 INFO:teuthology.orchestra.run.smithi099.stderr: local status: [, , , , 00000000001.85.6, , , , , , , , , , , , 00000000001.74.6, , , , , , , , , , , , , , ] 2023-05-31T20:51:43.778 INFO:teuthology.orchestra.run.smithi099.stderr: remote markers: [, , 00000000002.105.6, , 00000000001.85.6, , 00000000001.72.6, , , , , , , , , , 00000000001.74.6, , , , , , , , , , , , , , 00000000001.86.6] 2023-05-31T20:51:48.780 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:51:48.776+0000 7ff56cadc780 1 bucket sync caught up with source: 2023-05-31T20:51:48.780 INFO:teuthology.orchestra.run.smithi099.stderr: local status: [, , 00000000002.105.6, , 00000000001.85.6, , 00000000001.72.6, , , , , , , , , , 00000000001.74.6, , , , , , , , , , , , , , 00000000001.86.6] 2023-05-31T20:51:48.781 INFO:teuthology.orchestra.run.smithi099.stderr: remote markers: [, , 00000000002.105.6, , 00000000001.85.6, , 00000000001.72.6, , , , , , , , , , 00000000001.74.6, , , , , , , , , , , , , , 00000000001.86.6] 2023-05-31T20:51:48.781 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:51:48.776+0000 7ff56cadc780 0 bucket checkpoint complete 2023-05-31T20:51:48.801 DEBUG:rgw_multi.tests:bucket checkpoint: source=a2 target=a1 bucket=pklxkd-59 2023-05-31T20:51:48.802 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'pklxkd-59', '--source-zone', 'a2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-05-31T20:51:48.802 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'pklxkd-59', '--source-zone', 'a2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-05-31T20:51:48.803 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket pklxkd-59 --source-zone a2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2023-05-31T20:51:48.864 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2023-05-31T20:51:48.864 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2023-05-31T20:51:48.923 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:51:48.919+0000 7f3406645780 1 waiting for incremental sync to catch up: 2023-05-31T20:51:48.923 INFO:teuthology.orchestra.run.smithi099.stderr: local status: [, 00000000002.3584.6, 00000000002.3906.6, 00000000002.2037.6, 00000000003.60.6, 00000000004.2031.6, 00000000001.377.6, 00000000001.3914.6, 00000000003.2037.6, 00000000005.2299.6, 00000000004.2018.6] 2023-05-31T20:51:48.923 INFO:teuthology.orchestra.run.smithi099.stderr: remote markers: [, 00000000003.3594.6, 00000000002.3906.6, 00000000002.2037.6, 00000000004.64.6, 00000000004.2031.6, 00000000001.377.6, 00000000001.3914.6, 00000000003.2037.6, 00000000006.2306.6, 00000000004.2018.6] 2023-05-31T20:51:53.924 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:51:53.919+0000 7f3406645780 1 waiting for incremental sync to catch up: 2023-05-31T20:51:53.924 INFO:teuthology.orchestra.run.smithi099.stderr: local status: [, 00000000002.3584.6, 00000000002.3906.6, 00000000002.2037.6, 00000000003.60.6, 00000000004.2031.6, 00000000001.377.6, 00000000001.3914.6, 00000000003.2037.6, 00000000005.2299.6, 00000000004.2018.6] 2023-05-31T20:51:53.925 INFO:teuthology.orchestra.run.smithi099.stderr: remote markers: [, 00000000003.3594.6, 00000000002.3906.6, 00000000002.2037.6, 00000000004.64.6, 00000000004.2031.6, 00000000001.377.6, 00000000001.3914.6, 00000000003.2037.6, 00000000006.2306.6, 00000000004.2018.6] 2023-05-31T20:51:58.925 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:51:58.920+0000 7f3406645780 1 waiting for incremental sync to catch up: 2023-05-31T20:51:58.926 INFO:teuthology.orchestra.run.smithi099.stderr: local status: [, 00000000002.3584.6, 00000000002.3906.6, 00000000002.2037.6, 00000000003.60.6, 00000000004.2031.6, 00000000001.377.6, 00000000001.3914.6, 00000000003.2037.6, 00000000005.2299.6, 00000000004.2018.6] 2023-05-31T20:51:58.926 INFO:teuthology.orchestra.run.smithi099.stderr: remote markers: [, 00000000003.3594.6, 00000000002.3906.6, 00000000002.2037.6, 00000000004.64.6, 00000000004.2031.6, 00000000001.377.6, 00000000001.3914.6, 00000000003.2037.6, 00000000006.2306.6, 00000000004.2018.6] 2023-05-31T20:52:03.927 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:52:03.922+0000 7f3406645780 1 waiting for incremental sync to catch up: 2023-05-31T20:52:03.927 INFO:teuthology.orchestra.run.smithi099.stderr: local status: [, 00000000002.3584.6, 00000000002.3906.6, 00000000002.2037.6, 00000000003.60.6, 00000000004.2031.6, 00000000001.377.6, 00000000001.3914.6, 00000000003.2037.6, 00000000005.2299.6, 00000000004.2018.6] 2023-05-31T20:52:03.927 INFO:teuthology.orchestra.run.smithi099.stderr: remote markers: [, 00000000003.3594.6, 00000000002.3906.6, 00000000002.2037.6, 00000000004.64.6, 00000000004.2031.6, 00000000001.377.6, 00000000001.3914.6, 00000000003.2037.6, 00000000006.2306.6, 00000000004.2018.6] 2023-05-31T20:52:08.928 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:52:08.923+0000 7f3406645780 1 bucket sync caught up with source: 2023-05-31T20:52:08.928 INFO:teuthology.orchestra.run.smithi099.stderr: local status: [, 00000000003.3594.6, 00000000002.3906.6, 00000000002.2037.6, 00000000004.64.6, 00000000004.2031.6, 00000000001.377.6, 00000000001.3914.6, 00000000003.2037.6, 00000000006.2306.6, 00000000004.2018.6] 2023-05-31T20:52:08.928 INFO:teuthology.orchestra.run.smithi099.stderr: remote markers: [, 00000000003.3594.6, 00000000002.3906.6, 00000000002.2037.6, 00000000004.64.6, 00000000004.2031.6, 00000000001.377.6, 00000000001.3914.6, 00000000003.2037.6, 00000000006.2306.6, 00000000004.2018.6] 2023-05-31T20:52:08.929 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:52:08.923+0000 7f3406645780 0 bucket checkpoint complete 2023-05-31T20:52:08.947 INFO:tasks.rgw_multi.zone_rados:comparing bucket=pklxkd-59 zones={a2, a1} 2023-05-31T20:52:08.947 DEBUG:boto:path=/pklxkd-59/ 2023-05-31T20:52:08.948 DEBUG:boto:auth_path=/pklxkd-59/ 2023-05-31T20:52:08.948 DEBUG:boto:Method: HEAD 2023-05-31T20:52:08.949 DEBUG:boto:Path: /pklxkd-59/ 2023-05-31T20:52:08.949 DEBUG:boto:Data: 2023-05-31T20:52:08.949 DEBUG:boto:Headers: {} 2023-05-31T20:52:08.950 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:52:08.950 DEBUG:boto:Port: 8001 2023-05-31T20:52:08.950 DEBUG:boto:Params: {} 2023-05-31T20:52:08.951 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8001} 2023-05-31T20:52:08.951 DEBUG:boto:Token: None 2023-05-31T20:52:08.952 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:52:08 GMT /pklxkd-59/ 2023-05-31T20:52:08.952 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:rYaWaermX4lzv9wbwShPkRIZEFU= 2023-05-31T20:52:08.953 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:52:08 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:rYaWaermX4lzv9wbwShPkRIZEFU=', 'Content-Length': '0'} 2023-05-31T20:52:08.956 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '30'), ('X-RGW-Bytes-Used', '90'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000008f2a296975635363-006477b378-1c17-a2'), ('Content-Length', '0'), ('Date', 'Wed, 31 May 2023 20:52:08 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:52:08.957 DEBUG:boto:path=/pklxkd-59/ 2023-05-31T20:52:08.957 DEBUG:boto:auth_path=/pklxkd-59/ 2023-05-31T20:52:08.957 DEBUG:boto:Method: HEAD 2023-05-31T20:52:08.958 DEBUG:boto:Path: /pklxkd-59/ 2023-05-31T20:52:08.958 DEBUG:boto:Data: 2023-05-31T20:52:08.958 DEBUG:boto:Headers: {} 2023-05-31T20:52:08.959 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:52:08.959 DEBUG:boto:Port: 8000 2023-05-31T20:52:08.959 DEBUG:boto:Params: {} 2023-05-31T20:52:08.960 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2023-05-31T20:52:08.960 DEBUG:boto:Token: None 2023-05-31T20:52:08.961 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:52:08 GMT /pklxkd-59/ 2023-05-31T20:52:08.961 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:rYaWaermX4lzv9wbwShPkRIZEFU= 2023-05-31T20:52:08.961 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:52:08 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:rYaWaermX4lzv9wbwShPkRIZEFU=', 'Content-Length': '0'} 2023-05-31T20:52:08.965 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '30'), ('X-RGW-Bytes-Used', '90'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000008131faaa6bb9c333-006477b378-1630-a1'), ('Content-Length', '0'), ('Date', 'Wed, 31 May 2023 20:52:08 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:52:08.966 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2023-05-31T20:52:08.966 DEBUG:boto:path=/pklxkd-59/ 2023-05-31T20:52:08.967 DEBUG:boto:auth_path=/pklxkd-59/ 2023-05-31T20:52:08.967 DEBUG:boto:path=/pklxkd-59/?versions&max-keys=999 2023-05-31T20:52:08.967 DEBUG:boto:auth_path=/pklxkd-59/?versions&max-keys=999 2023-05-31T20:52:08.968 DEBUG:boto:Method: GET 2023-05-31T20:52:08.968 DEBUG:boto:Path: /pklxkd-59/?versions&max-keys=999 2023-05-31T20:52:08.968 DEBUG:boto:Data: 2023-05-31T20:52:08.969 DEBUG:boto:Headers: {} 2023-05-31T20:52:08.969 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:52:08.969 DEBUG:boto:Port: 8001 2023-05-31T20:52:08.970 DEBUG:boto:Params: {} 2023-05-31T20:52:08.970 DEBUG:boto:Token: None 2023-05-31T20:52:08.971 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:52:08 GMT /pklxkd-59/?versions 2023-05-31T20:52:08.971 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:PG9QF+15QmqqzHjiofZ4uKag46E= 2023-05-31T20:52:08.971 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:52:08 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:PG9QF+15QmqqzHjiofZ4uKag46E=', 'Content-Length': '0'} 2023-05-31T20:52:08.974 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000001d2e7309344b7cd4-006477b378-1c17-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:52:08 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:52:08.974 DEBUG:boto:b'pklxkd-59999falseobj11nulltrue2023-05-31T20:39:26.427Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj115nulltrue2023-05-31T20:46:33.958Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj116nulltrue2023-05-31T20:48:13.052Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj124nulltrue2023-05-31T20:50:33.587Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj145nulltrue2023-05-31T20:48:13.107Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj155nulltrue2023-05-31T20:50:33.605Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj17nulltrue2023-05-31T20:43:07.977Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj19nulltrue2023-05-31T20:44:50.557Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj22nulltrue2023-05-31T20:39:26.473Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj23nulltrue2023-05-31T20:46:33.813Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj29nulltrue2023-05-31T20:48:12.956Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj31nulltrue2023-05-31T20:50:33.460Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj33nulltrue2023-05-31T20:39:26.534Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj34nulltrue2023-05-31T20:43:07.994Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj38nulltrue2023-05-31T20:44:50.610Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj44nulltrue2023-05-31T20:39:26.565Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj46nulltrue2023-05-31T20:46:33.830Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj51nulltrue2023-05-31T20:43:08.011Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj55nulltrue2023-05-31T20:39:26.630Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj57nulltrue2023-05-31T20:44:50.627Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj58nulltrue2023-05-31T20:48:12.976Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj62nulltrue2023-05-31T20:50:33.515Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj68nulltrue2023-05-31T20:43:08.066Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj69nulltrue2023-05-31T20:46:33.886Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj76nulltrue2023-05-31T20:44:50.682Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj85nulltrue2023-05-31T20:43:08.086Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj87nulltrue2023-05-31T20:48:13.031Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj92nulltrue2023-05-31T20:46:33.904Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj93nulltrue2023-05-31T20:50:33.533Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj95nulltrue2023-05-31T20:44:50.705Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:52:08.976 DEBUG:tasks.rgw_multi.zone_rados:o=obj11 2023-05-31T20:52:08.977 DEBUG:tasks.rgw_multi.zone_rados:o=obj115 2023-05-31T20:52:08.977 DEBUG:tasks.rgw_multi.zone_rados:o=obj116 2023-05-31T20:52:08.977 DEBUG:tasks.rgw_multi.zone_rados:o=obj124 2023-05-31T20:52:08.978 DEBUG:tasks.rgw_multi.zone_rados:o=obj145 2023-05-31T20:52:08.978 DEBUG:tasks.rgw_multi.zone_rados:o=obj155 2023-05-31T20:52:08.978 DEBUG:tasks.rgw_multi.zone_rados:o=obj17 2023-05-31T20:52:08.979 DEBUG:tasks.rgw_multi.zone_rados:o=obj19 2023-05-31T20:52:08.979 DEBUG:tasks.rgw_multi.zone_rados:o=obj22 2023-05-31T20:52:08.979 DEBUG:tasks.rgw_multi.zone_rados:o=obj23 2023-05-31T20:52:08.980 DEBUG:tasks.rgw_multi.zone_rados:o=obj29 2023-05-31T20:52:08.980 DEBUG:tasks.rgw_multi.zone_rados:o=obj31 2023-05-31T20:52:08.980 DEBUG:tasks.rgw_multi.zone_rados:o=obj33 2023-05-31T20:52:08.980 DEBUG:tasks.rgw_multi.zone_rados:o=obj34 2023-05-31T20:52:08.981 DEBUG:tasks.rgw_multi.zone_rados:o=obj38 2023-05-31T20:52:08.981 DEBUG:tasks.rgw_multi.zone_rados:o=obj44 2023-05-31T20:52:08.981 DEBUG:tasks.rgw_multi.zone_rados:o=obj46 2023-05-31T20:52:08.982 DEBUG:tasks.rgw_multi.zone_rados:o=obj51 2023-05-31T20:52:08.982 DEBUG:tasks.rgw_multi.zone_rados:o=obj55 2023-05-31T20:52:08.982 DEBUG:tasks.rgw_multi.zone_rados:o=obj57 2023-05-31T20:52:08.982 DEBUG:tasks.rgw_multi.zone_rados:o=obj58 2023-05-31T20:52:08.983 DEBUG:tasks.rgw_multi.zone_rados:o=obj62 2023-05-31T20:52:08.983 DEBUG:tasks.rgw_multi.zone_rados:o=obj68 2023-05-31T20:52:08.983 DEBUG:tasks.rgw_multi.zone_rados:o=obj69 2023-05-31T20:52:08.984 DEBUG:tasks.rgw_multi.zone_rados:o=obj76 2023-05-31T20:52:08.984 DEBUG:tasks.rgw_multi.zone_rados:o=obj85 2023-05-31T20:52:08.984 DEBUG:tasks.rgw_multi.zone_rados:o=obj87 2023-05-31T20:52:08.985 DEBUG:tasks.rgw_multi.zone_rados:o=obj92 2023-05-31T20:52:08.985 DEBUG:tasks.rgw_multi.zone_rados:o=obj93 2023-05-31T20:52:08.985 DEBUG:tasks.rgw_multi.zone_rados:o=obj95 2023-05-31T20:52:08.985 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2023-05-31T20:52:08.986 DEBUG:boto:path=/pklxkd-59/ 2023-05-31T20:52:08.986 DEBUG:boto:auth_path=/pklxkd-59/ 2023-05-31T20:52:08.987 DEBUG:boto:path=/pklxkd-59/?versions&max-keys=999 2023-05-31T20:52:08.987 DEBUG:boto:auth_path=/pklxkd-59/?versions&max-keys=999 2023-05-31T20:52:08.987 DEBUG:boto:Method: GET 2023-05-31T20:52:08.988 DEBUG:boto:Path: /pklxkd-59/?versions&max-keys=999 2023-05-31T20:52:08.988 DEBUG:boto:Data: 2023-05-31T20:52:08.989 DEBUG:boto:Headers: {} 2023-05-31T20:52:08.989 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:52:08.989 DEBUG:boto:Port: 8000 2023-05-31T20:52:08.990 DEBUG:boto:Params: {} 2023-05-31T20:52:08.990 DEBUG:boto:Token: None 2023-05-31T20:52:08.990 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:52:08 GMT /pklxkd-59/?versions 2023-05-31T20:52:08.991 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:PG9QF+15QmqqzHjiofZ4uKag46E= 2023-05-31T20:52:08.991 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:52:08 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:PG9QF+15QmqqzHjiofZ4uKag46E=', 'Content-Length': '0'} 2023-05-31T20:52:08.995 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000007adf2d24b89ec1f9-006477b378-1630-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:52:08 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:52:08.995 DEBUG:boto:b'pklxkd-59999falseobj11nulltrue2023-05-31T20:39:26.427Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj115nulltrue2023-05-31T20:46:33.958Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj116nulltrue2023-05-31T20:48:13.052Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj124nulltrue2023-05-31T20:50:33.587Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj145nulltrue2023-05-31T20:48:13.107Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj155nulltrue2023-05-31T20:50:33.605Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj17nulltrue2023-05-31T20:43:07.977Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj19nulltrue2023-05-31T20:44:50.557Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj22nulltrue2023-05-31T20:39:26.473Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj23nulltrue2023-05-31T20:46:33.813Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj29nulltrue2023-05-31T20:48:12.956Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj31nulltrue2023-05-31T20:50:33.460Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj33nulltrue2023-05-31T20:39:26.534Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj34nulltrue2023-05-31T20:43:07.994Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj38nulltrue2023-05-31T20:44:50.610Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj44nulltrue2023-05-31T20:39:26.565Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj46nulltrue2023-05-31T20:46:33.830Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj51nulltrue2023-05-31T20:43:08.011Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj55nulltrue2023-05-31T20:39:26.630Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj57nulltrue2023-05-31T20:44:50.627Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj58nulltrue2023-05-31T20:48:12.976Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj62nulltrue2023-05-31T20:50:33.515Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj68nulltrue2023-05-31T20:43:08.066Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj69nulltrue2023-05-31T20:46:33.886Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj76nulltrue2023-05-31T20:44:50.682Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj85nulltrue2023-05-31T20:43:08.086Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj87nulltrue2023-05-31T20:48:13.031Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj92nulltrue2023-05-31T20:46:33.904Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj93nulltrue2023-05-31T20:50:33.533Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj95nulltrue2023-05-31T20:44:50.705Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:52:08.997 DEBUG:tasks.rgw_multi.zone_rados:o=obj11 2023-05-31T20:52:08.997 DEBUG:tasks.rgw_multi.zone_rados:o=obj115 2023-05-31T20:52:08.998 DEBUG:tasks.rgw_multi.zone_rados:o=obj116 2023-05-31T20:52:08.998 DEBUG:tasks.rgw_multi.zone_rados:o=obj124 2023-05-31T20:52:08.998 DEBUG:tasks.rgw_multi.zone_rados:o=obj145 2023-05-31T20:52:08.999 DEBUG:tasks.rgw_multi.zone_rados:o=obj155 2023-05-31T20:52:08.999 DEBUG:tasks.rgw_multi.zone_rados:o=obj17 2023-05-31T20:52:08.999 DEBUG:tasks.rgw_multi.zone_rados:o=obj19 2023-05-31T20:52:09.000 DEBUG:tasks.rgw_multi.zone_rados:o=obj22 2023-05-31T20:52:09.000 DEBUG:tasks.rgw_multi.zone_rados:o=obj23 2023-05-31T20:52:09.000 DEBUG:tasks.rgw_multi.zone_rados:o=obj29 2023-05-31T20:52:09.000 DEBUG:tasks.rgw_multi.zone_rados:o=obj31 2023-05-31T20:52:09.001 DEBUG:tasks.rgw_multi.zone_rados:o=obj33 2023-05-31T20:52:09.001 DEBUG:tasks.rgw_multi.zone_rados:o=obj34 2023-05-31T20:52:09.001 DEBUG:tasks.rgw_multi.zone_rados:o=obj38 2023-05-31T20:52:09.002 DEBUG:tasks.rgw_multi.zone_rados:o=obj44 2023-05-31T20:52:09.002 DEBUG:tasks.rgw_multi.zone_rados:o=obj46 2023-05-31T20:52:09.002 DEBUG:tasks.rgw_multi.zone_rados:o=obj51 2023-05-31T20:52:09.003 DEBUG:tasks.rgw_multi.zone_rados:o=obj55 2023-05-31T20:52:09.003 DEBUG:tasks.rgw_multi.zone_rados:o=obj57 2023-05-31T20:52:09.003 DEBUG:tasks.rgw_multi.zone_rados:o=obj58 2023-05-31T20:52:09.003 DEBUG:tasks.rgw_multi.zone_rados:o=obj62 2023-05-31T20:52:09.004 DEBUG:tasks.rgw_multi.zone_rados:o=obj68 2023-05-31T20:52:09.004 DEBUG:tasks.rgw_multi.zone_rados:o=obj69 2023-05-31T20:52:09.004 DEBUG:tasks.rgw_multi.zone_rados:o=obj76 2023-05-31T20:52:09.005 DEBUG:tasks.rgw_multi.zone_rados:o=obj85 2023-05-31T20:52:09.005 DEBUG:tasks.rgw_multi.zone_rados:o=obj87 2023-05-31T20:52:09.005 DEBUG:tasks.rgw_multi.zone_rados:o=obj92 2023-05-31T20:52:09.005 DEBUG:tasks.rgw_multi.zone_rados:o=obj93 2023-05-31T20:52:09.006 DEBUG:tasks.rgw_multi.zone_rados:o=obj95 2023-05-31T20:52:09.006 DEBUG:boto:path=/pklxkd-59/ 2023-05-31T20:52:09.006 DEBUG:boto:auth_path=/pklxkd-59/ 2023-05-31T20:52:09.007 DEBUG:boto:path=/pklxkd-59/?versions&max-keys=999 2023-05-31T20:52:09.007 DEBUG:boto:auth_path=/pklxkd-59/?versions&max-keys=999 2023-05-31T20:52:09.007 DEBUG:boto:Method: GET 2023-05-31T20:52:09.008 DEBUG:boto:Path: /pklxkd-59/?versions&max-keys=999 2023-05-31T20:52:09.008 DEBUG:boto:Data: 2023-05-31T20:52:09.008 DEBUG:boto:Headers: {} 2023-05-31T20:52:09.009 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:52:09.009 DEBUG:boto:Port: 8001 2023-05-31T20:52:09.009 DEBUG:boto:Params: {} 2023-05-31T20:52:09.010 DEBUG:boto:Token: None 2023-05-31T20:52:09.010 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:52:09 GMT /pklxkd-59/?versions 2023-05-31T20:52:09.011 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:v9PIOp9yove2aWt8+i9T6x45Fv8= 2023-05-31T20:52:09.011 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:52:09 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:v9PIOp9yove2aWt8+i9T6x45Fv8=', 'Content-Length': '0'} 2023-05-31T20:52:09.014 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000092b54fe0b04b5a6b-006477b379-1c17-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:52:09 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:52:09.014 DEBUG:boto:b'pklxkd-59999falseobj11nulltrue2023-05-31T20:39:26.427Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj115nulltrue2023-05-31T20:46:33.958Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj116nulltrue2023-05-31T20:48:13.052Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj124nulltrue2023-05-31T20:50:33.587Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj145nulltrue2023-05-31T20:48:13.107Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj155nulltrue2023-05-31T20:50:33.605Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj17nulltrue2023-05-31T20:43:07.977Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj19nulltrue2023-05-31T20:44:50.557Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj22nulltrue2023-05-31T20:39:26.473Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj23nulltrue2023-05-31T20:46:33.813Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj29nulltrue2023-05-31T20:48:12.956Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj31nulltrue2023-05-31T20:50:33.460Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj33nulltrue2023-05-31T20:39:26.534Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj34nulltrue2023-05-31T20:43:07.994Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj38nulltrue2023-05-31T20:44:50.610Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj44nulltrue2023-05-31T20:39:26.565Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj46nulltrue2023-05-31T20:46:33.830Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj51nulltrue2023-05-31T20:43:08.011Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj55nulltrue2023-05-31T20:39:26.630Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj57nulltrue2023-05-31T20:44:50.627Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj58nulltrue2023-05-31T20:48:12.976Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj62nulltrue2023-05-31T20:50:33.515Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj68nulltrue2023-05-31T20:43:08.066Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj69nulltrue2023-05-31T20:46:33.886Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj76nulltrue2023-05-31T20:44:50.682Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj85nulltrue2023-05-31T20:43:08.086Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj87nulltrue2023-05-31T20:48:13.031Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj92nulltrue2023-05-31T20:46:33.904Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj93nulltrue2023-05-31T20:50:33.533Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj95nulltrue2023-05-31T20:44:50.705Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:52:09.016 DEBUG:boto:path=/pklxkd-59/ 2023-05-31T20:52:09.016 DEBUG:boto:auth_path=/pklxkd-59/ 2023-05-31T20:52:09.017 DEBUG:boto:path=/pklxkd-59/?versions&max-keys=999 2023-05-31T20:52:09.017 DEBUG:boto:auth_path=/pklxkd-59/?versions&max-keys=999 2023-05-31T20:52:09.017 DEBUG:boto:Method: GET 2023-05-31T20:52:09.018 DEBUG:boto:Path: /pklxkd-59/?versions&max-keys=999 2023-05-31T20:52:09.018 DEBUG:boto:Data: 2023-05-31T20:52:09.018 DEBUG:boto:Headers: {} 2023-05-31T20:52:09.019 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:52:09.019 DEBUG:boto:Port: 8000 2023-05-31T20:52:09.020 DEBUG:boto:Params: {} 2023-05-31T20:52:09.020 DEBUG:boto:Token: None 2023-05-31T20:52:09.020 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:52:09 GMT /pklxkd-59/?versions 2023-05-31T20:52:09.021 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:v9PIOp9yove2aWt8+i9T6x45Fv8= 2023-05-31T20:52:09.021 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:52:09 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:v9PIOp9yove2aWt8+i9T6x45Fv8=', 'Content-Length': '0'} 2023-05-31T20:52:09.025 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000000c476398949a21d3-006477b379-1630-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:52:09 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:52:09.025 DEBUG:boto:b'pklxkd-59999falseobj11nulltrue2023-05-31T20:39:26.427Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj115nulltrue2023-05-31T20:46:33.958Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj116nulltrue2023-05-31T20:48:13.052Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj124nulltrue2023-05-31T20:50:33.587Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj145nulltrue2023-05-31T20:48:13.107Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj155nulltrue2023-05-31T20:50:33.605Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj17nulltrue2023-05-31T20:43:07.977Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj19nulltrue2023-05-31T20:44:50.557Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj22nulltrue2023-05-31T20:39:26.473Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj23nulltrue2023-05-31T20:46:33.813Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj29nulltrue2023-05-31T20:48:12.956Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj31nulltrue2023-05-31T20:50:33.460Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj33nulltrue2023-05-31T20:39:26.534Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj34nulltrue2023-05-31T20:43:07.994Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj38nulltrue2023-05-31T20:44:50.610Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj44nulltrue2023-05-31T20:39:26.565Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj46nulltrue2023-05-31T20:46:33.830Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj51nulltrue2023-05-31T20:43:08.011Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj55nulltrue2023-05-31T20:39:26.630Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj57nulltrue2023-05-31T20:44:50.627Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj58nulltrue2023-05-31T20:48:12.976Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj62nulltrue2023-05-31T20:50:33.515Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj68nulltrue2023-05-31T20:43:08.066Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj69nulltrue2023-05-31T20:46:33.886Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj76nulltrue2023-05-31T20:44:50.682Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj85nulltrue2023-05-31T20:43:08.086Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj87nulltrue2023-05-31T20:48:13.031Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj92nulltrue2023-05-31T20:46:33.904Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj93nulltrue2023-05-31T20:50:33.533Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj95nulltrue2023-05-31T20:44:50.705Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:52:09.027 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj11 2023-05-31T20:52:09.028 DEBUG:boto:path=/pklxkd-59/obj11 2023-05-31T20:52:09.028 DEBUG:boto:auth_path=/pklxkd-59/obj11 2023-05-31T20:52:09.028 DEBUG:boto:path=/pklxkd-59/obj11?versionId=null 2023-05-31T20:52:09.029 DEBUG:boto:auth_path=/pklxkd-59/obj11?versionId=null 2023-05-31T20:52:09.029 DEBUG:boto:Method: GET 2023-05-31T20:52:09.029 DEBUG:boto:Path: /pklxkd-59/obj11?versionId=null 2023-05-31T20:52:09.030 DEBUG:boto:Data: 2023-05-31T20:52:09.030 DEBUG:boto:Headers: {} 2023-05-31T20:52:09.030 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:52:09.031 DEBUG:boto:Port: 8001 2023-05-31T20:52:09.031 DEBUG:boto:Params: {} 2023-05-31T20:52:09.032 DEBUG:boto:Token: None 2023-05-31T20:52:09.032 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:52:09 GMT /pklxkd-59/obj11?versionId=null 2023-05-31T20:52:09.032 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:bCrrxmNdqlm6/2T18b9aH4W1V1E= 2023-05-31T20:52:09.033 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:52:09 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:bCrrxmNdqlm6/2T18b9aH4W1V1E=', 'Content-Length': '0'} 2023-05-31T20:52:09.035 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:39:26 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a9d7768159ef4156-006477b379-1c17-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:52:09 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:52:09.035 DEBUG:boto:path=/pklxkd-59/obj11 2023-05-31T20:52:09.036 DEBUG:boto:auth_path=/pklxkd-59/obj11 2023-05-31T20:52:09.036 DEBUG:boto:path=/pklxkd-59/obj11?versionId=null 2023-05-31T20:52:09.036 DEBUG:boto:auth_path=/pklxkd-59/obj11?versionId=null 2023-05-31T20:52:09.037 DEBUG:boto:Method: GET 2023-05-31T20:52:09.037 DEBUG:boto:Path: /pklxkd-59/obj11?versionId=null 2023-05-31T20:52:09.038 DEBUG:boto:Data: 2023-05-31T20:52:09.038 DEBUG:boto:Headers: {} 2023-05-31T20:52:09.038 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:52:09.039 DEBUG:boto:Port: 8000 2023-05-31T20:52:09.039 DEBUG:boto:Params: {} 2023-05-31T20:52:09.039 DEBUG:boto:Token: None 2023-05-31T20:52:09.040 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:52:09 GMT /pklxkd-59/obj11?versionId=null 2023-05-31T20:52:09.040 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:bCrrxmNdqlm6/2T18b9aH4W1V1E= 2023-05-31T20:52:09.041 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:52:09 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:bCrrxmNdqlm6/2T18b9aH4W1V1E=', 'Content-Length': '0'} 2023-05-31T20:52:09.042 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:39:26 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009b2492d30fb6d4ab-006477b379-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:52:09 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:52:09.084 DEBUG:boto:path=/pklxkd-59/obj11 2023-05-31T20:52:09.085 DEBUG:boto:auth_path=/pklxkd-59/obj11 2023-05-31T20:52:09.085 DEBUG:boto:path=/pklxkd-59/obj11?versionId=null 2023-05-31T20:52:09.085 DEBUG:boto:auth_path=/pklxkd-59/obj11?versionId=null 2023-05-31T20:52:09.086 DEBUG:boto:Method: HEAD 2023-05-31T20:52:09.086 DEBUG:boto:Path: /pklxkd-59/obj11?versionId=null 2023-05-31T20:52:09.086 DEBUG:boto:Data: 2023-05-31T20:52:09.087 DEBUG:boto:Headers: {} 2023-05-31T20:52:09.087 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:52:09.087 DEBUG:boto:Port: 8001 2023-05-31T20:52:09.088 DEBUG:boto:Params: {} 2023-05-31T20:52:09.088 DEBUG:boto:Token: None 2023-05-31T20:52:09.088 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:52:09 GMT /pklxkd-59/obj11?versionId=null 2023-05-31T20:52:09.089 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:6ECPo5CtV9dk/bXyGdF5v5WWTdM= 2023-05-31T20:52:09.089 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:52:09 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:6ECPo5CtV9dk/bXyGdF5v5WWTdM=', 'Content-Length': '0'} 2023-05-31T20:52:09.091 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:39:26 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a68c175ad474c7ae-006477b379-1c17-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:52:09 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:52:09.091 DEBUG:boto:path=/pklxkd-59/obj11 2023-05-31T20:52:09.092 DEBUG:boto:auth_path=/pklxkd-59/obj11 2023-05-31T20:52:09.092 DEBUG:boto:path=/pklxkd-59/obj11?versionId=null 2023-05-31T20:52:09.092 DEBUG:boto:auth_path=/pklxkd-59/obj11?versionId=null 2023-05-31T20:52:09.093 DEBUG:boto:Method: HEAD 2023-05-31T20:52:09.093 DEBUG:boto:Path: /pklxkd-59/obj11?versionId=null 2023-05-31T20:52:09.093 DEBUG:boto:Data: 2023-05-31T20:52:09.094 DEBUG:boto:Headers: {} 2023-05-31T20:52:09.094 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:52:09.094 DEBUG:boto:Port: 8000 2023-05-31T20:52:09.095 DEBUG:boto:Params: {} 2023-05-31T20:52:09.095 DEBUG:boto:Token: None 2023-05-31T20:52:09.095 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:52:09 GMT /pklxkd-59/obj11?versionId=null 2023-05-31T20:52:09.096 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:6ECPo5CtV9dk/bXyGdF5v5WWTdM= 2023-05-31T20:52:09.096 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:52:09 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:6ECPo5CtV9dk/bXyGdF5v5WWTdM=', 'Content-Length': '0'} 2023-05-31T20:52:09.098 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:39:26 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004ce73d768f4cbd3b-006477b379-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:52:09 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:52:09.098 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj11 2023-05-31T20:52:09.099 DEBUG:boto:path=/pklxkd-59/obj11 2023-05-31T20:52:09.099 DEBUG:boto:auth_path=/pklxkd-59/obj11 2023-05-31T20:52:09.099 DEBUG:boto:path=/pklxkd-59/obj11?versionId=null 2023-05-31T20:52:09.100 DEBUG:boto:auth_path=/pklxkd-59/obj11?versionId=null 2023-05-31T20:52:09.100 DEBUG:boto:Method: GET 2023-05-31T20:52:09.101 DEBUG:boto:Path: /pklxkd-59/obj11?versionId=null 2023-05-31T20:52:09.101 DEBUG:boto:Data: 2023-05-31T20:52:09.101 DEBUG:boto:Headers: {} 2023-05-31T20:52:09.102 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:52:09.102 DEBUG:boto:Port: 8001 2023-05-31T20:52:09.102 DEBUG:boto:Params: {} 2023-05-31T20:52:09.103 DEBUG:boto:Token: None 2023-05-31T20:52:09.103 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:52:09 GMT /pklxkd-59/obj11?versionId=null 2023-05-31T20:52:09.104 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:bCrrxmNdqlm6/2T18b9aH4W1V1E= 2023-05-31T20:52:09.104 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:52:09 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:bCrrxmNdqlm6/2T18b9aH4W1V1E=', 'Content-Length': '0'} 2023-05-31T20:52:09.106 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:39:26 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e33b14776a908a9c-006477b379-1c17-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:52:09 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:52:09.107 DEBUG:boto:path=/pklxkd-59/obj11 2023-05-31T20:52:09.107 DEBUG:boto:auth_path=/pklxkd-59/obj11 2023-05-31T20:52:09.107 DEBUG:boto:path=/pklxkd-59/obj11?versionId=null 2023-05-31T20:52:09.108 DEBUG:boto:auth_path=/pklxkd-59/obj11?versionId=null 2023-05-31T20:52:09.108 DEBUG:boto:Method: GET 2023-05-31T20:52:09.108 DEBUG:boto:Path: /pklxkd-59/obj11?versionId=null 2023-05-31T20:52:09.109 DEBUG:boto:Data: 2023-05-31T20:52:09.109 DEBUG:boto:Headers: {} 2023-05-31T20:52:09.109 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:52:09.110 DEBUG:boto:Port: 8000 2023-05-31T20:52:09.110 DEBUG:boto:Params: {} 2023-05-31T20:52:09.111 DEBUG:boto:Token: None 2023-05-31T20:52:09.111 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:52:09 GMT /pklxkd-59/obj11?versionId=null 2023-05-31T20:52:09.111 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:bCrrxmNdqlm6/2T18b9aH4W1V1E= 2023-05-31T20:52:09.112 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:52:09 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:bCrrxmNdqlm6/2T18b9aH4W1V1E=', 'Content-Length': '0'} 2023-05-31T20:52:09.114 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:39:26 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000028b5ac8e3bd13819-006477b379-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:52:09 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:52:09.114 DEBUG:boto:path=/pklxkd-59/ 2023-05-31T20:52:09.115 DEBUG:boto:auth_path=/pklxkd-59/ 2023-05-31T20:52:09.115 DEBUG:boto:path=/pklxkd-59/?prefix=obj11 2023-05-31T20:52:09.115 DEBUG:boto:auth_path=/pklxkd-59/?prefix=obj11 2023-05-31T20:52:09.116 DEBUG:boto:Method: GET 2023-05-31T20:52:09.116 DEBUG:boto:Path: /pklxkd-59/?prefix=obj11 2023-05-31T20:52:09.116 DEBUG:boto:Data: 2023-05-31T20:52:09.117 DEBUG:boto:Headers: {} 2023-05-31T20:52:09.117 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:52:09.117 DEBUG:boto:Port: 8001 2023-05-31T20:52:09.118 DEBUG:boto:Params: {} 2023-05-31T20:52:09.118 DEBUG:boto:Token: None 2023-05-31T20:52:09.119 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:52:09 GMT /pklxkd-59/ 2023-05-31T20:52:09.119 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:OXHTYn90MKDcz4kG+dbepWeHVJ4= 2023-05-31T20:52:09.119 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:52:09 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:OXHTYn90MKDcz4kG+dbepWeHVJ4=', 'Content-Length': '0'} 2023-05-31T20:52:09.122 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e241f4d9275f8bf1-006477b379-1c17-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:52:09 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:52:09.122 DEBUG:boto:b'pklxkd-59obj111000falseobj112023-05-31T20:39:26.427Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj1152023-05-31T20:46:33.958Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj1162023-05-31T20:48:13.052Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:52:09.123 DEBUG:boto:path=/pklxkd-59/ 2023-05-31T20:52:09.123 DEBUG:boto:auth_path=/pklxkd-59/ 2023-05-31T20:52:09.124 DEBUG:boto:path=/pklxkd-59/?prefix=obj11 2023-05-31T20:52:09.124 DEBUG:boto:auth_path=/pklxkd-59/?prefix=obj11 2023-05-31T20:52:09.124 DEBUG:boto:Method: GET 2023-05-31T20:52:09.125 DEBUG:boto:Path: /pklxkd-59/?prefix=obj11 2023-05-31T20:52:09.125 DEBUG:boto:Data: 2023-05-31T20:52:09.125 DEBUG:boto:Headers: {} 2023-05-31T20:52:09.126 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:52:09.126 DEBUG:boto:Port: 8000 2023-05-31T20:52:09.126 DEBUG:boto:Params: {} 2023-05-31T20:52:09.127 DEBUG:boto:Token: None 2023-05-31T20:52:09.127 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:52:09 GMT /pklxkd-59/ 2023-05-31T20:52:09.128 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:OXHTYn90MKDcz4kG+dbepWeHVJ4= 2023-05-31T20:52:09.128 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:52:09 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:OXHTYn90MKDcz4kG+dbepWeHVJ4=', 'Content-Length': '0'} 2023-05-31T20:52:09.132 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000006a5a00504c2ebc74-006477b379-1630-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:52:09 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:52:09.132 DEBUG:boto:b'pklxkd-59obj111000falseobj112023-05-31T20:39:26.427Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj1152023-05-31T20:46:33.958Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj1162023-05-31T20:48:13.052Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:52:09.133 DEBUG:boto:path=/pklxkd-59/obj11 2023-05-31T20:52:09.133 DEBUG:boto:auth_path=/pklxkd-59/obj11 2023-05-31T20:52:09.134 DEBUG:boto:Method: HEAD 2023-05-31T20:52:09.134 DEBUG:boto:Path: /pklxkd-59/obj11 2023-05-31T20:52:09.134 DEBUG:boto:Data: 2023-05-31T20:52:09.135 DEBUG:boto:Headers: {} 2023-05-31T20:52:09.135 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:52:09.135 DEBUG:boto:Port: 8001 2023-05-31T20:52:09.136 DEBUG:boto:Params: {} 2023-05-31T20:52:09.136 DEBUG:boto:Token: None 2023-05-31T20:52:09.136 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:52:09 GMT /pklxkd-59/obj11 2023-05-31T20:52:09.137 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:JvXFb9ghDIXsYT+RZah9APGeks0= 2023-05-31T20:52:09.137 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:52:09 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:JvXFb9ghDIXsYT+RZah9APGeks0=', 'Content-Length': '0'} 2023-05-31T20:52:09.139 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:39:26 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005faefff3b5914146-006477b379-1c17-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:52:09 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:52:09.139 DEBUG:boto:path=/pklxkd-59/obj11 2023-05-31T20:52:09.140 DEBUG:boto:auth_path=/pklxkd-59/obj11 2023-05-31T20:52:09.140 DEBUG:boto:Method: HEAD 2023-05-31T20:52:09.140 DEBUG:boto:Path: /pklxkd-59/obj11 2023-05-31T20:52:09.141 DEBUG:boto:Data: 2023-05-31T20:52:09.141 DEBUG:boto:Headers: {} 2023-05-31T20:52:09.141 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:52:09.142 DEBUG:boto:Port: 8000 2023-05-31T20:52:09.142 DEBUG:boto:Params: {} 2023-05-31T20:52:09.142 DEBUG:boto:Token: None 2023-05-31T20:52:09.143 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:52:09 GMT /pklxkd-59/obj11 2023-05-31T20:52:09.143 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:JvXFb9ghDIXsYT+RZah9APGeks0= 2023-05-31T20:52:09.144 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:52:09 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:JvXFb9ghDIXsYT+RZah9APGeks0=', 'Content-Length': '0'} 2023-05-31T20:52:09.145 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:39:26 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006f527e22e466198c-006477b379-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:52:09 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:52:09.146 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj11 2023-05-31T20:52:09.146 DEBUG:boto:path=/pklxkd-59/obj11 2023-05-31T20:52:09.146 DEBUG:boto:auth_path=/pklxkd-59/obj11 2023-05-31T20:52:09.147 DEBUG:boto:Method: GET 2023-05-31T20:52:09.147 DEBUG:boto:Path: /pklxkd-59/obj11 2023-05-31T20:52:09.147 DEBUG:boto:Data: 2023-05-31T20:52:09.148 DEBUG:boto:Headers: {} 2023-05-31T20:52:09.148 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:52:09.149 DEBUG:boto:Port: 8001 2023-05-31T20:52:09.149 DEBUG:boto:Params: {} 2023-05-31T20:52:09.149 DEBUG:boto:Token: None 2023-05-31T20:52:09.150 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:52:09 GMT /pklxkd-59/obj11 2023-05-31T20:52:09.150 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:iZ8V6a3LionZQPRBhHC2lupqUcY= 2023-05-31T20:52:09.151 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:52:09 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:iZ8V6a3LionZQPRBhHC2lupqUcY=', 'Content-Length': '0'} 2023-05-31T20:52:09.152 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:39:26 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f7e100864a52cac6-006477b379-1c17-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:52:09 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:52:09.196 DEBUG:boto:path=/pklxkd-59/obj11 2023-05-31T20:52:09.197 DEBUG:boto:auth_path=/pklxkd-59/obj11 2023-05-31T20:52:09.197 DEBUG:boto:Method: GET 2023-05-31T20:52:09.197 DEBUG:boto:Path: /pklxkd-59/obj11 2023-05-31T20:52:09.198 DEBUG:boto:Data: 2023-05-31T20:52:09.198 DEBUG:boto:Headers: {} 2023-05-31T20:52:09.198 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:52:09.199 DEBUG:boto:Port: 8000 2023-05-31T20:52:09.199 DEBUG:boto:Params: {} 2023-05-31T20:52:09.199 DEBUG:boto:Token: None 2023-05-31T20:52:09.200 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:52:09 GMT /pklxkd-59/obj11 2023-05-31T20:52:09.200 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:iZ8V6a3LionZQPRBhHC2lupqUcY= 2023-05-31T20:52:09.201 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:52:09 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:iZ8V6a3LionZQPRBhHC2lupqUcY=', 'Content-Length': '0'} 2023-05-31T20:52:09.203 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:39:26 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f685db2bc1dcadb9-006477b379-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:52:09 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:52:09.203 DEBUG:boto:path=/pklxkd-59/ 2023-05-31T20:52:09.204 DEBUG:boto:auth_path=/pklxkd-59/ 2023-05-31T20:52:09.204 DEBUG:boto:path=/pklxkd-59/?prefix=obj11 2023-05-31T20:52:09.204 DEBUG:boto:auth_path=/pklxkd-59/?prefix=obj11 2023-05-31T20:52:09.205 DEBUG:boto:Method: GET 2023-05-31T20:52:09.205 DEBUG:boto:Path: /pklxkd-59/?prefix=obj11 2023-05-31T20:52:09.205 DEBUG:boto:Data: 2023-05-31T20:52:09.206 DEBUG:boto:Headers: {} 2023-05-31T20:52:09.206 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:52:09.206 DEBUG:boto:Port: 8001 2023-05-31T20:52:09.207 DEBUG:boto:Params: {} 2023-05-31T20:52:09.207 DEBUG:boto:Token: None 2023-05-31T20:52:09.208 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:52:09 GMT /pklxkd-59/ 2023-05-31T20:52:09.208 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:OXHTYn90MKDcz4kG+dbepWeHVJ4= 2023-05-31T20:52:09.208 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:52:09 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:OXHTYn90MKDcz4kG+dbepWeHVJ4=', 'Content-Length': '0'} 2023-05-31T20:52:09.211 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b58501a1884ae490-006477b379-1c17-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:52:09 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:52:09.211 DEBUG:boto:b'pklxkd-59obj111000falseobj112023-05-31T20:39:26.427Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj1152023-05-31T20:46:33.958Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj1162023-05-31T20:48:13.052Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:52:09.212 DEBUG:boto:path=/pklxkd-59/ 2023-05-31T20:52:09.212 DEBUG:boto:auth_path=/pklxkd-59/ 2023-05-31T20:52:09.213 DEBUG:boto:path=/pklxkd-59/?prefix=obj11 2023-05-31T20:52:09.213 DEBUG:boto:auth_path=/pklxkd-59/?prefix=obj11 2023-05-31T20:52:09.213 DEBUG:boto:Method: GET 2023-05-31T20:52:09.214 DEBUG:boto:Path: /pklxkd-59/?prefix=obj11 2023-05-31T20:52:09.214 DEBUG:boto:Data: 2023-05-31T20:52:09.214 DEBUG:boto:Headers: {} 2023-05-31T20:52:09.215 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:52:09.215 DEBUG:boto:Port: 8000 2023-05-31T20:52:09.216 DEBUG:boto:Params: {} 2023-05-31T20:52:09.216 DEBUG:boto:Token: None 2023-05-31T20:52:09.216 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:52:09 GMT /pklxkd-59/ 2023-05-31T20:52:09.217 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:OXHTYn90MKDcz4kG+dbepWeHVJ4= 2023-05-31T20:52:09.217 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:52:09 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:OXHTYn90MKDcz4kG+dbepWeHVJ4=', 'Content-Length': '0'} 2023-05-31T20:52:09.221 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000006dc2301418bdda9b-006477b379-1630-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:52:09 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:52:09.221 DEBUG:boto:b'pklxkd-59obj111000falseobj112023-05-31T20:39:26.427Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj1152023-05-31T20:46:33.958Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj1162023-05-31T20:48:13.052Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:52:09.222 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj115 2023-05-31T20:52:09.222 DEBUG:boto:path=/pklxkd-59/obj115 2023-05-31T20:52:09.223 DEBUG:boto:auth_path=/pklxkd-59/obj115 2023-05-31T20:52:09.223 DEBUG:boto:path=/pklxkd-59/obj115?versionId=null 2023-05-31T20:52:09.223 DEBUG:boto:auth_path=/pklxkd-59/obj115?versionId=null 2023-05-31T20:52:09.224 DEBUG:boto:Method: GET 2023-05-31T20:52:09.224 DEBUG:boto:Path: /pklxkd-59/obj115?versionId=null 2023-05-31T20:52:09.224 DEBUG:boto:Data: 2023-05-31T20:52:09.225 DEBUG:boto:Headers: {} 2023-05-31T20:52:09.225 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:52:09.226 DEBUG:boto:Port: 8001 2023-05-31T20:52:09.226 DEBUG:boto:Params: {} 2023-05-31T20:52:09.226 DEBUG:boto:Token: None 2023-05-31T20:52:09.227 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:52:09 GMT /pklxkd-59/obj115?versionId=null 2023-05-31T20:52:09.227 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:E+2W9sV9lyF9n/ZBC/qdIWoY2Yk= 2023-05-31T20:52:09.228 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:52:09 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:E+2W9sV9lyF9n/ZBC/qdIWoY2Yk=', 'Content-Length': '0'} 2023-05-31T20:52:09.229 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:46:33 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000624fb356b3d0c796-006477b379-1c17-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:52:09 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:52:09.230 DEBUG:boto:path=/pklxkd-59/obj115 2023-05-31T20:52:09.230 DEBUG:boto:auth_path=/pklxkd-59/obj115 2023-05-31T20:52:09.231 DEBUG:boto:path=/pklxkd-59/obj115?versionId=null 2023-05-31T20:52:09.231 DEBUG:boto:auth_path=/pklxkd-59/obj115?versionId=null 2023-05-31T20:52:09.231 DEBUG:boto:Method: GET 2023-05-31T20:52:09.232 DEBUG:boto:Path: /pklxkd-59/obj115?versionId=null 2023-05-31T20:52:09.232 DEBUG:boto:Data: 2023-05-31T20:52:09.233 DEBUG:boto:Headers: {} 2023-05-31T20:52:09.233 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:52:09.233 DEBUG:boto:Port: 8000 2023-05-31T20:52:09.234 DEBUG:boto:Params: {} 2023-05-31T20:52:09.234 DEBUG:boto:Token: None 2023-05-31T20:52:09.234 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:52:09 GMT /pklxkd-59/obj115?versionId=null 2023-05-31T20:52:09.235 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:E+2W9sV9lyF9n/ZBC/qdIWoY2Yk= 2023-05-31T20:52:09.235 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:52:09 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:E+2W9sV9lyF9n/ZBC/qdIWoY2Yk=', 'Content-Length': '0'} 2023-05-31T20:52:09.237 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:46:33 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f4b1afc4a852296d-006477b379-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:52:09 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:52:09.238 DEBUG:boto:path=/pklxkd-59/obj115 2023-05-31T20:52:09.238 DEBUG:boto:auth_path=/pklxkd-59/obj115 2023-05-31T20:52:09.238 DEBUG:boto:path=/pklxkd-59/obj115?versionId=null 2023-05-31T20:52:09.239 DEBUG:boto:auth_path=/pklxkd-59/obj115?versionId=null 2023-05-31T20:52:09.239 DEBUG:boto:Method: HEAD 2023-05-31T20:52:09.239 DEBUG:boto:Path: /pklxkd-59/obj115?versionId=null 2023-05-31T20:52:09.240 DEBUG:boto:Data: 2023-05-31T20:52:09.240 DEBUG:boto:Headers: {} 2023-05-31T20:52:09.240 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:52:09.241 DEBUG:boto:Port: 8001 2023-05-31T20:52:09.241 DEBUG:boto:Params: {} 2023-05-31T20:52:09.241 DEBUG:boto:Token: None 2023-05-31T20:52:09.242 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:52:09 GMT /pklxkd-59/obj115?versionId=null 2023-05-31T20:52:09.242 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:JGPetf2mv65nSFJClnCPhvVPjV8= 2023-05-31T20:52:09.243 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:52:09 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:JGPetf2mv65nSFJClnCPhvVPjV8=', 'Content-Length': '0'} 2023-05-31T20:52:09.244 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:46:33 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000a4dba78c0e3dc8c-006477b379-1c17-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:52:09 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:52:09.245 DEBUG:boto:path=/pklxkd-59/obj115 2023-05-31T20:52:09.245 DEBUG:boto:auth_path=/pklxkd-59/obj115 2023-05-31T20:52:09.245 DEBUG:boto:path=/pklxkd-59/obj115?versionId=null 2023-05-31T20:52:09.246 DEBUG:boto:auth_path=/pklxkd-59/obj115?versionId=null 2023-05-31T20:52:09.246 DEBUG:boto:Method: HEAD 2023-05-31T20:52:09.246 DEBUG:boto:Path: /pklxkd-59/obj115?versionId=null 2023-05-31T20:52:09.247 DEBUG:boto:Data: 2023-05-31T20:52:09.247 DEBUG:boto:Headers: {} 2023-05-31T20:52:09.247 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:52:09.248 DEBUG:boto:Port: 8000 2023-05-31T20:52:09.248 DEBUG:boto:Params: {} 2023-05-31T20:52:09.248 DEBUG:boto:Token: None 2023-05-31T20:52:09.249 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:52:09 GMT /pklxkd-59/obj115?versionId=null 2023-05-31T20:52:09.249 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:JGPetf2mv65nSFJClnCPhvVPjV8= 2023-05-31T20:52:09.250 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:52:09 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:JGPetf2mv65nSFJClnCPhvVPjV8=', 'Content-Length': '0'} 2023-05-31T20:52:09.251 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:46:33 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000eb6bf16749bf4063-006477b379-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:52:09 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:52:09.252 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj115 2023-05-31T20:52:09.252 DEBUG:boto:path=/pklxkd-59/obj115 2023-05-31T20:52:09.253 DEBUG:boto:auth_path=/pklxkd-59/obj115 2023-05-31T20:52:09.253 DEBUG:boto:path=/pklxkd-59/obj115?versionId=null 2023-05-31T20:52:09.253 DEBUG:boto:auth_path=/pklxkd-59/obj115?versionId=null 2023-05-31T20:52:09.254 DEBUG:boto:Method: GET 2023-05-31T20:52:09.254 DEBUG:boto:Path: /pklxkd-59/obj115?versionId=null 2023-05-31T20:52:09.254 DEBUG:boto:Data: 2023-05-31T20:52:09.255 DEBUG:boto:Headers: {} 2023-05-31T20:52:09.255 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:52:09.256 DEBUG:boto:Port: 8001 2023-05-31T20:52:09.256 DEBUG:boto:Params: {} 2023-05-31T20:52:09.256 DEBUG:boto:Token: None 2023-05-31T20:52:09.257 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:52:09 GMT /pklxkd-59/obj115?versionId=null 2023-05-31T20:52:09.257 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:E+2W9sV9lyF9n/ZBC/qdIWoY2Yk= 2023-05-31T20:52:09.258 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:52:09 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:E+2W9sV9lyF9n/ZBC/qdIWoY2Yk=', 'Content-Length': '0'} 2023-05-31T20:52:09.259 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:46:33 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000554a4d032ce46b98-006477b379-1c17-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:52:09 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:52:09.260 DEBUG:boto:path=/pklxkd-59/obj115 2023-05-31T20:52:09.260 DEBUG:boto:auth_path=/pklxkd-59/obj115 2023-05-31T20:52:09.261 DEBUG:boto:path=/pklxkd-59/obj115?versionId=null 2023-05-31T20:52:09.261 DEBUG:boto:auth_path=/pklxkd-59/obj115?versionId=null 2023-05-31T20:52:09.262 DEBUG:boto:Method: GET 2023-05-31T20:52:09.262 DEBUG:boto:Path: /pklxkd-59/obj115?versionId=null 2023-05-31T20:52:09.262 DEBUG:boto:Data: 2023-05-31T20:52:09.263 DEBUG:boto:Headers: {} 2023-05-31T20:52:09.263 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:52:09.263 DEBUG:boto:Port: 8000 2023-05-31T20:52:09.264 DEBUG:boto:Params: {} 2023-05-31T20:52:09.264 DEBUG:boto:Token: None 2023-05-31T20:52:09.264 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:52:09 GMT /pklxkd-59/obj115?versionId=null 2023-05-31T20:52:09.265 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:E+2W9sV9lyF9n/ZBC/qdIWoY2Yk= 2023-05-31T20:52:09.265 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:52:09 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:E+2W9sV9lyF9n/ZBC/qdIWoY2Yk=', 'Content-Length': '0'} 2023-05-31T20:52:09.267 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:46:33 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000035c0f052c4c0ab6d-006477b379-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:52:09 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:52:09.308 DEBUG:boto:path=/pklxkd-59/ 2023-05-31T20:52:09.309 DEBUG:boto:auth_path=/pklxkd-59/ 2023-05-31T20:52:09.309 DEBUG:boto:path=/pklxkd-59/?prefix=obj115 2023-05-31T20:52:09.310 DEBUG:boto:auth_path=/pklxkd-59/?prefix=obj115 2023-05-31T20:52:09.310 DEBUG:boto:Method: GET 2023-05-31T20:52:09.310 DEBUG:boto:Path: /pklxkd-59/?prefix=obj115 2023-05-31T20:52:09.311 DEBUG:boto:Data: 2023-05-31T20:52:09.311 DEBUG:boto:Headers: {} 2023-05-31T20:52:09.311 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:52:09.312 DEBUG:boto:Port: 8001 2023-05-31T20:52:09.312 DEBUG:boto:Params: {} 2023-05-31T20:52:09.313 DEBUG:boto:Token: None 2023-05-31T20:52:09.313 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:52:09 GMT /pklxkd-59/ 2023-05-31T20:52:09.313 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:OXHTYn90MKDcz4kG+dbepWeHVJ4= 2023-05-31T20:52:09.314 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:52:09 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:OXHTYn90MKDcz4kG+dbepWeHVJ4=', 'Content-Length': '0'} 2023-05-31T20:52:09.316 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000079f9bc460d89c1f-006477b379-1c17-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:52:09 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:52:09.317 DEBUG:boto:b'pklxkd-59obj1151000falseobj1152023-05-31T20:46:33.958Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:52:09.318 DEBUG:boto:path=/pklxkd-59/ 2023-05-31T20:52:09.318 DEBUG:boto:auth_path=/pklxkd-59/ 2023-05-31T20:52:09.318 DEBUG:boto:path=/pklxkd-59/?prefix=obj115 2023-05-31T20:52:09.319 DEBUG:boto:auth_path=/pklxkd-59/?prefix=obj115 2023-05-31T20:52:09.319 DEBUG:boto:Method: GET 2023-05-31T20:52:09.319 DEBUG:boto:Path: /pklxkd-59/?prefix=obj115 2023-05-31T20:52:09.320 DEBUG:boto:Data: 2023-05-31T20:52:09.320 DEBUG:boto:Headers: {} 2023-05-31T20:52:09.321 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:52:09.321 DEBUG:boto:Port: 8000 2023-05-31T20:52:09.321 DEBUG:boto:Params: {} 2023-05-31T20:52:09.322 DEBUG:boto:Token: None 2023-05-31T20:52:09.322 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:52:09 GMT /pklxkd-59/ 2023-05-31T20:52:09.322 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:OXHTYn90MKDcz4kG+dbepWeHVJ4= 2023-05-31T20:52:09.323 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:52:09 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:OXHTYn90MKDcz4kG+dbepWeHVJ4=', 'Content-Length': '0'} 2023-05-31T20:52:09.327 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000513d1a05bffbf1a9-006477b379-1630-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:52:09 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:52:09.327 DEBUG:boto:b'pklxkd-59obj1151000falseobj1152023-05-31T20:46:33.958Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:52:09.328 DEBUG:boto:path=/pklxkd-59/obj115 2023-05-31T20:52:09.328 DEBUG:boto:auth_path=/pklxkd-59/obj115 2023-05-31T20:52:09.329 DEBUG:boto:Method: HEAD 2023-05-31T20:52:09.329 DEBUG:boto:Path: /pklxkd-59/obj115 2023-05-31T20:52:09.330 DEBUG:boto:Data: 2023-05-31T20:52:09.330 DEBUG:boto:Headers: {} 2023-05-31T20:52:09.330 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:52:09.331 DEBUG:boto:Port: 8001 2023-05-31T20:52:09.331 DEBUG:boto:Params: {} 2023-05-31T20:52:09.331 DEBUG:boto:Token: None 2023-05-31T20:52:09.332 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:52:09 GMT /pklxkd-59/obj115 2023-05-31T20:52:09.332 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:b8FzwDgJlFgeyNbh4oETuYYIDV0= 2023-05-31T20:52:09.333 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:52:09 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:b8FzwDgJlFgeyNbh4oETuYYIDV0=', 'Content-Length': '0'} 2023-05-31T20:52:09.334 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:46:33 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008fd2f9c988d0afd0-006477b379-1c17-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:52:09 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:52:09.335 DEBUG:boto:path=/pklxkd-59/obj115 2023-05-31T20:52:09.335 DEBUG:boto:auth_path=/pklxkd-59/obj115 2023-05-31T20:52:09.336 DEBUG:boto:Method: HEAD 2023-05-31T20:52:09.336 DEBUG:boto:Path: /pklxkd-59/obj115 2023-05-31T20:52:09.336 DEBUG:boto:Data: 2023-05-31T20:52:09.337 DEBUG:boto:Headers: {} 2023-05-31T20:52:09.337 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:52:09.337 DEBUG:boto:Port: 8000 2023-05-31T20:52:09.338 DEBUG:boto:Params: {} 2023-05-31T20:52:09.338 DEBUG:boto:Token: None 2023-05-31T20:52:09.338 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:52:09 GMT /pklxkd-59/obj115 2023-05-31T20:52:09.339 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:b8FzwDgJlFgeyNbh4oETuYYIDV0= 2023-05-31T20:52:09.339 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:52:09 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:b8FzwDgJlFgeyNbh4oETuYYIDV0=', 'Content-Length': '0'} 2023-05-31T20:52:09.341 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:46:33 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008e41b977c0ea2080-006477b379-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:52:09 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:52:09.341 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj115 2023-05-31T20:52:09.342 DEBUG:boto:path=/pklxkd-59/obj115 2023-05-31T20:52:09.342 DEBUG:boto:auth_path=/pklxkd-59/obj115 2023-05-31T20:52:09.342 DEBUG:boto:Method: GET 2023-05-31T20:52:09.343 DEBUG:boto:Path: /pklxkd-59/obj115 2023-05-31T20:52:09.343 DEBUG:boto:Data: 2023-05-31T20:52:09.344 DEBUG:boto:Headers: {} 2023-05-31T20:52:09.344 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:52:09.344 DEBUG:boto:Port: 8001 2023-05-31T20:52:09.345 DEBUG:boto:Params: {} 2023-05-31T20:52:09.345 DEBUG:boto:Token: None 2023-05-31T20:52:09.345 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:52:09 GMT /pklxkd-59/obj115 2023-05-31T20:52:09.346 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:COteMpWeRIye7rR7KonpT684zAk= 2023-05-31T20:52:09.346 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:52:09 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:COteMpWeRIye7rR7KonpT684zAk=', 'Content-Length': '0'} 2023-05-31T20:52:09.348 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:46:33 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000bf28a88b90032dbd-006477b379-1c17-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:52:09 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:52:09.388 DEBUG:boto:path=/pklxkd-59/obj115 2023-05-31T20:52:09.389 DEBUG:boto:auth_path=/pklxkd-59/obj115 2023-05-31T20:52:09.389 DEBUG:boto:Method: GET 2023-05-31T20:52:09.389 DEBUG:boto:Path: /pklxkd-59/obj115 2023-05-31T20:52:09.390 DEBUG:boto:Data: 2023-05-31T20:52:09.390 DEBUG:boto:Headers: {} 2023-05-31T20:52:09.390 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:52:09.391 DEBUG:boto:Port: 8000 2023-05-31T20:52:09.391 DEBUG:boto:Params: {} 2023-05-31T20:52:09.392 DEBUG:boto:Token: None 2023-05-31T20:52:09.392 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:52:09 GMT /pklxkd-59/obj115 2023-05-31T20:52:09.392 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:COteMpWeRIye7rR7KonpT684zAk= 2023-05-31T20:52:09.393 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:52:09 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:COteMpWeRIye7rR7KonpT684zAk=', 'Content-Length': '0'} 2023-05-31T20:52:09.395 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:46:33 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000087ca9663eb873d6-006477b379-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:52:09 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:52:09.395 DEBUG:boto:path=/pklxkd-59/ 2023-05-31T20:52:09.396 DEBUG:boto:auth_path=/pklxkd-59/ 2023-05-31T20:52:09.396 DEBUG:boto:path=/pklxkd-59/?prefix=obj115 2023-05-31T20:52:09.397 DEBUG:boto:auth_path=/pklxkd-59/?prefix=obj115 2023-05-31T20:52:09.397 DEBUG:boto:Method: GET 2023-05-31T20:52:09.397 DEBUG:boto:Path: /pklxkd-59/?prefix=obj115 2023-05-31T20:52:09.398 DEBUG:boto:Data: 2023-05-31T20:52:09.398 DEBUG:boto:Headers: {} 2023-05-31T20:52:09.398 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:52:09.399 DEBUG:boto:Port: 8001 2023-05-31T20:52:09.399 DEBUG:boto:Params: {} 2023-05-31T20:52:09.399 DEBUG:boto:Token: None 2023-05-31T20:52:09.400 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:52:09 GMT /pklxkd-59/ 2023-05-31T20:52:09.400 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:OXHTYn90MKDcz4kG+dbepWeHVJ4= 2023-05-31T20:52:09.401 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:52:09 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:OXHTYn90MKDcz4kG+dbepWeHVJ4=', 'Content-Length': '0'} 2023-05-31T20:52:09.403 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b8826b5112a042dc-006477b379-1c17-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:52:09 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:52:09.404 DEBUG:boto:b'pklxkd-59obj1151000falseobj1152023-05-31T20:46:33.958Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:52:09.404 DEBUG:boto:path=/pklxkd-59/ 2023-05-31T20:52:09.405 DEBUG:boto:auth_path=/pklxkd-59/ 2023-05-31T20:52:09.405 DEBUG:boto:path=/pklxkd-59/?prefix=obj115 2023-05-31T20:52:09.405 DEBUG:boto:auth_path=/pklxkd-59/?prefix=obj115 2023-05-31T20:52:09.406 DEBUG:boto:Method: GET 2023-05-31T20:52:09.406 DEBUG:boto:Path: /pklxkd-59/?prefix=obj115 2023-05-31T20:52:09.407 DEBUG:boto:Data: 2023-05-31T20:52:09.407 DEBUG:boto:Headers: {} 2023-05-31T20:52:09.407 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:52:09.408 DEBUG:boto:Port: 8000 2023-05-31T20:52:09.408 DEBUG:boto:Params: {} 2023-05-31T20:52:09.408 DEBUG:boto:Token: None 2023-05-31T20:52:09.409 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:52:09 GMT /pklxkd-59/ 2023-05-31T20:52:09.409 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:OXHTYn90MKDcz4kG+dbepWeHVJ4= 2023-05-31T20:52:09.410 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:52:09 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:OXHTYn90MKDcz4kG+dbepWeHVJ4=', 'Content-Length': '0'} 2023-05-31T20:52:09.413 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000020fe25ac96bece80-006477b379-1630-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:52:09 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:52:09.414 DEBUG:boto:b'pklxkd-59obj1151000falseobj1152023-05-31T20:46:33.958Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:52:09.414 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj116 2023-05-31T20:52:09.415 DEBUG:boto:path=/pklxkd-59/obj116 2023-05-31T20:52:09.415 DEBUG:boto:auth_path=/pklxkd-59/obj116 2023-05-31T20:52:09.416 DEBUG:boto:path=/pklxkd-59/obj116?versionId=null 2023-05-31T20:52:09.416 DEBUG:boto:auth_path=/pklxkd-59/obj116?versionId=null 2023-05-31T20:52:09.416 DEBUG:boto:Method: GET 2023-05-31T20:52:09.417 DEBUG:boto:Path: /pklxkd-59/obj116?versionId=null 2023-05-31T20:52:09.417 DEBUG:boto:Data: 2023-05-31T20:52:09.417 DEBUG:boto:Headers: {} 2023-05-31T20:52:09.418 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:52:09.418 DEBUG:boto:Port: 8001 2023-05-31T20:52:09.419 DEBUG:boto:Params: {} 2023-05-31T20:52:09.419 DEBUG:boto:Token: None 2023-05-31T20:52:09.419 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:52:09 GMT /pklxkd-59/obj116?versionId=null 2023-05-31T20:52:09.420 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:2v8dixo4sD3v5De5ngTNDa5Y7RI= 2023-05-31T20:52:09.420 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:52:09 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:2v8dixo4sD3v5De5ngTNDa5Y7RI=', 'Content-Length': '0'} 2023-05-31T20:52:09.422 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:48:13 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a736c4ec5c9c68ec-006477b379-1c17-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:52:09 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:52:09.423 DEBUG:boto:path=/pklxkd-59/obj116 2023-05-31T20:52:09.423 DEBUG:boto:auth_path=/pklxkd-59/obj116 2023-05-31T20:52:09.423 DEBUG:boto:path=/pklxkd-59/obj116?versionId=null 2023-05-31T20:52:09.424 DEBUG:boto:auth_path=/pklxkd-59/obj116?versionId=null 2023-05-31T20:52:09.424 DEBUG:boto:Method: GET 2023-05-31T20:52:09.425 DEBUG:boto:Path: /pklxkd-59/obj116?versionId=null 2023-05-31T20:52:09.425 DEBUG:boto:Data: 2023-05-31T20:52:09.425 DEBUG:boto:Headers: {} 2023-05-31T20:52:09.426 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:52:09.426 DEBUG:boto:Port: 8000 2023-05-31T20:52:09.426 DEBUG:boto:Params: {} 2023-05-31T20:52:09.427 DEBUG:boto:Token: None 2023-05-31T20:52:09.427 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:52:09 GMT /pklxkd-59/obj116?versionId=null 2023-05-31T20:52:09.428 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:2v8dixo4sD3v5De5ngTNDa5Y7RI= 2023-05-31T20:52:09.428 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:52:09 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:2v8dixo4sD3v5De5ngTNDa5Y7RI=', 'Content-Length': '0'} 2023-05-31T20:52:09.430 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:48:13 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000200b8644290a3fd2-006477b379-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:52:09 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:52:09.472 DEBUG:boto:path=/pklxkd-59/obj116 2023-05-31T20:52:09.473 DEBUG:boto:auth_path=/pklxkd-59/obj116 2023-05-31T20:52:09.473 DEBUG:boto:path=/pklxkd-59/obj116?versionId=null 2023-05-31T20:52:09.473 DEBUG:boto:auth_path=/pklxkd-59/obj116?versionId=null 2023-05-31T20:52:09.474 DEBUG:boto:Method: HEAD 2023-05-31T20:52:09.474 DEBUG:boto:Path: /pklxkd-59/obj116?versionId=null 2023-05-31T20:52:09.474 DEBUG:boto:Data: 2023-05-31T20:52:09.475 DEBUG:boto:Headers: {} 2023-05-31T20:52:09.475 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:52:09.475 DEBUG:boto:Port: 8001 2023-05-31T20:52:09.476 DEBUG:boto:Params: {} 2023-05-31T20:52:09.476 DEBUG:boto:Token: None 2023-05-31T20:52:09.476 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:52:09 GMT /pklxkd-59/obj116?versionId=null 2023-05-31T20:52:09.477 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:UxXoNZHlaC0s0202uVlWF5Wf1X8= 2023-05-31T20:52:09.477 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:52:09 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:UxXoNZHlaC0s0202uVlWF5Wf1X8=', 'Content-Length': '0'} 2023-05-31T20:52:09.479 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:48:13 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ee8c64f14c408922-006477b379-1c17-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:52:09 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:52:09.479 DEBUG:boto:path=/pklxkd-59/obj116 2023-05-31T20:52:09.480 DEBUG:boto:auth_path=/pklxkd-59/obj116 2023-05-31T20:52:09.480 DEBUG:boto:path=/pklxkd-59/obj116?versionId=null 2023-05-31T20:52:09.480 DEBUG:boto:auth_path=/pklxkd-59/obj116?versionId=null 2023-05-31T20:52:09.481 DEBUG:boto:Method: HEAD 2023-05-31T20:52:09.481 DEBUG:boto:Path: /pklxkd-59/obj116?versionId=null 2023-05-31T20:52:09.481 DEBUG:boto:Data: 2023-05-31T20:52:09.482 DEBUG:boto:Headers: {} 2023-05-31T20:52:09.482 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:52:09.482 DEBUG:boto:Port: 8000 2023-05-31T20:52:09.483 DEBUG:boto:Params: {} 2023-05-31T20:52:09.483 DEBUG:boto:Token: None 2023-05-31T20:52:09.484 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:52:09 GMT /pklxkd-59/obj116?versionId=null 2023-05-31T20:52:09.484 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:UxXoNZHlaC0s0202uVlWF5Wf1X8= 2023-05-31T20:52:09.484 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:52:09 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:UxXoNZHlaC0s0202uVlWF5Wf1X8=', 'Content-Length': '0'} 2023-05-31T20:52:09.486 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:48:13 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f708beee623f0bf8-006477b379-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:52:09 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:52:09.487 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj116 2023-05-31T20:52:09.487 DEBUG:boto:path=/pklxkd-59/obj116 2023-05-31T20:52:09.487 DEBUG:boto:auth_path=/pklxkd-59/obj116 2023-05-31T20:52:09.488 DEBUG:boto:path=/pklxkd-59/obj116?versionId=null 2023-05-31T20:52:09.488 DEBUG:boto:auth_path=/pklxkd-59/obj116?versionId=null 2023-05-31T20:52:09.489 DEBUG:boto:Method: GET 2023-05-31T20:52:09.489 DEBUG:boto:Path: /pklxkd-59/obj116?versionId=null 2023-05-31T20:52:09.489 DEBUG:boto:Data: 2023-05-31T20:52:09.490 DEBUG:boto:Headers: {} 2023-05-31T20:52:09.490 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:52:09.490 DEBUG:boto:Port: 8001 2023-05-31T20:52:09.491 DEBUG:boto:Params: {} 2023-05-31T20:52:09.491 DEBUG:boto:Token: None 2023-05-31T20:52:09.491 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:52:09 GMT /pklxkd-59/obj116?versionId=null 2023-05-31T20:52:09.492 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:2v8dixo4sD3v5De5ngTNDa5Y7RI= 2023-05-31T20:52:09.492 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:52:09 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:2v8dixo4sD3v5De5ngTNDa5Y7RI=', 'Content-Length': '0'} 2023-05-31T20:52:09.494 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:48:13 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000bd6e39182916da14-006477b379-1c17-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:52:09 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:52:09.495 DEBUG:boto:path=/pklxkd-59/obj116 2023-05-31T20:52:09.495 DEBUG:boto:auth_path=/pklxkd-59/obj116 2023-05-31T20:52:09.496 DEBUG:boto:path=/pklxkd-59/obj116?versionId=null 2023-05-31T20:52:09.496 DEBUG:boto:auth_path=/pklxkd-59/obj116?versionId=null 2023-05-31T20:52:09.496 DEBUG:boto:Method: GET 2023-05-31T20:52:09.497 DEBUG:boto:Path: /pklxkd-59/obj116?versionId=null 2023-05-31T20:52:09.497 DEBUG:boto:Data: 2023-05-31T20:52:09.497 DEBUG:boto:Headers: {} 2023-05-31T20:52:09.498 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:52:09.498 DEBUG:boto:Port: 8000 2023-05-31T20:52:09.498 DEBUG:boto:Params: {} 2023-05-31T20:52:09.499 DEBUG:boto:Token: None 2023-05-31T20:52:09.499 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:52:09 GMT /pklxkd-59/obj116?versionId=null 2023-05-31T20:52:09.500 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:2v8dixo4sD3v5De5ngTNDa5Y7RI= 2023-05-31T20:52:09.500 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:52:09 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:2v8dixo4sD3v5De5ngTNDa5Y7RI=', 'Content-Length': '0'} 2023-05-31T20:52:09.502 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:48:13 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a3beb458eb6c203a-006477b379-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:52:09 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:52:09.503 DEBUG:boto:path=/pklxkd-59/ 2023-05-31T20:52:09.503 DEBUG:boto:auth_path=/pklxkd-59/ 2023-05-31T20:52:09.503 DEBUG:boto:path=/pklxkd-59/?prefix=obj116 2023-05-31T20:52:09.504 DEBUG:boto:auth_path=/pklxkd-59/?prefix=obj116 2023-05-31T20:52:09.504 DEBUG:boto:Method: GET 2023-05-31T20:52:09.505 DEBUG:boto:Path: /pklxkd-59/?prefix=obj116 2023-05-31T20:52:09.505 DEBUG:boto:Data: 2023-05-31T20:52:09.505 DEBUG:boto:Headers: {} 2023-05-31T20:52:09.506 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:52:09.506 DEBUG:boto:Port: 8001 2023-05-31T20:52:09.506 DEBUG:boto:Params: {} 2023-05-31T20:52:09.507 DEBUG:boto:Token: None 2023-05-31T20:52:09.507 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:52:09 GMT /pklxkd-59/ 2023-05-31T20:52:09.508 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:OXHTYn90MKDcz4kG+dbepWeHVJ4= 2023-05-31T20:52:09.508 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:52:09 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:OXHTYn90MKDcz4kG+dbepWeHVJ4=', 'Content-Length': '0'} 2023-05-31T20:52:09.510 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000005f8f13745651a5ca-006477b379-1c17-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:52:09 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:52:09.511 DEBUG:boto:b'pklxkd-59obj1161000falseobj1162023-05-31T20:48:13.052Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:52:09.511 DEBUG:boto:path=/pklxkd-59/ 2023-05-31T20:52:09.512 DEBUG:boto:auth_path=/pklxkd-59/ 2023-05-31T20:52:09.512 DEBUG:boto:path=/pklxkd-59/?prefix=obj116 2023-05-31T20:52:09.513 DEBUG:boto:auth_path=/pklxkd-59/?prefix=obj116 2023-05-31T20:52:09.513 DEBUG:boto:Method: GET 2023-05-31T20:52:09.513 DEBUG:boto:Path: /pklxkd-59/?prefix=obj116 2023-05-31T20:52:09.514 DEBUG:boto:Data: 2023-05-31T20:52:09.514 DEBUG:boto:Headers: {} 2023-05-31T20:52:09.514 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:52:09.515 DEBUG:boto:Port: 8000 2023-05-31T20:52:09.515 DEBUG:boto:Params: {} 2023-05-31T20:52:09.515 DEBUG:boto:Token: None 2023-05-31T20:52:09.516 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:52:09 GMT /pklxkd-59/ 2023-05-31T20:52:09.516 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:OXHTYn90MKDcz4kG+dbepWeHVJ4= 2023-05-31T20:52:09.517 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:52:09 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:OXHTYn90MKDcz4kG+dbepWeHVJ4=', 'Content-Length': '0'} 2023-05-31T20:52:09.520 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000005b2bb54538665ab4-006477b379-1630-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:52:09 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:52:09.521 DEBUG:boto:b'pklxkd-59obj1161000falseobj1162023-05-31T20:48:13.052Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:52:09.521 DEBUG:boto:path=/pklxkd-59/obj116 2023-05-31T20:52:09.522 DEBUG:boto:auth_path=/pklxkd-59/obj116 2023-05-31T20:52:09.522 DEBUG:boto:Method: HEAD 2023-05-31T20:52:09.523 DEBUG:boto:Path: /pklxkd-59/obj116 2023-05-31T20:52:09.523 DEBUG:boto:Data: 2023-05-31T20:52:09.523 DEBUG:boto:Headers: {} 2023-05-31T20:52:09.524 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:52:09.524 DEBUG:boto:Port: 8001 2023-05-31T20:52:09.524 DEBUG:boto:Params: {} 2023-05-31T20:52:09.525 DEBUG:boto:Token: None 2023-05-31T20:52:09.525 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:52:09 GMT /pklxkd-59/obj116 2023-05-31T20:52:09.525 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:502HyURp4IhgoJ5RhWaPmjFK34U= 2023-05-31T20:52:09.526 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:52:09 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:502HyURp4IhgoJ5RhWaPmjFK34U=', 'Content-Length': '0'} 2023-05-31T20:52:09.527 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:48:13 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ab20d0a1c241afe7-006477b379-1c17-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:52:09 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:52:09.528 DEBUG:boto:path=/pklxkd-59/obj116 2023-05-31T20:52:09.528 DEBUG:boto:auth_path=/pklxkd-59/obj116 2023-05-31T20:52:09.529 DEBUG:boto:Method: HEAD 2023-05-31T20:52:09.529 DEBUG:boto:Path: /pklxkd-59/obj116 2023-05-31T20:52:09.529 DEBUG:boto:Data: 2023-05-31T20:52:09.530 DEBUG:boto:Headers: {} 2023-05-31T20:52:09.530 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:52:09.530 DEBUG:boto:Port: 8000 2023-05-31T20:52:09.531 DEBUG:boto:Params: {} 2023-05-31T20:52:09.531 DEBUG:boto:Token: None 2023-05-31T20:52:09.532 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:52:09 GMT /pklxkd-59/obj116 2023-05-31T20:52:09.532 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:502HyURp4IhgoJ5RhWaPmjFK34U= 2023-05-31T20:52:09.532 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:52:09 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:502HyURp4IhgoJ5RhWaPmjFK34U=', 'Content-Length': '0'} 2023-05-31T20:52:09.534 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:48:13 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002c5c725e213fd9f8-006477b379-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:52:09 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:52:09.535 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj116 2023-05-31T20:52:09.535 DEBUG:boto:path=/pklxkd-59/obj116 2023-05-31T20:52:09.535 DEBUG:boto:auth_path=/pklxkd-59/obj116 2023-05-31T20:52:09.536 DEBUG:boto:Method: GET 2023-05-31T20:52:09.536 DEBUG:boto:Path: /pklxkd-59/obj116 2023-05-31T20:52:09.536 DEBUG:boto:Data: 2023-05-31T20:52:09.537 DEBUG:boto:Headers: {} 2023-05-31T20:52:09.537 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:52:09.538 DEBUG:boto:Port: 8001 2023-05-31T20:52:09.538 DEBUG:boto:Params: {} 2023-05-31T20:52:09.538 DEBUG:boto:Token: None 2023-05-31T20:52:09.539 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:52:09 GMT /pklxkd-59/obj116 2023-05-31T20:52:09.539 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:aFE4V/OuaXJosltxiBI/Sldz4g8= 2023-05-31T20:52:09.539 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:52:09 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:aFE4V/OuaXJosltxiBI/Sldz4g8=', 'Content-Length': '0'} 2023-05-31T20:52:09.542 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:48:13 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000048c32a986737aad0-006477b379-1c17-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:52:09 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:52:09.542 DEBUG:boto:path=/pklxkd-59/obj116 2023-05-31T20:52:09.543 DEBUG:boto:auth_path=/pklxkd-59/obj116 2023-05-31T20:52:09.543 DEBUG:boto:Method: GET 2023-05-31T20:52:09.543 DEBUG:boto:Path: /pklxkd-59/obj116 2023-05-31T20:52:09.544 DEBUG:boto:Data: 2023-05-31T20:52:09.544 DEBUG:boto:Headers: {} 2023-05-31T20:52:09.544 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:52:09.545 DEBUG:boto:Port: 8000 2023-05-31T20:52:09.545 DEBUG:boto:Params: {} 2023-05-31T20:52:09.545 DEBUG:boto:Token: None 2023-05-31T20:52:09.546 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:52:09 GMT /pklxkd-59/obj116 2023-05-31T20:52:09.546 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:aFE4V/OuaXJosltxiBI/Sldz4g8= 2023-05-31T20:52:09.547 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:52:09 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:aFE4V/OuaXJosltxiBI/Sldz4g8=', 'Content-Length': '0'} 2023-05-31T20:52:09.548 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:48:13 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000adbbbd664a32f286-006477b379-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:52:09 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:52:09.549 DEBUG:boto:path=/pklxkd-59/ 2023-05-31T20:52:09.550 DEBUG:boto:auth_path=/pklxkd-59/ 2023-05-31T20:52:09.550 DEBUG:boto:path=/pklxkd-59/?prefix=obj116 2023-05-31T20:52:09.550 DEBUG:boto:auth_path=/pklxkd-59/?prefix=obj116 2023-05-31T20:52:09.551 DEBUG:boto:Method: GET 2023-05-31T20:52:09.551 DEBUG:boto:Path: /pklxkd-59/?prefix=obj116 2023-05-31T20:52:09.551 DEBUG:boto:Data: 2023-05-31T20:52:09.552 DEBUG:boto:Headers: {} 2023-05-31T20:52:09.552 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:52:09.552 DEBUG:boto:Port: 8001 2023-05-31T20:52:09.553 DEBUG:boto:Params: {} 2023-05-31T20:52:09.553 DEBUG:boto:Token: None 2023-05-31T20:52:09.553 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:52:09 GMT /pklxkd-59/ 2023-05-31T20:52:09.554 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:OXHTYn90MKDcz4kG+dbepWeHVJ4= 2023-05-31T20:52:09.554 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:52:09 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:OXHTYn90MKDcz4kG+dbepWeHVJ4=', 'Content-Length': '0'} 2023-05-31T20:52:09.556 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000038e44c4dc50ab633-006477b379-1c17-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:52:09 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:52:09.557 DEBUG:boto:b'pklxkd-59obj1161000falseobj1162023-05-31T20:48:13.052Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:52:09.558 DEBUG:boto:path=/pklxkd-59/ 2023-05-31T20:52:09.558 DEBUG:boto:auth_path=/pklxkd-59/ 2023-05-31T20:52:09.558 DEBUG:boto:path=/pklxkd-59/?prefix=obj116 2023-05-31T20:52:09.559 DEBUG:boto:auth_path=/pklxkd-59/?prefix=obj116 2023-05-31T20:52:09.559 DEBUG:boto:Method: GET 2023-05-31T20:52:09.560 DEBUG:boto:Path: /pklxkd-59/?prefix=obj116 2023-05-31T20:52:09.560 DEBUG:boto:Data: 2023-05-31T20:52:09.560 DEBUG:boto:Headers: {} 2023-05-31T20:52:09.561 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:52:09.561 DEBUG:boto:Port: 8000 2023-05-31T20:52:09.561 DEBUG:boto:Params: {} 2023-05-31T20:52:09.562 DEBUG:boto:Token: None 2023-05-31T20:52:09.562 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:52:09 GMT /pklxkd-59/ 2023-05-31T20:52:09.562 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:OXHTYn90MKDcz4kG+dbepWeHVJ4= 2023-05-31T20:52:09.563 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:52:09 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:OXHTYn90MKDcz4kG+dbepWeHVJ4=', 'Content-Length': '0'} 2023-05-31T20:52:09.566 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000001c4cec3b8b21279-006477b379-1630-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:52:09 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:52:09.608 DEBUG:boto:b'pklxkd-59obj1161000falseobj1162023-05-31T20:48:13.052Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:52:09.609 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj124 2023-05-31T20:52:09.609 DEBUG:boto:path=/pklxkd-59/obj124 2023-05-31T20:52:09.609 DEBUG:boto:auth_path=/pklxkd-59/obj124 2023-05-31T20:52:09.610 DEBUG:boto:path=/pklxkd-59/obj124?versionId=null 2023-05-31T20:52:09.610 DEBUG:boto:auth_path=/pklxkd-59/obj124?versionId=null 2023-05-31T20:52:09.611 DEBUG:boto:Method: GET 2023-05-31T20:52:09.611 DEBUG:boto:Path: /pklxkd-59/obj124?versionId=null 2023-05-31T20:52:09.611 DEBUG:boto:Data: 2023-05-31T20:52:09.612 DEBUG:boto:Headers: {} 2023-05-31T20:52:09.612 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:52:09.613 DEBUG:boto:Port: 8001 2023-05-31T20:52:09.613 DEBUG:boto:Params: {} 2023-05-31T20:52:09.613 DEBUG:boto:Token: None 2023-05-31T20:52:09.614 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:52:09 GMT /pklxkd-59/obj124?versionId=null 2023-05-31T20:52:09.614 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:oXL1A7fc2Unjr4B0WyVHifHGDCk= 2023-05-31T20:52:09.615 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:52:09 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:oXL1A7fc2Unjr4B0WyVHifHGDCk=', 'Content-Length': '0'} 2023-05-31T20:52:09.617 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:50:33 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000064379285680c5a2-006477b379-1c17-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:52:09 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:52:09.617 DEBUG:boto:path=/pklxkd-59/obj124 2023-05-31T20:52:09.618 DEBUG:boto:auth_path=/pklxkd-59/obj124 2023-05-31T20:52:09.618 DEBUG:boto:path=/pklxkd-59/obj124?versionId=null 2023-05-31T20:52:09.618 DEBUG:boto:auth_path=/pklxkd-59/obj124?versionId=null 2023-05-31T20:52:09.619 DEBUG:boto:Method: GET 2023-05-31T20:52:09.619 DEBUG:boto:Path: /pklxkd-59/obj124?versionId=null 2023-05-31T20:52:09.620 DEBUG:boto:Data: 2023-05-31T20:52:09.620 DEBUG:boto:Headers: {} 2023-05-31T20:52:09.620 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:52:09.621 DEBUG:boto:Port: 8000 2023-05-31T20:52:09.621 DEBUG:boto:Params: {} 2023-05-31T20:52:09.621 DEBUG:boto:Token: None 2023-05-31T20:52:09.622 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:52:09 GMT /pklxkd-59/obj124?versionId=null 2023-05-31T20:52:09.622 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:oXL1A7fc2Unjr4B0WyVHifHGDCk= 2023-05-31T20:52:09.623 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:52:09 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:oXL1A7fc2Unjr4B0WyVHifHGDCk=', 'Content-Length': '0'} 2023-05-31T20:52:09.625 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:50:33 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000553f68109695e44e-006477b379-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:52:09 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:52:09.625 DEBUG:boto:path=/pklxkd-59/obj124 2023-05-31T20:52:09.626 DEBUG:boto:auth_path=/pklxkd-59/obj124 2023-05-31T20:52:09.626 DEBUG:boto:path=/pklxkd-59/obj124?versionId=null 2023-05-31T20:52:09.626 DEBUG:boto:auth_path=/pklxkd-59/obj124?versionId=null 2023-05-31T20:52:09.627 DEBUG:boto:Method: HEAD 2023-05-31T20:52:09.627 DEBUG:boto:Path: /pklxkd-59/obj124?versionId=null 2023-05-31T20:52:09.628 DEBUG:boto:Data: 2023-05-31T20:52:09.628 DEBUG:boto:Headers: {} 2023-05-31T20:52:09.628 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:52:09.629 DEBUG:boto:Port: 8001 2023-05-31T20:52:09.629 DEBUG:boto:Params: {} 2023-05-31T20:52:09.629 DEBUG:boto:Token: None 2023-05-31T20:52:09.630 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:52:09 GMT /pklxkd-59/obj124?versionId=null 2023-05-31T20:52:09.630 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:cLk5APh9jeX/LQ8vF/xvZwsHK9A= 2023-05-31T20:52:09.630 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:52:09 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:cLk5APh9jeX/LQ8vF/xvZwsHK9A=', 'Content-Length': '0'} 2023-05-31T20:52:09.632 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:50:33 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002d357765ed8e6727-006477b379-1c17-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:52:09 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:52:09.633 DEBUG:boto:path=/pklxkd-59/obj124 2023-05-31T20:52:09.633 DEBUG:boto:auth_path=/pklxkd-59/obj124 2023-05-31T20:52:09.633 DEBUG:boto:path=/pklxkd-59/obj124?versionId=null 2023-05-31T20:52:09.634 DEBUG:boto:auth_path=/pklxkd-59/obj124?versionId=null 2023-05-31T20:52:09.634 DEBUG:boto:Method: HEAD 2023-05-31T20:52:09.634 DEBUG:boto:Path: /pklxkd-59/obj124?versionId=null 2023-05-31T20:52:09.635 DEBUG:boto:Data: 2023-05-31T20:52:09.635 DEBUG:boto:Headers: {} 2023-05-31T20:52:09.635 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:52:09.636 DEBUG:boto:Port: 8000 2023-05-31T20:52:09.636 DEBUG:boto:Params: {} 2023-05-31T20:52:09.636 DEBUG:boto:Token: None 2023-05-31T20:52:09.637 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:52:09 GMT /pklxkd-59/obj124?versionId=null 2023-05-31T20:52:09.637 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:cLk5APh9jeX/LQ8vF/xvZwsHK9A= 2023-05-31T20:52:09.638 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:52:09 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:cLk5APh9jeX/LQ8vF/xvZwsHK9A=', 'Content-Length': '0'} 2023-05-31T20:52:09.639 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:50:33 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001d4bf6bf5739b18e-006477b379-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:52:09 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:52:09.640 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj124 2023-05-31T20:52:09.640 DEBUG:boto:path=/pklxkd-59/obj124 2023-05-31T20:52:09.641 DEBUG:boto:auth_path=/pklxkd-59/obj124 2023-05-31T20:52:09.641 DEBUG:boto:path=/pklxkd-59/obj124?versionId=null 2023-05-31T20:52:09.641 DEBUG:boto:auth_path=/pklxkd-59/obj124?versionId=null 2023-05-31T20:52:09.642 DEBUG:boto:Method: GET 2023-05-31T20:52:09.642 DEBUG:boto:Path: /pklxkd-59/obj124?versionId=null 2023-05-31T20:52:09.642 DEBUG:boto:Data: 2023-05-31T20:52:09.643 DEBUG:boto:Headers: {} 2023-05-31T20:52:09.643 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:52:09.643 DEBUG:boto:Port: 8001 2023-05-31T20:52:09.644 DEBUG:boto:Params: {} 2023-05-31T20:52:09.644 DEBUG:boto:Token: None 2023-05-31T20:52:09.645 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:52:09 GMT /pklxkd-59/obj124?versionId=null 2023-05-31T20:52:09.645 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:oXL1A7fc2Unjr4B0WyVHifHGDCk= 2023-05-31T20:52:09.645 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:52:09 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:oXL1A7fc2Unjr4B0WyVHifHGDCk=', 'Content-Length': '0'} 2023-05-31T20:52:09.647 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:50:33 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006dca47a8bf25a910-006477b379-1c17-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:52:09 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:52:09.688 DEBUG:boto:path=/pklxkd-59/obj124 2023-05-31T20:52:09.689 DEBUG:boto:auth_path=/pklxkd-59/obj124 2023-05-31T20:52:09.689 DEBUG:boto:path=/pklxkd-59/obj124?versionId=null 2023-05-31T20:52:09.689 DEBUG:boto:auth_path=/pklxkd-59/obj124?versionId=null 2023-05-31T20:52:09.690 DEBUG:boto:Method: GET 2023-05-31T20:52:09.690 DEBUG:boto:Path: /pklxkd-59/obj124?versionId=null 2023-05-31T20:52:09.690 DEBUG:boto:Data: 2023-05-31T20:52:09.691 DEBUG:boto:Headers: {} 2023-05-31T20:52:09.691 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:52:09.691 DEBUG:boto:Port: 8000 2023-05-31T20:52:09.692 DEBUG:boto:Params: {} 2023-05-31T20:52:09.692 DEBUG:boto:Token: None 2023-05-31T20:52:09.693 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:52:09 GMT /pklxkd-59/obj124?versionId=null 2023-05-31T20:52:09.693 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:oXL1A7fc2Unjr4B0WyVHifHGDCk= 2023-05-31T20:52:09.693 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:52:09 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:oXL1A7fc2Unjr4B0WyVHifHGDCk=', 'Content-Length': '0'} 2023-05-31T20:52:09.695 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:50:33 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d4b9524d52b2dd97-006477b379-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:52:09 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:52:09.696 DEBUG:boto:path=/pklxkd-59/ 2023-05-31T20:52:09.696 DEBUG:boto:auth_path=/pklxkd-59/ 2023-05-31T20:52:09.697 DEBUG:boto:path=/pklxkd-59/?prefix=obj124 2023-05-31T20:52:09.697 DEBUG:boto:auth_path=/pklxkd-59/?prefix=obj124 2023-05-31T20:52:09.698 DEBUG:boto:Method: GET 2023-05-31T20:52:09.698 DEBUG:boto:Path: /pklxkd-59/?prefix=obj124 2023-05-31T20:52:09.698 DEBUG:boto:Data: 2023-05-31T20:52:09.699 DEBUG:boto:Headers: {} 2023-05-31T20:52:09.699 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:52:09.699 DEBUG:boto:Port: 8001 2023-05-31T20:52:09.700 DEBUG:boto:Params: {} 2023-05-31T20:52:09.700 DEBUG:boto:Token: None 2023-05-31T20:52:09.700 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:52:09 GMT /pklxkd-59/ 2023-05-31T20:52:09.701 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:OXHTYn90MKDcz4kG+dbepWeHVJ4= 2023-05-31T20:52:09.701 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:52:09 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:OXHTYn90MKDcz4kG+dbepWeHVJ4=', 'Content-Length': '0'} 2023-05-31T20:52:09.704 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000009da6b1e65f3acd61-006477b379-1c17-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:52:09 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:52:09.704 DEBUG:boto:b'pklxkd-59obj1241000falseobj1242023-05-31T20:50:33.587Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:52:09.705 DEBUG:boto:path=/pklxkd-59/ 2023-05-31T20:52:09.705 DEBUG:boto:auth_path=/pklxkd-59/ 2023-05-31T20:52:09.705 DEBUG:boto:path=/pklxkd-59/?prefix=obj124 2023-05-31T20:52:09.706 DEBUG:boto:auth_path=/pklxkd-59/?prefix=obj124 2023-05-31T20:52:09.706 DEBUG:boto:Method: GET 2023-05-31T20:52:09.707 DEBUG:boto:Path: /pklxkd-59/?prefix=obj124 2023-05-31T20:52:09.707 DEBUG:boto:Data: 2023-05-31T20:52:09.707 DEBUG:boto:Headers: {} 2023-05-31T20:52:09.708 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:52:09.708 DEBUG:boto:Port: 8000 2023-05-31T20:52:09.708 DEBUG:boto:Params: {} 2023-05-31T20:52:09.709 DEBUG:boto:Token: None 2023-05-31T20:52:09.709 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:52:09 GMT /pklxkd-59/ 2023-05-31T20:52:09.709 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:OXHTYn90MKDcz4kG+dbepWeHVJ4= 2023-05-31T20:52:09.710 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:52:09 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:OXHTYn90MKDcz4kG+dbepWeHVJ4=', 'Content-Length': '0'} 2023-05-31T20:52:09.714 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000074f3887a6d3de31d-006477b379-1630-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:52:09 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:52:09.714 DEBUG:boto:b'pklxkd-59obj1241000falseobj1242023-05-31T20:50:33.587Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:52:09.715 DEBUG:boto:path=/pklxkd-59/obj124 2023-05-31T20:52:09.715 DEBUG:boto:auth_path=/pklxkd-59/obj124 2023-05-31T20:52:09.715 DEBUG:boto:Method: HEAD 2023-05-31T20:52:09.716 DEBUG:boto:Path: /pklxkd-59/obj124 2023-05-31T20:52:09.716 DEBUG:boto:Data: 2023-05-31T20:52:09.716 DEBUG:boto:Headers: {} 2023-05-31T20:52:09.717 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:52:09.717 DEBUG:boto:Port: 8001 2023-05-31T20:52:09.717 DEBUG:boto:Params: {} 2023-05-31T20:52:09.718 DEBUG:boto:Token: None 2023-05-31T20:52:09.718 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:52:09 GMT /pklxkd-59/obj124 2023-05-31T20:52:09.719 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:JNSEu0G8HstU11OR0eYdPwGwNhc= 2023-05-31T20:52:09.719 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:52:09 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:JNSEu0G8HstU11OR0eYdPwGwNhc=', 'Content-Length': '0'} 2023-05-31T20:52:09.721 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:50:33 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004d66b767ef8c920d-006477b379-1c17-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:52:09 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:52:09.721 DEBUG:boto:path=/pklxkd-59/obj124 2023-05-31T20:52:09.722 DEBUG:boto:auth_path=/pklxkd-59/obj124 2023-05-31T20:52:09.722 DEBUG:boto:Method: HEAD 2023-05-31T20:52:09.722 DEBUG:boto:Path: /pklxkd-59/obj124 2023-05-31T20:52:09.723 DEBUG:boto:Data: 2023-05-31T20:52:09.723 DEBUG:boto:Headers: {} 2023-05-31T20:52:09.723 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:52:09.724 DEBUG:boto:Port: 8000 2023-05-31T20:52:09.724 DEBUG:boto:Params: {} 2023-05-31T20:52:09.724 DEBUG:boto:Token: None 2023-05-31T20:52:09.725 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:52:09 GMT /pklxkd-59/obj124 2023-05-31T20:52:09.725 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:JNSEu0G8HstU11OR0eYdPwGwNhc= 2023-05-31T20:52:09.726 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:52:09 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:JNSEu0G8HstU11OR0eYdPwGwNhc=', 'Content-Length': '0'} 2023-05-31T20:52:09.727 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:50:33 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000050d32b436c2f6583-006477b379-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:52:09 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:52:09.728 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj124 2023-05-31T20:52:09.728 DEBUG:boto:path=/pklxkd-59/obj124 2023-05-31T20:52:09.729 DEBUG:boto:auth_path=/pklxkd-59/obj124 2023-05-31T20:52:09.729 DEBUG:boto:Method: GET 2023-05-31T20:52:09.729 DEBUG:boto:Path: /pklxkd-59/obj124 2023-05-31T20:52:09.730 DEBUG:boto:Data: 2023-05-31T20:52:09.730 DEBUG:boto:Headers: {} 2023-05-31T20:52:09.731 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:52:09.731 DEBUG:boto:Port: 8001 2023-05-31T20:52:09.731 DEBUG:boto:Params: {} 2023-05-31T20:52:09.732 DEBUG:boto:Token: None 2023-05-31T20:52:09.732 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:52:09 GMT /pklxkd-59/obj124 2023-05-31T20:52:09.733 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:2NzBS+KdbWerSzANUGfUBY3iYqo= 2023-05-31T20:52:09.733 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:52:09 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:2NzBS+KdbWerSzANUGfUBY3iYqo=', 'Content-Length': '0'} 2023-05-31T20:52:09.735 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:50:33 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000066f768005dbdc1db-006477b379-1c17-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:52:09 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:52:09.735 DEBUG:boto:path=/pklxkd-59/obj124 2023-05-31T20:52:09.736 DEBUG:boto:auth_path=/pklxkd-59/obj124 2023-05-31T20:52:09.736 DEBUG:boto:Method: GET 2023-05-31T20:52:09.737 DEBUG:boto:Path: /pklxkd-59/obj124 2023-05-31T20:52:09.737 DEBUG:boto:Data: 2023-05-31T20:52:09.737 DEBUG:boto:Headers: {} 2023-05-31T20:52:09.738 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:52:09.738 DEBUG:boto:Port: 8000 2023-05-31T20:52:09.738 DEBUG:boto:Params: {} 2023-05-31T20:52:09.739 DEBUG:boto:Token: None 2023-05-31T20:52:09.739 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:52:09 GMT /pklxkd-59/obj124 2023-05-31T20:52:09.740 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:2NzBS+KdbWerSzANUGfUBY3iYqo= 2023-05-31T20:52:09.740 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:52:09 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:2NzBS+KdbWerSzANUGfUBY3iYqo=', 'Content-Length': '0'} 2023-05-31T20:52:09.742 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:50:33 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000002267339cc13fef9-006477b379-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:52:09 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:52:09.784 DEBUG:boto:path=/pklxkd-59/ 2023-05-31T20:52:09.785 DEBUG:boto:auth_path=/pklxkd-59/ 2023-05-31T20:52:09.785 DEBUG:boto:path=/pklxkd-59/?prefix=obj124 2023-05-31T20:52:09.785 DEBUG:boto:auth_path=/pklxkd-59/?prefix=obj124 2023-05-31T20:52:09.786 DEBUG:boto:Method: GET 2023-05-31T20:52:09.786 DEBUG:boto:Path: /pklxkd-59/?prefix=obj124 2023-05-31T20:52:09.786 DEBUG:boto:Data: 2023-05-31T20:52:09.787 DEBUG:boto:Headers: {} 2023-05-31T20:52:09.787 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:52:09.787 DEBUG:boto:Port: 8001 2023-05-31T20:52:09.788 DEBUG:boto:Params: {} 2023-05-31T20:52:09.788 DEBUG:boto:Token: None 2023-05-31T20:52:09.789 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:52:09 GMT /pklxkd-59/ 2023-05-31T20:52:09.789 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:OXHTYn90MKDcz4kG+dbepWeHVJ4= 2023-05-31T20:52:09.789 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:52:09 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:OXHTYn90MKDcz4kG+dbepWeHVJ4=', 'Content-Length': '0'} 2023-05-31T20:52:09.792 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b076e420d91d96bc-006477b379-1c17-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:52:09 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:52:09.792 DEBUG:boto:b'pklxkd-59obj1241000falseobj1242023-05-31T20:50:33.587Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:52:09.793 DEBUG:boto:path=/pklxkd-59/ 2023-05-31T20:52:09.793 DEBUG:boto:auth_path=/pklxkd-59/ 2023-05-31T20:52:09.794 DEBUG:boto:path=/pklxkd-59/?prefix=obj124 2023-05-31T20:52:09.794 DEBUG:boto:auth_path=/pklxkd-59/?prefix=obj124 2023-05-31T20:52:09.794 DEBUG:boto:Method: GET 2023-05-31T20:52:09.795 DEBUG:boto:Path: /pklxkd-59/?prefix=obj124 2023-05-31T20:52:09.795 DEBUG:boto:Data: 2023-05-31T20:52:09.795 DEBUG:boto:Headers: {} 2023-05-31T20:52:09.796 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:52:09.796 DEBUG:boto:Port: 8000 2023-05-31T20:52:09.797 DEBUG:boto:Params: {} 2023-05-31T20:52:09.797 DEBUG:boto:Token: None 2023-05-31T20:52:09.797 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:52:09 GMT /pklxkd-59/ 2023-05-31T20:52:09.798 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:OXHTYn90MKDcz4kG+dbepWeHVJ4= 2023-05-31T20:52:09.798 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:52:09 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:OXHTYn90MKDcz4kG+dbepWeHVJ4=', 'Content-Length': '0'} 2023-05-31T20:52:09.802 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000028027afed5a87566-006477b379-1630-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:52:09 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:52:09.802 DEBUG:boto:b'pklxkd-59obj1241000falseobj1242023-05-31T20:50:33.587Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:52:09.803 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj145 2023-05-31T20:52:09.803 DEBUG:boto:path=/pklxkd-59/obj145 2023-05-31T20:52:09.804 DEBUG:boto:auth_path=/pklxkd-59/obj145 2023-05-31T20:52:09.804 DEBUG:boto:path=/pklxkd-59/obj145?versionId=null 2023-05-31T20:52:09.804 DEBUG:boto:auth_path=/pklxkd-59/obj145?versionId=null 2023-05-31T20:52:09.805 DEBUG:boto:Method: GET 2023-05-31T20:52:09.805 DEBUG:boto:Path: /pklxkd-59/obj145?versionId=null 2023-05-31T20:52:09.805 DEBUG:boto:Data: 2023-05-31T20:52:09.806 DEBUG:boto:Headers: {} 2023-05-31T20:52:09.806 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:52:09.806 DEBUG:boto:Port: 8001 2023-05-31T20:52:09.807 DEBUG:boto:Params: {} 2023-05-31T20:52:09.807 DEBUG:boto:Token: None 2023-05-31T20:52:09.808 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:52:09 GMT /pklxkd-59/obj145?versionId=null 2023-05-31T20:52:09.808 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:d6nFdy7HAV77gCDZgTps0MCB4Xs= 2023-05-31T20:52:09.808 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:52:09 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:d6nFdy7HAV77gCDZgTps0MCB4Xs=', 'Content-Length': '0'} 2023-05-31T20:52:09.810 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:48:13 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000729ea84cd1fdcb12-006477b379-1c17-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:52:09 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:52:09.811 DEBUG:boto:path=/pklxkd-59/obj145 2023-05-31T20:52:09.811 DEBUG:boto:auth_path=/pklxkd-59/obj145 2023-05-31T20:52:09.811 DEBUG:boto:path=/pklxkd-59/obj145?versionId=null 2023-05-31T20:52:09.812 DEBUG:boto:auth_path=/pklxkd-59/obj145?versionId=null 2023-05-31T20:52:09.812 DEBUG:boto:Method: GET 2023-05-31T20:52:09.813 DEBUG:boto:Path: /pklxkd-59/obj145?versionId=null 2023-05-31T20:52:09.813 DEBUG:boto:Data: 2023-05-31T20:52:09.813 DEBUG:boto:Headers: {} 2023-05-31T20:52:09.814 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:52:09.814 DEBUG:boto:Port: 8000 2023-05-31T20:52:09.814 DEBUG:boto:Params: {} 2023-05-31T20:52:09.815 DEBUG:boto:Token: None 2023-05-31T20:52:09.815 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:52:09 GMT /pklxkd-59/obj145?versionId=null 2023-05-31T20:52:09.816 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:d6nFdy7HAV77gCDZgTps0MCB4Xs= 2023-05-31T20:52:09.816 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:52:09 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:d6nFdy7HAV77gCDZgTps0MCB4Xs=', 'Content-Length': '0'} 2023-05-31T20:52:09.818 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:48:13 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000654deff4aa532a54-006477b379-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:52:09 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:52:09.818 DEBUG:boto:path=/pklxkd-59/obj145 2023-05-31T20:52:09.819 DEBUG:boto:auth_path=/pklxkd-59/obj145 2023-05-31T20:52:09.819 DEBUG:boto:path=/pklxkd-59/obj145?versionId=null 2023-05-31T20:52:09.819 DEBUG:boto:auth_path=/pklxkd-59/obj145?versionId=null 2023-05-31T20:52:09.820 DEBUG:boto:Method: HEAD 2023-05-31T20:52:09.820 DEBUG:boto:Path: /pklxkd-59/obj145?versionId=null 2023-05-31T20:52:09.820 DEBUG:boto:Data: 2023-05-31T20:52:09.821 DEBUG:boto:Headers: {} 2023-05-31T20:52:09.821 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:52:09.821 DEBUG:boto:Port: 8001 2023-05-31T20:52:09.822 DEBUG:boto:Params: {} 2023-05-31T20:52:09.822 DEBUG:boto:Token: None 2023-05-31T20:52:09.822 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:52:09 GMT /pklxkd-59/obj145?versionId=null 2023-05-31T20:52:09.823 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:22KTZdmi9dDJ9b8ZxZcVjH90q5w= 2023-05-31T20:52:09.823 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:52:09 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:22KTZdmi9dDJ9b8ZxZcVjH90q5w=', 'Content-Length': '0'} 2023-05-31T20:52:09.825 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:48:13 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000407cc562fd13beb6-006477b379-1c17-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:52:09 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:52:09.825 DEBUG:boto:path=/pklxkd-59/obj145 2023-05-31T20:52:09.826 DEBUG:boto:auth_path=/pklxkd-59/obj145 2023-05-31T20:52:09.826 DEBUG:boto:path=/pklxkd-59/obj145?versionId=null 2023-05-31T20:52:09.826 DEBUG:boto:auth_path=/pklxkd-59/obj145?versionId=null 2023-05-31T20:52:09.827 DEBUG:boto:Method: HEAD 2023-05-31T20:52:09.827 DEBUG:boto:Path: /pklxkd-59/obj145?versionId=null 2023-05-31T20:52:09.827 DEBUG:boto:Data: 2023-05-31T20:52:09.828 DEBUG:boto:Headers: {} 2023-05-31T20:52:09.828 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:52:09.828 DEBUG:boto:Port: 8000 2023-05-31T20:52:09.829 DEBUG:boto:Params: {} 2023-05-31T20:52:09.829 DEBUG:boto:Token: None 2023-05-31T20:52:09.829 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:52:09 GMT /pklxkd-59/obj145?versionId=null 2023-05-31T20:52:09.830 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:22KTZdmi9dDJ9b8ZxZcVjH90q5w= 2023-05-31T20:52:09.830 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:52:09 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:22KTZdmi9dDJ9b8ZxZcVjH90q5w=', 'Content-Length': '0'} 2023-05-31T20:52:09.832 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:48:13 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000073b67a94d37b6d01-006477b379-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:52:09 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:52:09.833 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj145 2023-05-31T20:52:09.833 DEBUG:boto:path=/pklxkd-59/obj145 2023-05-31T20:52:09.833 DEBUG:boto:auth_path=/pklxkd-59/obj145 2023-05-31T20:52:09.834 DEBUG:boto:path=/pklxkd-59/obj145?versionId=null 2023-05-31T20:52:09.834 DEBUG:boto:auth_path=/pklxkd-59/obj145?versionId=null 2023-05-31T20:52:09.834 DEBUG:boto:Method: GET 2023-05-31T20:52:09.835 DEBUG:boto:Path: /pklxkd-59/obj145?versionId=null 2023-05-31T20:52:09.835 DEBUG:boto:Data: 2023-05-31T20:52:09.835 DEBUG:boto:Headers: {} 2023-05-31T20:52:09.836 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:52:09.836 DEBUG:boto:Port: 8001 2023-05-31T20:52:09.836 DEBUG:boto:Params: {} 2023-05-31T20:52:09.837 DEBUG:boto:Token: None 2023-05-31T20:52:09.837 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:52:09 GMT /pklxkd-59/obj145?versionId=null 2023-05-31T20:52:09.838 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:d6nFdy7HAV77gCDZgTps0MCB4Xs= 2023-05-31T20:52:09.838 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:52:09 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:d6nFdy7HAV77gCDZgTps0MCB4Xs=', 'Content-Length': '0'} 2023-05-31T20:52:09.840 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:48:13 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000cf038f57a9649a9f-006477b379-1c17-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:52:09 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:52:09.880 DEBUG:boto:path=/pklxkd-59/obj145 2023-05-31T20:52:09.881 DEBUG:boto:auth_path=/pklxkd-59/obj145 2023-05-31T20:52:09.881 DEBUG:boto:path=/pklxkd-59/obj145?versionId=null 2023-05-31T20:52:09.881 DEBUG:boto:auth_path=/pklxkd-59/obj145?versionId=null 2023-05-31T20:52:09.882 DEBUG:boto:Method: GET 2023-05-31T20:52:09.882 DEBUG:boto:Path: /pklxkd-59/obj145?versionId=null 2023-05-31T20:52:09.882 DEBUG:boto:Data: 2023-05-31T20:52:09.883 DEBUG:boto:Headers: {} 2023-05-31T20:52:09.883 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:52:09.883 DEBUG:boto:Port: 8000 2023-05-31T20:52:09.884 DEBUG:boto:Params: {} 2023-05-31T20:52:09.884 DEBUG:boto:Token: None 2023-05-31T20:52:09.885 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:52:09 GMT /pklxkd-59/obj145?versionId=null 2023-05-31T20:52:09.885 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:d6nFdy7HAV77gCDZgTps0MCB4Xs= 2023-05-31T20:52:09.885 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:52:09 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:d6nFdy7HAV77gCDZgTps0MCB4Xs=', 'Content-Length': '0'} 2023-05-31T20:52:09.887 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:48:13 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c4c18183e53914d1-006477b379-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:52:09 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:52:09.888 DEBUG:boto:path=/pklxkd-59/ 2023-05-31T20:52:09.888 DEBUG:boto:auth_path=/pklxkd-59/ 2023-05-31T20:52:09.888 DEBUG:boto:path=/pklxkd-59/?prefix=obj145 2023-05-31T20:52:09.889 DEBUG:boto:auth_path=/pklxkd-59/?prefix=obj145 2023-05-31T20:52:09.889 DEBUG:boto:Method: GET 2023-05-31T20:52:09.890 DEBUG:boto:Path: /pklxkd-59/?prefix=obj145 2023-05-31T20:52:09.890 DEBUG:boto:Data: 2023-05-31T20:52:09.890 DEBUG:boto:Headers: {} 2023-05-31T20:52:09.891 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:52:09.891 DEBUG:boto:Port: 8001 2023-05-31T20:52:09.891 DEBUG:boto:Params: {} 2023-05-31T20:52:09.892 DEBUG:boto:Token: None 2023-05-31T20:52:09.892 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:52:09 GMT /pklxkd-59/ 2023-05-31T20:52:09.892 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:OXHTYn90MKDcz4kG+dbepWeHVJ4= 2023-05-31T20:52:09.893 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:52:09 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:OXHTYn90MKDcz4kG+dbepWeHVJ4=', 'Content-Length': '0'} 2023-05-31T20:52:09.895 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000db17de050b862daa-006477b379-1c17-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:52:09 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:52:09.896 DEBUG:boto:b'pklxkd-59obj1451000falseobj1452023-05-31T20:48:13.107Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:52:09.896 DEBUG:boto:path=/pklxkd-59/ 2023-05-31T20:52:09.897 DEBUG:boto:auth_path=/pklxkd-59/ 2023-05-31T20:52:09.897 DEBUG:boto:path=/pklxkd-59/?prefix=obj145 2023-05-31T20:52:09.897 DEBUG:boto:auth_path=/pklxkd-59/?prefix=obj145 2023-05-31T20:52:09.898 DEBUG:boto:Method: GET 2023-05-31T20:52:09.898 DEBUG:boto:Path: /pklxkd-59/?prefix=obj145 2023-05-31T20:52:09.898 DEBUG:boto:Data: 2023-05-31T20:52:09.899 DEBUG:boto:Headers: {} 2023-05-31T20:52:09.899 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:52:09.899 DEBUG:boto:Port: 8000 2023-05-31T20:52:09.900 DEBUG:boto:Params: {} 2023-05-31T20:52:09.900 DEBUG:boto:Token: None 2023-05-31T20:52:09.901 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:52:09 GMT /pklxkd-59/ 2023-05-31T20:52:09.901 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:OXHTYn90MKDcz4kG+dbepWeHVJ4= 2023-05-31T20:52:09.901 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:52:09 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:OXHTYn90MKDcz4kG+dbepWeHVJ4=', 'Content-Length': '0'} 2023-05-31T20:52:09.905 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000046eab7ca9be70f5f-006477b379-1630-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:52:09 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:52:09.906 DEBUG:boto:b'pklxkd-59obj1451000falseobj1452023-05-31T20:48:13.107Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:52:09.906 DEBUG:boto:path=/pklxkd-59/obj145 2023-05-31T20:52:09.907 DEBUG:boto:auth_path=/pklxkd-59/obj145 2023-05-31T20:52:09.907 DEBUG:boto:Method: HEAD 2023-05-31T20:52:09.907 DEBUG:boto:Path: /pklxkd-59/obj145 2023-05-31T20:52:09.908 DEBUG:boto:Data: 2023-05-31T20:52:09.908 DEBUG:boto:Headers: {} 2023-05-31T20:52:09.908 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:52:09.909 DEBUG:boto:Port: 8001 2023-05-31T20:52:09.909 DEBUG:boto:Params: {} 2023-05-31T20:52:09.909 DEBUG:boto:Token: None 2023-05-31T20:52:09.910 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:52:09 GMT /pklxkd-59/obj145 2023-05-31T20:52:09.910 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:e4o6Ty1V1tAf7nIG5Ng7HaK4+EA= 2023-05-31T20:52:09.911 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:52:09 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:e4o6Ty1V1tAf7nIG5Ng7HaK4+EA=', 'Content-Length': '0'} 2023-05-31T20:52:09.912 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:48:13 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006c84a59fbf1a3628-006477b379-1c17-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:52:09 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:52:09.913 DEBUG:boto:path=/pklxkd-59/obj145 2023-05-31T20:52:09.913 DEBUG:boto:auth_path=/pklxkd-59/obj145 2023-05-31T20:52:09.914 DEBUG:boto:Method: HEAD 2023-05-31T20:52:09.914 DEBUG:boto:Path: /pklxkd-59/obj145 2023-05-31T20:52:09.914 DEBUG:boto:Data: 2023-05-31T20:52:09.915 DEBUG:boto:Headers: {} 2023-05-31T20:52:09.915 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:52:09.915 DEBUG:boto:Port: 8000 2023-05-31T20:52:09.916 DEBUG:boto:Params: {} 2023-05-31T20:52:09.916 DEBUG:boto:Token: None 2023-05-31T20:52:09.917 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:52:09 GMT /pklxkd-59/obj145 2023-05-31T20:52:09.917 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:e4o6Ty1V1tAf7nIG5Ng7HaK4+EA= 2023-05-31T20:52:09.917 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:52:09 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:e4o6Ty1V1tAf7nIG5Ng7HaK4+EA=', 'Content-Length': '0'} 2023-05-31T20:52:09.919 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:48:13 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000428f9a1b395fa7ac-006477b379-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:52:09 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:52:09.919 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj145 2023-05-31T20:52:09.920 DEBUG:boto:path=/pklxkd-59/obj145 2023-05-31T20:52:09.920 DEBUG:boto:auth_path=/pklxkd-59/obj145 2023-05-31T20:52:09.921 DEBUG:boto:Method: GET 2023-05-31T20:52:09.921 DEBUG:boto:Path: /pklxkd-59/obj145 2023-05-31T20:52:09.921 DEBUG:boto:Data: 2023-05-31T20:52:09.922 DEBUG:boto:Headers: {} 2023-05-31T20:52:09.922 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:52:09.922 DEBUG:boto:Port: 8001 2023-05-31T20:52:09.923 DEBUG:boto:Params: {} 2023-05-31T20:52:09.923 DEBUG:boto:Token: None 2023-05-31T20:52:09.924 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:52:09 GMT /pklxkd-59/obj145 2023-05-31T20:52:09.924 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:riHzGyT1wYAkhR3T6+XKdqsJeT8= 2023-05-31T20:52:09.924 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:52:09 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:riHzGyT1wYAkhR3T6+XKdqsJeT8=', 'Content-Length': '0'} 2023-05-31T20:52:09.926 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:48:13 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006f65a8c9b8c3a140-006477b379-1c17-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:52:09 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:52:09.927 DEBUG:boto:path=/pklxkd-59/obj145 2023-05-31T20:52:09.927 DEBUG:boto:auth_path=/pklxkd-59/obj145 2023-05-31T20:52:09.928 DEBUG:boto:Method: GET 2023-05-31T20:52:09.928 DEBUG:boto:Path: /pklxkd-59/obj145 2023-05-31T20:52:09.928 DEBUG:boto:Data: 2023-05-31T20:52:09.929 DEBUG:boto:Headers: {} 2023-05-31T20:52:09.929 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:52:09.929 DEBUG:boto:Port: 8000 2023-05-31T20:52:09.930 DEBUG:boto:Params: {} 2023-05-31T20:52:09.930 DEBUG:boto:Token: None 2023-05-31T20:52:09.931 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:52:09 GMT /pklxkd-59/obj145 2023-05-31T20:52:09.931 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:riHzGyT1wYAkhR3T6+XKdqsJeT8= 2023-05-31T20:52:09.931 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:52:09 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:riHzGyT1wYAkhR3T6+XKdqsJeT8=', 'Content-Length': '0'} 2023-05-31T20:52:09.933 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:48:13 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000097b2197f7c8a047f-006477b379-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:52:09 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:52:09.976 DEBUG:boto:path=/pklxkd-59/ 2023-05-31T20:52:09.977 DEBUG:boto:auth_path=/pklxkd-59/ 2023-05-31T20:52:09.977 DEBUG:boto:path=/pklxkd-59/?prefix=obj145 2023-05-31T20:52:09.977 DEBUG:boto:auth_path=/pklxkd-59/?prefix=obj145 2023-05-31T20:52:09.978 DEBUG:boto:Method: GET 2023-05-31T20:52:09.978 DEBUG:boto:Path: /pklxkd-59/?prefix=obj145 2023-05-31T20:52:09.978 DEBUG:boto:Data: 2023-05-31T20:52:09.979 DEBUG:boto:Headers: {} 2023-05-31T20:52:09.979 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:52:09.979 DEBUG:boto:Port: 8001 2023-05-31T20:52:09.980 DEBUG:boto:Params: {} 2023-05-31T20:52:09.980 DEBUG:boto:Token: None 2023-05-31T20:52:09.981 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:52:09 GMT /pklxkd-59/ 2023-05-31T20:52:09.981 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:OXHTYn90MKDcz4kG+dbepWeHVJ4= 2023-05-31T20:52:09.981 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:52:09 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:OXHTYn90MKDcz4kG+dbepWeHVJ4=', 'Content-Length': '0'} 2023-05-31T20:52:09.984 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000008f207ebd0041b7fc-006477b379-1c17-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:52:09 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:52:09.984 DEBUG:boto:b'pklxkd-59obj1451000falseobj1452023-05-31T20:48:13.107Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:52:09.985 DEBUG:boto:path=/pklxkd-59/ 2023-05-31T20:52:09.985 DEBUG:boto:auth_path=/pklxkd-59/ 2023-05-31T20:52:09.986 DEBUG:boto:path=/pklxkd-59/?prefix=obj145 2023-05-31T20:52:09.986 DEBUG:boto:auth_path=/pklxkd-59/?prefix=obj145 2023-05-31T20:52:09.986 DEBUG:boto:Method: GET 2023-05-31T20:52:09.987 DEBUG:boto:Path: /pklxkd-59/?prefix=obj145 2023-05-31T20:52:09.987 DEBUG:boto:Data: 2023-05-31T20:52:09.987 DEBUG:boto:Headers: {} 2023-05-31T20:52:09.988 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:52:09.988 DEBUG:boto:Port: 8000 2023-05-31T20:52:09.988 DEBUG:boto:Params: {} 2023-05-31T20:52:09.989 DEBUG:boto:Token: None 2023-05-31T20:52:09.989 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:52:09 GMT /pklxkd-59/ 2023-05-31T20:52:09.990 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:OXHTYn90MKDcz4kG+dbepWeHVJ4= 2023-05-31T20:52:09.990 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:52:09 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:OXHTYn90MKDcz4kG+dbepWeHVJ4=', 'Content-Length': '0'} 2023-05-31T20:52:09.994 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f786a88af52fa936-006477b379-1630-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:52:09 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:52:09.994 DEBUG:boto:b'pklxkd-59obj1451000falseobj1452023-05-31T20:48:13.107Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:52:09.995 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj155 2023-05-31T20:52:09.995 DEBUG:boto:path=/pklxkd-59/obj155 2023-05-31T20:52:09.996 DEBUG:boto:auth_path=/pklxkd-59/obj155 2023-05-31T20:52:09.996 DEBUG:boto:path=/pklxkd-59/obj155?versionId=null 2023-05-31T20:52:09.996 DEBUG:boto:auth_path=/pklxkd-59/obj155?versionId=null 2023-05-31T20:52:09.997 DEBUG:boto:Method: GET 2023-05-31T20:52:09.997 DEBUG:boto:Path: /pklxkd-59/obj155?versionId=null 2023-05-31T20:52:09.998 DEBUG:boto:Data: 2023-05-31T20:52:09.998 DEBUG:boto:Headers: {} 2023-05-31T20:52:09.998 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:52:09.999 DEBUG:boto:Port: 8001 2023-05-31T20:52:09.999 DEBUG:boto:Params: {} 2023-05-31T20:52:09.999 DEBUG:boto:Token: None 2023-05-31T20:52:10.000 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:52:10 GMT /pklxkd-59/obj155?versionId=null 2023-05-31T20:52:10.000 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:fQsc67Y1YkKyOEgCMs4GAE0m6L4= 2023-05-31T20:52:10.001 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:52:10 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:fQsc67Y1YkKyOEgCMs4GAE0m6L4=', 'Content-Length': '0'} 2023-05-31T20:52:10.003 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:50:33 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007b3f4bf3042b13e7-006477b379-1c17-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:52:09 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:52:10.004 DEBUG:boto:path=/pklxkd-59/obj155 2023-05-31T20:52:10.004 DEBUG:boto:auth_path=/pklxkd-59/obj155 2023-05-31T20:52:10.004 DEBUG:boto:path=/pklxkd-59/obj155?versionId=null 2023-05-31T20:52:10.005 DEBUG:boto:auth_path=/pklxkd-59/obj155?versionId=null 2023-05-31T20:52:10.005 DEBUG:boto:Method: GET 2023-05-31T20:52:10.005 DEBUG:boto:Path: /pklxkd-59/obj155?versionId=null 2023-05-31T20:52:10.006 DEBUG:boto:Data: 2023-05-31T20:52:10.006 DEBUG:boto:Headers: {} 2023-05-31T20:52:10.007 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:52:10.007 DEBUG:boto:Port: 8000 2023-05-31T20:52:10.007 DEBUG:boto:Params: {} 2023-05-31T20:52:10.008 DEBUG:boto:Token: None 2023-05-31T20:52:10.008 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:52:10 GMT /pklxkd-59/obj155?versionId=null 2023-05-31T20:52:10.008 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:fQsc67Y1YkKyOEgCMs4GAE0m6L4= 2023-05-31T20:52:10.009 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:52:10 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:fQsc67Y1YkKyOEgCMs4GAE0m6L4=', 'Content-Length': '0'} 2023-05-31T20:52:10.011 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:50:33 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e63ce3983614503c-006477b37a-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:52:10 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:52:10.011 DEBUG:boto:path=/pklxkd-59/obj155 2023-05-31T20:52:10.012 DEBUG:boto:auth_path=/pklxkd-59/obj155 2023-05-31T20:52:10.012 DEBUG:boto:path=/pklxkd-59/obj155?versionId=null 2023-05-31T20:52:10.012 DEBUG:boto:auth_path=/pklxkd-59/obj155?versionId=null 2023-05-31T20:52:10.013 DEBUG:boto:Method: HEAD 2023-05-31T20:52:10.013 DEBUG:boto:Path: /pklxkd-59/obj155?versionId=null 2023-05-31T20:52:10.013 DEBUG:boto:Data: 2023-05-31T20:52:10.014 DEBUG:boto:Headers: {} 2023-05-31T20:52:10.014 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:52:10.014 DEBUG:boto:Port: 8001 2023-05-31T20:52:10.015 DEBUG:boto:Params: {} 2023-05-31T20:52:10.015 DEBUG:boto:Token: None 2023-05-31T20:52:10.015 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:52:10 GMT /pklxkd-59/obj155?versionId=null 2023-05-31T20:52:10.016 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:R3MYBK5nTlsG1l2LI103Iz4qluM= 2023-05-31T20:52:10.016 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:52:10 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:R3MYBK5nTlsG1l2LI103Iz4qluM=', 'Content-Length': '0'} 2023-05-31T20:52:10.018 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:50:33 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000e02e8c289f31fbe-006477b37a-1c17-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:52:10 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:52:10.019 DEBUG:boto:path=/pklxkd-59/obj155 2023-05-31T20:52:10.019 DEBUG:boto:auth_path=/pklxkd-59/obj155 2023-05-31T20:52:10.019 DEBUG:boto:path=/pklxkd-59/obj155?versionId=null 2023-05-31T20:52:10.020 DEBUG:boto:auth_path=/pklxkd-59/obj155?versionId=null 2023-05-31T20:52:10.020 DEBUG:boto:Method: HEAD 2023-05-31T20:52:10.020 DEBUG:boto:Path: /pklxkd-59/obj155?versionId=null 2023-05-31T20:52:10.021 DEBUG:boto:Data: 2023-05-31T20:52:10.021 DEBUG:boto:Headers: {} 2023-05-31T20:52:10.021 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:52:10.022 DEBUG:boto:Port: 8000 2023-05-31T20:52:10.022 DEBUG:boto:Params: {} 2023-05-31T20:52:10.022 DEBUG:boto:Token: None 2023-05-31T20:52:10.023 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:52:10 GMT /pklxkd-59/obj155?versionId=null 2023-05-31T20:52:10.023 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:R3MYBK5nTlsG1l2LI103Iz4qluM= 2023-05-31T20:52:10.023 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:52:10 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:R3MYBK5nTlsG1l2LI103Iz4qluM=', 'Content-Length': '0'} 2023-05-31T20:52:10.025 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:50:33 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000017d396d0295a3af-006477b37a-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:52:10 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:52:10.026 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj155 2023-05-31T20:52:10.026 DEBUG:boto:path=/pklxkd-59/obj155 2023-05-31T20:52:10.027 DEBUG:boto:auth_path=/pklxkd-59/obj155 2023-05-31T20:52:10.027 DEBUG:boto:path=/pklxkd-59/obj155?versionId=null 2023-05-31T20:52:10.027 DEBUG:boto:auth_path=/pklxkd-59/obj155?versionId=null 2023-05-31T20:52:10.028 DEBUG:boto:Method: GET 2023-05-31T20:52:10.028 DEBUG:boto:Path: /pklxkd-59/obj155?versionId=null 2023-05-31T20:52:10.028 DEBUG:boto:Data: 2023-05-31T20:52:10.029 DEBUG:boto:Headers: {} 2023-05-31T20:52:10.029 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:52:10.029 DEBUG:boto:Port: 8001 2023-05-31T20:52:10.030 DEBUG:boto:Params: {} 2023-05-31T20:52:10.030 DEBUG:boto:Token: None 2023-05-31T20:52:10.031 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:52:10 GMT /pklxkd-59/obj155?versionId=null 2023-05-31T20:52:10.031 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:fQsc67Y1YkKyOEgCMs4GAE0m6L4= 2023-05-31T20:52:10.032 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:52:10 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:fQsc67Y1YkKyOEgCMs4GAE0m6L4=', 'Content-Length': '0'} 2023-05-31T20:52:10.033 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:50:33 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a9160780263765a0-006477b37a-1c17-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:52:10 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:52:10.034 DEBUG:boto:path=/pklxkd-59/obj155 2023-05-31T20:52:10.034 DEBUG:boto:auth_path=/pklxkd-59/obj155 2023-05-31T20:52:10.035 DEBUG:boto:path=/pklxkd-59/obj155?versionId=null 2023-05-31T20:52:10.035 DEBUG:boto:auth_path=/pklxkd-59/obj155?versionId=null 2023-05-31T20:52:10.035 DEBUG:boto:Method: GET 2023-05-31T20:52:10.036 DEBUG:boto:Path: /pklxkd-59/obj155?versionId=null 2023-05-31T20:52:10.036 DEBUG:boto:Data: 2023-05-31T20:52:10.037 DEBUG:boto:Headers: {} 2023-05-31T20:52:10.037 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:52:10.037 DEBUG:boto:Port: 8000 2023-05-31T20:52:10.038 DEBUG:boto:Params: {} 2023-05-31T20:52:10.038 DEBUG:boto:Token: None 2023-05-31T20:52:10.038 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:52:10 GMT /pklxkd-59/obj155?versionId=null 2023-05-31T20:52:10.039 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:fQsc67Y1YkKyOEgCMs4GAE0m6L4= 2023-05-31T20:52:10.039 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:52:10 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:fQsc67Y1YkKyOEgCMs4GAE0m6L4=', 'Content-Length': '0'} 2023-05-31T20:52:10.041 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:50:33 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000073f0623f05c46260-006477b37a-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:52:10 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:52:10.084 DEBUG:boto:path=/pklxkd-59/ 2023-05-31T20:52:10.085 DEBUG:boto:auth_path=/pklxkd-59/ 2023-05-31T20:52:10.085 DEBUG:boto:path=/pklxkd-59/?prefix=obj155 2023-05-31T20:52:10.085 DEBUG:boto:auth_path=/pklxkd-59/?prefix=obj155 2023-05-31T20:52:10.086 DEBUG:boto:Method: GET 2023-05-31T20:52:10.086 DEBUG:boto:Path: /pklxkd-59/?prefix=obj155 2023-05-31T20:52:10.086 DEBUG:boto:Data: 2023-05-31T20:52:10.087 DEBUG:boto:Headers: {} 2023-05-31T20:52:10.087 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:52:10.087 DEBUG:boto:Port: 8001 2023-05-31T20:52:10.088 DEBUG:boto:Params: {} 2023-05-31T20:52:10.088 DEBUG:boto:Token: None 2023-05-31T20:52:10.089 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:52:10 GMT /pklxkd-59/ 2023-05-31T20:52:10.089 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:1aCZ7BhkC9nnuJr6CPizGBU2/jg= 2023-05-31T20:52:10.089 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:52:10 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:1aCZ7BhkC9nnuJr6CPizGBU2/jg=', 'Content-Length': '0'} 2023-05-31T20:52:10.092 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d500833bd5510e51-006477b37a-1c17-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:52:10 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:52:10.092 DEBUG:boto:b'pklxkd-59obj1551000falseobj1552023-05-31T20:50:33.605Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:52:10.093 DEBUG:boto:path=/pklxkd-59/ 2023-05-31T20:52:10.093 DEBUG:boto:auth_path=/pklxkd-59/ 2023-05-31T20:52:10.094 DEBUG:boto:path=/pklxkd-59/?prefix=obj155 2023-05-31T20:52:10.094 DEBUG:boto:auth_path=/pklxkd-59/?prefix=obj155 2023-05-31T20:52:10.094 DEBUG:boto:Method: GET 2023-05-31T20:52:10.095 DEBUG:boto:Path: /pklxkd-59/?prefix=obj155 2023-05-31T20:52:10.095 DEBUG:boto:Data: 2023-05-31T20:52:10.095 DEBUG:boto:Headers: {} 2023-05-31T20:52:10.096 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:52:10.096 DEBUG:boto:Port: 8000 2023-05-31T20:52:10.096 DEBUG:boto:Params: {} 2023-05-31T20:52:10.097 DEBUG:boto:Token: None 2023-05-31T20:52:10.097 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:52:10 GMT /pklxkd-59/ 2023-05-31T20:52:10.098 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:1aCZ7BhkC9nnuJr6CPizGBU2/jg= 2023-05-31T20:52:10.098 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:52:10 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:1aCZ7BhkC9nnuJr6CPizGBU2/jg=', 'Content-Length': '0'} 2023-05-31T20:52:10.102 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ebecbdf9d7a99c66-006477b37a-1630-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:52:10 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:52:10.102 DEBUG:boto:b'pklxkd-59obj1551000falseobj1552023-05-31T20:50:33.605Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:52:10.103 DEBUG:boto:path=/pklxkd-59/obj155 2023-05-31T20:52:10.103 DEBUG:boto:auth_path=/pklxkd-59/obj155 2023-05-31T20:52:10.104 DEBUG:boto:Method: HEAD 2023-05-31T20:52:10.104 DEBUG:boto:Path: /pklxkd-59/obj155 2023-05-31T20:52:10.104 DEBUG:boto:Data: 2023-05-31T20:52:10.105 DEBUG:boto:Headers: {} 2023-05-31T20:52:10.105 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:52:10.105 DEBUG:boto:Port: 8001 2023-05-31T20:52:10.106 DEBUG:boto:Params: {} 2023-05-31T20:52:10.106 DEBUG:boto:Token: None 2023-05-31T20:52:10.106 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:52:10 GMT /pklxkd-59/obj155 2023-05-31T20:52:10.107 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:FkOKIhfAIYXT7CqCQeXX4JeTd5o= 2023-05-31T20:52:10.107 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:52:10 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:FkOKIhfAIYXT7CqCQeXX4JeTd5o=', 'Content-Length': '0'} 2023-05-31T20:52:10.109 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:50:33 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000097579d4814d1b597-006477b37a-1c17-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:52:10 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:52:10.109 DEBUG:boto:path=/pklxkd-59/obj155 2023-05-31T20:52:10.110 DEBUG:boto:auth_path=/pklxkd-59/obj155 2023-05-31T20:52:10.110 DEBUG:boto:Method: HEAD 2023-05-31T20:52:10.111 DEBUG:boto:Path: /pklxkd-59/obj155 2023-05-31T20:52:10.111 DEBUG:boto:Data: 2023-05-31T20:52:10.111 DEBUG:boto:Headers: {} 2023-05-31T20:52:10.112 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:52:10.112 DEBUG:boto:Port: 8000 2023-05-31T20:52:10.112 DEBUG:boto:Params: {} 2023-05-31T20:52:10.113 DEBUG:boto:Token: None 2023-05-31T20:52:10.113 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:52:10 GMT /pklxkd-59/obj155 2023-05-31T20:52:10.113 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:FkOKIhfAIYXT7CqCQeXX4JeTd5o= 2023-05-31T20:52:10.114 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:52:10 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:FkOKIhfAIYXT7CqCQeXX4JeTd5o=', 'Content-Length': '0'} 2023-05-31T20:52:10.115 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:50:33 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009d0218b1f540a60b-006477b37a-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:52:10 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:52:10.116 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj155 2023-05-31T20:52:10.116 DEBUG:boto:path=/pklxkd-59/obj155 2023-05-31T20:52:10.117 DEBUG:boto:auth_path=/pklxkd-59/obj155 2023-05-31T20:52:10.117 DEBUG:boto:Method: GET 2023-05-31T20:52:10.117 DEBUG:boto:Path: /pklxkd-59/obj155 2023-05-31T20:52:10.118 DEBUG:boto:Data: 2023-05-31T20:52:10.118 DEBUG:boto:Headers: {} 2023-05-31T20:52:10.119 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:52:10.119 DEBUG:boto:Port: 8001 2023-05-31T20:52:10.119 DEBUG:boto:Params: {} 2023-05-31T20:52:10.120 DEBUG:boto:Token: None 2023-05-31T20:52:10.120 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:52:10 GMT /pklxkd-59/obj155 2023-05-31T20:52:10.120 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:LQ4f82JXP2b8o6op5uGccWcQ11I= 2023-05-31T20:52:10.121 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:52:10 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:LQ4f82JXP2b8o6op5uGccWcQ11I=', 'Content-Length': '0'} 2023-05-31T20:52:10.123 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:50:33 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f92972ebbbcdcdee-006477b37a-1c17-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:52:10 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:52:10.164 DEBUG:boto:path=/pklxkd-59/obj155 2023-05-31T20:52:10.165 DEBUG:boto:auth_path=/pklxkd-59/obj155 2023-05-31T20:52:10.165 DEBUG:boto:Method: GET 2023-05-31T20:52:10.165 DEBUG:boto:Path: /pklxkd-59/obj155 2023-05-31T20:52:10.166 DEBUG:boto:Data: 2023-05-31T20:52:10.166 DEBUG:boto:Headers: {} 2023-05-31T20:52:10.166 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:52:10.167 DEBUG:boto:Port: 8000 2023-05-31T20:52:10.167 DEBUG:boto:Params: {} 2023-05-31T20:52:10.167 DEBUG:boto:Token: None 2023-05-31T20:52:10.168 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:52:10 GMT /pklxkd-59/obj155 2023-05-31T20:52:10.168 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:LQ4f82JXP2b8o6op5uGccWcQ11I= 2023-05-31T20:52:10.169 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:52:10 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:LQ4f82JXP2b8o6op5uGccWcQ11I=', 'Content-Length': '0'} 2023-05-31T20:52:10.170 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:50:33 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e857b30809fa98c5-006477b37a-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:52:10 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:52:10.171 DEBUG:boto:path=/pklxkd-59/ 2023-05-31T20:52:10.172 DEBUG:boto:auth_path=/pklxkd-59/ 2023-05-31T20:52:10.172 DEBUG:boto:path=/pklxkd-59/?prefix=obj155 2023-05-31T20:52:10.172 DEBUG:boto:auth_path=/pklxkd-59/?prefix=obj155 2023-05-31T20:52:10.173 DEBUG:boto:Method: GET 2023-05-31T20:52:10.173 DEBUG:boto:Path: /pklxkd-59/?prefix=obj155 2023-05-31T20:52:10.173 DEBUG:boto:Data: 2023-05-31T20:52:10.174 DEBUG:boto:Headers: {} 2023-05-31T20:52:10.174 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:52:10.174 DEBUG:boto:Port: 8001 2023-05-31T20:52:10.175 DEBUG:boto:Params: {} 2023-05-31T20:52:10.175 DEBUG:boto:Token: None 2023-05-31T20:52:10.175 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:52:10 GMT /pklxkd-59/ 2023-05-31T20:52:10.176 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:1aCZ7BhkC9nnuJr6CPizGBU2/jg= 2023-05-31T20:52:10.176 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:52:10 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:1aCZ7BhkC9nnuJr6CPizGBU2/jg=', 'Content-Length': '0'} 2023-05-31T20:52:10.179 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e188993317e51b37-006477b37a-1c17-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:52:10 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:52:10.179 DEBUG:boto:b'pklxkd-59obj1551000falseobj1552023-05-31T20:50:33.605Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:52:10.180 DEBUG:boto:path=/pklxkd-59/ 2023-05-31T20:52:10.180 DEBUG:boto:auth_path=/pklxkd-59/ 2023-05-31T20:52:10.181 DEBUG:boto:path=/pklxkd-59/?prefix=obj155 2023-05-31T20:52:10.181 DEBUG:boto:auth_path=/pklxkd-59/?prefix=obj155 2023-05-31T20:52:10.181 DEBUG:boto:Method: GET 2023-05-31T20:52:10.182 DEBUG:boto:Path: /pklxkd-59/?prefix=obj155 2023-05-31T20:52:10.182 DEBUG:boto:Data: 2023-05-31T20:52:10.182 DEBUG:boto:Headers: {} 2023-05-31T20:52:10.183 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:52:10.183 DEBUG:boto:Port: 8000 2023-05-31T20:52:10.183 DEBUG:boto:Params: {} 2023-05-31T20:52:10.184 DEBUG:boto:Token: None 2023-05-31T20:52:10.184 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:52:10 GMT /pklxkd-59/ 2023-05-31T20:52:10.185 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:1aCZ7BhkC9nnuJr6CPizGBU2/jg= 2023-05-31T20:52:10.185 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:52:10 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:1aCZ7BhkC9nnuJr6CPizGBU2/jg=', 'Content-Length': '0'} 2023-05-31T20:52:10.189 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000207980a90ff65075-006477b37a-1630-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:52:10 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:52:10.189 DEBUG:boto:b'pklxkd-59obj1551000falseobj1552023-05-31T20:50:33.605Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:52:10.190 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj17 2023-05-31T20:52:10.190 DEBUG:boto:path=/pklxkd-59/obj17 2023-05-31T20:52:10.190 DEBUG:boto:auth_path=/pklxkd-59/obj17 2023-05-31T20:52:10.191 DEBUG:boto:path=/pklxkd-59/obj17?versionId=null 2023-05-31T20:52:10.191 DEBUG:boto:auth_path=/pklxkd-59/obj17?versionId=null 2023-05-31T20:52:10.192 DEBUG:boto:Method: GET 2023-05-31T20:52:10.192 DEBUG:boto:Path: /pklxkd-59/obj17?versionId=null 2023-05-31T20:52:10.192 DEBUG:boto:Data: 2023-05-31T20:52:10.193 DEBUG:boto:Headers: {} 2023-05-31T20:52:10.193 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:52:10.193 DEBUG:boto:Port: 8001 2023-05-31T20:52:10.194 DEBUG:boto:Params: {} 2023-05-31T20:52:10.194 DEBUG:boto:Token: None 2023-05-31T20:52:10.194 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:52:10 GMT /pklxkd-59/obj17?versionId=null 2023-05-31T20:52:10.195 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:AsfZpkCHwBktDRme3a7R2Db/dcg= 2023-05-31T20:52:10.195 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:52:10 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:AsfZpkCHwBktDRme3a7R2Db/dcg=', 'Content-Length': '0'} 2023-05-31T20:52:10.197 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:43:07 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000744c3dbe21a3d274-006477b37a-1c17-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:52:10 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:52:10.198 DEBUG:boto:path=/pklxkd-59/obj17 2023-05-31T20:52:10.198 DEBUG:boto:auth_path=/pklxkd-59/obj17 2023-05-31T20:52:10.198 DEBUG:boto:path=/pklxkd-59/obj17?versionId=null 2023-05-31T20:52:10.199 DEBUG:boto:auth_path=/pklxkd-59/obj17?versionId=null 2023-05-31T20:52:10.199 DEBUG:boto:Method: GET 2023-05-31T20:52:10.200 DEBUG:boto:Path: /pklxkd-59/obj17?versionId=null 2023-05-31T20:52:10.200 DEBUG:boto:Data: 2023-05-31T20:52:10.200 DEBUG:boto:Headers: {} 2023-05-31T20:52:10.201 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:52:10.201 DEBUG:boto:Port: 8000 2023-05-31T20:52:10.201 DEBUG:boto:Params: {} 2023-05-31T20:52:10.202 DEBUG:boto:Token: None 2023-05-31T20:52:10.202 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:52:10 GMT /pklxkd-59/obj17?versionId=null 2023-05-31T20:52:10.203 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:AsfZpkCHwBktDRme3a7R2Db/dcg= 2023-05-31T20:52:10.203 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:52:10 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:AsfZpkCHwBktDRme3a7R2Db/dcg=', 'Content-Length': '0'} 2023-05-31T20:52:10.205 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:43:07 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a75349167c7c5cd3-006477b37a-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:52:10 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:52:10.248 DEBUG:boto:path=/pklxkd-59/obj17 2023-05-31T20:52:10.249 DEBUG:boto:auth_path=/pklxkd-59/obj17 2023-05-31T20:52:10.249 DEBUG:boto:path=/pklxkd-59/obj17?versionId=null 2023-05-31T20:52:10.249 DEBUG:boto:auth_path=/pklxkd-59/obj17?versionId=null 2023-05-31T20:52:10.250 DEBUG:boto:Method: HEAD 2023-05-31T20:52:10.250 DEBUG:boto:Path: /pklxkd-59/obj17?versionId=null 2023-05-31T20:52:10.250 DEBUG:boto:Data: 2023-05-31T20:52:10.251 DEBUG:boto:Headers: {} 2023-05-31T20:52:10.251 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:52:10.251 DEBUG:boto:Port: 8001 2023-05-31T20:52:10.252 DEBUG:boto:Params: {} 2023-05-31T20:52:10.252 DEBUG:boto:Token: None 2023-05-31T20:52:10.252 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:52:10 GMT /pklxkd-59/obj17?versionId=null 2023-05-31T20:52:10.253 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:p/vrqqf5njg3OuwZvCL0oXJHv/o= 2023-05-31T20:52:10.253 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:52:10 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:p/vrqqf5njg3OuwZvCL0oXJHv/o=', 'Content-Length': '0'} 2023-05-31T20:52:10.255 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:43:07 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005a702c380b0ed38c-006477b37a-1c17-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:52:10 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:52:10.255 DEBUG:boto:path=/pklxkd-59/obj17 2023-05-31T20:52:10.256 DEBUG:boto:auth_path=/pklxkd-59/obj17 2023-05-31T20:52:10.256 DEBUG:boto:path=/pklxkd-59/obj17?versionId=null 2023-05-31T20:52:10.257 DEBUG:boto:auth_path=/pklxkd-59/obj17?versionId=null 2023-05-31T20:52:10.257 DEBUG:boto:Method: HEAD 2023-05-31T20:52:10.257 DEBUG:boto:Path: /pklxkd-59/obj17?versionId=null 2023-05-31T20:52:10.258 DEBUG:boto:Data: 2023-05-31T20:52:10.258 DEBUG:boto:Headers: {} 2023-05-31T20:52:10.258 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:52:10.259 DEBUG:boto:Port: 8000 2023-05-31T20:52:10.259 DEBUG:boto:Params: {} 2023-05-31T20:52:10.259 DEBUG:boto:Token: None 2023-05-31T20:52:10.260 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:52:10 GMT /pklxkd-59/obj17?versionId=null 2023-05-31T20:52:10.260 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:p/vrqqf5njg3OuwZvCL0oXJHv/o= 2023-05-31T20:52:10.261 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:52:10 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:p/vrqqf5njg3OuwZvCL0oXJHv/o=', 'Content-Length': '0'} 2023-05-31T20:52:10.262 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:43:07 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000210a2d37c2151dba-006477b37a-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:52:10 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:52:10.263 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj17 2023-05-31T20:52:10.263 DEBUG:boto:path=/pklxkd-59/obj17 2023-05-31T20:52:10.264 DEBUG:boto:auth_path=/pklxkd-59/obj17 2023-05-31T20:52:10.264 DEBUG:boto:path=/pklxkd-59/obj17?versionId=null 2023-05-31T20:52:10.264 DEBUG:boto:auth_path=/pklxkd-59/obj17?versionId=null 2023-05-31T20:52:10.265 DEBUG:boto:Method: GET 2023-05-31T20:52:10.265 DEBUG:boto:Path: /pklxkd-59/obj17?versionId=null 2023-05-31T20:52:10.265 DEBUG:boto:Data: 2023-05-31T20:52:10.266 DEBUG:boto:Headers: {} 2023-05-31T20:52:10.266 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:52:10.266 DEBUG:boto:Port: 8001 2023-05-31T20:52:10.267 DEBUG:boto:Params: {} 2023-05-31T20:52:10.267 DEBUG:boto:Token: None 2023-05-31T20:52:10.268 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:52:10 GMT /pklxkd-59/obj17?versionId=null 2023-05-31T20:52:10.268 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:AsfZpkCHwBktDRme3a7R2Db/dcg= 2023-05-31T20:52:10.268 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:52:10 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:AsfZpkCHwBktDRme3a7R2Db/dcg=', 'Content-Length': '0'} 2023-05-31T20:52:10.270 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:43:07 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ebc5672b2ade9767-006477b37a-1c17-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:52:10 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:52:10.271 DEBUG:boto:path=/pklxkd-59/obj17 2023-05-31T20:52:10.271 DEBUG:boto:auth_path=/pklxkd-59/obj17 2023-05-31T20:52:10.272 DEBUG:boto:path=/pklxkd-59/obj17?versionId=null 2023-05-31T20:52:10.272 DEBUG:boto:auth_path=/pklxkd-59/obj17?versionId=null 2023-05-31T20:52:10.272 DEBUG:boto:Method: GET 2023-05-31T20:52:10.273 DEBUG:boto:Path: /pklxkd-59/obj17?versionId=null 2023-05-31T20:52:10.273 DEBUG:boto:Data: 2023-05-31T20:52:10.273 DEBUG:boto:Headers: {} 2023-05-31T20:52:10.274 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:52:10.274 DEBUG:boto:Port: 8000 2023-05-31T20:52:10.274 DEBUG:boto:Params: {} 2023-05-31T20:52:10.275 DEBUG:boto:Token: None 2023-05-31T20:52:10.276 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:52:10 GMT /pklxkd-59/obj17?versionId=null 2023-05-31T20:52:10.276 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:AsfZpkCHwBktDRme3a7R2Db/dcg= 2023-05-31T20:52:10.277 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:52:10 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:AsfZpkCHwBktDRme3a7R2Db/dcg=', 'Content-Length': '0'} 2023-05-31T20:52:10.279 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:43:07 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e4f4ad2b635acc69-006477b37a-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:52:10 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:52:10.279 DEBUG:boto:path=/pklxkd-59/ 2023-05-31T20:52:10.280 DEBUG:boto:auth_path=/pklxkd-59/ 2023-05-31T20:52:10.280 DEBUG:boto:path=/pklxkd-59/?prefix=obj17 2023-05-31T20:52:10.281 DEBUG:boto:auth_path=/pklxkd-59/?prefix=obj17 2023-05-31T20:52:10.281 DEBUG:boto:Method: GET 2023-05-31T20:52:10.281 DEBUG:boto:Path: /pklxkd-59/?prefix=obj17 2023-05-31T20:52:10.282 DEBUG:boto:Data: 2023-05-31T20:52:10.282 DEBUG:boto:Headers: {} 2023-05-31T20:52:10.282 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:52:10.283 DEBUG:boto:Port: 8001 2023-05-31T20:52:10.283 DEBUG:boto:Params: {} 2023-05-31T20:52:10.284 DEBUG:boto:Token: None 2023-05-31T20:52:10.284 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:52:10 GMT /pklxkd-59/ 2023-05-31T20:52:10.284 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:1aCZ7BhkC9nnuJr6CPizGBU2/jg= 2023-05-31T20:52:10.285 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:52:10 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:1aCZ7BhkC9nnuJr6CPizGBU2/jg=', 'Content-Length': '0'} 2023-05-31T20:52:10.287 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000cff90f7e10ef5a3c-006477b37a-1c17-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:52:10 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:52:10.288 DEBUG:boto:b'pklxkd-59obj171000falseobj172023-05-31T20:43:07.977Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:52:10.289 DEBUG:boto:path=/pklxkd-59/ 2023-05-31T20:52:10.289 DEBUG:boto:auth_path=/pklxkd-59/ 2023-05-31T20:52:10.289 DEBUG:boto:path=/pklxkd-59/?prefix=obj17 2023-05-31T20:52:10.290 DEBUG:boto:auth_path=/pklxkd-59/?prefix=obj17 2023-05-31T20:52:10.290 DEBUG:boto:Method: GET 2023-05-31T20:52:10.291 DEBUG:boto:Path: /pklxkd-59/?prefix=obj17 2023-05-31T20:52:10.291 DEBUG:boto:Data: 2023-05-31T20:52:10.291 DEBUG:boto:Headers: {} 2023-05-31T20:52:10.292 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:52:10.292 DEBUG:boto:Port: 8000 2023-05-31T20:52:10.293 DEBUG:boto:Params: {} 2023-05-31T20:52:10.293 DEBUG:boto:Token: None 2023-05-31T20:52:10.293 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:52:10 GMT /pklxkd-59/ 2023-05-31T20:52:10.294 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:1aCZ7BhkC9nnuJr6CPizGBU2/jg= 2023-05-31T20:52:10.294 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:52:10 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:1aCZ7BhkC9nnuJr6CPizGBU2/jg=', 'Content-Length': '0'} 2023-05-31T20:52:10.298 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000edb72d3ff31e9a78-006477b37a-1630-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:52:10 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:52:10.298 DEBUG:boto:b'pklxkd-59obj171000falseobj172023-05-31T20:43:07.977Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:52:10.299 DEBUG:boto:path=/pklxkd-59/obj17 2023-05-31T20:52:10.299 DEBUG:boto:auth_path=/pklxkd-59/obj17 2023-05-31T20:52:10.300 DEBUG:boto:Method: HEAD 2023-05-31T20:52:10.300 DEBUG:boto:Path: /pklxkd-59/obj17 2023-05-31T20:52:10.300 DEBUG:boto:Data: 2023-05-31T20:52:10.301 DEBUG:boto:Headers: {} 2023-05-31T20:52:10.301 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:52:10.301 DEBUG:boto:Port: 8001 2023-05-31T20:52:10.302 DEBUG:boto:Params: {} 2023-05-31T20:52:10.302 DEBUG:boto:Token: None 2023-05-31T20:52:10.303 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:52:10 GMT /pklxkd-59/obj17 2023-05-31T20:52:10.303 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:vduuQuRup4kYc134nEs/XJeqJc0= 2023-05-31T20:52:10.303 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:52:10 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:vduuQuRup4kYc134nEs/XJeqJc0=', 'Content-Length': '0'} 2023-05-31T20:52:10.305 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:43:07 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000444902ad197b1cf-006477b37a-1c17-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:52:10 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:52:10.306 DEBUG:boto:path=/pklxkd-59/obj17 2023-05-31T20:52:10.306 DEBUG:boto:auth_path=/pklxkd-59/obj17 2023-05-31T20:52:10.306 DEBUG:boto:Method: HEAD 2023-05-31T20:52:10.307 DEBUG:boto:Path: /pklxkd-59/obj17 2023-05-31T20:52:10.307 DEBUG:boto:Data: 2023-05-31T20:52:10.307 DEBUG:boto:Headers: {} 2023-05-31T20:52:10.308 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:52:10.308 DEBUG:boto:Port: 8000 2023-05-31T20:52:10.308 DEBUG:boto:Params: {} 2023-05-31T20:52:10.309 DEBUG:boto:Token: None 2023-05-31T20:52:10.309 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:52:10 GMT /pklxkd-59/obj17 2023-05-31T20:52:10.310 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:vduuQuRup4kYc134nEs/XJeqJc0= 2023-05-31T20:52:10.310 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:52:10 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:vduuQuRup4kYc134nEs/XJeqJc0=', 'Content-Length': '0'} 2023-05-31T20:52:10.312 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:43:07 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000046e4b34905db7d6a-006477b37a-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:52:10 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:52:10.312 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj17 2023-05-31T20:52:10.313 DEBUG:boto:path=/pklxkd-59/obj17 2023-05-31T20:52:10.313 DEBUG:boto:auth_path=/pklxkd-59/obj17 2023-05-31T20:52:10.314 DEBUG:boto:Method: GET 2023-05-31T20:52:10.314 DEBUG:boto:Path: /pklxkd-59/obj17 2023-05-31T20:52:10.314 DEBUG:boto:Data: 2023-05-31T20:52:10.315 DEBUG:boto:Headers: {} 2023-05-31T20:52:10.315 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:52:10.315 DEBUG:boto:Port: 8001 2023-05-31T20:52:10.316 DEBUG:boto:Params: {} 2023-05-31T20:52:10.316 DEBUG:boto:Token: None 2023-05-31T20:52:10.317 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:52:10 GMT /pklxkd-59/obj17 2023-05-31T20:52:10.317 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:zoEcvzc00Je+uPv4Wy3uNFmpXBU= 2023-05-31T20:52:10.317 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:52:10 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:zoEcvzc00Je+uPv4Wy3uNFmpXBU=', 'Content-Length': '0'} 2023-05-31T20:52:10.320 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:43:07 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b30c2ba27bdb8951-006477b37a-1c17-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:52:10 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:52:10.320 DEBUG:boto:path=/pklxkd-59/obj17 2023-05-31T20:52:10.321 DEBUG:boto:auth_path=/pklxkd-59/obj17 2023-05-31T20:52:10.321 DEBUG:boto:Method: GET 2023-05-31T20:52:10.321 DEBUG:boto:Path: /pklxkd-59/obj17 2023-05-31T20:52:10.322 DEBUG:boto:Data: 2023-05-31T20:52:10.322 DEBUG:boto:Headers: {} 2023-05-31T20:52:10.323 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:52:10.323 DEBUG:boto:Port: 8000 2023-05-31T20:52:10.323 DEBUG:boto:Params: {} 2023-05-31T20:52:10.324 DEBUG:boto:Token: None 2023-05-31T20:52:10.324 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:52:10 GMT /pklxkd-59/obj17 2023-05-31T20:52:10.325 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:zoEcvzc00Je+uPv4Wy3uNFmpXBU= 2023-05-31T20:52:10.325 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:52:10 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:zoEcvzc00Je+uPv4Wy3uNFmpXBU=', 'Content-Length': '0'} 2023-05-31T20:52:10.327 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:43:07 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f933acee7a71da31-006477b37a-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:52:10 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:52:10.327 DEBUG:boto:path=/pklxkd-59/ 2023-05-31T20:52:10.328 DEBUG:boto:auth_path=/pklxkd-59/ 2023-05-31T20:52:10.328 DEBUG:boto:path=/pklxkd-59/?prefix=obj17 2023-05-31T20:52:10.329 DEBUG:boto:auth_path=/pklxkd-59/?prefix=obj17 2023-05-31T20:52:10.329 DEBUG:boto:Method: GET 2023-05-31T20:52:10.329 DEBUG:boto:Path: /pklxkd-59/?prefix=obj17 2023-05-31T20:52:10.330 DEBUG:boto:Data: 2023-05-31T20:52:10.330 DEBUG:boto:Headers: {} 2023-05-31T20:52:10.330 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:52:10.331 DEBUG:boto:Port: 8001 2023-05-31T20:52:10.331 DEBUG:boto:Params: {} 2023-05-31T20:52:10.331 DEBUG:boto:Token: None 2023-05-31T20:52:10.332 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:52:10 GMT /pklxkd-59/ 2023-05-31T20:52:10.332 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:1aCZ7BhkC9nnuJr6CPizGBU2/jg= 2023-05-31T20:52:10.333 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:52:10 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:1aCZ7BhkC9nnuJr6CPizGBU2/jg=', 'Content-Length': '0'} 2023-05-31T20:52:10.335 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000009e907c1cc5492592-006477b37a-1c17-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:52:10 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:52:10.336 DEBUG:boto:b'pklxkd-59obj171000falseobj172023-05-31T20:43:07.977Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:52:10.336 DEBUG:boto:path=/pklxkd-59/ 2023-05-31T20:52:10.337 DEBUG:boto:auth_path=/pklxkd-59/ 2023-05-31T20:52:10.337 DEBUG:boto:path=/pklxkd-59/?prefix=obj17 2023-05-31T20:52:10.337 DEBUG:boto:auth_path=/pklxkd-59/?prefix=obj17 2023-05-31T20:52:10.338 DEBUG:boto:Method: GET 2023-05-31T20:52:10.338 DEBUG:boto:Path: /pklxkd-59/?prefix=obj17 2023-05-31T20:52:10.339 DEBUG:boto:Data: 2023-05-31T20:52:10.339 DEBUG:boto:Headers: {} 2023-05-31T20:52:10.339 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:52:10.340 DEBUG:boto:Port: 8000 2023-05-31T20:52:10.340 DEBUG:boto:Params: {} 2023-05-31T20:52:10.340 DEBUG:boto:Token: None 2023-05-31T20:52:10.341 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:52:10 GMT /pklxkd-59/ 2023-05-31T20:52:10.341 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:1aCZ7BhkC9nnuJr6CPizGBU2/jg= 2023-05-31T20:52:10.342 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:52:10 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:1aCZ7BhkC9nnuJr6CPizGBU2/jg=', 'Content-Length': '0'} 2023-05-31T20:52:10.345 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a3747372d6bfab72-006477b37a-1630-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:52:10 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:52:10.346 DEBUG:boto:b'pklxkd-59obj171000falseobj172023-05-31T20:43:07.977Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:52:10.346 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj19 2023-05-31T20:52:10.347 DEBUG:boto:path=/pklxkd-59/obj19 2023-05-31T20:52:10.347 DEBUG:boto:auth_path=/pklxkd-59/obj19 2023-05-31T20:52:10.347 DEBUG:boto:path=/pklxkd-59/obj19?versionId=null 2023-05-31T20:52:10.348 DEBUG:boto:auth_path=/pklxkd-59/obj19?versionId=null 2023-05-31T20:52:10.348 DEBUG:boto:Method: GET 2023-05-31T20:52:10.348 DEBUG:boto:Path: /pklxkd-59/obj19?versionId=null 2023-05-31T20:52:10.349 DEBUG:boto:Data: 2023-05-31T20:52:10.349 DEBUG:boto:Headers: {} 2023-05-31T20:52:10.350 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:52:10.350 DEBUG:boto:Port: 8001 2023-05-31T20:52:10.350 DEBUG:boto:Params: {} 2023-05-31T20:52:10.351 DEBUG:boto:Token: None 2023-05-31T20:52:10.351 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:52:10 GMT /pklxkd-59/obj19?versionId=null 2023-05-31T20:52:10.351 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:UM2PtRraVICmnFNeA7uDtfxR2+o= 2023-05-31T20:52:10.352 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:52:10 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:UM2PtRraVICmnFNeA7uDtfxR2+o=', 'Content-Length': '0'} 2023-05-31T20:52:10.354 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:44:50 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007c0a623e2b48290b-006477b37a-1c17-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:52:10 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:52:10.354 DEBUG:boto:path=/pklxkd-59/obj19 2023-05-31T20:52:10.355 DEBUG:boto:auth_path=/pklxkd-59/obj19 2023-05-31T20:52:10.355 DEBUG:boto:path=/pklxkd-59/obj19?versionId=null 2023-05-31T20:52:10.355 DEBUG:boto:auth_path=/pklxkd-59/obj19?versionId=null 2023-05-31T20:52:10.356 DEBUG:boto:Method: GET 2023-05-31T20:52:10.356 DEBUG:boto:Path: /pklxkd-59/obj19?versionId=null 2023-05-31T20:52:10.357 DEBUG:boto:Data: 2023-05-31T20:52:10.357 DEBUG:boto:Headers: {} 2023-05-31T20:52:10.357 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:52:10.358 DEBUG:boto:Port: 8000 2023-05-31T20:52:10.358 DEBUG:boto:Params: {} 2023-05-31T20:52:10.358 DEBUG:boto:Token: None 2023-05-31T20:52:10.359 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:52:10 GMT /pklxkd-59/obj19?versionId=null 2023-05-31T20:52:10.359 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:UM2PtRraVICmnFNeA7uDtfxR2+o= 2023-05-31T20:52:10.360 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:52:10 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:UM2PtRraVICmnFNeA7uDtfxR2+o=', 'Content-Length': '0'} 2023-05-31T20:52:10.362 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:44:50 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000009a9bbd826301fff-006477b37a-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:52:10 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:52:10.404 DEBUG:boto:path=/pklxkd-59/obj19 2023-05-31T20:52:10.405 DEBUG:boto:auth_path=/pklxkd-59/obj19 2023-05-31T20:52:10.405 DEBUG:boto:path=/pklxkd-59/obj19?versionId=null 2023-05-31T20:52:10.405 DEBUG:boto:auth_path=/pklxkd-59/obj19?versionId=null 2023-05-31T20:52:10.406 DEBUG:boto:Method: HEAD 2023-05-31T20:52:10.406 DEBUG:boto:Path: /pklxkd-59/obj19?versionId=null 2023-05-31T20:52:10.406 DEBUG:boto:Data: 2023-05-31T20:52:10.407 DEBUG:boto:Headers: {} 2023-05-31T20:52:10.407 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:52:10.407 DEBUG:boto:Port: 8001 2023-05-31T20:52:10.408 DEBUG:boto:Params: {} 2023-05-31T20:52:10.408 DEBUG:boto:Token: None 2023-05-31T20:52:10.409 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:52:10 GMT /pklxkd-59/obj19?versionId=null 2023-05-31T20:52:10.409 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:p7+BqgO6390+WpZzU1FKTrxZjcs= 2023-05-31T20:52:10.409 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:52:10 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:p7+BqgO6390+WpZzU1FKTrxZjcs=', 'Content-Length': '0'} 2023-05-31T20:52:10.411 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:44:50 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000045cacb0af286a05d-006477b37a-1c17-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:52:10 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:52:10.412 DEBUG:boto:path=/pklxkd-59/obj19 2023-05-31T20:52:10.412 DEBUG:boto:auth_path=/pklxkd-59/obj19 2023-05-31T20:52:10.413 DEBUG:boto:path=/pklxkd-59/obj19?versionId=null 2023-05-31T20:52:10.413 DEBUG:boto:auth_path=/pklxkd-59/obj19?versionId=null 2023-05-31T20:52:10.413 DEBUG:boto:Method: HEAD 2023-05-31T20:52:10.414 DEBUG:boto:Path: /pklxkd-59/obj19?versionId=null 2023-05-31T20:52:10.414 DEBUG:boto:Data: 2023-05-31T20:52:10.414 DEBUG:boto:Headers: {} 2023-05-31T20:52:10.415 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:52:10.415 DEBUG:boto:Port: 8000 2023-05-31T20:52:10.415 DEBUG:boto:Params: {} 2023-05-31T20:52:10.416 DEBUG:boto:Token: None 2023-05-31T20:52:10.416 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:52:10 GMT /pklxkd-59/obj19?versionId=null 2023-05-31T20:52:10.417 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:p7+BqgO6390+WpZzU1FKTrxZjcs= 2023-05-31T20:52:10.417 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:52:10 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:p7+BqgO6390+WpZzU1FKTrxZjcs=', 'Content-Length': '0'} 2023-05-31T20:52:10.419 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:44:50 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000009481b4714cd82b1-006477b37a-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:52:10 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:52:10.419 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj19 2023-05-31T20:52:10.420 DEBUG:boto:path=/pklxkd-59/obj19 2023-05-31T20:52:10.420 DEBUG:boto:auth_path=/pklxkd-59/obj19 2023-05-31T20:52:10.420 DEBUG:boto:path=/pklxkd-59/obj19?versionId=null 2023-05-31T20:52:10.421 DEBUG:boto:auth_path=/pklxkd-59/obj19?versionId=null 2023-05-31T20:52:10.421 DEBUG:boto:Method: GET 2023-05-31T20:52:10.421 DEBUG:boto:Path: /pklxkd-59/obj19?versionId=null 2023-05-31T20:52:10.422 DEBUG:boto:Data: 2023-05-31T20:52:10.422 DEBUG:boto:Headers: {} 2023-05-31T20:52:10.423 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:52:10.423 DEBUG:boto:Port: 8001 2023-05-31T20:52:10.423 DEBUG:boto:Params: {} 2023-05-31T20:52:10.424 DEBUG:boto:Token: None 2023-05-31T20:52:10.424 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:52:10 GMT /pklxkd-59/obj19?versionId=null 2023-05-31T20:52:10.424 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:UM2PtRraVICmnFNeA7uDtfxR2+o= 2023-05-31T20:52:10.425 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:52:10 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:UM2PtRraVICmnFNeA7uDtfxR2+o=', 'Content-Length': '0'} 2023-05-31T20:52:10.427 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:44:50 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000415051cafb13cf45-006477b37a-1c17-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:52:10 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:52:10.428 DEBUG:boto:path=/pklxkd-59/obj19 2023-05-31T20:52:10.428 DEBUG:boto:auth_path=/pklxkd-59/obj19 2023-05-31T20:52:10.428 DEBUG:boto:path=/pklxkd-59/obj19?versionId=null 2023-05-31T20:52:10.429 DEBUG:boto:auth_path=/pklxkd-59/obj19?versionId=null 2023-05-31T20:52:10.429 DEBUG:boto:Method: GET 2023-05-31T20:52:10.430 DEBUG:boto:Path: /pklxkd-59/obj19?versionId=null 2023-05-31T20:52:10.430 DEBUG:boto:Data: 2023-05-31T20:52:10.430 DEBUG:boto:Headers: {} 2023-05-31T20:52:10.431 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:52:10.431 DEBUG:boto:Port: 8000 2023-05-31T20:52:10.431 DEBUG:boto:Params: {} 2023-05-31T20:52:10.432 DEBUG:boto:Token: None 2023-05-31T20:52:10.432 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:52:10 GMT /pklxkd-59/obj19?versionId=null 2023-05-31T20:52:10.433 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:UM2PtRraVICmnFNeA7uDtfxR2+o= 2023-05-31T20:52:10.433 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:52:10 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:UM2PtRraVICmnFNeA7uDtfxR2+o=', 'Content-Length': '0'} 2023-05-31T20:52:10.435 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:44:50 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000087e1bda24b4d0dbf-006477b37a-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:52:10 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:52:10.435 DEBUG:boto:path=/pklxkd-59/ 2023-05-31T20:52:10.436 DEBUG:boto:auth_path=/pklxkd-59/ 2023-05-31T20:52:10.436 DEBUG:boto:path=/pklxkd-59/?prefix=obj19 2023-05-31T20:52:10.436 DEBUG:boto:auth_path=/pklxkd-59/?prefix=obj19 2023-05-31T20:52:10.437 DEBUG:boto:Method: GET 2023-05-31T20:52:10.437 DEBUG:boto:Path: /pklxkd-59/?prefix=obj19 2023-05-31T20:52:10.438 DEBUG:boto:Data: 2023-05-31T20:52:10.438 DEBUG:boto:Headers: {} 2023-05-31T20:52:10.438 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:52:10.439 DEBUG:boto:Port: 8001 2023-05-31T20:52:10.439 DEBUG:boto:Params: {} 2023-05-31T20:52:10.439 DEBUG:boto:Token: None 2023-05-31T20:52:10.440 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:52:10 GMT /pklxkd-59/ 2023-05-31T20:52:10.440 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:1aCZ7BhkC9nnuJr6CPizGBU2/jg= 2023-05-31T20:52:10.441 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:52:10 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:1aCZ7BhkC9nnuJr6CPizGBU2/jg=', 'Content-Length': '0'} 2023-05-31T20:52:10.443 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000001294adf4f3fefc87-006477b37a-1c17-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:52:10 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:52:10.444 DEBUG:boto:b'pklxkd-59obj191000falseobj192023-05-31T20:44:50.557Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:52:10.444 DEBUG:boto:path=/pklxkd-59/ 2023-05-31T20:52:10.445 DEBUG:boto:auth_path=/pklxkd-59/ 2023-05-31T20:52:10.445 DEBUG:boto:path=/pklxkd-59/?prefix=obj19 2023-05-31T20:52:10.445 DEBUG:boto:auth_path=/pklxkd-59/?prefix=obj19 2023-05-31T20:52:10.446 DEBUG:boto:Method: GET 2023-05-31T20:52:10.446 DEBUG:boto:Path: /pklxkd-59/?prefix=obj19 2023-05-31T20:52:10.446 DEBUG:boto:Data: 2023-05-31T20:52:10.447 DEBUG:boto:Headers: {} 2023-05-31T20:52:10.447 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:52:10.447 DEBUG:boto:Port: 8000 2023-05-31T20:52:10.448 DEBUG:boto:Params: {} 2023-05-31T20:52:10.448 DEBUG:boto:Token: None 2023-05-31T20:52:10.449 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:52:10 GMT /pklxkd-59/ 2023-05-31T20:52:10.449 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:1aCZ7BhkC9nnuJr6CPizGBU2/jg= 2023-05-31T20:52:10.449 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:52:10 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:1aCZ7BhkC9nnuJr6CPizGBU2/jg=', 'Content-Length': '0'} 2023-05-31T20:52:10.453 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000009627a1d1af46ea8-006477b37a-1630-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:52:10 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:52:10.454 DEBUG:boto:b'pklxkd-59obj191000falseobj192023-05-31T20:44:50.557Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:52:10.454 DEBUG:boto:path=/pklxkd-59/obj19 2023-05-31T20:52:10.455 DEBUG:boto:auth_path=/pklxkd-59/obj19 2023-05-31T20:52:10.455 DEBUG:boto:Method: HEAD 2023-05-31T20:52:10.455 DEBUG:boto:Path: /pklxkd-59/obj19 2023-05-31T20:52:10.456 DEBUG:boto:Data: 2023-05-31T20:52:10.456 DEBUG:boto:Headers: {} 2023-05-31T20:52:10.456 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:52:10.457 DEBUG:boto:Port: 8001 2023-05-31T20:52:10.457 DEBUG:boto:Params: {} 2023-05-31T20:52:10.457 DEBUG:boto:Token: None 2023-05-31T20:52:10.458 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:52:10 GMT /pklxkd-59/obj19 2023-05-31T20:52:10.458 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:i8XFrmoanBb0/mMLIoo58iQ7Ncg= 2023-05-31T20:52:10.459 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:52:10 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:i8XFrmoanBb0/mMLIoo58iQ7Ncg=', 'Content-Length': '0'} 2023-05-31T20:52:10.460 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:44:50 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009d3a6cc7588ff4fc-006477b37a-1c17-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:52:10 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:52:10.461 DEBUG:boto:path=/pklxkd-59/obj19 2023-05-31T20:52:10.461 DEBUG:boto:auth_path=/pklxkd-59/obj19 2023-05-31T20:52:10.462 DEBUG:boto:Method: HEAD 2023-05-31T20:52:10.462 DEBUG:boto:Path: /pklxkd-59/obj19 2023-05-31T20:52:10.462 DEBUG:boto:Data: 2023-05-31T20:52:10.463 DEBUG:boto:Headers: {} 2023-05-31T20:52:10.463 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:52:10.463 DEBUG:boto:Port: 8000 2023-05-31T20:52:10.464 DEBUG:boto:Params: {} 2023-05-31T20:52:10.464 DEBUG:boto:Token: None 2023-05-31T20:52:10.464 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:52:10 GMT /pklxkd-59/obj19 2023-05-31T20:52:10.465 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:i8XFrmoanBb0/mMLIoo58iQ7Ncg= 2023-05-31T20:52:10.465 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:52:10 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:i8XFrmoanBb0/mMLIoo58iQ7Ncg=', 'Content-Length': '0'} 2023-05-31T20:52:10.467 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:44:50 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a7633b70aa1c78e7-006477b37a-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:52:10 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:52:10.468 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj19 2023-05-31T20:52:10.468 DEBUG:boto:path=/pklxkd-59/obj19 2023-05-31T20:52:10.468 DEBUG:boto:auth_path=/pklxkd-59/obj19 2023-05-31T20:52:10.469 DEBUG:boto:Method: GET 2023-05-31T20:52:10.469 DEBUG:boto:Path: /pklxkd-59/obj19 2023-05-31T20:52:10.470 DEBUG:boto:Data: 2023-05-31T20:52:10.470 DEBUG:boto:Headers: {} 2023-05-31T20:52:10.470 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:52:10.471 DEBUG:boto:Port: 8001 2023-05-31T20:52:10.471 DEBUG:boto:Params: {} 2023-05-31T20:52:10.471 DEBUG:boto:Token: None 2023-05-31T20:52:10.472 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:52:10 GMT /pklxkd-59/obj19 2023-05-31T20:52:10.472 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:vh/NqRXBQ1MfZMV8WUxZwg02TEA= 2023-05-31T20:52:10.473 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:52:10 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:vh/NqRXBQ1MfZMV8WUxZwg02TEA=', 'Content-Length': '0'} 2023-05-31T20:52:10.474 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:44:50 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000113c1ba4bd85c006-006477b37a-1c17-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:52:10 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:52:10.516 DEBUG:boto:path=/pklxkd-59/obj19 2023-05-31T20:52:10.517 DEBUG:boto:auth_path=/pklxkd-59/obj19 2023-05-31T20:52:10.517 DEBUG:boto:Method: GET 2023-05-31T20:52:10.518 DEBUG:boto:Path: /pklxkd-59/obj19 2023-05-31T20:52:10.518 DEBUG:boto:Data: 2023-05-31T20:52:10.519 DEBUG:boto:Headers: {} 2023-05-31T20:52:10.519 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:52:10.519 DEBUG:boto:Port: 8000 2023-05-31T20:52:10.520 DEBUG:boto:Params: {} 2023-05-31T20:52:10.520 DEBUG:boto:Token: None 2023-05-31T20:52:10.521 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:52:10 GMT /pklxkd-59/obj19 2023-05-31T20:52:10.521 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:vh/NqRXBQ1MfZMV8WUxZwg02TEA= 2023-05-31T20:52:10.522 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:52:10 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:vh/NqRXBQ1MfZMV8WUxZwg02TEA=', 'Content-Length': '0'} 2023-05-31T20:52:10.523 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:44:50 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003b490bfe8441732c-006477b37a-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:52:10 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:52:10.524 DEBUG:boto:path=/pklxkd-59/ 2023-05-31T20:52:10.525 DEBUG:boto:auth_path=/pklxkd-59/ 2023-05-31T20:52:10.525 DEBUG:boto:path=/pklxkd-59/?prefix=obj19 2023-05-31T20:52:10.525 DEBUG:boto:auth_path=/pklxkd-59/?prefix=obj19 2023-05-31T20:52:10.526 DEBUG:boto:Method: GET 2023-05-31T20:52:10.526 DEBUG:boto:Path: /pklxkd-59/?prefix=obj19 2023-05-31T20:52:10.527 DEBUG:boto:Data: 2023-05-31T20:52:10.527 DEBUG:boto:Headers: {} 2023-05-31T20:52:10.527 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:52:10.528 DEBUG:boto:Port: 8001 2023-05-31T20:52:10.528 DEBUG:boto:Params: {} 2023-05-31T20:52:10.529 DEBUG:boto:Token: None 2023-05-31T20:52:10.529 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:52:10 GMT /pklxkd-59/ 2023-05-31T20:52:10.529 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:1aCZ7BhkC9nnuJr6CPizGBU2/jg= 2023-05-31T20:52:10.530 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:52:10 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:1aCZ7BhkC9nnuJr6CPizGBU2/jg=', 'Content-Length': '0'} 2023-05-31T20:52:10.532 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000001fd0e2e4167ce5d6-006477b37a-1c17-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:52:10 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:52:10.533 DEBUG:boto:b'pklxkd-59obj191000falseobj192023-05-31T20:44:50.557Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:52:10.534 DEBUG:boto:path=/pklxkd-59/ 2023-05-31T20:52:10.534 DEBUG:boto:auth_path=/pklxkd-59/ 2023-05-31T20:52:10.534 DEBUG:boto:path=/pklxkd-59/?prefix=obj19 2023-05-31T20:52:10.535 DEBUG:boto:auth_path=/pklxkd-59/?prefix=obj19 2023-05-31T20:52:10.535 DEBUG:boto:Method: GET 2023-05-31T20:52:10.536 DEBUG:boto:Path: /pklxkd-59/?prefix=obj19 2023-05-31T20:52:10.536 DEBUG:boto:Data: 2023-05-31T20:52:10.536 DEBUG:boto:Headers: {} 2023-05-31T20:52:10.537 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:52:10.537 DEBUG:boto:Port: 8000 2023-05-31T20:52:10.537 DEBUG:boto:Params: {} 2023-05-31T20:52:10.538 DEBUG:boto:Token: None 2023-05-31T20:52:10.538 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:52:10 GMT /pklxkd-59/ 2023-05-31T20:52:10.539 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:1aCZ7BhkC9nnuJr6CPizGBU2/jg= 2023-05-31T20:52:10.539 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:52:10 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:1aCZ7BhkC9nnuJr6CPizGBU2/jg=', 'Content-Length': '0'} 2023-05-31T20:52:10.543 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a86966b99e0b580e-006477b37a-1630-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:52:10 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:52:10.544 DEBUG:boto:b'pklxkd-59obj191000falseobj192023-05-31T20:44:50.557Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:52:10.544 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj22 2023-05-31T20:52:10.545 DEBUG:boto:path=/pklxkd-59/obj22 2023-05-31T20:52:10.545 DEBUG:boto:auth_path=/pklxkd-59/obj22 2023-05-31T20:52:10.546 DEBUG:boto:path=/pklxkd-59/obj22?versionId=null 2023-05-31T20:52:10.546 DEBUG:boto:auth_path=/pklxkd-59/obj22?versionId=null 2023-05-31T20:52:10.546 DEBUG:boto:Method: GET 2023-05-31T20:52:10.547 DEBUG:boto:Path: /pklxkd-59/obj22?versionId=null 2023-05-31T20:52:10.547 DEBUG:boto:Data: 2023-05-31T20:52:10.548 DEBUG:boto:Headers: {} 2023-05-31T20:52:10.548 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:52:10.549 DEBUG:boto:Port: 8001 2023-05-31T20:52:10.549 DEBUG:boto:Params: {} 2023-05-31T20:52:10.549 DEBUG:boto:Token: None 2023-05-31T20:52:10.550 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:52:10 GMT /pklxkd-59/obj22?versionId=null 2023-05-31T20:52:10.550 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:c4THCJHpQcv/zOcPR+VsW8hz2yg= 2023-05-31T20:52:10.551 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:52:10 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:c4THCJHpQcv/zOcPR+VsW8hz2yg=', 'Content-Length': '0'} 2023-05-31T20:52:10.553 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:39:26 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d2f96a331d7434a2-006477b37a-1c17-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:52:10 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:52:10.553 DEBUG:boto:path=/pklxkd-59/obj22 2023-05-31T20:52:10.554 DEBUG:boto:auth_path=/pklxkd-59/obj22 2023-05-31T20:52:10.554 DEBUG:boto:path=/pklxkd-59/obj22?versionId=null 2023-05-31T20:52:10.554 DEBUG:boto:auth_path=/pklxkd-59/obj22?versionId=null 2023-05-31T20:52:10.555 DEBUG:boto:Method: GET 2023-05-31T20:52:10.555 DEBUG:boto:Path: /pklxkd-59/obj22?versionId=null 2023-05-31T20:52:10.555 DEBUG:boto:Data: 2023-05-31T20:52:10.556 DEBUG:boto:Headers: {} 2023-05-31T20:52:10.556 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:52:10.556 DEBUG:boto:Port: 8000 2023-05-31T20:52:10.557 DEBUG:boto:Params: {} 2023-05-31T20:52:10.557 DEBUG:boto:Token: None 2023-05-31T20:52:10.558 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:52:10 GMT /pklxkd-59/obj22?versionId=null 2023-05-31T20:52:10.558 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:c4THCJHpQcv/zOcPR+VsW8hz2yg= 2023-05-31T20:52:10.558 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:52:10 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:c4THCJHpQcv/zOcPR+VsW8hz2yg=', 'Content-Length': '0'} 2023-05-31T20:52:10.560 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:39:26 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000cb01da34a1348c7-006477b37a-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:52:10 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:52:10.604 DEBUG:boto:path=/pklxkd-59/obj22 2023-05-31T20:52:10.605 DEBUG:boto:auth_path=/pklxkd-59/obj22 2023-05-31T20:52:10.605 DEBUG:boto:path=/pklxkd-59/obj22?versionId=null 2023-05-31T20:52:10.605 DEBUG:boto:auth_path=/pklxkd-59/obj22?versionId=null 2023-05-31T20:52:10.606 DEBUG:boto:Method: HEAD 2023-05-31T20:52:10.606 DEBUG:boto:Path: /pklxkd-59/obj22?versionId=null 2023-05-31T20:52:10.606 DEBUG:boto:Data: 2023-05-31T20:52:10.607 DEBUG:boto:Headers: {} 2023-05-31T20:52:10.607 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:52:10.607 DEBUG:boto:Port: 8001 2023-05-31T20:52:10.608 DEBUG:boto:Params: {} 2023-05-31T20:52:10.608 DEBUG:boto:Token: None 2023-05-31T20:52:10.608 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:52:10 GMT /pklxkd-59/obj22?versionId=null 2023-05-31T20:52:10.609 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:QVPBRsG0cMqcMSlDots7lghEn5E= 2023-05-31T20:52:10.609 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:52:10 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:QVPBRsG0cMqcMSlDots7lghEn5E=', 'Content-Length': '0'} 2023-05-31T20:52:10.611 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:39:26 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c13736da9b0b0156-006477b37a-1c17-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:52:10 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:52:10.612 DEBUG:boto:path=/pklxkd-59/obj22 2023-05-31T20:52:10.612 DEBUG:boto:auth_path=/pklxkd-59/obj22 2023-05-31T20:52:10.612 DEBUG:boto:path=/pklxkd-59/obj22?versionId=null 2023-05-31T20:52:10.613 DEBUG:boto:auth_path=/pklxkd-59/obj22?versionId=null 2023-05-31T20:52:10.613 DEBUG:boto:Method: HEAD 2023-05-31T20:52:10.613 DEBUG:boto:Path: /pklxkd-59/obj22?versionId=null 2023-05-31T20:52:10.614 DEBUG:boto:Data: 2023-05-31T20:52:10.614 DEBUG:boto:Headers: {} 2023-05-31T20:52:10.614 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:52:10.615 DEBUG:boto:Port: 8000 2023-05-31T20:52:10.615 DEBUG:boto:Params: {} 2023-05-31T20:52:10.615 DEBUG:boto:Token: None 2023-05-31T20:52:10.616 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:52:10 GMT /pklxkd-59/obj22?versionId=null 2023-05-31T20:52:10.616 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:QVPBRsG0cMqcMSlDots7lghEn5E= 2023-05-31T20:52:10.617 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:52:10 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:QVPBRsG0cMqcMSlDots7lghEn5E=', 'Content-Length': '0'} 2023-05-31T20:52:10.618 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:39:26 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009ccd02e5bbc0f4dd-006477b37a-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:52:10 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:52:10.619 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj22 2023-05-31T20:52:10.619 DEBUG:boto:path=/pklxkd-59/obj22 2023-05-31T20:52:10.620 DEBUG:boto:auth_path=/pklxkd-59/obj22 2023-05-31T20:52:10.620 DEBUG:boto:path=/pklxkd-59/obj22?versionId=null 2023-05-31T20:52:10.620 DEBUG:boto:auth_path=/pklxkd-59/obj22?versionId=null 2023-05-31T20:52:10.621 DEBUG:boto:Method: GET 2023-05-31T20:52:10.621 DEBUG:boto:Path: /pklxkd-59/obj22?versionId=null 2023-05-31T20:52:10.621 DEBUG:boto:Data: 2023-05-31T20:52:10.622 DEBUG:boto:Headers: {} 2023-05-31T20:52:10.622 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:52:10.622 DEBUG:boto:Port: 8001 2023-05-31T20:52:10.623 DEBUG:boto:Params: {} 2023-05-31T20:52:10.623 DEBUG:boto:Token: None 2023-05-31T20:52:10.624 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:52:10 GMT /pklxkd-59/obj22?versionId=null 2023-05-31T20:52:10.624 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:c4THCJHpQcv/zOcPR+VsW8hz2yg= 2023-05-31T20:52:10.625 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:52:10 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:c4THCJHpQcv/zOcPR+VsW8hz2yg=', 'Content-Length': '0'} 2023-05-31T20:52:10.626 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:39:26 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e27c792329fba791-006477b37a-1c17-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:52:10 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:52:10.627 DEBUG:boto:path=/pklxkd-59/obj22 2023-05-31T20:52:10.628 DEBUG:boto:auth_path=/pklxkd-59/obj22 2023-05-31T20:52:10.628 DEBUG:boto:path=/pklxkd-59/obj22?versionId=null 2023-05-31T20:52:10.628 DEBUG:boto:auth_path=/pklxkd-59/obj22?versionId=null 2023-05-31T20:52:10.629 DEBUG:boto:Method: GET 2023-05-31T20:52:10.629 DEBUG:boto:Path: /pklxkd-59/obj22?versionId=null 2023-05-31T20:52:10.629 DEBUG:boto:Data: 2023-05-31T20:52:10.630 DEBUG:boto:Headers: {} 2023-05-31T20:52:10.630 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:52:10.630 DEBUG:boto:Port: 8000 2023-05-31T20:52:10.631 DEBUG:boto:Params: {} 2023-05-31T20:52:10.631 DEBUG:boto:Token: None 2023-05-31T20:52:10.632 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:52:10 GMT /pklxkd-59/obj22?versionId=null 2023-05-31T20:52:10.632 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:c4THCJHpQcv/zOcPR+VsW8hz2yg= 2023-05-31T20:52:10.632 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:52:10 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:c4THCJHpQcv/zOcPR+VsW8hz2yg=', 'Content-Length': '0'} 2023-05-31T20:52:10.634 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:39:26 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000df280eb7011f5b49-006477b37a-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:52:10 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:52:10.635 DEBUG:boto:path=/pklxkd-59/ 2023-05-31T20:52:10.635 DEBUG:boto:auth_path=/pklxkd-59/ 2023-05-31T20:52:10.636 DEBUG:boto:path=/pklxkd-59/?prefix=obj22 2023-05-31T20:52:10.636 DEBUG:boto:auth_path=/pklxkd-59/?prefix=obj22 2023-05-31T20:52:10.636 DEBUG:boto:Method: GET 2023-05-31T20:52:10.637 DEBUG:boto:Path: /pklxkd-59/?prefix=obj22 2023-05-31T20:52:10.637 DEBUG:boto:Data: 2023-05-31T20:52:10.638 DEBUG:boto:Headers: {} 2023-05-31T20:52:10.638 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:52:10.638 DEBUG:boto:Port: 8001 2023-05-31T20:52:10.639 DEBUG:boto:Params: {} 2023-05-31T20:52:10.639 DEBUG:boto:Token: None 2023-05-31T20:52:10.639 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:52:10 GMT /pklxkd-59/ 2023-05-31T20:52:10.640 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:1aCZ7BhkC9nnuJr6CPizGBU2/jg= 2023-05-31T20:52:10.640 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:52:10 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:1aCZ7BhkC9nnuJr6CPizGBU2/jg=', 'Content-Length': '0'} 2023-05-31T20:52:10.643 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000003895b413f66d071e-006477b37a-1c17-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:52:10 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:52:10.643 DEBUG:boto:b'pklxkd-59obj221000falseobj222023-05-31T20:39:26.473Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:52:10.644 DEBUG:boto:path=/pklxkd-59/ 2023-05-31T20:52:10.645 DEBUG:boto:auth_path=/pklxkd-59/ 2023-05-31T20:52:10.645 DEBUG:boto:path=/pklxkd-59/?prefix=obj22 2023-05-31T20:52:10.645 DEBUG:boto:auth_path=/pklxkd-59/?prefix=obj22 2023-05-31T20:52:10.646 DEBUG:boto:Method: GET 2023-05-31T20:52:10.646 DEBUG:boto:Path: /pklxkd-59/?prefix=obj22 2023-05-31T20:52:10.646 DEBUG:boto:Data: 2023-05-31T20:52:10.647 DEBUG:boto:Headers: {} 2023-05-31T20:52:10.647 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:52:10.647 DEBUG:boto:Port: 8000 2023-05-31T20:52:10.648 DEBUG:boto:Params: {} 2023-05-31T20:52:10.648 DEBUG:boto:Token: None 2023-05-31T20:52:10.649 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:52:10 GMT /pklxkd-59/ 2023-05-31T20:52:10.649 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:1aCZ7BhkC9nnuJr6CPizGBU2/jg= 2023-05-31T20:52:10.649 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:52:10 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:1aCZ7BhkC9nnuJr6CPizGBU2/jg=', 'Content-Length': '0'} 2023-05-31T20:52:10.653 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000dadebba53e6b5f55-006477b37a-1630-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:52:10 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:52:10.654 DEBUG:boto:b'pklxkd-59obj221000falseobj222023-05-31T20:39:26.473Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:52:10.654 DEBUG:boto:path=/pklxkd-59/obj22 2023-05-31T20:52:10.655 DEBUG:boto:auth_path=/pklxkd-59/obj22 2023-05-31T20:52:10.655 DEBUG:boto:Method: HEAD 2023-05-31T20:52:10.655 DEBUG:boto:Path: /pklxkd-59/obj22 2023-05-31T20:52:10.656 DEBUG:boto:Data: 2023-05-31T20:52:10.656 DEBUG:boto:Headers: {} 2023-05-31T20:52:10.657 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:52:10.657 DEBUG:boto:Port: 8001 2023-05-31T20:52:10.657 DEBUG:boto:Params: {} 2023-05-31T20:52:10.658 DEBUG:boto:Token: None 2023-05-31T20:52:10.658 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:52:10 GMT /pklxkd-59/obj22 2023-05-31T20:52:10.658 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:d8n9Xm74KgjQwbB7Gn9F1dtD1Cc= 2023-05-31T20:52:10.659 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:52:10 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:d8n9Xm74KgjQwbB7Gn9F1dtD1Cc=', 'Content-Length': '0'} 2023-05-31T20:52:10.661 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:39:26 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000b6dc1f8f7fcca59-006477b37a-1c17-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:52:10 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:52:10.661 DEBUG:boto:path=/pklxkd-59/obj22 2023-05-31T20:52:10.662 DEBUG:boto:auth_path=/pklxkd-59/obj22 2023-05-31T20:52:10.662 DEBUG:boto:Method: HEAD 2023-05-31T20:52:10.662 DEBUG:boto:Path: /pklxkd-59/obj22 2023-05-31T20:52:10.663 DEBUG:boto:Data: 2023-05-31T20:52:10.663 DEBUG:boto:Headers: {} 2023-05-31T20:52:10.663 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:52:10.664 DEBUG:boto:Port: 8000 2023-05-31T20:52:10.664 DEBUG:boto:Params: {} 2023-05-31T20:52:10.664 DEBUG:boto:Token: None 2023-05-31T20:52:10.665 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:52:10 GMT /pklxkd-59/obj22 2023-05-31T20:52:10.665 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:d8n9Xm74KgjQwbB7Gn9F1dtD1Cc= 2023-05-31T20:52:10.665 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:52:10 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:d8n9Xm74KgjQwbB7Gn9F1dtD1Cc=', 'Content-Length': '0'} 2023-05-31T20:52:10.667 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:39:26 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000000d19e11807ae80c-006477b37a-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:52:10 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:52:10.668 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj22 2023-05-31T20:52:10.668 DEBUG:boto:path=/pklxkd-59/obj22 2023-05-31T20:52:10.669 DEBUG:boto:auth_path=/pklxkd-59/obj22 2023-05-31T20:52:10.669 DEBUG:boto:Method: GET 2023-05-31T20:52:10.669 DEBUG:boto:Path: /pklxkd-59/obj22 2023-05-31T20:52:10.670 DEBUG:boto:Data: 2023-05-31T20:52:10.670 DEBUG:boto:Headers: {} 2023-05-31T20:52:10.670 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:52:10.671 DEBUG:boto:Port: 8001 2023-05-31T20:52:10.671 DEBUG:boto:Params: {} 2023-05-31T20:52:10.671 DEBUG:boto:Token: None 2023-05-31T20:52:10.672 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:52:10 GMT /pklxkd-59/obj22 2023-05-31T20:52:10.672 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:1H2kDiLQLan7KbPaDwdVcdWMdno= 2023-05-31T20:52:10.673 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:52:10 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:1H2kDiLQLan7KbPaDwdVcdWMdno=', 'Content-Length': '0'} 2023-05-31T20:52:10.675 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:39:26 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000028ded24256013a4b-006477b37a-1c17-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:52:10 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:52:10.716 DEBUG:boto:path=/pklxkd-59/obj22 2023-05-31T20:52:10.717 DEBUG:boto:auth_path=/pklxkd-59/obj22 2023-05-31T20:52:10.717 DEBUG:boto:Method: GET 2023-05-31T20:52:10.717 DEBUG:boto:Path: /pklxkd-59/obj22 2023-05-31T20:52:10.718 DEBUG:boto:Data: 2023-05-31T20:52:10.718 DEBUG:boto:Headers: {} 2023-05-31T20:52:10.718 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:52:10.719 DEBUG:boto:Port: 8000 2023-05-31T20:52:10.719 DEBUG:boto:Params: {} 2023-05-31T20:52:10.720 DEBUG:boto:Token: None 2023-05-31T20:52:10.720 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:52:10 GMT /pklxkd-59/obj22 2023-05-31T20:52:10.720 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:1H2kDiLQLan7KbPaDwdVcdWMdno= 2023-05-31T20:52:10.721 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:52:10 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:1H2kDiLQLan7KbPaDwdVcdWMdno=', 'Content-Length': '0'} 2023-05-31T20:52:10.723 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:39:26 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d404f32651de6698-006477b37a-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:52:10 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:52:10.723 DEBUG:boto:path=/pklxkd-59/ 2023-05-31T20:52:10.724 DEBUG:boto:auth_path=/pklxkd-59/ 2023-05-31T20:52:10.724 DEBUG:boto:path=/pklxkd-59/?prefix=obj22 2023-05-31T20:52:10.724 DEBUG:boto:auth_path=/pklxkd-59/?prefix=obj22 2023-05-31T20:52:10.725 DEBUG:boto:Method: GET 2023-05-31T20:52:10.725 DEBUG:boto:Path: /pklxkd-59/?prefix=obj22 2023-05-31T20:52:10.725 DEBUG:boto:Data: 2023-05-31T20:52:10.726 DEBUG:boto:Headers: {} 2023-05-31T20:52:10.726 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:52:10.726 DEBUG:boto:Port: 8001 2023-05-31T20:52:10.727 DEBUG:boto:Params: {} 2023-05-31T20:52:10.727 DEBUG:boto:Token: None 2023-05-31T20:52:10.728 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:52:10 GMT /pklxkd-59/ 2023-05-31T20:52:10.728 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:1aCZ7BhkC9nnuJr6CPizGBU2/jg= 2023-05-31T20:52:10.728 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:52:10 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:1aCZ7BhkC9nnuJr6CPizGBU2/jg=', 'Content-Length': '0'} 2023-05-31T20:52:10.731 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a0544a92fa89377d-006477b37a-1c17-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:52:10 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:52:10.731 DEBUG:boto:b'pklxkd-59obj221000falseobj222023-05-31T20:39:26.473Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:52:10.732 DEBUG:boto:path=/pklxkd-59/ 2023-05-31T20:52:10.732 DEBUG:boto:auth_path=/pklxkd-59/ 2023-05-31T20:52:10.733 DEBUG:boto:path=/pklxkd-59/?prefix=obj22 2023-05-31T20:52:10.733 DEBUG:boto:auth_path=/pklxkd-59/?prefix=obj22 2023-05-31T20:52:10.733 DEBUG:boto:Method: GET 2023-05-31T20:52:10.734 DEBUG:boto:Path: /pklxkd-59/?prefix=obj22 2023-05-31T20:52:10.734 DEBUG:boto:Data: 2023-05-31T20:52:10.735 DEBUG:boto:Headers: {} 2023-05-31T20:52:10.735 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:52:10.735 DEBUG:boto:Port: 8000 2023-05-31T20:52:10.736 DEBUG:boto:Params: {} 2023-05-31T20:52:10.736 DEBUG:boto:Token: None 2023-05-31T20:52:10.736 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:52:10 GMT /pklxkd-59/ 2023-05-31T20:52:10.737 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:1aCZ7BhkC9nnuJr6CPizGBU2/jg= 2023-05-31T20:52:10.737 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:52:10 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:1aCZ7BhkC9nnuJr6CPizGBU2/jg=', 'Content-Length': '0'} 2023-05-31T20:52:10.741 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000bdad0af460a317ec-006477b37a-1630-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:52:10 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:52:10.742 DEBUG:boto:b'pklxkd-59obj221000falseobj222023-05-31T20:39:26.473Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:52:10.742 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj23 2023-05-31T20:52:10.743 DEBUG:boto:path=/pklxkd-59/obj23 2023-05-31T20:52:10.743 DEBUG:boto:auth_path=/pklxkd-59/obj23 2023-05-31T20:52:10.743 DEBUG:boto:path=/pklxkd-59/obj23?versionId=null 2023-05-31T20:52:10.744 DEBUG:boto:auth_path=/pklxkd-59/obj23?versionId=null 2023-05-31T20:52:10.744 DEBUG:boto:Method: GET 2023-05-31T20:52:10.745 DEBUG:boto:Path: /pklxkd-59/obj23?versionId=null 2023-05-31T20:52:10.745 DEBUG:boto:Data: 2023-05-31T20:52:10.745 DEBUG:boto:Headers: {} 2023-05-31T20:52:10.746 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:52:10.746 DEBUG:boto:Port: 8001 2023-05-31T20:52:10.746 DEBUG:boto:Params: {} 2023-05-31T20:52:10.747 DEBUG:boto:Token: None 2023-05-31T20:52:10.747 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:52:10 GMT /pklxkd-59/obj23?versionId=null 2023-05-31T20:52:10.748 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:aHUb3qeUj4Ot6nLKzF3oywImuuo= 2023-05-31T20:52:10.748 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:52:10 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:aHUb3qeUj4Ot6nLKzF3oywImuuo=', 'Content-Length': '0'} 2023-05-31T20:52:10.750 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:46:33 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d7820fdcd9ba1a6d-006477b37a-1c17-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:52:10 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:52:10.751 DEBUG:boto:path=/pklxkd-59/obj23 2023-05-31T20:52:10.751 DEBUG:boto:auth_path=/pklxkd-59/obj23 2023-05-31T20:52:10.751 DEBUG:boto:path=/pklxkd-59/obj23?versionId=null 2023-05-31T20:52:10.752 DEBUG:boto:auth_path=/pklxkd-59/obj23?versionId=null 2023-05-31T20:52:10.752 DEBUG:boto:Method: GET 2023-05-31T20:52:10.752 DEBUG:boto:Path: /pklxkd-59/obj23?versionId=null 2023-05-31T20:52:10.753 DEBUG:boto:Data: 2023-05-31T20:52:10.753 DEBUG:boto:Headers: {} 2023-05-31T20:52:10.753 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:52:10.754 DEBUG:boto:Port: 8000 2023-05-31T20:52:10.754 DEBUG:boto:Params: {} 2023-05-31T20:52:10.755 DEBUG:boto:Token: None 2023-05-31T20:52:10.755 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:52:10 GMT /pklxkd-59/obj23?versionId=null 2023-05-31T20:52:10.755 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:aHUb3qeUj4Ot6nLKzF3oywImuuo= 2023-05-31T20:52:10.756 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:52:10 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:aHUb3qeUj4Ot6nLKzF3oywImuuo=', 'Content-Length': '0'} 2023-05-31T20:52:10.758 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:46:33 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d3ad2d6cce51521f-006477b37a-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:52:10 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:52:10.800 DEBUG:boto:path=/pklxkd-59/obj23 2023-05-31T20:52:10.801 DEBUG:boto:auth_path=/pklxkd-59/obj23 2023-05-31T20:52:10.801 DEBUG:boto:path=/pklxkd-59/obj23?versionId=null 2023-05-31T20:52:10.801 DEBUG:boto:auth_path=/pklxkd-59/obj23?versionId=null 2023-05-31T20:52:10.802 DEBUG:boto:Method: HEAD 2023-05-31T20:52:10.802 DEBUG:boto:Path: /pklxkd-59/obj23?versionId=null 2023-05-31T20:52:10.802 DEBUG:boto:Data: 2023-05-31T20:52:10.803 DEBUG:boto:Headers: {} 2023-05-31T20:52:10.803 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:52:10.803 DEBUG:boto:Port: 8001 2023-05-31T20:52:10.804 DEBUG:boto:Params: {} 2023-05-31T20:52:10.804 DEBUG:boto:Token: None 2023-05-31T20:52:10.805 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:52:10 GMT /pklxkd-59/obj23?versionId=null 2023-05-31T20:52:10.805 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:yaeV4OcFubYIITyM/wTh9I7xkPA= 2023-05-31T20:52:10.805 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:52:10 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:yaeV4OcFubYIITyM/wTh9I7xkPA=', 'Content-Length': '0'} 2023-05-31T20:52:10.807 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:46:33 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000035349de24e0144d8-006477b37a-1c17-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:52:10 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:52:10.807 DEBUG:boto:path=/pklxkd-59/obj23 2023-05-31T20:52:10.808 DEBUG:boto:auth_path=/pklxkd-59/obj23 2023-05-31T20:52:10.808 DEBUG:boto:path=/pklxkd-59/obj23?versionId=null 2023-05-31T20:52:10.809 DEBUG:boto:auth_path=/pklxkd-59/obj23?versionId=null 2023-05-31T20:52:10.809 DEBUG:boto:Method: HEAD 2023-05-31T20:52:10.809 DEBUG:boto:Path: /pklxkd-59/obj23?versionId=null 2023-05-31T20:52:10.810 DEBUG:boto:Data: 2023-05-31T20:52:10.810 DEBUG:boto:Headers: {} 2023-05-31T20:52:10.810 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:52:10.811 DEBUG:boto:Port: 8000 2023-05-31T20:52:10.811 DEBUG:boto:Params: {} 2023-05-31T20:52:10.811 DEBUG:boto:Token: None 2023-05-31T20:52:10.812 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:52:10 GMT /pklxkd-59/obj23?versionId=null 2023-05-31T20:52:10.812 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:yaeV4OcFubYIITyM/wTh9I7xkPA= 2023-05-31T20:52:10.812 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:52:10 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:yaeV4OcFubYIITyM/wTh9I7xkPA=', 'Content-Length': '0'} 2023-05-31T20:52:10.814 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:46:33 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000afc48c20e2882605-006477b37a-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:52:10 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:52:10.815 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj23 2023-05-31T20:52:10.815 DEBUG:boto:path=/pklxkd-59/obj23 2023-05-31T20:52:10.815 DEBUG:boto:auth_path=/pklxkd-59/obj23 2023-05-31T20:52:10.816 DEBUG:boto:path=/pklxkd-59/obj23?versionId=null 2023-05-31T20:52:10.816 DEBUG:boto:auth_path=/pklxkd-59/obj23?versionId=null 2023-05-31T20:52:10.816 DEBUG:boto:Method: GET 2023-05-31T20:52:10.817 DEBUG:boto:Path: /pklxkd-59/obj23?versionId=null 2023-05-31T20:52:10.817 DEBUG:boto:Data: 2023-05-31T20:52:10.817 DEBUG:boto:Headers: {} 2023-05-31T20:52:10.818 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:52:10.818 DEBUG:boto:Port: 8001 2023-05-31T20:52:10.819 DEBUG:boto:Params: {} 2023-05-31T20:52:10.819 DEBUG:boto:Token: None 2023-05-31T20:52:10.819 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:52:10 GMT /pklxkd-59/obj23?versionId=null 2023-05-31T20:52:10.820 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:aHUb3qeUj4Ot6nLKzF3oywImuuo= 2023-05-31T20:52:10.820 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:52:10 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:aHUb3qeUj4Ot6nLKzF3oywImuuo=', 'Content-Length': '0'} 2023-05-31T20:52:10.822 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:46:33 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001ff6413cc0917511-006477b37a-1c17-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:52:10 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:52:10.823 DEBUG:boto:path=/pklxkd-59/obj23 2023-05-31T20:52:10.823 DEBUG:boto:auth_path=/pklxkd-59/obj23 2023-05-31T20:52:10.824 DEBUG:boto:path=/pklxkd-59/obj23?versionId=null 2023-05-31T20:52:10.824 DEBUG:boto:auth_path=/pklxkd-59/obj23?versionId=null 2023-05-31T20:52:10.824 DEBUG:boto:Method: GET 2023-05-31T20:52:10.825 DEBUG:boto:Path: /pklxkd-59/obj23?versionId=null 2023-05-31T20:52:10.825 DEBUG:boto:Data: 2023-05-31T20:52:10.825 DEBUG:boto:Headers: {} 2023-05-31T20:52:10.826 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:52:10.826 DEBUG:boto:Port: 8000 2023-05-31T20:52:10.826 DEBUG:boto:Params: {} 2023-05-31T20:52:10.827 DEBUG:boto:Token: None 2023-05-31T20:52:10.827 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:52:10 GMT /pklxkd-59/obj23?versionId=null 2023-05-31T20:52:10.828 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:aHUb3qeUj4Ot6nLKzF3oywImuuo= 2023-05-31T20:52:10.828 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:52:10 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:aHUb3qeUj4Ot6nLKzF3oywImuuo=', 'Content-Length': '0'} 2023-05-31T20:52:10.830 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:46:33 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000074040df81c53b0ee-006477b37a-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:52:10 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:52:10.831 DEBUG:boto:path=/pklxkd-59/ 2023-05-31T20:52:10.831 DEBUG:boto:auth_path=/pklxkd-59/ 2023-05-31T20:52:10.832 DEBUG:boto:path=/pklxkd-59/?prefix=obj23 2023-05-31T20:52:10.832 DEBUG:boto:auth_path=/pklxkd-59/?prefix=obj23 2023-05-31T20:52:10.832 DEBUG:boto:Method: GET 2023-05-31T20:52:10.833 DEBUG:boto:Path: /pklxkd-59/?prefix=obj23 2023-05-31T20:52:10.833 DEBUG:boto:Data: 2023-05-31T20:52:10.833 DEBUG:boto:Headers: {} 2023-05-31T20:52:10.834 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:52:10.834 DEBUG:boto:Port: 8001 2023-05-31T20:52:10.834 DEBUG:boto:Params: {} 2023-05-31T20:52:10.835 DEBUG:boto:Token: None 2023-05-31T20:52:10.835 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:52:10 GMT /pklxkd-59/ 2023-05-31T20:52:10.836 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:1aCZ7BhkC9nnuJr6CPizGBU2/jg= 2023-05-31T20:52:10.836 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:52:10 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:1aCZ7BhkC9nnuJr6CPizGBU2/jg=', 'Content-Length': '0'} 2023-05-31T20:52:10.839 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f02c8b44b82817ca-006477b37a-1c17-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:52:10 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:52:10.839 DEBUG:boto:b'pklxkd-59obj231000falseobj232023-05-31T20:46:33.813Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:52:10.840 DEBUG:boto:path=/pklxkd-59/ 2023-05-31T20:52:10.840 DEBUG:boto:auth_path=/pklxkd-59/ 2023-05-31T20:52:10.841 DEBUG:boto:path=/pklxkd-59/?prefix=obj23 2023-05-31T20:52:10.841 DEBUG:boto:auth_path=/pklxkd-59/?prefix=obj23 2023-05-31T20:52:10.841 DEBUG:boto:Method: GET 2023-05-31T20:52:10.842 DEBUG:boto:Path: /pklxkd-59/?prefix=obj23 2023-05-31T20:52:10.842 DEBUG:boto:Data: 2023-05-31T20:52:10.842 DEBUG:boto:Headers: {} 2023-05-31T20:52:10.843 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:52:10.843 DEBUG:boto:Port: 8000 2023-05-31T20:52:10.843 DEBUG:boto:Params: {} 2023-05-31T20:52:10.844 DEBUG:boto:Token: None 2023-05-31T20:52:10.844 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:52:10 GMT /pklxkd-59/ 2023-05-31T20:52:10.845 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:1aCZ7BhkC9nnuJr6CPizGBU2/jg= 2023-05-31T20:52:10.845 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:52:10 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:1aCZ7BhkC9nnuJr6CPizGBU2/jg=', 'Content-Length': '0'} 2023-05-31T20:52:10.849 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000008c7877ddc02b5a27-006477b37a-1630-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:52:10 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:52:10.849 DEBUG:boto:b'pklxkd-59obj231000falseobj232023-05-31T20:46:33.813Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:52:10.850 DEBUG:boto:path=/pklxkd-59/obj23 2023-05-31T20:52:10.850 DEBUG:boto:auth_path=/pklxkd-59/obj23 2023-05-31T20:52:10.850 DEBUG:boto:Method: HEAD 2023-05-31T20:52:10.851 DEBUG:boto:Path: /pklxkd-59/obj23 2023-05-31T20:52:10.851 DEBUG:boto:Data: 2023-05-31T20:52:10.852 DEBUG:boto:Headers: {} 2023-05-31T20:52:10.852 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:52:10.852 DEBUG:boto:Port: 8001 2023-05-31T20:52:10.853 DEBUG:boto:Params: {} 2023-05-31T20:52:10.853 DEBUG:boto:Token: None 2023-05-31T20:52:10.853 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:52:10 GMT /pklxkd-59/obj23 2023-05-31T20:52:10.854 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:B9EiUTkz6GcHgjhjFGyeG5K2nuc= 2023-05-31T20:52:10.854 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:52:10 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:B9EiUTkz6GcHgjhjFGyeG5K2nuc=', 'Content-Length': '0'} 2023-05-31T20:52:10.856 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:46:33 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000873b45fb6bf44374-006477b37a-1c17-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:52:10 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:52:10.857 DEBUG:boto:path=/pklxkd-59/obj23 2023-05-31T20:52:10.857 DEBUG:boto:auth_path=/pklxkd-59/obj23 2023-05-31T20:52:10.857 DEBUG:boto:Method: HEAD 2023-05-31T20:52:10.858 DEBUG:boto:Path: /pklxkd-59/obj23 2023-05-31T20:52:10.858 DEBUG:boto:Data: 2023-05-31T20:52:10.858 DEBUG:boto:Headers: {} 2023-05-31T20:52:10.859 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:52:10.859 DEBUG:boto:Port: 8000 2023-05-31T20:52:10.859 DEBUG:boto:Params: {} 2023-05-31T20:52:10.860 DEBUG:boto:Token: None 2023-05-31T20:52:10.860 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:52:10 GMT /pklxkd-59/obj23 2023-05-31T20:52:10.860 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:B9EiUTkz6GcHgjhjFGyeG5K2nuc= 2023-05-31T20:52:10.861 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:52:10 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:B9EiUTkz6GcHgjhjFGyeG5K2nuc=', 'Content-Length': '0'} 2023-05-31T20:52:10.863 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:46:33 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004d842c2b0ad01b89-006477b37a-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:52:10 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:52:10.863 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj23 2023-05-31T20:52:10.864 DEBUG:boto:path=/pklxkd-59/obj23 2023-05-31T20:52:10.864 DEBUG:boto:auth_path=/pklxkd-59/obj23 2023-05-31T20:52:10.864 DEBUG:boto:Method: GET 2023-05-31T20:52:10.865 DEBUG:boto:Path: /pklxkd-59/obj23 2023-05-31T20:52:10.865 DEBUG:boto:Data: 2023-05-31T20:52:10.865 DEBUG:boto:Headers: {} 2023-05-31T20:52:10.866 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:52:10.866 DEBUG:boto:Port: 8001 2023-05-31T20:52:10.867 DEBUG:boto:Params: {} 2023-05-31T20:52:10.867 DEBUG:boto:Token: None 2023-05-31T20:52:10.867 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:52:10 GMT /pklxkd-59/obj23 2023-05-31T20:52:10.868 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:uMRDLSmrVHDX5HmTG/JmZkMhOEU= 2023-05-31T20:52:10.868 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:52:10 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:uMRDLSmrVHDX5HmTG/JmZkMhOEU=', 'Content-Length': '0'} 2023-05-31T20:52:10.870 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:46:33 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a20518f37140429e-006477b37a-1c17-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:52:10 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:52:10.871 DEBUG:boto:path=/pklxkd-59/obj23 2023-05-31T20:52:10.871 DEBUG:boto:auth_path=/pklxkd-59/obj23 2023-05-31T20:52:10.871 DEBUG:boto:Method: GET 2023-05-31T20:52:10.872 DEBUG:boto:Path: /pklxkd-59/obj23 2023-05-31T20:52:10.872 DEBUG:boto:Data: 2023-05-31T20:52:10.873 DEBUG:boto:Headers: {} 2023-05-31T20:52:10.873 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:52:10.873 DEBUG:boto:Port: 8000 2023-05-31T20:52:10.874 DEBUG:boto:Params: {} 2023-05-31T20:52:10.874 DEBUG:boto:Token: None 2023-05-31T20:52:10.874 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:52:10 GMT /pklxkd-59/obj23 2023-05-31T20:52:10.875 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:uMRDLSmrVHDX5HmTG/JmZkMhOEU= 2023-05-31T20:52:10.875 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:52:10 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:uMRDLSmrVHDX5HmTG/JmZkMhOEU=', 'Content-Length': '0'} 2023-05-31T20:52:10.877 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:46:33 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000066b3533af36ef1a4-006477b37a-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:52:10 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:52:10.877 DEBUG:boto:path=/pklxkd-59/ 2023-05-31T20:52:10.878 DEBUG:boto:auth_path=/pklxkd-59/ 2023-05-31T20:52:10.878 DEBUG:boto:path=/pklxkd-59/?prefix=obj23 2023-05-31T20:52:10.879 DEBUG:boto:auth_path=/pklxkd-59/?prefix=obj23 2023-05-31T20:52:10.879 DEBUG:boto:Method: GET 2023-05-31T20:52:10.879 DEBUG:boto:Path: /pklxkd-59/?prefix=obj23 2023-05-31T20:52:10.880 DEBUG:boto:Data: 2023-05-31T20:52:10.880 DEBUG:boto:Headers: {} 2023-05-31T20:52:10.880 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:52:10.881 DEBUG:boto:Port: 8001 2023-05-31T20:52:10.881 DEBUG:boto:Params: {} 2023-05-31T20:52:10.882 DEBUG:boto:Token: None 2023-05-31T20:52:10.882 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:52:10 GMT /pklxkd-59/ 2023-05-31T20:52:10.882 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:1aCZ7BhkC9nnuJr6CPizGBU2/jg= 2023-05-31T20:52:10.883 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:52:10 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:1aCZ7BhkC9nnuJr6CPizGBU2/jg=', 'Content-Length': '0'} 2023-05-31T20:52:10.885 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000005432d2ba5887b65f-006477b37a-1c17-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:52:10 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:52:10.885 DEBUG:boto:b'pklxkd-59obj231000falseobj232023-05-31T20:46:33.813Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:52:10.886 DEBUG:boto:path=/pklxkd-59/ 2023-05-31T20:52:10.886 DEBUG:boto:auth_path=/pklxkd-59/ 2023-05-31T20:52:10.887 DEBUG:boto:path=/pklxkd-59/?prefix=obj23 2023-05-31T20:52:10.887 DEBUG:boto:auth_path=/pklxkd-59/?prefix=obj23 2023-05-31T20:52:10.888 DEBUG:boto:Method: GET 2023-05-31T20:52:10.888 DEBUG:boto:Path: /pklxkd-59/?prefix=obj23 2023-05-31T20:52:10.888 DEBUG:boto:Data: 2023-05-31T20:52:10.889 DEBUG:boto:Headers: {} 2023-05-31T20:52:10.889 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:52:10.889 DEBUG:boto:Port: 8000 2023-05-31T20:52:10.890 DEBUG:boto:Params: {} 2023-05-31T20:52:10.890 DEBUG:boto:Token: None 2023-05-31T20:52:10.890 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:52:10 GMT /pklxkd-59/ 2023-05-31T20:52:10.891 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:1aCZ7BhkC9nnuJr6CPizGBU2/jg= 2023-05-31T20:52:10.891 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:52:10 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:1aCZ7BhkC9nnuJr6CPizGBU2/jg=', 'Content-Length': '0'} 2023-05-31T20:52:10.895 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000006bf86caad56ca16c-006477b37a-1630-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:52:10 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:52:10.936 DEBUG:boto:b'pklxkd-59obj231000falseobj232023-05-31T20:46:33.813Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:52:10.937 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj29 2023-05-31T20:52:10.937 DEBUG:boto:path=/pklxkd-59/obj29 2023-05-31T20:52:10.938 DEBUG:boto:auth_path=/pklxkd-59/obj29 2023-05-31T20:52:10.938 DEBUG:boto:path=/pklxkd-59/obj29?versionId=null 2023-05-31T20:52:10.938 DEBUG:boto:auth_path=/pklxkd-59/obj29?versionId=null 2023-05-31T20:52:10.939 DEBUG:boto:Method: GET 2023-05-31T20:52:10.939 DEBUG:boto:Path: /pklxkd-59/obj29?versionId=null 2023-05-31T20:52:10.939 DEBUG:boto:Data: 2023-05-31T20:52:10.940 DEBUG:boto:Headers: {} 2023-05-31T20:52:10.940 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:52:10.941 DEBUG:boto:Port: 8001 2023-05-31T20:52:10.941 DEBUG:boto:Params: {} 2023-05-31T20:52:10.941 DEBUG:boto:Token: None 2023-05-31T20:52:10.942 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:52:10 GMT /pklxkd-59/obj29?versionId=null 2023-05-31T20:52:10.942 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:vQcZH1QaMzf6LL5gDUAO4Bys49U= 2023-05-31T20:52:10.943 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:52:10 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:vQcZH1QaMzf6LL5gDUAO4Bys49U=', 'Content-Length': '0'} 2023-05-31T20:52:10.945 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:48:12 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002cfdbe6859c93527-006477b37a-1c17-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:52:10 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:52:10.945 DEBUG:boto:path=/pklxkd-59/obj29 2023-05-31T20:52:10.946 DEBUG:boto:auth_path=/pklxkd-59/obj29 2023-05-31T20:52:10.946 DEBUG:boto:path=/pklxkd-59/obj29?versionId=null 2023-05-31T20:52:10.946 DEBUG:boto:auth_path=/pklxkd-59/obj29?versionId=null 2023-05-31T20:52:10.947 DEBUG:boto:Method: GET 2023-05-31T20:52:10.947 DEBUG:boto:Path: /pklxkd-59/obj29?versionId=null 2023-05-31T20:52:10.947 DEBUG:boto:Data: 2023-05-31T20:52:10.948 DEBUG:boto:Headers: {} 2023-05-31T20:52:10.948 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:52:10.948 DEBUG:boto:Port: 8000 2023-05-31T20:52:10.949 DEBUG:boto:Params: {} 2023-05-31T20:52:10.949 DEBUG:boto:Token: None 2023-05-31T20:52:10.950 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:52:10 GMT /pklxkd-59/obj29?versionId=null 2023-05-31T20:52:10.950 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:vQcZH1QaMzf6LL5gDUAO4Bys49U= 2023-05-31T20:52:10.950 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:52:10 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:vQcZH1QaMzf6LL5gDUAO4Bys49U=', 'Content-Length': '0'} 2023-05-31T20:52:10.952 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:48:12 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003b874249ec05c7bf-006477b37a-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:52:10 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:52:10.953 DEBUG:boto:path=/pklxkd-59/obj29 2023-05-31T20:52:10.953 DEBUG:boto:auth_path=/pklxkd-59/obj29 2023-05-31T20:52:10.954 DEBUG:boto:path=/pklxkd-59/obj29?versionId=null 2023-05-31T20:52:10.954 DEBUG:boto:auth_path=/pklxkd-59/obj29?versionId=null 2023-05-31T20:52:10.954 DEBUG:boto:Method: HEAD 2023-05-31T20:52:10.955 DEBUG:boto:Path: /pklxkd-59/obj29?versionId=null 2023-05-31T20:52:10.955 DEBUG:boto:Data: 2023-05-31T20:52:10.955 DEBUG:boto:Headers: {} 2023-05-31T20:52:10.956 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:52:10.956 DEBUG:boto:Port: 8001 2023-05-31T20:52:10.956 DEBUG:boto:Params: {} 2023-05-31T20:52:10.957 DEBUG:boto:Token: None 2023-05-31T20:52:10.957 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:52:10 GMT /pklxkd-59/obj29?versionId=null 2023-05-31T20:52:10.957 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:3XDLw5tRUdQ/pcHwSvpt0uExUJQ= 2023-05-31T20:52:10.958 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:52:10 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:3XDLw5tRUdQ/pcHwSvpt0uExUJQ=', 'Content-Length': '0'} 2023-05-31T20:52:10.960 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:48:12 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000bfc118a0bd6e536b-006477b37a-1c17-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:52:10 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:52:10.960 DEBUG:boto:path=/pklxkd-59/obj29 2023-05-31T20:52:10.961 DEBUG:boto:auth_path=/pklxkd-59/obj29 2023-05-31T20:52:10.961 DEBUG:boto:path=/pklxkd-59/obj29?versionId=null 2023-05-31T20:52:10.961 DEBUG:boto:auth_path=/pklxkd-59/obj29?versionId=null 2023-05-31T20:52:10.962 DEBUG:boto:Method: HEAD 2023-05-31T20:52:10.962 DEBUG:boto:Path: /pklxkd-59/obj29?versionId=null 2023-05-31T20:52:10.962 DEBUG:boto:Data: 2023-05-31T20:52:10.963 DEBUG:boto:Headers: {} 2023-05-31T20:52:10.963 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:52:10.963 DEBUG:boto:Port: 8000 2023-05-31T20:52:10.963 DEBUG:boto:Params: {} 2023-05-31T20:52:10.964 DEBUG:boto:Token: None 2023-05-31T20:52:10.964 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:52:10 GMT /pklxkd-59/obj29?versionId=null 2023-05-31T20:52:10.965 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:3XDLw5tRUdQ/pcHwSvpt0uExUJQ= 2023-05-31T20:52:10.965 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:52:10 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:3XDLw5tRUdQ/pcHwSvpt0uExUJQ=', 'Content-Length': '0'} 2023-05-31T20:52:10.967 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:48:12 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000041d39b6c4fb47b55-006477b37a-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:52:10 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:52:10.967 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj29 2023-05-31T20:52:10.968 DEBUG:boto:path=/pklxkd-59/obj29 2023-05-31T20:52:10.968 DEBUG:boto:auth_path=/pklxkd-59/obj29 2023-05-31T20:52:10.968 DEBUG:boto:path=/pklxkd-59/obj29?versionId=null 2023-05-31T20:52:10.969 DEBUG:boto:auth_path=/pklxkd-59/obj29?versionId=null 2023-05-31T20:52:10.969 DEBUG:boto:Method: GET 2023-05-31T20:52:10.970 DEBUG:boto:Path: /pklxkd-59/obj29?versionId=null 2023-05-31T20:52:10.970 DEBUG:boto:Data: 2023-05-31T20:52:10.970 DEBUG:boto:Headers: {} 2023-05-31T20:52:10.971 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:52:10.971 DEBUG:boto:Port: 8001 2023-05-31T20:52:10.971 DEBUG:boto:Params: {} 2023-05-31T20:52:10.972 DEBUG:boto:Token: None 2023-05-31T20:52:10.972 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:52:10 GMT /pklxkd-59/obj29?versionId=null 2023-05-31T20:52:10.972 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:vQcZH1QaMzf6LL5gDUAO4Bys49U= 2023-05-31T20:52:10.973 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:52:10 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:vQcZH1QaMzf6LL5gDUAO4Bys49U=', 'Content-Length': '0'} 2023-05-31T20:52:10.974 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:48:12 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c1a605286ea44cae-006477b37a-1c17-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:52:10 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:52:10.975 DEBUG:boto:path=/pklxkd-59/obj29 2023-05-31T20:52:10.975 DEBUG:boto:auth_path=/pklxkd-59/obj29 2023-05-31T20:52:10.976 DEBUG:boto:path=/pklxkd-59/obj29?versionId=null 2023-05-31T20:52:10.976 DEBUG:boto:auth_path=/pklxkd-59/obj29?versionId=null 2023-05-31T20:52:10.977 DEBUG:boto:Method: GET 2023-05-31T20:52:10.977 DEBUG:boto:Path: /pklxkd-59/obj29?versionId=null 2023-05-31T20:52:10.977 DEBUG:boto:Data: 2023-05-31T20:52:10.978 DEBUG:boto:Headers: {} 2023-05-31T20:52:10.978 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:52:10.978 DEBUG:boto:Port: 8000 2023-05-31T20:52:10.979 DEBUG:boto:Params: {} 2023-05-31T20:52:10.979 DEBUG:boto:Token: None 2023-05-31T20:52:10.980 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:52:10 GMT /pklxkd-59/obj29?versionId=null 2023-05-31T20:52:10.980 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:vQcZH1QaMzf6LL5gDUAO4Bys49U= 2023-05-31T20:52:10.980 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:52:10 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:vQcZH1QaMzf6LL5gDUAO4Bys49U=', 'Content-Length': '0'} 2023-05-31T20:52:10.982 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:48:12 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000039c71b8116a1b4d9-006477b37a-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:52:10 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:52:10.983 DEBUG:boto:path=/pklxkd-59/ 2023-05-31T20:52:10.983 DEBUG:boto:auth_path=/pklxkd-59/ 2023-05-31T20:52:10.984 DEBUG:boto:path=/pklxkd-59/?prefix=obj29 2023-05-31T20:52:10.984 DEBUG:boto:auth_path=/pklxkd-59/?prefix=obj29 2023-05-31T20:52:10.985 DEBUG:boto:Method: GET 2023-05-31T20:52:10.985 DEBUG:boto:Path: /pklxkd-59/?prefix=obj29 2023-05-31T20:52:10.985 DEBUG:boto:Data: 2023-05-31T20:52:10.986 DEBUG:boto:Headers: {} 2023-05-31T20:52:10.986 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:52:10.986 DEBUG:boto:Port: 8001 2023-05-31T20:52:10.987 DEBUG:boto:Params: {} 2023-05-31T20:52:10.987 DEBUG:boto:Token: None 2023-05-31T20:52:10.988 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:52:10 GMT /pklxkd-59/ 2023-05-31T20:52:10.988 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:1aCZ7BhkC9nnuJr6CPizGBU2/jg= 2023-05-31T20:52:10.988 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:52:10 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:1aCZ7BhkC9nnuJr6CPizGBU2/jg=', 'Content-Length': '0'} 2023-05-31T20:52:10.991 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000004d7337dc58e55147-006477b37a-1c17-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:52:10 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:52:10.991 DEBUG:boto:b'pklxkd-59obj291000falseobj292023-05-31T20:48:12.956Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:52:10.992 DEBUG:boto:path=/pklxkd-59/ 2023-05-31T20:52:10.992 DEBUG:boto:auth_path=/pklxkd-59/ 2023-05-31T20:52:10.993 DEBUG:boto:path=/pklxkd-59/?prefix=obj29 2023-05-31T20:52:10.993 DEBUG:boto:auth_path=/pklxkd-59/?prefix=obj29 2023-05-31T20:52:10.993 DEBUG:boto:Method: GET 2023-05-31T20:52:10.994 DEBUG:boto:Path: /pklxkd-59/?prefix=obj29 2023-05-31T20:52:10.994 DEBUG:boto:Data: 2023-05-31T20:52:10.994 DEBUG:boto:Headers: {} 2023-05-31T20:52:10.995 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:52:10.995 DEBUG:boto:Port: 8000 2023-05-31T20:52:10.995 DEBUG:boto:Params: {} 2023-05-31T20:52:10.996 DEBUG:boto:Token: None 2023-05-31T20:52:10.996 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:52:10 GMT /pklxkd-59/ 2023-05-31T20:52:10.997 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:1aCZ7BhkC9nnuJr6CPizGBU2/jg= 2023-05-31T20:52:10.997 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:52:10 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:1aCZ7BhkC9nnuJr6CPizGBU2/jg=', 'Content-Length': '0'} 2023-05-31T20:52:11.001 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000005d654a73d46247ed-006477b37a-1630-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:52:10 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:52:11.001 DEBUG:boto:b'pklxkd-59obj291000falseobj292023-05-31T20:48:12.956Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:52:11.002 DEBUG:boto:path=/pklxkd-59/obj29 2023-05-31T20:52:11.002 DEBUG:boto:auth_path=/pklxkd-59/obj29 2023-05-31T20:52:11.002 DEBUG:boto:Method: HEAD 2023-05-31T20:52:11.003 DEBUG:boto:Path: /pklxkd-59/obj29 2023-05-31T20:52:11.003 DEBUG:boto:Data: 2023-05-31T20:52:11.003 DEBUG:boto:Headers: {} 2023-05-31T20:52:11.004 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:52:11.004 DEBUG:boto:Port: 8001 2023-05-31T20:52:11.004 DEBUG:boto:Params: {} 2023-05-31T20:52:11.005 DEBUG:boto:Token: None 2023-05-31T20:52:11.005 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:52:11 GMT /pklxkd-59/obj29 2023-05-31T20:52:11.006 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:xcaQZQwGn5ZDiYNP01OdVGpMpPY= 2023-05-31T20:52:11.006 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:52:11 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:xcaQZQwGn5ZDiYNP01OdVGpMpPY=', 'Content-Length': '0'} 2023-05-31T20:52:11.008 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:48:12 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ebe71a5df34f7d48-006477b37b-1c17-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:52:11 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:52:11.008 DEBUG:boto:path=/pklxkd-59/obj29 2023-05-31T20:52:11.009 DEBUG:boto:auth_path=/pklxkd-59/obj29 2023-05-31T20:52:11.009 DEBUG:boto:Method: HEAD 2023-05-31T20:52:11.009 DEBUG:boto:Path: /pklxkd-59/obj29 2023-05-31T20:52:11.010 DEBUG:boto:Data: 2023-05-31T20:52:11.010 DEBUG:boto:Headers: {} 2023-05-31T20:52:11.010 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:52:11.011 DEBUG:boto:Port: 8000 2023-05-31T20:52:11.011 DEBUG:boto:Params: {} 2023-05-31T20:52:11.011 DEBUG:boto:Token: None 2023-05-31T20:52:11.012 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:52:11 GMT /pklxkd-59/obj29 2023-05-31T20:52:11.012 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:xcaQZQwGn5ZDiYNP01OdVGpMpPY= 2023-05-31T20:52:11.013 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:52:11 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:xcaQZQwGn5ZDiYNP01OdVGpMpPY=', 'Content-Length': '0'} 2023-05-31T20:52:11.014 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:48:12 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007f42ae476876c1f2-006477b37b-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:52:11 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:52:11.015 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj29 2023-05-31T20:52:11.015 DEBUG:boto:path=/pklxkd-59/obj29 2023-05-31T20:52:11.016 DEBUG:boto:auth_path=/pklxkd-59/obj29 2023-05-31T20:52:11.016 DEBUG:boto:Method: GET 2023-05-31T20:52:11.016 DEBUG:boto:Path: /pklxkd-59/obj29 2023-05-31T20:52:11.017 DEBUG:boto:Data: 2023-05-31T20:52:11.017 DEBUG:boto:Headers: {} 2023-05-31T20:52:11.017 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:52:11.018 DEBUG:boto:Port: 8001 2023-05-31T20:52:11.018 DEBUG:boto:Params: {} 2023-05-31T20:52:11.019 DEBUG:boto:Token: None 2023-05-31T20:52:11.019 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:52:11 GMT /pklxkd-59/obj29 2023-05-31T20:52:11.019 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:M2vp0GNdwB9ZCIcw5oDvtHV0zrE= 2023-05-31T20:52:11.020 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:52:11 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:M2vp0GNdwB9ZCIcw5oDvtHV0zrE=', 'Content-Length': '0'} 2023-05-31T20:52:11.022 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:48:12 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000948672b63b92f025-006477b37b-1c17-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:52:11 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:52:11.022 DEBUG:boto:path=/pklxkd-59/obj29 2023-05-31T20:52:11.023 DEBUG:boto:auth_path=/pklxkd-59/obj29 2023-05-31T20:52:11.023 DEBUG:boto:Method: GET 2023-05-31T20:52:11.023 DEBUG:boto:Path: /pklxkd-59/obj29 2023-05-31T20:52:11.024 DEBUG:boto:Data: 2023-05-31T20:52:11.024 DEBUG:boto:Headers: {} 2023-05-31T20:52:11.025 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:52:11.025 DEBUG:boto:Port: 8000 2023-05-31T20:52:11.025 DEBUG:boto:Params: {} 2023-05-31T20:52:11.026 DEBUG:boto:Token: None 2023-05-31T20:52:11.026 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:52:11 GMT /pklxkd-59/obj29 2023-05-31T20:52:11.027 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:M2vp0GNdwB9ZCIcw5oDvtHV0zrE= 2023-05-31T20:52:11.027 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:52:11 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:M2vp0GNdwB9ZCIcw5oDvtHV0zrE=', 'Content-Length': '0'} 2023-05-31T20:52:11.029 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:48:12 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005dbc1d365b54bbff-006477b37b-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:52:11 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:52:11.072 DEBUG:boto:path=/pklxkd-59/ 2023-05-31T20:52:11.073 DEBUG:boto:auth_path=/pklxkd-59/ 2023-05-31T20:52:11.073 DEBUG:boto:path=/pklxkd-59/?prefix=obj29 2023-05-31T20:52:11.074 DEBUG:boto:auth_path=/pklxkd-59/?prefix=obj29 2023-05-31T20:52:11.074 DEBUG:boto:Method: GET 2023-05-31T20:52:11.075 DEBUG:boto:Path: /pklxkd-59/?prefix=obj29 2023-05-31T20:52:11.075 DEBUG:boto:Data: 2023-05-31T20:52:11.075 DEBUG:boto:Headers: {} 2023-05-31T20:52:11.076 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:52:11.076 DEBUG:boto:Port: 8001 2023-05-31T20:52:11.076 DEBUG:boto:Params: {} 2023-05-31T20:52:11.077 DEBUG:boto:Token: None 2023-05-31T20:52:11.077 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:52:11 GMT /pklxkd-59/ 2023-05-31T20:52:11.078 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:HITZ+oTJfJ3JS1dgm5XbULiOORY= 2023-05-31T20:52:11.078 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:52:11 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:HITZ+oTJfJ3JS1dgm5XbULiOORY=', 'Content-Length': '0'} 2023-05-31T20:52:11.081 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000462bc9ac30dc9e4a-006477b37b-1c17-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:52:11 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:52:11.082 DEBUG:boto:b'pklxkd-59obj291000falseobj292023-05-31T20:48:12.956Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:52:11.082 DEBUG:boto:path=/pklxkd-59/ 2023-05-31T20:52:11.083 DEBUG:boto:auth_path=/pklxkd-59/ 2023-05-31T20:52:11.083 DEBUG:boto:path=/pklxkd-59/?prefix=obj29 2023-05-31T20:52:11.084 DEBUG:boto:auth_path=/pklxkd-59/?prefix=obj29 2023-05-31T20:52:11.084 DEBUG:boto:Method: GET 2023-05-31T20:52:11.084 DEBUG:boto:Path: /pklxkd-59/?prefix=obj29 2023-05-31T20:52:11.085 DEBUG:boto:Data: 2023-05-31T20:52:11.085 DEBUG:boto:Headers: {} 2023-05-31T20:52:11.085 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:52:11.086 DEBUG:boto:Port: 8000 2023-05-31T20:52:11.086 DEBUG:boto:Params: {} 2023-05-31T20:52:11.087 DEBUG:boto:Token: None 2023-05-31T20:52:11.087 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:52:11 GMT /pklxkd-59/ 2023-05-31T20:52:11.088 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:HITZ+oTJfJ3JS1dgm5XbULiOORY= 2023-05-31T20:52:11.088 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:52:11 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:HITZ+oTJfJ3JS1dgm5XbULiOORY=', 'Content-Length': '0'} 2023-05-31T20:52:11.091 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000006b0978ccc98fc9f-006477b37b-1630-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:52:11 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:52:11.092 DEBUG:boto:b'pklxkd-59obj291000falseobj292023-05-31T20:48:12.956Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:52:11.093 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj31 2023-05-31T20:52:11.093 DEBUG:boto:path=/pklxkd-59/obj31 2023-05-31T20:52:11.094 DEBUG:boto:auth_path=/pklxkd-59/obj31 2023-05-31T20:52:11.094 DEBUG:boto:path=/pklxkd-59/obj31?versionId=null 2023-05-31T20:52:11.094 DEBUG:boto:auth_path=/pklxkd-59/obj31?versionId=null 2023-05-31T20:52:11.095 DEBUG:boto:Method: GET 2023-05-31T20:52:11.095 DEBUG:boto:Path: /pklxkd-59/obj31?versionId=null 2023-05-31T20:52:11.095 DEBUG:boto:Data: 2023-05-31T20:52:11.096 DEBUG:boto:Headers: {} 2023-05-31T20:52:11.096 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:52:11.096 DEBUG:boto:Port: 8001 2023-05-31T20:52:11.097 DEBUG:boto:Params: {} 2023-05-31T20:52:11.097 DEBUG:boto:Token: None 2023-05-31T20:52:11.098 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:52:11 GMT /pklxkd-59/obj31?versionId=null 2023-05-31T20:52:11.098 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:f4pY+jUfBO+o3dRgakXvOzEMQKk= 2023-05-31T20:52:11.098 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:52:11 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:f4pY+jUfBO+o3dRgakXvOzEMQKk=', 'Content-Length': '0'} 2023-05-31T20:52:11.101 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:50:33 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000073dbe1ea022bcab9-006477b37b-1c17-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:52:11 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:52:11.144 DEBUG:boto:path=/pklxkd-59/obj31 2023-05-31T20:52:11.145 DEBUG:boto:auth_path=/pklxkd-59/obj31 2023-05-31T20:52:11.145 DEBUG:boto:path=/pklxkd-59/obj31?versionId=null 2023-05-31T20:52:11.146 DEBUG:boto:auth_path=/pklxkd-59/obj31?versionId=null 2023-05-31T20:52:11.146 DEBUG:boto:Method: GET 2023-05-31T20:52:11.147 DEBUG:boto:Path: /pklxkd-59/obj31?versionId=null 2023-05-31T20:52:11.147 DEBUG:boto:Data: 2023-05-31T20:52:11.147 DEBUG:boto:Headers: {} 2023-05-31T20:52:11.148 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:52:11.148 DEBUG:boto:Port: 8000 2023-05-31T20:52:11.148 DEBUG:boto:Params: {} 2023-05-31T20:52:11.149 DEBUG:boto:Token: None 2023-05-31T20:52:11.149 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:52:11 GMT /pklxkd-59/obj31?versionId=null 2023-05-31T20:52:11.150 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:f4pY+jUfBO+o3dRgakXvOzEMQKk= 2023-05-31T20:52:11.150 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:52:11 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:f4pY+jUfBO+o3dRgakXvOzEMQKk=', 'Content-Length': '0'} 2023-05-31T20:52:11.152 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:50:33 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d6735408815a106a-006477b37b-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:52:11 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:52:11.153 DEBUG:boto:path=/pklxkd-59/obj31 2023-05-31T20:52:11.153 DEBUG:boto:auth_path=/pklxkd-59/obj31 2023-05-31T20:52:11.154 DEBUG:boto:path=/pklxkd-59/obj31?versionId=null 2023-05-31T20:52:11.154 DEBUG:boto:auth_path=/pklxkd-59/obj31?versionId=null 2023-05-31T20:52:11.154 DEBUG:boto:Method: HEAD 2023-05-31T20:52:11.155 DEBUG:boto:Path: /pklxkd-59/obj31?versionId=null 2023-05-31T20:52:11.155 DEBUG:boto:Data: 2023-05-31T20:52:11.155 DEBUG:boto:Headers: {} 2023-05-31T20:52:11.156 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:52:11.156 DEBUG:boto:Port: 8001 2023-05-31T20:52:11.157 DEBUG:boto:Params: {} 2023-05-31T20:52:11.157 DEBUG:boto:Token: None 2023-05-31T20:52:11.157 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:52:11 GMT /pklxkd-59/obj31?versionId=null 2023-05-31T20:52:11.158 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:0B9+7aSvC8aSjYz2BX9d/VIBe1Y= 2023-05-31T20:52:11.158 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:52:11 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:0B9+7aSvC8aSjYz2BX9d/VIBe1Y=', 'Content-Length': '0'} 2023-05-31T20:52:11.160 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:50:33 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000df7607f27e32d94c-006477b37b-1c17-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:52:11 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:52:11.160 DEBUG:boto:path=/pklxkd-59/obj31 2023-05-31T20:52:11.161 DEBUG:boto:auth_path=/pklxkd-59/obj31 2023-05-31T20:52:11.161 DEBUG:boto:path=/pklxkd-59/obj31?versionId=null 2023-05-31T20:52:11.162 DEBUG:boto:auth_path=/pklxkd-59/obj31?versionId=null 2023-05-31T20:52:11.162 DEBUG:boto:Method: HEAD 2023-05-31T20:52:11.162 DEBUG:boto:Path: /pklxkd-59/obj31?versionId=null 2023-05-31T20:52:11.163 DEBUG:boto:Data: 2023-05-31T20:52:11.163 DEBUG:boto:Headers: {} 2023-05-31T20:52:11.163 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:52:11.163 DEBUG:boto:Port: 8000 2023-05-31T20:52:11.164 DEBUG:boto:Params: {} 2023-05-31T20:52:11.164 DEBUG:boto:Token: None 2023-05-31T20:52:11.165 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:52:11 GMT /pklxkd-59/obj31?versionId=null 2023-05-31T20:52:11.165 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:0B9+7aSvC8aSjYz2BX9d/VIBe1Y= 2023-05-31T20:52:11.165 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:52:11 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:0B9+7aSvC8aSjYz2BX9d/VIBe1Y=', 'Content-Length': '0'} 2023-05-31T20:52:11.167 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:50:33 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d14b2d797803b8ca-006477b37b-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:52:11 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:52:11.168 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj31 2023-05-31T20:52:11.168 DEBUG:boto:path=/pklxkd-59/obj31 2023-05-31T20:52:11.169 DEBUG:boto:auth_path=/pklxkd-59/obj31 2023-05-31T20:52:11.169 DEBUG:boto:path=/pklxkd-59/obj31?versionId=null 2023-05-31T20:52:11.169 DEBUG:boto:auth_path=/pklxkd-59/obj31?versionId=null 2023-05-31T20:52:11.170 DEBUG:boto:Method: GET 2023-05-31T20:52:11.170 DEBUG:boto:Path: /pklxkd-59/obj31?versionId=null 2023-05-31T20:52:11.170 DEBUG:boto:Data: 2023-05-31T20:52:11.171 DEBUG:boto:Headers: {} 2023-05-31T20:52:11.171 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:52:11.171 DEBUG:boto:Port: 8001 2023-05-31T20:52:11.172 DEBUG:boto:Params: {} 2023-05-31T20:52:11.172 DEBUG:boto:Token: None 2023-05-31T20:52:11.173 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:52:11 GMT /pklxkd-59/obj31?versionId=null 2023-05-31T20:52:11.173 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:f4pY+jUfBO+o3dRgakXvOzEMQKk= 2023-05-31T20:52:11.174 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:52:11 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:f4pY+jUfBO+o3dRgakXvOzEMQKk=', 'Content-Length': '0'} 2023-05-31T20:52:11.175 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:50:33 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a63445ac4ac5df63-006477b37b-1c17-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:52:11 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:52:11.176 DEBUG:boto:path=/pklxkd-59/obj31 2023-05-31T20:52:11.176 DEBUG:boto:auth_path=/pklxkd-59/obj31 2023-05-31T20:52:11.177 DEBUG:boto:path=/pklxkd-59/obj31?versionId=null 2023-05-31T20:52:11.177 DEBUG:boto:auth_path=/pklxkd-59/obj31?versionId=null 2023-05-31T20:52:11.177 DEBUG:boto:Method: GET 2023-05-31T20:52:11.178 DEBUG:boto:Path: /pklxkd-59/obj31?versionId=null 2023-05-31T20:52:11.178 DEBUG:boto:Data: 2023-05-31T20:52:11.179 DEBUG:boto:Headers: {} 2023-05-31T20:52:11.179 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:52:11.179 DEBUG:boto:Port: 8000 2023-05-31T20:52:11.180 DEBUG:boto:Params: {} 2023-05-31T20:52:11.180 DEBUG:boto:Token: None 2023-05-31T20:52:11.180 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:52:11 GMT /pklxkd-59/obj31?versionId=null 2023-05-31T20:52:11.181 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:f4pY+jUfBO+o3dRgakXvOzEMQKk= 2023-05-31T20:52:11.181 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:52:11 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:f4pY+jUfBO+o3dRgakXvOzEMQKk=', 'Content-Length': '0'} 2023-05-31T20:52:11.183 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:50:33 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000058f19a6d57de0dfb-006477b37b-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:52:11 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:52:11.184 DEBUG:boto:path=/pklxkd-59/ 2023-05-31T20:52:11.184 DEBUG:boto:auth_path=/pklxkd-59/ 2023-05-31T20:52:11.185 DEBUG:boto:path=/pklxkd-59/?prefix=obj31 2023-05-31T20:52:11.185 DEBUG:boto:auth_path=/pklxkd-59/?prefix=obj31 2023-05-31T20:52:11.185 DEBUG:boto:Method: GET 2023-05-31T20:52:11.186 DEBUG:boto:Path: /pklxkd-59/?prefix=obj31 2023-05-31T20:52:11.186 DEBUG:boto:Data: 2023-05-31T20:52:11.186 DEBUG:boto:Headers: {} 2023-05-31T20:52:11.187 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:52:11.187 DEBUG:boto:Port: 8001 2023-05-31T20:52:11.188 DEBUG:boto:Params: {} 2023-05-31T20:52:11.188 DEBUG:boto:Token: None 2023-05-31T20:52:11.188 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:52:11 GMT /pklxkd-59/ 2023-05-31T20:52:11.189 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:HITZ+oTJfJ3JS1dgm5XbULiOORY= 2023-05-31T20:52:11.189 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:52:11 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:HITZ+oTJfJ3JS1dgm5XbULiOORY=', 'Content-Length': '0'} 2023-05-31T20:52:11.192 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000071c5ca3399a68a54-006477b37b-1c17-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:52:11 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:52:11.192 DEBUG:boto:b'pklxkd-59obj311000falseobj312023-05-31T20:50:33.460Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:52:11.193 DEBUG:boto:path=/pklxkd-59/ 2023-05-31T20:52:11.193 DEBUG:boto:auth_path=/pklxkd-59/ 2023-05-31T20:52:11.194 DEBUG:boto:path=/pklxkd-59/?prefix=obj31 2023-05-31T20:52:11.194 DEBUG:boto:auth_path=/pklxkd-59/?prefix=obj31 2023-05-31T20:52:11.194 DEBUG:boto:Method: GET 2023-05-31T20:52:11.195 DEBUG:boto:Path: /pklxkd-59/?prefix=obj31 2023-05-31T20:52:11.195 DEBUG:boto:Data: 2023-05-31T20:52:11.195 DEBUG:boto:Headers: {} 2023-05-31T20:52:11.196 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:52:11.196 DEBUG:boto:Port: 8000 2023-05-31T20:52:11.197 DEBUG:boto:Params: {} 2023-05-31T20:52:11.197 DEBUG:boto:Token: None 2023-05-31T20:52:11.197 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:52:11 GMT /pklxkd-59/ 2023-05-31T20:52:11.198 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:HITZ+oTJfJ3JS1dgm5XbULiOORY= 2023-05-31T20:52:11.198 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:52:11 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:HITZ+oTJfJ3JS1dgm5XbULiOORY=', 'Content-Length': '0'} 2023-05-31T20:52:11.202 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a75289b01a5395b1-006477b37b-1630-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:52:11 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:52:11.203 DEBUG:boto:b'pklxkd-59obj311000falseobj312023-05-31T20:50:33.460Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:52:11.203 DEBUG:boto:path=/pklxkd-59/obj31 2023-05-31T20:52:11.204 DEBUG:boto:auth_path=/pklxkd-59/obj31 2023-05-31T20:52:11.204 DEBUG:boto:Method: HEAD 2023-05-31T20:52:11.204 DEBUG:boto:Path: /pklxkd-59/obj31 2023-05-31T20:52:11.205 DEBUG:boto:Data: 2023-05-31T20:52:11.205 DEBUG:boto:Headers: {} 2023-05-31T20:52:11.205 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:52:11.206 DEBUG:boto:Port: 8001 2023-05-31T20:52:11.206 DEBUG:boto:Params: {} 2023-05-31T20:52:11.206 DEBUG:boto:Token: None 2023-05-31T20:52:11.207 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:52:11 GMT /pklxkd-59/obj31 2023-05-31T20:52:11.207 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:ZcKC66GQhofx1RMZgxmca4F4xUo= 2023-05-31T20:52:11.208 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:52:11 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:ZcKC66GQhofx1RMZgxmca4F4xUo=', 'Content-Length': '0'} 2023-05-31T20:52:11.209 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:50:33 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009fd5519c5c6f1c3a-006477b37b-1c17-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:52:11 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:52:11.210 DEBUG:boto:path=/pklxkd-59/obj31 2023-05-31T20:52:11.210 DEBUG:boto:auth_path=/pklxkd-59/obj31 2023-05-31T20:52:11.211 DEBUG:boto:Method: HEAD 2023-05-31T20:52:11.211 DEBUG:boto:Path: /pklxkd-59/obj31 2023-05-31T20:52:11.211 DEBUG:boto:Data: 2023-05-31T20:52:11.212 DEBUG:boto:Headers: {} 2023-05-31T20:52:11.212 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:52:11.212 DEBUG:boto:Port: 8000 2023-05-31T20:52:11.213 DEBUG:boto:Params: {} 2023-05-31T20:52:11.213 DEBUG:boto:Token: None 2023-05-31T20:52:11.213 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:52:11 GMT /pklxkd-59/obj31 2023-05-31T20:52:11.214 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:ZcKC66GQhofx1RMZgxmca4F4xUo= 2023-05-31T20:52:11.214 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:52:11 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:ZcKC66GQhofx1RMZgxmca4F4xUo=', 'Content-Length': '0'} 2023-05-31T20:52:11.216 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:50:33 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a11651a4fc3dbb1e-006477b37b-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:52:11 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:52:11.216 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj31 2023-05-31T20:52:11.217 DEBUG:boto:path=/pklxkd-59/obj31 2023-05-31T20:52:11.217 DEBUG:boto:auth_path=/pklxkd-59/obj31 2023-05-31T20:52:11.218 DEBUG:boto:Method: GET 2023-05-31T20:52:11.218 DEBUG:boto:Path: /pklxkd-59/obj31 2023-05-31T20:52:11.219 DEBUG:boto:Data: 2023-05-31T20:52:11.219 DEBUG:boto:Headers: {} 2023-05-31T20:52:11.220 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:52:11.220 DEBUG:boto:Port: 8001 2023-05-31T20:52:11.220 DEBUG:boto:Params: {} 2023-05-31T20:52:11.221 DEBUG:boto:Token: None 2023-05-31T20:52:11.221 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:52:11 GMT /pklxkd-59/obj31 2023-05-31T20:52:11.222 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:vRd03+67BYDYOn6U3NbUE28kwCU= 2023-05-31T20:52:11.222 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:52:11 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:vRd03+67BYDYOn6U3NbUE28kwCU=', 'Content-Length': '0'} 2023-05-31T20:52:11.224 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:50:33 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009a46709523fb7581-006477b37b-1c17-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:52:11 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:52:11.225 DEBUG:boto:path=/pklxkd-59/obj31 2023-05-31T20:52:11.225 DEBUG:boto:auth_path=/pklxkd-59/obj31 2023-05-31T20:52:11.226 DEBUG:boto:Method: GET 2023-05-31T20:52:11.226 DEBUG:boto:Path: /pklxkd-59/obj31 2023-05-31T20:52:11.226 DEBUG:boto:Data: 2023-05-31T20:52:11.227 DEBUG:boto:Headers: {} 2023-05-31T20:52:11.227 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:52:11.228 DEBUG:boto:Port: 8000 2023-05-31T20:52:11.228 DEBUG:boto:Params: {} 2023-05-31T20:52:11.228 DEBUG:boto:Token: None 2023-05-31T20:52:11.229 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:52:11 GMT /pklxkd-59/obj31 2023-05-31T20:52:11.229 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:vRd03+67BYDYOn6U3NbUE28kwCU= 2023-05-31T20:52:11.230 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:52:11 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:vRd03+67BYDYOn6U3NbUE28kwCU=', 'Content-Length': '0'} 2023-05-31T20:52:11.232 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:50:33 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c54d6a3c25c0c436-006477b37b-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:52:11 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:52:11.272 DEBUG:boto:path=/pklxkd-59/ 2023-05-31T20:52:11.273 DEBUG:boto:auth_path=/pklxkd-59/ 2023-05-31T20:52:11.273 DEBUG:boto:path=/pklxkd-59/?prefix=obj31 2023-05-31T20:52:11.273 DEBUG:boto:auth_path=/pklxkd-59/?prefix=obj31 2023-05-31T20:52:11.274 DEBUG:boto:Method: GET 2023-05-31T20:52:11.274 DEBUG:boto:Path: /pklxkd-59/?prefix=obj31 2023-05-31T20:52:11.274 DEBUG:boto:Data: 2023-05-31T20:52:11.275 DEBUG:boto:Headers: {} 2023-05-31T20:52:11.275 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:52:11.275 DEBUG:boto:Port: 8001 2023-05-31T20:52:11.276 DEBUG:boto:Params: {} 2023-05-31T20:52:11.276 DEBUG:boto:Token: None 2023-05-31T20:52:11.277 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:52:11 GMT /pklxkd-59/ 2023-05-31T20:52:11.277 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:HITZ+oTJfJ3JS1dgm5XbULiOORY= 2023-05-31T20:52:11.277 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:52:11 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:HITZ+oTJfJ3JS1dgm5XbULiOORY=', 'Content-Length': '0'} 2023-05-31T20:52:11.280 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000006d0e38274cda2a06-006477b37b-1c17-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:52:11 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:52:11.280 DEBUG:boto:b'pklxkd-59obj311000falseobj312023-05-31T20:50:33.460Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:52:11.281 DEBUG:boto:path=/pklxkd-59/ 2023-05-31T20:52:11.281 DEBUG:boto:auth_path=/pklxkd-59/ 2023-05-31T20:52:11.282 DEBUG:boto:path=/pklxkd-59/?prefix=obj31 2023-05-31T20:52:11.282 DEBUG:boto:auth_path=/pklxkd-59/?prefix=obj31 2023-05-31T20:52:11.282 DEBUG:boto:Method: GET 2023-05-31T20:52:11.283 DEBUG:boto:Path: /pklxkd-59/?prefix=obj31 2023-05-31T20:52:11.283 DEBUG:boto:Data: 2023-05-31T20:52:11.283 DEBUG:boto:Headers: {} 2023-05-31T20:52:11.284 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:52:11.284 DEBUG:boto:Port: 8000 2023-05-31T20:52:11.284 DEBUG:boto:Params: {} 2023-05-31T20:52:11.285 DEBUG:boto:Token: None 2023-05-31T20:52:11.285 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:52:11 GMT /pklxkd-59/ 2023-05-31T20:52:11.286 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:HITZ+oTJfJ3JS1dgm5XbULiOORY= 2023-05-31T20:52:11.286 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:52:11 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:HITZ+oTJfJ3JS1dgm5XbULiOORY=', 'Content-Length': '0'} 2023-05-31T20:52:11.290 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000fb6f6ef734028e16-006477b37b-1630-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:52:11 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:52:11.290 DEBUG:boto:b'pklxkd-59obj311000falseobj312023-05-31T20:50:33.460Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:52:11.291 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj33 2023-05-31T20:52:11.291 DEBUG:boto:path=/pklxkd-59/obj33 2023-05-31T20:52:11.292 DEBUG:boto:auth_path=/pklxkd-59/obj33 2023-05-31T20:52:11.292 DEBUG:boto:path=/pklxkd-59/obj33?versionId=null 2023-05-31T20:52:11.292 DEBUG:boto:auth_path=/pklxkd-59/obj33?versionId=null 2023-05-31T20:52:11.293 DEBUG:boto:Method: GET 2023-05-31T20:52:11.293 DEBUG:boto:Path: /pklxkd-59/obj33?versionId=null 2023-05-31T20:52:11.293 DEBUG:boto:Data: 2023-05-31T20:52:11.294 DEBUG:boto:Headers: {} 2023-05-31T20:52:11.294 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:52:11.294 DEBUG:boto:Port: 8001 2023-05-31T20:52:11.295 DEBUG:boto:Params: {} 2023-05-31T20:52:11.295 DEBUG:boto:Token: None 2023-05-31T20:52:11.296 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:52:11 GMT /pklxkd-59/obj33?versionId=null 2023-05-31T20:52:11.296 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:75K/8GsVyos4L8OU9isMOWfLc9U= 2023-05-31T20:52:11.296 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:52:11 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:75K/8GsVyos4L8OU9isMOWfLc9U=', 'Content-Length': '0'} 2023-05-31T20:52:11.298 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:39:26 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d0a4b5b9189d98fa-006477b37b-1c17-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:52:11 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:52:11.299 DEBUG:boto:path=/pklxkd-59/obj33 2023-05-31T20:52:11.299 DEBUG:boto:auth_path=/pklxkd-59/obj33 2023-05-31T20:52:11.300 DEBUG:boto:path=/pklxkd-59/obj33?versionId=null 2023-05-31T20:52:11.300 DEBUG:boto:auth_path=/pklxkd-59/obj33?versionId=null 2023-05-31T20:52:11.301 DEBUG:boto:Method: GET 2023-05-31T20:52:11.301 DEBUG:boto:Path: /pklxkd-59/obj33?versionId=null 2023-05-31T20:52:11.301 DEBUG:boto:Data: 2023-05-31T20:52:11.302 DEBUG:boto:Headers: {} 2023-05-31T20:52:11.302 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:52:11.302 DEBUG:boto:Port: 8000 2023-05-31T20:52:11.303 DEBUG:boto:Params: {} 2023-05-31T20:52:11.303 DEBUG:boto:Token: None 2023-05-31T20:52:11.304 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:52:11 GMT /pklxkd-59/obj33?versionId=null 2023-05-31T20:52:11.304 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:75K/8GsVyos4L8OU9isMOWfLc9U= 2023-05-31T20:52:11.305 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:52:11 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:75K/8GsVyos4L8OU9isMOWfLc9U=', 'Content-Length': '0'} 2023-05-31T20:52:11.307 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:39:26 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000adc3ee4ffd98928d-006477b37b-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:52:11 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:52:11.307 DEBUG:boto:path=/pklxkd-59/obj33 2023-05-31T20:52:11.308 DEBUG:boto:auth_path=/pklxkd-59/obj33 2023-05-31T20:52:11.308 DEBUG:boto:path=/pklxkd-59/obj33?versionId=null 2023-05-31T20:52:11.308 DEBUG:boto:auth_path=/pklxkd-59/obj33?versionId=null 2023-05-31T20:52:11.309 DEBUG:boto:Method: HEAD 2023-05-31T20:52:11.309 DEBUG:boto:Path: /pklxkd-59/obj33?versionId=null 2023-05-31T20:52:11.309 DEBUG:boto:Data: 2023-05-31T20:52:11.310 DEBUG:boto:Headers: {} 2023-05-31T20:52:11.310 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:52:11.310 DEBUG:boto:Port: 8001 2023-05-31T20:52:11.311 DEBUG:boto:Params: {} 2023-05-31T20:52:11.311 DEBUG:boto:Token: None 2023-05-31T20:52:11.311 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:52:11 GMT /pklxkd-59/obj33?versionId=null 2023-05-31T20:52:11.312 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:sz/XJatv2BhA60dtNWNEIgstOZU= 2023-05-31T20:52:11.312 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:52:11 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:sz/XJatv2BhA60dtNWNEIgstOZU=', 'Content-Length': '0'} 2023-05-31T20:52:11.314 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:39:26 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008364eaf91f6ba94b-006477b37b-1c17-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:52:11 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:52:11.315 DEBUG:boto:path=/pklxkd-59/obj33 2023-05-31T20:52:11.315 DEBUG:boto:auth_path=/pklxkd-59/obj33 2023-05-31T20:52:11.315 DEBUG:boto:path=/pklxkd-59/obj33?versionId=null 2023-05-31T20:52:11.316 DEBUG:boto:auth_path=/pklxkd-59/obj33?versionId=null 2023-05-31T20:52:11.316 DEBUG:boto:Method: HEAD 2023-05-31T20:52:11.316 DEBUG:boto:Path: /pklxkd-59/obj33?versionId=null 2023-05-31T20:52:11.317 DEBUG:boto:Data: 2023-05-31T20:52:11.317 DEBUG:boto:Headers: {} 2023-05-31T20:52:11.317 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:52:11.318 DEBUG:boto:Port: 8000 2023-05-31T20:52:11.318 DEBUG:boto:Params: {} 2023-05-31T20:52:11.318 DEBUG:boto:Token: None 2023-05-31T20:52:11.319 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:52:11 GMT /pklxkd-59/obj33?versionId=null 2023-05-31T20:52:11.319 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:sz/XJatv2BhA60dtNWNEIgstOZU= 2023-05-31T20:52:11.320 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:52:11 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:sz/XJatv2BhA60dtNWNEIgstOZU=', 'Content-Length': '0'} 2023-05-31T20:52:11.322 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:39:26 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000235b4b957011bfa1-006477b37b-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:52:11 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:52:11.322 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj33 2023-05-31T20:52:11.323 DEBUG:boto:path=/pklxkd-59/obj33 2023-05-31T20:52:11.323 DEBUG:boto:auth_path=/pklxkd-59/obj33 2023-05-31T20:52:11.323 DEBUG:boto:path=/pklxkd-59/obj33?versionId=null 2023-05-31T20:52:11.324 DEBUG:boto:auth_path=/pklxkd-59/obj33?versionId=null 2023-05-31T20:52:11.324 DEBUG:boto:Method: GET 2023-05-31T20:52:11.324 DEBUG:boto:Path: /pklxkd-59/obj33?versionId=null 2023-05-31T20:52:11.325 DEBUG:boto:Data: 2023-05-31T20:52:11.325 DEBUG:boto:Headers: {} 2023-05-31T20:52:11.325 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:52:11.326 DEBUG:boto:Port: 8001 2023-05-31T20:52:11.326 DEBUG:boto:Params: {} 2023-05-31T20:52:11.326 DEBUG:boto:Token: None 2023-05-31T20:52:11.327 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:52:11 GMT /pklxkd-59/obj33?versionId=null 2023-05-31T20:52:11.327 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:75K/8GsVyos4L8OU9isMOWfLc9U= 2023-05-31T20:52:11.328 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:52:11 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:75K/8GsVyos4L8OU9isMOWfLc9U=', 'Content-Length': '0'} 2023-05-31T20:52:11.329 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:39:26 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009a2132cd8a9a418a-006477b37b-1c17-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:52:11 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:52:11.330 DEBUG:boto:path=/pklxkd-59/obj33 2023-05-31T20:52:11.330 DEBUG:boto:auth_path=/pklxkd-59/obj33 2023-05-31T20:52:11.331 DEBUG:boto:path=/pklxkd-59/obj33?versionId=null 2023-05-31T20:52:11.331 DEBUG:boto:auth_path=/pklxkd-59/obj33?versionId=null 2023-05-31T20:52:11.332 DEBUG:boto:Method: GET 2023-05-31T20:52:11.332 DEBUG:boto:Path: /pklxkd-59/obj33?versionId=null 2023-05-31T20:52:11.332 DEBUG:boto:Data: 2023-05-31T20:52:11.333 DEBUG:boto:Headers: {} 2023-05-31T20:52:11.333 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:52:11.333 DEBUG:boto:Port: 8000 2023-05-31T20:52:11.334 DEBUG:boto:Params: {} 2023-05-31T20:52:11.334 DEBUG:boto:Token: None 2023-05-31T20:52:11.335 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:52:11 GMT /pklxkd-59/obj33?versionId=null 2023-05-31T20:52:11.335 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:75K/8GsVyos4L8OU9isMOWfLc9U= 2023-05-31T20:52:11.335 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:52:11 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:75K/8GsVyos4L8OU9isMOWfLc9U=', 'Content-Length': '0'} 2023-05-31T20:52:11.337 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:39:26 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000da8e707110b9c455-006477b37b-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:52:11 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:52:11.338 DEBUG:boto:path=/pklxkd-59/ 2023-05-31T20:52:11.338 DEBUG:boto:auth_path=/pklxkd-59/ 2023-05-31T20:52:11.339 DEBUG:boto:path=/pklxkd-59/?prefix=obj33 2023-05-31T20:52:11.339 DEBUG:boto:auth_path=/pklxkd-59/?prefix=obj33 2023-05-31T20:52:11.339 DEBUG:boto:Method: GET 2023-05-31T20:52:11.340 DEBUG:boto:Path: /pklxkd-59/?prefix=obj33 2023-05-31T20:52:11.340 DEBUG:boto:Data: 2023-05-31T20:52:11.340 DEBUG:boto:Headers: {} 2023-05-31T20:52:11.341 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:52:11.341 DEBUG:boto:Port: 8001 2023-05-31T20:52:11.341 DEBUG:boto:Params: {} 2023-05-31T20:52:11.342 DEBUG:boto:Token: None 2023-05-31T20:52:11.342 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:52:11 GMT /pklxkd-59/ 2023-05-31T20:52:11.343 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:HITZ+oTJfJ3JS1dgm5XbULiOORY= 2023-05-31T20:52:11.343 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:52:11 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:HITZ+oTJfJ3JS1dgm5XbULiOORY=', 'Content-Length': '0'} 2023-05-31T20:52:11.345 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ad1bee1154e88450-006477b37b-1c17-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:52:11 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:52:11.346 DEBUG:boto:b'pklxkd-59obj331000falseobj332023-05-31T20:39:26.534Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:52:11.346 DEBUG:boto:path=/pklxkd-59/ 2023-05-31T20:52:11.347 DEBUG:boto:auth_path=/pklxkd-59/ 2023-05-31T20:52:11.347 DEBUG:boto:path=/pklxkd-59/?prefix=obj33 2023-05-31T20:52:11.348 DEBUG:boto:auth_path=/pklxkd-59/?prefix=obj33 2023-05-31T20:52:11.348 DEBUG:boto:Method: GET 2023-05-31T20:52:11.348 DEBUG:boto:Path: /pklxkd-59/?prefix=obj33 2023-05-31T20:52:11.349 DEBUG:boto:Data: 2023-05-31T20:52:11.349 DEBUG:boto:Headers: {} 2023-05-31T20:52:11.349 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:52:11.350 DEBUG:boto:Port: 8000 2023-05-31T20:52:11.350 DEBUG:boto:Params: {} 2023-05-31T20:52:11.350 DEBUG:boto:Token: None 2023-05-31T20:52:11.351 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:52:11 GMT /pklxkd-59/ 2023-05-31T20:52:11.351 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:HITZ+oTJfJ3JS1dgm5XbULiOORY= 2023-05-31T20:52:11.352 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:52:11 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:HITZ+oTJfJ3JS1dgm5XbULiOORY=', 'Content-Length': '0'} 2023-05-31T20:52:11.355 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a022ed41a983cae1-006477b37b-1630-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:52:11 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:52:11.396 DEBUG:boto:b'pklxkd-59obj331000falseobj332023-05-31T20:39:26.534Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:52:11.397 DEBUG:boto:path=/pklxkd-59/obj33 2023-05-31T20:52:11.397 DEBUG:boto:auth_path=/pklxkd-59/obj33 2023-05-31T20:52:11.398 DEBUG:boto:Method: HEAD 2023-05-31T20:52:11.398 DEBUG:boto:Path: /pklxkd-59/obj33 2023-05-31T20:52:11.398 DEBUG:boto:Data: 2023-05-31T20:52:11.399 DEBUG:boto:Headers: {} 2023-05-31T20:52:11.399 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:52:11.399 DEBUG:boto:Port: 8001 2023-05-31T20:52:11.399 DEBUG:boto:Params: {} 2023-05-31T20:52:11.400 DEBUG:boto:Token: None 2023-05-31T20:52:11.400 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:52:11 GMT /pklxkd-59/obj33 2023-05-31T20:52:11.401 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:MpCryF5Xig23X8E/oIgF1coifTU= 2023-05-31T20:52:11.401 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:52:11 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:MpCryF5Xig23X8E/oIgF1coifTU=', 'Content-Length': '0'} 2023-05-31T20:52:11.403 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:39:26 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000cd2698f58d713a62-006477b37b-1c17-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:52:11 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:52:11.403 DEBUG:boto:path=/pklxkd-59/obj33 2023-05-31T20:52:11.404 DEBUG:boto:auth_path=/pklxkd-59/obj33 2023-05-31T20:52:11.404 DEBUG:boto:Method: HEAD 2023-05-31T20:52:11.404 DEBUG:boto:Path: /pklxkd-59/obj33 2023-05-31T20:52:11.405 DEBUG:boto:Data: 2023-05-31T20:52:11.405 DEBUG:boto:Headers: {} 2023-05-31T20:52:11.405 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:52:11.406 DEBUG:boto:Port: 8000 2023-05-31T20:52:11.406 DEBUG:boto:Params: {} 2023-05-31T20:52:11.406 DEBUG:boto:Token: None 2023-05-31T20:52:11.407 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:52:11 GMT /pklxkd-59/obj33 2023-05-31T20:52:11.407 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:MpCryF5Xig23X8E/oIgF1coifTU= 2023-05-31T20:52:11.408 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:52:11 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:MpCryF5Xig23X8E/oIgF1coifTU=', 'Content-Length': '0'} 2023-05-31T20:52:11.409 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:39:26 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004e82624eea002ef6-006477b37b-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:52:11 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:52:11.410 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj33 2023-05-31T20:52:11.410 DEBUG:boto:path=/pklxkd-59/obj33 2023-05-31T20:52:11.411 DEBUG:boto:auth_path=/pklxkd-59/obj33 2023-05-31T20:52:11.411 DEBUG:boto:Method: GET 2023-05-31T20:52:11.412 DEBUG:boto:Path: /pklxkd-59/obj33 2023-05-31T20:52:11.412 DEBUG:boto:Data: 2023-05-31T20:52:11.412 DEBUG:boto:Headers: {} 2023-05-31T20:52:11.413 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:52:11.413 DEBUG:boto:Port: 8001 2023-05-31T20:52:11.413 DEBUG:boto:Params: {} 2023-05-31T20:52:11.414 DEBUG:boto:Token: None 2023-05-31T20:52:11.414 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:52:11 GMT /pklxkd-59/obj33 2023-05-31T20:52:11.415 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:LNngZV5iNi3j8GVpoaj3yCs26lE= 2023-05-31T20:52:11.415 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:52:11 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:LNngZV5iNi3j8GVpoaj3yCs26lE=', 'Content-Length': '0'} 2023-05-31T20:52:11.417 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:39:26 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a2f986cccc1e50f8-006477b37b-1c17-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:52:11 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:52:11.417 DEBUG:boto:path=/pklxkd-59/obj33 2023-05-31T20:52:11.418 DEBUG:boto:auth_path=/pklxkd-59/obj33 2023-05-31T20:52:11.418 DEBUG:boto:Method: GET 2023-05-31T20:52:11.419 DEBUG:boto:Path: /pklxkd-59/obj33 2023-05-31T20:52:11.419 DEBUG:boto:Data: 2023-05-31T20:52:11.419 DEBUG:boto:Headers: {} 2023-05-31T20:52:11.420 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:52:11.420 DEBUG:boto:Port: 8000 2023-05-31T20:52:11.420 DEBUG:boto:Params: {} 2023-05-31T20:52:11.421 DEBUG:boto:Token: None 2023-05-31T20:52:11.421 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:52:11 GMT /pklxkd-59/obj33 2023-05-31T20:52:11.422 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:LNngZV5iNi3j8GVpoaj3yCs26lE= 2023-05-31T20:52:11.422 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:52:11 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:LNngZV5iNi3j8GVpoaj3yCs26lE=', 'Content-Length': '0'} 2023-05-31T20:52:11.424 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:39:26 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007677ed8d9d3bd529-006477b37b-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:52:11 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:52:11.424 DEBUG:boto:path=/pklxkd-59/ 2023-05-31T20:52:11.425 DEBUG:boto:auth_path=/pklxkd-59/ 2023-05-31T20:52:11.425 DEBUG:boto:path=/pklxkd-59/?prefix=obj33 2023-05-31T20:52:11.425 DEBUG:boto:auth_path=/pklxkd-59/?prefix=obj33 2023-05-31T20:52:11.426 DEBUG:boto:Method: GET 2023-05-31T20:52:11.426 DEBUG:boto:Path: /pklxkd-59/?prefix=obj33 2023-05-31T20:52:11.426 DEBUG:boto:Data: 2023-05-31T20:52:11.427 DEBUG:boto:Headers: {} 2023-05-31T20:52:11.427 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:52:11.427 DEBUG:boto:Port: 8001 2023-05-31T20:52:11.428 DEBUG:boto:Params: {} 2023-05-31T20:52:11.428 DEBUG:boto:Token: None 2023-05-31T20:52:11.429 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:52:11 GMT /pklxkd-59/ 2023-05-31T20:52:11.429 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:HITZ+oTJfJ3JS1dgm5XbULiOORY= 2023-05-31T20:52:11.429 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:52:11 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:HITZ+oTJfJ3JS1dgm5XbULiOORY=', 'Content-Length': '0'} 2023-05-31T20:52:11.432 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000987f209fa273b637-006477b37b-1c17-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:52:11 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:52:11.432 DEBUG:boto:b'pklxkd-59obj331000falseobj332023-05-31T20:39:26.534Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:52:11.433 DEBUG:boto:path=/pklxkd-59/ 2023-05-31T20:52:11.433 DEBUG:boto:auth_path=/pklxkd-59/ 2023-05-31T20:52:11.434 DEBUG:boto:path=/pklxkd-59/?prefix=obj33 2023-05-31T20:52:11.434 DEBUG:boto:auth_path=/pklxkd-59/?prefix=obj33 2023-05-31T20:52:11.434 DEBUG:boto:Method: GET 2023-05-31T20:52:11.435 DEBUG:boto:Path: /pklxkd-59/?prefix=obj33 2023-05-31T20:52:11.435 DEBUG:boto:Data: 2023-05-31T20:52:11.435 DEBUG:boto:Headers: {} 2023-05-31T20:52:11.436 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:52:11.436 DEBUG:boto:Port: 8000 2023-05-31T20:52:11.436 DEBUG:boto:Params: {} 2023-05-31T20:52:11.437 DEBUG:boto:Token: None 2023-05-31T20:52:11.437 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:52:11 GMT /pklxkd-59/ 2023-05-31T20:52:11.438 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:HITZ+oTJfJ3JS1dgm5XbULiOORY= 2023-05-31T20:52:11.438 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:52:11 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:HITZ+oTJfJ3JS1dgm5XbULiOORY=', 'Content-Length': '0'} 2023-05-31T20:52:11.442 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b6343b7ee58df7cb-006477b37b-1630-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:52:11 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:52:11.442 DEBUG:boto:b'pklxkd-59obj331000falseobj332023-05-31T20:39:26.534Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:52:11.443 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj34 2023-05-31T20:52:11.443 DEBUG:boto:path=/pklxkd-59/obj34 2023-05-31T20:52:11.443 DEBUG:boto:auth_path=/pklxkd-59/obj34 2023-05-31T20:52:11.444 DEBUG:boto:path=/pklxkd-59/obj34?versionId=null 2023-05-31T20:52:11.444 DEBUG:boto:auth_path=/pklxkd-59/obj34?versionId=null 2023-05-31T20:52:11.445 DEBUG:boto:Method: GET 2023-05-31T20:52:11.445 DEBUG:boto:Path: /pklxkd-59/obj34?versionId=null 2023-05-31T20:52:11.445 DEBUG:boto:Data: 2023-05-31T20:52:11.446 DEBUG:boto:Headers: {} 2023-05-31T20:52:11.446 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:52:11.446 DEBUG:boto:Port: 8001 2023-05-31T20:52:11.447 DEBUG:boto:Params: {} 2023-05-31T20:52:11.447 DEBUG:boto:Token: None 2023-05-31T20:52:11.447 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:52:11 GMT /pklxkd-59/obj34?versionId=null 2023-05-31T20:52:11.448 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:goGnkjqZkolNu+6ZFCMxlC3SeY0= 2023-05-31T20:52:11.448 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:52:11 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:goGnkjqZkolNu+6ZFCMxlC3SeY0=', 'Content-Length': '0'} 2023-05-31T20:52:11.450 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:43:07 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000921ff49f176c7463-006477b37b-1c17-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:52:11 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:52:11.451 DEBUG:boto:path=/pklxkd-59/obj34 2023-05-31T20:52:11.451 DEBUG:boto:auth_path=/pklxkd-59/obj34 2023-05-31T20:52:11.452 DEBUG:boto:path=/pklxkd-59/obj34?versionId=null 2023-05-31T20:52:11.452 DEBUG:boto:auth_path=/pklxkd-59/obj34?versionId=null 2023-05-31T20:52:11.453 DEBUG:boto:Method: GET 2023-05-31T20:52:11.453 DEBUG:boto:Path: /pklxkd-59/obj34?versionId=null 2023-05-31T20:52:11.453 DEBUG:boto:Data: 2023-05-31T20:52:11.454 DEBUG:boto:Headers: {} 2023-05-31T20:52:11.454 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:52:11.454 DEBUG:boto:Port: 8000 2023-05-31T20:52:11.455 DEBUG:boto:Params: {} 2023-05-31T20:52:11.455 DEBUG:boto:Token: None 2023-05-31T20:52:11.455 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:52:11 GMT /pklxkd-59/obj34?versionId=null 2023-05-31T20:52:11.456 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:goGnkjqZkolNu+6ZFCMxlC3SeY0= 2023-05-31T20:52:11.456 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:52:11 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:goGnkjqZkolNu+6ZFCMxlC3SeY0=', 'Content-Length': '0'} 2023-05-31T20:52:11.458 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:43:07 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ee1d585dd8b76b8d-006477b37b-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:52:11 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:52:11.459 DEBUG:boto:path=/pklxkd-59/obj34 2023-05-31T20:52:11.459 DEBUG:boto:auth_path=/pklxkd-59/obj34 2023-05-31T20:52:11.460 DEBUG:boto:path=/pklxkd-59/obj34?versionId=null 2023-05-31T20:52:11.460 DEBUG:boto:auth_path=/pklxkd-59/obj34?versionId=null 2023-05-31T20:52:11.460 DEBUG:boto:Method: HEAD 2023-05-31T20:52:11.461 DEBUG:boto:Path: /pklxkd-59/obj34?versionId=null 2023-05-31T20:52:11.461 DEBUG:boto:Data: 2023-05-31T20:52:11.461 DEBUG:boto:Headers: {} 2023-05-31T20:52:11.462 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:52:11.462 DEBUG:boto:Port: 8001 2023-05-31T20:52:11.462 DEBUG:boto:Params: {} 2023-05-31T20:52:11.463 DEBUG:boto:Token: None 2023-05-31T20:52:11.463 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:52:11 GMT /pklxkd-59/obj34?versionId=null 2023-05-31T20:52:11.463 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:ACXj4GdwUZk9fWSAildFgsO4Opo= 2023-05-31T20:52:11.464 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:52:11 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:ACXj4GdwUZk9fWSAildFgsO4Opo=', 'Content-Length': '0'} 2023-05-31T20:52:11.466 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:43:07 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000038220a1224ff1db9-006477b37b-1c17-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:52:11 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:52:11.466 DEBUG:boto:path=/pklxkd-59/obj34 2023-05-31T20:52:11.466 DEBUG:boto:auth_path=/pklxkd-59/obj34 2023-05-31T20:52:11.467 DEBUG:boto:path=/pklxkd-59/obj34?versionId=null 2023-05-31T20:52:11.467 DEBUG:boto:auth_path=/pklxkd-59/obj34?versionId=null 2023-05-31T20:52:11.467 DEBUG:boto:Method: HEAD 2023-05-31T20:52:11.468 DEBUG:boto:Path: /pklxkd-59/obj34?versionId=null 2023-05-31T20:52:11.468 DEBUG:boto:Data: 2023-05-31T20:52:11.469 DEBUG:boto:Headers: {} 2023-05-31T20:52:11.469 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:52:11.469 DEBUG:boto:Port: 8000 2023-05-31T20:52:11.469 DEBUG:boto:Params: {} 2023-05-31T20:52:11.470 DEBUG:boto:Token: None 2023-05-31T20:52:11.470 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:52:11 GMT /pklxkd-59/obj34?versionId=null 2023-05-31T20:52:11.471 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:ACXj4GdwUZk9fWSAildFgsO4Opo= 2023-05-31T20:52:11.471 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:52:11 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:ACXj4GdwUZk9fWSAildFgsO4Opo=', 'Content-Length': '0'} 2023-05-31T20:52:11.473 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:43:07 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000080e142988666e3-006477b37b-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:52:11 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:52:11.473 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj34 2023-05-31T20:52:11.474 DEBUG:boto:path=/pklxkd-59/obj34 2023-05-31T20:52:11.474 DEBUG:boto:auth_path=/pklxkd-59/obj34 2023-05-31T20:52:11.474 DEBUG:boto:path=/pklxkd-59/obj34?versionId=null 2023-05-31T20:52:11.475 DEBUG:boto:auth_path=/pklxkd-59/obj34?versionId=null 2023-05-31T20:52:11.475 DEBUG:boto:Method: GET 2023-05-31T20:52:11.476 DEBUG:boto:Path: /pklxkd-59/obj34?versionId=null 2023-05-31T20:52:11.476 DEBUG:boto:Data: 2023-05-31T20:52:11.476 DEBUG:boto:Headers: {} 2023-05-31T20:52:11.477 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:52:11.477 DEBUG:boto:Port: 8001 2023-05-31T20:52:11.477 DEBUG:boto:Params: {} 2023-05-31T20:52:11.478 DEBUG:boto:Token: None 2023-05-31T20:52:11.478 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:52:11 GMT /pklxkd-59/obj34?versionId=null 2023-05-31T20:52:11.479 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:goGnkjqZkolNu+6ZFCMxlC3SeY0= 2023-05-31T20:52:11.479 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:52:11 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:goGnkjqZkolNu+6ZFCMxlC3SeY0=', 'Content-Length': '0'} 2023-05-31T20:52:11.481 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:43:07 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000013b96f34178216e9-006477b37b-1c17-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:52:11 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:52:11.481 DEBUG:boto:path=/pklxkd-59/obj34 2023-05-31T20:52:11.482 DEBUG:boto:auth_path=/pklxkd-59/obj34 2023-05-31T20:52:11.482 DEBUG:boto:path=/pklxkd-59/obj34?versionId=null 2023-05-31T20:52:11.482 DEBUG:boto:auth_path=/pklxkd-59/obj34?versionId=null 2023-05-31T20:52:11.483 DEBUG:boto:Method: GET 2023-05-31T20:52:11.483 DEBUG:boto:Path: /pklxkd-59/obj34?versionId=null 2023-05-31T20:52:11.483 DEBUG:boto:Data: 2023-05-31T20:52:11.484 DEBUG:boto:Headers: {} 2023-05-31T20:52:11.484 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:52:11.484 DEBUG:boto:Port: 8000 2023-05-31T20:52:11.485 DEBUG:boto:Params: {} 2023-05-31T20:52:11.485 DEBUG:boto:Token: None 2023-05-31T20:52:11.486 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:52:11 GMT /pklxkd-59/obj34?versionId=null 2023-05-31T20:52:11.486 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:goGnkjqZkolNu+6ZFCMxlC3SeY0= 2023-05-31T20:52:11.486 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:52:11 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:goGnkjqZkolNu+6ZFCMxlC3SeY0=', 'Content-Length': '0'} 2023-05-31T20:52:11.488 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:43:07 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004b26d24416d2e756-006477b37b-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:52:11 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:52:11.532 DEBUG:boto:path=/pklxkd-59/ 2023-05-31T20:52:11.533 DEBUG:boto:auth_path=/pklxkd-59/ 2023-05-31T20:52:11.533 DEBUG:boto:path=/pklxkd-59/?prefix=obj34 2023-05-31T20:52:11.533 DEBUG:boto:auth_path=/pklxkd-59/?prefix=obj34 2023-05-31T20:52:11.534 DEBUG:boto:Method: GET 2023-05-31T20:52:11.534 DEBUG:boto:Path: /pklxkd-59/?prefix=obj34 2023-05-31T20:52:11.534 DEBUG:boto:Data: 2023-05-31T20:52:11.535 DEBUG:boto:Headers: {} 2023-05-31T20:52:11.535 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:52:11.536 DEBUG:boto:Port: 8001 2023-05-31T20:52:11.536 DEBUG:boto:Params: {} 2023-05-31T20:52:11.536 DEBUG:boto:Token: None 2023-05-31T20:52:11.537 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:52:11 GMT /pklxkd-59/ 2023-05-31T20:52:11.537 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:HITZ+oTJfJ3JS1dgm5XbULiOORY= 2023-05-31T20:52:11.538 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:52:11 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:HITZ+oTJfJ3JS1dgm5XbULiOORY=', 'Content-Length': '0'} 2023-05-31T20:52:11.540 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b89d92bc7fa5aa8b-006477b37b-1c17-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:52:11 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:52:11.540 DEBUG:boto:b'pklxkd-59obj341000falseobj342023-05-31T20:43:07.994Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:52:11.541 DEBUG:boto:path=/pklxkd-59/ 2023-05-31T20:52:11.541 DEBUG:boto:auth_path=/pklxkd-59/ 2023-05-31T20:52:11.542 DEBUG:boto:path=/pklxkd-59/?prefix=obj34 2023-05-31T20:52:11.542 DEBUG:boto:auth_path=/pklxkd-59/?prefix=obj34 2023-05-31T20:52:11.543 DEBUG:boto:Method: GET 2023-05-31T20:52:11.543 DEBUG:boto:Path: /pklxkd-59/?prefix=obj34 2023-05-31T20:52:11.543 DEBUG:boto:Data: 2023-05-31T20:52:11.544 DEBUG:boto:Headers: {} 2023-05-31T20:52:11.544 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:52:11.544 DEBUG:boto:Port: 8000 2023-05-31T20:52:11.545 DEBUG:boto:Params: {} 2023-05-31T20:52:11.545 DEBUG:boto:Token: None 2023-05-31T20:52:11.545 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:52:11 GMT /pklxkd-59/ 2023-05-31T20:52:11.546 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:HITZ+oTJfJ3JS1dgm5XbULiOORY= 2023-05-31T20:52:11.546 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:52:11 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:HITZ+oTJfJ3JS1dgm5XbULiOORY=', 'Content-Length': '0'} 2023-05-31T20:52:11.550 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a7930fabb18a0326-006477b37b-1630-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:52:11 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:52:11.550 DEBUG:boto:b'pklxkd-59obj341000falseobj342023-05-31T20:43:07.994Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:52:11.551 DEBUG:boto:path=/pklxkd-59/obj34 2023-05-31T20:52:11.551 DEBUG:boto:auth_path=/pklxkd-59/obj34 2023-05-31T20:52:11.552 DEBUG:boto:Method: HEAD 2023-05-31T20:52:11.552 DEBUG:boto:Path: /pklxkd-59/obj34 2023-05-31T20:52:11.552 DEBUG:boto:Data: 2023-05-31T20:52:11.553 DEBUG:boto:Headers: {} 2023-05-31T20:52:11.553 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:52:11.553 DEBUG:boto:Port: 8001 2023-05-31T20:52:11.554 DEBUG:boto:Params: {} 2023-05-31T20:52:11.554 DEBUG:boto:Token: None 2023-05-31T20:52:11.554 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:52:11 GMT /pklxkd-59/obj34 2023-05-31T20:52:11.555 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:MqSgFXeDslDoNKvj1NnPBxEFU98= 2023-05-31T20:52:11.555 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:52:11 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:MqSgFXeDslDoNKvj1NnPBxEFU98=', 'Content-Length': '0'} 2023-05-31T20:52:11.557 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:43:07 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000bcf8a88d5e23f06d-006477b37b-1c17-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:52:11 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:52:11.558 DEBUG:boto:path=/pklxkd-59/obj34 2023-05-31T20:52:11.558 DEBUG:boto:auth_path=/pklxkd-59/obj34 2023-05-31T20:52:11.558 DEBUG:boto:Method: HEAD 2023-05-31T20:52:11.559 DEBUG:boto:Path: /pklxkd-59/obj34 2023-05-31T20:52:11.559 DEBUG:boto:Data: 2023-05-31T20:52:11.559 DEBUG:boto:Headers: {} 2023-05-31T20:52:11.560 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:52:11.560 DEBUG:boto:Port: 8000 2023-05-31T20:52:11.560 DEBUG:boto:Params: {} 2023-05-31T20:52:11.561 DEBUG:boto:Token: None 2023-05-31T20:52:11.561 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:52:11 GMT /pklxkd-59/obj34 2023-05-31T20:52:11.561 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:MqSgFXeDslDoNKvj1NnPBxEFU98= 2023-05-31T20:52:11.562 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:52:11 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:MqSgFXeDslDoNKvj1NnPBxEFU98=', 'Content-Length': '0'} 2023-05-31T20:52:11.563 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:43:07 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005ab9ba44c4028a9a-006477b37b-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:52:11 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:52:11.564 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj34 2023-05-31T20:52:11.564 DEBUG:boto:path=/pklxkd-59/obj34 2023-05-31T20:52:11.565 DEBUG:boto:auth_path=/pklxkd-59/obj34 2023-05-31T20:52:11.565 DEBUG:boto:Method: GET 2023-05-31T20:52:11.566 DEBUG:boto:Path: /pklxkd-59/obj34 2023-05-31T20:52:11.566 DEBUG:boto:Data: 2023-05-31T20:52:11.566 DEBUG:boto:Headers: {} 2023-05-31T20:52:11.567 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:52:11.567 DEBUG:boto:Port: 8001 2023-05-31T20:52:11.567 DEBUG:boto:Params: {} 2023-05-31T20:52:11.568 DEBUG:boto:Token: None 2023-05-31T20:52:11.568 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:52:11 GMT /pklxkd-59/obj34 2023-05-31T20:52:11.569 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:o3vELuC2LslCGT/52Ic2/p/MVKM= 2023-05-31T20:52:11.569 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:52:11 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:o3vELuC2LslCGT/52Ic2/p/MVKM=', 'Content-Length': '0'} 2023-05-31T20:52:11.571 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:43:07 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006f8e40788a1e0700-006477b37b-1c17-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:52:11 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:52:11.612 DEBUG:boto:path=/pklxkd-59/obj34 2023-05-31T20:52:11.613 DEBUG:boto:auth_path=/pklxkd-59/obj34 2023-05-31T20:52:11.613 DEBUG:boto:Method: GET 2023-05-31T20:52:11.613 DEBUG:boto:Path: /pklxkd-59/obj34 2023-05-31T20:52:11.614 DEBUG:boto:Data: 2023-05-31T20:52:11.614 DEBUG:boto:Headers: {} 2023-05-31T20:52:11.614 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:52:11.615 DEBUG:boto:Port: 8000 2023-05-31T20:52:11.615 DEBUG:boto:Params: {} 2023-05-31T20:52:11.615 DEBUG:boto:Token: None 2023-05-31T20:52:11.616 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:52:11 GMT /pklxkd-59/obj34 2023-05-31T20:52:11.616 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:o3vELuC2LslCGT/52Ic2/p/MVKM= 2023-05-31T20:52:11.617 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:52:11 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:o3vELuC2LslCGT/52Ic2/p/MVKM=', 'Content-Length': '0'} 2023-05-31T20:52:11.619 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:43:07 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000495dcf513635b243-006477b37b-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:52:11 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:52:11.619 DEBUG:boto:path=/pklxkd-59/ 2023-05-31T20:52:11.620 DEBUG:boto:auth_path=/pklxkd-59/ 2023-05-31T20:52:11.620 DEBUG:boto:path=/pklxkd-59/?prefix=obj34 2023-05-31T20:52:11.620 DEBUG:boto:auth_path=/pklxkd-59/?prefix=obj34 2023-05-31T20:52:11.621 DEBUG:boto:Method: GET 2023-05-31T20:52:11.621 DEBUG:boto:Path: /pklxkd-59/?prefix=obj34 2023-05-31T20:52:11.621 DEBUG:boto:Data: 2023-05-31T20:52:11.622 DEBUG:boto:Headers: {} 2023-05-31T20:52:11.622 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:52:11.622 DEBUG:boto:Port: 8001 2023-05-31T20:52:11.623 DEBUG:boto:Params: {} 2023-05-31T20:52:11.623 DEBUG:boto:Token: None 2023-05-31T20:52:11.624 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:52:11 GMT /pklxkd-59/ 2023-05-31T20:52:11.624 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:HITZ+oTJfJ3JS1dgm5XbULiOORY= 2023-05-31T20:52:11.624 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:52:11 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:HITZ+oTJfJ3JS1dgm5XbULiOORY=', 'Content-Length': '0'} 2023-05-31T20:52:11.627 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000cb01969d7e8b82bf-006477b37b-1c17-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:52:11 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:52:11.627 DEBUG:boto:b'pklxkd-59obj341000falseobj342023-05-31T20:43:07.994Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:52:11.628 DEBUG:boto:path=/pklxkd-59/ 2023-05-31T20:52:11.628 DEBUG:boto:auth_path=/pklxkd-59/ 2023-05-31T20:52:11.629 DEBUG:boto:path=/pklxkd-59/?prefix=obj34 2023-05-31T20:52:11.629 DEBUG:boto:auth_path=/pklxkd-59/?prefix=obj34 2023-05-31T20:52:11.629 DEBUG:boto:Method: GET 2023-05-31T20:52:11.630 DEBUG:boto:Path: /pklxkd-59/?prefix=obj34 2023-05-31T20:52:11.630 DEBUG:boto:Data: 2023-05-31T20:52:11.630 DEBUG:boto:Headers: {} 2023-05-31T20:52:11.631 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:52:11.631 DEBUG:boto:Port: 8000 2023-05-31T20:52:11.631 DEBUG:boto:Params: {} 2023-05-31T20:52:11.632 DEBUG:boto:Token: None 2023-05-31T20:52:11.632 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:52:11 GMT /pklxkd-59/ 2023-05-31T20:52:11.633 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:HITZ+oTJfJ3JS1dgm5XbULiOORY= 2023-05-31T20:52:11.633 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:52:11 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:HITZ+oTJfJ3JS1dgm5XbULiOORY=', 'Content-Length': '0'} 2023-05-31T20:52:11.636 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000bdd1ca3b3923b798-006477b37b-1630-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:52:11 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:52:11.637 DEBUG:boto:b'pklxkd-59obj341000falseobj342023-05-31T20:43:07.994Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:52:11.637 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj38 2023-05-31T20:52:11.638 DEBUG:boto:path=/pklxkd-59/obj38 2023-05-31T20:52:11.638 DEBUG:boto:auth_path=/pklxkd-59/obj38 2023-05-31T20:52:11.639 DEBUG:boto:path=/pklxkd-59/obj38?versionId=null 2023-05-31T20:52:11.639 DEBUG:boto:auth_path=/pklxkd-59/obj38?versionId=null 2023-05-31T20:52:11.639 DEBUG:boto:Method: GET 2023-05-31T20:52:11.640 DEBUG:boto:Path: /pklxkd-59/obj38?versionId=null 2023-05-31T20:52:11.640 DEBUG:boto:Data: 2023-05-31T20:52:11.640 DEBUG:boto:Headers: {} 2023-05-31T20:52:11.641 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:52:11.641 DEBUG:boto:Port: 8001 2023-05-31T20:52:11.641 DEBUG:boto:Params: {} 2023-05-31T20:52:11.642 DEBUG:boto:Token: None 2023-05-31T20:52:11.642 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:52:11 GMT /pklxkd-59/obj38?versionId=null 2023-05-31T20:52:11.643 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:z9hf38+vFXO4kfXYBTVZZbOoPkk= 2023-05-31T20:52:11.643 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:52:11 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:z9hf38+vFXO4kfXYBTVZZbOoPkk=', 'Content-Length': '0'} 2023-05-31T20:52:11.645 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:44:50 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000013fcd4ff281464dc-006477b37b-1c17-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:52:11 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:52:11.646 DEBUG:boto:path=/pklxkd-59/obj38 2023-05-31T20:52:11.646 DEBUG:boto:auth_path=/pklxkd-59/obj38 2023-05-31T20:52:11.646 DEBUG:boto:path=/pklxkd-59/obj38?versionId=null 2023-05-31T20:52:11.647 DEBUG:boto:auth_path=/pklxkd-59/obj38?versionId=null 2023-05-31T20:52:11.647 DEBUG:boto:Method: GET 2023-05-31T20:52:11.647 DEBUG:boto:Path: /pklxkd-59/obj38?versionId=null 2023-05-31T20:52:11.648 DEBUG:boto:Data: 2023-05-31T20:52:11.648 DEBUG:boto:Headers: {} 2023-05-31T20:52:11.649 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:52:11.649 DEBUG:boto:Port: 8000 2023-05-31T20:52:11.649 DEBUG:boto:Params: {} 2023-05-31T20:52:11.650 DEBUG:boto:Token: None 2023-05-31T20:52:11.650 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:52:11 GMT /pklxkd-59/obj38?versionId=null 2023-05-31T20:52:11.650 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:z9hf38+vFXO4kfXYBTVZZbOoPkk= 2023-05-31T20:52:11.651 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:52:11 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:z9hf38+vFXO4kfXYBTVZZbOoPkk=', 'Content-Length': '0'} 2023-05-31T20:52:11.653 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:44:50 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005b84ccede9426b44-006477b37b-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:52:11 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:52:11.696 DEBUG:boto:path=/pklxkd-59/obj38 2023-05-31T20:52:11.697 DEBUG:boto:auth_path=/pklxkd-59/obj38 2023-05-31T20:52:11.697 DEBUG:boto:path=/pklxkd-59/obj38?versionId=null 2023-05-31T20:52:11.697 DEBUG:boto:auth_path=/pklxkd-59/obj38?versionId=null 2023-05-31T20:52:11.698 DEBUG:boto:Method: HEAD 2023-05-31T20:52:11.698 DEBUG:boto:Path: /pklxkd-59/obj38?versionId=null 2023-05-31T20:52:11.698 DEBUG:boto:Data: 2023-05-31T20:52:11.699 DEBUG:boto:Headers: {} 2023-05-31T20:52:11.699 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:52:11.699 DEBUG:boto:Port: 8001 2023-05-31T20:52:11.700 DEBUG:boto:Params: {} 2023-05-31T20:52:11.700 DEBUG:boto:Token: None 2023-05-31T20:52:11.700 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:52:11 GMT /pklxkd-59/obj38?versionId=null 2023-05-31T20:52:11.701 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:Ep0j7X2S8PDtbquwa07/C0pN6oY= 2023-05-31T20:52:11.701 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:52:11 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:Ep0j7X2S8PDtbquwa07/C0pN6oY=', 'Content-Length': '0'} 2023-05-31T20:52:11.703 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:44:50 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000363a769c979919a8-006477b37b-1c17-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:52:11 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:52:11.703 DEBUG:boto:path=/pklxkd-59/obj38 2023-05-31T20:52:11.704 DEBUG:boto:auth_path=/pklxkd-59/obj38 2023-05-31T20:52:11.704 DEBUG:boto:path=/pklxkd-59/obj38?versionId=null 2023-05-31T20:52:11.705 DEBUG:boto:auth_path=/pklxkd-59/obj38?versionId=null 2023-05-31T20:52:11.705 DEBUG:boto:Method: HEAD 2023-05-31T20:52:11.705 DEBUG:boto:Path: /pklxkd-59/obj38?versionId=null 2023-05-31T20:52:11.706 DEBUG:boto:Data: 2023-05-31T20:52:11.706 DEBUG:boto:Headers: {} 2023-05-31T20:52:11.706 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:52:11.707 DEBUG:boto:Port: 8000 2023-05-31T20:52:11.707 DEBUG:boto:Params: {} 2023-05-31T20:52:11.707 DEBUG:boto:Token: None 2023-05-31T20:52:11.708 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:52:11 GMT /pklxkd-59/obj38?versionId=null 2023-05-31T20:52:11.708 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:Ep0j7X2S8PDtbquwa07/C0pN6oY= 2023-05-31T20:52:11.708 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:52:11 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:Ep0j7X2S8PDtbquwa07/C0pN6oY=', 'Content-Length': '0'} 2023-05-31T20:52:11.710 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:44:50 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003f5392b3923b4849-006477b37b-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:52:11 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:52:11.711 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj38 2023-05-31T20:52:11.711 DEBUG:boto:path=/pklxkd-59/obj38 2023-05-31T20:52:11.712 DEBUG:boto:auth_path=/pklxkd-59/obj38 2023-05-31T20:52:11.712 DEBUG:boto:path=/pklxkd-59/obj38?versionId=null 2023-05-31T20:52:11.712 DEBUG:boto:auth_path=/pklxkd-59/obj38?versionId=null 2023-05-31T20:52:11.713 DEBUG:boto:Method: GET 2023-05-31T20:52:11.713 DEBUG:boto:Path: /pklxkd-59/obj38?versionId=null 2023-05-31T20:52:11.713 DEBUG:boto:Data: 2023-05-31T20:52:11.714 DEBUG:boto:Headers: {} 2023-05-31T20:52:11.714 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:52:11.714 DEBUG:boto:Port: 8001 2023-05-31T20:52:11.715 DEBUG:boto:Params: {} 2023-05-31T20:52:11.715 DEBUG:boto:Token: None 2023-05-31T20:52:11.715 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:52:11 GMT /pklxkd-59/obj38?versionId=null 2023-05-31T20:52:11.716 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:z9hf38+vFXO4kfXYBTVZZbOoPkk= 2023-05-31T20:52:11.716 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:52:11 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:z9hf38+vFXO4kfXYBTVZZbOoPkk=', 'Content-Length': '0'} 2023-05-31T20:52:11.718 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:44:50 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007c6ed872b0cf321f-006477b37b-1c17-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:52:11 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:52:11.719 DEBUG:boto:path=/pklxkd-59/obj38 2023-05-31T20:52:11.719 DEBUG:boto:auth_path=/pklxkd-59/obj38 2023-05-31T20:52:11.720 DEBUG:boto:path=/pklxkd-59/obj38?versionId=null 2023-05-31T20:52:11.720 DEBUG:boto:auth_path=/pklxkd-59/obj38?versionId=null 2023-05-31T20:52:11.720 DEBUG:boto:Method: GET 2023-05-31T20:52:11.721 DEBUG:boto:Path: /pklxkd-59/obj38?versionId=null 2023-05-31T20:52:11.721 DEBUG:boto:Data: 2023-05-31T20:52:11.721 DEBUG:boto:Headers: {} 2023-05-31T20:52:11.722 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:52:11.722 DEBUG:boto:Port: 8000 2023-05-31T20:52:11.722 DEBUG:boto:Params: {} 2023-05-31T20:52:11.723 DEBUG:boto:Token: None 2023-05-31T20:52:11.723 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:52:11 GMT /pklxkd-59/obj38?versionId=null 2023-05-31T20:52:11.724 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:z9hf38+vFXO4kfXYBTVZZbOoPkk= 2023-05-31T20:52:11.724 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:52:11 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:z9hf38+vFXO4kfXYBTVZZbOoPkk=', 'Content-Length': '0'} 2023-05-31T20:52:11.726 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:44:50 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000097bb9754daaca4c0-006477b37b-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:52:11 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:52:11.727 DEBUG:boto:path=/pklxkd-59/ 2023-05-31T20:52:11.727 DEBUG:boto:auth_path=/pklxkd-59/ 2023-05-31T20:52:11.727 DEBUG:boto:path=/pklxkd-59/?prefix=obj38 2023-05-31T20:52:11.728 DEBUG:boto:auth_path=/pklxkd-59/?prefix=obj38 2023-05-31T20:52:11.728 DEBUG:boto:Method: GET 2023-05-31T20:52:11.728 DEBUG:boto:Path: /pklxkd-59/?prefix=obj38 2023-05-31T20:52:11.729 DEBUG:boto:Data: 2023-05-31T20:52:11.729 DEBUG:boto:Headers: {} 2023-05-31T20:52:11.729 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:52:11.730 DEBUG:boto:Port: 8001 2023-05-31T20:52:11.730 DEBUG:boto:Params: {} 2023-05-31T20:52:11.731 DEBUG:boto:Token: None 2023-05-31T20:52:11.731 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:52:11 GMT /pklxkd-59/ 2023-05-31T20:52:11.731 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:HITZ+oTJfJ3JS1dgm5XbULiOORY= 2023-05-31T20:52:11.732 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:52:11 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:HITZ+oTJfJ3JS1dgm5XbULiOORY=', 'Content-Length': '0'} 2023-05-31T20:52:11.734 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000055d910d6e45e41cd-006477b37b-1c17-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:52:11 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:52:11.735 DEBUG:boto:b'pklxkd-59obj381000falseobj382023-05-31T20:44:50.610Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:52:11.735 DEBUG:boto:path=/pklxkd-59/ 2023-05-31T20:52:11.736 DEBUG:boto:auth_path=/pklxkd-59/ 2023-05-31T20:52:11.736 DEBUG:boto:path=/pklxkd-59/?prefix=obj38 2023-05-31T20:52:11.737 DEBUG:boto:auth_path=/pklxkd-59/?prefix=obj38 2023-05-31T20:52:11.737 DEBUG:boto:Method: GET 2023-05-31T20:52:11.737 DEBUG:boto:Path: /pklxkd-59/?prefix=obj38 2023-05-31T20:52:11.738 DEBUG:boto:Data: 2023-05-31T20:52:11.738 DEBUG:boto:Headers: {} 2023-05-31T20:52:11.738 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:52:11.739 DEBUG:boto:Port: 8000 2023-05-31T20:52:11.739 DEBUG:boto:Params: {} 2023-05-31T20:52:11.739 DEBUG:boto:Token: None 2023-05-31T20:52:11.740 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:52:11 GMT /pklxkd-59/ 2023-05-31T20:52:11.740 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:HITZ+oTJfJ3JS1dgm5XbULiOORY= 2023-05-31T20:52:11.741 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:52:11 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:HITZ+oTJfJ3JS1dgm5XbULiOORY=', 'Content-Length': '0'} 2023-05-31T20:52:11.744 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c3219cee9b08c115-006477b37b-1630-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:52:11 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:52:11.744 DEBUG:boto:b'pklxkd-59obj381000falseobj382023-05-31T20:44:50.610Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:52:11.745 DEBUG:boto:path=/pklxkd-59/obj38 2023-05-31T20:52:11.745 DEBUG:boto:auth_path=/pklxkd-59/obj38 2023-05-31T20:52:11.746 DEBUG:boto:Method: HEAD 2023-05-31T20:52:11.746 DEBUG:boto:Path: /pklxkd-59/obj38 2023-05-31T20:52:11.746 DEBUG:boto:Data: 2023-05-31T20:52:11.747 DEBUG:boto:Headers: {} 2023-05-31T20:52:11.747 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:52:11.747 DEBUG:boto:Port: 8001 2023-05-31T20:52:11.748 DEBUG:boto:Params: {} 2023-05-31T20:52:11.748 DEBUG:boto:Token: None 2023-05-31T20:52:11.748 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:52:11 GMT /pklxkd-59/obj38 2023-05-31T20:52:11.749 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:ppiqm/lWMZhmUTmBEaMm1GWZefw= 2023-05-31T20:52:11.749 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:52:11 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:ppiqm/lWMZhmUTmBEaMm1GWZefw=', 'Content-Length': '0'} 2023-05-31T20:52:11.751 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:44:50 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000083450b9fc6ea4ccb-006477b37b-1c17-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:52:11 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:52:11.752 DEBUG:boto:path=/pklxkd-59/obj38 2023-05-31T20:52:11.752 DEBUG:boto:auth_path=/pklxkd-59/obj38 2023-05-31T20:52:11.752 DEBUG:boto:Method: HEAD 2023-05-31T20:52:11.753 DEBUG:boto:Path: /pklxkd-59/obj38 2023-05-31T20:52:11.753 DEBUG:boto:Data: 2023-05-31T20:52:11.753 DEBUG:boto:Headers: {} 2023-05-31T20:52:11.754 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:52:11.754 DEBUG:boto:Port: 8000 2023-05-31T20:52:11.754 DEBUG:boto:Params: {} 2023-05-31T20:52:11.755 DEBUG:boto:Token: None 2023-05-31T20:52:11.755 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:52:11 GMT /pklxkd-59/obj38 2023-05-31T20:52:11.755 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:ppiqm/lWMZhmUTmBEaMm1GWZefw= 2023-05-31T20:52:11.756 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:52:11 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:ppiqm/lWMZhmUTmBEaMm1GWZefw=', 'Content-Length': '0'} 2023-05-31T20:52:11.757 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:44:50 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009593078cf95fbab9-006477b37b-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:52:11 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:52:11.758 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj38 2023-05-31T20:52:11.758 DEBUG:boto:path=/pklxkd-59/obj38 2023-05-31T20:52:11.759 DEBUG:boto:auth_path=/pklxkd-59/obj38 2023-05-31T20:52:11.759 DEBUG:boto:Method: GET 2023-05-31T20:52:11.759 DEBUG:boto:Path: /pklxkd-59/obj38 2023-05-31T20:52:11.760 DEBUG:boto:Data: 2023-05-31T20:52:11.760 DEBUG:boto:Headers: {} 2023-05-31T20:52:11.761 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:52:11.761 DEBUG:boto:Port: 8001 2023-05-31T20:52:11.761 DEBUG:boto:Params: {} 2023-05-31T20:52:11.762 DEBUG:boto:Token: None 2023-05-31T20:52:11.762 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:52:11 GMT /pklxkd-59/obj38 2023-05-31T20:52:11.763 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:YzoTwh95bIPOHy2HfOCLw5oR3hw= 2023-05-31T20:52:11.763 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:52:11 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:YzoTwh95bIPOHy2HfOCLw5oR3hw=', 'Content-Length': '0'} 2023-05-31T20:52:11.765 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:44:50 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000cea491bf722a844e-006477b37b-1c17-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:52:11 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:52:11.765 DEBUG:boto:path=/pklxkd-59/obj38 2023-05-31T20:52:11.766 DEBUG:boto:auth_path=/pklxkd-59/obj38 2023-05-31T20:52:11.766 DEBUG:boto:Method: GET 2023-05-31T20:52:11.767 DEBUG:boto:Path: /pklxkd-59/obj38 2023-05-31T20:52:11.767 DEBUG:boto:Data: 2023-05-31T20:52:11.767 DEBUG:boto:Headers: {} 2023-05-31T20:52:11.768 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:52:11.768 DEBUG:boto:Port: 8000 2023-05-31T20:52:11.768 DEBUG:boto:Params: {} 2023-05-31T20:52:11.769 DEBUG:boto:Token: None 2023-05-31T20:52:11.769 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:52:11 GMT /pklxkd-59/obj38 2023-05-31T20:52:11.770 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:YzoTwh95bIPOHy2HfOCLw5oR3hw= 2023-05-31T20:52:11.770 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:52:11 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:YzoTwh95bIPOHy2HfOCLw5oR3hw=', 'Content-Length': '0'} 2023-05-31T20:52:11.772 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:44:50 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000bdbc504e358d042-006477b37b-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:52:11 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:52:11.772 DEBUG:boto:path=/pklxkd-59/ 2023-05-31T20:52:11.773 DEBUG:boto:auth_path=/pklxkd-59/ 2023-05-31T20:52:11.773 DEBUG:boto:path=/pklxkd-59/?prefix=obj38 2023-05-31T20:52:11.773 DEBUG:boto:auth_path=/pklxkd-59/?prefix=obj38 2023-05-31T20:52:11.774 DEBUG:boto:Method: GET 2023-05-31T20:52:11.774 DEBUG:boto:Path: /pklxkd-59/?prefix=obj38 2023-05-31T20:52:11.774 DEBUG:boto:Data: 2023-05-31T20:52:11.775 DEBUG:boto:Headers: {} 2023-05-31T20:52:11.775 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:52:11.775 DEBUG:boto:Port: 8001 2023-05-31T20:52:11.776 DEBUG:boto:Params: {} 2023-05-31T20:52:11.776 DEBUG:boto:Token: None 2023-05-31T20:52:11.777 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:52:11 GMT /pklxkd-59/ 2023-05-31T20:52:11.777 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:HITZ+oTJfJ3JS1dgm5XbULiOORY= 2023-05-31T20:52:11.777 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:52:11 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:HITZ+oTJfJ3JS1dgm5XbULiOORY=', 'Content-Length': '0'} 2023-05-31T20:52:11.780 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000be7ecfe2750bcf8f-006477b37b-1c17-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:52:11 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:52:11.780 DEBUG:boto:b'pklxkd-59obj381000falseobj382023-05-31T20:44:50.610Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:52:11.781 DEBUG:boto:path=/pklxkd-59/ 2023-05-31T20:52:11.781 DEBUG:boto:auth_path=/pklxkd-59/ 2023-05-31T20:52:11.782 DEBUG:boto:path=/pklxkd-59/?prefix=obj38 2023-05-31T20:52:11.782 DEBUG:boto:auth_path=/pklxkd-59/?prefix=obj38 2023-05-31T20:52:11.782 DEBUG:boto:Method: GET 2023-05-31T20:52:11.783 DEBUG:boto:Path: /pklxkd-59/?prefix=obj38 2023-05-31T20:52:11.783 DEBUG:boto:Data: 2023-05-31T20:52:11.783 DEBUG:boto:Headers: {} 2023-05-31T20:52:11.784 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:52:11.784 DEBUG:boto:Port: 8000 2023-05-31T20:52:11.784 DEBUG:boto:Params: {} 2023-05-31T20:52:11.785 DEBUG:boto:Token: None 2023-05-31T20:52:11.785 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:52:11 GMT /pklxkd-59/ 2023-05-31T20:52:11.786 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:HITZ+oTJfJ3JS1dgm5XbULiOORY= 2023-05-31T20:52:11.786 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:52:11 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:HITZ+oTJfJ3JS1dgm5XbULiOORY=', 'Content-Length': '0'} 2023-05-31T20:52:11.789 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ec067eac8cb6a799-006477b37b-1630-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:52:11 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:52:11.790 DEBUG:boto:b'pklxkd-59obj381000falseobj382023-05-31T20:44:50.610Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:52:11.791 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj44 2023-05-31T20:52:11.791 DEBUG:boto:path=/pklxkd-59/obj44 2023-05-31T20:52:11.791 DEBUG:boto:auth_path=/pklxkd-59/obj44 2023-05-31T20:52:11.792 DEBUG:boto:path=/pklxkd-59/obj44?versionId=null 2023-05-31T20:52:11.792 DEBUG:boto:auth_path=/pklxkd-59/obj44?versionId=null 2023-05-31T20:52:11.793 DEBUG:boto:Method: GET 2023-05-31T20:52:11.793 DEBUG:boto:Path: /pklxkd-59/obj44?versionId=null 2023-05-31T20:52:11.793 DEBUG:boto:Data: 2023-05-31T20:52:11.794 DEBUG:boto:Headers: {} 2023-05-31T20:52:11.794 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:52:11.794 DEBUG:boto:Port: 8001 2023-05-31T20:52:11.795 DEBUG:boto:Params: {} 2023-05-31T20:52:11.795 DEBUG:boto:Token: None 2023-05-31T20:52:11.795 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:52:11 GMT /pklxkd-59/obj44?versionId=null 2023-05-31T20:52:11.796 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:Erl/WvrRe5XmVNxcaY7m9pNHkAg= 2023-05-31T20:52:11.796 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:52:11 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:Erl/WvrRe5XmVNxcaY7m9pNHkAg=', 'Content-Length': '0'} 2023-05-31T20:52:11.798 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:39:26 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000032611d3b42089cac-006477b37b-1c17-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:52:11 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:52:11.799 DEBUG:boto:path=/pklxkd-59/obj44 2023-05-31T20:52:11.799 DEBUG:boto:auth_path=/pklxkd-59/obj44 2023-05-31T20:52:11.799 DEBUG:boto:path=/pklxkd-59/obj44?versionId=null 2023-05-31T20:52:11.800 DEBUG:boto:auth_path=/pklxkd-59/obj44?versionId=null 2023-05-31T20:52:11.800 DEBUG:boto:Method: GET 2023-05-31T20:52:11.800 DEBUG:boto:Path: /pklxkd-59/obj44?versionId=null 2023-05-31T20:52:11.801 DEBUG:boto:Data: 2023-05-31T20:52:11.801 DEBUG:boto:Headers: {} 2023-05-31T20:52:11.802 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:52:11.802 DEBUG:boto:Port: 8000 2023-05-31T20:52:11.802 DEBUG:boto:Params: {} 2023-05-31T20:52:11.803 DEBUG:boto:Token: None 2023-05-31T20:52:11.803 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:52:11 GMT /pklxkd-59/obj44?versionId=null 2023-05-31T20:52:11.803 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:Erl/WvrRe5XmVNxcaY7m9pNHkAg= 2023-05-31T20:52:11.804 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:52:11 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:Erl/WvrRe5XmVNxcaY7m9pNHkAg=', 'Content-Length': '0'} 2023-05-31T20:52:11.806 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:39:26 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004e4295c931e67df8-006477b37b-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:52:11 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:52:11.806 DEBUG:boto:path=/pklxkd-59/obj44 2023-05-31T20:52:11.807 DEBUG:boto:auth_path=/pklxkd-59/obj44 2023-05-31T20:52:11.807 DEBUG:boto:path=/pklxkd-59/obj44?versionId=null 2023-05-31T20:52:11.807 DEBUG:boto:auth_path=/pklxkd-59/obj44?versionId=null 2023-05-31T20:52:11.808 DEBUG:boto:Method: HEAD 2023-05-31T20:52:11.808 DEBUG:boto:Path: /pklxkd-59/obj44?versionId=null 2023-05-31T20:52:11.808 DEBUG:boto:Data: 2023-05-31T20:52:11.809 DEBUG:boto:Headers: {} 2023-05-31T20:52:11.809 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:52:11.809 DEBUG:boto:Port: 8001 2023-05-31T20:52:11.810 DEBUG:boto:Params: {} 2023-05-31T20:52:11.810 DEBUG:boto:Token: None 2023-05-31T20:52:11.810 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:52:11 GMT /pklxkd-59/obj44?versionId=null 2023-05-31T20:52:11.811 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:hWAmN4ndnKogQw2JgMbiVCM6xoo= 2023-05-31T20:52:11.811 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:52:11 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:hWAmN4ndnKogQw2JgMbiVCM6xoo=', 'Content-Length': '0'} 2023-05-31T20:52:11.813 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:39:26 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f0d69c94702d0999-006477b37b-1c17-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:52:11 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:52:11.813 DEBUG:boto:path=/pklxkd-59/obj44 2023-05-31T20:52:11.814 DEBUG:boto:auth_path=/pklxkd-59/obj44 2023-05-31T20:52:11.814 DEBUG:boto:path=/pklxkd-59/obj44?versionId=null 2023-05-31T20:52:11.814 DEBUG:boto:auth_path=/pklxkd-59/obj44?versionId=null 2023-05-31T20:52:11.815 DEBUG:boto:Method: HEAD 2023-05-31T20:52:11.815 DEBUG:boto:Path: /pklxkd-59/obj44?versionId=null 2023-05-31T20:52:11.815 DEBUG:boto:Data: 2023-05-31T20:52:11.816 DEBUG:boto:Headers: {} 2023-05-31T20:52:11.816 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:52:11.816 DEBUG:boto:Port: 8000 2023-05-31T20:52:11.817 DEBUG:boto:Params: {} 2023-05-31T20:52:11.817 DEBUG:boto:Token: None 2023-05-31T20:52:11.817 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:52:11 GMT /pklxkd-59/obj44?versionId=null 2023-05-31T20:52:11.818 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:hWAmN4ndnKogQw2JgMbiVCM6xoo= 2023-05-31T20:52:11.818 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:52:11 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:hWAmN4ndnKogQw2JgMbiVCM6xoo=', 'Content-Length': '0'} 2023-05-31T20:52:11.820 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:39:26 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000fa5df828f52a89cc-006477b37b-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:52:11 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:52:11.820 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj44 2023-05-31T20:52:11.821 DEBUG:boto:path=/pklxkd-59/obj44 2023-05-31T20:52:11.821 DEBUG:boto:auth_path=/pklxkd-59/obj44 2023-05-31T20:52:11.822 DEBUG:boto:path=/pklxkd-59/obj44?versionId=null 2023-05-31T20:52:11.822 DEBUG:boto:auth_path=/pklxkd-59/obj44?versionId=null 2023-05-31T20:52:11.822 DEBUG:boto:Method: GET 2023-05-31T20:52:11.823 DEBUG:boto:Path: /pklxkd-59/obj44?versionId=null 2023-05-31T20:52:11.823 DEBUG:boto:Data: 2023-05-31T20:52:11.823 DEBUG:boto:Headers: {} 2023-05-31T20:52:11.824 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:52:11.824 DEBUG:boto:Port: 8001 2023-05-31T20:52:11.824 DEBUG:boto:Params: {} 2023-05-31T20:52:11.825 DEBUG:boto:Token: None 2023-05-31T20:52:11.825 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:52:11 GMT /pklxkd-59/obj44?versionId=null 2023-05-31T20:52:11.826 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:Erl/WvrRe5XmVNxcaY7m9pNHkAg= 2023-05-31T20:52:11.826 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:52:11 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:Erl/WvrRe5XmVNxcaY7m9pNHkAg=', 'Content-Length': '0'} 2023-05-31T20:52:11.828 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:39:26 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000dc5df823c92715a-006477b37b-1c17-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:52:11 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:52:11.868 DEBUG:boto:path=/pklxkd-59/obj44 2023-05-31T20:52:11.869 DEBUG:boto:auth_path=/pklxkd-59/obj44 2023-05-31T20:52:11.869 DEBUG:boto:path=/pklxkd-59/obj44?versionId=null 2023-05-31T20:52:11.869 DEBUG:boto:auth_path=/pklxkd-59/obj44?versionId=null 2023-05-31T20:52:11.870 DEBUG:boto:Method: GET 2023-05-31T20:52:11.870 DEBUG:boto:Path: /pklxkd-59/obj44?versionId=null 2023-05-31T20:52:11.870 DEBUG:boto:Data: 2023-05-31T20:52:11.871 DEBUG:boto:Headers: {} 2023-05-31T20:52:11.871 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:52:11.871 DEBUG:boto:Port: 8000 2023-05-31T20:52:11.872 DEBUG:boto:Params: {} 2023-05-31T20:52:11.872 DEBUG:boto:Token: None 2023-05-31T20:52:11.873 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:52:11 GMT /pklxkd-59/obj44?versionId=null 2023-05-31T20:52:11.873 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:Erl/WvrRe5XmVNxcaY7m9pNHkAg= 2023-05-31T20:52:11.873 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:52:11 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:Erl/WvrRe5XmVNxcaY7m9pNHkAg=', 'Content-Length': '0'} 2023-05-31T20:52:11.875 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:39:26 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f505c9c5c9f072b3-006477b37b-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:52:11 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:52:11.876 DEBUG:boto:path=/pklxkd-59/ 2023-05-31T20:52:11.876 DEBUG:boto:auth_path=/pklxkd-59/ 2023-05-31T20:52:11.877 DEBUG:boto:path=/pklxkd-59/?prefix=obj44 2023-05-31T20:52:11.877 DEBUG:boto:auth_path=/pklxkd-59/?prefix=obj44 2023-05-31T20:52:11.878 DEBUG:boto:Method: GET 2023-05-31T20:52:11.878 DEBUG:boto:Path: /pklxkd-59/?prefix=obj44 2023-05-31T20:52:11.878 DEBUG:boto:Data: 2023-05-31T20:52:11.879 DEBUG:boto:Headers: {} 2023-05-31T20:52:11.879 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:52:11.879 DEBUG:boto:Port: 8001 2023-05-31T20:52:11.880 DEBUG:boto:Params: {} 2023-05-31T20:52:11.880 DEBUG:boto:Token: None 2023-05-31T20:52:11.880 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:52:11 GMT /pklxkd-59/ 2023-05-31T20:52:11.881 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:HITZ+oTJfJ3JS1dgm5XbULiOORY= 2023-05-31T20:52:11.881 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:52:11 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:HITZ+oTJfJ3JS1dgm5XbULiOORY=', 'Content-Length': '0'} 2023-05-31T20:52:11.884 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000855113308643b1cf-006477b37b-1c17-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:52:11 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:52:11.884 DEBUG:boto:b'pklxkd-59obj441000falseobj442023-05-31T20:39:26.565Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:52:11.885 DEBUG:boto:path=/pklxkd-59/ 2023-05-31T20:52:11.885 DEBUG:boto:auth_path=/pklxkd-59/ 2023-05-31T20:52:11.885 DEBUG:boto:path=/pklxkd-59/?prefix=obj44 2023-05-31T20:52:11.886 DEBUG:boto:auth_path=/pklxkd-59/?prefix=obj44 2023-05-31T20:52:11.886 DEBUG:boto:Method: GET 2023-05-31T20:52:11.886 DEBUG:boto:Path: /pklxkd-59/?prefix=obj44 2023-05-31T20:52:11.887 DEBUG:boto:Data: 2023-05-31T20:52:11.887 DEBUG:boto:Headers: {} 2023-05-31T20:52:11.888 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:52:11.888 DEBUG:boto:Port: 8000 2023-05-31T20:52:11.888 DEBUG:boto:Params: {} 2023-05-31T20:52:11.889 DEBUG:boto:Token: None 2023-05-31T20:52:11.889 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:52:11 GMT /pklxkd-59/ 2023-05-31T20:52:11.889 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:HITZ+oTJfJ3JS1dgm5XbULiOORY= 2023-05-31T20:52:11.890 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:52:11 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:HITZ+oTJfJ3JS1dgm5XbULiOORY=', 'Content-Length': '0'} 2023-05-31T20:52:11.893 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c71b643c92394bb5-006477b37b-1630-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:52:11 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:52:11.894 DEBUG:boto:b'pklxkd-59obj441000falseobj442023-05-31T20:39:26.565Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:52:11.894 DEBUG:boto:path=/pklxkd-59/obj44 2023-05-31T20:52:11.895 DEBUG:boto:auth_path=/pklxkd-59/obj44 2023-05-31T20:52:11.895 DEBUG:boto:Method: HEAD 2023-05-31T20:52:11.895 DEBUG:boto:Path: /pklxkd-59/obj44 2023-05-31T20:52:11.896 DEBUG:boto:Data: 2023-05-31T20:52:11.896 DEBUG:boto:Headers: {} 2023-05-31T20:52:11.896 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:52:11.897 DEBUG:boto:Port: 8001 2023-05-31T20:52:11.897 DEBUG:boto:Params: {} 2023-05-31T20:52:11.897 DEBUG:boto:Token: None 2023-05-31T20:52:11.898 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:52:11 GMT /pklxkd-59/obj44 2023-05-31T20:52:11.898 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:4be0gX12R56Z6UMyuOUSyyyVnVM= 2023-05-31T20:52:11.898 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:52:11 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:4be0gX12R56Z6UMyuOUSyyyVnVM=', 'Content-Length': '0'} 2023-05-31T20:52:11.900 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:39:26 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008b6f6dacd3579593-006477b37b-1c17-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:52:11 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:52:11.901 DEBUG:boto:path=/pklxkd-59/obj44 2023-05-31T20:52:11.901 DEBUG:boto:auth_path=/pklxkd-59/obj44 2023-05-31T20:52:11.901 DEBUG:boto:Method: HEAD 2023-05-31T20:52:11.902 DEBUG:boto:Path: /pklxkd-59/obj44 2023-05-31T20:52:11.902 DEBUG:boto:Data: 2023-05-31T20:52:11.902 DEBUG:boto:Headers: {} 2023-05-31T20:52:11.903 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:52:11.903 DEBUG:boto:Port: 8000 2023-05-31T20:52:11.903 DEBUG:boto:Params: {} 2023-05-31T20:52:11.904 DEBUG:boto:Token: None 2023-05-31T20:52:11.904 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:52:11 GMT /pklxkd-59/obj44 2023-05-31T20:52:11.904 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:4be0gX12R56Z6UMyuOUSyyyVnVM= 2023-05-31T20:52:11.905 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:52:11 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:4be0gX12R56Z6UMyuOUSyyyVnVM=', 'Content-Length': '0'} 2023-05-31T20:52:11.907 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:39:26 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e7ae3e85225fe139-006477b37b-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:52:11 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:52:11.907 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj44 2023-05-31T20:52:11.907 DEBUG:boto:path=/pklxkd-59/obj44 2023-05-31T20:52:11.908 DEBUG:boto:auth_path=/pklxkd-59/obj44 2023-05-31T20:52:11.908 DEBUG:boto:Method: GET 2023-05-31T20:52:11.909 DEBUG:boto:Path: /pklxkd-59/obj44 2023-05-31T20:52:11.909 DEBUG:boto:Data: 2023-05-31T20:52:11.909 DEBUG:boto:Headers: {} 2023-05-31T20:52:11.910 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:52:11.910 DEBUG:boto:Port: 8001 2023-05-31T20:52:11.910 DEBUG:boto:Params: {} 2023-05-31T20:52:11.911 DEBUG:boto:Token: None 2023-05-31T20:52:11.911 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:52:11 GMT /pklxkd-59/obj44 2023-05-31T20:52:11.912 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:5vLCJoAOtoDbYK6upgNH/1ppVjU= 2023-05-31T20:52:11.912 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:52:11 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:5vLCJoAOtoDbYK6upgNH/1ppVjU=', 'Content-Length': '0'} 2023-05-31T20:52:11.914 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:39:26 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000045d23d80443f9536-006477b37b-1c17-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:52:11 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:52:11.914 DEBUG:boto:path=/pklxkd-59/obj44 2023-05-31T20:52:11.915 DEBUG:boto:auth_path=/pklxkd-59/obj44 2023-05-31T20:52:11.915 DEBUG:boto:Method: GET 2023-05-31T20:52:11.915 DEBUG:boto:Path: /pklxkd-59/obj44 2023-05-31T20:52:11.916 DEBUG:boto:Data: 2023-05-31T20:52:11.916 DEBUG:boto:Headers: {} 2023-05-31T20:52:11.917 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:52:11.917 DEBUG:boto:Port: 8000 2023-05-31T20:52:11.917 DEBUG:boto:Params: {} 2023-05-31T20:52:11.918 DEBUG:boto:Token: None 2023-05-31T20:52:11.918 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:52:11 GMT /pklxkd-59/obj44 2023-05-31T20:52:11.919 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:5vLCJoAOtoDbYK6upgNH/1ppVjU= 2023-05-31T20:52:11.919 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:52:11 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:5vLCJoAOtoDbYK6upgNH/1ppVjU=', 'Content-Length': '0'} 2023-05-31T20:52:11.921 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:39:26 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002981cd806baed684-006477b37b-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:52:11 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:52:11.964 DEBUG:boto:path=/pklxkd-59/ 2023-05-31T20:52:11.965 DEBUG:boto:auth_path=/pklxkd-59/ 2023-05-31T20:52:11.965 DEBUG:boto:path=/pklxkd-59/?prefix=obj44 2023-05-31T20:52:11.965 DEBUG:boto:auth_path=/pklxkd-59/?prefix=obj44 2023-05-31T20:52:11.966 DEBUG:boto:Method: GET 2023-05-31T20:52:11.966 DEBUG:boto:Path: /pklxkd-59/?prefix=obj44 2023-05-31T20:52:11.966 DEBUG:boto:Data: 2023-05-31T20:52:11.967 DEBUG:boto:Headers: {} 2023-05-31T20:52:11.967 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:52:11.967 DEBUG:boto:Port: 8001 2023-05-31T20:52:11.968 DEBUG:boto:Params: {} 2023-05-31T20:52:11.968 DEBUG:boto:Token: None 2023-05-31T20:52:11.969 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:52:11 GMT /pklxkd-59/ 2023-05-31T20:52:11.969 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:HITZ+oTJfJ3JS1dgm5XbULiOORY= 2023-05-31T20:52:11.969 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:52:11 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:HITZ+oTJfJ3JS1dgm5XbULiOORY=', 'Content-Length': '0'} 2023-05-31T20:52:11.972 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000003bd2b01e1b0786a-006477b37b-1c17-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:52:11 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:52:11.972 DEBUG:boto:b'pklxkd-59obj441000falseobj442023-05-31T20:39:26.565Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:52:11.973 DEBUG:boto:path=/pklxkd-59/ 2023-05-31T20:52:11.973 DEBUG:boto:auth_path=/pklxkd-59/ 2023-05-31T20:52:11.974 DEBUG:boto:path=/pklxkd-59/?prefix=obj44 2023-05-31T20:52:11.974 DEBUG:boto:auth_path=/pklxkd-59/?prefix=obj44 2023-05-31T20:52:11.974 DEBUG:boto:Method: GET 2023-05-31T20:52:11.975 DEBUG:boto:Path: /pklxkd-59/?prefix=obj44 2023-05-31T20:52:11.975 DEBUG:boto:Data: 2023-05-31T20:52:11.975 DEBUG:boto:Headers: {} 2023-05-31T20:52:11.976 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:52:11.976 DEBUG:boto:Port: 8000 2023-05-31T20:52:11.977 DEBUG:boto:Params: {} 2023-05-31T20:52:11.977 DEBUG:boto:Token: None 2023-05-31T20:52:11.977 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:52:11 GMT /pklxkd-59/ 2023-05-31T20:52:11.978 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:HITZ+oTJfJ3JS1dgm5XbULiOORY= 2023-05-31T20:52:11.978 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:52:11 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:HITZ+oTJfJ3JS1dgm5XbULiOORY=', 'Content-Length': '0'} 2023-05-31T20:52:11.982 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000024d55451cfd0bd22-006477b37b-1630-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:52:11 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:52:11.982 DEBUG:boto:b'pklxkd-59obj441000falseobj442023-05-31T20:39:26.565Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:52:11.983 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj46 2023-05-31T20:52:11.983 DEBUG:boto:path=/pklxkd-59/obj46 2023-05-31T20:52:11.983 DEBUG:boto:auth_path=/pklxkd-59/obj46 2023-05-31T20:52:11.984 DEBUG:boto:path=/pklxkd-59/obj46?versionId=null 2023-05-31T20:52:11.984 DEBUG:boto:auth_path=/pklxkd-59/obj46?versionId=null 2023-05-31T20:52:11.984 DEBUG:boto:Method: GET 2023-05-31T20:52:11.985 DEBUG:boto:Path: /pklxkd-59/obj46?versionId=null 2023-05-31T20:52:11.985 DEBUG:boto:Data: 2023-05-31T20:52:11.985 DEBUG:boto:Headers: {} 2023-05-31T20:52:11.986 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:52:11.986 DEBUG:boto:Port: 8001 2023-05-31T20:52:11.987 DEBUG:boto:Params: {} 2023-05-31T20:52:11.987 DEBUG:boto:Token: None 2023-05-31T20:52:11.987 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:52:11 GMT /pklxkd-59/obj46?versionId=null 2023-05-31T20:52:11.988 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:PQfs/gXvQDfqgSqlYtZpTPndDN8= 2023-05-31T20:52:11.988 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:52:11 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:PQfs/gXvQDfqgSqlYtZpTPndDN8=', 'Content-Length': '0'} 2023-05-31T20:52:11.990 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:46:33 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ac2ba4bb19ec0cd8-006477b37b-1c17-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:52:11 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:52:11.991 DEBUG:boto:path=/pklxkd-59/obj46 2023-05-31T20:52:11.991 DEBUG:boto:auth_path=/pklxkd-59/obj46 2023-05-31T20:52:11.991 DEBUG:boto:path=/pklxkd-59/obj46?versionId=null 2023-05-31T20:52:11.992 DEBUG:boto:auth_path=/pklxkd-59/obj46?versionId=null 2023-05-31T20:52:11.992 DEBUG:boto:Method: GET 2023-05-31T20:52:11.993 DEBUG:boto:Path: /pklxkd-59/obj46?versionId=null 2023-05-31T20:52:11.993 DEBUG:boto:Data: 2023-05-31T20:52:11.993 DEBUG:boto:Headers: {} 2023-05-31T20:52:11.994 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:52:11.994 DEBUG:boto:Port: 8000 2023-05-31T20:52:11.994 DEBUG:boto:Params: {} 2023-05-31T20:52:11.995 DEBUG:boto:Token: None 2023-05-31T20:52:11.995 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:52:11 GMT /pklxkd-59/obj46?versionId=null 2023-05-31T20:52:11.996 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:PQfs/gXvQDfqgSqlYtZpTPndDN8= 2023-05-31T20:52:11.996 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:52:11 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:PQfs/gXvQDfqgSqlYtZpTPndDN8=', 'Content-Length': '0'} 2023-05-31T20:52:11.998 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:46:33 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000be8a99a800b6406-006477b37b-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:52:11 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:52:11.998 DEBUG:boto:path=/pklxkd-59/obj46 2023-05-31T20:52:11.999 DEBUG:boto:auth_path=/pklxkd-59/obj46 2023-05-31T20:52:11.999 DEBUG:boto:path=/pklxkd-59/obj46?versionId=null 2023-05-31T20:52:11.999 DEBUG:boto:auth_path=/pklxkd-59/obj46?versionId=null 2023-05-31T20:52:12.000 DEBUG:boto:Method: HEAD 2023-05-31T20:52:12.000 DEBUG:boto:Path: /pklxkd-59/obj46?versionId=null 2023-05-31T20:52:12.000 DEBUG:boto:Data: 2023-05-31T20:52:12.001 DEBUG:boto:Headers: {} 2023-05-31T20:52:12.001 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:52:12.001 DEBUG:boto:Port: 8001 2023-05-31T20:52:12.002 DEBUG:boto:Params: {} 2023-05-31T20:52:12.002 DEBUG:boto:Token: None 2023-05-31T20:52:12.002 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:52:12 GMT /pklxkd-59/obj46?versionId=null 2023-05-31T20:52:12.003 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:QXRZk7aeRe/I5ArRh7VKv/pB7As= 2023-05-31T20:52:12.003 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:52:12 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:QXRZk7aeRe/I5ArRh7VKv/pB7As=', 'Content-Length': '0'} 2023-05-31T20:52:12.005 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:46:33 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000c7a394326a224d0-006477b37c-1c17-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:52:12 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:52:12.005 DEBUG:boto:path=/pklxkd-59/obj46 2023-05-31T20:52:12.006 DEBUG:boto:auth_path=/pklxkd-59/obj46 2023-05-31T20:52:12.006 DEBUG:boto:path=/pklxkd-59/obj46?versionId=null 2023-05-31T20:52:12.006 DEBUG:boto:auth_path=/pklxkd-59/obj46?versionId=null 2023-05-31T20:52:12.007 DEBUG:boto:Method: HEAD 2023-05-31T20:52:12.007 DEBUG:boto:Path: /pklxkd-59/obj46?versionId=null 2023-05-31T20:52:12.007 DEBUG:boto:Data: 2023-05-31T20:52:12.008 DEBUG:boto:Headers: {} 2023-05-31T20:52:12.008 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:52:12.008 DEBUG:boto:Port: 8000 2023-05-31T20:52:12.009 DEBUG:boto:Params: {} 2023-05-31T20:52:12.009 DEBUG:boto:Token: None 2023-05-31T20:52:12.010 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:52:12 GMT /pklxkd-59/obj46?versionId=null 2023-05-31T20:52:12.010 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:QXRZk7aeRe/I5ArRh7VKv/pB7As= 2023-05-31T20:52:12.010 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:52:12 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:QXRZk7aeRe/I5ArRh7VKv/pB7As=', 'Content-Length': '0'} 2023-05-31T20:52:12.012 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:46:33 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000016366beac4f4cb5d-006477b37c-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:52:12 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:52:12.013 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj46 2023-05-31T20:52:12.013 DEBUG:boto:path=/pklxkd-59/obj46 2023-05-31T20:52:12.013 DEBUG:boto:auth_path=/pklxkd-59/obj46 2023-05-31T20:52:12.014 DEBUG:boto:path=/pklxkd-59/obj46?versionId=null 2023-05-31T20:52:12.014 DEBUG:boto:auth_path=/pklxkd-59/obj46?versionId=null 2023-05-31T20:52:12.014 DEBUG:boto:Method: GET 2023-05-31T20:52:12.015 DEBUG:boto:Path: /pklxkd-59/obj46?versionId=null 2023-05-31T20:52:12.015 DEBUG:boto:Data: 2023-05-31T20:52:12.016 DEBUG:boto:Headers: {} 2023-05-31T20:52:12.016 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:52:12.016 DEBUG:boto:Port: 8001 2023-05-31T20:52:12.017 DEBUG:boto:Params: {} 2023-05-31T20:52:12.017 DEBUG:boto:Token: None 2023-05-31T20:52:12.017 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:52:12 GMT /pklxkd-59/obj46?versionId=null 2023-05-31T20:52:12.018 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:otfmgiA8BfYAwP41uqGf6YvzXKQ= 2023-05-31T20:52:12.018 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:52:12 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:otfmgiA8BfYAwP41uqGf6YvzXKQ=', 'Content-Length': '0'} 2023-05-31T20:52:12.020 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:46:33 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000074c1574b2f909e54-006477b37c-1c17-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:52:12 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:52:12.060 DEBUG:boto:path=/pklxkd-59/obj46 2023-05-31T20:52:12.061 DEBUG:boto:auth_path=/pklxkd-59/obj46 2023-05-31T20:52:12.061 DEBUG:boto:path=/pklxkd-59/obj46?versionId=null 2023-05-31T20:52:12.061 DEBUG:boto:auth_path=/pklxkd-59/obj46?versionId=null 2023-05-31T20:52:12.062 DEBUG:boto:Method: GET 2023-05-31T20:52:12.062 DEBUG:boto:Path: /pklxkd-59/obj46?versionId=null 2023-05-31T20:52:12.062 DEBUG:boto:Data: 2023-05-31T20:52:12.063 DEBUG:boto:Headers: {} 2023-05-31T20:52:12.063 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:52:12.063 DEBUG:boto:Port: 8000 2023-05-31T20:52:12.064 DEBUG:boto:Params: {} 2023-05-31T20:52:12.064 DEBUG:boto:Token: None 2023-05-31T20:52:12.065 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:52:12 GMT /pklxkd-59/obj46?versionId=null 2023-05-31T20:52:12.065 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:otfmgiA8BfYAwP41uqGf6YvzXKQ= 2023-05-31T20:52:12.065 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:52:12 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:otfmgiA8BfYAwP41uqGf6YvzXKQ=', 'Content-Length': '0'} 2023-05-31T20:52:12.067 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:46:33 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f2b97c0b941b88ee-006477b37c-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:52:12 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:52:12.068 DEBUG:boto:path=/pklxkd-59/ 2023-05-31T20:52:12.068 DEBUG:boto:auth_path=/pklxkd-59/ 2023-05-31T20:52:12.069 DEBUG:boto:path=/pklxkd-59/?prefix=obj46 2023-05-31T20:52:12.069 DEBUG:boto:auth_path=/pklxkd-59/?prefix=obj46 2023-05-31T20:52:12.070 DEBUG:boto:Method: GET 2023-05-31T20:52:12.070 DEBUG:boto:Path: /pklxkd-59/?prefix=obj46 2023-05-31T20:52:12.070 DEBUG:boto:Data: 2023-05-31T20:52:12.071 DEBUG:boto:Headers: {} 2023-05-31T20:52:12.071 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:52:12.071 DEBUG:boto:Port: 8001 2023-05-31T20:52:12.072 DEBUG:boto:Params: {} 2023-05-31T20:52:12.072 DEBUG:boto:Token: None 2023-05-31T20:52:12.072 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:52:12 GMT /pklxkd-59/ 2023-05-31T20:52:12.073 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:pLqt3uLOcBEwAGLY0jYnxgHFOQo= 2023-05-31T20:52:12.073 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:52:12 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:pLqt3uLOcBEwAGLY0jYnxgHFOQo=', 'Content-Length': '0'} 2023-05-31T20:52:12.076 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000000637f888f05c3aa1-006477b37c-1c17-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:52:12 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:52:12.077 DEBUG:boto:b'pklxkd-59obj461000falseobj462023-05-31T20:46:33.830Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:52:12.077 DEBUG:boto:path=/pklxkd-59/ 2023-05-31T20:52:12.078 DEBUG:boto:auth_path=/pklxkd-59/ 2023-05-31T20:52:12.078 DEBUG:boto:path=/pklxkd-59/?prefix=obj46 2023-05-31T20:52:12.078 DEBUG:boto:auth_path=/pklxkd-59/?prefix=obj46 2023-05-31T20:52:12.079 DEBUG:boto:Method: GET 2023-05-31T20:52:12.079 DEBUG:boto:Path: /pklxkd-59/?prefix=obj46 2023-05-31T20:52:12.079 DEBUG:boto:Data: 2023-05-31T20:52:12.080 DEBUG:boto:Headers: {} 2023-05-31T20:52:12.080 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:52:12.080 DEBUG:boto:Port: 8000 2023-05-31T20:52:12.081 DEBUG:boto:Params: {} 2023-05-31T20:52:12.081 DEBUG:boto:Token: None 2023-05-31T20:52:12.082 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:52:12 GMT /pklxkd-59/ 2023-05-31T20:52:12.082 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:pLqt3uLOcBEwAGLY0jYnxgHFOQo= 2023-05-31T20:52:12.082 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:52:12 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:pLqt3uLOcBEwAGLY0jYnxgHFOQo=', 'Content-Length': '0'} 2023-05-31T20:52:12.086 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a4d4012ea1e5d933-006477b37c-1630-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:52:12 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:52:12.087 DEBUG:boto:b'pklxkd-59obj461000falseobj462023-05-31T20:46:33.830Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:52:12.087 DEBUG:boto:path=/pklxkd-59/obj46 2023-05-31T20:52:12.087 DEBUG:boto:auth_path=/pklxkd-59/obj46 2023-05-31T20:52:12.088 DEBUG:boto:Method: HEAD 2023-05-31T20:52:12.088 DEBUG:boto:Path: /pklxkd-59/obj46 2023-05-31T20:52:12.089 DEBUG:boto:Data: 2023-05-31T20:52:12.089 DEBUG:boto:Headers: {} 2023-05-31T20:52:12.089 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:52:12.089 DEBUG:boto:Port: 8001 2023-05-31T20:52:12.090 DEBUG:boto:Params: {} 2023-05-31T20:52:12.090 DEBUG:boto:Token: None 2023-05-31T20:52:12.091 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:52:12 GMT /pklxkd-59/obj46 2023-05-31T20:52:12.091 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:pf+7+52344KyApDcfQNDoZ7eTFI= 2023-05-31T20:52:12.091 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:52:12 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:pf+7+52344KyApDcfQNDoZ7eTFI=', 'Content-Length': '0'} 2023-05-31T20:52:12.093 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:46:33 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009d0146ca099a40fa-006477b37c-1c17-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:52:12 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:52:12.094 DEBUG:boto:path=/pklxkd-59/obj46 2023-05-31T20:52:12.094 DEBUG:boto:auth_path=/pklxkd-59/obj46 2023-05-31T20:52:12.094 DEBUG:boto:Method: HEAD 2023-05-31T20:52:12.095 DEBUG:boto:Path: /pklxkd-59/obj46 2023-05-31T20:52:12.095 DEBUG:boto:Data: 2023-05-31T20:52:12.095 DEBUG:boto:Headers: {} 2023-05-31T20:52:12.096 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:52:12.096 DEBUG:boto:Port: 8000 2023-05-31T20:52:12.096 DEBUG:boto:Params: {} 2023-05-31T20:52:12.097 DEBUG:boto:Token: None 2023-05-31T20:52:12.097 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:52:12 GMT /pklxkd-59/obj46 2023-05-31T20:52:12.097 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:pf+7+52344KyApDcfQNDoZ7eTFI= 2023-05-31T20:52:12.098 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:52:12 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:pf+7+52344KyApDcfQNDoZ7eTFI=', 'Content-Length': '0'} 2023-05-31T20:52:12.100 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:46:33 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009d0b1f5b16454f94-006477b37c-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:52:12 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:52:12.101 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj46 2023-05-31T20:52:12.101 DEBUG:boto:path=/pklxkd-59/obj46 2023-05-31T20:52:12.101 DEBUG:boto:auth_path=/pklxkd-59/obj46 2023-05-31T20:52:12.102 DEBUG:boto:Method: GET 2023-05-31T20:52:12.102 DEBUG:boto:Path: /pklxkd-59/obj46 2023-05-31T20:52:12.103 DEBUG:boto:Data: 2023-05-31T20:52:12.103 DEBUG:boto:Headers: {} 2023-05-31T20:52:12.103 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:52:12.104 DEBUG:boto:Port: 8001 2023-05-31T20:52:12.104 DEBUG:boto:Params: {} 2023-05-31T20:52:12.104 DEBUG:boto:Token: None 2023-05-31T20:52:12.105 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:52:12 GMT /pklxkd-59/obj46 2023-05-31T20:52:12.105 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:5Auouf8NXv4LrMO4SAEpUHcZDCs= 2023-05-31T20:52:12.106 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:52:12 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:5Auouf8NXv4LrMO4SAEpUHcZDCs=', 'Content-Length': '0'} 2023-05-31T20:52:12.107 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:46:33 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002907bff9d37ad800-006477b37c-1c17-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:52:12 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:52:12.148 DEBUG:boto:path=/pklxkd-59/obj46 2023-05-31T20:52:12.149 DEBUG:boto:auth_path=/pklxkd-59/obj46 2023-05-31T20:52:12.149 DEBUG:boto:Method: GET 2023-05-31T20:52:12.149 DEBUG:boto:Path: /pklxkd-59/obj46 2023-05-31T20:52:12.150 DEBUG:boto:Data: 2023-05-31T20:52:12.150 DEBUG:boto:Headers: {} 2023-05-31T20:52:12.150 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:52:12.151 DEBUG:boto:Port: 8000 2023-05-31T20:52:12.151 DEBUG:boto:Params: {} 2023-05-31T20:52:12.152 DEBUG:boto:Token: None 2023-05-31T20:52:12.152 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:52:12 GMT /pklxkd-59/obj46 2023-05-31T20:52:12.152 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:5Auouf8NXv4LrMO4SAEpUHcZDCs= 2023-05-31T20:52:12.153 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:52:12 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:5Auouf8NXv4LrMO4SAEpUHcZDCs=', 'Content-Length': '0'} 2023-05-31T20:52:12.155 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:46:33 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000367b28d3457ec2d1-006477b37c-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:52:12 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:52:12.156 DEBUG:boto:path=/pklxkd-59/ 2023-05-31T20:52:12.156 DEBUG:boto:auth_path=/pklxkd-59/ 2023-05-31T20:52:12.156 DEBUG:boto:path=/pklxkd-59/?prefix=obj46 2023-05-31T20:52:12.157 DEBUG:boto:auth_path=/pklxkd-59/?prefix=obj46 2023-05-31T20:52:12.157 DEBUG:boto:Method: GET 2023-05-31T20:52:12.157 DEBUG:boto:Path: /pklxkd-59/?prefix=obj46 2023-05-31T20:52:12.158 DEBUG:boto:Data: 2023-05-31T20:52:12.158 DEBUG:boto:Headers: {} 2023-05-31T20:52:12.159 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:52:12.159 DEBUG:boto:Port: 8001 2023-05-31T20:52:12.159 DEBUG:boto:Params: {} 2023-05-31T20:52:12.160 DEBUG:boto:Token: None 2023-05-31T20:52:12.160 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:52:12 GMT /pklxkd-59/ 2023-05-31T20:52:12.160 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:pLqt3uLOcBEwAGLY0jYnxgHFOQo= 2023-05-31T20:52:12.161 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:52:12 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:pLqt3uLOcBEwAGLY0jYnxgHFOQo=', 'Content-Length': '0'} 2023-05-31T20:52:12.163 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000009b13d57ff9fea19c-006477b37c-1c17-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:52:12 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:52:12.164 DEBUG:boto:b'pklxkd-59obj461000falseobj462023-05-31T20:46:33.830Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:52:12.164 DEBUG:boto:path=/pklxkd-59/ 2023-05-31T20:52:12.165 DEBUG:boto:auth_path=/pklxkd-59/ 2023-05-31T20:52:12.165 DEBUG:boto:path=/pklxkd-59/?prefix=obj46 2023-05-31T20:52:12.165 DEBUG:boto:auth_path=/pklxkd-59/?prefix=obj46 2023-05-31T20:52:12.166 DEBUG:boto:Method: GET 2023-05-31T20:52:12.166 DEBUG:boto:Path: /pklxkd-59/?prefix=obj46 2023-05-31T20:52:12.167 DEBUG:boto:Data: 2023-05-31T20:52:12.167 DEBUG:boto:Headers: {} 2023-05-31T20:52:12.167 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:52:12.168 DEBUG:boto:Port: 8000 2023-05-31T20:52:12.168 DEBUG:boto:Params: {} 2023-05-31T20:52:12.168 DEBUG:boto:Token: None 2023-05-31T20:52:12.169 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:52:12 GMT /pklxkd-59/ 2023-05-31T20:52:12.169 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:pLqt3uLOcBEwAGLY0jYnxgHFOQo= 2023-05-31T20:52:12.170 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:52:12 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:pLqt3uLOcBEwAGLY0jYnxgHFOQo=', 'Content-Length': '0'} 2023-05-31T20:52:12.173 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000004c5321aa1a430e06-006477b37c-1630-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:52:12 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:52:12.174 DEBUG:boto:b'pklxkd-59obj461000falseobj462023-05-31T20:46:33.830Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:52:12.174 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj51 2023-05-31T20:52:12.175 DEBUG:boto:path=/pklxkd-59/obj51 2023-05-31T20:52:12.175 DEBUG:boto:auth_path=/pklxkd-59/obj51 2023-05-31T20:52:12.175 DEBUG:boto:path=/pklxkd-59/obj51?versionId=null 2023-05-31T20:52:12.176 DEBUG:boto:auth_path=/pklxkd-59/obj51?versionId=null 2023-05-31T20:52:12.176 DEBUG:boto:Method: GET 2023-05-31T20:52:12.177 DEBUG:boto:Path: /pklxkd-59/obj51?versionId=null 2023-05-31T20:52:12.177 DEBUG:boto:Data: 2023-05-31T20:52:12.177 DEBUG:boto:Headers: {} 2023-05-31T20:52:12.178 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:52:12.178 DEBUG:boto:Port: 8001 2023-05-31T20:52:12.178 DEBUG:boto:Params: {} 2023-05-31T20:52:12.179 DEBUG:boto:Token: None 2023-05-31T20:52:12.179 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:52:12 GMT /pklxkd-59/obj51?versionId=null 2023-05-31T20:52:12.179 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:LF9ODj2pG7fXCDawBp7aajfWKwk= 2023-05-31T20:52:12.180 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:52:12 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:LF9ODj2pG7fXCDawBp7aajfWKwk=', 'Content-Length': '0'} 2023-05-31T20:52:12.182 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:43:08 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001ac31a5492df3706-006477b37c-1c17-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:52:12 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:52:12.182 DEBUG:boto:path=/pklxkd-59/obj51 2023-05-31T20:52:12.183 DEBUG:boto:auth_path=/pklxkd-59/obj51 2023-05-31T20:52:12.183 DEBUG:boto:path=/pklxkd-59/obj51?versionId=null 2023-05-31T20:52:12.183 DEBUG:boto:auth_path=/pklxkd-59/obj51?versionId=null 2023-05-31T20:52:12.184 DEBUG:boto:Method: GET 2023-05-31T20:52:12.184 DEBUG:boto:Path: /pklxkd-59/obj51?versionId=null 2023-05-31T20:52:12.185 DEBUG:boto:Data: 2023-05-31T20:52:12.185 DEBUG:boto:Headers: {} 2023-05-31T20:52:12.185 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:52:12.186 DEBUG:boto:Port: 8000 2023-05-31T20:52:12.186 DEBUG:boto:Params: {} 2023-05-31T20:52:12.186 DEBUG:boto:Token: None 2023-05-31T20:52:12.187 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:52:12 GMT /pklxkd-59/obj51?versionId=null 2023-05-31T20:52:12.187 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:LF9ODj2pG7fXCDawBp7aajfWKwk= 2023-05-31T20:52:12.188 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:52:12 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:LF9ODj2pG7fXCDawBp7aajfWKwk=', 'Content-Length': '0'} 2023-05-31T20:52:12.189 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:43:08 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f7836ebecdad903b-006477b37c-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:52:12 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:52:12.190 DEBUG:boto:path=/pklxkd-59/obj51 2023-05-31T20:52:12.190 DEBUG:boto:auth_path=/pklxkd-59/obj51 2023-05-31T20:52:12.191 DEBUG:boto:path=/pklxkd-59/obj51?versionId=null 2023-05-31T20:52:12.191 DEBUG:boto:auth_path=/pklxkd-59/obj51?versionId=null 2023-05-31T20:52:12.191 DEBUG:boto:Method: HEAD 2023-05-31T20:52:12.192 DEBUG:boto:Path: /pklxkd-59/obj51?versionId=null 2023-05-31T20:52:12.192 DEBUG:boto:Data: 2023-05-31T20:52:12.192 DEBUG:boto:Headers: {} 2023-05-31T20:52:12.193 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:52:12.193 DEBUG:boto:Port: 8001 2023-05-31T20:52:12.193 DEBUG:boto:Params: {} 2023-05-31T20:52:12.194 DEBUG:boto:Token: None 2023-05-31T20:52:12.194 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:52:12 GMT /pklxkd-59/obj51?versionId=null 2023-05-31T20:52:12.194 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:EXaCSLpBDuQu5SX/shm0jHjPSZI= 2023-05-31T20:52:12.195 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:52:12 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:EXaCSLpBDuQu5SX/shm0jHjPSZI=', 'Content-Length': '0'} 2023-05-31T20:52:12.196 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:43:08 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f28992ac81d67647-006477b37c-1c17-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:52:12 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:52:12.197 DEBUG:boto:path=/pklxkd-59/obj51 2023-05-31T20:52:12.197 DEBUG:boto:auth_path=/pklxkd-59/obj51 2023-05-31T20:52:12.198 DEBUG:boto:path=/pklxkd-59/obj51?versionId=null 2023-05-31T20:52:12.198 DEBUG:boto:auth_path=/pklxkd-59/obj51?versionId=null 2023-05-31T20:52:12.198 DEBUG:boto:Method: HEAD 2023-05-31T20:52:12.199 DEBUG:boto:Path: /pklxkd-59/obj51?versionId=null 2023-05-31T20:52:12.199 DEBUG:boto:Data: 2023-05-31T20:52:12.199 DEBUG:boto:Headers: {} 2023-05-31T20:52:12.200 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:52:12.200 DEBUG:boto:Port: 8000 2023-05-31T20:52:12.200 DEBUG:boto:Params: {} 2023-05-31T20:52:12.201 DEBUG:boto:Token: None 2023-05-31T20:52:12.201 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:52:12 GMT /pklxkd-59/obj51?versionId=null 2023-05-31T20:52:12.201 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:EXaCSLpBDuQu5SX/shm0jHjPSZI= 2023-05-31T20:52:12.202 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:52:12 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:EXaCSLpBDuQu5SX/shm0jHjPSZI=', 'Content-Length': '0'} 2023-05-31T20:52:12.204 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:43:08 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f2f37c80531b89fc-006477b37c-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:52:12 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:52:12.204 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj51 2023-05-31T20:52:12.205 DEBUG:boto:path=/pklxkd-59/obj51 2023-05-31T20:52:12.205 DEBUG:boto:auth_path=/pklxkd-59/obj51 2023-05-31T20:52:12.205 DEBUG:boto:path=/pklxkd-59/obj51?versionId=null 2023-05-31T20:52:12.206 DEBUG:boto:auth_path=/pklxkd-59/obj51?versionId=null 2023-05-31T20:52:12.206 DEBUG:boto:Method: GET 2023-05-31T20:52:12.206 DEBUG:boto:Path: /pklxkd-59/obj51?versionId=null 2023-05-31T20:52:12.207 DEBUG:boto:Data: 2023-05-31T20:52:12.207 DEBUG:boto:Headers: {} 2023-05-31T20:52:12.207 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:52:12.208 DEBUG:boto:Port: 8001 2023-05-31T20:52:12.208 DEBUG:boto:Params: {} 2023-05-31T20:52:12.209 DEBUG:boto:Token: None 2023-05-31T20:52:12.209 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:52:12 GMT /pklxkd-59/obj51?versionId=null 2023-05-31T20:52:12.209 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:LF9ODj2pG7fXCDawBp7aajfWKwk= 2023-05-31T20:52:12.210 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:52:12 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:LF9ODj2pG7fXCDawBp7aajfWKwk=', 'Content-Length': '0'} 2023-05-31T20:52:12.212 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:43:08 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d9aaecf6e2ae5428-006477b37c-1c17-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:52:12 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:52:12.212 DEBUG:boto:path=/pklxkd-59/obj51 2023-05-31T20:52:12.213 DEBUG:boto:auth_path=/pklxkd-59/obj51 2023-05-31T20:52:12.213 DEBUG:boto:path=/pklxkd-59/obj51?versionId=null 2023-05-31T20:52:12.213 DEBUG:boto:auth_path=/pklxkd-59/obj51?versionId=null 2023-05-31T20:52:12.214 DEBUG:boto:Method: GET 2023-05-31T20:52:12.214 DEBUG:boto:Path: /pklxkd-59/obj51?versionId=null 2023-05-31T20:52:12.215 DEBUG:boto:Data: 2023-05-31T20:52:12.215 DEBUG:boto:Headers: {} 2023-05-31T20:52:12.215 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:52:12.216 DEBUG:boto:Port: 8000 2023-05-31T20:52:12.216 DEBUG:boto:Params: {} 2023-05-31T20:52:12.216 DEBUG:boto:Token: None 2023-05-31T20:52:12.217 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:52:12 GMT /pklxkd-59/obj51?versionId=null 2023-05-31T20:52:12.217 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:LF9ODj2pG7fXCDawBp7aajfWKwk= 2023-05-31T20:52:12.218 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:52:12 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:LF9ODj2pG7fXCDawBp7aajfWKwk=', 'Content-Length': '0'} 2023-05-31T20:52:12.219 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:43:08 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004c5c3dadc63af192-006477b37c-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:52:12 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:52:12.220 DEBUG:boto:path=/pklxkd-59/ 2023-05-31T20:52:12.220 DEBUG:boto:auth_path=/pklxkd-59/ 2023-05-31T20:52:12.221 DEBUG:boto:path=/pklxkd-59/?prefix=obj51 2023-05-31T20:52:12.221 DEBUG:boto:auth_path=/pklxkd-59/?prefix=obj51 2023-05-31T20:52:12.222 DEBUG:boto:Method: GET 2023-05-31T20:52:12.222 DEBUG:boto:Path: /pklxkd-59/?prefix=obj51 2023-05-31T20:52:12.222 DEBUG:boto:Data: 2023-05-31T20:52:12.223 DEBUG:boto:Headers: {} 2023-05-31T20:52:12.223 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:52:12.223 DEBUG:boto:Port: 8001 2023-05-31T20:52:12.224 DEBUG:boto:Params: {} 2023-05-31T20:52:12.224 DEBUG:boto:Token: None 2023-05-31T20:52:12.224 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:52:12 GMT /pklxkd-59/ 2023-05-31T20:52:12.225 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:pLqt3uLOcBEwAGLY0jYnxgHFOQo= 2023-05-31T20:52:12.225 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:52:12 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:pLqt3uLOcBEwAGLY0jYnxgHFOQo=', 'Content-Length': '0'} 2023-05-31T20:52:12.228 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000004f81cabae643a334-006477b37c-1c17-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:52:12 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:52:12.228 DEBUG:boto:b'pklxkd-59obj511000falseobj512023-05-31T20:43:08.011Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:52:12.229 DEBUG:boto:path=/pklxkd-59/ 2023-05-31T20:52:12.229 DEBUG:boto:auth_path=/pklxkd-59/ 2023-05-31T20:52:12.230 DEBUG:boto:path=/pklxkd-59/?prefix=obj51 2023-05-31T20:52:12.230 DEBUG:boto:auth_path=/pklxkd-59/?prefix=obj51 2023-05-31T20:52:12.230 DEBUG:boto:Method: GET 2023-05-31T20:52:12.231 DEBUG:boto:Path: /pklxkd-59/?prefix=obj51 2023-05-31T20:52:12.231 DEBUG:boto:Data: 2023-05-31T20:52:12.231 DEBUG:boto:Headers: {} 2023-05-31T20:52:12.232 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:52:12.232 DEBUG:boto:Port: 8000 2023-05-31T20:52:12.233 DEBUG:boto:Params: {} 2023-05-31T20:52:12.233 DEBUG:boto:Token: None 2023-05-31T20:52:12.233 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:52:12 GMT /pklxkd-59/ 2023-05-31T20:52:12.234 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:pLqt3uLOcBEwAGLY0jYnxgHFOQo= 2023-05-31T20:52:12.234 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:52:12 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:pLqt3uLOcBEwAGLY0jYnxgHFOQo=', 'Content-Length': '0'} 2023-05-31T20:52:12.238 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000717827da4922114b-006477b37c-1630-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:52:12 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:52:12.238 DEBUG:boto:b'pklxkd-59obj511000falseobj512023-05-31T20:43:08.011Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:52:12.239 DEBUG:boto:path=/pklxkd-59/obj51 2023-05-31T20:52:12.239 DEBUG:boto:auth_path=/pklxkd-59/obj51 2023-05-31T20:52:12.240 DEBUG:boto:Method: HEAD 2023-05-31T20:52:12.240 DEBUG:boto:Path: /pklxkd-59/obj51 2023-05-31T20:52:12.240 DEBUG:boto:Data: 2023-05-31T20:52:12.241 DEBUG:boto:Headers: {} 2023-05-31T20:52:12.241 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:52:12.241 DEBUG:boto:Port: 8001 2023-05-31T20:52:12.242 DEBUG:boto:Params: {} 2023-05-31T20:52:12.242 DEBUG:boto:Token: None 2023-05-31T20:52:12.242 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:52:12 GMT /pklxkd-59/obj51 2023-05-31T20:52:12.243 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:RT0ji1PqGbvT7amynbD3RueZNts= 2023-05-31T20:52:12.243 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:52:12 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:RT0ji1PqGbvT7amynbD3RueZNts=', 'Content-Length': '0'} 2023-05-31T20:52:12.245 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:43:08 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000051860e392a08a1b1-006477b37c-1c17-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:52:12 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:52:12.246 DEBUG:boto:path=/pklxkd-59/obj51 2023-05-31T20:52:12.246 DEBUG:boto:auth_path=/pklxkd-59/obj51 2023-05-31T20:52:12.246 DEBUG:boto:Method: HEAD 2023-05-31T20:52:12.247 DEBUG:boto:Path: /pklxkd-59/obj51 2023-05-31T20:52:12.247 DEBUG:boto:Data: 2023-05-31T20:52:12.247 DEBUG:boto:Headers: {} 2023-05-31T20:52:12.248 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:52:12.248 DEBUG:boto:Port: 8000 2023-05-31T20:52:12.248 DEBUG:boto:Params: {} 2023-05-31T20:52:12.249 DEBUG:boto:Token: None 2023-05-31T20:52:12.249 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:52:12 GMT /pklxkd-59/obj51 2023-05-31T20:52:12.250 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:RT0ji1PqGbvT7amynbD3RueZNts= 2023-05-31T20:52:12.250 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:52:12 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:RT0ji1PqGbvT7amynbD3RueZNts=', 'Content-Length': '0'} 2023-05-31T20:52:12.252 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:43:08 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000049ac9defa0b887e5-006477b37c-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:52:12 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:52:12.252 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj51 2023-05-31T20:52:12.253 DEBUG:boto:path=/pklxkd-59/obj51 2023-05-31T20:52:12.253 DEBUG:boto:auth_path=/pklxkd-59/obj51 2023-05-31T20:52:12.253 DEBUG:boto:Method: GET 2023-05-31T20:52:12.254 DEBUG:boto:Path: /pklxkd-59/obj51 2023-05-31T20:52:12.254 DEBUG:boto:Data: 2023-05-31T20:52:12.254 DEBUG:boto:Headers: {} 2023-05-31T20:52:12.255 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:52:12.255 DEBUG:boto:Port: 8001 2023-05-31T20:52:12.255 DEBUG:boto:Params: {} 2023-05-31T20:52:12.256 DEBUG:boto:Token: None 2023-05-31T20:52:12.256 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:52:12 GMT /pklxkd-59/obj51 2023-05-31T20:52:12.257 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:INNwm7CIPEIMnbTQ2eVJA1Ljp4k= 2023-05-31T20:52:12.257 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:52:12 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:INNwm7CIPEIMnbTQ2eVJA1Ljp4k=', 'Content-Length': '0'} 2023-05-31T20:52:12.259 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:43:08 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002440e86931b191d5-006477b37c-1c17-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:52:12 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:52:12.300 DEBUG:boto:path=/pklxkd-59/obj51 2023-05-31T20:52:12.301 DEBUG:boto:auth_path=/pklxkd-59/obj51 2023-05-31T20:52:12.301 DEBUG:boto:Method: GET 2023-05-31T20:52:12.301 DEBUG:boto:Path: /pklxkd-59/obj51 2023-05-31T20:52:12.302 DEBUG:boto:Data: 2023-05-31T20:52:12.302 DEBUG:boto:Headers: {} 2023-05-31T20:52:12.302 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:52:12.303 DEBUG:boto:Port: 8000 2023-05-31T20:52:12.303 DEBUG:boto:Params: {} 2023-05-31T20:52:12.304 DEBUG:boto:Token: None 2023-05-31T20:52:12.304 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:52:12 GMT /pklxkd-59/obj51 2023-05-31T20:52:12.305 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:INNwm7CIPEIMnbTQ2eVJA1Ljp4k= 2023-05-31T20:52:12.305 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:52:12 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:INNwm7CIPEIMnbTQ2eVJA1Ljp4k=', 'Content-Length': '0'} 2023-05-31T20:52:12.307 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:43:08 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000095e2bad75928ae14-006477b37c-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:52:12 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:52:12.307 DEBUG:boto:path=/pklxkd-59/ 2023-05-31T20:52:12.308 DEBUG:boto:auth_path=/pklxkd-59/ 2023-05-31T20:52:12.308 DEBUG:boto:path=/pklxkd-59/?prefix=obj51 2023-05-31T20:52:12.309 DEBUG:boto:auth_path=/pklxkd-59/?prefix=obj51 2023-05-31T20:52:12.309 DEBUG:boto:Method: GET 2023-05-31T20:52:12.309 DEBUG:boto:Path: /pklxkd-59/?prefix=obj51 2023-05-31T20:52:12.310 DEBUG:boto:Data: 2023-05-31T20:52:12.310 DEBUG:boto:Headers: {} 2023-05-31T20:52:12.310 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:52:12.311 DEBUG:boto:Port: 8001 2023-05-31T20:52:12.311 DEBUG:boto:Params: {} 2023-05-31T20:52:12.311 DEBUG:boto:Token: None 2023-05-31T20:52:12.312 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:52:12 GMT /pklxkd-59/ 2023-05-31T20:52:12.312 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:pLqt3uLOcBEwAGLY0jYnxgHFOQo= 2023-05-31T20:52:12.313 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:52:12 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:pLqt3uLOcBEwAGLY0jYnxgHFOQo=', 'Content-Length': '0'} 2023-05-31T20:52:12.315 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000035ef09cee91e7d74-006477b37c-1c17-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:52:12 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:52:12.316 DEBUG:boto:b'pklxkd-59obj511000falseobj512023-05-31T20:43:08.011Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:52:12.316 DEBUG:boto:path=/pklxkd-59/ 2023-05-31T20:52:12.317 DEBUG:boto:auth_path=/pklxkd-59/ 2023-05-31T20:52:12.317 DEBUG:boto:path=/pklxkd-59/?prefix=obj51 2023-05-31T20:52:12.317 DEBUG:boto:auth_path=/pklxkd-59/?prefix=obj51 2023-05-31T20:52:12.318 DEBUG:boto:Method: GET 2023-05-31T20:52:12.318 DEBUG:boto:Path: /pklxkd-59/?prefix=obj51 2023-05-31T20:52:12.319 DEBUG:boto:Data: 2023-05-31T20:52:12.319 DEBUG:boto:Headers: {} 2023-05-31T20:52:12.319 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:52:12.320 DEBUG:boto:Port: 8000 2023-05-31T20:52:12.320 DEBUG:boto:Params: {} 2023-05-31T20:52:12.320 DEBUG:boto:Token: None 2023-05-31T20:52:12.321 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:52:12 GMT /pklxkd-59/ 2023-05-31T20:52:12.321 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:pLqt3uLOcBEwAGLY0jYnxgHFOQo= 2023-05-31T20:52:12.322 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:52:12 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:pLqt3uLOcBEwAGLY0jYnxgHFOQo=', 'Content-Length': '0'} 2023-05-31T20:52:12.325 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000afc24361b6a52027-006477b37c-1630-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:52:12 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:52:12.326 DEBUG:boto:b'pklxkd-59obj511000falseobj512023-05-31T20:43:08.011Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:52:12.326 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj55 2023-05-31T20:52:12.327 DEBUG:boto:path=/pklxkd-59/obj55 2023-05-31T20:52:12.327 DEBUG:boto:auth_path=/pklxkd-59/obj55 2023-05-31T20:52:12.327 DEBUG:boto:path=/pklxkd-59/obj55?versionId=null 2023-05-31T20:52:12.328 DEBUG:boto:auth_path=/pklxkd-59/obj55?versionId=null 2023-05-31T20:52:12.328 DEBUG:boto:Method: GET 2023-05-31T20:52:12.329 DEBUG:boto:Path: /pklxkd-59/obj55?versionId=null 2023-05-31T20:52:12.329 DEBUG:boto:Data: 2023-05-31T20:52:12.329 DEBUG:boto:Headers: {} 2023-05-31T20:52:12.330 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:52:12.330 DEBUG:boto:Port: 8001 2023-05-31T20:52:12.330 DEBUG:boto:Params: {} 2023-05-31T20:52:12.331 DEBUG:boto:Token: None 2023-05-31T20:52:12.331 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:52:12 GMT /pklxkd-59/obj55?versionId=null 2023-05-31T20:52:12.332 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:y800/UcwYHWSk73rMqHRWTKJpw0= 2023-05-31T20:52:12.332 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:52:12 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:y800/UcwYHWSk73rMqHRWTKJpw0=', 'Content-Length': '0'} 2023-05-31T20:52:12.334 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:39:26 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000be02ea076a9c4cc2-006477b37c-1c17-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:52:12 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:52:12.334 DEBUG:boto:path=/pklxkd-59/obj55 2023-05-31T20:52:12.335 DEBUG:boto:auth_path=/pklxkd-59/obj55 2023-05-31T20:52:12.335 DEBUG:boto:path=/pklxkd-59/obj55?versionId=null 2023-05-31T20:52:12.336 DEBUG:boto:auth_path=/pklxkd-59/obj55?versionId=null 2023-05-31T20:52:12.336 DEBUG:boto:Method: GET 2023-05-31T20:52:12.336 DEBUG:boto:Path: /pklxkd-59/obj55?versionId=null 2023-05-31T20:52:12.337 DEBUG:boto:Data: 2023-05-31T20:52:12.337 DEBUG:boto:Headers: {} 2023-05-31T20:52:12.337 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:52:12.338 DEBUG:boto:Port: 8000 2023-05-31T20:52:12.338 DEBUG:boto:Params: {} 2023-05-31T20:52:12.338 DEBUG:boto:Token: None 2023-05-31T20:52:12.339 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:52:12 GMT /pklxkd-59/obj55?versionId=null 2023-05-31T20:52:12.339 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:y800/UcwYHWSk73rMqHRWTKJpw0= 2023-05-31T20:52:12.340 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:52:12 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:y800/UcwYHWSk73rMqHRWTKJpw0=', 'Content-Length': '0'} 2023-05-31T20:52:12.342 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:39:26 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007f97f236a2dff405-006477b37c-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:52:12 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:52:12.384 DEBUG:boto:path=/pklxkd-59/obj55 2023-05-31T20:52:12.385 DEBUG:boto:auth_path=/pklxkd-59/obj55 2023-05-31T20:52:12.385 DEBUG:boto:path=/pklxkd-59/obj55?versionId=null 2023-05-31T20:52:12.385 DEBUG:boto:auth_path=/pklxkd-59/obj55?versionId=null 2023-05-31T20:52:12.386 DEBUG:boto:Method: HEAD 2023-05-31T20:52:12.386 DEBUG:boto:Path: /pklxkd-59/obj55?versionId=null 2023-05-31T20:52:12.386 DEBUG:boto:Data: 2023-05-31T20:52:12.387 DEBUG:boto:Headers: {} 2023-05-31T20:52:12.387 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:52:12.387 DEBUG:boto:Port: 8001 2023-05-31T20:52:12.388 DEBUG:boto:Params: {} 2023-05-31T20:52:12.388 DEBUG:boto:Token: None 2023-05-31T20:52:12.388 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:52:12 GMT /pklxkd-59/obj55?versionId=null 2023-05-31T20:52:12.389 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:0Ezyc9dGR/hU1piUMkRjh1E3FxA= 2023-05-31T20:52:12.389 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:52:12 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:0Ezyc9dGR/hU1piUMkRjh1E3FxA=', 'Content-Length': '0'} 2023-05-31T20:52:12.391 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:39:26 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b1566fdefe3e368d-006477b37c-1c17-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:52:12 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:52:12.392 DEBUG:boto:path=/pklxkd-59/obj55 2023-05-31T20:52:12.392 DEBUG:boto:auth_path=/pklxkd-59/obj55 2023-05-31T20:52:12.392 DEBUG:boto:path=/pklxkd-59/obj55?versionId=null 2023-05-31T20:52:12.393 DEBUG:boto:auth_path=/pklxkd-59/obj55?versionId=null 2023-05-31T20:52:12.393 DEBUG:boto:Method: HEAD 2023-05-31T20:52:12.393 DEBUG:boto:Path: /pklxkd-59/obj55?versionId=null 2023-05-31T20:52:12.394 DEBUG:boto:Data: 2023-05-31T20:52:12.394 DEBUG:boto:Headers: {} 2023-05-31T20:52:12.394 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:52:12.395 DEBUG:boto:Port: 8000 2023-05-31T20:52:12.395 DEBUG:boto:Params: {} 2023-05-31T20:52:12.395 DEBUG:boto:Token: None 2023-05-31T20:52:12.396 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:52:12 GMT /pklxkd-59/obj55?versionId=null 2023-05-31T20:52:12.396 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:0Ezyc9dGR/hU1piUMkRjh1E3FxA= 2023-05-31T20:52:12.397 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:52:12 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:0Ezyc9dGR/hU1piUMkRjh1E3FxA=', 'Content-Length': '0'} 2023-05-31T20:52:12.398 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:39:26 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000df3d511b683df9ec-006477b37c-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:52:12 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:52:12.399 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj55 2023-05-31T20:52:12.399 DEBUG:boto:path=/pklxkd-59/obj55 2023-05-31T20:52:12.400 DEBUG:boto:auth_path=/pklxkd-59/obj55 2023-05-31T20:52:12.400 DEBUG:boto:path=/pklxkd-59/obj55?versionId=null 2023-05-31T20:52:12.400 DEBUG:boto:auth_path=/pklxkd-59/obj55?versionId=null 2023-05-31T20:52:12.401 DEBUG:boto:Method: GET 2023-05-31T20:52:12.404 DEBUG:boto:Path: /pklxkd-59/obj55?versionId=null 2023-05-31T20:52:12.404 DEBUG:boto:Data: 2023-05-31T20:52:12.405 DEBUG:boto:Headers: {} 2023-05-31T20:52:12.405 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:52:12.405 DEBUG:boto:Port: 8001 2023-05-31T20:52:12.406 DEBUG:boto:Params: {} 2023-05-31T20:52:12.406 DEBUG:boto:Token: None 2023-05-31T20:52:12.406 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:52:12 GMT /pklxkd-59/obj55?versionId=null 2023-05-31T20:52:12.407 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:y800/UcwYHWSk73rMqHRWTKJpw0= 2023-05-31T20:52:12.407 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:52:12 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:y800/UcwYHWSk73rMqHRWTKJpw0=', 'Content-Length': '0'} 2023-05-31T20:52:12.409 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:39:26 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003736fb51187fd458-006477b37c-1c17-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:52:12 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:52:12.410 DEBUG:boto:path=/pklxkd-59/obj55 2023-05-31T20:52:12.410 DEBUG:boto:auth_path=/pklxkd-59/obj55 2023-05-31T20:52:12.411 DEBUG:boto:path=/pklxkd-59/obj55?versionId=null 2023-05-31T20:52:12.411 DEBUG:boto:auth_path=/pklxkd-59/obj55?versionId=null 2023-05-31T20:52:12.411 DEBUG:boto:Method: GET 2023-05-31T20:52:12.412 DEBUG:boto:Path: /pklxkd-59/obj55?versionId=null 2023-05-31T20:52:12.412 DEBUG:boto:Data: 2023-05-31T20:52:12.412 DEBUG:boto:Headers: {} 2023-05-31T20:52:12.413 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:52:12.413 DEBUG:boto:Port: 8000 2023-05-31T20:52:12.413 DEBUG:boto:Params: {} 2023-05-31T20:52:12.414 DEBUG:boto:Token: None 2023-05-31T20:52:12.414 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:52:12 GMT /pklxkd-59/obj55?versionId=null 2023-05-31T20:52:12.415 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:y800/UcwYHWSk73rMqHRWTKJpw0= 2023-05-31T20:52:12.416 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:52:12 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:y800/UcwYHWSk73rMqHRWTKJpw0=', 'Content-Length': '0'} 2023-05-31T20:52:12.418 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:39:26 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000016a44d541f10af27-006477b37c-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:52:12 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:52:12.418 DEBUG:boto:path=/pklxkd-59/ 2023-05-31T20:52:12.419 DEBUG:boto:auth_path=/pklxkd-59/ 2023-05-31T20:52:12.419 DEBUG:boto:path=/pklxkd-59/?prefix=obj55 2023-05-31T20:52:12.420 DEBUG:boto:auth_path=/pklxkd-59/?prefix=obj55 2023-05-31T20:52:12.420 DEBUG:boto:Method: GET 2023-05-31T20:52:12.420 DEBUG:boto:Path: /pklxkd-59/?prefix=obj55 2023-05-31T20:52:12.421 DEBUG:boto:Data: 2023-05-31T20:52:12.421 DEBUG:boto:Headers: {} 2023-05-31T20:52:12.421 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:52:12.422 DEBUG:boto:Port: 8001 2023-05-31T20:52:12.422 DEBUG:boto:Params: {} 2023-05-31T20:52:12.423 DEBUG:boto:Token: None 2023-05-31T20:52:12.423 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:52:12 GMT /pklxkd-59/ 2023-05-31T20:52:12.423 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:pLqt3uLOcBEwAGLY0jYnxgHFOQo= 2023-05-31T20:52:12.424 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:52:12 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:pLqt3uLOcBEwAGLY0jYnxgHFOQo=', 'Content-Length': '0'} 2023-05-31T20:52:12.426 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000008e21111b01522022-006477b37c-1c17-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:52:12 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:52:12.427 DEBUG:boto:b'pklxkd-59obj551000falseobj552023-05-31T20:39:26.630Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:52:12.427 DEBUG:boto:path=/pklxkd-59/ 2023-05-31T20:52:12.428 DEBUG:boto:auth_path=/pklxkd-59/ 2023-05-31T20:52:12.428 DEBUG:boto:path=/pklxkd-59/?prefix=obj55 2023-05-31T20:52:12.429 DEBUG:boto:auth_path=/pklxkd-59/?prefix=obj55 2023-05-31T20:52:12.429 DEBUG:boto:Method: GET 2023-05-31T20:52:12.429 DEBUG:boto:Path: /pklxkd-59/?prefix=obj55 2023-05-31T20:52:12.430 DEBUG:boto:Data: 2023-05-31T20:52:12.430 DEBUG:boto:Headers: {} 2023-05-31T20:52:12.430 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:52:12.431 DEBUG:boto:Port: 8000 2023-05-31T20:52:12.431 DEBUG:boto:Params: {} 2023-05-31T20:52:12.431 DEBUG:boto:Token: None 2023-05-31T20:52:12.432 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:52:12 GMT /pklxkd-59/ 2023-05-31T20:52:12.432 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:pLqt3uLOcBEwAGLY0jYnxgHFOQo= 2023-05-31T20:52:12.433 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:52:12 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:pLqt3uLOcBEwAGLY0jYnxgHFOQo=', 'Content-Length': '0'} 2023-05-31T20:52:12.436 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b005afce4a9e3eef-006477b37c-1630-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:52:12 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:52:12.437 DEBUG:boto:b'pklxkd-59obj551000falseobj552023-05-31T20:39:26.630Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:52:12.437 DEBUG:boto:path=/pklxkd-59/obj55 2023-05-31T20:52:12.438 DEBUG:boto:auth_path=/pklxkd-59/obj55 2023-05-31T20:52:12.438 DEBUG:boto:Method: HEAD 2023-05-31T20:52:12.438 DEBUG:boto:Path: /pklxkd-59/obj55 2023-05-31T20:52:12.439 DEBUG:boto:Data: 2023-05-31T20:52:12.439 DEBUG:boto:Headers: {} 2023-05-31T20:52:12.439 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:52:12.440 DEBUG:boto:Port: 8001 2023-05-31T20:52:12.440 DEBUG:boto:Params: {} 2023-05-31T20:52:12.440 DEBUG:boto:Token: None 2023-05-31T20:52:12.441 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:52:12 GMT /pklxkd-59/obj55 2023-05-31T20:52:12.441 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:ReadHtmAkSeC6XphEacc40XgWCE= 2023-05-31T20:52:12.441 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:52:12 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:ReadHtmAkSeC6XphEacc40XgWCE=', 'Content-Length': '0'} 2023-05-31T20:52:12.444 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:39:26 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000054bfcf47a684deb1-006477b37c-1c17-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:52:12 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:52:12.444 DEBUG:boto:path=/pklxkd-59/obj55 2023-05-31T20:52:12.445 DEBUG:boto:auth_path=/pklxkd-59/obj55 2023-05-31T20:52:12.445 DEBUG:boto:Method: HEAD 2023-05-31T20:52:12.445 DEBUG:boto:Path: /pklxkd-59/obj55 2023-05-31T20:52:12.446 DEBUG:boto:Data: 2023-05-31T20:52:12.446 DEBUG:boto:Headers: {} 2023-05-31T20:52:12.446 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:52:12.447 DEBUG:boto:Port: 8000 2023-05-31T20:52:12.447 DEBUG:boto:Params: {} 2023-05-31T20:52:12.447 DEBUG:boto:Token: None 2023-05-31T20:52:12.448 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:52:12 GMT /pklxkd-59/obj55 2023-05-31T20:52:12.448 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:ReadHtmAkSeC6XphEacc40XgWCE= 2023-05-31T20:52:12.448 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:52:12 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:ReadHtmAkSeC6XphEacc40XgWCE=', 'Content-Length': '0'} 2023-05-31T20:52:12.450 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:39:26 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000eaeb05512babbf83-006477b37c-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:52:12 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:52:12.451 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj55 2023-05-31T20:52:12.451 DEBUG:boto:path=/pklxkd-59/obj55 2023-05-31T20:52:12.452 DEBUG:boto:auth_path=/pklxkd-59/obj55 2023-05-31T20:52:12.452 DEBUG:boto:Method: GET 2023-05-31T20:52:12.452 DEBUG:boto:Path: /pklxkd-59/obj55 2023-05-31T20:52:12.453 DEBUG:boto:Data: 2023-05-31T20:52:12.453 DEBUG:boto:Headers: {} 2023-05-31T20:52:12.454 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:52:12.454 DEBUG:boto:Port: 8001 2023-05-31T20:52:12.454 DEBUG:boto:Params: {} 2023-05-31T20:52:12.455 DEBUG:boto:Token: None 2023-05-31T20:52:12.455 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:52:12 GMT /pklxkd-59/obj55 2023-05-31T20:52:12.455 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:Q905s05H9+pRO6geCZq2GJ+qgYk= 2023-05-31T20:52:12.456 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:52:12 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:Q905s05H9+pRO6geCZq2GJ+qgYk=', 'Content-Length': '0'} 2023-05-31T20:52:12.458 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:39:26 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007cdca3a9ff891575-006477b37c-1c17-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:52:12 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:52:12.458 DEBUG:boto:path=/pklxkd-59/obj55 2023-05-31T20:52:12.459 DEBUG:boto:auth_path=/pklxkd-59/obj55 2023-05-31T20:52:12.459 DEBUG:boto:Method: GET 2023-05-31T20:52:12.460 DEBUG:boto:Path: /pklxkd-59/obj55 2023-05-31T20:52:12.460 DEBUG:boto:Data: 2023-05-31T20:52:12.460 DEBUG:boto:Headers: {} 2023-05-31T20:52:12.461 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:52:12.461 DEBUG:boto:Port: 8000 2023-05-31T20:52:12.462 DEBUG:boto:Params: {} 2023-05-31T20:52:12.462 DEBUG:boto:Token: None 2023-05-31T20:52:12.462 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:52:12 GMT /pklxkd-59/obj55 2023-05-31T20:52:12.463 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:Q905s05H9+pRO6geCZq2GJ+qgYk= 2023-05-31T20:52:12.463 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:52:12 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:Q905s05H9+pRO6geCZq2GJ+qgYk=', 'Content-Length': '0'} 2023-05-31T20:52:12.465 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:39:26 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003aefd98345a3ea43-006477b37c-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:52:12 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:52:12.466 DEBUG:boto:path=/pklxkd-59/ 2023-05-31T20:52:12.466 DEBUG:boto:auth_path=/pklxkd-59/ 2023-05-31T20:52:12.467 DEBUG:boto:path=/pklxkd-59/?prefix=obj55 2023-05-31T20:52:12.467 DEBUG:boto:auth_path=/pklxkd-59/?prefix=obj55 2023-05-31T20:52:12.467 DEBUG:boto:Method: GET 2023-05-31T20:52:12.468 DEBUG:boto:Path: /pklxkd-59/?prefix=obj55 2023-05-31T20:52:12.468 DEBUG:boto:Data: 2023-05-31T20:52:12.468 DEBUG:boto:Headers: {} 2023-05-31T20:52:12.469 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:52:12.469 DEBUG:boto:Port: 8001 2023-05-31T20:52:12.469 DEBUG:boto:Params: {} 2023-05-31T20:52:12.470 DEBUG:boto:Token: None 2023-05-31T20:52:12.470 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:52:12 GMT /pklxkd-59/ 2023-05-31T20:52:12.471 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:pLqt3uLOcBEwAGLY0jYnxgHFOQo= 2023-05-31T20:52:12.471 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:52:12 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:pLqt3uLOcBEwAGLY0jYnxgHFOQo=', 'Content-Length': '0'} 2023-05-31T20:52:12.474 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000007aa7e6975f89f459-006477b37c-1c17-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:52:12 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:52:12.474 DEBUG:boto:b'pklxkd-59obj551000falseobj552023-05-31T20:39:26.630Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:52:12.476 DEBUG:boto:path=/pklxkd-59/ 2023-05-31T20:52:12.476 DEBUG:boto:auth_path=/pklxkd-59/ 2023-05-31T20:52:12.477 DEBUG:boto:path=/pklxkd-59/?prefix=obj55 2023-05-31T20:52:12.477 DEBUG:boto:auth_path=/pklxkd-59/?prefix=obj55 2023-05-31T20:52:12.478 DEBUG:boto:Method: GET 2023-05-31T20:52:12.478 DEBUG:boto:Path: /pklxkd-59/?prefix=obj55 2023-05-31T20:52:12.479 DEBUG:boto:Data: 2023-05-31T20:52:12.480 DEBUG:boto:Headers: {} 2023-05-31T20:52:12.481 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:52:12.481 DEBUG:boto:Port: 8000 2023-05-31T20:52:12.481 DEBUG:boto:Params: {} 2023-05-31T20:52:12.482 DEBUG:boto:Token: None 2023-05-31T20:52:12.482 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:52:12 GMT /pklxkd-59/ 2023-05-31T20:52:12.483 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:pLqt3uLOcBEwAGLY0jYnxgHFOQo= 2023-05-31T20:52:12.483 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:52:12 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:pLqt3uLOcBEwAGLY0jYnxgHFOQo=', 'Content-Length': '0'} 2023-05-31T20:52:12.487 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000009bb0a97d67cea204-006477b37c-1630-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:52:12 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:52:12.528 DEBUG:boto:b'pklxkd-59obj551000falseobj552023-05-31T20:39:26.630Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:52:12.529 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj57 2023-05-31T20:52:12.530 DEBUG:boto:path=/pklxkd-59/obj57 2023-05-31T20:52:12.530 DEBUG:boto:auth_path=/pklxkd-59/obj57 2023-05-31T20:52:12.530 DEBUG:boto:path=/pklxkd-59/obj57?versionId=null 2023-05-31T20:52:12.531 DEBUG:boto:auth_path=/pklxkd-59/obj57?versionId=null 2023-05-31T20:52:12.531 DEBUG:boto:Method: GET 2023-05-31T20:52:12.532 DEBUG:boto:Path: /pklxkd-59/obj57?versionId=null 2023-05-31T20:52:12.532 DEBUG:boto:Data: 2023-05-31T20:52:12.533 DEBUG:boto:Headers: {} 2023-05-31T20:52:12.533 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:52:12.533 DEBUG:boto:Port: 8001 2023-05-31T20:52:12.534 DEBUG:boto:Params: {} 2023-05-31T20:52:12.534 DEBUG:boto:Token: None 2023-05-31T20:52:12.535 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:52:12 GMT /pklxkd-59/obj57?versionId=null 2023-05-31T20:52:12.535 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:bG5+1L3dXAFTfjZ2BYUXu9zIrU4= 2023-05-31T20:52:12.536 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:52:12 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:bG5+1L3dXAFTfjZ2BYUXu9zIrU4=', 'Content-Length': '0'} 2023-05-31T20:52:12.538 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:44:50 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000223bcd37ff4da099-006477b37c-1c17-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:52:12 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:52:12.539 DEBUG:boto:path=/pklxkd-59/obj57 2023-05-31T20:52:12.539 DEBUG:boto:auth_path=/pklxkd-59/obj57 2023-05-31T20:52:12.540 DEBUG:boto:path=/pklxkd-59/obj57?versionId=null 2023-05-31T20:52:12.540 DEBUG:boto:auth_path=/pklxkd-59/obj57?versionId=null 2023-05-31T20:52:12.540 DEBUG:boto:Method: GET 2023-05-31T20:52:12.541 DEBUG:boto:Path: /pklxkd-59/obj57?versionId=null 2023-05-31T20:52:12.541 DEBUG:boto:Data: 2023-05-31T20:52:12.542 DEBUG:boto:Headers: {} 2023-05-31T20:52:12.542 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:52:12.542 DEBUG:boto:Port: 8000 2023-05-31T20:52:12.543 DEBUG:boto:Params: {} 2023-05-31T20:52:12.543 DEBUG:boto:Token: None 2023-05-31T20:52:12.544 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:52:12 GMT /pklxkd-59/obj57?versionId=null 2023-05-31T20:52:12.544 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:bG5+1L3dXAFTfjZ2BYUXu9zIrU4= 2023-05-31T20:52:12.545 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:52:12 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:bG5+1L3dXAFTfjZ2BYUXu9zIrU4=', 'Content-Length': '0'} 2023-05-31T20:52:12.547 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:44:50 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ddae356e39b67151-006477b37c-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:52:12 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:52:12.548 DEBUG:boto:path=/pklxkd-59/obj57 2023-05-31T20:52:12.548 DEBUG:boto:auth_path=/pklxkd-59/obj57 2023-05-31T20:52:12.548 DEBUG:boto:path=/pklxkd-59/obj57?versionId=null 2023-05-31T20:52:12.549 DEBUG:boto:auth_path=/pklxkd-59/obj57?versionId=null 2023-05-31T20:52:12.549 DEBUG:boto:Method: HEAD 2023-05-31T20:52:12.549 DEBUG:boto:Path: /pklxkd-59/obj57?versionId=null 2023-05-31T20:52:12.550 DEBUG:boto:Data: 2023-05-31T20:52:12.550 DEBUG:boto:Headers: {} 2023-05-31T20:52:12.550 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:52:12.551 DEBUG:boto:Port: 8001 2023-05-31T20:52:12.551 DEBUG:boto:Params: {} 2023-05-31T20:52:12.552 DEBUG:boto:Token: None 2023-05-31T20:52:12.552 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:52:12 GMT /pklxkd-59/obj57?versionId=null 2023-05-31T20:52:12.552 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:X+JdyKlwu8ATDh39rPLpvtN5MCo= 2023-05-31T20:52:12.553 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:52:12 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:X+JdyKlwu8ATDh39rPLpvtN5MCo=', 'Content-Length': '0'} 2023-05-31T20:52:12.555 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:44:50 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000076e3ecd68015fbb2-006477b37c-1c17-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:52:12 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:52:12.555 DEBUG:boto:path=/pklxkd-59/obj57 2023-05-31T20:52:12.556 DEBUG:boto:auth_path=/pklxkd-59/obj57 2023-05-31T20:52:12.556 DEBUG:boto:path=/pklxkd-59/obj57?versionId=null 2023-05-31T20:52:12.557 DEBUG:boto:auth_path=/pklxkd-59/obj57?versionId=null 2023-05-31T20:52:12.557 DEBUG:boto:Method: HEAD 2023-05-31T20:52:12.557 DEBUG:boto:Path: /pklxkd-59/obj57?versionId=null 2023-05-31T20:52:12.558 DEBUG:boto:Data: 2023-05-31T20:52:12.558 DEBUG:boto:Headers: {} 2023-05-31T20:52:12.558 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:52:12.559 DEBUG:boto:Port: 8000 2023-05-31T20:52:12.559 DEBUG:boto:Params: {} 2023-05-31T20:52:12.559 DEBUG:boto:Token: None 2023-05-31T20:52:12.560 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:52:12 GMT /pklxkd-59/obj57?versionId=null 2023-05-31T20:52:12.560 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:X+JdyKlwu8ATDh39rPLpvtN5MCo= 2023-05-31T20:52:12.561 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:52:12 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:X+JdyKlwu8ATDh39rPLpvtN5MCo=', 'Content-Length': '0'} 2023-05-31T20:52:12.563 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:44:50 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007803f682b84c0e37-006477b37c-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:52:12 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:52:12.563 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj57 2023-05-31T20:52:12.564 DEBUG:boto:path=/pklxkd-59/obj57 2023-05-31T20:52:12.564 DEBUG:boto:auth_path=/pklxkd-59/obj57 2023-05-31T20:52:12.564 DEBUG:boto:path=/pklxkd-59/obj57?versionId=null 2023-05-31T20:52:12.565 DEBUG:boto:auth_path=/pklxkd-59/obj57?versionId=null 2023-05-31T20:52:12.565 DEBUG:boto:Method: GET 2023-05-31T20:52:12.566 DEBUG:boto:Path: /pklxkd-59/obj57?versionId=null 2023-05-31T20:52:12.566 DEBUG:boto:Data: 2023-05-31T20:52:12.567 DEBUG:boto:Headers: {} 2023-05-31T20:52:12.567 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:52:12.568 DEBUG:boto:Port: 8001 2023-05-31T20:52:12.568 DEBUG:boto:Params: {} 2023-05-31T20:52:12.569 DEBUG:boto:Token: None 2023-05-31T20:52:12.569 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:52:12 GMT /pklxkd-59/obj57?versionId=null 2023-05-31T20:52:12.570 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:bG5+1L3dXAFTfjZ2BYUXu9zIrU4= 2023-05-31T20:52:12.570 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:52:12 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:bG5+1L3dXAFTfjZ2BYUXu9zIrU4=', 'Content-Length': '0'} 2023-05-31T20:52:12.572 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:44:50 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ad78a9abe3b6ced8-006477b37c-1c17-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:52:12 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:52:12.573 DEBUG:boto:path=/pklxkd-59/obj57 2023-05-31T20:52:12.573 DEBUG:boto:auth_path=/pklxkd-59/obj57 2023-05-31T20:52:12.573 DEBUG:boto:path=/pklxkd-59/obj57?versionId=null 2023-05-31T20:52:12.574 DEBUG:boto:auth_path=/pklxkd-59/obj57?versionId=null 2023-05-31T20:52:12.574 DEBUG:boto:Method: GET 2023-05-31T20:52:12.575 DEBUG:boto:Path: /pklxkd-59/obj57?versionId=null 2023-05-31T20:52:12.575 DEBUG:boto:Data: 2023-05-31T20:52:12.575 DEBUG:boto:Headers: {} 2023-05-31T20:52:12.576 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:52:12.576 DEBUG:boto:Port: 8000 2023-05-31T20:52:12.576 DEBUG:boto:Params: {} 2023-05-31T20:52:12.577 DEBUG:boto:Token: None 2023-05-31T20:52:12.577 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:52:12 GMT /pklxkd-59/obj57?versionId=null 2023-05-31T20:52:12.578 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:bG5+1L3dXAFTfjZ2BYUXu9zIrU4= 2023-05-31T20:52:12.578 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:52:12 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:bG5+1L3dXAFTfjZ2BYUXu9zIrU4=', 'Content-Length': '0'} 2023-05-31T20:52:12.580 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:44:50 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006f041e86b2d1607e-006477b37c-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:52:12 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:52:12.581 DEBUG:boto:path=/pklxkd-59/ 2023-05-31T20:52:12.581 DEBUG:boto:auth_path=/pklxkd-59/ 2023-05-31T20:52:12.582 DEBUG:boto:path=/pklxkd-59/?prefix=obj57 2023-05-31T20:52:12.582 DEBUG:boto:auth_path=/pklxkd-59/?prefix=obj57 2023-05-31T20:52:12.582 DEBUG:boto:Method: GET 2023-05-31T20:52:12.583 DEBUG:boto:Path: /pklxkd-59/?prefix=obj57 2023-05-31T20:52:12.583 DEBUG:boto:Data: 2023-05-31T20:52:12.583 DEBUG:boto:Headers: {} 2023-05-31T20:52:12.584 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:52:12.584 DEBUG:boto:Port: 8001 2023-05-31T20:52:12.584 DEBUG:boto:Params: {} 2023-05-31T20:52:12.585 DEBUG:boto:Token: None 2023-05-31T20:52:12.585 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:52:12 GMT /pklxkd-59/ 2023-05-31T20:52:12.586 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:pLqt3uLOcBEwAGLY0jYnxgHFOQo= 2023-05-31T20:52:12.586 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:52:12 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:pLqt3uLOcBEwAGLY0jYnxgHFOQo=', 'Content-Length': '0'} 2023-05-31T20:52:12.588 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000fe6a08314cdb89e8-006477b37c-1c17-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:52:12 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:52:12.589 DEBUG:boto:b'pklxkd-59obj571000falseobj572023-05-31T20:44:50.627Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:52:12.590 DEBUG:boto:path=/pklxkd-59/ 2023-05-31T20:52:12.590 DEBUG:boto:auth_path=/pklxkd-59/ 2023-05-31T20:52:12.591 DEBUG:boto:path=/pklxkd-59/?prefix=obj57 2023-05-31T20:52:12.591 DEBUG:boto:auth_path=/pklxkd-59/?prefix=obj57 2023-05-31T20:52:12.591 DEBUG:boto:Method: GET 2023-05-31T20:52:12.592 DEBUG:boto:Path: /pklxkd-59/?prefix=obj57 2023-05-31T20:52:12.592 DEBUG:boto:Data: 2023-05-31T20:52:12.593 DEBUG:boto:Headers: {} 2023-05-31T20:52:12.593 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:52:12.593 DEBUG:boto:Port: 8000 2023-05-31T20:52:12.594 DEBUG:boto:Params: {} 2023-05-31T20:52:12.594 DEBUG:boto:Token: None 2023-05-31T20:52:12.595 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:52:12 GMT /pklxkd-59/ 2023-05-31T20:52:12.595 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:pLqt3uLOcBEwAGLY0jYnxgHFOQo= 2023-05-31T20:52:12.596 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:52:12 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:pLqt3uLOcBEwAGLY0jYnxgHFOQo=', 'Content-Length': '0'} 2023-05-31T20:52:12.599 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000043d731383a652e5e-006477b37c-1630-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:52:12 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:52:12.600 DEBUG:boto:b'pklxkd-59obj571000falseobj572023-05-31T20:44:50.627Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:52:12.600 DEBUG:boto:path=/pklxkd-59/obj57 2023-05-31T20:52:12.601 DEBUG:boto:auth_path=/pklxkd-59/obj57 2023-05-31T20:52:12.601 DEBUG:boto:Method: HEAD 2023-05-31T20:52:12.601 DEBUG:boto:Path: /pklxkd-59/obj57 2023-05-31T20:52:12.602 DEBUG:boto:Data: 2023-05-31T20:52:12.602 DEBUG:boto:Headers: {} 2023-05-31T20:52:12.602 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:52:12.603 DEBUG:boto:Port: 8001 2023-05-31T20:52:12.603 DEBUG:boto:Params: {} 2023-05-31T20:52:12.603 DEBUG:boto:Token: None 2023-05-31T20:52:12.604 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:52:12 GMT /pklxkd-59/obj57 2023-05-31T20:52:12.604 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:erp4SUqQlkgXCqbrinvL3cSN/Bk= 2023-05-31T20:52:12.605 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:52:12 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:erp4SUqQlkgXCqbrinvL3cSN/Bk=', 'Content-Length': '0'} 2023-05-31T20:52:12.606 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:44:50 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d10245f594784057-006477b37c-1c17-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:52:12 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:52:12.607 DEBUG:boto:path=/pklxkd-59/obj57 2023-05-31T20:52:12.607 DEBUG:boto:auth_path=/pklxkd-59/obj57 2023-05-31T20:52:12.608 DEBUG:boto:Method: HEAD 2023-05-31T20:52:12.608 DEBUG:boto:Path: /pklxkd-59/obj57 2023-05-31T20:52:12.609 DEBUG:boto:Data: 2023-05-31T20:52:12.609 DEBUG:boto:Headers: {} 2023-05-31T20:52:12.609 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:52:12.610 DEBUG:boto:Port: 8000 2023-05-31T20:52:12.610 DEBUG:boto:Params: {} 2023-05-31T20:52:12.610 DEBUG:boto:Token: None 2023-05-31T20:52:12.611 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:52:12 GMT /pklxkd-59/obj57 2023-05-31T20:52:12.611 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:erp4SUqQlkgXCqbrinvL3cSN/Bk= 2023-05-31T20:52:12.611 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:52:12 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:erp4SUqQlkgXCqbrinvL3cSN/Bk=', 'Content-Length': '0'} 2023-05-31T20:52:12.613 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:44:50 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b55c86c9c5bb1b9c-006477b37c-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:52:12 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:52:12.614 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj57 2023-05-31T20:52:12.614 DEBUG:boto:path=/pklxkd-59/obj57 2023-05-31T20:52:12.614 DEBUG:boto:auth_path=/pklxkd-59/obj57 2023-05-31T20:52:12.615 DEBUG:boto:Method: GET 2023-05-31T20:52:12.615 DEBUG:boto:Path: /pklxkd-59/obj57 2023-05-31T20:52:12.616 DEBUG:boto:Data: 2023-05-31T20:52:12.616 DEBUG:boto:Headers: {} 2023-05-31T20:52:12.616 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:52:12.617 DEBUG:boto:Port: 8001 2023-05-31T20:52:12.617 DEBUG:boto:Params: {} 2023-05-31T20:52:12.618 DEBUG:boto:Token: None 2023-05-31T20:52:12.618 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:52:12 GMT /pklxkd-59/obj57 2023-05-31T20:52:12.619 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:mvDyBbZwt3pKrHThQN4l18Vgs9k= 2023-05-31T20:52:12.619 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:52:12 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:mvDyBbZwt3pKrHThQN4l18Vgs9k=', 'Content-Length': '0'} 2023-05-31T20:52:12.621 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:44:50 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a7cbb67bbf46ab80-006477b37c-1c17-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:52:12 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:52:12.664 DEBUG:boto:path=/pklxkd-59/obj57 2023-05-31T20:52:12.665 DEBUG:boto:auth_path=/pklxkd-59/obj57 2023-05-31T20:52:12.665 DEBUG:boto:Method: GET 2023-05-31T20:52:12.665 DEBUG:boto:Path: /pklxkd-59/obj57 2023-05-31T20:52:12.666 DEBUG:boto:Data: 2023-05-31T20:52:12.666 DEBUG:boto:Headers: {} 2023-05-31T20:52:12.666 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:52:12.667 DEBUG:boto:Port: 8000 2023-05-31T20:52:12.667 DEBUG:boto:Params: {} 2023-05-31T20:52:12.668 DEBUG:boto:Token: None 2023-05-31T20:52:12.668 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:52:12 GMT /pklxkd-59/obj57 2023-05-31T20:52:12.668 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:mvDyBbZwt3pKrHThQN4l18Vgs9k= 2023-05-31T20:52:12.669 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:52:12 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:mvDyBbZwt3pKrHThQN4l18Vgs9k=', 'Content-Length': '0'} 2023-05-31T20:52:12.671 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:44:50 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000accc124ee05e7645-006477b37c-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:52:12 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:52:12.671 DEBUG:boto:path=/pklxkd-59/ 2023-05-31T20:52:12.672 DEBUG:boto:auth_path=/pklxkd-59/ 2023-05-31T20:52:12.672 DEBUG:boto:path=/pklxkd-59/?prefix=obj57 2023-05-31T20:52:12.672 DEBUG:boto:auth_path=/pklxkd-59/?prefix=obj57 2023-05-31T20:52:12.673 DEBUG:boto:Method: GET 2023-05-31T20:52:12.673 DEBUG:boto:Path: /pklxkd-59/?prefix=obj57 2023-05-31T20:52:12.674 DEBUG:boto:Data: 2023-05-31T20:52:12.674 DEBUG:boto:Headers: {} 2023-05-31T20:52:12.674 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:52:12.675 DEBUG:boto:Port: 8001 2023-05-31T20:52:12.675 DEBUG:boto:Params: {} 2023-05-31T20:52:12.676 DEBUG:boto:Token: None 2023-05-31T20:52:12.676 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:52:12 GMT /pklxkd-59/ 2023-05-31T20:52:12.677 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:pLqt3uLOcBEwAGLY0jYnxgHFOQo= 2023-05-31T20:52:12.677 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:52:12 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:pLqt3uLOcBEwAGLY0jYnxgHFOQo=', 'Content-Length': '0'} 2023-05-31T20:52:12.679 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000003f4553d0c3eef19c-006477b37c-1c17-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:52:12 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:52:12.680 DEBUG:boto:b'pklxkd-59obj571000falseobj572023-05-31T20:44:50.627Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:52:12.681 DEBUG:boto:path=/pklxkd-59/ 2023-05-31T20:52:12.681 DEBUG:boto:auth_path=/pklxkd-59/ 2023-05-31T20:52:12.682 DEBUG:boto:path=/pklxkd-59/?prefix=obj57 2023-05-31T20:52:12.682 DEBUG:boto:auth_path=/pklxkd-59/?prefix=obj57 2023-05-31T20:52:12.682 DEBUG:boto:Method: GET 2023-05-31T20:52:12.683 DEBUG:boto:Path: /pklxkd-59/?prefix=obj57 2023-05-31T20:52:12.683 DEBUG:boto:Data: 2023-05-31T20:52:12.683 DEBUG:boto:Headers: {} 2023-05-31T20:52:12.684 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:52:12.684 DEBUG:boto:Port: 8000 2023-05-31T20:52:12.685 DEBUG:boto:Params: {} 2023-05-31T20:52:12.685 DEBUG:boto:Token: None 2023-05-31T20:52:12.685 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:52:12 GMT /pklxkd-59/ 2023-05-31T20:52:12.686 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:pLqt3uLOcBEwAGLY0jYnxgHFOQo= 2023-05-31T20:52:12.686 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:52:12 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:pLqt3uLOcBEwAGLY0jYnxgHFOQo=', 'Content-Length': '0'} 2023-05-31T20:52:12.690 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a0bf4a6bda6337a5-006477b37c-1630-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:52:12 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:52:12.691 DEBUG:boto:b'pklxkd-59obj571000falseobj572023-05-31T20:44:50.627Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:52:12.691 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj58 2023-05-31T20:52:12.692 DEBUG:boto:path=/pklxkd-59/obj58 2023-05-31T20:52:12.692 DEBUG:boto:auth_path=/pklxkd-59/obj58 2023-05-31T20:52:12.692 DEBUG:boto:path=/pklxkd-59/obj58?versionId=null 2023-05-31T20:52:12.693 DEBUG:boto:auth_path=/pklxkd-59/obj58?versionId=null 2023-05-31T20:52:12.693 DEBUG:boto:Method: GET 2023-05-31T20:52:12.693 DEBUG:boto:Path: /pklxkd-59/obj58?versionId=null 2023-05-31T20:52:12.694 DEBUG:boto:Data: 2023-05-31T20:52:12.694 DEBUG:boto:Headers: {} 2023-05-31T20:52:12.694 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:52:12.695 DEBUG:boto:Port: 8001 2023-05-31T20:52:12.695 DEBUG:boto:Params: {} 2023-05-31T20:52:12.695 DEBUG:boto:Token: None 2023-05-31T20:52:12.696 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:52:12 GMT /pklxkd-59/obj58?versionId=null 2023-05-31T20:52:12.696 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:UyE3SV9E6Ha62tzWgzLeig1tFMo= 2023-05-31T20:52:12.697 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:52:12 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:UyE3SV9E6Ha62tzWgzLeig1tFMo=', 'Content-Length': '0'} 2023-05-31T20:52:12.698 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:48:12 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000044b7d435eebc46b-006477b37c-1c17-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:52:12 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:52:12.699 DEBUG:boto:path=/pklxkd-59/obj58 2023-05-31T20:52:12.699 DEBUG:boto:auth_path=/pklxkd-59/obj58 2023-05-31T20:52:12.700 DEBUG:boto:path=/pklxkd-59/obj58?versionId=null 2023-05-31T20:52:12.700 DEBUG:boto:auth_path=/pklxkd-59/obj58?versionId=null 2023-05-31T20:52:12.701 DEBUG:boto:Method: GET 2023-05-31T20:52:12.701 DEBUG:boto:Path: /pklxkd-59/obj58?versionId=null 2023-05-31T20:52:12.701 DEBUG:boto:Data: 2023-05-31T20:52:12.702 DEBUG:boto:Headers: {} 2023-05-31T20:52:12.702 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:52:12.702 DEBUG:boto:Port: 8000 2023-05-31T20:52:12.703 DEBUG:boto:Params: {} 2023-05-31T20:52:12.703 DEBUG:boto:Token: None 2023-05-31T20:52:12.704 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:52:12 GMT /pklxkd-59/obj58?versionId=null 2023-05-31T20:52:12.704 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:UyE3SV9E6Ha62tzWgzLeig1tFMo= 2023-05-31T20:52:12.705 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:52:12 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:UyE3SV9E6Ha62tzWgzLeig1tFMo=', 'Content-Length': '0'} 2023-05-31T20:52:12.706 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:48:12 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000059f24029f6dd6cff-006477b37c-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:52:12 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:52:12.752 DEBUG:boto:path=/pklxkd-59/obj58 2023-05-31T20:52:12.753 DEBUG:boto:auth_path=/pklxkd-59/obj58 2023-05-31T20:52:12.753 DEBUG:boto:path=/pklxkd-59/obj58?versionId=null 2023-05-31T20:52:12.753 DEBUG:boto:auth_path=/pklxkd-59/obj58?versionId=null 2023-05-31T20:52:12.754 DEBUG:boto:Method: HEAD 2023-05-31T20:52:12.754 DEBUG:boto:Path: /pklxkd-59/obj58?versionId=null 2023-05-31T20:52:12.754 DEBUG:boto:Data: 2023-05-31T20:52:12.755 DEBUG:boto:Headers: {} 2023-05-31T20:52:12.755 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:52:12.755 DEBUG:boto:Port: 8001 2023-05-31T20:52:12.756 DEBUG:boto:Params: {} 2023-05-31T20:52:12.756 DEBUG:boto:Token: None 2023-05-31T20:52:12.757 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:52:12 GMT /pklxkd-59/obj58?versionId=null 2023-05-31T20:52:12.757 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:gpbI7miASqVP+eeXHOSCd919V48= 2023-05-31T20:52:12.757 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:52:12 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:gpbI7miASqVP+eeXHOSCd919V48=', 'Content-Length': '0'} 2023-05-31T20:52:12.759 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:48:12 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000517259950ee67e35-006477b37c-1c17-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:52:12 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:52:12.760 DEBUG:boto:path=/pklxkd-59/obj58 2023-05-31T20:52:12.760 DEBUG:boto:auth_path=/pklxkd-59/obj58 2023-05-31T20:52:12.760 DEBUG:boto:path=/pklxkd-59/obj58?versionId=null 2023-05-31T20:52:12.761 DEBUG:boto:auth_path=/pklxkd-59/obj58?versionId=null 2023-05-31T20:52:12.761 DEBUG:boto:Method: HEAD 2023-05-31T20:52:12.761 DEBUG:boto:Path: /pklxkd-59/obj58?versionId=null 2023-05-31T20:52:12.762 DEBUG:boto:Data: 2023-05-31T20:52:12.762 DEBUG:boto:Headers: {} 2023-05-31T20:52:12.762 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:52:12.763 DEBUG:boto:Port: 8000 2023-05-31T20:52:12.763 DEBUG:boto:Params: {} 2023-05-31T20:52:12.763 DEBUG:boto:Token: None 2023-05-31T20:52:12.764 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:52:12 GMT /pklxkd-59/obj58?versionId=null 2023-05-31T20:52:12.764 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:gpbI7miASqVP+eeXHOSCd919V48= 2023-05-31T20:52:12.765 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:52:12 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:gpbI7miASqVP+eeXHOSCd919V48=', 'Content-Length': '0'} 2023-05-31T20:52:12.767 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:48:12 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000082e8a4b163887ffb-006477b37c-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:52:12 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:52:12.767 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj58 2023-05-31T20:52:12.768 DEBUG:boto:path=/pklxkd-59/obj58 2023-05-31T20:52:12.768 DEBUG:boto:auth_path=/pklxkd-59/obj58 2023-05-31T20:52:12.769 DEBUG:boto:path=/pklxkd-59/obj58?versionId=null 2023-05-31T20:52:12.769 DEBUG:boto:auth_path=/pklxkd-59/obj58?versionId=null 2023-05-31T20:52:12.769 DEBUG:boto:Method: GET 2023-05-31T20:52:12.770 DEBUG:boto:Path: /pklxkd-59/obj58?versionId=null 2023-05-31T20:52:12.770 DEBUG:boto:Data: 2023-05-31T20:52:12.771 DEBUG:boto:Headers: {} 2023-05-31T20:52:12.771 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:52:12.772 DEBUG:boto:Port: 8001 2023-05-31T20:52:12.772 DEBUG:boto:Params: {} 2023-05-31T20:52:12.773 DEBUG:boto:Token: None 2023-05-31T20:52:12.773 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:52:12 GMT /pklxkd-59/obj58?versionId=null 2023-05-31T20:52:12.774 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:UyE3SV9E6Ha62tzWgzLeig1tFMo= 2023-05-31T20:52:12.774 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:52:12 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:UyE3SV9E6Ha62tzWgzLeig1tFMo=', 'Content-Length': '0'} 2023-05-31T20:52:12.776 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:48:12 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000abd7ab2608eea870-006477b37c-1c17-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:52:12 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:52:12.777 DEBUG:boto:path=/pklxkd-59/obj58 2023-05-31T20:52:12.777 DEBUG:boto:auth_path=/pklxkd-59/obj58 2023-05-31T20:52:12.777 DEBUG:boto:path=/pklxkd-59/obj58?versionId=null 2023-05-31T20:52:12.778 DEBUG:boto:auth_path=/pklxkd-59/obj58?versionId=null 2023-05-31T20:52:12.778 DEBUG:boto:Method: GET 2023-05-31T20:52:12.778 DEBUG:boto:Path: /pklxkd-59/obj58?versionId=null 2023-05-31T20:52:12.779 DEBUG:boto:Data: 2023-05-31T20:52:12.779 DEBUG:boto:Headers: {} 2023-05-31T20:52:12.779 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:52:12.780 DEBUG:boto:Port: 8000 2023-05-31T20:52:12.780 DEBUG:boto:Params: {} 2023-05-31T20:52:12.781 DEBUG:boto:Token: None 2023-05-31T20:52:12.781 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:52:12 GMT /pklxkd-59/obj58?versionId=null 2023-05-31T20:52:12.781 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:UyE3SV9E6Ha62tzWgzLeig1tFMo= 2023-05-31T20:52:12.782 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:52:12 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:UyE3SV9E6Ha62tzWgzLeig1tFMo=', 'Content-Length': '0'} 2023-05-31T20:52:12.784 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:48:12 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001066dec361bac17c-006477b37c-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:52:12 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:52:12.784 DEBUG:boto:path=/pklxkd-59/ 2023-05-31T20:52:12.785 DEBUG:boto:auth_path=/pklxkd-59/ 2023-05-31T20:52:12.785 DEBUG:boto:path=/pklxkd-59/?prefix=obj58 2023-05-31T20:52:12.785 DEBUG:boto:auth_path=/pklxkd-59/?prefix=obj58 2023-05-31T20:52:12.786 DEBUG:boto:Method: GET 2023-05-31T20:52:12.786 DEBUG:boto:Path: /pklxkd-59/?prefix=obj58 2023-05-31T20:52:12.786 DEBUG:boto:Data: 2023-05-31T20:52:12.787 DEBUG:boto:Headers: {} 2023-05-31T20:52:12.787 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:52:12.787 DEBUG:boto:Port: 8001 2023-05-31T20:52:12.788 DEBUG:boto:Params: {} 2023-05-31T20:52:12.788 DEBUG:boto:Token: None 2023-05-31T20:52:12.789 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:52:12 GMT /pklxkd-59/ 2023-05-31T20:52:12.789 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:pLqt3uLOcBEwAGLY0jYnxgHFOQo= 2023-05-31T20:52:12.790 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:52:12 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:pLqt3uLOcBEwAGLY0jYnxgHFOQo=', 'Content-Length': '0'} 2023-05-31T20:52:12.792 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000dcd50edbdc345299-006477b37c-1c17-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:52:12 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:52:12.792 DEBUG:boto:b'pklxkd-59obj581000falseobj582023-05-31T20:48:12.976Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:52:12.793 DEBUG:boto:path=/pklxkd-59/ 2023-05-31T20:52:12.793 DEBUG:boto:auth_path=/pklxkd-59/ 2023-05-31T20:52:12.794 DEBUG:boto:path=/pklxkd-59/?prefix=obj58 2023-05-31T20:52:12.794 DEBUG:boto:auth_path=/pklxkd-59/?prefix=obj58 2023-05-31T20:52:12.795 DEBUG:boto:Method: GET 2023-05-31T20:52:12.795 DEBUG:boto:Path: /pklxkd-59/?prefix=obj58 2023-05-31T20:52:12.795 DEBUG:boto:Data: 2023-05-31T20:52:12.796 DEBUG:boto:Headers: {} 2023-05-31T20:52:12.796 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:52:12.797 DEBUG:boto:Port: 8000 2023-05-31T20:52:12.797 DEBUG:boto:Params: {} 2023-05-31T20:52:12.798 DEBUG:boto:Token: None 2023-05-31T20:52:12.798 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:52:12 GMT /pklxkd-59/ 2023-05-31T20:52:12.799 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:pLqt3uLOcBEwAGLY0jYnxgHFOQo= 2023-05-31T20:52:12.799 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:52:12 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:pLqt3uLOcBEwAGLY0jYnxgHFOQo=', 'Content-Length': '0'} 2023-05-31T20:52:12.803 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000025257d8a6c14d2ef-006477b37c-1630-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:52:12 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:52:12.803 DEBUG:boto:b'pklxkd-59obj581000falseobj582023-05-31T20:48:12.976Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:52:12.804 DEBUG:boto:path=/pklxkd-59/obj58 2023-05-31T20:52:12.804 DEBUG:boto:auth_path=/pklxkd-59/obj58 2023-05-31T20:52:12.805 DEBUG:boto:Method: HEAD 2023-05-31T20:52:12.805 DEBUG:boto:Path: /pklxkd-59/obj58 2023-05-31T20:52:12.805 DEBUG:boto:Data: 2023-05-31T20:52:12.806 DEBUG:boto:Headers: {} 2023-05-31T20:52:12.806 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:52:12.806 DEBUG:boto:Port: 8001 2023-05-31T20:52:12.807 DEBUG:boto:Params: {} 2023-05-31T20:52:12.807 DEBUG:boto:Token: None 2023-05-31T20:52:12.807 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:52:12 GMT /pklxkd-59/obj58 2023-05-31T20:52:12.808 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:ZcaFipAezxsNXlHTT+iaPznlEIc= 2023-05-31T20:52:12.808 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:52:12 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:ZcaFipAezxsNXlHTT+iaPznlEIc=', 'Content-Length': '0'} 2023-05-31T20:52:12.810 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:48:12 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000fa7353f1694f2ea-006477b37c-1c17-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:52:12 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:52:12.810 DEBUG:boto:path=/pklxkd-59/obj58 2023-05-31T20:52:12.811 DEBUG:boto:auth_path=/pklxkd-59/obj58 2023-05-31T20:52:12.811 DEBUG:boto:Method: HEAD 2023-05-31T20:52:12.812 DEBUG:boto:Path: /pklxkd-59/obj58 2023-05-31T20:52:12.812 DEBUG:boto:Data: 2023-05-31T20:52:12.812 DEBUG:boto:Headers: {} 2023-05-31T20:52:12.813 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:52:12.813 DEBUG:boto:Port: 8000 2023-05-31T20:52:12.813 DEBUG:boto:Params: {} 2023-05-31T20:52:12.814 DEBUG:boto:Token: None 2023-05-31T20:52:12.814 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:52:12 GMT /pklxkd-59/obj58 2023-05-31T20:52:12.814 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:ZcaFipAezxsNXlHTT+iaPznlEIc= 2023-05-31T20:52:12.815 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:52:12 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:ZcaFipAezxsNXlHTT+iaPznlEIc=', 'Content-Length': '0'} 2023-05-31T20:52:12.817 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:48:12 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c912c5ddbab0e943-006477b37c-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:52:12 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:52:12.817 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj58 2023-05-31T20:52:12.818 DEBUG:boto:path=/pklxkd-59/obj58 2023-05-31T20:52:12.818 DEBUG:boto:auth_path=/pklxkd-59/obj58 2023-05-31T20:52:12.818 DEBUG:boto:Method: GET 2023-05-31T20:52:12.819 DEBUG:boto:Path: /pklxkd-59/obj58 2023-05-31T20:52:12.819 DEBUG:boto:Data: 2023-05-31T20:52:12.820 DEBUG:boto:Headers: {} 2023-05-31T20:52:12.820 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:52:12.820 DEBUG:boto:Port: 8001 2023-05-31T20:52:12.821 DEBUG:boto:Params: {} 2023-05-31T20:52:12.821 DEBUG:boto:Token: None 2023-05-31T20:52:12.821 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:52:12 GMT /pklxkd-59/obj58 2023-05-31T20:52:12.822 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:EyA3vQNYTur4/xIoyAUrWDoh+4M= 2023-05-31T20:52:12.822 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:52:12 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:EyA3vQNYTur4/xIoyAUrWDoh+4M=', 'Content-Length': '0'} 2023-05-31T20:52:12.824 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:48:12 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000fc4ab68f290808e7-006477b37c-1c17-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:52:12 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:52:12.825 DEBUG:boto:path=/pklxkd-59/obj58 2023-05-31T20:52:12.825 DEBUG:boto:auth_path=/pklxkd-59/obj58 2023-05-31T20:52:12.826 DEBUG:boto:Method: GET 2023-05-31T20:52:12.826 DEBUG:boto:Path: /pklxkd-59/obj58 2023-05-31T20:52:12.827 DEBUG:boto:Data: 2023-05-31T20:52:12.827 DEBUG:boto:Headers: {} 2023-05-31T20:52:12.828 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:52:12.828 DEBUG:boto:Port: 8000 2023-05-31T20:52:12.829 DEBUG:boto:Params: {} 2023-05-31T20:52:12.829 DEBUG:boto:Token: None 2023-05-31T20:52:12.830 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:52:12 GMT /pklxkd-59/obj58 2023-05-31T20:52:12.830 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:EyA3vQNYTur4/xIoyAUrWDoh+4M= 2023-05-31T20:52:12.830 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:52:12 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:EyA3vQNYTur4/xIoyAUrWDoh+4M=', 'Content-Length': '0'} 2023-05-31T20:52:12.832 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:48:12 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000815188cf9a960aab-006477b37c-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:52:12 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:52:12.876 DEBUG:boto:path=/pklxkd-59/ 2023-05-31T20:52:12.877 DEBUG:boto:auth_path=/pklxkd-59/ 2023-05-31T20:52:12.877 DEBUG:boto:path=/pklxkd-59/?prefix=obj58 2023-05-31T20:52:12.878 DEBUG:boto:auth_path=/pklxkd-59/?prefix=obj58 2023-05-31T20:52:12.878 DEBUG:boto:Method: GET 2023-05-31T20:52:12.878 DEBUG:boto:Path: /pklxkd-59/?prefix=obj58 2023-05-31T20:52:12.879 DEBUG:boto:Data: 2023-05-31T20:52:12.879 DEBUG:boto:Headers: {} 2023-05-31T20:52:12.879 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:52:12.880 DEBUG:boto:Port: 8001 2023-05-31T20:52:12.880 DEBUG:boto:Params: {} 2023-05-31T20:52:12.880 DEBUG:boto:Token: None 2023-05-31T20:52:12.881 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:52:12 GMT /pklxkd-59/ 2023-05-31T20:52:12.881 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:pLqt3uLOcBEwAGLY0jYnxgHFOQo= 2023-05-31T20:52:12.882 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:52:12 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:pLqt3uLOcBEwAGLY0jYnxgHFOQo=', 'Content-Length': '0'} 2023-05-31T20:52:12.884 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f34f8d5a8a57f75e-006477b37c-1c17-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:52:12 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:52:12.885 DEBUG:boto:b'pklxkd-59obj581000falseobj582023-05-31T20:48:12.976Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:52:12.886 DEBUG:boto:path=/pklxkd-59/ 2023-05-31T20:52:12.886 DEBUG:boto:auth_path=/pklxkd-59/ 2023-05-31T20:52:12.887 DEBUG:boto:path=/pklxkd-59/?prefix=obj58 2023-05-31T20:52:12.887 DEBUG:boto:auth_path=/pklxkd-59/?prefix=obj58 2023-05-31T20:52:12.887 DEBUG:boto:Method: GET 2023-05-31T20:52:12.888 DEBUG:boto:Path: /pklxkd-59/?prefix=obj58 2023-05-31T20:52:12.888 DEBUG:boto:Data: 2023-05-31T20:52:12.889 DEBUG:boto:Headers: {} 2023-05-31T20:52:12.889 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:52:12.889 DEBUG:boto:Port: 8000 2023-05-31T20:52:12.890 DEBUG:boto:Params: {} 2023-05-31T20:52:12.890 DEBUG:boto:Token: None 2023-05-31T20:52:12.890 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:52:12 GMT /pklxkd-59/ 2023-05-31T20:52:12.891 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:pLqt3uLOcBEwAGLY0jYnxgHFOQo= 2023-05-31T20:52:12.891 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:52:12 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:pLqt3uLOcBEwAGLY0jYnxgHFOQo=', 'Content-Length': '0'} 2023-05-31T20:52:12.895 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000010d8cf2b50af7801-006477b37c-1630-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:52:12 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:52:12.895 DEBUG:boto:b'pklxkd-59obj581000falseobj582023-05-31T20:48:12.976Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:52:12.896 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj62 2023-05-31T20:52:12.896 DEBUG:boto:path=/pklxkd-59/obj62 2023-05-31T20:52:12.897 DEBUG:boto:auth_path=/pklxkd-59/obj62 2023-05-31T20:52:12.897 DEBUG:boto:path=/pklxkd-59/obj62?versionId=null 2023-05-31T20:52:12.897 DEBUG:boto:auth_path=/pklxkd-59/obj62?versionId=null 2023-05-31T20:52:12.898 DEBUG:boto:Method: GET 2023-05-31T20:52:12.898 DEBUG:boto:Path: /pklxkd-59/obj62?versionId=null 2023-05-31T20:52:12.898 DEBUG:boto:Data: 2023-05-31T20:52:12.899 DEBUG:boto:Headers: {} 2023-05-31T20:52:12.899 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:52:12.900 DEBUG:boto:Port: 8001 2023-05-31T20:52:12.900 DEBUG:boto:Params: {} 2023-05-31T20:52:12.900 DEBUG:boto:Token: None 2023-05-31T20:52:12.901 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:52:12 GMT /pklxkd-59/obj62?versionId=null 2023-05-31T20:52:12.901 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:GRLmil/Z4aq8rHr804xBZzzHJHI= 2023-05-31T20:52:12.902 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:52:12 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:GRLmil/Z4aq8rHr804xBZzzHJHI=', 'Content-Length': '0'} 2023-05-31T20:52:12.904 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:50:33 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000120918d95e58cf85-006477b37c-1c17-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:52:12 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:52:12.904 DEBUG:boto:path=/pklxkd-59/obj62 2023-05-31T20:52:12.905 DEBUG:boto:auth_path=/pklxkd-59/obj62 2023-05-31T20:52:12.905 DEBUG:boto:path=/pklxkd-59/obj62?versionId=null 2023-05-31T20:52:12.905 DEBUG:boto:auth_path=/pklxkd-59/obj62?versionId=null 2023-05-31T20:52:12.906 DEBUG:boto:Method: GET 2023-05-31T20:52:12.906 DEBUG:boto:Path: /pklxkd-59/obj62?versionId=null 2023-05-31T20:52:12.907 DEBUG:boto:Data: 2023-05-31T20:52:12.907 DEBUG:boto:Headers: {} 2023-05-31T20:52:12.907 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:52:12.908 DEBUG:boto:Port: 8000 2023-05-31T20:52:12.908 DEBUG:boto:Params: {} 2023-05-31T20:52:12.909 DEBUG:boto:Token: None 2023-05-31T20:52:12.909 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:52:12 GMT /pklxkd-59/obj62?versionId=null 2023-05-31T20:52:12.910 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:GRLmil/Z4aq8rHr804xBZzzHJHI= 2023-05-31T20:52:12.910 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:52:12 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:GRLmil/Z4aq8rHr804xBZzzHJHI=', 'Content-Length': '0'} 2023-05-31T20:52:12.912 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:50:33 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000075f7899fc4ff3ba-006477b37c-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:52:12 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:52:12.913 DEBUG:boto:path=/pklxkd-59/obj62 2023-05-31T20:52:12.913 DEBUG:boto:auth_path=/pklxkd-59/obj62 2023-05-31T20:52:12.914 DEBUG:boto:path=/pklxkd-59/obj62?versionId=null 2023-05-31T20:52:12.914 DEBUG:boto:auth_path=/pklxkd-59/obj62?versionId=null 2023-05-31T20:52:12.915 DEBUG:boto:Method: HEAD 2023-05-31T20:52:12.915 DEBUG:boto:Path: /pklxkd-59/obj62?versionId=null 2023-05-31T20:52:12.915 DEBUG:boto:Data: 2023-05-31T20:52:12.916 DEBUG:boto:Headers: {} 2023-05-31T20:52:12.916 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:52:12.916 DEBUG:boto:Port: 8001 2023-05-31T20:52:12.917 DEBUG:boto:Params: {} 2023-05-31T20:52:12.917 DEBUG:boto:Token: None 2023-05-31T20:52:12.918 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:52:12 GMT /pklxkd-59/obj62?versionId=null 2023-05-31T20:52:12.918 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:5W274SxuiO2uRAZbKwt6ng7pVvE= 2023-05-31T20:52:12.918 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:52:12 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:5W274SxuiO2uRAZbKwt6ng7pVvE=', 'Content-Length': '0'} 2023-05-31T20:52:12.920 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:50:33 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000fd4c25f72f75ffe-006477b37c-1c17-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:52:12 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:52:12.921 DEBUG:boto:path=/pklxkd-59/obj62 2023-05-31T20:52:12.921 DEBUG:boto:auth_path=/pklxkd-59/obj62 2023-05-31T20:52:12.921 DEBUG:boto:path=/pklxkd-59/obj62?versionId=null 2023-05-31T20:52:12.922 DEBUG:boto:auth_path=/pklxkd-59/obj62?versionId=null 2023-05-31T20:52:12.922 DEBUG:boto:Method: HEAD 2023-05-31T20:52:12.922 DEBUG:boto:Path: /pklxkd-59/obj62?versionId=null 2023-05-31T20:52:12.923 DEBUG:boto:Data: 2023-05-31T20:52:12.923 DEBUG:boto:Headers: {} 2023-05-31T20:52:12.923 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:52:12.924 DEBUG:boto:Port: 8000 2023-05-31T20:52:12.924 DEBUG:boto:Params: {} 2023-05-31T20:52:12.924 DEBUG:boto:Token: None 2023-05-31T20:52:12.925 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:52:12 GMT /pklxkd-59/obj62?versionId=null 2023-05-31T20:52:12.925 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:5W274SxuiO2uRAZbKwt6ng7pVvE= 2023-05-31T20:52:12.925 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:52:12 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:5W274SxuiO2uRAZbKwt6ng7pVvE=', 'Content-Length': '0'} 2023-05-31T20:52:12.927 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:50:33 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007d3556b07cc92931-006477b37c-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:52:12 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:52:12.928 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj62 2023-05-31T20:52:12.928 DEBUG:boto:path=/pklxkd-59/obj62 2023-05-31T20:52:12.929 DEBUG:boto:auth_path=/pklxkd-59/obj62 2023-05-31T20:52:12.929 DEBUG:boto:path=/pklxkd-59/obj62?versionId=null 2023-05-31T20:52:12.929 DEBUG:boto:auth_path=/pklxkd-59/obj62?versionId=null 2023-05-31T20:52:12.930 DEBUG:boto:Method: GET 2023-05-31T20:52:12.930 DEBUG:boto:Path: /pklxkd-59/obj62?versionId=null 2023-05-31T20:52:12.931 DEBUG:boto:Data: 2023-05-31T20:52:12.931 DEBUG:boto:Headers: {} 2023-05-31T20:52:12.931 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:52:12.932 DEBUG:boto:Port: 8001 2023-05-31T20:52:12.932 DEBUG:boto:Params: {} 2023-05-31T20:52:12.933 DEBUG:boto:Token: None 2023-05-31T20:52:12.933 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:52:12 GMT /pklxkd-59/obj62?versionId=null 2023-05-31T20:52:12.934 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:GRLmil/Z4aq8rHr804xBZzzHJHI= 2023-05-31T20:52:12.934 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:52:12 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:GRLmil/Z4aq8rHr804xBZzzHJHI=', 'Content-Length': '0'} 2023-05-31T20:52:12.936 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:50:33 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000844b2083af5d03c0-006477b37c-1c17-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:52:12 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:52:12.937 DEBUG:boto:path=/pklxkd-59/obj62 2023-05-31T20:52:12.937 DEBUG:boto:auth_path=/pklxkd-59/obj62 2023-05-31T20:52:12.938 DEBUG:boto:path=/pklxkd-59/obj62?versionId=null 2023-05-31T20:52:12.938 DEBUG:boto:auth_path=/pklxkd-59/obj62?versionId=null 2023-05-31T20:52:12.938 DEBUG:boto:Method: GET 2023-05-31T20:52:12.939 DEBUG:boto:Path: /pklxkd-59/obj62?versionId=null 2023-05-31T20:52:12.939 DEBUG:boto:Data: 2023-05-31T20:52:12.939 DEBUG:boto:Headers: {} 2023-05-31T20:52:12.940 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:52:12.940 DEBUG:boto:Port: 8000 2023-05-31T20:52:12.940 DEBUG:boto:Params: {} 2023-05-31T20:52:12.941 DEBUG:boto:Token: None 2023-05-31T20:52:12.941 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:52:12 GMT /pklxkd-59/obj62?versionId=null 2023-05-31T20:52:12.942 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:GRLmil/Z4aq8rHr804xBZzzHJHI= 2023-05-31T20:52:12.942 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:52:12 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:GRLmil/Z4aq8rHr804xBZzzHJHI=', 'Content-Length': '0'} 2023-05-31T20:52:12.944 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:50:33 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001190f9a743a93496-006477b37c-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:52:12 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:52:12.992 DEBUG:boto:path=/pklxkd-59/ 2023-05-31T20:52:12.993 DEBUG:boto:auth_path=/pklxkd-59/ 2023-05-31T20:52:12.993 DEBUG:boto:path=/pklxkd-59/?prefix=obj62 2023-05-31T20:52:12.994 DEBUG:boto:auth_path=/pklxkd-59/?prefix=obj62 2023-05-31T20:52:12.994 DEBUG:boto:Method: GET 2023-05-31T20:52:12.994 DEBUG:boto:Path: /pklxkd-59/?prefix=obj62 2023-05-31T20:52:12.995 DEBUG:boto:Data: 2023-05-31T20:52:12.995 DEBUG:boto:Headers: {} 2023-05-31T20:52:12.996 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:52:12.996 DEBUG:boto:Port: 8001 2023-05-31T20:52:12.996 DEBUG:boto:Params: {} 2023-05-31T20:52:12.997 DEBUG:boto:Token: None 2023-05-31T20:52:12.997 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:52:12 GMT /pklxkd-59/ 2023-05-31T20:52:12.998 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:pLqt3uLOcBEwAGLY0jYnxgHFOQo= 2023-05-31T20:52:12.998 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:52:12 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:pLqt3uLOcBEwAGLY0jYnxgHFOQo=', 'Content-Length': '0'} 2023-05-31T20:52:13.000 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000008ae30ed6563c6782-006477b37c-1c17-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:52:12 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:52:13.001 DEBUG:boto:b'pklxkd-59obj621000falseobj622023-05-31T20:50:33.515Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:52:13.002 DEBUG:boto:path=/pklxkd-59/ 2023-05-31T20:52:13.002 DEBUG:boto:auth_path=/pklxkd-59/ 2023-05-31T20:52:13.003 DEBUG:boto:path=/pklxkd-59/?prefix=obj62 2023-05-31T20:52:13.003 DEBUG:boto:auth_path=/pklxkd-59/?prefix=obj62 2023-05-31T20:52:13.004 DEBUG:boto:Method: GET 2023-05-31T20:52:13.004 DEBUG:boto:Path: /pklxkd-59/?prefix=obj62 2023-05-31T20:52:13.004 DEBUG:boto:Data: 2023-05-31T20:52:13.005 DEBUG:boto:Headers: {} 2023-05-31T20:52:13.005 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:52:13.005 DEBUG:boto:Port: 8000 2023-05-31T20:52:13.006 DEBUG:boto:Params: {} 2023-05-31T20:52:13.006 DEBUG:boto:Token: None 2023-05-31T20:52:13.006 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:52:13 GMT /pklxkd-59/ 2023-05-31T20:52:13.007 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:2oppaLXJix1iBBaZf3oYtXHJCGM= 2023-05-31T20:52:13.007 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:52:13 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:2oppaLXJix1iBBaZf3oYtXHJCGM=', 'Content-Length': '0'} 2023-05-31T20:52:13.011 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000724f077dfbaf61ea-006477b37d-1630-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:52:13 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:52:13.011 DEBUG:boto:b'pklxkd-59obj621000falseobj622023-05-31T20:50:33.515Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:52:13.012 DEBUG:boto:path=/pklxkd-59/obj62 2023-05-31T20:52:13.012 DEBUG:boto:auth_path=/pklxkd-59/obj62 2023-05-31T20:52:13.013 DEBUG:boto:Method: HEAD 2023-05-31T20:52:13.013 DEBUG:boto:Path: /pklxkd-59/obj62 2023-05-31T20:52:13.013 DEBUG:boto:Data: 2023-05-31T20:52:13.014 DEBUG:boto:Headers: {} 2023-05-31T20:52:13.014 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:52:13.014 DEBUG:boto:Port: 8001 2023-05-31T20:52:13.015 DEBUG:boto:Params: {} 2023-05-31T20:52:13.015 DEBUG:boto:Token: None 2023-05-31T20:52:13.015 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:52:13 GMT /pklxkd-59/obj62 2023-05-31T20:52:13.016 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:3hIeL9qvF5PPLLwg3Bj+/qIziTw= 2023-05-31T20:52:13.016 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:52:13 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:3hIeL9qvF5PPLLwg3Bj+/qIziTw=', 'Content-Length': '0'} 2023-05-31T20:52:13.018 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:50:33 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000044ce518ff76190cc-006477b37d-1c17-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:52:13 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:52:13.018 DEBUG:boto:path=/pklxkd-59/obj62 2023-05-31T20:52:13.019 DEBUG:boto:auth_path=/pklxkd-59/obj62 2023-05-31T20:52:13.019 DEBUG:boto:Method: HEAD 2023-05-31T20:52:13.019 DEBUG:boto:Path: /pklxkd-59/obj62 2023-05-31T20:52:13.020 DEBUG:boto:Data: 2023-05-31T20:52:13.020 DEBUG:boto:Headers: {} 2023-05-31T20:52:13.021 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:52:13.021 DEBUG:boto:Port: 8000 2023-05-31T20:52:13.021 DEBUG:boto:Params: {} 2023-05-31T20:52:13.022 DEBUG:boto:Token: None 2023-05-31T20:52:13.022 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:52:13 GMT /pklxkd-59/obj62 2023-05-31T20:52:13.023 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:3hIeL9qvF5PPLLwg3Bj+/qIziTw= 2023-05-31T20:52:13.023 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:52:13 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:3hIeL9qvF5PPLLwg3Bj+/qIziTw=', 'Content-Length': '0'} 2023-05-31T20:52:13.025 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:50:33 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005f0648d12f15a920-006477b37d-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:52:13 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:52:13.025 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj62 2023-05-31T20:52:13.026 DEBUG:boto:path=/pklxkd-59/obj62 2023-05-31T20:52:13.026 DEBUG:boto:auth_path=/pklxkd-59/obj62 2023-05-31T20:52:13.027 DEBUG:boto:Method: GET 2023-05-31T20:52:13.027 DEBUG:boto:Path: /pklxkd-59/obj62 2023-05-31T20:52:13.028 DEBUG:boto:Data: 2023-05-31T20:52:13.028 DEBUG:boto:Headers: {} 2023-05-31T20:52:13.028 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:52:13.029 DEBUG:boto:Port: 8001 2023-05-31T20:52:13.029 DEBUG:boto:Params: {} 2023-05-31T20:52:13.030 DEBUG:boto:Token: None 2023-05-31T20:52:13.030 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:52:13 GMT /pklxkd-59/obj62 2023-05-31T20:52:13.031 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:BynUO0zacvyT6l0RJ/bEe8xOnOU= 2023-05-31T20:52:13.031 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:52:13 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:BynUO0zacvyT6l0RJ/bEe8xOnOU=', 'Content-Length': '0'} 2023-05-31T20:52:13.033 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:50:33 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000069afb2f4755e8cd3-006477b37d-1c17-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:52:13 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:52:13.080 DEBUG:boto:path=/pklxkd-59/obj62 2023-05-31T20:52:13.081 DEBUG:boto:auth_path=/pklxkd-59/obj62 2023-05-31T20:52:13.081 DEBUG:boto:Method: GET 2023-05-31T20:52:13.082 DEBUG:boto:Path: /pklxkd-59/obj62 2023-05-31T20:52:13.082 DEBUG:boto:Data: 2023-05-31T20:52:13.082 DEBUG:boto:Headers: {} 2023-05-31T20:52:13.083 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:52:13.083 DEBUG:boto:Port: 8000 2023-05-31T20:52:13.084 DEBUG:boto:Params: {} 2023-05-31T20:52:13.084 DEBUG:boto:Token: None 2023-05-31T20:52:13.085 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:52:13 GMT /pklxkd-59/obj62 2023-05-31T20:52:13.085 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:BynUO0zacvyT6l0RJ/bEe8xOnOU= 2023-05-31T20:52:13.086 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:52:13 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:BynUO0zacvyT6l0RJ/bEe8xOnOU=', 'Content-Length': '0'} 2023-05-31T20:52:13.087 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:50:33 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000684a18f110303be1-006477b37d-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:52:13 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:52:13.088 DEBUG:boto:path=/pklxkd-59/ 2023-05-31T20:52:13.089 DEBUG:boto:auth_path=/pklxkd-59/ 2023-05-31T20:52:13.089 DEBUG:boto:path=/pklxkd-59/?prefix=obj62 2023-05-31T20:52:13.089 DEBUG:boto:auth_path=/pklxkd-59/?prefix=obj62 2023-05-31T20:52:13.090 DEBUG:boto:Method: GET 2023-05-31T20:52:13.090 DEBUG:boto:Path: /pklxkd-59/?prefix=obj62 2023-05-31T20:52:13.090 DEBUG:boto:Data: 2023-05-31T20:52:13.091 DEBUG:boto:Headers: {} 2023-05-31T20:52:13.091 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:52:13.092 DEBUG:boto:Port: 8001 2023-05-31T20:52:13.092 DEBUG:boto:Params: {} 2023-05-31T20:52:13.092 DEBUG:boto:Token: None 2023-05-31T20:52:13.093 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:52:13 GMT /pklxkd-59/ 2023-05-31T20:52:13.093 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:2oppaLXJix1iBBaZf3oYtXHJCGM= 2023-05-31T20:52:13.094 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:52:13 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:2oppaLXJix1iBBaZf3oYtXHJCGM=', 'Content-Length': '0'} 2023-05-31T20:52:13.096 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000efb01603e42e3126-006477b37d-1c17-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:52:13 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:52:13.097 DEBUG:boto:b'pklxkd-59obj621000falseobj622023-05-31T20:50:33.515Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:52:13.097 DEBUG:boto:path=/pklxkd-59/ 2023-05-31T20:52:13.098 DEBUG:boto:auth_path=/pklxkd-59/ 2023-05-31T20:52:13.098 DEBUG:boto:path=/pklxkd-59/?prefix=obj62 2023-05-31T20:52:13.099 DEBUG:boto:auth_path=/pklxkd-59/?prefix=obj62 2023-05-31T20:52:13.099 DEBUG:boto:Method: GET 2023-05-31T20:52:13.099 DEBUG:boto:Path: /pklxkd-59/?prefix=obj62 2023-05-31T20:52:13.100 DEBUG:boto:Data: 2023-05-31T20:52:13.100 DEBUG:boto:Headers: {} 2023-05-31T20:52:13.100 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:52:13.101 DEBUG:boto:Port: 8000 2023-05-31T20:52:13.101 DEBUG:boto:Params: {} 2023-05-31T20:52:13.101 DEBUG:boto:Token: None 2023-05-31T20:52:13.102 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:52:13 GMT /pklxkd-59/ 2023-05-31T20:52:13.102 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:2oppaLXJix1iBBaZf3oYtXHJCGM= 2023-05-31T20:52:13.103 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:52:13 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:2oppaLXJix1iBBaZf3oYtXHJCGM=', 'Content-Length': '0'} 2023-05-31T20:52:13.106 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000088c8b443be44fddb-006477b37d-1630-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:52:13 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:52:13.107 DEBUG:boto:b'pklxkd-59obj621000falseobj622023-05-31T20:50:33.515Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:52:13.107 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj68 2023-05-31T20:52:13.108 DEBUG:boto:path=/pklxkd-59/obj68 2023-05-31T20:52:13.108 DEBUG:boto:auth_path=/pklxkd-59/obj68 2023-05-31T20:52:13.108 DEBUG:boto:path=/pklxkd-59/obj68?versionId=null 2023-05-31T20:52:13.109 DEBUG:boto:auth_path=/pklxkd-59/obj68?versionId=null 2023-05-31T20:52:13.109 DEBUG:boto:Method: GET 2023-05-31T20:52:13.110 DEBUG:boto:Path: /pklxkd-59/obj68?versionId=null 2023-05-31T20:52:13.110 DEBUG:boto:Data: 2023-05-31T20:52:13.110 DEBUG:boto:Headers: {} 2023-05-31T20:52:13.111 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:52:13.111 DEBUG:boto:Port: 8001 2023-05-31T20:52:13.111 DEBUG:boto:Params: {} 2023-05-31T20:52:13.112 DEBUG:boto:Token: None 2023-05-31T20:52:13.112 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:52:13 GMT /pklxkd-59/obj68?versionId=null 2023-05-31T20:52:13.113 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:2m1aHHHqPvZQxumYEJQVYdWI2wI= 2023-05-31T20:52:13.113 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:52:13 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:2m1aHHHqPvZQxumYEJQVYdWI2wI=', 'Content-Length': '0'} 2023-05-31T20:52:13.115 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:43:08 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e14907bd7e15ee7f-006477b37d-1c17-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:52:13 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:52:13.116 DEBUG:boto:path=/pklxkd-59/obj68 2023-05-31T20:52:13.117 DEBUG:boto:auth_path=/pklxkd-59/obj68 2023-05-31T20:52:13.117 DEBUG:boto:path=/pklxkd-59/obj68?versionId=null 2023-05-31T20:52:13.117 DEBUG:boto:auth_path=/pklxkd-59/obj68?versionId=null 2023-05-31T20:52:13.118 DEBUG:boto:Method: GET 2023-05-31T20:52:13.118 DEBUG:boto:Path: /pklxkd-59/obj68?versionId=null 2023-05-31T20:52:13.119 DEBUG:boto:Data: 2023-05-31T20:52:13.119 DEBUG:boto:Headers: {} 2023-05-31T20:52:13.120 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:52:13.120 DEBUG:boto:Port: 8000 2023-05-31T20:52:13.121 DEBUG:boto:Params: {} 2023-05-31T20:52:13.121 DEBUG:boto:Token: None 2023-05-31T20:52:13.122 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:52:13 GMT /pklxkd-59/obj68?versionId=null 2023-05-31T20:52:13.122 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:2m1aHHHqPvZQxumYEJQVYdWI2wI= 2023-05-31T20:52:13.123 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:52:13 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:2m1aHHHqPvZQxumYEJQVYdWI2wI=', 'Content-Length': '0'} 2023-05-31T20:52:13.125 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:43:08 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000748fa97cb543b4f0-006477b37d-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:52:13 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:52:13.172 DEBUG:boto:path=/pklxkd-59/obj68 2023-05-31T20:52:13.173 DEBUG:boto:auth_path=/pklxkd-59/obj68 2023-05-31T20:52:13.173 DEBUG:boto:path=/pklxkd-59/obj68?versionId=null 2023-05-31T20:52:13.173 DEBUG:boto:auth_path=/pklxkd-59/obj68?versionId=null 2023-05-31T20:52:13.174 DEBUG:boto:Method: HEAD 2023-05-31T20:52:13.174 DEBUG:boto:Path: /pklxkd-59/obj68?versionId=null 2023-05-31T20:52:13.174 DEBUG:boto:Data: 2023-05-31T20:52:13.175 DEBUG:boto:Headers: {} 2023-05-31T20:52:13.175 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:52:13.175 DEBUG:boto:Port: 8001 2023-05-31T20:52:13.176 DEBUG:boto:Params: {} 2023-05-31T20:52:13.176 DEBUG:boto:Token: None 2023-05-31T20:52:13.177 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:52:13 GMT /pklxkd-59/obj68?versionId=null 2023-05-31T20:52:13.177 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:4DQYAl6JFnQJ/RzPAYcMJdOzGJE= 2023-05-31T20:52:13.177 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:52:13 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:4DQYAl6JFnQJ/RzPAYcMJdOzGJE=', 'Content-Length': '0'} 2023-05-31T20:52:13.179 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:43:08 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b68ecbc3d78efb1a-006477b37d-1c17-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:52:13 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:52:13.180 DEBUG:boto:path=/pklxkd-59/obj68 2023-05-31T20:52:13.180 DEBUG:boto:auth_path=/pklxkd-59/obj68 2023-05-31T20:52:13.180 DEBUG:boto:path=/pklxkd-59/obj68?versionId=null 2023-05-31T20:52:13.181 DEBUG:boto:auth_path=/pklxkd-59/obj68?versionId=null 2023-05-31T20:52:13.181 DEBUG:boto:Method: HEAD 2023-05-31T20:52:13.181 DEBUG:boto:Path: /pklxkd-59/obj68?versionId=null 2023-05-31T20:52:13.182 DEBUG:boto:Data: 2023-05-31T20:52:13.182 DEBUG:boto:Headers: {} 2023-05-31T20:52:13.182 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:52:13.183 DEBUG:boto:Port: 8000 2023-05-31T20:52:13.183 DEBUG:boto:Params: {} 2023-05-31T20:52:13.183 DEBUG:boto:Token: None 2023-05-31T20:52:13.184 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:52:13 GMT /pklxkd-59/obj68?versionId=null 2023-05-31T20:52:13.184 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:4DQYAl6JFnQJ/RzPAYcMJdOzGJE= 2023-05-31T20:52:13.184 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:52:13 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:4DQYAl6JFnQJ/RzPAYcMJdOzGJE=', 'Content-Length': '0'} 2023-05-31T20:52:13.186 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:43:08 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008668c9f4f9a5153b-006477b37d-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:52:13 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:52:13.187 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj68 2023-05-31T20:52:13.187 DEBUG:boto:path=/pklxkd-59/obj68 2023-05-31T20:52:13.188 DEBUG:boto:auth_path=/pklxkd-59/obj68 2023-05-31T20:52:13.188 DEBUG:boto:path=/pklxkd-59/obj68?versionId=null 2023-05-31T20:52:13.188 DEBUG:boto:auth_path=/pklxkd-59/obj68?versionId=null 2023-05-31T20:52:13.189 DEBUG:boto:Method: GET 2023-05-31T20:52:13.189 DEBUG:boto:Path: /pklxkd-59/obj68?versionId=null 2023-05-31T20:52:13.190 DEBUG:boto:Data: 2023-05-31T20:52:13.190 DEBUG:boto:Headers: {} 2023-05-31T20:52:13.190 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:52:13.191 DEBUG:boto:Port: 8001 2023-05-31T20:52:13.191 DEBUG:boto:Params: {} 2023-05-31T20:52:13.192 DEBUG:boto:Token: None 2023-05-31T20:52:13.192 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:52:13 GMT /pklxkd-59/obj68?versionId=null 2023-05-31T20:52:13.193 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:2m1aHHHqPvZQxumYEJQVYdWI2wI= 2023-05-31T20:52:13.193 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:52:13 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:2m1aHHHqPvZQxumYEJQVYdWI2wI=', 'Content-Length': '0'} 2023-05-31T20:52:13.195 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:43:08 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f1d9a2b74730b9c1-006477b37d-1c17-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:52:13 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:52:13.196 DEBUG:boto:path=/pklxkd-59/obj68 2023-05-31T20:52:13.196 DEBUG:boto:auth_path=/pklxkd-59/obj68 2023-05-31T20:52:13.197 DEBUG:boto:path=/pklxkd-59/obj68?versionId=null 2023-05-31T20:52:13.197 DEBUG:boto:auth_path=/pklxkd-59/obj68?versionId=null 2023-05-31T20:52:13.198 DEBUG:boto:Method: GET 2023-05-31T20:52:13.198 DEBUG:boto:Path: /pklxkd-59/obj68?versionId=null 2023-05-31T20:52:13.199 DEBUG:boto:Data: 2023-05-31T20:52:13.199 DEBUG:boto:Headers: {} 2023-05-31T20:52:13.199 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:52:13.200 DEBUG:boto:Port: 8000 2023-05-31T20:52:13.200 DEBUG:boto:Params: {} 2023-05-31T20:52:13.201 DEBUG:boto:Token: None 2023-05-31T20:52:13.201 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:52:13 GMT /pklxkd-59/obj68?versionId=null 2023-05-31T20:52:13.202 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:2m1aHHHqPvZQxumYEJQVYdWI2wI= 2023-05-31T20:52:13.202 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:52:13 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:2m1aHHHqPvZQxumYEJQVYdWI2wI=', 'Content-Length': '0'} 2023-05-31T20:52:13.204 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:43:08 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000eac4e5561e490fce-006477b37d-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:52:13 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:52:13.204 DEBUG:boto:path=/pklxkd-59/ 2023-05-31T20:52:13.205 DEBUG:boto:auth_path=/pklxkd-59/ 2023-05-31T20:52:13.205 DEBUG:boto:path=/pklxkd-59/?prefix=obj68 2023-05-31T20:52:13.206 DEBUG:boto:auth_path=/pklxkd-59/?prefix=obj68 2023-05-31T20:52:13.206 DEBUG:boto:Method: GET 2023-05-31T20:52:13.206 DEBUG:boto:Path: /pklxkd-59/?prefix=obj68 2023-05-31T20:52:13.207 DEBUG:boto:Data: 2023-05-31T20:52:13.207 DEBUG:boto:Headers: {} 2023-05-31T20:52:13.207 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:52:13.208 DEBUG:boto:Port: 8001 2023-05-31T20:52:13.208 DEBUG:boto:Params: {} 2023-05-31T20:52:13.209 DEBUG:boto:Token: None 2023-05-31T20:52:13.209 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:52:13 GMT /pklxkd-59/ 2023-05-31T20:52:13.209 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:2oppaLXJix1iBBaZf3oYtXHJCGM= 2023-05-31T20:52:13.210 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:52:13 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:2oppaLXJix1iBBaZf3oYtXHJCGM=', 'Content-Length': '0'} 2023-05-31T20:52:13.212 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000103736c9f0ff86ac-006477b37d-1c17-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:52:13 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:52:13.213 DEBUG:boto:b'pklxkd-59obj681000falseobj682023-05-31T20:43:08.066Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:52:13.213 DEBUG:boto:path=/pklxkd-59/ 2023-05-31T20:52:13.214 DEBUG:boto:auth_path=/pklxkd-59/ 2023-05-31T20:52:13.214 DEBUG:boto:path=/pklxkd-59/?prefix=obj68 2023-05-31T20:52:13.214 DEBUG:boto:auth_path=/pklxkd-59/?prefix=obj68 2023-05-31T20:52:13.215 DEBUG:boto:Method: GET 2023-05-31T20:52:13.215 DEBUG:boto:Path: /pklxkd-59/?prefix=obj68 2023-05-31T20:52:13.215 DEBUG:boto:Data: 2023-05-31T20:52:13.216 DEBUG:boto:Headers: {} 2023-05-31T20:52:13.216 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:52:13.217 DEBUG:boto:Port: 8000 2023-05-31T20:52:13.217 DEBUG:boto:Params: {} 2023-05-31T20:52:13.217 DEBUG:boto:Token: None 2023-05-31T20:52:13.218 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:52:13 GMT /pklxkd-59/ 2023-05-31T20:52:13.218 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:2oppaLXJix1iBBaZf3oYtXHJCGM= 2023-05-31T20:52:13.219 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:52:13 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:2oppaLXJix1iBBaZf3oYtXHJCGM=', 'Content-Length': '0'} 2023-05-31T20:52:13.222 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000046159bf3d10dfd5e-006477b37d-1630-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:52:13 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:52:13.223 DEBUG:boto:b'pklxkd-59obj681000falseobj682023-05-31T20:43:08.066Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:52:13.223 DEBUG:boto:path=/pklxkd-59/obj68 2023-05-31T20:52:13.224 DEBUG:boto:auth_path=/pklxkd-59/obj68 2023-05-31T20:52:13.224 DEBUG:boto:Method: HEAD 2023-05-31T20:52:13.225 DEBUG:boto:Path: /pklxkd-59/obj68 2023-05-31T20:52:13.225 DEBUG:boto:Data: 2023-05-31T20:52:13.225 DEBUG:boto:Headers: {} 2023-05-31T20:52:13.226 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:52:13.226 DEBUG:boto:Port: 8001 2023-05-31T20:52:13.226 DEBUG:boto:Params: {} 2023-05-31T20:52:13.227 DEBUG:boto:Token: None 2023-05-31T20:52:13.227 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:52:13 GMT /pklxkd-59/obj68 2023-05-31T20:52:13.228 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:eODOCh+DcFFir9uZVCymHBk0AMg= 2023-05-31T20:52:13.228 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:52:13 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:eODOCh+DcFFir9uZVCymHBk0AMg=', 'Content-Length': '0'} 2023-05-31T20:52:13.230 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:43:08 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009eeac39b91f7cda5-006477b37d-1c17-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:52:13 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:52:13.231 DEBUG:boto:path=/pklxkd-59/obj68 2023-05-31T20:52:13.231 DEBUG:boto:auth_path=/pklxkd-59/obj68 2023-05-31T20:52:13.232 DEBUG:boto:Method: HEAD 2023-05-31T20:52:13.232 DEBUG:boto:Path: /pklxkd-59/obj68 2023-05-31T20:52:13.233 DEBUG:boto:Data: 2023-05-31T20:52:13.233 DEBUG:boto:Headers: {} 2023-05-31T20:52:13.233 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:52:13.234 DEBUG:boto:Port: 8000 2023-05-31T20:52:13.234 DEBUG:boto:Params: {} 2023-05-31T20:52:13.235 DEBUG:boto:Token: None 2023-05-31T20:52:13.235 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:52:13 GMT /pklxkd-59/obj68 2023-05-31T20:52:13.236 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:eODOCh+DcFFir9uZVCymHBk0AMg= 2023-05-31T20:52:13.236 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:52:13 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:eODOCh+DcFFir9uZVCymHBk0AMg=', 'Content-Length': '0'} 2023-05-31T20:52:13.238 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:43:08 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009696f81fc89e7f42-006477b37d-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:52:13 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:52:13.239 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj68 2023-05-31T20:52:13.239 DEBUG:boto:path=/pklxkd-59/obj68 2023-05-31T20:52:13.240 DEBUG:boto:auth_path=/pklxkd-59/obj68 2023-05-31T20:52:13.240 DEBUG:boto:Method: GET 2023-05-31T20:52:13.241 DEBUG:boto:Path: /pklxkd-59/obj68 2023-05-31T20:52:13.241 DEBUG:boto:Data: 2023-05-31T20:52:13.242 DEBUG:boto:Headers: {} 2023-05-31T20:52:13.242 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:52:13.242 DEBUG:boto:Port: 8001 2023-05-31T20:52:13.243 DEBUG:boto:Params: {} 2023-05-31T20:52:13.243 DEBUG:boto:Token: None 2023-05-31T20:52:13.244 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:52:13 GMT /pklxkd-59/obj68 2023-05-31T20:52:13.244 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:hX390JLYhmQSvy36967tQWXpKUQ= 2023-05-31T20:52:13.245 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:52:13 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:hX390JLYhmQSvy36967tQWXpKUQ=', 'Content-Length': '0'} 2023-05-31T20:52:13.247 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:43:08 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e1356b83ef7917a6-006477b37d-1c17-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:52:13 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:52:13.288 DEBUG:boto:path=/pklxkd-59/obj68 2023-05-31T20:52:13.289 DEBUG:boto:auth_path=/pklxkd-59/obj68 2023-05-31T20:52:13.289 DEBUG:boto:Method: GET 2023-05-31T20:52:13.289 DEBUG:boto:Path: /pklxkd-59/obj68 2023-05-31T20:52:13.290 DEBUG:boto:Data: 2023-05-31T20:52:13.290 DEBUG:boto:Headers: {} 2023-05-31T20:52:13.290 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:52:13.291 DEBUG:boto:Port: 8000 2023-05-31T20:52:13.291 DEBUG:boto:Params: {} 2023-05-31T20:52:13.292 DEBUG:boto:Token: None 2023-05-31T20:52:13.292 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:52:13 GMT /pklxkd-59/obj68 2023-05-31T20:52:13.293 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:hX390JLYhmQSvy36967tQWXpKUQ= 2023-05-31T20:52:13.293 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:52:13 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:hX390JLYhmQSvy36967tQWXpKUQ=', 'Content-Length': '0'} 2023-05-31T20:52:13.295 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:43:08 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000eb974d978674b815-006477b37d-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:52:13 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:52:13.296 DEBUG:boto:path=/pklxkd-59/ 2023-05-31T20:52:13.296 DEBUG:boto:auth_path=/pklxkd-59/ 2023-05-31T20:52:13.297 DEBUG:boto:path=/pklxkd-59/?prefix=obj68 2023-05-31T20:52:13.297 DEBUG:boto:auth_path=/pklxkd-59/?prefix=obj68 2023-05-31T20:52:13.297 DEBUG:boto:Method: GET 2023-05-31T20:52:13.298 DEBUG:boto:Path: /pklxkd-59/?prefix=obj68 2023-05-31T20:52:13.298 DEBUG:boto:Data: 2023-05-31T20:52:13.299 DEBUG:boto:Headers: {} 2023-05-31T20:52:13.299 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:52:13.299 DEBUG:boto:Port: 8001 2023-05-31T20:52:13.300 DEBUG:boto:Params: {} 2023-05-31T20:52:13.300 DEBUG:boto:Token: None 2023-05-31T20:52:13.301 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:52:13 GMT /pklxkd-59/ 2023-05-31T20:52:13.301 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:2oppaLXJix1iBBaZf3oYtXHJCGM= 2023-05-31T20:52:13.302 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:52:13 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:2oppaLXJix1iBBaZf3oYtXHJCGM=', 'Content-Length': '0'} 2023-05-31T20:52:13.304 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b78ebaec18eb95b0-006477b37d-1c17-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:52:13 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:52:13.305 DEBUG:boto:b'pklxkd-59obj681000falseobj682023-05-31T20:43:08.066Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:52:13.305 DEBUG:boto:path=/pklxkd-59/ 2023-05-31T20:52:13.306 DEBUG:boto:auth_path=/pklxkd-59/ 2023-05-31T20:52:13.306 DEBUG:boto:path=/pklxkd-59/?prefix=obj68 2023-05-31T20:52:13.307 DEBUG:boto:auth_path=/pklxkd-59/?prefix=obj68 2023-05-31T20:52:13.307 DEBUG:boto:Method: GET 2023-05-31T20:52:13.308 DEBUG:boto:Path: /pklxkd-59/?prefix=obj68 2023-05-31T20:52:13.308 DEBUG:boto:Data: 2023-05-31T20:52:13.309 DEBUG:boto:Headers: {} 2023-05-31T20:52:13.309 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:52:13.310 DEBUG:boto:Port: 8000 2023-05-31T20:52:13.310 DEBUG:boto:Params: {} 2023-05-31T20:52:13.310 DEBUG:boto:Token: None 2023-05-31T20:52:13.311 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:52:13 GMT /pklxkd-59/ 2023-05-31T20:52:13.311 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:2oppaLXJix1iBBaZf3oYtXHJCGM= 2023-05-31T20:52:13.312 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:52:13 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:2oppaLXJix1iBBaZf3oYtXHJCGM=', 'Content-Length': '0'} 2023-05-31T20:52:13.315 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000579d139e347327c8-006477b37d-1630-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:52:13 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:52:13.316 DEBUG:boto:b'pklxkd-59obj681000falseobj682023-05-31T20:43:08.066Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:52:13.317 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj69 2023-05-31T20:52:13.317 DEBUG:boto:path=/pklxkd-59/obj69 2023-05-31T20:52:13.317 DEBUG:boto:auth_path=/pklxkd-59/obj69 2023-05-31T20:52:13.318 DEBUG:boto:path=/pklxkd-59/obj69?versionId=null 2023-05-31T20:52:13.318 DEBUG:boto:auth_path=/pklxkd-59/obj69?versionId=null 2023-05-31T20:52:13.318 DEBUG:boto:Method: GET 2023-05-31T20:52:13.319 DEBUG:boto:Path: /pklxkd-59/obj69?versionId=null 2023-05-31T20:52:13.319 DEBUG:boto:Data: 2023-05-31T20:52:13.319 DEBUG:boto:Headers: {} 2023-05-31T20:52:13.320 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:52:13.320 DEBUG:boto:Port: 8001 2023-05-31T20:52:13.321 DEBUG:boto:Params: {} 2023-05-31T20:52:13.321 DEBUG:boto:Token: None 2023-05-31T20:52:13.321 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:52:13 GMT /pklxkd-59/obj69?versionId=null 2023-05-31T20:52:13.322 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:QuW83h816skaxxtb4U50AxGpTho= 2023-05-31T20:52:13.322 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:52:13 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:QuW83h816skaxxtb4U50AxGpTho=', 'Content-Length': '0'} 2023-05-31T20:52:13.324 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:46:33 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002bcb43ae2c644a83-006477b37d-1c17-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:52:13 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:52:13.325 DEBUG:boto:path=/pklxkd-59/obj69 2023-05-31T20:52:13.325 DEBUG:boto:auth_path=/pklxkd-59/obj69 2023-05-31T20:52:13.326 DEBUG:boto:path=/pklxkd-59/obj69?versionId=null 2023-05-31T20:52:13.326 DEBUG:boto:auth_path=/pklxkd-59/obj69?versionId=null 2023-05-31T20:52:13.327 DEBUG:boto:Method: GET 2023-05-31T20:52:13.327 DEBUG:boto:Path: /pklxkd-59/obj69?versionId=null 2023-05-31T20:52:13.327 DEBUG:boto:Data: 2023-05-31T20:52:13.328 DEBUG:boto:Headers: {} 2023-05-31T20:52:13.328 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:52:13.329 DEBUG:boto:Port: 8000 2023-05-31T20:52:13.329 DEBUG:boto:Params: {} 2023-05-31T20:52:13.330 DEBUG:boto:Token: None 2023-05-31T20:52:13.330 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:52:13 GMT /pklxkd-59/obj69?versionId=null 2023-05-31T20:52:13.331 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:QuW83h816skaxxtb4U50AxGpTho= 2023-05-31T20:52:13.331 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:52:13 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:QuW83h816skaxxtb4U50AxGpTho=', 'Content-Length': '0'} 2023-05-31T20:52:13.333 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:46:33 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000da16daaf3c86756c-006477b37d-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:52:13 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:52:13.334 DEBUG:boto:path=/pklxkd-59/obj69 2023-05-31T20:52:13.335 DEBUG:boto:auth_path=/pklxkd-59/obj69 2023-05-31T20:52:13.335 DEBUG:boto:path=/pklxkd-59/obj69?versionId=null 2023-05-31T20:52:13.335 DEBUG:boto:auth_path=/pklxkd-59/obj69?versionId=null 2023-05-31T20:52:13.336 DEBUG:boto:Method: HEAD 2023-05-31T20:52:13.336 DEBUG:boto:Path: /pklxkd-59/obj69?versionId=null 2023-05-31T20:52:13.336 DEBUG:boto:Data: 2023-05-31T20:52:13.337 DEBUG:boto:Headers: {} 2023-05-31T20:52:13.337 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:52:13.337 DEBUG:boto:Port: 8001 2023-05-31T20:52:13.338 DEBUG:boto:Params: {} 2023-05-31T20:52:13.338 DEBUG:boto:Token: None 2023-05-31T20:52:13.338 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:52:13 GMT /pklxkd-59/obj69?versionId=null 2023-05-31T20:52:13.339 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:4FgxgAzMUKkkJOjcjhVxOlpWO5M= 2023-05-31T20:52:13.339 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:52:13 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:4FgxgAzMUKkkJOjcjhVxOlpWO5M=', 'Content-Length': '0'} 2023-05-31T20:52:13.341 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:46:33 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001f1c38fcef5251b1-006477b37d-1c17-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:52:13 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:52:13.342 DEBUG:boto:path=/pklxkd-59/obj69 2023-05-31T20:52:13.342 DEBUG:boto:auth_path=/pklxkd-59/obj69 2023-05-31T20:52:13.342 DEBUG:boto:path=/pklxkd-59/obj69?versionId=null 2023-05-31T20:52:13.343 DEBUG:boto:auth_path=/pklxkd-59/obj69?versionId=null 2023-05-31T20:52:13.343 DEBUG:boto:Method: HEAD 2023-05-31T20:52:13.343 DEBUG:boto:Path: /pklxkd-59/obj69?versionId=null 2023-05-31T20:52:13.344 DEBUG:boto:Data: 2023-05-31T20:52:13.344 DEBUG:boto:Headers: {} 2023-05-31T20:52:13.344 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:52:13.345 DEBUG:boto:Port: 8000 2023-05-31T20:52:13.345 DEBUG:boto:Params: {} 2023-05-31T20:52:13.345 DEBUG:boto:Token: None 2023-05-31T20:52:13.346 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:52:13 GMT /pklxkd-59/obj69?versionId=null 2023-05-31T20:52:13.346 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:4FgxgAzMUKkkJOjcjhVxOlpWO5M= 2023-05-31T20:52:13.346 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:52:13 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:4FgxgAzMUKkkJOjcjhVxOlpWO5M=', 'Content-Length': '0'} 2023-05-31T20:52:13.348 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:46:33 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007c818d46c8bff37a-006477b37d-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:52:13 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:52:13.349 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj69 2023-05-31T20:52:13.349 DEBUG:boto:path=/pklxkd-59/obj69 2023-05-31T20:52:13.350 DEBUG:boto:auth_path=/pklxkd-59/obj69 2023-05-31T20:52:13.350 DEBUG:boto:path=/pklxkd-59/obj69?versionId=null 2023-05-31T20:52:13.350 DEBUG:boto:auth_path=/pklxkd-59/obj69?versionId=null 2023-05-31T20:52:13.351 DEBUG:boto:Method: GET 2023-05-31T20:52:13.351 DEBUG:boto:Path: /pklxkd-59/obj69?versionId=null 2023-05-31T20:52:13.352 DEBUG:boto:Data: 2023-05-31T20:52:13.352 DEBUG:boto:Headers: {} 2023-05-31T20:52:13.352 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:52:13.353 DEBUG:boto:Port: 8001 2023-05-31T20:52:13.353 DEBUG:boto:Params: {} 2023-05-31T20:52:13.354 DEBUG:boto:Token: None 2023-05-31T20:52:13.354 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:52:13 GMT /pklxkd-59/obj69?versionId=null 2023-05-31T20:52:13.355 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:QuW83h816skaxxtb4U50AxGpTho= 2023-05-31T20:52:13.355 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:52:13 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:QuW83h816skaxxtb4U50AxGpTho=', 'Content-Length': '0'} 2023-05-31T20:52:13.357 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:46:33 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e8ae7f410d92cc98-006477b37d-1c17-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:52:13 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:52:13.400 DEBUG:boto:path=/pklxkd-59/obj69 2023-05-31T20:52:13.401 DEBUG:boto:auth_path=/pklxkd-59/obj69 2023-05-31T20:52:13.401 DEBUG:boto:path=/pklxkd-59/obj69?versionId=null 2023-05-31T20:52:13.401 DEBUG:boto:auth_path=/pklxkd-59/obj69?versionId=null 2023-05-31T20:52:13.402 DEBUG:boto:Method: GET 2023-05-31T20:52:13.402 DEBUG:boto:Path: /pklxkd-59/obj69?versionId=null 2023-05-31T20:52:13.402 DEBUG:boto:Data: 2023-05-31T20:52:13.403 DEBUG:boto:Headers: {} 2023-05-31T20:52:13.403 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:52:13.403 DEBUG:boto:Port: 8000 2023-05-31T20:52:13.404 DEBUG:boto:Params: {} 2023-05-31T20:52:13.404 DEBUG:boto:Token: None 2023-05-31T20:52:13.405 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:52:13 GMT /pklxkd-59/obj69?versionId=null 2023-05-31T20:52:13.405 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:QuW83h816skaxxtb4U50AxGpTho= 2023-05-31T20:52:13.406 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:52:13 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:QuW83h816skaxxtb4U50AxGpTho=', 'Content-Length': '0'} 2023-05-31T20:52:13.407 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:46:33 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000478a8e8094fd14c-006477b37d-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:52:13 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:52:13.408 DEBUG:boto:path=/pklxkd-59/ 2023-05-31T20:52:13.408 DEBUG:boto:auth_path=/pklxkd-59/ 2023-05-31T20:52:13.409 DEBUG:boto:path=/pklxkd-59/?prefix=obj69 2023-05-31T20:52:13.409 DEBUG:boto:auth_path=/pklxkd-59/?prefix=obj69 2023-05-31T20:52:13.410 DEBUG:boto:Method: GET 2023-05-31T20:52:13.410 DEBUG:boto:Path: /pklxkd-59/?prefix=obj69 2023-05-31T20:52:13.411 DEBUG:boto:Data: 2023-05-31T20:52:13.411 DEBUG:boto:Headers: {} 2023-05-31T20:52:13.411 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:52:13.412 DEBUG:boto:Port: 8001 2023-05-31T20:52:13.412 DEBUG:boto:Params: {} 2023-05-31T20:52:13.413 DEBUG:boto:Token: None 2023-05-31T20:52:13.413 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:52:13 GMT /pklxkd-59/ 2023-05-31T20:52:13.414 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:2oppaLXJix1iBBaZf3oYtXHJCGM= 2023-05-31T20:52:13.414 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:52:13 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:2oppaLXJix1iBBaZf3oYtXHJCGM=', 'Content-Length': '0'} 2023-05-31T20:52:13.417 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000071c26d3260162507-006477b37d-1c17-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:52:13 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:52:13.417 DEBUG:boto:b'pklxkd-59obj691000falseobj692023-05-31T20:46:33.886Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:52:13.418 DEBUG:boto:path=/pklxkd-59/ 2023-05-31T20:52:13.419 DEBUG:boto:auth_path=/pklxkd-59/ 2023-05-31T20:52:13.419 DEBUG:boto:path=/pklxkd-59/?prefix=obj69 2023-05-31T20:52:13.419 DEBUG:boto:auth_path=/pklxkd-59/?prefix=obj69 2023-05-31T20:52:13.420 DEBUG:boto:Method: GET 2023-05-31T20:52:13.420 DEBUG:boto:Path: /pklxkd-59/?prefix=obj69 2023-05-31T20:52:13.420 DEBUG:boto:Data: 2023-05-31T20:52:13.421 DEBUG:boto:Headers: {} 2023-05-31T20:52:13.421 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:52:13.421 DEBUG:boto:Port: 8000 2023-05-31T20:52:13.422 DEBUG:boto:Params: {} 2023-05-31T20:52:13.422 DEBUG:boto:Token: None 2023-05-31T20:52:13.423 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:52:13 GMT /pklxkd-59/ 2023-05-31T20:52:13.423 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:2oppaLXJix1iBBaZf3oYtXHJCGM= 2023-05-31T20:52:13.423 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:52:13 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:2oppaLXJix1iBBaZf3oYtXHJCGM=', 'Content-Length': '0'} 2023-05-31T20:52:13.427 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000009cbebdb3ea385e5a-006477b37d-1630-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:52:13 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:52:13.427 DEBUG:boto:b'pklxkd-59obj691000falseobj692023-05-31T20:46:33.886Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:52:13.428 DEBUG:boto:path=/pklxkd-59/obj69 2023-05-31T20:52:13.428 DEBUG:boto:auth_path=/pklxkd-59/obj69 2023-05-31T20:52:13.429 DEBUG:boto:Method: HEAD 2023-05-31T20:52:13.429 DEBUG:boto:Path: /pklxkd-59/obj69 2023-05-31T20:52:13.429 DEBUG:boto:Data: 2023-05-31T20:52:13.430 DEBUG:boto:Headers: {} 2023-05-31T20:52:13.430 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:52:13.430 DEBUG:boto:Port: 8001 2023-05-31T20:52:13.431 DEBUG:boto:Params: {} 2023-05-31T20:52:13.431 DEBUG:boto:Token: None 2023-05-31T20:52:13.431 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:52:13 GMT /pklxkd-59/obj69 2023-05-31T20:52:13.432 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:pE95Diz4zqVJ38la6AwZHRnbWT4= 2023-05-31T20:52:13.432 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:52:13 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:pE95Diz4zqVJ38la6AwZHRnbWT4=', 'Content-Length': '0'} 2023-05-31T20:52:13.434 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:46:33 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001660603e491c262b-006477b37d-1c17-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:52:13 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:52:13.435 DEBUG:boto:path=/pklxkd-59/obj69 2023-05-31T20:52:13.435 DEBUG:boto:auth_path=/pklxkd-59/obj69 2023-05-31T20:52:13.435 DEBUG:boto:Method: HEAD 2023-05-31T20:52:13.436 DEBUG:boto:Path: /pklxkd-59/obj69 2023-05-31T20:52:13.436 DEBUG:boto:Data: 2023-05-31T20:52:13.436 DEBUG:boto:Headers: {} 2023-05-31T20:52:13.437 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:52:13.437 DEBUG:boto:Port: 8000 2023-05-31T20:52:13.437 DEBUG:boto:Params: {} 2023-05-31T20:52:13.438 DEBUG:boto:Token: None 2023-05-31T20:52:13.438 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:52:13 GMT /pklxkd-59/obj69 2023-05-31T20:52:13.439 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:pE95Diz4zqVJ38la6AwZHRnbWT4= 2023-05-31T20:52:13.439 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:52:13 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:pE95Diz4zqVJ38la6AwZHRnbWT4=', 'Content-Length': '0'} 2023-05-31T20:52:13.441 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:46:33 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ff56e08044027eb1-006477b37d-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:52:13 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:52:13.441 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj69 2023-05-31T20:52:13.442 DEBUG:boto:path=/pklxkd-59/obj69 2023-05-31T20:52:13.442 DEBUG:boto:auth_path=/pklxkd-59/obj69 2023-05-31T20:52:13.443 DEBUG:boto:Method: GET 2023-05-31T20:52:13.443 DEBUG:boto:Path: /pklxkd-59/obj69 2023-05-31T20:52:13.443 DEBUG:boto:Data: 2023-05-31T20:52:13.444 DEBUG:boto:Headers: {} 2023-05-31T20:52:13.444 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:52:13.445 DEBUG:boto:Port: 8001 2023-05-31T20:52:13.445 DEBUG:boto:Params: {} 2023-05-31T20:52:13.446 DEBUG:boto:Token: None 2023-05-31T20:52:13.446 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:52:13 GMT /pklxkd-59/obj69 2023-05-31T20:52:13.447 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:k6V8KCg9vC57tKgsyvj7QkUuPVk= 2023-05-31T20:52:13.447 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:52:13 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:k6V8KCg9vC57tKgsyvj7QkUuPVk=', 'Content-Length': '0'} 2023-05-31T20:52:13.449 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:46:33 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b2bc53f38968a369-006477b37d-1c17-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:52:13 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:52:13.450 DEBUG:boto:path=/pklxkd-59/obj69 2023-05-31T20:52:13.450 DEBUG:boto:auth_path=/pklxkd-59/obj69 2023-05-31T20:52:13.451 DEBUG:boto:Method: GET 2023-05-31T20:52:13.451 DEBUG:boto:Path: /pklxkd-59/obj69 2023-05-31T20:52:13.451 DEBUG:boto:Data: 2023-05-31T20:52:13.452 DEBUG:boto:Headers: {} 2023-05-31T20:52:13.452 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:52:13.453 DEBUG:boto:Port: 8000 2023-05-31T20:52:13.453 DEBUG:boto:Params: {} 2023-05-31T20:52:13.453 DEBUG:boto:Token: None 2023-05-31T20:52:13.454 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:52:13 GMT /pklxkd-59/obj69 2023-05-31T20:52:13.454 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:k6V8KCg9vC57tKgsyvj7QkUuPVk= 2023-05-31T20:52:13.454 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:52:13 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:k6V8KCg9vC57tKgsyvj7QkUuPVk=', 'Content-Length': '0'} 2023-05-31T20:52:13.456 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:46:33 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000302b0905f20f6270-006477b37d-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:52:13 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:52:13.457 DEBUG:boto:path=/pklxkd-59/ 2023-05-31T20:52:13.457 DEBUG:boto:auth_path=/pklxkd-59/ 2023-05-31T20:52:13.458 DEBUG:boto:path=/pklxkd-59/?prefix=obj69 2023-05-31T20:52:13.458 DEBUG:boto:auth_path=/pklxkd-59/?prefix=obj69 2023-05-31T20:52:13.458 DEBUG:boto:Method: GET 2023-05-31T20:52:13.459 DEBUG:boto:Path: /pklxkd-59/?prefix=obj69 2023-05-31T20:52:13.459 DEBUG:boto:Data: 2023-05-31T20:52:13.459 DEBUG:boto:Headers: {} 2023-05-31T20:52:13.460 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:52:13.460 DEBUG:boto:Port: 8001 2023-05-31T20:52:13.460 DEBUG:boto:Params: {} 2023-05-31T20:52:13.461 DEBUG:boto:Token: None 2023-05-31T20:52:13.461 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:52:13 GMT /pklxkd-59/ 2023-05-31T20:52:13.462 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:2oppaLXJix1iBBaZf3oYtXHJCGM= 2023-05-31T20:52:13.462 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:52:13 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:2oppaLXJix1iBBaZf3oYtXHJCGM=', 'Content-Length': '0'} 2023-05-31T20:52:13.464 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000626a452d541fb332-006477b37d-1c17-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:52:13 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:52:13.465 DEBUG:boto:b'pklxkd-59obj691000falseobj692023-05-31T20:46:33.886Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:52:13.466 DEBUG:boto:path=/pklxkd-59/ 2023-05-31T20:52:13.466 DEBUG:boto:auth_path=/pklxkd-59/ 2023-05-31T20:52:13.466 DEBUG:boto:path=/pklxkd-59/?prefix=obj69 2023-05-31T20:52:13.467 DEBUG:boto:auth_path=/pklxkd-59/?prefix=obj69 2023-05-31T20:52:13.467 DEBUG:boto:Method: GET 2023-05-31T20:52:13.468 DEBUG:boto:Path: /pklxkd-59/?prefix=obj69 2023-05-31T20:52:13.468 DEBUG:boto:Data: 2023-05-31T20:52:13.468 DEBUG:boto:Headers: {} 2023-05-31T20:52:13.469 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:52:13.469 DEBUG:boto:Port: 8000 2023-05-31T20:52:13.470 DEBUG:boto:Params: {} 2023-05-31T20:52:13.470 DEBUG:boto:Token: None 2023-05-31T20:52:13.471 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:52:13 GMT /pklxkd-59/ 2023-05-31T20:52:13.471 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:2oppaLXJix1iBBaZf3oYtXHJCGM= 2023-05-31T20:52:13.472 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:52:13 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:2oppaLXJix1iBBaZf3oYtXHJCGM=', 'Content-Length': '0'} 2023-05-31T20:52:13.475 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000005769e47e748a2853-006477b37d-1630-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:52:13 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:52:13.476 DEBUG:boto:b'pklxkd-59obj691000falseobj692023-05-31T20:46:33.886Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:52:13.476 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj76 2023-05-31T20:52:13.477 DEBUG:boto:path=/pklxkd-59/obj76 2023-05-31T20:52:13.477 DEBUG:boto:auth_path=/pklxkd-59/obj76 2023-05-31T20:52:13.478 DEBUG:boto:path=/pklxkd-59/obj76?versionId=null 2023-05-31T20:52:13.478 DEBUG:boto:auth_path=/pklxkd-59/obj76?versionId=null 2023-05-31T20:52:13.478 DEBUG:boto:Method: GET 2023-05-31T20:52:13.479 DEBUG:boto:Path: /pklxkd-59/obj76?versionId=null 2023-05-31T20:52:13.479 DEBUG:boto:Data: 2023-05-31T20:52:13.479 DEBUG:boto:Headers: {} 2023-05-31T20:52:13.480 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:52:13.480 DEBUG:boto:Port: 8001 2023-05-31T20:52:13.481 DEBUG:boto:Params: {} 2023-05-31T20:52:13.481 DEBUG:boto:Token: None 2023-05-31T20:52:13.482 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:52:13 GMT /pklxkd-59/obj76?versionId=null 2023-05-31T20:52:13.482 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:Lz2FJdTx4xK7lGDJOPzfzUpsB4U= 2023-05-31T20:52:13.482 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:52:13 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:Lz2FJdTx4xK7lGDJOPzfzUpsB4U=', 'Content-Length': '0'} 2023-05-31T20:52:13.484 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:44:50 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000087538ed460401ba1-006477b37d-1c17-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:52:13 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:52:13.528 DEBUG:boto:path=/pklxkd-59/obj76 2023-05-31T20:52:13.529 DEBUG:boto:auth_path=/pklxkd-59/obj76 2023-05-31T20:52:13.529 DEBUG:boto:path=/pklxkd-59/obj76?versionId=null 2023-05-31T20:52:13.530 DEBUG:boto:auth_path=/pklxkd-59/obj76?versionId=null 2023-05-31T20:52:13.530 DEBUG:boto:Method: GET 2023-05-31T20:52:13.530 DEBUG:boto:Path: /pklxkd-59/obj76?versionId=null 2023-05-31T20:52:13.531 DEBUG:boto:Data: 2023-05-31T20:52:13.531 DEBUG:boto:Headers: {} 2023-05-31T20:52:13.532 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:52:13.532 DEBUG:boto:Port: 8000 2023-05-31T20:52:13.533 DEBUG:boto:Params: {} 2023-05-31T20:52:13.533 DEBUG:boto:Token: None 2023-05-31T20:52:13.533 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:52:13 GMT /pklxkd-59/obj76?versionId=null 2023-05-31T20:52:13.534 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:Lz2FJdTx4xK7lGDJOPzfzUpsB4U= 2023-05-31T20:52:13.535 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:52:13 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:Lz2FJdTx4xK7lGDJOPzfzUpsB4U=', 'Content-Length': '0'} 2023-05-31T20:52:13.537 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:44:50 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007d3d53e095a82abf-006477b37d-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:52:13 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:52:13.537 DEBUG:boto:path=/pklxkd-59/obj76 2023-05-31T20:52:13.538 DEBUG:boto:auth_path=/pklxkd-59/obj76 2023-05-31T20:52:13.538 DEBUG:boto:path=/pklxkd-59/obj76?versionId=null 2023-05-31T20:52:13.538 DEBUG:boto:auth_path=/pklxkd-59/obj76?versionId=null 2023-05-31T20:52:13.539 DEBUG:boto:Method: HEAD 2023-05-31T20:52:13.539 DEBUG:boto:Path: /pklxkd-59/obj76?versionId=null 2023-05-31T20:52:13.539 DEBUG:boto:Data: 2023-05-31T20:52:13.540 DEBUG:boto:Headers: {} 2023-05-31T20:52:13.540 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:52:13.540 DEBUG:boto:Port: 8001 2023-05-31T20:52:13.541 DEBUG:boto:Params: {} 2023-05-31T20:52:13.541 DEBUG:boto:Token: None 2023-05-31T20:52:13.542 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:52:13 GMT /pklxkd-59/obj76?versionId=null 2023-05-31T20:52:13.542 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:yJ/5angLz+0oWgmZSXHuX4bsGtg= 2023-05-31T20:52:13.543 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:52:13 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:yJ/5angLz+0oWgmZSXHuX4bsGtg=', 'Content-Length': '0'} 2023-05-31T20:52:13.545 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:44:50 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000da1738856ca61748-006477b37d-1c17-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:52:13 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:52:13.545 DEBUG:boto:path=/pklxkd-59/obj76 2023-05-31T20:52:13.546 DEBUG:boto:auth_path=/pklxkd-59/obj76 2023-05-31T20:52:13.546 DEBUG:boto:path=/pklxkd-59/obj76?versionId=null 2023-05-31T20:52:13.546 DEBUG:boto:auth_path=/pklxkd-59/obj76?versionId=null 2023-05-31T20:52:13.547 DEBUG:boto:Method: HEAD 2023-05-31T20:52:13.547 DEBUG:boto:Path: /pklxkd-59/obj76?versionId=null 2023-05-31T20:52:13.547 DEBUG:boto:Data: 2023-05-31T20:52:13.548 DEBUG:boto:Headers: {} 2023-05-31T20:52:13.548 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:52:13.548 DEBUG:boto:Port: 8000 2023-05-31T20:52:13.549 DEBUG:boto:Params: {} 2023-05-31T20:52:13.549 DEBUG:boto:Token: None 2023-05-31T20:52:13.549 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:52:13 GMT /pklxkd-59/obj76?versionId=null 2023-05-31T20:52:13.550 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:yJ/5angLz+0oWgmZSXHuX4bsGtg= 2023-05-31T20:52:13.550 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:52:13 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:yJ/5angLz+0oWgmZSXHuX4bsGtg=', 'Content-Length': '0'} 2023-05-31T20:52:13.552 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:44:50 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000044e5e92490c45f51-006477b37d-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:52:13 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:52:13.553 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj76 2023-05-31T20:52:13.553 DEBUG:boto:path=/pklxkd-59/obj76 2023-05-31T20:52:13.553 DEBUG:boto:auth_path=/pklxkd-59/obj76 2023-05-31T20:52:13.554 DEBUG:boto:path=/pklxkd-59/obj76?versionId=null 2023-05-31T20:52:13.554 DEBUG:boto:auth_path=/pklxkd-59/obj76?versionId=null 2023-05-31T20:52:13.554 DEBUG:boto:Method: GET 2023-05-31T20:52:13.555 DEBUG:boto:Path: /pklxkd-59/obj76?versionId=null 2023-05-31T20:52:13.555 DEBUG:boto:Data: 2023-05-31T20:52:13.555 DEBUG:boto:Headers: {} 2023-05-31T20:52:13.556 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:52:13.556 DEBUG:boto:Port: 8001 2023-05-31T20:52:13.556 DEBUG:boto:Params: {} 2023-05-31T20:52:13.557 DEBUG:boto:Token: None 2023-05-31T20:52:13.557 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:52:13 GMT /pklxkd-59/obj76?versionId=null 2023-05-31T20:52:13.558 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:Lz2FJdTx4xK7lGDJOPzfzUpsB4U= 2023-05-31T20:52:13.558 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:52:13 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:Lz2FJdTx4xK7lGDJOPzfzUpsB4U=', 'Content-Length': '0'} 2023-05-31T20:52:13.560 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:44:50 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000fadddd318c561844-006477b37d-1c17-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:52:13 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:52:13.561 DEBUG:boto:path=/pklxkd-59/obj76 2023-05-31T20:52:13.561 DEBUG:boto:auth_path=/pklxkd-59/obj76 2023-05-31T20:52:13.562 DEBUG:boto:path=/pklxkd-59/obj76?versionId=null 2023-05-31T20:52:13.562 DEBUG:boto:auth_path=/pklxkd-59/obj76?versionId=null 2023-05-31T20:52:13.562 DEBUG:boto:Method: GET 2023-05-31T20:52:13.563 DEBUG:boto:Path: /pklxkd-59/obj76?versionId=null 2023-05-31T20:52:13.563 DEBUG:boto:Data: 2023-05-31T20:52:13.564 DEBUG:boto:Headers: {} 2023-05-31T20:52:13.564 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:52:13.565 DEBUG:boto:Port: 8000 2023-05-31T20:52:13.565 DEBUG:boto:Params: {} 2023-05-31T20:52:13.566 DEBUG:boto:Token: None 2023-05-31T20:52:13.566 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:52:13 GMT /pklxkd-59/obj76?versionId=null 2023-05-31T20:52:13.567 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:Lz2FJdTx4xK7lGDJOPzfzUpsB4U= 2023-05-31T20:52:13.567 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:52:13 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:Lz2FJdTx4xK7lGDJOPzfzUpsB4U=', 'Content-Length': '0'} 2023-05-31T20:52:13.569 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:44:50 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000698fd8881f63e3d5-006477b37d-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:52:13 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:52:13.612 DEBUG:boto:path=/pklxkd-59/ 2023-05-31T20:52:13.613 DEBUG:boto:auth_path=/pklxkd-59/ 2023-05-31T20:52:13.613 DEBUG:boto:path=/pklxkd-59/?prefix=obj76 2023-05-31T20:52:13.613 DEBUG:boto:auth_path=/pklxkd-59/?prefix=obj76 2023-05-31T20:52:13.614 DEBUG:boto:Method: GET 2023-05-31T20:52:13.614 DEBUG:boto:Path: /pklxkd-59/?prefix=obj76 2023-05-31T20:52:13.615 DEBUG:boto:Data: 2023-05-31T20:52:13.615 DEBUG:boto:Headers: {} 2023-05-31T20:52:13.616 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:52:13.616 DEBUG:boto:Port: 8001 2023-05-31T20:52:13.616 DEBUG:boto:Params: {} 2023-05-31T20:52:13.617 DEBUG:boto:Token: None 2023-05-31T20:52:13.618 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:52:13 GMT /pklxkd-59/ 2023-05-31T20:52:13.618 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:2oppaLXJix1iBBaZf3oYtXHJCGM= 2023-05-31T20:52:13.619 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:52:13 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:2oppaLXJix1iBBaZf3oYtXHJCGM=', 'Content-Length': '0'} 2023-05-31T20:52:13.621 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000018018469b732deea-006477b37d-1c17-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:52:13 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:52:13.622 DEBUG:boto:b'pklxkd-59obj761000falseobj762023-05-31T20:44:50.682Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:52:13.622 DEBUG:boto:path=/pklxkd-59/ 2023-05-31T20:52:13.623 DEBUG:boto:auth_path=/pklxkd-59/ 2023-05-31T20:52:13.623 DEBUG:boto:path=/pklxkd-59/?prefix=obj76 2023-05-31T20:52:13.623 DEBUG:boto:auth_path=/pklxkd-59/?prefix=obj76 2023-05-31T20:52:13.624 DEBUG:boto:Method: GET 2023-05-31T20:52:13.624 DEBUG:boto:Path: /pklxkd-59/?prefix=obj76 2023-05-31T20:52:13.625 DEBUG:boto:Data: 2023-05-31T20:52:13.625 DEBUG:boto:Headers: {} 2023-05-31T20:52:13.625 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:52:13.626 DEBUG:boto:Port: 8000 2023-05-31T20:52:13.626 DEBUG:boto:Params: {} 2023-05-31T20:52:13.626 DEBUG:boto:Token: None 2023-05-31T20:52:13.627 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:52:13 GMT /pklxkd-59/ 2023-05-31T20:52:13.627 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:2oppaLXJix1iBBaZf3oYtXHJCGM= 2023-05-31T20:52:13.628 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:52:13 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:2oppaLXJix1iBBaZf3oYtXHJCGM=', 'Content-Length': '0'} 2023-05-31T20:52:13.631 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c8f83f29a017dd20-006477b37d-1630-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:52:13 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:52:13.632 DEBUG:boto:b'pklxkd-59obj761000falseobj762023-05-31T20:44:50.682Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:52:13.632 DEBUG:boto:path=/pklxkd-59/obj76 2023-05-31T20:52:13.633 DEBUG:boto:auth_path=/pklxkd-59/obj76 2023-05-31T20:52:13.633 DEBUG:boto:Method: HEAD 2023-05-31T20:52:13.634 DEBUG:boto:Path: /pklxkd-59/obj76 2023-05-31T20:52:13.634 DEBUG:boto:Data: 2023-05-31T20:52:13.634 DEBUG:boto:Headers: {} 2023-05-31T20:52:13.635 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:52:13.635 DEBUG:boto:Port: 8001 2023-05-31T20:52:13.635 DEBUG:boto:Params: {} 2023-05-31T20:52:13.636 DEBUG:boto:Token: None 2023-05-31T20:52:13.636 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:52:13 GMT /pklxkd-59/obj76 2023-05-31T20:52:13.636 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:RQ7F6GoFGRylVucgLJ7I9+fSUAs= 2023-05-31T20:52:13.637 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:52:13 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:RQ7F6GoFGRylVucgLJ7I9+fSUAs=', 'Content-Length': '0'} 2023-05-31T20:52:13.639 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:44:50 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000beb03d204e3c1dae-006477b37d-1c17-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:52:13 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:52:13.639 DEBUG:boto:path=/pklxkd-59/obj76 2023-05-31T20:52:13.640 DEBUG:boto:auth_path=/pklxkd-59/obj76 2023-05-31T20:52:13.640 DEBUG:boto:Method: HEAD 2023-05-31T20:52:13.641 DEBUG:boto:Path: /pklxkd-59/obj76 2023-05-31T20:52:13.641 DEBUG:boto:Data: 2023-05-31T20:52:13.641 DEBUG:boto:Headers: {} 2023-05-31T20:52:13.642 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:52:13.642 DEBUG:boto:Port: 8000 2023-05-31T20:52:13.642 DEBUG:boto:Params: {} 2023-05-31T20:52:13.643 DEBUG:boto:Token: None 2023-05-31T20:52:13.643 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:52:13 GMT /pklxkd-59/obj76 2023-05-31T20:52:13.644 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:RQ7F6GoFGRylVucgLJ7I9+fSUAs= 2023-05-31T20:52:13.644 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:52:13 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:RQ7F6GoFGRylVucgLJ7I9+fSUAs=', 'Content-Length': '0'} 2023-05-31T20:52:13.646 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:44:50 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000faf8bf4555cbae28-006477b37d-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:52:13 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:52:13.647 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj76 2023-05-31T20:52:13.647 DEBUG:boto:path=/pklxkd-59/obj76 2023-05-31T20:52:13.648 DEBUG:boto:auth_path=/pklxkd-59/obj76 2023-05-31T20:52:13.648 DEBUG:boto:Method: GET 2023-05-31T20:52:13.648 DEBUG:boto:Path: /pklxkd-59/obj76 2023-05-31T20:52:13.649 DEBUG:boto:Data: 2023-05-31T20:52:13.649 DEBUG:boto:Headers: {} 2023-05-31T20:52:13.650 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:52:13.650 DEBUG:boto:Port: 8001 2023-05-31T20:52:13.650 DEBUG:boto:Params: {} 2023-05-31T20:52:13.651 DEBUG:boto:Token: None 2023-05-31T20:52:13.651 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:52:13 GMT /pklxkd-59/obj76 2023-05-31T20:52:13.651 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:E/maH8SmEKo48N+ah8qGNIEkw4s= 2023-05-31T20:52:13.652 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:52:13 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:E/maH8SmEKo48N+ah8qGNIEkw4s=', 'Content-Length': '0'} 2023-05-31T20:52:13.654 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:44:50 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000452d8d3a4e70ce51-006477b37d-1c17-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:52:13 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:52:13.696 DEBUG:boto:path=/pklxkd-59/obj76 2023-05-31T20:52:13.697 DEBUG:boto:auth_path=/pklxkd-59/obj76 2023-05-31T20:52:13.697 DEBUG:boto:Method: GET 2023-05-31T20:52:13.698 DEBUG:boto:Path: /pklxkd-59/obj76 2023-05-31T20:52:13.698 DEBUG:boto:Data: 2023-05-31T20:52:13.699 DEBUG:boto:Headers: {} 2023-05-31T20:52:13.699 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:52:13.700 DEBUG:boto:Port: 8000 2023-05-31T20:52:13.700 DEBUG:boto:Params: {} 2023-05-31T20:52:13.701 DEBUG:boto:Token: None 2023-05-31T20:52:13.701 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:52:13 GMT /pklxkd-59/obj76 2023-05-31T20:52:13.702 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:E/maH8SmEKo48N+ah8qGNIEkw4s= 2023-05-31T20:52:13.702 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:52:13 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:E/maH8SmEKo48N+ah8qGNIEkw4s=', 'Content-Length': '0'} 2023-05-31T20:52:13.704 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:44:50 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006d0fd88522625d8e-006477b37d-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:52:13 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:52:13.705 DEBUG:boto:path=/pklxkd-59/ 2023-05-31T20:52:13.705 DEBUG:boto:auth_path=/pklxkd-59/ 2023-05-31T20:52:13.705 DEBUG:boto:path=/pklxkd-59/?prefix=obj76 2023-05-31T20:52:13.706 DEBUG:boto:auth_path=/pklxkd-59/?prefix=obj76 2023-05-31T20:52:13.706 DEBUG:boto:Method: GET 2023-05-31T20:52:13.706 DEBUG:boto:Path: /pklxkd-59/?prefix=obj76 2023-05-31T20:52:13.707 DEBUG:boto:Data: 2023-05-31T20:52:13.707 DEBUG:boto:Headers: {} 2023-05-31T20:52:13.707 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:52:13.708 DEBUG:boto:Port: 8001 2023-05-31T20:52:13.708 DEBUG:boto:Params: {} 2023-05-31T20:52:13.709 DEBUG:boto:Token: None 2023-05-31T20:52:13.709 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:52:13 GMT /pklxkd-59/ 2023-05-31T20:52:13.709 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:2oppaLXJix1iBBaZf3oYtXHJCGM= 2023-05-31T20:52:13.710 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:52:13 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:2oppaLXJix1iBBaZf3oYtXHJCGM=', 'Content-Length': '0'} 2023-05-31T20:52:13.712 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d86a1399d6b4e6fd-006477b37d-1c17-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:52:13 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:52:13.713 DEBUG:boto:b'pklxkd-59obj761000falseobj762023-05-31T20:44:50.682Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:52:13.713 DEBUG:boto:path=/pklxkd-59/ 2023-05-31T20:52:13.714 DEBUG:boto:auth_path=/pklxkd-59/ 2023-05-31T20:52:13.714 DEBUG:boto:path=/pklxkd-59/?prefix=obj76 2023-05-31T20:52:13.714 DEBUG:boto:auth_path=/pklxkd-59/?prefix=obj76 2023-05-31T20:52:13.715 DEBUG:boto:Method: GET 2023-05-31T20:52:13.715 DEBUG:boto:Path: /pklxkd-59/?prefix=obj76 2023-05-31T20:52:13.715 DEBUG:boto:Data: 2023-05-31T20:52:13.716 DEBUG:boto:Headers: {} 2023-05-31T20:52:13.716 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:52:13.717 DEBUG:boto:Port: 8000 2023-05-31T20:52:13.717 DEBUG:boto:Params: {} 2023-05-31T20:52:13.717 DEBUG:boto:Token: None 2023-05-31T20:52:13.718 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:52:13 GMT /pklxkd-59/ 2023-05-31T20:52:13.718 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:2oppaLXJix1iBBaZf3oYtXHJCGM= 2023-05-31T20:52:13.719 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:52:13 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:2oppaLXJix1iBBaZf3oYtXHJCGM=', 'Content-Length': '0'} 2023-05-31T20:52:13.722 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000bb24818ac6da6054-006477b37d-1630-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:52:13 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:52:13.723 DEBUG:boto:b'pklxkd-59obj761000falseobj762023-05-31T20:44:50.682Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:52:13.724 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj85 2023-05-31T20:52:13.724 DEBUG:boto:path=/pklxkd-59/obj85 2023-05-31T20:52:13.724 DEBUG:boto:auth_path=/pklxkd-59/obj85 2023-05-31T20:52:13.725 DEBUG:boto:path=/pklxkd-59/obj85?versionId=null 2023-05-31T20:52:13.725 DEBUG:boto:auth_path=/pklxkd-59/obj85?versionId=null 2023-05-31T20:52:13.725 DEBUG:boto:Method: GET 2023-05-31T20:52:13.726 DEBUG:boto:Path: /pklxkd-59/obj85?versionId=null 2023-05-31T20:52:13.726 DEBUG:boto:Data: 2023-05-31T20:52:13.727 DEBUG:boto:Headers: {} 2023-05-31T20:52:13.727 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:52:13.727 DEBUG:boto:Port: 8001 2023-05-31T20:52:13.728 DEBUG:boto:Params: {} 2023-05-31T20:52:13.728 DEBUG:boto:Token: None 2023-05-31T20:52:13.728 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:52:13 GMT /pklxkd-59/obj85?versionId=null 2023-05-31T20:52:13.729 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:BE/OqhjK3mhOcG51pbQumL3UrSg= 2023-05-31T20:52:13.729 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:52:13 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:BE/OqhjK3mhOcG51pbQumL3UrSg=', 'Content-Length': '0'} 2023-05-31T20:52:13.731 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:43:08 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ba994ccf62a12f12-006477b37d-1c17-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:52:13 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:52:13.732 DEBUG:boto:path=/pklxkd-59/obj85 2023-05-31T20:52:13.732 DEBUG:boto:auth_path=/pklxkd-59/obj85 2023-05-31T20:52:13.733 DEBUG:boto:path=/pklxkd-59/obj85?versionId=null 2023-05-31T20:52:13.733 DEBUG:boto:auth_path=/pklxkd-59/obj85?versionId=null 2023-05-31T20:52:13.734 DEBUG:boto:Method: GET 2023-05-31T20:52:13.734 DEBUG:boto:Path: /pklxkd-59/obj85?versionId=null 2023-05-31T20:52:13.735 DEBUG:boto:Data: 2023-05-31T20:52:13.735 DEBUG:boto:Headers: {} 2023-05-31T20:52:13.735 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:52:13.736 DEBUG:boto:Port: 8000 2023-05-31T20:52:13.736 DEBUG:boto:Params: {} 2023-05-31T20:52:13.736 DEBUG:boto:Token: None 2023-05-31T20:52:13.737 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:52:13 GMT /pklxkd-59/obj85?versionId=null 2023-05-31T20:52:13.737 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:BE/OqhjK3mhOcG51pbQumL3UrSg= 2023-05-31T20:52:13.738 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:52:13 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:BE/OqhjK3mhOcG51pbQumL3UrSg=', 'Content-Length': '0'} 2023-05-31T20:52:13.740 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:43:08 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000086b75c5b45459907-006477b37d-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:52:13 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:52:13.740 DEBUG:boto:path=/pklxkd-59/obj85 2023-05-31T20:52:13.741 DEBUG:boto:auth_path=/pklxkd-59/obj85 2023-05-31T20:52:13.741 DEBUG:boto:path=/pklxkd-59/obj85?versionId=null 2023-05-31T20:52:13.741 DEBUG:boto:auth_path=/pklxkd-59/obj85?versionId=null 2023-05-31T20:52:13.742 DEBUG:boto:Method: HEAD 2023-05-31T20:52:13.742 DEBUG:boto:Path: /pklxkd-59/obj85?versionId=null 2023-05-31T20:52:13.742 DEBUG:boto:Data: 2023-05-31T20:52:13.743 DEBUG:boto:Headers: {} 2023-05-31T20:52:13.743 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:52:13.743 DEBUG:boto:Port: 8001 2023-05-31T20:52:13.744 DEBUG:boto:Params: {} 2023-05-31T20:52:13.744 DEBUG:boto:Token: None 2023-05-31T20:52:13.744 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:52:13 GMT /pklxkd-59/obj85?versionId=null 2023-05-31T20:52:13.745 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:BSYL2uxkCz8B/6Xahpakzxio7dE= 2023-05-31T20:52:13.745 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:52:13 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:BSYL2uxkCz8B/6Xahpakzxio7dE=', 'Content-Length': '0'} 2023-05-31T20:52:13.747 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:43:08 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000913c4f31d7f7daf8-006477b37d-1c17-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:52:13 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:52:13.747 DEBUG:boto:path=/pklxkd-59/obj85 2023-05-31T20:52:13.748 DEBUG:boto:auth_path=/pklxkd-59/obj85 2023-05-31T20:52:13.748 DEBUG:boto:path=/pklxkd-59/obj85?versionId=null 2023-05-31T20:52:13.748 DEBUG:boto:auth_path=/pklxkd-59/obj85?versionId=null 2023-05-31T20:52:13.749 DEBUG:boto:Method: HEAD 2023-05-31T20:52:13.749 DEBUG:boto:Path: /pklxkd-59/obj85?versionId=null 2023-05-31T20:52:13.750 DEBUG:boto:Data: 2023-05-31T20:52:13.750 DEBUG:boto:Headers: {} 2023-05-31T20:52:13.750 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:52:13.751 DEBUG:boto:Port: 8000 2023-05-31T20:52:13.751 DEBUG:boto:Params: {} 2023-05-31T20:52:13.752 DEBUG:boto:Token: None 2023-05-31T20:52:13.752 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:52:13 GMT /pklxkd-59/obj85?versionId=null 2023-05-31T20:52:13.752 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:BSYL2uxkCz8B/6Xahpakzxio7dE= 2023-05-31T20:52:13.753 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:52:13 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:BSYL2uxkCz8B/6Xahpakzxio7dE=', 'Content-Length': '0'} 2023-05-31T20:52:13.755 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:43:08 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c0809fb976ed2c6b-006477b37d-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:52:13 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:52:13.755 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj85 2023-05-31T20:52:13.756 DEBUG:boto:path=/pklxkd-59/obj85 2023-05-31T20:52:13.756 DEBUG:boto:auth_path=/pklxkd-59/obj85 2023-05-31T20:52:13.757 DEBUG:boto:path=/pklxkd-59/obj85?versionId=null 2023-05-31T20:52:13.757 DEBUG:boto:auth_path=/pklxkd-59/obj85?versionId=null 2023-05-31T20:52:13.758 DEBUG:boto:Method: GET 2023-05-31T20:52:13.758 DEBUG:boto:Path: /pklxkd-59/obj85?versionId=null 2023-05-31T20:52:13.759 DEBUG:boto:Data: 2023-05-31T20:52:13.759 DEBUG:boto:Headers: {} 2023-05-31T20:52:13.759 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:52:13.760 DEBUG:boto:Port: 8001 2023-05-31T20:52:13.760 DEBUG:boto:Params: {} 2023-05-31T20:52:13.760 DEBUG:boto:Token: None 2023-05-31T20:52:13.761 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:52:13 GMT /pklxkd-59/obj85?versionId=null 2023-05-31T20:52:13.761 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:BE/OqhjK3mhOcG51pbQumL3UrSg= 2023-05-31T20:52:13.762 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:52:13 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:BE/OqhjK3mhOcG51pbQumL3UrSg=', 'Content-Length': '0'} 2023-05-31T20:52:13.764 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:43:08 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009eea735168ca789b-006477b37d-1c17-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:52:13 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:52:13.764 DEBUG:boto:path=/pklxkd-59/obj85 2023-05-31T20:52:13.765 DEBUG:boto:auth_path=/pklxkd-59/obj85 2023-05-31T20:52:13.765 DEBUG:boto:path=/pklxkd-59/obj85?versionId=null 2023-05-31T20:52:13.765 DEBUG:boto:auth_path=/pklxkd-59/obj85?versionId=null 2023-05-31T20:52:13.766 DEBUG:boto:Method: GET 2023-05-31T20:52:13.766 DEBUG:boto:Path: /pklxkd-59/obj85?versionId=null 2023-05-31T20:52:13.766 DEBUG:boto:Data: 2023-05-31T20:52:13.767 DEBUG:boto:Headers: {} 2023-05-31T20:52:13.767 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:52:13.767 DEBUG:boto:Port: 8000 2023-05-31T20:52:13.768 DEBUG:boto:Params: {} 2023-05-31T20:52:13.768 DEBUG:boto:Token: None 2023-05-31T20:52:13.769 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:52:13 GMT /pklxkd-59/obj85?versionId=null 2023-05-31T20:52:13.769 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:BE/OqhjK3mhOcG51pbQumL3UrSg= 2023-05-31T20:52:13.769 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:52:13 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:BE/OqhjK3mhOcG51pbQumL3UrSg=', 'Content-Length': '0'} 2023-05-31T20:52:13.771 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:43:08 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000be941760dec835b6-006477b37d-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:52:13 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:52:13.772 DEBUG:boto:path=/pklxkd-59/ 2023-05-31T20:52:13.772 DEBUG:boto:auth_path=/pklxkd-59/ 2023-05-31T20:52:13.773 DEBUG:boto:path=/pklxkd-59/?prefix=obj85 2023-05-31T20:52:13.773 DEBUG:boto:auth_path=/pklxkd-59/?prefix=obj85 2023-05-31T20:52:13.773 DEBUG:boto:Method: GET 2023-05-31T20:52:13.774 DEBUG:boto:Path: /pklxkd-59/?prefix=obj85 2023-05-31T20:52:13.774 DEBUG:boto:Data: 2023-05-31T20:52:13.775 DEBUG:boto:Headers: {} 2023-05-31T20:52:13.775 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:52:13.775 DEBUG:boto:Port: 8001 2023-05-31T20:52:13.776 DEBUG:boto:Params: {} 2023-05-31T20:52:13.776 DEBUG:boto:Token: None 2023-05-31T20:52:13.777 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:52:13 GMT /pklxkd-59/ 2023-05-31T20:52:13.777 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:2oppaLXJix1iBBaZf3oYtXHJCGM= 2023-05-31T20:52:13.778 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:52:13 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:2oppaLXJix1iBBaZf3oYtXHJCGM=', 'Content-Length': '0'} 2023-05-31T20:52:13.780 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d8fa08d5aacf7ee9-006477b37d-1c17-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:52:13 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:52:13.781 DEBUG:boto:b'pklxkd-59obj851000falseobj852023-05-31T20:43:08.086Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:52:13.781 DEBUG:boto:path=/pklxkd-59/ 2023-05-31T20:52:13.782 DEBUG:boto:auth_path=/pklxkd-59/ 2023-05-31T20:52:13.782 DEBUG:boto:path=/pklxkd-59/?prefix=obj85 2023-05-31T20:52:13.783 DEBUG:boto:auth_path=/pklxkd-59/?prefix=obj85 2023-05-31T20:52:13.783 DEBUG:boto:Method: GET 2023-05-31T20:52:13.783 DEBUG:boto:Path: /pklxkd-59/?prefix=obj85 2023-05-31T20:52:13.784 DEBUG:boto:Data: 2023-05-31T20:52:13.784 DEBUG:boto:Headers: {} 2023-05-31T20:52:13.784 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:52:13.785 DEBUG:boto:Port: 8000 2023-05-31T20:52:13.785 DEBUG:boto:Params: {} 2023-05-31T20:52:13.786 DEBUG:boto:Token: None 2023-05-31T20:52:13.786 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:52:13 GMT /pklxkd-59/ 2023-05-31T20:52:13.787 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:2oppaLXJix1iBBaZf3oYtXHJCGM= 2023-05-31T20:52:13.787 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:52:13 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:2oppaLXJix1iBBaZf3oYtXHJCGM=', 'Content-Length': '0'} 2023-05-31T20:52:13.791 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000005c2378acc902ce1d-006477b37d-1630-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:52:13 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:52:13.791 DEBUG:boto:b'pklxkd-59obj851000falseobj852023-05-31T20:43:08.086Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:52:13.792 DEBUG:boto:path=/pklxkd-59/obj85 2023-05-31T20:52:13.792 DEBUG:boto:auth_path=/pklxkd-59/obj85 2023-05-31T20:52:13.793 DEBUG:boto:Method: HEAD 2023-05-31T20:52:13.793 DEBUG:boto:Path: /pklxkd-59/obj85 2023-05-31T20:52:13.794 DEBUG:boto:Data: 2023-05-31T20:52:13.794 DEBUG:boto:Headers: {} 2023-05-31T20:52:13.794 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:52:13.794 DEBUG:boto:Port: 8001 2023-05-31T20:52:13.795 DEBUG:boto:Params: {} 2023-05-31T20:52:13.795 DEBUG:boto:Token: None 2023-05-31T20:52:13.796 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:52:13 GMT /pklxkd-59/obj85 2023-05-31T20:52:13.796 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:MGwLxDnxv9mgPLBVMuzZpUlk7yQ= 2023-05-31T20:52:13.796 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:52:13 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:MGwLxDnxv9mgPLBVMuzZpUlk7yQ=', 'Content-Length': '0'} 2023-05-31T20:52:13.798 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:43:08 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f34a1bf53fbe8127-006477b37d-1c17-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:52:13 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:52:13.799 DEBUG:boto:path=/pklxkd-59/obj85 2023-05-31T20:52:13.799 DEBUG:boto:auth_path=/pklxkd-59/obj85 2023-05-31T20:52:13.799 DEBUG:boto:Method: HEAD 2023-05-31T20:52:13.800 DEBUG:boto:Path: /pklxkd-59/obj85 2023-05-31T20:52:13.800 DEBUG:boto:Data: 2023-05-31T20:52:13.800 DEBUG:boto:Headers: {} 2023-05-31T20:52:13.801 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:52:13.801 DEBUG:boto:Port: 8000 2023-05-31T20:52:13.801 DEBUG:boto:Params: {} 2023-05-31T20:52:13.802 DEBUG:boto:Token: None 2023-05-31T20:52:13.802 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:52:13 GMT /pklxkd-59/obj85 2023-05-31T20:52:13.802 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:MGwLxDnxv9mgPLBVMuzZpUlk7yQ= 2023-05-31T20:52:13.803 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:52:13 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:MGwLxDnxv9mgPLBVMuzZpUlk7yQ=', 'Content-Length': '0'} 2023-05-31T20:52:13.805 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:43:08 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e47d773a404ae145-006477b37d-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:52:13 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:52:13.805 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj85 2023-05-31T20:52:13.806 DEBUG:boto:path=/pklxkd-59/obj85 2023-05-31T20:52:13.806 DEBUG:boto:auth_path=/pklxkd-59/obj85 2023-05-31T20:52:13.806 DEBUG:boto:Method: GET 2023-05-31T20:52:13.807 DEBUG:boto:Path: /pklxkd-59/obj85 2023-05-31T20:52:13.807 DEBUG:boto:Data: 2023-05-31T20:52:13.808 DEBUG:boto:Headers: {} 2023-05-31T20:52:13.808 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:52:13.809 DEBUG:boto:Port: 8001 2023-05-31T20:52:13.809 DEBUG:boto:Params: {} 2023-05-31T20:52:13.809 DEBUG:boto:Token: None 2023-05-31T20:52:13.810 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:52:13 GMT /pklxkd-59/obj85 2023-05-31T20:52:13.810 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:OomL9GA2ecRQqzO6sWzTEXM4JvY= 2023-05-31T20:52:13.811 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:52:13 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:OomL9GA2ecRQqzO6sWzTEXM4JvY=', 'Content-Length': '0'} 2023-05-31T20:52:13.813 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:43:08 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005ecd3f26eea69e1c-006477b37d-1c17-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:52:13 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:52:13.856 DEBUG:boto:path=/pklxkd-59/obj85 2023-05-31T20:52:13.857 DEBUG:boto:auth_path=/pklxkd-59/obj85 2023-05-31T20:52:13.857 DEBUG:boto:Method: GET 2023-05-31T20:52:13.858 DEBUG:boto:Path: /pklxkd-59/obj85 2023-05-31T20:52:13.858 DEBUG:boto:Data: 2023-05-31T20:52:13.858 DEBUG:boto:Headers: {} 2023-05-31T20:52:13.859 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:52:13.859 DEBUG:boto:Port: 8000 2023-05-31T20:52:13.860 DEBUG:boto:Params: {} 2023-05-31T20:52:13.860 DEBUG:boto:Token: None 2023-05-31T20:52:13.861 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:52:13 GMT /pklxkd-59/obj85 2023-05-31T20:52:13.861 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:OomL9GA2ecRQqzO6sWzTEXM4JvY= 2023-05-31T20:52:13.862 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:52:13 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:OomL9GA2ecRQqzO6sWzTEXM4JvY=', 'Content-Length': '0'} 2023-05-31T20:52:13.863 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:43:08 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000207512654a42ab6-006477b37d-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:52:13 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:52:13.864 DEBUG:boto:path=/pklxkd-59/ 2023-05-31T20:52:13.865 DEBUG:boto:auth_path=/pklxkd-59/ 2023-05-31T20:52:13.865 DEBUG:boto:path=/pklxkd-59/?prefix=obj85 2023-05-31T20:52:13.866 DEBUG:boto:auth_path=/pklxkd-59/?prefix=obj85 2023-05-31T20:52:13.866 DEBUG:boto:Method: GET 2023-05-31T20:52:13.867 DEBUG:boto:Path: /pklxkd-59/?prefix=obj85 2023-05-31T20:52:13.867 DEBUG:boto:Data: 2023-05-31T20:52:13.867 DEBUG:boto:Headers: {} 2023-05-31T20:52:13.868 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:52:13.868 DEBUG:boto:Port: 8001 2023-05-31T20:52:13.868 DEBUG:boto:Params: {} 2023-05-31T20:52:13.869 DEBUG:boto:Token: None 2023-05-31T20:52:13.869 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:52:13 GMT /pklxkd-59/ 2023-05-31T20:52:13.870 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:2oppaLXJix1iBBaZf3oYtXHJCGM= 2023-05-31T20:52:13.870 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:52:13 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:2oppaLXJix1iBBaZf3oYtXHJCGM=', 'Content-Length': '0'} 2023-05-31T20:52:13.873 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000007a22981a01d70b09-006477b37d-1c17-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:52:13 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:52:13.873 DEBUG:boto:b'pklxkd-59obj851000falseobj852023-05-31T20:43:08.086Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:52:13.874 DEBUG:boto:path=/pklxkd-59/ 2023-05-31T20:52:13.874 DEBUG:boto:auth_path=/pklxkd-59/ 2023-05-31T20:52:13.874 DEBUG:boto:path=/pklxkd-59/?prefix=obj85 2023-05-31T20:52:13.875 DEBUG:boto:auth_path=/pklxkd-59/?prefix=obj85 2023-05-31T20:52:13.875 DEBUG:boto:Method: GET 2023-05-31T20:52:13.876 DEBUG:boto:Path: /pklxkd-59/?prefix=obj85 2023-05-31T20:52:13.876 DEBUG:boto:Data: 2023-05-31T20:52:13.876 DEBUG:boto:Headers: {} 2023-05-31T20:52:13.877 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:52:13.877 DEBUG:boto:Port: 8000 2023-05-31T20:52:13.877 DEBUG:boto:Params: {} 2023-05-31T20:52:13.878 DEBUG:boto:Token: None 2023-05-31T20:52:13.878 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:52:13 GMT /pklxkd-59/ 2023-05-31T20:52:13.879 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:2oppaLXJix1iBBaZf3oYtXHJCGM= 2023-05-31T20:52:13.879 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:52:13 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:2oppaLXJix1iBBaZf3oYtXHJCGM=', 'Content-Length': '0'} 2023-05-31T20:52:13.883 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a0bf7e24f4e9d61f-006477b37d-1630-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:52:13 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:52:13.883 DEBUG:boto:b'pklxkd-59obj851000falseobj852023-05-31T20:43:08.086Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:52:13.884 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj87 2023-05-31T20:52:13.884 DEBUG:boto:path=/pklxkd-59/obj87 2023-05-31T20:52:13.885 DEBUG:boto:auth_path=/pklxkd-59/obj87 2023-05-31T20:52:13.885 DEBUG:boto:path=/pklxkd-59/obj87?versionId=null 2023-05-31T20:52:13.886 DEBUG:boto:auth_path=/pklxkd-59/obj87?versionId=null 2023-05-31T20:52:13.886 DEBUG:boto:Method: GET 2023-05-31T20:52:13.887 DEBUG:boto:Path: /pklxkd-59/obj87?versionId=null 2023-05-31T20:52:13.887 DEBUG:boto:Data: 2023-05-31T20:52:13.887 DEBUG:boto:Headers: {} 2023-05-31T20:52:13.888 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:52:13.888 DEBUG:boto:Port: 8001 2023-05-31T20:52:13.889 DEBUG:boto:Params: {} 2023-05-31T20:52:13.889 DEBUG:boto:Token: None 2023-05-31T20:52:13.890 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:52:13 GMT /pklxkd-59/obj87?versionId=null 2023-05-31T20:52:13.890 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:+yxhuUyYXZCC6Bi9DlPpbunReIU= 2023-05-31T20:52:13.891 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:52:13 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:+yxhuUyYXZCC6Bi9DlPpbunReIU=', 'Content-Length': '0'} 2023-05-31T20:52:13.893 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:48:13 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000005f0bf50697dc683-006477b37d-1c17-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:52:13 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:52:13.894 DEBUG:boto:path=/pklxkd-59/obj87 2023-05-31T20:52:13.894 DEBUG:boto:auth_path=/pklxkd-59/obj87 2023-05-31T20:52:13.894 DEBUG:boto:path=/pklxkd-59/obj87?versionId=null 2023-05-31T20:52:13.895 DEBUG:boto:auth_path=/pklxkd-59/obj87?versionId=null 2023-05-31T20:52:13.895 DEBUG:boto:Method: GET 2023-05-31T20:52:13.895 DEBUG:boto:Path: /pklxkd-59/obj87?versionId=null 2023-05-31T20:52:13.896 DEBUG:boto:Data: 2023-05-31T20:52:13.896 DEBUG:boto:Headers: {} 2023-05-31T20:52:13.897 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:52:13.897 DEBUG:boto:Port: 8000 2023-05-31T20:52:13.897 DEBUG:boto:Params: {} 2023-05-31T20:52:13.898 DEBUG:boto:Token: None 2023-05-31T20:52:13.898 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:52:13 GMT /pklxkd-59/obj87?versionId=null 2023-05-31T20:52:13.899 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:+yxhuUyYXZCC6Bi9DlPpbunReIU= 2023-05-31T20:52:13.899 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:52:13 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:+yxhuUyYXZCC6Bi9DlPpbunReIU=', 'Content-Length': '0'} 2023-05-31T20:52:13.901 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:48:13 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000040b3280c6149006d-006477b37d-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:52:13 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:52:13.902 DEBUG:boto:path=/pklxkd-59/obj87 2023-05-31T20:52:13.902 DEBUG:boto:auth_path=/pklxkd-59/obj87 2023-05-31T20:52:13.902 DEBUG:boto:path=/pklxkd-59/obj87?versionId=null 2023-05-31T20:52:13.903 DEBUG:boto:auth_path=/pklxkd-59/obj87?versionId=null 2023-05-31T20:52:13.903 DEBUG:boto:Method: HEAD 2023-05-31T20:52:13.903 DEBUG:boto:Path: /pklxkd-59/obj87?versionId=null 2023-05-31T20:52:13.904 DEBUG:boto:Data: 2023-05-31T20:52:13.904 DEBUG:boto:Headers: {} 2023-05-31T20:52:13.904 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:52:13.905 DEBUG:boto:Port: 8001 2023-05-31T20:52:13.905 DEBUG:boto:Params: {} 2023-05-31T20:52:13.905 DEBUG:boto:Token: None 2023-05-31T20:52:13.906 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:52:13 GMT /pklxkd-59/obj87?versionId=null 2023-05-31T20:52:13.906 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:dJ9SIT5ou/cTVhQjqusTClGxj7g= 2023-05-31T20:52:13.906 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:52:13 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:dJ9SIT5ou/cTVhQjqusTClGxj7g=', 'Content-Length': '0'} 2023-05-31T20:52:13.908 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:48:13 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008f2d6a7d20a11d89-006477b37d-1c17-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:52:13 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:52:13.909 DEBUG:boto:path=/pklxkd-59/obj87 2023-05-31T20:52:13.909 DEBUG:boto:auth_path=/pklxkd-59/obj87 2023-05-31T20:52:13.910 DEBUG:boto:path=/pklxkd-59/obj87?versionId=null 2023-05-31T20:52:13.910 DEBUG:boto:auth_path=/pklxkd-59/obj87?versionId=null 2023-05-31T20:52:13.910 DEBUG:boto:Method: HEAD 2023-05-31T20:52:13.911 DEBUG:boto:Path: /pklxkd-59/obj87?versionId=null 2023-05-31T20:52:13.911 DEBUG:boto:Data: 2023-05-31T20:52:13.911 DEBUG:boto:Headers: {} 2023-05-31T20:52:13.912 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:52:13.912 DEBUG:boto:Port: 8000 2023-05-31T20:52:13.913 DEBUG:boto:Params: {} 2023-05-31T20:52:13.913 DEBUG:boto:Token: None 2023-05-31T20:52:13.913 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:52:13 GMT /pklxkd-59/obj87?versionId=null 2023-05-31T20:52:13.914 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:dJ9SIT5ou/cTVhQjqusTClGxj7g= 2023-05-31T20:52:13.914 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:52:13 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:dJ9SIT5ou/cTVhQjqusTClGxj7g=', 'Content-Length': '0'} 2023-05-31T20:52:13.916 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:48:13 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005d4f1d3874d3c223-006477b37d-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:52:13 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:52:13.917 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj87 2023-05-31T20:52:13.917 DEBUG:boto:path=/pklxkd-59/obj87 2023-05-31T20:52:13.918 DEBUG:boto:auth_path=/pklxkd-59/obj87 2023-05-31T20:52:13.918 DEBUG:boto:path=/pklxkd-59/obj87?versionId=null 2023-05-31T20:52:13.918 DEBUG:boto:auth_path=/pklxkd-59/obj87?versionId=null 2023-05-31T20:52:13.919 DEBUG:boto:Method: GET 2023-05-31T20:52:13.919 DEBUG:boto:Path: /pklxkd-59/obj87?versionId=null 2023-05-31T20:52:13.920 DEBUG:boto:Data: 2023-05-31T20:52:13.920 DEBUG:boto:Headers: {} 2023-05-31T20:52:13.921 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:52:13.921 DEBUG:boto:Port: 8001 2023-05-31T20:52:13.922 DEBUG:boto:Params: {} 2023-05-31T20:52:13.922 DEBUG:boto:Token: None 2023-05-31T20:52:13.922 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:52:13 GMT /pklxkd-59/obj87?versionId=null 2023-05-31T20:52:13.923 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:+yxhuUyYXZCC6Bi9DlPpbunReIU= 2023-05-31T20:52:13.923 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:52:13 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:+yxhuUyYXZCC6Bi9DlPpbunReIU=', 'Content-Length': '0'} 2023-05-31T20:52:13.925 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:48:13 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a18ba145a5e5e17b-006477b37d-1c17-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:52:13 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:52:13.968 DEBUG:boto:path=/pklxkd-59/obj87 2023-05-31T20:52:13.969 DEBUG:boto:auth_path=/pklxkd-59/obj87 2023-05-31T20:52:13.969 DEBUG:boto:path=/pklxkd-59/obj87?versionId=null 2023-05-31T20:52:13.970 DEBUG:boto:auth_path=/pklxkd-59/obj87?versionId=null 2023-05-31T20:52:13.970 DEBUG:boto:Method: GET 2023-05-31T20:52:13.971 DEBUG:boto:Path: /pklxkd-59/obj87?versionId=null 2023-05-31T20:52:13.971 DEBUG:boto:Data: 2023-05-31T20:52:13.971 DEBUG:boto:Headers: {} 2023-05-31T20:52:13.972 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:52:13.972 DEBUG:boto:Port: 8000 2023-05-31T20:52:13.973 DEBUG:boto:Params: {} 2023-05-31T20:52:13.973 DEBUG:boto:Token: None 2023-05-31T20:52:13.974 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:52:13 GMT /pklxkd-59/obj87?versionId=null 2023-05-31T20:52:13.974 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:+yxhuUyYXZCC6Bi9DlPpbunReIU= 2023-05-31T20:52:13.975 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:52:13 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:+yxhuUyYXZCC6Bi9DlPpbunReIU=', 'Content-Length': '0'} 2023-05-31T20:52:13.977 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:48:13 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000009b97c9dba62e9a6-006477b37d-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:52:13 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:52:13.978 DEBUG:boto:path=/pklxkd-59/ 2023-05-31T20:52:13.978 DEBUG:boto:auth_path=/pklxkd-59/ 2023-05-31T20:52:13.978 DEBUG:boto:path=/pklxkd-59/?prefix=obj87 2023-05-31T20:52:13.979 DEBUG:boto:auth_path=/pklxkd-59/?prefix=obj87 2023-05-31T20:52:13.979 DEBUG:boto:Method: GET 2023-05-31T20:52:13.980 DEBUG:boto:Path: /pklxkd-59/?prefix=obj87 2023-05-31T20:52:13.980 DEBUG:boto:Data: 2023-05-31T20:52:13.980 DEBUG:boto:Headers: {} 2023-05-31T20:52:13.981 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:52:13.981 DEBUG:boto:Port: 8001 2023-05-31T20:52:13.981 DEBUG:boto:Params: {} 2023-05-31T20:52:13.982 DEBUG:boto:Token: None 2023-05-31T20:52:13.982 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:52:13 GMT /pklxkd-59/ 2023-05-31T20:52:13.983 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:2oppaLXJix1iBBaZf3oYtXHJCGM= 2023-05-31T20:52:13.983 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:52:13 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:2oppaLXJix1iBBaZf3oYtXHJCGM=', 'Content-Length': '0'} 2023-05-31T20:52:13.985 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000003a3f81e255d33204-006477b37d-1c17-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:52:13 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:52:13.986 DEBUG:boto:b'pklxkd-59obj871000falseobj872023-05-31T20:48:13.031Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:52:13.986 DEBUG:boto:path=/pklxkd-59/ 2023-05-31T20:52:13.987 DEBUG:boto:auth_path=/pklxkd-59/ 2023-05-31T20:52:13.987 DEBUG:boto:path=/pklxkd-59/?prefix=obj87 2023-05-31T20:52:13.988 DEBUG:boto:auth_path=/pklxkd-59/?prefix=obj87 2023-05-31T20:52:13.988 DEBUG:boto:Method: GET 2023-05-31T20:52:13.988 DEBUG:boto:Path: /pklxkd-59/?prefix=obj87 2023-05-31T20:52:13.989 DEBUG:boto:Data: 2023-05-31T20:52:13.989 DEBUG:boto:Headers: {} 2023-05-31T20:52:13.989 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:52:13.990 DEBUG:boto:Port: 8000 2023-05-31T20:52:13.990 DEBUG:boto:Params: {} 2023-05-31T20:52:13.990 DEBUG:boto:Token: None 2023-05-31T20:52:13.991 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:52:13 GMT /pklxkd-59/ 2023-05-31T20:52:13.991 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:2oppaLXJix1iBBaZf3oYtXHJCGM= 2023-05-31T20:52:13.992 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:52:13 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:2oppaLXJix1iBBaZf3oYtXHJCGM=', 'Content-Length': '0'} 2023-05-31T20:52:13.996 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000053dcc37635d7805f-006477b37d-1630-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:52:13 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:52:14.036 DEBUG:boto:b'pklxkd-59obj871000falseobj872023-05-31T20:48:13.031Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:52:14.037 DEBUG:boto:path=/pklxkd-59/obj87 2023-05-31T20:52:14.037 DEBUG:boto:auth_path=/pklxkd-59/obj87 2023-05-31T20:52:14.038 DEBUG:boto:Method: HEAD 2023-05-31T20:52:14.038 DEBUG:boto:Path: /pklxkd-59/obj87 2023-05-31T20:52:14.038 DEBUG:boto:Data: 2023-05-31T20:52:14.039 DEBUG:boto:Headers: {} 2023-05-31T20:52:14.039 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:52:14.039 DEBUG:boto:Port: 8001 2023-05-31T20:52:14.040 DEBUG:boto:Params: {} 2023-05-31T20:52:14.040 DEBUG:boto:Token: None 2023-05-31T20:52:14.041 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:52:14 GMT /pklxkd-59/obj87 2023-05-31T20:52:14.041 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:9HxyQKz9B59ZO84I5xQsHTUy0x4= 2023-05-31T20:52:14.041 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:52:14 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:9HxyQKz9B59ZO84I5xQsHTUy0x4=', 'Content-Length': '0'} 2023-05-31T20:52:14.043 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:48:13 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000555a0eb8419208fa-006477b37e-1c17-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:52:14 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:52:14.044 DEBUG:boto:path=/pklxkd-59/obj87 2023-05-31T20:52:14.044 DEBUG:boto:auth_path=/pklxkd-59/obj87 2023-05-31T20:52:14.045 DEBUG:boto:Method: HEAD 2023-05-31T20:52:14.045 DEBUG:boto:Path: /pklxkd-59/obj87 2023-05-31T20:52:14.045 DEBUG:boto:Data: 2023-05-31T20:52:14.046 DEBUG:boto:Headers: {} 2023-05-31T20:52:14.046 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:52:14.047 DEBUG:boto:Port: 8000 2023-05-31T20:52:14.047 DEBUG:boto:Params: {} 2023-05-31T20:52:14.047 DEBUG:boto:Token: None 2023-05-31T20:52:14.048 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:52:14 GMT /pklxkd-59/obj87 2023-05-31T20:52:14.048 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:9HxyQKz9B59ZO84I5xQsHTUy0x4= 2023-05-31T20:52:14.049 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:52:14 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:9HxyQKz9B59ZO84I5xQsHTUy0x4=', 'Content-Length': '0'} 2023-05-31T20:52:14.050 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:48:13 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000075140f171b86e1fb-006477b37e-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:52:14 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:52:14.051 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj87 2023-05-31T20:52:14.052 DEBUG:boto:path=/pklxkd-59/obj87 2023-05-31T20:52:14.052 DEBUG:boto:auth_path=/pklxkd-59/obj87 2023-05-31T20:52:14.053 DEBUG:boto:Method: GET 2023-05-31T20:52:14.053 DEBUG:boto:Path: /pklxkd-59/obj87 2023-05-31T20:52:14.053 DEBUG:boto:Data: 2023-05-31T20:52:14.054 DEBUG:boto:Headers: {} 2023-05-31T20:52:14.054 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:52:14.055 DEBUG:boto:Port: 8001 2023-05-31T20:52:14.055 DEBUG:boto:Params: {} 2023-05-31T20:52:14.056 DEBUG:boto:Token: None 2023-05-31T20:52:14.056 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:52:14 GMT /pklxkd-59/obj87 2023-05-31T20:52:14.057 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:m0qkbbZpSIWEQ+Om5PhUijtrBjg= 2023-05-31T20:52:14.057 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:52:14 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:m0qkbbZpSIWEQ+Om5PhUijtrBjg=', 'Content-Length': '0'} 2023-05-31T20:52:14.059 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:48:13 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007f142e540f8c6c85-006477b37e-1c17-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:52:14 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:52:14.059 DEBUG:boto:path=/pklxkd-59/obj87 2023-05-31T20:52:14.060 DEBUG:boto:auth_path=/pklxkd-59/obj87 2023-05-31T20:52:14.060 DEBUG:boto:Method: GET 2023-05-31T20:52:14.060 DEBUG:boto:Path: /pklxkd-59/obj87 2023-05-31T20:52:14.061 DEBUG:boto:Data: 2023-05-31T20:52:14.061 DEBUG:boto:Headers: {} 2023-05-31T20:52:14.062 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:52:14.062 DEBUG:boto:Port: 8000 2023-05-31T20:52:14.062 DEBUG:boto:Params: {} 2023-05-31T20:52:14.063 DEBUG:boto:Token: None 2023-05-31T20:52:14.063 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:52:14 GMT /pklxkd-59/obj87 2023-05-31T20:52:14.063 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:m0qkbbZpSIWEQ+Om5PhUijtrBjg= 2023-05-31T20:52:14.064 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:52:14 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:m0qkbbZpSIWEQ+Om5PhUijtrBjg=', 'Content-Length': '0'} 2023-05-31T20:52:14.066 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:48:13 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009574851053c84d1d-006477b37e-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:52:14 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:52:14.066 DEBUG:boto:path=/pklxkd-59/ 2023-05-31T20:52:14.067 DEBUG:boto:auth_path=/pklxkd-59/ 2023-05-31T20:52:14.067 DEBUG:boto:path=/pklxkd-59/?prefix=obj87 2023-05-31T20:52:14.067 DEBUG:boto:auth_path=/pklxkd-59/?prefix=obj87 2023-05-31T20:52:14.068 DEBUG:boto:Method: GET 2023-05-31T20:52:14.068 DEBUG:boto:Path: /pklxkd-59/?prefix=obj87 2023-05-31T20:52:14.068 DEBUG:boto:Data: 2023-05-31T20:52:14.069 DEBUG:boto:Headers: {} 2023-05-31T20:52:14.069 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:52:14.069 DEBUG:boto:Port: 8001 2023-05-31T20:52:14.070 DEBUG:boto:Params: {} 2023-05-31T20:52:14.070 DEBUG:boto:Token: None 2023-05-31T20:52:14.071 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:52:14 GMT /pklxkd-59/ 2023-05-31T20:52:14.071 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:mdO1s5Hqbhwig9TZcTjTTOcY57g= 2023-05-31T20:52:14.072 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:52:14 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:mdO1s5Hqbhwig9TZcTjTTOcY57g=', 'Content-Length': '0'} 2023-05-31T20:52:14.074 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000bc08da5c624f424d-006477b37e-1c17-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:52:14 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:52:14.074 DEBUG:boto:b'pklxkd-59obj871000falseobj872023-05-31T20:48:13.031Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:52:14.075 DEBUG:boto:path=/pklxkd-59/ 2023-05-31T20:52:14.076 DEBUG:boto:auth_path=/pklxkd-59/ 2023-05-31T20:52:14.076 DEBUG:boto:path=/pklxkd-59/?prefix=obj87 2023-05-31T20:52:14.076 DEBUG:boto:auth_path=/pklxkd-59/?prefix=obj87 2023-05-31T20:52:14.077 DEBUG:boto:Method: GET 2023-05-31T20:52:14.077 DEBUG:boto:Path: /pklxkd-59/?prefix=obj87 2023-05-31T20:52:14.078 DEBUG:boto:Data: 2023-05-31T20:52:14.078 DEBUG:boto:Headers: {} 2023-05-31T20:52:14.078 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:52:14.079 DEBUG:boto:Port: 8000 2023-05-31T20:52:14.079 DEBUG:boto:Params: {} 2023-05-31T20:52:14.079 DEBUG:boto:Token: None 2023-05-31T20:52:14.080 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:52:14 GMT /pklxkd-59/ 2023-05-31T20:52:14.080 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:mdO1s5Hqbhwig9TZcTjTTOcY57g= 2023-05-31T20:52:14.081 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:52:14 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:mdO1s5Hqbhwig9TZcTjTTOcY57g=', 'Content-Length': '0'} 2023-05-31T20:52:14.085 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000007fe8f80bb047a2ea-006477b37e-1630-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:52:14 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:52:14.085 DEBUG:boto:b'pklxkd-59obj871000falseobj872023-05-31T20:48:13.031Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:52:14.086 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj92 2023-05-31T20:52:14.086 DEBUG:boto:path=/pklxkd-59/obj92 2023-05-31T20:52:14.087 DEBUG:boto:auth_path=/pklxkd-59/obj92 2023-05-31T20:52:14.087 DEBUG:boto:path=/pklxkd-59/obj92?versionId=null 2023-05-31T20:52:14.087 DEBUG:boto:auth_path=/pklxkd-59/obj92?versionId=null 2023-05-31T20:52:14.088 DEBUG:boto:Method: GET 2023-05-31T20:52:14.088 DEBUG:boto:Path: /pklxkd-59/obj92?versionId=null 2023-05-31T20:52:14.088 DEBUG:boto:Data: 2023-05-31T20:52:14.089 DEBUG:boto:Headers: {} 2023-05-31T20:52:14.089 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:52:14.089 DEBUG:boto:Port: 8001 2023-05-31T20:52:14.090 DEBUG:boto:Params: {} 2023-05-31T20:52:14.090 DEBUG:boto:Token: None 2023-05-31T20:52:14.091 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:52:14 GMT /pklxkd-59/obj92?versionId=null 2023-05-31T20:52:14.091 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:roXA84R2EylH0ZYwcA391DiwR5s= 2023-05-31T20:52:14.091 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:52:14 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:roXA84R2EylH0ZYwcA391DiwR5s=', 'Content-Length': '0'} 2023-05-31T20:52:14.093 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:46:33 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e6f54f32ca737b22-006477b37e-1c17-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:52:14 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:52:14.094 DEBUG:boto:path=/pklxkd-59/obj92 2023-05-31T20:52:14.094 DEBUG:boto:auth_path=/pklxkd-59/obj92 2023-05-31T20:52:14.095 DEBUG:boto:path=/pklxkd-59/obj92?versionId=null 2023-05-31T20:52:14.095 DEBUG:boto:auth_path=/pklxkd-59/obj92?versionId=null 2023-05-31T20:52:14.095 DEBUG:boto:Method: GET 2023-05-31T20:52:14.096 DEBUG:boto:Path: /pklxkd-59/obj92?versionId=null 2023-05-31T20:52:14.096 DEBUG:boto:Data: 2023-05-31T20:52:14.096 DEBUG:boto:Headers: {} 2023-05-31T20:52:14.097 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:52:14.097 DEBUG:boto:Port: 8000 2023-05-31T20:52:14.097 DEBUG:boto:Params: {} 2023-05-31T20:52:14.098 DEBUG:boto:Token: None 2023-05-31T20:52:14.098 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:52:14 GMT /pklxkd-59/obj92?versionId=null 2023-05-31T20:52:14.099 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:roXA84R2EylH0ZYwcA391DiwR5s= 2023-05-31T20:52:14.099 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:52:14 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:roXA84R2EylH0ZYwcA391DiwR5s=', 'Content-Length': '0'} 2023-05-31T20:52:14.101 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:46:33 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000022cc443095f483f-006477b37e-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:52:14 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:52:14.102 DEBUG:boto:path=/pklxkd-59/obj92 2023-05-31T20:52:14.102 DEBUG:boto:auth_path=/pklxkd-59/obj92 2023-05-31T20:52:14.102 DEBUG:boto:path=/pklxkd-59/obj92?versionId=null 2023-05-31T20:52:14.103 DEBUG:boto:auth_path=/pklxkd-59/obj92?versionId=null 2023-05-31T20:52:14.103 DEBUG:boto:Method: HEAD 2023-05-31T20:52:14.104 DEBUG:boto:Path: /pklxkd-59/obj92?versionId=null 2023-05-31T20:52:14.104 DEBUG:boto:Data: 2023-05-31T20:52:14.104 DEBUG:boto:Headers: {} 2023-05-31T20:52:14.105 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:52:14.105 DEBUG:boto:Port: 8001 2023-05-31T20:52:14.105 DEBUG:boto:Params: {} 2023-05-31T20:52:14.106 DEBUG:boto:Token: None 2023-05-31T20:52:14.106 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:52:14 GMT /pklxkd-59/obj92?versionId=null 2023-05-31T20:52:14.107 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:VwvrbS2WkliBNJ1nUBnONT1wCNQ= 2023-05-31T20:52:14.107 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:52:14 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:VwvrbS2WkliBNJ1nUBnONT1wCNQ=', 'Content-Length': '0'} 2023-05-31T20:52:14.109 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:46:33 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000049a823f0d4ffc72a-006477b37e-1c17-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:52:14 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:52:14.110 DEBUG:boto:path=/pklxkd-59/obj92 2023-05-31T20:52:14.110 DEBUG:boto:auth_path=/pklxkd-59/obj92 2023-05-31T20:52:14.111 DEBUG:boto:path=/pklxkd-59/obj92?versionId=null 2023-05-31T20:52:14.111 DEBUG:boto:auth_path=/pklxkd-59/obj92?versionId=null 2023-05-31T20:52:14.111 DEBUG:boto:Method: HEAD 2023-05-31T20:52:14.112 DEBUG:boto:Path: /pklxkd-59/obj92?versionId=null 2023-05-31T20:52:14.112 DEBUG:boto:Data: 2023-05-31T20:52:14.112 DEBUG:boto:Headers: {} 2023-05-31T20:52:14.113 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:52:14.113 DEBUG:boto:Port: 8000 2023-05-31T20:52:14.113 DEBUG:boto:Params: {} 2023-05-31T20:52:14.114 DEBUG:boto:Token: None 2023-05-31T20:52:14.114 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:52:14 GMT /pklxkd-59/obj92?versionId=null 2023-05-31T20:52:14.115 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:VwvrbS2WkliBNJ1nUBnONT1wCNQ= 2023-05-31T20:52:14.115 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:52:14 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:VwvrbS2WkliBNJ1nUBnONT1wCNQ=', 'Content-Length': '0'} 2023-05-31T20:52:14.117 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:46:33 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006a622d39fd877621-006477b37e-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:52:14 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:52:14.117 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj92 2023-05-31T20:52:14.118 DEBUG:boto:path=/pklxkd-59/obj92 2023-05-31T20:52:14.118 DEBUG:boto:auth_path=/pklxkd-59/obj92 2023-05-31T20:52:14.118 DEBUG:boto:path=/pklxkd-59/obj92?versionId=null 2023-05-31T20:52:14.119 DEBUG:boto:auth_path=/pklxkd-59/obj92?versionId=null 2023-05-31T20:52:14.119 DEBUG:boto:Method: GET 2023-05-31T20:52:14.119 DEBUG:boto:Path: /pklxkd-59/obj92?versionId=null 2023-05-31T20:52:14.120 DEBUG:boto:Data: 2023-05-31T20:52:14.120 DEBUG:boto:Headers: {} 2023-05-31T20:52:14.121 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:52:14.121 DEBUG:boto:Port: 8001 2023-05-31T20:52:14.121 DEBUG:boto:Params: {} 2023-05-31T20:52:14.122 DEBUG:boto:Token: None 2023-05-31T20:52:14.122 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:52:14 GMT /pklxkd-59/obj92?versionId=null 2023-05-31T20:52:14.122 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:roXA84R2EylH0ZYwcA391DiwR5s= 2023-05-31T20:52:14.123 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:52:14 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:roXA84R2EylH0ZYwcA391DiwR5s=', 'Content-Length': '0'} 2023-05-31T20:52:14.125 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:46:33 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f7df3cc262a2e81c-006477b37e-1c17-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:52:14 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:52:14.125 DEBUG:boto:path=/pklxkd-59/obj92 2023-05-31T20:52:14.126 DEBUG:boto:auth_path=/pklxkd-59/obj92 2023-05-31T20:52:14.126 DEBUG:boto:path=/pklxkd-59/obj92?versionId=null 2023-05-31T20:52:14.126 DEBUG:boto:auth_path=/pklxkd-59/obj92?versionId=null 2023-05-31T20:52:14.127 DEBUG:boto:Method: GET 2023-05-31T20:52:14.127 DEBUG:boto:Path: /pklxkd-59/obj92?versionId=null 2023-05-31T20:52:14.128 DEBUG:boto:Data: 2023-05-31T20:52:14.128 DEBUG:boto:Headers: {} 2023-05-31T20:52:14.128 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:52:14.129 DEBUG:boto:Port: 8000 2023-05-31T20:52:14.129 DEBUG:boto:Params: {} 2023-05-31T20:52:14.130 DEBUG:boto:Token: None 2023-05-31T20:52:14.130 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:52:14 GMT /pklxkd-59/obj92?versionId=null 2023-05-31T20:52:14.131 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:roXA84R2EylH0ZYwcA391DiwR5s= 2023-05-31T20:52:14.131 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:52:14 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:roXA84R2EylH0ZYwcA391DiwR5s=', 'Content-Length': '0'} 2023-05-31T20:52:14.133 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:46:33 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000041482877cb922299-006477b37e-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:52:14 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:52:14.176 DEBUG:boto:path=/pklxkd-59/ 2023-05-31T20:52:14.177 DEBUG:boto:auth_path=/pklxkd-59/ 2023-05-31T20:52:14.177 DEBUG:boto:path=/pklxkd-59/?prefix=obj92 2023-05-31T20:52:14.177 DEBUG:boto:auth_path=/pklxkd-59/?prefix=obj92 2023-05-31T20:52:14.178 DEBUG:boto:Method: GET 2023-05-31T20:52:14.178 DEBUG:boto:Path: /pklxkd-59/?prefix=obj92 2023-05-31T20:52:14.179 DEBUG:boto:Data: 2023-05-31T20:52:14.179 DEBUG:boto:Headers: {} 2023-05-31T20:52:14.179 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:52:14.180 DEBUG:boto:Port: 8001 2023-05-31T20:52:14.180 DEBUG:boto:Params: {} 2023-05-31T20:52:14.181 DEBUG:boto:Token: None 2023-05-31T20:52:14.181 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:52:14 GMT /pklxkd-59/ 2023-05-31T20:52:14.182 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:mdO1s5Hqbhwig9TZcTjTTOcY57g= 2023-05-31T20:52:14.182 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:52:14 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:mdO1s5Hqbhwig9TZcTjTTOcY57g=', 'Content-Length': '0'} 2023-05-31T20:52:14.184 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000003605af878677ce60-006477b37e-1c17-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:52:14 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:52:14.185 DEBUG:boto:b'pklxkd-59obj921000falseobj922023-05-31T20:46:33.904Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:52:14.186 DEBUG:boto:path=/pklxkd-59/ 2023-05-31T20:52:14.186 DEBUG:boto:auth_path=/pklxkd-59/ 2023-05-31T20:52:14.186 DEBUG:boto:path=/pklxkd-59/?prefix=obj92 2023-05-31T20:52:14.187 DEBUG:boto:auth_path=/pklxkd-59/?prefix=obj92 2023-05-31T20:52:14.187 DEBUG:boto:Method: GET 2023-05-31T20:52:14.188 DEBUG:boto:Path: /pklxkd-59/?prefix=obj92 2023-05-31T20:52:14.188 DEBUG:boto:Data: 2023-05-31T20:52:14.188 DEBUG:boto:Headers: {} 2023-05-31T20:52:14.189 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:52:14.189 DEBUG:boto:Port: 8000 2023-05-31T20:52:14.190 DEBUG:boto:Params: {} 2023-05-31T20:52:14.190 DEBUG:boto:Token: None 2023-05-31T20:52:14.191 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:52:14 GMT /pklxkd-59/ 2023-05-31T20:52:14.191 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:mdO1s5Hqbhwig9TZcTjTTOcY57g= 2023-05-31T20:52:14.192 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:52:14 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:mdO1s5Hqbhwig9TZcTjTTOcY57g=', 'Content-Length': '0'} 2023-05-31T20:52:14.196 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000106b591b2da0e54b-006477b37e-1630-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:52:14 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:52:14.196 DEBUG:boto:b'pklxkd-59obj921000falseobj922023-05-31T20:46:33.904Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:52:14.197 DEBUG:boto:path=/pklxkd-59/obj92 2023-05-31T20:52:14.197 DEBUG:boto:auth_path=/pklxkd-59/obj92 2023-05-31T20:52:14.198 DEBUG:boto:Method: HEAD 2023-05-31T20:52:14.198 DEBUG:boto:Path: /pklxkd-59/obj92 2023-05-31T20:52:14.198 DEBUG:boto:Data: 2023-05-31T20:52:14.199 DEBUG:boto:Headers: {} 2023-05-31T20:52:14.199 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:52:14.199 DEBUG:boto:Port: 8001 2023-05-31T20:52:14.200 DEBUG:boto:Params: {} 2023-05-31T20:52:14.200 DEBUG:boto:Token: None 2023-05-31T20:52:14.200 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:52:14 GMT /pklxkd-59/obj92 2023-05-31T20:52:14.201 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:X4QVzjl3qMi49FOCCFfIRsXOWPk= 2023-05-31T20:52:14.201 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:52:14 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:X4QVzjl3qMi49FOCCFfIRsXOWPk=', 'Content-Length': '0'} 2023-05-31T20:52:14.203 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:46:33 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007a7323efd0adf938-006477b37e-1c17-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:52:14 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:52:14.204 DEBUG:boto:path=/pklxkd-59/obj92 2023-05-31T20:52:14.204 DEBUG:boto:auth_path=/pklxkd-59/obj92 2023-05-31T20:52:14.204 DEBUG:boto:Method: HEAD 2023-05-31T20:52:14.205 DEBUG:boto:Path: /pklxkd-59/obj92 2023-05-31T20:52:14.205 DEBUG:boto:Data: 2023-05-31T20:52:14.205 DEBUG:boto:Headers: {} 2023-05-31T20:52:14.206 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:52:14.206 DEBUG:boto:Port: 8000 2023-05-31T20:52:14.206 DEBUG:boto:Params: {} 2023-05-31T20:52:14.207 DEBUG:boto:Token: None 2023-05-31T20:52:14.207 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:52:14 GMT /pklxkd-59/obj92 2023-05-31T20:52:14.208 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:X4QVzjl3qMi49FOCCFfIRsXOWPk= 2023-05-31T20:52:14.208 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:52:14 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:X4QVzjl3qMi49FOCCFfIRsXOWPk=', 'Content-Length': '0'} 2023-05-31T20:52:14.210 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:46:33 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f818e3bdf3bfdf8a-006477b37e-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:52:14 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:52:14.210 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj92 2023-05-31T20:52:14.211 DEBUG:boto:path=/pklxkd-59/obj92 2023-05-31T20:52:14.211 DEBUG:boto:auth_path=/pklxkd-59/obj92 2023-05-31T20:52:14.212 DEBUG:boto:Method: GET 2023-05-31T20:52:14.212 DEBUG:boto:Path: /pklxkd-59/obj92 2023-05-31T20:52:14.213 DEBUG:boto:Data: 2023-05-31T20:52:14.213 DEBUG:boto:Headers: {} 2023-05-31T20:52:14.213 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:52:14.214 DEBUG:boto:Port: 8001 2023-05-31T20:52:14.214 DEBUG:boto:Params: {} 2023-05-31T20:52:14.215 DEBUG:boto:Token: None 2023-05-31T20:52:14.215 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:52:14 GMT /pklxkd-59/obj92 2023-05-31T20:52:14.216 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:8KXevJ3I8xC3TFWP8E/R9FpJrKg= 2023-05-31T20:52:14.216 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:52:14 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:8KXevJ3I8xC3TFWP8E/R9FpJrKg=', 'Content-Length': '0'} 2023-05-31T20:52:14.218 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:46:33 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000000ebafc8df14ac0c-006477b37e-1c17-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:52:14 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:52:14.219 DEBUG:boto:path=/pklxkd-59/obj92 2023-05-31T20:52:14.219 DEBUG:boto:auth_path=/pklxkd-59/obj92 2023-05-31T20:52:14.220 DEBUG:boto:Method: GET 2023-05-31T20:52:14.220 DEBUG:boto:Path: /pklxkd-59/obj92 2023-05-31T20:52:14.221 DEBUG:boto:Data: 2023-05-31T20:52:14.221 DEBUG:boto:Headers: {} 2023-05-31T20:52:14.221 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:52:14.222 DEBUG:boto:Port: 8000 2023-05-31T20:52:14.222 DEBUG:boto:Params: {} 2023-05-31T20:52:14.222 DEBUG:boto:Token: None 2023-05-31T20:52:14.223 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:52:14 GMT /pklxkd-59/obj92 2023-05-31T20:52:14.223 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:8KXevJ3I8xC3TFWP8E/R9FpJrKg= 2023-05-31T20:52:14.224 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:52:14 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:8KXevJ3I8xC3TFWP8E/R9FpJrKg=', 'Content-Length': '0'} 2023-05-31T20:52:14.225 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:46:33 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000cd09cd1348c9634b-006477b37e-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:52:14 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:52:14.226 DEBUG:boto:path=/pklxkd-59/ 2023-05-31T20:52:14.226 DEBUG:boto:auth_path=/pklxkd-59/ 2023-05-31T20:52:14.227 DEBUG:boto:path=/pklxkd-59/?prefix=obj92 2023-05-31T20:52:14.227 DEBUG:boto:auth_path=/pklxkd-59/?prefix=obj92 2023-05-31T20:52:14.228 DEBUG:boto:Method: GET 2023-05-31T20:52:14.228 DEBUG:boto:Path: /pklxkd-59/?prefix=obj92 2023-05-31T20:52:14.228 DEBUG:boto:Data: 2023-05-31T20:52:14.229 DEBUG:boto:Headers: {} 2023-05-31T20:52:14.229 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:52:14.229 DEBUG:boto:Port: 8001 2023-05-31T20:52:14.230 DEBUG:boto:Params: {} 2023-05-31T20:52:14.230 DEBUG:boto:Token: None 2023-05-31T20:52:14.230 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:52:14 GMT /pklxkd-59/ 2023-05-31T20:52:14.231 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:mdO1s5Hqbhwig9TZcTjTTOcY57g= 2023-05-31T20:52:14.231 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:52:14 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:mdO1s5Hqbhwig9TZcTjTTOcY57g=', 'Content-Length': '0'} 2023-05-31T20:52:14.234 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000004219d547d39c8790-006477b37e-1c17-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:52:14 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:52:14.234 DEBUG:boto:b'pklxkd-59obj921000falseobj922023-05-31T20:46:33.904Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:52:14.235 DEBUG:boto:path=/pklxkd-59/ 2023-05-31T20:52:14.235 DEBUG:boto:auth_path=/pklxkd-59/ 2023-05-31T20:52:14.236 DEBUG:boto:path=/pklxkd-59/?prefix=obj92 2023-05-31T20:52:14.236 DEBUG:boto:auth_path=/pklxkd-59/?prefix=obj92 2023-05-31T20:52:14.237 DEBUG:boto:Method: GET 2023-05-31T20:52:14.237 DEBUG:boto:Path: /pklxkd-59/?prefix=obj92 2023-05-31T20:52:14.237 DEBUG:boto:Data: 2023-05-31T20:52:14.238 DEBUG:boto:Headers: {} 2023-05-31T20:52:14.238 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:52:14.239 DEBUG:boto:Port: 8000 2023-05-31T20:52:14.239 DEBUG:boto:Params: {} 2023-05-31T20:52:14.239 DEBUG:boto:Token: None 2023-05-31T20:52:14.240 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:52:14 GMT /pklxkd-59/ 2023-05-31T20:52:14.240 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:mdO1s5Hqbhwig9TZcTjTTOcY57g= 2023-05-31T20:52:14.241 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:52:14 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:mdO1s5Hqbhwig9TZcTjTTOcY57g=', 'Content-Length': '0'} 2023-05-31T20:52:14.245 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000bfa3e7b544f5acba-006477b37e-1630-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:52:14 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:52:14.245 DEBUG:boto:b'pklxkd-59obj921000falseobj922023-05-31T20:46:33.904Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:52:14.246 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj93 2023-05-31T20:52:14.246 DEBUG:boto:path=/pklxkd-59/obj93 2023-05-31T20:52:14.247 DEBUG:boto:auth_path=/pklxkd-59/obj93 2023-05-31T20:52:14.247 DEBUG:boto:path=/pklxkd-59/obj93?versionId=null 2023-05-31T20:52:14.248 DEBUG:boto:auth_path=/pklxkd-59/obj93?versionId=null 2023-05-31T20:52:14.248 DEBUG:boto:Method: GET 2023-05-31T20:52:14.249 DEBUG:boto:Path: /pklxkd-59/obj93?versionId=null 2023-05-31T20:52:14.249 DEBUG:boto:Data: 2023-05-31T20:52:14.250 DEBUG:boto:Headers: {} 2023-05-31T20:52:14.250 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:52:14.250 DEBUG:boto:Port: 8001 2023-05-31T20:52:14.251 DEBUG:boto:Params: {} 2023-05-31T20:52:14.251 DEBUG:boto:Token: None 2023-05-31T20:52:14.252 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:52:14 GMT /pklxkd-59/obj93?versionId=null 2023-05-31T20:52:14.252 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:h05FJFiOpoKB+FFa+exAXpG0Pqo= 2023-05-31T20:52:14.253 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:52:14 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:h05FJFiOpoKB+FFa+exAXpG0Pqo=', 'Content-Length': '0'} 2023-05-31T20:52:14.255 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:50:33 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008099157d8241be3a-006477b37e-1c17-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:52:14 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:52:14.296 DEBUG:boto:path=/pklxkd-59/obj93 2023-05-31T20:52:14.297 DEBUG:boto:auth_path=/pklxkd-59/obj93 2023-05-31T20:52:14.297 DEBUG:boto:path=/pklxkd-59/obj93?versionId=null 2023-05-31T20:52:14.298 DEBUG:boto:auth_path=/pklxkd-59/obj93?versionId=null 2023-05-31T20:52:14.298 DEBUG:boto:Method: GET 2023-05-31T20:52:14.298 DEBUG:boto:Path: /pklxkd-59/obj93?versionId=null 2023-05-31T20:52:14.299 DEBUG:boto:Data: 2023-05-31T20:52:14.299 DEBUG:boto:Headers: {} 2023-05-31T20:52:14.299 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:52:14.300 DEBUG:boto:Port: 8000 2023-05-31T20:52:14.300 DEBUG:boto:Params: {} 2023-05-31T20:52:14.300 DEBUG:boto:Token: None 2023-05-31T20:52:14.301 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:52:14 GMT /pklxkd-59/obj93?versionId=null 2023-05-31T20:52:14.302 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:h05FJFiOpoKB+FFa+exAXpG0Pqo= 2023-05-31T20:52:14.302 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:52:14 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:h05FJFiOpoKB+FFa+exAXpG0Pqo=', 'Content-Length': '0'} 2023-05-31T20:52:14.304 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:50:33 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000092437449cc03dcf6-006477b37e-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:52:14 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:52:14.305 DEBUG:boto:path=/pklxkd-59/obj93 2023-05-31T20:52:14.305 DEBUG:boto:auth_path=/pklxkd-59/obj93 2023-05-31T20:52:14.306 DEBUG:boto:path=/pklxkd-59/obj93?versionId=null 2023-05-31T20:52:14.306 DEBUG:boto:auth_path=/pklxkd-59/obj93?versionId=null 2023-05-31T20:52:14.306 DEBUG:boto:Method: HEAD 2023-05-31T20:52:14.307 DEBUG:boto:Path: /pklxkd-59/obj93?versionId=null 2023-05-31T20:52:14.307 DEBUG:boto:Data: 2023-05-31T20:52:14.307 DEBUG:boto:Headers: {} 2023-05-31T20:52:14.308 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:52:14.308 DEBUG:boto:Port: 8001 2023-05-31T20:52:14.308 DEBUG:boto:Params: {} 2023-05-31T20:52:14.309 DEBUG:boto:Token: None 2023-05-31T20:52:14.309 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:52:14 GMT /pklxkd-59/obj93?versionId=null 2023-05-31T20:52:14.310 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:nnwJaiTWMG8Fl+q7tVsFu5XiyPc= 2023-05-31T20:52:14.310 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:52:14 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:nnwJaiTWMG8Fl+q7tVsFu5XiyPc=', 'Content-Length': '0'} 2023-05-31T20:52:14.312 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:50:33 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000debda4b8ee1fbd70-006477b37e-1c17-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:52:14 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:52:14.312 DEBUG:boto:path=/pklxkd-59/obj93 2023-05-31T20:52:14.313 DEBUG:boto:auth_path=/pklxkd-59/obj93 2023-05-31T20:52:14.313 DEBUG:boto:path=/pklxkd-59/obj93?versionId=null 2023-05-31T20:52:14.313 DEBUG:boto:auth_path=/pklxkd-59/obj93?versionId=null 2023-05-31T20:52:14.314 DEBUG:boto:Method: HEAD 2023-05-31T20:52:14.314 DEBUG:boto:Path: /pklxkd-59/obj93?versionId=null 2023-05-31T20:52:14.314 DEBUG:boto:Data: 2023-05-31T20:52:14.315 DEBUG:boto:Headers: {} 2023-05-31T20:52:14.315 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:52:14.315 DEBUG:boto:Port: 8000 2023-05-31T20:52:14.316 DEBUG:boto:Params: {} 2023-05-31T20:52:14.316 DEBUG:boto:Token: None 2023-05-31T20:52:14.317 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:52:14 GMT /pklxkd-59/obj93?versionId=null 2023-05-31T20:52:14.317 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:nnwJaiTWMG8Fl+q7tVsFu5XiyPc= 2023-05-31T20:52:14.317 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:52:14 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:nnwJaiTWMG8Fl+q7tVsFu5XiyPc=', 'Content-Length': '0'} 2023-05-31T20:52:14.319 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:50:33 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000fa129d915006d334-006477b37e-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:52:14 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:52:14.320 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj93 2023-05-31T20:52:14.320 DEBUG:boto:path=/pklxkd-59/obj93 2023-05-31T20:52:14.321 DEBUG:boto:auth_path=/pklxkd-59/obj93 2023-05-31T20:52:14.321 DEBUG:boto:path=/pklxkd-59/obj93?versionId=null 2023-05-31T20:52:14.322 DEBUG:boto:auth_path=/pklxkd-59/obj93?versionId=null 2023-05-31T20:52:14.322 DEBUG:boto:Method: GET 2023-05-31T20:52:14.323 DEBUG:boto:Path: /pklxkd-59/obj93?versionId=null 2023-05-31T20:52:14.323 DEBUG:boto:Data: 2023-05-31T20:52:14.323 DEBUG:boto:Headers: {} 2023-05-31T20:52:14.324 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:52:14.324 DEBUG:boto:Port: 8001 2023-05-31T20:52:14.325 DEBUG:boto:Params: {} 2023-05-31T20:52:14.325 DEBUG:boto:Token: None 2023-05-31T20:52:14.326 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:52:14 GMT /pklxkd-59/obj93?versionId=null 2023-05-31T20:52:14.326 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:h05FJFiOpoKB+FFa+exAXpG0Pqo= 2023-05-31T20:52:14.327 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:52:14 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:h05FJFiOpoKB+FFa+exAXpG0Pqo=', 'Content-Length': '0'} 2023-05-31T20:52:14.329 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:50:33 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008a58ca88ed51000d-006477b37e-1c17-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:52:14 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:52:14.329 DEBUG:boto:path=/pklxkd-59/obj93 2023-05-31T20:52:14.330 DEBUG:boto:auth_path=/pklxkd-59/obj93 2023-05-31T20:52:14.330 DEBUG:boto:path=/pklxkd-59/obj93?versionId=null 2023-05-31T20:52:14.331 DEBUG:boto:auth_path=/pklxkd-59/obj93?versionId=null 2023-05-31T20:52:14.331 DEBUG:boto:Method: GET 2023-05-31T20:52:14.331 DEBUG:boto:Path: /pklxkd-59/obj93?versionId=null 2023-05-31T20:52:14.332 DEBUG:boto:Data: 2023-05-31T20:52:14.332 DEBUG:boto:Headers: {} 2023-05-31T20:52:14.332 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:52:14.333 DEBUG:boto:Port: 8000 2023-05-31T20:52:14.333 DEBUG:boto:Params: {} 2023-05-31T20:52:14.333 DEBUG:boto:Token: None 2023-05-31T20:52:14.334 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:52:14 GMT /pklxkd-59/obj93?versionId=null 2023-05-31T20:52:14.334 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:h05FJFiOpoKB+FFa+exAXpG0Pqo= 2023-05-31T20:52:14.335 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:52:14 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:h05FJFiOpoKB+FFa+exAXpG0Pqo=', 'Content-Length': '0'} 2023-05-31T20:52:14.336 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:50:33 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000bb4417ebf9ef7b7e-006477b37e-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:52:14 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:52:14.337 DEBUG:boto:path=/pklxkd-59/ 2023-05-31T20:52:14.338 DEBUG:boto:auth_path=/pklxkd-59/ 2023-05-31T20:52:14.338 DEBUG:boto:path=/pklxkd-59/?prefix=obj93 2023-05-31T20:52:14.338 DEBUG:boto:auth_path=/pklxkd-59/?prefix=obj93 2023-05-31T20:52:14.339 DEBUG:boto:Method: GET 2023-05-31T20:52:14.339 DEBUG:boto:Path: /pklxkd-59/?prefix=obj93 2023-05-31T20:52:14.339 DEBUG:boto:Data: 2023-05-31T20:52:14.340 DEBUG:boto:Headers: {} 2023-05-31T20:52:14.340 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:52:14.340 DEBUG:boto:Port: 8001 2023-05-31T20:52:14.341 DEBUG:boto:Params: {} 2023-05-31T20:52:14.341 DEBUG:boto:Token: None 2023-05-31T20:52:14.342 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:52:14 GMT /pklxkd-59/ 2023-05-31T20:52:14.342 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:mdO1s5Hqbhwig9TZcTjTTOcY57g= 2023-05-31T20:52:14.342 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:52:14 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:mdO1s5Hqbhwig9TZcTjTTOcY57g=', 'Content-Length': '0'} 2023-05-31T20:52:14.345 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000000faf047217ddf0b0-006477b37e-1c17-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:52:14 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:52:14.345 DEBUG:boto:b'pklxkd-59obj931000falseobj932023-05-31T20:50:33.533Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:52:14.346 DEBUG:boto:path=/pklxkd-59/ 2023-05-31T20:52:14.346 DEBUG:boto:auth_path=/pklxkd-59/ 2023-05-31T20:52:14.347 DEBUG:boto:path=/pklxkd-59/?prefix=obj93 2023-05-31T20:52:14.347 DEBUG:boto:auth_path=/pklxkd-59/?prefix=obj93 2023-05-31T20:52:14.348 DEBUG:boto:Method: GET 2023-05-31T20:52:14.348 DEBUG:boto:Path: /pklxkd-59/?prefix=obj93 2023-05-31T20:52:14.349 DEBUG:boto:Data: 2023-05-31T20:52:14.349 DEBUG:boto:Headers: {} 2023-05-31T20:52:14.349 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:52:14.350 DEBUG:boto:Port: 8000 2023-05-31T20:52:14.350 DEBUG:boto:Params: {} 2023-05-31T20:52:14.351 DEBUG:boto:Token: None 2023-05-31T20:52:14.351 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:52:14 GMT /pklxkd-59/ 2023-05-31T20:52:14.352 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:mdO1s5Hqbhwig9TZcTjTTOcY57g= 2023-05-31T20:52:14.352 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:52:14 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:mdO1s5Hqbhwig9TZcTjTTOcY57g=', 'Content-Length': '0'} 2023-05-31T20:52:14.356 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e04a41e15530e63c-006477b37e-1630-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:52:14 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:52:14.400 DEBUG:boto:b'pklxkd-59obj931000falseobj932023-05-31T20:50:33.533Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:52:14.401 DEBUG:boto:path=/pklxkd-59/obj93 2023-05-31T20:52:14.401 DEBUG:boto:auth_path=/pklxkd-59/obj93 2023-05-31T20:52:14.402 DEBUG:boto:Method: HEAD 2023-05-31T20:52:14.402 DEBUG:boto:Path: /pklxkd-59/obj93 2023-05-31T20:52:14.403 DEBUG:boto:Data: 2023-05-31T20:52:14.403 DEBUG:boto:Headers: {} 2023-05-31T20:52:14.403 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:52:14.404 DEBUG:boto:Port: 8001 2023-05-31T20:52:14.404 DEBUG:boto:Params: {} 2023-05-31T20:52:14.405 DEBUG:boto:Token: None 2023-05-31T20:52:14.405 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:52:14 GMT /pklxkd-59/obj93 2023-05-31T20:52:14.406 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:xcIrjJKFgrIGBWnkrAr3rdCId1k= 2023-05-31T20:52:14.406 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:52:14 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:xcIrjJKFgrIGBWnkrAr3rdCId1k=', 'Content-Length': '0'} 2023-05-31T20:52:14.408 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:50:33 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001909b831b6112fc9-006477b37e-1c17-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:52:14 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:52:14.409 DEBUG:boto:path=/pklxkd-59/obj93 2023-05-31T20:52:14.409 DEBUG:boto:auth_path=/pklxkd-59/obj93 2023-05-31T20:52:14.410 DEBUG:boto:Method: HEAD 2023-05-31T20:52:14.410 DEBUG:boto:Path: /pklxkd-59/obj93 2023-05-31T20:52:14.410 DEBUG:boto:Data: 2023-05-31T20:52:14.411 DEBUG:boto:Headers: {} 2023-05-31T20:52:14.411 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:52:14.411 DEBUG:boto:Port: 8000 2023-05-31T20:52:14.412 DEBUG:boto:Params: {} 2023-05-31T20:52:14.412 DEBUG:boto:Token: None 2023-05-31T20:52:14.413 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:52:14 GMT /pklxkd-59/obj93 2023-05-31T20:52:14.413 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:xcIrjJKFgrIGBWnkrAr3rdCId1k= 2023-05-31T20:52:14.413 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:52:14 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:xcIrjJKFgrIGBWnkrAr3rdCId1k=', 'Content-Length': '0'} 2023-05-31T20:52:14.415 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:50:33 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000046ca667608f14f51-006477b37e-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:52:14 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:52:14.416 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj93 2023-05-31T20:52:14.416 DEBUG:boto:path=/pklxkd-59/obj93 2023-05-31T20:52:14.420 DEBUG:boto:auth_path=/pklxkd-59/obj93 2023-05-31T20:52:14.420 DEBUG:boto:Method: GET 2023-05-31T20:52:14.421 DEBUG:boto:Path: /pklxkd-59/obj93 2023-05-31T20:52:14.421 DEBUG:boto:Data: 2023-05-31T20:52:14.422 DEBUG:boto:Headers: {} 2023-05-31T20:52:14.422 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:52:14.422 DEBUG:boto:Port: 8001 2023-05-31T20:52:14.423 DEBUG:boto:Params: {} 2023-05-31T20:52:14.423 DEBUG:boto:Token: None 2023-05-31T20:52:14.423 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:52:14 GMT /pklxkd-59/obj93 2023-05-31T20:52:14.424 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:gKTaQ4utFHnMFk8H6vxph8J7SUk= 2023-05-31T20:52:14.424 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:52:14 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:gKTaQ4utFHnMFk8H6vxph8J7SUk=', 'Content-Length': '0'} 2023-05-31T20:52:14.426 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:50:33 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000020d941802b5213fd-006477b37e-1c17-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:52:14 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:52:14.468 DEBUG:boto:path=/pklxkd-59/obj93 2023-05-31T20:52:14.469 DEBUG:boto:auth_path=/pklxkd-59/obj93 2023-05-31T20:52:14.469 DEBUG:boto:Method: GET 2023-05-31T20:52:14.470 DEBUG:boto:Path: /pklxkd-59/obj93 2023-05-31T20:52:14.470 DEBUG:boto:Data: 2023-05-31T20:52:14.470 DEBUG:boto:Headers: {} 2023-05-31T20:52:14.471 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:52:14.471 DEBUG:boto:Port: 8000 2023-05-31T20:52:14.471 DEBUG:boto:Params: {} 2023-05-31T20:52:14.472 DEBUG:boto:Token: None 2023-05-31T20:52:14.472 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:52:14 GMT /pklxkd-59/obj93 2023-05-31T20:52:14.473 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:gKTaQ4utFHnMFk8H6vxph8J7SUk= 2023-05-31T20:52:14.473 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:52:14 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:gKTaQ4utFHnMFk8H6vxph8J7SUk=', 'Content-Length': '0'} 2023-05-31T20:52:14.475 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:50:33 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000de1826228e82636b-006477b37e-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:52:14 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:52:14.476 DEBUG:boto:path=/pklxkd-59/ 2023-05-31T20:52:14.476 DEBUG:boto:auth_path=/pklxkd-59/ 2023-05-31T20:52:14.476 DEBUG:boto:path=/pklxkd-59/?prefix=obj93 2023-05-31T20:52:14.477 DEBUG:boto:auth_path=/pklxkd-59/?prefix=obj93 2023-05-31T20:52:14.477 DEBUG:boto:Method: GET 2023-05-31T20:52:14.477 DEBUG:boto:Path: /pklxkd-59/?prefix=obj93 2023-05-31T20:52:14.478 DEBUG:boto:Data: 2023-05-31T20:52:14.478 DEBUG:boto:Headers: {} 2023-05-31T20:52:14.479 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:52:14.479 DEBUG:boto:Port: 8001 2023-05-31T20:52:14.479 DEBUG:boto:Params: {} 2023-05-31T20:52:14.480 DEBUG:boto:Token: None 2023-05-31T20:52:14.480 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:52:14 GMT /pklxkd-59/ 2023-05-31T20:52:14.480 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:mdO1s5Hqbhwig9TZcTjTTOcY57g= 2023-05-31T20:52:14.481 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:52:14 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:mdO1s5Hqbhwig9TZcTjTTOcY57g=', 'Content-Length': '0'} 2023-05-31T20:52:14.483 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a621d7d3d73c553b-006477b37e-1c17-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:52:14 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:52:14.484 DEBUG:boto:b'pklxkd-59obj931000falseobj932023-05-31T20:50:33.533Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:52:14.485 DEBUG:boto:path=/pklxkd-59/ 2023-05-31T20:52:14.485 DEBUG:boto:auth_path=/pklxkd-59/ 2023-05-31T20:52:14.486 DEBUG:boto:path=/pklxkd-59/?prefix=obj93 2023-05-31T20:52:14.486 DEBUG:boto:auth_path=/pklxkd-59/?prefix=obj93 2023-05-31T20:52:14.486 DEBUG:boto:Method: GET 2023-05-31T20:52:14.487 DEBUG:boto:Path: /pklxkd-59/?prefix=obj93 2023-05-31T20:52:14.487 DEBUG:boto:Data: 2023-05-31T20:52:14.487 DEBUG:boto:Headers: {} 2023-05-31T20:52:14.488 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:52:14.488 DEBUG:boto:Port: 8000 2023-05-31T20:52:14.488 DEBUG:boto:Params: {} 2023-05-31T20:52:14.489 DEBUG:boto:Token: None 2023-05-31T20:52:14.489 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:52:14 GMT /pklxkd-59/ 2023-05-31T20:52:14.490 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:mdO1s5Hqbhwig9TZcTjTTOcY57g= 2023-05-31T20:52:14.490 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:52:14 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:mdO1s5Hqbhwig9TZcTjTTOcY57g=', 'Content-Length': '0'} 2023-05-31T20:52:14.494 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000004a66c8d01a7ce2d-006477b37e-1630-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:52:14 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:52:14.494 DEBUG:boto:b'pklxkd-59obj931000falseobj932023-05-31T20:50:33.533Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:52:14.495 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj95 2023-05-31T20:52:14.495 DEBUG:boto:path=/pklxkd-59/obj95 2023-05-31T20:52:14.496 DEBUG:boto:auth_path=/pklxkd-59/obj95 2023-05-31T20:52:14.496 DEBUG:boto:path=/pklxkd-59/obj95?versionId=null 2023-05-31T20:52:14.497 DEBUG:boto:auth_path=/pklxkd-59/obj95?versionId=null 2023-05-31T20:52:14.497 DEBUG:boto:Method: GET 2023-05-31T20:52:14.498 DEBUG:boto:Path: /pklxkd-59/obj95?versionId=null 2023-05-31T20:52:14.498 DEBUG:boto:Data: 2023-05-31T20:52:14.499 DEBUG:boto:Headers: {} 2023-05-31T20:52:14.499 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:52:14.499 DEBUG:boto:Port: 8001 2023-05-31T20:52:14.500 DEBUG:boto:Params: {} 2023-05-31T20:52:14.500 DEBUG:boto:Token: None 2023-05-31T20:52:14.500 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:52:14 GMT /pklxkd-59/obj95?versionId=null 2023-05-31T20:52:14.501 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:rpT6x3COkA0DbvYCvUjbetUr4Eo= 2023-05-31T20:52:14.501 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:52:14 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:rpT6x3COkA0DbvYCvUjbetUr4Eo=', 'Content-Length': '0'} 2023-05-31T20:52:14.503 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:44:50 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000cef8a3148f560636-006477b37e-1c17-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:52:14 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:52:14.504 DEBUG:boto:path=/pklxkd-59/obj95 2023-05-31T20:52:14.504 DEBUG:boto:auth_path=/pklxkd-59/obj95 2023-05-31T20:52:14.505 DEBUG:boto:path=/pklxkd-59/obj95?versionId=null 2023-05-31T20:52:14.505 DEBUG:boto:auth_path=/pklxkd-59/obj95?versionId=null 2023-05-31T20:52:14.506 DEBUG:boto:Method: GET 2023-05-31T20:52:14.506 DEBUG:boto:Path: /pklxkd-59/obj95?versionId=null 2023-05-31T20:52:14.506 DEBUG:boto:Data: 2023-05-31T20:52:14.507 DEBUG:boto:Headers: {} 2023-05-31T20:52:14.507 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:52:14.507 DEBUG:boto:Port: 8000 2023-05-31T20:52:14.508 DEBUG:boto:Params: {} 2023-05-31T20:52:14.508 DEBUG:boto:Token: None 2023-05-31T20:52:14.508 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:52:14 GMT /pklxkd-59/obj95?versionId=null 2023-05-31T20:52:14.509 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:rpT6x3COkA0DbvYCvUjbetUr4Eo= 2023-05-31T20:52:14.509 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:52:14 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:rpT6x3COkA0DbvYCvUjbetUr4Eo=', 'Content-Length': '0'} 2023-05-31T20:52:14.511 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:44:50 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000009f5c971a7219b2c-006477b37e-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:52:14 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:52:14.512 DEBUG:boto:path=/pklxkd-59/obj95 2023-05-31T20:52:14.512 DEBUG:boto:auth_path=/pklxkd-59/obj95 2023-05-31T20:52:14.512 DEBUG:boto:path=/pklxkd-59/obj95?versionId=null 2023-05-31T20:52:14.513 DEBUG:boto:auth_path=/pklxkd-59/obj95?versionId=null 2023-05-31T20:52:14.513 DEBUG:boto:Method: HEAD 2023-05-31T20:52:14.513 DEBUG:boto:Path: /pklxkd-59/obj95?versionId=null 2023-05-31T20:52:14.514 DEBUG:boto:Data: 2023-05-31T20:52:14.514 DEBUG:boto:Headers: {} 2023-05-31T20:52:14.515 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:52:14.515 DEBUG:boto:Port: 8001 2023-05-31T20:52:14.515 DEBUG:boto:Params: {} 2023-05-31T20:52:14.516 DEBUG:boto:Token: None 2023-05-31T20:52:14.516 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:52:14 GMT /pklxkd-59/obj95?versionId=null 2023-05-31T20:52:14.517 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:xZtRY2GHb7x19/EwS7bE6xJtJZo= 2023-05-31T20:52:14.517 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:52:14 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:xZtRY2GHb7x19/EwS7bE6xJtJZo=', 'Content-Length': '0'} 2023-05-31T20:52:14.519 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:44:50 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000043027deca529b731-006477b37e-1c17-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:52:14 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:52:14.520 DEBUG:boto:path=/pklxkd-59/obj95 2023-05-31T20:52:14.520 DEBUG:boto:auth_path=/pklxkd-59/obj95 2023-05-31T20:52:14.521 DEBUG:boto:path=/pklxkd-59/obj95?versionId=null 2023-05-31T20:52:14.521 DEBUG:boto:auth_path=/pklxkd-59/obj95?versionId=null 2023-05-31T20:52:14.522 DEBUG:boto:Method: HEAD 2023-05-31T20:52:14.522 DEBUG:boto:Path: /pklxkd-59/obj95?versionId=null 2023-05-31T20:52:14.522 DEBUG:boto:Data: 2023-05-31T20:52:14.523 DEBUG:boto:Headers: {} 2023-05-31T20:52:14.523 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:52:14.524 DEBUG:boto:Port: 8000 2023-05-31T20:52:14.524 DEBUG:boto:Params: {} 2023-05-31T20:52:14.524 DEBUG:boto:Token: None 2023-05-31T20:52:14.525 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:52:14 GMT /pklxkd-59/obj95?versionId=null 2023-05-31T20:52:14.525 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:xZtRY2GHb7x19/EwS7bE6xJtJZo= 2023-05-31T20:52:14.525 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:52:14 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:xZtRY2GHb7x19/EwS7bE6xJtJZo=', 'Content-Length': '0'} 2023-05-31T20:52:14.527 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:44:50 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000da59e4891df1dcb5-006477b37e-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:52:14 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:52:14.528 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj95 2023-05-31T20:52:14.528 DEBUG:boto:path=/pklxkd-59/obj95 2023-05-31T20:52:14.528 DEBUG:boto:auth_path=/pklxkd-59/obj95 2023-05-31T20:52:14.529 DEBUG:boto:path=/pklxkd-59/obj95?versionId=null 2023-05-31T20:52:14.529 DEBUG:boto:auth_path=/pklxkd-59/obj95?versionId=null 2023-05-31T20:52:14.529 DEBUG:boto:Method: GET 2023-05-31T20:52:14.530 DEBUG:boto:Path: /pklxkd-59/obj95?versionId=null 2023-05-31T20:52:14.530 DEBUG:boto:Data: 2023-05-31T20:52:14.530 DEBUG:boto:Headers: {} 2023-05-31T20:52:14.531 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:52:14.531 DEBUG:boto:Port: 8001 2023-05-31T20:52:14.531 DEBUG:boto:Params: {} 2023-05-31T20:52:14.532 DEBUG:boto:Token: None 2023-05-31T20:52:14.532 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:52:14 GMT /pklxkd-59/obj95?versionId=null 2023-05-31T20:52:14.533 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:rpT6x3COkA0DbvYCvUjbetUr4Eo= 2023-05-31T20:52:14.533 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:52:14 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:rpT6x3COkA0DbvYCvUjbetUr4Eo=', 'Content-Length': '0'} 2023-05-31T20:52:14.535 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:44:50 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ae3c2e79a9305b87-006477b37e-1c17-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:52:14 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:52:14.536 DEBUG:boto:path=/pklxkd-59/obj95 2023-05-31T20:52:14.536 DEBUG:boto:auth_path=/pklxkd-59/obj95 2023-05-31T20:52:14.536 DEBUG:boto:path=/pklxkd-59/obj95?versionId=null 2023-05-31T20:52:14.537 DEBUG:boto:auth_path=/pklxkd-59/obj95?versionId=null 2023-05-31T20:52:14.537 DEBUG:boto:Method: GET 2023-05-31T20:52:14.537 DEBUG:boto:Path: /pklxkd-59/obj95?versionId=null 2023-05-31T20:52:14.538 DEBUG:boto:Data: 2023-05-31T20:52:14.538 DEBUG:boto:Headers: {} 2023-05-31T20:52:14.539 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:52:14.539 DEBUG:boto:Port: 8000 2023-05-31T20:52:14.539 DEBUG:boto:Params: {} 2023-05-31T20:52:14.540 DEBUG:boto:Token: None 2023-05-31T20:52:14.540 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:52:14 GMT /pklxkd-59/obj95?versionId=null 2023-05-31T20:52:14.541 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:rpT6x3COkA0DbvYCvUjbetUr4Eo= 2023-05-31T20:52:14.541 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:52:14 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:rpT6x3COkA0DbvYCvUjbetUr4Eo=', 'Content-Length': '0'} 2023-05-31T20:52:14.543 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:44:50 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008f01c7f83fda13ea-006477b37e-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:52:14 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:52:14.544 DEBUG:boto:path=/pklxkd-59/ 2023-05-31T20:52:14.544 DEBUG:boto:auth_path=/pklxkd-59/ 2023-05-31T20:52:14.544 DEBUG:boto:path=/pklxkd-59/?prefix=obj95 2023-05-31T20:52:14.545 DEBUG:boto:auth_path=/pklxkd-59/?prefix=obj95 2023-05-31T20:52:14.545 DEBUG:boto:Method: GET 2023-05-31T20:52:14.546 DEBUG:boto:Path: /pklxkd-59/?prefix=obj95 2023-05-31T20:52:14.546 DEBUG:boto:Data: 2023-05-31T20:52:14.546 DEBUG:boto:Headers: {} 2023-05-31T20:52:14.547 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:52:14.547 DEBUG:boto:Port: 8001 2023-05-31T20:52:14.548 DEBUG:boto:Params: {} 2023-05-31T20:52:14.548 DEBUG:boto:Token: None 2023-05-31T20:52:14.549 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:52:14 GMT /pklxkd-59/ 2023-05-31T20:52:14.549 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:mdO1s5Hqbhwig9TZcTjTTOcY57g= 2023-05-31T20:52:14.550 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:52:14 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:mdO1s5Hqbhwig9TZcTjTTOcY57g=', 'Content-Length': '0'} 2023-05-31T20:52:14.552 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c05802e5a2ef9c9f-006477b37e-1c17-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:52:14 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:52:14.553 DEBUG:boto:b'pklxkd-59obj951000falseobj952023-05-31T20:44:50.705Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:52:14.553 DEBUG:boto:path=/pklxkd-59/ 2023-05-31T20:52:14.554 DEBUG:boto:auth_path=/pklxkd-59/ 2023-05-31T20:52:14.554 DEBUG:boto:path=/pklxkd-59/?prefix=obj95 2023-05-31T20:52:14.554 DEBUG:boto:auth_path=/pklxkd-59/?prefix=obj95 2023-05-31T20:52:14.555 DEBUG:boto:Method: GET 2023-05-31T20:52:14.555 DEBUG:boto:Path: /pklxkd-59/?prefix=obj95 2023-05-31T20:52:14.556 DEBUG:boto:Data: 2023-05-31T20:52:14.556 DEBUG:boto:Headers: {} 2023-05-31T20:52:14.556 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:52:14.557 DEBUG:boto:Port: 8000 2023-05-31T20:52:14.557 DEBUG:boto:Params: {} 2023-05-31T20:52:14.557 DEBUG:boto:Token: None 2023-05-31T20:52:14.558 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:52:14 GMT /pklxkd-59/ 2023-05-31T20:52:14.558 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:mdO1s5Hqbhwig9TZcTjTTOcY57g= 2023-05-31T20:52:14.559 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:52:14 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:mdO1s5Hqbhwig9TZcTjTTOcY57g=', 'Content-Length': '0'} 2023-05-31T20:52:14.562 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000187cb33963c295e3-006477b37e-1630-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:52:14 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:52:14.562 DEBUG:boto:b'pklxkd-59obj951000falseobj952023-05-31T20:44:50.705Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:52:14.563 DEBUG:boto:path=/pklxkd-59/obj95 2023-05-31T20:52:14.563 DEBUG:boto:auth_path=/pklxkd-59/obj95 2023-05-31T20:52:14.564 DEBUG:boto:Method: HEAD 2023-05-31T20:52:14.564 DEBUG:boto:Path: /pklxkd-59/obj95 2023-05-31T20:52:14.564 DEBUG:boto:Data: 2023-05-31T20:52:14.565 DEBUG:boto:Headers: {} 2023-05-31T20:52:14.565 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:52:14.565 DEBUG:boto:Port: 8001 2023-05-31T20:52:14.566 DEBUG:boto:Params: {} 2023-05-31T20:52:14.566 DEBUG:boto:Token: None 2023-05-31T20:52:14.566 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:52:14 GMT /pklxkd-59/obj95 2023-05-31T20:52:14.567 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:5JV7YscLvp1v5HWlU5bJSLtYhCg= 2023-05-31T20:52:14.567 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:52:14 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:5JV7YscLvp1v5HWlU5bJSLtYhCg=', 'Content-Length': '0'} 2023-05-31T20:52:14.569 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:44:50 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005cf52b30d3f9f51c-006477b37e-1c17-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:52:14 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:52:14.569 DEBUG:boto:path=/pklxkd-59/obj95 2023-05-31T20:52:14.570 DEBUG:boto:auth_path=/pklxkd-59/obj95 2023-05-31T20:52:14.570 DEBUG:boto:Method: HEAD 2023-05-31T20:52:14.570 DEBUG:boto:Path: /pklxkd-59/obj95 2023-05-31T20:52:14.571 DEBUG:boto:Data: 2023-05-31T20:52:14.571 DEBUG:boto:Headers: {} 2023-05-31T20:52:14.572 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:52:14.572 DEBUG:boto:Port: 8000 2023-05-31T20:52:14.572 DEBUG:boto:Params: {} 2023-05-31T20:52:14.573 DEBUG:boto:Token: None 2023-05-31T20:52:14.573 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:52:14 GMT /pklxkd-59/obj95 2023-05-31T20:52:14.574 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:5JV7YscLvp1v5HWlU5bJSLtYhCg= 2023-05-31T20:52:14.574 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:52:14 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:5JV7YscLvp1v5HWlU5bJSLtYhCg=', 'Content-Length': '0'} 2023-05-31T20:52:14.576 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:44:50 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e20b3ca6974c75a0-006477b37e-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:52:14 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:52:14.577 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj95 2023-05-31T20:52:14.577 DEBUG:boto:path=/pklxkd-59/obj95 2023-05-31T20:52:14.578 DEBUG:boto:auth_path=/pklxkd-59/obj95 2023-05-31T20:52:14.578 DEBUG:boto:Method: GET 2023-05-31T20:52:14.578 DEBUG:boto:Path: /pklxkd-59/obj95 2023-05-31T20:52:14.579 DEBUG:boto:Data: 2023-05-31T20:52:14.579 DEBUG:boto:Headers: {} 2023-05-31T20:52:14.580 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:52:14.580 DEBUG:boto:Port: 8001 2023-05-31T20:52:14.580 DEBUG:boto:Params: {} 2023-05-31T20:52:14.581 DEBUG:boto:Token: None 2023-05-31T20:52:14.581 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:52:14 GMT /pklxkd-59/obj95 2023-05-31T20:52:14.582 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:xERyk3187jzpkuIB0G3+TBhSLro= 2023-05-31T20:52:14.582 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:52:14 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:xERyk3187jzpkuIB0G3+TBhSLro=', 'Content-Length': '0'} 2023-05-31T20:52:14.584 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:44:50 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000056063d690f3bda2f-006477b37e-1c17-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:52:14 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:52:14.624 DEBUG:boto:path=/pklxkd-59/obj95 2023-05-31T20:52:14.625 DEBUG:boto:auth_path=/pklxkd-59/obj95 2023-05-31T20:52:14.625 DEBUG:boto:Method: GET 2023-05-31T20:52:14.626 DEBUG:boto:Path: /pklxkd-59/obj95 2023-05-31T20:52:14.626 DEBUG:boto:Data: 2023-05-31T20:52:14.627 DEBUG:boto:Headers: {} 2023-05-31T20:52:14.627 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:52:14.627 DEBUG:boto:Port: 8000 2023-05-31T20:52:14.628 DEBUG:boto:Params: {} 2023-05-31T20:52:14.628 DEBUG:boto:Token: None 2023-05-31T20:52:14.629 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:52:14 GMT /pklxkd-59/obj95 2023-05-31T20:52:14.630 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:xERyk3187jzpkuIB0G3+TBhSLro= 2023-05-31T20:52:14.630 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:52:14 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:xERyk3187jzpkuIB0G3+TBhSLro=', 'Content-Length': '0'} 2023-05-31T20:52:14.632 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:44:50 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008ad3192a26e81e14-006477b37e-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:52:14 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:52:14.633 DEBUG:boto:path=/pklxkd-59/ 2023-05-31T20:52:14.634 DEBUG:boto:auth_path=/pklxkd-59/ 2023-05-31T20:52:14.634 DEBUG:boto:path=/pklxkd-59/?prefix=obj95 2023-05-31T20:52:14.634 DEBUG:boto:auth_path=/pklxkd-59/?prefix=obj95 2023-05-31T20:52:14.635 DEBUG:boto:Method: GET 2023-05-31T20:52:14.635 DEBUG:boto:Path: /pklxkd-59/?prefix=obj95 2023-05-31T20:52:14.635 DEBUG:boto:Data: 2023-05-31T20:52:14.636 DEBUG:boto:Headers: {} 2023-05-31T20:52:14.636 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:52:14.636 DEBUG:boto:Port: 8001 2023-05-31T20:52:14.637 DEBUG:boto:Params: {} 2023-05-31T20:52:14.637 DEBUG:boto:Token: None 2023-05-31T20:52:14.638 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:52:14 GMT /pklxkd-59/ 2023-05-31T20:52:14.638 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:mdO1s5Hqbhwig9TZcTjTTOcY57g= 2023-05-31T20:52:14.639 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:52:14 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:mdO1s5Hqbhwig9TZcTjTTOcY57g=', 'Content-Length': '0'} 2023-05-31T20:52:14.641 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000008858e415e66b87c2-006477b37e-1c17-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:52:14 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:52:14.641 DEBUG:boto:b'pklxkd-59obj951000falseobj952023-05-31T20:44:50.705Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:52:14.642 DEBUG:boto:path=/pklxkd-59/ 2023-05-31T20:52:14.642 DEBUG:boto:auth_path=/pklxkd-59/ 2023-05-31T20:52:14.643 DEBUG:boto:path=/pklxkd-59/?prefix=obj95 2023-05-31T20:52:14.643 DEBUG:boto:auth_path=/pklxkd-59/?prefix=obj95 2023-05-31T20:52:14.643 DEBUG:boto:Method: GET 2023-05-31T20:52:14.644 DEBUG:boto:Path: /pklxkd-59/?prefix=obj95 2023-05-31T20:52:14.644 DEBUG:boto:Data: 2023-05-31T20:52:14.645 DEBUG:boto:Headers: {} 2023-05-31T20:52:14.645 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:52:14.645 DEBUG:boto:Port: 8000 2023-05-31T20:52:14.646 DEBUG:boto:Params: {} 2023-05-31T20:52:14.646 DEBUG:boto:Token: None 2023-05-31T20:52:14.646 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:52:14 GMT /pklxkd-59/ 2023-05-31T20:52:14.647 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:mdO1s5Hqbhwig9TZcTjTTOcY57g= 2023-05-31T20:52:14.647 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:52:14 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:mdO1s5Hqbhwig9TZcTjTTOcY57g=', 'Content-Length': '0'} 2023-05-31T20:52:14.651 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ad34736de83d66a6-006477b37e-1630-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:52:14 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:52:14.651 DEBUG:boto:b'pklxkd-59obj951000falseobj952023-05-31T20:44:50.705Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:52:14.652 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=pklxkd-59 zones={a2, a1} 2023-05-31T20:52:14.653 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'reshard', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'pklxkd-59', '--num-shards', '37', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-05-31T20:52:14.653 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'reshard', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'pklxkd-59', '--num-shards', '37', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-05-31T20:52:14.654 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket reshard --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --bucket pklxkd-59 --num-shards 37 --yes-i-really-mean-it --cluster c1 --debug-rgw 0 --debug-ms 0 2023-05-31T20:52:14.714 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2023-05-31T20:52:14.714 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2023-05-31T20:52:14.775 INFO:teuthology.orchestra.run.smithi099.stdout:tenant: 2023-05-31T20:52:14.776 INFO:teuthology.orchestra.run.smithi099.stdout:bucket name: pklxkd-59 2023-05-31T20:52:14.787 INFO:teuthology.orchestra.run.smithi099.stdout:total entries: 30 2023-05-31T20:52:14.897 DEBUG:boto:path=/pklxkd-59/ 2023-05-31T20:52:14.898 DEBUG:boto:auth_path=/pklxkd-59/ 2023-05-31T20:52:14.898 DEBUG:boto:Method: HEAD 2023-05-31T20:52:14.898 DEBUG:boto:Path: /pklxkd-59/ 2023-05-31T20:52:14.899 DEBUG:boto:Data: 2023-05-31T20:52:14.899 DEBUG:boto:Headers: {} 2023-05-31T20:52:14.900 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:52:14.900 DEBUG:boto:Port: 8000 2023-05-31T20:52:14.900 DEBUG:boto:Params: {} 2023-05-31T20:52:14.901 DEBUG:boto:Token: None 2023-05-31T20:52:14.901 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:52:14 GMT /pklxkd-59/ 2023-05-31T20:52:14.902 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:KkIlsnH7WD/BhGZvqp5wE3EKLMo= 2023-05-31T20:52:14.902 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:52:14 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:KkIlsnH7WD/BhGZvqp5wE3EKLMo=', 'Content-Length': '0'} 2023-05-31T20:52:14.906 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '30'), ('X-RGW-Bytes-Used', '90'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000007b2cc0e901a8c4f0-006477b37e-1630-a1'), ('Content-Length', '0'), ('Date', 'Wed, 31 May 2023 20:52:14 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:52:14.907 DEBUG:boto:path=/pklxkd-59/obj37 2023-05-31T20:52:14.907 DEBUG:boto:auth_path=/pklxkd-59/obj37 2023-05-31T20:52:14.907 DEBUG:boto:Method: PUT 2023-05-31T20:52:14.908 DEBUG:boto:Path: /pklxkd-59/obj37 2023-05-31T20:52:14.908 DEBUG:boto:Data: 2023-05-31T20:52:14.908 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-05-31T20:52:14.909 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:52:14.909 DEBUG:boto:Port: 8000 2023-05-31T20:52:14.909 DEBUG:boto:Params: {} 2023-05-31T20:52:14.910 DEBUG:boto:Token: None 2023-05-31T20:52:14.910 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Wed, 31 May 2023 20:52:14 GMT /pklxkd-59/obj37 2023-05-31T20:52:14.911 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:1I2RpCmE3LxTFsVkIIInA1PuvvY= 2023-05-31T20:52:14.911 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Wed, 31 May 2023 20:52:14 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:1I2RpCmE3LxTFsVkIIInA1PuvvY='} 2023-05-31T20:52:14.922 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000c689a84fd4eeb19d-006477b37e-1630-a1'), ('Date', 'Wed, 31 May 2023 20:52:14 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:52:14.923 DEBUG:boto:path=/pklxkd-59/ 2023-05-31T20:52:14.923 DEBUG:boto:auth_path=/pklxkd-59/ 2023-05-31T20:52:14.924 DEBUG:boto:Method: HEAD 2023-05-31T20:52:14.924 DEBUG:boto:Path: /pklxkd-59/ 2023-05-31T20:52:14.924 DEBUG:boto:Data: 2023-05-31T20:52:14.925 DEBUG:boto:Headers: {} 2023-05-31T20:52:14.925 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:52:14.926 DEBUG:boto:Port: 8000 2023-05-31T20:52:14.926 DEBUG:boto:Params: {} 2023-05-31T20:52:14.926 DEBUG:boto:Token: None 2023-05-31T20:52:14.927 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:52:14 GMT /pklxkd-59/ 2023-05-31T20:52:14.927 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:KkIlsnH7WD/BhGZvqp5wE3EKLMo= 2023-05-31T20:52:14.928 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:52:14 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:KkIlsnH7WD/BhGZvqp5wE3EKLMo=', 'Content-Length': '0'} 2023-05-31T20:52:14.931 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '31'), ('X-RGW-Bytes-Used', '93'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000c74eacf45f423774-006477b37e-1630-a1'), ('Content-Length', '0'), ('Date', 'Wed, 31 May 2023 20:52:14 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:52:14.932 DEBUG:boto:path=/pklxkd-59/obj74 2023-05-31T20:52:14.932 DEBUG:boto:auth_path=/pklxkd-59/obj74 2023-05-31T20:52:14.932 DEBUG:boto:Method: PUT 2023-05-31T20:52:14.933 DEBUG:boto:Path: /pklxkd-59/obj74 2023-05-31T20:52:14.933 DEBUG:boto:Data: 2023-05-31T20:52:14.933 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-05-31T20:52:14.934 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:52:14.934 DEBUG:boto:Port: 8000 2023-05-31T20:52:14.934 DEBUG:boto:Params: {} 2023-05-31T20:52:14.935 DEBUG:boto:Token: None 2023-05-31T20:52:14.935 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Wed, 31 May 2023 20:52:14 GMT /pklxkd-59/obj74 2023-05-31T20:52:14.936 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:GWxXG/4RLwSF1wA0rMAXGDEZGLM= 2023-05-31T20:52:14.936 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Wed, 31 May 2023 20:52:14 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:GWxXG/4RLwSF1wA0rMAXGDEZGLM='} 2023-05-31T20:52:14.980 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000007caf256caac960a5-006477b37e-1630-a1'), ('Date', 'Wed, 31 May 2023 20:52:14 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:52:14.981 DEBUG:boto:path=/pklxkd-59/ 2023-05-31T20:52:14.981 DEBUG:boto:auth_path=/pklxkd-59/ 2023-05-31T20:52:14.982 DEBUG:boto:Method: HEAD 2023-05-31T20:52:14.982 DEBUG:boto:Path: /pklxkd-59/ 2023-05-31T20:52:14.982 DEBUG:boto:Data: 2023-05-31T20:52:14.983 DEBUG:boto:Headers: {} 2023-05-31T20:52:14.983 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:52:14.983 DEBUG:boto:Port: 8000 2023-05-31T20:52:14.984 DEBUG:boto:Params: {} 2023-05-31T20:52:14.984 DEBUG:boto:Token: None 2023-05-31T20:52:14.985 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:52:14 GMT /pklxkd-59/ 2023-05-31T20:52:14.985 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:KkIlsnH7WD/BhGZvqp5wE3EKLMo= 2023-05-31T20:52:14.986 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:52:14 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:KkIlsnH7WD/BhGZvqp5wE3EKLMo=', 'Content-Length': '0'} 2023-05-31T20:52:14.989 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '32'), ('X-RGW-Bytes-Used', '96'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000f4de70859be4a465-006477b37e-1630-a1'), ('Content-Length', '0'), ('Date', 'Wed, 31 May 2023 20:52:14 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:52:14.990 DEBUG:boto:path=/pklxkd-59/obj111 2023-05-31T20:52:14.990 DEBUG:boto:auth_path=/pklxkd-59/obj111 2023-05-31T20:52:14.990 DEBUG:boto:Method: PUT 2023-05-31T20:52:14.991 DEBUG:boto:Path: /pklxkd-59/obj111 2023-05-31T20:52:14.991 DEBUG:boto:Data: 2023-05-31T20:52:14.992 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-05-31T20:52:14.992 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:52:14.992 DEBUG:boto:Port: 8000 2023-05-31T20:52:14.993 DEBUG:boto:Params: {} 2023-05-31T20:52:14.993 DEBUG:boto:Token: None 2023-05-31T20:52:14.994 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Wed, 31 May 2023 20:52:14 GMT /pklxkd-59/obj111 2023-05-31T20:52:14.994 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:pYDPo5wSVwQG0MIPK4KNC0Rjmos= 2023-05-31T20:52:14.995 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Wed, 31 May 2023 20:52:14 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:pYDPo5wSVwQG0MIPK4KNC0Rjmos='} 2023-05-31T20:52:15.003 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000007eda4c7b603f188f-006477b37e-1630-a1'), ('Date', 'Wed, 31 May 2023 20:52:15 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:52:15.004 DEBUG:boto:path=/pklxkd-59/ 2023-05-31T20:52:15.004 DEBUG:boto:auth_path=/pklxkd-59/ 2023-05-31T20:52:15.005 DEBUG:boto:Method: HEAD 2023-05-31T20:52:15.005 DEBUG:boto:Path: /pklxkd-59/ 2023-05-31T20:52:15.005 DEBUG:boto:Data: 2023-05-31T20:52:15.006 DEBUG:boto:Headers: {} 2023-05-31T20:52:15.006 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:52:15.006 DEBUG:boto:Port: 8000 2023-05-31T20:52:15.007 DEBUG:boto:Params: {} 2023-05-31T20:52:15.007 DEBUG:boto:Token: None 2023-05-31T20:52:15.008 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:52:15 GMT /pklxkd-59/ 2023-05-31T20:52:15.008 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:hod8tbBBeFryk3f04q3+GNjl0Us= 2023-05-31T20:52:15.008 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:52:15 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:hod8tbBBeFryk3f04q3+GNjl0Us=', 'Content-Length': '0'} 2023-05-31T20:52:15.012 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '33'), ('X-RGW-Bytes-Used', '99'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000eb3077fbc96c2493-006477b37f-1630-a1'), ('Content-Length', '0'), ('Date', 'Wed, 31 May 2023 20:52:15 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:52:15.012 DEBUG:boto:path=/pklxkd-59/obj148 2023-05-31T20:52:15.012 DEBUG:boto:auth_path=/pklxkd-59/obj148 2023-05-31T20:52:15.013 DEBUG:boto:Method: PUT 2023-05-31T20:52:15.013 DEBUG:boto:Path: /pklxkd-59/obj148 2023-05-31T20:52:15.014 DEBUG:boto:Data: 2023-05-31T20:52:15.014 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-05-31T20:52:15.014 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:52:15.015 DEBUG:boto:Port: 8000 2023-05-31T20:52:15.015 DEBUG:boto:Params: {} 2023-05-31T20:52:15.015 DEBUG:boto:Token: None 2023-05-31T20:52:15.016 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Wed, 31 May 2023 20:52:15 GMT /pklxkd-59/obj148 2023-05-31T20:52:15.016 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:Tovl9nQkgs6m/tjFV9mmH+w0aQs= 2023-05-31T20:52:15.016 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Wed, 31 May 2023 20:52:15 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:Tovl9nQkgs6m/tjFV9mmH+w0aQs='} 2023-05-31T20:52:15.060 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000009c0f68077eb25449-006477b37f-1630-a1'), ('Date', 'Wed, 31 May 2023 20:52:15 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:52:15.061 DEBUG:boto:path=/pklxkd-59/ 2023-05-31T20:52:15.061 DEBUG:boto:auth_path=/pklxkd-59/ 2023-05-31T20:52:15.062 DEBUG:boto:Method: HEAD 2023-05-31T20:52:15.062 DEBUG:boto:Path: /pklxkd-59/ 2023-05-31T20:52:15.062 DEBUG:boto:Data: 2023-05-31T20:52:15.063 DEBUG:boto:Headers: {} 2023-05-31T20:52:15.063 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:52:15.063 DEBUG:boto:Port: 8000 2023-05-31T20:52:15.064 DEBUG:boto:Params: {} 2023-05-31T20:52:15.064 DEBUG:boto:Token: None 2023-05-31T20:52:15.064 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:52:15 GMT /pklxkd-59/ 2023-05-31T20:52:15.065 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:hod8tbBBeFryk3f04q3+GNjl0Us= 2023-05-31T20:52:15.065 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:52:15 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:hod8tbBBeFryk3f04q3+GNjl0Us=', 'Content-Length': '0'} 2023-05-31T20:52:15.068 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '34'), ('X-RGW-Bytes-Used', '102'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000dc337f0180e3ed4a-006477b37f-1630-a1'), ('Content-Length', '0'), ('Date', 'Wed, 31 May 2023 20:52:15 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:52:15.069 DEBUG:boto:path=/pklxkd-59/obj185 2023-05-31T20:52:15.070 DEBUG:boto:auth_path=/pklxkd-59/obj185 2023-05-31T20:52:15.070 DEBUG:boto:Method: PUT 2023-05-31T20:52:15.070 DEBUG:boto:Path: /pklxkd-59/obj185 2023-05-31T20:52:15.071 DEBUG:boto:Data: 2023-05-31T20:52:15.071 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-05-31T20:52:15.071 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:52:15.072 DEBUG:boto:Port: 8000 2023-05-31T20:52:15.072 DEBUG:boto:Params: {} 2023-05-31T20:52:15.072 DEBUG:boto:Token: None 2023-05-31T20:52:15.073 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Wed, 31 May 2023 20:52:15 GMT /pklxkd-59/obj185 2023-05-31T20:52:15.073 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:cAa+L3xMyffVDjRbrzvNzwQ3m20= 2023-05-31T20:52:15.074 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Wed, 31 May 2023 20:52:15 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:cAa+L3xMyffVDjRbrzvNzwQ3m20='} 2023-05-31T20:52:15.084 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000092629d0c46e12ccf-006477b37f-1630-a1'), ('Date', 'Wed, 31 May 2023 20:52:15 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:52:15.085 DEBUG:rgw_multi.tests:bucket checkpoint: source=a1 target=a2 bucket=pklxkd-59 2023-05-31T20:52:15.085 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'pklxkd-59', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-05-31T20:52:15.086 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'pklxkd-59', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-05-31T20:52:15.086 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket pklxkd-59 --source-zone a1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2023-05-31T20:52:15.142 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2023-05-31T20:52:15.143 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2023-05-31T20:52:15.194 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:52:15.191+0000 7f60f262c780 1 waiting to reach latest gen 6, on 5.. 2023-05-31T20:52:20.195 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:52:20.191+0000 7f60f262c780 1 waiting to reach latest gen 6, on 5.. 2023-05-31T20:52:25.196 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:52:25.192+0000 7f60f262c780 1 waiting to reach latest gen 6, on 5.. 2023-05-31T20:52:30.199 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:52:30.195+0000 7f60f262c780 1 waiting for incremental sync to catch up: 2023-05-31T20:52:30.199 INFO:teuthology.orchestra.run.smithi099.stderr: local status: [, , , , , , , , , , , , 00000000002.68.6, , , , , , , , , , , , , , , , , , , , 00000000001.75.6, , , , 00000000001.83.6] 2023-05-31T20:52:30.200 INFO:teuthology.orchestra.run.smithi099.stderr: remote markers: [, , , , , , 00000000002.48.6, , , , , , 00000000002.68.6, , , , , 00000000002.63.6, , , , , , , , , , , , , , , 00000000001.75.6, , , , 00000000001.83.6] 2023-05-31T20:52:35.202 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:52:35.197+0000 7f60f262c780 1 waiting for incremental sync to catch up: 2023-05-31T20:52:35.202 INFO:teuthology.orchestra.run.smithi099.stderr: local status: [, , , , , , , , , , , , 00000000002.68.6, , , , , , , , , , , , , , , , , , , , 00000000001.75.6, , , , 00000000001.83.6] 2023-05-31T20:52:35.202 INFO:teuthology.orchestra.run.smithi099.stderr: remote markers: [, , , , , , 00000000002.48.6, , , , , , 00000000002.68.6, , , , , 00000000002.63.6, , , , , , , , , , , , , , , 00000000001.75.6, , , , 00000000001.83.6] 2023-05-31T20:52:40.204 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:52:40.199+0000 7f60f262c780 1 waiting for incremental sync to catch up: 2023-05-31T20:52:40.204 INFO:teuthology.orchestra.run.smithi099.stderr: local status: [, , , , , , , , , , , , 00000000002.68.6, , , , , , , , , , , , , , , , , , , , 00000000001.75.6, , , , 00000000001.83.6] 2023-05-31T20:52:40.204 INFO:teuthology.orchestra.run.smithi099.stderr: remote markers: [, , , , , , 00000000002.48.6, , , , , , 00000000002.68.6, , , , , 00000000002.63.6, , , , , , , , , , , , , , , 00000000001.75.6, , , , 00000000001.83.6] 2023-05-31T20:52:45.206 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:52:45.202+0000 7f60f262c780 1 waiting for incremental sync to catch up: 2023-05-31T20:52:45.207 INFO:teuthology.orchestra.run.smithi099.stderr: local status: [, , , , , , , , , , , , 00000000002.68.6, , , , , , , , , , , , , , , , , , , , 00000000001.75.6, , , , 00000000001.83.6] 2023-05-31T20:52:45.207 INFO:teuthology.orchestra.run.smithi099.stderr: remote markers: [, , , , , , 00000000002.48.6, , , , , , 00000000002.68.6, , , , , 00000000002.63.6, , , , , , , , , , , , , , , 00000000001.75.6, , , , 00000000001.83.6] 2023-05-31T20:52:50.209 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:52:50.204+0000 7f60f262c780 1 waiting for incremental sync to catch up: 2023-05-31T20:52:50.210 INFO:teuthology.orchestra.run.smithi099.stderr: local status: [, , , , , , 00000000002.48.6, , , , , , 00000000002.68.6, , , , , , , , , , , , , , , , , , , , 00000000001.75.6, , , , 00000000001.83.6] 2023-05-31T20:52:50.210 INFO:teuthology.orchestra.run.smithi099.stderr: remote markers: [, , , , , , 00000000002.48.6, , , , , , 00000000002.68.6, , , , , 00000000002.63.6, , , , , , , , , , , , , , , 00000000001.75.6, , , , 00000000001.83.6] 2023-05-31T20:52:55.212 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:52:55.207+0000 7f60f262c780 1 waiting for incremental sync to catch up: 2023-05-31T20:52:55.212 INFO:teuthology.orchestra.run.smithi099.stderr: local status: [, , , , , , 00000000002.48.6, , , , , , 00000000002.68.6, , , , , , , , , , , , , , , , , , , , 00000000001.75.6, , , , 00000000001.83.6] 2023-05-31T20:52:55.213 INFO:teuthology.orchestra.run.smithi099.stderr: remote markers: [, , , , , , 00000000002.48.6, , , , , , 00000000002.68.6, , , , , 00000000002.63.6, , , , , , , , , , , , , , , 00000000001.75.6, , , , 00000000001.83.6] 2023-05-31T20:53:00.214 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:53:00.211+0000 7f60f262c780 1 waiting for incremental sync to catch up: 2023-05-31T20:53:00.215 INFO:teuthology.orchestra.run.smithi099.stderr: local status: [, , , , , , 00000000002.48.6, , , , , , 00000000002.68.6, , , , , , , , , , , , , , , , , , , , 00000000001.75.6, , , , 00000000001.83.6] 2023-05-31T20:53:00.215 INFO:teuthology.orchestra.run.smithi099.stderr: remote markers: [, , , , , , 00000000002.48.6, , , , , , 00000000002.68.6, , , , , 00000000002.63.6, , , , , , , , , , , , , , , 00000000001.75.6, , , , 00000000001.83.6] 2023-05-31T20:53:05.217 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:53:05.213+0000 7f60f262c780 1 waiting for incremental sync to catch up: 2023-05-31T20:53:05.218 INFO:teuthology.orchestra.run.smithi099.stderr: local status: [, , , , , , 00000000002.48.6, , , , , , 00000000002.68.6, , , , , , , , , , , , , , , , , , , , 00000000001.75.6, , , , 00000000001.83.6] 2023-05-31T20:53:05.218 INFO:teuthology.orchestra.run.smithi099.stderr: remote markers: [, , , , , , 00000000002.48.6, , , , , , 00000000002.68.6, , , , , 00000000002.63.6, , , , , , , , , , , , , , , 00000000001.75.6, , , , 00000000001.83.6] 2023-05-31T20:53:10.219 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:53:10.216+0000 7f60f262c780 1 waiting for incremental sync to catch up: 2023-05-31T20:53:10.220 INFO:teuthology.orchestra.run.smithi099.stderr: local status: [, , , , , , 00000000002.48.6, , , , , , 00000000002.68.6, , , , , , , , , , , , , , , , , , , , 00000000001.75.6, , , , 00000000001.83.6] 2023-05-31T20:53:10.220 INFO:teuthology.orchestra.run.smithi099.stderr: remote markers: [, , , , , , 00000000002.48.6, , , , , , 00000000002.68.6, , , , , 00000000002.63.6, , , , , , , , , , , , , , , 00000000001.75.6, , , , 00000000001.83.6] 2023-05-31T20:53:15.222 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:53:15.218+0000 7f60f262c780 1 waiting for incremental sync to catch up: 2023-05-31T20:53:15.223 INFO:teuthology.orchestra.run.smithi099.stderr: local status: [, , , , , , 00000000002.48.6, , , , , , 00000000002.68.6, , , , , , , , , , , , , , , , , , , , 00000000001.75.6, , , , 00000000001.83.6] 2023-05-31T20:53:15.223 INFO:teuthology.orchestra.run.smithi099.stderr: remote markers: [, , , , , , 00000000002.48.6, , , , , , 00000000002.68.6, , , , , 00000000002.63.6, , , , , , , , , , , , , , , 00000000001.75.6, , , , 00000000001.83.6] 2023-05-31T20:53:20.225 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:53:20.220+0000 7f60f262c780 1 waiting for incremental sync to catch up: 2023-05-31T20:53:20.226 INFO:teuthology.orchestra.run.smithi099.stderr: local status: [, , , , , , 00000000002.48.6, , , , , , 00000000002.68.6, , , , , , , , , , , , , , , , , , , , 00000000001.75.6, , , , 00000000001.83.6] 2023-05-31T20:53:20.226 INFO:teuthology.orchestra.run.smithi099.stderr: remote markers: [, , , , , , 00000000002.48.6, , , , , , 00000000002.68.6, , , , , 00000000002.63.6, , , , , , , , , , , , , , , 00000000001.75.6, , , , 00000000001.83.6] 2023-05-31T20:53:25.228 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:53:25.223+0000 7f60f262c780 1 waiting for incremental sync to catch up: 2023-05-31T20:53:25.228 INFO:teuthology.orchestra.run.smithi099.stderr: local status: [, , , , , , 00000000002.48.6, , , , , , 00000000002.68.6, , , , , , , , , , , , , , , , , , , , 00000000001.75.6, , , , 00000000001.83.6] 2023-05-31T20:53:25.228 INFO:teuthology.orchestra.run.smithi099.stderr: remote markers: [, , , , , , 00000000002.48.6, , , , , , 00000000002.68.6, , , , , 00000000002.63.6, , , , , , , , , , , , , , , 00000000001.75.6, , , , 00000000001.83.6] 2023-05-31T20:53:30.230 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:53:30.225+0000 7f60f262c780 1 waiting for incremental sync to catch up: 2023-05-31T20:53:30.230 INFO:teuthology.orchestra.run.smithi099.stderr: local status: [, , , , , , 00000000002.48.6, , , , , , 00000000002.68.6, , , , , , , , , , , , , , , , , , , , 00000000001.75.6, , , , 00000000001.83.6] 2023-05-31T20:53:30.231 INFO:teuthology.orchestra.run.smithi099.stderr: remote markers: [, , , , , , 00000000002.48.6, , , , , , 00000000002.68.6, , , , , 00000000002.63.6, , , , , , , , , , , , , , , 00000000001.75.6, , , , 00000000001.83.6] 2023-05-31T20:53:35.233 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:53:35.228+0000 7f60f262c780 1 waiting for incremental sync to catch up: 2023-05-31T20:53:35.233 INFO:teuthology.orchestra.run.smithi099.stderr: local status: [, , , , , , 00000000002.48.6, , , , , , 00000000002.68.6, , , , , , , , , , , , , , , , , , , , 00000000001.75.6, , , , 00000000001.83.6] 2023-05-31T20:53:35.233 INFO:teuthology.orchestra.run.smithi099.stderr: remote markers: [, , , , , , 00000000002.48.6, , , , , , 00000000002.68.6, , , , , 00000000002.63.6, , , , , , , , , , , , , , , 00000000001.75.6, , , , 00000000001.83.6] 2023-05-31T20:53:40.235 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:53:40.231+0000 7f60f262c780 1 bucket sync caught up with source: 2023-05-31T20:53:40.236 INFO:teuthology.orchestra.run.smithi099.stderr: local status: [, , , , , , 00000000002.48.6, , , , , , 00000000002.68.6, , , , , 00000000002.63.6, , , , , , , , , , , , , , , 00000000001.75.6, , , , 00000000001.83.6] 2023-05-31T20:53:40.236 INFO:teuthology.orchestra.run.smithi099.stderr: remote markers: [, , , , , , 00000000002.48.6, , , , , , 00000000002.68.6, , , , , 00000000002.63.6, , , , , , , , , , , , , , , 00000000001.75.6, , , , 00000000001.83.6] 2023-05-31T20:53:40.236 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:53:40.231+0000 7f60f262c780 0 bucket checkpoint complete 2023-05-31T20:53:40.257 DEBUG:rgw_multi.tests:bucket checkpoint: source=a2 target=a1 bucket=pklxkd-59 2023-05-31T20:53:40.258 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'pklxkd-59', '--source-zone', 'a2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-05-31T20:53:40.258 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'pklxkd-59', '--source-zone', 'a2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-05-31T20:53:40.259 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket pklxkd-59 --source-zone a2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2023-05-31T20:53:40.320 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2023-05-31T20:53:40.321 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2023-05-31T20:53:40.370 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:53:40.366+0000 7f41286a3780 1 waiting for incremental sync to catch up: 2023-05-31T20:53:40.370 INFO:teuthology.orchestra.run.smithi099.stderr: local status: [, 00000000003.3594.6, 00000000002.3906.6, 00000000002.2037.6, 00000000005.69.6, 00000000004.2031.6, 00000000001.377.6, 00000000001.3914.6, 00000000004.2045.6, 00000000006.2306.6, 00000000006.2030.6] 2023-05-31T20:53:40.370 INFO:teuthology.orchestra.run.smithi099.stderr: remote markers: [, 00000000003.3594.6, 00000000002.3906.6, 00000000002.2037.6, 00000000005.69.6, 00000000004.2031.6, 00000000001.377.6, 00000000001.3914.6, 00000000004.2045.6, 00000000007.2311.6, 00000000006.2030.6] 2023-05-31T20:53:45.371 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:53:45.367+0000 7f41286a3780 1 waiting for incremental sync to catch up: 2023-05-31T20:53:45.371 INFO:teuthology.orchestra.run.smithi099.stderr: local status: [, 00000000003.3594.6, 00000000002.3906.6, 00000000002.2037.6, 00000000005.69.6, 00000000004.2031.6, 00000000001.377.6, 00000000001.3914.6, 00000000004.2045.6, 00000000006.2306.6, 00000000006.2030.6] 2023-05-31T20:53:45.371 INFO:teuthology.orchestra.run.smithi099.stderr: remote markers: [, 00000000003.3594.6, 00000000002.3906.6, 00000000002.2037.6, 00000000005.69.6, 00000000004.2031.6, 00000000001.377.6, 00000000001.3914.6, 00000000004.2045.6, 00000000007.2311.6, 00000000006.2030.6] 2023-05-31T20:53:50.372 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:53:50.368+0000 7f41286a3780 1 bucket sync caught up with source: 2023-05-31T20:53:50.373 INFO:teuthology.orchestra.run.smithi099.stderr: local status: [, 00000000003.3594.6, 00000000002.3906.6, 00000000002.2037.6, 00000000005.69.6, 00000000004.2031.6, 00000000001.377.6, 00000000001.3914.6, 00000000004.2045.6, 00000000007.2311.6, 00000000006.2030.6] 2023-05-31T20:53:50.373 INFO:teuthology.orchestra.run.smithi099.stderr: remote markers: [, 00000000003.3594.6, 00000000002.3906.6, 00000000002.2037.6, 00000000005.69.6, 00000000004.2031.6, 00000000001.377.6, 00000000001.3914.6, 00000000004.2045.6, 00000000007.2311.6, 00000000006.2030.6] 2023-05-31T20:53:50.373 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:53:50.368+0000 7f41286a3780 0 bucket checkpoint complete 2023-05-31T20:53:50.390 INFO:tasks.rgw_multi.zone_rados:comparing bucket=pklxkd-59 zones={a2, a1} 2023-05-31T20:53:50.391 DEBUG:boto:path=/pklxkd-59/ 2023-05-31T20:53:50.391 DEBUG:boto:auth_path=/pklxkd-59/ 2023-05-31T20:53:50.391 DEBUG:boto:Method: HEAD 2023-05-31T20:53:50.392 DEBUG:boto:Path: /pklxkd-59/ 2023-05-31T20:53:50.392 DEBUG:boto:Data: 2023-05-31T20:53:50.392 DEBUG:boto:Headers: {} 2023-05-31T20:53:50.393 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:53:50.393 DEBUG:boto:Port: 8001 2023-05-31T20:53:50.393 DEBUG:boto:Params: {} 2023-05-31T20:53:50.394 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8001} 2023-05-31T20:53:50.395 DEBUG:boto:Token: None 2023-05-31T20:53:50.395 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:53:50 GMT /pklxkd-59/ 2023-05-31T20:53:50.395 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:guh8ugYfhXsQUPfPEmyRqBl40NE= 2023-05-31T20:53:50.396 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:53:50 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:guh8ugYfhXsQUPfPEmyRqBl40NE=', 'Content-Length': '0'} 2023-05-31T20:53:50.399 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '35'), ('X-RGW-Bytes-Used', '105'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000002f075d11dcc1d0bd-006477b3de-1c17-a2'), ('Content-Length', '0'), ('Date', 'Wed, 31 May 2023 20:53:50 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:53:50.400 DEBUG:boto:path=/pklxkd-59/ 2023-05-31T20:53:50.400 DEBUG:boto:auth_path=/pklxkd-59/ 2023-05-31T20:53:50.401 DEBUG:boto:Method: HEAD 2023-05-31T20:53:50.401 DEBUG:boto:Path: /pklxkd-59/ 2023-05-31T20:53:50.401 DEBUG:boto:Data: 2023-05-31T20:53:50.402 DEBUG:boto:Headers: {} 2023-05-31T20:53:50.402 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:53:50.402 DEBUG:boto:Port: 8000 2023-05-31T20:53:50.403 DEBUG:boto:Params: {} 2023-05-31T20:53:50.403 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2023-05-31T20:53:50.403 DEBUG:boto:Token: None 2023-05-31T20:53:50.404 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:53:50 GMT /pklxkd-59/ 2023-05-31T20:53:50.404 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:guh8ugYfhXsQUPfPEmyRqBl40NE= 2023-05-31T20:53:50.405 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:53:50 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:guh8ugYfhXsQUPfPEmyRqBl40NE=', 'Content-Length': '0'} 2023-05-31T20:53:50.409 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '35'), ('X-RGW-Bytes-Used', '105'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000007f79ec74bd4003d2-006477b3de-1630-a1'), ('Content-Length', '0'), ('Date', 'Wed, 31 May 2023 20:53:50 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:53:50.410 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2023-05-31T20:53:50.410 DEBUG:boto:path=/pklxkd-59/ 2023-05-31T20:53:50.410 DEBUG:boto:auth_path=/pklxkd-59/ 2023-05-31T20:53:50.411 DEBUG:boto:path=/pklxkd-59/?versions&max-keys=999 2023-05-31T20:53:50.411 DEBUG:boto:auth_path=/pklxkd-59/?versions&max-keys=999 2023-05-31T20:53:50.411 DEBUG:boto:Method: GET 2023-05-31T20:53:50.412 DEBUG:boto:Path: /pklxkd-59/?versions&max-keys=999 2023-05-31T20:53:50.412 DEBUG:boto:Data: 2023-05-31T20:53:50.413 DEBUG:boto:Headers: {} 2023-05-31T20:53:50.413 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:53:50.413 DEBUG:boto:Port: 8001 2023-05-31T20:53:50.414 DEBUG:boto:Params: {} 2023-05-31T20:53:50.414 DEBUG:boto:Token: None 2023-05-31T20:53:50.414 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:53:50 GMT /pklxkd-59/?versions 2023-05-31T20:53:50.415 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:ezLKHeBFRhPbJSaGU4Y3bZiErDw= 2023-05-31T20:53:50.415 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:53:50 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:ezLKHeBFRhPbJSaGU4Y3bZiErDw=', 'Content-Length': '0'} 2023-05-31T20:53:50.418 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000343f9f69a8f7c9e2-006477b3de-1c17-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:53:50 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:53:50.419 DEBUG:boto:b'pklxkd-59999falseobj11nulltrue2023-05-31T20:39:26.427Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj111nulltrue2023-05-31T20:52:14.993Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj115nulltrue2023-05-31T20:46:33.958Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj116nulltrue2023-05-31T20:48:13.052Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj124nulltrue2023-05-31T20:50:33.587Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj145nulltrue2023-05-31T20:48:13.107Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj148nulltrue2023-05-31T20:52:15.015Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj155nulltrue2023-05-31T20:50:33.605Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj17nulltrue2023-05-31T20:43:07.977Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj185nulltrue2023-05-31T20:52:15.072Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj19nulltrue2023-05-31T20:44:50.557Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj22nulltrue2023-05-31T20:39:26.473Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj23nulltrue2023-05-31T20:46:33.813Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj29nulltrue2023-05-31T20:48:12.956Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj31nulltrue2023-05-31T20:50:33.460Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj33nulltrue2023-05-31T20:39:26.534Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj34nulltrue2023-05-31T20:43:07.994Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj37nulltrue2023-05-31T20:52:14.909Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj38nulltrue2023-05-31T20:44:50.610Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj44nulltrue2023-05-31T20:39:26.565Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj46nulltrue2023-05-31T20:46:33.830Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj51nulltrue2023-05-31T20:43:08.011Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj55nulltrue2023-05-31T20:39:26.630Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj57nulltrue2023-05-31T20:44:50.627Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj58nulltrue2023-05-31T20:48:12.976Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj62nulltrue2023-05-31T20:50:33.515Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj68nulltrue2023-05-31T20:43:08.066Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj69nulltrue2023-05-31T20:46:33.886Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj74nulltrue2023-05-31T20:52:14.934Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj76nulltrue2023-05-31T20:44:50.682Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj85nulltrue2023-05-31T20:43:08.086Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj87nulltrue2023-05-31T20:48:13.031Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj92nulltrue2023-05-31T20:46:33.904Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj93nulltrue2023-05-31T20:50:33.533Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj95nulltrue2023-05-31T20:44:50.705Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:53:50.421 DEBUG:tasks.rgw_multi.zone_rados:o=obj11 2023-05-31T20:53:50.421 DEBUG:tasks.rgw_multi.zone_rados:o=obj111 2023-05-31T20:53:50.422 DEBUG:tasks.rgw_multi.zone_rados:o=obj115 2023-05-31T20:53:50.422 DEBUG:tasks.rgw_multi.zone_rados:o=obj116 2023-05-31T20:53:50.422 DEBUG:tasks.rgw_multi.zone_rados:o=obj124 2023-05-31T20:53:50.422 DEBUG:tasks.rgw_multi.zone_rados:o=obj145 2023-05-31T20:53:50.423 DEBUG:tasks.rgw_multi.zone_rados:o=obj148 2023-05-31T20:53:50.423 DEBUG:tasks.rgw_multi.zone_rados:o=obj155 2023-05-31T20:53:50.423 DEBUG:tasks.rgw_multi.zone_rados:o=obj17 2023-05-31T20:53:50.424 DEBUG:tasks.rgw_multi.zone_rados:o=obj185 2023-05-31T20:53:50.424 DEBUG:tasks.rgw_multi.zone_rados:o=obj19 2023-05-31T20:53:50.424 DEBUG:tasks.rgw_multi.zone_rados:o=obj22 2023-05-31T20:53:50.425 DEBUG:tasks.rgw_multi.zone_rados:o=obj23 2023-05-31T20:53:50.425 DEBUG:tasks.rgw_multi.zone_rados:o=obj29 2023-05-31T20:53:50.425 DEBUG:tasks.rgw_multi.zone_rados:o=obj31 2023-05-31T20:53:50.425 DEBUG:tasks.rgw_multi.zone_rados:o=obj33 2023-05-31T20:53:50.426 DEBUG:tasks.rgw_multi.zone_rados:o=obj34 2023-05-31T20:53:50.426 DEBUG:tasks.rgw_multi.zone_rados:o=obj37 2023-05-31T20:53:50.426 DEBUG:tasks.rgw_multi.zone_rados:o=obj38 2023-05-31T20:53:50.427 DEBUG:tasks.rgw_multi.zone_rados:o=obj44 2023-05-31T20:53:50.427 DEBUG:tasks.rgw_multi.zone_rados:o=obj46 2023-05-31T20:53:50.427 DEBUG:tasks.rgw_multi.zone_rados:o=obj51 2023-05-31T20:53:50.427 DEBUG:tasks.rgw_multi.zone_rados:o=obj55 2023-05-31T20:53:50.428 DEBUG:tasks.rgw_multi.zone_rados:o=obj57 2023-05-31T20:53:50.428 DEBUG:tasks.rgw_multi.zone_rados:o=obj58 2023-05-31T20:53:50.428 DEBUG:tasks.rgw_multi.zone_rados:o=obj62 2023-05-31T20:53:50.429 DEBUG:tasks.rgw_multi.zone_rados:o=obj68 2023-05-31T20:53:50.429 DEBUG:tasks.rgw_multi.zone_rados:o=obj69 2023-05-31T20:53:50.429 DEBUG:tasks.rgw_multi.zone_rados:o=obj74 2023-05-31T20:53:50.430 DEBUG:tasks.rgw_multi.zone_rados:o=obj76 2023-05-31T20:53:50.430 DEBUG:tasks.rgw_multi.zone_rados:o=obj85 2023-05-31T20:53:50.430 DEBUG:tasks.rgw_multi.zone_rados:o=obj87 2023-05-31T20:53:50.430 DEBUG:tasks.rgw_multi.zone_rados:o=obj92 2023-05-31T20:53:50.431 DEBUG:tasks.rgw_multi.zone_rados:o=obj93 2023-05-31T20:53:50.431 DEBUG:tasks.rgw_multi.zone_rados:o=obj95 2023-05-31T20:53:50.431 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2023-05-31T20:53:50.432 DEBUG:boto:path=/pklxkd-59/ 2023-05-31T20:53:50.432 DEBUG:boto:auth_path=/pklxkd-59/ 2023-05-31T20:53:50.432 DEBUG:boto:path=/pklxkd-59/?versions&max-keys=999 2023-05-31T20:53:50.433 DEBUG:boto:auth_path=/pklxkd-59/?versions&max-keys=999 2023-05-31T20:53:50.433 DEBUG:boto:Method: GET 2023-05-31T20:53:50.433 DEBUG:boto:Path: /pklxkd-59/?versions&max-keys=999 2023-05-31T20:53:50.434 DEBUG:boto:Data: 2023-05-31T20:53:50.434 DEBUG:boto:Headers: {} 2023-05-31T20:53:50.435 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:53:50.435 DEBUG:boto:Port: 8000 2023-05-31T20:53:50.435 DEBUG:boto:Params: {} 2023-05-31T20:53:50.436 DEBUG:boto:Token: None 2023-05-31T20:53:50.436 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:53:50 GMT /pklxkd-59/?versions 2023-05-31T20:53:50.436 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:ezLKHeBFRhPbJSaGU4Y3bZiErDw= 2023-05-31T20:53:50.437 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:53:50 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:ezLKHeBFRhPbJSaGU4Y3bZiErDw=', 'Content-Length': '0'} 2023-05-31T20:53:50.441 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000425c789c0073fbc2-006477b3de-1630-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:53:50 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:53:50.441 DEBUG:boto:b'pklxkd-59999falseobj11nulltrue2023-05-31T20:39:26.427Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj111nulltrue2023-05-31T20:52:14.993Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj115nulltrue2023-05-31T20:46:33.958Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj116nulltrue2023-05-31T20:48:13.052Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj124nulltrue2023-05-31T20:50:33.587Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj145nulltrue2023-05-31T20:48:13.107Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj148nulltrue2023-05-31T20:52:15.015Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj155nulltrue2023-05-31T20:50:33.605Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj17nulltrue2023-05-31T20:43:07.977Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj185nulltrue2023-05-31T20:52:15.072Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj19nulltrue2023-05-31T20:44:50.557Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj22nulltrue2023-05-31T20:39:26.473Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj23nulltrue2023-05-31T20:46:33.813Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj29nulltrue2023-05-31T20:48:12.956Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj31nulltrue2023-05-31T20:50:33.460Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj33nulltrue2023-05-31T20:39:26.534Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj34nulltrue2023-05-31T20:43:07.994Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj37nulltrue2023-05-31T20:52:14.909Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj38nulltrue2023-05-31T20:44:50.610Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj44nulltrue2023-05-31T20:39:26.565Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj46nulltrue2023-05-31T20:46:33.830Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj51nulltrue2023-05-31T20:43:08.011Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj55nulltrue2023-05-31T20:39:26.630Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj57nulltrue2023-05-31T20:44:50.627Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj58nulltrue2023-05-31T20:48:12.976Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj62nulltrue2023-05-31T20:50:33.515Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj68nulltrue2023-05-31T20:43:08.066Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj69nulltrue2023-05-31T20:46:33.886Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj74nulltrue2023-05-31T20:52:14.934Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj76nulltrue2023-05-31T20:44:50.682Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj85nulltrue2023-05-31T20:43:08.086Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj87nulltrue2023-05-31T20:48:13.031Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj92nulltrue2023-05-31T20:46:33.904Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj93nulltrue2023-05-31T20:50:33.533Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj95nulltrue2023-05-31T20:44:50.705Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:53:50.443 DEBUG:tasks.rgw_multi.zone_rados:o=obj11 2023-05-31T20:53:50.444 DEBUG:tasks.rgw_multi.zone_rados:o=obj111 2023-05-31T20:53:50.444 DEBUG:tasks.rgw_multi.zone_rados:o=obj115 2023-05-31T20:53:50.444 DEBUG:tasks.rgw_multi.zone_rados:o=obj116 2023-05-31T20:53:50.445 DEBUG:tasks.rgw_multi.zone_rados:o=obj124 2023-05-31T20:53:50.445 DEBUG:tasks.rgw_multi.zone_rados:o=obj145 2023-05-31T20:53:50.445 DEBUG:tasks.rgw_multi.zone_rados:o=obj148 2023-05-31T20:53:50.446 DEBUG:tasks.rgw_multi.zone_rados:o=obj155 2023-05-31T20:53:50.446 DEBUG:tasks.rgw_multi.zone_rados:o=obj17 2023-05-31T20:53:50.446 DEBUG:tasks.rgw_multi.zone_rados:o=obj185 2023-05-31T20:53:50.447 DEBUG:tasks.rgw_multi.zone_rados:o=obj19 2023-05-31T20:53:50.447 DEBUG:tasks.rgw_multi.zone_rados:o=obj22 2023-05-31T20:53:50.447 DEBUG:tasks.rgw_multi.zone_rados:o=obj23 2023-05-31T20:53:50.447 DEBUG:tasks.rgw_multi.zone_rados:o=obj29 2023-05-31T20:53:50.448 DEBUG:tasks.rgw_multi.zone_rados:o=obj31 2023-05-31T20:53:50.448 DEBUG:tasks.rgw_multi.zone_rados:o=obj33 2023-05-31T20:53:50.448 DEBUG:tasks.rgw_multi.zone_rados:o=obj34 2023-05-31T20:53:50.449 DEBUG:tasks.rgw_multi.zone_rados:o=obj37 2023-05-31T20:53:50.449 DEBUG:tasks.rgw_multi.zone_rados:o=obj38 2023-05-31T20:53:50.449 DEBUG:tasks.rgw_multi.zone_rados:o=obj44 2023-05-31T20:53:50.450 DEBUG:tasks.rgw_multi.zone_rados:o=obj46 2023-05-31T20:53:50.450 DEBUG:tasks.rgw_multi.zone_rados:o=obj51 2023-05-31T20:53:50.450 DEBUG:tasks.rgw_multi.zone_rados:o=obj55 2023-05-31T20:53:50.451 DEBUG:tasks.rgw_multi.zone_rados:o=obj57 2023-05-31T20:53:50.451 DEBUG:tasks.rgw_multi.zone_rados:o=obj58 2023-05-31T20:53:50.451 DEBUG:tasks.rgw_multi.zone_rados:o=obj62 2023-05-31T20:53:50.451 DEBUG:tasks.rgw_multi.zone_rados:o=obj68 2023-05-31T20:53:50.452 DEBUG:tasks.rgw_multi.zone_rados:o=obj69 2023-05-31T20:53:50.452 DEBUG:tasks.rgw_multi.zone_rados:o=obj74 2023-05-31T20:53:50.452 DEBUG:tasks.rgw_multi.zone_rados:o=obj76 2023-05-31T20:53:50.453 DEBUG:tasks.rgw_multi.zone_rados:o=obj85 2023-05-31T20:53:50.453 DEBUG:tasks.rgw_multi.zone_rados:o=obj87 2023-05-31T20:53:50.453 DEBUG:tasks.rgw_multi.zone_rados:o=obj92 2023-05-31T20:53:50.454 DEBUG:tasks.rgw_multi.zone_rados:o=obj93 2023-05-31T20:53:50.454 DEBUG:tasks.rgw_multi.zone_rados:o=obj95 2023-05-31T20:53:50.454 DEBUG:boto:path=/pklxkd-59/ 2023-05-31T20:53:50.455 DEBUG:boto:auth_path=/pklxkd-59/ 2023-05-31T20:53:50.455 DEBUG:boto:path=/pklxkd-59/?versions&max-keys=999 2023-05-31T20:53:50.455 DEBUG:boto:auth_path=/pklxkd-59/?versions&max-keys=999 2023-05-31T20:53:50.456 DEBUG:boto:Method: GET 2023-05-31T20:53:50.456 DEBUG:boto:Path: /pklxkd-59/?versions&max-keys=999 2023-05-31T20:53:50.456 DEBUG:boto:Data: 2023-05-31T20:53:50.457 DEBUG:boto:Headers: {} 2023-05-31T20:53:50.457 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:53:50.457 DEBUG:boto:Port: 8001 2023-05-31T20:53:50.458 DEBUG:boto:Params: {} 2023-05-31T20:53:50.458 DEBUG:boto:Token: None 2023-05-31T20:53:50.459 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:53:50 GMT /pklxkd-59/?versions 2023-05-31T20:53:50.459 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:ezLKHeBFRhPbJSaGU4Y3bZiErDw= 2023-05-31T20:53:50.459 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:53:50 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:ezLKHeBFRhPbJSaGU4Y3bZiErDw=', 'Content-Length': '0'} 2023-05-31T20:53:50.462 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a69ff77785e26725-006477b3de-1c17-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:53:50 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:53:50.463 DEBUG:boto:b'pklxkd-59999falseobj11nulltrue2023-05-31T20:39:26.427Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj111nulltrue2023-05-31T20:52:14.993Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj115nulltrue2023-05-31T20:46:33.958Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj116nulltrue2023-05-31T20:48:13.052Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj124nulltrue2023-05-31T20:50:33.587Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj145nulltrue2023-05-31T20:48:13.107Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj148nulltrue2023-05-31T20:52:15.015Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj155nulltrue2023-05-31T20:50:33.605Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj17nulltrue2023-05-31T20:43:07.977Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj185nulltrue2023-05-31T20:52:15.072Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj19nulltrue2023-05-31T20:44:50.557Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj22nulltrue2023-05-31T20:39:26.473Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj23nulltrue2023-05-31T20:46:33.813Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj29nulltrue2023-05-31T20:48:12.956Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj31nulltrue2023-05-31T20:50:33.460Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj33nulltrue2023-05-31T20:39:26.534Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj34nulltrue2023-05-31T20:43:07.994Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj37nulltrue2023-05-31T20:52:14.909Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj38nulltrue2023-05-31T20:44:50.610Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj44nulltrue2023-05-31T20:39:26.565Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj46nulltrue2023-05-31T20:46:33.830Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj51nulltrue2023-05-31T20:43:08.011Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj55nulltrue2023-05-31T20:39:26.630Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj57nulltrue2023-05-31T20:44:50.627Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj58nulltrue2023-05-31T20:48:12.976Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj62nulltrue2023-05-31T20:50:33.515Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj68nulltrue2023-05-31T20:43:08.066Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj69nulltrue2023-05-31T20:46:33.886Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj74nulltrue2023-05-31T20:52:14.934Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj76nulltrue2023-05-31T20:44:50.682Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj85nulltrue2023-05-31T20:43:08.086Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj87nulltrue2023-05-31T20:48:13.031Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj92nulltrue2023-05-31T20:46:33.904Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj93nulltrue2023-05-31T20:50:33.533Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj95nulltrue2023-05-31T20:44:50.705Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:53:50.465 DEBUG:boto:path=/pklxkd-59/ 2023-05-31T20:53:50.465 DEBUG:boto:auth_path=/pklxkd-59/ 2023-05-31T20:53:50.465 DEBUG:boto:path=/pklxkd-59/?versions&max-keys=999 2023-05-31T20:53:50.466 DEBUG:boto:auth_path=/pklxkd-59/?versions&max-keys=999 2023-05-31T20:53:50.466 DEBUG:boto:Method: GET 2023-05-31T20:53:50.467 DEBUG:boto:Path: /pklxkd-59/?versions&max-keys=999 2023-05-31T20:53:50.467 DEBUG:boto:Data: 2023-05-31T20:53:50.467 DEBUG:boto:Headers: {} 2023-05-31T20:53:50.468 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:53:50.468 DEBUG:boto:Port: 8000 2023-05-31T20:53:50.469 DEBUG:boto:Params: {} 2023-05-31T20:53:50.469 DEBUG:boto:Token: None 2023-05-31T20:53:50.470 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:53:50 GMT /pklxkd-59/?versions 2023-05-31T20:53:50.470 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:ezLKHeBFRhPbJSaGU4Y3bZiErDw= 2023-05-31T20:53:50.470 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:53:50 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:ezLKHeBFRhPbJSaGU4Y3bZiErDw=', 'Content-Length': '0'} 2023-05-31T20:53:50.474 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000000f778612a2288a5c-006477b3de-1630-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:53:50 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:53:50.475 DEBUG:boto:b'pklxkd-59999falseobj11nulltrue2023-05-31T20:39:26.427Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj111nulltrue2023-05-31T20:52:14.993Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj115nulltrue2023-05-31T20:46:33.958Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj116nulltrue2023-05-31T20:48:13.052Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj124nulltrue2023-05-31T20:50:33.587Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj145nulltrue2023-05-31T20:48:13.107Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj148nulltrue2023-05-31T20:52:15.015Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj155nulltrue2023-05-31T20:50:33.605Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj17nulltrue2023-05-31T20:43:07.977Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj185nulltrue2023-05-31T20:52:15.072Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj19nulltrue2023-05-31T20:44:50.557Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj22nulltrue2023-05-31T20:39:26.473Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj23nulltrue2023-05-31T20:46:33.813Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj29nulltrue2023-05-31T20:48:12.956Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj31nulltrue2023-05-31T20:50:33.460Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj33nulltrue2023-05-31T20:39:26.534Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj34nulltrue2023-05-31T20:43:07.994Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj37nulltrue2023-05-31T20:52:14.909Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj38nulltrue2023-05-31T20:44:50.610Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj44nulltrue2023-05-31T20:39:26.565Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj46nulltrue2023-05-31T20:46:33.830Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj51nulltrue2023-05-31T20:43:08.011Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj55nulltrue2023-05-31T20:39:26.630Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj57nulltrue2023-05-31T20:44:50.627Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj58nulltrue2023-05-31T20:48:12.976Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj62nulltrue2023-05-31T20:50:33.515Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj68nulltrue2023-05-31T20:43:08.066Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj69nulltrue2023-05-31T20:46:33.886Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj74nulltrue2023-05-31T20:52:14.934Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj76nulltrue2023-05-31T20:44:50.682Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj85nulltrue2023-05-31T20:43:08.086Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj87nulltrue2023-05-31T20:48:13.031Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj92nulltrue2023-05-31T20:46:33.904Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj93nulltrue2023-05-31T20:50:33.533Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj95nulltrue2023-05-31T20:44:50.705Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:53:50.477 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj11 2023-05-31T20:53:50.477 DEBUG:boto:path=/pklxkd-59/obj11 2023-05-31T20:53:50.478 DEBUG:boto:auth_path=/pklxkd-59/obj11 2023-05-31T20:53:50.478 DEBUG:boto:path=/pklxkd-59/obj11?versionId=null 2023-05-31T20:53:50.479 DEBUG:boto:auth_path=/pklxkd-59/obj11?versionId=null 2023-05-31T20:53:50.479 DEBUG:boto:Method: GET 2023-05-31T20:53:50.479 DEBUG:boto:Path: /pklxkd-59/obj11?versionId=null 2023-05-31T20:53:50.480 DEBUG:boto:Data: 2023-05-31T20:53:50.480 DEBUG:boto:Headers: {} 2023-05-31T20:53:50.481 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:53:50.481 DEBUG:boto:Port: 8001 2023-05-31T20:53:50.481 DEBUG:boto:Params: {} 2023-05-31T20:53:50.482 DEBUG:boto:Token: None 2023-05-31T20:53:50.482 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:53:50 GMT /pklxkd-59/obj11?versionId=null 2023-05-31T20:53:50.482 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:VG9wInCcH/IuS+rDfoEZDwzk/+Y= 2023-05-31T20:53:50.483 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:53:50 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:VG9wInCcH/IuS+rDfoEZDwzk/+Y=', 'Content-Length': '0'} 2023-05-31T20:53:50.485 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:39:26 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000472830e822a62aa5-006477b3de-1c17-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:53:50 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:53:50.486 DEBUG:boto:path=/pklxkd-59/obj11 2023-05-31T20:53:50.486 DEBUG:boto:auth_path=/pklxkd-59/obj11 2023-05-31T20:53:50.486 DEBUG:boto:path=/pklxkd-59/obj11?versionId=null 2023-05-31T20:53:50.487 DEBUG:boto:auth_path=/pklxkd-59/obj11?versionId=null 2023-05-31T20:53:50.487 DEBUG:boto:Method: GET 2023-05-31T20:53:50.487 DEBUG:boto:Path: /pklxkd-59/obj11?versionId=null 2023-05-31T20:53:50.488 DEBUG:boto:Data: 2023-05-31T20:53:50.488 DEBUG:boto:Headers: {} 2023-05-31T20:53:50.489 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:53:50.489 DEBUG:boto:Port: 8000 2023-05-31T20:53:50.489 DEBUG:boto:Params: {} 2023-05-31T20:53:50.490 DEBUG:boto:Token: None 2023-05-31T20:53:50.490 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:53:50 GMT /pklxkd-59/obj11?versionId=null 2023-05-31T20:53:50.491 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:VG9wInCcH/IuS+rDfoEZDwzk/+Y= 2023-05-31T20:53:50.491 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:53:50 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:VG9wInCcH/IuS+rDfoEZDwzk/+Y=', 'Content-Length': '0'} 2023-05-31T20:53:50.493 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:39:26 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c433655e03bdc0e9-006477b3de-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:53:50 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:53:50.536 DEBUG:boto:path=/pklxkd-59/obj11 2023-05-31T20:53:50.537 DEBUG:boto:auth_path=/pklxkd-59/obj11 2023-05-31T20:53:50.537 DEBUG:boto:path=/pklxkd-59/obj11?versionId=null 2023-05-31T20:53:50.537 DEBUG:boto:auth_path=/pklxkd-59/obj11?versionId=null 2023-05-31T20:53:50.538 DEBUG:boto:Method: HEAD 2023-05-31T20:53:50.538 DEBUG:boto:Path: /pklxkd-59/obj11?versionId=null 2023-05-31T20:53:50.538 DEBUG:boto:Data: 2023-05-31T20:53:50.539 DEBUG:boto:Headers: {} 2023-05-31T20:53:50.539 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:53:50.539 DEBUG:boto:Port: 8001 2023-05-31T20:53:50.540 DEBUG:boto:Params: {} 2023-05-31T20:53:50.540 DEBUG:boto:Token: None 2023-05-31T20:53:50.540 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:53:50 GMT /pklxkd-59/obj11?versionId=null 2023-05-31T20:53:50.541 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:g5lPQgNfs8i10gwNWZhs4MmTvZ4= 2023-05-31T20:53:50.541 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:53:50 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:g5lPQgNfs8i10gwNWZhs4MmTvZ4=', 'Content-Length': '0'} 2023-05-31T20:53:50.543 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:39:26 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002be0c6c5b4b135e4-006477b3de-1c17-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:53:50 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:53:50.543 DEBUG:boto:path=/pklxkd-59/obj11 2023-05-31T20:53:50.544 DEBUG:boto:auth_path=/pklxkd-59/obj11 2023-05-31T20:53:50.544 DEBUG:boto:path=/pklxkd-59/obj11?versionId=null 2023-05-31T20:53:50.545 DEBUG:boto:auth_path=/pklxkd-59/obj11?versionId=null 2023-05-31T20:53:50.545 DEBUG:boto:Method: HEAD 2023-05-31T20:53:50.545 DEBUG:boto:Path: /pklxkd-59/obj11?versionId=null 2023-05-31T20:53:50.546 DEBUG:boto:Data: 2023-05-31T20:53:50.546 DEBUG:boto:Headers: {} 2023-05-31T20:53:50.546 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:53:50.547 DEBUG:boto:Port: 8000 2023-05-31T20:53:50.547 DEBUG:boto:Params: {} 2023-05-31T20:53:50.547 DEBUG:boto:Token: None 2023-05-31T20:53:50.548 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:53:50 GMT /pklxkd-59/obj11?versionId=null 2023-05-31T20:53:50.548 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:g5lPQgNfs8i10gwNWZhs4MmTvZ4= 2023-05-31T20:53:50.549 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:53:50 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:g5lPQgNfs8i10gwNWZhs4MmTvZ4=', 'Content-Length': '0'} 2023-05-31T20:53:50.550 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:39:26 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b0668c46f76fa3a7-006477b3de-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:53:50 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:53:50.551 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj11 2023-05-31T20:53:50.551 DEBUG:boto:path=/pklxkd-59/obj11 2023-05-31T20:53:50.552 DEBUG:boto:auth_path=/pklxkd-59/obj11 2023-05-31T20:53:50.552 DEBUG:boto:path=/pklxkd-59/obj11?versionId=null 2023-05-31T20:53:50.552 DEBUG:boto:auth_path=/pklxkd-59/obj11?versionId=null 2023-05-31T20:53:50.553 DEBUG:boto:Method: GET 2023-05-31T20:53:50.553 DEBUG:boto:Path: /pklxkd-59/obj11?versionId=null 2023-05-31T20:53:50.553 DEBUG:boto:Data: 2023-05-31T20:53:50.554 DEBUG:boto:Headers: {} 2023-05-31T20:53:50.554 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:53:50.554 DEBUG:boto:Port: 8001 2023-05-31T20:53:50.555 DEBUG:boto:Params: {} 2023-05-31T20:53:50.555 DEBUG:boto:Token: None 2023-05-31T20:53:50.556 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:53:50 GMT /pklxkd-59/obj11?versionId=null 2023-05-31T20:53:50.556 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:VG9wInCcH/IuS+rDfoEZDwzk/+Y= 2023-05-31T20:53:50.556 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:53:50 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:VG9wInCcH/IuS+rDfoEZDwzk/+Y=', 'Content-Length': '0'} 2023-05-31T20:53:50.558 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:39:26 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008d230b5c3af3b8e7-006477b3de-1c17-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:53:50 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:53:50.559 DEBUG:boto:path=/pklxkd-59/obj11 2023-05-31T20:53:50.559 DEBUG:boto:auth_path=/pklxkd-59/obj11 2023-05-31T20:53:50.560 DEBUG:boto:path=/pklxkd-59/obj11?versionId=null 2023-05-31T20:53:50.560 DEBUG:boto:auth_path=/pklxkd-59/obj11?versionId=null 2023-05-31T20:53:50.560 DEBUG:boto:Method: GET 2023-05-31T20:53:50.561 DEBUG:boto:Path: /pklxkd-59/obj11?versionId=null 2023-05-31T20:53:50.561 DEBUG:boto:Data: 2023-05-31T20:53:50.561 DEBUG:boto:Headers: {} 2023-05-31T20:53:50.562 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:53:50.562 DEBUG:boto:Port: 8000 2023-05-31T20:53:50.563 DEBUG:boto:Params: {} 2023-05-31T20:53:50.563 DEBUG:boto:Token: None 2023-05-31T20:53:50.564 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:53:50 GMT /pklxkd-59/obj11?versionId=null 2023-05-31T20:53:50.565 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:VG9wInCcH/IuS+rDfoEZDwzk/+Y= 2023-05-31T20:53:50.565 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:53:50 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:VG9wInCcH/IuS+rDfoEZDwzk/+Y=', 'Content-Length': '0'} 2023-05-31T20:53:50.567 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:39:26 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000844fa8ec0279eb7a-006477b3de-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:53:50 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:53:50.568 DEBUG:boto:path=/pklxkd-59/ 2023-05-31T20:53:50.569 DEBUG:boto:auth_path=/pklxkd-59/ 2023-05-31T20:53:50.569 DEBUG:boto:path=/pklxkd-59/?prefix=obj11 2023-05-31T20:53:50.569 DEBUG:boto:auth_path=/pklxkd-59/?prefix=obj11 2023-05-31T20:53:50.570 DEBUG:boto:Method: GET 2023-05-31T20:53:50.570 DEBUG:boto:Path: /pklxkd-59/?prefix=obj11 2023-05-31T20:53:50.570 DEBUG:boto:Data: 2023-05-31T20:53:50.571 DEBUG:boto:Headers: {} 2023-05-31T20:53:50.571 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:53:50.571 DEBUG:boto:Port: 8001 2023-05-31T20:53:50.572 DEBUG:boto:Params: {} 2023-05-31T20:53:50.572 DEBUG:boto:Token: None 2023-05-31T20:53:50.573 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:53:50 GMT /pklxkd-59/ 2023-05-31T20:53:50.573 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:Q0m7mtbP/vbUm0P5WD8EGCw+sDU= 2023-05-31T20:53:50.574 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:53:50 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:Q0m7mtbP/vbUm0P5WD8EGCw+sDU=', 'Content-Length': '0'} 2023-05-31T20:53:50.576 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000efd25f0079707fc6-006477b3de-1c17-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:53:50 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:53:50.577 DEBUG:boto:b'pklxkd-59obj111000falseobj112023-05-31T20:39:26.427Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj1112023-05-31T20:52:14.993Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj1152023-05-31T20:46:33.958Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj1162023-05-31T20:48:13.052Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:53:50.578 DEBUG:boto:path=/pklxkd-59/ 2023-05-31T20:53:50.578 DEBUG:boto:auth_path=/pklxkd-59/ 2023-05-31T20:53:50.578 DEBUG:boto:path=/pklxkd-59/?prefix=obj11 2023-05-31T20:53:50.579 DEBUG:boto:auth_path=/pklxkd-59/?prefix=obj11 2023-05-31T20:53:50.579 DEBUG:boto:Method: GET 2023-05-31T20:53:50.580 DEBUG:boto:Path: /pklxkd-59/?prefix=obj11 2023-05-31T20:53:50.580 DEBUG:boto:Data: 2023-05-31T20:53:50.580 DEBUG:boto:Headers: {} 2023-05-31T20:53:50.581 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:53:50.581 DEBUG:boto:Port: 8000 2023-05-31T20:53:50.581 DEBUG:boto:Params: {} 2023-05-31T20:53:50.582 DEBUG:boto:Token: None 2023-05-31T20:53:50.582 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:53:50 GMT /pklxkd-59/ 2023-05-31T20:53:50.583 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:Q0m7mtbP/vbUm0P5WD8EGCw+sDU= 2023-05-31T20:53:50.583 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:53:50 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:Q0m7mtbP/vbUm0P5WD8EGCw+sDU=', 'Content-Length': '0'} 2023-05-31T20:53:50.588 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000009cbdd26cf18f06d3-006477b3de-1630-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:53:50 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:53:50.589 DEBUG:boto:b'pklxkd-59obj111000falseobj112023-05-31T20:39:26.427Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj1112023-05-31T20:52:14.993Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj1152023-05-31T20:46:33.958Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj1162023-05-31T20:48:13.052Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:53:50.589 DEBUG:boto:path=/pklxkd-59/obj11 2023-05-31T20:53:50.590 DEBUG:boto:auth_path=/pklxkd-59/obj11 2023-05-31T20:53:50.590 DEBUG:boto:Method: HEAD 2023-05-31T20:53:50.591 DEBUG:boto:Path: /pklxkd-59/obj11 2023-05-31T20:53:50.591 DEBUG:boto:Data: 2023-05-31T20:53:50.591 DEBUG:boto:Headers: {} 2023-05-31T20:53:50.592 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:53:50.592 DEBUG:boto:Port: 8001 2023-05-31T20:53:50.593 DEBUG:boto:Params: {} 2023-05-31T20:53:50.593 DEBUG:boto:Token: None 2023-05-31T20:53:50.593 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:53:50 GMT /pklxkd-59/obj11 2023-05-31T20:53:50.594 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:ygpjKU+IpUdUOKvoBTKVQIXiAF0= 2023-05-31T20:53:50.594 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:53:50 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:ygpjKU+IpUdUOKvoBTKVQIXiAF0=', 'Content-Length': '0'} 2023-05-31T20:53:50.596 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:39:26 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000030daeaae2560190-006477b3de-1c17-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:53:50 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:53:50.596 DEBUG:boto:path=/pklxkd-59/obj11 2023-05-31T20:53:50.597 DEBUG:boto:auth_path=/pklxkd-59/obj11 2023-05-31T20:53:50.597 DEBUG:boto:Method: HEAD 2023-05-31T20:53:50.597 DEBUG:boto:Path: /pklxkd-59/obj11 2023-05-31T20:53:50.598 DEBUG:boto:Data: 2023-05-31T20:53:50.598 DEBUG:boto:Headers: {} 2023-05-31T20:53:50.599 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:53:50.599 DEBUG:boto:Port: 8000 2023-05-31T20:53:50.599 DEBUG:boto:Params: {} 2023-05-31T20:53:50.600 DEBUG:boto:Token: None 2023-05-31T20:53:50.600 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:53:50 GMT /pklxkd-59/obj11 2023-05-31T20:53:50.600 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:ygpjKU+IpUdUOKvoBTKVQIXiAF0= 2023-05-31T20:53:50.601 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:53:50 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:ygpjKU+IpUdUOKvoBTKVQIXiAF0=', 'Content-Length': '0'} 2023-05-31T20:53:50.602 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:39:26 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000326ab3cdacdad8e9-006477b3de-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:53:50 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:53:50.603 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj11 2023-05-31T20:53:50.603 DEBUG:boto:path=/pklxkd-59/obj11 2023-05-31T20:53:50.604 DEBUG:boto:auth_path=/pklxkd-59/obj11 2023-05-31T20:53:50.604 DEBUG:boto:Method: GET 2023-05-31T20:53:50.605 DEBUG:boto:Path: /pklxkd-59/obj11 2023-05-31T20:53:50.605 DEBUG:boto:Data: 2023-05-31T20:53:50.605 DEBUG:boto:Headers: {} 2023-05-31T20:53:50.606 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:53:50.606 DEBUG:boto:Port: 8001 2023-05-31T20:53:50.606 DEBUG:boto:Params: {} 2023-05-31T20:53:50.607 DEBUG:boto:Token: None 2023-05-31T20:53:50.607 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:53:50 GMT /pklxkd-59/obj11 2023-05-31T20:53:50.608 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:4+sGTOoDucRXc8+0xmCFU+MDo08= 2023-05-31T20:53:50.608 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:53:50 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:4+sGTOoDucRXc8+0xmCFU+MDo08=', 'Content-Length': '0'} 2023-05-31T20:53:50.610 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:39:26 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000064d67215e25faa63-006477b3de-1c17-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:53:50 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:53:50.611 DEBUG:boto:path=/pklxkd-59/obj11 2023-05-31T20:53:50.611 DEBUG:boto:auth_path=/pklxkd-59/obj11 2023-05-31T20:53:50.612 DEBUG:boto:Method: GET 2023-05-31T20:53:50.612 DEBUG:boto:Path: /pklxkd-59/obj11 2023-05-31T20:53:50.613 DEBUG:boto:Data: 2023-05-31T20:53:50.613 DEBUG:boto:Headers: {} 2023-05-31T20:53:50.613 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:53:50.614 DEBUG:boto:Port: 8000 2023-05-31T20:53:50.614 DEBUG:boto:Params: {} 2023-05-31T20:53:50.614 DEBUG:boto:Token: None 2023-05-31T20:53:50.615 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:53:50 GMT /pklxkd-59/obj11 2023-05-31T20:53:50.615 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:4+sGTOoDucRXc8+0xmCFU+MDo08= 2023-05-31T20:53:50.616 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:53:50 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:4+sGTOoDucRXc8+0xmCFU+MDo08=', 'Content-Length': '0'} 2023-05-31T20:53:50.618 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:39:26 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e5f7b74c3c6a9dfd-006477b3de-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:53:50 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:53:50.618 DEBUG:boto:path=/pklxkd-59/ 2023-05-31T20:53:50.619 DEBUG:boto:auth_path=/pklxkd-59/ 2023-05-31T20:53:50.619 DEBUG:boto:path=/pklxkd-59/?prefix=obj11 2023-05-31T20:53:50.619 DEBUG:boto:auth_path=/pklxkd-59/?prefix=obj11 2023-05-31T20:53:50.620 DEBUG:boto:Method: GET 2023-05-31T20:53:50.620 DEBUG:boto:Path: /pklxkd-59/?prefix=obj11 2023-05-31T20:53:50.620 DEBUG:boto:Data: 2023-05-31T20:53:50.621 DEBUG:boto:Headers: {} 2023-05-31T20:53:50.621 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:53:50.622 DEBUG:boto:Port: 8001 2023-05-31T20:53:50.622 DEBUG:boto:Params: {} 2023-05-31T20:53:50.623 DEBUG:boto:Token: None 2023-05-31T20:53:50.623 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:53:50 GMT /pklxkd-59/ 2023-05-31T20:53:50.623 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:Q0m7mtbP/vbUm0P5WD8EGCw+sDU= 2023-05-31T20:53:50.624 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:53:50 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:Q0m7mtbP/vbUm0P5WD8EGCw+sDU=', 'Content-Length': '0'} 2023-05-31T20:53:50.626 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000098b99b2aa01043a1-006477b3de-1c17-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:53:50 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:53:50.627 DEBUG:boto:b'pklxkd-59obj111000falseobj112023-05-31T20:39:26.427Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj1112023-05-31T20:52:14.993Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj1152023-05-31T20:46:33.958Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj1162023-05-31T20:48:13.052Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:53:50.628 DEBUG:boto:path=/pklxkd-59/ 2023-05-31T20:53:50.628 DEBUG:boto:auth_path=/pklxkd-59/ 2023-05-31T20:53:50.629 DEBUG:boto:path=/pklxkd-59/?prefix=obj11 2023-05-31T20:53:50.629 DEBUG:boto:auth_path=/pklxkd-59/?prefix=obj11 2023-05-31T20:53:50.630 DEBUG:boto:Method: GET 2023-05-31T20:53:50.630 DEBUG:boto:Path: /pklxkd-59/?prefix=obj11 2023-05-31T20:53:50.631 DEBUG:boto:Data: 2023-05-31T20:53:50.631 DEBUG:boto:Headers: {} 2023-05-31T20:53:50.631 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:53:50.632 DEBUG:boto:Port: 8000 2023-05-31T20:53:50.632 DEBUG:boto:Params: {} 2023-05-31T20:53:50.633 DEBUG:boto:Token: None 2023-05-31T20:53:50.633 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:53:50 GMT /pklxkd-59/ 2023-05-31T20:53:50.634 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:Q0m7mtbP/vbUm0P5WD8EGCw+sDU= 2023-05-31T20:53:50.634 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:53:50 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:Q0m7mtbP/vbUm0P5WD8EGCw+sDU=', 'Content-Length': '0'} 2023-05-31T20:53:50.638 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000087233bba17c1c7ec-006477b3de-1630-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:53:50 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:53:50.639 DEBUG:boto:b'pklxkd-59obj111000falseobj112023-05-31T20:39:26.427Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj1112023-05-31T20:52:14.993Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj1152023-05-31T20:46:33.958Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj1162023-05-31T20:48:13.052Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:53:50.640 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj111 2023-05-31T20:53:50.640 DEBUG:boto:path=/pklxkd-59/obj111 2023-05-31T20:53:50.640 DEBUG:boto:auth_path=/pklxkd-59/obj111 2023-05-31T20:53:50.641 DEBUG:boto:path=/pklxkd-59/obj111?versionId=null 2023-05-31T20:53:50.641 DEBUG:boto:auth_path=/pklxkd-59/obj111?versionId=null 2023-05-31T20:53:50.642 DEBUG:boto:Method: GET 2023-05-31T20:53:50.642 DEBUG:boto:Path: /pklxkd-59/obj111?versionId=null 2023-05-31T20:53:50.643 DEBUG:boto:Data: 2023-05-31T20:53:50.643 DEBUG:boto:Headers: {} 2023-05-31T20:53:50.643 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:53:50.644 DEBUG:boto:Port: 8001 2023-05-31T20:53:50.644 DEBUG:boto:Params: {} 2023-05-31T20:53:50.645 DEBUG:boto:Token: None 2023-05-31T20:53:50.645 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:53:50 GMT /pklxkd-59/obj111?versionId=null 2023-05-31T20:53:50.645 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:gXd+pygrFmW3kYjOi0z6kAp9IcY= 2023-05-31T20:53:50.646 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:53:50 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:gXd+pygrFmW3kYjOi0z6kAp9IcY=', 'Content-Length': '0'} 2023-05-31T20:53:50.648 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:52:14 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000054598aed33fb0e45-006477b3de-1c17-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:53:50 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:53:50.649 DEBUG:boto:path=/pklxkd-59/obj111 2023-05-31T20:53:50.649 DEBUG:boto:auth_path=/pklxkd-59/obj111 2023-05-31T20:53:50.650 DEBUG:boto:path=/pklxkd-59/obj111?versionId=null 2023-05-31T20:53:50.650 DEBUG:boto:auth_path=/pklxkd-59/obj111?versionId=null 2023-05-31T20:53:50.650 DEBUG:boto:Method: GET 2023-05-31T20:53:50.651 DEBUG:boto:Path: /pklxkd-59/obj111?versionId=null 2023-05-31T20:53:50.651 DEBUG:boto:Data: 2023-05-31T20:53:50.651 DEBUG:boto:Headers: {} 2023-05-31T20:53:50.652 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:53:50.652 DEBUG:boto:Port: 8000 2023-05-31T20:53:50.653 DEBUG:boto:Params: {} 2023-05-31T20:53:50.653 DEBUG:boto:Token: None 2023-05-31T20:53:50.653 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:53:50 GMT /pklxkd-59/obj111?versionId=null 2023-05-31T20:53:50.654 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:gXd+pygrFmW3kYjOi0z6kAp9IcY= 2023-05-31T20:53:50.654 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:53:50 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:gXd+pygrFmW3kYjOi0z6kAp9IcY=', 'Content-Length': '0'} 2023-05-31T20:53:50.656 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:52:14 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ec598fb7a672361c-006477b3de-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:53:50 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:53:50.657 DEBUG:boto:path=/pklxkd-59/obj111 2023-05-31T20:53:50.657 DEBUG:boto:auth_path=/pklxkd-59/obj111 2023-05-31T20:53:50.657 DEBUG:boto:path=/pklxkd-59/obj111?versionId=null 2023-05-31T20:53:50.658 DEBUG:boto:auth_path=/pklxkd-59/obj111?versionId=null 2023-05-31T20:53:50.658 DEBUG:boto:Method: HEAD 2023-05-31T20:53:50.658 DEBUG:boto:Path: /pklxkd-59/obj111?versionId=null 2023-05-31T20:53:50.659 DEBUG:boto:Data: 2023-05-31T20:53:50.659 DEBUG:boto:Headers: {} 2023-05-31T20:53:50.660 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:53:50.660 DEBUG:boto:Port: 8001 2023-05-31T20:53:50.661 DEBUG:boto:Params: {} 2023-05-31T20:53:50.661 DEBUG:boto:Token: None 2023-05-31T20:53:50.661 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:53:50 GMT /pklxkd-59/obj111?versionId=null 2023-05-31T20:53:50.662 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:3CLSVjJdmGyNnuVwRpt/DPwyiU4= 2023-05-31T20:53:50.662 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:53:50 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:3CLSVjJdmGyNnuVwRpt/DPwyiU4=', 'Content-Length': '0'} 2023-05-31T20:53:50.664 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:52:14 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000028c01af795551f23-006477b3de-1c17-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:53:50 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:53:50.665 DEBUG:boto:path=/pklxkd-59/obj111 2023-05-31T20:53:50.665 DEBUG:boto:auth_path=/pklxkd-59/obj111 2023-05-31T20:53:50.666 DEBUG:boto:path=/pklxkd-59/obj111?versionId=null 2023-05-31T20:53:50.666 DEBUG:boto:auth_path=/pklxkd-59/obj111?versionId=null 2023-05-31T20:53:50.667 DEBUG:boto:Method: HEAD 2023-05-31T20:53:50.667 DEBUG:boto:Path: /pklxkd-59/obj111?versionId=null 2023-05-31T20:53:50.667 DEBUG:boto:Data: 2023-05-31T20:53:50.668 DEBUG:boto:Headers: {} 2023-05-31T20:53:50.668 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:53:50.668 DEBUG:boto:Port: 8000 2023-05-31T20:53:50.669 DEBUG:boto:Params: {} 2023-05-31T20:53:50.669 DEBUG:boto:Token: None 2023-05-31T20:53:50.669 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:53:50 GMT /pklxkd-59/obj111?versionId=null 2023-05-31T20:53:50.670 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:3CLSVjJdmGyNnuVwRpt/DPwyiU4= 2023-05-31T20:53:50.670 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:53:50 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:3CLSVjJdmGyNnuVwRpt/DPwyiU4=', 'Content-Length': '0'} 2023-05-31T20:53:50.672 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:52:14 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001c178597282982c2-006477b3de-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:53:50 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:53:50.673 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj111 2023-05-31T20:53:50.673 DEBUG:boto:path=/pklxkd-59/obj111 2023-05-31T20:53:50.673 DEBUG:boto:auth_path=/pklxkd-59/obj111 2023-05-31T20:53:50.674 DEBUG:boto:path=/pklxkd-59/obj111?versionId=null 2023-05-31T20:53:50.674 DEBUG:boto:auth_path=/pklxkd-59/obj111?versionId=null 2023-05-31T20:53:50.674 DEBUG:boto:Method: GET 2023-05-31T20:53:50.675 DEBUG:boto:Path: /pklxkd-59/obj111?versionId=null 2023-05-31T20:53:50.675 DEBUG:boto:Data: 2023-05-31T20:53:50.676 DEBUG:boto:Headers: {} 2023-05-31T20:53:50.676 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:53:50.676 DEBUG:boto:Port: 8001 2023-05-31T20:53:50.677 DEBUG:boto:Params: {} 2023-05-31T20:53:50.677 DEBUG:boto:Token: None 2023-05-31T20:53:50.677 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:53:50 GMT /pklxkd-59/obj111?versionId=null 2023-05-31T20:53:50.678 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:gXd+pygrFmW3kYjOi0z6kAp9IcY= 2023-05-31T20:53:50.678 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:53:50 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:gXd+pygrFmW3kYjOi0z6kAp9IcY=', 'Content-Length': '0'} 2023-05-31T20:53:50.680 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:52:14 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000007a039ceb1e5aaa9-006477b3de-1c17-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:53:50 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:53:50.720 DEBUG:boto:path=/pklxkd-59/obj111 2023-05-31T20:53:50.721 DEBUG:boto:auth_path=/pklxkd-59/obj111 2023-05-31T20:53:50.721 DEBUG:boto:path=/pklxkd-59/obj111?versionId=null 2023-05-31T20:53:50.722 DEBUG:boto:auth_path=/pklxkd-59/obj111?versionId=null 2023-05-31T20:53:50.722 DEBUG:boto:Method: GET 2023-05-31T20:53:50.723 DEBUG:boto:Path: /pklxkd-59/obj111?versionId=null 2023-05-31T20:53:50.723 DEBUG:boto:Data: 2023-05-31T20:53:50.724 DEBUG:boto:Headers: {} 2023-05-31T20:53:50.725 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:53:50.725 DEBUG:boto:Port: 8000 2023-05-31T20:53:50.725 DEBUG:boto:Params: {} 2023-05-31T20:53:50.726 DEBUG:boto:Token: None 2023-05-31T20:53:50.727 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:53:50 GMT /pklxkd-59/obj111?versionId=null 2023-05-31T20:53:50.727 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:gXd+pygrFmW3kYjOi0z6kAp9IcY= 2023-05-31T20:53:50.728 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:53:50 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:gXd+pygrFmW3kYjOi0z6kAp9IcY=', 'Content-Length': '0'} 2023-05-31T20:53:50.729 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:52:14 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004a4462e87662e37b-006477b3de-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:53:50 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:53:50.730 DEBUG:boto:path=/pklxkd-59/ 2023-05-31T20:53:50.730 DEBUG:boto:auth_path=/pklxkd-59/ 2023-05-31T20:53:50.731 DEBUG:boto:path=/pklxkd-59/?prefix=obj111 2023-05-31T20:53:50.731 DEBUG:boto:auth_path=/pklxkd-59/?prefix=obj111 2023-05-31T20:53:50.732 DEBUG:boto:Method: GET 2023-05-31T20:53:50.732 DEBUG:boto:Path: /pklxkd-59/?prefix=obj111 2023-05-31T20:53:50.732 DEBUG:boto:Data: 2023-05-31T20:53:50.733 DEBUG:boto:Headers: {} 2023-05-31T20:53:50.733 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:53:50.733 DEBUG:boto:Port: 8001 2023-05-31T20:53:50.734 DEBUG:boto:Params: {} 2023-05-31T20:53:50.734 DEBUG:boto:Token: None 2023-05-31T20:53:50.734 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:53:50 GMT /pklxkd-59/ 2023-05-31T20:53:50.735 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:Q0m7mtbP/vbUm0P5WD8EGCw+sDU= 2023-05-31T20:53:50.735 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:53:50 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:Q0m7mtbP/vbUm0P5WD8EGCw+sDU=', 'Content-Length': '0'} 2023-05-31T20:53:50.738 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000430007ff621b1807-006477b3de-1c17-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:53:50 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:53:50.738 DEBUG:boto:b'pklxkd-59obj1111000falseobj1112023-05-31T20:52:14.993Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:53:50.739 DEBUG:boto:path=/pklxkd-59/ 2023-05-31T20:53:50.739 DEBUG:boto:auth_path=/pklxkd-59/ 2023-05-31T20:53:50.740 DEBUG:boto:path=/pklxkd-59/?prefix=obj111 2023-05-31T20:53:50.740 DEBUG:boto:auth_path=/pklxkd-59/?prefix=obj111 2023-05-31T20:53:50.740 DEBUG:boto:Method: GET 2023-05-31T20:53:50.741 DEBUG:boto:Path: /pklxkd-59/?prefix=obj111 2023-05-31T20:53:50.741 DEBUG:boto:Data: 2023-05-31T20:53:50.741 DEBUG:boto:Headers: {} 2023-05-31T20:53:50.742 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:53:50.742 DEBUG:boto:Port: 8000 2023-05-31T20:53:50.742 DEBUG:boto:Params: {} 2023-05-31T20:53:50.743 DEBUG:boto:Token: None 2023-05-31T20:53:50.743 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:53:50 GMT /pklxkd-59/ 2023-05-31T20:53:50.744 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:Q0m7mtbP/vbUm0P5WD8EGCw+sDU= 2023-05-31T20:53:50.744 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:53:50 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:Q0m7mtbP/vbUm0P5WD8EGCw+sDU=', 'Content-Length': '0'} 2023-05-31T20:53:50.748 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000027e72d4fd50373b4-006477b3de-1630-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:53:50 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:53:50.748 DEBUG:boto:b'pklxkd-59obj1111000falseobj1112023-05-31T20:52:14.993Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:53:50.749 DEBUG:boto:path=/pklxkd-59/obj111 2023-05-31T20:53:50.749 DEBUG:boto:auth_path=/pklxkd-59/obj111 2023-05-31T20:53:50.750 DEBUG:boto:Method: HEAD 2023-05-31T20:53:50.750 DEBUG:boto:Path: /pklxkd-59/obj111 2023-05-31T20:53:50.750 DEBUG:boto:Data: 2023-05-31T20:53:50.751 DEBUG:boto:Headers: {} 2023-05-31T20:53:50.751 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:53:50.751 DEBUG:boto:Port: 8001 2023-05-31T20:53:50.752 DEBUG:boto:Params: {} 2023-05-31T20:53:50.752 DEBUG:boto:Token: None 2023-05-31T20:53:50.752 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:53:50 GMT /pklxkd-59/obj111 2023-05-31T20:53:50.753 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:dIEEbJUjAtYtPSfMzhX4P3YeMks= 2023-05-31T20:53:50.753 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:53:50 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:dIEEbJUjAtYtPSfMzhX4P3YeMks=', 'Content-Length': '0'} 2023-05-31T20:53:50.755 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:52:14 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000bc4bb6ae2c60e846-006477b3de-1c17-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:53:50 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:53:50.756 DEBUG:boto:path=/pklxkd-59/obj111 2023-05-31T20:53:50.756 DEBUG:boto:auth_path=/pklxkd-59/obj111 2023-05-31T20:53:50.756 DEBUG:boto:Method: HEAD 2023-05-31T20:53:50.757 DEBUG:boto:Path: /pklxkd-59/obj111 2023-05-31T20:53:50.757 DEBUG:boto:Data: 2023-05-31T20:53:50.758 DEBUG:boto:Headers: {} 2023-05-31T20:53:50.758 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:53:50.758 DEBUG:boto:Port: 8000 2023-05-31T20:53:50.759 DEBUG:boto:Params: {} 2023-05-31T20:53:50.759 DEBUG:boto:Token: None 2023-05-31T20:53:50.759 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:53:50 GMT /pklxkd-59/obj111 2023-05-31T20:53:50.760 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:dIEEbJUjAtYtPSfMzhX4P3YeMks= 2023-05-31T20:53:50.760 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:53:50 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:dIEEbJUjAtYtPSfMzhX4P3YeMks=', 'Content-Length': '0'} 2023-05-31T20:53:50.762 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:52:14 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ec0c2c68764213f7-006477b3de-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:53:50 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:53:50.762 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj111 2023-05-31T20:53:50.763 DEBUG:boto:path=/pklxkd-59/obj111 2023-05-31T20:53:50.763 DEBUG:boto:auth_path=/pklxkd-59/obj111 2023-05-31T20:53:50.763 DEBUG:boto:Method: GET 2023-05-31T20:53:50.764 DEBUG:boto:Path: /pklxkd-59/obj111 2023-05-31T20:53:50.764 DEBUG:boto:Data: 2023-05-31T20:53:50.765 DEBUG:boto:Headers: {} 2023-05-31T20:53:50.765 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:53:50.765 DEBUG:boto:Port: 8001 2023-05-31T20:53:50.766 DEBUG:boto:Params: {} 2023-05-31T20:53:50.766 DEBUG:boto:Token: None 2023-05-31T20:53:50.767 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:53:50 GMT /pklxkd-59/obj111 2023-05-31T20:53:50.767 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:HjZoAVkzFLOhcpKiEwOLFj83sm8= 2023-05-31T20:53:50.768 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:53:50 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:HjZoAVkzFLOhcpKiEwOLFj83sm8=', 'Content-Length': '0'} 2023-05-31T20:53:50.769 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:52:14 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c683c842d19546d5-006477b3de-1c17-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:53:50 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:53:50.770 DEBUG:boto:path=/pklxkd-59/obj111 2023-05-31T20:53:50.770 DEBUG:boto:auth_path=/pklxkd-59/obj111 2023-05-31T20:53:50.771 DEBUG:boto:Method: GET 2023-05-31T20:53:50.771 DEBUG:boto:Path: /pklxkd-59/obj111 2023-05-31T20:53:50.772 DEBUG:boto:Data: 2023-05-31T20:53:50.772 DEBUG:boto:Headers: {} 2023-05-31T20:53:50.772 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:53:50.773 DEBUG:boto:Port: 8000 2023-05-31T20:53:50.773 DEBUG:boto:Params: {} 2023-05-31T20:53:50.773 DEBUG:boto:Token: None 2023-05-31T20:53:50.774 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:53:50 GMT /pklxkd-59/obj111 2023-05-31T20:53:50.774 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:HjZoAVkzFLOhcpKiEwOLFj83sm8= 2023-05-31T20:53:50.775 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:53:50 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:HjZoAVkzFLOhcpKiEwOLFj83sm8=', 'Content-Length': '0'} 2023-05-31T20:53:50.776 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:52:14 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002693b3433b6dc53b-006477b3de-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:53:50 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:53:50.777 DEBUG:boto:path=/pklxkd-59/ 2023-05-31T20:53:50.777 DEBUG:boto:auth_path=/pklxkd-59/ 2023-05-31T20:53:50.778 DEBUG:boto:path=/pklxkd-59/?prefix=obj111 2023-05-31T20:53:50.778 DEBUG:boto:auth_path=/pklxkd-59/?prefix=obj111 2023-05-31T20:53:50.778 DEBUG:boto:Method: GET 2023-05-31T20:53:50.779 DEBUG:boto:Path: /pklxkd-59/?prefix=obj111 2023-05-31T20:53:50.779 DEBUG:boto:Data: 2023-05-31T20:53:50.780 DEBUG:boto:Headers: {} 2023-05-31T20:53:50.780 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:53:50.780 DEBUG:boto:Port: 8001 2023-05-31T20:53:50.781 DEBUG:boto:Params: {} 2023-05-31T20:53:50.781 DEBUG:boto:Token: None 2023-05-31T20:53:50.781 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:53:50 GMT /pklxkd-59/ 2023-05-31T20:53:50.782 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:Q0m7mtbP/vbUm0P5WD8EGCw+sDU= 2023-05-31T20:53:50.782 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:53:50 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:Q0m7mtbP/vbUm0P5WD8EGCw+sDU=', 'Content-Length': '0'} 2023-05-31T20:53:50.784 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e72c1ad1647cb451-006477b3de-1c17-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:53:50 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:53:50.785 DEBUG:boto:b'pklxkd-59obj1111000falseobj1112023-05-31T20:52:14.993Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:53:50.786 DEBUG:boto:path=/pklxkd-59/ 2023-05-31T20:53:50.786 DEBUG:boto:auth_path=/pklxkd-59/ 2023-05-31T20:53:50.786 DEBUG:boto:path=/pklxkd-59/?prefix=obj111 2023-05-31T20:53:50.787 DEBUG:boto:auth_path=/pklxkd-59/?prefix=obj111 2023-05-31T20:53:50.787 DEBUG:boto:Method: GET 2023-05-31T20:53:50.787 DEBUG:boto:Path: /pklxkd-59/?prefix=obj111 2023-05-31T20:53:50.788 DEBUG:boto:Data: 2023-05-31T20:53:50.788 DEBUG:boto:Headers: {} 2023-05-31T20:53:50.789 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:53:50.789 DEBUG:boto:Port: 8000 2023-05-31T20:53:50.789 DEBUG:boto:Params: {} 2023-05-31T20:53:50.790 DEBUG:boto:Token: None 2023-05-31T20:53:50.790 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:53:50 GMT /pklxkd-59/ 2023-05-31T20:53:50.790 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:Q0m7mtbP/vbUm0P5WD8EGCw+sDU= 2023-05-31T20:53:50.791 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:53:50 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:Q0m7mtbP/vbUm0P5WD8EGCw+sDU=', 'Content-Length': '0'} 2023-05-31T20:53:50.795 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000009f6c75f595b96a2e-006477b3de-1630-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:53:50 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:53:50.795 DEBUG:boto:b'pklxkd-59obj1111000falseobj1112023-05-31T20:52:14.993Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:53:50.796 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj115 2023-05-31T20:53:50.796 DEBUG:boto:path=/pklxkd-59/obj115 2023-05-31T20:53:50.797 DEBUG:boto:auth_path=/pklxkd-59/obj115 2023-05-31T20:53:50.797 DEBUG:boto:path=/pklxkd-59/obj115?versionId=null 2023-05-31T20:53:50.797 DEBUG:boto:auth_path=/pklxkd-59/obj115?versionId=null 2023-05-31T20:53:50.798 DEBUG:boto:Method: GET 2023-05-31T20:53:50.798 DEBUG:boto:Path: /pklxkd-59/obj115?versionId=null 2023-05-31T20:53:50.798 DEBUG:boto:Data: 2023-05-31T20:53:50.799 DEBUG:boto:Headers: {} 2023-05-31T20:53:50.799 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:53:50.800 DEBUG:boto:Port: 8001 2023-05-31T20:53:50.800 DEBUG:boto:Params: {} 2023-05-31T20:53:50.800 DEBUG:boto:Token: None 2023-05-31T20:53:50.801 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:53:50 GMT /pklxkd-59/obj115?versionId=null 2023-05-31T20:53:50.801 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:YihhPjO8TsyGR72NArN2uTJMkk8= 2023-05-31T20:53:50.802 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:53:50 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:YihhPjO8TsyGR72NArN2uTJMkk8=', 'Content-Length': '0'} 2023-05-31T20:53:50.803 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:46:33 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000931a90091d666b9-006477b3de-1c17-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:53:50 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:53:50.844 DEBUG:boto:path=/pklxkd-59/obj115 2023-05-31T20:53:50.845 DEBUG:boto:auth_path=/pklxkd-59/obj115 2023-05-31T20:53:50.845 DEBUG:boto:path=/pklxkd-59/obj115?versionId=null 2023-05-31T20:53:50.845 DEBUG:boto:auth_path=/pklxkd-59/obj115?versionId=null 2023-05-31T20:53:50.846 DEBUG:boto:Method: GET 2023-05-31T20:53:50.846 DEBUG:boto:Path: /pklxkd-59/obj115?versionId=null 2023-05-31T20:53:50.846 DEBUG:boto:Data: 2023-05-31T20:53:50.847 DEBUG:boto:Headers: {} 2023-05-31T20:53:50.847 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:53:50.847 DEBUG:boto:Port: 8000 2023-05-31T20:53:50.848 DEBUG:boto:Params: {} 2023-05-31T20:53:50.848 DEBUG:boto:Token: None 2023-05-31T20:53:50.849 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:53:50 GMT /pklxkd-59/obj115?versionId=null 2023-05-31T20:53:50.849 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:YihhPjO8TsyGR72NArN2uTJMkk8= 2023-05-31T20:53:50.850 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:53:50 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:YihhPjO8TsyGR72NArN2uTJMkk8=', 'Content-Length': '0'} 2023-05-31T20:53:50.852 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:46:33 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009d1f0a6c4d50f592-006477b3de-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:53:50 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:53:50.852 DEBUG:boto:path=/pklxkd-59/obj115 2023-05-31T20:53:50.853 DEBUG:boto:auth_path=/pklxkd-59/obj115 2023-05-31T20:53:50.853 DEBUG:boto:path=/pklxkd-59/obj115?versionId=null 2023-05-31T20:53:50.853 DEBUG:boto:auth_path=/pklxkd-59/obj115?versionId=null 2023-05-31T20:53:50.854 DEBUG:boto:Method: HEAD 2023-05-31T20:53:50.854 DEBUG:boto:Path: /pklxkd-59/obj115?versionId=null 2023-05-31T20:53:50.854 DEBUG:boto:Data: 2023-05-31T20:53:50.855 DEBUG:boto:Headers: {} 2023-05-31T20:53:50.855 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:53:50.855 DEBUG:boto:Port: 8001 2023-05-31T20:53:50.856 DEBUG:boto:Params: {} 2023-05-31T20:53:50.856 DEBUG:boto:Token: None 2023-05-31T20:53:50.857 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:53:50 GMT /pklxkd-59/obj115?versionId=null 2023-05-31T20:53:50.857 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:b/+70aj9zQIQv4jOrIj2jtDv5K4= 2023-05-31T20:53:50.858 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:53:50 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:b/+70aj9zQIQv4jOrIj2jtDv5K4=', 'Content-Length': '0'} 2023-05-31T20:53:50.859 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:46:33 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008f87059c14e8d1db-006477b3de-1c17-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:53:50 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:53:50.860 DEBUG:boto:path=/pklxkd-59/obj115 2023-05-31T20:53:50.860 DEBUG:boto:auth_path=/pklxkd-59/obj115 2023-05-31T20:53:50.861 DEBUG:boto:path=/pklxkd-59/obj115?versionId=null 2023-05-31T20:53:50.861 DEBUG:boto:auth_path=/pklxkd-59/obj115?versionId=null 2023-05-31T20:53:50.862 DEBUG:boto:Method: HEAD 2023-05-31T20:53:50.862 DEBUG:boto:Path: /pklxkd-59/obj115?versionId=null 2023-05-31T20:53:50.862 DEBUG:boto:Data: 2023-05-31T20:53:50.863 DEBUG:boto:Headers: {} 2023-05-31T20:53:50.863 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:53:50.863 DEBUG:boto:Port: 8000 2023-05-31T20:53:50.864 DEBUG:boto:Params: {} 2023-05-31T20:53:50.864 DEBUG:boto:Token: None 2023-05-31T20:53:50.865 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:53:50 GMT /pklxkd-59/obj115?versionId=null 2023-05-31T20:53:50.865 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:b/+70aj9zQIQv4jOrIj2jtDv5K4= 2023-05-31T20:53:50.865 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:53:50 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:b/+70aj9zQIQv4jOrIj2jtDv5K4=', 'Content-Length': '0'} 2023-05-31T20:53:50.867 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:46:33 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001d95be81adc5f793-006477b3de-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:53:50 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:53:50.868 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj115 2023-05-31T20:53:50.868 DEBUG:boto:path=/pklxkd-59/obj115 2023-05-31T20:53:50.869 DEBUG:boto:auth_path=/pklxkd-59/obj115 2023-05-31T20:53:50.869 DEBUG:boto:path=/pklxkd-59/obj115?versionId=null 2023-05-31T20:53:50.869 DEBUG:boto:auth_path=/pklxkd-59/obj115?versionId=null 2023-05-31T20:53:50.870 DEBUG:boto:Method: GET 2023-05-31T20:53:50.870 DEBUG:boto:Path: /pklxkd-59/obj115?versionId=null 2023-05-31T20:53:50.871 DEBUG:boto:Data: 2023-05-31T20:53:50.871 DEBUG:boto:Headers: {} 2023-05-31T20:53:50.872 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:53:50.872 DEBUG:boto:Port: 8001 2023-05-31T20:53:50.872 DEBUG:boto:Params: {} 2023-05-31T20:53:50.873 DEBUG:boto:Token: None 2023-05-31T20:53:50.873 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:53:50 GMT /pklxkd-59/obj115?versionId=null 2023-05-31T20:53:50.874 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:YihhPjO8TsyGR72NArN2uTJMkk8= 2023-05-31T20:53:50.874 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:53:50 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:YihhPjO8TsyGR72NArN2uTJMkk8=', 'Content-Length': '0'} 2023-05-31T20:53:50.876 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:46:33 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000084538dc101b31010-006477b3de-1c17-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:53:50 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:53:50.876 DEBUG:boto:path=/pklxkd-59/obj115 2023-05-31T20:53:50.877 DEBUG:boto:auth_path=/pklxkd-59/obj115 2023-05-31T20:53:50.877 DEBUG:boto:path=/pklxkd-59/obj115?versionId=null 2023-05-31T20:53:50.877 DEBUG:boto:auth_path=/pklxkd-59/obj115?versionId=null 2023-05-31T20:53:50.878 DEBUG:boto:Method: GET 2023-05-31T20:53:50.878 DEBUG:boto:Path: /pklxkd-59/obj115?versionId=null 2023-05-31T20:53:50.879 DEBUG:boto:Data: 2023-05-31T20:53:50.879 DEBUG:boto:Headers: {} 2023-05-31T20:53:50.879 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:53:50.880 DEBUG:boto:Port: 8000 2023-05-31T20:53:50.880 DEBUG:boto:Params: {} 2023-05-31T20:53:50.881 DEBUG:boto:Token: None 2023-05-31T20:53:50.881 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:53:50 GMT /pklxkd-59/obj115?versionId=null 2023-05-31T20:53:50.882 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:YihhPjO8TsyGR72NArN2uTJMkk8= 2023-05-31T20:53:50.882 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:53:50 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:YihhPjO8TsyGR72NArN2uTJMkk8=', 'Content-Length': '0'} 2023-05-31T20:53:50.884 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:46:33 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000030c3227df30f27bc-006477b3de-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:53:50 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:53:50.885 DEBUG:boto:path=/pklxkd-59/ 2023-05-31T20:53:50.885 DEBUG:boto:auth_path=/pklxkd-59/ 2023-05-31T20:53:50.886 DEBUG:boto:path=/pklxkd-59/?prefix=obj115 2023-05-31T20:53:50.886 DEBUG:boto:auth_path=/pklxkd-59/?prefix=obj115 2023-05-31T20:53:50.887 DEBUG:boto:Method: GET 2023-05-31T20:53:50.887 DEBUG:boto:Path: /pklxkd-59/?prefix=obj115 2023-05-31T20:53:50.888 DEBUG:boto:Data: 2023-05-31T20:53:50.888 DEBUG:boto:Headers: {} 2023-05-31T20:53:50.888 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:53:50.889 DEBUG:boto:Port: 8001 2023-05-31T20:53:50.889 DEBUG:boto:Params: {} 2023-05-31T20:53:50.890 DEBUG:boto:Token: None 2023-05-31T20:53:50.890 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:53:50 GMT /pklxkd-59/ 2023-05-31T20:53:50.891 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:Q0m7mtbP/vbUm0P5WD8EGCw+sDU= 2023-05-31T20:53:50.891 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:53:50 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:Q0m7mtbP/vbUm0P5WD8EGCw+sDU=', 'Content-Length': '0'} 2023-05-31T20:53:50.893 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f78bd58d9cb6f374-006477b3de-1c17-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:53:50 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:53:50.894 DEBUG:boto:b'pklxkd-59obj1151000falseobj1152023-05-31T20:46:33.958Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:53:50.895 DEBUG:boto:path=/pklxkd-59/ 2023-05-31T20:53:50.895 DEBUG:boto:auth_path=/pklxkd-59/ 2023-05-31T20:53:50.896 DEBUG:boto:path=/pklxkd-59/?prefix=obj115 2023-05-31T20:53:50.896 DEBUG:boto:auth_path=/pklxkd-59/?prefix=obj115 2023-05-31T20:53:50.896 DEBUG:boto:Method: GET 2023-05-31T20:53:50.897 DEBUG:boto:Path: /pklxkd-59/?prefix=obj115 2023-05-31T20:53:50.897 DEBUG:boto:Data: 2023-05-31T20:53:50.898 DEBUG:boto:Headers: {} 2023-05-31T20:53:50.898 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:53:50.899 DEBUG:boto:Port: 8000 2023-05-31T20:53:50.899 DEBUG:boto:Params: {} 2023-05-31T20:53:50.899 DEBUG:boto:Token: None 2023-05-31T20:53:50.900 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:53:50 GMT /pklxkd-59/ 2023-05-31T20:53:50.900 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:Q0m7mtbP/vbUm0P5WD8EGCw+sDU= 2023-05-31T20:53:50.901 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:53:50 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:Q0m7mtbP/vbUm0P5WD8EGCw+sDU=', 'Content-Length': '0'} 2023-05-31T20:53:50.905 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000002e1d1969b86b0797-006477b3de-1630-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:53:50 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:53:50.906 DEBUG:boto:b'pklxkd-59obj1151000falseobj1152023-05-31T20:46:33.958Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:53:50.906 DEBUG:boto:path=/pklxkd-59/obj115 2023-05-31T20:53:50.907 DEBUG:boto:auth_path=/pklxkd-59/obj115 2023-05-31T20:53:50.907 DEBUG:boto:Method: HEAD 2023-05-31T20:53:50.908 DEBUG:boto:Path: /pklxkd-59/obj115 2023-05-31T20:53:50.908 DEBUG:boto:Data: 2023-05-31T20:53:50.909 DEBUG:boto:Headers: {} 2023-05-31T20:53:50.909 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:53:50.909 DEBUG:boto:Port: 8001 2023-05-31T20:53:50.910 DEBUG:boto:Params: {} 2023-05-31T20:53:50.910 DEBUG:boto:Token: None 2023-05-31T20:53:50.910 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:53:50 GMT /pklxkd-59/obj115 2023-05-31T20:53:50.911 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:aO6g4PO6nSMYcJg5CzW2T907z4g= 2023-05-31T20:53:50.912 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:53:50 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:aO6g4PO6nSMYcJg5CzW2T907z4g=', 'Content-Length': '0'} 2023-05-31T20:53:50.913 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:46:33 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000c0a13e0a315ee64-006477b3de-1c17-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:53:50 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:53:50.914 DEBUG:boto:path=/pklxkd-59/obj115 2023-05-31T20:53:50.914 DEBUG:boto:auth_path=/pklxkd-59/obj115 2023-05-31T20:53:50.914 DEBUG:boto:Method: HEAD 2023-05-31T20:53:50.915 DEBUG:boto:Path: /pklxkd-59/obj115 2023-05-31T20:53:50.915 DEBUG:boto:Data: 2023-05-31T20:53:50.916 DEBUG:boto:Headers: {} 2023-05-31T20:53:50.916 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:53:50.916 DEBUG:boto:Port: 8000 2023-05-31T20:53:50.917 DEBUG:boto:Params: {} 2023-05-31T20:53:50.917 DEBUG:boto:Token: None 2023-05-31T20:53:50.918 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:53:50 GMT /pklxkd-59/obj115 2023-05-31T20:53:50.918 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:aO6g4PO6nSMYcJg5CzW2T907z4g= 2023-05-31T20:53:50.919 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:53:50 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:aO6g4PO6nSMYcJg5CzW2T907z4g=', 'Content-Length': '0'} 2023-05-31T20:53:50.920 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:46:33 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000fde515e89e7266eb-006477b3de-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:53:50 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:53:50.921 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj115 2023-05-31T20:53:50.921 DEBUG:boto:path=/pklxkd-59/obj115 2023-05-31T20:53:50.921 DEBUG:boto:auth_path=/pklxkd-59/obj115 2023-05-31T20:53:50.922 DEBUG:boto:Method: GET 2023-05-31T20:53:50.922 DEBUG:boto:Path: /pklxkd-59/obj115 2023-05-31T20:53:50.922 DEBUG:boto:Data: 2023-05-31T20:53:50.923 DEBUG:boto:Headers: {} 2023-05-31T20:53:50.924 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:53:50.924 DEBUG:boto:Port: 8001 2023-05-31T20:53:50.924 DEBUG:boto:Params: {} 2023-05-31T20:53:50.925 DEBUG:boto:Token: None 2023-05-31T20:53:50.925 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:53:50 GMT /pklxkd-59/obj115 2023-05-31T20:53:50.926 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:Cmd20SMzD2PwbAg6lBHQMBja900= 2023-05-31T20:53:50.926 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:53:50 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:Cmd20SMzD2PwbAg6lBHQMBja900=', 'Content-Length': '0'} 2023-05-31T20:53:50.928 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:46:33 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000033837608a9673017-006477b3de-1c17-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:53:50 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:53:50.929 DEBUG:boto:path=/pklxkd-59/obj115 2023-05-31T20:53:50.929 DEBUG:boto:auth_path=/pklxkd-59/obj115 2023-05-31T20:53:50.930 DEBUG:boto:Method: GET 2023-05-31T20:53:50.930 DEBUG:boto:Path: /pklxkd-59/obj115 2023-05-31T20:53:50.931 DEBUG:boto:Data: 2023-05-31T20:53:50.931 DEBUG:boto:Headers: {} 2023-05-31T20:53:50.932 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:53:50.932 DEBUG:boto:Port: 8000 2023-05-31T20:53:50.932 DEBUG:boto:Params: {} 2023-05-31T20:53:50.933 DEBUG:boto:Token: None 2023-05-31T20:53:50.933 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:53:50 GMT /pklxkd-59/obj115 2023-05-31T20:53:50.934 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:Cmd20SMzD2PwbAg6lBHQMBja900= 2023-05-31T20:53:50.934 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:53:50 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:Cmd20SMzD2PwbAg6lBHQMBja900=', 'Content-Length': '0'} 2023-05-31T20:53:50.936 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:46:33 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e4c38fddb329ab50-006477b3de-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:53:50 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:53:50.976 DEBUG:boto:path=/pklxkd-59/ 2023-05-31T20:53:50.977 DEBUG:boto:auth_path=/pklxkd-59/ 2023-05-31T20:53:50.977 DEBUG:boto:path=/pklxkd-59/?prefix=obj115 2023-05-31T20:53:50.978 DEBUG:boto:auth_path=/pklxkd-59/?prefix=obj115 2023-05-31T20:53:50.978 DEBUG:boto:Method: GET 2023-05-31T20:53:50.979 DEBUG:boto:Path: /pklxkd-59/?prefix=obj115 2023-05-31T20:53:50.979 DEBUG:boto:Data: 2023-05-31T20:53:50.979 DEBUG:boto:Headers: {} 2023-05-31T20:53:50.980 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:53:50.980 DEBUG:boto:Port: 8001 2023-05-31T20:53:50.981 DEBUG:boto:Params: {} 2023-05-31T20:53:50.981 DEBUG:boto:Token: None 2023-05-31T20:53:50.982 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:53:50 GMT /pklxkd-59/ 2023-05-31T20:53:50.982 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:Q0m7mtbP/vbUm0P5WD8EGCw+sDU= 2023-05-31T20:53:50.983 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:53:50 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:Q0m7mtbP/vbUm0P5WD8EGCw+sDU=', 'Content-Length': '0'} 2023-05-31T20:53:50.985 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000008e32a394045a622e-006477b3de-1c17-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:53:50 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:53:50.986 DEBUG:boto:b'pklxkd-59obj1151000falseobj1152023-05-31T20:46:33.958Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:53:50.986 DEBUG:boto:path=/pklxkd-59/ 2023-05-31T20:53:50.987 DEBUG:boto:auth_path=/pklxkd-59/ 2023-05-31T20:53:50.987 DEBUG:boto:path=/pklxkd-59/?prefix=obj115 2023-05-31T20:53:50.988 DEBUG:boto:auth_path=/pklxkd-59/?prefix=obj115 2023-05-31T20:53:50.988 DEBUG:boto:Method: GET 2023-05-31T20:53:50.988 DEBUG:boto:Path: /pklxkd-59/?prefix=obj115 2023-05-31T20:53:50.989 DEBUG:boto:Data: 2023-05-31T20:53:50.989 DEBUG:boto:Headers: {} 2023-05-31T20:53:50.989 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:53:50.990 DEBUG:boto:Port: 8000 2023-05-31T20:53:50.990 DEBUG:boto:Params: {} 2023-05-31T20:53:50.991 DEBUG:boto:Token: None 2023-05-31T20:53:50.991 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:53:50 GMT /pklxkd-59/ 2023-05-31T20:53:50.992 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:Q0m7mtbP/vbUm0P5WD8EGCw+sDU= 2023-05-31T20:53:50.992 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:53:50 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:Q0m7mtbP/vbUm0P5WD8EGCw+sDU=', 'Content-Length': '0'} 2023-05-31T20:53:50.996 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000003ba55c535c0c2b1d-006477b3de-1630-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:53:50 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:53:50.996 DEBUG:boto:b'pklxkd-59obj1151000falseobj1152023-05-31T20:46:33.958Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:53:50.997 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj116 2023-05-31T20:53:50.998 DEBUG:boto:path=/pklxkd-59/obj116 2023-05-31T20:53:50.998 DEBUG:boto:auth_path=/pklxkd-59/obj116 2023-05-31T20:53:50.998 DEBUG:boto:path=/pklxkd-59/obj116?versionId=null 2023-05-31T20:53:50.999 DEBUG:boto:auth_path=/pklxkd-59/obj116?versionId=null 2023-05-31T20:53:50.999 DEBUG:boto:Method: GET 2023-05-31T20:53:50.999 DEBUG:boto:Path: /pklxkd-59/obj116?versionId=null 2023-05-31T20:53:51.000 DEBUG:boto:Data: 2023-05-31T20:53:51.000 DEBUG:boto:Headers: {} 2023-05-31T20:53:51.001 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:53:51.001 DEBUG:boto:Port: 8001 2023-05-31T20:53:51.002 DEBUG:boto:Params: {} 2023-05-31T20:53:51.002 DEBUG:boto:Token: None 2023-05-31T20:53:51.002 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:53:51 GMT /pklxkd-59/obj116?versionId=null 2023-05-31T20:53:51.003 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:7G5biF3QOefIdHdtJVYHed3juE0= 2023-05-31T20:53:51.003 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:53:51 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:7G5biF3QOefIdHdtJVYHed3juE0=', 'Content-Length': '0'} 2023-05-31T20:53:51.005 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:48:13 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000038346983112545d0-006477b3df-1c17-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:53:51 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:53:51.048 DEBUG:boto:path=/pklxkd-59/obj116 2023-05-31T20:53:51.049 DEBUG:boto:auth_path=/pklxkd-59/obj116 2023-05-31T20:53:51.049 DEBUG:boto:path=/pklxkd-59/obj116?versionId=null 2023-05-31T20:53:51.049 DEBUG:boto:auth_path=/pklxkd-59/obj116?versionId=null 2023-05-31T20:53:51.050 DEBUG:boto:Method: GET 2023-05-31T20:53:51.050 DEBUG:boto:Path: /pklxkd-59/obj116?versionId=null 2023-05-31T20:53:51.050 DEBUG:boto:Data: 2023-05-31T20:53:51.051 DEBUG:boto:Headers: {} 2023-05-31T20:53:51.051 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:53:51.052 DEBUG:boto:Port: 8000 2023-05-31T20:53:51.052 DEBUG:boto:Params: {} 2023-05-31T20:53:51.053 DEBUG:boto:Token: None 2023-05-31T20:53:51.053 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:53:51 GMT /pklxkd-59/obj116?versionId=null 2023-05-31T20:53:51.054 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:7G5biF3QOefIdHdtJVYHed3juE0= 2023-05-31T20:53:51.054 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:53:51 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:7G5biF3QOefIdHdtJVYHed3juE0=', 'Content-Length': '0'} 2023-05-31T20:53:51.056 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:48:13 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000072291c18894465ec-006477b3df-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:53:51 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:53:51.057 DEBUG:boto:path=/pklxkd-59/obj116 2023-05-31T20:53:51.057 DEBUG:boto:auth_path=/pklxkd-59/obj116 2023-05-31T20:53:51.058 DEBUG:boto:path=/pklxkd-59/obj116?versionId=null 2023-05-31T20:53:51.058 DEBUG:boto:auth_path=/pklxkd-59/obj116?versionId=null 2023-05-31T20:53:51.058 DEBUG:boto:Method: HEAD 2023-05-31T20:53:51.059 DEBUG:boto:Path: /pklxkd-59/obj116?versionId=null 2023-05-31T20:53:51.059 DEBUG:boto:Data: 2023-05-31T20:53:51.059 DEBUG:boto:Headers: {} 2023-05-31T20:53:51.060 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:53:51.060 DEBUG:boto:Port: 8001 2023-05-31T20:53:51.060 DEBUG:boto:Params: {} 2023-05-31T20:53:51.061 DEBUG:boto:Token: None 2023-05-31T20:53:51.061 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:53:51 GMT /pklxkd-59/obj116?versionId=null 2023-05-31T20:53:51.062 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:vReyt0q63aH5szMR4TRIunV6ZUc= 2023-05-31T20:53:51.062 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:53:51 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:vReyt0q63aH5szMR4TRIunV6ZUc=', 'Content-Length': '0'} 2023-05-31T20:53:51.064 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:48:13 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000059709d42fb0820b4-006477b3df-1c17-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:53:51 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:53:51.064 DEBUG:boto:path=/pklxkd-59/obj116 2023-05-31T20:53:51.065 DEBUG:boto:auth_path=/pklxkd-59/obj116 2023-05-31T20:53:51.065 DEBUG:boto:path=/pklxkd-59/obj116?versionId=null 2023-05-31T20:53:51.066 DEBUG:boto:auth_path=/pklxkd-59/obj116?versionId=null 2023-05-31T20:53:51.066 DEBUG:boto:Method: HEAD 2023-05-31T20:53:51.066 DEBUG:boto:Path: /pklxkd-59/obj116?versionId=null 2023-05-31T20:53:51.067 DEBUG:boto:Data: 2023-05-31T20:53:51.067 DEBUG:boto:Headers: {} 2023-05-31T20:53:51.067 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:53:51.068 DEBUG:boto:Port: 8000 2023-05-31T20:53:51.068 DEBUG:boto:Params: {} 2023-05-31T20:53:51.068 DEBUG:boto:Token: None 2023-05-31T20:53:51.069 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:53:51 GMT /pklxkd-59/obj116?versionId=null 2023-05-31T20:53:51.069 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:vReyt0q63aH5szMR4TRIunV6ZUc= 2023-05-31T20:53:51.070 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:53:51 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:vReyt0q63aH5szMR4TRIunV6ZUc=', 'Content-Length': '0'} 2023-05-31T20:53:51.071 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:48:13 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000010084443dd52d9c0-006477b3df-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:53:51 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:53:51.072 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj116 2023-05-31T20:53:51.072 DEBUG:boto:path=/pklxkd-59/obj116 2023-05-31T20:53:51.073 DEBUG:boto:auth_path=/pklxkd-59/obj116 2023-05-31T20:53:51.073 DEBUG:boto:path=/pklxkd-59/obj116?versionId=null 2023-05-31T20:53:51.074 DEBUG:boto:auth_path=/pklxkd-59/obj116?versionId=null 2023-05-31T20:53:51.074 DEBUG:boto:Method: GET 2023-05-31T20:53:51.074 DEBUG:boto:Path: /pklxkd-59/obj116?versionId=null 2023-05-31T20:53:51.075 DEBUG:boto:Data: 2023-05-31T20:53:51.075 DEBUG:boto:Headers: {} 2023-05-31T20:53:51.075 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:53:51.076 DEBUG:boto:Port: 8001 2023-05-31T20:53:51.076 DEBUG:boto:Params: {} 2023-05-31T20:53:51.077 DEBUG:boto:Token: None 2023-05-31T20:53:51.077 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:53:51 GMT /pklxkd-59/obj116?versionId=null 2023-05-31T20:53:51.078 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:7G5biF3QOefIdHdtJVYHed3juE0= 2023-05-31T20:53:51.078 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:53:51 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:7G5biF3QOefIdHdtJVYHed3juE0=', 'Content-Length': '0'} 2023-05-31T20:53:51.080 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:48:13 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008995a381d7eee099-006477b3df-1c17-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:53:51 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:53:51.081 DEBUG:boto:path=/pklxkd-59/obj116 2023-05-31T20:53:51.081 DEBUG:boto:auth_path=/pklxkd-59/obj116 2023-05-31T20:53:51.082 DEBUG:boto:path=/pklxkd-59/obj116?versionId=null 2023-05-31T20:53:51.082 DEBUG:boto:auth_path=/pklxkd-59/obj116?versionId=null 2023-05-31T20:53:51.083 DEBUG:boto:Method: GET 2023-05-31T20:53:51.083 DEBUG:boto:Path: /pklxkd-59/obj116?versionId=null 2023-05-31T20:53:51.084 DEBUG:boto:Data: 2023-05-31T20:53:51.084 DEBUG:boto:Headers: {} 2023-05-31T20:53:51.085 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:53:51.085 DEBUG:boto:Port: 8000 2023-05-31T20:53:51.086 DEBUG:boto:Params: {} 2023-05-31T20:53:51.086 DEBUG:boto:Token: None 2023-05-31T20:53:51.087 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:53:51 GMT /pklxkd-59/obj116?versionId=null 2023-05-31T20:53:51.087 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:7G5biF3QOefIdHdtJVYHed3juE0= 2023-05-31T20:53:51.087 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:53:51 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:7G5biF3QOefIdHdtJVYHed3juE0=', 'Content-Length': '0'} 2023-05-31T20:53:51.089 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:48:13 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004f122764a5692840-006477b3df-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:53:51 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:53:51.090 DEBUG:boto:path=/pklxkd-59/ 2023-05-31T20:53:51.090 DEBUG:boto:auth_path=/pklxkd-59/ 2023-05-31T20:53:51.090 DEBUG:boto:path=/pklxkd-59/?prefix=obj116 2023-05-31T20:53:51.091 DEBUG:boto:auth_path=/pklxkd-59/?prefix=obj116 2023-05-31T20:53:51.091 DEBUG:boto:Method: GET 2023-05-31T20:53:51.092 DEBUG:boto:Path: /pklxkd-59/?prefix=obj116 2023-05-31T20:53:51.092 DEBUG:boto:Data: 2023-05-31T20:53:51.093 DEBUG:boto:Headers: {} 2023-05-31T20:53:51.093 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:53:51.093 DEBUG:boto:Port: 8001 2023-05-31T20:53:51.094 DEBUG:boto:Params: {} 2023-05-31T20:53:51.094 DEBUG:boto:Token: None 2023-05-31T20:53:51.095 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:53:51 GMT /pklxkd-59/ 2023-05-31T20:53:51.095 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:lmsrlNekY57vtRsTzICLOlyPcM0= 2023-05-31T20:53:51.095 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:53:51 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:lmsrlNekY57vtRsTzICLOlyPcM0=', 'Content-Length': '0'} 2023-05-31T20:53:51.098 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d51c5fdb9f447308-006477b3df-1c17-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:53:51 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:53:51.098 DEBUG:boto:b'pklxkd-59obj1161000falseobj1162023-05-31T20:48:13.052Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:53:51.099 DEBUG:boto:path=/pklxkd-59/ 2023-05-31T20:53:51.099 DEBUG:boto:auth_path=/pklxkd-59/ 2023-05-31T20:53:51.100 DEBUG:boto:path=/pklxkd-59/?prefix=obj116 2023-05-31T20:53:51.100 DEBUG:boto:auth_path=/pklxkd-59/?prefix=obj116 2023-05-31T20:53:51.100 DEBUG:boto:Method: GET 2023-05-31T20:53:51.101 DEBUG:boto:Path: /pklxkd-59/?prefix=obj116 2023-05-31T20:53:51.101 DEBUG:boto:Data: 2023-05-31T20:53:51.101 DEBUG:boto:Headers: {} 2023-05-31T20:53:51.102 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:53:51.102 DEBUG:boto:Port: 8000 2023-05-31T20:53:51.102 DEBUG:boto:Params: {} 2023-05-31T20:53:51.103 DEBUG:boto:Token: None 2023-05-31T20:53:51.103 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:53:51 GMT /pklxkd-59/ 2023-05-31T20:53:51.104 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:lmsrlNekY57vtRsTzICLOlyPcM0= 2023-05-31T20:53:51.104 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:53:51 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:lmsrlNekY57vtRsTzICLOlyPcM0=', 'Content-Length': '0'} 2023-05-31T20:53:51.108 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000003454875361af919c-006477b3df-1630-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:53:51 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:53:51.109 DEBUG:boto:b'pklxkd-59obj1161000falseobj1162023-05-31T20:48:13.052Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:53:51.110 DEBUG:boto:path=/pklxkd-59/obj116 2023-05-31T20:53:51.110 DEBUG:boto:auth_path=/pklxkd-59/obj116 2023-05-31T20:53:51.110 DEBUG:boto:Method: HEAD 2023-05-31T20:53:51.111 DEBUG:boto:Path: /pklxkd-59/obj116 2023-05-31T20:53:51.111 DEBUG:boto:Data: 2023-05-31T20:53:51.111 DEBUG:boto:Headers: {} 2023-05-31T20:53:51.112 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:53:51.112 DEBUG:boto:Port: 8001 2023-05-31T20:53:51.112 DEBUG:boto:Params: {} 2023-05-31T20:53:51.113 DEBUG:boto:Token: None 2023-05-31T20:53:51.113 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:53:51 GMT /pklxkd-59/obj116 2023-05-31T20:53:51.113 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:IiJKDDQl1op1iKNd2pzJOR3zZLo= 2023-05-31T20:53:51.114 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:53:51 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:IiJKDDQl1op1iKNd2pzJOR3zZLo=', 'Content-Length': '0'} 2023-05-31T20:53:51.116 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:48:13 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002c64fd3982c64ff4-006477b3df-1c17-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:53:51 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:53:51.116 DEBUG:boto:path=/pklxkd-59/obj116 2023-05-31T20:53:51.117 DEBUG:boto:auth_path=/pklxkd-59/obj116 2023-05-31T20:53:51.117 DEBUG:boto:Method: HEAD 2023-05-31T20:53:51.117 DEBUG:boto:Path: /pklxkd-59/obj116 2023-05-31T20:53:51.118 DEBUG:boto:Data: 2023-05-31T20:53:51.118 DEBUG:boto:Headers: {} 2023-05-31T20:53:51.118 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:53:51.119 DEBUG:boto:Port: 8000 2023-05-31T20:53:51.119 DEBUG:boto:Params: {} 2023-05-31T20:53:51.119 DEBUG:boto:Token: None 2023-05-31T20:53:51.120 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:53:51 GMT /pklxkd-59/obj116 2023-05-31T20:53:51.120 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:IiJKDDQl1op1iKNd2pzJOR3zZLo= 2023-05-31T20:53:51.120 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:53:51 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:IiJKDDQl1op1iKNd2pzJOR3zZLo=', 'Content-Length': '0'} 2023-05-31T20:53:51.122 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:48:13 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000036d702936078f4af-006477b3df-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:53:51 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:53:51.123 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj116 2023-05-31T20:53:51.123 DEBUG:boto:path=/pklxkd-59/obj116 2023-05-31T20:53:51.123 DEBUG:boto:auth_path=/pklxkd-59/obj116 2023-05-31T20:53:51.124 DEBUG:boto:Method: GET 2023-05-31T20:53:51.124 DEBUG:boto:Path: /pklxkd-59/obj116 2023-05-31T20:53:51.125 DEBUG:boto:Data: 2023-05-31T20:53:51.125 DEBUG:boto:Headers: {} 2023-05-31T20:53:51.125 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:53:51.126 DEBUG:boto:Port: 8001 2023-05-31T20:53:51.126 DEBUG:boto:Params: {} 2023-05-31T20:53:51.126 DEBUG:boto:Token: None 2023-05-31T20:53:51.127 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:53:51 GMT /pklxkd-59/obj116 2023-05-31T20:53:51.127 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:qroFufNLgJQfsFWH5Zlc2Vu6HEo= 2023-05-31T20:53:51.128 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:53:51 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:qroFufNLgJQfsFWH5Zlc2Vu6HEo=', 'Content-Length': '0'} 2023-05-31T20:53:51.130 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:48:13 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007c78a9bb39bdca9e-006477b3df-1c17-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:53:51 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:53:51.131 DEBUG:boto:path=/pklxkd-59/obj116 2023-05-31T20:53:51.131 DEBUG:boto:auth_path=/pklxkd-59/obj116 2023-05-31T20:53:51.131 DEBUG:boto:Method: GET 2023-05-31T20:53:51.132 DEBUG:boto:Path: /pklxkd-59/obj116 2023-05-31T20:53:51.132 DEBUG:boto:Data: 2023-05-31T20:53:51.133 DEBUG:boto:Headers: {} 2023-05-31T20:53:51.133 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:53:51.133 DEBUG:boto:Port: 8000 2023-05-31T20:53:51.134 DEBUG:boto:Params: {} 2023-05-31T20:53:51.134 DEBUG:boto:Token: None 2023-05-31T20:53:51.134 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:53:51 GMT /pklxkd-59/obj116 2023-05-31T20:53:51.135 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:qroFufNLgJQfsFWH5Zlc2Vu6HEo= 2023-05-31T20:53:51.135 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:53:51 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:qroFufNLgJQfsFWH5Zlc2Vu6HEo=', 'Content-Length': '0'} 2023-05-31T20:53:51.137 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:48:13 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f885a69f3ae999a9-006477b3df-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:53:51 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:53:51.180 DEBUG:boto:path=/pklxkd-59/ 2023-05-31T20:53:51.181 DEBUG:boto:auth_path=/pklxkd-59/ 2023-05-31T20:53:51.181 DEBUG:boto:path=/pklxkd-59/?prefix=obj116 2023-05-31T20:53:51.182 DEBUG:boto:auth_path=/pklxkd-59/?prefix=obj116 2023-05-31T20:53:51.182 DEBUG:boto:Method: GET 2023-05-31T20:53:51.182 DEBUG:boto:Path: /pklxkd-59/?prefix=obj116 2023-05-31T20:53:51.183 DEBUG:boto:Data: 2023-05-31T20:53:51.183 DEBUG:boto:Headers: {} 2023-05-31T20:53:51.183 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:53:51.184 DEBUG:boto:Port: 8001 2023-05-31T20:53:51.184 DEBUG:boto:Params: {} 2023-05-31T20:53:51.185 DEBUG:boto:Token: None 2023-05-31T20:53:51.190 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:53:51 GMT /pklxkd-59/ 2023-05-31T20:53:51.190 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:lmsrlNekY57vtRsTzICLOlyPcM0= 2023-05-31T20:53:51.191 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:53:51 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:lmsrlNekY57vtRsTzICLOlyPcM0=', 'Content-Length': '0'} 2023-05-31T20:53:51.193 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e78e67c54e478d98-006477b3df-1c17-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:53:51 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:53:51.194 DEBUG:boto:b'pklxkd-59obj1161000falseobj1162023-05-31T20:48:13.052Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:53:51.194 DEBUG:boto:path=/pklxkd-59/ 2023-05-31T20:53:51.195 DEBUG:boto:auth_path=/pklxkd-59/ 2023-05-31T20:53:51.195 DEBUG:boto:path=/pklxkd-59/?prefix=obj116 2023-05-31T20:53:51.196 DEBUG:boto:auth_path=/pklxkd-59/?prefix=obj116 2023-05-31T20:53:51.196 DEBUG:boto:Method: GET 2023-05-31T20:53:51.196 DEBUG:boto:Path: /pklxkd-59/?prefix=obj116 2023-05-31T20:53:51.197 DEBUG:boto:Data: 2023-05-31T20:53:51.197 DEBUG:boto:Headers: {} 2023-05-31T20:53:51.197 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:53:51.198 DEBUG:boto:Port: 8000 2023-05-31T20:53:51.198 DEBUG:boto:Params: {} 2023-05-31T20:53:51.199 DEBUG:boto:Token: None 2023-05-31T20:53:51.199 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:53:51 GMT /pklxkd-59/ 2023-05-31T20:53:51.199 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:lmsrlNekY57vtRsTzICLOlyPcM0= 2023-05-31T20:53:51.200 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:53:51 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:lmsrlNekY57vtRsTzICLOlyPcM0=', 'Content-Length': '0'} 2023-05-31T20:53:51.204 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000007a2c92951fe6a547-006477b3df-1630-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:53:51 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:53:51.205 DEBUG:boto:b'pklxkd-59obj1161000falseobj1162023-05-31T20:48:13.052Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:53:51.205 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj124 2023-05-31T20:53:51.206 DEBUG:boto:path=/pklxkd-59/obj124 2023-05-31T20:53:51.206 DEBUG:boto:auth_path=/pklxkd-59/obj124 2023-05-31T20:53:51.206 DEBUG:boto:path=/pklxkd-59/obj124?versionId=null 2023-05-31T20:53:51.207 DEBUG:boto:auth_path=/pklxkd-59/obj124?versionId=null 2023-05-31T20:53:51.207 DEBUG:boto:Method: GET 2023-05-31T20:53:51.207 DEBUG:boto:Path: /pklxkd-59/obj124?versionId=null 2023-05-31T20:53:51.208 DEBUG:boto:Data: 2023-05-31T20:53:51.208 DEBUG:boto:Headers: {} 2023-05-31T20:53:51.209 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:53:51.209 DEBUG:boto:Port: 8001 2023-05-31T20:53:51.209 DEBUG:boto:Params: {} 2023-05-31T20:53:51.210 DEBUG:boto:Token: None 2023-05-31T20:53:51.210 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:53:51 GMT /pklxkd-59/obj124?versionId=null 2023-05-31T20:53:51.211 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:OXGevRlkDONxVzCI1+Q3mUIeLxE= 2023-05-31T20:53:51.211 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:53:51 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:OXGevRlkDONxVzCI1+Q3mUIeLxE=', 'Content-Length': '0'} 2023-05-31T20:53:51.213 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:50:33 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000710036abc60b8ddf-006477b3df-1c17-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:53:51 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:53:51.256 DEBUG:boto:path=/pklxkd-59/obj124 2023-05-31T20:53:51.257 DEBUG:boto:auth_path=/pklxkd-59/obj124 2023-05-31T20:53:51.257 DEBUG:boto:path=/pklxkd-59/obj124?versionId=null 2023-05-31T20:53:51.257 DEBUG:boto:auth_path=/pklxkd-59/obj124?versionId=null 2023-05-31T20:53:51.258 DEBUG:boto:Method: GET 2023-05-31T20:53:51.258 DEBUG:boto:Path: /pklxkd-59/obj124?versionId=null 2023-05-31T20:53:51.258 DEBUG:boto:Data: 2023-05-31T20:53:51.259 DEBUG:boto:Headers: {} 2023-05-31T20:53:51.259 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:53:51.259 DEBUG:boto:Port: 8000 2023-05-31T20:53:51.260 DEBUG:boto:Params: {} 2023-05-31T20:53:51.260 DEBUG:boto:Token: None 2023-05-31T20:53:51.261 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:53:51 GMT /pklxkd-59/obj124?versionId=null 2023-05-31T20:53:51.261 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:OXGevRlkDONxVzCI1+Q3mUIeLxE= 2023-05-31T20:53:51.261 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:53:51 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:OXGevRlkDONxVzCI1+Q3mUIeLxE=', 'Content-Length': '0'} 2023-05-31T20:53:51.263 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:50:33 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003b116f9f7f29a27d-006477b3df-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:53:51 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:53:51.264 DEBUG:boto:path=/pklxkd-59/obj124 2023-05-31T20:53:51.265 DEBUG:boto:auth_path=/pklxkd-59/obj124 2023-05-31T20:53:51.265 DEBUG:boto:path=/pklxkd-59/obj124?versionId=null 2023-05-31T20:53:51.265 DEBUG:boto:auth_path=/pklxkd-59/obj124?versionId=null 2023-05-31T20:53:51.266 DEBUG:boto:Method: HEAD 2023-05-31T20:53:51.266 DEBUG:boto:Path: /pklxkd-59/obj124?versionId=null 2023-05-31T20:53:51.266 DEBUG:boto:Data: 2023-05-31T20:53:51.267 DEBUG:boto:Headers: {} 2023-05-31T20:53:51.267 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:53:51.267 DEBUG:boto:Port: 8001 2023-05-31T20:53:51.268 DEBUG:boto:Params: {} 2023-05-31T20:53:51.268 DEBUG:boto:Token: None 2023-05-31T20:53:51.268 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:53:51 GMT /pklxkd-59/obj124?versionId=null 2023-05-31T20:53:51.269 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:D/JaJB9phAOeKYlRVQy/4aP6Ig8= 2023-05-31T20:53:51.269 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:53:51 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:D/JaJB9phAOeKYlRVQy/4aP6Ig8=', 'Content-Length': '0'} 2023-05-31T20:53:51.271 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:50:33 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d569c51c5073fb22-006477b3df-1c17-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:53:51 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:53:51.271 DEBUG:boto:path=/pklxkd-59/obj124 2023-05-31T20:53:51.272 DEBUG:boto:auth_path=/pklxkd-59/obj124 2023-05-31T20:53:51.272 DEBUG:boto:path=/pklxkd-59/obj124?versionId=null 2023-05-31T20:53:51.272 DEBUG:boto:auth_path=/pklxkd-59/obj124?versionId=null 2023-05-31T20:53:51.273 DEBUG:boto:Method: HEAD 2023-05-31T20:53:51.273 DEBUG:boto:Path: /pklxkd-59/obj124?versionId=null 2023-05-31T20:53:51.273 DEBUG:boto:Data: 2023-05-31T20:53:51.274 DEBUG:boto:Headers: {} 2023-05-31T20:53:51.274 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:53:51.274 DEBUG:boto:Port: 8000 2023-05-31T20:53:51.275 DEBUG:boto:Params: {} 2023-05-31T20:53:51.275 DEBUG:boto:Token: None 2023-05-31T20:53:51.275 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:53:51 GMT /pklxkd-59/obj124?versionId=null 2023-05-31T20:53:51.276 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:D/JaJB9phAOeKYlRVQy/4aP6Ig8= 2023-05-31T20:53:51.276 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:53:51 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:D/JaJB9phAOeKYlRVQy/4aP6Ig8=', 'Content-Length': '0'} 2023-05-31T20:53:51.278 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:50:33 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001a308c0e69b44434-006477b3df-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:53:51 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:53:51.279 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj124 2023-05-31T20:53:51.279 DEBUG:boto:path=/pklxkd-59/obj124 2023-05-31T20:53:51.279 DEBUG:boto:auth_path=/pklxkd-59/obj124 2023-05-31T20:53:51.280 DEBUG:boto:path=/pklxkd-59/obj124?versionId=null 2023-05-31T20:53:51.280 DEBUG:boto:auth_path=/pklxkd-59/obj124?versionId=null 2023-05-31T20:53:51.281 DEBUG:boto:Method: GET 2023-05-31T20:53:51.281 DEBUG:boto:Path: /pklxkd-59/obj124?versionId=null 2023-05-31T20:53:51.281 DEBUG:boto:Data: 2023-05-31T20:53:51.282 DEBUG:boto:Headers: {} 2023-05-31T20:53:51.282 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:53:51.282 DEBUG:boto:Port: 8001 2023-05-31T20:53:51.283 DEBUG:boto:Params: {} 2023-05-31T20:53:51.283 DEBUG:boto:Token: None 2023-05-31T20:53:51.284 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:53:51 GMT /pklxkd-59/obj124?versionId=null 2023-05-31T20:53:51.284 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:OXGevRlkDONxVzCI1+Q3mUIeLxE= 2023-05-31T20:53:51.284 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:53:51 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:OXGevRlkDONxVzCI1+Q3mUIeLxE=', 'Content-Length': '0'} 2023-05-31T20:53:51.286 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:50:33 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007c073fa58fabddcd-006477b3df-1c17-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:53:51 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:53:51.287 DEBUG:boto:path=/pklxkd-59/obj124 2023-05-31T20:53:51.287 DEBUG:boto:auth_path=/pklxkd-59/obj124 2023-05-31T20:53:51.287 DEBUG:boto:path=/pklxkd-59/obj124?versionId=null 2023-05-31T20:53:51.288 DEBUG:boto:auth_path=/pklxkd-59/obj124?versionId=null 2023-05-31T20:53:51.288 DEBUG:boto:Method: GET 2023-05-31T20:53:51.289 DEBUG:boto:Path: /pklxkd-59/obj124?versionId=null 2023-05-31T20:53:51.289 DEBUG:boto:Data: 2023-05-31T20:53:51.289 DEBUG:boto:Headers: {} 2023-05-31T20:53:51.290 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:53:51.290 DEBUG:boto:Port: 8000 2023-05-31T20:53:51.290 DEBUG:boto:Params: {} 2023-05-31T20:53:51.291 DEBUG:boto:Token: None 2023-05-31T20:53:51.291 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:53:51 GMT /pklxkd-59/obj124?versionId=null 2023-05-31T20:53:51.292 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:OXGevRlkDONxVzCI1+Q3mUIeLxE= 2023-05-31T20:53:51.292 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:53:51 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:OXGevRlkDONxVzCI1+Q3mUIeLxE=', 'Content-Length': '0'} 2023-05-31T20:53:51.294 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:50:33 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a3657fb92bb54b44-006477b3df-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:53:51 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:53:51.295 DEBUG:boto:path=/pklxkd-59/ 2023-05-31T20:53:51.295 DEBUG:boto:auth_path=/pklxkd-59/ 2023-05-31T20:53:51.295 DEBUG:boto:path=/pklxkd-59/?prefix=obj124 2023-05-31T20:53:51.296 DEBUG:boto:auth_path=/pklxkd-59/?prefix=obj124 2023-05-31T20:53:51.296 DEBUG:boto:Method: GET 2023-05-31T20:53:51.296 DEBUG:boto:Path: /pklxkd-59/?prefix=obj124 2023-05-31T20:53:51.297 DEBUG:boto:Data: 2023-05-31T20:53:51.297 DEBUG:boto:Headers: {} 2023-05-31T20:53:51.298 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:53:51.298 DEBUG:boto:Port: 8001 2023-05-31T20:53:51.298 DEBUG:boto:Params: {} 2023-05-31T20:53:51.299 DEBUG:boto:Token: None 2023-05-31T20:53:51.299 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:53:51 GMT /pklxkd-59/ 2023-05-31T20:53:51.299 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:lmsrlNekY57vtRsTzICLOlyPcM0= 2023-05-31T20:53:51.300 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:53:51 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:lmsrlNekY57vtRsTzICLOlyPcM0=', 'Content-Length': '0'} 2023-05-31T20:53:51.302 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000890873d7efbb5c8a-006477b3df-1c17-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:53:51 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:53:51.303 DEBUG:boto:b'pklxkd-59obj1241000falseobj1242023-05-31T20:50:33.587Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:53:51.303 DEBUG:boto:path=/pklxkd-59/ 2023-05-31T20:53:51.304 DEBUG:boto:auth_path=/pklxkd-59/ 2023-05-31T20:53:51.304 DEBUG:boto:path=/pklxkd-59/?prefix=obj124 2023-05-31T20:53:51.304 DEBUG:boto:auth_path=/pklxkd-59/?prefix=obj124 2023-05-31T20:53:51.305 DEBUG:boto:Method: GET 2023-05-31T20:53:51.305 DEBUG:boto:Path: /pklxkd-59/?prefix=obj124 2023-05-31T20:53:51.305 DEBUG:boto:Data: 2023-05-31T20:53:51.306 DEBUG:boto:Headers: {} 2023-05-31T20:53:51.306 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:53:51.307 DEBUG:boto:Port: 8000 2023-05-31T20:53:51.307 DEBUG:boto:Params: {} 2023-05-31T20:53:51.307 DEBUG:boto:Token: None 2023-05-31T20:53:51.308 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:53:51 GMT /pklxkd-59/ 2023-05-31T20:53:51.308 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:lmsrlNekY57vtRsTzICLOlyPcM0= 2023-05-31T20:53:51.309 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:53:51 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:lmsrlNekY57vtRsTzICLOlyPcM0=', 'Content-Length': '0'} 2023-05-31T20:53:51.312 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000af3ca0da2dd8cdf3-006477b3df-1630-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:53:51 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:53:51.313 DEBUG:boto:b'pklxkd-59obj1241000falseobj1242023-05-31T20:50:33.587Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:53:51.314 DEBUG:boto:path=/pklxkd-59/obj124 2023-05-31T20:53:51.314 DEBUG:boto:auth_path=/pklxkd-59/obj124 2023-05-31T20:53:51.314 DEBUG:boto:Method: HEAD 2023-05-31T20:53:51.315 DEBUG:boto:Path: /pklxkd-59/obj124 2023-05-31T20:53:51.315 DEBUG:boto:Data: 2023-05-31T20:53:51.315 DEBUG:boto:Headers: {} 2023-05-31T20:53:51.316 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:53:51.316 DEBUG:boto:Port: 8001 2023-05-31T20:53:51.316 DEBUG:boto:Params: {} 2023-05-31T20:53:51.317 DEBUG:boto:Token: None 2023-05-31T20:53:51.317 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:53:51 GMT /pklxkd-59/obj124 2023-05-31T20:53:51.318 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:ihK7ObMZOAIhbOKGMSUhjDwiQSM= 2023-05-31T20:53:51.318 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:53:51 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:ihK7ObMZOAIhbOKGMSUhjDwiQSM=', 'Content-Length': '0'} 2023-05-31T20:53:51.320 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:50:33 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000066896bb2f5d76c29-006477b3df-1c17-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:53:51 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:53:51.320 DEBUG:boto:path=/pklxkd-59/obj124 2023-05-31T20:53:51.321 DEBUG:boto:auth_path=/pklxkd-59/obj124 2023-05-31T20:53:51.321 DEBUG:boto:Method: HEAD 2023-05-31T20:53:51.321 DEBUG:boto:Path: /pklxkd-59/obj124 2023-05-31T20:53:51.322 DEBUG:boto:Data: 2023-05-31T20:53:51.322 DEBUG:boto:Headers: {} 2023-05-31T20:53:51.322 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:53:51.323 DEBUG:boto:Port: 8000 2023-05-31T20:53:51.323 DEBUG:boto:Params: {} 2023-05-31T20:53:51.323 DEBUG:boto:Token: None 2023-05-31T20:53:51.324 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:53:51 GMT /pklxkd-59/obj124 2023-05-31T20:53:51.324 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:ihK7ObMZOAIhbOKGMSUhjDwiQSM= 2023-05-31T20:53:51.325 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:53:51 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:ihK7ObMZOAIhbOKGMSUhjDwiQSM=', 'Content-Length': '0'} 2023-05-31T20:53:51.326 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:50:33 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e30c88438b5da90a-006477b3df-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:53:51 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:53:51.327 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj124 2023-05-31T20:53:51.327 DEBUG:boto:path=/pklxkd-59/obj124 2023-05-31T20:53:51.328 DEBUG:boto:auth_path=/pklxkd-59/obj124 2023-05-31T20:53:51.328 DEBUG:boto:Method: GET 2023-05-31T20:53:51.328 DEBUG:boto:Path: /pklxkd-59/obj124 2023-05-31T20:53:51.329 DEBUG:boto:Data: 2023-05-31T20:53:51.329 DEBUG:boto:Headers: {} 2023-05-31T20:53:51.329 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:53:51.330 DEBUG:boto:Port: 8001 2023-05-31T20:53:51.330 DEBUG:boto:Params: {} 2023-05-31T20:53:51.331 DEBUG:boto:Token: None 2023-05-31T20:53:51.331 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:53:51 GMT /pklxkd-59/obj124 2023-05-31T20:53:51.331 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:ERkBTG3oCDuOYf9d2Tv9SLGRCYo= 2023-05-31T20:53:51.332 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:53:51 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:ERkBTG3oCDuOYf9d2Tv9SLGRCYo=', 'Content-Length': '0'} 2023-05-31T20:53:51.333 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:50:33 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000238cad95437e13d4-006477b3df-1c17-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:53:51 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:53:51.334 DEBUG:boto:path=/pklxkd-59/obj124 2023-05-31T20:53:51.334 DEBUG:boto:auth_path=/pklxkd-59/obj124 2023-05-31T20:53:51.335 DEBUG:boto:Method: GET 2023-05-31T20:53:51.335 DEBUG:boto:Path: /pklxkd-59/obj124 2023-05-31T20:53:51.335 DEBUG:boto:Data: 2023-05-31T20:53:51.336 DEBUG:boto:Headers: {} 2023-05-31T20:53:51.336 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:53:51.337 DEBUG:boto:Port: 8000 2023-05-31T20:53:51.337 DEBUG:boto:Params: {} 2023-05-31T20:53:51.337 DEBUG:boto:Token: None 2023-05-31T20:53:51.338 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:53:51 GMT /pklxkd-59/obj124 2023-05-31T20:53:51.338 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:ERkBTG3oCDuOYf9d2Tv9SLGRCYo= 2023-05-31T20:53:51.339 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:53:51 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:ERkBTG3oCDuOYf9d2Tv9SLGRCYo=', 'Content-Length': '0'} 2023-05-31T20:53:51.340 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:50:33 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000062865447b7f6a078-006477b3df-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:53:51 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:53:51.341 DEBUG:boto:path=/pklxkd-59/ 2023-05-31T20:53:51.341 DEBUG:boto:auth_path=/pklxkd-59/ 2023-05-31T20:53:51.342 DEBUG:boto:path=/pklxkd-59/?prefix=obj124 2023-05-31T20:53:51.342 DEBUG:boto:auth_path=/pklxkd-59/?prefix=obj124 2023-05-31T20:53:51.342 DEBUG:boto:Method: GET 2023-05-31T20:53:51.343 DEBUG:boto:Path: /pklxkd-59/?prefix=obj124 2023-05-31T20:53:51.343 DEBUG:boto:Data: 2023-05-31T20:53:51.344 DEBUG:boto:Headers: {} 2023-05-31T20:53:51.344 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:53:51.344 DEBUG:boto:Port: 8001 2023-05-31T20:53:51.345 DEBUG:boto:Params: {} 2023-05-31T20:53:51.345 DEBUG:boto:Token: None 2023-05-31T20:53:51.345 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:53:51 GMT /pklxkd-59/ 2023-05-31T20:53:51.346 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:lmsrlNekY57vtRsTzICLOlyPcM0= 2023-05-31T20:53:51.346 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:53:51 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:lmsrlNekY57vtRsTzICLOlyPcM0=', 'Content-Length': '0'} 2023-05-31T20:53:51.349 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000001910b5884bb9ae51-006477b3df-1c17-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:53:51 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:53:51.349 DEBUG:boto:b'pklxkd-59obj1241000falseobj1242023-05-31T20:50:33.587Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:53:51.350 DEBUG:boto:path=/pklxkd-59/ 2023-05-31T20:53:51.350 DEBUG:boto:auth_path=/pklxkd-59/ 2023-05-31T20:53:51.351 DEBUG:boto:path=/pklxkd-59/?prefix=obj124 2023-05-31T20:53:51.351 DEBUG:boto:auth_path=/pklxkd-59/?prefix=obj124 2023-05-31T20:53:51.351 DEBUG:boto:Method: GET 2023-05-31T20:53:51.352 DEBUG:boto:Path: /pklxkd-59/?prefix=obj124 2023-05-31T20:53:51.352 DEBUG:boto:Data: 2023-05-31T20:53:51.352 DEBUG:boto:Headers: {} 2023-05-31T20:53:51.353 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:53:51.353 DEBUG:boto:Port: 8000 2023-05-31T20:53:51.353 DEBUG:boto:Params: {} 2023-05-31T20:53:51.354 DEBUG:boto:Token: None 2023-05-31T20:53:51.354 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:53:51 GMT /pklxkd-59/ 2023-05-31T20:53:51.355 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:lmsrlNekY57vtRsTzICLOlyPcM0= 2023-05-31T20:53:51.355 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:53:51 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:lmsrlNekY57vtRsTzICLOlyPcM0=', 'Content-Length': '0'} 2023-05-31T20:53:51.359 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000007877c4d79c5af09a-006477b3df-1630-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:53:51 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:53:51.400 DEBUG:boto:b'pklxkd-59obj1241000falseobj1242023-05-31T20:50:33.587Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:53:51.401 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj145 2023-05-31T20:53:51.401 DEBUG:boto:path=/pklxkd-59/obj145 2023-05-31T20:53:51.402 DEBUG:boto:auth_path=/pklxkd-59/obj145 2023-05-31T20:53:51.402 DEBUG:boto:path=/pklxkd-59/obj145?versionId=null 2023-05-31T20:53:51.402 DEBUG:boto:auth_path=/pklxkd-59/obj145?versionId=null 2023-05-31T20:53:51.403 DEBUG:boto:Method: GET 2023-05-31T20:53:51.403 DEBUG:boto:Path: /pklxkd-59/obj145?versionId=null 2023-05-31T20:53:51.403 DEBUG:boto:Data: 2023-05-31T20:53:51.404 DEBUG:boto:Headers: {} 2023-05-31T20:53:51.404 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:53:51.405 DEBUG:boto:Port: 8001 2023-05-31T20:53:51.405 DEBUG:boto:Params: {} 2023-05-31T20:53:51.406 DEBUG:boto:Token: None 2023-05-31T20:53:51.406 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:53:51 GMT /pklxkd-59/obj145?versionId=null 2023-05-31T20:53:51.407 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:mEKkiBipCyc907j066qEe2c1e14= 2023-05-31T20:53:51.407 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:53:51 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:mEKkiBipCyc907j066qEe2c1e14=', 'Content-Length': '0'} 2023-05-31T20:53:51.409 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:48:13 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000036fb540680a65912-006477b3df-1c17-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:53:51 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:53:51.410 DEBUG:boto:path=/pklxkd-59/obj145 2023-05-31T20:53:51.410 DEBUG:boto:auth_path=/pklxkd-59/obj145 2023-05-31T20:53:51.411 DEBUG:boto:path=/pklxkd-59/obj145?versionId=null 2023-05-31T20:53:51.411 DEBUG:boto:auth_path=/pklxkd-59/obj145?versionId=null 2023-05-31T20:53:51.411 DEBUG:boto:Method: GET 2023-05-31T20:53:51.412 DEBUG:boto:Path: /pklxkd-59/obj145?versionId=null 2023-05-31T20:53:51.412 DEBUG:boto:Data: 2023-05-31T20:53:51.413 DEBUG:boto:Headers: {} 2023-05-31T20:53:51.413 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:53:51.414 DEBUG:boto:Port: 8000 2023-05-31T20:53:51.414 DEBUG:boto:Params: {} 2023-05-31T20:53:51.414 DEBUG:boto:Token: None 2023-05-31T20:53:51.415 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:53:51 GMT /pklxkd-59/obj145?versionId=null 2023-05-31T20:53:51.415 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:mEKkiBipCyc907j066qEe2c1e14= 2023-05-31T20:53:51.416 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:53:51 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:mEKkiBipCyc907j066qEe2c1e14=', 'Content-Length': '0'} 2023-05-31T20:53:51.417 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:48:13 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000cefab9f421a50153-006477b3df-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:53:51 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:53:51.418 DEBUG:boto:path=/pklxkd-59/obj145 2023-05-31T20:53:51.418 DEBUG:boto:auth_path=/pklxkd-59/obj145 2023-05-31T20:53:51.419 DEBUG:boto:path=/pklxkd-59/obj145?versionId=null 2023-05-31T20:53:51.419 DEBUG:boto:auth_path=/pklxkd-59/obj145?versionId=null 2023-05-31T20:53:51.419 DEBUG:boto:Method: HEAD 2023-05-31T20:53:51.420 DEBUG:boto:Path: /pklxkd-59/obj145?versionId=null 2023-05-31T20:53:51.420 DEBUG:boto:Data: 2023-05-31T20:53:51.421 DEBUG:boto:Headers: {} 2023-05-31T20:53:51.421 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:53:51.422 DEBUG:boto:Port: 8001 2023-05-31T20:53:51.422 DEBUG:boto:Params: {} 2023-05-31T20:53:51.422 DEBUG:boto:Token: None 2023-05-31T20:53:51.423 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:53:51 GMT /pklxkd-59/obj145?versionId=null 2023-05-31T20:53:51.423 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:YZZUpXjgU6KRoFlyYqhQk3j8zPk= 2023-05-31T20:53:51.424 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:53:51 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:YZZUpXjgU6KRoFlyYqhQk3j8zPk=', 'Content-Length': '0'} 2023-05-31T20:53:51.426 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:48:13 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000087b69001a678bbae-006477b3df-1c17-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:53:51 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:53:51.426 DEBUG:boto:path=/pklxkd-59/obj145 2023-05-31T20:53:51.427 DEBUG:boto:auth_path=/pklxkd-59/obj145 2023-05-31T20:53:51.427 DEBUG:boto:path=/pklxkd-59/obj145?versionId=null 2023-05-31T20:53:51.428 DEBUG:boto:auth_path=/pklxkd-59/obj145?versionId=null 2023-05-31T20:53:51.428 DEBUG:boto:Method: HEAD 2023-05-31T20:53:51.429 DEBUG:boto:Path: /pklxkd-59/obj145?versionId=null 2023-05-31T20:53:51.429 DEBUG:boto:Data: 2023-05-31T20:53:51.430 DEBUG:boto:Headers: {} 2023-05-31T20:53:51.430 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:53:51.431 DEBUG:boto:Port: 8000 2023-05-31T20:53:51.431 DEBUG:boto:Params: {} 2023-05-31T20:53:51.431 DEBUG:boto:Token: None 2023-05-31T20:53:51.432 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:53:51 GMT /pklxkd-59/obj145?versionId=null 2023-05-31T20:53:51.432 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:YZZUpXjgU6KRoFlyYqhQk3j8zPk= 2023-05-31T20:53:51.433 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:53:51 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:YZZUpXjgU6KRoFlyYqhQk3j8zPk=', 'Content-Length': '0'} 2023-05-31T20:53:51.435 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:48:13 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000fac998fda79406cd-006477b3df-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:53:51 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:53:51.435 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj145 2023-05-31T20:53:51.436 DEBUG:boto:path=/pklxkd-59/obj145 2023-05-31T20:53:51.436 DEBUG:boto:auth_path=/pklxkd-59/obj145 2023-05-31T20:53:51.436 DEBUG:boto:path=/pklxkd-59/obj145?versionId=null 2023-05-31T20:53:51.437 DEBUG:boto:auth_path=/pklxkd-59/obj145?versionId=null 2023-05-31T20:53:51.437 DEBUG:boto:Method: GET 2023-05-31T20:53:51.437 DEBUG:boto:Path: /pklxkd-59/obj145?versionId=null 2023-05-31T20:53:51.438 DEBUG:boto:Data: 2023-05-31T20:53:51.438 DEBUG:boto:Headers: {} 2023-05-31T20:53:51.439 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:53:51.439 DEBUG:boto:Port: 8001 2023-05-31T20:53:51.439 DEBUG:boto:Params: {} 2023-05-31T20:53:51.440 DEBUG:boto:Token: None 2023-05-31T20:53:51.440 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:53:51 GMT /pklxkd-59/obj145?versionId=null 2023-05-31T20:53:51.441 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:mEKkiBipCyc907j066qEe2c1e14= 2023-05-31T20:53:51.441 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:53:51 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:mEKkiBipCyc907j066qEe2c1e14=', 'Content-Length': '0'} 2023-05-31T20:53:51.443 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:48:13 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004157cb70d07ca9a3-006477b3df-1c17-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:53:51 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:53:51.444 DEBUG:boto:path=/pklxkd-59/obj145 2023-05-31T20:53:51.444 DEBUG:boto:auth_path=/pklxkd-59/obj145 2023-05-31T20:53:51.445 DEBUG:boto:path=/pklxkd-59/obj145?versionId=null 2023-05-31T20:53:51.445 DEBUG:boto:auth_path=/pklxkd-59/obj145?versionId=null 2023-05-31T20:53:51.445 DEBUG:boto:Method: GET 2023-05-31T20:53:51.446 DEBUG:boto:Path: /pklxkd-59/obj145?versionId=null 2023-05-31T20:53:51.446 DEBUG:boto:Data: 2023-05-31T20:53:51.446 DEBUG:boto:Headers: {} 2023-05-31T20:53:51.447 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:53:51.447 DEBUG:boto:Port: 8000 2023-05-31T20:53:51.447 DEBUG:boto:Params: {} 2023-05-31T20:53:51.448 DEBUG:boto:Token: None 2023-05-31T20:53:51.448 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:53:51 GMT /pklxkd-59/obj145?versionId=null 2023-05-31T20:53:51.449 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:mEKkiBipCyc907j066qEe2c1e14= 2023-05-31T20:53:51.449 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:53:51 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:mEKkiBipCyc907j066qEe2c1e14=', 'Content-Length': '0'} 2023-05-31T20:53:51.451 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:48:13 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f7f3a213d860e640-006477b3df-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:53:51 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:53:51.452 DEBUG:boto:path=/pklxkd-59/ 2023-05-31T20:53:51.452 DEBUG:boto:auth_path=/pklxkd-59/ 2023-05-31T20:53:51.453 DEBUG:boto:path=/pklxkd-59/?prefix=obj145 2023-05-31T20:53:51.453 DEBUG:boto:auth_path=/pklxkd-59/?prefix=obj145 2023-05-31T20:53:51.453 DEBUG:boto:Method: GET 2023-05-31T20:53:51.454 DEBUG:boto:Path: /pklxkd-59/?prefix=obj145 2023-05-31T20:53:51.454 DEBUG:boto:Data: 2023-05-31T20:53:51.454 DEBUG:boto:Headers: {} 2023-05-31T20:53:51.455 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:53:51.455 DEBUG:boto:Port: 8001 2023-05-31T20:53:51.455 DEBUG:boto:Params: {} 2023-05-31T20:53:51.456 DEBUG:boto:Token: None 2023-05-31T20:53:51.456 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:53:51 GMT /pklxkd-59/ 2023-05-31T20:53:51.457 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:lmsrlNekY57vtRsTzICLOlyPcM0= 2023-05-31T20:53:51.457 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:53:51 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:lmsrlNekY57vtRsTzICLOlyPcM0=', 'Content-Length': '0'} 2023-05-31T20:53:51.459 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a00b7de124bf3764-006477b3df-1c17-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:53:51 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:53:51.460 DEBUG:boto:b'pklxkd-59obj1451000falseobj1452023-05-31T20:48:13.107Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:53:51.461 DEBUG:boto:path=/pklxkd-59/ 2023-05-31T20:53:51.461 DEBUG:boto:auth_path=/pklxkd-59/ 2023-05-31T20:53:51.461 DEBUG:boto:path=/pklxkd-59/?prefix=obj145 2023-05-31T20:53:51.462 DEBUG:boto:auth_path=/pklxkd-59/?prefix=obj145 2023-05-31T20:53:51.462 DEBUG:boto:Method: GET 2023-05-31T20:53:51.463 DEBUG:boto:Path: /pklxkd-59/?prefix=obj145 2023-05-31T20:53:51.463 DEBUG:boto:Data: 2023-05-31T20:53:51.463 DEBUG:boto:Headers: {} 2023-05-31T20:53:51.464 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:53:51.464 DEBUG:boto:Port: 8000 2023-05-31T20:53:51.464 DEBUG:boto:Params: {} 2023-05-31T20:53:51.465 DEBUG:boto:Token: None 2023-05-31T20:53:51.465 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:53:51 GMT /pklxkd-59/ 2023-05-31T20:53:51.466 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:lmsrlNekY57vtRsTzICLOlyPcM0= 2023-05-31T20:53:51.466 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:53:51 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:lmsrlNekY57vtRsTzICLOlyPcM0=', 'Content-Length': '0'} 2023-05-31T20:53:51.470 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000005eab8c4c26631b51-006477b3df-1630-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:53:51 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:53:51.471 DEBUG:boto:b'pklxkd-59obj1451000falseobj1452023-05-31T20:48:13.107Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:53:51.471 DEBUG:boto:path=/pklxkd-59/obj145 2023-05-31T20:53:51.472 DEBUG:boto:auth_path=/pklxkd-59/obj145 2023-05-31T20:53:51.472 DEBUG:boto:Method: HEAD 2023-05-31T20:53:51.472 DEBUG:boto:Path: /pklxkd-59/obj145 2023-05-31T20:53:51.473 DEBUG:boto:Data: 2023-05-31T20:53:51.473 DEBUG:boto:Headers: {} 2023-05-31T20:53:51.473 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:53:51.474 DEBUG:boto:Port: 8001 2023-05-31T20:53:51.474 DEBUG:boto:Params: {} 2023-05-31T20:53:51.474 DEBUG:boto:Token: None 2023-05-31T20:53:51.475 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:53:51 GMT /pklxkd-59/obj145 2023-05-31T20:53:51.475 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:aG74976r7MlPNaOJqDucyuk5Cl4= 2023-05-31T20:53:51.475 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:53:51 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:aG74976r7MlPNaOJqDucyuk5Cl4=', 'Content-Length': '0'} 2023-05-31T20:53:51.477 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:48:13 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000929cbac294e1e5a6-006477b3df-1c17-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:53:51 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:53:51.478 DEBUG:boto:path=/pklxkd-59/obj145 2023-05-31T20:53:51.478 DEBUG:boto:auth_path=/pklxkd-59/obj145 2023-05-31T20:53:51.478 DEBUG:boto:Method: HEAD 2023-05-31T20:53:51.479 DEBUG:boto:Path: /pklxkd-59/obj145 2023-05-31T20:53:51.479 DEBUG:boto:Data: 2023-05-31T20:53:51.479 DEBUG:boto:Headers: {} 2023-05-31T20:53:51.480 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:53:51.480 DEBUG:boto:Port: 8000 2023-05-31T20:53:51.480 DEBUG:boto:Params: {} 2023-05-31T20:53:51.481 DEBUG:boto:Token: None 2023-05-31T20:53:51.481 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:53:51 GMT /pklxkd-59/obj145 2023-05-31T20:53:51.482 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:aG74976r7MlPNaOJqDucyuk5Cl4= 2023-05-31T20:53:51.482 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:53:51 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:aG74976r7MlPNaOJqDucyuk5Cl4=', 'Content-Length': '0'} 2023-05-31T20:53:51.484 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:48:13 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b3c1d0d6a3e6e713-006477b3df-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:53:51 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:53:51.484 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj145 2023-05-31T20:53:51.485 DEBUG:boto:path=/pklxkd-59/obj145 2023-05-31T20:53:51.485 DEBUG:boto:auth_path=/pklxkd-59/obj145 2023-05-31T20:53:51.486 DEBUG:boto:Method: GET 2023-05-31T20:53:51.486 DEBUG:boto:Path: /pklxkd-59/obj145 2023-05-31T20:53:51.486 DEBUG:boto:Data: 2023-05-31T20:53:51.487 DEBUG:boto:Headers: {} 2023-05-31T20:53:51.487 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:53:51.487 DEBUG:boto:Port: 8001 2023-05-31T20:53:51.488 DEBUG:boto:Params: {} 2023-05-31T20:53:51.488 DEBUG:boto:Token: None 2023-05-31T20:53:51.489 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:53:51 GMT /pklxkd-59/obj145 2023-05-31T20:53:51.489 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:4jbjZtJgkaHiWoBXyqrmeaeFcME= 2023-05-31T20:53:51.489 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:53:51 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:4jbjZtJgkaHiWoBXyqrmeaeFcME=', 'Content-Length': '0'} 2023-05-31T20:53:51.491 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:48:13 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009c2a997070d757bf-006477b3df-1c17-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:53:51 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:53:51.492 DEBUG:boto:path=/pklxkd-59/obj145 2023-05-31T20:53:51.492 DEBUG:boto:auth_path=/pklxkd-59/obj145 2023-05-31T20:53:51.492 DEBUG:boto:Method: GET 2023-05-31T20:53:51.493 DEBUG:boto:Path: /pklxkd-59/obj145 2023-05-31T20:53:51.493 DEBUG:boto:Data: 2023-05-31T20:53:51.494 DEBUG:boto:Headers: {} 2023-05-31T20:53:51.494 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:53:51.494 DEBUG:boto:Port: 8000 2023-05-31T20:53:51.495 DEBUG:boto:Params: {} 2023-05-31T20:53:51.495 DEBUG:boto:Token: None 2023-05-31T20:53:51.495 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:53:51 GMT /pklxkd-59/obj145 2023-05-31T20:53:51.496 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:4jbjZtJgkaHiWoBXyqrmeaeFcME= 2023-05-31T20:53:51.496 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:53:51 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:4jbjZtJgkaHiWoBXyqrmeaeFcME=', 'Content-Length': '0'} 2023-05-31T20:53:51.498 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:48:13 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000072f899167234822d-006477b3df-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:53:51 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:53:51.499 DEBUG:boto:path=/pklxkd-59/ 2023-05-31T20:53:51.499 DEBUG:boto:auth_path=/pklxkd-59/ 2023-05-31T20:53:51.499 DEBUG:boto:path=/pklxkd-59/?prefix=obj145 2023-05-31T20:53:51.500 DEBUG:boto:auth_path=/pklxkd-59/?prefix=obj145 2023-05-31T20:53:51.500 DEBUG:boto:Method: GET 2023-05-31T20:53:51.501 DEBUG:boto:Path: /pklxkd-59/?prefix=obj145 2023-05-31T20:53:51.501 DEBUG:boto:Data: 2023-05-31T20:53:51.501 DEBUG:boto:Headers: {} 2023-05-31T20:53:51.502 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:53:51.502 DEBUG:boto:Port: 8001 2023-05-31T20:53:51.502 DEBUG:boto:Params: {} 2023-05-31T20:53:51.503 DEBUG:boto:Token: None 2023-05-31T20:53:51.503 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:53:51 GMT /pklxkd-59/ 2023-05-31T20:53:51.504 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:lmsrlNekY57vtRsTzICLOlyPcM0= 2023-05-31T20:53:51.504 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:53:51 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:lmsrlNekY57vtRsTzICLOlyPcM0=', 'Content-Length': '0'} 2023-05-31T20:53:51.506 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a9da610ad8cb6998-006477b3df-1c17-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:53:51 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:53:51.507 DEBUG:boto:b'pklxkd-59obj1451000falseobj1452023-05-31T20:48:13.107Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:53:51.507 DEBUG:boto:path=/pklxkd-59/ 2023-05-31T20:53:51.508 DEBUG:boto:auth_path=/pklxkd-59/ 2023-05-31T20:53:51.508 DEBUG:boto:path=/pklxkd-59/?prefix=obj145 2023-05-31T20:53:51.508 DEBUG:boto:auth_path=/pklxkd-59/?prefix=obj145 2023-05-31T20:53:51.509 DEBUG:boto:Method: GET 2023-05-31T20:53:51.509 DEBUG:boto:Path: /pklxkd-59/?prefix=obj145 2023-05-31T20:53:51.509 DEBUG:boto:Data: 2023-05-31T20:53:51.510 DEBUG:boto:Headers: {} 2023-05-31T20:53:51.510 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:53:51.511 DEBUG:boto:Port: 8000 2023-05-31T20:53:51.511 DEBUG:boto:Params: {} 2023-05-31T20:53:51.511 DEBUG:boto:Token: None 2023-05-31T20:53:51.512 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:53:51 GMT /pklxkd-59/ 2023-05-31T20:53:51.512 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:lmsrlNekY57vtRsTzICLOlyPcM0= 2023-05-31T20:53:51.513 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:53:51 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:lmsrlNekY57vtRsTzICLOlyPcM0=', 'Content-Length': '0'} 2023-05-31T20:53:51.516 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ae15ea47093acd47-006477b3df-1630-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:53:51 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:53:51.517 DEBUG:boto:b'pklxkd-59obj1451000falseobj1452023-05-31T20:48:13.107Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:53:51.518 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj148 2023-05-31T20:53:51.518 DEBUG:boto:path=/pklxkd-59/obj148 2023-05-31T20:53:51.518 DEBUG:boto:auth_path=/pklxkd-59/obj148 2023-05-31T20:53:51.519 DEBUG:boto:path=/pklxkd-59/obj148?versionId=null 2023-05-31T20:53:51.519 DEBUG:boto:auth_path=/pklxkd-59/obj148?versionId=null 2023-05-31T20:53:51.519 DEBUG:boto:Method: GET 2023-05-31T20:53:51.520 DEBUG:boto:Path: /pklxkd-59/obj148?versionId=null 2023-05-31T20:53:51.520 DEBUG:boto:Data: 2023-05-31T20:53:51.521 DEBUG:boto:Headers: {} 2023-05-31T20:53:51.521 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:53:51.521 DEBUG:boto:Port: 8001 2023-05-31T20:53:51.522 DEBUG:boto:Params: {} 2023-05-31T20:53:51.522 DEBUG:boto:Token: None 2023-05-31T20:53:51.523 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:53:51 GMT /pklxkd-59/obj148?versionId=null 2023-05-31T20:53:51.523 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:ur4gpaQufutQgsdYvZc72xAoSeo= 2023-05-31T20:53:51.523 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:53:51 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:ur4gpaQufutQgsdYvZc72xAoSeo=', 'Content-Length': '0'} 2023-05-31T20:53:51.525 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:52:15 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000038e9f7e3ad7c1195-006477b3df-1c17-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:53:51 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:53:51.526 DEBUG:boto:path=/pklxkd-59/obj148 2023-05-31T20:53:51.526 DEBUG:boto:auth_path=/pklxkd-59/obj148 2023-05-31T20:53:51.527 DEBUG:boto:path=/pklxkd-59/obj148?versionId=null 2023-05-31T20:53:51.527 DEBUG:boto:auth_path=/pklxkd-59/obj148?versionId=null 2023-05-31T20:53:51.528 DEBUG:boto:Method: GET 2023-05-31T20:53:51.528 DEBUG:boto:Path: /pklxkd-59/obj148?versionId=null 2023-05-31T20:53:51.528 DEBUG:boto:Data: 2023-05-31T20:53:51.529 DEBUG:boto:Headers: {} 2023-05-31T20:53:51.529 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:53:51.529 DEBUG:boto:Port: 8000 2023-05-31T20:53:51.530 DEBUG:boto:Params: {} 2023-05-31T20:53:51.530 DEBUG:boto:Token: None 2023-05-31T20:53:51.531 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:53:51 GMT /pklxkd-59/obj148?versionId=null 2023-05-31T20:53:51.532 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:ur4gpaQufutQgsdYvZc72xAoSeo= 2023-05-31T20:53:51.532 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:53:51 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:ur4gpaQufutQgsdYvZc72xAoSeo=', 'Content-Length': '0'} 2023-05-31T20:53:51.534 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:52:15 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b8c9647478a171e8-006477b3df-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:53:51 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:53:51.576 DEBUG:boto:path=/pklxkd-59/obj148 2023-05-31T20:53:51.577 DEBUG:boto:auth_path=/pklxkd-59/obj148 2023-05-31T20:53:51.577 DEBUG:boto:path=/pklxkd-59/obj148?versionId=null 2023-05-31T20:53:51.578 DEBUG:boto:auth_path=/pklxkd-59/obj148?versionId=null 2023-05-31T20:53:51.578 DEBUG:boto:Method: HEAD 2023-05-31T20:53:51.578 DEBUG:boto:Path: /pklxkd-59/obj148?versionId=null 2023-05-31T20:53:51.579 DEBUG:boto:Data: 2023-05-31T20:53:51.579 DEBUG:boto:Headers: {} 2023-05-31T20:53:51.579 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:53:51.580 DEBUG:boto:Port: 8001 2023-05-31T20:53:51.580 DEBUG:boto:Params: {} 2023-05-31T20:53:51.580 DEBUG:boto:Token: None 2023-05-31T20:53:51.581 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:53:51 GMT /pklxkd-59/obj148?versionId=null 2023-05-31T20:53:51.581 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:RSnBmigFr4maYa9wxA/n+RVFeuQ= 2023-05-31T20:53:51.582 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:53:51 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:RSnBmigFr4maYa9wxA/n+RVFeuQ=', 'Content-Length': '0'} 2023-05-31T20:53:51.583 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:52:15 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006eb119c01f606b48-006477b3df-1c17-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:53:51 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:53:51.584 DEBUG:boto:path=/pklxkd-59/obj148 2023-05-31T20:53:51.584 DEBUG:boto:auth_path=/pklxkd-59/obj148 2023-05-31T20:53:51.585 DEBUG:boto:path=/pklxkd-59/obj148?versionId=null 2023-05-31T20:53:51.585 DEBUG:boto:auth_path=/pklxkd-59/obj148?versionId=null 2023-05-31T20:53:51.585 DEBUG:boto:Method: HEAD 2023-05-31T20:53:51.586 DEBUG:boto:Path: /pklxkd-59/obj148?versionId=null 2023-05-31T20:53:51.586 DEBUG:boto:Data: 2023-05-31T20:53:51.586 DEBUG:boto:Headers: {} 2023-05-31T20:53:51.587 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:53:51.587 DEBUG:boto:Port: 8000 2023-05-31T20:53:51.587 DEBUG:boto:Params: {} 2023-05-31T20:53:51.588 DEBUG:boto:Token: None 2023-05-31T20:53:51.588 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:53:51 GMT /pklxkd-59/obj148?versionId=null 2023-05-31T20:53:51.589 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:RSnBmigFr4maYa9wxA/n+RVFeuQ= 2023-05-31T20:53:51.589 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:53:51 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:RSnBmigFr4maYa9wxA/n+RVFeuQ=', 'Content-Length': '0'} 2023-05-31T20:53:51.591 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:52:15 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004462008aea9c5445-006477b3df-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:53:51 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:53:51.591 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj148 2023-05-31T20:53:51.592 DEBUG:boto:path=/pklxkd-59/obj148 2023-05-31T20:53:51.592 DEBUG:boto:auth_path=/pklxkd-59/obj148 2023-05-31T20:53:51.592 DEBUG:boto:path=/pklxkd-59/obj148?versionId=null 2023-05-31T20:53:51.593 DEBUG:boto:auth_path=/pklxkd-59/obj148?versionId=null 2023-05-31T20:53:51.593 DEBUG:boto:Method: GET 2023-05-31T20:53:51.593 DEBUG:boto:Path: /pklxkd-59/obj148?versionId=null 2023-05-31T20:53:51.594 DEBUG:boto:Data: 2023-05-31T20:53:51.594 DEBUG:boto:Headers: {} 2023-05-31T20:53:51.595 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:53:51.595 DEBUG:boto:Port: 8001 2023-05-31T20:53:51.595 DEBUG:boto:Params: {} 2023-05-31T20:53:51.596 DEBUG:boto:Token: None 2023-05-31T20:53:51.596 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:53:51 GMT /pklxkd-59/obj148?versionId=null 2023-05-31T20:53:51.597 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:ur4gpaQufutQgsdYvZc72xAoSeo= 2023-05-31T20:53:51.597 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:53:51 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:ur4gpaQufutQgsdYvZc72xAoSeo=', 'Content-Length': '0'} 2023-05-31T20:53:51.599 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:52:15 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000066cfe5251a5bdd65-006477b3df-1c17-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:53:51 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:53:51.599 DEBUG:boto:path=/pklxkd-59/obj148 2023-05-31T20:53:51.600 DEBUG:boto:auth_path=/pklxkd-59/obj148 2023-05-31T20:53:51.600 DEBUG:boto:path=/pklxkd-59/obj148?versionId=null 2023-05-31T20:53:51.600 DEBUG:boto:auth_path=/pklxkd-59/obj148?versionId=null 2023-05-31T20:53:51.601 DEBUG:boto:Method: GET 2023-05-31T20:53:51.601 DEBUG:boto:Path: /pklxkd-59/obj148?versionId=null 2023-05-31T20:53:51.601 DEBUG:boto:Data: 2023-05-31T20:53:51.602 DEBUG:boto:Headers: {} 2023-05-31T20:53:51.602 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:53:51.603 DEBUG:boto:Port: 8000 2023-05-31T20:53:51.603 DEBUG:boto:Params: {} 2023-05-31T20:53:51.603 DEBUG:boto:Token: None 2023-05-31T20:53:51.604 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:53:51 GMT /pklxkd-59/obj148?versionId=null 2023-05-31T20:53:51.604 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:ur4gpaQufutQgsdYvZc72xAoSeo= 2023-05-31T20:53:51.605 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:53:51 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:ur4gpaQufutQgsdYvZc72xAoSeo=', 'Content-Length': '0'} 2023-05-31T20:53:51.606 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:52:15 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000067dcdc578a9a2a92-006477b3df-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:53:51 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:53:51.607 DEBUG:boto:path=/pklxkd-59/ 2023-05-31T20:53:51.607 DEBUG:boto:auth_path=/pklxkd-59/ 2023-05-31T20:53:51.608 DEBUG:boto:path=/pklxkd-59/?prefix=obj148 2023-05-31T20:53:51.608 DEBUG:boto:auth_path=/pklxkd-59/?prefix=obj148 2023-05-31T20:53:51.608 DEBUG:boto:Method: GET 2023-05-31T20:53:51.609 DEBUG:boto:Path: /pklxkd-59/?prefix=obj148 2023-05-31T20:53:51.609 DEBUG:boto:Data: 2023-05-31T20:53:51.609 DEBUG:boto:Headers: {} 2023-05-31T20:53:51.610 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:53:51.610 DEBUG:boto:Port: 8001 2023-05-31T20:53:51.611 DEBUG:boto:Params: {} 2023-05-31T20:53:51.611 DEBUG:boto:Token: None 2023-05-31T20:53:51.611 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:53:51 GMT /pklxkd-59/ 2023-05-31T20:53:51.612 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:lmsrlNekY57vtRsTzICLOlyPcM0= 2023-05-31T20:53:51.612 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:53:51 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:lmsrlNekY57vtRsTzICLOlyPcM0=', 'Content-Length': '0'} 2023-05-31T20:53:51.614 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000bc983ef96c3eaf42-006477b3df-1c17-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:53:51 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:53:51.615 DEBUG:boto:b'pklxkd-59obj1481000falseobj1482023-05-31T20:52:15.015Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:53:51.616 DEBUG:boto:path=/pklxkd-59/ 2023-05-31T20:53:51.616 DEBUG:boto:auth_path=/pklxkd-59/ 2023-05-31T20:53:51.616 DEBUG:boto:path=/pklxkd-59/?prefix=obj148 2023-05-31T20:53:51.617 DEBUG:boto:auth_path=/pklxkd-59/?prefix=obj148 2023-05-31T20:53:51.617 DEBUG:boto:Method: GET 2023-05-31T20:53:51.618 DEBUG:boto:Path: /pklxkd-59/?prefix=obj148 2023-05-31T20:53:51.618 DEBUG:boto:Data: 2023-05-31T20:53:51.618 DEBUG:boto:Headers: {} 2023-05-31T20:53:51.619 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:53:51.619 DEBUG:boto:Port: 8000 2023-05-31T20:53:51.619 DEBUG:boto:Params: {} 2023-05-31T20:53:51.620 DEBUG:boto:Token: None 2023-05-31T20:53:51.620 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:53:51 GMT /pklxkd-59/ 2023-05-31T20:53:51.621 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:lmsrlNekY57vtRsTzICLOlyPcM0= 2023-05-31T20:53:51.621 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:53:51 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:lmsrlNekY57vtRsTzICLOlyPcM0=', 'Content-Length': '0'} 2023-05-31T20:53:51.625 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000005cce658f71a32fc5-006477b3df-1630-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:53:51 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:53:51.625 DEBUG:boto:b'pklxkd-59obj1481000falseobj1482023-05-31T20:52:15.015Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:53:51.626 DEBUG:boto:path=/pklxkd-59/obj148 2023-05-31T20:53:51.626 DEBUG:boto:auth_path=/pklxkd-59/obj148 2023-05-31T20:53:51.627 DEBUG:boto:Method: HEAD 2023-05-31T20:53:51.627 DEBUG:boto:Path: /pklxkd-59/obj148 2023-05-31T20:53:51.627 DEBUG:boto:Data: 2023-05-31T20:53:51.628 DEBUG:boto:Headers: {} 2023-05-31T20:53:51.628 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:53:51.628 DEBUG:boto:Port: 8001 2023-05-31T20:53:51.629 DEBUG:boto:Params: {} 2023-05-31T20:53:51.629 DEBUG:boto:Token: None 2023-05-31T20:53:51.629 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:53:51 GMT /pklxkd-59/obj148 2023-05-31T20:53:51.630 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:+zcJSqqgK0Xd1l2Q/LBmiM2U2Vc= 2023-05-31T20:53:51.630 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:53:51 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:+zcJSqqgK0Xd1l2Q/LBmiM2U2Vc=', 'Content-Length': '0'} 2023-05-31T20:53:51.632 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:52:15 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000070babb6523e09611-006477b3df-1c17-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:53:51 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:53:51.632 DEBUG:boto:path=/pklxkd-59/obj148 2023-05-31T20:53:51.633 DEBUG:boto:auth_path=/pklxkd-59/obj148 2023-05-31T20:53:51.633 DEBUG:boto:Method: HEAD 2023-05-31T20:53:51.634 DEBUG:boto:Path: /pklxkd-59/obj148 2023-05-31T20:53:51.634 DEBUG:boto:Data: 2023-05-31T20:53:51.634 DEBUG:boto:Headers: {} 2023-05-31T20:53:51.635 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:53:51.635 DEBUG:boto:Port: 8000 2023-05-31T20:53:51.635 DEBUG:boto:Params: {} 2023-05-31T20:53:51.636 DEBUG:boto:Token: None 2023-05-31T20:53:51.636 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:53:51 GMT /pklxkd-59/obj148 2023-05-31T20:53:51.636 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:+zcJSqqgK0Xd1l2Q/LBmiM2U2Vc= 2023-05-31T20:53:51.637 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:53:51 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:+zcJSqqgK0Xd1l2Q/LBmiM2U2Vc=', 'Content-Length': '0'} 2023-05-31T20:53:51.639 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:52:15 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000028d312c7cfa03156-006477b3df-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:53:51 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:53:51.639 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj148 2023-05-31T20:53:51.640 DEBUG:boto:path=/pklxkd-59/obj148 2023-05-31T20:53:51.640 DEBUG:boto:auth_path=/pklxkd-59/obj148 2023-05-31T20:53:51.640 DEBUG:boto:Method: GET 2023-05-31T20:53:51.641 DEBUG:boto:Path: /pklxkd-59/obj148 2023-05-31T20:53:51.641 DEBUG:boto:Data: 2023-05-31T20:53:51.641 DEBUG:boto:Headers: {} 2023-05-31T20:53:51.642 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:53:51.642 DEBUG:boto:Port: 8001 2023-05-31T20:53:51.642 DEBUG:boto:Params: {} 2023-05-31T20:53:51.643 DEBUG:boto:Token: None 2023-05-31T20:53:51.643 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:53:51 GMT /pklxkd-59/obj148 2023-05-31T20:53:51.644 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:lgvhREiU1DrUkRZqmWQkJCfnKlI= 2023-05-31T20:53:51.644 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:53:51 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:lgvhREiU1DrUkRZqmWQkJCfnKlI=', 'Content-Length': '0'} 2023-05-31T20:53:51.646 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:52:15 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000086bdf417a6d2eb0b-006477b3df-1c17-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:53:51 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:53:51.646 DEBUG:boto:path=/pklxkd-59/obj148 2023-05-31T20:53:51.647 DEBUG:boto:auth_path=/pklxkd-59/obj148 2023-05-31T20:53:51.647 DEBUG:boto:Method: GET 2023-05-31T20:53:51.648 DEBUG:boto:Path: /pklxkd-59/obj148 2023-05-31T20:53:51.648 DEBUG:boto:Data: 2023-05-31T20:53:51.648 DEBUG:boto:Headers: {} 2023-05-31T20:53:51.649 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:53:51.649 DEBUG:boto:Port: 8000 2023-05-31T20:53:51.649 DEBUG:boto:Params: {} 2023-05-31T20:53:51.650 DEBUG:boto:Token: None 2023-05-31T20:53:51.650 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:53:51 GMT /pklxkd-59/obj148 2023-05-31T20:53:51.651 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:lgvhREiU1DrUkRZqmWQkJCfnKlI= 2023-05-31T20:53:51.651 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:53:51 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:lgvhREiU1DrUkRZqmWQkJCfnKlI=', 'Content-Length': '0'} 2023-05-31T20:53:51.653 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:52:15 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002e1d4c3b527b2e4c-006477b3df-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:53:51 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:53:51.654 DEBUG:boto:path=/pklxkd-59/ 2023-05-31T20:53:51.654 DEBUG:boto:auth_path=/pklxkd-59/ 2023-05-31T20:53:51.654 DEBUG:boto:path=/pklxkd-59/?prefix=obj148 2023-05-31T20:53:51.655 DEBUG:boto:auth_path=/pklxkd-59/?prefix=obj148 2023-05-31T20:53:51.655 DEBUG:boto:Method: GET 2023-05-31T20:53:51.655 DEBUG:boto:Path: /pklxkd-59/?prefix=obj148 2023-05-31T20:53:51.656 DEBUG:boto:Data: 2023-05-31T20:53:51.656 DEBUG:boto:Headers: {} 2023-05-31T20:53:51.657 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:53:51.657 DEBUG:boto:Port: 8001 2023-05-31T20:53:51.657 DEBUG:boto:Params: {} 2023-05-31T20:53:51.658 DEBUG:boto:Token: None 2023-05-31T20:53:51.658 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:53:51 GMT /pklxkd-59/ 2023-05-31T20:53:51.658 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:lmsrlNekY57vtRsTzICLOlyPcM0= 2023-05-31T20:53:51.659 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:53:51 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:lmsrlNekY57vtRsTzICLOlyPcM0=', 'Content-Length': '0'} 2023-05-31T20:53:51.661 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e9c85ea59c5dce46-006477b3df-1c17-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:53:51 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:53:51.662 DEBUG:boto:b'pklxkd-59obj1481000falseobj1482023-05-31T20:52:15.015Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:53:51.662 DEBUG:boto:path=/pklxkd-59/ 2023-05-31T20:53:51.663 DEBUG:boto:auth_path=/pklxkd-59/ 2023-05-31T20:53:51.663 DEBUG:boto:path=/pklxkd-59/?prefix=obj148 2023-05-31T20:53:51.663 DEBUG:boto:auth_path=/pklxkd-59/?prefix=obj148 2023-05-31T20:53:51.664 DEBUG:boto:Method: GET 2023-05-31T20:53:51.664 DEBUG:boto:Path: /pklxkd-59/?prefix=obj148 2023-05-31T20:53:51.665 DEBUG:boto:Data: 2023-05-31T20:53:51.665 DEBUG:boto:Headers: {} 2023-05-31T20:53:51.665 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:53:51.666 DEBUG:boto:Port: 8000 2023-05-31T20:53:51.666 DEBUG:boto:Params: {} 2023-05-31T20:53:51.666 DEBUG:boto:Token: None 2023-05-31T20:53:51.667 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:53:51 GMT /pklxkd-59/ 2023-05-31T20:53:51.667 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:lmsrlNekY57vtRsTzICLOlyPcM0= 2023-05-31T20:53:51.668 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:53:51 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:lmsrlNekY57vtRsTzICLOlyPcM0=', 'Content-Length': '0'} 2023-05-31T20:53:51.672 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000008e843bb2dcc83878-006477b3df-1630-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:53:51 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:53:51.712 DEBUG:boto:b'pklxkd-59obj1481000falseobj1482023-05-31T20:52:15.015Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:53:51.713 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj155 2023-05-31T20:53:51.713 DEBUG:boto:path=/pklxkd-59/obj155 2023-05-31T20:53:51.714 DEBUG:boto:auth_path=/pklxkd-59/obj155 2023-05-31T20:53:51.714 DEBUG:boto:path=/pklxkd-59/obj155?versionId=null 2023-05-31T20:53:51.714 DEBUG:boto:auth_path=/pklxkd-59/obj155?versionId=null 2023-05-31T20:53:51.715 DEBUG:boto:Method: GET 2023-05-31T20:53:51.715 DEBUG:boto:Path: /pklxkd-59/obj155?versionId=null 2023-05-31T20:53:51.715 DEBUG:boto:Data: 2023-05-31T20:53:51.716 DEBUG:boto:Headers: {} 2023-05-31T20:53:51.716 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:53:51.716 DEBUG:boto:Port: 8001 2023-05-31T20:53:51.717 DEBUG:boto:Params: {} 2023-05-31T20:53:51.717 DEBUG:boto:Token: None 2023-05-31T20:53:51.718 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:53:51 GMT /pklxkd-59/obj155?versionId=null 2023-05-31T20:53:51.718 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:YnBKvHXeHsB7n2xCBHiu9A/GEvc= 2023-05-31T20:53:51.718 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:53:51 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:YnBKvHXeHsB7n2xCBHiu9A/GEvc=', 'Content-Length': '0'} 2023-05-31T20:53:51.720 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:50:33 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005342134460f6fb4c-006477b3df-1c17-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:53:51 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:53:51.721 DEBUG:boto:path=/pklxkd-59/obj155 2023-05-31T20:53:51.721 DEBUG:boto:auth_path=/pklxkd-59/obj155 2023-05-31T20:53:51.722 DEBUG:boto:path=/pklxkd-59/obj155?versionId=null 2023-05-31T20:53:51.722 DEBUG:boto:auth_path=/pklxkd-59/obj155?versionId=null 2023-05-31T20:53:51.723 DEBUG:boto:Method: GET 2023-05-31T20:53:51.723 DEBUG:boto:Path: /pklxkd-59/obj155?versionId=null 2023-05-31T20:53:51.723 DEBUG:boto:Data: 2023-05-31T20:53:51.724 DEBUG:boto:Headers: {} 2023-05-31T20:53:51.724 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:53:51.724 DEBUG:boto:Port: 8000 2023-05-31T20:53:51.725 DEBUG:boto:Params: {} 2023-05-31T20:53:51.725 DEBUG:boto:Token: None 2023-05-31T20:53:51.725 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:53:51 GMT /pklxkd-59/obj155?versionId=null 2023-05-31T20:53:51.726 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:YnBKvHXeHsB7n2xCBHiu9A/GEvc= 2023-05-31T20:53:51.726 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:53:51 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:YnBKvHXeHsB7n2xCBHiu9A/GEvc=', 'Content-Length': '0'} 2023-05-31T20:53:51.728 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:50:33 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000062c4cd8dc21480f4-006477b3df-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:53:51 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:53:51.729 DEBUG:boto:path=/pklxkd-59/obj155 2023-05-31T20:53:51.729 DEBUG:boto:auth_path=/pklxkd-59/obj155 2023-05-31T20:53:51.730 DEBUG:boto:path=/pklxkd-59/obj155?versionId=null 2023-05-31T20:53:51.730 DEBUG:boto:auth_path=/pklxkd-59/obj155?versionId=null 2023-05-31T20:53:51.730 DEBUG:boto:Method: HEAD 2023-05-31T20:53:51.730 DEBUG:boto:Path: /pklxkd-59/obj155?versionId=null 2023-05-31T20:53:51.731 DEBUG:boto:Data: 2023-05-31T20:53:51.731 DEBUG:boto:Headers: {} 2023-05-31T20:53:51.731 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:53:51.732 DEBUG:boto:Port: 8001 2023-05-31T20:53:51.732 DEBUG:boto:Params: {} 2023-05-31T20:53:51.733 DEBUG:boto:Token: None 2023-05-31T20:53:51.733 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:53:51 GMT /pklxkd-59/obj155?versionId=null 2023-05-31T20:53:51.733 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:bIEQScUG4RwJrREKsWVqfnMaW3g= 2023-05-31T20:53:51.734 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:53:51 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:bIEQScUG4RwJrREKsWVqfnMaW3g=', 'Content-Length': '0'} 2023-05-31T20:53:51.736 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:50:33 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001e24a17b753940c4-006477b3df-1c17-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:53:51 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:53:51.736 DEBUG:boto:path=/pklxkd-59/obj155 2023-05-31T20:53:51.737 DEBUG:boto:auth_path=/pklxkd-59/obj155 2023-05-31T20:53:51.737 DEBUG:boto:path=/pklxkd-59/obj155?versionId=null 2023-05-31T20:53:51.737 DEBUG:boto:auth_path=/pklxkd-59/obj155?versionId=null 2023-05-31T20:53:51.738 DEBUG:boto:Method: HEAD 2023-05-31T20:53:51.738 DEBUG:boto:Path: /pklxkd-59/obj155?versionId=null 2023-05-31T20:53:51.738 DEBUG:boto:Data: 2023-05-31T20:53:51.739 DEBUG:boto:Headers: {} 2023-05-31T20:53:51.739 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:53:51.739 DEBUG:boto:Port: 8000 2023-05-31T20:53:51.739 DEBUG:boto:Params: {} 2023-05-31T20:53:51.740 DEBUG:boto:Token: None 2023-05-31T20:53:51.740 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:53:51 GMT /pklxkd-59/obj155?versionId=null 2023-05-31T20:53:51.741 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:bIEQScUG4RwJrREKsWVqfnMaW3g= 2023-05-31T20:53:51.741 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:53:51 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:bIEQScUG4RwJrREKsWVqfnMaW3g=', 'Content-Length': '0'} 2023-05-31T20:53:51.743 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:50:33 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000c82e47118bc2a25-006477b3df-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:53:51 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:53:51.743 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj155 2023-05-31T20:53:51.744 DEBUG:boto:path=/pklxkd-59/obj155 2023-05-31T20:53:51.744 DEBUG:boto:auth_path=/pklxkd-59/obj155 2023-05-31T20:53:51.745 DEBUG:boto:path=/pklxkd-59/obj155?versionId=null 2023-05-31T20:53:51.745 DEBUG:boto:auth_path=/pklxkd-59/obj155?versionId=null 2023-05-31T20:53:51.745 DEBUG:boto:Method: GET 2023-05-31T20:53:51.746 DEBUG:boto:Path: /pklxkd-59/obj155?versionId=null 2023-05-31T20:53:51.746 DEBUG:boto:Data: 2023-05-31T20:53:51.746 DEBUG:boto:Headers: {} 2023-05-31T20:53:51.747 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:53:51.747 DEBUG:boto:Port: 8001 2023-05-31T20:53:51.747 DEBUG:boto:Params: {} 2023-05-31T20:53:51.748 DEBUG:boto:Token: None 2023-05-31T20:53:51.748 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:53:51 GMT /pklxkd-59/obj155?versionId=null 2023-05-31T20:53:51.749 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:YnBKvHXeHsB7n2xCBHiu9A/GEvc= 2023-05-31T20:53:51.749 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:53:51 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:YnBKvHXeHsB7n2xCBHiu9A/GEvc=', 'Content-Length': '0'} 2023-05-31T20:53:51.751 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:50:33 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004a30612133b1493b-006477b3df-1c17-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:53:51 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:53:51.751 DEBUG:boto:path=/pklxkd-59/obj155 2023-05-31T20:53:51.752 DEBUG:boto:auth_path=/pklxkd-59/obj155 2023-05-31T20:53:51.752 DEBUG:boto:path=/pklxkd-59/obj155?versionId=null 2023-05-31T20:53:51.753 DEBUG:boto:auth_path=/pklxkd-59/obj155?versionId=null 2023-05-31T20:53:51.753 DEBUG:boto:Method: GET 2023-05-31T20:53:51.753 DEBUG:boto:Path: /pklxkd-59/obj155?versionId=null 2023-05-31T20:53:51.754 DEBUG:boto:Data: 2023-05-31T20:53:51.754 DEBUG:boto:Headers: {} 2023-05-31T20:53:51.754 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:53:51.755 DEBUG:boto:Port: 8000 2023-05-31T20:53:51.755 DEBUG:boto:Params: {} 2023-05-31T20:53:51.755 DEBUG:boto:Token: None 2023-05-31T20:53:51.756 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:53:51 GMT /pklxkd-59/obj155?versionId=null 2023-05-31T20:53:51.756 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:YnBKvHXeHsB7n2xCBHiu9A/GEvc= 2023-05-31T20:53:51.757 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:53:51 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:YnBKvHXeHsB7n2xCBHiu9A/GEvc=', 'Content-Length': '0'} 2023-05-31T20:53:51.759 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:50:33 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e50724bbdbe04dc1-006477b3df-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:53:51 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:53:51.759 DEBUG:boto:path=/pklxkd-59/ 2023-05-31T20:53:51.760 DEBUG:boto:auth_path=/pklxkd-59/ 2023-05-31T20:53:51.760 DEBUG:boto:path=/pklxkd-59/?prefix=obj155 2023-05-31T20:53:51.760 DEBUG:boto:auth_path=/pklxkd-59/?prefix=obj155 2023-05-31T20:53:51.761 DEBUG:boto:Method: GET 2023-05-31T20:53:51.761 DEBUG:boto:Path: /pklxkd-59/?prefix=obj155 2023-05-31T20:53:51.761 DEBUG:boto:Data: 2023-05-31T20:53:51.762 DEBUG:boto:Headers: {} 2023-05-31T20:53:51.762 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:53:51.763 DEBUG:boto:Port: 8001 2023-05-31T20:53:51.763 DEBUG:boto:Params: {} 2023-05-31T20:53:51.763 DEBUG:boto:Token: None 2023-05-31T20:53:51.764 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:53:51 GMT /pklxkd-59/ 2023-05-31T20:53:51.764 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:lmsrlNekY57vtRsTzICLOlyPcM0= 2023-05-31T20:53:51.765 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:53:51 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:lmsrlNekY57vtRsTzICLOlyPcM0=', 'Content-Length': '0'} 2023-05-31T20:53:51.767 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c34d24d16e26f03a-006477b3df-1c17-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:53:51 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:53:51.767 DEBUG:boto:b'pklxkd-59obj1551000falseobj1552023-05-31T20:50:33.605Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:53:51.768 DEBUG:boto:path=/pklxkd-59/ 2023-05-31T20:53:51.768 DEBUG:boto:auth_path=/pklxkd-59/ 2023-05-31T20:53:51.769 DEBUG:boto:path=/pklxkd-59/?prefix=obj155 2023-05-31T20:53:51.769 DEBUG:boto:auth_path=/pklxkd-59/?prefix=obj155 2023-05-31T20:53:51.769 DEBUG:boto:Method: GET 2023-05-31T20:53:51.770 DEBUG:boto:Path: /pklxkd-59/?prefix=obj155 2023-05-31T20:53:51.770 DEBUG:boto:Data: 2023-05-31T20:53:51.770 DEBUG:boto:Headers: {} 2023-05-31T20:53:51.771 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:53:51.771 DEBUG:boto:Port: 8000 2023-05-31T20:53:51.771 DEBUG:boto:Params: {} 2023-05-31T20:53:51.772 DEBUG:boto:Token: None 2023-05-31T20:53:51.772 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:53:51 GMT /pklxkd-59/ 2023-05-31T20:53:51.773 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:lmsrlNekY57vtRsTzICLOlyPcM0= 2023-05-31T20:53:51.773 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:53:51 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:lmsrlNekY57vtRsTzICLOlyPcM0=', 'Content-Length': '0'} 2023-05-31T20:53:51.777 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000250addcd44a4ad8c-006477b3df-1630-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:53:51 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:53:51.778 DEBUG:boto:b'pklxkd-59obj1551000falseobj1552023-05-31T20:50:33.605Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:53:51.778 DEBUG:boto:path=/pklxkd-59/obj155 2023-05-31T20:53:51.779 DEBUG:boto:auth_path=/pklxkd-59/obj155 2023-05-31T20:53:51.779 DEBUG:boto:Method: HEAD 2023-05-31T20:53:51.779 DEBUG:boto:Path: /pklxkd-59/obj155 2023-05-31T20:53:51.780 DEBUG:boto:Data: 2023-05-31T20:53:51.780 DEBUG:boto:Headers: {} 2023-05-31T20:53:51.780 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:53:51.781 DEBUG:boto:Port: 8001 2023-05-31T20:53:51.781 DEBUG:boto:Params: {} 2023-05-31T20:53:51.781 DEBUG:boto:Token: None 2023-05-31T20:53:51.782 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:53:51 GMT /pklxkd-59/obj155 2023-05-31T20:53:51.782 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:qwyMAf/DX0KQ6xBhVqffdAZCkk8= 2023-05-31T20:53:51.782 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:53:51 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:qwyMAf/DX0KQ6xBhVqffdAZCkk8=', 'Content-Length': '0'} 2023-05-31T20:53:51.784 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:50:33 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005c008f16e163869d-006477b3df-1c17-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:53:51 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:53:51.785 DEBUG:boto:path=/pklxkd-59/obj155 2023-05-31T20:53:51.785 DEBUG:boto:auth_path=/pklxkd-59/obj155 2023-05-31T20:53:51.786 DEBUG:boto:Method: HEAD 2023-05-31T20:53:51.786 DEBUG:boto:Path: /pklxkd-59/obj155 2023-05-31T20:53:51.786 DEBUG:boto:Data: 2023-05-31T20:53:51.787 DEBUG:boto:Headers: {} 2023-05-31T20:53:51.787 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:53:51.787 DEBUG:boto:Port: 8000 2023-05-31T20:53:51.788 DEBUG:boto:Params: {} 2023-05-31T20:53:51.788 DEBUG:boto:Token: None 2023-05-31T20:53:51.788 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:53:51 GMT /pklxkd-59/obj155 2023-05-31T20:53:51.789 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:qwyMAf/DX0KQ6xBhVqffdAZCkk8= 2023-05-31T20:53:51.789 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:53:51 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:qwyMAf/DX0KQ6xBhVqffdAZCkk8=', 'Content-Length': '0'} 2023-05-31T20:53:51.791 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:50:33 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002930bf1959e1b5c5-006477b3df-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:53:51 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:53:51.792 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj155 2023-05-31T20:53:51.792 DEBUG:boto:path=/pklxkd-59/obj155 2023-05-31T20:53:51.792 DEBUG:boto:auth_path=/pklxkd-59/obj155 2023-05-31T20:53:51.793 DEBUG:boto:Method: GET 2023-05-31T20:53:51.793 DEBUG:boto:Path: /pklxkd-59/obj155 2023-05-31T20:53:51.793 DEBUG:boto:Data: 2023-05-31T20:53:51.794 DEBUG:boto:Headers: {} 2023-05-31T20:53:51.794 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:53:51.795 DEBUG:boto:Port: 8001 2023-05-31T20:53:51.795 DEBUG:boto:Params: {} 2023-05-31T20:53:51.795 DEBUG:boto:Token: None 2023-05-31T20:53:51.796 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:53:51 GMT /pklxkd-59/obj155 2023-05-31T20:53:51.796 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:S+9xJlKgoAONgANK2hjqWsxohTU= 2023-05-31T20:53:51.797 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:53:51 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:S+9xJlKgoAONgANK2hjqWsxohTU=', 'Content-Length': '0'} 2023-05-31T20:53:51.798 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:50:33 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003ba3f2e722b16de4-006477b3df-1c17-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:53:51 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:53:51.840 DEBUG:boto:path=/pklxkd-59/obj155 2023-05-31T20:53:51.841 DEBUG:boto:auth_path=/pklxkd-59/obj155 2023-05-31T20:53:51.841 DEBUG:boto:Method: GET 2023-05-31T20:53:51.842 DEBUG:boto:Path: /pklxkd-59/obj155 2023-05-31T20:53:51.842 DEBUG:boto:Data: 2023-05-31T20:53:51.842 DEBUG:boto:Headers: {} 2023-05-31T20:53:51.843 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:53:51.843 DEBUG:boto:Port: 8000 2023-05-31T20:53:51.844 DEBUG:boto:Params: {} 2023-05-31T20:53:51.845 DEBUG:boto:Token: None 2023-05-31T20:53:51.845 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:53:51 GMT /pklxkd-59/obj155 2023-05-31T20:53:51.846 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:S+9xJlKgoAONgANK2hjqWsxohTU= 2023-05-31T20:53:51.847 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:53:51 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:S+9xJlKgoAONgANK2hjqWsxohTU=', 'Content-Length': '0'} 2023-05-31T20:53:51.849 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:50:33 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000042f2c2ce13dd04d0-006477b3df-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:53:51 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:53:51.851 DEBUG:boto:path=/pklxkd-59/ 2023-05-31T20:53:51.851 DEBUG:boto:auth_path=/pklxkd-59/ 2023-05-31T20:53:51.852 DEBUG:boto:path=/pklxkd-59/?prefix=obj155 2023-05-31T20:53:51.852 DEBUG:boto:auth_path=/pklxkd-59/?prefix=obj155 2023-05-31T20:53:51.853 DEBUG:boto:Method: GET 2023-05-31T20:53:51.853 DEBUG:boto:Path: /pklxkd-59/?prefix=obj155 2023-05-31T20:53:51.854 DEBUG:boto:Data: 2023-05-31T20:53:51.854 DEBUG:boto:Headers: {} 2023-05-31T20:53:51.854 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:53:51.855 DEBUG:boto:Port: 8001 2023-05-31T20:53:51.855 DEBUG:boto:Params: {} 2023-05-31T20:53:51.856 DEBUG:boto:Token: None 2023-05-31T20:53:51.856 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:53:51 GMT /pklxkd-59/ 2023-05-31T20:53:51.857 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:lmsrlNekY57vtRsTzICLOlyPcM0= 2023-05-31T20:53:51.857 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:53:51 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:lmsrlNekY57vtRsTzICLOlyPcM0=', 'Content-Length': '0'} 2023-05-31T20:53:51.860 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000278b51d23924c88f-006477b3df-1c17-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:53:51 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:53:51.861 DEBUG:boto:b'pklxkd-59obj1551000falseobj1552023-05-31T20:50:33.605Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:53:51.861 DEBUG:boto:path=/pklxkd-59/ 2023-05-31T20:53:51.862 DEBUG:boto:auth_path=/pklxkd-59/ 2023-05-31T20:53:51.863 DEBUG:boto:path=/pklxkd-59/?prefix=obj155 2023-05-31T20:53:51.863 DEBUG:boto:auth_path=/pklxkd-59/?prefix=obj155 2023-05-31T20:53:51.863 DEBUG:boto:Method: GET 2023-05-31T20:53:51.864 DEBUG:boto:Path: /pklxkd-59/?prefix=obj155 2023-05-31T20:53:51.864 DEBUG:boto:Data: 2023-05-31T20:53:51.864 DEBUG:boto:Headers: {} 2023-05-31T20:53:51.865 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:53:51.865 DEBUG:boto:Port: 8000 2023-05-31T20:53:51.866 DEBUG:boto:Params: {} 2023-05-31T20:53:51.866 DEBUG:boto:Token: None 2023-05-31T20:53:51.867 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:53:51 GMT /pklxkd-59/ 2023-05-31T20:53:51.867 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:lmsrlNekY57vtRsTzICLOlyPcM0= 2023-05-31T20:53:51.868 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:53:51 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:lmsrlNekY57vtRsTzICLOlyPcM0=', 'Content-Length': '0'} 2023-05-31T20:53:51.872 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000014ef822c9cda476-006477b3df-1630-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:53:51 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:53:51.873 DEBUG:boto:b'pklxkd-59obj1551000falseobj1552023-05-31T20:50:33.605Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:53:51.873 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj17 2023-05-31T20:53:51.873 DEBUG:boto:path=/pklxkd-59/obj17 2023-05-31T20:53:51.874 DEBUG:boto:auth_path=/pklxkd-59/obj17 2023-05-31T20:53:51.875 DEBUG:boto:path=/pklxkd-59/obj17?versionId=null 2023-05-31T20:53:51.875 DEBUG:boto:auth_path=/pklxkd-59/obj17?versionId=null 2023-05-31T20:53:51.875 DEBUG:boto:Method: GET 2023-05-31T20:53:51.876 DEBUG:boto:Path: /pklxkd-59/obj17?versionId=null 2023-05-31T20:53:51.876 DEBUG:boto:Data: 2023-05-31T20:53:51.876 DEBUG:boto:Headers: {} 2023-05-31T20:53:51.877 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:53:51.877 DEBUG:boto:Port: 8001 2023-05-31T20:53:51.877 DEBUG:boto:Params: {} 2023-05-31T20:53:51.878 DEBUG:boto:Token: None 2023-05-31T20:53:51.878 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:53:51 GMT /pklxkd-59/obj17?versionId=null 2023-05-31T20:53:51.879 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:MBVlfREoepRCRMQ9+BuLmHmtgRs= 2023-05-31T20:53:51.879 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:53:51 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:MBVlfREoepRCRMQ9+BuLmHmtgRs=', 'Content-Length': '0'} 2023-05-31T20:53:51.881 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:43:07 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d8ae50d0d5c1be29-006477b3df-1c17-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:53:51 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:53:51.882 DEBUG:boto:path=/pklxkd-59/obj17 2023-05-31T20:53:51.882 DEBUG:boto:auth_path=/pklxkd-59/obj17 2023-05-31T20:53:51.883 DEBUG:boto:path=/pklxkd-59/obj17?versionId=null 2023-05-31T20:53:51.883 DEBUG:boto:auth_path=/pklxkd-59/obj17?versionId=null 2023-05-31T20:53:51.883 DEBUG:boto:Method: GET 2023-05-31T20:53:51.884 DEBUG:boto:Path: /pklxkd-59/obj17?versionId=null 2023-05-31T20:53:51.884 DEBUG:boto:Data: 2023-05-31T20:53:51.885 DEBUG:boto:Headers: {} 2023-05-31T20:53:51.885 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:53:51.885 DEBUG:boto:Port: 8000 2023-05-31T20:53:51.886 DEBUG:boto:Params: {} 2023-05-31T20:53:51.886 DEBUG:boto:Token: None 2023-05-31T20:53:51.886 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:53:51 GMT /pklxkd-59/obj17?versionId=null 2023-05-31T20:53:51.887 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:MBVlfREoepRCRMQ9+BuLmHmtgRs= 2023-05-31T20:53:51.887 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:53:51 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:MBVlfREoepRCRMQ9+BuLmHmtgRs=', 'Content-Length': '0'} 2023-05-31T20:53:51.889 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:43:07 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009b3945572d0c3c8e-006477b3df-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:53:51 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:53:51.890 DEBUG:boto:path=/pklxkd-59/obj17 2023-05-31T20:53:51.890 DEBUG:boto:auth_path=/pklxkd-59/obj17 2023-05-31T20:53:51.891 DEBUG:boto:path=/pklxkd-59/obj17?versionId=null 2023-05-31T20:53:51.891 DEBUG:boto:auth_path=/pklxkd-59/obj17?versionId=null 2023-05-31T20:53:51.891 DEBUG:boto:Method: HEAD 2023-05-31T20:53:51.892 DEBUG:boto:Path: /pklxkd-59/obj17?versionId=null 2023-05-31T20:53:51.892 DEBUG:boto:Data: 2023-05-31T20:53:51.892 DEBUG:boto:Headers: {} 2023-05-31T20:53:51.893 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:53:51.893 DEBUG:boto:Port: 8001 2023-05-31T20:53:51.893 DEBUG:boto:Params: {} 2023-05-31T20:53:51.894 DEBUG:boto:Token: None 2023-05-31T20:53:51.894 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:53:51 GMT /pklxkd-59/obj17?versionId=null 2023-05-31T20:53:51.894 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:n6b4akFv8p/i5lau+7u94cAliLw= 2023-05-31T20:53:51.895 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:53:51 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:n6b4akFv8p/i5lau+7u94cAliLw=', 'Content-Length': '0'} 2023-05-31T20:53:51.897 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:43:07 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000014a383e723c3ac97-006477b3df-1c17-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:53:51 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:53:51.897 DEBUG:boto:path=/pklxkd-59/obj17 2023-05-31T20:53:51.898 DEBUG:boto:auth_path=/pklxkd-59/obj17 2023-05-31T20:53:51.898 DEBUG:boto:path=/pklxkd-59/obj17?versionId=null 2023-05-31T20:53:51.898 DEBUG:boto:auth_path=/pklxkd-59/obj17?versionId=null 2023-05-31T20:53:51.899 DEBUG:boto:Method: HEAD 2023-05-31T20:53:51.899 DEBUG:boto:Path: /pklxkd-59/obj17?versionId=null 2023-05-31T20:53:51.899 DEBUG:boto:Data: 2023-05-31T20:53:51.900 DEBUG:boto:Headers: {} 2023-05-31T20:53:51.900 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:53:51.900 DEBUG:boto:Port: 8000 2023-05-31T20:53:51.901 DEBUG:boto:Params: {} 2023-05-31T20:53:51.901 DEBUG:boto:Token: None 2023-05-31T20:53:51.901 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:53:51 GMT /pklxkd-59/obj17?versionId=null 2023-05-31T20:53:51.902 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:n6b4akFv8p/i5lau+7u94cAliLw= 2023-05-31T20:53:51.902 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:53:51 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:n6b4akFv8p/i5lau+7u94cAliLw=', 'Content-Length': '0'} 2023-05-31T20:53:51.904 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:43:07 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007ebec05c3f94ba7d-006477b3df-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:53:51 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:53:51.905 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj17 2023-05-31T20:53:51.905 DEBUG:boto:path=/pklxkd-59/obj17 2023-05-31T20:53:51.905 DEBUG:boto:auth_path=/pklxkd-59/obj17 2023-05-31T20:53:51.906 DEBUG:boto:path=/pklxkd-59/obj17?versionId=null 2023-05-31T20:53:51.906 DEBUG:boto:auth_path=/pklxkd-59/obj17?versionId=null 2023-05-31T20:53:51.906 DEBUG:boto:Method: GET 2023-05-31T20:53:51.907 DEBUG:boto:Path: /pklxkd-59/obj17?versionId=null 2023-05-31T20:53:51.907 DEBUG:boto:Data: 2023-05-31T20:53:51.908 DEBUG:boto:Headers: {} 2023-05-31T20:53:51.908 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:53:51.908 DEBUG:boto:Port: 8001 2023-05-31T20:53:51.909 DEBUG:boto:Params: {} 2023-05-31T20:53:51.909 DEBUG:boto:Token: None 2023-05-31T20:53:51.909 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:53:51 GMT /pklxkd-59/obj17?versionId=null 2023-05-31T20:53:51.910 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:MBVlfREoepRCRMQ9+BuLmHmtgRs= 2023-05-31T20:53:51.910 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:53:51 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:MBVlfREoepRCRMQ9+BuLmHmtgRs=', 'Content-Length': '0'} 2023-05-31T20:53:51.912 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:43:07 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b71649d3543b5aa0-006477b3df-1c17-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:53:51 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:53:51.952 DEBUG:boto:path=/pklxkd-59/obj17 2023-05-31T20:53:51.953 DEBUG:boto:auth_path=/pklxkd-59/obj17 2023-05-31T20:53:51.953 DEBUG:boto:path=/pklxkd-59/obj17?versionId=null 2023-05-31T20:53:51.953 DEBUG:boto:auth_path=/pklxkd-59/obj17?versionId=null 2023-05-31T20:53:51.954 DEBUG:boto:Method: GET 2023-05-31T20:53:51.954 DEBUG:boto:Path: /pklxkd-59/obj17?versionId=null 2023-05-31T20:53:51.955 DEBUG:boto:Data: 2023-05-31T20:53:51.955 DEBUG:boto:Headers: {} 2023-05-31T20:53:51.955 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:53:51.956 DEBUG:boto:Port: 8000 2023-05-31T20:53:51.956 DEBUG:boto:Params: {} 2023-05-31T20:53:51.957 DEBUG:boto:Token: None 2023-05-31T20:53:51.957 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:53:51 GMT /pklxkd-59/obj17?versionId=null 2023-05-31T20:53:51.958 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:MBVlfREoepRCRMQ9+BuLmHmtgRs= 2023-05-31T20:53:51.958 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:53:51 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:MBVlfREoepRCRMQ9+BuLmHmtgRs=', 'Content-Length': '0'} 2023-05-31T20:53:51.960 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:43:07 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000053ddb27e306d6f0a-006477b3df-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:53:51 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:53:51.961 DEBUG:boto:path=/pklxkd-59/ 2023-05-31T20:53:51.961 DEBUG:boto:auth_path=/pklxkd-59/ 2023-05-31T20:53:51.962 DEBUG:boto:path=/pklxkd-59/?prefix=obj17 2023-05-31T20:53:51.962 DEBUG:boto:auth_path=/pklxkd-59/?prefix=obj17 2023-05-31T20:53:51.963 DEBUG:boto:Method: GET 2023-05-31T20:53:51.963 DEBUG:boto:Path: /pklxkd-59/?prefix=obj17 2023-05-31T20:53:51.963 DEBUG:boto:Data: 2023-05-31T20:53:51.964 DEBUG:boto:Headers: {} 2023-05-31T20:53:51.964 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:53:51.964 DEBUG:boto:Port: 8001 2023-05-31T20:53:51.965 DEBUG:boto:Params: {} 2023-05-31T20:53:51.965 DEBUG:boto:Token: None 2023-05-31T20:53:51.966 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:53:51 GMT /pklxkd-59/ 2023-05-31T20:53:51.966 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:lmsrlNekY57vtRsTzICLOlyPcM0= 2023-05-31T20:53:51.966 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:53:51 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:lmsrlNekY57vtRsTzICLOlyPcM0=', 'Content-Length': '0'} 2023-05-31T20:53:51.969 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000002c1a3a00ab43b62d-006477b3df-1c17-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:53:51 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:53:51.969 DEBUG:boto:b'pklxkd-59obj171000falseobj172023-05-31T20:43:07.977Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:53:51.970 DEBUG:boto:path=/pklxkd-59/ 2023-05-31T20:53:51.970 DEBUG:boto:auth_path=/pklxkd-59/ 2023-05-31T20:53:51.971 DEBUG:boto:path=/pklxkd-59/?prefix=obj17 2023-05-31T20:53:51.971 DEBUG:boto:auth_path=/pklxkd-59/?prefix=obj17 2023-05-31T20:53:51.971 DEBUG:boto:Method: GET 2023-05-31T20:53:51.972 DEBUG:boto:Path: /pklxkd-59/?prefix=obj17 2023-05-31T20:53:51.972 DEBUG:boto:Data: 2023-05-31T20:53:51.972 DEBUG:boto:Headers: {} 2023-05-31T20:53:51.973 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:53:51.973 DEBUG:boto:Port: 8000 2023-05-31T20:53:51.973 DEBUG:boto:Params: {} 2023-05-31T20:53:51.974 DEBUG:boto:Token: None 2023-05-31T20:53:51.974 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:53:51 GMT /pklxkd-59/ 2023-05-31T20:53:51.975 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:lmsrlNekY57vtRsTzICLOlyPcM0= 2023-05-31T20:53:51.975 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:53:51 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:lmsrlNekY57vtRsTzICLOlyPcM0=', 'Content-Length': '0'} 2023-05-31T20:53:51.979 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000008935e587be0b07f3-006477b3df-1630-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:53:51 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:53:51.980 DEBUG:boto:b'pklxkd-59obj171000falseobj172023-05-31T20:43:07.977Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:53:51.980 DEBUG:boto:path=/pklxkd-59/obj17 2023-05-31T20:53:51.981 DEBUG:boto:auth_path=/pklxkd-59/obj17 2023-05-31T20:53:51.981 DEBUG:boto:Method: HEAD 2023-05-31T20:53:51.982 DEBUG:boto:Path: /pklxkd-59/obj17 2023-05-31T20:53:51.982 DEBUG:boto:Data: 2023-05-31T20:53:51.982 DEBUG:boto:Headers: {} 2023-05-31T20:53:51.983 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:53:51.983 DEBUG:boto:Port: 8001 2023-05-31T20:53:51.983 DEBUG:boto:Params: {} 2023-05-31T20:53:51.984 DEBUG:boto:Token: None 2023-05-31T20:53:51.984 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:53:51 GMT /pklxkd-59/obj17 2023-05-31T20:53:51.984 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:M9mCCBZMU4hDVpvVPIe9f80EcX4= 2023-05-31T20:53:51.985 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:53:51 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:M9mCCBZMU4hDVpvVPIe9f80EcX4=', 'Content-Length': '0'} 2023-05-31T20:53:51.986 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:43:07 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b777d97ddcf321cd-006477b3df-1c17-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:53:51 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:53:51.987 DEBUG:boto:path=/pklxkd-59/obj17 2023-05-31T20:53:51.987 DEBUG:boto:auth_path=/pklxkd-59/obj17 2023-05-31T20:53:51.988 DEBUG:boto:Method: HEAD 2023-05-31T20:53:51.988 DEBUG:boto:Path: /pklxkd-59/obj17 2023-05-31T20:53:51.988 DEBUG:boto:Data: 2023-05-31T20:53:51.989 DEBUG:boto:Headers: {} 2023-05-31T20:53:51.989 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:53:51.989 DEBUG:boto:Port: 8000 2023-05-31T20:53:51.990 DEBUG:boto:Params: {} 2023-05-31T20:53:51.990 DEBUG:boto:Token: None 2023-05-31T20:53:51.990 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:53:51 GMT /pklxkd-59/obj17 2023-05-31T20:53:51.991 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:M9mCCBZMU4hDVpvVPIe9f80EcX4= 2023-05-31T20:53:51.991 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:53:51 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:M9mCCBZMU4hDVpvVPIe9f80EcX4=', 'Content-Length': '0'} 2023-05-31T20:53:51.993 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:43:07 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007af2ac02382cc694-006477b3df-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:53:51 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:53:51.994 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj17 2023-05-31T20:53:51.994 DEBUG:boto:path=/pklxkd-59/obj17 2023-05-31T20:53:51.995 DEBUG:boto:auth_path=/pklxkd-59/obj17 2023-05-31T20:53:51.995 DEBUG:boto:Method: GET 2023-05-31T20:53:51.995 DEBUG:boto:Path: /pklxkd-59/obj17 2023-05-31T20:53:51.996 DEBUG:boto:Data: 2023-05-31T20:53:51.996 DEBUG:boto:Headers: {} 2023-05-31T20:53:51.997 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:53:51.997 DEBUG:boto:Port: 8001 2023-05-31T20:53:51.997 DEBUG:boto:Params: {} 2023-05-31T20:53:51.998 DEBUG:boto:Token: None 2023-05-31T20:53:51.998 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:53:51 GMT /pklxkd-59/obj17 2023-05-31T20:53:51.999 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:JSamSyqZiIAbg2oq7cOdC0kMUbk= 2023-05-31T20:53:51.999 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:53:51 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:JSamSyqZiIAbg2oq7cOdC0kMUbk=', 'Content-Length': '0'} 2023-05-31T20:53:52.001 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:43:07 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000058d5db491d096177-006477b3df-1c17-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:53:51 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:53:52.002 DEBUG:boto:path=/pklxkd-59/obj17 2023-05-31T20:53:52.002 DEBUG:boto:auth_path=/pklxkd-59/obj17 2023-05-31T20:53:52.003 DEBUG:boto:Method: GET 2023-05-31T20:53:52.003 DEBUG:boto:Path: /pklxkd-59/obj17 2023-05-31T20:53:52.003 DEBUG:boto:Data: 2023-05-31T20:53:52.004 DEBUG:boto:Headers: {} 2023-05-31T20:53:52.004 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:53:52.005 DEBUG:boto:Port: 8000 2023-05-31T20:53:52.005 DEBUG:boto:Params: {} 2023-05-31T20:53:52.006 DEBUG:boto:Token: None 2023-05-31T20:53:52.006 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:53:52 GMT /pklxkd-59/obj17 2023-05-31T20:53:52.006 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:u9P+0eWFVXOLr9IDZPSeIIIu/Qc= 2023-05-31T20:53:52.007 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:53:52 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:u9P+0eWFVXOLr9IDZPSeIIIu/Qc=', 'Content-Length': '0'} 2023-05-31T20:53:52.009 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:43:07 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003b2be73f1206d533-006477b3e0-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:53:52 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:53:52.052 DEBUG:boto:path=/pklxkd-59/ 2023-05-31T20:53:52.053 DEBUG:boto:auth_path=/pklxkd-59/ 2023-05-31T20:53:52.054 DEBUG:boto:path=/pklxkd-59/?prefix=obj17 2023-05-31T20:53:52.054 DEBUG:boto:auth_path=/pklxkd-59/?prefix=obj17 2023-05-31T20:53:52.055 DEBUG:boto:Method: GET 2023-05-31T20:53:52.055 DEBUG:boto:Path: /pklxkd-59/?prefix=obj17 2023-05-31T20:53:52.056 DEBUG:boto:Data: 2023-05-31T20:53:52.056 DEBUG:boto:Headers: {} 2023-05-31T20:53:52.056 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:53:52.057 DEBUG:boto:Port: 8001 2023-05-31T20:53:52.057 DEBUG:boto:Params: {} 2023-05-31T20:53:52.058 DEBUG:boto:Token: None 2023-05-31T20:53:52.058 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:53:52 GMT /pklxkd-59/ 2023-05-31T20:53:52.059 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:Q6NEL+7aW2LGJe6DiOLgBFJA3Bw= 2023-05-31T20:53:52.059 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:53:52 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:Q6NEL+7aW2LGJe6DiOLgBFJA3Bw=', 'Content-Length': '0'} 2023-05-31T20:53:52.062 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b21322f3446d78bc-006477b3e0-1c17-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:53:52 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:53:52.062 DEBUG:boto:b'pklxkd-59obj171000falseobj172023-05-31T20:43:07.977Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:53:52.063 DEBUG:boto:path=/pklxkd-59/ 2023-05-31T20:53:52.063 DEBUG:boto:auth_path=/pklxkd-59/ 2023-05-31T20:53:52.064 DEBUG:boto:path=/pklxkd-59/?prefix=obj17 2023-05-31T20:53:52.064 DEBUG:boto:auth_path=/pklxkd-59/?prefix=obj17 2023-05-31T20:53:52.065 DEBUG:boto:Method: GET 2023-05-31T20:53:52.065 DEBUG:boto:Path: /pklxkd-59/?prefix=obj17 2023-05-31T20:53:52.065 DEBUG:boto:Data: 2023-05-31T20:53:52.066 DEBUG:boto:Headers: {} 2023-05-31T20:53:52.066 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:53:52.067 DEBUG:boto:Port: 8000 2023-05-31T20:53:52.067 DEBUG:boto:Params: {} 2023-05-31T20:53:52.067 DEBUG:boto:Token: None 2023-05-31T20:53:52.068 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:53:52 GMT /pklxkd-59/ 2023-05-31T20:53:52.068 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:Q6NEL+7aW2LGJe6DiOLgBFJA3Bw= 2023-05-31T20:53:52.069 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:53:52 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:Q6NEL+7aW2LGJe6DiOLgBFJA3Bw=', 'Content-Length': '0'} 2023-05-31T20:53:52.073 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000006783a6f64a7f95e-006477b3e0-1630-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:53:52 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:53:52.073 DEBUG:boto:b'pklxkd-59obj171000falseobj172023-05-31T20:43:07.977Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:53:52.074 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj185 2023-05-31T20:53:52.075 DEBUG:boto:path=/pklxkd-59/obj185 2023-05-31T20:53:52.075 DEBUG:boto:auth_path=/pklxkd-59/obj185 2023-05-31T20:53:52.075 DEBUG:boto:path=/pklxkd-59/obj185?versionId=null 2023-05-31T20:53:52.076 DEBUG:boto:auth_path=/pklxkd-59/obj185?versionId=null 2023-05-31T20:53:52.076 DEBUG:boto:Method: GET 2023-05-31T20:53:52.077 DEBUG:boto:Path: /pklxkd-59/obj185?versionId=null 2023-05-31T20:53:52.077 DEBUG:boto:Data: 2023-05-31T20:53:52.077 DEBUG:boto:Headers: {} 2023-05-31T20:53:52.078 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:53:52.078 DEBUG:boto:Port: 8001 2023-05-31T20:53:52.078 DEBUG:boto:Params: {} 2023-05-31T20:53:52.079 DEBUG:boto:Token: None 2023-05-31T20:53:52.079 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:53:52 GMT /pklxkd-59/obj185?versionId=null 2023-05-31T20:53:52.080 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:ilSZ11jPNjPXy6vfKSmRWVyCh6o= 2023-05-31T20:53:52.080 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:53:52 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:ilSZ11jPNjPXy6vfKSmRWVyCh6o=', 'Content-Length': '0'} 2023-05-31T20:53:52.082 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:52:15 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004885e229f9702f20-006477b3e0-1c17-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:53:52 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:53:52.124 DEBUG:boto:path=/pklxkd-59/obj185 2023-05-31T20:53:52.125 DEBUG:boto:auth_path=/pklxkd-59/obj185 2023-05-31T20:53:52.125 DEBUG:boto:path=/pklxkd-59/obj185?versionId=null 2023-05-31T20:53:52.125 DEBUG:boto:auth_path=/pklxkd-59/obj185?versionId=null 2023-05-31T20:53:52.126 DEBUG:boto:Method: GET 2023-05-31T20:53:52.126 DEBUG:boto:Path: /pklxkd-59/obj185?versionId=null 2023-05-31T20:53:52.126 DEBUG:boto:Data: 2023-05-31T20:53:52.127 DEBUG:boto:Headers: {} 2023-05-31T20:53:52.127 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:53:52.128 DEBUG:boto:Port: 8000 2023-05-31T20:53:52.128 DEBUG:boto:Params: {} 2023-05-31T20:53:52.128 DEBUG:boto:Token: None 2023-05-31T20:53:52.129 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:53:52 GMT /pklxkd-59/obj185?versionId=null 2023-05-31T20:53:52.129 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:ilSZ11jPNjPXy6vfKSmRWVyCh6o= 2023-05-31T20:53:52.130 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:53:52 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:ilSZ11jPNjPXy6vfKSmRWVyCh6o=', 'Content-Length': '0'} 2023-05-31T20:53:52.132 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:52:15 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000067fa43011ea2942f-006477b3e0-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:53:52 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:53:52.132 DEBUG:boto:path=/pklxkd-59/obj185 2023-05-31T20:53:52.133 DEBUG:boto:auth_path=/pklxkd-59/obj185 2023-05-31T20:53:52.133 DEBUG:boto:path=/pklxkd-59/obj185?versionId=null 2023-05-31T20:53:52.133 DEBUG:boto:auth_path=/pklxkd-59/obj185?versionId=null 2023-05-31T20:53:52.134 DEBUG:boto:Method: HEAD 2023-05-31T20:53:52.134 DEBUG:boto:Path: /pklxkd-59/obj185?versionId=null 2023-05-31T20:53:52.135 DEBUG:boto:Data: 2023-05-31T20:53:52.135 DEBUG:boto:Headers: {} 2023-05-31T20:53:52.135 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:53:52.136 DEBUG:boto:Port: 8001 2023-05-31T20:53:52.136 DEBUG:boto:Params: {} 2023-05-31T20:53:52.136 DEBUG:boto:Token: None 2023-05-31T20:53:52.137 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:53:52 GMT /pklxkd-59/obj185?versionId=null 2023-05-31T20:53:52.137 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:m46YSq40bQdFIPgHlUxcqUuI0lI= 2023-05-31T20:53:52.137 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:53:52 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:m46YSq40bQdFIPgHlUxcqUuI0lI=', 'Content-Length': '0'} 2023-05-31T20:53:52.139 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:52:15 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000049c1283f296f3175-006477b3e0-1c17-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:53:52 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:53:52.140 DEBUG:boto:path=/pklxkd-59/obj185 2023-05-31T20:53:52.140 DEBUG:boto:auth_path=/pklxkd-59/obj185 2023-05-31T20:53:52.141 DEBUG:boto:path=/pklxkd-59/obj185?versionId=null 2023-05-31T20:53:52.141 DEBUG:boto:auth_path=/pklxkd-59/obj185?versionId=null 2023-05-31T20:53:52.141 DEBUG:boto:Method: HEAD 2023-05-31T20:53:52.142 DEBUG:boto:Path: /pklxkd-59/obj185?versionId=null 2023-05-31T20:53:52.142 DEBUG:boto:Data: 2023-05-31T20:53:52.142 DEBUG:boto:Headers: {} 2023-05-31T20:53:52.143 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:53:52.143 DEBUG:boto:Port: 8000 2023-05-31T20:53:52.143 DEBUG:boto:Params: {} 2023-05-31T20:53:52.144 DEBUG:boto:Token: None 2023-05-31T20:53:52.144 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:53:52 GMT /pklxkd-59/obj185?versionId=null 2023-05-31T20:53:52.145 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:m46YSq40bQdFIPgHlUxcqUuI0lI= 2023-05-31T20:53:52.145 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:53:52 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:m46YSq40bQdFIPgHlUxcqUuI0lI=', 'Content-Length': '0'} 2023-05-31T20:53:52.147 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:52:15 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c90bdbec2efab4ed-006477b3e0-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:53:52 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:53:52.147 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj185 2023-05-31T20:53:52.148 DEBUG:boto:path=/pklxkd-59/obj185 2023-05-31T20:53:52.148 DEBUG:boto:auth_path=/pklxkd-59/obj185 2023-05-31T20:53:52.148 DEBUG:boto:path=/pklxkd-59/obj185?versionId=null 2023-05-31T20:53:52.149 DEBUG:boto:auth_path=/pklxkd-59/obj185?versionId=null 2023-05-31T20:53:52.149 DEBUG:boto:Method: GET 2023-05-31T20:53:52.149 DEBUG:boto:Path: /pklxkd-59/obj185?versionId=null 2023-05-31T20:53:52.150 DEBUG:boto:Data: 2023-05-31T20:53:52.150 DEBUG:boto:Headers: {} 2023-05-31T20:53:52.150 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:53:52.151 DEBUG:boto:Port: 8001 2023-05-31T20:53:52.151 DEBUG:boto:Params: {} 2023-05-31T20:53:52.152 DEBUG:boto:Token: None 2023-05-31T20:53:52.152 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:53:52 GMT /pklxkd-59/obj185?versionId=null 2023-05-31T20:53:52.152 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:ilSZ11jPNjPXy6vfKSmRWVyCh6o= 2023-05-31T20:53:52.153 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:53:52 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:ilSZ11jPNjPXy6vfKSmRWVyCh6o=', 'Content-Length': '0'} 2023-05-31T20:53:52.155 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:52:15 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008c01cf95e5c21e83-006477b3e0-1c17-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:53:52 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:53:52.155 DEBUG:boto:path=/pklxkd-59/obj185 2023-05-31T20:53:52.156 DEBUG:boto:auth_path=/pklxkd-59/obj185 2023-05-31T20:53:52.156 DEBUG:boto:path=/pklxkd-59/obj185?versionId=null 2023-05-31T20:53:52.156 DEBUG:boto:auth_path=/pklxkd-59/obj185?versionId=null 2023-05-31T20:53:52.157 DEBUG:boto:Method: GET 2023-05-31T20:53:52.157 DEBUG:boto:Path: /pklxkd-59/obj185?versionId=null 2023-05-31T20:53:52.158 DEBUG:boto:Data: 2023-05-31T20:53:52.158 DEBUG:boto:Headers: {} 2023-05-31T20:53:52.158 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:53:52.159 DEBUG:boto:Port: 8000 2023-05-31T20:53:52.159 DEBUG:boto:Params: {} 2023-05-31T20:53:52.159 DEBUG:boto:Token: None 2023-05-31T20:53:52.160 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:53:52 GMT /pklxkd-59/obj185?versionId=null 2023-05-31T20:53:52.160 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:ilSZ11jPNjPXy6vfKSmRWVyCh6o= 2023-05-31T20:53:52.161 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:53:52 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:ilSZ11jPNjPXy6vfKSmRWVyCh6o=', 'Content-Length': '0'} 2023-05-31T20:53:52.162 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:52:15 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006df498b872dc47dc-006477b3e0-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:53:52 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:53:52.163 DEBUG:boto:path=/pklxkd-59/ 2023-05-31T20:53:52.164 DEBUG:boto:auth_path=/pklxkd-59/ 2023-05-31T20:53:52.164 DEBUG:boto:path=/pklxkd-59/?prefix=obj185 2023-05-31T20:53:52.164 DEBUG:boto:auth_path=/pklxkd-59/?prefix=obj185 2023-05-31T20:53:52.165 DEBUG:boto:Method: GET 2023-05-31T20:53:52.165 DEBUG:boto:Path: /pklxkd-59/?prefix=obj185 2023-05-31T20:53:52.165 DEBUG:boto:Data: 2023-05-31T20:53:52.166 DEBUG:boto:Headers: {} 2023-05-31T20:53:52.166 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:53:52.166 DEBUG:boto:Port: 8001 2023-05-31T20:53:52.167 DEBUG:boto:Params: {} 2023-05-31T20:53:52.167 DEBUG:boto:Token: None 2023-05-31T20:53:52.168 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:53:52 GMT /pklxkd-59/ 2023-05-31T20:53:52.168 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:Q6NEL+7aW2LGJe6DiOLgBFJA3Bw= 2023-05-31T20:53:52.168 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:53:52 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:Q6NEL+7aW2LGJe6DiOLgBFJA3Bw=', 'Content-Length': '0'} 2023-05-31T20:53:52.171 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000004f89e4348da42cdb-006477b3e0-1c17-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:53:52 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:53:52.171 DEBUG:boto:b'pklxkd-59obj1851000falseobj1852023-05-31T20:52:15.072Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:53:52.172 DEBUG:boto:path=/pklxkd-59/ 2023-05-31T20:53:52.172 DEBUG:boto:auth_path=/pklxkd-59/ 2023-05-31T20:53:52.173 DEBUG:boto:path=/pklxkd-59/?prefix=obj185 2023-05-31T20:53:52.173 DEBUG:boto:auth_path=/pklxkd-59/?prefix=obj185 2023-05-31T20:53:52.173 DEBUG:boto:Method: GET 2023-05-31T20:53:52.174 DEBUG:boto:Path: /pklxkd-59/?prefix=obj185 2023-05-31T20:53:52.174 DEBUG:boto:Data: 2023-05-31T20:53:52.174 DEBUG:boto:Headers: {} 2023-05-31T20:53:52.175 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:53:52.175 DEBUG:boto:Port: 8000 2023-05-31T20:53:52.175 DEBUG:boto:Params: {} 2023-05-31T20:53:52.176 DEBUG:boto:Token: None 2023-05-31T20:53:52.176 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:53:52 GMT /pklxkd-59/ 2023-05-31T20:53:52.177 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:Q6NEL+7aW2LGJe6DiOLgBFJA3Bw= 2023-05-31T20:53:52.177 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:53:52 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:Q6NEL+7aW2LGJe6DiOLgBFJA3Bw=', 'Content-Length': '0'} 2023-05-31T20:53:52.181 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d926b9e596559c11-006477b3e0-1630-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:53:52 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:53:52.182 DEBUG:boto:b'pklxkd-59obj1851000falseobj1852023-05-31T20:52:15.072Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:53:52.182 DEBUG:boto:path=/pklxkd-59/obj185 2023-05-31T20:53:52.183 DEBUG:boto:auth_path=/pklxkd-59/obj185 2023-05-31T20:53:52.183 DEBUG:boto:Method: HEAD 2023-05-31T20:53:52.183 DEBUG:boto:Path: /pklxkd-59/obj185 2023-05-31T20:53:52.184 DEBUG:boto:Data: 2023-05-31T20:53:52.184 DEBUG:boto:Headers: {} 2023-05-31T20:53:52.184 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:53:52.185 DEBUG:boto:Port: 8001 2023-05-31T20:53:52.185 DEBUG:boto:Params: {} 2023-05-31T20:53:52.185 DEBUG:boto:Token: None 2023-05-31T20:53:52.186 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:53:52 GMT /pklxkd-59/obj185 2023-05-31T20:53:52.186 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:QgmcHYdpGWbrDquDCzKWntSwSlU= 2023-05-31T20:53:52.187 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:53:52 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:QgmcHYdpGWbrDquDCzKWntSwSlU=', 'Content-Length': '0'} 2023-05-31T20:53:52.188 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:52:15 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e0303aeefc1ccc88-006477b3e0-1c17-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:53:52 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:53:52.189 DEBUG:boto:path=/pklxkd-59/obj185 2023-05-31T20:53:52.189 DEBUG:boto:auth_path=/pklxkd-59/obj185 2023-05-31T20:53:52.189 DEBUG:boto:Method: HEAD 2023-05-31T20:53:52.190 DEBUG:boto:Path: /pklxkd-59/obj185 2023-05-31T20:53:52.190 DEBUG:boto:Data: 2023-05-31T20:53:52.190 DEBUG:boto:Headers: {} 2023-05-31T20:53:52.191 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:53:52.191 DEBUG:boto:Port: 8000 2023-05-31T20:53:52.191 DEBUG:boto:Params: {} 2023-05-31T20:53:52.192 DEBUG:boto:Token: None 2023-05-31T20:53:52.192 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:53:52 GMT /pklxkd-59/obj185 2023-05-31T20:53:52.193 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:QgmcHYdpGWbrDquDCzKWntSwSlU= 2023-05-31T20:53:52.193 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:53:52 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:QgmcHYdpGWbrDquDCzKWntSwSlU=', 'Content-Length': '0'} 2023-05-31T20:53:52.195 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:52:15 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b286fd92aeca0828-006477b3e0-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:53:52 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:53:52.195 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj185 2023-05-31T20:53:52.196 DEBUG:boto:path=/pklxkd-59/obj185 2023-05-31T20:53:52.196 DEBUG:boto:auth_path=/pklxkd-59/obj185 2023-05-31T20:53:52.196 DEBUG:boto:Method: GET 2023-05-31T20:53:52.197 DEBUG:boto:Path: /pklxkd-59/obj185 2023-05-31T20:53:52.197 DEBUG:boto:Data: 2023-05-31T20:53:52.197 DEBUG:boto:Headers: {} 2023-05-31T20:53:52.198 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:53:52.198 DEBUG:boto:Port: 8001 2023-05-31T20:53:52.199 DEBUG:boto:Params: {} 2023-05-31T20:53:52.199 DEBUG:boto:Token: None 2023-05-31T20:53:52.199 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:53:52 GMT /pklxkd-59/obj185 2023-05-31T20:53:52.200 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:pBhXGTkp5rXfMHZ5ou5Di6JTs4I= 2023-05-31T20:53:52.200 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:53:52 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:pBhXGTkp5rXfMHZ5ou5Di6JTs4I=', 'Content-Length': '0'} 2023-05-31T20:53:52.202 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:52:15 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a732bb65ca91d487-006477b3e0-1c17-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:53:52 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:53:52.202 DEBUG:boto:path=/pklxkd-59/obj185 2023-05-31T20:53:52.203 DEBUG:boto:auth_path=/pklxkd-59/obj185 2023-05-31T20:53:52.203 DEBUG:boto:Method: GET 2023-05-31T20:53:52.204 DEBUG:boto:Path: /pklxkd-59/obj185 2023-05-31T20:53:52.204 DEBUG:boto:Data: 2023-05-31T20:53:52.204 DEBUG:boto:Headers: {} 2023-05-31T20:53:52.205 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:53:52.205 DEBUG:boto:Port: 8000 2023-05-31T20:53:52.205 DEBUG:boto:Params: {} 2023-05-31T20:53:52.206 DEBUG:boto:Token: None 2023-05-31T20:53:52.206 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:53:52 GMT /pklxkd-59/obj185 2023-05-31T20:53:52.207 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:pBhXGTkp5rXfMHZ5ou5Di6JTs4I= 2023-05-31T20:53:52.207 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:53:52 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:pBhXGTkp5rXfMHZ5ou5Di6JTs4I=', 'Content-Length': '0'} 2023-05-31T20:53:52.209 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:52:15 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003a3ec839f69c4dce-006477b3e0-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:53:52 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:53:52.210 DEBUG:boto:path=/pklxkd-59/ 2023-05-31T20:53:52.210 DEBUG:boto:auth_path=/pklxkd-59/ 2023-05-31T20:53:52.210 DEBUG:boto:path=/pklxkd-59/?prefix=obj185 2023-05-31T20:53:52.211 DEBUG:boto:auth_path=/pklxkd-59/?prefix=obj185 2023-05-31T20:53:52.211 DEBUG:boto:Method: GET 2023-05-31T20:53:52.211 DEBUG:boto:Path: /pklxkd-59/?prefix=obj185 2023-05-31T20:53:52.212 DEBUG:boto:Data: 2023-05-31T20:53:52.212 DEBUG:boto:Headers: {} 2023-05-31T20:53:52.212 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:53:52.213 DEBUG:boto:Port: 8001 2023-05-31T20:53:52.213 DEBUG:boto:Params: {} 2023-05-31T20:53:52.214 DEBUG:boto:Token: None 2023-05-31T20:53:52.214 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:53:52 GMT /pklxkd-59/ 2023-05-31T20:53:52.214 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:Q6NEL+7aW2LGJe6DiOLgBFJA3Bw= 2023-05-31T20:53:52.215 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:53:52 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:Q6NEL+7aW2LGJe6DiOLgBFJA3Bw=', 'Content-Length': '0'} 2023-05-31T20:53:52.217 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000cf711073521d4128-006477b3e0-1c17-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:53:52 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:53:52.217 DEBUG:boto:b'pklxkd-59obj1851000falseobj1852023-05-31T20:52:15.072Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:53:52.218 DEBUG:boto:path=/pklxkd-59/ 2023-05-31T20:53:52.218 DEBUG:boto:auth_path=/pklxkd-59/ 2023-05-31T20:53:52.219 DEBUG:boto:path=/pklxkd-59/?prefix=obj185 2023-05-31T20:53:52.219 DEBUG:boto:auth_path=/pklxkd-59/?prefix=obj185 2023-05-31T20:53:52.220 DEBUG:boto:Method: GET 2023-05-31T20:53:52.220 DEBUG:boto:Path: /pklxkd-59/?prefix=obj185 2023-05-31T20:53:52.220 DEBUG:boto:Data: 2023-05-31T20:53:52.221 DEBUG:boto:Headers: {} 2023-05-31T20:53:52.221 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:53:52.221 DEBUG:boto:Port: 8000 2023-05-31T20:53:52.222 DEBUG:boto:Params: {} 2023-05-31T20:53:52.222 DEBUG:boto:Token: None 2023-05-31T20:53:52.223 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:53:52 GMT /pklxkd-59/ 2023-05-31T20:53:52.223 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:Q6NEL+7aW2LGJe6DiOLgBFJA3Bw= 2023-05-31T20:53:52.223 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:53:52 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:Q6NEL+7aW2LGJe6DiOLgBFJA3Bw=', 'Content-Length': '0'} 2023-05-31T20:53:52.227 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000000fab8104ca992c2b-006477b3e0-1630-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:53:52 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:53:52.268 DEBUG:boto:b'pklxkd-59obj1851000falseobj1852023-05-31T20:52:15.072Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:53:52.269 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj19 2023-05-31T20:53:52.269 DEBUG:boto:path=/pklxkd-59/obj19 2023-05-31T20:53:52.270 DEBUG:boto:auth_path=/pklxkd-59/obj19 2023-05-31T20:53:52.270 DEBUG:boto:path=/pklxkd-59/obj19?versionId=null 2023-05-31T20:53:52.270 DEBUG:boto:auth_path=/pklxkd-59/obj19?versionId=null 2023-05-31T20:53:52.271 DEBUG:boto:Method: GET 2023-05-31T20:53:52.271 DEBUG:boto:Path: /pklxkd-59/obj19?versionId=null 2023-05-31T20:53:52.271 DEBUG:boto:Data: 2023-05-31T20:53:52.272 DEBUG:boto:Headers: {} 2023-05-31T20:53:52.272 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:53:52.273 DEBUG:boto:Port: 8001 2023-05-31T20:53:52.273 DEBUG:boto:Params: {} 2023-05-31T20:53:52.273 DEBUG:boto:Token: None 2023-05-31T20:53:52.274 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:53:52 GMT /pklxkd-59/obj19?versionId=null 2023-05-31T20:53:52.274 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:MyETkxgdBN42DObK7nA8K3h4qVU= 2023-05-31T20:53:52.275 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:53:52 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:MyETkxgdBN42DObK7nA8K3h4qVU=', 'Content-Length': '0'} 2023-05-31T20:53:52.276 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:44:50 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000d584c0f0a71f8ac-006477b3e0-1c17-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:53:52 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:53:52.277 DEBUG:boto:path=/pklxkd-59/obj19 2023-05-31T20:53:52.277 DEBUG:boto:auth_path=/pklxkd-59/obj19 2023-05-31T20:53:52.278 DEBUG:boto:path=/pklxkd-59/obj19?versionId=null 2023-05-31T20:53:52.278 DEBUG:boto:auth_path=/pklxkd-59/obj19?versionId=null 2023-05-31T20:53:52.279 DEBUG:boto:Method: GET 2023-05-31T20:53:52.279 DEBUG:boto:Path: /pklxkd-59/obj19?versionId=null 2023-05-31T20:53:52.279 DEBUG:boto:Data: 2023-05-31T20:53:52.280 DEBUG:boto:Headers: {} 2023-05-31T20:53:52.280 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:53:52.280 DEBUG:boto:Port: 8000 2023-05-31T20:53:52.281 DEBUG:boto:Params: {} 2023-05-31T20:53:52.281 DEBUG:boto:Token: None 2023-05-31T20:53:52.282 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:53:52 GMT /pklxkd-59/obj19?versionId=null 2023-05-31T20:53:52.282 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:MyETkxgdBN42DObK7nA8K3h4qVU= 2023-05-31T20:53:52.282 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:53:52 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:MyETkxgdBN42DObK7nA8K3h4qVU=', 'Content-Length': '0'} 2023-05-31T20:53:52.284 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:44:50 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000b20fdf5997e5687-006477b3e0-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:53:52 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:53:52.285 DEBUG:boto:path=/pklxkd-59/obj19 2023-05-31T20:53:52.285 DEBUG:boto:auth_path=/pklxkd-59/obj19 2023-05-31T20:53:52.286 DEBUG:boto:path=/pklxkd-59/obj19?versionId=null 2023-05-31T20:53:52.286 DEBUG:boto:auth_path=/pklxkd-59/obj19?versionId=null 2023-05-31T20:53:52.286 DEBUG:boto:Method: HEAD 2023-05-31T20:53:52.287 DEBUG:boto:Path: /pklxkd-59/obj19?versionId=null 2023-05-31T20:53:52.287 DEBUG:boto:Data: 2023-05-31T20:53:52.287 DEBUG:boto:Headers: {} 2023-05-31T20:53:52.288 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:53:52.288 DEBUG:boto:Port: 8001 2023-05-31T20:53:52.288 DEBUG:boto:Params: {} 2023-05-31T20:53:52.289 DEBUG:boto:Token: None 2023-05-31T20:53:52.289 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:53:52 GMT /pklxkd-59/obj19?versionId=null 2023-05-31T20:53:52.289 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:6wRGvS1MsRZwIaw+uDD33awtbt4= 2023-05-31T20:53:52.290 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:53:52 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:6wRGvS1MsRZwIaw+uDD33awtbt4=', 'Content-Length': '0'} 2023-05-31T20:53:52.292 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:44:50 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003fc97a4689d07240-006477b3e0-1c17-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:53:52 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:53:52.292 DEBUG:boto:path=/pklxkd-59/obj19 2023-05-31T20:53:52.293 DEBUG:boto:auth_path=/pklxkd-59/obj19 2023-05-31T20:53:52.293 DEBUG:boto:path=/pklxkd-59/obj19?versionId=null 2023-05-31T20:53:52.293 DEBUG:boto:auth_path=/pklxkd-59/obj19?versionId=null 2023-05-31T20:53:52.294 DEBUG:boto:Method: HEAD 2023-05-31T20:53:52.294 DEBUG:boto:Path: /pklxkd-59/obj19?versionId=null 2023-05-31T20:53:52.294 DEBUG:boto:Data: 2023-05-31T20:53:52.295 DEBUG:boto:Headers: {} 2023-05-31T20:53:52.295 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:53:52.295 DEBUG:boto:Port: 8000 2023-05-31T20:53:52.296 DEBUG:boto:Params: {} 2023-05-31T20:53:52.296 DEBUG:boto:Token: None 2023-05-31T20:53:52.297 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:53:52 GMT /pklxkd-59/obj19?versionId=null 2023-05-31T20:53:52.297 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:6wRGvS1MsRZwIaw+uDD33awtbt4= 2023-05-31T20:53:52.297 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:53:52 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:6wRGvS1MsRZwIaw+uDD33awtbt4=', 'Content-Length': '0'} 2023-05-31T20:53:52.299 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:44:50 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000441b6695356b3286-006477b3e0-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:53:52 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:53:52.300 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj19 2023-05-31T20:53:52.300 DEBUG:boto:path=/pklxkd-59/obj19 2023-05-31T20:53:52.301 DEBUG:boto:auth_path=/pklxkd-59/obj19 2023-05-31T20:53:52.301 DEBUG:boto:path=/pklxkd-59/obj19?versionId=null 2023-05-31T20:53:52.301 DEBUG:boto:auth_path=/pklxkd-59/obj19?versionId=null 2023-05-31T20:53:52.302 DEBUG:boto:Method: GET 2023-05-31T20:53:52.302 DEBUG:boto:Path: /pklxkd-59/obj19?versionId=null 2023-05-31T20:53:52.302 DEBUG:boto:Data: 2023-05-31T20:53:52.303 DEBUG:boto:Headers: {} 2023-05-31T20:53:52.303 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:53:52.303 DEBUG:boto:Port: 8001 2023-05-31T20:53:52.304 DEBUG:boto:Params: {} 2023-05-31T20:53:52.304 DEBUG:boto:Token: None 2023-05-31T20:53:52.305 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:53:52 GMT /pklxkd-59/obj19?versionId=null 2023-05-31T20:53:52.305 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:MyETkxgdBN42DObK7nA8K3h4qVU= 2023-05-31T20:53:52.306 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:53:52 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:MyETkxgdBN42DObK7nA8K3h4qVU=', 'Content-Length': '0'} 2023-05-31T20:53:52.307 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:44:50 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005729ff8bafe9e279-006477b3e0-1c17-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:53:52 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:53:52.348 DEBUG:boto:path=/pklxkd-59/obj19 2023-05-31T20:53:52.349 DEBUG:boto:auth_path=/pklxkd-59/obj19 2023-05-31T20:53:52.349 DEBUG:boto:path=/pklxkd-59/obj19?versionId=null 2023-05-31T20:53:52.349 DEBUG:boto:auth_path=/pklxkd-59/obj19?versionId=null 2023-05-31T20:53:52.350 DEBUG:boto:Method: GET 2023-05-31T20:53:52.350 DEBUG:boto:Path: /pklxkd-59/obj19?versionId=null 2023-05-31T20:53:52.350 DEBUG:boto:Data: 2023-05-31T20:53:52.351 DEBUG:boto:Headers: {} 2023-05-31T20:53:52.351 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:53:52.352 DEBUG:boto:Port: 8000 2023-05-31T20:53:52.352 DEBUG:boto:Params: {} 2023-05-31T20:53:52.352 DEBUG:boto:Token: None 2023-05-31T20:53:52.353 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:53:52 GMT /pklxkd-59/obj19?versionId=null 2023-05-31T20:53:52.353 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:MyETkxgdBN42DObK7nA8K3h4qVU= 2023-05-31T20:53:52.354 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:53:52 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:MyETkxgdBN42DObK7nA8K3h4qVU=', 'Content-Length': '0'} 2023-05-31T20:53:52.356 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:44:50 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001eb3c577de03ddd4-006477b3e0-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:53:52 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:53:52.356 DEBUG:boto:path=/pklxkd-59/ 2023-05-31T20:53:52.357 DEBUG:boto:auth_path=/pklxkd-59/ 2023-05-31T20:53:52.357 DEBUG:boto:path=/pklxkd-59/?prefix=obj19 2023-05-31T20:53:52.358 DEBUG:boto:auth_path=/pklxkd-59/?prefix=obj19 2023-05-31T20:53:52.358 DEBUG:boto:Method: GET 2023-05-31T20:53:52.358 DEBUG:boto:Path: /pklxkd-59/?prefix=obj19 2023-05-31T20:53:52.359 DEBUG:boto:Data: 2023-05-31T20:53:52.359 DEBUG:boto:Headers: {} 2023-05-31T20:53:52.360 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:53:52.360 DEBUG:boto:Port: 8001 2023-05-31T20:53:52.360 DEBUG:boto:Params: {} 2023-05-31T20:53:52.361 DEBUG:boto:Token: None 2023-05-31T20:53:52.361 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:53:52 GMT /pklxkd-59/ 2023-05-31T20:53:52.362 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:Q6NEL+7aW2LGJe6DiOLgBFJA3Bw= 2023-05-31T20:53:52.362 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:53:52 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:Q6NEL+7aW2LGJe6DiOLgBFJA3Bw=', 'Content-Length': '0'} 2023-05-31T20:53:52.364 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000dfb93ef8f6c914f2-006477b3e0-1c17-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:53:52 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:53:52.365 DEBUG:boto:b'pklxkd-59obj191000falseobj192023-05-31T20:44:50.557Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:53:52.365 DEBUG:boto:path=/pklxkd-59/ 2023-05-31T20:53:52.366 DEBUG:boto:auth_path=/pklxkd-59/ 2023-05-31T20:53:52.366 DEBUG:boto:path=/pklxkd-59/?prefix=obj19 2023-05-31T20:53:52.367 DEBUG:boto:auth_path=/pklxkd-59/?prefix=obj19 2023-05-31T20:53:52.367 DEBUG:boto:Method: GET 2023-05-31T20:53:52.367 DEBUG:boto:Path: /pklxkd-59/?prefix=obj19 2023-05-31T20:53:52.368 DEBUG:boto:Data: 2023-05-31T20:53:52.368 DEBUG:boto:Headers: {} 2023-05-31T20:53:52.368 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:53:52.369 DEBUG:boto:Port: 8000 2023-05-31T20:53:52.369 DEBUG:boto:Params: {} 2023-05-31T20:53:52.369 DEBUG:boto:Token: None 2023-05-31T20:53:52.370 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:53:52 GMT /pklxkd-59/ 2023-05-31T20:53:52.370 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:Q6NEL+7aW2LGJe6DiOLgBFJA3Bw= 2023-05-31T20:53:52.371 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:53:52 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:Q6NEL+7aW2LGJe6DiOLgBFJA3Bw=', 'Content-Length': '0'} 2023-05-31T20:53:52.374 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000caaf0873a7eeec46-006477b3e0-1630-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:53:52 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:53:52.375 DEBUG:boto:b'pklxkd-59obj191000falseobj192023-05-31T20:44:50.557Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:53:52.376 DEBUG:boto:path=/pklxkd-59/obj19 2023-05-31T20:53:52.376 DEBUG:boto:auth_path=/pklxkd-59/obj19 2023-05-31T20:53:52.376 DEBUG:boto:Method: HEAD 2023-05-31T20:53:52.377 DEBUG:boto:Path: /pklxkd-59/obj19 2023-05-31T20:53:52.377 DEBUG:boto:Data: 2023-05-31T20:53:52.377 DEBUG:boto:Headers: {} 2023-05-31T20:53:52.378 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:53:52.378 DEBUG:boto:Port: 8001 2023-05-31T20:53:52.378 DEBUG:boto:Params: {} 2023-05-31T20:53:52.379 DEBUG:boto:Token: None 2023-05-31T20:53:52.379 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:53:52 GMT /pklxkd-59/obj19 2023-05-31T20:53:52.380 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:1pmbkAuHbFPGWdGZhD3hFOz0qrU= 2023-05-31T20:53:52.380 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:53:52 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:1pmbkAuHbFPGWdGZhD3hFOz0qrU=', 'Content-Length': '0'} 2023-05-31T20:53:52.382 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:44:50 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005c8183ab03d1f5e9-006477b3e0-1c17-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:53:52 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:53:52.382 DEBUG:boto:path=/pklxkd-59/obj19 2023-05-31T20:53:52.383 DEBUG:boto:auth_path=/pklxkd-59/obj19 2023-05-31T20:53:52.383 DEBUG:boto:Method: HEAD 2023-05-31T20:53:52.383 DEBUG:boto:Path: /pklxkd-59/obj19 2023-05-31T20:53:52.384 DEBUG:boto:Data: 2023-05-31T20:53:52.384 DEBUG:boto:Headers: {} 2023-05-31T20:53:52.384 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:53:52.385 DEBUG:boto:Port: 8000 2023-05-31T20:53:52.385 DEBUG:boto:Params: {} 2023-05-31T20:53:52.385 DEBUG:boto:Token: None 2023-05-31T20:53:52.386 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:53:52 GMT /pklxkd-59/obj19 2023-05-31T20:53:52.386 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:1pmbkAuHbFPGWdGZhD3hFOz0qrU= 2023-05-31T20:53:52.387 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:53:52 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:1pmbkAuHbFPGWdGZhD3hFOz0qrU=', 'Content-Length': '0'} 2023-05-31T20:53:52.388 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:44:50 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000647079df96e2485f-006477b3e0-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:53:52 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:53:52.389 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj19 2023-05-31T20:53:52.389 DEBUG:boto:path=/pklxkd-59/obj19 2023-05-31T20:53:52.389 DEBUG:boto:auth_path=/pklxkd-59/obj19 2023-05-31T20:53:52.390 DEBUG:boto:Method: GET 2023-05-31T20:53:52.390 DEBUG:boto:Path: /pklxkd-59/obj19 2023-05-31T20:53:52.391 DEBUG:boto:Data: 2023-05-31T20:53:52.391 DEBUG:boto:Headers: {} 2023-05-31T20:53:52.391 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:53:52.392 DEBUG:boto:Port: 8001 2023-05-31T20:53:52.392 DEBUG:boto:Params: {} 2023-05-31T20:53:52.392 DEBUG:boto:Token: None 2023-05-31T20:53:52.393 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:53:52 GMT /pklxkd-59/obj19 2023-05-31T20:53:52.393 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:b1B7uvS4Zuc1rX7G68RO72XPI6k= 2023-05-31T20:53:52.394 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:53:52 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:b1B7uvS4Zuc1rX7G68RO72XPI6k=', 'Content-Length': '0'} 2023-05-31T20:53:52.395 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:44:50 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000aab50dcc7e685ee7-006477b3e0-1c17-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:53:52 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:53:52.396 DEBUG:boto:path=/pklxkd-59/obj19 2023-05-31T20:53:52.397 DEBUG:boto:auth_path=/pklxkd-59/obj19 2023-05-31T20:53:52.397 DEBUG:boto:Method: GET 2023-05-31T20:53:52.397 DEBUG:boto:Path: /pklxkd-59/obj19 2023-05-31T20:53:52.398 DEBUG:boto:Data: 2023-05-31T20:53:52.398 DEBUG:boto:Headers: {} 2023-05-31T20:53:52.398 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:53:52.399 DEBUG:boto:Port: 8000 2023-05-31T20:53:52.399 DEBUG:boto:Params: {} 2023-05-31T20:53:52.399 DEBUG:boto:Token: None 2023-05-31T20:53:52.400 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:53:52 GMT /pklxkd-59/obj19 2023-05-31T20:53:52.400 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:b1B7uvS4Zuc1rX7G68RO72XPI6k= 2023-05-31T20:53:52.401 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:53:52 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:b1B7uvS4Zuc1rX7G68RO72XPI6k=', 'Content-Length': '0'} 2023-05-31T20:53:52.402 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:44:50 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ff8ed7787790ca9a-006477b3e0-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:53:52 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:53:52.444 DEBUG:boto:path=/pklxkd-59/ 2023-05-31T20:53:52.445 DEBUG:boto:auth_path=/pklxkd-59/ 2023-05-31T20:53:52.445 DEBUG:boto:path=/pklxkd-59/?prefix=obj19 2023-05-31T20:53:52.445 DEBUG:boto:auth_path=/pklxkd-59/?prefix=obj19 2023-05-31T20:53:52.446 DEBUG:boto:Method: GET 2023-05-31T20:53:52.446 DEBUG:boto:Path: /pklxkd-59/?prefix=obj19 2023-05-31T20:53:52.447 DEBUG:boto:Data: 2023-05-31T20:53:52.447 DEBUG:boto:Headers: {} 2023-05-31T20:53:52.447 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:53:52.448 DEBUG:boto:Port: 8001 2023-05-31T20:53:52.448 DEBUG:boto:Params: {} 2023-05-31T20:53:52.448 DEBUG:boto:Token: None 2023-05-31T20:53:52.449 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:53:52 GMT /pklxkd-59/ 2023-05-31T20:53:52.449 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:Q6NEL+7aW2LGJe6DiOLgBFJA3Bw= 2023-05-31T20:53:52.450 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:53:52 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:Q6NEL+7aW2LGJe6DiOLgBFJA3Bw=', 'Content-Length': '0'} 2023-05-31T20:53:52.452 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000001455de8f51b61a7d-006477b3e0-1c17-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:53:52 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:53:52.453 DEBUG:boto:b'pklxkd-59obj191000falseobj192023-05-31T20:44:50.557Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:53:52.453 DEBUG:boto:path=/pklxkd-59/ 2023-05-31T20:53:52.454 DEBUG:boto:auth_path=/pklxkd-59/ 2023-05-31T20:53:52.454 DEBUG:boto:path=/pklxkd-59/?prefix=obj19 2023-05-31T20:53:52.454 DEBUG:boto:auth_path=/pklxkd-59/?prefix=obj19 2023-05-31T20:53:52.455 DEBUG:boto:Method: GET 2023-05-31T20:53:52.455 DEBUG:boto:Path: /pklxkd-59/?prefix=obj19 2023-05-31T20:53:52.455 DEBUG:boto:Data: 2023-05-31T20:53:52.456 DEBUG:boto:Headers: {} 2023-05-31T20:53:52.456 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:53:52.456 DEBUG:boto:Port: 8000 2023-05-31T20:53:52.457 DEBUG:boto:Params: {} 2023-05-31T20:53:52.457 DEBUG:boto:Token: None 2023-05-31T20:53:52.458 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:53:52 GMT /pklxkd-59/ 2023-05-31T20:53:52.458 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:Q6NEL+7aW2LGJe6DiOLgBFJA3Bw= 2023-05-31T20:53:52.458 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:53:52 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:Q6NEL+7aW2LGJe6DiOLgBFJA3Bw=', 'Content-Length': '0'} 2023-05-31T20:53:52.463 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000022b2b42679d86d84-006477b3e0-1630-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:53:52 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:53:52.463 DEBUG:boto:b'pklxkd-59obj191000falseobj192023-05-31T20:44:50.557Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:53:52.464 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj22 2023-05-31T20:53:52.464 DEBUG:boto:path=/pklxkd-59/obj22 2023-05-31T20:53:52.465 DEBUG:boto:auth_path=/pklxkd-59/obj22 2023-05-31T20:53:52.465 DEBUG:boto:path=/pklxkd-59/obj22?versionId=null 2023-05-31T20:53:52.465 DEBUG:boto:auth_path=/pklxkd-59/obj22?versionId=null 2023-05-31T20:53:52.466 DEBUG:boto:Method: GET 2023-05-31T20:53:52.466 DEBUG:boto:Path: /pklxkd-59/obj22?versionId=null 2023-05-31T20:53:52.466 DEBUG:boto:Data: 2023-05-31T20:53:52.467 DEBUG:boto:Headers: {} 2023-05-31T20:53:52.467 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:53:52.467 DEBUG:boto:Port: 8001 2023-05-31T20:53:52.468 DEBUG:boto:Params: {} 2023-05-31T20:53:52.468 DEBUG:boto:Token: None 2023-05-31T20:53:52.469 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:53:52 GMT /pklxkd-59/obj22?versionId=null 2023-05-31T20:53:52.469 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:XnYGrwR3lbWCyXmxGEmyOhwWSlk= 2023-05-31T20:53:52.470 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:53:52 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:XnYGrwR3lbWCyXmxGEmyOhwWSlk=', 'Content-Length': '0'} 2023-05-31T20:53:52.472 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:39:26 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005cd4bb54b15c0b63-006477b3e0-1c17-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:53:52 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:53:52.472 DEBUG:boto:path=/pklxkd-59/obj22 2023-05-31T20:53:52.473 DEBUG:boto:auth_path=/pklxkd-59/obj22 2023-05-31T20:53:52.473 DEBUG:boto:path=/pklxkd-59/obj22?versionId=null 2023-05-31T20:53:52.473 DEBUG:boto:auth_path=/pklxkd-59/obj22?versionId=null 2023-05-31T20:53:52.474 DEBUG:boto:Method: GET 2023-05-31T20:53:52.474 DEBUG:boto:Path: /pklxkd-59/obj22?versionId=null 2023-05-31T20:53:52.475 DEBUG:boto:Data: 2023-05-31T20:53:52.475 DEBUG:boto:Headers: {} 2023-05-31T20:53:52.475 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:53:52.476 DEBUG:boto:Port: 8000 2023-05-31T20:53:52.476 DEBUG:boto:Params: {} 2023-05-31T20:53:52.476 DEBUG:boto:Token: None 2023-05-31T20:53:52.477 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:53:52 GMT /pklxkd-59/obj22?versionId=null 2023-05-31T20:53:52.477 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:XnYGrwR3lbWCyXmxGEmyOhwWSlk= 2023-05-31T20:53:52.478 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:53:52 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:XnYGrwR3lbWCyXmxGEmyOhwWSlk=', 'Content-Length': '0'} 2023-05-31T20:53:52.479 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:39:26 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ae8d6b9552f159f3-006477b3e0-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:53:52 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:53:52.480 DEBUG:boto:path=/pklxkd-59/obj22 2023-05-31T20:53:52.480 DEBUG:boto:auth_path=/pklxkd-59/obj22 2023-05-31T20:53:52.481 DEBUG:boto:path=/pklxkd-59/obj22?versionId=null 2023-05-31T20:53:52.481 DEBUG:boto:auth_path=/pklxkd-59/obj22?versionId=null 2023-05-31T20:53:52.481 DEBUG:boto:Method: HEAD 2023-05-31T20:53:52.482 DEBUG:boto:Path: /pklxkd-59/obj22?versionId=null 2023-05-31T20:53:52.482 DEBUG:boto:Data: 2023-05-31T20:53:52.482 DEBUG:boto:Headers: {} 2023-05-31T20:53:52.483 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:53:52.483 DEBUG:boto:Port: 8001 2023-05-31T20:53:52.483 DEBUG:boto:Params: {} 2023-05-31T20:53:52.484 DEBUG:boto:Token: None 2023-05-31T20:53:52.484 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:53:52 GMT /pklxkd-59/obj22?versionId=null 2023-05-31T20:53:52.485 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:Hb+ZPeyjNfXnp01uHgsLlCop7oo= 2023-05-31T20:53:52.485 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:53:52 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:Hb+ZPeyjNfXnp01uHgsLlCop7oo=', 'Content-Length': '0'} 2023-05-31T20:53:52.487 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:39:26 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b34e4bc8e56c8836-006477b3e0-1c17-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:53:52 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:53:52.487 DEBUG:boto:path=/pklxkd-59/obj22 2023-05-31T20:53:52.488 DEBUG:boto:auth_path=/pklxkd-59/obj22 2023-05-31T20:53:52.488 DEBUG:boto:path=/pklxkd-59/obj22?versionId=null 2023-05-31T20:53:52.488 DEBUG:boto:auth_path=/pklxkd-59/obj22?versionId=null 2023-05-31T20:53:52.489 DEBUG:boto:Method: HEAD 2023-05-31T20:53:52.489 DEBUG:boto:Path: /pklxkd-59/obj22?versionId=null 2023-05-31T20:53:52.489 DEBUG:boto:Data: 2023-05-31T20:53:52.490 DEBUG:boto:Headers: {} 2023-05-31T20:53:52.490 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:53:52.490 DEBUG:boto:Port: 8000 2023-05-31T20:53:52.491 DEBUG:boto:Params: {} 2023-05-31T20:53:52.491 DEBUG:boto:Token: None 2023-05-31T20:53:52.492 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:53:52 GMT /pklxkd-59/obj22?versionId=null 2023-05-31T20:53:52.492 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:Hb+ZPeyjNfXnp01uHgsLlCop7oo= 2023-05-31T20:53:52.492 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:53:52 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:Hb+ZPeyjNfXnp01uHgsLlCop7oo=', 'Content-Length': '0'} 2023-05-31T20:53:52.494 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:39:26 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000455c682a5606f51-006477b3e0-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:53:52 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:53:52.495 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj22 2023-05-31T20:53:52.495 DEBUG:boto:path=/pklxkd-59/obj22 2023-05-31T20:53:52.496 DEBUG:boto:auth_path=/pklxkd-59/obj22 2023-05-31T20:53:52.496 DEBUG:boto:path=/pklxkd-59/obj22?versionId=null 2023-05-31T20:53:52.496 DEBUG:boto:auth_path=/pklxkd-59/obj22?versionId=null 2023-05-31T20:53:52.497 DEBUG:boto:Method: GET 2023-05-31T20:53:52.497 DEBUG:boto:Path: /pklxkd-59/obj22?versionId=null 2023-05-31T20:53:52.497 DEBUG:boto:Data: 2023-05-31T20:53:52.498 DEBUG:boto:Headers: {} 2023-05-31T20:53:52.498 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:53:52.498 DEBUG:boto:Port: 8001 2023-05-31T20:53:52.499 DEBUG:boto:Params: {} 2023-05-31T20:53:52.499 DEBUG:boto:Token: None 2023-05-31T20:53:52.500 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:53:52 GMT /pklxkd-59/obj22?versionId=null 2023-05-31T20:53:52.500 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:XnYGrwR3lbWCyXmxGEmyOhwWSlk= 2023-05-31T20:53:52.500 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:53:52 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:XnYGrwR3lbWCyXmxGEmyOhwWSlk=', 'Content-Length': '0'} 2023-05-31T20:53:52.502 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:39:26 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007b061e63b153bb57-006477b3e0-1c17-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:53:52 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:53:52.503 DEBUG:boto:path=/pklxkd-59/obj22 2023-05-31T20:53:52.503 DEBUG:boto:auth_path=/pklxkd-59/obj22 2023-05-31T20:53:52.503 DEBUG:boto:path=/pklxkd-59/obj22?versionId=null 2023-05-31T20:53:52.504 DEBUG:boto:auth_path=/pklxkd-59/obj22?versionId=null 2023-05-31T20:53:52.504 DEBUG:boto:Method: GET 2023-05-31T20:53:52.505 DEBUG:boto:Path: /pklxkd-59/obj22?versionId=null 2023-05-31T20:53:52.505 DEBUG:boto:Data: 2023-05-31T20:53:52.505 DEBUG:boto:Headers: {} 2023-05-31T20:53:52.506 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:53:52.506 DEBUG:boto:Port: 8000 2023-05-31T20:53:52.507 DEBUG:boto:Params: {} 2023-05-31T20:53:52.507 DEBUG:boto:Token: None 2023-05-31T20:53:52.507 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:53:52 GMT /pklxkd-59/obj22?versionId=null 2023-05-31T20:53:52.508 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:XnYGrwR3lbWCyXmxGEmyOhwWSlk= 2023-05-31T20:53:52.508 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:53:52 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:XnYGrwR3lbWCyXmxGEmyOhwWSlk=', 'Content-Length': '0'} 2023-05-31T20:53:52.510 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:39:26 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b1735f44d51c9c27-006477b3e0-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:53:52 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:53:52.511 DEBUG:boto:path=/pklxkd-59/ 2023-05-31T20:53:52.511 DEBUG:boto:auth_path=/pklxkd-59/ 2023-05-31T20:53:52.511 DEBUG:boto:path=/pklxkd-59/?prefix=obj22 2023-05-31T20:53:52.512 DEBUG:boto:auth_path=/pklxkd-59/?prefix=obj22 2023-05-31T20:53:52.512 DEBUG:boto:Method: GET 2023-05-31T20:53:52.513 DEBUG:boto:Path: /pklxkd-59/?prefix=obj22 2023-05-31T20:53:52.513 DEBUG:boto:Data: 2023-05-31T20:53:52.513 DEBUG:boto:Headers: {} 2023-05-31T20:53:52.514 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:53:52.514 DEBUG:boto:Port: 8001 2023-05-31T20:53:52.514 DEBUG:boto:Params: {} 2023-05-31T20:53:52.515 DEBUG:boto:Token: None 2023-05-31T20:53:52.515 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:53:52 GMT /pklxkd-59/ 2023-05-31T20:53:52.516 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:Q6NEL+7aW2LGJe6DiOLgBFJA3Bw= 2023-05-31T20:53:52.516 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:53:52 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:Q6NEL+7aW2LGJe6DiOLgBFJA3Bw=', 'Content-Length': '0'} 2023-05-31T20:53:52.518 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f0557e5f3864ed1d-006477b3e0-1c17-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:53:52 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:53:52.519 DEBUG:boto:b'pklxkd-59obj221000falseobj222023-05-31T20:39:26.473Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:53:52.519 DEBUG:boto:path=/pklxkd-59/ 2023-05-31T20:53:52.520 DEBUG:boto:auth_path=/pklxkd-59/ 2023-05-31T20:53:52.520 DEBUG:boto:path=/pklxkd-59/?prefix=obj22 2023-05-31T20:53:52.521 DEBUG:boto:auth_path=/pklxkd-59/?prefix=obj22 2023-05-31T20:53:52.521 DEBUG:boto:Method: GET 2023-05-31T20:53:52.521 DEBUG:boto:Path: /pklxkd-59/?prefix=obj22 2023-05-31T20:53:52.522 DEBUG:boto:Data: 2023-05-31T20:53:52.522 DEBUG:boto:Headers: {} 2023-05-31T20:53:52.522 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:53:52.523 DEBUG:boto:Port: 8000 2023-05-31T20:53:52.523 DEBUG:boto:Params: {} 2023-05-31T20:53:52.524 DEBUG:boto:Token: None 2023-05-31T20:53:52.524 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:53:52 GMT /pklxkd-59/ 2023-05-31T20:53:52.524 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:Q6NEL+7aW2LGJe6DiOLgBFJA3Bw= 2023-05-31T20:53:52.525 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:53:52 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:Q6NEL+7aW2LGJe6DiOLgBFJA3Bw=', 'Content-Length': '0'} 2023-05-31T20:53:52.529 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d5d82b70eebacd80-006477b3e0-1630-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:53:52 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:53:52.530 DEBUG:boto:b'pklxkd-59obj221000falseobj222023-05-31T20:39:26.473Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:53:52.530 DEBUG:boto:path=/pklxkd-59/obj22 2023-05-31T20:53:52.530 DEBUG:boto:auth_path=/pklxkd-59/obj22 2023-05-31T20:53:52.531 DEBUG:boto:Method: HEAD 2023-05-31T20:53:52.531 DEBUG:boto:Path: /pklxkd-59/obj22 2023-05-31T20:53:52.532 DEBUG:boto:Data: 2023-05-31T20:53:52.532 DEBUG:boto:Headers: {} 2023-05-31T20:53:52.532 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:53:52.533 DEBUG:boto:Port: 8001 2023-05-31T20:53:52.533 DEBUG:boto:Params: {} 2023-05-31T20:53:52.533 DEBUG:boto:Token: None 2023-05-31T20:53:52.534 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:53:52 GMT /pklxkd-59/obj22 2023-05-31T20:53:52.534 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:yqvSQPb+TMAvDQTa64uJ6NsfvpM= 2023-05-31T20:53:52.534 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:53:52 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:yqvSQPb+TMAvDQTa64uJ6NsfvpM=', 'Content-Length': '0'} 2023-05-31T20:53:52.536 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:39:26 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d084188edec3acb5-006477b3e0-1c17-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:53:52 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:53:52.537 DEBUG:boto:path=/pklxkd-59/obj22 2023-05-31T20:53:52.537 DEBUG:boto:auth_path=/pklxkd-59/obj22 2023-05-31T20:53:52.537 DEBUG:boto:Method: HEAD 2023-05-31T20:53:52.538 DEBUG:boto:Path: /pklxkd-59/obj22 2023-05-31T20:53:52.538 DEBUG:boto:Data: 2023-05-31T20:53:52.538 DEBUG:boto:Headers: {} 2023-05-31T20:53:52.539 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:53:52.539 DEBUG:boto:Port: 8000 2023-05-31T20:53:52.539 DEBUG:boto:Params: {} 2023-05-31T20:53:52.540 DEBUG:boto:Token: None 2023-05-31T20:53:52.540 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:53:52 GMT /pklxkd-59/obj22 2023-05-31T20:53:52.541 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:yqvSQPb+TMAvDQTa64uJ6NsfvpM= 2023-05-31T20:53:52.541 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:53:52 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:yqvSQPb+TMAvDQTa64uJ6NsfvpM=', 'Content-Length': '0'} 2023-05-31T20:53:52.543 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:39:26 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000cd5414ac60e5a277-006477b3e0-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:53:52 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:53:52.543 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj22 2023-05-31T20:53:52.543 DEBUG:boto:path=/pklxkd-59/obj22 2023-05-31T20:53:52.544 DEBUG:boto:auth_path=/pklxkd-59/obj22 2023-05-31T20:53:52.544 DEBUG:boto:Method: GET 2023-05-31T20:53:52.545 DEBUG:boto:Path: /pklxkd-59/obj22 2023-05-31T20:53:52.545 DEBUG:boto:Data: 2023-05-31T20:53:52.545 DEBUG:boto:Headers: {} 2023-05-31T20:53:52.546 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:53:52.546 DEBUG:boto:Port: 8001 2023-05-31T20:53:52.546 DEBUG:boto:Params: {} 2023-05-31T20:53:52.547 DEBUG:boto:Token: None 2023-05-31T20:53:52.547 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:53:52 GMT /pklxkd-59/obj22 2023-05-31T20:53:52.548 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:b27OFkX1BFd3eAiFmhLLAS9odms= 2023-05-31T20:53:52.548 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:53:52 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:b27OFkX1BFd3eAiFmhLLAS9odms=', 'Content-Length': '0'} 2023-05-31T20:53:52.550 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:39:26 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b0d0be3cedb2f730-006477b3e0-1c17-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:53:52 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:53:52.550 DEBUG:boto:path=/pklxkd-59/obj22 2023-05-31T20:53:52.551 DEBUG:boto:auth_path=/pklxkd-59/obj22 2023-05-31T20:53:52.551 DEBUG:boto:Method: GET 2023-05-31T20:53:52.551 DEBUG:boto:Path: /pklxkd-59/obj22 2023-05-31T20:53:52.552 DEBUG:boto:Data: 2023-05-31T20:53:52.552 DEBUG:boto:Headers: {} 2023-05-31T20:53:52.553 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:53:52.553 DEBUG:boto:Port: 8000 2023-05-31T20:53:52.553 DEBUG:boto:Params: {} 2023-05-31T20:53:52.554 DEBUG:boto:Token: None 2023-05-31T20:53:52.554 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:53:52 GMT /pklxkd-59/obj22 2023-05-31T20:53:52.554 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:b27OFkX1BFd3eAiFmhLLAS9odms= 2023-05-31T20:53:52.555 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:53:52 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:b27OFkX1BFd3eAiFmhLLAS9odms=', 'Content-Length': '0'} 2023-05-31T20:53:52.557 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:39:26 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000208ba952e004ce06-006477b3e0-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:53:52 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:53:52.600 DEBUG:boto:path=/pklxkd-59/ 2023-05-31T20:53:52.601 DEBUG:boto:auth_path=/pklxkd-59/ 2023-05-31T20:53:52.601 DEBUG:boto:path=/pklxkd-59/?prefix=obj22 2023-05-31T20:53:52.602 DEBUG:boto:auth_path=/pklxkd-59/?prefix=obj22 2023-05-31T20:53:52.602 DEBUG:boto:Method: GET 2023-05-31T20:53:52.602 DEBUG:boto:Path: /pklxkd-59/?prefix=obj22 2023-05-31T20:53:52.603 DEBUG:boto:Data: 2023-05-31T20:53:52.603 DEBUG:boto:Headers: {} 2023-05-31T20:53:52.603 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:53:52.604 DEBUG:boto:Port: 8001 2023-05-31T20:53:52.604 DEBUG:boto:Params: {} 2023-05-31T20:53:52.605 DEBUG:boto:Token: None 2023-05-31T20:53:52.605 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:53:52 GMT /pklxkd-59/ 2023-05-31T20:53:52.606 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:Q6NEL+7aW2LGJe6DiOLgBFJA3Bw= 2023-05-31T20:53:52.606 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:53:52 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:Q6NEL+7aW2LGJe6DiOLgBFJA3Bw=', 'Content-Length': '0'} 2023-05-31T20:53:52.609 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e5f8083665e9bfa1-006477b3e0-1c17-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:53:52 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:53:52.609 DEBUG:boto:b'pklxkd-59obj221000falseobj222023-05-31T20:39:26.473Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:53:52.610 DEBUG:boto:path=/pklxkd-59/ 2023-05-31T20:53:52.610 DEBUG:boto:auth_path=/pklxkd-59/ 2023-05-31T20:53:52.611 DEBUG:boto:path=/pklxkd-59/?prefix=obj22 2023-05-31T20:53:52.611 DEBUG:boto:auth_path=/pklxkd-59/?prefix=obj22 2023-05-31T20:53:52.611 DEBUG:boto:Method: GET 2023-05-31T20:53:52.612 DEBUG:boto:Path: /pklxkd-59/?prefix=obj22 2023-05-31T20:53:52.612 DEBUG:boto:Data: 2023-05-31T20:53:52.613 DEBUG:boto:Headers: {} 2023-05-31T20:53:52.613 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:53:52.613 DEBUG:boto:Port: 8000 2023-05-31T20:53:52.614 DEBUG:boto:Params: {} 2023-05-31T20:53:52.614 DEBUG:boto:Token: None 2023-05-31T20:53:52.614 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:53:52 GMT /pklxkd-59/ 2023-05-31T20:53:52.615 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:Q6NEL+7aW2LGJe6DiOLgBFJA3Bw= 2023-05-31T20:53:52.615 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:53:52 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:Q6NEL+7aW2LGJe6DiOLgBFJA3Bw=', 'Content-Length': '0'} 2023-05-31T20:53:52.619 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000fd51bf240e74c407-006477b3e0-1630-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:53:52 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:53:52.620 DEBUG:boto:b'pklxkd-59obj221000falseobj222023-05-31T20:39:26.473Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:53:52.621 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj23 2023-05-31T20:53:52.621 DEBUG:boto:path=/pklxkd-59/obj23 2023-05-31T20:53:52.621 DEBUG:boto:auth_path=/pklxkd-59/obj23 2023-05-31T20:53:52.622 DEBUG:boto:path=/pklxkd-59/obj23?versionId=null 2023-05-31T20:53:52.622 DEBUG:boto:auth_path=/pklxkd-59/obj23?versionId=null 2023-05-31T20:53:52.622 DEBUG:boto:Method: GET 2023-05-31T20:53:52.623 DEBUG:boto:Path: /pklxkd-59/obj23?versionId=null 2023-05-31T20:53:52.623 DEBUG:boto:Data: 2023-05-31T20:53:52.624 DEBUG:boto:Headers: {} 2023-05-31T20:53:52.624 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:53:52.624 DEBUG:boto:Port: 8001 2023-05-31T20:53:52.625 DEBUG:boto:Params: {} 2023-05-31T20:53:52.625 DEBUG:boto:Token: None 2023-05-31T20:53:52.625 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:53:52 GMT /pklxkd-59/obj23?versionId=null 2023-05-31T20:53:52.626 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:mdcb21ZPfjgJq92JaGvkFXB8IC8= 2023-05-31T20:53:52.626 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:53:52 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:mdcb21ZPfjgJq92JaGvkFXB8IC8=', 'Content-Length': '0'} 2023-05-31T20:53:52.628 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:46:33 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000220074e18ff1d3db-006477b3e0-1c17-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:53:52 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:53:52.672 DEBUG:boto:path=/pklxkd-59/obj23 2023-05-31T20:53:52.673 DEBUG:boto:auth_path=/pklxkd-59/obj23 2023-05-31T20:53:52.673 DEBUG:boto:path=/pklxkd-59/obj23?versionId=null 2023-05-31T20:53:52.673 DEBUG:boto:auth_path=/pklxkd-59/obj23?versionId=null 2023-05-31T20:53:52.674 DEBUG:boto:Method: GET 2023-05-31T20:53:52.674 DEBUG:boto:Path: /pklxkd-59/obj23?versionId=null 2023-05-31T20:53:52.674 DEBUG:boto:Data: 2023-05-31T20:53:52.675 DEBUG:boto:Headers: {} 2023-05-31T20:53:52.675 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:53:52.676 DEBUG:boto:Port: 8000 2023-05-31T20:53:52.676 DEBUG:boto:Params: {} 2023-05-31T20:53:52.676 DEBUG:boto:Token: None 2023-05-31T20:53:52.677 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:53:52 GMT /pklxkd-59/obj23?versionId=null 2023-05-31T20:53:52.677 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:mdcb21ZPfjgJq92JaGvkFXB8IC8= 2023-05-31T20:53:52.678 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:53:52 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:mdcb21ZPfjgJq92JaGvkFXB8IC8=', 'Content-Length': '0'} 2023-05-31T20:53:52.680 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:46:33 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000006194f2e36d5b533-006477b3e0-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:53:52 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:53:52.680 DEBUG:boto:path=/pklxkd-59/obj23 2023-05-31T20:53:52.681 DEBUG:boto:auth_path=/pklxkd-59/obj23 2023-05-31T20:53:52.681 DEBUG:boto:path=/pklxkd-59/obj23?versionId=null 2023-05-31T20:53:52.681 DEBUG:boto:auth_path=/pklxkd-59/obj23?versionId=null 2023-05-31T20:53:52.682 DEBUG:boto:Method: HEAD 2023-05-31T20:53:52.682 DEBUG:boto:Path: /pklxkd-59/obj23?versionId=null 2023-05-31T20:53:52.682 DEBUG:boto:Data: 2023-05-31T20:53:52.683 DEBUG:boto:Headers: {} 2023-05-31T20:53:52.683 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:53:52.683 DEBUG:boto:Port: 8001 2023-05-31T20:53:52.684 DEBUG:boto:Params: {} 2023-05-31T20:53:52.684 DEBUG:boto:Token: None 2023-05-31T20:53:52.684 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:53:52 GMT /pklxkd-59/obj23?versionId=null 2023-05-31T20:53:52.685 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:zheZHGfbb/KtkQM9SeHO7qZkLO4= 2023-05-31T20:53:52.685 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:53:52 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:zheZHGfbb/KtkQM9SeHO7qZkLO4=', 'Content-Length': '0'} 2023-05-31T20:53:52.687 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:46:33 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009f2f1c069ef81dd9-006477b3e0-1c17-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:53:52 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:53:52.687 DEBUG:boto:path=/pklxkd-59/obj23 2023-05-31T20:53:52.688 DEBUG:boto:auth_path=/pklxkd-59/obj23 2023-05-31T20:53:52.688 DEBUG:boto:path=/pklxkd-59/obj23?versionId=null 2023-05-31T20:53:52.689 DEBUG:boto:auth_path=/pklxkd-59/obj23?versionId=null 2023-05-31T20:53:52.689 DEBUG:boto:Method: HEAD 2023-05-31T20:53:52.689 DEBUG:boto:Path: /pklxkd-59/obj23?versionId=null 2023-05-31T20:53:52.690 DEBUG:boto:Data: 2023-05-31T20:53:52.690 DEBUG:boto:Headers: {} 2023-05-31T20:53:52.690 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:53:52.690 DEBUG:boto:Port: 8000 2023-05-31T20:53:52.691 DEBUG:boto:Params: {} 2023-05-31T20:53:52.691 DEBUG:boto:Token: None 2023-05-31T20:53:52.692 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:53:52 GMT /pklxkd-59/obj23?versionId=null 2023-05-31T20:53:52.692 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:zheZHGfbb/KtkQM9SeHO7qZkLO4= 2023-05-31T20:53:52.692 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:53:52 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:zheZHGfbb/KtkQM9SeHO7qZkLO4=', 'Content-Length': '0'} 2023-05-31T20:53:52.694 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:46:33 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000aa5466a36cee11aa-006477b3e0-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:53:52 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:53:52.695 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj23 2023-05-31T20:53:52.695 DEBUG:boto:path=/pklxkd-59/obj23 2023-05-31T20:53:52.696 DEBUG:boto:auth_path=/pklxkd-59/obj23 2023-05-31T20:53:52.696 DEBUG:boto:path=/pklxkd-59/obj23?versionId=null 2023-05-31T20:53:52.696 DEBUG:boto:auth_path=/pklxkd-59/obj23?versionId=null 2023-05-31T20:53:52.697 DEBUG:boto:Method: GET 2023-05-31T20:53:52.697 DEBUG:boto:Path: /pklxkd-59/obj23?versionId=null 2023-05-31T20:53:52.697 DEBUG:boto:Data: 2023-05-31T20:53:52.698 DEBUG:boto:Headers: {} 2023-05-31T20:53:52.698 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:53:52.698 DEBUG:boto:Port: 8001 2023-05-31T20:53:52.699 DEBUG:boto:Params: {} 2023-05-31T20:53:52.699 DEBUG:boto:Token: None 2023-05-31T20:53:52.700 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:53:52 GMT /pklxkd-59/obj23?versionId=null 2023-05-31T20:53:52.700 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:mdcb21ZPfjgJq92JaGvkFXB8IC8= 2023-05-31T20:53:52.701 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:53:52 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:mdcb21ZPfjgJq92JaGvkFXB8IC8=', 'Content-Length': '0'} 2023-05-31T20:53:52.702 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:46:33 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005f97cebc50ada0a3-006477b3e0-1c17-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:53:52 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:53:52.703 DEBUG:boto:path=/pklxkd-59/obj23 2023-05-31T20:53:52.703 DEBUG:boto:auth_path=/pklxkd-59/obj23 2023-05-31T20:53:52.704 DEBUG:boto:path=/pklxkd-59/obj23?versionId=null 2023-05-31T20:53:52.704 DEBUG:boto:auth_path=/pklxkd-59/obj23?versionId=null 2023-05-31T20:53:52.704 DEBUG:boto:Method: GET 2023-05-31T20:53:52.705 DEBUG:boto:Path: /pklxkd-59/obj23?versionId=null 2023-05-31T20:53:52.705 DEBUG:boto:Data: 2023-05-31T20:53:52.706 DEBUG:boto:Headers: {} 2023-05-31T20:53:52.706 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:53:52.706 DEBUG:boto:Port: 8000 2023-05-31T20:53:52.707 DEBUG:boto:Params: {} 2023-05-31T20:53:52.707 DEBUG:boto:Token: None 2023-05-31T20:53:52.708 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:53:52 GMT /pklxkd-59/obj23?versionId=null 2023-05-31T20:53:52.708 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:mdcb21ZPfjgJq92JaGvkFXB8IC8= 2023-05-31T20:53:52.708 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:53:52 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:mdcb21ZPfjgJq92JaGvkFXB8IC8=', 'Content-Length': '0'} 2023-05-31T20:53:52.710 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:46:33 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002443241bd6e32af0-006477b3e0-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:53:52 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:53:52.711 DEBUG:boto:path=/pklxkd-59/ 2023-05-31T20:53:52.711 DEBUG:boto:auth_path=/pklxkd-59/ 2023-05-31T20:53:52.712 DEBUG:boto:path=/pklxkd-59/?prefix=obj23 2023-05-31T20:53:52.712 DEBUG:boto:auth_path=/pklxkd-59/?prefix=obj23 2023-05-31T20:53:52.712 DEBUG:boto:Method: GET 2023-05-31T20:53:52.713 DEBUG:boto:Path: /pklxkd-59/?prefix=obj23 2023-05-31T20:53:52.713 DEBUG:boto:Data: 2023-05-31T20:53:52.713 DEBUG:boto:Headers: {} 2023-05-31T20:53:52.714 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:53:52.714 DEBUG:boto:Port: 8001 2023-05-31T20:53:52.714 DEBUG:boto:Params: {} 2023-05-31T20:53:52.715 DEBUG:boto:Token: None 2023-05-31T20:53:52.715 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:53:52 GMT /pklxkd-59/ 2023-05-31T20:53:52.716 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:Q6NEL+7aW2LGJe6DiOLgBFJA3Bw= 2023-05-31T20:53:52.716 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:53:52 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:Q6NEL+7aW2LGJe6DiOLgBFJA3Bw=', 'Content-Length': '0'} 2023-05-31T20:53:52.718 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b50cb5ccd6b73b28-006477b3e0-1c17-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:53:52 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:53:52.719 DEBUG:boto:b'pklxkd-59obj231000falseobj232023-05-31T20:46:33.813Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:53:52.719 DEBUG:boto:path=/pklxkd-59/ 2023-05-31T20:53:52.720 DEBUG:boto:auth_path=/pklxkd-59/ 2023-05-31T20:53:52.720 DEBUG:boto:path=/pklxkd-59/?prefix=obj23 2023-05-31T20:53:52.721 DEBUG:boto:auth_path=/pklxkd-59/?prefix=obj23 2023-05-31T20:53:52.721 DEBUG:boto:Method: GET 2023-05-31T20:53:52.721 DEBUG:boto:Path: /pklxkd-59/?prefix=obj23 2023-05-31T20:53:52.722 DEBUG:boto:Data: 2023-05-31T20:53:52.722 DEBUG:boto:Headers: {} 2023-05-31T20:53:52.722 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:53:52.723 DEBUG:boto:Port: 8000 2023-05-31T20:53:52.723 DEBUG:boto:Params: {} 2023-05-31T20:53:52.723 DEBUG:boto:Token: None 2023-05-31T20:53:52.724 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:53:52 GMT /pklxkd-59/ 2023-05-31T20:53:52.724 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:Q6NEL+7aW2LGJe6DiOLgBFJA3Bw= 2023-05-31T20:53:52.725 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:53:52 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:Q6NEL+7aW2LGJe6DiOLgBFJA3Bw=', 'Content-Length': '0'} 2023-05-31T20:53:52.728 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000085b48b5e8983f981-006477b3e0-1630-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:53:52 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:53:52.729 DEBUG:boto:b'pklxkd-59obj231000falseobj232023-05-31T20:46:33.813Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:53:52.730 DEBUG:boto:path=/pklxkd-59/obj23 2023-05-31T20:53:52.730 DEBUG:boto:auth_path=/pklxkd-59/obj23 2023-05-31T20:53:52.730 DEBUG:boto:Method: HEAD 2023-05-31T20:53:52.731 DEBUG:boto:Path: /pklxkd-59/obj23 2023-05-31T20:53:52.731 DEBUG:boto:Data: 2023-05-31T20:53:52.731 DEBUG:boto:Headers: {} 2023-05-31T20:53:52.732 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:53:52.732 DEBUG:boto:Port: 8001 2023-05-31T20:53:52.732 DEBUG:boto:Params: {} 2023-05-31T20:53:52.733 DEBUG:boto:Token: None 2023-05-31T20:53:52.733 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:53:52 GMT /pklxkd-59/obj23 2023-05-31T20:53:52.733 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:k2KOOC+8RAobVS8e6tsqFgBFNAU= 2023-05-31T20:53:52.734 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:53:52 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:k2KOOC+8RAobVS8e6tsqFgBFNAU=', 'Content-Length': '0'} 2023-05-31T20:53:52.735 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:46:33 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f8424c0230131102-006477b3e0-1c17-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:53:52 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:53:52.736 DEBUG:boto:path=/pklxkd-59/obj23 2023-05-31T20:53:52.736 DEBUG:boto:auth_path=/pklxkd-59/obj23 2023-05-31T20:53:52.737 DEBUG:boto:Method: HEAD 2023-05-31T20:53:52.737 DEBUG:boto:Path: /pklxkd-59/obj23 2023-05-31T20:53:52.737 DEBUG:boto:Data: 2023-05-31T20:53:52.738 DEBUG:boto:Headers: {} 2023-05-31T20:53:52.738 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:53:52.738 DEBUG:boto:Port: 8000 2023-05-31T20:53:52.739 DEBUG:boto:Params: {} 2023-05-31T20:53:52.739 DEBUG:boto:Token: None 2023-05-31T20:53:52.740 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:53:52 GMT /pklxkd-59/obj23 2023-05-31T20:53:52.740 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:k2KOOC+8RAobVS8e6tsqFgBFNAU= 2023-05-31T20:53:52.740 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:53:52 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:k2KOOC+8RAobVS8e6tsqFgBFNAU=', 'Content-Length': '0'} 2023-05-31T20:53:52.742 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:46:33 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000978bfdf1f16ca4f1-006477b3e0-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:53:52 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:53:52.743 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj23 2023-05-31T20:53:52.743 DEBUG:boto:path=/pklxkd-59/obj23 2023-05-31T20:53:52.743 DEBUG:boto:auth_path=/pklxkd-59/obj23 2023-05-31T20:53:52.744 DEBUG:boto:Method: GET 2023-05-31T20:53:52.744 DEBUG:boto:Path: /pklxkd-59/obj23 2023-05-31T20:53:52.744 DEBUG:boto:Data: 2023-05-31T20:53:52.745 DEBUG:boto:Headers: {} 2023-05-31T20:53:52.745 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:53:52.746 DEBUG:boto:Port: 8001 2023-05-31T20:53:52.746 DEBUG:boto:Params: {} 2023-05-31T20:53:52.746 DEBUG:boto:Token: None 2023-05-31T20:53:52.747 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:53:52 GMT /pklxkd-59/obj23 2023-05-31T20:53:52.747 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:CRHdBcvXMxxBHN8PF/lciSWKwRY= 2023-05-31T20:53:52.747 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:53:52 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:CRHdBcvXMxxBHN8PF/lciSWKwRY=', 'Content-Length': '0'} 2023-05-31T20:53:52.749 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:46:33 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000040f623743b4be4c2-006477b3e0-1c17-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:53:52 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:53:52.750 DEBUG:boto:path=/pklxkd-59/obj23 2023-05-31T20:53:52.750 DEBUG:boto:auth_path=/pklxkd-59/obj23 2023-05-31T20:53:52.751 DEBUG:boto:Method: GET 2023-05-31T20:53:52.751 DEBUG:boto:Path: /pklxkd-59/obj23 2023-05-31T20:53:52.751 DEBUG:boto:Data: 2023-05-31T20:53:52.752 DEBUG:boto:Headers: {} 2023-05-31T20:53:52.752 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:53:52.752 DEBUG:boto:Port: 8000 2023-05-31T20:53:52.753 DEBUG:boto:Params: {} 2023-05-31T20:53:52.753 DEBUG:boto:Token: None 2023-05-31T20:53:52.754 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:53:52 GMT /pklxkd-59/obj23 2023-05-31T20:53:52.754 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:CRHdBcvXMxxBHN8PF/lciSWKwRY= 2023-05-31T20:53:52.754 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:53:52 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:CRHdBcvXMxxBHN8PF/lciSWKwRY=', 'Content-Length': '0'} 2023-05-31T20:53:52.756 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:46:33 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006e6bf1d4a7f55e6e-006477b3e0-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:53:52 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:53:52.757 DEBUG:boto:path=/pklxkd-59/ 2023-05-31T20:53:52.757 DEBUG:boto:auth_path=/pklxkd-59/ 2023-05-31T20:53:52.758 DEBUG:boto:path=/pklxkd-59/?prefix=obj23 2023-05-31T20:53:52.758 DEBUG:boto:auth_path=/pklxkd-59/?prefix=obj23 2023-05-31T20:53:52.758 DEBUG:boto:Method: GET 2023-05-31T20:53:52.759 DEBUG:boto:Path: /pklxkd-59/?prefix=obj23 2023-05-31T20:53:52.759 DEBUG:boto:Data: 2023-05-31T20:53:52.759 DEBUG:boto:Headers: {} 2023-05-31T20:53:52.760 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:53:52.760 DEBUG:boto:Port: 8001 2023-05-31T20:53:52.761 DEBUG:boto:Params: {} 2023-05-31T20:53:52.761 DEBUG:boto:Token: None 2023-05-31T20:53:52.761 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:53:52 GMT /pklxkd-59/ 2023-05-31T20:53:52.762 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:Q6NEL+7aW2LGJe6DiOLgBFJA3Bw= 2023-05-31T20:53:52.762 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:53:52 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:Q6NEL+7aW2LGJe6DiOLgBFJA3Bw=', 'Content-Length': '0'} 2023-05-31T20:53:52.765 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000006f5a6ba6753b0c17-006477b3e0-1c17-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:53:52 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:53:52.765 DEBUG:boto:b'pklxkd-59obj231000falseobj232023-05-31T20:46:33.813Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:53:52.766 DEBUG:boto:path=/pklxkd-59/ 2023-05-31T20:53:52.766 DEBUG:boto:auth_path=/pklxkd-59/ 2023-05-31T20:53:52.766 DEBUG:boto:path=/pklxkd-59/?prefix=obj23 2023-05-31T20:53:52.767 DEBUG:boto:auth_path=/pklxkd-59/?prefix=obj23 2023-05-31T20:53:52.767 DEBUG:boto:Method: GET 2023-05-31T20:53:52.767 DEBUG:boto:Path: /pklxkd-59/?prefix=obj23 2023-05-31T20:53:52.768 DEBUG:boto:Data: 2023-05-31T20:53:52.768 DEBUG:boto:Headers: {} 2023-05-31T20:53:52.769 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:53:52.769 DEBUG:boto:Port: 8000 2023-05-31T20:53:52.769 DEBUG:boto:Params: {} 2023-05-31T20:53:52.770 DEBUG:boto:Token: None 2023-05-31T20:53:52.770 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:53:52 GMT /pklxkd-59/ 2023-05-31T20:53:52.770 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:Q6NEL+7aW2LGJe6DiOLgBFJA3Bw= 2023-05-31T20:53:52.771 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:53:52 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:Q6NEL+7aW2LGJe6DiOLgBFJA3Bw=', 'Content-Length': '0'} 2023-05-31T20:53:52.775 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000001c20dbecf213b5-006477b3e0-1630-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:53:52 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:53:52.816 DEBUG:boto:b'pklxkd-59obj231000falseobj232023-05-31T20:46:33.813Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:53:52.817 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj29 2023-05-31T20:53:52.817 DEBUG:boto:path=/pklxkd-59/obj29 2023-05-31T20:53:52.818 DEBUG:boto:auth_path=/pklxkd-59/obj29 2023-05-31T20:53:52.818 DEBUG:boto:path=/pklxkd-59/obj29?versionId=null 2023-05-31T20:53:52.818 DEBUG:boto:auth_path=/pklxkd-59/obj29?versionId=null 2023-05-31T20:53:52.819 DEBUG:boto:Method: GET 2023-05-31T20:53:52.819 DEBUG:boto:Path: /pklxkd-59/obj29?versionId=null 2023-05-31T20:53:52.819 DEBUG:boto:Data: 2023-05-31T20:53:52.820 DEBUG:boto:Headers: {} 2023-05-31T20:53:52.820 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:53:52.820 DEBUG:boto:Port: 8001 2023-05-31T20:53:52.821 DEBUG:boto:Params: {} 2023-05-31T20:53:52.821 DEBUG:boto:Token: None 2023-05-31T20:53:52.822 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:53:52 GMT /pklxkd-59/obj29?versionId=null 2023-05-31T20:53:52.822 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:raMzklBkQbkOsAS2ieyVeGqXGJ4= 2023-05-31T20:53:52.822 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:53:52 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:raMzklBkQbkOsAS2ieyVeGqXGJ4=', 'Content-Length': '0'} 2023-05-31T20:53:52.824 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:48:12 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005d1c70edf803d4e6-006477b3e0-1c17-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:53:52 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:53:52.825 DEBUG:boto:path=/pklxkd-59/obj29 2023-05-31T20:53:52.825 DEBUG:boto:auth_path=/pklxkd-59/obj29 2023-05-31T20:53:52.826 DEBUG:boto:path=/pklxkd-59/obj29?versionId=null 2023-05-31T20:53:52.826 DEBUG:boto:auth_path=/pklxkd-59/obj29?versionId=null 2023-05-31T20:53:52.827 DEBUG:boto:Method: GET 2023-05-31T20:53:52.827 DEBUG:boto:Path: /pklxkd-59/obj29?versionId=null 2023-05-31T20:53:52.827 DEBUG:boto:Data: 2023-05-31T20:53:52.828 DEBUG:boto:Headers: {} 2023-05-31T20:53:52.828 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:53:52.828 DEBUG:boto:Port: 8000 2023-05-31T20:53:52.829 DEBUG:boto:Params: {} 2023-05-31T20:53:52.829 DEBUG:boto:Token: None 2023-05-31T20:53:52.829 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:53:52 GMT /pklxkd-59/obj29?versionId=null 2023-05-31T20:53:52.830 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:raMzklBkQbkOsAS2ieyVeGqXGJ4= 2023-05-31T20:53:52.830 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:53:52 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:raMzklBkQbkOsAS2ieyVeGqXGJ4=', 'Content-Length': '0'} 2023-05-31T20:53:52.832 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:48:12 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003b59f4d4304a431a-006477b3e0-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:53:52 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:53:52.833 DEBUG:boto:path=/pklxkd-59/obj29 2023-05-31T20:53:52.833 DEBUG:boto:auth_path=/pklxkd-59/obj29 2023-05-31T20:53:52.834 DEBUG:boto:path=/pklxkd-59/obj29?versionId=null 2023-05-31T20:53:52.834 DEBUG:boto:auth_path=/pklxkd-59/obj29?versionId=null 2023-05-31T20:53:52.834 DEBUG:boto:Method: HEAD 2023-05-31T20:53:52.835 DEBUG:boto:Path: /pklxkd-59/obj29?versionId=null 2023-05-31T20:53:52.835 DEBUG:boto:Data: 2023-05-31T20:53:52.835 DEBUG:boto:Headers: {} 2023-05-31T20:53:52.836 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:53:52.836 DEBUG:boto:Port: 8001 2023-05-31T20:53:52.836 DEBUG:boto:Params: {} 2023-05-31T20:53:52.837 DEBUG:boto:Token: None 2023-05-31T20:53:52.837 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:53:52 GMT /pklxkd-59/obj29?versionId=null 2023-05-31T20:53:52.837 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:xGvQnzW6+lJd2hFrAy7mfHi76gw= 2023-05-31T20:53:52.838 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:53:52 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:xGvQnzW6+lJd2hFrAy7mfHi76gw=', 'Content-Length': '0'} 2023-05-31T20:53:52.840 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:48:12 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c755e21711887c9a-006477b3e0-1c17-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:53:52 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:53:52.840 DEBUG:boto:path=/pklxkd-59/obj29 2023-05-31T20:53:52.840 DEBUG:boto:auth_path=/pklxkd-59/obj29 2023-05-31T20:53:52.841 DEBUG:boto:path=/pklxkd-59/obj29?versionId=null 2023-05-31T20:53:52.841 DEBUG:boto:auth_path=/pklxkd-59/obj29?versionId=null 2023-05-31T20:53:52.841 DEBUG:boto:Method: HEAD 2023-05-31T20:53:52.842 DEBUG:boto:Path: /pklxkd-59/obj29?versionId=null 2023-05-31T20:53:52.842 DEBUG:boto:Data: 2023-05-31T20:53:52.842 DEBUG:boto:Headers: {} 2023-05-31T20:53:52.843 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:53:52.843 DEBUG:boto:Port: 8000 2023-05-31T20:53:52.843 DEBUG:boto:Params: {} 2023-05-31T20:53:52.844 DEBUG:boto:Token: None 2023-05-31T20:53:52.844 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:53:52 GMT /pklxkd-59/obj29?versionId=null 2023-05-31T20:53:52.845 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:xGvQnzW6+lJd2hFrAy7mfHi76gw= 2023-05-31T20:53:52.845 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:53:52 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:xGvQnzW6+lJd2hFrAy7mfHi76gw=', 'Content-Length': '0'} 2023-05-31T20:53:52.847 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:48:12 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000345cd5f824547ab1-006477b3e0-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:53:52 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:53:52.847 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj29 2023-05-31T20:53:52.848 DEBUG:boto:path=/pklxkd-59/obj29 2023-05-31T20:53:52.848 DEBUG:boto:auth_path=/pklxkd-59/obj29 2023-05-31T20:53:52.848 DEBUG:boto:path=/pklxkd-59/obj29?versionId=null 2023-05-31T20:53:52.849 DEBUG:boto:auth_path=/pklxkd-59/obj29?versionId=null 2023-05-31T20:53:52.849 DEBUG:boto:Method: GET 2023-05-31T20:53:52.849 DEBUG:boto:Path: /pklxkd-59/obj29?versionId=null 2023-05-31T20:53:52.850 DEBUG:boto:Data: 2023-05-31T20:53:52.850 DEBUG:boto:Headers: {} 2023-05-31T20:53:52.850 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:53:52.851 DEBUG:boto:Port: 8001 2023-05-31T20:53:52.851 DEBUG:boto:Params: {} 2023-05-31T20:53:52.852 DEBUG:boto:Token: None 2023-05-31T20:53:52.852 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:53:52 GMT /pklxkd-59/obj29?versionId=null 2023-05-31T20:53:52.852 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:raMzklBkQbkOsAS2ieyVeGqXGJ4= 2023-05-31T20:53:52.853 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:53:52 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:raMzklBkQbkOsAS2ieyVeGqXGJ4=', 'Content-Length': '0'} 2023-05-31T20:53:52.855 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:48:12 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000eba0e2ae708495c5-006477b3e0-1c17-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:53:52 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:53:52.855 DEBUG:boto:path=/pklxkd-59/obj29 2023-05-31T20:53:52.856 DEBUG:boto:auth_path=/pklxkd-59/obj29 2023-05-31T20:53:52.856 DEBUG:boto:path=/pklxkd-59/obj29?versionId=null 2023-05-31T20:53:52.857 DEBUG:boto:auth_path=/pklxkd-59/obj29?versionId=null 2023-05-31T20:53:52.857 DEBUG:boto:Method: GET 2023-05-31T20:53:52.857 DEBUG:boto:Path: /pklxkd-59/obj29?versionId=null 2023-05-31T20:53:52.858 DEBUG:boto:Data: 2023-05-31T20:53:52.858 DEBUG:boto:Headers: {} 2023-05-31T20:53:52.858 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:53:52.859 DEBUG:boto:Port: 8000 2023-05-31T20:53:52.859 DEBUG:boto:Params: {} 2023-05-31T20:53:52.859 DEBUG:boto:Token: None 2023-05-31T20:53:52.860 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:53:52 GMT /pklxkd-59/obj29?versionId=null 2023-05-31T20:53:52.860 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:raMzklBkQbkOsAS2ieyVeGqXGJ4= 2023-05-31T20:53:52.861 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:53:52 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:raMzklBkQbkOsAS2ieyVeGqXGJ4=', 'Content-Length': '0'} 2023-05-31T20:53:52.863 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:48:12 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000025d48f3734bb34ff-006477b3e0-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:53:52 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:53:52.863 DEBUG:boto:path=/pklxkd-59/ 2023-05-31T20:53:52.864 DEBUG:boto:auth_path=/pklxkd-59/ 2023-05-31T20:53:52.864 DEBUG:boto:path=/pklxkd-59/?prefix=obj29 2023-05-31T20:53:52.864 DEBUG:boto:auth_path=/pklxkd-59/?prefix=obj29 2023-05-31T20:53:52.865 DEBUG:boto:Method: GET 2023-05-31T20:53:52.865 DEBUG:boto:Path: /pklxkd-59/?prefix=obj29 2023-05-31T20:53:52.865 DEBUG:boto:Data: 2023-05-31T20:53:52.866 DEBUG:boto:Headers: {} 2023-05-31T20:53:52.866 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:53:52.866 DEBUG:boto:Port: 8001 2023-05-31T20:53:52.867 DEBUG:boto:Params: {} 2023-05-31T20:53:52.867 DEBUG:boto:Token: None 2023-05-31T20:53:52.868 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:53:52 GMT /pklxkd-59/ 2023-05-31T20:53:52.868 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:Q6NEL+7aW2LGJe6DiOLgBFJA3Bw= 2023-05-31T20:53:52.868 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:53:52 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:Q6NEL+7aW2LGJe6DiOLgBFJA3Bw=', 'Content-Length': '0'} 2023-05-31T20:53:52.871 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f3a26e3f0b61e740-006477b3e0-1c17-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:53:52 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:53:52.871 DEBUG:boto:b'pklxkd-59obj291000falseobj292023-05-31T20:48:12.956Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:53:52.872 DEBUG:boto:path=/pklxkd-59/ 2023-05-31T20:53:52.872 DEBUG:boto:auth_path=/pklxkd-59/ 2023-05-31T20:53:52.872 DEBUG:boto:path=/pklxkd-59/?prefix=obj29 2023-05-31T20:53:52.873 DEBUG:boto:auth_path=/pklxkd-59/?prefix=obj29 2023-05-31T20:53:52.873 DEBUG:boto:Method: GET 2023-05-31T20:53:52.874 DEBUG:boto:Path: /pklxkd-59/?prefix=obj29 2023-05-31T20:53:52.874 DEBUG:boto:Data: 2023-05-31T20:53:52.874 DEBUG:boto:Headers: {} 2023-05-31T20:53:52.875 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:53:52.875 DEBUG:boto:Port: 8000 2023-05-31T20:53:52.875 DEBUG:boto:Params: {} 2023-05-31T20:53:52.876 DEBUG:boto:Token: None 2023-05-31T20:53:52.876 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:53:52 GMT /pklxkd-59/ 2023-05-31T20:53:52.877 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:Q6NEL+7aW2LGJe6DiOLgBFJA3Bw= 2023-05-31T20:53:52.877 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:53:52 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:Q6NEL+7aW2LGJe6DiOLgBFJA3Bw=', 'Content-Length': '0'} 2023-05-31T20:53:52.881 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000af08ed7b2a1d5748-006477b3e0-1630-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:53:52 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:53:52.928 DEBUG:boto:b'pklxkd-59obj291000falseobj292023-05-31T20:48:12.956Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:53:52.929 DEBUG:boto:path=/pklxkd-59/obj29 2023-05-31T20:53:52.930 DEBUG:boto:auth_path=/pklxkd-59/obj29 2023-05-31T20:53:52.930 DEBUG:boto:Method: HEAD 2023-05-31T20:53:52.930 DEBUG:boto:Path: /pklxkd-59/obj29 2023-05-31T20:53:52.931 DEBUG:boto:Data: 2023-05-31T20:53:52.931 DEBUG:boto:Headers: {} 2023-05-31T20:53:52.931 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:53:52.932 DEBUG:boto:Port: 8001 2023-05-31T20:53:52.932 DEBUG:boto:Params: {} 2023-05-31T20:53:52.932 DEBUG:boto:Token: None 2023-05-31T20:53:52.933 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:53:52 GMT /pklxkd-59/obj29 2023-05-31T20:53:52.933 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:qkF4QeqmOlSAiHw2vzIU7eMxHHk= 2023-05-31T20:53:52.933 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:53:52 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:qkF4QeqmOlSAiHw2vzIU7eMxHHk=', 'Content-Length': '0'} 2023-05-31T20:53:52.935 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:48:12 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c804137967282b46-006477b3e0-1c17-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:53:52 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:53:52.936 DEBUG:boto:path=/pklxkd-59/obj29 2023-05-31T20:53:52.936 DEBUG:boto:auth_path=/pklxkd-59/obj29 2023-05-31T20:53:52.936 DEBUG:boto:Method: HEAD 2023-05-31T20:53:52.937 DEBUG:boto:Path: /pklxkd-59/obj29 2023-05-31T20:53:52.937 DEBUG:boto:Data: 2023-05-31T20:53:52.937 DEBUG:boto:Headers: {} 2023-05-31T20:53:52.938 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:53:52.938 DEBUG:boto:Port: 8000 2023-05-31T20:53:52.938 DEBUG:boto:Params: {} 2023-05-31T20:53:52.939 DEBUG:boto:Token: None 2023-05-31T20:53:52.939 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:53:52 GMT /pklxkd-59/obj29 2023-05-31T20:53:52.939 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:qkF4QeqmOlSAiHw2vzIU7eMxHHk= 2023-05-31T20:53:52.940 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:53:52 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:qkF4QeqmOlSAiHw2vzIU7eMxHHk=', 'Content-Length': '0'} 2023-05-31T20:53:52.942 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:48:12 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a164b39c2f0771d4-006477b3e0-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:53:52 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:53:52.942 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj29 2023-05-31T20:53:52.942 DEBUG:boto:path=/pklxkd-59/obj29 2023-05-31T20:53:52.943 DEBUG:boto:auth_path=/pklxkd-59/obj29 2023-05-31T20:53:52.943 DEBUG:boto:Method: GET 2023-05-31T20:53:52.944 DEBUG:boto:Path: /pklxkd-59/obj29 2023-05-31T20:53:52.944 DEBUG:boto:Data: 2023-05-31T20:53:52.944 DEBUG:boto:Headers: {} 2023-05-31T20:53:52.945 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:53:52.945 DEBUG:boto:Port: 8001 2023-05-31T20:53:52.945 DEBUG:boto:Params: {} 2023-05-31T20:53:52.946 DEBUG:boto:Token: None 2023-05-31T20:53:52.946 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:53:52 GMT /pklxkd-59/obj29 2023-05-31T20:53:52.947 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:8sHksI2ACejH8lVcwg736NGDUPA= 2023-05-31T20:53:52.947 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:53:52 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:8sHksI2ACejH8lVcwg736NGDUPA=', 'Content-Length': '0'} 2023-05-31T20:53:52.949 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:48:12 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000da61b20e9e5e95d4-006477b3e0-1c17-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:53:52 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:53:52.950 DEBUG:boto:path=/pklxkd-59/obj29 2023-05-31T20:53:52.950 DEBUG:boto:auth_path=/pklxkd-59/obj29 2023-05-31T20:53:52.950 DEBUG:boto:Method: GET 2023-05-31T20:53:52.951 DEBUG:boto:Path: /pklxkd-59/obj29 2023-05-31T20:53:52.951 DEBUG:boto:Data: 2023-05-31T20:53:52.951 DEBUG:boto:Headers: {} 2023-05-31T20:53:52.952 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:53:52.952 DEBUG:boto:Port: 8000 2023-05-31T20:53:52.952 DEBUG:boto:Params: {} 2023-05-31T20:53:52.953 DEBUG:boto:Token: None 2023-05-31T20:53:52.953 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:53:52 GMT /pklxkd-59/obj29 2023-05-31T20:53:52.954 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:8sHksI2ACejH8lVcwg736NGDUPA= 2023-05-31T20:53:52.954 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:53:52 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:8sHksI2ACejH8lVcwg736NGDUPA=', 'Content-Length': '0'} 2023-05-31T20:53:52.956 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:48:12 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007ed9fb52b69fe95a-006477b3e0-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:53:52 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:53:52.957 DEBUG:boto:path=/pklxkd-59/ 2023-05-31T20:53:52.957 DEBUG:boto:auth_path=/pklxkd-59/ 2023-05-31T20:53:52.957 DEBUG:boto:path=/pklxkd-59/?prefix=obj29 2023-05-31T20:53:52.958 DEBUG:boto:auth_path=/pklxkd-59/?prefix=obj29 2023-05-31T20:53:52.958 DEBUG:boto:Method: GET 2023-05-31T20:53:52.958 DEBUG:boto:Path: /pklxkd-59/?prefix=obj29 2023-05-31T20:53:52.959 DEBUG:boto:Data: 2023-05-31T20:53:52.959 DEBUG:boto:Headers: {} 2023-05-31T20:53:52.959 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:53:52.960 DEBUG:boto:Port: 8001 2023-05-31T20:53:52.960 DEBUG:boto:Params: {} 2023-05-31T20:53:52.961 DEBUG:boto:Token: None 2023-05-31T20:53:52.961 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:53:52 GMT /pklxkd-59/ 2023-05-31T20:53:52.961 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:Q6NEL+7aW2LGJe6DiOLgBFJA3Bw= 2023-05-31T20:53:52.962 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:53:52 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:Q6NEL+7aW2LGJe6DiOLgBFJA3Bw=', 'Content-Length': '0'} 2023-05-31T20:53:52.964 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a220e9f4b53e71aa-006477b3e0-1c17-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:53:52 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:53:53.008 DEBUG:boto:b'pklxkd-59obj291000falseobj292023-05-31T20:48:12.956Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:53:53.009 DEBUG:boto:path=/pklxkd-59/ 2023-05-31T20:53:53.009 DEBUG:boto:auth_path=/pklxkd-59/ 2023-05-31T20:53:53.010 DEBUG:boto:path=/pklxkd-59/?prefix=obj29 2023-05-31T20:53:53.010 DEBUG:boto:auth_path=/pklxkd-59/?prefix=obj29 2023-05-31T20:53:53.010 DEBUG:boto:Method: GET 2023-05-31T20:53:53.011 DEBUG:boto:Path: /pklxkd-59/?prefix=obj29 2023-05-31T20:53:53.011 DEBUG:boto:Data: 2023-05-31T20:53:53.011 DEBUG:boto:Headers: {} 2023-05-31T20:53:53.012 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:53:53.012 DEBUG:boto:Port: 8000 2023-05-31T20:53:53.012 DEBUG:boto:Params: {} 2023-05-31T20:53:53.013 DEBUG:boto:Token: None 2023-05-31T20:53:53.013 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:53:53 GMT /pklxkd-59/ 2023-05-31T20:53:53.014 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:diXwr3WzChuZ6w/0HLW0xEXCAKg= 2023-05-31T20:53:53.014 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:53:53 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:diXwr3WzChuZ6w/0HLW0xEXCAKg=', 'Content-Length': '0'} 2023-05-31T20:53:53.018 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ba265b186fb6aab6-006477b3e1-1630-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:53:53 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:53:53.019 DEBUG:boto:b'pklxkd-59obj291000falseobj292023-05-31T20:48:12.956Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:53:53.019 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj31 2023-05-31T20:53:53.020 DEBUG:boto:path=/pklxkd-59/obj31 2023-05-31T20:53:53.020 DEBUG:boto:auth_path=/pklxkd-59/obj31 2023-05-31T20:53:53.021 DEBUG:boto:path=/pklxkd-59/obj31?versionId=null 2023-05-31T20:53:53.021 DEBUG:boto:auth_path=/pklxkd-59/obj31?versionId=null 2023-05-31T20:53:53.021 DEBUG:boto:Method: GET 2023-05-31T20:53:53.022 DEBUG:boto:Path: /pklxkd-59/obj31?versionId=null 2023-05-31T20:53:53.022 DEBUG:boto:Data: 2023-05-31T20:53:53.022 DEBUG:boto:Headers: {} 2023-05-31T20:53:53.023 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:53:53.023 DEBUG:boto:Port: 8001 2023-05-31T20:53:53.023 DEBUG:boto:Params: {} 2023-05-31T20:53:53.024 DEBUG:boto:Token: None 2023-05-31T20:53:53.024 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:53:53 GMT /pklxkd-59/obj31?versionId=null 2023-05-31T20:53:53.025 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:+AzJSYqzOFuN6wiDZPimy94HIqI= 2023-05-31T20:53:53.025 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:53:53 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:+AzJSYqzOFuN6wiDZPimy94HIqI=', 'Content-Length': '0'} 2023-05-31T20:53:53.027 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:50:33 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000030da8d4575781bff-006477b3e1-1c17-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:53:53 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:53:53.028 DEBUG:boto:path=/pklxkd-59/obj31 2023-05-31T20:53:53.028 DEBUG:boto:auth_path=/pklxkd-59/obj31 2023-05-31T20:53:53.028 DEBUG:boto:path=/pklxkd-59/obj31?versionId=null 2023-05-31T20:53:53.029 DEBUG:boto:auth_path=/pklxkd-59/obj31?versionId=null 2023-05-31T20:53:53.029 DEBUG:boto:Method: GET 2023-05-31T20:53:53.029 DEBUG:boto:Path: /pklxkd-59/obj31?versionId=null 2023-05-31T20:53:53.030 DEBUG:boto:Data: 2023-05-31T20:53:53.030 DEBUG:boto:Headers: {} 2023-05-31T20:53:53.031 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:53:53.031 DEBUG:boto:Port: 8000 2023-05-31T20:53:53.031 DEBUG:boto:Params: {} 2023-05-31T20:53:53.032 DEBUG:boto:Token: None 2023-05-31T20:53:53.032 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:53:53 GMT /pklxkd-59/obj31?versionId=null 2023-05-31T20:53:53.032 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:+AzJSYqzOFuN6wiDZPimy94HIqI= 2023-05-31T20:53:53.033 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:53:53 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:+AzJSYqzOFuN6wiDZPimy94HIqI=', 'Content-Length': '0'} 2023-05-31T20:53:53.035 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:50:33 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000891b59c520da1a1f-006477b3e1-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:53:53 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:53:53.076 DEBUG:boto:path=/pklxkd-59/obj31 2023-05-31T20:53:53.077 DEBUG:boto:auth_path=/pklxkd-59/obj31 2023-05-31T20:53:53.077 DEBUG:boto:path=/pklxkd-59/obj31?versionId=null 2023-05-31T20:53:53.077 DEBUG:boto:auth_path=/pklxkd-59/obj31?versionId=null 2023-05-31T20:53:53.078 DEBUG:boto:Method: HEAD 2023-05-31T20:53:53.078 DEBUG:boto:Path: /pklxkd-59/obj31?versionId=null 2023-05-31T20:53:53.078 DEBUG:boto:Data: 2023-05-31T20:53:53.079 DEBUG:boto:Headers: {} 2023-05-31T20:53:53.079 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:53:53.079 DEBUG:boto:Port: 8001 2023-05-31T20:53:53.080 DEBUG:boto:Params: {} 2023-05-31T20:53:53.080 DEBUG:boto:Token: None 2023-05-31T20:53:53.080 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:53:53 GMT /pklxkd-59/obj31?versionId=null 2023-05-31T20:53:53.081 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:Siuoutb4mQXKdy9X9Wvp+XT15bE= 2023-05-31T20:53:53.081 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:53:53 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:Siuoutb4mQXKdy9X9Wvp+XT15bE=', 'Content-Length': '0'} 2023-05-31T20:53:53.083 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:50:33 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e2f2af01c9b815db-006477b3e1-1c17-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:53:53 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:53:53.083 DEBUG:boto:path=/pklxkd-59/obj31 2023-05-31T20:53:53.084 DEBUG:boto:auth_path=/pklxkd-59/obj31 2023-05-31T20:53:53.084 DEBUG:boto:path=/pklxkd-59/obj31?versionId=null 2023-05-31T20:53:53.084 DEBUG:boto:auth_path=/pklxkd-59/obj31?versionId=null 2023-05-31T20:53:53.085 DEBUG:boto:Method: HEAD 2023-05-31T20:53:53.085 DEBUG:boto:Path: /pklxkd-59/obj31?versionId=null 2023-05-31T20:53:53.085 DEBUG:boto:Data: 2023-05-31T20:53:53.086 DEBUG:boto:Headers: {} 2023-05-31T20:53:53.086 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:53:53.086 DEBUG:boto:Port: 8000 2023-05-31T20:53:53.087 DEBUG:boto:Params: {} 2023-05-31T20:53:53.087 DEBUG:boto:Token: None 2023-05-31T20:53:53.088 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:53:53 GMT /pklxkd-59/obj31?versionId=null 2023-05-31T20:53:53.088 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:Siuoutb4mQXKdy9X9Wvp+XT15bE= 2023-05-31T20:53:53.088 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:53:53 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:Siuoutb4mQXKdy9X9Wvp+XT15bE=', 'Content-Length': '0'} 2023-05-31T20:53:53.090 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:50:33 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e5e682c2e98f5352-006477b3e1-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:53:53 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:53:53.091 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj31 2023-05-31T20:53:53.091 DEBUG:boto:path=/pklxkd-59/obj31 2023-05-31T20:53:53.091 DEBUG:boto:auth_path=/pklxkd-59/obj31 2023-05-31T20:53:53.092 DEBUG:boto:path=/pklxkd-59/obj31?versionId=null 2023-05-31T20:53:53.092 DEBUG:boto:auth_path=/pklxkd-59/obj31?versionId=null 2023-05-31T20:53:53.092 DEBUG:boto:Method: GET 2023-05-31T20:53:53.093 DEBUG:boto:Path: /pklxkd-59/obj31?versionId=null 2023-05-31T20:53:53.093 DEBUG:boto:Data: 2023-05-31T20:53:53.094 DEBUG:boto:Headers: {} 2023-05-31T20:53:53.094 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:53:53.094 DEBUG:boto:Port: 8001 2023-05-31T20:53:53.095 DEBUG:boto:Params: {} 2023-05-31T20:53:53.095 DEBUG:boto:Token: None 2023-05-31T20:53:53.095 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:53:53 GMT /pklxkd-59/obj31?versionId=null 2023-05-31T20:53:53.096 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:+AzJSYqzOFuN6wiDZPimy94HIqI= 2023-05-31T20:53:53.096 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:53:53 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:+AzJSYqzOFuN6wiDZPimy94HIqI=', 'Content-Length': '0'} 2023-05-31T20:53:53.098 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:50:33 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005fb97672933a7f07-006477b3e1-1c17-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:53:53 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:53:53.098 DEBUG:boto:path=/pklxkd-59/obj31 2023-05-31T20:53:53.099 DEBUG:boto:auth_path=/pklxkd-59/obj31 2023-05-31T20:53:53.099 DEBUG:boto:path=/pklxkd-59/obj31?versionId=null 2023-05-31T20:53:53.100 DEBUG:boto:auth_path=/pklxkd-59/obj31?versionId=null 2023-05-31T20:53:53.100 DEBUG:boto:Method: GET 2023-05-31T20:53:53.100 DEBUG:boto:Path: /pklxkd-59/obj31?versionId=null 2023-05-31T20:53:53.101 DEBUG:boto:Data: 2023-05-31T20:53:53.101 DEBUG:boto:Headers: {} 2023-05-31T20:53:53.101 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:53:53.102 DEBUG:boto:Port: 8000 2023-05-31T20:53:53.102 DEBUG:boto:Params: {} 2023-05-31T20:53:53.102 DEBUG:boto:Token: None 2023-05-31T20:53:53.103 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:53:53 GMT /pklxkd-59/obj31?versionId=null 2023-05-31T20:53:53.103 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:+AzJSYqzOFuN6wiDZPimy94HIqI= 2023-05-31T20:53:53.104 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:53:53 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:+AzJSYqzOFuN6wiDZPimy94HIqI=', 'Content-Length': '0'} 2023-05-31T20:53:53.106 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:50:33 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000031428543478b4a-006477b3e1-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:53:53 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:53:53.106 DEBUG:boto:path=/pklxkd-59/ 2023-05-31T20:53:53.107 DEBUG:boto:auth_path=/pklxkd-59/ 2023-05-31T20:53:53.107 DEBUG:boto:path=/pklxkd-59/?prefix=obj31 2023-05-31T20:53:53.107 DEBUG:boto:auth_path=/pklxkd-59/?prefix=obj31 2023-05-31T20:53:53.108 DEBUG:boto:Method: GET 2023-05-31T20:53:53.108 DEBUG:boto:Path: /pklxkd-59/?prefix=obj31 2023-05-31T20:53:53.109 DEBUG:boto:Data: 2023-05-31T20:53:53.109 DEBUG:boto:Headers: {} 2023-05-31T20:53:53.109 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:53:53.110 DEBUG:boto:Port: 8001 2023-05-31T20:53:53.110 DEBUG:boto:Params: {} 2023-05-31T20:53:53.110 DEBUG:boto:Token: None 2023-05-31T20:53:53.111 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:53:53 GMT /pklxkd-59/ 2023-05-31T20:53:53.111 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:diXwr3WzChuZ6w/0HLW0xEXCAKg= 2023-05-31T20:53:53.112 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:53:53 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:diXwr3WzChuZ6w/0HLW0xEXCAKg=', 'Content-Length': '0'} 2023-05-31T20:53:53.114 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000005ee94746cee95779-006477b3e1-1c17-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:53:53 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:53:53.114 DEBUG:boto:b'pklxkd-59obj311000falseobj312023-05-31T20:50:33.460Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:53:53.115 DEBUG:boto:path=/pklxkd-59/ 2023-05-31T20:53:53.115 DEBUG:boto:auth_path=/pklxkd-59/ 2023-05-31T20:53:53.116 DEBUG:boto:path=/pklxkd-59/?prefix=obj31 2023-05-31T20:53:53.116 DEBUG:boto:auth_path=/pklxkd-59/?prefix=obj31 2023-05-31T20:53:53.116 DEBUG:boto:Method: GET 2023-05-31T20:53:53.117 DEBUG:boto:Path: /pklxkd-59/?prefix=obj31 2023-05-31T20:53:53.117 DEBUG:boto:Data: 2023-05-31T20:53:53.117 DEBUG:boto:Headers: {} 2023-05-31T20:53:53.118 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:53:53.118 DEBUG:boto:Port: 8000 2023-05-31T20:53:53.118 DEBUG:boto:Params: {} 2023-05-31T20:53:53.119 DEBUG:boto:Token: None 2023-05-31T20:53:53.119 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:53:53 GMT /pklxkd-59/ 2023-05-31T20:53:53.120 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:diXwr3WzChuZ6w/0HLW0xEXCAKg= 2023-05-31T20:53:53.120 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:53:53 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:diXwr3WzChuZ6w/0HLW0xEXCAKg=', 'Content-Length': '0'} 2023-05-31T20:53:53.124 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e742ab0b76d0132c-006477b3e1-1630-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:53:53 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:53:53.125 DEBUG:boto:b'pklxkd-59obj311000falseobj312023-05-31T20:50:33.460Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:53:53.125 DEBUG:boto:path=/pklxkd-59/obj31 2023-05-31T20:53:53.126 DEBUG:boto:auth_path=/pklxkd-59/obj31 2023-05-31T20:53:53.126 DEBUG:boto:Method: HEAD 2023-05-31T20:53:53.126 DEBUG:boto:Path: /pklxkd-59/obj31 2023-05-31T20:53:53.127 DEBUG:boto:Data: 2023-05-31T20:53:53.127 DEBUG:boto:Headers: {} 2023-05-31T20:53:53.127 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:53:53.128 DEBUG:boto:Port: 8001 2023-05-31T20:53:53.128 DEBUG:boto:Params: {} 2023-05-31T20:53:53.128 DEBUG:boto:Token: None 2023-05-31T20:53:53.129 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:53:53 GMT /pklxkd-59/obj31 2023-05-31T20:53:53.129 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:F3IYtCijg8bMMKOik56iLD9gbl0= 2023-05-31T20:53:53.129 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:53:53 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:F3IYtCijg8bMMKOik56iLD9gbl0=', 'Content-Length': '0'} 2023-05-31T20:53:53.131 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:50:33 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004874bc8d2e3bf8ac-006477b3e1-1c17-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:53:53 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:53:53.132 DEBUG:boto:path=/pklxkd-59/obj31 2023-05-31T20:53:53.132 DEBUG:boto:auth_path=/pklxkd-59/obj31 2023-05-31T20:53:53.132 DEBUG:boto:Method: HEAD 2023-05-31T20:53:53.133 DEBUG:boto:Path: /pklxkd-59/obj31 2023-05-31T20:53:53.133 DEBUG:boto:Data: 2023-05-31T20:53:53.133 DEBUG:boto:Headers: {} 2023-05-31T20:53:53.134 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:53:53.134 DEBUG:boto:Port: 8000 2023-05-31T20:53:53.134 DEBUG:boto:Params: {} 2023-05-31T20:53:53.135 DEBUG:boto:Token: None 2023-05-31T20:53:53.135 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:53:53 GMT /pklxkd-59/obj31 2023-05-31T20:53:53.135 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:F3IYtCijg8bMMKOik56iLD9gbl0= 2023-05-31T20:53:53.136 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:53:53 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:F3IYtCijg8bMMKOik56iLD9gbl0=', 'Content-Length': '0'} 2023-05-31T20:53:53.138 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:50:33 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000011314f3333c85106-006477b3e1-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:53:53 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:53:53.138 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj31 2023-05-31T20:53:53.139 DEBUG:boto:path=/pklxkd-59/obj31 2023-05-31T20:53:53.139 DEBUG:boto:auth_path=/pklxkd-59/obj31 2023-05-31T20:53:53.139 DEBUG:boto:Method: GET 2023-05-31T20:53:53.140 DEBUG:boto:Path: /pklxkd-59/obj31 2023-05-31T20:53:53.140 DEBUG:boto:Data: 2023-05-31T20:53:53.141 DEBUG:boto:Headers: {} 2023-05-31T20:53:53.141 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:53:53.141 DEBUG:boto:Port: 8001 2023-05-31T20:53:53.142 DEBUG:boto:Params: {} 2023-05-31T20:53:53.142 DEBUG:boto:Token: None 2023-05-31T20:53:53.142 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:53:53 GMT /pklxkd-59/obj31 2023-05-31T20:53:53.143 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:NZ9WbamOoute9SFQvWF5PTLos58= 2023-05-31T20:53:53.143 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:53:53 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:NZ9WbamOoute9SFQvWF5PTLos58=', 'Content-Length': '0'} 2023-05-31T20:53:53.145 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:50:33 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000fea7bd397254f677-006477b3e1-1c17-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:53:53 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:53:53.145 DEBUG:boto:path=/pklxkd-59/obj31 2023-05-31T20:53:53.146 DEBUG:boto:auth_path=/pklxkd-59/obj31 2023-05-31T20:53:53.146 DEBUG:boto:Method: GET 2023-05-31T20:53:53.147 DEBUG:boto:Path: /pklxkd-59/obj31 2023-05-31T20:53:53.147 DEBUG:boto:Data: 2023-05-31T20:53:53.147 DEBUG:boto:Headers: {} 2023-05-31T20:53:53.148 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:53:53.148 DEBUG:boto:Port: 8000 2023-05-31T20:53:53.148 DEBUG:boto:Params: {} 2023-05-31T20:53:53.149 DEBUG:boto:Token: None 2023-05-31T20:53:53.149 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:53:53 GMT /pklxkd-59/obj31 2023-05-31T20:53:53.150 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:NZ9WbamOoute9SFQvWF5PTLos58= 2023-05-31T20:53:53.150 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:53:53 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:NZ9WbamOoute9SFQvWF5PTLos58=', 'Content-Length': '0'} 2023-05-31T20:53:53.152 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:50:33 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009e9f0514b993a551-006477b3e1-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:53:53 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:53:53.152 DEBUG:boto:path=/pklxkd-59/ 2023-05-31T20:53:53.153 DEBUG:boto:auth_path=/pklxkd-59/ 2023-05-31T20:53:53.153 DEBUG:boto:path=/pklxkd-59/?prefix=obj31 2023-05-31T20:53:53.153 DEBUG:boto:auth_path=/pklxkd-59/?prefix=obj31 2023-05-31T20:53:53.154 DEBUG:boto:Method: GET 2023-05-31T20:53:53.154 DEBUG:boto:Path: /pklxkd-59/?prefix=obj31 2023-05-31T20:53:53.155 DEBUG:boto:Data: 2023-05-31T20:53:53.155 DEBUG:boto:Headers: {} 2023-05-31T20:53:53.155 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:53:53.156 DEBUG:boto:Port: 8001 2023-05-31T20:53:53.156 DEBUG:boto:Params: {} 2023-05-31T20:53:53.156 DEBUG:boto:Token: None 2023-05-31T20:53:53.157 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:53:53 GMT /pklxkd-59/ 2023-05-31T20:53:53.157 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:diXwr3WzChuZ6w/0HLW0xEXCAKg= 2023-05-31T20:53:53.158 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:53:53 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:diXwr3WzChuZ6w/0HLW0xEXCAKg=', 'Content-Length': '0'} 2023-05-31T20:53:53.160 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000025019d8e0617de4d-006477b3e1-1c17-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:53:53 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:53:53.160 DEBUG:boto:b'pklxkd-59obj311000falseobj312023-05-31T20:50:33.460Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:53:53.161 DEBUG:boto:path=/pklxkd-59/ 2023-05-31T20:53:53.161 DEBUG:boto:auth_path=/pklxkd-59/ 2023-05-31T20:53:53.162 DEBUG:boto:path=/pklxkd-59/?prefix=obj31 2023-05-31T20:53:53.162 DEBUG:boto:auth_path=/pklxkd-59/?prefix=obj31 2023-05-31T20:53:53.162 DEBUG:boto:Method: GET 2023-05-31T20:53:53.163 DEBUG:boto:Path: /pklxkd-59/?prefix=obj31 2023-05-31T20:53:53.163 DEBUG:boto:Data: 2023-05-31T20:53:53.164 DEBUG:boto:Headers: {} 2023-05-31T20:53:53.164 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:53:53.164 DEBUG:boto:Port: 8000 2023-05-31T20:53:53.165 DEBUG:boto:Params: {} 2023-05-31T20:53:53.165 DEBUG:boto:Token: None 2023-05-31T20:53:53.165 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:53:53 GMT /pklxkd-59/ 2023-05-31T20:53:53.166 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:diXwr3WzChuZ6w/0HLW0xEXCAKg= 2023-05-31T20:53:53.166 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:53:53 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:diXwr3WzChuZ6w/0HLW0xEXCAKg=', 'Content-Length': '0'} 2023-05-31T20:53:53.170 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000005f6cae2d328d0a21-006477b3e1-1630-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:53:53 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:53:53.171 DEBUG:boto:b'pklxkd-59obj311000falseobj312023-05-31T20:50:33.460Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:53:53.171 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj33 2023-05-31T20:53:53.172 DEBUG:boto:path=/pklxkd-59/obj33 2023-05-31T20:53:53.172 DEBUG:boto:auth_path=/pklxkd-59/obj33 2023-05-31T20:53:53.172 DEBUG:boto:path=/pklxkd-59/obj33?versionId=null 2023-05-31T20:53:53.173 DEBUG:boto:auth_path=/pklxkd-59/obj33?versionId=null 2023-05-31T20:53:53.173 DEBUG:boto:Method: GET 2023-05-31T20:53:53.173 DEBUG:boto:Path: /pklxkd-59/obj33?versionId=null 2023-05-31T20:53:53.174 DEBUG:boto:Data: 2023-05-31T20:53:53.174 DEBUG:boto:Headers: {} 2023-05-31T20:53:53.174 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:53:53.175 DEBUG:boto:Port: 8001 2023-05-31T20:53:53.175 DEBUG:boto:Params: {} 2023-05-31T20:53:53.176 DEBUG:boto:Token: None 2023-05-31T20:53:53.176 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:53:53 GMT /pklxkd-59/obj33?versionId=null 2023-05-31T20:53:53.176 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:jMYSPqXOjWu3ndS+B2Vexdtsogg= 2023-05-31T20:53:53.177 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:53:53 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:jMYSPqXOjWu3ndS+B2Vexdtsogg=', 'Content-Length': '0'} 2023-05-31T20:53:53.179 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:39:26 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000cfcc9823c301cc0b-006477b3e1-1c17-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:53:53 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:53:53.179 DEBUG:boto:path=/pklxkd-59/obj33 2023-05-31T20:53:53.180 DEBUG:boto:auth_path=/pklxkd-59/obj33 2023-05-31T20:53:53.180 DEBUG:boto:path=/pklxkd-59/obj33?versionId=null 2023-05-31T20:53:53.181 DEBUG:boto:auth_path=/pklxkd-59/obj33?versionId=null 2023-05-31T20:53:53.181 DEBUG:boto:Method: GET 2023-05-31T20:53:53.181 DEBUG:boto:Path: /pklxkd-59/obj33?versionId=null 2023-05-31T20:53:53.182 DEBUG:boto:Data: 2023-05-31T20:53:53.182 DEBUG:boto:Headers: {} 2023-05-31T20:53:53.182 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:53:53.183 DEBUG:boto:Port: 8000 2023-05-31T20:53:53.183 DEBUG:boto:Params: {} 2023-05-31T20:53:53.183 DEBUG:boto:Token: None 2023-05-31T20:53:53.184 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:53:53 GMT /pklxkd-59/obj33?versionId=null 2023-05-31T20:53:53.184 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:jMYSPqXOjWu3ndS+B2Vexdtsogg= 2023-05-31T20:53:53.185 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:53:53 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:jMYSPqXOjWu3ndS+B2Vexdtsogg=', 'Content-Length': '0'} 2023-05-31T20:53:53.187 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:39:26 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e1b7982f31f5c43f-006477b3e1-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:53:53 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:53:53.228 DEBUG:boto:path=/pklxkd-59/obj33 2023-05-31T20:53:53.229 DEBUG:boto:auth_path=/pklxkd-59/obj33 2023-05-31T20:53:53.229 DEBUG:boto:path=/pklxkd-59/obj33?versionId=null 2023-05-31T20:53:53.230 DEBUG:boto:auth_path=/pklxkd-59/obj33?versionId=null 2023-05-31T20:53:53.230 DEBUG:boto:Method: HEAD 2023-05-31T20:53:53.230 DEBUG:boto:Path: /pklxkd-59/obj33?versionId=null 2023-05-31T20:53:53.231 DEBUG:boto:Data: 2023-05-31T20:53:53.231 DEBUG:boto:Headers: {} 2023-05-31T20:53:53.231 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:53:53.232 DEBUG:boto:Port: 8001 2023-05-31T20:53:53.232 DEBUG:boto:Params: {} 2023-05-31T20:53:53.232 DEBUG:boto:Token: None 2023-05-31T20:53:53.233 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:53:53 GMT /pklxkd-59/obj33?versionId=null 2023-05-31T20:53:53.233 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:/ep2Cbii5FHP8+0Z76h/g7/7i8o= 2023-05-31T20:53:53.234 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:53:53 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:/ep2Cbii5FHP8+0Z76h/g7/7i8o=', 'Content-Length': '0'} 2023-05-31T20:53:53.235 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:39:26 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b48d7dd1184b17cd-006477b3e1-1c17-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:53:53 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:53:53.236 DEBUG:boto:path=/pklxkd-59/obj33 2023-05-31T20:53:53.236 DEBUG:boto:auth_path=/pklxkd-59/obj33 2023-05-31T20:53:53.237 DEBUG:boto:path=/pklxkd-59/obj33?versionId=null 2023-05-31T20:53:53.237 DEBUG:boto:auth_path=/pklxkd-59/obj33?versionId=null 2023-05-31T20:53:53.237 DEBUG:boto:Method: HEAD 2023-05-31T20:53:53.238 DEBUG:boto:Path: /pklxkd-59/obj33?versionId=null 2023-05-31T20:53:53.238 DEBUG:boto:Data: 2023-05-31T20:53:53.238 DEBUG:boto:Headers: {} 2023-05-31T20:53:53.239 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:53:53.239 DEBUG:boto:Port: 8000 2023-05-31T20:53:53.239 DEBUG:boto:Params: {} 2023-05-31T20:53:53.240 DEBUG:boto:Token: None 2023-05-31T20:53:53.240 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:53:53 GMT /pklxkd-59/obj33?versionId=null 2023-05-31T20:53:53.241 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:/ep2Cbii5FHP8+0Z76h/g7/7i8o= 2023-05-31T20:53:53.241 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:53:53 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:/ep2Cbii5FHP8+0Z76h/g7/7i8o=', 'Content-Length': '0'} 2023-05-31T20:53:53.243 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:39:26 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000090cd4d8b6a092ce5-006477b3e1-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:53:53 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:53:53.243 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj33 2023-05-31T20:53:53.244 DEBUG:boto:path=/pklxkd-59/obj33 2023-05-31T20:53:53.244 DEBUG:boto:auth_path=/pklxkd-59/obj33 2023-05-31T20:53:53.244 DEBUG:boto:path=/pklxkd-59/obj33?versionId=null 2023-05-31T20:53:53.245 DEBUG:boto:auth_path=/pklxkd-59/obj33?versionId=null 2023-05-31T20:53:53.245 DEBUG:boto:Method: GET 2023-05-31T20:53:53.246 DEBUG:boto:Path: /pklxkd-59/obj33?versionId=null 2023-05-31T20:53:53.246 DEBUG:boto:Data: 2023-05-31T20:53:53.246 DEBUG:boto:Headers: {} 2023-05-31T20:53:53.247 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:53:53.247 DEBUG:boto:Port: 8001 2023-05-31T20:53:53.247 DEBUG:boto:Params: {} 2023-05-31T20:53:53.248 DEBUG:boto:Token: None 2023-05-31T20:53:53.248 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:53:53 GMT /pklxkd-59/obj33?versionId=null 2023-05-31T20:53:53.249 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:jMYSPqXOjWu3ndS+B2Vexdtsogg= 2023-05-31T20:53:53.249 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:53:53 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:jMYSPqXOjWu3ndS+B2Vexdtsogg=', 'Content-Length': '0'} 2023-05-31T20:53:53.251 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:39:26 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000028aa76527bd3185b-006477b3e1-1c17-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:53:53 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:53:53.252 DEBUG:boto:path=/pklxkd-59/obj33 2023-05-31T20:53:53.252 DEBUG:boto:auth_path=/pklxkd-59/obj33 2023-05-31T20:53:53.252 DEBUG:boto:path=/pklxkd-59/obj33?versionId=null 2023-05-31T20:53:53.253 DEBUG:boto:auth_path=/pklxkd-59/obj33?versionId=null 2023-05-31T20:53:53.253 DEBUG:boto:Method: GET 2023-05-31T20:53:53.253 DEBUG:boto:Path: /pklxkd-59/obj33?versionId=null 2023-05-31T20:53:53.254 DEBUG:boto:Data: 2023-05-31T20:53:53.254 DEBUG:boto:Headers: {} 2023-05-31T20:53:53.254 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:53:53.255 DEBUG:boto:Port: 8000 2023-05-31T20:53:53.255 DEBUG:boto:Params: {} 2023-05-31T20:53:53.256 DEBUG:boto:Token: None 2023-05-31T20:53:53.256 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:53:53 GMT /pklxkd-59/obj33?versionId=null 2023-05-31T20:53:53.256 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:jMYSPqXOjWu3ndS+B2Vexdtsogg= 2023-05-31T20:53:53.257 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:53:53 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:jMYSPqXOjWu3ndS+B2Vexdtsogg=', 'Content-Length': '0'} 2023-05-31T20:53:53.259 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:39:26 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004c36d6729797e95e-006477b3e1-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:53:53 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:53:53.259 DEBUG:boto:path=/pklxkd-59/ 2023-05-31T20:53:53.260 DEBUG:boto:auth_path=/pklxkd-59/ 2023-05-31T20:53:53.260 DEBUG:boto:path=/pklxkd-59/?prefix=obj33 2023-05-31T20:53:53.260 DEBUG:boto:auth_path=/pklxkd-59/?prefix=obj33 2023-05-31T20:53:53.261 DEBUG:boto:Method: GET 2023-05-31T20:53:53.261 DEBUG:boto:Path: /pklxkd-59/?prefix=obj33 2023-05-31T20:53:53.262 DEBUG:boto:Data: 2023-05-31T20:53:53.262 DEBUG:boto:Headers: {} 2023-05-31T20:53:53.262 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:53:53.263 DEBUG:boto:Port: 8001 2023-05-31T20:53:53.263 DEBUG:boto:Params: {} 2023-05-31T20:53:53.263 DEBUG:boto:Token: None 2023-05-31T20:53:53.264 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:53:53 GMT /pklxkd-59/ 2023-05-31T20:53:53.264 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:diXwr3WzChuZ6w/0HLW0xEXCAKg= 2023-05-31T20:53:53.265 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:53:53 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:diXwr3WzChuZ6w/0HLW0xEXCAKg=', 'Content-Length': '0'} 2023-05-31T20:53:53.267 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f00a943b4ced140e-006477b3e1-1c17-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:53:53 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:53:53.267 DEBUG:boto:b'pklxkd-59obj331000falseobj332023-05-31T20:39:26.534Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:53:53.268 DEBUG:boto:path=/pklxkd-59/ 2023-05-31T20:53:53.268 DEBUG:boto:auth_path=/pklxkd-59/ 2023-05-31T20:53:53.269 DEBUG:boto:path=/pklxkd-59/?prefix=obj33 2023-05-31T20:53:53.269 DEBUG:boto:auth_path=/pklxkd-59/?prefix=obj33 2023-05-31T20:53:53.269 DEBUG:boto:Method: GET 2023-05-31T20:53:53.270 DEBUG:boto:Path: /pklxkd-59/?prefix=obj33 2023-05-31T20:53:53.270 DEBUG:boto:Data: 2023-05-31T20:53:53.270 DEBUG:boto:Headers: {} 2023-05-31T20:53:53.271 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:53:53.271 DEBUG:boto:Port: 8000 2023-05-31T20:53:53.271 DEBUG:boto:Params: {} 2023-05-31T20:53:53.272 DEBUG:boto:Token: None 2023-05-31T20:53:53.272 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:53:53 GMT /pklxkd-59/ 2023-05-31T20:53:53.273 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:diXwr3WzChuZ6w/0HLW0xEXCAKg= 2023-05-31T20:53:53.273 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:53:53 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:diXwr3WzChuZ6w/0HLW0xEXCAKg=', 'Content-Length': '0'} 2023-05-31T20:53:53.278 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ee546ba0977e047e-006477b3e1-1630-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:53:53 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:53:53.278 DEBUG:boto:b'pklxkd-59obj331000falseobj332023-05-31T20:39:26.534Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:53:53.279 DEBUG:boto:path=/pklxkd-59/obj33 2023-05-31T20:53:53.279 DEBUG:boto:auth_path=/pklxkd-59/obj33 2023-05-31T20:53:53.280 DEBUG:boto:Method: HEAD 2023-05-31T20:53:53.280 DEBUG:boto:Path: /pklxkd-59/obj33 2023-05-31T20:53:53.281 DEBUG:boto:Data: 2023-05-31T20:53:53.281 DEBUG:boto:Headers: {} 2023-05-31T20:53:53.281 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:53:53.282 DEBUG:boto:Port: 8001 2023-05-31T20:53:53.282 DEBUG:boto:Params: {} 2023-05-31T20:53:53.282 DEBUG:boto:Token: None 2023-05-31T20:53:53.283 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:53:53 GMT /pklxkd-59/obj33 2023-05-31T20:53:53.283 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:VRdYyFBM1JMdW8xXEoTdAyFd4sg= 2023-05-31T20:53:53.283 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:53:53 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:VRdYyFBM1JMdW8xXEoTdAyFd4sg=', 'Content-Length': '0'} 2023-05-31T20:53:53.285 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:39:26 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000079404edd74f3e924-006477b3e1-1c17-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:53:53 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:53:53.286 DEBUG:boto:path=/pklxkd-59/obj33 2023-05-31T20:53:53.286 DEBUG:boto:auth_path=/pklxkd-59/obj33 2023-05-31T20:53:53.286 DEBUG:boto:Method: HEAD 2023-05-31T20:53:53.287 DEBUG:boto:Path: /pklxkd-59/obj33 2023-05-31T20:53:53.287 DEBUG:boto:Data: 2023-05-31T20:53:53.287 DEBUG:boto:Headers: {} 2023-05-31T20:53:53.288 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:53:53.288 DEBUG:boto:Port: 8000 2023-05-31T20:53:53.289 DEBUG:boto:Params: {} 2023-05-31T20:53:53.289 DEBUG:boto:Token: None 2023-05-31T20:53:53.289 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:53:53 GMT /pklxkd-59/obj33 2023-05-31T20:53:53.290 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:VRdYyFBM1JMdW8xXEoTdAyFd4sg= 2023-05-31T20:53:53.290 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:53:53 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:VRdYyFBM1JMdW8xXEoTdAyFd4sg=', 'Content-Length': '0'} 2023-05-31T20:53:53.292 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:39:26 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003aa2c45fbd8aa331-006477b3e1-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:53:53 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:53:53.293 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj33 2023-05-31T20:53:53.293 DEBUG:boto:path=/pklxkd-59/obj33 2023-05-31T20:53:53.293 DEBUG:boto:auth_path=/pklxkd-59/obj33 2023-05-31T20:53:53.294 DEBUG:boto:Method: GET 2023-05-31T20:53:53.294 DEBUG:boto:Path: /pklxkd-59/obj33 2023-05-31T20:53:53.294 DEBUG:boto:Data: 2023-05-31T20:53:53.295 DEBUG:boto:Headers: {} 2023-05-31T20:53:53.295 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:53:53.295 DEBUG:boto:Port: 8001 2023-05-31T20:53:53.296 DEBUG:boto:Params: {} 2023-05-31T20:53:53.296 DEBUG:boto:Token: None 2023-05-31T20:53:53.297 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:53:53 GMT /pklxkd-59/obj33 2023-05-31T20:53:53.297 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:U6yFrwNRqiNP5ByE2bR2qIHFFhQ= 2023-05-31T20:53:53.297 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:53:53 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:U6yFrwNRqiNP5ByE2bR2qIHFFhQ=', 'Content-Length': '0'} 2023-05-31T20:53:53.299 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:39:26 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000322ffd1a10642e4a-006477b3e1-1c17-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:53:53 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:53:53.300 DEBUG:boto:path=/pklxkd-59/obj33 2023-05-31T20:53:53.300 DEBUG:boto:auth_path=/pklxkd-59/obj33 2023-05-31T20:53:53.301 DEBUG:boto:Method: GET 2023-05-31T20:53:53.301 DEBUG:boto:Path: /pklxkd-59/obj33 2023-05-31T20:53:53.301 DEBUG:boto:Data: 2023-05-31T20:53:53.302 DEBUG:boto:Headers: {} 2023-05-31T20:53:53.302 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:53:53.302 DEBUG:boto:Port: 8000 2023-05-31T20:53:53.303 DEBUG:boto:Params: {} 2023-05-31T20:53:53.303 DEBUG:boto:Token: None 2023-05-31T20:53:53.304 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:53:53 GMT /pklxkd-59/obj33 2023-05-31T20:53:53.304 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:U6yFrwNRqiNP5ByE2bR2qIHFFhQ= 2023-05-31T20:53:53.304 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:53:53 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:U6yFrwNRqiNP5ByE2bR2qIHFFhQ=', 'Content-Length': '0'} 2023-05-31T20:53:53.306 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:39:26 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000026fd78df7a1ffa65-006477b3e1-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:53:53 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:53:53.307 DEBUG:boto:path=/pklxkd-59/ 2023-05-31T20:53:53.307 DEBUG:boto:auth_path=/pklxkd-59/ 2023-05-31T20:53:53.308 DEBUG:boto:path=/pklxkd-59/?prefix=obj33 2023-05-31T20:53:53.308 DEBUG:boto:auth_path=/pklxkd-59/?prefix=obj33 2023-05-31T20:53:53.309 DEBUG:boto:Method: GET 2023-05-31T20:53:53.309 DEBUG:boto:Path: /pklxkd-59/?prefix=obj33 2023-05-31T20:53:53.309 DEBUG:boto:Data: 2023-05-31T20:53:53.310 DEBUG:boto:Headers: {} 2023-05-31T20:53:53.310 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:53:53.310 DEBUG:boto:Port: 8001 2023-05-31T20:53:53.311 DEBUG:boto:Params: {} 2023-05-31T20:53:53.311 DEBUG:boto:Token: None 2023-05-31T20:53:53.311 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:53:53 GMT /pklxkd-59/ 2023-05-31T20:53:53.312 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:diXwr3WzChuZ6w/0HLW0xEXCAKg= 2023-05-31T20:53:53.312 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:53:53 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:diXwr3WzChuZ6w/0HLW0xEXCAKg=', 'Content-Length': '0'} 2023-05-31T20:53:53.315 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000060461de535927e30-006477b3e1-1c17-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:53:53 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:53:53.315 DEBUG:boto:b'pklxkd-59obj331000falseobj332023-05-31T20:39:26.534Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:53:53.316 DEBUG:boto:path=/pklxkd-59/ 2023-05-31T20:53:53.316 DEBUG:boto:auth_path=/pklxkd-59/ 2023-05-31T20:53:53.317 DEBUG:boto:path=/pklxkd-59/?prefix=obj33 2023-05-31T20:53:53.317 DEBUG:boto:auth_path=/pklxkd-59/?prefix=obj33 2023-05-31T20:53:53.317 DEBUG:boto:Method: GET 2023-05-31T20:53:53.318 DEBUG:boto:Path: /pklxkd-59/?prefix=obj33 2023-05-31T20:53:53.318 DEBUG:boto:Data: 2023-05-31T20:53:53.318 DEBUG:boto:Headers: {} 2023-05-31T20:53:53.319 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:53:53.319 DEBUG:boto:Port: 8000 2023-05-31T20:53:53.319 DEBUG:boto:Params: {} 2023-05-31T20:53:53.320 DEBUG:boto:Token: None 2023-05-31T20:53:53.320 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:53:53 GMT /pklxkd-59/ 2023-05-31T20:53:53.321 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:diXwr3WzChuZ6w/0HLW0xEXCAKg= 2023-05-31T20:53:53.321 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:53:53 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:diXwr3WzChuZ6w/0HLW0xEXCAKg=', 'Content-Length': '0'} 2023-05-31T20:53:53.325 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000001ffc5919608a3054-006477b3e1-1630-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:53:53 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:53:53.326 DEBUG:boto:b'pklxkd-59obj331000falseobj332023-05-31T20:39:26.534Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:53:53.326 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj34 2023-05-31T20:53:53.327 DEBUG:boto:path=/pklxkd-59/obj34 2023-05-31T20:53:53.327 DEBUG:boto:auth_path=/pklxkd-59/obj34 2023-05-31T20:53:53.327 DEBUG:boto:path=/pklxkd-59/obj34?versionId=null 2023-05-31T20:53:53.328 DEBUG:boto:auth_path=/pklxkd-59/obj34?versionId=null 2023-05-31T20:53:53.328 DEBUG:boto:Method: GET 2023-05-31T20:53:53.328 DEBUG:boto:Path: /pklxkd-59/obj34?versionId=null 2023-05-31T20:53:53.329 DEBUG:boto:Data: 2023-05-31T20:53:53.329 DEBUG:boto:Headers: {} 2023-05-31T20:53:53.330 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:53:53.330 DEBUG:boto:Port: 8001 2023-05-31T20:53:53.330 DEBUG:boto:Params: {} 2023-05-31T20:53:53.331 DEBUG:boto:Token: None 2023-05-31T20:53:53.331 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:53:53 GMT /pklxkd-59/obj34?versionId=null 2023-05-31T20:53:53.331 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:CZ7bKxVX/g0kRnIJeAMZ44HRYBI= 2023-05-31T20:53:53.332 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:53:53 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:CZ7bKxVX/g0kRnIJeAMZ44HRYBI=', 'Content-Length': '0'} 2023-05-31T20:53:53.334 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:43:07 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000826486bca203b947-006477b3e1-1c17-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:53:53 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:53:53.334 DEBUG:boto:path=/pklxkd-59/obj34 2023-05-31T20:53:53.335 DEBUG:boto:auth_path=/pklxkd-59/obj34 2023-05-31T20:53:53.335 DEBUG:boto:path=/pklxkd-59/obj34?versionId=null 2023-05-31T20:53:53.335 DEBUG:boto:auth_path=/pklxkd-59/obj34?versionId=null 2023-05-31T20:53:53.336 DEBUG:boto:Method: GET 2023-05-31T20:53:53.336 DEBUG:boto:Path: /pklxkd-59/obj34?versionId=null 2023-05-31T20:53:53.337 DEBUG:boto:Data: 2023-05-31T20:53:53.337 DEBUG:boto:Headers: {} 2023-05-31T20:53:53.337 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:53:53.338 DEBUG:boto:Port: 8000 2023-05-31T20:53:53.338 DEBUG:boto:Params: {} 2023-05-31T20:53:53.338 DEBUG:boto:Token: None 2023-05-31T20:53:53.339 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:53:53 GMT /pklxkd-59/obj34?versionId=null 2023-05-31T20:53:53.339 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:CZ7bKxVX/g0kRnIJeAMZ44HRYBI= 2023-05-31T20:53:53.340 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:53:53 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:CZ7bKxVX/g0kRnIJeAMZ44HRYBI=', 'Content-Length': '0'} 2023-05-31T20:53:53.341 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:43:07 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007a2b35baecfdfe9c-006477b3e1-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:53:53 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:53:53.384 DEBUG:boto:path=/pklxkd-59/obj34 2023-05-31T20:53:53.385 DEBUG:boto:auth_path=/pklxkd-59/obj34 2023-05-31T20:53:53.385 DEBUG:boto:path=/pklxkd-59/obj34?versionId=null 2023-05-31T20:53:53.385 DEBUG:boto:auth_path=/pklxkd-59/obj34?versionId=null 2023-05-31T20:53:53.386 DEBUG:boto:Method: HEAD 2023-05-31T20:53:53.386 DEBUG:boto:Path: /pklxkd-59/obj34?versionId=null 2023-05-31T20:53:53.386 DEBUG:boto:Data: 2023-05-31T20:53:53.387 DEBUG:boto:Headers: {} 2023-05-31T20:53:53.387 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:53:53.387 DEBUG:boto:Port: 8001 2023-05-31T20:53:53.388 DEBUG:boto:Params: {} 2023-05-31T20:53:53.388 DEBUG:boto:Token: None 2023-05-31T20:53:53.388 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:53:53 GMT /pklxkd-59/obj34?versionId=null 2023-05-31T20:53:53.389 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:rW1fEJZcxvClryLu3mbXOzdKckw= 2023-05-31T20:53:53.389 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:53:53 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:rW1fEJZcxvClryLu3mbXOzdKckw=', 'Content-Length': '0'} 2023-05-31T20:53:53.391 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:43:07 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000121ae6d6ffe7135e-006477b3e1-1c17-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:53:53 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:53:53.392 DEBUG:boto:path=/pklxkd-59/obj34 2023-05-31T20:53:53.392 DEBUG:boto:auth_path=/pklxkd-59/obj34 2023-05-31T20:53:53.392 DEBUG:boto:path=/pklxkd-59/obj34?versionId=null 2023-05-31T20:53:53.393 DEBUG:boto:auth_path=/pklxkd-59/obj34?versionId=null 2023-05-31T20:53:53.393 DEBUG:boto:Method: HEAD 2023-05-31T20:53:53.393 DEBUG:boto:Path: /pklxkd-59/obj34?versionId=null 2023-05-31T20:53:53.394 DEBUG:boto:Data: 2023-05-31T20:53:53.394 DEBUG:boto:Headers: {} 2023-05-31T20:53:53.395 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:53:53.395 DEBUG:boto:Port: 8000 2023-05-31T20:53:53.395 DEBUG:boto:Params: {} 2023-05-31T20:53:53.396 DEBUG:boto:Token: None 2023-05-31T20:53:53.396 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:53:53 GMT /pklxkd-59/obj34?versionId=null 2023-05-31T20:53:53.396 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:rW1fEJZcxvClryLu3mbXOzdKckw= 2023-05-31T20:53:53.397 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:53:53 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:rW1fEJZcxvClryLu3mbXOzdKckw=', 'Content-Length': '0'} 2023-05-31T20:53:53.399 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:43:07 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b59c0803d73724a1-006477b3e1-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:53:53 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:53:53.399 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj34 2023-05-31T20:53:53.400 DEBUG:boto:path=/pklxkd-59/obj34 2023-05-31T20:53:53.400 DEBUG:boto:auth_path=/pklxkd-59/obj34 2023-05-31T20:53:53.400 DEBUG:boto:path=/pklxkd-59/obj34?versionId=null 2023-05-31T20:53:53.401 DEBUG:boto:auth_path=/pklxkd-59/obj34?versionId=null 2023-05-31T20:53:53.401 DEBUG:boto:Method: GET 2023-05-31T20:53:53.401 DEBUG:boto:Path: /pklxkd-59/obj34?versionId=null 2023-05-31T20:53:53.402 DEBUG:boto:Data: 2023-05-31T20:53:53.402 DEBUG:boto:Headers: {} 2023-05-31T20:53:53.402 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:53:53.403 DEBUG:boto:Port: 8001 2023-05-31T20:53:53.403 DEBUG:boto:Params: {} 2023-05-31T20:53:53.404 DEBUG:boto:Token: None 2023-05-31T20:53:53.404 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:53:53 GMT /pklxkd-59/obj34?versionId=null 2023-05-31T20:53:53.405 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:CZ7bKxVX/g0kRnIJeAMZ44HRYBI= 2023-05-31T20:53:53.405 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:53:53 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:CZ7bKxVX/g0kRnIJeAMZ44HRYBI=', 'Content-Length': '0'} 2023-05-31T20:53:53.407 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:43:07 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f3c9a29a26819ab6-006477b3e1-1c17-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:53:53 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:53:53.407 DEBUG:boto:path=/pklxkd-59/obj34 2023-05-31T20:53:53.408 DEBUG:boto:auth_path=/pklxkd-59/obj34 2023-05-31T20:53:53.408 DEBUG:boto:path=/pklxkd-59/obj34?versionId=null 2023-05-31T20:53:53.408 DEBUG:boto:auth_path=/pklxkd-59/obj34?versionId=null 2023-05-31T20:53:53.409 DEBUG:boto:Method: GET 2023-05-31T20:53:53.409 DEBUG:boto:Path: /pklxkd-59/obj34?versionId=null 2023-05-31T20:53:53.409 DEBUG:boto:Data: 2023-05-31T20:53:53.410 DEBUG:boto:Headers: {} 2023-05-31T20:53:53.410 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:53:53.410 DEBUG:boto:Port: 8000 2023-05-31T20:53:53.411 DEBUG:boto:Params: {} 2023-05-31T20:53:53.411 DEBUG:boto:Token: None 2023-05-31T20:53:53.412 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:53:53 GMT /pklxkd-59/obj34?versionId=null 2023-05-31T20:53:53.412 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:CZ7bKxVX/g0kRnIJeAMZ44HRYBI= 2023-05-31T20:53:53.412 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:53:53 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:CZ7bKxVX/g0kRnIJeAMZ44HRYBI=', 'Content-Length': '0'} 2023-05-31T20:53:53.414 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:43:07 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b79f4b38178c2fb9-006477b3e1-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:53:53 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:53:53.415 DEBUG:boto:path=/pklxkd-59/ 2023-05-31T20:53:53.415 DEBUG:boto:auth_path=/pklxkd-59/ 2023-05-31T20:53:53.415 DEBUG:boto:path=/pklxkd-59/?prefix=obj34 2023-05-31T20:53:53.416 DEBUG:boto:auth_path=/pklxkd-59/?prefix=obj34 2023-05-31T20:53:53.416 DEBUG:boto:Method: GET 2023-05-31T20:53:53.417 DEBUG:boto:Path: /pklxkd-59/?prefix=obj34 2023-05-31T20:53:53.417 DEBUG:boto:Data: 2023-05-31T20:53:53.417 DEBUG:boto:Headers: {} 2023-05-31T20:53:53.418 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:53:53.418 DEBUG:boto:Port: 8001 2023-05-31T20:53:53.418 DEBUG:boto:Params: {} 2023-05-31T20:53:53.419 DEBUG:boto:Token: None 2023-05-31T20:53:53.419 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:53:53 GMT /pklxkd-59/ 2023-05-31T20:53:53.419 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:diXwr3WzChuZ6w/0HLW0xEXCAKg= 2023-05-31T20:53:53.420 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:53:53 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:diXwr3WzChuZ6w/0HLW0xEXCAKg=', 'Content-Length': '0'} 2023-05-31T20:53:53.422 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000729a9a85bdc4fa88-006477b3e1-1c17-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:53:53 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:53:53.423 DEBUG:boto:b'pklxkd-59obj341000falseobj342023-05-31T20:43:07.994Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:53:53.423 DEBUG:boto:path=/pklxkd-59/ 2023-05-31T20:53:53.424 DEBUG:boto:auth_path=/pklxkd-59/ 2023-05-31T20:53:53.424 DEBUG:boto:path=/pklxkd-59/?prefix=obj34 2023-05-31T20:53:53.425 DEBUG:boto:auth_path=/pklxkd-59/?prefix=obj34 2023-05-31T20:53:53.425 DEBUG:boto:Method: GET 2023-05-31T20:53:53.425 DEBUG:boto:Path: /pklxkd-59/?prefix=obj34 2023-05-31T20:53:53.426 DEBUG:boto:Data: 2023-05-31T20:53:53.426 DEBUG:boto:Headers: {} 2023-05-31T20:53:53.426 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:53:53.427 DEBUG:boto:Port: 8000 2023-05-31T20:53:53.427 DEBUG:boto:Params: {} 2023-05-31T20:53:53.427 DEBUG:boto:Token: None 2023-05-31T20:53:53.428 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:53:53 GMT /pklxkd-59/ 2023-05-31T20:53:53.428 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:diXwr3WzChuZ6w/0HLW0xEXCAKg= 2023-05-31T20:53:53.429 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:53:53 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:diXwr3WzChuZ6w/0HLW0xEXCAKg=', 'Content-Length': '0'} 2023-05-31T20:53:53.432 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000067f31b19732594bd-006477b3e1-1630-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:53:53 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:53:53.433 DEBUG:boto:b'pklxkd-59obj341000falseobj342023-05-31T20:43:07.994Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:53:53.433 DEBUG:boto:path=/pklxkd-59/obj34 2023-05-31T20:53:53.434 DEBUG:boto:auth_path=/pklxkd-59/obj34 2023-05-31T20:53:53.434 DEBUG:boto:Method: HEAD 2023-05-31T20:53:53.434 DEBUG:boto:Path: /pklxkd-59/obj34 2023-05-31T20:53:53.435 DEBUG:boto:Data: 2023-05-31T20:53:53.435 DEBUG:boto:Headers: {} 2023-05-31T20:53:53.435 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:53:53.436 DEBUG:boto:Port: 8001 2023-05-31T20:53:53.436 DEBUG:boto:Params: {} 2023-05-31T20:53:53.437 DEBUG:boto:Token: None 2023-05-31T20:53:53.437 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:53:53 GMT /pklxkd-59/obj34 2023-05-31T20:53:53.437 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:CNS8r7Uv7mKDpWcjYAZcdoyFdac= 2023-05-31T20:53:53.438 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:53:53 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:CNS8r7Uv7mKDpWcjYAZcdoyFdac=', 'Content-Length': '0'} 2023-05-31T20:53:53.439 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:43:07 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004e2eca9f0b3745c0-006477b3e1-1c17-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:53:53 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:53:53.440 DEBUG:boto:path=/pklxkd-59/obj34 2023-05-31T20:53:53.440 DEBUG:boto:auth_path=/pklxkd-59/obj34 2023-05-31T20:53:53.440 DEBUG:boto:Method: HEAD 2023-05-31T20:53:53.441 DEBUG:boto:Path: /pklxkd-59/obj34 2023-05-31T20:53:53.441 DEBUG:boto:Data: 2023-05-31T20:53:53.441 DEBUG:boto:Headers: {} 2023-05-31T20:53:53.442 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:53:53.442 DEBUG:boto:Port: 8000 2023-05-31T20:53:53.442 DEBUG:boto:Params: {} 2023-05-31T20:53:53.443 DEBUG:boto:Token: None 2023-05-31T20:53:53.443 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:53:53 GMT /pklxkd-59/obj34 2023-05-31T20:53:53.444 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:CNS8r7Uv7mKDpWcjYAZcdoyFdac= 2023-05-31T20:53:53.444 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:53:53 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:CNS8r7Uv7mKDpWcjYAZcdoyFdac=', 'Content-Length': '0'} 2023-05-31T20:53:53.446 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:43:07 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a83a1caedd7c4faf-006477b3e1-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:53:53 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:53:53.446 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj34 2023-05-31T20:53:53.447 DEBUG:boto:path=/pklxkd-59/obj34 2023-05-31T20:53:53.447 DEBUG:boto:auth_path=/pklxkd-59/obj34 2023-05-31T20:53:53.447 DEBUG:boto:Method: GET 2023-05-31T20:53:53.448 DEBUG:boto:Path: /pklxkd-59/obj34 2023-05-31T20:53:53.448 DEBUG:boto:Data: 2023-05-31T20:53:53.449 DEBUG:boto:Headers: {} 2023-05-31T20:53:53.449 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:53:53.449 DEBUG:boto:Port: 8001 2023-05-31T20:53:53.450 DEBUG:boto:Params: {} 2023-05-31T20:53:53.450 DEBUG:boto:Token: None 2023-05-31T20:53:53.450 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:53:53 GMT /pklxkd-59/obj34 2023-05-31T20:53:53.451 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:1YV7W9+oLCjtEYwLhajRDghAnQU= 2023-05-31T20:53:53.451 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:53:53 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:1YV7W9+oLCjtEYwLhajRDghAnQU=', 'Content-Length': '0'} 2023-05-31T20:53:53.453 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:43:07 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c0225d0b9f8f1d64-006477b3e1-1c17-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:53:53 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:53:53.453 DEBUG:boto:path=/pklxkd-59/obj34 2023-05-31T20:53:53.454 DEBUG:boto:auth_path=/pklxkd-59/obj34 2023-05-31T20:53:53.454 DEBUG:boto:Method: GET 2023-05-31T20:53:53.455 DEBUG:boto:Path: /pklxkd-59/obj34 2023-05-31T20:53:53.455 DEBUG:boto:Data: 2023-05-31T20:53:53.455 DEBUG:boto:Headers: {} 2023-05-31T20:53:53.456 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:53:53.456 DEBUG:boto:Port: 8000 2023-05-31T20:53:53.456 DEBUG:boto:Params: {} 2023-05-31T20:53:53.457 DEBUG:boto:Token: None 2023-05-31T20:53:53.457 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:53:53 GMT /pklxkd-59/obj34 2023-05-31T20:53:53.458 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:1YV7W9+oLCjtEYwLhajRDghAnQU= 2023-05-31T20:53:53.458 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:53:53 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:1YV7W9+oLCjtEYwLhajRDghAnQU=', 'Content-Length': '0'} 2023-05-31T20:53:53.460 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:43:07 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000448030c3e6036fde-006477b3e1-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:53:53 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:53:53.460 DEBUG:boto:path=/pklxkd-59/ 2023-05-31T20:53:53.461 DEBUG:boto:auth_path=/pklxkd-59/ 2023-05-31T20:53:53.461 DEBUG:boto:path=/pklxkd-59/?prefix=obj34 2023-05-31T20:53:53.461 DEBUG:boto:auth_path=/pklxkd-59/?prefix=obj34 2023-05-31T20:53:53.462 DEBUG:boto:Method: GET 2023-05-31T20:53:53.462 DEBUG:boto:Path: /pklxkd-59/?prefix=obj34 2023-05-31T20:53:53.462 DEBUG:boto:Data: 2023-05-31T20:53:53.463 DEBUG:boto:Headers: {} 2023-05-31T20:53:53.463 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:53:53.463 DEBUG:boto:Port: 8001 2023-05-31T20:53:53.464 DEBUG:boto:Params: {} 2023-05-31T20:53:53.464 DEBUG:boto:Token: None 2023-05-31T20:53:53.465 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:53:53 GMT /pklxkd-59/ 2023-05-31T20:53:53.465 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:diXwr3WzChuZ6w/0HLW0xEXCAKg= 2023-05-31T20:53:53.465 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:53:53 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:diXwr3WzChuZ6w/0HLW0xEXCAKg=', 'Content-Length': '0'} 2023-05-31T20:53:53.468 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000422d34ddc31feb00-006477b3e1-1c17-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:53:53 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:53:53.508 DEBUG:boto:b'pklxkd-59obj341000falseobj342023-05-31T20:43:07.994Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:53:53.509 DEBUG:boto:path=/pklxkd-59/ 2023-05-31T20:53:53.509 DEBUG:boto:auth_path=/pklxkd-59/ 2023-05-31T20:53:53.510 DEBUG:boto:path=/pklxkd-59/?prefix=obj34 2023-05-31T20:53:53.510 DEBUG:boto:auth_path=/pklxkd-59/?prefix=obj34 2023-05-31T20:53:53.510 DEBUG:boto:Method: GET 2023-05-31T20:53:53.511 DEBUG:boto:Path: /pklxkd-59/?prefix=obj34 2023-05-31T20:53:53.511 DEBUG:boto:Data: 2023-05-31T20:53:53.511 DEBUG:boto:Headers: {} 2023-05-31T20:53:53.512 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:53:53.512 DEBUG:boto:Port: 8000 2023-05-31T20:53:53.512 DEBUG:boto:Params: {} 2023-05-31T20:53:53.513 DEBUG:boto:Token: None 2023-05-31T20:53:53.513 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:53:53 GMT /pklxkd-59/ 2023-05-31T20:53:53.514 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:diXwr3WzChuZ6w/0HLW0xEXCAKg= 2023-05-31T20:53:53.514 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:53:53 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:diXwr3WzChuZ6w/0HLW0xEXCAKg=', 'Content-Length': '0'} 2023-05-31T20:53:53.518 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000cfb9d3fd233523b1-006477b3e1-1630-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:53:53 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:53:53.519 DEBUG:boto:b'pklxkd-59obj341000falseobj342023-05-31T20:43:07.994Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:53:53.519 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj37 2023-05-31T20:53:53.520 DEBUG:boto:path=/pklxkd-59/obj37 2023-05-31T20:53:53.520 DEBUG:boto:auth_path=/pklxkd-59/obj37 2023-05-31T20:53:53.521 DEBUG:boto:path=/pklxkd-59/obj37?versionId=null 2023-05-31T20:53:53.521 DEBUG:boto:auth_path=/pklxkd-59/obj37?versionId=null 2023-05-31T20:53:53.521 DEBUG:boto:Method: GET 2023-05-31T20:53:53.522 DEBUG:boto:Path: /pklxkd-59/obj37?versionId=null 2023-05-31T20:53:53.522 DEBUG:boto:Data: 2023-05-31T20:53:53.522 DEBUG:boto:Headers: {} 2023-05-31T20:53:53.523 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:53:53.523 DEBUG:boto:Port: 8001 2023-05-31T20:53:53.523 DEBUG:boto:Params: {} 2023-05-31T20:53:53.524 DEBUG:boto:Token: None 2023-05-31T20:53:53.524 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:53:53 GMT /pklxkd-59/obj37?versionId=null 2023-05-31T20:53:53.525 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:lgXhjjr65gd4N9x7fUha3hKWLfo= 2023-05-31T20:53:53.525 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:53:53 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:lgXhjjr65gd4N9x7fUha3hKWLfo=', 'Content-Length': '0'} 2023-05-31T20:53:53.527 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:52:14 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ea186eb9abbcb060-006477b3e1-1c17-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:53:53 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:53:53.528 DEBUG:boto:path=/pklxkd-59/obj37 2023-05-31T20:53:53.529 DEBUG:boto:auth_path=/pklxkd-59/obj37 2023-05-31T20:53:53.529 DEBUG:boto:path=/pklxkd-59/obj37?versionId=null 2023-05-31T20:53:53.529 DEBUG:boto:auth_path=/pklxkd-59/obj37?versionId=null 2023-05-31T20:53:53.530 DEBUG:boto:Method: GET 2023-05-31T20:53:53.530 DEBUG:boto:Path: /pklxkd-59/obj37?versionId=null 2023-05-31T20:53:53.530 DEBUG:boto:Data: 2023-05-31T20:53:53.531 DEBUG:boto:Headers: {} 2023-05-31T20:53:53.531 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:53:53.531 DEBUG:boto:Port: 8000 2023-05-31T20:53:53.532 DEBUG:boto:Params: {} 2023-05-31T20:53:53.532 DEBUG:boto:Token: None 2023-05-31T20:53:53.533 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:53:53 GMT /pklxkd-59/obj37?versionId=null 2023-05-31T20:53:53.533 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:lgXhjjr65gd4N9x7fUha3hKWLfo= 2023-05-31T20:53:53.533 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:53:53 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:lgXhjjr65gd4N9x7fUha3hKWLfo=', 'Content-Length': '0'} 2023-05-31T20:53:53.535 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:52:14 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ca19c5e01bb8328a-006477b3e1-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:53:53 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:53:53.576 DEBUG:boto:path=/pklxkd-59/obj37 2023-05-31T20:53:53.577 DEBUG:boto:auth_path=/pklxkd-59/obj37 2023-05-31T20:53:53.577 DEBUG:boto:path=/pklxkd-59/obj37?versionId=null 2023-05-31T20:53:53.578 DEBUG:boto:auth_path=/pklxkd-59/obj37?versionId=null 2023-05-31T20:53:53.578 DEBUG:boto:Method: HEAD 2023-05-31T20:53:53.578 DEBUG:boto:Path: /pklxkd-59/obj37?versionId=null 2023-05-31T20:53:53.579 DEBUG:boto:Data: 2023-05-31T20:53:53.579 DEBUG:boto:Headers: {} 2023-05-31T20:53:53.579 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:53:53.580 DEBUG:boto:Port: 8001 2023-05-31T20:53:53.580 DEBUG:boto:Params: {} 2023-05-31T20:53:53.581 DEBUG:boto:Token: None 2023-05-31T20:53:53.581 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:53:53 GMT /pklxkd-59/obj37?versionId=null 2023-05-31T20:53:53.581 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:cFE2zTyLnnKChXGHotwB90hzLRE= 2023-05-31T20:53:53.582 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:53:53 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:cFE2zTyLnnKChXGHotwB90hzLRE=', 'Content-Length': '0'} 2023-05-31T20:53:53.584 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:52:14 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d35b5f975a579194-006477b3e1-1c17-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:53:53 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:53:53.584 DEBUG:boto:path=/pklxkd-59/obj37 2023-05-31T20:53:53.585 DEBUG:boto:auth_path=/pklxkd-59/obj37 2023-05-31T20:53:53.585 DEBUG:boto:path=/pklxkd-59/obj37?versionId=null 2023-05-31T20:53:53.585 DEBUG:boto:auth_path=/pklxkd-59/obj37?versionId=null 2023-05-31T20:53:53.586 DEBUG:boto:Method: HEAD 2023-05-31T20:53:53.586 DEBUG:boto:Path: /pklxkd-59/obj37?versionId=null 2023-05-31T20:53:53.586 DEBUG:boto:Data: 2023-05-31T20:53:53.587 DEBUG:boto:Headers: {} 2023-05-31T20:53:53.587 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:53:53.587 DEBUG:boto:Port: 8000 2023-05-31T20:53:53.588 DEBUG:boto:Params: {} 2023-05-31T20:53:53.588 DEBUG:boto:Token: None 2023-05-31T20:53:53.588 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:53:53 GMT /pklxkd-59/obj37?versionId=null 2023-05-31T20:53:53.589 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:cFE2zTyLnnKChXGHotwB90hzLRE= 2023-05-31T20:53:53.589 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:53:53 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:cFE2zTyLnnKChXGHotwB90hzLRE=', 'Content-Length': '0'} 2023-05-31T20:53:53.591 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:52:14 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000014defd82242f24a2-006477b3e1-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:53:53 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:53:53.592 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj37 2023-05-31T20:53:53.592 DEBUG:boto:path=/pklxkd-59/obj37 2023-05-31T20:53:53.593 DEBUG:boto:auth_path=/pklxkd-59/obj37 2023-05-31T20:53:53.593 DEBUG:boto:path=/pklxkd-59/obj37?versionId=null 2023-05-31T20:53:53.593 DEBUG:boto:auth_path=/pklxkd-59/obj37?versionId=null 2023-05-31T20:53:53.594 DEBUG:boto:Method: GET 2023-05-31T20:53:53.594 DEBUG:boto:Path: /pklxkd-59/obj37?versionId=null 2023-05-31T20:53:53.594 DEBUG:boto:Data: 2023-05-31T20:53:53.595 DEBUG:boto:Headers: {} 2023-05-31T20:53:53.595 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:53:53.595 DEBUG:boto:Port: 8001 2023-05-31T20:53:53.596 DEBUG:boto:Params: {} 2023-05-31T20:53:53.596 DEBUG:boto:Token: None 2023-05-31T20:53:53.597 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:53:53 GMT /pklxkd-59/obj37?versionId=null 2023-05-31T20:53:53.597 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:lgXhjjr65gd4N9x7fUha3hKWLfo= 2023-05-31T20:53:53.597 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:53:53 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:lgXhjjr65gd4N9x7fUha3hKWLfo=', 'Content-Length': '0'} 2023-05-31T20:53:53.599 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:52:14 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000129000148daad81d-006477b3e1-1c17-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:53:53 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:53:53.600 DEBUG:boto:path=/pklxkd-59/obj37 2023-05-31T20:53:53.600 DEBUG:boto:auth_path=/pklxkd-59/obj37 2023-05-31T20:53:53.601 DEBUG:boto:path=/pklxkd-59/obj37?versionId=null 2023-05-31T20:53:53.601 DEBUG:boto:auth_path=/pklxkd-59/obj37?versionId=null 2023-05-31T20:53:53.602 DEBUG:boto:Method: GET 2023-05-31T20:53:53.602 DEBUG:boto:Path: /pklxkd-59/obj37?versionId=null 2023-05-31T20:53:53.602 DEBUG:boto:Data: 2023-05-31T20:53:53.603 DEBUG:boto:Headers: {} 2023-05-31T20:53:53.603 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:53:53.603 DEBUG:boto:Port: 8000 2023-05-31T20:53:53.604 DEBUG:boto:Params: {} 2023-05-31T20:53:53.604 DEBUG:boto:Token: None 2023-05-31T20:53:53.604 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:53:53 GMT /pklxkd-59/obj37?versionId=null 2023-05-31T20:53:53.605 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:lgXhjjr65gd4N9x7fUha3hKWLfo= 2023-05-31T20:53:53.605 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:53:53 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:lgXhjjr65gd4N9x7fUha3hKWLfo=', 'Content-Length': '0'} 2023-05-31T20:53:53.607 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:52:14 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a44b55560a746292-006477b3e1-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:53:53 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:53:53.608 DEBUG:boto:path=/pklxkd-59/ 2023-05-31T20:53:53.608 DEBUG:boto:auth_path=/pklxkd-59/ 2023-05-31T20:53:53.609 DEBUG:boto:path=/pklxkd-59/?prefix=obj37 2023-05-31T20:53:53.609 DEBUG:boto:auth_path=/pklxkd-59/?prefix=obj37 2023-05-31T20:53:53.610 DEBUG:boto:Method: GET 2023-05-31T20:53:53.610 DEBUG:boto:Path: /pklxkd-59/?prefix=obj37 2023-05-31T20:53:53.610 DEBUG:boto:Data: 2023-05-31T20:53:53.611 DEBUG:boto:Headers: {} 2023-05-31T20:53:53.611 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:53:53.611 DEBUG:boto:Port: 8001 2023-05-31T20:53:53.612 DEBUG:boto:Params: {} 2023-05-31T20:53:53.612 DEBUG:boto:Token: None 2023-05-31T20:53:53.613 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:53:53 GMT /pklxkd-59/ 2023-05-31T20:53:53.613 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:diXwr3WzChuZ6w/0HLW0xEXCAKg= 2023-05-31T20:53:53.613 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:53:53 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:diXwr3WzChuZ6w/0HLW0xEXCAKg=', 'Content-Length': '0'} 2023-05-31T20:53:53.616 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ef8702b8536efaa9-006477b3e1-1c17-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:53:53 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:53:53.616 DEBUG:boto:b'pklxkd-59obj371000falseobj372023-05-31T20:52:14.909Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:53:53.617 DEBUG:boto:path=/pklxkd-59/ 2023-05-31T20:53:53.617 DEBUG:boto:auth_path=/pklxkd-59/ 2023-05-31T20:53:53.618 DEBUG:boto:path=/pklxkd-59/?prefix=obj37 2023-05-31T20:53:53.618 DEBUG:boto:auth_path=/pklxkd-59/?prefix=obj37 2023-05-31T20:53:53.618 DEBUG:boto:Method: GET 2023-05-31T20:53:53.619 DEBUG:boto:Path: /pklxkd-59/?prefix=obj37 2023-05-31T20:53:53.619 DEBUG:boto:Data: 2023-05-31T20:53:53.620 DEBUG:boto:Headers: {} 2023-05-31T20:53:53.620 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:53:53.620 DEBUG:boto:Port: 8000 2023-05-31T20:53:53.621 DEBUG:boto:Params: {} 2023-05-31T20:53:53.621 DEBUG:boto:Token: None 2023-05-31T20:53:53.621 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:53:53 GMT /pklxkd-59/ 2023-05-31T20:53:53.622 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:diXwr3WzChuZ6w/0HLW0xEXCAKg= 2023-05-31T20:53:53.622 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:53:53 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:diXwr3WzChuZ6w/0HLW0xEXCAKg=', 'Content-Length': '0'} 2023-05-31T20:53:53.626 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ca7b7f6d39eafcb0-006477b3e1-1630-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:53:53 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:53:53.627 DEBUG:boto:b'pklxkd-59obj371000falseobj372023-05-31T20:52:14.909Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:53:53.628 DEBUG:boto:path=/pklxkd-59/obj37 2023-05-31T20:53:53.628 DEBUG:boto:auth_path=/pklxkd-59/obj37 2023-05-31T20:53:53.628 DEBUG:boto:Method: HEAD 2023-05-31T20:53:53.629 DEBUG:boto:Path: /pklxkd-59/obj37 2023-05-31T20:53:53.629 DEBUG:boto:Data: 2023-05-31T20:53:53.629 DEBUG:boto:Headers: {} 2023-05-31T20:53:53.630 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:53:53.630 DEBUG:boto:Port: 8001 2023-05-31T20:53:53.630 DEBUG:boto:Params: {} 2023-05-31T20:53:53.631 DEBUG:boto:Token: None 2023-05-31T20:53:53.631 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:53:53 GMT /pklxkd-59/obj37 2023-05-31T20:53:53.632 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:ATOwm/8YP7gZWv4YzoThDRGjpIg= 2023-05-31T20:53:53.632 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:53:53 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:ATOwm/8YP7gZWv4YzoThDRGjpIg=', 'Content-Length': '0'} 2023-05-31T20:53:53.634 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:52:14 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000031987c975dda193-006477b3e1-1c17-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:53:53 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:53:53.634 DEBUG:boto:path=/pklxkd-59/obj37 2023-05-31T20:53:53.635 DEBUG:boto:auth_path=/pklxkd-59/obj37 2023-05-31T20:53:53.635 DEBUG:boto:Method: HEAD 2023-05-31T20:53:53.635 DEBUG:boto:Path: /pklxkd-59/obj37 2023-05-31T20:53:53.636 DEBUG:boto:Data: 2023-05-31T20:53:53.636 DEBUG:boto:Headers: {} 2023-05-31T20:53:53.636 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:53:53.637 DEBUG:boto:Port: 8000 2023-05-31T20:53:53.637 DEBUG:boto:Params: {} 2023-05-31T20:53:53.637 DEBUG:boto:Token: None 2023-05-31T20:53:53.638 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:53:53 GMT /pklxkd-59/obj37 2023-05-31T20:53:53.638 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:ATOwm/8YP7gZWv4YzoThDRGjpIg= 2023-05-31T20:53:53.639 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:53:53 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:ATOwm/8YP7gZWv4YzoThDRGjpIg=', 'Content-Length': '0'} 2023-05-31T20:53:53.640 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:52:14 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004034ff742e0b7bfc-006477b3e1-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:53:53 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:53:53.641 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj37 2023-05-31T20:53:53.641 DEBUG:boto:path=/pklxkd-59/obj37 2023-05-31T20:53:53.641 DEBUG:boto:auth_path=/pklxkd-59/obj37 2023-05-31T20:53:53.642 DEBUG:boto:Method: GET 2023-05-31T20:53:53.642 DEBUG:boto:Path: /pklxkd-59/obj37 2023-05-31T20:53:53.643 DEBUG:boto:Data: 2023-05-31T20:53:53.643 DEBUG:boto:Headers: {} 2023-05-31T20:53:53.643 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:53:53.644 DEBUG:boto:Port: 8001 2023-05-31T20:53:53.644 DEBUG:boto:Params: {} 2023-05-31T20:53:53.645 DEBUG:boto:Token: None 2023-05-31T20:53:53.645 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:53:53 GMT /pklxkd-59/obj37 2023-05-31T20:53:53.646 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:JlmOmaJxpKUXHcW3/xRzUUO7Cws= 2023-05-31T20:53:53.646 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:53:53 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:JlmOmaJxpKUXHcW3/xRzUUO7Cws=', 'Content-Length': '0'} 2023-05-31T20:53:53.648 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:52:14 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f85a4693fc39f3a9-006477b3e1-1c17-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:53:53 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:53:53.649 DEBUG:boto:path=/pklxkd-59/obj37 2023-05-31T20:53:53.649 DEBUG:boto:auth_path=/pklxkd-59/obj37 2023-05-31T20:53:53.649 DEBUG:boto:Method: GET 2023-05-31T20:53:53.650 DEBUG:boto:Path: /pklxkd-59/obj37 2023-05-31T20:53:53.650 DEBUG:boto:Data: 2023-05-31T20:53:53.651 DEBUG:boto:Headers: {} 2023-05-31T20:53:53.651 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:53:53.651 DEBUG:boto:Port: 8000 2023-05-31T20:53:53.652 DEBUG:boto:Params: {} 2023-05-31T20:53:53.652 DEBUG:boto:Token: None 2023-05-31T20:53:53.653 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:53:53 GMT /pklxkd-59/obj37 2023-05-31T20:53:53.653 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:JlmOmaJxpKUXHcW3/xRzUUO7Cws= 2023-05-31T20:53:53.654 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:53:53 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:JlmOmaJxpKUXHcW3/xRzUUO7Cws=', 'Content-Length': '0'} 2023-05-31T20:53:53.656 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:52:14 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b6313732597b942b-006477b3e1-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:53:53 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:53:53.696 DEBUG:boto:path=/pklxkd-59/ 2023-05-31T20:53:53.697 DEBUG:boto:auth_path=/pklxkd-59/ 2023-05-31T20:53:53.697 DEBUG:boto:path=/pklxkd-59/?prefix=obj37 2023-05-31T20:53:53.698 DEBUG:boto:auth_path=/pklxkd-59/?prefix=obj37 2023-05-31T20:53:53.698 DEBUG:boto:Method: GET 2023-05-31T20:53:53.698 DEBUG:boto:Path: /pklxkd-59/?prefix=obj37 2023-05-31T20:53:53.699 DEBUG:boto:Data: 2023-05-31T20:53:53.699 DEBUG:boto:Headers: {} 2023-05-31T20:53:53.700 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:53:53.700 DEBUG:boto:Port: 8001 2023-05-31T20:53:53.700 DEBUG:boto:Params: {} 2023-05-31T20:53:53.701 DEBUG:boto:Token: None 2023-05-31T20:53:53.701 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:53:53 GMT /pklxkd-59/ 2023-05-31T20:53:53.702 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:diXwr3WzChuZ6w/0HLW0xEXCAKg= 2023-05-31T20:53:53.702 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:53:53 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:diXwr3WzChuZ6w/0HLW0xEXCAKg=', 'Content-Length': '0'} 2023-05-31T20:53:53.705 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000001a047c4ad0e6ab04-006477b3e1-1c17-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:53:53 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:53:53.705 DEBUG:boto:b'pklxkd-59obj371000falseobj372023-05-31T20:52:14.909Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:53:53.706 DEBUG:boto:path=/pklxkd-59/ 2023-05-31T20:53:53.707 DEBUG:boto:auth_path=/pklxkd-59/ 2023-05-31T20:53:53.707 DEBUG:boto:path=/pklxkd-59/?prefix=obj37 2023-05-31T20:53:53.707 DEBUG:boto:auth_path=/pklxkd-59/?prefix=obj37 2023-05-31T20:53:53.708 DEBUG:boto:Method: GET 2023-05-31T20:53:53.708 DEBUG:boto:Path: /pklxkd-59/?prefix=obj37 2023-05-31T20:53:53.709 DEBUG:boto:Data: 2023-05-31T20:53:53.709 DEBUG:boto:Headers: {} 2023-05-31T20:53:53.709 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:53:53.710 DEBUG:boto:Port: 8000 2023-05-31T20:53:53.710 DEBUG:boto:Params: {} 2023-05-31T20:53:53.711 DEBUG:boto:Token: None 2023-05-31T20:53:53.711 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:53:53 GMT /pklxkd-59/ 2023-05-31T20:53:53.711 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:diXwr3WzChuZ6w/0HLW0xEXCAKg= 2023-05-31T20:53:53.712 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:53:53 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:diXwr3WzChuZ6w/0HLW0xEXCAKg=', 'Content-Length': '0'} 2023-05-31T20:53:53.716 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000005f6e94622be6b475-006477b3e1-1630-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:53:53 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:53:53.717 DEBUG:boto:b'pklxkd-59obj371000falseobj372023-05-31T20:52:14.909Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:53:53.717 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj38 2023-05-31T20:53:53.718 DEBUG:boto:path=/pklxkd-59/obj38 2023-05-31T20:53:53.718 DEBUG:boto:auth_path=/pklxkd-59/obj38 2023-05-31T20:53:53.719 DEBUG:boto:path=/pklxkd-59/obj38?versionId=null 2023-05-31T20:53:53.720 DEBUG:boto:auth_path=/pklxkd-59/obj38?versionId=null 2023-05-31T20:53:53.721 DEBUG:boto:Method: GET 2023-05-31T20:53:53.721 DEBUG:boto:Path: /pklxkd-59/obj38?versionId=null 2023-05-31T20:53:53.722 DEBUG:boto:Data: 2023-05-31T20:53:53.722 DEBUG:boto:Headers: {} 2023-05-31T20:53:53.722 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:53:53.723 DEBUG:boto:Port: 8001 2023-05-31T20:53:53.723 DEBUG:boto:Params: {} 2023-05-31T20:53:53.724 DEBUG:boto:Token: None 2023-05-31T20:53:53.724 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:53:53 GMT /pklxkd-59/obj38?versionId=null 2023-05-31T20:53:53.724 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:bmcyzSTdMG3Fhs0LpOehYBKhDF0= 2023-05-31T20:53:53.725 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:53:53 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:bmcyzSTdMG3Fhs0LpOehYBKhDF0=', 'Content-Length': '0'} 2023-05-31T20:53:53.727 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:44:50 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008083a686844745be-006477b3e1-1c17-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:53:53 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:53:53.728 DEBUG:boto:path=/pklxkd-59/obj38 2023-05-31T20:53:53.728 DEBUG:boto:auth_path=/pklxkd-59/obj38 2023-05-31T20:53:53.729 DEBUG:boto:path=/pklxkd-59/obj38?versionId=null 2023-05-31T20:53:53.730 DEBUG:boto:auth_path=/pklxkd-59/obj38?versionId=null 2023-05-31T20:53:53.731 DEBUG:boto:Method: GET 2023-05-31T20:53:53.731 DEBUG:boto:Path: /pklxkd-59/obj38?versionId=null 2023-05-31T20:53:53.731 DEBUG:boto:Data: 2023-05-31T20:53:53.732 DEBUG:boto:Headers: {} 2023-05-31T20:53:53.732 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:53:53.733 DEBUG:boto:Port: 8000 2023-05-31T20:53:53.733 DEBUG:boto:Params: {} 2023-05-31T20:53:53.733 DEBUG:boto:Token: None 2023-05-31T20:53:53.734 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:53:53 GMT /pklxkd-59/obj38?versionId=null 2023-05-31T20:53:53.734 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:bmcyzSTdMG3Fhs0LpOehYBKhDF0= 2023-05-31T20:53:53.735 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:53:53 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:bmcyzSTdMG3Fhs0LpOehYBKhDF0=', 'Content-Length': '0'} 2023-05-31T20:53:53.737 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:44:50 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000309b951c2a47233f-006477b3e1-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:53:53 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:53:53.737 DEBUG:boto:path=/pklxkd-59/obj38 2023-05-31T20:53:53.738 DEBUG:boto:auth_path=/pklxkd-59/obj38 2023-05-31T20:53:53.738 DEBUG:boto:path=/pklxkd-59/obj38?versionId=null 2023-05-31T20:53:53.739 DEBUG:boto:auth_path=/pklxkd-59/obj38?versionId=null 2023-05-31T20:53:53.739 DEBUG:boto:Method: HEAD 2023-05-31T20:53:53.739 DEBUG:boto:Path: /pklxkd-59/obj38?versionId=null 2023-05-31T20:53:53.740 DEBUG:boto:Data: 2023-05-31T20:53:53.740 DEBUG:boto:Headers: {} 2023-05-31T20:53:53.741 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:53:53.741 DEBUG:boto:Port: 8001 2023-05-31T20:53:53.741 DEBUG:boto:Params: {} 2023-05-31T20:53:53.742 DEBUG:boto:Token: None 2023-05-31T20:53:53.742 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:53:53 GMT /pklxkd-59/obj38?versionId=null 2023-05-31T20:53:53.743 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:HNuncT6ndzvo7EQAjGdJU1UImaA= 2023-05-31T20:53:53.743 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:53:53 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:HNuncT6ndzvo7EQAjGdJU1UImaA=', 'Content-Length': '0'} 2023-05-31T20:53:53.745 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:44:50 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ad2121dfffec2986-006477b3e1-1c17-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:53:53 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:53:53.745 DEBUG:boto:path=/pklxkd-59/obj38 2023-05-31T20:53:53.746 DEBUG:boto:auth_path=/pklxkd-59/obj38 2023-05-31T20:53:53.746 DEBUG:boto:path=/pklxkd-59/obj38?versionId=null 2023-05-31T20:53:53.746 DEBUG:boto:auth_path=/pklxkd-59/obj38?versionId=null 2023-05-31T20:53:53.747 DEBUG:boto:Method: HEAD 2023-05-31T20:53:53.747 DEBUG:boto:Path: /pklxkd-59/obj38?versionId=null 2023-05-31T20:53:53.747 DEBUG:boto:Data: 2023-05-31T20:53:53.748 DEBUG:boto:Headers: {} 2023-05-31T20:53:53.748 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:53:53.748 DEBUG:boto:Port: 8000 2023-05-31T20:53:53.749 DEBUG:boto:Params: {} 2023-05-31T20:53:53.749 DEBUG:boto:Token: None 2023-05-31T20:53:53.750 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:53:53 GMT /pklxkd-59/obj38?versionId=null 2023-05-31T20:53:53.750 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:HNuncT6ndzvo7EQAjGdJU1UImaA= 2023-05-31T20:53:53.750 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:53:53 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:HNuncT6ndzvo7EQAjGdJU1UImaA=', 'Content-Length': '0'} 2023-05-31T20:53:53.752 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:44:50 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000032b6d984ba2aa6c3-006477b3e1-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:53:53 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:53:53.753 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj38 2023-05-31T20:53:53.753 DEBUG:boto:path=/pklxkd-59/obj38 2023-05-31T20:53:53.754 DEBUG:boto:auth_path=/pklxkd-59/obj38 2023-05-31T20:53:53.754 DEBUG:boto:path=/pklxkd-59/obj38?versionId=null 2023-05-31T20:53:53.754 DEBUG:boto:auth_path=/pklxkd-59/obj38?versionId=null 2023-05-31T20:53:53.755 DEBUG:boto:Method: GET 2023-05-31T20:53:53.755 DEBUG:boto:Path: /pklxkd-59/obj38?versionId=null 2023-05-31T20:53:53.755 DEBUG:boto:Data: 2023-05-31T20:53:53.756 DEBUG:boto:Headers: {} 2023-05-31T20:53:53.756 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:53:53.756 DEBUG:boto:Port: 8001 2023-05-31T20:53:53.757 DEBUG:boto:Params: {} 2023-05-31T20:53:53.757 DEBUG:boto:Token: None 2023-05-31T20:53:53.758 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:53:53 GMT /pklxkd-59/obj38?versionId=null 2023-05-31T20:53:53.758 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:bmcyzSTdMG3Fhs0LpOehYBKhDF0= 2023-05-31T20:53:53.758 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:53:53 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:bmcyzSTdMG3Fhs0LpOehYBKhDF0=', 'Content-Length': '0'} 2023-05-31T20:53:53.760 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:44:50 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000083da19c6b8416d7-006477b3e1-1c17-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:53:53 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:53:53.804 DEBUG:boto:path=/pklxkd-59/obj38 2023-05-31T20:53:53.805 DEBUG:boto:auth_path=/pklxkd-59/obj38 2023-05-31T20:53:53.806 DEBUG:boto:path=/pklxkd-59/obj38?versionId=null 2023-05-31T20:53:53.806 DEBUG:boto:auth_path=/pklxkd-59/obj38?versionId=null 2023-05-31T20:53:53.807 DEBUG:boto:Method: GET 2023-05-31T20:53:53.807 DEBUG:boto:Path: /pklxkd-59/obj38?versionId=null 2023-05-31T20:53:53.808 DEBUG:boto:Data: 2023-05-31T20:53:53.808 DEBUG:boto:Headers: {} 2023-05-31T20:53:53.809 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:53:53.809 DEBUG:boto:Port: 8000 2023-05-31T20:53:53.810 DEBUG:boto:Params: {} 2023-05-31T20:53:53.810 DEBUG:boto:Token: None 2023-05-31T20:53:53.811 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:53:53 GMT /pklxkd-59/obj38?versionId=null 2023-05-31T20:53:53.812 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:bmcyzSTdMG3Fhs0LpOehYBKhDF0= 2023-05-31T20:53:53.812 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:53:53 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:bmcyzSTdMG3Fhs0LpOehYBKhDF0=', 'Content-Length': '0'} 2023-05-31T20:53:53.815 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:44:50 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e807604258675936-006477b3e1-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:53:53 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:53:53.816 DEBUG:boto:path=/pklxkd-59/ 2023-05-31T20:53:53.816 DEBUG:boto:auth_path=/pklxkd-59/ 2023-05-31T20:53:53.817 DEBUG:boto:path=/pklxkd-59/?prefix=obj38 2023-05-31T20:53:53.817 DEBUG:boto:auth_path=/pklxkd-59/?prefix=obj38 2023-05-31T20:53:53.818 DEBUG:boto:Method: GET 2023-05-31T20:53:53.818 DEBUG:boto:Path: /pklxkd-59/?prefix=obj38 2023-05-31T20:53:53.819 DEBUG:boto:Data: 2023-05-31T20:53:53.820 DEBUG:boto:Headers: {} 2023-05-31T20:53:53.820 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:53:53.821 DEBUG:boto:Port: 8001 2023-05-31T20:53:53.821 DEBUG:boto:Params: {} 2023-05-31T20:53:53.822 DEBUG:boto:Token: None 2023-05-31T20:53:53.822 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:53:53 GMT /pklxkd-59/ 2023-05-31T20:53:53.823 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:diXwr3WzChuZ6w/0HLW0xEXCAKg= 2023-05-31T20:53:53.823 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:53:53 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:diXwr3WzChuZ6w/0HLW0xEXCAKg=', 'Content-Length': '0'} 2023-05-31T20:53:53.826 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000016afad4974750caa-006477b3e1-1c17-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:53:53 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:53:53.827 DEBUG:boto:b'pklxkd-59obj381000falseobj382023-05-31T20:44:50.610Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:53:53.828 DEBUG:boto:path=/pklxkd-59/ 2023-05-31T20:53:53.829 DEBUG:boto:auth_path=/pklxkd-59/ 2023-05-31T20:53:53.829 DEBUG:boto:path=/pklxkd-59/?prefix=obj38 2023-05-31T20:53:53.830 DEBUG:boto:auth_path=/pklxkd-59/?prefix=obj38 2023-05-31T20:53:53.830 DEBUG:boto:Method: GET 2023-05-31T20:53:53.831 DEBUG:boto:Path: /pklxkd-59/?prefix=obj38 2023-05-31T20:53:53.831 DEBUG:boto:Data: 2023-05-31T20:53:53.832 DEBUG:boto:Headers: {} 2023-05-31T20:53:53.832 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:53:53.833 DEBUG:boto:Port: 8000 2023-05-31T20:53:53.833 DEBUG:boto:Params: {} 2023-05-31T20:53:53.834 DEBUG:boto:Token: None 2023-05-31T20:53:53.834 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:53:53 GMT /pklxkd-59/ 2023-05-31T20:53:53.835 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:diXwr3WzChuZ6w/0HLW0xEXCAKg= 2023-05-31T20:53:53.836 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:53:53 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:diXwr3WzChuZ6w/0HLW0xEXCAKg=', 'Content-Length': '0'} 2023-05-31T20:53:53.840 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d8645512e48cbef1-006477b3e1-1630-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:53:53 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:53:53.841 DEBUG:boto:b'pklxkd-59obj381000falseobj382023-05-31T20:44:50.610Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:53:53.841 DEBUG:boto:path=/pklxkd-59/obj38 2023-05-31T20:53:53.842 DEBUG:boto:auth_path=/pklxkd-59/obj38 2023-05-31T20:53:53.842 DEBUG:boto:Method: HEAD 2023-05-31T20:53:53.842 DEBUG:boto:Path: /pklxkd-59/obj38 2023-05-31T20:53:53.843 DEBUG:boto:Data: 2023-05-31T20:53:53.843 DEBUG:boto:Headers: {} 2023-05-31T20:53:53.843 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:53:53.844 DEBUG:boto:Port: 8001 2023-05-31T20:53:53.844 DEBUG:boto:Params: {} 2023-05-31T20:53:53.844 DEBUG:boto:Token: None 2023-05-31T20:53:53.845 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:53:53 GMT /pklxkd-59/obj38 2023-05-31T20:53:53.845 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:REjR6btyjfrvuoxSvIMiY+KaAdw= 2023-05-31T20:53:53.845 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:53:53 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:REjR6btyjfrvuoxSvIMiY+KaAdw=', 'Content-Length': '0'} 2023-05-31T20:53:53.847 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:44:50 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000087976429aa06871e-006477b3e1-1c17-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:53:53 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:53:53.848 DEBUG:boto:path=/pklxkd-59/obj38 2023-05-31T20:53:53.848 DEBUG:boto:auth_path=/pklxkd-59/obj38 2023-05-31T20:53:53.849 DEBUG:boto:Method: HEAD 2023-05-31T20:53:53.849 DEBUG:boto:Path: /pklxkd-59/obj38 2023-05-31T20:53:53.849 DEBUG:boto:Data: 2023-05-31T20:53:53.850 DEBUG:boto:Headers: {} 2023-05-31T20:53:53.850 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:53:53.850 DEBUG:boto:Port: 8000 2023-05-31T20:53:53.851 DEBUG:boto:Params: {} 2023-05-31T20:53:53.851 DEBUG:boto:Token: None 2023-05-31T20:53:53.851 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:53:53 GMT /pklxkd-59/obj38 2023-05-31T20:53:53.852 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:REjR6btyjfrvuoxSvIMiY+KaAdw= 2023-05-31T20:53:53.852 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:53:53 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:REjR6btyjfrvuoxSvIMiY+KaAdw=', 'Content-Length': '0'} 2023-05-31T20:53:53.854 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:44:50 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a4353841b32e7d0e-006477b3e1-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:53:53 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:53:53.855 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj38 2023-05-31T20:53:53.855 DEBUG:boto:path=/pklxkd-59/obj38 2023-05-31T20:53:53.856 DEBUG:boto:auth_path=/pklxkd-59/obj38 2023-05-31T20:53:53.856 DEBUG:boto:Method: GET 2023-05-31T20:53:53.856 DEBUG:boto:Path: /pklxkd-59/obj38 2023-05-31T20:53:53.857 DEBUG:boto:Data: 2023-05-31T20:53:53.857 DEBUG:boto:Headers: {} 2023-05-31T20:53:53.857 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:53:53.858 DEBUG:boto:Port: 8001 2023-05-31T20:53:53.858 DEBUG:boto:Params: {} 2023-05-31T20:53:53.859 DEBUG:boto:Token: None 2023-05-31T20:53:53.859 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:53:53 GMT /pklxkd-59/obj38 2023-05-31T20:53:53.860 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:wD7yezhiQsjyw6UgcclG9aKljrU= 2023-05-31T20:53:53.860 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:53:53 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:wD7yezhiQsjyw6UgcclG9aKljrU=', 'Content-Length': '0'} 2023-05-31T20:53:53.862 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:44:50 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000037e2ce25b70dc59b-006477b3e1-1c17-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:53:53 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:53:53.862 DEBUG:boto:path=/pklxkd-59/obj38 2023-05-31T20:53:53.863 DEBUG:boto:auth_path=/pklxkd-59/obj38 2023-05-31T20:53:53.863 DEBUG:boto:Method: GET 2023-05-31T20:53:53.863 DEBUG:boto:Path: /pklxkd-59/obj38 2023-05-31T20:53:53.864 DEBUG:boto:Data: 2023-05-31T20:53:53.864 DEBUG:boto:Headers: {} 2023-05-31T20:53:53.864 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:53:53.865 DEBUG:boto:Port: 8000 2023-05-31T20:53:53.865 DEBUG:boto:Params: {} 2023-05-31T20:53:53.866 DEBUG:boto:Token: None 2023-05-31T20:53:53.866 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:53:53 GMT /pklxkd-59/obj38 2023-05-31T20:53:53.866 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:wD7yezhiQsjyw6UgcclG9aKljrU= 2023-05-31T20:53:53.867 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:53:53 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:wD7yezhiQsjyw6UgcclG9aKljrU=', 'Content-Length': '0'} 2023-05-31T20:53:53.868 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:44:50 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000012ae0e3077934054-006477b3e1-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:53:53 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:53:53.869 DEBUG:boto:path=/pklxkd-59/ 2023-05-31T20:53:53.869 DEBUG:boto:auth_path=/pklxkd-59/ 2023-05-31T20:53:53.870 DEBUG:boto:path=/pklxkd-59/?prefix=obj38 2023-05-31T20:53:53.870 DEBUG:boto:auth_path=/pklxkd-59/?prefix=obj38 2023-05-31T20:53:53.870 DEBUG:boto:Method: GET 2023-05-31T20:53:53.871 DEBUG:boto:Path: /pklxkd-59/?prefix=obj38 2023-05-31T20:53:53.871 DEBUG:boto:Data: 2023-05-31T20:53:53.872 DEBUG:boto:Headers: {} 2023-05-31T20:53:53.872 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:53:53.872 DEBUG:boto:Port: 8001 2023-05-31T20:53:53.873 DEBUG:boto:Params: {} 2023-05-31T20:53:53.873 DEBUG:boto:Token: None 2023-05-31T20:53:53.873 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:53:53 GMT /pklxkd-59/ 2023-05-31T20:53:53.874 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:diXwr3WzChuZ6w/0HLW0xEXCAKg= 2023-05-31T20:53:53.874 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:53:53 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:diXwr3WzChuZ6w/0HLW0xEXCAKg=', 'Content-Length': '0'} 2023-05-31T20:53:53.877 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000059a14d03730a700b-006477b3e1-1c17-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:53:53 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:53:53.877 DEBUG:boto:b'pklxkd-59obj381000falseobj382023-05-31T20:44:50.610Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:53:53.878 DEBUG:boto:path=/pklxkd-59/ 2023-05-31T20:53:53.878 DEBUG:boto:auth_path=/pklxkd-59/ 2023-05-31T20:53:53.878 DEBUG:boto:path=/pklxkd-59/?prefix=obj38 2023-05-31T20:53:53.879 DEBUG:boto:auth_path=/pklxkd-59/?prefix=obj38 2023-05-31T20:53:53.879 DEBUG:boto:Method: GET 2023-05-31T20:53:53.879 DEBUG:boto:Path: /pklxkd-59/?prefix=obj38 2023-05-31T20:53:53.880 DEBUG:boto:Data: 2023-05-31T20:53:53.880 DEBUG:boto:Headers: {} 2023-05-31T20:53:53.881 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:53:53.881 DEBUG:boto:Port: 8000 2023-05-31T20:53:53.881 DEBUG:boto:Params: {} 2023-05-31T20:53:53.882 DEBUG:boto:Token: None 2023-05-31T20:53:53.882 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:53:53 GMT /pklxkd-59/ 2023-05-31T20:53:53.882 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:diXwr3WzChuZ6w/0HLW0xEXCAKg= 2023-05-31T20:53:53.883 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:53:53 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:diXwr3WzChuZ6w/0HLW0xEXCAKg=', 'Content-Length': '0'} 2023-05-31T20:53:53.887 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000027b604ef9f24a7c2-006477b3e1-1630-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:53:53 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:53:53.887 DEBUG:boto:b'pklxkd-59obj381000falseobj382023-05-31T20:44:50.610Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:53:53.888 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj44 2023-05-31T20:53:53.888 DEBUG:boto:path=/pklxkd-59/obj44 2023-05-31T20:53:53.889 DEBUG:boto:auth_path=/pklxkd-59/obj44 2023-05-31T20:53:53.889 DEBUG:boto:path=/pklxkd-59/obj44?versionId=null 2023-05-31T20:53:53.889 DEBUG:boto:auth_path=/pklxkd-59/obj44?versionId=null 2023-05-31T20:53:53.890 DEBUG:boto:Method: GET 2023-05-31T20:53:53.890 DEBUG:boto:Path: /pklxkd-59/obj44?versionId=null 2023-05-31T20:53:53.890 DEBUG:boto:Data: 2023-05-31T20:53:53.891 DEBUG:boto:Headers: {} 2023-05-31T20:53:53.891 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:53:53.891 DEBUG:boto:Port: 8001 2023-05-31T20:53:53.892 DEBUG:boto:Params: {} 2023-05-31T20:53:53.892 DEBUG:boto:Token: None 2023-05-31T20:53:53.893 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:53:53 GMT /pklxkd-59/obj44?versionId=null 2023-05-31T20:53:53.893 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:NrpS8BlQmvJid4G24+dg2XcneA0= 2023-05-31T20:53:53.893 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:53:53 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:NrpS8BlQmvJid4G24+dg2XcneA0=', 'Content-Length': '0'} 2023-05-31T20:53:53.895 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:39:26 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a5902882a069343c-006477b3e1-1c17-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:53:53 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:53:53.896 DEBUG:boto:path=/pklxkd-59/obj44 2023-05-31T20:53:53.896 DEBUG:boto:auth_path=/pklxkd-59/obj44 2023-05-31T20:53:53.897 DEBUG:boto:path=/pklxkd-59/obj44?versionId=null 2023-05-31T20:53:53.897 DEBUG:boto:auth_path=/pklxkd-59/obj44?versionId=null 2023-05-31T20:53:53.897 DEBUG:boto:Method: GET 2023-05-31T20:53:53.898 DEBUG:boto:Path: /pklxkd-59/obj44?versionId=null 2023-05-31T20:53:53.898 DEBUG:boto:Data: 2023-05-31T20:53:53.898 DEBUG:boto:Headers: {} 2023-05-31T20:53:53.899 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:53:53.899 DEBUG:boto:Port: 8000 2023-05-31T20:53:53.899 DEBUG:boto:Params: {} 2023-05-31T20:53:53.900 DEBUG:boto:Token: None 2023-05-31T20:53:53.900 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:53:53 GMT /pklxkd-59/obj44?versionId=null 2023-05-31T20:53:53.901 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:NrpS8BlQmvJid4G24+dg2XcneA0= 2023-05-31T20:53:53.901 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:53:53 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:NrpS8BlQmvJid4G24+dg2XcneA0=', 'Content-Length': '0'} 2023-05-31T20:53:53.903 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:39:26 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000070a3356e13867793-006477b3e1-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:53:53 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:53:53.904 DEBUG:boto:path=/pklxkd-59/obj44 2023-05-31T20:53:53.904 DEBUG:boto:auth_path=/pklxkd-59/obj44 2023-05-31T20:53:53.904 DEBUG:boto:path=/pklxkd-59/obj44?versionId=null 2023-05-31T20:53:53.905 DEBUG:boto:auth_path=/pklxkd-59/obj44?versionId=null 2023-05-31T20:53:53.905 DEBUG:boto:Method: HEAD 2023-05-31T20:53:53.905 DEBUG:boto:Path: /pklxkd-59/obj44?versionId=null 2023-05-31T20:53:53.906 DEBUG:boto:Data: 2023-05-31T20:53:53.906 DEBUG:boto:Headers: {} 2023-05-31T20:53:53.906 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:53:53.907 DEBUG:boto:Port: 8001 2023-05-31T20:53:53.907 DEBUG:boto:Params: {} 2023-05-31T20:53:53.907 DEBUG:boto:Token: None 2023-05-31T20:53:53.908 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:53:53 GMT /pklxkd-59/obj44?versionId=null 2023-05-31T20:53:53.908 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:3/xHCTIymkb0de68+sbdEI+UnzA= 2023-05-31T20:53:53.908 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:53:53 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:3/xHCTIymkb0de68+sbdEI+UnzA=', 'Content-Length': '0'} 2023-05-31T20:53:53.910 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:39:26 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a400ab9940039aa3-006477b3e1-1c17-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:53:53 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:53:53.911 DEBUG:boto:path=/pklxkd-59/obj44 2023-05-31T20:53:53.911 DEBUG:boto:auth_path=/pklxkd-59/obj44 2023-05-31T20:53:53.911 DEBUG:boto:path=/pklxkd-59/obj44?versionId=null 2023-05-31T20:53:53.912 DEBUG:boto:auth_path=/pklxkd-59/obj44?versionId=null 2023-05-31T20:53:53.912 DEBUG:boto:Method: HEAD 2023-05-31T20:53:53.912 DEBUG:boto:Path: /pklxkd-59/obj44?versionId=null 2023-05-31T20:53:53.913 DEBUG:boto:Data: 2023-05-31T20:53:53.913 DEBUG:boto:Headers: {} 2023-05-31T20:53:53.913 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:53:53.914 DEBUG:boto:Port: 8000 2023-05-31T20:53:53.914 DEBUG:boto:Params: {} 2023-05-31T20:53:53.914 DEBUG:boto:Token: None 2023-05-31T20:53:53.915 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:53:53 GMT /pklxkd-59/obj44?versionId=null 2023-05-31T20:53:53.915 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:3/xHCTIymkb0de68+sbdEI+UnzA= 2023-05-31T20:53:53.916 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:53:53 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:3/xHCTIymkb0de68+sbdEI+UnzA=', 'Content-Length': '0'} 2023-05-31T20:53:53.917 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:39:26 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005eb6e2ccf6edde01-006477b3e1-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:53:53 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:53:53.918 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj44 2023-05-31T20:53:53.918 DEBUG:boto:path=/pklxkd-59/obj44 2023-05-31T20:53:53.919 DEBUG:boto:auth_path=/pklxkd-59/obj44 2023-05-31T20:53:53.919 DEBUG:boto:path=/pklxkd-59/obj44?versionId=null 2023-05-31T20:53:53.919 DEBUG:boto:auth_path=/pklxkd-59/obj44?versionId=null 2023-05-31T20:53:53.920 DEBUG:boto:Method: GET 2023-05-31T20:53:53.920 DEBUG:boto:Path: /pklxkd-59/obj44?versionId=null 2023-05-31T20:53:53.920 DEBUG:boto:Data: 2023-05-31T20:53:53.921 DEBUG:boto:Headers: {} 2023-05-31T20:53:53.921 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:53:53.921 DEBUG:boto:Port: 8001 2023-05-31T20:53:53.922 DEBUG:boto:Params: {} 2023-05-31T20:53:53.922 DEBUG:boto:Token: None 2023-05-31T20:53:53.923 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:53:53 GMT /pklxkd-59/obj44?versionId=null 2023-05-31T20:53:53.923 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:NrpS8BlQmvJid4G24+dg2XcneA0= 2023-05-31T20:53:53.923 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:53:53 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:NrpS8BlQmvJid4G24+dg2XcneA0=', 'Content-Length': '0'} 2023-05-31T20:53:53.925 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:39:26 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000079020d38875d8032-006477b3e1-1c17-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:53:53 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:53:53.926 DEBUG:boto:path=/pklxkd-59/obj44 2023-05-31T20:53:53.926 DEBUG:boto:auth_path=/pklxkd-59/obj44 2023-05-31T20:53:53.926 DEBUG:boto:path=/pklxkd-59/obj44?versionId=null 2023-05-31T20:53:53.927 DEBUG:boto:auth_path=/pklxkd-59/obj44?versionId=null 2023-05-31T20:53:53.927 DEBUG:boto:Method: GET 2023-05-31T20:53:53.927 DEBUG:boto:Path: /pklxkd-59/obj44?versionId=null 2023-05-31T20:53:53.928 DEBUG:boto:Data: 2023-05-31T20:53:53.928 DEBUG:boto:Headers: {} 2023-05-31T20:53:53.929 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:53:53.929 DEBUG:boto:Port: 8000 2023-05-31T20:53:53.929 DEBUG:boto:Params: {} 2023-05-31T20:53:53.930 DEBUG:boto:Token: None 2023-05-31T20:53:53.930 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:53:53 GMT /pklxkd-59/obj44?versionId=null 2023-05-31T20:53:53.930 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:NrpS8BlQmvJid4G24+dg2XcneA0= 2023-05-31T20:53:53.931 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:53:53 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:NrpS8BlQmvJid4G24+dg2XcneA0=', 'Content-Length': '0'} 2023-05-31T20:53:53.933 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:39:26 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c50c377e35f0df07-006477b3e1-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:53:53 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:53:53.933 DEBUG:boto:path=/pklxkd-59/ 2023-05-31T20:53:53.934 DEBUG:boto:auth_path=/pklxkd-59/ 2023-05-31T20:53:53.934 DEBUG:boto:path=/pklxkd-59/?prefix=obj44 2023-05-31T20:53:53.934 DEBUG:boto:auth_path=/pklxkd-59/?prefix=obj44 2023-05-31T20:53:53.935 DEBUG:boto:Method: GET 2023-05-31T20:53:53.935 DEBUG:boto:Path: /pklxkd-59/?prefix=obj44 2023-05-31T20:53:53.935 DEBUG:boto:Data: 2023-05-31T20:53:53.936 DEBUG:boto:Headers: {} 2023-05-31T20:53:53.936 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:53:53.936 DEBUG:boto:Port: 8001 2023-05-31T20:53:53.937 DEBUG:boto:Params: {} 2023-05-31T20:53:53.937 DEBUG:boto:Token: None 2023-05-31T20:53:53.938 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:53:53 GMT /pklxkd-59/ 2023-05-31T20:53:53.938 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:diXwr3WzChuZ6w/0HLW0xEXCAKg= 2023-05-31T20:53:53.938 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:53:53 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:diXwr3WzChuZ6w/0HLW0xEXCAKg=', 'Content-Length': '0'} 2023-05-31T20:53:53.941 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000004ae7ec8755150919-006477b3e1-1c17-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:53:53 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:53:53.984 DEBUG:boto:b'pklxkd-59obj441000falseobj442023-05-31T20:39:26.565Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:53:53.985 DEBUG:boto:path=/pklxkd-59/ 2023-05-31T20:53:53.985 DEBUG:boto:auth_path=/pklxkd-59/ 2023-05-31T20:53:53.986 DEBUG:boto:path=/pklxkd-59/?prefix=obj44 2023-05-31T20:53:53.986 DEBUG:boto:auth_path=/pklxkd-59/?prefix=obj44 2023-05-31T20:53:53.986 DEBUG:boto:Method: GET 2023-05-31T20:53:53.987 DEBUG:boto:Path: /pklxkd-59/?prefix=obj44 2023-05-31T20:53:53.987 DEBUG:boto:Data: 2023-05-31T20:53:53.987 DEBUG:boto:Headers: {} 2023-05-31T20:53:53.988 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:53:53.988 DEBUG:boto:Port: 8000 2023-05-31T20:53:53.988 DEBUG:boto:Params: {} 2023-05-31T20:53:53.989 DEBUG:boto:Token: None 2023-05-31T20:53:53.989 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:53:53 GMT /pklxkd-59/ 2023-05-31T20:53:53.990 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:diXwr3WzChuZ6w/0HLW0xEXCAKg= 2023-05-31T20:53:53.990 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:53:53 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:diXwr3WzChuZ6w/0HLW0xEXCAKg=', 'Content-Length': '0'} 2023-05-31T20:53:53.994 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000090399537a802e805-006477b3e1-1630-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:53:53 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:53:53.995 DEBUG:boto:b'pklxkd-59obj441000falseobj442023-05-31T20:39:26.565Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:53:53.995 DEBUG:boto:path=/pklxkd-59/obj44 2023-05-31T20:53:53.996 DEBUG:boto:auth_path=/pklxkd-59/obj44 2023-05-31T20:53:53.996 DEBUG:boto:Method: HEAD 2023-05-31T20:53:53.996 DEBUG:boto:Path: /pklxkd-59/obj44 2023-05-31T20:53:53.997 DEBUG:boto:Data: 2023-05-31T20:53:53.997 DEBUG:boto:Headers: {} 2023-05-31T20:53:53.997 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:53:53.998 DEBUG:boto:Port: 8001 2023-05-31T20:53:53.998 DEBUG:boto:Params: {} 2023-05-31T20:53:53.998 DEBUG:boto:Token: None 2023-05-31T20:53:53.999 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:53:53 GMT /pklxkd-59/obj44 2023-05-31T20:53:53.999 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:J1VHz7tSFOXR/QSSJ8KZxo5eIOA= 2023-05-31T20:53:54.000 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:53:53 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:J1VHz7tSFOXR/QSSJ8KZxo5eIOA=', 'Content-Length': '0'} 2023-05-31T20:53:54.001 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:39:26 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b4ad6113f2ce124c-006477b3e1-1c17-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:53:53 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:53:54.002 DEBUG:boto:path=/pklxkd-59/obj44 2023-05-31T20:53:54.002 DEBUG:boto:auth_path=/pklxkd-59/obj44 2023-05-31T20:53:54.003 DEBUG:boto:Method: HEAD 2023-05-31T20:53:54.003 DEBUG:boto:Path: /pklxkd-59/obj44 2023-05-31T20:53:54.003 DEBUG:boto:Data: 2023-05-31T20:53:54.004 DEBUG:boto:Headers: {} 2023-05-31T20:53:54.004 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:53:54.004 DEBUG:boto:Port: 8000 2023-05-31T20:53:54.005 DEBUG:boto:Params: {} 2023-05-31T20:53:54.005 DEBUG:boto:Token: None 2023-05-31T20:53:54.006 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:53:54 GMT /pklxkd-59/obj44 2023-05-31T20:53:54.006 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:P7q6MA/G4Trn2s4aSmxAxu8xAow= 2023-05-31T20:53:54.006 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:53:54 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:P7q6MA/G4Trn2s4aSmxAxu8xAow=', 'Content-Length': '0'} 2023-05-31T20:53:54.008 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:39:26 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006ab910a90d070190-006477b3e2-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:53:54 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:53:54.009 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj44 2023-05-31T20:53:54.009 DEBUG:boto:path=/pklxkd-59/obj44 2023-05-31T20:53:54.009 DEBUG:boto:auth_path=/pklxkd-59/obj44 2023-05-31T20:53:54.010 DEBUG:boto:Method: GET 2023-05-31T20:53:54.010 DEBUG:boto:Path: /pklxkd-59/obj44 2023-05-31T20:53:54.011 DEBUG:boto:Data: 2023-05-31T20:53:54.011 DEBUG:boto:Headers: {} 2023-05-31T20:53:54.011 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:53:54.012 DEBUG:boto:Port: 8001 2023-05-31T20:53:54.012 DEBUG:boto:Params: {} 2023-05-31T20:53:54.012 DEBUG:boto:Token: None 2023-05-31T20:53:54.013 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:53:54 GMT /pklxkd-59/obj44 2023-05-31T20:53:54.013 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:SfxTG5dzglOaPdZ6ilqhD8p4bWE= 2023-05-31T20:53:54.014 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:53:54 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:SfxTG5dzglOaPdZ6ilqhD8p4bWE=', 'Content-Length': '0'} 2023-05-31T20:53:54.015 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:39:26 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a59b6fe7693444d4-006477b3e2-1c17-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:53:54 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:53:54.016 DEBUG:boto:path=/pklxkd-59/obj44 2023-05-31T20:53:54.016 DEBUG:boto:auth_path=/pklxkd-59/obj44 2023-05-31T20:53:54.017 DEBUG:boto:Method: GET 2023-05-31T20:53:54.017 DEBUG:boto:Path: /pklxkd-59/obj44 2023-05-31T20:53:54.017 DEBUG:boto:Data: 2023-05-31T20:53:54.018 DEBUG:boto:Headers: {} 2023-05-31T20:53:54.018 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:53:54.019 DEBUG:boto:Port: 8000 2023-05-31T20:53:54.019 DEBUG:boto:Params: {} 2023-05-31T20:53:54.019 DEBUG:boto:Token: None 2023-05-31T20:53:54.020 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:53:54 GMT /pklxkd-59/obj44 2023-05-31T20:53:54.020 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:SfxTG5dzglOaPdZ6ilqhD8p4bWE= 2023-05-31T20:53:54.021 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:53:54 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:SfxTG5dzglOaPdZ6ilqhD8p4bWE=', 'Content-Length': '0'} 2023-05-31T20:53:54.022 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:39:26 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009819a39996233e6d-006477b3e2-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:53:54 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:53:54.064 DEBUG:boto:path=/pklxkd-59/ 2023-05-31T20:53:54.065 DEBUG:boto:auth_path=/pklxkd-59/ 2023-05-31T20:53:54.065 DEBUG:boto:path=/pklxkd-59/?prefix=obj44 2023-05-31T20:53:54.065 DEBUG:boto:auth_path=/pklxkd-59/?prefix=obj44 2023-05-31T20:53:54.066 DEBUG:boto:Method: GET 2023-05-31T20:53:54.066 DEBUG:boto:Path: /pklxkd-59/?prefix=obj44 2023-05-31T20:53:54.066 DEBUG:boto:Data: 2023-05-31T20:53:54.067 DEBUG:boto:Headers: {} 2023-05-31T20:53:54.067 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:53:54.067 DEBUG:boto:Port: 8001 2023-05-31T20:53:54.068 DEBUG:boto:Params: {} 2023-05-31T20:53:54.068 DEBUG:boto:Token: None 2023-05-31T20:53:54.069 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:53:54 GMT /pklxkd-59/ 2023-05-31T20:53:54.069 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:X6lj3OWBH37r6jmqQli80RnWHOg= 2023-05-31T20:53:54.070 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:53:54 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:X6lj3OWBH37r6jmqQli80RnWHOg=', 'Content-Length': '0'} 2023-05-31T20:53:54.072 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000009bbe9f5a90a7caf4-006477b3e2-1c17-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:53:54 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:53:54.072 DEBUG:boto:b'pklxkd-59obj441000falseobj442023-05-31T20:39:26.565Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:53:54.073 DEBUG:boto:path=/pklxkd-59/ 2023-05-31T20:53:54.074 DEBUG:boto:auth_path=/pklxkd-59/ 2023-05-31T20:53:54.074 DEBUG:boto:path=/pklxkd-59/?prefix=obj44 2023-05-31T20:53:54.074 DEBUG:boto:auth_path=/pklxkd-59/?prefix=obj44 2023-05-31T20:53:54.075 DEBUG:boto:Method: GET 2023-05-31T20:53:54.075 DEBUG:boto:Path: /pklxkd-59/?prefix=obj44 2023-05-31T20:53:54.075 DEBUG:boto:Data: 2023-05-31T20:53:54.076 DEBUG:boto:Headers: {} 2023-05-31T20:53:54.076 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:53:54.076 DEBUG:boto:Port: 8000 2023-05-31T20:53:54.077 DEBUG:boto:Params: {} 2023-05-31T20:53:54.077 DEBUG:boto:Token: None 2023-05-31T20:53:54.078 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:53:54 GMT /pklxkd-59/ 2023-05-31T20:53:54.078 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:X6lj3OWBH37r6jmqQli80RnWHOg= 2023-05-31T20:53:54.078 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:53:54 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:X6lj3OWBH37r6jmqQli80RnWHOg=', 'Content-Length': '0'} 2023-05-31T20:53:54.082 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000084c6191a401de33f-006477b3e2-1630-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:53:54 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:53:54.083 DEBUG:boto:b'pklxkd-59obj441000falseobj442023-05-31T20:39:26.565Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:53:54.084 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj46 2023-05-31T20:53:54.084 DEBUG:boto:path=/pklxkd-59/obj46 2023-05-31T20:53:54.084 DEBUG:boto:auth_path=/pklxkd-59/obj46 2023-05-31T20:53:54.085 DEBUG:boto:path=/pklxkd-59/obj46?versionId=null 2023-05-31T20:53:54.085 DEBUG:boto:auth_path=/pklxkd-59/obj46?versionId=null 2023-05-31T20:53:54.085 DEBUG:boto:Method: GET 2023-05-31T20:53:54.086 DEBUG:boto:Path: /pklxkd-59/obj46?versionId=null 2023-05-31T20:53:54.086 DEBUG:boto:Data: 2023-05-31T20:53:54.087 DEBUG:boto:Headers: {} 2023-05-31T20:53:54.087 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:53:54.087 DEBUG:boto:Port: 8001 2023-05-31T20:53:54.088 DEBUG:boto:Params: {} 2023-05-31T20:53:54.088 DEBUG:boto:Token: None 2023-05-31T20:53:54.088 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:53:54 GMT /pklxkd-59/obj46?versionId=null 2023-05-31T20:53:54.089 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:EOpiWjFPUrr82u8ehbndRj0KCBg= 2023-05-31T20:53:54.089 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:53:54 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:EOpiWjFPUrr82u8ehbndRj0KCBg=', 'Content-Length': '0'} 2023-05-31T20:53:54.091 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:46:33 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f9eeeff483df4cad-006477b3e2-1c17-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:53:54 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:53:54.092 DEBUG:boto:path=/pklxkd-59/obj46 2023-05-31T20:53:54.092 DEBUG:boto:auth_path=/pklxkd-59/obj46 2023-05-31T20:53:54.093 DEBUG:boto:path=/pklxkd-59/obj46?versionId=null 2023-05-31T20:53:54.093 DEBUG:boto:auth_path=/pklxkd-59/obj46?versionId=null 2023-05-31T20:53:54.093 DEBUG:boto:Method: GET 2023-05-31T20:53:54.094 DEBUG:boto:Path: /pklxkd-59/obj46?versionId=null 2023-05-31T20:53:54.094 DEBUG:boto:Data: 2023-05-31T20:53:54.094 DEBUG:boto:Headers: {} 2023-05-31T20:53:54.095 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:53:54.095 DEBUG:boto:Port: 8000 2023-05-31T20:53:54.095 DEBUG:boto:Params: {} 2023-05-31T20:53:54.096 DEBUG:boto:Token: None 2023-05-31T20:53:54.096 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:53:54 GMT /pklxkd-59/obj46?versionId=null 2023-05-31T20:53:54.097 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:EOpiWjFPUrr82u8ehbndRj0KCBg= 2023-05-31T20:53:54.097 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:53:54 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:EOpiWjFPUrr82u8ehbndRj0KCBg=', 'Content-Length': '0'} 2023-05-31T20:53:54.099 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:46:33 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a9acd924ff176eed-006477b3e2-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:53:54 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:53:54.100 DEBUG:boto:path=/pklxkd-59/obj46 2023-05-31T20:53:54.100 DEBUG:boto:auth_path=/pklxkd-59/obj46 2023-05-31T20:53:54.100 DEBUG:boto:path=/pklxkd-59/obj46?versionId=null 2023-05-31T20:53:54.101 DEBUG:boto:auth_path=/pklxkd-59/obj46?versionId=null 2023-05-31T20:53:54.101 DEBUG:boto:Method: HEAD 2023-05-31T20:53:54.101 DEBUG:boto:Path: /pklxkd-59/obj46?versionId=null 2023-05-31T20:53:54.102 DEBUG:boto:Data: 2023-05-31T20:53:54.102 DEBUG:boto:Headers: {} 2023-05-31T20:53:54.102 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:53:54.103 DEBUG:boto:Port: 8001 2023-05-31T20:53:54.103 DEBUG:boto:Params: {} 2023-05-31T20:53:54.103 DEBUG:boto:Token: None 2023-05-31T20:53:54.104 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:53:54 GMT /pklxkd-59/obj46?versionId=null 2023-05-31T20:53:54.104 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:H9lO+VlJaZUxueerDSW6Tf3eUAY= 2023-05-31T20:53:54.105 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:53:54 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:H9lO+VlJaZUxueerDSW6Tf3eUAY=', 'Content-Length': '0'} 2023-05-31T20:53:54.106 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:46:33 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003f1d16c5211069a0-006477b3e2-1c17-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:53:54 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:53:54.107 DEBUG:boto:path=/pklxkd-59/obj46 2023-05-31T20:53:54.107 DEBUG:boto:auth_path=/pklxkd-59/obj46 2023-05-31T20:53:54.108 DEBUG:boto:path=/pklxkd-59/obj46?versionId=null 2023-05-31T20:53:54.108 DEBUG:boto:auth_path=/pklxkd-59/obj46?versionId=null 2023-05-31T20:53:54.108 DEBUG:boto:Method: HEAD 2023-05-31T20:53:54.109 DEBUG:boto:Path: /pklxkd-59/obj46?versionId=null 2023-05-31T20:53:54.109 DEBUG:boto:Data: 2023-05-31T20:53:54.109 DEBUG:boto:Headers: {} 2023-05-31T20:53:54.110 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:53:54.110 DEBUG:boto:Port: 8000 2023-05-31T20:53:54.110 DEBUG:boto:Params: {} 2023-05-31T20:53:54.111 DEBUG:boto:Token: None 2023-05-31T20:53:54.111 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:53:54 GMT /pklxkd-59/obj46?versionId=null 2023-05-31T20:53:54.112 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:H9lO+VlJaZUxueerDSW6Tf3eUAY= 2023-05-31T20:53:54.112 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:53:54 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:H9lO+VlJaZUxueerDSW6Tf3eUAY=', 'Content-Length': '0'} 2023-05-31T20:53:54.114 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:46:33 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f266c6522935147a-006477b3e2-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:53:54 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:53:54.114 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj46 2023-05-31T20:53:54.115 DEBUG:boto:path=/pklxkd-59/obj46 2023-05-31T20:53:54.115 DEBUG:boto:auth_path=/pklxkd-59/obj46 2023-05-31T20:53:54.115 DEBUG:boto:path=/pklxkd-59/obj46?versionId=null 2023-05-31T20:53:54.116 DEBUG:boto:auth_path=/pklxkd-59/obj46?versionId=null 2023-05-31T20:53:54.116 DEBUG:boto:Method: GET 2023-05-31T20:53:54.116 DEBUG:boto:Path: /pklxkd-59/obj46?versionId=null 2023-05-31T20:53:54.117 DEBUG:boto:Data: 2023-05-31T20:53:54.117 DEBUG:boto:Headers: {} 2023-05-31T20:53:54.118 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:53:54.118 DEBUG:boto:Port: 8001 2023-05-31T20:53:54.118 DEBUG:boto:Params: {} 2023-05-31T20:53:54.119 DEBUG:boto:Token: None 2023-05-31T20:53:54.119 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:53:54 GMT /pklxkd-59/obj46?versionId=null 2023-05-31T20:53:54.119 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:EOpiWjFPUrr82u8ehbndRj0KCBg= 2023-05-31T20:53:54.120 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:53:54 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:EOpiWjFPUrr82u8ehbndRj0KCBg=', 'Content-Length': '0'} 2023-05-31T20:53:54.122 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:46:33 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000fee34acafcfd5743-006477b3e2-1c17-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:53:54 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:53:54.122 DEBUG:boto:path=/pklxkd-59/obj46 2023-05-31T20:53:54.123 DEBUG:boto:auth_path=/pklxkd-59/obj46 2023-05-31T20:53:54.123 DEBUG:boto:path=/pklxkd-59/obj46?versionId=null 2023-05-31T20:53:54.124 DEBUG:boto:auth_path=/pklxkd-59/obj46?versionId=null 2023-05-31T20:53:54.124 DEBUG:boto:Method: GET 2023-05-31T20:53:54.124 DEBUG:boto:Path: /pklxkd-59/obj46?versionId=null 2023-05-31T20:53:54.125 DEBUG:boto:Data: 2023-05-31T20:53:54.125 DEBUG:boto:Headers: {} 2023-05-31T20:53:54.125 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:53:54.126 DEBUG:boto:Port: 8000 2023-05-31T20:53:54.126 DEBUG:boto:Params: {} 2023-05-31T20:53:54.126 DEBUG:boto:Token: None 2023-05-31T20:53:54.127 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:53:54 GMT /pklxkd-59/obj46?versionId=null 2023-05-31T20:53:54.127 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:EOpiWjFPUrr82u8ehbndRj0KCBg= 2023-05-31T20:53:54.128 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:53:54 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:EOpiWjFPUrr82u8ehbndRj0KCBg=', 'Content-Length': '0'} 2023-05-31T20:53:54.129 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:46:33 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009b4dda09bb168fbc-006477b3e2-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:53:54 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:53:54.130 DEBUG:boto:path=/pklxkd-59/ 2023-05-31T20:53:54.130 DEBUG:boto:auth_path=/pklxkd-59/ 2023-05-31T20:53:54.131 DEBUG:boto:path=/pklxkd-59/?prefix=obj46 2023-05-31T20:53:54.131 DEBUG:boto:auth_path=/pklxkd-59/?prefix=obj46 2023-05-31T20:53:54.132 DEBUG:boto:Method: GET 2023-05-31T20:53:54.132 DEBUG:boto:Path: /pklxkd-59/?prefix=obj46 2023-05-31T20:53:54.132 DEBUG:boto:Data: 2023-05-31T20:53:54.133 DEBUG:boto:Headers: {} 2023-05-31T20:53:54.133 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:53:54.133 DEBUG:boto:Port: 8001 2023-05-31T20:53:54.134 DEBUG:boto:Params: {} 2023-05-31T20:53:54.134 DEBUG:boto:Token: None 2023-05-31T20:53:54.134 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:53:54 GMT /pklxkd-59/ 2023-05-31T20:53:54.135 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:X6lj3OWBH37r6jmqQli80RnWHOg= 2023-05-31T20:53:54.135 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:53:54 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:X6lj3OWBH37r6jmqQli80RnWHOg=', 'Content-Length': '0'} 2023-05-31T20:53:54.137 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e1e958da35eef6e0-006477b3e2-1c17-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:53:54 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:53:54.138 DEBUG:boto:b'pklxkd-59obj461000falseobj462023-05-31T20:46:33.830Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:53:54.139 DEBUG:boto:path=/pklxkd-59/ 2023-05-31T20:53:54.139 DEBUG:boto:auth_path=/pklxkd-59/ 2023-05-31T20:53:54.139 DEBUG:boto:path=/pklxkd-59/?prefix=obj46 2023-05-31T20:53:54.140 DEBUG:boto:auth_path=/pklxkd-59/?prefix=obj46 2023-05-31T20:53:54.140 DEBUG:boto:Method: GET 2023-05-31T20:53:54.140 DEBUG:boto:Path: /pklxkd-59/?prefix=obj46 2023-05-31T20:53:54.141 DEBUG:boto:Data: 2023-05-31T20:53:54.141 DEBUG:boto:Headers: {} 2023-05-31T20:53:54.141 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:53:54.142 DEBUG:boto:Port: 8000 2023-05-31T20:53:54.142 DEBUG:boto:Params: {} 2023-05-31T20:53:54.143 DEBUG:boto:Token: None 2023-05-31T20:53:54.143 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:53:54 GMT /pklxkd-59/ 2023-05-31T20:53:54.143 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:X6lj3OWBH37r6jmqQli80RnWHOg= 2023-05-31T20:53:54.144 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:53:54 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:X6lj3OWBH37r6jmqQli80RnWHOg=', 'Content-Length': '0'} 2023-05-31T20:53:54.147 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a37d31235257d200-006477b3e2-1630-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:53:54 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:53:54.148 DEBUG:boto:b'pklxkd-59obj461000falseobj462023-05-31T20:46:33.830Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:53:54.149 DEBUG:boto:path=/pklxkd-59/obj46 2023-05-31T20:53:54.149 DEBUG:boto:auth_path=/pklxkd-59/obj46 2023-05-31T20:53:54.149 DEBUG:boto:Method: HEAD 2023-05-31T20:53:54.150 DEBUG:boto:Path: /pklxkd-59/obj46 2023-05-31T20:53:54.150 DEBUG:boto:Data: 2023-05-31T20:53:54.150 DEBUG:boto:Headers: {} 2023-05-31T20:53:54.151 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:53:54.151 DEBUG:boto:Port: 8001 2023-05-31T20:53:54.151 DEBUG:boto:Params: {} 2023-05-31T20:53:54.152 DEBUG:boto:Token: None 2023-05-31T20:53:54.152 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:53:54 GMT /pklxkd-59/obj46 2023-05-31T20:53:54.152 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:PGhX4tvBzyBF/r25rzhy3EhAb7M= 2023-05-31T20:53:54.153 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:53:54 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:PGhX4tvBzyBF/r25rzhy3EhAb7M=', 'Content-Length': '0'} 2023-05-31T20:53:54.154 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:46:33 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000061e1fbf96f44ed9e-006477b3e2-1c17-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:53:54 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:53:54.155 DEBUG:boto:path=/pklxkd-59/obj46 2023-05-31T20:53:54.155 DEBUG:boto:auth_path=/pklxkd-59/obj46 2023-05-31T20:53:54.156 DEBUG:boto:Method: HEAD 2023-05-31T20:53:54.156 DEBUG:boto:Path: /pklxkd-59/obj46 2023-05-31T20:53:54.156 DEBUG:boto:Data: 2023-05-31T20:53:54.157 DEBUG:boto:Headers: {} 2023-05-31T20:53:54.157 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:53:54.157 DEBUG:boto:Port: 8000 2023-05-31T20:53:54.158 DEBUG:boto:Params: {} 2023-05-31T20:53:54.158 DEBUG:boto:Token: None 2023-05-31T20:53:54.158 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:53:54 GMT /pklxkd-59/obj46 2023-05-31T20:53:54.159 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:PGhX4tvBzyBF/r25rzhy3EhAb7M= 2023-05-31T20:53:54.159 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:53:54 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:PGhX4tvBzyBF/r25rzhy3EhAb7M=', 'Content-Length': '0'} 2023-05-31T20:53:54.161 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:46:33 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000056c78dafb331647c-006477b3e2-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:53:54 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:53:54.162 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj46 2023-05-31T20:53:54.162 DEBUG:boto:path=/pklxkd-59/obj46 2023-05-31T20:53:54.162 DEBUG:boto:auth_path=/pklxkd-59/obj46 2023-05-31T20:53:54.163 DEBUG:boto:Method: GET 2023-05-31T20:53:54.163 DEBUG:boto:Path: /pklxkd-59/obj46 2023-05-31T20:53:54.163 DEBUG:boto:Data: 2023-05-31T20:53:54.164 DEBUG:boto:Headers: {} 2023-05-31T20:53:54.164 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:53:54.165 DEBUG:boto:Port: 8001 2023-05-31T20:53:54.165 DEBUG:boto:Params: {} 2023-05-31T20:53:54.165 DEBUG:boto:Token: None 2023-05-31T20:53:54.166 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:53:54 GMT /pklxkd-59/obj46 2023-05-31T20:53:54.166 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:A6bxvzvRBH2gvYCn0d3m7EcAzWQ= 2023-05-31T20:53:54.167 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:53:54 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:A6bxvzvRBH2gvYCn0d3m7EcAzWQ=', 'Content-Length': '0'} 2023-05-31T20:53:54.168 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:46:33 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ea4a049819c2de64-006477b3e2-1c17-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:53:54 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:53:54.212 DEBUG:boto:path=/pklxkd-59/obj46 2023-05-31T20:53:54.213 DEBUG:boto:auth_path=/pklxkd-59/obj46 2023-05-31T20:53:54.213 DEBUG:boto:Method: GET 2023-05-31T20:53:54.213 DEBUG:boto:Path: /pklxkd-59/obj46 2023-05-31T20:53:54.214 DEBUG:boto:Data: 2023-05-31T20:53:54.214 DEBUG:boto:Headers: {} 2023-05-31T20:53:54.214 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:53:54.215 DEBUG:boto:Port: 8000 2023-05-31T20:53:54.215 DEBUG:boto:Params: {} 2023-05-31T20:53:54.216 DEBUG:boto:Token: None 2023-05-31T20:53:54.216 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:53:54 GMT /pklxkd-59/obj46 2023-05-31T20:53:54.216 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:A6bxvzvRBH2gvYCn0d3m7EcAzWQ= 2023-05-31T20:53:54.217 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:53:54 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:A6bxvzvRBH2gvYCn0d3m7EcAzWQ=', 'Content-Length': '0'} 2023-05-31T20:53:54.219 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:46:33 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000cce88e71bc20a3fa-006477b3e2-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:53:54 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:53:54.219 DEBUG:boto:path=/pklxkd-59/ 2023-05-31T20:53:54.220 DEBUG:boto:auth_path=/pklxkd-59/ 2023-05-31T20:53:54.220 DEBUG:boto:path=/pklxkd-59/?prefix=obj46 2023-05-31T20:53:54.220 DEBUG:boto:auth_path=/pklxkd-59/?prefix=obj46 2023-05-31T20:53:54.221 DEBUG:boto:Method: GET 2023-05-31T20:53:54.221 DEBUG:boto:Path: /pklxkd-59/?prefix=obj46 2023-05-31T20:53:54.221 DEBUG:boto:Data: 2023-05-31T20:53:54.222 DEBUG:boto:Headers: {} 2023-05-31T20:53:54.222 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:53:54.222 DEBUG:boto:Port: 8001 2023-05-31T20:53:54.223 DEBUG:boto:Params: {} 2023-05-31T20:53:54.223 DEBUG:boto:Token: None 2023-05-31T20:53:54.224 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:53:54 GMT /pklxkd-59/ 2023-05-31T20:53:54.224 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:X6lj3OWBH37r6jmqQli80RnWHOg= 2023-05-31T20:53:54.224 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:53:54 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:X6lj3OWBH37r6jmqQli80RnWHOg=', 'Content-Length': '0'} 2023-05-31T20:53:54.227 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b31fd0050ea7755e-006477b3e2-1c17-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:53:54 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:53:54.227 DEBUG:boto:b'pklxkd-59obj461000falseobj462023-05-31T20:46:33.830Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:53:54.228 DEBUG:boto:path=/pklxkd-59/ 2023-05-31T20:53:54.228 DEBUG:boto:auth_path=/pklxkd-59/ 2023-05-31T20:53:54.229 DEBUG:boto:path=/pklxkd-59/?prefix=obj46 2023-05-31T20:53:54.229 DEBUG:boto:auth_path=/pklxkd-59/?prefix=obj46 2023-05-31T20:53:54.229 DEBUG:boto:Method: GET 2023-05-31T20:53:54.230 DEBUG:boto:Path: /pklxkd-59/?prefix=obj46 2023-05-31T20:53:54.230 DEBUG:boto:Data: 2023-05-31T20:53:54.230 DEBUG:boto:Headers: {} 2023-05-31T20:53:54.231 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:53:54.231 DEBUG:boto:Port: 8000 2023-05-31T20:53:54.231 DEBUG:boto:Params: {} 2023-05-31T20:53:54.232 DEBUG:boto:Token: None 2023-05-31T20:53:54.232 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:53:54 GMT /pklxkd-59/ 2023-05-31T20:53:54.233 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:X6lj3OWBH37r6jmqQli80RnWHOg= 2023-05-31T20:53:54.233 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:53:54 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:X6lj3OWBH37r6jmqQli80RnWHOg=', 'Content-Length': '0'} 2023-05-31T20:53:54.237 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c997b0e865637bfa-006477b3e2-1630-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:53:54 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:53:54.238 DEBUG:boto:b'pklxkd-59obj461000falseobj462023-05-31T20:46:33.830Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:53:54.238 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj51 2023-05-31T20:53:54.239 DEBUG:boto:path=/pklxkd-59/obj51 2023-05-31T20:53:54.239 DEBUG:boto:auth_path=/pklxkd-59/obj51 2023-05-31T20:53:54.239 DEBUG:boto:path=/pklxkd-59/obj51?versionId=null 2023-05-31T20:53:54.240 DEBUG:boto:auth_path=/pklxkd-59/obj51?versionId=null 2023-05-31T20:53:54.240 DEBUG:boto:Method: GET 2023-05-31T20:53:54.241 DEBUG:boto:Path: /pklxkd-59/obj51?versionId=null 2023-05-31T20:53:54.241 DEBUG:boto:Data: 2023-05-31T20:53:54.241 DEBUG:boto:Headers: {} 2023-05-31T20:53:54.242 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:53:54.242 DEBUG:boto:Port: 8001 2023-05-31T20:53:54.242 DEBUG:boto:Params: {} 2023-05-31T20:53:54.243 DEBUG:boto:Token: None 2023-05-31T20:53:54.243 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:53:54 GMT /pklxkd-59/obj51?versionId=null 2023-05-31T20:53:54.243 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:VkXvC/duxQJxFH0pyxSdGCyLo9A= 2023-05-31T20:53:54.244 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:53:54 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:VkXvC/duxQJxFH0pyxSdGCyLo9A=', 'Content-Length': '0'} 2023-05-31T20:53:54.246 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:43:08 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a9ff534bcb7071e2-006477b3e2-1c17-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:53:54 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:53:54.246 DEBUG:boto:path=/pklxkd-59/obj51 2023-05-31T20:53:54.247 DEBUG:boto:auth_path=/pklxkd-59/obj51 2023-05-31T20:53:54.247 DEBUG:boto:path=/pklxkd-59/obj51?versionId=null 2023-05-31T20:53:54.248 DEBUG:boto:auth_path=/pklxkd-59/obj51?versionId=null 2023-05-31T20:53:54.248 DEBUG:boto:Method: GET 2023-05-31T20:53:54.248 DEBUG:boto:Path: /pklxkd-59/obj51?versionId=null 2023-05-31T20:53:54.249 DEBUG:boto:Data: 2023-05-31T20:53:54.249 DEBUG:boto:Headers: {} 2023-05-31T20:53:54.249 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:53:54.250 DEBUG:boto:Port: 8000 2023-05-31T20:53:54.250 DEBUG:boto:Params: {} 2023-05-31T20:53:54.250 DEBUG:boto:Token: None 2023-05-31T20:53:54.251 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:53:54 GMT /pklxkd-59/obj51?versionId=null 2023-05-31T20:53:54.251 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:VkXvC/duxQJxFH0pyxSdGCyLo9A= 2023-05-31T20:53:54.252 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:53:54 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:VkXvC/duxQJxFH0pyxSdGCyLo9A=', 'Content-Length': '0'} 2023-05-31T20:53:54.253 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:43:08 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000058dfdd3c3eed4abf-006477b3e2-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:53:54 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:53:54.296 DEBUG:boto:path=/pklxkd-59/obj51 2023-05-31T20:53:54.297 DEBUG:boto:auth_path=/pklxkd-59/obj51 2023-05-31T20:53:54.297 DEBUG:boto:path=/pklxkd-59/obj51?versionId=null 2023-05-31T20:53:54.297 DEBUG:boto:auth_path=/pklxkd-59/obj51?versionId=null 2023-05-31T20:53:54.298 DEBUG:boto:Method: HEAD 2023-05-31T20:53:54.298 DEBUG:boto:Path: /pklxkd-59/obj51?versionId=null 2023-05-31T20:53:54.298 DEBUG:boto:Data: 2023-05-31T20:53:54.299 DEBUG:boto:Headers: {} 2023-05-31T20:53:54.299 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:53:54.299 DEBUG:boto:Port: 8001 2023-05-31T20:53:54.299 DEBUG:boto:Params: {} 2023-05-31T20:53:54.300 DEBUG:boto:Token: None 2023-05-31T20:53:54.300 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:53:54 GMT /pklxkd-59/obj51?versionId=null 2023-05-31T20:53:54.301 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:6hePLA+Iitk0kDzM+87dtwWjRVU= 2023-05-31T20:53:54.301 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:53:54 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:6hePLA+Iitk0kDzM+87dtwWjRVU=', 'Content-Length': '0'} 2023-05-31T20:53:54.303 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:43:08 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000055747414cb11ab52-006477b3e2-1c17-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:53:54 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:53:54.303 DEBUG:boto:path=/pklxkd-59/obj51 2023-05-31T20:53:54.304 DEBUG:boto:auth_path=/pklxkd-59/obj51 2023-05-31T20:53:54.304 DEBUG:boto:path=/pklxkd-59/obj51?versionId=null 2023-05-31T20:53:54.304 DEBUG:boto:auth_path=/pklxkd-59/obj51?versionId=null 2023-05-31T20:53:54.305 DEBUG:boto:Method: HEAD 2023-05-31T20:53:54.305 DEBUG:boto:Path: /pklxkd-59/obj51?versionId=null 2023-05-31T20:53:54.305 DEBUG:boto:Data: 2023-05-31T20:53:54.306 DEBUG:boto:Headers: {} 2023-05-31T20:53:54.306 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:53:54.306 DEBUG:boto:Port: 8000 2023-05-31T20:53:54.307 DEBUG:boto:Params: {} 2023-05-31T20:53:54.307 DEBUG:boto:Token: None 2023-05-31T20:53:54.307 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:53:54 GMT /pklxkd-59/obj51?versionId=null 2023-05-31T20:53:54.308 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:6hePLA+Iitk0kDzM+87dtwWjRVU= 2023-05-31T20:53:54.308 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:53:54 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:6hePLA+Iitk0kDzM+87dtwWjRVU=', 'Content-Length': '0'} 2023-05-31T20:53:54.310 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:43:08 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000dd6a8ff4087aec22-006477b3e2-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:53:54 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:53:54.311 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj51 2023-05-31T20:53:54.311 DEBUG:boto:path=/pklxkd-59/obj51 2023-05-31T20:53:54.311 DEBUG:boto:auth_path=/pklxkd-59/obj51 2023-05-31T20:53:54.312 DEBUG:boto:path=/pklxkd-59/obj51?versionId=null 2023-05-31T20:53:54.312 DEBUG:boto:auth_path=/pklxkd-59/obj51?versionId=null 2023-05-31T20:53:54.312 DEBUG:boto:Method: GET 2023-05-31T20:53:54.313 DEBUG:boto:Path: /pklxkd-59/obj51?versionId=null 2023-05-31T20:53:54.313 DEBUG:boto:Data: 2023-05-31T20:53:54.313 DEBUG:boto:Headers: {} 2023-05-31T20:53:54.314 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:53:54.314 DEBUG:boto:Port: 8001 2023-05-31T20:53:54.315 DEBUG:boto:Params: {} 2023-05-31T20:53:54.315 DEBUG:boto:Token: None 2023-05-31T20:53:54.315 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:53:54 GMT /pklxkd-59/obj51?versionId=null 2023-05-31T20:53:54.316 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:VkXvC/duxQJxFH0pyxSdGCyLo9A= 2023-05-31T20:53:54.316 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:53:54 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:VkXvC/duxQJxFH0pyxSdGCyLo9A=', 'Content-Length': '0'} 2023-05-31T20:53:54.318 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:43:08 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000bb6960b198f3d6e1-006477b3e2-1c17-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:53:54 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:53:54.319 DEBUG:boto:path=/pklxkd-59/obj51 2023-05-31T20:53:54.319 DEBUG:boto:auth_path=/pklxkd-59/obj51 2023-05-31T20:53:54.319 DEBUG:boto:path=/pklxkd-59/obj51?versionId=null 2023-05-31T20:53:54.320 DEBUG:boto:auth_path=/pklxkd-59/obj51?versionId=null 2023-05-31T20:53:54.320 DEBUG:boto:Method: GET 2023-05-31T20:53:54.321 DEBUG:boto:Path: /pklxkd-59/obj51?versionId=null 2023-05-31T20:53:54.321 DEBUG:boto:Data: 2023-05-31T20:53:54.321 DEBUG:boto:Headers: {} 2023-05-31T20:53:54.322 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:53:54.322 DEBUG:boto:Port: 8000 2023-05-31T20:53:54.322 DEBUG:boto:Params: {} 2023-05-31T20:53:54.323 DEBUG:boto:Token: None 2023-05-31T20:53:54.323 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:53:54 GMT /pklxkd-59/obj51?versionId=null 2023-05-31T20:53:54.323 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:VkXvC/duxQJxFH0pyxSdGCyLo9A= 2023-05-31T20:53:54.324 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:53:54 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:VkXvC/duxQJxFH0pyxSdGCyLo9A=', 'Content-Length': '0'} 2023-05-31T20:53:54.326 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:43:08 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000086ad25d108693719-006477b3e2-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:53:54 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:53:54.326 DEBUG:boto:path=/pklxkd-59/ 2023-05-31T20:53:54.327 DEBUG:boto:auth_path=/pklxkd-59/ 2023-05-31T20:53:54.327 DEBUG:boto:path=/pklxkd-59/?prefix=obj51 2023-05-31T20:53:54.328 DEBUG:boto:auth_path=/pklxkd-59/?prefix=obj51 2023-05-31T20:53:54.328 DEBUG:boto:Method: GET 2023-05-31T20:53:54.328 DEBUG:boto:Path: /pklxkd-59/?prefix=obj51 2023-05-31T20:53:54.329 DEBUG:boto:Data: 2023-05-31T20:53:54.329 DEBUG:boto:Headers: {} 2023-05-31T20:53:54.329 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:53:54.330 DEBUG:boto:Port: 8001 2023-05-31T20:53:54.330 DEBUG:boto:Params: {} 2023-05-31T20:53:54.331 DEBUG:boto:Token: None 2023-05-31T20:53:54.331 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:53:54 GMT /pklxkd-59/ 2023-05-31T20:53:54.331 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:X6lj3OWBH37r6jmqQli80RnWHOg= 2023-05-31T20:53:54.332 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:53:54 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:X6lj3OWBH37r6jmqQli80RnWHOg=', 'Content-Length': '0'} 2023-05-31T20:53:54.334 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000006c19cbe172cb0acf-006477b3e2-1c17-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:53:54 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:53:54.335 DEBUG:boto:b'pklxkd-59obj511000falseobj512023-05-31T20:43:08.011Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:53:54.335 DEBUG:boto:path=/pklxkd-59/ 2023-05-31T20:53:54.336 DEBUG:boto:auth_path=/pklxkd-59/ 2023-05-31T20:53:54.336 DEBUG:boto:path=/pklxkd-59/?prefix=obj51 2023-05-31T20:53:54.336 DEBUG:boto:auth_path=/pklxkd-59/?prefix=obj51 2023-05-31T20:53:54.337 DEBUG:boto:Method: GET 2023-05-31T20:53:54.337 DEBUG:boto:Path: /pklxkd-59/?prefix=obj51 2023-05-31T20:53:54.337 DEBUG:boto:Data: 2023-05-31T20:53:54.338 DEBUG:boto:Headers: {} 2023-05-31T20:53:54.338 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:53:54.338 DEBUG:boto:Port: 8000 2023-05-31T20:53:54.339 DEBUG:boto:Params: {} 2023-05-31T20:53:54.339 DEBUG:boto:Token: None 2023-05-31T20:53:54.340 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:53:54 GMT /pklxkd-59/ 2023-05-31T20:53:54.340 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:X6lj3OWBH37r6jmqQli80RnWHOg= 2023-05-31T20:53:54.340 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:53:54 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:X6lj3OWBH37r6jmqQli80RnWHOg=', 'Content-Length': '0'} 2023-05-31T20:53:54.345 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000008e3a8e6a93d007b7-006477b3e2-1630-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:53:54 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:53:54.345 DEBUG:boto:b'pklxkd-59obj511000falseobj512023-05-31T20:43:08.011Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:53:54.346 DEBUG:boto:path=/pklxkd-59/obj51 2023-05-31T20:53:54.346 DEBUG:boto:auth_path=/pklxkd-59/obj51 2023-05-31T20:53:54.347 DEBUG:boto:Method: HEAD 2023-05-31T20:53:54.347 DEBUG:boto:Path: /pklxkd-59/obj51 2023-05-31T20:53:54.347 DEBUG:boto:Data: 2023-05-31T20:53:54.348 DEBUG:boto:Headers: {} 2023-05-31T20:53:54.348 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:53:54.348 DEBUG:boto:Port: 8001 2023-05-31T20:53:54.349 DEBUG:boto:Params: {} 2023-05-31T20:53:54.349 DEBUG:boto:Token: None 2023-05-31T20:53:54.349 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:53:54 GMT /pklxkd-59/obj51 2023-05-31T20:53:54.350 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:coDygZwT9VDgHkz81wmjWoSbfD8= 2023-05-31T20:53:54.350 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:53:54 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:coDygZwT9VDgHkz81wmjWoSbfD8=', 'Content-Length': '0'} 2023-05-31T20:53:54.352 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:43:08 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000044691a230ec2c3e2-006477b3e2-1c17-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:53:54 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:53:54.353 DEBUG:boto:path=/pklxkd-59/obj51 2023-05-31T20:53:54.353 DEBUG:boto:auth_path=/pklxkd-59/obj51 2023-05-31T20:53:54.353 DEBUG:boto:Method: HEAD 2023-05-31T20:53:54.354 DEBUG:boto:Path: /pklxkd-59/obj51 2023-05-31T20:53:54.354 DEBUG:boto:Data: 2023-05-31T20:53:54.354 DEBUG:boto:Headers: {} 2023-05-31T20:53:54.355 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:53:54.355 DEBUG:boto:Port: 8000 2023-05-31T20:53:54.355 DEBUG:boto:Params: {} 2023-05-31T20:53:54.356 DEBUG:boto:Token: None 2023-05-31T20:53:54.356 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:53:54 GMT /pklxkd-59/obj51 2023-05-31T20:53:54.356 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:coDygZwT9VDgHkz81wmjWoSbfD8= 2023-05-31T20:53:54.357 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:53:54 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:coDygZwT9VDgHkz81wmjWoSbfD8=', 'Content-Length': '0'} 2023-05-31T20:53:54.359 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:43:08 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000104829e9224eaab3-006477b3e2-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:53:54 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:53:54.359 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj51 2023-05-31T20:53:54.360 DEBUG:boto:path=/pklxkd-59/obj51 2023-05-31T20:53:54.360 DEBUG:boto:auth_path=/pklxkd-59/obj51 2023-05-31T20:53:54.360 DEBUG:boto:Method: GET 2023-05-31T20:53:54.361 DEBUG:boto:Path: /pklxkd-59/obj51 2023-05-31T20:53:54.361 DEBUG:boto:Data: 2023-05-31T20:53:54.362 DEBUG:boto:Headers: {} 2023-05-31T20:53:54.362 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:53:54.362 DEBUG:boto:Port: 8001 2023-05-31T20:53:54.363 DEBUG:boto:Params: {} 2023-05-31T20:53:54.363 DEBUG:boto:Token: None 2023-05-31T20:53:54.363 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:53:54 GMT /pklxkd-59/obj51 2023-05-31T20:53:54.364 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:6B5eLFcz/sKym3YQgkDM+L/UCj0= 2023-05-31T20:53:54.364 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:53:54 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:6B5eLFcz/sKym3YQgkDM+L/UCj0=', 'Content-Length': '0'} 2023-05-31T20:53:54.366 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:43:08 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f5d5ad0c67553969-006477b3e2-1c17-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:53:54 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:53:54.367 DEBUG:boto:path=/pklxkd-59/obj51 2023-05-31T20:53:54.367 DEBUG:boto:auth_path=/pklxkd-59/obj51 2023-05-31T20:53:54.368 DEBUG:boto:Method: GET 2023-05-31T20:53:54.368 DEBUG:boto:Path: /pklxkd-59/obj51 2023-05-31T20:53:54.368 DEBUG:boto:Data: 2023-05-31T20:53:54.369 DEBUG:boto:Headers: {} 2023-05-31T20:53:54.369 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:53:54.369 DEBUG:boto:Port: 8000 2023-05-31T20:53:54.370 DEBUG:boto:Params: {} 2023-05-31T20:53:54.370 DEBUG:boto:Token: None 2023-05-31T20:53:54.370 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:53:54 GMT /pklxkd-59/obj51 2023-05-31T20:53:54.371 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:6B5eLFcz/sKym3YQgkDM+L/UCj0= 2023-05-31T20:53:54.371 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:53:54 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:6B5eLFcz/sKym3YQgkDM+L/UCj0=', 'Content-Length': '0'} 2023-05-31T20:53:54.373 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:43:08 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003e5a48cd395af728-006477b3e2-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:53:54 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:53:54.374 DEBUG:boto:path=/pklxkd-59/ 2023-05-31T20:53:54.374 DEBUG:boto:auth_path=/pklxkd-59/ 2023-05-31T20:53:54.375 DEBUG:boto:path=/pklxkd-59/?prefix=obj51 2023-05-31T20:53:54.375 DEBUG:boto:auth_path=/pklxkd-59/?prefix=obj51 2023-05-31T20:53:54.375 DEBUG:boto:Method: GET 2023-05-31T20:53:54.376 DEBUG:boto:Path: /pklxkd-59/?prefix=obj51 2023-05-31T20:53:54.376 DEBUG:boto:Data: 2023-05-31T20:53:54.377 DEBUG:boto:Headers: {} 2023-05-31T20:53:54.377 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:53:54.377 DEBUG:boto:Port: 8001 2023-05-31T20:53:54.378 DEBUG:boto:Params: {} 2023-05-31T20:53:54.378 DEBUG:boto:Token: None 2023-05-31T20:53:54.378 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:53:54 GMT /pklxkd-59/ 2023-05-31T20:53:54.379 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:X6lj3OWBH37r6jmqQli80RnWHOg= 2023-05-31T20:53:54.379 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:53:54 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:X6lj3OWBH37r6jmqQli80RnWHOg=', 'Content-Length': '0'} 2023-05-31T20:53:54.382 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000000e572dbac199f7c4-006477b3e2-1c17-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:53:54 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:53:54.383 DEBUG:boto:b'pklxkd-59obj511000falseobj512023-05-31T20:43:08.011Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:53:54.383 DEBUG:boto:path=/pklxkd-59/ 2023-05-31T20:53:54.384 DEBUG:boto:auth_path=/pklxkd-59/ 2023-05-31T20:53:54.384 DEBUG:boto:path=/pklxkd-59/?prefix=obj51 2023-05-31T20:53:54.384 DEBUG:boto:auth_path=/pklxkd-59/?prefix=obj51 2023-05-31T20:53:54.385 DEBUG:boto:Method: GET 2023-05-31T20:53:54.385 DEBUG:boto:Path: /pklxkd-59/?prefix=obj51 2023-05-31T20:53:54.385 DEBUG:boto:Data: 2023-05-31T20:53:54.386 DEBUG:boto:Headers: {} 2023-05-31T20:53:54.386 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:53:54.386 DEBUG:boto:Port: 8000 2023-05-31T20:53:54.387 DEBUG:boto:Params: {} 2023-05-31T20:53:54.387 DEBUG:boto:Token: None 2023-05-31T20:53:54.388 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:53:54 GMT /pklxkd-59/ 2023-05-31T20:53:54.388 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:X6lj3OWBH37r6jmqQli80RnWHOg= 2023-05-31T20:53:54.388 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:53:54 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:X6lj3OWBH37r6jmqQli80RnWHOg=', 'Content-Length': '0'} 2023-05-31T20:53:54.393 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000686ef909f985ce41-006477b3e2-1630-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:53:54 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:53:54.393 DEBUG:boto:b'pklxkd-59obj511000falseobj512023-05-31T20:43:08.011Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:53:54.394 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj55 2023-05-31T20:53:54.394 DEBUG:boto:path=/pklxkd-59/obj55 2023-05-31T20:53:54.395 DEBUG:boto:auth_path=/pklxkd-59/obj55 2023-05-31T20:53:54.395 DEBUG:boto:path=/pklxkd-59/obj55?versionId=null 2023-05-31T20:53:54.395 DEBUG:boto:auth_path=/pklxkd-59/obj55?versionId=null 2023-05-31T20:53:54.396 DEBUG:boto:Method: GET 2023-05-31T20:53:54.396 DEBUG:boto:Path: /pklxkd-59/obj55?versionId=null 2023-05-31T20:53:54.397 DEBUG:boto:Data: 2023-05-31T20:53:54.397 DEBUG:boto:Headers: {} 2023-05-31T20:53:54.397 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:53:54.398 DEBUG:boto:Port: 8001 2023-05-31T20:53:54.398 DEBUG:boto:Params: {} 2023-05-31T20:53:54.398 DEBUG:boto:Token: None 2023-05-31T20:53:54.399 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:53:54 GMT /pklxkd-59/obj55?versionId=null 2023-05-31T20:53:54.399 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:6t2Zri7w99fCdy9AhGCHp2ATagY= 2023-05-31T20:53:54.400 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:53:54 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:6t2Zri7w99fCdy9AhGCHp2ATagY=', 'Content-Length': '0'} 2023-05-31T20:53:54.401 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:39:26 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f0e82ebef30d0974-006477b3e2-1c17-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:53:54 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:53:54.402 DEBUG:boto:path=/pklxkd-59/obj55 2023-05-31T20:53:54.402 DEBUG:boto:auth_path=/pklxkd-59/obj55 2023-05-31T20:53:54.403 DEBUG:boto:path=/pklxkd-59/obj55?versionId=null 2023-05-31T20:53:54.403 DEBUG:boto:auth_path=/pklxkd-59/obj55?versionId=null 2023-05-31T20:53:54.403 DEBUG:boto:Method: GET 2023-05-31T20:53:54.404 DEBUG:boto:Path: /pklxkd-59/obj55?versionId=null 2023-05-31T20:53:54.404 DEBUG:boto:Data: 2023-05-31T20:53:54.405 DEBUG:boto:Headers: {} 2023-05-31T20:53:54.405 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:53:54.405 DEBUG:boto:Port: 8000 2023-05-31T20:53:54.406 DEBUG:boto:Params: {} 2023-05-31T20:53:54.406 DEBUG:boto:Token: None 2023-05-31T20:53:54.406 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:53:54 GMT /pklxkd-59/obj55?versionId=null 2023-05-31T20:53:54.407 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:6t2Zri7w99fCdy9AhGCHp2ATagY= 2023-05-31T20:53:54.407 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:53:54 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:6t2Zri7w99fCdy9AhGCHp2ATagY=', 'Content-Length': '0'} 2023-05-31T20:53:54.409 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:39:26 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000894faee0f788086d-006477b3e2-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:53:54 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:53:54.452 DEBUG:boto:path=/pklxkd-59/obj55 2023-05-31T20:53:54.453 DEBUG:boto:auth_path=/pklxkd-59/obj55 2023-05-31T20:53:54.453 DEBUG:boto:path=/pklxkd-59/obj55?versionId=null 2023-05-31T20:53:54.453 DEBUG:boto:auth_path=/pklxkd-59/obj55?versionId=null 2023-05-31T20:53:54.454 DEBUG:boto:Method: HEAD 2023-05-31T20:53:54.454 DEBUG:boto:Path: /pklxkd-59/obj55?versionId=null 2023-05-31T20:53:54.454 DEBUG:boto:Data: 2023-05-31T20:53:54.455 DEBUG:boto:Headers: {} 2023-05-31T20:53:54.455 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:53:54.455 DEBUG:boto:Port: 8001 2023-05-31T20:53:54.456 DEBUG:boto:Params: {} 2023-05-31T20:53:54.456 DEBUG:boto:Token: None 2023-05-31T20:53:54.456 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:53:54 GMT /pklxkd-59/obj55?versionId=null 2023-05-31T20:53:54.457 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:/Ov7Pey0qtS0wGlu7r4S741dofM= 2023-05-31T20:53:54.457 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:53:54 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:/Ov7Pey0qtS0wGlu7r4S741dofM=', 'Content-Length': '0'} 2023-05-31T20:53:54.459 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:39:26 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f9e02ce9f1cc8c15-006477b3e2-1c17-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:53:54 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:53:54.459 DEBUG:boto:path=/pklxkd-59/obj55 2023-05-31T20:53:54.460 DEBUG:boto:auth_path=/pklxkd-59/obj55 2023-05-31T20:53:54.460 DEBUG:boto:path=/pklxkd-59/obj55?versionId=null 2023-05-31T20:53:54.461 DEBUG:boto:auth_path=/pklxkd-59/obj55?versionId=null 2023-05-31T20:53:54.461 DEBUG:boto:Method: HEAD 2023-05-31T20:53:54.461 DEBUG:boto:Path: /pklxkd-59/obj55?versionId=null 2023-05-31T20:53:54.462 DEBUG:boto:Data: 2023-05-31T20:53:54.462 DEBUG:boto:Headers: {} 2023-05-31T20:53:54.462 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:53:54.463 DEBUG:boto:Port: 8000 2023-05-31T20:53:54.463 DEBUG:boto:Params: {} 2023-05-31T20:53:54.463 DEBUG:boto:Token: None 2023-05-31T20:53:54.464 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:53:54 GMT /pklxkd-59/obj55?versionId=null 2023-05-31T20:53:54.464 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:/Ov7Pey0qtS0wGlu7r4S741dofM= 2023-05-31T20:53:54.464 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:53:54 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:/Ov7Pey0qtS0wGlu7r4S741dofM=', 'Content-Length': '0'} 2023-05-31T20:53:54.466 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:39:26 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004ffe6fa0d2ad0030-006477b3e2-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:53:54 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:53:54.467 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj55 2023-05-31T20:53:54.467 DEBUG:boto:path=/pklxkd-59/obj55 2023-05-31T20:53:54.468 DEBUG:boto:auth_path=/pklxkd-59/obj55 2023-05-31T20:53:54.468 DEBUG:boto:path=/pklxkd-59/obj55?versionId=null 2023-05-31T20:53:54.468 DEBUG:boto:auth_path=/pklxkd-59/obj55?versionId=null 2023-05-31T20:53:54.469 DEBUG:boto:Method: GET 2023-05-31T20:53:54.469 DEBUG:boto:Path: /pklxkd-59/obj55?versionId=null 2023-05-31T20:53:54.469 DEBUG:boto:Data: 2023-05-31T20:53:54.470 DEBUG:boto:Headers: {} 2023-05-31T20:53:54.470 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:53:54.470 DEBUG:boto:Port: 8001 2023-05-31T20:53:54.471 DEBUG:boto:Params: {} 2023-05-31T20:53:54.471 DEBUG:boto:Token: None 2023-05-31T20:53:54.472 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:53:54 GMT /pklxkd-59/obj55?versionId=null 2023-05-31T20:53:54.472 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:6t2Zri7w99fCdy9AhGCHp2ATagY= 2023-05-31T20:53:54.472 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:53:54 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:6t2Zri7w99fCdy9AhGCHp2ATagY=', 'Content-Length': '0'} 2023-05-31T20:53:54.474 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:39:26 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000083f5c1fbe7501818-006477b3e2-1c17-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:53:54 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:53:54.475 DEBUG:boto:path=/pklxkd-59/obj55 2023-05-31T20:53:54.475 DEBUG:boto:auth_path=/pklxkd-59/obj55 2023-05-31T20:53:54.476 DEBUG:boto:path=/pklxkd-59/obj55?versionId=null 2023-05-31T20:53:54.476 DEBUG:boto:auth_path=/pklxkd-59/obj55?versionId=null 2023-05-31T20:53:54.476 DEBUG:boto:Method: GET 2023-05-31T20:53:54.477 DEBUG:boto:Path: /pklxkd-59/obj55?versionId=null 2023-05-31T20:53:54.477 DEBUG:boto:Data: 2023-05-31T20:53:54.478 DEBUG:boto:Headers: {} 2023-05-31T20:53:54.478 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:53:54.478 DEBUG:boto:Port: 8000 2023-05-31T20:53:54.479 DEBUG:boto:Params: {} 2023-05-31T20:53:54.479 DEBUG:boto:Token: None 2023-05-31T20:53:54.479 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:53:54 GMT /pklxkd-59/obj55?versionId=null 2023-05-31T20:53:54.480 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:6t2Zri7w99fCdy9AhGCHp2ATagY= 2023-05-31T20:53:54.480 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:53:54 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:6t2Zri7w99fCdy9AhGCHp2ATagY=', 'Content-Length': '0'} 2023-05-31T20:53:54.482 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:39:26 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a1c61be3a38c2d63-006477b3e2-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:53:54 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:53:54.483 DEBUG:boto:path=/pklxkd-59/ 2023-05-31T20:53:54.483 DEBUG:boto:auth_path=/pklxkd-59/ 2023-05-31T20:53:54.483 DEBUG:boto:path=/pklxkd-59/?prefix=obj55 2023-05-31T20:53:54.484 DEBUG:boto:auth_path=/pklxkd-59/?prefix=obj55 2023-05-31T20:53:54.484 DEBUG:boto:Method: GET 2023-05-31T20:53:54.485 DEBUG:boto:Path: /pklxkd-59/?prefix=obj55 2023-05-31T20:53:54.485 DEBUG:boto:Data: 2023-05-31T20:53:54.485 DEBUG:boto:Headers: {} 2023-05-31T20:53:54.486 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:53:54.486 DEBUG:boto:Port: 8001 2023-05-31T20:53:54.486 DEBUG:boto:Params: {} 2023-05-31T20:53:54.487 DEBUG:boto:Token: None 2023-05-31T20:53:54.487 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:53:54 GMT /pklxkd-59/ 2023-05-31T20:53:54.488 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:X6lj3OWBH37r6jmqQli80RnWHOg= 2023-05-31T20:53:54.488 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:53:54 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:X6lj3OWBH37r6jmqQli80RnWHOg=', 'Content-Length': '0'} 2023-05-31T20:53:54.490 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000002974717511b858a3-006477b3e2-1c17-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:53:54 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:53:54.491 DEBUG:boto:b'pklxkd-59obj551000falseobj552023-05-31T20:39:26.630Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:53:54.491 DEBUG:boto:path=/pklxkd-59/ 2023-05-31T20:53:54.492 DEBUG:boto:auth_path=/pklxkd-59/ 2023-05-31T20:53:54.492 DEBUG:boto:path=/pklxkd-59/?prefix=obj55 2023-05-31T20:53:54.493 DEBUG:boto:auth_path=/pklxkd-59/?prefix=obj55 2023-05-31T20:53:54.493 DEBUG:boto:Method: GET 2023-05-31T20:53:54.493 DEBUG:boto:Path: /pklxkd-59/?prefix=obj55 2023-05-31T20:53:54.494 DEBUG:boto:Data: 2023-05-31T20:53:54.494 DEBUG:boto:Headers: {} 2023-05-31T20:53:54.494 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:53:54.495 DEBUG:boto:Port: 8000 2023-05-31T20:53:54.495 DEBUG:boto:Params: {} 2023-05-31T20:53:54.495 DEBUG:boto:Token: None 2023-05-31T20:53:54.496 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:53:54 GMT /pklxkd-59/ 2023-05-31T20:53:54.496 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:X6lj3OWBH37r6jmqQli80RnWHOg= 2023-05-31T20:53:54.497 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:53:54 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:X6lj3OWBH37r6jmqQli80RnWHOg=', 'Content-Length': '0'} 2023-05-31T20:53:54.501 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ad8216ab3936cd97-006477b3e2-1630-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:53:54 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:53:54.501 DEBUG:boto:b'pklxkd-59obj551000falseobj552023-05-31T20:39:26.630Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:53:54.502 DEBUG:boto:path=/pklxkd-59/obj55 2023-05-31T20:53:54.502 DEBUG:boto:auth_path=/pklxkd-59/obj55 2023-05-31T20:53:54.503 DEBUG:boto:Method: HEAD 2023-05-31T20:53:54.503 DEBUG:boto:Path: /pklxkd-59/obj55 2023-05-31T20:53:54.503 DEBUG:boto:Data: 2023-05-31T20:53:54.504 DEBUG:boto:Headers: {} 2023-05-31T20:53:54.504 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:53:54.504 DEBUG:boto:Port: 8001 2023-05-31T20:53:54.505 DEBUG:boto:Params: {} 2023-05-31T20:53:54.505 DEBUG:boto:Token: None 2023-05-31T20:53:54.505 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:53:54 GMT /pklxkd-59/obj55 2023-05-31T20:53:54.506 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:Vmj7tmdO62B5dLrTcwYDQ00GgWg= 2023-05-31T20:53:54.506 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:53:54 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:Vmj7tmdO62B5dLrTcwYDQ00GgWg=', 'Content-Length': '0'} 2023-05-31T20:53:54.508 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:39:26 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000060c9e4424a2b6c99-006477b3e2-1c17-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:53:54 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:53:54.508 DEBUG:boto:path=/pklxkd-59/obj55 2023-05-31T20:53:54.509 DEBUG:boto:auth_path=/pklxkd-59/obj55 2023-05-31T20:53:54.509 DEBUG:boto:Method: HEAD 2023-05-31T20:53:54.509 DEBUG:boto:Path: /pklxkd-59/obj55 2023-05-31T20:53:54.510 DEBUG:boto:Data: 2023-05-31T20:53:54.510 DEBUG:boto:Headers: {} 2023-05-31T20:53:54.510 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:53:54.511 DEBUG:boto:Port: 8000 2023-05-31T20:53:54.511 DEBUG:boto:Params: {} 2023-05-31T20:53:54.511 DEBUG:boto:Token: None 2023-05-31T20:53:54.512 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:53:54 GMT /pklxkd-59/obj55 2023-05-31T20:53:54.512 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:Vmj7tmdO62B5dLrTcwYDQ00GgWg= 2023-05-31T20:53:54.513 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:53:54 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:Vmj7tmdO62B5dLrTcwYDQ00GgWg=', 'Content-Length': '0'} 2023-05-31T20:53:54.514 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:39:26 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d8087b2149792a44-006477b3e2-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:53:54 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:53:54.515 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj55 2023-05-31T20:53:54.515 DEBUG:boto:path=/pklxkd-59/obj55 2023-05-31T20:53:54.516 DEBUG:boto:auth_path=/pklxkd-59/obj55 2023-05-31T20:53:54.516 DEBUG:boto:Method: GET 2023-05-31T20:53:54.517 DEBUG:boto:Path: /pklxkd-59/obj55 2023-05-31T20:53:54.517 DEBUG:boto:Data: 2023-05-31T20:53:54.517 DEBUG:boto:Headers: {} 2023-05-31T20:53:54.518 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:53:54.518 DEBUG:boto:Port: 8001 2023-05-31T20:53:54.518 DEBUG:boto:Params: {} 2023-05-31T20:53:54.519 DEBUG:boto:Token: None 2023-05-31T20:53:54.519 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:53:54 GMT /pklxkd-59/obj55 2023-05-31T20:53:54.520 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:OlgKibBWq9jNEjXqm7MLwAqiuaA= 2023-05-31T20:53:54.520 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:53:54 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:OlgKibBWq9jNEjXqm7MLwAqiuaA=', 'Content-Length': '0'} 2023-05-31T20:53:54.522 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:39:26 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000081e9335a62b236cf-006477b3e2-1c17-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:53:54 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:53:54.522 DEBUG:boto:path=/pklxkd-59/obj55 2023-05-31T20:53:54.523 DEBUG:boto:auth_path=/pklxkd-59/obj55 2023-05-31T20:53:54.523 DEBUG:boto:Method: GET 2023-05-31T20:53:54.523 DEBUG:boto:Path: /pklxkd-59/obj55 2023-05-31T20:53:54.524 DEBUG:boto:Data: 2023-05-31T20:53:54.524 DEBUG:boto:Headers: {} 2023-05-31T20:53:54.525 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:53:54.525 DEBUG:boto:Port: 8000 2023-05-31T20:53:54.525 DEBUG:boto:Params: {} 2023-05-31T20:53:54.526 DEBUG:boto:Token: None 2023-05-31T20:53:54.526 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:53:54 GMT /pklxkd-59/obj55 2023-05-31T20:53:54.526 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:OlgKibBWq9jNEjXqm7MLwAqiuaA= 2023-05-31T20:53:54.527 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:53:54 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:OlgKibBWq9jNEjXqm7MLwAqiuaA=', 'Content-Length': '0'} 2023-05-31T20:53:54.529 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:39:26 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001029ebd81fce06aa-006477b3e2-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:53:54 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:53:54.529 DEBUG:boto:path=/pklxkd-59/ 2023-05-31T20:53:54.530 DEBUG:boto:auth_path=/pklxkd-59/ 2023-05-31T20:53:54.530 DEBUG:boto:path=/pklxkd-59/?prefix=obj55 2023-05-31T20:53:54.530 DEBUG:boto:auth_path=/pklxkd-59/?prefix=obj55 2023-05-31T20:53:54.531 DEBUG:boto:Method: GET 2023-05-31T20:53:54.531 DEBUG:boto:Path: /pklxkd-59/?prefix=obj55 2023-05-31T20:53:54.532 DEBUG:boto:Data: 2023-05-31T20:53:54.532 DEBUG:boto:Headers: {} 2023-05-31T20:53:54.532 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:53:54.533 DEBUG:boto:Port: 8001 2023-05-31T20:53:54.533 DEBUG:boto:Params: {} 2023-05-31T20:53:54.533 DEBUG:boto:Token: None 2023-05-31T20:53:54.534 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:53:54 GMT /pklxkd-59/ 2023-05-31T20:53:54.534 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:X6lj3OWBH37r6jmqQli80RnWHOg= 2023-05-31T20:53:54.535 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:53:54 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:X6lj3OWBH37r6jmqQli80RnWHOg=', 'Content-Length': '0'} 2023-05-31T20:53:54.537 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000007aa5cfe04d4fbe71-006477b3e2-1c17-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:53:54 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:53:54.537 DEBUG:boto:b'pklxkd-59obj551000falseobj552023-05-31T20:39:26.630Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:53:54.538 DEBUG:boto:path=/pklxkd-59/ 2023-05-31T20:53:54.538 DEBUG:boto:auth_path=/pklxkd-59/ 2023-05-31T20:53:54.539 DEBUG:boto:path=/pklxkd-59/?prefix=obj55 2023-05-31T20:53:54.539 DEBUG:boto:auth_path=/pklxkd-59/?prefix=obj55 2023-05-31T20:53:54.539 DEBUG:boto:Method: GET 2023-05-31T20:53:54.540 DEBUG:boto:Path: /pklxkd-59/?prefix=obj55 2023-05-31T20:53:54.540 DEBUG:boto:Data: 2023-05-31T20:53:54.540 DEBUG:boto:Headers: {} 2023-05-31T20:53:54.541 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:53:54.541 DEBUG:boto:Port: 8000 2023-05-31T20:53:54.541 DEBUG:boto:Params: {} 2023-05-31T20:53:54.542 DEBUG:boto:Token: None 2023-05-31T20:53:54.542 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:53:54 GMT /pklxkd-59/ 2023-05-31T20:53:54.543 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:X6lj3OWBH37r6jmqQli80RnWHOg= 2023-05-31T20:53:54.543 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:53:54 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:X6lj3OWBH37r6jmqQli80RnWHOg=', 'Content-Length': '0'} 2023-05-31T20:53:54.547 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000bc099e81c6e725a5-006477b3e2-1630-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:53:54 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:53:54.588 DEBUG:boto:b'pklxkd-59obj551000falseobj552023-05-31T20:39:26.630Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:53:54.589 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj57 2023-05-31T20:53:54.589 DEBUG:boto:path=/pklxkd-59/obj57 2023-05-31T20:53:54.590 DEBUG:boto:auth_path=/pklxkd-59/obj57 2023-05-31T20:53:54.590 DEBUG:boto:path=/pklxkd-59/obj57?versionId=null 2023-05-31T20:53:54.591 DEBUG:boto:auth_path=/pklxkd-59/obj57?versionId=null 2023-05-31T20:53:54.591 DEBUG:boto:Method: GET 2023-05-31T20:53:54.591 DEBUG:boto:Path: /pklxkd-59/obj57?versionId=null 2023-05-31T20:53:54.592 DEBUG:boto:Data: 2023-05-31T20:53:54.592 DEBUG:boto:Headers: {} 2023-05-31T20:53:54.592 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:53:54.593 DEBUG:boto:Port: 8001 2023-05-31T20:53:54.593 DEBUG:boto:Params: {} 2023-05-31T20:53:54.594 DEBUG:boto:Token: None 2023-05-31T20:53:54.594 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:53:54 GMT /pklxkd-59/obj57?versionId=null 2023-05-31T20:53:54.594 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:Orv+SIb71ogowQk/GUPr2ywTflQ= 2023-05-31T20:53:54.595 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:53:54 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:Orv+SIb71ogowQk/GUPr2ywTflQ=', 'Content-Length': '0'} 2023-05-31T20:53:54.597 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:44:50 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007ba0f35b190ff201-006477b3e2-1c17-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:53:54 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:53:54.598 DEBUG:boto:path=/pklxkd-59/obj57 2023-05-31T20:53:54.598 DEBUG:boto:auth_path=/pklxkd-59/obj57 2023-05-31T20:53:54.598 DEBUG:boto:path=/pklxkd-59/obj57?versionId=null 2023-05-31T20:53:54.599 DEBUG:boto:auth_path=/pklxkd-59/obj57?versionId=null 2023-05-31T20:53:54.599 DEBUG:boto:Method: GET 2023-05-31T20:53:54.599 DEBUG:boto:Path: /pklxkd-59/obj57?versionId=null 2023-05-31T20:53:54.600 DEBUG:boto:Data: 2023-05-31T20:53:54.600 DEBUG:boto:Headers: {} 2023-05-31T20:53:54.600 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:53:54.601 DEBUG:boto:Port: 8000 2023-05-31T20:53:54.601 DEBUG:boto:Params: {} 2023-05-31T20:53:54.602 DEBUG:boto:Token: None 2023-05-31T20:53:54.602 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:53:54 GMT /pklxkd-59/obj57?versionId=null 2023-05-31T20:53:54.602 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:Orv+SIb71ogowQk/GUPr2ywTflQ= 2023-05-31T20:53:54.603 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:53:54 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:Orv+SIb71ogowQk/GUPr2ywTflQ=', 'Content-Length': '0'} 2023-05-31T20:53:54.605 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:44:50 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a1ada4669754ac28-006477b3e2-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:53:54 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:53:54.606 DEBUG:boto:path=/pklxkd-59/obj57 2023-05-31T20:53:54.606 DEBUG:boto:auth_path=/pklxkd-59/obj57 2023-05-31T20:53:54.606 DEBUG:boto:path=/pklxkd-59/obj57?versionId=null 2023-05-31T20:53:54.607 DEBUG:boto:auth_path=/pklxkd-59/obj57?versionId=null 2023-05-31T20:53:54.607 DEBUG:boto:Method: HEAD 2023-05-31T20:53:54.607 DEBUG:boto:Path: /pklxkd-59/obj57?versionId=null 2023-05-31T20:53:54.608 DEBUG:boto:Data: 2023-05-31T20:53:54.608 DEBUG:boto:Headers: {} 2023-05-31T20:53:54.608 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:53:54.609 DEBUG:boto:Port: 8001 2023-05-31T20:53:54.609 DEBUG:boto:Params: {} 2023-05-31T20:53:54.610 DEBUG:boto:Token: None 2023-05-31T20:53:54.610 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:53:54 GMT /pklxkd-59/obj57?versionId=null 2023-05-31T20:53:54.610 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:kiQYs5HStnE0+YRdIjvwSro8mGE= 2023-05-31T20:53:54.611 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:53:54 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:kiQYs5HStnE0+YRdIjvwSro8mGE=', 'Content-Length': '0'} 2023-05-31T20:53:54.612 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:44:50 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c767d1cac460a10f-006477b3e2-1c17-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:53:54 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:53:54.613 DEBUG:boto:path=/pklxkd-59/obj57 2023-05-31T20:53:54.613 DEBUG:boto:auth_path=/pklxkd-59/obj57 2023-05-31T20:53:54.614 DEBUG:boto:path=/pklxkd-59/obj57?versionId=null 2023-05-31T20:53:54.614 DEBUG:boto:auth_path=/pklxkd-59/obj57?versionId=null 2023-05-31T20:53:54.614 DEBUG:boto:Method: HEAD 2023-05-31T20:53:54.615 DEBUG:boto:Path: /pklxkd-59/obj57?versionId=null 2023-05-31T20:53:54.615 DEBUG:boto:Data: 2023-05-31T20:53:54.615 DEBUG:boto:Headers: {} 2023-05-31T20:53:54.616 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:53:54.616 DEBUG:boto:Port: 8000 2023-05-31T20:53:54.616 DEBUG:boto:Params: {} 2023-05-31T20:53:54.617 DEBUG:boto:Token: None 2023-05-31T20:53:54.617 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:53:54 GMT /pklxkd-59/obj57?versionId=null 2023-05-31T20:53:54.618 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:kiQYs5HStnE0+YRdIjvwSro8mGE= 2023-05-31T20:53:54.618 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:53:54 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:kiQYs5HStnE0+YRdIjvwSro8mGE=', 'Content-Length': '0'} 2023-05-31T20:53:54.620 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:44:50 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e6240063f29f071b-006477b3e2-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:53:54 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:53:54.620 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj57 2023-05-31T20:53:54.621 DEBUG:boto:path=/pklxkd-59/obj57 2023-05-31T20:53:54.621 DEBUG:boto:auth_path=/pklxkd-59/obj57 2023-05-31T20:53:54.621 DEBUG:boto:path=/pklxkd-59/obj57?versionId=null 2023-05-31T20:53:54.622 DEBUG:boto:auth_path=/pklxkd-59/obj57?versionId=null 2023-05-31T20:53:54.622 DEBUG:boto:Method: GET 2023-05-31T20:53:54.622 DEBUG:boto:Path: /pklxkd-59/obj57?versionId=null 2023-05-31T20:53:54.623 DEBUG:boto:Data: 2023-05-31T20:53:54.623 DEBUG:boto:Headers: {} 2023-05-31T20:53:54.623 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:53:54.624 DEBUG:boto:Port: 8001 2023-05-31T20:53:54.624 DEBUG:boto:Params: {} 2023-05-31T20:53:54.625 DEBUG:boto:Token: None 2023-05-31T20:53:54.625 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:53:54 GMT /pklxkd-59/obj57?versionId=null 2023-05-31T20:53:54.625 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:Orv+SIb71ogowQk/GUPr2ywTflQ= 2023-05-31T20:53:54.626 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:53:54 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:Orv+SIb71ogowQk/GUPr2ywTflQ=', 'Content-Length': '0'} 2023-05-31T20:53:54.628 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:44:50 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000239bf319987af9b1-006477b3e2-1c17-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:53:54 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:53:54.668 DEBUG:boto:path=/pklxkd-59/obj57 2023-05-31T20:53:54.669 DEBUG:boto:auth_path=/pklxkd-59/obj57 2023-05-31T20:53:54.669 DEBUG:boto:path=/pklxkd-59/obj57?versionId=null 2023-05-31T20:53:54.669 DEBUG:boto:auth_path=/pklxkd-59/obj57?versionId=null 2023-05-31T20:53:54.670 DEBUG:boto:Method: GET 2023-05-31T20:53:54.670 DEBUG:boto:Path: /pklxkd-59/obj57?versionId=null 2023-05-31T20:53:54.671 DEBUG:boto:Data: 2023-05-31T20:53:54.671 DEBUG:boto:Headers: {} 2023-05-31T20:53:54.671 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:53:54.672 DEBUG:boto:Port: 8000 2023-05-31T20:53:54.672 DEBUG:boto:Params: {} 2023-05-31T20:53:54.673 DEBUG:boto:Token: None 2023-05-31T20:53:54.673 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:53:54 GMT /pklxkd-59/obj57?versionId=null 2023-05-31T20:53:54.673 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:Orv+SIb71ogowQk/GUPr2ywTflQ= 2023-05-31T20:53:54.674 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:53:54 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:Orv+SIb71ogowQk/GUPr2ywTflQ=', 'Content-Length': '0'} 2023-05-31T20:53:54.676 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:44:50 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b25e709362eac949-006477b3e2-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:53:54 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:53:54.677 DEBUG:boto:path=/pklxkd-59/ 2023-05-31T20:53:54.677 DEBUG:boto:auth_path=/pklxkd-59/ 2023-05-31T20:53:54.678 DEBUG:boto:path=/pklxkd-59/?prefix=obj57 2023-05-31T20:53:54.678 DEBUG:boto:auth_path=/pklxkd-59/?prefix=obj57 2023-05-31T20:53:54.678 DEBUG:boto:Method: GET 2023-05-31T20:53:54.679 DEBUG:boto:Path: /pklxkd-59/?prefix=obj57 2023-05-31T20:53:54.679 DEBUG:boto:Data: 2023-05-31T20:53:54.679 DEBUG:boto:Headers: {} 2023-05-31T20:53:54.680 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:53:54.680 DEBUG:boto:Port: 8001 2023-05-31T20:53:54.680 DEBUG:boto:Params: {} 2023-05-31T20:53:54.681 DEBUG:boto:Token: None 2023-05-31T20:53:54.681 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:53:54 GMT /pklxkd-59/ 2023-05-31T20:53:54.682 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:X6lj3OWBH37r6jmqQli80RnWHOg= 2023-05-31T20:53:54.682 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:53:54 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:X6lj3OWBH37r6jmqQli80RnWHOg=', 'Content-Length': '0'} 2023-05-31T20:53:54.684 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000fa8fee1394729f48-006477b3e2-1c17-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:53:54 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:53:54.685 DEBUG:boto:b'pklxkd-59obj571000falseobj572023-05-31T20:44:50.627Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:53:54.685 DEBUG:boto:path=/pklxkd-59/ 2023-05-31T20:53:54.686 DEBUG:boto:auth_path=/pklxkd-59/ 2023-05-31T20:53:54.686 DEBUG:boto:path=/pklxkd-59/?prefix=obj57 2023-05-31T20:53:54.687 DEBUG:boto:auth_path=/pklxkd-59/?prefix=obj57 2023-05-31T20:53:54.687 DEBUG:boto:Method: GET 2023-05-31T20:53:54.687 DEBUG:boto:Path: /pklxkd-59/?prefix=obj57 2023-05-31T20:53:54.688 DEBUG:boto:Data: 2023-05-31T20:53:54.688 DEBUG:boto:Headers: {} 2023-05-31T20:53:54.688 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:53:54.689 DEBUG:boto:Port: 8000 2023-05-31T20:53:54.689 DEBUG:boto:Params: {} 2023-05-31T20:53:54.689 DEBUG:boto:Token: None 2023-05-31T20:53:54.690 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:53:54 GMT /pklxkd-59/ 2023-05-31T20:53:54.690 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:X6lj3OWBH37r6jmqQli80RnWHOg= 2023-05-31T20:53:54.691 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:53:54 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:X6lj3OWBH37r6jmqQli80RnWHOg=', 'Content-Length': '0'} 2023-05-31T20:53:54.694 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f3aedd816be91806-006477b3e2-1630-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:53:54 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:53:54.736 DEBUG:boto:b'pklxkd-59obj571000falseobj572023-05-31T20:44:50.627Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:53:54.737 DEBUG:boto:path=/pklxkd-59/obj57 2023-05-31T20:53:54.737 DEBUG:boto:auth_path=/pklxkd-59/obj57 2023-05-31T20:53:54.738 DEBUG:boto:Method: HEAD 2023-05-31T20:53:54.738 DEBUG:boto:Path: /pklxkd-59/obj57 2023-05-31T20:53:54.738 DEBUG:boto:Data: 2023-05-31T20:53:54.739 DEBUG:boto:Headers: {} 2023-05-31T20:53:54.739 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:53:54.739 DEBUG:boto:Port: 8001 2023-05-31T20:53:54.740 DEBUG:boto:Params: {} 2023-05-31T20:53:54.740 DEBUG:boto:Token: None 2023-05-31T20:53:54.740 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:53:54 GMT /pklxkd-59/obj57 2023-05-31T20:53:54.741 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:q2B7nuAzcgQZKsrHVk1U+8+lPd0= 2023-05-31T20:53:54.741 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:53:54 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:q2B7nuAzcgQZKsrHVk1U+8+lPd0=', 'Content-Length': '0'} 2023-05-31T20:53:54.743 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:44:50 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004b6e181ca42ae1b4-006477b3e2-1c17-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:53:54 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:53:54.744 DEBUG:boto:path=/pklxkd-59/obj57 2023-05-31T20:53:54.744 DEBUG:boto:auth_path=/pklxkd-59/obj57 2023-05-31T20:53:54.744 DEBUG:boto:Method: HEAD 2023-05-31T20:53:54.745 DEBUG:boto:Path: /pklxkd-59/obj57 2023-05-31T20:53:54.745 DEBUG:boto:Data: 2023-05-31T20:53:54.745 DEBUG:boto:Headers: {} 2023-05-31T20:53:54.746 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:53:54.746 DEBUG:boto:Port: 8000 2023-05-31T20:53:54.746 DEBUG:boto:Params: {} 2023-05-31T20:53:54.747 DEBUG:boto:Token: None 2023-05-31T20:53:54.747 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:53:54 GMT /pklxkd-59/obj57 2023-05-31T20:53:54.747 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:q2B7nuAzcgQZKsrHVk1U+8+lPd0= 2023-05-31T20:53:54.748 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:53:54 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:q2B7nuAzcgQZKsrHVk1U+8+lPd0=', 'Content-Length': '0'} 2023-05-31T20:53:54.750 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:44:50 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a9f8a6025ed5ffc2-006477b3e2-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:53:54 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:53:54.750 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj57 2023-05-31T20:53:54.751 DEBUG:boto:path=/pklxkd-59/obj57 2023-05-31T20:53:54.751 DEBUG:boto:auth_path=/pklxkd-59/obj57 2023-05-31T20:53:54.751 DEBUG:boto:Method: GET 2023-05-31T20:53:54.752 DEBUG:boto:Path: /pklxkd-59/obj57 2023-05-31T20:53:54.752 DEBUG:boto:Data: 2023-05-31T20:53:54.753 DEBUG:boto:Headers: {} 2023-05-31T20:53:54.753 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:53:54.753 DEBUG:boto:Port: 8001 2023-05-31T20:53:54.754 DEBUG:boto:Params: {} 2023-05-31T20:53:54.754 DEBUG:boto:Token: None 2023-05-31T20:53:54.754 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:53:54 GMT /pklxkd-59/obj57 2023-05-31T20:53:54.755 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:11RpYWGxNNl3mNUTMs5xU8sqaTs= 2023-05-31T20:53:54.755 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:53:54 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:11RpYWGxNNl3mNUTMs5xU8sqaTs=', 'Content-Length': '0'} 2023-05-31T20:53:54.757 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:44:50 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e6ec0237855ec98b-006477b3e2-1c17-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:53:54 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:53:54.758 DEBUG:boto:path=/pklxkd-59/obj57 2023-05-31T20:53:54.758 DEBUG:boto:auth_path=/pklxkd-59/obj57 2023-05-31T20:53:54.758 DEBUG:boto:Method: GET 2023-05-31T20:53:54.759 DEBUG:boto:Path: /pklxkd-59/obj57 2023-05-31T20:53:54.759 DEBUG:boto:Data: 2023-05-31T20:53:54.760 DEBUG:boto:Headers: {} 2023-05-31T20:53:54.760 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:53:54.760 DEBUG:boto:Port: 8000 2023-05-31T20:53:54.761 DEBUG:boto:Params: {} 2023-05-31T20:53:54.761 DEBUG:boto:Token: None 2023-05-31T20:53:54.762 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:53:54 GMT /pklxkd-59/obj57 2023-05-31T20:53:54.762 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:11RpYWGxNNl3mNUTMs5xU8sqaTs= 2023-05-31T20:53:54.762 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:53:54 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:11RpYWGxNNl3mNUTMs5xU8sqaTs=', 'Content-Length': '0'} 2023-05-31T20:53:54.764 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:44:50 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a74e368e628927d8-006477b3e2-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:53:54 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:53:54.765 DEBUG:boto:path=/pklxkd-59/ 2023-05-31T20:53:54.765 DEBUG:boto:auth_path=/pklxkd-59/ 2023-05-31T20:53:54.766 DEBUG:boto:path=/pklxkd-59/?prefix=obj57 2023-05-31T20:53:54.766 DEBUG:boto:auth_path=/pklxkd-59/?prefix=obj57 2023-05-31T20:53:54.766 DEBUG:boto:Method: GET 2023-05-31T20:53:54.767 DEBUG:boto:Path: /pklxkd-59/?prefix=obj57 2023-05-31T20:53:54.767 DEBUG:boto:Data: 2023-05-31T20:53:54.767 DEBUG:boto:Headers: {} 2023-05-31T20:53:54.768 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:53:54.768 DEBUG:boto:Port: 8001 2023-05-31T20:53:54.769 DEBUG:boto:Params: {} 2023-05-31T20:53:54.769 DEBUG:boto:Token: None 2023-05-31T20:53:54.769 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:53:54 GMT /pklxkd-59/ 2023-05-31T20:53:54.770 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:X6lj3OWBH37r6jmqQli80RnWHOg= 2023-05-31T20:53:54.770 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:53:54 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:X6lj3OWBH37r6jmqQli80RnWHOg=', 'Content-Length': '0'} 2023-05-31T20:53:54.772 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000093b78ea4a10fcc1c-006477b3e2-1c17-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:53:54 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:53:54.773 DEBUG:boto:b'pklxkd-59obj571000falseobj572023-05-31T20:44:50.627Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:53:54.774 DEBUG:boto:path=/pklxkd-59/ 2023-05-31T20:53:54.774 DEBUG:boto:auth_path=/pklxkd-59/ 2023-05-31T20:53:54.774 DEBUG:boto:path=/pklxkd-59/?prefix=obj57 2023-05-31T20:53:54.775 DEBUG:boto:auth_path=/pklxkd-59/?prefix=obj57 2023-05-31T20:53:54.775 DEBUG:boto:Method: GET 2023-05-31T20:53:54.775 DEBUG:boto:Path: /pklxkd-59/?prefix=obj57 2023-05-31T20:53:54.776 DEBUG:boto:Data: 2023-05-31T20:53:54.776 DEBUG:boto:Headers: {} 2023-05-31T20:53:54.776 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:53:54.777 DEBUG:boto:Port: 8000 2023-05-31T20:53:54.777 DEBUG:boto:Params: {} 2023-05-31T20:53:54.777 DEBUG:boto:Token: None 2023-05-31T20:53:54.778 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:53:54 GMT /pklxkd-59/ 2023-05-31T20:53:54.778 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:X6lj3OWBH37r6jmqQli80RnWHOg= 2023-05-31T20:53:54.779 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:53:54 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:X6lj3OWBH37r6jmqQli80RnWHOg=', 'Content-Length': '0'} 2023-05-31T20:53:54.783 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000009ba668dba5c54ecb-006477b3e2-1630-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:53:54 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:53:54.783 DEBUG:boto:b'pklxkd-59obj571000falseobj572023-05-31T20:44:50.627Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:53:54.784 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj58 2023-05-31T20:53:54.784 DEBUG:boto:path=/pklxkd-59/obj58 2023-05-31T20:53:54.785 DEBUG:boto:auth_path=/pklxkd-59/obj58 2023-05-31T20:53:54.785 DEBUG:boto:path=/pklxkd-59/obj58?versionId=null 2023-05-31T20:53:54.785 DEBUG:boto:auth_path=/pklxkd-59/obj58?versionId=null 2023-05-31T20:53:54.786 DEBUG:boto:Method: GET 2023-05-31T20:53:54.786 DEBUG:boto:Path: /pklxkd-59/obj58?versionId=null 2023-05-31T20:53:54.787 DEBUG:boto:Data: 2023-05-31T20:53:54.787 DEBUG:boto:Headers: {} 2023-05-31T20:53:54.787 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:53:54.788 DEBUG:boto:Port: 8001 2023-05-31T20:53:54.788 DEBUG:boto:Params: {} 2023-05-31T20:53:54.788 DEBUG:boto:Token: None 2023-05-31T20:53:54.789 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:53:54 GMT /pklxkd-59/obj58?versionId=null 2023-05-31T20:53:54.789 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:rUITRzOhr2PcQKPeKu93kL3+zMU= 2023-05-31T20:53:54.790 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:53:54 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:rUITRzOhr2PcQKPeKu93kL3+zMU=', 'Content-Length': '0'} 2023-05-31T20:53:54.792 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:48:12 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f3a948650efe6843-006477b3e2-1c17-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:53:54 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:53:54.832 DEBUG:boto:path=/pklxkd-59/obj58 2023-05-31T20:53:54.833 DEBUG:boto:auth_path=/pklxkd-59/obj58 2023-05-31T20:53:54.833 DEBUG:boto:path=/pklxkd-59/obj58?versionId=null 2023-05-31T20:53:54.833 DEBUG:boto:auth_path=/pklxkd-59/obj58?versionId=null 2023-05-31T20:53:54.834 DEBUG:boto:Method: GET 2023-05-31T20:53:54.834 DEBUG:boto:Path: /pklxkd-59/obj58?versionId=null 2023-05-31T20:53:54.834 DEBUG:boto:Data: 2023-05-31T20:53:54.835 DEBUG:boto:Headers: {} 2023-05-31T20:53:54.835 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:53:54.835 DEBUG:boto:Port: 8000 2023-05-31T20:53:54.836 DEBUG:boto:Params: {} 2023-05-31T20:53:54.836 DEBUG:boto:Token: None 2023-05-31T20:53:54.837 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:53:54 GMT /pklxkd-59/obj58?versionId=null 2023-05-31T20:53:54.837 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:rUITRzOhr2PcQKPeKu93kL3+zMU= 2023-05-31T20:53:54.837 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:53:54 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:rUITRzOhr2PcQKPeKu93kL3+zMU=', 'Content-Length': '0'} 2023-05-31T20:53:54.839 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:48:12 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004c600fca3bda4353-006477b3e2-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:53:54 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:53:54.840 DEBUG:boto:path=/pklxkd-59/obj58 2023-05-31T20:53:54.840 DEBUG:boto:auth_path=/pklxkd-59/obj58 2023-05-31T20:53:54.841 DEBUG:boto:path=/pklxkd-59/obj58?versionId=null 2023-05-31T20:53:54.841 DEBUG:boto:auth_path=/pklxkd-59/obj58?versionId=null 2023-05-31T20:53:54.841 DEBUG:boto:Method: HEAD 2023-05-31T20:53:54.842 DEBUG:boto:Path: /pklxkd-59/obj58?versionId=null 2023-05-31T20:53:54.842 DEBUG:boto:Data: 2023-05-31T20:53:54.842 DEBUG:boto:Headers: {} 2023-05-31T20:53:54.843 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:53:54.843 DEBUG:boto:Port: 8001 2023-05-31T20:53:54.843 DEBUG:boto:Params: {} 2023-05-31T20:53:54.844 DEBUG:boto:Token: None 2023-05-31T20:53:54.844 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:53:54 GMT /pklxkd-59/obj58?versionId=null 2023-05-31T20:53:54.844 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:S7Lw5YmlV/FAxry8JYeoFriQPnM= 2023-05-31T20:53:54.845 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:53:54 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:S7Lw5YmlV/FAxry8JYeoFriQPnM=', 'Content-Length': '0'} 2023-05-31T20:53:54.847 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:48:12 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006b878cc67239fb6b-006477b3e2-1c17-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:53:54 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:53:54.847 DEBUG:boto:path=/pklxkd-59/obj58 2023-05-31T20:53:54.848 DEBUG:boto:auth_path=/pklxkd-59/obj58 2023-05-31T20:53:54.848 DEBUG:boto:path=/pklxkd-59/obj58?versionId=null 2023-05-31T20:53:54.848 DEBUG:boto:auth_path=/pklxkd-59/obj58?versionId=null 2023-05-31T20:53:54.849 DEBUG:boto:Method: HEAD 2023-05-31T20:53:54.849 DEBUG:boto:Path: /pklxkd-59/obj58?versionId=null 2023-05-31T20:53:54.849 DEBUG:boto:Data: 2023-05-31T20:53:54.850 DEBUG:boto:Headers: {} 2023-05-31T20:53:54.850 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:53:54.850 DEBUG:boto:Port: 8000 2023-05-31T20:53:54.851 DEBUG:boto:Params: {} 2023-05-31T20:53:54.851 DEBUG:boto:Token: None 2023-05-31T20:53:54.851 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:53:54 GMT /pklxkd-59/obj58?versionId=null 2023-05-31T20:53:54.852 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:S7Lw5YmlV/FAxry8JYeoFriQPnM= 2023-05-31T20:53:54.852 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:53:54 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:S7Lw5YmlV/FAxry8JYeoFriQPnM=', 'Content-Length': '0'} 2023-05-31T20:53:54.854 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:48:12 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f775d78a555ffd80-006477b3e2-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:53:54 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:53:54.855 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj58 2023-05-31T20:53:54.855 DEBUG:boto:path=/pklxkd-59/obj58 2023-05-31T20:53:54.855 DEBUG:boto:auth_path=/pklxkd-59/obj58 2023-05-31T20:53:54.856 DEBUG:boto:path=/pklxkd-59/obj58?versionId=null 2023-05-31T20:53:54.856 DEBUG:boto:auth_path=/pklxkd-59/obj58?versionId=null 2023-05-31T20:53:54.856 DEBUG:boto:Method: GET 2023-05-31T20:53:54.857 DEBUG:boto:Path: /pklxkd-59/obj58?versionId=null 2023-05-31T20:53:54.857 DEBUG:boto:Data: 2023-05-31T20:53:54.858 DEBUG:boto:Headers: {} 2023-05-31T20:53:54.858 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:53:54.858 DEBUG:boto:Port: 8001 2023-05-31T20:53:54.859 DEBUG:boto:Params: {} 2023-05-31T20:53:54.859 DEBUG:boto:Token: None 2023-05-31T20:53:54.859 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:53:54 GMT /pklxkd-59/obj58?versionId=null 2023-05-31T20:53:54.860 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:rUITRzOhr2PcQKPeKu93kL3+zMU= 2023-05-31T20:53:54.860 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:53:54 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:rUITRzOhr2PcQKPeKu93kL3+zMU=', 'Content-Length': '0'} 2023-05-31T20:53:54.862 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:48:12 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f51379bccef0b527-006477b3e2-1c17-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:53:54 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:53:54.863 DEBUG:boto:path=/pklxkd-59/obj58 2023-05-31T20:53:54.863 DEBUG:boto:auth_path=/pklxkd-59/obj58 2023-05-31T20:53:54.863 DEBUG:boto:path=/pklxkd-59/obj58?versionId=null 2023-05-31T20:53:54.864 DEBUG:boto:auth_path=/pklxkd-59/obj58?versionId=null 2023-05-31T20:53:54.864 DEBUG:boto:Method: GET 2023-05-31T20:53:54.864 DEBUG:boto:Path: /pklxkd-59/obj58?versionId=null 2023-05-31T20:53:54.865 DEBUG:boto:Data: 2023-05-31T20:53:54.865 DEBUG:boto:Headers: {} 2023-05-31T20:53:54.865 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:53:54.866 DEBUG:boto:Port: 8000 2023-05-31T20:53:54.866 DEBUG:boto:Params: {} 2023-05-31T20:53:54.867 DEBUG:boto:Token: None 2023-05-31T20:53:54.867 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:53:54 GMT /pklxkd-59/obj58?versionId=null 2023-05-31T20:53:54.867 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:rUITRzOhr2PcQKPeKu93kL3+zMU= 2023-05-31T20:53:54.868 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:53:54 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:rUITRzOhr2PcQKPeKu93kL3+zMU=', 'Content-Length': '0'} 2023-05-31T20:53:54.870 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:48:12 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000820dadcd97e6b6d8-006477b3e2-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:53:54 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:53:54.870 DEBUG:boto:path=/pklxkd-59/ 2023-05-31T20:53:54.871 DEBUG:boto:auth_path=/pklxkd-59/ 2023-05-31T20:53:54.871 DEBUG:boto:path=/pklxkd-59/?prefix=obj58 2023-05-31T20:53:54.872 DEBUG:boto:auth_path=/pklxkd-59/?prefix=obj58 2023-05-31T20:53:54.872 DEBUG:boto:Method: GET 2023-05-31T20:53:54.872 DEBUG:boto:Path: /pklxkd-59/?prefix=obj58 2023-05-31T20:53:54.873 DEBUG:boto:Data: 2023-05-31T20:53:54.873 DEBUG:boto:Headers: {} 2023-05-31T20:53:54.873 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:53:54.874 DEBUG:boto:Port: 8001 2023-05-31T20:53:54.874 DEBUG:boto:Params: {} 2023-05-31T20:53:54.874 DEBUG:boto:Token: None 2023-05-31T20:53:54.875 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:53:54 GMT /pklxkd-59/ 2023-05-31T20:53:54.875 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:X6lj3OWBH37r6jmqQli80RnWHOg= 2023-05-31T20:53:54.876 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:53:54 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:X6lj3OWBH37r6jmqQli80RnWHOg=', 'Content-Length': '0'} 2023-05-31T20:53:54.878 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b832d0c83f89bd96-006477b3e2-1c17-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:53:54 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:53:54.879 DEBUG:boto:b'pklxkd-59obj581000falseobj582023-05-31T20:48:12.976Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:53:54.879 DEBUG:boto:path=/pklxkd-59/ 2023-05-31T20:53:54.880 DEBUG:boto:auth_path=/pklxkd-59/ 2023-05-31T20:53:54.880 DEBUG:boto:path=/pklxkd-59/?prefix=obj58 2023-05-31T20:53:54.880 DEBUG:boto:auth_path=/pklxkd-59/?prefix=obj58 2023-05-31T20:53:54.881 DEBUG:boto:Method: GET 2023-05-31T20:53:54.881 DEBUG:boto:Path: /pklxkd-59/?prefix=obj58 2023-05-31T20:53:54.881 DEBUG:boto:Data: 2023-05-31T20:53:54.882 DEBUG:boto:Headers: {} 2023-05-31T20:53:54.882 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:53:54.882 DEBUG:boto:Port: 8000 2023-05-31T20:53:54.883 DEBUG:boto:Params: {} 2023-05-31T20:53:54.883 DEBUG:boto:Token: None 2023-05-31T20:53:54.884 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:53:54 GMT /pklxkd-59/ 2023-05-31T20:53:54.884 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:X6lj3OWBH37r6jmqQli80RnWHOg= 2023-05-31T20:53:54.885 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:53:54 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:X6lj3OWBH37r6jmqQli80RnWHOg=', 'Content-Length': '0'} 2023-05-31T20:53:54.888 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000bb9c60c640bc5a82-006477b3e2-1630-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:53:54 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:53:54.889 DEBUG:boto:b'pklxkd-59obj581000falseobj582023-05-31T20:48:12.976Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:53:54.889 DEBUG:boto:path=/pklxkd-59/obj58 2023-05-31T20:53:54.890 DEBUG:boto:auth_path=/pklxkd-59/obj58 2023-05-31T20:53:54.890 DEBUG:boto:Method: HEAD 2023-05-31T20:53:54.891 DEBUG:boto:Path: /pklxkd-59/obj58 2023-05-31T20:53:54.891 DEBUG:boto:Data: 2023-05-31T20:53:54.891 DEBUG:boto:Headers: {} 2023-05-31T20:53:54.892 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:53:54.892 DEBUG:boto:Port: 8001 2023-05-31T20:53:54.892 DEBUG:boto:Params: {} 2023-05-31T20:53:54.893 DEBUG:boto:Token: None 2023-05-31T20:53:54.893 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:53:54 GMT /pklxkd-59/obj58 2023-05-31T20:53:54.893 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:vLDwK4VxkgcC1qOB4FYs+KPjogo= 2023-05-31T20:53:54.894 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:53:54 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:vLDwK4VxkgcC1qOB4FYs+KPjogo=', 'Content-Length': '0'} 2023-05-31T20:53:54.895 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:48:12 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004f39a61766a96127-006477b3e2-1c17-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:53:54 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:53:54.896 DEBUG:boto:path=/pklxkd-59/obj58 2023-05-31T20:53:54.896 DEBUG:boto:auth_path=/pklxkd-59/obj58 2023-05-31T20:53:54.897 DEBUG:boto:Method: HEAD 2023-05-31T20:53:54.897 DEBUG:boto:Path: /pklxkd-59/obj58 2023-05-31T20:53:54.897 DEBUG:boto:Data: 2023-05-31T20:53:54.898 DEBUG:boto:Headers: {} 2023-05-31T20:53:54.898 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:53:54.898 DEBUG:boto:Port: 8000 2023-05-31T20:53:54.899 DEBUG:boto:Params: {} 2023-05-31T20:53:54.899 DEBUG:boto:Token: None 2023-05-31T20:53:54.900 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:53:54 GMT /pklxkd-59/obj58 2023-05-31T20:53:54.900 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:vLDwK4VxkgcC1qOB4FYs+KPjogo= 2023-05-31T20:53:54.900 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:53:54 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:vLDwK4VxkgcC1qOB4FYs+KPjogo=', 'Content-Length': '0'} 2023-05-31T20:53:54.902 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:48:12 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009c1659990e519cdf-006477b3e2-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:53:54 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:53:54.903 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj58 2023-05-31T20:53:54.903 DEBUG:boto:path=/pklxkd-59/obj58 2023-05-31T20:53:54.904 DEBUG:boto:auth_path=/pklxkd-59/obj58 2023-05-31T20:53:54.904 DEBUG:boto:Method: GET 2023-05-31T20:53:54.904 DEBUG:boto:Path: /pklxkd-59/obj58 2023-05-31T20:53:54.905 DEBUG:boto:Data: 2023-05-31T20:53:54.905 DEBUG:boto:Headers: {} 2023-05-31T20:53:54.905 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:53:54.906 DEBUG:boto:Port: 8001 2023-05-31T20:53:54.906 DEBUG:boto:Params: {} 2023-05-31T20:53:54.906 DEBUG:boto:Token: None 2023-05-31T20:53:54.907 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:53:54 GMT /pklxkd-59/obj58 2023-05-31T20:53:54.907 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:G5vmRDHFfYMhr7aG9pcm4WoyUG8= 2023-05-31T20:53:54.908 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:53:54 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:G5vmRDHFfYMhr7aG9pcm4WoyUG8=', 'Content-Length': '0'} 2023-05-31T20:53:54.909 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:48:12 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002d076a414f84eceb-006477b3e2-1c17-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:53:54 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:53:54.910 DEBUG:boto:path=/pklxkd-59/obj58 2023-05-31T20:53:54.910 DEBUG:boto:auth_path=/pklxkd-59/obj58 2023-05-31T20:53:54.911 DEBUG:boto:Method: GET 2023-05-31T20:53:54.911 DEBUG:boto:Path: /pklxkd-59/obj58 2023-05-31T20:53:54.912 DEBUG:boto:Data: 2023-05-31T20:53:54.912 DEBUG:boto:Headers: {} 2023-05-31T20:53:54.912 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:53:54.913 DEBUG:boto:Port: 8000 2023-05-31T20:53:54.913 DEBUG:boto:Params: {} 2023-05-31T20:53:54.913 DEBUG:boto:Token: None 2023-05-31T20:53:54.914 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:53:54 GMT /pklxkd-59/obj58 2023-05-31T20:53:54.914 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:G5vmRDHFfYMhr7aG9pcm4WoyUG8= 2023-05-31T20:53:54.915 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:53:54 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:G5vmRDHFfYMhr7aG9pcm4WoyUG8=', 'Content-Length': '0'} 2023-05-31T20:53:54.916 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:48:12 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b24fae98516345ac-006477b3e2-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:53:54 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:53:54.960 DEBUG:boto:path=/pklxkd-59/ 2023-05-31T20:53:54.961 DEBUG:boto:auth_path=/pklxkd-59/ 2023-05-31T20:53:54.961 DEBUG:boto:path=/pklxkd-59/?prefix=obj58 2023-05-31T20:53:54.961 DEBUG:boto:auth_path=/pklxkd-59/?prefix=obj58 2023-05-31T20:53:54.962 DEBUG:boto:Method: GET 2023-05-31T20:53:54.962 DEBUG:boto:Path: /pklxkd-59/?prefix=obj58 2023-05-31T20:53:54.962 DEBUG:boto:Data: 2023-05-31T20:53:54.963 DEBUG:boto:Headers: {} 2023-05-31T20:53:54.963 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:53:54.963 DEBUG:boto:Port: 8001 2023-05-31T20:53:54.964 DEBUG:boto:Params: {} 2023-05-31T20:53:54.964 DEBUG:boto:Token: None 2023-05-31T20:53:54.965 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:53:54 GMT /pklxkd-59/ 2023-05-31T20:53:54.965 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:X6lj3OWBH37r6jmqQli80RnWHOg= 2023-05-31T20:53:54.965 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:53:54 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:X6lj3OWBH37r6jmqQli80RnWHOg=', 'Content-Length': '0'} 2023-05-31T20:53:54.968 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000003f793afa623a0050-006477b3e2-1c17-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:53:54 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:53:54.968 DEBUG:boto:b'pklxkd-59obj581000falseobj582023-05-31T20:48:12.976Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:53:54.969 DEBUG:boto:path=/pklxkd-59/ 2023-05-31T20:53:54.969 DEBUG:boto:auth_path=/pklxkd-59/ 2023-05-31T20:53:54.970 DEBUG:boto:path=/pklxkd-59/?prefix=obj58 2023-05-31T20:53:54.970 DEBUG:boto:auth_path=/pklxkd-59/?prefix=obj58 2023-05-31T20:53:54.970 DEBUG:boto:Method: GET 2023-05-31T20:53:54.971 DEBUG:boto:Path: /pklxkd-59/?prefix=obj58 2023-05-31T20:53:54.971 DEBUG:boto:Data: 2023-05-31T20:53:54.971 DEBUG:boto:Headers: {} 2023-05-31T20:53:54.972 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:53:54.972 DEBUG:boto:Port: 8000 2023-05-31T20:53:54.972 DEBUG:boto:Params: {} 2023-05-31T20:53:54.973 DEBUG:boto:Token: None 2023-05-31T20:53:54.973 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:53:54 GMT /pklxkd-59/ 2023-05-31T20:53:54.974 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:X6lj3OWBH37r6jmqQli80RnWHOg= 2023-05-31T20:53:54.974 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:53:54 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:X6lj3OWBH37r6jmqQli80RnWHOg=', 'Content-Length': '0'} 2023-05-31T20:53:54.978 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000004a0fb25570cc9fc7-006477b3e2-1630-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:53:54 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:53:54.978 DEBUG:boto:b'pklxkd-59obj581000falseobj582023-05-31T20:48:12.976Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:53:54.979 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj62 2023-05-31T20:53:54.979 DEBUG:boto:path=/pklxkd-59/obj62 2023-05-31T20:53:54.980 DEBUG:boto:auth_path=/pklxkd-59/obj62 2023-05-31T20:53:54.980 DEBUG:boto:path=/pklxkd-59/obj62?versionId=null 2023-05-31T20:53:54.980 DEBUG:boto:auth_path=/pklxkd-59/obj62?versionId=null 2023-05-31T20:53:54.981 DEBUG:boto:Method: GET 2023-05-31T20:53:54.981 DEBUG:boto:Path: /pklxkd-59/obj62?versionId=null 2023-05-31T20:53:54.981 DEBUG:boto:Data: 2023-05-31T20:53:54.982 DEBUG:boto:Headers: {} 2023-05-31T20:53:54.982 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:53:54.982 DEBUG:boto:Port: 8001 2023-05-31T20:53:54.983 DEBUG:boto:Params: {} 2023-05-31T20:53:54.983 DEBUG:boto:Token: None 2023-05-31T20:53:54.984 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:53:54 GMT /pklxkd-59/obj62?versionId=null 2023-05-31T20:53:54.984 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:uL9L1nnX51FUup5gUcefrMoTvhY= 2023-05-31T20:53:54.984 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:53:54 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:uL9L1nnX51FUup5gUcefrMoTvhY=', 'Content-Length': '0'} 2023-05-31T20:53:54.986 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:50:33 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000fad2b2a85257b19f-006477b3e2-1c17-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:53:54 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:53:54.987 DEBUG:boto:path=/pklxkd-59/obj62 2023-05-31T20:53:54.987 DEBUG:boto:auth_path=/pklxkd-59/obj62 2023-05-31T20:53:54.988 DEBUG:boto:path=/pklxkd-59/obj62?versionId=null 2023-05-31T20:53:54.988 DEBUG:boto:auth_path=/pklxkd-59/obj62?versionId=null 2023-05-31T20:53:54.989 DEBUG:boto:Method: GET 2023-05-31T20:53:54.989 DEBUG:boto:Path: /pklxkd-59/obj62?versionId=null 2023-05-31T20:53:54.989 DEBUG:boto:Data: 2023-05-31T20:53:54.990 DEBUG:boto:Headers: {} 2023-05-31T20:53:54.990 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:53:54.990 DEBUG:boto:Port: 8000 2023-05-31T20:53:54.991 DEBUG:boto:Params: {} 2023-05-31T20:53:54.991 DEBUG:boto:Token: None 2023-05-31T20:53:54.992 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:53:54 GMT /pklxkd-59/obj62?versionId=null 2023-05-31T20:53:54.992 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:uL9L1nnX51FUup5gUcefrMoTvhY= 2023-05-31T20:53:54.992 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:53:54 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:uL9L1nnX51FUup5gUcefrMoTvhY=', 'Content-Length': '0'} 2023-05-31T20:53:54.994 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:50:33 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000108dc0b7d2982fd1-006477b3e2-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:53:54 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:53:54.995 DEBUG:boto:path=/pklxkd-59/obj62 2023-05-31T20:53:54.995 DEBUG:boto:auth_path=/pklxkd-59/obj62 2023-05-31T20:53:54.996 DEBUG:boto:path=/pklxkd-59/obj62?versionId=null 2023-05-31T20:53:54.996 DEBUG:boto:auth_path=/pklxkd-59/obj62?versionId=null 2023-05-31T20:53:54.996 DEBUG:boto:Method: HEAD 2023-05-31T20:53:54.997 DEBUG:boto:Path: /pklxkd-59/obj62?versionId=null 2023-05-31T20:53:54.997 DEBUG:boto:Data: 2023-05-31T20:53:54.997 DEBUG:boto:Headers: {} 2023-05-31T20:53:54.998 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:53:54.998 DEBUG:boto:Port: 8001 2023-05-31T20:53:54.998 DEBUG:boto:Params: {} 2023-05-31T20:53:54.999 DEBUG:boto:Token: None 2023-05-31T20:53:54.999 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:53:54 GMT /pklxkd-59/obj62?versionId=null 2023-05-31T20:53:54.999 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:FCxUuknoAtkmVQLvVJjKRE1Rn7c= 2023-05-31T20:53:55.000 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:53:54 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:FCxUuknoAtkmVQLvVJjKRE1Rn7c=', 'Content-Length': '0'} 2023-05-31T20:53:55.002 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:50:33 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004cdf909f1fa28068-006477b3e2-1c17-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:53:54 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:53:55.002 DEBUG:boto:path=/pklxkd-59/obj62 2023-05-31T20:53:55.003 DEBUG:boto:auth_path=/pklxkd-59/obj62 2023-05-31T20:53:55.003 DEBUG:boto:path=/pklxkd-59/obj62?versionId=null 2023-05-31T20:53:55.003 DEBUG:boto:auth_path=/pklxkd-59/obj62?versionId=null 2023-05-31T20:53:55.004 DEBUG:boto:Method: HEAD 2023-05-31T20:53:55.004 DEBUG:boto:Path: /pklxkd-59/obj62?versionId=null 2023-05-31T20:53:55.004 DEBUG:boto:Data: 2023-05-31T20:53:55.005 DEBUG:boto:Headers: {} 2023-05-31T20:53:55.005 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:53:55.005 DEBUG:boto:Port: 8000 2023-05-31T20:53:55.006 DEBUG:boto:Params: {} 2023-05-31T20:53:55.006 DEBUG:boto:Token: None 2023-05-31T20:53:55.006 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:53:55 GMT /pklxkd-59/obj62?versionId=null 2023-05-31T20:53:55.007 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:2QLtBNFgpSyWPOYH18S9jwwqyO4= 2023-05-31T20:53:55.007 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:53:55 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:2QLtBNFgpSyWPOYH18S9jwwqyO4=', 'Content-Length': '0'} 2023-05-31T20:53:55.009 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:50:33 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002ee4eebbb457a818-006477b3e3-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:53:55 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:53:55.010 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj62 2023-05-31T20:53:55.010 DEBUG:boto:path=/pklxkd-59/obj62 2023-05-31T20:53:55.010 DEBUG:boto:auth_path=/pklxkd-59/obj62 2023-05-31T20:53:55.011 DEBUG:boto:path=/pklxkd-59/obj62?versionId=null 2023-05-31T20:53:55.011 DEBUG:boto:auth_path=/pklxkd-59/obj62?versionId=null 2023-05-31T20:53:55.011 DEBUG:boto:Method: GET 2023-05-31T20:53:55.012 DEBUG:boto:Path: /pklxkd-59/obj62?versionId=null 2023-05-31T20:53:55.012 DEBUG:boto:Data: 2023-05-31T20:53:55.012 DEBUG:boto:Headers: {} 2023-05-31T20:53:55.013 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:53:55.013 DEBUG:boto:Port: 8001 2023-05-31T20:53:55.014 DEBUG:boto:Params: {} 2023-05-31T20:53:55.014 DEBUG:boto:Token: None 2023-05-31T20:53:55.014 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:53:55 GMT /pklxkd-59/obj62?versionId=null 2023-05-31T20:53:55.015 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:hqRnLv0PB2z+WOuxgbC4EvaC2jI= 2023-05-31T20:53:55.015 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:53:55 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:hqRnLv0PB2z+WOuxgbC4EvaC2jI=', 'Content-Length': '0'} 2023-05-31T20:53:55.017 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:50:33 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000da68cab8c20ab4ff-006477b3e3-1c17-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:53:55 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:53:55.017 DEBUG:boto:path=/pklxkd-59/obj62 2023-05-31T20:53:55.018 DEBUG:boto:auth_path=/pklxkd-59/obj62 2023-05-31T20:53:55.018 DEBUG:boto:path=/pklxkd-59/obj62?versionId=null 2023-05-31T20:53:55.019 DEBUG:boto:auth_path=/pklxkd-59/obj62?versionId=null 2023-05-31T20:53:55.019 DEBUG:boto:Method: GET 2023-05-31T20:53:55.019 DEBUG:boto:Path: /pklxkd-59/obj62?versionId=null 2023-05-31T20:53:55.020 DEBUG:boto:Data: 2023-05-31T20:53:55.020 DEBUG:boto:Headers: {} 2023-05-31T20:53:55.020 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:53:55.021 DEBUG:boto:Port: 8000 2023-05-31T20:53:55.021 DEBUG:boto:Params: {} 2023-05-31T20:53:55.021 DEBUG:boto:Token: None 2023-05-31T20:53:55.022 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:53:55 GMT /pklxkd-59/obj62?versionId=null 2023-05-31T20:53:55.022 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:hqRnLv0PB2z+WOuxgbC4EvaC2jI= 2023-05-31T20:53:55.023 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:53:55 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:hqRnLv0PB2z+WOuxgbC4EvaC2jI=', 'Content-Length': '0'} 2023-05-31T20:53:55.025 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:50:33 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000001371d5b0df45033-006477b3e3-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:53:55 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:53:55.068 DEBUG:boto:path=/pklxkd-59/ 2023-05-31T20:53:55.069 DEBUG:boto:auth_path=/pklxkd-59/ 2023-05-31T20:53:55.069 DEBUG:boto:path=/pklxkd-59/?prefix=obj62 2023-05-31T20:53:55.070 DEBUG:boto:auth_path=/pklxkd-59/?prefix=obj62 2023-05-31T20:53:55.070 DEBUG:boto:Method: GET 2023-05-31T20:53:55.070 DEBUG:boto:Path: /pklxkd-59/?prefix=obj62 2023-05-31T20:53:55.071 DEBUG:boto:Data: 2023-05-31T20:53:55.071 DEBUG:boto:Headers: {} 2023-05-31T20:53:55.071 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:53:55.072 DEBUG:boto:Port: 8001 2023-05-31T20:53:55.072 DEBUG:boto:Params: {} 2023-05-31T20:53:55.073 DEBUG:boto:Token: None 2023-05-31T20:53:55.073 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:53:55 GMT /pklxkd-59/ 2023-05-31T20:53:55.073 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:FBsfjfyGgIUCCYA4ReXasZ3hTno= 2023-05-31T20:53:55.074 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:53:55 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:FBsfjfyGgIUCCYA4ReXasZ3hTno=', 'Content-Length': '0'} 2023-05-31T20:53:55.076 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ee443c15ca7c66e5-006477b3e3-1c17-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:53:55 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:53:55.077 DEBUG:boto:b'pklxkd-59obj621000falseobj622023-05-31T20:50:33.515Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:53:55.077 DEBUG:boto:path=/pklxkd-59/ 2023-05-31T20:53:55.078 DEBUG:boto:auth_path=/pklxkd-59/ 2023-05-31T20:53:55.078 DEBUG:boto:path=/pklxkd-59/?prefix=obj62 2023-05-31T20:53:55.079 DEBUG:boto:auth_path=/pklxkd-59/?prefix=obj62 2023-05-31T20:53:55.079 DEBUG:boto:Method: GET 2023-05-31T20:53:55.079 DEBUG:boto:Path: /pklxkd-59/?prefix=obj62 2023-05-31T20:53:55.080 DEBUG:boto:Data: 2023-05-31T20:53:55.080 DEBUG:boto:Headers: {} 2023-05-31T20:53:55.080 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:53:55.081 DEBUG:boto:Port: 8000 2023-05-31T20:53:55.081 DEBUG:boto:Params: {} 2023-05-31T20:53:55.081 DEBUG:boto:Token: None 2023-05-31T20:53:55.082 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:53:55 GMT /pklxkd-59/ 2023-05-31T20:53:55.082 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:FBsfjfyGgIUCCYA4ReXasZ3hTno= 2023-05-31T20:53:55.083 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:53:55 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:FBsfjfyGgIUCCYA4ReXasZ3hTno=', 'Content-Length': '0'} 2023-05-31T20:53:55.087 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000079a03c9227523ed9-006477b3e3-1630-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:53:55 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:53:55.088 DEBUG:boto:b'pklxkd-59obj621000falseobj622023-05-31T20:50:33.515Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:53:55.088 DEBUG:boto:path=/pklxkd-59/obj62 2023-05-31T20:53:55.089 DEBUG:boto:auth_path=/pklxkd-59/obj62 2023-05-31T20:53:55.089 DEBUG:boto:Method: HEAD 2023-05-31T20:53:55.089 DEBUG:boto:Path: /pklxkd-59/obj62 2023-05-31T20:53:55.090 DEBUG:boto:Data: 2023-05-31T20:53:55.090 DEBUG:boto:Headers: {} 2023-05-31T20:53:55.090 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:53:55.091 DEBUG:boto:Port: 8001 2023-05-31T20:53:55.091 DEBUG:boto:Params: {} 2023-05-31T20:53:55.091 DEBUG:boto:Token: None 2023-05-31T20:53:55.092 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:53:55 GMT /pklxkd-59/obj62 2023-05-31T20:53:55.092 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:yG6t62A6+l8PkkGyjWY50zlgIps= 2023-05-31T20:53:55.093 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:53:55 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:yG6t62A6+l8PkkGyjWY50zlgIps=', 'Content-Length': '0'} 2023-05-31T20:53:55.094 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:50:33 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000039f475b1d8e4da32-006477b3e3-1c17-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:53:55 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:53:55.095 DEBUG:boto:path=/pklxkd-59/obj62 2023-05-31T20:53:55.095 DEBUG:boto:auth_path=/pklxkd-59/obj62 2023-05-31T20:53:55.096 DEBUG:boto:Method: HEAD 2023-05-31T20:53:55.096 DEBUG:boto:Path: /pklxkd-59/obj62 2023-05-31T20:53:55.096 DEBUG:boto:Data: 2023-05-31T20:53:55.097 DEBUG:boto:Headers: {} 2023-05-31T20:53:55.097 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:53:55.097 DEBUG:boto:Port: 8000 2023-05-31T20:53:55.098 DEBUG:boto:Params: {} 2023-05-31T20:53:55.098 DEBUG:boto:Token: None 2023-05-31T20:53:55.098 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:53:55 GMT /pklxkd-59/obj62 2023-05-31T20:53:55.099 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:yG6t62A6+l8PkkGyjWY50zlgIps= 2023-05-31T20:53:55.099 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:53:55 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:yG6t62A6+l8PkkGyjWY50zlgIps=', 'Content-Length': '0'} 2023-05-31T20:53:55.101 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:50:33 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000647aa10b4bdcdd7c-006477b3e3-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:53:55 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:53:55.102 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj62 2023-05-31T20:53:55.102 DEBUG:boto:path=/pklxkd-59/obj62 2023-05-31T20:53:55.102 DEBUG:boto:auth_path=/pklxkd-59/obj62 2023-05-31T20:53:55.103 DEBUG:boto:Method: GET 2023-05-31T20:53:55.103 DEBUG:boto:Path: /pklxkd-59/obj62 2023-05-31T20:53:55.104 DEBUG:boto:Data: 2023-05-31T20:53:55.104 DEBUG:boto:Headers: {} 2023-05-31T20:53:55.104 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:53:55.105 DEBUG:boto:Port: 8001 2023-05-31T20:53:55.105 DEBUG:boto:Params: {} 2023-05-31T20:53:55.105 DEBUG:boto:Token: None 2023-05-31T20:53:55.106 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:53:55 GMT /pklxkd-59/obj62 2023-05-31T20:53:55.106 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:OgxBYRUBDT2r5fGL0NCvY56B9O4= 2023-05-31T20:53:55.107 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:53:55 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:OgxBYRUBDT2r5fGL0NCvY56B9O4=', 'Content-Length': '0'} 2023-05-31T20:53:55.109 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:50:33 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000eff6ec45384b6338-006477b3e3-1c17-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:53:55 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:53:55.109 DEBUG:boto:path=/pklxkd-59/obj62 2023-05-31T20:53:55.110 DEBUG:boto:auth_path=/pklxkd-59/obj62 2023-05-31T20:53:55.110 DEBUG:boto:Method: GET 2023-05-31T20:53:55.110 DEBUG:boto:Path: /pklxkd-59/obj62 2023-05-31T20:53:55.111 DEBUG:boto:Data: 2023-05-31T20:53:55.111 DEBUG:boto:Headers: {} 2023-05-31T20:53:55.111 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:53:55.112 DEBUG:boto:Port: 8000 2023-05-31T20:53:55.112 DEBUG:boto:Params: {} 2023-05-31T20:53:55.113 DEBUG:boto:Token: None 2023-05-31T20:53:55.113 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:53:55 GMT /pklxkd-59/obj62 2023-05-31T20:53:55.113 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:OgxBYRUBDT2r5fGL0NCvY56B9O4= 2023-05-31T20:53:55.114 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:53:55 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:OgxBYRUBDT2r5fGL0NCvY56B9O4=', 'Content-Length': '0'} 2023-05-31T20:53:55.116 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:50:33 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000042cc3059cbc7605-006477b3e3-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:53:55 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:53:55.156 DEBUG:boto:path=/pklxkd-59/ 2023-05-31T20:53:55.157 DEBUG:boto:auth_path=/pklxkd-59/ 2023-05-31T20:53:55.157 DEBUG:boto:path=/pklxkd-59/?prefix=obj62 2023-05-31T20:53:55.157 DEBUG:boto:auth_path=/pklxkd-59/?prefix=obj62 2023-05-31T20:53:55.158 DEBUG:boto:Method: GET 2023-05-31T20:53:55.158 DEBUG:boto:Path: /pklxkd-59/?prefix=obj62 2023-05-31T20:53:55.158 DEBUG:boto:Data: 2023-05-31T20:53:55.159 DEBUG:boto:Headers: {} 2023-05-31T20:53:55.159 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:53:55.160 DEBUG:boto:Port: 8001 2023-05-31T20:53:55.160 DEBUG:boto:Params: {} 2023-05-31T20:53:55.160 DEBUG:boto:Token: None 2023-05-31T20:53:55.161 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:53:55 GMT /pklxkd-59/ 2023-05-31T20:53:55.161 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:FBsfjfyGgIUCCYA4ReXasZ3hTno= 2023-05-31T20:53:55.162 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:53:55 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:FBsfjfyGgIUCCYA4ReXasZ3hTno=', 'Content-Length': '0'} 2023-05-31T20:53:55.164 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000024386255529e9404-006477b3e3-1c17-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:53:55 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:53:55.164 DEBUG:boto:b'pklxkd-59obj621000falseobj622023-05-31T20:50:33.515Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:53:55.165 DEBUG:boto:path=/pklxkd-59/ 2023-05-31T20:53:55.165 DEBUG:boto:auth_path=/pklxkd-59/ 2023-05-31T20:53:55.166 DEBUG:boto:path=/pklxkd-59/?prefix=obj62 2023-05-31T20:53:55.166 DEBUG:boto:auth_path=/pklxkd-59/?prefix=obj62 2023-05-31T20:53:55.166 DEBUG:boto:Method: GET 2023-05-31T20:53:55.167 DEBUG:boto:Path: /pklxkd-59/?prefix=obj62 2023-05-31T20:53:55.167 DEBUG:boto:Data: 2023-05-31T20:53:55.168 DEBUG:boto:Headers: {} 2023-05-31T20:53:55.168 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:53:55.168 DEBUG:boto:Port: 8000 2023-05-31T20:53:55.169 DEBUG:boto:Params: {} 2023-05-31T20:53:55.169 DEBUG:boto:Token: None 2023-05-31T20:53:55.169 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:53:55 GMT /pklxkd-59/ 2023-05-31T20:53:55.170 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:FBsfjfyGgIUCCYA4ReXasZ3hTno= 2023-05-31T20:53:55.170 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:53:55 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:FBsfjfyGgIUCCYA4ReXasZ3hTno=', 'Content-Length': '0'} 2023-05-31T20:53:55.174 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000098fff6cd354ae862-006477b3e3-1630-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:53:55 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:53:55.175 DEBUG:boto:b'pklxkd-59obj621000falseobj622023-05-31T20:50:33.515Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:53:55.175 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj68 2023-05-31T20:53:55.176 DEBUG:boto:path=/pklxkd-59/obj68 2023-05-31T20:53:55.176 DEBUG:boto:auth_path=/pklxkd-59/obj68 2023-05-31T20:53:55.177 DEBUG:boto:path=/pklxkd-59/obj68?versionId=null 2023-05-31T20:53:55.177 DEBUG:boto:auth_path=/pklxkd-59/obj68?versionId=null 2023-05-31T20:53:55.177 DEBUG:boto:Method: GET 2023-05-31T20:53:55.178 DEBUG:boto:Path: /pklxkd-59/obj68?versionId=null 2023-05-31T20:53:55.178 DEBUG:boto:Data: 2023-05-31T20:53:55.178 DEBUG:boto:Headers: {} 2023-05-31T20:53:55.179 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:53:55.179 DEBUG:boto:Port: 8001 2023-05-31T20:53:55.179 DEBUG:boto:Params: {} 2023-05-31T20:53:55.180 DEBUG:boto:Token: None 2023-05-31T20:53:55.180 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:53:55 GMT /pklxkd-59/obj68?versionId=null 2023-05-31T20:53:55.181 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:1IQyWZjpRZDT2zvFmL1siqVqNyM= 2023-05-31T20:53:55.181 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:53:55 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:1IQyWZjpRZDT2zvFmL1siqVqNyM=', 'Content-Length': '0'} 2023-05-31T20:53:55.183 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:43:08 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d7f6797b0d6c7c90-006477b3e3-1c17-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:53:55 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:53:55.184 DEBUG:boto:path=/pklxkd-59/obj68 2023-05-31T20:53:55.184 DEBUG:boto:auth_path=/pklxkd-59/obj68 2023-05-31T20:53:55.184 DEBUG:boto:path=/pklxkd-59/obj68?versionId=null 2023-05-31T20:53:55.185 DEBUG:boto:auth_path=/pklxkd-59/obj68?versionId=null 2023-05-31T20:53:55.185 DEBUG:boto:Method: GET 2023-05-31T20:53:55.186 DEBUG:boto:Path: /pklxkd-59/obj68?versionId=null 2023-05-31T20:53:55.186 DEBUG:boto:Data: 2023-05-31T20:53:55.186 DEBUG:boto:Headers: {} 2023-05-31T20:53:55.187 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:53:55.187 DEBUG:boto:Port: 8000 2023-05-31T20:53:55.187 DEBUG:boto:Params: {} 2023-05-31T20:53:55.188 DEBUG:boto:Token: None 2023-05-31T20:53:55.188 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:53:55 GMT /pklxkd-59/obj68?versionId=null 2023-05-31T20:53:55.189 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:1IQyWZjpRZDT2zvFmL1siqVqNyM= 2023-05-31T20:53:55.189 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:53:55 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:1IQyWZjpRZDT2zvFmL1siqVqNyM=', 'Content-Length': '0'} 2023-05-31T20:53:55.191 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:43:08 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005f1f7e059c98e7e7-006477b3e3-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:53:55 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:53:55.192 DEBUG:boto:path=/pklxkd-59/obj68 2023-05-31T20:53:55.192 DEBUG:boto:auth_path=/pklxkd-59/obj68 2023-05-31T20:53:55.192 DEBUG:boto:path=/pklxkd-59/obj68?versionId=null 2023-05-31T20:53:55.193 DEBUG:boto:auth_path=/pklxkd-59/obj68?versionId=null 2023-05-31T20:53:55.193 DEBUG:boto:Method: HEAD 2023-05-31T20:53:55.193 DEBUG:boto:Path: /pklxkd-59/obj68?versionId=null 2023-05-31T20:53:55.194 DEBUG:boto:Data: 2023-05-31T20:53:55.194 DEBUG:boto:Headers: {} 2023-05-31T20:53:55.194 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:53:55.195 DEBUG:boto:Port: 8001 2023-05-31T20:53:55.195 DEBUG:boto:Params: {} 2023-05-31T20:53:55.195 DEBUG:boto:Token: None 2023-05-31T20:53:55.196 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:53:55 GMT /pklxkd-59/obj68?versionId=null 2023-05-31T20:53:55.196 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:EvhapkvfRKCow7SSwQsOYeRGyYY= 2023-05-31T20:53:55.197 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:53:55 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:EvhapkvfRKCow7SSwQsOYeRGyYY=', 'Content-Length': '0'} 2023-05-31T20:53:55.198 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:43:08 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f1c4070f324a8175-006477b3e3-1c17-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:53:55 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:53:55.199 DEBUG:boto:path=/pklxkd-59/obj68 2023-05-31T20:53:55.199 DEBUG:boto:auth_path=/pklxkd-59/obj68 2023-05-31T20:53:55.200 DEBUG:boto:path=/pklxkd-59/obj68?versionId=null 2023-05-31T20:53:55.200 DEBUG:boto:auth_path=/pklxkd-59/obj68?versionId=null 2023-05-31T20:53:55.200 DEBUG:boto:Method: HEAD 2023-05-31T20:53:55.201 DEBUG:boto:Path: /pklxkd-59/obj68?versionId=null 2023-05-31T20:53:55.201 DEBUG:boto:Data: 2023-05-31T20:53:55.201 DEBUG:boto:Headers: {} 2023-05-31T20:53:55.202 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:53:55.202 DEBUG:boto:Port: 8000 2023-05-31T20:53:55.202 DEBUG:boto:Params: {} 2023-05-31T20:53:55.203 DEBUG:boto:Token: None 2023-05-31T20:53:55.203 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:53:55 GMT /pklxkd-59/obj68?versionId=null 2023-05-31T20:53:55.204 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:EvhapkvfRKCow7SSwQsOYeRGyYY= 2023-05-31T20:53:55.204 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:53:55 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:EvhapkvfRKCow7SSwQsOYeRGyYY=', 'Content-Length': '0'} 2023-05-31T20:53:55.206 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:43:08 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001ca58c82fec205e6-006477b3e3-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:53:55 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:53:55.206 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj68 2023-05-31T20:53:55.207 DEBUG:boto:path=/pklxkd-59/obj68 2023-05-31T20:53:55.207 DEBUG:boto:auth_path=/pklxkd-59/obj68 2023-05-31T20:53:55.207 DEBUG:boto:path=/pklxkd-59/obj68?versionId=null 2023-05-31T20:53:55.208 DEBUG:boto:auth_path=/pklxkd-59/obj68?versionId=null 2023-05-31T20:53:55.208 DEBUG:boto:Method: GET 2023-05-31T20:53:55.208 DEBUG:boto:Path: /pklxkd-59/obj68?versionId=null 2023-05-31T20:53:55.209 DEBUG:boto:Data: 2023-05-31T20:53:55.209 DEBUG:boto:Headers: {} 2023-05-31T20:53:55.209 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:53:55.210 DEBUG:boto:Port: 8001 2023-05-31T20:53:55.210 DEBUG:boto:Params: {} 2023-05-31T20:53:55.210 DEBUG:boto:Token: None 2023-05-31T20:53:55.211 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:53:55 GMT /pklxkd-59/obj68?versionId=null 2023-05-31T20:53:55.211 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:1IQyWZjpRZDT2zvFmL1siqVqNyM= 2023-05-31T20:53:55.212 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:53:55 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:1IQyWZjpRZDT2zvFmL1siqVqNyM=', 'Content-Length': '0'} 2023-05-31T20:53:55.214 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:43:08 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c0896e3c7b25e752-006477b3e3-1c17-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:53:55 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:53:55.214 DEBUG:boto:path=/pklxkd-59/obj68 2023-05-31T20:53:55.215 DEBUG:boto:auth_path=/pklxkd-59/obj68 2023-05-31T20:53:55.215 DEBUG:boto:path=/pklxkd-59/obj68?versionId=null 2023-05-31T20:53:55.215 DEBUG:boto:auth_path=/pklxkd-59/obj68?versionId=null 2023-05-31T20:53:55.216 DEBUG:boto:Method: GET 2023-05-31T20:53:55.216 DEBUG:boto:Path: /pklxkd-59/obj68?versionId=null 2023-05-31T20:53:55.216 DEBUG:boto:Data: 2023-05-31T20:53:55.217 DEBUG:boto:Headers: {} 2023-05-31T20:53:55.217 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:53:55.217 DEBUG:boto:Port: 8000 2023-05-31T20:53:55.221 DEBUG:boto:Params: {} 2023-05-31T20:53:55.221 DEBUG:boto:Token: None 2023-05-31T20:53:55.221 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:53:55 GMT /pklxkd-59/obj68?versionId=null 2023-05-31T20:53:55.222 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:1IQyWZjpRZDT2zvFmL1siqVqNyM= 2023-05-31T20:53:55.222 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:53:55 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:1IQyWZjpRZDT2zvFmL1siqVqNyM=', 'Content-Length': '0'} 2023-05-31T20:53:55.224 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:43:08 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009b53712c990b28e0-006477b3e3-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:53:55 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:53:55.225 DEBUG:boto:path=/pklxkd-59/ 2023-05-31T20:53:55.225 DEBUG:boto:auth_path=/pklxkd-59/ 2023-05-31T20:53:55.226 DEBUG:boto:path=/pklxkd-59/?prefix=obj68 2023-05-31T20:53:55.226 DEBUG:boto:auth_path=/pklxkd-59/?prefix=obj68 2023-05-31T20:53:55.227 DEBUG:boto:Method: GET 2023-05-31T20:53:55.227 DEBUG:boto:Path: /pklxkd-59/?prefix=obj68 2023-05-31T20:53:55.227 DEBUG:boto:Data: 2023-05-31T20:53:55.228 DEBUG:boto:Headers: {} 2023-05-31T20:53:55.228 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:53:55.228 DEBUG:boto:Port: 8001 2023-05-31T20:53:55.229 DEBUG:boto:Params: {} 2023-05-31T20:53:55.229 DEBUG:boto:Token: None 2023-05-31T20:53:55.230 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:53:55 GMT /pklxkd-59/ 2023-05-31T20:53:55.230 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:FBsfjfyGgIUCCYA4ReXasZ3hTno= 2023-05-31T20:53:55.230 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:53:55 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:FBsfjfyGgIUCCYA4ReXasZ3hTno=', 'Content-Length': '0'} 2023-05-31T20:53:55.233 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000099ae27180a5afe1d-006477b3e3-1c17-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:53:55 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:53:55.233 DEBUG:boto:b'pklxkd-59obj681000falseobj682023-05-31T20:43:08.066Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:53:55.234 DEBUG:boto:path=/pklxkd-59/ 2023-05-31T20:53:55.234 DEBUG:boto:auth_path=/pklxkd-59/ 2023-05-31T20:53:55.235 DEBUG:boto:path=/pklxkd-59/?prefix=obj68 2023-05-31T20:53:55.235 DEBUG:boto:auth_path=/pklxkd-59/?prefix=obj68 2023-05-31T20:53:55.235 DEBUG:boto:Method: GET 2023-05-31T20:53:55.236 DEBUG:boto:Path: /pklxkd-59/?prefix=obj68 2023-05-31T20:53:55.236 DEBUG:boto:Data: 2023-05-31T20:53:55.236 DEBUG:boto:Headers: {} 2023-05-31T20:53:55.237 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:53:55.237 DEBUG:boto:Port: 8000 2023-05-31T20:53:55.237 DEBUG:boto:Params: {} 2023-05-31T20:53:55.238 DEBUG:boto:Token: None 2023-05-31T20:53:55.238 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:53:55 GMT /pklxkd-59/ 2023-05-31T20:53:55.239 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:FBsfjfyGgIUCCYA4ReXasZ3hTno= 2023-05-31T20:53:55.239 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:53:55 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:FBsfjfyGgIUCCYA4ReXasZ3hTno=', 'Content-Length': '0'} 2023-05-31T20:53:55.243 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000de6e928460f39c72-006477b3e3-1630-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:53:55 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:53:55.284 DEBUG:boto:b'pklxkd-59obj681000falseobj682023-05-31T20:43:08.066Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:53:55.285 DEBUG:boto:path=/pklxkd-59/obj68 2023-05-31T20:53:55.285 DEBUG:boto:auth_path=/pklxkd-59/obj68 2023-05-31T20:53:55.285 DEBUG:boto:Method: HEAD 2023-05-31T20:53:55.286 DEBUG:boto:Path: /pklxkd-59/obj68 2023-05-31T20:53:55.286 DEBUG:boto:Data: 2023-05-31T20:53:55.286 DEBUG:boto:Headers: {} 2023-05-31T20:53:55.287 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:53:55.287 DEBUG:boto:Port: 8001 2023-05-31T20:53:55.287 DEBUG:boto:Params: {} 2023-05-31T20:53:55.288 DEBUG:boto:Token: None 2023-05-31T20:53:55.288 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:53:55 GMT /pklxkd-59/obj68 2023-05-31T20:53:55.289 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:w+6AnYlOtj25rNoFSGkgdrLqULo= 2023-05-31T20:53:55.289 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:53:55 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:w+6AnYlOtj25rNoFSGkgdrLqULo=', 'Content-Length': '0'} 2023-05-31T20:53:55.291 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:43:08 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000036c5f7d7a30da645-006477b3e3-1c17-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:53:55 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:53:55.291 DEBUG:boto:path=/pklxkd-59/obj68 2023-05-31T20:53:55.292 DEBUG:boto:auth_path=/pklxkd-59/obj68 2023-05-31T20:53:55.292 DEBUG:boto:Method: HEAD 2023-05-31T20:53:55.293 DEBUG:boto:Path: /pklxkd-59/obj68 2023-05-31T20:53:55.293 DEBUG:boto:Data: 2023-05-31T20:53:55.293 DEBUG:boto:Headers: {} 2023-05-31T20:53:55.293 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:53:55.294 DEBUG:boto:Port: 8000 2023-05-31T20:53:55.294 DEBUG:boto:Params: {} 2023-05-31T20:53:55.295 DEBUG:boto:Token: None 2023-05-31T20:53:55.295 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:53:55 GMT /pklxkd-59/obj68 2023-05-31T20:53:55.295 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:w+6AnYlOtj25rNoFSGkgdrLqULo= 2023-05-31T20:53:55.296 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:53:55 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:w+6AnYlOtj25rNoFSGkgdrLqULo=', 'Content-Length': '0'} 2023-05-31T20:53:55.297 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:43:08 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f8ece75f2ee49193-006477b3e3-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:53:55 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:53:55.298 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj68 2023-05-31T20:53:55.298 DEBUG:boto:path=/pklxkd-59/obj68 2023-05-31T20:53:55.299 DEBUG:boto:auth_path=/pklxkd-59/obj68 2023-05-31T20:53:55.299 DEBUG:boto:Method: GET 2023-05-31T20:53:55.299 DEBUG:boto:Path: /pklxkd-59/obj68 2023-05-31T20:53:55.300 DEBUG:boto:Data: 2023-05-31T20:53:55.300 DEBUG:boto:Headers: {} 2023-05-31T20:53:55.300 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:53:55.301 DEBUG:boto:Port: 8001 2023-05-31T20:53:55.301 DEBUG:boto:Params: {} 2023-05-31T20:53:55.302 DEBUG:boto:Token: None 2023-05-31T20:53:55.302 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:53:55 GMT /pklxkd-59/obj68 2023-05-31T20:53:55.302 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:HZ+q5cQsFxa/F7As/YBMxqpOhss= 2023-05-31T20:53:55.303 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:53:55 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:HZ+q5cQsFxa/F7As/YBMxqpOhss=', 'Content-Length': '0'} 2023-05-31T20:53:55.305 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:43:08 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ee8091283b753456-006477b3e3-1c17-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:53:55 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:53:55.305 DEBUG:boto:path=/pklxkd-59/obj68 2023-05-31T20:53:55.306 DEBUG:boto:auth_path=/pklxkd-59/obj68 2023-05-31T20:53:55.306 DEBUG:boto:Method: GET 2023-05-31T20:53:55.307 DEBUG:boto:Path: /pklxkd-59/obj68 2023-05-31T20:53:55.307 DEBUG:boto:Data: 2023-05-31T20:53:55.307 DEBUG:boto:Headers: {} 2023-05-31T20:53:55.308 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:53:55.308 DEBUG:boto:Port: 8000 2023-05-31T20:53:55.308 DEBUG:boto:Params: {} 2023-05-31T20:53:55.309 DEBUG:boto:Token: None 2023-05-31T20:53:55.309 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:53:55 GMT /pklxkd-59/obj68 2023-05-31T20:53:55.309 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:HZ+q5cQsFxa/F7As/YBMxqpOhss= 2023-05-31T20:53:55.310 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:53:55 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:HZ+q5cQsFxa/F7As/YBMxqpOhss=', 'Content-Length': '0'} 2023-05-31T20:53:55.312 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:43:08 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009847872b1eda7b2b-006477b3e3-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:53:55 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:53:55.312 DEBUG:boto:path=/pklxkd-59/ 2023-05-31T20:53:55.313 DEBUG:boto:auth_path=/pklxkd-59/ 2023-05-31T20:53:55.313 DEBUG:boto:path=/pklxkd-59/?prefix=obj68 2023-05-31T20:53:55.313 DEBUG:boto:auth_path=/pklxkd-59/?prefix=obj68 2023-05-31T20:53:55.314 DEBUG:boto:Method: GET 2023-05-31T20:53:55.314 DEBUG:boto:Path: /pklxkd-59/?prefix=obj68 2023-05-31T20:53:55.314 DEBUG:boto:Data: 2023-05-31T20:53:55.315 DEBUG:boto:Headers: {} 2023-05-31T20:53:55.315 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:53:55.316 DEBUG:boto:Port: 8001 2023-05-31T20:53:55.316 DEBUG:boto:Params: {} 2023-05-31T20:53:55.316 DEBUG:boto:Token: None 2023-05-31T20:53:55.317 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:53:55 GMT /pklxkd-59/ 2023-05-31T20:53:55.317 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:FBsfjfyGgIUCCYA4ReXasZ3hTno= 2023-05-31T20:53:55.318 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:53:55 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:FBsfjfyGgIUCCYA4ReXasZ3hTno=', 'Content-Length': '0'} 2023-05-31T20:53:55.320 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000445fd010cb2be2ff-006477b3e3-1c17-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:53:55 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:53:55.360 DEBUG:boto:b'pklxkd-59obj681000falseobj682023-05-31T20:43:08.066Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:53:55.361 DEBUG:boto:path=/pklxkd-59/ 2023-05-31T20:53:55.361 DEBUG:boto:auth_path=/pklxkd-59/ 2023-05-31T20:53:55.362 DEBUG:boto:path=/pklxkd-59/?prefix=obj68 2023-05-31T20:53:55.362 DEBUG:boto:auth_path=/pklxkd-59/?prefix=obj68 2023-05-31T20:53:55.362 DEBUG:boto:Method: GET 2023-05-31T20:53:55.363 DEBUG:boto:Path: /pklxkd-59/?prefix=obj68 2023-05-31T20:53:55.363 DEBUG:boto:Data: 2023-05-31T20:53:55.363 DEBUG:boto:Headers: {} 2023-05-31T20:53:55.364 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:53:55.364 DEBUG:boto:Port: 8000 2023-05-31T20:53:55.364 DEBUG:boto:Params: {} 2023-05-31T20:53:55.365 DEBUG:boto:Token: None 2023-05-31T20:53:55.365 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:53:55 GMT /pklxkd-59/ 2023-05-31T20:53:55.366 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:FBsfjfyGgIUCCYA4ReXasZ3hTno= 2023-05-31T20:53:55.366 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:53:55 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:FBsfjfyGgIUCCYA4ReXasZ3hTno=', 'Content-Length': '0'} 2023-05-31T20:53:55.370 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000022f3475217d23ad9-006477b3e3-1630-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:53:55 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:53:55.371 DEBUG:boto:b'pklxkd-59obj681000falseobj682023-05-31T20:43:08.066Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:53:55.371 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj69 2023-05-31T20:53:55.372 DEBUG:boto:path=/pklxkd-59/obj69 2023-05-31T20:53:55.372 DEBUG:boto:auth_path=/pklxkd-59/obj69 2023-05-31T20:53:55.373 DEBUG:boto:path=/pklxkd-59/obj69?versionId=null 2023-05-31T20:53:55.373 DEBUG:boto:auth_path=/pklxkd-59/obj69?versionId=null 2023-05-31T20:53:55.374 DEBUG:boto:Method: GET 2023-05-31T20:53:55.374 DEBUG:boto:Path: /pklxkd-59/obj69?versionId=null 2023-05-31T20:53:55.375 DEBUG:boto:Data: 2023-05-31T20:53:55.375 DEBUG:boto:Headers: {} 2023-05-31T20:53:55.376 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:53:55.376 DEBUG:boto:Port: 8001 2023-05-31T20:53:55.377 DEBUG:boto:Params: {} 2023-05-31T20:53:55.377 DEBUG:boto:Token: None 2023-05-31T20:53:55.378 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:53:55 GMT /pklxkd-59/obj69?versionId=null 2023-05-31T20:53:55.378 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:vIxqMpmusxWu30/4+xi4SIuvABo= 2023-05-31T20:53:55.378 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:53:55 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:vIxqMpmusxWu30/4+xi4SIuvABo=', 'Content-Length': '0'} 2023-05-31T20:53:55.380 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:46:33 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000cc88a0000961a249-006477b3e3-1c17-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:53:55 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:53:55.381 DEBUG:boto:path=/pklxkd-59/obj69 2023-05-31T20:53:55.382 DEBUG:boto:auth_path=/pklxkd-59/obj69 2023-05-31T20:53:55.382 DEBUG:boto:path=/pklxkd-59/obj69?versionId=null 2023-05-31T20:53:55.383 DEBUG:boto:auth_path=/pklxkd-59/obj69?versionId=null 2023-05-31T20:53:55.383 DEBUG:boto:Method: GET 2023-05-31T20:53:55.384 DEBUG:boto:Path: /pklxkd-59/obj69?versionId=null 2023-05-31T20:53:55.384 DEBUG:boto:Data: 2023-05-31T20:53:55.385 DEBUG:boto:Headers: {} 2023-05-31T20:53:55.385 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:53:55.385 DEBUG:boto:Port: 8000 2023-05-31T20:53:55.386 DEBUG:boto:Params: {} 2023-05-31T20:53:55.386 DEBUG:boto:Token: None 2023-05-31T20:53:55.387 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:53:55 GMT /pklxkd-59/obj69?versionId=null 2023-05-31T20:53:55.387 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:vIxqMpmusxWu30/4+xi4SIuvABo= 2023-05-31T20:53:55.388 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:53:55 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:vIxqMpmusxWu30/4+xi4SIuvABo=', 'Content-Length': '0'} 2023-05-31T20:53:55.390 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:46:33 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000de36e453d2b870b4-006477b3e3-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:53:55 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:53:55.390 DEBUG:boto:path=/pklxkd-59/obj69 2023-05-31T20:53:55.391 DEBUG:boto:auth_path=/pklxkd-59/obj69 2023-05-31T20:53:55.391 DEBUG:boto:path=/pklxkd-59/obj69?versionId=null 2023-05-31T20:53:55.391 DEBUG:boto:auth_path=/pklxkd-59/obj69?versionId=null 2023-05-31T20:53:55.392 DEBUG:boto:Method: HEAD 2023-05-31T20:53:55.392 DEBUG:boto:Path: /pklxkd-59/obj69?versionId=null 2023-05-31T20:53:55.392 DEBUG:boto:Data: 2023-05-31T20:53:55.393 DEBUG:boto:Headers: {} 2023-05-31T20:53:55.393 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:53:55.393 DEBUG:boto:Port: 8001 2023-05-31T20:53:55.394 DEBUG:boto:Params: {} 2023-05-31T20:53:55.394 DEBUG:boto:Token: None 2023-05-31T20:53:55.394 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:53:55 GMT /pklxkd-59/obj69?versionId=null 2023-05-31T20:53:55.395 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:Kvbr8UW+bJkOVRnco22r/aqGHeE= 2023-05-31T20:53:55.395 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:53:55 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:Kvbr8UW+bJkOVRnco22r/aqGHeE=', 'Content-Length': '0'} 2023-05-31T20:53:55.397 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:46:33 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000169e9ad8b3162694-006477b3e3-1c17-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:53:55 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:53:55.398 DEBUG:boto:path=/pklxkd-59/obj69 2023-05-31T20:53:55.398 DEBUG:boto:auth_path=/pklxkd-59/obj69 2023-05-31T20:53:55.398 DEBUG:boto:path=/pklxkd-59/obj69?versionId=null 2023-05-31T20:53:55.399 DEBUG:boto:auth_path=/pklxkd-59/obj69?versionId=null 2023-05-31T20:53:55.399 DEBUG:boto:Method: HEAD 2023-05-31T20:53:55.399 DEBUG:boto:Path: /pklxkd-59/obj69?versionId=null 2023-05-31T20:53:55.400 DEBUG:boto:Data: 2023-05-31T20:53:55.400 DEBUG:boto:Headers: {} 2023-05-31T20:53:55.400 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:53:55.401 DEBUG:boto:Port: 8000 2023-05-31T20:53:55.401 DEBUG:boto:Params: {} 2023-05-31T20:53:55.402 DEBUG:boto:Token: None 2023-05-31T20:53:55.402 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:53:55 GMT /pklxkd-59/obj69?versionId=null 2023-05-31T20:53:55.402 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:Kvbr8UW+bJkOVRnco22r/aqGHeE= 2023-05-31T20:53:55.403 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:53:55 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:Kvbr8UW+bJkOVRnco22r/aqGHeE=', 'Content-Length': '0'} 2023-05-31T20:53:55.405 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:46:33 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000266379c38010e7af-006477b3e3-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:53:55 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:53:55.406 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj69 2023-05-31T20:53:55.407 DEBUG:boto:path=/pklxkd-59/obj69 2023-05-31T20:53:55.408 DEBUG:boto:auth_path=/pklxkd-59/obj69 2023-05-31T20:53:55.408 DEBUG:boto:path=/pklxkd-59/obj69?versionId=null 2023-05-31T20:53:55.409 DEBUG:boto:auth_path=/pklxkd-59/obj69?versionId=null 2023-05-31T20:53:55.410 DEBUG:boto:Method: GET 2023-05-31T20:53:55.410 DEBUG:boto:Path: /pklxkd-59/obj69?versionId=null 2023-05-31T20:53:55.411 DEBUG:boto:Data: 2023-05-31T20:53:55.411 DEBUG:boto:Headers: {} 2023-05-31T20:53:55.412 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:53:55.412 DEBUG:boto:Port: 8001 2023-05-31T20:53:55.412 DEBUG:boto:Params: {} 2023-05-31T20:53:55.413 DEBUG:boto:Token: None 2023-05-31T20:53:55.413 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:53:55 GMT /pklxkd-59/obj69?versionId=null 2023-05-31T20:53:55.414 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:vIxqMpmusxWu30/4+xi4SIuvABo= 2023-05-31T20:53:55.414 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:53:55 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:vIxqMpmusxWu30/4+xi4SIuvABo=', 'Content-Length': '0'} 2023-05-31T20:53:55.416 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:46:33 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000043aef77619363621-006477b3e3-1c17-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:53:55 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:53:55.456 DEBUG:boto:path=/pklxkd-59/obj69 2023-05-31T20:53:55.457 DEBUG:boto:auth_path=/pklxkd-59/obj69 2023-05-31T20:53:55.457 DEBUG:boto:path=/pklxkd-59/obj69?versionId=null 2023-05-31T20:53:55.457 DEBUG:boto:auth_path=/pklxkd-59/obj69?versionId=null 2023-05-31T20:53:55.458 DEBUG:boto:Method: GET 2023-05-31T20:53:55.458 DEBUG:boto:Path: /pklxkd-59/obj69?versionId=null 2023-05-31T20:53:55.459 DEBUG:boto:Data: 2023-05-31T20:53:55.459 DEBUG:boto:Headers: {} 2023-05-31T20:53:55.459 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:53:55.460 DEBUG:boto:Port: 8000 2023-05-31T20:53:55.460 DEBUG:boto:Params: {} 2023-05-31T20:53:55.461 DEBUG:boto:Token: None 2023-05-31T20:53:55.461 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:53:55 GMT /pklxkd-59/obj69?versionId=null 2023-05-31T20:53:55.462 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:vIxqMpmusxWu30/4+xi4SIuvABo= 2023-05-31T20:53:55.462 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:53:55 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:vIxqMpmusxWu30/4+xi4SIuvABo=', 'Content-Length': '0'} 2023-05-31T20:53:55.464 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:46:33 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000443681221bd70664-006477b3e3-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:53:55 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:53:55.464 DEBUG:boto:path=/pklxkd-59/ 2023-05-31T20:53:55.465 DEBUG:boto:auth_path=/pklxkd-59/ 2023-05-31T20:53:55.465 DEBUG:boto:path=/pklxkd-59/?prefix=obj69 2023-05-31T20:53:55.466 DEBUG:boto:auth_path=/pklxkd-59/?prefix=obj69 2023-05-31T20:53:55.466 DEBUG:boto:Method: GET 2023-05-31T20:53:55.466 DEBUG:boto:Path: /pklxkd-59/?prefix=obj69 2023-05-31T20:53:55.467 DEBUG:boto:Data: 2023-05-31T20:53:55.467 DEBUG:boto:Headers: {} 2023-05-31T20:53:55.468 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:53:55.468 DEBUG:boto:Port: 8001 2023-05-31T20:53:55.468 DEBUG:boto:Params: {} 2023-05-31T20:53:55.469 DEBUG:boto:Token: None 2023-05-31T20:53:55.469 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:53:55 GMT /pklxkd-59/ 2023-05-31T20:53:55.470 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:FBsfjfyGgIUCCYA4ReXasZ3hTno= 2023-05-31T20:53:55.470 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:53:55 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:FBsfjfyGgIUCCYA4ReXasZ3hTno=', 'Content-Length': '0'} 2023-05-31T20:53:55.472 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000009e96e2800d87a8dc-006477b3e3-1c17-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:53:55 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:53:55.473 DEBUG:boto:b'pklxkd-59obj691000falseobj692023-05-31T20:46:33.886Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:53:55.474 DEBUG:boto:path=/pklxkd-59/ 2023-05-31T20:53:55.474 DEBUG:boto:auth_path=/pklxkd-59/ 2023-05-31T20:53:55.475 DEBUG:boto:path=/pklxkd-59/?prefix=obj69 2023-05-31T20:53:55.475 DEBUG:boto:auth_path=/pklxkd-59/?prefix=obj69 2023-05-31T20:53:55.475 DEBUG:boto:Method: GET 2023-05-31T20:53:55.476 DEBUG:boto:Path: /pklxkd-59/?prefix=obj69 2023-05-31T20:53:55.476 DEBUG:boto:Data: 2023-05-31T20:53:55.477 DEBUG:boto:Headers: {} 2023-05-31T20:53:55.477 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:53:55.477 DEBUG:boto:Port: 8000 2023-05-31T20:53:55.478 DEBUG:boto:Params: {} 2023-05-31T20:53:55.478 DEBUG:boto:Token: None 2023-05-31T20:53:55.479 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:53:55 GMT /pklxkd-59/ 2023-05-31T20:53:55.479 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:FBsfjfyGgIUCCYA4ReXasZ3hTno= 2023-05-31T20:53:55.479 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:53:55 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:FBsfjfyGgIUCCYA4ReXasZ3hTno=', 'Content-Length': '0'} 2023-05-31T20:53:55.484 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000043dd11053d9dc6a6-006477b3e3-1630-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:53:55 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:53:55.484 DEBUG:boto:b'pklxkd-59obj691000falseobj692023-05-31T20:46:33.886Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:53:55.485 DEBUG:boto:path=/pklxkd-59/obj69 2023-05-31T20:53:55.485 DEBUG:boto:auth_path=/pklxkd-59/obj69 2023-05-31T20:53:55.486 DEBUG:boto:Method: HEAD 2023-05-31T20:53:55.486 DEBUG:boto:Path: /pklxkd-59/obj69 2023-05-31T20:53:55.486 DEBUG:boto:Data: 2023-05-31T20:53:55.487 DEBUG:boto:Headers: {} 2023-05-31T20:53:55.487 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:53:55.487 DEBUG:boto:Port: 8001 2023-05-31T20:53:55.488 DEBUG:boto:Params: {} 2023-05-31T20:53:55.488 DEBUG:boto:Token: None 2023-05-31T20:53:55.489 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:53:55 GMT /pklxkd-59/obj69 2023-05-31T20:53:55.489 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:yFfQgDAHR0r77xw6QelomuLjN2E= 2023-05-31T20:53:55.490 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:53:55 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:yFfQgDAHR0r77xw6QelomuLjN2E=', 'Content-Length': '0'} 2023-05-31T20:53:55.492 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:46:33 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000079221facfa147380-006477b3e3-1c17-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:53:55 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:53:55.492 DEBUG:boto:path=/pklxkd-59/obj69 2023-05-31T20:53:55.493 DEBUG:boto:auth_path=/pklxkd-59/obj69 2023-05-31T20:53:55.493 DEBUG:boto:Method: HEAD 2023-05-31T20:53:55.493 DEBUG:boto:Path: /pklxkd-59/obj69 2023-05-31T20:53:55.494 DEBUG:boto:Data: 2023-05-31T20:53:55.494 DEBUG:boto:Headers: {} 2023-05-31T20:53:55.494 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:53:55.495 DEBUG:boto:Port: 8000 2023-05-31T20:53:55.495 DEBUG:boto:Params: {} 2023-05-31T20:53:55.495 DEBUG:boto:Token: None 2023-05-31T20:53:55.496 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:53:55 GMT /pklxkd-59/obj69 2023-05-31T20:53:55.496 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:yFfQgDAHR0r77xw6QelomuLjN2E= 2023-05-31T20:53:55.497 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:53:55 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:yFfQgDAHR0r77xw6QelomuLjN2E=', 'Content-Length': '0'} 2023-05-31T20:53:55.498 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:46:33 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ff2360c05ac050fc-006477b3e3-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:53:55 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:53:55.499 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj69 2023-05-31T20:53:55.499 DEBUG:boto:path=/pklxkd-59/obj69 2023-05-31T20:53:55.500 DEBUG:boto:auth_path=/pklxkd-59/obj69 2023-05-31T20:53:55.500 DEBUG:boto:Method: GET 2023-05-31T20:53:55.500 DEBUG:boto:Path: /pklxkd-59/obj69 2023-05-31T20:53:55.501 DEBUG:boto:Data: 2023-05-31T20:53:55.501 DEBUG:boto:Headers: {} 2023-05-31T20:53:55.501 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:53:55.502 DEBUG:boto:Port: 8001 2023-05-31T20:53:55.502 DEBUG:boto:Params: {} 2023-05-31T20:53:55.503 DEBUG:boto:Token: None 2023-05-31T20:53:55.503 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:53:55 GMT /pklxkd-59/obj69 2023-05-31T20:53:55.503 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:w/cOG4aaVOGFh0Eq5h2Ec8tdGqM= 2023-05-31T20:53:55.504 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:53:55 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:w/cOG4aaVOGFh0Eq5h2Ec8tdGqM=', 'Content-Length': '0'} 2023-05-31T20:53:55.506 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:46:33 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006ed8dbcf032c30c7-006477b3e3-1c17-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:53:55 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:53:55.507 DEBUG:boto:path=/pklxkd-59/obj69 2023-05-31T20:53:55.507 DEBUG:boto:auth_path=/pklxkd-59/obj69 2023-05-31T20:53:55.507 DEBUG:boto:Method: GET 2023-05-31T20:53:55.508 DEBUG:boto:Path: /pklxkd-59/obj69 2023-05-31T20:53:55.508 DEBUG:boto:Data: 2023-05-31T20:53:55.508 DEBUG:boto:Headers: {} 2023-05-31T20:53:55.509 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:53:55.509 DEBUG:boto:Port: 8000 2023-05-31T20:53:55.509 DEBUG:boto:Params: {} 2023-05-31T20:53:55.510 DEBUG:boto:Token: None 2023-05-31T20:53:55.510 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:53:55 GMT /pklxkd-59/obj69 2023-05-31T20:53:55.511 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:w/cOG4aaVOGFh0Eq5h2Ec8tdGqM= 2023-05-31T20:53:55.511 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:53:55 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:w/cOG4aaVOGFh0Eq5h2Ec8tdGqM=', 'Content-Length': '0'} 2023-05-31T20:53:55.513 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:46:33 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001a7c5aff4a83692c-006477b3e3-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:53:55 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:53:55.514 DEBUG:boto:path=/pklxkd-59/ 2023-05-31T20:53:55.514 DEBUG:boto:auth_path=/pklxkd-59/ 2023-05-31T20:53:55.514 DEBUG:boto:path=/pklxkd-59/?prefix=obj69 2023-05-31T20:53:55.515 DEBUG:boto:auth_path=/pklxkd-59/?prefix=obj69 2023-05-31T20:53:55.515 DEBUG:boto:Method: GET 2023-05-31T20:53:55.515 DEBUG:boto:Path: /pklxkd-59/?prefix=obj69 2023-05-31T20:53:55.516 DEBUG:boto:Data: 2023-05-31T20:53:55.516 DEBUG:boto:Headers: {} 2023-05-31T20:53:55.517 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:53:55.517 DEBUG:boto:Port: 8001 2023-05-31T20:53:55.517 DEBUG:boto:Params: {} 2023-05-31T20:53:55.518 DEBUG:boto:Token: None 2023-05-31T20:53:55.518 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:53:55 GMT /pklxkd-59/ 2023-05-31T20:53:55.518 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:FBsfjfyGgIUCCYA4ReXasZ3hTno= 2023-05-31T20:53:55.519 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:53:55 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:FBsfjfyGgIUCCYA4ReXasZ3hTno=', 'Content-Length': '0'} 2023-05-31T20:53:55.521 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ffdcbb7fe9c48b95-006477b3e3-1c17-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:53:55 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:53:55.522 DEBUG:boto:b'pklxkd-59obj691000falseobj692023-05-31T20:46:33.886Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:53:55.522 DEBUG:boto:path=/pklxkd-59/ 2023-05-31T20:53:55.523 DEBUG:boto:auth_path=/pklxkd-59/ 2023-05-31T20:53:55.523 DEBUG:boto:path=/pklxkd-59/?prefix=obj69 2023-05-31T20:53:55.524 DEBUG:boto:auth_path=/pklxkd-59/?prefix=obj69 2023-05-31T20:53:55.524 DEBUG:boto:Method: GET 2023-05-31T20:53:55.524 DEBUG:boto:Path: /pklxkd-59/?prefix=obj69 2023-05-31T20:53:55.525 DEBUG:boto:Data: 2023-05-31T20:53:55.525 DEBUG:boto:Headers: {} 2023-05-31T20:53:55.525 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:53:55.526 DEBUG:boto:Port: 8000 2023-05-31T20:53:55.526 DEBUG:boto:Params: {} 2023-05-31T20:53:55.526 DEBUG:boto:Token: None 2023-05-31T20:53:55.527 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:53:55 GMT /pklxkd-59/ 2023-05-31T20:53:55.527 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:FBsfjfyGgIUCCYA4ReXasZ3hTno= 2023-05-31T20:53:55.528 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:53:55 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:FBsfjfyGgIUCCYA4ReXasZ3hTno=', 'Content-Length': '0'} 2023-05-31T20:53:55.532 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000001c1ad17658f08e3b-006477b3e3-1630-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:53:55 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:53:55.572 DEBUG:boto:b'pklxkd-59obj691000falseobj692023-05-31T20:46:33.886Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:53:55.573 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj74 2023-05-31T20:53:55.573 DEBUG:boto:path=/pklxkd-59/obj74 2023-05-31T20:53:55.574 DEBUG:boto:auth_path=/pklxkd-59/obj74 2023-05-31T20:53:55.574 DEBUG:boto:path=/pklxkd-59/obj74?versionId=null 2023-05-31T20:53:55.574 DEBUG:boto:auth_path=/pklxkd-59/obj74?versionId=null 2023-05-31T20:53:55.575 DEBUG:boto:Method: GET 2023-05-31T20:53:55.575 DEBUG:boto:Path: /pklxkd-59/obj74?versionId=null 2023-05-31T20:53:55.576 DEBUG:boto:Data: 2023-05-31T20:53:55.576 DEBUG:boto:Headers: {} 2023-05-31T20:53:55.576 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:53:55.577 DEBUG:boto:Port: 8001 2023-05-31T20:53:55.577 DEBUG:boto:Params: {} 2023-05-31T20:53:55.578 DEBUG:boto:Token: None 2023-05-31T20:53:55.578 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:53:55 GMT /pklxkd-59/obj74?versionId=null 2023-05-31T20:53:55.578 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:5mpvNXCjELA476OgM5lWN0R+fj8= 2023-05-31T20:53:55.579 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:53:55 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:5mpvNXCjELA476OgM5lWN0R+fj8=', 'Content-Length': '0'} 2023-05-31T20:53:55.581 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:52:14 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c8432da043c76e27-006477b3e3-1c17-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:53:55 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:53:55.582 DEBUG:boto:path=/pklxkd-59/obj74 2023-05-31T20:53:55.582 DEBUG:boto:auth_path=/pklxkd-59/obj74 2023-05-31T20:53:55.582 DEBUG:boto:path=/pklxkd-59/obj74?versionId=null 2023-05-31T20:53:55.583 DEBUG:boto:auth_path=/pklxkd-59/obj74?versionId=null 2023-05-31T20:53:55.583 DEBUG:boto:Method: GET 2023-05-31T20:53:55.584 DEBUG:boto:Path: /pklxkd-59/obj74?versionId=null 2023-05-31T20:53:55.584 DEBUG:boto:Data: 2023-05-31T20:53:55.585 DEBUG:boto:Headers: {} 2023-05-31T20:53:55.585 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:53:55.585 DEBUG:boto:Port: 8000 2023-05-31T20:53:55.586 DEBUG:boto:Params: {} 2023-05-31T20:53:55.587 DEBUG:boto:Token: None 2023-05-31T20:53:55.587 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:53:55 GMT /pklxkd-59/obj74?versionId=null 2023-05-31T20:53:55.587 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:5mpvNXCjELA476OgM5lWN0R+fj8= 2023-05-31T20:53:55.588 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:53:55 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:5mpvNXCjELA476OgM5lWN0R+fj8=', 'Content-Length': '0'} 2023-05-31T20:53:55.591 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:52:14 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000637c1360bfc9a9ae-006477b3e3-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:53:55 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:53:55.592 DEBUG:boto:path=/pklxkd-59/obj74 2023-05-31T20:53:55.593 DEBUG:boto:auth_path=/pklxkd-59/obj74 2023-05-31T20:53:55.593 DEBUG:boto:path=/pklxkd-59/obj74?versionId=null 2023-05-31T20:53:55.593 DEBUG:boto:auth_path=/pklxkd-59/obj74?versionId=null 2023-05-31T20:53:55.594 DEBUG:boto:Method: HEAD 2023-05-31T20:53:55.594 DEBUG:boto:Path: /pklxkd-59/obj74?versionId=null 2023-05-31T20:53:55.594 DEBUG:boto:Data: 2023-05-31T20:53:55.595 DEBUG:boto:Headers: {} 2023-05-31T20:53:55.595 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:53:55.595 DEBUG:boto:Port: 8001 2023-05-31T20:53:55.596 DEBUG:boto:Params: {} 2023-05-31T20:53:55.596 DEBUG:boto:Token: None 2023-05-31T20:53:55.596 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:53:55 GMT /pklxkd-59/obj74?versionId=null 2023-05-31T20:53:55.597 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:Geb3fN9Y9/BAftgPnqWig2DcFcY= 2023-05-31T20:53:55.597 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:53:55 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:Geb3fN9Y9/BAftgPnqWig2DcFcY=', 'Content-Length': '0'} 2023-05-31T20:53:55.599 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:52:14 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000024994adb059e5d24-006477b3e3-1c17-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:53:55 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:53:55.599 DEBUG:boto:path=/pklxkd-59/obj74 2023-05-31T20:53:55.600 DEBUG:boto:auth_path=/pklxkd-59/obj74 2023-05-31T20:53:55.600 DEBUG:boto:path=/pklxkd-59/obj74?versionId=null 2023-05-31T20:53:55.601 DEBUG:boto:auth_path=/pklxkd-59/obj74?versionId=null 2023-05-31T20:53:55.601 DEBUG:boto:Method: HEAD 2023-05-31T20:53:55.601 DEBUG:boto:Path: /pklxkd-59/obj74?versionId=null 2023-05-31T20:53:55.602 DEBUG:boto:Data: 2023-05-31T20:53:55.602 DEBUG:boto:Headers: {} 2023-05-31T20:53:55.602 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:53:55.603 DEBUG:boto:Port: 8000 2023-05-31T20:53:55.603 DEBUG:boto:Params: {} 2023-05-31T20:53:55.603 DEBUG:boto:Token: None 2023-05-31T20:53:55.604 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:53:55 GMT /pklxkd-59/obj74?versionId=null 2023-05-31T20:53:55.604 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:Geb3fN9Y9/BAftgPnqWig2DcFcY= 2023-05-31T20:53:55.604 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:53:55 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:Geb3fN9Y9/BAftgPnqWig2DcFcY=', 'Content-Length': '0'} 2023-05-31T20:53:55.606 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:52:14 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000727eb1b5ed913e4b-006477b3e3-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:53:55 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:53:55.607 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj74 2023-05-31T20:53:55.607 DEBUG:boto:path=/pklxkd-59/obj74 2023-05-31T20:53:55.608 DEBUG:boto:auth_path=/pklxkd-59/obj74 2023-05-31T20:53:55.608 DEBUG:boto:path=/pklxkd-59/obj74?versionId=null 2023-05-31T20:53:55.609 DEBUG:boto:auth_path=/pklxkd-59/obj74?versionId=null 2023-05-31T20:53:55.609 DEBUG:boto:Method: GET 2023-05-31T20:53:55.609 DEBUG:boto:Path: /pklxkd-59/obj74?versionId=null 2023-05-31T20:53:55.610 DEBUG:boto:Data: 2023-05-31T20:53:55.610 DEBUG:boto:Headers: {} 2023-05-31T20:53:55.610 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:53:55.611 DEBUG:boto:Port: 8001 2023-05-31T20:53:55.611 DEBUG:boto:Params: {} 2023-05-31T20:53:55.611 DEBUG:boto:Token: None 2023-05-31T20:53:55.612 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:53:55 GMT /pklxkd-59/obj74?versionId=null 2023-05-31T20:53:55.612 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:5mpvNXCjELA476OgM5lWN0R+fj8= 2023-05-31T20:53:55.613 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:53:55 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:5mpvNXCjELA476OgM5lWN0R+fj8=', 'Content-Length': '0'} 2023-05-31T20:53:55.614 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:52:14 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e977f978615a87f2-006477b3e3-1c17-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:53:55 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:53:55.656 DEBUG:boto:path=/pklxkd-59/obj74 2023-05-31T20:53:55.657 DEBUG:boto:auth_path=/pklxkd-59/obj74 2023-05-31T20:53:55.657 DEBUG:boto:path=/pklxkd-59/obj74?versionId=null 2023-05-31T20:53:55.657 DEBUG:boto:auth_path=/pklxkd-59/obj74?versionId=null 2023-05-31T20:53:55.658 DEBUG:boto:Method: GET 2023-05-31T20:53:55.658 DEBUG:boto:Path: /pklxkd-59/obj74?versionId=null 2023-05-31T20:53:55.658 DEBUG:boto:Data: 2023-05-31T20:53:55.659 DEBUG:boto:Headers: {} 2023-05-31T20:53:55.659 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:53:55.659 DEBUG:boto:Port: 8000 2023-05-31T20:53:55.660 DEBUG:boto:Params: {} 2023-05-31T20:53:55.660 DEBUG:boto:Token: None 2023-05-31T20:53:55.661 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:53:55 GMT /pklxkd-59/obj74?versionId=null 2023-05-31T20:53:55.661 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:5mpvNXCjELA476OgM5lWN0R+fj8= 2023-05-31T20:53:55.661 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:53:55 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:5mpvNXCjELA476OgM5lWN0R+fj8=', 'Content-Length': '0'} 2023-05-31T20:53:55.664 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:52:14 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000fbfb1ac0554a11bf-006477b3e3-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:53:55 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:53:55.664 DEBUG:boto:path=/pklxkd-59/ 2023-05-31T20:53:55.665 DEBUG:boto:auth_path=/pklxkd-59/ 2023-05-31T20:53:55.665 DEBUG:boto:path=/pklxkd-59/?prefix=obj74 2023-05-31T20:53:55.666 DEBUG:boto:auth_path=/pklxkd-59/?prefix=obj74 2023-05-31T20:53:55.666 DEBUG:boto:Method: GET 2023-05-31T20:53:55.666 DEBUG:boto:Path: /pklxkd-59/?prefix=obj74 2023-05-31T20:53:55.667 DEBUG:boto:Data: 2023-05-31T20:53:55.667 DEBUG:boto:Headers: {} 2023-05-31T20:53:55.667 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:53:55.668 DEBUG:boto:Port: 8001 2023-05-31T20:53:55.668 DEBUG:boto:Params: {} 2023-05-31T20:53:55.668 DEBUG:boto:Token: None 2023-05-31T20:53:55.669 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:53:55 GMT /pklxkd-59/ 2023-05-31T20:53:55.669 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:FBsfjfyGgIUCCYA4ReXasZ3hTno= 2023-05-31T20:53:55.670 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:53:55 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:FBsfjfyGgIUCCYA4ReXasZ3hTno=', 'Content-Length': '0'} 2023-05-31T20:53:55.672 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000030c21fe86e71e804-006477b3e3-1c17-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:53:55 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:53:55.672 DEBUG:boto:b'pklxkd-59obj741000falseobj742023-05-31T20:52:14.934Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:53:55.673 DEBUG:boto:path=/pklxkd-59/ 2023-05-31T20:53:55.673 DEBUG:boto:auth_path=/pklxkd-59/ 2023-05-31T20:53:55.674 DEBUG:boto:path=/pklxkd-59/?prefix=obj74 2023-05-31T20:53:55.674 DEBUG:boto:auth_path=/pklxkd-59/?prefix=obj74 2023-05-31T20:53:55.675 DEBUG:boto:Method: GET 2023-05-31T20:53:55.675 DEBUG:boto:Path: /pklxkd-59/?prefix=obj74 2023-05-31T20:53:55.675 DEBUG:boto:Data: 2023-05-31T20:53:55.676 DEBUG:boto:Headers: {} 2023-05-31T20:53:55.676 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:53:55.676 DEBUG:boto:Port: 8000 2023-05-31T20:53:55.677 DEBUG:boto:Params: {} 2023-05-31T20:53:55.677 DEBUG:boto:Token: None 2023-05-31T20:53:55.678 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:53:55 GMT /pklxkd-59/ 2023-05-31T20:53:55.678 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:FBsfjfyGgIUCCYA4ReXasZ3hTno= 2023-05-31T20:53:55.678 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:53:55 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:FBsfjfyGgIUCCYA4ReXasZ3hTno=', 'Content-Length': '0'} 2023-05-31T20:53:55.682 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e2579820402e138d-006477b3e3-1630-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:53:55 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:53:55.683 DEBUG:boto:b'pklxkd-59obj741000falseobj742023-05-31T20:52:14.934Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:53:55.684 DEBUG:boto:path=/pklxkd-59/obj74 2023-05-31T20:53:55.684 DEBUG:boto:auth_path=/pklxkd-59/obj74 2023-05-31T20:53:55.684 DEBUG:boto:Method: HEAD 2023-05-31T20:53:55.685 DEBUG:boto:Path: /pklxkd-59/obj74 2023-05-31T20:53:55.685 DEBUG:boto:Data: 2023-05-31T20:53:55.686 DEBUG:boto:Headers: {} 2023-05-31T20:53:55.686 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:53:55.686 DEBUG:boto:Port: 8001 2023-05-31T20:53:55.687 DEBUG:boto:Params: {} 2023-05-31T20:53:55.687 DEBUG:boto:Token: None 2023-05-31T20:53:55.687 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:53:55 GMT /pklxkd-59/obj74 2023-05-31T20:53:55.688 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:0XoXQqCcem5JaKzRpC7qwDPruW8= 2023-05-31T20:53:55.688 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:53:55 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:0XoXQqCcem5JaKzRpC7qwDPruW8=', 'Content-Length': '0'} 2023-05-31T20:53:55.690 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:52:14 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a397ba7d6e4c941f-006477b3e3-1c17-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:53:55 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:53:55.690 DEBUG:boto:path=/pklxkd-59/obj74 2023-05-31T20:53:55.691 DEBUG:boto:auth_path=/pklxkd-59/obj74 2023-05-31T20:53:55.691 DEBUG:boto:Method: HEAD 2023-05-31T20:53:55.691 DEBUG:boto:Path: /pklxkd-59/obj74 2023-05-31T20:53:55.692 DEBUG:boto:Data: 2023-05-31T20:53:55.692 DEBUG:boto:Headers: {} 2023-05-31T20:53:55.692 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:53:55.693 DEBUG:boto:Port: 8000 2023-05-31T20:53:55.694 DEBUG:boto:Params: {} 2023-05-31T20:53:55.694 DEBUG:boto:Token: None 2023-05-31T20:53:55.695 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:53:55 GMT /pklxkd-59/obj74 2023-05-31T20:53:55.696 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:0XoXQqCcem5JaKzRpC7qwDPruW8= 2023-05-31T20:53:55.697 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:53:55 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:0XoXQqCcem5JaKzRpC7qwDPruW8=', 'Content-Length': '0'} 2023-05-31T20:53:55.700 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:52:14 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f7f84b3cbc49c71f-006477b3e3-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:53:55 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:53:55.701 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj74 2023-05-31T20:53:55.702 DEBUG:boto:path=/pklxkd-59/obj74 2023-05-31T20:53:55.703 DEBUG:boto:auth_path=/pklxkd-59/obj74 2023-05-31T20:53:55.704 DEBUG:boto:Method: GET 2023-05-31T20:53:55.705 DEBUG:boto:Path: /pklxkd-59/obj74 2023-05-31T20:53:55.705 DEBUG:boto:Data: 2023-05-31T20:53:55.706 DEBUG:boto:Headers: {} 2023-05-31T20:53:55.707 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:53:55.708 DEBUG:boto:Port: 8001 2023-05-31T20:53:55.709 DEBUG:boto:Params: {} 2023-05-31T20:53:55.710 DEBUG:boto:Token: None 2023-05-31T20:53:55.711 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:53:55 GMT /pklxkd-59/obj74 2023-05-31T20:53:55.712 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:hpbykT4h3L1K3/JSEFtT4/tbFqY= 2023-05-31T20:53:55.713 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:53:55 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:hpbykT4h3L1K3/JSEFtT4/tbFqY=', 'Content-Length': '0'} 2023-05-31T20:53:55.715 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:52:14 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000def0091fc611786a-006477b3e3-1c17-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:53:55 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:53:55.716 DEBUG:boto:path=/pklxkd-59/obj74 2023-05-31T20:53:55.717 DEBUG:boto:auth_path=/pklxkd-59/obj74 2023-05-31T20:53:55.718 DEBUG:boto:Method: GET 2023-05-31T20:53:55.719 DEBUG:boto:Path: /pklxkd-59/obj74 2023-05-31T20:53:55.720 DEBUG:boto:Data: 2023-05-31T20:53:55.721 DEBUG:boto:Headers: {} 2023-05-31T20:53:55.722 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:53:55.723 DEBUG:boto:Port: 8000 2023-05-31T20:53:55.723 DEBUG:boto:Params: {} 2023-05-31T20:53:55.724 DEBUG:boto:Token: None 2023-05-31T20:53:55.725 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:53:55 GMT /pklxkd-59/obj74 2023-05-31T20:53:55.726 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:hpbykT4h3L1K3/JSEFtT4/tbFqY= 2023-05-31T20:53:55.727 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:53:55 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:hpbykT4h3L1K3/JSEFtT4/tbFqY=', 'Content-Length': '0'} 2023-05-31T20:53:55.730 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:52:14 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009edf77bf46e20a5c-006477b3e3-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:53:55 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:53:55.731 DEBUG:boto:path=/pklxkd-59/ 2023-05-31T20:53:55.732 DEBUG:boto:auth_path=/pklxkd-59/ 2023-05-31T20:53:55.733 DEBUG:boto:path=/pklxkd-59/?prefix=obj74 2023-05-31T20:53:55.734 DEBUG:boto:auth_path=/pklxkd-59/?prefix=obj74 2023-05-31T20:53:55.734 DEBUG:boto:Method: GET 2023-05-31T20:53:55.735 DEBUG:boto:Path: /pklxkd-59/?prefix=obj74 2023-05-31T20:53:55.736 DEBUG:boto:Data: 2023-05-31T20:53:55.736 DEBUG:boto:Headers: {} 2023-05-31T20:53:55.737 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:53:55.737 DEBUG:boto:Port: 8001 2023-05-31T20:53:55.737 DEBUG:boto:Params: {} 2023-05-31T20:53:55.738 DEBUG:boto:Token: None 2023-05-31T20:53:55.738 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:53:55 GMT /pklxkd-59/ 2023-05-31T20:53:55.738 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:FBsfjfyGgIUCCYA4ReXasZ3hTno= 2023-05-31T20:53:55.739 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:53:55 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:FBsfjfyGgIUCCYA4ReXasZ3hTno=', 'Content-Length': '0'} 2023-05-31T20:53:55.741 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000af1f109f047eebc7-006477b3e3-1c17-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:53:55 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:53:55.742 DEBUG:boto:b'pklxkd-59obj741000falseobj742023-05-31T20:52:14.934Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:53:55.742 DEBUG:boto:path=/pklxkd-59/ 2023-05-31T20:53:55.743 DEBUG:boto:auth_path=/pklxkd-59/ 2023-05-31T20:53:55.743 DEBUG:boto:path=/pklxkd-59/?prefix=obj74 2023-05-31T20:53:55.743 DEBUG:boto:auth_path=/pklxkd-59/?prefix=obj74 2023-05-31T20:53:55.744 DEBUG:boto:Method: GET 2023-05-31T20:53:55.744 DEBUG:boto:Path: /pklxkd-59/?prefix=obj74 2023-05-31T20:53:55.745 DEBUG:boto:Data: 2023-05-31T20:53:55.745 DEBUG:boto:Headers: {} 2023-05-31T20:53:55.745 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:53:55.746 DEBUG:boto:Port: 8000 2023-05-31T20:53:55.746 DEBUG:boto:Params: {} 2023-05-31T20:53:55.746 DEBUG:boto:Token: None 2023-05-31T20:53:55.747 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:53:55 GMT /pklxkd-59/ 2023-05-31T20:53:55.747 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:FBsfjfyGgIUCCYA4ReXasZ3hTno= 2023-05-31T20:53:55.748 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:53:55 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:FBsfjfyGgIUCCYA4ReXasZ3hTno=', 'Content-Length': '0'} 2023-05-31T20:53:55.752 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a887bb79ac2e17ad-006477b3e3-1630-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:53:55 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:53:55.752 DEBUG:boto:b'pklxkd-59obj741000falseobj742023-05-31T20:52:14.934Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:53:55.753 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj76 2023-05-31T20:53:55.753 DEBUG:boto:path=/pklxkd-59/obj76 2023-05-31T20:53:55.753 DEBUG:boto:auth_path=/pklxkd-59/obj76 2023-05-31T20:53:55.754 DEBUG:boto:path=/pklxkd-59/obj76?versionId=null 2023-05-31T20:53:55.754 DEBUG:boto:auth_path=/pklxkd-59/obj76?versionId=null 2023-05-31T20:53:55.755 DEBUG:boto:Method: GET 2023-05-31T20:53:55.755 DEBUG:boto:Path: /pklxkd-59/obj76?versionId=null 2023-05-31T20:53:55.755 DEBUG:boto:Data: 2023-05-31T20:53:55.756 DEBUG:boto:Headers: {} 2023-05-31T20:53:55.756 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:53:55.756 DEBUG:boto:Port: 8001 2023-05-31T20:53:55.757 DEBUG:boto:Params: {} 2023-05-31T20:53:55.757 DEBUG:boto:Token: None 2023-05-31T20:53:55.758 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:53:55 GMT /pklxkd-59/obj76?versionId=null 2023-05-31T20:53:55.758 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:+/JnXAxOyzYFc4S7pnkMSKiyA9E= 2023-05-31T20:53:55.758 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:53:55 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:+/JnXAxOyzYFc4S7pnkMSKiyA9E=', 'Content-Length': '0'} 2023-05-31T20:53:55.760 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:44:50 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000121d7b9352c2875e-006477b3e3-1c17-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:53:55 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:53:55.761 DEBUG:boto:path=/pklxkd-59/obj76 2023-05-31T20:53:55.761 DEBUG:boto:auth_path=/pklxkd-59/obj76 2023-05-31T20:53:55.762 DEBUG:boto:path=/pklxkd-59/obj76?versionId=null 2023-05-31T20:53:55.762 DEBUG:boto:auth_path=/pklxkd-59/obj76?versionId=null 2023-05-31T20:53:55.762 DEBUG:boto:Method: GET 2023-05-31T20:53:55.763 DEBUG:boto:Path: /pklxkd-59/obj76?versionId=null 2023-05-31T20:53:55.763 DEBUG:boto:Data: 2023-05-31T20:53:55.764 DEBUG:boto:Headers: {} 2023-05-31T20:53:55.764 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:53:55.764 DEBUG:boto:Port: 8000 2023-05-31T20:53:55.765 DEBUG:boto:Params: {} 2023-05-31T20:53:55.765 DEBUG:boto:Token: None 2023-05-31T20:53:55.766 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:53:55 GMT /pklxkd-59/obj76?versionId=null 2023-05-31T20:53:55.766 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:+/JnXAxOyzYFc4S7pnkMSKiyA9E= 2023-05-31T20:53:55.767 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:53:55 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:+/JnXAxOyzYFc4S7pnkMSKiyA9E=', 'Content-Length': '0'} 2023-05-31T20:53:55.769 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:44:50 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000053fbe032aa8a02d6-006477b3e3-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:53:55 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:53:55.769 DEBUG:boto:path=/pklxkd-59/obj76 2023-05-31T20:53:55.770 DEBUG:boto:auth_path=/pklxkd-59/obj76 2023-05-31T20:53:55.770 DEBUG:boto:path=/pklxkd-59/obj76?versionId=null 2023-05-31T20:53:55.770 DEBUG:boto:auth_path=/pklxkd-59/obj76?versionId=null 2023-05-31T20:53:55.771 DEBUG:boto:Method: HEAD 2023-05-31T20:53:55.771 DEBUG:boto:Path: /pklxkd-59/obj76?versionId=null 2023-05-31T20:53:55.771 DEBUG:boto:Data: 2023-05-31T20:53:55.772 DEBUG:boto:Headers: {} 2023-05-31T20:53:55.772 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:53:55.772 DEBUG:boto:Port: 8001 2023-05-31T20:53:55.773 DEBUG:boto:Params: {} 2023-05-31T20:53:55.773 DEBUG:boto:Token: None 2023-05-31T20:53:55.774 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:53:55 GMT /pklxkd-59/obj76?versionId=null 2023-05-31T20:53:55.774 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:flYGa5uurbsclQ+kMf/Dq9RORmg= 2023-05-31T20:53:55.774 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:53:55 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:flYGa5uurbsclQ+kMf/Dq9RORmg=', 'Content-Length': '0'} 2023-05-31T20:53:55.776 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:44:50 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000065071668548561ee-006477b3e3-1c17-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:53:55 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:53:55.777 DEBUG:boto:path=/pklxkd-59/obj76 2023-05-31T20:53:55.777 DEBUG:boto:auth_path=/pklxkd-59/obj76 2023-05-31T20:53:55.778 DEBUG:boto:path=/pklxkd-59/obj76?versionId=null 2023-05-31T20:53:55.778 DEBUG:boto:auth_path=/pklxkd-59/obj76?versionId=null 2023-05-31T20:53:55.778 DEBUG:boto:Method: HEAD 2023-05-31T20:53:55.779 DEBUG:boto:Path: /pklxkd-59/obj76?versionId=null 2023-05-31T20:53:55.779 DEBUG:boto:Data: 2023-05-31T20:53:55.779 DEBUG:boto:Headers: {} 2023-05-31T20:53:55.780 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:53:55.780 DEBUG:boto:Port: 8000 2023-05-31T20:53:55.780 DEBUG:boto:Params: {} 2023-05-31T20:53:55.781 DEBUG:boto:Token: None 2023-05-31T20:53:55.781 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:53:55 GMT /pklxkd-59/obj76?versionId=null 2023-05-31T20:53:55.782 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:flYGa5uurbsclQ+kMf/Dq9RORmg= 2023-05-31T20:53:55.782 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:53:55 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:flYGa5uurbsclQ+kMf/Dq9RORmg=', 'Content-Length': '0'} 2023-05-31T20:53:55.784 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:44:50 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d8eec696e4a81ba7-006477b3e3-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:53:55 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:53:55.784 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj76 2023-05-31T20:53:55.785 DEBUG:boto:path=/pklxkd-59/obj76 2023-05-31T20:53:55.785 DEBUG:boto:auth_path=/pklxkd-59/obj76 2023-05-31T20:53:55.786 DEBUG:boto:path=/pklxkd-59/obj76?versionId=null 2023-05-31T20:53:55.786 DEBUG:boto:auth_path=/pklxkd-59/obj76?versionId=null 2023-05-31T20:53:55.786 DEBUG:boto:Method: GET 2023-05-31T20:53:55.787 DEBUG:boto:Path: /pklxkd-59/obj76?versionId=null 2023-05-31T20:53:55.787 DEBUG:boto:Data: 2023-05-31T20:53:55.787 DEBUG:boto:Headers: {} 2023-05-31T20:53:55.788 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:53:55.788 DEBUG:boto:Port: 8001 2023-05-31T20:53:55.788 DEBUG:boto:Params: {} 2023-05-31T20:53:55.789 DEBUG:boto:Token: None 2023-05-31T20:53:55.789 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:53:55 GMT /pklxkd-59/obj76?versionId=null 2023-05-31T20:53:55.790 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:+/JnXAxOyzYFc4S7pnkMSKiyA9E= 2023-05-31T20:53:55.790 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:53:55 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:+/JnXAxOyzYFc4S7pnkMSKiyA9E=', 'Content-Length': '0'} 2023-05-31T20:53:55.792 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:44:50 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a9c90b72f4fbe396-006477b3e3-1c17-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:53:55 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:53:55.832 DEBUG:boto:path=/pklxkd-59/obj76 2023-05-31T20:53:55.833 DEBUG:boto:auth_path=/pklxkd-59/obj76 2023-05-31T20:53:55.833 DEBUG:boto:path=/pklxkd-59/obj76?versionId=null 2023-05-31T20:53:55.833 DEBUG:boto:auth_path=/pklxkd-59/obj76?versionId=null 2023-05-31T20:53:55.834 DEBUG:boto:Method: GET 2023-05-31T20:53:55.834 DEBUG:boto:Path: /pklxkd-59/obj76?versionId=null 2023-05-31T20:53:55.835 DEBUG:boto:Data: 2023-05-31T20:53:55.835 DEBUG:boto:Headers: {} 2023-05-31T20:53:55.835 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:53:55.836 DEBUG:boto:Port: 8000 2023-05-31T20:53:55.836 DEBUG:boto:Params: {} 2023-05-31T20:53:55.837 DEBUG:boto:Token: None 2023-05-31T20:53:55.837 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:53:55 GMT /pklxkd-59/obj76?versionId=null 2023-05-31T20:53:55.838 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:+/JnXAxOyzYFc4S7pnkMSKiyA9E= 2023-05-31T20:53:55.838 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:53:55 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:+/JnXAxOyzYFc4S7pnkMSKiyA9E=', 'Content-Length': '0'} 2023-05-31T20:53:55.840 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:44:50 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004003a0945c396f09-006477b3e3-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:53:55 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:53:55.841 DEBUG:boto:path=/pklxkd-59/ 2023-05-31T20:53:55.841 DEBUG:boto:auth_path=/pklxkd-59/ 2023-05-31T20:53:55.841 DEBUG:boto:path=/pklxkd-59/?prefix=obj76 2023-05-31T20:53:55.842 DEBUG:boto:auth_path=/pklxkd-59/?prefix=obj76 2023-05-31T20:53:55.842 DEBUG:boto:Method: GET 2023-05-31T20:53:55.843 DEBUG:boto:Path: /pklxkd-59/?prefix=obj76 2023-05-31T20:53:55.843 DEBUG:boto:Data: 2023-05-31T20:53:55.843 DEBUG:boto:Headers: {} 2023-05-31T20:53:55.844 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:53:55.844 DEBUG:boto:Port: 8001 2023-05-31T20:53:55.844 DEBUG:boto:Params: {} 2023-05-31T20:53:55.845 DEBUG:boto:Token: None 2023-05-31T20:53:55.845 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:53:55 GMT /pklxkd-59/ 2023-05-31T20:53:55.846 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:FBsfjfyGgIUCCYA4ReXasZ3hTno= 2023-05-31T20:53:55.846 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:53:55 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:FBsfjfyGgIUCCYA4ReXasZ3hTno=', 'Content-Length': '0'} 2023-05-31T20:53:55.848 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000dd1bdd1a2c35ec73-006477b3e3-1c17-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:53:55 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:53:55.849 DEBUG:boto:b'pklxkd-59obj761000falseobj762023-05-31T20:44:50.682Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:53:55.849 DEBUG:boto:path=/pklxkd-59/ 2023-05-31T20:53:55.850 DEBUG:boto:auth_path=/pklxkd-59/ 2023-05-31T20:53:55.850 DEBUG:boto:path=/pklxkd-59/?prefix=obj76 2023-05-31T20:53:55.851 DEBUG:boto:auth_path=/pklxkd-59/?prefix=obj76 2023-05-31T20:53:55.851 DEBUG:boto:Method: GET 2023-05-31T20:53:55.851 DEBUG:boto:Path: /pklxkd-59/?prefix=obj76 2023-05-31T20:53:55.852 DEBUG:boto:Data: 2023-05-31T20:53:55.852 DEBUG:boto:Headers: {} 2023-05-31T20:53:55.853 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:53:55.853 DEBUG:boto:Port: 8000 2023-05-31T20:53:55.854 DEBUG:boto:Params: {} 2023-05-31T20:53:55.854 DEBUG:boto:Token: None 2023-05-31T20:53:55.854 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:53:55 GMT /pklxkd-59/ 2023-05-31T20:53:55.855 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:FBsfjfyGgIUCCYA4ReXasZ3hTno= 2023-05-31T20:53:55.855 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:53:55 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:FBsfjfyGgIUCCYA4ReXasZ3hTno=', 'Content-Length': '0'} 2023-05-31T20:53:55.859 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d9387f81b397d0e1-006477b3e3-1630-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:53:55 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:53:55.860 DEBUG:boto:b'pklxkd-59obj761000falseobj762023-05-31T20:44:50.682Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:53:55.861 DEBUG:boto:path=/pklxkd-59/obj76 2023-05-31T20:53:55.861 DEBUG:boto:auth_path=/pklxkd-59/obj76 2023-05-31T20:53:55.861 DEBUG:boto:Method: HEAD 2023-05-31T20:53:55.862 DEBUG:boto:Path: /pklxkd-59/obj76 2023-05-31T20:53:55.862 DEBUG:boto:Data: 2023-05-31T20:53:55.862 DEBUG:boto:Headers: {} 2023-05-31T20:53:55.863 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:53:55.863 DEBUG:boto:Port: 8001 2023-05-31T20:53:55.863 DEBUG:boto:Params: {} 2023-05-31T20:53:55.864 DEBUG:boto:Token: None 2023-05-31T20:53:55.864 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:53:55 GMT /pklxkd-59/obj76 2023-05-31T20:53:55.865 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:v3qXnXp0lX6b2OePjdJlTTYJz6Y= 2023-05-31T20:53:55.865 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:53:55 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:v3qXnXp0lX6b2OePjdJlTTYJz6Y=', 'Content-Length': '0'} 2023-05-31T20:53:55.867 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:44:50 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c79603f2f565da74-006477b3e3-1c17-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:53:55 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:53:55.868 DEBUG:boto:path=/pklxkd-59/obj76 2023-05-31T20:53:55.868 DEBUG:boto:auth_path=/pklxkd-59/obj76 2023-05-31T20:53:55.868 DEBUG:boto:Method: HEAD 2023-05-31T20:53:55.869 DEBUG:boto:Path: /pklxkd-59/obj76 2023-05-31T20:53:55.869 DEBUG:boto:Data: 2023-05-31T20:53:55.869 DEBUG:boto:Headers: {} 2023-05-31T20:53:55.870 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:53:55.870 DEBUG:boto:Port: 8000 2023-05-31T20:53:55.870 DEBUG:boto:Params: {} 2023-05-31T20:53:55.871 DEBUG:boto:Token: None 2023-05-31T20:53:55.871 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:53:55 GMT /pklxkd-59/obj76 2023-05-31T20:53:55.871 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:v3qXnXp0lX6b2OePjdJlTTYJz6Y= 2023-05-31T20:53:55.872 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:53:55 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:v3qXnXp0lX6b2OePjdJlTTYJz6Y=', 'Content-Length': '0'} 2023-05-31T20:53:55.874 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:44:50 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000065092abc65863e38-006477b3e3-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:53:55 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:53:55.874 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj76 2023-05-31T20:53:55.874 DEBUG:boto:path=/pklxkd-59/obj76 2023-05-31T20:53:55.875 DEBUG:boto:auth_path=/pklxkd-59/obj76 2023-05-31T20:53:55.875 DEBUG:boto:Method: GET 2023-05-31T20:53:55.876 DEBUG:boto:Path: /pklxkd-59/obj76 2023-05-31T20:53:55.876 DEBUG:boto:Data: 2023-05-31T20:53:55.876 DEBUG:boto:Headers: {} 2023-05-31T20:53:55.877 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:53:55.877 DEBUG:boto:Port: 8001 2023-05-31T20:53:55.877 DEBUG:boto:Params: {} 2023-05-31T20:53:55.878 DEBUG:boto:Token: None 2023-05-31T20:53:55.878 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:53:55 GMT /pklxkd-59/obj76 2023-05-31T20:53:55.879 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:pKcN9s/qFf5ZLJKa6FuCscUb4Dg= 2023-05-31T20:53:55.879 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:53:55 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:pKcN9s/qFf5ZLJKa6FuCscUb4Dg=', 'Content-Length': '0'} 2023-05-31T20:53:55.881 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:44:50 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d3f9ab9c08ffd903-006477b3e3-1c17-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:53:55 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:53:55.882 DEBUG:boto:path=/pklxkd-59/obj76 2023-05-31T20:53:55.882 DEBUG:boto:auth_path=/pklxkd-59/obj76 2023-05-31T20:53:55.882 DEBUG:boto:Method: GET 2023-05-31T20:53:55.883 DEBUG:boto:Path: /pklxkd-59/obj76 2023-05-31T20:53:55.883 DEBUG:boto:Data: 2023-05-31T20:53:55.883 DEBUG:boto:Headers: {} 2023-05-31T20:53:55.884 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:53:55.884 DEBUG:boto:Port: 8000 2023-05-31T20:53:55.885 DEBUG:boto:Params: {} 2023-05-31T20:53:55.885 DEBUG:boto:Token: None 2023-05-31T20:53:55.885 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:53:55 GMT /pklxkd-59/obj76 2023-05-31T20:53:55.886 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:pKcN9s/qFf5ZLJKa6FuCscUb4Dg= 2023-05-31T20:53:55.886 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:53:55 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:pKcN9s/qFf5ZLJKa6FuCscUb4Dg=', 'Content-Length': '0'} 2023-05-31T20:53:55.888 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:44:50 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d5d7e67bcb5d7fae-006477b3e3-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:53:55 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:53:55.889 DEBUG:boto:path=/pklxkd-59/ 2023-05-31T20:53:55.889 DEBUG:boto:auth_path=/pklxkd-59/ 2023-05-31T20:53:55.889 DEBUG:boto:path=/pklxkd-59/?prefix=obj76 2023-05-31T20:53:55.890 DEBUG:boto:auth_path=/pklxkd-59/?prefix=obj76 2023-05-31T20:53:55.890 DEBUG:boto:Method: GET 2023-05-31T20:53:55.890 DEBUG:boto:Path: /pklxkd-59/?prefix=obj76 2023-05-31T20:53:55.891 DEBUG:boto:Data: 2023-05-31T20:53:55.891 DEBUG:boto:Headers: {} 2023-05-31T20:53:55.891 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:53:55.892 DEBUG:boto:Port: 8001 2023-05-31T20:53:55.892 DEBUG:boto:Params: {} 2023-05-31T20:53:55.893 DEBUG:boto:Token: None 2023-05-31T20:53:55.893 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:53:55 GMT /pklxkd-59/ 2023-05-31T20:53:55.893 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:FBsfjfyGgIUCCYA4ReXasZ3hTno= 2023-05-31T20:53:55.894 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:53:55 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:FBsfjfyGgIUCCYA4ReXasZ3hTno=', 'Content-Length': '0'} 2023-05-31T20:53:55.896 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000edaf5f94f29ea602-006477b3e3-1c17-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:53:55 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:53:55.940 DEBUG:boto:b'pklxkd-59obj761000falseobj762023-05-31T20:44:50.682Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:53:55.941 DEBUG:boto:path=/pklxkd-59/ 2023-05-31T20:53:55.941 DEBUG:boto:auth_path=/pklxkd-59/ 2023-05-31T20:53:55.942 DEBUG:boto:path=/pklxkd-59/?prefix=obj76 2023-05-31T20:53:55.942 DEBUG:boto:auth_path=/pklxkd-59/?prefix=obj76 2023-05-31T20:53:55.942 DEBUG:boto:Method: GET 2023-05-31T20:53:55.943 DEBUG:boto:Path: /pklxkd-59/?prefix=obj76 2023-05-31T20:53:55.943 DEBUG:boto:Data: 2023-05-31T20:53:55.944 DEBUG:boto:Headers: {} 2023-05-31T20:53:55.944 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:53:55.944 DEBUG:boto:Port: 8000 2023-05-31T20:53:55.945 DEBUG:boto:Params: {} 2023-05-31T20:53:55.945 DEBUG:boto:Token: None 2023-05-31T20:53:55.945 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:53:55 GMT /pklxkd-59/ 2023-05-31T20:53:55.946 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:FBsfjfyGgIUCCYA4ReXasZ3hTno= 2023-05-31T20:53:55.946 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:53:55 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:FBsfjfyGgIUCCYA4ReXasZ3hTno=', 'Content-Length': '0'} 2023-05-31T20:53:55.952 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000008264fd07deae4bf5-006477b3e3-1630-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:53:55 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:53:55.952 DEBUG:boto:b'pklxkd-59obj761000falseobj762023-05-31T20:44:50.682Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:53:55.953 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj85 2023-05-31T20:53:55.953 DEBUG:boto:path=/pklxkd-59/obj85 2023-05-31T20:53:55.954 DEBUG:boto:auth_path=/pklxkd-59/obj85 2023-05-31T20:53:55.954 DEBUG:boto:path=/pklxkd-59/obj85?versionId=null 2023-05-31T20:53:55.954 DEBUG:boto:auth_path=/pklxkd-59/obj85?versionId=null 2023-05-31T20:53:55.955 DEBUG:boto:Method: GET 2023-05-31T20:53:55.955 DEBUG:boto:Path: /pklxkd-59/obj85?versionId=null 2023-05-31T20:53:55.955 DEBUG:boto:Data: 2023-05-31T20:53:55.956 DEBUG:boto:Headers: {} 2023-05-31T20:53:55.956 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:53:55.957 DEBUG:boto:Port: 8001 2023-05-31T20:53:55.957 DEBUG:boto:Params: {} 2023-05-31T20:53:55.957 DEBUG:boto:Token: None 2023-05-31T20:53:55.958 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:53:55 GMT /pklxkd-59/obj85?versionId=null 2023-05-31T20:53:55.958 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:WuiXratVXMDBwXIwT/nKHUcyDKw= 2023-05-31T20:53:55.959 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:53:55 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:WuiXratVXMDBwXIwT/nKHUcyDKw=', 'Content-Length': '0'} 2023-05-31T20:53:55.960 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:43:08 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000fd47a100de1be339-006477b3e3-1c17-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:53:55 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:53:55.961 DEBUG:boto:path=/pklxkd-59/obj85 2023-05-31T20:53:55.961 DEBUG:boto:auth_path=/pklxkd-59/obj85 2023-05-31T20:53:55.962 DEBUG:boto:path=/pklxkd-59/obj85?versionId=null 2023-05-31T20:53:55.962 DEBUG:boto:auth_path=/pklxkd-59/obj85?versionId=null 2023-05-31T20:53:55.962 DEBUG:boto:Method: GET 2023-05-31T20:53:55.963 DEBUG:boto:Path: /pklxkd-59/obj85?versionId=null 2023-05-31T20:53:55.963 DEBUG:boto:Data: 2023-05-31T20:53:55.964 DEBUG:boto:Headers: {} 2023-05-31T20:53:55.964 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:53:55.964 DEBUG:boto:Port: 8000 2023-05-31T20:53:55.965 DEBUG:boto:Params: {} 2023-05-31T20:53:55.965 DEBUG:boto:Token: None 2023-05-31T20:53:55.965 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:53:55 GMT /pklxkd-59/obj85?versionId=null 2023-05-31T20:53:55.966 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:WuiXratVXMDBwXIwT/nKHUcyDKw= 2023-05-31T20:53:55.966 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:53:55 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:WuiXratVXMDBwXIwT/nKHUcyDKw=', 'Content-Length': '0'} 2023-05-31T20:53:55.968 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:43:08 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005085fdaa578aebfa-006477b3e3-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:53:55 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:53:55.969 DEBUG:boto:path=/pklxkd-59/obj85 2023-05-31T20:53:55.969 DEBUG:boto:auth_path=/pklxkd-59/obj85 2023-05-31T20:53:55.970 DEBUG:boto:path=/pklxkd-59/obj85?versionId=null 2023-05-31T20:53:55.970 DEBUG:boto:auth_path=/pklxkd-59/obj85?versionId=null 2023-05-31T20:53:55.970 DEBUG:boto:Method: HEAD 2023-05-31T20:53:55.971 DEBUG:boto:Path: /pklxkd-59/obj85?versionId=null 2023-05-31T20:53:55.971 DEBUG:boto:Data: 2023-05-31T20:53:55.971 DEBUG:boto:Headers: {} 2023-05-31T20:53:55.972 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:53:55.972 DEBUG:boto:Port: 8001 2023-05-31T20:53:55.972 DEBUG:boto:Params: {} 2023-05-31T20:53:55.973 DEBUG:boto:Token: None 2023-05-31T20:53:55.973 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:53:55 GMT /pklxkd-59/obj85?versionId=null 2023-05-31T20:53:55.973 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:4Y5UFwA6M+XKDINrM1X+PIQ8PCo= 2023-05-31T20:53:55.974 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:53:55 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:4Y5UFwA6M+XKDINrM1X+PIQ8PCo=', 'Content-Length': '0'} 2023-05-31T20:53:55.976 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:43:08 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000eaf7e0a4c39ee033-006477b3e3-1c17-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:53:55 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:53:55.976 DEBUG:boto:path=/pklxkd-59/obj85 2023-05-31T20:53:55.977 DEBUG:boto:auth_path=/pklxkd-59/obj85 2023-05-31T20:53:55.977 DEBUG:boto:path=/pklxkd-59/obj85?versionId=null 2023-05-31T20:53:55.977 DEBUG:boto:auth_path=/pklxkd-59/obj85?versionId=null 2023-05-31T20:53:55.978 DEBUG:boto:Method: HEAD 2023-05-31T20:53:55.978 DEBUG:boto:Path: /pklxkd-59/obj85?versionId=null 2023-05-31T20:53:55.978 DEBUG:boto:Data: 2023-05-31T20:53:55.979 DEBUG:boto:Headers: {} 2023-05-31T20:53:55.979 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:53:55.979 DEBUG:boto:Port: 8000 2023-05-31T20:53:55.980 DEBUG:boto:Params: {} 2023-05-31T20:53:55.980 DEBUG:boto:Token: None 2023-05-31T20:53:55.980 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:53:55 GMT /pklxkd-59/obj85?versionId=null 2023-05-31T20:53:55.981 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:4Y5UFwA6M+XKDINrM1X+PIQ8PCo= 2023-05-31T20:53:55.981 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:53:55 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:4Y5UFwA6M+XKDINrM1X+PIQ8PCo=', 'Content-Length': '0'} 2023-05-31T20:53:55.983 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:43:08 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009db21c8bba1a51c4-006477b3e3-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:53:55 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:53:55.983 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj85 2023-05-31T20:53:55.984 DEBUG:boto:path=/pklxkd-59/obj85 2023-05-31T20:53:55.984 DEBUG:boto:auth_path=/pklxkd-59/obj85 2023-05-31T20:53:55.985 DEBUG:boto:path=/pklxkd-59/obj85?versionId=null 2023-05-31T20:53:55.985 DEBUG:boto:auth_path=/pklxkd-59/obj85?versionId=null 2023-05-31T20:53:55.985 DEBUG:boto:Method: GET 2023-05-31T20:53:55.986 DEBUG:boto:Path: /pklxkd-59/obj85?versionId=null 2023-05-31T20:53:55.986 DEBUG:boto:Data: 2023-05-31T20:53:55.986 DEBUG:boto:Headers: {} 2023-05-31T20:53:55.987 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:53:55.987 DEBUG:boto:Port: 8001 2023-05-31T20:53:55.987 DEBUG:boto:Params: {} 2023-05-31T20:53:55.988 DEBUG:boto:Token: None 2023-05-31T20:53:55.988 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:53:55 GMT /pklxkd-59/obj85?versionId=null 2023-05-31T20:53:55.989 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:WuiXratVXMDBwXIwT/nKHUcyDKw= 2023-05-31T20:53:55.989 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:53:55 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:WuiXratVXMDBwXIwT/nKHUcyDKw=', 'Content-Length': '0'} 2023-05-31T20:53:55.991 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:43:08 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000297b343209535a4c-006477b3e3-1c17-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:53:55 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:53:55.992 DEBUG:boto:path=/pklxkd-59/obj85 2023-05-31T20:53:55.992 DEBUG:boto:auth_path=/pklxkd-59/obj85 2023-05-31T20:53:55.992 DEBUG:boto:path=/pklxkd-59/obj85?versionId=null 2023-05-31T20:53:55.993 DEBUG:boto:auth_path=/pklxkd-59/obj85?versionId=null 2023-05-31T20:53:55.993 DEBUG:boto:Method: GET 2023-05-31T20:53:55.993 DEBUG:boto:Path: /pklxkd-59/obj85?versionId=null 2023-05-31T20:53:55.994 DEBUG:boto:Data: 2023-05-31T20:53:55.994 DEBUG:boto:Headers: {} 2023-05-31T20:53:55.994 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:53:55.995 DEBUG:boto:Port: 8000 2023-05-31T20:53:55.995 DEBUG:boto:Params: {} 2023-05-31T20:53:55.996 DEBUG:boto:Token: None 2023-05-31T20:53:55.996 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:53:55 GMT /pklxkd-59/obj85?versionId=null 2023-05-31T20:53:55.996 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:WuiXratVXMDBwXIwT/nKHUcyDKw= 2023-05-31T20:53:55.997 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:53:55 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:WuiXratVXMDBwXIwT/nKHUcyDKw=', 'Content-Length': '0'} 2023-05-31T20:53:55.998 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:43:08 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000218e94282907f147-006477b3e3-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:53:55 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:53:55.999 DEBUG:boto:path=/pklxkd-59/ 2023-05-31T20:53:55.999 DEBUG:boto:auth_path=/pklxkd-59/ 2023-05-31T20:53:56.000 DEBUG:boto:path=/pklxkd-59/?prefix=obj85 2023-05-31T20:53:56.000 DEBUG:boto:auth_path=/pklxkd-59/?prefix=obj85 2023-05-31T20:53:56.001 DEBUG:boto:Method: GET 2023-05-31T20:53:56.001 DEBUG:boto:Path: /pklxkd-59/?prefix=obj85 2023-05-31T20:53:56.001 DEBUG:boto:Data: 2023-05-31T20:53:56.002 DEBUG:boto:Headers: {} 2023-05-31T20:53:56.002 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:53:56.002 DEBUG:boto:Port: 8001 2023-05-31T20:53:56.003 DEBUG:boto:Params: {} 2023-05-31T20:53:56.003 DEBUG:boto:Token: None 2023-05-31T20:53:56.003 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:53:56 GMT /pklxkd-59/ 2023-05-31T20:53:56.004 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:VxrsRCUGt15KLeSgquVd6DPBfJs= 2023-05-31T20:53:56.004 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:53:56 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:VxrsRCUGt15KLeSgquVd6DPBfJs=', 'Content-Length': '0'} 2023-05-31T20:53:56.007 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000073640ba28ce14313-006477b3e4-1c17-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:53:56 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:53:56.007 DEBUG:boto:b'pklxkd-59obj851000falseobj852023-05-31T20:43:08.086Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:53:56.008 DEBUG:boto:path=/pklxkd-59/ 2023-05-31T20:53:56.008 DEBUG:boto:auth_path=/pklxkd-59/ 2023-05-31T20:53:56.009 DEBUG:boto:path=/pklxkd-59/?prefix=obj85 2023-05-31T20:53:56.009 DEBUG:boto:auth_path=/pklxkd-59/?prefix=obj85 2023-05-31T20:53:56.009 DEBUG:boto:Method: GET 2023-05-31T20:53:56.010 DEBUG:boto:Path: /pklxkd-59/?prefix=obj85 2023-05-31T20:53:56.010 DEBUG:boto:Data: 2023-05-31T20:53:56.010 DEBUG:boto:Headers: {} 2023-05-31T20:53:56.011 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:53:56.011 DEBUG:boto:Port: 8000 2023-05-31T20:53:56.011 DEBUG:boto:Params: {} 2023-05-31T20:53:56.012 DEBUG:boto:Token: None 2023-05-31T20:53:56.012 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:53:56 GMT /pklxkd-59/ 2023-05-31T20:53:56.013 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:VxrsRCUGt15KLeSgquVd6DPBfJs= 2023-05-31T20:53:56.013 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:53:56 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:VxrsRCUGt15KLeSgquVd6DPBfJs=', 'Content-Length': '0'} 2023-05-31T20:53:56.017 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000007b2ad43604fac030-006477b3e4-1630-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:53:56 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:53:56.018 DEBUG:boto:b'pklxkd-59obj851000falseobj852023-05-31T20:43:08.086Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:53:56.018 DEBUG:boto:path=/pklxkd-59/obj85 2023-05-31T20:53:56.019 DEBUG:boto:auth_path=/pklxkd-59/obj85 2023-05-31T20:53:56.019 DEBUG:boto:Method: HEAD 2023-05-31T20:53:56.019 DEBUG:boto:Path: /pklxkd-59/obj85 2023-05-31T20:53:56.020 DEBUG:boto:Data: 2023-05-31T20:53:56.020 DEBUG:boto:Headers: {} 2023-05-31T20:53:56.020 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:53:56.021 DEBUG:boto:Port: 8001 2023-05-31T20:53:56.021 DEBUG:boto:Params: {} 2023-05-31T20:53:56.021 DEBUG:boto:Token: None 2023-05-31T20:53:56.022 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:53:56 GMT /pklxkd-59/obj85 2023-05-31T20:53:56.022 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:xke/MI9auc/lbVaQNR1oYVLlG9o= 2023-05-31T20:53:56.023 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:53:56 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:xke/MI9auc/lbVaQNR1oYVLlG9o=', 'Content-Length': '0'} 2023-05-31T20:53:56.024 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:43:08 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000103bf3e95f25504c-006477b3e4-1c17-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:53:56 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:53:56.025 DEBUG:boto:path=/pklxkd-59/obj85 2023-05-31T20:53:56.025 DEBUG:boto:auth_path=/pklxkd-59/obj85 2023-05-31T20:53:56.026 DEBUG:boto:Method: HEAD 2023-05-31T20:53:56.026 DEBUG:boto:Path: /pklxkd-59/obj85 2023-05-31T20:53:56.026 DEBUG:boto:Data: 2023-05-31T20:53:56.027 DEBUG:boto:Headers: {} 2023-05-31T20:53:56.027 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:53:56.027 DEBUG:boto:Port: 8000 2023-05-31T20:53:56.028 DEBUG:boto:Params: {} 2023-05-31T20:53:56.028 DEBUG:boto:Token: None 2023-05-31T20:53:56.028 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:53:56 GMT /pklxkd-59/obj85 2023-05-31T20:53:56.029 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:xke/MI9auc/lbVaQNR1oYVLlG9o= 2023-05-31T20:53:56.029 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:53:56 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:xke/MI9auc/lbVaQNR1oYVLlG9o=', 'Content-Length': '0'} 2023-05-31T20:53:56.031 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:43:08 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009eab6330914f2d76-006477b3e4-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:53:56 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:53:56.031 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj85 2023-05-31T20:53:56.032 DEBUG:boto:path=/pklxkd-59/obj85 2023-05-31T20:53:56.032 DEBUG:boto:auth_path=/pklxkd-59/obj85 2023-05-31T20:53:56.033 DEBUG:boto:Method: GET 2023-05-31T20:53:56.033 DEBUG:boto:Path: /pklxkd-59/obj85 2023-05-31T20:53:56.033 DEBUG:boto:Data: 2023-05-31T20:53:56.034 DEBUG:boto:Headers: {} 2023-05-31T20:53:56.034 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:53:56.034 DEBUG:boto:Port: 8001 2023-05-31T20:53:56.035 DEBUG:boto:Params: {} 2023-05-31T20:53:56.035 DEBUG:boto:Token: None 2023-05-31T20:53:56.036 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:53:56 GMT /pklxkd-59/obj85 2023-05-31T20:53:56.036 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:rjRKLmergj2/UAn6noyg4TTteUQ= 2023-05-31T20:53:56.036 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:53:56 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:rjRKLmergj2/UAn6noyg4TTteUQ=', 'Content-Length': '0'} 2023-05-31T20:53:56.038 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:43:08 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000710084496dd995e6-006477b3e4-1c17-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:53:56 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:53:56.080 DEBUG:boto:path=/pklxkd-59/obj85 2023-05-31T20:53:56.081 DEBUG:boto:auth_path=/pklxkd-59/obj85 2023-05-31T20:53:56.081 DEBUG:boto:Method: GET 2023-05-31T20:53:56.081 DEBUG:boto:Path: /pklxkd-59/obj85 2023-05-31T20:53:56.082 DEBUG:boto:Data: 2023-05-31T20:53:56.082 DEBUG:boto:Headers: {} 2023-05-31T20:53:56.082 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:53:56.083 DEBUG:boto:Port: 8000 2023-05-31T20:53:56.083 DEBUG:boto:Params: {} 2023-05-31T20:53:56.084 DEBUG:boto:Token: None 2023-05-31T20:53:56.084 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:53:56 GMT /pklxkd-59/obj85 2023-05-31T20:53:56.084 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:rjRKLmergj2/UAn6noyg4TTteUQ= 2023-05-31T20:53:56.085 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:53:56 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:rjRKLmergj2/UAn6noyg4TTteUQ=', 'Content-Length': '0'} 2023-05-31T20:53:56.087 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:43:08 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000064d3d1bb898f425d-006477b3e4-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:53:56 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:53:56.087 DEBUG:boto:path=/pklxkd-59/ 2023-05-31T20:53:56.088 DEBUG:boto:auth_path=/pklxkd-59/ 2023-05-31T20:53:56.088 DEBUG:boto:path=/pklxkd-59/?prefix=obj85 2023-05-31T20:53:56.088 DEBUG:boto:auth_path=/pklxkd-59/?prefix=obj85 2023-05-31T20:53:56.089 DEBUG:boto:Method: GET 2023-05-31T20:53:56.089 DEBUG:boto:Path: /pklxkd-59/?prefix=obj85 2023-05-31T20:53:56.089 DEBUG:boto:Data: 2023-05-31T20:53:56.090 DEBUG:boto:Headers: {} 2023-05-31T20:53:56.090 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:53:56.091 DEBUG:boto:Port: 8001 2023-05-31T20:53:56.091 DEBUG:boto:Params: {} 2023-05-31T20:53:56.091 DEBUG:boto:Token: None 2023-05-31T20:53:56.092 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:53:56 GMT /pklxkd-59/ 2023-05-31T20:53:56.092 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:VxrsRCUGt15KLeSgquVd6DPBfJs= 2023-05-31T20:53:56.093 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:53:56 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:VxrsRCUGt15KLeSgquVd6DPBfJs=', 'Content-Length': '0'} 2023-05-31T20:53:56.095 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000005971b7910bcfa541-006477b3e4-1c17-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:53:56 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:53:56.095 DEBUG:boto:b'pklxkd-59obj851000falseobj852023-05-31T20:43:08.086Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:53:56.096 DEBUG:boto:path=/pklxkd-59/ 2023-05-31T20:53:56.096 DEBUG:boto:auth_path=/pklxkd-59/ 2023-05-31T20:53:56.097 DEBUG:boto:path=/pklxkd-59/?prefix=obj85 2023-05-31T20:53:56.097 DEBUG:boto:auth_path=/pklxkd-59/?prefix=obj85 2023-05-31T20:53:56.097 DEBUG:boto:Method: GET 2023-05-31T20:53:56.098 DEBUG:boto:Path: /pklxkd-59/?prefix=obj85 2023-05-31T20:53:56.098 DEBUG:boto:Data: 2023-05-31T20:53:56.098 DEBUG:boto:Headers: {} 2023-05-31T20:53:56.099 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:53:56.099 DEBUG:boto:Port: 8000 2023-05-31T20:53:56.099 DEBUG:boto:Params: {} 2023-05-31T20:53:56.100 DEBUG:boto:Token: None 2023-05-31T20:53:56.100 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:53:56 GMT /pklxkd-59/ 2023-05-31T20:53:56.101 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:VxrsRCUGt15KLeSgquVd6DPBfJs= 2023-05-31T20:53:56.101 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:53:56 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:VxrsRCUGt15KLeSgquVd6DPBfJs=', 'Content-Length': '0'} 2023-05-31T20:53:56.107 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e7529c2923cdde43-006477b3e4-1630-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:53:56 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:53:56.107 DEBUG:boto:b'pklxkd-59obj851000falseobj852023-05-31T20:43:08.086Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:53:56.108 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj87 2023-05-31T20:53:56.108 DEBUG:boto:path=/pklxkd-59/obj87 2023-05-31T20:53:56.109 DEBUG:boto:auth_path=/pklxkd-59/obj87 2023-05-31T20:53:56.109 DEBUG:boto:path=/pklxkd-59/obj87?versionId=null 2023-05-31T20:53:56.109 DEBUG:boto:auth_path=/pklxkd-59/obj87?versionId=null 2023-05-31T20:53:56.110 DEBUG:boto:Method: GET 2023-05-31T20:53:56.110 DEBUG:boto:Path: /pklxkd-59/obj87?versionId=null 2023-05-31T20:53:56.110 DEBUG:boto:Data: 2023-05-31T20:53:56.111 DEBUG:boto:Headers: {} 2023-05-31T20:53:56.111 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:53:56.111 DEBUG:boto:Port: 8001 2023-05-31T20:53:56.112 DEBUG:boto:Params: {} 2023-05-31T20:53:56.112 DEBUG:boto:Token: None 2023-05-31T20:53:56.113 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:53:56 GMT /pklxkd-59/obj87?versionId=null 2023-05-31T20:53:56.113 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:81Bdgqb+opjdwsm7QisZzuiH6iA= 2023-05-31T20:53:56.113 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:53:56 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:81Bdgqb+opjdwsm7QisZzuiH6iA=', 'Content-Length': '0'} 2023-05-31T20:53:56.115 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:48:13 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ff14fd6379bf010b-006477b3e4-1c17-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:53:56 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:53:56.116 DEBUG:boto:path=/pklxkd-59/obj87 2023-05-31T20:53:56.117 DEBUG:boto:auth_path=/pklxkd-59/obj87 2023-05-31T20:53:56.117 DEBUG:boto:path=/pklxkd-59/obj87?versionId=null 2023-05-31T20:53:56.117 DEBUG:boto:auth_path=/pklxkd-59/obj87?versionId=null 2023-05-31T20:53:56.118 DEBUG:boto:Method: GET 2023-05-31T20:53:56.118 DEBUG:boto:Path: /pklxkd-59/obj87?versionId=null 2023-05-31T20:53:56.118 DEBUG:boto:Data: 2023-05-31T20:53:56.119 DEBUG:boto:Headers: {} 2023-05-31T20:53:56.119 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:53:56.119 DEBUG:boto:Port: 8000 2023-05-31T20:53:56.120 DEBUG:boto:Params: {} 2023-05-31T20:53:56.120 DEBUG:boto:Token: None 2023-05-31T20:53:56.121 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:53:56 GMT /pklxkd-59/obj87?versionId=null 2023-05-31T20:53:56.121 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:81Bdgqb+opjdwsm7QisZzuiH6iA= 2023-05-31T20:53:56.121 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:53:56 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:81Bdgqb+opjdwsm7QisZzuiH6iA=', 'Content-Length': '0'} 2023-05-31T20:53:56.123 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:48:13 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000039269313426a3c0e-006477b3e4-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:53:56 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:53:56.124 DEBUG:boto:path=/pklxkd-59/obj87 2023-05-31T20:53:56.124 DEBUG:boto:auth_path=/pklxkd-59/obj87 2023-05-31T20:53:56.125 DEBUG:boto:path=/pklxkd-59/obj87?versionId=null 2023-05-31T20:53:56.125 DEBUG:boto:auth_path=/pklxkd-59/obj87?versionId=null 2023-05-31T20:53:56.125 DEBUG:boto:Method: HEAD 2023-05-31T20:53:56.126 DEBUG:boto:Path: /pklxkd-59/obj87?versionId=null 2023-05-31T20:53:56.126 DEBUG:boto:Data: 2023-05-31T20:53:56.126 DEBUG:boto:Headers: {} 2023-05-31T20:53:56.127 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:53:56.127 DEBUG:boto:Port: 8001 2023-05-31T20:53:56.127 DEBUG:boto:Params: {} 2023-05-31T20:53:56.128 DEBUG:boto:Token: None 2023-05-31T20:53:56.128 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:53:56 GMT /pklxkd-59/obj87?versionId=null 2023-05-31T20:53:56.129 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:7xl+99UWa9i6EvgeNcb1VuWcZgA= 2023-05-31T20:53:56.129 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:53:56 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:7xl+99UWa9i6EvgeNcb1VuWcZgA=', 'Content-Length': '0'} 2023-05-31T20:53:56.131 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:48:13 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000bef8ee90960bf9fe-006477b3e4-1c17-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:53:56 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:53:56.131 DEBUG:boto:path=/pklxkd-59/obj87 2023-05-31T20:53:56.132 DEBUG:boto:auth_path=/pklxkd-59/obj87 2023-05-31T20:53:56.132 DEBUG:boto:path=/pklxkd-59/obj87?versionId=null 2023-05-31T20:53:56.132 DEBUG:boto:auth_path=/pklxkd-59/obj87?versionId=null 2023-05-31T20:53:56.133 DEBUG:boto:Method: HEAD 2023-05-31T20:53:56.133 DEBUG:boto:Path: /pklxkd-59/obj87?versionId=null 2023-05-31T20:53:56.133 DEBUG:boto:Data: 2023-05-31T20:53:56.134 DEBUG:boto:Headers: {} 2023-05-31T20:53:56.134 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:53:56.134 DEBUG:boto:Port: 8000 2023-05-31T20:53:56.135 DEBUG:boto:Params: {} 2023-05-31T20:53:56.135 DEBUG:boto:Token: None 2023-05-31T20:53:56.135 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:53:56 GMT /pklxkd-59/obj87?versionId=null 2023-05-31T20:53:56.136 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:7xl+99UWa9i6EvgeNcb1VuWcZgA= 2023-05-31T20:53:56.136 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:53:56 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:7xl+99UWa9i6EvgeNcb1VuWcZgA=', 'Content-Length': '0'} 2023-05-31T20:53:56.138 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:48:13 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003b800228341c1989-006477b3e4-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:53:56 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:53:56.139 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj87 2023-05-31T20:53:56.139 DEBUG:boto:path=/pklxkd-59/obj87 2023-05-31T20:53:56.139 DEBUG:boto:auth_path=/pklxkd-59/obj87 2023-05-31T20:53:56.140 DEBUG:boto:path=/pklxkd-59/obj87?versionId=null 2023-05-31T20:53:56.140 DEBUG:boto:auth_path=/pklxkd-59/obj87?versionId=null 2023-05-31T20:53:56.140 DEBUG:boto:Method: GET 2023-05-31T20:53:56.141 DEBUG:boto:Path: /pklxkd-59/obj87?versionId=null 2023-05-31T20:53:56.141 DEBUG:boto:Data: 2023-05-31T20:53:56.142 DEBUG:boto:Headers: {} 2023-05-31T20:53:56.142 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:53:56.142 DEBUG:boto:Port: 8001 2023-05-31T20:53:56.143 DEBUG:boto:Params: {} 2023-05-31T20:53:56.143 DEBUG:boto:Token: None 2023-05-31T20:53:56.143 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:53:56 GMT /pklxkd-59/obj87?versionId=null 2023-05-31T20:53:56.144 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:81Bdgqb+opjdwsm7QisZzuiH6iA= 2023-05-31T20:53:56.144 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:53:56 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:81Bdgqb+opjdwsm7QisZzuiH6iA=', 'Content-Length': '0'} 2023-05-31T20:53:56.146 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:48:13 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b4895a18f8d19c99-006477b3e4-1c17-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:53:56 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:53:56.147 DEBUG:boto:path=/pklxkd-59/obj87 2023-05-31T20:53:56.147 DEBUG:boto:auth_path=/pklxkd-59/obj87 2023-05-31T20:53:56.148 DEBUG:boto:path=/pklxkd-59/obj87?versionId=null 2023-05-31T20:53:56.148 DEBUG:boto:auth_path=/pklxkd-59/obj87?versionId=null 2023-05-31T20:53:56.148 DEBUG:boto:Method: GET 2023-05-31T20:53:56.149 DEBUG:boto:Path: /pklxkd-59/obj87?versionId=null 2023-05-31T20:53:56.149 DEBUG:boto:Data: 2023-05-31T20:53:56.150 DEBUG:boto:Headers: {} 2023-05-31T20:53:56.150 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:53:56.150 DEBUG:boto:Port: 8000 2023-05-31T20:53:56.151 DEBUG:boto:Params: {} 2023-05-31T20:53:56.151 DEBUG:boto:Token: None 2023-05-31T20:53:56.152 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:53:56 GMT /pklxkd-59/obj87?versionId=null 2023-05-31T20:53:56.152 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:81Bdgqb+opjdwsm7QisZzuiH6iA= 2023-05-31T20:53:56.152 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:53:56 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:81Bdgqb+opjdwsm7QisZzuiH6iA=', 'Content-Length': '0'} 2023-05-31T20:53:56.154 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:48:13 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008967921b37b66549-006477b3e4-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:53:56 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:53:56.155 DEBUG:boto:path=/pklxkd-59/ 2023-05-31T20:53:56.155 DEBUG:boto:auth_path=/pklxkd-59/ 2023-05-31T20:53:56.156 DEBUG:boto:path=/pklxkd-59/?prefix=obj87 2023-05-31T20:53:56.156 DEBUG:boto:auth_path=/pklxkd-59/?prefix=obj87 2023-05-31T20:53:56.156 DEBUG:boto:Method: GET 2023-05-31T20:53:56.157 DEBUG:boto:Path: /pklxkd-59/?prefix=obj87 2023-05-31T20:53:56.157 DEBUG:boto:Data: 2023-05-31T20:53:56.158 DEBUG:boto:Headers: {} 2023-05-31T20:53:56.158 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:53:56.158 DEBUG:boto:Port: 8001 2023-05-31T20:53:56.159 DEBUG:boto:Params: {} 2023-05-31T20:53:56.159 DEBUG:boto:Token: None 2023-05-31T20:53:56.159 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:53:56 GMT /pklxkd-59/ 2023-05-31T20:53:56.160 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:VxrsRCUGt15KLeSgquVd6DPBfJs= 2023-05-31T20:53:56.160 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:53:56 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:VxrsRCUGt15KLeSgquVd6DPBfJs=', 'Content-Length': '0'} 2023-05-31T20:53:56.163 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e5de2c04a2c2c4b8-006477b3e4-1c17-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:53:56 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:53:56.163 DEBUG:boto:b'pklxkd-59obj871000falseobj872023-05-31T20:48:13.031Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:53:56.164 DEBUG:boto:path=/pklxkd-59/ 2023-05-31T20:53:56.164 DEBUG:boto:auth_path=/pklxkd-59/ 2023-05-31T20:53:56.165 DEBUG:boto:path=/pklxkd-59/?prefix=obj87 2023-05-31T20:53:56.165 DEBUG:boto:auth_path=/pklxkd-59/?prefix=obj87 2023-05-31T20:53:56.165 DEBUG:boto:Method: GET 2023-05-31T20:53:56.166 DEBUG:boto:Path: /pklxkd-59/?prefix=obj87 2023-05-31T20:53:56.166 DEBUG:boto:Data: 2023-05-31T20:53:56.166 DEBUG:boto:Headers: {} 2023-05-31T20:53:56.167 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:53:56.167 DEBUG:boto:Port: 8000 2023-05-31T20:53:56.167 DEBUG:boto:Params: {} 2023-05-31T20:53:56.168 DEBUG:boto:Token: None 2023-05-31T20:53:56.168 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:53:56 GMT /pklxkd-59/ 2023-05-31T20:53:56.169 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:VxrsRCUGt15KLeSgquVd6DPBfJs= 2023-05-31T20:53:56.169 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:53:56 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:VxrsRCUGt15KLeSgquVd6DPBfJs=', 'Content-Length': '0'} 2023-05-31T20:53:56.173 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000007a0366c502944957-006477b3e4-1630-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:53:56 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:53:56.174 DEBUG:boto:b'pklxkd-59obj871000falseobj872023-05-31T20:48:13.031Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:53:56.174 DEBUG:boto:path=/pklxkd-59/obj87 2023-05-31T20:53:56.175 DEBUG:boto:auth_path=/pklxkd-59/obj87 2023-05-31T20:53:56.175 DEBUG:boto:Method: HEAD 2023-05-31T20:53:56.175 DEBUG:boto:Path: /pklxkd-59/obj87 2023-05-31T20:53:56.176 DEBUG:boto:Data: 2023-05-31T20:53:56.176 DEBUG:boto:Headers: {} 2023-05-31T20:53:56.176 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:53:56.177 DEBUG:boto:Port: 8001 2023-05-31T20:53:56.177 DEBUG:boto:Params: {} 2023-05-31T20:53:56.177 DEBUG:boto:Token: None 2023-05-31T20:53:56.178 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:53:56 GMT /pklxkd-59/obj87 2023-05-31T20:53:56.178 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:rGN0/EG1Hq1siEjWtH9LEyr8CyE= 2023-05-31T20:53:56.179 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:53:56 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:rGN0/EG1Hq1siEjWtH9LEyr8CyE=', 'Content-Length': '0'} 2023-05-31T20:53:56.180 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:48:13 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000fee04d72f47c13d0-006477b3e4-1c17-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:53:56 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:53:56.181 DEBUG:boto:path=/pklxkd-59/obj87 2023-05-31T20:53:56.181 DEBUG:boto:auth_path=/pklxkd-59/obj87 2023-05-31T20:53:56.182 DEBUG:boto:Method: HEAD 2023-05-31T20:53:56.182 DEBUG:boto:Path: /pklxkd-59/obj87 2023-05-31T20:53:56.182 DEBUG:boto:Data: 2023-05-31T20:53:56.183 DEBUG:boto:Headers: {} 2023-05-31T20:53:56.183 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:53:56.183 DEBUG:boto:Port: 8000 2023-05-31T20:53:56.184 DEBUG:boto:Params: {} 2023-05-31T20:53:56.184 DEBUG:boto:Token: None 2023-05-31T20:53:56.184 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:53:56 GMT /pklxkd-59/obj87 2023-05-31T20:53:56.185 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:rGN0/EG1Hq1siEjWtH9LEyr8CyE= 2023-05-31T20:53:56.185 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:53:56 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:rGN0/EG1Hq1siEjWtH9LEyr8CyE=', 'Content-Length': '0'} 2023-05-31T20:53:56.187 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:48:13 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c670a361802d7ab8-006477b3e4-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:53:56 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:53:56.188 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj87 2023-05-31T20:53:56.188 DEBUG:boto:path=/pklxkd-59/obj87 2023-05-31T20:53:56.188 DEBUG:boto:auth_path=/pklxkd-59/obj87 2023-05-31T20:53:56.189 DEBUG:boto:Method: GET 2023-05-31T20:53:56.189 DEBUG:boto:Path: /pklxkd-59/obj87 2023-05-31T20:53:56.190 DEBUG:boto:Data: 2023-05-31T20:53:56.190 DEBUG:boto:Headers: {} 2023-05-31T20:53:56.190 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:53:56.191 DEBUG:boto:Port: 8001 2023-05-31T20:53:56.191 DEBUG:boto:Params: {} 2023-05-31T20:53:56.191 DEBUG:boto:Token: None 2023-05-31T20:53:56.192 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:53:56 GMT /pklxkd-59/obj87 2023-05-31T20:53:56.192 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:j0JT+huq2gncr2o/vnIGkIvQsvY= 2023-05-31T20:53:56.193 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:53:56 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:j0JT+huq2gncr2o/vnIGkIvQsvY=', 'Content-Length': '0'} 2023-05-31T20:53:56.194 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:48:13 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b167eabdf52bd815-006477b3e4-1c17-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:53:56 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:53:56.195 DEBUG:boto:path=/pklxkd-59/obj87 2023-05-31T20:53:56.196 DEBUG:boto:auth_path=/pklxkd-59/obj87 2023-05-31T20:53:56.196 DEBUG:boto:Method: GET 2023-05-31T20:53:56.196 DEBUG:boto:Path: /pklxkd-59/obj87 2023-05-31T20:53:56.197 DEBUG:boto:Data: 2023-05-31T20:53:56.197 DEBUG:boto:Headers: {} 2023-05-31T20:53:56.197 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:53:56.198 DEBUG:boto:Port: 8000 2023-05-31T20:53:56.198 DEBUG:boto:Params: {} 2023-05-31T20:53:56.198 DEBUG:boto:Token: None 2023-05-31T20:53:56.199 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:53:56 GMT /pklxkd-59/obj87 2023-05-31T20:53:56.199 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:j0JT+huq2gncr2o/vnIGkIvQsvY= 2023-05-31T20:53:56.200 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:53:56 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:j0JT+huq2gncr2o/vnIGkIvQsvY=', 'Content-Length': '0'} 2023-05-31T20:53:56.202 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:48:13 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005612fb0597f1bef9-006477b3e4-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:53:56 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:53:56.202 DEBUG:boto:path=/pklxkd-59/ 2023-05-31T20:53:56.203 DEBUG:boto:auth_path=/pklxkd-59/ 2023-05-31T20:53:56.203 DEBUG:boto:path=/pklxkd-59/?prefix=obj87 2023-05-31T20:53:56.203 DEBUG:boto:auth_path=/pklxkd-59/?prefix=obj87 2023-05-31T20:53:56.204 DEBUG:boto:Method: GET 2023-05-31T20:53:56.204 DEBUG:boto:Path: /pklxkd-59/?prefix=obj87 2023-05-31T20:53:56.204 DEBUG:boto:Data: 2023-05-31T20:53:56.205 DEBUG:boto:Headers: {} 2023-05-31T20:53:56.205 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:53:56.206 DEBUG:boto:Port: 8001 2023-05-31T20:53:56.206 DEBUG:boto:Params: {} 2023-05-31T20:53:56.206 DEBUG:boto:Token: None 2023-05-31T20:53:56.207 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:53:56 GMT /pklxkd-59/ 2023-05-31T20:53:56.207 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:VxrsRCUGt15KLeSgquVd6DPBfJs= 2023-05-31T20:53:56.208 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:53:56 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:VxrsRCUGt15KLeSgquVd6DPBfJs=', 'Content-Length': '0'} 2023-05-31T20:53:56.210 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000003f4a96eb76cae6ca-006477b3e4-1c17-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:53:56 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:53:56.210 DEBUG:boto:b'pklxkd-59obj871000falseobj872023-05-31T20:48:13.031Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:53:56.211 DEBUG:boto:path=/pklxkd-59/ 2023-05-31T20:53:56.211 DEBUG:boto:auth_path=/pklxkd-59/ 2023-05-31T20:53:56.212 DEBUG:boto:path=/pklxkd-59/?prefix=obj87 2023-05-31T20:53:56.212 DEBUG:boto:auth_path=/pklxkd-59/?prefix=obj87 2023-05-31T20:53:56.212 DEBUG:boto:Method: GET 2023-05-31T20:53:56.213 DEBUG:boto:Path: /pklxkd-59/?prefix=obj87 2023-05-31T20:53:56.213 DEBUG:boto:Data: 2023-05-31T20:53:56.213 DEBUG:boto:Headers: {} 2023-05-31T20:53:56.214 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:53:56.214 DEBUG:boto:Port: 8000 2023-05-31T20:53:56.215 DEBUG:boto:Params: {} 2023-05-31T20:53:56.215 DEBUG:boto:Token: None 2023-05-31T20:53:56.216 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:53:56 GMT /pklxkd-59/ 2023-05-31T20:53:56.216 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:VxrsRCUGt15KLeSgquVd6DPBfJs= 2023-05-31T20:53:56.216 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:53:56 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:VxrsRCUGt15KLeSgquVd6DPBfJs=', 'Content-Length': '0'} 2023-05-31T20:53:56.220 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000794924cafcfa4854-006477b3e4-1630-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:53:56 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:53:56.221 DEBUG:boto:b'pklxkd-59obj871000falseobj872023-05-31T20:48:13.031Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:53:56.221 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj92 2023-05-31T20:53:56.222 DEBUG:boto:path=/pklxkd-59/obj92 2023-05-31T20:53:56.222 DEBUG:boto:auth_path=/pklxkd-59/obj92 2023-05-31T20:53:56.223 DEBUG:boto:path=/pklxkd-59/obj92?versionId=null 2023-05-31T20:53:56.223 DEBUG:boto:auth_path=/pklxkd-59/obj92?versionId=null 2023-05-31T20:53:56.223 DEBUG:boto:Method: GET 2023-05-31T20:53:56.224 DEBUG:boto:Path: /pklxkd-59/obj92?versionId=null 2023-05-31T20:53:56.224 DEBUG:boto:Data: 2023-05-31T20:53:56.224 DEBUG:boto:Headers: {} 2023-05-31T20:53:56.225 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:53:56.225 DEBUG:boto:Port: 8001 2023-05-31T20:53:56.225 DEBUG:boto:Params: {} 2023-05-31T20:53:56.226 DEBUG:boto:Token: None 2023-05-31T20:53:56.226 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:53:56 GMT /pklxkd-59/obj92?versionId=null 2023-05-31T20:53:56.227 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:z52ybUMhNdS0mV9Dr1OWe4zmL3Y= 2023-05-31T20:53:56.227 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:53:56 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:z52ybUMhNdS0mV9Dr1OWe4zmL3Y=', 'Content-Length': '0'} 2023-05-31T20:53:56.229 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:46:33 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007aa821f67dfcbacf-006477b3e4-1c17-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:53:56 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:53:56.229 DEBUG:boto:path=/pklxkd-59/obj92 2023-05-31T20:53:56.230 DEBUG:boto:auth_path=/pklxkd-59/obj92 2023-05-31T20:53:56.230 DEBUG:boto:path=/pklxkd-59/obj92?versionId=null 2023-05-31T20:53:56.231 DEBUG:boto:auth_path=/pklxkd-59/obj92?versionId=null 2023-05-31T20:53:56.231 DEBUG:boto:Method: GET 2023-05-31T20:53:56.231 DEBUG:boto:Path: /pklxkd-59/obj92?versionId=null 2023-05-31T20:53:56.232 DEBUG:boto:Data: 2023-05-31T20:53:56.232 DEBUG:boto:Headers: {} 2023-05-31T20:53:56.232 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:53:56.233 DEBUG:boto:Port: 8000 2023-05-31T20:53:56.233 DEBUG:boto:Params: {} 2023-05-31T20:53:56.233 DEBUG:boto:Token: None 2023-05-31T20:53:56.234 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:53:56 GMT /pklxkd-59/obj92?versionId=null 2023-05-31T20:53:56.234 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:z52ybUMhNdS0mV9Dr1OWe4zmL3Y= 2023-05-31T20:53:56.235 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:53:56 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:z52ybUMhNdS0mV9Dr1OWe4zmL3Y=', 'Content-Length': '0'} 2023-05-31T20:53:56.237 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:46:33 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000339ed24931851dfe-006477b3e4-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:53:56 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:53:56.280 DEBUG:boto:path=/pklxkd-59/obj92 2023-05-31T20:53:56.281 DEBUG:boto:auth_path=/pklxkd-59/obj92 2023-05-31T20:53:56.281 DEBUG:boto:path=/pklxkd-59/obj92?versionId=null 2023-05-31T20:53:56.281 DEBUG:boto:auth_path=/pklxkd-59/obj92?versionId=null 2023-05-31T20:53:56.282 DEBUG:boto:Method: HEAD 2023-05-31T20:53:56.282 DEBUG:boto:Path: /pklxkd-59/obj92?versionId=null 2023-05-31T20:53:56.282 DEBUG:boto:Data: 2023-05-31T20:53:56.283 DEBUG:boto:Headers: {} 2023-05-31T20:53:56.283 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:53:56.283 DEBUG:boto:Port: 8001 2023-05-31T20:53:56.284 DEBUG:boto:Params: {} 2023-05-31T20:53:56.284 DEBUG:boto:Token: None 2023-05-31T20:53:56.284 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:53:56 GMT /pklxkd-59/obj92?versionId=null 2023-05-31T20:53:56.285 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:5DdEozy9/qNF9BfV8yHnjGHPGk8= 2023-05-31T20:53:56.285 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:53:56 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:5DdEozy9/qNF9BfV8yHnjGHPGk8=', 'Content-Length': '0'} 2023-05-31T20:53:56.287 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:46:33 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000cc43247fe7196f02-006477b3e4-1c17-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:53:56 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:53:56.287 DEBUG:boto:path=/pklxkd-59/obj92 2023-05-31T20:53:56.288 DEBUG:boto:auth_path=/pklxkd-59/obj92 2023-05-31T20:53:56.288 DEBUG:boto:path=/pklxkd-59/obj92?versionId=null 2023-05-31T20:53:56.288 DEBUG:boto:auth_path=/pklxkd-59/obj92?versionId=null 2023-05-31T20:53:56.289 DEBUG:boto:Method: HEAD 2023-05-31T20:53:56.289 DEBUG:boto:Path: /pklxkd-59/obj92?versionId=null 2023-05-31T20:53:56.289 DEBUG:boto:Data: 2023-05-31T20:53:56.290 DEBUG:boto:Headers: {} 2023-05-31T20:53:56.290 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:53:56.290 DEBUG:boto:Port: 8000 2023-05-31T20:53:56.291 DEBUG:boto:Params: {} 2023-05-31T20:53:56.291 DEBUG:boto:Token: None 2023-05-31T20:53:56.292 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:53:56 GMT /pklxkd-59/obj92?versionId=null 2023-05-31T20:53:56.292 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:5DdEozy9/qNF9BfV8yHnjGHPGk8= 2023-05-31T20:53:56.292 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:53:56 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:5DdEozy9/qNF9BfV8yHnjGHPGk8=', 'Content-Length': '0'} 2023-05-31T20:53:56.294 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:46:33 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d43aae2060146113-006477b3e4-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:53:56 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:53:56.295 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj92 2023-05-31T20:53:56.295 DEBUG:boto:path=/pklxkd-59/obj92 2023-05-31T20:53:56.295 DEBUG:boto:auth_path=/pklxkd-59/obj92 2023-05-31T20:53:56.296 DEBUG:boto:path=/pklxkd-59/obj92?versionId=null 2023-05-31T20:53:56.296 DEBUG:boto:auth_path=/pklxkd-59/obj92?versionId=null 2023-05-31T20:53:56.297 DEBUG:boto:Method: GET 2023-05-31T20:53:56.297 DEBUG:boto:Path: /pklxkd-59/obj92?versionId=null 2023-05-31T20:53:56.297 DEBUG:boto:Data: 2023-05-31T20:53:56.298 DEBUG:boto:Headers: {} 2023-05-31T20:53:56.298 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:53:56.298 DEBUG:boto:Port: 8001 2023-05-31T20:53:56.299 DEBUG:boto:Params: {} 2023-05-31T20:53:56.299 DEBUG:boto:Token: None 2023-05-31T20:53:56.299 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:53:56 GMT /pklxkd-59/obj92?versionId=null 2023-05-31T20:53:56.300 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:z52ybUMhNdS0mV9Dr1OWe4zmL3Y= 2023-05-31T20:53:56.300 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:53:56 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:z52ybUMhNdS0mV9Dr1OWe4zmL3Y=', 'Content-Length': '0'} 2023-05-31T20:53:56.302 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:46:33 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000016b98c53047161f3-006477b3e4-1c17-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:53:56 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:53:56.303 DEBUG:boto:path=/pklxkd-59/obj92 2023-05-31T20:53:56.303 DEBUG:boto:auth_path=/pklxkd-59/obj92 2023-05-31T20:53:56.303 DEBUG:boto:path=/pklxkd-59/obj92?versionId=null 2023-05-31T20:53:56.304 DEBUG:boto:auth_path=/pklxkd-59/obj92?versionId=null 2023-05-31T20:53:56.304 DEBUG:boto:Method: GET 2023-05-31T20:53:56.304 DEBUG:boto:Path: /pklxkd-59/obj92?versionId=null 2023-05-31T20:53:56.305 DEBUG:boto:Data: 2023-05-31T20:53:56.305 DEBUG:boto:Headers: {} 2023-05-31T20:53:56.306 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:53:56.306 DEBUG:boto:Port: 8000 2023-05-31T20:53:56.306 DEBUG:boto:Params: {} 2023-05-31T20:53:56.307 DEBUG:boto:Token: None 2023-05-31T20:53:56.307 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:53:56 GMT /pklxkd-59/obj92?versionId=null 2023-05-31T20:53:56.307 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:z52ybUMhNdS0mV9Dr1OWe4zmL3Y= 2023-05-31T20:53:56.308 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:53:56 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:z52ybUMhNdS0mV9Dr1OWe4zmL3Y=', 'Content-Length': '0'} 2023-05-31T20:53:56.310 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:46:33 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000039d077fbe3a96171-006477b3e4-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:53:56 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:53:56.310 DEBUG:boto:path=/pklxkd-59/ 2023-05-31T20:53:56.311 DEBUG:boto:auth_path=/pklxkd-59/ 2023-05-31T20:53:56.311 DEBUG:boto:path=/pklxkd-59/?prefix=obj92 2023-05-31T20:53:56.311 DEBUG:boto:auth_path=/pklxkd-59/?prefix=obj92 2023-05-31T20:53:56.312 DEBUG:boto:Method: GET 2023-05-31T20:53:56.312 DEBUG:boto:Path: /pklxkd-59/?prefix=obj92 2023-05-31T20:53:56.312 DEBUG:boto:Data: 2023-05-31T20:53:56.313 DEBUG:boto:Headers: {} 2023-05-31T20:53:56.313 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:53:56.314 DEBUG:boto:Port: 8001 2023-05-31T20:53:56.314 DEBUG:boto:Params: {} 2023-05-31T20:53:56.314 DEBUG:boto:Token: None 2023-05-31T20:53:56.315 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:53:56 GMT /pklxkd-59/ 2023-05-31T20:53:56.315 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:VxrsRCUGt15KLeSgquVd6DPBfJs= 2023-05-31T20:53:56.316 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:53:56 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:VxrsRCUGt15KLeSgquVd6DPBfJs=', 'Content-Length': '0'} 2023-05-31T20:53:56.318 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000489f3f36fb62bbfc-006477b3e4-1c17-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:53:56 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:53:56.318 DEBUG:boto:b'pklxkd-59obj921000falseobj922023-05-31T20:46:33.904Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:53:56.319 DEBUG:boto:path=/pklxkd-59/ 2023-05-31T20:53:56.319 DEBUG:boto:auth_path=/pklxkd-59/ 2023-05-31T20:53:56.320 DEBUG:boto:path=/pklxkd-59/?prefix=obj92 2023-05-31T20:53:56.320 DEBUG:boto:auth_path=/pklxkd-59/?prefix=obj92 2023-05-31T20:53:56.320 DEBUG:boto:Method: GET 2023-05-31T20:53:56.321 DEBUG:boto:Path: /pklxkd-59/?prefix=obj92 2023-05-31T20:53:56.321 DEBUG:boto:Data: 2023-05-31T20:53:56.321 DEBUG:boto:Headers: {} 2023-05-31T20:53:56.322 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:53:56.322 DEBUG:boto:Port: 8000 2023-05-31T20:53:56.322 DEBUG:boto:Params: {} 2023-05-31T20:53:56.323 DEBUG:boto:Token: None 2023-05-31T20:53:56.323 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:53:56 GMT /pklxkd-59/ 2023-05-31T20:53:56.324 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:VxrsRCUGt15KLeSgquVd6DPBfJs= 2023-05-31T20:53:56.324 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:53:56 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:VxrsRCUGt15KLeSgquVd6DPBfJs=', 'Content-Length': '0'} 2023-05-31T20:53:56.328 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000069f752c530da70e1-006477b3e4-1630-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:53:56 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:53:56.329 DEBUG:boto:b'pklxkd-59obj921000falseobj922023-05-31T20:46:33.904Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:53:56.329 DEBUG:boto:path=/pklxkd-59/obj92 2023-05-31T20:53:56.330 DEBUG:boto:auth_path=/pklxkd-59/obj92 2023-05-31T20:53:56.330 DEBUG:boto:Method: HEAD 2023-05-31T20:53:56.330 DEBUG:boto:Path: /pklxkd-59/obj92 2023-05-31T20:53:56.331 DEBUG:boto:Data: 2023-05-31T20:53:56.331 DEBUG:boto:Headers: {} 2023-05-31T20:53:56.331 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:53:56.332 DEBUG:boto:Port: 8001 2023-05-31T20:53:56.332 DEBUG:boto:Params: {} 2023-05-31T20:53:56.332 DEBUG:boto:Token: None 2023-05-31T20:53:56.333 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:53:56 GMT /pklxkd-59/obj92 2023-05-31T20:53:56.333 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:Esx/lg9YbLodBGeP5/11Uw6lIUM= 2023-05-31T20:53:56.334 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:53:56 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:Esx/lg9YbLodBGeP5/11Uw6lIUM=', 'Content-Length': '0'} 2023-05-31T20:53:56.335 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:46:33 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a32cb9420dbd07a4-006477b3e4-1c17-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:53:56 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:53:56.336 DEBUG:boto:path=/pklxkd-59/obj92 2023-05-31T20:53:56.336 DEBUG:boto:auth_path=/pklxkd-59/obj92 2023-05-31T20:53:56.337 DEBUG:boto:Method: HEAD 2023-05-31T20:53:56.337 DEBUG:boto:Path: /pklxkd-59/obj92 2023-05-31T20:53:56.337 DEBUG:boto:Data: 2023-05-31T20:53:56.338 DEBUG:boto:Headers: {} 2023-05-31T20:53:56.338 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:53:56.338 DEBUG:boto:Port: 8000 2023-05-31T20:53:56.339 DEBUG:boto:Params: {} 2023-05-31T20:53:56.339 DEBUG:boto:Token: None 2023-05-31T20:53:56.339 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:53:56 GMT /pklxkd-59/obj92 2023-05-31T20:53:56.340 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:Esx/lg9YbLodBGeP5/11Uw6lIUM= 2023-05-31T20:53:56.340 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:53:56 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:Esx/lg9YbLodBGeP5/11Uw6lIUM=', 'Content-Length': '0'} 2023-05-31T20:53:56.342 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:46:33 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000029a93a4aa3bfa710-006477b3e4-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:53:56 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:53:56.343 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj92 2023-05-31T20:53:56.343 DEBUG:boto:path=/pklxkd-59/obj92 2023-05-31T20:53:56.343 DEBUG:boto:auth_path=/pklxkd-59/obj92 2023-05-31T20:53:56.344 DEBUG:boto:Method: GET 2023-05-31T20:53:56.344 DEBUG:boto:Path: /pklxkd-59/obj92 2023-05-31T20:53:56.344 DEBUG:boto:Data: 2023-05-31T20:53:56.345 DEBUG:boto:Headers: {} 2023-05-31T20:53:56.345 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:53:56.346 DEBUG:boto:Port: 8001 2023-05-31T20:53:56.346 DEBUG:boto:Params: {} 2023-05-31T20:53:56.346 DEBUG:boto:Token: None 2023-05-31T20:53:56.347 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:53:56 GMT /pklxkd-59/obj92 2023-05-31T20:53:56.347 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:TcxscdDm7aAHr5PxAuKcGI+yvQM= 2023-05-31T20:53:56.348 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:53:56 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:TcxscdDm7aAHr5PxAuKcGI+yvQM=', 'Content-Length': '0'} 2023-05-31T20:53:56.349 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:46:33 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000032ebfe347b4d1034-006477b3e4-1c17-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:53:56 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:53:56.350 DEBUG:boto:path=/pklxkd-59/obj92 2023-05-31T20:53:56.350 DEBUG:boto:auth_path=/pklxkd-59/obj92 2023-05-31T20:53:56.351 DEBUG:boto:Method: GET 2023-05-31T20:53:56.351 DEBUG:boto:Path: /pklxkd-59/obj92 2023-05-31T20:53:56.351 DEBUG:boto:Data: 2023-05-31T20:53:56.352 DEBUG:boto:Headers: {} 2023-05-31T20:53:56.352 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:53:56.352 DEBUG:boto:Port: 8000 2023-05-31T20:53:56.353 DEBUG:boto:Params: {} 2023-05-31T20:53:56.353 DEBUG:boto:Token: None 2023-05-31T20:53:56.354 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:53:56 GMT /pklxkd-59/obj92 2023-05-31T20:53:56.354 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:TcxscdDm7aAHr5PxAuKcGI+yvQM= 2023-05-31T20:53:56.354 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:53:56 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:TcxscdDm7aAHr5PxAuKcGI+yvQM=', 'Content-Length': '0'} 2023-05-31T20:53:56.356 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:46:33 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006782448414b90c92-006477b3e4-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:53:56 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:53:56.357 DEBUG:boto:path=/pklxkd-59/ 2023-05-31T20:53:56.357 DEBUG:boto:auth_path=/pklxkd-59/ 2023-05-31T20:53:56.358 DEBUG:boto:path=/pklxkd-59/?prefix=obj92 2023-05-31T20:53:56.358 DEBUG:boto:auth_path=/pklxkd-59/?prefix=obj92 2023-05-31T20:53:56.358 DEBUG:boto:Method: GET 2023-05-31T20:53:56.359 DEBUG:boto:Path: /pklxkd-59/?prefix=obj92 2023-05-31T20:53:56.359 DEBUG:boto:Data: 2023-05-31T20:53:56.359 DEBUG:boto:Headers: {} 2023-05-31T20:53:56.360 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:53:56.360 DEBUG:boto:Port: 8001 2023-05-31T20:53:56.360 DEBUG:boto:Params: {} 2023-05-31T20:53:56.361 DEBUG:boto:Token: None 2023-05-31T20:53:56.361 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:53:56 GMT /pklxkd-59/ 2023-05-31T20:53:56.362 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:VxrsRCUGt15KLeSgquVd6DPBfJs= 2023-05-31T20:53:56.362 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:53:56 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:VxrsRCUGt15KLeSgquVd6DPBfJs=', 'Content-Length': '0'} 2023-05-31T20:53:56.364 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000764d4d93c10cea5f-006477b3e4-1c17-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:53:56 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:53:56.365 DEBUG:boto:b'pklxkd-59obj921000falseobj922023-05-31T20:46:33.904Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:53:56.366 DEBUG:boto:path=/pklxkd-59/ 2023-05-31T20:53:56.366 DEBUG:boto:auth_path=/pklxkd-59/ 2023-05-31T20:53:56.366 DEBUG:boto:path=/pklxkd-59/?prefix=obj92 2023-05-31T20:53:56.367 DEBUG:boto:auth_path=/pklxkd-59/?prefix=obj92 2023-05-31T20:53:56.367 DEBUG:boto:Method: GET 2023-05-31T20:53:56.367 DEBUG:boto:Path: /pklxkd-59/?prefix=obj92 2023-05-31T20:53:56.368 DEBUG:boto:Data: 2023-05-31T20:53:56.368 DEBUG:boto:Headers: {} 2023-05-31T20:53:56.368 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:53:56.369 DEBUG:boto:Port: 8000 2023-05-31T20:53:56.369 DEBUG:boto:Params: {} 2023-05-31T20:53:56.370 DEBUG:boto:Token: None 2023-05-31T20:53:56.370 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:53:56 GMT /pklxkd-59/ 2023-05-31T20:53:56.370 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:VxrsRCUGt15KLeSgquVd6DPBfJs= 2023-05-31T20:53:56.371 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:53:56 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:VxrsRCUGt15KLeSgquVd6DPBfJs=', 'Content-Length': '0'} 2023-05-31T20:53:56.375 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b191e0fbfa4ec02c-006477b3e4-1630-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:53:56 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:53:56.416 DEBUG:boto:b'pklxkd-59obj921000falseobj922023-05-31T20:46:33.904Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:53:56.417 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj93 2023-05-31T20:53:56.417 DEBUG:boto:path=/pklxkd-59/obj93 2023-05-31T20:53:56.418 DEBUG:boto:auth_path=/pklxkd-59/obj93 2023-05-31T20:53:56.418 DEBUG:boto:path=/pklxkd-59/obj93?versionId=null 2023-05-31T20:53:56.418 DEBUG:boto:auth_path=/pklxkd-59/obj93?versionId=null 2023-05-31T20:53:56.419 DEBUG:boto:Method: GET 2023-05-31T20:53:56.419 DEBUG:boto:Path: /pklxkd-59/obj93?versionId=null 2023-05-31T20:53:56.419 DEBUG:boto:Data: 2023-05-31T20:53:56.420 DEBUG:boto:Headers: {} 2023-05-31T20:53:56.420 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:53:56.420 DEBUG:boto:Port: 8001 2023-05-31T20:53:56.421 DEBUG:boto:Params: {} 2023-05-31T20:53:56.421 DEBUG:boto:Token: None 2023-05-31T20:53:56.422 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:53:56 GMT /pklxkd-59/obj93?versionId=null 2023-05-31T20:53:56.422 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:+I2Ka0M67+KWIh0QWygZkq4VQX0= 2023-05-31T20:53:56.423 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:53:56 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:+I2Ka0M67+KWIh0QWygZkq4VQX0=', 'Content-Length': '0'} 2023-05-31T20:53:56.424 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:50:33 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f5b51ad6331953ac-006477b3e4-1c17-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:53:56 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:53:56.425 DEBUG:boto:path=/pklxkd-59/obj93 2023-05-31T20:53:56.425 DEBUG:boto:auth_path=/pklxkd-59/obj93 2023-05-31T20:53:56.426 DEBUG:boto:path=/pklxkd-59/obj93?versionId=null 2023-05-31T20:53:56.426 DEBUG:boto:auth_path=/pklxkd-59/obj93?versionId=null 2023-05-31T20:53:56.426 DEBUG:boto:Method: GET 2023-05-31T20:53:56.427 DEBUG:boto:Path: /pklxkd-59/obj93?versionId=null 2023-05-31T20:53:56.427 DEBUG:boto:Data: 2023-05-31T20:53:56.427 DEBUG:boto:Headers: {} 2023-05-31T20:53:56.428 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:53:56.428 DEBUG:boto:Port: 8000 2023-05-31T20:53:56.429 DEBUG:boto:Params: {} 2023-05-31T20:53:56.429 DEBUG:boto:Token: None 2023-05-31T20:53:56.429 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:53:56 GMT /pklxkd-59/obj93?versionId=null 2023-05-31T20:53:56.430 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:+I2Ka0M67+KWIh0QWygZkq4VQX0= 2023-05-31T20:53:56.430 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:53:56 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:+I2Ka0M67+KWIh0QWygZkq4VQX0=', 'Content-Length': '0'} 2023-05-31T20:53:56.432 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:50:33 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008c4c50e59b772e4a-006477b3e4-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:53:56 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:53:56.433 DEBUG:boto:path=/pklxkd-59/obj93 2023-05-31T20:53:56.433 DEBUG:boto:auth_path=/pklxkd-59/obj93 2023-05-31T20:53:56.434 DEBUG:boto:path=/pklxkd-59/obj93?versionId=null 2023-05-31T20:53:56.434 DEBUG:boto:auth_path=/pklxkd-59/obj93?versionId=null 2023-05-31T20:53:56.434 DEBUG:boto:Method: HEAD 2023-05-31T20:53:56.435 DEBUG:boto:Path: /pklxkd-59/obj93?versionId=null 2023-05-31T20:53:56.435 DEBUG:boto:Data: 2023-05-31T20:53:56.435 DEBUG:boto:Headers: {} 2023-05-31T20:53:56.436 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:53:56.436 DEBUG:boto:Port: 8001 2023-05-31T20:53:56.436 DEBUG:boto:Params: {} 2023-05-31T20:53:56.437 DEBUG:boto:Token: None 2023-05-31T20:53:56.437 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:53:56 GMT /pklxkd-59/obj93?versionId=null 2023-05-31T20:53:56.438 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:HqHhD0RTJKH0cIg5oS/PVcI7yp8= 2023-05-31T20:53:56.438 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:53:56 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:HqHhD0RTJKH0cIg5oS/PVcI7yp8=', 'Content-Length': '0'} 2023-05-31T20:53:56.440 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:50:33 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000aa31197252396cb6-006477b3e4-1c17-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:53:56 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:53:56.440 DEBUG:boto:path=/pklxkd-59/obj93 2023-05-31T20:53:56.441 DEBUG:boto:auth_path=/pklxkd-59/obj93 2023-05-31T20:53:56.441 DEBUG:boto:path=/pklxkd-59/obj93?versionId=null 2023-05-31T20:53:56.441 DEBUG:boto:auth_path=/pklxkd-59/obj93?versionId=null 2023-05-31T20:53:56.442 DEBUG:boto:Method: HEAD 2023-05-31T20:53:56.442 DEBUG:boto:Path: /pklxkd-59/obj93?versionId=null 2023-05-31T20:53:56.442 DEBUG:boto:Data: 2023-05-31T20:53:56.443 DEBUG:boto:Headers: {} 2023-05-31T20:53:56.443 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:53:56.444 DEBUG:boto:Port: 8000 2023-05-31T20:53:56.444 DEBUG:boto:Params: {} 2023-05-31T20:53:56.444 DEBUG:boto:Token: None 2023-05-31T20:53:56.445 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:53:56 GMT /pklxkd-59/obj93?versionId=null 2023-05-31T20:53:56.445 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:HqHhD0RTJKH0cIg5oS/PVcI7yp8= 2023-05-31T20:53:56.446 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:53:56 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:HqHhD0RTJKH0cIg5oS/PVcI7yp8=', 'Content-Length': '0'} 2023-05-31T20:53:56.447 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:50:33 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004bc77d33fcbd215b-006477b3e4-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:53:56 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:53:56.448 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj93 2023-05-31T20:53:56.448 DEBUG:boto:path=/pklxkd-59/obj93 2023-05-31T20:53:56.449 DEBUG:boto:auth_path=/pklxkd-59/obj93 2023-05-31T20:53:56.449 DEBUG:boto:path=/pklxkd-59/obj93?versionId=null 2023-05-31T20:53:56.449 DEBUG:boto:auth_path=/pklxkd-59/obj93?versionId=null 2023-05-31T20:53:56.450 DEBUG:boto:Method: GET 2023-05-31T20:53:56.450 DEBUG:boto:Path: /pklxkd-59/obj93?versionId=null 2023-05-31T20:53:56.450 DEBUG:boto:Data: 2023-05-31T20:53:56.451 DEBUG:boto:Headers: {} 2023-05-31T20:53:56.451 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:53:56.451 DEBUG:boto:Port: 8001 2023-05-31T20:53:56.452 DEBUG:boto:Params: {} 2023-05-31T20:53:56.452 DEBUG:boto:Token: None 2023-05-31T20:53:56.453 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:53:56 GMT /pklxkd-59/obj93?versionId=null 2023-05-31T20:53:56.453 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:+I2Ka0M67+KWIh0QWygZkq4VQX0= 2023-05-31T20:53:56.453 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:53:56 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:+I2Ka0M67+KWIh0QWygZkq4VQX0=', 'Content-Length': '0'} 2023-05-31T20:53:56.455 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:50:33 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003af19c332cc64125-006477b3e4-1c17-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:53:56 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:53:56.496 DEBUG:boto:path=/pklxkd-59/obj93 2023-05-31T20:53:56.497 DEBUG:boto:auth_path=/pklxkd-59/obj93 2023-05-31T20:53:56.497 DEBUG:boto:path=/pklxkd-59/obj93?versionId=null 2023-05-31T20:53:56.497 DEBUG:boto:auth_path=/pklxkd-59/obj93?versionId=null 2023-05-31T20:53:56.498 DEBUG:boto:Method: GET 2023-05-31T20:53:56.498 DEBUG:boto:Path: /pklxkd-59/obj93?versionId=null 2023-05-31T20:53:56.498 DEBUG:boto:Data: 2023-05-31T20:53:56.499 DEBUG:boto:Headers: {} 2023-05-31T20:53:56.499 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:53:56.499 DEBUG:boto:Port: 8000 2023-05-31T20:53:56.500 DEBUG:boto:Params: {} 2023-05-31T20:53:56.500 DEBUG:boto:Token: None 2023-05-31T20:53:56.501 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:53:56 GMT /pklxkd-59/obj93?versionId=null 2023-05-31T20:53:56.501 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:+I2Ka0M67+KWIh0QWygZkq4VQX0= 2023-05-31T20:53:56.502 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:53:56 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:+I2Ka0M67+KWIh0QWygZkq4VQX0=', 'Content-Length': '0'} 2023-05-31T20:53:56.503 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:50:33 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007e71270573aee207-006477b3e4-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:53:56 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:53:56.504 DEBUG:boto:path=/pklxkd-59/ 2023-05-31T20:53:56.504 DEBUG:boto:auth_path=/pklxkd-59/ 2023-05-31T20:53:56.505 DEBUG:boto:path=/pklxkd-59/?prefix=obj93 2023-05-31T20:53:56.505 DEBUG:boto:auth_path=/pklxkd-59/?prefix=obj93 2023-05-31T20:53:56.505 DEBUG:boto:Method: GET 2023-05-31T20:53:56.506 DEBUG:boto:Path: /pklxkd-59/?prefix=obj93 2023-05-31T20:53:56.506 DEBUG:boto:Data: 2023-05-31T20:53:56.507 DEBUG:boto:Headers: {} 2023-05-31T20:53:56.507 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:53:56.507 DEBUG:boto:Port: 8001 2023-05-31T20:53:56.508 DEBUG:boto:Params: {} 2023-05-31T20:53:56.508 DEBUG:boto:Token: None 2023-05-31T20:53:56.508 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:53:56 GMT /pklxkd-59/ 2023-05-31T20:53:56.509 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:VxrsRCUGt15KLeSgquVd6DPBfJs= 2023-05-31T20:53:56.509 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:53:56 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:VxrsRCUGt15KLeSgquVd6DPBfJs=', 'Content-Length': '0'} 2023-05-31T20:53:56.512 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000bb92d026d8f6eddd-006477b3e4-1c17-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:53:56 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:53:56.512 DEBUG:boto:b'pklxkd-59obj931000falseobj932023-05-31T20:50:33.533Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:53:56.513 DEBUG:boto:path=/pklxkd-59/ 2023-05-31T20:53:56.513 DEBUG:boto:auth_path=/pklxkd-59/ 2023-05-31T20:53:56.514 DEBUG:boto:path=/pklxkd-59/?prefix=obj93 2023-05-31T20:53:56.514 DEBUG:boto:auth_path=/pklxkd-59/?prefix=obj93 2023-05-31T20:53:56.514 DEBUG:boto:Method: GET 2023-05-31T20:53:56.515 DEBUG:boto:Path: /pklxkd-59/?prefix=obj93 2023-05-31T20:53:56.515 DEBUG:boto:Data: 2023-05-31T20:53:56.515 DEBUG:boto:Headers: {} 2023-05-31T20:53:56.516 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:53:56.516 DEBUG:boto:Port: 8000 2023-05-31T20:53:56.516 DEBUG:boto:Params: {} 2023-05-31T20:53:56.517 DEBUG:boto:Token: None 2023-05-31T20:53:56.517 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:53:56 GMT /pklxkd-59/ 2023-05-31T20:53:56.518 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:VxrsRCUGt15KLeSgquVd6DPBfJs= 2023-05-31T20:53:56.518 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:53:56 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:VxrsRCUGt15KLeSgquVd6DPBfJs=', 'Content-Length': '0'} 2023-05-31T20:53:56.522 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000003f4179fcca5a7e50-006477b3e4-1630-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:53:56 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:53:56.522 DEBUG:boto:b'pklxkd-59obj931000falseobj932023-05-31T20:50:33.533Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:53:56.523 DEBUG:boto:path=/pklxkd-59/obj93 2023-05-31T20:53:56.523 DEBUG:boto:auth_path=/pklxkd-59/obj93 2023-05-31T20:53:56.524 DEBUG:boto:Method: HEAD 2023-05-31T20:53:56.524 DEBUG:boto:Path: /pklxkd-59/obj93 2023-05-31T20:53:56.524 DEBUG:boto:Data: 2023-05-31T20:53:56.525 DEBUG:boto:Headers: {} 2023-05-31T20:53:56.525 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:53:56.525 DEBUG:boto:Port: 8001 2023-05-31T20:53:56.526 DEBUG:boto:Params: {} 2023-05-31T20:53:56.526 DEBUG:boto:Token: None 2023-05-31T20:53:56.526 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:53:56 GMT /pklxkd-59/obj93 2023-05-31T20:53:56.527 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:lPSbe3ye9vgYEaOOISUOysl1hPA= 2023-05-31T20:53:56.527 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:53:56 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:lPSbe3ye9vgYEaOOISUOysl1hPA=', 'Content-Length': '0'} 2023-05-31T20:53:56.529 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:50:33 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c1a5a93e3f80fd0a-006477b3e4-1c17-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:53:56 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:53:56.530 DEBUG:boto:path=/pklxkd-59/obj93 2023-05-31T20:53:56.530 DEBUG:boto:auth_path=/pklxkd-59/obj93 2023-05-31T20:53:56.530 DEBUG:boto:Method: HEAD 2023-05-31T20:53:56.531 DEBUG:boto:Path: /pklxkd-59/obj93 2023-05-31T20:53:56.531 DEBUG:boto:Data: 2023-05-31T20:53:56.531 DEBUG:boto:Headers: {} 2023-05-31T20:53:56.532 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:53:56.532 DEBUG:boto:Port: 8000 2023-05-31T20:53:56.532 DEBUG:boto:Params: {} 2023-05-31T20:53:56.533 DEBUG:boto:Token: None 2023-05-31T20:53:56.533 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:53:56 GMT /pklxkd-59/obj93 2023-05-31T20:53:56.533 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:lPSbe3ye9vgYEaOOISUOysl1hPA= 2023-05-31T20:53:56.534 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:53:56 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:lPSbe3ye9vgYEaOOISUOysl1hPA=', 'Content-Length': '0'} 2023-05-31T20:53:56.536 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:50:33 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000001cad823d13ce83f-006477b3e4-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:53:56 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:53:56.536 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj93 2023-05-31T20:53:56.537 DEBUG:boto:path=/pklxkd-59/obj93 2023-05-31T20:53:56.537 DEBUG:boto:auth_path=/pklxkd-59/obj93 2023-05-31T20:53:56.537 DEBUG:boto:Method: GET 2023-05-31T20:53:56.538 DEBUG:boto:Path: /pklxkd-59/obj93 2023-05-31T20:53:56.538 DEBUG:boto:Data: 2023-05-31T20:53:56.538 DEBUG:boto:Headers: {} 2023-05-31T20:53:56.539 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:53:56.539 DEBUG:boto:Port: 8001 2023-05-31T20:53:56.540 DEBUG:boto:Params: {} 2023-05-31T20:53:56.540 DEBUG:boto:Token: None 2023-05-31T20:53:56.540 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:53:56 GMT /pklxkd-59/obj93 2023-05-31T20:53:56.541 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:ryDhwiOQsn7c99QyYqIVwX76mt4= 2023-05-31T20:53:56.541 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:53:56 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:ryDhwiOQsn7c99QyYqIVwX76mt4=', 'Content-Length': '0'} 2023-05-31T20:53:56.543 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:50:33 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f860c705d1d56aa4-006477b3e4-1c17-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:53:56 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:53:56.544 DEBUG:boto:path=/pklxkd-59/obj93 2023-05-31T20:53:56.544 DEBUG:boto:auth_path=/pklxkd-59/obj93 2023-05-31T20:53:56.544 DEBUG:boto:Method: GET 2023-05-31T20:53:56.545 DEBUG:boto:Path: /pklxkd-59/obj93 2023-05-31T20:53:56.545 DEBUG:boto:Data: 2023-05-31T20:53:56.545 DEBUG:boto:Headers: {} 2023-05-31T20:53:56.546 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:53:56.546 DEBUG:boto:Port: 8000 2023-05-31T20:53:56.547 DEBUG:boto:Params: {} 2023-05-31T20:53:56.547 DEBUG:boto:Token: None 2023-05-31T20:53:56.547 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:53:56 GMT /pklxkd-59/obj93 2023-05-31T20:53:56.548 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:ryDhwiOQsn7c99QyYqIVwX76mt4= 2023-05-31T20:53:56.548 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:53:56 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:ryDhwiOQsn7c99QyYqIVwX76mt4=', 'Content-Length': '0'} 2023-05-31T20:53:56.550 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:50:33 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ecc5956a0c9633a4-006477b3e4-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:53:56 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:53:56.592 DEBUG:boto:path=/pklxkd-59/ 2023-05-31T20:53:56.593 DEBUG:boto:auth_path=/pklxkd-59/ 2023-05-31T20:53:56.593 DEBUG:boto:path=/pklxkd-59/?prefix=obj93 2023-05-31T20:53:56.594 DEBUG:boto:auth_path=/pklxkd-59/?prefix=obj93 2023-05-31T20:53:56.594 DEBUG:boto:Method: GET 2023-05-31T20:53:56.595 DEBUG:boto:Path: /pklxkd-59/?prefix=obj93 2023-05-31T20:53:56.595 DEBUG:boto:Data: 2023-05-31T20:53:56.595 DEBUG:boto:Headers: {} 2023-05-31T20:53:56.596 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:53:56.596 DEBUG:boto:Port: 8001 2023-05-31T20:53:56.597 DEBUG:boto:Params: {} 2023-05-31T20:53:56.597 DEBUG:boto:Token: None 2023-05-31T20:53:56.597 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:53:56 GMT /pklxkd-59/ 2023-05-31T20:53:56.598 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:VxrsRCUGt15KLeSgquVd6DPBfJs= 2023-05-31T20:53:56.598 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:53:56 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:VxrsRCUGt15KLeSgquVd6DPBfJs=', 'Content-Length': '0'} 2023-05-31T20:53:56.601 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000007101f73dac17ac6f-006477b3e4-1c17-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:53:56 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:53:56.601 DEBUG:boto:b'pklxkd-59obj931000falseobj932023-05-31T20:50:33.533Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:53:56.602 DEBUG:boto:path=/pklxkd-59/ 2023-05-31T20:53:56.603 DEBUG:boto:auth_path=/pklxkd-59/ 2023-05-31T20:53:56.603 DEBUG:boto:path=/pklxkd-59/?prefix=obj93 2023-05-31T20:53:56.603 DEBUG:boto:auth_path=/pklxkd-59/?prefix=obj93 2023-05-31T20:53:56.604 DEBUG:boto:Method: GET 2023-05-31T20:53:56.604 DEBUG:boto:Path: /pklxkd-59/?prefix=obj93 2023-05-31T20:53:56.605 DEBUG:boto:Data: 2023-05-31T20:53:56.605 DEBUG:boto:Headers: {} 2023-05-31T20:53:56.605 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:53:56.606 DEBUG:boto:Port: 8000 2023-05-31T20:53:56.606 DEBUG:boto:Params: {} 2023-05-31T20:53:56.607 DEBUG:boto:Token: None 2023-05-31T20:53:56.607 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:53:56 GMT /pklxkd-59/ 2023-05-31T20:53:56.607 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:VxrsRCUGt15KLeSgquVd6DPBfJs= 2023-05-31T20:53:56.608 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:53:56 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:VxrsRCUGt15KLeSgquVd6DPBfJs=', 'Content-Length': '0'} 2023-05-31T20:53:56.612 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000cb68182590758c78-006477b3e4-1630-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:53:56 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:53:56.612 DEBUG:boto:b'pklxkd-59obj931000falseobj932023-05-31T20:50:33.533Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:53:56.613 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj95 2023-05-31T20:53:56.613 DEBUG:boto:path=/pklxkd-59/obj95 2023-05-31T20:53:56.614 DEBUG:boto:auth_path=/pklxkd-59/obj95 2023-05-31T20:53:56.614 DEBUG:boto:path=/pklxkd-59/obj95?versionId=null 2023-05-31T20:53:56.614 DEBUG:boto:auth_path=/pklxkd-59/obj95?versionId=null 2023-05-31T20:53:56.615 DEBUG:boto:Method: GET 2023-05-31T20:53:56.615 DEBUG:boto:Path: /pklxkd-59/obj95?versionId=null 2023-05-31T20:53:56.616 DEBUG:boto:Data: 2023-05-31T20:53:56.616 DEBUG:boto:Headers: {} 2023-05-31T20:53:56.617 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:53:56.617 DEBUG:boto:Port: 8001 2023-05-31T20:53:56.617 DEBUG:boto:Params: {} 2023-05-31T20:53:56.618 DEBUG:boto:Token: None 2023-05-31T20:53:56.618 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:53:56 GMT /pklxkd-59/obj95?versionId=null 2023-05-31T20:53:56.619 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:5IIC4KWyR3UGnaGw4PDDf8zpAfE= 2023-05-31T20:53:56.619 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:53:56 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:5IIC4KWyR3UGnaGw4PDDf8zpAfE=', 'Content-Length': '0'} 2023-05-31T20:53:56.621 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:44:50 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f3bf7605eec0fbe4-006477b3e4-1c17-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:53:56 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:53:56.664 DEBUG:boto:path=/pklxkd-59/obj95 2023-05-31T20:53:56.665 DEBUG:boto:auth_path=/pklxkd-59/obj95 2023-05-31T20:53:56.665 DEBUG:boto:path=/pklxkd-59/obj95?versionId=null 2023-05-31T20:53:56.665 DEBUG:boto:auth_path=/pklxkd-59/obj95?versionId=null 2023-05-31T20:53:56.666 DEBUG:boto:Method: GET 2023-05-31T20:53:56.666 DEBUG:boto:Path: /pklxkd-59/obj95?versionId=null 2023-05-31T20:53:56.666 DEBUG:boto:Data: 2023-05-31T20:53:56.667 DEBUG:boto:Headers: {} 2023-05-31T20:53:56.667 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:53:56.667 DEBUG:boto:Port: 8000 2023-05-31T20:53:56.668 DEBUG:boto:Params: {} 2023-05-31T20:53:56.668 DEBUG:boto:Token: None 2023-05-31T20:53:56.669 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:53:56 GMT /pklxkd-59/obj95?versionId=null 2023-05-31T20:53:56.669 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:5IIC4KWyR3UGnaGw4PDDf8zpAfE= 2023-05-31T20:53:56.670 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:53:56 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:5IIC4KWyR3UGnaGw4PDDf8zpAfE=', 'Content-Length': '0'} 2023-05-31T20:53:56.671 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:44:50 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000091734bcd0b2030fe-006477b3e4-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:53:56 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:53:56.672 DEBUG:boto:path=/pklxkd-59/obj95 2023-05-31T20:53:56.672 DEBUG:boto:auth_path=/pklxkd-59/obj95 2023-05-31T20:53:56.673 DEBUG:boto:path=/pklxkd-59/obj95?versionId=null 2023-05-31T20:53:56.673 DEBUG:boto:auth_path=/pklxkd-59/obj95?versionId=null 2023-05-31T20:53:56.673 DEBUG:boto:Method: HEAD 2023-05-31T20:53:56.674 DEBUG:boto:Path: /pklxkd-59/obj95?versionId=null 2023-05-31T20:53:56.674 DEBUG:boto:Data: 2023-05-31T20:53:56.674 DEBUG:boto:Headers: {} 2023-05-31T20:53:56.675 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:53:56.675 DEBUG:boto:Port: 8001 2023-05-31T20:53:56.675 DEBUG:boto:Params: {} 2023-05-31T20:53:56.676 DEBUG:boto:Token: None 2023-05-31T20:53:56.676 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:53:56 GMT /pklxkd-59/obj95?versionId=null 2023-05-31T20:53:56.677 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:S9GAs07U1qisjIQy7s8g9uqqdsc= 2023-05-31T20:53:56.677 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:53:56 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:S9GAs07U1qisjIQy7s8g9uqqdsc=', 'Content-Length': '0'} 2023-05-31T20:53:56.679 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:44:50 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a4837015d05c1ca6-006477b3e4-1c17-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:53:56 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:53:56.679 DEBUG:boto:path=/pklxkd-59/obj95 2023-05-31T20:53:56.680 DEBUG:boto:auth_path=/pklxkd-59/obj95 2023-05-31T20:53:56.680 DEBUG:boto:path=/pklxkd-59/obj95?versionId=null 2023-05-31T20:53:56.680 DEBUG:boto:auth_path=/pklxkd-59/obj95?versionId=null 2023-05-31T20:53:56.681 DEBUG:boto:Method: HEAD 2023-05-31T20:53:56.681 DEBUG:boto:Path: /pklxkd-59/obj95?versionId=null 2023-05-31T20:53:56.681 DEBUG:boto:Data: 2023-05-31T20:53:56.682 DEBUG:boto:Headers: {} 2023-05-31T20:53:56.682 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:53:56.682 DEBUG:boto:Port: 8000 2023-05-31T20:53:56.683 DEBUG:boto:Params: {} 2023-05-31T20:53:56.683 DEBUG:boto:Token: None 2023-05-31T20:53:56.684 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:53:56 GMT /pklxkd-59/obj95?versionId=null 2023-05-31T20:53:56.684 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:S9GAs07U1qisjIQy7s8g9uqqdsc= 2023-05-31T20:53:56.684 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:53:56 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:S9GAs07U1qisjIQy7s8g9uqqdsc=', 'Content-Length': '0'} 2023-05-31T20:53:56.686 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:44:50 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000049f2b0f210b44e66-006477b3e4-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:53:56 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:53:56.687 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj95 2023-05-31T20:53:56.687 DEBUG:boto:path=/pklxkd-59/obj95 2023-05-31T20:53:56.687 DEBUG:boto:auth_path=/pklxkd-59/obj95 2023-05-31T20:53:56.688 DEBUG:boto:path=/pklxkd-59/obj95?versionId=null 2023-05-31T20:53:56.688 DEBUG:boto:auth_path=/pklxkd-59/obj95?versionId=null 2023-05-31T20:53:56.688 DEBUG:boto:Method: GET 2023-05-31T20:53:56.689 DEBUG:boto:Path: /pklxkd-59/obj95?versionId=null 2023-05-31T20:53:56.689 DEBUG:boto:Data: 2023-05-31T20:53:56.690 DEBUG:boto:Headers: {} 2023-05-31T20:53:56.690 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:53:56.690 DEBUG:boto:Port: 8001 2023-05-31T20:53:56.691 DEBUG:boto:Params: {} 2023-05-31T20:53:56.691 DEBUG:boto:Token: None 2023-05-31T20:53:56.692 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:53:56 GMT /pklxkd-59/obj95?versionId=null 2023-05-31T20:53:56.692 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:5IIC4KWyR3UGnaGw4PDDf8zpAfE= 2023-05-31T20:53:56.692 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:53:56 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:5IIC4KWyR3UGnaGw4PDDf8zpAfE=', 'Content-Length': '0'} 2023-05-31T20:53:56.694 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:44:50 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001406b23ab7240c3f-006477b3e4-1c17-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:53:56 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:53:56.695 DEBUG:boto:path=/pklxkd-59/obj95 2023-05-31T20:53:56.695 DEBUG:boto:auth_path=/pklxkd-59/obj95 2023-05-31T20:53:56.696 DEBUG:boto:path=/pklxkd-59/obj95?versionId=null 2023-05-31T20:53:56.696 DEBUG:boto:auth_path=/pklxkd-59/obj95?versionId=null 2023-05-31T20:53:56.696 DEBUG:boto:Method: GET 2023-05-31T20:53:56.697 DEBUG:boto:Path: /pklxkd-59/obj95?versionId=null 2023-05-31T20:53:56.697 DEBUG:boto:Data: 2023-05-31T20:53:56.698 DEBUG:boto:Headers: {} 2023-05-31T20:53:56.698 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:53:56.698 DEBUG:boto:Port: 8000 2023-05-31T20:53:56.699 DEBUG:boto:Params: {} 2023-05-31T20:53:56.699 DEBUG:boto:Token: None 2023-05-31T20:53:56.699 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:53:56 GMT /pklxkd-59/obj95?versionId=null 2023-05-31T20:53:56.700 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:5IIC4KWyR3UGnaGw4PDDf8zpAfE= 2023-05-31T20:53:56.700 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:53:56 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:5IIC4KWyR3UGnaGw4PDDf8zpAfE=', 'Content-Length': '0'} 2023-05-31T20:53:56.702 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:44:50 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000fbef0891247ec31d-006477b3e4-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:53:56 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:53:56.703 DEBUG:boto:path=/pklxkd-59/ 2023-05-31T20:53:56.703 DEBUG:boto:auth_path=/pklxkd-59/ 2023-05-31T20:53:56.704 DEBUG:boto:path=/pklxkd-59/?prefix=obj95 2023-05-31T20:53:56.704 DEBUG:boto:auth_path=/pklxkd-59/?prefix=obj95 2023-05-31T20:53:56.704 DEBUG:boto:Method: GET 2023-05-31T20:53:56.705 DEBUG:boto:Path: /pklxkd-59/?prefix=obj95 2023-05-31T20:53:56.705 DEBUG:boto:Data: 2023-05-31T20:53:56.706 DEBUG:boto:Headers: {} 2023-05-31T20:53:56.706 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:53:56.706 DEBUG:boto:Port: 8001 2023-05-31T20:53:56.707 DEBUG:boto:Params: {} 2023-05-31T20:53:56.707 DEBUG:boto:Token: None 2023-05-31T20:53:56.707 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:53:56 GMT /pklxkd-59/ 2023-05-31T20:53:56.708 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:VxrsRCUGt15KLeSgquVd6DPBfJs= 2023-05-31T20:53:56.708 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:53:56 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:VxrsRCUGt15KLeSgquVd6DPBfJs=', 'Content-Length': '0'} 2023-05-31T20:53:56.711 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c5cb30baa6aee0c8-006477b3e4-1c17-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:53:56 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:53:56.711 DEBUG:boto:b'pklxkd-59obj951000falseobj952023-05-31T20:44:50.705Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:53:56.712 DEBUG:boto:path=/pklxkd-59/ 2023-05-31T20:53:56.712 DEBUG:boto:auth_path=/pklxkd-59/ 2023-05-31T20:53:56.713 DEBUG:boto:path=/pklxkd-59/?prefix=obj95 2023-05-31T20:53:56.713 DEBUG:boto:auth_path=/pklxkd-59/?prefix=obj95 2023-05-31T20:53:56.713 DEBUG:boto:Method: GET 2023-05-31T20:53:56.714 DEBUG:boto:Path: /pklxkd-59/?prefix=obj95 2023-05-31T20:53:56.714 DEBUG:boto:Data: 2023-05-31T20:53:56.714 DEBUG:boto:Headers: {} 2023-05-31T20:53:56.715 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:53:56.715 DEBUG:boto:Port: 8000 2023-05-31T20:53:56.715 DEBUG:boto:Params: {} 2023-05-31T20:53:56.716 DEBUG:boto:Token: None 2023-05-31T20:53:56.717 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:53:56 GMT /pklxkd-59/ 2023-05-31T20:53:56.717 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:VxrsRCUGt15KLeSgquVd6DPBfJs= 2023-05-31T20:53:56.717 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:53:56 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:VxrsRCUGt15KLeSgquVd6DPBfJs=', 'Content-Length': '0'} 2023-05-31T20:53:56.721 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000bdfb5db78ba10f40-006477b3e4-1630-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:53:56 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:53:56.722 DEBUG:boto:b'pklxkd-59obj951000falseobj952023-05-31T20:44:50.705Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:53:56.722 DEBUG:boto:path=/pklxkd-59/obj95 2023-05-31T20:53:56.723 DEBUG:boto:auth_path=/pklxkd-59/obj95 2023-05-31T20:53:56.723 DEBUG:boto:Method: HEAD 2023-05-31T20:53:56.723 DEBUG:boto:Path: /pklxkd-59/obj95 2023-05-31T20:53:56.724 DEBUG:boto:Data: 2023-05-31T20:53:56.724 DEBUG:boto:Headers: {} 2023-05-31T20:53:56.724 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:53:56.725 DEBUG:boto:Port: 8001 2023-05-31T20:53:56.725 DEBUG:boto:Params: {} 2023-05-31T20:53:56.725 DEBUG:boto:Token: None 2023-05-31T20:53:56.726 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:53:56 GMT /pklxkd-59/obj95 2023-05-31T20:53:56.726 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:RNKWsYcHqikMMGyYfgezXiETsOk= 2023-05-31T20:53:56.727 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:53:56 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:RNKWsYcHqikMMGyYfgezXiETsOk=', 'Content-Length': '0'} 2023-05-31T20:53:56.728 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:44:50 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000233294e60ed163e2-006477b3e4-1c17-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:53:56 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:53:56.729 DEBUG:boto:path=/pklxkd-59/obj95 2023-05-31T20:53:56.729 DEBUG:boto:auth_path=/pklxkd-59/obj95 2023-05-31T20:53:56.730 DEBUG:boto:Method: HEAD 2023-05-31T20:53:56.730 DEBUG:boto:Path: /pklxkd-59/obj95 2023-05-31T20:53:56.730 DEBUG:boto:Data: 2023-05-31T20:53:56.731 DEBUG:boto:Headers: {} 2023-05-31T20:53:56.731 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:53:56.731 DEBUG:boto:Port: 8000 2023-05-31T20:53:56.732 DEBUG:boto:Params: {} 2023-05-31T20:53:56.732 DEBUG:boto:Token: None 2023-05-31T20:53:56.732 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:53:56 GMT /pklxkd-59/obj95 2023-05-31T20:53:56.733 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:RNKWsYcHqikMMGyYfgezXiETsOk= 2023-05-31T20:53:56.733 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:53:56 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:RNKWsYcHqikMMGyYfgezXiETsOk=', 'Content-Length': '0'} 2023-05-31T20:53:56.735 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:44:50 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002b40a1051314ca1b-006477b3e4-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:53:56 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:53:56.735 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj95 2023-05-31T20:53:56.736 DEBUG:boto:path=/pklxkd-59/obj95 2023-05-31T20:53:56.736 DEBUG:boto:auth_path=/pklxkd-59/obj95 2023-05-31T20:53:56.737 DEBUG:boto:Method: GET 2023-05-31T20:53:56.737 DEBUG:boto:Path: /pklxkd-59/obj95 2023-05-31T20:53:56.737 DEBUG:boto:Data: 2023-05-31T20:53:56.738 DEBUG:boto:Headers: {} 2023-05-31T20:53:56.738 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:53:56.738 DEBUG:boto:Port: 8001 2023-05-31T20:53:56.739 DEBUG:boto:Params: {} 2023-05-31T20:53:56.739 DEBUG:boto:Token: None 2023-05-31T20:53:56.740 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:53:56 GMT /pklxkd-59/obj95 2023-05-31T20:53:56.740 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:tb8KYcDBietaLPNl49f+7l8IwoU= 2023-05-31T20:53:56.740 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:53:56 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:tb8KYcDBietaLPNl49f+7l8IwoU=', 'Content-Length': '0'} 2023-05-31T20:53:56.742 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:44:50 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c6bffe37b2993ee5-006477b3e4-1c17-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:53:56 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:53:56.743 DEBUG:boto:path=/pklxkd-59/obj95 2023-05-31T20:53:56.743 DEBUG:boto:auth_path=/pklxkd-59/obj95 2023-05-31T20:53:56.744 DEBUG:boto:Method: GET 2023-05-31T20:53:56.744 DEBUG:boto:Path: /pklxkd-59/obj95 2023-05-31T20:53:56.744 DEBUG:boto:Data: 2023-05-31T20:53:56.745 DEBUG:boto:Headers: {} 2023-05-31T20:53:56.745 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:53:56.746 DEBUG:boto:Port: 8000 2023-05-31T20:53:56.746 DEBUG:boto:Params: {} 2023-05-31T20:53:56.746 DEBUG:boto:Token: None 2023-05-31T20:53:56.747 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:53:56 GMT /pklxkd-59/obj95 2023-05-31T20:53:56.747 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:tb8KYcDBietaLPNl49f+7l8IwoU= 2023-05-31T20:53:56.748 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:53:56 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:tb8KYcDBietaLPNl49f+7l8IwoU=', 'Content-Length': '0'} 2023-05-31T20:53:56.749 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:44:50 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007840dab7f2ad24c7-006477b3e4-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:53:56 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:53:56.750 DEBUG:boto:path=/pklxkd-59/ 2023-05-31T20:53:56.750 DEBUG:boto:auth_path=/pklxkd-59/ 2023-05-31T20:53:56.751 DEBUG:boto:path=/pklxkd-59/?prefix=obj95 2023-05-31T20:53:56.751 DEBUG:boto:auth_path=/pklxkd-59/?prefix=obj95 2023-05-31T20:53:56.752 DEBUG:boto:Method: GET 2023-05-31T20:53:56.752 DEBUG:boto:Path: /pklxkd-59/?prefix=obj95 2023-05-31T20:53:56.752 DEBUG:boto:Data: 2023-05-31T20:53:56.753 DEBUG:boto:Headers: {} 2023-05-31T20:53:56.753 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:53:56.753 DEBUG:boto:Port: 8001 2023-05-31T20:53:56.754 DEBUG:boto:Params: {} 2023-05-31T20:53:56.754 DEBUG:boto:Token: None 2023-05-31T20:53:56.754 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:53:56 GMT /pklxkd-59/ 2023-05-31T20:53:56.755 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:VxrsRCUGt15KLeSgquVd6DPBfJs= 2023-05-31T20:53:56.755 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:53:56 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:VxrsRCUGt15KLeSgquVd6DPBfJs=', 'Content-Length': '0'} 2023-05-31T20:53:56.758 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000004da11eb3634a109c-006477b3e4-1c17-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:53:56 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:53:56.758 DEBUG:boto:b'pklxkd-59obj951000falseobj952023-05-31T20:44:50.705Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:53:56.759 DEBUG:boto:path=/pklxkd-59/ 2023-05-31T20:53:56.759 DEBUG:boto:auth_path=/pklxkd-59/ 2023-05-31T20:53:56.760 DEBUG:boto:path=/pklxkd-59/?prefix=obj95 2023-05-31T20:53:56.760 DEBUG:boto:auth_path=/pklxkd-59/?prefix=obj95 2023-05-31T20:53:56.760 DEBUG:boto:Method: GET 2023-05-31T20:53:56.761 DEBUG:boto:Path: /pklxkd-59/?prefix=obj95 2023-05-31T20:53:56.761 DEBUG:boto:Data: 2023-05-31T20:53:56.762 DEBUG:boto:Headers: {} 2023-05-31T20:53:56.762 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:53:56.762 DEBUG:boto:Port: 8000 2023-05-31T20:53:56.763 DEBUG:boto:Params: {} 2023-05-31T20:53:56.763 DEBUG:boto:Token: None 2023-05-31T20:53:56.763 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:53:56 GMT /pklxkd-59/ 2023-05-31T20:53:56.764 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:VxrsRCUGt15KLeSgquVd6DPBfJs= 2023-05-31T20:53:56.764 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:53:56 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:VxrsRCUGt15KLeSgquVd6DPBfJs=', 'Content-Length': '0'} 2023-05-31T20:53:56.768 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000003f65bbb76277edde-006477b3e4-1630-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:53:56 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:53:56.769 DEBUG:boto:b'pklxkd-59obj951000falseobj952023-05-31T20:44:50.705Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:53:56.770 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=pklxkd-59 zones={a2, a1} 2023-05-31T20:53:56.770 DEBUG:tasks.rgw.c1.client.1:waiting for process to exit 2023-05-31T20:53:56.771 INFO:teuthology.orchestra.run:waiting for 300 2023-05-31T20:53:56.772 INFO:tasks.rgw.c1.client.1.smithi099.stdout:2023-05-31T20:53:56.767+0000 7fb066f79700 -1 received signal: Terminated from /usr/bin/python3 /bin/daemon-helper term radosgw --rgw-frontends beast port=8001 -n client.1 --cluster c1 -k /etc/ceph/c1.client.1.keyring --log-file /var/log/ceph/rgw.c1.client.1.log --rgw_ops_log_socket_path /home/ubuntu/cephtest/rgw.opslog.c1.client.1.sock --foreground --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm (PID: 154202) UID: 0 2023-05-31T20:53:56.772 INFO:tasks.rgw.c1.client.1.smithi099.stdout:2023-05-31T20:53:56.767+0000 7fb06b1e27c0 -1 shutting down 2023-05-31T20:54:44.209 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.1 is failed for ~0s 2023-05-31T20:54:45.577 INFO:tasks.rgw.c1.client.1:Stopped 2023-05-31T20:54:45.577 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'list', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'pklxkd-59', '--max-entries', '10000', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-31T20:54:45.578 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'list', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'pklxkd-59', '--max-entries', '10000', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-31T20:54:45.579 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket list --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --bucket pklxkd-59 --max-entries 10000 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-05-31T20:54:45.639 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2023-05-31T20:54:45.639 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2023-05-31T20:54:45.696 INFO:teuthology.orchestra.run.smithi099.stdout:[ 2023-05-31T20:54:45.696 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T20:54:45.696 INFO:teuthology.orchestra.run.smithi099.stdout: "name": "obj11", 2023-05-31T20:54:45.696 INFO:teuthology.orchestra.run.smithi099.stdout: "instance": "", 2023-05-31T20:54:45.697 INFO:teuthology.orchestra.run.smithi099.stdout: "ver": { 2023-05-31T20:54:45.697 INFO:teuthology.orchestra.run.smithi099.stdout: "pool": 13, 2023-05-31T20:54:45.697 INFO:teuthology.orchestra.run.smithi099.stdout: "epoch": 6 2023-05-31T20:54:45.697 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:54:45.697 INFO:teuthology.orchestra.run.smithi099.stdout: "locator": "", 2023-05-31T20:54:45.697 INFO:teuthology.orchestra.run.smithi099.stdout: "exists": true, 2023-05-31T20:54:45.697 INFO:teuthology.orchestra.run.smithi099.stdout: "meta": { 2023-05-31T20:54:45.698 INFO:teuthology.orchestra.run.smithi099.stdout: "category": 1, 2023-05-31T20:54:45.698 INFO:teuthology.orchestra.run.smithi099.stdout: "size": 16, 2023-05-31T20:54:45.698 INFO:teuthology.orchestra.run.smithi099.stdout: "mtime": "2023-05-31T20:39:26.427662Z", 2023-05-31T20:54:45.698 INFO:teuthology.orchestra.run.smithi099.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-05-31T20:54:45.698 INFO:teuthology.orchestra.run.smithi099.stdout: "storage_class": "", 2023-05-31T20:54:45.698 INFO:teuthology.orchestra.run.smithi099.stdout: "owner": "rgw-multisite-test-user", 2023-05-31T20:54:45.698 INFO:teuthology.orchestra.run.smithi099.stdout: "owner_display_name": "Multisite Test User", 2023-05-31T20:54:45.699 INFO:teuthology.orchestra.run.smithi099.stdout: "content_type": "application/octet-stream", 2023-05-31T20:54:45.699 INFO:teuthology.orchestra.run.smithi099.stdout: "accounted_size": 3, 2023-05-31T20:54:45.699 INFO:teuthology.orchestra.run.smithi099.stdout: "user_data": "", 2023-05-31T20:54:45.699 INFO:teuthology.orchestra.run.smithi099.stdout: "appendable": false 2023-05-31T20:54:45.699 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:54:45.699 INFO:teuthology.orchestra.run.smithi099.stdout: "tag": "_QOKwNR2WnjcPxJ_uQWDGHP047Xuk_vE", 2023-05-31T20:54:45.699 INFO:teuthology.orchestra.run.smithi099.stdout: "flags": 0, 2023-05-31T20:54:45.700 INFO:teuthology.orchestra.run.smithi099.stdout: "pending_map": [], 2023-05-31T20:54:45.700 INFO:teuthology.orchestra.run.smithi099.stdout: "versioned_epoch": 0 2023-05-31T20:54:45.700 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:54:45.700 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T20:54:45.700 INFO:teuthology.orchestra.run.smithi099.stdout: "name": "obj111", 2023-05-31T20:54:45.700 INFO:teuthology.orchestra.run.smithi099.stdout: "instance": "", 2023-05-31T20:54:45.701 INFO:teuthology.orchestra.run.smithi099.stdout: "ver": { 2023-05-31T20:54:45.701 INFO:teuthology.orchestra.run.smithi099.stdout: "pool": 13, 2023-05-31T20:54:45.701 INFO:teuthology.orchestra.run.smithi099.stdout: "epoch": 4 2023-05-31T20:54:45.701 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:54:45.701 INFO:teuthology.orchestra.run.smithi099.stdout: "locator": "", 2023-05-31T20:54:45.701 INFO:teuthology.orchestra.run.smithi099.stdout: "exists": true, 2023-05-31T20:54:45.701 INFO:teuthology.orchestra.run.smithi099.stdout: "meta": { 2023-05-31T20:54:45.702 INFO:teuthology.orchestra.run.smithi099.stdout: "category": 1, 2023-05-31T20:54:45.702 INFO:teuthology.orchestra.run.smithi099.stdout: "size": 16, 2023-05-31T20:54:45.702 INFO:teuthology.orchestra.run.smithi099.stdout: "mtime": "2023-05-31T20:52:14.993359Z", 2023-05-31T20:54:45.702 INFO:teuthology.orchestra.run.smithi099.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-05-31T20:54:45.702 INFO:teuthology.orchestra.run.smithi099.stdout: "storage_class": "", 2023-05-31T20:54:45.702 INFO:teuthology.orchestra.run.smithi099.stdout: "owner": "rgw-multisite-test-user", 2023-05-31T20:54:45.703 INFO:teuthology.orchestra.run.smithi099.stdout: "owner_display_name": "Multisite Test User", 2023-05-31T20:54:45.703 INFO:teuthology.orchestra.run.smithi099.stdout: "content_type": "application/octet-stream", 2023-05-31T20:54:45.703 INFO:teuthology.orchestra.run.smithi099.stdout: "accounted_size": 3, 2023-05-31T20:54:45.703 INFO:teuthology.orchestra.run.smithi099.stdout: "user_data": "", 2023-05-31T20:54:45.703 INFO:teuthology.orchestra.run.smithi099.stdout: "appendable": false 2023-05-31T20:54:45.703 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:54:45.703 INFO:teuthology.orchestra.run.smithi099.stdout: "tag": "_ecpekl2qRjqdO3lRlyOy4hlr5I5wMWa", 2023-05-31T20:54:45.704 INFO:teuthology.orchestra.run.smithi099.stdout: "flags": 0, 2023-05-31T20:54:45.704 INFO:teuthology.orchestra.run.smithi099.stdout: "pending_map": [], 2023-05-31T20:54:45.704 INFO:teuthology.orchestra.run.smithi099.stdout: "versioned_epoch": 0 2023-05-31T20:54:45.704 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:54:45.704 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T20:54:45.704 INFO:teuthology.orchestra.run.smithi099.stdout: "name": "obj115", 2023-05-31T20:54:45.705 INFO:teuthology.orchestra.run.smithi099.stdout: "instance": "", 2023-05-31T20:54:45.705 INFO:teuthology.orchestra.run.smithi099.stdout: "ver": { 2023-05-31T20:54:45.705 INFO:teuthology.orchestra.run.smithi099.stdout: "pool": 13, 2023-05-31T20:54:45.705 INFO:teuthology.orchestra.run.smithi099.stdout: "epoch": 4 2023-05-31T20:54:45.705 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:54:45.706 INFO:teuthology.orchestra.run.smithi099.stdout: "locator": "", 2023-05-31T20:54:45.706 INFO:teuthology.orchestra.run.smithi099.stdout: "exists": true, 2023-05-31T20:54:45.706 INFO:teuthology.orchestra.run.smithi099.stdout: "meta": { 2023-05-31T20:54:45.706 INFO:teuthology.orchestra.run.smithi099.stdout: "category": 1, 2023-05-31T20:54:45.706 INFO:teuthology.orchestra.run.smithi099.stdout: "size": 6, 2023-05-31T20:54:45.706 INFO:teuthology.orchestra.run.smithi099.stdout: "mtime": "2023-05-31T20:46:33.958937Z", 2023-05-31T20:54:45.707 INFO:teuthology.orchestra.run.smithi099.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-05-31T20:54:45.707 INFO:teuthology.orchestra.run.smithi099.stdout: "storage_class": "", 2023-05-31T20:54:45.707 INFO:teuthology.orchestra.run.smithi099.stdout: "owner": "rgw-multisite-test-user", 2023-05-31T20:54:45.707 INFO:teuthology.orchestra.run.smithi099.stdout: "owner_display_name": "Multisite Test User", 2023-05-31T20:54:45.707 INFO:teuthology.orchestra.run.smithi099.stdout: "content_type": "application/octet-stream", 2023-05-31T20:54:45.707 INFO:teuthology.orchestra.run.smithi099.stdout: "accounted_size": 3, 2023-05-31T20:54:45.707 INFO:teuthology.orchestra.run.smithi099.stdout: "user_data": "", 2023-05-31T20:54:45.708 INFO:teuthology.orchestra.run.smithi099.stdout: "appendable": false 2023-05-31T20:54:45.708 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:54:45.708 INFO:teuthology.orchestra.run.smithi099.stdout: "tag": "_ej93p72YD96XDaH5JjrIO3i7PcQd2mO", 2023-05-31T20:54:45.708 INFO:teuthology.orchestra.run.smithi099.stdout: "flags": 0, 2023-05-31T20:54:45.708 INFO:teuthology.orchestra.run.smithi099.stdout: "pending_map": [], 2023-05-31T20:54:45.708 INFO:teuthology.orchestra.run.smithi099.stdout: "versioned_epoch": 0 2023-05-31T20:54:45.709 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:54:45.709 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T20:54:45.709 INFO:teuthology.orchestra.run.smithi099.stdout: "name": "obj116", 2023-05-31T20:54:45.709 INFO:teuthology.orchestra.run.smithi099.stdout: "instance": "", 2023-05-31T20:54:45.709 INFO:teuthology.orchestra.run.smithi099.stdout: "ver": { 2023-05-31T20:54:45.709 INFO:teuthology.orchestra.run.smithi099.stdout: "pool": 13, 2023-05-31T20:54:45.709 INFO:teuthology.orchestra.run.smithi099.stdout: "epoch": 11 2023-05-31T20:54:45.710 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:54:45.710 INFO:teuthology.orchestra.run.smithi099.stdout: "locator": "", 2023-05-31T20:54:45.710 INFO:teuthology.orchestra.run.smithi099.stdout: "exists": true, 2023-05-31T20:54:45.710 INFO:teuthology.orchestra.run.smithi099.stdout: "meta": { 2023-05-31T20:54:45.710 INFO:teuthology.orchestra.run.smithi099.stdout: "category": 1, 2023-05-31T20:54:45.710 INFO:teuthology.orchestra.run.smithi099.stdout: "size": 16, 2023-05-31T20:54:45.711 INFO:teuthology.orchestra.run.smithi099.stdout: "mtime": "2023-05-31T20:48:13.052647Z", 2023-05-31T20:54:45.711 INFO:teuthology.orchestra.run.smithi099.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-05-31T20:54:45.711 INFO:teuthology.orchestra.run.smithi099.stdout: "storage_class": "", 2023-05-31T20:54:45.711 INFO:teuthology.orchestra.run.smithi099.stdout: "owner": "rgw-multisite-test-user", 2023-05-31T20:54:45.711 INFO:teuthology.orchestra.run.smithi099.stdout: "owner_display_name": "Multisite Test User", 2023-05-31T20:54:45.711 INFO:teuthology.orchestra.run.smithi099.stdout: "content_type": "application/octet-stream", 2023-05-31T20:54:45.711 INFO:teuthology.orchestra.run.smithi099.stdout: "accounted_size": 3, 2023-05-31T20:54:45.712 INFO:teuthology.orchestra.run.smithi099.stdout: "user_data": "", 2023-05-31T20:54:45.712 INFO:teuthology.orchestra.run.smithi099.stdout: "appendable": false 2023-05-31T20:54:45.712 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:54:45.713 INFO:teuthology.orchestra.run.smithi099.stdout: "tag": "_AKyrvmoxWTG-teUrfeqQedZXaLsozTD", 2023-05-31T20:54:45.713 INFO:teuthology.orchestra.run.smithi099.stdout: "flags": 0, 2023-05-31T20:54:45.713 INFO:teuthology.orchestra.run.smithi099.stdout: "pending_map": [], 2023-05-31T20:54:45.714 INFO:teuthology.orchestra.run.smithi099.stdout: "versioned_epoch": 0 2023-05-31T20:54:45.714 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:54:45.714 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T20:54:45.715 INFO:teuthology.orchestra.run.smithi099.stdout: "name": "obj124", 2023-05-31T20:54:45.715 INFO:teuthology.orchestra.run.smithi099.stdout: "instance": "", 2023-05-31T20:54:45.715 INFO:teuthology.orchestra.run.smithi099.stdout: "ver": { 2023-05-31T20:54:45.715 INFO:teuthology.orchestra.run.smithi099.stdout: "pool": 13, 2023-05-31T20:54:45.715 INFO:teuthology.orchestra.run.smithi099.stdout: "epoch": 7 2023-05-31T20:54:45.715 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:54:45.716 INFO:teuthology.orchestra.run.smithi099.stdout: "locator": "", 2023-05-31T20:54:45.716 INFO:teuthology.orchestra.run.smithi099.stdout: "exists": true, 2023-05-31T20:54:45.716 INFO:teuthology.orchestra.run.smithi099.stdout: "meta": { 2023-05-31T20:54:45.716 INFO:teuthology.orchestra.run.smithi099.stdout: "category": 1, 2023-05-31T20:54:45.716 INFO:teuthology.orchestra.run.smithi099.stdout: "size": 3, 2023-05-31T20:54:45.716 INFO:teuthology.orchestra.run.smithi099.stdout: "mtime": "2023-05-31T20:50:33.587761Z", 2023-05-31T20:54:45.717 INFO:teuthology.orchestra.run.smithi099.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-05-31T20:54:45.717 INFO:teuthology.orchestra.run.smithi099.stdout: "storage_class": "", 2023-05-31T20:54:45.717 INFO:teuthology.orchestra.run.smithi099.stdout: "owner": "rgw-multisite-test-user", 2023-05-31T20:54:45.717 INFO:teuthology.orchestra.run.smithi099.stdout: "owner_display_name": "Multisite Test User", 2023-05-31T20:54:45.717 INFO:teuthology.orchestra.run.smithi099.stdout: "content_type": "application/octet-stream", 2023-05-31T20:54:45.717 INFO:teuthology.orchestra.run.smithi099.stdout: "accounted_size": 3, 2023-05-31T20:54:45.717 INFO:teuthology.orchestra.run.smithi099.stdout: "user_data": "", 2023-05-31T20:54:45.718 INFO:teuthology.orchestra.run.smithi099.stdout: "appendable": false 2023-05-31T20:54:45.718 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:54:45.718 INFO:teuthology.orchestra.run.smithi099.stdout: "tag": "_ZllECGIJJIDmCcmTbeTBLJvBBfhaNfd", 2023-05-31T20:54:45.718 INFO:teuthology.orchestra.run.smithi099.stdout: "flags": 0, 2023-05-31T20:54:45.718 INFO:teuthology.orchestra.run.smithi099.stdout: "pending_map": [], 2023-05-31T20:54:45.718 INFO:teuthology.orchestra.run.smithi099.stdout: "versioned_epoch": 0 2023-05-31T20:54:45.718 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:54:45.719 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T20:54:45.719 INFO:teuthology.orchestra.run.smithi099.stdout: "name": "obj145", 2023-05-31T20:54:45.719 INFO:teuthology.orchestra.run.smithi099.stdout: "instance": "", 2023-05-31T20:54:45.719 INFO:teuthology.orchestra.run.smithi099.stdout: "ver": { 2023-05-31T20:54:45.719 INFO:teuthology.orchestra.run.smithi099.stdout: "pool": 13, 2023-05-31T20:54:45.719 INFO:teuthology.orchestra.run.smithi099.stdout: "epoch": 10 2023-05-31T20:54:45.719 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:54:45.720 INFO:teuthology.orchestra.run.smithi099.stdout: "locator": "", 2023-05-31T20:54:45.720 INFO:teuthology.orchestra.run.smithi099.stdout: "exists": true, 2023-05-31T20:54:45.720 INFO:teuthology.orchestra.run.smithi099.stdout: "meta": { 2023-05-31T20:54:45.720 INFO:teuthology.orchestra.run.smithi099.stdout: "category": 1, 2023-05-31T20:54:45.720 INFO:teuthology.orchestra.run.smithi099.stdout: "size": 5, 2023-05-31T20:54:45.720 INFO:teuthology.orchestra.run.smithi099.stdout: "mtime": "2023-05-31T20:48:13.107892Z", 2023-05-31T20:54:45.721 INFO:teuthology.orchestra.run.smithi099.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-05-31T20:54:45.721 INFO:teuthology.orchestra.run.smithi099.stdout: "storage_class": "", 2023-05-31T20:54:45.721 INFO:teuthology.orchestra.run.smithi099.stdout: "owner": "rgw-multisite-test-user", 2023-05-31T20:54:45.721 INFO:teuthology.orchestra.run.smithi099.stdout: "owner_display_name": "Multisite Test User", 2023-05-31T20:54:45.721 INFO:teuthology.orchestra.run.smithi099.stdout: "content_type": "application/octet-stream", 2023-05-31T20:54:45.721 INFO:teuthology.orchestra.run.smithi099.stdout: "accounted_size": 3, 2023-05-31T20:54:45.721 INFO:teuthology.orchestra.run.smithi099.stdout: "user_data": "", 2023-05-31T20:54:45.722 INFO:teuthology.orchestra.run.smithi099.stdout: "appendable": false 2023-05-31T20:54:45.722 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:54:45.722 INFO:teuthology.orchestra.run.smithi099.stdout: "tag": "_nqkp-xZCjHGxL6d3rV1ACV1Y3LKN3bz", 2023-05-31T20:54:45.722 INFO:teuthology.orchestra.run.smithi099.stdout: "flags": 0, 2023-05-31T20:54:45.722 INFO:teuthology.orchestra.run.smithi099.stdout: "pending_map": [], 2023-05-31T20:54:45.722 INFO:teuthology.orchestra.run.smithi099.stdout: "versioned_epoch": 0 2023-05-31T20:54:45.723 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:54:45.723 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T20:54:45.723 INFO:teuthology.orchestra.run.smithi099.stdout: "name": "obj148", 2023-05-31T20:54:45.724 INFO:teuthology.orchestra.run.smithi099.stdout: "instance": "", 2023-05-31T20:54:45.724 INFO:teuthology.orchestra.run.smithi099.stdout: "ver": { 2023-05-31T20:54:45.724 INFO:teuthology.orchestra.run.smithi099.stdout: "pool": 13, 2023-05-31T20:54:45.724 INFO:teuthology.orchestra.run.smithi099.stdout: "epoch": 3 2023-05-31T20:54:45.725 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:54:45.725 INFO:teuthology.orchestra.run.smithi099.stdout: "locator": "", 2023-05-31T20:54:45.725 INFO:teuthology.orchestra.run.smithi099.stdout: "exists": true, 2023-05-31T20:54:45.725 INFO:teuthology.orchestra.run.smithi099.stdout: "meta": { 2023-05-31T20:54:45.726 INFO:teuthology.orchestra.run.smithi099.stdout: "category": 1, 2023-05-31T20:54:45.726 INFO:teuthology.orchestra.run.smithi099.stdout: "size": 16, 2023-05-31T20:54:45.726 INFO:teuthology.orchestra.run.smithi099.stdout: "mtime": "2023-05-31T20:52:15.015083Z", 2023-05-31T20:54:45.726 INFO:teuthology.orchestra.run.smithi099.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-05-31T20:54:45.726 INFO:teuthology.orchestra.run.smithi099.stdout: "storage_class": "", 2023-05-31T20:54:45.726 INFO:teuthology.orchestra.run.smithi099.stdout: "owner": "rgw-multisite-test-user", 2023-05-31T20:54:45.726 INFO:teuthology.orchestra.run.smithi099.stdout: "owner_display_name": "Multisite Test User", 2023-05-31T20:54:45.727 INFO:teuthology.orchestra.run.smithi099.stdout: "content_type": "application/octet-stream", 2023-05-31T20:54:45.727 INFO:teuthology.orchestra.run.smithi099.stdout: "accounted_size": 3, 2023-05-31T20:54:45.727 INFO:teuthology.orchestra.run.smithi099.stdout: "user_data": "", 2023-05-31T20:54:45.727 INFO:teuthology.orchestra.run.smithi099.stdout: "appendable": false 2023-05-31T20:54:45.727 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:54:45.727 INFO:teuthology.orchestra.run.smithi099.stdout: "tag": "_KAM3eOjbcw213dOW-k_YQfufrYonlbX", 2023-05-31T20:54:45.727 INFO:teuthology.orchestra.run.smithi099.stdout: "flags": 0, 2023-05-31T20:54:45.728 INFO:teuthology.orchestra.run.smithi099.stdout: "pending_map": [], 2023-05-31T20:54:45.728 INFO:teuthology.orchestra.run.smithi099.stdout: "versioned_epoch": 0 2023-05-31T20:54:45.728 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:54:45.728 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T20:54:45.728 INFO:teuthology.orchestra.run.smithi099.stdout: "name": "obj155", 2023-05-31T20:54:45.729 INFO:teuthology.orchestra.run.smithi099.stdout: "instance": "", 2023-05-31T20:54:45.729 INFO:teuthology.orchestra.run.smithi099.stdout: "ver": { 2023-05-31T20:54:45.729 INFO:teuthology.orchestra.run.smithi099.stdout: "pool": 13, 2023-05-31T20:54:45.729 INFO:teuthology.orchestra.run.smithi099.stdout: "epoch": 13 2023-05-31T20:54:45.729 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:54:45.729 INFO:teuthology.orchestra.run.smithi099.stdout: "locator": "", 2023-05-31T20:54:45.730 INFO:teuthology.orchestra.run.smithi099.stdout: "exists": true, 2023-05-31T20:54:45.730 INFO:teuthology.orchestra.run.smithi099.stdout: "meta": { 2023-05-31T20:54:45.730 INFO:teuthology.orchestra.run.smithi099.stdout: "category": 1, 2023-05-31T20:54:45.730 INFO:teuthology.orchestra.run.smithi099.stdout: "size": 16, 2023-05-31T20:54:45.730 INFO:teuthology.orchestra.run.smithi099.stdout: "mtime": "2023-05-31T20:50:33.605763Z", 2023-05-31T20:54:45.730 INFO:teuthology.orchestra.run.smithi099.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-05-31T20:54:45.730 INFO:teuthology.orchestra.run.smithi099.stdout: "storage_class": "", 2023-05-31T20:54:45.731 INFO:teuthology.orchestra.run.smithi099.stdout: "owner": "rgw-multisite-test-user", 2023-05-31T20:54:45.731 INFO:teuthology.orchestra.run.smithi099.stdout: "owner_display_name": "Multisite Test User", 2023-05-31T20:54:45.731 INFO:teuthology.orchestra.run.smithi099.stdout: "content_type": "application/octet-stream", 2023-05-31T20:54:45.731 INFO:teuthology.orchestra.run.smithi099.stdout: "accounted_size": 3, 2023-05-31T20:54:45.731 INFO:teuthology.orchestra.run.smithi099.stdout: "user_data": "", 2023-05-31T20:54:45.731 INFO:teuthology.orchestra.run.smithi099.stdout: "appendable": false 2023-05-31T20:54:45.731 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:54:45.732 INFO:teuthology.orchestra.run.smithi099.stdout: "tag": "_zNpY00Umm43fYp2Vdr8lUoQa7HFPGBA", 2023-05-31T20:54:45.732 INFO:teuthology.orchestra.run.smithi099.stdout: "flags": 0, 2023-05-31T20:54:45.732 INFO:teuthology.orchestra.run.smithi099.stdout: "pending_map": [], 2023-05-31T20:54:45.732 INFO:teuthology.orchestra.run.smithi099.stdout: "versioned_epoch": 0 2023-05-31T20:54:45.732 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:54:45.732 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T20:54:45.733 INFO:teuthology.orchestra.run.smithi099.stdout: "name": "obj17", 2023-05-31T20:54:45.733 INFO:teuthology.orchestra.run.smithi099.stdout: "instance": "", 2023-05-31T20:54:45.733 INFO:teuthology.orchestra.run.smithi099.stdout: "ver": { 2023-05-31T20:54:45.733 INFO:teuthology.orchestra.run.smithi099.stdout: "pool": 13, 2023-05-31T20:54:45.733 INFO:teuthology.orchestra.run.smithi099.stdout: "epoch": 5 2023-05-31T20:54:45.733 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:54:45.733 INFO:teuthology.orchestra.run.smithi099.stdout: "locator": "", 2023-05-31T20:54:45.734 INFO:teuthology.orchestra.run.smithi099.stdout: "exists": true, 2023-05-31T20:54:45.734 INFO:teuthology.orchestra.run.smithi099.stdout: "meta": { 2023-05-31T20:54:45.734 INFO:teuthology.orchestra.run.smithi099.stdout: "category": 1, 2023-05-31T20:54:45.734 INFO:teuthology.orchestra.run.smithi099.stdout: "size": 3, 2023-05-31T20:54:45.734 INFO:teuthology.orchestra.run.smithi099.stdout: "mtime": "2023-05-31T20:43:07.977285Z", 2023-05-31T20:54:45.734 INFO:teuthology.orchestra.run.smithi099.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-05-31T20:54:45.734 INFO:teuthology.orchestra.run.smithi099.stdout: "storage_class": "", 2023-05-31T20:54:45.735 INFO:teuthology.orchestra.run.smithi099.stdout: "owner": "rgw-multisite-test-user", 2023-05-31T20:54:45.735 INFO:teuthology.orchestra.run.smithi099.stdout: "owner_display_name": "Multisite Test User", 2023-05-31T20:54:45.735 INFO:teuthology.orchestra.run.smithi099.stdout: "content_type": "application/octet-stream", 2023-05-31T20:54:45.735 INFO:teuthology.orchestra.run.smithi099.stdout: "accounted_size": 3, 2023-05-31T20:54:45.735 INFO:teuthology.orchestra.run.smithi099.stdout: "user_data": "", 2023-05-31T20:54:45.735 INFO:teuthology.orchestra.run.smithi099.stdout: "appendable": false 2023-05-31T20:54:45.735 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:54:45.736 INFO:teuthology.orchestra.run.smithi099.stdout: "tag": "_NEzZyzHjEMoKjKBKTkNC5XuXrnXe5vw", 2023-05-31T20:54:45.736 INFO:teuthology.orchestra.run.smithi099.stdout: "flags": 0, 2023-05-31T20:54:45.736 INFO:teuthology.orchestra.run.smithi099.stdout: "pending_map": [], 2023-05-31T20:54:45.736 INFO:teuthology.orchestra.run.smithi099.stdout: "versioned_epoch": 0 2023-05-31T20:54:45.736 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:54:45.736 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T20:54:45.737 INFO:teuthology.orchestra.run.smithi099.stdout: "name": "obj185", 2023-05-31T20:54:45.737 INFO:teuthology.orchestra.run.smithi099.stdout: "instance": "", 2023-05-31T20:54:45.737 INFO:teuthology.orchestra.run.smithi099.stdout: "ver": { 2023-05-31T20:54:45.737 INFO:teuthology.orchestra.run.smithi099.stdout: "pool": 13, 2023-05-31T20:54:45.737 INFO:teuthology.orchestra.run.smithi099.stdout: "epoch": 9 2023-05-31T20:54:45.737 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:54:45.737 INFO:teuthology.orchestra.run.smithi099.stdout: "locator": "", 2023-05-31T20:54:45.738 INFO:teuthology.orchestra.run.smithi099.stdout: "exists": true, 2023-05-31T20:54:45.738 INFO:teuthology.orchestra.run.smithi099.stdout: "meta": { 2023-05-31T20:54:45.738 INFO:teuthology.orchestra.run.smithi099.stdout: "category": 1, 2023-05-31T20:54:45.738 INFO:teuthology.orchestra.run.smithi099.stdout: "size": 3, 2023-05-31T20:54:45.738 INFO:teuthology.orchestra.run.smithi099.stdout: "mtime": "2023-05-31T20:52:15.072437Z", 2023-05-31T20:54:45.738 INFO:teuthology.orchestra.run.smithi099.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-05-31T20:54:45.739 INFO:teuthology.orchestra.run.smithi099.stdout: "storage_class": "", 2023-05-31T20:54:45.739 INFO:teuthology.orchestra.run.smithi099.stdout: "owner": "rgw-multisite-test-user", 2023-05-31T20:54:45.739 INFO:teuthology.orchestra.run.smithi099.stdout: "owner_display_name": "Multisite Test User", 2023-05-31T20:54:45.739 INFO:teuthology.orchestra.run.smithi099.stdout: "content_type": "application/octet-stream", 2023-05-31T20:54:45.739 INFO:teuthology.orchestra.run.smithi099.stdout: "accounted_size": 3, 2023-05-31T20:54:45.739 INFO:teuthology.orchestra.run.smithi099.stdout: "user_data": "", 2023-05-31T20:54:45.739 INFO:teuthology.orchestra.run.smithi099.stdout: "appendable": false 2023-05-31T20:54:45.740 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:54:45.740 INFO:teuthology.orchestra.run.smithi099.stdout: "tag": "_c0-zPc3a8iCmT8iPn__cOgjY1_4tJuW", 2023-05-31T20:54:45.740 INFO:teuthology.orchestra.run.smithi099.stdout: "flags": 0, 2023-05-31T20:54:45.740 INFO:teuthology.orchestra.run.smithi099.stdout: "pending_map": [], 2023-05-31T20:54:45.740 INFO:teuthology.orchestra.run.smithi099.stdout: "versioned_epoch": 0 2023-05-31T20:54:45.740 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:54:45.741 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T20:54:45.741 INFO:teuthology.orchestra.run.smithi099.stdout: "name": "obj19", 2023-05-31T20:54:45.741 INFO:teuthology.orchestra.run.smithi099.stdout: "instance": "", 2023-05-31T20:54:45.741 INFO:teuthology.orchestra.run.smithi099.stdout: "ver": { 2023-05-31T20:54:45.741 INFO:teuthology.orchestra.run.smithi099.stdout: "pool": 13, 2023-05-31T20:54:45.741 INFO:teuthology.orchestra.run.smithi099.stdout: "epoch": 8 2023-05-31T20:54:45.741 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:54:45.742 INFO:teuthology.orchestra.run.smithi099.stdout: "locator": "", 2023-05-31T20:54:45.742 INFO:teuthology.orchestra.run.smithi099.stdout: "exists": true, 2023-05-31T20:54:45.742 INFO:teuthology.orchestra.run.smithi099.stdout: "meta": { 2023-05-31T20:54:45.742 INFO:teuthology.orchestra.run.smithi099.stdout: "category": 1, 2023-05-31T20:54:45.742 INFO:teuthology.orchestra.run.smithi099.stdout: "size": 16, 2023-05-31T20:54:45.742 INFO:teuthology.orchestra.run.smithi099.stdout: "mtime": "2023-05-31T20:44:50.557118Z", 2023-05-31T20:54:45.742 INFO:teuthology.orchestra.run.smithi099.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-05-31T20:54:45.743 INFO:teuthology.orchestra.run.smithi099.stdout: "storage_class": "", 2023-05-31T20:54:45.743 INFO:teuthology.orchestra.run.smithi099.stdout: "owner": "rgw-multisite-test-user", 2023-05-31T20:54:45.743 INFO:teuthology.orchestra.run.smithi099.stdout: "owner_display_name": "Multisite Test User", 2023-05-31T20:54:45.743 INFO:teuthology.orchestra.run.smithi099.stdout: "content_type": "application/octet-stream", 2023-05-31T20:54:45.743 INFO:teuthology.orchestra.run.smithi099.stdout: "accounted_size": 3, 2023-05-31T20:54:45.743 INFO:teuthology.orchestra.run.smithi099.stdout: "user_data": "", 2023-05-31T20:54:45.744 INFO:teuthology.orchestra.run.smithi099.stdout: "appendable": false 2023-05-31T20:54:45.744 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:54:45.744 INFO:teuthology.orchestra.run.smithi099.stdout: "tag": "_H7SECQtaLIAoR06J_1hfLSpTscYnvMV", 2023-05-31T20:54:45.744 INFO:teuthology.orchestra.run.smithi099.stdout: "flags": 0, 2023-05-31T20:54:45.744 INFO:teuthology.orchestra.run.smithi099.stdout: "pending_map": [], 2023-05-31T20:54:45.744 INFO:teuthology.orchestra.run.smithi099.stdout: "versioned_epoch": 0 2023-05-31T20:54:45.744 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:54:45.745 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T20:54:45.745 INFO:teuthology.orchestra.run.smithi099.stdout: "name": "obj22", 2023-05-31T20:54:45.745 INFO:teuthology.orchestra.run.smithi099.stdout: "instance": "", 2023-05-31T20:54:45.745 INFO:teuthology.orchestra.run.smithi099.stdout: "ver": { 2023-05-31T20:54:45.745 INFO:teuthology.orchestra.run.smithi099.stdout: "pool": 13, 2023-05-31T20:54:45.745 INFO:teuthology.orchestra.run.smithi099.stdout: "epoch": 7 2023-05-31T20:54:45.745 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:54:45.746 INFO:teuthology.orchestra.run.smithi099.stdout: "locator": "", 2023-05-31T20:54:45.746 INFO:teuthology.orchestra.run.smithi099.stdout: "exists": true, 2023-05-31T20:54:45.746 INFO:teuthology.orchestra.run.smithi099.stdout: "meta": { 2023-05-31T20:54:45.746 INFO:teuthology.orchestra.run.smithi099.stdout: "category": 1, 2023-05-31T20:54:45.746 INFO:teuthology.orchestra.run.smithi099.stdout: "size": 16, 2023-05-31T20:54:45.746 INFO:teuthology.orchestra.run.smithi099.stdout: "mtime": "2023-05-31T20:39:26.473296Z", 2023-05-31T20:54:45.746 INFO:teuthology.orchestra.run.smithi099.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-05-31T20:54:45.747 INFO:teuthology.orchestra.run.smithi099.stdout: "storage_class": "", 2023-05-31T20:54:45.747 INFO:teuthology.orchestra.run.smithi099.stdout: "owner": "rgw-multisite-test-user", 2023-05-31T20:54:45.747 INFO:teuthology.orchestra.run.smithi099.stdout: "owner_display_name": "Multisite Test User", 2023-05-31T20:54:45.747 INFO:teuthology.orchestra.run.smithi099.stdout: "content_type": "application/octet-stream", 2023-05-31T20:54:45.747 INFO:teuthology.orchestra.run.smithi099.stdout: "accounted_size": 3, 2023-05-31T20:54:45.747 INFO:teuthology.orchestra.run.smithi099.stdout: "user_data": "", 2023-05-31T20:54:45.747 INFO:teuthology.orchestra.run.smithi099.stdout: "appendable": false 2023-05-31T20:54:45.748 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:54:45.748 INFO:teuthology.orchestra.run.smithi099.stdout: "tag": "_3-W3IG3cmI2R8PO79ATyKbmwxiFBaBP", 2023-05-31T20:54:45.748 INFO:teuthology.orchestra.run.smithi099.stdout: "flags": 0, 2023-05-31T20:54:45.748 INFO:teuthology.orchestra.run.smithi099.stdout: "pending_map": [], 2023-05-31T20:54:45.748 INFO:teuthology.orchestra.run.smithi099.stdout: "versioned_epoch": 0 2023-05-31T20:54:45.749 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:54:45.749 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T20:54:45.749 INFO:teuthology.orchestra.run.smithi099.stdout: "name": "obj23", 2023-05-31T20:54:45.749 INFO:teuthology.orchestra.run.smithi099.stdout: "instance": "", 2023-05-31T20:54:45.749 INFO:teuthology.orchestra.run.smithi099.stdout: "ver": { 2023-05-31T20:54:45.749 INFO:teuthology.orchestra.run.smithi099.stdout: "pool": 13, 2023-05-31T20:54:45.749 INFO:teuthology.orchestra.run.smithi099.stdout: "epoch": 5 2023-05-31T20:54:45.750 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:54:45.750 INFO:teuthology.orchestra.run.smithi099.stdout: "locator": "", 2023-05-31T20:54:45.750 INFO:teuthology.orchestra.run.smithi099.stdout: "exists": true, 2023-05-31T20:54:45.750 INFO:teuthology.orchestra.run.smithi099.stdout: "meta": { 2023-05-31T20:54:45.750 INFO:teuthology.orchestra.run.smithi099.stdout: "category": 1, 2023-05-31T20:54:45.750 INFO:teuthology.orchestra.run.smithi099.stdout: "size": 6, 2023-05-31T20:54:45.750 INFO:teuthology.orchestra.run.smithi099.stdout: "mtime": "2023-05-31T20:46:33.813577Z", 2023-05-31T20:54:45.751 INFO:teuthology.orchestra.run.smithi099.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-05-31T20:54:45.751 INFO:teuthology.orchestra.run.smithi099.stdout: "storage_class": "", 2023-05-31T20:54:45.751 INFO:teuthology.orchestra.run.smithi099.stdout: "owner": "rgw-multisite-test-user", 2023-05-31T20:54:45.751 INFO:teuthology.orchestra.run.smithi099.stdout: "owner_display_name": "Multisite Test User", 2023-05-31T20:54:45.751 INFO:teuthology.orchestra.run.smithi099.stdout: "content_type": "application/octet-stream", 2023-05-31T20:54:45.752 INFO:teuthology.orchestra.run.smithi099.stdout: "accounted_size": 3, 2023-05-31T20:54:45.752 INFO:teuthology.orchestra.run.smithi099.stdout: "user_data": "", 2023-05-31T20:54:45.752 INFO:teuthology.orchestra.run.smithi099.stdout: "appendable": false 2023-05-31T20:54:45.752 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:54:45.752 INFO:teuthology.orchestra.run.smithi099.stdout: "tag": "_m9vorJaNd3uG-ezD_Lve_5YhlfSMP3e", 2023-05-31T20:54:45.752 INFO:teuthology.orchestra.run.smithi099.stdout: "flags": 0, 2023-05-31T20:54:45.752 INFO:teuthology.orchestra.run.smithi099.stdout: "pending_map": [], 2023-05-31T20:54:45.753 INFO:teuthology.orchestra.run.smithi099.stdout: "versioned_epoch": 0 2023-05-31T20:54:45.753 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:54:45.753 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T20:54:45.753 INFO:teuthology.orchestra.run.smithi099.stdout: "name": "obj29", 2023-05-31T20:54:45.753 INFO:teuthology.orchestra.run.smithi099.stdout: "instance": "", 2023-05-31T20:54:45.753 INFO:teuthology.orchestra.run.smithi099.stdout: "ver": { 2023-05-31T20:54:45.754 INFO:teuthology.orchestra.run.smithi099.stdout: "pool": 13, 2023-05-31T20:54:45.754 INFO:teuthology.orchestra.run.smithi099.stdout: "epoch": 2 2023-05-31T20:54:45.754 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:54:45.754 INFO:teuthology.orchestra.run.smithi099.stdout: "locator": "", 2023-05-31T20:54:45.754 INFO:teuthology.orchestra.run.smithi099.stdout: "exists": true, 2023-05-31T20:54:45.754 INFO:teuthology.orchestra.run.smithi099.stdout: "meta": { 2023-05-31T20:54:45.754 INFO:teuthology.orchestra.run.smithi099.stdout: "category": 1, 2023-05-31T20:54:45.755 INFO:teuthology.orchestra.run.smithi099.stdout: "size": 3, 2023-05-31T20:54:45.755 INFO:teuthology.orchestra.run.smithi099.stdout: "mtime": "2023-05-31T20:48:12.956068Z", 2023-05-31T20:54:45.755 INFO:teuthology.orchestra.run.smithi099.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-05-31T20:54:45.755 INFO:teuthology.orchestra.run.smithi099.stdout: "storage_class": "", 2023-05-31T20:54:45.755 INFO:teuthology.orchestra.run.smithi099.stdout: "owner": "rgw-multisite-test-user", 2023-05-31T20:54:45.755 INFO:teuthology.orchestra.run.smithi099.stdout: "owner_display_name": "Multisite Test User", 2023-05-31T20:54:45.755 INFO:teuthology.orchestra.run.smithi099.stdout: "content_type": "application/octet-stream", 2023-05-31T20:54:45.756 INFO:teuthology.orchestra.run.smithi099.stdout: "accounted_size": 3, 2023-05-31T20:54:45.756 INFO:teuthology.orchestra.run.smithi099.stdout: "user_data": "", 2023-05-31T20:54:45.756 INFO:teuthology.orchestra.run.smithi099.stdout: "appendable": false 2023-05-31T20:54:45.756 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:54:45.756 INFO:teuthology.orchestra.run.smithi099.stdout: "tag": "_8LM22Sx1nXj9bATslnQI2jREvd6u9Gr", 2023-05-31T20:54:45.756 INFO:teuthology.orchestra.run.smithi099.stdout: "flags": 0, 2023-05-31T20:54:45.756 INFO:teuthology.orchestra.run.smithi099.stdout: "pending_map": [], 2023-05-31T20:54:45.757 INFO:teuthology.orchestra.run.smithi099.stdout: "versioned_epoch": 0 2023-05-31T20:54:45.757 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:54:45.757 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T20:54:45.757 INFO:teuthology.orchestra.run.smithi099.stdout: "name": "obj31", 2023-05-31T20:54:45.757 INFO:teuthology.orchestra.run.smithi099.stdout: "instance": "", 2023-05-31T20:54:45.757 INFO:teuthology.orchestra.run.smithi099.stdout: "ver": { 2023-05-31T20:54:45.757 INFO:teuthology.orchestra.run.smithi099.stdout: "pool": 13, 2023-05-31T20:54:45.758 INFO:teuthology.orchestra.run.smithi099.stdout: "epoch": 4 2023-05-31T20:54:45.758 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:54:45.758 INFO:teuthology.orchestra.run.smithi099.stdout: "locator": "", 2023-05-31T20:54:45.758 INFO:teuthology.orchestra.run.smithi099.stdout: "exists": true, 2023-05-31T20:54:45.758 INFO:teuthology.orchestra.run.smithi099.stdout: "meta": { 2023-05-31T20:54:45.758 INFO:teuthology.orchestra.run.smithi099.stdout: "category": 1, 2023-05-31T20:54:45.759 INFO:teuthology.orchestra.run.smithi099.stdout: "size": 3, 2023-05-31T20:54:45.759 INFO:teuthology.orchestra.run.smithi099.stdout: "mtime": "2023-05-31T20:50:33.460419Z", 2023-05-31T20:54:45.759 INFO:teuthology.orchestra.run.smithi099.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-05-31T20:54:45.759 INFO:teuthology.orchestra.run.smithi099.stdout: "storage_class": "", 2023-05-31T20:54:45.759 INFO:teuthology.orchestra.run.smithi099.stdout: "owner": "rgw-multisite-test-user", 2023-05-31T20:54:45.759 INFO:teuthology.orchestra.run.smithi099.stdout: "owner_display_name": "Multisite Test User", 2023-05-31T20:54:45.759 INFO:teuthology.orchestra.run.smithi099.stdout: "content_type": "application/octet-stream", 2023-05-31T20:54:45.760 INFO:teuthology.orchestra.run.smithi099.stdout: "accounted_size": 3, 2023-05-31T20:54:45.760 INFO:teuthology.orchestra.run.smithi099.stdout: "user_data": "", 2023-05-31T20:54:45.760 INFO:teuthology.orchestra.run.smithi099.stdout: "appendable": false 2023-05-31T20:54:45.760 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:54:45.760 INFO:teuthology.orchestra.run.smithi099.stdout: "tag": "_bZrlri4weZoXDnh1adBbM7CdM7CcEFE", 2023-05-31T20:54:45.760 INFO:teuthology.orchestra.run.smithi099.stdout: "flags": 0, 2023-05-31T20:54:45.761 INFO:teuthology.orchestra.run.smithi099.stdout: "pending_map": [], 2023-05-31T20:54:45.761 INFO:teuthology.orchestra.run.smithi099.stdout: "versioned_epoch": 0 2023-05-31T20:54:45.761 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:54:45.761 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T20:54:45.761 INFO:teuthology.orchestra.run.smithi099.stdout: "name": "obj33", 2023-05-31T20:54:45.761 INFO:teuthology.orchestra.run.smithi099.stdout: "instance": "", 2023-05-31T20:54:45.761 INFO:teuthology.orchestra.run.smithi099.stdout: "ver": { 2023-05-31T20:54:45.762 INFO:teuthology.orchestra.run.smithi099.stdout: "pool": 13, 2023-05-31T20:54:45.762 INFO:teuthology.orchestra.run.smithi099.stdout: "epoch": 18 2023-05-31T20:54:45.762 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:54:45.762 INFO:teuthology.orchestra.run.smithi099.stdout: "locator": "", 2023-05-31T20:54:45.763 INFO:teuthology.orchestra.run.smithi099.stdout: "exists": true, 2023-05-31T20:54:45.763 INFO:teuthology.orchestra.run.smithi099.stdout: "meta": { 2023-05-31T20:54:45.763 INFO:teuthology.orchestra.run.smithi099.stdout: "category": 1, 2023-05-31T20:54:45.763 INFO:teuthology.orchestra.run.smithi099.stdout: "size": 6, 2023-05-31T20:54:45.763 INFO:teuthology.orchestra.run.smithi099.stdout: "mtime": "2023-05-31T20:39:26.534549Z", 2023-05-31T20:54:45.764 INFO:teuthology.orchestra.run.smithi099.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-05-31T20:54:45.764 INFO:teuthology.orchestra.run.smithi099.stdout: "storage_class": "", 2023-05-31T20:54:45.764 INFO:teuthology.orchestra.run.smithi099.stdout: "owner": "rgw-multisite-test-user", 2023-05-31T20:54:45.764 INFO:teuthology.orchestra.run.smithi099.stdout: "owner_display_name": "Multisite Test User", 2023-05-31T20:54:45.764 INFO:teuthology.orchestra.run.smithi099.stdout: "content_type": "application/octet-stream", 2023-05-31T20:54:45.764 INFO:teuthology.orchestra.run.smithi099.stdout: "accounted_size": 3, 2023-05-31T20:54:45.765 INFO:teuthology.orchestra.run.smithi099.stdout: "user_data": "", 2023-05-31T20:54:45.765 INFO:teuthology.orchestra.run.smithi099.stdout: "appendable": false 2023-05-31T20:54:45.765 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:54:45.765 INFO:teuthology.orchestra.run.smithi099.stdout: "tag": "_-4RBje-8l9GN9v5aK3u67ojqaJLuyPH", 2023-05-31T20:54:45.765 INFO:teuthology.orchestra.run.smithi099.stdout: "flags": 0, 2023-05-31T20:54:45.765 INFO:teuthology.orchestra.run.smithi099.stdout: "pending_map": [], 2023-05-31T20:54:45.765 INFO:teuthology.orchestra.run.smithi099.stdout: "versioned_epoch": 0 2023-05-31T20:54:45.766 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:54:45.766 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T20:54:45.766 INFO:teuthology.orchestra.run.smithi099.stdout: "name": "obj34", 2023-05-31T20:54:45.766 INFO:teuthology.orchestra.run.smithi099.stdout: "instance": "", 2023-05-31T20:54:45.766 INFO:teuthology.orchestra.run.smithi099.stdout: "ver": { 2023-05-31T20:54:45.766 INFO:teuthology.orchestra.run.smithi099.stdout: "pool": 13, 2023-05-31T20:54:45.767 INFO:teuthology.orchestra.run.smithi099.stdout: "epoch": 8 2023-05-31T20:54:45.767 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:54:45.767 INFO:teuthology.orchestra.run.smithi099.stdout: "locator": "", 2023-05-31T20:54:45.767 INFO:teuthology.orchestra.run.smithi099.stdout: "exists": true, 2023-05-31T20:54:45.767 INFO:teuthology.orchestra.run.smithi099.stdout: "meta": { 2023-05-31T20:54:45.767 INFO:teuthology.orchestra.run.smithi099.stdout: "category": 1, 2023-05-31T20:54:45.767 INFO:teuthology.orchestra.run.smithi099.stdout: "size": 16, 2023-05-31T20:54:45.768 INFO:teuthology.orchestra.run.smithi099.stdout: "mtime": "2023-05-31T20:43:07.994147Z", 2023-05-31T20:54:45.768 INFO:teuthology.orchestra.run.smithi099.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-05-31T20:54:45.768 INFO:teuthology.orchestra.run.smithi099.stdout: "storage_class": "", 2023-05-31T20:54:45.768 INFO:teuthology.orchestra.run.smithi099.stdout: "owner": "rgw-multisite-test-user", 2023-05-31T20:54:45.768 INFO:teuthology.orchestra.run.smithi099.stdout: "owner_display_name": "Multisite Test User", 2023-05-31T20:54:45.768 INFO:teuthology.orchestra.run.smithi099.stdout: "content_type": "application/octet-stream", 2023-05-31T20:54:45.768 INFO:teuthology.orchestra.run.smithi099.stdout: "accounted_size": 3, 2023-05-31T20:54:45.769 INFO:teuthology.orchestra.run.smithi099.stdout: "user_data": "", 2023-05-31T20:54:45.769 INFO:teuthology.orchestra.run.smithi099.stdout: "appendable": false 2023-05-31T20:54:45.769 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:54:45.769 INFO:teuthology.orchestra.run.smithi099.stdout: "tag": "_SkUDvG2-E5J0vuYKzDap32jpry6p10l", 2023-05-31T20:54:45.769 INFO:teuthology.orchestra.run.smithi099.stdout: "flags": 0, 2023-05-31T20:54:45.769 INFO:teuthology.orchestra.run.smithi099.stdout: "pending_map": [], 2023-05-31T20:54:45.769 INFO:teuthology.orchestra.run.smithi099.stdout: "versioned_epoch": 0 2023-05-31T20:54:45.770 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:54:45.770 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T20:54:45.770 INFO:teuthology.orchestra.run.smithi099.stdout: "name": "obj37", 2023-05-31T20:54:45.770 INFO:teuthology.orchestra.run.smithi099.stdout: "instance": "", 2023-05-31T20:54:45.770 INFO:teuthology.orchestra.run.smithi099.stdout: "ver": { 2023-05-31T20:54:45.770 INFO:teuthology.orchestra.run.smithi099.stdout: "pool": 13, 2023-05-31T20:54:45.770 INFO:teuthology.orchestra.run.smithi099.stdout: "epoch": 4 2023-05-31T20:54:45.771 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:54:45.771 INFO:teuthology.orchestra.run.smithi099.stdout: "locator": "", 2023-05-31T20:54:45.771 INFO:teuthology.orchestra.run.smithi099.stdout: "exists": true, 2023-05-31T20:54:45.771 INFO:teuthology.orchestra.run.smithi099.stdout: "meta": { 2023-05-31T20:54:45.771 INFO:teuthology.orchestra.run.smithi099.stdout: "category": 1, 2023-05-31T20:54:45.771 INFO:teuthology.orchestra.run.smithi099.stdout: "size": 16, 2023-05-31T20:54:45.771 INFO:teuthology.orchestra.run.smithi099.stdout: "mtime": "2023-05-31T20:52:14.909599Z", 2023-05-31T20:54:45.772 INFO:teuthology.orchestra.run.smithi099.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-05-31T20:54:45.772 INFO:teuthology.orchestra.run.smithi099.stdout: "storage_class": "", 2023-05-31T20:54:45.772 INFO:teuthology.orchestra.run.smithi099.stdout: "owner": "rgw-multisite-test-user", 2023-05-31T20:54:45.772 INFO:teuthology.orchestra.run.smithi099.stdout: "owner_display_name": "Multisite Test User", 2023-05-31T20:54:45.772 INFO:teuthology.orchestra.run.smithi099.stdout: "content_type": "application/octet-stream", 2023-05-31T20:54:45.772 INFO:teuthology.orchestra.run.smithi099.stdout: "accounted_size": 3, 2023-05-31T20:54:45.773 INFO:teuthology.orchestra.run.smithi099.stdout: "user_data": "", 2023-05-31T20:54:45.773 INFO:teuthology.orchestra.run.smithi099.stdout: "appendable": false 2023-05-31T20:54:45.773 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:54:45.773 INFO:teuthology.orchestra.run.smithi099.stdout: "tag": "_L-1eIfBqqraXaRpdoMl2Qc9JgodC42g", 2023-05-31T20:54:45.773 INFO:teuthology.orchestra.run.smithi099.stdout: "flags": 0, 2023-05-31T20:54:45.773 INFO:teuthology.orchestra.run.smithi099.stdout: "pending_map": [], 2023-05-31T20:54:45.773 INFO:teuthology.orchestra.run.smithi099.stdout: "versioned_epoch": 0 2023-05-31T20:54:45.774 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:54:45.774 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T20:54:45.774 INFO:teuthology.orchestra.run.smithi099.stdout: "name": "obj38", 2023-05-31T20:54:45.774 INFO:teuthology.orchestra.run.smithi099.stdout: "instance": "", 2023-05-31T20:54:45.774 INFO:teuthology.orchestra.run.smithi099.stdout: "ver": { 2023-05-31T20:54:45.774 INFO:teuthology.orchestra.run.smithi099.stdout: "pool": 13, 2023-05-31T20:54:45.774 INFO:teuthology.orchestra.run.smithi099.stdout: "epoch": 6 2023-05-31T20:54:45.775 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:54:45.775 INFO:teuthology.orchestra.run.smithi099.stdout: "locator": "", 2023-05-31T20:54:45.775 INFO:teuthology.orchestra.run.smithi099.stdout: "exists": true, 2023-05-31T20:54:45.775 INFO:teuthology.orchestra.run.smithi099.stdout: "meta": { 2023-05-31T20:54:45.775 INFO:teuthology.orchestra.run.smithi099.stdout: "category": 1, 2023-05-31T20:54:45.775 INFO:teuthology.orchestra.run.smithi099.stdout: "size": 3, 2023-05-31T20:54:45.775 INFO:teuthology.orchestra.run.smithi099.stdout: "mtime": "2023-05-31T20:44:50.610918Z", 2023-05-31T20:54:45.776 INFO:teuthology.orchestra.run.smithi099.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-05-31T20:54:45.776 INFO:teuthology.orchestra.run.smithi099.stdout: "storage_class": "", 2023-05-31T20:54:45.776 INFO:teuthology.orchestra.run.smithi099.stdout: "owner": "rgw-multisite-test-user", 2023-05-31T20:54:45.776 INFO:teuthology.orchestra.run.smithi099.stdout: "owner_display_name": "Multisite Test User", 2023-05-31T20:54:45.776 INFO:teuthology.orchestra.run.smithi099.stdout: "content_type": "application/octet-stream", 2023-05-31T20:54:45.776 INFO:teuthology.orchestra.run.smithi099.stdout: "accounted_size": 3, 2023-05-31T20:54:45.776 INFO:teuthology.orchestra.run.smithi099.stdout: "user_data": "", 2023-05-31T20:54:45.777 INFO:teuthology.orchestra.run.smithi099.stdout: "appendable": false 2023-05-31T20:54:45.777 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:54:45.777 INFO:teuthology.orchestra.run.smithi099.stdout: "tag": "_t3E7dFQGjJ6cZ96jvfFxqZWfts6NA0u", 2023-05-31T20:54:45.777 INFO:teuthology.orchestra.run.smithi099.stdout: "flags": 0, 2023-05-31T20:54:45.777 INFO:teuthology.orchestra.run.smithi099.stdout: "pending_map": [], 2023-05-31T20:54:45.777 INFO:teuthology.orchestra.run.smithi099.stdout: "versioned_epoch": 0 2023-05-31T20:54:45.777 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:54:45.778 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T20:54:45.778 INFO:teuthology.orchestra.run.smithi099.stdout: "name": "obj44", 2023-05-31T20:54:45.778 INFO:teuthology.orchestra.run.smithi099.stdout: "instance": "", 2023-05-31T20:54:45.778 INFO:teuthology.orchestra.run.smithi099.stdout: "ver": { 2023-05-31T20:54:45.778 INFO:teuthology.orchestra.run.smithi099.stdout: "pool": 13, 2023-05-31T20:54:45.778 INFO:teuthology.orchestra.run.smithi099.stdout: "epoch": 8 2023-05-31T20:54:45.778 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:54:45.779 INFO:teuthology.orchestra.run.smithi099.stdout: "locator": "", 2023-05-31T20:54:45.779 INFO:teuthology.orchestra.run.smithi099.stdout: "exists": true, 2023-05-31T20:54:45.779 INFO:teuthology.orchestra.run.smithi099.stdout: "meta": { 2023-05-31T20:54:45.779 INFO:teuthology.orchestra.run.smithi099.stdout: "category": 1, 2023-05-31T20:54:45.779 INFO:teuthology.orchestra.run.smithi099.stdout: "size": 5, 2023-05-31T20:54:45.779 INFO:teuthology.orchestra.run.smithi099.stdout: "mtime": "2023-05-31T20:39:26.565648Z", 2023-05-31T20:54:45.779 INFO:teuthology.orchestra.run.smithi099.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-05-31T20:54:45.780 INFO:teuthology.orchestra.run.smithi099.stdout: "storage_class": "", 2023-05-31T20:54:45.780 INFO:teuthology.orchestra.run.smithi099.stdout: "owner": "rgw-multisite-test-user", 2023-05-31T20:54:45.780 INFO:teuthology.orchestra.run.smithi099.stdout: "owner_display_name": "Multisite Test User", 2023-05-31T20:54:45.780 INFO:teuthology.orchestra.run.smithi099.stdout: "content_type": "application/octet-stream", 2023-05-31T20:54:45.780 INFO:teuthology.orchestra.run.smithi099.stdout: "accounted_size": 3, 2023-05-31T20:54:45.780 INFO:teuthology.orchestra.run.smithi099.stdout: "user_data": "", 2023-05-31T20:54:45.781 INFO:teuthology.orchestra.run.smithi099.stdout: "appendable": false 2023-05-31T20:54:45.781 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:54:45.781 INFO:teuthology.orchestra.run.smithi099.stdout: "tag": "_pNKzFz-vS26xJ-v5B2jCiwX2F5RomAs", 2023-05-31T20:54:45.781 INFO:teuthology.orchestra.run.smithi099.stdout: "flags": 0, 2023-05-31T20:54:45.781 INFO:teuthology.orchestra.run.smithi099.stdout: "pending_map": [], 2023-05-31T20:54:45.781 INFO:teuthology.orchestra.run.smithi099.stdout: "versioned_epoch": 0 2023-05-31T20:54:45.781 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:54:45.782 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T20:54:45.782 INFO:teuthology.orchestra.run.smithi099.stdout: "name": "obj46", 2023-05-31T20:54:45.782 INFO:teuthology.orchestra.run.smithi099.stdout: "instance": "", 2023-05-31T20:54:45.782 INFO:teuthology.orchestra.run.smithi099.stdout: "ver": { 2023-05-31T20:54:45.782 INFO:teuthology.orchestra.run.smithi099.stdout: "pool": 13, 2023-05-31T20:54:45.782 INFO:teuthology.orchestra.run.smithi099.stdout: "epoch": 4 2023-05-31T20:54:45.782 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:54:45.783 INFO:teuthology.orchestra.run.smithi099.stdout: "locator": "", 2023-05-31T20:54:45.783 INFO:teuthology.orchestra.run.smithi099.stdout: "exists": true, 2023-05-31T20:54:45.783 INFO:teuthology.orchestra.run.smithi099.stdout: "meta": { 2023-05-31T20:54:45.783 INFO:teuthology.orchestra.run.smithi099.stdout: "category": 1, 2023-05-31T20:54:45.783 INFO:teuthology.orchestra.run.smithi099.stdout: "size": 6, 2023-05-31T20:54:45.783 INFO:teuthology.orchestra.run.smithi099.stdout: "mtime": "2023-05-31T20:46:33.830640Z", 2023-05-31T20:54:45.783 INFO:teuthology.orchestra.run.smithi099.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-05-31T20:54:45.784 INFO:teuthology.orchestra.run.smithi099.stdout: "storage_class": "", 2023-05-31T20:54:45.784 INFO:teuthology.orchestra.run.smithi099.stdout: "owner": "rgw-multisite-test-user", 2023-05-31T20:54:45.784 INFO:teuthology.orchestra.run.smithi099.stdout: "owner_display_name": "Multisite Test User", 2023-05-31T20:54:45.784 INFO:teuthology.orchestra.run.smithi099.stdout: "content_type": "application/octet-stream", 2023-05-31T20:54:45.784 INFO:teuthology.orchestra.run.smithi099.stdout: "accounted_size": 3, 2023-05-31T20:54:45.784 INFO:teuthology.orchestra.run.smithi099.stdout: "user_data": "", 2023-05-31T20:54:45.784 INFO:teuthology.orchestra.run.smithi099.stdout: "appendable": false 2023-05-31T20:54:45.785 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:54:45.785 INFO:teuthology.orchestra.run.smithi099.stdout: "tag": "_Idisp9Pd-hAY4fgJqP4_nPBeI9ZKsrj", 2023-05-31T20:54:45.785 INFO:teuthology.orchestra.run.smithi099.stdout: "flags": 0, 2023-05-31T20:54:45.785 INFO:teuthology.orchestra.run.smithi099.stdout: "pending_map": [], 2023-05-31T20:54:45.785 INFO:teuthology.orchestra.run.smithi099.stdout: "versioned_epoch": 0 2023-05-31T20:54:45.785 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:54:45.785 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T20:54:45.786 INFO:teuthology.orchestra.run.smithi099.stdout: "name": "obj51", 2023-05-31T20:54:45.786 INFO:teuthology.orchestra.run.smithi099.stdout: "instance": "", 2023-05-31T20:54:45.786 INFO:teuthology.orchestra.run.smithi099.stdout: "ver": { 2023-05-31T20:54:45.786 INFO:teuthology.orchestra.run.smithi099.stdout: "pool": 13, 2023-05-31T20:54:45.786 INFO:teuthology.orchestra.run.smithi099.stdout: "epoch": 4 2023-05-31T20:54:45.786 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:54:45.786 INFO:teuthology.orchestra.run.smithi099.stdout: "locator": "", 2023-05-31T20:54:45.787 INFO:teuthology.orchestra.run.smithi099.stdout: "exists": true, 2023-05-31T20:54:45.787 INFO:teuthology.orchestra.run.smithi099.stdout: "meta": { 2023-05-31T20:54:45.787 INFO:teuthology.orchestra.run.smithi099.stdout: "category": 1, 2023-05-31T20:54:45.787 INFO:teuthology.orchestra.run.smithi099.stdout: "size": 16, 2023-05-31T20:54:45.787 INFO:teuthology.orchestra.run.smithi099.stdout: "mtime": "2023-05-31T20:43:08.011797Z", 2023-05-31T20:54:45.787 INFO:teuthology.orchestra.run.smithi099.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-05-31T20:54:45.787 INFO:teuthology.orchestra.run.smithi099.stdout: "storage_class": "", 2023-05-31T20:54:45.788 INFO:teuthology.orchestra.run.smithi099.stdout: "owner": "rgw-multisite-test-user", 2023-05-31T20:54:45.788 INFO:teuthology.orchestra.run.smithi099.stdout: "owner_display_name": "Multisite Test User", 2023-05-31T20:54:45.788 INFO:teuthology.orchestra.run.smithi099.stdout: "content_type": "application/octet-stream", 2023-05-31T20:54:45.788 INFO:teuthology.orchestra.run.smithi099.stdout: "accounted_size": 3, 2023-05-31T20:54:45.788 INFO:teuthology.orchestra.run.smithi099.stdout: "user_data": "", 2023-05-31T20:54:45.788 INFO:teuthology.orchestra.run.smithi099.stdout: "appendable": false 2023-05-31T20:54:45.789 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:54:45.789 INFO:teuthology.orchestra.run.smithi099.stdout: "tag": "_RVz5ucLmyU7bQ_Zq2mZfjyEvmP-EHl2", 2023-05-31T20:54:45.789 INFO:teuthology.orchestra.run.smithi099.stdout: "flags": 0, 2023-05-31T20:54:45.789 INFO:teuthology.orchestra.run.smithi099.stdout: "pending_map": [], 2023-05-31T20:54:45.789 INFO:teuthology.orchestra.run.smithi099.stdout: "versioned_epoch": 0 2023-05-31T20:54:45.789 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:54:45.789 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T20:54:45.790 INFO:teuthology.orchestra.run.smithi099.stdout: "name": "obj55", 2023-05-31T20:54:45.790 INFO:teuthology.orchestra.run.smithi099.stdout: "instance": "", 2023-05-31T20:54:45.790 INFO:teuthology.orchestra.run.smithi099.stdout: "ver": { 2023-05-31T20:54:45.790 INFO:teuthology.orchestra.run.smithi099.stdout: "pool": 13, 2023-05-31T20:54:45.790 INFO:teuthology.orchestra.run.smithi099.stdout: "epoch": 5 2023-05-31T20:54:45.790 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:54:45.790 INFO:teuthology.orchestra.run.smithi099.stdout: "locator": "", 2023-05-31T20:54:45.791 INFO:teuthology.orchestra.run.smithi099.stdout: "exists": true, 2023-05-31T20:54:45.791 INFO:teuthology.orchestra.run.smithi099.stdout: "meta": { 2023-05-31T20:54:45.791 INFO:teuthology.orchestra.run.smithi099.stdout: "category": 1, 2023-05-31T20:54:45.791 INFO:teuthology.orchestra.run.smithi099.stdout: "size": 16, 2023-05-31T20:54:45.791 INFO:teuthology.orchestra.run.smithi099.stdout: "mtime": "2023-05-31T20:39:26.630472Z", 2023-05-31T20:54:45.791 INFO:teuthology.orchestra.run.smithi099.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-05-31T20:54:45.791 INFO:teuthology.orchestra.run.smithi099.stdout: "storage_class": "", 2023-05-31T20:54:45.792 INFO:teuthology.orchestra.run.smithi099.stdout: "owner": "rgw-multisite-test-user", 2023-05-31T20:54:45.792 INFO:teuthology.orchestra.run.smithi099.stdout: "owner_display_name": "Multisite Test User", 2023-05-31T20:54:45.792 INFO:teuthology.orchestra.run.smithi099.stdout: "content_type": "application/octet-stream", 2023-05-31T20:54:45.792 INFO:teuthology.orchestra.run.smithi099.stdout: "accounted_size": 3, 2023-05-31T20:54:45.792 INFO:teuthology.orchestra.run.smithi099.stdout: "user_data": "", 2023-05-31T20:54:45.792 INFO:teuthology.orchestra.run.smithi099.stdout: "appendable": false 2023-05-31T20:54:45.793 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:54:45.793 INFO:teuthology.orchestra.run.smithi099.stdout: "tag": "_rgQJTWhEYvJ-GRM7DbeZFWBRK_UUn5D", 2023-05-31T20:54:45.793 INFO:teuthology.orchestra.run.smithi099.stdout: "flags": 0, 2023-05-31T20:54:45.793 INFO:teuthology.orchestra.run.smithi099.stdout: "pending_map": [], 2023-05-31T20:54:45.793 INFO:teuthology.orchestra.run.smithi099.stdout: "versioned_epoch": 0 2023-05-31T20:54:45.793 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:54:45.793 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T20:54:45.794 INFO:teuthology.orchestra.run.smithi099.stdout: "name": "obj57", 2023-05-31T20:54:45.794 INFO:teuthology.orchestra.run.smithi099.stdout: "instance": "", 2023-05-31T20:54:45.794 INFO:teuthology.orchestra.run.smithi099.stdout: "ver": { 2023-05-31T20:54:45.794 INFO:teuthology.orchestra.run.smithi099.stdout: "pool": 13, 2023-05-31T20:54:45.794 INFO:teuthology.orchestra.run.smithi099.stdout: "epoch": 3 2023-05-31T20:54:45.794 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:54:45.794 INFO:teuthology.orchestra.run.smithi099.stdout: "locator": "", 2023-05-31T20:54:45.795 INFO:teuthology.orchestra.run.smithi099.stdout: "exists": true, 2023-05-31T20:54:45.795 INFO:teuthology.orchestra.run.smithi099.stdout: "meta": { 2023-05-31T20:54:45.795 INFO:teuthology.orchestra.run.smithi099.stdout: "category": 1, 2023-05-31T20:54:45.795 INFO:teuthology.orchestra.run.smithi099.stdout: "size": 5, 2023-05-31T20:54:45.795 INFO:teuthology.orchestra.run.smithi099.stdout: "mtime": "2023-05-31T20:44:50.627910Z", 2023-05-31T20:54:45.795 INFO:teuthology.orchestra.run.smithi099.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-05-31T20:54:45.795 INFO:teuthology.orchestra.run.smithi099.stdout: "storage_class": "", 2023-05-31T20:54:45.796 INFO:teuthology.orchestra.run.smithi099.stdout: "owner": "rgw-multisite-test-user", 2023-05-31T20:54:45.796 INFO:teuthology.orchestra.run.smithi099.stdout: "owner_display_name": "Multisite Test User", 2023-05-31T20:54:45.796 INFO:teuthology.orchestra.run.smithi099.stdout: "content_type": "application/octet-stream", 2023-05-31T20:54:45.796 INFO:teuthology.orchestra.run.smithi099.stdout: "accounted_size": 3, 2023-05-31T20:54:45.796 INFO:teuthology.orchestra.run.smithi099.stdout: "user_data": "", 2023-05-31T20:54:45.796 INFO:teuthology.orchestra.run.smithi099.stdout: "appendable": false 2023-05-31T20:54:45.797 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:54:45.797 INFO:teuthology.orchestra.run.smithi099.stdout: "tag": "_PNx-zYU15OsX05T4oGhUtjnlcgHp2yT", 2023-05-31T20:54:45.797 INFO:teuthology.orchestra.run.smithi099.stdout: "flags": 0, 2023-05-31T20:54:45.797 INFO:teuthology.orchestra.run.smithi099.stdout: "pending_map": [], 2023-05-31T20:54:45.797 INFO:teuthology.orchestra.run.smithi099.stdout: "versioned_epoch": 0 2023-05-31T20:54:45.797 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:54:45.797 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T20:54:45.798 INFO:teuthology.orchestra.run.smithi099.stdout: "name": "obj58", 2023-05-31T20:54:45.798 INFO:teuthology.orchestra.run.smithi099.stdout: "instance": "", 2023-05-31T20:54:45.798 INFO:teuthology.orchestra.run.smithi099.stdout: "ver": { 2023-05-31T20:54:45.798 INFO:teuthology.orchestra.run.smithi099.stdout: "pool": 13, 2023-05-31T20:54:45.798 INFO:teuthology.orchestra.run.smithi099.stdout: "epoch": 6 2023-05-31T20:54:45.798 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:54:45.798 INFO:teuthology.orchestra.run.smithi099.stdout: "locator": "", 2023-05-31T20:54:45.799 INFO:teuthology.orchestra.run.smithi099.stdout: "exists": true, 2023-05-31T20:54:45.799 INFO:teuthology.orchestra.run.smithi099.stdout: "meta": { 2023-05-31T20:54:45.799 INFO:teuthology.orchestra.run.smithi099.stdout: "category": 1, 2023-05-31T20:54:45.800 INFO:teuthology.orchestra.run.smithi099.stdout: "size": 3, 2023-05-31T20:54:45.800 INFO:teuthology.orchestra.run.smithi099.stdout: "mtime": "2023-05-31T20:48:12.976191Z", 2023-05-31T20:54:45.800 INFO:teuthology.orchestra.run.smithi099.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-05-31T20:54:45.800 INFO:teuthology.orchestra.run.smithi099.stdout: "storage_class": "", 2023-05-31T20:54:45.801 INFO:teuthology.orchestra.run.smithi099.stdout: "owner": "rgw-multisite-test-user", 2023-05-31T20:54:45.801 INFO:teuthology.orchestra.run.smithi099.stdout: "owner_display_name": "Multisite Test User", 2023-05-31T20:54:45.801 INFO:teuthology.orchestra.run.smithi099.stdout: "content_type": "application/octet-stream", 2023-05-31T20:54:45.801 INFO:teuthology.orchestra.run.smithi099.stdout: "accounted_size": 3, 2023-05-31T20:54:45.801 INFO:teuthology.orchestra.run.smithi099.stdout: "user_data": "", 2023-05-31T20:54:45.801 INFO:teuthology.orchestra.run.smithi099.stdout: "appendable": false 2023-05-31T20:54:45.802 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:54:45.802 INFO:teuthology.orchestra.run.smithi099.stdout: "tag": "_kOwPkuG-Z7gq6CKgJy_mbWYrU4xse2w", 2023-05-31T20:54:45.802 INFO:teuthology.orchestra.run.smithi099.stdout: "flags": 0, 2023-05-31T20:54:45.802 INFO:teuthology.orchestra.run.smithi099.stdout: "pending_map": [], 2023-05-31T20:54:45.802 INFO:teuthology.orchestra.run.smithi099.stdout: "versioned_epoch": 0 2023-05-31T20:54:45.802 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:54:45.802 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T20:54:45.803 INFO:teuthology.orchestra.run.smithi099.stdout: "name": "obj62", 2023-05-31T20:54:45.803 INFO:teuthology.orchestra.run.smithi099.stdout: "instance": "", 2023-05-31T20:54:45.803 INFO:teuthology.orchestra.run.smithi099.stdout: "ver": { 2023-05-31T20:54:45.803 INFO:teuthology.orchestra.run.smithi099.stdout: "pool": 13, 2023-05-31T20:54:45.803 INFO:teuthology.orchestra.run.smithi099.stdout: "epoch": 3 2023-05-31T20:54:45.803 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:54:45.803 INFO:teuthology.orchestra.run.smithi099.stdout: "locator": "", 2023-05-31T20:54:45.804 INFO:teuthology.orchestra.run.smithi099.stdout: "exists": true, 2023-05-31T20:54:45.804 INFO:teuthology.orchestra.run.smithi099.stdout: "meta": { 2023-05-31T20:54:45.804 INFO:teuthology.orchestra.run.smithi099.stdout: "category": 1, 2023-05-31T20:54:45.804 INFO:teuthology.orchestra.run.smithi099.stdout: "size": 5, 2023-05-31T20:54:45.804 INFO:teuthology.orchestra.run.smithi099.stdout: "mtime": "2023-05-31T20:50:33.515966Z", 2023-05-31T20:54:45.804 INFO:teuthology.orchestra.run.smithi099.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-05-31T20:54:45.805 INFO:teuthology.orchestra.run.smithi099.stdout: "storage_class": "", 2023-05-31T20:54:45.805 INFO:teuthology.orchestra.run.smithi099.stdout: "owner": "rgw-multisite-test-user", 2023-05-31T20:54:45.805 INFO:teuthology.orchestra.run.smithi099.stdout: "owner_display_name": "Multisite Test User", 2023-05-31T20:54:45.805 INFO:teuthology.orchestra.run.smithi099.stdout: "content_type": "application/octet-stream", 2023-05-31T20:54:45.806 INFO:teuthology.orchestra.run.smithi099.stdout: "accounted_size": 3, 2023-05-31T20:54:45.806 INFO:teuthology.orchestra.run.smithi099.stdout: "user_data": "", 2023-05-31T20:54:45.806 INFO:teuthology.orchestra.run.smithi099.stdout: "appendable": false 2023-05-31T20:54:45.806 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:54:45.806 INFO:teuthology.orchestra.run.smithi099.stdout: "tag": "_7Vya0uwCseD81DHfz6OWtF2KaV8RRM-", 2023-05-31T20:54:45.806 INFO:teuthology.orchestra.run.smithi099.stdout: "flags": 0, 2023-05-31T20:54:45.807 INFO:teuthology.orchestra.run.smithi099.stdout: "pending_map": [], 2023-05-31T20:54:45.807 INFO:teuthology.orchestra.run.smithi099.stdout: "versioned_epoch": 0 2023-05-31T20:54:45.807 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:54:45.808 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T20:54:45.808 INFO:teuthology.orchestra.run.smithi099.stdout: "name": "obj68", 2023-05-31T20:54:45.808 INFO:teuthology.orchestra.run.smithi099.stdout: "instance": "", 2023-05-31T20:54:45.809 INFO:teuthology.orchestra.run.smithi099.stdout: "ver": { 2023-05-31T20:54:45.809 INFO:teuthology.orchestra.run.smithi099.stdout: "pool": 13, 2023-05-31T20:54:45.809 INFO:teuthology.orchestra.run.smithi099.stdout: "epoch": 5 2023-05-31T20:54:45.809 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:54:45.810 INFO:teuthology.orchestra.run.smithi099.stdout: "locator": "", 2023-05-31T20:54:45.810 INFO:teuthology.orchestra.run.smithi099.stdout: "exists": true, 2023-05-31T20:54:45.810 INFO:teuthology.orchestra.run.smithi099.stdout: "meta": { 2023-05-31T20:54:45.810 INFO:teuthology.orchestra.run.smithi099.stdout: "category": 1, 2023-05-31T20:54:45.810 INFO:teuthology.orchestra.run.smithi099.stdout: "size": 6, 2023-05-31T20:54:45.811 INFO:teuthology.orchestra.run.smithi099.stdout: "mtime": "2023-05-31T20:43:08.066803Z", 2023-05-31T20:54:45.811 INFO:teuthology.orchestra.run.smithi099.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-05-31T20:54:45.811 INFO:teuthology.orchestra.run.smithi099.stdout: "storage_class": "", 2023-05-31T20:54:45.811 INFO:teuthology.orchestra.run.smithi099.stdout: "owner": "rgw-multisite-test-user", 2023-05-31T20:54:45.812 INFO:teuthology.orchestra.run.smithi099.stdout: "owner_display_name": "Multisite Test User", 2023-05-31T20:54:45.812 INFO:teuthology.orchestra.run.smithi099.stdout: "content_type": "application/octet-stream", 2023-05-31T20:54:45.812 INFO:teuthology.orchestra.run.smithi099.stdout: "accounted_size": 3, 2023-05-31T20:54:45.812 INFO:teuthology.orchestra.run.smithi099.stdout: "user_data": "", 2023-05-31T20:54:45.812 INFO:teuthology.orchestra.run.smithi099.stdout: "appendable": false 2023-05-31T20:54:45.812 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:54:45.813 INFO:teuthology.orchestra.run.smithi099.stdout: "tag": "_BUg4FgZG_VtkDhQJMknYp715ueQVIaN", 2023-05-31T20:54:45.813 INFO:teuthology.orchestra.run.smithi099.stdout: "flags": 0, 2023-05-31T20:54:45.813 INFO:teuthology.orchestra.run.smithi099.stdout: "pending_map": [], 2023-05-31T20:54:45.813 INFO:teuthology.orchestra.run.smithi099.stdout: "versioned_epoch": 0 2023-05-31T20:54:45.813 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:54:45.813 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T20:54:45.813 INFO:teuthology.orchestra.run.smithi099.stdout: "name": "obj69", 2023-05-31T20:54:45.814 INFO:teuthology.orchestra.run.smithi099.stdout: "instance": "", 2023-05-31T20:54:45.814 INFO:teuthology.orchestra.run.smithi099.stdout: "ver": { 2023-05-31T20:54:45.814 INFO:teuthology.orchestra.run.smithi099.stdout: "pool": 13, 2023-05-31T20:54:45.814 INFO:teuthology.orchestra.run.smithi099.stdout: "epoch": 1 2023-05-31T20:54:45.814 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:54:45.814 INFO:teuthology.orchestra.run.smithi099.stdout: "locator": "", 2023-05-31T20:54:45.814 INFO:teuthology.orchestra.run.smithi099.stdout: "exists": true, 2023-05-31T20:54:45.815 INFO:teuthology.orchestra.run.smithi099.stdout: "meta": { 2023-05-31T20:54:45.815 INFO:teuthology.orchestra.run.smithi099.stdout: "category": 1, 2023-05-31T20:54:45.815 INFO:teuthology.orchestra.run.smithi099.stdout: "size": 16, 2023-05-31T20:54:45.815 INFO:teuthology.orchestra.run.smithi099.stdout: "mtime": "2023-05-31T20:46:33.886933Z", 2023-05-31T20:54:45.815 INFO:teuthology.orchestra.run.smithi099.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-05-31T20:54:45.815 INFO:teuthology.orchestra.run.smithi099.stdout: "storage_class": "", 2023-05-31T20:54:45.815 INFO:teuthology.orchestra.run.smithi099.stdout: "owner": "rgw-multisite-test-user", 2023-05-31T20:54:45.816 INFO:teuthology.orchestra.run.smithi099.stdout: "owner_display_name": "Multisite Test User", 2023-05-31T20:54:45.816 INFO:teuthology.orchestra.run.smithi099.stdout: "content_type": "application/octet-stream", 2023-05-31T20:54:45.816 INFO:teuthology.orchestra.run.smithi099.stdout: "accounted_size": 3, 2023-05-31T20:54:45.816 INFO:teuthology.orchestra.run.smithi099.stdout: "user_data": "", 2023-05-31T20:54:45.816 INFO:teuthology.orchestra.run.smithi099.stdout: "appendable": false 2023-05-31T20:54:45.817 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:54:45.817 INFO:teuthology.orchestra.run.smithi099.stdout: "tag": "_QAQGSmUDZ0tgezNnP49sWN_1GSWITO4", 2023-05-31T20:54:45.817 INFO:teuthology.orchestra.run.smithi099.stdout: "flags": 0, 2023-05-31T20:54:45.818 INFO:teuthology.orchestra.run.smithi099.stdout: "pending_map": [], 2023-05-31T20:54:45.818 INFO:teuthology.orchestra.run.smithi099.stdout: "versioned_epoch": 0 2023-05-31T20:54:45.818 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:54:45.818 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T20:54:45.819 INFO:teuthology.orchestra.run.smithi099.stdout: "name": "obj74", 2023-05-31T20:54:45.819 INFO:teuthology.orchestra.run.smithi099.stdout: "instance": "", 2023-05-31T20:54:45.819 INFO:teuthology.orchestra.run.smithi099.stdout: "ver": { 2023-05-31T20:54:45.819 INFO:teuthology.orchestra.run.smithi099.stdout: "pool": 13, 2023-05-31T20:54:45.819 INFO:teuthology.orchestra.run.smithi099.stdout: "epoch": 6 2023-05-31T20:54:45.820 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:54:45.820 INFO:teuthology.orchestra.run.smithi099.stdout: "locator": "", 2023-05-31T20:54:45.820 INFO:teuthology.orchestra.run.smithi099.stdout: "exists": true, 2023-05-31T20:54:45.820 INFO:teuthology.orchestra.run.smithi099.stdout: "meta": { 2023-05-31T20:54:45.821 INFO:teuthology.orchestra.run.smithi099.stdout: "category": 1, 2023-05-31T20:54:45.821 INFO:teuthology.orchestra.run.smithi099.stdout: "size": 16, 2023-05-31T20:54:45.821 INFO:teuthology.orchestra.run.smithi099.stdout: "mtime": "2023-05-31T20:52:14.934725Z", 2023-05-31T20:54:45.821 INFO:teuthology.orchestra.run.smithi099.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-05-31T20:54:45.821 INFO:teuthology.orchestra.run.smithi099.stdout: "storage_class": "", 2023-05-31T20:54:45.821 INFO:teuthology.orchestra.run.smithi099.stdout: "owner": "rgw-multisite-test-user", 2023-05-31T20:54:45.821 INFO:teuthology.orchestra.run.smithi099.stdout: "owner_display_name": "Multisite Test User", 2023-05-31T20:54:45.822 INFO:teuthology.orchestra.run.smithi099.stdout: "content_type": "application/octet-stream", 2023-05-31T20:54:45.822 INFO:teuthology.orchestra.run.smithi099.stdout: "accounted_size": 3, 2023-05-31T20:54:45.822 INFO:teuthology.orchestra.run.smithi099.stdout: "user_data": "", 2023-05-31T20:54:45.822 INFO:teuthology.orchestra.run.smithi099.stdout: "appendable": false 2023-05-31T20:54:45.822 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:54:45.822 INFO:teuthology.orchestra.run.smithi099.stdout: "tag": "_AGLMHIWmbt26ckS-3uG_80injVSZg6l", 2023-05-31T20:54:45.822 INFO:teuthology.orchestra.run.smithi099.stdout: "flags": 0, 2023-05-31T20:54:45.823 INFO:teuthology.orchestra.run.smithi099.stdout: "pending_map": [], 2023-05-31T20:54:45.823 INFO:teuthology.orchestra.run.smithi099.stdout: "versioned_epoch": 0 2023-05-31T20:54:45.823 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:54:45.823 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T20:54:45.823 INFO:teuthology.orchestra.run.smithi099.stdout: "name": "obj76", 2023-05-31T20:54:45.823 INFO:teuthology.orchestra.run.smithi099.stdout: "instance": "", 2023-05-31T20:54:45.823 INFO:teuthology.orchestra.run.smithi099.stdout: "ver": { 2023-05-31T20:54:45.824 INFO:teuthology.orchestra.run.smithi099.stdout: "pool": 13, 2023-05-31T20:54:45.824 INFO:teuthology.orchestra.run.smithi099.stdout: "epoch": 5 2023-05-31T20:54:45.824 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:54:45.824 INFO:teuthology.orchestra.run.smithi099.stdout: "locator": "", 2023-05-31T20:54:45.824 INFO:teuthology.orchestra.run.smithi099.stdout: "exists": true, 2023-05-31T20:54:45.824 INFO:teuthology.orchestra.run.smithi099.stdout: "meta": { 2023-05-31T20:54:45.824 INFO:teuthology.orchestra.run.smithi099.stdout: "category": 1, 2023-05-31T20:54:45.825 INFO:teuthology.orchestra.run.smithi099.stdout: "size": 16, 2023-05-31T20:54:45.825 INFO:teuthology.orchestra.run.smithi099.stdout: "mtime": "2023-05-31T20:44:50.682916Z", 2023-05-31T20:54:45.825 INFO:teuthology.orchestra.run.smithi099.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-05-31T20:54:45.825 INFO:teuthology.orchestra.run.smithi099.stdout: "storage_class": "", 2023-05-31T20:54:45.825 INFO:teuthology.orchestra.run.smithi099.stdout: "owner": "rgw-multisite-test-user", 2023-05-31T20:54:45.825 INFO:teuthology.orchestra.run.smithi099.stdout: "owner_display_name": "Multisite Test User", 2023-05-31T20:54:45.825 INFO:teuthology.orchestra.run.smithi099.stdout: "content_type": "application/octet-stream", 2023-05-31T20:54:45.826 INFO:teuthology.orchestra.run.smithi099.stdout: "accounted_size": 3, 2023-05-31T20:54:45.826 INFO:teuthology.orchestra.run.smithi099.stdout: "user_data": "", 2023-05-31T20:54:45.826 INFO:teuthology.orchestra.run.smithi099.stdout: "appendable": false 2023-05-31T20:54:45.826 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:54:45.826 INFO:teuthology.orchestra.run.smithi099.stdout: "tag": "_GE78ydfVh6VfdUHL7mG-4mNZySfJ_g_", 2023-05-31T20:54:45.826 INFO:teuthology.orchestra.run.smithi099.stdout: "flags": 0, 2023-05-31T20:54:45.826 INFO:teuthology.orchestra.run.smithi099.stdout: "pending_map": [], 2023-05-31T20:54:45.827 INFO:teuthology.orchestra.run.smithi099.stdout: "versioned_epoch": 0 2023-05-31T20:54:45.827 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:54:45.827 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T20:54:45.827 INFO:teuthology.orchestra.run.smithi099.stdout: "name": "obj85", 2023-05-31T20:54:45.827 INFO:teuthology.orchestra.run.smithi099.stdout: "instance": "", 2023-05-31T20:54:45.827 INFO:teuthology.orchestra.run.smithi099.stdout: "ver": { 2023-05-31T20:54:45.827 INFO:teuthology.orchestra.run.smithi099.stdout: "pool": 13, 2023-05-31T20:54:45.828 INFO:teuthology.orchestra.run.smithi099.stdout: "epoch": 5 2023-05-31T20:54:45.828 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:54:45.828 INFO:teuthology.orchestra.run.smithi099.stdout: "locator": "", 2023-05-31T20:54:45.828 INFO:teuthology.orchestra.run.smithi099.stdout: "exists": true, 2023-05-31T20:54:45.828 INFO:teuthology.orchestra.run.smithi099.stdout: "meta": { 2023-05-31T20:54:45.828 INFO:teuthology.orchestra.run.smithi099.stdout: "category": 1, 2023-05-31T20:54:45.828 INFO:teuthology.orchestra.run.smithi099.stdout: "size": 16, 2023-05-31T20:54:45.829 INFO:teuthology.orchestra.run.smithi099.stdout: "mtime": "2023-05-31T20:43:08.086089Z", 2023-05-31T20:54:45.829 INFO:teuthology.orchestra.run.smithi099.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-05-31T20:54:45.829 INFO:teuthology.orchestra.run.smithi099.stdout: "storage_class": "", 2023-05-31T20:54:45.829 INFO:teuthology.orchestra.run.smithi099.stdout: "owner": "rgw-multisite-test-user", 2023-05-31T20:54:45.829 INFO:teuthology.orchestra.run.smithi099.stdout: "owner_display_name": "Multisite Test User", 2023-05-31T20:54:45.829 INFO:teuthology.orchestra.run.smithi099.stdout: "content_type": "application/octet-stream", 2023-05-31T20:54:45.830 INFO:teuthology.orchestra.run.smithi099.stdout: "accounted_size": 3, 2023-05-31T20:54:45.830 INFO:teuthology.orchestra.run.smithi099.stdout: "user_data": "", 2023-05-31T20:54:45.830 INFO:teuthology.orchestra.run.smithi099.stdout: "appendable": false 2023-05-31T20:54:45.830 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:54:45.830 INFO:teuthology.orchestra.run.smithi099.stdout: "tag": "_BP8bpBv9wMJ7CMtiFJ5LOhgta7o9leH", 2023-05-31T20:54:45.830 INFO:teuthology.orchestra.run.smithi099.stdout: "flags": 0, 2023-05-31T20:54:45.830 INFO:teuthology.orchestra.run.smithi099.stdout: "pending_map": [], 2023-05-31T20:54:45.831 INFO:teuthology.orchestra.run.smithi099.stdout: "versioned_epoch": 0 2023-05-31T20:54:45.831 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:54:45.831 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T20:54:45.831 INFO:teuthology.orchestra.run.smithi099.stdout: "name": "obj87", 2023-05-31T20:54:45.831 INFO:teuthology.orchestra.run.smithi099.stdout: "instance": "", 2023-05-31T20:54:45.831 INFO:teuthology.orchestra.run.smithi099.stdout: "ver": { 2023-05-31T20:54:45.831 INFO:teuthology.orchestra.run.smithi099.stdout: "pool": 13, 2023-05-31T20:54:45.832 INFO:teuthology.orchestra.run.smithi099.stdout: "epoch": 8 2023-05-31T20:54:45.832 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:54:45.832 INFO:teuthology.orchestra.run.smithi099.stdout: "locator": "", 2023-05-31T20:54:45.832 INFO:teuthology.orchestra.run.smithi099.stdout: "exists": true, 2023-05-31T20:54:45.833 INFO:teuthology.orchestra.run.smithi099.stdout: "meta": { 2023-05-31T20:54:45.833 INFO:teuthology.orchestra.run.smithi099.stdout: "category": 1, 2023-05-31T20:54:45.833 INFO:teuthology.orchestra.run.smithi099.stdout: "size": 5, 2023-05-31T20:54:45.834 INFO:teuthology.orchestra.run.smithi099.stdout: "mtime": "2023-05-31T20:48:13.031366Z", 2023-05-31T20:54:45.834 INFO:teuthology.orchestra.run.smithi099.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-05-31T20:54:45.834 INFO:teuthology.orchestra.run.smithi099.stdout: "storage_class": "", 2023-05-31T20:54:45.834 INFO:teuthology.orchestra.run.smithi099.stdout: "owner": "rgw-multisite-test-user", 2023-05-31T20:54:45.834 INFO:teuthology.orchestra.run.smithi099.stdout: "owner_display_name": "Multisite Test User", 2023-05-31T20:54:45.834 INFO:teuthology.orchestra.run.smithi099.stdout: "content_type": "application/octet-stream", 2023-05-31T20:54:45.834 INFO:teuthology.orchestra.run.smithi099.stdout: "accounted_size": 3, 2023-05-31T20:54:45.835 INFO:teuthology.orchestra.run.smithi099.stdout: "user_data": "", 2023-05-31T20:54:45.835 INFO:teuthology.orchestra.run.smithi099.stdout: "appendable": false 2023-05-31T20:54:45.835 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:54:45.835 INFO:teuthology.orchestra.run.smithi099.stdout: "tag": "_vKIEJ_jC0hWqntLOIpG3LHxu4f-7kfy", 2023-05-31T20:54:45.835 INFO:teuthology.orchestra.run.smithi099.stdout: "flags": 0, 2023-05-31T20:54:45.835 INFO:teuthology.orchestra.run.smithi099.stdout: "pending_map": [], 2023-05-31T20:54:45.835 INFO:teuthology.orchestra.run.smithi099.stdout: "versioned_epoch": 0 2023-05-31T20:54:45.836 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:54:45.836 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T20:54:45.836 INFO:teuthology.orchestra.run.smithi099.stdout: "name": "obj92", 2023-05-31T20:54:45.836 INFO:teuthology.orchestra.run.smithi099.stdout: "instance": "", 2023-05-31T20:54:45.836 INFO:teuthology.orchestra.run.smithi099.stdout: "ver": { 2023-05-31T20:54:45.836 INFO:teuthology.orchestra.run.smithi099.stdout: "pool": 13, 2023-05-31T20:54:45.836 INFO:teuthology.orchestra.run.smithi099.stdout: "epoch": 5 2023-05-31T20:54:45.837 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:54:45.837 INFO:teuthology.orchestra.run.smithi099.stdout: "locator": "", 2023-05-31T20:54:45.837 INFO:teuthology.orchestra.run.smithi099.stdout: "exists": true, 2023-05-31T20:54:45.837 INFO:teuthology.orchestra.run.smithi099.stdout: "meta": { 2023-05-31T20:54:45.837 INFO:teuthology.orchestra.run.smithi099.stdout: "category": 1, 2023-05-31T20:54:45.837 INFO:teuthology.orchestra.run.smithi099.stdout: "size": 16, 2023-05-31T20:54:45.837 INFO:teuthology.orchestra.run.smithi099.stdout: "mtime": "2023-05-31T20:46:33.904998Z", 2023-05-31T20:54:45.838 INFO:teuthology.orchestra.run.smithi099.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-05-31T20:54:45.838 INFO:teuthology.orchestra.run.smithi099.stdout: "storage_class": "", 2023-05-31T20:54:45.838 INFO:teuthology.orchestra.run.smithi099.stdout: "owner": "rgw-multisite-test-user", 2023-05-31T20:54:45.838 INFO:teuthology.orchestra.run.smithi099.stdout: "owner_display_name": "Multisite Test User", 2023-05-31T20:54:45.838 INFO:teuthology.orchestra.run.smithi099.stdout: "content_type": "application/octet-stream", 2023-05-31T20:54:45.838 INFO:teuthology.orchestra.run.smithi099.stdout: "accounted_size": 3, 2023-05-31T20:54:45.839 INFO:teuthology.orchestra.run.smithi099.stdout: "user_data": "", 2023-05-31T20:54:45.839 INFO:teuthology.orchestra.run.smithi099.stdout: "appendable": false 2023-05-31T20:54:45.839 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:54:45.839 INFO:teuthology.orchestra.run.smithi099.stdout: "tag": "_kQL1O_S1_rYY5D8au7IIyZ2KXiJv3TW", 2023-05-31T20:54:45.839 INFO:teuthology.orchestra.run.smithi099.stdout: "flags": 0, 2023-05-31T20:54:45.839 INFO:teuthology.orchestra.run.smithi099.stdout: "pending_map": [], 2023-05-31T20:54:45.840 INFO:teuthology.orchestra.run.smithi099.stdout: "versioned_epoch": 0 2023-05-31T20:54:45.840 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:54:45.840 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T20:54:45.840 INFO:teuthology.orchestra.run.smithi099.stdout: "name": "obj93", 2023-05-31T20:54:45.840 INFO:teuthology.orchestra.run.smithi099.stdout: "instance": "", 2023-05-31T20:54:45.840 INFO:teuthology.orchestra.run.smithi099.stdout: "ver": { 2023-05-31T20:54:45.840 INFO:teuthology.orchestra.run.smithi099.stdout: "pool": 13, 2023-05-31T20:54:45.841 INFO:teuthology.orchestra.run.smithi099.stdout: "epoch": 3 2023-05-31T20:54:45.841 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:54:45.841 INFO:teuthology.orchestra.run.smithi099.stdout: "locator": "", 2023-05-31T20:54:45.841 INFO:teuthology.orchestra.run.smithi099.stdout: "exists": true, 2023-05-31T20:54:45.841 INFO:teuthology.orchestra.run.smithi099.stdout: "meta": { 2023-05-31T20:54:45.841 INFO:teuthology.orchestra.run.smithi099.stdout: "category": 1, 2023-05-31T20:54:45.841 INFO:teuthology.orchestra.run.smithi099.stdout: "size": 16, 2023-05-31T20:54:45.842 INFO:teuthology.orchestra.run.smithi099.stdout: "mtime": "2023-05-31T20:50:33.533538Z", 2023-05-31T20:54:45.842 INFO:teuthology.orchestra.run.smithi099.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-05-31T20:54:45.842 INFO:teuthology.orchestra.run.smithi099.stdout: "storage_class": "", 2023-05-31T20:54:45.842 INFO:teuthology.orchestra.run.smithi099.stdout: "owner": "rgw-multisite-test-user", 2023-05-31T20:54:45.842 INFO:teuthology.orchestra.run.smithi099.stdout: "owner_display_name": "Multisite Test User", 2023-05-31T20:54:45.842 INFO:teuthology.orchestra.run.smithi099.stdout: "content_type": "application/octet-stream", 2023-05-31T20:54:45.842 INFO:teuthology.orchestra.run.smithi099.stdout: "accounted_size": 3, 2023-05-31T20:54:45.843 INFO:teuthology.orchestra.run.smithi099.stdout: "user_data": "", 2023-05-31T20:54:45.843 INFO:teuthology.orchestra.run.smithi099.stdout: "appendable": false 2023-05-31T20:54:45.843 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:54:45.843 INFO:teuthology.orchestra.run.smithi099.stdout: "tag": "_ezgt6uBWOIW1SXbEZyJiTsyN9mHk8t7", 2023-05-31T20:54:45.843 INFO:teuthology.orchestra.run.smithi099.stdout: "flags": 0, 2023-05-31T20:54:45.843 INFO:teuthology.orchestra.run.smithi099.stdout: "pending_map": [], 2023-05-31T20:54:45.843 INFO:teuthology.orchestra.run.smithi099.stdout: "versioned_epoch": 0 2023-05-31T20:54:45.844 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:54:45.844 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T20:54:45.844 INFO:teuthology.orchestra.run.smithi099.stdout: "name": "obj95", 2023-05-31T20:54:45.844 INFO:teuthology.orchestra.run.smithi099.stdout: "instance": "", 2023-05-31T20:54:45.844 INFO:teuthology.orchestra.run.smithi099.stdout: "ver": { 2023-05-31T20:54:45.844 INFO:teuthology.orchestra.run.smithi099.stdout: "pool": 13, 2023-05-31T20:54:45.845 INFO:teuthology.orchestra.run.smithi099.stdout: "epoch": 6 2023-05-31T20:54:45.845 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:54:45.845 INFO:teuthology.orchestra.run.smithi099.stdout: "locator": "", 2023-05-31T20:54:45.845 INFO:teuthology.orchestra.run.smithi099.stdout: "exists": true, 2023-05-31T20:54:45.845 INFO:teuthology.orchestra.run.smithi099.stdout: "meta": { 2023-05-31T20:54:45.845 INFO:teuthology.orchestra.run.smithi099.stdout: "category": 1, 2023-05-31T20:54:45.845 INFO:teuthology.orchestra.run.smithi099.stdout: "size": 16, 2023-05-31T20:54:45.846 INFO:teuthology.orchestra.run.smithi099.stdout: "mtime": "2023-05-31T20:44:50.705817Z", 2023-05-31T20:54:45.846 INFO:teuthology.orchestra.run.smithi099.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-05-31T20:54:45.846 INFO:teuthology.orchestra.run.smithi099.stdout: "storage_class": "", 2023-05-31T20:54:45.846 INFO:teuthology.orchestra.run.smithi099.stdout: "owner": "rgw-multisite-test-user", 2023-05-31T20:54:45.846 INFO:teuthology.orchestra.run.smithi099.stdout: "owner_display_name": "Multisite Test User", 2023-05-31T20:54:45.846 INFO:teuthology.orchestra.run.smithi099.stdout: "content_type": "application/octet-stream", 2023-05-31T20:54:45.846 INFO:teuthology.orchestra.run.smithi099.stdout: "accounted_size": 3, 2023-05-31T20:54:45.847 INFO:teuthology.orchestra.run.smithi099.stdout: "user_data": "", 2023-05-31T20:54:45.847 INFO:teuthology.orchestra.run.smithi099.stdout: "appendable": false 2023-05-31T20:54:45.847 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:54:45.847 INFO:teuthology.orchestra.run.smithi099.stdout: "tag": "_8uh9jSSuXeo2R4JvZSbiQyEbqPQgVsK", 2023-05-31T20:54:45.847 INFO:teuthology.orchestra.run.smithi099.stdout: "flags": 0, 2023-05-31T20:54:45.847 INFO:teuthology.orchestra.run.smithi099.stdout: "pending_map": [], 2023-05-31T20:54:45.847 INFO:teuthology.orchestra.run.smithi099.stdout: "versioned_epoch": 0 2023-05-31T20:54:45.848 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T20:54:45.848 INFO:teuthology.orchestra.run.smithi099.stdout:] 2023-05-31T20:54:45.848 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['object', 'rm', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'pklxkd-59', '--object', 'obj11', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-05-31T20:54:45.849 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'object', 'rm', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'pklxkd-59', '--object', 'obj11', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-05-31T20:54:45.849 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 object rm --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --bucket pklxkd-59 --object obj11 --cluster c1 --debug-rgw 0 --debug-ms 0 2023-05-31T20:54:45.902 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2023-05-31T20:54:45.902 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2023-05-31T20:54:46.031 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['object', 'rm', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'pklxkd-59', '--object', 'obj111', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-05-31T20:54:46.031 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'object', 'rm', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'pklxkd-59', '--object', 'obj111', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-05-31T20:54:46.032 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 object rm --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --bucket pklxkd-59 --object obj111 --cluster c1 --debug-rgw 0 --debug-ms 0 2023-05-31T20:54:46.083 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2023-05-31T20:54:46.084 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2023-05-31T20:54:46.208 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['object', 'rm', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'pklxkd-59', '--object', 'obj115', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-05-31T20:54:46.209 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'object', 'rm', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'pklxkd-59', '--object', 'obj115', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-05-31T20:54:46.209 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 object rm --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --bucket pklxkd-59 --object obj115 --cluster c1 --debug-rgw 0 --debug-ms 0 2023-05-31T20:54:46.259 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2023-05-31T20:54:46.260 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2023-05-31T20:54:46.370 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['object', 'rm', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'pklxkd-59', '--object', 'obj116', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-05-31T20:54:46.371 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'object', 'rm', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'pklxkd-59', '--object', 'obj116', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-05-31T20:54:46.371 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 object rm --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --bucket pklxkd-59 --object obj116 --cluster c1 --debug-rgw 0 --debug-ms 0 2023-05-31T20:54:46.423 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2023-05-31T20:54:46.423 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2023-05-31T20:54:46.542 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['object', 'rm', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'pklxkd-59', '--object', 'obj124', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-05-31T20:54:46.542 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'object', 'rm', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'pklxkd-59', '--object', 'obj124', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-05-31T20:54:46.543 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 object rm --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --bucket pklxkd-59 --object obj124 --cluster c1 --debug-rgw 0 --debug-ms 0 2023-05-31T20:54:46.593 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2023-05-31T20:54:46.594 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2023-05-31T20:54:46.717 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['object', 'rm', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'pklxkd-59', '--object', 'obj145', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-05-31T20:54:46.717 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'object', 'rm', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'pklxkd-59', '--object', 'obj145', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-05-31T20:54:46.718 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 object rm --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --bucket pklxkd-59 --object obj145 --cluster c1 --debug-rgw 0 --debug-ms 0 2023-05-31T20:54:46.770 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2023-05-31T20:54:46.771 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2023-05-31T20:54:46.870 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['object', 'rm', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'pklxkd-59', '--object', 'obj148', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-05-31T20:54:46.871 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'object', 'rm', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'pklxkd-59', '--object', 'obj148', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-05-31T20:54:46.871 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 object rm --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --bucket pklxkd-59 --object obj148 --cluster c1 --debug-rgw 0 --debug-ms 0 2023-05-31T20:54:46.922 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2023-05-31T20:54:46.922 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2023-05-31T20:54:47.042 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['object', 'rm', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'pklxkd-59', '--object', 'obj155', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-05-31T20:54:47.043 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'object', 'rm', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'pklxkd-59', '--object', 'obj155', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-05-31T20:54:47.044 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 object rm --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --bucket pklxkd-59 --object obj155 --cluster c1 --debug-rgw 0 --debug-ms 0 2023-05-31T20:54:47.094 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2023-05-31T20:54:47.094 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2023-05-31T20:54:47.213 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['object', 'rm', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'pklxkd-59', '--object', 'obj17', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-05-31T20:54:47.214 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'object', 'rm', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'pklxkd-59', '--object', 'obj17', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-05-31T20:54:47.214 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 object rm --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --bucket pklxkd-59 --object obj17 --cluster c1 --debug-rgw 0 --debug-ms 0 2023-05-31T20:54:47.266 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2023-05-31T20:54:47.266 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2023-05-31T20:54:47.382 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['object', 'rm', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'pklxkd-59', '--object', 'obj185', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-05-31T20:54:47.383 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'object', 'rm', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'pklxkd-59', '--object', 'obj185', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-05-31T20:54:47.383 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 object rm --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --bucket pklxkd-59 --object obj185 --cluster c1 --debug-rgw 0 --debug-ms 0 2023-05-31T20:54:47.436 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2023-05-31T20:54:47.436 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2023-05-31T20:54:47.571 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['object', 'rm', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'pklxkd-59', '--object', 'obj19', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-05-31T20:54:47.571 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'object', 'rm', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'pklxkd-59', '--object', 'obj19', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-05-31T20:54:47.572 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 object rm --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --bucket pklxkd-59 --object obj19 --cluster c1 --debug-rgw 0 --debug-ms 0 2023-05-31T20:54:47.623 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2023-05-31T20:54:47.623 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2023-05-31T20:54:47.727 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['object', 'rm', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'pklxkd-59', '--object', 'obj22', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-05-31T20:54:47.728 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'object', 'rm', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'pklxkd-59', '--object', 'obj22', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-05-31T20:54:47.728 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 object rm --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --bucket pklxkd-59 --object obj22 --cluster c1 --debug-rgw 0 --debug-ms 0 2023-05-31T20:54:47.779 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2023-05-31T20:54:47.780 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2023-05-31T20:54:47.891 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['object', 'rm', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'pklxkd-59', '--object', 'obj23', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-05-31T20:54:47.891 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'object', 'rm', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'pklxkd-59', '--object', 'obj23', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-05-31T20:54:47.892 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 object rm --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --bucket pklxkd-59 --object obj23 --cluster c1 --debug-rgw 0 --debug-ms 0 2023-05-31T20:54:47.944 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2023-05-31T20:54:47.944 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2023-05-31T20:54:48.062 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['object', 'rm', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'pklxkd-59', '--object', 'obj29', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-05-31T20:54:48.063 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'object', 'rm', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'pklxkd-59', '--object', 'obj29', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-05-31T20:54:48.063 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 object rm --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --bucket pklxkd-59 --object obj29 --cluster c1 --debug-rgw 0 --debug-ms 0 2023-05-31T20:54:48.115 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2023-05-31T20:54:48.116 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2023-05-31T20:54:48.236 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['object', 'rm', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'pklxkd-59', '--object', 'obj31', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-05-31T20:54:48.237 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'object', 'rm', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'pklxkd-59', '--object', 'obj31', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-05-31T20:54:48.237 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 object rm --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --bucket pklxkd-59 --object obj31 --cluster c1 --debug-rgw 0 --debug-ms 0 2023-05-31T20:54:48.287 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2023-05-31T20:54:48.287 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2023-05-31T20:54:48.407 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['object', 'rm', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'pklxkd-59', '--object', 'obj33', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-05-31T20:54:48.408 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'object', 'rm', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'pklxkd-59', '--object', 'obj33', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-05-31T20:54:48.408 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 object rm --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --bucket pklxkd-59 --object obj33 --cluster c1 --debug-rgw 0 --debug-ms 0 2023-05-31T20:54:48.460 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2023-05-31T20:54:48.460 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2023-05-31T20:54:48.589 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['object', 'rm', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'pklxkd-59', '--object', 'obj34', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-05-31T20:54:48.590 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'object', 'rm', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'pklxkd-59', '--object', 'obj34', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-05-31T20:54:48.590 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 object rm --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --bucket pklxkd-59 --object obj34 --cluster c1 --debug-rgw 0 --debug-ms 0 2023-05-31T20:54:48.641 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2023-05-31T20:54:48.642 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2023-05-31T20:54:48.759 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['object', 'rm', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'pklxkd-59', '--object', 'obj37', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-05-31T20:54:48.759 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'object', 'rm', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'pklxkd-59', '--object', 'obj37', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-05-31T20:54:48.760 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 object rm --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --bucket pklxkd-59 --object obj37 --cluster c1 --debug-rgw 0 --debug-ms 0 2023-05-31T20:54:48.811 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2023-05-31T20:54:48.811 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2023-05-31T20:54:48.924 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['object', 'rm', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'pklxkd-59', '--object', 'obj38', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-05-31T20:54:48.925 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'object', 'rm', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'pklxkd-59', '--object', 'obj38', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-05-31T20:54:48.925 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 object rm --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --bucket pklxkd-59 --object obj38 --cluster c1 --debug-rgw 0 --debug-ms 0 2023-05-31T20:54:48.977 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2023-05-31T20:54:48.977 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2023-05-31T20:54:49.087 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['object', 'rm', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'pklxkd-59', '--object', 'obj44', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-05-31T20:54:49.087 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'object', 'rm', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'pklxkd-59', '--object', 'obj44', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-05-31T20:54:49.088 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 object rm --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --bucket pklxkd-59 --object obj44 --cluster c1 --debug-rgw 0 --debug-ms 0 2023-05-31T20:54:49.139 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2023-05-31T20:54:49.139 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2023-05-31T20:54:49.273 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['object', 'rm', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'pklxkd-59', '--object', 'obj46', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-05-31T20:54:49.273 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'object', 'rm', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'pklxkd-59', '--object', 'obj46', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-05-31T20:54:49.274 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 object rm --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --bucket pklxkd-59 --object obj46 --cluster c1 --debug-rgw 0 --debug-ms 0 2023-05-31T20:54:49.326 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2023-05-31T20:54:49.326 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2023-05-31T20:54:49.441 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['object', 'rm', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'pklxkd-59', '--object', 'obj51', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-05-31T20:54:49.442 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'object', 'rm', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'pklxkd-59', '--object', 'obj51', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-05-31T20:54:49.442 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 object rm --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --bucket pklxkd-59 --object obj51 --cluster c1 --debug-rgw 0 --debug-ms 0 2023-05-31T20:54:49.493 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2023-05-31T20:54:49.493 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2023-05-31T20:54:49.612 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['object', 'rm', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'pklxkd-59', '--object', 'obj55', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-05-31T20:54:49.612 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'object', 'rm', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'pklxkd-59', '--object', 'obj55', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-05-31T20:54:49.613 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 object rm --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --bucket pklxkd-59 --object obj55 --cluster c1 --debug-rgw 0 --debug-ms 0 2023-05-31T20:54:49.663 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2023-05-31T20:54:49.663 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2023-05-31T20:54:49.771 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['object', 'rm', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'pklxkd-59', '--object', 'obj57', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-05-31T20:54:49.771 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'object', 'rm', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'pklxkd-59', '--object', 'obj57', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-05-31T20:54:49.772 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 object rm --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --bucket pklxkd-59 --object obj57 --cluster c1 --debug-rgw 0 --debug-ms 0 2023-05-31T20:54:49.814 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.1 has been restored 2023-05-31T20:54:49.822 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2023-05-31T20:54:49.822 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2023-05-31T20:54:49.933 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['object', 'rm', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'pklxkd-59', '--object', 'obj58', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-05-31T20:54:49.934 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'object', 'rm', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'pklxkd-59', '--object', 'obj58', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-05-31T20:54:49.934 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 object rm --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --bucket pklxkd-59 --object obj58 --cluster c1 --debug-rgw 0 --debug-ms 0 2023-05-31T20:54:49.988 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2023-05-31T20:54:49.988 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2023-05-31T20:54:50.122 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['object', 'rm', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'pklxkd-59', '--object', 'obj62', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-05-31T20:54:50.122 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'object', 'rm', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'pklxkd-59', '--object', 'obj62', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-05-31T20:54:50.123 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 object rm --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --bucket pklxkd-59 --object obj62 --cluster c1 --debug-rgw 0 --debug-ms 0 2023-05-31T20:54:50.173 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2023-05-31T20:54:50.174 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2023-05-31T20:54:50.290 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['object', 'rm', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'pklxkd-59', '--object', 'obj68', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-05-31T20:54:50.290 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'object', 'rm', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'pklxkd-59', '--object', 'obj68', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-05-31T20:54:50.291 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 object rm --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --bucket pklxkd-59 --object obj68 --cluster c1 --debug-rgw 0 --debug-ms 0 2023-05-31T20:54:50.342 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2023-05-31T20:54:50.343 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2023-05-31T20:54:50.475 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['object', 'rm', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'pklxkd-59', '--object', 'obj69', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-05-31T20:54:50.475 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'object', 'rm', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'pklxkd-59', '--object', 'obj69', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-05-31T20:54:50.476 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 object rm --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --bucket pklxkd-59 --object obj69 --cluster c1 --debug-rgw 0 --debug-ms 0 2023-05-31T20:54:50.527 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2023-05-31T20:54:50.527 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2023-05-31T20:54:50.649 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['object', 'rm', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'pklxkd-59', '--object', 'obj74', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-05-31T20:54:50.649 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'object', 'rm', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'pklxkd-59', '--object', 'obj74', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-05-31T20:54:50.650 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 object rm --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --bucket pklxkd-59 --object obj74 --cluster c1 --debug-rgw 0 --debug-ms 0 2023-05-31T20:54:50.701 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2023-05-31T20:54:50.701 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2023-05-31T20:54:50.791 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['object', 'rm', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'pklxkd-59', '--object', 'obj76', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-05-31T20:54:50.792 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'object', 'rm', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'pklxkd-59', '--object', 'obj76', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-05-31T20:54:50.793 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 object rm --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --bucket pklxkd-59 --object obj76 --cluster c1 --debug-rgw 0 --debug-ms 0 2023-05-31T20:54:50.843 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2023-05-31T20:54:50.844 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2023-05-31T20:54:50.974 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['object', 'rm', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'pklxkd-59', '--object', 'obj85', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-05-31T20:54:50.975 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'object', 'rm', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'pklxkd-59', '--object', 'obj85', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-05-31T20:54:50.975 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 object rm --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --bucket pklxkd-59 --object obj85 --cluster c1 --debug-rgw 0 --debug-ms 0 2023-05-31T20:54:51.026 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2023-05-31T20:54:51.027 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2023-05-31T20:54:51.125 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['object', 'rm', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'pklxkd-59', '--object', 'obj87', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-05-31T20:54:51.126 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'object', 'rm', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'pklxkd-59', '--object', 'obj87', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-05-31T20:54:51.126 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 object rm --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --bucket pklxkd-59 --object obj87 --cluster c1 --debug-rgw 0 --debug-ms 0 2023-05-31T20:54:51.177 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2023-05-31T20:54:51.177 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2023-05-31T20:54:51.296 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['object', 'rm', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'pklxkd-59', '--object', 'obj92', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-05-31T20:54:51.296 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'object', 'rm', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'pklxkd-59', '--object', 'obj92', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-05-31T20:54:51.297 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 object rm --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --bucket pklxkd-59 --object obj92 --cluster c1 --debug-rgw 0 --debug-ms 0 2023-05-31T20:54:51.347 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2023-05-31T20:54:51.348 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2023-05-31T20:54:51.456 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['object', 'rm', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'pklxkd-59', '--object', 'obj93', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-05-31T20:54:51.456 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'object', 'rm', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'pklxkd-59', '--object', 'obj93', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-05-31T20:54:51.457 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 object rm --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --bucket pklxkd-59 --object obj93 --cluster c1 --debug-rgw 0 --debug-ms 0 2023-05-31T20:54:51.508 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2023-05-31T20:54:51.508 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2023-05-31T20:54:51.626 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['object', 'rm', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'pklxkd-59', '--object', 'obj95', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-05-31T20:54:51.626 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'object', 'rm', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'pklxkd-59', '--object', 'obj95', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-05-31T20:54:51.627 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 object rm --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --bucket pklxkd-59 --object obj95 --cluster c1 --debug-rgw 0 --debug-ms 0 2023-05-31T20:54:51.678 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2023-05-31T20:54:51.679 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2023-05-31T20:54:51.791 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'trim', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'pklxkd-59', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-05-31T20:54:51.792 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'trim', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'pklxkd-59', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-05-31T20:54:51.792 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog trim --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --bucket pklxkd-59 --cluster c1 --debug-rgw 0 --debug-ms 0 2023-05-31T20:54:51.841 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2023-05-31T20:54:51.841 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2023-05-31T20:54:51.914 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'init', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'pklxkd-59', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-05-31T20:54:51.914 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'init', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'pklxkd-59', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-05-31T20:54:51.915 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync init --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --bucket pklxkd-59 --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 2023-05-31T20:54:51.966 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2023-05-31T20:54:51.967 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2023-05-31T20:54:52.023 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'run', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'pklxkd-59', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-05-31T20:54:52.023 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'run', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'pklxkd-59', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-05-31T20:54:52.024 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync run --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --bucket pklxkd-59 --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 2023-05-31T20:54:52.077 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2023-05-31T20:54:52.077 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2023-05-31T20:54:52.172 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:54:52.167+0000 7f259334b780 20 RGW-SYNC:bucket[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24<-pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]: sync status for source bucket: init. lease is: not taken. stop indications is: 0 2023-05-31T20:54:52.172 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:54:52.167+0000 7f259334b780 5 RGW-SYNC:bucket[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24<-pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]: waiting on bucket lease 2023-05-31T20:54:52.178 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:54:52.174+0000 7f259334b780 20 RGW-SYNC:bucket[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24<-pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]: status after acquiring the lock is: init 2023-05-31T20:54:52.197 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:54:52.192+0000 7f259334b780 20 RGW-SYNC:bucket[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24<-pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:full_sync[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]: listing bucket for full sync 2023-05-31T20:54:52.244 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:54:52.238+0000 7f259334b780 20 RGW-SYNC:bucket[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24<-pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:full_sync[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]: [full sync] syncing object: pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0/obj11 2023-05-31T20:54:52.244 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:54:52.238+0000 7f259334b780 20 RGW-SYNC:bucket[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24<-pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:full_sync[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:entry[obj11]: bucket sync single entry (source_zone=3d163b1e-09c7-47e2-ae87-add4aace3f38) b=bucket sync single entry (source_zone=3d163b1e-09c7-47e2-ae87-add4aace3f38) b= log_entry=obj11 op=0 op_state=1 2023-05-31T20:54:52.245 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:54:52.238+0000 7f259334b780 5 RGW-SYNC:bucket[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24<-pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:full_sync[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:entry[obj11]: bucket sync: sync obj: 3d163b1e-09c7-47e2-ae87-add4aace3f38/:pklxkd-59[3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24])/obj11[0] 2023-05-31T20:54:52.245 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:54:52.238+0000 7f259334b780 20 RGW-SYNC:bucket[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24<-pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:full_sync[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]: [full sync] syncing object: pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0/obj111 2023-05-31T20:54:52.245 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:54:52.238+0000 7f259334b780 20 RGW-SYNC:bucket[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24<-pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:full_sync[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:entry[obj111]: bucket sync single entry (source_zone=3d163b1e-09c7-47e2-ae87-add4aace3f38) b=bucket sync single entry (source_zone=3d163b1e-09c7-47e2-ae87-add4aace3f38) b= log_entry=obj111 op=0 op_state=1 2023-05-31T20:54:52.245 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:54:52.238+0000 7f259334b780 0 RGW-SYNC:bucket[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24<-pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:full_sync[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:entry[obj111]: : injecting data sync error on key=obj111 2023-05-31T20:54:52.245 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:54:52.238+0000 7f259334b780 10 RGW-SYNC:bucket[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24<-pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:full_sync[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:entry[obj111]: failed, retcode=-5 ((5) Input/output error) 2023-05-31T20:54:52.245 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:54:52.238+0000 7f259334b780 0 RGW-SYNC:bucket[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24<-pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:full_sync[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:entry[obj111]: ERROR: failed to sync object: pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0/obj111 2023-05-31T20:54:52.246 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:54:52.238+0000 7f259334b780 20 RGW-SYNC:bucket[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24<-pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:full_sync[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]: [full sync] syncing object: pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0/obj115 2023-05-31T20:54:52.246 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:54:52.238+0000 7f259334b780 20 RGW-SYNC:bucket[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24<-pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:full_sync[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:entry[obj115]: bucket sync single entry (source_zone=3d163b1e-09c7-47e2-ae87-add4aace3f38) b=bucket sync single entry (source_zone=3d163b1e-09c7-47e2-ae87-add4aace3f38) b= log_entry=obj115 op=0 op_state=1 2023-05-31T20:54:52.246 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:54:52.238+0000 7f259334b780 5 RGW-SYNC:bucket[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24<-pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:full_sync[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:entry[obj115]: bucket sync: sync obj: 3d163b1e-09c7-47e2-ae87-add4aace3f38/:pklxkd-59[3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24])/obj115[0] 2023-05-31T20:54:52.246 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:54:52.238+0000 7f259334b780 20 RGW-SYNC:bucket[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24<-pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:full_sync[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]: [full sync] syncing object: pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0/obj116 2023-05-31T20:54:52.246 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:54:52.238+0000 7f259334b780 20 RGW-SYNC:bucket[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24<-pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:full_sync[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:entry[obj116]: bucket sync single entry (source_zone=3d163b1e-09c7-47e2-ae87-add4aace3f38) b=bucket sync single entry (source_zone=3d163b1e-09c7-47e2-ae87-add4aace3f38) b= log_entry=obj116 op=0 op_state=1 2023-05-31T20:54:52.246 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:54:52.238+0000 7f259334b780 5 RGW-SYNC:bucket[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24<-pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:full_sync[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:entry[obj116]: bucket sync: sync obj: 3d163b1e-09c7-47e2-ae87-add4aace3f38/:pklxkd-59[3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24])/obj116[0] 2023-05-31T20:54:52.246 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:54:52.238+0000 7f259334b780 20 RGW-SYNC:bucket[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24<-pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:full_sync[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]: [full sync] syncing object: pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0/obj124 2023-05-31T20:54:52.247 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:54:52.238+0000 7f259334b780 20 RGW-SYNC:bucket[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24<-pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:full_sync[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:entry[obj124]: bucket sync single entry (source_zone=3d163b1e-09c7-47e2-ae87-add4aace3f38) b=bucket sync single entry (source_zone=3d163b1e-09c7-47e2-ae87-add4aace3f38) b= log_entry=obj124 op=0 op_state=1 2023-05-31T20:54:52.247 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:54:52.238+0000 7f259334b780 5 RGW-SYNC:bucket[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24<-pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:full_sync[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:entry[obj124]: bucket sync: sync obj: 3d163b1e-09c7-47e2-ae87-add4aace3f38/:pklxkd-59[3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24])/obj124[0] 2023-05-31T20:54:52.247 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:54:52.238+0000 7f259334b780 20 RGW-SYNC:bucket[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24<-pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:full_sync[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]: [full sync] syncing object: pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0/obj145 2023-05-31T20:54:52.247 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:54:52.238+0000 7f259334b780 20 RGW-SYNC:bucket[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24<-pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:full_sync[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:entry[obj145]: bucket sync single entry (source_zone=3d163b1e-09c7-47e2-ae87-add4aace3f38) b=bucket sync single entry (source_zone=3d163b1e-09c7-47e2-ae87-add4aace3f38) b= log_entry=obj145 op=0 op_state=1 2023-05-31T20:54:52.247 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:54:52.238+0000 7f259334b780 5 RGW-SYNC:bucket[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24<-pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:full_sync[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:entry[obj145]: bucket sync: sync obj: 3d163b1e-09c7-47e2-ae87-add4aace3f38/:pklxkd-59[3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24])/obj145[0] 2023-05-31T20:54:52.247 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:54:52.238+0000 7f259334b780 20 RGW-SYNC:bucket[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24<-pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:full_sync[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]: [full sync] syncing object: pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0/obj148 2023-05-31T20:54:52.248 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:54:52.238+0000 7f259334b780 20 RGW-SYNC:bucket[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24<-pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:full_sync[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:entry[obj148]: bucket sync single entry (source_zone=3d163b1e-09c7-47e2-ae87-add4aace3f38) b=bucket sync single entry (source_zone=3d163b1e-09c7-47e2-ae87-add4aace3f38) b= log_entry=obj148 op=0 op_state=1 2023-05-31T20:54:52.248 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:54:52.238+0000 7f259334b780 5 RGW-SYNC:bucket[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24<-pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:full_sync[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:entry[obj148]: bucket sync: sync obj: 3d163b1e-09c7-47e2-ae87-add4aace3f38/:pklxkd-59[3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24])/obj148[0] 2023-05-31T20:54:52.248 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:54:52.238+0000 7f259334b780 20 RGW-SYNC:bucket[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24<-pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:full_sync[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]: [full sync] syncing object: pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0/obj155 2023-05-31T20:54:52.248 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:54:52.238+0000 7f259334b780 20 RGW-SYNC:bucket[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24<-pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:full_sync[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:entry[obj155]: bucket sync single entry (source_zone=3d163b1e-09c7-47e2-ae87-add4aace3f38) b=bucket sync single entry (source_zone=3d163b1e-09c7-47e2-ae87-add4aace3f38) b= log_entry=obj155 op=0 op_state=1 2023-05-31T20:54:52.248 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:54:52.238+0000 7f259334b780 0 RGW-SYNC:bucket[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24<-pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:full_sync[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:entry[obj155]: : injecting data sync error on key=obj155 2023-05-31T20:54:52.248 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:54:52.238+0000 7f259334b780 10 RGW-SYNC:bucket[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24<-pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:full_sync[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:entry[obj155]: failed, retcode=-5 ((5) Input/output error) 2023-05-31T20:54:52.248 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:54:52.238+0000 7f259334b780 0 RGW-SYNC:bucket[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24<-pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:full_sync[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:entry[obj155]: ERROR: failed to sync object: pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0/obj155 2023-05-31T20:54:52.249 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:54:52.238+0000 7f259334b780 20 RGW-SYNC:bucket[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24<-pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:full_sync[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]: [full sync] syncing object: pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0/obj17 2023-05-31T20:54:52.249 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:54:52.238+0000 7f259334b780 20 RGW-SYNC:bucket[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24<-pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:full_sync[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:entry[obj17]: bucket sync single entry (source_zone=3d163b1e-09c7-47e2-ae87-add4aace3f38) b=bucket sync single entry (source_zone=3d163b1e-09c7-47e2-ae87-add4aace3f38) b= log_entry=obj17 op=0 op_state=1 2023-05-31T20:54:52.249 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:54:52.238+0000 7f259334b780 5 RGW-SYNC:bucket[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24<-pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:full_sync[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:entry[obj17]: bucket sync: sync obj: 3d163b1e-09c7-47e2-ae87-add4aace3f38/:pklxkd-59[3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24])/obj17[0] 2023-05-31T20:54:52.249 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:54:52.238+0000 7f259334b780 20 RGW-SYNC:bucket[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24<-pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:full_sync[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]: [full sync] syncing object: pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0/obj185 2023-05-31T20:54:52.249 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:54:52.238+0000 7f259334b780 20 RGW-SYNC:bucket[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24<-pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:full_sync[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:entry[obj185]: bucket sync single entry (source_zone=3d163b1e-09c7-47e2-ae87-add4aace3f38) b=bucket sync single entry (source_zone=3d163b1e-09c7-47e2-ae87-add4aace3f38) b= log_entry=obj185 op=0 op_state=1 2023-05-31T20:54:52.249 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:54:52.238+0000 7f259334b780 5 RGW-SYNC:bucket[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24<-pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:full_sync[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:entry[obj185]: bucket sync: sync obj: 3d163b1e-09c7-47e2-ae87-add4aace3f38/:pklxkd-59[3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24])/obj185[0] 2023-05-31T20:54:52.250 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:54:52.238+0000 7f259334b780 20 RGW-SYNC:bucket[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24<-pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:full_sync[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]: [full sync] syncing object: pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0/obj19 2023-05-31T20:54:52.250 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:54:52.238+0000 7f259334b780 20 RGW-SYNC:bucket[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24<-pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:full_sync[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:entry[obj19]: bucket sync single entry (source_zone=3d163b1e-09c7-47e2-ae87-add4aace3f38) b=bucket sync single entry (source_zone=3d163b1e-09c7-47e2-ae87-add4aace3f38) b= log_entry=obj19 op=0 op_state=1 2023-05-31T20:54:52.250 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:54:52.238+0000 7f259334b780 5 RGW-SYNC:bucket[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24<-pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:full_sync[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:entry[obj19]: bucket sync: sync obj: 3d163b1e-09c7-47e2-ae87-add4aace3f38/:pklxkd-59[3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24])/obj19[0] 2023-05-31T20:54:52.250 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:54:52.238+0000 7f259334b780 20 RGW-SYNC:bucket[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24<-pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:full_sync[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]: [full sync] syncing object: pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0/obj22 2023-05-31T20:54:52.250 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:54:52.238+0000 7f259334b780 20 RGW-SYNC:bucket[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24<-pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:full_sync[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:entry[obj22]: bucket sync single entry (source_zone=3d163b1e-09c7-47e2-ae87-add4aace3f38) b=bucket sync single entry (source_zone=3d163b1e-09c7-47e2-ae87-add4aace3f38) b= log_entry=obj22 op=0 op_state=1 2023-05-31T20:54:52.250 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:54:52.238+0000 7f259334b780 0 RGW-SYNC:bucket[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24<-pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:full_sync[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:entry[obj22]: : injecting data sync error on key=obj22 2023-05-31T20:54:52.250 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:54:52.238+0000 7f259334b780 10 RGW-SYNC:bucket[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24<-pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:full_sync[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:entry[obj22]: failed, retcode=-5 ((5) Input/output error) 2023-05-31T20:54:52.251 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:54:52.238+0000 7f259334b780 0 RGW-SYNC:bucket[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24<-pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:full_sync[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:entry[obj22]: ERROR: failed to sync object: pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0/obj22 2023-05-31T20:54:52.251 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:54:52.238+0000 7f259334b780 20 RGW-SYNC:bucket[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24<-pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:full_sync[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]: [full sync] syncing object: pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0/obj23 2023-05-31T20:54:52.251 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:54:52.238+0000 7f259334b780 20 RGW-SYNC:bucket[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24<-pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:full_sync[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:entry[obj23]: bucket sync single entry (source_zone=3d163b1e-09c7-47e2-ae87-add4aace3f38) b=bucket sync single entry (source_zone=3d163b1e-09c7-47e2-ae87-add4aace3f38) b= log_entry=obj23 op=0 op_state=1 2023-05-31T20:54:52.251 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:54:52.238+0000 7f259334b780 5 RGW-SYNC:bucket[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24<-pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:full_sync[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:entry[obj23]: bucket sync: sync obj: 3d163b1e-09c7-47e2-ae87-add4aace3f38/:pklxkd-59[3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24])/obj23[0] 2023-05-31T20:54:52.251 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:54:52.239+0000 7f259334b780 20 RGW-SYNC:bucket[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24<-pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:full_sync[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]: [full sync] syncing object: pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0/obj29 2023-05-31T20:54:52.251 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:54:52.239+0000 7f259334b780 20 RGW-SYNC:bucket[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24<-pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:full_sync[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:entry[obj29]: bucket sync single entry (source_zone=3d163b1e-09c7-47e2-ae87-add4aace3f38) b=bucket sync single entry (source_zone=3d163b1e-09c7-47e2-ae87-add4aace3f38) b= log_entry=obj29 op=0 op_state=1 2023-05-31T20:54:52.251 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:54:52.239+0000 7f259334b780 0 RGW-SYNC:bucket[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24<-pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:full_sync[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:entry[obj29]: : injecting data sync error on key=obj29 2023-05-31T20:54:52.252 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:54:52.239+0000 7f259334b780 10 RGW-SYNC:bucket[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24<-pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:full_sync[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:entry[obj29]: failed, retcode=-5 ((5) Input/output error) 2023-05-31T20:54:52.252 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:54:52.239+0000 7f259334b780 0 RGW-SYNC:bucket[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24<-pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:full_sync[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:entry[obj29]: ERROR: failed to sync object: pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0/obj29 2023-05-31T20:54:52.252 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:54:52.239+0000 7f259334b780 20 RGW-SYNC:bucket[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24<-pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:full_sync[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]: [full sync] syncing object: pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0/obj31 2023-05-31T20:54:52.252 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:54:52.239+0000 7f259334b780 20 RGW-SYNC:bucket[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24<-pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:full_sync[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:entry[obj31]: bucket sync single entry (source_zone=3d163b1e-09c7-47e2-ae87-add4aace3f38) b=bucket sync single entry (source_zone=3d163b1e-09c7-47e2-ae87-add4aace3f38) b= log_entry=obj31 op=0 op_state=1 2023-05-31T20:54:52.252 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:54:52.239+0000 7f259334b780 5 RGW-SYNC:bucket[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24<-pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:full_sync[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:entry[obj31]: bucket sync: sync obj: 3d163b1e-09c7-47e2-ae87-add4aace3f38/:pklxkd-59[3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24])/obj31[0] 2023-05-31T20:54:52.252 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:54:52.239+0000 7f259334b780 20 RGW-SYNC:bucket[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24<-pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:full_sync[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]: [full sync] syncing object: pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0/obj33 2023-05-31T20:54:52.253 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:54:52.239+0000 7f259334b780 20 RGW-SYNC:bucket[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24<-pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:full_sync[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:entry[obj33]: bucket sync single entry (source_zone=3d163b1e-09c7-47e2-ae87-add4aace3f38) b=bucket sync single entry (source_zone=3d163b1e-09c7-47e2-ae87-add4aace3f38) b= log_entry=obj33 op=0 op_state=1 2023-05-31T20:54:52.253 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:54:52.239+0000 7f259334b780 5 RGW-SYNC:bucket[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24<-pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:full_sync[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:entry[obj33]: bucket sync: sync obj: 3d163b1e-09c7-47e2-ae87-add4aace3f38/:pklxkd-59[3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24])/obj33[0] 2023-05-31T20:54:52.253 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:54:52.239+0000 7f259334b780 20 RGW-SYNC:bucket[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24<-pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:full_sync[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]: [full sync] syncing object: pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0/obj34 2023-05-31T20:54:52.253 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:54:52.239+0000 7f259334b780 20 RGW-SYNC:bucket[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24<-pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:full_sync[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:entry[obj34]: bucket sync single entry (source_zone=3d163b1e-09c7-47e2-ae87-add4aace3f38) b=bucket sync single entry (source_zone=3d163b1e-09c7-47e2-ae87-add4aace3f38) b= log_entry=obj34 op=0 op_state=1 2023-05-31T20:54:52.253 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:54:52.239+0000 7f259334b780 0 RGW-SYNC:bucket[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24<-pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:full_sync[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:entry[obj34]: : injecting data sync error on key=obj34 2023-05-31T20:54:52.253 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:54:52.239+0000 7f259334b780 10 RGW-SYNC:bucket[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24<-pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:full_sync[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:entry[obj34]: failed, retcode=-5 ((5) Input/output error) 2023-05-31T20:54:52.253 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:54:52.239+0000 7f259334b780 0 RGW-SYNC:bucket[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24<-pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:full_sync[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:entry[obj34]: ERROR: failed to sync object: pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0/obj34 2023-05-31T20:54:52.254 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:54:52.239+0000 7f259334b780 20 RGW-SYNC:bucket[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24<-pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:full_sync[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]: [full sync] syncing object: pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0/obj37 2023-05-31T20:54:52.254 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:54:52.239+0000 7f259334b780 20 RGW-SYNC:bucket[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24<-pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:full_sync[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:entry[obj37]: bucket sync single entry (source_zone=3d163b1e-09c7-47e2-ae87-add4aace3f38) b=bucket sync single entry (source_zone=3d163b1e-09c7-47e2-ae87-add4aace3f38) b= log_entry=obj37 op=0 op_state=1 2023-05-31T20:54:52.254 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:54:52.239+0000 7f259334b780 5 RGW-SYNC:bucket[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24<-pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:full_sync[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:entry[obj37]: bucket sync: sync obj: 3d163b1e-09c7-47e2-ae87-add4aace3f38/:pklxkd-59[3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24])/obj37[0] 2023-05-31T20:54:52.254 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:54:52.239+0000 7f259334b780 20 RGW-SYNC:bucket[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24<-pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:full_sync[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]: [full sync] syncing object: pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0/obj38 2023-05-31T20:54:52.254 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:54:52.239+0000 7f259334b780 20 RGW-SYNC:bucket[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24<-pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:full_sync[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:entry[obj38]: bucket sync single entry (source_zone=3d163b1e-09c7-47e2-ae87-add4aace3f38) b=bucket sync single entry (source_zone=3d163b1e-09c7-47e2-ae87-add4aace3f38) b= log_entry=obj38 op=0 op_state=1 2023-05-31T20:54:52.254 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:54:52.239+0000 7f259334b780 5 RGW-SYNC:bucket[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24<-pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:full_sync[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:entry[obj38]: bucket sync: sync obj: 3d163b1e-09c7-47e2-ae87-add4aace3f38/:pklxkd-59[3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24])/obj38[0] 2023-05-31T20:54:52.255 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:54:52.239+0000 7f259334b780 20 RGW-SYNC:bucket[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24<-pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:full_sync[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]: [full sync] syncing object: pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0/obj44 2023-05-31T20:54:52.255 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:54:52.239+0000 7f259334b780 20 RGW-SYNC:bucket[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24<-pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:full_sync[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:entry[obj44]: bucket sync single entry (source_zone=3d163b1e-09c7-47e2-ae87-add4aace3f38) b=bucket sync single entry (source_zone=3d163b1e-09c7-47e2-ae87-add4aace3f38) b= log_entry=obj44 op=0 op_state=1 2023-05-31T20:54:52.255 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:54:52.239+0000 7f259334b780 5 RGW-SYNC:bucket[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24<-pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:full_sync[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:entry[obj44]: bucket sync: sync obj: 3d163b1e-09c7-47e2-ae87-add4aace3f38/:pklxkd-59[3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24])/obj44[0] 2023-05-31T20:54:52.255 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:54:52.239+0000 7f259334b780 20 RGW-SYNC:bucket[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24<-pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:full_sync[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]: [full sync] syncing object: pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0/obj46 2023-05-31T20:54:52.255 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:54:52.239+0000 7f259334b780 20 RGW-SYNC:bucket[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24<-pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:full_sync[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:entry[obj46]: bucket sync single entry (source_zone=3d163b1e-09c7-47e2-ae87-add4aace3f38) b=bucket sync single entry (source_zone=3d163b1e-09c7-47e2-ae87-add4aace3f38) b= log_entry=obj46 op=0 op_state=1 2023-05-31T20:54:52.255 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:54:52.239+0000 7f259334b780 5 RGW-SYNC:bucket[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24<-pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:full_sync[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:entry[obj46]: bucket sync: sync obj: 3d163b1e-09c7-47e2-ae87-add4aace3f38/:pklxkd-59[3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24])/obj46[0] 2023-05-31T20:54:52.256 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:54:52.242+0000 7f259334b780 10 RGW-SYNC:bucket[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24<-pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:full_sync[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]: a sync operation returned error 2023-05-31T20:54:52.256 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:54:52.242+0000 7f259334b780 20 RGW-SYNC:bucket[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24<-pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:full_sync[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]: [full sync] syncing object: pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0/obj51 2023-05-31T20:54:52.256 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:54:52.242+0000 7f259334b780 20 RGW-SYNC:bucket[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24<-pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:full_sync[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:entry[obj51]: bucket sync single entry (source_zone=3d163b1e-09c7-47e2-ae87-add4aace3f38) b=bucket sync single entry (source_zone=3d163b1e-09c7-47e2-ae87-add4aace3f38) b= log_entry=obj51 op=0 op_state=1 2023-05-31T20:54:52.257 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:54:52.242+0000 7f259334b780 5 RGW-SYNC:bucket[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24<-pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:full_sync[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:entry[obj51]: bucket sync: sync obj: 3d163b1e-09c7-47e2-ae87-add4aace3f38/:pklxkd-59[3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24])/obj51[0] 2023-05-31T20:54:52.257 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:54:52.244+0000 7f259334b780 10 RGW-SYNC:bucket[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24<-pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:full_sync[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]: a sync operation returned error 2023-05-31T20:54:52.257 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:54:52.244+0000 7f259334b780 20 RGW-SYNC:bucket[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24<-pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:full_sync[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]: [full sync] syncing object: pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0/obj55 2023-05-31T20:54:52.257 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:54:52.244+0000 7f259334b780 20 RGW-SYNC:bucket[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24<-pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:full_sync[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:entry[obj55]: bucket sync single entry (source_zone=3d163b1e-09c7-47e2-ae87-add4aace3f38) b=bucket sync single entry (source_zone=3d163b1e-09c7-47e2-ae87-add4aace3f38) b= log_entry=obj55 op=0 op_state=1 2023-05-31T20:54:52.257 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:54:52.244+0000 7f259334b780 5 RGW-SYNC:bucket[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24<-pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:full_sync[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:entry[obj55]: bucket sync: sync obj: 3d163b1e-09c7-47e2-ae87-add4aace3f38/:pklxkd-59[3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24])/obj55[0] 2023-05-31T20:54:52.257 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:54:52.244+0000 7f259334b780 10 RGW-SYNC:bucket[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24<-pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:full_sync[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]: a sync operation returned error 2023-05-31T20:54:52.257 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:54:52.244+0000 7f259334b780 20 RGW-SYNC:bucket[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24<-pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:full_sync[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]: [full sync] syncing object: pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0/obj57 2023-05-31T20:54:52.258 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:54:52.244+0000 7f259334b780 20 RGW-SYNC:bucket[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24<-pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:full_sync[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:entry[obj57]: bucket sync single entry (source_zone=3d163b1e-09c7-47e2-ae87-add4aace3f38) b=bucket sync single entry (source_zone=3d163b1e-09c7-47e2-ae87-add4aace3f38) b= log_entry=obj57 op=0 op_state=1 2023-05-31T20:54:52.258 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:54:52.244+0000 7f259334b780 5 RGW-SYNC:bucket[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24<-pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:full_sync[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:entry[obj57]: bucket sync: sync obj: 3d163b1e-09c7-47e2-ae87-add4aace3f38/:pklxkd-59[3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24])/obj57[0] 2023-05-31T20:54:52.258 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:54:52.245+0000 7f259334b780 10 RGW-SYNC:bucket[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24<-pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:full_sync[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]: a sync operation returned error 2023-05-31T20:54:52.258 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:54:52.245+0000 7f259334b780 20 RGW-SYNC:bucket[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24<-pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:full_sync[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]: [full sync] syncing object: pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0/obj58 2023-05-31T20:54:52.258 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:54:52.245+0000 7f259334b780 20 RGW-SYNC:bucket[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24<-pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:full_sync[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:entry[obj58]: bucket sync single entry (source_zone=3d163b1e-09c7-47e2-ae87-add4aace3f38) b=bucket sync single entry (source_zone=3d163b1e-09c7-47e2-ae87-add4aace3f38) b= log_entry=obj58 op=0 op_state=1 2023-05-31T20:54:52.258 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:54:52.245+0000 7f259334b780 5 RGW-SYNC:bucket[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24<-pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:full_sync[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:entry[obj58]: bucket sync: sync obj: 3d163b1e-09c7-47e2-ae87-add4aace3f38/:pklxkd-59[3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24])/obj58[0] 2023-05-31T20:54:52.259 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:54:52.251+0000 7f259334b780 10 RGW-SYNC:bucket[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24<-pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:full_sync[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]: a sync operation returned error 2023-05-31T20:54:52.259 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:54:52.251+0000 7f259334b780 20 RGW-SYNC:bucket[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24<-pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:full_sync[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]: [full sync] syncing object: pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0/obj62 2023-05-31T20:54:52.259 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:54:52.251+0000 7f259334b780 20 RGW-SYNC:bucket[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24<-pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:full_sync[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:entry[obj62]: bucket sync single entry (source_zone=3d163b1e-09c7-47e2-ae87-add4aace3f38) b=bucket sync single entry (source_zone=3d163b1e-09c7-47e2-ae87-add4aace3f38) b= log_entry=obj62 op=0 op_state=1 2023-05-31T20:54:52.259 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:54:52.251+0000 7f259334b780 5 RGW-SYNC:bucket[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24<-pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:full_sync[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:entry[obj62]: bucket sync: sync obj: 3d163b1e-09c7-47e2-ae87-add4aace3f38/:pklxkd-59[3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24])/obj62[0] 2023-05-31T20:54:52.276 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:54:52.271+0000 7f259334b780 10 RGW-SYNC:bucket[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24<-pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:full_sync[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:entry[obj124]: success 2023-05-31T20:54:52.276 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:54:52.271+0000 7f259334b780 20 RGW-SYNC:bucket[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24<-pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:full_sync[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]: [full sync] syncing object: pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0/obj68 2023-05-31T20:54:52.276 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:54:52.271+0000 7f259334b780 20 RGW-SYNC:bucket[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24<-pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:full_sync[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:entry[obj68]: bucket sync single entry (source_zone=3d163b1e-09c7-47e2-ae87-add4aace3f38) b=bucket sync single entry (source_zone=3d163b1e-09c7-47e2-ae87-add4aace3f38) b= log_entry=obj68 op=0 op_state=1 2023-05-31T20:54:52.276 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:54:52.271+0000 7f259334b780 5 RGW-SYNC:bucket[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24<-pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:full_sync[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:entry[obj68]: bucket sync: sync obj: 3d163b1e-09c7-47e2-ae87-add4aace3f38/:pklxkd-59[3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24])/obj68[0] 2023-05-31T20:54:52.278 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:54:52.273+0000 7f259334b780 10 RGW-SYNC:bucket[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24<-pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:full_sync[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:entry[obj44]: success 2023-05-31T20:54:52.278 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:54:52.273+0000 7f259334b780 20 RGW-SYNC:bucket[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24<-pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:full_sync[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]: [full sync] syncing object: pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0/obj69 2023-05-31T20:54:52.278 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:54:52.273+0000 7f259334b780 20 RGW-SYNC:bucket[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24<-pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:full_sync[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:entry[obj69]: bucket sync single entry (source_zone=3d163b1e-09c7-47e2-ae87-add4aace3f38) b=bucket sync single entry (source_zone=3d163b1e-09c7-47e2-ae87-add4aace3f38) b= log_entry=obj69 op=0 op_state=1 2023-05-31T20:54:52.278 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:54:52.273+0000 7f259334b780 5 RGW-SYNC:bucket[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24<-pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:full_sync[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:entry[obj69]: bucket sync: sync obj: 3d163b1e-09c7-47e2-ae87-add4aace3f38/:pklxkd-59[3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24])/obj69[0] 2023-05-31T20:54:52.278 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:54:52.273+0000 7f259334b780 10 RGW-SYNC:bucket[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24<-pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:full_sync[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:entry[obj148]: success 2023-05-31T20:54:52.279 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:54:52.273+0000 7f259334b780 20 RGW-SYNC:bucket[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24<-pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:full_sync[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]: [full sync] syncing object: pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0/obj74 2023-05-31T20:54:52.279 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:54:52.273+0000 7f259334b780 20 RGW-SYNC:bucket[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24<-pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:full_sync[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:entry[obj74]: bucket sync single entry (source_zone=3d163b1e-09c7-47e2-ae87-add4aace3f38) b=bucket sync single entry (source_zone=3d163b1e-09c7-47e2-ae87-add4aace3f38) b= log_entry=obj74 op=0 op_state=1 2023-05-31T20:54:52.279 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:54:52.273+0000 7f259334b780 5 RGW-SYNC:bucket[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24<-pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:full_sync[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:entry[obj74]: bucket sync: sync obj: 3d163b1e-09c7-47e2-ae87-add4aace3f38/:pklxkd-59[3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24])/obj74[0] 2023-05-31T20:54:52.280 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:54:52.275+0000 7f259334b780 10 RGW-SYNC:bucket[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24<-pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:full_sync[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:entry[obj11]: success 2023-05-31T20:54:52.280 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:54:52.275+0000 7f259334b780 10 RGW-SYNC:bucket[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24<-pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:full_sync[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:entry[obj115]: success 2023-05-31T20:54:52.280 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:54:52.275+0000 7f259334b780 10 RGW-SYNC:bucket[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24<-pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:full_sync[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:entry[obj37]: success 2023-05-31T20:54:52.280 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:54:52.275+0000 7f259334b780 10 RGW-SYNC:bucket[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24<-pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:full_sync[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:entry[obj19]: success 2023-05-31T20:54:52.281 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:54:52.275+0000 7f259334b780 20 RGW-SYNC:bucket[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24<-pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:full_sync[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]: [full sync] syncing object: pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0/obj76 2023-05-31T20:54:52.281 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:54:52.275+0000 7f259334b780 20 RGW-SYNC:bucket[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24<-pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:full_sync[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:entry[obj76]: bucket sync single entry (source_zone=3d163b1e-09c7-47e2-ae87-add4aace3f38) b=bucket sync single entry (source_zone=3d163b1e-09c7-47e2-ae87-add4aace3f38) b= log_entry=obj76 op=0 op_state=1 2023-05-31T20:54:52.281 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:54:52.275+0000 7f259334b780 5 RGW-SYNC:bucket[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24<-pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:full_sync[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:entry[obj76]: bucket sync: sync obj: 3d163b1e-09c7-47e2-ae87-add4aace3f38/:pklxkd-59[3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24])/obj76[0] 2023-05-31T20:54:52.281 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:54:52.275+0000 7f259334b780 20 RGW-SYNC:bucket[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24<-pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:full_sync[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]: [full sync] syncing object: pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0/obj85 2023-05-31T20:54:52.281 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:54:52.275+0000 7f259334b780 20 RGW-SYNC:bucket[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24<-pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:full_sync[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:entry[obj85]: bucket sync single entry (source_zone=3d163b1e-09c7-47e2-ae87-add4aace3f38) b=bucket sync single entry (source_zone=3d163b1e-09c7-47e2-ae87-add4aace3f38) b= log_entry=obj85 op=0 op_state=1 2023-05-31T20:54:52.281 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:54:52.275+0000 7f259334b780 5 RGW-SYNC:bucket[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24<-pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:full_sync[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:entry[obj85]: bucket sync: sync obj: 3d163b1e-09c7-47e2-ae87-add4aace3f38/:pklxkd-59[3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24])/obj85[0] 2023-05-31T20:54:52.281 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:54:52.275+0000 7f259334b780 20 RGW-SYNC:bucket[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24<-pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:full_sync[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]: [full sync] syncing object: pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0/obj87 2023-05-31T20:54:52.282 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:54:52.275+0000 7f259334b780 20 RGW-SYNC:bucket[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24<-pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:full_sync[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:entry[obj87]: bucket sync single entry (source_zone=3d163b1e-09c7-47e2-ae87-add4aace3f38) b=bucket sync single entry (source_zone=3d163b1e-09c7-47e2-ae87-add4aace3f38) b= log_entry=obj87 op=0 op_state=1 2023-05-31T20:54:52.282 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:54:52.275+0000 7f259334b780 5 RGW-SYNC:bucket[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24<-pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:full_sync[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:entry[obj87]: bucket sync: sync obj: 3d163b1e-09c7-47e2-ae87-add4aace3f38/:pklxkd-59[3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24])/obj87[0] 2023-05-31T20:54:52.282 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:54:52.275+0000 7f259334b780 20 RGW-SYNC:bucket[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24<-pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:full_sync[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]: [full sync] syncing object: pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0/obj92 2023-05-31T20:54:52.282 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:54:52.275+0000 7f259334b780 20 RGW-SYNC:bucket[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24<-pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:full_sync[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:entry[obj92]: bucket sync single entry (source_zone=3d163b1e-09c7-47e2-ae87-add4aace3f38) b=bucket sync single entry (source_zone=3d163b1e-09c7-47e2-ae87-add4aace3f38) b= log_entry=obj92 op=0 op_state=1 2023-05-31T20:54:52.282 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:54:52.275+0000 7f259334b780 5 RGW-SYNC:bucket[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24<-pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:full_sync[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:entry[obj92]: bucket sync: sync obj: 3d163b1e-09c7-47e2-ae87-add4aace3f38/:pklxkd-59[3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24])/obj92[0] 2023-05-31T20:54:52.283 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:54:52.278+0000 7f259334b780 10 RGW-SYNC:bucket[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24<-pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:full_sync[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:entry[obj31]: success 2023-05-31T20:54:52.284 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:54:52.278+0000 7f259334b780 20 RGW-SYNC:bucket[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24<-pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:full_sync[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]: [full sync] syncing object: pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0/obj93 2023-05-31T20:54:52.284 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:54:52.278+0000 7f259334b780 20 RGW-SYNC:bucket[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24<-pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:full_sync[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:entry[obj93]: bucket sync single entry (source_zone=3d163b1e-09c7-47e2-ae87-add4aace3f38) b=bucket sync single entry (source_zone=3d163b1e-09c7-47e2-ae87-add4aace3f38) b= log_entry=obj93 op=0 op_state=1 2023-05-31T20:54:52.284 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:54:52.278+0000 7f259334b780 5 RGW-SYNC:bucket[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24<-pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:full_sync[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:entry[obj93]: bucket sync: sync obj: 3d163b1e-09c7-47e2-ae87-add4aace3f38/:pklxkd-59[3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24])/obj93[0] 2023-05-31T20:54:52.284 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:54:52.278+0000 7f259334b780 10 RGW-SYNC:bucket[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24<-pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:full_sync[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:entry[obj185]: success 2023-05-31T20:54:52.284 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:54:52.278+0000 7f259334b780 10 RGW-SYNC:bucket[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24<-pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:full_sync[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:entry[obj145]: success 2023-05-31T20:54:52.285 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:54:52.278+0000 7f259334b780 20 RGW-SYNC:bucket[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24<-pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:full_sync[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]: [full sync] syncing object: pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0/obj95 2023-05-31T20:54:52.285 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:54:52.278+0000 7f259334b780 20 RGW-SYNC:bucket[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24<-pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:full_sync[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:entry[obj95]: bucket sync single entry (source_zone=3d163b1e-09c7-47e2-ae87-add4aace3f38) b=bucket sync single entry (source_zone=3d163b1e-09c7-47e2-ae87-add4aace3f38) b= log_entry=obj95 op=0 op_state=1 2023-05-31T20:54:52.285 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:54:52.278+0000 7f259334b780 5 RGW-SYNC:bucket[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24<-pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:full_sync[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:entry[obj95]: bucket sync: sync obj: 3d163b1e-09c7-47e2-ae87-add4aace3f38/:pklxkd-59[3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24])/obj95[0] 2023-05-31T20:54:52.285 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:54:52.278+0000 7f259334b780 10 RGW-SYNC:bucket[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24<-pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:full_sync[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:entry[obj23]: success 2023-05-31T20:54:52.285 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:54:52.278+0000 7f259334b780 10 RGW-SYNC:bucket[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24<-pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:full_sync[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:entry[obj46]: success 2023-05-31T20:54:52.286 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:54:52.279+0000 7f259334b780 10 RGW-SYNC:bucket[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24<-pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:full_sync[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:entry[obj38]: success 2023-05-31T20:54:52.286 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:54:52.281+0000 7f259334b780 10 RGW-SYNC:bucket[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24<-pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:full_sync[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:entry[obj33]: success 2023-05-31T20:54:52.288 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:54:52.284+0000 7f259334b780 10 RGW-SYNC:bucket[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24<-pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:full_sync[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:entry[obj17]: success 2023-05-31T20:54:52.288 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:54:52.284+0000 7f259334b780 10 RGW-SYNC:bucket[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24<-pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:full_sync[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:entry[obj116]: success 2023-05-31T20:54:52.303 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:54:52.298+0000 7f259334b780 10 RGW-SYNC:bucket[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24<-pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:full_sync[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:entry[obj57]: success 2023-05-31T20:54:52.306 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:54:52.301+0000 7f259334b780 10 RGW-SYNC:bucket[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24<-pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:full_sync[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:entry[obj62]: success 2023-05-31T20:54:52.306 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:54:52.301+0000 7f259334b780 10 RGW-SYNC:bucket[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24<-pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:full_sync[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:entry[obj51]: success 2023-05-31T20:54:52.306 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:54:52.302+0000 7f259334b780 10 RGW-SYNC:bucket[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24<-pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:full_sync[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:entry[obj55]: success 2023-05-31T20:54:52.307 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:54:52.302+0000 7f259334b780 10 RGW-SYNC:bucket[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24<-pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:full_sync[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:entry[obj58]: success 2023-05-31T20:54:52.322 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:54:52.317+0000 7f259334b780 10 RGW-SYNC:bucket[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24<-pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:full_sync[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:entry[obj68]: success 2023-05-31T20:54:52.325 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:54:52.320+0000 7f259334b780 10 RGW-SYNC:bucket[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24<-pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:full_sync[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:entry[obj69]: success 2023-05-31T20:54:52.326 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:54:52.321+0000 7f259334b780 10 RGW-SYNC:bucket[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24<-pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:full_sync[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:entry[obj74]: success 2023-05-31T20:54:52.327 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:54:52.322+0000 7f259334b780 10 RGW-SYNC:bucket[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24<-pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:full_sync[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:entry[obj76]: success 2023-05-31T20:54:52.328 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:54:52.323+0000 7f259334b780 10 RGW-SYNC:bucket[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24<-pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:full_sync[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:entry[obj87]: success 2023-05-31T20:54:52.329 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:54:52.325+0000 7f259334b780 10 RGW-SYNC:bucket[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24<-pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:full_sync[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:entry[obj92]: success 2023-05-31T20:54:52.334 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:54:52.329+0000 7f259334b780 10 RGW-SYNC:bucket[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24<-pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:full_sync[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:entry[obj85]: success 2023-05-31T20:54:52.334 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:54:52.329+0000 7f259334b780 10 RGW-SYNC:bucket[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24<-pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:full_sync[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:entry[obj93]: success 2023-05-31T20:54:52.334 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:54:52.329+0000 7f259334b780 10 RGW-SYNC:bucket[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24<-pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:full_sync[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:entry[obj95]: success 2023-05-31T20:54:52.334 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:54:52.329+0000 7f259334b780 20 RGW-SYNC:bucket[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24<-pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:full_sync[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]: updating marker oid=bucket.full-sync-status.3d163b1e-09c7-47e2-ae87-add4aace3f38:pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24 marker=obj11 2023-05-31T20:54:52.336 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:54:52.331+0000 7f259334b780 10 RGW-SYNC:bucket[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24<-pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:full_sync[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]: backing out with sync_status=-5 2023-05-31T20:54:52.336 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:54:52.332+0000 7f259334b780 20 RGW-SYNC:bucket[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24<-pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]: ERROR: full sync failed. error: -5 2023-05-31T20:54:52.342 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:54:52.337+0000 7f259334b780 20 RGW-SYNC:bucket[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24<-pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]: sync status for source bucket: full. lease is: not taken. stop indications is: 0 2023-05-31T20:54:52.342 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:54:52.337+0000 7f259334b780 5 RGW-SYNC:bucket[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24<-pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]: waiting on bucket lease 2023-05-31T20:54:52.346 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:54:52.342+0000 7f259334b780 20 RGW-SYNC:bucket[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24<-pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]: status after acquiring the lock is: full 2023-05-31T20:54:52.364 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:54:52.359+0000 7f259334b780 20 RGW-SYNC:bucket[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24<-pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:full_sync[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]: listing bucket for full sync 2023-05-31T20:54:52.411 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:54:52.405+0000 7f259334b780 20 RGW-SYNC:bucket[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24<-pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:full_sync[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]: [full sync] syncing object: pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0/obj11 2023-05-31T20:54:52.411 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:54:52.405+0000 7f259334b780 20 RGW-SYNC:bucket[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24<-pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:full_sync[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:entry[obj11]: bucket sync single entry (source_zone=3d163b1e-09c7-47e2-ae87-add4aace3f38) b=bucket sync single entry (source_zone=3d163b1e-09c7-47e2-ae87-add4aace3f38) b= log_entry=obj11 op=0 op_state=1 2023-05-31T20:54:52.411 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:54:52.405+0000 7f259334b780 5 RGW-SYNC:bucket[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24<-pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:full_sync[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:entry[obj11]: bucket sync: sync obj: 3d163b1e-09c7-47e2-ae87-add4aace3f38/:pklxkd-59[3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24])/obj11[0] 2023-05-31T20:54:52.411 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:54:52.405+0000 7f259334b780 20 RGW-SYNC:bucket[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24<-pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:full_sync[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]: [full sync] syncing object: pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0/obj111 2023-05-31T20:54:52.412 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:54:52.405+0000 7f259334b780 20 RGW-SYNC:bucket[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24<-pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:full_sync[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:entry[obj111]: bucket sync single entry (source_zone=3d163b1e-09c7-47e2-ae87-add4aace3f38) b=bucket sync single entry (source_zone=3d163b1e-09c7-47e2-ae87-add4aace3f38) b= log_entry=obj111 op=0 op_state=1 2023-05-31T20:54:52.412 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:54:52.405+0000 7f259334b780 5 RGW-SYNC:bucket[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24<-pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:full_sync[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:entry[obj111]: bucket sync: sync obj: 3d163b1e-09c7-47e2-ae87-add4aace3f38/:pklxkd-59[3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24])/obj111[0] 2023-05-31T20:54:52.412 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:54:52.405+0000 7f259334b780 20 RGW-SYNC:bucket[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24<-pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:full_sync[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]: [full sync] syncing object: pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0/obj115 2023-05-31T20:54:52.412 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:54:52.405+0000 7f259334b780 20 RGW-SYNC:bucket[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24<-pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:full_sync[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:entry[obj115]: bucket sync single entry (source_zone=3d163b1e-09c7-47e2-ae87-add4aace3f38) b=bucket sync single entry (source_zone=3d163b1e-09c7-47e2-ae87-add4aace3f38) b= log_entry=obj115 op=0 op_state=1 2023-05-31T20:54:52.412 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:54:52.405+0000 7f259334b780 5 RGW-SYNC:bucket[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24<-pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:full_sync[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:entry[obj115]: bucket sync: sync obj: 3d163b1e-09c7-47e2-ae87-add4aace3f38/:pklxkd-59[3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24])/obj115[0] 2023-05-31T20:54:52.412 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:54:52.405+0000 7f259334b780 20 RGW-SYNC:bucket[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24<-pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:full_sync[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]: [full sync] syncing object: pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0/obj116 2023-05-31T20:54:52.413 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:54:52.405+0000 7f259334b780 20 RGW-SYNC:bucket[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24<-pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:full_sync[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:entry[obj116]: bucket sync single entry (source_zone=3d163b1e-09c7-47e2-ae87-add4aace3f38) b=bucket sync single entry (source_zone=3d163b1e-09c7-47e2-ae87-add4aace3f38) b= log_entry=obj116 op=0 op_state=1 2023-05-31T20:54:52.413 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:54:52.405+0000 7f259334b780 5 RGW-SYNC:bucket[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24<-pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:full_sync[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:entry[obj116]: bucket sync: sync obj: 3d163b1e-09c7-47e2-ae87-add4aace3f38/:pklxkd-59[3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24])/obj116[0] 2023-05-31T20:54:52.413 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:54:52.405+0000 7f259334b780 20 RGW-SYNC:bucket[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24<-pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:full_sync[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]: [full sync] syncing object: pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0/obj124 2023-05-31T20:54:52.413 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:54:52.405+0000 7f259334b780 20 RGW-SYNC:bucket[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24<-pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:full_sync[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:entry[obj124]: bucket sync single entry (source_zone=3d163b1e-09c7-47e2-ae87-add4aace3f38) b=bucket sync single entry (source_zone=3d163b1e-09c7-47e2-ae87-add4aace3f38) b= log_entry=obj124 op=0 op_state=1 2023-05-31T20:54:52.413 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:54:52.405+0000 7f259334b780 5 RGW-SYNC:bucket[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24<-pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:full_sync[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:entry[obj124]: bucket sync: sync obj: 3d163b1e-09c7-47e2-ae87-add4aace3f38/:pklxkd-59[3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24])/obj124[0] 2023-05-31T20:54:52.413 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:54:52.405+0000 7f259334b780 20 RGW-SYNC:bucket[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24<-pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:full_sync[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]: [full sync] syncing object: pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0/obj145 2023-05-31T20:54:52.414 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:54:52.405+0000 7f259334b780 20 RGW-SYNC:bucket[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24<-pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:full_sync[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:entry[obj145]: bucket sync single entry (source_zone=3d163b1e-09c7-47e2-ae87-add4aace3f38) b=bucket sync single entry (source_zone=3d163b1e-09c7-47e2-ae87-add4aace3f38) b= log_entry=obj145 op=0 op_state=1 2023-05-31T20:54:52.414 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:54:52.405+0000 7f259334b780 5 RGW-SYNC:bucket[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24<-pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:full_sync[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:entry[obj145]: bucket sync: sync obj: 3d163b1e-09c7-47e2-ae87-add4aace3f38/:pklxkd-59[3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24])/obj145[0] 2023-05-31T20:54:52.414 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:54:52.405+0000 7f259334b780 20 RGW-SYNC:bucket[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24<-pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:full_sync[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]: [full sync] syncing object: pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0/obj148 2023-05-31T20:54:52.414 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:54:52.405+0000 7f259334b780 20 RGW-SYNC:bucket[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24<-pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:full_sync[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:entry[obj148]: bucket sync single entry (source_zone=3d163b1e-09c7-47e2-ae87-add4aace3f38) b=bucket sync single entry (source_zone=3d163b1e-09c7-47e2-ae87-add4aace3f38) b= log_entry=obj148 op=0 op_state=1 2023-05-31T20:54:52.414 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:54:52.405+0000 7f259334b780 5 RGW-SYNC:bucket[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24<-pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:full_sync[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:entry[obj148]: bucket sync: sync obj: 3d163b1e-09c7-47e2-ae87-add4aace3f38/:pklxkd-59[3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24])/obj148[0] 2023-05-31T20:54:52.414 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:54:52.405+0000 7f259334b780 20 RGW-SYNC:bucket[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24<-pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:full_sync[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]: [full sync] syncing object: pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0/obj155 2023-05-31T20:54:52.414 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:54:52.405+0000 7f259334b780 20 RGW-SYNC:bucket[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24<-pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:full_sync[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:entry[obj155]: bucket sync single entry (source_zone=3d163b1e-09c7-47e2-ae87-add4aace3f38) b=bucket sync single entry (source_zone=3d163b1e-09c7-47e2-ae87-add4aace3f38) b= log_entry=obj155 op=0 op_state=1 2023-05-31T20:54:52.415 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:54:52.405+0000 7f259334b780 5 RGW-SYNC:bucket[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24<-pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:full_sync[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:entry[obj155]: bucket sync: sync obj: 3d163b1e-09c7-47e2-ae87-add4aace3f38/:pklxkd-59[3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24])/obj155[0] 2023-05-31T20:54:52.415 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:54:52.405+0000 7f259334b780 20 RGW-SYNC:bucket[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24<-pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:full_sync[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]: [full sync] syncing object: pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0/obj17 2023-05-31T20:54:52.415 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:54:52.405+0000 7f259334b780 20 RGW-SYNC:bucket[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24<-pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:full_sync[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:entry[obj17]: bucket sync single entry (source_zone=3d163b1e-09c7-47e2-ae87-add4aace3f38) b=bucket sync single entry (source_zone=3d163b1e-09c7-47e2-ae87-add4aace3f38) b= log_entry=obj17 op=0 op_state=1 2023-05-31T20:54:52.415 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:54:52.405+0000 7f259334b780 5 RGW-SYNC:bucket[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24<-pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:full_sync[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:entry[obj17]: bucket sync: sync obj: 3d163b1e-09c7-47e2-ae87-add4aace3f38/:pklxkd-59[3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24])/obj17[0] 2023-05-31T20:54:52.415 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:54:52.405+0000 7f259334b780 20 RGW-SYNC:bucket[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24<-pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:full_sync[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]: [full sync] syncing object: pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0/obj185 2023-05-31T20:54:52.415 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:54:52.405+0000 7f259334b780 20 RGW-SYNC:bucket[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24<-pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:full_sync[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:entry[obj185]: bucket sync single entry (source_zone=3d163b1e-09c7-47e2-ae87-add4aace3f38) b=bucket sync single entry (source_zone=3d163b1e-09c7-47e2-ae87-add4aace3f38) b= log_entry=obj185 op=0 op_state=1 2023-05-31T20:54:52.416 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:54:52.405+0000 7f259334b780 5 RGW-SYNC:bucket[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24<-pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:full_sync[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:entry[obj185]: bucket sync: sync obj: 3d163b1e-09c7-47e2-ae87-add4aace3f38/:pklxkd-59[3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24])/obj185[0] 2023-05-31T20:54:52.416 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:54:52.405+0000 7f259334b780 20 RGW-SYNC:bucket[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24<-pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:full_sync[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]: [full sync] syncing object: pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0/obj19 2023-05-31T20:54:52.416 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:54:52.405+0000 7f259334b780 20 RGW-SYNC:bucket[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24<-pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:full_sync[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:entry[obj19]: bucket sync single entry (source_zone=3d163b1e-09c7-47e2-ae87-add4aace3f38) b=bucket sync single entry (source_zone=3d163b1e-09c7-47e2-ae87-add4aace3f38) b= log_entry=obj19 op=0 op_state=1 2023-05-31T20:54:52.416 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:54:52.405+0000 7f259334b780 5 RGW-SYNC:bucket[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24<-pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:full_sync[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:entry[obj19]: bucket sync: sync obj: 3d163b1e-09c7-47e2-ae87-add4aace3f38/:pklxkd-59[3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24])/obj19[0] 2023-05-31T20:54:52.416 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:54:52.405+0000 7f259334b780 20 RGW-SYNC:bucket[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24<-pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:full_sync[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]: [full sync] syncing object: pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0/obj22 2023-05-31T20:54:52.416 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:54:52.405+0000 7f259334b780 20 RGW-SYNC:bucket[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24<-pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:full_sync[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:entry[obj22]: bucket sync single entry (source_zone=3d163b1e-09c7-47e2-ae87-add4aace3f38) b=bucket sync single entry (source_zone=3d163b1e-09c7-47e2-ae87-add4aace3f38) b= log_entry=obj22 op=0 op_state=1 2023-05-31T20:54:52.416 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:54:52.405+0000 7f259334b780 5 RGW-SYNC:bucket[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24<-pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:full_sync[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:entry[obj22]: bucket sync: sync obj: 3d163b1e-09c7-47e2-ae87-add4aace3f38/:pklxkd-59[3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24])/obj22[0] 2023-05-31T20:54:52.417 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:54:52.405+0000 7f259334b780 20 RGW-SYNC:bucket[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24<-pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:full_sync[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]: [full sync] syncing object: pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0/obj23 2023-05-31T20:54:52.417 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:54:52.405+0000 7f259334b780 20 RGW-SYNC:bucket[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24<-pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:full_sync[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:entry[obj23]: bucket sync single entry (source_zone=3d163b1e-09c7-47e2-ae87-add4aace3f38) b=bucket sync single entry (source_zone=3d163b1e-09c7-47e2-ae87-add4aace3f38) b= log_entry=obj23 op=0 op_state=1 2023-05-31T20:54:52.417 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:54:52.405+0000 7f259334b780 5 RGW-SYNC:bucket[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24<-pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:full_sync[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:entry[obj23]: bucket sync: sync obj: 3d163b1e-09c7-47e2-ae87-add4aace3f38/:pklxkd-59[3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24])/obj23[0] 2023-05-31T20:54:52.417 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:54:52.406+0000 7f259334b780 20 RGW-SYNC:bucket[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24<-pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:full_sync[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]: [full sync] syncing object: pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0/obj29 2023-05-31T20:54:52.417 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:54:52.406+0000 7f259334b780 20 RGW-SYNC:bucket[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24<-pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:full_sync[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:entry[obj29]: bucket sync single entry (source_zone=3d163b1e-09c7-47e2-ae87-add4aace3f38) b=bucket sync single entry (source_zone=3d163b1e-09c7-47e2-ae87-add4aace3f38) b= log_entry=obj29 op=0 op_state=1 2023-05-31T20:54:52.417 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:54:52.406+0000 7f259334b780 5 RGW-SYNC:bucket[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24<-pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:full_sync[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:entry[obj29]: bucket sync: sync obj: 3d163b1e-09c7-47e2-ae87-add4aace3f38/:pklxkd-59[3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24])/obj29[0] 2023-05-31T20:54:52.418 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:54:52.406+0000 7f259334b780 20 RGW-SYNC:bucket[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24<-pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:full_sync[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]: [full sync] syncing object: pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0/obj31 2023-05-31T20:54:52.418 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:54:52.406+0000 7f259334b780 20 RGW-SYNC:bucket[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24<-pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:full_sync[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:entry[obj31]: bucket sync single entry (source_zone=3d163b1e-09c7-47e2-ae87-add4aace3f38) b=bucket sync single entry (source_zone=3d163b1e-09c7-47e2-ae87-add4aace3f38) b= log_entry=obj31 op=0 op_state=1 2023-05-31T20:54:52.418 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:54:52.406+0000 7f259334b780 5 RGW-SYNC:bucket[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24<-pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:full_sync[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:entry[obj31]: bucket sync: sync obj: 3d163b1e-09c7-47e2-ae87-add4aace3f38/:pklxkd-59[3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24])/obj31[0] 2023-05-31T20:54:52.418 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:54:52.406+0000 7f259334b780 20 RGW-SYNC:bucket[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24<-pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:full_sync[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]: [full sync] syncing object: pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0/obj33 2023-05-31T20:54:52.418 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:54:52.406+0000 7f259334b780 20 RGW-SYNC:bucket[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24<-pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:full_sync[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:entry[obj33]: bucket sync single entry (source_zone=3d163b1e-09c7-47e2-ae87-add4aace3f38) b=bucket sync single entry (source_zone=3d163b1e-09c7-47e2-ae87-add4aace3f38) b= log_entry=obj33 op=0 op_state=1 2023-05-31T20:54:52.418 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:54:52.406+0000 7f259334b780 5 RGW-SYNC:bucket[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24<-pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:full_sync[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:entry[obj33]: bucket sync: sync obj: 3d163b1e-09c7-47e2-ae87-add4aace3f38/:pklxkd-59[3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24])/obj33[0] 2023-05-31T20:54:52.418 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:54:52.406+0000 7f259334b780 20 RGW-SYNC:bucket[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24<-pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:full_sync[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]: [full sync] syncing object: pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0/obj34 2023-05-31T20:54:52.419 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:54:52.406+0000 7f259334b780 20 RGW-SYNC:bucket[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24<-pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:full_sync[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:entry[obj34]: bucket sync single entry (source_zone=3d163b1e-09c7-47e2-ae87-add4aace3f38) b=bucket sync single entry (source_zone=3d163b1e-09c7-47e2-ae87-add4aace3f38) b= log_entry=obj34 op=0 op_state=1 2023-05-31T20:54:52.419 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:54:52.406+0000 7f259334b780 5 RGW-SYNC:bucket[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24<-pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:full_sync[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:entry[obj34]: bucket sync: sync obj: 3d163b1e-09c7-47e2-ae87-add4aace3f38/:pklxkd-59[3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24])/obj34[0] 2023-05-31T20:54:52.419 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:54:52.406+0000 7f259334b780 20 RGW-SYNC:bucket[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24<-pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:full_sync[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]: [full sync] syncing object: pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0/obj37 2023-05-31T20:54:52.419 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:54:52.406+0000 7f259334b780 20 RGW-SYNC:bucket[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24<-pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:full_sync[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:entry[obj37]: bucket sync single entry (source_zone=3d163b1e-09c7-47e2-ae87-add4aace3f38) b=bucket sync single entry (source_zone=3d163b1e-09c7-47e2-ae87-add4aace3f38) b= log_entry=obj37 op=0 op_state=1 2023-05-31T20:54:52.419 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:54:52.406+0000 7f259334b780 5 RGW-SYNC:bucket[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24<-pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:full_sync[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:entry[obj37]: bucket sync: sync obj: 3d163b1e-09c7-47e2-ae87-add4aace3f38/:pklxkd-59[3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24])/obj37[0] 2023-05-31T20:54:52.419 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:54:52.406+0000 7f259334b780 20 RGW-SYNC:bucket[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24<-pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:full_sync[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]: [full sync] syncing object: pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0/obj38 2023-05-31T20:54:52.419 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:54:52.406+0000 7f259334b780 20 RGW-SYNC:bucket[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24<-pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:full_sync[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:entry[obj38]: bucket sync single entry (source_zone=3d163b1e-09c7-47e2-ae87-add4aace3f38) b=bucket sync single entry (source_zone=3d163b1e-09c7-47e2-ae87-add4aace3f38) b= log_entry=obj38 op=0 op_state=1 2023-05-31T20:54:52.420 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:54:52.406+0000 7f259334b780 5 RGW-SYNC:bucket[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24<-pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:full_sync[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:entry[obj38]: bucket sync: sync obj: 3d163b1e-09c7-47e2-ae87-add4aace3f38/:pklxkd-59[3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24])/obj38[0] 2023-05-31T20:54:52.420 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:54:52.406+0000 7f259334b780 20 RGW-SYNC:bucket[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24<-pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:full_sync[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]: [full sync] syncing object: pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0/obj44 2023-05-31T20:54:52.420 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:54:52.406+0000 7f259334b780 20 RGW-SYNC:bucket[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24<-pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:full_sync[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:entry[obj44]: bucket sync single entry (source_zone=3d163b1e-09c7-47e2-ae87-add4aace3f38) b=bucket sync single entry (source_zone=3d163b1e-09c7-47e2-ae87-add4aace3f38) b= log_entry=obj44 op=0 op_state=1 2023-05-31T20:54:52.420 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:54:52.406+0000 7f259334b780 5 RGW-SYNC:bucket[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24<-pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:full_sync[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:entry[obj44]: bucket sync: sync obj: 3d163b1e-09c7-47e2-ae87-add4aace3f38/:pklxkd-59[3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24])/obj44[0] 2023-05-31T20:54:52.420 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:54:52.406+0000 7f259334b780 20 RGW-SYNC:bucket[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24<-pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:full_sync[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]: [full sync] syncing object: pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0/obj46 2023-05-31T20:54:52.420 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:54:52.406+0000 7f259334b780 20 RGW-SYNC:bucket[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24<-pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:full_sync[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:entry[obj46]: bucket sync single entry (source_zone=3d163b1e-09c7-47e2-ae87-add4aace3f38) b=bucket sync single entry (source_zone=3d163b1e-09c7-47e2-ae87-add4aace3f38) b= log_entry=obj46 op=0 op_state=1 2023-05-31T20:54:52.421 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:54:52.406+0000 7f259334b780 5 RGW-SYNC:bucket[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24<-pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:full_sync[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:entry[obj46]: bucket sync: sync obj: 3d163b1e-09c7-47e2-ae87-add4aace3f38/:pklxkd-59[3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24])/obj46[0] 2023-05-31T20:54:52.422 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:54:52.409+0000 7f259334b780 10 RGW-SYNC:bucket[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24<-pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:full_sync[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:entry[obj148]: success 2023-05-31T20:54:52.422 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:54:52.409+0000 7f259334b780 20 RGW-SYNC:bucket[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24<-pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:full_sync[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]: [full sync] syncing object: pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0/obj51 2023-05-31T20:54:52.422 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:54:52.409+0000 7f259334b780 20 RGW-SYNC:bucket[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24<-pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:full_sync[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:entry[obj51]: bucket sync single entry (source_zone=3d163b1e-09c7-47e2-ae87-add4aace3f38) b=bucket sync single entry (source_zone=3d163b1e-09c7-47e2-ae87-add4aace3f38) b= log_entry=obj51 op=0 op_state=1 2023-05-31T20:54:52.423 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:54:52.409+0000 7f259334b780 5 RGW-SYNC:bucket[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24<-pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:full_sync[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:entry[obj51]: bucket sync: sync obj: 3d163b1e-09c7-47e2-ae87-add4aace3f38/:pklxkd-59[3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24])/obj51[0] 2023-05-31T20:54:52.423 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:54:52.410+0000 7f259334b780 10 RGW-SYNC:bucket[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24<-pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:full_sync[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:entry[obj17]: success 2023-05-31T20:54:52.423 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:54:52.410+0000 7f259334b780 20 RGW-SYNC:bucket[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24<-pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:full_sync[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]: [full sync] syncing object: pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0/obj55 2023-05-31T20:54:52.423 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:54:52.410+0000 7f259334b780 20 RGW-SYNC:bucket[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24<-pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:full_sync[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:entry[obj55]: bucket sync single entry (source_zone=3d163b1e-09c7-47e2-ae87-add4aace3f38) b=bucket sync single entry (source_zone=3d163b1e-09c7-47e2-ae87-add4aace3f38) b= log_entry=obj55 op=0 op_state=1 2023-05-31T20:54:52.423 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:54:52.410+0000 7f259334b780 5 RGW-SYNC:bucket[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24<-pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:full_sync[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:entry[obj55]: bucket sync: sync obj: 3d163b1e-09c7-47e2-ae87-add4aace3f38/:pklxkd-59[3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24])/obj55[0] 2023-05-31T20:54:52.423 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:54:52.411+0000 7f259334b780 10 RGW-SYNC:bucket[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24<-pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:full_sync[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:entry[obj124]: success 2023-05-31T20:54:52.424 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:54:52.411+0000 7f259334b780 10 RGW-SYNC:bucket[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24<-pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:full_sync[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:entry[obj23]: success 2023-05-31T20:54:52.424 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:54:52.411+0000 7f259334b780 10 RGW-SYNC:bucket[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24<-pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:full_sync[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:entry[obj115]: success 2023-05-31T20:54:52.424 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:54:52.411+0000 7f259334b780 10 RGW-SYNC:bucket[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24<-pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:full_sync[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:entry[obj11]: success 2023-05-31T20:54:52.424 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:54:52.411+0000 7f259334b780 10 RGW-SYNC:bucket[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24<-pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:full_sync[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:entry[obj31]: success 2023-05-31T20:54:52.424 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:54:52.411+0000 7f259334b780 10 RGW-SYNC:bucket[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24<-pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:full_sync[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:entry[obj185]: success 2023-05-31T20:54:52.424 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:54:52.411+0000 7f259334b780 20 RGW-SYNC:bucket[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24<-pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:full_sync[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]: [full sync] syncing object: pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0/obj57 2023-05-31T20:54:52.424 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:54:52.411+0000 7f259334b780 20 RGW-SYNC:bucket[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24<-pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:full_sync[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:entry[obj57]: bucket sync single entry (source_zone=3d163b1e-09c7-47e2-ae87-add4aace3f38) b=bucket sync single entry (source_zone=3d163b1e-09c7-47e2-ae87-add4aace3f38) b= log_entry=obj57 op=0 op_state=1 2023-05-31T20:54:52.425 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:54:52.411+0000 7f259334b780 5 RGW-SYNC:bucket[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24<-pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:full_sync[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:entry[obj57]: bucket sync: sync obj: 3d163b1e-09c7-47e2-ae87-add4aace3f38/:pklxkd-59[3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24])/obj57[0] 2023-05-31T20:54:52.425 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:54:52.411+0000 7f259334b780 20 RGW-SYNC:bucket[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24<-pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:full_sync[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]: [full sync] syncing object: pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0/obj58 2023-05-31T20:54:52.425 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:54:52.411+0000 7f259334b780 20 RGW-SYNC:bucket[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24<-pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:full_sync[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:entry[obj58]: bucket sync single entry (source_zone=3d163b1e-09c7-47e2-ae87-add4aace3f38) b=bucket sync single entry (source_zone=3d163b1e-09c7-47e2-ae87-add4aace3f38) b= log_entry=obj58 op=0 op_state=1 2023-05-31T20:54:52.425 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:54:52.411+0000 7f259334b780 5 RGW-SYNC:bucket[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24<-pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:full_sync[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:entry[obj58]: bucket sync: sync obj: 3d163b1e-09c7-47e2-ae87-add4aace3f38/:pklxkd-59[3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24])/obj58[0] 2023-05-31T20:54:52.425 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:54:52.411+0000 7f259334b780 20 RGW-SYNC:bucket[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24<-pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:full_sync[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]: [full sync] syncing object: pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0/obj62 2023-05-31T20:54:52.425 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:54:52.411+0000 7f259334b780 20 RGW-SYNC:bucket[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24<-pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:full_sync[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:entry[obj62]: bucket sync single entry (source_zone=3d163b1e-09c7-47e2-ae87-add4aace3f38) b=bucket sync single entry (source_zone=3d163b1e-09c7-47e2-ae87-add4aace3f38) b= log_entry=obj62 op=0 op_state=1 2023-05-31T20:54:52.426 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:54:52.411+0000 7f259334b780 5 RGW-SYNC:bucket[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24<-pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:full_sync[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:entry[obj62]: bucket sync: sync obj: 3d163b1e-09c7-47e2-ae87-add4aace3f38/:pklxkd-59[3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24])/obj62[0] 2023-05-31T20:54:52.426 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:54:52.411+0000 7f259334b780 20 RGW-SYNC:bucket[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24<-pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:full_sync[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]: [full sync] syncing object: pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0/obj68 2023-05-31T20:54:52.426 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:54:52.411+0000 7f259334b780 20 RGW-SYNC:bucket[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24<-pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:full_sync[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:entry[obj68]: bucket sync single entry (source_zone=3d163b1e-09c7-47e2-ae87-add4aace3f38) b=bucket sync single entry (source_zone=3d163b1e-09c7-47e2-ae87-add4aace3f38) b= log_entry=obj68 op=0 op_state=1 2023-05-31T20:54:52.426 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:54:52.411+0000 7f259334b780 0 RGW-SYNC:bucket[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24<-pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:full_sync[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:entry[obj68]: : injecting data sync error on key=obj68 2023-05-31T20:54:52.426 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:54:52.411+0000 7f259334b780 10 RGW-SYNC:bucket[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24<-pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:full_sync[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:entry[obj68]: failed, retcode=-5 ((5) Input/output error) 2023-05-31T20:54:52.426 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:54:52.411+0000 7f259334b780 0 RGW-SYNC:bucket[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24<-pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:full_sync[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:entry[obj68]: ERROR: failed to sync object: pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0/obj68 2023-05-31T20:54:52.426 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:54:52.411+0000 7f259334b780 20 RGW-SYNC:bucket[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24<-pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:full_sync[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]: [full sync] syncing object: pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0/obj69 2023-05-31T20:54:52.427 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:54:52.411+0000 7f259334b780 20 RGW-SYNC:bucket[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24<-pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:full_sync[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:entry[obj69]: bucket sync single entry (source_zone=3d163b1e-09c7-47e2-ae87-add4aace3f38) b=bucket sync single entry (source_zone=3d163b1e-09c7-47e2-ae87-add4aace3f38) b= log_entry=obj69 op=0 op_state=1 2023-05-31T20:54:52.427 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:54:52.411+0000 7f259334b780 5 RGW-SYNC:bucket[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24<-pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:full_sync[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:entry[obj69]: bucket sync: sync obj: 3d163b1e-09c7-47e2-ae87-add4aace3f38/:pklxkd-59[3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24])/obj69[0] 2023-05-31T20:54:52.427 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:54:52.411+0000 7f259334b780 20 RGW-SYNC:bucket[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24<-pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:full_sync[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]: [full sync] syncing object: pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0/obj74 2023-05-31T20:54:52.427 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:54:52.411+0000 7f259334b780 20 RGW-SYNC:bucket[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24<-pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:full_sync[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:entry[obj74]: bucket sync single entry (source_zone=3d163b1e-09c7-47e2-ae87-add4aace3f38) b=bucket sync single entry (source_zone=3d163b1e-09c7-47e2-ae87-add4aace3f38) b= log_entry=obj74 op=0 op_state=1 2023-05-31T20:54:52.427 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:54:52.411+0000 7f259334b780 5 RGW-SYNC:bucket[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24<-pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:full_sync[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:entry[obj74]: bucket sync: sync obj: 3d163b1e-09c7-47e2-ae87-add4aace3f38/:pklxkd-59[3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24])/obj74[0] 2023-05-31T20:54:52.427 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:54:52.412+0000 7f259334b780 10 RGW-SYNC:bucket[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24<-pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:full_sync[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:entry[obj116]: success 2023-05-31T20:54:52.428 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:54:52.412+0000 7f259334b780 10 RGW-SYNC:bucket[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24<-pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:full_sync[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:entry[obj46]: success 2023-05-31T20:54:52.428 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:54:52.412+0000 7f259334b780 10 RGW-SYNC:bucket[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24<-pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:full_sync[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:entry[obj44]: success 2023-05-31T20:54:52.428 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:54:52.412+0000 7f259334b780 10 RGW-SYNC:bucket[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24<-pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:full_sync[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:entry[obj37]: success 2023-05-31T20:54:52.428 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:54:52.412+0000 7f259334b780 10 RGW-SYNC:bucket[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24<-pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:full_sync[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:entry[obj19]: success 2023-05-31T20:54:52.428 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:54:52.412+0000 7f259334b780 10 RGW-SYNC:bucket[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24<-pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:full_sync[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:entry[obj145]: success 2023-05-31T20:54:52.428 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:54:52.412+0000 7f259334b780 20 RGW-SYNC:bucket[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24<-pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:full_sync[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]: [full sync] syncing object: pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0/obj76 2023-05-31T20:54:52.428 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:54:52.412+0000 7f259334b780 20 RGW-SYNC:bucket[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24<-pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:full_sync[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:entry[obj76]: bucket sync single entry (source_zone=3d163b1e-09c7-47e2-ae87-add4aace3f38) b=bucket sync single entry (source_zone=3d163b1e-09c7-47e2-ae87-add4aace3f38) b= log_entry=obj76 op=0 op_state=1 2023-05-31T20:54:52.429 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:54:52.412+0000 7f259334b780 0 RGW-SYNC:bucket[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24<-pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:full_sync[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:entry[obj76]: : injecting data sync error on key=obj76 2023-05-31T20:54:52.429 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:54:52.412+0000 7f259334b780 10 RGW-SYNC:bucket[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24<-pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:full_sync[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:entry[obj76]: failed, retcode=-5 ((5) Input/output error) 2023-05-31T20:54:52.429 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:54:52.412+0000 7f259334b780 0 RGW-SYNC:bucket[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24<-pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:full_sync[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:entry[obj76]: ERROR: failed to sync object: pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0/obj76 2023-05-31T20:54:52.429 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:54:52.412+0000 7f259334b780 20 RGW-SYNC:bucket[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24<-pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:full_sync[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]: [full sync] syncing object: pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0/obj85 2023-05-31T20:54:52.429 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:54:52.412+0000 7f259334b780 20 RGW-SYNC:bucket[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24<-pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:full_sync[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:entry[obj85]: bucket sync single entry (source_zone=3d163b1e-09c7-47e2-ae87-add4aace3f38) b=bucket sync single entry (source_zone=3d163b1e-09c7-47e2-ae87-add4aace3f38) b= log_entry=obj85 op=0 op_state=1 2023-05-31T20:54:52.429 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:54:52.412+0000 7f259334b780 10 RGW-SYNC:bucket[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24<-pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:full_sync[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:entry[obj38]: success 2023-05-31T20:54:52.429 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:54:52.412+0000 7f259334b780 5 RGW-SYNC:bucket[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24<-pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:full_sync[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:entry[obj85]: bucket sync: sync obj: 3d163b1e-09c7-47e2-ae87-add4aace3f38/:pklxkd-59[3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24])/obj85[0] 2023-05-31T20:54:52.430 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:54:52.412+0000 7f259334b780 20 RGW-SYNC:bucket[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24<-pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:full_sync[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]: [full sync] syncing object: pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0/obj87 2023-05-31T20:54:52.430 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:54:52.412+0000 7f259334b780 20 RGW-SYNC:bucket[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24<-pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:full_sync[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:entry[obj87]: bucket sync single entry (source_zone=3d163b1e-09c7-47e2-ae87-add4aace3f38) b=bucket sync single entry (source_zone=3d163b1e-09c7-47e2-ae87-add4aace3f38) b= log_entry=obj87 op=0 op_state=1 2023-05-31T20:54:52.430 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:54:52.412+0000 7f259334b780 0 RGW-SYNC:bucket[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24<-pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:full_sync[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:entry[obj87]: : injecting data sync error on key=obj87 2023-05-31T20:54:52.430 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:54:52.412+0000 7f259334b780 10 RGW-SYNC:bucket[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24<-pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:full_sync[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:entry[obj87]: failed, retcode=-5 ((5) Input/output error) 2023-05-31T20:54:52.430 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:54:52.412+0000 7f259334b780 0 RGW-SYNC:bucket[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24<-pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:full_sync[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:entry[obj87]: ERROR: failed to sync object: pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0/obj87 2023-05-31T20:54:52.430 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:54:52.412+0000 7f259334b780 20 RGW-SYNC:bucket[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24<-pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:full_sync[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]: [full sync] syncing object: pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0/obj92 2023-05-31T20:54:52.431 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:54:52.412+0000 7f259334b780 20 RGW-SYNC:bucket[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24<-pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:full_sync[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:entry[obj92]: bucket sync single entry (source_zone=3d163b1e-09c7-47e2-ae87-add4aace3f38) b=bucket sync single entry (source_zone=3d163b1e-09c7-47e2-ae87-add4aace3f38) b= log_entry=obj92 op=0 op_state=1 2023-05-31T20:54:52.431 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:54:52.412+0000 7f259334b780 5 RGW-SYNC:bucket[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24<-pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:full_sync[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:entry[obj92]: bucket sync: sync obj: 3d163b1e-09c7-47e2-ae87-add4aace3f38/:pklxkd-59[3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24])/obj92[0] 2023-05-31T20:54:52.431 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:54:52.412+0000 7f259334b780 20 RGW-SYNC:bucket[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24<-pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:full_sync[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]: [full sync] syncing object: pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0/obj93 2023-05-31T20:54:52.431 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:54:52.412+0000 7f259334b780 20 RGW-SYNC:bucket[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24<-pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:full_sync[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:entry[obj93]: bucket sync single entry (source_zone=3d163b1e-09c7-47e2-ae87-add4aace3f38) b=bucket sync single entry (source_zone=3d163b1e-09c7-47e2-ae87-add4aace3f38) b= log_entry=obj93 op=0 op_state=1 2023-05-31T20:54:52.431 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:54:52.412+0000 7f259334b780 10 RGW-SYNC:bucket[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24<-pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:full_sync[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:entry[obj33]: success 2023-05-31T20:54:52.431 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:54:52.412+0000 7f259334b780 5 RGW-SYNC:bucket[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24<-pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:full_sync[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:entry[obj93]: bucket sync: sync obj: 3d163b1e-09c7-47e2-ae87-add4aace3f38/:pklxkd-59[3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24])/obj93[0] 2023-05-31T20:54:52.431 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:54:52.412+0000 7f259334b780 20 RGW-SYNC:bucket[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24<-pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:full_sync[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]: [full sync] syncing object: pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0/obj95 2023-05-31T20:54:52.432 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:54:52.412+0000 7f259334b780 20 RGW-SYNC:bucket[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24<-pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:full_sync[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:entry[obj95]: bucket sync single entry (source_zone=3d163b1e-09c7-47e2-ae87-add4aace3f38) b=bucket sync single entry (source_zone=3d163b1e-09c7-47e2-ae87-add4aace3f38) b= log_entry=obj95 op=0 op_state=1 2023-05-31T20:54:52.432 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:54:52.412+0000 7f259334b780 5 RGW-SYNC:bucket[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24<-pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:full_sync[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:entry[obj95]: bucket sync: sync obj: 3d163b1e-09c7-47e2-ae87-add4aace3f38/:pklxkd-59[3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24])/obj95[0] 2023-05-31T20:54:52.432 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:54:52.415+0000 7f259334b780 10 RGW-SYNC:bucket[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24<-pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:full_sync[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:entry[obj51]: success 2023-05-31T20:54:52.432 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:54:52.416+0000 7f259334b780 10 RGW-SYNC:bucket[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24<-pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:full_sync[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:entry[obj74]: success 2023-05-31T20:54:52.432 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:54:52.416+0000 7f259334b780 10 RGW-SYNC:bucket[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24<-pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:full_sync[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:entry[obj55]: success 2023-05-31T20:54:52.432 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:54:52.416+0000 7f259334b780 10 RGW-SYNC:bucket[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24<-pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:full_sync[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:entry[obj62]: success 2023-05-31T20:54:52.432 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:54:52.416+0000 7f259334b780 10 RGW-SYNC:bucket[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24<-pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:full_sync[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]: a sync operation returned error 2023-05-31T20:54:52.433 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:54:52.416+0000 7f259334b780 10 RGW-SYNC:bucket[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24<-pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:full_sync[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]: a sync operation returned error 2023-05-31T20:54:52.433 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:54:52.416+0000 7f259334b780 10 RGW-SYNC:bucket[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24<-pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:full_sync[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:entry[obj58]: success 2023-05-31T20:54:52.433 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:54:52.417+0000 7f259334b780 10 RGW-SYNC:bucket[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24<-pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:full_sync[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]: a sync operation returned error 2023-05-31T20:54:52.433 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:54:52.417+0000 7f259334b780 10 RGW-SYNC:bucket[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24<-pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:full_sync[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:entry[obj93]: success 2023-05-31T20:54:52.433 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:54:52.417+0000 7f259334b780 10 RGW-SYNC:bucket[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24<-pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:full_sync[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:entry[obj85]: success 2023-05-31T20:54:52.434 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:54:52.418+0000 7f259334b780 10 RGW-SYNC:bucket[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24<-pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:full_sync[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:entry[obj57]: success 2023-05-31T20:54:52.434 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:54:52.418+0000 7f259334b780 10 RGW-SYNC:bucket[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24<-pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:full_sync[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:entry[obj95]: success 2023-05-31T20:54:52.435 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:54:52.418+0000 7f259334b780 10 RGW-SYNC:bucket[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24<-pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:full_sync[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:entry[obj92]: success 2023-05-31T20:54:52.435 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:54:52.418+0000 7f259334b780 10 RGW-SYNC:bucket[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24<-pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:full_sync[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:entry[obj69]: success 2023-05-31T20:54:52.435 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:54:52.418+0000 7f259334b780 10 RGW-SYNC:bucket[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24<-pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:full_sync[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:entry[obj22]: success 2023-05-31T20:54:52.435 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:54:52.419+0000 7f259334b780 10 RGW-SYNC:bucket[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24<-pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:full_sync[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:entry[obj155]: success 2023-05-31T20:54:52.435 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:54:52.420+0000 7f259334b780 10 RGW-SYNC:bucket[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24<-pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:full_sync[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:entry[obj34]: success 2023-05-31T20:54:52.435 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:54:52.421+0000 7f259334b780 10 RGW-SYNC:bucket[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24<-pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:full_sync[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:entry[obj111]: success 2023-05-31T20:54:52.436 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:54:52.421+0000 7f259334b780 20 RGW-SYNC:bucket[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24<-pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:full_sync[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]: updating marker oid=bucket.full-sync-status.3d163b1e-09c7-47e2-ae87-add4aace3f38:pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24 marker=obj23 2023-05-31T20:54:52.436 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:54:52.423+0000 7f259334b780 10 RGW-SYNC:bucket[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24<-pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:full_sync[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:entry[obj29]: success 2023-05-31T20:54:52.436 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:54:52.424+0000 7f259334b780 20 RGW-SYNC:bucket[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24<-pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:full_sync[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]: updating marker oid=bucket.full-sync-status.3d163b1e-09c7-47e2-ae87-add4aace3f38:pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24 marker=obj62 2023-05-31T20:54:52.436 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:54:52.425+0000 7f259334b780 10 RGW-SYNC:bucket[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24<-pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:full_sync[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]: backing out with sync_status=-5 2023-05-31T20:54:52.436 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:54:52.425+0000 7f259334b780 20 RGW-SYNC:bucket[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24<-pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]: ERROR: full sync failed. error: -5 2023-05-31T20:54:52.436 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:54:52.426+0000 7f259334b780 20 RGW-SYNC:bucket[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24<-pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]: sync status for source bucket: full. lease is: not taken. stop indications is: 0 2023-05-31T20:54:52.436 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:54:52.426+0000 7f259334b780 5 RGW-SYNC:bucket[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24<-pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]: waiting on bucket lease 2023-05-31T20:54:52.476 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:54:52.471+0000 7f259334b780 20 RGW-SYNC:bucket[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24<-pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]: status after acquiring the lock is: full 2023-05-31T20:54:52.494 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:54:52.489+0000 7f259334b780 20 RGW-SYNC:bucket[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24<-pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:full_sync[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]: listing bucket for full sync 2023-05-31T20:54:52.541 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:54:52.534+0000 7f259334b780 20 RGW-SYNC:bucket[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24<-pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:full_sync[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]: [full sync] syncing object: pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0/obj11 2023-05-31T20:54:52.541 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:54:52.534+0000 7f259334b780 20 RGW-SYNC:bucket[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24<-pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:full_sync[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:entry[obj11]: bucket sync single entry (source_zone=3d163b1e-09c7-47e2-ae87-add4aace3f38) b=bucket sync single entry (source_zone=3d163b1e-09c7-47e2-ae87-add4aace3f38) b= log_entry=obj11 op=0 op_state=1 2023-05-31T20:54:52.541 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:54:52.534+0000 7f259334b780 5 RGW-SYNC:bucket[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24<-pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:full_sync[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:entry[obj11]: bucket sync: sync obj: 3d163b1e-09c7-47e2-ae87-add4aace3f38/:pklxkd-59[3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24])/obj11[0] 2023-05-31T20:54:52.541 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:54:52.534+0000 7f259334b780 20 RGW-SYNC:bucket[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24<-pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:full_sync[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]: [full sync] syncing object: pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0/obj111 2023-05-31T20:54:52.542 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:54:52.534+0000 7f259334b780 20 RGW-SYNC:bucket[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24<-pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:full_sync[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:entry[obj111]: bucket sync single entry (source_zone=3d163b1e-09c7-47e2-ae87-add4aace3f38) b=bucket sync single entry (source_zone=3d163b1e-09c7-47e2-ae87-add4aace3f38) b= log_entry=obj111 op=0 op_state=1 2023-05-31T20:54:52.542 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:54:52.534+0000 7f259334b780 5 RGW-SYNC:bucket[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24<-pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:full_sync[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:entry[obj111]: bucket sync: sync obj: 3d163b1e-09c7-47e2-ae87-add4aace3f38/:pklxkd-59[3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24])/obj111[0] 2023-05-31T20:54:52.542 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:54:52.534+0000 7f259334b780 20 RGW-SYNC:bucket[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24<-pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:full_sync[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]: [full sync] syncing object: pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0/obj115 2023-05-31T20:54:52.542 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:54:52.534+0000 7f259334b780 20 RGW-SYNC:bucket[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24<-pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:full_sync[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:entry[obj115]: bucket sync single entry (source_zone=3d163b1e-09c7-47e2-ae87-add4aace3f38) b=bucket sync single entry (source_zone=3d163b1e-09c7-47e2-ae87-add4aace3f38) b= log_entry=obj115 op=0 op_state=1 2023-05-31T20:54:52.543 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:54:52.534+0000 7f259334b780 5 RGW-SYNC:bucket[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24<-pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:full_sync[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:entry[obj115]: bucket sync: sync obj: 3d163b1e-09c7-47e2-ae87-add4aace3f38/:pklxkd-59[3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24])/obj115[0] 2023-05-31T20:54:52.543 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:54:52.534+0000 7f259334b780 20 RGW-SYNC:bucket[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24<-pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:full_sync[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]: [full sync] syncing object: pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0/obj116 2023-05-31T20:54:52.543 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:54:52.534+0000 7f259334b780 20 RGW-SYNC:bucket[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24<-pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:full_sync[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:entry[obj116]: bucket sync single entry (source_zone=3d163b1e-09c7-47e2-ae87-add4aace3f38) b=bucket sync single entry (source_zone=3d163b1e-09c7-47e2-ae87-add4aace3f38) b= log_entry=obj116 op=0 op_state=1 2023-05-31T20:54:52.543 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:54:52.534+0000 7f259334b780 0 RGW-SYNC:bucket[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24<-pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:full_sync[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:entry[obj116]: : injecting data sync error on key=obj116 2023-05-31T20:54:52.543 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:54:52.534+0000 7f259334b780 10 RGW-SYNC:bucket[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24<-pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:full_sync[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:entry[obj116]: failed, retcode=-5 ((5) Input/output error) 2023-05-31T20:54:52.543 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:54:52.534+0000 7f259334b780 0 RGW-SYNC:bucket[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24<-pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:full_sync[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:entry[obj116]: ERROR: failed to sync object: pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0/obj116 2023-05-31T20:54:52.544 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:54:52.534+0000 7f259334b780 20 RGW-SYNC:bucket[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24<-pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:full_sync[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]: [full sync] syncing object: pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0/obj124 2023-05-31T20:54:52.544 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:54:52.534+0000 7f259334b780 20 RGW-SYNC:bucket[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24<-pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:full_sync[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:entry[obj124]: bucket sync single entry (source_zone=3d163b1e-09c7-47e2-ae87-add4aace3f38) b=bucket sync single entry (source_zone=3d163b1e-09c7-47e2-ae87-add4aace3f38) b= log_entry=obj124 op=0 op_state=1 2023-05-31T20:54:52.544 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:54:52.534+0000 7f259334b780 5 RGW-SYNC:bucket[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24<-pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:full_sync[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:entry[obj124]: bucket sync: sync obj: 3d163b1e-09c7-47e2-ae87-add4aace3f38/:pklxkd-59[3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24])/obj124[0] 2023-05-31T20:54:52.544 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:54:52.534+0000 7f259334b780 20 RGW-SYNC:bucket[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24<-pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:full_sync[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]: [full sync] syncing object: pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0/obj145 2023-05-31T20:54:52.544 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:54:52.534+0000 7f259334b780 20 RGW-SYNC:bucket[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24<-pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:full_sync[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:entry[obj145]: bucket sync single entry (source_zone=3d163b1e-09c7-47e2-ae87-add4aace3f38) b=bucket sync single entry (source_zone=3d163b1e-09c7-47e2-ae87-add4aace3f38) b= log_entry=obj145 op=0 op_state=1 2023-05-31T20:54:52.545 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:54:52.534+0000 7f259334b780 5 RGW-SYNC:bucket[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24<-pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:full_sync[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:entry[obj145]: bucket sync: sync obj: 3d163b1e-09c7-47e2-ae87-add4aace3f38/:pklxkd-59[3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24])/obj145[0] 2023-05-31T20:54:52.545 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:54:52.534+0000 7f259334b780 20 RGW-SYNC:bucket[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24<-pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:full_sync[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]: [full sync] syncing object: pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0/obj148 2023-05-31T20:54:52.545 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:54:52.534+0000 7f259334b780 20 RGW-SYNC:bucket[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24<-pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:full_sync[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:entry[obj148]: bucket sync single entry (source_zone=3d163b1e-09c7-47e2-ae87-add4aace3f38) b=bucket sync single entry (source_zone=3d163b1e-09c7-47e2-ae87-add4aace3f38) b= log_entry=obj148 op=0 op_state=1 2023-05-31T20:54:52.545 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:54:52.534+0000 7f259334b780 5 RGW-SYNC:bucket[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24<-pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:full_sync[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:entry[obj148]: bucket sync: sync obj: 3d163b1e-09c7-47e2-ae87-add4aace3f38/:pklxkd-59[3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24])/obj148[0] 2023-05-31T20:54:52.545 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:54:52.535+0000 7f259334b780 20 RGW-SYNC:bucket[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24<-pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:full_sync[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]: [full sync] syncing object: pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0/obj155 2023-05-31T20:54:52.545 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:54:52.535+0000 7f259334b780 20 RGW-SYNC:bucket[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24<-pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:full_sync[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:entry[obj155]: bucket sync single entry (source_zone=3d163b1e-09c7-47e2-ae87-add4aace3f38) b=bucket sync single entry (source_zone=3d163b1e-09c7-47e2-ae87-add4aace3f38) b= log_entry=obj155 op=0 op_state=1 2023-05-31T20:54:52.546 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:54:52.535+0000 7f259334b780 0 RGW-SYNC:bucket[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24<-pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:full_sync[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:entry[obj155]: : injecting data sync error on key=obj155 2023-05-31T20:54:52.546 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:54:52.535+0000 7f259334b780 10 RGW-SYNC:bucket[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24<-pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:full_sync[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:entry[obj155]: failed, retcode=-5 ((5) Input/output error) 2023-05-31T20:54:52.546 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:54:52.535+0000 7f259334b780 0 RGW-SYNC:bucket[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24<-pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:full_sync[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:entry[obj155]: ERROR: failed to sync object: pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0/obj155 2023-05-31T20:54:52.546 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:54:52.535+0000 7f259334b780 20 RGW-SYNC:bucket[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24<-pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:full_sync[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]: [full sync] syncing object: pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0/obj17 2023-05-31T20:54:52.547 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:54:52.535+0000 7f259334b780 20 RGW-SYNC:bucket[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24<-pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:full_sync[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:entry[obj17]: bucket sync single entry (source_zone=3d163b1e-09c7-47e2-ae87-add4aace3f38) b=bucket sync single entry (source_zone=3d163b1e-09c7-47e2-ae87-add4aace3f38) b= log_entry=obj17 op=0 op_state=1 2023-05-31T20:54:52.547 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:54:52.535+0000 7f259334b780 5 RGW-SYNC:bucket[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24<-pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:full_sync[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:entry[obj17]: bucket sync: sync obj: 3d163b1e-09c7-47e2-ae87-add4aace3f38/:pklxkd-59[3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24])/obj17[0] 2023-05-31T20:54:52.547 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:54:52.535+0000 7f259334b780 20 RGW-SYNC:bucket[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24<-pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:full_sync[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]: [full sync] syncing object: pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0/obj185 2023-05-31T20:54:52.547 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:54:52.535+0000 7f259334b780 20 RGW-SYNC:bucket[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24<-pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:full_sync[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:entry[obj185]: bucket sync single entry (source_zone=3d163b1e-09c7-47e2-ae87-add4aace3f38) b=bucket sync single entry (source_zone=3d163b1e-09c7-47e2-ae87-add4aace3f38) b= log_entry=obj185 op=0 op_state=1 2023-05-31T20:54:52.547 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:54:52.535+0000 7f259334b780 5 RGW-SYNC:bucket[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24<-pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:full_sync[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:entry[obj185]: bucket sync: sync obj: 3d163b1e-09c7-47e2-ae87-add4aace3f38/:pklxkd-59[3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24])/obj185[0] 2023-05-31T20:54:52.547 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:54:52.535+0000 7f259334b780 20 RGW-SYNC:bucket[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24<-pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:full_sync[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]: [full sync] syncing object: pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0/obj19 2023-05-31T20:54:52.548 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:54:52.535+0000 7f259334b780 20 RGW-SYNC:bucket[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24<-pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:full_sync[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:entry[obj19]: bucket sync single entry (source_zone=3d163b1e-09c7-47e2-ae87-add4aace3f38) b=bucket sync single entry (source_zone=3d163b1e-09c7-47e2-ae87-add4aace3f38) b= log_entry=obj19 op=0 op_state=1 2023-05-31T20:54:52.548 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:54:52.535+0000 7f259334b780 5 RGW-SYNC:bucket[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24<-pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:full_sync[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:entry[obj19]: bucket sync: sync obj: 3d163b1e-09c7-47e2-ae87-add4aace3f38/:pklxkd-59[3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24])/obj19[0] 2023-05-31T20:54:52.548 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:54:52.535+0000 7f259334b780 20 RGW-SYNC:bucket[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24<-pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:full_sync[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]: [full sync] syncing object: pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0/obj22 2023-05-31T20:54:52.548 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:54:52.535+0000 7f259334b780 20 RGW-SYNC:bucket[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24<-pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:full_sync[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:entry[obj22]: bucket sync single entry (source_zone=3d163b1e-09c7-47e2-ae87-add4aace3f38) b=bucket sync single entry (source_zone=3d163b1e-09c7-47e2-ae87-add4aace3f38) b= log_entry=obj22 op=0 op_state=1 2023-05-31T20:54:52.548 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:54:52.535+0000 7f259334b780 5 RGW-SYNC:bucket[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24<-pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:full_sync[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:entry[obj22]: bucket sync: sync obj: 3d163b1e-09c7-47e2-ae87-add4aace3f38/:pklxkd-59[3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24])/obj22[0] 2023-05-31T20:54:52.548 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:54:52.535+0000 7f259334b780 20 RGW-SYNC:bucket[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24<-pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:full_sync[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]: [full sync] syncing object: pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0/obj23 2023-05-31T20:54:52.549 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:54:52.535+0000 7f259334b780 20 RGW-SYNC:bucket[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24<-pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:full_sync[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:entry[obj23]: bucket sync single entry (source_zone=3d163b1e-09c7-47e2-ae87-add4aace3f38) b=bucket sync single entry (source_zone=3d163b1e-09c7-47e2-ae87-add4aace3f38) b= log_entry=obj23 op=0 op_state=1 2023-05-31T20:54:52.549 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:54:52.535+0000 7f259334b780 5 RGW-SYNC:bucket[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24<-pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:full_sync[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:entry[obj23]: bucket sync: sync obj: 3d163b1e-09c7-47e2-ae87-add4aace3f38/:pklxkd-59[3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24])/obj23[0] 2023-05-31T20:54:52.549 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:54:52.535+0000 7f259334b780 20 RGW-SYNC:bucket[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24<-pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:full_sync[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]: [full sync] syncing object: pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0/obj29 2023-05-31T20:54:52.549 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:54:52.535+0000 7f259334b780 20 RGW-SYNC:bucket[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24<-pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:full_sync[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:entry[obj29]: bucket sync single entry (source_zone=3d163b1e-09c7-47e2-ae87-add4aace3f38) b=bucket sync single entry (source_zone=3d163b1e-09c7-47e2-ae87-add4aace3f38) b= log_entry=obj29 op=0 op_state=1 2023-05-31T20:54:52.549 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:54:52.535+0000 7f259334b780 5 RGW-SYNC:bucket[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24<-pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:full_sync[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:entry[obj29]: bucket sync: sync obj: 3d163b1e-09c7-47e2-ae87-add4aace3f38/:pklxkd-59[3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24])/obj29[0] 2023-05-31T20:54:52.549 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:54:52.535+0000 7f259334b780 20 RGW-SYNC:bucket[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24<-pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:full_sync[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]: [full sync] syncing object: pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0/obj31 2023-05-31T20:54:52.550 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:54:52.535+0000 7f259334b780 20 RGW-SYNC:bucket[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24<-pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:full_sync[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:entry[obj31]: bucket sync single entry (source_zone=3d163b1e-09c7-47e2-ae87-add4aace3f38) b=bucket sync single entry (source_zone=3d163b1e-09c7-47e2-ae87-add4aace3f38) b= log_entry=obj31 op=0 op_state=1 2023-05-31T20:54:52.550 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:54:52.535+0000 7f259334b780 5 RGW-SYNC:bucket[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24<-pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:full_sync[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:entry[obj31]: bucket sync: sync obj: 3d163b1e-09c7-47e2-ae87-add4aace3f38/:pklxkd-59[3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24])/obj31[0] 2023-05-31T20:54:52.550 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:54:52.535+0000 7f259334b780 20 RGW-SYNC:bucket[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24<-pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:full_sync[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]: [full sync] syncing object: pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0/obj33 2023-05-31T20:54:52.550 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:54:52.535+0000 7f259334b780 20 RGW-SYNC:bucket[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24<-pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:full_sync[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:entry[obj33]: bucket sync single entry (source_zone=3d163b1e-09c7-47e2-ae87-add4aace3f38) b=bucket sync single entry (source_zone=3d163b1e-09c7-47e2-ae87-add4aace3f38) b= log_entry=obj33 op=0 op_state=1 2023-05-31T20:54:52.551 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:54:52.535+0000 7f259334b780 5 RGW-SYNC:bucket[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24<-pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:full_sync[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:entry[obj33]: bucket sync: sync obj: 3d163b1e-09c7-47e2-ae87-add4aace3f38/:pklxkd-59[3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24])/obj33[0] 2023-05-31T20:54:52.551 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:54:52.535+0000 7f259334b780 20 RGW-SYNC:bucket[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24<-pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:full_sync[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]: [full sync] syncing object: pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0/obj34 2023-05-31T20:54:52.551 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:54:52.535+0000 7f259334b780 20 RGW-SYNC:bucket[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24<-pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:full_sync[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:entry[obj34]: bucket sync single entry (source_zone=3d163b1e-09c7-47e2-ae87-add4aace3f38) b=bucket sync single entry (source_zone=3d163b1e-09c7-47e2-ae87-add4aace3f38) b= log_entry=obj34 op=0 op_state=1 2023-05-31T20:54:52.551 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:54:52.535+0000 7f259334b780 5 RGW-SYNC:bucket[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24<-pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:full_sync[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:entry[obj34]: bucket sync: sync obj: 3d163b1e-09c7-47e2-ae87-add4aace3f38/:pklxkd-59[3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24])/obj34[0] 2023-05-31T20:54:52.551 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:54:52.535+0000 7f259334b780 20 RGW-SYNC:bucket[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24<-pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:full_sync[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]: [full sync] syncing object: pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0/obj37 2023-05-31T20:54:52.551 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:54:52.535+0000 7f259334b780 20 RGW-SYNC:bucket[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24<-pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:full_sync[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:entry[obj37]: bucket sync single entry (source_zone=3d163b1e-09c7-47e2-ae87-add4aace3f38) b=bucket sync single entry (source_zone=3d163b1e-09c7-47e2-ae87-add4aace3f38) b= log_entry=obj37 op=0 op_state=1 2023-05-31T20:54:52.552 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:54:52.535+0000 7f259334b780 5 RGW-SYNC:bucket[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24<-pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:full_sync[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:entry[obj37]: bucket sync: sync obj: 3d163b1e-09c7-47e2-ae87-add4aace3f38/:pklxkd-59[3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24])/obj37[0] 2023-05-31T20:54:52.552 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:54:52.535+0000 7f259334b780 20 RGW-SYNC:bucket[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24<-pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:full_sync[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]: [full sync] syncing object: pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0/obj38 2023-05-31T20:54:52.552 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:54:52.535+0000 7f259334b780 20 RGW-SYNC:bucket[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24<-pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:full_sync[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:entry[obj38]: bucket sync single entry (source_zone=3d163b1e-09c7-47e2-ae87-add4aace3f38) b=bucket sync single entry (source_zone=3d163b1e-09c7-47e2-ae87-add4aace3f38) b= log_entry=obj38 op=0 op_state=1 2023-05-31T20:54:52.552 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:54:52.535+0000 7f259334b780 5 RGW-SYNC:bucket[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24<-pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:full_sync[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:entry[obj38]: bucket sync: sync obj: 3d163b1e-09c7-47e2-ae87-add4aace3f38/:pklxkd-59[3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24])/obj38[0] 2023-05-31T20:54:52.552 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:54:52.535+0000 7f259334b780 20 RGW-SYNC:bucket[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24<-pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:full_sync[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]: [full sync] syncing object: pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0/obj44 2023-05-31T20:54:52.552 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:54:52.535+0000 7f259334b780 20 RGW-SYNC:bucket[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24<-pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:full_sync[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:entry[obj44]: bucket sync single entry (source_zone=3d163b1e-09c7-47e2-ae87-add4aace3f38) b=bucket sync single entry (source_zone=3d163b1e-09c7-47e2-ae87-add4aace3f38) b= log_entry=obj44 op=0 op_state=1 2023-05-31T20:54:52.552 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:54:52.535+0000 7f259334b780 5 RGW-SYNC:bucket[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24<-pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:full_sync[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:entry[obj44]: bucket sync: sync obj: 3d163b1e-09c7-47e2-ae87-add4aace3f38/:pklxkd-59[3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24])/obj44[0] 2023-05-31T20:54:52.553 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:54:52.535+0000 7f259334b780 20 RGW-SYNC:bucket[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24<-pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:full_sync[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]: [full sync] syncing object: pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0/obj46 2023-05-31T20:54:52.553 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:54:52.535+0000 7f259334b780 20 RGW-SYNC:bucket[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24<-pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:full_sync[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:entry[obj46]: bucket sync single entry (source_zone=3d163b1e-09c7-47e2-ae87-add4aace3f38) b=bucket sync single entry (source_zone=3d163b1e-09c7-47e2-ae87-add4aace3f38) b= log_entry=obj46 op=0 op_state=1 2023-05-31T20:54:52.553 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:54:52.535+0000 7f259334b780 5 RGW-SYNC:bucket[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24<-pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:full_sync[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:entry[obj46]: bucket sync: sync obj: 3d163b1e-09c7-47e2-ae87-add4aace3f38/:pklxkd-59[3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24])/obj46[0] 2023-05-31T20:54:52.555 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:54:52.539+0000 7f259334b780 10 RGW-SYNC:bucket[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24<-pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:full_sync[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:entry[obj44]: success 2023-05-31T20:54:52.555 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:54:52.539+0000 7f259334b780 10 RGW-SYNC:bucket[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24<-pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:full_sync[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:entry[obj111]: success 2023-05-31T20:54:52.555 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:54:52.539+0000 7f259334b780 20 RGW-SYNC:bucket[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24<-pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:full_sync[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]: [full sync] syncing object: pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0/obj51 2023-05-31T20:54:52.556 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:54:52.539+0000 7f259334b780 20 RGW-SYNC:bucket[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24<-pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:full_sync[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:entry[obj51]: bucket sync single entry (source_zone=3d163b1e-09c7-47e2-ae87-add4aace3f38) b=bucket sync single entry (source_zone=3d163b1e-09c7-47e2-ae87-add4aace3f38) b= log_entry=obj51 op=0 op_state=1 2023-05-31T20:54:52.556 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:54:52.539+0000 7f259334b780 5 RGW-SYNC:bucket[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24<-pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:full_sync[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:entry[obj51]: bucket sync: sync obj: 3d163b1e-09c7-47e2-ae87-add4aace3f38/:pklxkd-59[3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24])/obj51[0] 2023-05-31T20:54:52.556 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:54:52.539+0000 7f259334b780 20 RGW-SYNC:bucket[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24<-pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:full_sync[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]: [full sync] syncing object: pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0/obj55 2023-05-31T20:54:52.556 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:54:52.539+0000 7f259334b780 20 RGW-SYNC:bucket[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24<-pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:full_sync[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:entry[obj55]: bucket sync single entry (source_zone=3d163b1e-09c7-47e2-ae87-add4aace3f38) b=bucket sync single entry (source_zone=3d163b1e-09c7-47e2-ae87-add4aace3f38) b= log_entry=obj55 op=0 op_state=1 2023-05-31T20:54:52.556 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:54:52.539+0000 7f259334b780 5 RGW-SYNC:bucket[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24<-pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:full_sync[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:entry[obj55]: bucket sync: sync obj: 3d163b1e-09c7-47e2-ae87-add4aace3f38/:pklxkd-59[3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24])/obj55[0] 2023-05-31T20:54:52.556 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:54:52.539+0000 7f259334b780 10 RGW-SYNC:bucket[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24<-pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:full_sync[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:entry[obj17]: success 2023-05-31T20:54:52.557 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:54:52.539+0000 7f259334b780 10 RGW-SYNC:bucket[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24<-pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:full_sync[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:entry[obj185]: success 2023-05-31T20:54:52.557 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:54:52.539+0000 7f259334b780 20 RGW-SYNC:bucket[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24<-pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:full_sync[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]: [full sync] syncing object: pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0/obj57 2023-05-31T20:54:52.557 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:54:52.539+0000 7f259334b780 20 RGW-SYNC:bucket[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24<-pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:full_sync[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:entry[obj57]: bucket sync single entry (source_zone=3d163b1e-09c7-47e2-ae87-add4aace3f38) b=bucket sync single entry (source_zone=3d163b1e-09c7-47e2-ae87-add4aace3f38) b= log_entry=obj57 op=0 op_state=1 2023-05-31T20:54:52.557 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:54:52.539+0000 7f259334b780 5 RGW-SYNC:bucket[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24<-pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:full_sync[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:entry[obj57]: bucket sync: sync obj: 3d163b1e-09c7-47e2-ae87-add4aace3f38/:pklxkd-59[3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24])/obj57[0] 2023-05-31T20:54:52.557 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:54:52.539+0000 7f259334b780 20 RGW-SYNC:bucket[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24<-pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:full_sync[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]: [full sync] syncing object: pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0/obj58 2023-05-31T20:54:52.557 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:54:52.539+0000 7f259334b780 20 RGW-SYNC:bucket[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24<-pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:full_sync[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:entry[obj58]: bucket sync single entry (source_zone=3d163b1e-09c7-47e2-ae87-add4aace3f38) b=bucket sync single entry (source_zone=3d163b1e-09c7-47e2-ae87-add4aace3f38) b= log_entry=obj58 op=0 op_state=1 2023-05-31T20:54:52.558 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:54:52.539+0000 7f259334b780 5 RGW-SYNC:bucket[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24<-pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:full_sync[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:entry[obj58]: bucket sync: sync obj: 3d163b1e-09c7-47e2-ae87-add4aace3f38/:pklxkd-59[3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24])/obj58[0] 2023-05-31T20:54:52.558 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:54:52.539+0000 7f259334b780 10 RGW-SYNC:bucket[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24<-pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:full_sync[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:entry[obj145]: success 2023-05-31T20:54:52.558 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:54:52.539+0000 7f259334b780 20 RGW-SYNC:bucket[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24<-pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:full_sync[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]: [full sync] syncing object: pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0/obj62 2023-05-31T20:54:52.558 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:54:52.539+0000 7f259334b780 20 RGW-SYNC:bucket[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24<-pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:full_sync[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:entry[obj62]: bucket sync single entry (source_zone=3d163b1e-09c7-47e2-ae87-add4aace3f38) b=bucket sync single entry (source_zone=3d163b1e-09c7-47e2-ae87-add4aace3f38) b= log_entry=obj62 op=0 op_state=1 2023-05-31T20:54:52.558 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:54:52.539+0000 7f259334b780 0 RGW-SYNC:bucket[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24<-pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:full_sync[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:entry[obj62]: : injecting data sync error on key=obj62 2023-05-31T20:54:52.558 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:54:52.539+0000 7f259334b780 10 RGW-SYNC:bucket[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24<-pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:full_sync[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:entry[obj62]: failed, retcode=-5 ((5) Input/output error) 2023-05-31T20:54:52.559 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:54:52.539+0000 7f259334b780 0 RGW-SYNC:bucket[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24<-pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:full_sync[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:entry[obj62]: ERROR: failed to sync object: pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0/obj62 2023-05-31T20:54:52.559 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:54:52.539+0000 7f259334b780 10 RGW-SYNC:bucket[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24<-pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:full_sync[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]: a sync operation returned error 2023-05-31T20:54:52.559 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:54:52.539+0000 7f259334b780 20 RGW-SYNC:bucket[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24<-pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:full_sync[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]: [full sync] syncing object: pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0/obj68 2023-05-31T20:54:52.559 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:54:52.539+0000 7f259334b780 20 RGW-SYNC:bucket[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24<-pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:full_sync[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:entry[obj68]: bucket sync single entry (source_zone=3d163b1e-09c7-47e2-ae87-add4aace3f38) b=bucket sync single entry (source_zone=3d163b1e-09c7-47e2-ae87-add4aace3f38) b= log_entry=obj68 op=0 op_state=1 2023-05-31T20:54:52.559 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:54:52.539+0000 7f259334b780 5 RGW-SYNC:bucket[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24<-pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:full_sync[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:entry[obj68]: bucket sync: sync obj: 3d163b1e-09c7-47e2-ae87-add4aace3f38/:pklxkd-59[3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24])/obj68[0] 2023-05-31T20:54:52.560 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:54:52.540+0000 7f259334b780 10 RGW-SYNC:bucket[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24<-pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:full_sync[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:entry[obj38]: success 2023-05-31T20:54:52.560 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:54:52.540+0000 7f259334b780 10 RGW-SYNC:bucket[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24<-pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:full_sync[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:entry[obj31]: success 2023-05-31T20:54:52.560 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:54:52.540+0000 7f259334b780 20 RGW-SYNC:bucket[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24<-pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:full_sync[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]: [full sync] syncing object: pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0/obj69 2023-05-31T20:54:52.560 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:54:52.540+0000 7f259334b780 20 RGW-SYNC:bucket[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24<-pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:full_sync[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:entry[obj69]: bucket sync single entry (source_zone=3d163b1e-09c7-47e2-ae87-add4aace3f38) b=bucket sync single entry (source_zone=3d163b1e-09c7-47e2-ae87-add4aace3f38) b= log_entry=obj69 op=0 op_state=1 2023-05-31T20:54:52.561 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:54:52.540+0000 7f259334b780 5 RGW-SYNC:bucket[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24<-pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:full_sync[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:entry[obj69]: bucket sync: sync obj: 3d163b1e-09c7-47e2-ae87-add4aace3f38/:pklxkd-59[3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24])/obj69[0] 2023-05-31T20:54:52.561 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:54:52.540+0000 7f259334b780 20 RGW-SYNC:bucket[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24<-pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:full_sync[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]: [full sync] syncing object: pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0/obj74 2023-05-31T20:54:52.561 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:54:52.540+0000 7f259334b780 20 RGW-SYNC:bucket[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24<-pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:full_sync[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:entry[obj74]: bucket sync single entry (source_zone=3d163b1e-09c7-47e2-ae87-add4aace3f38) b=bucket sync single entry (source_zone=3d163b1e-09c7-47e2-ae87-add4aace3f38) b= log_entry=obj74 op=0 op_state=1 2023-05-31T20:54:52.561 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:54:52.540+0000 7f259334b780 10 RGW-SYNC:bucket[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24<-pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:full_sync[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:entry[obj115]: success 2023-05-31T20:54:52.561 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:54:52.540+0000 7f259334b780 5 RGW-SYNC:bucket[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24<-pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:full_sync[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:entry[obj74]: bucket sync: sync obj: 3d163b1e-09c7-47e2-ae87-add4aace3f38/:pklxkd-59[3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24])/obj74[0] 2023-05-31T20:54:52.561 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:54:52.540+0000 7f259334b780 20 RGW-SYNC:bucket[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24<-pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:full_sync[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]: [full sync] syncing object: pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0/obj76 2023-05-31T20:54:52.561 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:54:52.540+0000 7f259334b780 20 RGW-SYNC:bucket[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24<-pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:full_sync[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:entry[obj76]: bucket sync single entry (source_zone=3d163b1e-09c7-47e2-ae87-add4aace3f38) b=bucket sync single entry (source_zone=3d163b1e-09c7-47e2-ae87-add4aace3f38) b= log_entry=obj76 op=0 op_state=1 2023-05-31T20:54:52.562 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:54:52.540+0000 7f259334b780 5 RGW-SYNC:bucket[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24<-pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:full_sync[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:entry[obj76]: bucket sync: sync obj: 3d163b1e-09c7-47e2-ae87-add4aace3f38/:pklxkd-59[3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24])/obj76[0] 2023-05-31T20:54:52.562 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:54:52.540+0000 7f259334b780 10 RGW-SYNC:bucket[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24<-pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:full_sync[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:entry[obj37]: success 2023-05-31T20:54:52.562 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:54:52.540+0000 7f259334b780 10 RGW-SYNC:bucket[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24<-pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:full_sync[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:entry[obj23]: success 2023-05-31T20:54:52.562 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:54:52.540+0000 7f259334b780 20 RGW-SYNC:bucket[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24<-pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:full_sync[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]: [full sync] syncing object: pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0/obj85 2023-05-31T20:54:52.562 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:54:52.540+0000 7f259334b780 20 RGW-SYNC:bucket[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24<-pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:full_sync[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:entry[obj85]: bucket sync single entry (source_zone=3d163b1e-09c7-47e2-ae87-add4aace3f38) b=bucket sync single entry (source_zone=3d163b1e-09c7-47e2-ae87-add4aace3f38) b= log_entry=obj85 op=0 op_state=1 2023-05-31T20:54:52.562 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:54:52.540+0000 7f259334b780 10 RGW-SYNC:bucket[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24<-pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:full_sync[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:entry[obj11]: success 2023-05-31T20:54:52.563 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:54:52.540+0000 7f259334b780 20 RGW-SYNC:bucket[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24<-pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:full_sync[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]: updating marker oid=bucket.full-sync-status.3d163b1e-09c7-47e2-ae87-add4aace3f38:pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24 marker=obj115 2023-05-31T20:54:52.563 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:54:52.540+0000 7f259334b780 0 RGW-SYNC:bucket[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24<-pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:full_sync[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:entry[obj85]: : injecting data sync error on key=obj85 2023-05-31T20:54:52.563 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:54:52.540+0000 7f259334b780 10 RGW-SYNC:bucket[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24<-pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:full_sync[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:entry[obj85]: failed, retcode=-5 ((5) Input/output error) 2023-05-31T20:54:52.563 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:54:52.540+0000 7f259334b780 0 RGW-SYNC:bucket[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24<-pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:full_sync[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:entry[obj85]: ERROR: failed to sync object: pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0/obj85 2023-05-31T20:54:52.563 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:54:52.540+0000 7f259334b780 20 RGW-SYNC:bucket[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24<-pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:full_sync[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]: [full sync] syncing object: pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0/obj87 2023-05-31T20:54:52.563 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:54:52.540+0000 7f259334b780 20 RGW-SYNC:bucket[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24<-pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:full_sync[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:entry[obj87]: bucket sync single entry (source_zone=3d163b1e-09c7-47e2-ae87-add4aace3f38) b=bucket sync single entry (source_zone=3d163b1e-09c7-47e2-ae87-add4aace3f38) b= log_entry=obj87 op=0 op_state=1 2023-05-31T20:54:52.564 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:54:52.540+0000 7f259334b780 0 RGW-SYNC:bucket[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24<-pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:full_sync[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:entry[obj87]: : injecting data sync error on key=obj87 2023-05-31T20:54:52.564 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:54:52.540+0000 7f259334b780 10 RGW-SYNC:bucket[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24<-pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:full_sync[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:entry[obj87]: failed, retcode=-5 ((5) Input/output error) 2023-05-31T20:54:52.564 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:54:52.540+0000 7f259334b780 0 RGW-SYNC:bucket[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24<-pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:full_sync[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:entry[obj87]: ERROR: failed to sync object: pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0/obj87 2023-05-31T20:54:52.564 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:54:52.540+0000 7f259334b780 10 RGW-SYNC:bucket[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24<-pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:full_sync[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:entry[obj29]: success 2023-05-31T20:54:52.565 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:54:52.540+0000 7f259334b780 20 RGW-SYNC:bucket[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24<-pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:full_sync[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]: [full sync] syncing object: pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0/obj92 2023-05-31T20:54:52.565 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:54:52.540+0000 7f259334b780 20 RGW-SYNC:bucket[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24<-pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:full_sync[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:entry[obj92]: bucket sync single entry (source_zone=3d163b1e-09c7-47e2-ae87-add4aace3f38) b=bucket sync single entry (source_zone=3d163b1e-09c7-47e2-ae87-add4aace3f38) b= log_entry=obj92 op=0 op_state=1 2023-05-31T20:54:52.565 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:54:52.541+0000 7f259334b780 5 RGW-SYNC:bucket[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24<-pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:full_sync[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:entry[obj92]: bucket sync: sync obj: 3d163b1e-09c7-47e2-ae87-add4aace3f38/:pklxkd-59[3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24])/obj92[0] 2023-05-31T20:54:52.565 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:54:52.541+0000 7f259334b780 10 RGW-SYNC:bucket[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24<-pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:full_sync[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:entry[obj148]: success 2023-05-31T20:54:52.565 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:54:52.541+0000 7f259334b780 10 RGW-SYNC:bucket[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24<-pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:full_sync[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:entry[obj22]: success 2023-05-31T20:54:52.565 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:54:52.541+0000 7f259334b780 10 RGW-SYNC:bucket[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24<-pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:full_sync[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:entry[obj33]: success 2023-05-31T20:54:52.566 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:54:52.541+0000 7f259334b780 10 RGW-SYNC:bucket[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24<-pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:full_sync[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]: a sync operation returned error 2023-05-31T20:54:52.566 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:54:52.541+0000 7f259334b780 20 RGW-SYNC:bucket[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24<-pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:full_sync[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]: [full sync] syncing object: pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0/obj93 2023-05-31T20:54:52.566 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:54:52.541+0000 7f259334b780 20 RGW-SYNC:bucket[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24<-pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:full_sync[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:entry[obj93]: bucket sync single entry (source_zone=3d163b1e-09c7-47e2-ae87-add4aace3f38) b=bucket sync single entry (source_zone=3d163b1e-09c7-47e2-ae87-add4aace3f38) b= log_entry=obj93 op=0 op_state=1 2023-05-31T20:54:52.566 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:54:52.541+0000 7f259334b780 0 RGW-SYNC:bucket[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24<-pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:full_sync[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:entry[obj93]: : injecting data sync error on key=obj93 2023-05-31T20:54:52.566 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:54:52.541+0000 7f259334b780 10 RGW-SYNC:bucket[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24<-pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:full_sync[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:entry[obj93]: failed, retcode=-5 ((5) Input/output error) 2023-05-31T20:54:52.566 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:54:52.541+0000 7f259334b780 0 RGW-SYNC:bucket[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24<-pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:full_sync[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:entry[obj93]: ERROR: failed to sync object: pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0/obj93 2023-05-31T20:54:52.567 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:54:52.541+0000 7f259334b780 20 RGW-SYNC:bucket[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24<-pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:full_sync[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]: [full sync] syncing object: pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0/obj95 2023-05-31T20:54:52.567 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:54:52.541+0000 7f259334b780 20 RGW-SYNC:bucket[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24<-pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:full_sync[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:entry[obj95]: bucket sync single entry (source_zone=3d163b1e-09c7-47e2-ae87-add4aace3f38) b=bucket sync single entry (source_zone=3d163b1e-09c7-47e2-ae87-add4aace3f38) b= log_entry=obj95 op=0 op_state=1 2023-05-31T20:54:52.567 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:54:52.541+0000 7f259334b780 5 RGW-SYNC:bucket[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24<-pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:full_sync[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:entry[obj95]: bucket sync: sync obj: 3d163b1e-09c7-47e2-ae87-add4aace3f38/:pklxkd-59[3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24])/obj95[0] 2023-05-31T20:54:52.567 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:54:52.543+0000 7f259334b780 10 RGW-SYNC:bucket[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24<-pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:full_sync[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:entry[obj19]: success 2023-05-31T20:54:52.567 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:54:52.543+0000 7f259334b780 10 RGW-SYNC:bucket[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24<-pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:full_sync[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:entry[obj34]: success 2023-05-31T20:54:52.567 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:54:52.543+0000 7f259334b780 10 RGW-SYNC:bucket[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24<-pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:full_sync[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:entry[obj46]: success 2023-05-31T20:54:52.567 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:54:52.543+0000 7f259334b780 10 RGW-SYNC:bucket[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24<-pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:full_sync[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:entry[obj124]: success 2023-05-31T20:54:52.568 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:54:52.544+0000 7f259334b780 10 RGW-SYNC:bucket[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24<-pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:full_sync[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:entry[obj51]: success 2023-05-31T20:54:52.568 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:54:52.544+0000 7f259334b780 10 RGW-SYNC:bucket[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24<-pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:full_sync[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:entry[obj76]: success 2023-05-31T20:54:52.568 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:54:52.544+0000 7f259334b780 10 RGW-SYNC:bucket[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24<-pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:full_sync[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:entry[obj58]: success 2023-05-31T20:54:52.568 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:54:52.544+0000 7f259334b780 10 RGW-SYNC:bucket[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24<-pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:full_sync[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:entry[obj57]: success 2023-05-31T20:54:52.568 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:54:52.544+0000 7f259334b780 10 RGW-SYNC:bucket[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24<-pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:full_sync[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:entry[obj69]: success 2023-05-31T20:54:52.568 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:54:52.544+0000 7f259334b780 10 RGW-SYNC:bucket[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24<-pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:full_sync[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:entry[obj92]: success 2023-05-31T20:54:52.569 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:54:52.544+0000 7f259334b780 10 RGW-SYNC:bucket[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24<-pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:full_sync[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:entry[obj68]: success 2023-05-31T20:54:52.569 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:54:52.544+0000 7f259334b780 10 RGW-SYNC:bucket[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24<-pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:full_sync[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:entry[obj74]: success 2023-05-31T20:54:52.569 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:54:52.544+0000 7f259334b780 10 RGW-SYNC:bucket[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24<-pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:full_sync[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:entry[obj55]: success 2023-05-31T20:54:52.569 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:54:52.545+0000 7f259334b780 10 RGW-SYNC:bucket[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24<-pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:full_sync[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:entry[obj95]: success 2023-05-31T20:54:52.569 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:54:52.545+0000 7f259334b780 10 RGW-SYNC:bucket[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24<-pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:full_sync[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]: a sync operation returned error 2023-05-31T20:54:52.569 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:54:52.548+0000 7f259334b780 10 RGW-SYNC:bucket[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24<-pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:full_sync[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]: a sync operation returned error 2023-05-31T20:54:52.569 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:54:52.548+0000 7f259334b780 10 RGW-SYNC:bucket[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24<-pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:full_sync[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]: a sync operation returned error 2023-05-31T20:54:52.570 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:54:52.548+0000 7f259334b780 10 RGW-SYNC:bucket[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24<-pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:full_sync[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]: a sync operation returned error 2023-05-31T20:54:52.570 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:54:52.548+0000 7f259334b780 10 RGW-SYNC:bucket[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24<-pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:full_sync[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]: backing out with sync_status=-5 2023-05-31T20:54:52.570 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:54:52.548+0000 7f259334b780 20 RGW-SYNC:bucket[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24<-pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]: ERROR: full sync failed. error: -5 2023-05-31T20:54:52.570 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:54:52.551+0000 7f259334b780 20 RGW-SYNC:bucket[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24<-pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]: sync status for source bucket: full. lease is: not taken. stop indications is: 0 2023-05-31T20:54:52.571 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:54:52.551+0000 7f259334b780 5 RGW-SYNC:bucket[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24<-pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]: waiting on bucket lease 2023-05-31T20:54:52.603 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:54:52.598+0000 7f259334b780 20 RGW-SYNC:bucket[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24<-pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]: status after acquiring the lock is: full 2023-05-31T20:54:52.623 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:54:52.618+0000 7f259334b780 20 RGW-SYNC:bucket[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24<-pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:full_sync[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]: listing bucket for full sync 2023-05-31T20:54:52.670 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:54:52.664+0000 7f259334b780 20 RGW-SYNC:bucket[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24<-pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:full_sync[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]: [full sync] syncing object: pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0/obj11 2023-05-31T20:54:52.671 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:54:52.664+0000 7f259334b780 20 RGW-SYNC:bucket[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24<-pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:full_sync[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:entry[obj11]: bucket sync single entry (source_zone=3d163b1e-09c7-47e2-ae87-add4aace3f38) b=bucket sync single entry (source_zone=3d163b1e-09c7-47e2-ae87-add4aace3f38) b= log_entry=obj11 op=0 op_state=1 2023-05-31T20:54:52.671 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:54:52.664+0000 7f259334b780 5 RGW-SYNC:bucket[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24<-pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:full_sync[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:entry[obj11]: bucket sync: sync obj: 3d163b1e-09c7-47e2-ae87-add4aace3f38/:pklxkd-59[3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24])/obj11[0] 2023-05-31T20:54:52.671 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:54:52.664+0000 7f259334b780 20 RGW-SYNC:bucket[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24<-pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:full_sync[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]: [full sync] syncing object: pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0/obj111 2023-05-31T20:54:52.671 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:54:52.664+0000 7f259334b780 20 RGW-SYNC:bucket[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24<-pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:full_sync[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:entry[obj111]: bucket sync single entry (source_zone=3d163b1e-09c7-47e2-ae87-add4aace3f38) b=bucket sync single entry (source_zone=3d163b1e-09c7-47e2-ae87-add4aace3f38) b= log_entry=obj111 op=0 op_state=1 2023-05-31T20:54:52.671 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:54:52.664+0000 7f259334b780 0 RGW-SYNC:bucket[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24<-pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:full_sync[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:entry[obj111]: : injecting data sync error on key=obj111 2023-05-31T20:54:52.671 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:54:52.664+0000 7f259334b780 10 RGW-SYNC:bucket[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24<-pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:full_sync[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:entry[obj111]: failed, retcode=-5 ((5) Input/output error) 2023-05-31T20:54:52.672 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:54:52.664+0000 7f259334b780 0 RGW-SYNC:bucket[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24<-pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:full_sync[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:entry[obj111]: ERROR: failed to sync object: pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0/obj111 2023-05-31T20:54:52.672 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:54:52.664+0000 7f259334b780 20 RGW-SYNC:bucket[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24<-pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:full_sync[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]: [full sync] syncing object: pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0/obj115 2023-05-31T20:54:52.672 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:54:52.664+0000 7f259334b780 20 RGW-SYNC:bucket[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24<-pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:full_sync[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:entry[obj115]: bucket sync single entry (source_zone=3d163b1e-09c7-47e2-ae87-add4aace3f38) b=bucket sync single entry (source_zone=3d163b1e-09c7-47e2-ae87-add4aace3f38) b= log_entry=obj115 op=0 op_state=1 2023-05-31T20:54:52.672 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:54:52.664+0000 7f259334b780 5 RGW-SYNC:bucket[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24<-pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:full_sync[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:entry[obj115]: bucket sync: sync obj: 3d163b1e-09c7-47e2-ae87-add4aace3f38/:pklxkd-59[3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24])/obj115[0] 2023-05-31T20:54:52.672 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:54:52.664+0000 7f259334b780 20 RGW-SYNC:bucket[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24<-pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:full_sync[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]: [full sync] syncing object: pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0/obj116 2023-05-31T20:54:52.673 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:54:52.664+0000 7f259334b780 20 RGW-SYNC:bucket[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24<-pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:full_sync[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:entry[obj116]: bucket sync single entry (source_zone=3d163b1e-09c7-47e2-ae87-add4aace3f38) b=bucket sync single entry (source_zone=3d163b1e-09c7-47e2-ae87-add4aace3f38) b= log_entry=obj116 op=0 op_state=1 2023-05-31T20:54:52.673 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:54:52.664+0000 7f259334b780 5 RGW-SYNC:bucket[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24<-pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:full_sync[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:entry[obj116]: bucket sync: sync obj: 3d163b1e-09c7-47e2-ae87-add4aace3f38/:pklxkd-59[3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24])/obj116[0] 2023-05-31T20:54:52.673 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:54:52.664+0000 7f259334b780 20 RGW-SYNC:bucket[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24<-pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:full_sync[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]: [full sync] syncing object: pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0/obj124 2023-05-31T20:54:52.673 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:54:52.664+0000 7f259334b780 20 RGW-SYNC:bucket[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24<-pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:full_sync[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:entry[obj124]: bucket sync single entry (source_zone=3d163b1e-09c7-47e2-ae87-add4aace3f38) b=bucket sync single entry (source_zone=3d163b1e-09c7-47e2-ae87-add4aace3f38) b= log_entry=obj124 op=0 op_state=1 2023-05-31T20:54:52.673 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:54:52.664+0000 7f259334b780 0 RGW-SYNC:bucket[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24<-pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:full_sync[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:entry[obj124]: : injecting data sync error on key=obj124 2023-05-31T20:54:52.673 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:54:52.664+0000 7f259334b780 10 RGW-SYNC:bucket[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24<-pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:full_sync[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:entry[obj124]: failed, retcode=-5 ((5) Input/output error) 2023-05-31T20:54:52.674 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:54:52.664+0000 7f259334b780 0 RGW-SYNC:bucket[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24<-pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:full_sync[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:entry[obj124]: ERROR: failed to sync object: pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0/obj124 2023-05-31T20:54:52.674 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:54:52.664+0000 7f259334b780 20 RGW-SYNC:bucket[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24<-pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:full_sync[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]: [full sync] syncing object: pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0/obj145 2023-05-31T20:54:52.674 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:54:52.664+0000 7f259334b780 20 RGW-SYNC:bucket[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24<-pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:full_sync[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:entry[obj145]: bucket sync single entry (source_zone=3d163b1e-09c7-47e2-ae87-add4aace3f38) b=bucket sync single entry (source_zone=3d163b1e-09c7-47e2-ae87-add4aace3f38) b= log_entry=obj145 op=0 op_state=1 2023-05-31T20:54:52.674 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:54:52.664+0000 7f259334b780 5 RGW-SYNC:bucket[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24<-pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:full_sync[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:entry[obj145]: bucket sync: sync obj: 3d163b1e-09c7-47e2-ae87-add4aace3f38/:pklxkd-59[3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24])/obj145[0] 2023-05-31T20:54:52.674 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:54:52.664+0000 7f259334b780 20 RGW-SYNC:bucket[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24<-pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:full_sync[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]: [full sync] syncing object: pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0/obj148 2023-05-31T20:54:52.674 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:54:52.664+0000 7f259334b780 20 RGW-SYNC:bucket[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24<-pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:full_sync[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:entry[obj148]: bucket sync single entry (source_zone=3d163b1e-09c7-47e2-ae87-add4aace3f38) b=bucket sync single entry (source_zone=3d163b1e-09c7-47e2-ae87-add4aace3f38) b= log_entry=obj148 op=0 op_state=1 2023-05-31T20:54:52.675 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:54:52.664+0000 7f259334b780 0 RGW-SYNC:bucket[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24<-pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:full_sync[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:entry[obj148]: : injecting data sync error on key=obj148 2023-05-31T20:54:52.675 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:54:52.664+0000 7f259334b780 10 RGW-SYNC:bucket[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24<-pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:full_sync[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:entry[obj148]: failed, retcode=-5 ((5) Input/output error) 2023-05-31T20:54:52.675 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:54:52.664+0000 7f259334b780 0 RGW-SYNC:bucket[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24<-pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:full_sync[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:entry[obj148]: ERROR: failed to sync object: pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0/obj148 2023-05-31T20:54:52.675 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:54:52.664+0000 7f259334b780 20 RGW-SYNC:bucket[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24<-pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:full_sync[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]: [full sync] syncing object: pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0/obj155 2023-05-31T20:54:52.675 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:54:52.664+0000 7f259334b780 20 RGW-SYNC:bucket[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24<-pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:full_sync[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:entry[obj155]: bucket sync single entry (source_zone=3d163b1e-09c7-47e2-ae87-add4aace3f38) b=bucket sync single entry (source_zone=3d163b1e-09c7-47e2-ae87-add4aace3f38) b= log_entry=obj155 op=0 op_state=1 2023-05-31T20:54:52.675 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:54:52.664+0000 7f259334b780 5 RGW-SYNC:bucket[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24<-pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:full_sync[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:entry[obj155]: bucket sync: sync obj: 3d163b1e-09c7-47e2-ae87-add4aace3f38/:pklxkd-59[3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24])/obj155[0] 2023-05-31T20:54:52.676 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:54:52.664+0000 7f259334b780 20 RGW-SYNC:bucket[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24<-pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:full_sync[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]: [full sync] syncing object: pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0/obj17 2023-05-31T20:54:52.676 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:54:52.664+0000 7f259334b780 20 RGW-SYNC:bucket[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24<-pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:full_sync[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:entry[obj17]: bucket sync single entry (source_zone=3d163b1e-09c7-47e2-ae87-add4aace3f38) b=bucket sync single entry (source_zone=3d163b1e-09c7-47e2-ae87-add4aace3f38) b= log_entry=obj17 op=0 op_state=1 2023-05-31T20:54:52.676 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:54:52.664+0000 7f259334b780 0 RGW-SYNC:bucket[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24<-pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:full_sync[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:entry[obj17]: : injecting data sync error on key=obj17 2023-05-31T20:54:52.676 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:54:52.664+0000 7f259334b780 10 RGW-SYNC:bucket[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24<-pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:full_sync[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:entry[obj17]: failed, retcode=-5 ((5) Input/output error) 2023-05-31T20:54:52.676 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:54:52.664+0000 7f259334b780 0 RGW-SYNC:bucket[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24<-pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:full_sync[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:entry[obj17]: ERROR: failed to sync object: pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0/obj17 2023-05-31T20:54:52.676 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:54:52.664+0000 7f259334b780 20 RGW-SYNC:bucket[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24<-pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:full_sync[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]: [full sync] syncing object: pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0/obj185 2023-05-31T20:54:52.677 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:54:52.664+0000 7f259334b780 20 RGW-SYNC:bucket[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24<-pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:full_sync[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:entry[obj185]: bucket sync single entry (source_zone=3d163b1e-09c7-47e2-ae87-add4aace3f38) b=bucket sync single entry (source_zone=3d163b1e-09c7-47e2-ae87-add4aace3f38) b= log_entry=obj185 op=0 op_state=1 2023-05-31T20:54:52.677 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:54:52.664+0000 7f259334b780 5 RGW-SYNC:bucket[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24<-pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:full_sync[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:entry[obj185]: bucket sync: sync obj: 3d163b1e-09c7-47e2-ae87-add4aace3f38/:pklxkd-59[3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24])/obj185[0] 2023-05-31T20:54:52.677 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:54:52.665+0000 7f259334b780 20 RGW-SYNC:bucket[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24<-pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:full_sync[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]: [full sync] syncing object: pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0/obj19 2023-05-31T20:54:52.677 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:54:52.665+0000 7f259334b780 20 RGW-SYNC:bucket[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24<-pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:full_sync[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:entry[obj19]: bucket sync single entry (source_zone=3d163b1e-09c7-47e2-ae87-add4aace3f38) b=bucket sync single entry (source_zone=3d163b1e-09c7-47e2-ae87-add4aace3f38) b= log_entry=obj19 op=0 op_state=1 2023-05-31T20:54:52.677 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:54:52.665+0000 7f259334b780 5 RGW-SYNC:bucket[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24<-pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:full_sync[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:entry[obj19]: bucket sync: sync obj: 3d163b1e-09c7-47e2-ae87-add4aace3f38/:pklxkd-59[3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24])/obj19[0] 2023-05-31T20:54:52.677 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:54:52.665+0000 7f259334b780 20 RGW-SYNC:bucket[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24<-pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:full_sync[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]: [full sync] syncing object: pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0/obj22 2023-05-31T20:54:52.677 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:54:52.665+0000 7f259334b780 20 RGW-SYNC:bucket[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24<-pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:full_sync[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:entry[obj22]: bucket sync single entry (source_zone=3d163b1e-09c7-47e2-ae87-add4aace3f38) b=bucket sync single entry (source_zone=3d163b1e-09c7-47e2-ae87-add4aace3f38) b= log_entry=obj22 op=0 op_state=1 2023-05-31T20:54:52.678 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:54:52.665+0000 7f259334b780 5 RGW-SYNC:bucket[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24<-pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:full_sync[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:entry[obj22]: bucket sync: sync obj: 3d163b1e-09c7-47e2-ae87-add4aace3f38/:pklxkd-59[3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24])/obj22[0] 2023-05-31T20:54:52.678 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:54:52.665+0000 7f259334b780 20 RGW-SYNC:bucket[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24<-pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:full_sync[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]: [full sync] syncing object: pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0/obj23 2023-05-31T20:54:52.678 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:54:52.665+0000 7f259334b780 20 RGW-SYNC:bucket[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24<-pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:full_sync[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:entry[obj23]: bucket sync single entry (source_zone=3d163b1e-09c7-47e2-ae87-add4aace3f38) b=bucket sync single entry (source_zone=3d163b1e-09c7-47e2-ae87-add4aace3f38) b= log_entry=obj23 op=0 op_state=1 2023-05-31T20:54:52.678 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:54:52.665+0000 7f259334b780 5 RGW-SYNC:bucket[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24<-pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:full_sync[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:entry[obj23]: bucket sync: sync obj: 3d163b1e-09c7-47e2-ae87-add4aace3f38/:pklxkd-59[3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24])/obj23[0] 2023-05-31T20:54:52.678 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:54:52.665+0000 7f259334b780 20 RGW-SYNC:bucket[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24<-pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:full_sync[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]: [full sync] syncing object: pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0/obj29 2023-05-31T20:54:52.678 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:54:52.665+0000 7f259334b780 20 RGW-SYNC:bucket[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24<-pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:full_sync[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:entry[obj29]: bucket sync single entry (source_zone=3d163b1e-09c7-47e2-ae87-add4aace3f38) b=bucket sync single entry (source_zone=3d163b1e-09c7-47e2-ae87-add4aace3f38) b= log_entry=obj29 op=0 op_state=1 2023-05-31T20:54:52.679 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:54:52.665+0000 7f259334b780 5 RGW-SYNC:bucket[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24<-pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:full_sync[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:entry[obj29]: bucket sync: sync obj: 3d163b1e-09c7-47e2-ae87-add4aace3f38/:pklxkd-59[3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24])/obj29[0] 2023-05-31T20:54:52.679 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:54:52.665+0000 7f259334b780 20 RGW-SYNC:bucket[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24<-pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:full_sync[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]: [full sync] syncing object: pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0/obj31 2023-05-31T20:54:52.679 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:54:52.665+0000 7f259334b780 20 RGW-SYNC:bucket[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24<-pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:full_sync[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:entry[obj31]: bucket sync single entry (source_zone=3d163b1e-09c7-47e2-ae87-add4aace3f38) b=bucket sync single entry (source_zone=3d163b1e-09c7-47e2-ae87-add4aace3f38) b= log_entry=obj31 op=0 op_state=1 2023-05-31T20:54:52.679 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:54:52.665+0000 7f259334b780 5 RGW-SYNC:bucket[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24<-pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:full_sync[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:entry[obj31]: bucket sync: sync obj: 3d163b1e-09c7-47e2-ae87-add4aace3f38/:pklxkd-59[3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24])/obj31[0] 2023-05-31T20:54:52.679 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:54:52.665+0000 7f259334b780 20 RGW-SYNC:bucket[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24<-pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:full_sync[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]: [full sync] syncing object: pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0/obj33 2023-05-31T20:54:52.679 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:54:52.665+0000 7f259334b780 20 RGW-SYNC:bucket[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24<-pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:full_sync[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:entry[obj33]: bucket sync single entry (source_zone=3d163b1e-09c7-47e2-ae87-add4aace3f38) b=bucket sync single entry (source_zone=3d163b1e-09c7-47e2-ae87-add4aace3f38) b= log_entry=obj33 op=0 op_state=1 2023-05-31T20:54:52.679 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:54:52.665+0000 7f259334b780 5 RGW-SYNC:bucket[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24<-pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:full_sync[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:entry[obj33]: bucket sync: sync obj: 3d163b1e-09c7-47e2-ae87-add4aace3f38/:pklxkd-59[3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24])/obj33[0] 2023-05-31T20:54:52.680 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:54:52.665+0000 7f259334b780 20 RGW-SYNC:bucket[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24<-pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:full_sync[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]: [full sync] syncing object: pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0/obj34 2023-05-31T20:54:52.680 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:54:52.665+0000 7f259334b780 20 RGW-SYNC:bucket[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24<-pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:full_sync[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:entry[obj34]: bucket sync single entry (source_zone=3d163b1e-09c7-47e2-ae87-add4aace3f38) b=bucket sync single entry (source_zone=3d163b1e-09c7-47e2-ae87-add4aace3f38) b= log_entry=obj34 op=0 op_state=1 2023-05-31T20:54:52.680 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:54:52.665+0000 7f259334b780 5 RGW-SYNC:bucket[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24<-pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:full_sync[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:entry[obj34]: bucket sync: sync obj: 3d163b1e-09c7-47e2-ae87-add4aace3f38/:pklxkd-59[3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24])/obj34[0] 2023-05-31T20:54:52.680 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:54:52.665+0000 7f259334b780 20 RGW-SYNC:bucket[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24<-pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:full_sync[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]: [full sync] syncing object: pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0/obj37 2023-05-31T20:54:52.680 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:54:52.665+0000 7f259334b780 20 RGW-SYNC:bucket[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24<-pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:full_sync[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:entry[obj37]: bucket sync single entry (source_zone=3d163b1e-09c7-47e2-ae87-add4aace3f38) b=bucket sync single entry (source_zone=3d163b1e-09c7-47e2-ae87-add4aace3f38) b= log_entry=obj37 op=0 op_state=1 2023-05-31T20:54:52.680 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:54:52.665+0000 7f259334b780 5 RGW-SYNC:bucket[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24<-pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:full_sync[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:entry[obj37]: bucket sync: sync obj: 3d163b1e-09c7-47e2-ae87-add4aace3f38/:pklxkd-59[3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24])/obj37[0] 2023-05-31T20:54:52.681 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:54:52.665+0000 7f259334b780 20 RGW-SYNC:bucket[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24<-pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:full_sync[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]: [full sync] syncing object: pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0/obj38 2023-05-31T20:54:52.681 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:54:52.665+0000 7f259334b780 20 RGW-SYNC:bucket[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24<-pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:full_sync[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:entry[obj38]: bucket sync single entry (source_zone=3d163b1e-09c7-47e2-ae87-add4aace3f38) b=bucket sync single entry (source_zone=3d163b1e-09c7-47e2-ae87-add4aace3f38) b= log_entry=obj38 op=0 op_state=1 2023-05-31T20:54:52.681 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:54:52.665+0000 7f259334b780 5 RGW-SYNC:bucket[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24<-pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:full_sync[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:entry[obj38]: bucket sync: sync obj: 3d163b1e-09c7-47e2-ae87-add4aace3f38/:pklxkd-59[3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24])/obj38[0] 2023-05-31T20:54:52.681 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:54:52.665+0000 7f259334b780 20 RGW-SYNC:bucket[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24<-pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:full_sync[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]: [full sync] syncing object: pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0/obj44 2023-05-31T20:54:52.681 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:54:52.665+0000 7f259334b780 20 RGW-SYNC:bucket[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24<-pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:full_sync[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:entry[obj44]: bucket sync single entry (source_zone=3d163b1e-09c7-47e2-ae87-add4aace3f38) b=bucket sync single entry (source_zone=3d163b1e-09c7-47e2-ae87-add4aace3f38) b= log_entry=obj44 op=0 op_state=1 2023-05-31T20:54:52.681 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:54:52.665+0000 7f259334b780 5 RGW-SYNC:bucket[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24<-pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:full_sync[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:entry[obj44]: bucket sync: sync obj: 3d163b1e-09c7-47e2-ae87-add4aace3f38/:pklxkd-59[3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24])/obj44[0] 2023-05-31T20:54:52.684 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:54:52.666+0000 7f259334b780 20 RGW-SYNC:bucket[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24<-pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:full_sync[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]: [full sync] syncing object: pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0/obj46 2023-05-31T20:54:52.684 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:54:52.666+0000 7f259334b780 20 RGW-SYNC:bucket[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24<-pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:full_sync[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:entry[obj46]: bucket sync single entry (source_zone=3d163b1e-09c7-47e2-ae87-add4aace3f38) b=bucket sync single entry (source_zone=3d163b1e-09c7-47e2-ae87-add4aace3f38) b= log_entry=obj46 op=0 op_state=1 2023-05-31T20:54:52.684 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:54:52.666+0000 7f259334b780 5 RGW-SYNC:bucket[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24<-pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:full_sync[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:entry[obj46]: bucket sync: sync obj: 3d163b1e-09c7-47e2-ae87-add4aace3f38/:pklxkd-59[3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24])/obj46[0] 2023-05-31T20:54:52.684 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:54:52.667+0000 7f259334b780 10 RGW-SYNC:bucket[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24<-pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:full_sync[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:entry[obj11]: success 2023-05-31T20:54:52.684 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:54:52.667+0000 7f259334b780 20 RGW-SYNC:bucket[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24<-pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:full_sync[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]: [full sync] syncing object: pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0/obj51 2023-05-31T20:54:52.685 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:54:52.667+0000 7f259334b780 20 RGW-SYNC:bucket[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24<-pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:full_sync[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:entry[obj51]: bucket sync single entry (source_zone=3d163b1e-09c7-47e2-ae87-add4aace3f38) b=bucket sync single entry (source_zone=3d163b1e-09c7-47e2-ae87-add4aace3f38) b= log_entry=obj51 op=0 op_state=1 2023-05-31T20:54:52.685 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:54:52.667+0000 7f259334b780 5 RGW-SYNC:bucket[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24<-pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:full_sync[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:entry[obj51]: bucket sync: sync obj: 3d163b1e-09c7-47e2-ae87-add4aace3f38/:pklxkd-59[3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24])/obj51[0] 2023-05-31T20:54:52.685 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:54:52.669+0000 7f259334b780 10 RGW-SYNC:bucket[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24<-pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:full_sync[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:entry[obj116]: success 2023-05-31T20:54:52.685 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:54:52.669+0000 7f259334b780 10 RGW-SYNC:bucket[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24<-pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:full_sync[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:entry[obj23]: success 2023-05-31T20:54:52.685 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:54:52.669+0000 7f259334b780 20 RGW-SYNC:bucket[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24<-pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:full_sync[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]: [full sync] syncing object: pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0/obj55 2023-05-31T20:54:52.685 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:54:52.669+0000 7f259334b780 20 RGW-SYNC:bucket[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24<-pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:full_sync[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:entry[obj55]: bucket sync single entry (source_zone=3d163b1e-09c7-47e2-ae87-add4aace3f38) b=bucket sync single entry (source_zone=3d163b1e-09c7-47e2-ae87-add4aace3f38) b= log_entry=obj55 op=0 op_state=1 2023-05-31T20:54:52.686 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:54:52.669+0000 7f259334b780 0 RGW-SYNC:bucket[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24<-pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:full_sync[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:entry[obj55]: : injecting data sync error on key=obj55 2023-05-31T20:54:52.686 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:54:52.669+0000 7f259334b780 10 RGW-SYNC:bucket[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24<-pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:full_sync[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:entry[obj55]: failed, retcode=-5 ((5) Input/output error) 2023-05-31T20:54:52.686 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:54:52.669+0000 7f259334b780 0 RGW-SYNC:bucket[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24<-pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:full_sync[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:entry[obj55]: ERROR: failed to sync object: pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0/obj55 2023-05-31T20:54:52.686 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:54:52.669+0000 7f259334b780 20 RGW-SYNC:bucket[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24<-pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:full_sync[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]: [full sync] syncing object: pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0/obj57 2023-05-31T20:54:52.686 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:54:52.669+0000 7f259334b780 20 RGW-SYNC:bucket[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24<-pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:full_sync[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:entry[obj57]: bucket sync single entry (source_zone=3d163b1e-09c7-47e2-ae87-add4aace3f38) b=bucket sync single entry (source_zone=3d163b1e-09c7-47e2-ae87-add4aace3f38) b= log_entry=obj57 op=0 op_state=1 2023-05-31T20:54:52.686 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:54:52.669+0000 7f259334b780 0 RGW-SYNC:bucket[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24<-pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:full_sync[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:entry[obj57]: : injecting data sync error on key=obj57 2023-05-31T20:54:52.686 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:54:52.669+0000 7f259334b780 10 RGW-SYNC:bucket[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24<-pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:full_sync[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:entry[obj57]: failed, retcode=-5 ((5) Input/output error) 2023-05-31T20:54:52.687 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:54:52.669+0000 7f259334b780 0 RGW-SYNC:bucket[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24<-pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:full_sync[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:entry[obj57]: ERROR: failed to sync object: pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0/obj57 2023-05-31T20:54:52.687 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:54:52.669+0000 7f259334b780 10 RGW-SYNC:bucket[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24<-pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:full_sync[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:entry[obj115]: success 2023-05-31T20:54:52.687 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:54:52.669+0000 7f259334b780 20 RGW-SYNC:bucket[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24<-pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:full_sync[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]: [full sync] syncing object: pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0/obj58 2023-05-31T20:54:52.687 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:54:52.669+0000 7f259334b780 20 RGW-SYNC:bucket[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24<-pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:full_sync[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:entry[obj58]: bucket sync single entry (source_zone=3d163b1e-09c7-47e2-ae87-add4aace3f38) b=bucket sync single entry (source_zone=3d163b1e-09c7-47e2-ae87-add4aace3f38) b= log_entry=obj58 op=0 op_state=1 2023-05-31T20:54:52.687 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:54:52.669+0000 7f259334b780 5 RGW-SYNC:bucket[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24<-pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:full_sync[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:entry[obj58]: bucket sync: sync obj: 3d163b1e-09c7-47e2-ae87-add4aace3f38/:pklxkd-59[3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24])/obj58[0] 2023-05-31T20:54:52.687 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:54:52.669+0000 7f259334b780 10 RGW-SYNC:bucket[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24<-pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:full_sync[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:entry[obj185]: success 2023-05-31T20:54:52.688 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:54:52.669+0000 7f259334b780 10 RGW-SYNC:bucket[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24<-pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:full_sync[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:entry[obj31]: success 2023-05-31T20:54:52.688 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:54:52.669+0000 7f259334b780 20 RGW-SYNC:bucket[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24<-pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:full_sync[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]: [full sync] syncing object: pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0/obj62 2023-05-31T20:54:52.688 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:54:52.669+0000 7f259334b780 20 RGW-SYNC:bucket[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24<-pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:full_sync[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:entry[obj62]: bucket sync single entry (source_zone=3d163b1e-09c7-47e2-ae87-add4aace3f38) b=bucket sync single entry (source_zone=3d163b1e-09c7-47e2-ae87-add4aace3f38) b= log_entry=obj62 op=0 op_state=1 2023-05-31T20:54:52.688 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:54:52.669+0000 7f259334b780 5 RGW-SYNC:bucket[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24<-pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:full_sync[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:entry[obj62]: bucket sync: sync obj: 3d163b1e-09c7-47e2-ae87-add4aace3f38/:pklxkd-59[3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24])/obj62[0] 2023-05-31T20:54:52.688 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:54:52.669+0000 7f259334b780 20 RGW-SYNC:bucket[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24<-pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:full_sync[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]: [full sync] syncing object: pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0/obj68 2023-05-31T20:54:52.689 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:54:52.669+0000 7f259334b780 20 RGW-SYNC:bucket[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24<-pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:full_sync[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:entry[obj68]: bucket sync single entry (source_zone=3d163b1e-09c7-47e2-ae87-add4aace3f38) b=bucket sync single entry (source_zone=3d163b1e-09c7-47e2-ae87-add4aace3f38) b= log_entry=obj68 op=0 op_state=1 2023-05-31T20:54:52.689 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:54:52.669+0000 7f259334b780 5 RGW-SYNC:bucket[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24<-pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:full_sync[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:entry[obj68]: bucket sync: sync obj: 3d163b1e-09c7-47e2-ae87-add4aace3f38/:pklxkd-59[3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24])/obj68[0] 2023-05-31T20:54:52.689 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:54:52.670+0000 7f259334b780 10 RGW-SYNC:bucket[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24<-pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:full_sync[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:entry[obj145]: success 2023-05-31T20:54:52.689 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:54:52.670+0000 7f259334b780 10 RGW-SYNC:bucket[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24<-pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:full_sync[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:entry[obj19]: success 2023-05-31T20:54:52.689 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:54:52.670+0000 7f259334b780 10 RGW-SYNC:bucket[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24<-pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:full_sync[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:entry[obj34]: success 2023-05-31T20:54:52.689 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:54:52.670+0000 7f259334b780 10 RGW-SYNC:bucket[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24<-pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:full_sync[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:entry[obj29]: success 2023-05-31T20:54:52.689 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:54:52.670+0000 7f259334b780 20 RGW-SYNC:bucket[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24<-pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:full_sync[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]: [full sync] syncing object: pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0/obj69 2023-05-31T20:54:52.690 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:54:52.670+0000 7f259334b780 20 RGW-SYNC:bucket[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24<-pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:full_sync[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:entry[obj69]: bucket sync single entry (source_zone=3d163b1e-09c7-47e2-ae87-add4aace3f38) b=bucket sync single entry (source_zone=3d163b1e-09c7-47e2-ae87-add4aace3f38) b= log_entry=obj69 op=0 op_state=1 2023-05-31T20:54:52.690 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:54:52.670+0000 7f259334b780 5 RGW-SYNC:bucket[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24<-pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:full_sync[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:entry[obj69]: bucket sync: sync obj: 3d163b1e-09c7-47e2-ae87-add4aace3f38/:pklxkd-59[3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24])/obj69[0] 2023-05-31T20:54:52.690 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:54:52.670+0000 7f259334b780 20 RGW-SYNC:bucket[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24<-pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:full_sync[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]: [full sync] syncing object: pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0/obj74 2023-05-31T20:54:52.690 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:54:52.670+0000 7f259334b780 20 RGW-SYNC:bucket[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24<-pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:full_sync[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:entry[obj74]: bucket sync single entry (source_zone=3d163b1e-09c7-47e2-ae87-add4aace3f38) b=bucket sync single entry (source_zone=3d163b1e-09c7-47e2-ae87-add4aace3f38) b= log_entry=obj74 op=0 op_state=1 2023-05-31T20:54:52.690 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:54:52.670+0000 7f259334b780 5 RGW-SYNC:bucket[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24<-pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:full_sync[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:entry[obj74]: bucket sync: sync obj: 3d163b1e-09c7-47e2-ae87-add4aace3f38/:pklxkd-59[3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24])/obj74[0] 2023-05-31T20:54:52.690 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:54:52.670+0000 7f259334b780 20 RGW-SYNC:bucket[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24<-pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:full_sync[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]: [full sync] syncing object: pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0/obj76 2023-05-31T20:54:52.691 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:54:52.670+0000 7f259334b780 20 RGW-SYNC:bucket[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24<-pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:full_sync[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:entry[obj76]: bucket sync single entry (source_zone=3d163b1e-09c7-47e2-ae87-add4aace3f38) b=bucket sync single entry (source_zone=3d163b1e-09c7-47e2-ae87-add4aace3f38) b= log_entry=obj76 op=0 op_state=1 2023-05-31T20:54:52.691 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:54:52.670+0000 7f259334b780 5 RGW-SYNC:bucket[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24<-pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:full_sync[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:entry[obj76]: bucket sync: sync obj: 3d163b1e-09c7-47e2-ae87-add4aace3f38/:pklxkd-59[3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24])/obj76[0] 2023-05-31T20:54:52.691 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:54:52.670+0000 7f259334b780 20 RGW-SYNC:bucket[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24<-pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:full_sync[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]: [full sync] syncing object: pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0/obj85 2023-05-31T20:54:52.691 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:54:52.670+0000 7f259334b780 20 RGW-SYNC:bucket[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24<-pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:full_sync[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:entry[obj85]: bucket sync single entry (source_zone=3d163b1e-09c7-47e2-ae87-add4aace3f38) b=bucket sync single entry (source_zone=3d163b1e-09c7-47e2-ae87-add4aace3f38) b= log_entry=obj85 op=0 op_state=1 2023-05-31T20:54:52.691 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:54:52.670+0000 7f259334b780 5 RGW-SYNC:bucket[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24<-pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:full_sync[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:entry[obj85]: bucket sync: sync obj: 3d163b1e-09c7-47e2-ae87-add4aace3f38/:pklxkd-59[3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24])/obj85[0] 2023-05-31T20:54:52.691 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:54:52.672+0000 7f259334b780 10 RGW-SYNC:bucket[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24<-pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:full_sync[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]: a sync operation returned error 2023-05-31T20:54:52.692 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:54:52.672+0000 7f259334b780 20 RGW-SYNC:bucket[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24<-pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:full_sync[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]: [full sync] syncing object: pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0/obj87 2023-05-31T20:54:52.692 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:54:52.672+0000 7f259334b780 20 RGW-SYNC:bucket[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24<-pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:full_sync[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:entry[obj87]: bucket sync single entry (source_zone=3d163b1e-09c7-47e2-ae87-add4aace3f38) b=bucket sync single entry (source_zone=3d163b1e-09c7-47e2-ae87-add4aace3f38) b= log_entry=obj87 op=0 op_state=1 2023-05-31T20:54:52.692 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:54:52.672+0000 7f259334b780 10 RGW-SYNC:bucket[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24<-pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:full_sync[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:entry[obj44]: success 2023-05-31T20:54:52.692 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:54:52.672+0000 7f259334b780 10 RGW-SYNC:bucket[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24<-pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:full_sync[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:entry[obj155]: success 2023-05-31T20:54:52.692 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:54:52.672+0000 7f259334b780 10 RGW-SYNC:bucket[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24<-pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:full_sync[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:entry[obj33]: success 2023-05-31T20:54:52.692 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:54:52.672+0000 7f259334b780 10 RGW-SYNC:bucket[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24<-pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:full_sync[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:entry[obj37]: success 2023-05-31T20:54:52.693 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:54:52.672+0000 7f259334b780 10 RGW-SYNC:bucket[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24<-pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:full_sync[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:entry[obj38]: success 2023-05-31T20:54:52.693 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:54:52.672+0000 7f259334b780 10 RGW-SYNC:bucket[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24<-pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:full_sync[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:entry[obj22]: success 2023-05-31T20:54:52.693 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:54:52.672+0000 7f259334b780 10 RGW-SYNC:bucket[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24<-pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:full_sync[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:entry[obj46]: success 2023-05-31T20:54:52.693 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:54:52.672+0000 7f259334b780 5 RGW-SYNC:bucket[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24<-pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:full_sync[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:entry[obj87]: bucket sync: sync obj: 3d163b1e-09c7-47e2-ae87-add4aace3f38/:pklxkd-59[3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24])/obj87[0] 2023-05-31T20:54:52.693 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:54:52.672+0000 7f259334b780 10 RGW-SYNC:bucket[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24<-pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:full_sync[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]: a sync operation returned error 2023-05-31T20:54:52.693 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:54:52.672+0000 7f259334b780 10 RGW-SYNC:bucket[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24<-pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:full_sync[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]: a sync operation returned error 2023-05-31T20:54:52.693 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:54:52.672+0000 7f259334b780 10 RGW-SYNC:bucket[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24<-pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:full_sync[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]: a sync operation returned error 2023-05-31T20:54:52.694 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:54:52.672+0000 7f259334b780 20 RGW-SYNC:bucket[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24<-pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:full_sync[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]: [full sync] syncing object: pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0/obj92 2023-05-31T20:54:52.694 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:54:52.672+0000 7f259334b780 20 RGW-SYNC:bucket[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24<-pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:full_sync[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:entry[obj92]: bucket sync single entry (source_zone=3d163b1e-09c7-47e2-ae87-add4aace3f38) b=bucket sync single entry (source_zone=3d163b1e-09c7-47e2-ae87-add4aace3f38) b= log_entry=obj92 op=0 op_state=1 2023-05-31T20:54:52.694 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:54:52.672+0000 7f259334b780 5 RGW-SYNC:bucket[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24<-pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:full_sync[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:entry[obj92]: bucket sync: sync obj: 3d163b1e-09c7-47e2-ae87-add4aace3f38/:pklxkd-59[3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24])/obj92[0] 2023-05-31T20:54:52.694 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:54:52.672+0000 7f259334b780 20 RGW-SYNC:bucket[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24<-pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:full_sync[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]: [full sync] syncing object: pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0/obj93 2023-05-31T20:54:52.694 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:54:52.672+0000 7f259334b780 20 RGW-SYNC:bucket[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24<-pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:full_sync[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:entry[obj93]: bucket sync single entry (source_zone=3d163b1e-09c7-47e2-ae87-add4aace3f38) b=bucket sync single entry (source_zone=3d163b1e-09c7-47e2-ae87-add4aace3f38) b= log_entry=obj93 op=0 op_state=1 2023-05-31T20:54:52.694 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:54:52.672+0000 7f259334b780 5 RGW-SYNC:bucket[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24<-pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:full_sync[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:entry[obj93]: bucket sync: sync obj: 3d163b1e-09c7-47e2-ae87-add4aace3f38/:pklxkd-59[3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24])/obj93[0] 2023-05-31T20:54:52.695 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:54:52.672+0000 7f259334b780 20 RGW-SYNC:bucket[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24<-pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:full_sync[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]: [full sync] syncing object: pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0/obj95 2023-05-31T20:54:52.695 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:54:52.672+0000 7f259334b780 20 RGW-SYNC:bucket[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24<-pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:full_sync[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:entry[obj95]: bucket sync single entry (source_zone=3d163b1e-09c7-47e2-ae87-add4aace3f38) b=bucket sync single entry (source_zone=3d163b1e-09c7-47e2-ae87-add4aace3f38) b= log_entry=obj95 op=0 op_state=1 2023-05-31T20:54:52.695 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:54:52.672+0000 7f259334b780 5 RGW-SYNC:bucket[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24<-pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:full_sync[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:entry[obj95]: bucket sync: sync obj: 3d163b1e-09c7-47e2-ae87-add4aace3f38/:pklxkd-59[3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24])/obj95[0] 2023-05-31T20:54:52.695 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:54:52.673+0000 7f259334b780 10 RGW-SYNC:bucket[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24<-pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:full_sync[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]: a sync operation returned error 2023-05-31T20:54:52.695 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:54:52.674+0000 7f259334b780 10 RGW-SYNC:bucket[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24<-pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:full_sync[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:entry[obj51]: success 2023-05-31T20:54:52.695 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:54:52.674+0000 7f259334b780 10 RGW-SYNC:bucket[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24<-pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:full_sync[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:entry[obj68]: success 2023-05-31T20:54:52.696 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:54:52.674+0000 7f259334b780 10 RGW-SYNC:bucket[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24<-pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:full_sync[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:entry[obj58]: success 2023-05-31T20:54:52.696 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:54:52.674+0000 7f259334b780 10 RGW-SYNC:bucket[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24<-pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:full_sync[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:entry[obj74]: success 2023-05-31T20:54:52.696 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:54:52.674+0000 7f259334b780 10 RGW-SYNC:bucket[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24<-pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:full_sync[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:entry[obj62]: success 2023-05-31T20:54:52.696 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:54:52.674+0000 7f259334b780 10 RGW-SYNC:bucket[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24<-pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:full_sync[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:entry[obj76]: success 2023-05-31T20:54:52.696 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:54:52.674+0000 7f259334b780 10 RGW-SYNC:bucket[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24<-pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:full_sync[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:entry[obj69]: success 2023-05-31T20:54:52.696 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:54:52.674+0000 7f259334b780 10 RGW-SYNC:bucket[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24<-pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:full_sync[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:entry[obj85]: success 2023-05-31T20:54:52.696 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:54:52.674+0000 7f259334b780 10 RGW-SYNC:bucket[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24<-pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:full_sync[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]: a sync operation returned error 2023-05-31T20:54:52.697 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:54:52.674+0000 7f259334b780 10 RGW-SYNC:bucket[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24<-pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:full_sync[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:entry[obj93]: success 2023-05-31T20:54:52.697 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:54:52.675+0000 7f259334b780 10 RGW-SYNC:bucket[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24<-pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:full_sync[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:entry[obj87]: success 2023-05-31T20:54:52.697 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:54:52.675+0000 7f259334b780 10 RGW-SYNC:bucket[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24<-pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:full_sync[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:entry[obj92]: success 2023-05-31T20:54:52.697 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:54:52.675+0000 7f259334b780 10 RGW-SYNC:bucket[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24<-pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:full_sync[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:entry[obj95]: success 2023-05-31T20:54:52.697 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:54:52.675+0000 7f259334b780 20 RGW-SYNC:bucket[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24<-pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:full_sync[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]: updating marker oid=bucket.full-sync-status.3d163b1e-09c7-47e2-ae87-add4aace3f38:pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24 marker=obj11 2023-05-31T20:54:52.697 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:54:52.676+0000 7f259334b780 10 RGW-SYNC:bucket[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24<-pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:full_sync[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]: backing out with sync_status=-5 2023-05-31T20:54:52.698 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:54:52.676+0000 7f259334b780 20 RGW-SYNC:bucket[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24<-pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]: ERROR: full sync failed. error: -5 2023-05-31T20:54:52.698 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:54:52.678+0000 7f259334b780 20 RGW-SYNC:bucket[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24<-pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]: sync status for source bucket: full. lease is: not taken. stop indications is: 0 2023-05-31T20:54:52.698 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:54:52.678+0000 7f259334b780 5 RGW-SYNC:bucket[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24<-pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]: waiting on bucket lease 2023-05-31T20:54:52.727 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:54:52.722+0000 7f259334b780 20 RGW-SYNC:bucket[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24<-pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]: status after acquiring the lock is: full 2023-05-31T20:54:52.746 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:54:52.741+0000 7f259334b780 20 RGW-SYNC:bucket[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24<-pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:full_sync[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]: listing bucket for full sync 2023-05-31T20:54:52.792 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:54:52.786+0000 7f259334b780 20 RGW-SYNC:bucket[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24<-pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:full_sync[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]: [full sync] syncing object: pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0/obj11 2023-05-31T20:54:52.793 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:54:52.786+0000 7f259334b780 20 RGW-SYNC:bucket[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24<-pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:full_sync[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:entry[obj11]: bucket sync single entry (source_zone=3d163b1e-09c7-47e2-ae87-add4aace3f38) b=bucket sync single entry (source_zone=3d163b1e-09c7-47e2-ae87-add4aace3f38) b= log_entry=obj11 op=0 op_state=1 2023-05-31T20:54:52.793 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:54:52.786+0000 7f259334b780 5 RGW-SYNC:bucket[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24<-pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:full_sync[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:entry[obj11]: bucket sync: sync obj: 3d163b1e-09c7-47e2-ae87-add4aace3f38/:pklxkd-59[3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24])/obj11[0] 2023-05-31T20:54:52.793 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:54:52.786+0000 7f259334b780 20 RGW-SYNC:bucket[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24<-pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:full_sync[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]: [full sync] syncing object: pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0/obj111 2023-05-31T20:54:52.793 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:54:52.786+0000 7f259334b780 20 RGW-SYNC:bucket[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24<-pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:full_sync[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:entry[obj111]: bucket sync single entry (source_zone=3d163b1e-09c7-47e2-ae87-add4aace3f38) b=bucket sync single entry (source_zone=3d163b1e-09c7-47e2-ae87-add4aace3f38) b= log_entry=obj111 op=0 op_state=1 2023-05-31T20:54:52.793 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:54:52.786+0000 7f259334b780 5 RGW-SYNC:bucket[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24<-pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:full_sync[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:entry[obj111]: bucket sync: sync obj: 3d163b1e-09c7-47e2-ae87-add4aace3f38/:pklxkd-59[3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24])/obj111[0] 2023-05-31T20:54:52.793 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:54:52.786+0000 7f259334b780 20 RGW-SYNC:bucket[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24<-pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:full_sync[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]: [full sync] syncing object: pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0/obj115 2023-05-31T20:54:52.794 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:54:52.786+0000 7f259334b780 20 RGW-SYNC:bucket[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24<-pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:full_sync[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:entry[obj115]: bucket sync single entry (source_zone=3d163b1e-09c7-47e2-ae87-add4aace3f38) b=bucket sync single entry (source_zone=3d163b1e-09c7-47e2-ae87-add4aace3f38) b= log_entry=obj115 op=0 op_state=1 2023-05-31T20:54:52.794 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:54:52.786+0000 7f259334b780 5 RGW-SYNC:bucket[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24<-pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:full_sync[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:entry[obj115]: bucket sync: sync obj: 3d163b1e-09c7-47e2-ae87-add4aace3f38/:pklxkd-59[3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24])/obj115[0] 2023-05-31T20:54:52.794 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:54:52.786+0000 7f259334b780 20 RGW-SYNC:bucket[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24<-pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:full_sync[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]: [full sync] syncing object: pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0/obj116 2023-05-31T20:54:52.794 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:54:52.786+0000 7f259334b780 20 RGW-SYNC:bucket[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24<-pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:full_sync[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:entry[obj116]: bucket sync single entry (source_zone=3d163b1e-09c7-47e2-ae87-add4aace3f38) b=bucket sync single entry (source_zone=3d163b1e-09c7-47e2-ae87-add4aace3f38) b= log_entry=obj116 op=0 op_state=1 2023-05-31T20:54:52.794 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:54:52.786+0000 7f259334b780 5 RGW-SYNC:bucket[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24<-pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:full_sync[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:entry[obj116]: bucket sync: sync obj: 3d163b1e-09c7-47e2-ae87-add4aace3f38/:pklxkd-59[3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24])/obj116[0] 2023-05-31T20:54:52.794 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:54:52.786+0000 7f259334b780 20 RGW-SYNC:bucket[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24<-pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:full_sync[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]: [full sync] syncing object: pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0/obj124 2023-05-31T20:54:52.794 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:54:52.786+0000 7f259334b780 20 RGW-SYNC:bucket[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24<-pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:full_sync[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:entry[obj124]: bucket sync single entry (source_zone=3d163b1e-09c7-47e2-ae87-add4aace3f38) b=bucket sync single entry (source_zone=3d163b1e-09c7-47e2-ae87-add4aace3f38) b= log_entry=obj124 op=0 op_state=1 2023-05-31T20:54:52.795 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:54:52.786+0000 7f259334b780 5 RGW-SYNC:bucket[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24<-pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:full_sync[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:entry[obj124]: bucket sync: sync obj: 3d163b1e-09c7-47e2-ae87-add4aace3f38/:pklxkd-59[3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24])/obj124[0] 2023-05-31T20:54:52.795 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:54:52.786+0000 7f259334b780 20 RGW-SYNC:bucket[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24<-pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:full_sync[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]: [full sync] syncing object: pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0/obj145 2023-05-31T20:54:52.795 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:54:52.786+0000 7f259334b780 20 RGW-SYNC:bucket[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24<-pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:full_sync[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:entry[obj145]: bucket sync single entry (source_zone=3d163b1e-09c7-47e2-ae87-add4aace3f38) b=bucket sync single entry (source_zone=3d163b1e-09c7-47e2-ae87-add4aace3f38) b= log_entry=obj145 op=0 op_state=1 2023-05-31T20:54:52.795 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:54:52.786+0000 7f259334b780 0 RGW-SYNC:bucket[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24<-pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:full_sync[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:entry[obj145]: : injecting data sync error on key=obj145 2023-05-31T20:54:52.795 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:54:52.786+0000 7f259334b780 10 RGW-SYNC:bucket[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24<-pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:full_sync[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:entry[obj145]: failed, retcode=-5 ((5) Input/output error) 2023-05-31T20:54:52.795 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:54:52.786+0000 7f259334b780 0 RGW-SYNC:bucket[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24<-pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:full_sync[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:entry[obj145]: ERROR: failed to sync object: pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0/obj145 2023-05-31T20:54:52.796 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:54:52.786+0000 7f259334b780 20 RGW-SYNC:bucket[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24<-pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:full_sync[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]: [full sync] syncing object: pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0/obj148 2023-05-31T20:54:52.796 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:54:52.786+0000 7f259334b780 20 RGW-SYNC:bucket[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24<-pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:full_sync[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:entry[obj148]: bucket sync single entry (source_zone=3d163b1e-09c7-47e2-ae87-add4aace3f38) b=bucket sync single entry (source_zone=3d163b1e-09c7-47e2-ae87-add4aace3f38) b= log_entry=obj148 op=0 op_state=1 2023-05-31T20:54:52.796 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:54:52.786+0000 7f259334b780 5 RGW-SYNC:bucket[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24<-pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:full_sync[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:entry[obj148]: bucket sync: sync obj: 3d163b1e-09c7-47e2-ae87-add4aace3f38/:pklxkd-59[3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24])/obj148[0] 2023-05-31T20:54:52.796 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:54:52.786+0000 7f259334b780 20 RGW-SYNC:bucket[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24<-pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:full_sync[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]: [full sync] syncing object: pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0/obj155 2023-05-31T20:54:52.796 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:54:52.786+0000 7f259334b780 20 RGW-SYNC:bucket[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24<-pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:full_sync[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:entry[obj155]: bucket sync single entry (source_zone=3d163b1e-09c7-47e2-ae87-add4aace3f38) b=bucket sync single entry (source_zone=3d163b1e-09c7-47e2-ae87-add4aace3f38) b= log_entry=obj155 op=0 op_state=1 2023-05-31T20:54:52.796 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:54:52.786+0000 7f259334b780 5 RGW-SYNC:bucket[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24<-pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:full_sync[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:entry[obj155]: bucket sync: sync obj: 3d163b1e-09c7-47e2-ae87-add4aace3f38/:pklxkd-59[3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24])/obj155[0] 2023-05-31T20:54:52.797 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:54:52.786+0000 7f259334b780 20 RGW-SYNC:bucket[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24<-pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:full_sync[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]: [full sync] syncing object: pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0/obj17 2023-05-31T20:54:52.797 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:54:52.786+0000 7f259334b780 20 RGW-SYNC:bucket[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24<-pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:full_sync[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:entry[obj17]: bucket sync single entry (source_zone=3d163b1e-09c7-47e2-ae87-add4aace3f38) b=bucket sync single entry (source_zone=3d163b1e-09c7-47e2-ae87-add4aace3f38) b= log_entry=obj17 op=0 op_state=1 2023-05-31T20:54:52.797 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:54:52.786+0000 7f259334b780 5 RGW-SYNC:bucket[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24<-pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:full_sync[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:entry[obj17]: bucket sync: sync obj: 3d163b1e-09c7-47e2-ae87-add4aace3f38/:pklxkd-59[3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24])/obj17[0] 2023-05-31T20:54:52.797 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:54:52.786+0000 7f259334b780 20 RGW-SYNC:bucket[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24<-pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:full_sync[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]: [full sync] syncing object: pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0/obj185 2023-05-31T20:54:52.797 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:54:52.786+0000 7f259334b780 20 RGW-SYNC:bucket[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24<-pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:full_sync[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:entry[obj185]: bucket sync single entry (source_zone=3d163b1e-09c7-47e2-ae87-add4aace3f38) b=bucket sync single entry (source_zone=3d163b1e-09c7-47e2-ae87-add4aace3f38) b= log_entry=obj185 op=0 op_state=1 2023-05-31T20:54:52.797 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:54:52.786+0000 7f259334b780 5 RGW-SYNC:bucket[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24<-pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:full_sync[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:entry[obj185]: bucket sync: sync obj: 3d163b1e-09c7-47e2-ae87-add4aace3f38/:pklxkd-59[3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24])/obj185[0] 2023-05-31T20:54:52.797 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:54:52.786+0000 7f259334b780 20 RGW-SYNC:bucket[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24<-pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:full_sync[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]: [full sync] syncing object: pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0/obj19 2023-05-31T20:54:52.798 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:54:52.786+0000 7f259334b780 20 RGW-SYNC:bucket[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24<-pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:full_sync[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:entry[obj19]: bucket sync single entry (source_zone=3d163b1e-09c7-47e2-ae87-add4aace3f38) b=bucket sync single entry (source_zone=3d163b1e-09c7-47e2-ae87-add4aace3f38) b= log_entry=obj19 op=0 op_state=1 2023-05-31T20:54:52.798 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:54:52.786+0000 7f259334b780 5 RGW-SYNC:bucket[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24<-pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:full_sync[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:entry[obj19]: bucket sync: sync obj: 3d163b1e-09c7-47e2-ae87-add4aace3f38/:pklxkd-59[3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24])/obj19[0] 2023-05-31T20:54:52.798 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:54:52.786+0000 7f259334b780 20 RGW-SYNC:bucket[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24<-pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:full_sync[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]: [full sync] syncing object: pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0/obj22 2023-05-31T20:54:52.798 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:54:52.786+0000 7f259334b780 20 RGW-SYNC:bucket[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24<-pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:full_sync[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:entry[obj22]: bucket sync single entry (source_zone=3d163b1e-09c7-47e2-ae87-add4aace3f38) b=bucket sync single entry (source_zone=3d163b1e-09c7-47e2-ae87-add4aace3f38) b= log_entry=obj22 op=0 op_state=1 2023-05-31T20:54:52.798 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:54:52.786+0000 7f259334b780 5 RGW-SYNC:bucket[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24<-pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:full_sync[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:entry[obj22]: bucket sync: sync obj: 3d163b1e-09c7-47e2-ae87-add4aace3f38/:pklxkd-59[3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24])/obj22[0] 2023-05-31T20:54:52.798 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:54:52.786+0000 7f259334b780 20 RGW-SYNC:bucket[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24<-pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:full_sync[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]: [full sync] syncing object: pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0/obj23 2023-05-31T20:54:52.799 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:54:52.786+0000 7f259334b780 20 RGW-SYNC:bucket[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24<-pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:full_sync[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:entry[obj23]: bucket sync single entry (source_zone=3d163b1e-09c7-47e2-ae87-add4aace3f38) b=bucket sync single entry (source_zone=3d163b1e-09c7-47e2-ae87-add4aace3f38) b= log_entry=obj23 op=0 op_state=1 2023-05-31T20:54:52.799 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:54:52.786+0000 7f259334b780 5 RGW-SYNC:bucket[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24<-pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:full_sync[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:entry[obj23]: bucket sync: sync obj: 3d163b1e-09c7-47e2-ae87-add4aace3f38/:pklxkd-59[3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24])/obj23[0] 2023-05-31T20:54:52.799 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:54:52.786+0000 7f259334b780 20 RGW-SYNC:bucket[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24<-pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:full_sync[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]: [full sync] syncing object: pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0/obj29 2023-05-31T20:54:52.799 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:54:52.786+0000 7f259334b780 20 RGW-SYNC:bucket[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24<-pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:full_sync[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:entry[obj29]: bucket sync single entry (source_zone=3d163b1e-09c7-47e2-ae87-add4aace3f38) b=bucket sync single entry (source_zone=3d163b1e-09c7-47e2-ae87-add4aace3f38) b= log_entry=obj29 op=0 op_state=1 2023-05-31T20:54:52.799 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:54:52.786+0000 7f259334b780 0 RGW-SYNC:bucket[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24<-pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:full_sync[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:entry[obj29]: : injecting data sync error on key=obj29 2023-05-31T20:54:52.799 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:54:52.787+0000 7f259334b780 10 RGW-SYNC:bucket[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24<-pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:full_sync[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:entry[obj29]: failed, retcode=-5 ((5) Input/output error) 2023-05-31T20:54:52.799 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:54:52.787+0000 7f259334b780 0 RGW-SYNC:bucket[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24<-pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:full_sync[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:entry[obj29]: ERROR: failed to sync object: pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0/obj29 2023-05-31T20:54:52.800 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:54:52.787+0000 7f259334b780 20 RGW-SYNC:bucket[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24<-pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:full_sync[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]: [full sync] syncing object: pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0/obj31 2023-05-31T20:54:52.800 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:54:52.787+0000 7f259334b780 20 RGW-SYNC:bucket[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24<-pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:full_sync[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:entry[obj31]: bucket sync single entry (source_zone=3d163b1e-09c7-47e2-ae87-add4aace3f38) b=bucket sync single entry (source_zone=3d163b1e-09c7-47e2-ae87-add4aace3f38) b= log_entry=obj31 op=0 op_state=1 2023-05-31T20:54:52.800 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:54:52.787+0000 7f259334b780 5 RGW-SYNC:bucket[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24<-pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:full_sync[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:entry[obj31]: bucket sync: sync obj: 3d163b1e-09c7-47e2-ae87-add4aace3f38/:pklxkd-59[3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24])/obj31[0] 2023-05-31T20:54:52.800 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:54:52.787+0000 7f259334b780 20 RGW-SYNC:bucket[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24<-pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:full_sync[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]: [full sync] syncing object: pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0/obj33 2023-05-31T20:54:52.800 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:54:52.787+0000 7f259334b780 20 RGW-SYNC:bucket[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24<-pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:full_sync[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:entry[obj33]: bucket sync single entry (source_zone=3d163b1e-09c7-47e2-ae87-add4aace3f38) b=bucket sync single entry (source_zone=3d163b1e-09c7-47e2-ae87-add4aace3f38) b= log_entry=obj33 op=0 op_state=1 2023-05-31T20:54:52.800 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:54:52.787+0000 7f259334b780 5 RGW-SYNC:bucket[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24<-pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:full_sync[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:entry[obj33]: bucket sync: sync obj: 3d163b1e-09c7-47e2-ae87-add4aace3f38/:pklxkd-59[3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24])/obj33[0] 2023-05-31T20:54:52.801 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:54:52.787+0000 7f259334b780 20 RGW-SYNC:bucket[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24<-pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:full_sync[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]: [full sync] syncing object: pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0/obj34 2023-05-31T20:54:52.801 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:54:52.787+0000 7f259334b780 20 RGW-SYNC:bucket[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24<-pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:full_sync[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:entry[obj34]: bucket sync single entry (source_zone=3d163b1e-09c7-47e2-ae87-add4aace3f38) b=bucket sync single entry (source_zone=3d163b1e-09c7-47e2-ae87-add4aace3f38) b= log_entry=obj34 op=0 op_state=1 2023-05-31T20:54:52.801 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:54:52.787+0000 7f259334b780 5 RGW-SYNC:bucket[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24<-pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:full_sync[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:entry[obj34]: bucket sync: sync obj: 3d163b1e-09c7-47e2-ae87-add4aace3f38/:pklxkd-59[3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24])/obj34[0] 2023-05-31T20:54:52.801 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:54:52.787+0000 7f259334b780 20 RGW-SYNC:bucket[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24<-pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:full_sync[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]: [full sync] syncing object: pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0/obj37 2023-05-31T20:54:52.801 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:54:52.787+0000 7f259334b780 20 RGW-SYNC:bucket[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24<-pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:full_sync[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:entry[obj37]: bucket sync single entry (source_zone=3d163b1e-09c7-47e2-ae87-add4aace3f38) b=bucket sync single entry (source_zone=3d163b1e-09c7-47e2-ae87-add4aace3f38) b= log_entry=obj37 op=0 op_state=1 2023-05-31T20:54:52.801 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:54:52.787+0000 7f259334b780 5 RGW-SYNC:bucket[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24<-pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:full_sync[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:entry[obj37]: bucket sync: sync obj: 3d163b1e-09c7-47e2-ae87-add4aace3f38/:pklxkd-59[3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24])/obj37[0] 2023-05-31T20:54:52.801 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:54:52.787+0000 7f259334b780 20 RGW-SYNC:bucket[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24<-pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:full_sync[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]: [full sync] syncing object: pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0/obj38 2023-05-31T20:54:52.802 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:54:52.787+0000 7f259334b780 20 RGW-SYNC:bucket[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24<-pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:full_sync[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:entry[obj38]: bucket sync single entry (source_zone=3d163b1e-09c7-47e2-ae87-add4aace3f38) b=bucket sync single entry (source_zone=3d163b1e-09c7-47e2-ae87-add4aace3f38) b= log_entry=obj38 op=0 op_state=1 2023-05-31T20:54:52.802 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:54:52.787+0000 7f259334b780 5 RGW-SYNC:bucket[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24<-pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:full_sync[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:entry[obj38]: bucket sync: sync obj: 3d163b1e-09c7-47e2-ae87-add4aace3f38/:pklxkd-59[3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24])/obj38[0] 2023-05-31T20:54:52.802 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:54:52.787+0000 7f259334b780 20 RGW-SYNC:bucket[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24<-pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:full_sync[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]: [full sync] syncing object: pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0/obj44 2023-05-31T20:54:52.802 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:54:52.787+0000 7f259334b780 20 RGW-SYNC:bucket[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24<-pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:full_sync[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:entry[obj44]: bucket sync single entry (source_zone=3d163b1e-09c7-47e2-ae87-add4aace3f38) b=bucket sync single entry (source_zone=3d163b1e-09c7-47e2-ae87-add4aace3f38) b= log_entry=obj44 op=0 op_state=1 2023-05-31T20:54:52.802 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:54:52.787+0000 7f259334b780 5 RGW-SYNC:bucket[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24<-pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:full_sync[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:entry[obj44]: bucket sync: sync obj: 3d163b1e-09c7-47e2-ae87-add4aace3f38/:pklxkd-59[3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24])/obj44[0] 2023-05-31T20:54:52.802 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:54:52.787+0000 7f259334b780 20 RGW-SYNC:bucket[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24<-pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:full_sync[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]: [full sync] syncing object: pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0/obj46 2023-05-31T20:54:52.803 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:54:52.787+0000 7f259334b780 20 RGW-SYNC:bucket[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24<-pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:full_sync[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:entry[obj46]: bucket sync single entry (source_zone=3d163b1e-09c7-47e2-ae87-add4aace3f38) b=bucket sync single entry (source_zone=3d163b1e-09c7-47e2-ae87-add4aace3f38) b= log_entry=obj46 op=0 op_state=1 2023-05-31T20:54:52.803 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:54:52.787+0000 7f259334b780 5 RGW-SYNC:bucket[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24<-pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:full_sync[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:entry[obj46]: bucket sync: sync obj: 3d163b1e-09c7-47e2-ae87-add4aace3f38/:pklxkd-59[3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24])/obj46[0] 2023-05-31T20:54:52.804 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:54:52.791+0000 7f259334b780 10 RGW-SYNC:bucket[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24<-pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:full_sync[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:entry[obj11]: success 2023-05-31T20:54:52.805 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:54:52.791+0000 7f259334b780 10 RGW-SYNC:bucket[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24<-pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:full_sync[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:entry[obj124]: success 2023-05-31T20:54:52.805 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:54:52.791+0000 7f259334b780 10 RGW-SYNC:bucket[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24<-pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:full_sync[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:entry[obj37]: success 2023-05-31T20:54:52.805 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:54:52.791+0000 7f259334b780 10 RGW-SYNC:bucket[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24<-pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:full_sync[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:entry[obj31]: success 2023-05-31T20:54:52.805 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:54:52.791+0000 7f259334b780 10 RGW-SYNC:bucket[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24<-pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:full_sync[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:entry[obj116]: success 2023-05-31T20:54:52.805 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:54:52.791+0000 7f259334b780 10 RGW-SYNC:bucket[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24<-pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:full_sync[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:entry[obj111]: success 2023-05-31T20:54:52.806 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:54:52.791+0000 7f259334b780 10 RGW-SYNC:bucket[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24<-pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:full_sync[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:entry[obj33]: success 2023-05-31T20:54:52.806 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:54:52.791+0000 7f259334b780 10 RGW-SYNC:bucket[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24<-pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:full_sync[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:entry[obj148]: success 2023-05-31T20:54:52.806 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:54:52.791+0000 7f259334b780 10 RGW-SYNC:bucket[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24<-pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:full_sync[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:entry[obj19]: success 2023-05-31T20:54:52.806 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:54:52.791+0000 7f259334b780 20 RGW-SYNC:bucket[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24<-pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:full_sync[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]: [full sync] syncing object: pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0/obj51 2023-05-31T20:54:52.806 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:54:52.791+0000 7f259334b780 20 RGW-SYNC:bucket[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24<-pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:full_sync[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:entry[obj51]: bucket sync single entry (source_zone=3d163b1e-09c7-47e2-ae87-add4aace3f38) b=bucket sync single entry (source_zone=3d163b1e-09c7-47e2-ae87-add4aace3f38) b= log_entry=obj51 op=0 op_state=1 2023-05-31T20:54:52.806 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:54:52.791+0000 7f259334b780 5 RGW-SYNC:bucket[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24<-pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:full_sync[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:entry[obj51]: bucket sync: sync obj: 3d163b1e-09c7-47e2-ae87-add4aace3f38/:pklxkd-59[3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24])/obj51[0] 2023-05-31T20:54:52.806 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:54:52.791+0000 7f259334b780 20 RGW-SYNC:bucket[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24<-pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:full_sync[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]: [full sync] syncing object: pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0/obj55 2023-05-31T20:54:52.807 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:54:52.791+0000 7f259334b780 20 RGW-SYNC:bucket[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24<-pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:full_sync[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:entry[obj55]: bucket sync single entry (source_zone=3d163b1e-09c7-47e2-ae87-add4aace3f38) b=bucket sync single entry (source_zone=3d163b1e-09c7-47e2-ae87-add4aace3f38) b= log_entry=obj55 op=0 op_state=1 2023-05-31T20:54:52.807 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:54:52.791+0000 7f259334b780 5 RGW-SYNC:bucket[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24<-pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:full_sync[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:entry[obj55]: bucket sync: sync obj: 3d163b1e-09c7-47e2-ae87-add4aace3f38/:pklxkd-59[3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24])/obj55[0] 2023-05-31T20:54:52.807 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:54:52.791+0000 7f259334b780 20 RGW-SYNC:bucket[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24<-pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:full_sync[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]: [full sync] syncing object: pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0/obj57 2023-05-31T20:54:52.807 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:54:52.792+0000 7f259334b780 20 RGW-SYNC:bucket[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24<-pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:full_sync[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:entry[obj57]: bucket sync single entry (source_zone=3d163b1e-09c7-47e2-ae87-add4aace3f38) b=bucket sync single entry (source_zone=3d163b1e-09c7-47e2-ae87-add4aace3f38) b= log_entry=obj57 op=0 op_state=1 2023-05-31T20:54:52.807 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:54:52.792+0000 7f259334b780 5 RGW-SYNC:bucket[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24<-pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:full_sync[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:entry[obj57]: bucket sync: sync obj: 3d163b1e-09c7-47e2-ae87-add4aace3f38/:pklxkd-59[3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24])/obj57[0] 2023-05-31T20:54:52.807 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:54:52.792+0000 7f259334b780 20 RGW-SYNC:bucket[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24<-pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:full_sync[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]: [full sync] syncing object: pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0/obj58 2023-05-31T20:54:52.808 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:54:52.792+0000 7f259334b780 20 RGW-SYNC:bucket[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24<-pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:full_sync[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:entry[obj58]: bucket sync single entry (source_zone=3d163b1e-09c7-47e2-ae87-add4aace3f38) b=bucket sync single entry (source_zone=3d163b1e-09c7-47e2-ae87-add4aace3f38) b= log_entry=obj58 op=0 op_state=1 2023-05-31T20:54:52.808 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:54:52.792+0000 7f259334b780 5 RGW-SYNC:bucket[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24<-pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:full_sync[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:entry[obj58]: bucket sync: sync obj: 3d163b1e-09c7-47e2-ae87-add4aace3f38/:pklxkd-59[3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24])/obj58[0] 2023-05-31T20:54:52.808 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:54:52.792+0000 7f259334b780 20 RGW-SYNC:bucket[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24<-pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:full_sync[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]: [full sync] syncing object: pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0/obj62 2023-05-31T20:54:52.808 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:54:52.792+0000 7f259334b780 20 RGW-SYNC:bucket[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24<-pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:full_sync[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:entry[obj62]: bucket sync single entry (source_zone=3d163b1e-09c7-47e2-ae87-add4aace3f38) b=bucket sync single entry (source_zone=3d163b1e-09c7-47e2-ae87-add4aace3f38) b= log_entry=obj62 op=0 op_state=1 2023-05-31T20:54:52.808 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:54:52.792+0000 7f259334b780 5 RGW-SYNC:bucket[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24<-pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:full_sync[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:entry[obj62]: bucket sync: sync obj: 3d163b1e-09c7-47e2-ae87-add4aace3f38/:pklxkd-59[3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24])/obj62[0] 2023-05-31T20:54:52.808 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:54:52.792+0000 7f259334b780 20 RGW-SYNC:bucket[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24<-pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:full_sync[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]: [full sync] syncing object: pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0/obj68 2023-05-31T20:54:52.809 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:54:52.792+0000 7f259334b780 20 RGW-SYNC:bucket[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24<-pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:full_sync[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:entry[obj68]: bucket sync single entry (source_zone=3d163b1e-09c7-47e2-ae87-add4aace3f38) b=bucket sync single entry (source_zone=3d163b1e-09c7-47e2-ae87-add4aace3f38) b= log_entry=obj68 op=0 op_state=1 2023-05-31T20:54:52.809 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:54:52.792+0000 7f259334b780 10 RGW-SYNC:bucket[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24<-pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:full_sync[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:entry[obj23]: success 2023-05-31T20:54:52.809 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:54:52.792+0000 7f259334b780 10 RGW-SYNC:bucket[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24<-pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:full_sync[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:entry[obj115]: success 2023-05-31T20:54:52.809 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:54:52.792+0000 7f259334b780 20 RGW-SYNC:bucket[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24<-pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:full_sync[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]: updating marker oid=bucket.full-sync-status.3d163b1e-09c7-47e2-ae87-add4aace3f38:pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24 marker=obj124 2023-05-31T20:54:52.809 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:54:52.792+0000 7f259334b780 5 RGW-SYNC:bucket[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24<-pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:full_sync[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:entry[obj68]: bucket sync: sync obj: 3d163b1e-09c7-47e2-ae87-add4aace3f38/:pklxkd-59[3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24])/obj68[0] 2023-05-31T20:54:52.809 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:54:52.792+0000 7f259334b780 20 RGW-SYNC:bucket[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24<-pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:full_sync[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]: [full sync] syncing object: pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0/obj69 2023-05-31T20:54:52.809 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:54:52.792+0000 7f259334b780 20 RGW-SYNC:bucket[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24<-pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:full_sync[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:entry[obj69]: bucket sync single entry (source_zone=3d163b1e-09c7-47e2-ae87-add4aace3f38) b=bucket sync single entry (source_zone=3d163b1e-09c7-47e2-ae87-add4aace3f38) b= log_entry=obj69 op=0 op_state=1 2023-05-31T20:54:52.810 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:54:52.792+0000 7f259334b780 5 RGW-SYNC:bucket[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24<-pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:full_sync[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:entry[obj69]: bucket sync: sync obj: 3d163b1e-09c7-47e2-ae87-add4aace3f38/:pklxkd-59[3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24])/obj69[0] 2023-05-31T20:54:52.810 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:54:52.792+0000 7f259334b780 20 RGW-SYNC:bucket[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24<-pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:full_sync[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]: [full sync] syncing object: pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0/obj74 2023-05-31T20:54:52.810 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:54:52.792+0000 7f259334b780 20 RGW-SYNC:bucket[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24<-pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:full_sync[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:entry[obj74]: bucket sync single entry (source_zone=3d163b1e-09c7-47e2-ae87-add4aace3f38) b=bucket sync single entry (source_zone=3d163b1e-09c7-47e2-ae87-add4aace3f38) b= log_entry=obj74 op=0 op_state=1 2023-05-31T20:54:52.810 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:54:52.792+0000 7f259334b780 5 RGW-SYNC:bucket[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24<-pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:full_sync[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:entry[obj74]: bucket sync: sync obj: 3d163b1e-09c7-47e2-ae87-add4aace3f38/:pklxkd-59[3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24])/obj74[0] 2023-05-31T20:54:52.810 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:54:52.792+0000 7f259334b780 20 RGW-SYNC:bucket[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24<-pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:full_sync[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]: [full sync] syncing object: pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0/obj76 2023-05-31T20:54:52.810 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:54:52.792+0000 7f259334b780 20 RGW-SYNC:bucket[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24<-pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:full_sync[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:entry[obj76]: bucket sync single entry (source_zone=3d163b1e-09c7-47e2-ae87-add4aace3f38) b=bucket sync single entry (source_zone=3d163b1e-09c7-47e2-ae87-add4aace3f38) b= log_entry=obj76 op=0 op_state=1 2023-05-31T20:54:52.811 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:54:52.792+0000 7f259334b780 5 RGW-SYNC:bucket[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24<-pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:full_sync[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:entry[obj76]: bucket sync: sync obj: 3d163b1e-09c7-47e2-ae87-add4aace3f38/:pklxkd-59[3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24])/obj76[0] 2023-05-31T20:54:52.811 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:54:52.792+0000 7f259334b780 20 RGW-SYNC:bucket[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24<-pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:full_sync[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]: [full sync] syncing object: pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0/obj85 2023-05-31T20:54:52.811 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:54:52.792+0000 7f259334b780 20 RGW-SYNC:bucket[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24<-pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:full_sync[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:entry[obj85]: bucket sync single entry (source_zone=3d163b1e-09c7-47e2-ae87-add4aace3f38) b=bucket sync single entry (source_zone=3d163b1e-09c7-47e2-ae87-add4aace3f38) b= log_entry=obj85 op=0 op_state=1 2023-05-31T20:54:52.811 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:54:52.792+0000 7f259334b780 0 RGW-SYNC:bucket[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24<-pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:full_sync[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:entry[obj85]: : injecting data sync error on key=obj85 2023-05-31T20:54:52.811 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:54:52.792+0000 7f259334b780 10 RGW-SYNC:bucket[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24<-pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:full_sync[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:entry[obj85]: failed, retcode=-5 ((5) Input/output error) 2023-05-31T20:54:52.811 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:54:52.792+0000 7f259334b780 0 RGW-SYNC:bucket[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24<-pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:full_sync[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:entry[obj85]: ERROR: failed to sync object: pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0/obj85 2023-05-31T20:54:52.811 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:54:52.792+0000 7f259334b780 10 RGW-SYNC:bucket[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24<-pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:full_sync[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]: a sync operation returned error 2023-05-31T20:54:52.812 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:54:52.792+0000 7f259334b780 20 RGW-SYNC:bucket[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24<-pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:full_sync[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]: [full sync] syncing object: pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0/obj87 2023-05-31T20:54:52.812 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:54:52.792+0000 7f259334b780 20 RGW-SYNC:bucket[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24<-pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:full_sync[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:entry[obj87]: bucket sync single entry (source_zone=3d163b1e-09c7-47e2-ae87-add4aace3f38) b=bucket sync single entry (source_zone=3d163b1e-09c7-47e2-ae87-add4aace3f38) b= log_entry=obj87 op=0 op_state=1 2023-05-31T20:54:52.812 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:54:52.792+0000 7f259334b780 5 RGW-SYNC:bucket[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24<-pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:full_sync[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:entry[obj87]: bucket sync: sync obj: 3d163b1e-09c7-47e2-ae87-add4aace3f38/:pklxkd-59[3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24])/obj87[0] 2023-05-31T20:54:52.812 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:54:52.794+0000 7f259334b780 10 RGW-SYNC:bucket[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24<-pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:full_sync[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]: a sync operation returned error 2023-05-31T20:54:52.812 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:54:52.794+0000 7f259334b780 20 RGW-SYNC:bucket[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24<-pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:full_sync[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]: [full sync] syncing object: pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0/obj92 2023-05-31T20:54:52.812 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:54:52.794+0000 7f259334b780 20 RGW-SYNC:bucket[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24<-pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:full_sync[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:entry[obj92]: bucket sync single entry (source_zone=3d163b1e-09c7-47e2-ae87-add4aace3f38) b=bucket sync single entry (source_zone=3d163b1e-09c7-47e2-ae87-add4aace3f38) b= log_entry=obj92 op=0 op_state=1 2023-05-31T20:54:52.812 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:54:52.794+0000 7f259334b780 10 RGW-SYNC:bucket[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24<-pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:full_sync[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:entry[obj34]: success 2023-05-31T20:54:52.813 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:54:52.794+0000 7f259334b780 10 RGW-SYNC:bucket[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24<-pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:full_sync[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:entry[obj17]: success 2023-05-31T20:54:52.813 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:54:52.794+0000 7f259334b780 10 RGW-SYNC:bucket[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24<-pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:full_sync[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:entry[obj22]: success 2023-05-31T20:54:52.813 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:54:52.794+0000 7f259334b780 10 RGW-SYNC:bucket[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24<-pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:full_sync[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:entry[obj185]: success 2023-05-31T20:54:52.813 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:54:52.794+0000 7f259334b780 10 RGW-SYNC:bucket[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24<-pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:full_sync[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:entry[obj38]: success 2023-05-31T20:54:52.813 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:54:52.794+0000 7f259334b780 10 RGW-SYNC:bucket[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24<-pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:full_sync[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:entry[obj155]: success 2023-05-31T20:54:52.813 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:54:52.794+0000 7f259334b780 10 RGW-SYNC:bucket[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24<-pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:full_sync[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:entry[obj44]: success 2023-05-31T20:54:52.814 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:54:52.794+0000 7f259334b780 10 RGW-SYNC:bucket[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24<-pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:full_sync[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:entry[obj46]: success 2023-05-31T20:54:52.814 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:54:52.794+0000 7f259334b780 5 RGW-SYNC:bucket[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24<-pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:full_sync[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:entry[obj92]: bucket sync: sync obj: 3d163b1e-09c7-47e2-ae87-add4aace3f38/:pklxkd-59[3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24])/obj92[0] 2023-05-31T20:54:52.814 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:54:52.794+0000 7f259334b780 20 RGW-SYNC:bucket[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24<-pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:full_sync[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]: [full sync] syncing object: pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0/obj93 2023-05-31T20:54:52.814 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:54:52.794+0000 7f259334b780 20 RGW-SYNC:bucket[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24<-pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:full_sync[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:entry[obj93]: bucket sync single entry (source_zone=3d163b1e-09c7-47e2-ae87-add4aace3f38) b=bucket sync single entry (source_zone=3d163b1e-09c7-47e2-ae87-add4aace3f38) b= log_entry=obj93 op=0 op_state=1 2023-05-31T20:54:52.814 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:54:52.794+0000 7f259334b780 5 RGW-SYNC:bucket[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24<-pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:full_sync[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:entry[obj93]: bucket sync: sync obj: 3d163b1e-09c7-47e2-ae87-add4aace3f38/:pklxkd-59[3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24])/obj93[0] 2023-05-31T20:54:52.814 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:54:52.794+0000 7f259334b780 20 RGW-SYNC:bucket[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24<-pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:full_sync[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]: [full sync] syncing object: pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0/obj95 2023-05-31T20:54:52.814 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:54:52.794+0000 7f259334b780 20 RGW-SYNC:bucket[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24<-pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:full_sync[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:entry[obj95]: bucket sync single entry (source_zone=3d163b1e-09c7-47e2-ae87-add4aace3f38) b=bucket sync single entry (source_zone=3d163b1e-09c7-47e2-ae87-add4aace3f38) b= log_entry=obj95 op=0 op_state=1 2023-05-31T20:54:52.815 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:54:52.794+0000 7f259334b780 5 RGW-SYNC:bucket[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24<-pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:full_sync[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:entry[obj95]: bucket sync: sync obj: 3d163b1e-09c7-47e2-ae87-add4aace3f38/:pklxkd-59[3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24])/obj95[0] 2023-05-31T20:54:52.815 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:54:52.795+0000 7f259334b780 10 RGW-SYNC:bucket[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24<-pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:full_sync[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:entry[obj87]: success 2023-05-31T20:54:52.815 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:54:52.795+0000 7f259334b780 10 RGW-SYNC:bucket[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24<-pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:full_sync[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:entry[obj74]: success 2023-05-31T20:54:52.815 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:54:52.795+0000 7f259334b780 10 RGW-SYNC:bucket[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24<-pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:full_sync[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:entry[obj55]: success 2023-05-31T20:54:52.815 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:54:52.795+0000 7f259334b780 10 RGW-SYNC:bucket[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24<-pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:full_sync[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:entry[obj62]: success 2023-05-31T20:54:52.815 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:54:52.796+0000 7f259334b780 10 RGW-SYNC:bucket[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24<-pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:full_sync[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:entry[obj57]: success 2023-05-31T20:54:52.815 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:54:52.796+0000 7f259334b780 10 RGW-SYNC:bucket[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24<-pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:full_sync[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:entry[obj51]: success 2023-05-31T20:54:52.816 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:54:52.796+0000 7f259334b780 10 RGW-SYNC:bucket[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24<-pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:full_sync[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:entry[obj68]: success 2023-05-31T20:54:52.816 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:54:52.797+0000 7f259334b780 10 RGW-SYNC:bucket[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24<-pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:full_sync[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:entry[obj93]: success 2023-05-31T20:54:52.816 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:54:52.797+0000 7f259334b780 10 RGW-SYNC:bucket[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24<-pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:full_sync[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:entry[obj76]: success 2023-05-31T20:54:52.816 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:54:52.797+0000 7f259334b780 10 RGW-SYNC:bucket[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24<-pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:full_sync[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:entry[obj69]: success 2023-05-31T20:54:52.816 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:54:52.797+0000 7f259334b780 10 RGW-SYNC:bucket[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24<-pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:full_sync[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:entry[obj95]: success 2023-05-31T20:54:52.816 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:54:52.797+0000 7f259334b780 10 RGW-SYNC:bucket[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24<-pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:full_sync[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:entry[obj58]: success 2023-05-31T20:54:52.817 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:54:52.797+0000 7f259334b780 10 RGW-SYNC:bucket[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24<-pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:full_sync[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:entry[obj92]: success 2023-05-31T20:54:52.817 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:54:52.798+0000 7f259334b780 10 RGW-SYNC:bucket[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24<-pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:full_sync[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]: a sync operation returned error 2023-05-31T20:54:52.817 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:54:52.798+0000 7f259334b780 10 RGW-SYNC:bucket[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24<-pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:full_sync[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]: backing out with sync_status=-5 2023-05-31T20:54:52.817 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:54:52.798+0000 7f259334b780 20 RGW-SYNC:bucket[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24<-pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]: ERROR: full sync failed. error: -5 2023-05-31T20:54:52.817 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:54:52.799+0000 7f259334b780 20 RGW-SYNC:bucket[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24<-pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]: sync status for source bucket: full. lease is: not taken. stop indications is: 0 2023-05-31T20:54:52.817 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:54:52.799+0000 7f259334b780 5 RGW-SYNC:bucket[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24<-pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]: waiting on bucket lease 2023-05-31T20:54:52.850 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:54:52.845+0000 7f259334b780 20 RGW-SYNC:bucket[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24<-pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]: status after acquiring the lock is: full 2023-05-31T20:54:52.863 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:54:52.858+0000 7f259334b780 20 RGW-SYNC:bucket[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24<-pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:full_sync[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]: listing bucket for full sync 2023-05-31T20:54:52.909 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:54:52.904+0000 7f259334b780 20 RGW-SYNC:bucket[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24<-pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:full_sync[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]: [full sync] syncing object: pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0/obj11 2023-05-31T20:54:52.910 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:54:52.904+0000 7f259334b780 20 RGW-SYNC:bucket[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24<-pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:full_sync[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:entry[obj11]: bucket sync single entry (source_zone=3d163b1e-09c7-47e2-ae87-add4aace3f38) b=bucket sync single entry (source_zone=3d163b1e-09c7-47e2-ae87-add4aace3f38) b= log_entry=obj11 op=0 op_state=1 2023-05-31T20:54:52.910 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:54:52.904+0000 7f259334b780 5 RGW-SYNC:bucket[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24<-pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:full_sync[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:entry[obj11]: bucket sync: sync obj: 3d163b1e-09c7-47e2-ae87-add4aace3f38/:pklxkd-59[3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24])/obj11[0] 2023-05-31T20:54:52.910 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:54:52.904+0000 7f259334b780 20 RGW-SYNC:bucket[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24<-pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:full_sync[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]: [full sync] syncing object: pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0/obj111 2023-05-31T20:54:52.910 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:54:52.904+0000 7f259334b780 20 RGW-SYNC:bucket[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24<-pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:full_sync[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:entry[obj111]: bucket sync single entry (source_zone=3d163b1e-09c7-47e2-ae87-add4aace3f38) b=bucket sync single entry (source_zone=3d163b1e-09c7-47e2-ae87-add4aace3f38) b= log_entry=obj111 op=0 op_state=1 2023-05-31T20:54:52.910 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:54:52.904+0000 7f259334b780 5 RGW-SYNC:bucket[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24<-pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:full_sync[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:entry[obj111]: bucket sync: sync obj: 3d163b1e-09c7-47e2-ae87-add4aace3f38/:pklxkd-59[3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24])/obj111[0] 2023-05-31T20:54:52.911 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:54:52.904+0000 7f259334b780 20 RGW-SYNC:bucket[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24<-pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:full_sync[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]: [full sync] syncing object: pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0/obj115 2023-05-31T20:54:52.911 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:54:52.904+0000 7f259334b780 20 RGW-SYNC:bucket[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24<-pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:full_sync[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:entry[obj115]: bucket sync single entry (source_zone=3d163b1e-09c7-47e2-ae87-add4aace3f38) b=bucket sync single entry (source_zone=3d163b1e-09c7-47e2-ae87-add4aace3f38) b= log_entry=obj115 op=0 op_state=1 2023-05-31T20:54:52.911 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:54:52.904+0000 7f259334b780 5 RGW-SYNC:bucket[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24<-pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:full_sync[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:entry[obj115]: bucket sync: sync obj: 3d163b1e-09c7-47e2-ae87-add4aace3f38/:pklxkd-59[3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24])/obj115[0] 2023-05-31T20:54:52.911 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:54:52.904+0000 7f259334b780 20 RGW-SYNC:bucket[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24<-pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:full_sync[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]: [full sync] syncing object: pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0/obj116 2023-05-31T20:54:52.911 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:54:52.904+0000 7f259334b780 20 RGW-SYNC:bucket[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24<-pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:full_sync[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:entry[obj116]: bucket sync single entry (source_zone=3d163b1e-09c7-47e2-ae87-add4aace3f38) b=bucket sync single entry (source_zone=3d163b1e-09c7-47e2-ae87-add4aace3f38) b= log_entry=obj116 op=0 op_state=1 2023-05-31T20:54:52.911 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:54:52.904+0000 7f259334b780 5 RGW-SYNC:bucket[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24<-pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:full_sync[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:entry[obj116]: bucket sync: sync obj: 3d163b1e-09c7-47e2-ae87-add4aace3f38/:pklxkd-59[3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24])/obj116[0] 2023-05-31T20:54:52.911 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:54:52.904+0000 7f259334b780 20 RGW-SYNC:bucket[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24<-pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:full_sync[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]: [full sync] syncing object: pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0/obj124 2023-05-31T20:54:52.912 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:54:52.904+0000 7f259334b780 20 RGW-SYNC:bucket[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24<-pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:full_sync[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:entry[obj124]: bucket sync single entry (source_zone=3d163b1e-09c7-47e2-ae87-add4aace3f38) b=bucket sync single entry (source_zone=3d163b1e-09c7-47e2-ae87-add4aace3f38) b= log_entry=obj124 op=0 op_state=1 2023-05-31T20:54:52.912 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:54:52.904+0000 7f259334b780 5 RGW-SYNC:bucket[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24<-pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:full_sync[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:entry[obj124]: bucket sync: sync obj: 3d163b1e-09c7-47e2-ae87-add4aace3f38/:pklxkd-59[3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24])/obj124[0] 2023-05-31T20:54:52.912 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:54:52.904+0000 7f259334b780 20 RGW-SYNC:bucket[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24<-pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:full_sync[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]: [full sync] syncing object: pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0/obj145 2023-05-31T20:54:52.912 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:54:52.904+0000 7f259334b780 20 RGW-SYNC:bucket[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24<-pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:full_sync[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:entry[obj145]: bucket sync single entry (source_zone=3d163b1e-09c7-47e2-ae87-add4aace3f38) b=bucket sync single entry (source_zone=3d163b1e-09c7-47e2-ae87-add4aace3f38) b= log_entry=obj145 op=0 op_state=1 2023-05-31T20:54:52.912 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:54:52.904+0000 7f259334b780 5 RGW-SYNC:bucket[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24<-pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:full_sync[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:entry[obj145]: bucket sync: sync obj: 3d163b1e-09c7-47e2-ae87-add4aace3f38/:pklxkd-59[3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24])/obj145[0] 2023-05-31T20:54:52.912 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:54:52.904+0000 7f259334b780 20 RGW-SYNC:bucket[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24<-pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:full_sync[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]: [full sync] syncing object: pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0/obj148 2023-05-31T20:54:52.913 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:54:52.904+0000 7f259334b780 20 RGW-SYNC:bucket[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24<-pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:full_sync[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:entry[obj148]: bucket sync single entry (source_zone=3d163b1e-09c7-47e2-ae87-add4aace3f38) b=bucket sync single entry (source_zone=3d163b1e-09c7-47e2-ae87-add4aace3f38) b= log_entry=obj148 op=0 op_state=1 2023-05-31T20:54:52.913 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:54:52.904+0000 7f259334b780 5 RGW-SYNC:bucket[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24<-pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:full_sync[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:entry[obj148]: bucket sync: sync obj: 3d163b1e-09c7-47e2-ae87-add4aace3f38/:pklxkd-59[3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24])/obj148[0] 2023-05-31T20:54:52.913 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:54:52.904+0000 7f259334b780 20 RGW-SYNC:bucket[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24<-pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:full_sync[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]: [full sync] syncing object: pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0/obj155 2023-05-31T20:54:52.913 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:54:52.904+0000 7f259334b780 20 RGW-SYNC:bucket[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24<-pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:full_sync[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:entry[obj155]: bucket sync single entry (source_zone=3d163b1e-09c7-47e2-ae87-add4aace3f38) b=bucket sync single entry (source_zone=3d163b1e-09c7-47e2-ae87-add4aace3f38) b= log_entry=obj155 op=0 op_state=1 2023-05-31T20:54:52.913 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:54:52.904+0000 7f259334b780 5 RGW-SYNC:bucket[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24<-pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:full_sync[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:entry[obj155]: bucket sync: sync obj: 3d163b1e-09c7-47e2-ae87-add4aace3f38/:pklxkd-59[3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24])/obj155[0] 2023-05-31T20:54:52.913 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:54:52.904+0000 7f259334b780 20 RGW-SYNC:bucket[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24<-pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:full_sync[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]: [full sync] syncing object: pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0/obj17 2023-05-31T20:54:52.914 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:54:52.904+0000 7f259334b780 20 RGW-SYNC:bucket[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24<-pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:full_sync[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:entry[obj17]: bucket sync single entry (source_zone=3d163b1e-09c7-47e2-ae87-add4aace3f38) b=bucket sync single entry (source_zone=3d163b1e-09c7-47e2-ae87-add4aace3f38) b= log_entry=obj17 op=0 op_state=1 2023-05-31T20:54:52.914 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:54:52.904+0000 7f259334b780 5 RGW-SYNC:bucket[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24<-pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:full_sync[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:entry[obj17]: bucket sync: sync obj: 3d163b1e-09c7-47e2-ae87-add4aace3f38/:pklxkd-59[3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24])/obj17[0] 2023-05-31T20:54:52.914 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:54:52.904+0000 7f259334b780 20 RGW-SYNC:bucket[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24<-pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:full_sync[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]: [full sync] syncing object: pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0/obj185 2023-05-31T20:54:52.914 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:54:52.904+0000 7f259334b780 20 RGW-SYNC:bucket[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24<-pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:full_sync[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:entry[obj185]: bucket sync single entry (source_zone=3d163b1e-09c7-47e2-ae87-add4aace3f38) b=bucket sync single entry (source_zone=3d163b1e-09c7-47e2-ae87-add4aace3f38) b= log_entry=obj185 op=0 op_state=1 2023-05-31T20:54:52.914 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:54:52.904+0000 7f259334b780 5 RGW-SYNC:bucket[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24<-pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:full_sync[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:entry[obj185]: bucket sync: sync obj: 3d163b1e-09c7-47e2-ae87-add4aace3f38/:pklxkd-59[3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24])/obj185[0] 2023-05-31T20:54:52.914 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:54:52.904+0000 7f259334b780 20 RGW-SYNC:bucket[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24<-pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:full_sync[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]: [full sync] syncing object: pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0/obj19 2023-05-31T20:54:52.914 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:54:52.904+0000 7f259334b780 20 RGW-SYNC:bucket[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24<-pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:full_sync[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:entry[obj19]: bucket sync single entry (source_zone=3d163b1e-09c7-47e2-ae87-add4aace3f38) b=bucket sync single entry (source_zone=3d163b1e-09c7-47e2-ae87-add4aace3f38) b= log_entry=obj19 op=0 op_state=1 2023-05-31T20:54:52.915 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:54:52.904+0000 7f259334b780 5 RGW-SYNC:bucket[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24<-pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:full_sync[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:entry[obj19]: bucket sync: sync obj: 3d163b1e-09c7-47e2-ae87-add4aace3f38/:pklxkd-59[3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24])/obj19[0] 2023-05-31T20:54:52.915 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:54:52.904+0000 7f259334b780 20 RGW-SYNC:bucket[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24<-pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:full_sync[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]: [full sync] syncing object: pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0/obj22 2023-05-31T20:54:52.915 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:54:52.904+0000 7f259334b780 20 RGW-SYNC:bucket[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24<-pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:full_sync[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:entry[obj22]: bucket sync single entry (source_zone=3d163b1e-09c7-47e2-ae87-add4aace3f38) b=bucket sync single entry (source_zone=3d163b1e-09c7-47e2-ae87-add4aace3f38) b= log_entry=obj22 op=0 op_state=1 2023-05-31T20:54:52.915 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:54:52.904+0000 7f259334b780 5 RGW-SYNC:bucket[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24<-pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:full_sync[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:entry[obj22]: bucket sync: sync obj: 3d163b1e-09c7-47e2-ae87-add4aace3f38/:pklxkd-59[3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24])/obj22[0] 2023-05-31T20:54:52.915 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:54:52.904+0000 7f259334b780 20 RGW-SYNC:bucket[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24<-pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:full_sync[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]: [full sync] syncing object: pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0/obj23 2023-05-31T20:54:52.915 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:54:52.904+0000 7f259334b780 20 RGW-SYNC:bucket[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24<-pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:full_sync[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:entry[obj23]: bucket sync single entry (source_zone=3d163b1e-09c7-47e2-ae87-add4aace3f38) b=bucket sync single entry (source_zone=3d163b1e-09c7-47e2-ae87-add4aace3f38) b= log_entry=obj23 op=0 op_state=1 2023-05-31T20:54:52.916 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:54:52.904+0000 7f259334b780 5 RGW-SYNC:bucket[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24<-pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:full_sync[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:entry[obj23]: bucket sync: sync obj: 3d163b1e-09c7-47e2-ae87-add4aace3f38/:pklxkd-59[3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24])/obj23[0] 2023-05-31T20:54:52.916 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:54:52.905+0000 7f259334b780 20 RGW-SYNC:bucket[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24<-pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:full_sync[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]: [full sync] syncing object: pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0/obj29 2023-05-31T20:54:52.916 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:54:52.905+0000 7f259334b780 20 RGW-SYNC:bucket[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24<-pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:full_sync[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:entry[obj29]: bucket sync single entry (source_zone=3d163b1e-09c7-47e2-ae87-add4aace3f38) b=bucket sync single entry (source_zone=3d163b1e-09c7-47e2-ae87-add4aace3f38) b= log_entry=obj29 op=0 op_state=1 2023-05-31T20:54:52.916 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:54:52.905+0000 7f259334b780 5 RGW-SYNC:bucket[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24<-pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:full_sync[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:entry[obj29]: bucket sync: sync obj: 3d163b1e-09c7-47e2-ae87-add4aace3f38/:pklxkd-59[3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24])/obj29[0] 2023-05-31T20:54:52.917 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:54:52.905+0000 7f259334b780 20 RGW-SYNC:bucket[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24<-pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:full_sync[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]: [full sync] syncing object: pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0/obj31 2023-05-31T20:54:52.917 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:54:52.905+0000 7f259334b780 20 RGW-SYNC:bucket[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24<-pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:full_sync[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:entry[obj31]: bucket sync single entry (source_zone=3d163b1e-09c7-47e2-ae87-add4aace3f38) b=bucket sync single entry (source_zone=3d163b1e-09c7-47e2-ae87-add4aace3f38) b= log_entry=obj31 op=0 op_state=1 2023-05-31T20:54:52.918 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:54:52.905+0000 7f259334b780 5 RGW-SYNC:bucket[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24<-pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:full_sync[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:entry[obj31]: bucket sync: sync obj: 3d163b1e-09c7-47e2-ae87-add4aace3f38/:pklxkd-59[3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24])/obj31[0] 2023-05-31T20:54:52.918 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:54:52.905+0000 7f259334b780 20 RGW-SYNC:bucket[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24<-pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:full_sync[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]: [full sync] syncing object: pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0/obj33 2023-05-31T20:54:52.918 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:54:52.905+0000 7f259334b780 20 RGW-SYNC:bucket[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24<-pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:full_sync[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:entry[obj33]: bucket sync single entry (source_zone=3d163b1e-09c7-47e2-ae87-add4aace3f38) b=bucket sync single entry (source_zone=3d163b1e-09c7-47e2-ae87-add4aace3f38) b= log_entry=obj33 op=0 op_state=1 2023-05-31T20:54:52.918 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:54:52.905+0000 7f259334b780 5 RGW-SYNC:bucket[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24<-pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:full_sync[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:entry[obj33]: bucket sync: sync obj: 3d163b1e-09c7-47e2-ae87-add4aace3f38/:pklxkd-59[3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24])/obj33[0] 2023-05-31T20:54:52.918 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:54:52.905+0000 7f259334b780 20 RGW-SYNC:bucket[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24<-pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:full_sync[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]: [full sync] syncing object: pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0/obj34 2023-05-31T20:54:52.919 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:54:52.905+0000 7f259334b780 20 RGW-SYNC:bucket[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24<-pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:full_sync[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:entry[obj34]: bucket sync single entry (source_zone=3d163b1e-09c7-47e2-ae87-add4aace3f38) b=bucket sync single entry (source_zone=3d163b1e-09c7-47e2-ae87-add4aace3f38) b= log_entry=obj34 op=0 op_state=1 2023-05-31T20:54:52.919 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:54:52.905+0000 7f259334b780 5 RGW-SYNC:bucket[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24<-pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:full_sync[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:entry[obj34]: bucket sync: sync obj: 3d163b1e-09c7-47e2-ae87-add4aace3f38/:pklxkd-59[3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24])/obj34[0] 2023-05-31T20:54:52.919 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:54:52.905+0000 7f259334b780 20 RGW-SYNC:bucket[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24<-pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:full_sync[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]: [full sync] syncing object: pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0/obj37 2023-05-31T20:54:52.919 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:54:52.905+0000 7f259334b780 20 RGW-SYNC:bucket[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24<-pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:full_sync[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:entry[obj37]: bucket sync single entry (source_zone=3d163b1e-09c7-47e2-ae87-add4aace3f38) b=bucket sync single entry (source_zone=3d163b1e-09c7-47e2-ae87-add4aace3f38) b= log_entry=obj37 op=0 op_state=1 2023-05-31T20:54:52.919 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:54:52.905+0000 7f259334b780 5 RGW-SYNC:bucket[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24<-pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:full_sync[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:entry[obj37]: bucket sync: sync obj: 3d163b1e-09c7-47e2-ae87-add4aace3f38/:pklxkd-59[3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24])/obj37[0] 2023-05-31T20:54:52.919 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:54:52.905+0000 7f259334b780 20 RGW-SYNC:bucket[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24<-pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:full_sync[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]: [full sync] syncing object: pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0/obj38 2023-05-31T20:54:52.920 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:54:52.905+0000 7f259334b780 20 RGW-SYNC:bucket[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24<-pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:full_sync[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:entry[obj38]: bucket sync single entry (source_zone=3d163b1e-09c7-47e2-ae87-add4aace3f38) b=bucket sync single entry (source_zone=3d163b1e-09c7-47e2-ae87-add4aace3f38) b= log_entry=obj38 op=0 op_state=1 2023-05-31T20:54:52.920 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:54:52.905+0000 7f259334b780 5 RGW-SYNC:bucket[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24<-pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:full_sync[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:entry[obj38]: bucket sync: sync obj: 3d163b1e-09c7-47e2-ae87-add4aace3f38/:pklxkd-59[3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24])/obj38[0] 2023-05-31T20:54:52.920 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:54:52.905+0000 7f259334b780 20 RGW-SYNC:bucket[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24<-pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:full_sync[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]: [full sync] syncing object: pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0/obj44 2023-05-31T20:54:52.920 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:54:52.905+0000 7f259334b780 20 RGW-SYNC:bucket[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24<-pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:full_sync[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:entry[obj44]: bucket sync single entry (source_zone=3d163b1e-09c7-47e2-ae87-add4aace3f38) b=bucket sync single entry (source_zone=3d163b1e-09c7-47e2-ae87-add4aace3f38) b= log_entry=obj44 op=0 op_state=1 2023-05-31T20:54:52.920 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:54:52.905+0000 7f259334b780 5 RGW-SYNC:bucket[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24<-pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:full_sync[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:entry[obj44]: bucket sync: sync obj: 3d163b1e-09c7-47e2-ae87-add4aace3f38/:pklxkd-59[3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24])/obj44[0] 2023-05-31T20:54:52.920 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:54:52.905+0000 7f259334b780 20 RGW-SYNC:bucket[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24<-pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:full_sync[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]: [full sync] syncing object: pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0/obj46 2023-05-31T20:54:52.921 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:54:52.905+0000 7f259334b780 20 RGW-SYNC:bucket[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24<-pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:full_sync[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:entry[obj46]: bucket sync single entry (source_zone=3d163b1e-09c7-47e2-ae87-add4aace3f38) b=bucket sync single entry (source_zone=3d163b1e-09c7-47e2-ae87-add4aace3f38) b= log_entry=obj46 op=0 op_state=1 2023-05-31T20:54:52.921 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:54:52.905+0000 7f259334b780 5 RGW-SYNC:bucket[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24<-pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:full_sync[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:entry[obj46]: bucket sync: sync obj: 3d163b1e-09c7-47e2-ae87-add4aace3f38/:pklxkd-59[3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24])/obj46[0] 2023-05-31T20:54:52.921 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:54:52.908+0000 7f259334b780 10 RGW-SYNC:bucket[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24<-pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:full_sync[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:entry[obj11]: success 2023-05-31T20:54:52.921 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:54:52.908+0000 7f259334b780 20 RGW-SYNC:bucket[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24<-pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:full_sync[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]: [full sync] syncing object: pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0/obj51 2023-05-31T20:54:52.921 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:54:52.908+0000 7f259334b780 20 RGW-SYNC:bucket[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24<-pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:full_sync[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:entry[obj51]: bucket sync single entry (source_zone=3d163b1e-09c7-47e2-ae87-add4aace3f38) b=bucket sync single entry (source_zone=3d163b1e-09c7-47e2-ae87-add4aace3f38) b= log_entry=obj51 op=0 op_state=1 2023-05-31T20:54:52.921 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:54:52.908+0000 7f259334b780 5 RGW-SYNC:bucket[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24<-pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:full_sync[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:entry[obj51]: bucket sync: sync obj: 3d163b1e-09c7-47e2-ae87-add4aace3f38/:pklxkd-59[3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24])/obj51[0] 2023-05-31T20:54:52.921 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:54:52.908+0000 7f259334b780 10 RGW-SYNC:bucket[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24<-pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:full_sync[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:entry[obj116]: success 2023-05-31T20:54:52.922 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:54:52.908+0000 7f259334b780 10 RGW-SYNC:bucket[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24<-pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:full_sync[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:entry[obj124]: success 2023-05-31T20:54:52.922 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:54:52.908+0000 7f259334b780 20 RGW-SYNC:bucket[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24<-pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:full_sync[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]: [full sync] syncing object: pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0/obj55 2023-05-31T20:54:52.922 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:54:52.908+0000 7f259334b780 20 RGW-SYNC:bucket[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24<-pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:full_sync[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:entry[obj55]: bucket sync single entry (source_zone=3d163b1e-09c7-47e2-ae87-add4aace3f38) b=bucket sync single entry (source_zone=3d163b1e-09c7-47e2-ae87-add4aace3f38) b= log_entry=obj55 op=0 op_state=1 2023-05-31T20:54:52.922 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:54:52.908+0000 7f259334b780 5 RGW-SYNC:bucket[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24<-pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:full_sync[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:entry[obj55]: bucket sync: sync obj: 3d163b1e-09c7-47e2-ae87-add4aace3f38/:pklxkd-59[3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24])/obj55[0] 2023-05-31T20:54:52.922 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:54:52.908+0000 7f259334b780 20 RGW-SYNC:bucket[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24<-pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:full_sync[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]: [full sync] syncing object: pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0/obj57 2023-05-31T20:54:52.922 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:54:52.908+0000 7f259334b780 20 RGW-SYNC:bucket[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24<-pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:full_sync[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:entry[obj57]: bucket sync single entry (source_zone=3d163b1e-09c7-47e2-ae87-add4aace3f38) b=bucket sync single entry (source_zone=3d163b1e-09c7-47e2-ae87-add4aace3f38) b= log_entry=obj57 op=0 op_state=1 2023-05-31T20:54:52.922 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:54:52.908+0000 7f259334b780 5 RGW-SYNC:bucket[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24<-pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:full_sync[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:entry[obj57]: bucket sync: sync obj: 3d163b1e-09c7-47e2-ae87-add4aace3f38/:pklxkd-59[3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24])/obj57[0] 2023-05-31T20:54:52.923 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:54:52.909+0000 7f259334b780 10 RGW-SYNC:bucket[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24<-pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:full_sync[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:entry[obj111]: success 2023-05-31T20:54:52.923 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:54:52.909+0000 7f259334b780 20 RGW-SYNC:bucket[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24<-pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:full_sync[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]: [full sync] syncing object: pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0/obj58 2023-05-31T20:54:52.923 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:54:52.909+0000 7f259334b780 20 RGW-SYNC:bucket[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24<-pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:full_sync[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:entry[obj58]: bucket sync single entry (source_zone=3d163b1e-09c7-47e2-ae87-add4aace3f38) b=bucket sync single entry (source_zone=3d163b1e-09c7-47e2-ae87-add4aace3f38) b= log_entry=obj58 op=0 op_state=1 2023-05-31T20:54:52.923 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:54:52.909+0000 7f259334b780 5 RGW-SYNC:bucket[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24<-pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:full_sync[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:entry[obj58]: bucket sync: sync obj: 3d163b1e-09c7-47e2-ae87-add4aace3f38/:pklxkd-59[3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24])/obj58[0] 2023-05-31T20:54:52.923 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:54:52.910+0000 7f259334b780 10 RGW-SYNC:bucket[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24<-pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:full_sync[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:entry[obj23]: success 2023-05-31T20:54:52.923 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:54:52.910+0000 7f259334b780 20 RGW-SYNC:bucket[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24<-pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:full_sync[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]: [full sync] syncing object: pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0/obj62 2023-05-31T20:54:52.924 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:54:52.910+0000 7f259334b780 20 RGW-SYNC:bucket[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24<-pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:full_sync[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:entry[obj62]: bucket sync single entry (source_zone=3d163b1e-09c7-47e2-ae87-add4aace3f38) b=bucket sync single entry (source_zone=3d163b1e-09c7-47e2-ae87-add4aace3f38) b= log_entry=obj62 op=0 op_state=1 2023-05-31T20:54:52.924 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:54:52.910+0000 7f259334b780 5 RGW-SYNC:bucket[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24<-pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:full_sync[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:entry[obj62]: bucket sync: sync obj: 3d163b1e-09c7-47e2-ae87-add4aace3f38/:pklxkd-59[3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24])/obj62[0] 2023-05-31T20:54:52.924 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:54:52.910+0000 7f259334b780 10 RGW-SYNC:bucket[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24<-pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:full_sync[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:entry[obj148]: success 2023-05-31T20:54:52.924 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:54:52.910+0000 7f259334b780 20 RGW-SYNC:bucket[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24<-pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:full_sync[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]: [full sync] syncing object: pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0/obj68 2023-05-31T20:54:52.924 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:54:52.910+0000 7f259334b780 20 RGW-SYNC:bucket[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24<-pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:full_sync[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:entry[obj68]: bucket sync single entry (source_zone=3d163b1e-09c7-47e2-ae87-add4aace3f38) b=bucket sync single entry (source_zone=3d163b1e-09c7-47e2-ae87-add4aace3f38) b= log_entry=obj68 op=0 op_state=1 2023-05-31T20:54:52.924 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:54:52.910+0000 7f259334b780 5 RGW-SYNC:bucket[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24<-pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:full_sync[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:entry[obj68]: bucket sync: sync obj: 3d163b1e-09c7-47e2-ae87-add4aace3f38/:pklxkd-59[3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24])/obj68[0] 2023-05-31T20:54:52.925 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:54:52.910+0000 7f259334b780 10 RGW-SYNC:bucket[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24<-pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:full_sync[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:entry[obj115]: success 2023-05-31T20:54:52.925 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:54:52.910+0000 7f259334b780 10 RGW-SYNC:bucket[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24<-pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:full_sync[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:entry[obj38]: success 2023-05-31T20:54:52.925 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:54:52.910+0000 7f259334b780 10 RGW-SYNC:bucket[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24<-pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:full_sync[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:entry[obj44]: success 2023-05-31T20:54:52.925 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:54:52.910+0000 7f259334b780 10 RGW-SYNC:bucket[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24<-pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:full_sync[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:entry[obj46]: success 2023-05-31T20:54:52.925 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:54:52.910+0000 7f259334b780 10 RGW-SYNC:bucket[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24<-pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:full_sync[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:entry[obj34]: success 2023-05-31T20:54:52.925 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:54:52.911+0000 7f259334b780 10 RGW-SYNC:bucket[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24<-pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:full_sync[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:entry[obj185]: success 2023-05-31T20:54:52.925 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:54:52.911+0000 7f259334b780 10 RGW-SYNC:bucket[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24<-pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:full_sync[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:entry[obj145]: success 2023-05-31T20:54:52.926 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:54:52.911+0000 7f259334b780 20 RGW-SYNC:bucket[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24<-pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:full_sync[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]: updating marker oid=bucket.full-sync-status.3d163b1e-09c7-47e2-ae87-add4aace3f38:pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24 marker=obj148 2023-05-31T20:54:52.926 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:54:52.911+0000 7f259334b780 10 RGW-SYNC:bucket[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24<-pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:full_sync[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:entry[obj29]: success 2023-05-31T20:54:52.926 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:54:52.911+0000 7f259334b780 10 RGW-SYNC:bucket[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24<-pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:full_sync[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:entry[obj37]: success 2023-05-31T20:54:52.926 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:54:52.911+0000 7f259334b780 20 RGW-SYNC:bucket[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24<-pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:full_sync[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]: [full sync] syncing object: pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0/obj69 2023-05-31T20:54:52.926 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:54:52.911+0000 7f259334b780 20 RGW-SYNC:bucket[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24<-pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:full_sync[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:entry[obj69]: bucket sync single entry (source_zone=3d163b1e-09c7-47e2-ae87-add4aace3f38) b=bucket sync single entry (source_zone=3d163b1e-09c7-47e2-ae87-add4aace3f38) b= log_entry=obj69 op=0 op_state=1 2023-05-31T20:54:52.926 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:54:52.911+0000 7f259334b780 5 RGW-SYNC:bucket[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24<-pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:full_sync[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:entry[obj69]: bucket sync: sync obj: 3d163b1e-09c7-47e2-ae87-add4aace3f38/:pklxkd-59[3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24])/obj69[0] 2023-05-31T20:54:52.926 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:54:52.911+0000 7f259334b780 20 RGW-SYNC:bucket[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24<-pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:full_sync[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]: [full sync] syncing object: pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0/obj74 2023-05-31T20:54:52.927 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:54:52.911+0000 7f259334b780 20 RGW-SYNC:bucket[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24<-pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:full_sync[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:entry[obj74]: bucket sync single entry (source_zone=3d163b1e-09c7-47e2-ae87-add4aace3f38) b=bucket sync single entry (source_zone=3d163b1e-09c7-47e2-ae87-add4aace3f38) b= log_entry=obj74 op=0 op_state=1 2023-05-31T20:54:52.927 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:54:52.911+0000 7f259334b780 5 RGW-SYNC:bucket[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24<-pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:full_sync[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:entry[obj74]: bucket sync: sync obj: 3d163b1e-09c7-47e2-ae87-add4aace3f38/:pklxkd-59[3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24])/obj74[0] 2023-05-31T20:54:52.927 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:54:52.911+0000 7f259334b780 20 RGW-SYNC:bucket[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24<-pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:full_sync[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]: [full sync] syncing object: pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0/obj76 2023-05-31T20:54:52.927 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:54:52.911+0000 7f259334b780 20 RGW-SYNC:bucket[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24<-pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:full_sync[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:entry[obj76]: bucket sync single entry (source_zone=3d163b1e-09c7-47e2-ae87-add4aace3f38) b=bucket sync single entry (source_zone=3d163b1e-09c7-47e2-ae87-add4aace3f38) b= log_entry=obj76 op=0 op_state=1 2023-05-31T20:54:52.927 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:54:52.911+0000 7f259334b780 5 RGW-SYNC:bucket[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24<-pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:full_sync[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:entry[obj76]: bucket sync: sync obj: 3d163b1e-09c7-47e2-ae87-add4aace3f38/:pklxkd-59[3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24])/obj76[0] 2023-05-31T20:54:52.928 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:54:52.911+0000 7f259334b780 20 RGW-SYNC:bucket[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24<-pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:full_sync[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]: [full sync] syncing object: pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0/obj85 2023-05-31T20:54:52.928 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:54:52.911+0000 7f259334b780 20 RGW-SYNC:bucket[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24<-pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:full_sync[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:entry[obj85]: bucket sync single entry (source_zone=3d163b1e-09c7-47e2-ae87-add4aace3f38) b=bucket sync single entry (source_zone=3d163b1e-09c7-47e2-ae87-add4aace3f38) b= log_entry=obj85 op=0 op_state=1 2023-05-31T20:54:52.928 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:54:52.911+0000 7f259334b780 5 RGW-SYNC:bucket[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24<-pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:full_sync[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:entry[obj85]: bucket sync: sync obj: 3d163b1e-09c7-47e2-ae87-add4aace3f38/:pklxkd-59[3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24])/obj85[0] 2023-05-31T20:54:52.928 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:54:52.911+0000 7f259334b780 20 RGW-SYNC:bucket[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24<-pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:full_sync[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]: [full sync] syncing object: pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0/obj87 2023-05-31T20:54:52.928 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:54:52.911+0000 7f259334b780 20 RGW-SYNC:bucket[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24<-pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:full_sync[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:entry[obj87]: bucket sync single entry (source_zone=3d163b1e-09c7-47e2-ae87-add4aace3f38) b=bucket sync single entry (source_zone=3d163b1e-09c7-47e2-ae87-add4aace3f38) b= log_entry=obj87 op=0 op_state=1 2023-05-31T20:54:52.928 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:54:52.911+0000 7f259334b780 5 RGW-SYNC:bucket[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24<-pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:full_sync[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:entry[obj87]: bucket sync: sync obj: 3d163b1e-09c7-47e2-ae87-add4aace3f38/:pklxkd-59[3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24])/obj87[0] 2023-05-31T20:54:52.929 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:54:52.911+0000 7f259334b780 20 RGW-SYNC:bucket[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24<-pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:full_sync[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]: [full sync] syncing object: pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0/obj92 2023-05-31T20:54:52.929 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:54:52.911+0000 7f259334b780 20 RGW-SYNC:bucket[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24<-pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:full_sync[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:entry[obj92]: bucket sync single entry (source_zone=3d163b1e-09c7-47e2-ae87-add4aace3f38) b=bucket sync single entry (source_zone=3d163b1e-09c7-47e2-ae87-add4aace3f38) b= log_entry=obj92 op=0 op_state=1 2023-05-31T20:54:52.929 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:54:52.911+0000 7f259334b780 5 RGW-SYNC:bucket[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24<-pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:full_sync[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:entry[obj92]: bucket sync: sync obj: 3d163b1e-09c7-47e2-ae87-add4aace3f38/:pklxkd-59[3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24])/obj92[0] 2023-05-31T20:54:52.929 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:54:52.911+0000 7f259334b780 20 RGW-SYNC:bucket[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24<-pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:full_sync[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]: [full sync] syncing object: pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0/obj93 2023-05-31T20:54:52.929 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:54:52.911+0000 7f259334b780 20 RGW-SYNC:bucket[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24<-pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:full_sync[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:entry[obj93]: bucket sync single entry (source_zone=3d163b1e-09c7-47e2-ae87-add4aace3f38) b=bucket sync single entry (source_zone=3d163b1e-09c7-47e2-ae87-add4aace3f38) b= log_entry=obj93 op=0 op_state=1 2023-05-31T20:54:52.929 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:54:52.911+0000 7f259334b780 5 RGW-SYNC:bucket[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24<-pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:full_sync[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:entry[obj93]: bucket sync: sync obj: 3d163b1e-09c7-47e2-ae87-add4aace3f38/:pklxkd-59[3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24])/obj93[0] 2023-05-31T20:54:52.929 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:54:52.911+0000 7f259334b780 20 RGW-SYNC:bucket[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24<-pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:full_sync[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]: [full sync] syncing object: pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0/obj95 2023-05-31T20:54:52.930 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:54:52.911+0000 7f259334b780 20 RGW-SYNC:bucket[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24<-pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:full_sync[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:entry[obj95]: bucket sync single entry (source_zone=3d163b1e-09c7-47e2-ae87-add4aace3f38) b=bucket sync single entry (source_zone=3d163b1e-09c7-47e2-ae87-add4aace3f38) b= log_entry=obj95 op=0 op_state=1 2023-05-31T20:54:52.930 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:54:52.911+0000 7f259334b780 5 RGW-SYNC:bucket[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24<-pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:full_sync[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:entry[obj95]: bucket sync: sync obj: 3d163b1e-09c7-47e2-ae87-add4aace3f38/:pklxkd-59[3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24])/obj95[0] 2023-05-31T20:54:52.933 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:54:52.913+0000 7f259334b780 10 RGW-SYNC:bucket[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24<-pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:full_sync[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:entry[obj31]: success 2023-05-31T20:54:52.933 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:54:52.913+0000 7f259334b780 10 RGW-SYNC:bucket[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24<-pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:full_sync[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:entry[obj155]: success 2023-05-31T20:54:52.933 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:54:52.913+0000 7f259334b780 10 RGW-SYNC:bucket[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24<-pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:full_sync[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:entry[obj55]: success 2023-05-31T20:54:52.933 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:54:52.913+0000 7f259334b780 10 RGW-SYNC:bucket[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24<-pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:full_sync[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:entry[obj33]: success 2023-05-31T20:54:52.934 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:54:52.913+0000 7f259334b780 10 RGW-SYNC:bucket[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24<-pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:full_sync[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:entry[obj22]: success 2023-05-31T20:54:52.934 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:54:52.913+0000 7f259334b780 10 RGW-SYNC:bucket[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24<-pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:full_sync[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:entry[obj17]: success 2023-05-31T20:54:52.934 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:54:52.913+0000 7f259334b780 10 RGW-SYNC:bucket[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24<-pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:full_sync[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:entry[obj57]: success 2023-05-31T20:54:52.934 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:54:52.913+0000 7f259334b780 10 RGW-SYNC:bucket[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24<-pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:full_sync[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:entry[obj19]: success 2023-05-31T20:54:52.934 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:54:52.913+0000 7f259334b780 20 RGW-SYNC:bucket[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24<-pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:full_sync[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]: updating marker oid=bucket.full-sync-status.3d163b1e-09c7-47e2-ae87-add4aace3f38:pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24 marker=obj46 2023-05-31T20:54:52.934 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:54:52.915+0000 7f259334b780 10 RGW-SYNC:bucket[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24<-pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:full_sync[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:entry[obj51]: success 2023-05-31T20:54:52.934 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:54:52.915+0000 7f259334b780 10 RGW-SYNC:bucket[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24<-pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:full_sync[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:entry[obj68]: success 2023-05-31T20:54:52.935 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:54:52.915+0000 7f259334b780 10 RGW-SYNC:bucket[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24<-pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:full_sync[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:entry[obj62]: success 2023-05-31T20:54:52.935 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:54:52.915+0000 7f259334b780 10 RGW-SYNC:bucket[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24<-pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:full_sync[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:entry[obj69]: success 2023-05-31T20:54:52.935 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:54:52.915+0000 7f259334b780 10 RGW-SYNC:bucket[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24<-pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:full_sync[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:entry[obj74]: success 2023-05-31T20:54:52.935 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:54:52.915+0000 7f259334b780 10 RGW-SYNC:bucket[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24<-pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:full_sync[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:entry[obj92]: success 2023-05-31T20:54:52.935 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:54:52.915+0000 7f259334b780 10 RGW-SYNC:bucket[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24<-pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:full_sync[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:entry[obj87]: success 2023-05-31T20:54:52.935 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:54:52.915+0000 7f259334b780 10 RGW-SYNC:bucket[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24<-pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:full_sync[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:entry[obj76]: success 2023-05-31T20:54:52.935 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:54:52.915+0000 7f259334b780 10 RGW-SYNC:bucket[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24<-pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:full_sync[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:entry[obj93]: success 2023-05-31T20:54:52.936 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:54:52.916+0000 7f259334b780 10 RGW-SYNC:bucket[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24<-pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:full_sync[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:entry[obj58]: success 2023-05-31T20:54:52.936 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:54:52.916+0000 7f259334b780 20 RGW-SYNC:bucket[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24<-pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:full_sync[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]: updating marker oid=bucket.full-sync-status.3d163b1e-09c7-47e2-ae87-add4aace3f38:pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24 marker=obj76 2023-05-31T20:54:52.936 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:54:52.916+0000 7f259334b780 10 RGW-SYNC:bucket[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24<-pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:full_sync[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:entry[obj95]: success 2023-05-31T20:54:52.936 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:54:52.916+0000 7f259334b780 10 RGW-SYNC:bucket[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24<-pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:full_sync[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:entry[obj85]: success 2023-05-31T20:54:52.936 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:54:52.916+0000 7f259334b780 20 RGW-SYNC:bucket[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24<-pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:full_sync[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]: updating marker oid=bucket.full-sync-status.3d163b1e-09c7-47e2-ae87-add4aace3f38:pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24 marker=obj95 2023-05-31T20:54:52.936 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:54:52.918+0000 7f259334b780 5 RGW-SYNC:bucket[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24<-pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:full_sync[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]: set bucket state=incremental 2023-05-31T20:54:52.937 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:54:52.919+0000 7f259334b780 5 RGW-SYNC:bucket[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24<-pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:full_sync[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]: bucket status objv={r=_cuC9z89M-x5ERndAlfJizr3:158,w=:0} 2023-05-31T20:54:52.937 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:54:52.920+0000 7f259334b780 10 RGW-SYNC:bucket[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24<-pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]: WARNING: requested sync of past generation 0 < 6, returning success 2023-05-31T20:54:52.937 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:54:52.922+0000 7f259334b780 20 RGW-SYNC:bucket[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24<-pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:1]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-05-31T20:54:52.937 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:54:52.922+0000 7f259334b780 20 RGW-SYNC:bucket[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24<-pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:2]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-05-31T20:54:52.937 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:54:52.922+0000 7f259334b780 20 RGW-SYNC:bucket[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24<-pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-05-31T20:54:52.937 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:54:52.922+0000 7f259334b780 20 RGW-SYNC:bucket[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24<-pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:5]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-05-31T20:54:52.937 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:54:52.923+0000 7f259334b780 20 RGW-SYNC:bucket[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24<-pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:4]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-05-31T20:54:52.938 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:54:52.923+0000 7f259334b780 20 RGW-SYNC:bucket[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24<-pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]: sync status for source bucket shard: 2 2023-05-31T20:54:52.938 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:54:52.923+0000 7f259334b780 20 RGW-SYNC:bucket[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24<-pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]:inc_sync[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:0]: listing bilog for incremental sync; position= 2023-05-31T20:54:52.938 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:54:52.923+0000 7f259334b780 20 RGW-SYNC:bucket[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24<-pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:2]: sync status for source bucket shard: 2 2023-05-31T20:54:52.938 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:54:52.923+0000 7f259334b780 20 RGW-SYNC:bucket[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24<-pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:2]:inc_sync[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:2]: listing bilog for incremental sync; position= 2023-05-31T20:54:52.938 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:54:52.923+0000 7f259334b780 20 RGW-SYNC:bucket[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24<-pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:3]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-05-31T20:54:52.938 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:54:52.923+0000 7f259334b780 20 RGW-SYNC:bucket[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24<-pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:1]: sync status for source bucket shard: 2 2023-05-31T20:54:52.939 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:54:52.923+0000 7f259334b780 20 RGW-SYNC:bucket[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24<-pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:5]: sync status for source bucket shard: 2 2023-05-31T20:54:52.939 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:54:52.923+0000 7f259334b780 20 RGW-SYNC:bucket[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24<-pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:1]:inc_sync[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:1]: listing bilog for incremental sync; position= 2023-05-31T20:54:52.939 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:54:52.923+0000 7f259334b780 20 RGW-SYNC:bucket[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24<-pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:5]:inc_sync[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:5]: listing bilog for incremental sync; position= 2023-05-31T20:54:52.939 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:54:52.923+0000 7f259334b780 20 RGW-SYNC:bucket[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24<-pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:6]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-05-31T20:54:52.939 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:54:52.923+0000 7f259334b780 20 RGW-SYNC:bucket[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24<-pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:4]: sync status for source bucket shard: 2 2023-05-31T20:54:52.939 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:54:52.923+0000 7f259334b780 20 RGW-SYNC:bucket[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24<-pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:4]:inc_sync[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:4]: listing bilog for incremental sync; position= 2023-05-31T20:54:52.939 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:54:52.924+0000 7f259334b780 20 RGW-SYNC:bucket[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24<-pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:7]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-05-31T20:54:52.940 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:54:52.924+0000 7f259334b780 20 RGW-SYNC:bucket[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24<-pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:9]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-05-31T20:54:52.940 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:54:52.924+0000 7f259334b780 20 RGW-SYNC:bucket[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24<-pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:8]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-05-31T20:54:52.940 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:54:52.924+0000 7f259334b780 20 RGW-SYNC:bucket[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24<-pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:10]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-05-31T20:54:52.940 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:54:52.924+0000 7f259334b780 20 RGW-SYNC:bucket[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24<-pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:11]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-05-31T20:54:52.940 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:54:52.924+0000 7f259334b780 20 RGW-SYNC:bucket[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24<-pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:12]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-05-31T20:54:52.940 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:54:52.924+0000 7f259334b780 20 RGW-SYNC:bucket[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24<-pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:3]: sync status for source bucket shard: 2 2023-05-31T20:54:52.941 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:54:52.924+0000 7f259334b780 20 RGW-SYNC:bucket[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24<-pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:3]:inc_sync[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:3]: listing bilog for incremental sync; position= 2023-05-31T20:54:52.941 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:54:52.925+0000 7f259334b780 20 RGW-SYNC:bucket[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24<-pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:6]: sync status for source bucket shard: 2 2023-05-31T20:54:52.941 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:54:52.925+0000 7f259334b780 20 RGW-SYNC:bucket[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24<-pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:6]:inc_sync[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:6]: listing bilog for incremental sync; position=00000000002.48.6 2023-05-31T20:54:52.941 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:54:52.925+0000 7f259334b780 20 RGW-SYNC:bucket[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24<-pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:13]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-05-31T20:54:52.941 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:54:52.925+0000 7f259334b780 20 RGW-SYNC:bucket[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24<-pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:15]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-05-31T20:54:52.941 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:54:52.925+0000 7f259334b780 20 RGW-SYNC:bucket[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24<-pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:14]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-05-31T20:54:52.941 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:54:52.925+0000 7f259334b780 20 RGW-SYNC:bucket[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24<-pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:16]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-05-31T20:54:52.942 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:54:52.925+0000 7f259334b780 20 RGW-SYNC:bucket[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24<-pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:17]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-05-31T20:54:52.942 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:54:52.925+0000 7f259334b780 20 RGW-SYNC:bucket[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24<-pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:11]: sync status for source bucket shard: 2 2023-05-31T20:54:52.942 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:54:52.925+0000 7f259334b780 20 RGW-SYNC:bucket[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24<-pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:15]: sync status for source bucket shard: 2 2023-05-31T20:54:52.942 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:54:52.925+0000 7f259334b780 20 RGW-SYNC:bucket[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24<-pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:11]:inc_sync[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:11]: listing bilog for incremental sync; position= 2023-05-31T20:54:52.942 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:54:52.925+0000 7f259334b780 20 RGW-SYNC:bucket[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24<-pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:15]:inc_sync[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:15]: listing bilog for incremental sync; position= 2023-05-31T20:54:52.942 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:54:52.926+0000 7f259334b780 20 RGW-SYNC:bucket[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24<-pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:20]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-05-31T20:54:52.942 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:54:52.926+0000 7f259334b780 20 RGW-SYNC:bucket[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24<-pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:7]: sync status for source bucket shard: 2 2023-05-31T20:54:52.943 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:54:52.926+0000 7f259334b780 20 RGW-SYNC:bucket[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24<-pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:9]: sync status for source bucket shard: 2 2023-05-31T20:54:52.943 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:54:52.926+0000 7f259334b780 20 RGW-SYNC:bucket[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24<-pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:7]:inc_sync[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:7]: listing bilog for incremental sync; position= 2023-05-31T20:54:52.943 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:54:52.926+0000 7f259334b780 20 RGW-SYNC:bucket[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24<-pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:9]:inc_sync[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:9]: listing bilog for incremental sync; position= 2023-05-31T20:54:52.943 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:54:52.926+0000 7f259334b780 20 RGW-SYNC:bucket[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24<-pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:10]: sync status for source bucket shard: 2 2023-05-31T20:54:52.943 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:54:52.926+0000 7f259334b780 20 RGW-SYNC:bucket[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24<-pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:8]: sync status for source bucket shard: 2 2023-05-31T20:54:52.943 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:54:52.926+0000 7f259334b780 20 RGW-SYNC:bucket[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24<-pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:13]: sync status for source bucket shard: 2 2023-05-31T20:54:52.943 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:54:52.926+0000 7f259334b780 20 RGW-SYNC:bucket[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24<-pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:10]:inc_sync[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:10]: listing bilog for incremental sync; position= 2023-05-31T20:54:52.944 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:54:52.926+0000 7f259334b780 20 RGW-SYNC:bucket[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24<-pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:8]:inc_sync[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:8]: listing bilog for incremental sync; position= 2023-05-31T20:54:52.944 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:54:52.926+0000 7f259334b780 20 RGW-SYNC:bucket[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24<-pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:13]:inc_sync[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:13]: listing bilog for incremental sync; position= 2023-05-31T20:54:52.944 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:54:52.927+0000 7f259334b780 20 RGW-SYNC:bucket[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24<-pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:18]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-05-31T20:54:52.944 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:54:52.927+0000 7f259334b780 20 RGW-SYNC:bucket[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24<-pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:19]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-05-31T20:54:52.944 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:54:52.927+0000 7f259334b780 20 RGW-SYNC:bucket[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24<-pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:21]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-05-31T20:54:52.944 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:54:52.927+0000 7f259334b780 20 RGW-SYNC:bucket[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24<-pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:22]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-05-31T20:54:52.944 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:54:52.927+0000 7f259334b780 20 RGW-SYNC:bucket[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24<-pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:23]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-05-31T20:54:52.945 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:54:52.927+0000 7f259334b780 20 RGW-SYNC:bucket[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24<-pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:24]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-05-31T20:54:52.945 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:54:52.927+0000 7f259334b780 20 RGW-SYNC:bucket[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24<-pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:25]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-05-31T20:54:52.945 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:54:52.927+0000 7f259334b780 20 RGW-SYNC:bucket[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24<-pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:26]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-05-31T20:54:52.945 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:54:52.928+0000 7f259334b780 20 RGW-SYNC:bucket[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24<-pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:17]: sync status for source bucket shard: 2 2023-05-31T20:54:52.945 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:54:52.928+0000 7f259334b780 20 RGW-SYNC:bucket[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24<-pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:17]:inc_sync[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:17]: listing bilog for incremental sync; position=00000000002.63.6 2023-05-31T20:54:52.947 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:54:52.928+0000 7f259334b780 20 RGW-SYNC:bucket[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24<-pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:27]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-05-31T20:54:52.947 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:54:52.928+0000 7f259334b780 20 RGW-SYNC:bucket[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24<-pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:20]: sync status for source bucket shard: 2 2023-05-31T20:54:52.947 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:54:52.928+0000 7f259334b780 20 RGW-SYNC:bucket[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24<-pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:20]:inc_sync[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:20]: listing bilog for incremental sync; position= 2023-05-31T20:54:52.948 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:54:52.928+0000 7f259334b780 20 RGW-SYNC:bucket[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24<-pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:28]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-05-31T20:54:52.948 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:54:52.929+0000 7f259334b780 20 RGW-SYNC:bucket[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24<-pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:29]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-05-31T20:54:52.948 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:54:52.929+0000 7f259334b780 20 RGW-SYNC:bucket[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24<-pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:30]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-05-31T20:54:52.948 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:54:52.929+0000 7f259334b780 20 RGW-SYNC:bucket[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24<-pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:23]: sync status for source bucket shard: 2 2023-05-31T20:54:52.948 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:54:52.929+0000 7f259334b780 20 RGW-SYNC:bucket[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24<-pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:19]: sync status for source bucket shard: 2 2023-05-31T20:54:52.948 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:54:52.929+0000 7f259334b780 20 RGW-SYNC:bucket[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24<-pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:24]: sync status for source bucket shard: 2 2023-05-31T20:54:52.949 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:54:52.929+0000 7f259334b780 20 RGW-SYNC:bucket[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24<-pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:26]: sync status for source bucket shard: 2 2023-05-31T20:54:52.949 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:54:52.929+0000 7f259334b780 20 RGW-SYNC:bucket[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24<-pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:23]:inc_sync[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:23]: listing bilog for incremental sync; position= 2023-05-31T20:54:52.949 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:54:52.929+0000 7f259334b780 20 RGW-SYNC:bucket[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24<-pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:19]:inc_sync[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:19]: listing bilog for incremental sync; position= 2023-05-31T20:54:52.949 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:54:52.929+0000 7f259334b780 20 RGW-SYNC:bucket[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24<-pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:24]:inc_sync[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:24]: listing bilog for incremental sync; position= 2023-05-31T20:54:52.949 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:54:52.929+0000 7f259334b780 20 RGW-SYNC:bucket[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24<-pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:25]: sync status for source bucket shard: 2 2023-05-31T20:54:52.949 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:54:52.929+0000 7f259334b780 20 RGW-SYNC:bucket[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24<-pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:26]:inc_sync[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:26]: listing bilog for incremental sync; position= 2023-05-31T20:54:52.949 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:54:52.929+0000 7f259334b780 20 RGW-SYNC:bucket[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24<-pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:25]:inc_sync[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:25]: listing bilog for incremental sync; position= 2023-05-31T20:54:52.950 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:54:52.929+0000 7f259334b780 20 RGW-SYNC:bucket[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24<-pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:31]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-05-31T20:54:52.950 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:54:52.929+0000 7f259334b780 20 RGW-SYNC:bucket[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24<-pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:32]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-05-31T20:54:52.950 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:54:52.930+0000 7f259334b780 20 RGW-SYNC:bucket[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24<-pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:21]: sync status for source bucket shard: 2 2023-05-31T20:54:52.950 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:54:52.930+0000 7f259334b780 20 RGW-SYNC:bucket[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24<-pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:21]:inc_sync[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:21]: listing bilog for incremental sync; position= 2023-05-31T20:54:52.950 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:54:52.930+0000 7f259334b780 20 RGW-SYNC:bucket[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24<-pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:22]: sync status for source bucket shard: 2 2023-05-31T20:54:52.950 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:54:52.930+0000 7f259334b780 20 RGW-SYNC:bucket[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24<-pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:28]: sync status for source bucket shard: 2 2023-05-31T20:54:52.951 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:54:52.930+0000 7f259334b780 20 RGW-SYNC:bucket[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24<-pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:27]: sync status for source bucket shard: 2 2023-05-31T20:54:52.951 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:54:52.930+0000 7f259334b780 20 RGW-SYNC:bucket[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24<-pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:22]:inc_sync[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:22]: listing bilog for incremental sync; position= 2023-05-31T20:54:52.951 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:54:52.930+0000 7f259334b780 20 RGW-SYNC:bucket[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24<-pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:28]:inc_sync[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:28]: listing bilog for incremental sync; position= 2023-05-31T20:54:52.951 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:54:52.930+0000 7f259334b780 20 RGW-SYNC:bucket[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24<-pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:27]:inc_sync[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:27]: listing bilog for incremental sync; position= 2023-05-31T20:54:52.951 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:54:52.930+0000 7f259334b780 20 RGW-SYNC:bucket[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24<-pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:29]: sync status for source bucket shard: 2 2023-05-31T20:54:52.951 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:54:52.930+0000 7f259334b780 20 RGW-SYNC:bucket[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24<-pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:29]:inc_sync[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:29]: listing bilog for incremental sync; position= 2023-05-31T20:54:52.951 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:54:52.931+0000 7f259334b780 20 RGW-SYNC:bucket[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24<-pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:30]: sync status for source bucket shard: 2 2023-05-31T20:54:52.952 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:54:52.931+0000 7f259334b780 20 RGW-SYNC:bucket[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24<-pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:30]:inc_sync[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:30]: listing bilog for incremental sync; position= 2023-05-31T20:54:52.952 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:54:52.931+0000 7f259334b780 20 RGW-SYNC:bucket[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24<-pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:32]: sync status for source bucket shard: 2 2023-05-31T20:54:52.952 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:54:52.931+0000 7f259334b780 20 RGW-SYNC:bucket[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24<-pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:31]: sync status for source bucket shard: 2 2023-05-31T20:54:52.952 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:54:52.931+0000 7f259334b780 20 RGW-SYNC:bucket[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24<-pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:32]:inc_sync[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:32]: listing bilog for incremental sync; position=00000000001.75.6 2023-05-31T20:54:52.952 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:54:52.931+0000 7f259334b780 20 RGW-SYNC:bucket[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24<-pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:31]:inc_sync[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:31]: listing bilog for incremental sync; position= 2023-05-31T20:54:52.953 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:54:52.931+0000 7f259334b780 20 RGW-SYNC:bucket[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24<-pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:33]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-05-31T20:54:52.953 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:54:52.931+0000 7f259334b780 20 RGW-SYNC:bucket[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24<-pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:34]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-05-31T20:54:52.953 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:54:52.931+0000 7f259334b780 20 RGW-SYNC:bucket[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24<-pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:35]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-05-31T20:54:52.953 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:54:52.931+0000 7f259334b780 20 RGW-SYNC:bucket[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24<-pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:36]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-05-31T20:54:52.953 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:54:52.931+0000 7f259334b780 20 RGW-SYNC:bucket[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24<-pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:14]: sync status for source bucket shard: 2 2023-05-31T20:54:52.953 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:54:52.931+0000 7f259334b780 20 RGW-SYNC:bucket[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24<-pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:16]: sync status for source bucket shard: 2 2023-05-31T20:54:52.954 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:54:52.931+0000 7f259334b780 20 RGW-SYNC:bucket[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24<-pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:12]: sync status for source bucket shard: 2 2023-05-31T20:54:52.954 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:54:52.931+0000 7f259334b780 20 RGW-SYNC:bucket[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24<-pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:14]:inc_sync[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:14]: listing bilog for incremental sync; position= 2023-05-31T20:54:52.954 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:54:52.931+0000 7f259334b780 20 RGW-SYNC:bucket[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24<-pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:16]:inc_sync[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:16]: listing bilog for incremental sync; position= 2023-05-31T20:54:52.954 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:54:52.931+0000 7f259334b780 20 RGW-SYNC:bucket[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24<-pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:12]:inc_sync[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:12]: listing bilog for incremental sync; position=00000000002.68.6 2023-05-31T20:54:52.954 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:54:52.932+0000 7f259334b780 20 RGW-SYNC:bucket[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24<-pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:18]: sync status for source bucket shard: 2 2023-05-31T20:54:52.954 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:54:52.932+0000 7f259334b780 20 RGW-SYNC:bucket[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24<-pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:18]:inc_sync[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:18]: listing bilog for incremental sync; position= 2023-05-31T20:54:52.954 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:54:52.932+0000 7f259334b780 20 RGW-SYNC:bucket[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24<-pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:33]: sync status for source bucket shard: 2 2023-05-31T20:54:52.955 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:54:52.932+0000 7f259334b780 20 RGW-SYNC:bucket[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24<-pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:35]: sync status for source bucket shard: 2 2023-05-31T20:54:52.955 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:54:52.932+0000 7f259334b780 20 RGW-SYNC:bucket[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24<-pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:34]: sync status for source bucket shard: 2 2023-05-31T20:54:52.955 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:54:52.932+0000 7f259334b780 20 RGW-SYNC:bucket[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24<-pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:36]: sync status for source bucket shard: 2 2023-05-31T20:54:52.955 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:54:52.932+0000 7f259334b780 20 RGW-SYNC:bucket[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24<-pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:33]:inc_sync[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:33]: listing bilog for incremental sync; position= 2023-05-31T20:54:52.955 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:54:52.932+0000 7f259334b780 20 RGW-SYNC:bucket[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24<-pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:35]:inc_sync[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:35]: listing bilog for incremental sync; position= 2023-05-31T20:54:52.955 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:54:52.932+0000 7f259334b780 20 RGW-SYNC:bucket[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24<-pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:34]:inc_sync[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:34]: listing bilog for incremental sync; position= 2023-05-31T20:54:52.955 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:54:52.932+0000 7f259334b780 20 RGW-SYNC:bucket[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24<-pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:36]:inc_sync[pklxkd-59:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.24:36]: listing bilog for incremental sync; position=00000000001.83.6 2023-05-31T20:54:53.003 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'list', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'pklxkd-59', '--max-entries', '10000', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-31T20:54:53.003 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'list', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'pklxkd-59', '--max-entries', '10000', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-31T20:54:53.004 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket list --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --bucket pklxkd-59 --max-entries 10000 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-05-31T20:54:53.094 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2023-05-31T20:54:53.095 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2023-05-31T20:54:53.143 INFO:teuthology.orchestra.run.smithi099.stdout:[ 2023-05-31T20:54:53.143 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T20:54:53.143 INFO:teuthology.orchestra.run.smithi099.stdout: "name": "obj11", 2023-05-31T20:54:53.143 INFO:teuthology.orchestra.run.smithi099.stdout: "instance": "", 2023-05-31T20:54:53.143 INFO:teuthology.orchestra.run.smithi099.stdout: "ver": { 2023-05-31T20:54:53.144 INFO:teuthology.orchestra.run.smithi099.stdout: "pool": 9, 2023-05-31T20:54:53.144 INFO:teuthology.orchestra.run.smithi099.stdout: "epoch": 22 2023-05-31T20:54:53.144 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:54:53.144 INFO:teuthology.orchestra.run.smithi099.stdout: "locator": "", 2023-05-31T20:54:53.144 INFO:teuthology.orchestra.run.smithi099.stdout: "exists": true, 2023-05-31T20:54:53.144 INFO:teuthology.orchestra.run.smithi099.stdout: "meta": { 2023-05-31T20:54:53.145 INFO:teuthology.orchestra.run.smithi099.stdout: "category": 1, 2023-05-31T20:54:53.145 INFO:teuthology.orchestra.run.smithi099.stdout: "size": 16, 2023-05-31T20:54:53.145 INFO:teuthology.orchestra.run.smithi099.stdout: "mtime": "2023-05-31T20:39:26.427662Z", 2023-05-31T20:54:53.145 INFO:teuthology.orchestra.run.smithi099.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-05-31T20:54:53.145 INFO:teuthology.orchestra.run.smithi099.stdout: "storage_class": "", 2023-05-31T20:54:53.145 INFO:teuthology.orchestra.run.smithi099.stdout: "owner": "rgw-multisite-test-user", 2023-05-31T20:54:53.145 INFO:teuthology.orchestra.run.smithi099.stdout: "owner_display_name": "Multisite Test User", 2023-05-31T20:54:53.146 INFO:teuthology.orchestra.run.smithi099.stdout: "content_type": "application/octet-stream", 2023-05-31T20:54:53.146 INFO:teuthology.orchestra.run.smithi099.stdout: "accounted_size": 3, 2023-05-31T20:54:53.146 INFO:teuthology.orchestra.run.smithi099.stdout: "user_data": "", 2023-05-31T20:54:53.146 INFO:teuthology.orchestra.run.smithi099.stdout: "appendable": false 2023-05-31T20:54:53.146 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:54:53.146 INFO:teuthology.orchestra.run.smithi099.stdout: "tag": "3d163b1e-09c7-47e2-ae87-add4aace3f38.5680.9116791658085997217", 2023-05-31T20:54:53.146 INFO:teuthology.orchestra.run.smithi099.stdout: "flags": 0, 2023-05-31T20:54:53.147 INFO:teuthology.orchestra.run.smithi099.stdout: "pending_map": [], 2023-05-31T20:54:53.147 INFO:teuthology.orchestra.run.smithi099.stdout: "versioned_epoch": 0 2023-05-31T20:54:53.147 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:54:53.147 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T20:54:53.147 INFO:teuthology.orchestra.run.smithi099.stdout: "name": "obj111", 2023-05-31T20:54:53.147 INFO:teuthology.orchestra.run.smithi099.stdout: "instance": "", 2023-05-31T20:54:53.147 INFO:teuthology.orchestra.run.smithi099.stdout: "ver": { 2023-05-31T20:54:53.148 INFO:teuthology.orchestra.run.smithi099.stdout: "pool": 9, 2023-05-31T20:54:53.148 INFO:teuthology.orchestra.run.smithi099.stdout: "epoch": 3 2023-05-31T20:54:53.148 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:54:53.148 INFO:teuthology.orchestra.run.smithi099.stdout: "locator": "", 2023-05-31T20:54:53.148 INFO:teuthology.orchestra.run.smithi099.stdout: "exists": true, 2023-05-31T20:54:53.148 INFO:teuthology.orchestra.run.smithi099.stdout: "meta": { 2023-05-31T20:54:53.149 INFO:teuthology.orchestra.run.smithi099.stdout: "category": 1, 2023-05-31T20:54:53.149 INFO:teuthology.orchestra.run.smithi099.stdout: "size": 3, 2023-05-31T20:54:53.149 INFO:teuthology.orchestra.run.smithi099.stdout: "mtime": "2023-05-31T20:52:14.993359Z", 2023-05-31T20:54:53.149 INFO:teuthology.orchestra.run.smithi099.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-05-31T20:54:53.149 INFO:teuthology.orchestra.run.smithi099.stdout: "storage_class": "", 2023-05-31T20:54:53.149 INFO:teuthology.orchestra.run.smithi099.stdout: "owner": "rgw-multisite-test-user", 2023-05-31T20:54:53.149 INFO:teuthology.orchestra.run.smithi099.stdout: "owner_display_name": "Multisite Test User", 2023-05-31T20:54:53.150 INFO:teuthology.orchestra.run.smithi099.stdout: "content_type": "application/octet-stream", 2023-05-31T20:54:53.150 INFO:teuthology.orchestra.run.smithi099.stdout: "accounted_size": 3, 2023-05-31T20:54:53.150 INFO:teuthology.orchestra.run.smithi099.stdout: "user_data": "", 2023-05-31T20:54:53.150 INFO:teuthology.orchestra.run.smithi099.stdout: "appendable": false 2023-05-31T20:54:53.150 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:54:53.150 INFO:teuthology.orchestra.run.smithi099.stdout: "tag": "3d163b1e-09c7-47e2-ae87-add4aace3f38.5680.9140702486481279119", 2023-05-31T20:54:53.150 INFO:teuthology.orchestra.run.smithi099.stdout: "flags": 0, 2023-05-31T20:54:53.151 INFO:teuthology.orchestra.run.smithi099.stdout: "pending_map": [], 2023-05-31T20:54:53.151 INFO:teuthology.orchestra.run.smithi099.stdout: "versioned_epoch": 0 2023-05-31T20:54:53.151 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:54:53.151 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T20:54:53.151 INFO:teuthology.orchestra.run.smithi099.stdout: "name": "obj115", 2023-05-31T20:54:53.151 INFO:teuthology.orchestra.run.smithi099.stdout: "instance": "", 2023-05-31T20:54:53.151 INFO:teuthology.orchestra.run.smithi099.stdout: "ver": { 2023-05-31T20:54:53.152 INFO:teuthology.orchestra.run.smithi099.stdout: "pool": 9, 2023-05-31T20:54:53.152 INFO:teuthology.orchestra.run.smithi099.stdout: "epoch": 14 2023-05-31T20:54:53.152 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:54:53.152 INFO:teuthology.orchestra.run.smithi099.stdout: "locator": "", 2023-05-31T20:54:53.152 INFO:teuthology.orchestra.run.smithi099.stdout: "exists": true, 2023-05-31T20:54:53.152 INFO:teuthology.orchestra.run.smithi099.stdout: "meta": { 2023-05-31T20:54:53.153 INFO:teuthology.orchestra.run.smithi099.stdout: "category": 1, 2023-05-31T20:54:53.153 INFO:teuthology.orchestra.run.smithi099.stdout: "size": 6, 2023-05-31T20:54:53.153 INFO:teuthology.orchestra.run.smithi099.stdout: "mtime": "2023-05-31T20:46:33.958937Z", 2023-05-31T20:54:53.153 INFO:teuthology.orchestra.run.smithi099.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-05-31T20:54:53.153 INFO:teuthology.orchestra.run.smithi099.stdout: "storage_class": "", 2023-05-31T20:54:53.153 INFO:teuthology.orchestra.run.smithi099.stdout: "owner": "rgw-multisite-test-user", 2023-05-31T20:54:53.154 INFO:teuthology.orchestra.run.smithi099.stdout: "owner_display_name": "Multisite Test User", 2023-05-31T20:54:53.154 INFO:teuthology.orchestra.run.smithi099.stdout: "content_type": "application/octet-stream", 2023-05-31T20:54:53.154 INFO:teuthology.orchestra.run.smithi099.stdout: "accounted_size": 3, 2023-05-31T20:54:53.154 INFO:teuthology.orchestra.run.smithi099.stdout: "user_data": "", 2023-05-31T20:54:53.154 INFO:teuthology.orchestra.run.smithi099.stdout: "appendable": false 2023-05-31T20:54:53.154 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:54:53.155 INFO:teuthology.orchestra.run.smithi099.stdout: "tag": "3d163b1e-09c7-47e2-ae87-add4aace3f38.5680.5956930270309595350", 2023-05-31T20:54:53.155 INFO:teuthology.orchestra.run.smithi099.stdout: "flags": 0, 2023-05-31T20:54:53.155 INFO:teuthology.orchestra.run.smithi099.stdout: "pending_map": [], 2023-05-31T20:54:53.155 INFO:teuthology.orchestra.run.smithi099.stdout: "versioned_epoch": 0 2023-05-31T20:54:53.155 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:54:53.155 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T20:54:53.155 INFO:teuthology.orchestra.run.smithi099.stdout: "name": "obj116", 2023-05-31T20:54:53.156 INFO:teuthology.orchestra.run.smithi099.stdout: "instance": "", 2023-05-31T20:54:53.156 INFO:teuthology.orchestra.run.smithi099.stdout: "ver": { 2023-05-31T20:54:53.156 INFO:teuthology.orchestra.run.smithi099.stdout: "pool": 9, 2023-05-31T20:54:53.156 INFO:teuthology.orchestra.run.smithi099.stdout: "epoch": 11 2023-05-31T20:54:53.156 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:54:53.156 INFO:teuthology.orchestra.run.smithi099.stdout: "locator": "", 2023-05-31T20:54:53.156 INFO:teuthology.orchestra.run.smithi099.stdout: "exists": true, 2023-05-31T20:54:53.157 INFO:teuthology.orchestra.run.smithi099.stdout: "meta": { 2023-05-31T20:54:53.157 INFO:teuthology.orchestra.run.smithi099.stdout: "category": 1, 2023-05-31T20:54:53.157 INFO:teuthology.orchestra.run.smithi099.stdout: "size": 16, 2023-05-31T20:54:53.157 INFO:teuthology.orchestra.run.smithi099.stdout: "mtime": "2023-05-31T20:48:13.052647Z", 2023-05-31T20:54:53.157 INFO:teuthology.orchestra.run.smithi099.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-05-31T20:54:53.157 INFO:teuthology.orchestra.run.smithi099.stdout: "storage_class": "", 2023-05-31T20:54:53.157 INFO:teuthology.orchestra.run.smithi099.stdout: "owner": "rgw-multisite-test-user", 2023-05-31T20:54:53.158 INFO:teuthology.orchestra.run.smithi099.stdout: "owner_display_name": "Multisite Test User", 2023-05-31T20:54:53.158 INFO:teuthology.orchestra.run.smithi099.stdout: "content_type": "application/octet-stream", 2023-05-31T20:54:53.158 INFO:teuthology.orchestra.run.smithi099.stdout: "accounted_size": 3, 2023-05-31T20:54:53.158 INFO:teuthology.orchestra.run.smithi099.stdout: "user_data": "", 2023-05-31T20:54:53.158 INFO:teuthology.orchestra.run.smithi099.stdout: "appendable": false 2023-05-31T20:54:53.158 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:54:53.158 INFO:teuthology.orchestra.run.smithi099.stdout: "tag": "3d163b1e-09c7-47e2-ae87-add4aace3f38.5680.10874767853196308622", 2023-05-31T20:54:53.159 INFO:teuthology.orchestra.run.smithi099.stdout: "flags": 0, 2023-05-31T20:54:53.159 INFO:teuthology.orchestra.run.smithi099.stdout: "pending_map": [], 2023-05-31T20:54:53.159 INFO:teuthology.orchestra.run.smithi099.stdout: "versioned_epoch": 0 2023-05-31T20:54:53.159 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:54:53.159 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T20:54:53.159 INFO:teuthology.orchestra.run.smithi099.stdout: "name": "obj124", 2023-05-31T20:54:53.160 INFO:teuthology.orchestra.run.smithi099.stdout: "instance": "", 2023-05-31T20:54:53.160 INFO:teuthology.orchestra.run.smithi099.stdout: "ver": { 2023-05-31T20:54:53.160 INFO:teuthology.orchestra.run.smithi099.stdout: "pool": 9, 2023-05-31T20:54:53.160 INFO:teuthology.orchestra.run.smithi099.stdout: "epoch": 23 2023-05-31T20:54:53.160 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:54:53.160 INFO:teuthology.orchestra.run.smithi099.stdout: "locator": "", 2023-05-31T20:54:53.161 INFO:teuthology.orchestra.run.smithi099.stdout: "exists": true, 2023-05-31T20:54:53.161 INFO:teuthology.orchestra.run.smithi099.stdout: "meta": { 2023-05-31T20:54:53.161 INFO:teuthology.orchestra.run.smithi099.stdout: "category": 1, 2023-05-31T20:54:53.161 INFO:teuthology.orchestra.run.smithi099.stdout: "size": 5, 2023-05-31T20:54:53.161 INFO:teuthology.orchestra.run.smithi099.stdout: "mtime": "2023-05-31T20:50:33.587761Z", 2023-05-31T20:54:53.161 INFO:teuthology.orchestra.run.smithi099.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-05-31T20:54:53.161 INFO:teuthology.orchestra.run.smithi099.stdout: "storage_class": "", 2023-05-31T20:54:53.162 INFO:teuthology.orchestra.run.smithi099.stdout: "owner": "rgw-multisite-test-user", 2023-05-31T20:54:53.162 INFO:teuthology.orchestra.run.smithi099.stdout: "owner_display_name": "Multisite Test User", 2023-05-31T20:54:53.162 INFO:teuthology.orchestra.run.smithi099.stdout: "content_type": "application/octet-stream", 2023-05-31T20:54:53.162 INFO:teuthology.orchestra.run.smithi099.stdout: "accounted_size": 3, 2023-05-31T20:54:53.162 INFO:teuthology.orchestra.run.smithi099.stdout: "user_data": "", 2023-05-31T20:54:53.162 INFO:teuthology.orchestra.run.smithi099.stdout: "appendable": false 2023-05-31T20:54:53.162 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:54:53.163 INFO:teuthology.orchestra.run.smithi099.stdout: "tag": "3d163b1e-09c7-47e2-ae87-add4aace3f38.5680.9290701452469207553", 2023-05-31T20:54:53.163 INFO:teuthology.orchestra.run.smithi099.stdout: "flags": 0, 2023-05-31T20:54:53.163 INFO:teuthology.orchestra.run.smithi099.stdout: "pending_map": [], 2023-05-31T20:54:53.163 INFO:teuthology.orchestra.run.smithi099.stdout: "versioned_epoch": 0 2023-05-31T20:54:53.163 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:54:53.163 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T20:54:53.163 INFO:teuthology.orchestra.run.smithi099.stdout: "name": "obj145", 2023-05-31T20:54:53.164 INFO:teuthology.orchestra.run.smithi099.stdout: "instance": "", 2023-05-31T20:54:53.164 INFO:teuthology.orchestra.run.smithi099.stdout: "ver": { 2023-05-31T20:54:53.164 INFO:teuthology.orchestra.run.smithi099.stdout: "pool": 9, 2023-05-31T20:54:53.164 INFO:teuthology.orchestra.run.smithi099.stdout: "epoch": 12 2023-05-31T20:54:53.164 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:54:53.164 INFO:teuthology.orchestra.run.smithi099.stdout: "locator": "", 2023-05-31T20:54:53.164 INFO:teuthology.orchestra.run.smithi099.stdout: "exists": true, 2023-05-31T20:54:53.165 INFO:teuthology.orchestra.run.smithi099.stdout: "meta": { 2023-05-31T20:54:53.165 INFO:teuthology.orchestra.run.smithi099.stdout: "category": 1, 2023-05-31T20:54:53.165 INFO:teuthology.orchestra.run.smithi099.stdout: "size": 5, 2023-05-31T20:54:53.165 INFO:teuthology.orchestra.run.smithi099.stdout: "mtime": "2023-05-31T20:48:13.107892Z", 2023-05-31T20:54:53.165 INFO:teuthology.orchestra.run.smithi099.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-05-31T20:54:53.165 INFO:teuthology.orchestra.run.smithi099.stdout: "storage_class": "", 2023-05-31T20:54:53.166 INFO:teuthology.orchestra.run.smithi099.stdout: "owner": "rgw-multisite-test-user", 2023-05-31T20:54:53.166 INFO:teuthology.orchestra.run.smithi099.stdout: "owner_display_name": "Multisite Test User", 2023-05-31T20:54:53.166 INFO:teuthology.orchestra.run.smithi099.stdout: "content_type": "application/octet-stream", 2023-05-31T20:54:53.166 INFO:teuthology.orchestra.run.smithi099.stdout: "accounted_size": 3, 2023-05-31T20:54:53.166 INFO:teuthology.orchestra.run.smithi099.stdout: "user_data": "", 2023-05-31T20:54:53.166 INFO:teuthology.orchestra.run.smithi099.stdout: "appendable": false 2023-05-31T20:54:53.166 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:54:53.167 INFO:teuthology.orchestra.run.smithi099.stdout: "tag": "3d163b1e-09c7-47e2-ae87-add4aace3f38.5680.15585921013774236177", 2023-05-31T20:54:53.167 INFO:teuthology.orchestra.run.smithi099.stdout: "flags": 0, 2023-05-31T20:54:53.167 INFO:teuthology.orchestra.run.smithi099.stdout: "pending_map": [], 2023-05-31T20:54:53.167 INFO:teuthology.orchestra.run.smithi099.stdout: "versioned_epoch": 0 2023-05-31T20:54:53.167 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:54:53.167 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T20:54:53.167 INFO:teuthology.orchestra.run.smithi099.stdout: "name": "obj148", 2023-05-31T20:54:53.168 INFO:teuthology.orchestra.run.smithi099.stdout: "instance": "", 2023-05-31T20:54:53.168 INFO:teuthology.orchestra.run.smithi099.stdout: "ver": { 2023-05-31T20:54:53.168 INFO:teuthology.orchestra.run.smithi099.stdout: "pool": 9, 2023-05-31T20:54:53.168 INFO:teuthology.orchestra.run.smithi099.stdout: "epoch": 4 2023-05-31T20:54:53.168 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:54:53.168 INFO:teuthology.orchestra.run.smithi099.stdout: "locator": "", 2023-05-31T20:54:53.168 INFO:teuthology.orchestra.run.smithi099.stdout: "exists": true, 2023-05-31T20:54:53.169 INFO:teuthology.orchestra.run.smithi099.stdout: "meta": { 2023-05-31T20:54:53.169 INFO:teuthology.orchestra.run.smithi099.stdout: "category": 1, 2023-05-31T20:54:53.169 INFO:teuthology.orchestra.run.smithi099.stdout: "size": 6, 2023-05-31T20:54:53.169 INFO:teuthology.orchestra.run.smithi099.stdout: "mtime": "2023-05-31T20:52:15.015083Z", 2023-05-31T20:54:53.169 INFO:teuthology.orchestra.run.smithi099.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-05-31T20:54:53.169 INFO:teuthology.orchestra.run.smithi099.stdout: "storage_class": "", 2023-05-31T20:54:53.169 INFO:teuthology.orchestra.run.smithi099.stdout: "owner": "rgw-multisite-test-user", 2023-05-31T20:54:53.170 INFO:teuthology.orchestra.run.smithi099.stdout: "owner_display_name": "Multisite Test User", 2023-05-31T20:54:53.170 INFO:teuthology.orchestra.run.smithi099.stdout: "content_type": "application/octet-stream", 2023-05-31T20:54:53.170 INFO:teuthology.orchestra.run.smithi099.stdout: "accounted_size": 3, 2023-05-31T20:54:53.170 INFO:teuthology.orchestra.run.smithi099.stdout: "user_data": "", 2023-05-31T20:54:53.170 INFO:teuthology.orchestra.run.smithi099.stdout: "appendable": false 2023-05-31T20:54:53.170 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:54:53.170 INFO:teuthology.orchestra.run.smithi099.stdout: "tag": "3d163b1e-09c7-47e2-ae87-add4aace3f38.5680.11245321175967093833", 2023-05-31T20:54:53.171 INFO:teuthology.orchestra.run.smithi099.stdout: "flags": 0, 2023-05-31T20:54:53.171 INFO:teuthology.orchestra.run.smithi099.stdout: "pending_map": [], 2023-05-31T20:54:53.171 INFO:teuthology.orchestra.run.smithi099.stdout: "versioned_epoch": 0 2023-05-31T20:54:53.171 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:54:53.171 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T20:54:53.171 INFO:teuthology.orchestra.run.smithi099.stdout: "name": "obj155", 2023-05-31T20:54:53.171 INFO:teuthology.orchestra.run.smithi099.stdout: "instance": "", 2023-05-31T20:54:53.172 INFO:teuthology.orchestra.run.smithi099.stdout: "ver": { 2023-05-31T20:54:53.172 INFO:teuthology.orchestra.run.smithi099.stdout: "pool": 9, 2023-05-31T20:54:53.172 INFO:teuthology.orchestra.run.smithi099.stdout: "epoch": 5 2023-05-31T20:54:53.172 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:54:53.172 INFO:teuthology.orchestra.run.smithi099.stdout: "locator": "", 2023-05-31T20:54:53.172 INFO:teuthology.orchestra.run.smithi099.stdout: "exists": true, 2023-05-31T20:54:53.173 INFO:teuthology.orchestra.run.smithi099.stdout: "meta": { 2023-05-31T20:54:53.173 INFO:teuthology.orchestra.run.smithi099.stdout: "category": 1, 2023-05-31T20:54:53.173 INFO:teuthology.orchestra.run.smithi099.stdout: "size": 6, 2023-05-31T20:54:53.173 INFO:teuthology.orchestra.run.smithi099.stdout: "mtime": "2023-05-31T20:50:33.605763Z", 2023-05-31T20:54:53.173 INFO:teuthology.orchestra.run.smithi099.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-05-31T20:54:53.173 INFO:teuthology.orchestra.run.smithi099.stdout: "storage_class": "", 2023-05-31T20:54:53.173 INFO:teuthology.orchestra.run.smithi099.stdout: "owner": "rgw-multisite-test-user", 2023-05-31T20:54:53.174 INFO:teuthology.orchestra.run.smithi099.stdout: "owner_display_name": "Multisite Test User", 2023-05-31T20:54:53.174 INFO:teuthology.orchestra.run.smithi099.stdout: "content_type": "application/octet-stream", 2023-05-31T20:54:53.174 INFO:teuthology.orchestra.run.smithi099.stdout: "accounted_size": 3, 2023-05-31T20:54:53.174 INFO:teuthology.orchestra.run.smithi099.stdout: "user_data": "", 2023-05-31T20:54:53.174 INFO:teuthology.orchestra.run.smithi099.stdout: "appendable": false 2023-05-31T20:54:53.174 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:54:53.174 INFO:teuthology.orchestra.run.smithi099.stdout: "tag": "3d163b1e-09c7-47e2-ae87-add4aace3f38.5680.17814659581524855812", 2023-05-31T20:54:53.175 INFO:teuthology.orchestra.run.smithi099.stdout: "flags": 0, 2023-05-31T20:54:53.175 INFO:teuthology.orchestra.run.smithi099.stdout: "pending_map": [], 2023-05-31T20:54:53.175 INFO:teuthology.orchestra.run.smithi099.stdout: "versioned_epoch": 0 2023-05-31T20:54:53.175 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:54:53.175 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T20:54:53.175 INFO:teuthology.orchestra.run.smithi099.stdout: "name": "obj17", 2023-05-31T20:54:53.175 INFO:teuthology.orchestra.run.smithi099.stdout: "instance": "", 2023-05-31T20:54:53.176 INFO:teuthology.orchestra.run.smithi099.stdout: "ver": { 2023-05-31T20:54:53.176 INFO:teuthology.orchestra.run.smithi099.stdout: "pool": 9, 2023-05-31T20:54:53.176 INFO:teuthology.orchestra.run.smithi099.stdout: "epoch": 7 2023-05-31T20:54:53.176 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:54:53.176 INFO:teuthology.orchestra.run.smithi099.stdout: "locator": "", 2023-05-31T20:54:53.176 INFO:teuthology.orchestra.run.smithi099.stdout: "exists": true, 2023-05-31T20:54:53.176 INFO:teuthology.orchestra.run.smithi099.stdout: "meta": { 2023-05-31T20:54:53.177 INFO:teuthology.orchestra.run.smithi099.stdout: "category": 1, 2023-05-31T20:54:53.177 INFO:teuthology.orchestra.run.smithi099.stdout: "size": 16, 2023-05-31T20:54:53.177 INFO:teuthology.orchestra.run.smithi099.stdout: "mtime": "2023-05-31T20:43:07.977285Z", 2023-05-31T20:54:53.177 INFO:teuthology.orchestra.run.smithi099.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-05-31T20:54:53.177 INFO:teuthology.orchestra.run.smithi099.stdout: "storage_class": "", 2023-05-31T20:54:53.177 INFO:teuthology.orchestra.run.smithi099.stdout: "owner": "rgw-multisite-test-user", 2023-05-31T20:54:53.177 INFO:teuthology.orchestra.run.smithi099.stdout: "owner_display_name": "Multisite Test User", 2023-05-31T20:54:53.178 INFO:teuthology.orchestra.run.smithi099.stdout: "content_type": "application/octet-stream", 2023-05-31T20:54:53.178 INFO:teuthology.orchestra.run.smithi099.stdout: "accounted_size": 3, 2023-05-31T20:54:53.178 INFO:teuthology.orchestra.run.smithi099.stdout: "user_data": "", 2023-05-31T20:54:53.178 INFO:teuthology.orchestra.run.smithi099.stdout: "appendable": false 2023-05-31T20:54:53.178 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:54:53.178 INFO:teuthology.orchestra.run.smithi099.stdout: "tag": "3d163b1e-09c7-47e2-ae87-add4aace3f38.5680.8165500486622100550", 2023-05-31T20:54:53.178 INFO:teuthology.orchestra.run.smithi099.stdout: "flags": 0, 2023-05-31T20:54:53.179 INFO:teuthology.orchestra.run.smithi099.stdout: "pending_map": [], 2023-05-31T20:54:53.179 INFO:teuthology.orchestra.run.smithi099.stdout: "versioned_epoch": 0 2023-05-31T20:54:53.179 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:54:53.179 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T20:54:53.179 INFO:teuthology.orchestra.run.smithi099.stdout: "name": "obj185", 2023-05-31T20:54:53.179 INFO:teuthology.orchestra.run.smithi099.stdout: "instance": "", 2023-05-31T20:54:53.180 INFO:teuthology.orchestra.run.smithi099.stdout: "ver": { 2023-05-31T20:54:53.180 INFO:teuthology.orchestra.run.smithi099.stdout: "pool": 9, 2023-05-31T20:54:53.180 INFO:teuthology.orchestra.run.smithi099.stdout: "epoch": 14 2023-05-31T20:54:53.180 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:54:53.180 INFO:teuthology.orchestra.run.smithi099.stdout: "locator": "", 2023-05-31T20:54:53.180 INFO:teuthology.orchestra.run.smithi099.stdout: "exists": true, 2023-05-31T20:54:53.181 INFO:teuthology.orchestra.run.smithi099.stdout: "meta": { 2023-05-31T20:54:53.181 INFO:teuthology.orchestra.run.smithi099.stdout: "category": 1, 2023-05-31T20:54:53.181 INFO:teuthology.orchestra.run.smithi099.stdout: "size": 3, 2023-05-31T20:54:53.181 INFO:teuthology.orchestra.run.smithi099.stdout: "mtime": "2023-05-31T20:52:15.072437Z", 2023-05-31T20:54:53.181 INFO:teuthology.orchestra.run.smithi099.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-05-31T20:54:53.181 INFO:teuthology.orchestra.run.smithi099.stdout: "storage_class": "", 2023-05-31T20:54:53.181 INFO:teuthology.orchestra.run.smithi099.stdout: "owner": "rgw-multisite-test-user", 2023-05-31T20:54:53.182 INFO:teuthology.orchestra.run.smithi099.stdout: "owner_display_name": "Multisite Test User", 2023-05-31T20:54:53.182 INFO:teuthology.orchestra.run.smithi099.stdout: "content_type": "application/octet-stream", 2023-05-31T20:54:53.182 INFO:teuthology.orchestra.run.smithi099.stdout: "accounted_size": 3, 2023-05-31T20:54:53.182 INFO:teuthology.orchestra.run.smithi099.stdout: "user_data": "", 2023-05-31T20:54:53.182 INFO:teuthology.orchestra.run.smithi099.stdout: "appendable": false 2023-05-31T20:54:53.182 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:54:53.183 INFO:teuthology.orchestra.run.smithi099.stdout: "tag": "3d163b1e-09c7-47e2-ae87-add4aace3f38.5680.10548165953309453519", 2023-05-31T20:54:53.183 INFO:teuthology.orchestra.run.smithi099.stdout: "flags": 0, 2023-05-31T20:54:53.183 INFO:teuthology.orchestra.run.smithi099.stdout: "pending_map": [], 2023-05-31T20:54:53.183 INFO:teuthology.orchestra.run.smithi099.stdout: "versioned_epoch": 0 2023-05-31T20:54:53.183 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:54:53.183 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T20:54:53.183 INFO:teuthology.orchestra.run.smithi099.stdout: "name": "obj19", 2023-05-31T20:54:53.184 INFO:teuthology.orchestra.run.smithi099.stdout: "instance": "", 2023-05-31T20:54:53.184 INFO:teuthology.orchestra.run.smithi099.stdout: "ver": { 2023-05-31T20:54:53.184 INFO:teuthology.orchestra.run.smithi099.stdout: "pool": 9, 2023-05-31T20:54:53.184 INFO:teuthology.orchestra.run.smithi099.stdout: "epoch": 6 2023-05-31T20:54:53.184 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:54:53.184 INFO:teuthology.orchestra.run.smithi099.stdout: "locator": "", 2023-05-31T20:54:53.185 INFO:teuthology.orchestra.run.smithi099.stdout: "exists": true, 2023-05-31T20:54:53.185 INFO:teuthology.orchestra.run.smithi099.stdout: "meta": { 2023-05-31T20:54:53.185 INFO:teuthology.orchestra.run.smithi099.stdout: "category": 1, 2023-05-31T20:54:53.185 INFO:teuthology.orchestra.run.smithi099.stdout: "size": 6, 2023-05-31T20:54:53.185 INFO:teuthology.orchestra.run.smithi099.stdout: "mtime": "2023-05-31T20:44:50.557118Z", 2023-05-31T20:54:53.185 INFO:teuthology.orchestra.run.smithi099.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-05-31T20:54:53.185 INFO:teuthology.orchestra.run.smithi099.stdout: "storage_class": "", 2023-05-31T20:54:53.186 INFO:teuthology.orchestra.run.smithi099.stdout: "owner": "rgw-multisite-test-user", 2023-05-31T20:54:53.186 INFO:teuthology.orchestra.run.smithi099.stdout: "owner_display_name": "Multisite Test User", 2023-05-31T20:54:53.186 INFO:teuthology.orchestra.run.smithi099.stdout: "content_type": "application/octet-stream", 2023-05-31T20:54:53.186 INFO:teuthology.orchestra.run.smithi099.stdout: "accounted_size": 3, 2023-05-31T20:54:53.186 INFO:teuthology.orchestra.run.smithi099.stdout: "user_data": "", 2023-05-31T20:54:53.186 INFO:teuthology.orchestra.run.smithi099.stdout: "appendable": false 2023-05-31T20:54:53.186 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:54:53.187 INFO:teuthology.orchestra.run.smithi099.stdout: "tag": "3d163b1e-09c7-47e2-ae87-add4aace3f38.5680.10409482175799089750", 2023-05-31T20:54:53.187 INFO:teuthology.orchestra.run.smithi099.stdout: "flags": 0, 2023-05-31T20:54:53.187 INFO:teuthology.orchestra.run.smithi099.stdout: "pending_map": [], 2023-05-31T20:54:53.187 INFO:teuthology.orchestra.run.smithi099.stdout: "versioned_epoch": 0 2023-05-31T20:54:53.187 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:54:53.187 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T20:54:53.187 INFO:teuthology.orchestra.run.smithi099.stdout: "name": "obj22", 2023-05-31T20:54:53.188 INFO:teuthology.orchestra.run.smithi099.stdout: "instance": "", 2023-05-31T20:54:53.188 INFO:teuthology.orchestra.run.smithi099.stdout: "ver": { 2023-05-31T20:54:53.188 INFO:teuthology.orchestra.run.smithi099.stdout: "pool": 9, 2023-05-31T20:54:53.188 INFO:teuthology.orchestra.run.smithi099.stdout: "epoch": 5 2023-05-31T20:54:53.188 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:54:53.188 INFO:teuthology.orchestra.run.smithi099.stdout: "locator": "", 2023-05-31T20:54:53.189 INFO:teuthology.orchestra.run.smithi099.stdout: "exists": true, 2023-05-31T20:54:53.189 INFO:teuthology.orchestra.run.smithi099.stdout: "meta": { 2023-05-31T20:54:53.189 INFO:teuthology.orchestra.run.smithi099.stdout: "category": 1, 2023-05-31T20:54:53.189 INFO:teuthology.orchestra.run.smithi099.stdout: "size": 6, 2023-05-31T20:54:53.189 INFO:teuthology.orchestra.run.smithi099.stdout: "mtime": "2023-05-31T20:39:26.473296Z", 2023-05-31T20:54:53.189 INFO:teuthology.orchestra.run.smithi099.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-05-31T20:54:53.189 INFO:teuthology.orchestra.run.smithi099.stdout: "storage_class": "", 2023-05-31T20:54:53.190 INFO:teuthology.orchestra.run.smithi099.stdout: "owner": "rgw-multisite-test-user", 2023-05-31T20:54:53.190 INFO:teuthology.orchestra.run.smithi099.stdout: "owner_display_name": "Multisite Test User", 2023-05-31T20:54:53.190 INFO:teuthology.orchestra.run.smithi099.stdout: "content_type": "application/octet-stream", 2023-05-31T20:54:53.190 INFO:teuthology.orchestra.run.smithi099.stdout: "accounted_size": 3, 2023-05-31T20:54:53.190 INFO:teuthology.orchestra.run.smithi099.stdout: "user_data": "", 2023-05-31T20:54:53.190 INFO:teuthology.orchestra.run.smithi099.stdout: "appendable": false 2023-05-31T20:54:53.190 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:54:53.191 INFO:teuthology.orchestra.run.smithi099.stdout: "tag": "3d163b1e-09c7-47e2-ae87-add4aace3f38.5680.15063889672733459713", 2023-05-31T20:54:53.191 INFO:teuthology.orchestra.run.smithi099.stdout: "flags": 0, 2023-05-31T20:54:53.191 INFO:teuthology.orchestra.run.smithi099.stdout: "pending_map": [], 2023-05-31T20:54:53.191 INFO:teuthology.orchestra.run.smithi099.stdout: "versioned_epoch": 0 2023-05-31T20:54:53.191 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:54:53.191 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T20:54:53.191 INFO:teuthology.orchestra.run.smithi099.stdout: "name": "obj23", 2023-05-31T20:54:53.192 INFO:teuthology.orchestra.run.smithi099.stdout: "instance": "", 2023-05-31T20:54:53.192 INFO:teuthology.orchestra.run.smithi099.stdout: "ver": { 2023-05-31T20:54:53.192 INFO:teuthology.orchestra.run.smithi099.stdout: "pool": 9, 2023-05-31T20:54:53.192 INFO:teuthology.orchestra.run.smithi099.stdout: "epoch": 7 2023-05-31T20:54:53.192 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:54:53.192 INFO:teuthology.orchestra.run.smithi099.stdout: "locator": "", 2023-05-31T20:54:53.193 INFO:teuthology.orchestra.run.smithi099.stdout: "exists": true, 2023-05-31T20:54:53.193 INFO:teuthology.orchestra.run.smithi099.stdout: "meta": { 2023-05-31T20:54:53.193 INFO:teuthology.orchestra.run.smithi099.stdout: "category": 1, 2023-05-31T20:54:53.193 INFO:teuthology.orchestra.run.smithi099.stdout: "size": 5, 2023-05-31T20:54:53.193 INFO:teuthology.orchestra.run.smithi099.stdout: "mtime": "2023-05-31T20:46:33.813577Z", 2023-05-31T20:54:53.193 INFO:teuthology.orchestra.run.smithi099.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-05-31T20:54:53.193 INFO:teuthology.orchestra.run.smithi099.stdout: "storage_class": "", 2023-05-31T20:54:53.194 INFO:teuthology.orchestra.run.smithi099.stdout: "owner": "rgw-multisite-test-user", 2023-05-31T20:54:53.194 INFO:teuthology.orchestra.run.smithi099.stdout: "owner_display_name": "Multisite Test User", 2023-05-31T20:54:53.194 INFO:teuthology.orchestra.run.smithi099.stdout: "content_type": "application/octet-stream", 2023-05-31T20:54:53.194 INFO:teuthology.orchestra.run.smithi099.stdout: "accounted_size": 3, 2023-05-31T20:54:53.194 INFO:teuthology.orchestra.run.smithi099.stdout: "user_data": "", 2023-05-31T20:54:53.194 INFO:teuthology.orchestra.run.smithi099.stdout: "appendable": false 2023-05-31T20:54:53.194 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:54:53.195 INFO:teuthology.orchestra.run.smithi099.stdout: "tag": "3d163b1e-09c7-47e2-ae87-add4aace3f38.5680.819440386057841109", 2023-05-31T20:54:53.195 INFO:teuthology.orchestra.run.smithi099.stdout: "flags": 0, 2023-05-31T20:54:53.195 INFO:teuthology.orchestra.run.smithi099.stdout: "pending_map": [], 2023-05-31T20:54:53.195 INFO:teuthology.orchestra.run.smithi099.stdout: "versioned_epoch": 0 2023-05-31T20:54:53.195 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:54:53.195 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T20:54:53.195 INFO:teuthology.orchestra.run.smithi099.stdout: "name": "obj29", 2023-05-31T20:54:53.196 INFO:teuthology.orchestra.run.smithi099.stdout: "instance": "", 2023-05-31T20:54:53.196 INFO:teuthology.orchestra.run.smithi099.stdout: "ver": { 2023-05-31T20:54:53.196 INFO:teuthology.orchestra.run.smithi099.stdout: "pool": 9, 2023-05-31T20:54:53.196 INFO:teuthology.orchestra.run.smithi099.stdout: "epoch": 2 2023-05-31T20:54:53.196 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:54:53.196 INFO:teuthology.orchestra.run.smithi099.stdout: "locator": "", 2023-05-31T20:54:53.196 INFO:teuthology.orchestra.run.smithi099.stdout: "exists": true, 2023-05-31T20:54:53.197 INFO:teuthology.orchestra.run.smithi099.stdout: "meta": { 2023-05-31T20:54:53.197 INFO:teuthology.orchestra.run.smithi099.stdout: "category": 1, 2023-05-31T20:54:53.197 INFO:teuthology.orchestra.run.smithi099.stdout: "size": 16, 2023-05-31T20:54:53.197 INFO:teuthology.orchestra.run.smithi099.stdout: "mtime": "2023-05-31T20:48:12.956068Z", 2023-05-31T20:54:53.197 INFO:teuthology.orchestra.run.smithi099.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-05-31T20:54:53.197 INFO:teuthology.orchestra.run.smithi099.stdout: "storage_class": "", 2023-05-31T20:54:53.197 INFO:teuthology.orchestra.run.smithi099.stdout: "owner": "rgw-multisite-test-user", 2023-05-31T20:54:53.198 INFO:teuthology.orchestra.run.smithi099.stdout: "owner_display_name": "Multisite Test User", 2023-05-31T20:54:53.198 INFO:teuthology.orchestra.run.smithi099.stdout: "content_type": "application/octet-stream", 2023-05-31T20:54:53.198 INFO:teuthology.orchestra.run.smithi099.stdout: "accounted_size": 3, 2023-05-31T20:54:53.198 INFO:teuthology.orchestra.run.smithi099.stdout: "user_data": "", 2023-05-31T20:54:53.198 INFO:teuthology.orchestra.run.smithi099.stdout: "appendable": false 2023-05-31T20:54:53.198 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:54:53.198 INFO:teuthology.orchestra.run.smithi099.stdout: "tag": "3d163b1e-09c7-47e2-ae87-add4aace3f38.5680.7261702054336868042", 2023-05-31T20:54:53.199 INFO:teuthology.orchestra.run.smithi099.stdout: "flags": 0, 2023-05-31T20:54:53.199 INFO:teuthology.orchestra.run.smithi099.stdout: "pending_map": [], 2023-05-31T20:54:53.199 INFO:teuthology.orchestra.run.smithi099.stdout: "versioned_epoch": 0 2023-05-31T20:54:53.199 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:54:53.199 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T20:54:53.199 INFO:teuthology.orchestra.run.smithi099.stdout: "name": "obj31", 2023-05-31T20:54:53.199 INFO:teuthology.orchestra.run.smithi099.stdout: "instance": "", 2023-05-31T20:54:53.200 INFO:teuthology.orchestra.run.smithi099.stdout: "ver": { 2023-05-31T20:54:53.200 INFO:teuthology.orchestra.run.smithi099.stdout: "pool": 9, 2023-05-31T20:54:53.200 INFO:teuthology.orchestra.run.smithi099.stdout: "epoch": 6 2023-05-31T20:54:53.200 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:54:53.200 INFO:teuthology.orchestra.run.smithi099.stdout: "locator": "", 2023-05-31T20:54:53.200 INFO:teuthology.orchestra.run.smithi099.stdout: "exists": true, 2023-05-31T20:54:53.201 INFO:teuthology.orchestra.run.smithi099.stdout: "meta": { 2023-05-31T20:54:53.201 INFO:teuthology.orchestra.run.smithi099.stdout: "category": 1, 2023-05-31T20:54:53.201 INFO:teuthology.orchestra.run.smithi099.stdout: "size": 3, 2023-05-31T20:54:53.201 INFO:teuthology.orchestra.run.smithi099.stdout: "mtime": "2023-05-31T20:50:33.460419Z", 2023-05-31T20:54:53.201 INFO:teuthology.orchestra.run.smithi099.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-05-31T20:54:53.201 INFO:teuthology.orchestra.run.smithi099.stdout: "storage_class": "", 2023-05-31T20:54:53.201 INFO:teuthology.orchestra.run.smithi099.stdout: "owner": "rgw-multisite-test-user", 2023-05-31T20:54:53.201 INFO:teuthology.orchestra.run.smithi099.stdout: "owner_display_name": "Multisite Test User", 2023-05-31T20:54:53.202 INFO:teuthology.orchestra.run.smithi099.stdout: "content_type": "application/octet-stream", 2023-05-31T20:54:53.202 INFO:teuthology.orchestra.run.smithi099.stdout: "accounted_size": 3, 2023-05-31T20:54:53.202 INFO:teuthology.orchestra.run.smithi099.stdout: "user_data": "", 2023-05-31T20:54:53.202 INFO:teuthology.orchestra.run.smithi099.stdout: "appendable": false 2023-05-31T20:54:53.202 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:54:53.202 INFO:teuthology.orchestra.run.smithi099.stdout: "tag": "3d163b1e-09c7-47e2-ae87-add4aace3f38.5680.7733967597836860348", 2023-05-31T20:54:53.202 INFO:teuthology.orchestra.run.smithi099.stdout: "flags": 0, 2023-05-31T20:54:53.203 INFO:teuthology.orchestra.run.smithi099.stdout: "pending_map": [], 2023-05-31T20:54:53.203 INFO:teuthology.orchestra.run.smithi099.stdout: "versioned_epoch": 0 2023-05-31T20:54:53.203 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:54:53.203 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T20:54:53.203 INFO:teuthology.orchestra.run.smithi099.stdout: "name": "obj33", 2023-05-31T20:54:53.203 INFO:teuthology.orchestra.run.smithi099.stdout: "instance": "", 2023-05-31T20:54:53.203 INFO:teuthology.orchestra.run.smithi099.stdout: "ver": { 2023-05-31T20:54:53.204 INFO:teuthology.orchestra.run.smithi099.stdout: "pool": 9, 2023-05-31T20:54:53.204 INFO:teuthology.orchestra.run.smithi099.stdout: "epoch": 8 2023-05-31T20:54:53.204 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:54:53.204 INFO:teuthology.orchestra.run.smithi099.stdout: "locator": "", 2023-05-31T20:54:53.204 INFO:teuthology.orchestra.run.smithi099.stdout: "exists": true, 2023-05-31T20:54:53.204 INFO:teuthology.orchestra.run.smithi099.stdout: "meta": { 2023-05-31T20:54:53.205 INFO:teuthology.orchestra.run.smithi099.stdout: "category": 1, 2023-05-31T20:54:53.205 INFO:teuthology.orchestra.run.smithi099.stdout: "size": 3, 2023-05-31T20:54:53.205 INFO:teuthology.orchestra.run.smithi099.stdout: "mtime": "2023-05-31T20:39:26.534549Z", 2023-05-31T20:54:53.205 INFO:teuthology.orchestra.run.smithi099.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-05-31T20:54:53.205 INFO:teuthology.orchestra.run.smithi099.stdout: "storage_class": "", 2023-05-31T20:54:53.205 INFO:teuthology.orchestra.run.smithi099.stdout: "owner": "rgw-multisite-test-user", 2023-05-31T20:54:53.205 INFO:teuthology.orchestra.run.smithi099.stdout: "owner_display_name": "Multisite Test User", 2023-05-31T20:54:53.206 INFO:teuthology.orchestra.run.smithi099.stdout: "content_type": "application/octet-stream", 2023-05-31T20:54:53.206 INFO:teuthology.orchestra.run.smithi099.stdout: "accounted_size": 3, 2023-05-31T20:54:53.206 INFO:teuthology.orchestra.run.smithi099.stdout: "user_data": "", 2023-05-31T20:54:53.206 INFO:teuthology.orchestra.run.smithi099.stdout: "appendable": false 2023-05-31T20:54:53.206 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:54:53.206 INFO:teuthology.orchestra.run.smithi099.stdout: "tag": "3d163b1e-09c7-47e2-ae87-add4aace3f38.5680.15091333072801631050", 2023-05-31T20:54:53.206 INFO:teuthology.orchestra.run.smithi099.stdout: "flags": 0, 2023-05-31T20:54:53.207 INFO:teuthology.orchestra.run.smithi099.stdout: "pending_map": [], 2023-05-31T20:54:53.207 INFO:teuthology.orchestra.run.smithi099.stdout: "versioned_epoch": 0 2023-05-31T20:54:53.207 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:54:53.207 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T20:54:53.207 INFO:teuthology.orchestra.run.smithi099.stdout: "name": "obj34", 2023-05-31T20:54:53.207 INFO:teuthology.orchestra.run.smithi099.stdout: "instance": "", 2023-05-31T20:54:53.207 INFO:teuthology.orchestra.run.smithi099.stdout: "ver": { 2023-05-31T20:54:53.208 INFO:teuthology.orchestra.run.smithi099.stdout: "pool": 9, 2023-05-31T20:54:53.208 INFO:teuthology.orchestra.run.smithi099.stdout: "epoch": 9 2023-05-31T20:54:53.208 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:54:53.208 INFO:teuthology.orchestra.run.smithi099.stdout: "locator": "", 2023-05-31T20:54:53.208 INFO:teuthology.orchestra.run.smithi099.stdout: "exists": true, 2023-05-31T20:54:53.208 INFO:teuthology.orchestra.run.smithi099.stdout: "meta": { 2023-05-31T20:54:53.208 INFO:teuthology.orchestra.run.smithi099.stdout: "category": 1, 2023-05-31T20:54:53.209 INFO:teuthology.orchestra.run.smithi099.stdout: "size": 5, 2023-05-31T20:54:53.209 INFO:teuthology.orchestra.run.smithi099.stdout: "mtime": "2023-05-31T20:43:07.994147Z", 2023-05-31T20:54:53.209 INFO:teuthology.orchestra.run.smithi099.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-05-31T20:54:53.209 INFO:teuthology.orchestra.run.smithi099.stdout: "storage_class": "", 2023-05-31T20:54:53.209 INFO:teuthology.orchestra.run.smithi099.stdout: "owner": "rgw-multisite-test-user", 2023-05-31T20:54:53.209 INFO:teuthology.orchestra.run.smithi099.stdout: "owner_display_name": "Multisite Test User", 2023-05-31T20:54:53.209 INFO:teuthology.orchestra.run.smithi099.stdout: "content_type": "application/octet-stream", 2023-05-31T20:54:53.210 INFO:teuthology.orchestra.run.smithi099.stdout: "accounted_size": 3, 2023-05-31T20:54:53.210 INFO:teuthology.orchestra.run.smithi099.stdout: "user_data": "", 2023-05-31T20:54:53.210 INFO:teuthology.orchestra.run.smithi099.stdout: "appendable": false 2023-05-31T20:54:53.210 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:54:53.210 INFO:teuthology.orchestra.run.smithi099.stdout: "tag": "3d163b1e-09c7-47e2-ae87-add4aace3f38.5680.9152043840104363135", 2023-05-31T20:54:53.210 INFO:teuthology.orchestra.run.smithi099.stdout: "flags": 0, 2023-05-31T20:54:53.210 INFO:teuthology.orchestra.run.smithi099.stdout: "pending_map": [], 2023-05-31T20:54:53.211 INFO:teuthology.orchestra.run.smithi099.stdout: "versioned_epoch": 0 2023-05-31T20:54:53.211 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:54:53.211 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T20:54:53.211 INFO:teuthology.orchestra.run.smithi099.stdout: "name": "obj37", 2023-05-31T20:54:53.211 INFO:teuthology.orchestra.run.smithi099.stdout: "instance": "", 2023-05-31T20:54:53.211 INFO:teuthology.orchestra.run.smithi099.stdout: "ver": { 2023-05-31T20:54:53.211 INFO:teuthology.orchestra.run.smithi099.stdout: "pool": 9, 2023-05-31T20:54:53.212 INFO:teuthology.orchestra.run.smithi099.stdout: "epoch": 4 2023-05-31T20:54:53.212 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:54:53.212 INFO:teuthology.orchestra.run.smithi099.stdout: "locator": "", 2023-05-31T20:54:53.212 INFO:teuthology.orchestra.run.smithi099.stdout: "exists": true, 2023-05-31T20:54:53.212 INFO:teuthology.orchestra.run.smithi099.stdout: "meta": { 2023-05-31T20:54:53.212 INFO:teuthology.orchestra.run.smithi099.stdout: "category": 1, 2023-05-31T20:54:53.213 INFO:teuthology.orchestra.run.smithi099.stdout: "size": 16, 2023-05-31T20:54:53.213 INFO:teuthology.orchestra.run.smithi099.stdout: "mtime": "2023-05-31T20:52:14.909599Z", 2023-05-31T20:54:53.213 INFO:teuthology.orchestra.run.smithi099.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-05-31T20:54:53.213 INFO:teuthology.orchestra.run.smithi099.stdout: "storage_class": "", 2023-05-31T20:54:53.213 INFO:teuthology.orchestra.run.smithi099.stdout: "owner": "rgw-multisite-test-user", 2023-05-31T20:54:53.213 INFO:teuthology.orchestra.run.smithi099.stdout: "owner_display_name": "Multisite Test User", 2023-05-31T20:54:53.213 INFO:teuthology.orchestra.run.smithi099.stdout: "content_type": "application/octet-stream", 2023-05-31T20:54:53.214 INFO:teuthology.orchestra.run.smithi099.stdout: "accounted_size": 3, 2023-05-31T20:54:53.214 INFO:teuthology.orchestra.run.smithi099.stdout: "user_data": "", 2023-05-31T20:54:53.214 INFO:teuthology.orchestra.run.smithi099.stdout: "appendable": false 2023-05-31T20:54:53.214 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:54:53.214 INFO:teuthology.orchestra.run.smithi099.stdout: "tag": "3d163b1e-09c7-47e2-ae87-add4aace3f38.5680.14306150752147386781", 2023-05-31T20:54:53.214 INFO:teuthology.orchestra.run.smithi099.stdout: "flags": 0, 2023-05-31T20:54:53.214 INFO:teuthology.orchestra.run.smithi099.stdout: "pending_map": [], 2023-05-31T20:54:53.215 INFO:teuthology.orchestra.run.smithi099.stdout: "versioned_epoch": 0 2023-05-31T20:54:53.215 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:54:53.215 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T20:54:53.215 INFO:teuthology.orchestra.run.smithi099.stdout: "name": "obj38", 2023-05-31T20:54:53.215 INFO:teuthology.orchestra.run.smithi099.stdout: "instance": "", 2023-05-31T20:54:53.215 INFO:teuthology.orchestra.run.smithi099.stdout: "ver": { 2023-05-31T20:54:53.215 INFO:teuthology.orchestra.run.smithi099.stdout: "pool": 9, 2023-05-31T20:54:53.216 INFO:teuthology.orchestra.run.smithi099.stdout: "epoch": 13 2023-05-31T20:54:53.216 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:54:53.216 INFO:teuthology.orchestra.run.smithi099.stdout: "locator": "", 2023-05-31T20:54:53.216 INFO:teuthology.orchestra.run.smithi099.stdout: "exists": true, 2023-05-31T20:54:53.216 INFO:teuthology.orchestra.run.smithi099.stdout: "meta": { 2023-05-31T20:54:53.216 INFO:teuthology.orchestra.run.smithi099.stdout: "category": 1, 2023-05-31T20:54:53.216 INFO:teuthology.orchestra.run.smithi099.stdout: "size": 16, 2023-05-31T20:54:53.217 INFO:teuthology.orchestra.run.smithi099.stdout: "mtime": "2023-05-31T20:44:50.610918Z", 2023-05-31T20:54:53.217 INFO:teuthology.orchestra.run.smithi099.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-05-31T20:54:53.217 INFO:teuthology.orchestra.run.smithi099.stdout: "storage_class": "", 2023-05-31T20:54:53.217 INFO:teuthology.orchestra.run.smithi099.stdout: "owner": "rgw-multisite-test-user", 2023-05-31T20:54:53.217 INFO:teuthology.orchestra.run.smithi099.stdout: "owner_display_name": "Multisite Test User", 2023-05-31T20:54:53.217 INFO:teuthology.orchestra.run.smithi099.stdout: "content_type": "application/octet-stream", 2023-05-31T20:54:53.217 INFO:teuthology.orchestra.run.smithi099.stdout: "accounted_size": 3, 2023-05-31T20:54:53.218 INFO:teuthology.orchestra.run.smithi099.stdout: "user_data": "", 2023-05-31T20:54:53.218 INFO:teuthology.orchestra.run.smithi099.stdout: "appendable": false 2023-05-31T20:54:53.218 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:54:53.218 INFO:teuthology.orchestra.run.smithi099.stdout: "tag": "3d163b1e-09c7-47e2-ae87-add4aace3f38.5680.3587300964602934833", 2023-05-31T20:54:53.218 INFO:teuthology.orchestra.run.smithi099.stdout: "flags": 0, 2023-05-31T20:54:53.218 INFO:teuthology.orchestra.run.smithi099.stdout: "pending_map": [], 2023-05-31T20:54:53.218 INFO:teuthology.orchestra.run.smithi099.stdout: "versioned_epoch": 0 2023-05-31T20:54:53.219 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:54:53.219 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T20:54:53.219 INFO:teuthology.orchestra.run.smithi099.stdout: "name": "obj44", 2023-05-31T20:54:53.219 INFO:teuthology.orchestra.run.smithi099.stdout: "instance": "", 2023-05-31T20:54:53.219 INFO:teuthology.orchestra.run.smithi099.stdout: "ver": { 2023-05-31T20:54:53.219 INFO:teuthology.orchestra.run.smithi099.stdout: "pool": 9, 2023-05-31T20:54:53.219 INFO:teuthology.orchestra.run.smithi099.stdout: "epoch": 6 2023-05-31T20:54:53.220 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:54:53.220 INFO:teuthology.orchestra.run.smithi099.stdout: "locator": "", 2023-05-31T20:54:53.220 INFO:teuthology.orchestra.run.smithi099.stdout: "exists": true, 2023-05-31T20:54:53.220 INFO:teuthology.orchestra.run.smithi099.stdout: "meta": { 2023-05-31T20:54:53.220 INFO:teuthology.orchestra.run.smithi099.stdout: "category": 1, 2023-05-31T20:54:53.220 INFO:teuthology.orchestra.run.smithi099.stdout: "size": 5, 2023-05-31T20:54:53.220 INFO:teuthology.orchestra.run.smithi099.stdout: "mtime": "2023-05-31T20:39:26.565648Z", 2023-05-31T20:54:53.221 INFO:teuthology.orchestra.run.smithi099.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-05-31T20:54:53.221 INFO:teuthology.orchestra.run.smithi099.stdout: "storage_class": "", 2023-05-31T20:54:53.221 INFO:teuthology.orchestra.run.smithi099.stdout: "owner": "rgw-multisite-test-user", 2023-05-31T20:54:53.221 INFO:teuthology.orchestra.run.smithi099.stdout: "owner_display_name": "Multisite Test User", 2023-05-31T20:54:53.221 INFO:teuthology.orchestra.run.smithi099.stdout: "content_type": "application/octet-stream", 2023-05-31T20:54:53.221 INFO:teuthology.orchestra.run.smithi099.stdout: "accounted_size": 3, 2023-05-31T20:54:53.221 INFO:teuthology.orchestra.run.smithi099.stdout: "user_data": "", 2023-05-31T20:54:53.222 INFO:teuthology.orchestra.run.smithi099.stdout: "appendable": false 2023-05-31T20:54:53.222 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:54:53.222 INFO:teuthology.orchestra.run.smithi099.stdout: "tag": "3d163b1e-09c7-47e2-ae87-add4aace3f38.5680.1537521458725935231", 2023-05-31T20:54:53.222 INFO:teuthology.orchestra.run.smithi099.stdout: "flags": 0, 2023-05-31T20:54:53.222 INFO:teuthology.orchestra.run.smithi099.stdout: "pending_map": [], 2023-05-31T20:54:53.222 INFO:teuthology.orchestra.run.smithi099.stdout: "versioned_epoch": 0 2023-05-31T20:54:53.222 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:54:53.223 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T20:54:53.223 INFO:teuthology.orchestra.run.smithi099.stdout: "name": "obj46", 2023-05-31T20:54:53.223 INFO:teuthology.orchestra.run.smithi099.stdout: "instance": "", 2023-05-31T20:54:53.223 INFO:teuthology.orchestra.run.smithi099.stdout: "ver": { 2023-05-31T20:54:53.223 INFO:teuthology.orchestra.run.smithi099.stdout: "pool": 9, 2023-05-31T20:54:53.223 INFO:teuthology.orchestra.run.smithi099.stdout: "epoch": 4 2023-05-31T20:54:53.224 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:54:53.224 INFO:teuthology.orchestra.run.smithi099.stdout: "locator": "", 2023-05-31T20:54:53.224 INFO:teuthology.orchestra.run.smithi099.stdout: "exists": true, 2023-05-31T20:54:53.224 INFO:teuthology.orchestra.run.smithi099.stdout: "meta": { 2023-05-31T20:54:53.224 INFO:teuthology.orchestra.run.smithi099.stdout: "category": 1, 2023-05-31T20:54:53.224 INFO:teuthology.orchestra.run.smithi099.stdout: "size": 6, 2023-05-31T20:54:53.224 INFO:teuthology.orchestra.run.smithi099.stdout: "mtime": "2023-05-31T20:46:33.830640Z", 2023-05-31T20:54:53.225 INFO:teuthology.orchestra.run.smithi099.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-05-31T20:54:53.225 INFO:teuthology.orchestra.run.smithi099.stdout: "storage_class": "", 2023-05-31T20:54:53.225 INFO:teuthology.orchestra.run.smithi099.stdout: "owner": "rgw-multisite-test-user", 2023-05-31T20:54:53.225 INFO:teuthology.orchestra.run.smithi099.stdout: "owner_display_name": "Multisite Test User", 2023-05-31T20:54:53.225 INFO:teuthology.orchestra.run.smithi099.stdout: "content_type": "application/octet-stream", 2023-05-31T20:54:53.225 INFO:teuthology.orchestra.run.smithi099.stdout: "accounted_size": 3, 2023-05-31T20:54:53.225 INFO:teuthology.orchestra.run.smithi099.stdout: "user_data": "", 2023-05-31T20:54:53.226 INFO:teuthology.orchestra.run.smithi099.stdout: "appendable": false 2023-05-31T20:54:53.226 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:54:53.226 INFO:teuthology.orchestra.run.smithi099.stdout: "tag": "3d163b1e-09c7-47e2-ae87-add4aace3f38.5680.11796791252089577861", 2023-05-31T20:54:53.226 INFO:teuthology.orchestra.run.smithi099.stdout: "flags": 0, 2023-05-31T20:54:53.226 INFO:teuthology.orchestra.run.smithi099.stdout: "pending_map": [], 2023-05-31T20:54:53.226 INFO:teuthology.orchestra.run.smithi099.stdout: "versioned_epoch": 0 2023-05-31T20:54:53.226 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:54:53.227 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T20:54:53.227 INFO:teuthology.orchestra.run.smithi099.stdout: "name": "obj51", 2023-05-31T20:54:53.227 INFO:teuthology.orchestra.run.smithi099.stdout: "instance": "", 2023-05-31T20:54:53.227 INFO:teuthology.orchestra.run.smithi099.stdout: "ver": { 2023-05-31T20:54:53.227 INFO:teuthology.orchestra.run.smithi099.stdout: "pool": 9, 2023-05-31T20:54:53.227 INFO:teuthology.orchestra.run.smithi099.stdout: "epoch": 8 2023-05-31T20:54:53.227 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:54:53.228 INFO:teuthology.orchestra.run.smithi099.stdout: "locator": "", 2023-05-31T20:54:53.228 INFO:teuthology.orchestra.run.smithi099.stdout: "exists": true, 2023-05-31T20:54:53.228 INFO:teuthology.orchestra.run.smithi099.stdout: "meta": { 2023-05-31T20:54:53.228 INFO:teuthology.orchestra.run.smithi099.stdout: "category": 1, 2023-05-31T20:54:53.228 INFO:teuthology.orchestra.run.smithi099.stdout: "size": 16, 2023-05-31T20:54:53.228 INFO:teuthology.orchestra.run.smithi099.stdout: "mtime": "2023-05-31T20:43:08.011797Z", 2023-05-31T20:54:53.228 INFO:teuthology.orchestra.run.smithi099.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-05-31T20:54:53.229 INFO:teuthology.orchestra.run.smithi099.stdout: "storage_class": "", 2023-05-31T20:54:53.229 INFO:teuthology.orchestra.run.smithi099.stdout: "owner": "rgw-multisite-test-user", 2023-05-31T20:54:53.229 INFO:teuthology.orchestra.run.smithi099.stdout: "owner_display_name": "Multisite Test User", 2023-05-31T20:54:53.229 INFO:teuthology.orchestra.run.smithi099.stdout: "content_type": "application/octet-stream", 2023-05-31T20:54:53.229 INFO:teuthology.orchestra.run.smithi099.stdout: "accounted_size": 3, 2023-05-31T20:54:53.229 INFO:teuthology.orchestra.run.smithi099.stdout: "user_data": "", 2023-05-31T20:54:53.229 INFO:teuthology.orchestra.run.smithi099.stdout: "appendable": false 2023-05-31T20:54:53.230 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:54:53.230 INFO:teuthology.orchestra.run.smithi099.stdout: "tag": "3d163b1e-09c7-47e2-ae87-add4aace3f38.5680.4660573097437846228", 2023-05-31T20:54:53.230 INFO:teuthology.orchestra.run.smithi099.stdout: "flags": 0, 2023-05-31T20:54:53.230 INFO:teuthology.orchestra.run.smithi099.stdout: "pending_map": [], 2023-05-31T20:54:53.230 INFO:teuthology.orchestra.run.smithi099.stdout: "versioned_epoch": 0 2023-05-31T20:54:53.230 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:54:53.230 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T20:54:53.231 INFO:teuthology.orchestra.run.smithi099.stdout: "name": "obj55", 2023-05-31T20:54:53.231 INFO:teuthology.orchestra.run.smithi099.stdout: "instance": "", 2023-05-31T20:54:53.231 INFO:teuthology.orchestra.run.smithi099.stdout: "ver": { 2023-05-31T20:54:53.231 INFO:teuthology.orchestra.run.smithi099.stdout: "pool": 9, 2023-05-31T20:54:53.231 INFO:teuthology.orchestra.run.smithi099.stdout: "epoch": 12 2023-05-31T20:54:53.231 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:54:53.231 INFO:teuthology.orchestra.run.smithi099.stdout: "locator": "", 2023-05-31T20:54:53.232 INFO:teuthology.orchestra.run.smithi099.stdout: "exists": true, 2023-05-31T20:54:53.232 INFO:teuthology.orchestra.run.smithi099.stdout: "meta": { 2023-05-31T20:54:53.232 INFO:teuthology.orchestra.run.smithi099.stdout: "category": 1, 2023-05-31T20:54:53.232 INFO:teuthology.orchestra.run.smithi099.stdout: "size": 5, 2023-05-31T20:54:53.232 INFO:teuthology.orchestra.run.smithi099.stdout: "mtime": "2023-05-31T20:39:26.630472Z", 2023-05-31T20:54:53.232 INFO:teuthology.orchestra.run.smithi099.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-05-31T20:54:53.233 INFO:teuthology.orchestra.run.smithi099.stdout: "storage_class": "", 2023-05-31T20:54:53.233 INFO:teuthology.orchestra.run.smithi099.stdout: "owner": "rgw-multisite-test-user", 2023-05-31T20:54:53.233 INFO:teuthology.orchestra.run.smithi099.stdout: "owner_display_name": "Multisite Test User", 2023-05-31T20:54:53.233 INFO:teuthology.orchestra.run.smithi099.stdout: "content_type": "application/octet-stream", 2023-05-31T20:54:53.233 INFO:teuthology.orchestra.run.smithi099.stdout: "accounted_size": 3, 2023-05-31T20:54:53.233 INFO:teuthology.orchestra.run.smithi099.stdout: "user_data": "", 2023-05-31T20:54:53.233 INFO:teuthology.orchestra.run.smithi099.stdout: "appendable": false 2023-05-31T20:54:53.234 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:54:53.234 INFO:teuthology.orchestra.run.smithi099.stdout: "tag": "3d163b1e-09c7-47e2-ae87-add4aace3f38.5680.14260246798228966448", 2023-05-31T20:54:53.234 INFO:teuthology.orchestra.run.smithi099.stdout: "flags": 0, 2023-05-31T20:54:53.234 INFO:teuthology.orchestra.run.smithi099.stdout: "pending_map": [], 2023-05-31T20:54:53.234 INFO:teuthology.orchestra.run.smithi099.stdout: "versioned_epoch": 0 2023-05-31T20:54:53.234 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:54:53.234 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T20:54:53.235 INFO:teuthology.orchestra.run.smithi099.stdout: "name": "obj57", 2023-05-31T20:54:53.235 INFO:teuthology.orchestra.run.smithi099.stdout: "instance": "", 2023-05-31T20:54:53.235 INFO:teuthology.orchestra.run.smithi099.stdout: "ver": { 2023-05-31T20:54:53.235 INFO:teuthology.orchestra.run.smithi099.stdout: "pool": 9, 2023-05-31T20:54:53.235 INFO:teuthology.orchestra.run.smithi099.stdout: "epoch": 3 2023-05-31T20:54:53.235 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:54:53.235 INFO:teuthology.orchestra.run.smithi099.stdout: "locator": "", 2023-05-31T20:54:53.236 INFO:teuthology.orchestra.run.smithi099.stdout: "exists": true, 2023-05-31T20:54:53.236 INFO:teuthology.orchestra.run.smithi099.stdout: "meta": { 2023-05-31T20:54:53.236 INFO:teuthology.orchestra.run.smithi099.stdout: "category": 1, 2023-05-31T20:54:53.236 INFO:teuthology.orchestra.run.smithi099.stdout: "size": 16, 2023-05-31T20:54:53.236 INFO:teuthology.orchestra.run.smithi099.stdout: "mtime": "2023-05-31T20:44:50.627910Z", 2023-05-31T20:54:53.236 INFO:teuthology.orchestra.run.smithi099.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-05-31T20:54:53.236 INFO:teuthology.orchestra.run.smithi099.stdout: "storage_class": "", 2023-05-31T20:54:53.237 INFO:teuthology.orchestra.run.smithi099.stdout: "owner": "rgw-multisite-test-user", 2023-05-31T20:54:53.237 INFO:teuthology.orchestra.run.smithi099.stdout: "owner_display_name": "Multisite Test User", 2023-05-31T20:54:53.237 INFO:teuthology.orchestra.run.smithi099.stdout: "content_type": "application/octet-stream", 2023-05-31T20:54:53.237 INFO:teuthology.orchestra.run.smithi099.stdout: "accounted_size": 3, 2023-05-31T20:54:53.237 INFO:teuthology.orchestra.run.smithi099.stdout: "user_data": "", 2023-05-31T20:54:53.237 INFO:teuthology.orchestra.run.smithi099.stdout: "appendable": false 2023-05-31T20:54:53.237 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:54:53.238 INFO:teuthology.orchestra.run.smithi099.stdout: "tag": "3d163b1e-09c7-47e2-ae87-add4aace3f38.5680.16602446054877755122", 2023-05-31T20:54:53.238 INFO:teuthology.orchestra.run.smithi099.stdout: "flags": 0, 2023-05-31T20:54:53.238 INFO:teuthology.orchestra.run.smithi099.stdout: "pending_map": [], 2023-05-31T20:54:53.238 INFO:teuthology.orchestra.run.smithi099.stdout: "versioned_epoch": 0 2023-05-31T20:54:53.238 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:54:53.238 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T20:54:53.238 INFO:teuthology.orchestra.run.smithi099.stdout: "name": "obj58", 2023-05-31T20:54:53.239 INFO:teuthology.orchestra.run.smithi099.stdout: "instance": "", 2023-05-31T20:54:53.239 INFO:teuthology.orchestra.run.smithi099.stdout: "ver": { 2023-05-31T20:54:53.239 INFO:teuthology.orchestra.run.smithi099.stdout: "pool": 9, 2023-05-31T20:54:53.239 INFO:teuthology.orchestra.run.smithi099.stdout: "epoch": 6 2023-05-31T20:54:53.239 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:54:53.239 INFO:teuthology.orchestra.run.smithi099.stdout: "locator": "", 2023-05-31T20:54:53.239 INFO:teuthology.orchestra.run.smithi099.stdout: "exists": true, 2023-05-31T20:54:53.240 INFO:teuthology.orchestra.run.smithi099.stdout: "meta": { 2023-05-31T20:54:53.240 INFO:teuthology.orchestra.run.smithi099.stdout: "category": 1, 2023-05-31T20:54:53.240 INFO:teuthology.orchestra.run.smithi099.stdout: "size": 16, 2023-05-31T20:54:53.240 INFO:teuthology.orchestra.run.smithi099.stdout: "mtime": "2023-05-31T20:48:12.976191Z", 2023-05-31T20:54:53.240 INFO:teuthology.orchestra.run.smithi099.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-05-31T20:54:53.240 INFO:teuthology.orchestra.run.smithi099.stdout: "storage_class": "", 2023-05-31T20:54:53.241 INFO:teuthology.orchestra.run.smithi099.stdout: "owner": "rgw-multisite-test-user", 2023-05-31T20:54:53.241 INFO:teuthology.orchestra.run.smithi099.stdout: "owner_display_name": "Multisite Test User", 2023-05-31T20:54:53.241 INFO:teuthology.orchestra.run.smithi099.stdout: "content_type": "application/octet-stream", 2023-05-31T20:54:53.241 INFO:teuthology.orchestra.run.smithi099.stdout: "accounted_size": 3, 2023-05-31T20:54:53.241 INFO:teuthology.orchestra.run.smithi099.stdout: "user_data": "", 2023-05-31T20:54:53.241 INFO:teuthology.orchestra.run.smithi099.stdout: "appendable": false 2023-05-31T20:54:53.241 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:54:53.242 INFO:teuthology.orchestra.run.smithi099.stdout: "tag": "3d163b1e-09c7-47e2-ae87-add4aace3f38.5680.6681636744803174373", 2023-05-31T20:54:53.242 INFO:teuthology.orchestra.run.smithi099.stdout: "flags": 0, 2023-05-31T20:54:53.242 INFO:teuthology.orchestra.run.smithi099.stdout: "pending_map": [], 2023-05-31T20:54:53.242 INFO:teuthology.orchestra.run.smithi099.stdout: "versioned_epoch": 0 2023-05-31T20:54:53.242 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:54:53.242 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T20:54:53.242 INFO:teuthology.orchestra.run.smithi099.stdout: "name": "obj62", 2023-05-31T20:54:53.242 INFO:teuthology.orchestra.run.smithi099.stdout: "instance": "", 2023-05-31T20:54:53.243 INFO:teuthology.orchestra.run.smithi099.stdout: "ver": { 2023-05-31T20:54:53.243 INFO:teuthology.orchestra.run.smithi099.stdout: "pool": 9, 2023-05-31T20:54:53.243 INFO:teuthology.orchestra.run.smithi099.stdout: "epoch": 3 2023-05-31T20:54:53.243 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:54:53.243 INFO:teuthology.orchestra.run.smithi099.stdout: "locator": "", 2023-05-31T20:54:53.243 INFO:teuthology.orchestra.run.smithi099.stdout: "exists": true, 2023-05-31T20:54:53.243 INFO:teuthology.orchestra.run.smithi099.stdout: "meta": { 2023-05-31T20:54:53.244 INFO:teuthology.orchestra.run.smithi099.stdout: "category": 1, 2023-05-31T20:54:53.244 INFO:teuthology.orchestra.run.smithi099.stdout: "size": 5, 2023-05-31T20:54:53.244 INFO:teuthology.orchestra.run.smithi099.stdout: "mtime": "2023-05-31T20:50:33.515966Z", 2023-05-31T20:54:53.244 INFO:teuthology.orchestra.run.smithi099.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-05-31T20:54:53.244 INFO:teuthology.orchestra.run.smithi099.stdout: "storage_class": "", 2023-05-31T20:54:53.244 INFO:teuthology.orchestra.run.smithi099.stdout: "owner": "rgw-multisite-test-user", 2023-05-31T20:54:53.245 INFO:teuthology.orchestra.run.smithi099.stdout: "owner_display_name": "Multisite Test User", 2023-05-31T20:54:53.245 INFO:teuthology.orchestra.run.smithi099.stdout: "content_type": "application/octet-stream", 2023-05-31T20:54:53.245 INFO:teuthology.orchestra.run.smithi099.stdout: "accounted_size": 3, 2023-05-31T20:54:53.245 INFO:teuthology.orchestra.run.smithi099.stdout: "user_data": "", 2023-05-31T20:54:53.245 INFO:teuthology.orchestra.run.smithi099.stdout: "appendable": false 2023-05-31T20:54:53.245 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:54:53.245 INFO:teuthology.orchestra.run.smithi099.stdout: "tag": "3d163b1e-09c7-47e2-ae87-add4aace3f38.5680.4429903888965660935", 2023-05-31T20:54:53.246 INFO:teuthology.orchestra.run.smithi099.stdout: "flags": 0, 2023-05-31T20:54:53.246 INFO:teuthology.orchestra.run.smithi099.stdout: "pending_map": [], 2023-05-31T20:54:53.246 INFO:teuthology.orchestra.run.smithi099.stdout: "versioned_epoch": 0 2023-05-31T20:54:53.246 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:54:53.246 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T20:54:53.246 INFO:teuthology.orchestra.run.smithi099.stdout: "name": "obj68", 2023-05-31T20:54:53.246 INFO:teuthology.orchestra.run.smithi099.stdout: "instance": "", 2023-05-31T20:54:53.247 INFO:teuthology.orchestra.run.smithi099.stdout: "ver": { 2023-05-31T20:54:53.247 INFO:teuthology.orchestra.run.smithi099.stdout: "pool": 9, 2023-05-31T20:54:53.247 INFO:teuthology.orchestra.run.smithi099.stdout: "epoch": 13 2023-05-31T20:54:53.247 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:54:53.247 INFO:teuthology.orchestra.run.smithi099.stdout: "locator": "", 2023-05-31T20:54:53.247 INFO:teuthology.orchestra.run.smithi099.stdout: "exists": true, 2023-05-31T20:54:53.247 INFO:teuthology.orchestra.run.smithi099.stdout: "meta": { 2023-05-31T20:54:53.248 INFO:teuthology.orchestra.run.smithi099.stdout: "category": 1, 2023-05-31T20:54:53.248 INFO:teuthology.orchestra.run.smithi099.stdout: "size": 16, 2023-05-31T20:54:53.248 INFO:teuthology.orchestra.run.smithi099.stdout: "mtime": "2023-05-31T20:43:08.066803Z", 2023-05-31T20:54:53.248 INFO:teuthology.orchestra.run.smithi099.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-05-31T20:54:53.248 INFO:teuthology.orchestra.run.smithi099.stdout: "storage_class": "", 2023-05-31T20:54:53.248 INFO:teuthology.orchestra.run.smithi099.stdout: "owner": "rgw-multisite-test-user", 2023-05-31T20:54:53.248 INFO:teuthology.orchestra.run.smithi099.stdout: "owner_display_name": "Multisite Test User", 2023-05-31T20:54:53.249 INFO:teuthology.orchestra.run.smithi099.stdout: "content_type": "application/octet-stream", 2023-05-31T20:54:53.249 INFO:teuthology.orchestra.run.smithi099.stdout: "accounted_size": 3, 2023-05-31T20:54:53.249 INFO:teuthology.orchestra.run.smithi099.stdout: "user_data": "", 2023-05-31T20:54:53.249 INFO:teuthology.orchestra.run.smithi099.stdout: "appendable": false 2023-05-31T20:54:53.249 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:54:53.249 INFO:teuthology.orchestra.run.smithi099.stdout: "tag": "3d163b1e-09c7-47e2-ae87-add4aace3f38.5680.5649902209982896197", 2023-05-31T20:54:53.249 INFO:teuthology.orchestra.run.smithi099.stdout: "flags": 0, 2023-05-31T20:54:53.250 INFO:teuthology.orchestra.run.smithi099.stdout: "pending_map": [], 2023-05-31T20:54:53.250 INFO:teuthology.orchestra.run.smithi099.stdout: "versioned_epoch": 0 2023-05-31T20:54:53.250 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:54:53.250 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T20:54:53.250 INFO:teuthology.orchestra.run.smithi099.stdout: "name": "obj69", 2023-05-31T20:54:53.250 INFO:teuthology.orchestra.run.smithi099.stdout: "instance": "", 2023-05-31T20:54:53.250 INFO:teuthology.orchestra.run.smithi099.stdout: "ver": { 2023-05-31T20:54:53.251 INFO:teuthology.orchestra.run.smithi099.stdout: "pool": 9, 2023-05-31T20:54:53.251 INFO:teuthology.orchestra.run.smithi099.stdout: "epoch": 1 2023-05-31T20:54:53.251 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:54:53.251 INFO:teuthology.orchestra.run.smithi099.stdout: "locator": "", 2023-05-31T20:54:53.251 INFO:teuthology.orchestra.run.smithi099.stdout: "exists": true, 2023-05-31T20:54:53.251 INFO:teuthology.orchestra.run.smithi099.stdout: "meta": { 2023-05-31T20:54:53.251 INFO:teuthology.orchestra.run.smithi099.stdout: "category": 1, 2023-05-31T20:54:53.252 INFO:teuthology.orchestra.run.smithi099.stdout: "size": 6, 2023-05-31T20:54:53.252 INFO:teuthology.orchestra.run.smithi099.stdout: "mtime": "2023-05-31T20:46:33.886933Z", 2023-05-31T20:54:53.252 INFO:teuthology.orchestra.run.smithi099.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-05-31T20:54:53.252 INFO:teuthology.orchestra.run.smithi099.stdout: "storage_class": "", 2023-05-31T20:54:53.252 INFO:teuthology.orchestra.run.smithi099.stdout: "owner": "rgw-multisite-test-user", 2023-05-31T20:54:53.252 INFO:teuthology.orchestra.run.smithi099.stdout: "owner_display_name": "Multisite Test User", 2023-05-31T20:54:53.252 INFO:teuthology.orchestra.run.smithi099.stdout: "content_type": "application/octet-stream", 2023-05-31T20:54:53.253 INFO:teuthology.orchestra.run.smithi099.stdout: "accounted_size": 3, 2023-05-31T20:54:53.253 INFO:teuthology.orchestra.run.smithi099.stdout: "user_data": "", 2023-05-31T20:54:53.253 INFO:teuthology.orchestra.run.smithi099.stdout: "appendable": false 2023-05-31T20:54:53.253 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:54:53.253 INFO:teuthology.orchestra.run.smithi099.stdout: "tag": "3d163b1e-09c7-47e2-ae87-add4aace3f38.5680.10976639480736641454", 2023-05-31T20:54:53.253 INFO:teuthology.orchestra.run.smithi099.stdout: "flags": 0, 2023-05-31T20:54:53.254 INFO:teuthology.orchestra.run.smithi099.stdout: "pending_map": [], 2023-05-31T20:54:53.254 INFO:teuthology.orchestra.run.smithi099.stdout: "versioned_epoch": 0 2023-05-31T20:54:53.254 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:54:53.254 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T20:54:53.254 INFO:teuthology.orchestra.run.smithi099.stdout: "name": "obj74", 2023-05-31T20:54:53.254 INFO:teuthology.orchestra.run.smithi099.stdout: "instance": "", 2023-05-31T20:54:53.255 INFO:teuthology.orchestra.run.smithi099.stdout: "ver": { 2023-05-31T20:54:53.255 INFO:teuthology.orchestra.run.smithi099.stdout: "pool": 9, 2023-05-31T20:54:53.255 INFO:teuthology.orchestra.run.smithi099.stdout: "epoch": 25 2023-05-31T20:54:53.255 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:54:53.255 INFO:teuthology.orchestra.run.smithi099.stdout: "locator": "", 2023-05-31T20:54:53.255 INFO:teuthology.orchestra.run.smithi099.stdout: "exists": true, 2023-05-31T20:54:53.255 INFO:teuthology.orchestra.run.smithi099.stdout: "meta": { 2023-05-31T20:54:53.256 INFO:teuthology.orchestra.run.smithi099.stdout: "category": 1, 2023-05-31T20:54:53.256 INFO:teuthology.orchestra.run.smithi099.stdout: "size": 6, 2023-05-31T20:54:53.256 INFO:teuthology.orchestra.run.smithi099.stdout: "mtime": "2023-05-31T20:52:14.934725Z", 2023-05-31T20:54:53.256 INFO:teuthology.orchestra.run.smithi099.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-05-31T20:54:53.256 INFO:teuthology.orchestra.run.smithi099.stdout: "storage_class": "", 2023-05-31T20:54:53.256 INFO:teuthology.orchestra.run.smithi099.stdout: "owner": "rgw-multisite-test-user", 2023-05-31T20:54:53.256 INFO:teuthology.orchestra.run.smithi099.stdout: "owner_display_name": "Multisite Test User", 2023-05-31T20:54:53.257 INFO:teuthology.orchestra.run.smithi099.stdout: "content_type": "application/octet-stream", 2023-05-31T20:54:53.257 INFO:teuthology.orchestra.run.smithi099.stdout: "accounted_size": 3, 2023-05-31T20:54:53.257 INFO:teuthology.orchestra.run.smithi099.stdout: "user_data": "", 2023-05-31T20:54:53.257 INFO:teuthology.orchestra.run.smithi099.stdout: "appendable": false 2023-05-31T20:54:53.257 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:54:53.257 INFO:teuthology.orchestra.run.smithi099.stdout: "tag": "3d163b1e-09c7-47e2-ae87-add4aace3f38.5680.8984440930279448741", 2023-05-31T20:54:53.257 INFO:teuthology.orchestra.run.smithi099.stdout: "flags": 0, 2023-05-31T20:54:53.258 INFO:teuthology.orchestra.run.smithi099.stdout: "pending_map": [], 2023-05-31T20:54:53.258 INFO:teuthology.orchestra.run.smithi099.stdout: "versioned_epoch": 0 2023-05-31T20:54:53.258 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:54:53.258 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T20:54:53.258 INFO:teuthology.orchestra.run.smithi099.stdout: "name": "obj76", 2023-05-31T20:54:53.258 INFO:teuthology.orchestra.run.smithi099.stdout: "instance": "", 2023-05-31T20:54:53.258 INFO:teuthology.orchestra.run.smithi099.stdout: "ver": { 2023-05-31T20:54:53.259 INFO:teuthology.orchestra.run.smithi099.stdout: "pool": 9, 2023-05-31T20:54:53.259 INFO:teuthology.orchestra.run.smithi099.stdout: "epoch": 13 2023-05-31T20:54:53.259 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:54:53.259 INFO:teuthology.orchestra.run.smithi099.stdout: "locator": "", 2023-05-31T20:54:53.259 INFO:teuthology.orchestra.run.smithi099.stdout: "exists": true, 2023-05-31T20:54:53.259 INFO:teuthology.orchestra.run.smithi099.stdout: "meta": { 2023-05-31T20:54:53.260 INFO:teuthology.orchestra.run.smithi099.stdout: "category": 1, 2023-05-31T20:54:53.260 INFO:teuthology.orchestra.run.smithi099.stdout: "size": 3, 2023-05-31T20:54:53.260 INFO:teuthology.orchestra.run.smithi099.stdout: "mtime": "2023-05-31T20:44:50.682916Z", 2023-05-31T20:54:53.260 INFO:teuthology.orchestra.run.smithi099.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-05-31T20:54:53.260 INFO:teuthology.orchestra.run.smithi099.stdout: "storage_class": "", 2023-05-31T20:54:53.260 INFO:teuthology.orchestra.run.smithi099.stdout: "owner": "rgw-multisite-test-user", 2023-05-31T20:54:53.261 INFO:teuthology.orchestra.run.smithi099.stdout: "owner_display_name": "Multisite Test User", 2023-05-31T20:54:53.261 INFO:teuthology.orchestra.run.smithi099.stdout: "content_type": "application/octet-stream", 2023-05-31T20:54:53.261 INFO:teuthology.orchestra.run.smithi099.stdout: "accounted_size": 3, 2023-05-31T20:54:53.261 INFO:teuthology.orchestra.run.smithi099.stdout: "user_data": "", 2023-05-31T20:54:53.261 INFO:teuthology.orchestra.run.smithi099.stdout: "appendable": false 2023-05-31T20:54:53.261 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:54:53.261 INFO:teuthology.orchestra.run.smithi099.stdout: "tag": "3d163b1e-09c7-47e2-ae87-add4aace3f38.5680.6136035549046558794", 2023-05-31T20:54:53.262 INFO:teuthology.orchestra.run.smithi099.stdout: "flags": 0, 2023-05-31T20:54:53.262 INFO:teuthology.orchestra.run.smithi099.stdout: "pending_map": [], 2023-05-31T20:54:53.262 INFO:teuthology.orchestra.run.smithi099.stdout: "versioned_epoch": 0 2023-05-31T20:54:53.262 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:54:53.262 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T20:54:53.262 INFO:teuthology.orchestra.run.smithi099.stdout: "name": "obj85", 2023-05-31T20:54:53.262 INFO:teuthology.orchestra.run.smithi099.stdout: "instance": "", 2023-05-31T20:54:53.263 INFO:teuthology.orchestra.run.smithi099.stdout: "ver": { 2023-05-31T20:54:53.263 INFO:teuthology.orchestra.run.smithi099.stdout: "pool": 9, 2023-05-31T20:54:53.264 INFO:teuthology.orchestra.run.smithi099.stdout: "epoch": 5 2023-05-31T20:54:53.264 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:54:53.264 INFO:teuthology.orchestra.run.smithi099.stdout: "locator": "", 2023-05-31T20:54:53.264 INFO:teuthology.orchestra.run.smithi099.stdout: "exists": true, 2023-05-31T20:54:53.264 INFO:teuthology.orchestra.run.smithi099.stdout: "meta": { 2023-05-31T20:54:53.265 INFO:teuthology.orchestra.run.smithi099.stdout: "category": 1, 2023-05-31T20:54:53.265 INFO:teuthology.orchestra.run.smithi099.stdout: "size": 3, 2023-05-31T20:54:53.265 INFO:teuthology.orchestra.run.smithi099.stdout: "mtime": "2023-05-31T20:43:08.086089Z", 2023-05-31T20:54:53.265 INFO:teuthology.orchestra.run.smithi099.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-05-31T20:54:53.265 INFO:teuthology.orchestra.run.smithi099.stdout: "storage_class": "", 2023-05-31T20:54:53.265 INFO:teuthology.orchestra.run.smithi099.stdout: "owner": "rgw-multisite-test-user", 2023-05-31T20:54:53.265 INFO:teuthology.orchestra.run.smithi099.stdout: "owner_display_name": "Multisite Test User", 2023-05-31T20:54:53.266 INFO:teuthology.orchestra.run.smithi099.stdout: "content_type": "application/octet-stream", 2023-05-31T20:54:53.266 INFO:teuthology.orchestra.run.smithi099.stdout: "accounted_size": 3, 2023-05-31T20:54:53.266 INFO:teuthology.orchestra.run.smithi099.stdout: "user_data": "", 2023-05-31T20:54:53.266 INFO:teuthology.orchestra.run.smithi099.stdout: "appendable": false 2023-05-31T20:54:53.266 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:54:53.266 INFO:teuthology.orchestra.run.smithi099.stdout: "tag": "3d163b1e-09c7-47e2-ae87-add4aace3f38.5680.13616303383336706590", 2023-05-31T20:54:53.267 INFO:teuthology.orchestra.run.smithi099.stdout: "flags": 0, 2023-05-31T20:54:53.267 INFO:teuthology.orchestra.run.smithi099.stdout: "pending_map": [], 2023-05-31T20:54:53.267 INFO:teuthology.orchestra.run.smithi099.stdout: "versioned_epoch": 0 2023-05-31T20:54:53.267 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:54:53.267 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T20:54:53.267 INFO:teuthology.orchestra.run.smithi099.stdout: "name": "obj87", 2023-05-31T20:54:53.267 INFO:teuthology.orchestra.run.smithi099.stdout: "instance": "", 2023-05-31T20:54:53.268 INFO:teuthology.orchestra.run.smithi099.stdout: "ver": { 2023-05-31T20:54:53.268 INFO:teuthology.orchestra.run.smithi099.stdout: "pool": 9, 2023-05-31T20:54:53.268 INFO:teuthology.orchestra.run.smithi099.stdout: "epoch": 12 2023-05-31T20:54:53.268 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:54:53.268 INFO:teuthology.orchestra.run.smithi099.stdout: "locator": "", 2023-05-31T20:54:53.268 INFO:teuthology.orchestra.run.smithi099.stdout: "exists": true, 2023-05-31T20:54:53.268 INFO:teuthology.orchestra.run.smithi099.stdout: "meta": { 2023-05-31T20:54:53.269 INFO:teuthology.orchestra.run.smithi099.stdout: "category": 1, 2023-05-31T20:54:53.269 INFO:teuthology.orchestra.run.smithi099.stdout: "size": 6, 2023-05-31T20:54:53.269 INFO:teuthology.orchestra.run.smithi099.stdout: "mtime": "2023-05-31T20:48:13.031366Z", 2023-05-31T20:54:53.269 INFO:teuthology.orchestra.run.smithi099.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-05-31T20:54:53.269 INFO:teuthology.orchestra.run.smithi099.stdout: "storage_class": "", 2023-05-31T20:54:53.269 INFO:teuthology.orchestra.run.smithi099.stdout: "owner": "rgw-multisite-test-user", 2023-05-31T20:54:53.269 INFO:teuthology.orchestra.run.smithi099.stdout: "owner_display_name": "Multisite Test User", 2023-05-31T20:54:53.270 INFO:teuthology.orchestra.run.smithi099.stdout: "content_type": "application/octet-stream", 2023-05-31T20:54:53.270 INFO:teuthology.orchestra.run.smithi099.stdout: "accounted_size": 3, 2023-05-31T20:54:53.270 INFO:teuthology.orchestra.run.smithi099.stdout: "user_data": "", 2023-05-31T20:54:53.270 INFO:teuthology.orchestra.run.smithi099.stdout: "appendable": false 2023-05-31T20:54:53.270 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:54:53.270 INFO:teuthology.orchestra.run.smithi099.stdout: "tag": "3d163b1e-09c7-47e2-ae87-add4aace3f38.5680.6806200294514102254", 2023-05-31T20:54:53.270 INFO:teuthology.orchestra.run.smithi099.stdout: "flags": 0, 2023-05-31T20:54:53.271 INFO:teuthology.orchestra.run.smithi099.stdout: "pending_map": [], 2023-05-31T20:54:53.271 INFO:teuthology.orchestra.run.smithi099.stdout: "versioned_epoch": 0 2023-05-31T20:54:53.271 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:54:53.271 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T20:54:53.271 INFO:teuthology.orchestra.run.smithi099.stdout: "name": "obj92", 2023-05-31T20:54:53.271 INFO:teuthology.orchestra.run.smithi099.stdout: "instance": "", 2023-05-31T20:54:53.272 INFO:teuthology.orchestra.run.smithi099.stdout: "ver": { 2023-05-31T20:54:53.272 INFO:teuthology.orchestra.run.smithi099.stdout: "pool": 9, 2023-05-31T20:54:53.272 INFO:teuthology.orchestra.run.smithi099.stdout: "epoch": 25 2023-05-31T20:54:53.272 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:54:53.272 INFO:teuthology.orchestra.run.smithi099.stdout: "locator": "", 2023-05-31T20:54:53.272 INFO:teuthology.orchestra.run.smithi099.stdout: "exists": true, 2023-05-31T20:54:53.272 INFO:teuthology.orchestra.run.smithi099.stdout: "meta": { 2023-05-31T20:54:53.273 INFO:teuthology.orchestra.run.smithi099.stdout: "category": 1, 2023-05-31T20:54:53.273 INFO:teuthology.orchestra.run.smithi099.stdout: "size": 16, 2023-05-31T20:54:53.273 INFO:teuthology.orchestra.run.smithi099.stdout: "mtime": "2023-05-31T20:46:33.904998Z", 2023-05-31T20:54:53.273 INFO:teuthology.orchestra.run.smithi099.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-05-31T20:54:53.273 INFO:teuthology.orchestra.run.smithi099.stdout: "storage_class": "", 2023-05-31T20:54:53.273 INFO:teuthology.orchestra.run.smithi099.stdout: "owner": "rgw-multisite-test-user", 2023-05-31T20:54:53.273 INFO:teuthology.orchestra.run.smithi099.stdout: "owner_display_name": "Multisite Test User", 2023-05-31T20:54:53.274 INFO:teuthology.orchestra.run.smithi099.stdout: "content_type": "application/octet-stream", 2023-05-31T20:54:53.274 INFO:teuthology.orchestra.run.smithi099.stdout: "accounted_size": 3, 2023-05-31T20:54:53.274 INFO:teuthology.orchestra.run.smithi099.stdout: "user_data": "", 2023-05-31T20:54:53.274 INFO:teuthology.orchestra.run.smithi099.stdout: "appendable": false 2023-05-31T20:54:53.274 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:54:53.274 INFO:teuthology.orchestra.run.smithi099.stdout: "tag": "3d163b1e-09c7-47e2-ae87-add4aace3f38.5680.5358348178921291786", 2023-05-31T20:54:53.274 INFO:teuthology.orchestra.run.smithi099.stdout: "flags": 0, 2023-05-31T20:54:53.275 INFO:teuthology.orchestra.run.smithi099.stdout: "pending_map": [], 2023-05-31T20:54:53.275 INFO:teuthology.orchestra.run.smithi099.stdout: "versioned_epoch": 0 2023-05-31T20:54:53.275 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:54:53.275 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T20:54:53.275 INFO:teuthology.orchestra.run.smithi099.stdout: "name": "obj93", 2023-05-31T20:54:53.275 INFO:teuthology.orchestra.run.smithi099.stdout: "instance": "", 2023-05-31T20:54:53.275 INFO:teuthology.orchestra.run.smithi099.stdout: "ver": { 2023-05-31T20:54:53.276 INFO:teuthology.orchestra.run.smithi099.stdout: "pool": 9, 2023-05-31T20:54:53.276 INFO:teuthology.orchestra.run.smithi099.stdout: "epoch": 3 2023-05-31T20:54:53.276 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:54:53.276 INFO:teuthology.orchestra.run.smithi099.stdout: "locator": "", 2023-05-31T20:54:53.276 INFO:teuthology.orchestra.run.smithi099.stdout: "exists": true, 2023-05-31T20:54:53.276 INFO:teuthology.orchestra.run.smithi099.stdout: "meta": { 2023-05-31T20:54:53.277 INFO:teuthology.orchestra.run.smithi099.stdout: "category": 1, 2023-05-31T20:54:53.277 INFO:teuthology.orchestra.run.smithi099.stdout: "size": 6, 2023-05-31T20:54:53.277 INFO:teuthology.orchestra.run.smithi099.stdout: "mtime": "2023-05-31T20:50:33.533538Z", 2023-05-31T20:54:53.277 INFO:teuthology.orchestra.run.smithi099.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-05-31T20:54:53.277 INFO:teuthology.orchestra.run.smithi099.stdout: "storage_class": "", 2023-05-31T20:54:53.277 INFO:teuthology.orchestra.run.smithi099.stdout: "owner": "rgw-multisite-test-user", 2023-05-31T20:54:53.277 INFO:teuthology.orchestra.run.smithi099.stdout: "owner_display_name": "Multisite Test User", 2023-05-31T20:54:53.278 INFO:teuthology.orchestra.run.smithi099.stdout: "content_type": "application/octet-stream", 2023-05-31T20:54:53.278 INFO:teuthology.orchestra.run.smithi099.stdout: "accounted_size": 3, 2023-05-31T20:54:53.278 INFO:teuthology.orchestra.run.smithi099.stdout: "user_data": "", 2023-05-31T20:54:53.278 INFO:teuthology.orchestra.run.smithi099.stdout: "appendable": false 2023-05-31T20:54:53.278 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:54:53.278 INFO:teuthology.orchestra.run.smithi099.stdout: "tag": "3d163b1e-09c7-47e2-ae87-add4aace3f38.5680.14799555098245270540", 2023-05-31T20:54:53.278 INFO:teuthology.orchestra.run.smithi099.stdout: "flags": 0, 2023-05-31T20:54:53.279 INFO:teuthology.orchestra.run.smithi099.stdout: "pending_map": [], 2023-05-31T20:54:53.279 INFO:teuthology.orchestra.run.smithi099.stdout: "versioned_epoch": 0 2023-05-31T20:54:53.279 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:54:53.279 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T20:54:53.279 INFO:teuthology.orchestra.run.smithi099.stdout: "name": "obj95", 2023-05-31T20:54:53.279 INFO:teuthology.orchestra.run.smithi099.stdout: "instance": "", 2023-05-31T20:54:53.279 INFO:teuthology.orchestra.run.smithi099.stdout: "ver": { 2023-05-31T20:54:53.280 INFO:teuthology.orchestra.run.smithi099.stdout: "pool": 9, 2023-05-31T20:54:53.280 INFO:teuthology.orchestra.run.smithi099.stdout: "epoch": 10 2023-05-31T20:54:53.280 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:54:53.280 INFO:teuthology.orchestra.run.smithi099.stdout: "locator": "", 2023-05-31T20:54:53.280 INFO:teuthology.orchestra.run.smithi099.stdout: "exists": true, 2023-05-31T20:54:53.280 INFO:teuthology.orchestra.run.smithi099.stdout: "meta": { 2023-05-31T20:54:53.281 INFO:teuthology.orchestra.run.smithi099.stdout: "category": 1, 2023-05-31T20:54:53.281 INFO:teuthology.orchestra.run.smithi099.stdout: "size": 5, 2023-05-31T20:54:53.281 INFO:teuthology.orchestra.run.smithi099.stdout: "mtime": "2023-05-31T20:44:50.705817Z", 2023-05-31T20:54:53.281 INFO:teuthology.orchestra.run.smithi099.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-05-31T20:54:53.281 INFO:teuthology.orchestra.run.smithi099.stdout: "storage_class": "", 2023-05-31T20:54:53.281 INFO:teuthology.orchestra.run.smithi099.stdout: "owner": "rgw-multisite-test-user", 2023-05-31T20:54:53.282 INFO:teuthology.orchestra.run.smithi099.stdout: "owner_display_name": "Multisite Test User", 2023-05-31T20:54:53.282 INFO:teuthology.orchestra.run.smithi099.stdout: "content_type": "application/octet-stream", 2023-05-31T20:54:53.282 INFO:teuthology.orchestra.run.smithi099.stdout: "accounted_size": 3, 2023-05-31T20:54:53.282 INFO:teuthology.orchestra.run.smithi099.stdout: "user_data": "", 2023-05-31T20:54:53.282 INFO:teuthology.orchestra.run.smithi099.stdout: "appendable": false 2023-05-31T20:54:53.282 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:54:53.282 INFO:teuthology.orchestra.run.smithi099.stdout: "tag": "3d163b1e-09c7-47e2-ae87-add4aace3f38.5680.1113219356438931954", 2023-05-31T20:54:53.283 INFO:teuthology.orchestra.run.smithi099.stdout: "flags": 0, 2023-05-31T20:54:53.283 INFO:teuthology.orchestra.run.smithi099.stdout: "pending_map": [], 2023-05-31T20:54:53.283 INFO:teuthology.orchestra.run.smithi099.stdout: "versioned_epoch": 0 2023-05-31T20:54:53.283 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T20:54:53.283 INFO:teuthology.orchestra.run.smithi099.stdout:] 2023-05-31T20:54:53.284 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'list', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'pklxkd-59', '--max-entries', '10000', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-31T20:54:53.284 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'list', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'pklxkd-59', '--max-entries', '10000', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-31T20:54:53.285 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket list --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --bucket pklxkd-59 --max-entries 10000 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-05-31T20:54:53.336 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2023-05-31T20:54:53.337 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2023-05-31T20:54:53.392 INFO:teuthology.orchestra.run.smithi099.stdout:[ 2023-05-31T20:54:53.392 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T20:54:53.392 INFO:teuthology.orchestra.run.smithi099.stdout: "name": "obj11", 2023-05-31T20:54:53.392 INFO:teuthology.orchestra.run.smithi099.stdout: "instance": "", 2023-05-31T20:54:53.392 INFO:teuthology.orchestra.run.smithi099.stdout: "ver": { 2023-05-31T20:54:53.393 INFO:teuthology.orchestra.run.smithi099.stdout: "pool": 13, 2023-05-31T20:54:53.393 INFO:teuthology.orchestra.run.smithi099.stdout: "epoch": 10 2023-05-31T20:54:53.393 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:54:53.393 INFO:teuthology.orchestra.run.smithi099.stdout: "locator": "", 2023-05-31T20:54:53.393 INFO:teuthology.orchestra.run.smithi099.stdout: "exists": true, 2023-05-31T20:54:53.393 INFO:teuthology.orchestra.run.smithi099.stdout: "meta": { 2023-05-31T20:54:53.394 INFO:teuthology.orchestra.run.smithi099.stdout: "category": 1, 2023-05-31T20:54:53.394 INFO:teuthology.orchestra.run.smithi099.stdout: "size": 16, 2023-05-31T20:54:53.394 INFO:teuthology.orchestra.run.smithi099.stdout: "mtime": "2023-05-31T20:39:26.427662Z", 2023-05-31T20:54:53.394 INFO:teuthology.orchestra.run.smithi099.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-05-31T20:54:53.394 INFO:teuthology.orchestra.run.smithi099.stdout: "storage_class": "", 2023-05-31T20:54:53.394 INFO:teuthology.orchestra.run.smithi099.stdout: "owner": "rgw-multisite-test-user", 2023-05-31T20:54:53.394 INFO:teuthology.orchestra.run.smithi099.stdout: "owner_display_name": "Multisite Test User", 2023-05-31T20:54:53.395 INFO:teuthology.orchestra.run.smithi099.stdout: "content_type": "application/octet-stream", 2023-05-31T20:54:53.395 INFO:teuthology.orchestra.run.smithi099.stdout: "accounted_size": 3, 2023-05-31T20:54:53.395 INFO:teuthology.orchestra.run.smithi099.stdout: "user_data": "", 2023-05-31T20:54:53.395 INFO:teuthology.orchestra.run.smithi099.stdout: "appendable": false 2023-05-31T20:54:53.395 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:54:53.395 INFO:teuthology.orchestra.run.smithi099.stdout: "tag": "_1FBjGTjtkf9KLJ_lA60gf-jDNurOyL1", 2023-05-31T20:54:53.395 INFO:teuthology.orchestra.run.smithi099.stdout: "flags": 0, 2023-05-31T20:54:53.396 INFO:teuthology.orchestra.run.smithi099.stdout: "pending_map": [], 2023-05-31T20:54:53.396 INFO:teuthology.orchestra.run.smithi099.stdout: "versioned_epoch": 0 2023-05-31T20:54:53.396 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:54:53.396 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T20:54:53.396 INFO:teuthology.orchestra.run.smithi099.stdout: "name": "obj111", 2023-05-31T20:54:53.396 INFO:teuthology.orchestra.run.smithi099.stdout: "instance": "", 2023-05-31T20:54:53.397 INFO:teuthology.orchestra.run.smithi099.stdout: "ver": { 2023-05-31T20:54:53.397 INFO:teuthology.orchestra.run.smithi099.stdout: "pool": 13, 2023-05-31T20:54:53.397 INFO:teuthology.orchestra.run.smithi099.stdout: "epoch": 8 2023-05-31T20:54:53.397 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:54:53.397 INFO:teuthology.orchestra.run.smithi099.stdout: "locator": "", 2023-05-31T20:54:53.397 INFO:teuthology.orchestra.run.smithi099.stdout: "exists": true, 2023-05-31T20:54:53.397 INFO:teuthology.orchestra.run.smithi099.stdout: "meta": { 2023-05-31T20:54:53.398 INFO:teuthology.orchestra.run.smithi099.stdout: "category": 1, 2023-05-31T20:54:53.398 INFO:teuthology.orchestra.run.smithi099.stdout: "size": 16, 2023-05-31T20:54:53.398 INFO:teuthology.orchestra.run.smithi099.stdout: "mtime": "2023-05-31T20:52:14.993359Z", 2023-05-31T20:54:53.398 INFO:teuthology.orchestra.run.smithi099.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-05-31T20:54:53.398 INFO:teuthology.orchestra.run.smithi099.stdout: "storage_class": "", 2023-05-31T20:54:53.398 INFO:teuthology.orchestra.run.smithi099.stdout: "owner": "rgw-multisite-test-user", 2023-05-31T20:54:53.398 INFO:teuthology.orchestra.run.smithi099.stdout: "owner_display_name": "Multisite Test User", 2023-05-31T20:54:53.399 INFO:teuthology.orchestra.run.smithi099.stdout: "content_type": "application/octet-stream", 2023-05-31T20:54:53.399 INFO:teuthology.orchestra.run.smithi099.stdout: "accounted_size": 3, 2023-05-31T20:54:53.399 INFO:teuthology.orchestra.run.smithi099.stdout: "user_data": "", 2023-05-31T20:54:53.399 INFO:teuthology.orchestra.run.smithi099.stdout: "appendable": false 2023-05-31T20:54:53.399 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:54:53.399 INFO:teuthology.orchestra.run.smithi099.stdout: "tag": "_c8PF1yvw8Wgay8a-EP9Hcx2a67YsCqy", 2023-05-31T20:54:53.399 INFO:teuthology.orchestra.run.smithi099.stdout: "flags": 0, 2023-05-31T20:54:53.400 INFO:teuthology.orchestra.run.smithi099.stdout: "pending_map": [], 2023-05-31T20:54:53.400 INFO:teuthology.orchestra.run.smithi099.stdout: "versioned_epoch": 0 2023-05-31T20:54:53.400 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:54:53.400 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T20:54:53.400 INFO:teuthology.orchestra.run.smithi099.stdout: "name": "obj115", 2023-05-31T20:54:53.400 INFO:teuthology.orchestra.run.smithi099.stdout: "instance": "", 2023-05-31T20:54:53.400 INFO:teuthology.orchestra.run.smithi099.stdout: "ver": { 2023-05-31T20:54:53.401 INFO:teuthology.orchestra.run.smithi099.stdout: "pool": 13, 2023-05-31T20:54:53.401 INFO:teuthology.orchestra.run.smithi099.stdout: "epoch": 6 2023-05-31T20:54:53.401 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:54:53.401 INFO:teuthology.orchestra.run.smithi099.stdout: "locator": "", 2023-05-31T20:54:53.401 INFO:teuthology.orchestra.run.smithi099.stdout: "exists": true, 2023-05-31T20:54:53.401 INFO:teuthology.orchestra.run.smithi099.stdout: "meta": { 2023-05-31T20:54:53.401 INFO:teuthology.orchestra.run.smithi099.stdout: "category": 1, 2023-05-31T20:54:53.402 INFO:teuthology.orchestra.run.smithi099.stdout: "size": 3, 2023-05-31T20:54:53.402 INFO:teuthology.orchestra.run.smithi099.stdout: "mtime": "2023-05-31T20:46:33.958937Z", 2023-05-31T20:54:53.402 INFO:teuthology.orchestra.run.smithi099.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-05-31T20:54:53.402 INFO:teuthology.orchestra.run.smithi099.stdout: "storage_class": "", 2023-05-31T20:54:53.402 INFO:teuthology.orchestra.run.smithi099.stdout: "owner": "rgw-multisite-test-user", 2023-05-31T20:54:53.402 INFO:teuthology.orchestra.run.smithi099.stdout: "owner_display_name": "Multisite Test User", 2023-05-31T20:54:53.402 INFO:teuthology.orchestra.run.smithi099.stdout: "content_type": "application/octet-stream", 2023-05-31T20:54:53.403 INFO:teuthology.orchestra.run.smithi099.stdout: "accounted_size": 3, 2023-05-31T20:54:53.403 INFO:teuthology.orchestra.run.smithi099.stdout: "user_data": "", 2023-05-31T20:54:53.403 INFO:teuthology.orchestra.run.smithi099.stdout: "appendable": false 2023-05-31T20:54:53.403 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:54:53.403 INFO:teuthology.orchestra.run.smithi099.stdout: "tag": "_UFUtv_zAuQ9UAzvoCDMntTqN7mGwQWP", 2023-05-31T20:54:53.403 INFO:teuthology.orchestra.run.smithi099.stdout: "flags": 0, 2023-05-31T20:54:53.403 INFO:teuthology.orchestra.run.smithi099.stdout: "pending_map": [], 2023-05-31T20:54:53.404 INFO:teuthology.orchestra.run.smithi099.stdout: "versioned_epoch": 0 2023-05-31T20:54:53.404 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:54:53.404 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T20:54:53.404 INFO:teuthology.orchestra.run.smithi099.stdout: "name": "obj116", 2023-05-31T20:54:53.404 INFO:teuthology.orchestra.run.smithi099.stdout: "instance": "", 2023-05-31T20:54:53.404 INFO:teuthology.orchestra.run.smithi099.stdout: "ver": { 2023-05-31T20:54:53.405 INFO:teuthology.orchestra.run.smithi099.stdout: "pool": 13, 2023-05-31T20:54:53.405 INFO:teuthology.orchestra.run.smithi099.stdout: "epoch": 14 2023-05-31T20:54:53.405 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:54:53.405 INFO:teuthology.orchestra.run.smithi099.stdout: "locator": "", 2023-05-31T20:54:53.405 INFO:teuthology.orchestra.run.smithi099.stdout: "exists": true, 2023-05-31T20:54:53.405 INFO:teuthology.orchestra.run.smithi099.stdout: "meta": { 2023-05-31T20:54:53.405 INFO:teuthology.orchestra.run.smithi099.stdout: "category": 1, 2023-05-31T20:54:53.406 INFO:teuthology.orchestra.run.smithi099.stdout: "size": 16, 2023-05-31T20:54:53.406 INFO:teuthology.orchestra.run.smithi099.stdout: "mtime": "2023-05-31T20:48:13.052647Z", 2023-05-31T20:54:53.406 INFO:teuthology.orchestra.run.smithi099.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-05-31T20:54:53.406 INFO:teuthology.orchestra.run.smithi099.stdout: "storage_class": "", 2023-05-31T20:54:53.406 INFO:teuthology.orchestra.run.smithi099.stdout: "owner": "rgw-multisite-test-user", 2023-05-31T20:54:53.406 INFO:teuthology.orchestra.run.smithi099.stdout: "owner_display_name": "Multisite Test User", 2023-05-31T20:54:53.406 INFO:teuthology.orchestra.run.smithi099.stdout: "content_type": "application/octet-stream", 2023-05-31T20:54:53.407 INFO:teuthology.orchestra.run.smithi099.stdout: "accounted_size": 3, 2023-05-31T20:54:53.407 INFO:teuthology.orchestra.run.smithi099.stdout: "user_data": "", 2023-05-31T20:54:53.407 INFO:teuthology.orchestra.run.smithi099.stdout: "appendable": false 2023-05-31T20:54:53.407 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:54:53.407 INFO:teuthology.orchestra.run.smithi099.stdout: "tag": "_Q3-ZT9MwF0yK-mNoZ5RrsVdFou9ekFy", 2023-05-31T20:54:53.407 INFO:teuthology.orchestra.run.smithi099.stdout: "flags": 0, 2023-05-31T20:54:53.407 INFO:teuthology.orchestra.run.smithi099.stdout: "pending_map": [], 2023-05-31T20:54:53.408 INFO:teuthology.orchestra.run.smithi099.stdout: "versioned_epoch": 0 2023-05-31T20:54:53.408 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:54:53.408 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T20:54:53.408 INFO:teuthology.orchestra.run.smithi099.stdout: "name": "obj124", 2023-05-31T20:54:53.408 INFO:teuthology.orchestra.run.smithi099.stdout: "instance": "", 2023-05-31T20:54:53.408 INFO:teuthology.orchestra.run.smithi099.stdout: "ver": { 2023-05-31T20:54:53.408 INFO:teuthology.orchestra.run.smithi099.stdout: "pool": 13, 2023-05-31T20:54:53.409 INFO:teuthology.orchestra.run.smithi099.stdout: "epoch": 11 2023-05-31T20:54:53.409 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:54:53.409 INFO:teuthology.orchestra.run.smithi099.stdout: "locator": "", 2023-05-31T20:54:53.409 INFO:teuthology.orchestra.run.smithi099.stdout: "exists": true, 2023-05-31T20:54:53.409 INFO:teuthology.orchestra.run.smithi099.stdout: "meta": { 2023-05-31T20:54:53.409 INFO:teuthology.orchestra.run.smithi099.stdout: "category": 1, 2023-05-31T20:54:53.409 INFO:teuthology.orchestra.run.smithi099.stdout: "size": 5, 2023-05-31T20:54:53.410 INFO:teuthology.orchestra.run.smithi099.stdout: "mtime": "2023-05-31T20:50:33.587761Z", 2023-05-31T20:54:53.410 INFO:teuthology.orchestra.run.smithi099.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-05-31T20:54:53.410 INFO:teuthology.orchestra.run.smithi099.stdout: "storage_class": "", 2023-05-31T20:54:53.410 INFO:teuthology.orchestra.run.smithi099.stdout: "owner": "rgw-multisite-test-user", 2023-05-31T20:54:53.410 INFO:teuthology.orchestra.run.smithi099.stdout: "owner_display_name": "Multisite Test User", 2023-05-31T20:54:53.410 INFO:teuthology.orchestra.run.smithi099.stdout: "content_type": "application/octet-stream", 2023-05-31T20:54:53.410 INFO:teuthology.orchestra.run.smithi099.stdout: "accounted_size": 3, 2023-05-31T20:54:53.411 INFO:teuthology.orchestra.run.smithi099.stdout: "user_data": "", 2023-05-31T20:54:53.411 INFO:teuthology.orchestra.run.smithi099.stdout: "appendable": false 2023-05-31T20:54:53.411 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:54:53.411 INFO:teuthology.orchestra.run.smithi099.stdout: "tag": "_CxgiILPzCYwiIVZ13FU28QOnbkLNqZ_", 2023-05-31T20:54:53.411 INFO:teuthology.orchestra.run.smithi099.stdout: "flags": 0, 2023-05-31T20:54:53.411 INFO:teuthology.orchestra.run.smithi099.stdout: "pending_map": [], 2023-05-31T20:54:53.411 INFO:teuthology.orchestra.run.smithi099.stdout: "versioned_epoch": 0 2023-05-31T20:54:53.412 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:54:53.412 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T20:54:53.412 INFO:teuthology.orchestra.run.smithi099.stdout: "name": "obj145", 2023-05-31T20:54:53.412 INFO:teuthology.orchestra.run.smithi099.stdout: "instance": "", 2023-05-31T20:54:53.412 INFO:teuthology.orchestra.run.smithi099.stdout: "ver": { 2023-05-31T20:54:53.412 INFO:teuthology.orchestra.run.smithi099.stdout: "pool": 13, 2023-05-31T20:54:53.413 INFO:teuthology.orchestra.run.smithi099.stdout: "epoch": 15 2023-05-31T20:54:53.413 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:54:53.413 INFO:teuthology.orchestra.run.smithi099.stdout: "locator": "", 2023-05-31T20:54:53.413 INFO:teuthology.orchestra.run.smithi099.stdout: "exists": true, 2023-05-31T20:54:53.413 INFO:teuthology.orchestra.run.smithi099.stdout: "meta": { 2023-05-31T20:54:53.413 INFO:teuthology.orchestra.run.smithi099.stdout: "category": 1, 2023-05-31T20:54:53.413 INFO:teuthology.orchestra.run.smithi099.stdout: "size": 16, 2023-05-31T20:54:53.414 INFO:teuthology.orchestra.run.smithi099.stdout: "mtime": "2023-05-31T20:48:13.107892Z", 2023-05-31T20:54:53.414 INFO:teuthology.orchestra.run.smithi099.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-05-31T20:54:53.414 INFO:teuthology.orchestra.run.smithi099.stdout: "storage_class": "", 2023-05-31T20:54:53.414 INFO:teuthology.orchestra.run.smithi099.stdout: "owner": "rgw-multisite-test-user", 2023-05-31T20:54:53.414 INFO:teuthology.orchestra.run.smithi099.stdout: "owner_display_name": "Multisite Test User", 2023-05-31T20:54:53.414 INFO:teuthology.orchestra.run.smithi099.stdout: "content_type": "application/octet-stream", 2023-05-31T20:54:53.414 INFO:teuthology.orchestra.run.smithi099.stdout: "accounted_size": 3, 2023-05-31T20:54:53.415 INFO:teuthology.orchestra.run.smithi099.stdout: "user_data": "", 2023-05-31T20:54:53.415 INFO:teuthology.orchestra.run.smithi099.stdout: "appendable": false 2023-05-31T20:54:53.415 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:54:53.415 INFO:teuthology.orchestra.run.smithi099.stdout: "tag": "_1s3BESpY-vcE3lTm4WdPkanTkuV9xBf", 2023-05-31T20:54:53.415 INFO:teuthology.orchestra.run.smithi099.stdout: "flags": 0, 2023-05-31T20:54:53.415 INFO:teuthology.orchestra.run.smithi099.stdout: "pending_map": [], 2023-05-31T20:54:53.415 INFO:teuthology.orchestra.run.smithi099.stdout: "versioned_epoch": 0 2023-05-31T20:54:53.416 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:54:53.416 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T20:54:53.416 INFO:teuthology.orchestra.run.smithi099.stdout: "name": "obj148", 2023-05-31T20:54:53.416 INFO:teuthology.orchestra.run.smithi099.stdout: "instance": "", 2023-05-31T20:54:53.416 INFO:teuthology.orchestra.run.smithi099.stdout: "ver": { 2023-05-31T20:54:53.416 INFO:teuthology.orchestra.run.smithi099.stdout: "pool": 13, 2023-05-31T20:54:53.416 INFO:teuthology.orchestra.run.smithi099.stdout: "epoch": 7 2023-05-31T20:54:53.417 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:54:53.417 INFO:teuthology.orchestra.run.smithi099.stdout: "locator": "", 2023-05-31T20:54:53.417 INFO:teuthology.orchestra.run.smithi099.stdout: "exists": true, 2023-05-31T20:54:53.417 INFO:teuthology.orchestra.run.smithi099.stdout: "meta": { 2023-05-31T20:54:53.417 INFO:teuthology.orchestra.run.smithi099.stdout: "category": 1, 2023-05-31T20:54:53.417 INFO:teuthology.orchestra.run.smithi099.stdout: "size": 6, 2023-05-31T20:54:53.417 INFO:teuthology.orchestra.run.smithi099.stdout: "mtime": "2023-05-31T20:52:15.015083Z", 2023-05-31T20:54:53.418 INFO:teuthology.orchestra.run.smithi099.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-05-31T20:54:53.418 INFO:teuthology.orchestra.run.smithi099.stdout: "storage_class": "", 2023-05-31T20:54:53.418 INFO:teuthology.orchestra.run.smithi099.stdout: "owner": "rgw-multisite-test-user", 2023-05-31T20:54:53.418 INFO:teuthology.orchestra.run.smithi099.stdout: "owner_display_name": "Multisite Test User", 2023-05-31T20:54:53.418 INFO:teuthology.orchestra.run.smithi099.stdout: "content_type": "application/octet-stream", 2023-05-31T20:54:53.418 INFO:teuthology.orchestra.run.smithi099.stdout: "accounted_size": 3, 2023-05-31T20:54:53.418 INFO:teuthology.orchestra.run.smithi099.stdout: "user_data": "", 2023-05-31T20:54:53.419 INFO:teuthology.orchestra.run.smithi099.stdout: "appendable": false 2023-05-31T20:54:53.419 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:54:53.419 INFO:teuthology.orchestra.run.smithi099.stdout: "tag": "_IT_uXkrerJIwfegYrG_DC7oZm_Xuul9", 2023-05-31T20:54:53.419 INFO:teuthology.orchestra.run.smithi099.stdout: "flags": 0, 2023-05-31T20:54:53.419 INFO:teuthology.orchestra.run.smithi099.stdout: "pending_map": [], 2023-05-31T20:54:53.419 INFO:teuthology.orchestra.run.smithi099.stdout: "versioned_epoch": 0 2023-05-31T20:54:53.419 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:54:53.420 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T20:54:53.420 INFO:teuthology.orchestra.run.smithi099.stdout: "name": "obj155", 2023-05-31T20:54:53.420 INFO:teuthology.orchestra.run.smithi099.stdout: "instance": "", 2023-05-31T20:54:53.420 INFO:teuthology.orchestra.run.smithi099.stdout: "ver": { 2023-05-31T20:54:53.420 INFO:teuthology.orchestra.run.smithi099.stdout: "pool": 13, 2023-05-31T20:54:53.420 INFO:teuthology.orchestra.run.smithi099.stdout: "epoch": 15 2023-05-31T20:54:53.421 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:54:53.421 INFO:teuthology.orchestra.run.smithi099.stdout: "locator": "", 2023-05-31T20:54:53.421 INFO:teuthology.orchestra.run.smithi099.stdout: "exists": true, 2023-05-31T20:54:53.421 INFO:teuthology.orchestra.run.smithi099.stdout: "meta": { 2023-05-31T20:54:53.421 INFO:teuthology.orchestra.run.smithi099.stdout: "category": 1, 2023-05-31T20:54:53.421 INFO:teuthology.orchestra.run.smithi099.stdout: "size": 6, 2023-05-31T20:54:53.421 INFO:teuthology.orchestra.run.smithi099.stdout: "mtime": "2023-05-31T20:50:33.605763Z", 2023-05-31T20:54:53.421 INFO:teuthology.orchestra.run.smithi099.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-05-31T20:54:53.422 INFO:teuthology.orchestra.run.smithi099.stdout: "storage_class": "", 2023-05-31T20:54:53.422 INFO:teuthology.orchestra.run.smithi099.stdout: "owner": "rgw-multisite-test-user", 2023-05-31T20:54:53.422 INFO:teuthology.orchestra.run.smithi099.stdout: "owner_display_name": "Multisite Test User", 2023-05-31T20:54:53.422 INFO:teuthology.orchestra.run.smithi099.stdout: "content_type": "application/octet-stream", 2023-05-31T20:54:53.422 INFO:teuthology.orchestra.run.smithi099.stdout: "accounted_size": 3, 2023-05-31T20:54:53.422 INFO:teuthology.orchestra.run.smithi099.stdout: "user_data": "", 2023-05-31T20:54:53.422 INFO:teuthology.orchestra.run.smithi099.stdout: "appendable": false 2023-05-31T20:54:53.423 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:54:53.423 INFO:teuthology.orchestra.run.smithi099.stdout: "tag": "__4aQ4qqQ_boRZV50pz2KuGPKsffvXRV", 2023-05-31T20:54:53.423 INFO:teuthology.orchestra.run.smithi099.stdout: "flags": 0, 2023-05-31T20:54:53.423 INFO:teuthology.orchestra.run.smithi099.stdout: "pending_map": [], 2023-05-31T20:54:53.423 INFO:teuthology.orchestra.run.smithi099.stdout: "versioned_epoch": 0 2023-05-31T20:54:53.423 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:54:53.423 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T20:54:53.424 INFO:teuthology.orchestra.run.smithi099.stdout: "name": "obj17", 2023-05-31T20:54:53.424 INFO:teuthology.orchestra.run.smithi099.stdout: "instance": "", 2023-05-31T20:54:53.424 INFO:teuthology.orchestra.run.smithi099.stdout: "ver": { 2023-05-31T20:54:53.424 INFO:teuthology.orchestra.run.smithi099.stdout: "pool": 13, 2023-05-31T20:54:53.424 INFO:teuthology.orchestra.run.smithi099.stdout: "epoch": 7 2023-05-31T20:54:53.424 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:54:53.424 INFO:teuthology.orchestra.run.smithi099.stdout: "locator": "", 2023-05-31T20:54:53.425 INFO:teuthology.orchestra.run.smithi099.stdout: "exists": true, 2023-05-31T20:54:53.425 INFO:teuthology.orchestra.run.smithi099.stdout: "meta": { 2023-05-31T20:54:53.425 INFO:teuthology.orchestra.run.smithi099.stdout: "category": 1, 2023-05-31T20:54:53.425 INFO:teuthology.orchestra.run.smithi099.stdout: "size": 5, 2023-05-31T20:54:53.425 INFO:teuthology.orchestra.run.smithi099.stdout: "mtime": "2023-05-31T20:43:07.977285Z", 2023-05-31T20:54:53.425 INFO:teuthology.orchestra.run.smithi099.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-05-31T20:54:53.425 INFO:teuthology.orchestra.run.smithi099.stdout: "storage_class": "", 2023-05-31T20:54:53.426 INFO:teuthology.orchestra.run.smithi099.stdout: "owner": "rgw-multisite-test-user", 2023-05-31T20:54:53.426 INFO:teuthology.orchestra.run.smithi099.stdout: "owner_display_name": "Multisite Test User", 2023-05-31T20:54:53.426 INFO:teuthology.orchestra.run.smithi099.stdout: "content_type": "application/octet-stream", 2023-05-31T20:54:53.426 INFO:teuthology.orchestra.run.smithi099.stdout: "accounted_size": 3, 2023-05-31T20:54:53.426 INFO:teuthology.orchestra.run.smithi099.stdout: "user_data": "", 2023-05-31T20:54:53.426 INFO:teuthology.orchestra.run.smithi099.stdout: "appendable": false 2023-05-31T20:54:53.426 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:54:53.427 INFO:teuthology.orchestra.run.smithi099.stdout: "tag": "_EzTJtr2OVlXCw506gL02JdPq3cvG80m", 2023-05-31T20:54:53.427 INFO:teuthology.orchestra.run.smithi099.stdout: "flags": 0, 2023-05-31T20:54:53.427 INFO:teuthology.orchestra.run.smithi099.stdout: "pending_map": [], 2023-05-31T20:54:53.427 INFO:teuthology.orchestra.run.smithi099.stdout: "versioned_epoch": 0 2023-05-31T20:54:53.427 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:54:53.427 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T20:54:53.427 INFO:teuthology.orchestra.run.smithi099.stdout: "name": "obj185", 2023-05-31T20:54:53.428 INFO:teuthology.orchestra.run.smithi099.stdout: "instance": "", 2023-05-31T20:54:53.428 INFO:teuthology.orchestra.run.smithi099.stdout: "ver": { 2023-05-31T20:54:53.428 INFO:teuthology.orchestra.run.smithi099.stdout: "pool": 13, 2023-05-31T20:54:53.428 INFO:teuthology.orchestra.run.smithi099.stdout: "epoch": 11 2023-05-31T20:54:53.428 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:54:53.428 INFO:teuthology.orchestra.run.smithi099.stdout: "locator": "", 2023-05-31T20:54:53.429 INFO:teuthology.orchestra.run.smithi099.stdout: "exists": true, 2023-05-31T20:54:53.429 INFO:teuthology.orchestra.run.smithi099.stdout: "meta": { 2023-05-31T20:54:53.429 INFO:teuthology.orchestra.run.smithi099.stdout: "category": 1, 2023-05-31T20:54:53.429 INFO:teuthology.orchestra.run.smithi099.stdout: "size": 16, 2023-05-31T20:54:53.429 INFO:teuthology.orchestra.run.smithi099.stdout: "mtime": "2023-05-31T20:52:15.072437Z", 2023-05-31T20:54:53.429 INFO:teuthology.orchestra.run.smithi099.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-05-31T20:54:53.429 INFO:teuthology.orchestra.run.smithi099.stdout: "storage_class": "", 2023-05-31T20:54:53.430 INFO:teuthology.orchestra.run.smithi099.stdout: "owner": "rgw-multisite-test-user", 2023-05-31T20:54:53.430 INFO:teuthology.orchestra.run.smithi099.stdout: "owner_display_name": "Multisite Test User", 2023-05-31T20:54:53.430 INFO:teuthology.orchestra.run.smithi099.stdout: "content_type": "application/octet-stream", 2023-05-31T20:54:53.430 INFO:teuthology.orchestra.run.smithi099.stdout: "accounted_size": 3, 2023-05-31T20:54:53.430 INFO:teuthology.orchestra.run.smithi099.stdout: "user_data": "", 2023-05-31T20:54:53.430 INFO:teuthology.orchestra.run.smithi099.stdout: "appendable": false 2023-05-31T20:54:53.430 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:54:53.431 INFO:teuthology.orchestra.run.smithi099.stdout: "tag": "_SIdihEEGFGp7SK9TB58HoDnT5q66VXe", 2023-05-31T20:54:53.431 INFO:teuthology.orchestra.run.smithi099.stdout: "flags": 0, 2023-05-31T20:54:53.431 INFO:teuthology.orchestra.run.smithi099.stdout: "pending_map": [], 2023-05-31T20:54:53.431 INFO:teuthology.orchestra.run.smithi099.stdout: "versioned_epoch": 0 2023-05-31T20:54:53.431 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:54:53.431 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T20:54:53.431 INFO:teuthology.orchestra.run.smithi099.stdout: "name": "obj19", 2023-05-31T20:54:53.432 INFO:teuthology.orchestra.run.smithi099.stdout: "instance": "", 2023-05-31T20:54:53.432 INFO:teuthology.orchestra.run.smithi099.stdout: "ver": { 2023-05-31T20:54:53.432 INFO:teuthology.orchestra.run.smithi099.stdout: "pool": 13, 2023-05-31T20:54:53.432 INFO:teuthology.orchestra.run.smithi099.stdout: "epoch": 11 2023-05-31T20:54:53.432 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:54:53.432 INFO:teuthology.orchestra.run.smithi099.stdout: "locator": "", 2023-05-31T20:54:53.432 INFO:teuthology.orchestra.run.smithi099.stdout: "exists": true, 2023-05-31T20:54:53.433 INFO:teuthology.orchestra.run.smithi099.stdout: "meta": { 2023-05-31T20:54:53.433 INFO:teuthology.orchestra.run.smithi099.stdout: "category": 1, 2023-05-31T20:54:53.433 INFO:teuthology.orchestra.run.smithi099.stdout: "size": 16, 2023-05-31T20:54:53.433 INFO:teuthology.orchestra.run.smithi099.stdout: "mtime": "2023-05-31T20:44:50.557118Z", 2023-05-31T20:54:53.433 INFO:teuthology.orchestra.run.smithi099.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-05-31T20:54:53.433 INFO:teuthology.orchestra.run.smithi099.stdout: "storage_class": "", 2023-05-31T20:54:53.434 INFO:teuthology.orchestra.run.smithi099.stdout: "owner": "rgw-multisite-test-user", 2023-05-31T20:54:53.434 INFO:teuthology.orchestra.run.smithi099.stdout: "owner_display_name": "Multisite Test User", 2023-05-31T20:54:53.434 INFO:teuthology.orchestra.run.smithi099.stdout: "content_type": "application/octet-stream", 2023-05-31T20:54:53.434 INFO:teuthology.orchestra.run.smithi099.stdout: "accounted_size": 3, 2023-05-31T20:54:53.434 INFO:teuthology.orchestra.run.smithi099.stdout: "user_data": "", 2023-05-31T20:54:53.434 INFO:teuthology.orchestra.run.smithi099.stdout: "appendable": false 2023-05-31T20:54:53.434 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:54:53.435 INFO:teuthology.orchestra.run.smithi099.stdout: "tag": "_-CWubXhotbaW-Vgk8uqjeseSmHLmbG9", 2023-05-31T20:54:53.435 INFO:teuthology.orchestra.run.smithi099.stdout: "flags": 0, 2023-05-31T20:54:53.435 INFO:teuthology.orchestra.run.smithi099.stdout: "pending_map": [], 2023-05-31T20:54:53.435 INFO:teuthology.orchestra.run.smithi099.stdout: "versioned_epoch": 0 2023-05-31T20:54:53.435 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:54:53.435 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T20:54:53.435 INFO:teuthology.orchestra.run.smithi099.stdout: "name": "obj22", 2023-05-31T20:54:53.436 INFO:teuthology.orchestra.run.smithi099.stdout: "instance": "", 2023-05-31T20:54:53.436 INFO:teuthology.orchestra.run.smithi099.stdout: "ver": { 2023-05-31T20:54:53.436 INFO:teuthology.orchestra.run.smithi099.stdout: "pool": 13, 2023-05-31T20:54:53.436 INFO:teuthology.orchestra.run.smithi099.stdout: "epoch": 12 2023-05-31T20:54:53.436 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:54:53.436 INFO:teuthology.orchestra.run.smithi099.stdout: "locator": "", 2023-05-31T20:54:53.436 INFO:teuthology.orchestra.run.smithi099.stdout: "exists": true, 2023-05-31T20:54:53.437 INFO:teuthology.orchestra.run.smithi099.stdout: "meta": { 2023-05-31T20:54:53.437 INFO:teuthology.orchestra.run.smithi099.stdout: "category": 1, 2023-05-31T20:54:53.437 INFO:teuthology.orchestra.run.smithi099.stdout: "size": 6, 2023-05-31T20:54:53.437 INFO:teuthology.orchestra.run.smithi099.stdout: "mtime": "2023-05-31T20:39:26.473296Z", 2023-05-31T20:54:53.437 INFO:teuthology.orchestra.run.smithi099.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-05-31T20:54:53.437 INFO:teuthology.orchestra.run.smithi099.stdout: "storage_class": "", 2023-05-31T20:54:53.437 INFO:teuthology.orchestra.run.smithi099.stdout: "owner": "rgw-multisite-test-user", 2023-05-31T20:54:53.438 INFO:teuthology.orchestra.run.smithi099.stdout: "owner_display_name": "Multisite Test User", 2023-05-31T20:54:53.438 INFO:teuthology.orchestra.run.smithi099.stdout: "content_type": "application/octet-stream", 2023-05-31T20:54:53.438 INFO:teuthology.orchestra.run.smithi099.stdout: "accounted_size": 3, 2023-05-31T20:54:53.438 INFO:teuthology.orchestra.run.smithi099.stdout: "user_data": "", 2023-05-31T20:54:53.438 INFO:teuthology.orchestra.run.smithi099.stdout: "appendable": false 2023-05-31T20:54:53.438 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:54:53.438 INFO:teuthology.orchestra.run.smithi099.stdout: "tag": "_aEM-0ra24u0QswRMGDC3U8aNYE0VQBB", 2023-05-31T20:54:53.439 INFO:teuthology.orchestra.run.smithi099.stdout: "flags": 0, 2023-05-31T20:54:53.439 INFO:teuthology.orchestra.run.smithi099.stdout: "pending_map": [], 2023-05-31T20:54:53.439 INFO:teuthology.orchestra.run.smithi099.stdout: "versioned_epoch": 0 2023-05-31T20:54:53.439 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:54:53.439 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T20:54:53.439 INFO:teuthology.orchestra.run.smithi099.stdout: "name": "obj23", 2023-05-31T20:54:53.439 INFO:teuthology.orchestra.run.smithi099.stdout: "instance": "", 2023-05-31T20:54:53.440 INFO:teuthology.orchestra.run.smithi099.stdout: "ver": { 2023-05-31T20:54:53.440 INFO:teuthology.orchestra.run.smithi099.stdout: "pool": 13, 2023-05-31T20:54:53.440 INFO:teuthology.orchestra.run.smithi099.stdout: "epoch": 7 2023-05-31T20:54:53.440 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:54:53.440 INFO:teuthology.orchestra.run.smithi099.stdout: "locator": "", 2023-05-31T20:54:53.440 INFO:teuthology.orchestra.run.smithi099.stdout: "exists": true, 2023-05-31T20:54:53.441 INFO:teuthology.orchestra.run.smithi099.stdout: "meta": { 2023-05-31T20:54:53.441 INFO:teuthology.orchestra.run.smithi099.stdout: "category": 1, 2023-05-31T20:54:53.441 INFO:teuthology.orchestra.run.smithi099.stdout: "size": 16, 2023-05-31T20:54:53.441 INFO:teuthology.orchestra.run.smithi099.stdout: "mtime": "2023-05-31T20:46:33.813577Z", 2023-05-31T20:54:53.441 INFO:teuthology.orchestra.run.smithi099.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-05-31T20:54:53.441 INFO:teuthology.orchestra.run.smithi099.stdout: "storage_class": "", 2023-05-31T20:54:53.441 INFO:teuthology.orchestra.run.smithi099.stdout: "owner": "rgw-multisite-test-user", 2023-05-31T20:54:53.442 INFO:teuthology.orchestra.run.smithi099.stdout: "owner_display_name": "Multisite Test User", 2023-05-31T20:54:53.442 INFO:teuthology.orchestra.run.smithi099.stdout: "content_type": "application/octet-stream", 2023-05-31T20:54:53.442 INFO:teuthology.orchestra.run.smithi099.stdout: "accounted_size": 3, 2023-05-31T20:54:53.442 INFO:teuthology.orchestra.run.smithi099.stdout: "user_data": "", 2023-05-31T20:54:53.442 INFO:teuthology.orchestra.run.smithi099.stdout: "appendable": false 2023-05-31T20:54:53.442 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:54:53.442 INFO:teuthology.orchestra.run.smithi099.stdout: "tag": "_SDTENy7Z6rN7oYjAh_uv660l-_T-HJ_", 2023-05-31T20:54:53.443 INFO:teuthology.orchestra.run.smithi099.stdout: "flags": 0, 2023-05-31T20:54:53.443 INFO:teuthology.orchestra.run.smithi099.stdout: "pending_map": [], 2023-05-31T20:54:53.443 INFO:teuthology.orchestra.run.smithi099.stdout: "versioned_epoch": 0 2023-05-31T20:54:53.443 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:54:53.443 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T20:54:53.443 INFO:teuthology.orchestra.run.smithi099.stdout: "name": "obj29", 2023-05-31T20:54:53.443 INFO:teuthology.orchestra.run.smithi099.stdout: "instance": "", 2023-05-31T20:54:53.444 INFO:teuthology.orchestra.run.smithi099.stdout: "ver": { 2023-05-31T20:54:53.444 INFO:teuthology.orchestra.run.smithi099.stdout: "pool": 13, 2023-05-31T20:54:53.444 INFO:teuthology.orchestra.run.smithi099.stdout: "epoch": 4 2023-05-31T20:54:53.444 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:54:53.444 INFO:teuthology.orchestra.run.smithi099.stdout: "locator": "", 2023-05-31T20:54:53.444 INFO:teuthology.orchestra.run.smithi099.stdout: "exists": true, 2023-05-31T20:54:53.444 INFO:teuthology.orchestra.run.smithi099.stdout: "meta": { 2023-05-31T20:54:53.445 INFO:teuthology.orchestra.run.smithi099.stdout: "category": 1, 2023-05-31T20:54:53.445 INFO:teuthology.orchestra.run.smithi099.stdout: "size": 5, 2023-05-31T20:54:53.445 INFO:teuthology.orchestra.run.smithi099.stdout: "mtime": "2023-05-31T20:48:12.956068Z", 2023-05-31T20:54:53.445 INFO:teuthology.orchestra.run.smithi099.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-05-31T20:54:53.445 INFO:teuthology.orchestra.run.smithi099.stdout: "storage_class": "", 2023-05-31T20:54:53.445 INFO:teuthology.orchestra.run.smithi099.stdout: "owner": "rgw-multisite-test-user", 2023-05-31T20:54:53.445 INFO:teuthology.orchestra.run.smithi099.stdout: "owner_display_name": "Multisite Test User", 2023-05-31T20:54:53.446 INFO:teuthology.orchestra.run.smithi099.stdout: "content_type": "application/octet-stream", 2023-05-31T20:54:53.446 INFO:teuthology.orchestra.run.smithi099.stdout: "accounted_size": 3, 2023-05-31T20:54:53.446 INFO:teuthology.orchestra.run.smithi099.stdout: "user_data": "", 2023-05-31T20:54:53.446 INFO:teuthology.orchestra.run.smithi099.stdout: "appendable": false 2023-05-31T20:54:53.446 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:54:53.446 INFO:teuthology.orchestra.run.smithi099.stdout: "tag": "_6pBNLJGGdDGBhl-iDeICUY5j4Ef7CER", 2023-05-31T20:54:53.446 INFO:teuthology.orchestra.run.smithi099.stdout: "flags": 0, 2023-05-31T20:54:53.447 INFO:teuthology.orchestra.run.smithi099.stdout: "pending_map": [], 2023-05-31T20:54:53.447 INFO:teuthology.orchestra.run.smithi099.stdout: "versioned_epoch": 0 2023-05-31T20:54:53.447 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:54:53.447 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T20:54:53.447 INFO:teuthology.orchestra.run.smithi099.stdout: "name": "obj31", 2023-05-31T20:54:53.447 INFO:teuthology.orchestra.run.smithi099.stdout: "instance": "", 2023-05-31T20:54:53.447 INFO:teuthology.orchestra.run.smithi099.stdout: "ver": { 2023-05-31T20:54:53.448 INFO:teuthology.orchestra.run.smithi099.stdout: "pool": 13, 2023-05-31T20:54:53.448 INFO:teuthology.orchestra.run.smithi099.stdout: "epoch": 6 2023-05-31T20:54:53.448 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:54:53.448 INFO:teuthology.orchestra.run.smithi099.stdout: "locator": "", 2023-05-31T20:54:53.448 INFO:teuthology.orchestra.run.smithi099.stdout: "exists": true, 2023-05-31T20:54:53.448 INFO:teuthology.orchestra.run.smithi099.stdout: "meta": { 2023-05-31T20:54:53.449 INFO:teuthology.orchestra.run.smithi099.stdout: "category": 1, 2023-05-31T20:54:53.449 INFO:teuthology.orchestra.run.smithi099.stdout: "size": 16, 2023-05-31T20:54:53.449 INFO:teuthology.orchestra.run.smithi099.stdout: "mtime": "2023-05-31T20:50:33.460419Z", 2023-05-31T20:54:53.449 INFO:teuthology.orchestra.run.smithi099.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-05-31T20:54:53.449 INFO:teuthology.orchestra.run.smithi099.stdout: "storage_class": "", 2023-05-31T20:54:53.449 INFO:teuthology.orchestra.run.smithi099.stdout: "owner": "rgw-multisite-test-user", 2023-05-31T20:54:53.449 INFO:teuthology.orchestra.run.smithi099.stdout: "owner_display_name": "Multisite Test User", 2023-05-31T20:54:53.450 INFO:teuthology.orchestra.run.smithi099.stdout: "content_type": "application/octet-stream", 2023-05-31T20:54:53.450 INFO:teuthology.orchestra.run.smithi099.stdout: "accounted_size": 3, 2023-05-31T20:54:53.450 INFO:teuthology.orchestra.run.smithi099.stdout: "user_data": "", 2023-05-31T20:54:53.450 INFO:teuthology.orchestra.run.smithi099.stdout: "appendable": false 2023-05-31T20:54:53.450 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:54:53.450 INFO:teuthology.orchestra.run.smithi099.stdout: "tag": "_XAQWyCkag-NqncA0SMYYl2nO1mVJBkA", 2023-05-31T20:54:53.450 INFO:teuthology.orchestra.run.smithi099.stdout: "flags": 0, 2023-05-31T20:54:53.451 INFO:teuthology.orchestra.run.smithi099.stdout: "pending_map": [], 2023-05-31T20:54:53.451 INFO:teuthology.orchestra.run.smithi099.stdout: "versioned_epoch": 0 2023-05-31T20:54:53.451 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:54:53.451 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T20:54:53.451 INFO:teuthology.orchestra.run.smithi099.stdout: "name": "obj33", 2023-05-31T20:54:53.451 INFO:teuthology.orchestra.run.smithi099.stdout: "instance": "", 2023-05-31T20:54:53.451 INFO:teuthology.orchestra.run.smithi099.stdout: "ver": { 2023-05-31T20:54:53.452 INFO:teuthology.orchestra.run.smithi099.stdout: "pool": 13, 2023-05-31T20:54:53.452 INFO:teuthology.orchestra.run.smithi099.stdout: "epoch": 20 2023-05-31T20:54:53.452 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:54:53.452 INFO:teuthology.orchestra.run.smithi099.stdout: "locator": "", 2023-05-31T20:54:53.452 INFO:teuthology.orchestra.run.smithi099.stdout: "exists": true, 2023-05-31T20:54:53.452 INFO:teuthology.orchestra.run.smithi099.stdout: "meta": { 2023-05-31T20:54:53.452 INFO:teuthology.orchestra.run.smithi099.stdout: "category": 1, 2023-05-31T20:54:53.453 INFO:teuthology.orchestra.run.smithi099.stdout: "size": 16, 2023-05-31T20:54:53.453 INFO:teuthology.orchestra.run.smithi099.stdout: "mtime": "2023-05-31T20:39:26.534549Z", 2023-05-31T20:54:53.453 INFO:teuthology.orchestra.run.smithi099.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-05-31T20:54:53.453 INFO:teuthology.orchestra.run.smithi099.stdout: "storage_class": "", 2023-05-31T20:54:53.453 INFO:teuthology.orchestra.run.smithi099.stdout: "owner": "rgw-multisite-test-user", 2023-05-31T20:54:53.453 INFO:teuthology.orchestra.run.smithi099.stdout: "owner_display_name": "Multisite Test User", 2023-05-31T20:54:53.454 INFO:teuthology.orchestra.run.smithi099.stdout: "content_type": "application/octet-stream", 2023-05-31T20:54:53.454 INFO:teuthology.orchestra.run.smithi099.stdout: "accounted_size": 3, 2023-05-31T20:54:53.454 INFO:teuthology.orchestra.run.smithi099.stdout: "user_data": "", 2023-05-31T20:54:53.454 INFO:teuthology.orchestra.run.smithi099.stdout: "appendable": false 2023-05-31T20:54:53.454 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:54:53.454 INFO:teuthology.orchestra.run.smithi099.stdout: "tag": "_VM7vlH7R9QATDtv9Xm6PlgY_Tq7dMMb", 2023-05-31T20:54:53.455 INFO:teuthology.orchestra.run.smithi099.stdout: "flags": 0, 2023-05-31T20:54:53.455 INFO:teuthology.orchestra.run.smithi099.stdout: "pending_map": [], 2023-05-31T20:54:53.455 INFO:teuthology.orchestra.run.smithi099.stdout: "versioned_epoch": 0 2023-05-31T20:54:53.455 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:54:53.455 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T20:54:53.455 INFO:teuthology.orchestra.run.smithi099.stdout: "name": "obj34", 2023-05-31T20:54:53.455 INFO:teuthology.orchestra.run.smithi099.stdout: "instance": "", 2023-05-31T20:54:53.456 INFO:teuthology.orchestra.run.smithi099.stdout: "ver": { 2023-05-31T20:54:53.456 INFO:teuthology.orchestra.run.smithi099.stdout: "pool": 13, 2023-05-31T20:54:53.456 INFO:teuthology.orchestra.run.smithi099.stdout: "epoch": 10 2023-05-31T20:54:53.456 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:54:53.456 INFO:teuthology.orchestra.run.smithi099.stdout: "locator": "", 2023-05-31T20:54:53.456 INFO:teuthology.orchestra.run.smithi099.stdout: "exists": true, 2023-05-31T20:54:53.457 INFO:teuthology.orchestra.run.smithi099.stdout: "meta": { 2023-05-31T20:54:53.457 INFO:teuthology.orchestra.run.smithi099.stdout: "category": 1, 2023-05-31T20:54:53.457 INFO:teuthology.orchestra.run.smithi099.stdout: "size": 3, 2023-05-31T20:54:53.457 INFO:teuthology.orchestra.run.smithi099.stdout: "mtime": "2023-05-31T20:43:07.994147Z", 2023-05-31T20:54:53.457 INFO:teuthology.orchestra.run.smithi099.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-05-31T20:54:53.457 INFO:teuthology.orchestra.run.smithi099.stdout: "storage_class": "", 2023-05-31T20:54:53.457 INFO:teuthology.orchestra.run.smithi099.stdout: "owner": "rgw-multisite-test-user", 2023-05-31T20:54:53.458 INFO:teuthology.orchestra.run.smithi099.stdout: "owner_display_name": "Multisite Test User", 2023-05-31T20:54:53.458 INFO:teuthology.orchestra.run.smithi099.stdout: "content_type": "application/octet-stream", 2023-05-31T20:54:53.458 INFO:teuthology.orchestra.run.smithi099.stdout: "accounted_size": 3, 2023-05-31T20:54:53.458 INFO:teuthology.orchestra.run.smithi099.stdout: "user_data": "", 2023-05-31T20:54:53.458 INFO:teuthology.orchestra.run.smithi099.stdout: "appendable": false 2023-05-31T20:54:53.458 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:54:53.458 INFO:teuthology.orchestra.run.smithi099.stdout: "tag": "_eEHfjimQVba_0oArtdY8kT8jbDWj8XB", 2023-05-31T20:54:53.459 INFO:teuthology.orchestra.run.smithi099.stdout: "flags": 0, 2023-05-31T20:54:53.459 INFO:teuthology.orchestra.run.smithi099.stdout: "pending_map": [], 2023-05-31T20:54:53.459 INFO:teuthology.orchestra.run.smithi099.stdout: "versioned_epoch": 0 2023-05-31T20:54:53.459 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:54:53.459 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T20:54:53.459 INFO:teuthology.orchestra.run.smithi099.stdout: "name": "obj37", 2023-05-31T20:54:53.459 INFO:teuthology.orchestra.run.smithi099.stdout: "instance": "", 2023-05-31T20:54:53.460 INFO:teuthology.orchestra.run.smithi099.stdout: "ver": { 2023-05-31T20:54:53.460 INFO:teuthology.orchestra.run.smithi099.stdout: "pool": 13, 2023-05-31T20:54:53.460 INFO:teuthology.orchestra.run.smithi099.stdout: "epoch": 7 2023-05-31T20:54:53.460 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:54:53.460 INFO:teuthology.orchestra.run.smithi099.stdout: "locator": "", 2023-05-31T20:54:53.460 INFO:teuthology.orchestra.run.smithi099.stdout: "exists": true, 2023-05-31T20:54:53.460 INFO:teuthology.orchestra.run.smithi099.stdout: "meta": { 2023-05-31T20:54:53.461 INFO:teuthology.orchestra.run.smithi099.stdout: "category": 1, 2023-05-31T20:54:53.461 INFO:teuthology.orchestra.run.smithi099.stdout: "size": 6, 2023-05-31T20:54:53.461 INFO:teuthology.orchestra.run.smithi099.stdout: "mtime": "2023-05-31T20:52:14.909599Z", 2023-05-31T20:54:53.461 INFO:teuthology.orchestra.run.smithi099.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-05-31T20:54:53.461 INFO:teuthology.orchestra.run.smithi099.stdout: "storage_class": "", 2023-05-31T20:54:53.461 INFO:teuthology.orchestra.run.smithi099.stdout: "owner": "rgw-multisite-test-user", 2023-05-31T20:54:53.462 INFO:teuthology.orchestra.run.smithi099.stdout: "owner_display_name": "Multisite Test User", 2023-05-31T20:54:53.462 INFO:teuthology.orchestra.run.smithi099.stdout: "content_type": "application/octet-stream", 2023-05-31T20:54:53.462 INFO:teuthology.orchestra.run.smithi099.stdout: "accounted_size": 3, 2023-05-31T20:54:53.462 INFO:teuthology.orchestra.run.smithi099.stdout: "user_data": "", 2023-05-31T20:54:53.462 INFO:teuthology.orchestra.run.smithi099.stdout: "appendable": false 2023-05-31T20:54:53.462 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:54:53.462 INFO:teuthology.orchestra.run.smithi099.stdout: "tag": "_yhcxWH-6MZRLwuw28M2JBWdQGUsLTT8", 2023-05-31T20:54:53.463 INFO:teuthology.orchestra.run.smithi099.stdout: "flags": 0, 2023-05-31T20:54:53.463 INFO:teuthology.orchestra.run.smithi099.stdout: "pending_map": [], 2023-05-31T20:54:53.463 INFO:teuthology.orchestra.run.smithi099.stdout: "versioned_epoch": 0 2023-05-31T20:54:53.463 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:54:53.463 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T20:54:53.463 INFO:teuthology.orchestra.run.smithi099.stdout: "name": "obj38", 2023-05-31T20:54:53.464 INFO:teuthology.orchestra.run.smithi099.stdout: "instance": "", 2023-05-31T20:54:53.464 INFO:teuthology.orchestra.run.smithi099.stdout: "ver": { 2023-05-31T20:54:53.464 INFO:teuthology.orchestra.run.smithi099.stdout: "pool": 13, 2023-05-31T20:54:53.464 INFO:teuthology.orchestra.run.smithi099.stdout: "epoch": 9 2023-05-31T20:54:53.464 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:54:53.464 INFO:teuthology.orchestra.run.smithi099.stdout: "locator": "", 2023-05-31T20:54:53.465 INFO:teuthology.orchestra.run.smithi099.stdout: "exists": true, 2023-05-31T20:54:53.465 INFO:teuthology.orchestra.run.smithi099.stdout: "meta": { 2023-05-31T20:54:53.465 INFO:teuthology.orchestra.run.smithi099.stdout: "category": 1, 2023-05-31T20:54:53.465 INFO:teuthology.orchestra.run.smithi099.stdout: "size": 16, 2023-05-31T20:54:53.465 INFO:teuthology.orchestra.run.smithi099.stdout: "mtime": "2023-05-31T20:44:50.610918Z", 2023-05-31T20:54:53.465 INFO:teuthology.orchestra.run.smithi099.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-05-31T20:54:53.465 INFO:teuthology.orchestra.run.smithi099.stdout: "storage_class": "", 2023-05-31T20:54:53.466 INFO:teuthology.orchestra.run.smithi099.stdout: "owner": "rgw-multisite-test-user", 2023-05-31T20:54:53.466 INFO:teuthology.orchestra.run.smithi099.stdout: "owner_display_name": "Multisite Test User", 2023-05-31T20:54:53.466 INFO:teuthology.orchestra.run.smithi099.stdout: "content_type": "application/octet-stream", 2023-05-31T20:54:53.466 INFO:teuthology.orchestra.run.smithi099.stdout: "accounted_size": 3, 2023-05-31T20:54:53.466 INFO:teuthology.orchestra.run.smithi099.stdout: "user_data": "", 2023-05-31T20:54:53.466 INFO:teuthology.orchestra.run.smithi099.stdout: "appendable": false 2023-05-31T20:54:53.466 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:54:53.467 INFO:teuthology.orchestra.run.smithi099.stdout: "tag": "_XKyvFP7jpeKBNa-2FHrsYTyljuwjnCr", 2023-05-31T20:54:53.467 INFO:teuthology.orchestra.run.smithi099.stdout: "flags": 0, 2023-05-31T20:54:53.467 INFO:teuthology.orchestra.run.smithi099.stdout: "pending_map": [], 2023-05-31T20:54:53.467 INFO:teuthology.orchestra.run.smithi099.stdout: "versioned_epoch": 0 2023-05-31T20:54:53.467 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:54:53.467 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T20:54:53.467 INFO:teuthology.orchestra.run.smithi099.stdout: "name": "obj44", 2023-05-31T20:54:53.468 INFO:teuthology.orchestra.run.smithi099.stdout: "instance": "", 2023-05-31T20:54:53.468 INFO:teuthology.orchestra.run.smithi099.stdout: "ver": { 2023-05-31T20:54:53.468 INFO:teuthology.orchestra.run.smithi099.stdout: "pool": 13, 2023-05-31T20:54:53.468 INFO:teuthology.orchestra.run.smithi099.stdout: "epoch": 10 2023-05-31T20:54:53.468 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:54:53.468 INFO:teuthology.orchestra.run.smithi099.stdout: "locator": "", 2023-05-31T20:54:53.468 INFO:teuthology.orchestra.run.smithi099.stdout: "exists": true, 2023-05-31T20:54:53.469 INFO:teuthology.orchestra.run.smithi099.stdout: "meta": { 2023-05-31T20:54:53.469 INFO:teuthology.orchestra.run.smithi099.stdout: "category": 1, 2023-05-31T20:54:53.469 INFO:teuthology.orchestra.run.smithi099.stdout: "size": 3, 2023-05-31T20:54:53.469 INFO:teuthology.orchestra.run.smithi099.stdout: "mtime": "2023-05-31T20:39:26.565648Z", 2023-05-31T20:54:53.469 INFO:teuthology.orchestra.run.smithi099.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-05-31T20:54:53.469 INFO:teuthology.orchestra.run.smithi099.stdout: "storage_class": "", 2023-05-31T20:54:53.469 INFO:teuthology.orchestra.run.smithi099.stdout: "owner": "rgw-multisite-test-user", 2023-05-31T20:54:53.470 INFO:teuthology.orchestra.run.smithi099.stdout: "owner_display_name": "Multisite Test User", 2023-05-31T20:54:53.470 INFO:teuthology.orchestra.run.smithi099.stdout: "content_type": "application/octet-stream", 2023-05-31T20:54:53.470 INFO:teuthology.orchestra.run.smithi099.stdout: "accounted_size": 3, 2023-05-31T20:54:53.470 INFO:teuthology.orchestra.run.smithi099.stdout: "user_data": "", 2023-05-31T20:54:53.470 INFO:teuthology.orchestra.run.smithi099.stdout: "appendable": false 2023-05-31T20:54:53.470 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:54:53.470 INFO:teuthology.orchestra.run.smithi099.stdout: "tag": "_PIuaAUbZLQr32FBK8rYt6z4oTAAABe4", 2023-05-31T20:54:53.471 INFO:teuthology.orchestra.run.smithi099.stdout: "flags": 0, 2023-05-31T20:54:53.471 INFO:teuthology.orchestra.run.smithi099.stdout: "pending_map": [], 2023-05-31T20:54:53.471 INFO:teuthology.orchestra.run.smithi099.stdout: "versioned_epoch": 0 2023-05-31T20:54:53.471 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:54:53.471 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T20:54:53.471 INFO:teuthology.orchestra.run.smithi099.stdout: "name": "obj46", 2023-05-31T20:54:53.471 INFO:teuthology.orchestra.run.smithi099.stdout: "instance": "", 2023-05-31T20:54:53.472 INFO:teuthology.orchestra.run.smithi099.stdout: "ver": { 2023-05-31T20:54:53.472 INFO:teuthology.orchestra.run.smithi099.stdout: "pool": 13, 2023-05-31T20:54:53.472 INFO:teuthology.orchestra.run.smithi099.stdout: "epoch": 6 2023-05-31T20:54:53.472 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:54:53.472 INFO:teuthology.orchestra.run.smithi099.stdout: "locator": "", 2023-05-31T20:54:53.472 INFO:teuthology.orchestra.run.smithi099.stdout: "exists": true, 2023-05-31T20:54:53.473 INFO:teuthology.orchestra.run.smithi099.stdout: "meta": { 2023-05-31T20:54:53.473 INFO:teuthology.orchestra.run.smithi099.stdout: "category": 1, 2023-05-31T20:54:53.473 INFO:teuthology.orchestra.run.smithi099.stdout: "size": 5, 2023-05-31T20:54:53.473 INFO:teuthology.orchestra.run.smithi099.stdout: "mtime": "2023-05-31T20:46:33.830640Z", 2023-05-31T20:54:53.473 INFO:teuthology.orchestra.run.smithi099.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-05-31T20:54:53.473 INFO:teuthology.orchestra.run.smithi099.stdout: "storage_class": "", 2023-05-31T20:54:53.473 INFO:teuthology.orchestra.run.smithi099.stdout: "owner": "rgw-multisite-test-user", 2023-05-31T20:54:53.474 INFO:teuthology.orchestra.run.smithi099.stdout: "owner_display_name": "Multisite Test User", 2023-05-31T20:54:53.474 INFO:teuthology.orchestra.run.smithi099.stdout: "content_type": "application/octet-stream", 2023-05-31T20:54:53.474 INFO:teuthology.orchestra.run.smithi099.stdout: "accounted_size": 3, 2023-05-31T20:54:53.474 INFO:teuthology.orchestra.run.smithi099.stdout: "user_data": "", 2023-05-31T20:54:53.474 INFO:teuthology.orchestra.run.smithi099.stdout: "appendable": false 2023-05-31T20:54:53.474 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:54:53.474 INFO:teuthology.orchestra.run.smithi099.stdout: "tag": "_pGxLNsAhe6EoZLJHQWGIS-CuCTAb9so", 2023-05-31T20:54:53.475 INFO:teuthology.orchestra.run.smithi099.stdout: "flags": 0, 2023-05-31T20:54:53.475 INFO:teuthology.orchestra.run.smithi099.stdout: "pending_map": [], 2023-05-31T20:54:53.475 INFO:teuthology.orchestra.run.smithi099.stdout: "versioned_epoch": 0 2023-05-31T20:54:53.475 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:54:53.475 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T20:54:53.475 INFO:teuthology.orchestra.run.smithi099.stdout: "name": "obj51", 2023-05-31T20:54:53.475 INFO:teuthology.orchestra.run.smithi099.stdout: "instance": "", 2023-05-31T20:54:53.476 INFO:teuthology.orchestra.run.smithi099.stdout: "ver": { 2023-05-31T20:54:53.476 INFO:teuthology.orchestra.run.smithi099.stdout: "pool": 13, 2023-05-31T20:54:53.476 INFO:teuthology.orchestra.run.smithi099.stdout: "epoch": 6 2023-05-31T20:54:53.476 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:54:53.476 INFO:teuthology.orchestra.run.smithi099.stdout: "locator": "", 2023-05-31T20:54:53.476 INFO:teuthology.orchestra.run.smithi099.stdout: "exists": true, 2023-05-31T20:54:53.477 INFO:teuthology.orchestra.run.smithi099.stdout: "meta": { 2023-05-31T20:54:53.477 INFO:teuthology.orchestra.run.smithi099.stdout: "category": 1, 2023-05-31T20:54:53.477 INFO:teuthology.orchestra.run.smithi099.stdout: "size": 16, 2023-05-31T20:54:53.477 INFO:teuthology.orchestra.run.smithi099.stdout: "mtime": "2023-05-31T20:43:08.011797Z", 2023-05-31T20:54:53.477 INFO:teuthology.orchestra.run.smithi099.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-05-31T20:54:53.477 INFO:teuthology.orchestra.run.smithi099.stdout: "storage_class": "", 2023-05-31T20:54:53.477 INFO:teuthology.orchestra.run.smithi099.stdout: "owner": "rgw-multisite-test-user", 2023-05-31T20:54:53.478 INFO:teuthology.orchestra.run.smithi099.stdout: "owner_display_name": "Multisite Test User", 2023-05-31T20:54:53.478 INFO:teuthology.orchestra.run.smithi099.stdout: "content_type": "application/octet-stream", 2023-05-31T20:54:53.478 INFO:teuthology.orchestra.run.smithi099.stdout: "accounted_size": 3, 2023-05-31T20:54:53.478 INFO:teuthology.orchestra.run.smithi099.stdout: "user_data": "", 2023-05-31T20:54:53.478 INFO:teuthology.orchestra.run.smithi099.stdout: "appendable": false 2023-05-31T20:54:53.478 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:54:53.478 INFO:teuthology.orchestra.run.smithi099.stdout: "tag": "_ev7ce80s-Y2J__h_ZOe3uyUTeYm2uyI", 2023-05-31T20:54:53.479 INFO:teuthology.orchestra.run.smithi099.stdout: "flags": 0, 2023-05-31T20:54:53.479 INFO:teuthology.orchestra.run.smithi099.stdout: "pending_map": [], 2023-05-31T20:54:53.479 INFO:teuthology.orchestra.run.smithi099.stdout: "versioned_epoch": 0 2023-05-31T20:54:53.479 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:54:53.479 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T20:54:53.479 INFO:teuthology.orchestra.run.smithi099.stdout: "name": "obj55", 2023-05-31T20:54:53.479 INFO:teuthology.orchestra.run.smithi099.stdout: "instance": "", 2023-05-31T20:54:53.480 INFO:teuthology.orchestra.run.smithi099.stdout: "ver": { 2023-05-31T20:54:53.480 INFO:teuthology.orchestra.run.smithi099.stdout: "pool": 13, 2023-05-31T20:54:53.480 INFO:teuthology.orchestra.run.smithi099.stdout: "epoch": 10 2023-05-31T20:54:53.480 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:54:53.480 INFO:teuthology.orchestra.run.smithi099.stdout: "locator": "", 2023-05-31T20:54:53.480 INFO:teuthology.orchestra.run.smithi099.stdout: "exists": true, 2023-05-31T20:54:53.480 INFO:teuthology.orchestra.run.smithi099.stdout: "meta": { 2023-05-31T20:54:53.481 INFO:teuthology.orchestra.run.smithi099.stdout: "category": 1, 2023-05-31T20:54:53.481 INFO:teuthology.orchestra.run.smithi099.stdout: "size": 3, 2023-05-31T20:54:53.481 INFO:teuthology.orchestra.run.smithi099.stdout: "mtime": "2023-05-31T20:39:26.630472Z", 2023-05-31T20:54:53.481 INFO:teuthology.orchestra.run.smithi099.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-05-31T20:54:53.481 INFO:teuthology.orchestra.run.smithi099.stdout: "storage_class": "", 2023-05-31T20:54:53.481 INFO:teuthology.orchestra.run.smithi099.stdout: "owner": "rgw-multisite-test-user", 2023-05-31T20:54:53.482 INFO:teuthology.orchestra.run.smithi099.stdout: "owner_display_name": "Multisite Test User", 2023-05-31T20:54:53.482 INFO:teuthology.orchestra.run.smithi099.stdout: "content_type": "application/octet-stream", 2023-05-31T20:54:53.482 INFO:teuthology.orchestra.run.smithi099.stdout: "accounted_size": 3, 2023-05-31T20:54:53.482 INFO:teuthology.orchestra.run.smithi099.stdout: "user_data": "", 2023-05-31T20:54:53.482 INFO:teuthology.orchestra.run.smithi099.stdout: "appendable": false 2023-05-31T20:54:53.482 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:54:53.482 INFO:teuthology.orchestra.run.smithi099.stdout: "tag": "_zgZrtYYmCCxChOb7fl4uzzCdJm79Pas", 2023-05-31T20:54:53.483 INFO:teuthology.orchestra.run.smithi099.stdout: "flags": 0, 2023-05-31T20:54:53.483 INFO:teuthology.orchestra.run.smithi099.stdout: "pending_map": [], 2023-05-31T20:54:53.483 INFO:teuthology.orchestra.run.smithi099.stdout: "versioned_epoch": 0 2023-05-31T20:54:53.483 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:54:53.483 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T20:54:53.483 INFO:teuthology.orchestra.run.smithi099.stdout: "name": "obj57", 2023-05-31T20:54:53.484 INFO:teuthology.orchestra.run.smithi099.stdout: "instance": "", 2023-05-31T20:54:53.484 INFO:teuthology.orchestra.run.smithi099.stdout: "ver": { 2023-05-31T20:54:53.484 INFO:teuthology.orchestra.run.smithi099.stdout: "pool": 13, 2023-05-31T20:54:53.484 INFO:teuthology.orchestra.run.smithi099.stdout: "epoch": 8 2023-05-31T20:54:53.484 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:54:53.484 INFO:teuthology.orchestra.run.smithi099.stdout: "locator": "", 2023-05-31T20:54:53.485 INFO:teuthology.orchestra.run.smithi099.stdout: "exists": true, 2023-05-31T20:54:53.485 INFO:teuthology.orchestra.run.smithi099.stdout: "meta": { 2023-05-31T20:54:53.485 INFO:teuthology.orchestra.run.smithi099.stdout: "category": 1, 2023-05-31T20:54:53.485 INFO:teuthology.orchestra.run.smithi099.stdout: "size": 16, 2023-05-31T20:54:53.485 INFO:teuthology.orchestra.run.smithi099.stdout: "mtime": "2023-05-31T20:44:50.627910Z", 2023-05-31T20:54:53.485 INFO:teuthology.orchestra.run.smithi099.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-05-31T20:54:53.485 INFO:teuthology.orchestra.run.smithi099.stdout: "storage_class": "", 2023-05-31T20:54:53.486 INFO:teuthology.orchestra.run.smithi099.stdout: "owner": "rgw-multisite-test-user", 2023-05-31T20:54:53.486 INFO:teuthology.orchestra.run.smithi099.stdout: "owner_display_name": "Multisite Test User", 2023-05-31T20:54:53.486 INFO:teuthology.orchestra.run.smithi099.stdout: "content_type": "application/octet-stream", 2023-05-31T20:54:53.486 INFO:teuthology.orchestra.run.smithi099.stdout: "accounted_size": 3, 2023-05-31T20:54:53.486 INFO:teuthology.orchestra.run.smithi099.stdout: "user_data": "", 2023-05-31T20:54:53.486 INFO:teuthology.orchestra.run.smithi099.stdout: "appendable": false 2023-05-31T20:54:53.486 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:54:53.487 INFO:teuthology.orchestra.run.smithi099.stdout: "tag": "_6PcJWeFQm8nyZyhxCp3fcl3XtgP3kpa", 2023-05-31T20:54:53.487 INFO:teuthology.orchestra.run.smithi099.stdout: "flags": 0, 2023-05-31T20:54:53.487 INFO:teuthology.orchestra.run.smithi099.stdout: "pending_map": [], 2023-05-31T20:54:53.487 INFO:teuthology.orchestra.run.smithi099.stdout: "versioned_epoch": 0 2023-05-31T20:54:53.487 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:54:53.487 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T20:54:53.487 INFO:teuthology.orchestra.run.smithi099.stdout: "name": "obj58", 2023-05-31T20:54:53.488 INFO:teuthology.orchestra.run.smithi099.stdout: "instance": "", 2023-05-31T20:54:53.488 INFO:teuthology.orchestra.run.smithi099.stdout: "ver": { 2023-05-31T20:54:53.488 INFO:teuthology.orchestra.run.smithi099.stdout: "pool": 13, 2023-05-31T20:54:53.488 INFO:teuthology.orchestra.run.smithi099.stdout: "epoch": 9 2023-05-31T20:54:53.488 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:54:53.488 INFO:teuthology.orchestra.run.smithi099.stdout: "locator": "", 2023-05-31T20:54:53.488 INFO:teuthology.orchestra.run.smithi099.stdout: "exists": true, 2023-05-31T20:54:53.489 INFO:teuthology.orchestra.run.smithi099.stdout: "meta": { 2023-05-31T20:54:53.489 INFO:teuthology.orchestra.run.smithi099.stdout: "category": 1, 2023-05-31T20:54:53.489 INFO:teuthology.orchestra.run.smithi099.stdout: "size": 6, 2023-05-31T20:54:53.489 INFO:teuthology.orchestra.run.smithi099.stdout: "mtime": "2023-05-31T20:48:12.976191Z", 2023-05-31T20:54:53.489 INFO:teuthology.orchestra.run.smithi099.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-05-31T20:54:53.489 INFO:teuthology.orchestra.run.smithi099.stdout: "storage_class": "", 2023-05-31T20:54:53.489 INFO:teuthology.orchestra.run.smithi099.stdout: "owner": "rgw-multisite-test-user", 2023-05-31T20:54:53.490 INFO:teuthology.orchestra.run.smithi099.stdout: "owner_display_name": "Multisite Test User", 2023-05-31T20:54:53.490 INFO:teuthology.orchestra.run.smithi099.stdout: "content_type": "application/octet-stream", 2023-05-31T20:54:53.490 INFO:teuthology.orchestra.run.smithi099.stdout: "accounted_size": 3, 2023-05-31T20:54:53.490 INFO:teuthology.orchestra.run.smithi099.stdout: "user_data": "", 2023-05-31T20:54:53.490 INFO:teuthology.orchestra.run.smithi099.stdout: "appendable": false 2023-05-31T20:54:53.490 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:54:53.491 INFO:teuthology.orchestra.run.smithi099.stdout: "tag": "_mBFZER1SdYxKPXWX5ZkBo9a7BSEuyw-", 2023-05-31T20:54:53.491 INFO:teuthology.orchestra.run.smithi099.stdout: "flags": 0, 2023-05-31T20:54:53.491 INFO:teuthology.orchestra.run.smithi099.stdout: "pending_map": [], 2023-05-31T20:54:53.491 INFO:teuthology.orchestra.run.smithi099.stdout: "versioned_epoch": 0 2023-05-31T20:54:53.491 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:54:53.491 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T20:54:53.491 INFO:teuthology.orchestra.run.smithi099.stdout: "name": "obj62", 2023-05-31T20:54:53.492 INFO:teuthology.orchestra.run.smithi099.stdout: "instance": "", 2023-05-31T20:54:53.492 INFO:teuthology.orchestra.run.smithi099.stdout: "ver": { 2023-05-31T20:54:53.492 INFO:teuthology.orchestra.run.smithi099.stdout: "pool": 13, 2023-05-31T20:54:53.492 INFO:teuthology.orchestra.run.smithi099.stdout: "epoch": 5 2023-05-31T20:54:53.492 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:54:53.492 INFO:teuthology.orchestra.run.smithi099.stdout: "locator": "", 2023-05-31T20:54:53.492 INFO:teuthology.orchestra.run.smithi099.stdout: "exists": true, 2023-05-31T20:54:53.493 INFO:teuthology.orchestra.run.smithi099.stdout: "meta": { 2023-05-31T20:54:53.493 INFO:teuthology.orchestra.run.smithi099.stdout: "category": 1, 2023-05-31T20:54:53.493 INFO:teuthology.orchestra.run.smithi099.stdout: "size": 16, 2023-05-31T20:54:53.493 INFO:teuthology.orchestra.run.smithi099.stdout: "mtime": "2023-05-31T20:50:33.515966Z", 2023-05-31T20:54:53.493 INFO:teuthology.orchestra.run.smithi099.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-05-31T20:54:53.493 INFO:teuthology.orchestra.run.smithi099.stdout: "storage_class": "", 2023-05-31T20:54:53.493 INFO:teuthology.orchestra.run.smithi099.stdout: "owner": "rgw-multisite-test-user", 2023-05-31T20:54:53.494 INFO:teuthology.orchestra.run.smithi099.stdout: "owner_display_name": "Multisite Test User", 2023-05-31T20:54:53.494 INFO:teuthology.orchestra.run.smithi099.stdout: "content_type": "application/octet-stream", 2023-05-31T20:54:53.494 INFO:teuthology.orchestra.run.smithi099.stdout: "accounted_size": 3, 2023-05-31T20:54:53.494 INFO:teuthology.orchestra.run.smithi099.stdout: "user_data": "", 2023-05-31T20:54:53.494 INFO:teuthology.orchestra.run.smithi099.stdout: "appendable": false 2023-05-31T20:54:53.494 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:54:53.494 INFO:teuthology.orchestra.run.smithi099.stdout: "tag": "_eaKK7dx1ypoUYZGvyHEtmBI7ydn816H", 2023-05-31T20:54:53.495 INFO:teuthology.orchestra.run.smithi099.stdout: "flags": 0, 2023-05-31T20:54:53.495 INFO:teuthology.orchestra.run.smithi099.stdout: "pending_map": [], 2023-05-31T20:54:53.495 INFO:teuthology.orchestra.run.smithi099.stdout: "versioned_epoch": 0 2023-05-31T20:54:53.495 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:54:53.495 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T20:54:53.495 INFO:teuthology.orchestra.run.smithi099.stdout: "name": "obj68", 2023-05-31T20:54:53.495 INFO:teuthology.orchestra.run.smithi099.stdout: "instance": "", 2023-05-31T20:54:53.496 INFO:teuthology.orchestra.run.smithi099.stdout: "ver": { 2023-05-31T20:54:53.496 INFO:teuthology.orchestra.run.smithi099.stdout: "pool": 13, 2023-05-31T20:54:53.496 INFO:teuthology.orchestra.run.smithi099.stdout: "epoch": 7 2023-05-31T20:54:53.496 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:54:53.496 INFO:teuthology.orchestra.run.smithi099.stdout: "locator": "", 2023-05-31T20:54:53.496 INFO:teuthology.orchestra.run.smithi099.stdout: "exists": true, 2023-05-31T20:54:53.496 INFO:teuthology.orchestra.run.smithi099.stdout: "meta": { 2023-05-31T20:54:53.497 INFO:teuthology.orchestra.run.smithi099.stdout: "category": 1, 2023-05-31T20:54:53.497 INFO:teuthology.orchestra.run.smithi099.stdout: "size": 16, 2023-05-31T20:54:53.497 INFO:teuthology.orchestra.run.smithi099.stdout: "mtime": "2023-05-31T20:43:08.066803Z", 2023-05-31T20:54:53.497 INFO:teuthology.orchestra.run.smithi099.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-05-31T20:54:53.497 INFO:teuthology.orchestra.run.smithi099.stdout: "storage_class": "", 2023-05-31T20:54:53.497 INFO:teuthology.orchestra.run.smithi099.stdout: "owner": "rgw-multisite-test-user", 2023-05-31T20:54:53.497 INFO:teuthology.orchestra.run.smithi099.stdout: "owner_display_name": "Multisite Test User", 2023-05-31T20:54:53.498 INFO:teuthology.orchestra.run.smithi099.stdout: "content_type": "application/octet-stream", 2023-05-31T20:54:53.498 INFO:teuthology.orchestra.run.smithi099.stdout: "accounted_size": 3, 2023-05-31T20:54:53.498 INFO:teuthology.orchestra.run.smithi099.stdout: "user_data": "", 2023-05-31T20:54:53.498 INFO:teuthology.orchestra.run.smithi099.stdout: "appendable": false 2023-05-31T20:54:53.498 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:54:53.498 INFO:teuthology.orchestra.run.smithi099.stdout: "tag": "_-g-VlK6GpWIvV8-gk5lJwlYGD71QSzL", 2023-05-31T20:54:53.498 INFO:teuthology.orchestra.run.smithi099.stdout: "flags": 0, 2023-05-31T20:54:53.499 INFO:teuthology.orchestra.run.smithi099.stdout: "pending_map": [], 2023-05-31T20:54:53.499 INFO:teuthology.orchestra.run.smithi099.stdout: "versioned_epoch": 0 2023-05-31T20:54:53.499 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:54:53.499 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T20:54:53.499 INFO:teuthology.orchestra.run.smithi099.stdout: "name": "obj69", 2023-05-31T20:54:53.499 INFO:teuthology.orchestra.run.smithi099.stdout: "instance": "", 2023-05-31T20:54:53.499 INFO:teuthology.orchestra.run.smithi099.stdout: "ver": { 2023-05-31T20:54:53.500 INFO:teuthology.orchestra.run.smithi099.stdout: "pool": 13, 2023-05-31T20:54:53.500 INFO:teuthology.orchestra.run.smithi099.stdout: "epoch": 3 2023-05-31T20:54:53.500 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:54:53.500 INFO:teuthology.orchestra.run.smithi099.stdout: "locator": "", 2023-05-31T20:54:53.500 INFO:teuthology.orchestra.run.smithi099.stdout: "exists": true, 2023-05-31T20:54:53.500 INFO:teuthology.orchestra.run.smithi099.stdout: "meta": { 2023-05-31T20:54:53.501 INFO:teuthology.orchestra.run.smithi099.stdout: "category": 1, 2023-05-31T20:54:53.501 INFO:teuthology.orchestra.run.smithi099.stdout: "size": 16, 2023-05-31T20:54:53.501 INFO:teuthology.orchestra.run.smithi099.stdout: "mtime": "2023-05-31T20:46:33.886933Z", 2023-05-31T20:54:53.501 INFO:teuthology.orchestra.run.smithi099.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-05-31T20:54:53.501 INFO:teuthology.orchestra.run.smithi099.stdout: "storage_class": "", 2023-05-31T20:54:53.501 INFO:teuthology.orchestra.run.smithi099.stdout: "owner": "rgw-multisite-test-user", 2023-05-31T20:54:53.501 INFO:teuthology.orchestra.run.smithi099.stdout: "owner_display_name": "Multisite Test User", 2023-05-31T20:54:53.502 INFO:teuthology.orchestra.run.smithi099.stdout: "content_type": "application/octet-stream", 2023-05-31T20:54:53.502 INFO:teuthology.orchestra.run.smithi099.stdout: "accounted_size": 3, 2023-05-31T20:54:53.502 INFO:teuthology.orchestra.run.smithi099.stdout: "user_data": "", 2023-05-31T20:54:53.502 INFO:teuthology.orchestra.run.smithi099.stdout: "appendable": false 2023-05-31T20:54:53.502 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:54:53.502 INFO:teuthology.orchestra.run.smithi099.stdout: "tag": "_uVhCIkkpyt1UGKHE2B7Vy3CpqBtduUW", 2023-05-31T20:54:53.502 INFO:teuthology.orchestra.run.smithi099.stdout: "flags": 0, 2023-05-31T20:54:53.503 INFO:teuthology.orchestra.run.smithi099.stdout: "pending_map": [], 2023-05-31T20:54:53.503 INFO:teuthology.orchestra.run.smithi099.stdout: "versioned_epoch": 0 2023-05-31T20:54:53.503 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:54:53.503 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T20:54:53.503 INFO:teuthology.orchestra.run.smithi099.stdout: "name": "obj74", 2023-05-31T20:54:53.503 INFO:teuthology.orchestra.run.smithi099.stdout: "instance": "", 2023-05-31T20:54:53.503 INFO:teuthology.orchestra.run.smithi099.stdout: "ver": { 2023-05-31T20:54:53.504 INFO:teuthology.orchestra.run.smithi099.stdout: "pool": 13, 2023-05-31T20:54:53.504 INFO:teuthology.orchestra.run.smithi099.stdout: "epoch": 8 2023-05-31T20:54:53.504 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:54:53.504 INFO:teuthology.orchestra.run.smithi099.stdout: "locator": "", 2023-05-31T20:54:53.504 INFO:teuthology.orchestra.run.smithi099.stdout: "exists": true, 2023-05-31T20:54:53.504 INFO:teuthology.orchestra.run.smithi099.stdout: "meta": { 2023-05-31T20:54:53.504 INFO:teuthology.orchestra.run.smithi099.stdout: "category": 1, 2023-05-31T20:54:53.505 INFO:teuthology.orchestra.run.smithi099.stdout: "size": 5, 2023-05-31T20:54:53.505 INFO:teuthology.orchestra.run.smithi099.stdout: "mtime": "2023-05-31T20:52:14.934725Z", 2023-05-31T20:54:53.505 INFO:teuthology.orchestra.run.smithi099.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-05-31T20:54:53.505 INFO:teuthology.orchestra.run.smithi099.stdout: "storage_class": "", 2023-05-31T20:54:53.505 INFO:teuthology.orchestra.run.smithi099.stdout: "owner": "rgw-multisite-test-user", 2023-05-31T20:54:53.505 INFO:teuthology.orchestra.run.smithi099.stdout: "owner_display_name": "Multisite Test User", 2023-05-31T20:54:53.505 INFO:teuthology.orchestra.run.smithi099.stdout: "content_type": "application/octet-stream", 2023-05-31T20:54:53.506 INFO:teuthology.orchestra.run.smithi099.stdout: "accounted_size": 3, 2023-05-31T20:54:53.506 INFO:teuthology.orchestra.run.smithi099.stdout: "user_data": "", 2023-05-31T20:54:53.506 INFO:teuthology.orchestra.run.smithi099.stdout: "appendable": false 2023-05-31T20:54:53.506 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:54:53.506 INFO:teuthology.orchestra.run.smithi099.stdout: "tag": "_lrGKVfNSuFZA8snSFJGmHwryZwvsFUa", 2023-05-31T20:54:53.506 INFO:teuthology.orchestra.run.smithi099.stdout: "flags": 0, 2023-05-31T20:54:53.506 INFO:teuthology.orchestra.run.smithi099.stdout: "pending_map": [], 2023-05-31T20:54:53.507 INFO:teuthology.orchestra.run.smithi099.stdout: "versioned_epoch": 0 2023-05-31T20:54:53.507 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:54:53.507 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T20:54:53.507 INFO:teuthology.orchestra.run.smithi099.stdout: "name": "obj76", 2023-05-31T20:54:53.507 INFO:teuthology.orchestra.run.smithi099.stdout: "instance": "", 2023-05-31T20:54:53.507 INFO:teuthology.orchestra.run.smithi099.stdout: "ver": { 2023-05-31T20:54:53.507 INFO:teuthology.orchestra.run.smithi099.stdout: "pool": 13, 2023-05-31T20:54:53.508 INFO:teuthology.orchestra.run.smithi099.stdout: "epoch": 7 2023-05-31T20:54:53.508 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:54:53.508 INFO:teuthology.orchestra.run.smithi099.stdout: "locator": "", 2023-05-31T20:54:53.508 INFO:teuthology.orchestra.run.smithi099.stdout: "exists": true, 2023-05-31T20:54:53.508 INFO:teuthology.orchestra.run.smithi099.stdout: "meta": { 2023-05-31T20:54:53.508 INFO:teuthology.orchestra.run.smithi099.stdout: "category": 1, 2023-05-31T20:54:53.508 INFO:teuthology.orchestra.run.smithi099.stdout: "size": 5, 2023-05-31T20:54:53.509 INFO:teuthology.orchestra.run.smithi099.stdout: "mtime": "2023-05-31T20:44:50.682916Z", 2023-05-31T20:54:53.509 INFO:teuthology.orchestra.run.smithi099.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-05-31T20:54:53.509 INFO:teuthology.orchestra.run.smithi099.stdout: "storage_class": "", 2023-05-31T20:54:53.509 INFO:teuthology.orchestra.run.smithi099.stdout: "owner": "rgw-multisite-test-user", 2023-05-31T20:54:53.509 INFO:teuthology.orchestra.run.smithi099.stdout: "owner_display_name": "Multisite Test User", 2023-05-31T20:54:53.509 INFO:teuthology.orchestra.run.smithi099.stdout: "content_type": "application/octet-stream", 2023-05-31T20:54:53.509 INFO:teuthology.orchestra.run.smithi099.stdout: "accounted_size": 3, 2023-05-31T20:54:53.510 INFO:teuthology.orchestra.run.smithi099.stdout: "user_data": "", 2023-05-31T20:54:53.510 INFO:teuthology.orchestra.run.smithi099.stdout: "appendable": false 2023-05-31T20:54:53.510 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:54:53.510 INFO:teuthology.orchestra.run.smithi099.stdout: "tag": "_Svh21sRTRFq72Ls-jnuzGKJLIEZQYR1", 2023-05-31T20:54:53.510 INFO:teuthology.orchestra.run.smithi099.stdout: "flags": 0, 2023-05-31T20:54:53.510 INFO:teuthology.orchestra.run.smithi099.stdout: "pending_map": [], 2023-05-31T20:54:53.511 INFO:teuthology.orchestra.run.smithi099.stdout: "versioned_epoch": 0 2023-05-31T20:54:53.511 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:54:53.511 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T20:54:53.511 INFO:teuthology.orchestra.run.smithi099.stdout: "name": "obj85", 2023-05-31T20:54:53.511 INFO:teuthology.orchestra.run.smithi099.stdout: "instance": "", 2023-05-31T20:54:53.511 INFO:teuthology.orchestra.run.smithi099.stdout: "ver": { 2023-05-31T20:54:53.511 INFO:teuthology.orchestra.run.smithi099.stdout: "pool": 13, 2023-05-31T20:54:53.511 INFO:teuthology.orchestra.run.smithi099.stdout: "epoch": 10 2023-05-31T20:54:53.512 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:54:53.512 INFO:teuthology.orchestra.run.smithi099.stdout: "locator": "", 2023-05-31T20:54:53.512 INFO:teuthology.orchestra.run.smithi099.stdout: "exists": true, 2023-05-31T20:54:53.512 INFO:teuthology.orchestra.run.smithi099.stdout: "meta": { 2023-05-31T20:54:53.512 INFO:teuthology.orchestra.run.smithi099.stdout: "category": 1, 2023-05-31T20:54:53.512 INFO:teuthology.orchestra.run.smithi099.stdout: "size": 3, 2023-05-31T20:54:53.513 INFO:teuthology.orchestra.run.smithi099.stdout: "mtime": "2023-05-31T20:43:08.086089Z", 2023-05-31T20:54:53.513 INFO:teuthology.orchestra.run.smithi099.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-05-31T20:54:53.513 INFO:teuthology.orchestra.run.smithi099.stdout: "storage_class": "", 2023-05-31T20:54:53.513 INFO:teuthology.orchestra.run.smithi099.stdout: "owner": "rgw-multisite-test-user", 2023-05-31T20:54:53.513 INFO:teuthology.orchestra.run.smithi099.stdout: "owner_display_name": "Multisite Test User", 2023-05-31T20:54:53.513 INFO:teuthology.orchestra.run.smithi099.stdout: "content_type": "application/octet-stream", 2023-05-31T20:54:53.513 INFO:teuthology.orchestra.run.smithi099.stdout: "accounted_size": 3, 2023-05-31T20:54:53.514 INFO:teuthology.orchestra.run.smithi099.stdout: "user_data": "", 2023-05-31T20:54:53.514 INFO:teuthology.orchestra.run.smithi099.stdout: "appendable": false 2023-05-31T20:54:53.514 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:54:53.514 INFO:teuthology.orchestra.run.smithi099.stdout: "tag": "_IklzCcrcGemqehp_5gUbqtDVBVeoXZY", 2023-05-31T20:54:53.514 INFO:teuthology.orchestra.run.smithi099.stdout: "flags": 0, 2023-05-31T20:54:53.514 INFO:teuthology.orchestra.run.smithi099.stdout: "pending_map": [], 2023-05-31T20:54:53.514 INFO:teuthology.orchestra.run.smithi099.stdout: "versioned_epoch": 0 2023-05-31T20:54:53.514 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:54:53.515 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T20:54:53.515 INFO:teuthology.orchestra.run.smithi099.stdout: "name": "obj87", 2023-05-31T20:54:53.515 INFO:teuthology.orchestra.run.smithi099.stdout: "instance": "", 2023-05-31T20:54:53.515 INFO:teuthology.orchestra.run.smithi099.stdout: "ver": { 2023-05-31T20:54:53.515 INFO:teuthology.orchestra.run.smithi099.stdout: "pool": 13, 2023-05-31T20:54:53.515 INFO:teuthology.orchestra.run.smithi099.stdout: "epoch": 10 2023-05-31T20:54:53.515 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:54:53.516 INFO:teuthology.orchestra.run.smithi099.stdout: "locator": "", 2023-05-31T20:54:53.517 INFO:teuthology.orchestra.run.smithi099.stdout: "exists": true, 2023-05-31T20:54:53.517 INFO:teuthology.orchestra.run.smithi099.stdout: "meta": { 2023-05-31T20:54:53.517 INFO:teuthology.orchestra.run.smithi099.stdout: "category": 1, 2023-05-31T20:54:53.517 INFO:teuthology.orchestra.run.smithi099.stdout: "size": 16, 2023-05-31T20:54:53.517 INFO:teuthology.orchestra.run.smithi099.stdout: "mtime": "2023-05-31T20:48:13.031366Z", 2023-05-31T20:54:53.517 INFO:teuthology.orchestra.run.smithi099.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-05-31T20:54:53.518 INFO:teuthology.orchestra.run.smithi099.stdout: "storage_class": "", 2023-05-31T20:54:53.518 INFO:teuthology.orchestra.run.smithi099.stdout: "owner": "rgw-multisite-test-user", 2023-05-31T20:54:53.518 INFO:teuthology.orchestra.run.smithi099.stdout: "owner_display_name": "Multisite Test User", 2023-05-31T20:54:53.518 INFO:teuthology.orchestra.run.smithi099.stdout: "content_type": "application/octet-stream", 2023-05-31T20:54:53.518 INFO:teuthology.orchestra.run.smithi099.stdout: "accounted_size": 3, 2023-05-31T20:54:53.518 INFO:teuthology.orchestra.run.smithi099.stdout: "user_data": "", 2023-05-31T20:54:53.518 INFO:teuthology.orchestra.run.smithi099.stdout: "appendable": false 2023-05-31T20:54:53.519 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:54:53.519 INFO:teuthology.orchestra.run.smithi099.stdout: "tag": "_u6Duf_yfupDOJy8twJE7aMJKqMyIek2", 2023-05-31T20:54:53.519 INFO:teuthology.orchestra.run.smithi099.stdout: "flags": 0, 2023-05-31T20:54:53.519 INFO:teuthology.orchestra.run.smithi099.stdout: "pending_map": [], 2023-05-31T20:54:53.519 INFO:teuthology.orchestra.run.smithi099.stdout: "versioned_epoch": 0 2023-05-31T20:54:53.519 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:54:53.520 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T20:54:53.520 INFO:teuthology.orchestra.run.smithi099.stdout: "name": "obj92", 2023-05-31T20:54:53.520 INFO:teuthology.orchestra.run.smithi099.stdout: "instance": "", 2023-05-31T20:54:53.520 INFO:teuthology.orchestra.run.smithi099.stdout: "ver": { 2023-05-31T20:54:53.520 INFO:teuthology.orchestra.run.smithi099.stdout: "pool": 13, 2023-05-31T20:54:53.520 INFO:teuthology.orchestra.run.smithi099.stdout: "epoch": 7 2023-05-31T20:54:53.520 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:54:53.521 INFO:teuthology.orchestra.run.smithi099.stdout: "locator": "", 2023-05-31T20:54:53.521 INFO:teuthology.orchestra.run.smithi099.stdout: "exists": true, 2023-05-31T20:54:53.521 INFO:teuthology.orchestra.run.smithi099.stdout: "meta": { 2023-05-31T20:54:53.521 INFO:teuthology.orchestra.run.smithi099.stdout: "category": 1, 2023-05-31T20:54:53.521 INFO:teuthology.orchestra.run.smithi099.stdout: "size": 6, 2023-05-31T20:54:53.521 INFO:teuthology.orchestra.run.smithi099.stdout: "mtime": "2023-05-31T20:46:33.904998Z", 2023-05-31T20:54:53.521 INFO:teuthology.orchestra.run.smithi099.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-05-31T20:54:53.522 INFO:teuthology.orchestra.run.smithi099.stdout: "storage_class": "", 2023-05-31T20:54:53.522 INFO:teuthology.orchestra.run.smithi099.stdout: "owner": "rgw-multisite-test-user", 2023-05-31T20:54:53.522 INFO:teuthology.orchestra.run.smithi099.stdout: "owner_display_name": "Multisite Test User", 2023-05-31T20:54:53.522 INFO:teuthology.orchestra.run.smithi099.stdout: "content_type": "application/octet-stream", 2023-05-31T20:54:53.522 INFO:teuthology.orchestra.run.smithi099.stdout: "accounted_size": 3, 2023-05-31T20:54:53.522 INFO:teuthology.orchestra.run.smithi099.stdout: "user_data": "", 2023-05-31T20:54:53.522 INFO:teuthology.orchestra.run.smithi099.stdout: "appendable": false 2023-05-31T20:54:53.523 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:54:53.523 INFO:teuthology.orchestra.run.smithi099.stdout: "tag": "_NYFuilhsfIUVg55_z-ppR6tRnREV2i2", 2023-05-31T20:54:53.523 INFO:teuthology.orchestra.run.smithi099.stdout: "flags": 0, 2023-05-31T20:54:53.523 INFO:teuthology.orchestra.run.smithi099.stdout: "pending_map": [], 2023-05-31T20:54:53.523 INFO:teuthology.orchestra.run.smithi099.stdout: "versioned_epoch": 0 2023-05-31T20:54:53.523 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:54:53.523 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T20:54:53.524 INFO:teuthology.orchestra.run.smithi099.stdout: "name": "obj93", 2023-05-31T20:54:53.524 INFO:teuthology.orchestra.run.smithi099.stdout: "instance": "", 2023-05-31T20:54:53.524 INFO:teuthology.orchestra.run.smithi099.stdout: "ver": { 2023-05-31T20:54:53.524 INFO:teuthology.orchestra.run.smithi099.stdout: "pool": 13, 2023-05-31T20:54:53.524 INFO:teuthology.orchestra.run.smithi099.stdout: "epoch": 5 2023-05-31T20:54:53.524 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:54:53.525 INFO:teuthology.orchestra.run.smithi099.stdout: "locator": "", 2023-05-31T20:54:53.525 INFO:teuthology.orchestra.run.smithi099.stdout: "exists": true, 2023-05-31T20:54:53.525 INFO:teuthology.orchestra.run.smithi099.stdout: "meta": { 2023-05-31T20:54:53.525 INFO:teuthology.orchestra.run.smithi099.stdout: "category": 1, 2023-05-31T20:54:53.525 INFO:teuthology.orchestra.run.smithi099.stdout: "size": 16, 2023-05-31T20:54:53.525 INFO:teuthology.orchestra.run.smithi099.stdout: "mtime": "2023-05-31T20:50:33.533538Z", 2023-05-31T20:54:53.525 INFO:teuthology.orchestra.run.smithi099.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-05-31T20:54:53.526 INFO:teuthology.orchestra.run.smithi099.stdout: "storage_class": "", 2023-05-31T20:54:53.526 INFO:teuthology.orchestra.run.smithi099.stdout: "owner": "rgw-multisite-test-user", 2023-05-31T20:54:53.526 INFO:teuthology.orchestra.run.smithi099.stdout: "owner_display_name": "Multisite Test User", 2023-05-31T20:54:53.526 INFO:teuthology.orchestra.run.smithi099.stdout: "content_type": "application/octet-stream", 2023-05-31T20:54:53.526 INFO:teuthology.orchestra.run.smithi099.stdout: "accounted_size": 3, 2023-05-31T20:54:53.526 INFO:teuthology.orchestra.run.smithi099.stdout: "user_data": "", 2023-05-31T20:54:53.526 INFO:teuthology.orchestra.run.smithi099.stdout: "appendable": false 2023-05-31T20:54:53.527 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:54:53.527 INFO:teuthology.orchestra.run.smithi099.stdout: "tag": "_cDZPnBwzWd11q8OtetfBNacZbiznhSJ", 2023-05-31T20:54:53.527 INFO:teuthology.orchestra.run.smithi099.stdout: "flags": 0, 2023-05-31T20:54:53.527 INFO:teuthology.orchestra.run.smithi099.stdout: "pending_map": [], 2023-05-31T20:54:53.527 INFO:teuthology.orchestra.run.smithi099.stdout: "versioned_epoch": 0 2023-05-31T20:54:53.527 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:54:53.527 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T20:54:53.528 INFO:teuthology.orchestra.run.smithi099.stdout: "name": "obj95", 2023-05-31T20:54:53.528 INFO:teuthology.orchestra.run.smithi099.stdout: "instance": "", 2023-05-31T20:54:53.528 INFO:teuthology.orchestra.run.smithi099.stdout: "ver": { 2023-05-31T20:54:53.528 INFO:teuthology.orchestra.run.smithi099.stdout: "pool": 13, 2023-05-31T20:54:53.528 INFO:teuthology.orchestra.run.smithi099.stdout: "epoch": 8 2023-05-31T20:54:53.528 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:54:53.528 INFO:teuthology.orchestra.run.smithi099.stdout: "locator": "", 2023-05-31T20:54:53.529 INFO:teuthology.orchestra.run.smithi099.stdout: "exists": true, 2023-05-31T20:54:53.529 INFO:teuthology.orchestra.run.smithi099.stdout: "meta": { 2023-05-31T20:54:53.529 INFO:teuthology.orchestra.run.smithi099.stdout: "category": 1, 2023-05-31T20:54:53.529 INFO:teuthology.orchestra.run.smithi099.stdout: "size": 5, 2023-05-31T20:54:53.529 INFO:teuthology.orchestra.run.smithi099.stdout: "mtime": "2023-05-31T20:44:50.705817Z", 2023-05-31T20:54:53.529 INFO:teuthology.orchestra.run.smithi099.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-05-31T20:54:53.529 INFO:teuthology.orchestra.run.smithi099.stdout: "storage_class": "", 2023-05-31T20:54:53.530 INFO:teuthology.orchestra.run.smithi099.stdout: "owner": "rgw-multisite-test-user", 2023-05-31T20:54:53.530 INFO:teuthology.orchestra.run.smithi099.stdout: "owner_display_name": "Multisite Test User", 2023-05-31T20:54:53.530 INFO:teuthology.orchestra.run.smithi099.stdout: "content_type": "application/octet-stream", 2023-05-31T20:54:53.530 INFO:teuthology.orchestra.run.smithi099.stdout: "accounted_size": 3, 2023-05-31T20:54:53.530 INFO:teuthology.orchestra.run.smithi099.stdout: "user_data": "", 2023-05-31T20:54:53.530 INFO:teuthology.orchestra.run.smithi099.stdout: "appendable": false 2023-05-31T20:54:53.530 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:54:53.531 INFO:teuthology.orchestra.run.smithi099.stdout: "tag": "_f_nzOnbpJ2sU6DEpmDZnIWPNpc_Pi4j", 2023-05-31T20:54:53.531 INFO:teuthology.orchestra.run.smithi099.stdout: "flags": 0, 2023-05-31T20:54:53.531 INFO:teuthology.orchestra.run.smithi099.stdout: "pending_map": [], 2023-05-31T20:54:53.531 INFO:teuthology.orchestra.run.smithi099.stdout: "versioned_epoch": 0 2023-05-31T20:54:53.531 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T20:54:53.531 INFO:teuthology.orchestra.run.smithi099.stdout:] 2023-05-31T20:54:53.532 INFO:tasks.rgw.c1.client.1:Restarting daemon 2023-05-31T20:54:53.532 DEBUG:teuthology.orchestra.run.smithi099:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term radosgw --rgw-frontends 'beast port=8001' -n client.1 --cluster c1 -k /etc/ceph/c1.client.1.keyring --log-file /var/log/ceph/rgw.c1.client.1.log --rgw_ops_log_socket_path /home/ubuntu/cephtest/rgw.opslog.c1.client.1.sock --foreground --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm | sudo tee /var/log/ceph/rgw.c1.client.1.stdout 2>&1 2023-05-31T20:54:53.535 INFO:tasks.rgw.c1.client.1:Started 2023-05-31T20:54:53.535 DEBUG:teuthology.orchestra.run.smithi099:> curl http://smithi099.front.sepia.ceph.com:8001 2023-05-31T20:54:53.556 DEBUG:teuthology.orchestra.run:got remote process result: 7 2023-05-31T20:54:53.557 INFO:teuthology.orchestra.run.smithi099.stderr: % Total % Received % Xferd Average Speed Time Time Time Current 2023-05-31T20:54:53.557 INFO:teuthology.orchestra.run.smithi099.stderr: Dload Upload Total Spent Left Speed 2023-05-31T20:54:53.557 INFO:teuthology.orchestra.run.smithi099.stderr: 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0curl: (7) Failed to connect to smithi099.front.sepia.ceph.com port 8001: Connection refused 2023-05-31T20:54:54.559 DEBUG:teuthology.orchestra.run.smithi099:> curl http://smithi099.front.sepia.ceph.com:8001 2023-05-31T20:54:54.581 INFO:teuthology.orchestra.run.smithi099.stderr: % Total % Received % Xferd Average Speed Time Time Time Current 2023-05-31T20:54:54.581 INFO:teuthology.orchestra.run.smithi099.stderr: Dload Upload Total Spent Left Speed 2023-05-31T20:54:54.581 INFO:teuthology.orchestra.run.smithi099.stderr: 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 100 214 0 214 0 0 208k 0 --:--:-- --:--:-- --:--:-- 208k 2023-05-31T20:54:54.582 INFO:teuthology.orchestra.run.smithi099.stdout:anonymous 2023-05-31T20:54:54.582 DEBUG:rgw_multi.tests:bucket checkpoint: source=a1 target=a2 bucket=pklxkd-59 2023-05-31T20:54:54.583 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'pklxkd-59', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-05-31T20:54:54.583 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'pklxkd-59', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-05-31T20:54:54.584 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket pklxkd-59 --source-zone a1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2023-05-31T20:54:54.678 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2023-05-31T20:54:54.678 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2023-05-31T20:54:54.743 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:54:54.738+0000 7f1880a4a780 1 bucket sync caught up with source: 2023-05-31T20:54:54.743 INFO:teuthology.orchestra.run.smithi099.stderr: local status: [, , , , , , 00000000002.48.6, , , , , , 00000000002.68.6, , , , , 00000000002.63.6, , , , , , , , , , , , , , , 00000000001.75.6, , , , 00000000001.83.6] 2023-05-31T20:54:54.743 INFO:teuthology.orchestra.run.smithi099.stderr: remote markers: [, , , , , , 00000000002.48.6, , , , , , 00000000002.68.6, , , , , 00000000002.63.6, , , , , , , , , , , , , , , 00000000001.75.6, , , , 00000000001.83.6] 2023-05-31T20:54:54.743 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:54:54.738+0000 7f1880a4a780 0 bucket checkpoint complete 2023-05-31T20:54:54.764 DEBUG:rgw_multi.tests:bucket checkpoint: source=a2 target=a1 bucket=pklxkd-59 2023-05-31T20:54:54.765 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'pklxkd-59', '--source-zone', 'a2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-05-31T20:54:54.765 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'pklxkd-59', '--source-zone', 'a2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-05-31T20:54:54.766 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket pklxkd-59 --source-zone a2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2023-05-31T20:54:54.821 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2023-05-31T20:54:54.821 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2023-05-31T20:54:54.866 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:54:54.861+0000 7fe84b253780 1 bucket sync caught up with source: 2023-05-31T20:54:54.866 INFO:teuthology.orchestra.run.smithi099.stderr: local status: [, 00000000009.3611.6, 00000000006.3938.6, 00000000006.2050.6, 00000000015.91.6, 00000000012.2080.6, 00000000003.388.6, 00000000003.3940.6, 00000000012.2081.6, 00000000021.2341.6, 00000000018.2056.6] 2023-05-31T20:54:54.866 INFO:teuthology.orchestra.run.smithi099.stderr: remote markers: [, 00000000009.3611.6, 00000000006.3938.6, 00000000006.2050.6, 00000000015.91.6, 00000000012.2080.6, 00000000003.388.6, 00000000003.3940.6, 00000000012.2081.6, 00000000021.2341.6, 00000000018.2056.6] 2023-05-31T20:54:54.866 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:54:54.861+0000 7fe84b253780 0 bucket checkpoint complete 2023-05-31T20:54:54.879 INFO:tasks.rgw_multi.zone_rados:comparing bucket=pklxkd-59 zones={a2, a1} 2023-05-31T20:54:54.880 DEBUG:boto:path=/pklxkd-59/ 2023-05-31T20:54:54.880 DEBUG:boto:auth_path=/pklxkd-59/ 2023-05-31T20:54:54.881 DEBUG:boto:Method: HEAD 2023-05-31T20:54:54.881 DEBUG:boto:Path: /pklxkd-59/ 2023-05-31T20:54:54.881 DEBUG:boto:Data: 2023-05-31T20:54:54.882 DEBUG:boto:Headers: {} 2023-05-31T20:54:54.882 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:54:54.882 DEBUG:boto:Port: 8001 2023-05-31T20:54:54.883 DEBUG:boto:Params: {} 2023-05-31T20:54:54.883 DEBUG:boto:Token: None 2023-05-31T20:54:54.883 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:54:54 GMT /pklxkd-59/ 2023-05-31T20:54:54.884 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:ZkCSsspsHMYS/LjKZSJNygMcRXA= 2023-05-31T20:54:54.884 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:54:54 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:ZkCSsspsHMYS/LjKZSJNygMcRXA=', 'Content-Length': '0'} 2023-05-31T20:54:54.885 DEBUG:boto:encountered RemoteDisconnected exception, reconnecting 2023-05-31T20:54:54.925 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8001} 2023-05-31T20:54:55.358 DEBUG:boto:Token: None 2023-05-31T20:54:55.358 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:54:54 GMT /pklxkd-59/ 2023-05-31T20:54:55.359 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:ZkCSsspsHMYS/LjKZSJNygMcRXA= 2023-05-31T20:54:55.359 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:54:54 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:ZkCSsspsHMYS/LjKZSJNygMcRXA=', 'Content-Length': '0'} 2023-05-31T20:54:55.364 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '35'), ('X-RGW-Bytes-Used', '105'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000d09f6fece46007fd-006477b41f-1e2e-a2'), ('Content-Length', '0'), ('Date', 'Wed, 31 May 2023 20:54:55 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:54:55.364 DEBUG:boto:path=/pklxkd-59/ 2023-05-31T20:54:55.365 DEBUG:boto:auth_path=/pklxkd-59/ 2023-05-31T20:54:55.365 DEBUG:boto:Method: HEAD 2023-05-31T20:54:55.365 DEBUG:boto:Path: /pklxkd-59/ 2023-05-31T20:54:55.366 DEBUG:boto:Data: 2023-05-31T20:54:55.366 DEBUG:boto:Headers: {} 2023-05-31T20:54:55.366 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:54:55.367 DEBUG:boto:Port: 8000 2023-05-31T20:54:55.367 DEBUG:boto:Params: {} 2023-05-31T20:54:55.367 DEBUG:boto:Token: None 2023-05-31T20:54:55.368 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:54:55 GMT /pklxkd-59/ 2023-05-31T20:54:55.368 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:0CZ1fh5+Y/Wu8fmED4ZQt3kAPNM= 2023-05-31T20:54:55.369 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:54:55 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:0CZ1fh5+Y/Wu8fmED4ZQt3kAPNM=', 'Content-Length': '0'} 2023-05-31T20:54:55.372 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '35'), ('X-RGW-Bytes-Used', '105'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000c22b3c28c04b74dc-006477b41f-1630-a1'), ('Content-Length', '0'), ('Date', 'Wed, 31 May 2023 20:54:55 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:54:55.373 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2023-05-31T20:54:55.373 DEBUG:boto:path=/pklxkd-59/ 2023-05-31T20:54:55.374 DEBUG:boto:auth_path=/pklxkd-59/ 2023-05-31T20:54:55.374 DEBUG:boto:path=/pklxkd-59/?versions&max-keys=999 2023-05-31T20:54:55.374 DEBUG:boto:auth_path=/pklxkd-59/?versions&max-keys=999 2023-05-31T20:54:55.375 DEBUG:boto:Method: GET 2023-05-31T20:54:55.375 DEBUG:boto:Path: /pklxkd-59/?versions&max-keys=999 2023-05-31T20:54:55.375 DEBUG:boto:Data: 2023-05-31T20:54:55.376 DEBUG:boto:Headers: {} 2023-05-31T20:54:55.376 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:54:55.376 DEBUG:boto:Port: 8001 2023-05-31T20:54:55.377 DEBUG:boto:Params: {} 2023-05-31T20:54:55.377 DEBUG:boto:Token: None 2023-05-31T20:54:55.377 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:54:55 GMT /pklxkd-59/?versions 2023-05-31T20:54:55.378 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:4h7JJTjcnp3JcdxhS9xudNP+BDw= 2023-05-31T20:54:55.378 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:54:55 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:4h7JJTjcnp3JcdxhS9xudNP+BDw=', 'Content-Length': '0'} 2023-05-31T20:54:55.381 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b25b43fec22fe8d9-006477b41f-1e2e-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:54:55 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:54:55.381 DEBUG:boto:b'pklxkd-59999falseobj11nulltrue2023-05-31T20:39:26.427Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj111nulltrue2023-05-31T20:52:14.993Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj115nulltrue2023-05-31T20:46:33.958Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj116nulltrue2023-05-31T20:48:13.052Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj124nulltrue2023-05-31T20:50:33.587Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj145nulltrue2023-05-31T20:48:13.107Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj148nulltrue2023-05-31T20:52:15.015Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj155nulltrue2023-05-31T20:50:33.605Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj17nulltrue2023-05-31T20:43:07.977Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj185nulltrue2023-05-31T20:52:15.072Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj19nulltrue2023-05-31T20:44:50.557Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj22nulltrue2023-05-31T20:39:26.473Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj23nulltrue2023-05-31T20:46:33.813Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj29nulltrue2023-05-31T20:48:12.956Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj31nulltrue2023-05-31T20:50:33.460Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj33nulltrue2023-05-31T20:39:26.534Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj34nulltrue2023-05-31T20:43:07.994Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj37nulltrue2023-05-31T20:52:14.909Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj38nulltrue2023-05-31T20:44:50.610Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj44nulltrue2023-05-31T20:39:26.565Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj46nulltrue2023-05-31T20:46:33.830Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj51nulltrue2023-05-31T20:43:08.011Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj55nulltrue2023-05-31T20:39:26.630Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj57nulltrue2023-05-31T20:44:50.627Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj58nulltrue2023-05-31T20:48:12.976Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj62nulltrue2023-05-31T20:50:33.515Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj68nulltrue2023-05-31T20:43:08.066Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj69nulltrue2023-05-31T20:46:33.886Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj74nulltrue2023-05-31T20:52:14.934Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj76nulltrue2023-05-31T20:44:50.682Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj85nulltrue2023-05-31T20:43:08.086Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj87nulltrue2023-05-31T20:48:13.031Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj92nulltrue2023-05-31T20:46:33.904Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj93nulltrue2023-05-31T20:50:33.533Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj95nulltrue2023-05-31T20:44:50.705Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:54:55.383 DEBUG:tasks.rgw_multi.zone_rados:o=obj11 2023-05-31T20:54:55.384 DEBUG:tasks.rgw_multi.zone_rados:o=obj111 2023-05-31T20:54:55.384 DEBUG:tasks.rgw_multi.zone_rados:o=obj115 2023-05-31T20:54:55.384 DEBUG:tasks.rgw_multi.zone_rados:o=obj116 2023-05-31T20:54:55.385 DEBUG:tasks.rgw_multi.zone_rados:o=obj124 2023-05-31T20:54:55.385 DEBUG:tasks.rgw_multi.zone_rados:o=obj145 2023-05-31T20:54:55.385 DEBUG:tasks.rgw_multi.zone_rados:o=obj148 2023-05-31T20:54:55.385 DEBUG:tasks.rgw_multi.zone_rados:o=obj155 2023-05-31T20:54:55.386 DEBUG:tasks.rgw_multi.zone_rados:o=obj17 2023-05-31T20:54:55.386 DEBUG:tasks.rgw_multi.zone_rados:o=obj185 2023-05-31T20:54:55.386 DEBUG:tasks.rgw_multi.zone_rados:o=obj19 2023-05-31T20:54:55.387 DEBUG:tasks.rgw_multi.zone_rados:o=obj22 2023-05-31T20:54:55.387 DEBUG:tasks.rgw_multi.zone_rados:o=obj23 2023-05-31T20:54:55.387 DEBUG:tasks.rgw_multi.zone_rados:o=obj29 2023-05-31T20:54:55.387 DEBUG:tasks.rgw_multi.zone_rados:o=obj31 2023-05-31T20:54:55.388 DEBUG:tasks.rgw_multi.zone_rados:o=obj33 2023-05-31T20:54:55.388 DEBUG:tasks.rgw_multi.zone_rados:o=obj34 2023-05-31T20:54:55.388 DEBUG:tasks.rgw_multi.zone_rados:o=obj37 2023-05-31T20:54:55.389 DEBUG:tasks.rgw_multi.zone_rados:o=obj38 2023-05-31T20:54:55.389 DEBUG:tasks.rgw_multi.zone_rados:o=obj44 2023-05-31T20:54:55.389 DEBUG:tasks.rgw_multi.zone_rados:o=obj46 2023-05-31T20:54:55.389 DEBUG:tasks.rgw_multi.zone_rados:o=obj51 2023-05-31T20:54:55.390 DEBUG:tasks.rgw_multi.zone_rados:o=obj55 2023-05-31T20:54:55.390 DEBUG:tasks.rgw_multi.zone_rados:o=obj57 2023-05-31T20:54:55.390 DEBUG:tasks.rgw_multi.zone_rados:o=obj58 2023-05-31T20:54:55.391 DEBUG:tasks.rgw_multi.zone_rados:o=obj62 2023-05-31T20:54:55.391 DEBUG:tasks.rgw_multi.zone_rados:o=obj68 2023-05-31T20:54:55.391 DEBUG:tasks.rgw_multi.zone_rados:o=obj69 2023-05-31T20:54:55.392 DEBUG:tasks.rgw_multi.zone_rados:o=obj74 2023-05-31T20:54:55.392 DEBUG:tasks.rgw_multi.zone_rados:o=obj76 2023-05-31T20:54:55.392 DEBUG:tasks.rgw_multi.zone_rados:o=obj85 2023-05-31T20:54:55.393 DEBUG:tasks.rgw_multi.zone_rados:o=obj87 2023-05-31T20:54:55.393 DEBUG:tasks.rgw_multi.zone_rados:o=obj92 2023-05-31T20:54:55.393 DEBUG:tasks.rgw_multi.zone_rados:o=obj93 2023-05-31T20:54:55.394 DEBUG:tasks.rgw_multi.zone_rados:o=obj95 2023-05-31T20:54:55.394 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2023-05-31T20:54:55.394 DEBUG:boto:path=/pklxkd-59/ 2023-05-31T20:54:55.395 DEBUG:boto:auth_path=/pklxkd-59/ 2023-05-31T20:54:55.395 DEBUG:boto:path=/pklxkd-59/?versions&max-keys=999 2023-05-31T20:54:55.395 DEBUG:boto:auth_path=/pklxkd-59/?versions&max-keys=999 2023-05-31T20:54:55.396 DEBUG:boto:Method: GET 2023-05-31T20:54:55.396 DEBUG:boto:Path: /pklxkd-59/?versions&max-keys=999 2023-05-31T20:54:55.396 DEBUG:boto:Data: 2023-05-31T20:54:55.397 DEBUG:boto:Headers: {} 2023-05-31T20:54:55.397 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:54:55.397 DEBUG:boto:Port: 8000 2023-05-31T20:54:55.398 DEBUG:boto:Params: {} 2023-05-31T20:54:55.398 DEBUG:boto:Token: None 2023-05-31T20:54:55.399 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:54:55 GMT /pklxkd-59/?versions 2023-05-31T20:54:55.399 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:4h7JJTjcnp3JcdxhS9xudNP+BDw= 2023-05-31T20:54:55.399 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:54:55 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:4h7JJTjcnp3JcdxhS9xudNP+BDw=', 'Content-Length': '0'} 2023-05-31T20:54:55.403 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000006e7900edda75c85b-006477b41f-1630-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:54:55 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:54:55.404 DEBUG:boto:b'pklxkd-59999falseobj11nulltrue2023-05-31T20:39:26.427Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj111nulltrue2023-05-31T20:52:14.993Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj115nulltrue2023-05-31T20:46:33.958Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj116nulltrue2023-05-31T20:48:13.052Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj124nulltrue2023-05-31T20:50:33.587Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj145nulltrue2023-05-31T20:48:13.107Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj148nulltrue2023-05-31T20:52:15.015Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj155nulltrue2023-05-31T20:50:33.605Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj17nulltrue2023-05-31T20:43:07.977Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj185nulltrue2023-05-31T20:52:15.072Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj19nulltrue2023-05-31T20:44:50.557Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj22nulltrue2023-05-31T20:39:26.473Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj23nulltrue2023-05-31T20:46:33.813Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj29nulltrue2023-05-31T20:48:12.956Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj31nulltrue2023-05-31T20:50:33.460Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj33nulltrue2023-05-31T20:39:26.534Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj34nulltrue2023-05-31T20:43:07.994Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj37nulltrue2023-05-31T20:52:14.909Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj38nulltrue2023-05-31T20:44:50.610Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj44nulltrue2023-05-31T20:39:26.565Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj46nulltrue2023-05-31T20:46:33.830Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj51nulltrue2023-05-31T20:43:08.011Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj55nulltrue2023-05-31T20:39:26.630Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj57nulltrue2023-05-31T20:44:50.627Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj58nulltrue2023-05-31T20:48:12.976Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj62nulltrue2023-05-31T20:50:33.515Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj68nulltrue2023-05-31T20:43:08.066Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj69nulltrue2023-05-31T20:46:33.886Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj74nulltrue2023-05-31T20:52:14.934Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj76nulltrue2023-05-31T20:44:50.682Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj85nulltrue2023-05-31T20:43:08.086Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj87nulltrue2023-05-31T20:48:13.031Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj92nulltrue2023-05-31T20:46:33.904Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj93nulltrue2023-05-31T20:50:33.533Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj95nulltrue2023-05-31T20:44:50.705Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:54:55.406 DEBUG:tasks.rgw_multi.zone_rados:o=obj11 2023-05-31T20:54:55.406 DEBUG:tasks.rgw_multi.zone_rados:o=obj111 2023-05-31T20:54:55.407 DEBUG:tasks.rgw_multi.zone_rados:o=obj115 2023-05-31T20:54:55.407 DEBUG:tasks.rgw_multi.zone_rados:o=obj116 2023-05-31T20:54:55.407 DEBUG:tasks.rgw_multi.zone_rados:o=obj124 2023-05-31T20:54:55.407 DEBUG:tasks.rgw_multi.zone_rados:o=obj145 2023-05-31T20:54:55.408 DEBUG:tasks.rgw_multi.zone_rados:o=obj148 2023-05-31T20:54:55.408 DEBUG:tasks.rgw_multi.zone_rados:o=obj155 2023-05-31T20:54:55.408 DEBUG:tasks.rgw_multi.zone_rados:o=obj17 2023-05-31T20:54:55.409 DEBUG:tasks.rgw_multi.zone_rados:o=obj185 2023-05-31T20:54:55.409 DEBUG:tasks.rgw_multi.zone_rados:o=obj19 2023-05-31T20:54:55.409 DEBUG:tasks.rgw_multi.zone_rados:o=obj22 2023-05-31T20:54:55.410 DEBUG:tasks.rgw_multi.zone_rados:o=obj23 2023-05-31T20:54:55.410 DEBUG:tasks.rgw_multi.zone_rados:o=obj29 2023-05-31T20:54:55.410 DEBUG:tasks.rgw_multi.zone_rados:o=obj31 2023-05-31T20:54:55.410 DEBUG:tasks.rgw_multi.zone_rados:o=obj33 2023-05-31T20:54:55.411 DEBUG:tasks.rgw_multi.zone_rados:o=obj34 2023-05-31T20:54:55.411 DEBUG:tasks.rgw_multi.zone_rados:o=obj37 2023-05-31T20:54:55.411 DEBUG:tasks.rgw_multi.zone_rados:o=obj38 2023-05-31T20:54:55.412 DEBUG:tasks.rgw_multi.zone_rados:o=obj44 2023-05-31T20:54:55.412 DEBUG:tasks.rgw_multi.zone_rados:o=obj46 2023-05-31T20:54:55.412 DEBUG:tasks.rgw_multi.zone_rados:o=obj51 2023-05-31T20:54:55.412 DEBUG:tasks.rgw_multi.zone_rados:o=obj55 2023-05-31T20:54:55.413 DEBUG:tasks.rgw_multi.zone_rados:o=obj57 2023-05-31T20:54:55.413 DEBUG:tasks.rgw_multi.zone_rados:o=obj58 2023-05-31T20:54:55.413 DEBUG:tasks.rgw_multi.zone_rados:o=obj62 2023-05-31T20:54:55.414 DEBUG:tasks.rgw_multi.zone_rados:o=obj68 2023-05-31T20:54:55.414 DEBUG:tasks.rgw_multi.zone_rados:o=obj69 2023-05-31T20:54:55.414 DEBUG:tasks.rgw_multi.zone_rados:o=obj74 2023-05-31T20:54:55.414 DEBUG:tasks.rgw_multi.zone_rados:o=obj76 2023-05-31T20:54:55.415 DEBUG:tasks.rgw_multi.zone_rados:o=obj85 2023-05-31T20:54:55.415 DEBUG:tasks.rgw_multi.zone_rados:o=obj87 2023-05-31T20:54:55.415 DEBUG:tasks.rgw_multi.zone_rados:o=obj92 2023-05-31T20:54:55.416 DEBUG:tasks.rgw_multi.zone_rados:o=obj93 2023-05-31T20:54:55.416 DEBUG:tasks.rgw_multi.zone_rados:o=obj95 2023-05-31T20:54:55.416 DEBUG:boto:path=/pklxkd-59/ 2023-05-31T20:54:55.417 DEBUG:boto:auth_path=/pklxkd-59/ 2023-05-31T20:54:55.417 DEBUG:boto:path=/pklxkd-59/?versions&max-keys=999 2023-05-31T20:54:55.417 DEBUG:boto:auth_path=/pklxkd-59/?versions&max-keys=999 2023-05-31T20:54:55.418 DEBUG:boto:Method: GET 2023-05-31T20:54:55.418 DEBUG:boto:Path: /pklxkd-59/?versions&max-keys=999 2023-05-31T20:54:55.418 DEBUG:boto:Data: 2023-05-31T20:54:55.419 DEBUG:boto:Headers: {} 2023-05-31T20:54:55.419 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:54:55.419 DEBUG:boto:Port: 8001 2023-05-31T20:54:55.420 DEBUG:boto:Params: {} 2023-05-31T20:54:55.420 DEBUG:boto:Token: None 2023-05-31T20:54:55.420 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:54:55 GMT /pklxkd-59/?versions 2023-05-31T20:54:55.421 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:4h7JJTjcnp3JcdxhS9xudNP+BDw= 2023-05-31T20:54:55.421 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:54:55 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:4h7JJTjcnp3JcdxhS9xudNP+BDw=', 'Content-Length': '0'} 2023-05-31T20:54:55.424 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b4f6898df7aa33db-006477b41f-1e2e-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:54:55 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:54:55.424 DEBUG:boto:b'pklxkd-59999falseobj11nulltrue2023-05-31T20:39:26.427Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj111nulltrue2023-05-31T20:52:14.993Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj115nulltrue2023-05-31T20:46:33.958Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj116nulltrue2023-05-31T20:48:13.052Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj124nulltrue2023-05-31T20:50:33.587Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj145nulltrue2023-05-31T20:48:13.107Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj148nulltrue2023-05-31T20:52:15.015Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj155nulltrue2023-05-31T20:50:33.605Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj17nulltrue2023-05-31T20:43:07.977Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj185nulltrue2023-05-31T20:52:15.072Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj19nulltrue2023-05-31T20:44:50.557Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj22nulltrue2023-05-31T20:39:26.473Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj23nulltrue2023-05-31T20:46:33.813Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj29nulltrue2023-05-31T20:48:12.956Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj31nulltrue2023-05-31T20:50:33.460Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj33nulltrue2023-05-31T20:39:26.534Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj34nulltrue2023-05-31T20:43:07.994Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj37nulltrue2023-05-31T20:52:14.909Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj38nulltrue2023-05-31T20:44:50.610Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj44nulltrue2023-05-31T20:39:26.565Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj46nulltrue2023-05-31T20:46:33.830Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj51nulltrue2023-05-31T20:43:08.011Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj55nulltrue2023-05-31T20:39:26.630Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj57nulltrue2023-05-31T20:44:50.627Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj58nulltrue2023-05-31T20:48:12.976Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj62nulltrue2023-05-31T20:50:33.515Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj68nulltrue2023-05-31T20:43:08.066Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj69nulltrue2023-05-31T20:46:33.886Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj74nulltrue2023-05-31T20:52:14.934Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj76nulltrue2023-05-31T20:44:50.682Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj85nulltrue2023-05-31T20:43:08.086Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj87nulltrue2023-05-31T20:48:13.031Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj92nulltrue2023-05-31T20:46:33.904Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj93nulltrue2023-05-31T20:50:33.533Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj95nulltrue2023-05-31T20:44:50.705Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:54:55.426 DEBUG:boto:path=/pklxkd-59/ 2023-05-31T20:54:55.427 DEBUG:boto:auth_path=/pklxkd-59/ 2023-05-31T20:54:55.427 DEBUG:boto:path=/pklxkd-59/?versions&max-keys=999 2023-05-31T20:54:55.427 DEBUG:boto:auth_path=/pklxkd-59/?versions&max-keys=999 2023-05-31T20:54:55.428 DEBUG:boto:Method: GET 2023-05-31T20:54:55.428 DEBUG:boto:Path: /pklxkd-59/?versions&max-keys=999 2023-05-31T20:54:55.428 DEBUG:boto:Data: 2023-05-31T20:54:55.429 DEBUG:boto:Headers: {} 2023-05-31T20:54:55.429 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:54:55.429 DEBUG:boto:Port: 8000 2023-05-31T20:54:55.430 DEBUG:boto:Params: {} 2023-05-31T20:54:55.430 DEBUG:boto:Token: None 2023-05-31T20:54:55.431 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:54:55 GMT /pklxkd-59/?versions 2023-05-31T20:54:55.431 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:4h7JJTjcnp3JcdxhS9xudNP+BDw= 2023-05-31T20:54:55.431 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:54:55 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:4h7JJTjcnp3JcdxhS9xudNP+BDw=', 'Content-Length': '0'} 2023-05-31T20:54:55.435 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000839524065a8dce3e-006477b41f-1630-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:54:55 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:54:55.436 DEBUG:boto:b'pklxkd-59999falseobj11nulltrue2023-05-31T20:39:26.427Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj111nulltrue2023-05-31T20:52:14.993Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj115nulltrue2023-05-31T20:46:33.958Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj116nulltrue2023-05-31T20:48:13.052Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj124nulltrue2023-05-31T20:50:33.587Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj145nulltrue2023-05-31T20:48:13.107Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj148nulltrue2023-05-31T20:52:15.015Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj155nulltrue2023-05-31T20:50:33.605Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj17nulltrue2023-05-31T20:43:07.977Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj185nulltrue2023-05-31T20:52:15.072Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj19nulltrue2023-05-31T20:44:50.557Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj22nulltrue2023-05-31T20:39:26.473Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj23nulltrue2023-05-31T20:46:33.813Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj29nulltrue2023-05-31T20:48:12.956Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj31nulltrue2023-05-31T20:50:33.460Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj33nulltrue2023-05-31T20:39:26.534Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj34nulltrue2023-05-31T20:43:07.994Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj37nulltrue2023-05-31T20:52:14.909Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj38nulltrue2023-05-31T20:44:50.610Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj44nulltrue2023-05-31T20:39:26.565Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj46nulltrue2023-05-31T20:46:33.830Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj51nulltrue2023-05-31T20:43:08.011Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj55nulltrue2023-05-31T20:39:26.630Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj57nulltrue2023-05-31T20:44:50.627Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj58nulltrue2023-05-31T20:48:12.976Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj62nulltrue2023-05-31T20:50:33.515Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj68nulltrue2023-05-31T20:43:08.066Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj69nulltrue2023-05-31T20:46:33.886Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj74nulltrue2023-05-31T20:52:14.934Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj76nulltrue2023-05-31T20:44:50.682Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj85nulltrue2023-05-31T20:43:08.086Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj87nulltrue2023-05-31T20:48:13.031Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj92nulltrue2023-05-31T20:46:33.904Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj93nulltrue2023-05-31T20:50:33.533Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj95nulltrue2023-05-31T20:44:50.705Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:54:55.438 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj11 2023-05-31T20:54:55.438 DEBUG:boto:path=/pklxkd-59/obj11 2023-05-31T20:54:55.439 DEBUG:boto:auth_path=/pklxkd-59/obj11 2023-05-31T20:54:55.439 DEBUG:boto:path=/pklxkd-59/obj11?versionId=null 2023-05-31T20:54:55.439 DEBUG:boto:auth_path=/pklxkd-59/obj11?versionId=null 2023-05-31T20:54:55.440 DEBUG:boto:Method: GET 2023-05-31T20:54:55.440 DEBUG:boto:Path: /pklxkd-59/obj11?versionId=null 2023-05-31T20:54:55.440 DEBUG:boto:Data: 2023-05-31T20:54:55.441 DEBUG:boto:Headers: {} 2023-05-31T20:54:55.441 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:54:55.442 DEBUG:boto:Port: 8001 2023-05-31T20:54:55.442 DEBUG:boto:Params: {} 2023-05-31T20:54:55.442 DEBUG:boto:Token: None 2023-05-31T20:54:55.443 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:54:55 GMT /pklxkd-59/obj11?versionId=null 2023-05-31T20:54:55.443 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:Ojadgnp4g1MC/Yu8s8ZWSEdeqhs= 2023-05-31T20:54:55.444 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:54:55 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:Ojadgnp4g1MC/Yu8s8ZWSEdeqhs=', 'Content-Length': '0'} 2023-05-31T20:54:55.446 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:39:26 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ae4019002b54f9ec-006477b41f-1e2e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:54:55 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:54:55.447 DEBUG:boto:path=/pklxkd-59/obj11 2023-05-31T20:54:55.447 DEBUG:boto:auth_path=/pklxkd-59/obj11 2023-05-31T20:54:55.448 DEBUG:boto:path=/pklxkd-59/obj11?versionId=null 2023-05-31T20:54:55.448 DEBUG:boto:auth_path=/pklxkd-59/obj11?versionId=null 2023-05-31T20:54:55.448 DEBUG:boto:Method: GET 2023-05-31T20:54:55.449 DEBUG:boto:Path: /pklxkd-59/obj11?versionId=null 2023-05-31T20:54:55.449 DEBUG:boto:Data: 2023-05-31T20:54:55.449 DEBUG:boto:Headers: {} 2023-05-31T20:54:55.450 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:54:55.450 DEBUG:boto:Port: 8000 2023-05-31T20:54:55.450 DEBUG:boto:Params: {} 2023-05-31T20:54:55.451 DEBUG:boto:Token: None 2023-05-31T20:54:55.451 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:54:55 GMT /pklxkd-59/obj11?versionId=null 2023-05-31T20:54:55.452 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:Ojadgnp4g1MC/Yu8s8ZWSEdeqhs= 2023-05-31T20:54:55.452 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:54:55 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:Ojadgnp4g1MC/Yu8s8ZWSEdeqhs=', 'Content-Length': '0'} 2023-05-31T20:54:55.454 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:39:26 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000063fc596188b0697c-006477b41f-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:54:55 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:54:55.496 DEBUG:boto:path=/pklxkd-59/obj11 2023-05-31T20:54:55.497 DEBUG:boto:auth_path=/pklxkd-59/obj11 2023-05-31T20:54:55.497 DEBUG:boto:path=/pklxkd-59/obj11?versionId=null 2023-05-31T20:54:55.497 DEBUG:boto:auth_path=/pklxkd-59/obj11?versionId=null 2023-05-31T20:54:55.498 DEBUG:boto:Method: HEAD 2023-05-31T20:54:55.498 DEBUG:boto:Path: /pklxkd-59/obj11?versionId=null 2023-05-31T20:54:55.498 DEBUG:boto:Data: 2023-05-31T20:54:55.499 DEBUG:boto:Headers: {} 2023-05-31T20:54:55.499 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:54:55.499 DEBUG:boto:Port: 8001 2023-05-31T20:54:55.500 DEBUG:boto:Params: {} 2023-05-31T20:54:55.500 DEBUG:boto:Token: None 2023-05-31T20:54:55.500 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:54:55 GMT /pklxkd-59/obj11?versionId=null 2023-05-31T20:54:55.501 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:ZhmqfgD7K3D+W8eMd1fE2CPedgw= 2023-05-31T20:54:55.501 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:54:55 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:ZhmqfgD7K3D+W8eMd1fE2CPedgw=', 'Content-Length': '0'} 2023-05-31T20:54:55.503 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:39:26 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000830ccebeef819087-006477b41f-1e2e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:54:55 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:54:55.504 DEBUG:boto:path=/pklxkd-59/obj11 2023-05-31T20:54:55.504 DEBUG:boto:auth_path=/pklxkd-59/obj11 2023-05-31T20:54:55.504 DEBUG:boto:path=/pklxkd-59/obj11?versionId=null 2023-05-31T20:54:55.505 DEBUG:boto:auth_path=/pklxkd-59/obj11?versionId=null 2023-05-31T20:54:55.505 DEBUG:boto:Method: HEAD 2023-05-31T20:54:55.505 DEBUG:boto:Path: /pklxkd-59/obj11?versionId=null 2023-05-31T20:54:55.506 DEBUG:boto:Data: 2023-05-31T20:54:55.506 DEBUG:boto:Headers: {} 2023-05-31T20:54:55.507 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:54:55.507 DEBUG:boto:Port: 8000 2023-05-31T20:54:55.507 DEBUG:boto:Params: {} 2023-05-31T20:54:55.508 DEBUG:boto:Token: None 2023-05-31T20:54:55.508 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:54:55 GMT /pklxkd-59/obj11?versionId=null 2023-05-31T20:54:55.508 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:ZhmqfgD7K3D+W8eMd1fE2CPedgw= 2023-05-31T20:54:55.509 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:54:55 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:ZhmqfgD7K3D+W8eMd1fE2CPedgw=', 'Content-Length': '0'} 2023-05-31T20:54:55.510 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:39:26 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000068c27db0ce8b5a37-006477b41f-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:54:55 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:54:55.511 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj11 2023-05-31T20:54:55.511 DEBUG:boto:path=/pklxkd-59/obj11 2023-05-31T20:54:55.512 DEBUG:boto:auth_path=/pklxkd-59/obj11 2023-05-31T20:54:55.512 DEBUG:boto:path=/pklxkd-59/obj11?versionId=null 2023-05-31T20:54:55.512 DEBUG:boto:auth_path=/pklxkd-59/obj11?versionId=null 2023-05-31T20:54:55.513 DEBUG:boto:Method: GET 2023-05-31T20:54:55.513 DEBUG:boto:Path: /pklxkd-59/obj11?versionId=null 2023-05-31T20:54:55.513 DEBUG:boto:Data: 2023-05-31T20:54:55.514 DEBUG:boto:Headers: {} 2023-05-31T20:54:55.514 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:54:55.514 DEBUG:boto:Port: 8001 2023-05-31T20:54:55.515 DEBUG:boto:Params: {} 2023-05-31T20:54:55.515 DEBUG:boto:Token: None 2023-05-31T20:54:55.516 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:54:55 GMT /pklxkd-59/obj11?versionId=null 2023-05-31T20:54:55.516 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:Ojadgnp4g1MC/Yu8s8ZWSEdeqhs= 2023-05-31T20:54:55.516 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:54:55 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:Ojadgnp4g1MC/Yu8s8ZWSEdeqhs=', 'Content-Length': '0'} 2023-05-31T20:54:55.518 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:39:26 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000da88ce9e2d4a45f7-006477b41f-1e2e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:54:55 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:54:55.519 DEBUG:boto:path=/pklxkd-59/obj11 2023-05-31T20:54:55.519 DEBUG:boto:auth_path=/pklxkd-59/obj11 2023-05-31T20:54:55.520 DEBUG:boto:path=/pklxkd-59/obj11?versionId=null 2023-05-31T20:54:55.520 DEBUG:boto:auth_path=/pklxkd-59/obj11?versionId=null 2023-05-31T20:54:55.520 DEBUG:boto:Method: GET 2023-05-31T20:54:55.521 DEBUG:boto:Path: /pklxkd-59/obj11?versionId=null 2023-05-31T20:54:55.521 DEBUG:boto:Data: 2023-05-31T20:54:55.521 DEBUG:boto:Headers: {} 2023-05-31T20:54:55.522 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:54:55.522 DEBUG:boto:Port: 8000 2023-05-31T20:54:55.522 DEBUG:boto:Params: {} 2023-05-31T20:54:55.523 DEBUG:boto:Token: None 2023-05-31T20:54:55.523 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:54:55 GMT /pklxkd-59/obj11?versionId=null 2023-05-31T20:54:55.524 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:Ojadgnp4g1MC/Yu8s8ZWSEdeqhs= 2023-05-31T20:54:55.524 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:54:55 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:Ojadgnp4g1MC/Yu8s8ZWSEdeqhs=', 'Content-Length': '0'} 2023-05-31T20:54:55.526 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:39:26 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000058cec186f8cd67b6-006477b41f-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:54:55 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:54:55.526 DEBUG:boto:path=/pklxkd-59/ 2023-05-31T20:54:55.527 DEBUG:boto:auth_path=/pklxkd-59/ 2023-05-31T20:54:55.527 DEBUG:boto:path=/pklxkd-59/?prefix=obj11 2023-05-31T20:54:55.527 DEBUG:boto:auth_path=/pklxkd-59/?prefix=obj11 2023-05-31T20:54:55.528 DEBUG:boto:Method: GET 2023-05-31T20:54:55.528 DEBUG:boto:Path: /pklxkd-59/?prefix=obj11 2023-05-31T20:54:55.529 DEBUG:boto:Data: 2023-05-31T20:54:55.529 DEBUG:boto:Headers: {} 2023-05-31T20:54:55.529 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:54:55.530 DEBUG:boto:Port: 8001 2023-05-31T20:54:55.530 DEBUG:boto:Params: {} 2023-05-31T20:54:55.530 DEBUG:boto:Token: None 2023-05-31T20:54:55.531 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:54:55 GMT /pklxkd-59/ 2023-05-31T20:54:55.531 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:RywX8LlH8rRUpedRaKmQfbksV7k= 2023-05-31T20:54:55.532 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:54:55 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:RywX8LlH8rRUpedRaKmQfbksV7k=', 'Content-Length': '0'} 2023-05-31T20:54:55.534 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000006f8c7d65a2c8ea80-006477b41f-1e2e-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:54:55 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:54:55.534 DEBUG:boto:b'pklxkd-59obj111000falseobj112023-05-31T20:39:26.427Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj1112023-05-31T20:52:14.993Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj1152023-05-31T20:46:33.958Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj1162023-05-31T20:48:13.052Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:54:55.535 DEBUG:boto:path=/pklxkd-59/ 2023-05-31T20:54:55.535 DEBUG:boto:auth_path=/pklxkd-59/ 2023-05-31T20:54:55.536 DEBUG:boto:path=/pklxkd-59/?prefix=obj11 2023-05-31T20:54:55.536 DEBUG:boto:auth_path=/pklxkd-59/?prefix=obj11 2023-05-31T20:54:55.536 DEBUG:boto:Method: GET 2023-05-31T20:54:55.537 DEBUG:boto:Path: /pklxkd-59/?prefix=obj11 2023-05-31T20:54:55.538 DEBUG:boto:Data: 2023-05-31T20:54:55.538 DEBUG:boto:Headers: {} 2023-05-31T20:54:55.539 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:54:55.539 DEBUG:boto:Port: 8000 2023-05-31T20:54:55.540 DEBUG:boto:Params: {} 2023-05-31T20:54:55.540 DEBUG:boto:Token: None 2023-05-31T20:54:55.540 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:54:55 GMT /pklxkd-59/ 2023-05-31T20:54:55.541 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:RywX8LlH8rRUpedRaKmQfbksV7k= 2023-05-31T20:54:55.541 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:54:55 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:RywX8LlH8rRUpedRaKmQfbksV7k=', 'Content-Length': '0'} 2023-05-31T20:54:55.545 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000002e2ae4a24b65f120-006477b41f-1630-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:54:55 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:54:55.546 DEBUG:boto:b'pklxkd-59obj111000falseobj112023-05-31T20:39:26.427Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj1112023-05-31T20:52:14.993Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj1152023-05-31T20:46:33.958Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj1162023-05-31T20:48:13.052Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:54:55.546 DEBUG:boto:path=/pklxkd-59/obj11 2023-05-31T20:54:55.547 DEBUG:boto:auth_path=/pklxkd-59/obj11 2023-05-31T20:54:55.547 DEBUG:boto:Method: HEAD 2023-05-31T20:54:55.547 DEBUG:boto:Path: /pklxkd-59/obj11 2023-05-31T20:54:55.548 DEBUG:boto:Data: 2023-05-31T20:54:55.548 DEBUG:boto:Headers: {} 2023-05-31T20:54:55.548 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:54:55.549 DEBUG:boto:Port: 8001 2023-05-31T20:54:55.549 DEBUG:boto:Params: {} 2023-05-31T20:54:55.549 DEBUG:boto:Token: None 2023-05-31T20:54:55.550 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:54:55 GMT /pklxkd-59/obj11 2023-05-31T20:54:55.550 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:4sYmcKTULmHb8ek4tNDySAQfU20= 2023-05-31T20:54:55.551 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:54:55 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:4sYmcKTULmHb8ek4tNDySAQfU20=', 'Content-Length': '0'} 2023-05-31T20:54:55.552 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:39:26 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004ce7874db52b1a61-006477b41f-1e2e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:54:55 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:54:55.553 DEBUG:boto:path=/pklxkd-59/obj11 2023-05-31T20:54:55.553 DEBUG:boto:auth_path=/pklxkd-59/obj11 2023-05-31T20:54:55.553 DEBUG:boto:Method: HEAD 2023-05-31T20:54:55.554 DEBUG:boto:Path: /pklxkd-59/obj11 2023-05-31T20:54:55.554 DEBUG:boto:Data: 2023-05-31T20:54:55.554 DEBUG:boto:Headers: {} 2023-05-31T20:54:55.555 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:54:55.555 DEBUG:boto:Port: 8000 2023-05-31T20:54:55.555 DEBUG:boto:Params: {} 2023-05-31T20:54:55.556 DEBUG:boto:Token: None 2023-05-31T20:54:55.556 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:54:55 GMT /pklxkd-59/obj11 2023-05-31T20:54:55.557 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:4sYmcKTULmHb8ek4tNDySAQfU20= 2023-05-31T20:54:55.557 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:54:55 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:4sYmcKTULmHb8ek4tNDySAQfU20=', 'Content-Length': '0'} 2023-05-31T20:54:55.559 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:39:26 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000074116045f14b6588-006477b41f-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:54:55 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:54:55.559 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj11 2023-05-31T20:54:55.560 DEBUG:boto:path=/pklxkd-59/obj11 2023-05-31T20:54:55.560 DEBUG:boto:auth_path=/pklxkd-59/obj11 2023-05-31T20:54:55.561 DEBUG:boto:Method: GET 2023-05-31T20:54:55.561 DEBUG:boto:Path: /pklxkd-59/obj11 2023-05-31T20:54:55.561 DEBUG:boto:Data: 2023-05-31T20:54:55.562 DEBUG:boto:Headers: {} 2023-05-31T20:54:55.562 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:54:55.562 DEBUG:boto:Port: 8001 2023-05-31T20:54:55.563 DEBUG:boto:Params: {} 2023-05-31T20:54:55.563 DEBUG:boto:Token: None 2023-05-31T20:54:55.563 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:54:55 GMT /pklxkd-59/obj11 2023-05-31T20:54:55.564 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:msP/VN++kR2gtnJUdmcS5CfkdDk= 2023-05-31T20:54:55.564 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:54:55 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:msP/VN++kR2gtnJUdmcS5CfkdDk=', 'Content-Length': '0'} 2023-05-31T20:54:55.566 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:39:26 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f8f874409b526da8-006477b41f-1e2e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:54:55 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:54:55.566 DEBUG:boto:path=/pklxkd-59/obj11 2023-05-31T20:54:55.567 DEBUG:boto:auth_path=/pklxkd-59/obj11 2023-05-31T20:54:55.567 DEBUG:boto:Method: GET 2023-05-31T20:54:55.568 DEBUG:boto:Path: /pklxkd-59/obj11 2023-05-31T20:54:55.568 DEBUG:boto:Data: 2023-05-31T20:54:55.568 DEBUG:boto:Headers: {} 2023-05-31T20:54:55.569 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:54:55.569 DEBUG:boto:Port: 8000 2023-05-31T20:54:55.570 DEBUG:boto:Params: {} 2023-05-31T20:54:55.570 DEBUG:boto:Token: None 2023-05-31T20:54:55.570 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:54:55 GMT /pklxkd-59/obj11 2023-05-31T20:54:55.571 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:msP/VN++kR2gtnJUdmcS5CfkdDk= 2023-05-31T20:54:55.571 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:54:55 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:msP/VN++kR2gtnJUdmcS5CfkdDk=', 'Content-Length': '0'} 2023-05-31T20:54:55.573 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:39:26 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000036916bd730392e10-006477b41f-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:54:55 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:54:55.574 DEBUG:boto:path=/pklxkd-59/ 2023-05-31T20:54:55.574 DEBUG:boto:auth_path=/pklxkd-59/ 2023-05-31T20:54:55.574 DEBUG:boto:path=/pklxkd-59/?prefix=obj11 2023-05-31T20:54:55.575 DEBUG:boto:auth_path=/pklxkd-59/?prefix=obj11 2023-05-31T20:54:55.575 DEBUG:boto:Method: GET 2023-05-31T20:54:55.575 DEBUG:boto:Path: /pklxkd-59/?prefix=obj11 2023-05-31T20:54:55.576 DEBUG:boto:Data: 2023-05-31T20:54:55.576 DEBUG:boto:Headers: {} 2023-05-31T20:54:55.576 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:54:55.577 DEBUG:boto:Port: 8001 2023-05-31T20:54:55.577 DEBUG:boto:Params: {} 2023-05-31T20:54:55.577 DEBUG:boto:Token: None 2023-05-31T20:54:55.578 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:54:55 GMT /pklxkd-59/ 2023-05-31T20:54:55.578 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:RywX8LlH8rRUpedRaKmQfbksV7k= 2023-05-31T20:54:55.579 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:54:55 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:RywX8LlH8rRUpedRaKmQfbksV7k=', 'Content-Length': '0'} 2023-05-31T20:54:55.581 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d675a541b195f78c-006477b41f-1e2e-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:54:55 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:54:55.581 DEBUG:boto:b'pklxkd-59obj111000falseobj112023-05-31T20:39:26.427Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj1112023-05-31T20:52:14.993Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj1152023-05-31T20:46:33.958Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj1162023-05-31T20:48:13.052Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:54:55.582 DEBUG:boto:path=/pklxkd-59/ 2023-05-31T20:54:55.583 DEBUG:boto:auth_path=/pklxkd-59/ 2023-05-31T20:54:55.583 DEBUG:boto:path=/pklxkd-59/?prefix=obj11 2023-05-31T20:54:55.583 DEBUG:boto:auth_path=/pklxkd-59/?prefix=obj11 2023-05-31T20:54:55.584 DEBUG:boto:Method: GET 2023-05-31T20:54:55.584 DEBUG:boto:Path: /pklxkd-59/?prefix=obj11 2023-05-31T20:54:55.584 DEBUG:boto:Data: 2023-05-31T20:54:55.585 DEBUG:boto:Headers: {} 2023-05-31T20:54:55.585 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:54:55.585 DEBUG:boto:Port: 8000 2023-05-31T20:54:55.586 DEBUG:boto:Params: {} 2023-05-31T20:54:55.586 DEBUG:boto:Token: None 2023-05-31T20:54:55.587 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:54:55 GMT /pklxkd-59/ 2023-05-31T20:54:55.587 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:RywX8LlH8rRUpedRaKmQfbksV7k= 2023-05-31T20:54:55.587 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:54:55 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:RywX8LlH8rRUpedRaKmQfbksV7k=', 'Content-Length': '0'} 2023-05-31T20:54:55.591 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000002d2e8bf6e1148c0a-006477b41f-1630-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:54:55 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:54:55.592 DEBUG:boto:b'pklxkd-59obj111000falseobj112023-05-31T20:39:26.427Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj1112023-05-31T20:52:14.993Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj1152023-05-31T20:46:33.958Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj1162023-05-31T20:48:13.052Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:54:55.592 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj111 2023-05-31T20:54:55.593 DEBUG:boto:path=/pklxkd-59/obj111 2023-05-31T20:54:55.593 DEBUG:boto:auth_path=/pklxkd-59/obj111 2023-05-31T20:54:55.594 DEBUG:boto:path=/pklxkd-59/obj111?versionId=null 2023-05-31T20:54:55.594 DEBUG:boto:auth_path=/pklxkd-59/obj111?versionId=null 2023-05-31T20:54:55.594 DEBUG:boto:Method: GET 2023-05-31T20:54:55.595 DEBUG:boto:Path: /pklxkd-59/obj111?versionId=null 2023-05-31T20:54:55.595 DEBUG:boto:Data: 2023-05-31T20:54:55.595 DEBUG:boto:Headers: {} 2023-05-31T20:54:55.596 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:54:55.596 DEBUG:boto:Port: 8001 2023-05-31T20:54:55.596 DEBUG:boto:Params: {} 2023-05-31T20:54:55.597 DEBUG:boto:Token: None 2023-05-31T20:54:55.597 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:54:55 GMT /pklxkd-59/obj111?versionId=null 2023-05-31T20:54:55.598 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:oAsaCNdwCEgngYJkzMEJIpWm1hY= 2023-05-31T20:54:55.598 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:54:55 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:oAsaCNdwCEgngYJkzMEJIpWm1hY=', 'Content-Length': '0'} 2023-05-31T20:54:55.600 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:52:14 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c0b1f2b442a59a2c-006477b41f-1e2e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:54:55 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:54:55.644 DEBUG:boto:path=/pklxkd-59/obj111 2023-05-31T20:54:55.645 DEBUG:boto:auth_path=/pklxkd-59/obj111 2023-05-31T20:54:55.645 DEBUG:boto:path=/pklxkd-59/obj111?versionId=null 2023-05-31T20:54:55.645 DEBUG:boto:auth_path=/pklxkd-59/obj111?versionId=null 2023-05-31T20:54:55.646 DEBUG:boto:Method: GET 2023-05-31T20:54:55.646 DEBUG:boto:Path: /pklxkd-59/obj111?versionId=null 2023-05-31T20:54:55.646 DEBUG:boto:Data: 2023-05-31T20:54:55.647 DEBUG:boto:Headers: {} 2023-05-31T20:54:55.647 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:54:55.647 DEBUG:boto:Port: 8000 2023-05-31T20:54:55.648 DEBUG:boto:Params: {} 2023-05-31T20:54:55.648 DEBUG:boto:Token: None 2023-05-31T20:54:55.649 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:54:55 GMT /pklxkd-59/obj111?versionId=null 2023-05-31T20:54:55.649 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:oAsaCNdwCEgngYJkzMEJIpWm1hY= 2023-05-31T20:54:55.650 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:54:55 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:oAsaCNdwCEgngYJkzMEJIpWm1hY=', 'Content-Length': '0'} 2023-05-31T20:54:55.651 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:52:14 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000431e9bcc1093b869-006477b41f-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:54:55 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:54:55.652 DEBUG:boto:path=/pklxkd-59/obj111 2023-05-31T20:54:55.652 DEBUG:boto:auth_path=/pklxkd-59/obj111 2023-05-31T20:54:55.653 DEBUG:boto:path=/pklxkd-59/obj111?versionId=null 2023-05-31T20:54:55.653 DEBUG:boto:auth_path=/pklxkd-59/obj111?versionId=null 2023-05-31T20:54:55.653 DEBUG:boto:Method: HEAD 2023-05-31T20:54:55.654 DEBUG:boto:Path: /pklxkd-59/obj111?versionId=null 2023-05-31T20:54:55.654 DEBUG:boto:Data: 2023-05-31T20:54:55.654 DEBUG:boto:Headers: {} 2023-05-31T20:54:55.655 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:54:55.655 DEBUG:boto:Port: 8001 2023-05-31T20:54:55.655 DEBUG:boto:Params: {} 2023-05-31T20:54:55.656 DEBUG:boto:Token: None 2023-05-31T20:54:55.656 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:54:55 GMT /pklxkd-59/obj111?versionId=null 2023-05-31T20:54:55.656 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:d/qzR0XA5hvvmA4wH+FiOeRFQcg= 2023-05-31T20:54:55.657 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:54:55 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:d/qzR0XA5hvvmA4wH+FiOeRFQcg=', 'Content-Length': '0'} 2023-05-31T20:54:55.658 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:52:14 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001878c2d1de9a2174-006477b41f-1e2e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:54:55 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:54:55.659 DEBUG:boto:path=/pklxkd-59/obj111 2023-05-31T20:54:55.659 DEBUG:boto:auth_path=/pklxkd-59/obj111 2023-05-31T20:54:55.660 DEBUG:boto:path=/pklxkd-59/obj111?versionId=null 2023-05-31T20:54:55.660 DEBUG:boto:auth_path=/pklxkd-59/obj111?versionId=null 2023-05-31T20:54:55.660 DEBUG:boto:Method: HEAD 2023-05-31T20:54:55.661 DEBUG:boto:Path: /pklxkd-59/obj111?versionId=null 2023-05-31T20:54:55.661 DEBUG:boto:Data: 2023-05-31T20:54:55.661 DEBUG:boto:Headers: {} 2023-05-31T20:54:55.662 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:54:55.662 DEBUG:boto:Port: 8000 2023-05-31T20:54:55.662 DEBUG:boto:Params: {} 2023-05-31T20:54:55.663 DEBUG:boto:Token: None 2023-05-31T20:54:55.663 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:54:55 GMT /pklxkd-59/obj111?versionId=null 2023-05-31T20:54:55.664 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:d/qzR0XA5hvvmA4wH+FiOeRFQcg= 2023-05-31T20:54:55.664 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:54:55 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:d/qzR0XA5hvvmA4wH+FiOeRFQcg=', 'Content-Length': '0'} 2023-05-31T20:54:55.666 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:52:14 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006164e8dc4a558fbc-006477b41f-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:54:55 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:54:55.666 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj111 2023-05-31T20:54:55.666 DEBUG:boto:path=/pklxkd-59/obj111 2023-05-31T20:54:55.667 DEBUG:boto:auth_path=/pklxkd-59/obj111 2023-05-31T20:54:55.667 DEBUG:boto:path=/pklxkd-59/obj111?versionId=null 2023-05-31T20:54:55.668 DEBUG:boto:auth_path=/pklxkd-59/obj111?versionId=null 2023-05-31T20:54:55.668 DEBUG:boto:Method: GET 2023-05-31T20:54:55.668 DEBUG:boto:Path: /pklxkd-59/obj111?versionId=null 2023-05-31T20:54:55.669 DEBUG:boto:Data: 2023-05-31T20:54:55.669 DEBUG:boto:Headers: {} 2023-05-31T20:54:55.669 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:54:55.670 DEBUG:boto:Port: 8001 2023-05-31T20:54:55.670 DEBUG:boto:Params: {} 2023-05-31T20:54:55.671 DEBUG:boto:Token: None 2023-05-31T20:54:55.671 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:54:55 GMT /pklxkd-59/obj111?versionId=null 2023-05-31T20:54:55.671 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:oAsaCNdwCEgngYJkzMEJIpWm1hY= 2023-05-31T20:54:55.672 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:54:55 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:oAsaCNdwCEgngYJkzMEJIpWm1hY=', 'Content-Length': '0'} 2023-05-31T20:54:55.673 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:52:14 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c32ef963f5359b9a-006477b41f-1e2e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:54:55 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:54:55.674 DEBUG:boto:path=/pklxkd-59/obj111 2023-05-31T20:54:55.674 DEBUG:boto:auth_path=/pklxkd-59/obj111 2023-05-31T20:54:55.675 DEBUG:boto:path=/pklxkd-59/obj111?versionId=null 2023-05-31T20:54:55.675 DEBUG:boto:auth_path=/pklxkd-59/obj111?versionId=null 2023-05-31T20:54:55.676 DEBUG:boto:Method: GET 2023-05-31T20:54:55.676 DEBUG:boto:Path: /pklxkd-59/obj111?versionId=null 2023-05-31T20:54:55.676 DEBUG:boto:Data: 2023-05-31T20:54:55.677 DEBUG:boto:Headers: {} 2023-05-31T20:54:55.677 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:54:55.677 DEBUG:boto:Port: 8000 2023-05-31T20:54:55.678 DEBUG:boto:Params: {} 2023-05-31T20:54:55.678 DEBUG:boto:Token: None 2023-05-31T20:54:55.679 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:54:55 GMT /pklxkd-59/obj111?versionId=null 2023-05-31T20:54:55.679 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:oAsaCNdwCEgngYJkzMEJIpWm1hY= 2023-05-31T20:54:55.679 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:54:55 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:oAsaCNdwCEgngYJkzMEJIpWm1hY=', 'Content-Length': '0'} 2023-05-31T20:54:55.681 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:52:14 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000046bc64cbda0a9479-006477b41f-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:54:55 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:54:55.682 DEBUG:boto:path=/pklxkd-59/ 2023-05-31T20:54:55.682 DEBUG:boto:auth_path=/pklxkd-59/ 2023-05-31T20:54:55.683 DEBUG:boto:path=/pklxkd-59/?prefix=obj111 2023-05-31T20:54:55.683 DEBUG:boto:auth_path=/pklxkd-59/?prefix=obj111 2023-05-31T20:54:55.683 DEBUG:boto:Method: GET 2023-05-31T20:54:55.684 DEBUG:boto:Path: /pklxkd-59/?prefix=obj111 2023-05-31T20:54:55.684 DEBUG:boto:Data: 2023-05-31T20:54:55.684 DEBUG:boto:Headers: {} 2023-05-31T20:54:55.685 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:54:55.685 DEBUG:boto:Port: 8001 2023-05-31T20:54:55.685 DEBUG:boto:Params: {} 2023-05-31T20:54:55.686 DEBUG:boto:Token: None 2023-05-31T20:54:55.686 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:54:55 GMT /pklxkd-59/ 2023-05-31T20:54:55.687 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:RywX8LlH8rRUpedRaKmQfbksV7k= 2023-05-31T20:54:55.687 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:54:55 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:RywX8LlH8rRUpedRaKmQfbksV7k=', 'Content-Length': '0'} 2023-05-31T20:54:55.689 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000071471a2320ef338d-006477b41f-1e2e-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:54:55 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:54:55.690 DEBUG:boto:b'pklxkd-59obj1111000falseobj1112023-05-31T20:52:14.993Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:54:55.690 DEBUG:boto:path=/pklxkd-59/ 2023-05-31T20:54:55.691 DEBUG:boto:auth_path=/pklxkd-59/ 2023-05-31T20:54:55.691 DEBUG:boto:path=/pklxkd-59/?prefix=obj111 2023-05-31T20:54:55.691 DEBUG:boto:auth_path=/pklxkd-59/?prefix=obj111 2023-05-31T20:54:55.692 DEBUG:boto:Method: GET 2023-05-31T20:54:55.692 DEBUG:boto:Path: /pklxkd-59/?prefix=obj111 2023-05-31T20:54:55.693 DEBUG:boto:Data: 2023-05-31T20:54:55.693 DEBUG:boto:Headers: {} 2023-05-31T20:54:55.693 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:54:55.694 DEBUG:boto:Port: 8000 2023-05-31T20:54:55.694 DEBUG:boto:Params: {} 2023-05-31T20:54:55.694 DEBUG:boto:Token: None 2023-05-31T20:54:55.695 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:54:55 GMT /pklxkd-59/ 2023-05-31T20:54:55.695 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:RywX8LlH8rRUpedRaKmQfbksV7k= 2023-05-31T20:54:55.696 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:54:55 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:RywX8LlH8rRUpedRaKmQfbksV7k=', 'Content-Length': '0'} 2023-05-31T20:54:55.699 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e31c5107a3a9fdc2-006477b41f-1630-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:54:55 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:54:55.700 DEBUG:boto:b'pklxkd-59obj1111000falseobj1112023-05-31T20:52:14.993Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:54:55.700 DEBUG:boto:path=/pklxkd-59/obj111 2023-05-31T20:54:55.701 DEBUG:boto:auth_path=/pklxkd-59/obj111 2023-05-31T20:54:55.701 DEBUG:boto:Method: HEAD 2023-05-31T20:54:55.701 DEBUG:boto:Path: /pklxkd-59/obj111 2023-05-31T20:54:55.702 DEBUG:boto:Data: 2023-05-31T20:54:55.702 DEBUG:boto:Headers: {} 2023-05-31T20:54:55.702 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:54:55.703 DEBUG:boto:Port: 8001 2023-05-31T20:54:55.703 DEBUG:boto:Params: {} 2023-05-31T20:54:55.704 DEBUG:boto:Token: None 2023-05-31T20:54:55.704 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:54:55 GMT /pklxkd-59/obj111 2023-05-31T20:54:55.704 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:XZYbetYaPFwSLeCSguK1VjyraVM= 2023-05-31T20:54:55.705 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:54:55 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:XZYbetYaPFwSLeCSguK1VjyraVM=', 'Content-Length': '0'} 2023-05-31T20:54:55.706 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:52:14 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f9d125613c998511-006477b41f-1e2e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:54:55 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:54:55.707 DEBUG:boto:path=/pklxkd-59/obj111 2023-05-31T20:54:55.707 DEBUG:boto:auth_path=/pklxkd-59/obj111 2023-05-31T20:54:55.708 DEBUG:boto:Method: HEAD 2023-05-31T20:54:55.708 DEBUG:boto:Path: /pklxkd-59/obj111 2023-05-31T20:54:55.708 DEBUG:boto:Data: 2023-05-31T20:54:55.709 DEBUG:boto:Headers: {} 2023-05-31T20:54:55.709 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:54:55.709 DEBUG:boto:Port: 8000 2023-05-31T20:54:55.710 DEBUG:boto:Params: {} 2023-05-31T20:54:55.710 DEBUG:boto:Token: None 2023-05-31T20:54:55.710 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:54:55 GMT /pklxkd-59/obj111 2023-05-31T20:54:55.711 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:XZYbetYaPFwSLeCSguK1VjyraVM= 2023-05-31T20:54:55.711 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:54:55 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:XZYbetYaPFwSLeCSguK1VjyraVM=', 'Content-Length': '0'} 2023-05-31T20:54:55.713 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:52:14 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003dfde8945969fb4f-006477b41f-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:54:55 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:54:55.713 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj111 2023-05-31T20:54:55.714 DEBUG:boto:path=/pklxkd-59/obj111 2023-05-31T20:54:55.714 DEBUG:boto:auth_path=/pklxkd-59/obj111 2023-05-31T20:54:55.714 DEBUG:boto:Method: GET 2023-05-31T20:54:55.715 DEBUG:boto:Path: /pklxkd-59/obj111 2023-05-31T20:54:55.715 DEBUG:boto:Data: 2023-05-31T20:54:55.715 DEBUG:boto:Headers: {} 2023-05-31T20:54:55.716 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:54:55.716 DEBUG:boto:Port: 8001 2023-05-31T20:54:55.716 DEBUG:boto:Params: {} 2023-05-31T20:54:55.717 DEBUG:boto:Token: None 2023-05-31T20:54:55.717 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:54:55 GMT /pklxkd-59/obj111 2023-05-31T20:54:55.718 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:nQzizQ96QaobIqtCSEzUKA/oL/s= 2023-05-31T20:54:55.718 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:54:55 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:nQzizQ96QaobIqtCSEzUKA/oL/s=', 'Content-Length': '0'} 2023-05-31T20:54:55.720 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:52:14 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009324371790ab55c3-006477b41f-1e2e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:54:55 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:54:55.720 DEBUG:boto:path=/pklxkd-59/obj111 2023-05-31T20:54:55.721 DEBUG:boto:auth_path=/pklxkd-59/obj111 2023-05-31T20:54:55.721 DEBUG:boto:Method: GET 2023-05-31T20:54:55.721 DEBUG:boto:Path: /pklxkd-59/obj111 2023-05-31T20:54:55.722 DEBUG:boto:Data: 2023-05-31T20:54:55.722 DEBUG:boto:Headers: {} 2023-05-31T20:54:55.723 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:54:55.723 DEBUG:boto:Port: 8000 2023-05-31T20:54:55.723 DEBUG:boto:Params: {} 2023-05-31T20:54:55.724 DEBUG:boto:Token: None 2023-05-31T20:54:55.724 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:54:55 GMT /pklxkd-59/obj111 2023-05-31T20:54:55.724 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:nQzizQ96QaobIqtCSEzUKA/oL/s= 2023-05-31T20:54:55.725 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:54:55 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:nQzizQ96QaobIqtCSEzUKA/oL/s=', 'Content-Length': '0'} 2023-05-31T20:54:55.727 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:52:14 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f15c5305032f5827-006477b41f-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:54:55 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:54:55.727 DEBUG:boto:path=/pklxkd-59/ 2023-05-31T20:54:55.728 DEBUG:boto:auth_path=/pklxkd-59/ 2023-05-31T20:54:55.728 DEBUG:boto:path=/pklxkd-59/?prefix=obj111 2023-05-31T20:54:55.728 DEBUG:boto:auth_path=/pklxkd-59/?prefix=obj111 2023-05-31T20:54:55.729 DEBUG:boto:Method: GET 2023-05-31T20:54:55.729 DEBUG:boto:Path: /pklxkd-59/?prefix=obj111 2023-05-31T20:54:55.729 DEBUG:boto:Data: 2023-05-31T20:54:55.730 DEBUG:boto:Headers: {} 2023-05-31T20:54:55.730 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:54:55.730 DEBUG:boto:Port: 8001 2023-05-31T20:54:55.731 DEBUG:boto:Params: {} 2023-05-31T20:54:55.731 DEBUG:boto:Token: None 2023-05-31T20:54:55.732 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:54:55 GMT /pklxkd-59/ 2023-05-31T20:54:55.732 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:RywX8LlH8rRUpedRaKmQfbksV7k= 2023-05-31T20:54:55.732 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:54:55 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:RywX8LlH8rRUpedRaKmQfbksV7k=', 'Content-Length': '0'} 2023-05-31T20:54:55.735 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b3a78614280bc33f-006477b41f-1e2e-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:54:55 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:54:55.735 DEBUG:boto:b'pklxkd-59obj1111000falseobj1112023-05-31T20:52:14.993Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:54:55.736 DEBUG:boto:path=/pklxkd-59/ 2023-05-31T20:54:55.736 DEBUG:boto:auth_path=/pklxkd-59/ 2023-05-31T20:54:55.737 DEBUG:boto:path=/pklxkd-59/?prefix=obj111 2023-05-31T20:54:55.737 DEBUG:boto:auth_path=/pklxkd-59/?prefix=obj111 2023-05-31T20:54:55.737 DEBUG:boto:Method: GET 2023-05-31T20:54:55.738 DEBUG:boto:Path: /pklxkd-59/?prefix=obj111 2023-05-31T20:54:55.738 DEBUG:boto:Data: 2023-05-31T20:54:55.738 DEBUG:boto:Headers: {} 2023-05-31T20:54:55.739 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:54:55.739 DEBUG:boto:Port: 8000 2023-05-31T20:54:55.740 DEBUG:boto:Params: {} 2023-05-31T20:54:55.741 DEBUG:boto:Token: None 2023-05-31T20:54:55.741 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:54:55 GMT /pklxkd-59/ 2023-05-31T20:54:55.742 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:RywX8LlH8rRUpedRaKmQfbksV7k= 2023-05-31T20:54:55.743 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:54:55 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:RywX8LlH8rRUpedRaKmQfbksV7k=', 'Content-Length': '0'} 2023-05-31T20:54:55.746 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000888b912e63a51a8b-006477b41f-1630-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:54:55 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:54:55.747 DEBUG:boto:b'pklxkd-59obj1111000falseobj1112023-05-31T20:52:14.993Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:54:55.747 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj115 2023-05-31T20:54:55.748 DEBUG:boto:path=/pklxkd-59/obj115 2023-05-31T20:54:55.748 DEBUG:boto:auth_path=/pklxkd-59/obj115 2023-05-31T20:54:55.749 DEBUG:boto:path=/pklxkd-59/obj115?versionId=null 2023-05-31T20:54:55.749 DEBUG:boto:auth_path=/pklxkd-59/obj115?versionId=null 2023-05-31T20:54:55.750 DEBUG:boto:Method: GET 2023-05-31T20:54:55.751 DEBUG:boto:Path: /pklxkd-59/obj115?versionId=null 2023-05-31T20:54:55.751 DEBUG:boto:Data: 2023-05-31T20:54:55.752 DEBUG:boto:Headers: {} 2023-05-31T20:54:55.752 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:54:55.753 DEBUG:boto:Port: 8001 2023-05-31T20:54:55.753 DEBUG:boto:Params: {} 2023-05-31T20:54:55.754 DEBUG:boto:Token: None 2023-05-31T20:54:55.754 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:54:55 GMT /pklxkd-59/obj115?versionId=null 2023-05-31T20:54:55.754 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:RNv/zKyBd+qtjt0ojdjOMwmdap0= 2023-05-31T20:54:55.755 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:54:55 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:RNv/zKyBd+qtjt0ojdjOMwmdap0=', 'Content-Length': '0'} 2023-05-31T20:54:55.757 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:46:33 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004764795b8d2636c6-006477b41f-1e2e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:54:55 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:54:55.758 DEBUG:boto:path=/pklxkd-59/obj115 2023-05-31T20:54:55.758 DEBUG:boto:auth_path=/pklxkd-59/obj115 2023-05-31T20:54:55.758 DEBUG:boto:path=/pklxkd-59/obj115?versionId=null 2023-05-31T20:54:55.759 DEBUG:boto:auth_path=/pklxkd-59/obj115?versionId=null 2023-05-31T20:54:55.759 DEBUG:boto:Method: GET 2023-05-31T20:54:55.760 DEBUG:boto:Path: /pklxkd-59/obj115?versionId=null 2023-05-31T20:54:55.760 DEBUG:boto:Data: 2023-05-31T20:54:55.760 DEBUG:boto:Headers: {} 2023-05-31T20:54:55.761 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:54:55.762 DEBUG:boto:Port: 8000 2023-05-31T20:54:55.762 DEBUG:boto:Params: {} 2023-05-31T20:54:55.762 DEBUG:boto:Token: None 2023-05-31T20:54:55.763 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:54:55 GMT /pklxkd-59/obj115?versionId=null 2023-05-31T20:54:55.764 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:RNv/zKyBd+qtjt0ojdjOMwmdap0= 2023-05-31T20:54:55.765 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:54:55 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:RNv/zKyBd+qtjt0ojdjOMwmdap0=', 'Content-Length': '0'} 2023-05-31T20:54:55.767 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:46:33 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006411fa442044d28b-006477b41f-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:54:55 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:54:55.808 DEBUG:boto:path=/pklxkd-59/obj115 2023-05-31T20:54:55.809 DEBUG:boto:auth_path=/pklxkd-59/obj115 2023-05-31T20:54:55.809 DEBUG:boto:path=/pklxkd-59/obj115?versionId=null 2023-05-31T20:54:55.809 DEBUG:boto:auth_path=/pklxkd-59/obj115?versionId=null 2023-05-31T20:54:55.810 DEBUG:boto:Method: HEAD 2023-05-31T20:54:55.810 DEBUG:boto:Path: /pklxkd-59/obj115?versionId=null 2023-05-31T20:54:55.810 DEBUG:boto:Data: 2023-05-31T20:54:55.811 DEBUG:boto:Headers: {} 2023-05-31T20:54:55.811 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:54:55.811 DEBUG:boto:Port: 8001 2023-05-31T20:54:55.812 DEBUG:boto:Params: {} 2023-05-31T20:54:55.812 DEBUG:boto:Token: None 2023-05-31T20:54:55.812 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:54:55 GMT /pklxkd-59/obj115?versionId=null 2023-05-31T20:54:55.813 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:+zLObjrVILFUdMbQYsg2pKGNc+8= 2023-05-31T20:54:55.813 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:54:55 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:+zLObjrVILFUdMbQYsg2pKGNc+8=', 'Content-Length': '0'} 2023-05-31T20:54:55.815 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:46:33 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008619057f910e2a72-006477b41f-1e2e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:54:55 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:54:55.816 DEBUG:boto:path=/pklxkd-59/obj115 2023-05-31T20:54:55.816 DEBUG:boto:auth_path=/pklxkd-59/obj115 2023-05-31T20:54:55.816 DEBUG:boto:path=/pklxkd-59/obj115?versionId=null 2023-05-31T20:54:55.817 DEBUG:boto:auth_path=/pklxkd-59/obj115?versionId=null 2023-05-31T20:54:55.817 DEBUG:boto:Method: HEAD 2023-05-31T20:54:55.817 DEBUG:boto:Path: /pklxkd-59/obj115?versionId=null 2023-05-31T20:54:55.818 DEBUG:boto:Data: 2023-05-31T20:54:55.818 DEBUG:boto:Headers: {} 2023-05-31T20:54:55.818 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:54:55.819 DEBUG:boto:Port: 8000 2023-05-31T20:54:55.819 DEBUG:boto:Params: {} 2023-05-31T20:54:55.819 DEBUG:boto:Token: None 2023-05-31T20:54:55.820 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:54:55 GMT /pklxkd-59/obj115?versionId=null 2023-05-31T20:54:55.820 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:+zLObjrVILFUdMbQYsg2pKGNc+8= 2023-05-31T20:54:55.821 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:54:55 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:+zLObjrVILFUdMbQYsg2pKGNc+8=', 'Content-Length': '0'} 2023-05-31T20:54:55.822 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:46:33 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000886f148a47a0e560-006477b41f-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:54:55 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:54:55.823 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj115 2023-05-31T20:54:55.823 DEBUG:boto:path=/pklxkd-59/obj115 2023-05-31T20:54:55.824 DEBUG:boto:auth_path=/pklxkd-59/obj115 2023-05-31T20:54:55.824 DEBUG:boto:path=/pklxkd-59/obj115?versionId=null 2023-05-31T20:54:55.824 DEBUG:boto:auth_path=/pklxkd-59/obj115?versionId=null 2023-05-31T20:54:55.825 DEBUG:boto:Method: GET 2023-05-31T20:54:55.825 DEBUG:boto:Path: /pklxkd-59/obj115?versionId=null 2023-05-31T20:54:55.826 DEBUG:boto:Data: 2023-05-31T20:54:55.826 DEBUG:boto:Headers: {} 2023-05-31T20:54:55.826 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:54:55.827 DEBUG:boto:Port: 8001 2023-05-31T20:54:55.827 DEBUG:boto:Params: {} 2023-05-31T20:54:55.827 DEBUG:boto:Token: None 2023-05-31T20:54:55.828 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:54:55 GMT /pklxkd-59/obj115?versionId=null 2023-05-31T20:54:55.828 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:RNv/zKyBd+qtjt0ojdjOMwmdap0= 2023-05-31T20:54:55.829 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:54:55 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:RNv/zKyBd+qtjt0ojdjOMwmdap0=', 'Content-Length': '0'} 2023-05-31T20:54:55.830 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:46:33 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001430763ca2837958-006477b41f-1e2e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:54:55 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:54:55.831 DEBUG:boto:path=/pklxkd-59/obj115 2023-05-31T20:54:55.831 DEBUG:boto:auth_path=/pklxkd-59/obj115 2023-05-31T20:54:55.832 DEBUG:boto:path=/pklxkd-59/obj115?versionId=null 2023-05-31T20:54:55.832 DEBUG:boto:auth_path=/pklxkd-59/obj115?versionId=null 2023-05-31T20:54:55.832 DEBUG:boto:Method: GET 2023-05-31T20:54:55.833 DEBUG:boto:Path: /pklxkd-59/obj115?versionId=null 2023-05-31T20:54:55.833 DEBUG:boto:Data: 2023-05-31T20:54:55.833 DEBUG:boto:Headers: {} 2023-05-31T20:54:55.834 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:54:55.834 DEBUG:boto:Port: 8000 2023-05-31T20:54:55.835 DEBUG:boto:Params: {} 2023-05-31T20:54:55.835 DEBUG:boto:Token: None 2023-05-31T20:54:55.835 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:54:55 GMT /pklxkd-59/obj115?versionId=null 2023-05-31T20:54:55.836 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:RNv/zKyBd+qtjt0ojdjOMwmdap0= 2023-05-31T20:54:55.836 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:54:55 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:RNv/zKyBd+qtjt0ojdjOMwmdap0=', 'Content-Length': '0'} 2023-05-31T20:54:55.838 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:46:33 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000055215271a8d34739-006477b41f-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:54:55 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:54:55.839 DEBUG:boto:path=/pklxkd-59/ 2023-05-31T20:54:55.839 DEBUG:boto:auth_path=/pklxkd-59/ 2023-05-31T20:54:55.839 DEBUG:boto:path=/pklxkd-59/?prefix=obj115 2023-05-31T20:54:55.840 DEBUG:boto:auth_path=/pklxkd-59/?prefix=obj115 2023-05-31T20:54:55.840 DEBUG:boto:Method: GET 2023-05-31T20:54:55.841 DEBUG:boto:Path: /pklxkd-59/?prefix=obj115 2023-05-31T20:54:55.841 DEBUG:boto:Data: 2023-05-31T20:54:55.841 DEBUG:boto:Headers: {} 2023-05-31T20:54:55.842 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:54:55.842 DEBUG:boto:Port: 8001 2023-05-31T20:54:55.842 DEBUG:boto:Params: {} 2023-05-31T20:54:55.843 DEBUG:boto:Token: None 2023-05-31T20:54:55.843 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:54:55 GMT /pklxkd-59/ 2023-05-31T20:54:55.843 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:RywX8LlH8rRUpedRaKmQfbksV7k= 2023-05-31T20:54:55.844 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:54:55 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:RywX8LlH8rRUpedRaKmQfbksV7k=', 'Content-Length': '0'} 2023-05-31T20:54:55.846 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000923080c793edb1a9-006477b41f-1e2e-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:54:55 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:54:55.847 DEBUG:boto:b'pklxkd-59obj1151000falseobj1152023-05-31T20:46:33.958Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:54:55.847 DEBUG:boto:path=/pklxkd-59/ 2023-05-31T20:54:55.848 DEBUG:boto:auth_path=/pklxkd-59/ 2023-05-31T20:54:55.848 DEBUG:boto:path=/pklxkd-59/?prefix=obj115 2023-05-31T20:54:55.848 DEBUG:boto:auth_path=/pklxkd-59/?prefix=obj115 2023-05-31T20:54:55.849 DEBUG:boto:Method: GET 2023-05-31T20:54:55.849 DEBUG:boto:Path: /pklxkd-59/?prefix=obj115 2023-05-31T20:54:55.850 DEBUG:boto:Data: 2023-05-31T20:54:55.850 DEBUG:boto:Headers: {} 2023-05-31T20:54:55.850 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:54:55.851 DEBUG:boto:Port: 8000 2023-05-31T20:54:55.851 DEBUG:boto:Params: {} 2023-05-31T20:54:55.851 DEBUG:boto:Token: None 2023-05-31T20:54:55.852 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:54:55 GMT /pklxkd-59/ 2023-05-31T20:54:55.852 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:RywX8LlH8rRUpedRaKmQfbksV7k= 2023-05-31T20:54:55.853 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:54:55 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:RywX8LlH8rRUpedRaKmQfbksV7k=', 'Content-Length': '0'} 2023-05-31T20:54:55.856 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000000d57cc4dc8ed063c-006477b41f-1630-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:54:55 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:54:55.857 DEBUG:boto:b'pklxkd-59obj1151000falseobj1152023-05-31T20:46:33.958Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:54:55.857 DEBUG:boto:path=/pklxkd-59/obj115 2023-05-31T20:54:55.857 DEBUG:boto:auth_path=/pklxkd-59/obj115 2023-05-31T20:54:55.858 DEBUG:boto:Method: HEAD 2023-05-31T20:54:55.858 DEBUG:boto:Path: /pklxkd-59/obj115 2023-05-31T20:54:55.859 DEBUG:boto:Data: 2023-05-31T20:54:55.859 DEBUG:boto:Headers: {} 2023-05-31T20:54:55.859 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:54:55.859 DEBUG:boto:Port: 8001 2023-05-31T20:54:55.860 DEBUG:boto:Params: {} 2023-05-31T20:54:55.860 DEBUG:boto:Token: None 2023-05-31T20:54:55.861 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:54:55 GMT /pklxkd-59/obj115 2023-05-31T20:54:55.861 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:BW0QfA7mIuYhdJwjN1JnPG4FfLg= 2023-05-31T20:54:55.862 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:54:55 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:BW0QfA7mIuYhdJwjN1JnPG4FfLg=', 'Content-Length': '0'} 2023-05-31T20:54:55.863 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:46:33 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000228b5560c8f7b61f-006477b41f-1e2e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:54:55 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:54:55.864 DEBUG:boto:path=/pklxkd-59/obj115 2023-05-31T20:54:55.864 DEBUG:boto:auth_path=/pklxkd-59/obj115 2023-05-31T20:54:55.864 DEBUG:boto:Method: HEAD 2023-05-31T20:54:55.865 DEBUG:boto:Path: /pklxkd-59/obj115 2023-05-31T20:54:55.865 DEBUG:boto:Data: 2023-05-31T20:54:55.865 DEBUG:boto:Headers: {} 2023-05-31T20:54:55.866 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:54:55.866 DEBUG:boto:Port: 8000 2023-05-31T20:54:55.866 DEBUG:boto:Params: {} 2023-05-31T20:54:55.867 DEBUG:boto:Token: None 2023-05-31T20:54:55.867 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:54:55 GMT /pklxkd-59/obj115 2023-05-31T20:54:55.868 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:BW0QfA7mIuYhdJwjN1JnPG4FfLg= 2023-05-31T20:54:55.868 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:54:55 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:BW0QfA7mIuYhdJwjN1JnPG4FfLg=', 'Content-Length': '0'} 2023-05-31T20:54:55.870 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:46:33 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008fa2c282ce97a1be-006477b41f-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:54:55 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:54:55.870 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj115 2023-05-31T20:54:55.871 DEBUG:boto:path=/pklxkd-59/obj115 2023-05-31T20:54:55.871 DEBUG:boto:auth_path=/pklxkd-59/obj115 2023-05-31T20:54:55.871 DEBUG:boto:Method: GET 2023-05-31T20:54:55.872 DEBUG:boto:Path: /pklxkd-59/obj115 2023-05-31T20:54:55.872 DEBUG:boto:Data: 2023-05-31T20:54:55.873 DEBUG:boto:Headers: {} 2023-05-31T20:54:55.873 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:54:55.873 DEBUG:boto:Port: 8001 2023-05-31T20:54:55.874 DEBUG:boto:Params: {} 2023-05-31T20:54:55.874 DEBUG:boto:Token: None 2023-05-31T20:54:55.874 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:54:55 GMT /pklxkd-59/obj115 2023-05-31T20:54:55.875 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:Bccfs1qYJGH4yKwrTSWNfFlmK/Q= 2023-05-31T20:54:55.876 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:54:55 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:Bccfs1qYJGH4yKwrTSWNfFlmK/Q=', 'Content-Length': '0'} 2023-05-31T20:54:55.877 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:46:33 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f27b27811cd43925-006477b41f-1e2e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:54:55 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:54:55.878 DEBUG:boto:path=/pklxkd-59/obj115 2023-05-31T20:54:55.879 DEBUG:boto:auth_path=/pklxkd-59/obj115 2023-05-31T20:54:55.880 DEBUG:boto:Method: GET 2023-05-31T20:54:55.880 DEBUG:boto:Path: /pklxkd-59/obj115 2023-05-31T20:54:55.880 DEBUG:boto:Data: 2023-05-31T20:54:55.881 DEBUG:boto:Headers: {} 2023-05-31T20:54:55.882 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:54:55.882 DEBUG:boto:Port: 8000 2023-05-31T20:54:55.882 DEBUG:boto:Params: {} 2023-05-31T20:54:55.883 DEBUG:boto:Token: None 2023-05-31T20:54:55.883 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:54:55 GMT /pklxkd-59/obj115 2023-05-31T20:54:55.884 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:Bccfs1qYJGH4yKwrTSWNfFlmK/Q= 2023-05-31T20:54:55.885 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:54:55 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:Bccfs1qYJGH4yKwrTSWNfFlmK/Q=', 'Content-Length': '0'} 2023-05-31T20:54:55.886 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:46:33 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000cfcb0605f4c8355d-006477b41f-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:54:55 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:54:55.887 DEBUG:boto:path=/pklxkd-59/ 2023-05-31T20:54:55.887 DEBUG:boto:auth_path=/pklxkd-59/ 2023-05-31T20:54:55.888 DEBUG:boto:path=/pklxkd-59/?prefix=obj115 2023-05-31T20:54:55.888 DEBUG:boto:auth_path=/pklxkd-59/?prefix=obj115 2023-05-31T20:54:55.888 DEBUG:boto:Method: GET 2023-05-31T20:54:55.889 DEBUG:boto:Path: /pklxkd-59/?prefix=obj115 2023-05-31T20:54:55.889 DEBUG:boto:Data: 2023-05-31T20:54:55.889 DEBUG:boto:Headers: {} 2023-05-31T20:54:55.890 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:54:55.890 DEBUG:boto:Port: 8001 2023-05-31T20:54:55.890 DEBUG:boto:Params: {} 2023-05-31T20:54:55.891 DEBUG:boto:Token: None 2023-05-31T20:54:55.891 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:54:55 GMT /pklxkd-59/ 2023-05-31T20:54:55.892 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:RywX8LlH8rRUpedRaKmQfbksV7k= 2023-05-31T20:54:55.892 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:54:55 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:RywX8LlH8rRUpedRaKmQfbksV7k=', 'Content-Length': '0'} 2023-05-31T20:54:55.894 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000126864c2bdbd3de6-006477b41f-1e2e-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:54:55 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:54:55.895 DEBUG:boto:b'pklxkd-59obj1151000falseobj1152023-05-31T20:46:33.958Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:54:55.895 DEBUG:boto:path=/pklxkd-59/ 2023-05-31T20:54:55.896 DEBUG:boto:auth_path=/pklxkd-59/ 2023-05-31T20:54:55.896 DEBUG:boto:path=/pklxkd-59/?prefix=obj115 2023-05-31T20:54:55.897 DEBUG:boto:auth_path=/pklxkd-59/?prefix=obj115 2023-05-31T20:54:55.897 DEBUG:boto:Method: GET 2023-05-31T20:54:55.897 DEBUG:boto:Path: /pklxkd-59/?prefix=obj115 2023-05-31T20:54:55.898 DEBUG:boto:Data: 2023-05-31T20:54:55.898 DEBUG:boto:Headers: {} 2023-05-31T20:54:55.898 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:54:55.899 DEBUG:boto:Port: 8000 2023-05-31T20:54:55.899 DEBUG:boto:Params: {} 2023-05-31T20:54:55.900 DEBUG:boto:Token: None 2023-05-31T20:54:55.900 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:54:55 GMT /pklxkd-59/ 2023-05-31T20:54:55.901 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:RywX8LlH8rRUpedRaKmQfbksV7k= 2023-05-31T20:54:55.901 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:54:55 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:RywX8LlH8rRUpedRaKmQfbksV7k=', 'Content-Length': '0'} 2023-05-31T20:54:55.904 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f71bcbf8276ceef0-006477b41f-1630-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:54:55 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:54:55.905 DEBUG:boto:b'pklxkd-59obj1151000falseobj1152023-05-31T20:46:33.958Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:54:55.906 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj116 2023-05-31T20:54:55.906 DEBUG:boto:path=/pklxkd-59/obj116 2023-05-31T20:54:55.906 DEBUG:boto:auth_path=/pklxkd-59/obj116 2023-05-31T20:54:55.907 DEBUG:boto:path=/pklxkd-59/obj116?versionId=null 2023-05-31T20:54:55.907 DEBUG:boto:auth_path=/pklxkd-59/obj116?versionId=null 2023-05-31T20:54:55.908 DEBUG:boto:Method: GET 2023-05-31T20:54:55.908 DEBUG:boto:Path: /pklxkd-59/obj116?versionId=null 2023-05-31T20:54:55.909 DEBUG:boto:Data: 2023-05-31T20:54:55.909 DEBUG:boto:Headers: {} 2023-05-31T20:54:55.909 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:54:55.910 DEBUG:boto:Port: 8001 2023-05-31T20:54:55.910 DEBUG:boto:Params: {} 2023-05-31T20:54:55.911 DEBUG:boto:Token: None 2023-05-31T20:54:55.911 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:54:55 GMT /pklxkd-59/obj116?versionId=null 2023-05-31T20:54:55.911 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:K5NK6zP+8WgCGq+i1ZJPFHfcMO8= 2023-05-31T20:54:55.912 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:54:55 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:K5NK6zP+8WgCGq+i1ZJPFHfcMO8=', 'Content-Length': '0'} 2023-05-31T20:54:55.914 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:48:13 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000012f772a31b30e138-006477b41f-1e2e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:54:55 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:54:55.915 DEBUG:boto:path=/pklxkd-59/obj116 2023-05-31T20:54:55.915 DEBUG:boto:auth_path=/pklxkd-59/obj116 2023-05-31T20:54:55.915 DEBUG:boto:path=/pklxkd-59/obj116?versionId=null 2023-05-31T20:54:55.916 DEBUG:boto:auth_path=/pklxkd-59/obj116?versionId=null 2023-05-31T20:54:55.916 DEBUG:boto:Method: GET 2023-05-31T20:54:55.916 DEBUG:boto:Path: /pklxkd-59/obj116?versionId=null 2023-05-31T20:54:55.917 DEBUG:boto:Data: 2023-05-31T20:54:55.917 DEBUG:boto:Headers: {} 2023-05-31T20:54:55.918 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:54:55.918 DEBUG:boto:Port: 8000 2023-05-31T20:54:55.918 DEBUG:boto:Params: {} 2023-05-31T20:54:55.919 DEBUG:boto:Token: None 2023-05-31T20:54:55.919 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:54:55 GMT /pklxkd-59/obj116?versionId=null 2023-05-31T20:54:55.919 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:K5NK6zP+8WgCGq+i1ZJPFHfcMO8= 2023-05-31T20:54:55.920 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:54:55 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:K5NK6zP+8WgCGq+i1ZJPFHfcMO8=', 'Content-Length': '0'} 2023-05-31T20:54:55.922 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:48:13 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000507390a2e46c913c-006477b41f-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:54:55 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:54:55.964 DEBUG:boto:path=/pklxkd-59/obj116 2023-05-31T20:54:55.965 DEBUG:boto:auth_path=/pklxkd-59/obj116 2023-05-31T20:54:55.965 DEBUG:boto:path=/pklxkd-59/obj116?versionId=null 2023-05-31T20:54:55.965 DEBUG:boto:auth_path=/pklxkd-59/obj116?versionId=null 2023-05-31T20:54:55.966 DEBUG:boto:Method: HEAD 2023-05-31T20:54:55.966 DEBUG:boto:Path: /pklxkd-59/obj116?versionId=null 2023-05-31T20:54:55.966 DEBUG:boto:Data: 2023-05-31T20:54:55.967 DEBUG:boto:Headers: {} 2023-05-31T20:54:55.967 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:54:55.967 DEBUG:boto:Port: 8001 2023-05-31T20:54:55.968 DEBUG:boto:Params: {} 2023-05-31T20:54:55.968 DEBUG:boto:Token: None 2023-05-31T20:54:55.969 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:54:55 GMT /pklxkd-59/obj116?versionId=null 2023-05-31T20:54:55.969 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:kl9LFEoZsuSCJx+T7x8DwagV/J8= 2023-05-31T20:54:55.969 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:54:55 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:kl9LFEoZsuSCJx+T7x8DwagV/J8=', 'Content-Length': '0'} 2023-05-31T20:54:55.971 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:48:13 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000558c9207295d295e-006477b41f-1e2e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:54:55 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:54:55.972 DEBUG:boto:path=/pklxkd-59/obj116 2023-05-31T20:54:55.972 DEBUG:boto:auth_path=/pklxkd-59/obj116 2023-05-31T20:54:55.972 DEBUG:boto:path=/pklxkd-59/obj116?versionId=null 2023-05-31T20:54:55.973 DEBUG:boto:auth_path=/pklxkd-59/obj116?versionId=null 2023-05-31T20:54:55.973 DEBUG:boto:Method: HEAD 2023-05-31T20:54:55.973 DEBUG:boto:Path: /pklxkd-59/obj116?versionId=null 2023-05-31T20:54:55.974 DEBUG:boto:Data: 2023-05-31T20:54:55.974 DEBUG:boto:Headers: {} 2023-05-31T20:54:55.974 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:54:55.975 DEBUG:boto:Port: 8000 2023-05-31T20:54:55.975 DEBUG:boto:Params: {} 2023-05-31T20:54:55.975 DEBUG:boto:Token: None 2023-05-31T20:54:55.976 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:54:55 GMT /pklxkd-59/obj116?versionId=null 2023-05-31T20:54:55.976 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:kl9LFEoZsuSCJx+T7x8DwagV/J8= 2023-05-31T20:54:55.977 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:54:55 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:kl9LFEoZsuSCJx+T7x8DwagV/J8=', 'Content-Length': '0'} 2023-05-31T20:54:55.978 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:48:13 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c17695451aaccbe0-006477b41f-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:54:55 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:54:55.979 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj116 2023-05-31T20:54:55.979 DEBUG:boto:path=/pklxkd-59/obj116 2023-05-31T20:54:55.980 DEBUG:boto:auth_path=/pklxkd-59/obj116 2023-05-31T20:54:55.980 DEBUG:boto:path=/pklxkd-59/obj116?versionId=null 2023-05-31T20:54:55.980 DEBUG:boto:auth_path=/pklxkd-59/obj116?versionId=null 2023-05-31T20:54:55.981 DEBUG:boto:Method: GET 2023-05-31T20:54:55.981 DEBUG:boto:Path: /pklxkd-59/obj116?versionId=null 2023-05-31T20:54:55.982 DEBUG:boto:Data: 2023-05-31T20:54:55.982 DEBUG:boto:Headers: {} 2023-05-31T20:54:55.982 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:54:55.983 DEBUG:boto:Port: 8001 2023-05-31T20:54:55.983 DEBUG:boto:Params: {} 2023-05-31T20:54:55.984 DEBUG:boto:Token: None 2023-05-31T20:54:55.984 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:54:55 GMT /pklxkd-59/obj116?versionId=null 2023-05-31T20:54:55.984 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:K5NK6zP+8WgCGq+i1ZJPFHfcMO8= 2023-05-31T20:54:55.985 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:54:55 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:K5NK6zP+8WgCGq+i1ZJPFHfcMO8=', 'Content-Length': '0'} 2023-05-31T20:54:55.987 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:48:13 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c04b351391e9a7de-006477b41f-1e2e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:54:55 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:54:55.988 DEBUG:boto:path=/pklxkd-59/obj116 2023-05-31T20:54:55.988 DEBUG:boto:auth_path=/pklxkd-59/obj116 2023-05-31T20:54:55.988 DEBUG:boto:path=/pklxkd-59/obj116?versionId=null 2023-05-31T20:54:55.989 DEBUG:boto:auth_path=/pklxkd-59/obj116?versionId=null 2023-05-31T20:54:55.989 DEBUG:boto:Method: GET 2023-05-31T20:54:55.989 DEBUG:boto:Path: /pklxkd-59/obj116?versionId=null 2023-05-31T20:54:55.990 DEBUG:boto:Data: 2023-05-31T20:54:55.990 DEBUG:boto:Headers: {} 2023-05-31T20:54:55.991 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:54:55.991 DEBUG:boto:Port: 8000 2023-05-31T20:54:55.992 DEBUG:boto:Params: {} 2023-05-31T20:54:55.992 DEBUG:boto:Token: None 2023-05-31T20:54:55.993 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:54:55 GMT /pklxkd-59/obj116?versionId=null 2023-05-31T20:54:55.993 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:K5NK6zP+8WgCGq+i1ZJPFHfcMO8= 2023-05-31T20:54:55.994 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:54:55 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:K5NK6zP+8WgCGq+i1ZJPFHfcMO8=', 'Content-Length': '0'} 2023-05-31T20:54:55.995 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:48:13 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001e700773a032e467-006477b41f-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:54:55 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:54:55.996 DEBUG:boto:path=/pklxkd-59/ 2023-05-31T20:54:55.996 DEBUG:boto:auth_path=/pklxkd-59/ 2023-05-31T20:54:55.997 DEBUG:boto:path=/pklxkd-59/?prefix=obj116 2023-05-31T20:54:55.997 DEBUG:boto:auth_path=/pklxkd-59/?prefix=obj116 2023-05-31T20:54:55.997 DEBUG:boto:Method: GET 2023-05-31T20:54:55.998 DEBUG:boto:Path: /pklxkd-59/?prefix=obj116 2023-05-31T20:54:55.998 DEBUG:boto:Data: 2023-05-31T20:54:55.999 DEBUG:boto:Headers: {} 2023-05-31T20:54:55.999 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:54:55.999 DEBUG:boto:Port: 8001 2023-05-31T20:54:56.000 DEBUG:boto:Params: {} 2023-05-31T20:54:56.000 DEBUG:boto:Token: None 2023-05-31T20:54:56.001 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:54:56 GMT /pklxkd-59/ 2023-05-31T20:54:56.001 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:jiU8KdFvayyCgdc2KNAmenbHZMQ= 2023-05-31T20:54:56.001 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:54:56 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:jiU8KdFvayyCgdc2KNAmenbHZMQ=', 'Content-Length': '0'} 2023-05-31T20:54:56.004 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b269bbd92db31beb-006477b41f-1e2e-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:54:55 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:54:56.004 DEBUG:boto:b'pklxkd-59obj1161000falseobj1162023-05-31T20:48:13.052Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:54:56.005 DEBUG:boto:path=/pklxkd-59/ 2023-05-31T20:54:56.005 DEBUG:boto:auth_path=/pklxkd-59/ 2023-05-31T20:54:56.006 DEBUG:boto:path=/pklxkd-59/?prefix=obj116 2023-05-31T20:54:56.006 DEBUG:boto:auth_path=/pklxkd-59/?prefix=obj116 2023-05-31T20:54:56.006 DEBUG:boto:Method: GET 2023-05-31T20:54:56.007 DEBUG:boto:Path: /pklxkd-59/?prefix=obj116 2023-05-31T20:54:56.007 DEBUG:boto:Data: 2023-05-31T20:54:56.008 DEBUG:boto:Headers: {} 2023-05-31T20:54:56.008 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:54:56.008 DEBUG:boto:Port: 8000 2023-05-31T20:54:56.009 DEBUG:boto:Params: {} 2023-05-31T20:54:56.009 DEBUG:boto:Token: None 2023-05-31T20:54:56.010 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:54:56 GMT /pklxkd-59/ 2023-05-31T20:54:56.010 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:jiU8KdFvayyCgdc2KNAmenbHZMQ= 2023-05-31T20:54:56.010 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:54:56 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:jiU8KdFvayyCgdc2KNAmenbHZMQ=', 'Content-Length': '0'} 2023-05-31T20:54:56.014 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ec27a2bb0bbe7f6c-006477b420-1630-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:54:56 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:54:56.015 DEBUG:boto:b'pklxkd-59obj1161000falseobj1162023-05-31T20:48:13.052Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:54:56.015 DEBUG:boto:path=/pklxkd-59/obj116 2023-05-31T20:54:56.016 DEBUG:boto:auth_path=/pklxkd-59/obj116 2023-05-31T20:54:56.016 DEBUG:boto:Method: HEAD 2023-05-31T20:54:56.016 DEBUG:boto:Path: /pklxkd-59/obj116 2023-05-31T20:54:56.017 DEBUG:boto:Data: 2023-05-31T20:54:56.017 DEBUG:boto:Headers: {} 2023-05-31T20:54:56.017 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:54:56.018 DEBUG:boto:Port: 8001 2023-05-31T20:54:56.018 DEBUG:boto:Params: {} 2023-05-31T20:54:56.018 DEBUG:boto:Token: None 2023-05-31T20:54:56.019 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:54:56 GMT /pklxkd-59/obj116 2023-05-31T20:54:56.019 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:BUK+luBiUbxwpJNhye+icAZC94c= 2023-05-31T20:54:56.019 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:54:56 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:BUK+luBiUbxwpJNhye+icAZC94c=', 'Content-Length': '0'} 2023-05-31T20:54:56.021 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:48:13 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ce112c35da99cd78-006477b420-1e2e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:54:56 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:54:56.022 DEBUG:boto:path=/pklxkd-59/obj116 2023-05-31T20:54:56.022 DEBUG:boto:auth_path=/pklxkd-59/obj116 2023-05-31T20:54:56.022 DEBUG:boto:Method: HEAD 2023-05-31T20:54:56.023 DEBUG:boto:Path: /pklxkd-59/obj116 2023-05-31T20:54:56.023 DEBUG:boto:Data: 2023-05-31T20:54:56.023 DEBUG:boto:Headers: {} 2023-05-31T20:54:56.024 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:54:56.024 DEBUG:boto:Port: 8000 2023-05-31T20:54:56.024 DEBUG:boto:Params: {} 2023-05-31T20:54:56.025 DEBUG:boto:Token: None 2023-05-31T20:54:56.025 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:54:56 GMT /pklxkd-59/obj116 2023-05-31T20:54:56.025 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:BUK+luBiUbxwpJNhye+icAZC94c= 2023-05-31T20:54:56.026 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:54:56 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:BUK+luBiUbxwpJNhye+icAZC94c=', 'Content-Length': '0'} 2023-05-31T20:54:56.027 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:48:13 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007710c8de01a56c8b-006477b420-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:54:56 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:54:56.028 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj116 2023-05-31T20:54:56.028 DEBUG:boto:path=/pklxkd-59/obj116 2023-05-31T20:54:56.029 DEBUG:boto:auth_path=/pklxkd-59/obj116 2023-05-31T20:54:56.029 DEBUG:boto:Method: GET 2023-05-31T20:54:56.029 DEBUG:boto:Path: /pklxkd-59/obj116 2023-05-31T20:54:56.030 DEBUG:boto:Data: 2023-05-31T20:54:56.030 DEBUG:boto:Headers: {} 2023-05-31T20:54:56.030 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:54:56.031 DEBUG:boto:Port: 8001 2023-05-31T20:54:56.031 DEBUG:boto:Params: {} 2023-05-31T20:54:56.032 DEBUG:boto:Token: None 2023-05-31T20:54:56.032 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:54:56 GMT /pklxkd-59/obj116 2023-05-31T20:54:56.032 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:tMcl/fCXYm8X2PfqUFDQo28Gwrk= 2023-05-31T20:54:56.033 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:54:56 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:tMcl/fCXYm8X2PfqUFDQo28Gwrk=', 'Content-Length': '0'} 2023-05-31T20:54:56.035 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:48:13 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e13822fcd4250ba3-006477b420-1e2e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:54:56 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:54:56.076 DEBUG:boto:path=/pklxkd-59/obj116 2023-05-31T20:54:56.077 DEBUG:boto:auth_path=/pklxkd-59/obj116 2023-05-31T20:54:56.077 DEBUG:boto:Method: GET 2023-05-31T20:54:56.077 DEBUG:boto:Path: /pklxkd-59/obj116 2023-05-31T20:54:56.078 DEBUG:boto:Data: 2023-05-31T20:54:56.078 DEBUG:boto:Headers: {} 2023-05-31T20:54:56.079 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:54:56.079 DEBUG:boto:Port: 8000 2023-05-31T20:54:56.079 DEBUG:boto:Params: {} 2023-05-31T20:54:56.080 DEBUG:boto:Token: None 2023-05-31T20:54:56.080 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:54:56 GMT /pklxkd-59/obj116 2023-05-31T20:54:56.081 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:tMcl/fCXYm8X2PfqUFDQo28Gwrk= 2023-05-31T20:54:56.081 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:54:56 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:tMcl/fCXYm8X2PfqUFDQo28Gwrk=', 'Content-Length': '0'} 2023-05-31T20:54:56.083 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:48:13 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000309e1d2df6a3ae76-006477b420-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:54:56 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:54:56.084 DEBUG:boto:path=/pklxkd-59/ 2023-05-31T20:54:56.084 DEBUG:boto:auth_path=/pklxkd-59/ 2023-05-31T20:54:56.085 DEBUG:boto:path=/pklxkd-59/?prefix=obj116 2023-05-31T20:54:56.085 DEBUG:boto:auth_path=/pklxkd-59/?prefix=obj116 2023-05-31T20:54:56.085 DEBUG:boto:Method: GET 2023-05-31T20:54:56.086 DEBUG:boto:Path: /pklxkd-59/?prefix=obj116 2023-05-31T20:54:56.086 DEBUG:boto:Data: 2023-05-31T20:54:56.087 DEBUG:boto:Headers: {} 2023-05-31T20:54:56.087 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:54:56.087 DEBUG:boto:Port: 8001 2023-05-31T20:54:56.088 DEBUG:boto:Params: {} 2023-05-31T20:54:56.088 DEBUG:boto:Token: None 2023-05-31T20:54:56.089 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:54:56 GMT /pklxkd-59/ 2023-05-31T20:54:56.089 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:jiU8KdFvayyCgdc2KNAmenbHZMQ= 2023-05-31T20:54:56.089 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:54:56 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:jiU8KdFvayyCgdc2KNAmenbHZMQ=', 'Content-Length': '0'} 2023-05-31T20:54:56.092 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a3b192412f2e3ee2-006477b420-1e2e-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:54:56 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:54:56.093 DEBUG:boto:b'pklxkd-59obj1161000falseobj1162023-05-31T20:48:13.052Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:54:56.094 DEBUG:boto:path=/pklxkd-59/ 2023-05-31T20:54:56.094 DEBUG:boto:auth_path=/pklxkd-59/ 2023-05-31T20:54:56.094 DEBUG:boto:path=/pklxkd-59/?prefix=obj116 2023-05-31T20:54:56.095 DEBUG:boto:auth_path=/pklxkd-59/?prefix=obj116 2023-05-31T20:54:56.095 DEBUG:boto:Method: GET 2023-05-31T20:54:56.096 DEBUG:boto:Path: /pklxkd-59/?prefix=obj116 2023-05-31T20:54:56.096 DEBUG:boto:Data: 2023-05-31T20:54:56.096 DEBUG:boto:Headers: {} 2023-05-31T20:54:56.097 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:54:56.097 DEBUG:boto:Port: 8000 2023-05-31T20:54:56.097 DEBUG:boto:Params: {} 2023-05-31T20:54:56.098 DEBUG:boto:Token: None 2023-05-31T20:54:56.098 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:54:56 GMT /pklxkd-59/ 2023-05-31T20:54:56.099 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:jiU8KdFvayyCgdc2KNAmenbHZMQ= 2023-05-31T20:54:56.099 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:54:56 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:jiU8KdFvayyCgdc2KNAmenbHZMQ=', 'Content-Length': '0'} 2023-05-31T20:54:56.103 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ffa784361f17f3a0-006477b420-1630-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:54:56 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:54:56.103 DEBUG:boto:b'pklxkd-59obj1161000falseobj1162023-05-31T20:48:13.052Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:54:56.104 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj124 2023-05-31T20:54:56.104 DEBUG:boto:path=/pklxkd-59/obj124 2023-05-31T20:54:56.105 DEBUG:boto:auth_path=/pklxkd-59/obj124 2023-05-31T20:54:56.105 DEBUG:boto:path=/pklxkd-59/obj124?versionId=null 2023-05-31T20:54:56.106 DEBUG:boto:auth_path=/pklxkd-59/obj124?versionId=null 2023-05-31T20:54:56.106 DEBUG:boto:Method: GET 2023-05-31T20:54:56.106 DEBUG:boto:Path: /pklxkd-59/obj124?versionId=null 2023-05-31T20:54:56.107 DEBUG:boto:Data: 2023-05-31T20:54:56.107 DEBUG:boto:Headers: {} 2023-05-31T20:54:56.108 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:54:56.108 DEBUG:boto:Port: 8001 2023-05-31T20:54:56.108 DEBUG:boto:Params: {} 2023-05-31T20:54:56.109 DEBUG:boto:Token: None 2023-05-31T20:54:56.109 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:54:56 GMT /pklxkd-59/obj124?versionId=null 2023-05-31T20:54:56.110 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:rCPdl6Wj4dfxAqU9pf7e2kN6Hmg= 2023-05-31T20:54:56.110 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:54:56 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:rCPdl6Wj4dfxAqU9pf7e2kN6Hmg=', 'Content-Length': '0'} 2023-05-31T20:54:56.112 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:50:33 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f35fb49553167f7d-006477b420-1e2e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:54:56 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:54:56.113 DEBUG:boto:path=/pklxkd-59/obj124 2023-05-31T20:54:56.113 DEBUG:boto:auth_path=/pklxkd-59/obj124 2023-05-31T20:54:56.114 DEBUG:boto:path=/pklxkd-59/obj124?versionId=null 2023-05-31T20:54:56.114 DEBUG:boto:auth_path=/pklxkd-59/obj124?versionId=null 2023-05-31T20:54:56.115 DEBUG:boto:Method: GET 2023-05-31T20:54:56.115 DEBUG:boto:Path: /pklxkd-59/obj124?versionId=null 2023-05-31T20:54:56.115 DEBUG:boto:Data: 2023-05-31T20:54:56.116 DEBUG:boto:Headers: {} 2023-05-31T20:54:56.116 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:54:56.116 DEBUG:boto:Port: 8000 2023-05-31T20:54:56.117 DEBUG:boto:Params: {} 2023-05-31T20:54:56.117 DEBUG:boto:Token: None 2023-05-31T20:54:56.118 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:54:56 GMT /pklxkd-59/obj124?versionId=null 2023-05-31T20:54:56.118 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:rCPdl6Wj4dfxAqU9pf7e2kN6Hmg= 2023-05-31T20:54:56.119 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:54:56 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:rCPdl6Wj4dfxAqU9pf7e2kN6Hmg=', 'Content-Length': '0'} 2023-05-31T20:54:56.120 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:50:33 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008421cde9fe2696b7-006477b420-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:54:56 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:54:56.164 DEBUG:boto:path=/pklxkd-59/obj124 2023-05-31T20:54:56.165 DEBUG:boto:auth_path=/pklxkd-59/obj124 2023-05-31T20:54:56.165 DEBUG:boto:path=/pklxkd-59/obj124?versionId=null 2023-05-31T20:54:56.165 DEBUG:boto:auth_path=/pklxkd-59/obj124?versionId=null 2023-05-31T20:54:56.166 DEBUG:boto:Method: HEAD 2023-05-31T20:54:56.166 DEBUG:boto:Path: /pklxkd-59/obj124?versionId=null 2023-05-31T20:54:56.166 DEBUG:boto:Data: 2023-05-31T20:54:56.167 DEBUG:boto:Headers: {} 2023-05-31T20:54:56.167 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:54:56.167 DEBUG:boto:Port: 8001 2023-05-31T20:54:56.168 DEBUG:boto:Params: {} 2023-05-31T20:54:56.168 DEBUG:boto:Token: None 2023-05-31T20:54:56.168 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:54:56 GMT /pklxkd-59/obj124?versionId=null 2023-05-31T20:54:56.169 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:pT9bF5YHd/DUy/LX2ChAxWzkMkY= 2023-05-31T20:54:56.169 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:54:56 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:pT9bF5YHd/DUy/LX2ChAxWzkMkY=', 'Content-Length': '0'} 2023-05-31T20:54:56.171 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:50:33 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f144da53f9a3e1e5-006477b420-1e2e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:54:56 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:54:56.172 DEBUG:boto:path=/pklxkd-59/obj124 2023-05-31T20:54:56.172 DEBUG:boto:auth_path=/pklxkd-59/obj124 2023-05-31T20:54:56.172 DEBUG:boto:path=/pklxkd-59/obj124?versionId=null 2023-05-31T20:54:56.173 DEBUG:boto:auth_path=/pklxkd-59/obj124?versionId=null 2023-05-31T20:54:56.173 DEBUG:boto:Method: HEAD 2023-05-31T20:54:56.173 DEBUG:boto:Path: /pklxkd-59/obj124?versionId=null 2023-05-31T20:54:56.174 DEBUG:boto:Data: 2023-05-31T20:54:56.174 DEBUG:boto:Headers: {} 2023-05-31T20:54:56.174 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:54:56.175 DEBUG:boto:Port: 8000 2023-05-31T20:54:56.175 DEBUG:boto:Params: {} 2023-05-31T20:54:56.176 DEBUG:boto:Token: None 2023-05-31T20:54:56.176 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:54:56 GMT /pklxkd-59/obj124?versionId=null 2023-05-31T20:54:56.176 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:pT9bF5YHd/DUy/LX2ChAxWzkMkY= 2023-05-31T20:54:56.177 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:54:56 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:pT9bF5YHd/DUy/LX2ChAxWzkMkY=', 'Content-Length': '0'} 2023-05-31T20:54:56.178 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:50:33 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c379fe546e6bd946-006477b420-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:54:56 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:54:56.179 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj124 2023-05-31T20:54:56.179 DEBUG:boto:path=/pklxkd-59/obj124 2023-05-31T20:54:56.180 DEBUG:boto:auth_path=/pklxkd-59/obj124 2023-05-31T20:54:56.180 DEBUG:boto:path=/pklxkd-59/obj124?versionId=null 2023-05-31T20:54:56.180 DEBUG:boto:auth_path=/pklxkd-59/obj124?versionId=null 2023-05-31T20:54:56.181 DEBUG:boto:Method: GET 2023-05-31T20:54:56.181 DEBUG:boto:Path: /pklxkd-59/obj124?versionId=null 2023-05-31T20:54:56.181 DEBUG:boto:Data: 2023-05-31T20:54:56.182 DEBUG:boto:Headers: {} 2023-05-31T20:54:56.182 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:54:56.182 DEBUG:boto:Port: 8001 2023-05-31T20:54:56.183 DEBUG:boto:Params: {} 2023-05-31T20:54:56.183 DEBUG:boto:Token: None 2023-05-31T20:54:56.184 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:54:56 GMT /pklxkd-59/obj124?versionId=null 2023-05-31T20:54:56.184 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:rCPdl6Wj4dfxAqU9pf7e2kN6Hmg= 2023-05-31T20:54:56.185 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:54:56 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:rCPdl6Wj4dfxAqU9pf7e2kN6Hmg=', 'Content-Length': '0'} 2023-05-31T20:54:56.186 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:50:33 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009f9106dcd6718219-006477b420-1e2e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:54:56 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:54:56.187 DEBUG:boto:path=/pklxkd-59/obj124 2023-05-31T20:54:56.187 DEBUG:boto:auth_path=/pklxkd-59/obj124 2023-05-31T20:54:56.188 DEBUG:boto:path=/pklxkd-59/obj124?versionId=null 2023-05-31T20:54:56.188 DEBUG:boto:auth_path=/pklxkd-59/obj124?versionId=null 2023-05-31T20:54:56.188 DEBUG:boto:Method: GET 2023-05-31T20:54:56.189 DEBUG:boto:Path: /pklxkd-59/obj124?versionId=null 2023-05-31T20:54:56.189 DEBUG:boto:Data: 2023-05-31T20:54:56.189 DEBUG:boto:Headers: {} 2023-05-31T20:54:56.190 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:54:56.190 DEBUG:boto:Port: 8000 2023-05-31T20:54:56.191 DEBUG:boto:Params: {} 2023-05-31T20:54:56.191 DEBUG:boto:Token: None 2023-05-31T20:54:56.191 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:54:56 GMT /pklxkd-59/obj124?versionId=null 2023-05-31T20:54:56.192 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:rCPdl6Wj4dfxAqU9pf7e2kN6Hmg= 2023-05-31T20:54:56.192 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:54:56 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:rCPdl6Wj4dfxAqU9pf7e2kN6Hmg=', 'Content-Length': '0'} 2023-05-31T20:54:56.194 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:50:33 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000003505a9613953668-006477b420-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:54:56 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:54:56.194 DEBUG:boto:path=/pklxkd-59/ 2023-05-31T20:54:56.195 DEBUG:boto:auth_path=/pklxkd-59/ 2023-05-31T20:54:56.195 DEBUG:boto:path=/pklxkd-59/?prefix=obj124 2023-05-31T20:54:56.195 DEBUG:boto:auth_path=/pklxkd-59/?prefix=obj124 2023-05-31T20:54:56.196 DEBUG:boto:Method: GET 2023-05-31T20:54:56.196 DEBUG:boto:Path: /pklxkd-59/?prefix=obj124 2023-05-31T20:54:56.197 DEBUG:boto:Data: 2023-05-31T20:54:56.197 DEBUG:boto:Headers: {} 2023-05-31T20:54:56.197 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:54:56.198 DEBUG:boto:Port: 8001 2023-05-31T20:54:56.198 DEBUG:boto:Params: {} 2023-05-31T20:54:56.198 DEBUG:boto:Token: None 2023-05-31T20:54:56.199 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:54:56 GMT /pklxkd-59/ 2023-05-31T20:54:56.199 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:jiU8KdFvayyCgdc2KNAmenbHZMQ= 2023-05-31T20:54:56.200 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:54:56 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:jiU8KdFvayyCgdc2KNAmenbHZMQ=', 'Content-Length': '0'} 2023-05-31T20:54:56.202 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a5914cafb640b03f-006477b420-1e2e-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:54:56 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:54:56.202 DEBUG:boto:b'pklxkd-59obj1241000falseobj1242023-05-31T20:50:33.587Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:54:56.203 DEBUG:boto:path=/pklxkd-59/ 2023-05-31T20:54:56.203 DEBUG:boto:auth_path=/pklxkd-59/ 2023-05-31T20:54:56.204 DEBUG:boto:path=/pklxkd-59/?prefix=obj124 2023-05-31T20:54:56.204 DEBUG:boto:auth_path=/pklxkd-59/?prefix=obj124 2023-05-31T20:54:56.204 DEBUG:boto:Method: GET 2023-05-31T20:54:56.205 DEBUG:boto:Path: /pklxkd-59/?prefix=obj124 2023-05-31T20:54:56.205 DEBUG:boto:Data: 2023-05-31T20:54:56.206 DEBUG:boto:Headers: {} 2023-05-31T20:54:56.206 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:54:56.206 DEBUG:boto:Port: 8000 2023-05-31T20:54:56.207 DEBUG:boto:Params: {} 2023-05-31T20:54:56.207 DEBUG:boto:Token: None 2023-05-31T20:54:56.208 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:54:56 GMT /pklxkd-59/ 2023-05-31T20:54:56.208 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:jiU8KdFvayyCgdc2KNAmenbHZMQ= 2023-05-31T20:54:56.208 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:54:56 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:jiU8KdFvayyCgdc2KNAmenbHZMQ=', 'Content-Length': '0'} 2023-05-31T20:54:56.212 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000003b1c5a039842d116-006477b420-1630-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:54:56 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:54:56.213 DEBUG:boto:b'pklxkd-59obj1241000falseobj1242023-05-31T20:50:33.587Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:54:56.213 DEBUG:boto:path=/pklxkd-59/obj124 2023-05-31T20:54:56.214 DEBUG:boto:auth_path=/pklxkd-59/obj124 2023-05-31T20:54:56.214 DEBUG:boto:Method: HEAD 2023-05-31T20:54:56.214 DEBUG:boto:Path: /pklxkd-59/obj124 2023-05-31T20:54:56.215 DEBUG:boto:Data: 2023-05-31T20:54:56.215 DEBUG:boto:Headers: {} 2023-05-31T20:54:56.215 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:54:56.216 DEBUG:boto:Port: 8001 2023-05-31T20:54:56.216 DEBUG:boto:Params: {} 2023-05-31T20:54:56.216 DEBUG:boto:Token: None 2023-05-31T20:54:56.217 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:54:56 GMT /pklxkd-59/obj124 2023-05-31T20:54:56.217 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:6BjQco5WykMT1fgnM015s2LJhOo= 2023-05-31T20:54:56.218 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:54:56 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:6BjQco5WykMT1fgnM015s2LJhOo=', 'Content-Length': '0'} 2023-05-31T20:54:56.219 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:50:33 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003fcb9dc3a297ab64-006477b420-1e2e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:54:56 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:54:56.220 DEBUG:boto:path=/pklxkd-59/obj124 2023-05-31T20:54:56.220 DEBUG:boto:auth_path=/pklxkd-59/obj124 2023-05-31T20:54:56.221 DEBUG:boto:Method: HEAD 2023-05-31T20:54:56.221 DEBUG:boto:Path: /pklxkd-59/obj124 2023-05-31T20:54:56.221 DEBUG:boto:Data: 2023-05-31T20:54:56.222 DEBUG:boto:Headers: {} 2023-05-31T20:54:56.222 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:54:56.222 DEBUG:boto:Port: 8000 2023-05-31T20:54:56.223 DEBUG:boto:Params: {} 2023-05-31T20:54:56.223 DEBUG:boto:Token: None 2023-05-31T20:54:56.224 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:54:56 GMT /pklxkd-59/obj124 2023-05-31T20:54:56.224 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:6BjQco5WykMT1fgnM015s2LJhOo= 2023-05-31T20:54:56.224 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:54:56 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:6BjQco5WykMT1fgnM015s2LJhOo=', 'Content-Length': '0'} 2023-05-31T20:54:56.226 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:50:33 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006718ea3a0d1114c2-006477b420-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:54:56 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:54:56.227 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj124 2023-05-31T20:54:56.227 DEBUG:boto:path=/pklxkd-59/obj124 2023-05-31T20:54:56.227 DEBUG:boto:auth_path=/pklxkd-59/obj124 2023-05-31T20:54:56.228 DEBUG:boto:Method: GET 2023-05-31T20:54:56.228 DEBUG:boto:Path: /pklxkd-59/obj124 2023-05-31T20:54:56.228 DEBUG:boto:Data: 2023-05-31T20:54:56.229 DEBUG:boto:Headers: {} 2023-05-31T20:54:56.229 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:54:56.230 DEBUG:boto:Port: 8001 2023-05-31T20:54:56.230 DEBUG:boto:Params: {} 2023-05-31T20:54:56.230 DEBUG:boto:Token: None 2023-05-31T20:54:56.231 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:54:56 GMT /pklxkd-59/obj124 2023-05-31T20:54:56.231 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:qZF91OGqnAGvI6K6ih6/up2CUmM= 2023-05-31T20:54:56.232 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:54:56 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:qZF91OGqnAGvI6K6ih6/up2CUmM=', 'Content-Length': '0'} 2023-05-31T20:54:56.233 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:50:33 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008da691a2aeae9d4f-006477b420-1e2e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:54:56 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:54:56.234 DEBUG:boto:path=/pklxkd-59/obj124 2023-05-31T20:54:56.234 DEBUG:boto:auth_path=/pklxkd-59/obj124 2023-05-31T20:54:56.234 DEBUG:boto:Method: GET 2023-05-31T20:54:56.235 DEBUG:boto:Path: /pklxkd-59/obj124 2023-05-31T20:54:56.235 DEBUG:boto:Data: 2023-05-31T20:54:56.236 DEBUG:boto:Headers: {} 2023-05-31T20:54:56.236 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:54:56.236 DEBUG:boto:Port: 8000 2023-05-31T20:54:56.237 DEBUG:boto:Params: {} 2023-05-31T20:54:56.237 DEBUG:boto:Token: None 2023-05-31T20:54:56.238 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:54:56 GMT /pklxkd-59/obj124 2023-05-31T20:54:56.238 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:qZF91OGqnAGvI6K6ih6/up2CUmM= 2023-05-31T20:54:56.238 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:54:56 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:qZF91OGqnAGvI6K6ih6/up2CUmM=', 'Content-Length': '0'} 2023-05-31T20:54:56.240 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:50:33 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000972f2633d4300da6-006477b420-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:54:56 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:54:56.240 DEBUG:boto:path=/pklxkd-59/ 2023-05-31T20:54:56.241 DEBUG:boto:auth_path=/pklxkd-59/ 2023-05-31T20:54:56.241 DEBUG:boto:path=/pklxkd-59/?prefix=obj124 2023-05-31T20:54:56.242 DEBUG:boto:auth_path=/pklxkd-59/?prefix=obj124 2023-05-31T20:54:56.242 DEBUG:boto:Method: GET 2023-05-31T20:54:56.242 DEBUG:boto:Path: /pklxkd-59/?prefix=obj124 2023-05-31T20:54:56.243 DEBUG:boto:Data: 2023-05-31T20:54:56.243 DEBUG:boto:Headers: {} 2023-05-31T20:54:56.243 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:54:56.244 DEBUG:boto:Port: 8001 2023-05-31T20:54:56.244 DEBUG:boto:Params: {} 2023-05-31T20:54:56.244 DEBUG:boto:Token: None 2023-05-31T20:54:56.245 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:54:56 GMT /pklxkd-59/ 2023-05-31T20:54:56.245 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:jiU8KdFvayyCgdc2KNAmenbHZMQ= 2023-05-31T20:54:56.246 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:54:56 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:jiU8KdFvayyCgdc2KNAmenbHZMQ=', 'Content-Length': '0'} 2023-05-31T20:54:56.248 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000efdac7d34982c5cd-006477b420-1e2e-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:54:56 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:54:56.248 DEBUG:boto:b'pklxkd-59obj1241000falseobj1242023-05-31T20:50:33.587Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:54:56.249 DEBUG:boto:path=/pklxkd-59/ 2023-05-31T20:54:56.250 DEBUG:boto:auth_path=/pklxkd-59/ 2023-05-31T20:54:56.250 DEBUG:boto:path=/pklxkd-59/?prefix=obj124 2023-05-31T20:54:56.250 DEBUG:boto:auth_path=/pklxkd-59/?prefix=obj124 2023-05-31T20:54:56.251 DEBUG:boto:Method: GET 2023-05-31T20:54:56.251 DEBUG:boto:Path: /pklxkd-59/?prefix=obj124 2023-05-31T20:54:56.251 DEBUG:boto:Data: 2023-05-31T20:54:56.252 DEBUG:boto:Headers: {} 2023-05-31T20:54:56.252 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:54:56.252 DEBUG:boto:Port: 8000 2023-05-31T20:54:56.253 DEBUG:boto:Params: {} 2023-05-31T20:54:56.253 DEBUG:boto:Token: None 2023-05-31T20:54:56.253 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:54:56 GMT /pklxkd-59/ 2023-05-31T20:54:56.254 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:jiU8KdFvayyCgdc2KNAmenbHZMQ= 2023-05-31T20:54:56.254 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:54:56 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:jiU8KdFvayyCgdc2KNAmenbHZMQ=', 'Content-Length': '0'} 2023-05-31T20:54:56.258 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d477f98a4f636bf6-006477b420-1630-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:54:56 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:54:56.258 DEBUG:boto:b'pklxkd-59obj1241000falseobj1242023-05-31T20:50:33.587Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:54:56.259 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj145 2023-05-31T20:54:56.259 DEBUG:boto:path=/pklxkd-59/obj145 2023-05-31T20:54:56.260 DEBUG:boto:auth_path=/pklxkd-59/obj145 2023-05-31T20:54:56.260 DEBUG:boto:path=/pklxkd-59/obj145?versionId=null 2023-05-31T20:54:56.260 DEBUG:boto:auth_path=/pklxkd-59/obj145?versionId=null 2023-05-31T20:54:56.261 DEBUG:boto:Method: GET 2023-05-31T20:54:56.261 DEBUG:boto:Path: /pklxkd-59/obj145?versionId=null 2023-05-31T20:54:56.262 DEBUG:boto:Data: 2023-05-31T20:54:56.262 DEBUG:boto:Headers: {} 2023-05-31T20:54:56.262 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:54:56.263 DEBUG:boto:Port: 8001 2023-05-31T20:54:56.263 DEBUG:boto:Params: {} 2023-05-31T20:54:56.264 DEBUG:boto:Token: None 2023-05-31T20:54:56.264 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:54:56 GMT /pklxkd-59/obj145?versionId=null 2023-05-31T20:54:56.264 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:TWQSCvx9nT3C8WFmRbgCb0PQJho= 2023-05-31T20:54:56.265 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:54:56 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:TWQSCvx9nT3C8WFmRbgCb0PQJho=', 'Content-Length': '0'} 2023-05-31T20:54:56.267 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:48:13 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c6a773df1cb5f7b7-006477b420-1e2e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:54:56 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:54:56.267 DEBUG:boto:path=/pklxkd-59/obj145 2023-05-31T20:54:56.268 DEBUG:boto:auth_path=/pklxkd-59/obj145 2023-05-31T20:54:56.268 DEBUG:boto:path=/pklxkd-59/obj145?versionId=null 2023-05-31T20:54:56.269 DEBUG:boto:auth_path=/pklxkd-59/obj145?versionId=null 2023-05-31T20:54:56.269 DEBUG:boto:Method: GET 2023-05-31T20:54:56.269 DEBUG:boto:Path: /pklxkd-59/obj145?versionId=null 2023-05-31T20:54:56.270 DEBUG:boto:Data: 2023-05-31T20:54:56.270 DEBUG:boto:Headers: {} 2023-05-31T20:54:56.270 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:54:56.271 DEBUG:boto:Port: 8000 2023-05-31T20:54:56.271 DEBUG:boto:Params: {} 2023-05-31T20:54:56.271 DEBUG:boto:Token: None 2023-05-31T20:54:56.272 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:54:56 GMT /pklxkd-59/obj145?versionId=null 2023-05-31T20:54:56.272 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:TWQSCvx9nT3C8WFmRbgCb0PQJho= 2023-05-31T20:54:56.273 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:54:56 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:TWQSCvx9nT3C8WFmRbgCb0PQJho=', 'Content-Length': '0'} 2023-05-31T20:54:56.275 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:48:13 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000058f0486caa861855-006477b420-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:54:56 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:54:56.316 DEBUG:boto:path=/pklxkd-59/obj145 2023-05-31T20:54:56.317 DEBUG:boto:auth_path=/pklxkd-59/obj145 2023-05-31T20:54:56.317 DEBUG:boto:path=/pklxkd-59/obj145?versionId=null 2023-05-31T20:54:56.317 DEBUG:boto:auth_path=/pklxkd-59/obj145?versionId=null 2023-05-31T20:54:56.318 DEBUG:boto:Method: HEAD 2023-05-31T20:54:56.318 DEBUG:boto:Path: /pklxkd-59/obj145?versionId=null 2023-05-31T20:54:56.318 DEBUG:boto:Data: 2023-05-31T20:54:56.319 DEBUG:boto:Headers: {} 2023-05-31T20:54:56.319 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:54:56.319 DEBUG:boto:Port: 8001 2023-05-31T20:54:56.320 DEBUG:boto:Params: {} 2023-05-31T20:54:56.320 DEBUG:boto:Token: None 2023-05-31T20:54:56.321 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:54:56 GMT /pklxkd-59/obj145?versionId=null 2023-05-31T20:54:56.321 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:ZQpB1mqS8tJSJbxfYSB6JIwE2bw= 2023-05-31T20:54:56.321 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:54:56 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:ZQpB1mqS8tJSJbxfYSB6JIwE2bw=', 'Content-Length': '0'} 2023-05-31T20:54:56.323 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:48:13 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002cf408e35f17adab-006477b420-1e2e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:54:56 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:54:56.324 DEBUG:boto:path=/pklxkd-59/obj145 2023-05-31T20:54:56.324 DEBUG:boto:auth_path=/pklxkd-59/obj145 2023-05-31T20:54:56.324 DEBUG:boto:path=/pklxkd-59/obj145?versionId=null 2023-05-31T20:54:56.325 DEBUG:boto:auth_path=/pklxkd-59/obj145?versionId=null 2023-05-31T20:54:56.325 DEBUG:boto:Method: HEAD 2023-05-31T20:54:56.325 DEBUG:boto:Path: /pklxkd-59/obj145?versionId=null 2023-05-31T20:54:56.326 DEBUG:boto:Data: 2023-05-31T20:54:56.326 DEBUG:boto:Headers: {} 2023-05-31T20:54:56.326 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:54:56.327 DEBUG:boto:Port: 8000 2023-05-31T20:54:56.327 DEBUG:boto:Params: {} 2023-05-31T20:54:56.327 DEBUG:boto:Token: None 2023-05-31T20:54:56.328 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:54:56 GMT /pklxkd-59/obj145?versionId=null 2023-05-31T20:54:56.328 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:ZQpB1mqS8tJSJbxfYSB6JIwE2bw= 2023-05-31T20:54:56.329 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:54:56 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:ZQpB1mqS8tJSJbxfYSB6JIwE2bw=', 'Content-Length': '0'} 2023-05-31T20:54:56.330 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:48:13 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006b4351a6f43fc58a-006477b420-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:54:56 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:54:56.331 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj145 2023-05-31T20:54:56.331 DEBUG:boto:path=/pklxkd-59/obj145 2023-05-31T20:54:56.332 DEBUG:boto:auth_path=/pklxkd-59/obj145 2023-05-31T20:54:56.332 DEBUG:boto:path=/pklxkd-59/obj145?versionId=null 2023-05-31T20:54:56.332 DEBUG:boto:auth_path=/pklxkd-59/obj145?versionId=null 2023-05-31T20:54:56.333 DEBUG:boto:Method: GET 2023-05-31T20:54:56.333 DEBUG:boto:Path: /pklxkd-59/obj145?versionId=null 2023-05-31T20:54:56.333 DEBUG:boto:Data: 2023-05-31T20:54:56.334 DEBUG:boto:Headers: {} 2023-05-31T20:54:56.334 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:54:56.334 DEBUG:boto:Port: 8001 2023-05-31T20:54:56.335 DEBUG:boto:Params: {} 2023-05-31T20:54:56.335 DEBUG:boto:Token: None 2023-05-31T20:54:56.336 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:54:56 GMT /pklxkd-59/obj145?versionId=null 2023-05-31T20:54:56.336 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:TWQSCvx9nT3C8WFmRbgCb0PQJho= 2023-05-31T20:54:56.337 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:54:56 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:TWQSCvx9nT3C8WFmRbgCb0PQJho=', 'Content-Length': '0'} 2023-05-31T20:54:56.338 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:48:13 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000088541794f0626773-006477b420-1e2e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:54:56 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:54:56.339 DEBUG:boto:path=/pklxkd-59/obj145 2023-05-31T20:54:56.339 DEBUG:boto:auth_path=/pklxkd-59/obj145 2023-05-31T20:54:56.340 DEBUG:boto:path=/pklxkd-59/obj145?versionId=null 2023-05-31T20:54:56.340 DEBUG:boto:auth_path=/pklxkd-59/obj145?versionId=null 2023-05-31T20:54:56.340 DEBUG:boto:Method: GET 2023-05-31T20:54:56.341 DEBUG:boto:Path: /pklxkd-59/obj145?versionId=null 2023-05-31T20:54:56.341 DEBUG:boto:Data: 2023-05-31T20:54:56.341 DEBUG:boto:Headers: {} 2023-05-31T20:54:56.342 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:54:56.342 DEBUG:boto:Port: 8000 2023-05-31T20:54:56.342 DEBUG:boto:Params: {} 2023-05-31T20:54:56.343 DEBUG:boto:Token: None 2023-05-31T20:54:56.343 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:54:56 GMT /pklxkd-59/obj145?versionId=null 2023-05-31T20:54:56.344 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:TWQSCvx9nT3C8WFmRbgCb0PQJho= 2023-05-31T20:54:56.344 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:54:56 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:TWQSCvx9nT3C8WFmRbgCb0PQJho=', 'Content-Length': '0'} 2023-05-31T20:54:56.346 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:48:13 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a72895cc1665699e-006477b420-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:54:56 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:54:56.346 DEBUG:boto:path=/pklxkd-59/ 2023-05-31T20:54:56.347 DEBUG:boto:auth_path=/pklxkd-59/ 2023-05-31T20:54:56.347 DEBUG:boto:path=/pklxkd-59/?prefix=obj145 2023-05-31T20:54:56.347 DEBUG:boto:auth_path=/pklxkd-59/?prefix=obj145 2023-05-31T20:54:56.348 DEBUG:boto:Method: GET 2023-05-31T20:54:56.348 DEBUG:boto:Path: /pklxkd-59/?prefix=obj145 2023-05-31T20:54:56.348 DEBUG:boto:Data: 2023-05-31T20:54:56.349 DEBUG:boto:Headers: {} 2023-05-31T20:54:56.349 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:54:56.350 DEBUG:boto:Port: 8001 2023-05-31T20:54:56.350 DEBUG:boto:Params: {} 2023-05-31T20:54:56.350 DEBUG:boto:Token: None 2023-05-31T20:54:56.351 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:54:56 GMT /pklxkd-59/ 2023-05-31T20:54:56.351 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:jiU8KdFvayyCgdc2KNAmenbHZMQ= 2023-05-31T20:54:56.352 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:54:56 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:jiU8KdFvayyCgdc2KNAmenbHZMQ=', 'Content-Length': '0'} 2023-05-31T20:54:56.354 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000010412bad0c307954-006477b420-1e2e-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:54:56 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:54:56.354 DEBUG:boto:b'pklxkd-59obj1451000falseobj1452023-05-31T20:48:13.107Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:54:56.355 DEBUG:boto:path=/pklxkd-59/ 2023-05-31T20:54:56.355 DEBUG:boto:auth_path=/pklxkd-59/ 2023-05-31T20:54:56.356 DEBUG:boto:path=/pklxkd-59/?prefix=obj145 2023-05-31T20:54:56.356 DEBUG:boto:auth_path=/pklxkd-59/?prefix=obj145 2023-05-31T20:54:56.356 DEBUG:boto:Method: GET 2023-05-31T20:54:56.357 DEBUG:boto:Path: /pklxkd-59/?prefix=obj145 2023-05-31T20:54:56.357 DEBUG:boto:Data: 2023-05-31T20:54:56.357 DEBUG:boto:Headers: {} 2023-05-31T20:54:56.358 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:54:56.358 DEBUG:boto:Port: 8000 2023-05-31T20:54:56.358 DEBUG:boto:Params: {} 2023-05-31T20:54:56.359 DEBUG:boto:Token: None 2023-05-31T20:54:56.359 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:54:56 GMT /pklxkd-59/ 2023-05-31T20:54:56.360 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:jiU8KdFvayyCgdc2KNAmenbHZMQ= 2023-05-31T20:54:56.360 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:54:56 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:jiU8KdFvayyCgdc2KNAmenbHZMQ=', 'Content-Length': '0'} 2023-05-31T20:54:56.364 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000fb44eb76666d87a0-006477b420-1630-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:54:56 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:54:56.364 DEBUG:boto:b'pklxkd-59obj1451000falseobj1452023-05-31T20:48:13.107Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:54:56.365 DEBUG:boto:path=/pklxkd-59/obj145 2023-05-31T20:54:56.365 DEBUG:boto:auth_path=/pklxkd-59/obj145 2023-05-31T20:54:56.366 DEBUG:boto:Method: HEAD 2023-05-31T20:54:56.366 DEBUG:boto:Path: /pklxkd-59/obj145 2023-05-31T20:54:56.366 DEBUG:boto:Data: 2023-05-31T20:54:56.367 DEBUG:boto:Headers: {} 2023-05-31T20:54:56.367 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:54:56.367 DEBUG:boto:Port: 8001 2023-05-31T20:54:56.368 DEBUG:boto:Params: {} 2023-05-31T20:54:56.368 DEBUG:boto:Token: None 2023-05-31T20:54:56.368 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:54:56 GMT /pklxkd-59/obj145 2023-05-31T20:54:56.369 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:pA+MF64wRlE2VUWwzC1b/heqS7Q= 2023-05-31T20:54:56.369 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:54:56 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:pA+MF64wRlE2VUWwzC1b/heqS7Q=', 'Content-Length': '0'} 2023-05-31T20:54:56.371 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:48:13 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003b91218a7c1e51ee-006477b420-1e2e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:54:56 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:54:56.372 DEBUG:boto:path=/pklxkd-59/obj145 2023-05-31T20:54:56.372 DEBUG:boto:auth_path=/pklxkd-59/obj145 2023-05-31T20:54:56.372 DEBUG:boto:Method: HEAD 2023-05-31T20:54:56.373 DEBUG:boto:Path: /pklxkd-59/obj145 2023-05-31T20:54:56.373 DEBUG:boto:Data: 2023-05-31T20:54:56.373 DEBUG:boto:Headers: {} 2023-05-31T20:54:56.374 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:54:56.374 DEBUG:boto:Port: 8000 2023-05-31T20:54:56.375 DEBUG:boto:Params: {} 2023-05-31T20:54:56.375 DEBUG:boto:Token: None 2023-05-31T20:54:56.375 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:54:56 GMT /pklxkd-59/obj145 2023-05-31T20:54:56.376 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:pA+MF64wRlE2VUWwzC1b/heqS7Q= 2023-05-31T20:54:56.376 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:54:56 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:pA+MF64wRlE2VUWwzC1b/heqS7Q=', 'Content-Length': '0'} 2023-05-31T20:54:56.378 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:48:13 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000d34b55b4a0a4c7f-006477b420-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:54:56 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:54:56.378 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj145 2023-05-31T20:54:56.379 DEBUG:boto:path=/pklxkd-59/obj145 2023-05-31T20:54:56.379 DEBUG:boto:auth_path=/pklxkd-59/obj145 2023-05-31T20:54:56.380 DEBUG:boto:Method: GET 2023-05-31T20:54:56.380 DEBUG:boto:Path: /pklxkd-59/obj145 2023-05-31T20:54:56.380 DEBUG:boto:Data: 2023-05-31T20:54:56.381 DEBUG:boto:Headers: {} 2023-05-31T20:54:56.381 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:54:56.382 DEBUG:boto:Port: 8001 2023-05-31T20:54:56.382 DEBUG:boto:Params: {} 2023-05-31T20:54:56.382 DEBUG:boto:Token: None 2023-05-31T20:54:56.383 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:54:56 GMT /pklxkd-59/obj145 2023-05-31T20:54:56.383 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:/AMsbeWNusEdymf1pAeSQanG3CM= 2023-05-31T20:54:56.384 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:54:56 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:/AMsbeWNusEdymf1pAeSQanG3CM=', 'Content-Length': '0'} 2023-05-31T20:54:56.385 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:48:13 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c8441220b8f73ccd-006477b420-1e2e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:54:56 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:54:56.386 DEBUG:boto:path=/pklxkd-59/obj145 2023-05-31T20:54:56.387 DEBUG:boto:auth_path=/pklxkd-59/obj145 2023-05-31T20:54:56.387 DEBUG:boto:Method: GET 2023-05-31T20:54:56.387 DEBUG:boto:Path: /pklxkd-59/obj145 2023-05-31T20:54:56.388 DEBUG:boto:Data: 2023-05-31T20:54:56.388 DEBUG:boto:Headers: {} 2023-05-31T20:54:56.389 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:54:56.389 DEBUG:boto:Port: 8000 2023-05-31T20:54:56.389 DEBUG:boto:Params: {} 2023-05-31T20:54:56.390 DEBUG:boto:Token: None 2023-05-31T20:54:56.390 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:54:56 GMT /pklxkd-59/obj145 2023-05-31T20:54:56.391 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:/AMsbeWNusEdymf1pAeSQanG3CM= 2023-05-31T20:54:56.391 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:54:56 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:/AMsbeWNusEdymf1pAeSQanG3CM=', 'Content-Length': '0'} 2023-05-31T20:54:56.393 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:48:13 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000047aaf89934a607d-006477b420-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:54:56 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:54:56.394 DEBUG:boto:path=/pklxkd-59/ 2023-05-31T20:54:56.394 DEBUG:boto:auth_path=/pklxkd-59/ 2023-05-31T20:54:56.395 DEBUG:boto:path=/pklxkd-59/?prefix=obj145 2023-05-31T20:54:56.395 DEBUG:boto:auth_path=/pklxkd-59/?prefix=obj145 2023-05-31T20:54:56.395 DEBUG:boto:Method: GET 2023-05-31T20:54:56.396 DEBUG:boto:Path: /pklxkd-59/?prefix=obj145 2023-05-31T20:54:56.396 DEBUG:boto:Data: 2023-05-31T20:54:56.397 DEBUG:boto:Headers: {} 2023-05-31T20:54:56.397 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:54:56.398 DEBUG:boto:Port: 8001 2023-05-31T20:54:56.398 DEBUG:boto:Params: {} 2023-05-31T20:54:56.398 DEBUG:boto:Token: None 2023-05-31T20:54:56.399 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:54:56 GMT /pklxkd-59/ 2023-05-31T20:54:56.399 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:jiU8KdFvayyCgdc2KNAmenbHZMQ= 2023-05-31T20:54:56.400 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:54:56 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:jiU8KdFvayyCgdc2KNAmenbHZMQ=', 'Content-Length': '0'} 2023-05-31T20:54:56.402 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c18f11b6a96ed948-006477b420-1e2e-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:54:56 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:54:56.403 DEBUG:boto:b'pklxkd-59obj1451000falseobj1452023-05-31T20:48:13.107Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:54:56.403 DEBUG:boto:path=/pklxkd-59/ 2023-05-31T20:54:56.404 DEBUG:boto:auth_path=/pklxkd-59/ 2023-05-31T20:54:56.404 DEBUG:boto:path=/pklxkd-59/?prefix=obj145 2023-05-31T20:54:56.405 DEBUG:boto:auth_path=/pklxkd-59/?prefix=obj145 2023-05-31T20:54:56.405 DEBUG:boto:Method: GET 2023-05-31T20:54:56.405 DEBUG:boto:Path: /pklxkd-59/?prefix=obj145 2023-05-31T20:54:56.406 DEBUG:boto:Data: 2023-05-31T20:54:56.406 DEBUG:boto:Headers: {} 2023-05-31T20:54:56.406 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:54:56.407 DEBUG:boto:Port: 8000 2023-05-31T20:54:56.407 DEBUG:boto:Params: {} 2023-05-31T20:54:56.408 DEBUG:boto:Token: None 2023-05-31T20:54:56.408 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:54:56 GMT /pklxkd-59/ 2023-05-31T20:54:56.409 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:jiU8KdFvayyCgdc2KNAmenbHZMQ= 2023-05-31T20:54:56.409 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:54:56 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:jiU8KdFvayyCgdc2KNAmenbHZMQ=', 'Content-Length': '0'} 2023-05-31T20:54:56.413 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b9296ab4379600cb-006477b420-1630-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:54:56 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:54:56.413 DEBUG:boto:b'pklxkd-59obj1451000falseobj1452023-05-31T20:48:13.107Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:54:56.414 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj148 2023-05-31T20:54:56.414 DEBUG:boto:path=/pklxkd-59/obj148 2023-05-31T20:54:56.415 DEBUG:boto:auth_path=/pklxkd-59/obj148 2023-05-31T20:54:56.415 DEBUG:boto:path=/pklxkd-59/obj148?versionId=null 2023-05-31T20:54:56.415 DEBUG:boto:auth_path=/pklxkd-59/obj148?versionId=null 2023-05-31T20:54:56.416 DEBUG:boto:Method: GET 2023-05-31T20:54:56.416 DEBUG:boto:Path: /pklxkd-59/obj148?versionId=null 2023-05-31T20:54:56.416 DEBUG:boto:Data: 2023-05-31T20:54:56.417 DEBUG:boto:Headers: {} 2023-05-31T20:54:56.417 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:54:56.417 DEBUG:boto:Port: 8001 2023-05-31T20:54:56.418 DEBUG:boto:Params: {} 2023-05-31T20:54:56.418 DEBUG:boto:Token: None 2023-05-31T20:54:56.419 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:54:56 GMT /pklxkd-59/obj148?versionId=null 2023-05-31T20:54:56.419 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:DYzQ1UIC7+xdaBcOvhga370m00U= 2023-05-31T20:54:56.420 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:54:56 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:DYzQ1UIC7+xdaBcOvhga370m00U=', 'Content-Length': '0'} 2023-05-31T20:54:56.422 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:52:15 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005bd1926f4884c782-006477b420-1e2e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:54:56 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:54:56.423 DEBUG:boto:path=/pklxkd-59/obj148 2023-05-31T20:54:56.423 DEBUG:boto:auth_path=/pklxkd-59/obj148 2023-05-31T20:54:56.423 DEBUG:boto:path=/pklxkd-59/obj148?versionId=null 2023-05-31T20:54:56.424 DEBUG:boto:auth_path=/pklxkd-59/obj148?versionId=null 2023-05-31T20:54:56.424 DEBUG:boto:Method: GET 2023-05-31T20:54:56.425 DEBUG:boto:Path: /pklxkd-59/obj148?versionId=null 2023-05-31T20:54:56.425 DEBUG:boto:Data: 2023-05-31T20:54:56.425 DEBUG:boto:Headers: {} 2023-05-31T20:54:56.426 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:54:56.426 DEBUG:boto:Port: 8000 2023-05-31T20:54:56.426 DEBUG:boto:Params: {} 2023-05-31T20:54:56.427 DEBUG:boto:Token: None 2023-05-31T20:54:56.427 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:54:56 GMT /pklxkd-59/obj148?versionId=null 2023-05-31T20:54:56.427 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:DYzQ1UIC7+xdaBcOvhga370m00U= 2023-05-31T20:54:56.428 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:54:56 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:DYzQ1UIC7+xdaBcOvhga370m00U=', 'Content-Length': '0'} 2023-05-31T20:54:56.430 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:52:15 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000bd6ce5ecfb258cff-006477b420-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:54:56 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:54:56.472 DEBUG:boto:path=/pklxkd-59/obj148 2023-05-31T20:54:56.473 DEBUG:boto:auth_path=/pklxkd-59/obj148 2023-05-31T20:54:56.473 DEBUG:boto:path=/pklxkd-59/obj148?versionId=null 2023-05-31T20:54:56.473 DEBUG:boto:auth_path=/pklxkd-59/obj148?versionId=null 2023-05-31T20:54:56.474 DEBUG:boto:Method: HEAD 2023-05-31T20:54:56.474 DEBUG:boto:Path: /pklxkd-59/obj148?versionId=null 2023-05-31T20:54:56.474 DEBUG:boto:Data: 2023-05-31T20:54:56.475 DEBUG:boto:Headers: {} 2023-05-31T20:54:56.475 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:54:56.475 DEBUG:boto:Port: 8001 2023-05-31T20:54:56.476 DEBUG:boto:Params: {} 2023-05-31T20:54:56.476 DEBUG:boto:Token: None 2023-05-31T20:54:56.477 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:54:56 GMT /pklxkd-59/obj148?versionId=null 2023-05-31T20:54:56.477 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:48ce0KGs3WckahzpnGZiGVaq4dQ= 2023-05-31T20:54:56.477 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:54:56 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:48ce0KGs3WckahzpnGZiGVaq4dQ=', 'Content-Length': '0'} 2023-05-31T20:54:56.479 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:52:15 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005deb88b5ad0c0d8b-006477b420-1e2e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:54:56 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:54:56.480 DEBUG:boto:path=/pklxkd-59/obj148 2023-05-31T20:54:56.480 DEBUG:boto:auth_path=/pklxkd-59/obj148 2023-05-31T20:54:56.480 DEBUG:boto:path=/pklxkd-59/obj148?versionId=null 2023-05-31T20:54:56.481 DEBUG:boto:auth_path=/pklxkd-59/obj148?versionId=null 2023-05-31T20:54:56.481 DEBUG:boto:Method: HEAD 2023-05-31T20:54:56.481 DEBUG:boto:Path: /pklxkd-59/obj148?versionId=null 2023-05-31T20:54:56.482 DEBUG:boto:Data: 2023-05-31T20:54:56.482 DEBUG:boto:Headers: {} 2023-05-31T20:54:56.482 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:54:56.483 DEBUG:boto:Port: 8000 2023-05-31T20:54:56.483 DEBUG:boto:Params: {} 2023-05-31T20:54:56.483 DEBUG:boto:Token: None 2023-05-31T20:54:56.484 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:54:56 GMT /pklxkd-59/obj148?versionId=null 2023-05-31T20:54:56.484 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:48ce0KGs3WckahzpnGZiGVaq4dQ= 2023-05-31T20:54:56.485 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:54:56 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:48ce0KGs3WckahzpnGZiGVaq4dQ=', 'Content-Length': '0'} 2023-05-31T20:54:56.486 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:52:15 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000041b6e2b314e06abe-006477b420-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:54:56 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:54:56.487 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj148 2023-05-31T20:54:56.487 DEBUG:boto:path=/pklxkd-59/obj148 2023-05-31T20:54:56.488 DEBUG:boto:auth_path=/pklxkd-59/obj148 2023-05-31T20:54:56.488 DEBUG:boto:path=/pklxkd-59/obj148?versionId=null 2023-05-31T20:54:56.489 DEBUG:boto:auth_path=/pklxkd-59/obj148?versionId=null 2023-05-31T20:54:56.489 DEBUG:boto:Method: GET 2023-05-31T20:54:56.489 DEBUG:boto:Path: /pklxkd-59/obj148?versionId=null 2023-05-31T20:54:56.490 DEBUG:boto:Data: 2023-05-31T20:54:56.490 DEBUG:boto:Headers: {} 2023-05-31T20:54:56.490 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:54:56.491 DEBUG:boto:Port: 8001 2023-05-31T20:54:56.491 DEBUG:boto:Params: {} 2023-05-31T20:54:56.491 DEBUG:boto:Token: None 2023-05-31T20:54:56.492 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:54:56 GMT /pklxkd-59/obj148?versionId=null 2023-05-31T20:54:56.492 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:DYzQ1UIC7+xdaBcOvhga370m00U= 2023-05-31T20:54:56.493 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:54:56 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:DYzQ1UIC7+xdaBcOvhga370m00U=', 'Content-Length': '0'} 2023-05-31T20:54:56.494 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:52:15 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000ccb446536c1c2f7-006477b420-1e2e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:54:56 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:54:56.495 DEBUG:boto:path=/pklxkd-59/obj148 2023-05-31T20:54:56.495 DEBUG:boto:auth_path=/pklxkd-59/obj148 2023-05-31T20:54:56.496 DEBUG:boto:path=/pklxkd-59/obj148?versionId=null 2023-05-31T20:54:56.496 DEBUG:boto:auth_path=/pklxkd-59/obj148?versionId=null 2023-05-31T20:54:56.497 DEBUG:boto:Method: GET 2023-05-31T20:54:56.497 DEBUG:boto:Path: /pklxkd-59/obj148?versionId=null 2023-05-31T20:54:56.497 DEBUG:boto:Data: 2023-05-31T20:54:56.498 DEBUG:boto:Headers: {} 2023-05-31T20:54:56.498 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:54:56.498 DEBUG:boto:Port: 8000 2023-05-31T20:54:56.499 DEBUG:boto:Params: {} 2023-05-31T20:54:56.499 DEBUG:boto:Token: None 2023-05-31T20:54:56.500 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:54:56 GMT /pklxkd-59/obj148?versionId=null 2023-05-31T20:54:56.500 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:DYzQ1UIC7+xdaBcOvhga370m00U= 2023-05-31T20:54:56.501 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:54:56 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:DYzQ1UIC7+xdaBcOvhga370m00U=', 'Content-Length': '0'} 2023-05-31T20:54:56.502 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:52:15 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000dd23d42841ef8087-006477b420-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:54:56 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:54:56.503 DEBUG:boto:path=/pklxkd-59/ 2023-05-31T20:54:56.503 DEBUG:boto:auth_path=/pklxkd-59/ 2023-05-31T20:54:56.503 DEBUG:boto:path=/pklxkd-59/?prefix=obj148 2023-05-31T20:54:56.504 DEBUG:boto:auth_path=/pklxkd-59/?prefix=obj148 2023-05-31T20:54:56.504 DEBUG:boto:Method: GET 2023-05-31T20:54:56.505 DEBUG:boto:Path: /pklxkd-59/?prefix=obj148 2023-05-31T20:54:56.505 DEBUG:boto:Data: 2023-05-31T20:54:56.505 DEBUG:boto:Headers: {} 2023-05-31T20:54:56.506 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:54:56.506 DEBUG:boto:Port: 8001 2023-05-31T20:54:56.506 DEBUG:boto:Params: {} 2023-05-31T20:54:56.507 DEBUG:boto:Token: None 2023-05-31T20:54:56.507 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:54:56 GMT /pklxkd-59/ 2023-05-31T20:54:56.507 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:jiU8KdFvayyCgdc2KNAmenbHZMQ= 2023-05-31T20:54:56.508 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:54:56 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:jiU8KdFvayyCgdc2KNAmenbHZMQ=', 'Content-Length': '0'} 2023-05-31T20:54:56.510 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ca830b4298a7a8ef-006477b420-1e2e-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:54:56 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:54:56.511 DEBUG:boto:b'pklxkd-59obj1481000falseobj1482023-05-31T20:52:15.015Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:54:56.512 DEBUG:boto:path=/pklxkd-59/ 2023-05-31T20:54:56.512 DEBUG:boto:auth_path=/pklxkd-59/ 2023-05-31T20:54:56.512 DEBUG:boto:path=/pklxkd-59/?prefix=obj148 2023-05-31T20:54:56.513 DEBUG:boto:auth_path=/pklxkd-59/?prefix=obj148 2023-05-31T20:54:56.513 DEBUG:boto:Method: GET 2023-05-31T20:54:56.513 DEBUG:boto:Path: /pklxkd-59/?prefix=obj148 2023-05-31T20:54:56.514 DEBUG:boto:Data: 2023-05-31T20:54:56.514 DEBUG:boto:Headers: {} 2023-05-31T20:54:56.515 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:54:56.515 DEBUG:boto:Port: 8000 2023-05-31T20:54:56.515 DEBUG:boto:Params: {} 2023-05-31T20:54:56.516 DEBUG:boto:Token: None 2023-05-31T20:54:56.516 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:54:56 GMT /pklxkd-59/ 2023-05-31T20:54:56.516 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:jiU8KdFvayyCgdc2KNAmenbHZMQ= 2023-05-31T20:54:56.517 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:54:56 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:jiU8KdFvayyCgdc2KNAmenbHZMQ=', 'Content-Length': '0'} 2023-05-31T20:54:56.521 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f016b720064a17a7-006477b420-1630-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:54:56 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:54:56.521 DEBUG:boto:b'pklxkd-59obj1481000falseobj1482023-05-31T20:52:15.015Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:54:56.522 DEBUG:boto:path=/pklxkd-59/obj148 2023-05-31T20:54:56.522 DEBUG:boto:auth_path=/pklxkd-59/obj148 2023-05-31T20:54:56.523 DEBUG:boto:Method: HEAD 2023-05-31T20:54:56.523 DEBUG:boto:Path: /pklxkd-59/obj148 2023-05-31T20:54:56.523 DEBUG:boto:Data: 2023-05-31T20:54:56.524 DEBUG:boto:Headers: {} 2023-05-31T20:54:56.524 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:54:56.524 DEBUG:boto:Port: 8001 2023-05-31T20:54:56.525 DEBUG:boto:Params: {} 2023-05-31T20:54:56.525 DEBUG:boto:Token: None 2023-05-31T20:54:56.525 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:54:56 GMT /pklxkd-59/obj148 2023-05-31T20:54:56.526 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:iIQrb+ELsLMMHIuAB8NJTKBBiXM= 2023-05-31T20:54:56.526 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:54:56 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:iIQrb+ELsLMMHIuAB8NJTKBBiXM=', 'Content-Length': '0'} 2023-05-31T20:54:56.528 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:52:15 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d33a3729d911d94b-006477b420-1e2e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:54:56 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:54:56.528 DEBUG:boto:path=/pklxkd-59/obj148 2023-05-31T20:54:56.529 DEBUG:boto:auth_path=/pklxkd-59/obj148 2023-05-31T20:54:56.529 DEBUG:boto:Method: HEAD 2023-05-31T20:54:56.530 DEBUG:boto:Path: /pklxkd-59/obj148 2023-05-31T20:54:56.530 DEBUG:boto:Data: 2023-05-31T20:54:56.530 DEBUG:boto:Headers: {} 2023-05-31T20:54:56.531 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:54:56.531 DEBUG:boto:Port: 8000 2023-05-31T20:54:56.531 DEBUG:boto:Params: {} 2023-05-31T20:54:56.532 DEBUG:boto:Token: None 2023-05-31T20:54:56.532 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:54:56 GMT /pklxkd-59/obj148 2023-05-31T20:54:56.532 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:iIQrb+ELsLMMHIuAB8NJTKBBiXM= 2023-05-31T20:54:56.533 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:54:56 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:iIQrb+ELsLMMHIuAB8NJTKBBiXM=', 'Content-Length': '0'} 2023-05-31T20:54:56.534 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:52:15 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000036e02884918987a-006477b420-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:54:56 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:54:56.535 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj148 2023-05-31T20:54:56.535 DEBUG:boto:path=/pklxkd-59/obj148 2023-05-31T20:54:56.536 DEBUG:boto:auth_path=/pklxkd-59/obj148 2023-05-31T20:54:56.536 DEBUG:boto:Method: GET 2023-05-31T20:54:56.536 DEBUG:boto:Path: /pklxkd-59/obj148 2023-05-31T20:54:56.537 DEBUG:boto:Data: 2023-05-31T20:54:56.537 DEBUG:boto:Headers: {} 2023-05-31T20:54:56.537 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:54:56.538 DEBUG:boto:Port: 8001 2023-05-31T20:54:56.538 DEBUG:boto:Params: {} 2023-05-31T20:54:56.539 DEBUG:boto:Token: None 2023-05-31T20:54:56.539 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:54:56 GMT /pklxkd-59/obj148 2023-05-31T20:54:56.540 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:ofTf2dn0MmTY0oYpm7P6LuVR2lw= 2023-05-31T20:54:56.540 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:54:56 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:ofTf2dn0MmTY0oYpm7P6LuVR2lw=', 'Content-Length': '0'} 2023-05-31T20:54:56.542 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:52:15 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009f41d4e138f95c95-006477b420-1e2e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:54:56 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:54:56.542 DEBUG:boto:path=/pklxkd-59/obj148 2023-05-31T20:54:56.543 DEBUG:boto:auth_path=/pklxkd-59/obj148 2023-05-31T20:54:56.543 DEBUG:boto:Method: GET 2023-05-31T20:54:56.543 DEBUG:boto:Path: /pklxkd-59/obj148 2023-05-31T20:54:56.544 DEBUG:boto:Data: 2023-05-31T20:54:56.544 DEBUG:boto:Headers: {} 2023-05-31T20:54:56.545 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:54:56.545 DEBUG:boto:Port: 8000 2023-05-31T20:54:56.545 DEBUG:boto:Params: {} 2023-05-31T20:54:56.546 DEBUG:boto:Token: None 2023-05-31T20:54:56.546 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:54:56 GMT /pklxkd-59/obj148 2023-05-31T20:54:56.546 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:ofTf2dn0MmTY0oYpm7P6LuVR2lw= 2023-05-31T20:54:56.547 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:54:56 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:ofTf2dn0MmTY0oYpm7P6LuVR2lw=', 'Content-Length': '0'} 2023-05-31T20:54:56.548 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:52:15 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003798b94138f2cdc2-006477b420-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:54:56 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:54:56.549 DEBUG:boto:path=/pklxkd-59/ 2023-05-31T20:54:56.549 DEBUG:boto:auth_path=/pklxkd-59/ 2023-05-31T20:54:56.550 DEBUG:boto:path=/pklxkd-59/?prefix=obj148 2023-05-31T20:54:56.550 DEBUG:boto:auth_path=/pklxkd-59/?prefix=obj148 2023-05-31T20:54:56.551 DEBUG:boto:Method: GET 2023-05-31T20:54:56.551 DEBUG:boto:Path: /pklxkd-59/?prefix=obj148 2023-05-31T20:54:56.551 DEBUG:boto:Data: 2023-05-31T20:54:56.552 DEBUG:boto:Headers: {} 2023-05-31T20:54:56.552 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:54:56.552 DEBUG:boto:Port: 8001 2023-05-31T20:54:56.553 DEBUG:boto:Params: {} 2023-05-31T20:54:56.553 DEBUG:boto:Token: None 2023-05-31T20:54:56.553 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:54:56 GMT /pklxkd-59/ 2023-05-31T20:54:56.554 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:jiU8KdFvayyCgdc2KNAmenbHZMQ= 2023-05-31T20:54:56.554 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:54:56 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:jiU8KdFvayyCgdc2KNAmenbHZMQ=', 'Content-Length': '0'} 2023-05-31T20:54:56.557 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000000e5f641c77cdd621-006477b420-1e2e-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:54:56 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:54:56.557 DEBUG:boto:b'pklxkd-59obj1481000falseobj1482023-05-31T20:52:15.015Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:54:56.558 DEBUG:boto:path=/pklxkd-59/ 2023-05-31T20:54:56.558 DEBUG:boto:auth_path=/pklxkd-59/ 2023-05-31T20:54:56.558 DEBUG:boto:path=/pklxkd-59/?prefix=obj148 2023-05-31T20:54:56.559 DEBUG:boto:auth_path=/pklxkd-59/?prefix=obj148 2023-05-31T20:54:56.559 DEBUG:boto:Method: GET 2023-05-31T20:54:56.560 DEBUG:boto:Path: /pklxkd-59/?prefix=obj148 2023-05-31T20:54:56.560 DEBUG:boto:Data: 2023-05-31T20:54:56.560 DEBUG:boto:Headers: {} 2023-05-31T20:54:56.561 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:54:56.561 DEBUG:boto:Port: 8000 2023-05-31T20:54:56.561 DEBUG:boto:Params: {} 2023-05-31T20:54:56.562 DEBUG:boto:Token: None 2023-05-31T20:54:56.562 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:54:56 GMT /pklxkd-59/ 2023-05-31T20:54:56.563 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:jiU8KdFvayyCgdc2KNAmenbHZMQ= 2023-05-31T20:54:56.563 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:54:56 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:jiU8KdFvayyCgdc2KNAmenbHZMQ=', 'Content-Length': '0'} 2023-05-31T20:54:56.567 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000283bca06b9aa65f1-006477b420-1630-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:54:56 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:54:56.608 DEBUG:boto:b'pklxkd-59obj1481000falseobj1482023-05-31T20:52:15.015Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:54:56.609 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj155 2023-05-31T20:54:56.609 DEBUG:boto:path=/pklxkd-59/obj155 2023-05-31T20:54:56.610 DEBUG:boto:auth_path=/pklxkd-59/obj155 2023-05-31T20:54:56.610 DEBUG:boto:path=/pklxkd-59/obj155?versionId=null 2023-05-31T20:54:56.610 DEBUG:boto:auth_path=/pklxkd-59/obj155?versionId=null 2023-05-31T20:54:56.611 DEBUG:boto:Method: GET 2023-05-31T20:54:56.611 DEBUG:boto:Path: /pklxkd-59/obj155?versionId=null 2023-05-31T20:54:56.611 DEBUG:boto:Data: 2023-05-31T20:54:56.612 DEBUG:boto:Headers: {} 2023-05-31T20:54:56.612 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:54:56.613 DEBUG:boto:Port: 8001 2023-05-31T20:54:56.613 DEBUG:boto:Params: {} 2023-05-31T20:54:56.613 DEBUG:boto:Token: None 2023-05-31T20:54:56.614 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:54:56 GMT /pklxkd-59/obj155?versionId=null 2023-05-31T20:54:56.614 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:bmWENRrYd4RzQF3j5aw1aJe8e1U= 2023-05-31T20:54:56.615 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:54:56 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:bmWENRrYd4RzQF3j5aw1aJe8e1U=', 'Content-Length': '0'} 2023-05-31T20:54:56.617 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:50:33 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b37e51a2864c85c6-006477b420-1e2e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:54:56 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:54:56.617 DEBUG:boto:path=/pklxkd-59/obj155 2023-05-31T20:54:56.618 DEBUG:boto:auth_path=/pklxkd-59/obj155 2023-05-31T20:54:56.618 DEBUG:boto:path=/pklxkd-59/obj155?versionId=null 2023-05-31T20:54:56.618 DEBUG:boto:auth_path=/pklxkd-59/obj155?versionId=null 2023-05-31T20:54:56.619 DEBUG:boto:Method: GET 2023-05-31T20:54:56.619 DEBUG:boto:Path: /pklxkd-59/obj155?versionId=null 2023-05-31T20:54:56.620 DEBUG:boto:Data: 2023-05-31T20:54:56.620 DEBUG:boto:Headers: {} 2023-05-31T20:54:56.620 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:54:56.621 DEBUG:boto:Port: 8000 2023-05-31T20:54:56.621 DEBUG:boto:Params: {} 2023-05-31T20:54:56.621 DEBUG:boto:Token: None 2023-05-31T20:54:56.622 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:54:56 GMT /pklxkd-59/obj155?versionId=null 2023-05-31T20:54:56.622 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:bmWENRrYd4RzQF3j5aw1aJe8e1U= 2023-05-31T20:54:56.623 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:54:56 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:bmWENRrYd4RzQF3j5aw1aJe8e1U=', 'Content-Length': '0'} 2023-05-31T20:54:56.624 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:50:33 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d32ddae282469613-006477b420-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:54:56 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:54:56.625 DEBUG:boto:path=/pklxkd-59/obj155 2023-05-31T20:54:56.625 DEBUG:boto:auth_path=/pklxkd-59/obj155 2023-05-31T20:54:56.626 DEBUG:boto:path=/pklxkd-59/obj155?versionId=null 2023-05-31T20:54:56.626 DEBUG:boto:auth_path=/pklxkd-59/obj155?versionId=null 2023-05-31T20:54:56.626 DEBUG:boto:Method: HEAD 2023-05-31T20:54:56.627 DEBUG:boto:Path: /pklxkd-59/obj155?versionId=null 2023-05-31T20:54:56.627 DEBUG:boto:Data: 2023-05-31T20:54:56.627 DEBUG:boto:Headers: {} 2023-05-31T20:54:56.628 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:54:56.628 DEBUG:boto:Port: 8001 2023-05-31T20:54:56.628 DEBUG:boto:Params: {} 2023-05-31T20:54:56.629 DEBUG:boto:Token: None 2023-05-31T20:54:56.629 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:54:56 GMT /pklxkd-59/obj155?versionId=null 2023-05-31T20:54:56.629 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:IySU55c18+gQXvS3R/uuFXy2PmI= 2023-05-31T20:54:56.630 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:54:56 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:IySU55c18+gQXvS3R/uuFXy2PmI=', 'Content-Length': '0'} 2023-05-31T20:54:56.632 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:50:33 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000bb86c2f7aabc113b-006477b420-1e2e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:54:56 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:54:56.632 DEBUG:boto:path=/pklxkd-59/obj155 2023-05-31T20:54:56.632 DEBUG:boto:auth_path=/pklxkd-59/obj155 2023-05-31T20:54:56.633 DEBUG:boto:path=/pklxkd-59/obj155?versionId=null 2023-05-31T20:54:56.633 DEBUG:boto:auth_path=/pklxkd-59/obj155?versionId=null 2023-05-31T20:54:56.633 DEBUG:boto:Method: HEAD 2023-05-31T20:54:56.634 DEBUG:boto:Path: /pklxkd-59/obj155?versionId=null 2023-05-31T20:54:56.634 DEBUG:boto:Data: 2023-05-31T20:54:56.634 DEBUG:boto:Headers: {} 2023-05-31T20:54:56.635 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:54:56.635 DEBUG:boto:Port: 8000 2023-05-31T20:54:56.635 DEBUG:boto:Params: {} 2023-05-31T20:54:56.636 DEBUG:boto:Token: None 2023-05-31T20:54:56.636 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:54:56 GMT /pklxkd-59/obj155?versionId=null 2023-05-31T20:54:56.637 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:IySU55c18+gQXvS3R/uuFXy2PmI= 2023-05-31T20:54:56.637 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:54:56 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:IySU55c18+gQXvS3R/uuFXy2PmI=', 'Content-Length': '0'} 2023-05-31T20:54:56.639 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:50:33 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000025bd3fcac059ba0a-006477b420-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:54:56 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:54:56.639 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj155 2023-05-31T20:54:56.640 DEBUG:boto:path=/pklxkd-59/obj155 2023-05-31T20:54:56.640 DEBUG:boto:auth_path=/pklxkd-59/obj155 2023-05-31T20:54:56.640 DEBUG:boto:path=/pklxkd-59/obj155?versionId=null 2023-05-31T20:54:56.641 DEBUG:boto:auth_path=/pklxkd-59/obj155?versionId=null 2023-05-31T20:54:56.641 DEBUG:boto:Method: GET 2023-05-31T20:54:56.641 DEBUG:boto:Path: /pklxkd-59/obj155?versionId=null 2023-05-31T20:54:56.642 DEBUG:boto:Data: 2023-05-31T20:54:56.642 DEBUG:boto:Headers: {} 2023-05-31T20:54:56.643 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:54:56.643 DEBUG:boto:Port: 8001 2023-05-31T20:54:56.643 DEBUG:boto:Params: {} 2023-05-31T20:54:56.644 DEBUG:boto:Token: None 2023-05-31T20:54:56.644 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:54:56 GMT /pklxkd-59/obj155?versionId=null 2023-05-31T20:54:56.644 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:bmWENRrYd4RzQF3j5aw1aJe8e1U= 2023-05-31T20:54:56.645 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:54:56 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:bmWENRrYd4RzQF3j5aw1aJe8e1U=', 'Content-Length': '0'} 2023-05-31T20:54:56.646 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:50:33 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000059866998a8e533a7-006477b420-1e2e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:54:56 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:54:56.688 DEBUG:boto:path=/pklxkd-59/obj155 2023-05-31T20:54:56.689 DEBUG:boto:auth_path=/pklxkd-59/obj155 2023-05-31T20:54:56.689 DEBUG:boto:path=/pklxkd-59/obj155?versionId=null 2023-05-31T20:54:56.689 DEBUG:boto:auth_path=/pklxkd-59/obj155?versionId=null 2023-05-31T20:54:56.690 DEBUG:boto:Method: GET 2023-05-31T20:54:56.690 DEBUG:boto:Path: /pklxkd-59/obj155?versionId=null 2023-05-31T20:54:56.690 DEBUG:boto:Data: 2023-05-31T20:54:56.691 DEBUG:boto:Headers: {} 2023-05-31T20:54:56.691 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:54:56.691 DEBUG:boto:Port: 8000 2023-05-31T20:54:56.692 DEBUG:boto:Params: {} 2023-05-31T20:54:56.692 DEBUG:boto:Token: None 2023-05-31T20:54:56.693 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:54:56 GMT /pklxkd-59/obj155?versionId=null 2023-05-31T20:54:56.693 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:bmWENRrYd4RzQF3j5aw1aJe8e1U= 2023-05-31T20:54:56.693 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:54:56 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:bmWENRrYd4RzQF3j5aw1aJe8e1U=', 'Content-Length': '0'} 2023-05-31T20:54:56.695 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:50:33 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006202b864b9266753-006477b420-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:54:56 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:54:56.696 DEBUG:boto:path=/pklxkd-59/ 2023-05-31T20:54:56.696 DEBUG:boto:auth_path=/pklxkd-59/ 2023-05-31T20:54:56.697 DEBUG:boto:path=/pklxkd-59/?prefix=obj155 2023-05-31T20:54:56.697 DEBUG:boto:auth_path=/pklxkd-59/?prefix=obj155 2023-05-31T20:54:56.697 DEBUG:boto:Method: GET 2023-05-31T20:54:56.698 DEBUG:boto:Path: /pklxkd-59/?prefix=obj155 2023-05-31T20:54:56.698 DEBUG:boto:Data: 2023-05-31T20:54:56.698 DEBUG:boto:Headers: {} 2023-05-31T20:54:56.699 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:54:56.699 DEBUG:boto:Port: 8001 2023-05-31T20:54:56.699 DEBUG:boto:Params: {} 2023-05-31T20:54:56.700 DEBUG:boto:Token: None 2023-05-31T20:54:56.700 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:54:56 GMT /pklxkd-59/ 2023-05-31T20:54:56.701 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:jiU8KdFvayyCgdc2KNAmenbHZMQ= 2023-05-31T20:54:56.701 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:54:56 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:jiU8KdFvayyCgdc2KNAmenbHZMQ=', 'Content-Length': '0'} 2023-05-31T20:54:56.703 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000002c8711d07d2988bd-006477b420-1e2e-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:54:56 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:54:56.704 DEBUG:boto:b'pklxkd-59obj1551000falseobj1552023-05-31T20:50:33.605Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:54:56.704 DEBUG:boto:path=/pklxkd-59/ 2023-05-31T20:54:56.705 DEBUG:boto:auth_path=/pklxkd-59/ 2023-05-31T20:54:56.705 DEBUG:boto:path=/pklxkd-59/?prefix=obj155 2023-05-31T20:54:56.706 DEBUG:boto:auth_path=/pklxkd-59/?prefix=obj155 2023-05-31T20:54:56.706 DEBUG:boto:Method: GET 2023-05-31T20:54:56.706 DEBUG:boto:Path: /pklxkd-59/?prefix=obj155 2023-05-31T20:54:56.707 DEBUG:boto:Data: 2023-05-31T20:54:56.707 DEBUG:boto:Headers: {} 2023-05-31T20:54:56.707 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:54:56.708 DEBUG:boto:Port: 8000 2023-05-31T20:54:56.708 DEBUG:boto:Params: {} 2023-05-31T20:54:56.709 DEBUG:boto:Token: None 2023-05-31T20:54:56.709 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:54:56 GMT /pklxkd-59/ 2023-05-31T20:54:56.709 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:jiU8KdFvayyCgdc2KNAmenbHZMQ= 2023-05-31T20:54:56.710 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:54:56 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:jiU8KdFvayyCgdc2KNAmenbHZMQ=', 'Content-Length': '0'} 2023-05-31T20:54:56.714 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000009a71366a13946299-006477b420-1630-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:54:56 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:54:56.715 DEBUG:boto:b'pklxkd-59obj1551000falseobj1552023-05-31T20:50:33.605Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:54:56.716 DEBUG:boto:path=/pklxkd-59/obj155 2023-05-31T20:54:56.716 DEBUG:boto:auth_path=/pklxkd-59/obj155 2023-05-31T20:54:56.716 DEBUG:boto:Method: HEAD 2023-05-31T20:54:56.717 DEBUG:boto:Path: /pklxkd-59/obj155 2023-05-31T20:54:56.717 DEBUG:boto:Data: 2023-05-31T20:54:56.717 DEBUG:boto:Headers: {} 2023-05-31T20:54:56.718 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:54:56.718 DEBUG:boto:Port: 8001 2023-05-31T20:54:56.718 DEBUG:boto:Params: {} 2023-05-31T20:54:56.719 DEBUG:boto:Token: None 2023-05-31T20:54:56.719 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:54:56 GMT /pklxkd-59/obj155 2023-05-31T20:54:56.719 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:I/tHUtupylS78pPM7mYRSbWsibk= 2023-05-31T20:54:56.720 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:54:56 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:I/tHUtupylS78pPM7mYRSbWsibk=', 'Content-Length': '0'} 2023-05-31T20:54:56.721 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:50:33 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000003d4455a3ad95f97-006477b420-1e2e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:54:56 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:54:56.722 DEBUG:boto:path=/pklxkd-59/obj155 2023-05-31T20:54:56.722 DEBUG:boto:auth_path=/pklxkd-59/obj155 2023-05-31T20:54:56.723 DEBUG:boto:Method: HEAD 2023-05-31T20:54:56.723 DEBUG:boto:Path: /pklxkd-59/obj155 2023-05-31T20:54:56.723 DEBUG:boto:Data: 2023-05-31T20:54:56.724 DEBUG:boto:Headers: {} 2023-05-31T20:54:56.724 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:54:56.724 DEBUG:boto:Port: 8000 2023-05-31T20:54:56.725 DEBUG:boto:Params: {} 2023-05-31T20:54:56.725 DEBUG:boto:Token: None 2023-05-31T20:54:56.725 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:54:56 GMT /pklxkd-59/obj155 2023-05-31T20:54:56.726 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:I/tHUtupylS78pPM7mYRSbWsibk= 2023-05-31T20:54:56.726 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:54:56 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:I/tHUtupylS78pPM7mYRSbWsibk=', 'Content-Length': '0'} 2023-05-31T20:54:56.728 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:50:33 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ddc4d0a251aaea70-006477b420-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:54:56 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:54:56.729 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj155 2023-05-31T20:54:56.729 DEBUG:boto:path=/pklxkd-59/obj155 2023-05-31T20:54:56.729 DEBUG:boto:auth_path=/pklxkd-59/obj155 2023-05-31T20:54:56.730 DEBUG:boto:Method: GET 2023-05-31T20:54:56.730 DEBUG:boto:Path: /pklxkd-59/obj155 2023-05-31T20:54:56.730 DEBUG:boto:Data: 2023-05-31T20:54:56.731 DEBUG:boto:Headers: {} 2023-05-31T20:54:56.731 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:54:56.732 DEBUG:boto:Port: 8001 2023-05-31T20:54:56.732 DEBUG:boto:Params: {} 2023-05-31T20:54:56.732 DEBUG:boto:Token: None 2023-05-31T20:54:56.733 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:54:56 GMT /pklxkd-59/obj155 2023-05-31T20:54:56.733 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:VxBzbtNhphGf4szhkEz4J78VRjo= 2023-05-31T20:54:56.734 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:54:56 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:VxBzbtNhphGf4szhkEz4J78VRjo=', 'Content-Length': '0'} 2023-05-31T20:54:56.735 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:50:33 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000d33ef79648eb4d0-006477b420-1e2e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:54:56 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:54:56.736 DEBUG:boto:path=/pklxkd-59/obj155 2023-05-31T20:54:56.736 DEBUG:boto:auth_path=/pklxkd-59/obj155 2023-05-31T20:54:56.736 DEBUG:boto:Method: GET 2023-05-31T20:54:56.737 DEBUG:boto:Path: /pklxkd-59/obj155 2023-05-31T20:54:56.737 DEBUG:boto:Data: 2023-05-31T20:54:56.738 DEBUG:boto:Headers: {} 2023-05-31T20:54:56.738 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:54:56.738 DEBUG:boto:Port: 8000 2023-05-31T20:54:56.739 DEBUG:boto:Params: {} 2023-05-31T20:54:56.739 DEBUG:boto:Token: None 2023-05-31T20:54:56.740 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:54:56 GMT /pklxkd-59/obj155 2023-05-31T20:54:56.741 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:VxBzbtNhphGf4szhkEz4J78VRjo= 2023-05-31T20:54:56.741 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:54:56 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:VxBzbtNhphGf4szhkEz4J78VRjo=', 'Content-Length': '0'} 2023-05-31T20:54:56.743 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:50:33 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000058e3a3782e754c1b-006477b420-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:54:56 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:54:56.784 DEBUG:boto:path=/pklxkd-59/ 2023-05-31T20:54:56.785 DEBUG:boto:auth_path=/pklxkd-59/ 2023-05-31T20:54:56.785 DEBUG:boto:path=/pklxkd-59/?prefix=obj155 2023-05-31T20:54:56.786 DEBUG:boto:auth_path=/pklxkd-59/?prefix=obj155 2023-05-31T20:54:56.786 DEBUG:boto:Method: GET 2023-05-31T20:54:56.786 DEBUG:boto:Path: /pklxkd-59/?prefix=obj155 2023-05-31T20:54:56.787 DEBUG:boto:Data: 2023-05-31T20:54:56.787 DEBUG:boto:Headers: {} 2023-05-31T20:54:56.788 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:54:56.788 DEBUG:boto:Port: 8001 2023-05-31T20:54:56.788 DEBUG:boto:Params: {} 2023-05-31T20:54:56.789 DEBUG:boto:Token: None 2023-05-31T20:54:56.789 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:54:56 GMT /pklxkd-59/ 2023-05-31T20:54:56.790 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:jiU8KdFvayyCgdc2KNAmenbHZMQ= 2023-05-31T20:54:56.790 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:54:56 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:jiU8KdFvayyCgdc2KNAmenbHZMQ=', 'Content-Length': '0'} 2023-05-31T20:54:56.792 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b62cc94413f2c1b5-006477b420-1e2e-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:54:56 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:54:56.793 DEBUG:boto:b'pklxkd-59obj1551000falseobj1552023-05-31T20:50:33.605Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:54:56.794 DEBUG:boto:path=/pklxkd-59/ 2023-05-31T20:54:56.794 DEBUG:boto:auth_path=/pklxkd-59/ 2023-05-31T20:54:56.794 DEBUG:boto:path=/pklxkd-59/?prefix=obj155 2023-05-31T20:54:56.795 DEBUG:boto:auth_path=/pklxkd-59/?prefix=obj155 2023-05-31T20:54:56.795 DEBUG:boto:Method: GET 2023-05-31T20:54:56.795 DEBUG:boto:Path: /pklxkd-59/?prefix=obj155 2023-05-31T20:54:56.796 DEBUG:boto:Data: 2023-05-31T20:54:56.796 DEBUG:boto:Headers: {} 2023-05-31T20:54:56.796 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:54:56.797 DEBUG:boto:Port: 8000 2023-05-31T20:54:56.797 DEBUG:boto:Params: {} 2023-05-31T20:54:56.798 DEBUG:boto:Token: None 2023-05-31T20:54:56.798 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:54:56 GMT /pklxkd-59/ 2023-05-31T20:54:56.798 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:jiU8KdFvayyCgdc2KNAmenbHZMQ= 2023-05-31T20:54:56.799 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:54:56 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:jiU8KdFvayyCgdc2KNAmenbHZMQ=', 'Content-Length': '0'} 2023-05-31T20:54:56.802 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000006023d03c9dead1d8-006477b420-1630-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:54:56 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:54:56.803 DEBUG:boto:b'pklxkd-59obj1551000falseobj1552023-05-31T20:50:33.605Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:54:56.803 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj17 2023-05-31T20:54:56.804 DEBUG:boto:path=/pklxkd-59/obj17 2023-05-31T20:54:56.804 DEBUG:boto:auth_path=/pklxkd-59/obj17 2023-05-31T20:54:56.805 DEBUG:boto:path=/pklxkd-59/obj17?versionId=null 2023-05-31T20:54:56.805 DEBUG:boto:auth_path=/pklxkd-59/obj17?versionId=null 2023-05-31T20:54:56.805 DEBUG:boto:Method: GET 2023-05-31T20:54:56.806 DEBUG:boto:Path: /pklxkd-59/obj17?versionId=null 2023-05-31T20:54:56.806 DEBUG:boto:Data: 2023-05-31T20:54:56.806 DEBUG:boto:Headers: {} 2023-05-31T20:54:56.807 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:54:56.807 DEBUG:boto:Port: 8001 2023-05-31T20:54:56.807 DEBUG:boto:Params: {} 2023-05-31T20:54:56.808 DEBUG:boto:Token: None 2023-05-31T20:54:56.808 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:54:56 GMT /pklxkd-59/obj17?versionId=null 2023-05-31T20:54:56.809 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:rIfTAQRWj4rZCfXnDz/r88XJtws= 2023-05-31T20:54:56.809 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:54:56 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:rIfTAQRWj4rZCfXnDz/r88XJtws=', 'Content-Length': '0'} 2023-05-31T20:54:56.812 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:43:07 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003ab5598e45408e35-006477b420-1e2e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:54:56 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:54:56.812 DEBUG:boto:path=/pklxkd-59/obj17 2023-05-31T20:54:56.813 DEBUG:boto:auth_path=/pklxkd-59/obj17 2023-05-31T20:54:56.813 DEBUG:boto:path=/pklxkd-59/obj17?versionId=null 2023-05-31T20:54:56.813 DEBUG:boto:auth_path=/pklxkd-59/obj17?versionId=null 2023-05-31T20:54:56.814 DEBUG:boto:Method: GET 2023-05-31T20:54:56.814 DEBUG:boto:Path: /pklxkd-59/obj17?versionId=null 2023-05-31T20:54:56.815 DEBUG:boto:Data: 2023-05-31T20:54:56.815 DEBUG:boto:Headers: {} 2023-05-31T20:54:56.815 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:54:56.816 DEBUG:boto:Port: 8000 2023-05-31T20:54:56.816 DEBUG:boto:Params: {} 2023-05-31T20:54:56.816 DEBUG:boto:Token: None 2023-05-31T20:54:56.817 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:54:56 GMT /pklxkd-59/obj17?versionId=null 2023-05-31T20:54:56.817 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:rIfTAQRWj4rZCfXnDz/r88XJtws= 2023-05-31T20:54:56.818 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:54:56 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:rIfTAQRWj4rZCfXnDz/r88XJtws=', 'Content-Length': '0'} 2023-05-31T20:54:56.819 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:43:07 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f1076e10797c0ba3-006477b420-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:54:56 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:54:56.820 DEBUG:boto:path=/pklxkd-59/obj17 2023-05-31T20:54:56.820 DEBUG:boto:auth_path=/pklxkd-59/obj17 2023-05-31T20:54:56.821 DEBUG:boto:path=/pklxkd-59/obj17?versionId=null 2023-05-31T20:54:56.821 DEBUG:boto:auth_path=/pklxkd-59/obj17?versionId=null 2023-05-31T20:54:56.821 DEBUG:boto:Method: HEAD 2023-05-31T20:54:56.822 DEBUG:boto:Path: /pklxkd-59/obj17?versionId=null 2023-05-31T20:54:56.822 DEBUG:boto:Data: 2023-05-31T20:54:56.822 DEBUG:boto:Headers: {} 2023-05-31T20:54:56.823 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:54:56.823 DEBUG:boto:Port: 8001 2023-05-31T20:54:56.823 DEBUG:boto:Params: {} 2023-05-31T20:54:56.824 DEBUG:boto:Token: None 2023-05-31T20:54:56.824 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:54:56 GMT /pklxkd-59/obj17?versionId=null 2023-05-31T20:54:56.825 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:aQbSwzEo03xb6Qw+I8QzOF1nvhE= 2023-05-31T20:54:56.825 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:54:56 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:aQbSwzEo03xb6Qw+I8QzOF1nvhE=', 'Content-Length': '0'} 2023-05-31T20:54:56.827 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:43:07 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c78c01e3537311c9-006477b420-1e2e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:54:56 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:54:56.827 DEBUG:boto:path=/pklxkd-59/obj17 2023-05-31T20:54:56.828 DEBUG:boto:auth_path=/pklxkd-59/obj17 2023-05-31T20:54:56.828 DEBUG:boto:path=/pklxkd-59/obj17?versionId=null 2023-05-31T20:54:56.828 DEBUG:boto:auth_path=/pklxkd-59/obj17?versionId=null 2023-05-31T20:54:56.829 DEBUG:boto:Method: HEAD 2023-05-31T20:54:56.829 DEBUG:boto:Path: /pklxkd-59/obj17?versionId=null 2023-05-31T20:54:56.829 DEBUG:boto:Data: 2023-05-31T20:54:56.830 DEBUG:boto:Headers: {} 2023-05-31T20:54:56.830 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:54:56.830 DEBUG:boto:Port: 8000 2023-05-31T20:54:56.831 DEBUG:boto:Params: {} 2023-05-31T20:54:56.831 DEBUG:boto:Token: None 2023-05-31T20:54:56.831 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:54:56 GMT /pklxkd-59/obj17?versionId=null 2023-05-31T20:54:56.832 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:aQbSwzEo03xb6Qw+I8QzOF1nvhE= 2023-05-31T20:54:56.832 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:54:56 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:aQbSwzEo03xb6Qw+I8QzOF1nvhE=', 'Content-Length': '0'} 2023-05-31T20:54:56.834 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:43:07 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a148936e426e9a6b-006477b420-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:54:56 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:54:56.835 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj17 2023-05-31T20:54:56.835 DEBUG:boto:path=/pklxkd-59/obj17 2023-05-31T20:54:56.835 DEBUG:boto:auth_path=/pklxkd-59/obj17 2023-05-31T20:54:56.836 DEBUG:boto:path=/pklxkd-59/obj17?versionId=null 2023-05-31T20:54:56.836 DEBUG:boto:auth_path=/pklxkd-59/obj17?versionId=null 2023-05-31T20:54:56.836 DEBUG:boto:Method: GET 2023-05-31T20:54:56.837 DEBUG:boto:Path: /pklxkd-59/obj17?versionId=null 2023-05-31T20:54:56.837 DEBUG:boto:Data: 2023-05-31T20:54:56.837 DEBUG:boto:Headers: {} 2023-05-31T20:54:56.838 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:54:56.838 DEBUG:boto:Port: 8001 2023-05-31T20:54:56.838 DEBUG:boto:Params: {} 2023-05-31T20:54:56.839 DEBUG:boto:Token: None 2023-05-31T20:54:56.839 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:54:56 GMT /pklxkd-59/obj17?versionId=null 2023-05-31T20:54:56.840 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:rIfTAQRWj4rZCfXnDz/r88XJtws= 2023-05-31T20:54:56.840 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:54:56 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:rIfTAQRWj4rZCfXnDz/r88XJtws=', 'Content-Length': '0'} 2023-05-31T20:54:56.842 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:43:07 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000631af6c1f2120654-006477b420-1e2e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:54:56 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:54:56.842 DEBUG:boto:path=/pklxkd-59/obj17 2023-05-31T20:54:56.843 DEBUG:boto:auth_path=/pklxkd-59/obj17 2023-05-31T20:54:56.843 DEBUG:boto:path=/pklxkd-59/obj17?versionId=null 2023-05-31T20:54:56.843 DEBUG:boto:auth_path=/pklxkd-59/obj17?versionId=null 2023-05-31T20:54:56.844 DEBUG:boto:Method: GET 2023-05-31T20:54:56.844 DEBUG:boto:Path: /pklxkd-59/obj17?versionId=null 2023-05-31T20:54:56.845 DEBUG:boto:Data: 2023-05-31T20:54:56.845 DEBUG:boto:Headers: {} 2023-05-31T20:54:56.845 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:54:56.846 DEBUG:boto:Port: 8000 2023-05-31T20:54:56.846 DEBUG:boto:Params: {} 2023-05-31T20:54:56.847 DEBUG:boto:Token: None 2023-05-31T20:54:56.847 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:54:56 GMT /pklxkd-59/obj17?versionId=null 2023-05-31T20:54:56.848 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:rIfTAQRWj4rZCfXnDz/r88XJtws= 2023-05-31T20:54:56.848 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:54:56 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:rIfTAQRWj4rZCfXnDz/r88XJtws=', 'Content-Length': '0'} 2023-05-31T20:54:56.850 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:43:07 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004279056c24c31fff-006477b420-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:54:56 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:54:56.851 DEBUG:boto:path=/pklxkd-59/ 2023-05-31T20:54:56.851 DEBUG:boto:auth_path=/pklxkd-59/ 2023-05-31T20:54:56.851 DEBUG:boto:path=/pklxkd-59/?prefix=obj17 2023-05-31T20:54:56.852 DEBUG:boto:auth_path=/pklxkd-59/?prefix=obj17 2023-05-31T20:54:56.852 DEBUG:boto:Method: GET 2023-05-31T20:54:56.852 DEBUG:boto:Path: /pklxkd-59/?prefix=obj17 2023-05-31T20:54:56.853 DEBUG:boto:Data: 2023-05-31T20:54:56.853 DEBUG:boto:Headers: {} 2023-05-31T20:54:56.853 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:54:56.854 DEBUG:boto:Port: 8001 2023-05-31T20:54:56.854 DEBUG:boto:Params: {} 2023-05-31T20:54:56.855 DEBUG:boto:Token: None 2023-05-31T20:54:56.855 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:54:56 GMT /pklxkd-59/ 2023-05-31T20:54:56.855 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:jiU8KdFvayyCgdc2KNAmenbHZMQ= 2023-05-31T20:54:56.856 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:54:56 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:jiU8KdFvayyCgdc2KNAmenbHZMQ=', 'Content-Length': '0'} 2023-05-31T20:54:56.858 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d46ce928a2ce8996-006477b420-1e2e-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:54:56 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:54:56.858 DEBUG:boto:b'pklxkd-59obj171000falseobj172023-05-31T20:43:07.977Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:54:56.859 DEBUG:boto:path=/pklxkd-59/ 2023-05-31T20:54:56.859 DEBUG:boto:auth_path=/pklxkd-59/ 2023-05-31T20:54:56.860 DEBUG:boto:path=/pklxkd-59/?prefix=obj17 2023-05-31T20:54:56.860 DEBUG:boto:auth_path=/pklxkd-59/?prefix=obj17 2023-05-31T20:54:56.860 DEBUG:boto:Method: GET 2023-05-31T20:54:56.861 DEBUG:boto:Path: /pklxkd-59/?prefix=obj17 2023-05-31T20:54:56.861 DEBUG:boto:Data: 2023-05-31T20:54:56.861 DEBUG:boto:Headers: {} 2023-05-31T20:54:56.862 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:54:56.862 DEBUG:boto:Port: 8000 2023-05-31T20:54:56.862 DEBUG:boto:Params: {} 2023-05-31T20:54:56.863 DEBUG:boto:Token: None 2023-05-31T20:54:56.863 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:54:56 GMT /pklxkd-59/ 2023-05-31T20:54:56.864 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:jiU8KdFvayyCgdc2KNAmenbHZMQ= 2023-05-31T20:54:56.864 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:54:56 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:jiU8KdFvayyCgdc2KNAmenbHZMQ=', 'Content-Length': '0'} 2023-05-31T20:54:56.868 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000fddf175e0b75f35c-006477b420-1630-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:54:56 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:54:56.868 DEBUG:boto:b'pklxkd-59obj171000falseobj172023-05-31T20:43:07.977Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:54:56.869 DEBUG:boto:path=/pklxkd-59/obj17 2023-05-31T20:54:56.869 DEBUG:boto:auth_path=/pklxkd-59/obj17 2023-05-31T20:54:56.869 DEBUG:boto:Method: HEAD 2023-05-31T20:54:56.870 DEBUG:boto:Path: /pklxkd-59/obj17 2023-05-31T20:54:56.870 DEBUG:boto:Data: 2023-05-31T20:54:56.870 DEBUG:boto:Headers: {} 2023-05-31T20:54:56.871 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:54:56.871 DEBUG:boto:Port: 8001 2023-05-31T20:54:56.871 DEBUG:boto:Params: {} 2023-05-31T20:54:56.872 DEBUG:boto:Token: None 2023-05-31T20:54:56.872 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:54:56 GMT /pklxkd-59/obj17 2023-05-31T20:54:56.873 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:siYLwvG1X44CgQKgmzp8mXfwnwI= 2023-05-31T20:54:56.873 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:54:56 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:siYLwvG1X44CgQKgmzp8mXfwnwI=', 'Content-Length': '0'} 2023-05-31T20:54:56.875 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:43:07 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009e9acfa69104e981-006477b420-1e2e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:54:56 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:54:56.875 DEBUG:boto:path=/pklxkd-59/obj17 2023-05-31T20:54:56.876 DEBUG:boto:auth_path=/pklxkd-59/obj17 2023-05-31T20:54:56.876 DEBUG:boto:Method: HEAD 2023-05-31T20:54:56.876 DEBUG:boto:Path: /pklxkd-59/obj17 2023-05-31T20:54:56.877 DEBUG:boto:Data: 2023-05-31T20:54:56.877 DEBUG:boto:Headers: {} 2023-05-31T20:54:56.877 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:54:56.878 DEBUG:boto:Port: 8000 2023-05-31T20:54:56.878 DEBUG:boto:Params: {} 2023-05-31T20:54:56.878 DEBUG:boto:Token: None 2023-05-31T20:54:56.879 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:54:56 GMT /pklxkd-59/obj17 2023-05-31T20:54:56.879 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:siYLwvG1X44CgQKgmzp8mXfwnwI= 2023-05-31T20:54:56.879 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:54:56 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:siYLwvG1X44CgQKgmzp8mXfwnwI=', 'Content-Length': '0'} 2023-05-31T20:54:56.881 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:43:07 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000078b0f1b86ed90cdf-006477b420-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:54:56 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:54:56.882 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj17 2023-05-31T20:54:56.882 DEBUG:boto:path=/pklxkd-59/obj17 2023-05-31T20:54:56.882 DEBUG:boto:auth_path=/pklxkd-59/obj17 2023-05-31T20:54:56.883 DEBUG:boto:Method: GET 2023-05-31T20:54:56.883 DEBUG:boto:Path: /pklxkd-59/obj17 2023-05-31T20:54:56.883 DEBUG:boto:Data: 2023-05-31T20:54:56.884 DEBUG:boto:Headers: {} 2023-05-31T20:54:56.884 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:54:56.885 DEBUG:boto:Port: 8001 2023-05-31T20:54:56.885 DEBUG:boto:Params: {} 2023-05-31T20:54:56.885 DEBUG:boto:Token: None 2023-05-31T20:54:56.886 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:54:56 GMT /pklxkd-59/obj17 2023-05-31T20:54:56.886 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:Np3/aDIlPt54q+4Clj5g7p5bUBs= 2023-05-31T20:54:56.887 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:54:56 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:Np3/aDIlPt54q+4Clj5g7p5bUBs=', 'Content-Length': '0'} 2023-05-31T20:54:56.888 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:43:07 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000012e84948d4a5df25-006477b420-1e2e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:54:56 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:54:56.889 DEBUG:boto:path=/pklxkd-59/obj17 2023-05-31T20:54:56.889 DEBUG:boto:auth_path=/pklxkd-59/obj17 2023-05-31T20:54:56.889 DEBUG:boto:Method: GET 2023-05-31T20:54:56.890 DEBUG:boto:Path: /pklxkd-59/obj17 2023-05-31T20:54:56.890 DEBUG:boto:Data: 2023-05-31T20:54:56.891 DEBUG:boto:Headers: {} 2023-05-31T20:54:56.891 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:54:56.891 DEBUG:boto:Port: 8000 2023-05-31T20:54:56.892 DEBUG:boto:Params: {} 2023-05-31T20:54:56.892 DEBUG:boto:Token: None 2023-05-31T20:54:56.892 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:54:56 GMT /pklxkd-59/obj17 2023-05-31T20:54:56.893 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:Np3/aDIlPt54q+4Clj5g7p5bUBs= 2023-05-31T20:54:56.893 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:54:56 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:Np3/aDIlPt54q+4Clj5g7p5bUBs=', 'Content-Length': '0'} 2023-05-31T20:54:56.895 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:43:07 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ad561f82ad07a235-006477b420-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:54:56 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:54:56.936 DEBUG:boto:path=/pklxkd-59/ 2023-05-31T20:54:56.937 DEBUG:boto:auth_path=/pklxkd-59/ 2023-05-31T20:54:56.937 DEBUG:boto:path=/pklxkd-59/?prefix=obj17 2023-05-31T20:54:56.937 DEBUG:boto:auth_path=/pklxkd-59/?prefix=obj17 2023-05-31T20:54:56.938 DEBUG:boto:Method: GET 2023-05-31T20:54:56.938 DEBUG:boto:Path: /pklxkd-59/?prefix=obj17 2023-05-31T20:54:56.938 DEBUG:boto:Data: 2023-05-31T20:54:56.939 DEBUG:boto:Headers: {} 2023-05-31T20:54:56.939 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:54:56.939 DEBUG:boto:Port: 8001 2023-05-31T20:54:56.940 DEBUG:boto:Params: {} 2023-05-31T20:54:56.940 DEBUG:boto:Token: None 2023-05-31T20:54:56.941 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:54:56 GMT /pklxkd-59/ 2023-05-31T20:54:56.941 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:jiU8KdFvayyCgdc2KNAmenbHZMQ= 2023-05-31T20:54:56.941 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:54:56 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:jiU8KdFvayyCgdc2KNAmenbHZMQ=', 'Content-Length': '0'} 2023-05-31T20:54:56.944 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f0d13454aa442713-006477b420-1e2e-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:54:56 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:54:56.944 DEBUG:boto:b'pklxkd-59obj171000falseobj172023-05-31T20:43:07.977Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:54:56.945 DEBUG:boto:path=/pklxkd-59/ 2023-05-31T20:54:56.945 DEBUG:boto:auth_path=/pklxkd-59/ 2023-05-31T20:54:56.946 DEBUG:boto:path=/pklxkd-59/?prefix=obj17 2023-05-31T20:54:56.946 DEBUG:boto:auth_path=/pklxkd-59/?prefix=obj17 2023-05-31T20:54:56.946 DEBUG:boto:Method: GET 2023-05-31T20:54:56.947 DEBUG:boto:Path: /pklxkd-59/?prefix=obj17 2023-05-31T20:54:56.947 DEBUG:boto:Data: 2023-05-31T20:54:56.947 DEBUG:boto:Headers: {} 2023-05-31T20:54:56.948 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:54:56.948 DEBUG:boto:Port: 8000 2023-05-31T20:54:56.948 DEBUG:boto:Params: {} 2023-05-31T20:54:56.949 DEBUG:boto:Token: None 2023-05-31T20:54:56.949 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:54:56 GMT /pklxkd-59/ 2023-05-31T20:54:56.950 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:jiU8KdFvayyCgdc2KNAmenbHZMQ= 2023-05-31T20:54:56.950 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:54:56 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:jiU8KdFvayyCgdc2KNAmenbHZMQ=', 'Content-Length': '0'} 2023-05-31T20:54:56.954 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000066b8ae9f32bf8891-006477b420-1630-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:54:56 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:54:56.954 DEBUG:boto:b'pklxkd-59obj171000falseobj172023-05-31T20:43:07.977Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:54:56.955 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj185 2023-05-31T20:54:56.955 DEBUG:boto:path=/pklxkd-59/obj185 2023-05-31T20:54:56.956 DEBUG:boto:auth_path=/pklxkd-59/obj185 2023-05-31T20:54:56.956 DEBUG:boto:path=/pklxkd-59/obj185?versionId=null 2023-05-31T20:54:56.956 DEBUG:boto:auth_path=/pklxkd-59/obj185?versionId=null 2023-05-31T20:54:56.957 DEBUG:boto:Method: GET 2023-05-31T20:54:56.957 DEBUG:boto:Path: /pklxkd-59/obj185?versionId=null 2023-05-31T20:54:56.957 DEBUG:boto:Data: 2023-05-31T20:54:56.958 DEBUG:boto:Headers: {} 2023-05-31T20:54:56.958 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:54:56.958 DEBUG:boto:Port: 8001 2023-05-31T20:54:56.959 DEBUG:boto:Params: {} 2023-05-31T20:54:56.959 DEBUG:boto:Token: None 2023-05-31T20:54:56.960 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:54:56 GMT /pklxkd-59/obj185?versionId=null 2023-05-31T20:54:56.960 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:s7zyK538cJo169BtBx+W1kAadZw= 2023-05-31T20:54:56.960 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:54:56 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:s7zyK538cJo169BtBx+W1kAadZw=', 'Content-Length': '0'} 2023-05-31T20:54:56.962 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:52:15 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002866cd17c8aafc07-006477b420-1e2e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:54:56 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:54:56.963 DEBUG:boto:path=/pklxkd-59/obj185 2023-05-31T20:54:56.964 DEBUG:boto:auth_path=/pklxkd-59/obj185 2023-05-31T20:54:56.964 DEBUG:boto:path=/pklxkd-59/obj185?versionId=null 2023-05-31T20:54:56.964 DEBUG:boto:auth_path=/pklxkd-59/obj185?versionId=null 2023-05-31T20:54:56.965 DEBUG:boto:Method: GET 2023-05-31T20:54:56.965 DEBUG:boto:Path: /pklxkd-59/obj185?versionId=null 2023-05-31T20:54:56.965 DEBUG:boto:Data: 2023-05-31T20:54:56.966 DEBUG:boto:Headers: {} 2023-05-31T20:54:56.966 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:54:56.966 DEBUG:boto:Port: 8000 2023-05-31T20:54:56.967 DEBUG:boto:Params: {} 2023-05-31T20:54:56.967 DEBUG:boto:Token: None 2023-05-31T20:54:56.967 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:54:56 GMT /pklxkd-59/obj185?versionId=null 2023-05-31T20:54:56.968 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:s7zyK538cJo169BtBx+W1kAadZw= 2023-05-31T20:54:56.968 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:54:56 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:s7zyK538cJo169BtBx+W1kAadZw=', 'Content-Length': '0'} 2023-05-31T20:54:56.970 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:52:15 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000032905c727d0a07c2-006477b420-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:54:56 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:54:56.971 DEBUG:boto:path=/pklxkd-59/obj185 2023-05-31T20:54:56.971 DEBUG:boto:auth_path=/pklxkd-59/obj185 2023-05-31T20:54:56.971 DEBUG:boto:path=/pklxkd-59/obj185?versionId=null 2023-05-31T20:54:56.972 DEBUG:boto:auth_path=/pklxkd-59/obj185?versionId=null 2023-05-31T20:54:56.972 DEBUG:boto:Method: HEAD 2023-05-31T20:54:56.972 DEBUG:boto:Path: /pklxkd-59/obj185?versionId=null 2023-05-31T20:54:56.973 DEBUG:boto:Data: 2023-05-31T20:54:56.973 DEBUG:boto:Headers: {} 2023-05-31T20:54:56.973 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:54:56.974 DEBUG:boto:Port: 8001 2023-05-31T20:54:56.974 DEBUG:boto:Params: {} 2023-05-31T20:54:56.974 DEBUG:boto:Token: None 2023-05-31T20:54:56.975 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:54:56 GMT /pklxkd-59/obj185?versionId=null 2023-05-31T20:54:56.975 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:8SJGgeZZ7AEoFwNPJRmsniZ9gqk= 2023-05-31T20:54:56.976 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:54:56 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:8SJGgeZZ7AEoFwNPJRmsniZ9gqk=', 'Content-Length': '0'} 2023-05-31T20:54:56.977 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:52:15 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000665c3b5dc90bb2c3-006477b420-1e2e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:54:56 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:54:56.978 DEBUG:boto:path=/pklxkd-59/obj185 2023-05-31T20:54:56.978 DEBUG:boto:auth_path=/pklxkd-59/obj185 2023-05-31T20:54:56.979 DEBUG:boto:path=/pklxkd-59/obj185?versionId=null 2023-05-31T20:54:56.979 DEBUG:boto:auth_path=/pklxkd-59/obj185?versionId=null 2023-05-31T20:54:56.979 DEBUG:boto:Method: HEAD 2023-05-31T20:54:56.980 DEBUG:boto:Path: /pklxkd-59/obj185?versionId=null 2023-05-31T20:54:56.980 DEBUG:boto:Data: 2023-05-31T20:54:56.980 DEBUG:boto:Headers: {} 2023-05-31T20:54:56.981 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:54:56.981 DEBUG:boto:Port: 8000 2023-05-31T20:54:56.981 DEBUG:boto:Params: {} 2023-05-31T20:54:56.982 DEBUG:boto:Token: None 2023-05-31T20:54:56.982 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:54:56 GMT /pklxkd-59/obj185?versionId=null 2023-05-31T20:54:56.982 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:8SJGgeZZ7AEoFwNPJRmsniZ9gqk= 2023-05-31T20:54:56.983 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:54:56 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:8SJGgeZZ7AEoFwNPJRmsniZ9gqk=', 'Content-Length': '0'} 2023-05-31T20:54:56.984 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:52:15 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c4cb007e43f1f8c4-006477b420-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:54:56 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:54:56.985 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj185 2023-05-31T20:54:56.985 DEBUG:boto:path=/pklxkd-59/obj185 2023-05-31T20:54:56.986 DEBUG:boto:auth_path=/pklxkd-59/obj185 2023-05-31T20:54:56.986 DEBUG:boto:path=/pklxkd-59/obj185?versionId=null 2023-05-31T20:54:56.986 DEBUG:boto:auth_path=/pklxkd-59/obj185?versionId=null 2023-05-31T20:54:56.987 DEBUG:boto:Method: GET 2023-05-31T20:54:56.987 DEBUG:boto:Path: /pklxkd-59/obj185?versionId=null 2023-05-31T20:54:56.988 DEBUG:boto:Data: 2023-05-31T20:54:56.988 DEBUG:boto:Headers: {} 2023-05-31T20:54:56.988 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:54:56.989 DEBUG:boto:Port: 8001 2023-05-31T20:54:56.989 DEBUG:boto:Params: {} 2023-05-31T20:54:56.990 DEBUG:boto:Token: None 2023-05-31T20:54:56.990 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:54:56 GMT /pklxkd-59/obj185?versionId=null 2023-05-31T20:54:56.990 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:s7zyK538cJo169BtBx+W1kAadZw= 2023-05-31T20:54:56.991 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:54:56 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:s7zyK538cJo169BtBx+W1kAadZw=', 'Content-Length': '0'} 2023-05-31T20:54:56.993 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:52:15 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000043aa1024156c6268-006477b420-1e2e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:54:56 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:54:56.993 DEBUG:boto:path=/pklxkd-59/obj185 2023-05-31T20:54:56.994 DEBUG:boto:auth_path=/pklxkd-59/obj185 2023-05-31T20:54:56.994 DEBUG:boto:path=/pklxkd-59/obj185?versionId=null 2023-05-31T20:54:56.994 DEBUG:boto:auth_path=/pklxkd-59/obj185?versionId=null 2023-05-31T20:54:56.995 DEBUG:boto:Method: GET 2023-05-31T20:54:56.995 DEBUG:boto:Path: /pklxkd-59/obj185?versionId=null 2023-05-31T20:54:56.995 DEBUG:boto:Data: 2023-05-31T20:54:56.996 DEBUG:boto:Headers: {} 2023-05-31T20:54:56.996 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:54:56.997 DEBUG:boto:Port: 8000 2023-05-31T20:54:56.997 DEBUG:boto:Params: {} 2023-05-31T20:54:56.997 DEBUG:boto:Token: None 2023-05-31T20:54:56.998 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:54:56 GMT /pklxkd-59/obj185?versionId=null 2023-05-31T20:54:56.998 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:s7zyK538cJo169BtBx+W1kAadZw= 2023-05-31T20:54:56.998 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:54:56 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:s7zyK538cJo169BtBx+W1kAadZw=', 'Content-Length': '0'} 2023-05-31T20:54:57.000 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:52:15 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a6cda8f374ea292c-006477b420-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:54:56 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:54:57.044 DEBUG:boto:path=/pklxkd-59/ 2023-05-31T20:54:57.045 DEBUG:boto:auth_path=/pklxkd-59/ 2023-05-31T20:54:57.045 DEBUG:boto:path=/pklxkd-59/?prefix=obj185 2023-05-31T20:54:57.045 DEBUG:boto:auth_path=/pklxkd-59/?prefix=obj185 2023-05-31T20:54:57.046 DEBUG:boto:Method: GET 2023-05-31T20:54:57.046 DEBUG:boto:Path: /pklxkd-59/?prefix=obj185 2023-05-31T20:54:57.046 DEBUG:boto:Data: 2023-05-31T20:54:57.047 DEBUG:boto:Headers: {} 2023-05-31T20:54:57.047 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:54:57.047 DEBUG:boto:Port: 8001 2023-05-31T20:54:57.048 DEBUG:boto:Params: {} 2023-05-31T20:54:57.048 DEBUG:boto:Token: None 2023-05-31T20:54:57.049 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:54:57 GMT /pklxkd-59/ 2023-05-31T20:54:57.049 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:aisxEfj8evvfMLEyPoSaposCVmU= 2023-05-31T20:54:57.049 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:54:57 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:aisxEfj8evvfMLEyPoSaposCVmU=', 'Content-Length': '0'} 2023-05-31T20:54:57.052 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000009ae9f4577fabd23c-006477b421-1e2e-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:54:57 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:54:57.052 DEBUG:boto:b'pklxkd-59obj1851000falseobj1852023-05-31T20:52:15.072Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:54:57.053 DEBUG:boto:path=/pklxkd-59/ 2023-05-31T20:54:57.053 DEBUG:boto:auth_path=/pklxkd-59/ 2023-05-31T20:54:57.054 DEBUG:boto:path=/pklxkd-59/?prefix=obj185 2023-05-31T20:54:57.054 DEBUG:boto:auth_path=/pklxkd-59/?prefix=obj185 2023-05-31T20:54:57.054 DEBUG:boto:Method: GET 2023-05-31T20:54:57.055 DEBUG:boto:Path: /pklxkd-59/?prefix=obj185 2023-05-31T20:54:57.055 DEBUG:boto:Data: 2023-05-31T20:54:57.055 DEBUG:boto:Headers: {} 2023-05-31T20:54:57.056 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:54:57.056 DEBUG:boto:Port: 8000 2023-05-31T20:54:57.056 DEBUG:boto:Params: {} 2023-05-31T20:54:57.057 DEBUG:boto:Token: None 2023-05-31T20:54:57.057 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:54:57 GMT /pklxkd-59/ 2023-05-31T20:54:57.058 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:aisxEfj8evvfMLEyPoSaposCVmU= 2023-05-31T20:54:57.058 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:54:57 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:aisxEfj8evvfMLEyPoSaposCVmU=', 'Content-Length': '0'} 2023-05-31T20:54:57.062 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ff58c2693b28070d-006477b421-1630-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:54:57 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:54:57.062 DEBUG:boto:b'pklxkd-59obj1851000falseobj1852023-05-31T20:52:15.072Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:54:57.063 DEBUG:boto:path=/pklxkd-59/obj185 2023-05-31T20:54:57.063 DEBUG:boto:auth_path=/pklxkd-59/obj185 2023-05-31T20:54:57.064 DEBUG:boto:Method: HEAD 2023-05-31T20:54:57.064 DEBUG:boto:Path: /pklxkd-59/obj185 2023-05-31T20:54:57.064 DEBUG:boto:Data: 2023-05-31T20:54:57.065 DEBUG:boto:Headers: {} 2023-05-31T20:54:57.065 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:54:57.065 DEBUG:boto:Port: 8001 2023-05-31T20:54:57.066 DEBUG:boto:Params: {} 2023-05-31T20:54:57.066 DEBUG:boto:Token: None 2023-05-31T20:54:57.067 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:54:57 GMT /pklxkd-59/obj185 2023-05-31T20:54:57.067 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:e5CaC9tmGyKkIrOHwWD2oBwsee0= 2023-05-31T20:54:57.067 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:54:57 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:e5CaC9tmGyKkIrOHwWD2oBwsee0=', 'Content-Length': '0'} 2023-05-31T20:54:57.069 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:52:15 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005f204a29854a5650-006477b421-1e2e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:54:57 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:54:57.069 DEBUG:boto:path=/pklxkd-59/obj185 2023-05-31T20:54:57.070 DEBUG:boto:auth_path=/pklxkd-59/obj185 2023-05-31T20:54:57.070 DEBUG:boto:Method: HEAD 2023-05-31T20:54:57.070 DEBUG:boto:Path: /pklxkd-59/obj185 2023-05-31T20:54:57.071 DEBUG:boto:Data: 2023-05-31T20:54:57.071 DEBUG:boto:Headers: {} 2023-05-31T20:54:57.071 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:54:57.072 DEBUG:boto:Port: 8000 2023-05-31T20:54:57.072 DEBUG:boto:Params: {} 2023-05-31T20:54:57.073 DEBUG:boto:Token: None 2023-05-31T20:54:57.073 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:54:57 GMT /pklxkd-59/obj185 2023-05-31T20:54:57.073 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:e5CaC9tmGyKkIrOHwWD2oBwsee0= 2023-05-31T20:54:57.074 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:54:57 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:e5CaC9tmGyKkIrOHwWD2oBwsee0=', 'Content-Length': '0'} 2023-05-31T20:54:57.075 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:52:15 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d5f1fba1abba0ec5-006477b421-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:54:57 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:54:57.076 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj185 2023-05-31T20:54:57.076 DEBUG:boto:path=/pklxkd-59/obj185 2023-05-31T20:54:57.077 DEBUG:boto:auth_path=/pklxkd-59/obj185 2023-05-31T20:54:57.077 DEBUG:boto:Method: GET 2023-05-31T20:54:57.078 DEBUG:boto:Path: /pklxkd-59/obj185 2023-05-31T20:54:57.078 DEBUG:boto:Data: 2023-05-31T20:54:57.078 DEBUG:boto:Headers: {} 2023-05-31T20:54:57.079 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:54:57.079 DEBUG:boto:Port: 8001 2023-05-31T20:54:57.079 DEBUG:boto:Params: {} 2023-05-31T20:54:57.080 DEBUG:boto:Token: None 2023-05-31T20:54:57.080 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:54:57 GMT /pklxkd-59/obj185 2023-05-31T20:54:57.081 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:opdl5rzsfiOETfdiMj4zFwk0rG4= 2023-05-31T20:54:57.081 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:54:57 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:opdl5rzsfiOETfdiMj4zFwk0rG4=', 'Content-Length': '0'} 2023-05-31T20:54:57.082 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:52:15 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004723d7a567b755e8-006477b421-1e2e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:54:57 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:54:57.124 DEBUG:boto:path=/pklxkd-59/obj185 2023-05-31T20:54:57.125 DEBUG:boto:auth_path=/pklxkd-59/obj185 2023-05-31T20:54:57.125 DEBUG:boto:Method: GET 2023-05-31T20:54:57.125 DEBUG:boto:Path: /pklxkd-59/obj185 2023-05-31T20:54:57.126 DEBUG:boto:Data: 2023-05-31T20:54:57.126 DEBUG:boto:Headers: {} 2023-05-31T20:54:57.126 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:54:57.127 DEBUG:boto:Port: 8000 2023-05-31T20:54:57.127 DEBUG:boto:Params: {} 2023-05-31T20:54:57.127 DEBUG:boto:Token: None 2023-05-31T20:54:57.128 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:54:57 GMT /pklxkd-59/obj185 2023-05-31T20:54:57.128 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:opdl5rzsfiOETfdiMj4zFwk0rG4= 2023-05-31T20:54:57.129 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:54:57 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:opdl5rzsfiOETfdiMj4zFwk0rG4=', 'Content-Length': '0'} 2023-05-31T20:54:57.131 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:52:15 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d6a40db67e73c6e8-006477b421-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:54:57 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:54:57.131 DEBUG:boto:path=/pklxkd-59/ 2023-05-31T20:54:57.132 DEBUG:boto:auth_path=/pklxkd-59/ 2023-05-31T20:54:57.132 DEBUG:boto:path=/pklxkd-59/?prefix=obj185 2023-05-31T20:54:57.133 DEBUG:boto:auth_path=/pklxkd-59/?prefix=obj185 2023-05-31T20:54:57.133 DEBUG:boto:Method: GET 2023-05-31T20:54:57.133 DEBUG:boto:Path: /pklxkd-59/?prefix=obj185 2023-05-31T20:54:57.134 DEBUG:boto:Data: 2023-05-31T20:54:57.134 DEBUG:boto:Headers: {} 2023-05-31T20:54:57.134 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:54:57.135 DEBUG:boto:Port: 8001 2023-05-31T20:54:57.135 DEBUG:boto:Params: {} 2023-05-31T20:54:57.135 DEBUG:boto:Token: None 2023-05-31T20:54:57.136 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:54:57 GMT /pklxkd-59/ 2023-05-31T20:54:57.136 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:aisxEfj8evvfMLEyPoSaposCVmU= 2023-05-31T20:54:57.137 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:54:57 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:aisxEfj8evvfMLEyPoSaposCVmU=', 'Content-Length': '0'} 2023-05-31T20:54:57.139 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000006fd70232da4876b4-006477b421-1e2e-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:54:57 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:54:57.139 DEBUG:boto:b'pklxkd-59obj1851000falseobj1852023-05-31T20:52:15.072Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:54:57.140 DEBUG:boto:path=/pklxkd-59/ 2023-05-31T20:54:57.140 DEBUG:boto:auth_path=/pklxkd-59/ 2023-05-31T20:54:57.141 DEBUG:boto:path=/pklxkd-59/?prefix=obj185 2023-05-31T20:54:57.141 DEBUG:boto:auth_path=/pklxkd-59/?prefix=obj185 2023-05-31T20:54:57.141 DEBUG:boto:Method: GET 2023-05-31T20:54:57.142 DEBUG:boto:Path: /pklxkd-59/?prefix=obj185 2023-05-31T20:54:57.142 DEBUG:boto:Data: 2023-05-31T20:54:57.143 DEBUG:boto:Headers: {} 2023-05-31T20:54:57.143 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:54:57.143 DEBUG:boto:Port: 8000 2023-05-31T20:54:57.144 DEBUG:boto:Params: {} 2023-05-31T20:54:57.144 DEBUG:boto:Token: None 2023-05-31T20:54:57.144 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:54:57 GMT /pklxkd-59/ 2023-05-31T20:54:57.145 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:aisxEfj8evvfMLEyPoSaposCVmU= 2023-05-31T20:54:57.145 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:54:57 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:aisxEfj8evvfMLEyPoSaposCVmU=', 'Content-Length': '0'} 2023-05-31T20:54:57.149 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000080cb106e241d5972-006477b421-1630-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:54:57 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:54:57.150 DEBUG:boto:b'pklxkd-59obj1851000falseobj1852023-05-31T20:52:15.072Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:54:57.150 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj19 2023-05-31T20:54:57.151 DEBUG:boto:path=/pklxkd-59/obj19 2023-05-31T20:54:57.151 DEBUG:boto:auth_path=/pklxkd-59/obj19 2023-05-31T20:54:57.151 DEBUG:boto:path=/pklxkd-59/obj19?versionId=null 2023-05-31T20:54:57.152 DEBUG:boto:auth_path=/pklxkd-59/obj19?versionId=null 2023-05-31T20:54:57.152 DEBUG:boto:Method: GET 2023-05-31T20:54:57.152 DEBUG:boto:Path: /pklxkd-59/obj19?versionId=null 2023-05-31T20:54:57.153 DEBUG:boto:Data: 2023-05-31T20:54:57.153 DEBUG:boto:Headers: {} 2023-05-31T20:54:57.154 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:54:57.154 DEBUG:boto:Port: 8001 2023-05-31T20:54:57.154 DEBUG:boto:Params: {} 2023-05-31T20:54:57.155 DEBUG:boto:Token: None 2023-05-31T20:54:57.155 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:54:57 GMT /pklxkd-59/obj19?versionId=null 2023-05-31T20:54:57.156 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:G+nC+rwrzmTxDDVkV5ZQTE/337I= 2023-05-31T20:54:57.156 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:54:57 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:G+nC+rwrzmTxDDVkV5ZQTE/337I=', 'Content-Length': '0'} 2023-05-31T20:54:57.158 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:44:50 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000093cc1452276638d8-006477b421-1e2e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:54:57 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:54:57.159 DEBUG:boto:path=/pklxkd-59/obj19 2023-05-31T20:54:57.159 DEBUG:boto:auth_path=/pklxkd-59/obj19 2023-05-31T20:54:57.160 DEBUG:boto:path=/pklxkd-59/obj19?versionId=null 2023-05-31T20:54:57.160 DEBUG:boto:auth_path=/pklxkd-59/obj19?versionId=null 2023-05-31T20:54:57.160 DEBUG:boto:Method: GET 2023-05-31T20:54:57.161 DEBUG:boto:Path: /pklxkd-59/obj19?versionId=null 2023-05-31T20:54:57.161 DEBUG:boto:Data: 2023-05-31T20:54:57.161 DEBUG:boto:Headers: {} 2023-05-31T20:54:57.162 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:54:57.162 DEBUG:boto:Port: 8000 2023-05-31T20:54:57.162 DEBUG:boto:Params: {} 2023-05-31T20:54:57.163 DEBUG:boto:Token: None 2023-05-31T20:54:57.163 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:54:57 GMT /pklxkd-59/obj19?versionId=null 2023-05-31T20:54:57.164 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:G+nC+rwrzmTxDDVkV5ZQTE/337I= 2023-05-31T20:54:57.164 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:54:57 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:G+nC+rwrzmTxDDVkV5ZQTE/337I=', 'Content-Length': '0'} 2023-05-31T20:54:57.166 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:44:50 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f299007eeb7eecb0-006477b421-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:54:57 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:54:57.208 DEBUG:boto:path=/pklxkd-59/obj19 2023-05-31T20:54:57.209 DEBUG:boto:auth_path=/pklxkd-59/obj19 2023-05-31T20:54:57.209 DEBUG:boto:path=/pklxkd-59/obj19?versionId=null 2023-05-31T20:54:57.209 DEBUG:boto:auth_path=/pklxkd-59/obj19?versionId=null 2023-05-31T20:54:57.210 DEBUG:boto:Method: HEAD 2023-05-31T20:54:57.210 DEBUG:boto:Path: /pklxkd-59/obj19?versionId=null 2023-05-31T20:54:57.210 DEBUG:boto:Data: 2023-05-31T20:54:57.211 DEBUG:boto:Headers: {} 2023-05-31T20:54:57.211 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:54:57.211 DEBUG:boto:Port: 8001 2023-05-31T20:54:57.212 DEBUG:boto:Params: {} 2023-05-31T20:54:57.212 DEBUG:boto:Token: None 2023-05-31T20:54:57.212 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:54:57 GMT /pklxkd-59/obj19?versionId=null 2023-05-31T20:54:57.213 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:UFSJ1uVM051H2j5m8d4Zv/Bx1yM= 2023-05-31T20:54:57.213 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:54:57 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:UFSJ1uVM051H2j5m8d4Zv/Bx1yM=', 'Content-Length': '0'} 2023-05-31T20:54:57.215 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:44:50 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a855fe79aa16757a-006477b421-1e2e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:54:57 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:54:57.216 DEBUG:boto:path=/pklxkd-59/obj19 2023-05-31T20:54:57.216 DEBUG:boto:auth_path=/pklxkd-59/obj19 2023-05-31T20:54:57.216 DEBUG:boto:path=/pklxkd-59/obj19?versionId=null 2023-05-31T20:54:57.217 DEBUG:boto:auth_path=/pklxkd-59/obj19?versionId=null 2023-05-31T20:54:57.217 DEBUG:boto:Method: HEAD 2023-05-31T20:54:57.217 DEBUG:boto:Path: /pklxkd-59/obj19?versionId=null 2023-05-31T20:54:57.218 DEBUG:boto:Data: 2023-05-31T20:54:57.218 DEBUG:boto:Headers: {} 2023-05-31T20:54:57.218 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:54:57.219 DEBUG:boto:Port: 8000 2023-05-31T20:54:57.219 DEBUG:boto:Params: {} 2023-05-31T20:54:57.219 DEBUG:boto:Token: None 2023-05-31T20:54:57.220 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:54:57 GMT /pklxkd-59/obj19?versionId=null 2023-05-31T20:54:57.220 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:UFSJ1uVM051H2j5m8d4Zv/Bx1yM= 2023-05-31T20:54:57.221 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:54:57 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:UFSJ1uVM051H2j5m8d4Zv/Bx1yM=', 'Content-Length': '0'} 2023-05-31T20:54:57.222 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:44:50 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d60e81eb4cb07ca1-006477b421-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:54:57 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:54:57.223 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj19 2023-05-31T20:54:57.223 DEBUG:boto:path=/pklxkd-59/obj19 2023-05-31T20:54:57.224 DEBUG:boto:auth_path=/pklxkd-59/obj19 2023-05-31T20:54:57.224 DEBUG:boto:path=/pklxkd-59/obj19?versionId=null 2023-05-31T20:54:57.224 DEBUG:boto:auth_path=/pklxkd-59/obj19?versionId=null 2023-05-31T20:54:57.225 DEBUG:boto:Method: GET 2023-05-31T20:54:57.225 DEBUG:boto:Path: /pklxkd-59/obj19?versionId=null 2023-05-31T20:54:57.225 DEBUG:boto:Data: 2023-05-31T20:54:57.226 DEBUG:boto:Headers: {} 2023-05-31T20:54:57.226 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:54:57.226 DEBUG:boto:Port: 8001 2023-05-31T20:54:57.227 DEBUG:boto:Params: {} 2023-05-31T20:54:57.227 DEBUG:boto:Token: None 2023-05-31T20:54:57.228 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:54:57 GMT /pklxkd-59/obj19?versionId=null 2023-05-31T20:54:57.228 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:G+nC+rwrzmTxDDVkV5ZQTE/337I= 2023-05-31T20:54:57.228 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:54:57 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:G+nC+rwrzmTxDDVkV5ZQTE/337I=', 'Content-Length': '0'} 2023-05-31T20:54:57.230 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:44:50 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000278da88f6c34d3d9-006477b421-1e2e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:54:57 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:54:57.231 DEBUG:boto:path=/pklxkd-59/obj19 2023-05-31T20:54:57.231 DEBUG:boto:auth_path=/pklxkd-59/obj19 2023-05-31T20:54:57.231 DEBUG:boto:path=/pklxkd-59/obj19?versionId=null 2023-05-31T20:54:57.232 DEBUG:boto:auth_path=/pklxkd-59/obj19?versionId=null 2023-05-31T20:54:57.232 DEBUG:boto:Method: GET 2023-05-31T20:54:57.232 DEBUG:boto:Path: /pklxkd-59/obj19?versionId=null 2023-05-31T20:54:57.233 DEBUG:boto:Data: 2023-05-31T20:54:57.233 DEBUG:boto:Headers: {} 2023-05-31T20:54:57.233 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:54:57.234 DEBUG:boto:Port: 8000 2023-05-31T20:54:57.234 DEBUG:boto:Params: {} 2023-05-31T20:54:57.235 DEBUG:boto:Token: None 2023-05-31T20:54:57.235 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:54:57 GMT /pklxkd-59/obj19?versionId=null 2023-05-31T20:54:57.235 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:G+nC+rwrzmTxDDVkV5ZQTE/337I= 2023-05-31T20:54:57.236 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:54:57 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:G+nC+rwrzmTxDDVkV5ZQTE/337I=', 'Content-Length': '0'} 2023-05-31T20:54:57.237 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:44:50 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000addcf4cfd327f3a7-006477b421-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:54:57 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:54:57.238 DEBUG:boto:path=/pklxkd-59/ 2023-05-31T20:54:57.238 DEBUG:boto:auth_path=/pklxkd-59/ 2023-05-31T20:54:57.239 DEBUG:boto:path=/pklxkd-59/?prefix=obj19 2023-05-31T20:54:57.239 DEBUG:boto:auth_path=/pklxkd-59/?prefix=obj19 2023-05-31T20:54:57.239 DEBUG:boto:Method: GET 2023-05-31T20:54:57.240 DEBUG:boto:Path: /pklxkd-59/?prefix=obj19 2023-05-31T20:54:57.240 DEBUG:boto:Data: 2023-05-31T20:54:57.241 DEBUG:boto:Headers: {} 2023-05-31T20:54:57.241 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:54:57.241 DEBUG:boto:Port: 8001 2023-05-31T20:54:57.242 DEBUG:boto:Params: {} 2023-05-31T20:54:57.242 DEBUG:boto:Token: None 2023-05-31T20:54:57.242 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:54:57 GMT /pklxkd-59/ 2023-05-31T20:54:57.243 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:aisxEfj8evvfMLEyPoSaposCVmU= 2023-05-31T20:54:57.243 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:54:57 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:aisxEfj8evvfMLEyPoSaposCVmU=', 'Content-Length': '0'} 2023-05-31T20:54:57.245 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000007a9017fb29ef6dc-006477b421-1e2e-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:54:57 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:54:57.246 DEBUG:boto:b'pklxkd-59obj191000falseobj192023-05-31T20:44:50.557Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:54:57.247 DEBUG:boto:path=/pklxkd-59/ 2023-05-31T20:54:57.247 DEBUG:boto:auth_path=/pklxkd-59/ 2023-05-31T20:54:57.247 DEBUG:boto:path=/pklxkd-59/?prefix=obj19 2023-05-31T20:54:57.248 DEBUG:boto:auth_path=/pklxkd-59/?prefix=obj19 2023-05-31T20:54:57.248 DEBUG:boto:Method: GET 2023-05-31T20:54:57.248 DEBUG:boto:Path: /pklxkd-59/?prefix=obj19 2023-05-31T20:54:57.249 DEBUG:boto:Data: 2023-05-31T20:54:57.249 DEBUG:boto:Headers: {} 2023-05-31T20:54:57.250 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:54:57.250 DEBUG:boto:Port: 8000 2023-05-31T20:54:57.251 DEBUG:boto:Params: {} 2023-05-31T20:54:57.251 DEBUG:boto:Token: None 2023-05-31T20:54:57.252 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:54:57 GMT /pklxkd-59/ 2023-05-31T20:54:57.253 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:aisxEfj8evvfMLEyPoSaposCVmU= 2023-05-31T20:54:57.253 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:54:57 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:aisxEfj8evvfMLEyPoSaposCVmU=', 'Content-Length': '0'} 2023-05-31T20:54:57.257 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000bf0bfc2adb83b695-006477b421-1630-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:54:57 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:54:57.257 DEBUG:boto:b'pklxkd-59obj191000falseobj192023-05-31T20:44:50.557Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:54:57.258 DEBUG:boto:path=/pklxkd-59/obj19 2023-05-31T20:54:57.258 DEBUG:boto:auth_path=/pklxkd-59/obj19 2023-05-31T20:54:57.259 DEBUG:boto:Method: HEAD 2023-05-31T20:54:57.259 DEBUG:boto:Path: /pklxkd-59/obj19 2023-05-31T20:54:57.259 DEBUG:boto:Data: 2023-05-31T20:54:57.260 DEBUG:boto:Headers: {} 2023-05-31T20:54:57.260 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:54:57.260 DEBUG:boto:Port: 8001 2023-05-31T20:54:57.261 DEBUG:boto:Params: {} 2023-05-31T20:54:57.261 DEBUG:boto:Token: None 2023-05-31T20:54:57.261 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:54:57 GMT /pklxkd-59/obj19 2023-05-31T20:54:57.262 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:gKp1ttvVRq5wy1ubpH8XqEyZkas= 2023-05-31T20:54:57.262 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:54:57 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:gKp1ttvVRq5wy1ubpH8XqEyZkas=', 'Content-Length': '0'} 2023-05-31T20:54:57.264 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:44:50 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a3dc92783cb873f4-006477b421-1e2e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:54:57 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:54:57.265 DEBUG:boto:path=/pklxkd-59/obj19 2023-05-31T20:54:57.265 DEBUG:boto:auth_path=/pklxkd-59/obj19 2023-05-31T20:54:57.265 DEBUG:boto:Method: HEAD 2023-05-31T20:54:57.266 DEBUG:boto:Path: /pklxkd-59/obj19 2023-05-31T20:54:57.266 DEBUG:boto:Data: 2023-05-31T20:54:57.266 DEBUG:boto:Headers: {} 2023-05-31T20:54:57.267 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:54:57.267 DEBUG:boto:Port: 8000 2023-05-31T20:54:57.267 DEBUG:boto:Params: {} 2023-05-31T20:54:57.268 DEBUG:boto:Token: None 2023-05-31T20:54:57.268 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:54:57 GMT /pklxkd-59/obj19 2023-05-31T20:54:57.269 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:gKp1ttvVRq5wy1ubpH8XqEyZkas= 2023-05-31T20:54:57.269 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:54:57 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:gKp1ttvVRq5wy1ubpH8XqEyZkas=', 'Content-Length': '0'} 2023-05-31T20:54:57.271 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:44:50 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001f04badbe2fcf90f-006477b421-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:54:57 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:54:57.271 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj19 2023-05-31T20:54:57.272 DEBUG:boto:path=/pklxkd-59/obj19 2023-05-31T20:54:57.272 DEBUG:boto:auth_path=/pklxkd-59/obj19 2023-05-31T20:54:57.272 DEBUG:boto:Method: GET 2023-05-31T20:54:57.273 DEBUG:boto:Path: /pklxkd-59/obj19 2023-05-31T20:54:57.273 DEBUG:boto:Data: 2023-05-31T20:54:57.273 DEBUG:boto:Headers: {} 2023-05-31T20:54:57.274 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:54:57.274 DEBUG:boto:Port: 8001 2023-05-31T20:54:57.275 DEBUG:boto:Params: {} 2023-05-31T20:54:57.275 DEBUG:boto:Token: None 2023-05-31T20:54:57.275 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:54:57 GMT /pklxkd-59/obj19 2023-05-31T20:54:57.276 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:ksRI/LIDVZ1vxtjZLcy7UORfC0M= 2023-05-31T20:54:57.276 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:54:57 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:ksRI/LIDVZ1vxtjZLcy7UORfC0M=', 'Content-Length': '0'} 2023-05-31T20:54:57.278 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:44:50 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000757a62ff960b1d19-006477b421-1e2e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:54:57 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:54:57.279 DEBUG:boto:path=/pklxkd-59/obj19 2023-05-31T20:54:57.279 DEBUG:boto:auth_path=/pklxkd-59/obj19 2023-05-31T20:54:57.279 DEBUG:boto:Method: GET 2023-05-31T20:54:57.280 DEBUG:boto:Path: /pklxkd-59/obj19 2023-05-31T20:54:57.280 DEBUG:boto:Data: 2023-05-31T20:54:57.280 DEBUG:boto:Headers: {} 2023-05-31T20:54:57.281 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:54:57.281 DEBUG:boto:Port: 8000 2023-05-31T20:54:57.282 DEBUG:boto:Params: {} 2023-05-31T20:54:57.282 DEBUG:boto:Token: None 2023-05-31T20:54:57.282 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:54:57 GMT /pklxkd-59/obj19 2023-05-31T20:54:57.283 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:ksRI/LIDVZ1vxtjZLcy7UORfC0M= 2023-05-31T20:54:57.283 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:54:57 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:ksRI/LIDVZ1vxtjZLcy7UORfC0M=', 'Content-Length': '0'} 2023-05-31T20:54:57.285 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:44:50 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005b5ae3006a7043f2-006477b421-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:54:57 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:54:57.286 DEBUG:boto:path=/pklxkd-59/ 2023-05-31T20:54:57.286 DEBUG:boto:auth_path=/pklxkd-59/ 2023-05-31T20:54:57.287 DEBUG:boto:path=/pklxkd-59/?prefix=obj19 2023-05-31T20:54:57.287 DEBUG:boto:auth_path=/pklxkd-59/?prefix=obj19 2023-05-31T20:54:57.287 DEBUG:boto:Method: GET 2023-05-31T20:54:57.288 DEBUG:boto:Path: /pklxkd-59/?prefix=obj19 2023-05-31T20:54:57.288 DEBUG:boto:Data: 2023-05-31T20:54:57.289 DEBUG:boto:Headers: {} 2023-05-31T20:54:57.289 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:54:57.289 DEBUG:boto:Port: 8001 2023-05-31T20:54:57.290 DEBUG:boto:Params: {} 2023-05-31T20:54:57.290 DEBUG:boto:Token: None 2023-05-31T20:54:57.290 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:54:57 GMT /pklxkd-59/ 2023-05-31T20:54:57.291 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:aisxEfj8evvfMLEyPoSaposCVmU= 2023-05-31T20:54:57.291 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:54:57 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:aisxEfj8evvfMLEyPoSaposCVmU=', 'Content-Length': '0'} 2023-05-31T20:54:57.294 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a14cdf3ff678b97e-006477b421-1e2e-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:54:57 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:54:57.294 DEBUG:boto:b'pklxkd-59obj191000falseobj192023-05-31T20:44:50.557Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:54:57.295 DEBUG:boto:path=/pklxkd-59/ 2023-05-31T20:54:57.295 DEBUG:boto:auth_path=/pklxkd-59/ 2023-05-31T20:54:57.295 DEBUG:boto:path=/pklxkd-59/?prefix=obj19 2023-05-31T20:54:57.296 DEBUG:boto:auth_path=/pklxkd-59/?prefix=obj19 2023-05-31T20:54:57.296 DEBUG:boto:Method: GET 2023-05-31T20:54:57.296 DEBUG:boto:Path: /pklxkd-59/?prefix=obj19 2023-05-31T20:54:57.297 DEBUG:boto:Data: 2023-05-31T20:54:57.297 DEBUG:boto:Headers: {} 2023-05-31T20:54:57.298 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:54:57.298 DEBUG:boto:Port: 8000 2023-05-31T20:54:57.298 DEBUG:boto:Params: {} 2023-05-31T20:54:57.299 DEBUG:boto:Token: None 2023-05-31T20:54:57.299 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:54:57 GMT /pklxkd-59/ 2023-05-31T20:54:57.299 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:aisxEfj8evvfMLEyPoSaposCVmU= 2023-05-31T20:54:57.300 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:54:57 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:aisxEfj8evvfMLEyPoSaposCVmU=', 'Content-Length': '0'} 2023-05-31T20:54:57.304 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000672b6e438bfc1c14-006477b421-1630-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:54:57 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:54:57.344 DEBUG:boto:b'pklxkd-59obj191000falseobj192023-05-31T20:44:50.557Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:54:57.345 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj22 2023-05-31T20:54:57.345 DEBUG:boto:path=/pklxkd-59/obj22 2023-05-31T20:54:57.345 DEBUG:boto:auth_path=/pklxkd-59/obj22 2023-05-31T20:54:57.346 DEBUG:boto:path=/pklxkd-59/obj22?versionId=null 2023-05-31T20:54:57.346 DEBUG:boto:auth_path=/pklxkd-59/obj22?versionId=null 2023-05-31T20:54:57.346 DEBUG:boto:Method: GET 2023-05-31T20:54:57.347 DEBUG:boto:Path: /pklxkd-59/obj22?versionId=null 2023-05-31T20:54:57.347 DEBUG:boto:Data: 2023-05-31T20:54:57.348 DEBUG:boto:Headers: {} 2023-05-31T20:54:57.348 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:54:57.348 DEBUG:boto:Port: 8001 2023-05-31T20:54:57.349 DEBUG:boto:Params: {} 2023-05-31T20:54:57.349 DEBUG:boto:Token: None 2023-05-31T20:54:57.349 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:54:57 GMT /pklxkd-59/obj22?versionId=null 2023-05-31T20:54:57.350 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:J16JD0bJVh2Fvhkdz8Yy615dbXM= 2023-05-31T20:54:57.350 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:54:57 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:J16JD0bJVh2Fvhkdz8Yy615dbXM=', 'Content-Length': '0'} 2023-05-31T20:54:57.352 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:39:26 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ffab24367b0433a3-006477b421-1e2e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:54:57 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:54:57.353 DEBUG:boto:path=/pklxkd-59/obj22 2023-05-31T20:54:57.353 DEBUG:boto:auth_path=/pklxkd-59/obj22 2023-05-31T20:54:57.354 DEBUG:boto:path=/pklxkd-59/obj22?versionId=null 2023-05-31T20:54:57.354 DEBUG:boto:auth_path=/pklxkd-59/obj22?versionId=null 2023-05-31T20:54:57.354 DEBUG:boto:Method: GET 2023-05-31T20:54:57.355 DEBUG:boto:Path: /pklxkd-59/obj22?versionId=null 2023-05-31T20:54:57.355 DEBUG:boto:Data: 2023-05-31T20:54:57.355 DEBUG:boto:Headers: {} 2023-05-31T20:54:57.356 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:54:57.356 DEBUG:boto:Port: 8000 2023-05-31T20:54:57.357 DEBUG:boto:Params: {} 2023-05-31T20:54:57.357 DEBUG:boto:Token: None 2023-05-31T20:54:57.357 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:54:57 GMT /pklxkd-59/obj22?versionId=null 2023-05-31T20:54:57.358 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:J16JD0bJVh2Fvhkdz8Yy615dbXM= 2023-05-31T20:54:57.358 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:54:57 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:J16JD0bJVh2Fvhkdz8Yy615dbXM=', 'Content-Length': '0'} 2023-05-31T20:54:57.360 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:39:26 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000046a350347d415bba-006477b421-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:54:57 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:54:57.361 DEBUG:boto:path=/pklxkd-59/obj22 2023-05-31T20:54:57.361 DEBUG:boto:auth_path=/pklxkd-59/obj22 2023-05-31T20:54:57.361 DEBUG:boto:path=/pklxkd-59/obj22?versionId=null 2023-05-31T20:54:57.362 DEBUG:boto:auth_path=/pklxkd-59/obj22?versionId=null 2023-05-31T20:54:57.362 DEBUG:boto:Method: HEAD 2023-05-31T20:54:57.362 DEBUG:boto:Path: /pklxkd-59/obj22?versionId=null 2023-05-31T20:54:57.363 DEBUG:boto:Data: 2023-05-31T20:54:57.363 DEBUG:boto:Headers: {} 2023-05-31T20:54:57.363 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:54:57.364 DEBUG:boto:Port: 8001 2023-05-31T20:54:57.364 DEBUG:boto:Params: {} 2023-05-31T20:54:57.365 DEBUG:boto:Token: None 2023-05-31T20:54:57.365 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:54:57 GMT /pklxkd-59/obj22?versionId=null 2023-05-31T20:54:57.365 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:qPTbe0Ls2ZtAa0ZvL26yWIzwK/g= 2023-05-31T20:54:57.366 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:54:57 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:qPTbe0Ls2ZtAa0ZvL26yWIzwK/g=', 'Content-Length': '0'} 2023-05-31T20:54:57.367 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:39:26 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f9a0598ab62fd55e-006477b421-1e2e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:54:57 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:54:57.368 DEBUG:boto:path=/pklxkd-59/obj22 2023-05-31T20:54:57.368 DEBUG:boto:auth_path=/pklxkd-59/obj22 2023-05-31T20:54:57.369 DEBUG:boto:path=/pklxkd-59/obj22?versionId=null 2023-05-31T20:54:57.369 DEBUG:boto:auth_path=/pklxkd-59/obj22?versionId=null 2023-05-31T20:54:57.369 DEBUG:boto:Method: HEAD 2023-05-31T20:54:57.370 DEBUG:boto:Path: /pklxkd-59/obj22?versionId=null 2023-05-31T20:54:57.370 DEBUG:boto:Data: 2023-05-31T20:54:57.370 DEBUG:boto:Headers: {} 2023-05-31T20:54:57.371 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:54:57.371 DEBUG:boto:Port: 8000 2023-05-31T20:54:57.371 DEBUG:boto:Params: {} 2023-05-31T20:54:57.372 DEBUG:boto:Token: None 2023-05-31T20:54:57.372 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:54:57 GMT /pklxkd-59/obj22?versionId=null 2023-05-31T20:54:57.372 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:qPTbe0Ls2ZtAa0ZvL26yWIzwK/g= 2023-05-31T20:54:57.373 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:54:57 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:qPTbe0Ls2ZtAa0ZvL26yWIzwK/g=', 'Content-Length': '0'} 2023-05-31T20:54:57.374 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:39:26 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ec9a6c0e69f26c85-006477b421-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:54:57 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:54:57.375 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj22 2023-05-31T20:54:57.375 DEBUG:boto:path=/pklxkd-59/obj22 2023-05-31T20:54:57.376 DEBUG:boto:auth_path=/pklxkd-59/obj22 2023-05-31T20:54:57.376 DEBUG:boto:path=/pklxkd-59/obj22?versionId=null 2023-05-31T20:54:57.377 DEBUG:boto:auth_path=/pklxkd-59/obj22?versionId=null 2023-05-31T20:54:57.377 DEBUG:boto:Method: GET 2023-05-31T20:54:57.377 DEBUG:boto:Path: /pklxkd-59/obj22?versionId=null 2023-05-31T20:54:57.378 DEBUG:boto:Data: 2023-05-31T20:54:57.378 DEBUG:boto:Headers: {} 2023-05-31T20:54:57.378 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:54:57.379 DEBUG:boto:Port: 8001 2023-05-31T20:54:57.379 DEBUG:boto:Params: {} 2023-05-31T20:54:57.379 DEBUG:boto:Token: None 2023-05-31T20:54:57.380 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:54:57 GMT /pklxkd-59/obj22?versionId=null 2023-05-31T20:54:57.380 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:J16JD0bJVh2Fvhkdz8Yy615dbXM= 2023-05-31T20:54:57.381 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:54:57 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:J16JD0bJVh2Fvhkdz8Yy615dbXM=', 'Content-Length': '0'} 2023-05-31T20:54:57.382 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:39:26 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009d974d98fa74266a-006477b421-1e2e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:54:57 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:54:57.424 DEBUG:boto:path=/pklxkd-59/obj22 2023-05-31T20:54:57.425 DEBUG:boto:auth_path=/pklxkd-59/obj22 2023-05-31T20:54:57.425 DEBUG:boto:path=/pklxkd-59/obj22?versionId=null 2023-05-31T20:54:57.425 DEBUG:boto:auth_path=/pklxkd-59/obj22?versionId=null 2023-05-31T20:54:57.426 DEBUG:boto:Method: GET 2023-05-31T20:54:57.426 DEBUG:boto:Path: /pklxkd-59/obj22?versionId=null 2023-05-31T20:54:57.426 DEBUG:boto:Data: 2023-05-31T20:54:57.427 DEBUG:boto:Headers: {} 2023-05-31T20:54:57.427 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:54:57.427 DEBUG:boto:Port: 8000 2023-05-31T20:54:57.428 DEBUG:boto:Params: {} 2023-05-31T20:54:57.428 DEBUG:boto:Token: None 2023-05-31T20:54:57.429 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:54:57 GMT /pklxkd-59/obj22?versionId=null 2023-05-31T20:54:57.429 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:J16JD0bJVh2Fvhkdz8Yy615dbXM= 2023-05-31T20:54:57.429 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:54:57 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:J16JD0bJVh2Fvhkdz8Yy615dbXM=', 'Content-Length': '0'} 2023-05-31T20:54:57.432 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:39:26 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000bf2bd564b0e12f7a-006477b421-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:54:57 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:54:57.432 DEBUG:boto:path=/pklxkd-59/ 2023-05-31T20:54:57.433 DEBUG:boto:auth_path=/pklxkd-59/ 2023-05-31T20:54:57.433 DEBUG:boto:path=/pklxkd-59/?prefix=obj22 2023-05-31T20:54:57.433 DEBUG:boto:auth_path=/pklxkd-59/?prefix=obj22 2023-05-31T20:54:57.434 DEBUG:boto:Method: GET 2023-05-31T20:54:57.434 DEBUG:boto:Path: /pklxkd-59/?prefix=obj22 2023-05-31T20:54:57.435 DEBUG:boto:Data: 2023-05-31T20:54:57.435 DEBUG:boto:Headers: {} 2023-05-31T20:54:57.435 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:54:57.436 DEBUG:boto:Port: 8001 2023-05-31T20:54:57.436 DEBUG:boto:Params: {} 2023-05-31T20:54:57.436 DEBUG:boto:Token: None 2023-05-31T20:54:57.437 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:54:57 GMT /pklxkd-59/ 2023-05-31T20:54:57.437 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:aisxEfj8evvfMLEyPoSaposCVmU= 2023-05-31T20:54:57.438 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:54:57 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:aisxEfj8evvfMLEyPoSaposCVmU=', 'Content-Length': '0'} 2023-05-31T20:54:57.440 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000657d4b2ff3b2fc38-006477b421-1e2e-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:54:57 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:54:57.440 DEBUG:boto:b'pklxkd-59obj221000falseobj222023-05-31T20:39:26.473Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:54:57.441 DEBUG:boto:path=/pklxkd-59/ 2023-05-31T20:54:57.442 DEBUG:boto:auth_path=/pklxkd-59/ 2023-05-31T20:54:57.442 DEBUG:boto:path=/pklxkd-59/?prefix=obj22 2023-05-31T20:54:57.442 DEBUG:boto:auth_path=/pklxkd-59/?prefix=obj22 2023-05-31T20:54:57.443 DEBUG:boto:Method: GET 2023-05-31T20:54:57.443 DEBUG:boto:Path: /pklxkd-59/?prefix=obj22 2023-05-31T20:54:57.443 DEBUG:boto:Data: 2023-05-31T20:54:57.444 DEBUG:boto:Headers: {} 2023-05-31T20:54:57.444 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:54:57.444 DEBUG:boto:Port: 8000 2023-05-31T20:54:57.445 DEBUG:boto:Params: {} 2023-05-31T20:54:57.445 DEBUG:boto:Token: None 2023-05-31T20:54:57.445 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:54:57 GMT /pklxkd-59/ 2023-05-31T20:54:57.446 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:aisxEfj8evvfMLEyPoSaposCVmU= 2023-05-31T20:54:57.446 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:54:57 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:aisxEfj8evvfMLEyPoSaposCVmU=', 'Content-Length': '0'} 2023-05-31T20:54:57.450 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000000d3a2e9b98ab5aaa-006477b421-1630-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:54:57 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:54:57.451 DEBUG:boto:b'pklxkd-59obj221000falseobj222023-05-31T20:39:26.473Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:54:57.451 DEBUG:boto:path=/pklxkd-59/obj22 2023-05-31T20:54:57.452 DEBUG:boto:auth_path=/pklxkd-59/obj22 2023-05-31T20:54:57.452 DEBUG:boto:Method: HEAD 2023-05-31T20:54:57.452 DEBUG:boto:Path: /pklxkd-59/obj22 2023-05-31T20:54:57.453 DEBUG:boto:Data: 2023-05-31T20:54:57.453 DEBUG:boto:Headers: {} 2023-05-31T20:54:57.453 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:54:57.454 DEBUG:boto:Port: 8001 2023-05-31T20:54:57.454 DEBUG:boto:Params: {} 2023-05-31T20:54:57.454 DEBUG:boto:Token: None 2023-05-31T20:54:57.455 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:54:57 GMT /pklxkd-59/obj22 2023-05-31T20:54:57.455 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:vPOc0Hqv/5ZoU6PhXCUIRWocm0c= 2023-05-31T20:54:57.456 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:54:57 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:vPOc0Hqv/5ZoU6PhXCUIRWocm0c=', 'Content-Length': '0'} 2023-05-31T20:54:57.457 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:39:26 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e5d2da7fd4c6253a-006477b421-1e2e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:54:57 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:54:57.458 DEBUG:boto:path=/pklxkd-59/obj22 2023-05-31T20:54:57.458 DEBUG:boto:auth_path=/pklxkd-59/obj22 2023-05-31T20:54:57.459 DEBUG:boto:Method: HEAD 2023-05-31T20:54:57.459 DEBUG:boto:Path: /pklxkd-59/obj22 2023-05-31T20:54:57.459 DEBUG:boto:Data: 2023-05-31T20:54:57.460 DEBUG:boto:Headers: {} 2023-05-31T20:54:57.460 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:54:57.460 DEBUG:boto:Port: 8000 2023-05-31T20:54:57.461 DEBUG:boto:Params: {} 2023-05-31T20:54:57.461 DEBUG:boto:Token: None 2023-05-31T20:54:57.461 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:54:57 GMT /pklxkd-59/obj22 2023-05-31T20:54:57.462 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:vPOc0Hqv/5ZoU6PhXCUIRWocm0c= 2023-05-31T20:54:57.462 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:54:57 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:vPOc0Hqv/5ZoU6PhXCUIRWocm0c=', 'Content-Length': '0'} 2023-05-31T20:54:57.464 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:39:26 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006d3202aba7d44052-006477b421-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:54:57 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:54:57.464 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj22 2023-05-31T20:54:57.465 DEBUG:boto:path=/pklxkd-59/obj22 2023-05-31T20:54:57.465 DEBUG:boto:auth_path=/pklxkd-59/obj22 2023-05-31T20:54:57.466 DEBUG:boto:Method: GET 2023-05-31T20:54:57.466 DEBUG:boto:Path: /pklxkd-59/obj22 2023-05-31T20:54:57.466 DEBUG:boto:Data: 2023-05-31T20:54:57.467 DEBUG:boto:Headers: {} 2023-05-31T20:54:57.467 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:54:57.467 DEBUG:boto:Port: 8001 2023-05-31T20:54:57.468 DEBUG:boto:Params: {} 2023-05-31T20:54:57.468 DEBUG:boto:Token: None 2023-05-31T20:54:57.468 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:54:57 GMT /pklxkd-59/obj22 2023-05-31T20:54:57.469 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:41QcNOWI06w3A9XhsxlOFq6++1g= 2023-05-31T20:54:57.469 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:54:57 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:41QcNOWI06w3A9XhsxlOFq6++1g=', 'Content-Length': '0'} 2023-05-31T20:54:57.471 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:39:26 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000050270f6ba82fd062-006477b421-1e2e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:54:57 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:54:57.472 DEBUG:boto:path=/pklxkd-59/obj22 2023-05-31T20:54:57.472 DEBUG:boto:auth_path=/pklxkd-59/obj22 2023-05-31T20:54:57.472 DEBUG:boto:Method: GET 2023-05-31T20:54:57.473 DEBUG:boto:Path: /pklxkd-59/obj22 2023-05-31T20:54:57.473 DEBUG:boto:Data: 2023-05-31T20:54:57.473 DEBUG:boto:Headers: {} 2023-05-31T20:54:57.474 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:54:57.474 DEBUG:boto:Port: 8000 2023-05-31T20:54:57.474 DEBUG:boto:Params: {} 2023-05-31T20:54:57.475 DEBUG:boto:Token: None 2023-05-31T20:54:57.475 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:54:57 GMT /pklxkd-59/obj22 2023-05-31T20:54:57.476 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:41QcNOWI06w3A9XhsxlOFq6++1g= 2023-05-31T20:54:57.476 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:54:57 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:41QcNOWI06w3A9XhsxlOFq6++1g=', 'Content-Length': '0'} 2023-05-31T20:54:57.478 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:39:26 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009509b15a8a90f813-006477b421-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:54:57 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:54:57.520 DEBUG:boto:path=/pklxkd-59/ 2023-05-31T20:54:57.521 DEBUG:boto:auth_path=/pklxkd-59/ 2023-05-31T20:54:57.521 DEBUG:boto:path=/pklxkd-59/?prefix=obj22 2023-05-31T20:54:57.521 DEBUG:boto:auth_path=/pklxkd-59/?prefix=obj22 2023-05-31T20:54:57.522 DEBUG:boto:Method: GET 2023-05-31T20:54:57.522 DEBUG:boto:Path: /pklxkd-59/?prefix=obj22 2023-05-31T20:54:57.522 DEBUG:boto:Data: 2023-05-31T20:54:57.523 DEBUG:boto:Headers: {} 2023-05-31T20:54:57.523 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:54:57.523 DEBUG:boto:Port: 8001 2023-05-31T20:54:57.524 DEBUG:boto:Params: {} 2023-05-31T20:54:57.524 DEBUG:boto:Token: None 2023-05-31T20:54:57.525 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:54:57 GMT /pklxkd-59/ 2023-05-31T20:54:57.525 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:aisxEfj8evvfMLEyPoSaposCVmU= 2023-05-31T20:54:57.525 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:54:57 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:aisxEfj8evvfMLEyPoSaposCVmU=', 'Content-Length': '0'} 2023-05-31T20:54:57.528 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c4e915eeb19cbce8-006477b421-1e2e-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:54:57 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:54:57.528 DEBUG:boto:b'pklxkd-59obj221000falseobj222023-05-31T20:39:26.473Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:54:57.529 DEBUG:boto:path=/pklxkd-59/ 2023-05-31T20:54:57.529 DEBUG:boto:auth_path=/pklxkd-59/ 2023-05-31T20:54:57.530 DEBUG:boto:path=/pklxkd-59/?prefix=obj22 2023-05-31T20:54:57.530 DEBUG:boto:auth_path=/pklxkd-59/?prefix=obj22 2023-05-31T20:54:57.530 DEBUG:boto:Method: GET 2023-05-31T20:54:57.531 DEBUG:boto:Path: /pklxkd-59/?prefix=obj22 2023-05-31T20:54:57.531 DEBUG:boto:Data: 2023-05-31T20:54:57.531 DEBUG:boto:Headers: {} 2023-05-31T20:54:57.532 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:54:57.532 DEBUG:boto:Port: 8000 2023-05-31T20:54:57.532 DEBUG:boto:Params: {} 2023-05-31T20:54:57.533 DEBUG:boto:Token: None 2023-05-31T20:54:57.533 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:54:57 GMT /pklxkd-59/ 2023-05-31T20:54:57.534 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:aisxEfj8evvfMLEyPoSaposCVmU= 2023-05-31T20:54:57.534 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:54:57 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:aisxEfj8evvfMLEyPoSaposCVmU=', 'Content-Length': '0'} 2023-05-31T20:54:57.538 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000098d16ae856404900-006477b421-1630-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:54:57 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:54:57.539 DEBUG:boto:b'pklxkd-59obj221000falseobj222023-05-31T20:39:26.473Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:54:57.539 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj23 2023-05-31T20:54:57.540 DEBUG:boto:path=/pklxkd-59/obj23 2023-05-31T20:54:57.540 DEBUG:boto:auth_path=/pklxkd-59/obj23 2023-05-31T20:54:57.540 DEBUG:boto:path=/pklxkd-59/obj23?versionId=null 2023-05-31T20:54:57.541 DEBUG:boto:auth_path=/pklxkd-59/obj23?versionId=null 2023-05-31T20:54:57.541 DEBUG:boto:Method: GET 2023-05-31T20:54:57.542 DEBUG:boto:Path: /pklxkd-59/obj23?versionId=null 2023-05-31T20:54:57.542 DEBUG:boto:Data: 2023-05-31T20:54:57.542 DEBUG:boto:Headers: {} 2023-05-31T20:54:57.543 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:54:57.543 DEBUG:boto:Port: 8001 2023-05-31T20:54:57.543 DEBUG:boto:Params: {} 2023-05-31T20:54:57.544 DEBUG:boto:Token: None 2023-05-31T20:54:57.544 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:54:57 GMT /pklxkd-59/obj23?versionId=null 2023-05-31T20:54:57.545 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:habOo2my3mTZt+e4+5iAvCEZ5jM= 2023-05-31T20:54:57.545 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:54:57 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:habOo2my3mTZt+e4+5iAvCEZ5jM=', 'Content-Length': '0'} 2023-05-31T20:54:57.547 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:46:33 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000086cda774b4255afd-006477b421-1e2e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:54:57 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:54:57.548 DEBUG:boto:path=/pklxkd-59/obj23 2023-05-31T20:54:57.548 DEBUG:boto:auth_path=/pklxkd-59/obj23 2023-05-31T20:54:57.548 DEBUG:boto:path=/pklxkd-59/obj23?versionId=null 2023-05-31T20:54:57.549 DEBUG:boto:auth_path=/pklxkd-59/obj23?versionId=null 2023-05-31T20:54:57.549 DEBUG:boto:Method: GET 2023-05-31T20:54:57.550 DEBUG:boto:Path: /pklxkd-59/obj23?versionId=null 2023-05-31T20:54:57.550 DEBUG:boto:Data: 2023-05-31T20:54:57.550 DEBUG:boto:Headers: {} 2023-05-31T20:54:57.551 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:54:57.551 DEBUG:boto:Port: 8000 2023-05-31T20:54:57.551 DEBUG:boto:Params: {} 2023-05-31T20:54:57.552 DEBUG:boto:Token: None 2023-05-31T20:54:57.552 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:54:57 GMT /pklxkd-59/obj23?versionId=null 2023-05-31T20:54:57.553 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:habOo2my3mTZt+e4+5iAvCEZ5jM= 2023-05-31T20:54:57.553 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:54:57 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:habOo2my3mTZt+e4+5iAvCEZ5jM=', 'Content-Length': '0'} 2023-05-31T20:54:57.555 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:46:33 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003ed75f15dfea821d-006477b421-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:54:57 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:54:57.555 DEBUG:boto:path=/pklxkd-59/obj23 2023-05-31T20:54:57.556 DEBUG:boto:auth_path=/pklxkd-59/obj23 2023-05-31T20:54:57.556 DEBUG:boto:path=/pklxkd-59/obj23?versionId=null 2023-05-31T20:54:57.556 DEBUG:boto:auth_path=/pklxkd-59/obj23?versionId=null 2023-05-31T20:54:57.557 DEBUG:boto:Method: HEAD 2023-05-31T20:54:57.557 DEBUG:boto:Path: /pklxkd-59/obj23?versionId=null 2023-05-31T20:54:57.557 DEBUG:boto:Data: 2023-05-31T20:54:57.558 DEBUG:boto:Headers: {} 2023-05-31T20:54:57.558 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:54:57.558 DEBUG:boto:Port: 8001 2023-05-31T20:54:57.559 DEBUG:boto:Params: {} 2023-05-31T20:54:57.559 DEBUG:boto:Token: None 2023-05-31T20:54:57.559 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:54:57 GMT /pklxkd-59/obj23?versionId=null 2023-05-31T20:54:57.560 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:nNtdVJdVdx47qKhJ/KremaIrskA= 2023-05-31T20:54:57.560 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:54:57 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:nNtdVJdVdx47qKhJ/KremaIrskA=', 'Content-Length': '0'} 2023-05-31T20:54:57.562 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:46:33 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c9fac60835d6b5fa-006477b421-1e2e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:54:57 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:54:57.563 DEBUG:boto:path=/pklxkd-59/obj23 2023-05-31T20:54:57.563 DEBUG:boto:auth_path=/pklxkd-59/obj23 2023-05-31T20:54:57.563 DEBUG:boto:path=/pklxkd-59/obj23?versionId=null 2023-05-31T20:54:57.564 DEBUG:boto:auth_path=/pklxkd-59/obj23?versionId=null 2023-05-31T20:54:57.564 DEBUG:boto:Method: HEAD 2023-05-31T20:54:57.565 DEBUG:boto:Path: /pklxkd-59/obj23?versionId=null 2023-05-31T20:54:57.565 DEBUG:boto:Data: 2023-05-31T20:54:57.565 DEBUG:boto:Headers: {} 2023-05-31T20:54:57.566 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:54:57.566 DEBUG:boto:Port: 8000 2023-05-31T20:54:57.566 DEBUG:boto:Params: {} 2023-05-31T20:54:57.567 DEBUG:boto:Token: None 2023-05-31T20:54:57.567 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:54:57 GMT /pklxkd-59/obj23?versionId=null 2023-05-31T20:54:57.567 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:nNtdVJdVdx47qKhJ/KremaIrskA= 2023-05-31T20:54:57.568 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:54:57 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:nNtdVJdVdx47qKhJ/KremaIrskA=', 'Content-Length': '0'} 2023-05-31T20:54:57.570 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:46:33 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009616552e500dd06f-006477b421-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:54:57 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:54:57.570 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj23 2023-05-31T20:54:57.571 DEBUG:boto:path=/pklxkd-59/obj23 2023-05-31T20:54:57.571 DEBUG:boto:auth_path=/pklxkd-59/obj23 2023-05-31T20:54:57.571 DEBUG:boto:path=/pklxkd-59/obj23?versionId=null 2023-05-31T20:54:57.572 DEBUG:boto:auth_path=/pklxkd-59/obj23?versionId=null 2023-05-31T20:54:57.572 DEBUG:boto:Method: GET 2023-05-31T20:54:57.572 DEBUG:boto:Path: /pklxkd-59/obj23?versionId=null 2023-05-31T20:54:57.573 DEBUG:boto:Data: 2023-05-31T20:54:57.573 DEBUG:boto:Headers: {} 2023-05-31T20:54:57.573 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:54:57.574 DEBUG:boto:Port: 8001 2023-05-31T20:54:57.574 DEBUG:boto:Params: {} 2023-05-31T20:54:57.575 DEBUG:boto:Token: None 2023-05-31T20:54:57.575 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:54:57 GMT /pklxkd-59/obj23?versionId=null 2023-05-31T20:54:57.575 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:habOo2my3mTZt+e4+5iAvCEZ5jM= 2023-05-31T20:54:57.576 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:54:57 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:habOo2my3mTZt+e4+5iAvCEZ5jM=', 'Content-Length': '0'} 2023-05-31T20:54:57.577 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:46:33 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a5ebc6e39cf09fa4-006477b421-1e2e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:54:57 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:54:57.578 DEBUG:boto:path=/pklxkd-59/obj23 2023-05-31T20:54:57.578 DEBUG:boto:auth_path=/pklxkd-59/obj23 2023-05-31T20:54:57.579 DEBUG:boto:path=/pklxkd-59/obj23?versionId=null 2023-05-31T20:54:57.579 DEBUG:boto:auth_path=/pklxkd-59/obj23?versionId=null 2023-05-31T20:54:57.580 DEBUG:boto:Method: GET 2023-05-31T20:54:57.580 DEBUG:boto:Path: /pklxkd-59/obj23?versionId=null 2023-05-31T20:54:57.580 DEBUG:boto:Data: 2023-05-31T20:54:57.581 DEBUG:boto:Headers: {} 2023-05-31T20:54:57.581 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:54:57.581 DEBUG:boto:Port: 8000 2023-05-31T20:54:57.582 DEBUG:boto:Params: {} 2023-05-31T20:54:57.582 DEBUG:boto:Token: None 2023-05-31T20:54:57.582 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:54:57 GMT /pklxkd-59/obj23?versionId=null 2023-05-31T20:54:57.583 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:habOo2my3mTZt+e4+5iAvCEZ5jM= 2023-05-31T20:54:57.583 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:54:57 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:habOo2my3mTZt+e4+5iAvCEZ5jM=', 'Content-Length': '0'} 2023-05-31T20:54:57.585 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:46:33 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000187ea601638f1bb5-006477b421-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:54:57 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:54:57.586 DEBUG:boto:path=/pklxkd-59/ 2023-05-31T20:54:57.586 DEBUG:boto:auth_path=/pklxkd-59/ 2023-05-31T20:54:57.586 DEBUG:boto:path=/pklxkd-59/?prefix=obj23 2023-05-31T20:54:57.587 DEBUG:boto:auth_path=/pklxkd-59/?prefix=obj23 2023-05-31T20:54:57.587 DEBUG:boto:Method: GET 2023-05-31T20:54:57.588 DEBUG:boto:Path: /pklxkd-59/?prefix=obj23 2023-05-31T20:54:57.588 DEBUG:boto:Data: 2023-05-31T20:54:57.588 DEBUG:boto:Headers: {} 2023-05-31T20:54:57.589 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:54:57.589 DEBUG:boto:Port: 8001 2023-05-31T20:54:57.589 DEBUG:boto:Params: {} 2023-05-31T20:54:57.590 DEBUG:boto:Token: None 2023-05-31T20:54:57.590 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:54:57 GMT /pklxkd-59/ 2023-05-31T20:54:57.591 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:aisxEfj8evvfMLEyPoSaposCVmU= 2023-05-31T20:54:57.591 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:54:57 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:aisxEfj8evvfMLEyPoSaposCVmU=', 'Content-Length': '0'} 2023-05-31T20:54:57.594 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000001798f28a1655e5e-006477b421-1e2e-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:54:57 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:54:57.594 DEBUG:boto:b'pklxkd-59obj231000falseobj232023-05-31T20:46:33.813Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:54:57.595 DEBUG:boto:path=/pklxkd-59/ 2023-05-31T20:54:57.595 DEBUG:boto:auth_path=/pklxkd-59/ 2023-05-31T20:54:57.595 DEBUG:boto:path=/pklxkd-59/?prefix=obj23 2023-05-31T20:54:57.596 DEBUG:boto:auth_path=/pklxkd-59/?prefix=obj23 2023-05-31T20:54:57.596 DEBUG:boto:Method: GET 2023-05-31T20:54:57.596 DEBUG:boto:Path: /pklxkd-59/?prefix=obj23 2023-05-31T20:54:57.597 DEBUG:boto:Data: 2023-05-31T20:54:57.597 DEBUG:boto:Headers: {} 2023-05-31T20:54:57.598 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:54:57.598 DEBUG:boto:Port: 8000 2023-05-31T20:54:57.598 DEBUG:boto:Params: {} 2023-05-31T20:54:57.599 DEBUG:boto:Token: None 2023-05-31T20:54:57.599 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:54:57 GMT /pklxkd-59/ 2023-05-31T20:54:57.599 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:aisxEfj8evvfMLEyPoSaposCVmU= 2023-05-31T20:54:57.600 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:54:57 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:aisxEfj8evvfMLEyPoSaposCVmU=', 'Content-Length': '0'} 2023-05-31T20:54:57.604 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ce8b34e270509552-006477b421-1630-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:54:57 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:54:57.644 DEBUG:boto:b'pklxkd-59obj231000falseobj232023-05-31T20:46:33.813Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:54:57.645 DEBUG:boto:path=/pklxkd-59/obj23 2023-05-31T20:54:57.645 DEBUG:boto:auth_path=/pklxkd-59/obj23 2023-05-31T20:54:57.645 DEBUG:boto:Method: HEAD 2023-05-31T20:54:57.646 DEBUG:boto:Path: /pklxkd-59/obj23 2023-05-31T20:54:57.646 DEBUG:boto:Data: 2023-05-31T20:54:57.647 DEBUG:boto:Headers: {} 2023-05-31T20:54:57.647 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:54:57.647 DEBUG:boto:Port: 8001 2023-05-31T20:54:57.647 DEBUG:boto:Params: {} 2023-05-31T20:54:57.648 DEBUG:boto:Token: None 2023-05-31T20:54:57.648 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:54:57 GMT /pklxkd-59/obj23 2023-05-31T20:54:57.649 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:8ehYdQXxYRpN753Ue+MLsE28fCU= 2023-05-31T20:54:57.649 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:54:57 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:8ehYdQXxYRpN753Ue+MLsE28fCU=', 'Content-Length': '0'} 2023-05-31T20:54:57.651 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:46:33 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e5ebff3639d42595-006477b421-1e2e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:54:57 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:54:57.652 DEBUG:boto:path=/pklxkd-59/obj23 2023-05-31T20:54:57.652 DEBUG:boto:auth_path=/pklxkd-59/obj23 2023-05-31T20:54:57.652 DEBUG:boto:Method: HEAD 2023-05-31T20:54:57.653 DEBUG:boto:Path: /pklxkd-59/obj23 2023-05-31T20:54:57.653 DEBUG:boto:Data: 2023-05-31T20:54:57.653 DEBUG:boto:Headers: {} 2023-05-31T20:54:57.654 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:54:57.654 DEBUG:boto:Port: 8000 2023-05-31T20:54:57.654 DEBUG:boto:Params: {} 2023-05-31T20:54:57.655 DEBUG:boto:Token: None 2023-05-31T20:54:57.655 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:54:57 GMT /pklxkd-59/obj23 2023-05-31T20:54:57.655 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:8ehYdQXxYRpN753Ue+MLsE28fCU= 2023-05-31T20:54:57.656 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:54:57 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:8ehYdQXxYRpN753Ue+MLsE28fCU=', 'Content-Length': '0'} 2023-05-31T20:54:57.657 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:46:33 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f5623da2b3f407f4-006477b421-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:54:57 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:54:57.658 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj23 2023-05-31T20:54:57.658 DEBUG:boto:path=/pklxkd-59/obj23 2023-05-31T20:54:57.659 DEBUG:boto:auth_path=/pklxkd-59/obj23 2023-05-31T20:54:57.659 DEBUG:boto:Method: GET 2023-05-31T20:54:57.660 DEBUG:boto:Path: /pklxkd-59/obj23 2023-05-31T20:54:57.660 DEBUG:boto:Data: 2023-05-31T20:54:57.660 DEBUG:boto:Headers: {} 2023-05-31T20:54:57.661 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:54:57.661 DEBUG:boto:Port: 8001 2023-05-31T20:54:57.661 DEBUG:boto:Params: {} 2023-05-31T20:54:57.662 DEBUG:boto:Token: None 2023-05-31T20:54:57.662 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:54:57 GMT /pklxkd-59/obj23 2023-05-31T20:54:57.662 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:zyx7QXuesB9LlTYesFTnoFtt8mo= 2023-05-31T20:54:57.663 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:54:57 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:zyx7QXuesB9LlTYesFTnoFtt8mo=', 'Content-Length': '0'} 2023-05-31T20:54:57.665 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:46:33 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e28d46fe482e39f0-006477b421-1e2e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:54:57 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:54:57.665 DEBUG:boto:path=/pklxkd-59/obj23 2023-05-31T20:54:57.666 DEBUG:boto:auth_path=/pklxkd-59/obj23 2023-05-31T20:54:57.666 DEBUG:boto:Method: GET 2023-05-31T20:54:57.667 DEBUG:boto:Path: /pklxkd-59/obj23 2023-05-31T20:54:57.667 DEBUG:boto:Data: 2023-05-31T20:54:57.667 DEBUG:boto:Headers: {} 2023-05-31T20:54:57.668 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:54:57.668 DEBUG:boto:Port: 8000 2023-05-31T20:54:57.668 DEBUG:boto:Params: {} 2023-05-31T20:54:57.669 DEBUG:boto:Token: None 2023-05-31T20:54:57.669 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:54:57 GMT /pklxkd-59/obj23 2023-05-31T20:54:57.670 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:zyx7QXuesB9LlTYesFTnoFtt8mo= 2023-05-31T20:54:57.670 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:54:57 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:zyx7QXuesB9LlTYesFTnoFtt8mo=', 'Content-Length': '0'} 2023-05-31T20:54:57.672 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:46:33 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000091607cf8e5d9eba3-006477b421-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:54:57 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:54:57.673 DEBUG:boto:path=/pklxkd-59/ 2023-05-31T20:54:57.673 DEBUG:boto:auth_path=/pklxkd-59/ 2023-05-31T20:54:57.673 DEBUG:boto:path=/pklxkd-59/?prefix=obj23 2023-05-31T20:54:57.674 DEBUG:boto:auth_path=/pklxkd-59/?prefix=obj23 2023-05-31T20:54:57.674 DEBUG:boto:Method: GET 2023-05-31T20:54:57.674 DEBUG:boto:Path: /pklxkd-59/?prefix=obj23 2023-05-31T20:54:57.675 DEBUG:boto:Data: 2023-05-31T20:54:57.675 DEBUG:boto:Headers: {} 2023-05-31T20:54:57.675 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:54:57.676 DEBUG:boto:Port: 8001 2023-05-31T20:54:57.676 DEBUG:boto:Params: {} 2023-05-31T20:54:57.676 DEBUG:boto:Token: None 2023-05-31T20:54:57.677 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:54:57 GMT /pklxkd-59/ 2023-05-31T20:54:57.677 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:aisxEfj8evvfMLEyPoSaposCVmU= 2023-05-31T20:54:57.678 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:54:57 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:aisxEfj8evvfMLEyPoSaposCVmU=', 'Content-Length': '0'} 2023-05-31T20:54:57.680 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000037bd8d9b9c423deb-006477b421-1e2e-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:54:57 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:54:57.681 DEBUG:boto:b'pklxkd-59obj231000falseobj232023-05-31T20:46:33.813Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:54:57.681 DEBUG:boto:path=/pklxkd-59/ 2023-05-31T20:54:57.682 DEBUG:boto:auth_path=/pklxkd-59/ 2023-05-31T20:54:57.682 DEBUG:boto:path=/pklxkd-59/?prefix=obj23 2023-05-31T20:54:57.682 DEBUG:boto:auth_path=/pklxkd-59/?prefix=obj23 2023-05-31T20:54:57.683 DEBUG:boto:Method: GET 2023-05-31T20:54:57.683 DEBUG:boto:Path: /pklxkd-59/?prefix=obj23 2023-05-31T20:54:57.683 DEBUG:boto:Data: 2023-05-31T20:54:57.684 DEBUG:boto:Headers: {} 2023-05-31T20:54:57.684 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:54:57.684 DEBUG:boto:Port: 8000 2023-05-31T20:54:57.685 DEBUG:boto:Params: {} 2023-05-31T20:54:57.685 DEBUG:boto:Token: None 2023-05-31T20:54:57.686 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:54:57 GMT /pklxkd-59/ 2023-05-31T20:54:57.686 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:aisxEfj8evvfMLEyPoSaposCVmU= 2023-05-31T20:54:57.686 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:54:57 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:aisxEfj8evvfMLEyPoSaposCVmU=', 'Content-Length': '0'} 2023-05-31T20:54:57.690 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000040b959f70e9eaf62-006477b421-1630-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:54:57 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:54:57.691 DEBUG:boto:b'pklxkd-59obj231000falseobj232023-05-31T20:46:33.813Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:54:57.691 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj29 2023-05-31T20:54:57.692 DEBUG:boto:path=/pklxkd-59/obj29 2023-05-31T20:54:57.692 DEBUG:boto:auth_path=/pklxkd-59/obj29 2023-05-31T20:54:57.693 DEBUG:boto:path=/pklxkd-59/obj29?versionId=null 2023-05-31T20:54:57.693 DEBUG:boto:auth_path=/pklxkd-59/obj29?versionId=null 2023-05-31T20:54:57.693 DEBUG:boto:Method: GET 2023-05-31T20:54:57.694 DEBUG:boto:Path: /pklxkd-59/obj29?versionId=null 2023-05-31T20:54:57.694 DEBUG:boto:Data: 2023-05-31T20:54:57.694 DEBUG:boto:Headers: {} 2023-05-31T20:54:57.695 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:54:57.695 DEBUG:boto:Port: 8001 2023-05-31T20:54:57.695 DEBUG:boto:Params: {} 2023-05-31T20:54:57.696 DEBUG:boto:Token: None 2023-05-31T20:54:57.696 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:54:57 GMT /pklxkd-59/obj29?versionId=null 2023-05-31T20:54:57.697 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:NtGdgedE29WromgudJzRAZ6R130= 2023-05-31T20:54:57.697 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:54:57 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:NtGdgedE29WromgudJzRAZ6R130=', 'Content-Length': '0'} 2023-05-31T20:54:57.699 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:48:12 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ec00511ee19e142c-006477b421-1e2e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:54:57 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:54:57.744 DEBUG:boto:path=/pklxkd-59/obj29 2023-05-31T20:54:57.745 DEBUG:boto:auth_path=/pklxkd-59/obj29 2023-05-31T20:54:57.745 DEBUG:boto:path=/pklxkd-59/obj29?versionId=null 2023-05-31T20:54:57.746 DEBUG:boto:auth_path=/pklxkd-59/obj29?versionId=null 2023-05-31T20:54:57.746 DEBUG:boto:Method: GET 2023-05-31T20:54:57.746 DEBUG:boto:Path: /pklxkd-59/obj29?versionId=null 2023-05-31T20:54:57.747 DEBUG:boto:Data: 2023-05-31T20:54:57.747 DEBUG:boto:Headers: {} 2023-05-31T20:54:57.747 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:54:57.748 DEBUG:boto:Port: 8000 2023-05-31T20:54:57.748 DEBUG:boto:Params: {} 2023-05-31T20:54:57.749 DEBUG:boto:Token: None 2023-05-31T20:54:57.749 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:54:57 GMT /pklxkd-59/obj29?versionId=null 2023-05-31T20:54:57.749 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:NtGdgedE29WromgudJzRAZ6R130= 2023-05-31T20:54:57.750 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:54:57 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:NtGdgedE29WromgudJzRAZ6R130=', 'Content-Length': '0'} 2023-05-31T20:54:57.752 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:48:12 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000541ec9819204e0d0-006477b421-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:54:57 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:54:57.753 DEBUG:boto:path=/pklxkd-59/obj29 2023-05-31T20:54:57.753 DEBUG:boto:auth_path=/pklxkd-59/obj29 2023-05-31T20:54:57.754 DEBUG:boto:path=/pklxkd-59/obj29?versionId=null 2023-05-31T20:54:57.754 DEBUG:boto:auth_path=/pklxkd-59/obj29?versionId=null 2023-05-31T20:54:57.754 DEBUG:boto:Method: HEAD 2023-05-31T20:54:57.755 DEBUG:boto:Path: /pklxkd-59/obj29?versionId=null 2023-05-31T20:54:57.755 DEBUG:boto:Data: 2023-05-31T20:54:57.755 DEBUG:boto:Headers: {} 2023-05-31T20:54:57.756 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:54:57.756 DEBUG:boto:Port: 8001 2023-05-31T20:54:57.756 DEBUG:boto:Params: {} 2023-05-31T20:54:57.757 DEBUG:boto:Token: None 2023-05-31T20:54:57.757 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:54:57 GMT /pklxkd-59/obj29?versionId=null 2023-05-31T20:54:57.758 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:EJdazOuIHd0H6YgFI7IXu5IPcaQ= 2023-05-31T20:54:57.758 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:54:57 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:EJdazOuIHd0H6YgFI7IXu5IPcaQ=', 'Content-Length': '0'} 2023-05-31T20:54:57.760 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:48:12 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000096e9e037cd59de87-006477b421-1e2e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:54:57 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:54:57.760 DEBUG:boto:path=/pklxkd-59/obj29 2023-05-31T20:54:57.761 DEBUG:boto:auth_path=/pklxkd-59/obj29 2023-05-31T20:54:57.761 DEBUG:boto:path=/pklxkd-59/obj29?versionId=null 2023-05-31T20:54:57.761 DEBUG:boto:auth_path=/pklxkd-59/obj29?versionId=null 2023-05-31T20:54:57.762 DEBUG:boto:Method: HEAD 2023-05-31T20:54:57.762 DEBUG:boto:Path: /pklxkd-59/obj29?versionId=null 2023-05-31T20:54:57.762 DEBUG:boto:Data: 2023-05-31T20:54:57.763 DEBUG:boto:Headers: {} 2023-05-31T20:54:57.763 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:54:57.763 DEBUG:boto:Port: 8000 2023-05-31T20:54:57.764 DEBUG:boto:Params: {} 2023-05-31T20:54:57.764 DEBUG:boto:Token: None 2023-05-31T20:54:57.765 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:54:57 GMT /pklxkd-59/obj29?versionId=null 2023-05-31T20:54:57.765 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:EJdazOuIHd0H6YgFI7IXu5IPcaQ= 2023-05-31T20:54:57.766 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:54:57 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:EJdazOuIHd0H6YgFI7IXu5IPcaQ=', 'Content-Length': '0'} 2023-05-31T20:54:57.767 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:48:12 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ad80c4920bd2475c-006477b421-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:54:57 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:54:57.768 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj29 2023-05-31T20:54:57.768 DEBUG:boto:path=/pklxkd-59/obj29 2023-05-31T20:54:57.769 DEBUG:boto:auth_path=/pklxkd-59/obj29 2023-05-31T20:54:57.769 DEBUG:boto:path=/pklxkd-59/obj29?versionId=null 2023-05-31T20:54:57.769 DEBUG:boto:auth_path=/pklxkd-59/obj29?versionId=null 2023-05-31T20:54:57.770 DEBUG:boto:Method: GET 2023-05-31T20:54:57.770 DEBUG:boto:Path: /pklxkd-59/obj29?versionId=null 2023-05-31T20:54:57.771 DEBUG:boto:Data: 2023-05-31T20:54:57.771 DEBUG:boto:Headers: {} 2023-05-31T20:54:57.771 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:54:57.772 DEBUG:boto:Port: 8001 2023-05-31T20:54:57.772 DEBUG:boto:Params: {} 2023-05-31T20:54:57.772 DEBUG:boto:Token: None 2023-05-31T20:54:57.773 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:54:57 GMT /pklxkd-59/obj29?versionId=null 2023-05-31T20:54:57.773 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:NtGdgedE29WromgudJzRAZ6R130= 2023-05-31T20:54:57.774 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:54:57 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:NtGdgedE29WromgudJzRAZ6R130=', 'Content-Length': '0'} 2023-05-31T20:54:57.775 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:48:12 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000051e7383a7e9d8b04-006477b421-1e2e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:54:57 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:54:57.776 DEBUG:boto:path=/pklxkd-59/obj29 2023-05-31T20:54:57.777 DEBUG:boto:auth_path=/pklxkd-59/obj29 2023-05-31T20:54:57.777 DEBUG:boto:path=/pklxkd-59/obj29?versionId=null 2023-05-31T20:54:57.777 DEBUG:boto:auth_path=/pklxkd-59/obj29?versionId=null 2023-05-31T20:54:57.778 DEBUG:boto:Method: GET 2023-05-31T20:54:57.778 DEBUG:boto:Path: /pklxkd-59/obj29?versionId=null 2023-05-31T20:54:57.778 DEBUG:boto:Data: 2023-05-31T20:54:57.779 DEBUG:boto:Headers: {} 2023-05-31T20:54:57.779 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:54:57.780 DEBUG:boto:Port: 8000 2023-05-31T20:54:57.780 DEBUG:boto:Params: {} 2023-05-31T20:54:57.780 DEBUG:boto:Token: None 2023-05-31T20:54:57.781 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:54:57 GMT /pklxkd-59/obj29?versionId=null 2023-05-31T20:54:57.781 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:NtGdgedE29WromgudJzRAZ6R130= 2023-05-31T20:54:57.782 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:54:57 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:NtGdgedE29WromgudJzRAZ6R130=', 'Content-Length': '0'} 2023-05-31T20:54:57.784 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:48:12 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e3c3ca81073c480d-006477b421-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:54:57 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:54:57.784 DEBUG:boto:path=/pklxkd-59/ 2023-05-31T20:54:57.785 DEBUG:boto:auth_path=/pklxkd-59/ 2023-05-31T20:54:57.785 DEBUG:boto:path=/pklxkd-59/?prefix=obj29 2023-05-31T20:54:57.785 DEBUG:boto:auth_path=/pklxkd-59/?prefix=obj29 2023-05-31T20:54:57.786 DEBUG:boto:Method: GET 2023-05-31T20:54:57.786 DEBUG:boto:Path: /pklxkd-59/?prefix=obj29 2023-05-31T20:54:57.786 DEBUG:boto:Data: 2023-05-31T20:54:57.787 DEBUG:boto:Headers: {} 2023-05-31T20:54:57.787 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:54:57.787 DEBUG:boto:Port: 8001 2023-05-31T20:54:57.788 DEBUG:boto:Params: {} 2023-05-31T20:54:57.788 DEBUG:boto:Token: None 2023-05-31T20:54:57.789 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:54:57 GMT /pklxkd-59/ 2023-05-31T20:54:57.789 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:aisxEfj8evvfMLEyPoSaposCVmU= 2023-05-31T20:54:57.790 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:54:57 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:aisxEfj8evvfMLEyPoSaposCVmU=', 'Content-Length': '0'} 2023-05-31T20:54:57.792 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c1b6836e64069efb-006477b421-1e2e-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:54:57 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:54:57.792 DEBUG:boto:b'pklxkd-59obj291000falseobj292023-05-31T20:48:12.956Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:54:57.793 DEBUG:boto:path=/pklxkd-59/ 2023-05-31T20:54:57.793 DEBUG:boto:auth_path=/pklxkd-59/ 2023-05-31T20:54:57.794 DEBUG:boto:path=/pklxkd-59/?prefix=obj29 2023-05-31T20:54:57.794 DEBUG:boto:auth_path=/pklxkd-59/?prefix=obj29 2023-05-31T20:54:57.794 DEBUG:boto:Method: GET 2023-05-31T20:54:57.795 DEBUG:boto:Path: /pklxkd-59/?prefix=obj29 2023-05-31T20:54:57.795 DEBUG:boto:Data: 2023-05-31T20:54:57.795 DEBUG:boto:Headers: {} 2023-05-31T20:54:57.796 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:54:57.796 DEBUG:boto:Port: 8000 2023-05-31T20:54:57.797 DEBUG:boto:Params: {} 2023-05-31T20:54:57.797 DEBUG:boto:Token: None 2023-05-31T20:54:57.797 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:54:57 GMT /pklxkd-59/ 2023-05-31T20:54:57.798 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:aisxEfj8evvfMLEyPoSaposCVmU= 2023-05-31T20:54:57.798 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:54:57 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:aisxEfj8evvfMLEyPoSaposCVmU=', 'Content-Length': '0'} 2023-05-31T20:54:57.802 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000004ac390462f5cd984-006477b421-1630-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:54:57 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:54:57.803 DEBUG:boto:b'pklxkd-59obj291000falseobj292023-05-31T20:48:12.956Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:54:57.803 DEBUG:boto:path=/pklxkd-59/obj29 2023-05-31T20:54:57.804 DEBUG:boto:auth_path=/pklxkd-59/obj29 2023-05-31T20:54:57.804 DEBUG:boto:Method: HEAD 2023-05-31T20:54:57.805 DEBUG:boto:Path: /pklxkd-59/obj29 2023-05-31T20:54:57.805 DEBUG:boto:Data: 2023-05-31T20:54:57.805 DEBUG:boto:Headers: {} 2023-05-31T20:54:57.806 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:54:57.806 DEBUG:boto:Port: 8001 2023-05-31T20:54:57.806 DEBUG:boto:Params: {} 2023-05-31T20:54:57.807 DEBUG:boto:Token: None 2023-05-31T20:54:57.807 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:54:57 GMT /pklxkd-59/obj29 2023-05-31T20:54:57.808 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:O+I7WAOo0HuAZ1mTLkk3VzYxIiA= 2023-05-31T20:54:57.808 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:54:57 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:O+I7WAOo0HuAZ1mTLkk3VzYxIiA=', 'Content-Length': '0'} 2023-05-31T20:54:57.810 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:48:12 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006071c9029ac20d64-006477b421-1e2e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:54:57 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:54:57.810 DEBUG:boto:path=/pklxkd-59/obj29 2023-05-31T20:54:57.811 DEBUG:boto:auth_path=/pklxkd-59/obj29 2023-05-31T20:54:57.811 DEBUG:boto:Method: HEAD 2023-05-31T20:54:57.811 DEBUG:boto:Path: /pklxkd-59/obj29 2023-05-31T20:54:57.812 DEBUG:boto:Data: 2023-05-31T20:54:57.812 DEBUG:boto:Headers: {} 2023-05-31T20:54:57.813 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:54:57.813 DEBUG:boto:Port: 8000 2023-05-31T20:54:57.813 DEBUG:boto:Params: {} 2023-05-31T20:54:57.814 DEBUG:boto:Token: None 2023-05-31T20:54:57.814 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:54:57 GMT /pklxkd-59/obj29 2023-05-31T20:54:57.814 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:O+I7WAOo0HuAZ1mTLkk3VzYxIiA= 2023-05-31T20:54:57.815 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:54:57 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:O+I7WAOo0HuAZ1mTLkk3VzYxIiA=', 'Content-Length': '0'} 2023-05-31T20:54:57.816 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:48:12 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000062cc9d6343b0d533-006477b421-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:54:57 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:54:57.817 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj29 2023-05-31T20:54:57.817 DEBUG:boto:path=/pklxkd-59/obj29 2023-05-31T20:54:57.818 DEBUG:boto:auth_path=/pklxkd-59/obj29 2023-05-31T20:54:57.818 DEBUG:boto:Method: GET 2023-05-31T20:54:57.818 DEBUG:boto:Path: /pklxkd-59/obj29 2023-05-31T20:54:57.819 DEBUG:boto:Data: 2023-05-31T20:54:57.819 DEBUG:boto:Headers: {} 2023-05-31T20:54:57.820 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:54:57.820 DEBUG:boto:Port: 8001 2023-05-31T20:54:57.820 DEBUG:boto:Params: {} 2023-05-31T20:54:57.821 DEBUG:boto:Token: None 2023-05-31T20:54:57.821 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:54:57 GMT /pklxkd-59/obj29 2023-05-31T20:54:57.821 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:XRq77g+H0D8f4HCtFEmbWFXpmwc= 2023-05-31T20:54:57.822 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:54:57 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:XRq77g+H0D8f4HCtFEmbWFXpmwc=', 'Content-Length': '0'} 2023-05-31T20:54:57.824 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:48:12 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000fd6d80cfca958236-006477b421-1e2e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:54:57 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:54:57.824 DEBUG:boto:path=/pklxkd-59/obj29 2023-05-31T20:54:57.825 DEBUG:boto:auth_path=/pklxkd-59/obj29 2023-05-31T20:54:57.825 DEBUG:boto:Method: GET 2023-05-31T20:54:57.825 DEBUG:boto:Path: /pklxkd-59/obj29 2023-05-31T20:54:57.826 DEBUG:boto:Data: 2023-05-31T20:54:57.826 DEBUG:boto:Headers: {} 2023-05-31T20:54:57.827 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:54:57.827 DEBUG:boto:Port: 8000 2023-05-31T20:54:57.827 DEBUG:boto:Params: {} 2023-05-31T20:54:57.828 DEBUG:boto:Token: None 2023-05-31T20:54:57.828 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:54:57 GMT /pklxkd-59/obj29 2023-05-31T20:54:57.828 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:XRq77g+H0D8f4HCtFEmbWFXpmwc= 2023-05-31T20:54:57.829 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:54:57 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:XRq77g+H0D8f4HCtFEmbWFXpmwc=', 'Content-Length': '0'} 2023-05-31T20:54:57.831 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:48:12 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000063c3fbf72097e8f9-006477b421-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:54:57 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:54:57.872 DEBUG:boto:path=/pklxkd-59/ 2023-05-31T20:54:57.873 DEBUG:boto:auth_path=/pklxkd-59/ 2023-05-31T20:54:57.873 DEBUG:boto:path=/pklxkd-59/?prefix=obj29 2023-05-31T20:54:57.873 DEBUG:boto:auth_path=/pklxkd-59/?prefix=obj29 2023-05-31T20:54:57.874 DEBUG:boto:Method: GET 2023-05-31T20:54:57.874 DEBUG:boto:Path: /pklxkd-59/?prefix=obj29 2023-05-31T20:54:57.874 DEBUG:boto:Data: 2023-05-31T20:54:57.875 DEBUG:boto:Headers: {} 2023-05-31T20:54:57.875 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:54:57.876 DEBUG:boto:Port: 8001 2023-05-31T20:54:57.876 DEBUG:boto:Params: {} 2023-05-31T20:54:57.876 DEBUG:boto:Token: None 2023-05-31T20:54:57.877 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:54:57 GMT /pklxkd-59/ 2023-05-31T20:54:57.877 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:aisxEfj8evvfMLEyPoSaposCVmU= 2023-05-31T20:54:57.878 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:54:57 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:aisxEfj8evvfMLEyPoSaposCVmU=', 'Content-Length': '0'} 2023-05-31T20:54:57.880 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b3d9a26a5b0837f4-006477b421-1e2e-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:54:57 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:54:57.881 DEBUG:boto:b'pklxkd-59obj291000falseobj292023-05-31T20:48:12.956Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:54:57.881 DEBUG:boto:path=/pklxkd-59/ 2023-05-31T20:54:57.882 DEBUG:boto:auth_path=/pklxkd-59/ 2023-05-31T20:54:57.882 DEBUG:boto:path=/pklxkd-59/?prefix=obj29 2023-05-31T20:54:57.882 DEBUG:boto:auth_path=/pklxkd-59/?prefix=obj29 2023-05-31T20:54:57.883 DEBUG:boto:Method: GET 2023-05-31T20:54:57.883 DEBUG:boto:Path: /pklxkd-59/?prefix=obj29 2023-05-31T20:54:57.883 DEBUG:boto:Data: 2023-05-31T20:54:57.884 DEBUG:boto:Headers: {} 2023-05-31T20:54:57.884 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:54:57.884 DEBUG:boto:Port: 8000 2023-05-31T20:54:57.885 DEBUG:boto:Params: {} 2023-05-31T20:54:57.885 DEBUG:boto:Token: None 2023-05-31T20:54:57.886 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:54:57 GMT /pklxkd-59/ 2023-05-31T20:54:57.886 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:aisxEfj8evvfMLEyPoSaposCVmU= 2023-05-31T20:54:57.886 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:54:57 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:aisxEfj8evvfMLEyPoSaposCVmU=', 'Content-Length': '0'} 2023-05-31T20:54:57.891 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e2530064e53544d5-006477b421-1630-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:54:57 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:54:57.891 DEBUG:boto:b'pklxkd-59obj291000falseobj292023-05-31T20:48:12.956Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:54:57.892 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj31 2023-05-31T20:54:57.892 DEBUG:boto:path=/pklxkd-59/obj31 2023-05-31T20:54:57.892 DEBUG:boto:auth_path=/pklxkd-59/obj31 2023-05-31T20:54:57.893 DEBUG:boto:path=/pklxkd-59/obj31?versionId=null 2023-05-31T20:54:57.893 DEBUG:boto:auth_path=/pklxkd-59/obj31?versionId=null 2023-05-31T20:54:57.894 DEBUG:boto:Method: GET 2023-05-31T20:54:57.894 DEBUG:boto:Path: /pklxkd-59/obj31?versionId=null 2023-05-31T20:54:57.894 DEBUG:boto:Data: 2023-05-31T20:54:57.895 DEBUG:boto:Headers: {} 2023-05-31T20:54:57.895 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:54:57.895 DEBUG:boto:Port: 8001 2023-05-31T20:54:57.896 DEBUG:boto:Params: {} 2023-05-31T20:54:57.896 DEBUG:boto:Token: None 2023-05-31T20:54:57.896 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:54:57 GMT /pklxkd-59/obj31?versionId=null 2023-05-31T20:54:57.897 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:Llq2rUvdEiQC5EopAyFd1hqRp5Y= 2023-05-31T20:54:57.897 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:54:57 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:Llq2rUvdEiQC5EopAyFd1hqRp5Y=', 'Content-Length': '0'} 2023-05-31T20:54:57.899 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:50:33 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f0cbaefbd56f51f9-006477b421-1e2e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:54:57 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:54:57.900 DEBUG:boto:path=/pklxkd-59/obj31 2023-05-31T20:54:57.900 DEBUG:boto:auth_path=/pklxkd-59/obj31 2023-05-31T20:54:57.901 DEBUG:boto:path=/pklxkd-59/obj31?versionId=null 2023-05-31T20:54:57.901 DEBUG:boto:auth_path=/pklxkd-59/obj31?versionId=null 2023-05-31T20:54:57.901 DEBUG:boto:Method: GET 2023-05-31T20:54:57.902 DEBUG:boto:Path: /pklxkd-59/obj31?versionId=null 2023-05-31T20:54:57.902 DEBUG:boto:Data: 2023-05-31T20:54:57.902 DEBUG:boto:Headers: {} 2023-05-31T20:54:57.903 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:54:57.903 DEBUG:boto:Port: 8000 2023-05-31T20:54:57.903 DEBUG:boto:Params: {} 2023-05-31T20:54:57.904 DEBUG:boto:Token: None 2023-05-31T20:54:57.904 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:54:57 GMT /pklxkd-59/obj31?versionId=null 2023-05-31T20:54:57.905 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:Llq2rUvdEiQC5EopAyFd1hqRp5Y= 2023-05-31T20:54:57.905 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:54:57 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:Llq2rUvdEiQC5EopAyFd1hqRp5Y=', 'Content-Length': '0'} 2023-05-31T20:54:57.907 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:50:33 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000091f10fa48ab2144a-006477b421-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:54:57 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:54:57.907 DEBUG:boto:path=/pklxkd-59/obj31 2023-05-31T20:54:57.908 DEBUG:boto:auth_path=/pklxkd-59/obj31 2023-05-31T20:54:57.908 DEBUG:boto:path=/pklxkd-59/obj31?versionId=null 2023-05-31T20:54:57.909 DEBUG:boto:auth_path=/pklxkd-59/obj31?versionId=null 2023-05-31T20:54:57.909 DEBUG:boto:Method: HEAD 2023-05-31T20:54:57.909 DEBUG:boto:Path: /pklxkd-59/obj31?versionId=null 2023-05-31T20:54:57.910 DEBUG:boto:Data: 2023-05-31T20:54:57.910 DEBUG:boto:Headers: {} 2023-05-31T20:54:57.910 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:54:57.911 DEBUG:boto:Port: 8001 2023-05-31T20:54:57.911 DEBUG:boto:Params: {} 2023-05-31T20:54:57.911 DEBUG:boto:Token: None 2023-05-31T20:54:57.912 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:54:57 GMT /pklxkd-59/obj31?versionId=null 2023-05-31T20:54:57.912 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:3MRhFNN3dmSGLtQQ4mG7oLY5JLg= 2023-05-31T20:54:57.913 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:54:57 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:3MRhFNN3dmSGLtQQ4mG7oLY5JLg=', 'Content-Length': '0'} 2023-05-31T20:54:57.915 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:50:33 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000079ebe66c751be62a-006477b421-1e2e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:54:57 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:54:57.915 DEBUG:boto:path=/pklxkd-59/obj31 2023-05-31T20:54:57.915 DEBUG:boto:auth_path=/pklxkd-59/obj31 2023-05-31T20:54:57.916 DEBUG:boto:path=/pklxkd-59/obj31?versionId=null 2023-05-31T20:54:57.916 DEBUG:boto:auth_path=/pklxkd-59/obj31?versionId=null 2023-05-31T20:54:57.917 DEBUG:boto:Method: HEAD 2023-05-31T20:54:57.917 DEBUG:boto:Path: /pklxkd-59/obj31?versionId=null 2023-05-31T20:54:57.917 DEBUG:boto:Data: 2023-05-31T20:54:57.918 DEBUG:boto:Headers: {} 2023-05-31T20:54:57.918 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:54:57.919 DEBUG:boto:Port: 8000 2023-05-31T20:54:57.919 DEBUG:boto:Params: {} 2023-05-31T20:54:57.919 DEBUG:boto:Token: None 2023-05-31T20:54:57.920 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:54:57 GMT /pklxkd-59/obj31?versionId=null 2023-05-31T20:54:57.920 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:3MRhFNN3dmSGLtQQ4mG7oLY5JLg= 2023-05-31T20:54:57.921 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:54:57 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:3MRhFNN3dmSGLtQQ4mG7oLY5JLg=', 'Content-Length': '0'} 2023-05-31T20:54:57.922 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:50:33 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000023c849c155a3bcc9-006477b421-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:54:57 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:54:57.923 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj31 2023-05-31T20:54:57.923 DEBUG:boto:path=/pklxkd-59/obj31 2023-05-31T20:54:57.924 DEBUG:boto:auth_path=/pklxkd-59/obj31 2023-05-31T20:54:57.924 DEBUG:boto:path=/pklxkd-59/obj31?versionId=null 2023-05-31T20:54:57.925 DEBUG:boto:auth_path=/pklxkd-59/obj31?versionId=null 2023-05-31T20:54:57.925 DEBUG:boto:Method: GET 2023-05-31T20:54:57.925 DEBUG:boto:Path: /pklxkd-59/obj31?versionId=null 2023-05-31T20:54:57.926 DEBUG:boto:Data: 2023-05-31T20:54:57.926 DEBUG:boto:Headers: {} 2023-05-31T20:54:57.926 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:54:57.927 DEBUG:boto:Port: 8001 2023-05-31T20:54:57.927 DEBUG:boto:Params: {} 2023-05-31T20:54:57.927 DEBUG:boto:Token: None 2023-05-31T20:54:57.928 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:54:57 GMT /pklxkd-59/obj31?versionId=null 2023-05-31T20:54:57.928 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:Llq2rUvdEiQC5EopAyFd1hqRp5Y= 2023-05-31T20:54:57.929 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:54:57 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:Llq2rUvdEiQC5EopAyFd1hqRp5Y=', 'Content-Length': '0'} 2023-05-31T20:54:57.930 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:50:33 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ad9d3d7d25ed7c67-006477b421-1e2e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:54:57 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:54:57.931 DEBUG:boto:path=/pklxkd-59/obj31 2023-05-31T20:54:57.931 DEBUG:boto:auth_path=/pklxkd-59/obj31 2023-05-31T20:54:57.932 DEBUG:boto:path=/pklxkd-59/obj31?versionId=null 2023-05-31T20:54:57.932 DEBUG:boto:auth_path=/pklxkd-59/obj31?versionId=null 2023-05-31T20:54:57.933 DEBUG:boto:Method: GET 2023-05-31T20:54:57.933 DEBUG:boto:Path: /pklxkd-59/obj31?versionId=null 2023-05-31T20:54:57.933 DEBUG:boto:Data: 2023-05-31T20:54:57.934 DEBUG:boto:Headers: {} 2023-05-31T20:54:57.934 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:54:57.934 DEBUG:boto:Port: 8000 2023-05-31T20:54:57.935 DEBUG:boto:Params: {} 2023-05-31T20:54:57.935 DEBUG:boto:Token: None 2023-05-31T20:54:57.935 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:54:57 GMT /pklxkd-59/obj31?versionId=null 2023-05-31T20:54:57.936 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:Llq2rUvdEiQC5EopAyFd1hqRp5Y= 2023-05-31T20:54:57.936 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:54:57 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:Llq2rUvdEiQC5EopAyFd1hqRp5Y=', 'Content-Length': '0'} 2023-05-31T20:54:57.938 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:50:33 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000081858308c2a78a5a-006477b421-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:54:57 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:54:57.939 DEBUG:boto:path=/pklxkd-59/ 2023-05-31T20:54:57.939 DEBUG:boto:auth_path=/pklxkd-59/ 2023-05-31T20:54:57.940 DEBUG:boto:path=/pklxkd-59/?prefix=obj31 2023-05-31T20:54:57.940 DEBUG:boto:auth_path=/pklxkd-59/?prefix=obj31 2023-05-31T20:54:57.941 DEBUG:boto:Method: GET 2023-05-31T20:54:57.941 DEBUG:boto:Path: /pklxkd-59/?prefix=obj31 2023-05-31T20:54:57.941 DEBUG:boto:Data: 2023-05-31T20:54:57.942 DEBUG:boto:Headers: {} 2023-05-31T20:54:57.942 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:54:57.942 DEBUG:boto:Port: 8001 2023-05-31T20:54:57.943 DEBUG:boto:Params: {} 2023-05-31T20:54:57.943 DEBUG:boto:Token: None 2023-05-31T20:54:57.944 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:54:57 GMT /pklxkd-59/ 2023-05-31T20:54:57.944 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:aisxEfj8evvfMLEyPoSaposCVmU= 2023-05-31T20:54:57.944 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:54:57 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:aisxEfj8evvfMLEyPoSaposCVmU=', 'Content-Length': '0'} 2023-05-31T20:54:57.947 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000006fe606fd0e133727-006477b421-1e2e-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:54:57 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:54:57.947 DEBUG:boto:b'pklxkd-59obj311000falseobj312023-05-31T20:50:33.460Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:54:57.948 DEBUG:boto:path=/pklxkd-59/ 2023-05-31T20:54:57.948 DEBUG:boto:auth_path=/pklxkd-59/ 2023-05-31T20:54:57.949 DEBUG:boto:path=/pklxkd-59/?prefix=obj31 2023-05-31T20:54:57.949 DEBUG:boto:auth_path=/pklxkd-59/?prefix=obj31 2023-05-31T20:54:57.949 DEBUG:boto:Method: GET 2023-05-31T20:54:57.950 DEBUG:boto:Path: /pklxkd-59/?prefix=obj31 2023-05-31T20:54:57.950 DEBUG:boto:Data: 2023-05-31T20:54:57.951 DEBUG:boto:Headers: {} 2023-05-31T20:54:57.951 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:54:57.951 DEBUG:boto:Port: 8000 2023-05-31T20:54:57.952 DEBUG:boto:Params: {} 2023-05-31T20:54:57.952 DEBUG:boto:Token: None 2023-05-31T20:54:57.952 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:54:57 GMT /pklxkd-59/ 2023-05-31T20:54:57.953 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:aisxEfj8evvfMLEyPoSaposCVmU= 2023-05-31T20:54:57.953 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:54:57 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:aisxEfj8evvfMLEyPoSaposCVmU=', 'Content-Length': '0'} 2023-05-31T20:54:57.957 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000054b63b213cf099ac-006477b421-1630-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:54:57 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:54:57.958 DEBUG:boto:b'pklxkd-59obj311000falseobj312023-05-31T20:50:33.460Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:54:57.958 DEBUG:boto:path=/pklxkd-59/obj31 2023-05-31T20:54:57.959 DEBUG:boto:auth_path=/pklxkd-59/obj31 2023-05-31T20:54:57.959 DEBUG:boto:Method: HEAD 2023-05-31T20:54:57.959 DEBUG:boto:Path: /pklxkd-59/obj31 2023-05-31T20:54:57.960 DEBUG:boto:Data: 2023-05-31T20:54:57.960 DEBUG:boto:Headers: {} 2023-05-31T20:54:57.961 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:54:57.961 DEBUG:boto:Port: 8001 2023-05-31T20:54:57.961 DEBUG:boto:Params: {} 2023-05-31T20:54:57.962 DEBUG:boto:Token: None 2023-05-31T20:54:57.962 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:54:57 GMT /pklxkd-59/obj31 2023-05-31T20:54:57.962 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:0lYF51Uh6tLyYknmFIvX5Rc6SsY= 2023-05-31T20:54:57.963 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:54:57 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:0lYF51Uh6tLyYknmFIvX5Rc6SsY=', 'Content-Length': '0'} 2023-05-31T20:54:57.965 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:50:33 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000011d55c4855e61a66-006477b421-1e2e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:54:57 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:54:57.965 DEBUG:boto:path=/pklxkd-59/obj31 2023-05-31T20:54:57.966 DEBUG:boto:auth_path=/pklxkd-59/obj31 2023-05-31T20:54:57.966 DEBUG:boto:Method: HEAD 2023-05-31T20:54:57.966 DEBUG:boto:Path: /pklxkd-59/obj31 2023-05-31T20:54:57.967 DEBUG:boto:Data: 2023-05-31T20:54:57.967 DEBUG:boto:Headers: {} 2023-05-31T20:54:57.967 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:54:57.968 DEBUG:boto:Port: 8000 2023-05-31T20:54:57.968 DEBUG:boto:Params: {} 2023-05-31T20:54:57.968 DEBUG:boto:Token: None 2023-05-31T20:54:57.969 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:54:57 GMT /pklxkd-59/obj31 2023-05-31T20:54:57.969 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:0lYF51Uh6tLyYknmFIvX5Rc6SsY= 2023-05-31T20:54:57.970 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:54:57 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:0lYF51Uh6tLyYknmFIvX5Rc6SsY=', 'Content-Length': '0'} 2023-05-31T20:54:57.971 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:50:33 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000036b09f9ab0bb2f33-006477b421-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:54:57 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:54:57.972 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj31 2023-05-31T20:54:57.972 DEBUG:boto:path=/pklxkd-59/obj31 2023-05-31T20:54:57.973 DEBUG:boto:auth_path=/pklxkd-59/obj31 2023-05-31T20:54:57.973 DEBUG:boto:Method: GET 2023-05-31T20:54:57.974 DEBUG:boto:Path: /pklxkd-59/obj31 2023-05-31T20:54:57.974 DEBUG:boto:Data: 2023-05-31T20:54:57.974 DEBUG:boto:Headers: {} 2023-05-31T20:54:57.975 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:54:57.975 DEBUG:boto:Port: 8001 2023-05-31T20:54:57.975 DEBUG:boto:Params: {} 2023-05-31T20:54:57.976 DEBUG:boto:Token: None 2023-05-31T20:54:57.976 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:54:57 GMT /pklxkd-59/obj31 2023-05-31T20:54:57.977 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:gnFxkih1ak1pvMZ+BYbIZKa1WJs= 2023-05-31T20:54:57.977 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:54:57 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:gnFxkih1ak1pvMZ+BYbIZKa1WJs=', 'Content-Length': '0'} 2023-05-31T20:54:57.979 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:50:33 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001d74920f2a7086d8-006477b421-1e2e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:54:57 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:54:58.020 DEBUG:boto:path=/pklxkd-59/obj31 2023-05-31T20:54:58.021 DEBUG:boto:auth_path=/pklxkd-59/obj31 2023-05-31T20:54:58.021 DEBUG:boto:Method: GET 2023-05-31T20:54:58.022 DEBUG:boto:Path: /pklxkd-59/obj31 2023-05-31T20:54:58.022 DEBUG:boto:Data: 2023-05-31T20:54:58.022 DEBUG:boto:Headers: {} 2023-05-31T20:54:58.023 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:54:58.023 DEBUG:boto:Port: 8000 2023-05-31T20:54:58.023 DEBUG:boto:Params: {} 2023-05-31T20:54:58.024 DEBUG:boto:Token: None 2023-05-31T20:54:58.024 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:54:58 GMT /pklxkd-59/obj31 2023-05-31T20:54:58.025 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:V3t/EHulWlsijq4CwT3FTGuEKRk= 2023-05-31T20:54:58.025 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:54:58 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:V3t/EHulWlsijq4CwT3FTGuEKRk=', 'Content-Length': '0'} 2023-05-31T20:54:58.027 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:50:33 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000fc747f0b16b370ea-006477b422-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:54:58 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:54:58.027 DEBUG:boto:path=/pklxkd-59/ 2023-05-31T20:54:58.028 DEBUG:boto:auth_path=/pklxkd-59/ 2023-05-31T20:54:58.028 DEBUG:boto:path=/pklxkd-59/?prefix=obj31 2023-05-31T20:54:58.029 DEBUG:boto:auth_path=/pklxkd-59/?prefix=obj31 2023-05-31T20:54:58.029 DEBUG:boto:Method: GET 2023-05-31T20:54:58.029 DEBUG:boto:Path: /pklxkd-59/?prefix=obj31 2023-05-31T20:54:58.030 DEBUG:boto:Data: 2023-05-31T20:54:58.030 DEBUG:boto:Headers: {} 2023-05-31T20:54:58.030 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:54:58.031 DEBUG:boto:Port: 8001 2023-05-31T20:54:58.031 DEBUG:boto:Params: {} 2023-05-31T20:54:58.031 DEBUG:boto:Token: None 2023-05-31T20:54:58.032 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:54:58 GMT /pklxkd-59/ 2023-05-31T20:54:58.032 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:B6nEAA7PON5axhJ7EDo9Lg1oe5U= 2023-05-31T20:54:58.033 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:54:58 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:B6nEAA7PON5axhJ7EDo9Lg1oe5U=', 'Content-Length': '0'} 2023-05-31T20:54:58.035 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000eea6a62f4339dd83-006477b422-1e2e-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:54:58 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:54:58.036 DEBUG:boto:b'pklxkd-59obj311000falseobj312023-05-31T20:50:33.460Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:54:58.037 DEBUG:boto:path=/pklxkd-59/ 2023-05-31T20:54:58.037 DEBUG:boto:auth_path=/pklxkd-59/ 2023-05-31T20:54:58.037 DEBUG:boto:path=/pklxkd-59/?prefix=obj31 2023-05-31T20:54:58.038 DEBUG:boto:auth_path=/pklxkd-59/?prefix=obj31 2023-05-31T20:54:58.038 DEBUG:boto:Method: GET 2023-05-31T20:54:58.038 DEBUG:boto:Path: /pklxkd-59/?prefix=obj31 2023-05-31T20:54:58.039 DEBUG:boto:Data: 2023-05-31T20:54:58.039 DEBUG:boto:Headers: {} 2023-05-31T20:54:58.039 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:54:58.040 DEBUG:boto:Port: 8000 2023-05-31T20:54:58.040 DEBUG:boto:Params: {} 2023-05-31T20:54:58.041 DEBUG:boto:Token: None 2023-05-31T20:54:58.041 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:54:58 GMT /pklxkd-59/ 2023-05-31T20:54:58.041 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:B6nEAA7PON5axhJ7EDo9Lg1oe5U= 2023-05-31T20:54:58.042 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:54:58 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:B6nEAA7PON5axhJ7EDo9Lg1oe5U=', 'Content-Length': '0'} 2023-05-31T20:54:58.046 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000075aba0055680d308-006477b422-1630-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:54:58 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:54:58.046 DEBUG:boto:b'pklxkd-59obj311000falseobj312023-05-31T20:50:33.460Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:54:58.047 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj33 2023-05-31T20:54:58.047 DEBUG:boto:path=/pklxkd-59/obj33 2023-05-31T20:54:58.048 DEBUG:boto:auth_path=/pklxkd-59/obj33 2023-05-31T20:54:58.048 DEBUG:boto:path=/pklxkd-59/obj33?versionId=null 2023-05-31T20:54:58.048 DEBUG:boto:auth_path=/pklxkd-59/obj33?versionId=null 2023-05-31T20:54:58.049 DEBUG:boto:Method: GET 2023-05-31T20:54:58.049 DEBUG:boto:Path: /pklxkd-59/obj33?versionId=null 2023-05-31T20:54:58.049 DEBUG:boto:Data: 2023-05-31T20:54:58.050 DEBUG:boto:Headers: {} 2023-05-31T20:54:58.050 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:54:58.050 DEBUG:boto:Port: 8001 2023-05-31T20:54:58.051 DEBUG:boto:Params: {} 2023-05-31T20:54:58.051 DEBUG:boto:Token: None 2023-05-31T20:54:58.052 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:54:58 GMT /pklxkd-59/obj33?versionId=null 2023-05-31T20:54:58.052 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:TJoGVe8G/x7D1bq3GhXOVoe0P2E= 2023-05-31T20:54:58.052 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:54:58 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:TJoGVe8G/x7D1bq3GhXOVoe0P2E=', 'Content-Length': '0'} 2023-05-31T20:54:58.055 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:39:26 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000135b2c053fe1df98-006477b422-1e2e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:54:58 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:54:58.055 DEBUG:boto:path=/pklxkd-59/obj33 2023-05-31T20:54:58.056 DEBUG:boto:auth_path=/pklxkd-59/obj33 2023-05-31T20:54:58.056 DEBUG:boto:path=/pklxkd-59/obj33?versionId=null 2023-05-31T20:54:58.056 DEBUG:boto:auth_path=/pklxkd-59/obj33?versionId=null 2023-05-31T20:54:58.057 DEBUG:boto:Method: GET 2023-05-31T20:54:58.057 DEBUG:boto:Path: /pklxkd-59/obj33?versionId=null 2023-05-31T20:54:58.057 DEBUG:boto:Data: 2023-05-31T20:54:58.058 DEBUG:boto:Headers: {} 2023-05-31T20:54:58.058 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:54:58.058 DEBUG:boto:Port: 8000 2023-05-31T20:54:58.059 DEBUG:boto:Params: {} 2023-05-31T20:54:58.059 DEBUG:boto:Token: None 2023-05-31T20:54:58.060 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:54:58 GMT /pklxkd-59/obj33?versionId=null 2023-05-31T20:54:58.060 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:TJoGVe8G/x7D1bq3GhXOVoe0P2E= 2023-05-31T20:54:58.061 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:54:58 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:TJoGVe8G/x7D1bq3GhXOVoe0P2E=', 'Content-Length': '0'} 2023-05-31T20:54:58.062 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:39:26 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000780129dcc8c3f438-006477b422-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:54:58 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:54:58.063 DEBUG:boto:path=/pklxkd-59/obj33 2023-05-31T20:54:58.063 DEBUG:boto:auth_path=/pklxkd-59/obj33 2023-05-31T20:54:58.064 DEBUG:boto:path=/pklxkd-59/obj33?versionId=null 2023-05-31T20:54:58.064 DEBUG:boto:auth_path=/pklxkd-59/obj33?versionId=null 2023-05-31T20:54:58.064 DEBUG:boto:Method: HEAD 2023-05-31T20:54:58.065 DEBUG:boto:Path: /pklxkd-59/obj33?versionId=null 2023-05-31T20:54:58.065 DEBUG:boto:Data: 2023-05-31T20:54:58.065 DEBUG:boto:Headers: {} 2023-05-31T20:54:58.066 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:54:58.066 DEBUG:boto:Port: 8001 2023-05-31T20:54:58.067 DEBUG:boto:Params: {} 2023-05-31T20:54:58.067 DEBUG:boto:Token: None 2023-05-31T20:54:58.067 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:54:58 GMT /pklxkd-59/obj33?versionId=null 2023-05-31T20:54:58.068 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:K7BBHbexSf79sVjwCKxfgZZC/p0= 2023-05-31T20:54:58.068 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:54:58 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:K7BBHbexSf79sVjwCKxfgZZC/p0=', 'Content-Length': '0'} 2023-05-31T20:54:58.070 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:39:26 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001b50fa9b55a6611e-006477b422-1e2e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:54:58 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:54:58.070 DEBUG:boto:path=/pklxkd-59/obj33 2023-05-31T20:54:58.071 DEBUG:boto:auth_path=/pklxkd-59/obj33 2023-05-31T20:54:58.071 DEBUG:boto:path=/pklxkd-59/obj33?versionId=null 2023-05-31T20:54:58.071 DEBUG:boto:auth_path=/pklxkd-59/obj33?versionId=null 2023-05-31T20:54:58.072 DEBUG:boto:Method: HEAD 2023-05-31T20:54:58.072 DEBUG:boto:Path: /pklxkd-59/obj33?versionId=null 2023-05-31T20:54:58.072 DEBUG:boto:Data: 2023-05-31T20:54:58.073 DEBUG:boto:Headers: {} 2023-05-31T20:54:58.073 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:54:58.073 DEBUG:boto:Port: 8000 2023-05-31T20:54:58.074 DEBUG:boto:Params: {} 2023-05-31T20:54:58.074 DEBUG:boto:Token: None 2023-05-31T20:54:58.074 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:54:58 GMT /pklxkd-59/obj33?versionId=null 2023-05-31T20:54:58.075 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:K7BBHbexSf79sVjwCKxfgZZC/p0= 2023-05-31T20:54:58.075 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:54:58 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:K7BBHbexSf79sVjwCKxfgZZC/p0=', 'Content-Length': '0'} 2023-05-31T20:54:58.077 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:39:26 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000cf831620903a6c3c-006477b422-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:54:58 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:54:58.078 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj33 2023-05-31T20:54:58.078 DEBUG:boto:path=/pklxkd-59/obj33 2023-05-31T20:54:58.079 DEBUG:boto:auth_path=/pklxkd-59/obj33 2023-05-31T20:54:58.079 DEBUG:boto:path=/pklxkd-59/obj33?versionId=null 2023-05-31T20:54:58.079 DEBUG:boto:auth_path=/pklxkd-59/obj33?versionId=null 2023-05-31T20:54:58.080 DEBUG:boto:Method: GET 2023-05-31T20:54:58.080 DEBUG:boto:Path: /pklxkd-59/obj33?versionId=null 2023-05-31T20:54:58.081 DEBUG:boto:Data: 2023-05-31T20:54:58.081 DEBUG:boto:Headers: {} 2023-05-31T20:54:58.081 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:54:58.082 DEBUG:boto:Port: 8001 2023-05-31T20:54:58.082 DEBUG:boto:Params: {} 2023-05-31T20:54:58.082 DEBUG:boto:Token: None 2023-05-31T20:54:58.083 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:54:58 GMT /pklxkd-59/obj33?versionId=null 2023-05-31T20:54:58.083 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:TJoGVe8G/x7D1bq3GhXOVoe0P2E= 2023-05-31T20:54:58.084 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:54:58 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:TJoGVe8G/x7D1bq3GhXOVoe0P2E=', 'Content-Length': '0'} 2023-05-31T20:54:58.085 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:39:26 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000079c38e6cc46ab7a6-006477b422-1e2e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:54:58 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:54:58.086 DEBUG:boto:path=/pklxkd-59/obj33 2023-05-31T20:54:58.087 DEBUG:boto:auth_path=/pklxkd-59/obj33 2023-05-31T20:54:58.087 DEBUG:boto:path=/pklxkd-59/obj33?versionId=null 2023-05-31T20:54:58.087 DEBUG:boto:auth_path=/pklxkd-59/obj33?versionId=null 2023-05-31T20:54:58.088 DEBUG:boto:Method: GET 2023-05-31T20:54:58.088 DEBUG:boto:Path: /pklxkd-59/obj33?versionId=null 2023-05-31T20:54:58.088 DEBUG:boto:Data: 2023-05-31T20:54:58.089 DEBUG:boto:Headers: {} 2023-05-31T20:54:58.089 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:54:58.090 DEBUG:boto:Port: 8000 2023-05-31T20:54:58.090 DEBUG:boto:Params: {} 2023-05-31T20:54:58.090 DEBUG:boto:Token: None 2023-05-31T20:54:58.091 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:54:58 GMT /pklxkd-59/obj33?versionId=null 2023-05-31T20:54:58.091 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:TJoGVe8G/x7D1bq3GhXOVoe0P2E= 2023-05-31T20:54:58.092 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:54:58 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:TJoGVe8G/x7D1bq3GhXOVoe0P2E=', 'Content-Length': '0'} 2023-05-31T20:54:58.094 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:39:26 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000890e2d775f5fa32b-006477b422-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:54:58 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:54:58.094 DEBUG:boto:path=/pklxkd-59/ 2023-05-31T20:54:58.095 DEBUG:boto:auth_path=/pklxkd-59/ 2023-05-31T20:54:58.095 DEBUG:boto:path=/pklxkd-59/?prefix=obj33 2023-05-31T20:54:58.095 DEBUG:boto:auth_path=/pklxkd-59/?prefix=obj33 2023-05-31T20:54:58.096 DEBUG:boto:Method: GET 2023-05-31T20:54:58.096 DEBUG:boto:Path: /pklxkd-59/?prefix=obj33 2023-05-31T20:54:58.096 DEBUG:boto:Data: 2023-05-31T20:54:58.097 DEBUG:boto:Headers: {} 2023-05-31T20:54:58.097 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:54:58.097 DEBUG:boto:Port: 8001 2023-05-31T20:54:58.098 DEBUG:boto:Params: {} 2023-05-31T20:54:58.098 DEBUG:boto:Token: None 2023-05-31T20:54:58.099 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:54:58 GMT /pklxkd-59/ 2023-05-31T20:54:58.099 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:B6nEAA7PON5axhJ7EDo9Lg1oe5U= 2023-05-31T20:54:58.099 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:54:58 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:B6nEAA7PON5axhJ7EDo9Lg1oe5U=', 'Content-Length': '0'} 2023-05-31T20:54:58.102 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c25a009fbc592472-006477b422-1e2e-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:54:58 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:54:58.102 DEBUG:boto:b'pklxkd-59obj331000falseobj332023-05-31T20:39:26.534Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:54:58.103 DEBUG:boto:path=/pklxkd-59/ 2023-05-31T20:54:58.103 DEBUG:boto:auth_path=/pklxkd-59/ 2023-05-31T20:54:58.104 DEBUG:boto:path=/pklxkd-59/?prefix=obj33 2023-05-31T20:54:58.104 DEBUG:boto:auth_path=/pklxkd-59/?prefix=obj33 2023-05-31T20:54:58.104 DEBUG:boto:Method: GET 2023-05-31T20:54:58.105 DEBUG:boto:Path: /pklxkd-59/?prefix=obj33 2023-05-31T20:54:58.105 DEBUG:boto:Data: 2023-05-31T20:54:58.105 DEBUG:boto:Headers: {} 2023-05-31T20:54:58.106 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:54:58.106 DEBUG:boto:Port: 8000 2023-05-31T20:54:58.106 DEBUG:boto:Params: {} 2023-05-31T20:54:58.107 DEBUG:boto:Token: None 2023-05-31T20:54:58.107 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:54:58 GMT /pklxkd-59/ 2023-05-31T20:54:58.108 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:B6nEAA7PON5axhJ7EDo9Lg1oe5U= 2023-05-31T20:54:58.108 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:54:58 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:B6nEAA7PON5axhJ7EDo9Lg1oe5U=', 'Content-Length': '0'} 2023-05-31T20:54:58.112 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000edaa4066522ed70c-006477b422-1630-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:54:58 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:54:58.112 DEBUG:boto:b'pklxkd-59obj331000falseobj332023-05-31T20:39:26.534Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:54:58.113 DEBUG:boto:path=/pklxkd-59/obj33 2023-05-31T20:54:58.113 DEBUG:boto:auth_path=/pklxkd-59/obj33 2023-05-31T20:54:58.114 DEBUG:boto:Method: HEAD 2023-05-31T20:54:58.114 DEBUG:boto:Path: /pklxkd-59/obj33 2023-05-31T20:54:58.114 DEBUG:boto:Data: 2023-05-31T20:54:58.115 DEBUG:boto:Headers: {} 2023-05-31T20:54:58.115 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:54:58.115 DEBUG:boto:Port: 8001 2023-05-31T20:54:58.116 DEBUG:boto:Params: {} 2023-05-31T20:54:58.116 DEBUG:boto:Token: None 2023-05-31T20:54:58.117 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:54:58 GMT /pklxkd-59/obj33 2023-05-31T20:54:58.117 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:qGmuWMt9dtEGAy/a7lHCWG9GTW8= 2023-05-31T20:54:58.117 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:54:58 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:qGmuWMt9dtEGAy/a7lHCWG9GTW8=', 'Content-Length': '0'} 2023-05-31T20:54:58.119 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:39:26 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000cbce8aed1aced10c-006477b422-1e2e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:54:58 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:54:58.120 DEBUG:boto:path=/pklxkd-59/obj33 2023-05-31T20:54:58.120 DEBUG:boto:auth_path=/pklxkd-59/obj33 2023-05-31T20:54:58.121 DEBUG:boto:Method: HEAD 2023-05-31T20:54:58.121 DEBUG:boto:Path: /pklxkd-59/obj33 2023-05-31T20:54:58.121 DEBUG:boto:Data: 2023-05-31T20:54:58.122 DEBUG:boto:Headers: {} 2023-05-31T20:54:58.122 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:54:58.122 DEBUG:boto:Port: 8000 2023-05-31T20:54:58.123 DEBUG:boto:Params: {} 2023-05-31T20:54:58.123 DEBUG:boto:Token: None 2023-05-31T20:54:58.123 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:54:58 GMT /pklxkd-59/obj33 2023-05-31T20:54:58.124 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:qGmuWMt9dtEGAy/a7lHCWG9GTW8= 2023-05-31T20:54:58.124 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:54:58 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:qGmuWMt9dtEGAy/a7lHCWG9GTW8=', 'Content-Length': '0'} 2023-05-31T20:54:58.126 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:39:26 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c38b1074afa1b83d-006477b422-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:54:58 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:54:58.126 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj33 2023-05-31T20:54:58.127 DEBUG:boto:path=/pklxkd-59/obj33 2023-05-31T20:54:58.127 DEBUG:boto:auth_path=/pklxkd-59/obj33 2023-05-31T20:54:58.127 DEBUG:boto:Method: GET 2023-05-31T20:54:58.128 DEBUG:boto:Path: /pklxkd-59/obj33 2023-05-31T20:54:58.128 DEBUG:boto:Data: 2023-05-31T20:54:58.128 DEBUG:boto:Headers: {} 2023-05-31T20:54:58.129 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:54:58.129 DEBUG:boto:Port: 8001 2023-05-31T20:54:58.129 DEBUG:boto:Params: {} 2023-05-31T20:54:58.130 DEBUG:boto:Token: None 2023-05-31T20:54:58.130 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:54:58 GMT /pklxkd-59/obj33 2023-05-31T20:54:58.131 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:6rpTEoALo8G2MlVaDmVSbKSchRo= 2023-05-31T20:54:58.131 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:54:58 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:6rpTEoALo8G2MlVaDmVSbKSchRo=', 'Content-Length': '0'} 2023-05-31T20:54:58.133 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:39:26 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002d678c069c967f3f-006477b422-1e2e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:54:58 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:54:58.134 DEBUG:boto:path=/pklxkd-59/obj33 2023-05-31T20:54:58.134 DEBUG:boto:auth_path=/pklxkd-59/obj33 2023-05-31T20:54:58.134 DEBUG:boto:Method: GET 2023-05-31T20:54:58.135 DEBUG:boto:Path: /pklxkd-59/obj33 2023-05-31T20:54:58.135 DEBUG:boto:Data: 2023-05-31T20:54:58.136 DEBUG:boto:Headers: {} 2023-05-31T20:54:58.136 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:54:58.136 DEBUG:boto:Port: 8000 2023-05-31T20:54:58.137 DEBUG:boto:Params: {} 2023-05-31T20:54:58.137 DEBUG:boto:Token: None 2023-05-31T20:54:58.138 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:54:58 GMT /pklxkd-59/obj33 2023-05-31T20:54:58.138 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:6rpTEoALo8G2MlVaDmVSbKSchRo= 2023-05-31T20:54:58.139 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:54:58 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:6rpTEoALo8G2MlVaDmVSbKSchRo=', 'Content-Length': '0'} 2023-05-31T20:54:58.141 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:39:26 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003c62af71744e8c8b-006477b422-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:54:58 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:54:58.141 DEBUG:boto:path=/pklxkd-59/ 2023-05-31T20:54:58.142 DEBUG:boto:auth_path=/pklxkd-59/ 2023-05-31T20:54:58.142 DEBUG:boto:path=/pklxkd-59/?prefix=obj33 2023-05-31T20:54:58.143 DEBUG:boto:auth_path=/pklxkd-59/?prefix=obj33 2023-05-31T20:54:58.143 DEBUG:boto:Method: GET 2023-05-31T20:54:58.143 DEBUG:boto:Path: /pklxkd-59/?prefix=obj33 2023-05-31T20:54:58.144 DEBUG:boto:Data: 2023-05-31T20:54:58.144 DEBUG:boto:Headers: {} 2023-05-31T20:54:58.145 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:54:58.145 DEBUG:boto:Port: 8001 2023-05-31T20:54:58.145 DEBUG:boto:Params: {} 2023-05-31T20:54:58.146 DEBUG:boto:Token: None 2023-05-31T20:54:58.146 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:54:58 GMT /pklxkd-59/ 2023-05-31T20:54:58.147 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:B6nEAA7PON5axhJ7EDo9Lg1oe5U= 2023-05-31T20:54:58.147 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:54:58 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:B6nEAA7PON5axhJ7EDo9Lg1oe5U=', 'Content-Length': '0'} 2023-05-31T20:54:58.149 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ac249abcfdc62f65-006477b422-1e2e-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:54:58 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:54:58.192 DEBUG:boto:b'pklxkd-59obj331000falseobj332023-05-31T20:39:26.534Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:54:58.193 DEBUG:boto:path=/pklxkd-59/ 2023-05-31T20:54:58.193 DEBUG:boto:auth_path=/pklxkd-59/ 2023-05-31T20:54:58.194 DEBUG:boto:path=/pklxkd-59/?prefix=obj33 2023-05-31T20:54:58.194 DEBUG:boto:auth_path=/pklxkd-59/?prefix=obj33 2023-05-31T20:54:58.195 DEBUG:boto:Method: GET 2023-05-31T20:54:58.195 DEBUG:boto:Path: /pklxkd-59/?prefix=obj33 2023-05-31T20:54:58.195 DEBUG:boto:Data: 2023-05-31T20:54:58.196 DEBUG:boto:Headers: {} 2023-05-31T20:54:58.196 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:54:58.197 DEBUG:boto:Port: 8000 2023-05-31T20:54:58.197 DEBUG:boto:Params: {} 2023-05-31T20:54:58.197 DEBUG:boto:Token: None 2023-05-31T20:54:58.198 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:54:58 GMT /pklxkd-59/ 2023-05-31T20:54:58.198 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:B6nEAA7PON5axhJ7EDo9Lg1oe5U= 2023-05-31T20:54:58.199 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:54:58 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:B6nEAA7PON5axhJ7EDo9Lg1oe5U=', 'Content-Length': '0'} 2023-05-31T20:54:58.203 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000073295f9e75e57daf-006477b422-1630-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:54:58 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:54:58.204 DEBUG:boto:b'pklxkd-59obj331000falseobj332023-05-31T20:39:26.534Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:54:58.204 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj34 2023-05-31T20:54:58.205 DEBUG:boto:path=/pklxkd-59/obj34 2023-05-31T20:54:58.205 DEBUG:boto:auth_path=/pklxkd-59/obj34 2023-05-31T20:54:58.206 DEBUG:boto:path=/pklxkd-59/obj34?versionId=null 2023-05-31T20:54:58.206 DEBUG:boto:auth_path=/pklxkd-59/obj34?versionId=null 2023-05-31T20:54:58.207 DEBUG:boto:Method: GET 2023-05-31T20:54:58.207 DEBUG:boto:Path: /pklxkd-59/obj34?versionId=null 2023-05-31T20:54:58.208 DEBUG:boto:Data: 2023-05-31T20:54:58.208 DEBUG:boto:Headers: {} 2023-05-31T20:54:58.209 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:54:58.210 DEBUG:boto:Port: 8001 2023-05-31T20:54:58.210 DEBUG:boto:Params: {} 2023-05-31T20:54:58.211 DEBUG:boto:Token: None 2023-05-31T20:54:58.211 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:54:58 GMT /pklxkd-59/obj34?versionId=null 2023-05-31T20:54:58.212 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:Qhai2VbvSzYZjTfy6GtgNT5zu40= 2023-05-31T20:54:58.212 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:54:58 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:Qhai2VbvSzYZjTfy6GtgNT5zu40=', 'Content-Length': '0'} 2023-05-31T20:54:58.215 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:43:07 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004633ffc5e6eb4e31-006477b422-1e2e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:54:58 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:54:58.216 DEBUG:boto:path=/pklxkd-59/obj34 2023-05-31T20:54:58.216 DEBUG:boto:auth_path=/pklxkd-59/obj34 2023-05-31T20:54:58.217 DEBUG:boto:path=/pklxkd-59/obj34?versionId=null 2023-05-31T20:54:58.217 DEBUG:boto:auth_path=/pklxkd-59/obj34?versionId=null 2023-05-31T20:54:58.218 DEBUG:boto:Method: GET 2023-05-31T20:54:58.218 DEBUG:boto:Path: /pklxkd-59/obj34?versionId=null 2023-05-31T20:54:58.219 DEBUG:boto:Data: 2023-05-31T20:54:58.219 DEBUG:boto:Headers: {} 2023-05-31T20:54:58.220 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:54:58.220 DEBUG:boto:Port: 8000 2023-05-31T20:54:58.221 DEBUG:boto:Params: {} 2023-05-31T20:54:58.221 DEBUG:boto:Token: None 2023-05-31T20:54:58.222 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:54:58 GMT /pklxkd-59/obj34?versionId=null 2023-05-31T20:54:58.222 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:Qhai2VbvSzYZjTfy6GtgNT5zu40= 2023-05-31T20:54:58.223 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:54:58 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:Qhai2VbvSzYZjTfy6GtgNT5zu40=', 'Content-Length': '0'} 2023-05-31T20:54:58.225 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:43:07 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001906c083b0778d7a-006477b422-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:54:58 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:54:58.268 DEBUG:boto:path=/pklxkd-59/obj34 2023-05-31T20:54:58.269 DEBUG:boto:auth_path=/pklxkd-59/obj34 2023-05-31T20:54:58.269 DEBUG:boto:path=/pklxkd-59/obj34?versionId=null 2023-05-31T20:54:58.269 DEBUG:boto:auth_path=/pklxkd-59/obj34?versionId=null 2023-05-31T20:54:58.270 DEBUG:boto:Method: HEAD 2023-05-31T20:54:58.270 DEBUG:boto:Path: /pklxkd-59/obj34?versionId=null 2023-05-31T20:54:58.271 DEBUG:boto:Data: 2023-05-31T20:54:58.271 DEBUG:boto:Headers: {} 2023-05-31T20:54:58.271 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:54:58.272 DEBUG:boto:Port: 8001 2023-05-31T20:54:58.272 DEBUG:boto:Params: {} 2023-05-31T20:54:58.273 DEBUG:boto:Token: None 2023-05-31T20:54:58.273 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:54:58 GMT /pklxkd-59/obj34?versionId=null 2023-05-31T20:54:58.274 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:mq22AE1MlqmvKTOyzyjM9Ed92U4= 2023-05-31T20:54:58.274 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:54:58 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:mq22AE1MlqmvKTOyzyjM9Ed92U4=', 'Content-Length': '0'} 2023-05-31T20:54:58.276 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:43:07 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004b83a02d2f04a80f-006477b422-1e2e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:54:58 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:54:58.276 DEBUG:boto:path=/pklxkd-59/obj34 2023-05-31T20:54:58.277 DEBUG:boto:auth_path=/pklxkd-59/obj34 2023-05-31T20:54:58.277 DEBUG:boto:path=/pklxkd-59/obj34?versionId=null 2023-05-31T20:54:58.278 DEBUG:boto:auth_path=/pklxkd-59/obj34?versionId=null 2023-05-31T20:54:58.278 DEBUG:boto:Method: HEAD 2023-05-31T20:54:58.278 DEBUG:boto:Path: /pklxkd-59/obj34?versionId=null 2023-05-31T20:54:58.279 DEBUG:boto:Data: 2023-05-31T20:54:58.279 DEBUG:boto:Headers: {} 2023-05-31T20:54:58.280 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:54:58.280 DEBUG:boto:Port: 8000 2023-05-31T20:54:58.280 DEBUG:boto:Params: {} 2023-05-31T20:54:58.281 DEBUG:boto:Token: None 2023-05-31T20:54:58.281 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:54:58 GMT /pklxkd-59/obj34?versionId=null 2023-05-31T20:54:58.282 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:mq22AE1MlqmvKTOyzyjM9Ed92U4= 2023-05-31T20:54:58.282 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:54:58 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:mq22AE1MlqmvKTOyzyjM9Ed92U4=', 'Content-Length': '0'} 2023-05-31T20:54:58.284 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:43:07 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f3575c890b9ea657-006477b422-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:54:58 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:54:58.285 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj34 2023-05-31T20:54:58.285 DEBUG:boto:path=/pklxkd-59/obj34 2023-05-31T20:54:58.286 DEBUG:boto:auth_path=/pklxkd-59/obj34 2023-05-31T20:54:58.286 DEBUG:boto:path=/pklxkd-59/obj34?versionId=null 2023-05-31T20:54:58.286 DEBUG:boto:auth_path=/pklxkd-59/obj34?versionId=null 2023-05-31T20:54:58.287 DEBUG:boto:Method: GET 2023-05-31T20:54:58.287 DEBUG:boto:Path: /pklxkd-59/obj34?versionId=null 2023-05-31T20:54:58.288 DEBUG:boto:Data: 2023-05-31T20:54:58.288 DEBUG:boto:Headers: {} 2023-05-31T20:54:58.289 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:54:58.289 DEBUG:boto:Port: 8001 2023-05-31T20:54:58.290 DEBUG:boto:Params: {} 2023-05-31T20:54:58.290 DEBUG:boto:Token: None 2023-05-31T20:54:58.290 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:54:58 GMT /pklxkd-59/obj34?versionId=null 2023-05-31T20:54:58.291 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:Qhai2VbvSzYZjTfy6GtgNT5zu40= 2023-05-31T20:54:58.291 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:54:58 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:Qhai2VbvSzYZjTfy6GtgNT5zu40=', 'Content-Length': '0'} 2023-05-31T20:54:58.293 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:43:07 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f7c58ea1371baf78-006477b422-1e2e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:54:58 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:54:58.294 DEBUG:boto:path=/pklxkd-59/obj34 2023-05-31T20:54:58.294 DEBUG:boto:auth_path=/pklxkd-59/obj34 2023-05-31T20:54:58.295 DEBUG:boto:path=/pklxkd-59/obj34?versionId=null 2023-05-31T20:54:58.295 DEBUG:boto:auth_path=/pklxkd-59/obj34?versionId=null 2023-05-31T20:54:58.295 DEBUG:boto:Method: GET 2023-05-31T20:54:58.296 DEBUG:boto:Path: /pklxkd-59/obj34?versionId=null 2023-05-31T20:54:58.296 DEBUG:boto:Data: 2023-05-31T20:54:58.297 DEBUG:boto:Headers: {} 2023-05-31T20:54:58.297 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:54:58.298 DEBUG:boto:Port: 8000 2023-05-31T20:54:58.298 DEBUG:boto:Params: {} 2023-05-31T20:54:58.298 DEBUG:boto:Token: None 2023-05-31T20:54:58.299 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:54:58 GMT /pklxkd-59/obj34?versionId=null 2023-05-31T20:54:58.299 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:Qhai2VbvSzYZjTfy6GtgNT5zu40= 2023-05-31T20:54:58.300 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:54:58 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:Qhai2VbvSzYZjTfy6GtgNT5zu40=', 'Content-Length': '0'} 2023-05-31T20:54:58.302 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:43:07 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007b146205820eec94-006477b422-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:54:58 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:54:58.302 DEBUG:boto:path=/pklxkd-59/ 2023-05-31T20:54:58.303 DEBUG:boto:auth_path=/pklxkd-59/ 2023-05-31T20:54:58.303 DEBUG:boto:path=/pklxkd-59/?prefix=obj34 2023-05-31T20:54:58.304 DEBUG:boto:auth_path=/pklxkd-59/?prefix=obj34 2023-05-31T20:54:58.304 DEBUG:boto:Method: GET 2023-05-31T20:54:58.305 DEBUG:boto:Path: /pklxkd-59/?prefix=obj34 2023-05-31T20:54:58.305 DEBUG:boto:Data: 2023-05-31T20:54:58.305 DEBUG:boto:Headers: {} 2023-05-31T20:54:58.306 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:54:58.306 DEBUG:boto:Port: 8001 2023-05-31T20:54:58.307 DEBUG:boto:Params: {} 2023-05-31T20:54:58.307 DEBUG:boto:Token: None 2023-05-31T20:54:58.308 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:54:58 GMT /pklxkd-59/ 2023-05-31T20:54:58.308 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:B6nEAA7PON5axhJ7EDo9Lg1oe5U= 2023-05-31T20:54:58.309 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:54:58 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:B6nEAA7PON5axhJ7EDo9Lg1oe5U=', 'Content-Length': '0'} 2023-05-31T20:54:58.311 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f733ccd376cd3f56-006477b422-1e2e-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:54:58 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:54:58.311 DEBUG:boto:b'pklxkd-59obj341000falseobj342023-05-31T20:43:07.994Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:54:58.312 DEBUG:boto:path=/pklxkd-59/ 2023-05-31T20:54:58.313 DEBUG:boto:auth_path=/pklxkd-59/ 2023-05-31T20:54:58.313 DEBUG:boto:path=/pklxkd-59/?prefix=obj34 2023-05-31T20:54:58.314 DEBUG:boto:auth_path=/pklxkd-59/?prefix=obj34 2023-05-31T20:54:58.314 DEBUG:boto:Method: GET 2023-05-31T20:54:58.315 DEBUG:boto:Path: /pklxkd-59/?prefix=obj34 2023-05-31T20:54:58.315 DEBUG:boto:Data: 2023-05-31T20:54:58.315 DEBUG:boto:Headers: {} 2023-05-31T20:54:58.316 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:54:58.316 DEBUG:boto:Port: 8000 2023-05-31T20:54:58.317 DEBUG:boto:Params: {} 2023-05-31T20:54:58.317 DEBUG:boto:Token: None 2023-05-31T20:54:58.318 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:54:58 GMT /pklxkd-59/ 2023-05-31T20:54:58.318 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:B6nEAA7PON5axhJ7EDo9Lg1oe5U= 2023-05-31T20:54:58.319 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:54:58 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:B6nEAA7PON5axhJ7EDo9Lg1oe5U=', 'Content-Length': '0'} 2023-05-31T20:54:58.323 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000007bbf353a1d841996-006477b422-1630-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:54:58 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:54:58.324 DEBUG:boto:b'pklxkd-59obj341000falseobj342023-05-31T20:43:07.994Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:54:58.324 DEBUG:boto:path=/pklxkd-59/obj34 2023-05-31T20:54:58.325 DEBUG:boto:auth_path=/pklxkd-59/obj34 2023-05-31T20:54:58.325 DEBUG:boto:Method: HEAD 2023-05-31T20:54:58.326 DEBUG:boto:Path: /pklxkd-59/obj34 2023-05-31T20:54:58.326 DEBUG:boto:Data: 2023-05-31T20:54:58.326 DEBUG:boto:Headers: {} 2023-05-31T20:54:58.327 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:54:58.327 DEBUG:boto:Port: 8001 2023-05-31T20:54:58.328 DEBUG:boto:Params: {} 2023-05-31T20:54:58.328 DEBUG:boto:Token: None 2023-05-31T20:54:58.329 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:54:58 GMT /pklxkd-59/obj34 2023-05-31T20:54:58.329 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:cBHrKN3M29n8MvoDtm3iuES3imc= 2023-05-31T20:54:58.330 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:54:58 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:cBHrKN3M29n8MvoDtm3iuES3imc=', 'Content-Length': '0'} 2023-05-31T20:54:58.332 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:43:07 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005a0e7c76f1e2425e-006477b422-1e2e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:54:58 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:54:58.332 DEBUG:boto:path=/pklxkd-59/obj34 2023-05-31T20:54:58.333 DEBUG:boto:auth_path=/pklxkd-59/obj34 2023-05-31T20:54:58.333 DEBUG:boto:Method: HEAD 2023-05-31T20:54:58.334 DEBUG:boto:Path: /pklxkd-59/obj34 2023-05-31T20:54:58.334 DEBUG:boto:Data: 2023-05-31T20:54:58.335 DEBUG:boto:Headers: {} 2023-05-31T20:54:58.335 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:54:58.335 DEBUG:boto:Port: 8000 2023-05-31T20:54:58.336 DEBUG:boto:Params: {} 2023-05-31T20:54:58.336 DEBUG:boto:Token: None 2023-05-31T20:54:58.337 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:54:58 GMT /pklxkd-59/obj34 2023-05-31T20:54:58.337 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:cBHrKN3M29n8MvoDtm3iuES3imc= 2023-05-31T20:54:58.338 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:54:58 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:cBHrKN3M29n8MvoDtm3iuES3imc=', 'Content-Length': '0'} 2023-05-31T20:54:58.339 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:43:07 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b6f1c0419745e046-006477b422-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:54:58 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:54:58.340 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj34 2023-05-31T20:54:58.340 DEBUG:boto:path=/pklxkd-59/obj34 2023-05-31T20:54:58.341 DEBUG:boto:auth_path=/pklxkd-59/obj34 2023-05-31T20:54:58.341 DEBUG:boto:Method: GET 2023-05-31T20:54:58.341 DEBUG:boto:Path: /pklxkd-59/obj34 2023-05-31T20:54:58.342 DEBUG:boto:Data: 2023-05-31T20:54:58.342 DEBUG:boto:Headers: {} 2023-05-31T20:54:58.342 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:54:58.343 DEBUG:boto:Port: 8001 2023-05-31T20:54:58.343 DEBUG:boto:Params: {} 2023-05-31T20:54:58.344 DEBUG:boto:Token: None 2023-05-31T20:54:58.344 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:54:58 GMT /pklxkd-59/obj34 2023-05-31T20:54:58.344 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:9PJFs8AC7f4HY/XJIDl/kpYRAag= 2023-05-31T20:54:58.345 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:54:58 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:9PJFs8AC7f4HY/XJIDl/kpYRAag=', 'Content-Length': '0'} 2023-05-31T20:54:58.347 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:43:07 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001de7542404ed1eba-006477b422-1e2e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:54:58 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:54:58.347 DEBUG:boto:path=/pklxkd-59/obj34 2023-05-31T20:54:58.348 DEBUG:boto:auth_path=/pklxkd-59/obj34 2023-05-31T20:54:58.348 DEBUG:boto:Method: GET 2023-05-31T20:54:58.348 DEBUG:boto:Path: /pklxkd-59/obj34 2023-05-31T20:54:58.349 DEBUG:boto:Data: 2023-05-31T20:54:58.349 DEBUG:boto:Headers: {} 2023-05-31T20:54:58.350 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:54:58.350 DEBUG:boto:Port: 8000 2023-05-31T20:54:58.350 DEBUG:boto:Params: {} 2023-05-31T20:54:58.351 DEBUG:boto:Token: None 2023-05-31T20:54:58.351 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:54:58 GMT /pklxkd-59/obj34 2023-05-31T20:54:58.351 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:9PJFs8AC7f4HY/XJIDl/kpYRAag= 2023-05-31T20:54:58.352 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:54:58 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:9PJFs8AC7f4HY/XJIDl/kpYRAag=', 'Content-Length': '0'} 2023-05-31T20:54:58.354 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:43:07 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000cf0ee7177a93382b-006477b422-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:54:58 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:54:58.396 DEBUG:boto:path=/pklxkd-59/ 2023-05-31T20:54:58.397 DEBUG:boto:auth_path=/pklxkd-59/ 2023-05-31T20:54:58.397 DEBUG:boto:path=/pklxkd-59/?prefix=obj34 2023-05-31T20:54:58.397 DEBUG:boto:auth_path=/pklxkd-59/?prefix=obj34 2023-05-31T20:54:58.398 DEBUG:boto:Method: GET 2023-05-31T20:54:58.398 DEBUG:boto:Path: /pklxkd-59/?prefix=obj34 2023-05-31T20:54:58.398 DEBUG:boto:Data: 2023-05-31T20:54:58.399 DEBUG:boto:Headers: {} 2023-05-31T20:54:58.399 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:54:58.399 DEBUG:boto:Port: 8001 2023-05-31T20:54:58.400 DEBUG:boto:Params: {} 2023-05-31T20:54:58.400 DEBUG:boto:Token: None 2023-05-31T20:54:58.401 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:54:58 GMT /pklxkd-59/ 2023-05-31T20:54:58.401 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:B6nEAA7PON5axhJ7EDo9Lg1oe5U= 2023-05-31T20:54:58.401 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:54:58 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:B6nEAA7PON5axhJ7EDo9Lg1oe5U=', 'Content-Length': '0'} 2023-05-31T20:54:58.404 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000079fb4904d9026fcf-006477b422-1e2e-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:54:58 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:54:58.404 DEBUG:boto:b'pklxkd-59obj341000falseobj342023-05-31T20:43:07.994Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:54:58.405 DEBUG:boto:path=/pklxkd-59/ 2023-05-31T20:54:58.405 DEBUG:boto:auth_path=/pklxkd-59/ 2023-05-31T20:54:58.406 DEBUG:boto:path=/pklxkd-59/?prefix=obj34 2023-05-31T20:54:58.406 DEBUG:boto:auth_path=/pklxkd-59/?prefix=obj34 2023-05-31T20:54:58.407 DEBUG:boto:Method: GET 2023-05-31T20:54:58.407 DEBUG:boto:Path: /pklxkd-59/?prefix=obj34 2023-05-31T20:54:58.407 DEBUG:boto:Data: 2023-05-31T20:54:58.408 DEBUG:boto:Headers: {} 2023-05-31T20:54:58.408 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:54:58.408 DEBUG:boto:Port: 8000 2023-05-31T20:54:58.409 DEBUG:boto:Params: {} 2023-05-31T20:54:58.409 DEBUG:boto:Token: None 2023-05-31T20:54:58.409 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:54:58 GMT /pklxkd-59/ 2023-05-31T20:54:58.410 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:B6nEAA7PON5axhJ7EDo9Lg1oe5U= 2023-05-31T20:54:58.410 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:54:58 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:B6nEAA7PON5axhJ7EDo9Lg1oe5U=', 'Content-Length': '0'} 2023-05-31T20:54:58.414 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000033c754a177edd397-006477b422-1630-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:54:58 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:54:58.415 DEBUG:boto:b'pklxkd-59obj341000falseobj342023-05-31T20:43:07.994Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:54:58.415 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj37 2023-05-31T20:54:58.416 DEBUG:boto:path=/pklxkd-59/obj37 2023-05-31T20:54:58.416 DEBUG:boto:auth_path=/pklxkd-59/obj37 2023-05-31T20:54:58.416 DEBUG:boto:path=/pklxkd-59/obj37?versionId=null 2023-05-31T20:54:58.417 DEBUG:boto:auth_path=/pklxkd-59/obj37?versionId=null 2023-05-31T20:54:58.417 DEBUG:boto:Method: GET 2023-05-31T20:54:58.417 DEBUG:boto:Path: /pklxkd-59/obj37?versionId=null 2023-05-31T20:54:58.418 DEBUG:boto:Data: 2023-05-31T20:54:58.418 DEBUG:boto:Headers: {} 2023-05-31T20:54:58.418 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:54:58.419 DEBUG:boto:Port: 8001 2023-05-31T20:54:58.419 DEBUG:boto:Params: {} 2023-05-31T20:54:58.420 DEBUG:boto:Token: None 2023-05-31T20:54:58.420 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:54:58 GMT /pklxkd-59/obj37?versionId=null 2023-05-31T20:54:58.421 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:IlToC/hJbUVLkr7GY5cDvOKWHGE= 2023-05-31T20:54:58.421 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:54:58 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:IlToC/hJbUVLkr7GY5cDvOKWHGE=', 'Content-Length': '0'} 2023-05-31T20:54:58.423 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:52:14 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000092f0d32035a90b29-006477b422-1e2e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:54:58 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:54:58.424 DEBUG:boto:path=/pklxkd-59/obj37 2023-05-31T20:54:58.424 DEBUG:boto:auth_path=/pklxkd-59/obj37 2023-05-31T20:54:58.425 DEBUG:boto:path=/pklxkd-59/obj37?versionId=null 2023-05-31T20:54:58.425 DEBUG:boto:auth_path=/pklxkd-59/obj37?versionId=null 2023-05-31T20:54:58.425 DEBUG:boto:Method: GET 2023-05-31T20:54:58.426 DEBUG:boto:Path: /pklxkd-59/obj37?versionId=null 2023-05-31T20:54:58.426 DEBUG:boto:Data: 2023-05-31T20:54:58.426 DEBUG:boto:Headers: {} 2023-05-31T20:54:58.427 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:54:58.427 DEBUG:boto:Port: 8000 2023-05-31T20:54:58.427 DEBUG:boto:Params: {} 2023-05-31T20:54:58.428 DEBUG:boto:Token: None 2023-05-31T20:54:58.428 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:54:58 GMT /pklxkd-59/obj37?versionId=null 2023-05-31T20:54:58.429 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:IlToC/hJbUVLkr7GY5cDvOKWHGE= 2023-05-31T20:54:58.429 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:54:58 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:IlToC/hJbUVLkr7GY5cDvOKWHGE=', 'Content-Length': '0'} 2023-05-31T20:54:58.431 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:52:14 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ccba3e36fbaf8362-006477b422-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:54:58 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:54:58.432 DEBUG:boto:path=/pklxkd-59/obj37 2023-05-31T20:54:58.432 DEBUG:boto:auth_path=/pklxkd-59/obj37 2023-05-31T20:54:58.433 DEBUG:boto:path=/pklxkd-59/obj37?versionId=null 2023-05-31T20:54:58.433 DEBUG:boto:auth_path=/pklxkd-59/obj37?versionId=null 2023-05-31T20:54:58.433 DEBUG:boto:Method: HEAD 2023-05-31T20:54:58.434 DEBUG:boto:Path: /pklxkd-59/obj37?versionId=null 2023-05-31T20:54:58.434 DEBUG:boto:Data: 2023-05-31T20:54:58.434 DEBUG:boto:Headers: {} 2023-05-31T20:54:58.434 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:54:58.435 DEBUG:boto:Port: 8001 2023-05-31T20:54:58.435 DEBUG:boto:Params: {} 2023-05-31T20:54:58.436 DEBUG:boto:Token: None 2023-05-31T20:54:58.436 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:54:58 GMT /pklxkd-59/obj37?versionId=null 2023-05-31T20:54:58.436 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:d6UaCw3YNxZCir7owo1nP8DFPM8= 2023-05-31T20:54:58.437 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:54:58 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:d6UaCw3YNxZCir7owo1nP8DFPM8=', 'Content-Length': '0'} 2023-05-31T20:54:58.439 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:52:14 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ca0bd97677b7df7d-006477b422-1e2e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:54:58 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:54:58.439 DEBUG:boto:path=/pklxkd-59/obj37 2023-05-31T20:54:58.440 DEBUG:boto:auth_path=/pklxkd-59/obj37 2023-05-31T20:54:58.440 DEBUG:boto:path=/pklxkd-59/obj37?versionId=null 2023-05-31T20:54:58.440 DEBUG:boto:auth_path=/pklxkd-59/obj37?versionId=null 2023-05-31T20:54:58.441 DEBUG:boto:Method: HEAD 2023-05-31T20:54:58.441 DEBUG:boto:Path: /pklxkd-59/obj37?versionId=null 2023-05-31T20:54:58.441 DEBUG:boto:Data: 2023-05-31T20:54:58.442 DEBUG:boto:Headers: {} 2023-05-31T20:54:58.442 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:54:58.442 DEBUG:boto:Port: 8000 2023-05-31T20:54:58.443 DEBUG:boto:Params: {} 2023-05-31T20:54:58.443 DEBUG:boto:Token: None 2023-05-31T20:54:58.443 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:54:58 GMT /pklxkd-59/obj37?versionId=null 2023-05-31T20:54:58.444 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:d6UaCw3YNxZCir7owo1nP8DFPM8= 2023-05-31T20:54:58.444 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:54:58 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:d6UaCw3YNxZCir7owo1nP8DFPM8=', 'Content-Length': '0'} 2023-05-31T20:54:58.446 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:52:14 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009288964e4068000a-006477b422-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:54:58 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:54:58.446 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj37 2023-05-31T20:54:58.447 DEBUG:boto:path=/pklxkd-59/obj37 2023-05-31T20:54:58.447 DEBUG:boto:auth_path=/pklxkd-59/obj37 2023-05-31T20:54:58.448 DEBUG:boto:path=/pklxkd-59/obj37?versionId=null 2023-05-31T20:54:58.448 DEBUG:boto:auth_path=/pklxkd-59/obj37?versionId=null 2023-05-31T20:54:58.448 DEBUG:boto:Method: GET 2023-05-31T20:54:58.449 DEBUG:boto:Path: /pklxkd-59/obj37?versionId=null 2023-05-31T20:54:58.449 DEBUG:boto:Data: 2023-05-31T20:54:58.449 DEBUG:boto:Headers: {} 2023-05-31T20:54:58.450 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:54:58.450 DEBUG:boto:Port: 8001 2023-05-31T20:54:58.450 DEBUG:boto:Params: {} 2023-05-31T20:54:58.451 DEBUG:boto:Token: None 2023-05-31T20:54:58.451 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:54:58 GMT /pklxkd-59/obj37?versionId=null 2023-05-31T20:54:58.452 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:IlToC/hJbUVLkr7GY5cDvOKWHGE= 2023-05-31T20:54:58.452 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:54:58 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:IlToC/hJbUVLkr7GY5cDvOKWHGE=', 'Content-Length': '0'} 2023-05-31T20:54:58.454 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:52:14 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001ed7e82666352fea-006477b422-1e2e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:54:58 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:54:58.455 DEBUG:boto:path=/pklxkd-59/obj37 2023-05-31T20:54:58.455 DEBUG:boto:auth_path=/pklxkd-59/obj37 2023-05-31T20:54:58.455 DEBUG:boto:path=/pklxkd-59/obj37?versionId=null 2023-05-31T20:54:58.456 DEBUG:boto:auth_path=/pklxkd-59/obj37?versionId=null 2023-05-31T20:54:58.456 DEBUG:boto:Method: GET 2023-05-31T20:54:58.456 DEBUG:boto:Path: /pklxkd-59/obj37?versionId=null 2023-05-31T20:54:58.457 DEBUG:boto:Data: 2023-05-31T20:54:58.457 DEBUG:boto:Headers: {} 2023-05-31T20:54:58.458 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:54:58.458 DEBUG:boto:Port: 8000 2023-05-31T20:54:58.458 DEBUG:boto:Params: {} 2023-05-31T20:54:58.459 DEBUG:boto:Token: None 2023-05-31T20:54:58.459 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:54:58 GMT /pklxkd-59/obj37?versionId=null 2023-05-31T20:54:58.459 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:IlToC/hJbUVLkr7GY5cDvOKWHGE= 2023-05-31T20:54:58.460 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:54:58 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:IlToC/hJbUVLkr7GY5cDvOKWHGE=', 'Content-Length': '0'} 2023-05-31T20:54:58.462 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:52:14 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006393cd496ac7607b-006477b422-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:54:58 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:54:58.463 DEBUG:boto:path=/pklxkd-59/ 2023-05-31T20:54:58.463 DEBUG:boto:auth_path=/pklxkd-59/ 2023-05-31T20:54:58.463 DEBUG:boto:path=/pklxkd-59/?prefix=obj37 2023-05-31T20:54:58.464 DEBUG:boto:auth_path=/pklxkd-59/?prefix=obj37 2023-05-31T20:54:58.464 DEBUG:boto:Method: GET 2023-05-31T20:54:58.464 DEBUG:boto:Path: /pklxkd-59/?prefix=obj37 2023-05-31T20:54:58.465 DEBUG:boto:Data: 2023-05-31T20:54:58.465 DEBUG:boto:Headers: {} 2023-05-31T20:54:58.465 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:54:58.466 DEBUG:boto:Port: 8001 2023-05-31T20:54:58.466 DEBUG:boto:Params: {} 2023-05-31T20:54:58.467 DEBUG:boto:Token: None 2023-05-31T20:54:58.467 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:54:58 GMT /pklxkd-59/ 2023-05-31T20:54:58.467 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:B6nEAA7PON5axhJ7EDo9Lg1oe5U= 2023-05-31T20:54:58.468 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:54:58 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:B6nEAA7PON5axhJ7EDo9Lg1oe5U=', 'Content-Length': '0'} 2023-05-31T20:54:58.470 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000dfc78e284d71473c-006477b422-1e2e-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:54:58 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:54:58.471 DEBUG:boto:b'pklxkd-59obj371000falseobj372023-05-31T20:52:14.909Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:54:58.471 DEBUG:boto:path=/pklxkd-59/ 2023-05-31T20:54:58.472 DEBUG:boto:auth_path=/pklxkd-59/ 2023-05-31T20:54:58.472 DEBUG:boto:path=/pklxkd-59/?prefix=obj37 2023-05-31T20:54:58.472 DEBUG:boto:auth_path=/pklxkd-59/?prefix=obj37 2023-05-31T20:54:58.473 DEBUG:boto:Method: GET 2023-05-31T20:54:58.473 DEBUG:boto:Path: /pklxkd-59/?prefix=obj37 2023-05-31T20:54:58.474 DEBUG:boto:Data: 2023-05-31T20:54:58.474 DEBUG:boto:Headers: {} 2023-05-31T20:54:58.474 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:54:58.475 DEBUG:boto:Port: 8000 2023-05-31T20:54:58.475 DEBUG:boto:Params: {} 2023-05-31T20:54:58.475 DEBUG:boto:Token: None 2023-05-31T20:54:58.476 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:54:58 GMT /pklxkd-59/ 2023-05-31T20:54:58.476 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:B6nEAA7PON5axhJ7EDo9Lg1oe5U= 2023-05-31T20:54:58.477 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:54:58 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:B6nEAA7PON5axhJ7EDo9Lg1oe5U=', 'Content-Length': '0'} 2023-05-31T20:54:58.481 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000945c6e412f25c728-006477b422-1630-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:54:58 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:54:58.481 DEBUG:boto:b'pklxkd-59obj371000falseobj372023-05-31T20:52:14.909Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:54:58.482 DEBUG:boto:path=/pklxkd-59/obj37 2023-05-31T20:54:58.482 DEBUG:boto:auth_path=/pklxkd-59/obj37 2023-05-31T20:54:58.483 DEBUG:boto:Method: HEAD 2023-05-31T20:54:58.483 DEBUG:boto:Path: /pklxkd-59/obj37 2023-05-31T20:54:58.483 DEBUG:boto:Data: 2023-05-31T20:54:58.484 DEBUG:boto:Headers: {} 2023-05-31T20:54:58.484 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:54:58.484 DEBUG:boto:Port: 8001 2023-05-31T20:54:58.485 DEBUG:boto:Params: {} 2023-05-31T20:54:58.485 DEBUG:boto:Token: None 2023-05-31T20:54:58.485 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:54:58 GMT /pklxkd-59/obj37 2023-05-31T20:54:58.486 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:/cOj0yk2N+Iuhth3kBO0CU/BMgI= 2023-05-31T20:54:58.486 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:54:58 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:/cOj0yk2N+Iuhth3kBO0CU/BMgI=', 'Content-Length': '0'} 2023-05-31T20:54:58.488 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:52:14 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000012fc075c74a0554-006477b422-1e2e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:54:58 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:54:58.488 DEBUG:boto:path=/pklxkd-59/obj37 2023-05-31T20:54:58.489 DEBUG:boto:auth_path=/pklxkd-59/obj37 2023-05-31T20:54:58.489 DEBUG:boto:Method: HEAD 2023-05-31T20:54:58.489 DEBUG:boto:Path: /pklxkd-59/obj37 2023-05-31T20:54:58.490 DEBUG:boto:Data: 2023-05-31T20:54:58.490 DEBUG:boto:Headers: {} 2023-05-31T20:54:58.490 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:54:58.491 DEBUG:boto:Port: 8000 2023-05-31T20:54:58.491 DEBUG:boto:Params: {} 2023-05-31T20:54:58.491 DEBUG:boto:Token: None 2023-05-31T20:54:58.492 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:54:58 GMT /pklxkd-59/obj37 2023-05-31T20:54:58.492 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:/cOj0yk2N+Iuhth3kBO0CU/BMgI= 2023-05-31T20:54:58.493 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:54:58 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:/cOj0yk2N+Iuhth3kBO0CU/BMgI=', 'Content-Length': '0'} 2023-05-31T20:54:58.494 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:52:14 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a82502ff0b3e145f-006477b422-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:54:58 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:54:58.495 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj37 2023-05-31T20:54:58.495 DEBUG:boto:path=/pklxkd-59/obj37 2023-05-31T20:54:58.496 DEBUG:boto:auth_path=/pklxkd-59/obj37 2023-05-31T20:54:58.496 DEBUG:boto:Method: GET 2023-05-31T20:54:58.496 DEBUG:boto:Path: /pklxkd-59/obj37 2023-05-31T20:54:58.497 DEBUG:boto:Data: 2023-05-31T20:54:58.497 DEBUG:boto:Headers: {} 2023-05-31T20:54:58.498 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:54:58.498 DEBUG:boto:Port: 8001 2023-05-31T20:54:58.498 DEBUG:boto:Params: {} 2023-05-31T20:54:58.499 DEBUG:boto:Token: None 2023-05-31T20:54:58.499 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:54:58 GMT /pklxkd-59/obj37 2023-05-31T20:54:58.499 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:psIKqDwTgGQrO8hAkVSHLTE9bw0= 2023-05-31T20:54:58.500 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:54:58 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:psIKqDwTgGQrO8hAkVSHLTE9bw0=', 'Content-Length': '0'} 2023-05-31T20:54:58.502 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:52:14 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009e92e4b0817edab7-006477b422-1e2e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:54:58 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:54:58.502 DEBUG:boto:path=/pklxkd-59/obj37 2023-05-31T20:54:58.503 DEBUG:boto:auth_path=/pklxkd-59/obj37 2023-05-31T20:54:58.503 DEBUG:boto:Method: GET 2023-05-31T20:54:58.503 DEBUG:boto:Path: /pklxkd-59/obj37 2023-05-31T20:54:58.504 DEBUG:boto:Data: 2023-05-31T20:54:58.504 DEBUG:boto:Headers: {} 2023-05-31T20:54:58.504 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:54:58.505 DEBUG:boto:Port: 8000 2023-05-31T20:54:58.505 DEBUG:boto:Params: {} 2023-05-31T20:54:58.506 DEBUG:boto:Token: None 2023-05-31T20:54:58.506 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:54:58 GMT /pklxkd-59/obj37 2023-05-31T20:54:58.506 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:psIKqDwTgGQrO8hAkVSHLTE9bw0= 2023-05-31T20:54:58.507 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:54:58 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:psIKqDwTgGQrO8hAkVSHLTE9bw0=', 'Content-Length': '0'} 2023-05-31T20:54:58.509 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:52:14 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005c2148c229dbf4d2-006477b422-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:54:58 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:54:58.552 DEBUG:boto:path=/pklxkd-59/ 2023-05-31T20:54:58.553 DEBUG:boto:auth_path=/pklxkd-59/ 2023-05-31T20:54:58.553 DEBUG:boto:path=/pklxkd-59/?prefix=obj37 2023-05-31T20:54:58.553 DEBUG:boto:auth_path=/pklxkd-59/?prefix=obj37 2023-05-31T20:54:58.554 DEBUG:boto:Method: GET 2023-05-31T20:54:58.554 DEBUG:boto:Path: /pklxkd-59/?prefix=obj37 2023-05-31T20:54:58.554 DEBUG:boto:Data: 2023-05-31T20:54:58.555 DEBUG:boto:Headers: {} 2023-05-31T20:54:58.555 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:54:58.556 DEBUG:boto:Port: 8001 2023-05-31T20:54:58.556 DEBUG:boto:Params: {} 2023-05-31T20:54:58.556 DEBUG:boto:Token: None 2023-05-31T20:54:58.557 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:54:58 GMT /pklxkd-59/ 2023-05-31T20:54:58.557 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:B6nEAA7PON5axhJ7EDo9Lg1oe5U= 2023-05-31T20:54:58.558 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:54:58 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:B6nEAA7PON5axhJ7EDo9Lg1oe5U=', 'Content-Length': '0'} 2023-05-31T20:54:58.560 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000df72de4edefa2586-006477b422-1e2e-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:54:58 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:54:58.561 DEBUG:boto:b'pklxkd-59obj371000falseobj372023-05-31T20:52:14.909Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:54:58.561 DEBUG:boto:path=/pklxkd-59/ 2023-05-31T20:54:58.562 DEBUG:boto:auth_path=/pklxkd-59/ 2023-05-31T20:54:58.562 DEBUG:boto:path=/pklxkd-59/?prefix=obj37 2023-05-31T20:54:58.562 DEBUG:boto:auth_path=/pklxkd-59/?prefix=obj37 2023-05-31T20:54:58.563 DEBUG:boto:Method: GET 2023-05-31T20:54:58.563 DEBUG:boto:Path: /pklxkd-59/?prefix=obj37 2023-05-31T20:54:58.563 DEBUG:boto:Data: 2023-05-31T20:54:58.564 DEBUG:boto:Headers: {} 2023-05-31T20:54:58.564 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:54:58.564 DEBUG:boto:Port: 8000 2023-05-31T20:54:58.565 DEBUG:boto:Params: {} 2023-05-31T20:54:58.565 DEBUG:boto:Token: None 2023-05-31T20:54:58.566 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:54:58 GMT /pklxkd-59/ 2023-05-31T20:54:58.566 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:B6nEAA7PON5axhJ7EDo9Lg1oe5U= 2023-05-31T20:54:58.566 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:54:58 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:B6nEAA7PON5axhJ7EDo9Lg1oe5U=', 'Content-Length': '0'} 2023-05-31T20:54:58.571 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000001ba2e996daa028a5-006477b422-1630-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:54:58 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:54:58.571 DEBUG:boto:b'pklxkd-59obj371000falseobj372023-05-31T20:52:14.909Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:54:58.572 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj38 2023-05-31T20:54:58.572 DEBUG:boto:path=/pklxkd-59/obj38 2023-05-31T20:54:58.572 DEBUG:boto:auth_path=/pklxkd-59/obj38 2023-05-31T20:54:58.573 DEBUG:boto:path=/pklxkd-59/obj38?versionId=null 2023-05-31T20:54:58.573 DEBUG:boto:auth_path=/pklxkd-59/obj38?versionId=null 2023-05-31T20:54:58.573 DEBUG:boto:Method: GET 2023-05-31T20:54:58.574 DEBUG:boto:Path: /pklxkd-59/obj38?versionId=null 2023-05-31T20:54:58.574 DEBUG:boto:Data: 2023-05-31T20:54:58.575 DEBUG:boto:Headers: {} 2023-05-31T20:54:58.575 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:54:58.575 DEBUG:boto:Port: 8001 2023-05-31T20:54:58.576 DEBUG:boto:Params: {} 2023-05-31T20:54:58.576 DEBUG:boto:Token: None 2023-05-31T20:54:58.576 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:54:58 GMT /pklxkd-59/obj38?versionId=null 2023-05-31T20:54:58.577 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:9D5hlwjpq5UMwU7CdfvMLIS3wHM= 2023-05-31T20:54:58.577 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:54:58 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:9D5hlwjpq5UMwU7CdfvMLIS3wHM=', 'Content-Length': '0'} 2023-05-31T20:54:58.579 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:44:50 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000066d4b72313358ad6-006477b422-1e2e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:54:58 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:54:58.580 DEBUG:boto:path=/pklxkd-59/obj38 2023-05-31T20:54:58.580 DEBUG:boto:auth_path=/pklxkd-59/obj38 2023-05-31T20:54:58.581 DEBUG:boto:path=/pklxkd-59/obj38?versionId=null 2023-05-31T20:54:58.581 DEBUG:boto:auth_path=/pklxkd-59/obj38?versionId=null 2023-05-31T20:54:58.582 DEBUG:boto:Method: GET 2023-05-31T20:54:58.582 DEBUG:boto:Path: /pklxkd-59/obj38?versionId=null 2023-05-31T20:54:58.582 DEBUG:boto:Data: 2023-05-31T20:54:58.583 DEBUG:boto:Headers: {} 2023-05-31T20:54:58.583 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:54:58.583 DEBUG:boto:Port: 8000 2023-05-31T20:54:58.584 DEBUG:boto:Params: {} 2023-05-31T20:54:58.584 DEBUG:boto:Token: None 2023-05-31T20:54:58.585 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:54:58 GMT /pklxkd-59/obj38?versionId=null 2023-05-31T20:54:58.585 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:9D5hlwjpq5UMwU7CdfvMLIS3wHM= 2023-05-31T20:54:58.585 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:54:58 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:9D5hlwjpq5UMwU7CdfvMLIS3wHM=', 'Content-Length': '0'} 2023-05-31T20:54:58.587 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:44:50 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004d666d204974413c-006477b422-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:54:58 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:54:58.588 DEBUG:boto:path=/pklxkd-59/obj38 2023-05-31T20:54:58.588 DEBUG:boto:auth_path=/pklxkd-59/obj38 2023-05-31T20:54:58.589 DEBUG:boto:path=/pklxkd-59/obj38?versionId=null 2023-05-31T20:54:58.589 DEBUG:boto:auth_path=/pklxkd-59/obj38?versionId=null 2023-05-31T20:54:58.589 DEBUG:boto:Method: HEAD 2023-05-31T20:54:58.590 DEBUG:boto:Path: /pklxkd-59/obj38?versionId=null 2023-05-31T20:54:58.590 DEBUG:boto:Data: 2023-05-31T20:54:58.590 DEBUG:boto:Headers: {} 2023-05-31T20:54:58.591 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:54:58.591 DEBUG:boto:Port: 8001 2023-05-31T20:54:58.591 DEBUG:boto:Params: {} 2023-05-31T20:54:58.592 DEBUG:boto:Token: None 2023-05-31T20:54:58.592 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:54:58 GMT /pklxkd-59/obj38?versionId=null 2023-05-31T20:54:58.593 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:1z1rQU6oZRuFajSSx9wHDe82M2I= 2023-05-31T20:54:58.593 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:54:58 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:1z1rQU6oZRuFajSSx9wHDe82M2I=', 'Content-Length': '0'} 2023-05-31T20:54:58.595 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:44:50 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f22b8735c6c98226-006477b422-1e2e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:54:58 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:54:58.595 DEBUG:boto:path=/pklxkd-59/obj38 2023-05-31T20:54:58.596 DEBUG:boto:auth_path=/pklxkd-59/obj38 2023-05-31T20:54:58.596 DEBUG:boto:path=/pklxkd-59/obj38?versionId=null 2023-05-31T20:54:58.596 DEBUG:boto:auth_path=/pklxkd-59/obj38?versionId=null 2023-05-31T20:54:58.597 DEBUG:boto:Method: HEAD 2023-05-31T20:54:58.597 DEBUG:boto:Path: /pklxkd-59/obj38?versionId=null 2023-05-31T20:54:58.597 DEBUG:boto:Data: 2023-05-31T20:54:58.598 DEBUG:boto:Headers: {} 2023-05-31T20:54:58.598 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:54:58.598 DEBUG:boto:Port: 8000 2023-05-31T20:54:58.599 DEBUG:boto:Params: {} 2023-05-31T20:54:58.599 DEBUG:boto:Token: None 2023-05-31T20:54:58.599 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:54:58 GMT /pklxkd-59/obj38?versionId=null 2023-05-31T20:54:58.600 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:1z1rQU6oZRuFajSSx9wHDe82M2I= 2023-05-31T20:54:58.600 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:54:58 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:1z1rQU6oZRuFajSSx9wHDe82M2I=', 'Content-Length': '0'} 2023-05-31T20:54:58.602 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:44:50 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b08e5ca9629447c9-006477b422-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:54:58 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:54:58.603 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj38 2023-05-31T20:54:58.603 DEBUG:boto:path=/pklxkd-59/obj38 2023-05-31T20:54:58.603 DEBUG:boto:auth_path=/pklxkd-59/obj38 2023-05-31T20:54:58.604 DEBUG:boto:path=/pklxkd-59/obj38?versionId=null 2023-05-31T20:54:58.604 DEBUG:boto:auth_path=/pklxkd-59/obj38?versionId=null 2023-05-31T20:54:58.604 DEBUG:boto:Method: GET 2023-05-31T20:54:58.605 DEBUG:boto:Path: /pklxkd-59/obj38?versionId=null 2023-05-31T20:54:58.605 DEBUG:boto:Data: 2023-05-31T20:54:58.605 DEBUG:boto:Headers: {} 2023-05-31T20:54:58.606 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:54:58.606 DEBUG:boto:Port: 8001 2023-05-31T20:54:58.606 DEBUG:boto:Params: {} 2023-05-31T20:54:58.607 DEBUG:boto:Token: None 2023-05-31T20:54:58.607 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:54:58 GMT /pklxkd-59/obj38?versionId=null 2023-05-31T20:54:58.608 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:9D5hlwjpq5UMwU7CdfvMLIS3wHM= 2023-05-31T20:54:58.608 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:54:58 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:9D5hlwjpq5UMwU7CdfvMLIS3wHM=', 'Content-Length': '0'} 2023-05-31T20:54:58.610 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:44:50 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b54e53befd81b95f-006477b422-1e2e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:54:58 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:54:58.610 DEBUG:boto:path=/pklxkd-59/obj38 2023-05-31T20:54:58.611 DEBUG:boto:auth_path=/pklxkd-59/obj38 2023-05-31T20:54:58.611 DEBUG:boto:path=/pklxkd-59/obj38?versionId=null 2023-05-31T20:54:58.612 DEBUG:boto:auth_path=/pklxkd-59/obj38?versionId=null 2023-05-31T20:54:58.612 DEBUG:boto:Method: GET 2023-05-31T20:54:58.612 DEBUG:boto:Path: /pklxkd-59/obj38?versionId=null 2023-05-31T20:54:58.613 DEBUG:boto:Data: 2023-05-31T20:54:58.613 DEBUG:boto:Headers: {} 2023-05-31T20:54:58.614 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:54:58.614 DEBUG:boto:Port: 8000 2023-05-31T20:54:58.614 DEBUG:boto:Params: {} 2023-05-31T20:54:58.615 DEBUG:boto:Token: None 2023-05-31T20:54:58.615 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:54:58 GMT /pklxkd-59/obj38?versionId=null 2023-05-31T20:54:58.615 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:9D5hlwjpq5UMwU7CdfvMLIS3wHM= 2023-05-31T20:54:58.616 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:54:58 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:9D5hlwjpq5UMwU7CdfvMLIS3wHM=', 'Content-Length': '0'} 2023-05-31T20:54:58.618 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:44:50 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f15d9ae42b553b9e-006477b422-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:54:58 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:54:58.619 DEBUG:boto:path=/pklxkd-59/ 2023-05-31T20:54:58.619 DEBUG:boto:auth_path=/pklxkd-59/ 2023-05-31T20:54:58.619 DEBUG:boto:path=/pklxkd-59/?prefix=obj38 2023-05-31T20:54:58.620 DEBUG:boto:auth_path=/pklxkd-59/?prefix=obj38 2023-05-31T20:54:58.620 DEBUG:boto:Method: GET 2023-05-31T20:54:58.620 DEBUG:boto:Path: /pklxkd-59/?prefix=obj38 2023-05-31T20:54:58.621 DEBUG:boto:Data: 2023-05-31T20:54:58.621 DEBUG:boto:Headers: {} 2023-05-31T20:54:58.622 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:54:58.622 DEBUG:boto:Port: 8001 2023-05-31T20:54:58.622 DEBUG:boto:Params: {} 2023-05-31T20:54:58.623 DEBUG:boto:Token: None 2023-05-31T20:54:58.623 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:54:58 GMT /pklxkd-59/ 2023-05-31T20:54:58.624 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:B6nEAA7PON5axhJ7EDo9Lg1oe5U= 2023-05-31T20:54:58.624 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:54:58 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:B6nEAA7PON5axhJ7EDo9Lg1oe5U=', 'Content-Length': '0'} 2023-05-31T20:54:58.627 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000cc0df6f02169932b-006477b422-1e2e-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:54:58 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:54:58.628 DEBUG:boto:b'pklxkd-59obj381000falseobj382023-05-31T20:44:50.610Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:54:58.628 DEBUG:boto:path=/pklxkd-59/ 2023-05-31T20:54:58.629 DEBUG:boto:auth_path=/pklxkd-59/ 2023-05-31T20:54:58.629 DEBUG:boto:path=/pklxkd-59/?prefix=obj38 2023-05-31T20:54:58.629 DEBUG:boto:auth_path=/pklxkd-59/?prefix=obj38 2023-05-31T20:54:58.630 DEBUG:boto:Method: GET 2023-05-31T20:54:58.630 DEBUG:boto:Path: /pklxkd-59/?prefix=obj38 2023-05-31T20:54:58.630 DEBUG:boto:Data: 2023-05-31T20:54:58.631 DEBUG:boto:Headers: {} 2023-05-31T20:54:58.631 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:54:58.632 DEBUG:boto:Port: 8000 2023-05-31T20:54:58.632 DEBUG:boto:Params: {} 2023-05-31T20:54:58.632 DEBUG:boto:Token: None 2023-05-31T20:54:58.633 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:54:58 GMT /pklxkd-59/ 2023-05-31T20:54:58.633 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:B6nEAA7PON5axhJ7EDo9Lg1oe5U= 2023-05-31T20:54:58.634 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:54:58 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:B6nEAA7PON5axhJ7EDo9Lg1oe5U=', 'Content-Length': '0'} 2023-05-31T20:54:58.637 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000000d9bafaf99e7e6d6-006477b422-1630-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:54:58 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:54:58.638 DEBUG:boto:b'pklxkd-59obj381000falseobj382023-05-31T20:44:50.610Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:54:58.638 DEBUG:boto:path=/pklxkd-59/obj38 2023-05-31T20:54:58.639 DEBUG:boto:auth_path=/pklxkd-59/obj38 2023-05-31T20:54:58.639 DEBUG:boto:Method: HEAD 2023-05-31T20:54:58.640 DEBUG:boto:Path: /pklxkd-59/obj38 2023-05-31T20:54:58.640 DEBUG:boto:Data: 2023-05-31T20:54:58.640 DEBUG:boto:Headers: {} 2023-05-31T20:54:58.641 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:54:58.641 DEBUG:boto:Port: 8001 2023-05-31T20:54:58.641 DEBUG:boto:Params: {} 2023-05-31T20:54:58.642 DEBUG:boto:Token: None 2023-05-31T20:54:58.642 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:54:58 GMT /pklxkd-59/obj38 2023-05-31T20:54:58.642 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:iSBMSlyWNadzAiDo9S9thEiQv3o= 2023-05-31T20:54:58.643 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:54:58 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:iSBMSlyWNadzAiDo9S9thEiQv3o=', 'Content-Length': '0'} 2023-05-31T20:54:58.645 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:44:50 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000002872fd9e890659c-006477b422-1e2e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:54:58 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:54:58.645 DEBUG:boto:path=/pklxkd-59/obj38 2023-05-31T20:54:58.645 DEBUG:boto:auth_path=/pklxkd-59/obj38 2023-05-31T20:54:58.646 DEBUG:boto:Method: HEAD 2023-05-31T20:54:58.646 DEBUG:boto:Path: /pklxkd-59/obj38 2023-05-31T20:54:58.647 DEBUG:boto:Data: 2023-05-31T20:54:58.647 DEBUG:boto:Headers: {} 2023-05-31T20:54:58.647 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:54:58.648 DEBUG:boto:Port: 8000 2023-05-31T20:54:58.648 DEBUG:boto:Params: {} 2023-05-31T20:54:58.648 DEBUG:boto:Token: None 2023-05-31T20:54:58.649 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:54:58 GMT /pklxkd-59/obj38 2023-05-31T20:54:58.649 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:iSBMSlyWNadzAiDo9S9thEiQv3o= 2023-05-31T20:54:58.649 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:54:58 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:iSBMSlyWNadzAiDo9S9thEiQv3o=', 'Content-Length': '0'} 2023-05-31T20:54:58.651 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:44:50 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001251ede1b869bdc5-006477b422-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:54:58 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:54:58.652 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj38 2023-05-31T20:54:58.652 DEBUG:boto:path=/pklxkd-59/obj38 2023-05-31T20:54:58.653 DEBUG:boto:auth_path=/pklxkd-59/obj38 2023-05-31T20:54:58.653 DEBUG:boto:Method: GET 2023-05-31T20:54:58.653 DEBUG:boto:Path: /pklxkd-59/obj38 2023-05-31T20:54:58.654 DEBUG:boto:Data: 2023-05-31T20:54:58.654 DEBUG:boto:Headers: {} 2023-05-31T20:54:58.654 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:54:58.655 DEBUG:boto:Port: 8001 2023-05-31T20:54:58.655 DEBUG:boto:Params: {} 2023-05-31T20:54:58.656 DEBUG:boto:Token: None 2023-05-31T20:54:58.656 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:54:58 GMT /pklxkd-59/obj38 2023-05-31T20:54:58.657 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:3Oh26ib4ScHHkfjSV/Af50UUQGo= 2023-05-31T20:54:58.657 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:54:58 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:3Oh26ib4ScHHkfjSV/Af50UUQGo=', 'Content-Length': '0'} 2023-05-31T20:54:58.659 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:44:50 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c94e560dc6b12760-006477b422-1e2e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:54:58 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:54:58.700 DEBUG:boto:path=/pklxkd-59/obj38 2023-05-31T20:54:58.701 DEBUG:boto:auth_path=/pklxkd-59/obj38 2023-05-31T20:54:58.701 DEBUG:boto:Method: GET 2023-05-31T20:54:58.701 DEBUG:boto:Path: /pklxkd-59/obj38 2023-05-31T20:54:58.702 DEBUG:boto:Data: 2023-05-31T20:54:58.702 DEBUG:boto:Headers: {} 2023-05-31T20:54:58.703 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:54:58.703 DEBUG:boto:Port: 8000 2023-05-31T20:54:58.703 DEBUG:boto:Params: {} 2023-05-31T20:54:58.704 DEBUG:boto:Token: None 2023-05-31T20:54:58.704 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:54:58 GMT /pklxkd-59/obj38 2023-05-31T20:54:58.705 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:3Oh26ib4ScHHkfjSV/Af50UUQGo= 2023-05-31T20:54:58.705 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:54:58 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:3Oh26ib4ScHHkfjSV/Af50UUQGo=', 'Content-Length': '0'} 2023-05-31T20:54:58.707 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:44:50 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000bf15c632e0ba34ef-006477b422-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:54:58 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:54:58.707 DEBUG:boto:path=/pklxkd-59/ 2023-05-31T20:54:58.708 DEBUG:boto:auth_path=/pklxkd-59/ 2023-05-31T20:54:58.708 DEBUG:boto:path=/pklxkd-59/?prefix=obj38 2023-05-31T20:54:58.708 DEBUG:boto:auth_path=/pklxkd-59/?prefix=obj38 2023-05-31T20:54:58.709 DEBUG:boto:Method: GET 2023-05-31T20:54:58.709 DEBUG:boto:Path: /pklxkd-59/?prefix=obj38 2023-05-31T20:54:58.709 DEBUG:boto:Data: 2023-05-31T20:54:58.710 DEBUG:boto:Headers: {} 2023-05-31T20:54:58.710 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:54:58.711 DEBUG:boto:Port: 8001 2023-05-31T20:54:58.711 DEBUG:boto:Params: {} 2023-05-31T20:54:58.711 DEBUG:boto:Token: None 2023-05-31T20:54:58.712 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:54:58 GMT /pklxkd-59/ 2023-05-31T20:54:58.712 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:B6nEAA7PON5axhJ7EDo9Lg1oe5U= 2023-05-31T20:54:58.713 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:54:58 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:B6nEAA7PON5axhJ7EDo9Lg1oe5U=', 'Content-Length': '0'} 2023-05-31T20:54:58.715 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000007436964186aec01-006477b422-1e2e-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:54:58 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:54:58.715 DEBUG:boto:b'pklxkd-59obj381000falseobj382023-05-31T20:44:50.610Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:54:58.716 DEBUG:boto:path=/pklxkd-59/ 2023-05-31T20:54:58.717 DEBUG:boto:auth_path=/pklxkd-59/ 2023-05-31T20:54:58.717 DEBUG:boto:path=/pklxkd-59/?prefix=obj38 2023-05-31T20:54:58.717 DEBUG:boto:auth_path=/pklxkd-59/?prefix=obj38 2023-05-31T20:54:58.718 DEBUG:boto:Method: GET 2023-05-31T20:54:58.718 DEBUG:boto:Path: /pklxkd-59/?prefix=obj38 2023-05-31T20:54:58.718 DEBUG:boto:Data: 2023-05-31T20:54:58.719 DEBUG:boto:Headers: {} 2023-05-31T20:54:58.719 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:54:58.719 DEBUG:boto:Port: 8000 2023-05-31T20:54:58.720 DEBUG:boto:Params: {} 2023-05-31T20:54:58.720 DEBUG:boto:Token: None 2023-05-31T20:54:58.721 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:54:58 GMT /pklxkd-59/ 2023-05-31T20:54:58.721 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:B6nEAA7PON5axhJ7EDo9Lg1oe5U= 2023-05-31T20:54:58.722 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:54:58 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:B6nEAA7PON5axhJ7EDo9Lg1oe5U=', 'Content-Length': '0'} 2023-05-31T20:54:58.725 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a4f5b9d46d222b52-006477b422-1630-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:54:58 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:54:58.726 DEBUG:boto:b'pklxkd-59obj381000falseobj382023-05-31T20:44:50.610Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:54:58.727 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj44 2023-05-31T20:54:58.727 DEBUG:boto:path=/pklxkd-59/obj44 2023-05-31T20:54:58.727 DEBUG:boto:auth_path=/pklxkd-59/obj44 2023-05-31T20:54:58.728 DEBUG:boto:path=/pklxkd-59/obj44?versionId=null 2023-05-31T20:54:58.728 DEBUG:boto:auth_path=/pklxkd-59/obj44?versionId=null 2023-05-31T20:54:58.728 DEBUG:boto:Method: GET 2023-05-31T20:54:58.729 DEBUG:boto:Path: /pklxkd-59/obj44?versionId=null 2023-05-31T20:54:58.729 DEBUG:boto:Data: 2023-05-31T20:54:58.729 DEBUG:boto:Headers: {} 2023-05-31T20:54:58.730 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:54:58.730 DEBUG:boto:Port: 8001 2023-05-31T20:54:58.731 DEBUG:boto:Params: {} 2023-05-31T20:54:58.731 DEBUG:boto:Token: None 2023-05-31T20:54:58.731 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:54:58 GMT /pklxkd-59/obj44?versionId=null 2023-05-31T20:54:58.732 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:KtjMmeyyvZO/UIt4eCL0Pm2byKU= 2023-05-31T20:54:58.732 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:54:58 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:KtjMmeyyvZO/UIt4eCL0Pm2byKU=', 'Content-Length': '0'} 2023-05-31T20:54:58.734 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:39:26 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000638dd5d1db6bb16a-006477b422-1e2e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:54:58 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:54:58.735 DEBUG:boto:path=/pklxkd-59/obj44 2023-05-31T20:54:58.735 DEBUG:boto:auth_path=/pklxkd-59/obj44 2023-05-31T20:54:58.736 DEBUG:boto:path=/pklxkd-59/obj44?versionId=null 2023-05-31T20:54:58.736 DEBUG:boto:auth_path=/pklxkd-59/obj44?versionId=null 2023-05-31T20:54:58.736 DEBUG:boto:Method: GET 2023-05-31T20:54:58.737 DEBUG:boto:Path: /pklxkd-59/obj44?versionId=null 2023-05-31T20:54:58.737 DEBUG:boto:Data: 2023-05-31T20:54:58.737 DEBUG:boto:Headers: {} 2023-05-31T20:54:58.738 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:54:58.739 DEBUG:boto:Port: 8000 2023-05-31T20:54:58.739 DEBUG:boto:Params: {} 2023-05-31T20:54:58.739 DEBUG:boto:Token: None 2023-05-31T20:54:58.740 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:54:58 GMT /pklxkd-59/obj44?versionId=null 2023-05-31T20:54:58.740 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:KtjMmeyyvZO/UIt4eCL0Pm2byKU= 2023-05-31T20:54:58.741 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:54:58 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:KtjMmeyyvZO/UIt4eCL0Pm2byKU=', 'Content-Length': '0'} 2023-05-31T20:54:58.743 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:39:26 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e555d7d538461ff1-006477b422-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:54:58 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:54:58.743 DEBUG:boto:path=/pklxkd-59/obj44 2023-05-31T20:54:58.744 DEBUG:boto:auth_path=/pklxkd-59/obj44 2023-05-31T20:54:58.744 DEBUG:boto:path=/pklxkd-59/obj44?versionId=null 2023-05-31T20:54:58.744 DEBUG:boto:auth_path=/pklxkd-59/obj44?versionId=null 2023-05-31T20:54:58.745 DEBUG:boto:Method: HEAD 2023-05-31T20:54:58.745 DEBUG:boto:Path: /pklxkd-59/obj44?versionId=null 2023-05-31T20:54:58.745 DEBUG:boto:Data: 2023-05-31T20:54:58.746 DEBUG:boto:Headers: {} 2023-05-31T20:54:58.746 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:54:58.746 DEBUG:boto:Port: 8001 2023-05-31T20:54:58.747 DEBUG:boto:Params: {} 2023-05-31T20:54:58.747 DEBUG:boto:Token: None 2023-05-31T20:54:58.748 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:54:58 GMT /pklxkd-59/obj44?versionId=null 2023-05-31T20:54:58.748 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:BTrcJ6JcVQB3rjqvA/8ORm1hwN0= 2023-05-31T20:54:58.748 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:54:58 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:BTrcJ6JcVQB3rjqvA/8ORm1hwN0=', 'Content-Length': '0'} 2023-05-31T20:54:58.750 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:39:26 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a5cac0ba2d3d9b03-006477b422-1e2e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:54:58 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:54:58.751 DEBUG:boto:path=/pklxkd-59/obj44 2023-05-31T20:54:58.751 DEBUG:boto:auth_path=/pklxkd-59/obj44 2023-05-31T20:54:58.752 DEBUG:boto:path=/pklxkd-59/obj44?versionId=null 2023-05-31T20:54:58.752 DEBUG:boto:auth_path=/pklxkd-59/obj44?versionId=null 2023-05-31T20:54:58.752 DEBUG:boto:Method: HEAD 2023-05-31T20:54:58.753 DEBUG:boto:Path: /pklxkd-59/obj44?versionId=null 2023-05-31T20:54:58.753 DEBUG:boto:Data: 2023-05-31T20:54:58.753 DEBUG:boto:Headers: {} 2023-05-31T20:54:58.754 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:54:58.754 DEBUG:boto:Port: 8000 2023-05-31T20:54:58.754 DEBUG:boto:Params: {} 2023-05-31T20:54:58.755 DEBUG:boto:Token: None 2023-05-31T20:54:58.755 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:54:58 GMT /pklxkd-59/obj44?versionId=null 2023-05-31T20:54:58.756 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:BTrcJ6JcVQB3rjqvA/8ORm1hwN0= 2023-05-31T20:54:58.756 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:54:58 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:BTrcJ6JcVQB3rjqvA/8ORm1hwN0=', 'Content-Length': '0'} 2023-05-31T20:54:58.758 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:39:26 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c753a4df6b8f640d-006477b422-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:54:58 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:54:58.759 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj44 2023-05-31T20:54:58.759 DEBUG:boto:path=/pklxkd-59/obj44 2023-05-31T20:54:58.759 DEBUG:boto:auth_path=/pklxkd-59/obj44 2023-05-31T20:54:58.760 DEBUG:boto:path=/pklxkd-59/obj44?versionId=null 2023-05-31T20:54:58.760 DEBUG:boto:auth_path=/pklxkd-59/obj44?versionId=null 2023-05-31T20:54:58.761 DEBUG:boto:Method: GET 2023-05-31T20:54:58.761 DEBUG:boto:Path: /pklxkd-59/obj44?versionId=null 2023-05-31T20:54:58.761 DEBUG:boto:Data: 2023-05-31T20:54:58.762 DEBUG:boto:Headers: {} 2023-05-31T20:54:58.762 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:54:58.762 DEBUG:boto:Port: 8001 2023-05-31T20:54:58.763 DEBUG:boto:Params: {} 2023-05-31T20:54:58.763 DEBUG:boto:Token: None 2023-05-31T20:54:58.764 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:54:58 GMT /pklxkd-59/obj44?versionId=null 2023-05-31T20:54:58.764 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:KtjMmeyyvZO/UIt4eCL0Pm2byKU= 2023-05-31T20:54:58.764 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:54:58 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:KtjMmeyyvZO/UIt4eCL0Pm2byKU=', 'Content-Length': '0'} 2023-05-31T20:54:58.766 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:39:26 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000916a4afbed1b018b-006477b422-1e2e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:54:58 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:54:58.767 DEBUG:boto:path=/pklxkd-59/obj44 2023-05-31T20:54:58.768 DEBUG:boto:auth_path=/pklxkd-59/obj44 2023-05-31T20:54:58.769 DEBUG:boto:path=/pklxkd-59/obj44?versionId=null 2023-05-31T20:54:58.769 DEBUG:boto:auth_path=/pklxkd-59/obj44?versionId=null 2023-05-31T20:54:58.771 DEBUG:boto:Method: GET 2023-05-31T20:54:58.772 DEBUG:boto:Path: /pklxkd-59/obj44?versionId=null 2023-05-31T20:54:58.772 DEBUG:boto:Data: 2023-05-31T20:54:58.772 DEBUG:boto:Headers: {} 2023-05-31T20:54:58.773 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:54:58.773 DEBUG:boto:Port: 8000 2023-05-31T20:54:58.773 DEBUG:boto:Params: {} 2023-05-31T20:54:58.774 DEBUG:boto:Token: None 2023-05-31T20:54:58.774 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:54:58 GMT /pklxkd-59/obj44?versionId=null 2023-05-31T20:54:58.775 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:KtjMmeyyvZO/UIt4eCL0Pm2byKU= 2023-05-31T20:54:58.775 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:54:58 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:KtjMmeyyvZO/UIt4eCL0Pm2byKU=', 'Content-Length': '0'} 2023-05-31T20:54:58.777 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:39:26 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000010f8b7dd2dfb8ff0-006477b422-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:54:58 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:54:58.778 DEBUG:boto:path=/pklxkd-59/ 2023-05-31T20:54:58.778 DEBUG:boto:auth_path=/pklxkd-59/ 2023-05-31T20:54:58.779 DEBUG:boto:path=/pklxkd-59/?prefix=obj44 2023-05-31T20:54:58.779 DEBUG:boto:auth_path=/pklxkd-59/?prefix=obj44 2023-05-31T20:54:58.780 DEBUG:boto:Method: GET 2023-05-31T20:54:58.780 DEBUG:boto:Path: /pklxkd-59/?prefix=obj44 2023-05-31T20:54:58.780 DEBUG:boto:Data: 2023-05-31T20:54:58.781 DEBUG:boto:Headers: {} 2023-05-31T20:54:58.781 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:54:58.781 DEBUG:boto:Port: 8001 2023-05-31T20:54:58.782 DEBUG:boto:Params: {} 2023-05-31T20:54:58.782 DEBUG:boto:Token: None 2023-05-31T20:54:58.783 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:54:58 GMT /pklxkd-59/ 2023-05-31T20:54:58.783 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:B6nEAA7PON5axhJ7EDo9Lg1oe5U= 2023-05-31T20:54:58.784 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:54:58 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:B6nEAA7PON5axhJ7EDo9Lg1oe5U=', 'Content-Length': '0'} 2023-05-31T20:54:58.786 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f3454efbc9779483-006477b422-1e2e-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:54:58 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:54:58.786 DEBUG:boto:b'pklxkd-59obj441000falseobj442023-05-31T20:39:26.565Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:54:58.787 DEBUG:boto:path=/pklxkd-59/ 2023-05-31T20:54:58.788 DEBUG:boto:auth_path=/pklxkd-59/ 2023-05-31T20:54:58.788 DEBUG:boto:path=/pklxkd-59/?prefix=obj44 2023-05-31T20:54:58.788 DEBUG:boto:auth_path=/pklxkd-59/?prefix=obj44 2023-05-31T20:54:58.789 DEBUG:boto:Method: GET 2023-05-31T20:54:58.789 DEBUG:boto:Path: /pklxkd-59/?prefix=obj44 2023-05-31T20:54:58.789 DEBUG:boto:Data: 2023-05-31T20:54:58.790 DEBUG:boto:Headers: {} 2023-05-31T20:54:58.790 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:54:58.790 DEBUG:boto:Port: 8000 2023-05-31T20:54:58.791 DEBUG:boto:Params: {} 2023-05-31T20:54:58.791 DEBUG:boto:Token: None 2023-05-31T20:54:58.792 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:54:58 GMT /pklxkd-59/ 2023-05-31T20:54:58.792 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:B6nEAA7PON5axhJ7EDo9Lg1oe5U= 2023-05-31T20:54:58.793 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:54:58 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:B6nEAA7PON5axhJ7EDo9Lg1oe5U=', 'Content-Length': '0'} 2023-05-31T20:54:58.796 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000006e3128d1021a78af-006477b422-1630-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:54:58 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:54:58.797 DEBUG:boto:b'pklxkd-59obj441000falseobj442023-05-31T20:39:26.565Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:54:58.797 DEBUG:boto:path=/pklxkd-59/obj44 2023-05-31T20:54:58.798 DEBUG:boto:auth_path=/pklxkd-59/obj44 2023-05-31T20:54:58.798 DEBUG:boto:Method: HEAD 2023-05-31T20:54:58.799 DEBUG:boto:Path: /pklxkd-59/obj44 2023-05-31T20:54:58.799 DEBUG:boto:Data: 2023-05-31T20:54:58.799 DEBUG:boto:Headers: {} 2023-05-31T20:54:58.800 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:54:58.800 DEBUG:boto:Port: 8001 2023-05-31T20:54:58.800 DEBUG:boto:Params: {} 2023-05-31T20:54:58.801 DEBUG:boto:Token: None 2023-05-31T20:54:58.801 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:54:58 GMT /pklxkd-59/obj44 2023-05-31T20:54:58.801 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:vkIU93HLcJLWYMQXTGjwZd+jg9M= 2023-05-31T20:54:58.802 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:54:58 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:vkIU93HLcJLWYMQXTGjwZd+jg9M=', 'Content-Length': '0'} 2023-05-31T20:54:58.804 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:39:26 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000d2f503685b9626d-006477b422-1e2e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:54:58 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:54:58.804 DEBUG:boto:path=/pklxkd-59/obj44 2023-05-31T20:54:58.805 DEBUG:boto:auth_path=/pklxkd-59/obj44 2023-05-31T20:54:58.805 DEBUG:boto:Method: HEAD 2023-05-31T20:54:58.805 DEBUG:boto:Path: /pklxkd-59/obj44 2023-05-31T20:54:58.806 DEBUG:boto:Data: 2023-05-31T20:54:58.806 DEBUG:boto:Headers: {} 2023-05-31T20:54:58.806 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:54:58.807 DEBUG:boto:Port: 8000 2023-05-31T20:54:58.807 DEBUG:boto:Params: {} 2023-05-31T20:54:58.807 DEBUG:boto:Token: None 2023-05-31T20:54:58.808 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:54:58 GMT /pklxkd-59/obj44 2023-05-31T20:54:58.808 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:vkIU93HLcJLWYMQXTGjwZd+jg9M= 2023-05-31T20:54:58.809 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:54:58 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:vkIU93HLcJLWYMQXTGjwZd+jg9M=', 'Content-Length': '0'} 2023-05-31T20:54:58.810 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:39:26 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000051e4af0ab1ae5321-006477b422-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:54:58 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:54:58.811 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj44 2023-05-31T20:54:58.811 DEBUG:boto:path=/pklxkd-59/obj44 2023-05-31T20:54:58.812 DEBUG:boto:auth_path=/pklxkd-59/obj44 2023-05-31T20:54:58.812 DEBUG:boto:Method: GET 2023-05-31T20:54:58.813 DEBUG:boto:Path: /pklxkd-59/obj44 2023-05-31T20:54:58.813 DEBUG:boto:Data: 2023-05-31T20:54:58.814 DEBUG:boto:Headers: {} 2023-05-31T20:54:58.814 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:54:58.814 DEBUG:boto:Port: 8001 2023-05-31T20:54:58.815 DEBUG:boto:Params: {} 2023-05-31T20:54:58.815 DEBUG:boto:Token: None 2023-05-31T20:54:58.816 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:54:58 GMT /pklxkd-59/obj44 2023-05-31T20:54:58.816 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:xN2hb4tgsbxNYfKclHWdttfZugA= 2023-05-31T20:54:58.817 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:54:58 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:xN2hb4tgsbxNYfKclHWdttfZugA=', 'Content-Length': '0'} 2023-05-31T20:54:58.819 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:39:26 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000083708c601660061d-006477b422-1e2e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:54:58 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:54:58.819 DEBUG:boto:path=/pklxkd-59/obj44 2023-05-31T20:54:58.820 DEBUG:boto:auth_path=/pklxkd-59/obj44 2023-05-31T20:54:58.820 DEBUG:boto:Method: GET 2023-05-31T20:54:58.820 DEBUG:boto:Path: /pklxkd-59/obj44 2023-05-31T20:54:58.821 DEBUG:boto:Data: 2023-05-31T20:54:58.821 DEBUG:boto:Headers: {} 2023-05-31T20:54:58.821 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:54:58.822 DEBUG:boto:Port: 8000 2023-05-31T20:54:58.822 DEBUG:boto:Params: {} 2023-05-31T20:54:58.823 DEBUG:boto:Token: None 2023-05-31T20:54:58.823 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:54:58 GMT /pklxkd-59/obj44 2023-05-31T20:54:58.824 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:xN2hb4tgsbxNYfKclHWdttfZugA= 2023-05-31T20:54:58.824 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:54:58 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:xN2hb4tgsbxNYfKclHWdttfZugA=', 'Content-Length': '0'} 2023-05-31T20:54:58.826 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:39:26 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e21298bb5d8b9114-006477b422-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:54:58 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:54:58.868 DEBUG:boto:path=/pklxkd-59/ 2023-05-31T20:54:58.869 DEBUG:boto:auth_path=/pklxkd-59/ 2023-05-31T20:54:58.869 DEBUG:boto:path=/pklxkd-59/?prefix=obj44 2023-05-31T20:54:58.869 DEBUG:boto:auth_path=/pklxkd-59/?prefix=obj44 2023-05-31T20:54:58.870 DEBUG:boto:Method: GET 2023-05-31T20:54:58.871 DEBUG:boto:Path: /pklxkd-59/?prefix=obj44 2023-05-31T20:54:58.871 DEBUG:boto:Data: 2023-05-31T20:54:58.871 DEBUG:boto:Headers: {} 2023-05-31T20:54:58.872 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:54:58.872 DEBUG:boto:Port: 8001 2023-05-31T20:54:58.872 DEBUG:boto:Params: {} 2023-05-31T20:54:58.873 DEBUG:boto:Token: None 2023-05-31T20:54:58.873 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:54:58 GMT /pklxkd-59/ 2023-05-31T20:54:58.874 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:B6nEAA7PON5axhJ7EDo9Lg1oe5U= 2023-05-31T20:54:58.874 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:54:58 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:B6nEAA7PON5axhJ7EDo9Lg1oe5U=', 'Content-Length': '0'} 2023-05-31T20:54:58.877 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000006f52fd20a5868753-006477b422-1e2e-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:54:58 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:54:58.877 DEBUG:boto:b'pklxkd-59obj441000falseobj442023-05-31T20:39:26.565Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:54:58.878 DEBUG:boto:path=/pklxkd-59/ 2023-05-31T20:54:58.878 DEBUG:boto:auth_path=/pklxkd-59/ 2023-05-31T20:54:58.879 DEBUG:boto:path=/pklxkd-59/?prefix=obj44 2023-05-31T20:54:58.879 DEBUG:boto:auth_path=/pklxkd-59/?prefix=obj44 2023-05-31T20:54:58.879 DEBUG:boto:Method: GET 2023-05-31T20:54:58.880 DEBUG:boto:Path: /pklxkd-59/?prefix=obj44 2023-05-31T20:54:58.880 DEBUG:boto:Data: 2023-05-31T20:54:58.881 DEBUG:boto:Headers: {} 2023-05-31T20:54:58.881 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:54:58.882 DEBUG:boto:Port: 8000 2023-05-31T20:54:58.882 DEBUG:boto:Params: {} 2023-05-31T20:54:58.882 DEBUG:boto:Token: None 2023-05-31T20:54:58.883 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:54:58 GMT /pklxkd-59/ 2023-05-31T20:54:58.883 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:B6nEAA7PON5axhJ7EDo9Lg1oe5U= 2023-05-31T20:54:58.884 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:54:58 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:B6nEAA7PON5axhJ7EDo9Lg1oe5U=', 'Content-Length': '0'} 2023-05-31T20:54:58.888 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000006fa3f53babe41074-006477b422-1630-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:54:58 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:54:58.889 DEBUG:boto:b'pklxkd-59obj441000falseobj442023-05-31T20:39:26.565Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:54:58.889 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj46 2023-05-31T20:54:58.890 DEBUG:boto:path=/pklxkd-59/obj46 2023-05-31T20:54:58.890 DEBUG:boto:auth_path=/pklxkd-59/obj46 2023-05-31T20:54:58.891 DEBUG:boto:path=/pklxkd-59/obj46?versionId=null 2023-05-31T20:54:58.891 DEBUG:boto:auth_path=/pklxkd-59/obj46?versionId=null 2023-05-31T20:54:58.892 DEBUG:boto:Method: GET 2023-05-31T20:54:58.892 DEBUG:boto:Path: /pklxkd-59/obj46?versionId=null 2023-05-31T20:54:58.893 DEBUG:boto:Data: 2023-05-31T20:54:58.893 DEBUG:boto:Headers: {} 2023-05-31T20:54:58.893 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:54:58.894 DEBUG:boto:Port: 8001 2023-05-31T20:54:58.894 DEBUG:boto:Params: {} 2023-05-31T20:54:58.894 DEBUG:boto:Token: None 2023-05-31T20:54:58.895 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:54:58 GMT /pklxkd-59/obj46?versionId=null 2023-05-31T20:54:58.895 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:rbbnMQvwlpwq5QjAjloSfoTl1JA= 2023-05-31T20:54:58.896 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:54:58 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:rbbnMQvwlpwq5QjAjloSfoTl1JA=', 'Content-Length': '0'} 2023-05-31T20:54:58.898 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:46:33 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003c7ce9a2e7aab375-006477b422-1e2e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:54:58 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:54:58.899 DEBUG:boto:path=/pklxkd-59/obj46 2023-05-31T20:54:58.899 DEBUG:boto:auth_path=/pklxkd-59/obj46 2023-05-31T20:54:58.900 DEBUG:boto:path=/pklxkd-59/obj46?versionId=null 2023-05-31T20:54:58.900 DEBUG:boto:auth_path=/pklxkd-59/obj46?versionId=null 2023-05-31T20:54:58.900 DEBUG:boto:Method: GET 2023-05-31T20:54:58.901 DEBUG:boto:Path: /pklxkd-59/obj46?versionId=null 2023-05-31T20:54:58.901 DEBUG:boto:Data: 2023-05-31T20:54:58.902 DEBUG:boto:Headers: {} 2023-05-31T20:54:58.902 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:54:58.902 DEBUG:boto:Port: 8000 2023-05-31T20:54:58.903 DEBUG:boto:Params: {} 2023-05-31T20:54:58.903 DEBUG:boto:Token: None 2023-05-31T20:54:58.904 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:54:58 GMT /pklxkd-59/obj46?versionId=null 2023-05-31T20:54:58.905 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:rbbnMQvwlpwq5QjAjloSfoTl1JA= 2023-05-31T20:54:58.905 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:54:58 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:rbbnMQvwlpwq5QjAjloSfoTl1JA=', 'Content-Length': '0'} 2023-05-31T20:54:58.907 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:46:33 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c41e2cb1219ad7d1-006477b422-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:54:58 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:54:58.908 DEBUG:boto:path=/pklxkd-59/obj46 2023-05-31T20:54:58.908 DEBUG:boto:auth_path=/pklxkd-59/obj46 2023-05-31T20:54:58.908 DEBUG:boto:path=/pklxkd-59/obj46?versionId=null 2023-05-31T20:54:58.909 DEBUG:boto:auth_path=/pklxkd-59/obj46?versionId=null 2023-05-31T20:54:58.909 DEBUG:boto:Method: HEAD 2023-05-31T20:54:58.909 DEBUG:boto:Path: /pklxkd-59/obj46?versionId=null 2023-05-31T20:54:58.910 DEBUG:boto:Data: 2023-05-31T20:54:58.910 DEBUG:boto:Headers: {} 2023-05-31T20:54:58.910 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:54:58.911 DEBUG:boto:Port: 8001 2023-05-31T20:54:58.911 DEBUG:boto:Params: {} 2023-05-31T20:54:58.911 DEBUG:boto:Token: None 2023-05-31T20:54:58.912 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:54:58 GMT /pklxkd-59/obj46?versionId=null 2023-05-31T20:54:58.912 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:T7NoLzGrzAm5EynwDtFjE2NFQtY= 2023-05-31T20:54:58.913 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:54:58 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:T7NoLzGrzAm5EynwDtFjE2NFQtY=', 'Content-Length': '0'} 2023-05-31T20:54:58.914 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:46:33 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000097f5a421a000fd98-006477b422-1e2e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:54:58 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:54:58.915 DEBUG:boto:path=/pklxkd-59/obj46 2023-05-31T20:54:58.915 DEBUG:boto:auth_path=/pklxkd-59/obj46 2023-05-31T20:54:58.916 DEBUG:boto:path=/pklxkd-59/obj46?versionId=null 2023-05-31T20:54:58.916 DEBUG:boto:auth_path=/pklxkd-59/obj46?versionId=null 2023-05-31T20:54:58.917 DEBUG:boto:Method: HEAD 2023-05-31T20:54:58.917 DEBUG:boto:Path: /pklxkd-59/obj46?versionId=null 2023-05-31T20:54:58.917 DEBUG:boto:Data: 2023-05-31T20:54:58.918 DEBUG:boto:Headers: {} 2023-05-31T20:54:58.918 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:54:58.918 DEBUG:boto:Port: 8000 2023-05-31T20:54:58.919 DEBUG:boto:Params: {} 2023-05-31T20:54:58.919 DEBUG:boto:Token: None 2023-05-31T20:54:58.920 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:54:58 GMT /pklxkd-59/obj46?versionId=null 2023-05-31T20:54:58.920 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:T7NoLzGrzAm5EynwDtFjE2NFQtY= 2023-05-31T20:54:58.921 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:54:58 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:T7NoLzGrzAm5EynwDtFjE2NFQtY=', 'Content-Length': '0'} 2023-05-31T20:54:58.922 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:46:33 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009a87a8ea16a93d56-006477b422-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:54:58 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:54:58.923 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj46 2023-05-31T20:54:58.923 DEBUG:boto:path=/pklxkd-59/obj46 2023-05-31T20:54:58.924 DEBUG:boto:auth_path=/pklxkd-59/obj46 2023-05-31T20:54:58.924 DEBUG:boto:path=/pklxkd-59/obj46?versionId=null 2023-05-31T20:54:58.924 DEBUG:boto:auth_path=/pklxkd-59/obj46?versionId=null 2023-05-31T20:54:58.925 DEBUG:boto:Method: GET 2023-05-31T20:54:58.925 DEBUG:boto:Path: /pklxkd-59/obj46?versionId=null 2023-05-31T20:54:58.926 DEBUG:boto:Data: 2023-05-31T20:54:58.926 DEBUG:boto:Headers: {} 2023-05-31T20:54:58.926 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:54:58.927 DEBUG:boto:Port: 8001 2023-05-31T20:54:58.927 DEBUG:boto:Params: {} 2023-05-31T20:54:58.928 DEBUG:boto:Token: None 2023-05-31T20:54:58.928 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:54:58 GMT /pklxkd-59/obj46?versionId=null 2023-05-31T20:54:58.929 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:rbbnMQvwlpwq5QjAjloSfoTl1JA= 2023-05-31T20:54:58.929 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:54:58 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:rbbnMQvwlpwq5QjAjloSfoTl1JA=', 'Content-Length': '0'} 2023-05-31T20:54:58.931 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:46:33 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008cda325fdedc0bc5-006477b422-1e2e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:54:58 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:54:58.932 DEBUG:boto:path=/pklxkd-59/obj46 2023-05-31T20:54:58.932 DEBUG:boto:auth_path=/pklxkd-59/obj46 2023-05-31T20:54:58.933 DEBUG:boto:path=/pklxkd-59/obj46?versionId=null 2023-05-31T20:54:58.933 DEBUG:boto:auth_path=/pklxkd-59/obj46?versionId=null 2023-05-31T20:54:58.933 DEBUG:boto:Method: GET 2023-05-31T20:54:58.934 DEBUG:boto:Path: /pklxkd-59/obj46?versionId=null 2023-05-31T20:54:58.934 DEBUG:boto:Data: 2023-05-31T20:54:58.934 DEBUG:boto:Headers: {} 2023-05-31T20:54:58.935 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:54:58.935 DEBUG:boto:Port: 8000 2023-05-31T20:54:58.936 DEBUG:boto:Params: {} 2023-05-31T20:54:58.936 DEBUG:boto:Token: None 2023-05-31T20:54:58.937 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:54:58 GMT /pklxkd-59/obj46?versionId=null 2023-05-31T20:54:58.937 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:rbbnMQvwlpwq5QjAjloSfoTl1JA= 2023-05-31T20:54:58.938 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:54:58 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:rbbnMQvwlpwq5QjAjloSfoTl1JA=', 'Content-Length': '0'} 2023-05-31T20:54:58.939 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:46:33 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e43a937bf26ad215-006477b422-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:54:58 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:54:58.940 DEBUG:boto:path=/pklxkd-59/ 2023-05-31T20:54:58.941 DEBUG:boto:auth_path=/pklxkd-59/ 2023-05-31T20:54:58.941 DEBUG:boto:path=/pklxkd-59/?prefix=obj46 2023-05-31T20:54:58.941 DEBUG:boto:auth_path=/pklxkd-59/?prefix=obj46 2023-05-31T20:54:58.942 DEBUG:boto:Method: GET 2023-05-31T20:54:58.942 DEBUG:boto:Path: /pklxkd-59/?prefix=obj46 2023-05-31T20:54:58.943 DEBUG:boto:Data: 2023-05-31T20:54:58.943 DEBUG:boto:Headers: {} 2023-05-31T20:54:58.943 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:54:58.944 DEBUG:boto:Port: 8001 2023-05-31T20:54:58.944 DEBUG:boto:Params: {} 2023-05-31T20:54:58.945 DEBUG:boto:Token: None 2023-05-31T20:54:58.945 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:54:58 GMT /pklxkd-59/ 2023-05-31T20:54:58.946 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:B6nEAA7PON5axhJ7EDo9Lg1oe5U= 2023-05-31T20:54:58.946 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:54:58 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:B6nEAA7PON5axhJ7EDo9Lg1oe5U=', 'Content-Length': '0'} 2023-05-31T20:54:58.948 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b8e275afd3b1dfb5-006477b422-1e2e-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:54:58 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:54:58.949 DEBUG:boto:b'pklxkd-59obj461000falseobj462023-05-31T20:46:33.830Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:54:58.950 DEBUG:boto:path=/pklxkd-59/ 2023-05-31T20:54:58.950 DEBUG:boto:auth_path=/pklxkd-59/ 2023-05-31T20:54:58.950 DEBUG:boto:path=/pklxkd-59/?prefix=obj46 2023-05-31T20:54:58.951 DEBUG:boto:auth_path=/pklxkd-59/?prefix=obj46 2023-05-31T20:54:58.951 DEBUG:boto:Method: GET 2023-05-31T20:54:58.952 DEBUG:boto:Path: /pklxkd-59/?prefix=obj46 2023-05-31T20:54:58.952 DEBUG:boto:Data: 2023-05-31T20:54:58.953 DEBUG:boto:Headers: {} 2023-05-31T20:54:58.953 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:54:58.953 DEBUG:boto:Port: 8000 2023-05-31T20:54:58.954 DEBUG:boto:Params: {} 2023-05-31T20:54:58.954 DEBUG:boto:Token: None 2023-05-31T20:54:58.954 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:54:58 GMT /pklxkd-59/ 2023-05-31T20:54:58.955 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:B6nEAA7PON5axhJ7EDo9Lg1oe5U= 2023-05-31T20:54:58.956 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:54:58 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:B6nEAA7PON5axhJ7EDo9Lg1oe5U=', 'Content-Length': '0'} 2023-05-31T20:54:58.960 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000045386018db799e2d-006477b422-1630-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:54:58 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:54:58.960 DEBUG:boto:b'pklxkd-59obj461000falseobj462023-05-31T20:46:33.830Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:54:58.961 DEBUG:boto:path=/pklxkd-59/obj46 2023-05-31T20:54:58.961 DEBUG:boto:auth_path=/pklxkd-59/obj46 2023-05-31T20:54:58.962 DEBUG:boto:Method: HEAD 2023-05-31T20:54:58.962 DEBUG:boto:Path: /pklxkd-59/obj46 2023-05-31T20:54:58.962 DEBUG:boto:Data: 2023-05-31T20:54:58.963 DEBUG:boto:Headers: {} 2023-05-31T20:54:58.963 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:54:58.963 DEBUG:boto:Port: 8001 2023-05-31T20:54:58.963 DEBUG:boto:Params: {} 2023-05-31T20:54:58.964 DEBUG:boto:Token: None 2023-05-31T20:54:58.964 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:54:58 GMT /pklxkd-59/obj46 2023-05-31T20:54:58.965 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:uWkme1sIwgGO0QmzTTx5dIss8RY= 2023-05-31T20:54:58.965 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:54:58 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:uWkme1sIwgGO0QmzTTx5dIss8RY=', 'Content-Length': '0'} 2023-05-31T20:54:58.967 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:46:33 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000bb0a6a35e55f22a7-006477b422-1e2e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:54:58 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:54:58.968 DEBUG:boto:path=/pklxkd-59/obj46 2023-05-31T20:54:58.968 DEBUG:boto:auth_path=/pklxkd-59/obj46 2023-05-31T20:54:58.969 DEBUG:boto:Method: HEAD 2023-05-31T20:54:58.969 DEBUG:boto:Path: /pklxkd-59/obj46 2023-05-31T20:54:58.969 DEBUG:boto:Data: 2023-05-31T20:54:58.970 DEBUG:boto:Headers: {} 2023-05-31T20:54:58.970 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:54:58.970 DEBUG:boto:Port: 8000 2023-05-31T20:54:58.971 DEBUG:boto:Params: {} 2023-05-31T20:54:58.971 DEBUG:boto:Token: None 2023-05-31T20:54:58.972 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:54:58 GMT /pklxkd-59/obj46 2023-05-31T20:54:58.972 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:uWkme1sIwgGO0QmzTTx5dIss8RY= 2023-05-31T20:54:58.972 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:54:58 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:uWkme1sIwgGO0QmzTTx5dIss8RY=', 'Content-Length': '0'} 2023-05-31T20:54:58.974 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:46:33 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000081d86897c6161305-006477b422-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:54:58 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:54:58.975 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj46 2023-05-31T20:54:58.975 DEBUG:boto:path=/pklxkd-59/obj46 2023-05-31T20:54:58.976 DEBUG:boto:auth_path=/pklxkd-59/obj46 2023-05-31T20:54:58.976 DEBUG:boto:Method: GET 2023-05-31T20:54:58.976 DEBUG:boto:Path: /pklxkd-59/obj46 2023-05-31T20:54:58.977 DEBUG:boto:Data: 2023-05-31T20:54:58.977 DEBUG:boto:Headers: {} 2023-05-31T20:54:58.977 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:54:58.978 DEBUG:boto:Port: 8001 2023-05-31T20:54:58.978 DEBUG:boto:Params: {} 2023-05-31T20:54:58.979 DEBUG:boto:Token: None 2023-05-31T20:54:58.979 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:54:58 GMT /pklxkd-59/obj46 2023-05-31T20:54:58.980 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:L67N9RhcxWPnlsIHWkIq5Ifxxr0= 2023-05-31T20:54:58.980 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:54:58 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:L67N9RhcxWPnlsIHWkIq5Ifxxr0=', 'Content-Length': '0'} 2023-05-31T20:54:58.982 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:46:33 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000aa0a2af5daef08b7-006477b422-1e2e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:54:58 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:54:59.024 DEBUG:boto:path=/pklxkd-59/obj46 2023-05-31T20:54:59.025 DEBUG:boto:auth_path=/pklxkd-59/obj46 2023-05-31T20:54:59.025 DEBUG:boto:Method: GET 2023-05-31T20:54:59.025 DEBUG:boto:Path: /pklxkd-59/obj46 2023-05-31T20:54:59.026 DEBUG:boto:Data: 2023-05-31T20:54:59.026 DEBUG:boto:Headers: {} 2023-05-31T20:54:59.026 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:54:59.027 DEBUG:boto:Port: 8000 2023-05-31T20:54:59.027 DEBUG:boto:Params: {} 2023-05-31T20:54:59.028 DEBUG:boto:Token: None 2023-05-31T20:54:59.028 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:54:59 GMT /pklxkd-59/obj46 2023-05-31T20:54:59.028 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:/8yLCZwKEBjYnzPKkHCWgR9GkOI= 2023-05-31T20:54:59.029 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:54:59 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:/8yLCZwKEBjYnzPKkHCWgR9GkOI=', 'Content-Length': '0'} 2023-05-31T20:54:59.031 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:46:33 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006d9a10af05141840-006477b423-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:54:59 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:54:59.031 DEBUG:boto:path=/pklxkd-59/ 2023-05-31T20:54:59.032 DEBUG:boto:auth_path=/pklxkd-59/ 2023-05-31T20:54:59.032 DEBUG:boto:path=/pklxkd-59/?prefix=obj46 2023-05-31T20:54:59.033 DEBUG:boto:auth_path=/pklxkd-59/?prefix=obj46 2023-05-31T20:54:59.033 DEBUG:boto:Method: GET 2023-05-31T20:54:59.033 DEBUG:boto:Path: /pklxkd-59/?prefix=obj46 2023-05-31T20:54:59.034 DEBUG:boto:Data: 2023-05-31T20:54:59.034 DEBUG:boto:Headers: {} 2023-05-31T20:54:59.034 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:54:59.035 DEBUG:boto:Port: 8001 2023-05-31T20:54:59.035 DEBUG:boto:Params: {} 2023-05-31T20:54:59.035 DEBUG:boto:Token: None 2023-05-31T20:54:59.036 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:54:59 GMT /pklxkd-59/ 2023-05-31T20:54:59.036 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:tAvGnTMQPosMaETd601jyw6cOdE= 2023-05-31T20:54:59.037 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:54:59 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:tAvGnTMQPosMaETd601jyw6cOdE=', 'Content-Length': '0'} 2023-05-31T20:54:59.039 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b196cd5f80b1ff74-006477b423-1e2e-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:54:59 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:54:59.039 DEBUG:boto:b'pklxkd-59obj461000falseobj462023-05-31T20:46:33.830Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:54:59.040 DEBUG:boto:path=/pklxkd-59/ 2023-05-31T20:54:59.040 DEBUG:boto:auth_path=/pklxkd-59/ 2023-05-31T20:54:59.041 DEBUG:boto:path=/pklxkd-59/?prefix=obj46 2023-05-31T20:54:59.041 DEBUG:boto:auth_path=/pklxkd-59/?prefix=obj46 2023-05-31T20:54:59.042 DEBUG:boto:Method: GET 2023-05-31T20:54:59.042 DEBUG:boto:Path: /pklxkd-59/?prefix=obj46 2023-05-31T20:54:59.042 DEBUG:boto:Data: 2023-05-31T20:54:59.043 DEBUG:boto:Headers: {} 2023-05-31T20:54:59.043 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:54:59.043 DEBUG:boto:Port: 8000 2023-05-31T20:54:59.044 DEBUG:boto:Params: {} 2023-05-31T20:54:59.044 DEBUG:boto:Token: None 2023-05-31T20:54:59.045 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:54:59 GMT /pklxkd-59/ 2023-05-31T20:54:59.045 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:tAvGnTMQPosMaETd601jyw6cOdE= 2023-05-31T20:54:59.045 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:54:59 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:tAvGnTMQPosMaETd601jyw6cOdE=', 'Content-Length': '0'} 2023-05-31T20:54:59.049 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000fc4e016bd94a8227-006477b423-1630-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:54:59 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:54:59.050 DEBUG:boto:b'pklxkd-59obj461000falseobj462023-05-31T20:46:33.830Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:54:59.050 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj51 2023-05-31T20:54:59.051 DEBUG:boto:path=/pklxkd-59/obj51 2023-05-31T20:54:59.051 DEBUG:boto:auth_path=/pklxkd-59/obj51 2023-05-31T20:54:59.052 DEBUG:boto:path=/pklxkd-59/obj51?versionId=null 2023-05-31T20:54:59.052 DEBUG:boto:auth_path=/pklxkd-59/obj51?versionId=null 2023-05-31T20:54:59.052 DEBUG:boto:Method: GET 2023-05-31T20:54:59.053 DEBUG:boto:Path: /pklxkd-59/obj51?versionId=null 2023-05-31T20:54:59.053 DEBUG:boto:Data: 2023-05-31T20:54:59.053 DEBUG:boto:Headers: {} 2023-05-31T20:54:59.054 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:54:59.054 DEBUG:boto:Port: 8001 2023-05-31T20:54:59.054 DEBUG:boto:Params: {} 2023-05-31T20:54:59.055 DEBUG:boto:Token: None 2023-05-31T20:54:59.055 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:54:59 GMT /pklxkd-59/obj51?versionId=null 2023-05-31T20:54:59.056 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:KtKRRZJTGUSUHYaFvxwNB4dOnkk= 2023-05-31T20:54:59.056 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:54:59 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:KtKRRZJTGUSUHYaFvxwNB4dOnkk=', 'Content-Length': '0'} 2023-05-31T20:54:59.058 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:43:08 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000bf1904a78aa021c5-006477b423-1e2e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:54:59 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:54:59.059 DEBUG:boto:path=/pklxkd-59/obj51 2023-05-31T20:54:59.059 DEBUG:boto:auth_path=/pklxkd-59/obj51 2023-05-31T20:54:59.060 DEBUG:boto:path=/pklxkd-59/obj51?versionId=null 2023-05-31T20:54:59.060 DEBUG:boto:auth_path=/pklxkd-59/obj51?versionId=null 2023-05-31T20:54:59.060 DEBUG:boto:Method: GET 2023-05-31T20:54:59.061 DEBUG:boto:Path: /pklxkd-59/obj51?versionId=null 2023-05-31T20:54:59.061 DEBUG:boto:Data: 2023-05-31T20:54:59.062 DEBUG:boto:Headers: {} 2023-05-31T20:54:59.062 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:54:59.062 DEBUG:boto:Port: 8000 2023-05-31T20:54:59.063 DEBUG:boto:Params: {} 2023-05-31T20:54:59.063 DEBUG:boto:Token: None 2023-05-31T20:54:59.063 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:54:59 GMT /pklxkd-59/obj51?versionId=null 2023-05-31T20:54:59.064 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:KtKRRZJTGUSUHYaFvxwNB4dOnkk= 2023-05-31T20:54:59.064 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:54:59 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:KtKRRZJTGUSUHYaFvxwNB4dOnkk=', 'Content-Length': '0'} 2023-05-31T20:54:59.066 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:43:08 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e0d2213003bce09e-006477b423-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:54:59 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:54:59.108 DEBUG:boto:path=/pklxkd-59/obj51 2023-05-31T20:54:59.109 DEBUG:boto:auth_path=/pklxkd-59/obj51 2023-05-31T20:54:59.109 DEBUG:boto:path=/pklxkd-59/obj51?versionId=null 2023-05-31T20:54:59.109 DEBUG:boto:auth_path=/pklxkd-59/obj51?versionId=null 2023-05-31T20:54:59.110 DEBUG:boto:Method: HEAD 2023-05-31T20:54:59.110 DEBUG:boto:Path: /pklxkd-59/obj51?versionId=null 2023-05-31T20:54:59.111 DEBUG:boto:Data: 2023-05-31T20:54:59.111 DEBUG:boto:Headers: {} 2023-05-31T20:54:59.111 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:54:59.112 DEBUG:boto:Port: 8001 2023-05-31T20:54:59.112 DEBUG:boto:Params: {} 2023-05-31T20:54:59.112 DEBUG:boto:Token: None 2023-05-31T20:54:59.113 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:54:59 GMT /pklxkd-59/obj51?versionId=null 2023-05-31T20:54:59.113 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:k8rKXZs8OM5Q6gO4dNdR6+qQTuM= 2023-05-31T20:54:59.114 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:54:59 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:k8rKXZs8OM5Q6gO4dNdR6+qQTuM=', 'Content-Length': '0'} 2023-05-31T20:54:59.116 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:43:08 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005a683d05795cd76b-006477b423-1e2e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:54:59 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:54:59.117 DEBUG:boto:path=/pklxkd-59/obj51 2023-05-31T20:54:59.117 DEBUG:boto:auth_path=/pklxkd-59/obj51 2023-05-31T20:54:59.117 DEBUG:boto:path=/pklxkd-59/obj51?versionId=null 2023-05-31T20:54:59.118 DEBUG:boto:auth_path=/pklxkd-59/obj51?versionId=null 2023-05-31T20:54:59.118 DEBUG:boto:Method: HEAD 2023-05-31T20:54:59.118 DEBUG:boto:Path: /pklxkd-59/obj51?versionId=null 2023-05-31T20:54:59.119 DEBUG:boto:Data: 2023-05-31T20:54:59.119 DEBUG:boto:Headers: {} 2023-05-31T20:54:59.119 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:54:59.120 DEBUG:boto:Port: 8000 2023-05-31T20:54:59.120 DEBUG:boto:Params: {} 2023-05-31T20:54:59.121 DEBUG:boto:Token: None 2023-05-31T20:54:59.121 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:54:59 GMT /pklxkd-59/obj51?versionId=null 2023-05-31T20:54:59.122 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:k8rKXZs8OM5Q6gO4dNdR6+qQTuM= 2023-05-31T20:54:59.122 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:54:59 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:k8rKXZs8OM5Q6gO4dNdR6+qQTuM=', 'Content-Length': '0'} 2023-05-31T20:54:59.124 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:43:08 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000270a7e5727afb57d-006477b423-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:54:59 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:54:59.125 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj51 2023-05-31T20:54:59.125 DEBUG:boto:path=/pklxkd-59/obj51 2023-05-31T20:54:59.125 DEBUG:boto:auth_path=/pklxkd-59/obj51 2023-05-31T20:54:59.126 DEBUG:boto:path=/pklxkd-59/obj51?versionId=null 2023-05-31T20:54:59.126 DEBUG:boto:auth_path=/pklxkd-59/obj51?versionId=null 2023-05-31T20:54:59.127 DEBUG:boto:Method: GET 2023-05-31T20:54:59.127 DEBUG:boto:Path: /pklxkd-59/obj51?versionId=null 2023-05-31T20:54:59.127 DEBUG:boto:Data: 2023-05-31T20:54:59.128 DEBUG:boto:Headers: {} 2023-05-31T20:54:59.128 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:54:59.129 DEBUG:boto:Port: 8001 2023-05-31T20:54:59.129 DEBUG:boto:Params: {} 2023-05-31T20:54:59.129 DEBUG:boto:Token: None 2023-05-31T20:54:59.130 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:54:59 GMT /pklxkd-59/obj51?versionId=null 2023-05-31T20:54:59.130 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:KtKRRZJTGUSUHYaFvxwNB4dOnkk= 2023-05-31T20:54:59.131 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:54:59 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:KtKRRZJTGUSUHYaFvxwNB4dOnkk=', 'Content-Length': '0'} 2023-05-31T20:54:59.132 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:43:08 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000000c6fe480da49b42-006477b423-1e2e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:54:59 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:54:59.133 DEBUG:boto:path=/pklxkd-59/obj51 2023-05-31T20:54:59.134 DEBUG:boto:auth_path=/pklxkd-59/obj51 2023-05-31T20:54:59.134 DEBUG:boto:path=/pklxkd-59/obj51?versionId=null 2023-05-31T20:54:59.134 DEBUG:boto:auth_path=/pklxkd-59/obj51?versionId=null 2023-05-31T20:54:59.135 DEBUG:boto:Method: GET 2023-05-31T20:54:59.135 DEBUG:boto:Path: /pklxkd-59/obj51?versionId=null 2023-05-31T20:54:59.135 DEBUG:boto:Data: 2023-05-31T20:54:59.136 DEBUG:boto:Headers: {} 2023-05-31T20:54:59.136 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:54:59.136 DEBUG:boto:Port: 8000 2023-05-31T20:54:59.137 DEBUG:boto:Params: {} 2023-05-31T20:54:59.137 DEBUG:boto:Token: None 2023-05-31T20:54:59.138 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:54:59 GMT /pklxkd-59/obj51?versionId=null 2023-05-31T20:54:59.138 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:KtKRRZJTGUSUHYaFvxwNB4dOnkk= 2023-05-31T20:54:59.138 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:54:59 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:KtKRRZJTGUSUHYaFvxwNB4dOnkk=', 'Content-Length': '0'} 2023-05-31T20:54:59.141 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:43:08 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008d58645716e7e2ac-006477b423-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:54:59 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:54:59.141 DEBUG:boto:path=/pklxkd-59/ 2023-05-31T20:54:59.142 DEBUG:boto:auth_path=/pklxkd-59/ 2023-05-31T20:54:59.142 DEBUG:boto:path=/pklxkd-59/?prefix=obj51 2023-05-31T20:54:59.143 DEBUG:boto:auth_path=/pklxkd-59/?prefix=obj51 2023-05-31T20:54:59.143 DEBUG:boto:Method: GET 2023-05-31T20:54:59.143 DEBUG:boto:Path: /pklxkd-59/?prefix=obj51 2023-05-31T20:54:59.144 DEBUG:boto:Data: 2023-05-31T20:54:59.144 DEBUG:boto:Headers: {} 2023-05-31T20:54:59.144 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:54:59.145 DEBUG:boto:Port: 8001 2023-05-31T20:54:59.145 DEBUG:boto:Params: {} 2023-05-31T20:54:59.146 DEBUG:boto:Token: None 2023-05-31T20:54:59.146 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:54:59 GMT /pklxkd-59/ 2023-05-31T20:54:59.147 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:tAvGnTMQPosMaETd601jyw6cOdE= 2023-05-31T20:54:59.147 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:54:59 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:tAvGnTMQPosMaETd601jyw6cOdE=', 'Content-Length': '0'} 2023-05-31T20:54:59.151 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000005cc66ea6dbfcd708-006477b423-1e2e-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:54:59 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:54:59.152 DEBUG:boto:b'pklxkd-59obj511000falseobj512023-05-31T20:43:08.011Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:54:59.153 DEBUG:boto:path=/pklxkd-59/ 2023-05-31T20:54:59.153 DEBUG:boto:auth_path=/pklxkd-59/ 2023-05-31T20:54:59.154 DEBUG:boto:path=/pklxkd-59/?prefix=obj51 2023-05-31T20:54:59.154 DEBUG:boto:auth_path=/pklxkd-59/?prefix=obj51 2023-05-31T20:54:59.154 DEBUG:boto:Method: GET 2023-05-31T20:54:59.155 DEBUG:boto:Path: /pklxkd-59/?prefix=obj51 2023-05-31T20:54:59.155 DEBUG:boto:Data: 2023-05-31T20:54:59.155 DEBUG:boto:Headers: {} 2023-05-31T20:54:59.156 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:54:59.156 DEBUG:boto:Port: 8000 2023-05-31T20:54:59.157 DEBUG:boto:Params: {} 2023-05-31T20:54:59.157 DEBUG:boto:Token: None 2023-05-31T20:54:59.158 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:54:59 GMT /pklxkd-59/ 2023-05-31T20:54:59.158 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:tAvGnTMQPosMaETd601jyw6cOdE= 2023-05-31T20:54:59.159 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:54:59 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:tAvGnTMQPosMaETd601jyw6cOdE=', 'Content-Length': '0'} 2023-05-31T20:54:59.162 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d9c44e6166ddfc8b-006477b423-1630-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:54:59 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:54:59.163 DEBUG:boto:b'pklxkd-59obj511000falseobj512023-05-31T20:43:08.011Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:54:59.164 DEBUG:boto:path=/pklxkd-59/obj51 2023-05-31T20:54:59.164 DEBUG:boto:auth_path=/pklxkd-59/obj51 2023-05-31T20:54:59.164 DEBUG:boto:Method: HEAD 2023-05-31T20:54:59.165 DEBUG:boto:Path: /pklxkd-59/obj51 2023-05-31T20:54:59.165 DEBUG:boto:Data: 2023-05-31T20:54:59.165 DEBUG:boto:Headers: {} 2023-05-31T20:54:59.166 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:54:59.166 DEBUG:boto:Port: 8001 2023-05-31T20:54:59.166 DEBUG:boto:Params: {} 2023-05-31T20:54:59.167 DEBUG:boto:Token: None 2023-05-31T20:54:59.167 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:54:59 GMT /pklxkd-59/obj51 2023-05-31T20:54:59.168 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:MfGHLGhNkyzAO9pr2fJqbvxgV2c= 2023-05-31T20:54:59.168 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:54:59 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:MfGHLGhNkyzAO9pr2fJqbvxgV2c=', 'Content-Length': '0'} 2023-05-31T20:54:59.170 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:43:08 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004a2d2cd80f02d11e-006477b423-1e2e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:54:59 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:54:59.171 DEBUG:boto:path=/pklxkd-59/obj51 2023-05-31T20:54:59.171 DEBUG:boto:auth_path=/pklxkd-59/obj51 2023-05-31T20:54:59.172 DEBUG:boto:Method: HEAD 2023-05-31T20:54:59.172 DEBUG:boto:Path: /pklxkd-59/obj51 2023-05-31T20:54:59.172 DEBUG:boto:Data: 2023-05-31T20:54:59.173 DEBUG:boto:Headers: {} 2023-05-31T20:54:59.173 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:54:59.173 DEBUG:boto:Port: 8000 2023-05-31T20:54:59.174 DEBUG:boto:Params: {} 2023-05-31T20:54:59.174 DEBUG:boto:Token: None 2023-05-31T20:54:59.174 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:54:59 GMT /pklxkd-59/obj51 2023-05-31T20:54:59.175 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:MfGHLGhNkyzAO9pr2fJqbvxgV2c= 2023-05-31T20:54:59.175 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:54:59 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:MfGHLGhNkyzAO9pr2fJqbvxgV2c=', 'Content-Length': '0'} 2023-05-31T20:54:59.177 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:43:08 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003536e6ad79ea3088-006477b423-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:54:59 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:54:59.178 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj51 2023-05-31T20:54:59.178 DEBUG:boto:path=/pklxkd-59/obj51 2023-05-31T20:54:59.179 DEBUG:boto:auth_path=/pklxkd-59/obj51 2023-05-31T20:54:59.179 DEBUG:boto:Method: GET 2023-05-31T20:54:59.179 DEBUG:boto:Path: /pklxkd-59/obj51 2023-05-31T20:54:59.180 DEBUG:boto:Data: 2023-05-31T20:54:59.180 DEBUG:boto:Headers: {} 2023-05-31T20:54:59.181 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:54:59.181 DEBUG:boto:Port: 8001 2023-05-31T20:54:59.182 DEBUG:boto:Params: {} 2023-05-31T20:54:59.182 DEBUG:boto:Token: None 2023-05-31T20:54:59.183 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:54:59 GMT /pklxkd-59/obj51 2023-05-31T20:54:59.183 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:JlRqBR8PXdRPOBgYJ+1O6WV+Km8= 2023-05-31T20:54:59.183 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:54:59 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:JlRqBR8PXdRPOBgYJ+1O6WV+Km8=', 'Content-Length': '0'} 2023-05-31T20:54:59.185 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:43:08 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c98e59309accf8c0-006477b423-1e2e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:54:59 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:54:59.186 DEBUG:boto:path=/pklxkd-59/obj51 2023-05-31T20:54:59.186 DEBUG:boto:auth_path=/pklxkd-59/obj51 2023-05-31T20:54:59.187 DEBUG:boto:Method: GET 2023-05-31T20:54:59.187 DEBUG:boto:Path: /pklxkd-59/obj51 2023-05-31T20:54:59.187 DEBUG:boto:Data: 2023-05-31T20:54:59.188 DEBUG:boto:Headers: {} 2023-05-31T20:54:59.188 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:54:59.188 DEBUG:boto:Port: 8000 2023-05-31T20:54:59.189 DEBUG:boto:Params: {} 2023-05-31T20:54:59.189 DEBUG:boto:Token: None 2023-05-31T20:54:59.190 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:54:59 GMT /pklxkd-59/obj51 2023-05-31T20:54:59.190 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:JlRqBR8PXdRPOBgYJ+1O6WV+Km8= 2023-05-31T20:54:59.191 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:54:59 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:JlRqBR8PXdRPOBgYJ+1O6WV+Km8=', 'Content-Length': '0'} 2023-05-31T20:54:59.193 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:43:08 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000df91d724c1439582-006477b423-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:54:59 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:54:59.236 DEBUG:boto:path=/pklxkd-59/ 2023-05-31T20:54:59.237 DEBUG:boto:auth_path=/pklxkd-59/ 2023-05-31T20:54:59.237 DEBUG:boto:path=/pklxkd-59/?prefix=obj51 2023-05-31T20:54:59.237 DEBUG:boto:auth_path=/pklxkd-59/?prefix=obj51 2023-05-31T20:54:59.238 DEBUG:boto:Method: GET 2023-05-31T20:54:59.238 DEBUG:boto:Path: /pklxkd-59/?prefix=obj51 2023-05-31T20:54:59.239 DEBUG:boto:Data: 2023-05-31T20:54:59.239 DEBUG:boto:Headers: {} 2023-05-31T20:54:59.240 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:54:59.240 DEBUG:boto:Port: 8001 2023-05-31T20:54:59.240 DEBUG:boto:Params: {} 2023-05-31T20:54:59.241 DEBUG:boto:Token: None 2023-05-31T20:54:59.241 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:54:59 GMT /pklxkd-59/ 2023-05-31T20:54:59.242 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:tAvGnTMQPosMaETd601jyw6cOdE= 2023-05-31T20:54:59.242 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:54:59 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:tAvGnTMQPosMaETd601jyw6cOdE=', 'Content-Length': '0'} 2023-05-31T20:54:59.244 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000011028972b711a5b1-006477b423-1e2e-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:54:59 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:54:59.245 DEBUG:boto:b'pklxkd-59obj511000falseobj512023-05-31T20:43:08.011Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:54:59.246 DEBUG:boto:path=/pklxkd-59/ 2023-05-31T20:54:59.246 DEBUG:boto:auth_path=/pklxkd-59/ 2023-05-31T20:54:59.247 DEBUG:boto:path=/pklxkd-59/?prefix=obj51 2023-05-31T20:54:59.247 DEBUG:boto:auth_path=/pklxkd-59/?prefix=obj51 2023-05-31T20:54:59.247 DEBUG:boto:Method: GET 2023-05-31T20:54:59.248 DEBUG:boto:Path: /pklxkd-59/?prefix=obj51 2023-05-31T20:54:59.249 DEBUG:boto:Data: 2023-05-31T20:54:59.249 DEBUG:boto:Headers: {} 2023-05-31T20:54:59.250 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:54:59.250 DEBUG:boto:Port: 8000 2023-05-31T20:54:59.250 DEBUG:boto:Params: {} 2023-05-31T20:54:59.251 DEBUG:boto:Token: None 2023-05-31T20:54:59.251 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:54:59 GMT /pklxkd-59/ 2023-05-31T20:54:59.252 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:tAvGnTMQPosMaETd601jyw6cOdE= 2023-05-31T20:54:59.252 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:54:59 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:tAvGnTMQPosMaETd601jyw6cOdE=', 'Content-Length': '0'} 2023-05-31T20:54:59.256 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000004fd74b07bfa1df28-006477b423-1630-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:54:59 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:54:59.257 DEBUG:boto:b'pklxkd-59obj511000falseobj512023-05-31T20:43:08.011Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:54:59.257 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj55 2023-05-31T20:54:59.258 DEBUG:boto:path=/pklxkd-59/obj55 2023-05-31T20:54:59.258 DEBUG:boto:auth_path=/pklxkd-59/obj55 2023-05-31T20:54:59.259 DEBUG:boto:path=/pklxkd-59/obj55?versionId=null 2023-05-31T20:54:59.259 DEBUG:boto:auth_path=/pklxkd-59/obj55?versionId=null 2023-05-31T20:54:59.260 DEBUG:boto:Method: GET 2023-05-31T20:54:59.260 DEBUG:boto:Path: /pklxkd-59/obj55?versionId=null 2023-05-31T20:54:59.261 DEBUG:boto:Data: 2023-05-31T20:54:59.261 DEBUG:boto:Headers: {} 2023-05-31T20:54:59.261 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:54:59.262 DEBUG:boto:Port: 8001 2023-05-31T20:54:59.262 DEBUG:boto:Params: {} 2023-05-31T20:54:59.263 DEBUG:boto:Token: None 2023-05-31T20:54:59.263 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:54:59 GMT /pklxkd-59/obj55?versionId=null 2023-05-31T20:54:59.264 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:Sy7SpTKJTAAt0SCZViSKeb6f1bY= 2023-05-31T20:54:59.264 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:54:59 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:Sy7SpTKJTAAt0SCZViSKeb6f1bY=', 'Content-Length': '0'} 2023-05-31T20:54:59.266 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:39:26 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000998f10e119006d17-006477b423-1e2e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:54:59 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:54:59.308 DEBUG:boto:path=/pklxkd-59/obj55 2023-05-31T20:54:59.309 DEBUG:boto:auth_path=/pklxkd-59/obj55 2023-05-31T20:54:59.309 DEBUG:boto:path=/pklxkd-59/obj55?versionId=null 2023-05-31T20:54:59.309 DEBUG:boto:auth_path=/pklxkd-59/obj55?versionId=null 2023-05-31T20:54:59.310 DEBUG:boto:Method: GET 2023-05-31T20:54:59.310 DEBUG:boto:Path: /pklxkd-59/obj55?versionId=null 2023-05-31T20:54:59.310 DEBUG:boto:Data: 2023-05-31T20:54:59.311 DEBUG:boto:Headers: {} 2023-05-31T20:54:59.311 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:54:59.312 DEBUG:boto:Port: 8000 2023-05-31T20:54:59.312 DEBUG:boto:Params: {} 2023-05-31T20:54:59.313 DEBUG:boto:Token: None 2023-05-31T20:54:59.313 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:54:59 GMT /pklxkd-59/obj55?versionId=null 2023-05-31T20:54:59.313 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:Sy7SpTKJTAAt0SCZViSKeb6f1bY= 2023-05-31T20:54:59.314 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:54:59 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:Sy7SpTKJTAAt0SCZViSKeb6f1bY=', 'Content-Length': '0'} 2023-05-31T20:54:59.316 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:39:26 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d66db5119f61b7fe-006477b423-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:54:59 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:54:59.317 DEBUG:boto:path=/pklxkd-59/obj55 2023-05-31T20:54:59.317 DEBUG:boto:auth_path=/pklxkd-59/obj55 2023-05-31T20:54:59.317 DEBUG:boto:path=/pklxkd-59/obj55?versionId=null 2023-05-31T20:54:59.318 DEBUG:boto:auth_path=/pklxkd-59/obj55?versionId=null 2023-05-31T20:54:59.318 DEBUG:boto:Method: HEAD 2023-05-31T20:54:59.318 DEBUG:boto:Path: /pklxkd-59/obj55?versionId=null 2023-05-31T20:54:59.319 DEBUG:boto:Data: 2023-05-31T20:54:59.319 DEBUG:boto:Headers: {} 2023-05-31T20:54:59.319 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:54:59.320 DEBUG:boto:Port: 8001 2023-05-31T20:54:59.320 DEBUG:boto:Params: {} 2023-05-31T20:54:59.320 DEBUG:boto:Token: None 2023-05-31T20:54:59.321 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:54:59 GMT /pklxkd-59/obj55?versionId=null 2023-05-31T20:54:59.321 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:SUlJzwFtr/57Ni0z6F/0YfLihRs= 2023-05-31T20:54:59.321 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:54:59 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:SUlJzwFtr/57Ni0z6F/0YfLihRs=', 'Content-Length': '0'} 2023-05-31T20:54:59.323 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:39:26 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e3d9f95ee7a12caa-006477b423-1e2e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:54:59 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:54:59.324 DEBUG:boto:path=/pklxkd-59/obj55 2023-05-31T20:54:59.324 DEBUG:boto:auth_path=/pklxkd-59/obj55 2023-05-31T20:54:59.325 DEBUG:boto:path=/pklxkd-59/obj55?versionId=null 2023-05-31T20:54:59.325 DEBUG:boto:auth_path=/pklxkd-59/obj55?versionId=null 2023-05-31T20:54:59.325 DEBUG:boto:Method: HEAD 2023-05-31T20:54:59.326 DEBUG:boto:Path: /pklxkd-59/obj55?versionId=null 2023-05-31T20:54:59.326 DEBUG:boto:Data: 2023-05-31T20:54:59.327 DEBUG:boto:Headers: {} 2023-05-31T20:54:59.327 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:54:59.328 DEBUG:boto:Port: 8000 2023-05-31T20:54:59.328 DEBUG:boto:Params: {} 2023-05-31T20:54:59.328 DEBUG:boto:Token: None 2023-05-31T20:54:59.329 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:54:59 GMT /pklxkd-59/obj55?versionId=null 2023-05-31T20:54:59.329 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:SUlJzwFtr/57Ni0z6F/0YfLihRs= 2023-05-31T20:54:59.330 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:54:59 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:SUlJzwFtr/57Ni0z6F/0YfLihRs=', 'Content-Length': '0'} 2023-05-31T20:54:59.331 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:39:26 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c5ac4d034f3a271d-006477b423-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:54:59 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:54:59.332 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj55 2023-05-31T20:54:59.332 DEBUG:boto:path=/pklxkd-59/obj55 2023-05-31T20:54:59.333 DEBUG:boto:auth_path=/pklxkd-59/obj55 2023-05-31T20:54:59.333 DEBUG:boto:path=/pklxkd-59/obj55?versionId=null 2023-05-31T20:54:59.334 DEBUG:boto:auth_path=/pklxkd-59/obj55?versionId=null 2023-05-31T20:54:59.334 DEBUG:boto:Method: GET 2023-05-31T20:54:59.334 DEBUG:boto:Path: /pklxkd-59/obj55?versionId=null 2023-05-31T20:54:59.335 DEBUG:boto:Data: 2023-05-31T20:54:59.335 DEBUG:boto:Headers: {} 2023-05-31T20:54:59.336 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:54:59.336 DEBUG:boto:Port: 8001 2023-05-31T20:54:59.336 DEBUG:boto:Params: {} 2023-05-31T20:54:59.337 DEBUG:boto:Token: None 2023-05-31T20:54:59.337 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:54:59 GMT /pklxkd-59/obj55?versionId=null 2023-05-31T20:54:59.338 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:Sy7SpTKJTAAt0SCZViSKeb6f1bY= 2023-05-31T20:54:59.338 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:54:59 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:Sy7SpTKJTAAt0SCZViSKeb6f1bY=', 'Content-Length': '0'} 2023-05-31T20:54:59.340 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:39:26 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001a860cbac343bbe9-006477b423-1e2e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:54:59 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:54:59.341 DEBUG:boto:path=/pklxkd-59/obj55 2023-05-31T20:54:59.341 DEBUG:boto:auth_path=/pklxkd-59/obj55 2023-05-31T20:54:59.341 DEBUG:boto:path=/pklxkd-59/obj55?versionId=null 2023-05-31T20:54:59.342 DEBUG:boto:auth_path=/pklxkd-59/obj55?versionId=null 2023-05-31T20:54:59.342 DEBUG:boto:Method: GET 2023-05-31T20:54:59.343 DEBUG:boto:Path: /pklxkd-59/obj55?versionId=null 2023-05-31T20:54:59.343 DEBUG:boto:Data: 2023-05-31T20:54:59.343 DEBUG:boto:Headers: {} 2023-05-31T20:54:59.344 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:54:59.344 DEBUG:boto:Port: 8000 2023-05-31T20:54:59.345 DEBUG:boto:Params: {} 2023-05-31T20:54:59.345 DEBUG:boto:Token: None 2023-05-31T20:54:59.346 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:54:59 GMT /pklxkd-59/obj55?versionId=null 2023-05-31T20:54:59.346 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:Sy7SpTKJTAAt0SCZViSKeb6f1bY= 2023-05-31T20:54:59.346 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:54:59 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:Sy7SpTKJTAAt0SCZViSKeb6f1bY=', 'Content-Length': '0'} 2023-05-31T20:54:59.348 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:39:26 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e70fc00759d4d62a-006477b423-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:54:59 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:54:59.349 DEBUG:boto:path=/pklxkd-59/ 2023-05-31T20:54:59.349 DEBUG:boto:auth_path=/pklxkd-59/ 2023-05-31T20:54:59.350 DEBUG:boto:path=/pklxkd-59/?prefix=obj55 2023-05-31T20:54:59.350 DEBUG:boto:auth_path=/pklxkd-59/?prefix=obj55 2023-05-31T20:54:59.350 DEBUG:boto:Method: GET 2023-05-31T20:54:59.351 DEBUG:boto:Path: /pklxkd-59/?prefix=obj55 2023-05-31T20:54:59.351 DEBUG:boto:Data: 2023-05-31T20:54:59.352 DEBUG:boto:Headers: {} 2023-05-31T20:54:59.352 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:54:59.353 DEBUG:boto:Port: 8001 2023-05-31T20:54:59.353 DEBUG:boto:Params: {} 2023-05-31T20:54:59.353 DEBUG:boto:Token: None 2023-05-31T20:54:59.354 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:54:59 GMT /pklxkd-59/ 2023-05-31T20:54:59.354 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:tAvGnTMQPosMaETd601jyw6cOdE= 2023-05-31T20:54:59.354 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:54:59 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:tAvGnTMQPosMaETd601jyw6cOdE=', 'Content-Length': '0'} 2023-05-31T20:54:59.357 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000aaeca43480711130-006477b423-1e2e-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:54:59 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:54:59.357 DEBUG:boto:b'pklxkd-59obj551000falseobj552023-05-31T20:39:26.630Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:54:59.358 DEBUG:boto:path=/pklxkd-59/ 2023-05-31T20:54:59.359 DEBUG:boto:auth_path=/pklxkd-59/ 2023-05-31T20:54:59.359 DEBUG:boto:path=/pklxkd-59/?prefix=obj55 2023-05-31T20:54:59.359 DEBUG:boto:auth_path=/pklxkd-59/?prefix=obj55 2023-05-31T20:54:59.360 DEBUG:boto:Method: GET 2023-05-31T20:54:59.360 DEBUG:boto:Path: /pklxkd-59/?prefix=obj55 2023-05-31T20:54:59.361 DEBUG:boto:Data: 2023-05-31T20:54:59.361 DEBUG:boto:Headers: {} 2023-05-31T20:54:59.361 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:54:59.362 DEBUG:boto:Port: 8000 2023-05-31T20:54:59.362 DEBUG:boto:Params: {} 2023-05-31T20:54:59.363 DEBUG:boto:Token: None 2023-05-31T20:54:59.363 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:54:59 GMT /pklxkd-59/ 2023-05-31T20:54:59.364 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:tAvGnTMQPosMaETd601jyw6cOdE= 2023-05-31T20:54:59.364 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:54:59 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:tAvGnTMQPosMaETd601jyw6cOdE=', 'Content-Length': '0'} 2023-05-31T20:54:59.368 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000601139051529b53a-006477b423-1630-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:54:59 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:54:59.368 DEBUG:boto:b'pklxkd-59obj551000falseobj552023-05-31T20:39:26.630Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:54:59.369 DEBUG:boto:path=/pklxkd-59/obj55 2023-05-31T20:54:59.369 DEBUG:boto:auth_path=/pklxkd-59/obj55 2023-05-31T20:54:59.370 DEBUG:boto:Method: HEAD 2023-05-31T20:54:59.370 DEBUG:boto:Path: /pklxkd-59/obj55 2023-05-31T20:54:59.370 DEBUG:boto:Data: 2023-05-31T20:54:59.371 DEBUG:boto:Headers: {} 2023-05-31T20:54:59.371 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:54:59.372 DEBUG:boto:Port: 8001 2023-05-31T20:54:59.372 DEBUG:boto:Params: {} 2023-05-31T20:54:59.373 DEBUG:boto:Token: None 2023-05-31T20:54:59.373 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:54:59 GMT /pklxkd-59/obj55 2023-05-31T20:54:59.373 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:eOVPGm9O4mvuS9CZweB+VA/Mi70= 2023-05-31T20:54:59.374 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:54:59 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:eOVPGm9O4mvuS9CZweB+VA/Mi70=', 'Content-Length': '0'} 2023-05-31T20:54:59.376 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:39:26 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ede738bf16e3ea5f-006477b423-1e2e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:54:59 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:54:59.376 DEBUG:boto:path=/pklxkd-59/obj55 2023-05-31T20:54:59.376 DEBUG:boto:auth_path=/pklxkd-59/obj55 2023-05-31T20:54:59.377 DEBUG:boto:Method: HEAD 2023-05-31T20:54:59.377 DEBUG:boto:Path: /pklxkd-59/obj55 2023-05-31T20:54:59.377 DEBUG:boto:Data: 2023-05-31T20:54:59.378 DEBUG:boto:Headers: {} 2023-05-31T20:54:59.378 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:54:59.379 DEBUG:boto:Port: 8000 2023-05-31T20:54:59.379 DEBUG:boto:Params: {} 2023-05-31T20:54:59.380 DEBUG:boto:Token: None 2023-05-31T20:54:59.380 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:54:59 GMT /pklxkd-59/obj55 2023-05-31T20:54:59.380 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:eOVPGm9O4mvuS9CZweB+VA/Mi70= 2023-05-31T20:54:59.381 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:54:59 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:eOVPGm9O4mvuS9CZweB+VA/Mi70=', 'Content-Length': '0'} 2023-05-31T20:54:59.382 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:39:26 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009fa0958150eeeb36-006477b423-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:54:59 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:54:59.383 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj55 2023-05-31T20:54:59.383 DEBUG:boto:path=/pklxkd-59/obj55 2023-05-31T20:54:59.384 DEBUG:boto:auth_path=/pklxkd-59/obj55 2023-05-31T20:54:59.384 DEBUG:boto:Method: GET 2023-05-31T20:54:59.385 DEBUG:boto:Path: /pklxkd-59/obj55 2023-05-31T20:54:59.385 DEBUG:boto:Data: 2023-05-31T20:54:59.386 DEBUG:boto:Headers: {} 2023-05-31T20:54:59.386 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:54:59.387 DEBUG:boto:Port: 8001 2023-05-31T20:54:59.387 DEBUG:boto:Params: {} 2023-05-31T20:54:59.387 DEBUG:boto:Token: None 2023-05-31T20:54:59.388 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:54:59 GMT /pklxkd-59/obj55 2023-05-31T20:54:59.388 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:Q8fBmDG4nrhCXFfjzy2+JEowr3E= 2023-05-31T20:54:59.389 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:54:59 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:Q8fBmDG4nrhCXFfjzy2+JEowr3E=', 'Content-Length': '0'} 2023-05-31T20:54:59.391 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:39:26 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000bc882962ac8c6e31-006477b423-1e2e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:54:59 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:54:59.436 DEBUG:boto:path=/pklxkd-59/obj55 2023-05-31T20:54:59.437 DEBUG:boto:auth_path=/pklxkd-59/obj55 2023-05-31T20:54:59.437 DEBUG:boto:Method: GET 2023-05-31T20:54:59.437 DEBUG:boto:Path: /pklxkd-59/obj55 2023-05-31T20:54:59.438 DEBUG:boto:Data: 2023-05-31T20:54:59.438 DEBUG:boto:Headers: {} 2023-05-31T20:54:59.439 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:54:59.439 DEBUG:boto:Port: 8000 2023-05-31T20:54:59.439 DEBUG:boto:Params: {} 2023-05-31T20:54:59.440 DEBUG:boto:Token: None 2023-05-31T20:54:59.441 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:54:59 GMT /pklxkd-59/obj55 2023-05-31T20:54:59.441 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:Q8fBmDG4nrhCXFfjzy2+JEowr3E= 2023-05-31T20:54:59.442 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:54:59 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:Q8fBmDG4nrhCXFfjzy2+JEowr3E=', 'Content-Length': '0'} 2023-05-31T20:54:59.444 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:39:26 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009a1e324a807912a9-006477b423-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:54:59 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:54:59.444 DEBUG:boto:path=/pklxkd-59/ 2023-05-31T20:54:59.445 DEBUG:boto:auth_path=/pklxkd-59/ 2023-05-31T20:54:59.445 DEBUG:boto:path=/pklxkd-59/?prefix=obj55 2023-05-31T20:54:59.445 DEBUG:boto:auth_path=/pklxkd-59/?prefix=obj55 2023-05-31T20:54:59.446 DEBUG:boto:Method: GET 2023-05-31T20:54:59.446 DEBUG:boto:Path: /pklxkd-59/?prefix=obj55 2023-05-31T20:54:59.446 DEBUG:boto:Data: 2023-05-31T20:54:59.447 DEBUG:boto:Headers: {} 2023-05-31T20:54:59.447 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:54:59.448 DEBUG:boto:Port: 8001 2023-05-31T20:54:59.448 DEBUG:boto:Params: {} 2023-05-31T20:54:59.448 DEBUG:boto:Token: None 2023-05-31T20:54:59.449 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:54:59 GMT /pklxkd-59/ 2023-05-31T20:54:59.449 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:tAvGnTMQPosMaETd601jyw6cOdE= 2023-05-31T20:54:59.450 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:54:59 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:tAvGnTMQPosMaETd601jyw6cOdE=', 'Content-Length': '0'} 2023-05-31T20:54:59.452 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d027b5fb7d9db333-006477b423-1e2e-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:54:59 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:54:59.453 DEBUG:boto:b'pklxkd-59obj551000falseobj552023-05-31T20:39:26.630Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:54:59.454 DEBUG:boto:path=/pklxkd-59/ 2023-05-31T20:54:59.455 DEBUG:boto:auth_path=/pklxkd-59/ 2023-05-31T20:54:59.456 DEBUG:boto:path=/pklxkd-59/?prefix=obj55 2023-05-31T20:54:59.456 DEBUG:boto:auth_path=/pklxkd-59/?prefix=obj55 2023-05-31T20:54:59.457 DEBUG:boto:Method: GET 2023-05-31T20:54:59.458 DEBUG:boto:Path: /pklxkd-59/?prefix=obj55 2023-05-31T20:54:59.459 DEBUG:boto:Data: 2023-05-31T20:54:59.459 DEBUG:boto:Headers: {} 2023-05-31T20:54:59.460 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:54:59.461 DEBUG:boto:Port: 8000 2023-05-31T20:54:59.461 DEBUG:boto:Params: {} 2023-05-31T20:54:59.462 DEBUG:boto:Token: None 2023-05-31T20:54:59.463 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:54:59 GMT /pklxkd-59/ 2023-05-31T20:54:59.464 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:tAvGnTMQPosMaETd601jyw6cOdE= 2023-05-31T20:54:59.465 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:54:59 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:tAvGnTMQPosMaETd601jyw6cOdE=', 'Content-Length': '0'} 2023-05-31T20:54:59.469 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000748504c03fe3a0e8-006477b423-1630-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:54:59 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:54:59.470 DEBUG:boto:b'pklxkd-59obj551000falseobj552023-05-31T20:39:26.630Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:54:59.471 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj57 2023-05-31T20:54:59.472 DEBUG:boto:path=/pklxkd-59/obj57 2023-05-31T20:54:59.472 DEBUG:boto:auth_path=/pklxkd-59/obj57 2023-05-31T20:54:59.473 DEBUG:boto:path=/pklxkd-59/obj57?versionId=null 2023-05-31T20:54:59.474 DEBUG:boto:auth_path=/pklxkd-59/obj57?versionId=null 2023-05-31T20:54:59.475 DEBUG:boto:Method: GET 2023-05-31T20:54:59.475 DEBUG:boto:Path: /pklxkd-59/obj57?versionId=null 2023-05-31T20:54:59.476 DEBUG:boto:Data: 2023-05-31T20:54:59.477 DEBUG:boto:Headers: {} 2023-05-31T20:54:59.478 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:54:59.478 DEBUG:boto:Port: 8001 2023-05-31T20:54:59.479 DEBUG:boto:Params: {} 2023-05-31T20:54:59.480 DEBUG:boto:Token: None 2023-05-31T20:54:59.481 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:54:59 GMT /pklxkd-59/obj57?versionId=null 2023-05-31T20:54:59.481 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:Fa+k4uaU6sJGuBLEz/Tf6a+xNVY= 2023-05-31T20:54:59.482 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:54:59 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:Fa+k4uaU6sJGuBLEz/Tf6a+xNVY=', 'Content-Length': '0'} 2023-05-31T20:54:59.485 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:44:50 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008d8414a17ebef7e2-006477b423-1e2e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:54:59 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:54:59.486 DEBUG:boto:path=/pklxkd-59/obj57 2023-05-31T20:54:59.487 DEBUG:boto:auth_path=/pklxkd-59/obj57 2023-05-31T20:54:59.487 DEBUG:boto:path=/pklxkd-59/obj57?versionId=null 2023-05-31T20:54:59.488 DEBUG:boto:auth_path=/pklxkd-59/obj57?versionId=null 2023-05-31T20:54:59.489 DEBUG:boto:Method: GET 2023-05-31T20:54:59.490 DEBUG:boto:Path: /pklxkd-59/obj57?versionId=null 2023-05-31T20:54:59.490 DEBUG:boto:Data: 2023-05-31T20:54:59.491 DEBUG:boto:Headers: {} 2023-05-31T20:54:59.492 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:54:59.493 DEBUG:boto:Port: 8000 2023-05-31T20:54:59.493 DEBUG:boto:Params: {} 2023-05-31T20:54:59.494 DEBUG:boto:Token: None 2023-05-31T20:54:59.495 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:54:59 GMT /pklxkd-59/obj57?versionId=null 2023-05-31T20:54:59.496 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:Fa+k4uaU6sJGuBLEz/Tf6a+xNVY= 2023-05-31T20:54:59.497 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:54:59 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:Fa+k4uaU6sJGuBLEz/Tf6a+xNVY=', 'Content-Length': '0'} 2023-05-31T20:54:59.498 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:44:50 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000029fc55aa42c76f80-006477b423-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:54:59 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:54:59.499 DEBUG:boto:path=/pklxkd-59/obj57 2023-05-31T20:54:59.499 DEBUG:boto:auth_path=/pklxkd-59/obj57 2023-05-31T20:54:59.500 DEBUG:boto:path=/pklxkd-59/obj57?versionId=null 2023-05-31T20:54:59.500 DEBUG:boto:auth_path=/pklxkd-59/obj57?versionId=null 2023-05-31T20:54:59.500 DEBUG:boto:Method: HEAD 2023-05-31T20:54:59.501 DEBUG:boto:Path: /pklxkd-59/obj57?versionId=null 2023-05-31T20:54:59.501 DEBUG:boto:Data: 2023-05-31T20:54:59.501 DEBUG:boto:Headers: {} 2023-05-31T20:54:59.502 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:54:59.502 DEBUG:boto:Port: 8001 2023-05-31T20:54:59.502 DEBUG:boto:Params: {} 2023-05-31T20:54:59.503 DEBUG:boto:Token: None 2023-05-31T20:54:59.503 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:54:59 GMT /pklxkd-59/obj57?versionId=null 2023-05-31T20:54:59.503 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:6+Yo6mgt1rtxaf872a7mPWbdzUM= 2023-05-31T20:54:59.504 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:54:59 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:6+Yo6mgt1rtxaf872a7mPWbdzUM=', 'Content-Length': '0'} 2023-05-31T20:54:59.506 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:44:50 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005c7549ebbebc87a7-006477b423-1e2e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:54:59 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:54:59.506 DEBUG:boto:path=/pklxkd-59/obj57 2023-05-31T20:54:59.507 DEBUG:boto:auth_path=/pklxkd-59/obj57 2023-05-31T20:54:59.507 DEBUG:boto:path=/pklxkd-59/obj57?versionId=null 2023-05-31T20:54:59.507 DEBUG:boto:auth_path=/pklxkd-59/obj57?versionId=null 2023-05-31T20:54:59.508 DEBUG:boto:Method: HEAD 2023-05-31T20:54:59.508 DEBUG:boto:Path: /pklxkd-59/obj57?versionId=null 2023-05-31T20:54:59.508 DEBUG:boto:Data: 2023-05-31T20:54:59.509 DEBUG:boto:Headers: {} 2023-05-31T20:54:59.509 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:54:59.509 DEBUG:boto:Port: 8000 2023-05-31T20:54:59.510 DEBUG:boto:Params: {} 2023-05-31T20:54:59.510 DEBUG:boto:Token: None 2023-05-31T20:54:59.510 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:54:59 GMT /pklxkd-59/obj57?versionId=null 2023-05-31T20:54:59.511 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:6+Yo6mgt1rtxaf872a7mPWbdzUM= 2023-05-31T20:54:59.512 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:54:59 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:6+Yo6mgt1rtxaf872a7mPWbdzUM=', 'Content-Length': '0'} 2023-05-31T20:54:59.514 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:44:50 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000fabd9a5980ce3db3-006477b423-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:54:59 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:54:59.514 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj57 2023-05-31T20:54:59.515 DEBUG:boto:path=/pklxkd-59/obj57 2023-05-31T20:54:59.515 DEBUG:boto:auth_path=/pklxkd-59/obj57 2023-05-31T20:54:59.515 DEBUG:boto:path=/pklxkd-59/obj57?versionId=null 2023-05-31T20:54:59.516 DEBUG:boto:auth_path=/pklxkd-59/obj57?versionId=null 2023-05-31T20:54:59.516 DEBUG:boto:Method: GET 2023-05-31T20:54:59.517 DEBUG:boto:Path: /pklxkd-59/obj57?versionId=null 2023-05-31T20:54:59.517 DEBUG:boto:Data: 2023-05-31T20:54:59.517 DEBUG:boto:Headers: {} 2023-05-31T20:54:59.518 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:54:59.518 DEBUG:boto:Port: 8001 2023-05-31T20:54:59.518 DEBUG:boto:Params: {} 2023-05-31T20:54:59.519 DEBUG:boto:Token: None 2023-05-31T20:54:59.519 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:54:59 GMT /pklxkd-59/obj57?versionId=null 2023-05-31T20:54:59.519 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:Fa+k4uaU6sJGuBLEz/Tf6a+xNVY= 2023-05-31T20:54:59.520 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:54:59 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:Fa+k4uaU6sJGuBLEz/Tf6a+xNVY=', 'Content-Length': '0'} 2023-05-31T20:54:59.522 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:44:50 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c1618d43517f7536-006477b423-1e2e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:54:59 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:54:59.564 DEBUG:boto:path=/pklxkd-59/obj57 2023-05-31T20:54:59.565 DEBUG:boto:auth_path=/pklxkd-59/obj57 2023-05-31T20:54:59.565 DEBUG:boto:path=/pklxkd-59/obj57?versionId=null 2023-05-31T20:54:59.566 DEBUG:boto:auth_path=/pklxkd-59/obj57?versionId=null 2023-05-31T20:54:59.566 DEBUG:boto:Method: GET 2023-05-31T20:54:59.566 DEBUG:boto:Path: /pklxkd-59/obj57?versionId=null 2023-05-31T20:54:59.567 DEBUG:boto:Data: 2023-05-31T20:54:59.567 DEBUG:boto:Headers: {} 2023-05-31T20:54:59.567 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:54:59.568 DEBUG:boto:Port: 8000 2023-05-31T20:54:59.568 DEBUG:boto:Params: {} 2023-05-31T20:54:59.568 DEBUG:boto:Token: None 2023-05-31T20:54:59.569 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:54:59 GMT /pklxkd-59/obj57?versionId=null 2023-05-31T20:54:59.569 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:Fa+k4uaU6sJGuBLEz/Tf6a+xNVY= 2023-05-31T20:54:59.570 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:54:59 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:Fa+k4uaU6sJGuBLEz/Tf6a+xNVY=', 'Content-Length': '0'} 2023-05-31T20:54:59.572 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:44:50 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000288d7d45200131d6-006477b423-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:54:59 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:54:59.572 DEBUG:boto:path=/pklxkd-59/ 2023-05-31T20:54:59.573 DEBUG:boto:auth_path=/pklxkd-59/ 2023-05-31T20:54:59.573 DEBUG:boto:path=/pklxkd-59/?prefix=obj57 2023-05-31T20:54:59.574 DEBUG:boto:auth_path=/pklxkd-59/?prefix=obj57 2023-05-31T20:54:59.574 DEBUG:boto:Method: GET 2023-05-31T20:54:59.574 DEBUG:boto:Path: /pklxkd-59/?prefix=obj57 2023-05-31T20:54:59.575 DEBUG:boto:Data: 2023-05-31T20:54:59.575 DEBUG:boto:Headers: {} 2023-05-31T20:54:59.575 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:54:59.576 DEBUG:boto:Port: 8001 2023-05-31T20:54:59.576 DEBUG:boto:Params: {} 2023-05-31T20:54:59.576 DEBUG:boto:Token: None 2023-05-31T20:54:59.577 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:54:59 GMT /pklxkd-59/ 2023-05-31T20:54:59.577 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:tAvGnTMQPosMaETd601jyw6cOdE= 2023-05-31T20:54:59.578 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:54:59 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:tAvGnTMQPosMaETd601jyw6cOdE=', 'Content-Length': '0'} 2023-05-31T20:54:59.580 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000055cd1d8dc36b93a7-006477b423-1e2e-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:54:59 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:54:59.581 DEBUG:boto:b'pklxkd-59obj571000falseobj572023-05-31T20:44:50.627Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:54:59.581 DEBUG:boto:path=/pklxkd-59/ 2023-05-31T20:54:59.582 DEBUG:boto:auth_path=/pklxkd-59/ 2023-05-31T20:54:59.582 DEBUG:boto:path=/pklxkd-59/?prefix=obj57 2023-05-31T20:54:59.582 DEBUG:boto:auth_path=/pklxkd-59/?prefix=obj57 2023-05-31T20:54:59.583 DEBUG:boto:Method: GET 2023-05-31T20:54:59.583 DEBUG:boto:Path: /pklxkd-59/?prefix=obj57 2023-05-31T20:54:59.583 DEBUG:boto:Data: 2023-05-31T20:54:59.584 DEBUG:boto:Headers: {} 2023-05-31T20:54:59.584 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:54:59.585 DEBUG:boto:Port: 8000 2023-05-31T20:54:59.585 DEBUG:boto:Params: {} 2023-05-31T20:54:59.586 DEBUG:boto:Token: None 2023-05-31T20:54:59.586 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:54:59 GMT /pklxkd-59/ 2023-05-31T20:54:59.586 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:tAvGnTMQPosMaETd601jyw6cOdE= 2023-05-31T20:54:59.587 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:54:59 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:tAvGnTMQPosMaETd601jyw6cOdE=', 'Content-Length': '0'} 2023-05-31T20:54:59.591 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000048f3c5a4d49ba079-006477b423-1630-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:54:59 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:54:59.591 DEBUG:boto:b'pklxkd-59obj571000falseobj572023-05-31T20:44:50.627Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:54:59.592 DEBUG:boto:path=/pklxkd-59/obj57 2023-05-31T20:54:59.592 DEBUG:boto:auth_path=/pklxkd-59/obj57 2023-05-31T20:54:59.593 DEBUG:boto:Method: HEAD 2023-05-31T20:54:59.593 DEBUG:boto:Path: /pklxkd-59/obj57 2023-05-31T20:54:59.593 DEBUG:boto:Data: 2023-05-31T20:54:59.594 DEBUG:boto:Headers: {} 2023-05-31T20:54:59.594 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:54:59.594 DEBUG:boto:Port: 8001 2023-05-31T20:54:59.595 DEBUG:boto:Params: {} 2023-05-31T20:54:59.595 DEBUG:boto:Token: None 2023-05-31T20:54:59.596 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:54:59 GMT /pklxkd-59/obj57 2023-05-31T20:54:59.596 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:EC1wf0jtoU1VG/oEMLbR2eJKS6Q= 2023-05-31T20:54:59.596 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:54:59 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:EC1wf0jtoU1VG/oEMLbR2eJKS6Q=', 'Content-Length': '0'} 2023-05-31T20:54:59.598 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:44:50 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004d66f74b90507450-006477b423-1e2e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:54:59 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:54:59.599 DEBUG:boto:path=/pklxkd-59/obj57 2023-05-31T20:54:59.599 DEBUG:boto:auth_path=/pklxkd-59/obj57 2023-05-31T20:54:59.600 DEBUG:boto:Method: HEAD 2023-05-31T20:54:59.600 DEBUG:boto:Path: /pklxkd-59/obj57 2023-05-31T20:54:59.600 DEBUG:boto:Data: 2023-05-31T20:54:59.601 DEBUG:boto:Headers: {} 2023-05-31T20:54:59.601 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:54:59.601 DEBUG:boto:Port: 8000 2023-05-31T20:54:59.602 DEBUG:boto:Params: {} 2023-05-31T20:54:59.602 DEBUG:boto:Token: None 2023-05-31T20:54:59.602 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:54:59 GMT /pklxkd-59/obj57 2023-05-31T20:54:59.603 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:EC1wf0jtoU1VG/oEMLbR2eJKS6Q= 2023-05-31T20:54:59.603 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:54:59 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:EC1wf0jtoU1VG/oEMLbR2eJKS6Q=', 'Content-Length': '0'} 2023-05-31T20:54:59.605 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:44:50 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004bf12b0edcc073d4-006477b423-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:54:59 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:54:59.605 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj57 2023-05-31T20:54:59.606 DEBUG:boto:path=/pklxkd-59/obj57 2023-05-31T20:54:59.606 DEBUG:boto:auth_path=/pklxkd-59/obj57 2023-05-31T20:54:59.607 DEBUG:boto:Method: GET 2023-05-31T20:54:59.607 DEBUG:boto:Path: /pklxkd-59/obj57 2023-05-31T20:54:59.607 DEBUG:boto:Data: 2023-05-31T20:54:59.608 DEBUG:boto:Headers: {} 2023-05-31T20:54:59.608 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:54:59.608 DEBUG:boto:Port: 8001 2023-05-31T20:54:59.609 DEBUG:boto:Params: {} 2023-05-31T20:54:59.609 DEBUG:boto:Token: None 2023-05-31T20:54:59.610 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:54:59 GMT /pklxkd-59/obj57 2023-05-31T20:54:59.610 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:urx1ZgMR+aW+sS5pfKg7Mr3bitk= 2023-05-31T20:54:59.610 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:54:59 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:urx1ZgMR+aW+sS5pfKg7Mr3bitk=', 'Content-Length': '0'} 2023-05-31T20:54:59.612 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:44:50 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c4982d0b330d0052-006477b423-1e2e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:54:59 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:54:59.613 DEBUG:boto:path=/pklxkd-59/obj57 2023-05-31T20:54:59.613 DEBUG:boto:auth_path=/pklxkd-59/obj57 2023-05-31T20:54:59.614 DEBUG:boto:Method: GET 2023-05-31T20:54:59.614 DEBUG:boto:Path: /pklxkd-59/obj57 2023-05-31T20:54:59.614 DEBUG:boto:Data: 2023-05-31T20:54:59.615 DEBUG:boto:Headers: {} 2023-05-31T20:54:59.615 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:54:59.615 DEBUG:boto:Port: 8000 2023-05-31T20:54:59.616 DEBUG:boto:Params: {} 2023-05-31T20:54:59.616 DEBUG:boto:Token: None 2023-05-31T20:54:59.617 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:54:59 GMT /pklxkd-59/obj57 2023-05-31T20:54:59.617 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:urx1ZgMR+aW+sS5pfKg7Mr3bitk= 2023-05-31T20:54:59.618 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:54:59 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:urx1ZgMR+aW+sS5pfKg7Mr3bitk=', 'Content-Length': '0'} 2023-05-31T20:54:59.619 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:44:50 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007e8616cdb06fcb92-006477b423-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:54:59 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:54:59.660 DEBUG:boto:path=/pklxkd-59/ 2023-05-31T20:54:59.661 DEBUG:boto:auth_path=/pklxkd-59/ 2023-05-31T20:54:59.661 DEBUG:boto:path=/pklxkd-59/?prefix=obj57 2023-05-31T20:54:59.661 DEBUG:boto:auth_path=/pklxkd-59/?prefix=obj57 2023-05-31T20:54:59.662 DEBUG:boto:Method: GET 2023-05-31T20:54:59.662 DEBUG:boto:Path: /pklxkd-59/?prefix=obj57 2023-05-31T20:54:59.662 DEBUG:boto:Data: 2023-05-31T20:54:59.663 DEBUG:boto:Headers: {} 2023-05-31T20:54:59.663 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:54:59.664 DEBUG:boto:Port: 8001 2023-05-31T20:54:59.664 DEBUG:boto:Params: {} 2023-05-31T20:54:59.664 DEBUG:boto:Token: None 2023-05-31T20:54:59.665 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:54:59 GMT /pklxkd-59/ 2023-05-31T20:54:59.665 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:tAvGnTMQPosMaETd601jyw6cOdE= 2023-05-31T20:54:59.666 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:54:59 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:tAvGnTMQPosMaETd601jyw6cOdE=', 'Content-Length': '0'} 2023-05-31T20:54:59.668 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b34e8ae885cbe5d1-006477b423-1e2e-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:54:59 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:54:59.668 DEBUG:boto:b'pklxkd-59obj571000falseobj572023-05-31T20:44:50.627Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:54:59.669 DEBUG:boto:path=/pklxkd-59/ 2023-05-31T20:54:59.669 DEBUG:boto:auth_path=/pklxkd-59/ 2023-05-31T20:54:59.670 DEBUG:boto:path=/pklxkd-59/?prefix=obj57 2023-05-31T20:54:59.670 DEBUG:boto:auth_path=/pklxkd-59/?prefix=obj57 2023-05-31T20:54:59.671 DEBUG:boto:Method: GET 2023-05-31T20:54:59.671 DEBUG:boto:Path: /pklxkd-59/?prefix=obj57 2023-05-31T20:54:59.672 DEBUG:boto:Data: 2023-05-31T20:54:59.672 DEBUG:boto:Headers: {} 2023-05-31T20:54:59.672 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:54:59.673 DEBUG:boto:Port: 8000 2023-05-31T20:54:59.673 DEBUG:boto:Params: {} 2023-05-31T20:54:59.673 DEBUG:boto:Token: None 2023-05-31T20:54:59.674 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:54:59 GMT /pklxkd-59/ 2023-05-31T20:54:59.674 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:tAvGnTMQPosMaETd601jyw6cOdE= 2023-05-31T20:54:59.675 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:54:59 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:tAvGnTMQPosMaETd601jyw6cOdE=', 'Content-Length': '0'} 2023-05-31T20:54:59.679 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000004d45d3d111948345-006477b423-1630-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:54:59 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:54:59.680 DEBUG:boto:b'pklxkd-59obj571000falseobj572023-05-31T20:44:50.627Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:54:59.680 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj58 2023-05-31T20:54:59.681 DEBUG:boto:path=/pklxkd-59/obj58 2023-05-31T20:54:59.681 DEBUG:boto:auth_path=/pklxkd-59/obj58 2023-05-31T20:54:59.681 DEBUG:boto:path=/pklxkd-59/obj58?versionId=null 2023-05-31T20:54:59.682 DEBUG:boto:auth_path=/pklxkd-59/obj58?versionId=null 2023-05-31T20:54:59.682 DEBUG:boto:Method: GET 2023-05-31T20:54:59.682 DEBUG:boto:Path: /pklxkd-59/obj58?versionId=null 2023-05-31T20:54:59.683 DEBUG:boto:Data: 2023-05-31T20:54:59.683 DEBUG:boto:Headers: {} 2023-05-31T20:54:59.684 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:54:59.684 DEBUG:boto:Port: 8001 2023-05-31T20:54:59.684 DEBUG:boto:Params: {} 2023-05-31T20:54:59.685 DEBUG:boto:Token: None 2023-05-31T20:54:59.685 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:54:59 GMT /pklxkd-59/obj58?versionId=null 2023-05-31T20:54:59.686 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:3YKWCPuaSHR7AKtd1Jw5KZQmmi8= 2023-05-31T20:54:59.686 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:54:59 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:3YKWCPuaSHR7AKtd1Jw5KZQmmi8=', 'Content-Length': '0'} 2023-05-31T20:54:59.688 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:48:12 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000fab2d3eb7a2697dc-006477b423-1e2e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:54:59 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:54:59.689 DEBUG:boto:path=/pklxkd-59/obj58 2023-05-31T20:54:59.689 DEBUG:boto:auth_path=/pklxkd-59/obj58 2023-05-31T20:54:59.690 DEBUG:boto:path=/pklxkd-59/obj58?versionId=null 2023-05-31T20:54:59.690 DEBUG:boto:auth_path=/pklxkd-59/obj58?versionId=null 2023-05-31T20:54:59.690 DEBUG:boto:Method: GET 2023-05-31T20:54:59.691 DEBUG:boto:Path: /pklxkd-59/obj58?versionId=null 2023-05-31T20:54:59.691 DEBUG:boto:Data: 2023-05-31T20:54:59.691 DEBUG:boto:Headers: {} 2023-05-31T20:54:59.692 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:54:59.692 DEBUG:boto:Port: 8000 2023-05-31T20:54:59.692 DEBUG:boto:Params: {} 2023-05-31T20:54:59.693 DEBUG:boto:Token: None 2023-05-31T20:54:59.693 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:54:59 GMT /pklxkd-59/obj58?versionId=null 2023-05-31T20:54:59.694 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:3YKWCPuaSHR7AKtd1Jw5KZQmmi8= 2023-05-31T20:54:59.694 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:54:59 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:3YKWCPuaSHR7AKtd1Jw5KZQmmi8=', 'Content-Length': '0'} 2023-05-31T20:54:59.696 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:48:12 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000046bcb7c00495aca7-006477b423-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:54:59 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:54:59.696 DEBUG:boto:path=/pklxkd-59/obj58 2023-05-31T20:54:59.697 DEBUG:boto:auth_path=/pklxkd-59/obj58 2023-05-31T20:54:59.697 DEBUG:boto:path=/pklxkd-59/obj58?versionId=null 2023-05-31T20:54:59.698 DEBUG:boto:auth_path=/pklxkd-59/obj58?versionId=null 2023-05-31T20:54:59.698 DEBUG:boto:Method: HEAD 2023-05-31T20:54:59.698 DEBUG:boto:Path: /pklxkd-59/obj58?versionId=null 2023-05-31T20:54:59.699 DEBUG:boto:Data: 2023-05-31T20:54:59.699 DEBUG:boto:Headers: {} 2023-05-31T20:54:59.699 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:54:59.700 DEBUG:boto:Port: 8001 2023-05-31T20:54:59.700 DEBUG:boto:Params: {} 2023-05-31T20:54:59.700 DEBUG:boto:Token: None 2023-05-31T20:54:59.701 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:54:59 GMT /pklxkd-59/obj58?versionId=null 2023-05-31T20:54:59.701 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:cqZctbOV1TEWwD804aUKFJFeoCM= 2023-05-31T20:54:59.701 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:54:59 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:cqZctbOV1TEWwD804aUKFJFeoCM=', 'Content-Length': '0'} 2023-05-31T20:54:59.703 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:48:12 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009a3e87eb15a2f4af-006477b423-1e2e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:54:59 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:54:59.704 DEBUG:boto:path=/pklxkd-59/obj58 2023-05-31T20:54:59.704 DEBUG:boto:auth_path=/pklxkd-59/obj58 2023-05-31T20:54:59.704 DEBUG:boto:path=/pklxkd-59/obj58?versionId=null 2023-05-31T20:54:59.705 DEBUG:boto:auth_path=/pklxkd-59/obj58?versionId=null 2023-05-31T20:54:59.705 DEBUG:boto:Method: HEAD 2023-05-31T20:54:59.705 DEBUG:boto:Path: /pklxkd-59/obj58?versionId=null 2023-05-31T20:54:59.706 DEBUG:boto:Data: 2023-05-31T20:54:59.706 DEBUG:boto:Headers: {} 2023-05-31T20:54:59.706 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:54:59.707 DEBUG:boto:Port: 8000 2023-05-31T20:54:59.707 DEBUG:boto:Params: {} 2023-05-31T20:54:59.708 DEBUG:boto:Token: None 2023-05-31T20:54:59.708 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:54:59 GMT /pklxkd-59/obj58?versionId=null 2023-05-31T20:54:59.708 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:cqZctbOV1TEWwD804aUKFJFeoCM= 2023-05-31T20:54:59.709 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:54:59 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:cqZctbOV1TEWwD804aUKFJFeoCM=', 'Content-Length': '0'} 2023-05-31T20:54:59.710 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:48:12 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000cc57459939dacb50-006477b423-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:54:59 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:54:59.711 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj58 2023-05-31T20:54:59.711 DEBUG:boto:path=/pklxkd-59/obj58 2023-05-31T20:54:59.712 DEBUG:boto:auth_path=/pklxkd-59/obj58 2023-05-31T20:54:59.712 DEBUG:boto:path=/pklxkd-59/obj58?versionId=null 2023-05-31T20:54:59.712 DEBUG:boto:auth_path=/pklxkd-59/obj58?versionId=null 2023-05-31T20:54:59.713 DEBUG:boto:Method: GET 2023-05-31T20:54:59.713 DEBUG:boto:Path: /pklxkd-59/obj58?versionId=null 2023-05-31T20:54:59.713 DEBUG:boto:Data: 2023-05-31T20:54:59.714 DEBUG:boto:Headers: {} 2023-05-31T20:54:59.714 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:54:59.715 DEBUG:boto:Port: 8001 2023-05-31T20:54:59.715 DEBUG:boto:Params: {} 2023-05-31T20:54:59.715 DEBUG:boto:Token: None 2023-05-31T20:54:59.716 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:54:59 GMT /pklxkd-59/obj58?versionId=null 2023-05-31T20:54:59.716 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:3YKWCPuaSHR7AKtd1Jw5KZQmmi8= 2023-05-31T20:54:59.717 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:54:59 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:3YKWCPuaSHR7AKtd1Jw5KZQmmi8=', 'Content-Length': '0'} 2023-05-31T20:54:59.718 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:48:12 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000093a41880a5fb39a0-006477b423-1e2e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:54:59 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:54:59.760 DEBUG:boto:path=/pklxkd-59/obj58 2023-05-31T20:54:59.761 DEBUG:boto:auth_path=/pklxkd-59/obj58 2023-05-31T20:54:59.762 DEBUG:boto:path=/pklxkd-59/obj58?versionId=null 2023-05-31T20:54:59.763 DEBUG:boto:auth_path=/pklxkd-59/obj58?versionId=null 2023-05-31T20:54:59.764 DEBUG:boto:Method: GET 2023-05-31T20:54:59.765 DEBUG:boto:Path: /pklxkd-59/obj58?versionId=null 2023-05-31T20:54:59.766 DEBUG:boto:Data: 2023-05-31T20:54:59.767 DEBUG:boto:Headers: {} 2023-05-31T20:54:59.768 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:54:59.769 DEBUG:boto:Port: 8000 2023-05-31T20:54:59.769 DEBUG:boto:Params: {} 2023-05-31T20:54:59.770 DEBUG:boto:Token: None 2023-05-31T20:54:59.770 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:54:59 GMT /pklxkd-59/obj58?versionId=null 2023-05-31T20:54:59.771 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:3YKWCPuaSHR7AKtd1Jw5KZQmmi8= 2023-05-31T20:54:59.772 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:54:59 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:3YKWCPuaSHR7AKtd1Jw5KZQmmi8=', 'Content-Length': '0'} 2023-05-31T20:54:59.774 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:48:12 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000971b4d52a6783988-006477b423-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:54:59 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:54:59.775 DEBUG:boto:path=/pklxkd-59/ 2023-05-31T20:54:59.775 DEBUG:boto:auth_path=/pklxkd-59/ 2023-05-31T20:54:59.776 DEBUG:boto:path=/pklxkd-59/?prefix=obj58 2023-05-31T20:54:59.776 DEBUG:boto:auth_path=/pklxkd-59/?prefix=obj58 2023-05-31T20:54:59.777 DEBUG:boto:Method: GET 2023-05-31T20:54:59.777 DEBUG:boto:Path: /pklxkd-59/?prefix=obj58 2023-05-31T20:54:59.777 DEBUG:boto:Data: 2023-05-31T20:54:59.778 DEBUG:boto:Headers: {} 2023-05-31T20:54:59.778 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:54:59.778 DEBUG:boto:Port: 8001 2023-05-31T20:54:59.779 DEBUG:boto:Params: {} 2023-05-31T20:54:59.779 DEBUG:boto:Token: None 2023-05-31T20:54:59.780 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:54:59 GMT /pklxkd-59/ 2023-05-31T20:54:59.780 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:tAvGnTMQPosMaETd601jyw6cOdE= 2023-05-31T20:54:59.781 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:54:59 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:tAvGnTMQPosMaETd601jyw6cOdE=', 'Content-Length': '0'} 2023-05-31T20:54:59.783 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000007c2d6da5a43f6b9a-006477b423-1e2e-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:54:59 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:54:59.784 DEBUG:boto:b'pklxkd-59obj581000falseobj582023-05-31T20:48:12.976Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:54:59.785 DEBUG:boto:path=/pklxkd-59/ 2023-05-31T20:54:59.785 DEBUG:boto:auth_path=/pklxkd-59/ 2023-05-31T20:54:59.785 DEBUG:boto:path=/pklxkd-59/?prefix=obj58 2023-05-31T20:54:59.786 DEBUG:boto:auth_path=/pklxkd-59/?prefix=obj58 2023-05-31T20:54:59.786 DEBUG:boto:Method: GET 2023-05-31T20:54:59.786 DEBUG:boto:Path: /pklxkd-59/?prefix=obj58 2023-05-31T20:54:59.787 DEBUG:boto:Data: 2023-05-31T20:54:59.787 DEBUG:boto:Headers: {} 2023-05-31T20:54:59.787 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:54:59.788 DEBUG:boto:Port: 8000 2023-05-31T20:54:59.788 DEBUG:boto:Params: {} 2023-05-31T20:54:59.789 DEBUG:boto:Token: None 2023-05-31T20:54:59.789 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:54:59 GMT /pklxkd-59/ 2023-05-31T20:54:59.789 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:tAvGnTMQPosMaETd601jyw6cOdE= 2023-05-31T20:54:59.790 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:54:59 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:tAvGnTMQPosMaETd601jyw6cOdE=', 'Content-Length': '0'} 2023-05-31T20:54:59.794 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000777b9fa1cc6dcda1-006477b423-1630-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:54:59 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:54:59.795 DEBUG:boto:b'pklxkd-59obj581000falseobj582023-05-31T20:48:12.976Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:54:59.795 DEBUG:boto:path=/pklxkd-59/obj58 2023-05-31T20:54:59.796 DEBUG:boto:auth_path=/pklxkd-59/obj58 2023-05-31T20:54:59.796 DEBUG:boto:Method: HEAD 2023-05-31T20:54:59.796 DEBUG:boto:Path: /pklxkd-59/obj58 2023-05-31T20:54:59.797 DEBUG:boto:Data: 2023-05-31T20:54:59.797 DEBUG:boto:Headers: {} 2023-05-31T20:54:59.797 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:54:59.798 DEBUG:boto:Port: 8001 2023-05-31T20:54:59.798 DEBUG:boto:Params: {} 2023-05-31T20:54:59.798 DEBUG:boto:Token: None 2023-05-31T20:54:59.799 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:54:59 GMT /pklxkd-59/obj58 2023-05-31T20:54:59.799 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:YdbT1zQtmzOBdkVSdmdOeotydlE= 2023-05-31T20:54:59.800 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:54:59 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:YdbT1zQtmzOBdkVSdmdOeotydlE=', 'Content-Length': '0'} 2023-05-31T20:54:59.801 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:48:12 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000026641f1d54dd800e-006477b423-1e2e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:54:59 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:54:59.802 DEBUG:boto:path=/pklxkd-59/obj58 2023-05-31T20:54:59.802 DEBUG:boto:auth_path=/pklxkd-59/obj58 2023-05-31T20:54:59.803 DEBUG:boto:Method: HEAD 2023-05-31T20:54:59.803 DEBUG:boto:Path: /pklxkd-59/obj58 2023-05-31T20:54:59.803 DEBUG:boto:Data: 2023-05-31T20:54:59.804 DEBUG:boto:Headers: {} 2023-05-31T20:54:59.804 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:54:59.805 DEBUG:boto:Port: 8000 2023-05-31T20:54:59.805 DEBUG:boto:Params: {} 2023-05-31T20:54:59.805 DEBUG:boto:Token: None 2023-05-31T20:54:59.806 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:54:59 GMT /pklxkd-59/obj58 2023-05-31T20:54:59.806 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:YdbT1zQtmzOBdkVSdmdOeotydlE= 2023-05-31T20:54:59.806 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:54:59 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:YdbT1zQtmzOBdkVSdmdOeotydlE=', 'Content-Length': '0'} 2023-05-31T20:54:59.808 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:48:12 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ff9755a6d2970f94-006477b423-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:54:59 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:54:59.809 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj58 2023-05-31T20:54:59.809 DEBUG:boto:path=/pklxkd-59/obj58 2023-05-31T20:54:59.809 DEBUG:boto:auth_path=/pklxkd-59/obj58 2023-05-31T20:54:59.810 DEBUG:boto:Method: GET 2023-05-31T20:54:59.810 DEBUG:boto:Path: /pklxkd-59/obj58 2023-05-31T20:54:59.811 DEBUG:boto:Data: 2023-05-31T20:54:59.811 DEBUG:boto:Headers: {} 2023-05-31T20:54:59.811 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:54:59.812 DEBUG:boto:Port: 8001 2023-05-31T20:54:59.812 DEBUG:boto:Params: {} 2023-05-31T20:54:59.813 DEBUG:boto:Token: None 2023-05-31T20:54:59.813 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:54:59 GMT /pklxkd-59/obj58 2023-05-31T20:54:59.813 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:KYhy+runv0P1if0evEzab27TZkg= 2023-05-31T20:54:59.814 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:54:59 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:KYhy+runv0P1if0evEzab27TZkg=', 'Content-Length': '0'} 2023-05-31T20:54:59.815 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:48:12 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e5e54bec4a25742a-006477b423-1e2e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:54:59 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:54:59.816 DEBUG:boto:path=/pklxkd-59/obj58 2023-05-31T20:54:59.817 DEBUG:boto:auth_path=/pklxkd-59/obj58 2023-05-31T20:54:59.817 DEBUG:boto:Method: GET 2023-05-31T20:54:59.817 DEBUG:boto:Path: /pklxkd-59/obj58 2023-05-31T20:54:59.818 DEBUG:boto:Data: 2023-05-31T20:54:59.818 DEBUG:boto:Headers: {} 2023-05-31T20:54:59.818 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:54:59.819 DEBUG:boto:Port: 8000 2023-05-31T20:54:59.819 DEBUG:boto:Params: {} 2023-05-31T20:54:59.819 DEBUG:boto:Token: None 2023-05-31T20:54:59.820 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:54:59 GMT /pklxkd-59/obj58 2023-05-31T20:54:59.820 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:KYhy+runv0P1if0evEzab27TZkg= 2023-05-31T20:54:59.821 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:54:59 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:KYhy+runv0P1if0evEzab27TZkg=', 'Content-Length': '0'} 2023-05-31T20:54:59.823 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:48:12 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000048d1de79f1840e8e-006477b423-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:54:59 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:54:59.864 DEBUG:boto:path=/pklxkd-59/ 2023-05-31T20:54:59.865 DEBUG:boto:auth_path=/pklxkd-59/ 2023-05-31T20:54:59.865 DEBUG:boto:path=/pklxkd-59/?prefix=obj58 2023-05-31T20:54:59.865 DEBUG:boto:auth_path=/pklxkd-59/?prefix=obj58 2023-05-31T20:54:59.866 DEBUG:boto:Method: GET 2023-05-31T20:54:59.866 DEBUG:boto:Path: /pklxkd-59/?prefix=obj58 2023-05-31T20:54:59.866 DEBUG:boto:Data: 2023-05-31T20:54:59.867 DEBUG:boto:Headers: {} 2023-05-31T20:54:59.867 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:54:59.867 DEBUG:boto:Port: 8001 2023-05-31T20:54:59.868 DEBUG:boto:Params: {} 2023-05-31T20:54:59.868 DEBUG:boto:Token: None 2023-05-31T20:54:59.869 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:54:59 GMT /pklxkd-59/ 2023-05-31T20:54:59.869 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:tAvGnTMQPosMaETd601jyw6cOdE= 2023-05-31T20:54:59.870 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:54:59 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:tAvGnTMQPosMaETd601jyw6cOdE=', 'Content-Length': '0'} 2023-05-31T20:54:59.872 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c9626d0d9444cd75-006477b423-1e2e-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:54:59 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:54:59.873 DEBUG:boto:b'pklxkd-59obj581000falseobj582023-05-31T20:48:12.976Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:54:59.873 DEBUG:boto:path=/pklxkd-59/ 2023-05-31T20:54:59.874 DEBUG:boto:auth_path=/pklxkd-59/ 2023-05-31T20:54:59.874 DEBUG:boto:path=/pklxkd-59/?prefix=obj58 2023-05-31T20:54:59.874 DEBUG:boto:auth_path=/pklxkd-59/?prefix=obj58 2023-05-31T20:54:59.875 DEBUG:boto:Method: GET 2023-05-31T20:54:59.875 DEBUG:boto:Path: /pklxkd-59/?prefix=obj58 2023-05-31T20:54:59.875 DEBUG:boto:Data: 2023-05-31T20:54:59.876 DEBUG:boto:Headers: {} 2023-05-31T20:54:59.876 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:54:59.877 DEBUG:boto:Port: 8000 2023-05-31T20:54:59.877 DEBUG:boto:Params: {} 2023-05-31T20:54:59.877 DEBUG:boto:Token: None 2023-05-31T20:54:59.878 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:54:59 GMT /pklxkd-59/ 2023-05-31T20:54:59.878 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:tAvGnTMQPosMaETd601jyw6cOdE= 2023-05-31T20:54:59.879 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:54:59 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:tAvGnTMQPosMaETd601jyw6cOdE=', 'Content-Length': '0'} 2023-05-31T20:54:59.883 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a56b310c2b180855-006477b423-1630-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:54:59 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:54:59.883 DEBUG:boto:b'pklxkd-59obj581000falseobj582023-05-31T20:48:12.976Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:54:59.884 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj62 2023-05-31T20:54:59.884 DEBUG:boto:path=/pklxkd-59/obj62 2023-05-31T20:54:59.885 DEBUG:boto:auth_path=/pklxkd-59/obj62 2023-05-31T20:54:59.885 DEBUG:boto:path=/pklxkd-59/obj62?versionId=null 2023-05-31T20:54:59.885 DEBUG:boto:auth_path=/pklxkd-59/obj62?versionId=null 2023-05-31T20:54:59.886 DEBUG:boto:Method: GET 2023-05-31T20:54:59.886 DEBUG:boto:Path: /pklxkd-59/obj62?versionId=null 2023-05-31T20:54:59.887 DEBUG:boto:Data: 2023-05-31T20:54:59.887 DEBUG:boto:Headers: {} 2023-05-31T20:54:59.887 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:54:59.888 DEBUG:boto:Port: 8001 2023-05-31T20:54:59.888 DEBUG:boto:Params: {} 2023-05-31T20:54:59.889 DEBUG:boto:Token: None 2023-05-31T20:54:59.889 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:54:59 GMT /pklxkd-59/obj62?versionId=null 2023-05-31T20:54:59.889 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:3rxOpEiNIps7y75ieoSyWP5nfkA= 2023-05-31T20:54:59.890 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:54:59 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:3rxOpEiNIps7y75ieoSyWP5nfkA=', 'Content-Length': '0'} 2023-05-31T20:54:59.892 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:50:33 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000230374e3e3d0db78-006477b423-1e2e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:54:59 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:54:59.893 DEBUG:boto:path=/pklxkd-59/obj62 2023-05-31T20:54:59.893 DEBUG:boto:auth_path=/pklxkd-59/obj62 2023-05-31T20:54:59.893 DEBUG:boto:path=/pklxkd-59/obj62?versionId=null 2023-05-31T20:54:59.894 DEBUG:boto:auth_path=/pklxkd-59/obj62?versionId=null 2023-05-31T20:54:59.894 DEBUG:boto:Method: GET 2023-05-31T20:54:59.894 DEBUG:boto:Path: /pklxkd-59/obj62?versionId=null 2023-05-31T20:54:59.895 DEBUG:boto:Data: 2023-05-31T20:54:59.895 DEBUG:boto:Headers: {} 2023-05-31T20:54:59.895 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:54:59.896 DEBUG:boto:Port: 8000 2023-05-31T20:54:59.896 DEBUG:boto:Params: {} 2023-05-31T20:54:59.897 DEBUG:boto:Token: None 2023-05-31T20:54:59.897 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:54:59 GMT /pklxkd-59/obj62?versionId=null 2023-05-31T20:54:59.897 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:3rxOpEiNIps7y75ieoSyWP5nfkA= 2023-05-31T20:54:59.898 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:54:59 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:3rxOpEiNIps7y75ieoSyWP5nfkA=', 'Content-Length': '0'} 2023-05-31T20:54:59.900 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:50:33 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009b23070b44bd567f-006477b423-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:54:59 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:54:59.900 DEBUG:boto:path=/pklxkd-59/obj62 2023-05-31T20:54:59.901 DEBUG:boto:auth_path=/pklxkd-59/obj62 2023-05-31T20:54:59.901 DEBUG:boto:path=/pklxkd-59/obj62?versionId=null 2023-05-31T20:54:59.901 DEBUG:boto:auth_path=/pklxkd-59/obj62?versionId=null 2023-05-31T20:54:59.902 DEBUG:boto:Method: HEAD 2023-05-31T20:54:59.902 DEBUG:boto:Path: /pklxkd-59/obj62?versionId=null 2023-05-31T20:54:59.903 DEBUG:boto:Data: 2023-05-31T20:54:59.903 DEBUG:boto:Headers: {} 2023-05-31T20:54:59.903 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:54:59.904 DEBUG:boto:Port: 8001 2023-05-31T20:54:59.904 DEBUG:boto:Params: {} 2023-05-31T20:54:59.904 DEBUG:boto:Token: None 2023-05-31T20:54:59.905 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:54:59 GMT /pklxkd-59/obj62?versionId=null 2023-05-31T20:54:59.905 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:lBRVBD1YUjfISYqo4psRiHJQFWY= 2023-05-31T20:54:59.905 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:54:59 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:lBRVBD1YUjfISYqo4psRiHJQFWY=', 'Content-Length': '0'} 2023-05-31T20:54:59.907 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:50:33 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008cc93addab4cb87a-006477b423-1e2e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:54:59 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:54:59.908 DEBUG:boto:path=/pklxkd-59/obj62 2023-05-31T20:54:59.908 DEBUG:boto:auth_path=/pklxkd-59/obj62 2023-05-31T20:54:59.909 DEBUG:boto:path=/pklxkd-59/obj62?versionId=null 2023-05-31T20:54:59.909 DEBUG:boto:auth_path=/pklxkd-59/obj62?versionId=null 2023-05-31T20:54:59.909 DEBUG:boto:Method: HEAD 2023-05-31T20:54:59.910 DEBUG:boto:Path: /pklxkd-59/obj62?versionId=null 2023-05-31T20:54:59.910 DEBUG:boto:Data: 2023-05-31T20:54:59.910 DEBUG:boto:Headers: {} 2023-05-31T20:54:59.911 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:54:59.911 DEBUG:boto:Port: 8000 2023-05-31T20:54:59.911 DEBUG:boto:Params: {} 2023-05-31T20:54:59.912 DEBUG:boto:Token: None 2023-05-31T20:54:59.912 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:54:59 GMT /pklxkd-59/obj62?versionId=null 2023-05-31T20:54:59.912 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:lBRVBD1YUjfISYqo4psRiHJQFWY= 2023-05-31T20:54:59.913 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:54:59 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:lBRVBD1YUjfISYqo4psRiHJQFWY=', 'Content-Length': '0'} 2023-05-31T20:54:59.915 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:50:33 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ac8ba1b8d42128a7-006477b423-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:54:59 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:54:59.915 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj62 2023-05-31T20:54:59.916 DEBUG:boto:path=/pklxkd-59/obj62 2023-05-31T20:54:59.916 DEBUG:boto:auth_path=/pklxkd-59/obj62 2023-05-31T20:54:59.916 DEBUG:boto:path=/pklxkd-59/obj62?versionId=null 2023-05-31T20:54:59.917 DEBUG:boto:auth_path=/pklxkd-59/obj62?versionId=null 2023-05-31T20:54:59.917 DEBUG:boto:Method: GET 2023-05-31T20:54:59.917 DEBUG:boto:Path: /pklxkd-59/obj62?versionId=null 2023-05-31T20:54:59.918 DEBUG:boto:Data: 2023-05-31T20:54:59.918 DEBUG:boto:Headers: {} 2023-05-31T20:54:59.919 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:54:59.919 DEBUG:boto:Port: 8001 2023-05-31T20:54:59.919 DEBUG:boto:Params: {} 2023-05-31T20:54:59.920 DEBUG:boto:Token: None 2023-05-31T20:54:59.920 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:54:59 GMT /pklxkd-59/obj62?versionId=null 2023-05-31T20:54:59.921 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:3rxOpEiNIps7y75ieoSyWP5nfkA= 2023-05-31T20:54:59.921 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:54:59 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:3rxOpEiNIps7y75ieoSyWP5nfkA=', 'Content-Length': '0'} 2023-05-31T20:54:59.923 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:50:33 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e3a62ee1580a6bae-006477b423-1e2e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:54:59 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:54:59.923 DEBUG:boto:path=/pklxkd-59/obj62 2023-05-31T20:54:59.924 DEBUG:boto:auth_path=/pklxkd-59/obj62 2023-05-31T20:54:59.924 DEBUG:boto:path=/pklxkd-59/obj62?versionId=null 2023-05-31T20:54:59.925 DEBUG:boto:auth_path=/pklxkd-59/obj62?versionId=null 2023-05-31T20:54:59.925 DEBUG:boto:Method: GET 2023-05-31T20:54:59.925 DEBUG:boto:Path: /pklxkd-59/obj62?versionId=null 2023-05-31T20:54:59.926 DEBUG:boto:Data: 2023-05-31T20:54:59.926 DEBUG:boto:Headers: {} 2023-05-31T20:54:59.926 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:54:59.927 DEBUG:boto:Port: 8000 2023-05-31T20:54:59.927 DEBUG:boto:Params: {} 2023-05-31T20:54:59.927 DEBUG:boto:Token: None 2023-05-31T20:54:59.928 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:54:59 GMT /pklxkd-59/obj62?versionId=null 2023-05-31T20:54:59.928 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:3rxOpEiNIps7y75ieoSyWP5nfkA= 2023-05-31T20:54:59.929 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:54:59 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:3rxOpEiNIps7y75ieoSyWP5nfkA=', 'Content-Length': '0'} 2023-05-31T20:54:59.931 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:50:33 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004515a9790dfd356f-006477b423-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:54:59 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:54:59.931 DEBUG:boto:path=/pklxkd-59/ 2023-05-31T20:54:59.932 DEBUG:boto:auth_path=/pklxkd-59/ 2023-05-31T20:54:59.932 DEBUG:boto:path=/pklxkd-59/?prefix=obj62 2023-05-31T20:54:59.932 DEBUG:boto:auth_path=/pklxkd-59/?prefix=obj62 2023-05-31T20:54:59.933 DEBUG:boto:Method: GET 2023-05-31T20:54:59.933 DEBUG:boto:Path: /pklxkd-59/?prefix=obj62 2023-05-31T20:54:59.934 DEBUG:boto:Data: 2023-05-31T20:54:59.934 DEBUG:boto:Headers: {} 2023-05-31T20:54:59.934 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:54:59.935 DEBUG:boto:Port: 8001 2023-05-31T20:54:59.935 DEBUG:boto:Params: {} 2023-05-31T20:54:59.935 DEBUG:boto:Token: None 2023-05-31T20:54:59.936 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:54:59 GMT /pklxkd-59/ 2023-05-31T20:54:59.936 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:tAvGnTMQPosMaETd601jyw6cOdE= 2023-05-31T20:54:59.937 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:54:59 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:tAvGnTMQPosMaETd601jyw6cOdE=', 'Content-Length': '0'} 2023-05-31T20:54:59.939 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000024970c51def7ece2-006477b423-1e2e-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:54:59 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:54:59.940 DEBUG:boto:b'pklxkd-59obj621000falseobj622023-05-31T20:50:33.515Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:54:59.941 DEBUG:boto:path=/pklxkd-59/ 2023-05-31T20:54:59.941 DEBUG:boto:auth_path=/pklxkd-59/ 2023-05-31T20:54:59.941 DEBUG:boto:path=/pklxkd-59/?prefix=obj62 2023-05-31T20:54:59.942 DEBUG:boto:auth_path=/pklxkd-59/?prefix=obj62 2023-05-31T20:54:59.942 DEBUG:boto:Method: GET 2023-05-31T20:54:59.942 DEBUG:boto:Path: /pklxkd-59/?prefix=obj62 2023-05-31T20:54:59.943 DEBUG:boto:Data: 2023-05-31T20:54:59.943 DEBUG:boto:Headers: {} 2023-05-31T20:54:59.943 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:54:59.944 DEBUG:boto:Port: 8000 2023-05-31T20:54:59.944 DEBUG:boto:Params: {} 2023-05-31T20:54:59.944 DEBUG:boto:Token: None 2023-05-31T20:54:59.945 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:54:59 GMT /pklxkd-59/ 2023-05-31T20:54:59.945 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:tAvGnTMQPosMaETd601jyw6cOdE= 2023-05-31T20:54:59.946 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:54:59 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:tAvGnTMQPosMaETd601jyw6cOdE=', 'Content-Length': '0'} 2023-05-31T20:54:59.950 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000073e8105c4175cf5f-006477b423-1630-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:54:59 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:54:59.951 DEBUG:boto:b'pklxkd-59obj621000falseobj622023-05-31T20:50:33.515Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:54:59.952 DEBUG:boto:path=/pklxkd-59/obj62 2023-05-31T20:54:59.952 DEBUG:boto:auth_path=/pklxkd-59/obj62 2023-05-31T20:54:59.953 DEBUG:boto:Method: HEAD 2023-05-31T20:54:59.953 DEBUG:boto:Path: /pklxkd-59/obj62 2023-05-31T20:54:59.953 DEBUG:boto:Data: 2023-05-31T20:54:59.954 DEBUG:boto:Headers: {} 2023-05-31T20:54:59.954 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:54:59.954 DEBUG:boto:Port: 8001 2023-05-31T20:54:59.955 DEBUG:boto:Params: {} 2023-05-31T20:54:59.955 DEBUG:boto:Token: None 2023-05-31T20:54:59.955 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:54:59 GMT /pklxkd-59/obj62 2023-05-31T20:54:59.956 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:AzabjwHHt/sFVB8hp9GAu5qVS+o= 2023-05-31T20:54:59.956 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:54:59 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:AzabjwHHt/sFVB8hp9GAu5qVS+o=', 'Content-Length': '0'} 2023-05-31T20:54:59.958 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:50:33 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000047f06981d17d1451-006477b423-1e2e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:54:59 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:54:59.959 DEBUG:boto:path=/pklxkd-59/obj62 2023-05-31T20:54:59.959 DEBUG:boto:auth_path=/pklxkd-59/obj62 2023-05-31T20:54:59.959 DEBUG:boto:Method: HEAD 2023-05-31T20:54:59.960 DEBUG:boto:Path: /pklxkd-59/obj62 2023-05-31T20:54:59.960 DEBUG:boto:Data: 2023-05-31T20:54:59.961 DEBUG:boto:Headers: {} 2023-05-31T20:54:59.961 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:54:59.961 DEBUG:boto:Port: 8000 2023-05-31T20:54:59.962 DEBUG:boto:Params: {} 2023-05-31T20:54:59.962 DEBUG:boto:Token: None 2023-05-31T20:54:59.962 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:54:59 GMT /pklxkd-59/obj62 2023-05-31T20:54:59.963 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:AzabjwHHt/sFVB8hp9GAu5qVS+o= 2023-05-31T20:54:59.963 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:54:59 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:AzabjwHHt/sFVB8hp9GAu5qVS+o=', 'Content-Length': '0'} 2023-05-31T20:54:59.965 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:50:33 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f203c1ea6154d7aa-006477b423-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:54:59 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:54:59.966 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj62 2023-05-31T20:54:59.966 DEBUG:boto:path=/pklxkd-59/obj62 2023-05-31T20:54:59.966 DEBUG:boto:auth_path=/pklxkd-59/obj62 2023-05-31T20:54:59.967 DEBUG:boto:Method: GET 2023-05-31T20:54:59.967 DEBUG:boto:Path: /pklxkd-59/obj62 2023-05-31T20:54:59.968 DEBUG:boto:Data: 2023-05-31T20:54:59.968 DEBUG:boto:Headers: {} 2023-05-31T20:54:59.968 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:54:59.969 DEBUG:boto:Port: 8001 2023-05-31T20:54:59.969 DEBUG:boto:Params: {} 2023-05-31T20:54:59.969 DEBUG:boto:Token: None 2023-05-31T20:54:59.970 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:54:59 GMT /pklxkd-59/obj62 2023-05-31T20:54:59.970 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:eWlP6E4QCTAo/TBrB2YbH0LyeUc= 2023-05-31T20:54:59.971 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:54:59 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:eWlP6E4QCTAo/TBrB2YbH0LyeUc=', 'Content-Length': '0'} 2023-05-31T20:54:59.973 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:50:33 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000019f72d40fd9bb412-006477b423-1e2e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:54:59 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:54:59.973 DEBUG:boto:path=/pklxkd-59/obj62 2023-05-31T20:54:59.974 DEBUG:boto:auth_path=/pklxkd-59/obj62 2023-05-31T20:54:59.974 DEBUG:boto:Method: GET 2023-05-31T20:54:59.974 DEBUG:boto:Path: /pklxkd-59/obj62 2023-05-31T20:54:59.975 DEBUG:boto:Data: 2023-05-31T20:54:59.975 DEBUG:boto:Headers: {} 2023-05-31T20:54:59.975 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:54:59.976 DEBUG:boto:Port: 8000 2023-05-31T20:54:59.976 DEBUG:boto:Params: {} 2023-05-31T20:54:59.977 DEBUG:boto:Token: None 2023-05-31T20:54:59.977 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:54:59 GMT /pklxkd-59/obj62 2023-05-31T20:54:59.977 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:eWlP6E4QCTAo/TBrB2YbH0LyeUc= 2023-05-31T20:54:59.978 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:54:59 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:eWlP6E4QCTAo/TBrB2YbH0LyeUc=', 'Content-Length': '0'} 2023-05-31T20:54:59.980 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:50:33 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009e451a2a9c5fff51-006477b423-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:54:59 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:55:00.024 DEBUG:boto:path=/pklxkd-59/ 2023-05-31T20:55:00.025 DEBUG:boto:auth_path=/pklxkd-59/ 2023-05-31T20:55:00.025 DEBUG:boto:path=/pklxkd-59/?prefix=obj62 2023-05-31T20:55:00.025 DEBUG:boto:auth_path=/pklxkd-59/?prefix=obj62 2023-05-31T20:55:00.026 DEBUG:boto:Method: GET 2023-05-31T20:55:00.026 DEBUG:boto:Path: /pklxkd-59/?prefix=obj62 2023-05-31T20:55:00.026 DEBUG:boto:Data: 2023-05-31T20:55:00.027 DEBUG:boto:Headers: {} 2023-05-31T20:55:00.027 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:55:00.028 DEBUG:boto:Port: 8001 2023-05-31T20:55:00.028 DEBUG:boto:Params: {} 2023-05-31T20:55:00.028 DEBUG:boto:Token: None 2023-05-31T20:55:00.029 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:55:00 GMT /pklxkd-59/ 2023-05-31T20:55:00.029 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:3AIDo5PzlXImNUFq+iJAv4ABZFs= 2023-05-31T20:55:00.030 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:55:00 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:3AIDo5PzlXImNUFq+iJAv4ABZFs=', 'Content-Length': '0'} 2023-05-31T20:55:00.032 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000607942c8703b5102-006477b424-1e2e-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:55:00 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:55:00.032 DEBUG:boto:b'pklxkd-59obj621000falseobj622023-05-31T20:50:33.515Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:55:00.033 DEBUG:boto:path=/pklxkd-59/ 2023-05-31T20:55:00.034 DEBUG:boto:auth_path=/pklxkd-59/ 2023-05-31T20:55:00.034 DEBUG:boto:path=/pklxkd-59/?prefix=obj62 2023-05-31T20:55:00.034 DEBUG:boto:auth_path=/pklxkd-59/?prefix=obj62 2023-05-31T20:55:00.035 DEBUG:boto:Method: GET 2023-05-31T20:55:00.035 DEBUG:boto:Path: /pklxkd-59/?prefix=obj62 2023-05-31T20:55:00.035 DEBUG:boto:Data: 2023-05-31T20:55:00.036 DEBUG:boto:Headers: {} 2023-05-31T20:55:00.036 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:55:00.037 DEBUG:boto:Port: 8000 2023-05-31T20:55:00.037 DEBUG:boto:Params: {} 2023-05-31T20:55:00.037 DEBUG:boto:Token: None 2023-05-31T20:55:00.038 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:55:00 GMT /pklxkd-59/ 2023-05-31T20:55:00.038 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:3AIDo5PzlXImNUFq+iJAv4ABZFs= 2023-05-31T20:55:00.039 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:55:00 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:3AIDo5PzlXImNUFq+iJAv4ABZFs=', 'Content-Length': '0'} 2023-05-31T20:55:00.042 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000008d7641cf67de9120-006477b424-1630-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:55:00 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:55:00.043 DEBUG:boto:b'pklxkd-59obj621000falseobj622023-05-31T20:50:33.515Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:55:00.044 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj68 2023-05-31T20:55:00.044 DEBUG:boto:path=/pklxkd-59/obj68 2023-05-31T20:55:00.044 DEBUG:boto:auth_path=/pklxkd-59/obj68 2023-05-31T20:55:00.045 DEBUG:boto:path=/pklxkd-59/obj68?versionId=null 2023-05-31T20:55:00.045 DEBUG:boto:auth_path=/pklxkd-59/obj68?versionId=null 2023-05-31T20:55:00.046 DEBUG:boto:Method: GET 2023-05-31T20:55:00.046 DEBUG:boto:Path: /pklxkd-59/obj68?versionId=null 2023-05-31T20:55:00.046 DEBUG:boto:Data: 2023-05-31T20:55:00.047 DEBUG:boto:Headers: {} 2023-05-31T20:55:00.047 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:55:00.048 DEBUG:boto:Port: 8001 2023-05-31T20:55:00.048 DEBUG:boto:Params: {} 2023-05-31T20:55:00.048 DEBUG:boto:Token: None 2023-05-31T20:55:00.049 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:55:00 GMT /pklxkd-59/obj68?versionId=null 2023-05-31T20:55:00.049 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:IdqW3DDcDfyNLDSVlDH0Vk1Yv5I= 2023-05-31T20:55:00.050 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:55:00 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:IdqW3DDcDfyNLDSVlDH0Vk1Yv5I=', 'Content-Length': '0'} 2023-05-31T20:55:00.052 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:43:08 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d74ba3e2f0bdf5f1-006477b424-1e2e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:55:00 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:55:00.052 DEBUG:boto:path=/pklxkd-59/obj68 2023-05-31T20:55:00.053 DEBUG:boto:auth_path=/pklxkd-59/obj68 2023-05-31T20:55:00.053 DEBUG:boto:path=/pklxkd-59/obj68?versionId=null 2023-05-31T20:55:00.054 DEBUG:boto:auth_path=/pklxkd-59/obj68?versionId=null 2023-05-31T20:55:00.054 DEBUG:boto:Method: GET 2023-05-31T20:55:00.054 DEBUG:boto:Path: /pklxkd-59/obj68?versionId=null 2023-05-31T20:55:00.055 DEBUG:boto:Data: 2023-05-31T20:55:00.055 DEBUG:boto:Headers: {} 2023-05-31T20:55:00.055 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:55:00.056 DEBUG:boto:Port: 8000 2023-05-31T20:55:00.056 DEBUG:boto:Params: {} 2023-05-31T20:55:00.056 DEBUG:boto:Token: None 2023-05-31T20:55:00.057 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:55:00 GMT /pklxkd-59/obj68?versionId=null 2023-05-31T20:55:00.057 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:IdqW3DDcDfyNLDSVlDH0Vk1Yv5I= 2023-05-31T20:55:00.058 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:55:00 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:IdqW3DDcDfyNLDSVlDH0Vk1Yv5I=', 'Content-Length': '0'} 2023-05-31T20:55:00.060 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:43:08 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000bec7f952fa665bdf-006477b424-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:55:00 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:55:00.060 DEBUG:boto:path=/pklxkd-59/obj68 2023-05-31T20:55:00.061 DEBUG:boto:auth_path=/pklxkd-59/obj68 2023-05-31T20:55:00.061 DEBUG:boto:path=/pklxkd-59/obj68?versionId=null 2023-05-31T20:55:00.062 DEBUG:boto:auth_path=/pklxkd-59/obj68?versionId=null 2023-05-31T20:55:00.062 DEBUG:boto:Method: HEAD 2023-05-31T20:55:00.062 DEBUG:boto:Path: /pklxkd-59/obj68?versionId=null 2023-05-31T20:55:00.063 DEBUG:boto:Data: 2023-05-31T20:55:00.063 DEBUG:boto:Headers: {} 2023-05-31T20:55:00.063 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:55:00.064 DEBUG:boto:Port: 8001 2023-05-31T20:55:00.064 DEBUG:boto:Params: {} 2023-05-31T20:55:00.064 DEBUG:boto:Token: None 2023-05-31T20:55:00.065 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:55:00 GMT /pklxkd-59/obj68?versionId=null 2023-05-31T20:55:00.065 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:yeYLUYDjHRzBMC9Qzcd+8OUv3PU= 2023-05-31T20:55:00.066 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:55:00 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:yeYLUYDjHRzBMC9Qzcd+8OUv3PU=', 'Content-Length': '0'} 2023-05-31T20:55:00.067 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:43:08 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002c572a5a6bdf3e70-006477b424-1e2e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:55:00 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:55:00.068 DEBUG:boto:path=/pklxkd-59/obj68 2023-05-31T20:55:00.068 DEBUG:boto:auth_path=/pklxkd-59/obj68 2023-05-31T20:55:00.069 DEBUG:boto:path=/pklxkd-59/obj68?versionId=null 2023-05-31T20:55:00.069 DEBUG:boto:auth_path=/pklxkd-59/obj68?versionId=null 2023-05-31T20:55:00.070 DEBUG:boto:Method: HEAD 2023-05-31T20:55:00.070 DEBUG:boto:Path: /pklxkd-59/obj68?versionId=null 2023-05-31T20:55:00.070 DEBUG:boto:Data: 2023-05-31T20:55:00.071 DEBUG:boto:Headers: {} 2023-05-31T20:55:00.071 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:55:00.072 DEBUG:boto:Port: 8000 2023-05-31T20:55:00.072 DEBUG:boto:Params: {} 2023-05-31T20:55:00.072 DEBUG:boto:Token: None 2023-05-31T20:55:00.073 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:55:00 GMT /pklxkd-59/obj68?versionId=null 2023-05-31T20:55:00.073 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:yeYLUYDjHRzBMC9Qzcd+8OUv3PU= 2023-05-31T20:55:00.074 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:55:00 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:yeYLUYDjHRzBMC9Qzcd+8OUv3PU=', 'Content-Length': '0'} 2023-05-31T20:55:00.076 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:43:08 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000059f2d5b4c38ec82e-006477b424-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:55:00 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:55:00.076 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj68 2023-05-31T20:55:00.077 DEBUG:boto:path=/pklxkd-59/obj68 2023-05-31T20:55:00.077 DEBUG:boto:auth_path=/pklxkd-59/obj68 2023-05-31T20:55:00.077 DEBUG:boto:path=/pklxkd-59/obj68?versionId=null 2023-05-31T20:55:00.078 DEBUG:boto:auth_path=/pklxkd-59/obj68?versionId=null 2023-05-31T20:55:00.078 DEBUG:boto:Method: GET 2023-05-31T20:55:00.079 DEBUG:boto:Path: /pklxkd-59/obj68?versionId=null 2023-05-31T20:55:00.079 DEBUG:boto:Data: 2023-05-31T20:55:00.079 DEBUG:boto:Headers: {} 2023-05-31T20:55:00.080 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:55:00.080 DEBUG:boto:Port: 8001 2023-05-31T20:55:00.081 DEBUG:boto:Params: {} 2023-05-31T20:55:00.081 DEBUG:boto:Token: None 2023-05-31T20:55:00.082 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:55:00 GMT /pklxkd-59/obj68?versionId=null 2023-05-31T20:55:00.082 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:IdqW3DDcDfyNLDSVlDH0Vk1Yv5I= 2023-05-31T20:55:00.083 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:55:00 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:IdqW3DDcDfyNLDSVlDH0Vk1Yv5I=', 'Content-Length': '0'} 2023-05-31T20:55:00.084 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:43:08 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000063af8981a41cb26d-006477b424-1e2e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:55:00 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:55:00.085 DEBUG:boto:path=/pklxkd-59/obj68 2023-05-31T20:55:00.086 DEBUG:boto:auth_path=/pklxkd-59/obj68 2023-05-31T20:55:00.086 DEBUG:boto:path=/pklxkd-59/obj68?versionId=null 2023-05-31T20:55:00.086 DEBUG:boto:auth_path=/pklxkd-59/obj68?versionId=null 2023-05-31T20:55:00.087 DEBUG:boto:Method: GET 2023-05-31T20:55:00.087 DEBUG:boto:Path: /pklxkd-59/obj68?versionId=null 2023-05-31T20:55:00.088 DEBUG:boto:Data: 2023-05-31T20:55:00.088 DEBUG:boto:Headers: {} 2023-05-31T20:55:00.088 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:55:00.089 DEBUG:boto:Port: 8000 2023-05-31T20:55:00.089 DEBUG:boto:Params: {} 2023-05-31T20:55:00.090 DEBUG:boto:Token: None 2023-05-31T20:55:00.090 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:55:00 GMT /pklxkd-59/obj68?versionId=null 2023-05-31T20:55:00.091 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:IdqW3DDcDfyNLDSVlDH0Vk1Yv5I= 2023-05-31T20:55:00.091 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:55:00 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:IdqW3DDcDfyNLDSVlDH0Vk1Yv5I=', 'Content-Length': '0'} 2023-05-31T20:55:00.093 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:43:08 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000cfe7c1aeb11dadaf-006477b424-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:55:00 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:55:00.094 DEBUG:boto:path=/pklxkd-59/ 2023-05-31T20:55:00.094 DEBUG:boto:auth_path=/pklxkd-59/ 2023-05-31T20:55:00.095 DEBUG:boto:path=/pklxkd-59/?prefix=obj68 2023-05-31T20:55:00.095 DEBUG:boto:auth_path=/pklxkd-59/?prefix=obj68 2023-05-31T20:55:00.095 DEBUG:boto:Method: GET 2023-05-31T20:55:00.096 DEBUG:boto:Path: /pklxkd-59/?prefix=obj68 2023-05-31T20:55:00.096 DEBUG:boto:Data: 2023-05-31T20:55:00.097 DEBUG:boto:Headers: {} 2023-05-31T20:55:00.097 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:55:00.097 DEBUG:boto:Port: 8001 2023-05-31T20:55:00.098 DEBUG:boto:Params: {} 2023-05-31T20:55:00.098 DEBUG:boto:Token: None 2023-05-31T20:55:00.099 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:55:00 GMT /pklxkd-59/ 2023-05-31T20:55:00.099 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:3AIDo5PzlXImNUFq+iJAv4ABZFs= 2023-05-31T20:55:00.100 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:55:00 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:3AIDo5PzlXImNUFq+iJAv4ABZFs=', 'Content-Length': '0'} 2023-05-31T20:55:00.102 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000bd9a8b92641b1f85-006477b424-1e2e-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:55:00 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:55:00.144 DEBUG:boto:b'pklxkd-59obj681000falseobj682023-05-31T20:43:08.066Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:55:00.145 DEBUG:boto:path=/pklxkd-59/ 2023-05-31T20:55:00.145 DEBUG:boto:auth_path=/pklxkd-59/ 2023-05-31T20:55:00.146 DEBUG:boto:path=/pklxkd-59/?prefix=obj68 2023-05-31T20:55:00.146 DEBUG:boto:auth_path=/pklxkd-59/?prefix=obj68 2023-05-31T20:55:00.147 DEBUG:boto:Method: GET 2023-05-31T20:55:00.147 DEBUG:boto:Path: /pklxkd-59/?prefix=obj68 2023-05-31T20:55:00.147 DEBUG:boto:Data: 2023-05-31T20:55:00.148 DEBUG:boto:Headers: {} 2023-05-31T20:55:00.148 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:55:00.149 DEBUG:boto:Port: 8000 2023-05-31T20:55:00.149 DEBUG:boto:Params: {} 2023-05-31T20:55:00.149 DEBUG:boto:Token: None 2023-05-31T20:55:00.150 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:55:00 GMT /pklxkd-59/ 2023-05-31T20:55:00.150 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:3AIDo5PzlXImNUFq+iJAv4ABZFs= 2023-05-31T20:55:00.151 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:55:00 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:3AIDo5PzlXImNUFq+iJAv4ABZFs=', 'Content-Length': '0'} 2023-05-31T20:55:00.155 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000fc17a9127a5f2b90-006477b424-1630-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:55:00 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:55:00.155 DEBUG:boto:b'pklxkd-59obj681000falseobj682023-05-31T20:43:08.066Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:55:00.156 DEBUG:boto:path=/pklxkd-59/obj68 2023-05-31T20:55:00.156 DEBUG:boto:auth_path=/pklxkd-59/obj68 2023-05-31T20:55:00.157 DEBUG:boto:Method: HEAD 2023-05-31T20:55:00.157 DEBUG:boto:Path: /pklxkd-59/obj68 2023-05-31T20:55:00.157 DEBUG:boto:Data: 2023-05-31T20:55:00.158 DEBUG:boto:Headers: {} 2023-05-31T20:55:00.158 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:55:00.158 DEBUG:boto:Port: 8001 2023-05-31T20:55:00.159 DEBUG:boto:Params: {} 2023-05-31T20:55:00.159 DEBUG:boto:Token: None 2023-05-31T20:55:00.160 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:55:00 GMT /pklxkd-59/obj68 2023-05-31T20:55:00.160 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:EjKmzqAxuorC+CAy54YXsle1esA= 2023-05-31T20:55:00.160 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:55:00 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:EjKmzqAxuorC+CAy54YXsle1esA=', 'Content-Length': '0'} 2023-05-31T20:55:00.162 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:43:08 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000de1ad064ec676081-006477b424-1e2e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:55:00 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:55:00.163 DEBUG:boto:path=/pklxkd-59/obj68 2023-05-31T20:55:00.163 DEBUG:boto:auth_path=/pklxkd-59/obj68 2023-05-31T20:55:00.164 DEBUG:boto:Method: HEAD 2023-05-31T20:55:00.164 DEBUG:boto:Path: /pklxkd-59/obj68 2023-05-31T20:55:00.164 DEBUG:boto:Data: 2023-05-31T20:55:00.165 DEBUG:boto:Headers: {} 2023-05-31T20:55:00.165 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:55:00.165 DEBUG:boto:Port: 8000 2023-05-31T20:55:00.166 DEBUG:boto:Params: {} 2023-05-31T20:55:00.166 DEBUG:boto:Token: None 2023-05-31T20:55:00.167 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:55:00 GMT /pklxkd-59/obj68 2023-05-31T20:55:00.167 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:EjKmzqAxuorC+CAy54YXsle1esA= 2023-05-31T20:55:00.167 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:55:00 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:EjKmzqAxuorC+CAy54YXsle1esA=', 'Content-Length': '0'} 2023-05-31T20:55:00.169 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:43:08 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000316fbc08778569f6-006477b424-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:55:00 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:55:00.170 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj68 2023-05-31T20:55:00.170 DEBUG:boto:path=/pklxkd-59/obj68 2023-05-31T20:55:00.170 DEBUG:boto:auth_path=/pklxkd-59/obj68 2023-05-31T20:55:00.171 DEBUG:boto:Method: GET 2023-05-31T20:55:00.171 DEBUG:boto:Path: /pklxkd-59/obj68 2023-05-31T20:55:00.172 DEBUG:boto:Data: 2023-05-31T20:55:00.172 DEBUG:boto:Headers: {} 2023-05-31T20:55:00.172 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:55:00.173 DEBUG:boto:Port: 8001 2023-05-31T20:55:00.173 DEBUG:boto:Params: {} 2023-05-31T20:55:00.173 DEBUG:boto:Token: None 2023-05-31T20:55:00.174 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:55:00 GMT /pklxkd-59/obj68 2023-05-31T20:55:00.174 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:ApC+getXs5f7hLF8lWtJco2Gatw= 2023-05-31T20:55:00.175 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:55:00 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:ApC+getXs5f7hLF8lWtJco2Gatw=', 'Content-Length': '0'} 2023-05-31T20:55:00.176 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:43:08 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c17fbb7bcbdfbb2a-006477b424-1e2e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:55:00 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:55:00.177 DEBUG:boto:path=/pklxkd-59/obj68 2023-05-31T20:55:00.177 DEBUG:boto:auth_path=/pklxkd-59/obj68 2023-05-31T20:55:00.178 DEBUG:boto:Method: GET 2023-05-31T20:55:00.178 DEBUG:boto:Path: /pklxkd-59/obj68 2023-05-31T20:55:00.179 DEBUG:boto:Data: 2023-05-31T20:55:00.179 DEBUG:boto:Headers: {} 2023-05-31T20:55:00.179 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:55:00.180 DEBUG:boto:Port: 8000 2023-05-31T20:55:00.180 DEBUG:boto:Params: {} 2023-05-31T20:55:00.181 DEBUG:boto:Token: None 2023-05-31T20:55:00.181 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:55:00 GMT /pklxkd-59/obj68 2023-05-31T20:55:00.181 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:ApC+getXs5f7hLF8lWtJco2Gatw= 2023-05-31T20:55:00.182 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:55:00 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:ApC+getXs5f7hLF8lWtJco2Gatw=', 'Content-Length': '0'} 2023-05-31T20:55:00.183 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:43:08 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000056208ca48d6233fd-006477b424-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:55:00 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:55:00.224 DEBUG:boto:path=/pklxkd-59/ 2023-05-31T20:55:00.225 DEBUG:boto:auth_path=/pklxkd-59/ 2023-05-31T20:55:00.225 DEBUG:boto:path=/pklxkd-59/?prefix=obj68 2023-05-31T20:55:00.225 DEBUG:boto:auth_path=/pklxkd-59/?prefix=obj68 2023-05-31T20:55:00.226 DEBUG:boto:Method: GET 2023-05-31T20:55:00.226 DEBUG:boto:Path: /pklxkd-59/?prefix=obj68 2023-05-31T20:55:00.227 DEBUG:boto:Data: 2023-05-31T20:55:00.227 DEBUG:boto:Headers: {} 2023-05-31T20:55:00.227 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:55:00.228 DEBUG:boto:Port: 8001 2023-05-31T20:55:00.228 DEBUG:boto:Params: {} 2023-05-31T20:55:00.228 DEBUG:boto:Token: None 2023-05-31T20:55:00.229 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:55:00 GMT /pklxkd-59/ 2023-05-31T20:55:00.229 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:3AIDo5PzlXImNUFq+iJAv4ABZFs= 2023-05-31T20:55:00.230 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:55:00 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:3AIDo5PzlXImNUFq+iJAv4ABZFs=', 'Content-Length': '0'} 2023-05-31T20:55:00.232 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000eefe5b87be6092f2-006477b424-1e2e-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:55:00 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:55:00.233 DEBUG:boto:b'pklxkd-59obj681000falseobj682023-05-31T20:43:08.066Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:55:00.234 DEBUG:boto:path=/pklxkd-59/ 2023-05-31T20:55:00.234 DEBUG:boto:auth_path=/pklxkd-59/ 2023-05-31T20:55:00.234 DEBUG:boto:path=/pklxkd-59/?prefix=obj68 2023-05-31T20:55:00.235 DEBUG:boto:auth_path=/pklxkd-59/?prefix=obj68 2023-05-31T20:55:00.235 DEBUG:boto:Method: GET 2023-05-31T20:55:00.235 DEBUG:boto:Path: /pklxkd-59/?prefix=obj68 2023-05-31T20:55:00.236 DEBUG:boto:Data: 2023-05-31T20:55:00.236 DEBUG:boto:Headers: {} 2023-05-31T20:55:00.237 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:55:00.237 DEBUG:boto:Port: 8000 2023-05-31T20:55:00.237 DEBUG:boto:Params: {} 2023-05-31T20:55:00.238 DEBUG:boto:Token: None 2023-05-31T20:55:00.238 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:55:00 GMT /pklxkd-59/ 2023-05-31T20:55:00.239 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:3AIDo5PzlXImNUFq+iJAv4ABZFs= 2023-05-31T20:55:00.239 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:55:00 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:3AIDo5PzlXImNUFq+iJAv4ABZFs=', 'Content-Length': '0'} 2023-05-31T20:55:00.243 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f156cec52eea5da1-006477b424-1630-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:55:00 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:55:00.243 DEBUG:boto:b'pklxkd-59obj681000falseobj682023-05-31T20:43:08.066Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:55:00.244 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj69 2023-05-31T20:55:00.245 DEBUG:boto:path=/pklxkd-59/obj69 2023-05-31T20:55:00.245 DEBUG:boto:auth_path=/pklxkd-59/obj69 2023-05-31T20:55:00.245 DEBUG:boto:path=/pklxkd-59/obj69?versionId=null 2023-05-31T20:55:00.246 DEBUG:boto:auth_path=/pklxkd-59/obj69?versionId=null 2023-05-31T20:55:00.246 DEBUG:boto:Method: GET 2023-05-31T20:55:00.246 DEBUG:boto:Path: /pklxkd-59/obj69?versionId=null 2023-05-31T20:55:00.247 DEBUG:boto:Data: 2023-05-31T20:55:00.247 DEBUG:boto:Headers: {} 2023-05-31T20:55:00.247 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:55:00.248 DEBUG:boto:Port: 8001 2023-05-31T20:55:00.248 DEBUG:boto:Params: {} 2023-05-31T20:55:00.249 DEBUG:boto:Token: None 2023-05-31T20:55:00.249 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:55:00 GMT /pklxkd-59/obj69?versionId=null 2023-05-31T20:55:00.250 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:4hCeTMxgjQw7SmGkeHdcl1ilzL4= 2023-05-31T20:55:00.250 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:55:00 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:4hCeTMxgjQw7SmGkeHdcl1ilzL4=', 'Content-Length': '0'} 2023-05-31T20:55:00.252 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:46:33 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f40caa8506ef25e9-006477b424-1e2e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:55:00 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:55:00.253 DEBUG:boto:path=/pklxkd-59/obj69 2023-05-31T20:55:00.253 DEBUG:boto:auth_path=/pklxkd-59/obj69 2023-05-31T20:55:00.254 DEBUG:boto:path=/pklxkd-59/obj69?versionId=null 2023-05-31T20:55:00.254 DEBUG:boto:auth_path=/pklxkd-59/obj69?versionId=null 2023-05-31T20:55:00.254 DEBUG:boto:Method: GET 2023-05-31T20:55:00.255 DEBUG:boto:Path: /pklxkd-59/obj69?versionId=null 2023-05-31T20:55:00.255 DEBUG:boto:Data: 2023-05-31T20:55:00.255 DEBUG:boto:Headers: {} 2023-05-31T20:55:00.256 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:55:00.256 DEBUG:boto:Port: 8000 2023-05-31T20:55:00.257 DEBUG:boto:Params: {} 2023-05-31T20:55:00.257 DEBUG:boto:Token: None 2023-05-31T20:55:00.257 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:55:00 GMT /pklxkd-59/obj69?versionId=null 2023-05-31T20:55:00.258 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:4hCeTMxgjQw7SmGkeHdcl1ilzL4= 2023-05-31T20:55:00.258 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:55:00 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:4hCeTMxgjQw7SmGkeHdcl1ilzL4=', 'Content-Length': '0'} 2023-05-31T20:55:00.260 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:46:33 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000013301d12aac9342d-006477b424-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:55:00 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:55:00.261 DEBUG:boto:path=/pklxkd-59/obj69 2023-05-31T20:55:00.261 DEBUG:boto:auth_path=/pklxkd-59/obj69 2023-05-31T20:55:00.262 DEBUG:boto:path=/pklxkd-59/obj69?versionId=null 2023-05-31T20:55:00.262 DEBUG:boto:auth_path=/pklxkd-59/obj69?versionId=null 2023-05-31T20:55:00.262 DEBUG:boto:Method: HEAD 2023-05-31T20:55:00.263 DEBUG:boto:Path: /pklxkd-59/obj69?versionId=null 2023-05-31T20:55:00.263 DEBUG:boto:Data: 2023-05-31T20:55:00.263 DEBUG:boto:Headers: {} 2023-05-31T20:55:00.264 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:55:00.264 DEBUG:boto:Port: 8001 2023-05-31T20:55:00.265 DEBUG:boto:Params: {} 2023-05-31T20:55:00.265 DEBUG:boto:Token: None 2023-05-31T20:55:00.265 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:55:00 GMT /pklxkd-59/obj69?versionId=null 2023-05-31T20:55:00.266 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:Lx257SLykfFZZux3qzVEdF09rLs= 2023-05-31T20:55:00.266 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:55:00 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:Lx257SLykfFZZux3qzVEdF09rLs=', 'Content-Length': '0'} 2023-05-31T20:55:00.268 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:46:33 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f8800a7c5a648e85-006477b424-1e2e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:55:00 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:55:00.268 DEBUG:boto:path=/pklxkd-59/obj69 2023-05-31T20:55:00.269 DEBUG:boto:auth_path=/pklxkd-59/obj69 2023-05-31T20:55:00.269 DEBUG:boto:path=/pklxkd-59/obj69?versionId=null 2023-05-31T20:55:00.269 DEBUG:boto:auth_path=/pklxkd-59/obj69?versionId=null 2023-05-31T20:55:00.270 DEBUG:boto:Method: HEAD 2023-05-31T20:55:00.270 DEBUG:boto:Path: /pklxkd-59/obj69?versionId=null 2023-05-31T20:55:00.270 DEBUG:boto:Data: 2023-05-31T20:55:00.271 DEBUG:boto:Headers: {} 2023-05-31T20:55:00.271 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:55:00.271 DEBUG:boto:Port: 8000 2023-05-31T20:55:00.272 DEBUG:boto:Params: {} 2023-05-31T20:55:00.272 DEBUG:boto:Token: None 2023-05-31T20:55:00.273 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:55:00 GMT /pklxkd-59/obj69?versionId=null 2023-05-31T20:55:00.273 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:Lx257SLykfFZZux3qzVEdF09rLs= 2023-05-31T20:55:00.273 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:55:00 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:Lx257SLykfFZZux3qzVEdF09rLs=', 'Content-Length': '0'} 2023-05-31T20:55:00.275 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:46:33 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000062c593faa7d7a377-006477b424-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:55:00 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:55:00.276 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj69 2023-05-31T20:55:00.276 DEBUG:boto:path=/pklxkd-59/obj69 2023-05-31T20:55:00.276 DEBUG:boto:auth_path=/pklxkd-59/obj69 2023-05-31T20:55:00.277 DEBUG:boto:path=/pklxkd-59/obj69?versionId=null 2023-05-31T20:55:00.277 DEBUG:boto:auth_path=/pklxkd-59/obj69?versionId=null 2023-05-31T20:55:00.277 DEBUG:boto:Method: GET 2023-05-31T20:55:00.278 DEBUG:boto:Path: /pklxkd-59/obj69?versionId=null 2023-05-31T20:55:00.278 DEBUG:boto:Data: 2023-05-31T20:55:00.279 DEBUG:boto:Headers: {} 2023-05-31T20:55:00.279 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:55:00.279 DEBUG:boto:Port: 8001 2023-05-31T20:55:00.280 DEBUG:boto:Params: {} 2023-05-31T20:55:00.280 DEBUG:boto:Token: None 2023-05-31T20:55:00.281 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:55:00 GMT /pklxkd-59/obj69?versionId=null 2023-05-31T20:55:00.281 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:4hCeTMxgjQw7SmGkeHdcl1ilzL4= 2023-05-31T20:55:00.282 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:55:00 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:4hCeTMxgjQw7SmGkeHdcl1ilzL4=', 'Content-Length': '0'} 2023-05-31T20:55:00.283 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:46:33 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006be0a7e1a4daa294-006477b424-1e2e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:55:00 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:55:00.324 DEBUG:boto:path=/pklxkd-59/obj69 2023-05-31T20:55:00.325 DEBUG:boto:auth_path=/pklxkd-59/obj69 2023-05-31T20:55:00.325 DEBUG:boto:path=/pklxkd-59/obj69?versionId=null 2023-05-31T20:55:00.325 DEBUG:boto:auth_path=/pklxkd-59/obj69?versionId=null 2023-05-31T20:55:00.326 DEBUG:boto:Method: GET 2023-05-31T20:55:00.326 DEBUG:boto:Path: /pklxkd-59/obj69?versionId=null 2023-05-31T20:55:00.327 DEBUG:boto:Data: 2023-05-31T20:55:00.327 DEBUG:boto:Headers: {} 2023-05-31T20:55:00.327 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:55:00.328 DEBUG:boto:Port: 8000 2023-05-31T20:55:00.328 DEBUG:boto:Params: {} 2023-05-31T20:55:00.328 DEBUG:boto:Token: None 2023-05-31T20:55:00.329 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:55:00 GMT /pklxkd-59/obj69?versionId=null 2023-05-31T20:55:00.329 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:4hCeTMxgjQw7SmGkeHdcl1ilzL4= 2023-05-31T20:55:00.330 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:55:00 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:4hCeTMxgjQw7SmGkeHdcl1ilzL4=', 'Content-Length': '0'} 2023-05-31T20:55:00.332 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:46:33 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003e0f7eb6de5308da-006477b424-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:55:00 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:55:00.332 DEBUG:boto:path=/pklxkd-59/ 2023-05-31T20:55:00.333 DEBUG:boto:auth_path=/pklxkd-59/ 2023-05-31T20:55:00.333 DEBUG:boto:path=/pklxkd-59/?prefix=obj69 2023-05-31T20:55:00.333 DEBUG:boto:auth_path=/pklxkd-59/?prefix=obj69 2023-05-31T20:55:00.334 DEBUG:boto:Method: GET 2023-05-31T20:55:00.334 DEBUG:boto:Path: /pklxkd-59/?prefix=obj69 2023-05-31T20:55:00.335 DEBUG:boto:Data: 2023-05-31T20:55:00.335 DEBUG:boto:Headers: {} 2023-05-31T20:55:00.335 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:55:00.336 DEBUG:boto:Port: 8001 2023-05-31T20:55:00.336 DEBUG:boto:Params: {} 2023-05-31T20:55:00.337 DEBUG:boto:Token: None 2023-05-31T20:55:00.337 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:55:00 GMT /pklxkd-59/ 2023-05-31T20:55:00.337 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:3AIDo5PzlXImNUFq+iJAv4ABZFs= 2023-05-31T20:55:00.338 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:55:00 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:3AIDo5PzlXImNUFq+iJAv4ABZFs=', 'Content-Length': '0'} 2023-05-31T20:55:00.340 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000003a003b04e2455d3c-006477b424-1e2e-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:55:00 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:55:00.341 DEBUG:boto:b'pklxkd-59obj691000falseobj692023-05-31T20:46:33.886Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:55:00.342 DEBUG:boto:path=/pklxkd-59/ 2023-05-31T20:55:00.342 DEBUG:boto:auth_path=/pklxkd-59/ 2023-05-31T20:55:00.342 DEBUG:boto:path=/pklxkd-59/?prefix=obj69 2023-05-31T20:55:00.343 DEBUG:boto:auth_path=/pklxkd-59/?prefix=obj69 2023-05-31T20:55:00.343 DEBUG:boto:Method: GET 2023-05-31T20:55:00.343 DEBUG:boto:Path: /pklxkd-59/?prefix=obj69 2023-05-31T20:55:00.344 DEBUG:boto:Data: 2023-05-31T20:55:00.344 DEBUG:boto:Headers: {} 2023-05-31T20:55:00.344 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:55:00.345 DEBUG:boto:Port: 8000 2023-05-31T20:55:00.345 DEBUG:boto:Params: {} 2023-05-31T20:55:00.346 DEBUG:boto:Token: None 2023-05-31T20:55:00.346 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:55:00 GMT /pklxkd-59/ 2023-05-31T20:55:00.347 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:3AIDo5PzlXImNUFq+iJAv4ABZFs= 2023-05-31T20:55:00.347 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:55:00 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:3AIDo5PzlXImNUFq+iJAv4ABZFs=', 'Content-Length': '0'} 2023-05-31T20:55:00.351 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000da5f2f9dd4e50e27-006477b424-1630-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:55:00 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:55:00.351 DEBUG:boto:b'pklxkd-59obj691000falseobj692023-05-31T20:46:33.886Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:55:00.352 DEBUG:boto:path=/pklxkd-59/obj69 2023-05-31T20:55:00.352 DEBUG:boto:auth_path=/pklxkd-59/obj69 2023-05-31T20:55:00.353 DEBUG:boto:Method: HEAD 2023-05-31T20:55:00.353 DEBUG:boto:Path: /pklxkd-59/obj69 2023-05-31T20:55:00.354 DEBUG:boto:Data: 2023-05-31T20:55:00.354 DEBUG:boto:Headers: {} 2023-05-31T20:55:00.354 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:55:00.355 DEBUG:boto:Port: 8001 2023-05-31T20:55:00.355 DEBUG:boto:Params: {} 2023-05-31T20:55:00.355 DEBUG:boto:Token: None 2023-05-31T20:55:00.356 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:55:00 GMT /pklxkd-59/obj69 2023-05-31T20:55:00.356 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:6LqXIdHxdQhpwikCGw0mGtgOtxE= 2023-05-31T20:55:00.356 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:55:00 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:6LqXIdHxdQhpwikCGw0mGtgOtxE=', 'Content-Length': '0'} 2023-05-31T20:55:00.358 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:46:33 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008481d931bfaf7044-006477b424-1e2e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:55:00 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:55:00.359 DEBUG:boto:path=/pklxkd-59/obj69 2023-05-31T20:55:00.359 DEBUG:boto:auth_path=/pklxkd-59/obj69 2023-05-31T20:55:00.360 DEBUG:boto:Method: HEAD 2023-05-31T20:55:00.360 DEBUG:boto:Path: /pklxkd-59/obj69 2023-05-31T20:55:00.360 DEBUG:boto:Data: 2023-05-31T20:55:00.361 DEBUG:boto:Headers: {} 2023-05-31T20:55:00.361 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:55:00.361 DEBUG:boto:Port: 8000 2023-05-31T20:55:00.362 DEBUG:boto:Params: {} 2023-05-31T20:55:00.362 DEBUG:boto:Token: None 2023-05-31T20:55:00.362 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:55:00 GMT /pklxkd-59/obj69 2023-05-31T20:55:00.363 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:6LqXIdHxdQhpwikCGw0mGtgOtxE= 2023-05-31T20:55:00.363 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:55:00 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:6LqXIdHxdQhpwikCGw0mGtgOtxE=', 'Content-Length': '0'} 2023-05-31T20:55:00.365 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:46:33 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000be601a66b54d273d-006477b424-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:55:00 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:55:00.365 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj69 2023-05-31T20:55:00.366 DEBUG:boto:path=/pklxkd-59/obj69 2023-05-31T20:55:00.366 DEBUG:boto:auth_path=/pklxkd-59/obj69 2023-05-31T20:55:00.366 DEBUG:boto:Method: GET 2023-05-31T20:55:00.367 DEBUG:boto:Path: /pklxkd-59/obj69 2023-05-31T20:55:00.367 DEBUG:boto:Data: 2023-05-31T20:55:00.368 DEBUG:boto:Headers: {} 2023-05-31T20:55:00.368 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:55:00.368 DEBUG:boto:Port: 8001 2023-05-31T20:55:00.369 DEBUG:boto:Params: {} 2023-05-31T20:55:00.369 DEBUG:boto:Token: None 2023-05-31T20:55:00.370 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:55:00 GMT /pklxkd-59/obj69 2023-05-31T20:55:00.370 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:ntPCMAOQewwA9VclA1iHlDBz0dU= 2023-05-31T20:55:00.370 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:55:00 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:ntPCMAOQewwA9VclA1iHlDBz0dU=', 'Content-Length': '0'} 2023-05-31T20:55:00.372 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:46:33 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004d7dff52e0149d13-006477b424-1e2e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:55:00 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:55:00.373 DEBUG:boto:path=/pklxkd-59/obj69 2023-05-31T20:55:00.373 DEBUG:boto:auth_path=/pklxkd-59/obj69 2023-05-31T20:55:00.374 DEBUG:boto:Method: GET 2023-05-31T20:55:00.374 DEBUG:boto:Path: /pklxkd-59/obj69 2023-05-31T20:55:00.374 DEBUG:boto:Data: 2023-05-31T20:55:00.375 DEBUG:boto:Headers: {} 2023-05-31T20:55:00.375 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:55:00.376 DEBUG:boto:Port: 8000 2023-05-31T20:55:00.376 DEBUG:boto:Params: {} 2023-05-31T20:55:00.376 DEBUG:boto:Token: None 2023-05-31T20:55:00.377 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:55:00 GMT /pklxkd-59/obj69 2023-05-31T20:55:00.377 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:ntPCMAOQewwA9VclA1iHlDBz0dU= 2023-05-31T20:55:00.378 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:55:00 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:ntPCMAOQewwA9VclA1iHlDBz0dU=', 'Content-Length': '0'} 2023-05-31T20:55:00.379 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:46:33 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009e2624703bae87c8-006477b424-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:55:00 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:55:00.380 DEBUG:boto:path=/pklxkd-59/ 2023-05-31T20:55:00.380 DEBUG:boto:auth_path=/pklxkd-59/ 2023-05-31T20:55:00.381 DEBUG:boto:path=/pklxkd-59/?prefix=obj69 2023-05-31T20:55:00.381 DEBUG:boto:auth_path=/pklxkd-59/?prefix=obj69 2023-05-31T20:55:00.382 DEBUG:boto:Method: GET 2023-05-31T20:55:00.382 DEBUG:boto:Path: /pklxkd-59/?prefix=obj69 2023-05-31T20:55:00.382 DEBUG:boto:Data: 2023-05-31T20:55:00.383 DEBUG:boto:Headers: {} 2023-05-31T20:55:00.383 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:55:00.383 DEBUG:boto:Port: 8001 2023-05-31T20:55:00.384 DEBUG:boto:Params: {} 2023-05-31T20:55:00.384 DEBUG:boto:Token: None 2023-05-31T20:55:00.385 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:55:00 GMT /pklxkd-59/ 2023-05-31T20:55:00.385 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:3AIDo5PzlXImNUFq+iJAv4ABZFs= 2023-05-31T20:55:00.386 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:55:00 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:3AIDo5PzlXImNUFq+iJAv4ABZFs=', 'Content-Length': '0'} 2023-05-31T20:55:00.388 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000494fd1d70974fef1-006477b424-1e2e-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:55:00 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:55:00.428 DEBUG:boto:b'pklxkd-59obj691000falseobj692023-05-31T20:46:33.886Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:55:00.429 DEBUG:boto:path=/pklxkd-59/ 2023-05-31T20:55:00.430 DEBUG:boto:auth_path=/pklxkd-59/ 2023-05-31T20:55:00.430 DEBUG:boto:path=/pklxkd-59/?prefix=obj69 2023-05-31T20:55:00.430 DEBUG:boto:auth_path=/pklxkd-59/?prefix=obj69 2023-05-31T20:55:00.431 DEBUG:boto:Method: GET 2023-05-31T20:55:00.431 DEBUG:boto:Path: /pklxkd-59/?prefix=obj69 2023-05-31T20:55:00.431 DEBUG:boto:Data: 2023-05-31T20:55:00.432 DEBUG:boto:Headers: {} 2023-05-31T20:55:00.432 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:55:00.433 DEBUG:boto:Port: 8000 2023-05-31T20:55:00.433 DEBUG:boto:Params: {} 2023-05-31T20:55:00.433 DEBUG:boto:Token: None 2023-05-31T20:55:00.434 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:55:00 GMT /pklxkd-59/ 2023-05-31T20:55:00.434 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:3AIDo5PzlXImNUFq+iJAv4ABZFs= 2023-05-31T20:55:00.435 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:55:00 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:3AIDo5PzlXImNUFq+iJAv4ABZFs=', 'Content-Length': '0'} 2023-05-31T20:55:00.439 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000bdc51846b668295e-006477b424-1630-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:55:00 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:55:00.439 DEBUG:boto:b'pklxkd-59obj691000falseobj692023-05-31T20:46:33.886Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:55:00.440 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj74 2023-05-31T20:55:00.441 DEBUG:boto:path=/pklxkd-59/obj74 2023-05-31T20:55:00.441 DEBUG:boto:auth_path=/pklxkd-59/obj74 2023-05-31T20:55:00.441 DEBUG:boto:path=/pklxkd-59/obj74?versionId=null 2023-05-31T20:55:00.442 DEBUG:boto:auth_path=/pklxkd-59/obj74?versionId=null 2023-05-31T20:55:00.442 DEBUG:boto:Method: GET 2023-05-31T20:55:00.442 DEBUG:boto:Path: /pklxkd-59/obj74?versionId=null 2023-05-31T20:55:00.443 DEBUG:boto:Data: 2023-05-31T20:55:00.443 DEBUG:boto:Headers: {} 2023-05-31T20:55:00.443 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:55:00.444 DEBUG:boto:Port: 8001 2023-05-31T20:55:00.444 DEBUG:boto:Params: {} 2023-05-31T20:55:00.445 DEBUG:boto:Token: None 2023-05-31T20:55:00.445 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:55:00 GMT /pklxkd-59/obj74?versionId=null 2023-05-31T20:55:00.445 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:AEtkP+C3564Rc/I6NCJzN50rAxc= 2023-05-31T20:55:00.446 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:55:00 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:AEtkP+C3564Rc/I6NCJzN50rAxc=', 'Content-Length': '0'} 2023-05-31T20:55:00.448 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:52:14 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009f743aad11a22dc1-006477b424-1e2e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:55:00 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:55:00.449 DEBUG:boto:path=/pklxkd-59/obj74 2023-05-31T20:55:00.449 DEBUG:boto:auth_path=/pklxkd-59/obj74 2023-05-31T20:55:00.449 DEBUG:boto:path=/pklxkd-59/obj74?versionId=null 2023-05-31T20:55:00.450 DEBUG:boto:auth_path=/pklxkd-59/obj74?versionId=null 2023-05-31T20:55:00.450 DEBUG:boto:Method: GET 2023-05-31T20:55:00.450 DEBUG:boto:Path: /pklxkd-59/obj74?versionId=null 2023-05-31T20:55:00.451 DEBUG:boto:Data: 2023-05-31T20:55:00.451 DEBUG:boto:Headers: {} 2023-05-31T20:55:00.452 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:55:00.452 DEBUG:boto:Port: 8000 2023-05-31T20:55:00.452 DEBUG:boto:Params: {} 2023-05-31T20:55:00.453 DEBUG:boto:Token: None 2023-05-31T20:55:00.453 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:55:00 GMT /pklxkd-59/obj74?versionId=null 2023-05-31T20:55:00.454 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:AEtkP+C3564Rc/I6NCJzN50rAxc= 2023-05-31T20:55:00.454 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:55:00 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:AEtkP+C3564Rc/I6NCJzN50rAxc=', 'Content-Length': '0'} 2023-05-31T20:55:00.456 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:52:14 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002327e028113beaeb-006477b424-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:55:00 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:55:00.496 DEBUG:boto:path=/pklxkd-59/obj74 2023-05-31T20:55:00.497 DEBUG:boto:auth_path=/pklxkd-59/obj74 2023-05-31T20:55:00.497 DEBUG:boto:path=/pklxkd-59/obj74?versionId=null 2023-05-31T20:55:00.497 DEBUG:boto:auth_path=/pklxkd-59/obj74?versionId=null 2023-05-31T20:55:00.498 DEBUG:boto:Method: HEAD 2023-05-31T20:55:00.498 DEBUG:boto:Path: /pklxkd-59/obj74?versionId=null 2023-05-31T20:55:00.498 DEBUG:boto:Data: 2023-05-31T20:55:00.499 DEBUG:boto:Headers: {} 2023-05-31T20:55:00.499 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:55:00.499 DEBUG:boto:Port: 8001 2023-05-31T20:55:00.500 DEBUG:boto:Params: {} 2023-05-31T20:55:00.500 DEBUG:boto:Token: None 2023-05-31T20:55:00.500 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:55:00 GMT /pklxkd-59/obj74?versionId=null 2023-05-31T20:55:00.501 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:0aRWIAyMBFtg0gAp+f2XUNixNPY= 2023-05-31T20:55:00.501 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:55:00 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:0aRWIAyMBFtg0gAp+f2XUNixNPY=', 'Content-Length': '0'} 2023-05-31T20:55:00.503 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:52:14 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ea0ca2c91ef7ab4f-006477b424-1e2e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:55:00 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:55:00.504 DEBUG:boto:path=/pklxkd-59/obj74 2023-05-31T20:55:00.504 DEBUG:boto:auth_path=/pklxkd-59/obj74 2023-05-31T20:55:00.504 DEBUG:boto:path=/pklxkd-59/obj74?versionId=null 2023-05-31T20:55:00.505 DEBUG:boto:auth_path=/pklxkd-59/obj74?versionId=null 2023-05-31T20:55:00.505 DEBUG:boto:Method: HEAD 2023-05-31T20:55:00.505 DEBUG:boto:Path: /pklxkd-59/obj74?versionId=null 2023-05-31T20:55:00.506 DEBUG:boto:Data: 2023-05-31T20:55:00.506 DEBUG:boto:Headers: {} 2023-05-31T20:55:00.506 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:55:00.507 DEBUG:boto:Port: 8000 2023-05-31T20:55:00.507 DEBUG:boto:Params: {} 2023-05-31T20:55:00.507 DEBUG:boto:Token: None 2023-05-31T20:55:00.508 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:55:00 GMT /pklxkd-59/obj74?versionId=null 2023-05-31T20:55:00.508 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:0aRWIAyMBFtg0gAp+f2XUNixNPY= 2023-05-31T20:55:00.509 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:55:00 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:0aRWIAyMBFtg0gAp+f2XUNixNPY=', 'Content-Length': '0'} 2023-05-31T20:55:00.510 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:52:14 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c400c8720ecfa07d-006477b424-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:55:00 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:55:00.511 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj74 2023-05-31T20:55:00.511 DEBUG:boto:path=/pklxkd-59/obj74 2023-05-31T20:55:00.511 DEBUG:boto:auth_path=/pklxkd-59/obj74 2023-05-31T20:55:00.512 DEBUG:boto:path=/pklxkd-59/obj74?versionId=null 2023-05-31T20:55:00.512 DEBUG:boto:auth_path=/pklxkd-59/obj74?versionId=null 2023-05-31T20:55:00.513 DEBUG:boto:Method: GET 2023-05-31T20:55:00.513 DEBUG:boto:Path: /pklxkd-59/obj74?versionId=null 2023-05-31T20:55:00.513 DEBUG:boto:Data: 2023-05-31T20:55:00.514 DEBUG:boto:Headers: {} 2023-05-31T20:55:00.514 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:55:00.514 DEBUG:boto:Port: 8001 2023-05-31T20:55:00.515 DEBUG:boto:Params: {} 2023-05-31T20:55:00.515 DEBUG:boto:Token: None 2023-05-31T20:55:00.516 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:55:00 GMT /pklxkd-59/obj74?versionId=null 2023-05-31T20:55:00.516 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:AEtkP+C3564Rc/I6NCJzN50rAxc= 2023-05-31T20:55:00.516 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:55:00 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:AEtkP+C3564Rc/I6NCJzN50rAxc=', 'Content-Length': '0'} 2023-05-31T20:55:00.518 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:52:14 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003548dc57afd17364-006477b424-1e2e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:55:00 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:55:00.519 DEBUG:boto:path=/pklxkd-59/obj74 2023-05-31T20:55:00.519 DEBUG:boto:auth_path=/pklxkd-59/obj74 2023-05-31T20:55:00.519 DEBUG:boto:path=/pklxkd-59/obj74?versionId=null 2023-05-31T20:55:00.520 DEBUG:boto:auth_path=/pklxkd-59/obj74?versionId=null 2023-05-31T20:55:00.520 DEBUG:boto:Method: GET 2023-05-31T20:55:00.521 DEBUG:boto:Path: /pklxkd-59/obj74?versionId=null 2023-05-31T20:55:00.521 DEBUG:boto:Data: 2023-05-31T20:55:00.521 DEBUG:boto:Headers: {} 2023-05-31T20:55:00.522 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:55:00.522 DEBUG:boto:Port: 8000 2023-05-31T20:55:00.522 DEBUG:boto:Params: {} 2023-05-31T20:55:00.523 DEBUG:boto:Token: None 2023-05-31T20:55:00.523 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:55:00 GMT /pklxkd-59/obj74?versionId=null 2023-05-31T20:55:00.524 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:AEtkP+C3564Rc/I6NCJzN50rAxc= 2023-05-31T20:55:00.524 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:55:00 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:AEtkP+C3564Rc/I6NCJzN50rAxc=', 'Content-Length': '0'} 2023-05-31T20:55:00.526 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:52:14 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002e4499047a223b8e-006477b424-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:55:00 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:55:00.526 DEBUG:boto:path=/pklxkd-59/ 2023-05-31T20:55:00.527 DEBUG:boto:auth_path=/pklxkd-59/ 2023-05-31T20:55:00.527 DEBUG:boto:path=/pklxkd-59/?prefix=obj74 2023-05-31T20:55:00.528 DEBUG:boto:auth_path=/pklxkd-59/?prefix=obj74 2023-05-31T20:55:00.528 DEBUG:boto:Method: GET 2023-05-31T20:55:00.528 DEBUG:boto:Path: /pklxkd-59/?prefix=obj74 2023-05-31T20:55:00.529 DEBUG:boto:Data: 2023-05-31T20:55:00.529 DEBUG:boto:Headers: {} 2023-05-31T20:55:00.529 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:55:00.530 DEBUG:boto:Port: 8001 2023-05-31T20:55:00.530 DEBUG:boto:Params: {} 2023-05-31T20:55:00.531 DEBUG:boto:Token: None 2023-05-31T20:55:00.531 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:55:00 GMT /pklxkd-59/ 2023-05-31T20:55:00.531 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:3AIDo5PzlXImNUFq+iJAv4ABZFs= 2023-05-31T20:55:00.532 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:55:00 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:3AIDo5PzlXImNUFq+iJAv4ABZFs=', 'Content-Length': '0'} 2023-05-31T20:55:00.534 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000005c12de694545ac09-006477b424-1e2e-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:55:00 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:55:00.535 DEBUG:boto:b'pklxkd-59obj741000falseobj742023-05-31T20:52:14.934Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:55:00.536 DEBUG:boto:path=/pklxkd-59/ 2023-05-31T20:55:00.536 DEBUG:boto:auth_path=/pklxkd-59/ 2023-05-31T20:55:00.536 DEBUG:boto:path=/pklxkd-59/?prefix=obj74 2023-05-31T20:55:00.537 DEBUG:boto:auth_path=/pklxkd-59/?prefix=obj74 2023-05-31T20:55:00.537 DEBUG:boto:Method: GET 2023-05-31T20:55:00.538 DEBUG:boto:Path: /pklxkd-59/?prefix=obj74 2023-05-31T20:55:00.538 DEBUG:boto:Data: 2023-05-31T20:55:00.538 DEBUG:boto:Headers: {} 2023-05-31T20:55:00.539 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:55:00.539 DEBUG:boto:Port: 8000 2023-05-31T20:55:00.539 DEBUG:boto:Params: {} 2023-05-31T20:55:00.540 DEBUG:boto:Token: None 2023-05-31T20:55:00.540 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:55:00 GMT /pklxkd-59/ 2023-05-31T20:55:00.541 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:3AIDo5PzlXImNUFq+iJAv4ABZFs= 2023-05-31T20:55:00.541 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:55:00 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:3AIDo5PzlXImNUFq+iJAv4ABZFs=', 'Content-Length': '0'} 2023-05-31T20:55:00.545 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ee9f648e8a8bc397-006477b424-1630-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:55:00 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:55:00.545 DEBUG:boto:b'pklxkd-59obj741000falseobj742023-05-31T20:52:14.934Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:55:00.546 DEBUG:boto:path=/pklxkd-59/obj74 2023-05-31T20:55:00.546 DEBUG:boto:auth_path=/pklxkd-59/obj74 2023-05-31T20:55:00.547 DEBUG:boto:Method: HEAD 2023-05-31T20:55:00.547 DEBUG:boto:Path: /pklxkd-59/obj74 2023-05-31T20:55:00.548 DEBUG:boto:Data: 2023-05-31T20:55:00.548 DEBUG:boto:Headers: {} 2023-05-31T20:55:00.548 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:55:00.549 DEBUG:boto:Port: 8001 2023-05-31T20:55:00.549 DEBUG:boto:Params: {} 2023-05-31T20:55:00.549 DEBUG:boto:Token: None 2023-05-31T20:55:00.550 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:55:00 GMT /pklxkd-59/obj74 2023-05-31T20:55:00.550 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:DL+BpBryK2zlHKQQ8qrYJY4ddE0= 2023-05-31T20:55:00.551 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:55:00 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:DL+BpBryK2zlHKQQ8qrYJY4ddE0=', 'Content-Length': '0'} 2023-05-31T20:55:00.552 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:52:14 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000270a95c8b3b590cc-006477b424-1e2e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:55:00 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:55:00.553 DEBUG:boto:path=/pklxkd-59/obj74 2023-05-31T20:55:00.553 DEBUG:boto:auth_path=/pklxkd-59/obj74 2023-05-31T20:55:00.554 DEBUG:boto:Method: HEAD 2023-05-31T20:55:00.554 DEBUG:boto:Path: /pklxkd-59/obj74 2023-05-31T20:55:00.554 DEBUG:boto:Data: 2023-05-31T20:55:00.555 DEBUG:boto:Headers: {} 2023-05-31T20:55:00.555 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:55:00.555 DEBUG:boto:Port: 8000 2023-05-31T20:55:00.556 DEBUG:boto:Params: {} 2023-05-31T20:55:00.556 DEBUG:boto:Token: None 2023-05-31T20:55:00.557 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:55:00 GMT /pklxkd-59/obj74 2023-05-31T20:55:00.557 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:DL+BpBryK2zlHKQQ8qrYJY4ddE0= 2023-05-31T20:55:00.557 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:55:00 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:DL+BpBryK2zlHKQQ8qrYJY4ddE0=', 'Content-Length': '0'} 2023-05-31T20:55:00.559 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:52:14 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f9959a2133218454-006477b424-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:55:00 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:55:00.560 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj74 2023-05-31T20:55:00.560 DEBUG:boto:path=/pklxkd-59/obj74 2023-05-31T20:55:00.561 DEBUG:boto:auth_path=/pklxkd-59/obj74 2023-05-31T20:55:00.562 DEBUG:boto:Method: GET 2023-05-31T20:55:00.562 DEBUG:boto:Path: /pklxkd-59/obj74 2023-05-31T20:55:00.563 DEBUG:boto:Data: 2023-05-31T20:55:00.564 DEBUG:boto:Headers: {} 2023-05-31T20:55:00.564 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:55:00.565 DEBUG:boto:Port: 8001 2023-05-31T20:55:00.566 DEBUG:boto:Params: {} 2023-05-31T20:55:00.567 DEBUG:boto:Token: None 2023-05-31T20:55:00.567 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:55:00 GMT /pklxkd-59/obj74 2023-05-31T20:55:00.568 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:2iSt5ptt8+2EM9FgNQiZqbgUbdo= 2023-05-31T20:55:00.569 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:55:00 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:2iSt5ptt8+2EM9FgNQiZqbgUbdo=', 'Content-Length': '0'} 2023-05-31T20:55:00.571 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:52:14 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009eeb9931ff814d01-006477b424-1e2e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:55:00 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:55:00.572 DEBUG:boto:path=/pklxkd-59/obj74 2023-05-31T20:55:00.573 DEBUG:boto:auth_path=/pklxkd-59/obj74 2023-05-31T20:55:00.574 DEBUG:boto:Method: GET 2023-05-31T20:55:00.575 DEBUG:boto:Path: /pklxkd-59/obj74 2023-05-31T20:55:00.575 DEBUG:boto:Data: 2023-05-31T20:55:00.576 DEBUG:boto:Headers: {} 2023-05-31T20:55:00.577 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:55:00.577 DEBUG:boto:Port: 8000 2023-05-31T20:55:00.578 DEBUG:boto:Params: {} 2023-05-31T20:55:00.579 DEBUG:boto:Token: None 2023-05-31T20:55:00.579 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:55:00 GMT /pklxkd-59/obj74 2023-05-31T20:55:00.580 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:2iSt5ptt8+2EM9FgNQiZqbgUbdo= 2023-05-31T20:55:00.581 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:55:00 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:2iSt5ptt8+2EM9FgNQiZqbgUbdo=', 'Content-Length': '0'} 2023-05-31T20:55:00.583 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:52:14 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004609e3f0ce814374-006477b424-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:55:00 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:55:00.584 DEBUG:boto:path=/pklxkd-59/ 2023-05-31T20:55:00.585 DEBUG:boto:auth_path=/pklxkd-59/ 2023-05-31T20:55:00.586 DEBUG:boto:path=/pklxkd-59/?prefix=obj74 2023-05-31T20:55:00.586 DEBUG:boto:auth_path=/pklxkd-59/?prefix=obj74 2023-05-31T20:55:00.587 DEBUG:boto:Method: GET 2023-05-31T20:55:00.588 DEBUG:boto:Path: /pklxkd-59/?prefix=obj74 2023-05-31T20:55:00.588 DEBUG:boto:Data: 2023-05-31T20:55:00.589 DEBUG:boto:Headers: {} 2023-05-31T20:55:00.590 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:55:00.590 DEBUG:boto:Port: 8001 2023-05-31T20:55:00.591 DEBUG:boto:Params: {} 2023-05-31T20:55:00.592 DEBUG:boto:Token: None 2023-05-31T20:55:00.593 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:55:00 GMT /pklxkd-59/ 2023-05-31T20:55:00.593 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:3AIDo5PzlXImNUFq+iJAv4ABZFs= 2023-05-31T20:55:00.594 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:55:00 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:3AIDo5PzlXImNUFq+iJAv4ABZFs=', 'Content-Length': '0'} 2023-05-31T20:55:00.597 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000007d3c4ec50fbf9e4b-006477b424-1e2e-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:55:00 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:55:00.598 DEBUG:boto:b'pklxkd-59obj741000falseobj742023-05-31T20:52:14.934Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:55:00.599 DEBUG:boto:path=/pklxkd-59/ 2023-05-31T20:55:00.599 DEBUG:boto:auth_path=/pklxkd-59/ 2023-05-31T20:55:00.600 DEBUG:boto:path=/pklxkd-59/?prefix=obj74 2023-05-31T20:55:00.601 DEBUG:boto:auth_path=/pklxkd-59/?prefix=obj74 2023-05-31T20:55:00.601 DEBUG:boto:Method: GET 2023-05-31T20:55:00.602 DEBUG:boto:Path: /pklxkd-59/?prefix=obj74 2023-05-31T20:55:00.603 DEBUG:boto:Data: 2023-05-31T20:55:00.604 DEBUG:boto:Headers: {} 2023-05-31T20:55:00.604 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:55:00.605 DEBUG:boto:Port: 8000 2023-05-31T20:55:00.606 DEBUG:boto:Params: {} 2023-05-31T20:55:00.606 DEBUG:boto:Token: None 2023-05-31T20:55:00.607 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:55:00 GMT /pklxkd-59/ 2023-05-31T20:55:00.608 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:3AIDo5PzlXImNUFq+iJAv4ABZFs= 2023-05-31T20:55:00.609 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:55:00 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:3AIDo5PzlXImNUFq+iJAv4ABZFs=', 'Content-Length': '0'} 2023-05-31T20:55:00.613 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000088bcd96429d75ab4-006477b424-1630-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:55:00 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:55:00.614 DEBUG:boto:b'pklxkd-59obj741000falseobj742023-05-31T20:52:14.934Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:55:00.615 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj76 2023-05-31T20:55:00.615 DEBUG:boto:path=/pklxkd-59/obj76 2023-05-31T20:55:00.616 DEBUG:boto:auth_path=/pklxkd-59/obj76 2023-05-31T20:55:00.617 DEBUG:boto:path=/pklxkd-59/obj76?versionId=null 2023-05-31T20:55:00.617 DEBUG:boto:auth_path=/pklxkd-59/obj76?versionId=null 2023-05-31T20:55:00.618 DEBUG:boto:Method: GET 2023-05-31T20:55:00.619 DEBUG:boto:Path: /pklxkd-59/obj76?versionId=null 2023-05-31T20:55:00.619 DEBUG:boto:Data: 2023-05-31T20:55:00.620 DEBUG:boto:Headers: {} 2023-05-31T20:55:00.621 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:55:00.621 DEBUG:boto:Port: 8001 2023-05-31T20:55:00.622 DEBUG:boto:Params: {} 2023-05-31T20:55:00.623 DEBUG:boto:Token: None 2023-05-31T20:55:00.624 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:55:00 GMT /pklxkd-59/obj76?versionId=null 2023-05-31T20:55:00.624 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:MLlWHml4wcOb2BFM3vBpTCrrOg8= 2023-05-31T20:55:00.625 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:55:00 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:MLlWHml4wcOb2BFM3vBpTCrrOg8=', 'Content-Length': '0'} 2023-05-31T20:55:00.628 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:44:50 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006b9ad7a5571dbcf5-006477b424-1e2e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:55:00 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:55:00.668 DEBUG:boto:path=/pklxkd-59/obj76 2023-05-31T20:55:00.669 DEBUG:boto:auth_path=/pklxkd-59/obj76 2023-05-31T20:55:00.669 DEBUG:boto:path=/pklxkd-59/obj76?versionId=null 2023-05-31T20:55:00.669 DEBUG:boto:auth_path=/pklxkd-59/obj76?versionId=null 2023-05-31T20:55:00.670 DEBUG:boto:Method: GET 2023-05-31T20:55:00.670 DEBUG:boto:Path: /pklxkd-59/obj76?versionId=null 2023-05-31T20:55:00.670 DEBUG:boto:Data: 2023-05-31T20:55:00.671 DEBUG:boto:Headers: {} 2023-05-31T20:55:00.671 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:55:00.672 DEBUG:boto:Port: 8000 2023-05-31T20:55:00.672 DEBUG:boto:Params: {} 2023-05-31T20:55:00.672 DEBUG:boto:Token: None 2023-05-31T20:55:00.673 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:55:00 GMT /pklxkd-59/obj76?versionId=null 2023-05-31T20:55:00.673 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:MLlWHml4wcOb2BFM3vBpTCrrOg8= 2023-05-31T20:55:00.674 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:55:00 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:MLlWHml4wcOb2BFM3vBpTCrrOg8=', 'Content-Length': '0'} 2023-05-31T20:55:00.675 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:44:50 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002e75d7f83b9508a3-006477b424-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:55:00 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:55:00.676 DEBUG:boto:path=/pklxkd-59/obj76 2023-05-31T20:55:00.677 DEBUG:boto:auth_path=/pklxkd-59/obj76 2023-05-31T20:55:00.677 DEBUG:boto:path=/pklxkd-59/obj76?versionId=null 2023-05-31T20:55:00.677 DEBUG:boto:auth_path=/pklxkd-59/obj76?versionId=null 2023-05-31T20:55:00.678 DEBUG:boto:Method: HEAD 2023-05-31T20:55:00.678 DEBUG:boto:Path: /pklxkd-59/obj76?versionId=null 2023-05-31T20:55:00.678 DEBUG:boto:Data: 2023-05-31T20:55:00.679 DEBUG:boto:Headers: {} 2023-05-31T20:55:00.679 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:55:00.679 DEBUG:boto:Port: 8001 2023-05-31T20:55:00.679 DEBUG:boto:Params: {} 2023-05-31T20:55:00.680 DEBUG:boto:Token: None 2023-05-31T20:55:00.680 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:55:00 GMT /pklxkd-59/obj76?versionId=null 2023-05-31T20:55:00.681 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:cJKZJkm9+TbIsMYi3pgJDrQMPig= 2023-05-31T20:55:00.681 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:55:00 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:cJKZJkm9+TbIsMYi3pgJDrQMPig=', 'Content-Length': '0'} 2023-05-31T20:55:00.683 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:44:50 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005c95a257d5555b00-006477b424-1e2e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:55:00 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:55:00.683 DEBUG:boto:path=/pklxkd-59/obj76 2023-05-31T20:55:00.684 DEBUG:boto:auth_path=/pklxkd-59/obj76 2023-05-31T20:55:00.684 DEBUG:boto:path=/pklxkd-59/obj76?versionId=null 2023-05-31T20:55:00.684 DEBUG:boto:auth_path=/pklxkd-59/obj76?versionId=null 2023-05-31T20:55:00.685 DEBUG:boto:Method: HEAD 2023-05-31T20:55:00.685 DEBUG:boto:Path: /pklxkd-59/obj76?versionId=null 2023-05-31T20:55:00.685 DEBUG:boto:Data: 2023-05-31T20:55:00.686 DEBUG:boto:Headers: {} 2023-05-31T20:55:00.686 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:55:00.686 DEBUG:boto:Port: 8000 2023-05-31T20:55:00.687 DEBUG:boto:Params: {} 2023-05-31T20:55:00.687 DEBUG:boto:Token: None 2023-05-31T20:55:00.688 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:55:00 GMT /pklxkd-59/obj76?versionId=null 2023-05-31T20:55:00.688 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:cJKZJkm9+TbIsMYi3pgJDrQMPig= 2023-05-31T20:55:00.688 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:55:00 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:cJKZJkm9+TbIsMYi3pgJDrQMPig=', 'Content-Length': '0'} 2023-05-31T20:55:00.690 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:44:50 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000bb36dc22b0845857-006477b424-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:55:00 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:55:00.691 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj76 2023-05-31T20:55:00.691 DEBUG:boto:path=/pklxkd-59/obj76 2023-05-31T20:55:00.691 DEBUG:boto:auth_path=/pklxkd-59/obj76 2023-05-31T20:55:00.692 DEBUG:boto:path=/pklxkd-59/obj76?versionId=null 2023-05-31T20:55:00.692 DEBUG:boto:auth_path=/pklxkd-59/obj76?versionId=null 2023-05-31T20:55:00.693 DEBUG:boto:Method: GET 2023-05-31T20:55:00.693 DEBUG:boto:Path: /pklxkd-59/obj76?versionId=null 2023-05-31T20:55:00.693 DEBUG:boto:Data: 2023-05-31T20:55:00.694 DEBUG:boto:Headers: {} 2023-05-31T20:55:00.694 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:55:00.694 DEBUG:boto:Port: 8001 2023-05-31T20:55:00.695 DEBUG:boto:Params: {} 2023-05-31T20:55:00.695 DEBUG:boto:Token: None 2023-05-31T20:55:00.696 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:55:00 GMT /pklxkd-59/obj76?versionId=null 2023-05-31T20:55:00.696 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:MLlWHml4wcOb2BFM3vBpTCrrOg8= 2023-05-31T20:55:00.696 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:55:00 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:MLlWHml4wcOb2BFM3vBpTCrrOg8=', 'Content-Length': '0'} 2023-05-31T20:55:00.698 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:44:50 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000066ff11324b8b0be-006477b424-1e2e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:55:00 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:55:00.699 DEBUG:boto:path=/pklxkd-59/obj76 2023-05-31T20:55:00.699 DEBUG:boto:auth_path=/pklxkd-59/obj76 2023-05-31T20:55:00.699 DEBUG:boto:path=/pklxkd-59/obj76?versionId=null 2023-05-31T20:55:00.700 DEBUG:boto:auth_path=/pklxkd-59/obj76?versionId=null 2023-05-31T20:55:00.700 DEBUG:boto:Method: GET 2023-05-31T20:55:00.701 DEBUG:boto:Path: /pklxkd-59/obj76?versionId=null 2023-05-31T20:55:00.701 DEBUG:boto:Data: 2023-05-31T20:55:00.701 DEBUG:boto:Headers: {} 2023-05-31T20:55:00.702 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:55:00.702 DEBUG:boto:Port: 8000 2023-05-31T20:55:00.702 DEBUG:boto:Params: {} 2023-05-31T20:55:00.703 DEBUG:boto:Token: None 2023-05-31T20:55:00.703 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:55:00 GMT /pklxkd-59/obj76?versionId=null 2023-05-31T20:55:00.704 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:MLlWHml4wcOb2BFM3vBpTCrrOg8= 2023-05-31T20:55:00.704 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:55:00 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:MLlWHml4wcOb2BFM3vBpTCrrOg8=', 'Content-Length': '0'} 2023-05-31T20:55:00.706 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:44:50 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005bc88753a5387f06-006477b424-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:55:00 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:55:00.706 DEBUG:boto:path=/pklxkd-59/ 2023-05-31T20:55:00.707 DEBUG:boto:auth_path=/pklxkd-59/ 2023-05-31T20:55:00.707 DEBUG:boto:path=/pklxkd-59/?prefix=obj76 2023-05-31T20:55:00.708 DEBUG:boto:auth_path=/pklxkd-59/?prefix=obj76 2023-05-31T20:55:00.708 DEBUG:boto:Method: GET 2023-05-31T20:55:00.708 DEBUG:boto:Path: /pklxkd-59/?prefix=obj76 2023-05-31T20:55:00.709 DEBUG:boto:Data: 2023-05-31T20:55:00.709 DEBUG:boto:Headers: {} 2023-05-31T20:55:00.709 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:55:00.710 DEBUG:boto:Port: 8001 2023-05-31T20:55:00.710 DEBUG:boto:Params: {} 2023-05-31T20:55:00.710 DEBUG:boto:Token: None 2023-05-31T20:55:00.711 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:55:00 GMT /pklxkd-59/ 2023-05-31T20:55:00.711 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:3AIDo5PzlXImNUFq+iJAv4ABZFs= 2023-05-31T20:55:00.712 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:55:00 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:3AIDo5PzlXImNUFq+iJAv4ABZFs=', 'Content-Length': '0'} 2023-05-31T20:55:00.714 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000005e34a8b82d93683-006477b424-1e2e-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:55:00 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:55:00.715 DEBUG:boto:b'pklxkd-59obj761000falseobj762023-05-31T20:44:50.682Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:55:00.715 DEBUG:boto:path=/pklxkd-59/ 2023-05-31T20:55:00.716 DEBUG:boto:auth_path=/pklxkd-59/ 2023-05-31T20:55:00.716 DEBUG:boto:path=/pklxkd-59/?prefix=obj76 2023-05-31T20:55:00.717 DEBUG:boto:auth_path=/pklxkd-59/?prefix=obj76 2023-05-31T20:55:00.717 DEBUG:boto:Method: GET 2023-05-31T20:55:00.717 DEBUG:boto:Path: /pklxkd-59/?prefix=obj76 2023-05-31T20:55:00.718 DEBUG:boto:Data: 2023-05-31T20:55:00.718 DEBUG:boto:Headers: {} 2023-05-31T20:55:00.718 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:55:00.719 DEBUG:boto:Port: 8000 2023-05-31T20:55:00.719 DEBUG:boto:Params: {} 2023-05-31T20:55:00.719 DEBUG:boto:Token: None 2023-05-31T20:55:00.720 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:55:00 GMT /pklxkd-59/ 2023-05-31T20:55:00.720 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:3AIDo5PzlXImNUFq+iJAv4ABZFs= 2023-05-31T20:55:00.721 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:55:00 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:3AIDo5PzlXImNUFq+iJAv4ABZFs=', 'Content-Length': '0'} 2023-05-31T20:55:00.725 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ffdd0ba712dfed81-006477b424-1630-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:55:00 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:55:00.725 DEBUG:boto:b'pklxkd-59obj761000falseobj762023-05-31T20:44:50.682Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:55:00.726 DEBUG:boto:path=/pklxkd-59/obj76 2023-05-31T20:55:00.726 DEBUG:boto:auth_path=/pklxkd-59/obj76 2023-05-31T20:55:00.727 DEBUG:boto:Method: HEAD 2023-05-31T20:55:00.727 DEBUG:boto:Path: /pklxkd-59/obj76 2023-05-31T20:55:00.727 DEBUG:boto:Data: 2023-05-31T20:55:00.728 DEBUG:boto:Headers: {} 2023-05-31T20:55:00.728 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:55:00.729 DEBUG:boto:Port: 8001 2023-05-31T20:55:00.729 DEBUG:boto:Params: {} 2023-05-31T20:55:00.729 DEBUG:boto:Token: None 2023-05-31T20:55:00.730 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:55:00 GMT /pklxkd-59/obj76 2023-05-31T20:55:00.730 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:TwFzCy5yalVLUEXEtQ4asAsVP1A= 2023-05-31T20:55:00.730 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:55:00 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:TwFzCy5yalVLUEXEtQ4asAsVP1A=', 'Content-Length': '0'} 2023-05-31T20:55:00.732 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:44:50 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000805e88037aef2914-006477b424-1e2e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:55:00 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:55:00.733 DEBUG:boto:path=/pklxkd-59/obj76 2023-05-31T20:55:00.733 DEBUG:boto:auth_path=/pklxkd-59/obj76 2023-05-31T20:55:00.734 DEBUG:boto:Method: HEAD 2023-05-31T20:55:00.734 DEBUG:boto:Path: /pklxkd-59/obj76 2023-05-31T20:55:00.734 DEBUG:boto:Data: 2023-05-31T20:55:00.735 DEBUG:boto:Headers: {} 2023-05-31T20:55:00.735 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:55:00.735 DEBUG:boto:Port: 8000 2023-05-31T20:55:00.736 DEBUG:boto:Params: {} 2023-05-31T20:55:00.736 DEBUG:boto:Token: None 2023-05-31T20:55:00.736 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:55:00 GMT /pklxkd-59/obj76 2023-05-31T20:55:00.737 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:TwFzCy5yalVLUEXEtQ4asAsVP1A= 2023-05-31T20:55:00.737 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:55:00 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:TwFzCy5yalVLUEXEtQ4asAsVP1A=', 'Content-Length': '0'} 2023-05-31T20:55:00.739 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:44:50 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000099deb1f86f7404b4-006477b424-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:55:00 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:55:00.739 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj76 2023-05-31T20:55:00.740 DEBUG:boto:path=/pklxkd-59/obj76 2023-05-31T20:55:00.740 DEBUG:boto:auth_path=/pklxkd-59/obj76 2023-05-31T20:55:00.741 DEBUG:boto:Method: GET 2023-05-31T20:55:00.741 DEBUG:boto:Path: /pklxkd-59/obj76 2023-05-31T20:55:00.741 DEBUG:boto:Data: 2023-05-31T20:55:00.742 DEBUG:boto:Headers: {} 2023-05-31T20:55:00.742 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:55:00.742 DEBUG:boto:Port: 8001 2023-05-31T20:55:00.743 DEBUG:boto:Params: {} 2023-05-31T20:55:00.743 DEBUG:boto:Token: None 2023-05-31T20:55:00.744 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:55:00 GMT /pklxkd-59/obj76 2023-05-31T20:55:00.744 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:szeXagYwn/zUuMwkNsURyLJgskw= 2023-05-31T20:55:00.745 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:55:00 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:szeXagYwn/zUuMwkNsURyLJgskw=', 'Content-Length': '0'} 2023-05-31T20:55:00.746 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:44:50 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003c2bc730cb96682e-006477b424-1e2e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:55:00 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:55:00.747 DEBUG:boto:path=/pklxkd-59/obj76 2023-05-31T20:55:00.747 DEBUG:boto:auth_path=/pklxkd-59/obj76 2023-05-31T20:55:00.748 DEBUG:boto:Method: GET 2023-05-31T20:55:00.748 DEBUG:boto:Path: /pklxkd-59/obj76 2023-05-31T20:55:00.748 DEBUG:boto:Data: 2023-05-31T20:55:00.749 DEBUG:boto:Headers: {} 2023-05-31T20:55:00.749 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:55:00.749 DEBUG:boto:Port: 8000 2023-05-31T20:55:00.750 DEBUG:boto:Params: {} 2023-05-31T20:55:00.750 DEBUG:boto:Token: None 2023-05-31T20:55:00.751 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:55:00 GMT /pklxkd-59/obj76 2023-05-31T20:55:00.751 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:szeXagYwn/zUuMwkNsURyLJgskw= 2023-05-31T20:55:00.751 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:55:00 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:szeXagYwn/zUuMwkNsURyLJgskw=', 'Content-Length': '0'} 2023-05-31T20:55:00.753 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:44:50 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c1f792b141a4519c-006477b424-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:55:00 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:55:00.796 DEBUG:boto:path=/pklxkd-59/ 2023-05-31T20:55:00.797 DEBUG:boto:auth_path=/pklxkd-59/ 2023-05-31T20:55:00.797 DEBUG:boto:path=/pklxkd-59/?prefix=obj76 2023-05-31T20:55:00.798 DEBUG:boto:auth_path=/pklxkd-59/?prefix=obj76 2023-05-31T20:55:00.798 DEBUG:boto:Method: GET 2023-05-31T20:55:00.798 DEBUG:boto:Path: /pklxkd-59/?prefix=obj76 2023-05-31T20:55:00.799 DEBUG:boto:Data: 2023-05-31T20:55:00.799 DEBUG:boto:Headers: {} 2023-05-31T20:55:00.799 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:55:00.800 DEBUG:boto:Port: 8001 2023-05-31T20:55:00.800 DEBUG:boto:Params: {} 2023-05-31T20:55:00.801 DEBUG:boto:Token: None 2023-05-31T20:55:00.801 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:55:00 GMT /pklxkd-59/ 2023-05-31T20:55:00.801 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:3AIDo5PzlXImNUFq+iJAv4ABZFs= 2023-05-31T20:55:00.802 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:55:00 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:3AIDo5PzlXImNUFq+iJAv4ABZFs=', 'Content-Length': '0'} 2023-05-31T20:55:00.804 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000019da9a89d645a60f-006477b424-1e2e-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:55:00 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:55:00.805 DEBUG:boto:b'pklxkd-59obj761000falseobj762023-05-31T20:44:50.682Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:55:00.805 DEBUG:boto:path=/pklxkd-59/ 2023-05-31T20:55:00.806 DEBUG:boto:auth_path=/pklxkd-59/ 2023-05-31T20:55:00.806 DEBUG:boto:path=/pklxkd-59/?prefix=obj76 2023-05-31T20:55:00.806 DEBUG:boto:auth_path=/pklxkd-59/?prefix=obj76 2023-05-31T20:55:00.807 DEBUG:boto:Method: GET 2023-05-31T20:55:00.807 DEBUG:boto:Path: /pklxkd-59/?prefix=obj76 2023-05-31T20:55:00.807 DEBUG:boto:Data: 2023-05-31T20:55:00.808 DEBUG:boto:Headers: {} 2023-05-31T20:55:00.808 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:55:00.809 DEBUG:boto:Port: 8000 2023-05-31T20:55:00.809 DEBUG:boto:Params: {} 2023-05-31T20:55:00.809 DEBUG:boto:Token: None 2023-05-31T20:55:00.810 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:55:00 GMT /pklxkd-59/ 2023-05-31T20:55:00.810 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:3AIDo5PzlXImNUFq+iJAv4ABZFs= 2023-05-31T20:55:00.811 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:55:00 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:3AIDo5PzlXImNUFq+iJAv4ABZFs=', 'Content-Length': '0'} 2023-05-31T20:55:00.815 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b8f7f50f5a168fe6-006477b424-1630-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:55:00 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:55:00.815 DEBUG:boto:b'pklxkd-59obj761000falseobj762023-05-31T20:44:50.682Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:55:00.816 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj85 2023-05-31T20:55:00.817 DEBUG:boto:path=/pklxkd-59/obj85 2023-05-31T20:55:00.817 DEBUG:boto:auth_path=/pklxkd-59/obj85 2023-05-31T20:55:00.818 DEBUG:boto:path=/pklxkd-59/obj85?versionId=null 2023-05-31T20:55:00.818 DEBUG:boto:auth_path=/pklxkd-59/obj85?versionId=null 2023-05-31T20:55:00.818 DEBUG:boto:Method: GET 2023-05-31T20:55:00.819 DEBUG:boto:Path: /pklxkd-59/obj85?versionId=null 2023-05-31T20:55:00.819 DEBUG:boto:Data: 2023-05-31T20:55:00.819 DEBUG:boto:Headers: {} 2023-05-31T20:55:00.820 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:55:00.820 DEBUG:boto:Port: 8001 2023-05-31T20:55:00.820 DEBUG:boto:Params: {} 2023-05-31T20:55:00.821 DEBUG:boto:Token: None 2023-05-31T20:55:00.821 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:55:00 GMT /pklxkd-59/obj85?versionId=null 2023-05-31T20:55:00.822 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:a3NdKZSYiz5n18vQtxRmYc7yxw8= 2023-05-31T20:55:00.822 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:55:00 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:a3NdKZSYiz5n18vQtxRmYc7yxw8=', 'Content-Length': '0'} 2023-05-31T20:55:00.824 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:43:08 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009465017d14f7c8e3-006477b424-1e2e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:55:00 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:55:00.868 DEBUG:boto:path=/pklxkd-59/obj85 2023-05-31T20:55:00.869 DEBUG:boto:auth_path=/pklxkd-59/obj85 2023-05-31T20:55:00.869 DEBUG:boto:path=/pklxkd-59/obj85?versionId=null 2023-05-31T20:55:00.869 DEBUG:boto:auth_path=/pklxkd-59/obj85?versionId=null 2023-05-31T20:55:00.870 DEBUG:boto:Method: GET 2023-05-31T20:55:00.870 DEBUG:boto:Path: /pklxkd-59/obj85?versionId=null 2023-05-31T20:55:00.870 DEBUG:boto:Data: 2023-05-31T20:55:00.871 DEBUG:boto:Headers: {} 2023-05-31T20:55:00.871 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:55:00.872 DEBUG:boto:Port: 8000 2023-05-31T20:55:00.872 DEBUG:boto:Params: {} 2023-05-31T20:55:00.873 DEBUG:boto:Token: None 2023-05-31T20:55:00.873 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:55:00 GMT /pklxkd-59/obj85?versionId=null 2023-05-31T20:55:00.873 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:a3NdKZSYiz5n18vQtxRmYc7yxw8= 2023-05-31T20:55:00.874 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:55:00 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:a3NdKZSYiz5n18vQtxRmYc7yxw8=', 'Content-Length': '0'} 2023-05-31T20:55:00.876 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:43:08 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002ca11a0f56e8280e-006477b424-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:55:00 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:55:00.877 DEBUG:boto:path=/pklxkd-59/obj85 2023-05-31T20:55:00.877 DEBUG:boto:auth_path=/pklxkd-59/obj85 2023-05-31T20:55:00.877 DEBUG:boto:path=/pklxkd-59/obj85?versionId=null 2023-05-31T20:55:00.878 DEBUG:boto:auth_path=/pklxkd-59/obj85?versionId=null 2023-05-31T20:55:00.878 DEBUG:boto:Method: HEAD 2023-05-31T20:55:00.878 DEBUG:boto:Path: /pklxkd-59/obj85?versionId=null 2023-05-31T20:55:00.879 DEBUG:boto:Data: 2023-05-31T20:55:00.879 DEBUG:boto:Headers: {} 2023-05-31T20:55:00.879 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:55:00.880 DEBUG:boto:Port: 8001 2023-05-31T20:55:00.880 DEBUG:boto:Params: {} 2023-05-31T20:55:00.880 DEBUG:boto:Token: None 2023-05-31T20:55:00.881 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:55:00 GMT /pklxkd-59/obj85?versionId=null 2023-05-31T20:55:00.881 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:8/uExEyQdxgNwOuPrOA7FfbU8IY= 2023-05-31T20:55:00.882 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:55:00 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:8/uExEyQdxgNwOuPrOA7FfbU8IY=', 'Content-Length': '0'} 2023-05-31T20:55:00.883 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:43:08 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000bd94dc72f9d961e4-006477b424-1e2e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:55:00 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:55:00.884 DEBUG:boto:path=/pklxkd-59/obj85 2023-05-31T20:55:00.884 DEBUG:boto:auth_path=/pklxkd-59/obj85 2023-05-31T20:55:00.885 DEBUG:boto:path=/pklxkd-59/obj85?versionId=null 2023-05-31T20:55:00.885 DEBUG:boto:auth_path=/pklxkd-59/obj85?versionId=null 2023-05-31T20:55:00.885 DEBUG:boto:Method: HEAD 2023-05-31T20:55:00.886 DEBUG:boto:Path: /pklxkd-59/obj85?versionId=null 2023-05-31T20:55:00.886 DEBUG:boto:Data: 2023-05-31T20:55:00.886 DEBUG:boto:Headers: {} 2023-05-31T20:55:00.887 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:55:00.887 DEBUG:boto:Port: 8000 2023-05-31T20:55:00.887 DEBUG:boto:Params: {} 2023-05-31T20:55:00.888 DEBUG:boto:Token: None 2023-05-31T20:55:00.888 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:55:00 GMT /pklxkd-59/obj85?versionId=null 2023-05-31T20:55:00.889 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:8/uExEyQdxgNwOuPrOA7FfbU8IY= 2023-05-31T20:55:00.889 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:55:00 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:8/uExEyQdxgNwOuPrOA7FfbU8IY=', 'Content-Length': '0'} 2023-05-31T20:55:00.891 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:43:08 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a3ede6129f82ae2f-006477b424-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:55:00 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:55:00.891 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj85 2023-05-31T20:55:00.892 DEBUG:boto:path=/pklxkd-59/obj85 2023-05-31T20:55:00.892 DEBUG:boto:auth_path=/pklxkd-59/obj85 2023-05-31T20:55:00.892 DEBUG:boto:path=/pklxkd-59/obj85?versionId=null 2023-05-31T20:55:00.893 DEBUG:boto:auth_path=/pklxkd-59/obj85?versionId=null 2023-05-31T20:55:00.893 DEBUG:boto:Method: GET 2023-05-31T20:55:00.893 DEBUG:boto:Path: /pklxkd-59/obj85?versionId=null 2023-05-31T20:55:00.894 DEBUG:boto:Data: 2023-05-31T20:55:00.894 DEBUG:boto:Headers: {} 2023-05-31T20:55:00.894 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:55:00.895 DEBUG:boto:Port: 8001 2023-05-31T20:55:00.895 DEBUG:boto:Params: {} 2023-05-31T20:55:00.896 DEBUG:boto:Token: None 2023-05-31T20:55:00.896 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:55:00 GMT /pklxkd-59/obj85?versionId=null 2023-05-31T20:55:00.896 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:a3NdKZSYiz5n18vQtxRmYc7yxw8= 2023-05-31T20:55:00.897 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:55:00 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:a3NdKZSYiz5n18vQtxRmYc7yxw8=', 'Content-Length': '0'} 2023-05-31T20:55:00.899 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:43:08 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000bf25dc7720c5b24d-006477b424-1e2e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:55:00 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:55:00.899 DEBUG:boto:path=/pklxkd-59/obj85 2023-05-31T20:55:00.900 DEBUG:boto:auth_path=/pklxkd-59/obj85 2023-05-31T20:55:00.900 DEBUG:boto:path=/pklxkd-59/obj85?versionId=null 2023-05-31T20:55:00.900 DEBUG:boto:auth_path=/pklxkd-59/obj85?versionId=null 2023-05-31T20:55:00.901 DEBUG:boto:Method: GET 2023-05-31T20:55:00.901 DEBUG:boto:Path: /pklxkd-59/obj85?versionId=null 2023-05-31T20:55:00.901 DEBUG:boto:Data: 2023-05-31T20:55:00.902 DEBUG:boto:Headers: {} 2023-05-31T20:55:00.902 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:55:00.903 DEBUG:boto:Port: 8000 2023-05-31T20:55:00.903 DEBUG:boto:Params: {} 2023-05-31T20:55:00.903 DEBUG:boto:Token: None 2023-05-31T20:55:00.904 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:55:00 GMT /pklxkd-59/obj85?versionId=null 2023-05-31T20:55:00.904 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:a3NdKZSYiz5n18vQtxRmYc7yxw8= 2023-05-31T20:55:00.905 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:55:00 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:a3NdKZSYiz5n18vQtxRmYc7yxw8=', 'Content-Length': '0'} 2023-05-31T20:55:00.907 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:43:08 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ed8ce49743b71e72-006477b424-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:55:00 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:55:00.907 DEBUG:boto:path=/pklxkd-59/ 2023-05-31T20:55:00.908 DEBUG:boto:auth_path=/pklxkd-59/ 2023-05-31T20:55:00.908 DEBUG:boto:path=/pklxkd-59/?prefix=obj85 2023-05-31T20:55:00.908 DEBUG:boto:auth_path=/pklxkd-59/?prefix=obj85 2023-05-31T20:55:00.909 DEBUG:boto:Method: GET 2023-05-31T20:55:00.909 DEBUG:boto:Path: /pklxkd-59/?prefix=obj85 2023-05-31T20:55:00.909 DEBUG:boto:Data: 2023-05-31T20:55:00.910 DEBUG:boto:Headers: {} 2023-05-31T20:55:00.910 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:55:00.910 DEBUG:boto:Port: 8001 2023-05-31T20:55:00.911 DEBUG:boto:Params: {} 2023-05-31T20:55:00.911 DEBUG:boto:Token: None 2023-05-31T20:55:00.912 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:55:00 GMT /pklxkd-59/ 2023-05-31T20:55:00.912 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:3AIDo5PzlXImNUFq+iJAv4ABZFs= 2023-05-31T20:55:00.913 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:55:00 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:3AIDo5PzlXImNUFq+iJAv4ABZFs=', 'Content-Length': '0'} 2023-05-31T20:55:00.915 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000067954e6758491143-006477b424-1e2e-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:55:00 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:55:00.915 DEBUG:boto:b'pklxkd-59obj851000falseobj852023-05-31T20:43:08.086Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:55:00.916 DEBUG:boto:path=/pklxkd-59/ 2023-05-31T20:55:00.917 DEBUG:boto:auth_path=/pklxkd-59/ 2023-05-31T20:55:00.917 DEBUG:boto:path=/pklxkd-59/?prefix=obj85 2023-05-31T20:55:00.917 DEBUG:boto:auth_path=/pklxkd-59/?prefix=obj85 2023-05-31T20:55:00.918 DEBUG:boto:Method: GET 2023-05-31T20:55:00.918 DEBUG:boto:Path: /pklxkd-59/?prefix=obj85 2023-05-31T20:55:00.918 DEBUG:boto:Data: 2023-05-31T20:55:00.919 DEBUG:boto:Headers: {} 2023-05-31T20:55:00.919 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:55:00.920 DEBUG:boto:Port: 8000 2023-05-31T20:55:00.920 DEBUG:boto:Params: {} 2023-05-31T20:55:00.920 DEBUG:boto:Token: None 2023-05-31T20:55:00.921 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:55:00 GMT /pklxkd-59/ 2023-05-31T20:55:00.921 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:3AIDo5PzlXImNUFq+iJAv4ABZFs= 2023-05-31T20:55:00.922 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:55:00 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:3AIDo5PzlXImNUFq+iJAv4ABZFs=', 'Content-Length': '0'} 2023-05-31T20:55:00.926 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000db05cf9848ec7cbb-006477b424-1630-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:55:00 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:55:00.926 DEBUG:boto:b'pklxkd-59obj851000falseobj852023-05-31T20:43:08.086Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:55:00.927 DEBUG:boto:path=/pklxkd-59/obj85 2023-05-31T20:55:00.927 DEBUG:boto:auth_path=/pklxkd-59/obj85 2023-05-31T20:55:00.927 DEBUG:boto:Method: HEAD 2023-05-31T20:55:00.928 DEBUG:boto:Path: /pklxkd-59/obj85 2023-05-31T20:55:00.928 DEBUG:boto:Data: 2023-05-31T20:55:00.928 DEBUG:boto:Headers: {} 2023-05-31T20:55:00.929 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:55:00.929 DEBUG:boto:Port: 8001 2023-05-31T20:55:00.929 DEBUG:boto:Params: {} 2023-05-31T20:55:00.930 DEBUG:boto:Token: None 2023-05-31T20:55:00.930 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:55:00 GMT /pklxkd-59/obj85 2023-05-31T20:55:00.931 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:iVA5jmd0UwEUKfEWZgtoKjsnogQ= 2023-05-31T20:55:00.931 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:55:00 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:iVA5jmd0UwEUKfEWZgtoKjsnogQ=', 'Content-Length': '0'} 2023-05-31T20:55:00.933 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:43:08 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000050aa843e92f78ee7-006477b424-1e2e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:55:00 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:55:00.933 DEBUG:boto:path=/pklxkd-59/obj85 2023-05-31T20:55:00.934 DEBUG:boto:auth_path=/pklxkd-59/obj85 2023-05-31T20:55:00.934 DEBUG:boto:Method: HEAD 2023-05-31T20:55:00.934 DEBUG:boto:Path: /pklxkd-59/obj85 2023-05-31T20:55:00.935 DEBUG:boto:Data: 2023-05-31T20:55:00.935 DEBUG:boto:Headers: {} 2023-05-31T20:55:00.935 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:55:00.936 DEBUG:boto:Port: 8000 2023-05-31T20:55:00.936 DEBUG:boto:Params: {} 2023-05-31T20:55:00.937 DEBUG:boto:Token: None 2023-05-31T20:55:00.937 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:55:00 GMT /pklxkd-59/obj85 2023-05-31T20:55:00.937 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:iVA5jmd0UwEUKfEWZgtoKjsnogQ= 2023-05-31T20:55:00.938 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:55:00 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:iVA5jmd0UwEUKfEWZgtoKjsnogQ=', 'Content-Length': '0'} 2023-05-31T20:55:00.940 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:43:08 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000068b720097a55d2e6-006477b424-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:55:00 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:55:00.940 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj85 2023-05-31T20:55:00.941 DEBUG:boto:path=/pklxkd-59/obj85 2023-05-31T20:55:00.941 DEBUG:boto:auth_path=/pklxkd-59/obj85 2023-05-31T20:55:00.941 DEBUG:boto:Method: GET 2023-05-31T20:55:00.942 DEBUG:boto:Path: /pklxkd-59/obj85 2023-05-31T20:55:00.942 DEBUG:boto:Data: 2023-05-31T20:55:00.942 DEBUG:boto:Headers: {} 2023-05-31T20:55:00.943 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:55:00.943 DEBUG:boto:Port: 8001 2023-05-31T20:55:00.943 DEBUG:boto:Params: {} 2023-05-31T20:55:00.944 DEBUG:boto:Token: None 2023-05-31T20:55:00.944 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:55:00 GMT /pklxkd-59/obj85 2023-05-31T20:55:00.945 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:F3F2AUM+fhaYi1VMNHpigAW3/M8= 2023-05-31T20:55:00.945 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:55:00 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:F3F2AUM+fhaYi1VMNHpigAW3/M8=', 'Content-Length': '0'} 2023-05-31T20:55:00.947 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:43:08 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006ca1afe823c248bb-006477b424-1e2e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:55:00 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:55:00.947 DEBUG:boto:path=/pklxkd-59/obj85 2023-05-31T20:55:00.948 DEBUG:boto:auth_path=/pklxkd-59/obj85 2023-05-31T20:55:00.948 DEBUG:boto:Method: GET 2023-05-31T20:55:00.949 DEBUG:boto:Path: /pklxkd-59/obj85 2023-05-31T20:55:00.949 DEBUG:boto:Data: 2023-05-31T20:55:00.949 DEBUG:boto:Headers: {} 2023-05-31T20:55:00.950 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:55:00.950 DEBUG:boto:Port: 8000 2023-05-31T20:55:00.950 DEBUG:boto:Params: {} 2023-05-31T20:55:00.951 DEBUG:boto:Token: None 2023-05-31T20:55:00.951 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:55:00 GMT /pklxkd-59/obj85 2023-05-31T20:55:00.952 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:F3F2AUM+fhaYi1VMNHpigAW3/M8= 2023-05-31T20:55:00.952 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:55:00 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:F3F2AUM+fhaYi1VMNHpigAW3/M8=', 'Content-Length': '0'} 2023-05-31T20:55:00.954 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:43:08 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000657efddc529ea5f1-006477b424-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:55:00 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:55:00.954 DEBUG:boto:path=/pklxkd-59/ 2023-05-31T20:55:00.955 DEBUG:boto:auth_path=/pklxkd-59/ 2023-05-31T20:55:00.955 DEBUG:boto:path=/pklxkd-59/?prefix=obj85 2023-05-31T20:55:00.956 DEBUG:boto:auth_path=/pklxkd-59/?prefix=obj85 2023-05-31T20:55:00.956 DEBUG:boto:Method: GET 2023-05-31T20:55:00.956 DEBUG:boto:Path: /pklxkd-59/?prefix=obj85 2023-05-31T20:55:00.957 DEBUG:boto:Data: 2023-05-31T20:55:00.957 DEBUG:boto:Headers: {} 2023-05-31T20:55:00.957 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:55:00.958 DEBUG:boto:Port: 8001 2023-05-31T20:55:00.958 DEBUG:boto:Params: {} 2023-05-31T20:55:00.958 DEBUG:boto:Token: None 2023-05-31T20:55:00.959 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:55:00 GMT /pklxkd-59/ 2023-05-31T20:55:00.959 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:3AIDo5PzlXImNUFq+iJAv4ABZFs= 2023-05-31T20:55:00.960 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:55:00 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:3AIDo5PzlXImNUFq+iJAv4ABZFs=', 'Content-Length': '0'} 2023-05-31T20:55:00.962 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b266c209a701b76d-006477b424-1e2e-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:55:00 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:55:00.962 DEBUG:boto:b'pklxkd-59obj851000falseobj852023-05-31T20:43:08.086Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:55:00.963 DEBUG:boto:path=/pklxkd-59/ 2023-05-31T20:55:00.963 DEBUG:boto:auth_path=/pklxkd-59/ 2023-05-31T20:55:00.964 DEBUG:boto:path=/pklxkd-59/?prefix=obj85 2023-05-31T20:55:00.964 DEBUG:boto:auth_path=/pklxkd-59/?prefix=obj85 2023-05-31T20:55:00.965 DEBUG:boto:Method: GET 2023-05-31T20:55:00.965 DEBUG:boto:Path: /pklxkd-59/?prefix=obj85 2023-05-31T20:55:00.965 DEBUG:boto:Data: 2023-05-31T20:55:00.966 DEBUG:boto:Headers: {} 2023-05-31T20:55:00.966 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:55:00.966 DEBUG:boto:Port: 8000 2023-05-31T20:55:00.967 DEBUG:boto:Params: {} 2023-05-31T20:55:00.967 DEBUG:boto:Token: None 2023-05-31T20:55:00.968 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:55:00 GMT /pklxkd-59/ 2023-05-31T20:55:00.968 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:3AIDo5PzlXImNUFq+iJAv4ABZFs= 2023-05-31T20:55:00.968 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:55:00 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:3AIDo5PzlXImNUFq+iJAv4ABZFs=', 'Content-Length': '0'} 2023-05-31T20:55:00.972 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000dd1d78ea7f686694-006477b424-1630-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:55:00 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:55:00.973 DEBUG:boto:b'pklxkd-59obj851000falseobj852023-05-31T20:43:08.086Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:55:00.973 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj87 2023-05-31T20:55:00.974 DEBUG:boto:path=/pklxkd-59/obj87 2023-05-31T20:55:00.974 DEBUG:boto:auth_path=/pklxkd-59/obj87 2023-05-31T20:55:00.975 DEBUG:boto:path=/pklxkd-59/obj87?versionId=null 2023-05-31T20:55:00.975 DEBUG:boto:auth_path=/pklxkd-59/obj87?versionId=null 2023-05-31T20:55:00.975 DEBUG:boto:Method: GET 2023-05-31T20:55:00.976 DEBUG:boto:Path: /pklxkd-59/obj87?versionId=null 2023-05-31T20:55:00.976 DEBUG:boto:Data: 2023-05-31T20:55:00.976 DEBUG:boto:Headers: {} 2023-05-31T20:55:00.977 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:55:00.977 DEBUG:boto:Port: 8001 2023-05-31T20:55:00.977 DEBUG:boto:Params: {} 2023-05-31T20:55:00.978 DEBUG:boto:Token: None 2023-05-31T20:55:00.978 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:55:00 GMT /pklxkd-59/obj87?versionId=null 2023-05-31T20:55:00.979 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:24foOpOk2WTcox1hMHhpi2F4MXI= 2023-05-31T20:55:00.979 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:55:00 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:24foOpOk2WTcox1hMHhpi2F4MXI=', 'Content-Length': '0'} 2023-05-31T20:55:00.981 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:48:13 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009a2d14fea2c8f866-006477b424-1e2e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:55:00 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:55:00.982 DEBUG:boto:path=/pklxkd-59/obj87 2023-05-31T20:55:00.982 DEBUG:boto:auth_path=/pklxkd-59/obj87 2023-05-31T20:55:00.983 DEBUG:boto:path=/pklxkd-59/obj87?versionId=null 2023-05-31T20:55:00.983 DEBUG:boto:auth_path=/pklxkd-59/obj87?versionId=null 2023-05-31T20:55:00.983 DEBUG:boto:Method: GET 2023-05-31T20:55:00.984 DEBUG:boto:Path: /pklxkd-59/obj87?versionId=null 2023-05-31T20:55:00.984 DEBUG:boto:Data: 2023-05-31T20:55:00.984 DEBUG:boto:Headers: {} 2023-05-31T20:55:00.985 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:55:00.985 DEBUG:boto:Port: 8000 2023-05-31T20:55:00.985 DEBUG:boto:Params: {} 2023-05-31T20:55:00.986 DEBUG:boto:Token: None 2023-05-31T20:55:00.986 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:55:00 GMT /pklxkd-59/obj87?versionId=null 2023-05-31T20:55:00.987 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:24foOpOk2WTcox1hMHhpi2F4MXI= 2023-05-31T20:55:00.987 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:55:00 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:24foOpOk2WTcox1hMHhpi2F4MXI=', 'Content-Length': '0'} 2023-05-31T20:55:00.989 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:48:13 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000bb4c58f0b597d5f9-006477b424-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:55:00 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:55:01.032 DEBUG:boto:path=/pklxkd-59/obj87 2023-05-31T20:55:01.033 DEBUG:boto:auth_path=/pklxkd-59/obj87 2023-05-31T20:55:01.033 DEBUG:boto:path=/pklxkd-59/obj87?versionId=null 2023-05-31T20:55:01.033 DEBUG:boto:auth_path=/pklxkd-59/obj87?versionId=null 2023-05-31T20:55:01.034 DEBUG:boto:Method: HEAD 2023-05-31T20:55:01.034 DEBUG:boto:Path: /pklxkd-59/obj87?versionId=null 2023-05-31T20:55:01.034 DEBUG:boto:Data: 2023-05-31T20:55:01.035 DEBUG:boto:Headers: {} 2023-05-31T20:55:01.035 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:55:01.035 DEBUG:boto:Port: 8001 2023-05-31T20:55:01.036 DEBUG:boto:Params: {} 2023-05-31T20:55:01.036 DEBUG:boto:Token: None 2023-05-31T20:55:01.036 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:55:01 GMT /pklxkd-59/obj87?versionId=null 2023-05-31T20:55:01.037 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:uyQJp6sEgv/dTOz2FJOsa/PWV8c= 2023-05-31T20:55:01.037 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:55:01 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:uyQJp6sEgv/dTOz2FJOsa/PWV8c=', 'Content-Length': '0'} 2023-05-31T20:55:01.039 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:48:13 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d3e512d2f8dcd3d4-006477b425-1e2e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:55:01 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:55:01.040 DEBUG:boto:path=/pklxkd-59/obj87 2023-05-31T20:55:01.040 DEBUG:boto:auth_path=/pklxkd-59/obj87 2023-05-31T20:55:01.040 DEBUG:boto:path=/pklxkd-59/obj87?versionId=null 2023-05-31T20:55:01.041 DEBUG:boto:auth_path=/pklxkd-59/obj87?versionId=null 2023-05-31T20:55:01.041 DEBUG:boto:Method: HEAD 2023-05-31T20:55:01.041 DEBUG:boto:Path: /pklxkd-59/obj87?versionId=null 2023-05-31T20:55:01.042 DEBUG:boto:Data: 2023-05-31T20:55:01.042 DEBUG:boto:Headers: {} 2023-05-31T20:55:01.042 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:55:01.043 DEBUG:boto:Port: 8000 2023-05-31T20:55:01.043 DEBUG:boto:Params: {} 2023-05-31T20:55:01.043 DEBUG:boto:Token: None 2023-05-31T20:55:01.044 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:55:01 GMT /pklxkd-59/obj87?versionId=null 2023-05-31T20:55:01.044 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:uyQJp6sEgv/dTOz2FJOsa/PWV8c= 2023-05-31T20:55:01.045 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:55:01 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:uyQJp6sEgv/dTOz2FJOsa/PWV8c=', 'Content-Length': '0'} 2023-05-31T20:55:01.046 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:48:13 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e9ce94bbcbd6b2a8-006477b425-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:55:01 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:55:01.047 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj87 2023-05-31T20:55:01.047 DEBUG:boto:path=/pklxkd-59/obj87 2023-05-31T20:55:01.048 DEBUG:boto:auth_path=/pklxkd-59/obj87 2023-05-31T20:55:01.048 DEBUG:boto:path=/pklxkd-59/obj87?versionId=null 2023-05-31T20:55:01.048 DEBUG:boto:auth_path=/pklxkd-59/obj87?versionId=null 2023-05-31T20:55:01.049 DEBUG:boto:Method: GET 2023-05-31T20:55:01.049 DEBUG:boto:Path: /pklxkd-59/obj87?versionId=null 2023-05-31T20:55:01.049 DEBUG:boto:Data: 2023-05-31T20:55:01.050 DEBUG:boto:Headers: {} 2023-05-31T20:55:01.050 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:55:01.050 DEBUG:boto:Port: 8001 2023-05-31T20:55:01.051 DEBUG:boto:Params: {} 2023-05-31T20:55:01.051 DEBUG:boto:Token: None 2023-05-31T20:55:01.052 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:55:01 GMT /pklxkd-59/obj87?versionId=null 2023-05-31T20:55:01.052 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:HKqFiJyao6K5gBTdATe/BFIfDmk= 2023-05-31T20:55:01.052 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:55:01 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:HKqFiJyao6K5gBTdATe/BFIfDmk=', 'Content-Length': '0'} 2023-05-31T20:55:01.054 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:48:13 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009d153a64c506fcdd-006477b425-1e2e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:55:01 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:55:01.055 DEBUG:boto:path=/pklxkd-59/obj87 2023-05-31T20:55:01.055 DEBUG:boto:auth_path=/pklxkd-59/obj87 2023-05-31T20:55:01.056 DEBUG:boto:path=/pklxkd-59/obj87?versionId=null 2023-05-31T20:55:01.056 DEBUG:boto:auth_path=/pklxkd-59/obj87?versionId=null 2023-05-31T20:55:01.056 DEBUG:boto:Method: GET 2023-05-31T20:55:01.057 DEBUG:boto:Path: /pklxkd-59/obj87?versionId=null 2023-05-31T20:55:01.057 DEBUG:boto:Data: 2023-05-31T20:55:01.057 DEBUG:boto:Headers: {} 2023-05-31T20:55:01.058 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:55:01.058 DEBUG:boto:Port: 8000 2023-05-31T20:55:01.058 DEBUG:boto:Params: {} 2023-05-31T20:55:01.059 DEBUG:boto:Token: None 2023-05-31T20:55:01.059 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:55:01 GMT /pklxkd-59/obj87?versionId=null 2023-05-31T20:55:01.060 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:HKqFiJyao6K5gBTdATe/BFIfDmk= 2023-05-31T20:55:01.060 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:55:01 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:HKqFiJyao6K5gBTdATe/BFIfDmk=', 'Content-Length': '0'} 2023-05-31T20:55:01.062 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:48:13 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000608ef6f891b6389e-006477b425-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:55:01 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:55:01.063 DEBUG:boto:path=/pklxkd-59/ 2023-05-31T20:55:01.063 DEBUG:boto:auth_path=/pklxkd-59/ 2023-05-31T20:55:01.063 DEBUG:boto:path=/pklxkd-59/?prefix=obj87 2023-05-31T20:55:01.064 DEBUG:boto:auth_path=/pklxkd-59/?prefix=obj87 2023-05-31T20:55:01.064 DEBUG:boto:Method: GET 2023-05-31T20:55:01.065 DEBUG:boto:Path: /pklxkd-59/?prefix=obj87 2023-05-31T20:55:01.065 DEBUG:boto:Data: 2023-05-31T20:55:01.065 DEBUG:boto:Headers: {} 2023-05-31T20:55:01.066 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:55:01.066 DEBUG:boto:Port: 8001 2023-05-31T20:55:01.066 DEBUG:boto:Params: {} 2023-05-31T20:55:01.067 DEBUG:boto:Token: None 2023-05-31T20:55:01.067 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:55:01 GMT /pklxkd-59/ 2023-05-31T20:55:01.068 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:d7ZqE7fjjpja5oroUaADG51170Q= 2023-05-31T20:55:01.068 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:55:01 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:d7ZqE7fjjpja5oroUaADG51170Q=', 'Content-Length': '0'} 2023-05-31T20:55:01.070 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000af7d0c58b7799c46-006477b425-1e2e-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:55:01 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:55:01.071 DEBUG:boto:b'pklxkd-59obj871000falseobj872023-05-31T20:48:13.031Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:55:01.071 DEBUG:boto:path=/pklxkd-59/ 2023-05-31T20:55:01.072 DEBUG:boto:auth_path=/pklxkd-59/ 2023-05-31T20:55:01.072 DEBUG:boto:path=/pklxkd-59/?prefix=obj87 2023-05-31T20:55:01.073 DEBUG:boto:auth_path=/pklxkd-59/?prefix=obj87 2023-05-31T20:55:01.073 DEBUG:boto:Method: GET 2023-05-31T20:55:01.073 DEBUG:boto:Path: /pklxkd-59/?prefix=obj87 2023-05-31T20:55:01.074 DEBUG:boto:Data: 2023-05-31T20:55:01.074 DEBUG:boto:Headers: {} 2023-05-31T20:55:01.074 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:55:01.075 DEBUG:boto:Port: 8000 2023-05-31T20:55:01.075 DEBUG:boto:Params: {} 2023-05-31T20:55:01.076 DEBUG:boto:Token: None 2023-05-31T20:55:01.076 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:55:01 GMT /pklxkd-59/ 2023-05-31T20:55:01.076 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:d7ZqE7fjjpja5oroUaADG51170Q= 2023-05-31T20:55:01.077 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:55:01 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:d7ZqE7fjjpja5oroUaADG51170Q=', 'Content-Length': '0'} 2023-05-31T20:55:01.081 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000004986bfe73527ffc9-006477b425-1630-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:55:01 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:55:01.081 DEBUG:boto:b'pklxkd-59obj871000falseobj872023-05-31T20:48:13.031Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:55:01.082 DEBUG:boto:path=/pklxkd-59/obj87 2023-05-31T20:55:01.082 DEBUG:boto:auth_path=/pklxkd-59/obj87 2023-05-31T20:55:01.083 DEBUG:boto:Method: HEAD 2023-05-31T20:55:01.083 DEBUG:boto:Path: /pklxkd-59/obj87 2023-05-31T20:55:01.083 DEBUG:boto:Data: 2023-05-31T20:55:01.084 DEBUG:boto:Headers: {} 2023-05-31T20:55:01.084 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:55:01.084 DEBUG:boto:Port: 8001 2023-05-31T20:55:01.085 DEBUG:boto:Params: {} 2023-05-31T20:55:01.085 DEBUG:boto:Token: None 2023-05-31T20:55:01.085 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:55:01 GMT /pklxkd-59/obj87 2023-05-31T20:55:01.086 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:lrmbxw7+cm/1tyk8bxCf2fzXrM0= 2023-05-31T20:55:01.086 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:55:01 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:lrmbxw7+cm/1tyk8bxCf2fzXrM0=', 'Content-Length': '0'} 2023-05-31T20:55:01.088 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:48:13 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002fe351a9aa800be9-006477b425-1e2e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:55:01 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:55:01.089 DEBUG:boto:path=/pklxkd-59/obj87 2023-05-31T20:55:01.089 DEBUG:boto:auth_path=/pklxkd-59/obj87 2023-05-31T20:55:01.089 DEBUG:boto:Method: HEAD 2023-05-31T20:55:01.090 DEBUG:boto:Path: /pklxkd-59/obj87 2023-05-31T20:55:01.090 DEBUG:boto:Data: 2023-05-31T20:55:01.090 DEBUG:boto:Headers: {} 2023-05-31T20:55:01.091 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:55:01.091 DEBUG:boto:Port: 8000 2023-05-31T20:55:01.091 DEBUG:boto:Params: {} 2023-05-31T20:55:01.092 DEBUG:boto:Token: None 2023-05-31T20:55:01.092 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:55:01 GMT /pklxkd-59/obj87 2023-05-31T20:55:01.092 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:lrmbxw7+cm/1tyk8bxCf2fzXrM0= 2023-05-31T20:55:01.093 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:55:01 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:lrmbxw7+cm/1tyk8bxCf2fzXrM0=', 'Content-Length': '0'} 2023-05-31T20:55:01.095 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:48:13 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f1f883229cf459ce-006477b425-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:55:01 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:55:01.095 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj87 2023-05-31T20:55:01.095 DEBUG:boto:path=/pklxkd-59/obj87 2023-05-31T20:55:01.096 DEBUG:boto:auth_path=/pklxkd-59/obj87 2023-05-31T20:55:01.096 DEBUG:boto:Method: GET 2023-05-31T20:55:01.097 DEBUG:boto:Path: /pklxkd-59/obj87 2023-05-31T20:55:01.097 DEBUG:boto:Data: 2023-05-31T20:55:01.097 DEBUG:boto:Headers: {} 2023-05-31T20:55:01.098 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:55:01.098 DEBUG:boto:Port: 8001 2023-05-31T20:55:01.098 DEBUG:boto:Params: {} 2023-05-31T20:55:01.099 DEBUG:boto:Token: None 2023-05-31T20:55:01.099 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:55:01 GMT /pklxkd-59/obj87 2023-05-31T20:55:01.100 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:CPPDFpZQBOuttYX/zjILG+bs1Kk= 2023-05-31T20:55:01.100 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:55:01 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:CPPDFpZQBOuttYX/zjILG+bs1Kk=', 'Content-Length': '0'} 2023-05-31T20:55:01.102 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:48:13 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e734215b8227363d-006477b425-1e2e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:55:01 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:55:01.103 DEBUG:boto:path=/pklxkd-59/obj87 2023-05-31T20:55:01.103 DEBUG:boto:auth_path=/pklxkd-59/obj87 2023-05-31T20:55:01.104 DEBUG:boto:Method: GET 2023-05-31T20:55:01.104 DEBUG:boto:Path: /pklxkd-59/obj87 2023-05-31T20:55:01.104 DEBUG:boto:Data: 2023-05-31T20:55:01.105 DEBUG:boto:Headers: {} 2023-05-31T20:55:01.105 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:55:01.106 DEBUG:boto:Port: 8000 2023-05-31T20:55:01.106 DEBUG:boto:Params: {} 2023-05-31T20:55:01.106 DEBUG:boto:Token: None 2023-05-31T20:55:01.107 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:55:01 GMT /pklxkd-59/obj87 2023-05-31T20:55:01.107 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:CPPDFpZQBOuttYX/zjILG+bs1Kk= 2023-05-31T20:55:01.108 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:55:01 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:CPPDFpZQBOuttYX/zjILG+bs1Kk=', 'Content-Length': '0'} 2023-05-31T20:55:01.109 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:48:13 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000057675906a66fb146-006477b425-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:55:01 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:55:01.110 DEBUG:boto:path=/pklxkd-59/ 2023-05-31T20:55:01.110 DEBUG:boto:auth_path=/pklxkd-59/ 2023-05-31T20:55:01.111 DEBUG:boto:path=/pklxkd-59/?prefix=obj87 2023-05-31T20:55:01.111 DEBUG:boto:auth_path=/pklxkd-59/?prefix=obj87 2023-05-31T20:55:01.111 DEBUG:boto:Method: GET 2023-05-31T20:55:01.112 DEBUG:boto:Path: /pklxkd-59/?prefix=obj87 2023-05-31T20:55:01.112 DEBUG:boto:Data: 2023-05-31T20:55:01.112 DEBUG:boto:Headers: {} 2023-05-31T20:55:01.113 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:55:01.113 DEBUG:boto:Port: 8001 2023-05-31T20:55:01.114 DEBUG:boto:Params: {} 2023-05-31T20:55:01.114 DEBUG:boto:Token: None 2023-05-31T20:55:01.114 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:55:01 GMT /pklxkd-59/ 2023-05-31T20:55:01.115 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:d7ZqE7fjjpja5oroUaADG51170Q= 2023-05-31T20:55:01.115 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:55:01 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:d7ZqE7fjjpja5oroUaADG51170Q=', 'Content-Length': '0'} 2023-05-31T20:55:01.118 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000087941ab67a7d72dd-006477b425-1e2e-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:55:01 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:55:01.118 DEBUG:boto:b'pklxkd-59obj871000falseobj872023-05-31T20:48:13.031Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:55:01.119 DEBUG:boto:path=/pklxkd-59/ 2023-05-31T20:55:01.119 DEBUG:boto:auth_path=/pklxkd-59/ 2023-05-31T20:55:01.119 DEBUG:boto:path=/pklxkd-59/?prefix=obj87 2023-05-31T20:55:01.120 DEBUG:boto:auth_path=/pklxkd-59/?prefix=obj87 2023-05-31T20:55:01.120 DEBUG:boto:Method: GET 2023-05-31T20:55:01.121 DEBUG:boto:Path: /pklxkd-59/?prefix=obj87 2023-05-31T20:55:01.121 DEBUG:boto:Data: 2023-05-31T20:55:01.121 DEBUG:boto:Headers: {} 2023-05-31T20:55:01.122 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:55:01.122 DEBUG:boto:Port: 8000 2023-05-31T20:55:01.122 DEBUG:boto:Params: {} 2023-05-31T20:55:01.123 DEBUG:boto:Token: None 2023-05-31T20:55:01.123 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:55:01 GMT /pklxkd-59/ 2023-05-31T20:55:01.124 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:d7ZqE7fjjpja5oroUaADG51170Q= 2023-05-31T20:55:01.124 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:55:01 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:d7ZqE7fjjpja5oroUaADG51170Q=', 'Content-Length': '0'} 2023-05-31T20:55:01.128 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b3cc682212624571-006477b425-1630-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:55:01 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:55:01.129 DEBUG:boto:b'pklxkd-59obj871000falseobj872023-05-31T20:48:13.031Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:55:01.129 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj92 2023-05-31T20:55:01.130 DEBUG:boto:path=/pklxkd-59/obj92 2023-05-31T20:55:01.130 DEBUG:boto:auth_path=/pklxkd-59/obj92 2023-05-31T20:55:01.130 DEBUG:boto:path=/pklxkd-59/obj92?versionId=null 2023-05-31T20:55:01.131 DEBUG:boto:auth_path=/pklxkd-59/obj92?versionId=null 2023-05-31T20:55:01.131 DEBUG:boto:Method: GET 2023-05-31T20:55:01.131 DEBUG:boto:Path: /pklxkd-59/obj92?versionId=null 2023-05-31T20:55:01.132 DEBUG:boto:Data: 2023-05-31T20:55:01.132 DEBUG:boto:Headers: {} 2023-05-31T20:55:01.133 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:55:01.133 DEBUG:boto:Port: 8001 2023-05-31T20:55:01.133 DEBUG:boto:Params: {} 2023-05-31T20:55:01.134 DEBUG:boto:Token: None 2023-05-31T20:55:01.134 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:55:01 GMT /pklxkd-59/obj92?versionId=null 2023-05-31T20:55:01.134 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:MONAYlyGIX779+BvVSUIbhMbr44= 2023-05-31T20:55:01.135 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:55:01 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:MONAYlyGIX779+BvVSUIbhMbr44=', 'Content-Length': '0'} 2023-05-31T20:55:01.137 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:46:33 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000334d9db18c2221ba-006477b425-1e2e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:55:01 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:55:01.180 DEBUG:boto:path=/pklxkd-59/obj92 2023-05-31T20:55:01.181 DEBUG:boto:auth_path=/pklxkd-59/obj92 2023-05-31T20:55:01.181 DEBUG:boto:path=/pklxkd-59/obj92?versionId=null 2023-05-31T20:55:01.181 DEBUG:boto:auth_path=/pklxkd-59/obj92?versionId=null 2023-05-31T20:55:01.182 DEBUG:boto:Method: GET 2023-05-31T20:55:01.182 DEBUG:boto:Path: /pklxkd-59/obj92?versionId=null 2023-05-31T20:55:01.182 DEBUG:boto:Data: 2023-05-31T20:55:01.183 DEBUG:boto:Headers: {} 2023-05-31T20:55:01.183 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:55:01.184 DEBUG:boto:Port: 8000 2023-05-31T20:55:01.184 DEBUG:boto:Params: {} 2023-05-31T20:55:01.184 DEBUG:boto:Token: None 2023-05-31T20:55:01.185 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:55:01 GMT /pklxkd-59/obj92?versionId=null 2023-05-31T20:55:01.185 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:MONAYlyGIX779+BvVSUIbhMbr44= 2023-05-31T20:55:01.186 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:55:01 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:MONAYlyGIX779+BvVSUIbhMbr44=', 'Content-Length': '0'} 2023-05-31T20:55:01.187 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:46:33 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000059d5acc64f2ee79f-006477b425-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:55:01 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:55:01.188 DEBUG:boto:path=/pklxkd-59/obj92 2023-05-31T20:55:01.188 DEBUG:boto:auth_path=/pklxkd-59/obj92 2023-05-31T20:55:01.189 DEBUG:boto:path=/pklxkd-59/obj92?versionId=null 2023-05-31T20:55:01.189 DEBUG:boto:auth_path=/pklxkd-59/obj92?versionId=null 2023-05-31T20:55:01.189 DEBUG:boto:Method: HEAD 2023-05-31T20:55:01.190 DEBUG:boto:Path: /pklxkd-59/obj92?versionId=null 2023-05-31T20:55:01.190 DEBUG:boto:Data: 2023-05-31T20:55:01.190 DEBUG:boto:Headers: {} 2023-05-31T20:55:01.191 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:55:01.191 DEBUG:boto:Port: 8001 2023-05-31T20:55:01.191 DEBUG:boto:Params: {} 2023-05-31T20:55:01.192 DEBUG:boto:Token: None 2023-05-31T20:55:01.192 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:55:01 GMT /pklxkd-59/obj92?versionId=null 2023-05-31T20:55:01.193 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:XqLYd05y9TJ2Fsvl9ld+0oORccQ= 2023-05-31T20:55:01.193 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:55:01 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:XqLYd05y9TJ2Fsvl9ld+0oORccQ=', 'Content-Length': '0'} 2023-05-31T20:55:01.195 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:46:33 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009d13d7120ecd0f83-006477b425-1e2e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:55:01 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:55:01.195 DEBUG:boto:path=/pklxkd-59/obj92 2023-05-31T20:55:01.196 DEBUG:boto:auth_path=/pklxkd-59/obj92 2023-05-31T20:55:01.196 DEBUG:boto:path=/pklxkd-59/obj92?versionId=null 2023-05-31T20:55:01.196 DEBUG:boto:auth_path=/pklxkd-59/obj92?versionId=null 2023-05-31T20:55:01.197 DEBUG:boto:Method: HEAD 2023-05-31T20:55:01.197 DEBUG:boto:Path: /pklxkd-59/obj92?versionId=null 2023-05-31T20:55:01.197 DEBUG:boto:Data: 2023-05-31T20:55:01.198 DEBUG:boto:Headers: {} 2023-05-31T20:55:01.198 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:55:01.198 DEBUG:boto:Port: 8000 2023-05-31T20:55:01.199 DEBUG:boto:Params: {} 2023-05-31T20:55:01.199 DEBUG:boto:Token: None 2023-05-31T20:55:01.200 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:55:01 GMT /pklxkd-59/obj92?versionId=null 2023-05-31T20:55:01.200 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:XqLYd05y9TJ2Fsvl9ld+0oORccQ= 2023-05-31T20:55:01.200 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:55:01 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:XqLYd05y9TJ2Fsvl9ld+0oORccQ=', 'Content-Length': '0'} 2023-05-31T20:55:01.202 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:46:33 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000cb395cd8191abf61-006477b425-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:55:01 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:55:01.203 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj92 2023-05-31T20:55:01.203 DEBUG:boto:path=/pklxkd-59/obj92 2023-05-31T20:55:01.203 DEBUG:boto:auth_path=/pklxkd-59/obj92 2023-05-31T20:55:01.204 DEBUG:boto:path=/pklxkd-59/obj92?versionId=null 2023-05-31T20:55:01.204 DEBUG:boto:auth_path=/pklxkd-59/obj92?versionId=null 2023-05-31T20:55:01.204 DEBUG:boto:Method: GET 2023-05-31T20:55:01.205 DEBUG:boto:Path: /pklxkd-59/obj92?versionId=null 2023-05-31T20:55:01.205 DEBUG:boto:Data: 2023-05-31T20:55:01.205 DEBUG:boto:Headers: {} 2023-05-31T20:55:01.206 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:55:01.206 DEBUG:boto:Port: 8001 2023-05-31T20:55:01.207 DEBUG:boto:Params: {} 2023-05-31T20:55:01.207 DEBUG:boto:Token: None 2023-05-31T20:55:01.207 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:55:01 GMT /pklxkd-59/obj92?versionId=null 2023-05-31T20:55:01.208 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:MONAYlyGIX779+BvVSUIbhMbr44= 2023-05-31T20:55:01.208 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:55:01 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:MONAYlyGIX779+BvVSUIbhMbr44=', 'Content-Length': '0'} 2023-05-31T20:55:01.210 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:46:33 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c5c34bd99a4794b6-006477b425-1e2e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:55:01 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:55:01.211 DEBUG:boto:path=/pklxkd-59/obj92 2023-05-31T20:55:01.211 DEBUG:boto:auth_path=/pklxkd-59/obj92 2023-05-31T20:55:01.212 DEBUG:boto:path=/pklxkd-59/obj92?versionId=null 2023-05-31T20:55:01.212 DEBUG:boto:auth_path=/pklxkd-59/obj92?versionId=null 2023-05-31T20:55:01.213 DEBUG:boto:Method: GET 2023-05-31T20:55:01.213 DEBUG:boto:Path: /pklxkd-59/obj92?versionId=null 2023-05-31T20:55:01.213 DEBUG:boto:Data: 2023-05-31T20:55:01.214 DEBUG:boto:Headers: {} 2023-05-31T20:55:01.214 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:55:01.214 DEBUG:boto:Port: 8000 2023-05-31T20:55:01.215 DEBUG:boto:Params: {} 2023-05-31T20:55:01.215 DEBUG:boto:Token: None 2023-05-31T20:55:01.216 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:55:01 GMT /pklxkd-59/obj92?versionId=null 2023-05-31T20:55:01.216 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:MONAYlyGIX779+BvVSUIbhMbr44= 2023-05-31T20:55:01.217 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:55:01 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:MONAYlyGIX779+BvVSUIbhMbr44=', 'Content-Length': '0'} 2023-05-31T20:55:01.219 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:46:33 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002786970a4c3272ba-006477b425-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:55:01 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:55:01.219 DEBUG:boto:path=/pklxkd-59/ 2023-05-31T20:55:01.220 DEBUG:boto:auth_path=/pklxkd-59/ 2023-05-31T20:55:01.220 DEBUG:boto:path=/pklxkd-59/?prefix=obj92 2023-05-31T20:55:01.220 DEBUG:boto:auth_path=/pklxkd-59/?prefix=obj92 2023-05-31T20:55:01.221 DEBUG:boto:Method: GET 2023-05-31T20:55:01.221 DEBUG:boto:Path: /pklxkd-59/?prefix=obj92 2023-05-31T20:55:01.221 DEBUG:boto:Data: 2023-05-31T20:55:01.222 DEBUG:boto:Headers: {} 2023-05-31T20:55:01.222 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:55:01.222 DEBUG:boto:Port: 8001 2023-05-31T20:55:01.223 DEBUG:boto:Params: {} 2023-05-31T20:55:01.223 DEBUG:boto:Token: None 2023-05-31T20:55:01.224 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:55:01 GMT /pklxkd-59/ 2023-05-31T20:55:01.224 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:d7ZqE7fjjpja5oroUaADG51170Q= 2023-05-31T20:55:01.225 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:55:01 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:d7ZqE7fjjpja5oroUaADG51170Q=', 'Content-Length': '0'} 2023-05-31T20:55:01.227 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000003d51aaa983f0d3b0-006477b425-1e2e-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:55:01 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:55:01.228 DEBUG:boto:b'pklxkd-59obj921000falseobj922023-05-31T20:46:33.904Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:55:01.228 DEBUG:boto:path=/pklxkd-59/ 2023-05-31T20:55:01.229 DEBUG:boto:auth_path=/pklxkd-59/ 2023-05-31T20:55:01.229 DEBUG:boto:path=/pklxkd-59/?prefix=obj92 2023-05-31T20:55:01.229 DEBUG:boto:auth_path=/pklxkd-59/?prefix=obj92 2023-05-31T20:55:01.230 DEBUG:boto:Method: GET 2023-05-31T20:55:01.230 DEBUG:boto:Path: /pklxkd-59/?prefix=obj92 2023-05-31T20:55:01.231 DEBUG:boto:Data: 2023-05-31T20:55:01.231 DEBUG:boto:Headers: {} 2023-05-31T20:55:01.231 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:55:01.232 DEBUG:boto:Port: 8000 2023-05-31T20:55:01.232 DEBUG:boto:Params: {} 2023-05-31T20:55:01.232 DEBUG:boto:Token: None 2023-05-31T20:55:01.233 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:55:01 GMT /pklxkd-59/ 2023-05-31T20:55:01.233 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:d7ZqE7fjjpja5oroUaADG51170Q= 2023-05-31T20:55:01.234 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:55:01 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:d7ZqE7fjjpja5oroUaADG51170Q=', 'Content-Length': '0'} 2023-05-31T20:55:01.238 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000fad7cece713946a0-006477b425-1630-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:55:01 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:55:01.238 DEBUG:boto:b'pklxkd-59obj921000falseobj922023-05-31T20:46:33.904Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:55:01.239 DEBUG:boto:path=/pklxkd-59/obj92 2023-05-31T20:55:01.239 DEBUG:boto:auth_path=/pklxkd-59/obj92 2023-05-31T20:55:01.240 DEBUG:boto:Method: HEAD 2023-05-31T20:55:01.240 DEBUG:boto:Path: /pklxkd-59/obj92 2023-05-31T20:55:01.240 DEBUG:boto:Data: 2023-05-31T20:55:01.241 DEBUG:boto:Headers: {} 2023-05-31T20:55:01.241 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:55:01.241 DEBUG:boto:Port: 8001 2023-05-31T20:55:01.242 DEBUG:boto:Params: {} 2023-05-31T20:55:01.242 DEBUG:boto:Token: None 2023-05-31T20:55:01.242 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:55:01 GMT /pklxkd-59/obj92 2023-05-31T20:55:01.243 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:mWLC4YzIgDARoa0HIJSffqdtZWs= 2023-05-31T20:55:01.243 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:55:01 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:mWLC4YzIgDARoa0HIJSffqdtZWs=', 'Content-Length': '0'} 2023-05-31T20:55:01.245 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:46:33 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005a94e2585ede1a97-006477b425-1e2e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:55:01 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:55:01.246 DEBUG:boto:path=/pklxkd-59/obj92 2023-05-31T20:55:01.246 DEBUG:boto:auth_path=/pklxkd-59/obj92 2023-05-31T20:55:01.246 DEBUG:boto:Method: HEAD 2023-05-31T20:55:01.247 DEBUG:boto:Path: /pklxkd-59/obj92 2023-05-31T20:55:01.247 DEBUG:boto:Data: 2023-05-31T20:55:01.247 DEBUG:boto:Headers: {} 2023-05-31T20:55:01.248 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:55:01.248 DEBUG:boto:Port: 8000 2023-05-31T20:55:01.248 DEBUG:boto:Params: {} 2023-05-31T20:55:01.249 DEBUG:boto:Token: None 2023-05-31T20:55:01.249 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:55:01 GMT /pklxkd-59/obj92 2023-05-31T20:55:01.250 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:mWLC4YzIgDARoa0HIJSffqdtZWs= 2023-05-31T20:55:01.250 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:55:01 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:mWLC4YzIgDARoa0HIJSffqdtZWs=', 'Content-Length': '0'} 2023-05-31T20:55:01.252 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:46:33 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000080867d0778dd0865-006477b425-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:55:01 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:55:01.252 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj92 2023-05-31T20:55:01.253 DEBUG:boto:path=/pklxkd-59/obj92 2023-05-31T20:55:01.253 DEBUG:boto:auth_path=/pklxkd-59/obj92 2023-05-31T20:55:01.253 DEBUG:boto:Method: GET 2023-05-31T20:55:01.254 DEBUG:boto:Path: /pklxkd-59/obj92 2023-05-31T20:55:01.254 DEBUG:boto:Data: 2023-05-31T20:55:01.255 DEBUG:boto:Headers: {} 2023-05-31T20:55:01.255 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:55:01.255 DEBUG:boto:Port: 8001 2023-05-31T20:55:01.256 DEBUG:boto:Params: {} 2023-05-31T20:55:01.256 DEBUG:boto:Token: None 2023-05-31T20:55:01.256 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:55:01 GMT /pklxkd-59/obj92 2023-05-31T20:55:01.257 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:iHzulvLCHFJ2h9wD1Zej0izb3y4= 2023-05-31T20:55:01.257 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:55:01 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:iHzulvLCHFJ2h9wD1Zej0izb3y4=', 'Content-Length': '0'} 2023-05-31T20:55:01.259 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:46:33 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f012c1b87e913138-006477b425-1e2e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:55:01 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:55:01.260 DEBUG:boto:path=/pklxkd-59/obj92 2023-05-31T20:55:01.260 DEBUG:boto:auth_path=/pklxkd-59/obj92 2023-05-31T20:55:01.260 DEBUG:boto:Method: GET 2023-05-31T20:55:01.261 DEBUG:boto:Path: /pklxkd-59/obj92 2023-05-31T20:55:01.261 DEBUG:boto:Data: 2023-05-31T20:55:01.262 DEBUG:boto:Headers: {} 2023-05-31T20:55:01.262 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:55:01.262 DEBUG:boto:Port: 8000 2023-05-31T20:55:01.263 DEBUG:boto:Params: {} 2023-05-31T20:55:01.263 DEBUG:boto:Token: None 2023-05-31T20:55:01.263 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:55:01 GMT /pklxkd-59/obj92 2023-05-31T20:55:01.264 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:iHzulvLCHFJ2h9wD1Zej0izb3y4= 2023-05-31T20:55:01.264 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:55:01 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:iHzulvLCHFJ2h9wD1Zej0izb3y4=', 'Content-Length': '0'} 2023-05-31T20:55:01.266 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:46:33 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000034b090bd151b46f9-006477b425-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:55:01 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:55:01.308 DEBUG:boto:path=/pklxkd-59/ 2023-05-31T20:55:01.309 DEBUG:boto:auth_path=/pklxkd-59/ 2023-05-31T20:55:01.309 DEBUG:boto:path=/pklxkd-59/?prefix=obj92 2023-05-31T20:55:01.309 DEBUG:boto:auth_path=/pklxkd-59/?prefix=obj92 2023-05-31T20:55:01.310 DEBUG:boto:Method: GET 2023-05-31T20:55:01.310 DEBUG:boto:Path: /pklxkd-59/?prefix=obj92 2023-05-31T20:55:01.310 DEBUG:boto:Data: 2023-05-31T20:55:01.311 DEBUG:boto:Headers: {} 2023-05-31T20:55:01.311 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:55:01.311 DEBUG:boto:Port: 8001 2023-05-31T20:55:01.312 DEBUG:boto:Params: {} 2023-05-31T20:55:01.312 DEBUG:boto:Token: None 2023-05-31T20:55:01.313 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:55:01 GMT /pklxkd-59/ 2023-05-31T20:55:01.313 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:d7ZqE7fjjpja5oroUaADG51170Q= 2023-05-31T20:55:01.313 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:55:01 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:d7ZqE7fjjpja5oroUaADG51170Q=', 'Content-Length': '0'} 2023-05-31T20:55:01.316 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000003c91c6ef9af112cd-006477b425-1e2e-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:55:01 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:55:01.316 DEBUG:boto:b'pklxkd-59obj921000falseobj922023-05-31T20:46:33.904Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:55:01.317 DEBUG:boto:path=/pklxkd-59/ 2023-05-31T20:55:01.317 DEBUG:boto:auth_path=/pklxkd-59/ 2023-05-31T20:55:01.318 DEBUG:boto:path=/pklxkd-59/?prefix=obj92 2023-05-31T20:55:01.318 DEBUG:boto:auth_path=/pklxkd-59/?prefix=obj92 2023-05-31T20:55:01.318 DEBUG:boto:Method: GET 2023-05-31T20:55:01.319 DEBUG:boto:Path: /pklxkd-59/?prefix=obj92 2023-05-31T20:55:01.319 DEBUG:boto:Data: 2023-05-31T20:55:01.320 DEBUG:boto:Headers: {} 2023-05-31T20:55:01.320 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:55:01.320 DEBUG:boto:Port: 8000 2023-05-31T20:55:01.321 DEBUG:boto:Params: {} 2023-05-31T20:55:01.321 DEBUG:boto:Token: None 2023-05-31T20:55:01.321 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:55:01 GMT /pklxkd-59/ 2023-05-31T20:55:01.322 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:d7ZqE7fjjpja5oroUaADG51170Q= 2023-05-31T20:55:01.322 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:55:01 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:d7ZqE7fjjpja5oroUaADG51170Q=', 'Content-Length': '0'} 2023-05-31T20:55:01.326 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000384570c24cc6dcbd-006477b425-1630-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:55:01 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:55:01.327 DEBUG:boto:b'pklxkd-59obj921000falseobj922023-05-31T20:46:33.904Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:55:01.327 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj93 2023-05-31T20:55:01.328 DEBUG:boto:path=/pklxkd-59/obj93 2023-05-31T20:55:01.328 DEBUG:boto:auth_path=/pklxkd-59/obj93 2023-05-31T20:55:01.329 DEBUG:boto:path=/pklxkd-59/obj93?versionId=null 2023-05-31T20:55:01.329 DEBUG:boto:auth_path=/pklxkd-59/obj93?versionId=null 2023-05-31T20:55:01.329 DEBUG:boto:Method: GET 2023-05-31T20:55:01.330 DEBUG:boto:Path: /pklxkd-59/obj93?versionId=null 2023-05-31T20:55:01.330 DEBUG:boto:Data: 2023-05-31T20:55:01.330 DEBUG:boto:Headers: {} 2023-05-31T20:55:01.331 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:55:01.331 DEBUG:boto:Port: 8001 2023-05-31T20:55:01.331 DEBUG:boto:Params: {} 2023-05-31T20:55:01.332 DEBUG:boto:Token: None 2023-05-31T20:55:01.332 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:55:01 GMT /pklxkd-59/obj93?versionId=null 2023-05-31T20:55:01.333 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:2zhnEHPYzgmgX3wQt3YpiL/jnHk= 2023-05-31T20:55:01.333 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:55:01 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:2zhnEHPYzgmgX3wQt3YpiL/jnHk=', 'Content-Length': '0'} 2023-05-31T20:55:01.335 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:50:33 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c31b19cb440bae4b-006477b425-1e2e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:55:01 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:55:01.336 DEBUG:boto:path=/pklxkd-59/obj93 2023-05-31T20:55:01.336 DEBUG:boto:auth_path=/pklxkd-59/obj93 2023-05-31T20:55:01.337 DEBUG:boto:path=/pklxkd-59/obj93?versionId=null 2023-05-31T20:55:01.337 DEBUG:boto:auth_path=/pklxkd-59/obj93?versionId=null 2023-05-31T20:55:01.337 DEBUG:boto:Method: GET 2023-05-31T20:55:01.338 DEBUG:boto:Path: /pklxkd-59/obj93?versionId=null 2023-05-31T20:55:01.338 DEBUG:boto:Data: 2023-05-31T20:55:01.338 DEBUG:boto:Headers: {} 2023-05-31T20:55:01.339 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:55:01.339 DEBUG:boto:Port: 8000 2023-05-31T20:55:01.340 DEBUG:boto:Params: {} 2023-05-31T20:55:01.340 DEBUG:boto:Token: None 2023-05-31T20:55:01.340 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:55:01 GMT /pklxkd-59/obj93?versionId=null 2023-05-31T20:55:01.341 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:2zhnEHPYzgmgX3wQt3YpiL/jnHk= 2023-05-31T20:55:01.341 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:55:01 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:2zhnEHPYzgmgX3wQt3YpiL/jnHk=', 'Content-Length': '0'} 2023-05-31T20:55:01.343 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:50:33 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000922dde3cb9fdd61e-006477b425-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:55:01 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:55:01.344 DEBUG:boto:path=/pklxkd-59/obj93 2023-05-31T20:55:01.344 DEBUG:boto:auth_path=/pklxkd-59/obj93 2023-05-31T20:55:01.344 DEBUG:boto:path=/pklxkd-59/obj93?versionId=null 2023-05-31T20:55:01.345 DEBUG:boto:auth_path=/pklxkd-59/obj93?versionId=null 2023-05-31T20:55:01.345 DEBUG:boto:Method: HEAD 2023-05-31T20:55:01.345 DEBUG:boto:Path: /pklxkd-59/obj93?versionId=null 2023-05-31T20:55:01.346 DEBUG:boto:Data: 2023-05-31T20:55:01.346 DEBUG:boto:Headers: {} 2023-05-31T20:55:01.346 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:55:01.347 DEBUG:boto:Port: 8001 2023-05-31T20:55:01.347 DEBUG:boto:Params: {} 2023-05-31T20:55:01.347 DEBUG:boto:Token: None 2023-05-31T20:55:01.348 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:55:01 GMT /pklxkd-59/obj93?versionId=null 2023-05-31T20:55:01.348 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:Jqf56o9MEMNsGdhyLz8qV1nfKrI= 2023-05-31T20:55:01.349 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:55:01 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:Jqf56o9MEMNsGdhyLz8qV1nfKrI=', 'Content-Length': '0'} 2023-05-31T20:55:01.350 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:50:33 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ac5ab4683c8c1c0a-006477b425-1e2e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:55:01 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:55:01.351 DEBUG:boto:path=/pklxkd-59/obj93 2023-05-31T20:55:01.351 DEBUG:boto:auth_path=/pklxkd-59/obj93 2023-05-31T20:55:01.352 DEBUG:boto:path=/pklxkd-59/obj93?versionId=null 2023-05-31T20:55:01.352 DEBUG:boto:auth_path=/pklxkd-59/obj93?versionId=null 2023-05-31T20:55:01.352 DEBUG:boto:Method: HEAD 2023-05-31T20:55:01.353 DEBUG:boto:Path: /pklxkd-59/obj93?versionId=null 2023-05-31T20:55:01.353 DEBUG:boto:Data: 2023-05-31T20:55:01.353 DEBUG:boto:Headers: {} 2023-05-31T20:55:01.354 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:55:01.354 DEBUG:boto:Port: 8000 2023-05-31T20:55:01.354 DEBUG:boto:Params: {} 2023-05-31T20:55:01.355 DEBUG:boto:Token: None 2023-05-31T20:55:01.355 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:55:01 GMT /pklxkd-59/obj93?versionId=null 2023-05-31T20:55:01.355 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:Jqf56o9MEMNsGdhyLz8qV1nfKrI= 2023-05-31T20:55:01.356 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:55:01 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:Jqf56o9MEMNsGdhyLz8qV1nfKrI=', 'Content-Length': '0'} 2023-05-31T20:55:01.357 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:50:33 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e45f101428b89e2a-006477b425-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:55:01 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:55:01.358 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj93 2023-05-31T20:55:01.358 DEBUG:boto:path=/pklxkd-59/obj93 2023-05-31T20:55:01.359 DEBUG:boto:auth_path=/pklxkd-59/obj93 2023-05-31T20:55:01.359 DEBUG:boto:path=/pklxkd-59/obj93?versionId=null 2023-05-31T20:55:01.359 DEBUG:boto:auth_path=/pklxkd-59/obj93?versionId=null 2023-05-31T20:55:01.360 DEBUG:boto:Method: GET 2023-05-31T20:55:01.360 DEBUG:boto:Path: /pklxkd-59/obj93?versionId=null 2023-05-31T20:55:01.360 DEBUG:boto:Data: 2023-05-31T20:55:01.361 DEBUG:boto:Headers: {} 2023-05-31T20:55:01.361 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:55:01.362 DEBUG:boto:Port: 8001 2023-05-31T20:55:01.362 DEBUG:boto:Params: {} 2023-05-31T20:55:01.362 DEBUG:boto:Token: None 2023-05-31T20:55:01.363 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:55:01 GMT /pklxkd-59/obj93?versionId=null 2023-05-31T20:55:01.363 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:2zhnEHPYzgmgX3wQt3YpiL/jnHk= 2023-05-31T20:55:01.364 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:55:01 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:2zhnEHPYzgmgX3wQt3YpiL/jnHk=', 'Content-Length': '0'} 2023-05-31T20:55:01.365 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:50:33 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007b9a9254d3919299-006477b425-1e2e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:55:01 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:55:01.366 DEBUG:boto:path=/pklxkd-59/obj93 2023-05-31T20:55:01.366 DEBUG:boto:auth_path=/pklxkd-59/obj93 2023-05-31T20:55:01.367 DEBUG:boto:path=/pklxkd-59/obj93?versionId=null 2023-05-31T20:55:01.367 DEBUG:boto:auth_path=/pklxkd-59/obj93?versionId=null 2023-05-31T20:55:01.367 DEBUG:boto:Method: GET 2023-05-31T20:55:01.368 DEBUG:boto:Path: /pklxkd-59/obj93?versionId=null 2023-05-31T20:55:01.368 DEBUG:boto:Data: 2023-05-31T20:55:01.369 DEBUG:boto:Headers: {} 2023-05-31T20:55:01.369 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:55:01.369 DEBUG:boto:Port: 8000 2023-05-31T20:55:01.370 DEBUG:boto:Params: {} 2023-05-31T20:55:01.370 DEBUG:boto:Token: None 2023-05-31T20:55:01.370 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:55:01 GMT /pklxkd-59/obj93?versionId=null 2023-05-31T20:55:01.371 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:2zhnEHPYzgmgX3wQt3YpiL/jnHk= 2023-05-31T20:55:01.371 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:55:01 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:2zhnEHPYzgmgX3wQt3YpiL/jnHk=', 'Content-Length': '0'} 2023-05-31T20:55:01.373 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:50:33 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003a998e070411626d-006477b425-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:55:01 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:55:01.374 DEBUG:boto:path=/pklxkd-59/ 2023-05-31T20:55:01.374 DEBUG:boto:auth_path=/pklxkd-59/ 2023-05-31T20:55:01.375 DEBUG:boto:path=/pklxkd-59/?prefix=obj93 2023-05-31T20:55:01.375 DEBUG:boto:auth_path=/pklxkd-59/?prefix=obj93 2023-05-31T20:55:01.375 DEBUG:boto:Method: GET 2023-05-31T20:55:01.376 DEBUG:boto:Path: /pklxkd-59/?prefix=obj93 2023-05-31T20:55:01.376 DEBUG:boto:Data: 2023-05-31T20:55:01.376 DEBUG:boto:Headers: {} 2023-05-31T20:55:01.377 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:55:01.377 DEBUG:boto:Port: 8001 2023-05-31T20:55:01.377 DEBUG:boto:Params: {} 2023-05-31T20:55:01.378 DEBUG:boto:Token: None 2023-05-31T20:55:01.378 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:55:01 GMT /pklxkd-59/ 2023-05-31T20:55:01.379 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:d7ZqE7fjjpja5oroUaADG51170Q= 2023-05-31T20:55:01.379 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:55:01 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:d7ZqE7fjjpja5oroUaADG51170Q=', 'Content-Length': '0'} 2023-05-31T20:55:01.381 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000004d3117a5ef4c1dc2-006477b425-1e2e-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:55:01 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:55:01.382 DEBUG:boto:b'pklxkd-59obj931000falseobj932023-05-31T20:50:33.533Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:55:01.383 DEBUG:boto:path=/pklxkd-59/ 2023-05-31T20:55:01.383 DEBUG:boto:auth_path=/pklxkd-59/ 2023-05-31T20:55:01.383 DEBUG:boto:path=/pklxkd-59/?prefix=obj93 2023-05-31T20:55:01.384 DEBUG:boto:auth_path=/pklxkd-59/?prefix=obj93 2023-05-31T20:55:01.384 DEBUG:boto:Method: GET 2023-05-31T20:55:01.385 DEBUG:boto:Path: /pklxkd-59/?prefix=obj93 2023-05-31T20:55:01.385 DEBUG:boto:Data: 2023-05-31T20:55:01.385 DEBUG:boto:Headers: {} 2023-05-31T20:55:01.386 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:55:01.386 DEBUG:boto:Port: 8000 2023-05-31T20:55:01.386 DEBUG:boto:Params: {} 2023-05-31T20:55:01.387 DEBUG:boto:Token: None 2023-05-31T20:55:01.387 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:55:01 GMT /pklxkd-59/ 2023-05-31T20:55:01.387 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:d7ZqE7fjjpja5oroUaADG51170Q= 2023-05-31T20:55:01.388 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:55:01 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:d7ZqE7fjjpja5oroUaADG51170Q=', 'Content-Length': '0'} 2023-05-31T20:55:01.392 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000069dda2bf045a7d0b-006477b425-1630-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:55:01 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:55:01.432 DEBUG:boto:b'pklxkd-59obj931000falseobj932023-05-31T20:50:33.533Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:55:01.433 DEBUG:boto:path=/pklxkd-59/obj93 2023-05-31T20:55:01.433 DEBUG:boto:auth_path=/pklxkd-59/obj93 2023-05-31T20:55:01.434 DEBUG:boto:Method: HEAD 2023-05-31T20:55:01.434 DEBUG:boto:Path: /pklxkd-59/obj93 2023-05-31T20:55:01.434 DEBUG:boto:Data: 2023-05-31T20:55:01.435 DEBUG:boto:Headers: {} 2023-05-31T20:55:01.435 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:55:01.435 DEBUG:boto:Port: 8001 2023-05-31T20:55:01.436 DEBUG:boto:Params: {} 2023-05-31T20:55:01.436 DEBUG:boto:Token: None 2023-05-31T20:55:01.436 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:55:01 GMT /pklxkd-59/obj93 2023-05-31T20:55:01.437 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:+x21AgS2bclXfCfD+kX2A5KsxvY= 2023-05-31T20:55:01.437 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:55:01 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:+x21AgS2bclXfCfD+kX2A5KsxvY=', 'Content-Length': '0'} 2023-05-31T20:55:01.439 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:50:33 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003003137af3dc4719-006477b425-1e2e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:55:01 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:55:01.440 DEBUG:boto:path=/pklxkd-59/obj93 2023-05-31T20:55:01.440 DEBUG:boto:auth_path=/pklxkd-59/obj93 2023-05-31T20:55:01.440 DEBUG:boto:Method: HEAD 2023-05-31T20:55:01.441 DEBUG:boto:Path: /pklxkd-59/obj93 2023-05-31T20:55:01.441 DEBUG:boto:Data: 2023-05-31T20:55:01.441 DEBUG:boto:Headers: {} 2023-05-31T20:55:01.442 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:55:01.442 DEBUG:boto:Port: 8000 2023-05-31T20:55:01.442 DEBUG:boto:Params: {} 2023-05-31T20:55:01.443 DEBUG:boto:Token: None 2023-05-31T20:55:01.443 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:55:01 GMT /pklxkd-59/obj93 2023-05-31T20:55:01.443 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:+x21AgS2bclXfCfD+kX2A5KsxvY= 2023-05-31T20:55:01.444 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:55:01 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:+x21AgS2bclXfCfD+kX2A5KsxvY=', 'Content-Length': '0'} 2023-05-31T20:55:01.446 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:50:33 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000088b2b6f2f7499300-006477b425-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:55:01 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:55:01.446 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj93 2023-05-31T20:55:01.447 DEBUG:boto:path=/pklxkd-59/obj93 2023-05-31T20:55:01.447 DEBUG:boto:auth_path=/pklxkd-59/obj93 2023-05-31T20:55:01.447 DEBUG:boto:Method: GET 2023-05-31T20:55:01.448 DEBUG:boto:Path: /pklxkd-59/obj93 2023-05-31T20:55:01.448 DEBUG:boto:Data: 2023-05-31T20:55:01.449 DEBUG:boto:Headers: {} 2023-05-31T20:55:01.449 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:55:01.449 DEBUG:boto:Port: 8001 2023-05-31T20:55:01.450 DEBUG:boto:Params: {} 2023-05-31T20:55:01.450 DEBUG:boto:Token: None 2023-05-31T20:55:01.450 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:55:01 GMT /pklxkd-59/obj93 2023-05-31T20:55:01.451 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:ugEhw4/XwCkhijU2K1Wskm8vTGk= 2023-05-31T20:55:01.451 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:55:01 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:ugEhw4/XwCkhijU2K1Wskm8vTGk=', 'Content-Length': '0'} 2023-05-31T20:55:01.453 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:50:33 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000047cecedf15a539b6-006477b425-1e2e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:55:01 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:55:01.453 DEBUG:boto:path=/pklxkd-59/obj93 2023-05-31T20:55:01.454 DEBUG:boto:auth_path=/pklxkd-59/obj93 2023-05-31T20:55:01.454 DEBUG:boto:Method: GET 2023-05-31T20:55:01.455 DEBUG:boto:Path: /pklxkd-59/obj93 2023-05-31T20:55:01.455 DEBUG:boto:Data: 2023-05-31T20:55:01.455 DEBUG:boto:Headers: {} 2023-05-31T20:55:01.456 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:55:01.456 DEBUG:boto:Port: 8000 2023-05-31T20:55:01.457 DEBUG:boto:Params: {} 2023-05-31T20:55:01.457 DEBUG:boto:Token: None 2023-05-31T20:55:01.457 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:55:01 GMT /pklxkd-59/obj93 2023-05-31T20:55:01.458 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:ugEhw4/XwCkhijU2K1Wskm8vTGk= 2023-05-31T20:55:01.458 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:55:01 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:ugEhw4/XwCkhijU2K1Wskm8vTGk=', 'Content-Length': '0'} 2023-05-31T20:55:01.460 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:50:33 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000440786f0843b5f65-006477b425-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:55:01 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:55:01.461 DEBUG:boto:path=/pklxkd-59/ 2023-05-31T20:55:01.461 DEBUG:boto:auth_path=/pklxkd-59/ 2023-05-31T20:55:01.462 DEBUG:boto:path=/pklxkd-59/?prefix=obj93 2023-05-31T20:55:01.462 DEBUG:boto:auth_path=/pklxkd-59/?prefix=obj93 2023-05-31T20:55:01.462 DEBUG:boto:Method: GET 2023-05-31T20:55:01.463 DEBUG:boto:Path: /pklxkd-59/?prefix=obj93 2023-05-31T20:55:01.463 DEBUG:boto:Data: 2023-05-31T20:55:01.463 DEBUG:boto:Headers: {} 2023-05-31T20:55:01.464 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:55:01.464 DEBUG:boto:Port: 8001 2023-05-31T20:55:01.464 DEBUG:boto:Params: {} 2023-05-31T20:55:01.465 DEBUG:boto:Token: None 2023-05-31T20:55:01.465 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:55:01 GMT /pklxkd-59/ 2023-05-31T20:55:01.466 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:d7ZqE7fjjpja5oroUaADG51170Q= 2023-05-31T20:55:01.466 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:55:01 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:d7ZqE7fjjpja5oroUaADG51170Q=', 'Content-Length': '0'} 2023-05-31T20:55:01.468 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b7bacb06b1e60eac-006477b425-1e2e-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:55:01 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:55:01.512 DEBUG:boto:b'pklxkd-59obj931000falseobj932023-05-31T20:50:33.533Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:55:01.513 DEBUG:boto:path=/pklxkd-59/ 2023-05-31T20:55:01.513 DEBUG:boto:auth_path=/pklxkd-59/ 2023-05-31T20:55:01.514 DEBUG:boto:path=/pklxkd-59/?prefix=obj93 2023-05-31T20:55:01.514 DEBUG:boto:auth_path=/pklxkd-59/?prefix=obj93 2023-05-31T20:55:01.514 DEBUG:boto:Method: GET 2023-05-31T20:55:01.515 DEBUG:boto:Path: /pklxkd-59/?prefix=obj93 2023-05-31T20:55:01.515 DEBUG:boto:Data: 2023-05-31T20:55:01.515 DEBUG:boto:Headers: {} 2023-05-31T20:55:01.516 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:55:01.516 DEBUG:boto:Port: 8000 2023-05-31T20:55:01.516 DEBUG:boto:Params: {} 2023-05-31T20:55:01.517 DEBUG:boto:Token: None 2023-05-31T20:55:01.517 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:55:01 GMT /pklxkd-59/ 2023-05-31T20:55:01.518 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:d7ZqE7fjjpja5oroUaADG51170Q= 2023-05-31T20:55:01.518 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:55:01 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:d7ZqE7fjjpja5oroUaADG51170Q=', 'Content-Length': '0'} 2023-05-31T20:55:01.522 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b7ea7946a5d92d5d-006477b425-1630-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:55:01 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:55:01.523 DEBUG:boto:b'pklxkd-59obj931000falseobj932023-05-31T20:50:33.533Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:55:01.523 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj95 2023-05-31T20:55:01.523 DEBUG:boto:path=/pklxkd-59/obj95 2023-05-31T20:55:01.524 DEBUG:boto:auth_path=/pklxkd-59/obj95 2023-05-31T20:55:01.524 DEBUG:boto:path=/pklxkd-59/obj95?versionId=null 2023-05-31T20:55:01.525 DEBUG:boto:auth_path=/pklxkd-59/obj95?versionId=null 2023-05-31T20:55:01.525 DEBUG:boto:Method: GET 2023-05-31T20:55:01.525 DEBUG:boto:Path: /pklxkd-59/obj95?versionId=null 2023-05-31T20:55:01.526 DEBUG:boto:Data: 2023-05-31T20:55:01.526 DEBUG:boto:Headers: {} 2023-05-31T20:55:01.526 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:55:01.527 DEBUG:boto:Port: 8001 2023-05-31T20:55:01.527 DEBUG:boto:Params: {} 2023-05-31T20:55:01.527 DEBUG:boto:Token: None 2023-05-31T20:55:01.528 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:55:01 GMT /pklxkd-59/obj95?versionId=null 2023-05-31T20:55:01.528 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:ruMpFL42jJZmwPOSKGIQwjeeKBw= 2023-05-31T20:55:01.529 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:55:01 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:ruMpFL42jJZmwPOSKGIQwjeeKBw=', 'Content-Length': '0'} 2023-05-31T20:55:01.531 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:44:50 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e835fc22bfa449cb-006477b425-1e2e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:55:01 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:55:01.532 DEBUG:boto:path=/pklxkd-59/obj95 2023-05-31T20:55:01.532 DEBUG:boto:auth_path=/pklxkd-59/obj95 2023-05-31T20:55:01.532 DEBUG:boto:path=/pklxkd-59/obj95?versionId=null 2023-05-31T20:55:01.533 DEBUG:boto:auth_path=/pklxkd-59/obj95?versionId=null 2023-05-31T20:55:01.533 DEBUG:boto:Method: GET 2023-05-31T20:55:01.533 DEBUG:boto:Path: /pklxkd-59/obj95?versionId=null 2023-05-31T20:55:01.534 DEBUG:boto:Data: 2023-05-31T20:55:01.534 DEBUG:boto:Headers: {} 2023-05-31T20:55:01.535 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:55:01.535 DEBUG:boto:Port: 8000 2023-05-31T20:55:01.535 DEBUG:boto:Params: {} 2023-05-31T20:55:01.536 DEBUG:boto:Token: None 2023-05-31T20:55:01.536 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:55:01 GMT /pklxkd-59/obj95?versionId=null 2023-05-31T20:55:01.536 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:ruMpFL42jJZmwPOSKGIQwjeeKBw= 2023-05-31T20:55:01.537 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:55:01 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:ruMpFL42jJZmwPOSKGIQwjeeKBw=', 'Content-Length': '0'} 2023-05-31T20:55:01.539 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:44:50 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000753e61bfd00f4201-006477b425-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:55:01 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:55:01.539 DEBUG:boto:path=/pklxkd-59/obj95 2023-05-31T20:55:01.540 DEBUG:boto:auth_path=/pklxkd-59/obj95 2023-05-31T20:55:01.540 DEBUG:boto:path=/pklxkd-59/obj95?versionId=null 2023-05-31T20:55:01.540 DEBUG:boto:auth_path=/pklxkd-59/obj95?versionId=null 2023-05-31T20:55:01.541 DEBUG:boto:Method: HEAD 2023-05-31T20:55:01.541 DEBUG:boto:Path: /pklxkd-59/obj95?versionId=null 2023-05-31T20:55:01.541 DEBUG:boto:Data: 2023-05-31T20:55:01.542 DEBUG:boto:Headers: {} 2023-05-31T20:55:01.542 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:55:01.542 DEBUG:boto:Port: 8001 2023-05-31T20:55:01.543 DEBUG:boto:Params: {} 2023-05-31T20:55:01.543 DEBUG:boto:Token: None 2023-05-31T20:55:01.543 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:55:01 GMT /pklxkd-59/obj95?versionId=null 2023-05-31T20:55:01.544 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:XKoOaVdSxiWpJMd838HWhiAo3JA= 2023-05-31T20:55:01.544 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:55:01 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:XKoOaVdSxiWpJMd838HWhiAo3JA=', 'Content-Length': '0'} 2023-05-31T20:55:01.546 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:44:50 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007871db571428afef-006477b425-1e2e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:55:01 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:55:01.546 DEBUG:boto:path=/pklxkd-59/obj95 2023-05-31T20:55:01.547 DEBUG:boto:auth_path=/pklxkd-59/obj95 2023-05-31T20:55:01.547 DEBUG:boto:path=/pklxkd-59/obj95?versionId=null 2023-05-31T20:55:01.547 DEBUG:boto:auth_path=/pklxkd-59/obj95?versionId=null 2023-05-31T20:55:01.548 DEBUG:boto:Method: HEAD 2023-05-31T20:55:01.548 DEBUG:boto:Path: /pklxkd-59/obj95?versionId=null 2023-05-31T20:55:01.549 DEBUG:boto:Data: 2023-05-31T20:55:01.549 DEBUG:boto:Headers: {} 2023-05-31T20:55:01.549 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:55:01.549 DEBUG:boto:Port: 8000 2023-05-31T20:55:01.550 DEBUG:boto:Params: {} 2023-05-31T20:55:01.550 DEBUG:boto:Token: None 2023-05-31T20:55:01.551 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:55:01 GMT /pklxkd-59/obj95?versionId=null 2023-05-31T20:55:01.551 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:XKoOaVdSxiWpJMd838HWhiAo3JA= 2023-05-31T20:55:01.551 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:55:01 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:XKoOaVdSxiWpJMd838HWhiAo3JA=', 'Content-Length': '0'} 2023-05-31T20:55:01.553 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:44:50 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000018e9112bf728d5eb-006477b425-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:55:01 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:55:01.554 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj95 2023-05-31T20:55:01.554 DEBUG:boto:path=/pklxkd-59/obj95 2023-05-31T20:55:01.554 DEBUG:boto:auth_path=/pklxkd-59/obj95 2023-05-31T20:55:01.555 DEBUG:boto:path=/pklxkd-59/obj95?versionId=null 2023-05-31T20:55:01.555 DEBUG:boto:auth_path=/pklxkd-59/obj95?versionId=null 2023-05-31T20:55:01.555 DEBUG:boto:Method: GET 2023-05-31T20:55:01.556 DEBUG:boto:Path: /pklxkd-59/obj95?versionId=null 2023-05-31T20:55:01.556 DEBUG:boto:Data: 2023-05-31T20:55:01.557 DEBUG:boto:Headers: {} 2023-05-31T20:55:01.557 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:55:01.557 DEBUG:boto:Port: 8001 2023-05-31T20:55:01.558 DEBUG:boto:Params: {} 2023-05-31T20:55:01.558 DEBUG:boto:Token: None 2023-05-31T20:55:01.558 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:55:01 GMT /pklxkd-59/obj95?versionId=null 2023-05-31T20:55:01.559 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:ruMpFL42jJZmwPOSKGIQwjeeKBw= 2023-05-31T20:55:01.559 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:55:01 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:ruMpFL42jJZmwPOSKGIQwjeeKBw=', 'Content-Length': '0'} 2023-05-31T20:55:01.561 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:44:50 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e86ab11f477dd07e-006477b425-1e2e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:55:01 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:55:01.561 DEBUG:boto:path=/pklxkd-59/obj95 2023-05-31T20:55:01.562 DEBUG:boto:auth_path=/pklxkd-59/obj95 2023-05-31T20:55:01.562 DEBUG:boto:path=/pklxkd-59/obj95?versionId=null 2023-05-31T20:55:01.563 DEBUG:boto:auth_path=/pklxkd-59/obj95?versionId=null 2023-05-31T20:55:01.563 DEBUG:boto:Method: GET 2023-05-31T20:55:01.563 DEBUG:boto:Path: /pklxkd-59/obj95?versionId=null 2023-05-31T20:55:01.564 DEBUG:boto:Data: 2023-05-31T20:55:01.564 DEBUG:boto:Headers: {} 2023-05-31T20:55:01.564 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:55:01.565 DEBUG:boto:Port: 8000 2023-05-31T20:55:01.565 DEBUG:boto:Params: {} 2023-05-31T20:55:01.565 DEBUG:boto:Token: None 2023-05-31T20:55:01.566 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:55:01 GMT /pklxkd-59/obj95?versionId=null 2023-05-31T20:55:01.566 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:ruMpFL42jJZmwPOSKGIQwjeeKBw= 2023-05-31T20:55:01.567 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:55:01 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:ruMpFL42jJZmwPOSKGIQwjeeKBw=', 'Content-Length': '0'} 2023-05-31T20:55:01.568 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:44:50 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c40243ba51531b4e-006477b425-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:55:01 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:55:01.612 DEBUG:boto:path=/pklxkd-59/ 2023-05-31T20:55:01.613 DEBUG:boto:auth_path=/pklxkd-59/ 2023-05-31T20:55:01.613 DEBUG:boto:path=/pklxkd-59/?prefix=obj95 2023-05-31T20:55:01.613 DEBUG:boto:auth_path=/pklxkd-59/?prefix=obj95 2023-05-31T20:55:01.614 DEBUG:boto:Method: GET 2023-05-31T20:55:01.614 DEBUG:boto:Path: /pklxkd-59/?prefix=obj95 2023-05-31T20:55:01.614 DEBUG:boto:Data: 2023-05-31T20:55:01.615 DEBUG:boto:Headers: {} 2023-05-31T20:55:01.615 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:55:01.616 DEBUG:boto:Port: 8001 2023-05-31T20:55:01.616 DEBUG:boto:Params: {} 2023-05-31T20:55:01.616 DEBUG:boto:Token: None 2023-05-31T20:55:01.617 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:55:01 GMT /pklxkd-59/ 2023-05-31T20:55:01.617 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:d7ZqE7fjjpja5oroUaADG51170Q= 2023-05-31T20:55:01.618 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:55:01 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:d7ZqE7fjjpja5oroUaADG51170Q=', 'Content-Length': '0'} 2023-05-31T20:55:01.620 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000868985f3c34ea86d-006477b425-1e2e-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:55:01 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:55:01.620 DEBUG:boto:b'pklxkd-59obj951000falseobj952023-05-31T20:44:50.705Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:55:01.621 DEBUG:boto:path=/pklxkd-59/ 2023-05-31T20:55:01.621 DEBUG:boto:auth_path=/pklxkd-59/ 2023-05-31T20:55:01.622 DEBUG:boto:path=/pklxkd-59/?prefix=obj95 2023-05-31T20:55:01.622 DEBUG:boto:auth_path=/pklxkd-59/?prefix=obj95 2023-05-31T20:55:01.623 DEBUG:boto:Method: GET 2023-05-31T20:55:01.623 DEBUG:boto:Path: /pklxkd-59/?prefix=obj95 2023-05-31T20:55:01.623 DEBUG:boto:Data: 2023-05-31T20:55:01.624 DEBUG:boto:Headers: {} 2023-05-31T20:55:01.624 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:55:01.624 DEBUG:boto:Port: 8000 2023-05-31T20:55:01.625 DEBUG:boto:Params: {} 2023-05-31T20:55:01.625 DEBUG:boto:Token: None 2023-05-31T20:55:01.625 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:55:01 GMT /pklxkd-59/ 2023-05-31T20:55:01.626 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:d7ZqE7fjjpja5oroUaADG51170Q= 2023-05-31T20:55:01.626 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:55:01 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:d7ZqE7fjjpja5oroUaADG51170Q=', 'Content-Length': '0'} 2023-05-31T20:55:01.630 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000453edc102ec91a3f-006477b425-1630-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:55:01 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:55:01.631 DEBUG:boto:b'pklxkd-59obj951000falseobj952023-05-31T20:44:50.705Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:55:01.632 DEBUG:boto:path=/pklxkd-59/obj95 2023-05-31T20:55:01.632 DEBUG:boto:auth_path=/pklxkd-59/obj95 2023-05-31T20:55:01.632 DEBUG:boto:Method: HEAD 2023-05-31T20:55:01.633 DEBUG:boto:Path: /pklxkd-59/obj95 2023-05-31T20:55:01.633 DEBUG:boto:Data: 2023-05-31T20:55:01.633 DEBUG:boto:Headers: {} 2023-05-31T20:55:01.634 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:55:01.634 DEBUG:boto:Port: 8001 2023-05-31T20:55:01.634 DEBUG:boto:Params: {} 2023-05-31T20:55:01.635 DEBUG:boto:Token: None 2023-05-31T20:55:01.635 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:55:01 GMT /pklxkd-59/obj95 2023-05-31T20:55:01.635 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:uTHZsvMPfrpLwV/HXa03Ed5Zb20= 2023-05-31T20:55:01.636 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:55:01 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:uTHZsvMPfrpLwV/HXa03Ed5Zb20=', 'Content-Length': '0'} 2023-05-31T20:55:01.638 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:44:50 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c74ffadf8a824009-006477b425-1e2e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:55:01 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:55:01.638 DEBUG:boto:path=/pklxkd-59/obj95 2023-05-31T20:55:01.638 DEBUG:boto:auth_path=/pklxkd-59/obj95 2023-05-31T20:55:01.639 DEBUG:boto:Method: HEAD 2023-05-31T20:55:01.639 DEBUG:boto:Path: /pklxkd-59/obj95 2023-05-31T20:55:01.639 DEBUG:boto:Data: 2023-05-31T20:55:01.640 DEBUG:boto:Headers: {} 2023-05-31T20:55:01.640 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:55:01.641 DEBUG:boto:Port: 8000 2023-05-31T20:55:01.641 DEBUG:boto:Params: {} 2023-05-31T20:55:01.641 DEBUG:boto:Token: None 2023-05-31T20:55:01.642 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:55:01 GMT /pklxkd-59/obj95 2023-05-31T20:55:01.642 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:uTHZsvMPfrpLwV/HXa03Ed5Zb20= 2023-05-31T20:55:01.642 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:55:01 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:uTHZsvMPfrpLwV/HXa03Ed5Zb20=', 'Content-Length': '0'} 2023-05-31T20:55:01.644 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:44:50 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e8f5171541ac2c28-006477b425-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:55:01 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:55:01.645 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj95 2023-05-31T20:55:01.645 DEBUG:boto:path=/pklxkd-59/obj95 2023-05-31T20:55:01.646 DEBUG:boto:auth_path=/pklxkd-59/obj95 2023-05-31T20:55:01.646 DEBUG:boto:Method: GET 2023-05-31T20:55:01.646 DEBUG:boto:Path: /pklxkd-59/obj95 2023-05-31T20:55:01.647 DEBUG:boto:Data: 2023-05-31T20:55:01.647 DEBUG:boto:Headers: {} 2023-05-31T20:55:01.647 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:55:01.648 DEBUG:boto:Port: 8001 2023-05-31T20:55:01.648 DEBUG:boto:Params: {} 2023-05-31T20:55:01.648 DEBUG:boto:Token: None 2023-05-31T20:55:01.649 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:55:01 GMT /pklxkd-59/obj95 2023-05-31T20:55:01.649 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:GV2Q/3Wun+EndHewk6qePaiAH8U= 2023-05-31T20:55:01.650 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:55:01 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:GV2Q/3Wun+EndHewk6qePaiAH8U=', 'Content-Length': '0'} 2023-05-31T20:55:01.652 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:44:50 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000cd8f1623a3b3f419-006477b425-1e2e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:55:01 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:55:01.652 DEBUG:boto:path=/pklxkd-59/obj95 2023-05-31T20:55:01.653 DEBUG:boto:auth_path=/pklxkd-59/obj95 2023-05-31T20:55:01.653 DEBUG:boto:Method: GET 2023-05-31T20:55:01.653 DEBUG:boto:Path: /pklxkd-59/obj95 2023-05-31T20:55:01.654 DEBUG:boto:Data: 2023-05-31T20:55:01.654 DEBUG:boto:Headers: {} 2023-05-31T20:55:01.654 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:55:01.655 DEBUG:boto:Port: 8000 2023-05-31T20:55:01.655 DEBUG:boto:Params: {} 2023-05-31T20:55:01.656 DEBUG:boto:Token: None 2023-05-31T20:55:01.656 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:55:01 GMT /pklxkd-59/obj95 2023-05-31T20:55:01.656 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:GV2Q/3Wun+EndHewk6qePaiAH8U= 2023-05-31T20:55:01.657 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:55:01 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:GV2Q/3Wun+EndHewk6qePaiAH8U=', 'Content-Length': '0'} 2023-05-31T20:55:01.659 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:44:50 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000cdc0c0159c05ea40-006477b425-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:55:01 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:55:01.700 DEBUG:boto:path=/pklxkd-59/ 2023-05-31T20:55:01.701 DEBUG:boto:auth_path=/pklxkd-59/ 2023-05-31T20:55:01.702 DEBUG:boto:path=/pklxkd-59/?prefix=obj95 2023-05-31T20:55:01.702 DEBUG:boto:auth_path=/pklxkd-59/?prefix=obj95 2023-05-31T20:55:01.702 DEBUG:boto:Method: GET 2023-05-31T20:55:01.703 DEBUG:boto:Path: /pklxkd-59/?prefix=obj95 2023-05-31T20:55:01.703 DEBUG:boto:Data: 2023-05-31T20:55:01.703 DEBUG:boto:Headers: {} 2023-05-31T20:55:01.704 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:55:01.704 DEBUG:boto:Port: 8001 2023-05-31T20:55:01.704 DEBUG:boto:Params: {} 2023-05-31T20:55:01.705 DEBUG:boto:Token: None 2023-05-31T20:55:01.705 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:55:01 GMT /pklxkd-59/ 2023-05-31T20:55:01.706 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:d7ZqE7fjjpja5oroUaADG51170Q= 2023-05-31T20:55:01.706 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:55:01 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:d7ZqE7fjjpja5oroUaADG51170Q=', 'Content-Length': '0'} 2023-05-31T20:55:01.709 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000fb0ae2f7234e15af-006477b425-1e2e-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:55:01 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:55:01.709 DEBUG:boto:b'pklxkd-59obj951000falseobj952023-05-31T20:44:50.705Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:55:01.710 DEBUG:boto:path=/pklxkd-59/ 2023-05-31T20:55:01.710 DEBUG:boto:auth_path=/pklxkd-59/ 2023-05-31T20:55:01.711 DEBUG:boto:path=/pklxkd-59/?prefix=obj95 2023-05-31T20:55:01.711 DEBUG:boto:auth_path=/pklxkd-59/?prefix=obj95 2023-05-31T20:55:01.711 DEBUG:boto:Method: GET 2023-05-31T20:55:01.712 DEBUG:boto:Path: /pklxkd-59/?prefix=obj95 2023-05-31T20:55:01.712 DEBUG:boto:Data: 2023-05-31T20:55:01.713 DEBUG:boto:Headers: {} 2023-05-31T20:55:01.713 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:55:01.713 DEBUG:boto:Port: 8000 2023-05-31T20:55:01.714 DEBUG:boto:Params: {} 2023-05-31T20:55:01.714 DEBUG:boto:Token: None 2023-05-31T20:55:01.715 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:55:01 GMT /pklxkd-59/ 2023-05-31T20:55:01.715 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:d7ZqE7fjjpja5oroUaADG51170Q= 2023-05-31T20:55:01.716 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:55:01 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:d7ZqE7fjjpja5oroUaADG51170Q=', 'Content-Length': '0'} 2023-05-31T20:55:01.720 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000038693161a2ae297d-006477b425-1630-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:55:01 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:55:01.720 DEBUG:boto:b'pklxkd-59obj951000falseobj952023-05-31T20:44:50.705Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:55:01.721 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=pklxkd-59 zones={a2, a1} 2023-05-31T20:55:01.722 INFO:tasks.rgw_multisite_tests:Create several generations of objects, trash them, then run bucket sync init ... ok 2023-05-31T20:55:01.727 INFO:rgw_multi.tests:create role zone=a1 name=roles-pklxkd-1 2023-05-31T20:55:01.727 DEBUG:boto:Method: POST 2023-05-31T20:55:01.727 DEBUG:boto:Path: / 2023-05-31T20:55:01.728 DEBUG:boto:Data: 2023-05-31T20:55:01.728 DEBUG:boto:Headers: {} 2023-05-31T20:55:01.728 DEBUG:boto:Host: smithi099.front.sepia.ceph.com 2023-05-31T20:55:01.729 DEBUG:boto:Port: 8000 2023-05-31T20:55:01.729 DEBUG:boto:Params: {'RoleName': 'roles-pklxkd-1', 'AssumeRolePolicyDocument': '{"Version":"2012-10-17","Statement":[{"Effect":"Allow","Principal":{"AWS":["arn:aws:iam:::user/testuser"]},"Action":["sts:AssumeRole"]}]}', 'Path': '', 'Action': 'CreateRole', 'Version': '2010-05-08'} 2023-05-31T20:55:01.730 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2023-05-31T20:55:01.730 DEBUG:boto:Token: None 2023-05-31T20:55:01.730 DEBUG:boto:CanonicalRequest: POST / host:smithi099.front.sepia.ceph.com:8000 x-amz-date:20230531T205501Z host;x-amz-date 548b337e94aa8b4f4da1713e3d74bdd7aa9cb140af27a89221ca45ebb70e6dbd 2023-05-31T20:55:01.731 DEBUG:boto:StringToSign: AWS4-HMAC-SHA256 20230531T205501Z 20230531/front/smithi099/aws4_request faee090e8551dafcb6bd598668c7b4fa88b5501c0089807dca4060fd2c567c49 2023-05-31T20:55:01.731 DEBUG:boto:Signature: 91bb7f78c1e66e61e825881d2f7ad9ee4d281583fe700553b7503a7db2cbb5bb 2023-05-31T20:55:01.732 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'X-Amz-Date': '20230531T205501Z', 'Content-Type': 'application/x-www-form-urlencoded; charset=UTF-8', 'Content-Length': '325', 'Authorization': 'AWS4-HMAC-SHA256 Credential=Q80H11OLNU1S48PPYL26/20230531/front/smithi099/aws4_request,SignedHeaders=host;x-amz-date,Signature=91bb7f78c1e66e61e825881d2f7ad9ee4d281583fe700553b7503a7db2cbb5bb', 'Host': 'smithi099.front.sepia.ceph.com:8000'} 2023-05-31T20:55:01.744 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000dbf0a90a3a14aea0-006477b425-1630-a1'), ('Content-Type', 'application/xml'), ('Content-Length', '723'), ('Date', 'Wed, 31 May 2023 20:55:01 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:55:01.744 DEBUG:boto:b'315bba75-8952-4f92-bf4b-6459a819a42broles-pklxkd-1/arn:aws:iam:::role/roles-pklxkd-12023-05-31T20:55:01.732Z3600{"Version":"2012-10-17","Statement":[{"Effect":"Allow","Principal":{"AWS":["arn:aws:iam:::user/testuser"]},"Action":["sts:AssumeRole"]}]}tx00000dbf0a90a3a14aea0-006477b425-1630-a1' 2023-05-31T20:55:01.748 INFO:rgw_multi.tests:create role zone=a2 name=roles-pklxkd-2 2023-05-31T20:55:01.748 DEBUG:boto:Method: POST 2023-05-31T20:55:01.749 DEBUG:boto:Path: / 2023-05-31T20:55:01.749 DEBUG:boto:Data: 2023-05-31T20:55:01.749 DEBUG:boto:Headers: {} 2023-05-31T20:55:01.750 DEBUG:boto:Host: smithi099.front.sepia.ceph.com 2023-05-31T20:55:01.750 DEBUG:boto:Port: 8001 2023-05-31T20:55:01.750 DEBUG:boto:Params: {'RoleName': 'roles-pklxkd-2', 'AssumeRolePolicyDocument': '{"Version":"2012-10-17","Statement":[{"Effect":"Allow","Principal":{"AWS":["arn:aws:iam:::user/testuser"]},"Action":["sts:AssumeRole"]}]}', 'Path': '', 'Action': 'CreateRole', 'Version': '2010-05-08'} 2023-05-31T20:55:01.751 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8001} 2023-05-31T20:55:01.751 DEBUG:boto:Token: None 2023-05-31T20:55:01.752 DEBUG:boto:CanonicalRequest: POST / host:smithi099.front.sepia.ceph.com:8001 x-amz-date:20230531T205501Z host;x-amz-date 14dc48084991ab0a748f0d718bf60c45c09dd1f9cec393029491b8de5741c89b 2023-05-31T20:55:01.752 DEBUG:boto:StringToSign: AWS4-HMAC-SHA256 20230531T205501Z 20230531/front/smithi099/aws4_request 3ca2e4ab1843086dd6338d5af81167954982abbb43e65d94db122e997fed316f 2023-05-31T20:55:01.753 DEBUG:boto:Signature: b68782189a910dce612ecc026c07bbfb446641fcef5d75c909ece73ca3a80dd8 2023-05-31T20:55:01.753 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'X-Amz-Date': '20230531T205501Z', 'Content-Type': 'application/x-www-form-urlencoded; charset=UTF-8', 'Content-Length': '325', 'Authorization': 'AWS4-HMAC-SHA256 Credential=Q80H11OLNU1S48PPYL26/20230531/front/smithi099/aws4_request,SignedHeaders=host;x-amz-date,Signature=b68782189a910dce612ecc026c07bbfb446641fcef5d75c909ece73ca3a80dd8', 'Host': 'smithi099.front.sepia.ceph.com:8001'} 2023-05-31T20:55:01.775 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000ee764670743fb4c1-006477b425-1e2e-a2'), ('Content-Type', 'application/xml'), ('Content-Length', '723'), ('Date', 'Wed, 31 May 2023 20:55:01 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:55:01.775 DEBUG:boto:b'935318be-8452-4a09-9b0e-e7491b2d22e4roles-pklxkd-2/arn:aws:iam:::role/roles-pklxkd-22023-05-31T20:55:01.765Z3600{"Version":"2012-10-17","Statement":[{"Effect":"Allow","Principal":{"AWS":["arn:aws:iam:::user/testuser"]},"Action":["sts:AssumeRole"]}]}tx00000ee764670743fb4c1-006477b425-1e2e-a2' 2023-05-31T20:55:01.776 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-31T20:55:01.777 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-31T20:55:01.777 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-05-31T20:55:01.831 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2023-05-31T20:55:01.832 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2023-05-31T20:55:01.892 INFO:teuthology.orchestra.run.smithi099.stderr:No --period given, using current period=2e6123d2-d695-4350-9a01-61a455bc782b 2023-05-31T20:55:01.916 INFO:teuthology.orchestra.run.smithi099.stdout:[ 2023-05-31T20:55:01.916 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T20:55:01.916 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1685566501.757183_260.1", 2023-05-31T20:55:01.916 INFO:teuthology.orchestra.run.smithi099.stdout: "last_update": "2023-05-31T20:55:01.757183Z" 2023-05-31T20:55:01.917 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:55:01.917 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T20:55:01.917 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1685566501.735717_251.1", 2023-05-31T20:55:01.917 INFO:teuthology.orchestra.run.smithi099.stdout: "last_update": "2023-05-31T20:55:01.735717Z" 2023-05-31T20:55:01.917 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:55:01.917 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T20:55:01.917 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1685562970.274572_93.1", 2023-05-31T20:55:01.918 INFO:teuthology.orchestra.run.smithi099.stdout: "last_update": "2023-05-31T19:56:10.274572Z" 2023-05-31T20:55:01.918 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:55:01.918 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T20:55:01.918 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1685566334.798037_482.1", 2023-05-31T20:55:01.918 INFO:teuthology.orchestra.run.smithi099.stdout: "last_update": "2023-05-31T20:52:14.798037Z" 2023-05-31T20:55:01.918 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T20:55:01.918 INFO:teuthology.orchestra.run.smithi099.stdout:] 2023-05-31T20:55:01.919 DEBUG:rgw_multi.tests:master meta markers={0: '1_1685566501.757183_260.1', 1: '1_1685566501.735717_251.1', 2: '1_1685562970.274572_93.1', 3: '1_1685566334.798037_482.1'} 2023-05-31T20:55:01.920 INFO:rgw_multi.tests:starting meta checkpoint for zone=a2 2023-05-31T20:55:01.920 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-31T20:55:01.921 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-31T20:55:01.921 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-05-31T20:55:01.969 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2023-05-31T20:55:01.970 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2023-05-31T20:55:02.011 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:55:02.007+0000 7fd9c466d780 20 RGW-SYNC:meta: read sync status 2023-05-31T20:55:02.012 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:55:02.008+0000 7fd9c466d780 20 RGW-SYNC:meta: read sync status 2023-05-31T20:55:02.026 INFO:teuthology.orchestra.run.smithi099.stdout:{ 2023-05-31T20:55:02.026 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_status": { 2023-05-31T20:55:02.026 INFO:teuthology.orchestra.run.smithi099.stdout: "info": { 2023-05-31T20:55:02.026 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "sync", 2023-05-31T20:55:02.027 INFO:teuthology.orchestra.run.smithi099.stdout: "num_shards": 4, 2023-05-31T20:55:02.027 INFO:teuthology.orchestra.run.smithi099.stdout: "period": "2e6123d2-d695-4350-9a01-61a455bc782b", 2023-05-31T20:55:02.027 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2023-05-31T20:55:02.027 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:55:02.027 INFO:teuthology.orchestra.run.smithi099.stdout: "markers": [ 2023-05-31T20:55:02.027 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T20:55:02.028 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 0, 2023-05-31T20:55:02.028 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T20:55:02.028 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2023-05-31T20:55:02.028 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1685562631.516089_59.1", 2023-05-31T20:55:02.028 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T20:55:02.028 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 1, 2023-05-31T20:55:02.028 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T20:55:02.029 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2023-05-31T19:50:31.516089Z", 2023-05-31T20:55:02.029 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2023-05-31T20:55:02.029 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T20:55:02.029 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:55:02.029 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T20:55:02.029 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 1, 2023-05-31T20:55:02.029 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T20:55:02.030 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2023-05-31T20:55:02.030 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "", 2023-05-31T20:55:02.030 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T20:55:02.030 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2023-05-31T20:55:02.030 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T20:55:02.030 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "0.000000", 2023-05-31T20:55:02.031 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2023-05-31T20:55:02.031 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T20:55:02.031 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:55:02.031 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T20:55:02.031 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 2, 2023-05-31T20:55:02.031 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T20:55:02.031 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2023-05-31T20:55:02.031 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1685562970.274572_93.1", 2023-05-31T20:55:02.032 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T20:55:02.032 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2023-05-31T20:55:02.032 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T20:55:02.032 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2023-05-31T19:56:10.274572Z", 2023-05-31T20:55:02.032 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2023-05-31T20:55:02.032 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T20:55:02.033 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:55:02.033 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T20:55:02.033 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 3, 2023-05-31T20:55:02.033 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T20:55:02.033 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2023-05-31T20:55:02.033 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1685566334.798037_482.1", 2023-05-31T20:55:02.033 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T20:55:02.034 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2023-05-31T20:55:02.034 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T20:55:02.034 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2023-05-31T20:52:14.798037Z", 2023-05-31T20:55:02.034 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2023-05-31T20:55:02.034 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T20:55:02.034 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T20:55:02.034 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2023-05-31T20:55:02.035 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:55:02.035 INFO:teuthology.orchestra.run.smithi099.stdout: "full_sync": { 2023-05-31T20:55:02.035 INFO:teuthology.orchestra.run.smithi099.stdout: "total": 1, 2023-05-31T20:55:02.035 INFO:teuthology.orchestra.run.smithi099.stdout: "complete": 1 2023-05-31T20:55:02.035 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:55:02.035 INFO:teuthology.orchestra.run.smithi099.stdout: "current_time": "2023-05-31T20:55:02Z" 2023-05-31T20:55:02.035 INFO:teuthology.orchestra.run.smithi099.stdout:} 2023-05-31T20:55:02.036 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "2e6123d2-d695-4350-9a01-61a455bc782b", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1685562631.516089_59.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-05-31T19:50:31.516089Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1685562970.274572_93.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-05-31T19:56:10.274572Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1685566334.798037_482.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-05-31T20:52:14.798037Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-05-31T20:55:02Z" } 2023-05-31T20:55:02.037 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1685562631.516089_59.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-05-31T19:50:31.516089Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1685562970.274572_93.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-05-31T19:56:10.274572Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1685566334.798037_482.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-05-31T20:52:14.798037Z', 'realm_epoch': 2}}] 2023-05-31T20:55:02.037 DEBUG:rgw_multi.tests:log_status={0: '1_1685566501.757183_260.1', 1: '1_1685566501.735717_251.1', 2: '1_1685562970.274572_93.1', 3: '1_1685566334.798037_482.1'} 2023-05-31T20:55:02.037 DEBUG:rgw_multi.tests:sync_status={0: '1_1685562631.516089_59.1', 1: '', 2: '1_1685562970.274572_93.1', 3: '1_1685566334.798037_482.1'} 2023-05-31T20:55:02.038 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1685566501.757183_260.1 target=1_1685562631.516089_59.1, shard=1 master=1_1685566501.735717_251.1 target= 2023-05-31T20:55:07.039 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-31T20:55:07.039 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-31T20:55:07.040 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-05-31T20:55:07.098 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2023-05-31T20:55:07.098 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2023-05-31T20:55:07.135 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:55:07.131+0000 7f7eb3204780 20 RGW-SYNC:meta: read sync status 2023-05-31T20:55:07.136 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:55:07.132+0000 7f7eb3204780 20 RGW-SYNC:meta: read sync status 2023-05-31T20:55:07.154 INFO:teuthology.orchestra.run.smithi099.stdout:{ 2023-05-31T20:55:07.154 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_status": { 2023-05-31T20:55:07.154 INFO:teuthology.orchestra.run.smithi099.stdout: "info": { 2023-05-31T20:55:07.155 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "sync", 2023-05-31T20:55:07.155 INFO:teuthology.orchestra.run.smithi099.stdout: "num_shards": 4, 2023-05-31T20:55:07.155 INFO:teuthology.orchestra.run.smithi099.stdout: "period": "2e6123d2-d695-4350-9a01-61a455bc782b", 2023-05-31T20:55:07.155 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2023-05-31T20:55:07.155 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:55:07.155 INFO:teuthology.orchestra.run.smithi099.stdout: "markers": [ 2023-05-31T20:55:07.155 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T20:55:07.156 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 0, 2023-05-31T20:55:07.156 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T20:55:07.156 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2023-05-31T20:55:07.156 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1685562631.516089_59.1", 2023-05-31T20:55:07.156 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T20:55:07.156 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 1, 2023-05-31T20:55:07.157 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T20:55:07.157 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2023-05-31T19:50:31.516089Z", 2023-05-31T20:55:07.157 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2023-05-31T20:55:07.157 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T20:55:07.157 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:55:07.157 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T20:55:07.157 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 1, 2023-05-31T20:55:07.158 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T20:55:07.158 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2023-05-31T20:55:07.158 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "", 2023-05-31T20:55:07.158 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T20:55:07.158 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2023-05-31T20:55:07.158 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T20:55:07.158 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "0.000000", 2023-05-31T20:55:07.159 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2023-05-31T20:55:07.159 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T20:55:07.159 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:55:07.159 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T20:55:07.159 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 2, 2023-05-31T20:55:07.159 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T20:55:07.159 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2023-05-31T20:55:07.160 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1685562970.274572_93.1", 2023-05-31T20:55:07.160 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T20:55:07.160 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2023-05-31T20:55:07.160 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T20:55:07.160 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2023-05-31T19:56:10.274572Z", 2023-05-31T20:55:07.160 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2023-05-31T20:55:07.160 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T20:55:07.161 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:55:07.161 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T20:55:07.161 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 3, 2023-05-31T20:55:07.161 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T20:55:07.161 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2023-05-31T20:55:07.161 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1685566334.798037_482.1", 2023-05-31T20:55:07.161 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T20:55:07.162 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2023-05-31T20:55:07.162 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T20:55:07.162 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2023-05-31T20:52:14.798037Z", 2023-05-31T20:55:07.162 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2023-05-31T20:55:07.162 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T20:55:07.162 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T20:55:07.162 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2023-05-31T20:55:07.163 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:55:07.163 INFO:teuthology.orchestra.run.smithi099.stdout: "full_sync": { 2023-05-31T20:55:07.163 INFO:teuthology.orchestra.run.smithi099.stdout: "total": 1, 2023-05-31T20:55:07.163 INFO:teuthology.orchestra.run.smithi099.stdout: "complete": 1 2023-05-31T20:55:07.163 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:55:07.163 INFO:teuthology.orchestra.run.smithi099.stdout: "current_time": "2023-05-31T20:55:07Z" 2023-05-31T20:55:07.163 INFO:teuthology.orchestra.run.smithi099.stdout:} 2023-05-31T20:55:07.164 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "2e6123d2-d695-4350-9a01-61a455bc782b", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1685562631.516089_59.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-05-31T19:50:31.516089Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1685562970.274572_93.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-05-31T19:56:10.274572Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1685566334.798037_482.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-05-31T20:52:14.798037Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-05-31T20:55:07Z" } 2023-05-31T20:55:07.165 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1685562631.516089_59.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-05-31T19:50:31.516089Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1685562970.274572_93.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-05-31T19:56:10.274572Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1685566334.798037_482.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-05-31T20:52:14.798037Z', 'realm_epoch': 2}}] 2023-05-31T20:55:07.165 DEBUG:rgw_multi.tests:log_status={0: '1_1685566501.757183_260.1', 1: '1_1685566501.735717_251.1', 2: '1_1685562970.274572_93.1', 3: '1_1685566334.798037_482.1'} 2023-05-31T20:55:07.165 DEBUG:rgw_multi.tests:sync_status={0: '1_1685562631.516089_59.1', 1: '', 2: '1_1685562970.274572_93.1', 3: '1_1685566334.798037_482.1'} 2023-05-31T20:55:07.166 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1685566501.757183_260.1 target=1_1685562631.516089_59.1, shard=1 master=1_1685566501.735717_251.1 target= 2023-05-31T20:55:12.167 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-31T20:55:12.168 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-31T20:55:12.168 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-05-31T20:55:12.222 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2023-05-31T20:55:12.222 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2023-05-31T20:55:12.269 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:55:12.265+0000 7f0baf205780 20 RGW-SYNC:meta: read sync status 2023-05-31T20:55:12.270 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:55:12.266+0000 7f0baf205780 20 RGW-SYNC:meta: read sync status 2023-05-31T20:55:12.284 INFO:teuthology.orchestra.run.smithi099.stdout:{ 2023-05-31T20:55:12.284 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_status": { 2023-05-31T20:55:12.284 INFO:teuthology.orchestra.run.smithi099.stdout: "info": { 2023-05-31T20:55:12.284 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "sync", 2023-05-31T20:55:12.285 INFO:teuthology.orchestra.run.smithi099.stdout: "num_shards": 4, 2023-05-31T20:55:12.285 INFO:teuthology.orchestra.run.smithi099.stdout: "period": "2e6123d2-d695-4350-9a01-61a455bc782b", 2023-05-31T20:55:12.285 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2023-05-31T20:55:12.285 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:55:12.285 INFO:teuthology.orchestra.run.smithi099.stdout: "markers": [ 2023-05-31T20:55:12.285 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T20:55:12.286 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 0, 2023-05-31T20:55:12.286 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T20:55:12.286 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2023-05-31T20:55:12.286 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1685562631.516089_59.1", 2023-05-31T20:55:12.286 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T20:55:12.286 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 1, 2023-05-31T20:55:12.286 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T20:55:12.287 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2023-05-31T19:50:31.516089Z", 2023-05-31T20:55:12.287 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2023-05-31T20:55:12.287 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T20:55:12.287 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:55:12.287 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T20:55:12.287 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 1, 2023-05-31T20:55:12.287 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T20:55:12.288 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2023-05-31T20:55:12.288 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "", 2023-05-31T20:55:12.288 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T20:55:12.288 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2023-05-31T20:55:12.288 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T20:55:12.288 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "0.000000", 2023-05-31T20:55:12.288 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2023-05-31T20:55:12.289 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T20:55:12.289 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:55:12.289 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T20:55:12.289 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 2, 2023-05-31T20:55:12.289 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T20:55:12.289 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2023-05-31T20:55:12.289 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1685562970.274572_93.1", 2023-05-31T20:55:12.290 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T20:55:12.290 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2023-05-31T20:55:12.290 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T20:55:12.290 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2023-05-31T19:56:10.274572Z", 2023-05-31T20:55:12.290 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2023-05-31T20:55:12.290 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T20:55:12.290 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:55:12.291 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T20:55:12.291 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 3, 2023-05-31T20:55:12.291 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T20:55:12.291 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2023-05-31T20:55:12.291 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1685566334.798037_482.1", 2023-05-31T20:55:12.291 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T20:55:12.291 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2023-05-31T20:55:12.292 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T20:55:12.292 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2023-05-31T20:52:14.798037Z", 2023-05-31T20:55:12.292 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2023-05-31T20:55:12.292 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T20:55:12.292 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T20:55:12.292 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2023-05-31T20:55:12.292 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:55:12.293 INFO:teuthology.orchestra.run.smithi099.stdout: "full_sync": { 2023-05-31T20:55:12.293 INFO:teuthology.orchestra.run.smithi099.stdout: "total": 1, 2023-05-31T20:55:12.293 INFO:teuthology.orchestra.run.smithi099.stdout: "complete": 1 2023-05-31T20:55:12.293 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:55:12.293 INFO:teuthology.orchestra.run.smithi099.stdout: "current_time": "2023-05-31T20:55:12Z" 2023-05-31T20:55:12.293 INFO:teuthology.orchestra.run.smithi099.stdout:} 2023-05-31T20:55:12.294 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "2e6123d2-d695-4350-9a01-61a455bc782b", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1685562631.516089_59.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-05-31T19:50:31.516089Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1685562970.274572_93.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-05-31T19:56:10.274572Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1685566334.798037_482.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-05-31T20:52:14.798037Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-05-31T20:55:12Z" } 2023-05-31T20:55:12.295 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1685562631.516089_59.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-05-31T19:50:31.516089Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1685562970.274572_93.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-05-31T19:56:10.274572Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1685566334.798037_482.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-05-31T20:52:14.798037Z', 'realm_epoch': 2}}] 2023-05-31T20:55:12.295 DEBUG:rgw_multi.tests:log_status={0: '1_1685566501.757183_260.1', 1: '1_1685566501.735717_251.1', 2: '1_1685562970.274572_93.1', 3: '1_1685566334.798037_482.1'} 2023-05-31T20:55:12.295 DEBUG:rgw_multi.tests:sync_status={0: '1_1685562631.516089_59.1', 1: '', 2: '1_1685562970.274572_93.1', 3: '1_1685566334.798037_482.1'} 2023-05-31T20:55:12.296 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1685566501.757183_260.1 target=1_1685562631.516089_59.1, shard=1 master=1_1685566501.735717_251.1 target= 2023-05-31T20:55:17.296 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-31T20:55:17.297 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-31T20:55:17.297 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-05-31T20:55:17.354 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2023-05-31T20:55:17.354 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2023-05-31T20:55:17.402 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:55:17.398+0000 7f1646b54780 20 RGW-SYNC:meta: read sync status 2023-05-31T20:55:17.403 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:55:17.399+0000 7f1646b54780 20 RGW-SYNC:meta: read sync status 2023-05-31T20:55:17.424 INFO:teuthology.orchestra.run.smithi099.stdout:{ 2023-05-31T20:55:17.424 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_status": { 2023-05-31T20:55:17.424 INFO:teuthology.orchestra.run.smithi099.stdout: "info": { 2023-05-31T20:55:17.424 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "sync", 2023-05-31T20:55:17.425 INFO:teuthology.orchestra.run.smithi099.stdout: "num_shards": 4, 2023-05-31T20:55:17.425 INFO:teuthology.orchestra.run.smithi099.stdout: "period": "2e6123d2-d695-4350-9a01-61a455bc782b", 2023-05-31T20:55:17.425 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2023-05-31T20:55:17.425 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:55:17.425 INFO:teuthology.orchestra.run.smithi099.stdout: "markers": [ 2023-05-31T20:55:17.425 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T20:55:17.425 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 0, 2023-05-31T20:55:17.426 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T20:55:17.426 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2023-05-31T20:55:17.426 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1685562631.516089_59.1", 2023-05-31T20:55:17.426 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T20:55:17.426 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 1, 2023-05-31T20:55:17.426 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T20:55:17.426 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2023-05-31T19:50:31.516089Z", 2023-05-31T20:55:17.427 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2023-05-31T20:55:17.427 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T20:55:17.427 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:55:17.427 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T20:55:17.427 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 1, 2023-05-31T20:55:17.427 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T20:55:17.427 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2023-05-31T20:55:17.428 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "", 2023-05-31T20:55:17.428 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T20:55:17.428 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2023-05-31T20:55:17.428 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T20:55:17.428 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "0.000000", 2023-05-31T20:55:17.428 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2023-05-31T20:55:17.428 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T20:55:17.429 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:55:17.429 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T20:55:17.429 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 2, 2023-05-31T20:55:17.429 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T20:55:17.429 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2023-05-31T20:55:17.429 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1685562970.274572_93.1", 2023-05-31T20:55:17.429 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T20:55:17.430 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2023-05-31T20:55:17.430 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T20:55:17.430 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2023-05-31T19:56:10.274572Z", 2023-05-31T20:55:17.430 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2023-05-31T20:55:17.430 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T20:55:17.430 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:55:17.430 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T20:55:17.431 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 3, 2023-05-31T20:55:17.431 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T20:55:17.431 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2023-05-31T20:55:17.431 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1685566334.798037_482.1", 2023-05-31T20:55:17.431 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T20:55:17.431 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2023-05-31T20:55:17.431 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T20:55:17.432 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2023-05-31T20:52:14.798037Z", 2023-05-31T20:55:17.432 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2023-05-31T20:55:17.432 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T20:55:17.432 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T20:55:17.432 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2023-05-31T20:55:17.432 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:55:17.432 INFO:teuthology.orchestra.run.smithi099.stdout: "full_sync": { 2023-05-31T20:55:17.433 INFO:teuthology.orchestra.run.smithi099.stdout: "total": 1, 2023-05-31T20:55:17.433 INFO:teuthology.orchestra.run.smithi099.stdout: "complete": 1 2023-05-31T20:55:17.433 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:55:17.433 INFO:teuthology.orchestra.run.smithi099.stdout: "current_time": "2023-05-31T20:55:17Z" 2023-05-31T20:55:17.433 INFO:teuthology.orchestra.run.smithi099.stdout:} 2023-05-31T20:55:17.434 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "2e6123d2-d695-4350-9a01-61a455bc782b", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1685562631.516089_59.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-05-31T19:50:31.516089Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1685562970.274572_93.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-05-31T19:56:10.274572Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1685566334.798037_482.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-05-31T20:52:14.798037Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-05-31T20:55:17Z" } 2023-05-31T20:55:17.434 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1685562631.516089_59.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-05-31T19:50:31.516089Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1685562970.274572_93.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-05-31T19:56:10.274572Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1685566334.798037_482.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-05-31T20:52:14.798037Z', 'realm_epoch': 2}}] 2023-05-31T20:55:17.435 DEBUG:rgw_multi.tests:log_status={0: '1_1685566501.757183_260.1', 1: '1_1685566501.735717_251.1', 2: '1_1685562970.274572_93.1', 3: '1_1685566334.798037_482.1'} 2023-05-31T20:55:17.435 DEBUG:rgw_multi.tests:sync_status={0: '1_1685562631.516089_59.1', 1: '', 2: '1_1685562970.274572_93.1', 3: '1_1685566334.798037_482.1'} 2023-05-31T20:55:17.435 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1685566501.757183_260.1 target=1_1685562631.516089_59.1, shard=1 master=1_1685566501.735717_251.1 target= 2023-05-31T20:55:22.436 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-31T20:55:22.437 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-31T20:55:22.437 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-05-31T20:55:22.494 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2023-05-31T20:55:22.495 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2023-05-31T20:55:22.535 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:55:22.531+0000 7f7291768780 20 RGW-SYNC:meta: read sync status 2023-05-31T20:55:22.536 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:55:22.532+0000 7f7291768780 20 RGW-SYNC:meta: read sync status 2023-05-31T20:55:22.555 INFO:teuthology.orchestra.run.smithi099.stdout:{ 2023-05-31T20:55:22.555 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_status": { 2023-05-31T20:55:22.556 INFO:teuthology.orchestra.run.smithi099.stdout: "info": { 2023-05-31T20:55:22.556 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "sync", 2023-05-31T20:55:22.556 INFO:teuthology.orchestra.run.smithi099.stdout: "num_shards": 4, 2023-05-31T20:55:22.556 INFO:teuthology.orchestra.run.smithi099.stdout: "period": "2e6123d2-d695-4350-9a01-61a455bc782b", 2023-05-31T20:55:22.556 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2023-05-31T20:55:22.556 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:55:22.557 INFO:teuthology.orchestra.run.smithi099.stdout: "markers": [ 2023-05-31T20:55:22.557 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T20:55:22.557 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 0, 2023-05-31T20:55:22.557 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T20:55:22.557 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2023-05-31T20:55:22.557 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1685562631.516089_59.1", 2023-05-31T20:55:22.558 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T20:55:22.558 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 1, 2023-05-31T20:55:22.558 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T20:55:22.558 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2023-05-31T19:50:31.516089Z", 2023-05-31T20:55:22.558 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2023-05-31T20:55:22.558 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T20:55:22.558 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:55:22.559 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T20:55:22.559 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 1, 2023-05-31T20:55:22.559 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T20:55:22.559 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2023-05-31T20:55:22.559 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "", 2023-05-31T20:55:22.559 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T20:55:22.559 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2023-05-31T20:55:22.560 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T20:55:22.560 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "0.000000", 2023-05-31T20:55:22.560 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2023-05-31T20:55:22.560 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T20:55:22.560 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:55:22.560 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T20:55:22.560 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 2, 2023-05-31T20:55:22.561 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T20:55:22.561 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2023-05-31T20:55:22.561 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1685562970.274572_93.1", 2023-05-31T20:55:22.561 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T20:55:22.561 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2023-05-31T20:55:22.561 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T20:55:22.561 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2023-05-31T19:56:10.274572Z", 2023-05-31T20:55:22.562 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2023-05-31T20:55:22.562 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T20:55:22.562 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:55:22.562 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T20:55:22.562 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 3, 2023-05-31T20:55:22.562 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T20:55:22.563 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2023-05-31T20:55:22.563 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1685566334.798037_482.1", 2023-05-31T20:55:22.563 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T20:55:22.563 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2023-05-31T20:55:22.563 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T20:55:22.563 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2023-05-31T20:52:14.798037Z", 2023-05-31T20:55:22.563 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2023-05-31T20:55:22.563 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T20:55:22.564 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T20:55:22.564 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2023-05-31T20:55:22.564 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:55:22.564 INFO:teuthology.orchestra.run.smithi099.stdout: "full_sync": { 2023-05-31T20:55:22.564 INFO:teuthology.orchestra.run.smithi099.stdout: "total": 1, 2023-05-31T20:55:22.564 INFO:teuthology.orchestra.run.smithi099.stdout: "complete": 1 2023-05-31T20:55:22.564 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:55:22.565 INFO:teuthology.orchestra.run.smithi099.stdout: "current_time": "2023-05-31T20:55:22Z" 2023-05-31T20:55:22.565 INFO:teuthology.orchestra.run.smithi099.stdout:} 2023-05-31T20:55:22.566 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "2e6123d2-d695-4350-9a01-61a455bc782b", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1685562631.516089_59.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-05-31T19:50:31.516089Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1685562970.274572_93.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-05-31T19:56:10.274572Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1685566334.798037_482.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-05-31T20:52:14.798037Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-05-31T20:55:22Z" } 2023-05-31T20:55:22.566 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1685562631.516089_59.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-05-31T19:50:31.516089Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1685562970.274572_93.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-05-31T19:56:10.274572Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1685566334.798037_482.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-05-31T20:52:14.798037Z', 'realm_epoch': 2}}] 2023-05-31T20:55:22.567 DEBUG:rgw_multi.tests:log_status={0: '1_1685566501.757183_260.1', 1: '1_1685566501.735717_251.1', 2: '1_1685562970.274572_93.1', 3: '1_1685566334.798037_482.1'} 2023-05-31T20:55:22.567 DEBUG:rgw_multi.tests:sync_status={0: '1_1685562631.516089_59.1', 1: '', 2: '1_1685562970.274572_93.1', 3: '1_1685566334.798037_482.1'} 2023-05-31T20:55:22.567 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1685566501.757183_260.1 target=1_1685562631.516089_59.1, shard=1 master=1_1685566501.735717_251.1 target= 2023-05-31T20:55:27.568 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-31T20:55:27.569 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-31T20:55:27.569 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-05-31T20:55:27.626 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2023-05-31T20:55:27.626 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2023-05-31T20:55:27.669 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:55:27.664+0000 7fdb5f9e8780 20 RGW-SYNC:meta: read sync status 2023-05-31T20:55:27.670 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:55:27.665+0000 7fdb5f9e8780 20 RGW-SYNC:meta: read sync status 2023-05-31T20:55:27.686 INFO:teuthology.orchestra.run.smithi099.stdout:{ 2023-05-31T20:55:27.686 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_status": { 2023-05-31T20:55:27.687 INFO:teuthology.orchestra.run.smithi099.stdout: "info": { 2023-05-31T20:55:27.687 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "sync", 2023-05-31T20:55:27.687 INFO:teuthology.orchestra.run.smithi099.stdout: "num_shards": 4, 2023-05-31T20:55:27.687 INFO:teuthology.orchestra.run.smithi099.stdout: "period": "2e6123d2-d695-4350-9a01-61a455bc782b", 2023-05-31T20:55:27.687 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2023-05-31T20:55:27.687 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:55:27.688 INFO:teuthology.orchestra.run.smithi099.stdout: "markers": [ 2023-05-31T20:55:27.688 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T20:55:27.688 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 0, 2023-05-31T20:55:27.688 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T20:55:27.688 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2023-05-31T20:55:27.688 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1685562631.516089_59.1", 2023-05-31T20:55:27.688 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T20:55:27.689 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 1, 2023-05-31T20:55:27.689 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T20:55:27.689 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2023-05-31T19:50:31.516089Z", 2023-05-31T20:55:27.689 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2023-05-31T20:55:27.689 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T20:55:27.689 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:55:27.690 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T20:55:27.690 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 1, 2023-05-31T20:55:27.690 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T20:55:27.690 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2023-05-31T20:55:27.690 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "", 2023-05-31T20:55:27.690 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T20:55:27.691 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2023-05-31T20:55:27.691 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T20:55:27.691 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "0.000000", 2023-05-31T20:55:27.691 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2023-05-31T20:55:27.691 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T20:55:27.691 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:55:27.691 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T20:55:27.692 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 2, 2023-05-31T20:55:27.692 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T20:55:27.692 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2023-05-31T20:55:27.692 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1685562970.274572_93.1", 2023-05-31T20:55:27.692 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T20:55:27.692 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2023-05-31T20:55:27.692 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T20:55:27.693 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2023-05-31T19:56:10.274572Z", 2023-05-31T20:55:27.693 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2023-05-31T20:55:27.693 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T20:55:27.693 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:55:27.693 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T20:55:27.693 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 3, 2023-05-31T20:55:27.693 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T20:55:27.694 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2023-05-31T20:55:27.694 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1685566334.798037_482.1", 2023-05-31T20:55:27.694 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T20:55:27.694 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2023-05-31T20:55:27.694 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T20:55:27.694 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2023-05-31T20:52:14.798037Z", 2023-05-31T20:55:27.695 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2023-05-31T20:55:27.695 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T20:55:27.695 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T20:55:27.695 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2023-05-31T20:55:27.695 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:55:27.695 INFO:teuthology.orchestra.run.smithi099.stdout: "full_sync": { 2023-05-31T20:55:27.695 INFO:teuthology.orchestra.run.smithi099.stdout: "total": 1, 2023-05-31T20:55:27.696 INFO:teuthology.orchestra.run.smithi099.stdout: "complete": 1 2023-05-31T20:55:27.696 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:55:27.696 INFO:teuthology.orchestra.run.smithi099.stdout: "current_time": "2023-05-31T20:55:27Z" 2023-05-31T20:55:27.696 INFO:teuthology.orchestra.run.smithi099.stdout:} 2023-05-31T20:55:27.697 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "2e6123d2-d695-4350-9a01-61a455bc782b", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1685562631.516089_59.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-05-31T19:50:31.516089Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1685562970.274572_93.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-05-31T19:56:10.274572Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1685566334.798037_482.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-05-31T20:52:14.798037Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-05-31T20:55:27Z" } 2023-05-31T20:55:27.697 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1685562631.516089_59.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-05-31T19:50:31.516089Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1685562970.274572_93.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-05-31T19:56:10.274572Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1685566334.798037_482.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-05-31T20:52:14.798037Z', 'realm_epoch': 2}}] 2023-05-31T20:55:27.698 DEBUG:rgw_multi.tests:log_status={0: '1_1685566501.757183_260.1', 1: '1_1685566501.735717_251.1', 2: '1_1685562970.274572_93.1', 3: '1_1685566334.798037_482.1'} 2023-05-31T20:55:27.698 DEBUG:rgw_multi.tests:sync_status={0: '1_1685562631.516089_59.1', 1: '', 2: '1_1685562970.274572_93.1', 3: '1_1685566334.798037_482.1'} 2023-05-31T20:55:27.698 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1685566501.757183_260.1 target=1_1685562631.516089_59.1, shard=1 master=1_1685566501.735717_251.1 target= 2023-05-31T20:55:32.699 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-31T20:55:32.700 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-31T20:55:32.700 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-05-31T20:55:32.757 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2023-05-31T20:55:32.757 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2023-05-31T20:55:32.802 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:55:32.797+0000 7fba66eab780 20 RGW-SYNC:meta: read sync status 2023-05-31T20:55:32.803 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:55:32.798+0000 7fba66eab780 20 RGW-SYNC:meta: read sync status 2023-05-31T20:55:32.826 INFO:teuthology.orchestra.run.smithi099.stdout:{ 2023-05-31T20:55:32.826 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_status": { 2023-05-31T20:55:32.826 INFO:teuthology.orchestra.run.smithi099.stdout: "info": { 2023-05-31T20:55:32.826 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "sync", 2023-05-31T20:55:32.827 INFO:teuthology.orchestra.run.smithi099.stdout: "num_shards": 4, 2023-05-31T20:55:32.827 INFO:teuthology.orchestra.run.smithi099.stdout: "period": "2e6123d2-d695-4350-9a01-61a455bc782b", 2023-05-31T20:55:32.827 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2023-05-31T20:55:32.827 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:55:32.827 INFO:teuthology.orchestra.run.smithi099.stdout: "markers": [ 2023-05-31T20:55:32.828 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T20:55:32.828 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 0, 2023-05-31T20:55:32.828 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T20:55:32.828 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2023-05-31T20:55:32.828 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1685562631.516089_59.1", 2023-05-31T20:55:32.828 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T20:55:32.829 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 1, 2023-05-31T20:55:32.829 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T20:55:32.829 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2023-05-31T19:50:31.516089Z", 2023-05-31T20:55:32.829 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2023-05-31T20:55:32.829 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T20:55:32.829 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:55:32.829 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T20:55:32.830 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 1, 2023-05-31T20:55:32.830 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T20:55:32.830 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2023-05-31T20:55:32.830 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "", 2023-05-31T20:55:32.830 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T20:55:32.830 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2023-05-31T20:55:32.830 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T20:55:32.831 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "0.000000", 2023-05-31T20:55:32.831 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2023-05-31T20:55:32.831 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T20:55:32.831 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:55:32.831 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T20:55:32.831 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 2, 2023-05-31T20:55:32.832 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T20:55:32.832 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2023-05-31T20:55:32.832 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1685562970.274572_93.1", 2023-05-31T20:55:32.833 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T20:55:32.833 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2023-05-31T20:55:32.833 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T20:55:32.834 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2023-05-31T19:56:10.274572Z", 2023-05-31T20:55:32.834 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2023-05-31T20:55:32.834 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T20:55:32.834 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:55:32.835 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T20:55:32.835 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 3, 2023-05-31T20:55:32.835 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T20:55:32.835 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2023-05-31T20:55:32.835 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1685566334.798037_482.1", 2023-05-31T20:55:32.836 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T20:55:32.836 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2023-05-31T20:55:32.836 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T20:55:32.836 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2023-05-31T20:52:14.798037Z", 2023-05-31T20:55:32.836 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2023-05-31T20:55:32.837 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T20:55:32.837 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T20:55:32.838 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2023-05-31T20:55:32.839 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:55:32.839 INFO:teuthology.orchestra.run.smithi099.stdout: "full_sync": { 2023-05-31T20:55:32.839 INFO:teuthology.orchestra.run.smithi099.stdout: "total": 1, 2023-05-31T20:55:32.839 INFO:teuthology.orchestra.run.smithi099.stdout: "complete": 1 2023-05-31T20:55:32.839 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:55:32.839 INFO:teuthology.orchestra.run.smithi099.stdout: "current_time": "2023-05-31T20:55:32Z" 2023-05-31T20:55:32.840 INFO:teuthology.orchestra.run.smithi099.stdout:} 2023-05-31T20:55:32.841 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "2e6123d2-d695-4350-9a01-61a455bc782b", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1685562631.516089_59.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-05-31T19:50:31.516089Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1685562970.274572_93.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-05-31T19:56:10.274572Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1685566334.798037_482.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-05-31T20:52:14.798037Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-05-31T20:55:32Z" } 2023-05-31T20:55:32.841 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1685562631.516089_59.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-05-31T19:50:31.516089Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1685562970.274572_93.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-05-31T19:56:10.274572Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1685566334.798037_482.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-05-31T20:52:14.798037Z', 'realm_epoch': 2}}] 2023-05-31T20:55:32.842 DEBUG:rgw_multi.tests:log_status={0: '1_1685566501.757183_260.1', 1: '1_1685566501.735717_251.1', 2: '1_1685562970.274572_93.1', 3: '1_1685566334.798037_482.1'} 2023-05-31T20:55:32.843 DEBUG:rgw_multi.tests:sync_status={0: '1_1685562631.516089_59.1', 1: '', 2: '1_1685562970.274572_93.1', 3: '1_1685566334.798037_482.1'} 2023-05-31T20:55:32.843 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1685566501.757183_260.1 target=1_1685562631.516089_59.1, shard=1 master=1_1685566501.735717_251.1 target= 2023-05-31T20:55:37.844 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-31T20:55:37.845 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-31T20:55:37.845 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-05-31T20:55:37.904 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2023-05-31T20:55:37.904 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2023-05-31T20:55:37.938 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:55:37.934+0000 7febaa51f780 20 RGW-SYNC:meta: read sync status 2023-05-31T20:55:37.939 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:55:37.935+0000 7febaa51f780 20 RGW-SYNC:meta: read sync status 2023-05-31T20:55:37.957 INFO:teuthology.orchestra.run.smithi099.stdout:{ 2023-05-31T20:55:37.957 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_status": { 2023-05-31T20:55:37.957 INFO:teuthology.orchestra.run.smithi099.stdout: "info": { 2023-05-31T20:55:37.957 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "sync", 2023-05-31T20:55:37.957 INFO:teuthology.orchestra.run.smithi099.stdout: "num_shards": 4, 2023-05-31T20:55:37.957 INFO:teuthology.orchestra.run.smithi099.stdout: "period": "2e6123d2-d695-4350-9a01-61a455bc782b", 2023-05-31T20:55:37.958 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2023-05-31T20:55:37.958 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:55:37.958 INFO:teuthology.orchestra.run.smithi099.stdout: "markers": [ 2023-05-31T20:55:37.958 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T20:55:37.958 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 0, 2023-05-31T20:55:37.958 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T20:55:37.959 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2023-05-31T20:55:37.959 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1685562631.516089_59.1", 2023-05-31T20:55:37.959 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T20:55:37.959 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 1, 2023-05-31T20:55:37.959 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T20:55:37.959 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2023-05-31T19:50:31.516089Z", 2023-05-31T20:55:37.960 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2023-05-31T20:55:37.960 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T20:55:37.960 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:55:37.960 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T20:55:37.960 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 1, 2023-05-31T20:55:37.960 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T20:55:37.961 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2023-05-31T20:55:37.961 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "", 2023-05-31T20:55:37.961 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T20:55:37.961 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2023-05-31T20:55:37.961 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T20:55:37.961 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "0.000000", 2023-05-31T20:55:37.961 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2023-05-31T20:55:37.962 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T20:55:37.962 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:55:37.962 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T20:55:37.962 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 2, 2023-05-31T20:55:37.962 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T20:55:37.962 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2023-05-31T20:55:37.963 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1685562970.274572_93.1", 2023-05-31T20:55:37.963 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T20:55:37.963 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2023-05-31T20:55:37.963 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T20:55:37.963 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2023-05-31T19:56:10.274572Z", 2023-05-31T20:55:37.963 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2023-05-31T20:55:37.963 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T20:55:37.964 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:55:37.964 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T20:55:37.964 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 3, 2023-05-31T20:55:37.964 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T20:55:37.964 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2023-05-31T20:55:37.964 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1685566334.798037_482.1", 2023-05-31T20:55:37.965 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T20:55:37.965 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2023-05-31T20:55:37.965 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T20:55:37.965 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2023-05-31T20:52:14.798037Z", 2023-05-31T20:55:37.965 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2023-05-31T20:55:37.965 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T20:55:37.965 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T20:55:37.966 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2023-05-31T20:55:37.966 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:55:37.966 INFO:teuthology.orchestra.run.smithi099.stdout: "full_sync": { 2023-05-31T20:55:37.966 INFO:teuthology.orchestra.run.smithi099.stdout: "total": 1, 2023-05-31T20:55:37.966 INFO:teuthology.orchestra.run.smithi099.stdout: "complete": 1 2023-05-31T20:55:37.966 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:55:37.967 INFO:teuthology.orchestra.run.smithi099.stdout: "current_time": "2023-05-31T20:55:37Z" 2023-05-31T20:55:37.967 INFO:teuthology.orchestra.run.smithi099.stdout:} 2023-05-31T20:55:37.967 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "2e6123d2-d695-4350-9a01-61a455bc782b", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1685562631.516089_59.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-05-31T19:50:31.516089Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1685562970.274572_93.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-05-31T19:56:10.274572Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1685566334.798037_482.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-05-31T20:52:14.798037Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-05-31T20:55:37Z" } 2023-05-31T20:55:37.968 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1685562631.516089_59.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-05-31T19:50:31.516089Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1685562970.274572_93.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-05-31T19:56:10.274572Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1685566334.798037_482.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-05-31T20:52:14.798037Z', 'realm_epoch': 2}}] 2023-05-31T20:55:37.968 DEBUG:rgw_multi.tests:log_status={0: '1_1685566501.757183_260.1', 1: '1_1685566501.735717_251.1', 2: '1_1685562970.274572_93.1', 3: '1_1685566334.798037_482.1'} 2023-05-31T20:55:37.969 DEBUG:rgw_multi.tests:sync_status={0: '1_1685562631.516089_59.1', 1: '', 2: '1_1685562970.274572_93.1', 3: '1_1685566334.798037_482.1'} 2023-05-31T20:55:37.969 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1685566501.757183_260.1 target=1_1685562631.516089_59.1, shard=1 master=1_1685566501.735717_251.1 target= 2023-05-31T20:55:42.970 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-31T20:55:42.971 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-31T20:55:42.971 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-05-31T20:55:43.028 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2023-05-31T20:55:43.028 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2023-05-31T20:55:43.079 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:55:43.074+0000 7ffaafd38780 20 RGW-SYNC:meta: read sync status 2023-05-31T20:55:43.080 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:55:43.076+0000 7ffaafd38780 20 RGW-SYNC:meta: read sync status 2023-05-31T20:55:43.100 INFO:teuthology.orchestra.run.smithi099.stdout:{ 2023-05-31T20:55:43.100 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_status": { 2023-05-31T20:55:43.100 INFO:teuthology.orchestra.run.smithi099.stdout: "info": { 2023-05-31T20:55:43.100 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "sync", 2023-05-31T20:55:43.101 INFO:teuthology.orchestra.run.smithi099.stdout: "num_shards": 4, 2023-05-31T20:55:43.101 INFO:teuthology.orchestra.run.smithi099.stdout: "period": "2e6123d2-d695-4350-9a01-61a455bc782b", 2023-05-31T20:55:43.101 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2023-05-31T20:55:43.101 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:55:43.101 INFO:teuthology.orchestra.run.smithi099.stdout: "markers": [ 2023-05-31T20:55:43.101 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T20:55:43.102 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 0, 2023-05-31T20:55:43.102 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T20:55:43.102 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2023-05-31T20:55:43.102 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1685566501.757183_260.1", 2023-05-31T20:55:43.102 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T20:55:43.102 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 1, 2023-05-31T20:55:43.102 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T20:55:43.103 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2023-05-31T20:55:01.757183Z", 2023-05-31T20:55:43.103 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2023-05-31T20:55:43.103 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T20:55:43.103 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:55:43.103 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T20:55:43.103 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 1, 2023-05-31T20:55:43.103 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T20:55:43.104 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2023-05-31T20:55:43.104 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1685566501.735717_251.1", 2023-05-31T20:55:43.104 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T20:55:43.104 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2023-05-31T20:55:43.104 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T20:55:43.104 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2023-05-31T20:55:01.735717Z", 2023-05-31T20:55:43.104 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2023-05-31T20:55:43.105 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T20:55:43.105 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:55:43.105 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T20:55:43.105 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 2, 2023-05-31T20:55:43.105 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T20:55:43.105 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2023-05-31T20:55:43.105 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1685562970.274572_93.1", 2023-05-31T20:55:43.106 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T20:55:43.106 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2023-05-31T20:55:43.106 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T20:55:43.106 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2023-05-31T19:56:10.274572Z", 2023-05-31T20:55:43.106 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2023-05-31T20:55:43.106 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T20:55:43.106 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:55:43.107 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T20:55:43.107 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 3, 2023-05-31T20:55:43.107 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T20:55:43.107 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2023-05-31T20:55:43.107 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1685566334.798037_482.1", 2023-05-31T20:55:43.107 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T20:55:43.107 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2023-05-31T20:55:43.108 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T20:55:43.108 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2023-05-31T20:52:14.798037Z", 2023-05-31T20:55:43.108 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2023-05-31T20:55:43.108 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T20:55:43.108 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T20:55:43.108 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2023-05-31T20:55:43.108 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:55:43.109 INFO:teuthology.orchestra.run.smithi099.stdout: "full_sync": { 2023-05-31T20:55:43.109 INFO:teuthology.orchestra.run.smithi099.stdout: "total": 1, 2023-05-31T20:55:43.109 INFO:teuthology.orchestra.run.smithi099.stdout: "complete": 1 2023-05-31T20:55:43.109 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:55:43.109 INFO:teuthology.orchestra.run.smithi099.stdout: "current_time": "2023-05-31T20:55:43Z" 2023-05-31T20:55:43.109 INFO:teuthology.orchestra.run.smithi099.stdout:} 2023-05-31T20:55:43.110 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "2e6123d2-d695-4350-9a01-61a455bc782b", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1685566501.757183_260.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-05-31T20:55:01.757183Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1685566501.735717_251.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-05-31T20:55:01.735717Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1685562970.274572_93.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-05-31T19:56:10.274572Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1685566334.798037_482.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-05-31T20:52:14.798037Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-05-31T20:55:43Z" } 2023-05-31T20:55:43.111 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1685566501.757183_260.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-05-31T20:55:01.757183Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1685566501.735717_251.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-05-31T20:55:01.735717Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1685562970.274572_93.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-05-31T19:56:10.274572Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1685566334.798037_482.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-05-31T20:52:14.798037Z', 'realm_epoch': 2}}] 2023-05-31T20:55:43.111 DEBUG:rgw_multi.tests:log_status={0: '1_1685566501.757183_260.1', 1: '1_1685566501.735717_251.1', 2: '1_1685562970.274572_93.1', 3: '1_1685566334.798037_482.1'} 2023-05-31T20:55:43.111 DEBUG:rgw_multi.tests:sync_status={0: '1_1685566501.757183_260.1', 1: '1_1685566501.735717_251.1', 2: '1_1685562970.274572_93.1', 3: '1_1685566334.798037_482.1'} 2023-05-31T20:55:43.112 INFO:rgw_multi.tests:finish meta checkpoint for zone=a2 2023-05-31T20:55:43.112 INFO:tasks.rgw_multi.zone_rados:comparing role=roles-pklxkd-1 zones={a2, a1} 2023-05-31T20:55:43.112 DEBUG:boto:Method: POST 2023-05-31T20:55:43.113 DEBUG:boto:Path: / 2023-05-31T20:55:43.113 DEBUG:boto:Data: 2023-05-31T20:55:43.113 DEBUG:boto:Headers: {} 2023-05-31T20:55:43.114 DEBUG:boto:Host: smithi099.front.sepia.ceph.com 2023-05-31T20:55:43.114 DEBUG:boto:Port: 8001 2023-05-31T20:55:43.114 DEBUG:boto:Params: {'RoleName': 'roles-pklxkd-1', 'Action': 'GetRole', 'Version': '2010-05-08'} 2023-05-31T20:55:43.115 DEBUG:boto:Token: None 2023-05-31T20:55:43.115 DEBUG:boto:CanonicalRequest: POST / host:smithi099.front.sepia.ceph.com:8001 x-amz-date:20230531T205543Z host;x-amz-date ca2e30fd9554699b1bb2bbf0af2c0d841e5b87dd54b6db71af5f1fb9e82eb5bb 2023-05-31T20:55:43.116 DEBUG:boto:StringToSign: AWS4-HMAC-SHA256 20230531T205543Z 20230531/front/smithi099/aws4_request 42a5f7d912a1702d9488fba74089659b921118549b57afdde62f5a2648631575 2023-05-31T20:55:43.116 DEBUG:boto:Signature: 4d84752bf44aa893eae14e829b307671e936d1aa259b234550cfb735dc131b0d 2023-05-31T20:55:43.117 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'X-Amz-Date': '20230531T205543Z', 'Content-Type': 'application/x-www-form-urlencoded; charset=UTF-8', 'Content-Length': '57', 'Authorization': 'AWS4-HMAC-SHA256 Credential=Q80H11OLNU1S48PPYL26/20230531/front/smithi099/aws4_request,SignedHeaders=host;x-amz-date,Signature=4d84752bf44aa893eae14e829b307671e936d1aa259b234550cfb735dc131b0d', 'Host': 'smithi099.front.sepia.ceph.com:8001'} 2023-05-31T20:55:43.119 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000df4b5eaa9d3e53f6-006477b44f-1e2e-a2'), ('Content-Type', 'application/xml'), ('Content-Length', '711'), ('Date', 'Wed, 31 May 2023 20:55:43 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:55:43.119 DEBUG:boto:b'tx00000df4b5eaa9d3e53f6-006477b44f-1e2e-a2315bba75-8952-4f92-bf4b-6459a819a42broles-pklxkd-1/arn:aws:iam:::role/roles-pklxkd-12023-05-31T20:55:38.806Z3600{"Version":"2012-10-17","Statement":[{"Effect":"Allow","Principal":{"AWS":["arn:aws:iam:::user/testuser"]},"Action":["sts:AssumeRole"]}]}' 2023-05-31T20:55:43.120 DEBUG:boto:Method: POST 2023-05-31T20:55:43.120 DEBUG:boto:Path: / 2023-05-31T20:55:43.121 DEBUG:boto:Data: 2023-05-31T20:55:43.121 DEBUG:boto:Headers: {} 2023-05-31T20:55:43.121 DEBUG:boto:Host: smithi099.front.sepia.ceph.com 2023-05-31T20:55:43.122 DEBUG:boto:Port: 8000 2023-05-31T20:55:43.122 DEBUG:boto:Params: {'RoleName': 'roles-pklxkd-1', 'Action': 'GetRole', 'Version': '2010-05-08'} 2023-05-31T20:55:43.122 DEBUG:boto:Token: None 2023-05-31T20:55:43.123 DEBUG:boto:CanonicalRequest: POST / host:smithi099.front.sepia.ceph.com:8000 x-amz-date:20230531T205543Z host;x-amz-date ca2e30fd9554699b1bb2bbf0af2c0d841e5b87dd54b6db71af5f1fb9e82eb5bb 2023-05-31T20:55:43.123 DEBUG:boto:StringToSign: AWS4-HMAC-SHA256 20230531T205543Z 20230531/front/smithi099/aws4_request 3c904a0e53047607d2855d780ef727ad30ebed98ad17aa839d97bcc3aa82da0a 2023-05-31T20:55:43.124 DEBUG:boto:Signature: baff695cf17b7ec63a807e0afe8f297d896f32189610aa7eefb7926153737d8e 2023-05-31T20:55:43.124 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'X-Amz-Date': '20230531T205543Z', 'Content-Type': 'application/x-www-form-urlencoded; charset=UTF-8', 'Content-Length': '57', 'Authorization': 'AWS4-HMAC-SHA256 Credential=Q80H11OLNU1S48PPYL26/20230531/front/smithi099/aws4_request,SignedHeaders=host;x-amz-date,Signature=baff695cf17b7ec63a807e0afe8f297d896f32189610aa7eefb7926153737d8e', 'Host': 'smithi099.front.sepia.ceph.com:8000'} 2023-05-31T20:55:43.125 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000f98882eb7c782caa-006477b44f-1630-a1'), ('Content-Type', 'application/xml'), ('Content-Length', '711'), ('Date', 'Wed, 31 May 2023 20:55:43 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:55:43.126 DEBUG:boto:b'tx00000f98882eb7c782caa-006477b44f-1630-a1315bba75-8952-4f92-bf4b-6459a819a42broles-pklxkd-1/arn:aws:iam:::role/roles-pklxkd-12023-05-31T20:55:01.732Z3600{"Version":"2012-10-17","Statement":[{"Effect":"Allow","Principal":{"AWS":["arn:aws:iam:::user/testuser"]},"Action":["sts:AssumeRole"]}]}' 2023-05-31T20:55:43.126 DEBUG:tasks.rgw_multi.zone_rados:comparing role name=roles-pklxkd-1 2023-05-31T20:55:43.127 INFO:tasks.rgw_multi.zone_rados:success, role identical: role=roles-pklxkd-1 zones={a2, a1} 2023-05-31T20:55:43.127 INFO:tasks.rgw_multi.zone_rados:comparing role=roles-pklxkd-2 zones={a1, a2} 2023-05-31T20:55:43.127 DEBUG:boto:Method: POST 2023-05-31T20:55:43.128 DEBUG:boto:Path: / 2023-05-31T20:55:43.128 DEBUG:boto:Data: 2023-05-31T20:55:43.128 DEBUG:boto:Headers: {} 2023-05-31T20:55:43.129 DEBUG:boto:Host: smithi099.front.sepia.ceph.com 2023-05-31T20:55:43.129 DEBUG:boto:Port: 8000 2023-05-31T20:55:43.129 DEBUG:boto:Params: {'RoleName': 'roles-pklxkd-2', 'Action': 'GetRole', 'Version': '2010-05-08'} 2023-05-31T20:55:43.130 DEBUG:boto:Token: None 2023-05-31T20:55:43.130 DEBUG:boto:CanonicalRequest: POST / host:smithi099.front.sepia.ceph.com:8000 x-amz-date:20230531T205543Z host;x-amz-date b7f81c882f17f69893a9910da9ff43500bdedfedb39679ea6772df1aa81068a5 2023-05-31T20:55:43.131 DEBUG:boto:StringToSign: AWS4-HMAC-SHA256 20230531T205543Z 20230531/front/smithi099/aws4_request 7f1452be83a643bb73b639523c82efc516f783323a6dd7fc3d37e57e257fed0a 2023-05-31T20:55:43.131 DEBUG:boto:Signature: febf88f63caf24c0f89bbaf3fbcffb02013aeb5785d5a9168f8d902a0d0c0bde 2023-05-31T20:55:43.131 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'X-Amz-Date': '20230531T205543Z', 'Content-Type': 'application/x-www-form-urlencoded; charset=UTF-8', 'Content-Length': '57', 'Authorization': 'AWS4-HMAC-SHA256 Credential=Q80H11OLNU1S48PPYL26/20230531/front/smithi099/aws4_request,SignedHeaders=host;x-amz-date,Signature=febf88f63caf24c0f89bbaf3fbcffb02013aeb5785d5a9168f8d902a0d0c0bde', 'Host': 'smithi099.front.sepia.ceph.com:8000'} 2023-05-31T20:55:43.132 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000d57d58a9da9e0a9d-006477b44f-1630-a1'), ('Content-Type', 'application/xml'), ('Content-Length', '711'), ('Date', 'Wed, 31 May 2023 20:55:43 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:55:43.133 DEBUG:boto:b'tx00000d57d58a9da9e0a9d-006477b44f-1630-a1935318be-8452-4a09-9b0e-e7491b2d22e4roles-pklxkd-2/arn:aws:iam:::role/roles-pklxkd-22023-05-31T20:55:01.753Z3600{"Version":"2012-10-17","Statement":[{"Effect":"Allow","Principal":{"AWS":["arn:aws:iam:::user/testuser"]},"Action":["sts:AssumeRole"]}]}' 2023-05-31T20:55:43.134 DEBUG:boto:Method: POST 2023-05-31T20:55:43.134 DEBUG:boto:Path: / 2023-05-31T20:55:43.134 DEBUG:boto:Data: 2023-05-31T20:55:43.135 DEBUG:boto:Headers: {} 2023-05-31T20:55:43.135 DEBUG:boto:Host: smithi099.front.sepia.ceph.com 2023-05-31T20:55:43.135 DEBUG:boto:Port: 8001 2023-05-31T20:55:43.136 DEBUG:boto:Params: {'RoleName': 'roles-pklxkd-2', 'Action': 'GetRole', 'Version': '2010-05-08'} 2023-05-31T20:55:43.136 DEBUG:boto:Token: None 2023-05-31T20:55:43.137 DEBUG:boto:CanonicalRequest: POST / host:smithi099.front.sepia.ceph.com:8001 x-amz-date:20230531T205543Z host;x-amz-date b7f81c882f17f69893a9910da9ff43500bdedfedb39679ea6772df1aa81068a5 2023-05-31T20:55:43.137 DEBUG:boto:StringToSign: AWS4-HMAC-SHA256 20230531T205543Z 20230531/front/smithi099/aws4_request 8f3ff7daf2b0970f400ac95826636188f9626a0928b0c096bdb393cecbfe71f1 2023-05-31T20:55:43.137 DEBUG:boto:Signature: 8c39a1b5d6b092972d7d79823337b6d967c67350983d554c59f43c99da704818 2023-05-31T20:55:43.138 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'X-Amz-Date': '20230531T205543Z', 'Content-Type': 'application/x-www-form-urlencoded; charset=UTF-8', 'Content-Length': '57', 'Authorization': 'AWS4-HMAC-SHA256 Credential=Q80H11OLNU1S48PPYL26/20230531/front/smithi099/aws4_request,SignedHeaders=host;x-amz-date,Signature=8c39a1b5d6b092972d7d79823337b6d967c67350983d554c59f43c99da704818', 'Host': 'smithi099.front.sepia.ceph.com:8001'} 2023-05-31T20:55:43.140 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx000006e412f643a19983d-006477b44f-1e2e-a2'), ('Content-Type', 'application/xml'), ('Content-Length', '711'), ('Date', 'Wed, 31 May 2023 20:55:43 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:55:43.140 DEBUG:boto:b'tx000006e412f643a19983d-006477b44f-1e2e-a2935318be-8452-4a09-9b0e-e7491b2d22e4roles-pklxkd-2/arn:aws:iam:::role/roles-pklxkd-22023-05-31T20:55:01.753Z3600{"Version":"2012-10-17","Statement":[{"Effect":"Allow","Principal":{"AWS":["arn:aws:iam:::user/testuser"]},"Action":["sts:AssumeRole"]}]}' 2023-05-31T20:55:43.141 DEBUG:tasks.rgw_multi.zone_rados:comparing role name=roles-pklxkd-2 2023-05-31T20:55:43.141 INFO:tasks.rgw_multi.zone_rados:success, role identical: role=roles-pklxkd-2 zones={a1, a2} 2023-05-31T20:55:43.142 INFO:tasks.rgw_multisite_tests:rgw_multi.tests.test_role_sync ... ok 2023-05-31T20:55:43.142 DEBUG:boto:path=/pklxkd-60/ 2023-05-31T20:55:43.143 DEBUG:boto:auth_path=/pklxkd-60/ 2023-05-31T20:55:43.143 DEBUG:boto:Method: PUT 2023-05-31T20:55:43.144 DEBUG:boto:Path: /pklxkd-60/ 2023-05-31T20:55:43.144 DEBUG:boto:Data: 2023-05-31T20:55:43.144 DEBUG:boto:Headers: {} 2023-05-31T20:55:43.145 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:55:43.145 DEBUG:boto:Port: 8000 2023-05-31T20:55:43.145 DEBUG:boto:Params: {} 2023-05-31T20:55:43.146 DEBUG:boto:Token: None 2023-05-31T20:55:43.146 DEBUG:boto:StringToSign: PUT Wed, 31 May 2023 20:55:43 GMT /pklxkd-60/ 2023-05-31T20:55:43.146 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:Z4ALJS0w8vQu/VdLLQ8cLlNQzU8= 2023-05-31T20:55:43.147 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:55:43 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:Z4ALJS0w8vQu/VdLLQ8cLlNQzU8=', 'Content-Length': '0'} 2023-05-31T20:55:43.178 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000474714d27a350c0e-006477b44f-1630-a1'), ('Content-Length', '0'), ('Date', 'Wed, 31 May 2023 20:55:43 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:55:43.178 DEBUG:rgw_multi.tests:created bucket=pklxkd-60 2023-05-31T20:55:43.179 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-31T20:55:43.179 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-31T20:55:43.180 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-05-31T20:55:43.228 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2023-05-31T20:55:43.228 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2023-05-31T20:55:43.285 INFO:teuthology.orchestra.run.smithi099.stderr:No --period given, using current period=2e6123d2-d695-4350-9a01-61a455bc782b 2023-05-31T20:55:43.299 INFO:teuthology.orchestra.run.smithi099.stdout:[ 2023-05-31T20:55:43.299 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T20:55:43.299 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1685566501.757183_260.1", 2023-05-31T20:55:43.299 INFO:teuthology.orchestra.run.smithi099.stdout: "last_update": "2023-05-31T20:55:01.757183Z" 2023-05-31T20:55:43.300 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:55:43.300 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T20:55:43.300 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1685566501.735717_251.1", 2023-05-31T20:55:43.300 INFO:teuthology.orchestra.run.smithi099.stdout: "last_update": "2023-05-31T20:55:01.735717Z" 2023-05-31T20:55:43.300 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:55:43.300 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T20:55:43.301 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1685562970.274572_93.1", 2023-05-31T20:55:43.301 INFO:teuthology.orchestra.run.smithi099.stdout: "last_update": "2023-05-31T19:56:10.274572Z" 2023-05-31T20:55:43.301 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:55:43.301 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T20:55:43.301 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1685566543.171347_487.1", 2023-05-31T20:55:43.302 INFO:teuthology.orchestra.run.smithi099.stdout: "last_update": "2023-05-31T20:55:43.171347Z" 2023-05-31T20:55:43.302 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T20:55:43.302 INFO:teuthology.orchestra.run.smithi099.stdout:] 2023-05-31T20:55:43.303 DEBUG:rgw_multi.tests:master meta markers={0: '1_1685566501.757183_260.1', 1: '1_1685566501.735717_251.1', 2: '1_1685562970.274572_93.1', 3: '1_1685566543.171347_487.1'} 2023-05-31T20:55:43.304 INFO:rgw_multi.tests:starting meta checkpoint for zone=a2 2023-05-31T20:55:43.304 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-31T20:55:43.304 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-31T20:55:43.305 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-05-31T20:55:43.353 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2023-05-31T20:55:43.353 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2023-05-31T20:55:43.384 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:55:43.379+0000 7f86dc9f3780 20 RGW-SYNC:meta: read sync status 2023-05-31T20:55:43.385 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:55:43.380+0000 7f86dc9f3780 20 RGW-SYNC:meta: read sync status 2023-05-31T20:55:43.398 INFO:teuthology.orchestra.run.smithi099.stdout:{ 2023-05-31T20:55:43.399 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_status": { 2023-05-31T20:55:43.399 INFO:teuthology.orchestra.run.smithi099.stdout: "info": { 2023-05-31T20:55:43.399 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "sync", 2023-05-31T20:55:43.399 INFO:teuthology.orchestra.run.smithi099.stdout: "num_shards": 4, 2023-05-31T20:55:43.399 INFO:teuthology.orchestra.run.smithi099.stdout: "period": "2e6123d2-d695-4350-9a01-61a455bc782b", 2023-05-31T20:55:43.400 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2023-05-31T20:55:43.400 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:55:43.400 INFO:teuthology.orchestra.run.smithi099.stdout: "markers": [ 2023-05-31T20:55:43.400 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T20:55:43.400 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 0, 2023-05-31T20:55:43.400 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T20:55:43.400 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2023-05-31T20:55:43.401 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1685566501.757183_260.1", 2023-05-31T20:55:43.401 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T20:55:43.401 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 1, 2023-05-31T20:55:43.401 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T20:55:43.401 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2023-05-31T20:55:01.757183Z", 2023-05-31T20:55:43.401 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2023-05-31T20:55:43.402 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T20:55:43.402 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:55:43.402 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T20:55:43.402 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 1, 2023-05-31T20:55:43.402 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T20:55:43.402 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2023-05-31T20:55:43.402 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1685566501.735717_251.1", 2023-05-31T20:55:43.403 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T20:55:43.403 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2023-05-31T20:55:43.403 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T20:55:43.403 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2023-05-31T20:55:01.735717Z", 2023-05-31T20:55:43.403 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2023-05-31T20:55:43.403 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T20:55:43.403 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:55:43.404 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T20:55:43.404 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 2, 2023-05-31T20:55:43.404 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T20:55:43.404 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2023-05-31T20:55:43.404 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1685562970.274572_93.1", 2023-05-31T20:55:43.404 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T20:55:43.404 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2023-05-31T20:55:43.405 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T20:55:43.405 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2023-05-31T19:56:10.274572Z", 2023-05-31T20:55:43.405 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2023-05-31T20:55:43.405 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T20:55:43.405 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:55:43.405 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T20:55:43.405 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 3, 2023-05-31T20:55:43.406 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T20:55:43.406 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2023-05-31T20:55:43.406 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1685566334.798037_482.1", 2023-05-31T20:55:43.406 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T20:55:43.406 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2023-05-31T20:55:43.406 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T20:55:43.406 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2023-05-31T20:52:14.798037Z", 2023-05-31T20:55:43.407 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2023-05-31T20:55:43.407 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T20:55:43.407 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T20:55:43.407 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2023-05-31T20:55:43.407 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:55:43.407 INFO:teuthology.orchestra.run.smithi099.stdout: "full_sync": { 2023-05-31T20:55:43.408 INFO:teuthology.orchestra.run.smithi099.stdout: "total": 1, 2023-05-31T20:55:43.408 INFO:teuthology.orchestra.run.smithi099.stdout: "complete": 1 2023-05-31T20:55:43.408 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:55:43.408 INFO:teuthology.orchestra.run.smithi099.stdout: "current_time": "2023-05-31T20:55:43Z" 2023-05-31T20:55:43.408 INFO:teuthology.orchestra.run.smithi099.stdout:} 2023-05-31T20:55:43.409 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "2e6123d2-d695-4350-9a01-61a455bc782b", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1685566501.757183_260.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-05-31T20:55:01.757183Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1685566501.735717_251.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-05-31T20:55:01.735717Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1685562970.274572_93.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-05-31T19:56:10.274572Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1685566334.798037_482.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-05-31T20:52:14.798037Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-05-31T20:55:43Z" } 2023-05-31T20:55:43.410 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1685566501.757183_260.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-05-31T20:55:01.757183Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1685566501.735717_251.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-05-31T20:55:01.735717Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1685562970.274572_93.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-05-31T19:56:10.274572Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1685566334.798037_482.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-05-31T20:52:14.798037Z', 'realm_epoch': 2}}] 2023-05-31T20:55:43.410 DEBUG:rgw_multi.tests:log_status={0: '1_1685566501.757183_260.1', 1: '1_1685566501.735717_251.1', 2: '1_1685562970.274572_93.1', 3: '1_1685566543.171347_487.1'} 2023-05-31T20:55:43.410 DEBUG:rgw_multi.tests:sync_status={0: '1_1685566501.757183_260.1', 1: '1_1685566501.735717_251.1', 2: '1_1685562970.274572_93.1', 3: '1_1685566334.798037_482.1'} 2023-05-31T20:55:43.411 WARNING:rgw_multi.tests:zone a2 behind master: shard=3 master=1_1685566543.171347_487.1 target=1_1685566334.798037_482.1 2023-05-31T20:55:48.411 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-31T20:55:48.412 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-31T20:55:48.413 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-05-31T20:55:48.470 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2023-05-31T20:55:48.471 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2023-05-31T20:55:48.521 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:55:48.518+0000 7fe1db43a780 20 RGW-SYNC:meta: read sync status 2023-05-31T20:55:48.522 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:55:48.519+0000 7fe1db43a780 20 RGW-SYNC:meta: read sync status 2023-05-31T20:55:48.539 INFO:teuthology.orchestra.run.smithi099.stdout:{ 2023-05-31T20:55:48.539 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_status": { 2023-05-31T20:55:48.539 INFO:teuthology.orchestra.run.smithi099.stdout: "info": { 2023-05-31T20:55:48.540 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "sync", 2023-05-31T20:55:48.540 INFO:teuthology.orchestra.run.smithi099.stdout: "num_shards": 4, 2023-05-31T20:55:48.540 INFO:teuthology.orchestra.run.smithi099.stdout: "period": "2e6123d2-d695-4350-9a01-61a455bc782b", 2023-05-31T20:55:48.540 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2023-05-31T20:55:48.540 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:55:48.540 INFO:teuthology.orchestra.run.smithi099.stdout: "markers": [ 2023-05-31T20:55:48.541 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T20:55:48.541 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 0, 2023-05-31T20:55:48.541 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T20:55:48.541 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2023-05-31T20:55:48.541 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1685566501.757183_260.1", 2023-05-31T20:55:48.541 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T20:55:48.541 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 1, 2023-05-31T20:55:48.542 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T20:55:48.542 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2023-05-31T20:55:01.757183Z", 2023-05-31T20:55:48.542 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2023-05-31T20:55:48.542 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T20:55:48.542 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:55:48.542 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T20:55:48.542 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 1, 2023-05-31T20:55:48.543 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T20:55:48.543 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2023-05-31T20:55:48.543 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1685566501.735717_251.1", 2023-05-31T20:55:48.543 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T20:55:48.543 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2023-05-31T20:55:48.543 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T20:55:48.543 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2023-05-31T20:55:01.735717Z", 2023-05-31T20:55:48.544 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2023-05-31T20:55:48.544 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T20:55:48.544 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:55:48.544 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T20:55:48.544 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 2, 2023-05-31T20:55:48.544 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T20:55:48.545 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2023-05-31T20:55:48.545 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1685562970.274572_93.1", 2023-05-31T20:55:48.545 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T20:55:48.545 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2023-05-31T20:55:48.545 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T20:55:48.545 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2023-05-31T19:56:10.274572Z", 2023-05-31T20:55:48.545 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2023-05-31T20:55:48.545 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T20:55:48.546 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:55:48.546 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T20:55:48.546 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 3, 2023-05-31T20:55:48.546 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T20:55:48.546 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2023-05-31T20:55:48.546 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1685566334.798037_482.1", 2023-05-31T20:55:48.547 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T20:55:48.547 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2023-05-31T20:55:48.547 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T20:55:48.547 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2023-05-31T20:52:14.798037Z", 2023-05-31T20:55:48.547 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2023-05-31T20:55:48.547 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T20:55:48.547 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T20:55:48.548 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2023-05-31T20:55:48.548 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:55:48.548 INFO:teuthology.orchestra.run.smithi099.stdout: "full_sync": { 2023-05-31T20:55:48.548 INFO:teuthology.orchestra.run.smithi099.stdout: "total": 1, 2023-05-31T20:55:48.548 INFO:teuthology.orchestra.run.smithi099.stdout: "complete": 1 2023-05-31T20:55:48.548 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:55:48.548 INFO:teuthology.orchestra.run.smithi099.stdout: "current_time": "2023-05-31T20:55:48Z" 2023-05-31T20:55:48.549 INFO:teuthology.orchestra.run.smithi099.stdout:} 2023-05-31T20:55:48.549 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "2e6123d2-d695-4350-9a01-61a455bc782b", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1685566501.757183_260.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-05-31T20:55:01.757183Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1685566501.735717_251.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-05-31T20:55:01.735717Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1685562970.274572_93.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-05-31T19:56:10.274572Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1685566334.798037_482.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-05-31T20:52:14.798037Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-05-31T20:55:48Z" } 2023-05-31T20:55:48.550 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1685566501.757183_260.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-05-31T20:55:01.757183Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1685566501.735717_251.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-05-31T20:55:01.735717Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1685562970.274572_93.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-05-31T19:56:10.274572Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1685566334.798037_482.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-05-31T20:52:14.798037Z', 'realm_epoch': 2}}] 2023-05-31T20:55:48.550 DEBUG:rgw_multi.tests:log_status={0: '1_1685566501.757183_260.1', 1: '1_1685566501.735717_251.1', 2: '1_1685562970.274572_93.1', 3: '1_1685566543.171347_487.1'} 2023-05-31T20:55:48.551 DEBUG:rgw_multi.tests:sync_status={0: '1_1685566501.757183_260.1', 1: '1_1685566501.735717_251.1', 2: '1_1685562970.274572_93.1', 3: '1_1685566334.798037_482.1'} 2023-05-31T20:55:48.551 WARNING:rgw_multi.tests:zone a2 behind master: shard=3 master=1_1685566543.171347_487.1 target=1_1685566334.798037_482.1 2023-05-31T20:55:53.552 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-31T20:55:53.553 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-31T20:55:53.553 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-05-31T20:55:53.610 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2023-05-31T20:55:53.610 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2023-05-31T20:55:53.649 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:55:53.646+0000 7f40c7441780 20 RGW-SYNC:meta: read sync status 2023-05-31T20:55:53.650 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:55:53.647+0000 7f40c7441780 20 RGW-SYNC:meta: read sync status 2023-05-31T20:55:53.666 INFO:teuthology.orchestra.run.smithi099.stdout:{ 2023-05-31T20:55:53.666 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_status": { 2023-05-31T20:55:53.666 INFO:teuthology.orchestra.run.smithi099.stdout: "info": { 2023-05-31T20:55:53.667 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "sync", 2023-05-31T20:55:53.667 INFO:teuthology.orchestra.run.smithi099.stdout: "num_shards": 4, 2023-05-31T20:55:53.667 INFO:teuthology.orchestra.run.smithi099.stdout: "period": "2e6123d2-d695-4350-9a01-61a455bc782b", 2023-05-31T20:55:53.667 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2023-05-31T20:55:53.667 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:55:53.667 INFO:teuthology.orchestra.run.smithi099.stdout: "markers": [ 2023-05-31T20:55:53.668 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T20:55:53.668 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 0, 2023-05-31T20:55:53.668 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T20:55:53.668 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2023-05-31T20:55:53.668 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1685566501.757183_260.1", 2023-05-31T20:55:53.668 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T20:55:53.669 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 1, 2023-05-31T20:55:53.669 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T20:55:53.669 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2023-05-31T20:55:01.757183Z", 2023-05-31T20:55:53.669 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2023-05-31T20:55:53.669 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T20:55:53.669 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:55:53.669 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T20:55:53.670 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 1, 2023-05-31T20:55:53.670 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T20:55:53.670 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2023-05-31T20:55:53.670 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1685566501.735717_251.1", 2023-05-31T20:55:53.670 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T20:55:53.670 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2023-05-31T20:55:53.670 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T20:55:53.671 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2023-05-31T20:55:01.735717Z", 2023-05-31T20:55:53.671 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2023-05-31T20:55:53.671 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T20:55:53.671 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:55:53.671 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T20:55:53.671 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 2, 2023-05-31T20:55:53.671 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T20:55:53.672 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2023-05-31T20:55:53.672 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1685562970.274572_93.1", 2023-05-31T20:55:53.672 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T20:55:53.672 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2023-05-31T20:55:53.672 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T20:55:53.672 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2023-05-31T19:56:10.274572Z", 2023-05-31T20:55:53.672 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2023-05-31T20:55:53.673 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T20:55:53.673 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:55:53.673 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T20:55:53.673 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 3, 2023-05-31T20:55:53.673 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T20:55:53.673 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2023-05-31T20:55:53.673 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1685566334.798037_482.1", 2023-05-31T20:55:53.674 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T20:55:53.674 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2023-05-31T20:55:53.674 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T20:55:53.674 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2023-05-31T20:52:14.798037Z", 2023-05-31T20:55:53.674 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2023-05-31T20:55:53.674 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T20:55:53.674 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T20:55:53.675 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2023-05-31T20:55:53.675 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:55:53.675 INFO:teuthology.orchestra.run.smithi099.stdout: "full_sync": { 2023-05-31T20:55:53.675 INFO:teuthology.orchestra.run.smithi099.stdout: "total": 1, 2023-05-31T20:55:53.675 INFO:teuthology.orchestra.run.smithi099.stdout: "complete": 1 2023-05-31T20:55:53.675 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:55:53.675 INFO:teuthology.orchestra.run.smithi099.stdout: "current_time": "2023-05-31T20:55:53Z" 2023-05-31T20:55:53.676 INFO:teuthology.orchestra.run.smithi099.stdout:} 2023-05-31T20:55:53.676 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "2e6123d2-d695-4350-9a01-61a455bc782b", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1685566501.757183_260.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-05-31T20:55:01.757183Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1685566501.735717_251.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-05-31T20:55:01.735717Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1685562970.274572_93.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-05-31T19:56:10.274572Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1685566334.798037_482.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-05-31T20:52:14.798037Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-05-31T20:55:53Z" } 2023-05-31T20:55:53.677 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1685566501.757183_260.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-05-31T20:55:01.757183Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1685566501.735717_251.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-05-31T20:55:01.735717Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1685562970.274572_93.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-05-31T19:56:10.274572Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1685566334.798037_482.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-05-31T20:52:14.798037Z', 'realm_epoch': 2}}] 2023-05-31T20:55:53.677 DEBUG:rgw_multi.tests:log_status={0: '1_1685566501.757183_260.1', 1: '1_1685566501.735717_251.1', 2: '1_1685562970.274572_93.1', 3: '1_1685566543.171347_487.1'} 2023-05-31T20:55:53.678 DEBUG:rgw_multi.tests:sync_status={0: '1_1685566501.757183_260.1', 1: '1_1685566501.735717_251.1', 2: '1_1685562970.274572_93.1', 3: '1_1685566334.798037_482.1'} 2023-05-31T20:55:53.678 WARNING:rgw_multi.tests:zone a2 behind master: shard=3 master=1_1685566543.171347_487.1 target=1_1685566334.798037_482.1 2023-05-31T20:55:58.679 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-31T20:55:58.680 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-31T20:55:58.681 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-05-31T20:55:58.736 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2023-05-31T20:55:58.736 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2023-05-31T20:55:58.778 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:55:58.774+0000 7f167a074780 20 RGW-SYNC:meta: read sync status 2023-05-31T20:55:58.779 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:55:58.775+0000 7f167a074780 20 RGW-SYNC:meta: read sync status 2023-05-31T20:55:58.793 INFO:teuthology.orchestra.run.smithi099.stdout:{ 2023-05-31T20:55:58.794 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_status": { 2023-05-31T20:55:58.794 INFO:teuthology.orchestra.run.smithi099.stdout: "info": { 2023-05-31T20:55:58.794 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "sync", 2023-05-31T20:55:58.794 INFO:teuthology.orchestra.run.smithi099.stdout: "num_shards": 4, 2023-05-31T20:55:58.794 INFO:teuthology.orchestra.run.smithi099.stdout: "period": "2e6123d2-d695-4350-9a01-61a455bc782b", 2023-05-31T20:55:58.794 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2023-05-31T20:55:58.795 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:55:58.795 INFO:teuthology.orchestra.run.smithi099.stdout: "markers": [ 2023-05-31T20:55:58.795 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T20:55:58.795 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 0, 2023-05-31T20:55:58.795 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T20:55:58.795 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2023-05-31T20:55:58.795 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1685566501.757183_260.1", 2023-05-31T20:55:58.796 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T20:55:58.796 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 1, 2023-05-31T20:55:58.796 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T20:55:58.796 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2023-05-31T20:55:01.757183Z", 2023-05-31T20:55:58.796 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2023-05-31T20:55:58.796 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T20:55:58.797 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:55:58.797 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T20:55:58.797 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 1, 2023-05-31T20:55:58.797 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T20:55:58.797 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2023-05-31T20:55:58.797 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1685566501.735717_251.1", 2023-05-31T20:55:58.797 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T20:55:58.798 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2023-05-31T20:55:58.798 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T20:55:58.798 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2023-05-31T20:55:01.735717Z", 2023-05-31T20:55:58.798 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2023-05-31T20:55:58.798 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T20:55:58.798 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:55:58.798 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T20:55:58.799 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 2, 2023-05-31T20:55:58.799 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T20:55:58.799 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2023-05-31T20:55:58.799 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1685562970.274572_93.1", 2023-05-31T20:55:58.799 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T20:55:58.799 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2023-05-31T20:55:58.799 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T20:55:58.800 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2023-05-31T19:56:10.274572Z", 2023-05-31T20:55:58.800 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2023-05-31T20:55:58.800 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T20:55:58.800 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:55:58.800 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T20:55:58.800 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 3, 2023-05-31T20:55:58.801 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T20:55:58.801 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2023-05-31T20:55:58.801 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1685566543.171347_487.1", 2023-05-31T20:55:58.801 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T20:55:58.801 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2023-05-31T20:55:58.801 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T20:55:58.801 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2023-05-31T20:55:43.171347Z", 2023-05-31T20:55:58.802 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2023-05-31T20:55:58.802 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T20:55:58.802 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T20:55:58.802 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2023-05-31T20:55:58.802 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:55:58.802 INFO:teuthology.orchestra.run.smithi099.stdout: "full_sync": { 2023-05-31T20:55:58.802 INFO:teuthology.orchestra.run.smithi099.stdout: "total": 1, 2023-05-31T20:55:58.803 INFO:teuthology.orchestra.run.smithi099.stdout: "complete": 1 2023-05-31T20:55:58.803 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:55:58.803 INFO:teuthology.orchestra.run.smithi099.stdout: "current_time": "2023-05-31T20:55:58Z" 2023-05-31T20:55:58.803 INFO:teuthology.orchestra.run.smithi099.stdout:} 2023-05-31T20:55:58.804 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "2e6123d2-d695-4350-9a01-61a455bc782b", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1685566501.757183_260.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-05-31T20:55:01.757183Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1685566501.735717_251.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-05-31T20:55:01.735717Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1685562970.274572_93.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-05-31T19:56:10.274572Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1685566543.171347_487.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-05-31T20:55:43.171347Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-05-31T20:55:58Z" } 2023-05-31T20:55:58.804 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1685566501.757183_260.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-05-31T20:55:01.757183Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1685566501.735717_251.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-05-31T20:55:01.735717Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1685562970.274572_93.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-05-31T19:56:10.274572Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1685566543.171347_487.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-05-31T20:55:43.171347Z', 'realm_epoch': 2}}] 2023-05-31T20:55:58.805 DEBUG:rgw_multi.tests:log_status={0: '1_1685566501.757183_260.1', 1: '1_1685566501.735717_251.1', 2: '1_1685562970.274572_93.1', 3: '1_1685566543.171347_487.1'} 2023-05-31T20:55:58.805 DEBUG:rgw_multi.tests:sync_status={0: '1_1685566501.757183_260.1', 1: '1_1685566501.735717_251.1', 2: '1_1685562970.274572_93.1', 3: '1_1685566543.171347_487.1'} 2023-05-31T20:55:58.805 INFO:rgw_multi.tests:finish meta checkpoint for zone=a2 2023-05-31T20:55:58.806 DEBUG:tasks.rgw.c1.client.1:waiting for process to exit 2023-05-31T20:55:58.806 INFO:teuthology.orchestra.run:waiting for 300 2023-05-31T20:55:58.807 INFO:tasks.rgw.c1.client.1.smithi099.stdout:2023-05-31T20:55:58.803+0000 7fd22d2ce700 -1 received signal: Terminated from /usr/bin/python3 /bin/daemon-helper term radosgw --rgw-frontends beast port=8001 -n client.1 --cluster c1 -k /etc/ceph/c1.client.1.keyring --log-file /var/log/ceph/rgw.c1.client.1.log --rgw_ops_log_socket_path /home/ubuntu/cephtest/rgw.opslog.c1.client.1.sock --foreground --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm (PID: 162029) UID: 0 2023-05-31T20:55:58.807 INFO:tasks.rgw.c1.client.1.smithi099.stdout:2023-05-31T20:55:58.803+0000 7fd2315377c0 -1 shutting down 2023-05-31T20:56:03.895 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.1 is failed for ~0s 2023-05-31T20:56:04.909 INFO:tasks.rgw.c1.client.1:Stopped 2023-05-31T20:56:04.909 DEBUG:boto:path=/pklxkd-60/ 2023-05-31T20:56:04.910 DEBUG:boto:auth_path=/pklxkd-60/ 2023-05-31T20:56:04.910 DEBUG:boto:Method: HEAD 2023-05-31T20:56:04.910 DEBUG:boto:Path: /pklxkd-60/ 2023-05-31T20:56:04.911 DEBUG:boto:Data: 2023-05-31T20:56:04.911 DEBUG:boto:Headers: {} 2023-05-31T20:56:04.911 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:56:04.912 DEBUG:boto:Port: 8000 2023-05-31T20:56:04.912 DEBUG:boto:Params: {} 2023-05-31T20:56:04.913 DEBUG:boto:Token: None 2023-05-31T20:56:04.913 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:56:04 GMT /pklxkd-60/ 2023-05-31T20:56:04.913 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:hkwP+Lfm0Sic1uqR2r/nZFLnrRg= 2023-05-31T20:56:04.914 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:56:04 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:hkwP+Lfm0Sic1uqR2r/nZFLnrRg=', 'Content-Length': '0'} 2023-05-31T20:56:04.917 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000006a2921c9bc31a31-006477b464-1630-a1'), ('Content-Length', '0'), ('Date', 'Wed, 31 May 2023 20:56:04 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:56:04.918 DEBUG:boto:path=/pklxkd-60/obj11 2023-05-31T20:56:04.918 DEBUG:boto:auth_path=/pklxkd-60/obj11 2023-05-31T20:56:04.919 DEBUG:boto:Method: PUT 2023-05-31T20:56:04.919 DEBUG:boto:Path: /pklxkd-60/obj11 2023-05-31T20:56:04.919 DEBUG:boto:Data: 2023-05-31T20:56:04.920 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-05-31T20:56:04.920 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:56:04.920 DEBUG:boto:Port: 8000 2023-05-31T20:56:04.921 DEBUG:boto:Params: {} 2023-05-31T20:56:04.921 DEBUG:boto:Token: None 2023-05-31T20:56:04.921 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Wed, 31 May 2023 20:56:04 GMT /pklxkd-60/obj11 2023-05-31T20:56:04.922 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:COyESP2F3z5WnwPa+FSKxm60Vrw= 2023-05-31T20:56:04.922 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Wed, 31 May 2023 20:56:04 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:COyESP2F3z5WnwPa+FSKxm60Vrw='} 2023-05-31T20:56:04.933 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000089ab91661701923d-006477b464-1630-a1'), ('Date', 'Wed, 31 May 2023 20:56:04 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:56:04.934 DEBUG:boto:path=/pklxkd-60/ 2023-05-31T20:56:04.934 DEBUG:boto:auth_path=/pklxkd-60/ 2023-05-31T20:56:04.934 DEBUG:boto:Method: HEAD 2023-05-31T20:56:04.935 DEBUG:boto:Path: /pklxkd-60/ 2023-05-31T20:56:04.935 DEBUG:boto:Data: 2023-05-31T20:56:04.935 DEBUG:boto:Headers: {} 2023-05-31T20:56:04.936 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:56:04.936 DEBUG:boto:Port: 8000 2023-05-31T20:56:04.936 DEBUG:boto:Params: {} 2023-05-31T20:56:04.937 DEBUG:boto:Token: None 2023-05-31T20:56:04.937 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:56:04 GMT /pklxkd-60/ 2023-05-31T20:56:04.938 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:hkwP+Lfm0Sic1uqR2r/nZFLnrRg= 2023-05-31T20:56:04.938 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:56:04 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:hkwP+Lfm0Sic1uqR2r/nZFLnrRg=', 'Content-Length': '0'} 2023-05-31T20:56:04.940 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '3'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000024d707c4af23bda-006477b464-1630-a1'), ('Content-Length', '0'), ('Date', 'Wed, 31 May 2023 20:56:04 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:56:04.941 DEBUG:boto:path=/pklxkd-60/obj22 2023-05-31T20:56:04.941 DEBUG:boto:auth_path=/pklxkd-60/obj22 2023-05-31T20:56:04.941 DEBUG:boto:Method: PUT 2023-05-31T20:56:04.942 DEBUG:boto:Path: /pklxkd-60/obj22 2023-05-31T20:56:04.942 DEBUG:boto:Data: 2023-05-31T20:56:04.942 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-05-31T20:56:04.943 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:56:04.943 DEBUG:boto:Port: 8000 2023-05-31T20:56:04.943 DEBUG:boto:Params: {} 2023-05-31T20:56:04.944 DEBUG:boto:Token: None 2023-05-31T20:56:04.944 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Wed, 31 May 2023 20:56:04 GMT /pklxkd-60/obj22 2023-05-31T20:56:04.945 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:PbxFJ4IfXo9rfQxJ0nDghVx574Y= 2023-05-31T20:56:04.945 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Wed, 31 May 2023 20:56:04 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:PbxFJ4IfXo9rfQxJ0nDghVx574Y='} 2023-05-31T20:56:04.988 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000002f60f39dceb0f991-006477b464-1630-a1'), ('Date', 'Wed, 31 May 2023 20:56:04 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:56:04.989 DEBUG:boto:path=/pklxkd-60/ 2023-05-31T20:56:04.989 DEBUG:boto:auth_path=/pklxkd-60/ 2023-05-31T20:56:04.990 DEBUG:boto:Method: HEAD 2023-05-31T20:56:04.990 DEBUG:boto:Path: /pklxkd-60/ 2023-05-31T20:56:04.990 DEBUG:boto:Data: 2023-05-31T20:56:04.991 DEBUG:boto:Headers: {} 2023-05-31T20:56:04.991 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:56:04.991 DEBUG:boto:Port: 8000 2023-05-31T20:56:04.991 DEBUG:boto:Params: {} 2023-05-31T20:56:04.992 DEBUG:boto:Token: None 2023-05-31T20:56:04.992 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:56:04 GMT /pklxkd-60/ 2023-05-31T20:56:04.993 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:hkwP+Lfm0Sic1uqR2r/nZFLnrRg= 2023-05-31T20:56:04.993 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:56:04 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:hkwP+Lfm0Sic1uqR2r/nZFLnrRg=', 'Content-Length': '0'} 2023-05-31T20:56:04.995 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '2'), ('X-RGW-Bytes-Used', '6'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000071324f1d7708b361-006477b464-1630-a1'), ('Content-Length', '0'), ('Date', 'Wed, 31 May 2023 20:56:04 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:56:04.996 DEBUG:boto:path=/pklxkd-60/obj33 2023-05-31T20:56:04.996 DEBUG:boto:auth_path=/pklxkd-60/obj33 2023-05-31T20:56:04.996 DEBUG:boto:Method: PUT 2023-05-31T20:56:04.997 DEBUG:boto:Path: /pklxkd-60/obj33 2023-05-31T20:56:04.997 DEBUG:boto:Data: 2023-05-31T20:56:04.997 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-05-31T20:56:04.998 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:56:04.998 DEBUG:boto:Port: 8000 2023-05-31T20:56:04.998 DEBUG:boto:Params: {} 2023-05-31T20:56:04.999 DEBUG:boto:Token: None 2023-05-31T20:56:04.999 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Wed, 31 May 2023 20:56:04 GMT /pklxkd-60/obj33 2023-05-31T20:56:05.000 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:ms0t6H0U83D/K2MDOfavSxYJxy8= 2023-05-31T20:56:05.000 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Wed, 31 May 2023 20:56:04 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:ms0t6H0U83D/K2MDOfavSxYJxy8='} 2023-05-31T20:56:05.010 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000005c70d610f629f954-006477b464-1630-a1'), ('Date', 'Wed, 31 May 2023 20:56:05 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:56:05.011 DEBUG:boto:path=/pklxkd-60/ 2023-05-31T20:56:05.011 DEBUG:boto:auth_path=/pklxkd-60/ 2023-05-31T20:56:05.012 DEBUG:boto:Method: HEAD 2023-05-31T20:56:05.012 DEBUG:boto:Path: /pklxkd-60/ 2023-05-31T20:56:05.012 DEBUG:boto:Data: 2023-05-31T20:56:05.013 DEBUG:boto:Headers: {} 2023-05-31T20:56:05.013 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:56:05.013 DEBUG:boto:Port: 8000 2023-05-31T20:56:05.014 DEBUG:boto:Params: {} 2023-05-31T20:56:05.014 DEBUG:boto:Token: None 2023-05-31T20:56:05.014 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:56:05 GMT /pklxkd-60/ 2023-05-31T20:56:05.015 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:dXk+4adc/1RL89xUDhcLgV8+M5M= 2023-05-31T20:56:05.015 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:56:05 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:dXk+4adc/1RL89xUDhcLgV8+M5M=', 'Content-Length': '0'} 2023-05-31T20:56:05.017 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '3'), ('X-RGW-Bytes-Used', '9'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000d8d43251c319f09a-006477b465-1630-a1'), ('Content-Length', '0'), ('Date', 'Wed, 31 May 2023 20:56:05 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:56:05.018 DEBUG:boto:path=/pklxkd-60/obj44 2023-05-31T20:56:05.018 DEBUG:boto:auth_path=/pklxkd-60/obj44 2023-05-31T20:56:05.019 DEBUG:boto:Method: PUT 2023-05-31T20:56:05.019 DEBUG:boto:Path: /pklxkd-60/obj44 2023-05-31T20:56:05.019 DEBUG:boto:Data: 2023-05-31T20:56:05.020 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-05-31T20:56:05.020 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:56:05.020 DEBUG:boto:Port: 8000 2023-05-31T20:56:05.021 DEBUG:boto:Params: {} 2023-05-31T20:56:05.021 DEBUG:boto:Token: None 2023-05-31T20:56:05.021 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Wed, 31 May 2023 20:56:05 GMT /pklxkd-60/obj44 2023-05-31T20:56:05.022 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:OSXDKSwd/xD0iLIwyfIWmCrnbeg= 2023-05-31T20:56:05.022 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Wed, 31 May 2023 20:56:05 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:OSXDKSwd/xD0iLIwyfIWmCrnbeg='} 2023-05-31T20:56:05.064 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000ba84296e0b6a104d-006477b465-1630-a1'), ('Date', 'Wed, 31 May 2023 20:56:05 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:56:05.065 DEBUG:boto:path=/pklxkd-60/ 2023-05-31T20:56:05.065 DEBUG:boto:auth_path=/pklxkd-60/ 2023-05-31T20:56:05.066 DEBUG:boto:Method: HEAD 2023-05-31T20:56:05.066 DEBUG:boto:Path: /pklxkd-60/ 2023-05-31T20:56:05.066 DEBUG:boto:Data: 2023-05-31T20:56:05.067 DEBUG:boto:Headers: {} 2023-05-31T20:56:05.067 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:56:05.067 DEBUG:boto:Port: 8000 2023-05-31T20:56:05.068 DEBUG:boto:Params: {} 2023-05-31T20:56:05.068 DEBUG:boto:Token: None 2023-05-31T20:56:05.068 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:56:05 GMT /pklxkd-60/ 2023-05-31T20:56:05.069 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:dXk+4adc/1RL89xUDhcLgV8+M5M= 2023-05-31T20:56:05.069 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:56:05 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:dXk+4adc/1RL89xUDhcLgV8+M5M=', 'Content-Length': '0'} 2023-05-31T20:56:05.071 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '4'), ('X-RGW-Bytes-Used', '12'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000002af2693df5edd545-006477b465-1630-a1'), ('Content-Length', '0'), ('Date', 'Wed, 31 May 2023 20:56:05 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:56:05.072 DEBUG:boto:path=/pklxkd-60/obj55 2023-05-31T20:56:05.072 DEBUG:boto:auth_path=/pklxkd-60/obj55 2023-05-31T20:56:05.073 DEBUG:boto:Method: PUT 2023-05-31T20:56:05.073 DEBUG:boto:Path: /pklxkd-60/obj55 2023-05-31T20:56:05.073 DEBUG:boto:Data: 2023-05-31T20:56:05.074 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-05-31T20:56:05.074 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:56:05.074 DEBUG:boto:Port: 8000 2023-05-31T20:56:05.075 DEBUG:boto:Params: {} 2023-05-31T20:56:05.075 DEBUG:boto:Token: None 2023-05-31T20:56:05.075 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Wed, 31 May 2023 20:56:05 GMT /pklxkd-60/obj55 2023-05-31T20:56:05.076 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:COHM7zWKDbswa/jCVEbBVaP+q8E= 2023-05-31T20:56:05.076 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Wed, 31 May 2023 20:56:05 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:COHM7zWKDbswa/jCVEbBVaP+q8E='} 2023-05-31T20:56:05.082 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000753e6a65547e7258-006477b465-1630-a1'), ('Date', 'Wed, 31 May 2023 20:56:05 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:56:05.083 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'init', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-05-31T20:56:05.083 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'init', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-05-31T20:56:05.083 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync init --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 2023-05-31T20:56:05.137 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2023-05-31T20:56:05.138 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2023-05-31T20:56:05.196 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:56:05.192+0000 7ff303f7a780 5 RGW-SYNC:data: waiting on data sync status lease 2023-05-31T20:56:05.197 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:56:05.193+0000 7ff303f7a780 5 RGW-SYNC:data: acquired data sync status lease 2023-05-31T20:56:05.227 INFO:tasks.rgw.c1.client.1:Restarting daemon 2023-05-31T20:56:05.227 DEBUG:teuthology.orchestra.run.smithi099:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term radosgw --rgw-frontends 'beast port=8001' -n client.1 --cluster c1 -k /etc/ceph/c1.client.1.keyring --log-file /var/log/ceph/rgw.c1.client.1.log --rgw_ops_log_socket_path /home/ubuntu/cephtest/rgw.opslog.c1.client.1.sock --foreground --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm | sudo tee /var/log/ceph/rgw.c1.client.1.stdout 2>&1 2023-05-31T20:56:05.229 INFO:tasks.rgw.c1.client.1:Started 2023-05-31T20:56:05.230 DEBUG:teuthology.orchestra.run.smithi099:> curl http://smithi099.front.sepia.ceph.com:8001 2023-05-31T20:56:05.250 INFO:teuthology.orchestra.run.smithi099.stderr: % Total % Received % Xferd Average Speed Time Time Time Current 2023-05-31T20:56:05.250 INFO:teuthology.orchestra.run.smithi099.stderr: Dload Upload Total Spent Left Speed 2023-05-31T20:56:05.251 DEBUG:teuthology.orchestra.run:got remote process result: 7 2023-05-31T20:56:05.252 INFO:teuthology.orchestra.run.smithi099.stderr: 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0curl: (7) Failed to connect to smithi099.front.sepia.ceph.com port 8001: Connection refused 2023-05-31T20:56:06.253 DEBUG:teuthology.orchestra.run.smithi099:> curl http://smithi099.front.sepia.ceph.com:8001 2023-05-31T20:56:06.275 INFO:teuthology.orchestra.run.smithi099.stderr: % Total % Received % Xferd Average Speed Time Time Time Current 2023-05-31T20:56:06.276 INFO:teuthology.orchestra.run.smithi099.stderr: Dload Upload Total Spent Left Speed 2023-05-31T20:56:06.276 INFO:teuthology.orchestra.run.smithi099.stderr: 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 100 214 0 214 0 0 208k 0 --:--:-- --:--:-- --:--:-- 208k 2023-05-31T20:56:06.277 INFO:teuthology.orchestra.run.smithi099.stdout:anonymous 2023-05-31T20:56:06.277 DEBUG:rgw_multi.tests:bucket checkpoint: source=a1 target=a2 bucket=pklxkd-60 2023-05-31T20:56:06.278 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'pklxkd-60', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-05-31T20:56:06.278 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'pklxkd-60', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-05-31T20:56:06.279 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket pklxkd-60 --source-zone a1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2023-05-31T20:56:06.372 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2023-05-31T20:56:06.372 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2023-05-31T20:56:06.429 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:56:06.425+0000 7fc789d48780 1 waiting to reach incremental sync.. 2023-05-31T20:56:09.600 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.1 has been restored 2023-05-31T20:56:11.430 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:56:11.425+0000 7fc789d48780 1 waiting to reach incremental sync.. 2023-05-31T20:56:16.430 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:56:16.426+0000 7fc789d48780 1 waiting to reach incremental sync.. 2023-05-31T20:56:21.431 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:56:21.426+0000 7fc789d48780 1 waiting to reach incremental sync.. 2023-05-31T20:56:26.431 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:56:26.426+0000 7fc789d48780 1 waiting to reach incremental sync.. 2023-05-31T20:56:31.432 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:56:31.428+0000 7fc789d48780 1 waiting to reach incremental sync.. 2023-05-31T20:56:36.432 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:56:36.428+0000 7fc789d48780 1 waiting to reach incremental sync.. 2023-05-31T20:56:41.433 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:56:41.429+0000 7fc789d48780 1 waiting to reach incremental sync.. 2023-05-31T20:56:46.433 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:56:46.429+0000 7fc789d48780 1 waiting to reach incremental sync.. 2023-05-31T20:56:51.434 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:56:51.429+0000 7fc789d48780 1 waiting to reach incremental sync.. 2023-05-31T20:56:56.435 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:56:56.430+0000 7fc789d48780 1 waiting to reach incremental sync.. 2023-05-31T20:57:01.435 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:57:01.431+0000 7fc789d48780 1 waiting to reach incremental sync.. 2023-05-31T20:57:06.436 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:57:06.431+0000 7fc789d48780 1 waiting to reach incremental sync.. 2023-05-31T20:57:11.436 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:57:11.431+0000 7fc789d48780 1 waiting to reach incremental sync.. 2023-05-31T20:57:16.437 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:57:16.433+0000 7fc789d48780 1 waiting to reach incremental sync.. 2023-05-31T20:57:21.437 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:57:21.434+0000 7fc789d48780 1 waiting to reach incremental sync.. 2023-05-31T20:57:26.438 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:57:26.434+0000 7fc789d48780 1 waiting to reach incremental sync.. 2023-05-31T20:57:31.438 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:57:31.435+0000 7fc789d48780 1 waiting to reach incremental sync.. 2023-05-31T20:57:36.439 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:57:36.434+0000 7fc789d48780 1 waiting to reach incremental sync.. 2023-05-31T20:57:41.439 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:57:41.435+0000 7fc789d48780 1 waiting to reach incremental sync.. 2023-05-31T20:57:46.440 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:57:46.435+0000 7fc789d48780 1 waiting to reach incremental sync.. 2023-05-31T20:57:51.440 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:57:51.436+0000 7fc789d48780 1 waiting to reach incremental sync.. 2023-05-31T20:57:56.441 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:57:56.437+0000 7fc789d48780 1 waiting to reach incremental sync.. 2023-05-31T20:58:01.441 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:58:01.438+0000 7fc789d48780 1 waiting to reach incremental sync.. 2023-05-31T20:58:06.442 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:58:06.438+0000 7fc789d48780 1 waiting to reach incremental sync.. 2023-05-31T20:58:11.443 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:58:11.439+0000 7fc789d48780 1 waiting to reach incremental sync.. 2023-05-31T20:58:16.444 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:58:16.439+0000 7fc789d48780 1 waiting to reach incremental sync.. 2023-05-31T20:58:21.444 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:58:21.440+0000 7fc789d48780 1 waiting to reach incremental sync.. 2023-05-31T20:58:26.445 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:58:26.440+0000 7fc789d48780 1 waiting to reach incremental sync.. 2023-05-31T20:58:31.446 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:58:31.441+0000 7fc789d48780 1 waiting to reach incremental sync.. 2023-05-31T20:58:36.446 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:58:36.442+0000 7fc789d48780 1 waiting to reach incremental sync.. 2023-05-31T20:58:41.447 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:58:41.443+0000 7fc789d48780 1 waiting to reach incremental sync.. 2023-05-31T20:58:46.447 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:58:46.444+0000 7fc789d48780 1 waiting to reach incremental sync.. 2023-05-31T20:58:51.448 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:58:51.444+0000 7fc789d48780 1 waiting to reach incremental sync.. 2023-05-31T20:58:56.449 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:58:56.445+0000 7fc789d48780 1 waiting to reach incremental sync.. 2023-05-31T20:59:01.449 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:59:01.444+0000 7fc789d48780 1 waiting to reach incremental sync.. 2023-05-31T20:59:06.450 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:59:06.445+0000 7fc789d48780 1 waiting to reach incremental sync.. 2023-05-31T20:59:11.450 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:59:11.446+0000 7fc789d48780 1 waiting to reach incremental sync.. 2023-05-31T20:59:16.451 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:59:16.447+0000 7fc789d48780 1 waiting to reach incremental sync.. 2023-05-31T20:59:21.452 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:59:21.448+0000 7fc789d48780 1 waiting to reach incremental sync.. 2023-05-31T20:59:26.453 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:59:26.449+0000 7fc789d48780 1 waiting to reach incremental sync.. 2023-05-31T20:59:31.454 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:59:31.451+0000 7fc789d48780 1 bucket sync caught up with source: 2023-05-31T20:59:31.455 INFO:teuthology.orchestra.run.smithi099.stderr: local status: [, , , , , , , , 00000000001.50.6, 00000000002.75.6, 00000000002.77.6] 2023-05-31T20:59:31.455 INFO:teuthology.orchestra.run.smithi099.stderr: remote markers: [, , , , , , , , 00000000001.50.6, 00000000002.75.6, 00000000002.77.6] 2023-05-31T20:59:31.455 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:59:31.451+0000 7fc789d48780 0 bucket checkpoint complete 2023-05-31T20:59:31.482 DEBUG:rgw_multi.tests:bucket checkpoint: source=a2 target=a1 bucket=pklxkd-60 2023-05-31T20:59:31.482 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'pklxkd-60', '--source-zone', 'a2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-05-31T20:59:31.483 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'pklxkd-60', '--source-zone', 'a2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-05-31T20:59:31.483 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket pklxkd-60 --source-zone a2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2023-05-31T20:59:31.542 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2023-05-31T20:59:31.542 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2023-05-31T20:59:31.579 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:59:31.575+0000 7f93b1e39780 1 bucket sync caught up with source: 2023-05-31T20:59:31.579 INFO:teuthology.orchestra.run.smithi099.stderr: local status: [, , , , , , , , 00000000001.1717.6, 00000000002.2087.6, 00000000002.2062.6] 2023-05-31T20:59:31.579 INFO:teuthology.orchestra.run.smithi099.stderr: remote markers: [, , , , , , , , 00000000001.1717.6, 00000000002.2087.6, 00000000002.2062.6] 2023-05-31T20:59:31.580 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:59:31.575+0000 7f93b1e39780 0 bucket checkpoint complete 2023-05-31T20:59:31.593 INFO:tasks.rgw_multi.zone_rados:comparing bucket=pklxkd-60 zones={a2, a1} 2023-05-31T20:59:31.594 DEBUG:boto:path=/pklxkd-60/ 2023-05-31T20:59:31.594 DEBUG:boto:auth_path=/pklxkd-60/ 2023-05-31T20:59:31.594 DEBUG:boto:Method: HEAD 2023-05-31T20:59:31.595 DEBUG:boto:Path: /pklxkd-60/ 2023-05-31T20:59:31.595 DEBUG:boto:Data: 2023-05-31T20:59:31.595 DEBUG:boto:Headers: {} 2023-05-31T20:59:31.596 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:59:31.596 DEBUG:boto:Port: 8001 2023-05-31T20:59:31.596 DEBUG:boto:Params: {} 2023-05-31T20:59:31.597 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8001} 2023-05-31T20:59:31.597 DEBUG:boto:Token: None 2023-05-31T20:59:31.598 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:59:31 GMT /pklxkd-60/ 2023-05-31T20:59:31.598 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:r7k9GdlPiqqKJZB57hQFeEej3T0= 2023-05-31T20:59:31.599 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:59:31 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:r7k9GdlPiqqKJZB57hQFeEej3T0=', 'Content-Length': '0'} 2023-05-31T20:59:31.603 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '5'), ('X-RGW-Bytes-Used', '15'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000000221bceae16cdf1e-006477b533-1ec6-a2'), ('Content-Length', '0'), ('Date', 'Wed, 31 May 2023 20:59:31 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:59:31.604 DEBUG:boto:path=/pklxkd-60/ 2023-05-31T20:59:31.604 DEBUG:boto:auth_path=/pklxkd-60/ 2023-05-31T20:59:31.605 DEBUG:boto:Method: HEAD 2023-05-31T20:59:31.605 DEBUG:boto:Path: /pklxkd-60/ 2023-05-31T20:59:31.605 DEBUG:boto:Data: 2023-05-31T20:59:31.606 DEBUG:boto:Headers: {} 2023-05-31T20:59:31.606 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:59:31.606 DEBUG:boto:Port: 8000 2023-05-31T20:59:31.607 DEBUG:boto:Params: {} 2023-05-31T20:59:31.607 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2023-05-31T20:59:31.608 DEBUG:boto:Token: None 2023-05-31T20:59:31.608 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:59:31 GMT /pklxkd-60/ 2023-05-31T20:59:31.608 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:r7k9GdlPiqqKJZB57hQFeEej3T0= 2023-05-31T20:59:31.609 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:59:31 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:r7k9GdlPiqqKJZB57hQFeEej3T0=', 'Content-Length': '0'} 2023-05-31T20:59:31.612 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '5'), ('X-RGW-Bytes-Used', '15'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000b6ddbf3f54f98f9e-006477b533-1630-a1'), ('Content-Length', '0'), ('Date', 'Wed, 31 May 2023 20:59:31 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:59:31.613 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2023-05-31T20:59:31.613 DEBUG:boto:path=/pklxkd-60/ 2023-05-31T20:59:31.614 DEBUG:boto:auth_path=/pklxkd-60/ 2023-05-31T20:59:31.614 DEBUG:boto:path=/pklxkd-60/?versions&max-keys=999 2023-05-31T20:59:31.614 DEBUG:boto:auth_path=/pklxkd-60/?versions&max-keys=999 2023-05-31T20:59:31.615 DEBUG:boto:Method: GET 2023-05-31T20:59:31.615 DEBUG:boto:Path: /pklxkd-60/?versions&max-keys=999 2023-05-31T20:59:31.616 DEBUG:boto:Data: 2023-05-31T20:59:31.616 DEBUG:boto:Headers: {} 2023-05-31T20:59:31.616 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:59:31.617 DEBUG:boto:Port: 8001 2023-05-31T20:59:31.617 DEBUG:boto:Params: {} 2023-05-31T20:59:31.617 DEBUG:boto:Token: None 2023-05-31T20:59:31.618 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:59:31 GMT /pklxkd-60/?versions 2023-05-31T20:59:31.618 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:bm5WTxLWZVshAI9IiAfOEPYwXsI= 2023-05-31T20:59:31.619 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:59:31 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:bm5WTxLWZVshAI9IiAfOEPYwXsI=', 'Content-Length': '0'} 2023-05-31T20:59:31.621 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000002a2cbbed951989b5-006477b533-1ec6-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:59:31 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:59:31.621 DEBUG:boto:b'pklxkd-60999falseobj11nulltrue2023-05-31T20:56:04.921Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj22nulltrue2023-05-31T20:56:04.944Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj33nulltrue2023-05-31T20:56:04.999Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj44nulltrue2023-05-31T20:56:05.021Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj55nulltrue2023-05-31T20:56:05.074Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:59:31.622 DEBUG:tasks.rgw_multi.zone_rados:o=obj11 2023-05-31T20:59:31.623 DEBUG:tasks.rgw_multi.zone_rados:o=obj22 2023-05-31T20:59:31.623 DEBUG:tasks.rgw_multi.zone_rados:o=obj33 2023-05-31T20:59:31.623 DEBUG:tasks.rgw_multi.zone_rados:o=obj44 2023-05-31T20:59:31.624 DEBUG:tasks.rgw_multi.zone_rados:o=obj55 2023-05-31T20:59:31.624 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2023-05-31T20:59:31.624 DEBUG:boto:path=/pklxkd-60/ 2023-05-31T20:59:31.625 DEBUG:boto:auth_path=/pklxkd-60/ 2023-05-31T20:59:31.625 DEBUG:boto:path=/pklxkd-60/?versions&max-keys=999 2023-05-31T20:59:31.625 DEBUG:boto:auth_path=/pklxkd-60/?versions&max-keys=999 2023-05-31T20:59:31.626 DEBUG:boto:Method: GET 2023-05-31T20:59:31.626 DEBUG:boto:Path: /pklxkd-60/?versions&max-keys=999 2023-05-31T20:59:31.626 DEBUG:boto:Data: 2023-05-31T20:59:31.627 DEBUG:boto:Headers: {} 2023-05-31T20:59:31.627 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:59:31.627 DEBUG:boto:Port: 8000 2023-05-31T20:59:31.628 DEBUG:boto:Params: {} 2023-05-31T20:59:31.628 DEBUG:boto:Token: None 2023-05-31T20:59:31.628 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:59:31 GMT /pklxkd-60/?versions 2023-05-31T20:59:31.629 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:bm5WTxLWZVshAI9IiAfOEPYwXsI= 2023-05-31T20:59:31.629 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:59:31 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:bm5WTxLWZVshAI9IiAfOEPYwXsI=', 'Content-Length': '0'} 2023-05-31T20:59:31.631 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000007662932929ce882c-006477b533-1630-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:59:31 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:59:31.632 DEBUG:boto:b'pklxkd-60999falseobj11nulltrue2023-05-31T20:56:04.921Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj22nulltrue2023-05-31T20:56:04.944Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj33nulltrue2023-05-31T20:56:04.999Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj44nulltrue2023-05-31T20:56:05.021Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj55nulltrue2023-05-31T20:56:05.074Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:59:31.633 DEBUG:tasks.rgw_multi.zone_rados:o=obj11 2023-05-31T20:59:31.633 DEBUG:tasks.rgw_multi.zone_rados:o=obj22 2023-05-31T20:59:31.633 DEBUG:tasks.rgw_multi.zone_rados:o=obj33 2023-05-31T20:59:31.634 DEBUG:tasks.rgw_multi.zone_rados:o=obj44 2023-05-31T20:59:31.634 DEBUG:tasks.rgw_multi.zone_rados:o=obj55 2023-05-31T20:59:31.634 DEBUG:boto:path=/pklxkd-60/ 2023-05-31T20:59:31.635 DEBUG:boto:auth_path=/pklxkd-60/ 2023-05-31T20:59:31.635 DEBUG:boto:path=/pklxkd-60/?versions&max-keys=999 2023-05-31T20:59:31.635 DEBUG:boto:auth_path=/pklxkd-60/?versions&max-keys=999 2023-05-31T20:59:31.636 DEBUG:boto:Method: GET 2023-05-31T20:59:31.636 DEBUG:boto:Path: /pklxkd-60/?versions&max-keys=999 2023-05-31T20:59:31.636 DEBUG:boto:Data: 2023-05-31T20:59:31.637 DEBUG:boto:Headers: {} 2023-05-31T20:59:31.637 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:59:31.637 DEBUG:boto:Port: 8001 2023-05-31T20:59:31.638 DEBUG:boto:Params: {} 2023-05-31T20:59:31.638 DEBUG:boto:Token: None 2023-05-31T20:59:31.638 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:59:31 GMT /pklxkd-60/?versions 2023-05-31T20:59:31.639 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:bm5WTxLWZVshAI9IiAfOEPYwXsI= 2023-05-31T20:59:31.639 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:59:31 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:bm5WTxLWZVshAI9IiAfOEPYwXsI=', 'Content-Length': '0'} 2023-05-31T20:59:31.641 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000005450904a9921321d-006477b533-1ec6-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:59:31 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:59:31.642 DEBUG:boto:b'pklxkd-60999falseobj11nulltrue2023-05-31T20:56:04.921Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj22nulltrue2023-05-31T20:56:04.944Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj33nulltrue2023-05-31T20:56:04.999Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj44nulltrue2023-05-31T20:56:05.021Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj55nulltrue2023-05-31T20:56:05.074Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:59:31.642 DEBUG:boto:path=/pklxkd-60/ 2023-05-31T20:59:31.643 DEBUG:boto:auth_path=/pklxkd-60/ 2023-05-31T20:59:31.643 DEBUG:boto:path=/pklxkd-60/?versions&max-keys=999 2023-05-31T20:59:31.643 DEBUG:boto:auth_path=/pklxkd-60/?versions&max-keys=999 2023-05-31T20:59:31.644 DEBUG:boto:Method: GET 2023-05-31T20:59:31.644 DEBUG:boto:Path: /pklxkd-60/?versions&max-keys=999 2023-05-31T20:59:31.645 DEBUG:boto:Data: 2023-05-31T20:59:31.645 DEBUG:boto:Headers: {} 2023-05-31T20:59:31.645 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:59:31.646 DEBUG:boto:Port: 8000 2023-05-31T20:59:31.646 DEBUG:boto:Params: {} 2023-05-31T20:59:31.646 DEBUG:boto:Token: None 2023-05-31T20:59:31.647 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:59:31 GMT /pklxkd-60/?versions 2023-05-31T20:59:31.647 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:bm5WTxLWZVshAI9IiAfOEPYwXsI= 2023-05-31T20:59:31.647 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:59:31 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:bm5WTxLWZVshAI9IiAfOEPYwXsI=', 'Content-Length': '0'} 2023-05-31T20:59:31.650 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000095a5c11893bbbf9f-006477b533-1630-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:59:31 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:59:31.650 DEBUG:boto:b'pklxkd-60999falseobj11nulltrue2023-05-31T20:56:04.921Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj22nulltrue2023-05-31T20:56:04.944Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj33nulltrue2023-05-31T20:56:04.999Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj44nulltrue2023-05-31T20:56:05.021Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj55nulltrue2023-05-31T20:56:05.074Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:59:31.651 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj11 2023-05-31T20:59:31.651 DEBUG:boto:path=/pklxkd-60/obj11 2023-05-31T20:59:31.652 DEBUG:boto:auth_path=/pklxkd-60/obj11 2023-05-31T20:59:31.652 DEBUG:boto:path=/pklxkd-60/obj11?versionId=null 2023-05-31T20:59:31.652 DEBUG:boto:auth_path=/pklxkd-60/obj11?versionId=null 2023-05-31T20:59:31.653 DEBUG:boto:Method: GET 2023-05-31T20:59:31.653 DEBUG:boto:Path: /pklxkd-60/obj11?versionId=null 2023-05-31T20:59:31.653 DEBUG:boto:Data: 2023-05-31T20:59:31.654 DEBUG:boto:Headers: {} 2023-05-31T20:59:31.654 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:59:31.654 DEBUG:boto:Port: 8001 2023-05-31T20:59:31.655 DEBUG:boto:Params: {} 2023-05-31T20:59:31.655 DEBUG:boto:Token: None 2023-05-31T20:59:31.656 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:59:31 GMT /pklxkd-60/obj11?versionId=null 2023-05-31T20:59:31.656 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:J0YTnNQZBW6AEX4w5wJn+pP23zg= 2023-05-31T20:59:31.656 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:59:31 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:J0YTnNQZBW6AEX4w5wJn+pP23zg=', 'Content-Length': '0'} 2023-05-31T20:59:31.658 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:56:04 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000047fba3d093925c33-006477b533-1ec6-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:59:31 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:59:31.659 DEBUG:boto:path=/pklxkd-60/obj11 2023-05-31T20:59:31.659 DEBUG:boto:auth_path=/pklxkd-60/obj11 2023-05-31T20:59:31.660 DEBUG:boto:path=/pklxkd-60/obj11?versionId=null 2023-05-31T20:59:31.660 DEBUG:boto:auth_path=/pklxkd-60/obj11?versionId=null 2023-05-31T20:59:31.660 DEBUG:boto:Method: GET 2023-05-31T20:59:31.661 DEBUG:boto:Path: /pklxkd-60/obj11?versionId=null 2023-05-31T20:59:31.661 DEBUG:boto:Data: 2023-05-31T20:59:31.661 DEBUG:boto:Headers: {} 2023-05-31T20:59:31.662 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:59:31.662 DEBUG:boto:Port: 8000 2023-05-31T20:59:31.663 DEBUG:boto:Params: {} 2023-05-31T20:59:31.663 DEBUG:boto:Token: None 2023-05-31T20:59:31.663 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:59:31 GMT /pklxkd-60/obj11?versionId=null 2023-05-31T20:59:31.664 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:J0YTnNQZBW6AEX4w5wJn+pP23zg= 2023-05-31T20:59:31.664 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:59:31 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:J0YTnNQZBW6AEX4w5wJn+pP23zg=', 'Content-Length': '0'} 2023-05-31T20:59:31.666 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:56:04 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000093e4cec6d5f5fdc4-006477b533-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:59:31 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:59:31.666 DEBUG:boto:path=/pklxkd-60/obj11 2023-05-31T20:59:31.667 DEBUG:boto:auth_path=/pklxkd-60/obj11 2023-05-31T20:59:31.667 DEBUG:boto:path=/pklxkd-60/obj11?versionId=null 2023-05-31T20:59:31.668 DEBUG:boto:auth_path=/pklxkd-60/obj11?versionId=null 2023-05-31T20:59:31.668 DEBUG:boto:Method: HEAD 2023-05-31T20:59:31.668 DEBUG:boto:Path: /pklxkd-60/obj11?versionId=null 2023-05-31T20:59:31.669 DEBUG:boto:Data: 2023-05-31T20:59:31.669 DEBUG:boto:Headers: {} 2023-05-31T20:59:31.669 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:59:31.670 DEBUG:boto:Port: 8001 2023-05-31T20:59:31.670 DEBUG:boto:Params: {} 2023-05-31T20:59:31.670 DEBUG:boto:Token: None 2023-05-31T20:59:31.671 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:59:31 GMT /pklxkd-60/obj11?versionId=null 2023-05-31T20:59:31.671 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:z7GXKxaRIeoOKOzvWqof0tI3lVw= 2023-05-31T20:59:31.671 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:59:31 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:z7GXKxaRIeoOKOzvWqof0tI3lVw=', 'Content-Length': '0'} 2023-05-31T20:59:31.673 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:56:04 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005ff1091c9357e264-006477b533-1ec6-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:59:31 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:59:31.673 DEBUG:boto:path=/pklxkd-60/obj11 2023-05-31T20:59:31.674 DEBUG:boto:auth_path=/pklxkd-60/obj11 2023-05-31T20:59:31.674 DEBUG:boto:path=/pklxkd-60/obj11?versionId=null 2023-05-31T20:59:31.675 DEBUG:boto:auth_path=/pklxkd-60/obj11?versionId=null 2023-05-31T20:59:31.675 DEBUG:boto:Method: HEAD 2023-05-31T20:59:31.675 DEBUG:boto:Path: /pklxkd-60/obj11?versionId=null 2023-05-31T20:59:31.676 DEBUG:boto:Data: 2023-05-31T20:59:31.676 DEBUG:boto:Headers: {} 2023-05-31T20:59:31.676 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:59:31.677 DEBUG:boto:Port: 8000 2023-05-31T20:59:31.677 DEBUG:boto:Params: {} 2023-05-31T20:59:31.677 DEBUG:boto:Token: None 2023-05-31T20:59:31.678 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:59:31 GMT /pklxkd-60/obj11?versionId=null 2023-05-31T20:59:31.678 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:z7GXKxaRIeoOKOzvWqof0tI3lVw= 2023-05-31T20:59:31.678 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:59:31 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:z7GXKxaRIeoOKOzvWqof0tI3lVw=', 'Content-Length': '0'} 2023-05-31T20:59:31.680 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:56:04 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000074ed5c6342e55f71-006477b533-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:59:31 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:59:31.680 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj11 2023-05-31T20:59:31.681 DEBUG:boto:path=/pklxkd-60/obj11 2023-05-31T20:59:31.681 DEBUG:boto:auth_path=/pklxkd-60/obj11 2023-05-31T20:59:31.681 DEBUG:boto:path=/pklxkd-60/obj11?versionId=null 2023-05-31T20:59:31.682 DEBUG:boto:auth_path=/pklxkd-60/obj11?versionId=null 2023-05-31T20:59:31.682 DEBUG:boto:Method: GET 2023-05-31T20:59:31.682 DEBUG:boto:Path: /pklxkd-60/obj11?versionId=null 2023-05-31T20:59:31.683 DEBUG:boto:Data: 2023-05-31T20:59:31.683 DEBUG:boto:Headers: {} 2023-05-31T20:59:31.684 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:59:31.684 DEBUG:boto:Port: 8001 2023-05-31T20:59:31.684 DEBUG:boto:Params: {} 2023-05-31T20:59:31.685 DEBUG:boto:Token: None 2023-05-31T20:59:31.685 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:59:31 GMT /pklxkd-60/obj11?versionId=null 2023-05-31T20:59:31.685 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:J0YTnNQZBW6AEX4w5wJn+pP23zg= 2023-05-31T20:59:31.686 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:59:31 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:J0YTnNQZBW6AEX4w5wJn+pP23zg=', 'Content-Length': '0'} 2023-05-31T20:59:31.687 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:56:04 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000aecb2328f9b8e555-006477b533-1ec6-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:59:31 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:59:31.688 DEBUG:boto:path=/pklxkd-60/obj11 2023-05-31T20:59:31.688 DEBUG:boto:auth_path=/pklxkd-60/obj11 2023-05-31T20:59:31.689 DEBUG:boto:path=/pklxkd-60/obj11?versionId=null 2023-05-31T20:59:31.689 DEBUG:boto:auth_path=/pklxkd-60/obj11?versionId=null 2023-05-31T20:59:31.690 DEBUG:boto:Method: GET 2023-05-31T20:59:31.690 DEBUG:boto:Path: /pklxkd-60/obj11?versionId=null 2023-05-31T20:59:31.690 DEBUG:boto:Data: 2023-05-31T20:59:31.691 DEBUG:boto:Headers: {} 2023-05-31T20:59:31.691 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:59:31.691 DEBUG:boto:Port: 8000 2023-05-31T20:59:31.692 DEBUG:boto:Params: {} 2023-05-31T20:59:31.692 DEBUG:boto:Token: None 2023-05-31T20:59:31.692 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:59:31 GMT /pklxkd-60/obj11?versionId=null 2023-05-31T20:59:31.693 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:J0YTnNQZBW6AEX4w5wJn+pP23zg= 2023-05-31T20:59:31.693 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:59:31 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:J0YTnNQZBW6AEX4w5wJn+pP23zg=', 'Content-Length': '0'} 2023-05-31T20:59:31.695 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:56:04 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000fd70030993f316a-006477b533-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:59:31 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:59:31.696 DEBUG:boto:path=/pklxkd-60/ 2023-05-31T20:59:31.696 DEBUG:boto:auth_path=/pklxkd-60/ 2023-05-31T20:59:31.696 DEBUG:boto:path=/pklxkd-60/?prefix=obj11 2023-05-31T20:59:31.697 DEBUG:boto:auth_path=/pklxkd-60/?prefix=obj11 2023-05-31T20:59:31.697 DEBUG:boto:Method: GET 2023-05-31T20:59:31.698 DEBUG:boto:Path: /pklxkd-60/?prefix=obj11 2023-05-31T20:59:31.698 DEBUG:boto:Data: 2023-05-31T20:59:31.698 DEBUG:boto:Headers: {} 2023-05-31T20:59:31.699 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:59:31.699 DEBUG:boto:Port: 8001 2023-05-31T20:59:31.699 DEBUG:boto:Params: {} 2023-05-31T20:59:31.700 DEBUG:boto:Token: None 2023-05-31T20:59:31.700 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:59:31 GMT /pklxkd-60/ 2023-05-31T20:59:31.701 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:HeUfiHGYRLKd5l6XdeLlk05+4W8= 2023-05-31T20:59:31.701 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:59:31 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:HeUfiHGYRLKd5l6XdeLlk05+4W8=', 'Content-Length': '0'} 2023-05-31T20:59:31.703 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000004075eb04795a43de-006477b533-1ec6-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:59:31 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:59:31.703 DEBUG:boto:b'pklxkd-60obj111000falseobj112023-05-31T20:56:04.921Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:59:31.704 DEBUG:boto:path=/pklxkd-60/ 2023-05-31T20:59:31.705 DEBUG:boto:auth_path=/pklxkd-60/ 2023-05-31T20:59:31.705 DEBUG:boto:path=/pklxkd-60/?prefix=obj11 2023-05-31T20:59:31.705 DEBUG:boto:auth_path=/pklxkd-60/?prefix=obj11 2023-05-31T20:59:31.706 DEBUG:boto:Method: GET 2023-05-31T20:59:31.706 DEBUG:boto:Path: /pklxkd-60/?prefix=obj11 2023-05-31T20:59:31.706 DEBUG:boto:Data: 2023-05-31T20:59:31.707 DEBUG:boto:Headers: {} 2023-05-31T20:59:31.707 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:59:31.707 DEBUG:boto:Port: 8000 2023-05-31T20:59:31.708 DEBUG:boto:Params: {} 2023-05-31T20:59:31.708 DEBUG:boto:Token: None 2023-05-31T20:59:31.709 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:59:31 GMT /pklxkd-60/ 2023-05-31T20:59:31.709 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:HeUfiHGYRLKd5l6XdeLlk05+4W8= 2023-05-31T20:59:31.709 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:59:31 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:HeUfiHGYRLKd5l6XdeLlk05+4W8=', 'Content-Length': '0'} 2023-05-31T20:59:31.711 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000fb064a30c7cc58c9-006477b533-1630-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:59:31 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:59:31.712 DEBUG:boto:b'pklxkd-60obj111000falseobj112023-05-31T20:56:04.921Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:59:31.712 DEBUG:boto:path=/pklxkd-60/obj11 2023-05-31T20:59:31.713 DEBUG:boto:auth_path=/pklxkd-60/obj11 2023-05-31T20:59:31.713 DEBUG:boto:Method: HEAD 2023-05-31T20:59:31.713 DEBUG:boto:Path: /pklxkd-60/obj11 2023-05-31T20:59:31.714 DEBUG:boto:Data: 2023-05-31T20:59:31.714 DEBUG:boto:Headers: {} 2023-05-31T20:59:31.714 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:59:31.715 DEBUG:boto:Port: 8001 2023-05-31T20:59:31.715 DEBUG:boto:Params: {} 2023-05-31T20:59:31.715 DEBUG:boto:Token: None 2023-05-31T20:59:31.716 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:59:31 GMT /pklxkd-60/obj11 2023-05-31T20:59:31.716 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:cY3KAzXqKm5z+0WzE6SxjxDnxEs= 2023-05-31T20:59:31.717 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:59:31 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:cY3KAzXqKm5z+0WzE6SxjxDnxEs=', 'Content-Length': '0'} 2023-05-31T20:59:31.718 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:56:04 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000282713c79a6f536-006477b533-1ec6-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:59:31 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:59:31.719 DEBUG:boto:path=/pklxkd-60/obj11 2023-05-31T20:59:31.719 DEBUG:boto:auth_path=/pklxkd-60/obj11 2023-05-31T20:59:31.719 DEBUG:boto:Method: HEAD 2023-05-31T20:59:31.720 DEBUG:boto:Path: /pklxkd-60/obj11 2023-05-31T20:59:31.720 DEBUG:boto:Data: 2023-05-31T20:59:31.720 DEBUG:boto:Headers: {} 2023-05-31T20:59:31.721 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:59:31.721 DEBUG:boto:Port: 8000 2023-05-31T20:59:31.721 DEBUG:boto:Params: {} 2023-05-31T20:59:31.722 DEBUG:boto:Token: None 2023-05-31T20:59:31.722 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:59:31 GMT /pklxkd-60/obj11 2023-05-31T20:59:31.723 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:cY3KAzXqKm5z+0WzE6SxjxDnxEs= 2023-05-31T20:59:31.723 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:59:31 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:cY3KAzXqKm5z+0WzE6SxjxDnxEs=', 'Content-Length': '0'} 2023-05-31T20:59:31.724 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:56:04 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000148368532c5ef7ae-006477b533-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:59:31 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:59:31.725 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj11 2023-05-31T20:59:31.725 DEBUG:boto:path=/pklxkd-60/obj11 2023-05-31T20:59:31.726 DEBUG:boto:auth_path=/pklxkd-60/obj11 2023-05-31T20:59:31.726 DEBUG:boto:Method: GET 2023-05-31T20:59:31.726 DEBUG:boto:Path: /pklxkd-60/obj11 2023-05-31T20:59:31.727 DEBUG:boto:Data: 2023-05-31T20:59:31.727 DEBUG:boto:Headers: {} 2023-05-31T20:59:31.727 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:59:31.728 DEBUG:boto:Port: 8001 2023-05-31T20:59:31.728 DEBUG:boto:Params: {} 2023-05-31T20:59:31.728 DEBUG:boto:Token: None 2023-05-31T20:59:31.729 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:59:31 GMT /pklxkd-60/obj11 2023-05-31T20:59:31.729 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:/K1vyHVukr/DCbJoeEv3Nvjyn48= 2023-05-31T20:59:31.730 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:59:31 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:/K1vyHVukr/DCbJoeEv3Nvjyn48=', 'Content-Length': '0'} 2023-05-31T20:59:31.731 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:56:04 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a54d918f79eb38ba-006477b533-1ec6-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:59:31 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:59:31.732 DEBUG:boto:path=/pklxkd-60/obj11 2023-05-31T20:59:31.732 DEBUG:boto:auth_path=/pklxkd-60/obj11 2023-05-31T20:59:31.733 DEBUG:boto:Method: GET 2023-05-31T20:59:31.733 DEBUG:boto:Path: /pklxkd-60/obj11 2023-05-31T20:59:31.733 DEBUG:boto:Data: 2023-05-31T20:59:31.734 DEBUG:boto:Headers: {} 2023-05-31T20:59:31.734 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:59:31.734 DEBUG:boto:Port: 8000 2023-05-31T20:59:31.735 DEBUG:boto:Params: {} 2023-05-31T20:59:31.735 DEBUG:boto:Token: None 2023-05-31T20:59:31.735 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:59:31 GMT /pklxkd-60/obj11 2023-05-31T20:59:31.736 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:/K1vyHVukr/DCbJoeEv3Nvjyn48= 2023-05-31T20:59:31.736 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:59:31 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:/K1vyHVukr/DCbJoeEv3Nvjyn48=', 'Content-Length': '0'} 2023-05-31T20:59:31.738 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:56:04 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000093ffe391396d9fe9-006477b533-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:59:31 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:59:31.739 DEBUG:boto:path=/pklxkd-60/ 2023-05-31T20:59:31.739 DEBUG:boto:auth_path=/pklxkd-60/ 2023-05-31T20:59:31.740 DEBUG:boto:path=/pklxkd-60/?prefix=obj11 2023-05-31T20:59:31.740 DEBUG:boto:auth_path=/pklxkd-60/?prefix=obj11 2023-05-31T20:59:31.740 DEBUG:boto:Method: GET 2023-05-31T20:59:31.741 DEBUG:boto:Path: /pklxkd-60/?prefix=obj11 2023-05-31T20:59:31.741 DEBUG:boto:Data: 2023-05-31T20:59:31.741 DEBUG:boto:Headers: {} 2023-05-31T20:59:31.742 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:59:31.742 DEBUG:boto:Port: 8001 2023-05-31T20:59:31.742 DEBUG:boto:Params: {} 2023-05-31T20:59:31.743 DEBUG:boto:Token: None 2023-05-31T20:59:31.743 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:59:31 GMT /pklxkd-60/ 2023-05-31T20:59:31.744 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:HeUfiHGYRLKd5l6XdeLlk05+4W8= 2023-05-31T20:59:31.744 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:59:31 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:HeUfiHGYRLKd5l6XdeLlk05+4W8=', 'Content-Length': '0'} 2023-05-31T20:59:31.746 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c3ce5a4231d93669-006477b533-1ec6-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:59:31 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:59:31.747 DEBUG:boto:b'pklxkd-60obj111000falseobj112023-05-31T20:56:04.921Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:59:31.747 DEBUG:boto:path=/pklxkd-60/ 2023-05-31T20:59:31.748 DEBUG:boto:auth_path=/pklxkd-60/ 2023-05-31T20:59:31.748 DEBUG:boto:path=/pklxkd-60/?prefix=obj11 2023-05-31T20:59:31.748 DEBUG:boto:auth_path=/pklxkd-60/?prefix=obj11 2023-05-31T20:59:31.749 DEBUG:boto:Method: GET 2023-05-31T20:59:31.749 DEBUG:boto:Path: /pklxkd-60/?prefix=obj11 2023-05-31T20:59:31.749 DEBUG:boto:Data: 2023-05-31T20:59:31.750 DEBUG:boto:Headers: {} 2023-05-31T20:59:31.750 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:59:31.750 DEBUG:boto:Port: 8000 2023-05-31T20:59:31.751 DEBUG:boto:Params: {} 2023-05-31T20:59:31.751 DEBUG:boto:Token: None 2023-05-31T20:59:31.752 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:59:31 GMT /pklxkd-60/ 2023-05-31T20:59:31.752 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:HeUfiHGYRLKd5l6XdeLlk05+4W8= 2023-05-31T20:59:31.752 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:59:31 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:HeUfiHGYRLKd5l6XdeLlk05+4W8=', 'Content-Length': '0'} 2023-05-31T20:59:31.754 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000024e0b73a26b84de3-006477b533-1630-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:59:31 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:59:31.755 DEBUG:boto:b'pklxkd-60obj111000falseobj112023-05-31T20:56:04.921Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:59:31.755 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj22 2023-05-31T20:59:31.756 DEBUG:boto:path=/pklxkd-60/obj22 2023-05-31T20:59:31.756 DEBUG:boto:auth_path=/pklxkd-60/obj22 2023-05-31T20:59:31.756 DEBUG:boto:path=/pklxkd-60/obj22?versionId=null 2023-05-31T20:59:31.757 DEBUG:boto:auth_path=/pklxkd-60/obj22?versionId=null 2023-05-31T20:59:31.757 DEBUG:boto:Method: GET 2023-05-31T20:59:31.758 DEBUG:boto:Path: /pklxkd-60/obj22?versionId=null 2023-05-31T20:59:31.758 DEBUG:boto:Data: 2023-05-31T20:59:31.758 DEBUG:boto:Headers: {} 2023-05-31T20:59:31.759 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:59:31.759 DEBUG:boto:Port: 8001 2023-05-31T20:59:31.759 DEBUG:boto:Params: {} 2023-05-31T20:59:31.760 DEBUG:boto:Token: None 2023-05-31T20:59:31.760 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:59:31 GMT /pklxkd-60/obj22?versionId=null 2023-05-31T20:59:31.761 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:T07bf3HWCfBBH8Q8Qx0MxhqV2jQ= 2023-05-31T20:59:31.761 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:59:31 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:T07bf3HWCfBBH8Q8Qx0MxhqV2jQ=', 'Content-Length': '0'} 2023-05-31T20:59:31.762 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:56:04 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000042343c0eddd61231-006477b533-1ec6-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:59:31 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:59:31.763 DEBUG:boto:path=/pklxkd-60/obj22 2023-05-31T20:59:31.763 DEBUG:boto:auth_path=/pklxkd-60/obj22 2023-05-31T20:59:31.764 DEBUG:boto:path=/pklxkd-60/obj22?versionId=null 2023-05-31T20:59:31.764 DEBUG:boto:auth_path=/pklxkd-60/obj22?versionId=null 2023-05-31T20:59:31.765 DEBUG:boto:Method: GET 2023-05-31T20:59:31.765 DEBUG:boto:Path: /pklxkd-60/obj22?versionId=null 2023-05-31T20:59:31.765 DEBUG:boto:Data: 2023-05-31T20:59:31.766 DEBUG:boto:Headers: {} 2023-05-31T20:59:31.766 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:59:31.766 DEBUG:boto:Port: 8000 2023-05-31T20:59:31.767 DEBUG:boto:Params: {} 2023-05-31T20:59:31.767 DEBUG:boto:Token: None 2023-05-31T20:59:31.767 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:59:31 GMT /pklxkd-60/obj22?versionId=null 2023-05-31T20:59:31.768 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:T07bf3HWCfBBH8Q8Qx0MxhqV2jQ= 2023-05-31T20:59:31.768 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:59:31 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:T07bf3HWCfBBH8Q8Qx0MxhqV2jQ=', 'Content-Length': '0'} 2023-05-31T20:59:31.770 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:56:04 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000fbfd08ae5299a28d-006477b533-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:59:31 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:59:31.770 DEBUG:boto:path=/pklxkd-60/obj22 2023-05-31T20:59:31.771 DEBUG:boto:auth_path=/pklxkd-60/obj22 2023-05-31T20:59:31.771 DEBUG:boto:path=/pklxkd-60/obj22?versionId=null 2023-05-31T20:59:31.771 DEBUG:boto:auth_path=/pklxkd-60/obj22?versionId=null 2023-05-31T20:59:31.772 DEBUG:boto:Method: HEAD 2023-05-31T20:59:31.772 DEBUG:boto:Path: /pklxkd-60/obj22?versionId=null 2023-05-31T20:59:31.772 DEBUG:boto:Data: 2023-05-31T20:59:31.773 DEBUG:boto:Headers: {} 2023-05-31T20:59:31.773 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:59:31.773 DEBUG:boto:Port: 8001 2023-05-31T20:59:31.774 DEBUG:boto:Params: {} 2023-05-31T20:59:31.774 DEBUG:boto:Token: None 2023-05-31T20:59:31.775 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:59:31 GMT /pklxkd-60/obj22?versionId=null 2023-05-31T20:59:31.775 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:nVLEL7iRCxK3KuA95ex8+1HwYR0= 2023-05-31T20:59:31.775 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:59:31 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:nVLEL7iRCxK3KuA95ex8+1HwYR0=', 'Content-Length': '0'} 2023-05-31T20:59:31.777 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:56:04 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f8a66d6b0a1d331d-006477b533-1ec6-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:59:31 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:59:31.778 DEBUG:boto:path=/pklxkd-60/obj22 2023-05-31T20:59:31.778 DEBUG:boto:auth_path=/pklxkd-60/obj22 2023-05-31T20:59:31.778 DEBUG:boto:path=/pklxkd-60/obj22?versionId=null 2023-05-31T20:59:31.779 DEBUG:boto:auth_path=/pklxkd-60/obj22?versionId=null 2023-05-31T20:59:31.779 DEBUG:boto:Method: HEAD 2023-05-31T20:59:31.779 DEBUG:boto:Path: /pklxkd-60/obj22?versionId=null 2023-05-31T20:59:31.780 DEBUG:boto:Data: 2023-05-31T20:59:31.780 DEBUG:boto:Headers: {} 2023-05-31T20:59:31.780 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:59:31.781 DEBUG:boto:Port: 8000 2023-05-31T20:59:31.781 DEBUG:boto:Params: {} 2023-05-31T20:59:31.781 DEBUG:boto:Token: None 2023-05-31T20:59:31.782 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:59:31 GMT /pklxkd-60/obj22?versionId=null 2023-05-31T20:59:31.782 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:nVLEL7iRCxK3KuA95ex8+1HwYR0= 2023-05-31T20:59:31.782 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:59:31 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:nVLEL7iRCxK3KuA95ex8+1HwYR0=', 'Content-Length': '0'} 2023-05-31T20:59:31.784 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:56:04 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f5f7ff810172acb9-006477b533-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:59:31 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:59:31.785 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj22 2023-05-31T20:59:31.785 DEBUG:boto:path=/pklxkd-60/obj22 2023-05-31T20:59:31.785 DEBUG:boto:auth_path=/pklxkd-60/obj22 2023-05-31T20:59:31.786 DEBUG:boto:path=/pklxkd-60/obj22?versionId=null 2023-05-31T20:59:31.786 DEBUG:boto:auth_path=/pklxkd-60/obj22?versionId=null 2023-05-31T20:59:31.786 DEBUG:boto:Method: GET 2023-05-31T20:59:31.787 DEBUG:boto:Path: /pklxkd-60/obj22?versionId=null 2023-05-31T20:59:31.787 DEBUG:boto:Data: 2023-05-31T20:59:31.787 DEBUG:boto:Headers: {} 2023-05-31T20:59:31.788 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:59:31.788 DEBUG:boto:Port: 8001 2023-05-31T20:59:31.789 DEBUG:boto:Params: {} 2023-05-31T20:59:31.789 DEBUG:boto:Token: None 2023-05-31T20:59:31.789 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:59:31 GMT /pklxkd-60/obj22?versionId=null 2023-05-31T20:59:31.790 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:T07bf3HWCfBBH8Q8Qx0MxhqV2jQ= 2023-05-31T20:59:31.790 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:59:31 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:T07bf3HWCfBBH8Q8Qx0MxhqV2jQ=', 'Content-Length': '0'} 2023-05-31T20:59:31.792 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:56:04 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000b63b2554f1358e6-006477b533-1ec6-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:59:31 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:59:31.832 DEBUG:boto:path=/pklxkd-60/obj22 2023-05-31T20:59:31.833 DEBUG:boto:auth_path=/pklxkd-60/obj22 2023-05-31T20:59:31.833 DEBUG:boto:path=/pklxkd-60/obj22?versionId=null 2023-05-31T20:59:31.833 DEBUG:boto:auth_path=/pklxkd-60/obj22?versionId=null 2023-05-31T20:59:31.834 DEBUG:boto:Method: GET 2023-05-31T20:59:31.834 DEBUG:boto:Path: /pklxkd-60/obj22?versionId=null 2023-05-31T20:59:31.834 DEBUG:boto:Data: 2023-05-31T20:59:31.835 DEBUG:boto:Headers: {} 2023-05-31T20:59:31.835 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:59:31.835 DEBUG:boto:Port: 8000 2023-05-31T20:59:31.836 DEBUG:boto:Params: {} 2023-05-31T20:59:31.836 DEBUG:boto:Token: None 2023-05-31T20:59:31.837 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:59:31 GMT /pklxkd-60/obj22?versionId=null 2023-05-31T20:59:31.837 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:T07bf3HWCfBBH8Q8Qx0MxhqV2jQ= 2023-05-31T20:59:31.837 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:59:31 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:T07bf3HWCfBBH8Q8Qx0MxhqV2jQ=', 'Content-Length': '0'} 2023-05-31T20:59:31.839 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:56:04 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b65486487528b2ab-006477b533-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:59:31 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:59:31.840 DEBUG:boto:path=/pklxkd-60/ 2023-05-31T20:59:31.840 DEBUG:boto:auth_path=/pklxkd-60/ 2023-05-31T20:59:31.840 DEBUG:boto:path=/pklxkd-60/?prefix=obj22 2023-05-31T20:59:31.841 DEBUG:boto:auth_path=/pklxkd-60/?prefix=obj22 2023-05-31T20:59:31.841 DEBUG:boto:Method: GET 2023-05-31T20:59:31.842 DEBUG:boto:Path: /pklxkd-60/?prefix=obj22 2023-05-31T20:59:31.842 DEBUG:boto:Data: 2023-05-31T20:59:31.842 DEBUG:boto:Headers: {} 2023-05-31T20:59:31.843 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:59:31.843 DEBUG:boto:Port: 8001 2023-05-31T20:59:31.843 DEBUG:boto:Params: {} 2023-05-31T20:59:31.844 DEBUG:boto:Token: None 2023-05-31T20:59:31.844 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:59:31 GMT /pklxkd-60/ 2023-05-31T20:59:31.845 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:HeUfiHGYRLKd5l6XdeLlk05+4W8= 2023-05-31T20:59:31.845 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:59:31 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:HeUfiHGYRLKd5l6XdeLlk05+4W8=', 'Content-Length': '0'} 2023-05-31T20:59:31.847 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000002efd49ea320acf22-006477b533-1ec6-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:59:31 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:59:31.847 DEBUG:boto:b'pklxkd-60obj221000falseobj222023-05-31T20:56:04.944Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:59:31.848 DEBUG:boto:path=/pklxkd-60/ 2023-05-31T20:59:31.848 DEBUG:boto:auth_path=/pklxkd-60/ 2023-05-31T20:59:31.849 DEBUG:boto:path=/pklxkd-60/?prefix=obj22 2023-05-31T20:59:31.849 DEBUG:boto:auth_path=/pklxkd-60/?prefix=obj22 2023-05-31T20:59:31.849 DEBUG:boto:Method: GET 2023-05-31T20:59:31.850 DEBUG:boto:Path: /pklxkd-60/?prefix=obj22 2023-05-31T20:59:31.850 DEBUG:boto:Data: 2023-05-31T20:59:31.850 DEBUG:boto:Headers: {} 2023-05-31T20:59:31.851 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:59:31.851 DEBUG:boto:Port: 8000 2023-05-31T20:59:31.851 DEBUG:boto:Params: {} 2023-05-31T20:59:31.852 DEBUG:boto:Token: None 2023-05-31T20:59:31.852 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:59:31 GMT /pklxkd-60/ 2023-05-31T20:59:31.853 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:HeUfiHGYRLKd5l6XdeLlk05+4W8= 2023-05-31T20:59:31.853 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:59:31 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:HeUfiHGYRLKd5l6XdeLlk05+4W8=', 'Content-Length': '0'} 2023-05-31T20:59:31.855 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000008fe93bbaf2fe5436-006477b533-1630-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:59:31 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:59:31.856 DEBUG:boto:b'pklxkd-60obj221000falseobj222023-05-31T20:56:04.944Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:59:31.856 DEBUG:boto:path=/pklxkd-60/obj22 2023-05-31T20:59:31.856 DEBUG:boto:auth_path=/pklxkd-60/obj22 2023-05-31T20:59:31.857 DEBUG:boto:Method: HEAD 2023-05-31T20:59:31.857 DEBUG:boto:Path: /pklxkd-60/obj22 2023-05-31T20:59:31.858 DEBUG:boto:Data: 2023-05-31T20:59:31.858 DEBUG:boto:Headers: {} 2023-05-31T20:59:31.858 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:59:31.858 DEBUG:boto:Port: 8001 2023-05-31T20:59:31.859 DEBUG:boto:Params: {} 2023-05-31T20:59:31.859 DEBUG:boto:Token: None 2023-05-31T20:59:31.860 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:59:31 GMT /pklxkd-60/obj22 2023-05-31T20:59:31.860 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:+n6FnHuA2HCeBVUibgxw7eAcdFY= 2023-05-31T20:59:31.860 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:59:31 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:+n6FnHuA2HCeBVUibgxw7eAcdFY=', 'Content-Length': '0'} 2023-05-31T20:59:31.862 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:56:04 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007e945d4c82a2e7b1-006477b533-1ec6-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:59:31 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:59:31.862 DEBUG:boto:path=/pklxkd-60/obj22 2023-05-31T20:59:31.863 DEBUG:boto:auth_path=/pklxkd-60/obj22 2023-05-31T20:59:31.863 DEBUG:boto:Method: HEAD 2023-05-31T20:59:31.863 DEBUG:boto:Path: /pklxkd-60/obj22 2023-05-31T20:59:31.864 DEBUG:boto:Data: 2023-05-31T20:59:31.864 DEBUG:boto:Headers: {} 2023-05-31T20:59:31.864 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:59:31.865 DEBUG:boto:Port: 8000 2023-05-31T20:59:31.865 DEBUG:boto:Params: {} 2023-05-31T20:59:31.865 DEBUG:boto:Token: None 2023-05-31T20:59:31.866 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:59:31 GMT /pklxkd-60/obj22 2023-05-31T20:59:31.866 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:+n6FnHuA2HCeBVUibgxw7eAcdFY= 2023-05-31T20:59:31.867 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:59:31 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:+n6FnHuA2HCeBVUibgxw7eAcdFY=', 'Content-Length': '0'} 2023-05-31T20:59:31.868 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:56:04 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000dcbd97d553839720-006477b533-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:59:31 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:59:31.869 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj22 2023-05-31T20:59:31.869 DEBUG:boto:path=/pklxkd-60/obj22 2023-05-31T20:59:31.869 DEBUG:boto:auth_path=/pklxkd-60/obj22 2023-05-31T20:59:31.870 DEBUG:boto:Method: GET 2023-05-31T20:59:31.870 DEBUG:boto:Path: /pklxkd-60/obj22 2023-05-31T20:59:31.870 DEBUG:boto:Data: 2023-05-31T20:59:31.871 DEBUG:boto:Headers: {} 2023-05-31T20:59:31.871 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:59:31.871 DEBUG:boto:Port: 8001 2023-05-31T20:59:31.872 DEBUG:boto:Params: {} 2023-05-31T20:59:31.872 DEBUG:boto:Token: None 2023-05-31T20:59:31.873 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:59:31 GMT /pklxkd-60/obj22 2023-05-31T20:59:31.873 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:64cy0BvxdvDrPUej+20qv25gSAs= 2023-05-31T20:59:31.873 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:59:31 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:64cy0BvxdvDrPUej+20qv25gSAs=', 'Content-Length': '0'} 2023-05-31T20:59:31.875 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:56:04 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000057a1c8c81e83c24b-006477b533-1ec6-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:59:31 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:59:31.876 DEBUG:boto:path=/pklxkd-60/obj22 2023-05-31T20:59:31.876 DEBUG:boto:auth_path=/pklxkd-60/obj22 2023-05-31T20:59:31.876 DEBUG:boto:Method: GET 2023-05-31T20:59:31.877 DEBUG:boto:Path: /pklxkd-60/obj22 2023-05-31T20:59:31.877 DEBUG:boto:Data: 2023-05-31T20:59:31.878 DEBUG:boto:Headers: {} 2023-05-31T20:59:31.878 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:59:31.878 DEBUG:boto:Port: 8000 2023-05-31T20:59:31.879 DEBUG:boto:Params: {} 2023-05-31T20:59:31.879 DEBUG:boto:Token: None 2023-05-31T20:59:31.879 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:59:31 GMT /pklxkd-60/obj22 2023-05-31T20:59:31.880 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:64cy0BvxdvDrPUej+20qv25gSAs= 2023-05-31T20:59:31.880 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:59:31 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:64cy0BvxdvDrPUej+20qv25gSAs=', 'Content-Length': '0'} 2023-05-31T20:59:31.882 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:56:04 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001d260b7864de59e1-006477b533-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:59:31 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:59:31.882 DEBUG:boto:path=/pklxkd-60/ 2023-05-31T20:59:31.883 DEBUG:boto:auth_path=/pklxkd-60/ 2023-05-31T20:59:31.883 DEBUG:boto:path=/pklxkd-60/?prefix=obj22 2023-05-31T20:59:31.883 DEBUG:boto:auth_path=/pklxkd-60/?prefix=obj22 2023-05-31T20:59:31.884 DEBUG:boto:Method: GET 2023-05-31T20:59:31.884 DEBUG:boto:Path: /pklxkd-60/?prefix=obj22 2023-05-31T20:59:31.885 DEBUG:boto:Data: 2023-05-31T20:59:31.885 DEBUG:boto:Headers: {} 2023-05-31T20:59:31.885 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:59:31.886 DEBUG:boto:Port: 8001 2023-05-31T20:59:31.886 DEBUG:boto:Params: {} 2023-05-31T20:59:31.886 DEBUG:boto:Token: None 2023-05-31T20:59:31.887 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:59:31 GMT /pklxkd-60/ 2023-05-31T20:59:31.887 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:HeUfiHGYRLKd5l6XdeLlk05+4W8= 2023-05-31T20:59:31.888 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:59:31 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:HeUfiHGYRLKd5l6XdeLlk05+4W8=', 'Content-Length': '0'} 2023-05-31T20:59:31.890 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000004fb4c6308bb0460-006477b533-1ec6-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:59:31 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:59:31.892 DEBUG:boto:b'pklxkd-60obj221000falseobj222023-05-31T20:56:04.944Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:59:31.893 DEBUG:boto:path=/pklxkd-60/ 2023-05-31T20:59:31.893 DEBUG:boto:auth_path=/pklxkd-60/ 2023-05-31T20:59:31.894 DEBUG:boto:path=/pklxkd-60/?prefix=obj22 2023-05-31T20:59:31.894 DEBUG:boto:auth_path=/pklxkd-60/?prefix=obj22 2023-05-31T20:59:31.894 DEBUG:boto:Method: GET 2023-05-31T20:59:31.895 DEBUG:boto:Path: /pklxkd-60/?prefix=obj22 2023-05-31T20:59:31.895 DEBUG:boto:Data: 2023-05-31T20:59:31.895 DEBUG:boto:Headers: {} 2023-05-31T20:59:31.896 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:59:31.896 DEBUG:boto:Port: 8000 2023-05-31T20:59:31.896 DEBUG:boto:Params: {} 2023-05-31T20:59:31.897 DEBUG:boto:Token: None 2023-05-31T20:59:31.897 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:59:31 GMT /pklxkd-60/ 2023-05-31T20:59:31.898 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:HeUfiHGYRLKd5l6XdeLlk05+4W8= 2023-05-31T20:59:31.898 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:59:31 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:HeUfiHGYRLKd5l6XdeLlk05+4W8=', 'Content-Length': '0'} 2023-05-31T20:59:31.900 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000883b8a8159186391-006477b533-1630-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:59:31 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:59:31.901 DEBUG:boto:b'pklxkd-60obj221000falseobj222023-05-31T20:56:04.944Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:59:31.901 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj33 2023-05-31T20:59:31.902 DEBUG:boto:path=/pklxkd-60/obj33 2023-05-31T20:59:31.902 DEBUG:boto:auth_path=/pklxkd-60/obj33 2023-05-31T20:59:31.902 DEBUG:boto:path=/pklxkd-60/obj33?versionId=null 2023-05-31T20:59:31.903 DEBUG:boto:auth_path=/pklxkd-60/obj33?versionId=null 2023-05-31T20:59:31.903 DEBUG:boto:Method: GET 2023-05-31T20:59:31.904 DEBUG:boto:Path: /pklxkd-60/obj33?versionId=null 2023-05-31T20:59:31.904 DEBUG:boto:Data: 2023-05-31T20:59:31.904 DEBUG:boto:Headers: {} 2023-05-31T20:59:31.905 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:59:31.905 DEBUG:boto:Port: 8001 2023-05-31T20:59:31.905 DEBUG:boto:Params: {} 2023-05-31T20:59:31.906 DEBUG:boto:Token: None 2023-05-31T20:59:31.906 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:59:31 GMT /pklxkd-60/obj33?versionId=null 2023-05-31T20:59:31.907 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:aYdJ5rvQ74llibTeu9jar2BzVj0= 2023-05-31T20:59:31.907 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:59:31 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:aYdJ5rvQ74llibTeu9jar2BzVj0=', 'Content-Length': '0'} 2023-05-31T20:59:31.908 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:56:04 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ccbf480a112b5365-006477b533-1ec6-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:59:31 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:59:31.909 DEBUG:boto:path=/pklxkd-60/obj33 2023-05-31T20:59:31.910 DEBUG:boto:auth_path=/pklxkd-60/obj33 2023-05-31T20:59:31.910 DEBUG:boto:path=/pklxkd-60/obj33?versionId=null 2023-05-31T20:59:31.910 DEBUG:boto:auth_path=/pklxkd-60/obj33?versionId=null 2023-05-31T20:59:31.911 DEBUG:boto:Method: GET 2023-05-31T20:59:31.911 DEBUG:boto:Path: /pklxkd-60/obj33?versionId=null 2023-05-31T20:59:31.911 DEBUG:boto:Data: 2023-05-31T20:59:31.912 DEBUG:boto:Headers: {} 2023-05-31T20:59:31.912 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:59:31.912 DEBUG:boto:Port: 8000 2023-05-31T20:59:31.913 DEBUG:boto:Params: {} 2023-05-31T20:59:31.913 DEBUG:boto:Token: None 2023-05-31T20:59:31.914 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:59:31 GMT /pklxkd-60/obj33?versionId=null 2023-05-31T20:59:31.914 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:aYdJ5rvQ74llibTeu9jar2BzVj0= 2023-05-31T20:59:31.914 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:59:31 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:aYdJ5rvQ74llibTeu9jar2BzVj0=', 'Content-Length': '0'} 2023-05-31T20:59:31.916 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:56:04 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003ef1e04e77b3c60d-006477b533-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:59:31 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:59:31.960 DEBUG:boto:path=/pklxkd-60/obj33 2023-05-31T20:59:31.961 DEBUG:boto:auth_path=/pklxkd-60/obj33 2023-05-31T20:59:31.961 DEBUG:boto:path=/pklxkd-60/obj33?versionId=null 2023-05-31T20:59:31.961 DEBUG:boto:auth_path=/pklxkd-60/obj33?versionId=null 2023-05-31T20:59:31.962 DEBUG:boto:Method: HEAD 2023-05-31T20:59:31.962 DEBUG:boto:Path: /pklxkd-60/obj33?versionId=null 2023-05-31T20:59:31.962 DEBUG:boto:Data: 2023-05-31T20:59:31.963 DEBUG:boto:Headers: {} 2023-05-31T20:59:31.963 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:59:31.963 DEBUG:boto:Port: 8001 2023-05-31T20:59:31.964 DEBUG:boto:Params: {} 2023-05-31T20:59:31.964 DEBUG:boto:Token: None 2023-05-31T20:59:31.964 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:59:31 GMT /pklxkd-60/obj33?versionId=null 2023-05-31T20:59:31.965 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:MTqt4ugH/nWjxgEPP4QPwM31Cuk= 2023-05-31T20:59:31.965 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:59:31 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:MTqt4ugH/nWjxgEPP4QPwM31Cuk=', 'Content-Length': '0'} 2023-05-31T20:59:31.967 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:56:04 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000fef96d9b573395d-006477b533-1ec6-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:59:31 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:59:31.967 DEBUG:boto:path=/pklxkd-60/obj33 2023-05-31T20:59:31.968 DEBUG:boto:auth_path=/pklxkd-60/obj33 2023-05-31T20:59:31.968 DEBUG:boto:path=/pklxkd-60/obj33?versionId=null 2023-05-31T20:59:31.968 DEBUG:boto:auth_path=/pklxkd-60/obj33?versionId=null 2023-05-31T20:59:31.969 DEBUG:boto:Method: HEAD 2023-05-31T20:59:31.969 DEBUG:boto:Path: /pklxkd-60/obj33?versionId=null 2023-05-31T20:59:31.969 DEBUG:boto:Data: 2023-05-31T20:59:31.970 DEBUG:boto:Headers: {} 2023-05-31T20:59:31.970 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:59:31.970 DEBUG:boto:Port: 8000 2023-05-31T20:59:31.971 DEBUG:boto:Params: {} 2023-05-31T20:59:31.971 DEBUG:boto:Token: None 2023-05-31T20:59:31.971 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:59:31 GMT /pklxkd-60/obj33?versionId=null 2023-05-31T20:59:31.972 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:MTqt4ugH/nWjxgEPP4QPwM31Cuk= 2023-05-31T20:59:31.972 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:59:31 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:MTqt4ugH/nWjxgEPP4QPwM31Cuk=', 'Content-Length': '0'} 2023-05-31T20:59:31.974 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:56:04 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006f19bcd5c913d210-006477b533-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:59:31 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:59:31.974 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj33 2023-05-31T20:59:31.974 DEBUG:boto:path=/pklxkd-60/obj33 2023-05-31T20:59:31.975 DEBUG:boto:auth_path=/pklxkd-60/obj33 2023-05-31T20:59:31.975 DEBUG:boto:path=/pklxkd-60/obj33?versionId=null 2023-05-31T20:59:31.976 DEBUG:boto:auth_path=/pklxkd-60/obj33?versionId=null 2023-05-31T20:59:31.976 DEBUG:boto:Method: GET 2023-05-31T20:59:31.976 DEBUG:boto:Path: /pklxkd-60/obj33?versionId=null 2023-05-31T20:59:31.977 DEBUG:boto:Data: 2023-05-31T20:59:31.977 DEBUG:boto:Headers: {} 2023-05-31T20:59:31.977 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:59:31.978 DEBUG:boto:Port: 8001 2023-05-31T20:59:31.978 DEBUG:boto:Params: {} 2023-05-31T20:59:31.978 DEBUG:boto:Token: None 2023-05-31T20:59:31.979 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:59:31 GMT /pklxkd-60/obj33?versionId=null 2023-05-31T20:59:31.979 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:aYdJ5rvQ74llibTeu9jar2BzVj0= 2023-05-31T20:59:31.980 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:59:31 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:aYdJ5rvQ74llibTeu9jar2BzVj0=', 'Content-Length': '0'} 2023-05-31T20:59:31.981 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:56:04 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005d1887cc76964cb3-006477b533-1ec6-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:59:31 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:59:31.982 DEBUG:boto:path=/pklxkd-60/obj33 2023-05-31T20:59:31.982 DEBUG:boto:auth_path=/pklxkd-60/obj33 2023-05-31T20:59:31.983 DEBUG:boto:path=/pklxkd-60/obj33?versionId=null 2023-05-31T20:59:31.983 DEBUG:boto:auth_path=/pklxkd-60/obj33?versionId=null 2023-05-31T20:59:31.983 DEBUG:boto:Method: GET 2023-05-31T20:59:31.984 DEBUG:boto:Path: /pklxkd-60/obj33?versionId=null 2023-05-31T20:59:31.984 DEBUG:boto:Data: 2023-05-31T20:59:31.984 DEBUG:boto:Headers: {} 2023-05-31T20:59:31.985 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:59:31.985 DEBUG:boto:Port: 8000 2023-05-31T20:59:31.985 DEBUG:boto:Params: {} 2023-05-31T20:59:31.986 DEBUG:boto:Token: None 2023-05-31T20:59:31.986 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:59:31 GMT /pklxkd-60/obj33?versionId=null 2023-05-31T20:59:31.987 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:aYdJ5rvQ74llibTeu9jar2BzVj0= 2023-05-31T20:59:31.987 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:59:31 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:aYdJ5rvQ74llibTeu9jar2BzVj0=', 'Content-Length': '0'} 2023-05-31T20:59:31.989 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:56:04 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000057729f47eea5431b-006477b533-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:59:31 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:59:31.989 DEBUG:boto:path=/pklxkd-60/ 2023-05-31T20:59:31.990 DEBUG:boto:auth_path=/pklxkd-60/ 2023-05-31T20:59:31.990 DEBUG:boto:path=/pklxkd-60/?prefix=obj33 2023-05-31T20:59:31.990 DEBUG:boto:auth_path=/pklxkd-60/?prefix=obj33 2023-05-31T20:59:31.991 DEBUG:boto:Method: GET 2023-05-31T20:59:31.991 DEBUG:boto:Path: /pklxkd-60/?prefix=obj33 2023-05-31T20:59:31.991 DEBUG:boto:Data: 2023-05-31T20:59:31.992 DEBUG:boto:Headers: {} 2023-05-31T20:59:31.992 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:59:31.993 DEBUG:boto:Port: 8001 2023-05-31T20:59:31.993 DEBUG:boto:Params: {} 2023-05-31T20:59:31.993 DEBUG:boto:Token: None 2023-05-31T20:59:31.994 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:59:31 GMT /pklxkd-60/ 2023-05-31T20:59:31.994 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:HeUfiHGYRLKd5l6XdeLlk05+4W8= 2023-05-31T20:59:31.995 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:59:31 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:HeUfiHGYRLKd5l6XdeLlk05+4W8=', 'Content-Length': '0'} 2023-05-31T20:59:31.997 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000001ed95d00256c84f0-006477b533-1ec6-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:59:31 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:59:31.997 DEBUG:boto:b'pklxkd-60obj331000falseobj332023-05-31T20:56:04.999Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:59:31.998 DEBUG:boto:path=/pklxkd-60/ 2023-05-31T20:59:31.998 DEBUG:boto:auth_path=/pklxkd-60/ 2023-05-31T20:59:31.998 DEBUG:boto:path=/pklxkd-60/?prefix=obj33 2023-05-31T20:59:31.999 DEBUG:boto:auth_path=/pklxkd-60/?prefix=obj33 2023-05-31T20:59:31.999 DEBUG:boto:Method: GET 2023-05-31T20:59:32.000 DEBUG:boto:Path: /pklxkd-60/?prefix=obj33 2023-05-31T20:59:32.000 DEBUG:boto:Data: 2023-05-31T20:59:32.000 DEBUG:boto:Headers: {} 2023-05-31T20:59:32.001 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:59:32.001 DEBUG:boto:Port: 8000 2023-05-31T20:59:32.001 DEBUG:boto:Params: {} 2023-05-31T20:59:32.002 DEBUG:boto:Token: None 2023-05-31T20:59:32.002 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:59:32 GMT /pklxkd-60/ 2023-05-31T20:59:32.002 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:iXEoowlsr71lqS5OnCcdNHqCxxs= 2023-05-31T20:59:32.003 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:59:32 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:iXEoowlsr71lqS5OnCcdNHqCxxs=', 'Content-Length': '0'} 2023-05-31T20:59:32.005 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000795c7aade7844506-006477b534-1630-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:59:32 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:59:32.005 DEBUG:boto:b'pklxkd-60obj331000falseobj332023-05-31T20:56:04.999Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:59:32.006 DEBUG:boto:path=/pklxkd-60/obj33 2023-05-31T20:59:32.006 DEBUG:boto:auth_path=/pklxkd-60/obj33 2023-05-31T20:59:32.007 DEBUG:boto:Method: HEAD 2023-05-31T20:59:32.007 DEBUG:boto:Path: /pklxkd-60/obj33 2023-05-31T20:59:32.007 DEBUG:boto:Data: 2023-05-31T20:59:32.008 DEBUG:boto:Headers: {} 2023-05-31T20:59:32.008 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:59:32.008 DEBUG:boto:Port: 8001 2023-05-31T20:59:32.009 DEBUG:boto:Params: {} 2023-05-31T20:59:32.009 DEBUG:boto:Token: None 2023-05-31T20:59:32.009 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:59:32 GMT /pklxkd-60/obj33 2023-05-31T20:59:32.010 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:2RsRFX1Zb+LJFFBbFvcsJ/AQsd8= 2023-05-31T20:59:32.010 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:59:32 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:2RsRFX1Zb+LJFFBbFvcsJ/AQsd8=', 'Content-Length': '0'} 2023-05-31T20:59:32.012 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:56:04 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000069d198e157d308d2-006477b534-1ec6-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:59:32 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:59:32.012 DEBUG:boto:path=/pklxkd-60/obj33 2023-05-31T20:59:32.013 DEBUG:boto:auth_path=/pklxkd-60/obj33 2023-05-31T20:59:32.013 DEBUG:boto:Method: HEAD 2023-05-31T20:59:32.013 DEBUG:boto:Path: /pklxkd-60/obj33 2023-05-31T20:59:32.014 DEBUG:boto:Data: 2023-05-31T20:59:32.014 DEBUG:boto:Headers: {} 2023-05-31T20:59:32.014 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:59:32.015 DEBUG:boto:Port: 8000 2023-05-31T20:59:32.015 DEBUG:boto:Params: {} 2023-05-31T20:59:32.015 DEBUG:boto:Token: None 2023-05-31T20:59:32.016 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:59:32 GMT /pklxkd-60/obj33 2023-05-31T20:59:32.016 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:2RsRFX1Zb+LJFFBbFvcsJ/AQsd8= 2023-05-31T20:59:32.016 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:59:32 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:2RsRFX1Zb+LJFFBbFvcsJ/AQsd8=', 'Content-Length': '0'} 2023-05-31T20:59:32.018 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:56:04 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c39a4ac4fedfc512-006477b534-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:59:32 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:59:32.018 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj33 2023-05-31T20:59:32.019 DEBUG:boto:path=/pklxkd-60/obj33 2023-05-31T20:59:32.019 DEBUG:boto:auth_path=/pklxkd-60/obj33 2023-05-31T20:59:32.020 DEBUG:boto:Method: GET 2023-05-31T20:59:32.020 DEBUG:boto:Path: /pklxkd-60/obj33 2023-05-31T20:59:32.020 DEBUG:boto:Data: 2023-05-31T20:59:32.021 DEBUG:boto:Headers: {} 2023-05-31T20:59:32.021 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:59:32.021 DEBUG:boto:Port: 8001 2023-05-31T20:59:32.022 DEBUG:boto:Params: {} 2023-05-31T20:59:32.022 DEBUG:boto:Token: None 2023-05-31T20:59:32.023 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:59:32 GMT /pklxkd-60/obj33 2023-05-31T20:59:32.023 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:tGCQ2fXmUCWm0dzfgqCv4Xl+QL8= 2023-05-31T20:59:32.023 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:59:32 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:tGCQ2fXmUCWm0dzfgqCv4Xl+QL8=', 'Content-Length': '0'} 2023-05-31T20:59:32.025 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:56:04 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000cc9224a2eb016271-006477b534-1ec6-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:59:32 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:59:32.026 DEBUG:boto:path=/pklxkd-60/obj33 2023-05-31T20:59:32.026 DEBUG:boto:auth_path=/pklxkd-60/obj33 2023-05-31T20:59:32.026 DEBUG:boto:Method: GET 2023-05-31T20:59:32.027 DEBUG:boto:Path: /pklxkd-60/obj33 2023-05-31T20:59:32.027 DEBUG:boto:Data: 2023-05-31T20:59:32.028 DEBUG:boto:Headers: {} 2023-05-31T20:59:32.028 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:59:32.028 DEBUG:boto:Port: 8000 2023-05-31T20:59:32.029 DEBUG:boto:Params: {} 2023-05-31T20:59:32.029 DEBUG:boto:Token: None 2023-05-31T20:59:32.029 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:59:32 GMT /pklxkd-60/obj33 2023-05-31T20:59:32.030 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:tGCQ2fXmUCWm0dzfgqCv4Xl+QL8= 2023-05-31T20:59:32.030 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:59:32 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:tGCQ2fXmUCWm0dzfgqCv4Xl+QL8=', 'Content-Length': '0'} 2023-05-31T20:59:32.032 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:56:04 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002f6c02d07aaa35b5-006477b534-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:59:32 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:59:32.072 DEBUG:boto:path=/pklxkd-60/ 2023-05-31T20:59:32.073 DEBUG:boto:auth_path=/pklxkd-60/ 2023-05-31T20:59:32.073 DEBUG:boto:path=/pklxkd-60/?prefix=obj33 2023-05-31T20:59:32.073 DEBUG:boto:auth_path=/pklxkd-60/?prefix=obj33 2023-05-31T20:59:32.074 DEBUG:boto:Method: GET 2023-05-31T20:59:32.074 DEBUG:boto:Path: /pklxkd-60/?prefix=obj33 2023-05-31T20:59:32.074 DEBUG:boto:Data: 2023-05-31T20:59:32.075 DEBUG:boto:Headers: {} 2023-05-31T20:59:32.075 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:59:32.075 DEBUG:boto:Port: 8001 2023-05-31T20:59:32.076 DEBUG:boto:Params: {} 2023-05-31T20:59:32.076 DEBUG:boto:Token: None 2023-05-31T20:59:32.077 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:59:32 GMT /pklxkd-60/ 2023-05-31T20:59:32.077 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:iXEoowlsr71lqS5OnCcdNHqCxxs= 2023-05-31T20:59:32.078 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:59:32 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:iXEoowlsr71lqS5OnCcdNHqCxxs=', 'Content-Length': '0'} 2023-05-31T20:59:32.080 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000291222be2705f258-006477b534-1ec6-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:59:32 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:59:32.080 DEBUG:boto:b'pklxkd-60obj331000falseobj332023-05-31T20:56:04.999Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:59:32.081 DEBUG:boto:path=/pklxkd-60/ 2023-05-31T20:59:32.081 DEBUG:boto:auth_path=/pklxkd-60/ 2023-05-31T20:59:32.082 DEBUG:boto:path=/pklxkd-60/?prefix=obj33 2023-05-31T20:59:32.082 DEBUG:boto:auth_path=/pklxkd-60/?prefix=obj33 2023-05-31T20:59:32.082 DEBUG:boto:Method: GET 2023-05-31T20:59:32.083 DEBUG:boto:Path: /pklxkd-60/?prefix=obj33 2023-05-31T20:59:32.083 DEBUG:boto:Data: 2023-05-31T20:59:32.084 DEBUG:boto:Headers: {} 2023-05-31T20:59:32.084 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:59:32.084 DEBUG:boto:Port: 8000 2023-05-31T20:59:32.085 DEBUG:boto:Params: {} 2023-05-31T20:59:32.085 DEBUG:boto:Token: None 2023-05-31T20:59:32.085 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:59:32 GMT /pklxkd-60/ 2023-05-31T20:59:32.086 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:iXEoowlsr71lqS5OnCcdNHqCxxs= 2023-05-31T20:59:32.086 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:59:32 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:iXEoowlsr71lqS5OnCcdNHqCxxs=', 'Content-Length': '0'} 2023-05-31T20:59:32.088 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000010e7d783864ef6b-006477b534-1630-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:59:32 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:59:32.089 DEBUG:boto:b'pklxkd-60obj331000falseobj332023-05-31T20:56:04.999Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:59:32.089 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj44 2023-05-31T20:59:32.090 DEBUG:boto:path=/pklxkd-60/obj44 2023-05-31T20:59:32.090 DEBUG:boto:auth_path=/pklxkd-60/obj44 2023-05-31T20:59:32.091 DEBUG:boto:path=/pklxkd-60/obj44?versionId=null 2023-05-31T20:59:32.091 DEBUG:boto:auth_path=/pklxkd-60/obj44?versionId=null 2023-05-31T20:59:32.091 DEBUG:boto:Method: GET 2023-05-31T20:59:32.092 DEBUG:boto:Path: /pklxkd-60/obj44?versionId=null 2023-05-31T20:59:32.092 DEBUG:boto:Data: 2023-05-31T20:59:32.092 DEBUG:boto:Headers: {} 2023-05-31T20:59:32.093 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:59:32.093 DEBUG:boto:Port: 8001 2023-05-31T20:59:32.093 DEBUG:boto:Params: {} 2023-05-31T20:59:32.094 DEBUG:boto:Token: None 2023-05-31T20:59:32.094 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:59:32 GMT /pklxkd-60/obj44?versionId=null 2023-05-31T20:59:32.095 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:42irMQ3/o6dzWUQ871E5uwpLj7E= 2023-05-31T20:59:32.095 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:59:32 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:42irMQ3/o6dzWUQ871E5uwpLj7E=', 'Content-Length': '0'} 2023-05-31T20:59:32.097 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:56:05 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002a64e581e29c8c5c-006477b534-1ec6-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:59:32 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:59:32.097 DEBUG:boto:path=/pklxkd-60/obj44 2023-05-31T20:59:32.098 DEBUG:boto:auth_path=/pklxkd-60/obj44 2023-05-31T20:59:32.098 DEBUG:boto:path=/pklxkd-60/obj44?versionId=null 2023-05-31T20:59:32.098 DEBUG:boto:auth_path=/pklxkd-60/obj44?versionId=null 2023-05-31T20:59:32.099 DEBUG:boto:Method: GET 2023-05-31T20:59:32.099 DEBUG:boto:Path: /pklxkd-60/obj44?versionId=null 2023-05-31T20:59:32.099 DEBUG:boto:Data: 2023-05-31T20:59:32.100 DEBUG:boto:Headers: {} 2023-05-31T20:59:32.100 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:59:32.101 DEBUG:boto:Port: 8000 2023-05-31T20:59:32.101 DEBUG:boto:Params: {} 2023-05-31T20:59:32.101 DEBUG:boto:Token: None 2023-05-31T20:59:32.102 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:59:32 GMT /pklxkd-60/obj44?versionId=null 2023-05-31T20:59:32.102 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:42irMQ3/o6dzWUQ871E5uwpLj7E= 2023-05-31T20:59:32.103 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:59:32 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:42irMQ3/o6dzWUQ871E5uwpLj7E=', 'Content-Length': '0'} 2023-05-31T20:59:32.104 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:56:05 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000001b7e6cd1a43e1ab-006477b534-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:59:32 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:59:32.105 DEBUG:boto:path=/pklxkd-60/obj44 2023-05-31T20:59:32.105 DEBUG:boto:auth_path=/pklxkd-60/obj44 2023-05-31T20:59:32.105 DEBUG:boto:path=/pklxkd-60/obj44?versionId=null 2023-05-31T20:59:32.106 DEBUG:boto:auth_path=/pklxkd-60/obj44?versionId=null 2023-05-31T20:59:32.106 DEBUG:boto:Method: HEAD 2023-05-31T20:59:32.106 DEBUG:boto:Path: /pklxkd-60/obj44?versionId=null 2023-05-31T20:59:32.107 DEBUG:boto:Data: 2023-05-31T20:59:32.107 DEBUG:boto:Headers: {} 2023-05-31T20:59:32.107 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:59:32.108 DEBUG:boto:Port: 8001 2023-05-31T20:59:32.108 DEBUG:boto:Params: {} 2023-05-31T20:59:32.108 DEBUG:boto:Token: None 2023-05-31T20:59:32.109 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:59:32 GMT /pklxkd-60/obj44?versionId=null 2023-05-31T20:59:32.109 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:4zVCugWP+QSnsmt0SthInm0aBr8= 2023-05-31T20:59:32.110 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:59:32 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:4zVCugWP+QSnsmt0SthInm0aBr8=', 'Content-Length': '0'} 2023-05-31T20:59:32.111 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:56:05 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000916cd6d43b6737a3-006477b534-1ec6-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:59:32 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:59:32.111 DEBUG:boto:path=/pklxkd-60/obj44 2023-05-31T20:59:32.112 DEBUG:boto:auth_path=/pklxkd-60/obj44 2023-05-31T20:59:32.112 DEBUG:boto:path=/pklxkd-60/obj44?versionId=null 2023-05-31T20:59:32.113 DEBUG:boto:auth_path=/pklxkd-60/obj44?versionId=null 2023-05-31T20:59:32.113 DEBUG:boto:Method: HEAD 2023-05-31T20:59:32.113 DEBUG:boto:Path: /pklxkd-60/obj44?versionId=null 2023-05-31T20:59:32.114 DEBUG:boto:Data: 2023-05-31T20:59:32.114 DEBUG:boto:Headers: {} 2023-05-31T20:59:32.114 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:59:32.115 DEBUG:boto:Port: 8000 2023-05-31T20:59:32.115 DEBUG:boto:Params: {} 2023-05-31T20:59:32.115 DEBUG:boto:Token: None 2023-05-31T20:59:32.116 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:59:32 GMT /pklxkd-60/obj44?versionId=null 2023-05-31T20:59:32.116 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:4zVCugWP+QSnsmt0SthInm0aBr8= 2023-05-31T20:59:32.116 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:59:32 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:4zVCugWP+QSnsmt0SthInm0aBr8=', 'Content-Length': '0'} 2023-05-31T20:59:32.118 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:56:05 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000002bbcf84e57b0e40-006477b534-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:59:32 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:59:32.118 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj44 2023-05-31T20:59:32.119 DEBUG:boto:path=/pklxkd-60/obj44 2023-05-31T20:59:32.119 DEBUG:boto:auth_path=/pklxkd-60/obj44 2023-05-31T20:59:32.120 DEBUG:boto:path=/pklxkd-60/obj44?versionId=null 2023-05-31T20:59:32.120 DEBUG:boto:auth_path=/pklxkd-60/obj44?versionId=null 2023-05-31T20:59:32.120 DEBUG:boto:Method: GET 2023-05-31T20:59:32.121 DEBUG:boto:Path: /pklxkd-60/obj44?versionId=null 2023-05-31T20:59:32.121 DEBUG:boto:Data: 2023-05-31T20:59:32.121 DEBUG:boto:Headers: {} 2023-05-31T20:59:32.122 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:59:32.122 DEBUG:boto:Port: 8001 2023-05-31T20:59:32.122 DEBUG:boto:Params: {} 2023-05-31T20:59:32.123 DEBUG:boto:Token: None 2023-05-31T20:59:32.123 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:59:32 GMT /pklxkd-60/obj44?versionId=null 2023-05-31T20:59:32.124 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:42irMQ3/o6dzWUQ871E5uwpLj7E= 2023-05-31T20:59:32.124 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:59:32 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:42irMQ3/o6dzWUQ871E5uwpLj7E=', 'Content-Length': '0'} 2023-05-31T20:59:32.126 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:56:05 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008f605c654124c2cd-006477b534-1ec6-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:59:32 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:59:32.168 DEBUG:boto:path=/pklxkd-60/obj44 2023-05-31T20:59:32.169 DEBUG:boto:auth_path=/pklxkd-60/obj44 2023-05-31T20:59:32.169 DEBUG:boto:path=/pklxkd-60/obj44?versionId=null 2023-05-31T20:59:32.169 DEBUG:boto:auth_path=/pklxkd-60/obj44?versionId=null 2023-05-31T20:59:32.170 DEBUG:boto:Method: GET 2023-05-31T20:59:32.170 DEBUG:boto:Path: /pklxkd-60/obj44?versionId=null 2023-05-31T20:59:32.170 DEBUG:boto:Data: 2023-05-31T20:59:32.171 DEBUG:boto:Headers: {} 2023-05-31T20:59:32.171 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:59:32.171 DEBUG:boto:Port: 8000 2023-05-31T20:59:32.172 DEBUG:boto:Params: {} 2023-05-31T20:59:32.172 DEBUG:boto:Token: None 2023-05-31T20:59:32.173 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:59:32 GMT /pklxkd-60/obj44?versionId=null 2023-05-31T20:59:32.173 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:42irMQ3/o6dzWUQ871E5uwpLj7E= 2023-05-31T20:59:32.173 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:59:32 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:42irMQ3/o6dzWUQ871E5uwpLj7E=', 'Content-Length': '0'} 2023-05-31T20:59:32.175 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:56:05 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009db72c145283a8be-006477b534-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:59:32 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:59:32.176 DEBUG:boto:path=/pklxkd-60/ 2023-05-31T20:59:32.176 DEBUG:boto:auth_path=/pklxkd-60/ 2023-05-31T20:59:32.177 DEBUG:boto:path=/pklxkd-60/?prefix=obj44 2023-05-31T20:59:32.177 DEBUG:boto:auth_path=/pklxkd-60/?prefix=obj44 2023-05-31T20:59:32.177 DEBUG:boto:Method: GET 2023-05-31T20:59:32.178 DEBUG:boto:Path: /pklxkd-60/?prefix=obj44 2023-05-31T20:59:32.178 DEBUG:boto:Data: 2023-05-31T20:59:32.178 DEBUG:boto:Headers: {} 2023-05-31T20:59:32.179 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:59:32.179 DEBUG:boto:Port: 8001 2023-05-31T20:59:32.179 DEBUG:boto:Params: {} 2023-05-31T20:59:32.180 DEBUG:boto:Token: None 2023-05-31T20:59:32.180 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:59:32 GMT /pklxkd-60/ 2023-05-31T20:59:32.181 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:iXEoowlsr71lqS5OnCcdNHqCxxs= 2023-05-31T20:59:32.181 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:59:32 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:iXEoowlsr71lqS5OnCcdNHqCxxs=', 'Content-Length': '0'} 2023-05-31T20:59:32.183 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ad446c361929a886-006477b534-1ec6-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:59:32 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:59:32.184 DEBUG:boto:b'pklxkd-60obj441000falseobj442023-05-31T20:56:05.021Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:59:32.184 DEBUG:boto:path=/pklxkd-60/ 2023-05-31T20:59:32.185 DEBUG:boto:auth_path=/pklxkd-60/ 2023-05-31T20:59:32.185 DEBUG:boto:path=/pklxkd-60/?prefix=obj44 2023-05-31T20:59:32.185 DEBUG:boto:auth_path=/pklxkd-60/?prefix=obj44 2023-05-31T20:59:32.186 DEBUG:boto:Method: GET 2023-05-31T20:59:32.186 DEBUG:boto:Path: /pklxkd-60/?prefix=obj44 2023-05-31T20:59:32.186 DEBUG:boto:Data: 2023-05-31T20:59:32.187 DEBUG:boto:Headers: {} 2023-05-31T20:59:32.187 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:59:32.188 DEBUG:boto:Port: 8000 2023-05-31T20:59:32.188 DEBUG:boto:Params: {} 2023-05-31T20:59:32.188 DEBUG:boto:Token: None 2023-05-31T20:59:32.189 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:59:32 GMT /pklxkd-60/ 2023-05-31T20:59:32.189 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:iXEoowlsr71lqS5OnCcdNHqCxxs= 2023-05-31T20:59:32.190 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:59:32 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:iXEoowlsr71lqS5OnCcdNHqCxxs=', 'Content-Length': '0'} 2023-05-31T20:59:32.191 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000014e4f247b22937e5-006477b534-1630-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:59:32 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:59:32.192 DEBUG:boto:b'pklxkd-60obj441000falseobj442023-05-31T20:56:05.021Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:59:32.192 DEBUG:boto:path=/pklxkd-60/obj44 2023-05-31T20:59:32.193 DEBUG:boto:auth_path=/pklxkd-60/obj44 2023-05-31T20:59:32.193 DEBUG:boto:Method: HEAD 2023-05-31T20:59:32.194 DEBUG:boto:Path: /pklxkd-60/obj44 2023-05-31T20:59:32.194 DEBUG:boto:Data: 2023-05-31T20:59:32.194 DEBUG:boto:Headers: {} 2023-05-31T20:59:32.195 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:59:32.195 DEBUG:boto:Port: 8001 2023-05-31T20:59:32.195 DEBUG:boto:Params: {} 2023-05-31T20:59:32.196 DEBUG:boto:Token: None 2023-05-31T20:59:32.196 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:59:32 GMT /pklxkd-60/obj44 2023-05-31T20:59:32.196 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:0JrAfiJuBDvZOYSNNqG3XxNcq5g= 2023-05-31T20:59:32.197 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:59:32 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:0JrAfiJuBDvZOYSNNqG3XxNcq5g=', 'Content-Length': '0'} 2023-05-31T20:59:32.198 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:56:05 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002ff391997fea575a-006477b534-1ec6-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:59:32 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:59:32.199 DEBUG:boto:path=/pklxkd-60/obj44 2023-05-31T20:59:32.199 DEBUG:boto:auth_path=/pklxkd-60/obj44 2023-05-31T20:59:32.200 DEBUG:boto:Method: HEAD 2023-05-31T20:59:32.200 DEBUG:boto:Path: /pklxkd-60/obj44 2023-05-31T20:59:32.200 DEBUG:boto:Data: 2023-05-31T20:59:32.201 DEBUG:boto:Headers: {} 2023-05-31T20:59:32.201 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:59:32.201 DEBUG:boto:Port: 8000 2023-05-31T20:59:32.202 DEBUG:boto:Params: {} 2023-05-31T20:59:32.202 DEBUG:boto:Token: None 2023-05-31T20:59:32.202 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:59:32 GMT /pklxkd-60/obj44 2023-05-31T20:59:32.203 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:0JrAfiJuBDvZOYSNNqG3XxNcq5g= 2023-05-31T20:59:32.203 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:59:32 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:0JrAfiJuBDvZOYSNNqG3XxNcq5g=', 'Content-Length': '0'} 2023-05-31T20:59:32.205 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:56:05 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000086b34d76307a1bd1-006477b534-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:59:32 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:59:32.205 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj44 2023-05-31T20:59:32.206 DEBUG:boto:path=/pklxkd-60/obj44 2023-05-31T20:59:32.206 DEBUG:boto:auth_path=/pklxkd-60/obj44 2023-05-31T20:59:32.207 DEBUG:boto:Method: GET 2023-05-31T20:59:32.207 DEBUG:boto:Path: /pklxkd-60/obj44 2023-05-31T20:59:32.207 DEBUG:boto:Data: 2023-05-31T20:59:32.208 DEBUG:boto:Headers: {} 2023-05-31T20:59:32.208 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:59:32.208 DEBUG:boto:Port: 8001 2023-05-31T20:59:32.209 DEBUG:boto:Params: {} 2023-05-31T20:59:32.209 DEBUG:boto:Token: None 2023-05-31T20:59:32.210 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:59:32 GMT /pklxkd-60/obj44 2023-05-31T20:59:32.210 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:N00L6/YXc+Ag1T5ywEC5Pz7Alrg= 2023-05-31T20:59:32.210 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:59:32 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:N00L6/YXc+Ag1T5ywEC5Pz7Alrg=', 'Content-Length': '0'} 2023-05-31T20:59:32.212 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:56:05 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b80fbd8242895936-006477b534-1ec6-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:59:32 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:59:32.212 DEBUG:boto:path=/pklxkd-60/obj44 2023-05-31T20:59:32.213 DEBUG:boto:auth_path=/pklxkd-60/obj44 2023-05-31T20:59:32.213 DEBUG:boto:Method: GET 2023-05-31T20:59:32.214 DEBUG:boto:Path: /pklxkd-60/obj44 2023-05-31T20:59:32.214 DEBUG:boto:Data: 2023-05-31T20:59:32.214 DEBUG:boto:Headers: {} 2023-05-31T20:59:32.215 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:59:32.215 DEBUG:boto:Port: 8000 2023-05-31T20:59:32.215 DEBUG:boto:Params: {} 2023-05-31T20:59:32.216 DEBUG:boto:Token: None 2023-05-31T20:59:32.216 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:59:32 GMT /pklxkd-60/obj44 2023-05-31T20:59:32.217 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:N00L6/YXc+Ag1T5ywEC5Pz7Alrg= 2023-05-31T20:59:32.217 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:59:32 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:N00L6/YXc+Ag1T5ywEC5Pz7Alrg=', 'Content-Length': '0'} 2023-05-31T20:59:32.219 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:56:05 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006fb97f1fb7ad9872-006477b534-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:59:32 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:59:32.219 DEBUG:boto:path=/pklxkd-60/ 2023-05-31T20:59:32.220 DEBUG:boto:auth_path=/pklxkd-60/ 2023-05-31T20:59:32.220 DEBUG:boto:path=/pklxkd-60/?prefix=obj44 2023-05-31T20:59:32.220 DEBUG:boto:auth_path=/pklxkd-60/?prefix=obj44 2023-05-31T20:59:32.221 DEBUG:boto:Method: GET 2023-05-31T20:59:32.221 DEBUG:boto:Path: /pklxkd-60/?prefix=obj44 2023-05-31T20:59:32.222 DEBUG:boto:Data: 2023-05-31T20:59:32.222 DEBUG:boto:Headers: {} 2023-05-31T20:59:32.222 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:59:32.223 DEBUG:boto:Port: 8001 2023-05-31T20:59:32.223 DEBUG:boto:Params: {} 2023-05-31T20:59:32.223 DEBUG:boto:Token: None 2023-05-31T20:59:32.224 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:59:32 GMT /pklxkd-60/ 2023-05-31T20:59:32.224 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:iXEoowlsr71lqS5OnCcdNHqCxxs= 2023-05-31T20:59:32.225 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:59:32 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:iXEoowlsr71lqS5OnCcdNHqCxxs=', 'Content-Length': '0'} 2023-05-31T20:59:32.226 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000006e4f84eb9402a7f7-006477b534-1ec6-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:59:32 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:59:32.227 DEBUG:boto:b'pklxkd-60obj441000falseobj442023-05-31T20:56:05.021Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:59:32.227 DEBUG:boto:path=/pklxkd-60/ 2023-05-31T20:59:32.228 DEBUG:boto:auth_path=/pklxkd-60/ 2023-05-31T20:59:32.228 DEBUG:boto:path=/pklxkd-60/?prefix=obj44 2023-05-31T20:59:32.229 DEBUG:boto:auth_path=/pklxkd-60/?prefix=obj44 2023-05-31T20:59:32.229 DEBUG:boto:Method: GET 2023-05-31T20:59:32.229 DEBUG:boto:Path: /pklxkd-60/?prefix=obj44 2023-05-31T20:59:32.230 DEBUG:boto:Data: 2023-05-31T20:59:32.230 DEBUG:boto:Headers: {} 2023-05-31T20:59:32.230 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:59:32.231 DEBUG:boto:Port: 8000 2023-05-31T20:59:32.231 DEBUG:boto:Params: {} 2023-05-31T20:59:32.231 DEBUG:boto:Token: None 2023-05-31T20:59:32.232 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:59:32 GMT /pklxkd-60/ 2023-05-31T20:59:32.232 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:iXEoowlsr71lqS5OnCcdNHqCxxs= 2023-05-31T20:59:32.233 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:59:32 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:iXEoowlsr71lqS5OnCcdNHqCxxs=', 'Content-Length': '0'} 2023-05-31T20:59:32.235 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000062e454506d26fccd-006477b534-1630-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:59:32 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:59:32.236 DEBUG:boto:b'pklxkd-60obj441000falseobj442023-05-31T20:56:05.021Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:59:32.236 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj55 2023-05-31T20:59:32.237 DEBUG:boto:path=/pklxkd-60/obj55 2023-05-31T20:59:32.237 DEBUG:boto:auth_path=/pklxkd-60/obj55 2023-05-31T20:59:32.237 DEBUG:boto:path=/pklxkd-60/obj55?versionId=null 2023-05-31T20:59:32.238 DEBUG:boto:auth_path=/pklxkd-60/obj55?versionId=null 2023-05-31T20:59:32.238 DEBUG:boto:Method: GET 2023-05-31T20:59:32.238 DEBUG:boto:Path: /pklxkd-60/obj55?versionId=null 2023-05-31T20:59:32.239 DEBUG:boto:Data: 2023-05-31T20:59:32.239 DEBUG:boto:Headers: {} 2023-05-31T20:59:32.239 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:59:32.240 DEBUG:boto:Port: 8001 2023-05-31T20:59:32.240 DEBUG:boto:Params: {} 2023-05-31T20:59:32.240 DEBUG:boto:Token: None 2023-05-31T20:59:32.241 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:59:32 GMT /pklxkd-60/obj55?versionId=null 2023-05-31T20:59:32.241 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:56yQfPtF0CjwqOkvNokAeR8Y8Dw= 2023-05-31T20:59:32.242 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:59:32 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:56yQfPtF0CjwqOkvNokAeR8Y8Dw=', 'Content-Length': '0'} 2023-05-31T20:59:32.244 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:56:05 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000859348dafadd74dc-006477b534-1ec6-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:59:32 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:59:32.244 DEBUG:boto:path=/pklxkd-60/obj55 2023-05-31T20:59:32.245 DEBUG:boto:auth_path=/pklxkd-60/obj55 2023-05-31T20:59:32.245 DEBUG:boto:path=/pklxkd-60/obj55?versionId=null 2023-05-31T20:59:32.245 DEBUG:boto:auth_path=/pklxkd-60/obj55?versionId=null 2023-05-31T20:59:32.246 DEBUG:boto:Method: GET 2023-05-31T20:59:32.246 DEBUG:boto:Path: /pklxkd-60/obj55?versionId=null 2023-05-31T20:59:32.246 DEBUG:boto:Data: 2023-05-31T20:59:32.247 DEBUG:boto:Headers: {} 2023-05-31T20:59:32.247 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:59:32.247 DEBUG:boto:Port: 8000 2023-05-31T20:59:32.248 DEBUG:boto:Params: {} 2023-05-31T20:59:32.248 DEBUG:boto:Token: None 2023-05-31T20:59:32.249 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:59:32 GMT /pklxkd-60/obj55?versionId=null 2023-05-31T20:59:32.249 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:56yQfPtF0CjwqOkvNokAeR8Y8Dw= 2023-05-31T20:59:32.249 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:59:32 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:56yQfPtF0CjwqOkvNokAeR8Y8Dw=', 'Content-Length': '0'} 2023-05-31T20:59:32.251 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:56:05 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d8b9fb35d152ae67-006477b534-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:59:32 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:59:32.252 DEBUG:boto:path=/pklxkd-60/obj55 2023-05-31T20:59:32.252 DEBUG:boto:auth_path=/pklxkd-60/obj55 2023-05-31T20:59:32.253 DEBUG:boto:path=/pklxkd-60/obj55?versionId=null 2023-05-31T20:59:32.253 DEBUG:boto:auth_path=/pklxkd-60/obj55?versionId=null 2023-05-31T20:59:32.253 DEBUG:boto:Method: HEAD 2023-05-31T20:59:32.254 DEBUG:boto:Path: /pklxkd-60/obj55?versionId=null 2023-05-31T20:59:32.254 DEBUG:boto:Data: 2023-05-31T20:59:32.254 DEBUG:boto:Headers: {} 2023-05-31T20:59:32.255 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:59:32.255 DEBUG:boto:Port: 8001 2023-05-31T20:59:32.255 DEBUG:boto:Params: {} 2023-05-31T20:59:32.256 DEBUG:boto:Token: None 2023-05-31T20:59:32.256 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:59:32 GMT /pklxkd-60/obj55?versionId=null 2023-05-31T20:59:32.256 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:wfvDhvkSCYT0qfNdtI62rFlfaGI= 2023-05-31T20:59:32.257 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:59:32 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:wfvDhvkSCYT0qfNdtI62rFlfaGI=', 'Content-Length': '0'} 2023-05-31T20:59:32.258 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:56:05 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007901e57172f72bac-006477b534-1ec6-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:59:32 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:59:32.259 DEBUG:boto:path=/pklxkd-60/obj55 2023-05-31T20:59:32.259 DEBUG:boto:auth_path=/pklxkd-60/obj55 2023-05-31T20:59:32.260 DEBUG:boto:path=/pklxkd-60/obj55?versionId=null 2023-05-31T20:59:32.260 DEBUG:boto:auth_path=/pklxkd-60/obj55?versionId=null 2023-05-31T20:59:32.260 DEBUG:boto:Method: HEAD 2023-05-31T20:59:32.261 DEBUG:boto:Path: /pklxkd-60/obj55?versionId=null 2023-05-31T20:59:32.261 DEBUG:boto:Data: 2023-05-31T20:59:32.261 DEBUG:boto:Headers: {} 2023-05-31T20:59:32.262 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:59:32.262 DEBUG:boto:Port: 8000 2023-05-31T20:59:32.262 DEBUG:boto:Params: {} 2023-05-31T20:59:32.263 DEBUG:boto:Token: None 2023-05-31T20:59:32.263 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:59:32 GMT /pklxkd-60/obj55?versionId=null 2023-05-31T20:59:32.264 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:wfvDhvkSCYT0qfNdtI62rFlfaGI= 2023-05-31T20:59:32.264 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:59:32 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:wfvDhvkSCYT0qfNdtI62rFlfaGI=', 'Content-Length': '0'} 2023-05-31T20:59:32.265 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:56:05 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001b8503e90573ba3a-006477b534-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:59:32 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:59:32.266 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj55 2023-05-31T20:59:32.266 DEBUG:boto:path=/pklxkd-60/obj55 2023-05-31T20:59:32.267 DEBUG:boto:auth_path=/pklxkd-60/obj55 2023-05-31T20:59:32.267 DEBUG:boto:path=/pklxkd-60/obj55?versionId=null 2023-05-31T20:59:32.267 DEBUG:boto:auth_path=/pklxkd-60/obj55?versionId=null 2023-05-31T20:59:32.268 DEBUG:boto:Method: GET 2023-05-31T20:59:32.268 DEBUG:boto:Path: /pklxkd-60/obj55?versionId=null 2023-05-31T20:59:32.268 DEBUG:boto:Data: 2023-05-31T20:59:32.269 DEBUG:boto:Headers: {} 2023-05-31T20:59:32.269 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:59:32.269 DEBUG:boto:Port: 8001 2023-05-31T20:59:32.270 DEBUG:boto:Params: {} 2023-05-31T20:59:32.270 DEBUG:boto:Token: None 2023-05-31T20:59:32.271 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:59:32 GMT /pklxkd-60/obj55?versionId=null 2023-05-31T20:59:32.271 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:56yQfPtF0CjwqOkvNokAeR8Y8Dw= 2023-05-31T20:59:32.271 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:59:32 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:56yQfPtF0CjwqOkvNokAeR8Y8Dw=', 'Content-Length': '0'} 2023-05-31T20:59:32.273 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:56:05 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e44876f911d78b45-006477b534-1ec6-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:59:32 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:59:32.274 DEBUG:boto:path=/pklxkd-60/obj55 2023-05-31T20:59:32.274 DEBUG:boto:auth_path=/pklxkd-60/obj55 2023-05-31T20:59:32.274 DEBUG:boto:path=/pklxkd-60/obj55?versionId=null 2023-05-31T20:59:32.275 DEBUG:boto:auth_path=/pklxkd-60/obj55?versionId=null 2023-05-31T20:59:32.275 DEBUG:boto:Method: GET 2023-05-31T20:59:32.275 DEBUG:boto:Path: /pklxkd-60/obj55?versionId=null 2023-05-31T20:59:32.276 DEBUG:boto:Data: 2023-05-31T20:59:32.276 DEBUG:boto:Headers: {} 2023-05-31T20:59:32.276 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:59:32.277 DEBUG:boto:Port: 8000 2023-05-31T20:59:32.277 DEBUG:boto:Params: {} 2023-05-31T20:59:32.278 DEBUG:boto:Token: None 2023-05-31T20:59:32.278 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:59:32 GMT /pklxkd-60/obj55?versionId=null 2023-05-31T20:59:32.278 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:56yQfPtF0CjwqOkvNokAeR8Y8Dw= 2023-05-31T20:59:32.279 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:59:32 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:56yQfPtF0CjwqOkvNokAeR8Y8Dw=', 'Content-Length': '0'} 2023-05-31T20:59:32.280 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:56:05 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000381377a52bdb0328-006477b534-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:59:32 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:59:32.281 DEBUG:boto:path=/pklxkd-60/ 2023-05-31T20:59:32.281 DEBUG:boto:auth_path=/pklxkd-60/ 2023-05-31T20:59:32.282 DEBUG:boto:path=/pklxkd-60/?prefix=obj55 2023-05-31T20:59:32.282 DEBUG:boto:auth_path=/pklxkd-60/?prefix=obj55 2023-05-31T20:59:32.282 DEBUG:boto:Method: GET 2023-05-31T20:59:32.283 DEBUG:boto:Path: /pklxkd-60/?prefix=obj55 2023-05-31T20:59:32.283 DEBUG:boto:Data: 2023-05-31T20:59:32.283 DEBUG:boto:Headers: {} 2023-05-31T20:59:32.284 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:59:32.284 DEBUG:boto:Port: 8001 2023-05-31T20:59:32.284 DEBUG:boto:Params: {} 2023-05-31T20:59:32.285 DEBUG:boto:Token: None 2023-05-31T20:59:32.285 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:59:32 GMT /pklxkd-60/ 2023-05-31T20:59:32.286 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:iXEoowlsr71lqS5OnCcdNHqCxxs= 2023-05-31T20:59:32.286 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:59:32 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:iXEoowlsr71lqS5OnCcdNHqCxxs=', 'Content-Length': '0'} 2023-05-31T20:59:32.288 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000004c65d4d0b1578aba-006477b534-1ec6-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:59:32 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:59:32.289 DEBUG:boto:b'pklxkd-60obj551000falseobj552023-05-31T20:56:05.074Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:59:32.289 DEBUG:boto:path=/pklxkd-60/ 2023-05-31T20:59:32.290 DEBUG:boto:auth_path=/pklxkd-60/ 2023-05-31T20:59:32.290 DEBUG:boto:path=/pklxkd-60/?prefix=obj55 2023-05-31T20:59:32.290 DEBUG:boto:auth_path=/pklxkd-60/?prefix=obj55 2023-05-31T20:59:32.291 DEBUG:boto:Method: GET 2023-05-31T20:59:32.291 DEBUG:boto:Path: /pklxkd-60/?prefix=obj55 2023-05-31T20:59:32.291 DEBUG:boto:Data: 2023-05-31T20:59:32.292 DEBUG:boto:Headers: {} 2023-05-31T20:59:32.292 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:59:32.292 DEBUG:boto:Port: 8000 2023-05-31T20:59:32.293 DEBUG:boto:Params: {} 2023-05-31T20:59:32.293 DEBUG:boto:Token: None 2023-05-31T20:59:32.294 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:59:32 GMT /pklxkd-60/ 2023-05-31T20:59:32.294 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:iXEoowlsr71lqS5OnCcdNHqCxxs= 2023-05-31T20:59:32.294 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:59:32 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:iXEoowlsr71lqS5OnCcdNHqCxxs=', 'Content-Length': '0'} 2023-05-31T20:59:32.297 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000001f3adfd7a7909e1f-006477b534-1630-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:59:32 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:59:32.297 DEBUG:boto:b'pklxkd-60obj551000falseobj552023-05-31T20:56:05.074Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:59:32.298 DEBUG:boto:path=/pklxkd-60/obj55 2023-05-31T20:59:32.298 DEBUG:boto:auth_path=/pklxkd-60/obj55 2023-05-31T20:59:32.298 DEBUG:boto:Method: HEAD 2023-05-31T20:59:32.299 DEBUG:boto:Path: /pklxkd-60/obj55 2023-05-31T20:59:32.299 DEBUG:boto:Data: 2023-05-31T20:59:32.299 DEBUG:boto:Headers: {} 2023-05-31T20:59:32.300 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:59:32.300 DEBUG:boto:Port: 8001 2023-05-31T20:59:32.300 DEBUG:boto:Params: {} 2023-05-31T20:59:32.301 DEBUG:boto:Token: None 2023-05-31T20:59:32.301 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:59:32 GMT /pklxkd-60/obj55 2023-05-31T20:59:32.302 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:3xvUPltoN3RhFmRLPNHO3j647B4= 2023-05-31T20:59:32.302 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:59:32 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:3xvUPltoN3RhFmRLPNHO3j647B4=', 'Content-Length': '0'} 2023-05-31T20:59:32.303 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:56:05 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e830b85a0c6f1851-006477b534-1ec6-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:59:32 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:59:32.304 DEBUG:boto:path=/pklxkd-60/obj55 2023-05-31T20:59:32.304 DEBUG:boto:auth_path=/pklxkd-60/obj55 2023-05-31T20:59:32.305 DEBUG:boto:Method: HEAD 2023-05-31T20:59:32.305 DEBUG:boto:Path: /pklxkd-60/obj55 2023-05-31T20:59:32.305 DEBUG:boto:Data: 2023-05-31T20:59:32.306 DEBUG:boto:Headers: {} 2023-05-31T20:59:32.306 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:59:32.306 DEBUG:boto:Port: 8000 2023-05-31T20:59:32.307 DEBUG:boto:Params: {} 2023-05-31T20:59:32.307 DEBUG:boto:Token: None 2023-05-31T20:59:32.307 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:59:32 GMT /pklxkd-60/obj55 2023-05-31T20:59:32.308 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:3xvUPltoN3RhFmRLPNHO3j647B4= 2023-05-31T20:59:32.308 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:59:32 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:3xvUPltoN3RhFmRLPNHO3j647B4=', 'Content-Length': '0'} 2023-05-31T20:59:32.309 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:56:05 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000075a86a7cb12e8010-006477b534-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:59:32 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:59:32.310 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj55 2023-05-31T20:59:32.310 DEBUG:boto:path=/pklxkd-60/obj55 2023-05-31T20:59:32.311 DEBUG:boto:auth_path=/pklxkd-60/obj55 2023-05-31T20:59:32.311 DEBUG:boto:Method: GET 2023-05-31T20:59:32.311 DEBUG:boto:Path: /pklxkd-60/obj55 2023-05-31T20:59:32.312 DEBUG:boto:Data: 2023-05-31T20:59:32.312 DEBUG:boto:Headers: {} 2023-05-31T20:59:32.313 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:59:32.313 DEBUG:boto:Port: 8001 2023-05-31T20:59:32.313 DEBUG:boto:Params: {} 2023-05-31T20:59:32.314 DEBUG:boto:Token: None 2023-05-31T20:59:32.314 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:59:32 GMT /pklxkd-60/obj55 2023-05-31T20:59:32.315 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:GvCXjcNk1ezWucqkgnuk8qAiTsM= 2023-05-31T20:59:32.315 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:59:32 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:GvCXjcNk1ezWucqkgnuk8qAiTsM=', 'Content-Length': '0'} 2023-05-31T20:59:32.316 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:56:05 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008280db61a984a016-006477b534-1ec6-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:59:32 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:59:32.317 DEBUG:boto:path=/pklxkd-60/obj55 2023-05-31T20:59:32.317 DEBUG:boto:auth_path=/pklxkd-60/obj55 2023-05-31T20:59:32.318 DEBUG:boto:Method: GET 2023-05-31T20:59:32.318 DEBUG:boto:Path: /pklxkd-60/obj55 2023-05-31T20:59:32.319 DEBUG:boto:Data: 2023-05-31T20:59:32.319 DEBUG:boto:Headers: {} 2023-05-31T20:59:32.319 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:59:32.320 DEBUG:boto:Port: 8000 2023-05-31T20:59:32.320 DEBUG:boto:Params: {} 2023-05-31T20:59:32.320 DEBUG:boto:Token: None 2023-05-31T20:59:32.321 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:59:32 GMT /pklxkd-60/obj55 2023-05-31T20:59:32.321 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:GvCXjcNk1ezWucqkgnuk8qAiTsM= 2023-05-31T20:59:32.322 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:59:32 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:GvCXjcNk1ezWucqkgnuk8qAiTsM=', 'Content-Length': '0'} 2023-05-31T20:59:32.323 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:56:05 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000956da6f84e25d226-006477b534-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:59:32 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:59:32.364 DEBUG:boto:path=/pklxkd-60/ 2023-05-31T20:59:32.365 DEBUG:boto:auth_path=/pklxkd-60/ 2023-05-31T20:59:32.365 DEBUG:boto:path=/pklxkd-60/?prefix=obj55 2023-05-31T20:59:32.365 DEBUG:boto:auth_path=/pklxkd-60/?prefix=obj55 2023-05-31T20:59:32.366 DEBUG:boto:Method: GET 2023-05-31T20:59:32.366 DEBUG:boto:Path: /pklxkd-60/?prefix=obj55 2023-05-31T20:59:32.366 DEBUG:boto:Data: 2023-05-31T20:59:32.367 DEBUG:boto:Headers: {} 2023-05-31T20:59:32.367 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T20:59:32.367 DEBUG:boto:Port: 8001 2023-05-31T20:59:32.368 DEBUG:boto:Params: {} 2023-05-31T20:59:32.368 DEBUG:boto:Token: None 2023-05-31T20:59:32.369 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:59:32 GMT /pklxkd-60/ 2023-05-31T20:59:32.369 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:iXEoowlsr71lqS5OnCcdNHqCxxs= 2023-05-31T20:59:32.369 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:59:32 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:iXEoowlsr71lqS5OnCcdNHqCxxs=', 'Content-Length': '0'} 2023-05-31T20:59:32.371 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b4a8d7dd9f478506-006477b534-1ec6-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:59:32 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:59:32.372 DEBUG:boto:b'pklxkd-60obj551000falseobj552023-05-31T20:56:05.074Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:59:32.373 DEBUG:boto:path=/pklxkd-60/ 2023-05-31T20:59:32.373 DEBUG:boto:auth_path=/pklxkd-60/ 2023-05-31T20:59:32.373 DEBUG:boto:path=/pklxkd-60/?prefix=obj55 2023-05-31T20:59:32.374 DEBUG:boto:auth_path=/pklxkd-60/?prefix=obj55 2023-05-31T20:59:32.374 DEBUG:boto:Method: GET 2023-05-31T20:59:32.375 DEBUG:boto:Path: /pklxkd-60/?prefix=obj55 2023-05-31T20:59:32.375 DEBUG:boto:Data: 2023-05-31T20:59:32.375 DEBUG:boto:Headers: {} 2023-05-31T20:59:32.376 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:59:32.376 DEBUG:boto:Port: 8000 2023-05-31T20:59:32.376 DEBUG:boto:Params: {} 2023-05-31T20:59:32.377 DEBUG:boto:Token: None 2023-05-31T20:59:32.377 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 20:59:32 GMT /pklxkd-60/ 2023-05-31T20:59:32.378 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:iXEoowlsr71lqS5OnCcdNHqCxxs= 2023-05-31T20:59:32.378 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:59:32 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:iXEoowlsr71lqS5OnCcdNHqCxxs=', 'Content-Length': '0'} 2023-05-31T20:59:32.380 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000095cbc1ef1125c147-006477b534-1630-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:59:32 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:59:32.381 DEBUG:boto:b'pklxkd-60obj551000falseobj552023-05-31T20:56:05.074Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T20:59:32.381 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=pklxkd-60 zones={a2, a1} 2023-05-31T20:59:32.382 DEBUG:rgw_multi.tests:data checkpoint: source=a1 target=a2 2023-05-31T20:59:32.382 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['datalog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-31T20:59:32.382 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'datalog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-31T20:59:32.383 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 datalog status --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-05-31T20:59:32.432 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2023-05-31T20:59:32.432 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2023-05-31T20:59:32.514 INFO:teuthology.orchestra.run.smithi099.stdout:[ 2023-05-31T20:59:32.514 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T20:59:32.515 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "00000000000000000000:00000000000000025273", 2023-05-31T20:59:32.515 INFO:teuthology.orchestra.run.smithi099.stdout: "last_update": "2023-05-31T20:56:10.639409Z" 2023-05-31T20:59:32.515 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:59:32.515 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T20:59:32.515 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "00000000000000000000:00000000000000018334", 2023-05-31T20:59:32.515 INFO:teuthology.orchestra.run.smithi099.stdout: "last_update": "2023-05-31T20:52:14.863321Z" 2023-05-31T20:59:32.516 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:59:32.516 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T20:59:32.516 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "00000000000000000000:00000000000000030599", 2023-05-31T20:59:32.516 INFO:teuthology.orchestra.run.smithi099.stdout: "last_update": "2023-05-31T20:56:05.024110Z" 2023-05-31T20:59:32.516 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:59:32.516 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T20:59:32.516 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "00000000000000000000:00000000000000026500", 2023-05-31T20:59:32.517 INFO:teuthology.orchestra.run.smithi099.stdout: "last_update": "2023-05-31T20:56:10.641181Z" 2023-05-31T20:59:32.517 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T20:59:32.517 INFO:teuthology.orchestra.run.smithi099.stdout:] 2023-05-31T20:59:32.518 DEBUG:rgw_multi.tests:data markers for zone=a1 markers={0: '00000000000000000000:00000000000000025273', 1: '00000000000000000000:00000000000000018334', 2: '00000000000000000000:00000000000000030599', 3: '00000000000000000000:00000000000000026500'} 2023-05-31T20:59:32.518 INFO:rgw_multi.tests:starting data checkpoint for target_zone=a2 source_zone=a1 2023-05-31T20:59:32.519 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-31T20:59:32.519 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-31T20:59:32.519 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-05-31T20:59:32.570 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2023-05-31T20:59:32.570 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2023-05-31T20:59:32.641 INFO:teuthology.orchestra.run.smithi099.stdout:{ 2023-05-31T20:59:32.641 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_status": { 2023-05-31T20:59:32.641 INFO:teuthology.orchestra.run.smithi099.stdout: "info": { 2023-05-31T20:59:32.641 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "sync", 2023-05-31T20:59:32.641 INFO:teuthology.orchestra.run.smithi099.stdout: "num_shards": 4, 2023-05-31T20:59:32.642 INFO:teuthology.orchestra.run.smithi099.stdout: "instance_id": 1039642245673841108 2023-05-31T20:59:32.642 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:59:32.642 INFO:teuthology.orchestra.run.smithi099.stdout: "markers": [ 2023-05-31T20:59:32.642 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T20:59:32.642 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 0, 2023-05-31T20:59:32.642 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T20:59:32.642 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "incremental-sync", 2023-05-31T20:59:32.643 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "00000000000000000000:00000000000000025273", 2023-05-31T20:59:32.643 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T20:59:32.643 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2023-05-31T20:59:32.643 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T20:59:32.643 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2023-05-31T20:56:10.639409Z" 2023-05-31T20:59:32.643 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T20:59:32.643 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:59:32.644 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T20:59:32.644 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 1, 2023-05-31T20:59:32.644 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T20:59:32.644 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "incremental-sync", 2023-05-31T20:59:32.644 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "00000000000000000000:00000000000000018334", 2023-05-31T20:59:32.644 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T20:59:32.645 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2023-05-31T20:59:32.645 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T20:59:32.645 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2023-05-31T20:52:14.863321Z" 2023-05-31T20:59:32.645 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T20:59:32.645 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:59:32.645 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T20:59:32.645 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 2, 2023-05-31T20:59:32.646 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T20:59:32.646 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "incremental-sync", 2023-05-31T20:59:32.646 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "00000000000000000000:00000000000000030599", 2023-05-31T20:59:32.646 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T20:59:32.646 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 51, 2023-05-31T20:59:32.646 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T20:59:32.646 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2023-05-31T20:56:05.024110Z" 2023-05-31T20:59:32.647 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T20:59:32.647 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:59:32.647 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T20:59:32.647 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 3, 2023-05-31T20:59:32.647 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T20:59:32.647 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "incremental-sync", 2023-05-31T20:59:32.647 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "00000000000000000000:00000000000000026500", 2023-05-31T20:59:32.648 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T20:59:32.648 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 9, 2023-05-31T20:59:32.648 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T20:59:32.648 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2023-05-31T20:56:10.641181Z" 2023-05-31T20:59:32.648 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T20:59:32.648 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T20:59:32.648 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2023-05-31T20:59:32.649 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:59:32.649 INFO:teuthology.orchestra.run.smithi099.stdout: "full_sync": { 2023-05-31T20:59:32.649 INFO:teuthology.orchestra.run.smithi099.stdout: "total": 60, 2023-05-31T20:59:32.649 INFO:teuthology.orchestra.run.smithi099.stdout: "complete": 60 2023-05-31T20:59:32.649 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:59:32.649 INFO:teuthology.orchestra.run.smithi099.stdout: "current_time": "2023-05-31T20:59:32Z" 2023-05-31T20:59:32.650 INFO:teuthology.orchestra.run.smithi099.stdout:} 2023-05-31T20:59:32.650 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 1039642245673841108 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000025273", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-05-31T20:56:10.639409Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000018334", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-05-31T20:52:14.863321Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000030599", "next_step_marker": "", "total_entries": 51, "pos": 0, "timestamp": "2023-05-31T20:56:05.024110Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000026500", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2023-05-31T20:56:10.641181Z" } } ] }, "full_sync": { "total": 60, "complete": 60 }, "current_time": "2023-05-31T20:59:32Z" } 2023-05-31T20:59:32.651 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000025273', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-05-31T20:56:10.639409Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000018334', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-05-31T20:52:14.863321Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000030599', 'next_step_marker': '', 'total_entries': 51, 'pos': 0, 'timestamp': '2023-05-31T20:56:05.024110Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000026500', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2023-05-31T20:56:10.641181Z'}}] 2023-05-31T20:59:32.651 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000025273', 1: '00000000000000000000:00000000000000018334', 2: '00000000000000000000:00000000000000030599', 3: '00000000000000000000:00000000000000026500'} 2023-05-31T20:59:32.652 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000025273', 1: '00000000000000000000:00000000000000018334', 2: '00000000000000000000:00000000000000030599', 3: '00000000000000000000:00000000000000026500'} 2023-05-31T20:59:32.652 INFO:rgw_multi.tests:finished data checkpoint for target_zone=a2 source_zone=a1 2023-05-31T20:59:32.652 DEBUG:rgw_multi.tests:data checkpoint: source=a2 target=a1 2023-05-31T20:59:32.653 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['datalog', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-31T20:59:32.653 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'datalog', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-31T20:59:32.653 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 datalog status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-05-31T20:59:32.701 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2023-05-31T20:59:32.701 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2023-05-31T20:59:32.758 INFO:teuthology.orchestra.run.smithi099.stdout:[ 2023-05-31T20:59:32.758 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T20:59:32.758 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "00000000000000000000:00000000000000009842", 2023-05-31T20:59:32.758 INFO:teuthology.orchestra.run.smithi099.stdout: "last_update": "2023-05-31T20:58:17.353002Z" 2023-05-31T20:59:32.758 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:59:32.758 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T20:59:32.759 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "00000000000000000000:00000000000000004601", 2023-05-31T20:59:32.759 INFO:teuthology.orchestra.run.smithi099.stdout: "last_update": "2023-05-31T20:54:52.979815Z" 2023-05-31T20:59:32.759 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:59:32.759 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T20:59:32.759 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "00000000000000000000:00000000000000014040", 2023-05-31T20:59:32.759 INFO:teuthology.orchestra.run.smithi099.stdout: "last_update": "2023-05-31T20:58:05.542682Z" 2023-05-31T20:59:32.760 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:59:32.760 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T20:59:32.760 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "00000000000000000000:00000000000000012915", 2023-05-31T20:59:32.760 INFO:teuthology.orchestra.run.smithi099.stdout: "last_update": "2023-05-31T20:58:17.354615Z" 2023-05-31T20:59:32.760 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T20:59:32.760 INFO:teuthology.orchestra.run.smithi099.stdout:] 2023-05-31T20:59:32.761 DEBUG:rgw_multi.tests:data markers for zone=a2 markers={0: '00000000000000000000:00000000000000009842', 1: '00000000000000000000:00000000000000004601', 2: '00000000000000000000:00000000000000014040', 3: '00000000000000000000:00000000000000012915'} 2023-05-31T20:59:32.761 INFO:rgw_multi.tests:starting data checkpoint for target_zone=a1 source_zone=a2 2023-05-31T20:59:32.762 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-31T20:59:32.762 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-31T20:59:32.763 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --source-zone a2 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-05-31T20:59:32.813 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2023-05-31T20:59:32.813 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2023-05-31T20:59:32.881 INFO:teuthology.orchestra.run.smithi099.stdout:{ 2023-05-31T20:59:32.881 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_status": { 2023-05-31T20:59:32.882 INFO:teuthology.orchestra.run.smithi099.stdout: "info": { 2023-05-31T20:59:32.882 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "sync", 2023-05-31T20:59:32.882 INFO:teuthology.orchestra.run.smithi099.stdout: "num_shards": 4, 2023-05-31T20:59:32.882 INFO:teuthology.orchestra.run.smithi099.stdout: "instance_id": 17941554979570305020 2023-05-31T20:59:32.882 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:59:32.882 INFO:teuthology.orchestra.run.smithi099.stdout: "markers": [ 2023-05-31T20:59:32.883 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T20:59:32.883 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 0, 2023-05-31T20:59:32.883 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T20:59:32.883 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "incremental-sync", 2023-05-31T20:59:32.883 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "00000000000000000000:00000000000000009842", 2023-05-31T20:59:32.883 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T20:59:32.883 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2023-05-31T20:59:32.884 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T20:59:32.884 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2023-05-31T20:58:17.353002Z" 2023-05-31T20:59:32.884 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T20:59:32.884 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:59:32.884 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T20:59:32.884 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 1, 2023-05-31T20:59:32.885 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T20:59:32.885 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "incremental-sync", 2023-05-31T20:59:32.885 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "00000000000000000000:00000000000000004601", 2023-05-31T20:59:32.885 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T20:59:32.885 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2023-05-31T20:59:32.885 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T20:59:32.885 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2023-05-31T20:54:52.979815Z" 2023-05-31T20:59:32.886 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T20:59:32.886 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:59:32.886 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T20:59:32.886 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 2, 2023-05-31T20:59:32.886 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T20:59:32.886 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "incremental-sync", 2023-05-31T20:59:32.887 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "00000000000000000000:00000000000000014040", 2023-05-31T20:59:32.887 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T20:59:32.887 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2023-05-31T20:59:32.887 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T20:59:32.887 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2023-05-31T20:58:05.542682Z" 2023-05-31T20:59:32.887 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T20:59:32.887 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:59:32.888 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T20:59:32.888 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 3, 2023-05-31T20:59:32.888 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T20:59:32.888 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "incremental-sync", 2023-05-31T20:59:32.888 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "00000000000000000000:00000000000000012915", 2023-05-31T20:59:32.888 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T20:59:32.888 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2023-05-31T20:59:32.889 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T20:59:32.889 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2023-05-31T20:58:17.354615Z" 2023-05-31T20:59:32.889 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T20:59:32.889 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T20:59:32.889 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2023-05-31T20:59:32.889 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:59:32.889 INFO:teuthology.orchestra.run.smithi099.stdout: "full_sync": { 2023-05-31T20:59:32.890 INFO:teuthology.orchestra.run.smithi099.stdout: "total": 0, 2023-05-31T20:59:32.890 INFO:teuthology.orchestra.run.smithi099.stdout: "complete": 0 2023-05-31T20:59:32.890 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:59:32.890 INFO:teuthology.orchestra.run.smithi099.stdout: "current_time": "2023-05-31T20:59:32Z" 2023-05-31T20:59:32.890 INFO:teuthology.orchestra.run.smithi099.stdout:} 2023-05-31T20:59:32.891 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 17941554979570305020 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000009842", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-05-31T20:58:17.353002Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000004601", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-05-31T20:54:52.979815Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000014040", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-05-31T20:58:05.542682Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000012915", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-05-31T20:58:17.354615Z" } } ] }, "full_sync": { "total": 0, "complete": 0 }, "current_time": "2023-05-31T20:59:32Z" } 2023-05-31T20:59:32.892 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000009842', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-05-31T20:58:17.353002Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000004601', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-05-31T20:54:52.979815Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000014040', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-05-31T20:58:05.542682Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000012915', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-05-31T20:58:17.354615Z'}}] 2023-05-31T20:59:32.892 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000009842', 1: '00000000000000000000:00000000000000004601', 2: '00000000000000000000:00000000000000014040', 3: '00000000000000000000:00000000000000012915'} 2023-05-31T20:59:32.893 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000009842', 1: '00000000000000000000:00000000000000004601', 2: '00000000000000000000:00000000000000014040', 3: '00000000000000000000:00000000000000012915'} 2023-05-31T20:59:32.893 INFO:rgw_multi.tests:finished data checkpoint for target_zone=a1 source_zone=a2 2023-05-31T20:59:32.893 INFO:tasks.rgw_multisite_tests:rgw_multi.tests.test_bucket_full_sync_after_data_sync_init ... ok 2023-05-31T20:59:32.895 DEBUG:boto:path=/pklxkd-61/ 2023-05-31T20:59:32.895 DEBUG:boto:auth_path=/pklxkd-61/ 2023-05-31T20:59:32.895 DEBUG:boto:Method: PUT 2023-05-31T20:59:32.896 DEBUG:boto:Path: /pklxkd-61/ 2023-05-31T20:59:32.896 DEBUG:boto:Data: 2023-05-31T20:59:32.896 DEBUG:boto:Headers: {} 2023-05-31T20:59:32.897 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:59:32.897 DEBUG:boto:Port: 8000 2023-05-31T20:59:32.897 DEBUG:boto:Params: {} 2023-05-31T20:59:32.898 DEBUG:boto:Token: None 2023-05-31T20:59:32.898 DEBUG:boto:StringToSign: PUT Wed, 31 May 2023 20:59:32 GMT /pklxkd-61/ 2023-05-31T20:59:32.899 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:zLc113YHnFZGBVxpCpzbfBJcgAo= 2023-05-31T20:59:32.899 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:59:32 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:zLc113YHnFZGBVxpCpzbfBJcgAo=', 'Content-Length': '0'} 2023-05-31T20:59:32.916 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000eb1324ac3f5060fa-006477b534-1630-a1'), ('Content-Length', '0'), ('Date', 'Wed, 31 May 2023 20:59:32 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:59:32.917 DEBUG:rgw_multi.tests:created bucket=pklxkd-61 2023-05-31T20:59:32.917 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-31T20:59:32.918 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-31T20:59:32.918 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-05-31T20:59:32.966 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2023-05-31T20:59:32.966 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2023-05-31T20:59:33.017 INFO:teuthology.orchestra.run.smithi099.stderr:No --period given, using current period=2e6123d2-d695-4350-9a01-61a455bc782b 2023-05-31T20:59:33.041 INFO:teuthology.orchestra.run.smithi099.stdout:[ 2023-05-31T20:59:33.042 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T20:59:33.042 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1685566501.757183_260.1", 2023-05-31T20:59:33.042 INFO:teuthology.orchestra.run.smithi099.stdout: "last_update": "2023-05-31T20:55:01.757183Z" 2023-05-31T20:59:33.042 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:59:33.042 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T20:59:33.042 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1685566501.735717_251.1", 2023-05-31T20:59:33.042 INFO:teuthology.orchestra.run.smithi099.stdout: "last_update": "2023-05-31T20:55:01.735717Z" 2023-05-31T20:59:33.043 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:59:33.043 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T20:59:33.043 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1685562970.274572_93.1", 2023-05-31T20:59:33.043 INFO:teuthology.orchestra.run.smithi099.stdout: "last_update": "2023-05-31T19:56:10.274572Z" 2023-05-31T20:59:33.043 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:59:33.043 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T20:59:33.043 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1685566772.911499_492.1", 2023-05-31T20:59:33.044 INFO:teuthology.orchestra.run.smithi099.stdout: "last_update": "2023-05-31T20:59:32.911499Z" 2023-05-31T20:59:33.044 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T20:59:33.044 INFO:teuthology.orchestra.run.smithi099.stdout:] 2023-05-31T20:59:33.045 DEBUG:rgw_multi.tests:master meta markers={0: '1_1685566501.757183_260.1', 1: '1_1685566501.735717_251.1', 2: '1_1685562970.274572_93.1', 3: '1_1685566772.911499_492.1'} 2023-05-31T20:59:33.045 INFO:rgw_multi.tests:starting meta checkpoint for zone=a2 2023-05-31T20:59:33.046 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-31T20:59:33.046 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-31T20:59:33.047 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-05-31T20:59:33.097 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2023-05-31T20:59:33.097 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2023-05-31T20:59:33.141 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:59:33.138+0000 7f938a810780 20 RGW-SYNC:meta: read sync status 2023-05-31T20:59:33.142 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:59:33.139+0000 7f938a810780 20 RGW-SYNC:meta: read sync status 2023-05-31T20:59:33.161 INFO:teuthology.orchestra.run.smithi099.stdout:{ 2023-05-31T20:59:33.161 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_status": { 2023-05-31T20:59:33.161 INFO:teuthology.orchestra.run.smithi099.stdout: "info": { 2023-05-31T20:59:33.161 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "sync", 2023-05-31T20:59:33.162 INFO:teuthology.orchestra.run.smithi099.stdout: "num_shards": 4, 2023-05-31T20:59:33.162 INFO:teuthology.orchestra.run.smithi099.stdout: "period": "2e6123d2-d695-4350-9a01-61a455bc782b", 2023-05-31T20:59:33.162 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2023-05-31T20:59:33.162 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:59:33.162 INFO:teuthology.orchestra.run.smithi099.stdout: "markers": [ 2023-05-31T20:59:33.162 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T20:59:33.162 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 0, 2023-05-31T20:59:33.163 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T20:59:33.163 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2023-05-31T20:59:33.163 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1685566501.757183_260.1", 2023-05-31T20:59:33.163 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T20:59:33.163 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 1, 2023-05-31T20:59:33.163 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T20:59:33.163 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2023-05-31T20:55:01.757183Z", 2023-05-31T20:59:33.164 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2023-05-31T20:59:33.164 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T20:59:33.164 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:59:33.164 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T20:59:33.164 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 1, 2023-05-31T20:59:33.164 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T20:59:33.164 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2023-05-31T20:59:33.165 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1685566501.735717_251.1", 2023-05-31T20:59:33.165 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T20:59:33.165 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2023-05-31T20:59:33.165 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T20:59:33.165 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2023-05-31T20:55:01.735717Z", 2023-05-31T20:59:33.165 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2023-05-31T20:59:33.165 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T20:59:33.166 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:59:33.166 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T20:59:33.166 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 2, 2023-05-31T20:59:33.166 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T20:59:33.166 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2023-05-31T20:59:33.166 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1685562970.274572_93.1", 2023-05-31T20:59:33.166 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T20:59:33.167 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2023-05-31T20:59:33.167 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T20:59:33.167 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2023-05-31T19:56:10.274572Z", 2023-05-31T20:59:33.167 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2023-05-31T20:59:33.167 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T20:59:33.167 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:59:33.167 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T20:59:33.168 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 3, 2023-05-31T20:59:33.168 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T20:59:33.168 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2023-05-31T20:59:33.168 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1685566772.905669_489.1", 2023-05-31T20:59:33.168 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T20:59:33.168 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2023-05-31T20:59:33.169 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T20:59:33.169 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2023-05-31T20:59:32.905669Z", 2023-05-31T20:59:33.169 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2023-05-31T20:59:33.169 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T20:59:33.169 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T20:59:33.169 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2023-05-31T20:59:33.169 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:59:33.170 INFO:teuthology.orchestra.run.smithi099.stdout: "full_sync": { 2023-05-31T20:59:33.170 INFO:teuthology.orchestra.run.smithi099.stdout: "total": 1, 2023-05-31T20:59:33.170 INFO:teuthology.orchestra.run.smithi099.stdout: "complete": 1 2023-05-31T20:59:33.170 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:59:33.170 INFO:teuthology.orchestra.run.smithi099.stdout: "current_time": "2023-05-31T20:59:33Z" 2023-05-31T20:59:33.170 INFO:teuthology.orchestra.run.smithi099.stdout:} 2023-05-31T20:59:33.171 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "2e6123d2-d695-4350-9a01-61a455bc782b", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1685566501.757183_260.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-05-31T20:55:01.757183Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1685566501.735717_251.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-05-31T20:55:01.735717Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1685562970.274572_93.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-05-31T19:56:10.274572Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1685566772.905669_489.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-05-31T20:59:32.905669Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-05-31T20:59:33Z" } 2023-05-31T20:59:33.171 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1685566501.757183_260.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-05-31T20:55:01.757183Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1685566501.735717_251.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-05-31T20:55:01.735717Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1685562970.274572_93.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-05-31T19:56:10.274572Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1685566772.905669_489.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-05-31T20:59:32.905669Z', 'realm_epoch': 2}}] 2023-05-31T20:59:33.172 DEBUG:rgw_multi.tests:log_status={0: '1_1685566501.757183_260.1', 1: '1_1685566501.735717_251.1', 2: '1_1685562970.274572_93.1', 3: '1_1685566772.911499_492.1'} 2023-05-31T20:59:33.172 DEBUG:rgw_multi.tests:sync_status={0: '1_1685566501.757183_260.1', 1: '1_1685566501.735717_251.1', 2: '1_1685562970.274572_93.1', 3: '1_1685566772.905669_489.1'} 2023-05-31T20:59:33.173 WARNING:rgw_multi.tests:zone a2 behind master: shard=3 master=1_1685566772.911499_492.1 target=1_1685566772.905669_489.1 2023-05-31T20:59:38.174 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-31T20:59:38.175 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-31T20:59:38.175 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-05-31T20:59:38.232 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2023-05-31T20:59:38.233 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2023-05-31T20:59:38.283 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:59:38.280+0000 7fb7e5f92780 20 RGW-SYNC:meta: read sync status 2023-05-31T20:59:38.284 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:59:38.281+0000 7fb7e5f92780 20 RGW-SYNC:meta: read sync status 2023-05-31T20:59:38.304 INFO:teuthology.orchestra.run.smithi099.stdout:{ 2023-05-31T20:59:38.304 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_status": { 2023-05-31T20:59:38.304 INFO:teuthology.orchestra.run.smithi099.stdout: "info": { 2023-05-31T20:59:38.304 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "sync", 2023-05-31T20:59:38.304 INFO:teuthology.orchestra.run.smithi099.stdout: "num_shards": 4, 2023-05-31T20:59:38.305 INFO:teuthology.orchestra.run.smithi099.stdout: "period": "2e6123d2-d695-4350-9a01-61a455bc782b", 2023-05-31T20:59:38.305 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2023-05-31T20:59:38.305 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:59:38.305 INFO:teuthology.orchestra.run.smithi099.stdout: "markers": [ 2023-05-31T20:59:38.305 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T20:59:38.306 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 0, 2023-05-31T20:59:38.306 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T20:59:38.306 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2023-05-31T20:59:38.306 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1685566501.757183_260.1", 2023-05-31T20:59:38.306 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T20:59:38.306 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 1, 2023-05-31T20:59:38.306 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T20:59:38.307 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2023-05-31T20:55:01.757183Z", 2023-05-31T20:59:38.307 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2023-05-31T20:59:38.307 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T20:59:38.307 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:59:38.307 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T20:59:38.307 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 1, 2023-05-31T20:59:38.308 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T20:59:38.308 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2023-05-31T20:59:38.308 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1685566501.735717_251.1", 2023-05-31T20:59:38.308 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T20:59:38.308 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2023-05-31T20:59:38.308 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T20:59:38.308 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2023-05-31T20:55:01.735717Z", 2023-05-31T20:59:38.309 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2023-05-31T20:59:38.309 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T20:59:38.309 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:59:38.309 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T20:59:38.309 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 2, 2023-05-31T20:59:38.309 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T20:59:38.309 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2023-05-31T20:59:38.310 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1685562970.274572_93.1", 2023-05-31T20:59:38.310 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T20:59:38.310 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2023-05-31T20:59:38.310 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T20:59:38.310 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2023-05-31T19:56:10.274572Z", 2023-05-31T20:59:38.310 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2023-05-31T20:59:38.310 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T20:59:38.311 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:59:38.311 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T20:59:38.311 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 3, 2023-05-31T20:59:38.311 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T20:59:38.311 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2023-05-31T20:59:38.311 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1685566772.911499_492.1", 2023-05-31T20:59:38.311 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T20:59:38.312 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2023-05-31T20:59:38.312 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T20:59:38.312 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2023-05-31T20:59:32.911499Z", 2023-05-31T20:59:38.312 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2023-05-31T20:59:38.312 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T20:59:38.312 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T20:59:38.312 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2023-05-31T20:59:38.313 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:59:38.313 INFO:teuthology.orchestra.run.smithi099.stdout: "full_sync": { 2023-05-31T20:59:38.313 INFO:teuthology.orchestra.run.smithi099.stdout: "total": 1, 2023-05-31T20:59:38.313 INFO:teuthology.orchestra.run.smithi099.stdout: "complete": 1 2023-05-31T20:59:38.313 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T20:59:38.313 INFO:teuthology.orchestra.run.smithi099.stdout: "current_time": "2023-05-31T20:59:38Z" 2023-05-31T20:59:38.313 INFO:teuthology.orchestra.run.smithi099.stdout:} 2023-05-31T20:59:38.314 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "2e6123d2-d695-4350-9a01-61a455bc782b", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1685566501.757183_260.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-05-31T20:55:01.757183Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1685566501.735717_251.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-05-31T20:55:01.735717Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1685562970.274572_93.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-05-31T19:56:10.274572Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1685566772.911499_492.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-05-31T20:59:32.911499Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-05-31T20:59:38Z" } 2023-05-31T20:59:38.315 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1685566501.757183_260.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-05-31T20:55:01.757183Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1685566501.735717_251.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-05-31T20:55:01.735717Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1685562970.274572_93.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-05-31T19:56:10.274572Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1685566772.911499_492.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-05-31T20:59:32.911499Z', 'realm_epoch': 2}}] 2023-05-31T20:59:38.315 DEBUG:rgw_multi.tests:log_status={0: '1_1685566501.757183_260.1', 1: '1_1685566501.735717_251.1', 2: '1_1685562970.274572_93.1', 3: '1_1685566772.911499_492.1'} 2023-05-31T20:59:38.316 DEBUG:rgw_multi.tests:sync_status={0: '1_1685566501.757183_260.1', 1: '1_1685566501.735717_251.1', 2: '1_1685562970.274572_93.1', 3: '1_1685566772.911499_492.1'} 2023-05-31T20:59:38.316 INFO:rgw_multi.tests:finish meta checkpoint for zone=a2 2023-05-31T20:59:38.316 DEBUG:tasks.rgw.c1.client.1:waiting for process to exit 2023-05-31T20:59:38.317 INFO:teuthology.orchestra.run:waiting for 300 2023-05-31T20:59:38.317 INFO:tasks.rgw.c1.client.1.smithi099.stdout:2023-05-31T20:59:38.313+0000 7f6046086700 -1 received signal: Terminated from /usr/bin/python3 /bin/daemon-helper term radosgw --rgw-frontends beast port=8001 -n client.1 --cluster c1 -k /etc/ceph/c1.client.1.keyring --log-file /var/log/ceph/rgw.c1.client.1.log --rgw_ops_log_socket_path /home/ubuntu/cephtest/rgw.opslog.c1.client.1.sock --foreground --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm (PID: 164688) UID: 0 2023-05-31T20:59:38.318 INFO:tasks.rgw.c1.client.1.smithi099.stdout:2023-05-31T20:59:38.313+0000 7f604a2ef7c0 -1 shutting down 2023-05-31T20:59:40.572 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.1 is failed for ~0s 2023-05-31T20:59:44.419 INFO:tasks.rgw.c1.client.1:Stopped 2023-05-31T20:59:44.420 DEBUG:boto:path=/pklxkd-61/ 2023-05-31T20:59:44.420 DEBUG:boto:auth_path=/pklxkd-61/ 2023-05-31T20:59:44.420 DEBUG:boto:Method: HEAD 2023-05-31T20:59:44.421 DEBUG:boto:Path: /pklxkd-61/ 2023-05-31T20:59:44.421 DEBUG:boto:Data: 2023-05-31T20:59:44.421 DEBUG:boto:Headers: {} 2023-05-31T20:59:44.422 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:59:44.422 DEBUG:boto:Port: 8000 2023-05-31T20:59:44.422 DEBUG:boto:Params: {} 2023-05-31T20:59:44.423 DEBUG:boto:Token: None 2023-05-31T20:59:44.423 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:59:44 GMT /pklxkd-61/ 2023-05-31T20:59:44.424 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:y6Eqs0bJZw/IZkRHhGlQZXh+Yls= 2023-05-31T20:59:44.424 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:59:44 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:y6Eqs0bJZw/IZkRHhGlQZXh+Yls=', 'Content-Length': '0'} 2023-05-31T20:59:44.427 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000007cd3f3d18b0a7757-006477b540-1630-a1'), ('Content-Length', '0'), ('Date', 'Wed, 31 May 2023 20:59:44 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:59:44.427 DEBUG:boto:path=/pklxkd-61/obj11 2023-05-31T20:59:44.428 DEBUG:boto:auth_path=/pklxkd-61/obj11 2023-05-31T20:59:44.428 DEBUG:boto:Method: PUT 2023-05-31T20:59:44.429 DEBUG:boto:Path: /pklxkd-61/obj11 2023-05-31T20:59:44.429 DEBUG:boto:Data: 2023-05-31T20:59:44.429 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-05-31T20:59:44.430 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:59:44.430 DEBUG:boto:Port: 8000 2023-05-31T20:59:44.430 DEBUG:boto:Params: {} 2023-05-31T20:59:44.431 DEBUG:boto:Token: None 2023-05-31T20:59:44.431 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Wed, 31 May 2023 20:59:44 GMT /pklxkd-61/obj11 2023-05-31T20:59:44.431 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:GrPfb4gro2hzl3mOTHtTrnGQAkw= 2023-05-31T20:59:44.432 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Wed, 31 May 2023 20:59:44 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:GrPfb4gro2hzl3mOTHtTrnGQAkw='} 2023-05-31T20:59:44.439 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000a9e0374dd71f6056-006477b540-1630-a1'), ('Date', 'Wed, 31 May 2023 20:59:44 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:59:44.440 DEBUG:boto:path=/pklxkd-61/ 2023-05-31T20:59:44.440 DEBUG:boto:auth_path=/pklxkd-61/ 2023-05-31T20:59:44.440 DEBUG:boto:Method: HEAD 2023-05-31T20:59:44.441 DEBUG:boto:Path: /pklxkd-61/ 2023-05-31T20:59:44.441 DEBUG:boto:Data: 2023-05-31T20:59:44.441 DEBUG:boto:Headers: {} 2023-05-31T20:59:44.442 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:59:44.442 DEBUG:boto:Port: 8000 2023-05-31T20:59:44.442 DEBUG:boto:Params: {} 2023-05-31T20:59:44.443 DEBUG:boto:Token: None 2023-05-31T20:59:44.443 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:59:44 GMT /pklxkd-61/ 2023-05-31T20:59:44.443 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:y6Eqs0bJZw/IZkRHhGlQZXh+Yls= 2023-05-31T20:59:44.444 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:59:44 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:y6Eqs0bJZw/IZkRHhGlQZXh+Yls=', 'Content-Length': '0'} 2023-05-31T20:59:44.446 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '3'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000067b9dc82979237dc-006477b540-1630-a1'), ('Content-Length', '0'), ('Date', 'Wed, 31 May 2023 20:59:44 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:59:44.447 DEBUG:boto:path=/pklxkd-61/obj22 2023-05-31T20:59:44.447 DEBUG:boto:auth_path=/pklxkd-61/obj22 2023-05-31T20:59:44.447 DEBUG:boto:Method: PUT 2023-05-31T20:59:44.448 DEBUG:boto:Path: /pklxkd-61/obj22 2023-05-31T20:59:44.448 DEBUG:boto:Data: 2023-05-31T20:59:44.448 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-05-31T20:59:44.449 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:59:44.449 DEBUG:boto:Port: 8000 2023-05-31T20:59:44.449 DEBUG:boto:Params: {} 2023-05-31T20:59:44.450 DEBUG:boto:Token: None 2023-05-31T20:59:44.450 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Wed, 31 May 2023 20:59:44 GMT /pklxkd-61/obj22 2023-05-31T20:59:44.450 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:B45Yp32vw1qxoEOjeT4oApMt0rs= 2023-05-31T20:59:44.451 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Wed, 31 May 2023 20:59:44 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:B45Yp32vw1qxoEOjeT4oApMt0rs='} 2023-05-31T20:59:44.455 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000028440106702f8da-006477b540-1630-a1'), ('Date', 'Wed, 31 May 2023 20:59:44 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:59:44.456 DEBUG:boto:path=/pklxkd-61/ 2023-05-31T20:59:44.456 DEBUG:boto:auth_path=/pklxkd-61/ 2023-05-31T20:59:44.457 DEBUG:boto:Method: HEAD 2023-05-31T20:59:44.457 DEBUG:boto:Path: /pklxkd-61/ 2023-05-31T20:59:44.457 DEBUG:boto:Data: 2023-05-31T20:59:44.458 DEBUG:boto:Headers: {} 2023-05-31T20:59:44.458 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:59:44.458 DEBUG:boto:Port: 8000 2023-05-31T20:59:44.459 DEBUG:boto:Params: {} 2023-05-31T20:59:44.459 DEBUG:boto:Token: None 2023-05-31T20:59:44.459 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:59:44 GMT /pklxkd-61/ 2023-05-31T20:59:44.460 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:y6Eqs0bJZw/IZkRHhGlQZXh+Yls= 2023-05-31T20:59:44.460 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:59:44 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:y6Eqs0bJZw/IZkRHhGlQZXh+Yls=', 'Content-Length': '0'} 2023-05-31T20:59:44.463 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '2'), ('X-RGW-Bytes-Used', '6'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000df2c52c5312e72ff-006477b540-1630-a1'), ('Content-Length', '0'), ('Date', 'Wed, 31 May 2023 20:59:44 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:59:44.463 DEBUG:boto:path=/pklxkd-61/obj33 2023-05-31T20:59:44.464 DEBUG:boto:auth_path=/pklxkd-61/obj33 2023-05-31T20:59:44.464 DEBUG:boto:Method: PUT 2023-05-31T20:59:44.464 DEBUG:boto:Path: /pklxkd-61/obj33 2023-05-31T20:59:44.465 DEBUG:boto:Data: 2023-05-31T20:59:44.465 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-05-31T20:59:44.465 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:59:44.466 DEBUG:boto:Port: 8000 2023-05-31T20:59:44.466 DEBUG:boto:Params: {} 2023-05-31T20:59:44.466 DEBUG:boto:Token: None 2023-05-31T20:59:44.467 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Wed, 31 May 2023 20:59:44 GMT /pklxkd-61/obj33 2023-05-31T20:59:44.467 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:QPCzGlx+MSbNotfwPqC2JyJOSBw= 2023-05-31T20:59:44.468 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Wed, 31 May 2023 20:59:44 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:QPCzGlx+MSbNotfwPqC2JyJOSBw='} 2023-05-31T20:59:44.512 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000ed4ebc32631403a6-006477b540-1630-a1'), ('Date', 'Wed, 31 May 2023 20:59:44 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:59:44.513 DEBUG:boto:path=/pklxkd-61/ 2023-05-31T20:59:44.513 DEBUG:boto:auth_path=/pklxkd-61/ 2023-05-31T20:59:44.514 DEBUG:boto:Method: HEAD 2023-05-31T20:59:44.514 DEBUG:boto:Path: /pklxkd-61/ 2023-05-31T20:59:44.514 DEBUG:boto:Data: 2023-05-31T20:59:44.515 DEBUG:boto:Headers: {} 2023-05-31T20:59:44.515 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:59:44.515 DEBUG:boto:Port: 8000 2023-05-31T20:59:44.516 DEBUG:boto:Params: {} 2023-05-31T20:59:44.516 DEBUG:boto:Token: None 2023-05-31T20:59:44.517 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:59:44 GMT /pklxkd-61/ 2023-05-31T20:59:44.517 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:y6Eqs0bJZw/IZkRHhGlQZXh+Yls= 2023-05-31T20:59:44.517 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:59:44 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:y6Eqs0bJZw/IZkRHhGlQZXh+Yls=', 'Content-Length': '0'} 2023-05-31T20:59:44.520 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '3'), ('X-RGW-Bytes-Used', '9'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000d255c49b30698e0a-006477b540-1630-a1'), ('Content-Length', '0'), ('Date', 'Wed, 31 May 2023 20:59:44 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:59:44.521 DEBUG:boto:path=/pklxkd-61/obj44 2023-05-31T20:59:44.521 DEBUG:boto:auth_path=/pklxkd-61/obj44 2023-05-31T20:59:44.521 DEBUG:boto:Method: PUT 2023-05-31T20:59:44.522 DEBUG:boto:Path: /pklxkd-61/obj44 2023-05-31T20:59:44.522 DEBUG:boto:Data: 2023-05-31T20:59:44.522 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-05-31T20:59:44.523 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:59:44.523 DEBUG:boto:Port: 8000 2023-05-31T20:59:44.523 DEBUG:boto:Params: {} 2023-05-31T20:59:44.524 DEBUG:boto:Token: None 2023-05-31T20:59:44.524 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Wed, 31 May 2023 20:59:44 GMT /pklxkd-61/obj44 2023-05-31T20:59:44.524 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:Ti18WAe4RHgjkvkCaADOagPpuAc= 2023-05-31T20:59:44.525 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Wed, 31 May 2023 20:59:44 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:Ti18WAe4RHgjkvkCaADOagPpuAc='} 2023-05-31T20:59:44.531 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000007a3822220db3096f-006477b540-1630-a1'), ('Date', 'Wed, 31 May 2023 20:59:44 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:59:44.532 DEBUG:boto:path=/pklxkd-61/ 2023-05-31T20:59:44.532 DEBUG:boto:auth_path=/pklxkd-61/ 2023-05-31T20:59:44.532 DEBUG:boto:Method: HEAD 2023-05-31T20:59:44.533 DEBUG:boto:Path: /pklxkd-61/ 2023-05-31T20:59:44.533 DEBUG:boto:Data: 2023-05-31T20:59:44.533 DEBUG:boto:Headers: {} 2023-05-31T20:59:44.534 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:59:44.534 DEBUG:boto:Port: 8000 2023-05-31T20:59:44.534 DEBUG:boto:Params: {} 2023-05-31T20:59:44.535 DEBUG:boto:Token: None 2023-05-31T20:59:44.535 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:59:44 GMT /pklxkd-61/ 2023-05-31T20:59:44.535 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:y6Eqs0bJZw/IZkRHhGlQZXh+Yls= 2023-05-31T20:59:44.536 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:59:44 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:y6Eqs0bJZw/IZkRHhGlQZXh+Yls=', 'Content-Length': '0'} 2023-05-31T20:59:44.538 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '4'), ('X-RGW-Bytes-Used', '12'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000010479565d624b3d5-006477b540-1630-a1'), ('Content-Length', '0'), ('Date', 'Wed, 31 May 2023 20:59:44 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:59:44.539 DEBUG:boto:path=/pklxkd-61/obj55 2023-05-31T20:59:44.539 DEBUG:boto:auth_path=/pklxkd-61/obj55 2023-05-31T20:59:44.539 DEBUG:boto:Method: PUT 2023-05-31T20:59:44.540 DEBUG:boto:Path: /pklxkd-61/obj55 2023-05-31T20:59:44.540 DEBUG:boto:Data: 2023-05-31T20:59:44.541 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-05-31T20:59:44.541 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:59:44.541 DEBUG:boto:Port: 8000 2023-05-31T20:59:44.542 DEBUG:boto:Params: {} 2023-05-31T20:59:44.542 DEBUG:boto:Token: None 2023-05-31T20:59:44.542 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Wed, 31 May 2023 20:59:44 GMT /pklxkd-61/obj55 2023-05-31T20:59:44.543 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:LAP0rh8/MrHVugXr922dQR5cjM0= 2023-05-31T20:59:44.543 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Wed, 31 May 2023 20:59:44 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:LAP0rh8/MrHVugXr922dQR5cjM0='} 2023-05-31T20:59:44.547 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000bce4d82eb4c95988-006477b540-1630-a1'), ('Date', 'Wed, 31 May 2023 20:59:44 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:59:44.548 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'reshard', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'pklxkd-61', '--num-shards', '17', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-05-31T20:59:44.548 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'reshard', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'pklxkd-61', '--num-shards', '17', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-05-31T20:59:44.549 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket reshard --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --bucket pklxkd-61 --num-shards 17 --yes-i-really-mean-it --cluster c1 --debug-rgw 0 --debug-ms 0 2023-05-31T20:59:44.607 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2023-05-31T20:59:44.607 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2023-05-31T20:59:44.680 INFO:teuthology.orchestra.run.smithi099.stdout:tenant: 2023-05-31T20:59:44.681 INFO:teuthology.orchestra.run.smithi099.stdout:bucket name: pklxkd-61 2023-05-31T20:59:44.684 INFO:teuthology.orchestra.run.smithi099.stdout:total entries: 5 2023-05-31T20:59:44.759 DEBUG:boto:path=/pklxkd-61/ 2023-05-31T20:59:44.759 DEBUG:boto:auth_path=/pklxkd-61/ 2023-05-31T20:59:44.759 DEBUG:boto:Method: HEAD 2023-05-31T20:59:44.760 DEBUG:boto:Path: /pklxkd-61/ 2023-05-31T20:59:44.760 DEBUG:boto:Data: 2023-05-31T20:59:44.760 DEBUG:boto:Headers: {} 2023-05-31T20:59:44.761 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:59:44.761 DEBUG:boto:Port: 8000 2023-05-31T20:59:44.761 DEBUG:boto:Params: {} 2023-05-31T20:59:44.762 DEBUG:boto:Token: None 2023-05-31T20:59:44.762 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:59:44 GMT /pklxkd-61/ 2023-05-31T20:59:44.763 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:y6Eqs0bJZw/IZkRHhGlQZXh+Yls= 2023-05-31T20:59:44.763 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:59:44 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:y6Eqs0bJZw/IZkRHhGlQZXh+Yls=', 'Content-Length': '0'} 2023-05-31T20:59:44.765 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '5'), ('X-RGW-Bytes-Used', '15'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000b10c2583e4b826f3-006477b540-1630-a1'), ('Content-Length', '0'), ('Date', 'Wed, 31 May 2023 20:59:44 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:59:44.766 DEBUG:boto:path=/pklxkd-61/obj17 2023-05-31T20:59:44.766 DEBUG:boto:auth_path=/pklxkd-61/obj17 2023-05-31T20:59:44.767 DEBUG:boto:Method: PUT 2023-05-31T20:59:44.767 DEBUG:boto:Path: /pklxkd-61/obj17 2023-05-31T20:59:44.767 DEBUG:boto:Data: 2023-05-31T20:59:44.768 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-05-31T20:59:44.768 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:59:44.768 DEBUG:boto:Port: 8000 2023-05-31T20:59:44.769 DEBUG:boto:Params: {} 2023-05-31T20:59:44.769 DEBUG:boto:Token: None 2023-05-31T20:59:44.769 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Wed, 31 May 2023 20:59:44 GMT /pklxkd-61/obj17 2023-05-31T20:59:44.770 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:bG1yFhv2mnsf12zqTLQkUoPdxUk= 2023-05-31T20:59:44.770 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Wed, 31 May 2023 20:59:44 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:bG1yFhv2mnsf12zqTLQkUoPdxUk='} 2023-05-31T20:59:44.812 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000a535dadfd09459c1-006477b540-1630-a1'), ('Date', 'Wed, 31 May 2023 20:59:44 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:59:44.813 DEBUG:boto:path=/pklxkd-61/ 2023-05-31T20:59:44.813 DEBUG:boto:auth_path=/pklxkd-61/ 2023-05-31T20:59:44.813 DEBUG:boto:Method: HEAD 2023-05-31T20:59:44.814 DEBUG:boto:Path: /pklxkd-61/ 2023-05-31T20:59:44.814 DEBUG:boto:Data: 2023-05-31T20:59:44.814 DEBUG:boto:Headers: {} 2023-05-31T20:59:44.815 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:59:44.815 DEBUG:boto:Port: 8000 2023-05-31T20:59:44.815 DEBUG:boto:Params: {} 2023-05-31T20:59:44.816 DEBUG:boto:Token: None 2023-05-31T20:59:44.816 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:59:44 GMT /pklxkd-61/ 2023-05-31T20:59:44.816 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:y6Eqs0bJZw/IZkRHhGlQZXh+Yls= 2023-05-31T20:59:44.817 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:59:44 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:y6Eqs0bJZw/IZkRHhGlQZXh+Yls=', 'Content-Length': '0'} 2023-05-31T20:59:44.819 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '6'), ('X-RGW-Bytes-Used', '18'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000091fa36203c08a82c-006477b540-1630-a1'), ('Content-Length', '0'), ('Date', 'Wed, 31 May 2023 20:59:44 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:59:44.820 DEBUG:boto:path=/pklxkd-61/obj34 2023-05-31T20:59:44.820 DEBUG:boto:auth_path=/pklxkd-61/obj34 2023-05-31T20:59:44.820 DEBUG:boto:Method: PUT 2023-05-31T20:59:44.821 DEBUG:boto:Path: /pklxkd-61/obj34 2023-05-31T20:59:44.821 DEBUG:boto:Data: 2023-05-31T20:59:44.821 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-05-31T20:59:44.822 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:59:44.822 DEBUG:boto:Port: 8000 2023-05-31T20:59:44.822 DEBUG:boto:Params: {} 2023-05-31T20:59:44.823 DEBUG:boto:Token: None 2023-05-31T20:59:44.823 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Wed, 31 May 2023 20:59:44 GMT /pklxkd-61/obj34 2023-05-31T20:59:44.824 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:gnGz0Q+D6baeHTFS7mTWgR/B0Ag= 2023-05-31T20:59:44.824 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Wed, 31 May 2023 20:59:44 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:gnGz0Q+D6baeHTFS7mTWgR/B0Ag='} 2023-05-31T20:59:44.830 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000cc381b1ebe1bd447-006477b540-1630-a1'), ('Date', 'Wed, 31 May 2023 20:59:44 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:59:44.831 DEBUG:boto:path=/pklxkd-61/ 2023-05-31T20:59:44.831 DEBUG:boto:auth_path=/pklxkd-61/ 2023-05-31T20:59:44.832 DEBUG:boto:Method: HEAD 2023-05-31T20:59:44.832 DEBUG:boto:Path: /pklxkd-61/ 2023-05-31T20:59:44.832 DEBUG:boto:Data: 2023-05-31T20:59:44.833 DEBUG:boto:Headers: {} 2023-05-31T20:59:44.833 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:59:44.833 DEBUG:boto:Port: 8000 2023-05-31T20:59:44.834 DEBUG:boto:Params: {} 2023-05-31T20:59:44.834 DEBUG:boto:Token: None 2023-05-31T20:59:44.834 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:59:44 GMT /pklxkd-61/ 2023-05-31T20:59:44.835 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:y6Eqs0bJZw/IZkRHhGlQZXh+Yls= 2023-05-31T20:59:44.835 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:59:44 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:y6Eqs0bJZw/IZkRHhGlQZXh+Yls=', 'Content-Length': '0'} 2023-05-31T20:59:44.837 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '7'), ('X-RGW-Bytes-Used', '21'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000096865d6a44a7ecd-006477b540-1630-a1'), ('Content-Length', '0'), ('Date', 'Wed, 31 May 2023 20:59:44 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:59:44.838 DEBUG:boto:path=/pklxkd-61/obj51 2023-05-31T20:59:44.838 DEBUG:boto:auth_path=/pklxkd-61/obj51 2023-05-31T20:59:44.839 DEBUG:boto:Method: PUT 2023-05-31T20:59:44.839 DEBUG:boto:Path: /pklxkd-61/obj51 2023-05-31T20:59:44.839 DEBUG:boto:Data: 2023-05-31T20:59:44.840 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-05-31T20:59:44.840 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:59:44.840 DEBUG:boto:Port: 8000 2023-05-31T20:59:44.841 DEBUG:boto:Params: {} 2023-05-31T20:59:44.841 DEBUG:boto:Token: None 2023-05-31T20:59:44.842 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Wed, 31 May 2023 20:59:44 GMT /pklxkd-61/obj51 2023-05-31T20:59:44.842 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:i/60yz2OvKYQWq/xxsob5MAZlmw= 2023-05-31T20:59:44.842 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Wed, 31 May 2023 20:59:44 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:i/60yz2OvKYQWq/xxsob5MAZlmw='} 2023-05-31T20:59:44.884 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000cce209ad644df74c-006477b540-1630-a1'), ('Date', 'Wed, 31 May 2023 20:59:44 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:59:44.885 DEBUG:boto:path=/pklxkd-61/ 2023-05-31T20:59:44.885 DEBUG:boto:auth_path=/pklxkd-61/ 2023-05-31T20:59:44.885 DEBUG:boto:Method: HEAD 2023-05-31T20:59:44.886 DEBUG:boto:Path: /pklxkd-61/ 2023-05-31T20:59:44.886 DEBUG:boto:Data: 2023-05-31T20:59:44.886 DEBUG:boto:Headers: {} 2023-05-31T20:59:44.887 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:59:44.887 DEBUG:boto:Port: 8000 2023-05-31T20:59:44.887 DEBUG:boto:Params: {} 2023-05-31T20:59:44.888 DEBUG:boto:Token: None 2023-05-31T20:59:44.888 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:59:44 GMT /pklxkd-61/ 2023-05-31T20:59:44.888 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:y6Eqs0bJZw/IZkRHhGlQZXh+Yls= 2023-05-31T20:59:44.889 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:59:44 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:y6Eqs0bJZw/IZkRHhGlQZXh+Yls=', 'Content-Length': '0'} 2023-05-31T20:59:44.891 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '8'), ('X-RGW-Bytes-Used', '24'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000056fd5cc8cbb90ced-006477b540-1630-a1'), ('Content-Length', '0'), ('Date', 'Wed, 31 May 2023 20:59:44 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:59:44.892 DEBUG:boto:path=/pklxkd-61/obj68 2023-05-31T20:59:44.892 DEBUG:boto:auth_path=/pklxkd-61/obj68 2023-05-31T20:59:44.892 DEBUG:boto:Method: PUT 2023-05-31T20:59:44.893 DEBUG:boto:Path: /pklxkd-61/obj68 2023-05-31T20:59:44.893 DEBUG:boto:Data: 2023-05-31T20:59:44.893 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-05-31T20:59:44.894 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:59:44.894 DEBUG:boto:Port: 8000 2023-05-31T20:59:44.894 DEBUG:boto:Params: {} 2023-05-31T20:59:44.895 DEBUG:boto:Token: None 2023-05-31T20:59:44.895 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Wed, 31 May 2023 20:59:44 GMT /pklxkd-61/obj68 2023-05-31T20:59:44.896 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:mPWXJoaa+dmxlcIvXJd7rOFnkl4= 2023-05-31T20:59:44.896 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Wed, 31 May 2023 20:59:44 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:mPWXJoaa+dmxlcIvXJd7rOFnkl4='} 2023-05-31T20:59:44.902 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000000ff014e8f6199859-006477b540-1630-a1'), ('Date', 'Wed, 31 May 2023 20:59:44 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:59:44.903 DEBUG:boto:path=/pklxkd-61/ 2023-05-31T20:59:44.903 DEBUG:boto:auth_path=/pklxkd-61/ 2023-05-31T20:59:44.904 DEBUG:boto:Method: HEAD 2023-05-31T20:59:44.904 DEBUG:boto:Path: /pklxkd-61/ 2023-05-31T20:59:44.904 DEBUG:boto:Data: 2023-05-31T20:59:44.905 DEBUG:boto:Headers: {} 2023-05-31T20:59:44.905 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:59:44.905 DEBUG:boto:Port: 8000 2023-05-31T20:59:44.905 DEBUG:boto:Params: {} 2023-05-31T20:59:44.906 DEBUG:boto:Token: None 2023-05-31T20:59:44.906 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:59:44 GMT /pklxkd-61/ 2023-05-31T20:59:44.907 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:y6Eqs0bJZw/IZkRHhGlQZXh+Yls= 2023-05-31T20:59:44.907 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:59:44 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:y6Eqs0bJZw/IZkRHhGlQZXh+Yls=', 'Content-Length': '0'} 2023-05-31T20:59:44.909 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '9'), ('X-RGW-Bytes-Used', '27'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000dac72bcca9000128-006477b540-1630-a1'), ('Content-Length', '0'), ('Date', 'Wed, 31 May 2023 20:59:44 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:59:44.909 DEBUG:boto:path=/pklxkd-61/obj85 2023-05-31T20:59:44.910 DEBUG:boto:auth_path=/pklxkd-61/obj85 2023-05-31T20:59:44.910 DEBUG:boto:Method: PUT 2023-05-31T20:59:44.911 DEBUG:boto:Path: /pklxkd-61/obj85 2023-05-31T20:59:44.911 DEBUG:boto:Data: 2023-05-31T20:59:44.911 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-05-31T20:59:44.912 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:59:44.912 DEBUG:boto:Port: 8000 2023-05-31T20:59:44.912 DEBUG:boto:Params: {} 2023-05-31T20:59:44.913 DEBUG:boto:Token: None 2023-05-31T20:59:44.913 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Wed, 31 May 2023 20:59:44 GMT /pklxkd-61/obj85 2023-05-31T20:59:44.913 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:nGFl53H47QJP1Ai1KsTIcqX5apU= 2023-05-31T20:59:44.914 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Wed, 31 May 2023 20:59:44 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:nGFl53H47QJP1Ai1KsTIcqX5apU='} 2023-05-31T20:59:44.956 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000002b75788e2ca632eb-006477b540-1630-a1'), ('Date', 'Wed, 31 May 2023 20:59:44 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:59:44.957 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'reshard', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'pklxkd-61', '--num-shards', '19', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-05-31T20:59:44.957 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'reshard', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'pklxkd-61', '--num-shards', '19', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-05-31T20:59:44.958 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket reshard --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --bucket pklxkd-61 --num-shards 19 --yes-i-really-mean-it --cluster c1 --debug-rgw 0 --debug-ms 0 2023-05-31T20:59:45.010 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2023-05-31T20:59:45.010 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2023-05-31T20:59:45.066 INFO:teuthology.orchestra.run.smithi099.stdout:tenant: 2023-05-31T20:59:45.067 INFO:teuthology.orchestra.run.smithi099.stdout:bucket name: pklxkd-61 2023-05-31T20:59:45.073 INFO:teuthology.orchestra.run.smithi099.stdout:total entries: 10 2023-05-31T20:59:45.153 DEBUG:boto:path=/pklxkd-61/ 2023-05-31T20:59:45.154 DEBUG:boto:auth_path=/pklxkd-61/ 2023-05-31T20:59:45.154 DEBUG:boto:Method: HEAD 2023-05-31T20:59:45.155 DEBUG:boto:Path: /pklxkd-61/ 2023-05-31T20:59:45.155 DEBUG:boto:Data: 2023-05-31T20:59:45.155 DEBUG:boto:Headers: {} 2023-05-31T20:59:45.156 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:59:45.156 DEBUG:boto:Port: 8000 2023-05-31T20:59:45.156 DEBUG:boto:Params: {} 2023-05-31T20:59:45.157 DEBUG:boto:Token: None 2023-05-31T20:59:45.157 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:59:45 GMT /pklxkd-61/ 2023-05-31T20:59:45.158 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:KDZjyCz7mNQWuhX5fKZ7n337Nqk= 2023-05-31T20:59:45.158 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:59:45 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:KDZjyCz7mNQWuhX5fKZ7n337Nqk=', 'Content-Length': '0'} 2023-05-31T20:59:45.161 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '10'), ('X-RGW-Bytes-Used', '30'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000715f50f40bfbff10-006477b541-1630-a1'), ('Content-Length', '0'), ('Date', 'Wed, 31 May 2023 20:59:45 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:59:45.161 DEBUG:boto:path=/pklxkd-61/obj19 2023-05-31T20:59:45.162 DEBUG:boto:auth_path=/pklxkd-61/obj19 2023-05-31T20:59:45.162 DEBUG:boto:Method: PUT 2023-05-31T20:59:45.163 DEBUG:boto:Path: /pklxkd-61/obj19 2023-05-31T20:59:45.163 DEBUG:boto:Data: 2023-05-31T20:59:45.163 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-05-31T20:59:45.164 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:59:45.164 DEBUG:boto:Port: 8000 2023-05-31T20:59:45.164 DEBUG:boto:Params: {} 2023-05-31T20:59:45.165 DEBUG:boto:Token: None 2023-05-31T20:59:45.165 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Wed, 31 May 2023 20:59:45 GMT /pklxkd-61/obj19 2023-05-31T20:59:45.165 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:arCmY87lL6amPj2LeftfCwxbROU= 2023-05-31T20:59:45.166 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Wed, 31 May 2023 20:59:45 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:arCmY87lL6amPj2LeftfCwxbROU='} 2023-05-31T20:59:45.171 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000c31d975e8b067571-006477b541-1630-a1'), ('Date', 'Wed, 31 May 2023 20:59:45 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:59:45.172 DEBUG:boto:path=/pklxkd-61/ 2023-05-31T20:59:45.172 DEBUG:boto:auth_path=/pklxkd-61/ 2023-05-31T20:59:45.172 DEBUG:boto:Method: HEAD 2023-05-31T20:59:45.173 DEBUG:boto:Path: /pklxkd-61/ 2023-05-31T20:59:45.173 DEBUG:boto:Data: 2023-05-31T20:59:45.173 DEBUG:boto:Headers: {} 2023-05-31T20:59:45.174 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:59:45.174 DEBUG:boto:Port: 8000 2023-05-31T20:59:45.174 DEBUG:boto:Params: {} 2023-05-31T20:59:45.175 DEBUG:boto:Token: None 2023-05-31T20:59:45.175 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:59:45 GMT /pklxkd-61/ 2023-05-31T20:59:45.176 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:KDZjyCz7mNQWuhX5fKZ7n337Nqk= 2023-05-31T20:59:45.176 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:59:45 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:KDZjyCz7mNQWuhX5fKZ7n337Nqk=', 'Content-Length': '0'} 2023-05-31T20:59:45.178 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '11'), ('X-RGW-Bytes-Used', '33'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000018cd0b581813aa86-006477b541-1630-a1'), ('Content-Length', '0'), ('Date', 'Wed, 31 May 2023 20:59:45 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:59:45.179 DEBUG:boto:path=/pklxkd-61/obj38 2023-05-31T20:59:45.179 DEBUG:boto:auth_path=/pklxkd-61/obj38 2023-05-31T20:59:45.180 DEBUG:boto:Method: PUT 2023-05-31T20:59:45.180 DEBUG:boto:Path: /pklxkd-61/obj38 2023-05-31T20:59:45.180 DEBUG:boto:Data: 2023-05-31T20:59:45.181 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-05-31T20:59:45.181 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:59:45.181 DEBUG:boto:Port: 8000 2023-05-31T20:59:45.182 DEBUG:boto:Params: {} 2023-05-31T20:59:45.182 DEBUG:boto:Token: None 2023-05-31T20:59:45.182 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Wed, 31 May 2023 20:59:45 GMT /pklxkd-61/obj38 2023-05-31T20:59:45.183 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:zgzNSAcMWEFAnqZVJdn1SlaFEdc= 2023-05-31T20:59:45.183 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Wed, 31 May 2023 20:59:45 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:zgzNSAcMWEFAnqZVJdn1SlaFEdc='} 2023-05-31T20:59:45.187 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000003cbbb9611e213c47-006477b541-1630-a1'), ('Date', 'Wed, 31 May 2023 20:59:45 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:59:45.188 DEBUG:boto:path=/pklxkd-61/ 2023-05-31T20:59:45.188 DEBUG:boto:auth_path=/pklxkd-61/ 2023-05-31T20:59:45.188 DEBUG:boto:Method: HEAD 2023-05-31T20:59:45.189 DEBUG:boto:Path: /pklxkd-61/ 2023-05-31T20:59:45.189 DEBUG:boto:Data: 2023-05-31T20:59:45.189 DEBUG:boto:Headers: {} 2023-05-31T20:59:45.190 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:59:45.190 DEBUG:boto:Port: 8000 2023-05-31T20:59:45.190 DEBUG:boto:Params: {} 2023-05-31T20:59:45.191 DEBUG:boto:Token: None 2023-05-31T20:59:45.191 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:59:45 GMT /pklxkd-61/ 2023-05-31T20:59:45.191 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:KDZjyCz7mNQWuhX5fKZ7n337Nqk= 2023-05-31T20:59:45.192 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:59:45 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:KDZjyCz7mNQWuhX5fKZ7n337Nqk=', 'Content-Length': '0'} 2023-05-31T20:59:45.194 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '12'), ('X-RGW-Bytes-Used', '36'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000b8fc2ffc05769dc7-006477b541-1630-a1'), ('Content-Length', '0'), ('Date', 'Wed, 31 May 2023 20:59:45 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:59:45.195 DEBUG:boto:path=/pklxkd-61/obj57 2023-05-31T20:59:45.195 DEBUG:boto:auth_path=/pklxkd-61/obj57 2023-05-31T20:59:45.195 DEBUG:boto:Method: PUT 2023-05-31T20:59:45.196 DEBUG:boto:Path: /pklxkd-61/obj57 2023-05-31T20:59:45.196 DEBUG:boto:Data: 2023-05-31T20:59:45.196 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-05-31T20:59:45.197 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:59:45.197 DEBUG:boto:Port: 8000 2023-05-31T20:59:45.197 DEBUG:boto:Params: {} 2023-05-31T20:59:45.198 DEBUG:boto:Token: None 2023-05-31T20:59:45.198 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Wed, 31 May 2023 20:59:45 GMT /pklxkd-61/obj57 2023-05-31T20:59:45.199 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:K+foZN8HV2JxECV3XUA/o1+kgGc= 2023-05-31T20:59:45.199 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Wed, 31 May 2023 20:59:45 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:K+foZN8HV2JxECV3XUA/o1+kgGc='} 2023-05-31T20:59:45.244 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000002dc4599d349df09f-006477b541-1630-a1'), ('Date', 'Wed, 31 May 2023 20:59:45 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:59:45.245 DEBUG:boto:path=/pklxkd-61/ 2023-05-31T20:59:45.246 DEBUG:boto:auth_path=/pklxkd-61/ 2023-05-31T20:59:45.246 DEBUG:boto:Method: HEAD 2023-05-31T20:59:45.246 DEBUG:boto:Path: /pklxkd-61/ 2023-05-31T20:59:45.247 DEBUG:boto:Data: 2023-05-31T20:59:45.247 DEBUG:boto:Headers: {} 2023-05-31T20:59:45.247 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:59:45.248 DEBUG:boto:Port: 8000 2023-05-31T20:59:45.248 DEBUG:boto:Params: {} 2023-05-31T20:59:45.249 DEBUG:boto:Token: None 2023-05-31T20:59:45.249 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:59:45 GMT /pklxkd-61/ 2023-05-31T20:59:45.250 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:KDZjyCz7mNQWuhX5fKZ7n337Nqk= 2023-05-31T20:59:45.250 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:59:45 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:KDZjyCz7mNQWuhX5fKZ7n337Nqk=', 'Content-Length': '0'} 2023-05-31T20:59:45.253 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '13'), ('X-RGW-Bytes-Used', '39'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000001d3a11162987d3a2-006477b541-1630-a1'), ('Content-Length', '0'), ('Date', 'Wed, 31 May 2023 20:59:45 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:59:45.254 DEBUG:boto:path=/pklxkd-61/obj76 2023-05-31T20:59:45.254 DEBUG:boto:auth_path=/pklxkd-61/obj76 2023-05-31T20:59:45.254 DEBUG:boto:Method: PUT 2023-05-31T20:59:45.255 DEBUG:boto:Path: /pklxkd-61/obj76 2023-05-31T20:59:45.255 DEBUG:boto:Data: 2023-05-31T20:59:45.255 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-05-31T20:59:45.256 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:59:45.256 DEBUG:boto:Port: 8000 2023-05-31T20:59:45.256 DEBUG:boto:Params: {} 2023-05-31T20:59:45.257 DEBUG:boto:Token: None 2023-05-31T20:59:45.257 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Wed, 31 May 2023 20:59:45 GMT /pklxkd-61/obj76 2023-05-31T20:59:45.257 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:N8Eusf8H7AwSR2gNX07uGDmjqw0= 2023-05-31T20:59:45.258 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Wed, 31 May 2023 20:59:45 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:N8Eusf8H7AwSR2gNX07uGDmjqw0='} 2023-05-31T20:59:45.264 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000756052c33833b7a7-006477b541-1630-a1'), ('Date', 'Wed, 31 May 2023 20:59:45 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:59:45.264 DEBUG:boto:path=/pklxkd-61/ 2023-05-31T20:59:45.265 DEBUG:boto:auth_path=/pklxkd-61/ 2023-05-31T20:59:45.265 DEBUG:boto:Method: HEAD 2023-05-31T20:59:45.265 DEBUG:boto:Path: /pklxkd-61/ 2023-05-31T20:59:45.266 DEBUG:boto:Data: 2023-05-31T20:59:45.266 DEBUG:boto:Headers: {} 2023-05-31T20:59:45.266 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:59:45.267 DEBUG:boto:Port: 8000 2023-05-31T20:59:45.267 DEBUG:boto:Params: {} 2023-05-31T20:59:45.267 DEBUG:boto:Token: None 2023-05-31T20:59:45.268 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:59:45 GMT /pklxkd-61/ 2023-05-31T20:59:45.268 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:KDZjyCz7mNQWuhX5fKZ7n337Nqk= 2023-05-31T20:59:45.269 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:59:45 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:KDZjyCz7mNQWuhX5fKZ7n337Nqk=', 'Content-Length': '0'} 2023-05-31T20:59:45.271 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '14'), ('X-RGW-Bytes-Used', '42'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000bdac9251a2c51173-006477b541-1630-a1'), ('Content-Length', '0'), ('Date', 'Wed, 31 May 2023 20:59:45 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:59:45.272 DEBUG:boto:path=/pklxkd-61/obj95 2023-05-31T20:59:45.272 DEBUG:boto:auth_path=/pklxkd-61/obj95 2023-05-31T20:59:45.272 DEBUG:boto:Method: PUT 2023-05-31T20:59:45.273 DEBUG:boto:Path: /pklxkd-61/obj95 2023-05-31T20:59:45.273 DEBUG:boto:Data: 2023-05-31T20:59:45.273 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-05-31T20:59:45.274 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:59:45.274 DEBUG:boto:Port: 8000 2023-05-31T20:59:45.274 DEBUG:boto:Params: {} 2023-05-31T20:59:45.275 DEBUG:boto:Token: None 2023-05-31T20:59:45.275 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Wed, 31 May 2023 20:59:45 GMT /pklxkd-61/obj95 2023-05-31T20:59:45.276 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:Hq/vy8ulV6e6mSoK5ytMOSSuvjM= 2023-05-31T20:59:45.276 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Wed, 31 May 2023 20:59:45 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:Hq/vy8ulV6e6mSoK5ytMOSSuvjM='} 2023-05-31T20:59:45.320 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000f2d5d3469ceff119-006477b541-1630-a1'), ('Date', 'Wed, 31 May 2023 20:59:45 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:59:45.321 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'reshard', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'pklxkd-61', '--num-shards', '23', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-05-31T20:59:45.322 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'reshard', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'pklxkd-61', '--num-shards', '23', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-05-31T20:59:45.322 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket reshard --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --bucket pklxkd-61 --num-shards 23 --yes-i-really-mean-it --cluster c1 --debug-rgw 0 --debug-ms 0 2023-05-31T20:59:45.374 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2023-05-31T20:59:45.375 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2023-05-31T20:59:45.428 INFO:teuthology.orchestra.run.smithi099.stdout:tenant: 2023-05-31T20:59:45.428 INFO:teuthology.orchestra.run.smithi099.stdout:bucket name: pklxkd-61 2023-05-31T20:59:45.434 INFO:teuthology.orchestra.run.smithi099.stdout:total entries: 15 2023-05-31T20:59:45.506 DEBUG:boto:path=/pklxkd-61/ 2023-05-31T20:59:45.506 DEBUG:boto:auth_path=/pklxkd-61/ 2023-05-31T20:59:45.507 DEBUG:boto:Method: HEAD 2023-05-31T20:59:45.507 DEBUG:boto:Path: /pklxkd-61/ 2023-05-31T20:59:45.508 DEBUG:boto:Data: 2023-05-31T20:59:45.508 DEBUG:boto:Headers: {} 2023-05-31T20:59:45.508 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:59:45.509 DEBUG:boto:Port: 8000 2023-05-31T20:59:45.509 DEBUG:boto:Params: {} 2023-05-31T20:59:45.509 DEBUG:boto:Token: None 2023-05-31T20:59:45.510 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:59:45 GMT /pklxkd-61/ 2023-05-31T20:59:45.510 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:KDZjyCz7mNQWuhX5fKZ7n337Nqk= 2023-05-31T20:59:45.511 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:59:45 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:KDZjyCz7mNQWuhX5fKZ7n337Nqk=', 'Content-Length': '0'} 2023-05-31T20:59:45.514 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '15'), ('X-RGW-Bytes-Used', '45'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000002cdadb30756dded3-006477b541-1630-a1'), ('Content-Length', '0'), ('Date', 'Wed, 31 May 2023 20:59:45 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:59:45.514 DEBUG:boto:path=/pklxkd-61/obj23 2023-05-31T20:59:45.515 DEBUG:boto:auth_path=/pklxkd-61/obj23 2023-05-31T20:59:45.515 DEBUG:boto:Method: PUT 2023-05-31T20:59:45.515 DEBUG:boto:Path: /pklxkd-61/obj23 2023-05-31T20:59:45.516 DEBUG:boto:Data: 2023-05-31T20:59:45.516 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-05-31T20:59:45.516 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:59:45.517 DEBUG:boto:Port: 8000 2023-05-31T20:59:45.517 DEBUG:boto:Params: {} 2023-05-31T20:59:45.518 DEBUG:boto:Token: None 2023-05-31T20:59:45.518 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Wed, 31 May 2023 20:59:45 GMT /pklxkd-61/obj23 2023-05-31T20:59:45.518 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:aHAEZR4GYU4RT4TRJsmFhxFQ5jQ= 2023-05-31T20:59:45.519 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Wed, 31 May 2023 20:59:45 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:aHAEZR4GYU4RT4TRJsmFhxFQ5jQ='} 2023-05-31T20:59:45.528 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000003231bbe37586710b-006477b541-1630-a1'), ('Date', 'Wed, 31 May 2023 20:59:45 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:59:45.528 DEBUG:boto:path=/pklxkd-61/ 2023-05-31T20:59:45.529 DEBUG:boto:auth_path=/pklxkd-61/ 2023-05-31T20:59:45.529 DEBUG:boto:Method: HEAD 2023-05-31T20:59:45.530 DEBUG:boto:Path: /pklxkd-61/ 2023-05-31T20:59:45.530 DEBUG:boto:Data: 2023-05-31T20:59:45.530 DEBUG:boto:Headers: {} 2023-05-31T20:59:45.531 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:59:45.531 DEBUG:boto:Port: 8000 2023-05-31T20:59:45.531 DEBUG:boto:Params: {} 2023-05-31T20:59:45.532 DEBUG:boto:Token: None 2023-05-31T20:59:45.532 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:59:45 GMT /pklxkd-61/ 2023-05-31T20:59:45.532 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:KDZjyCz7mNQWuhX5fKZ7n337Nqk= 2023-05-31T20:59:45.533 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:59:45 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:KDZjyCz7mNQWuhX5fKZ7n337Nqk=', 'Content-Length': '0'} 2023-05-31T20:59:45.535 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '16'), ('X-RGW-Bytes-Used', '48'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000bfccb3dcb138e6cf-006477b541-1630-a1'), ('Content-Length', '0'), ('Date', 'Wed, 31 May 2023 20:59:45 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:59:45.536 DEBUG:boto:path=/pklxkd-61/obj46 2023-05-31T20:59:45.536 DEBUG:boto:auth_path=/pklxkd-61/obj46 2023-05-31T20:59:45.537 DEBUG:boto:Method: PUT 2023-05-31T20:59:45.537 DEBUG:boto:Path: /pklxkd-61/obj46 2023-05-31T20:59:45.537 DEBUG:boto:Data: 2023-05-31T20:59:45.538 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-05-31T20:59:45.538 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:59:45.538 DEBUG:boto:Port: 8000 2023-05-31T20:59:45.539 DEBUG:boto:Params: {} 2023-05-31T20:59:45.539 DEBUG:boto:Token: None 2023-05-31T20:59:45.539 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Wed, 31 May 2023 20:59:45 GMT /pklxkd-61/obj46 2023-05-31T20:59:45.540 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:VIeg81uFn15rDb4bbipZEtKvwBQ= 2023-05-31T20:59:45.540 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Wed, 31 May 2023 20:59:45 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:VIeg81uFn15rDb4bbipZEtKvwBQ='} 2023-05-31T20:59:45.548 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000772960269e2b4119-006477b541-1630-a1'), ('Date', 'Wed, 31 May 2023 20:59:45 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:59:45.549 DEBUG:boto:path=/pklxkd-61/ 2023-05-31T20:59:45.549 DEBUG:boto:auth_path=/pklxkd-61/ 2023-05-31T20:59:45.549 DEBUG:boto:Method: HEAD 2023-05-31T20:59:45.550 DEBUG:boto:Path: /pklxkd-61/ 2023-05-31T20:59:45.550 DEBUG:boto:Data: 2023-05-31T20:59:45.550 DEBUG:boto:Headers: {} 2023-05-31T20:59:45.551 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:59:45.551 DEBUG:boto:Port: 8000 2023-05-31T20:59:45.551 DEBUG:boto:Params: {} 2023-05-31T20:59:45.552 DEBUG:boto:Token: None 2023-05-31T20:59:45.552 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:59:45 GMT /pklxkd-61/ 2023-05-31T20:59:45.553 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:KDZjyCz7mNQWuhX5fKZ7n337Nqk= 2023-05-31T20:59:45.553 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:59:45 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:KDZjyCz7mNQWuhX5fKZ7n337Nqk=', 'Content-Length': '0'} 2023-05-31T20:59:45.556 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '17'), ('X-RGW-Bytes-Used', '51'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000001a11353b57373216-006477b541-1630-a1'), ('Content-Length', '0'), ('Date', 'Wed, 31 May 2023 20:59:45 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:59:45.556 DEBUG:boto:path=/pklxkd-61/obj69 2023-05-31T20:59:45.557 DEBUG:boto:auth_path=/pklxkd-61/obj69 2023-05-31T20:59:45.557 DEBUG:boto:Method: PUT 2023-05-31T20:59:45.558 DEBUG:boto:Path: /pklxkd-61/obj69 2023-05-31T20:59:45.558 DEBUG:boto:Data: 2023-05-31T20:59:45.558 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-05-31T20:59:45.559 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:59:45.559 DEBUG:boto:Port: 8000 2023-05-31T20:59:45.559 DEBUG:boto:Params: {} 2023-05-31T20:59:45.560 DEBUG:boto:Token: None 2023-05-31T20:59:45.560 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Wed, 31 May 2023 20:59:45 GMT /pklxkd-61/obj69 2023-05-31T20:59:45.560 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:TbA0+bRVlE7ZdNqn+HF7Gd2/ZzI= 2023-05-31T20:59:45.561 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Wed, 31 May 2023 20:59:45 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:TbA0+bRVlE7ZdNqn+HF7Gd2/ZzI='} 2023-05-31T20:59:45.604 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000f8e21b0725347818-006477b541-1630-a1'), ('Date', 'Wed, 31 May 2023 20:59:45 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:59:45.605 DEBUG:boto:path=/pklxkd-61/ 2023-05-31T20:59:45.606 DEBUG:boto:auth_path=/pklxkd-61/ 2023-05-31T20:59:45.606 DEBUG:boto:Method: HEAD 2023-05-31T20:59:45.606 DEBUG:boto:Path: /pklxkd-61/ 2023-05-31T20:59:45.607 DEBUG:boto:Data: 2023-05-31T20:59:45.607 DEBUG:boto:Headers: {} 2023-05-31T20:59:45.607 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:59:45.608 DEBUG:boto:Port: 8000 2023-05-31T20:59:45.608 DEBUG:boto:Params: {} 2023-05-31T20:59:45.608 DEBUG:boto:Token: None 2023-05-31T20:59:45.609 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:59:45 GMT /pklxkd-61/ 2023-05-31T20:59:45.609 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:KDZjyCz7mNQWuhX5fKZ7n337Nqk= 2023-05-31T20:59:45.610 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:59:45 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:KDZjyCz7mNQWuhX5fKZ7n337Nqk=', 'Content-Length': '0'} 2023-05-31T20:59:45.612 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '18'), ('X-RGW-Bytes-Used', '54'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000ad37efef31f566f3-006477b541-1630-a1'), ('Content-Length', '0'), ('Date', 'Wed, 31 May 2023 20:59:45 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:59:45.613 DEBUG:boto:path=/pklxkd-61/obj92 2023-05-31T20:59:45.613 DEBUG:boto:auth_path=/pklxkd-61/obj92 2023-05-31T20:59:45.614 DEBUG:boto:Method: PUT 2023-05-31T20:59:45.614 DEBUG:boto:Path: /pklxkd-61/obj92 2023-05-31T20:59:45.614 DEBUG:boto:Data: 2023-05-31T20:59:45.615 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-05-31T20:59:45.615 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:59:45.615 DEBUG:boto:Port: 8000 2023-05-31T20:59:45.616 DEBUG:boto:Params: {} 2023-05-31T20:59:45.616 DEBUG:boto:Token: None 2023-05-31T20:59:45.616 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Wed, 31 May 2023 20:59:45 GMT /pklxkd-61/obj92 2023-05-31T20:59:45.617 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:phchrVz/V1tNqu4u0YlsVv/OdNQ= 2023-05-31T20:59:45.617 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Wed, 31 May 2023 20:59:45 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:phchrVz/V1tNqu4u0YlsVv/OdNQ='} 2023-05-31T20:59:45.625 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000e5308560144a6af9-006477b541-1630-a1'), ('Date', 'Wed, 31 May 2023 20:59:45 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:59:45.626 DEBUG:boto:path=/pklxkd-61/ 2023-05-31T20:59:45.626 DEBUG:boto:auth_path=/pklxkd-61/ 2023-05-31T20:59:45.627 DEBUG:boto:Method: HEAD 2023-05-31T20:59:45.627 DEBUG:boto:Path: /pklxkd-61/ 2023-05-31T20:59:45.627 DEBUG:boto:Data: 2023-05-31T20:59:45.628 DEBUG:boto:Headers: {} 2023-05-31T20:59:45.628 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:59:45.628 DEBUG:boto:Port: 8000 2023-05-31T20:59:45.629 DEBUG:boto:Params: {} 2023-05-31T20:59:45.629 DEBUG:boto:Token: None 2023-05-31T20:59:45.629 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:59:45 GMT /pklxkd-61/ 2023-05-31T20:59:45.630 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:KDZjyCz7mNQWuhX5fKZ7n337Nqk= 2023-05-31T20:59:45.630 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:59:45 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:KDZjyCz7mNQWuhX5fKZ7n337Nqk=', 'Content-Length': '0'} 2023-05-31T20:59:45.633 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '19'), ('X-RGW-Bytes-Used', '57'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000aa93da4377705e2b-006477b541-1630-a1'), ('Content-Length', '0'), ('Date', 'Wed, 31 May 2023 20:59:45 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:59:45.633 DEBUG:boto:path=/pklxkd-61/obj115 2023-05-31T20:59:45.634 DEBUG:boto:auth_path=/pklxkd-61/obj115 2023-05-31T20:59:45.634 DEBUG:boto:Method: PUT 2023-05-31T20:59:45.635 DEBUG:boto:Path: /pklxkd-61/obj115 2023-05-31T20:59:45.635 DEBUG:boto:Data: 2023-05-31T20:59:45.635 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-05-31T20:59:45.636 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:59:45.636 DEBUG:boto:Port: 8000 2023-05-31T20:59:45.636 DEBUG:boto:Params: {} 2023-05-31T20:59:45.637 DEBUG:boto:Token: None 2023-05-31T20:59:45.637 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Wed, 31 May 2023 20:59:45 GMT /pklxkd-61/obj115 2023-05-31T20:59:45.637 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:FCGdNz+7vREEc7PtGZI9X2KZA+M= 2023-05-31T20:59:45.638 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Wed, 31 May 2023 20:59:45 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:FCGdNz+7vREEc7PtGZI9X2KZA+M='} 2023-05-31T20:59:45.680 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000f42c67294da3db17-006477b541-1630-a1'), ('Date', 'Wed, 31 May 2023 20:59:45 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:59:45.681 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'reshard', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'pklxkd-61', '--num-shards', '29', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-05-31T20:59:45.682 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'reshard', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'pklxkd-61', '--num-shards', '29', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-05-31T20:59:45.682 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket reshard --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --bucket pklxkd-61 --num-shards 29 --yes-i-really-mean-it --cluster c1 --debug-rgw 0 --debug-ms 0 2023-05-31T20:59:45.734 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2023-05-31T20:59:45.734 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2023-05-31T20:59:45.788 INFO:teuthology.orchestra.run.smithi099.stdout:tenant: 2023-05-31T20:59:45.789 INFO:teuthology.orchestra.run.smithi099.stdout:bucket name: pklxkd-61 2023-05-31T20:59:45.796 INFO:teuthology.orchestra.run.smithi099.stdout:total entries: 20 2023-05-31T20:59:45.877 DEBUG:boto:path=/pklxkd-61/ 2023-05-31T20:59:45.878 DEBUG:boto:auth_path=/pklxkd-61/ 2023-05-31T20:59:45.878 DEBUG:boto:Method: HEAD 2023-05-31T20:59:45.878 DEBUG:boto:Path: /pklxkd-61/ 2023-05-31T20:59:45.879 DEBUG:boto:Data: 2023-05-31T20:59:45.879 DEBUG:boto:Headers: {} 2023-05-31T20:59:45.879 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:59:45.880 DEBUG:boto:Port: 8000 2023-05-31T20:59:45.880 DEBUG:boto:Params: {} 2023-05-31T20:59:45.880 DEBUG:boto:Token: None 2023-05-31T20:59:45.881 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:59:45 GMT /pklxkd-61/ 2023-05-31T20:59:45.881 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:KDZjyCz7mNQWuhX5fKZ7n337Nqk= 2023-05-31T20:59:45.882 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:59:45 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:KDZjyCz7mNQWuhX5fKZ7n337Nqk=', 'Content-Length': '0'} 2023-05-31T20:59:45.885 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '20'), ('X-RGW-Bytes-Used', '60'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000ecbe3216c29fc2f9-006477b541-1630-a1'), ('Content-Length', '0'), ('Date', 'Wed, 31 May 2023 20:59:45 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:59:45.885 DEBUG:boto:path=/pklxkd-61/obj29 2023-05-31T20:59:45.885 DEBUG:boto:auth_path=/pklxkd-61/obj29 2023-05-31T20:59:45.886 DEBUG:boto:Method: PUT 2023-05-31T20:59:45.886 DEBUG:boto:Path: /pklxkd-61/obj29 2023-05-31T20:59:45.887 DEBUG:boto:Data: 2023-05-31T20:59:45.887 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-05-31T20:59:45.887 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:59:45.887 DEBUG:boto:Port: 8000 2023-05-31T20:59:45.888 DEBUG:boto:Params: {} 2023-05-31T20:59:45.888 DEBUG:boto:Token: None 2023-05-31T20:59:45.889 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Wed, 31 May 2023 20:59:45 GMT /pklxkd-61/obj29 2023-05-31T20:59:45.889 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:fdjuvqvsb1Vq6Na1Gz2NFEtwgN0= 2023-05-31T20:59:45.889 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Wed, 31 May 2023 20:59:45 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:fdjuvqvsb1Vq6Na1Gz2NFEtwgN0='} 2023-05-31T20:59:45.894 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000003244e4cc48ff7f79-006477b541-1630-a1'), ('Date', 'Wed, 31 May 2023 20:59:45 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:59:45.895 DEBUG:boto:path=/pklxkd-61/ 2023-05-31T20:59:45.895 DEBUG:boto:auth_path=/pklxkd-61/ 2023-05-31T20:59:45.895 DEBUG:boto:Method: HEAD 2023-05-31T20:59:45.896 DEBUG:boto:Path: /pklxkd-61/ 2023-05-31T20:59:45.896 DEBUG:boto:Data: 2023-05-31T20:59:45.896 DEBUG:boto:Headers: {} 2023-05-31T20:59:45.897 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:59:45.897 DEBUG:boto:Port: 8000 2023-05-31T20:59:45.897 DEBUG:boto:Params: {} 2023-05-31T20:59:45.898 DEBUG:boto:Token: None 2023-05-31T20:59:45.898 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:59:45 GMT /pklxkd-61/ 2023-05-31T20:59:45.899 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:KDZjyCz7mNQWuhX5fKZ7n337Nqk= 2023-05-31T20:59:45.899 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:59:45 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:KDZjyCz7mNQWuhX5fKZ7n337Nqk=', 'Content-Length': '0'} 2023-05-31T20:59:45.902 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '21'), ('X-RGW-Bytes-Used', '63'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000999b6df6703555db-006477b541-1630-a1'), ('Content-Length', '0'), ('Date', 'Wed, 31 May 2023 20:59:45 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:59:45.902 DEBUG:boto:path=/pklxkd-61/obj58 2023-05-31T20:59:45.903 DEBUG:boto:auth_path=/pklxkd-61/obj58 2023-05-31T20:59:45.903 DEBUG:boto:Method: PUT 2023-05-31T20:59:45.903 DEBUG:boto:Path: /pklxkd-61/obj58 2023-05-31T20:59:45.904 DEBUG:boto:Data: 2023-05-31T20:59:45.904 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-05-31T20:59:45.904 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:59:45.905 DEBUG:boto:Port: 8000 2023-05-31T20:59:45.905 DEBUG:boto:Params: {} 2023-05-31T20:59:45.905 DEBUG:boto:Token: None 2023-05-31T20:59:45.906 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Wed, 31 May 2023 20:59:45 GMT /pklxkd-61/obj58 2023-05-31T20:59:45.906 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:wDFaY/Rz69dqV7VfDEUu17hjR3Y= 2023-05-31T20:59:45.906 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Wed, 31 May 2023 20:59:45 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:wDFaY/Rz69dqV7VfDEUu17hjR3Y='} 2023-05-31T20:59:45.948 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000009be24b90dc17765a-006477b541-1630-a1'), ('Date', 'Wed, 31 May 2023 20:59:45 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:59:45.949 DEBUG:boto:path=/pklxkd-61/ 2023-05-31T20:59:45.949 DEBUG:boto:auth_path=/pklxkd-61/ 2023-05-31T20:59:45.949 DEBUG:boto:Method: HEAD 2023-05-31T20:59:45.950 DEBUG:boto:Path: /pklxkd-61/ 2023-05-31T20:59:45.950 DEBUG:boto:Data: 2023-05-31T20:59:45.950 DEBUG:boto:Headers: {} 2023-05-31T20:59:45.951 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:59:45.951 DEBUG:boto:Port: 8000 2023-05-31T20:59:45.951 DEBUG:boto:Params: {} 2023-05-31T20:59:45.952 DEBUG:boto:Token: None 2023-05-31T20:59:45.952 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:59:45 GMT /pklxkd-61/ 2023-05-31T20:59:45.952 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:KDZjyCz7mNQWuhX5fKZ7n337Nqk= 2023-05-31T20:59:45.953 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:59:45 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:KDZjyCz7mNQWuhX5fKZ7n337Nqk=', 'Content-Length': '0'} 2023-05-31T20:59:45.956 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '22'), ('X-RGW-Bytes-Used', '66'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000e6702d6d83afe8a8-006477b541-1630-a1'), ('Content-Length', '0'), ('Date', 'Wed, 31 May 2023 20:59:45 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:59:45.956 DEBUG:boto:path=/pklxkd-61/obj87 2023-05-31T20:59:45.956 DEBUG:boto:auth_path=/pklxkd-61/obj87 2023-05-31T20:59:45.957 DEBUG:boto:Method: PUT 2023-05-31T20:59:45.957 DEBUG:boto:Path: /pklxkd-61/obj87 2023-05-31T20:59:45.957 DEBUG:boto:Data: 2023-05-31T20:59:45.958 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-05-31T20:59:45.958 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:59:45.958 DEBUG:boto:Port: 8000 2023-05-31T20:59:45.959 DEBUG:boto:Params: {} 2023-05-31T20:59:45.959 DEBUG:boto:Token: None 2023-05-31T20:59:45.960 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Wed, 31 May 2023 20:59:45 GMT /pklxkd-61/obj87 2023-05-31T20:59:45.960 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:7AR0tuOWzK5OdgJDACis9ml3jiM= 2023-05-31T20:59:45.960 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Wed, 31 May 2023 20:59:45 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:7AR0tuOWzK5OdgJDACis9ml3jiM='} 2023-05-31T20:59:45.965 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000001673e860cd6b27b8-006477b541-1630-a1'), ('Date', 'Wed, 31 May 2023 20:59:45 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:59:45.966 DEBUG:boto:path=/pklxkd-61/ 2023-05-31T20:59:45.966 DEBUG:boto:auth_path=/pklxkd-61/ 2023-05-31T20:59:45.967 DEBUG:boto:Method: HEAD 2023-05-31T20:59:45.967 DEBUG:boto:Path: /pklxkd-61/ 2023-05-31T20:59:45.967 DEBUG:boto:Data: 2023-05-31T20:59:45.968 DEBUG:boto:Headers: {} 2023-05-31T20:59:45.968 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:59:45.968 DEBUG:boto:Port: 8000 2023-05-31T20:59:45.969 DEBUG:boto:Params: {} 2023-05-31T20:59:45.969 DEBUG:boto:Token: None 2023-05-31T20:59:45.970 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:59:45 GMT /pklxkd-61/ 2023-05-31T20:59:45.970 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:KDZjyCz7mNQWuhX5fKZ7n337Nqk= 2023-05-31T20:59:45.970 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:59:45 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:KDZjyCz7mNQWuhX5fKZ7n337Nqk=', 'Content-Length': '0'} 2023-05-31T20:59:45.973 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '23'), ('X-RGW-Bytes-Used', '69'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000051e5b1a0364b92d9-006477b541-1630-a1'), ('Content-Length', '0'), ('Date', 'Wed, 31 May 2023 20:59:45 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:59:45.974 DEBUG:boto:path=/pklxkd-61/obj116 2023-05-31T20:59:45.974 DEBUG:boto:auth_path=/pklxkd-61/obj116 2023-05-31T20:59:45.975 DEBUG:boto:Method: PUT 2023-05-31T20:59:45.975 DEBUG:boto:Path: /pklxkd-61/obj116 2023-05-31T20:59:45.975 DEBUG:boto:Data: 2023-05-31T20:59:45.976 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-05-31T20:59:45.976 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:59:45.976 DEBUG:boto:Port: 8000 2023-05-31T20:59:45.977 DEBUG:boto:Params: {} 2023-05-31T20:59:45.977 DEBUG:boto:Token: None 2023-05-31T20:59:45.977 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Wed, 31 May 2023 20:59:45 GMT /pklxkd-61/obj116 2023-05-31T20:59:45.978 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:8oMZ+kbVSEAeZIwRAe5tjfm8dB8= 2023-05-31T20:59:45.978 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Wed, 31 May 2023 20:59:45 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:8oMZ+kbVSEAeZIwRAe5tjfm8dB8='} 2023-05-31T20:59:46.024 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000009eb9d0a2777be4d3-006477b541-1630-a1'), ('Date', 'Wed, 31 May 2023 20:59:45 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:59:46.025 DEBUG:boto:path=/pklxkd-61/ 2023-05-31T20:59:46.025 DEBUG:boto:auth_path=/pklxkd-61/ 2023-05-31T20:59:46.025 DEBUG:boto:Method: HEAD 2023-05-31T20:59:46.026 DEBUG:boto:Path: /pklxkd-61/ 2023-05-31T20:59:46.026 DEBUG:boto:Data: 2023-05-31T20:59:46.026 DEBUG:boto:Headers: {} 2023-05-31T20:59:46.027 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:59:46.027 DEBUG:boto:Port: 8000 2023-05-31T20:59:46.028 DEBUG:boto:Params: {} 2023-05-31T20:59:46.028 DEBUG:boto:Token: None 2023-05-31T20:59:46.029 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:59:46 GMT /pklxkd-61/ 2023-05-31T20:59:46.029 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:G8cBsrdzcEiVIzypZ7uR8NP7fm8= 2023-05-31T20:59:46.029 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:59:46 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:G8cBsrdzcEiVIzypZ7uR8NP7fm8=', 'Content-Length': '0'} 2023-05-31T20:59:46.033 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '24'), ('X-RGW-Bytes-Used', '72'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000cf84a1325ad6d572-006477b542-1630-a1'), ('Content-Length', '0'), ('Date', 'Wed, 31 May 2023 20:59:46 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:59:46.033 DEBUG:boto:path=/pklxkd-61/obj145 2023-05-31T20:59:46.034 DEBUG:boto:auth_path=/pklxkd-61/obj145 2023-05-31T20:59:46.034 DEBUG:boto:Method: PUT 2023-05-31T20:59:46.035 DEBUG:boto:Path: /pklxkd-61/obj145 2023-05-31T20:59:46.035 DEBUG:boto:Data: 2023-05-31T20:59:46.035 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-05-31T20:59:46.036 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:59:46.036 DEBUG:boto:Port: 8000 2023-05-31T20:59:46.037 DEBUG:boto:Params: {} 2023-05-31T20:59:46.037 DEBUG:boto:Token: None 2023-05-31T20:59:46.038 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Wed, 31 May 2023 20:59:46 GMT /pklxkd-61/obj145 2023-05-31T20:59:46.038 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:nGIPF0vI09DbDrGUd9RAQPiCsco= 2023-05-31T20:59:46.039 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Wed, 31 May 2023 20:59:46 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:nGIPF0vI09DbDrGUd9RAQPiCsco='} 2023-05-31T20:59:46.044 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000705937e9a168617a-006477b542-1630-a1'), ('Date', 'Wed, 31 May 2023 20:59:46 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:59:46.045 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'reshard', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'pklxkd-61', '--num-shards', '31', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-05-31T20:59:46.045 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'reshard', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'pklxkd-61', '--num-shards', '31', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-05-31T20:59:46.046 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket reshard --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --bucket pklxkd-61 --num-shards 31 --yes-i-really-mean-it --cluster c1 --debug-rgw 0 --debug-ms 0 2023-05-31T20:59:46.099 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2023-05-31T20:59:46.100 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2023-05-31T20:59:46.155 INFO:teuthology.orchestra.run.smithi099.stdout:tenant: 2023-05-31T20:59:46.156 INFO:teuthology.orchestra.run.smithi099.stdout:bucket name: pklxkd-61 2023-05-31T20:59:46.165 INFO:teuthology.orchestra.run.smithi099.stdout:total entries: 25 2023-05-31T20:59:46.180 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.1 has been restored 2023-05-31T20:59:46.269 DEBUG:boto:path=/pklxkd-61/ 2023-05-31T20:59:46.270 DEBUG:boto:auth_path=/pklxkd-61/ 2023-05-31T20:59:46.270 DEBUG:boto:Method: HEAD 2023-05-31T20:59:46.271 DEBUG:boto:Path: /pklxkd-61/ 2023-05-31T20:59:46.271 DEBUG:boto:Data: 2023-05-31T20:59:46.271 DEBUG:boto:Headers: {} 2023-05-31T20:59:46.272 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:59:46.272 DEBUG:boto:Port: 8000 2023-05-31T20:59:46.272 DEBUG:boto:Params: {} 2023-05-31T20:59:46.273 DEBUG:boto:Token: None 2023-05-31T20:59:46.273 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:59:46 GMT /pklxkd-61/ 2023-05-31T20:59:46.273 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:G8cBsrdzcEiVIzypZ7uR8NP7fm8= 2023-05-31T20:59:46.274 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:59:46 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:G8cBsrdzcEiVIzypZ7uR8NP7fm8=', 'Content-Length': '0'} 2023-05-31T20:59:46.277 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '25'), ('X-RGW-Bytes-Used', '75'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000109a9d9d6b597ab5-006477b542-1630-a1'), ('Content-Length', '0'), ('Date', 'Wed, 31 May 2023 20:59:46 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:59:46.278 DEBUG:boto:path=/pklxkd-61/obj31 2023-05-31T20:59:46.278 DEBUG:boto:auth_path=/pklxkd-61/obj31 2023-05-31T20:59:46.278 DEBUG:boto:Method: PUT 2023-05-31T20:59:46.279 DEBUG:boto:Path: /pklxkd-61/obj31 2023-05-31T20:59:46.279 DEBUG:boto:Data: 2023-05-31T20:59:46.279 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-05-31T20:59:46.280 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:59:46.280 DEBUG:boto:Port: 8000 2023-05-31T20:59:46.280 DEBUG:boto:Params: {} 2023-05-31T20:59:46.281 DEBUG:boto:Token: None 2023-05-31T20:59:46.281 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Wed, 31 May 2023 20:59:46 GMT /pklxkd-61/obj31 2023-05-31T20:59:46.282 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:TgUtmssNsc4YXpbxkfN3Ec2CGk4= 2023-05-31T20:59:46.282 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Wed, 31 May 2023 20:59:46 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:TgUtmssNsc4YXpbxkfN3Ec2CGk4='} 2023-05-31T20:59:46.328 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000002e2bca5841343817-006477b542-1630-a1'), ('Date', 'Wed, 31 May 2023 20:59:46 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:59:46.329 DEBUG:boto:path=/pklxkd-61/ 2023-05-31T20:59:46.330 DEBUG:boto:auth_path=/pklxkd-61/ 2023-05-31T20:59:46.330 DEBUG:boto:Method: HEAD 2023-05-31T20:59:46.330 DEBUG:boto:Path: /pklxkd-61/ 2023-05-31T20:59:46.331 DEBUG:boto:Data: 2023-05-31T20:59:46.331 DEBUG:boto:Headers: {} 2023-05-31T20:59:46.331 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:59:46.332 DEBUG:boto:Port: 8000 2023-05-31T20:59:46.332 DEBUG:boto:Params: {} 2023-05-31T20:59:46.332 DEBUG:boto:Token: None 2023-05-31T20:59:46.333 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:59:46 GMT /pklxkd-61/ 2023-05-31T20:59:46.333 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:G8cBsrdzcEiVIzypZ7uR8NP7fm8= 2023-05-31T20:59:46.334 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:59:46 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:G8cBsrdzcEiVIzypZ7uR8NP7fm8=', 'Content-Length': '0'} 2023-05-31T20:59:46.337 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '26'), ('X-RGW-Bytes-Used', '78'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000d01c94ce486c1526-006477b542-1630-a1'), ('Content-Length', '0'), ('Date', 'Wed, 31 May 2023 20:59:46 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:59:46.337 DEBUG:boto:path=/pklxkd-61/obj62 2023-05-31T20:59:46.338 DEBUG:boto:auth_path=/pklxkd-61/obj62 2023-05-31T20:59:46.338 DEBUG:boto:Method: PUT 2023-05-31T20:59:46.338 DEBUG:boto:Path: /pklxkd-61/obj62 2023-05-31T20:59:46.339 DEBUG:boto:Data: 2023-05-31T20:59:46.339 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-05-31T20:59:46.339 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:59:46.340 DEBUG:boto:Port: 8000 2023-05-31T20:59:46.340 DEBUG:boto:Params: {} 2023-05-31T20:59:46.340 DEBUG:boto:Token: None 2023-05-31T20:59:46.341 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Wed, 31 May 2023 20:59:46 GMT /pklxkd-61/obj62 2023-05-31T20:59:46.341 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:KMRnO5REUG441Lhh7YvOgI0b9G0= 2023-05-31T20:59:46.342 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Wed, 31 May 2023 20:59:46 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:KMRnO5REUG441Lhh7YvOgI0b9G0='} 2023-05-31T20:59:46.347 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000db7ec566d1ac3668-006477b542-1630-a1'), ('Date', 'Wed, 31 May 2023 20:59:46 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:59:46.347 DEBUG:boto:path=/pklxkd-61/ 2023-05-31T20:59:46.347 DEBUG:boto:auth_path=/pklxkd-61/ 2023-05-31T20:59:46.348 DEBUG:boto:Method: HEAD 2023-05-31T20:59:46.348 DEBUG:boto:Path: /pklxkd-61/ 2023-05-31T20:59:46.348 DEBUG:boto:Data: 2023-05-31T20:59:46.349 DEBUG:boto:Headers: {} 2023-05-31T20:59:46.349 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:59:46.349 DEBUG:boto:Port: 8000 2023-05-31T20:59:46.350 DEBUG:boto:Params: {} 2023-05-31T20:59:46.350 DEBUG:boto:Token: None 2023-05-31T20:59:46.350 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:59:46 GMT /pklxkd-61/ 2023-05-31T20:59:46.351 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:G8cBsrdzcEiVIzypZ7uR8NP7fm8= 2023-05-31T20:59:46.351 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:59:46 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:G8cBsrdzcEiVIzypZ7uR8NP7fm8=', 'Content-Length': '0'} 2023-05-31T20:59:46.354 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '27'), ('X-RGW-Bytes-Used', '81'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000f15461c970722a6c-006477b542-1630-a1'), ('Content-Length', '0'), ('Date', 'Wed, 31 May 2023 20:59:46 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:59:46.355 DEBUG:boto:path=/pklxkd-61/obj93 2023-05-31T20:59:46.356 DEBUG:boto:auth_path=/pklxkd-61/obj93 2023-05-31T20:59:46.356 DEBUG:boto:Method: PUT 2023-05-31T20:59:46.356 DEBUG:boto:Path: /pklxkd-61/obj93 2023-05-31T20:59:46.357 DEBUG:boto:Data: 2023-05-31T20:59:46.357 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-05-31T20:59:46.357 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:59:46.358 DEBUG:boto:Port: 8000 2023-05-31T20:59:46.358 DEBUG:boto:Params: {} 2023-05-31T20:59:46.358 DEBUG:boto:Token: None 2023-05-31T20:59:46.359 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Wed, 31 May 2023 20:59:46 GMT /pklxkd-61/obj93 2023-05-31T20:59:46.359 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:B/LOcM4yuSNp5r6XFLuHjVRUhxg= 2023-05-31T20:59:46.360 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Wed, 31 May 2023 20:59:46 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:B/LOcM4yuSNp5r6XFLuHjVRUhxg='} 2023-05-31T20:59:46.408 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000e25a352397484113-006477b542-1630-a1'), ('Date', 'Wed, 31 May 2023 20:59:46 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:59:46.409 DEBUG:boto:path=/pklxkd-61/ 2023-05-31T20:59:46.409 DEBUG:boto:auth_path=/pklxkd-61/ 2023-05-31T20:59:46.410 DEBUG:boto:Method: HEAD 2023-05-31T20:59:46.410 DEBUG:boto:Path: /pklxkd-61/ 2023-05-31T20:59:46.410 DEBUG:boto:Data: 2023-05-31T20:59:46.411 DEBUG:boto:Headers: {} 2023-05-31T20:59:46.411 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:59:46.411 DEBUG:boto:Port: 8000 2023-05-31T20:59:46.411 DEBUG:boto:Params: {} 2023-05-31T20:59:46.412 DEBUG:boto:Token: None 2023-05-31T20:59:46.412 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:59:46 GMT /pklxkd-61/ 2023-05-31T20:59:46.413 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:G8cBsrdzcEiVIzypZ7uR8NP7fm8= 2023-05-31T20:59:46.413 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:59:46 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:G8cBsrdzcEiVIzypZ7uR8NP7fm8=', 'Content-Length': '0'} 2023-05-31T20:59:46.416 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '28'), ('X-RGW-Bytes-Used', '84'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000e481927b5b6d0513-006477b542-1630-a1'), ('Content-Length', '0'), ('Date', 'Wed, 31 May 2023 20:59:46 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:59:46.417 DEBUG:boto:path=/pklxkd-61/obj124 2023-05-31T20:59:46.417 DEBUG:boto:auth_path=/pklxkd-61/obj124 2023-05-31T20:59:46.417 DEBUG:boto:Method: PUT 2023-05-31T20:59:46.418 DEBUG:boto:Path: /pklxkd-61/obj124 2023-05-31T20:59:46.418 DEBUG:boto:Data: 2023-05-31T20:59:46.418 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-05-31T20:59:46.419 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:59:46.419 DEBUG:boto:Port: 8000 2023-05-31T20:59:46.419 DEBUG:boto:Params: {} 2023-05-31T20:59:46.420 DEBUG:boto:Token: None 2023-05-31T20:59:46.420 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Wed, 31 May 2023 20:59:46 GMT /pklxkd-61/obj124 2023-05-31T20:59:46.420 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:aSSlnFBQx7UlHlZ6Xq/Cww4Gba4= 2023-05-31T20:59:46.421 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Wed, 31 May 2023 20:59:46 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:aSSlnFBQx7UlHlZ6Xq/Cww4Gba4='} 2023-05-31T20:59:46.426 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000003fa481a993e83db8-006477b542-1630-a1'), ('Date', 'Wed, 31 May 2023 20:59:46 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:59:46.427 DEBUG:boto:path=/pklxkd-61/ 2023-05-31T20:59:46.427 DEBUG:boto:auth_path=/pklxkd-61/ 2023-05-31T20:59:46.427 DEBUG:boto:Method: HEAD 2023-05-31T20:59:46.428 DEBUG:boto:Path: /pklxkd-61/ 2023-05-31T20:59:46.428 DEBUG:boto:Data: 2023-05-31T20:59:46.428 DEBUG:boto:Headers: {} 2023-05-31T20:59:46.429 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:59:46.429 DEBUG:boto:Port: 8000 2023-05-31T20:59:46.429 DEBUG:boto:Params: {} 2023-05-31T20:59:46.430 DEBUG:boto:Token: None 2023-05-31T20:59:46.430 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:59:46 GMT /pklxkd-61/ 2023-05-31T20:59:46.431 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:G8cBsrdzcEiVIzypZ7uR8NP7fm8= 2023-05-31T20:59:46.431 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:59:46 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:G8cBsrdzcEiVIzypZ7uR8NP7fm8=', 'Content-Length': '0'} 2023-05-31T20:59:46.434 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '29'), ('X-RGW-Bytes-Used', '87'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000c4eabe0d8b1f32e0-006477b542-1630-a1'), ('Content-Length', '0'), ('Date', 'Wed, 31 May 2023 20:59:46 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:59:46.435 DEBUG:boto:path=/pklxkd-61/obj155 2023-05-31T20:59:46.435 DEBUG:boto:auth_path=/pklxkd-61/obj155 2023-05-31T20:59:46.435 DEBUG:boto:Method: PUT 2023-05-31T20:59:46.436 DEBUG:boto:Path: /pklxkd-61/obj155 2023-05-31T20:59:46.436 DEBUG:boto:Data: 2023-05-31T20:59:46.437 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-05-31T20:59:46.437 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:59:46.437 DEBUG:boto:Port: 8000 2023-05-31T20:59:46.438 DEBUG:boto:Params: {} 2023-05-31T20:59:46.438 DEBUG:boto:Token: None 2023-05-31T20:59:46.438 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Wed, 31 May 2023 20:59:46 GMT /pklxkd-61/obj155 2023-05-31T20:59:46.439 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:rf553h9zk0s+BuEx53NfVu7iUI8= 2023-05-31T20:59:46.439 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Wed, 31 May 2023 20:59:46 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:rf553h9zk0s+BuEx53NfVu7iUI8='} 2023-05-31T20:59:46.485 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000223fa9d2fa4eb66b-006477b542-1630-a1'), ('Date', 'Wed, 31 May 2023 20:59:46 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:59:46.697 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'reshard', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'pklxkd-61', '--num-shards', '37', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-05-31T20:59:46.698 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'reshard', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'pklxkd-61', '--num-shards', '37', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-05-31T20:59:46.698 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket reshard --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --bucket pklxkd-61 --num-shards 37 --yes-i-really-mean-it --cluster c1 --debug-rgw 0 --debug-ms 0 2023-05-31T20:59:46.751 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2023-05-31T20:59:46.751 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2023-05-31T20:59:46.811 INFO:teuthology.orchestra.run.smithi099.stdout:tenant: 2023-05-31T20:59:46.811 INFO:teuthology.orchestra.run.smithi099.stdout:bucket name: pklxkd-61 2023-05-31T20:59:46.822 INFO:teuthology.orchestra.run.smithi099.stdout:total entries: 30 2023-05-31T20:59:46.904 DEBUG:boto:path=/pklxkd-61/ 2023-05-31T20:59:46.905 DEBUG:boto:auth_path=/pklxkd-61/ 2023-05-31T20:59:46.905 DEBUG:boto:Method: HEAD 2023-05-31T20:59:46.906 DEBUG:boto:Path: /pklxkd-61/ 2023-05-31T20:59:46.906 DEBUG:boto:Data: 2023-05-31T20:59:46.906 DEBUG:boto:Headers: {} 2023-05-31T20:59:46.907 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:59:46.907 DEBUG:boto:Port: 8000 2023-05-31T20:59:46.907 DEBUG:boto:Params: {} 2023-05-31T20:59:46.908 DEBUG:boto:Token: None 2023-05-31T20:59:46.908 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:59:46 GMT /pklxkd-61/ 2023-05-31T20:59:46.908 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:G8cBsrdzcEiVIzypZ7uR8NP7fm8= 2023-05-31T20:59:46.909 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:59:46 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:G8cBsrdzcEiVIzypZ7uR8NP7fm8=', 'Content-Length': '0'} 2023-05-31T20:59:46.912 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '30'), ('X-RGW-Bytes-Used', '90'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000001050f12022c965a2-006477b542-1630-a1'), ('Content-Length', '0'), ('Date', 'Wed, 31 May 2023 20:59:46 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:59:46.913 DEBUG:boto:path=/pklxkd-61/obj37 2023-05-31T20:59:46.913 DEBUG:boto:auth_path=/pklxkd-61/obj37 2023-05-31T20:59:46.913 DEBUG:boto:Method: PUT 2023-05-31T20:59:46.914 DEBUG:boto:Path: /pklxkd-61/obj37 2023-05-31T20:59:46.914 DEBUG:boto:Data: 2023-05-31T20:59:46.914 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-05-31T20:59:46.915 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:59:46.915 DEBUG:boto:Port: 8000 2023-05-31T20:59:46.915 DEBUG:boto:Params: {} 2023-05-31T20:59:46.916 DEBUG:boto:Token: None 2023-05-31T20:59:46.916 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Wed, 31 May 2023 20:59:46 GMT /pklxkd-61/obj37 2023-05-31T20:59:46.917 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:cJxXugM9cPHI0YTEB2vNpzDfess= 2023-05-31T20:59:46.917 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Wed, 31 May 2023 20:59:46 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:cJxXugM9cPHI0YTEB2vNpzDfess='} 2023-05-31T20:59:46.922 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000b732f28406d66f42-006477b542-1630-a1'), ('Date', 'Wed, 31 May 2023 20:59:46 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:59:46.923 DEBUG:boto:path=/pklxkd-61/ 2023-05-31T20:59:46.923 DEBUG:boto:auth_path=/pklxkd-61/ 2023-05-31T20:59:46.923 DEBUG:boto:Method: HEAD 2023-05-31T20:59:46.924 DEBUG:boto:Path: /pklxkd-61/ 2023-05-31T20:59:46.924 DEBUG:boto:Data: 2023-05-31T20:59:46.924 DEBUG:boto:Headers: {} 2023-05-31T20:59:46.925 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:59:46.925 DEBUG:boto:Port: 8000 2023-05-31T20:59:46.925 DEBUG:boto:Params: {} 2023-05-31T20:59:46.926 DEBUG:boto:Token: None 2023-05-31T20:59:46.926 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:59:46 GMT /pklxkd-61/ 2023-05-31T20:59:46.927 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:G8cBsrdzcEiVIzypZ7uR8NP7fm8= 2023-05-31T20:59:46.927 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:59:46 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:G8cBsrdzcEiVIzypZ7uR8NP7fm8=', 'Content-Length': '0'} 2023-05-31T20:59:46.930 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '31'), ('X-RGW-Bytes-Used', '93'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000064fb7acfe08b5e4d-006477b542-1630-a1'), ('Content-Length', '0'), ('Date', 'Wed, 31 May 2023 20:59:46 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:59:46.931 DEBUG:boto:path=/pklxkd-61/obj74 2023-05-31T20:59:46.931 DEBUG:boto:auth_path=/pklxkd-61/obj74 2023-05-31T20:59:46.931 DEBUG:boto:Method: PUT 2023-05-31T20:59:46.932 DEBUG:boto:Path: /pklxkd-61/obj74 2023-05-31T20:59:46.932 DEBUG:boto:Data: 2023-05-31T20:59:46.932 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-05-31T20:59:46.933 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:59:46.933 DEBUG:boto:Port: 8000 2023-05-31T20:59:46.933 DEBUG:boto:Params: {} 2023-05-31T20:59:46.934 DEBUG:boto:Token: None 2023-05-31T20:59:46.934 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Wed, 31 May 2023 20:59:46 GMT /pklxkd-61/obj74 2023-05-31T20:59:46.934 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:JTwFeTCrEkQFtNhqU9IlwnN5c6I= 2023-05-31T20:59:46.935 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Wed, 31 May 2023 20:59:46 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:JTwFeTCrEkQFtNhqU9IlwnN5c6I='} 2023-05-31T20:59:46.940 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000009ae5cfaa5fb4b136-006477b542-1630-a1'), ('Date', 'Wed, 31 May 2023 20:59:46 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:59:46.941 DEBUG:boto:path=/pklxkd-61/ 2023-05-31T20:59:46.941 DEBUG:boto:auth_path=/pklxkd-61/ 2023-05-31T20:59:46.941 DEBUG:boto:Method: HEAD 2023-05-31T20:59:46.942 DEBUG:boto:Path: /pklxkd-61/ 2023-05-31T20:59:46.942 DEBUG:boto:Data: 2023-05-31T20:59:46.942 DEBUG:boto:Headers: {} 2023-05-31T20:59:46.943 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:59:46.943 DEBUG:boto:Port: 8000 2023-05-31T20:59:46.943 DEBUG:boto:Params: {} 2023-05-31T20:59:46.944 DEBUG:boto:Token: None 2023-05-31T20:59:46.944 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:59:46 GMT /pklxkd-61/ 2023-05-31T20:59:46.944 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:G8cBsrdzcEiVIzypZ7uR8NP7fm8= 2023-05-31T20:59:46.945 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:59:46 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:G8cBsrdzcEiVIzypZ7uR8NP7fm8=', 'Content-Length': '0'} 2023-05-31T20:59:46.948 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '32'), ('X-RGW-Bytes-Used', '96'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000003995c01bc6989a75-006477b542-1630-a1'), ('Content-Length', '0'), ('Date', 'Wed, 31 May 2023 20:59:46 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:59:46.948 DEBUG:boto:path=/pklxkd-61/obj111 2023-05-31T20:59:46.949 DEBUG:boto:auth_path=/pklxkd-61/obj111 2023-05-31T20:59:46.949 DEBUG:boto:Method: PUT 2023-05-31T20:59:46.950 DEBUG:boto:Path: /pklxkd-61/obj111 2023-05-31T20:59:46.950 DEBUG:boto:Data: 2023-05-31T20:59:46.950 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-05-31T20:59:46.950 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:59:46.951 DEBUG:boto:Port: 8000 2023-05-31T20:59:46.951 DEBUG:boto:Params: {} 2023-05-31T20:59:46.952 DEBUG:boto:Token: None 2023-05-31T20:59:46.952 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Wed, 31 May 2023 20:59:46 GMT /pklxkd-61/obj111 2023-05-31T20:59:46.952 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:2m/NNqkWnB40RCA89U+hqv03seU= 2023-05-31T20:59:46.953 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Wed, 31 May 2023 20:59:46 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:2m/NNqkWnB40RCA89U+hqv03seU='} 2023-05-31T20:59:46.996 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000a0850db2fc7afeef-006477b542-1630-a1'), ('Date', 'Wed, 31 May 2023 20:59:46 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:59:46.997 DEBUG:boto:path=/pklxkd-61/ 2023-05-31T20:59:46.997 DEBUG:boto:auth_path=/pklxkd-61/ 2023-05-31T20:59:46.997 DEBUG:boto:Method: HEAD 2023-05-31T20:59:46.998 DEBUG:boto:Path: /pklxkd-61/ 2023-05-31T20:59:46.998 DEBUG:boto:Data: 2023-05-31T20:59:46.998 DEBUG:boto:Headers: {} 2023-05-31T20:59:46.999 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:59:46.999 DEBUG:boto:Port: 8000 2023-05-31T20:59:46.999 DEBUG:boto:Params: {} 2023-05-31T20:59:47.000 DEBUG:boto:Token: None 2023-05-31T20:59:47.000 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:59:47 GMT /pklxkd-61/ 2023-05-31T20:59:47.001 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:mGiEdPHgykMq4AGHufUoifOM1Ak= 2023-05-31T20:59:47.001 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:59:47 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:mGiEdPHgykMq4AGHufUoifOM1Ak=', 'Content-Length': '0'} 2023-05-31T20:59:47.004 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '33'), ('X-RGW-Bytes-Used', '99'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000006f312d58ce6d6106-006477b542-1630-a1'), ('Content-Length', '0'), ('Date', 'Wed, 31 May 2023 20:59:46 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:59:47.005 DEBUG:boto:path=/pklxkd-61/obj148 2023-05-31T20:59:47.005 DEBUG:boto:auth_path=/pklxkd-61/obj148 2023-05-31T20:59:47.005 DEBUG:boto:Method: PUT 2023-05-31T20:59:47.006 DEBUG:boto:Path: /pklxkd-61/obj148 2023-05-31T20:59:47.006 DEBUG:boto:Data: 2023-05-31T20:59:47.006 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-05-31T20:59:47.007 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:59:47.007 DEBUG:boto:Port: 8000 2023-05-31T20:59:47.007 DEBUG:boto:Params: {} 2023-05-31T20:59:47.008 DEBUG:boto:Token: None 2023-05-31T20:59:47.008 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Wed, 31 May 2023 20:59:47 GMT /pklxkd-61/obj148 2023-05-31T20:59:47.008 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:1YSJlRbad90GT11zGivh5FXgjBI= 2023-05-31T20:59:47.009 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Wed, 31 May 2023 20:59:47 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:1YSJlRbad90GT11zGivh5FXgjBI='} 2023-05-31T20:59:47.020 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000006e2f544811d0ee13-006477b543-1630-a1'), ('Date', 'Wed, 31 May 2023 20:59:47 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:59:47.020 DEBUG:boto:path=/pklxkd-61/ 2023-05-31T20:59:47.021 DEBUG:boto:auth_path=/pklxkd-61/ 2023-05-31T20:59:47.021 DEBUG:boto:Method: HEAD 2023-05-31T20:59:47.021 DEBUG:boto:Path: /pklxkd-61/ 2023-05-31T20:59:47.022 DEBUG:boto:Data: 2023-05-31T20:59:47.022 DEBUG:boto:Headers: {} 2023-05-31T20:59:47.022 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:59:47.023 DEBUG:boto:Port: 8000 2023-05-31T20:59:47.023 DEBUG:boto:Params: {} 2023-05-31T20:59:47.023 DEBUG:boto:Token: None 2023-05-31T20:59:47.024 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 20:59:47 GMT /pklxkd-61/ 2023-05-31T20:59:47.024 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:mGiEdPHgykMq4AGHufUoifOM1Ak= 2023-05-31T20:59:47.024 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:59:47 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:mGiEdPHgykMq4AGHufUoifOM1Ak=', 'Content-Length': '0'} 2023-05-31T20:59:47.028 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '34'), ('X-RGW-Bytes-Used', '102'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000c8ed0c5061d61f5b-006477b543-1630-a1'), ('Content-Length', '0'), ('Date', 'Wed, 31 May 2023 20:59:47 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:59:47.028 DEBUG:boto:path=/pklxkd-61/obj185 2023-05-31T20:59:47.029 DEBUG:boto:auth_path=/pklxkd-61/obj185 2023-05-31T20:59:47.029 DEBUG:boto:Method: PUT 2023-05-31T20:59:47.029 DEBUG:boto:Path: /pklxkd-61/obj185 2023-05-31T20:59:47.030 DEBUG:boto:Data: 2023-05-31T20:59:47.030 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-05-31T20:59:47.030 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T20:59:47.030 DEBUG:boto:Port: 8000 2023-05-31T20:59:47.031 DEBUG:boto:Params: {} 2023-05-31T20:59:47.031 DEBUG:boto:Token: None 2023-05-31T20:59:47.032 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Wed, 31 May 2023 20:59:47 GMT /pklxkd-61/obj185 2023-05-31T20:59:47.032 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:0sT+WuF1m6TMHXtQD4xM8RDzR5Q= 2023-05-31T20:59:47.032 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Wed, 31 May 2023 20:59:47 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:0sT+WuF1m6TMHXtQD4xM8RDzR5Q='} 2023-05-31T20:59:47.041 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000e29d9e172979323b-006477b543-1630-a1'), ('Date', 'Wed, 31 May 2023 20:59:47 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T20:59:47.042 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'init', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-05-31T20:59:47.042 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'init', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-05-31T20:59:47.043 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync init --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 2023-05-31T20:59:47.094 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2023-05-31T20:59:47.094 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2023-05-31T20:59:47.146 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:59:47.141+0000 7f2e623b5780 5 RGW-SYNC:data: waiting on data sync status lease 2023-05-31T20:59:47.147 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:59:47.142+0000 7f2e623b5780 5 RGW-SYNC:data: acquired data sync status lease 2023-05-31T20:59:47.171 INFO:tasks.rgw.c1.client.1:Restarting daemon 2023-05-31T20:59:47.171 DEBUG:teuthology.orchestra.run.smithi099:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term radosgw --rgw-frontends 'beast port=8001' -n client.1 --cluster c1 -k /etc/ceph/c1.client.1.keyring --log-file /var/log/ceph/rgw.c1.client.1.log --rgw_ops_log_socket_path /home/ubuntu/cephtest/rgw.opslog.c1.client.1.sock --foreground --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm | sudo tee /var/log/ceph/rgw.c1.client.1.stdout 2>&1 2023-05-31T20:59:47.173 INFO:tasks.rgw.c1.client.1:Started 2023-05-31T20:59:47.174 DEBUG:teuthology.orchestra.run.smithi099:> curl http://smithi099.front.sepia.ceph.com:8001 2023-05-31T20:59:47.194 INFO:teuthology.orchestra.run.smithi099.stderr: % Total % Received % Xferd Average Speed Time Time Time Current 2023-05-31T20:59:47.195 INFO:teuthology.orchestra.run.smithi099.stderr: Dload Upload Total Spent Left Speed 2023-05-31T20:59:47.195 INFO:teuthology.orchestra.run.smithi099.stderr: 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0curl: (7) Failed to connect to smithi099.front.sepia.ceph.com port 8001: Connection refused 2023-05-31T20:59:47.195 DEBUG:teuthology.orchestra.run:got remote process result: 7 2023-05-31T20:59:48.196 DEBUG:teuthology.orchestra.run.smithi099:> curl http://smithi099.front.sepia.ceph.com:8001 2023-05-31T20:59:48.219 INFO:teuthology.orchestra.run.smithi099.stderr: % Total % Received % Xferd Average Speed Time Time Time Current 2023-05-31T20:59:48.220 INFO:teuthology.orchestra.run.smithi099.stderr: Dload Upload Total Spent Left Speed 2023-05-31T20:59:48.237 INFO:teuthology.orchestra.run.smithi099.stderr: 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 100 214 0 214 0 0 11888 0 --:--:-- --:--:-- --:--:-- 11888 2023-05-31T20:59:48.238 INFO:teuthology.orchestra.run.smithi099.stdout:anonymous 2023-05-31T20:59:48.239 DEBUG:rgw_multi.tests:bucket checkpoint: source=a1 target=a2 bucket=pklxkd-61 2023-05-31T20:59:48.239 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'pklxkd-61', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-05-31T20:59:48.240 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'pklxkd-61', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-05-31T20:59:48.241 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket pklxkd-61 --source-zone a1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2023-05-31T20:59:48.335 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2023-05-31T20:59:48.335 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2023-05-31T20:59:48.387 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:59:48.382+0000 7f6bd04ea780 1 waiting to reach incremental sync.. 2023-05-31T20:59:53.387 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:59:53.382+0000 7f6bd04ea780 1 waiting to reach incremental sync.. 2023-05-31T20:59:58.388 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T20:59:58.383+0000 7f6bd04ea780 1 waiting to reach incremental sync.. 2023-05-31T21:00:03.388 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T21:00:03.384+0000 7f6bd04ea780 1 waiting to reach incremental sync.. 2023-05-31T21:00:08.389 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T21:00:08.385+0000 7f6bd04ea780 1 waiting to reach incremental sync.. 2023-05-31T21:00:13.390 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T21:00:13.386+0000 7f6bd04ea780 1 waiting to reach incremental sync.. 2023-05-31T21:00:18.390 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T21:00:18.386+0000 7f6bd04ea780 1 waiting to reach incremental sync.. 2023-05-31T21:00:23.391 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T21:00:23.386+0000 7f6bd04ea780 1 waiting to reach incremental sync.. 2023-05-31T21:00:28.391 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T21:00:28.387+0000 7f6bd04ea780 1 waiting to reach incremental sync.. 2023-05-31T21:00:33.392 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T21:00:33.387+0000 7f6bd04ea780 1 waiting to reach incremental sync.. 2023-05-31T21:00:38.393 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T21:00:38.388+0000 7f6bd04ea780 1 waiting to reach incremental sync.. 2023-05-31T21:00:43.393 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T21:00:43.389+0000 7f6bd04ea780 1 waiting to reach incremental sync.. 2023-05-31T21:00:48.394 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T21:00:48.390+0000 7f6bd04ea780 1 waiting to reach incremental sync.. 2023-05-31T21:00:53.394 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T21:00:53.391+0000 7f6bd04ea780 1 waiting to reach incremental sync.. 2023-05-31T21:00:58.395 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T21:00:58.391+0000 7f6bd04ea780 1 waiting to reach incremental sync.. 2023-05-31T21:01:03.395 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T21:01:03.391+0000 7f6bd04ea780 1 waiting to reach incremental sync.. 2023-05-31T21:01:08.396 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T21:01:08.391+0000 7f6bd04ea780 1 waiting to reach incremental sync.. 2023-05-31T21:01:13.397 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T21:01:13.392+0000 7f6bd04ea780 1 waiting to reach incremental sync.. 2023-05-31T21:01:18.397 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T21:01:18.392+0000 7f6bd04ea780 1 waiting to reach incremental sync.. 2023-05-31T21:01:23.398 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T21:01:23.393+0000 7f6bd04ea780 1 waiting to reach incremental sync.. 2023-05-31T21:01:28.399 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T21:01:28.395+0000 7f6bd04ea780 1 waiting to reach incremental sync.. 2023-05-31T21:01:33.399 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T21:01:33.396+0000 7f6bd04ea780 1 waiting to reach incremental sync.. 2023-05-31T21:01:38.400 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T21:01:38.396+0000 7f6bd04ea780 1 waiting to reach incremental sync.. 2023-05-31T21:01:43.401 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T21:01:43.397+0000 7f6bd04ea780 1 waiting to reach incremental sync.. 2023-05-31T21:01:48.402 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T21:01:48.397+0000 7f6bd04ea780 1 waiting to reach incremental sync.. 2023-05-31T21:01:53.402 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T21:01:53.398+0000 7f6bd04ea780 1 waiting to reach incremental sync.. 2023-05-31T21:01:58.403 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T21:01:58.398+0000 7f6bd04ea780 1 waiting to reach incremental sync.. 2023-05-31T21:02:03.403 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T21:02:03.399+0000 7f6bd04ea780 1 waiting to reach incremental sync.. 2023-05-31T21:02:08.404 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T21:02:08.399+0000 7f6bd04ea780 1 waiting to reach incremental sync.. 2023-05-31T21:02:13.405 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T21:02:13.401+0000 7f6bd04ea780 1 waiting to reach incremental sync.. 2023-05-31T21:02:18.406 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T21:02:18.402+0000 7f6bd04ea780 1 waiting to reach incremental sync.. 2023-05-31T21:02:23.406 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T21:02:23.402+0000 7f6bd04ea780 1 waiting to reach incremental sync.. 2023-05-31T21:02:28.407 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T21:02:28.403+0000 7f6bd04ea780 1 waiting to reach incremental sync.. 2023-05-31T21:02:33.408 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T21:02:33.403+0000 7f6bd04ea780 1 waiting to reach incremental sync.. 2023-05-31T21:02:38.408 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T21:02:38.404+0000 7f6bd04ea780 1 waiting to reach incremental sync.. 2023-05-31T21:02:43.412 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T21:02:43.404+0000 7f6bd04ea780 1 waiting to reach incremental sync.. 2023-05-31T21:02:48.410 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T21:02:48.405+0000 7f6bd04ea780 1 waiting to reach incremental sync.. 2023-05-31T21:02:53.410 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T21:02:53.407+0000 7f6bd04ea780 1 waiting to reach incremental sync.. 2023-05-31T21:02:58.411 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T21:02:58.407+0000 7f6bd04ea780 1 waiting to reach incremental sync.. 2023-05-31T21:03:03.412 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T21:03:03.408+0000 7f6bd04ea780 1 waiting to reach incremental sync.. 2023-05-31T21:03:08.412 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T21:03:08.408+0000 7f6bd04ea780 1 waiting to reach incremental sync.. 2023-05-31T21:03:13.413 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T21:03:13.408+0000 7f6bd04ea780 1 waiting to reach incremental sync.. 2023-05-31T21:03:18.416 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T21:03:18.412+0000 7f6bd04ea780 1 bucket sync caught up with source: 2023-05-31T21:03:18.417 INFO:teuthology.orchestra.run.smithi099.stderr: local status: [, , , , , , 00000000002.124.6, , , , , , 00000000002.96.6, , , , , 00000000002.122.6, , , , , , , , , , , , , , , 00000000001.109.6, , , , 00000000001.94.6] 2023-05-31T21:03:18.417 INFO:teuthology.orchestra.run.smithi099.stderr: remote markers: [, , , , , , 00000000002.124.6, , , , , , 00000000002.96.6, , , , , 00000000002.122.6, , , , , , , , , , , , , , , 00000000001.109.6, , , , 00000000001.94.6] 2023-05-31T21:03:18.417 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T21:03:18.412+0000 7f6bd04ea780 0 bucket checkpoint complete 2023-05-31T21:03:18.437 DEBUG:rgw_multi.tests:bucket checkpoint: source=a2 target=a1 bucket=pklxkd-61 2023-05-31T21:03:18.438 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'pklxkd-61', '--source-zone', 'a2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-05-31T21:03:18.438 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'pklxkd-61', '--source-zone', 'a2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-05-31T21:03:18.438 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket pklxkd-61 --source-zone a2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2023-05-31T21:03:18.500 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2023-05-31T21:03:18.500 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2023-05-31T21:03:18.541 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T21:03:18.536+0000 7fbf4b6b5780 1 bucket sync caught up with source: 2023-05-31T21:03:18.542 INFO:teuthology.orchestra.run.smithi099.stderr: local status: [, 00000000003.1759.6, 00000000002.1346.6, 00000000002.1737.6, 00000000005.1095.6, 00000000004.2268.6, 00000000001.2090.6, 00000000001.2290.6, 00000000004.2267.6, 00000000007.1757.6, 00000000006.1089.6] 2023-05-31T21:03:18.542 INFO:teuthology.orchestra.run.smithi099.stderr: remote markers: [, 00000000003.1759.6, 00000000002.1346.6, 00000000002.1737.6, 00000000005.1095.6, 00000000004.2268.6, 00000000001.2090.6, 00000000001.2290.6, 00000000004.2267.6, 00000000007.1757.6, 00000000006.1089.6] 2023-05-31T21:03:18.542 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T21:03:18.536+0000 7fbf4b6b5780 0 bucket checkpoint complete 2023-05-31T21:03:18.555 INFO:tasks.rgw_multi.zone_rados:comparing bucket=pklxkd-61 zones={a2, a1} 2023-05-31T21:03:18.556 DEBUG:boto:path=/pklxkd-61/ 2023-05-31T21:03:18.556 DEBUG:boto:auth_path=/pklxkd-61/ 2023-05-31T21:03:18.557 DEBUG:boto:Method: HEAD 2023-05-31T21:03:18.557 DEBUG:boto:Path: /pklxkd-61/ 2023-05-31T21:03:18.557 DEBUG:boto:Data: 2023-05-31T21:03:18.558 DEBUG:boto:Headers: {} 2023-05-31T21:03:18.558 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T21:03:18.558 DEBUG:boto:Port: 8001 2023-05-31T21:03:18.559 DEBUG:boto:Params: {} 2023-05-31T21:03:18.559 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8001} 2023-05-31T21:03:18.560 DEBUG:boto:Token: None 2023-05-31T21:03:18.560 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 21:03:18 GMT /pklxkd-61/ 2023-05-31T21:03:18.561 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:FkeDmev8LSgxSmqIl3FuisDMmmw= 2023-05-31T21:03:18.561 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 21:03:18 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:FkeDmev8LSgxSmqIl3FuisDMmmw=', 'Content-Length': '0'} 2023-05-31T21:03:18.565 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '35'), ('X-RGW-Bytes-Used', '105'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000831951e65a3cb07f-006477b616-1f4e-a2'), ('Content-Length', '0'), ('Date', 'Wed, 31 May 2023 21:03:18 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:03:18.566 DEBUG:boto:path=/pklxkd-61/ 2023-05-31T21:03:18.566 DEBUG:boto:auth_path=/pklxkd-61/ 2023-05-31T21:03:18.567 DEBUG:boto:Method: HEAD 2023-05-31T21:03:18.567 DEBUG:boto:Path: /pklxkd-61/ 2023-05-31T21:03:18.567 DEBUG:boto:Data: 2023-05-31T21:03:18.568 DEBUG:boto:Headers: {} 2023-05-31T21:03:18.569 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T21:03:18.569 DEBUG:boto:Port: 8000 2023-05-31T21:03:18.569 DEBUG:boto:Params: {} 2023-05-31T21:03:18.570 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2023-05-31T21:03:18.570 DEBUG:boto:Token: None 2023-05-31T21:03:18.571 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 21:03:18 GMT /pklxkd-61/ 2023-05-31T21:03:18.571 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:FkeDmev8LSgxSmqIl3FuisDMmmw= 2023-05-31T21:03:18.571 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 21:03:18 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:FkeDmev8LSgxSmqIl3FuisDMmmw=', 'Content-Length': '0'} 2023-05-31T21:03:18.575 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '35'), ('X-RGW-Bytes-Used', '105'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000005f0e8295e705e975-006477b616-1630-a1'), ('Content-Length', '0'), ('Date', 'Wed, 31 May 2023 21:03:18 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:03:18.576 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2023-05-31T21:03:18.576 DEBUG:boto:path=/pklxkd-61/ 2023-05-31T21:03:18.577 DEBUG:boto:auth_path=/pklxkd-61/ 2023-05-31T21:03:18.577 DEBUG:boto:path=/pklxkd-61/?versions&max-keys=999 2023-05-31T21:03:18.577 DEBUG:boto:auth_path=/pklxkd-61/?versions&max-keys=999 2023-05-31T21:03:18.578 DEBUG:boto:Method: GET 2023-05-31T21:03:18.578 DEBUG:boto:Path: /pklxkd-61/?versions&max-keys=999 2023-05-31T21:03:18.578 DEBUG:boto:Data: 2023-05-31T21:03:18.579 DEBUG:boto:Headers: {} 2023-05-31T21:03:18.579 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T21:03:18.579 DEBUG:boto:Port: 8001 2023-05-31T21:03:18.580 DEBUG:boto:Params: {} 2023-05-31T21:03:18.580 DEBUG:boto:Token: None 2023-05-31T21:03:18.581 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 21:03:18 GMT /pklxkd-61/?versions 2023-05-31T21:03:18.581 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:Apa6wjvbcwEVIV91DGYY3STgyBQ= 2023-05-31T21:03:18.581 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 21:03:18 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:Apa6wjvbcwEVIV91DGYY3STgyBQ=', 'Content-Length': '0'} 2023-05-31T21:03:18.584 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000151c52cf989075c8-006477b616-1f4e-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 21:03:18 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:03:18.585 DEBUG:boto:b'pklxkd-61999falseobj11nulltrue2023-05-31T20:59:44.431Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj111nulltrue2023-05-31T20:59:46.951Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj115nulltrue2023-05-31T20:59:45.636Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj116nulltrue2023-05-31T20:59:45.977Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj124nulltrue2023-05-31T20:59:46.419Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj145nulltrue2023-05-31T20:59:46.038Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj148nulltrue2023-05-31T20:59:47.007Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj155nulltrue2023-05-31T20:59:46.438Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj17nulltrue2023-05-31T20:59:44.769Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj185nulltrue2023-05-31T20:59:47.031Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj19nulltrue2023-05-31T20:59:45.164Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj22nulltrue2023-05-31T20:59:44.449Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj23nulltrue2023-05-31T20:59:45.517Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj29nulltrue2023-05-31T20:59:45.888Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj31nulltrue2023-05-31T20:59:46.280Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj33nulltrue2023-05-31T20:59:44.466Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj34nulltrue2023-05-31T20:59:44.822Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj37nulltrue2023-05-31T20:59:46.915Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj38nulltrue2023-05-31T20:59:45.182Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj44nulltrue2023-05-31T20:59:44.523Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj46nulltrue2023-05-31T20:59:45.539Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj51nulltrue2023-05-31T20:59:44.841Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj55nulltrue2023-05-31T20:59:44.541Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj57nulltrue2023-05-31T20:59:45.197Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj58nulltrue2023-05-31T20:59:45.905Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj62nulltrue2023-05-31T20:59:46.340Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj68nulltrue2023-05-31T20:59:44.894Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj69nulltrue2023-05-31T20:59:45.559Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj74nulltrue2023-05-31T20:59:46.933Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj76nulltrue2023-05-31T20:59:45.256Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj85nulltrue2023-05-31T20:59:44.912Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj87nulltrue2023-05-31T20:59:45.959Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj92nulltrue2023-05-31T20:59:45.616Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj93nulltrue2023-05-31T20:59:46.358Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj95nulltrue2023-05-31T20:59:45.274Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T21:03:18.587 DEBUG:tasks.rgw_multi.zone_rados:o=obj11 2023-05-31T21:03:18.587 DEBUG:tasks.rgw_multi.zone_rados:o=obj111 2023-05-31T21:03:18.588 DEBUG:tasks.rgw_multi.zone_rados:o=obj115 2023-05-31T21:03:18.588 DEBUG:tasks.rgw_multi.zone_rados:o=obj116 2023-05-31T21:03:18.588 DEBUG:tasks.rgw_multi.zone_rados:o=obj124 2023-05-31T21:03:18.589 DEBUG:tasks.rgw_multi.zone_rados:o=obj145 2023-05-31T21:03:18.589 DEBUG:tasks.rgw_multi.zone_rados:o=obj148 2023-05-31T21:03:18.589 DEBUG:tasks.rgw_multi.zone_rados:o=obj155 2023-05-31T21:03:18.589 DEBUG:tasks.rgw_multi.zone_rados:o=obj17 2023-05-31T21:03:18.590 DEBUG:tasks.rgw_multi.zone_rados:o=obj185 2023-05-31T21:03:18.590 DEBUG:tasks.rgw_multi.zone_rados:o=obj19 2023-05-31T21:03:18.590 DEBUG:tasks.rgw_multi.zone_rados:o=obj22 2023-05-31T21:03:18.591 DEBUG:tasks.rgw_multi.zone_rados:o=obj23 2023-05-31T21:03:18.591 DEBUG:tasks.rgw_multi.zone_rados:o=obj29 2023-05-31T21:03:18.591 DEBUG:tasks.rgw_multi.zone_rados:o=obj31 2023-05-31T21:03:18.592 DEBUG:tasks.rgw_multi.zone_rados:o=obj33 2023-05-31T21:03:18.592 DEBUG:tasks.rgw_multi.zone_rados:o=obj34 2023-05-31T21:03:18.592 DEBUG:tasks.rgw_multi.zone_rados:o=obj37 2023-05-31T21:03:18.592 DEBUG:tasks.rgw_multi.zone_rados:o=obj38 2023-05-31T21:03:18.593 DEBUG:tasks.rgw_multi.zone_rados:o=obj44 2023-05-31T21:03:18.593 DEBUG:tasks.rgw_multi.zone_rados:o=obj46 2023-05-31T21:03:18.594 DEBUG:tasks.rgw_multi.zone_rados:o=obj51 2023-05-31T21:03:18.594 DEBUG:tasks.rgw_multi.zone_rados:o=obj55 2023-05-31T21:03:18.594 DEBUG:tasks.rgw_multi.zone_rados:o=obj57 2023-05-31T21:03:18.594 DEBUG:tasks.rgw_multi.zone_rados:o=obj58 2023-05-31T21:03:18.595 DEBUG:tasks.rgw_multi.zone_rados:o=obj62 2023-05-31T21:03:18.595 DEBUG:tasks.rgw_multi.zone_rados:o=obj68 2023-05-31T21:03:18.595 DEBUG:tasks.rgw_multi.zone_rados:o=obj69 2023-05-31T21:03:18.596 DEBUG:tasks.rgw_multi.zone_rados:o=obj74 2023-05-31T21:03:18.596 DEBUG:tasks.rgw_multi.zone_rados:o=obj76 2023-05-31T21:03:18.596 DEBUG:tasks.rgw_multi.zone_rados:o=obj85 2023-05-31T21:03:18.597 DEBUG:tasks.rgw_multi.zone_rados:o=obj87 2023-05-31T21:03:18.597 DEBUG:tasks.rgw_multi.zone_rados:o=obj92 2023-05-31T21:03:18.597 DEBUG:tasks.rgw_multi.zone_rados:o=obj93 2023-05-31T21:03:18.598 DEBUG:tasks.rgw_multi.zone_rados:o=obj95 2023-05-31T21:03:18.598 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2023-05-31T21:03:18.598 DEBUG:boto:path=/pklxkd-61/ 2023-05-31T21:03:18.599 DEBUG:boto:auth_path=/pklxkd-61/ 2023-05-31T21:03:18.599 DEBUG:boto:path=/pklxkd-61/?versions&max-keys=999 2023-05-31T21:03:18.599 DEBUG:boto:auth_path=/pklxkd-61/?versions&max-keys=999 2023-05-31T21:03:18.600 DEBUG:boto:Method: GET 2023-05-31T21:03:18.600 DEBUG:boto:Path: /pklxkd-61/?versions&max-keys=999 2023-05-31T21:03:18.600 DEBUG:boto:Data: 2023-05-31T21:03:18.601 DEBUG:boto:Headers: {} 2023-05-31T21:03:18.601 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T21:03:18.601 DEBUG:boto:Port: 8000 2023-05-31T21:03:18.602 DEBUG:boto:Params: {} 2023-05-31T21:03:18.602 DEBUG:boto:Token: None 2023-05-31T21:03:18.602 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 21:03:18 GMT /pklxkd-61/?versions 2023-05-31T21:03:18.603 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:Apa6wjvbcwEVIV91DGYY3STgyBQ= 2023-05-31T21:03:18.603 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 21:03:18 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:Apa6wjvbcwEVIV91DGYY3STgyBQ=', 'Content-Length': '0'} 2023-05-31T21:03:18.607 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c58bdf9d2ddf5580-006477b616-1630-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 21:03:18 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:03:18.608 DEBUG:boto:b'pklxkd-61999falseobj11nulltrue2023-05-31T20:59:44.431Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj111nulltrue2023-05-31T20:59:46.951Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj115nulltrue2023-05-31T20:59:45.636Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj116nulltrue2023-05-31T20:59:45.977Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj124nulltrue2023-05-31T20:59:46.419Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj145nulltrue2023-05-31T20:59:46.038Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj148nulltrue2023-05-31T20:59:47.007Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj155nulltrue2023-05-31T20:59:46.438Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj17nulltrue2023-05-31T20:59:44.769Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj185nulltrue2023-05-31T20:59:47.031Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj19nulltrue2023-05-31T20:59:45.164Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj22nulltrue2023-05-31T20:59:44.449Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj23nulltrue2023-05-31T20:59:45.517Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj29nulltrue2023-05-31T20:59:45.888Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj31nulltrue2023-05-31T20:59:46.280Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj33nulltrue2023-05-31T20:59:44.466Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj34nulltrue2023-05-31T20:59:44.822Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj37nulltrue2023-05-31T20:59:46.915Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj38nulltrue2023-05-31T20:59:45.182Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj44nulltrue2023-05-31T20:59:44.523Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj46nulltrue2023-05-31T20:59:45.539Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj51nulltrue2023-05-31T20:59:44.841Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj55nulltrue2023-05-31T20:59:44.541Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj57nulltrue2023-05-31T20:59:45.197Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj58nulltrue2023-05-31T20:59:45.905Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj62nulltrue2023-05-31T20:59:46.340Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj68nulltrue2023-05-31T20:59:44.894Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj69nulltrue2023-05-31T20:59:45.559Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj74nulltrue2023-05-31T20:59:46.933Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj76nulltrue2023-05-31T20:59:45.256Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj85nulltrue2023-05-31T20:59:44.912Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj87nulltrue2023-05-31T20:59:45.959Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj92nulltrue2023-05-31T20:59:45.616Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj93nulltrue2023-05-31T20:59:46.358Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj95nulltrue2023-05-31T20:59:45.274Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T21:03:18.610 DEBUG:tasks.rgw_multi.zone_rados:o=obj11 2023-05-31T21:03:18.610 DEBUG:tasks.rgw_multi.zone_rados:o=obj111 2023-05-31T21:03:18.610 DEBUG:tasks.rgw_multi.zone_rados:o=obj115 2023-05-31T21:03:18.611 DEBUG:tasks.rgw_multi.zone_rados:o=obj116 2023-05-31T21:03:18.611 DEBUG:tasks.rgw_multi.zone_rados:o=obj124 2023-05-31T21:03:18.611 DEBUG:tasks.rgw_multi.zone_rados:o=obj145 2023-05-31T21:03:18.611 DEBUG:tasks.rgw_multi.zone_rados:o=obj148 2023-05-31T21:03:18.612 DEBUG:tasks.rgw_multi.zone_rados:o=obj155 2023-05-31T21:03:18.612 DEBUG:tasks.rgw_multi.zone_rados:o=obj17 2023-05-31T21:03:18.612 DEBUG:tasks.rgw_multi.zone_rados:o=obj185 2023-05-31T21:03:18.613 DEBUG:tasks.rgw_multi.zone_rados:o=obj19 2023-05-31T21:03:18.613 DEBUG:tasks.rgw_multi.zone_rados:o=obj22 2023-05-31T21:03:18.613 DEBUG:tasks.rgw_multi.zone_rados:o=obj23 2023-05-31T21:03:18.614 DEBUG:tasks.rgw_multi.zone_rados:o=obj29 2023-05-31T21:03:18.614 DEBUG:tasks.rgw_multi.zone_rados:o=obj31 2023-05-31T21:03:18.614 DEBUG:tasks.rgw_multi.zone_rados:o=obj33 2023-05-31T21:03:18.615 DEBUG:tasks.rgw_multi.zone_rados:o=obj34 2023-05-31T21:03:18.615 DEBUG:tasks.rgw_multi.zone_rados:o=obj37 2023-05-31T21:03:18.616 DEBUG:tasks.rgw_multi.zone_rados:o=obj38 2023-05-31T21:03:18.617 DEBUG:tasks.rgw_multi.zone_rados:o=obj44 2023-05-31T21:03:18.618 DEBUG:tasks.rgw_multi.zone_rados:o=obj46 2023-05-31T21:03:18.618 DEBUG:tasks.rgw_multi.zone_rados:o=obj51 2023-05-31T21:03:18.619 DEBUG:tasks.rgw_multi.zone_rados:o=obj55 2023-05-31T21:03:18.620 DEBUG:tasks.rgw_multi.zone_rados:o=obj57 2023-05-31T21:03:18.620 DEBUG:tasks.rgw_multi.zone_rados:o=obj58 2023-05-31T21:03:18.621 DEBUG:tasks.rgw_multi.zone_rados:o=obj62 2023-05-31T21:03:18.622 DEBUG:tasks.rgw_multi.zone_rados:o=obj68 2023-05-31T21:03:18.622 DEBUG:tasks.rgw_multi.zone_rados:o=obj69 2023-05-31T21:03:18.623 DEBUG:tasks.rgw_multi.zone_rados:o=obj74 2023-05-31T21:03:18.623 DEBUG:tasks.rgw_multi.zone_rados:o=obj76 2023-05-31T21:03:18.624 DEBUG:tasks.rgw_multi.zone_rados:o=obj85 2023-05-31T21:03:18.625 DEBUG:tasks.rgw_multi.zone_rados:o=obj87 2023-05-31T21:03:18.625 DEBUG:tasks.rgw_multi.zone_rados:o=obj92 2023-05-31T21:03:18.625 DEBUG:tasks.rgw_multi.zone_rados:o=obj93 2023-05-31T21:03:18.626 DEBUG:tasks.rgw_multi.zone_rados:o=obj95 2023-05-31T21:03:18.626 DEBUG:boto:path=/pklxkd-61/ 2023-05-31T21:03:18.626 DEBUG:boto:auth_path=/pklxkd-61/ 2023-05-31T21:03:18.627 DEBUG:boto:path=/pklxkd-61/?versions&max-keys=999 2023-05-31T21:03:18.627 DEBUG:boto:auth_path=/pklxkd-61/?versions&max-keys=999 2023-05-31T21:03:18.627 DEBUG:boto:Method: GET 2023-05-31T21:03:18.628 DEBUG:boto:Path: /pklxkd-61/?versions&max-keys=999 2023-05-31T21:03:18.628 DEBUG:boto:Data: 2023-05-31T21:03:18.628 DEBUG:boto:Headers: {} 2023-05-31T21:03:18.629 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T21:03:18.629 DEBUG:boto:Port: 8001 2023-05-31T21:03:18.629 DEBUG:boto:Params: {} 2023-05-31T21:03:18.630 DEBUG:boto:Token: None 2023-05-31T21:03:18.630 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 21:03:18 GMT /pklxkd-61/?versions 2023-05-31T21:03:18.631 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:Apa6wjvbcwEVIV91DGYY3STgyBQ= 2023-05-31T21:03:18.631 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 21:03:18 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:Apa6wjvbcwEVIV91DGYY3STgyBQ=', 'Content-Length': '0'} 2023-05-31T21:03:18.633 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d8a1d0b36c959d15-006477b616-1f4e-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 21:03:18 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:03:18.634 DEBUG:boto:b'pklxkd-61999falseobj11nulltrue2023-05-31T20:59:44.431Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj111nulltrue2023-05-31T20:59:46.951Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj115nulltrue2023-05-31T20:59:45.636Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj116nulltrue2023-05-31T20:59:45.977Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj124nulltrue2023-05-31T20:59:46.419Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj145nulltrue2023-05-31T20:59:46.038Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj148nulltrue2023-05-31T20:59:47.007Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj155nulltrue2023-05-31T20:59:46.438Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj17nulltrue2023-05-31T20:59:44.769Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj185nulltrue2023-05-31T20:59:47.031Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj19nulltrue2023-05-31T20:59:45.164Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj22nulltrue2023-05-31T20:59:44.449Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj23nulltrue2023-05-31T20:59:45.517Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj29nulltrue2023-05-31T20:59:45.888Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj31nulltrue2023-05-31T20:59:46.280Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj33nulltrue2023-05-31T20:59:44.466Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj34nulltrue2023-05-31T20:59:44.822Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj37nulltrue2023-05-31T20:59:46.915Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj38nulltrue2023-05-31T20:59:45.182Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj44nulltrue2023-05-31T20:59:44.523Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj46nulltrue2023-05-31T20:59:45.539Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj51nulltrue2023-05-31T20:59:44.841Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj55nulltrue2023-05-31T20:59:44.541Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj57nulltrue2023-05-31T20:59:45.197Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj58nulltrue2023-05-31T20:59:45.905Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj62nulltrue2023-05-31T20:59:46.340Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj68nulltrue2023-05-31T20:59:44.894Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj69nulltrue2023-05-31T20:59:45.559Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj74nulltrue2023-05-31T20:59:46.933Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj76nulltrue2023-05-31T20:59:45.256Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj85nulltrue2023-05-31T20:59:44.912Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj87nulltrue2023-05-31T20:59:45.959Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj92nulltrue2023-05-31T20:59:45.616Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj93nulltrue2023-05-31T20:59:46.358Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj95nulltrue2023-05-31T20:59:45.274Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T21:03:18.636 DEBUG:boto:path=/pklxkd-61/ 2023-05-31T21:03:18.636 DEBUG:boto:auth_path=/pklxkd-61/ 2023-05-31T21:03:18.637 DEBUG:boto:path=/pklxkd-61/?versions&max-keys=999 2023-05-31T21:03:18.637 DEBUG:boto:auth_path=/pklxkd-61/?versions&max-keys=999 2023-05-31T21:03:18.637 DEBUG:boto:Method: GET 2023-05-31T21:03:18.638 DEBUG:boto:Path: /pklxkd-61/?versions&max-keys=999 2023-05-31T21:03:18.638 DEBUG:boto:Data: 2023-05-31T21:03:18.638 DEBUG:boto:Headers: {} 2023-05-31T21:03:18.639 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T21:03:18.639 DEBUG:boto:Port: 8000 2023-05-31T21:03:18.639 DEBUG:boto:Params: {} 2023-05-31T21:03:18.640 DEBUG:boto:Token: None 2023-05-31T21:03:18.640 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 21:03:18 GMT /pklxkd-61/?versions 2023-05-31T21:03:18.641 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:Apa6wjvbcwEVIV91DGYY3STgyBQ= 2023-05-31T21:03:18.641 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 21:03:18 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:Apa6wjvbcwEVIV91DGYY3STgyBQ=', 'Content-Length': '0'} 2023-05-31T21:03:18.644 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000943ffab517cb7aed-006477b616-1630-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 21:03:18 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:03:18.645 DEBUG:boto:b'pklxkd-61999falseobj11nulltrue2023-05-31T20:59:44.431Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj111nulltrue2023-05-31T20:59:46.951Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj115nulltrue2023-05-31T20:59:45.636Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj116nulltrue2023-05-31T20:59:45.977Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj124nulltrue2023-05-31T20:59:46.419Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj145nulltrue2023-05-31T20:59:46.038Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj148nulltrue2023-05-31T20:59:47.007Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj155nulltrue2023-05-31T20:59:46.438Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj17nulltrue2023-05-31T20:59:44.769Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj185nulltrue2023-05-31T20:59:47.031Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj19nulltrue2023-05-31T20:59:45.164Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj22nulltrue2023-05-31T20:59:44.449Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj23nulltrue2023-05-31T20:59:45.517Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj29nulltrue2023-05-31T20:59:45.888Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj31nulltrue2023-05-31T20:59:46.280Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj33nulltrue2023-05-31T20:59:44.466Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj34nulltrue2023-05-31T20:59:44.822Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj37nulltrue2023-05-31T20:59:46.915Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj38nulltrue2023-05-31T20:59:45.182Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj44nulltrue2023-05-31T20:59:44.523Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj46nulltrue2023-05-31T20:59:45.539Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj51nulltrue2023-05-31T20:59:44.841Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj55nulltrue2023-05-31T20:59:44.541Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj57nulltrue2023-05-31T20:59:45.197Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj58nulltrue2023-05-31T20:59:45.905Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj62nulltrue2023-05-31T20:59:46.340Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj68nulltrue2023-05-31T20:59:44.894Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj69nulltrue2023-05-31T20:59:45.559Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj74nulltrue2023-05-31T20:59:46.933Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj76nulltrue2023-05-31T20:59:45.256Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj85nulltrue2023-05-31T20:59:44.912Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj87nulltrue2023-05-31T20:59:45.959Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj92nulltrue2023-05-31T20:59:45.616Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj93nulltrue2023-05-31T20:59:46.358Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj95nulltrue2023-05-31T20:59:45.274Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T21:03:18.649 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj11 2023-05-31T21:03:18.649 DEBUG:boto:path=/pklxkd-61/obj11 2023-05-31T21:03:18.649 DEBUG:boto:auth_path=/pklxkd-61/obj11 2023-05-31T21:03:18.650 DEBUG:boto:path=/pklxkd-61/obj11?versionId=null 2023-05-31T21:03:18.650 DEBUG:boto:auth_path=/pklxkd-61/obj11?versionId=null 2023-05-31T21:03:18.650 DEBUG:boto:Method: GET 2023-05-31T21:03:18.651 DEBUG:boto:Path: /pklxkd-61/obj11?versionId=null 2023-05-31T21:03:18.651 DEBUG:boto:Data: 2023-05-31T21:03:18.652 DEBUG:boto:Headers: {} 2023-05-31T21:03:18.652 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T21:03:18.652 DEBUG:boto:Port: 8001 2023-05-31T21:03:18.653 DEBUG:boto:Params: {} 2023-05-31T21:03:18.653 DEBUG:boto:Token: None 2023-05-31T21:03:18.654 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 21:03:18 GMT /pklxkd-61/obj11?versionId=null 2023-05-31T21:03:18.654 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:dhPbjlugGoa5lObMLT0HKksQN84= 2023-05-31T21:03:18.654 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 21:03:18 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:dhPbjlugGoa5lObMLT0HKksQN84=', 'Content-Length': '0'} 2023-05-31T21:03:18.656 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:59:44 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000086eba8c6177e712b-006477b616-1f4e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 21:03:18 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:03:18.657 DEBUG:boto:path=/pklxkd-61/obj11 2023-05-31T21:03:18.657 DEBUG:boto:auth_path=/pklxkd-61/obj11 2023-05-31T21:03:18.658 DEBUG:boto:path=/pklxkd-61/obj11?versionId=null 2023-05-31T21:03:18.658 DEBUG:boto:auth_path=/pklxkd-61/obj11?versionId=null 2023-05-31T21:03:18.658 DEBUG:boto:Method: GET 2023-05-31T21:03:18.659 DEBUG:boto:Path: /pklxkd-61/obj11?versionId=null 2023-05-31T21:03:18.659 DEBUG:boto:Data: 2023-05-31T21:03:18.660 DEBUG:boto:Headers: {} 2023-05-31T21:03:18.660 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T21:03:18.660 DEBUG:boto:Port: 8000 2023-05-31T21:03:18.661 DEBUG:boto:Params: {} 2023-05-31T21:03:18.661 DEBUG:boto:Token: None 2023-05-31T21:03:18.662 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 21:03:18 GMT /pklxkd-61/obj11?versionId=null 2023-05-31T21:03:18.662 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:dhPbjlugGoa5lObMLT0HKksQN84= 2023-05-31T21:03:18.662 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 21:03:18 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:dhPbjlugGoa5lObMLT0HKksQN84=', 'Content-Length': '0'} 2023-05-31T21:03:18.664 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:59:44 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007bd2bf75a2e6815d-006477b616-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 21:03:18 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:03:18.665 DEBUG:boto:path=/pklxkd-61/obj11 2023-05-31T21:03:18.665 DEBUG:boto:auth_path=/pklxkd-61/obj11 2023-05-31T21:03:18.666 DEBUG:boto:path=/pklxkd-61/obj11?versionId=null 2023-05-31T21:03:18.666 DEBUG:boto:auth_path=/pklxkd-61/obj11?versionId=null 2023-05-31T21:03:18.666 DEBUG:boto:Method: HEAD 2023-05-31T21:03:18.667 DEBUG:boto:Path: /pklxkd-61/obj11?versionId=null 2023-05-31T21:03:18.667 DEBUG:boto:Data: 2023-05-31T21:03:18.667 DEBUG:boto:Headers: {} 2023-05-31T21:03:18.668 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T21:03:18.668 DEBUG:boto:Port: 8001 2023-05-31T21:03:18.669 DEBUG:boto:Params: {} 2023-05-31T21:03:18.669 DEBUG:boto:Token: None 2023-05-31T21:03:18.669 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 21:03:18 GMT /pklxkd-61/obj11?versionId=null 2023-05-31T21:03:18.670 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:s54ZGWFOlh/NmS3NHiaglDl9vcU= 2023-05-31T21:03:18.670 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 21:03:18 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:s54ZGWFOlh/NmS3NHiaglDl9vcU=', 'Content-Length': '0'} 2023-05-31T21:03:18.672 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:59:44 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d5ce8ad8fec49986-006477b616-1f4e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 21:03:18 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:03:18.672 DEBUG:boto:path=/pklxkd-61/obj11 2023-05-31T21:03:18.673 DEBUG:boto:auth_path=/pklxkd-61/obj11 2023-05-31T21:03:18.673 DEBUG:boto:path=/pklxkd-61/obj11?versionId=null 2023-05-31T21:03:18.673 DEBUG:boto:auth_path=/pklxkd-61/obj11?versionId=null 2023-05-31T21:03:18.674 DEBUG:boto:Method: HEAD 2023-05-31T21:03:18.674 DEBUG:boto:Path: /pklxkd-61/obj11?versionId=null 2023-05-31T21:03:18.675 DEBUG:boto:Data: 2023-05-31T21:03:18.675 DEBUG:boto:Headers: {} 2023-05-31T21:03:18.676 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T21:03:18.676 DEBUG:boto:Port: 8000 2023-05-31T21:03:18.676 DEBUG:boto:Params: {} 2023-05-31T21:03:18.677 DEBUG:boto:Token: None 2023-05-31T21:03:18.677 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 21:03:18 GMT /pklxkd-61/obj11?versionId=null 2023-05-31T21:03:18.677 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:s54ZGWFOlh/NmS3NHiaglDl9vcU= 2023-05-31T21:03:18.678 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 21:03:18 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:s54ZGWFOlh/NmS3NHiaglDl9vcU=', 'Content-Length': '0'} 2023-05-31T21:03:18.679 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:59:44 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009e336b970aee0e1c-006477b616-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 21:03:18 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:03:18.680 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj11 2023-05-31T21:03:18.680 DEBUG:boto:path=/pklxkd-61/obj11 2023-05-31T21:03:18.681 DEBUG:boto:auth_path=/pklxkd-61/obj11 2023-05-31T21:03:18.681 DEBUG:boto:path=/pklxkd-61/obj11?versionId=null 2023-05-31T21:03:18.681 DEBUG:boto:auth_path=/pklxkd-61/obj11?versionId=null 2023-05-31T21:03:18.682 DEBUG:boto:Method: GET 2023-05-31T21:03:18.682 DEBUG:boto:Path: /pklxkd-61/obj11?versionId=null 2023-05-31T21:03:18.682 DEBUG:boto:Data: 2023-05-31T21:03:18.683 DEBUG:boto:Headers: {} 2023-05-31T21:03:18.683 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T21:03:18.683 DEBUG:boto:Port: 8001 2023-05-31T21:03:18.684 DEBUG:boto:Params: {} 2023-05-31T21:03:18.684 DEBUG:boto:Token: None 2023-05-31T21:03:18.685 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 21:03:18 GMT /pklxkd-61/obj11?versionId=null 2023-05-31T21:03:18.685 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:dhPbjlugGoa5lObMLT0HKksQN84= 2023-05-31T21:03:18.685 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 21:03:18 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:dhPbjlugGoa5lObMLT0HKksQN84=', 'Content-Length': '0'} 2023-05-31T21:03:18.687 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:59:44 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000e3ca49825c1ac30-006477b616-1f4e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 21:03:18 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:03:18.728 DEBUG:boto:path=/pklxkd-61/obj11 2023-05-31T21:03:18.729 DEBUG:boto:auth_path=/pklxkd-61/obj11 2023-05-31T21:03:18.729 DEBUG:boto:path=/pklxkd-61/obj11?versionId=null 2023-05-31T21:03:18.730 DEBUG:boto:auth_path=/pklxkd-61/obj11?versionId=null 2023-05-31T21:03:18.730 DEBUG:boto:Method: GET 2023-05-31T21:03:18.730 DEBUG:boto:Path: /pklxkd-61/obj11?versionId=null 2023-05-31T21:03:18.731 DEBUG:boto:Data: 2023-05-31T21:03:18.731 DEBUG:boto:Headers: {} 2023-05-31T21:03:18.731 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T21:03:18.732 DEBUG:boto:Port: 8000 2023-05-31T21:03:18.733 DEBUG:boto:Params: {} 2023-05-31T21:03:18.733 DEBUG:boto:Token: None 2023-05-31T21:03:18.734 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 21:03:18 GMT /pklxkd-61/obj11?versionId=null 2023-05-31T21:03:18.735 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:dhPbjlugGoa5lObMLT0HKksQN84= 2023-05-31T21:03:18.735 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 21:03:18 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:dhPbjlugGoa5lObMLT0HKksQN84=', 'Content-Length': '0'} 2023-05-31T21:03:18.738 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:59:44 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b7d36e76d4a79d25-006477b616-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 21:03:18 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:03:18.739 DEBUG:boto:path=/pklxkd-61/ 2023-05-31T21:03:18.739 DEBUG:boto:auth_path=/pklxkd-61/ 2023-05-31T21:03:18.740 DEBUG:boto:path=/pklxkd-61/?prefix=obj11 2023-05-31T21:03:18.741 DEBUG:boto:auth_path=/pklxkd-61/?prefix=obj11 2023-05-31T21:03:18.741 DEBUG:boto:Method: GET 2023-05-31T21:03:18.742 DEBUG:boto:Path: /pklxkd-61/?prefix=obj11 2023-05-31T21:03:18.742 DEBUG:boto:Data: 2023-05-31T21:03:18.743 DEBUG:boto:Headers: {} 2023-05-31T21:03:18.743 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T21:03:18.744 DEBUG:boto:Port: 8001 2023-05-31T21:03:18.744 DEBUG:boto:Params: {} 2023-05-31T21:03:18.745 DEBUG:boto:Token: None 2023-05-31T21:03:18.746 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 21:03:18 GMT /pklxkd-61/ 2023-05-31T21:03:18.746 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:qoQ+UVg4HBmCbZSqS0auFpBGtNM= 2023-05-31T21:03:18.747 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 21:03:18 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:qoQ+UVg4HBmCbZSqS0auFpBGtNM=', 'Content-Length': '0'} 2023-05-31T21:03:18.750 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000070b1648359b91af-006477b616-1f4e-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 21:03:18 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:03:18.750 DEBUG:boto:b'pklxkd-61obj111000falseobj112023-05-31T20:59:44.431Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj1112023-05-31T20:59:46.951Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj1152023-05-31T20:59:45.636Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj1162023-05-31T20:59:45.977Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T21:03:18.751 DEBUG:boto:path=/pklxkd-61/ 2023-05-31T21:03:18.752 DEBUG:boto:auth_path=/pklxkd-61/ 2023-05-31T21:03:18.752 DEBUG:boto:path=/pklxkd-61/?prefix=obj11 2023-05-31T21:03:18.752 DEBUG:boto:auth_path=/pklxkd-61/?prefix=obj11 2023-05-31T21:03:18.753 DEBUG:boto:Method: GET 2023-05-31T21:03:18.753 DEBUG:boto:Path: /pklxkd-61/?prefix=obj11 2023-05-31T21:03:18.754 DEBUG:boto:Data: 2023-05-31T21:03:18.754 DEBUG:boto:Headers: {} 2023-05-31T21:03:18.754 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T21:03:18.755 DEBUG:boto:Port: 8000 2023-05-31T21:03:18.755 DEBUG:boto:Params: {} 2023-05-31T21:03:18.755 DEBUG:boto:Token: None 2023-05-31T21:03:18.756 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 21:03:18 GMT /pklxkd-61/ 2023-05-31T21:03:18.756 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:qoQ+UVg4HBmCbZSqS0auFpBGtNM= 2023-05-31T21:03:18.757 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 21:03:18 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:qoQ+UVg4HBmCbZSqS0auFpBGtNM=', 'Content-Length': '0'} 2023-05-31T21:03:18.760 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000033a6da36cfe5f12d-006477b616-1630-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 21:03:18 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:03:18.761 DEBUG:boto:b'pklxkd-61obj111000falseobj112023-05-31T20:59:44.431Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj1112023-05-31T20:59:46.951Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj1152023-05-31T20:59:45.636Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj1162023-05-31T20:59:45.977Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T21:03:18.762 DEBUG:boto:path=/pklxkd-61/obj11 2023-05-31T21:03:18.762 DEBUG:boto:auth_path=/pklxkd-61/obj11 2023-05-31T21:03:18.762 DEBUG:boto:Method: HEAD 2023-05-31T21:03:18.763 DEBUG:boto:Path: /pklxkd-61/obj11 2023-05-31T21:03:18.763 DEBUG:boto:Data: 2023-05-31T21:03:18.763 DEBUG:boto:Headers: {} 2023-05-31T21:03:18.764 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T21:03:18.764 DEBUG:boto:Port: 8001 2023-05-31T21:03:18.764 DEBUG:boto:Params: {} 2023-05-31T21:03:18.765 DEBUG:boto:Token: None 2023-05-31T21:03:18.765 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 21:03:18 GMT /pklxkd-61/obj11 2023-05-31T21:03:18.766 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:W8rqJhCq3ybyvYhPqU5swcGLuaE= 2023-05-31T21:03:18.766 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 21:03:18 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:W8rqJhCq3ybyvYhPqU5swcGLuaE=', 'Content-Length': '0'} 2023-05-31T21:03:18.768 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:59:44 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000097347eb4120522ab-006477b616-1f4e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 21:03:18 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:03:18.768 DEBUG:boto:path=/pklxkd-61/obj11 2023-05-31T21:03:18.769 DEBUG:boto:auth_path=/pklxkd-61/obj11 2023-05-31T21:03:18.769 DEBUG:boto:Method: HEAD 2023-05-31T21:03:18.769 DEBUG:boto:Path: /pklxkd-61/obj11 2023-05-31T21:03:18.770 DEBUG:boto:Data: 2023-05-31T21:03:18.770 DEBUG:boto:Headers: {} 2023-05-31T21:03:18.770 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T21:03:18.771 DEBUG:boto:Port: 8000 2023-05-31T21:03:18.771 DEBUG:boto:Params: {} 2023-05-31T21:03:18.771 DEBUG:boto:Token: None 2023-05-31T21:03:18.772 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 21:03:18 GMT /pklxkd-61/obj11 2023-05-31T21:03:18.772 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:W8rqJhCq3ybyvYhPqU5swcGLuaE= 2023-05-31T21:03:18.773 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 21:03:18 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:W8rqJhCq3ybyvYhPqU5swcGLuaE=', 'Content-Length': '0'} 2023-05-31T21:03:18.774 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:59:44 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d82e3a9225595415-006477b616-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 21:03:18 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:03:18.774 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj11 2023-05-31T21:03:18.775 DEBUG:boto:path=/pklxkd-61/obj11 2023-05-31T21:03:18.775 DEBUG:boto:auth_path=/pklxkd-61/obj11 2023-05-31T21:03:18.776 DEBUG:boto:Method: GET 2023-05-31T21:03:18.776 DEBUG:boto:Path: /pklxkd-61/obj11 2023-05-31T21:03:18.776 DEBUG:boto:Data: 2023-05-31T21:03:18.777 DEBUG:boto:Headers: {} 2023-05-31T21:03:18.777 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T21:03:18.777 DEBUG:boto:Port: 8001 2023-05-31T21:03:18.778 DEBUG:boto:Params: {} 2023-05-31T21:03:18.778 DEBUG:boto:Token: None 2023-05-31T21:03:18.779 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 21:03:18 GMT /pklxkd-61/obj11 2023-05-31T21:03:18.779 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:FzizjJ1iG4KMmdtOwPwUYNXYM7c= 2023-05-31T21:03:18.779 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 21:03:18 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:FzizjJ1iG4KMmdtOwPwUYNXYM7c=', 'Content-Length': '0'} 2023-05-31T21:03:18.781 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:59:44 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002a2e3ac659713f71-006477b616-1f4e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 21:03:18 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:03:18.782 DEBUG:boto:path=/pklxkd-61/obj11 2023-05-31T21:03:18.782 DEBUG:boto:auth_path=/pklxkd-61/obj11 2023-05-31T21:03:18.782 DEBUG:boto:Method: GET 2023-05-31T21:03:18.783 DEBUG:boto:Path: /pklxkd-61/obj11 2023-05-31T21:03:18.783 DEBUG:boto:Data: 2023-05-31T21:03:18.784 DEBUG:boto:Headers: {} 2023-05-31T21:03:18.784 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T21:03:18.784 DEBUG:boto:Port: 8000 2023-05-31T21:03:18.785 DEBUG:boto:Params: {} 2023-05-31T21:03:18.785 DEBUG:boto:Token: None 2023-05-31T21:03:18.785 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 21:03:18 GMT /pklxkd-61/obj11 2023-05-31T21:03:18.786 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:FzizjJ1iG4KMmdtOwPwUYNXYM7c= 2023-05-31T21:03:18.786 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 21:03:18 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:FzizjJ1iG4KMmdtOwPwUYNXYM7c=', 'Content-Length': '0'} 2023-05-31T21:03:18.788 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:59:44 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000682b63ab508d0385-006477b616-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 21:03:18 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:03:18.788 DEBUG:boto:path=/pklxkd-61/ 2023-05-31T21:03:18.789 DEBUG:boto:auth_path=/pklxkd-61/ 2023-05-31T21:03:18.789 DEBUG:boto:path=/pklxkd-61/?prefix=obj11 2023-05-31T21:03:18.790 DEBUG:boto:auth_path=/pklxkd-61/?prefix=obj11 2023-05-31T21:03:18.790 DEBUG:boto:Method: GET 2023-05-31T21:03:18.790 DEBUG:boto:Path: /pklxkd-61/?prefix=obj11 2023-05-31T21:03:18.791 DEBUG:boto:Data: 2023-05-31T21:03:18.791 DEBUG:boto:Headers: {} 2023-05-31T21:03:18.791 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T21:03:18.792 DEBUG:boto:Port: 8001 2023-05-31T21:03:18.792 DEBUG:boto:Params: {} 2023-05-31T21:03:18.792 DEBUG:boto:Token: None 2023-05-31T21:03:18.793 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 21:03:18 GMT /pklxkd-61/ 2023-05-31T21:03:18.793 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:qoQ+UVg4HBmCbZSqS0auFpBGtNM= 2023-05-31T21:03:18.794 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 21:03:18 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:qoQ+UVg4HBmCbZSqS0auFpBGtNM=', 'Content-Length': '0'} 2023-05-31T21:03:18.796 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f5a92988132a676e-006477b616-1f4e-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 21:03:18 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:03:18.797 DEBUG:boto:b'pklxkd-61obj111000falseobj112023-05-31T20:59:44.431Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj1112023-05-31T20:59:46.951Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj1152023-05-31T20:59:45.636Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj1162023-05-31T20:59:45.977Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T21:03:18.797 DEBUG:boto:path=/pklxkd-61/ 2023-05-31T21:03:18.798 DEBUG:boto:auth_path=/pklxkd-61/ 2023-05-31T21:03:18.798 DEBUG:boto:path=/pklxkd-61/?prefix=obj11 2023-05-31T21:03:18.798 DEBUG:boto:auth_path=/pklxkd-61/?prefix=obj11 2023-05-31T21:03:18.799 DEBUG:boto:Method: GET 2023-05-31T21:03:18.799 DEBUG:boto:Path: /pklxkd-61/?prefix=obj11 2023-05-31T21:03:18.800 DEBUG:boto:Data: 2023-05-31T21:03:18.800 DEBUG:boto:Headers: {} 2023-05-31T21:03:18.800 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T21:03:18.801 DEBUG:boto:Port: 8000 2023-05-31T21:03:18.801 DEBUG:boto:Params: {} 2023-05-31T21:03:18.801 DEBUG:boto:Token: None 2023-05-31T21:03:18.802 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 21:03:18 GMT /pklxkd-61/ 2023-05-31T21:03:18.802 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:qoQ+UVg4HBmCbZSqS0auFpBGtNM= 2023-05-31T21:03:18.803 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 21:03:18 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:qoQ+UVg4HBmCbZSqS0auFpBGtNM=', 'Content-Length': '0'} 2023-05-31T21:03:18.806 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000006ed4f229817af66c-006477b616-1630-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 21:03:18 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:03:18.806 DEBUG:boto:b'pklxkd-61obj111000falseobj112023-05-31T20:59:44.431Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj1112023-05-31T20:59:46.951Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj1152023-05-31T20:59:45.636Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj1162023-05-31T20:59:45.977Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T21:03:18.807 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj111 2023-05-31T21:03:18.808 DEBUG:boto:path=/pklxkd-61/obj111 2023-05-31T21:03:18.808 DEBUG:boto:auth_path=/pklxkd-61/obj111 2023-05-31T21:03:18.808 DEBUG:boto:path=/pklxkd-61/obj111?versionId=null 2023-05-31T21:03:18.809 DEBUG:boto:auth_path=/pklxkd-61/obj111?versionId=null 2023-05-31T21:03:18.809 DEBUG:boto:Method: GET 2023-05-31T21:03:18.809 DEBUG:boto:Path: /pklxkd-61/obj111?versionId=null 2023-05-31T21:03:18.810 DEBUG:boto:Data: 2023-05-31T21:03:18.810 DEBUG:boto:Headers: {} 2023-05-31T21:03:18.810 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T21:03:18.811 DEBUG:boto:Port: 8001 2023-05-31T21:03:18.811 DEBUG:boto:Params: {} 2023-05-31T21:03:18.811 DEBUG:boto:Token: None 2023-05-31T21:03:18.812 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 21:03:18 GMT /pklxkd-61/obj111?versionId=null 2023-05-31T21:03:18.812 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:+xqVFb5j1zcw85Z5oWodMNqyMQk= 2023-05-31T21:03:18.813 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 21:03:18 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:+xqVFb5j1zcw85Z5oWodMNqyMQk=', 'Content-Length': '0'} 2023-05-31T21:03:18.814 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:59:46 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e193aec247218398-006477b616-1f4e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 21:03:18 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:03:18.856 DEBUG:boto:path=/pklxkd-61/obj111 2023-05-31T21:03:18.857 DEBUG:boto:auth_path=/pklxkd-61/obj111 2023-05-31T21:03:18.857 DEBUG:boto:path=/pklxkd-61/obj111?versionId=null 2023-05-31T21:03:18.857 DEBUG:boto:auth_path=/pklxkd-61/obj111?versionId=null 2023-05-31T21:03:18.858 DEBUG:boto:Method: GET 2023-05-31T21:03:18.858 DEBUG:boto:Path: /pklxkd-61/obj111?versionId=null 2023-05-31T21:03:18.858 DEBUG:boto:Data: 2023-05-31T21:03:18.859 DEBUG:boto:Headers: {} 2023-05-31T21:03:18.859 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T21:03:18.859 DEBUG:boto:Port: 8000 2023-05-31T21:03:18.860 DEBUG:boto:Params: {} 2023-05-31T21:03:18.860 DEBUG:boto:Token: None 2023-05-31T21:03:18.861 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 21:03:18 GMT /pklxkd-61/obj111?versionId=null 2023-05-31T21:03:18.861 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:+xqVFb5j1zcw85Z5oWodMNqyMQk= 2023-05-31T21:03:18.862 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 21:03:18 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:+xqVFb5j1zcw85Z5oWodMNqyMQk=', 'Content-Length': '0'} 2023-05-31T21:03:18.863 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:59:46 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002e7e5a7298cf6f95-006477b616-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 21:03:18 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:03:18.864 DEBUG:boto:path=/pklxkd-61/obj111 2023-05-31T21:03:18.864 DEBUG:boto:auth_path=/pklxkd-61/obj111 2023-05-31T21:03:18.865 DEBUG:boto:path=/pklxkd-61/obj111?versionId=null 2023-05-31T21:03:18.865 DEBUG:boto:auth_path=/pklxkd-61/obj111?versionId=null 2023-05-31T21:03:18.866 DEBUG:boto:Method: HEAD 2023-05-31T21:03:18.866 DEBUG:boto:Path: /pklxkd-61/obj111?versionId=null 2023-05-31T21:03:18.866 DEBUG:boto:Data: 2023-05-31T21:03:18.867 DEBUG:boto:Headers: {} 2023-05-31T21:03:18.867 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T21:03:18.867 DEBUG:boto:Port: 8001 2023-05-31T21:03:18.867 DEBUG:boto:Params: {} 2023-05-31T21:03:18.868 DEBUG:boto:Token: None 2023-05-31T21:03:18.868 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 21:03:18 GMT /pklxkd-61/obj111?versionId=null 2023-05-31T21:03:18.869 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:aB1+vpIyOpyN+C9TCMze+frHer4= 2023-05-31T21:03:18.869 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 21:03:18 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:aB1+vpIyOpyN+C9TCMze+frHer4=', 'Content-Length': '0'} 2023-05-31T21:03:18.871 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:59:46 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000093112a544ba1dc52-006477b616-1f4e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 21:03:18 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:03:18.871 DEBUG:boto:path=/pklxkd-61/obj111 2023-05-31T21:03:18.872 DEBUG:boto:auth_path=/pklxkd-61/obj111 2023-05-31T21:03:18.872 DEBUG:boto:path=/pklxkd-61/obj111?versionId=null 2023-05-31T21:03:18.872 DEBUG:boto:auth_path=/pklxkd-61/obj111?versionId=null 2023-05-31T21:03:18.873 DEBUG:boto:Method: HEAD 2023-05-31T21:03:18.873 DEBUG:boto:Path: /pklxkd-61/obj111?versionId=null 2023-05-31T21:03:18.873 DEBUG:boto:Data: 2023-05-31T21:03:18.874 DEBUG:boto:Headers: {} 2023-05-31T21:03:18.874 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T21:03:18.874 DEBUG:boto:Port: 8000 2023-05-31T21:03:18.875 DEBUG:boto:Params: {} 2023-05-31T21:03:18.875 DEBUG:boto:Token: None 2023-05-31T21:03:18.875 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 21:03:18 GMT /pklxkd-61/obj111?versionId=null 2023-05-31T21:03:18.876 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:aB1+vpIyOpyN+C9TCMze+frHer4= 2023-05-31T21:03:18.876 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 21:03:18 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:aB1+vpIyOpyN+C9TCMze+frHer4=', 'Content-Length': '0'} 2023-05-31T21:03:18.878 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:59:46 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003ba409362e224dba-006477b616-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 21:03:18 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:03:18.878 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj111 2023-05-31T21:03:18.879 DEBUG:boto:path=/pklxkd-61/obj111 2023-05-31T21:03:18.879 DEBUG:boto:auth_path=/pklxkd-61/obj111 2023-05-31T21:03:18.880 DEBUG:boto:path=/pklxkd-61/obj111?versionId=null 2023-05-31T21:03:18.880 DEBUG:boto:auth_path=/pklxkd-61/obj111?versionId=null 2023-05-31T21:03:18.880 DEBUG:boto:Method: GET 2023-05-31T21:03:18.881 DEBUG:boto:Path: /pklxkd-61/obj111?versionId=null 2023-05-31T21:03:18.881 DEBUG:boto:Data: 2023-05-31T21:03:18.881 DEBUG:boto:Headers: {} 2023-05-31T21:03:18.882 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T21:03:18.882 DEBUG:boto:Port: 8001 2023-05-31T21:03:18.882 DEBUG:boto:Params: {} 2023-05-31T21:03:18.883 DEBUG:boto:Token: None 2023-05-31T21:03:18.883 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 21:03:18 GMT /pklxkd-61/obj111?versionId=null 2023-05-31T21:03:18.884 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:+xqVFb5j1zcw85Z5oWodMNqyMQk= 2023-05-31T21:03:18.884 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 21:03:18 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:+xqVFb5j1zcw85Z5oWodMNqyMQk=', 'Content-Length': '0'} 2023-05-31T21:03:18.886 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:59:46 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000eaf1926c07c55a58-006477b616-1f4e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 21:03:18 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:03:18.886 DEBUG:boto:path=/pklxkd-61/obj111 2023-05-31T21:03:18.887 DEBUG:boto:auth_path=/pklxkd-61/obj111 2023-05-31T21:03:18.887 DEBUG:boto:path=/pklxkd-61/obj111?versionId=null 2023-05-31T21:03:18.888 DEBUG:boto:auth_path=/pklxkd-61/obj111?versionId=null 2023-05-31T21:03:18.888 DEBUG:boto:Method: GET 2023-05-31T21:03:18.888 DEBUG:boto:Path: /pklxkd-61/obj111?versionId=null 2023-05-31T21:03:18.889 DEBUG:boto:Data: 2023-05-31T21:03:18.889 DEBUG:boto:Headers: {} 2023-05-31T21:03:18.890 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T21:03:18.890 DEBUG:boto:Port: 8000 2023-05-31T21:03:18.890 DEBUG:boto:Params: {} 2023-05-31T21:03:18.891 DEBUG:boto:Token: None 2023-05-31T21:03:18.891 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 21:03:18 GMT /pklxkd-61/obj111?versionId=null 2023-05-31T21:03:18.891 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:+xqVFb5j1zcw85Z5oWodMNqyMQk= 2023-05-31T21:03:18.892 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 21:03:18 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:+xqVFb5j1zcw85Z5oWodMNqyMQk=', 'Content-Length': '0'} 2023-05-31T21:03:18.894 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:59:46 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000def2512b71982e3f-006477b616-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 21:03:18 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:03:18.894 DEBUG:boto:path=/pklxkd-61/ 2023-05-31T21:03:18.895 DEBUG:boto:auth_path=/pklxkd-61/ 2023-05-31T21:03:18.895 DEBUG:boto:path=/pklxkd-61/?prefix=obj111 2023-05-31T21:03:18.895 DEBUG:boto:auth_path=/pklxkd-61/?prefix=obj111 2023-05-31T21:03:18.896 DEBUG:boto:Method: GET 2023-05-31T21:03:18.896 DEBUG:boto:Path: /pklxkd-61/?prefix=obj111 2023-05-31T21:03:18.896 DEBUG:boto:Data: 2023-05-31T21:03:18.897 DEBUG:boto:Headers: {} 2023-05-31T21:03:18.897 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T21:03:18.897 DEBUG:boto:Port: 8001 2023-05-31T21:03:18.898 DEBUG:boto:Params: {} 2023-05-31T21:03:18.898 DEBUG:boto:Token: None 2023-05-31T21:03:18.899 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 21:03:18 GMT /pklxkd-61/ 2023-05-31T21:03:18.899 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:qoQ+UVg4HBmCbZSqS0auFpBGtNM= 2023-05-31T21:03:18.899 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 21:03:18 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:qoQ+UVg4HBmCbZSqS0auFpBGtNM=', 'Content-Length': '0'} 2023-05-31T21:03:18.901 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000000ddedeac0e5eb198-006477b616-1f4e-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 21:03:18 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:03:18.902 DEBUG:boto:b'pklxkd-61obj1111000falseobj1112023-05-31T20:59:46.951Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T21:03:18.903 DEBUG:boto:path=/pklxkd-61/ 2023-05-31T21:03:18.903 DEBUG:boto:auth_path=/pklxkd-61/ 2023-05-31T21:03:18.903 DEBUG:boto:path=/pklxkd-61/?prefix=obj111 2023-05-31T21:03:18.904 DEBUG:boto:auth_path=/pklxkd-61/?prefix=obj111 2023-05-31T21:03:18.904 DEBUG:boto:Method: GET 2023-05-31T21:03:18.904 DEBUG:boto:Path: /pklxkd-61/?prefix=obj111 2023-05-31T21:03:18.905 DEBUG:boto:Data: 2023-05-31T21:03:18.905 DEBUG:boto:Headers: {} 2023-05-31T21:03:18.905 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T21:03:18.906 DEBUG:boto:Port: 8000 2023-05-31T21:03:18.906 DEBUG:boto:Params: {} 2023-05-31T21:03:18.907 DEBUG:boto:Token: None 2023-05-31T21:03:18.907 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 21:03:18 GMT /pklxkd-61/ 2023-05-31T21:03:18.907 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:qoQ+UVg4HBmCbZSqS0auFpBGtNM= 2023-05-31T21:03:18.908 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 21:03:18 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:qoQ+UVg4HBmCbZSqS0auFpBGtNM=', 'Content-Length': '0'} 2023-05-31T21:03:18.911 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000006c631d9e8cd062bc-006477b616-1630-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 21:03:18 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:03:18.912 DEBUG:boto:b'pklxkd-61obj1111000falseobj1112023-05-31T20:59:46.951Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T21:03:18.912 DEBUG:boto:path=/pklxkd-61/obj111 2023-05-31T21:03:18.913 DEBUG:boto:auth_path=/pklxkd-61/obj111 2023-05-31T21:03:18.913 DEBUG:boto:Method: HEAD 2023-05-31T21:03:18.913 DEBUG:boto:Path: /pklxkd-61/obj111 2023-05-31T21:03:18.914 DEBUG:boto:Data: 2023-05-31T21:03:18.914 DEBUG:boto:Headers: {} 2023-05-31T21:03:18.914 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T21:03:18.915 DEBUG:boto:Port: 8001 2023-05-31T21:03:18.915 DEBUG:boto:Params: {} 2023-05-31T21:03:18.916 DEBUG:boto:Token: None 2023-05-31T21:03:18.916 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 21:03:18 GMT /pklxkd-61/obj111 2023-05-31T21:03:18.916 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:S+VJj+QkV7gwd7IwVA6ZBGuuVVM= 2023-05-31T21:03:18.917 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 21:03:18 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:S+VJj+QkV7gwd7IwVA6ZBGuuVVM=', 'Content-Length': '0'} 2023-05-31T21:03:18.918 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:59:46 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000021566573d8e41360-006477b616-1f4e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 21:03:18 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:03:18.919 DEBUG:boto:path=/pklxkd-61/obj111 2023-05-31T21:03:18.919 DEBUG:boto:auth_path=/pklxkd-61/obj111 2023-05-31T21:03:18.920 DEBUG:boto:Method: HEAD 2023-05-31T21:03:18.920 DEBUG:boto:Path: /pklxkd-61/obj111 2023-05-31T21:03:18.920 DEBUG:boto:Data: 2023-05-31T21:03:18.921 DEBUG:boto:Headers: {} 2023-05-31T21:03:18.921 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T21:03:18.921 DEBUG:boto:Port: 8000 2023-05-31T21:03:18.922 DEBUG:boto:Params: {} 2023-05-31T21:03:18.922 DEBUG:boto:Token: None 2023-05-31T21:03:18.923 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 21:03:18 GMT /pklxkd-61/obj111 2023-05-31T21:03:18.923 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:S+VJj+QkV7gwd7IwVA6ZBGuuVVM= 2023-05-31T21:03:18.923 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 21:03:18 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:S+VJj+QkV7gwd7IwVA6ZBGuuVVM=', 'Content-Length': '0'} 2023-05-31T21:03:18.925 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:59:46 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a8d189a5860434b1-006477b616-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 21:03:18 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:03:18.926 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj111 2023-05-31T21:03:18.926 DEBUG:boto:path=/pklxkd-61/obj111 2023-05-31T21:03:18.926 DEBUG:boto:auth_path=/pklxkd-61/obj111 2023-05-31T21:03:18.927 DEBUG:boto:Method: GET 2023-05-31T21:03:18.927 DEBUG:boto:Path: /pklxkd-61/obj111 2023-05-31T21:03:18.927 DEBUG:boto:Data: 2023-05-31T21:03:18.928 DEBUG:boto:Headers: {} 2023-05-31T21:03:18.928 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T21:03:18.928 DEBUG:boto:Port: 8001 2023-05-31T21:03:18.929 DEBUG:boto:Params: {} 2023-05-31T21:03:18.929 DEBUG:boto:Token: None 2023-05-31T21:03:18.930 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 21:03:18 GMT /pklxkd-61/obj111 2023-05-31T21:03:18.930 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:vr5RxNeDdt4AtXX80H3D0Bf6au8= 2023-05-31T21:03:18.930 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 21:03:18 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:vr5RxNeDdt4AtXX80H3D0Bf6au8=', 'Content-Length': '0'} 2023-05-31T21:03:18.932 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:59:46 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009b4b39c2624fa48e-006477b616-1f4e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 21:03:18 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:03:18.933 DEBUG:boto:path=/pklxkd-61/obj111 2023-05-31T21:03:18.933 DEBUG:boto:auth_path=/pklxkd-61/obj111 2023-05-31T21:03:18.933 DEBUG:boto:Method: GET 2023-05-31T21:03:18.934 DEBUG:boto:Path: /pklxkd-61/obj111 2023-05-31T21:03:18.934 DEBUG:boto:Data: 2023-05-31T21:03:18.935 DEBUG:boto:Headers: {} 2023-05-31T21:03:18.935 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T21:03:18.935 DEBUG:boto:Port: 8000 2023-05-31T21:03:18.936 DEBUG:boto:Params: {} 2023-05-31T21:03:18.936 DEBUG:boto:Token: None 2023-05-31T21:03:18.937 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 21:03:18 GMT /pklxkd-61/obj111 2023-05-31T21:03:18.937 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:vr5RxNeDdt4AtXX80H3D0Bf6au8= 2023-05-31T21:03:18.937 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 21:03:18 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:vr5RxNeDdt4AtXX80H3D0Bf6au8=', 'Content-Length': '0'} 2023-05-31T21:03:18.939 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:59:46 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001e8397f728cdf4fe-006477b616-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 21:03:18 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:03:18.940 DEBUG:boto:path=/pklxkd-61/ 2023-05-31T21:03:18.940 DEBUG:boto:auth_path=/pklxkd-61/ 2023-05-31T21:03:18.941 DEBUG:boto:path=/pklxkd-61/?prefix=obj111 2023-05-31T21:03:18.941 DEBUG:boto:auth_path=/pklxkd-61/?prefix=obj111 2023-05-31T21:03:18.941 DEBUG:boto:Method: GET 2023-05-31T21:03:18.942 DEBUG:boto:Path: /pklxkd-61/?prefix=obj111 2023-05-31T21:03:18.942 DEBUG:boto:Data: 2023-05-31T21:03:18.942 DEBUG:boto:Headers: {} 2023-05-31T21:03:18.943 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T21:03:18.943 DEBUG:boto:Port: 8001 2023-05-31T21:03:18.943 DEBUG:boto:Params: {} 2023-05-31T21:03:18.944 DEBUG:boto:Token: None 2023-05-31T21:03:18.944 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 21:03:18 GMT /pklxkd-61/ 2023-05-31T21:03:18.945 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:qoQ+UVg4HBmCbZSqS0auFpBGtNM= 2023-05-31T21:03:18.945 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 21:03:18 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:qoQ+UVg4HBmCbZSqS0auFpBGtNM=', 'Content-Length': '0'} 2023-05-31T21:03:18.947 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000cb423dab26b7b568-006477b616-1f4e-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 21:03:18 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:03:18.948 DEBUG:boto:b'pklxkd-61obj1111000falseobj1112023-05-31T20:59:46.951Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T21:03:18.949 DEBUG:boto:path=/pklxkd-61/ 2023-05-31T21:03:18.949 DEBUG:boto:auth_path=/pklxkd-61/ 2023-05-31T21:03:18.949 DEBUG:boto:path=/pklxkd-61/?prefix=obj111 2023-05-31T21:03:18.950 DEBUG:boto:auth_path=/pklxkd-61/?prefix=obj111 2023-05-31T21:03:18.950 DEBUG:boto:Method: GET 2023-05-31T21:03:18.950 DEBUG:boto:Path: /pklxkd-61/?prefix=obj111 2023-05-31T21:03:18.951 DEBUG:boto:Data: 2023-05-31T21:03:18.951 DEBUG:boto:Headers: {} 2023-05-31T21:03:18.952 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T21:03:18.952 DEBUG:boto:Port: 8000 2023-05-31T21:03:18.952 DEBUG:boto:Params: {} 2023-05-31T21:03:18.953 DEBUG:boto:Token: None 2023-05-31T21:03:18.953 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 21:03:18 GMT /pklxkd-61/ 2023-05-31T21:03:18.954 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:qoQ+UVg4HBmCbZSqS0auFpBGtNM= 2023-05-31T21:03:18.954 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 21:03:18 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:qoQ+UVg4HBmCbZSqS0auFpBGtNM=', 'Content-Length': '0'} 2023-05-31T21:03:18.958 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f62ee075a66cd11e-006477b616-1630-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 21:03:18 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:03:18.959 DEBUG:boto:b'pklxkd-61obj1111000falseobj1112023-05-31T20:59:46.951Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T21:03:18.959 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj115 2023-05-31T21:03:18.960 DEBUG:boto:path=/pklxkd-61/obj115 2023-05-31T21:03:18.960 DEBUG:boto:auth_path=/pklxkd-61/obj115 2023-05-31T21:03:18.960 DEBUG:boto:path=/pklxkd-61/obj115?versionId=null 2023-05-31T21:03:18.961 DEBUG:boto:auth_path=/pklxkd-61/obj115?versionId=null 2023-05-31T21:03:18.961 DEBUG:boto:Method: GET 2023-05-31T21:03:18.962 DEBUG:boto:Path: /pklxkd-61/obj115?versionId=null 2023-05-31T21:03:18.962 DEBUG:boto:Data: 2023-05-31T21:03:18.962 DEBUG:boto:Headers: {} 2023-05-31T21:03:18.963 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T21:03:18.963 DEBUG:boto:Port: 8001 2023-05-31T21:03:18.963 DEBUG:boto:Params: {} 2023-05-31T21:03:18.964 DEBUG:boto:Token: None 2023-05-31T21:03:18.964 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 21:03:18 GMT /pklxkd-61/obj115?versionId=null 2023-05-31T21:03:18.965 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:C+hA0JtyW0zJmZ9XnY4dfYSC3Mc= 2023-05-31T21:03:18.965 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 21:03:18 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:C+hA0JtyW0zJmZ9XnY4dfYSC3Mc=', 'Content-Length': '0'} 2023-05-31T21:03:18.967 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:59:45 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ee2a0c4fdc78458e-006477b616-1f4e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 21:03:18 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:03:18.967 DEBUG:boto:path=/pklxkd-61/obj115 2023-05-31T21:03:18.968 DEBUG:boto:auth_path=/pklxkd-61/obj115 2023-05-31T21:03:18.968 DEBUG:boto:path=/pklxkd-61/obj115?versionId=null 2023-05-31T21:03:18.969 DEBUG:boto:auth_path=/pklxkd-61/obj115?versionId=null 2023-05-31T21:03:18.969 DEBUG:boto:Method: GET 2023-05-31T21:03:18.969 DEBUG:boto:Path: /pklxkd-61/obj115?versionId=null 2023-05-31T21:03:18.970 DEBUG:boto:Data: 2023-05-31T21:03:18.970 DEBUG:boto:Headers: {} 2023-05-31T21:03:18.970 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T21:03:18.971 DEBUG:boto:Port: 8000 2023-05-31T21:03:18.971 DEBUG:boto:Params: {} 2023-05-31T21:03:18.971 DEBUG:boto:Token: None 2023-05-31T21:03:18.972 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 21:03:18 GMT /pklxkd-61/obj115?versionId=null 2023-05-31T21:03:18.972 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:C+hA0JtyW0zJmZ9XnY4dfYSC3Mc= 2023-05-31T21:03:18.973 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 21:03:18 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:C+hA0JtyW0zJmZ9XnY4dfYSC3Mc=', 'Content-Length': '0'} 2023-05-31T21:03:18.974 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:59:45 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d6eeda2fdab50ed4-006477b616-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 21:03:18 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:03:19.016 DEBUG:boto:path=/pklxkd-61/obj115 2023-05-31T21:03:19.017 DEBUG:boto:auth_path=/pklxkd-61/obj115 2023-05-31T21:03:19.017 DEBUG:boto:path=/pklxkd-61/obj115?versionId=null 2023-05-31T21:03:19.017 DEBUG:boto:auth_path=/pklxkd-61/obj115?versionId=null 2023-05-31T21:03:19.018 DEBUG:boto:Method: HEAD 2023-05-31T21:03:19.018 DEBUG:boto:Path: /pklxkd-61/obj115?versionId=null 2023-05-31T21:03:19.018 DEBUG:boto:Data: 2023-05-31T21:03:19.019 DEBUG:boto:Headers: {} 2023-05-31T21:03:19.019 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T21:03:19.019 DEBUG:boto:Port: 8001 2023-05-31T21:03:19.020 DEBUG:boto:Params: {} 2023-05-31T21:03:19.020 DEBUG:boto:Token: None 2023-05-31T21:03:19.021 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 21:03:19 GMT /pklxkd-61/obj115?versionId=null 2023-05-31T21:03:19.021 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:KBgT7PwaqXJBVwfm4vp/HoNRsfU= 2023-05-31T21:03:19.021 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 21:03:19 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:KBgT7PwaqXJBVwfm4vp/HoNRsfU=', 'Content-Length': '0'} 2023-05-31T21:03:19.023 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:59:45 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005ca79247084c0328-006477b617-1f4e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 21:03:19 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:03:19.024 DEBUG:boto:path=/pklxkd-61/obj115 2023-05-31T21:03:19.024 DEBUG:boto:auth_path=/pklxkd-61/obj115 2023-05-31T21:03:19.024 DEBUG:boto:path=/pklxkd-61/obj115?versionId=null 2023-05-31T21:03:19.025 DEBUG:boto:auth_path=/pklxkd-61/obj115?versionId=null 2023-05-31T21:03:19.025 DEBUG:boto:Method: HEAD 2023-05-31T21:03:19.025 DEBUG:boto:Path: /pklxkd-61/obj115?versionId=null 2023-05-31T21:03:19.026 DEBUG:boto:Data: 2023-05-31T21:03:19.026 DEBUG:boto:Headers: {} 2023-05-31T21:03:19.026 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T21:03:19.027 DEBUG:boto:Port: 8000 2023-05-31T21:03:19.027 DEBUG:boto:Params: {} 2023-05-31T21:03:19.027 DEBUG:boto:Token: None 2023-05-31T21:03:19.028 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 21:03:19 GMT /pklxkd-61/obj115?versionId=null 2023-05-31T21:03:19.028 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:KBgT7PwaqXJBVwfm4vp/HoNRsfU= 2023-05-31T21:03:19.029 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 21:03:19 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:KBgT7PwaqXJBVwfm4vp/HoNRsfU=', 'Content-Length': '0'} 2023-05-31T21:03:19.030 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:59:45 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d2950cb3a498eee0-006477b617-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 21:03:19 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:03:19.031 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj115 2023-05-31T21:03:19.031 DEBUG:boto:path=/pklxkd-61/obj115 2023-05-31T21:03:19.032 DEBUG:boto:auth_path=/pklxkd-61/obj115 2023-05-31T21:03:19.032 DEBUG:boto:path=/pklxkd-61/obj115?versionId=null 2023-05-31T21:03:19.032 DEBUG:boto:auth_path=/pklxkd-61/obj115?versionId=null 2023-05-31T21:03:19.033 DEBUG:boto:Method: GET 2023-05-31T21:03:19.033 DEBUG:boto:Path: /pklxkd-61/obj115?versionId=null 2023-05-31T21:03:19.033 DEBUG:boto:Data: 2023-05-31T21:03:19.034 DEBUG:boto:Headers: {} 2023-05-31T21:03:19.034 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T21:03:19.035 DEBUG:boto:Port: 8001 2023-05-31T21:03:19.035 DEBUG:boto:Params: {} 2023-05-31T21:03:19.035 DEBUG:boto:Token: None 2023-05-31T21:03:19.036 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 21:03:19 GMT /pklxkd-61/obj115?versionId=null 2023-05-31T21:03:19.036 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:zcyNFAKKaXfTocUs6X850grOLZc= 2023-05-31T21:03:19.037 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 21:03:19 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:zcyNFAKKaXfTocUs6X850grOLZc=', 'Content-Length': '0'} 2023-05-31T21:03:19.038 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:59:45 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000bc23488b705528fe-006477b617-1f4e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 21:03:19 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:03:19.039 DEBUG:boto:path=/pklxkd-61/obj115 2023-05-31T21:03:19.040 DEBUG:boto:auth_path=/pklxkd-61/obj115 2023-05-31T21:03:19.040 DEBUG:boto:path=/pklxkd-61/obj115?versionId=null 2023-05-31T21:03:19.040 DEBUG:boto:auth_path=/pklxkd-61/obj115?versionId=null 2023-05-31T21:03:19.041 DEBUG:boto:Method: GET 2023-05-31T21:03:19.041 DEBUG:boto:Path: /pklxkd-61/obj115?versionId=null 2023-05-31T21:03:19.041 DEBUG:boto:Data: 2023-05-31T21:03:19.042 DEBUG:boto:Headers: {} 2023-05-31T21:03:19.042 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T21:03:19.043 DEBUG:boto:Port: 8000 2023-05-31T21:03:19.043 DEBUG:boto:Params: {} 2023-05-31T21:03:19.043 DEBUG:boto:Token: None 2023-05-31T21:03:19.044 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 21:03:19 GMT /pklxkd-61/obj115?versionId=null 2023-05-31T21:03:19.044 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:zcyNFAKKaXfTocUs6X850grOLZc= 2023-05-31T21:03:19.045 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 21:03:19 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:zcyNFAKKaXfTocUs6X850grOLZc=', 'Content-Length': '0'} 2023-05-31T21:03:19.046 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:59:45 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d87e7a55cd2ccd6a-006477b617-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 21:03:19 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:03:19.047 DEBUG:boto:path=/pklxkd-61/ 2023-05-31T21:03:19.047 DEBUG:boto:auth_path=/pklxkd-61/ 2023-05-31T21:03:19.048 DEBUG:boto:path=/pklxkd-61/?prefix=obj115 2023-05-31T21:03:19.048 DEBUG:boto:auth_path=/pklxkd-61/?prefix=obj115 2023-05-31T21:03:19.049 DEBUG:boto:Method: GET 2023-05-31T21:03:19.049 DEBUG:boto:Path: /pklxkd-61/?prefix=obj115 2023-05-31T21:03:19.049 DEBUG:boto:Data: 2023-05-31T21:03:19.050 DEBUG:boto:Headers: {} 2023-05-31T21:03:19.050 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T21:03:19.050 DEBUG:boto:Port: 8001 2023-05-31T21:03:19.051 DEBUG:boto:Params: {} 2023-05-31T21:03:19.051 DEBUG:boto:Token: None 2023-05-31T21:03:19.052 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 21:03:19 GMT /pklxkd-61/ 2023-05-31T21:03:19.052 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:hLB/3UScfc36qJjImbkC6+I98Sc= 2023-05-31T21:03:19.053 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 21:03:19 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:hLB/3UScfc36qJjImbkC6+I98Sc=', 'Content-Length': '0'} 2023-05-31T21:03:19.055 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000cdf2505b872d70af-006477b617-1f4e-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 21:03:19 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:03:19.055 DEBUG:boto:b'pklxkd-61obj1151000falseobj1152023-05-31T20:59:45.636Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T21:03:19.056 DEBUG:boto:path=/pklxkd-61/ 2023-05-31T21:03:19.056 DEBUG:boto:auth_path=/pklxkd-61/ 2023-05-31T21:03:19.057 DEBUG:boto:path=/pklxkd-61/?prefix=obj115 2023-05-31T21:03:19.057 DEBUG:boto:auth_path=/pklxkd-61/?prefix=obj115 2023-05-31T21:03:19.057 DEBUG:boto:Method: GET 2023-05-31T21:03:19.058 DEBUG:boto:Path: /pklxkd-61/?prefix=obj115 2023-05-31T21:03:19.058 DEBUG:boto:Data: 2023-05-31T21:03:19.058 DEBUG:boto:Headers: {} 2023-05-31T21:03:19.059 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T21:03:19.059 DEBUG:boto:Port: 8000 2023-05-31T21:03:19.059 DEBUG:boto:Params: {} 2023-05-31T21:03:19.060 DEBUG:boto:Token: None 2023-05-31T21:03:19.060 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 21:03:19 GMT /pklxkd-61/ 2023-05-31T21:03:19.061 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:hLB/3UScfc36qJjImbkC6+I98Sc= 2023-05-31T21:03:19.061 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 21:03:19 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:hLB/3UScfc36qJjImbkC6+I98Sc=', 'Content-Length': '0'} 2023-05-31T21:03:19.065 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000004cdfe447ca89e89d-006477b617-1630-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 21:03:19 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:03:19.065 DEBUG:boto:b'pklxkd-61obj1151000falseobj1152023-05-31T20:59:45.636Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T21:03:19.066 DEBUG:boto:path=/pklxkd-61/obj115 2023-05-31T21:03:19.066 DEBUG:boto:auth_path=/pklxkd-61/obj115 2023-05-31T21:03:19.067 DEBUG:boto:Method: HEAD 2023-05-31T21:03:19.067 DEBUG:boto:Path: /pklxkd-61/obj115 2023-05-31T21:03:19.067 DEBUG:boto:Data: 2023-05-31T21:03:19.068 DEBUG:boto:Headers: {} 2023-05-31T21:03:19.068 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T21:03:19.068 DEBUG:boto:Port: 8001 2023-05-31T21:03:19.069 DEBUG:boto:Params: {} 2023-05-31T21:03:19.069 DEBUG:boto:Token: None 2023-05-31T21:03:19.070 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 21:03:19 GMT /pklxkd-61/obj115 2023-05-31T21:03:19.070 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:8V9jR0I7wR2Fa5fUYwJm1Rpy0Ng= 2023-05-31T21:03:19.071 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 21:03:19 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:8V9jR0I7wR2Fa5fUYwJm1Rpy0Ng=', 'Content-Length': '0'} 2023-05-31T21:03:19.072 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:59:45 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000858c529568d47298-006477b617-1f4e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 21:03:19 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:03:19.073 DEBUG:boto:path=/pklxkd-61/obj115 2023-05-31T21:03:19.073 DEBUG:boto:auth_path=/pklxkd-61/obj115 2023-05-31T21:03:19.073 DEBUG:boto:Method: HEAD 2023-05-31T21:03:19.074 DEBUG:boto:Path: /pklxkd-61/obj115 2023-05-31T21:03:19.074 DEBUG:boto:Data: 2023-05-31T21:03:19.074 DEBUG:boto:Headers: {} 2023-05-31T21:03:19.075 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T21:03:19.075 DEBUG:boto:Port: 8000 2023-05-31T21:03:19.075 DEBUG:boto:Params: {} 2023-05-31T21:03:19.076 DEBUG:boto:Token: None 2023-05-31T21:03:19.076 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 21:03:19 GMT /pklxkd-61/obj115 2023-05-31T21:03:19.077 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:8V9jR0I7wR2Fa5fUYwJm1Rpy0Ng= 2023-05-31T21:03:19.077 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 21:03:19 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:8V9jR0I7wR2Fa5fUYwJm1Rpy0Ng=', 'Content-Length': '0'} 2023-05-31T21:03:19.079 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:59:45 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d472a1dd11fa4b49-006477b617-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 21:03:19 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:03:19.080 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj115 2023-05-31T21:03:19.080 DEBUG:boto:path=/pklxkd-61/obj115 2023-05-31T21:03:19.081 DEBUG:boto:auth_path=/pklxkd-61/obj115 2023-05-31T21:03:19.081 DEBUG:boto:Method: GET 2023-05-31T21:03:19.081 DEBUG:boto:Path: /pklxkd-61/obj115 2023-05-31T21:03:19.082 DEBUG:boto:Data: 2023-05-31T21:03:19.082 DEBUG:boto:Headers: {} 2023-05-31T21:03:19.083 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T21:03:19.083 DEBUG:boto:Port: 8001 2023-05-31T21:03:19.083 DEBUG:boto:Params: {} 2023-05-31T21:03:19.084 DEBUG:boto:Token: None 2023-05-31T21:03:19.084 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 21:03:19 GMT /pklxkd-61/obj115 2023-05-31T21:03:19.085 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:y8qLsbarHndK0oFBzSFbKR6M3AM= 2023-05-31T21:03:19.085 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 21:03:19 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:y8qLsbarHndK0oFBzSFbKR6M3AM=', 'Content-Length': '0'} 2023-05-31T21:03:19.087 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:59:45 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000032d1e66719940632-006477b617-1f4e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 21:03:19 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:03:19.128 DEBUG:boto:path=/pklxkd-61/obj115 2023-05-31T21:03:19.129 DEBUG:boto:auth_path=/pklxkd-61/obj115 2023-05-31T21:03:19.129 DEBUG:boto:Method: GET 2023-05-31T21:03:19.130 DEBUG:boto:Path: /pklxkd-61/obj115 2023-05-31T21:03:19.130 DEBUG:boto:Data: 2023-05-31T21:03:19.131 DEBUG:boto:Headers: {} 2023-05-31T21:03:19.131 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T21:03:19.132 DEBUG:boto:Port: 8000 2023-05-31T21:03:19.132 DEBUG:boto:Params: {} 2023-05-31T21:03:19.133 DEBUG:boto:Token: None 2023-05-31T21:03:19.133 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 21:03:19 GMT /pklxkd-61/obj115 2023-05-31T21:03:19.134 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:y8qLsbarHndK0oFBzSFbKR6M3AM= 2023-05-31T21:03:19.134 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 21:03:19 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:y8qLsbarHndK0oFBzSFbKR6M3AM=', 'Content-Length': '0'} 2023-05-31T21:03:19.136 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:59:45 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006bbca964af391b4d-006477b617-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 21:03:19 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:03:19.136 DEBUG:boto:path=/pklxkd-61/ 2023-05-31T21:03:19.137 DEBUG:boto:auth_path=/pklxkd-61/ 2023-05-31T21:03:19.137 DEBUG:boto:path=/pklxkd-61/?prefix=obj115 2023-05-31T21:03:19.138 DEBUG:boto:auth_path=/pklxkd-61/?prefix=obj115 2023-05-31T21:03:19.138 DEBUG:boto:Method: GET 2023-05-31T21:03:19.138 DEBUG:boto:Path: /pklxkd-61/?prefix=obj115 2023-05-31T21:03:19.139 DEBUG:boto:Data: 2023-05-31T21:03:19.139 DEBUG:boto:Headers: {} 2023-05-31T21:03:19.139 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T21:03:19.140 DEBUG:boto:Port: 8001 2023-05-31T21:03:19.140 DEBUG:boto:Params: {} 2023-05-31T21:03:19.141 DEBUG:boto:Token: None 2023-05-31T21:03:19.141 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 21:03:19 GMT /pklxkd-61/ 2023-05-31T21:03:19.141 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:hLB/3UScfc36qJjImbkC6+I98Sc= 2023-05-31T21:03:19.142 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 21:03:19 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:hLB/3UScfc36qJjImbkC6+I98Sc=', 'Content-Length': '0'} 2023-05-31T21:03:19.144 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000fe864665c8da62b2-006477b617-1f4e-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 21:03:19 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:03:19.145 DEBUG:boto:b'pklxkd-61obj1151000falseobj1152023-05-31T20:59:45.636Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T21:03:19.145 DEBUG:boto:path=/pklxkd-61/ 2023-05-31T21:03:19.146 DEBUG:boto:auth_path=/pklxkd-61/ 2023-05-31T21:03:19.146 DEBUG:boto:path=/pklxkd-61/?prefix=obj115 2023-05-31T21:03:19.146 DEBUG:boto:auth_path=/pklxkd-61/?prefix=obj115 2023-05-31T21:03:19.147 DEBUG:boto:Method: GET 2023-05-31T21:03:19.147 DEBUG:boto:Path: /pklxkd-61/?prefix=obj115 2023-05-31T21:03:19.147 DEBUG:boto:Data: 2023-05-31T21:03:19.148 DEBUG:boto:Headers: {} 2023-05-31T21:03:19.148 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T21:03:19.149 DEBUG:boto:Port: 8000 2023-05-31T21:03:19.149 DEBUG:boto:Params: {} 2023-05-31T21:03:19.149 DEBUG:boto:Token: None 2023-05-31T21:03:19.150 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 21:03:19 GMT /pklxkd-61/ 2023-05-31T21:03:19.150 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:hLB/3UScfc36qJjImbkC6+I98Sc= 2023-05-31T21:03:19.151 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 21:03:19 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:hLB/3UScfc36qJjImbkC6+I98Sc=', 'Content-Length': '0'} 2023-05-31T21:03:19.154 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000411d5fa95a726b3c-006477b617-1630-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 21:03:19 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:03:19.154 DEBUG:boto:b'pklxkd-61obj1151000falseobj1152023-05-31T20:59:45.636Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T21:03:19.155 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj116 2023-05-31T21:03:19.156 DEBUG:boto:path=/pklxkd-61/obj116 2023-05-31T21:03:19.156 DEBUG:boto:auth_path=/pklxkd-61/obj116 2023-05-31T21:03:19.156 DEBUG:boto:path=/pklxkd-61/obj116?versionId=null 2023-05-31T21:03:19.157 DEBUG:boto:auth_path=/pklxkd-61/obj116?versionId=null 2023-05-31T21:03:19.157 DEBUG:boto:Method: GET 2023-05-31T21:03:19.158 DEBUG:boto:Path: /pklxkd-61/obj116?versionId=null 2023-05-31T21:03:19.158 DEBUG:boto:Data: 2023-05-31T21:03:19.158 DEBUG:boto:Headers: {} 2023-05-31T21:03:19.159 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T21:03:19.159 DEBUG:boto:Port: 8001 2023-05-31T21:03:19.159 DEBUG:boto:Params: {} 2023-05-31T21:03:19.160 DEBUG:boto:Token: None 2023-05-31T21:03:19.160 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 21:03:19 GMT /pklxkd-61/obj116?versionId=null 2023-05-31T21:03:19.161 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:1jQ6HoMl0F+hNJ2y3NCXMGy6das= 2023-05-31T21:03:19.161 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 21:03:19 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:1jQ6HoMl0F+hNJ2y3NCXMGy6das=', 'Content-Length': '0'} 2023-05-31T21:03:19.163 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:59:45 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000031b677d53e119a81-006477b617-1f4e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 21:03:19 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:03:19.163 DEBUG:boto:path=/pklxkd-61/obj116 2023-05-31T21:03:19.164 DEBUG:boto:auth_path=/pklxkd-61/obj116 2023-05-31T21:03:19.164 DEBUG:boto:path=/pklxkd-61/obj116?versionId=null 2023-05-31T21:03:19.164 DEBUG:boto:auth_path=/pklxkd-61/obj116?versionId=null 2023-05-31T21:03:19.165 DEBUG:boto:Method: GET 2023-05-31T21:03:19.165 DEBUG:boto:Path: /pklxkd-61/obj116?versionId=null 2023-05-31T21:03:19.165 DEBUG:boto:Data: 2023-05-31T21:03:19.166 DEBUG:boto:Headers: {} 2023-05-31T21:03:19.166 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T21:03:19.166 DEBUG:boto:Port: 8000 2023-05-31T21:03:19.167 DEBUG:boto:Params: {} 2023-05-31T21:03:19.167 DEBUG:boto:Token: None 2023-05-31T21:03:19.168 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 21:03:19 GMT /pklxkd-61/obj116?versionId=null 2023-05-31T21:03:19.168 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:1jQ6HoMl0F+hNJ2y3NCXMGy6das= 2023-05-31T21:03:19.169 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 21:03:19 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:1jQ6HoMl0F+hNJ2y3NCXMGy6das=', 'Content-Length': '0'} 2023-05-31T21:03:19.170 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:59:45 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002a7b51d5e286d8bb-006477b617-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 21:03:19 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:03:19.212 DEBUG:boto:path=/pklxkd-61/obj116 2023-05-31T21:03:19.213 DEBUG:boto:auth_path=/pklxkd-61/obj116 2023-05-31T21:03:19.213 DEBUG:boto:path=/pklxkd-61/obj116?versionId=null 2023-05-31T21:03:19.213 DEBUG:boto:auth_path=/pklxkd-61/obj116?versionId=null 2023-05-31T21:03:19.214 DEBUG:boto:Method: HEAD 2023-05-31T21:03:19.214 DEBUG:boto:Path: /pklxkd-61/obj116?versionId=null 2023-05-31T21:03:19.214 DEBUG:boto:Data: 2023-05-31T21:03:19.215 DEBUG:boto:Headers: {} 2023-05-31T21:03:19.215 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T21:03:19.215 DEBUG:boto:Port: 8001 2023-05-31T21:03:19.216 DEBUG:boto:Params: {} 2023-05-31T21:03:19.216 DEBUG:boto:Token: None 2023-05-31T21:03:19.217 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 21:03:19 GMT /pklxkd-61/obj116?versionId=null 2023-05-31T21:03:19.217 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:k1LXlfXsU6oEzIN9OasxZrA+O7I= 2023-05-31T21:03:19.217 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 21:03:19 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:k1LXlfXsU6oEzIN9OasxZrA+O7I=', 'Content-Length': '0'} 2023-05-31T21:03:19.219 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:59:45 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000bd60ba2a3d3c5f3a-006477b617-1f4e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 21:03:19 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:03:19.219 DEBUG:boto:path=/pklxkd-61/obj116 2023-05-31T21:03:19.220 DEBUG:boto:auth_path=/pklxkd-61/obj116 2023-05-31T21:03:19.220 DEBUG:boto:path=/pklxkd-61/obj116?versionId=null 2023-05-31T21:03:19.221 DEBUG:boto:auth_path=/pklxkd-61/obj116?versionId=null 2023-05-31T21:03:19.221 DEBUG:boto:Method: HEAD 2023-05-31T21:03:19.221 DEBUG:boto:Path: /pklxkd-61/obj116?versionId=null 2023-05-31T21:03:19.222 DEBUG:boto:Data: 2023-05-31T21:03:19.222 DEBUG:boto:Headers: {} 2023-05-31T21:03:19.222 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T21:03:19.223 DEBUG:boto:Port: 8000 2023-05-31T21:03:19.223 DEBUG:boto:Params: {} 2023-05-31T21:03:19.223 DEBUG:boto:Token: None 2023-05-31T21:03:19.224 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 21:03:19 GMT /pklxkd-61/obj116?versionId=null 2023-05-31T21:03:19.224 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:k1LXlfXsU6oEzIN9OasxZrA+O7I= 2023-05-31T21:03:19.224 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 21:03:19 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:k1LXlfXsU6oEzIN9OasxZrA+O7I=', 'Content-Length': '0'} 2023-05-31T21:03:19.226 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:59:45 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004ba1011e401e00d7-006477b617-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 21:03:19 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:03:19.227 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj116 2023-05-31T21:03:19.227 DEBUG:boto:path=/pklxkd-61/obj116 2023-05-31T21:03:19.228 DEBUG:boto:auth_path=/pklxkd-61/obj116 2023-05-31T21:03:19.228 DEBUG:boto:path=/pklxkd-61/obj116?versionId=null 2023-05-31T21:03:19.228 DEBUG:boto:auth_path=/pklxkd-61/obj116?versionId=null 2023-05-31T21:03:19.229 DEBUG:boto:Method: GET 2023-05-31T21:03:19.229 DEBUG:boto:Path: /pklxkd-61/obj116?versionId=null 2023-05-31T21:03:19.229 DEBUG:boto:Data: 2023-05-31T21:03:19.230 DEBUG:boto:Headers: {} 2023-05-31T21:03:19.230 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T21:03:19.230 DEBUG:boto:Port: 8001 2023-05-31T21:03:19.231 DEBUG:boto:Params: {} 2023-05-31T21:03:19.231 DEBUG:boto:Token: None 2023-05-31T21:03:19.232 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 21:03:19 GMT /pklxkd-61/obj116?versionId=null 2023-05-31T21:03:19.232 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:1jQ6HoMl0F+hNJ2y3NCXMGy6das= 2023-05-31T21:03:19.232 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 21:03:19 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:1jQ6HoMl0F+hNJ2y3NCXMGy6das=', 'Content-Length': '0'} 2023-05-31T21:03:19.234 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:59:45 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f5cf84633ebefd89-006477b617-1f4e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 21:03:19 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:03:19.235 DEBUG:boto:path=/pklxkd-61/obj116 2023-05-31T21:03:19.235 DEBUG:boto:auth_path=/pklxkd-61/obj116 2023-05-31T21:03:19.235 DEBUG:boto:path=/pklxkd-61/obj116?versionId=null 2023-05-31T21:03:19.236 DEBUG:boto:auth_path=/pklxkd-61/obj116?versionId=null 2023-05-31T21:03:19.236 DEBUG:boto:Method: GET 2023-05-31T21:03:19.237 DEBUG:boto:Path: /pklxkd-61/obj116?versionId=null 2023-05-31T21:03:19.237 DEBUG:boto:Data: 2023-05-31T21:03:19.237 DEBUG:boto:Headers: {} 2023-05-31T21:03:19.238 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T21:03:19.238 DEBUG:boto:Port: 8000 2023-05-31T21:03:19.238 DEBUG:boto:Params: {} 2023-05-31T21:03:19.239 DEBUG:boto:Token: None 2023-05-31T21:03:19.239 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 21:03:19 GMT /pklxkd-61/obj116?versionId=null 2023-05-31T21:03:19.240 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:1jQ6HoMl0F+hNJ2y3NCXMGy6das= 2023-05-31T21:03:19.240 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 21:03:19 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:1jQ6HoMl0F+hNJ2y3NCXMGy6das=', 'Content-Length': '0'} 2023-05-31T21:03:19.242 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:59:45 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006a763c7a47219eb5-006477b617-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 21:03:19 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:03:19.242 DEBUG:boto:path=/pklxkd-61/ 2023-05-31T21:03:19.243 DEBUG:boto:auth_path=/pklxkd-61/ 2023-05-31T21:03:19.243 DEBUG:boto:path=/pklxkd-61/?prefix=obj116 2023-05-31T21:03:19.243 DEBUG:boto:auth_path=/pklxkd-61/?prefix=obj116 2023-05-31T21:03:19.244 DEBUG:boto:Method: GET 2023-05-31T21:03:19.244 DEBUG:boto:Path: /pklxkd-61/?prefix=obj116 2023-05-31T21:03:19.245 DEBUG:boto:Data: 2023-05-31T21:03:19.245 DEBUG:boto:Headers: {} 2023-05-31T21:03:19.245 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T21:03:19.246 DEBUG:boto:Port: 8001 2023-05-31T21:03:19.246 DEBUG:boto:Params: {} 2023-05-31T21:03:19.247 DEBUG:boto:Token: None 2023-05-31T21:03:19.247 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 21:03:19 GMT /pklxkd-61/ 2023-05-31T21:03:19.247 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:hLB/3UScfc36qJjImbkC6+I98Sc= 2023-05-31T21:03:19.248 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 21:03:19 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:hLB/3UScfc36qJjImbkC6+I98Sc=', 'Content-Length': '0'} 2023-05-31T21:03:19.250 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f282d0ea8abc2e2c-006477b617-1f4e-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 21:03:19 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:03:19.250 DEBUG:boto:b'pklxkd-61obj1161000falseobj1162023-05-31T20:59:45.977Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T21:03:19.251 DEBUG:boto:path=/pklxkd-61/ 2023-05-31T21:03:19.252 DEBUG:boto:auth_path=/pklxkd-61/ 2023-05-31T21:03:19.252 DEBUG:boto:path=/pklxkd-61/?prefix=obj116 2023-05-31T21:03:19.252 DEBUG:boto:auth_path=/pklxkd-61/?prefix=obj116 2023-05-31T21:03:19.253 DEBUG:boto:Method: GET 2023-05-31T21:03:19.253 DEBUG:boto:Path: /pklxkd-61/?prefix=obj116 2023-05-31T21:03:19.253 DEBUG:boto:Data: 2023-05-31T21:03:19.254 DEBUG:boto:Headers: {} 2023-05-31T21:03:19.254 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T21:03:19.255 DEBUG:boto:Port: 8000 2023-05-31T21:03:19.255 DEBUG:boto:Params: {} 2023-05-31T21:03:19.255 DEBUG:boto:Token: None 2023-05-31T21:03:19.256 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 21:03:19 GMT /pklxkd-61/ 2023-05-31T21:03:19.256 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:hLB/3UScfc36qJjImbkC6+I98Sc= 2023-05-31T21:03:19.257 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 21:03:19 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:hLB/3UScfc36qJjImbkC6+I98Sc=', 'Content-Length': '0'} 2023-05-31T21:03:19.260 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000001923110e35ddb520-006477b617-1630-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 21:03:19 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:03:19.261 DEBUG:boto:b'pklxkd-61obj1161000falseobj1162023-05-31T20:59:45.977Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T21:03:19.262 DEBUG:boto:path=/pklxkd-61/obj116 2023-05-31T21:03:19.262 DEBUG:boto:auth_path=/pklxkd-61/obj116 2023-05-31T21:03:19.262 DEBUG:boto:Method: HEAD 2023-05-31T21:03:19.263 DEBUG:boto:Path: /pklxkd-61/obj116 2023-05-31T21:03:19.263 DEBUG:boto:Data: 2023-05-31T21:03:19.263 DEBUG:boto:Headers: {} 2023-05-31T21:03:19.264 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T21:03:19.264 DEBUG:boto:Port: 8001 2023-05-31T21:03:19.264 DEBUG:boto:Params: {} 2023-05-31T21:03:19.265 DEBUG:boto:Token: None 2023-05-31T21:03:19.265 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 21:03:19 GMT /pklxkd-61/obj116 2023-05-31T21:03:19.265 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:am7HKD5A5rKibReWlbdyrHRlm6M= 2023-05-31T21:03:19.266 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 21:03:19 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:am7HKD5A5rKibReWlbdyrHRlm6M=', 'Content-Length': '0'} 2023-05-31T21:03:19.267 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:59:45 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000009ba0477800ebcab-006477b617-1f4e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 21:03:19 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:03:19.268 DEBUG:boto:path=/pklxkd-61/obj116 2023-05-31T21:03:19.268 DEBUG:boto:auth_path=/pklxkd-61/obj116 2023-05-31T21:03:19.269 DEBUG:boto:Method: HEAD 2023-05-31T21:03:19.269 DEBUG:boto:Path: /pklxkd-61/obj116 2023-05-31T21:03:19.269 DEBUG:boto:Data: 2023-05-31T21:03:19.270 DEBUG:boto:Headers: {} 2023-05-31T21:03:19.270 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T21:03:19.270 DEBUG:boto:Port: 8000 2023-05-31T21:03:19.271 DEBUG:boto:Params: {} 2023-05-31T21:03:19.271 DEBUG:boto:Token: None 2023-05-31T21:03:19.271 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 21:03:19 GMT /pklxkd-61/obj116 2023-05-31T21:03:19.272 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:am7HKD5A5rKibReWlbdyrHRlm6M= 2023-05-31T21:03:19.272 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 21:03:19 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:am7HKD5A5rKibReWlbdyrHRlm6M=', 'Content-Length': '0'} 2023-05-31T21:03:19.274 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:59:45 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004525e80e62b58b91-006477b617-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 21:03:19 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:03:19.274 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj116 2023-05-31T21:03:19.275 DEBUG:boto:path=/pklxkd-61/obj116 2023-05-31T21:03:19.275 DEBUG:boto:auth_path=/pklxkd-61/obj116 2023-05-31T21:03:19.276 DEBUG:boto:Method: GET 2023-05-31T21:03:19.276 DEBUG:boto:Path: /pklxkd-61/obj116 2023-05-31T21:03:19.276 DEBUG:boto:Data: 2023-05-31T21:03:19.277 DEBUG:boto:Headers: {} 2023-05-31T21:03:19.277 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T21:03:19.277 DEBUG:boto:Port: 8001 2023-05-31T21:03:19.278 DEBUG:boto:Params: {} 2023-05-31T21:03:19.278 DEBUG:boto:Token: None 2023-05-31T21:03:19.278 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 21:03:19 GMT /pklxkd-61/obj116 2023-05-31T21:03:19.279 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:H1huhSg0pyBw6tR9UgMHmY7sNCg= 2023-05-31T21:03:19.279 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 21:03:19 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:H1huhSg0pyBw6tR9UgMHmY7sNCg=', 'Content-Length': '0'} 2023-05-31T21:03:19.281 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:59:45 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003c865066b9f8b427-006477b617-1f4e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 21:03:19 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:03:19.281 DEBUG:boto:path=/pklxkd-61/obj116 2023-05-31T21:03:19.282 DEBUG:boto:auth_path=/pklxkd-61/obj116 2023-05-31T21:03:19.282 DEBUG:boto:Method: GET 2023-05-31T21:03:19.282 DEBUG:boto:Path: /pklxkd-61/obj116 2023-05-31T21:03:19.283 DEBUG:boto:Data: 2023-05-31T21:03:19.283 DEBUG:boto:Headers: {} 2023-05-31T21:03:19.283 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T21:03:19.284 DEBUG:boto:Port: 8000 2023-05-31T21:03:19.284 DEBUG:boto:Params: {} 2023-05-31T21:03:19.285 DEBUG:boto:Token: None 2023-05-31T21:03:19.285 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 21:03:19 GMT /pklxkd-61/obj116 2023-05-31T21:03:19.285 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:H1huhSg0pyBw6tR9UgMHmY7sNCg= 2023-05-31T21:03:19.286 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 21:03:19 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:H1huhSg0pyBw6tR9UgMHmY7sNCg=', 'Content-Length': '0'} 2023-05-31T21:03:19.287 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:59:45 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e7afa0c9d80f5687-006477b617-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 21:03:19 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:03:19.288 DEBUG:boto:path=/pklxkd-61/ 2023-05-31T21:03:19.288 DEBUG:boto:auth_path=/pklxkd-61/ 2023-05-31T21:03:19.289 DEBUG:boto:path=/pklxkd-61/?prefix=obj116 2023-05-31T21:03:19.289 DEBUG:boto:auth_path=/pklxkd-61/?prefix=obj116 2023-05-31T21:03:19.289 DEBUG:boto:Method: GET 2023-05-31T21:03:19.290 DEBUG:boto:Path: /pklxkd-61/?prefix=obj116 2023-05-31T21:03:19.290 DEBUG:boto:Data: 2023-05-31T21:03:19.290 DEBUG:boto:Headers: {} 2023-05-31T21:03:19.291 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T21:03:19.291 DEBUG:boto:Port: 8001 2023-05-31T21:03:19.292 DEBUG:boto:Params: {} 2023-05-31T21:03:19.292 DEBUG:boto:Token: None 2023-05-31T21:03:19.293 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 21:03:19 GMT /pklxkd-61/ 2023-05-31T21:03:19.293 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:hLB/3UScfc36qJjImbkC6+I98Sc= 2023-05-31T21:03:19.293 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 21:03:19 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:hLB/3UScfc36qJjImbkC6+I98Sc=', 'Content-Length': '0'} 2023-05-31T21:03:19.295 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000006aaefd7262489045-006477b617-1f4e-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 21:03:19 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:03:19.336 DEBUG:boto:b'pklxkd-61obj1161000falseobj1162023-05-31T20:59:45.977Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T21:03:19.337 DEBUG:boto:path=/pklxkd-61/ 2023-05-31T21:03:19.337 DEBUG:boto:auth_path=/pklxkd-61/ 2023-05-31T21:03:19.338 DEBUG:boto:path=/pklxkd-61/?prefix=obj116 2023-05-31T21:03:19.338 DEBUG:boto:auth_path=/pklxkd-61/?prefix=obj116 2023-05-31T21:03:19.338 DEBUG:boto:Method: GET 2023-05-31T21:03:19.339 DEBUG:boto:Path: /pklxkd-61/?prefix=obj116 2023-05-31T21:03:19.339 DEBUG:boto:Data: 2023-05-31T21:03:19.339 DEBUG:boto:Headers: {} 2023-05-31T21:03:19.340 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T21:03:19.340 DEBUG:boto:Port: 8000 2023-05-31T21:03:19.340 DEBUG:boto:Params: {} 2023-05-31T21:03:19.341 DEBUG:boto:Token: None 2023-05-31T21:03:19.341 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 21:03:19 GMT /pklxkd-61/ 2023-05-31T21:03:19.342 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:hLB/3UScfc36qJjImbkC6+I98Sc= 2023-05-31T21:03:19.342 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 21:03:19 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:hLB/3UScfc36qJjImbkC6+I98Sc=', 'Content-Length': '0'} 2023-05-31T21:03:19.345 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e6c783fdce3a9ea1-006477b617-1630-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 21:03:19 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:03:19.346 DEBUG:boto:b'pklxkd-61obj1161000falseobj1162023-05-31T20:59:45.977Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T21:03:19.347 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj124 2023-05-31T21:03:19.347 DEBUG:boto:path=/pklxkd-61/obj124 2023-05-31T21:03:19.347 DEBUG:boto:auth_path=/pklxkd-61/obj124 2023-05-31T21:03:19.348 DEBUG:boto:path=/pklxkd-61/obj124?versionId=null 2023-05-31T21:03:19.348 DEBUG:boto:auth_path=/pklxkd-61/obj124?versionId=null 2023-05-31T21:03:19.348 DEBUG:boto:Method: GET 2023-05-31T21:03:19.349 DEBUG:boto:Path: /pklxkd-61/obj124?versionId=null 2023-05-31T21:03:19.349 DEBUG:boto:Data: 2023-05-31T21:03:19.349 DEBUG:boto:Headers: {} 2023-05-31T21:03:19.350 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T21:03:19.350 DEBUG:boto:Port: 8001 2023-05-31T21:03:19.351 DEBUG:boto:Params: {} 2023-05-31T21:03:19.351 DEBUG:boto:Token: None 2023-05-31T21:03:19.351 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 21:03:19 GMT /pklxkd-61/obj124?versionId=null 2023-05-31T21:03:19.352 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:x5kwXjB+pRJKNS1SjQ2pKae82WU= 2023-05-31T21:03:19.352 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 21:03:19 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:x5kwXjB+pRJKNS1SjQ2pKae82WU=', 'Content-Length': '0'} 2023-05-31T21:03:19.354 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:59:46 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000074c84cc1c12f361c-006477b617-1f4e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 21:03:19 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:03:19.354 DEBUG:boto:path=/pklxkd-61/obj124 2023-05-31T21:03:19.355 DEBUG:boto:auth_path=/pklxkd-61/obj124 2023-05-31T21:03:19.355 DEBUG:boto:path=/pklxkd-61/obj124?versionId=null 2023-05-31T21:03:19.356 DEBUG:boto:auth_path=/pklxkd-61/obj124?versionId=null 2023-05-31T21:03:19.356 DEBUG:boto:Method: GET 2023-05-31T21:03:19.356 DEBUG:boto:Path: /pklxkd-61/obj124?versionId=null 2023-05-31T21:03:19.357 DEBUG:boto:Data: 2023-05-31T21:03:19.357 DEBUG:boto:Headers: {} 2023-05-31T21:03:19.357 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T21:03:19.358 DEBUG:boto:Port: 8000 2023-05-31T21:03:19.358 DEBUG:boto:Params: {} 2023-05-31T21:03:19.359 DEBUG:boto:Token: None 2023-05-31T21:03:19.359 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 21:03:19 GMT /pklxkd-61/obj124?versionId=null 2023-05-31T21:03:19.359 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:x5kwXjB+pRJKNS1SjQ2pKae82WU= 2023-05-31T21:03:19.360 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 21:03:19 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:x5kwXjB+pRJKNS1SjQ2pKae82WU=', 'Content-Length': '0'} 2023-05-31T21:03:19.361 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:59:46 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e854cd7e8daee283-006477b617-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 21:03:19 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:03:19.404 DEBUG:boto:path=/pklxkd-61/obj124 2023-05-31T21:03:19.405 DEBUG:boto:auth_path=/pklxkd-61/obj124 2023-05-31T21:03:19.405 DEBUG:boto:path=/pklxkd-61/obj124?versionId=null 2023-05-31T21:03:19.405 DEBUG:boto:auth_path=/pklxkd-61/obj124?versionId=null 2023-05-31T21:03:19.406 DEBUG:boto:Method: HEAD 2023-05-31T21:03:19.406 DEBUG:boto:Path: /pklxkd-61/obj124?versionId=null 2023-05-31T21:03:19.406 DEBUG:boto:Data: 2023-05-31T21:03:19.407 DEBUG:boto:Headers: {} 2023-05-31T21:03:19.407 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T21:03:19.407 DEBUG:boto:Port: 8001 2023-05-31T21:03:19.408 DEBUG:boto:Params: {} 2023-05-31T21:03:19.408 DEBUG:boto:Token: None 2023-05-31T21:03:19.408 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 21:03:19 GMT /pklxkd-61/obj124?versionId=null 2023-05-31T21:03:19.409 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:SEkmfbK/A0NgV9sHKhhHw8Kga+U= 2023-05-31T21:03:19.409 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 21:03:19 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:SEkmfbK/A0NgV9sHKhhHw8Kga+U=', 'Content-Length': '0'} 2023-05-31T21:03:19.411 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:59:46 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b79cd2e7534436f2-006477b617-1f4e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 21:03:19 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:03:19.411 DEBUG:boto:path=/pklxkd-61/obj124 2023-05-31T21:03:19.412 DEBUG:boto:auth_path=/pklxkd-61/obj124 2023-05-31T21:03:19.412 DEBUG:boto:path=/pklxkd-61/obj124?versionId=null 2023-05-31T21:03:19.412 DEBUG:boto:auth_path=/pklxkd-61/obj124?versionId=null 2023-05-31T21:03:19.413 DEBUG:boto:Method: HEAD 2023-05-31T21:03:19.413 DEBUG:boto:Path: /pklxkd-61/obj124?versionId=null 2023-05-31T21:03:19.413 DEBUG:boto:Data: 2023-05-31T21:03:19.414 DEBUG:boto:Headers: {} 2023-05-31T21:03:19.414 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T21:03:19.414 DEBUG:boto:Port: 8000 2023-05-31T21:03:19.415 DEBUG:boto:Params: {} 2023-05-31T21:03:19.415 DEBUG:boto:Token: None 2023-05-31T21:03:19.416 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 21:03:19 GMT /pklxkd-61/obj124?versionId=null 2023-05-31T21:03:19.416 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:SEkmfbK/A0NgV9sHKhhHw8Kga+U= 2023-05-31T21:03:19.416 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 21:03:19 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:SEkmfbK/A0NgV9sHKhhHw8Kga+U=', 'Content-Length': '0'} 2023-05-31T21:03:19.418 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:59:46 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000040f5ce608faa529f-006477b617-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 21:03:19 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:03:19.418 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj124 2023-05-31T21:03:19.419 DEBUG:boto:path=/pklxkd-61/obj124 2023-05-31T21:03:19.419 DEBUG:boto:auth_path=/pklxkd-61/obj124 2023-05-31T21:03:19.419 DEBUG:boto:path=/pklxkd-61/obj124?versionId=null 2023-05-31T21:03:19.420 DEBUG:boto:auth_path=/pklxkd-61/obj124?versionId=null 2023-05-31T21:03:19.420 DEBUG:boto:Method: GET 2023-05-31T21:03:19.420 DEBUG:boto:Path: /pklxkd-61/obj124?versionId=null 2023-05-31T21:03:19.421 DEBUG:boto:Data: 2023-05-31T21:03:19.421 DEBUG:boto:Headers: {} 2023-05-31T21:03:19.422 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T21:03:19.422 DEBUG:boto:Port: 8001 2023-05-31T21:03:19.422 DEBUG:boto:Params: {} 2023-05-31T21:03:19.423 DEBUG:boto:Token: None 2023-05-31T21:03:19.423 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 21:03:19 GMT /pklxkd-61/obj124?versionId=null 2023-05-31T21:03:19.423 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:x5kwXjB+pRJKNS1SjQ2pKae82WU= 2023-05-31T21:03:19.424 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 21:03:19 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:x5kwXjB+pRJKNS1SjQ2pKae82WU=', 'Content-Length': '0'} 2023-05-31T21:03:19.425 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:59:46 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008eccd56f98701ebf-006477b617-1f4e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 21:03:19 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:03:19.426 DEBUG:boto:path=/pklxkd-61/obj124 2023-05-31T21:03:19.426 DEBUG:boto:auth_path=/pklxkd-61/obj124 2023-05-31T21:03:19.427 DEBUG:boto:path=/pklxkd-61/obj124?versionId=null 2023-05-31T21:03:19.427 DEBUG:boto:auth_path=/pklxkd-61/obj124?versionId=null 2023-05-31T21:03:19.427 DEBUG:boto:Method: GET 2023-05-31T21:03:19.428 DEBUG:boto:Path: /pklxkd-61/obj124?versionId=null 2023-05-31T21:03:19.428 DEBUG:boto:Data: 2023-05-31T21:03:19.429 DEBUG:boto:Headers: {} 2023-05-31T21:03:19.429 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T21:03:19.429 DEBUG:boto:Port: 8000 2023-05-31T21:03:19.430 DEBUG:boto:Params: {} 2023-05-31T21:03:19.430 DEBUG:boto:Token: None 2023-05-31T21:03:19.430 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 21:03:19 GMT /pklxkd-61/obj124?versionId=null 2023-05-31T21:03:19.431 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:x5kwXjB+pRJKNS1SjQ2pKae82WU= 2023-05-31T21:03:19.431 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 21:03:19 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:x5kwXjB+pRJKNS1SjQ2pKae82WU=', 'Content-Length': '0'} 2023-05-31T21:03:19.433 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:59:46 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000becf7ec355dbc19-006477b617-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 21:03:19 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:03:19.434 DEBUG:boto:path=/pklxkd-61/ 2023-05-31T21:03:19.434 DEBUG:boto:auth_path=/pklxkd-61/ 2023-05-31T21:03:19.434 DEBUG:boto:path=/pklxkd-61/?prefix=obj124 2023-05-31T21:03:19.435 DEBUG:boto:auth_path=/pklxkd-61/?prefix=obj124 2023-05-31T21:03:19.435 DEBUG:boto:Method: GET 2023-05-31T21:03:19.435 DEBUG:boto:Path: /pklxkd-61/?prefix=obj124 2023-05-31T21:03:19.436 DEBUG:boto:Data: 2023-05-31T21:03:19.436 DEBUG:boto:Headers: {} 2023-05-31T21:03:19.437 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T21:03:19.437 DEBUG:boto:Port: 8001 2023-05-31T21:03:19.437 DEBUG:boto:Params: {} 2023-05-31T21:03:19.438 DEBUG:boto:Token: None 2023-05-31T21:03:19.438 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 21:03:19 GMT /pklxkd-61/ 2023-05-31T21:03:19.438 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:hLB/3UScfc36qJjImbkC6+I98Sc= 2023-05-31T21:03:19.439 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 21:03:19 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:hLB/3UScfc36qJjImbkC6+I98Sc=', 'Content-Length': '0'} 2023-05-31T21:03:19.441 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000dc438f773df51177-006477b617-1f4e-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 21:03:19 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:03:19.442 DEBUG:boto:b'pklxkd-61obj1241000falseobj1242023-05-31T20:59:46.419Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T21:03:19.442 DEBUG:boto:path=/pklxkd-61/ 2023-05-31T21:03:19.443 DEBUG:boto:auth_path=/pklxkd-61/ 2023-05-31T21:03:19.443 DEBUG:boto:path=/pklxkd-61/?prefix=obj124 2023-05-31T21:03:19.443 DEBUG:boto:auth_path=/pklxkd-61/?prefix=obj124 2023-05-31T21:03:19.444 DEBUG:boto:Method: GET 2023-05-31T21:03:19.444 DEBUG:boto:Path: /pklxkd-61/?prefix=obj124 2023-05-31T21:03:19.444 DEBUG:boto:Data: 2023-05-31T21:03:19.445 DEBUG:boto:Headers: {} 2023-05-31T21:03:19.445 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T21:03:19.445 DEBUG:boto:Port: 8000 2023-05-31T21:03:19.446 DEBUG:boto:Params: {} 2023-05-31T21:03:19.446 DEBUG:boto:Token: None 2023-05-31T21:03:19.447 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 21:03:19 GMT /pklxkd-61/ 2023-05-31T21:03:19.447 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:hLB/3UScfc36qJjImbkC6+I98Sc= 2023-05-31T21:03:19.447 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 21:03:19 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:hLB/3UScfc36qJjImbkC6+I98Sc=', 'Content-Length': '0'} 2023-05-31T21:03:19.451 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000455550b532f74108-006477b617-1630-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 21:03:19 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:03:19.451 DEBUG:boto:b'pklxkd-61obj1241000falseobj1242023-05-31T20:59:46.419Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T21:03:19.452 DEBUG:boto:path=/pklxkd-61/obj124 2023-05-31T21:03:19.452 DEBUG:boto:auth_path=/pklxkd-61/obj124 2023-05-31T21:03:19.453 DEBUG:boto:Method: HEAD 2023-05-31T21:03:19.453 DEBUG:boto:Path: /pklxkd-61/obj124 2023-05-31T21:03:19.453 DEBUG:boto:Data: 2023-05-31T21:03:19.454 DEBUG:boto:Headers: {} 2023-05-31T21:03:19.454 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T21:03:19.454 DEBUG:boto:Port: 8001 2023-05-31T21:03:19.455 DEBUG:boto:Params: {} 2023-05-31T21:03:19.455 DEBUG:boto:Token: None 2023-05-31T21:03:19.455 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 21:03:19 GMT /pklxkd-61/obj124 2023-05-31T21:03:19.456 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:LWuZ0rRrGDRt7ttKeQepqfldaCo= 2023-05-31T21:03:19.456 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 21:03:19 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:LWuZ0rRrGDRt7ttKeQepqfldaCo=', 'Content-Length': '0'} 2023-05-31T21:03:19.458 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:59:46 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000556ec26f5f84a0f5-006477b617-1f4e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 21:03:19 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:03:19.458 DEBUG:boto:path=/pklxkd-61/obj124 2023-05-31T21:03:19.459 DEBUG:boto:auth_path=/pklxkd-61/obj124 2023-05-31T21:03:19.459 DEBUG:boto:Method: HEAD 2023-05-31T21:03:19.459 DEBUG:boto:Path: /pklxkd-61/obj124 2023-05-31T21:03:19.460 DEBUG:boto:Data: 2023-05-31T21:03:19.460 DEBUG:boto:Headers: {} 2023-05-31T21:03:19.460 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T21:03:19.461 DEBUG:boto:Port: 8000 2023-05-31T21:03:19.461 DEBUG:boto:Params: {} 2023-05-31T21:03:19.461 DEBUG:boto:Token: None 2023-05-31T21:03:19.462 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 21:03:19 GMT /pklxkd-61/obj124 2023-05-31T21:03:19.462 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:LWuZ0rRrGDRt7ttKeQepqfldaCo= 2023-05-31T21:03:19.462 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 21:03:19 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:LWuZ0rRrGDRt7ttKeQepqfldaCo=', 'Content-Length': '0'} 2023-05-31T21:03:19.464 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:59:46 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d3cdbd1a01406142-006477b617-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 21:03:19 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:03:19.465 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj124 2023-05-31T21:03:19.465 DEBUG:boto:path=/pklxkd-61/obj124 2023-05-31T21:03:19.465 DEBUG:boto:auth_path=/pklxkd-61/obj124 2023-05-31T21:03:19.466 DEBUG:boto:Method: GET 2023-05-31T21:03:19.466 DEBUG:boto:Path: /pklxkd-61/obj124 2023-05-31T21:03:19.466 DEBUG:boto:Data: 2023-05-31T21:03:19.467 DEBUG:boto:Headers: {} 2023-05-31T21:03:19.467 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T21:03:19.468 DEBUG:boto:Port: 8001 2023-05-31T21:03:19.468 DEBUG:boto:Params: {} 2023-05-31T21:03:19.468 DEBUG:boto:Token: None 2023-05-31T21:03:19.469 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 21:03:19 GMT /pklxkd-61/obj124 2023-05-31T21:03:19.469 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:GtjBgfCZZOLLMN6oiCW5eVfPKBw= 2023-05-31T21:03:19.470 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 21:03:19 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:GtjBgfCZZOLLMN6oiCW5eVfPKBw=', 'Content-Length': '0'} 2023-05-31T21:03:19.471 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:59:46 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000cde43fe8d0b21808-006477b617-1f4e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 21:03:19 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:03:19.472 DEBUG:boto:path=/pklxkd-61/obj124 2023-05-31T21:03:19.472 DEBUG:boto:auth_path=/pklxkd-61/obj124 2023-05-31T21:03:19.472 DEBUG:boto:Method: GET 2023-05-31T21:03:19.473 DEBUG:boto:Path: /pklxkd-61/obj124 2023-05-31T21:03:19.473 DEBUG:boto:Data: 2023-05-31T21:03:19.473 DEBUG:boto:Headers: {} 2023-05-31T21:03:19.474 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T21:03:19.474 DEBUG:boto:Port: 8000 2023-05-31T21:03:19.475 DEBUG:boto:Params: {} 2023-05-31T21:03:19.475 DEBUG:boto:Token: None 2023-05-31T21:03:19.475 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 21:03:19 GMT /pklxkd-61/obj124 2023-05-31T21:03:19.476 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:GtjBgfCZZOLLMN6oiCW5eVfPKBw= 2023-05-31T21:03:19.476 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 21:03:19 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:GtjBgfCZZOLLMN6oiCW5eVfPKBw=', 'Content-Length': '0'} 2023-05-31T21:03:19.478 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:59:46 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001bfb8efad0e223e8-006477b617-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 21:03:19 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:03:19.520 DEBUG:boto:path=/pklxkd-61/ 2023-05-31T21:03:19.521 DEBUG:boto:auth_path=/pklxkd-61/ 2023-05-31T21:03:19.521 DEBUG:boto:path=/pklxkd-61/?prefix=obj124 2023-05-31T21:03:19.521 DEBUG:boto:auth_path=/pklxkd-61/?prefix=obj124 2023-05-31T21:03:19.522 DEBUG:boto:Method: GET 2023-05-31T21:03:19.522 DEBUG:boto:Path: /pklxkd-61/?prefix=obj124 2023-05-31T21:03:19.522 DEBUG:boto:Data: 2023-05-31T21:03:19.523 DEBUG:boto:Headers: {} 2023-05-31T21:03:19.523 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T21:03:19.523 DEBUG:boto:Port: 8001 2023-05-31T21:03:19.524 DEBUG:boto:Params: {} 2023-05-31T21:03:19.524 DEBUG:boto:Token: None 2023-05-31T21:03:19.525 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 21:03:19 GMT /pklxkd-61/ 2023-05-31T21:03:19.525 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:hLB/3UScfc36qJjImbkC6+I98Sc= 2023-05-31T21:03:19.525 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 21:03:19 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:hLB/3UScfc36qJjImbkC6+I98Sc=', 'Content-Length': '0'} 2023-05-31T21:03:19.528 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000002349b8786cb5622d-006477b617-1f4e-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 21:03:19 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:03:19.529 DEBUG:boto:b'pklxkd-61obj1241000falseobj1242023-05-31T20:59:46.419Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T21:03:19.529 DEBUG:boto:path=/pklxkd-61/ 2023-05-31T21:03:19.530 DEBUG:boto:auth_path=/pklxkd-61/ 2023-05-31T21:03:19.530 DEBUG:boto:path=/pklxkd-61/?prefix=obj124 2023-05-31T21:03:19.530 DEBUG:boto:auth_path=/pklxkd-61/?prefix=obj124 2023-05-31T21:03:19.531 DEBUG:boto:Method: GET 2023-05-31T21:03:19.531 DEBUG:boto:Path: /pklxkd-61/?prefix=obj124 2023-05-31T21:03:19.531 DEBUG:boto:Data: 2023-05-31T21:03:19.532 DEBUG:boto:Headers: {} 2023-05-31T21:03:19.532 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T21:03:19.532 DEBUG:boto:Port: 8000 2023-05-31T21:03:19.533 DEBUG:boto:Params: {} 2023-05-31T21:03:19.533 DEBUG:boto:Token: None 2023-05-31T21:03:19.534 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 21:03:19 GMT /pklxkd-61/ 2023-05-31T21:03:19.534 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:hLB/3UScfc36qJjImbkC6+I98Sc= 2023-05-31T21:03:19.534 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 21:03:19 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:hLB/3UScfc36qJjImbkC6+I98Sc=', 'Content-Length': '0'} 2023-05-31T21:03:19.538 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000009358d7dc0e265c46-006477b617-1630-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 21:03:19 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:03:19.539 DEBUG:boto:b'pklxkd-61obj1241000falseobj1242023-05-31T20:59:46.419Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T21:03:19.539 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj145 2023-05-31T21:03:19.540 DEBUG:boto:path=/pklxkd-61/obj145 2023-05-31T21:03:19.540 DEBUG:boto:auth_path=/pklxkd-61/obj145 2023-05-31T21:03:19.540 DEBUG:boto:path=/pklxkd-61/obj145?versionId=null 2023-05-31T21:03:19.541 DEBUG:boto:auth_path=/pklxkd-61/obj145?versionId=null 2023-05-31T21:03:19.541 DEBUG:boto:Method: GET 2023-05-31T21:03:19.541 DEBUG:boto:Path: /pklxkd-61/obj145?versionId=null 2023-05-31T21:03:19.542 DEBUG:boto:Data: 2023-05-31T21:03:19.542 DEBUG:boto:Headers: {} 2023-05-31T21:03:19.543 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T21:03:19.543 DEBUG:boto:Port: 8001 2023-05-31T21:03:19.543 DEBUG:boto:Params: {} 2023-05-31T21:03:19.544 DEBUG:boto:Token: None 2023-05-31T21:03:19.544 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 21:03:19 GMT /pklxkd-61/obj145?versionId=null 2023-05-31T21:03:19.544 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:GU9na2OvQps7fATi4Pi2RjbYeoU= 2023-05-31T21:03:19.545 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 21:03:19 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:GU9na2OvQps7fATi4Pi2RjbYeoU=', 'Content-Length': '0'} 2023-05-31T21:03:19.546 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:59:46 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000318ef262795d7cf-006477b617-1f4e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 21:03:19 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:03:19.588 DEBUG:boto:path=/pklxkd-61/obj145 2023-05-31T21:03:19.589 DEBUG:boto:auth_path=/pklxkd-61/obj145 2023-05-31T21:03:19.589 DEBUG:boto:path=/pklxkd-61/obj145?versionId=null 2023-05-31T21:03:19.589 DEBUG:boto:auth_path=/pklxkd-61/obj145?versionId=null 2023-05-31T21:03:19.590 DEBUG:boto:Method: GET 2023-05-31T21:03:19.590 DEBUG:boto:Path: /pklxkd-61/obj145?versionId=null 2023-05-31T21:03:19.590 DEBUG:boto:Data: 2023-05-31T21:03:19.591 DEBUG:boto:Headers: {} 2023-05-31T21:03:19.591 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T21:03:19.591 DEBUG:boto:Port: 8000 2023-05-31T21:03:19.592 DEBUG:boto:Params: {} 2023-05-31T21:03:19.592 DEBUG:boto:Token: None 2023-05-31T21:03:19.593 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 21:03:19 GMT /pklxkd-61/obj145?versionId=null 2023-05-31T21:03:19.593 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:GU9na2OvQps7fATi4Pi2RjbYeoU= 2023-05-31T21:03:19.593 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 21:03:19 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:GU9na2OvQps7fATi4Pi2RjbYeoU=', 'Content-Length': '0'} 2023-05-31T21:03:19.595 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:59:46 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003569bc170bd0ddbb-006477b617-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 21:03:19 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:03:19.596 DEBUG:boto:path=/pklxkd-61/obj145 2023-05-31T21:03:19.596 DEBUG:boto:auth_path=/pklxkd-61/obj145 2023-05-31T21:03:19.596 DEBUG:boto:path=/pklxkd-61/obj145?versionId=null 2023-05-31T21:03:19.597 DEBUG:boto:auth_path=/pklxkd-61/obj145?versionId=null 2023-05-31T21:03:19.597 DEBUG:boto:Method: HEAD 2023-05-31T21:03:19.597 DEBUG:boto:Path: /pklxkd-61/obj145?versionId=null 2023-05-31T21:03:19.598 DEBUG:boto:Data: 2023-05-31T21:03:19.598 DEBUG:boto:Headers: {} 2023-05-31T21:03:19.598 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T21:03:19.599 DEBUG:boto:Port: 8001 2023-05-31T21:03:19.599 DEBUG:boto:Params: {} 2023-05-31T21:03:19.599 DEBUG:boto:Token: None 2023-05-31T21:03:19.600 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 21:03:19 GMT /pklxkd-61/obj145?versionId=null 2023-05-31T21:03:19.600 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:ByKrW/Q5pn+YbX7h73WCS3GKCFs= 2023-05-31T21:03:19.601 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 21:03:19 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:ByKrW/Q5pn+YbX7h73WCS3GKCFs=', 'Content-Length': '0'} 2023-05-31T21:03:19.602 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:59:46 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000138d37f1ea7357a8-006477b617-1f4e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 21:03:19 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:03:19.603 DEBUG:boto:path=/pklxkd-61/obj145 2023-05-31T21:03:19.603 DEBUG:boto:auth_path=/pklxkd-61/obj145 2023-05-31T21:03:19.604 DEBUG:boto:path=/pklxkd-61/obj145?versionId=null 2023-05-31T21:03:19.604 DEBUG:boto:auth_path=/pklxkd-61/obj145?versionId=null 2023-05-31T21:03:19.604 DEBUG:boto:Method: HEAD 2023-05-31T21:03:19.605 DEBUG:boto:Path: /pklxkd-61/obj145?versionId=null 2023-05-31T21:03:19.605 DEBUG:boto:Data: 2023-05-31T21:03:19.605 DEBUG:boto:Headers: {} 2023-05-31T21:03:19.606 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T21:03:19.606 DEBUG:boto:Port: 8000 2023-05-31T21:03:19.606 DEBUG:boto:Params: {} 2023-05-31T21:03:19.607 DEBUG:boto:Token: None 2023-05-31T21:03:19.607 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 21:03:19 GMT /pklxkd-61/obj145?versionId=null 2023-05-31T21:03:19.607 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:ByKrW/Q5pn+YbX7h73WCS3GKCFs= 2023-05-31T21:03:19.608 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 21:03:19 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:ByKrW/Q5pn+YbX7h73WCS3GKCFs=', 'Content-Length': '0'} 2023-05-31T21:03:19.610 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:59:46 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f24890e92b24e901-006477b617-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 21:03:19 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:03:19.610 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj145 2023-05-31T21:03:19.611 DEBUG:boto:path=/pklxkd-61/obj145 2023-05-31T21:03:19.611 DEBUG:boto:auth_path=/pklxkd-61/obj145 2023-05-31T21:03:19.611 DEBUG:boto:path=/pklxkd-61/obj145?versionId=null 2023-05-31T21:03:19.612 DEBUG:boto:auth_path=/pklxkd-61/obj145?versionId=null 2023-05-31T21:03:19.612 DEBUG:boto:Method: GET 2023-05-31T21:03:19.612 DEBUG:boto:Path: /pklxkd-61/obj145?versionId=null 2023-05-31T21:03:19.613 DEBUG:boto:Data: 2023-05-31T21:03:19.613 DEBUG:boto:Headers: {} 2023-05-31T21:03:19.613 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T21:03:19.614 DEBUG:boto:Port: 8001 2023-05-31T21:03:19.614 DEBUG:boto:Params: {} 2023-05-31T21:03:19.615 DEBUG:boto:Token: None 2023-05-31T21:03:19.615 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 21:03:19 GMT /pklxkd-61/obj145?versionId=null 2023-05-31T21:03:19.615 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:GU9na2OvQps7fATi4Pi2RjbYeoU= 2023-05-31T21:03:19.616 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 21:03:19 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:GU9na2OvQps7fATi4Pi2RjbYeoU=', 'Content-Length': '0'} 2023-05-31T21:03:19.617 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:59:46 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000060e2932072c992de-006477b617-1f4e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 21:03:19 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:03:19.618 DEBUG:boto:path=/pklxkd-61/obj145 2023-05-31T21:03:19.618 DEBUG:boto:auth_path=/pklxkd-61/obj145 2023-05-31T21:03:19.619 DEBUG:boto:path=/pklxkd-61/obj145?versionId=null 2023-05-31T21:03:19.619 DEBUG:boto:auth_path=/pklxkd-61/obj145?versionId=null 2023-05-31T21:03:19.619 DEBUG:boto:Method: GET 2023-05-31T21:03:19.620 DEBUG:boto:Path: /pklxkd-61/obj145?versionId=null 2023-05-31T21:03:19.620 DEBUG:boto:Data: 2023-05-31T21:03:19.621 DEBUG:boto:Headers: {} 2023-05-31T21:03:19.621 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T21:03:19.621 DEBUG:boto:Port: 8000 2023-05-31T21:03:19.622 DEBUG:boto:Params: {} 2023-05-31T21:03:19.622 DEBUG:boto:Token: None 2023-05-31T21:03:19.622 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 21:03:19 GMT /pklxkd-61/obj145?versionId=null 2023-05-31T21:03:19.623 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:GU9na2OvQps7fATi4Pi2RjbYeoU= 2023-05-31T21:03:19.623 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 21:03:19 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:GU9na2OvQps7fATi4Pi2RjbYeoU=', 'Content-Length': '0'} 2023-05-31T21:03:19.625 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:59:46 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000327cf3cb88e05268-006477b617-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 21:03:19 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:03:19.626 DEBUG:boto:path=/pklxkd-61/ 2023-05-31T21:03:19.626 DEBUG:boto:auth_path=/pklxkd-61/ 2023-05-31T21:03:19.626 DEBUG:boto:path=/pklxkd-61/?prefix=obj145 2023-05-31T21:03:19.627 DEBUG:boto:auth_path=/pklxkd-61/?prefix=obj145 2023-05-31T21:03:19.627 DEBUG:boto:Method: GET 2023-05-31T21:03:19.628 DEBUG:boto:Path: /pklxkd-61/?prefix=obj145 2023-05-31T21:03:19.628 DEBUG:boto:Data: 2023-05-31T21:03:19.628 DEBUG:boto:Headers: {} 2023-05-31T21:03:19.629 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T21:03:19.629 DEBUG:boto:Port: 8001 2023-05-31T21:03:19.629 DEBUG:boto:Params: {} 2023-05-31T21:03:19.630 DEBUG:boto:Token: None 2023-05-31T21:03:19.630 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 21:03:19 GMT /pklxkd-61/ 2023-05-31T21:03:19.630 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:hLB/3UScfc36qJjImbkC6+I98Sc= 2023-05-31T21:03:19.631 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 21:03:19 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:hLB/3UScfc36qJjImbkC6+I98Sc=', 'Content-Length': '0'} 2023-05-31T21:03:19.633 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000003265c4f5a5354674-006477b617-1f4e-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 21:03:19 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:03:19.633 DEBUG:boto:b'pklxkd-61obj1451000falseobj1452023-05-31T20:59:46.038Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T21:03:19.634 DEBUG:boto:path=/pklxkd-61/ 2023-05-31T21:03:19.634 DEBUG:boto:auth_path=/pklxkd-61/ 2023-05-31T21:03:19.635 DEBUG:boto:path=/pklxkd-61/?prefix=obj145 2023-05-31T21:03:19.635 DEBUG:boto:auth_path=/pklxkd-61/?prefix=obj145 2023-05-31T21:03:19.636 DEBUG:boto:Method: GET 2023-05-31T21:03:19.636 DEBUG:boto:Path: /pklxkd-61/?prefix=obj145 2023-05-31T21:03:19.636 DEBUG:boto:Data: 2023-05-31T21:03:19.637 DEBUG:boto:Headers: {} 2023-05-31T21:03:19.637 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T21:03:19.637 DEBUG:boto:Port: 8000 2023-05-31T21:03:19.638 DEBUG:boto:Params: {} 2023-05-31T21:03:19.638 DEBUG:boto:Token: None 2023-05-31T21:03:19.639 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 21:03:19 GMT /pklxkd-61/ 2023-05-31T21:03:19.639 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:hLB/3UScfc36qJjImbkC6+I98Sc= 2023-05-31T21:03:19.639 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 21:03:19 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:hLB/3UScfc36qJjImbkC6+I98Sc=', 'Content-Length': '0'} 2023-05-31T21:03:19.643 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c13cd8517e4b92c2-006477b617-1630-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 21:03:19 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:03:19.643 DEBUG:boto:b'pklxkd-61obj1451000falseobj1452023-05-31T20:59:46.038Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T21:03:19.644 DEBUG:boto:path=/pklxkd-61/obj145 2023-05-31T21:03:19.644 DEBUG:boto:auth_path=/pklxkd-61/obj145 2023-05-31T21:03:19.645 DEBUG:boto:Method: HEAD 2023-05-31T21:03:19.645 DEBUG:boto:Path: /pklxkd-61/obj145 2023-05-31T21:03:19.645 DEBUG:boto:Data: 2023-05-31T21:03:19.646 DEBUG:boto:Headers: {} 2023-05-31T21:03:19.646 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T21:03:19.646 DEBUG:boto:Port: 8001 2023-05-31T21:03:19.647 DEBUG:boto:Params: {} 2023-05-31T21:03:19.647 DEBUG:boto:Token: None 2023-05-31T21:03:19.647 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 21:03:19 GMT /pklxkd-61/obj145 2023-05-31T21:03:19.648 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:8W9HLkK3QQks21wr1J86UrJ27GU= 2023-05-31T21:03:19.648 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 21:03:19 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:8W9HLkK3QQks21wr1J86UrJ27GU=', 'Content-Length': '0'} 2023-05-31T21:03:19.650 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:59:46 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000001b954ab55e63dd8-006477b617-1f4e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 21:03:19 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:03:19.651 DEBUG:boto:path=/pklxkd-61/obj145 2023-05-31T21:03:19.651 DEBUG:boto:auth_path=/pklxkd-61/obj145 2023-05-31T21:03:19.651 DEBUG:boto:Method: HEAD 2023-05-31T21:03:19.652 DEBUG:boto:Path: /pklxkd-61/obj145 2023-05-31T21:03:19.652 DEBUG:boto:Data: 2023-05-31T21:03:19.652 DEBUG:boto:Headers: {} 2023-05-31T21:03:19.653 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T21:03:19.653 DEBUG:boto:Port: 8000 2023-05-31T21:03:19.653 DEBUG:boto:Params: {} 2023-05-31T21:03:19.654 DEBUG:boto:Token: None 2023-05-31T21:03:19.654 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 21:03:19 GMT /pklxkd-61/obj145 2023-05-31T21:03:19.654 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:8W9HLkK3QQks21wr1J86UrJ27GU= 2023-05-31T21:03:19.655 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 21:03:19 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:8W9HLkK3QQks21wr1J86UrJ27GU=', 'Content-Length': '0'} 2023-05-31T21:03:19.656 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:59:46 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ad24cdbc61edacc7-006477b617-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 21:03:19 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:03:19.657 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj145 2023-05-31T21:03:19.657 DEBUG:boto:path=/pklxkd-61/obj145 2023-05-31T21:03:19.658 DEBUG:boto:auth_path=/pklxkd-61/obj145 2023-05-31T21:03:19.658 DEBUG:boto:Method: GET 2023-05-31T21:03:19.658 DEBUG:boto:Path: /pklxkd-61/obj145 2023-05-31T21:03:19.659 DEBUG:boto:Data: 2023-05-31T21:03:19.659 DEBUG:boto:Headers: {} 2023-05-31T21:03:19.659 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T21:03:19.660 DEBUG:boto:Port: 8001 2023-05-31T21:03:19.660 DEBUG:boto:Params: {} 2023-05-31T21:03:19.660 DEBUG:boto:Token: None 2023-05-31T21:03:19.661 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 21:03:19 GMT /pklxkd-61/obj145 2023-05-31T21:03:19.661 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:XWXn1XFq6+7QDJ5wNOnChIYMQDg= 2023-05-31T21:03:19.662 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 21:03:19 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:XWXn1XFq6+7QDJ5wNOnChIYMQDg=', 'Content-Length': '0'} 2023-05-31T21:03:19.663 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:59:46 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e3efbb71903aff3f-006477b617-1f4e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 21:03:19 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:03:19.664 DEBUG:boto:path=/pklxkd-61/obj145 2023-05-31T21:03:19.664 DEBUG:boto:auth_path=/pklxkd-61/obj145 2023-05-31T21:03:19.665 DEBUG:boto:Method: GET 2023-05-31T21:03:19.665 DEBUG:boto:Path: /pklxkd-61/obj145 2023-05-31T21:03:19.665 DEBUG:boto:Data: 2023-05-31T21:03:19.666 DEBUG:boto:Headers: {} 2023-05-31T21:03:19.666 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T21:03:19.667 DEBUG:boto:Port: 8000 2023-05-31T21:03:19.667 DEBUG:boto:Params: {} 2023-05-31T21:03:19.667 DEBUG:boto:Token: None 2023-05-31T21:03:19.668 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 21:03:19 GMT /pklxkd-61/obj145 2023-05-31T21:03:19.668 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:XWXn1XFq6+7QDJ5wNOnChIYMQDg= 2023-05-31T21:03:19.669 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 21:03:19 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:XWXn1XFq6+7QDJ5wNOnChIYMQDg=', 'Content-Length': '0'} 2023-05-31T21:03:19.670 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:59:46 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000cc3defdf3ba9d572-006477b617-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 21:03:19 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:03:19.712 DEBUG:boto:path=/pklxkd-61/ 2023-05-31T21:03:19.713 DEBUG:boto:auth_path=/pklxkd-61/ 2023-05-31T21:03:19.713 DEBUG:boto:path=/pklxkd-61/?prefix=obj145 2023-05-31T21:03:19.714 DEBUG:boto:auth_path=/pklxkd-61/?prefix=obj145 2023-05-31T21:03:19.714 DEBUG:boto:Method: GET 2023-05-31T21:03:19.714 DEBUG:boto:Path: /pklxkd-61/?prefix=obj145 2023-05-31T21:03:19.715 DEBUG:boto:Data: 2023-05-31T21:03:19.715 DEBUG:boto:Headers: {} 2023-05-31T21:03:19.715 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T21:03:19.716 DEBUG:boto:Port: 8001 2023-05-31T21:03:19.716 DEBUG:boto:Params: {} 2023-05-31T21:03:19.716 DEBUG:boto:Token: None 2023-05-31T21:03:19.717 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 21:03:19 GMT /pklxkd-61/ 2023-05-31T21:03:19.717 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:hLB/3UScfc36qJjImbkC6+I98Sc= 2023-05-31T21:03:19.718 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 21:03:19 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:hLB/3UScfc36qJjImbkC6+I98Sc=', 'Content-Length': '0'} 2023-05-31T21:03:19.720 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000039e899d743f67d7d-006477b617-1f4e-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 21:03:19 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:03:19.720 DEBUG:boto:b'pklxkd-61obj1451000falseobj1452023-05-31T20:59:46.038Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T21:03:19.721 DEBUG:boto:path=/pklxkd-61/ 2023-05-31T21:03:19.722 DEBUG:boto:auth_path=/pklxkd-61/ 2023-05-31T21:03:19.722 DEBUG:boto:path=/pklxkd-61/?prefix=obj145 2023-05-31T21:03:19.722 DEBUG:boto:auth_path=/pklxkd-61/?prefix=obj145 2023-05-31T21:03:19.723 DEBUG:boto:Method: GET 2023-05-31T21:03:19.723 DEBUG:boto:Path: /pklxkd-61/?prefix=obj145 2023-05-31T21:03:19.723 DEBUG:boto:Data: 2023-05-31T21:03:19.724 DEBUG:boto:Headers: {} 2023-05-31T21:03:19.724 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T21:03:19.724 DEBUG:boto:Port: 8000 2023-05-31T21:03:19.725 DEBUG:boto:Params: {} 2023-05-31T21:03:19.725 DEBUG:boto:Token: None 2023-05-31T21:03:19.726 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 21:03:19 GMT /pklxkd-61/ 2023-05-31T21:03:19.726 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:hLB/3UScfc36qJjImbkC6+I98Sc= 2023-05-31T21:03:19.726 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 21:03:19 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:hLB/3UScfc36qJjImbkC6+I98Sc=', 'Content-Length': '0'} 2023-05-31T21:03:19.730 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a310bb9232d4a2a7-006477b617-1630-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 21:03:19 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:03:19.731 DEBUG:boto:b'pklxkd-61obj1451000falseobj1452023-05-31T20:59:46.038Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T21:03:19.732 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj148 2023-05-31T21:03:19.733 DEBUG:boto:path=/pklxkd-61/obj148 2023-05-31T21:03:19.733 DEBUG:boto:auth_path=/pklxkd-61/obj148 2023-05-31T21:03:19.734 DEBUG:boto:path=/pklxkd-61/obj148?versionId=null 2023-05-31T21:03:19.734 DEBUG:boto:auth_path=/pklxkd-61/obj148?versionId=null 2023-05-31T21:03:19.735 DEBUG:boto:Method: GET 2023-05-31T21:03:19.735 DEBUG:boto:Path: /pklxkd-61/obj148?versionId=null 2023-05-31T21:03:19.735 DEBUG:boto:Data: 2023-05-31T21:03:19.736 DEBUG:boto:Headers: {} 2023-05-31T21:03:19.736 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T21:03:19.736 DEBUG:boto:Port: 8001 2023-05-31T21:03:19.737 DEBUG:boto:Params: {} 2023-05-31T21:03:19.737 DEBUG:boto:Token: None 2023-05-31T21:03:19.737 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 21:03:19 GMT /pklxkd-61/obj148?versionId=null 2023-05-31T21:03:19.738 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:DQN74p2v0lGpc2jgnnNcLEvVpU0= 2023-05-31T21:03:19.738 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 21:03:19 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:DQN74p2v0lGpc2jgnnNcLEvVpU0=', 'Content-Length': '0'} 2023-05-31T21:03:19.740 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:59:47 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000086bca62a7b921d88-006477b617-1f4e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 21:03:19 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:03:19.740 DEBUG:boto:path=/pklxkd-61/obj148 2023-05-31T21:03:19.741 DEBUG:boto:auth_path=/pklxkd-61/obj148 2023-05-31T21:03:19.741 DEBUG:boto:path=/pklxkd-61/obj148?versionId=null 2023-05-31T21:03:19.742 DEBUG:boto:auth_path=/pklxkd-61/obj148?versionId=null 2023-05-31T21:03:19.742 DEBUG:boto:Method: GET 2023-05-31T21:03:19.742 DEBUG:boto:Path: /pklxkd-61/obj148?versionId=null 2023-05-31T21:03:19.743 DEBUG:boto:Data: 2023-05-31T21:03:19.743 DEBUG:boto:Headers: {} 2023-05-31T21:03:19.744 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T21:03:19.744 DEBUG:boto:Port: 8000 2023-05-31T21:03:19.744 DEBUG:boto:Params: {} 2023-05-31T21:03:19.745 DEBUG:boto:Token: None 2023-05-31T21:03:19.745 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 21:03:19 GMT /pklxkd-61/obj148?versionId=null 2023-05-31T21:03:19.745 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:DQN74p2v0lGpc2jgnnNcLEvVpU0= 2023-05-31T21:03:19.746 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 21:03:19 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:DQN74p2v0lGpc2jgnnNcLEvVpU0=', 'Content-Length': '0'} 2023-05-31T21:03:19.748 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:59:47 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000a3bda0e15e27e31-006477b617-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 21:03:19 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:03:19.748 DEBUG:boto:path=/pklxkd-61/obj148 2023-05-31T21:03:19.749 DEBUG:boto:auth_path=/pklxkd-61/obj148 2023-05-31T21:03:19.749 DEBUG:boto:path=/pklxkd-61/obj148?versionId=null 2023-05-31T21:03:19.749 DEBUG:boto:auth_path=/pklxkd-61/obj148?versionId=null 2023-05-31T21:03:19.750 DEBUG:boto:Method: HEAD 2023-05-31T21:03:19.750 DEBUG:boto:Path: /pklxkd-61/obj148?versionId=null 2023-05-31T21:03:19.750 DEBUG:boto:Data: 2023-05-31T21:03:19.751 DEBUG:boto:Headers: {} 2023-05-31T21:03:19.751 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T21:03:19.751 DEBUG:boto:Port: 8001 2023-05-31T21:03:19.752 DEBUG:boto:Params: {} 2023-05-31T21:03:19.752 DEBUG:boto:Token: None 2023-05-31T21:03:19.752 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 21:03:19 GMT /pklxkd-61/obj148?versionId=null 2023-05-31T21:03:19.753 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:n9ZC3H0OCjiYdsWoqHNHV/hppH4= 2023-05-31T21:03:19.753 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 21:03:19 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:n9ZC3H0OCjiYdsWoqHNHV/hppH4=', 'Content-Length': '0'} 2023-05-31T21:03:19.755 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:59:47 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000852ad4b2693ae596-006477b617-1f4e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 21:03:19 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:03:19.755 DEBUG:boto:path=/pklxkd-61/obj148 2023-05-31T21:03:19.756 DEBUG:boto:auth_path=/pklxkd-61/obj148 2023-05-31T21:03:19.756 DEBUG:boto:path=/pklxkd-61/obj148?versionId=null 2023-05-31T21:03:19.756 DEBUG:boto:auth_path=/pklxkd-61/obj148?versionId=null 2023-05-31T21:03:19.757 DEBUG:boto:Method: HEAD 2023-05-31T21:03:19.757 DEBUG:boto:Path: /pklxkd-61/obj148?versionId=null 2023-05-31T21:03:19.757 DEBUG:boto:Data: 2023-05-31T21:03:19.758 DEBUG:boto:Headers: {} 2023-05-31T21:03:19.758 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T21:03:19.758 DEBUG:boto:Port: 8000 2023-05-31T21:03:19.759 DEBUG:boto:Params: {} 2023-05-31T21:03:19.759 DEBUG:boto:Token: None 2023-05-31T21:03:19.759 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 21:03:19 GMT /pklxkd-61/obj148?versionId=null 2023-05-31T21:03:19.760 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:n9ZC3H0OCjiYdsWoqHNHV/hppH4= 2023-05-31T21:03:19.760 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 21:03:19 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:n9ZC3H0OCjiYdsWoqHNHV/hppH4=', 'Content-Length': '0'} 2023-05-31T21:03:19.762 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:59:47 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000071ecec98ab902aae-006477b617-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 21:03:19 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:03:19.762 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj148 2023-05-31T21:03:19.763 DEBUG:boto:path=/pklxkd-61/obj148 2023-05-31T21:03:19.763 DEBUG:boto:auth_path=/pklxkd-61/obj148 2023-05-31T21:03:19.763 DEBUG:boto:path=/pklxkd-61/obj148?versionId=null 2023-05-31T21:03:19.764 DEBUG:boto:auth_path=/pklxkd-61/obj148?versionId=null 2023-05-31T21:03:19.764 DEBUG:boto:Method: GET 2023-05-31T21:03:19.765 DEBUG:boto:Path: /pklxkd-61/obj148?versionId=null 2023-05-31T21:03:19.765 DEBUG:boto:Data: 2023-05-31T21:03:19.765 DEBUG:boto:Headers: {} 2023-05-31T21:03:19.766 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T21:03:19.766 DEBUG:boto:Port: 8001 2023-05-31T21:03:19.766 DEBUG:boto:Params: {} 2023-05-31T21:03:19.767 DEBUG:boto:Token: None 2023-05-31T21:03:19.767 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 21:03:19 GMT /pklxkd-61/obj148?versionId=null 2023-05-31T21:03:19.768 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:DQN74p2v0lGpc2jgnnNcLEvVpU0= 2023-05-31T21:03:19.768 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 21:03:19 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:DQN74p2v0lGpc2jgnnNcLEvVpU0=', 'Content-Length': '0'} 2023-05-31T21:03:19.769 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:59:47 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000082beb523e1d3d869-006477b617-1f4e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 21:03:19 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:03:19.770 DEBUG:boto:path=/pklxkd-61/obj148 2023-05-31T21:03:19.770 DEBUG:boto:auth_path=/pklxkd-61/obj148 2023-05-31T21:03:19.771 DEBUG:boto:path=/pklxkd-61/obj148?versionId=null 2023-05-31T21:03:19.771 DEBUG:boto:auth_path=/pklxkd-61/obj148?versionId=null 2023-05-31T21:03:19.771 DEBUG:boto:Method: GET 2023-05-31T21:03:19.772 DEBUG:boto:Path: /pklxkd-61/obj148?versionId=null 2023-05-31T21:03:19.772 DEBUG:boto:Data: 2023-05-31T21:03:19.773 DEBUG:boto:Headers: {} 2023-05-31T21:03:19.773 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T21:03:19.773 DEBUG:boto:Port: 8000 2023-05-31T21:03:19.774 DEBUG:boto:Params: {} 2023-05-31T21:03:19.774 DEBUG:boto:Token: None 2023-05-31T21:03:19.774 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 21:03:19 GMT /pklxkd-61/obj148?versionId=null 2023-05-31T21:03:19.775 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:DQN74p2v0lGpc2jgnnNcLEvVpU0= 2023-05-31T21:03:19.775 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 21:03:19 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:DQN74p2v0lGpc2jgnnNcLEvVpU0=', 'Content-Length': '0'} 2023-05-31T21:03:19.777 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:59:47 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b34785fd6c4038dc-006477b617-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 21:03:19 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:03:19.777 DEBUG:boto:path=/pklxkd-61/ 2023-05-31T21:03:19.778 DEBUG:boto:auth_path=/pklxkd-61/ 2023-05-31T21:03:19.778 DEBUG:boto:path=/pklxkd-61/?prefix=obj148 2023-05-31T21:03:19.778 DEBUG:boto:auth_path=/pklxkd-61/?prefix=obj148 2023-05-31T21:03:19.779 DEBUG:boto:Method: GET 2023-05-31T21:03:19.779 DEBUG:boto:Path: /pklxkd-61/?prefix=obj148 2023-05-31T21:03:19.779 DEBUG:boto:Data: 2023-05-31T21:03:19.780 DEBUG:boto:Headers: {} 2023-05-31T21:03:19.780 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T21:03:19.780 DEBUG:boto:Port: 8001 2023-05-31T21:03:19.781 DEBUG:boto:Params: {} 2023-05-31T21:03:19.781 DEBUG:boto:Token: None 2023-05-31T21:03:19.782 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 21:03:19 GMT /pklxkd-61/ 2023-05-31T21:03:19.782 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:hLB/3UScfc36qJjImbkC6+I98Sc= 2023-05-31T21:03:19.782 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 21:03:19 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:hLB/3UScfc36qJjImbkC6+I98Sc=', 'Content-Length': '0'} 2023-05-31T21:03:19.785 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d6f6851fc965483c-006477b617-1f4e-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 21:03:19 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:03:19.785 DEBUG:boto:b'pklxkd-61obj1481000falseobj1482023-05-31T20:59:47.007Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T21:03:19.786 DEBUG:boto:path=/pklxkd-61/ 2023-05-31T21:03:19.786 DEBUG:boto:auth_path=/pklxkd-61/ 2023-05-31T21:03:19.787 DEBUG:boto:path=/pklxkd-61/?prefix=obj148 2023-05-31T21:03:19.787 DEBUG:boto:auth_path=/pklxkd-61/?prefix=obj148 2023-05-31T21:03:19.787 DEBUG:boto:Method: GET 2023-05-31T21:03:19.788 DEBUG:boto:Path: /pklxkd-61/?prefix=obj148 2023-05-31T21:03:19.788 DEBUG:boto:Data: 2023-05-31T21:03:19.788 DEBUG:boto:Headers: {} 2023-05-31T21:03:19.789 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T21:03:19.789 DEBUG:boto:Port: 8000 2023-05-31T21:03:19.789 DEBUG:boto:Params: {} 2023-05-31T21:03:19.790 DEBUG:boto:Token: None 2023-05-31T21:03:19.790 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 21:03:19 GMT /pklxkd-61/ 2023-05-31T21:03:19.791 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:hLB/3UScfc36qJjImbkC6+I98Sc= 2023-05-31T21:03:19.791 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 21:03:19 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:hLB/3UScfc36qJjImbkC6+I98Sc=', 'Content-Length': '0'} 2023-05-31T21:03:19.794 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000006ce116a127a6a576-006477b617-1630-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 21:03:19 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:03:19.836 DEBUG:boto:b'pklxkd-61obj1481000falseobj1482023-05-31T20:59:47.007Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T21:03:19.837 DEBUG:boto:path=/pklxkd-61/obj148 2023-05-31T21:03:19.837 DEBUG:boto:auth_path=/pklxkd-61/obj148 2023-05-31T21:03:19.838 DEBUG:boto:Method: HEAD 2023-05-31T21:03:19.838 DEBUG:boto:Path: /pklxkd-61/obj148 2023-05-31T21:03:19.838 DEBUG:boto:Data: 2023-05-31T21:03:19.839 DEBUG:boto:Headers: {} 2023-05-31T21:03:19.839 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T21:03:19.839 DEBUG:boto:Port: 8001 2023-05-31T21:03:19.840 DEBUG:boto:Params: {} 2023-05-31T21:03:19.840 DEBUG:boto:Token: None 2023-05-31T21:03:19.840 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 21:03:19 GMT /pklxkd-61/obj148 2023-05-31T21:03:19.841 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:M1IdSSdHs4VTNRWIFzi9ECokMpI= 2023-05-31T21:03:19.841 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 21:03:19 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:M1IdSSdHs4VTNRWIFzi9ECokMpI=', 'Content-Length': '0'} 2023-05-31T21:03:19.843 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:59:47 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000eb364951453967d0-006477b617-1f4e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 21:03:19 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:03:19.844 DEBUG:boto:path=/pklxkd-61/obj148 2023-05-31T21:03:19.844 DEBUG:boto:auth_path=/pklxkd-61/obj148 2023-05-31T21:03:19.844 DEBUG:boto:Method: HEAD 2023-05-31T21:03:19.845 DEBUG:boto:Path: /pklxkd-61/obj148 2023-05-31T21:03:19.845 DEBUG:boto:Data: 2023-05-31T21:03:19.845 DEBUG:boto:Headers: {} 2023-05-31T21:03:19.846 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T21:03:19.846 DEBUG:boto:Port: 8000 2023-05-31T21:03:19.846 DEBUG:boto:Params: {} 2023-05-31T21:03:19.847 DEBUG:boto:Token: None 2023-05-31T21:03:19.847 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 21:03:19 GMT /pklxkd-61/obj148 2023-05-31T21:03:19.847 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:M1IdSSdHs4VTNRWIFzi9ECokMpI= 2023-05-31T21:03:19.848 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 21:03:19 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:M1IdSSdHs4VTNRWIFzi9ECokMpI=', 'Content-Length': '0'} 2023-05-31T21:03:19.849 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:59:47 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000064fb40ba42c19396-006477b617-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 21:03:19 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:03:19.850 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj148 2023-05-31T21:03:19.851 DEBUG:boto:path=/pklxkd-61/obj148 2023-05-31T21:03:19.851 DEBUG:boto:auth_path=/pklxkd-61/obj148 2023-05-31T21:03:19.852 DEBUG:boto:Method: GET 2023-05-31T21:03:19.852 DEBUG:boto:Path: /pklxkd-61/obj148 2023-05-31T21:03:19.853 DEBUG:boto:Data: 2023-05-31T21:03:19.853 DEBUG:boto:Headers: {} 2023-05-31T21:03:19.853 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T21:03:19.854 DEBUG:boto:Port: 8001 2023-05-31T21:03:19.854 DEBUG:boto:Params: {} 2023-05-31T21:03:19.854 DEBUG:boto:Token: None 2023-05-31T21:03:19.855 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 21:03:19 GMT /pklxkd-61/obj148 2023-05-31T21:03:19.855 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:8gL7281GWa7CcC0xROdieref81c= 2023-05-31T21:03:19.856 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 21:03:19 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:8gL7281GWa7CcC0xROdieref81c=', 'Content-Length': '0'} 2023-05-31T21:03:19.858 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:59:47 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000050877a151aa36244-006477b617-1f4e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 21:03:19 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:03:19.858 DEBUG:boto:path=/pklxkd-61/obj148 2023-05-31T21:03:19.859 DEBUG:boto:auth_path=/pklxkd-61/obj148 2023-05-31T21:03:19.859 DEBUG:boto:Method: GET 2023-05-31T21:03:19.859 DEBUG:boto:Path: /pklxkd-61/obj148 2023-05-31T21:03:19.860 DEBUG:boto:Data: 2023-05-31T21:03:19.860 DEBUG:boto:Headers: {} 2023-05-31T21:03:19.860 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T21:03:19.861 DEBUG:boto:Port: 8000 2023-05-31T21:03:19.861 DEBUG:boto:Params: {} 2023-05-31T21:03:19.862 DEBUG:boto:Token: None 2023-05-31T21:03:19.862 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 21:03:19 GMT /pklxkd-61/obj148 2023-05-31T21:03:19.863 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:8gL7281GWa7CcC0xROdieref81c= 2023-05-31T21:03:19.863 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 21:03:19 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:8gL7281GWa7CcC0xROdieref81c=', 'Content-Length': '0'} 2023-05-31T21:03:19.865 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:59:47 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e469bd9622984bc9-006477b617-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 21:03:19 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:03:19.865 DEBUG:boto:path=/pklxkd-61/ 2023-05-31T21:03:19.866 DEBUG:boto:auth_path=/pklxkd-61/ 2023-05-31T21:03:19.866 DEBUG:boto:path=/pklxkd-61/?prefix=obj148 2023-05-31T21:03:19.866 DEBUG:boto:auth_path=/pklxkd-61/?prefix=obj148 2023-05-31T21:03:19.867 DEBUG:boto:Method: GET 2023-05-31T21:03:19.867 DEBUG:boto:Path: /pklxkd-61/?prefix=obj148 2023-05-31T21:03:19.868 DEBUG:boto:Data: 2023-05-31T21:03:19.868 DEBUG:boto:Headers: {} 2023-05-31T21:03:19.868 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T21:03:19.869 DEBUG:boto:Port: 8001 2023-05-31T21:03:19.869 DEBUG:boto:Params: {} 2023-05-31T21:03:19.870 DEBUG:boto:Token: None 2023-05-31T21:03:19.870 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 21:03:19 GMT /pklxkd-61/ 2023-05-31T21:03:19.870 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:hLB/3UScfc36qJjImbkC6+I98Sc= 2023-05-31T21:03:19.871 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 21:03:19 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:hLB/3UScfc36qJjImbkC6+I98Sc=', 'Content-Length': '0'} 2023-05-31T21:03:19.873 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f5923b6650ae6d28-006477b617-1f4e-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 21:03:19 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:03:19.874 DEBUG:boto:b'pklxkd-61obj1481000falseobj1482023-05-31T20:59:47.007Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T21:03:19.874 DEBUG:boto:path=/pklxkd-61/ 2023-05-31T21:03:19.875 DEBUG:boto:auth_path=/pklxkd-61/ 2023-05-31T21:03:19.875 DEBUG:boto:path=/pklxkd-61/?prefix=obj148 2023-05-31T21:03:19.876 DEBUG:boto:auth_path=/pklxkd-61/?prefix=obj148 2023-05-31T21:03:19.876 DEBUG:boto:Method: GET 2023-05-31T21:03:19.876 DEBUG:boto:Path: /pklxkd-61/?prefix=obj148 2023-05-31T21:03:19.877 DEBUG:boto:Data: 2023-05-31T21:03:19.877 DEBUG:boto:Headers: {} 2023-05-31T21:03:19.877 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T21:03:19.878 DEBUG:boto:Port: 8000 2023-05-31T21:03:19.878 DEBUG:boto:Params: {} 2023-05-31T21:03:19.878 DEBUG:boto:Token: None 2023-05-31T21:03:19.879 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 21:03:19 GMT /pklxkd-61/ 2023-05-31T21:03:19.879 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:hLB/3UScfc36qJjImbkC6+I98Sc= 2023-05-31T21:03:19.880 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 21:03:19 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:hLB/3UScfc36qJjImbkC6+I98Sc=', 'Content-Length': '0'} 2023-05-31T21:03:19.883 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000003ec7f1067622c312-006477b617-1630-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 21:03:19 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:03:19.883 DEBUG:boto:b'pklxkd-61obj1481000falseobj1482023-05-31T20:59:47.007Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T21:03:19.884 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj155 2023-05-31T21:03:19.885 DEBUG:boto:path=/pklxkd-61/obj155 2023-05-31T21:03:19.885 DEBUG:boto:auth_path=/pklxkd-61/obj155 2023-05-31T21:03:19.885 DEBUG:boto:path=/pklxkd-61/obj155?versionId=null 2023-05-31T21:03:19.886 DEBUG:boto:auth_path=/pklxkd-61/obj155?versionId=null 2023-05-31T21:03:19.886 DEBUG:boto:Method: GET 2023-05-31T21:03:19.886 DEBUG:boto:Path: /pklxkd-61/obj155?versionId=null 2023-05-31T21:03:19.887 DEBUG:boto:Data: 2023-05-31T21:03:19.887 DEBUG:boto:Headers: {} 2023-05-31T21:03:19.887 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T21:03:19.888 DEBUG:boto:Port: 8001 2023-05-31T21:03:19.888 DEBUG:boto:Params: {} 2023-05-31T21:03:19.889 DEBUG:boto:Token: None 2023-05-31T21:03:19.889 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 21:03:19 GMT /pklxkd-61/obj155?versionId=null 2023-05-31T21:03:19.890 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:DDVqqoChGvDt53t9HZg48e9F5tA= 2023-05-31T21:03:19.890 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 21:03:19 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:DDVqqoChGvDt53t9HZg48e9F5tA=', 'Content-Length': '0'} 2023-05-31T21:03:19.892 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:59:46 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009d84f5a512251f75-006477b617-1f4e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 21:03:19 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:03:19.932 DEBUG:boto:path=/pklxkd-61/obj155 2023-05-31T21:03:19.933 DEBUG:boto:auth_path=/pklxkd-61/obj155 2023-05-31T21:03:19.933 DEBUG:boto:path=/pklxkd-61/obj155?versionId=null 2023-05-31T21:03:19.933 DEBUG:boto:auth_path=/pklxkd-61/obj155?versionId=null 2023-05-31T21:03:19.934 DEBUG:boto:Method: GET 2023-05-31T21:03:19.934 DEBUG:boto:Path: /pklxkd-61/obj155?versionId=null 2023-05-31T21:03:19.934 DEBUG:boto:Data: 2023-05-31T21:03:19.935 DEBUG:boto:Headers: {} 2023-05-31T21:03:19.935 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T21:03:19.935 DEBUG:boto:Port: 8000 2023-05-31T21:03:19.936 DEBUG:boto:Params: {} 2023-05-31T21:03:19.936 DEBUG:boto:Token: None 2023-05-31T21:03:19.937 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 21:03:19 GMT /pklxkd-61/obj155?versionId=null 2023-05-31T21:03:19.937 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:DDVqqoChGvDt53t9HZg48e9F5tA= 2023-05-31T21:03:19.938 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 21:03:19 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:DDVqqoChGvDt53t9HZg48e9F5tA=', 'Content-Length': '0'} 2023-05-31T21:03:19.939 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:59:46 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a0b159cbb439c957-006477b617-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 21:03:19 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:03:19.940 DEBUG:boto:path=/pklxkd-61/obj155 2023-05-31T21:03:19.940 DEBUG:boto:auth_path=/pklxkd-61/obj155 2023-05-31T21:03:19.941 DEBUG:boto:path=/pklxkd-61/obj155?versionId=null 2023-05-31T21:03:19.941 DEBUG:boto:auth_path=/pklxkd-61/obj155?versionId=null 2023-05-31T21:03:19.941 DEBUG:boto:Method: HEAD 2023-05-31T21:03:19.942 DEBUG:boto:Path: /pklxkd-61/obj155?versionId=null 2023-05-31T21:03:19.942 DEBUG:boto:Data: 2023-05-31T21:03:19.942 DEBUG:boto:Headers: {} 2023-05-31T21:03:19.943 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T21:03:19.943 DEBUG:boto:Port: 8001 2023-05-31T21:03:19.943 DEBUG:boto:Params: {} 2023-05-31T21:03:19.944 DEBUG:boto:Token: None 2023-05-31T21:03:19.944 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 21:03:19 GMT /pklxkd-61/obj155?versionId=null 2023-05-31T21:03:19.945 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:skzdMFU7StXskkB4Z6isACEHJeE= 2023-05-31T21:03:19.945 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 21:03:19 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:skzdMFU7StXskkB4Z6isACEHJeE=', 'Content-Length': '0'} 2023-05-31T21:03:19.947 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:59:46 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000000473a8e878e1aa0-006477b617-1f4e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 21:03:19 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:03:19.947 DEBUG:boto:path=/pklxkd-61/obj155 2023-05-31T21:03:19.948 DEBUG:boto:auth_path=/pklxkd-61/obj155 2023-05-31T21:03:19.948 DEBUG:boto:path=/pklxkd-61/obj155?versionId=null 2023-05-31T21:03:19.948 DEBUG:boto:auth_path=/pklxkd-61/obj155?versionId=null 2023-05-31T21:03:19.949 DEBUG:boto:Method: HEAD 2023-05-31T21:03:19.949 DEBUG:boto:Path: /pklxkd-61/obj155?versionId=null 2023-05-31T21:03:19.949 DEBUG:boto:Data: 2023-05-31T21:03:19.950 DEBUG:boto:Headers: {} 2023-05-31T21:03:19.950 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T21:03:19.950 DEBUG:boto:Port: 8000 2023-05-31T21:03:19.951 DEBUG:boto:Params: {} 2023-05-31T21:03:19.951 DEBUG:boto:Token: None 2023-05-31T21:03:19.952 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 21:03:19 GMT /pklxkd-61/obj155?versionId=null 2023-05-31T21:03:19.952 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:skzdMFU7StXskkB4Z6isACEHJeE= 2023-05-31T21:03:19.952 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 21:03:19 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:skzdMFU7StXskkB4Z6isACEHJeE=', 'Content-Length': '0'} 2023-05-31T21:03:19.954 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:59:46 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000cb78f0f334456070-006477b617-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 21:03:19 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:03:19.954 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj155 2023-05-31T21:03:19.955 DEBUG:boto:path=/pklxkd-61/obj155 2023-05-31T21:03:19.955 DEBUG:boto:auth_path=/pklxkd-61/obj155 2023-05-31T21:03:19.956 DEBUG:boto:path=/pklxkd-61/obj155?versionId=null 2023-05-31T21:03:19.956 DEBUG:boto:auth_path=/pklxkd-61/obj155?versionId=null 2023-05-31T21:03:19.956 DEBUG:boto:Method: GET 2023-05-31T21:03:19.957 DEBUG:boto:Path: /pklxkd-61/obj155?versionId=null 2023-05-31T21:03:19.957 DEBUG:boto:Data: 2023-05-31T21:03:19.957 DEBUG:boto:Headers: {} 2023-05-31T21:03:19.958 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T21:03:19.958 DEBUG:boto:Port: 8001 2023-05-31T21:03:19.959 DEBUG:boto:Params: {} 2023-05-31T21:03:19.959 DEBUG:boto:Token: None 2023-05-31T21:03:19.959 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 21:03:19 GMT /pklxkd-61/obj155?versionId=null 2023-05-31T21:03:19.960 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:DDVqqoChGvDt53t9HZg48e9F5tA= 2023-05-31T21:03:19.960 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 21:03:19 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:DDVqqoChGvDt53t9HZg48e9F5tA=', 'Content-Length': '0'} 2023-05-31T21:03:19.962 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:59:46 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000046fbd3e5e23b003e-006477b617-1f4e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 21:03:19 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:03:19.962 DEBUG:boto:path=/pklxkd-61/obj155 2023-05-31T21:03:19.963 DEBUG:boto:auth_path=/pklxkd-61/obj155 2023-05-31T21:03:19.963 DEBUG:boto:path=/pklxkd-61/obj155?versionId=null 2023-05-31T21:03:19.964 DEBUG:boto:auth_path=/pklxkd-61/obj155?versionId=null 2023-05-31T21:03:19.964 DEBUG:boto:Method: GET 2023-05-31T21:03:19.965 DEBUG:boto:Path: /pklxkd-61/obj155?versionId=null 2023-05-31T21:03:19.965 DEBUG:boto:Data: 2023-05-31T21:03:19.965 DEBUG:boto:Headers: {} 2023-05-31T21:03:19.966 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T21:03:19.966 DEBUG:boto:Port: 8000 2023-05-31T21:03:19.966 DEBUG:boto:Params: {} 2023-05-31T21:03:19.967 DEBUG:boto:Token: None 2023-05-31T21:03:19.967 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 21:03:19 GMT /pklxkd-61/obj155?versionId=null 2023-05-31T21:03:19.967 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:DDVqqoChGvDt53t9HZg48e9F5tA= 2023-05-31T21:03:19.968 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 21:03:19 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:DDVqqoChGvDt53t9HZg48e9F5tA=', 'Content-Length': '0'} 2023-05-31T21:03:19.970 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:59:46 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001fb2cf9b29ecf79d-006477b617-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 21:03:19 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:03:19.970 DEBUG:boto:path=/pklxkd-61/ 2023-05-31T21:03:19.971 DEBUG:boto:auth_path=/pklxkd-61/ 2023-05-31T21:03:19.971 DEBUG:boto:path=/pklxkd-61/?prefix=obj155 2023-05-31T21:03:19.972 DEBUG:boto:auth_path=/pklxkd-61/?prefix=obj155 2023-05-31T21:03:19.972 DEBUG:boto:Method: GET 2023-05-31T21:03:19.972 DEBUG:boto:Path: /pklxkd-61/?prefix=obj155 2023-05-31T21:03:19.973 DEBUG:boto:Data: 2023-05-31T21:03:19.973 DEBUG:boto:Headers: {} 2023-05-31T21:03:19.973 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T21:03:19.974 DEBUG:boto:Port: 8001 2023-05-31T21:03:19.974 DEBUG:boto:Params: {} 2023-05-31T21:03:19.975 DEBUG:boto:Token: None 2023-05-31T21:03:19.975 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 21:03:19 GMT /pklxkd-61/ 2023-05-31T21:03:19.975 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:hLB/3UScfc36qJjImbkC6+I98Sc= 2023-05-31T21:03:19.976 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 21:03:19 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:hLB/3UScfc36qJjImbkC6+I98Sc=', 'Content-Length': '0'} 2023-05-31T21:03:19.978 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000bf3be0c0eb548c22-006477b617-1f4e-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 21:03:19 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:03:19.978 DEBUG:boto:b'pklxkd-61obj1551000falseobj1552023-05-31T20:59:46.438Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T21:03:19.979 DEBUG:boto:path=/pklxkd-61/ 2023-05-31T21:03:19.979 DEBUG:boto:auth_path=/pklxkd-61/ 2023-05-31T21:03:19.980 DEBUG:boto:path=/pklxkd-61/?prefix=obj155 2023-05-31T21:03:19.980 DEBUG:boto:auth_path=/pklxkd-61/?prefix=obj155 2023-05-31T21:03:19.981 DEBUG:boto:Method: GET 2023-05-31T21:03:19.981 DEBUG:boto:Path: /pklxkd-61/?prefix=obj155 2023-05-31T21:03:19.981 DEBUG:boto:Data: 2023-05-31T21:03:19.982 DEBUG:boto:Headers: {} 2023-05-31T21:03:19.982 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T21:03:19.982 DEBUG:boto:Port: 8000 2023-05-31T21:03:19.983 DEBUG:boto:Params: {} 2023-05-31T21:03:19.983 DEBUG:boto:Token: None 2023-05-31T21:03:19.983 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 21:03:19 GMT /pklxkd-61/ 2023-05-31T21:03:19.984 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:hLB/3UScfc36qJjImbkC6+I98Sc= 2023-05-31T21:03:19.984 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 21:03:19 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:hLB/3UScfc36qJjImbkC6+I98Sc=', 'Content-Length': '0'} 2023-05-31T21:03:19.988 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000037bf00695a6cd779-006477b617-1630-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 21:03:19 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:03:19.988 DEBUG:boto:b'pklxkd-61obj1551000falseobj1552023-05-31T20:59:46.438Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T21:03:19.989 DEBUG:boto:path=/pklxkd-61/obj155 2023-05-31T21:03:19.989 DEBUG:boto:auth_path=/pklxkd-61/obj155 2023-05-31T21:03:19.990 DEBUG:boto:Method: HEAD 2023-05-31T21:03:19.990 DEBUG:boto:Path: /pklxkd-61/obj155 2023-05-31T21:03:19.990 DEBUG:boto:Data: 2023-05-31T21:03:19.991 DEBUG:boto:Headers: {} 2023-05-31T21:03:19.991 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T21:03:19.991 DEBUG:boto:Port: 8001 2023-05-31T21:03:19.992 DEBUG:boto:Params: {} 2023-05-31T21:03:19.992 DEBUG:boto:Token: None 2023-05-31T21:03:19.993 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 21:03:19 GMT /pklxkd-61/obj155 2023-05-31T21:03:19.993 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:GCNKzs/gwfgHJO9YQ+uQXjllv2Q= 2023-05-31T21:03:19.993 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 21:03:19 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:GCNKzs/gwfgHJO9YQ+uQXjllv2Q=', 'Content-Length': '0'} 2023-05-31T21:03:19.995 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:59:46 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008c2fae7cec14c278-006477b617-1f4e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 21:03:19 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:03:19.996 DEBUG:boto:path=/pklxkd-61/obj155 2023-05-31T21:03:19.996 DEBUG:boto:auth_path=/pklxkd-61/obj155 2023-05-31T21:03:19.996 DEBUG:boto:Method: HEAD 2023-05-31T21:03:19.997 DEBUG:boto:Path: /pklxkd-61/obj155 2023-05-31T21:03:19.997 DEBUG:boto:Data: 2023-05-31T21:03:19.998 DEBUG:boto:Headers: {} 2023-05-31T21:03:19.998 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T21:03:19.998 DEBUG:boto:Port: 8000 2023-05-31T21:03:19.999 DEBUG:boto:Params: {} 2023-05-31T21:03:19.999 DEBUG:boto:Token: None 2023-05-31T21:03:19.999 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 21:03:19 GMT /pklxkd-61/obj155 2023-05-31T21:03:20.000 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:GCNKzs/gwfgHJO9YQ+uQXjllv2Q= 2023-05-31T21:03:20.000 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 21:03:19 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:GCNKzs/gwfgHJO9YQ+uQXjllv2Q=', 'Content-Length': '0'} 2023-05-31T21:03:20.002 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:59:46 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000be597439cc581734-006477b617-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 21:03:19 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:03:20.002 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj155 2023-05-31T21:03:20.003 DEBUG:boto:path=/pklxkd-61/obj155 2023-05-31T21:03:20.003 DEBUG:boto:auth_path=/pklxkd-61/obj155 2023-05-31T21:03:20.003 DEBUG:boto:Method: GET 2023-05-31T21:03:20.004 DEBUG:boto:Path: /pklxkd-61/obj155 2023-05-31T21:03:20.004 DEBUG:boto:Data: 2023-05-31T21:03:20.005 DEBUG:boto:Headers: {} 2023-05-31T21:03:20.005 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T21:03:20.005 DEBUG:boto:Port: 8001 2023-05-31T21:03:20.006 DEBUG:boto:Params: {} 2023-05-31T21:03:20.006 DEBUG:boto:Token: None 2023-05-31T21:03:20.007 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 21:03:20 GMT /pklxkd-61/obj155 2023-05-31T21:03:20.007 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:YdG1iLjVC2kpViBqvQRWWP8ff0I= 2023-05-31T21:03:20.007 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 21:03:20 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:YdG1iLjVC2kpViBqvQRWWP8ff0I=', 'Content-Length': '0'} 2023-05-31T21:03:20.009 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:59:46 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f3fb3d2616edeb61-006477b618-1f4e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 21:03:20 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:03:20.010 DEBUG:boto:path=/pklxkd-61/obj155 2023-05-31T21:03:20.010 DEBUG:boto:auth_path=/pklxkd-61/obj155 2023-05-31T21:03:20.010 DEBUG:boto:Method: GET 2023-05-31T21:03:20.011 DEBUG:boto:Path: /pklxkd-61/obj155 2023-05-31T21:03:20.011 DEBUG:boto:Data: 2023-05-31T21:03:20.012 DEBUG:boto:Headers: {} 2023-05-31T21:03:20.012 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T21:03:20.012 DEBUG:boto:Port: 8000 2023-05-31T21:03:20.013 DEBUG:boto:Params: {} 2023-05-31T21:03:20.013 DEBUG:boto:Token: None 2023-05-31T21:03:20.014 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 21:03:20 GMT /pklxkd-61/obj155 2023-05-31T21:03:20.015 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:YdG1iLjVC2kpViBqvQRWWP8ff0I= 2023-05-31T21:03:20.016 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 21:03:20 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:YdG1iLjVC2kpViBqvQRWWP8ff0I=', 'Content-Length': '0'} 2023-05-31T21:03:20.018 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:59:46 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000425a8db104afd2d2-006477b618-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 21:03:20 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:03:20.019 DEBUG:boto:path=/pklxkd-61/ 2023-05-31T21:03:20.019 DEBUG:boto:auth_path=/pklxkd-61/ 2023-05-31T21:03:20.020 DEBUG:boto:path=/pklxkd-61/?prefix=obj155 2023-05-31T21:03:20.020 DEBUG:boto:auth_path=/pklxkd-61/?prefix=obj155 2023-05-31T21:03:20.021 DEBUG:boto:Method: GET 2023-05-31T21:03:20.021 DEBUG:boto:Path: /pklxkd-61/?prefix=obj155 2023-05-31T21:03:20.021 DEBUG:boto:Data: 2023-05-31T21:03:20.022 DEBUG:boto:Headers: {} 2023-05-31T21:03:20.022 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T21:03:20.022 DEBUG:boto:Port: 8001 2023-05-31T21:03:20.023 DEBUG:boto:Params: {} 2023-05-31T21:03:20.023 DEBUG:boto:Token: None 2023-05-31T21:03:20.024 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 21:03:20 GMT /pklxkd-61/ 2023-05-31T21:03:20.024 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:2L5YtQy7Tf5tqdFPiN197mxB4Yg= 2023-05-31T21:03:20.025 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 21:03:20 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:2L5YtQy7Tf5tqdFPiN197mxB4Yg=', 'Content-Length': '0'} 2023-05-31T21:03:20.027 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000005f298918432a92ca-006477b618-1f4e-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 21:03:20 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:03:20.027 DEBUG:boto:b'pklxkd-61obj1551000falseobj1552023-05-31T20:59:46.438Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T21:03:20.028 DEBUG:boto:path=/pklxkd-61/ 2023-05-31T21:03:20.029 DEBUG:boto:auth_path=/pklxkd-61/ 2023-05-31T21:03:20.029 DEBUG:boto:path=/pklxkd-61/?prefix=obj155 2023-05-31T21:03:20.029 DEBUG:boto:auth_path=/pklxkd-61/?prefix=obj155 2023-05-31T21:03:20.030 DEBUG:boto:Method: GET 2023-05-31T21:03:20.030 DEBUG:boto:Path: /pklxkd-61/?prefix=obj155 2023-05-31T21:03:20.030 DEBUG:boto:Data: 2023-05-31T21:03:20.031 DEBUG:boto:Headers: {} 2023-05-31T21:03:20.031 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T21:03:20.031 DEBUG:boto:Port: 8000 2023-05-31T21:03:20.032 DEBUG:boto:Params: {} 2023-05-31T21:03:20.032 DEBUG:boto:Token: None 2023-05-31T21:03:20.033 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 21:03:20 GMT /pklxkd-61/ 2023-05-31T21:03:20.033 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:2L5YtQy7Tf5tqdFPiN197mxB4Yg= 2023-05-31T21:03:20.033 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 21:03:20 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:2L5YtQy7Tf5tqdFPiN197mxB4Yg=', 'Content-Length': '0'} 2023-05-31T21:03:20.037 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000dec12cf03dd983ff-006477b618-1630-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 21:03:20 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:03:20.037 DEBUG:boto:b'pklxkd-61obj1551000falseobj1552023-05-31T20:59:46.438Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T21:03:20.038 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj17 2023-05-31T21:03:20.038 DEBUG:boto:path=/pklxkd-61/obj17 2023-05-31T21:03:20.039 DEBUG:boto:auth_path=/pklxkd-61/obj17 2023-05-31T21:03:20.039 DEBUG:boto:path=/pklxkd-61/obj17?versionId=null 2023-05-31T21:03:20.039 DEBUG:boto:auth_path=/pklxkd-61/obj17?versionId=null 2023-05-31T21:03:20.040 DEBUG:boto:Method: GET 2023-05-31T21:03:20.040 DEBUG:boto:Path: /pklxkd-61/obj17?versionId=null 2023-05-31T21:03:20.041 DEBUG:boto:Data: 2023-05-31T21:03:20.041 DEBUG:boto:Headers: {} 2023-05-31T21:03:20.041 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T21:03:20.042 DEBUG:boto:Port: 8001 2023-05-31T21:03:20.042 DEBUG:boto:Params: {} 2023-05-31T21:03:20.042 DEBUG:boto:Token: None 2023-05-31T21:03:20.043 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 21:03:20 GMT /pklxkd-61/obj17?versionId=null 2023-05-31T21:03:20.043 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:EGrgPqms4/WOoNmXNttYK2n4kxo= 2023-05-31T21:03:20.044 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 21:03:20 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:EGrgPqms4/WOoNmXNttYK2n4kxo=', 'Content-Length': '0'} 2023-05-31T21:03:20.045 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:59:44 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000065f5f845d727056a-006477b618-1f4e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 21:03:20 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:03:20.046 DEBUG:boto:path=/pklxkd-61/obj17 2023-05-31T21:03:20.046 DEBUG:boto:auth_path=/pklxkd-61/obj17 2023-05-31T21:03:20.047 DEBUG:boto:path=/pklxkd-61/obj17?versionId=null 2023-05-31T21:03:20.047 DEBUG:boto:auth_path=/pklxkd-61/obj17?versionId=null 2023-05-31T21:03:20.047 DEBUG:boto:Method: GET 2023-05-31T21:03:20.048 DEBUG:boto:Path: /pklxkd-61/obj17?versionId=null 2023-05-31T21:03:20.048 DEBUG:boto:Data: 2023-05-31T21:03:20.049 DEBUG:boto:Headers: {} 2023-05-31T21:03:20.049 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T21:03:20.049 DEBUG:boto:Port: 8000 2023-05-31T21:03:20.050 DEBUG:boto:Params: {} 2023-05-31T21:03:20.050 DEBUG:boto:Token: None 2023-05-31T21:03:20.050 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 21:03:20 GMT /pklxkd-61/obj17?versionId=null 2023-05-31T21:03:20.051 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:EGrgPqms4/WOoNmXNttYK2n4kxo= 2023-05-31T21:03:20.051 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 21:03:20 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:EGrgPqms4/WOoNmXNttYK2n4kxo=', 'Content-Length': '0'} 2023-05-31T21:03:20.054 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:59:44 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e2c1a3602297147a-006477b618-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 21:03:20 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:03:20.100 DEBUG:boto:path=/pklxkd-61/obj17 2023-05-31T21:03:20.101 DEBUG:boto:auth_path=/pklxkd-61/obj17 2023-05-31T21:03:20.101 DEBUG:boto:path=/pklxkd-61/obj17?versionId=null 2023-05-31T21:03:20.101 DEBUG:boto:auth_path=/pklxkd-61/obj17?versionId=null 2023-05-31T21:03:20.102 DEBUG:boto:Method: HEAD 2023-05-31T21:03:20.102 DEBUG:boto:Path: /pklxkd-61/obj17?versionId=null 2023-05-31T21:03:20.102 DEBUG:boto:Data: 2023-05-31T21:03:20.103 DEBUG:boto:Headers: {} 2023-05-31T21:03:20.103 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T21:03:20.103 DEBUG:boto:Port: 8001 2023-05-31T21:03:20.104 DEBUG:boto:Params: {} 2023-05-31T21:03:20.104 DEBUG:boto:Token: None 2023-05-31T21:03:20.105 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 21:03:20 GMT /pklxkd-61/obj17?versionId=null 2023-05-31T21:03:20.105 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:6YYlA0pXSUS3RWsCnsuIpneWruw= 2023-05-31T21:03:20.106 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 21:03:20 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:6YYlA0pXSUS3RWsCnsuIpneWruw=', 'Content-Length': '0'} 2023-05-31T21:03:20.107 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:59:44 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000cc5489c952f552d3-006477b618-1f4e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 21:03:20 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:03:20.108 DEBUG:boto:path=/pklxkd-61/obj17 2023-05-31T21:03:20.108 DEBUG:boto:auth_path=/pklxkd-61/obj17 2023-05-31T21:03:20.109 DEBUG:boto:path=/pklxkd-61/obj17?versionId=null 2023-05-31T21:03:20.109 DEBUG:boto:auth_path=/pklxkd-61/obj17?versionId=null 2023-05-31T21:03:20.109 DEBUG:boto:Method: HEAD 2023-05-31T21:03:20.110 DEBUG:boto:Path: /pklxkd-61/obj17?versionId=null 2023-05-31T21:03:20.110 DEBUG:boto:Data: 2023-05-31T21:03:20.110 DEBUG:boto:Headers: {} 2023-05-31T21:03:20.111 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T21:03:20.111 DEBUG:boto:Port: 8000 2023-05-31T21:03:20.111 DEBUG:boto:Params: {} 2023-05-31T21:03:20.112 DEBUG:boto:Token: None 2023-05-31T21:03:20.112 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 21:03:20 GMT /pklxkd-61/obj17?versionId=null 2023-05-31T21:03:20.113 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:6YYlA0pXSUS3RWsCnsuIpneWruw= 2023-05-31T21:03:20.113 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 21:03:20 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:6YYlA0pXSUS3RWsCnsuIpneWruw=', 'Content-Length': '0'} 2023-05-31T21:03:20.115 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:59:44 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ce89addfa7c90572-006477b618-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 21:03:20 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:03:20.115 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj17 2023-05-31T21:03:20.115 DEBUG:boto:path=/pklxkd-61/obj17 2023-05-31T21:03:20.116 DEBUG:boto:auth_path=/pklxkd-61/obj17 2023-05-31T21:03:20.116 DEBUG:boto:path=/pklxkd-61/obj17?versionId=null 2023-05-31T21:03:20.117 DEBUG:boto:auth_path=/pklxkd-61/obj17?versionId=null 2023-05-31T21:03:20.117 DEBUG:boto:Method: GET 2023-05-31T21:03:20.117 DEBUG:boto:Path: /pklxkd-61/obj17?versionId=null 2023-05-31T21:03:20.118 DEBUG:boto:Data: 2023-05-31T21:03:20.118 DEBUG:boto:Headers: {} 2023-05-31T21:03:20.119 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T21:03:20.120 DEBUG:boto:Port: 8001 2023-05-31T21:03:20.120 DEBUG:boto:Params: {} 2023-05-31T21:03:20.121 DEBUG:boto:Token: None 2023-05-31T21:03:20.122 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 21:03:20 GMT /pklxkd-61/obj17?versionId=null 2023-05-31T21:03:20.122 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:EGrgPqms4/WOoNmXNttYK2n4kxo= 2023-05-31T21:03:20.123 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 21:03:20 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:EGrgPqms4/WOoNmXNttYK2n4kxo=', 'Content-Length': '0'} 2023-05-31T21:03:20.124 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:59:44 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000050f781c50ef4ab97-006477b618-1f4e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 21:03:20 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:03:20.125 DEBUG:boto:path=/pklxkd-61/obj17 2023-05-31T21:03:20.125 DEBUG:boto:auth_path=/pklxkd-61/obj17 2023-05-31T21:03:20.126 DEBUG:boto:path=/pklxkd-61/obj17?versionId=null 2023-05-31T21:03:20.126 DEBUG:boto:auth_path=/pklxkd-61/obj17?versionId=null 2023-05-31T21:03:20.126 DEBUG:boto:Method: GET 2023-05-31T21:03:20.127 DEBUG:boto:Path: /pklxkd-61/obj17?versionId=null 2023-05-31T21:03:20.127 DEBUG:boto:Data: 2023-05-31T21:03:20.127 DEBUG:boto:Headers: {} 2023-05-31T21:03:20.128 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T21:03:20.128 DEBUG:boto:Port: 8000 2023-05-31T21:03:20.129 DEBUG:boto:Params: {} 2023-05-31T21:03:20.129 DEBUG:boto:Token: None 2023-05-31T21:03:20.130 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 21:03:20 GMT /pklxkd-61/obj17?versionId=null 2023-05-31T21:03:20.131 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:EGrgPqms4/WOoNmXNttYK2n4kxo= 2023-05-31T21:03:20.132 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 21:03:20 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:EGrgPqms4/WOoNmXNttYK2n4kxo=', 'Content-Length': '0'} 2023-05-31T21:03:20.134 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:59:44 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b565b535fb23cadd-006477b618-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 21:03:20 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:03:20.135 DEBUG:boto:path=/pklxkd-61/ 2023-05-31T21:03:20.136 DEBUG:boto:auth_path=/pklxkd-61/ 2023-05-31T21:03:20.136 DEBUG:boto:path=/pklxkd-61/?prefix=obj17 2023-05-31T21:03:20.137 DEBUG:boto:auth_path=/pklxkd-61/?prefix=obj17 2023-05-31T21:03:20.137 DEBUG:boto:Method: GET 2023-05-31T21:03:20.137 DEBUG:boto:Path: /pklxkd-61/?prefix=obj17 2023-05-31T21:03:20.138 DEBUG:boto:Data: 2023-05-31T21:03:20.138 DEBUG:boto:Headers: {} 2023-05-31T21:03:20.138 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T21:03:20.139 DEBUG:boto:Port: 8001 2023-05-31T21:03:20.139 DEBUG:boto:Params: {} 2023-05-31T21:03:20.139 DEBUG:boto:Token: None 2023-05-31T21:03:20.140 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 21:03:20 GMT /pklxkd-61/ 2023-05-31T21:03:20.140 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:2L5YtQy7Tf5tqdFPiN197mxB4Yg= 2023-05-31T21:03:20.141 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 21:03:20 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:2L5YtQy7Tf5tqdFPiN197mxB4Yg=', 'Content-Length': '0'} 2023-05-31T21:03:20.143 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b53f623fd39bed9a-006477b618-1f4e-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 21:03:20 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:03:20.143 DEBUG:boto:b'pklxkd-61obj171000falseobj172023-05-31T20:59:44.769Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T21:03:20.144 DEBUG:boto:path=/pklxkd-61/ 2023-05-31T21:03:20.145 DEBUG:boto:auth_path=/pklxkd-61/ 2023-05-31T21:03:20.145 DEBUG:boto:path=/pklxkd-61/?prefix=obj17 2023-05-31T21:03:20.145 DEBUG:boto:auth_path=/pklxkd-61/?prefix=obj17 2023-05-31T21:03:20.146 DEBUG:boto:Method: GET 2023-05-31T21:03:20.146 DEBUG:boto:Path: /pklxkd-61/?prefix=obj17 2023-05-31T21:03:20.146 DEBUG:boto:Data: 2023-05-31T21:03:20.147 DEBUG:boto:Headers: {} 2023-05-31T21:03:20.147 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T21:03:20.147 DEBUG:boto:Port: 8000 2023-05-31T21:03:20.148 DEBUG:boto:Params: {} 2023-05-31T21:03:20.148 DEBUG:boto:Token: None 2023-05-31T21:03:20.149 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 21:03:20 GMT /pklxkd-61/ 2023-05-31T21:03:20.149 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:2L5YtQy7Tf5tqdFPiN197mxB4Yg= 2023-05-31T21:03:20.149 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 21:03:20 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:2L5YtQy7Tf5tqdFPiN197mxB4Yg=', 'Content-Length': '0'} 2023-05-31T21:03:20.153 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000eacf5a45fb25ad39-006477b618-1630-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 21:03:20 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:03:20.153 DEBUG:boto:b'pklxkd-61obj171000falseobj172023-05-31T20:59:44.769Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T21:03:20.154 DEBUG:boto:path=/pklxkd-61/obj17 2023-05-31T21:03:20.154 DEBUG:boto:auth_path=/pklxkd-61/obj17 2023-05-31T21:03:20.154 DEBUG:boto:Method: HEAD 2023-05-31T21:03:20.155 DEBUG:boto:Path: /pklxkd-61/obj17 2023-05-31T21:03:20.155 DEBUG:boto:Data: 2023-05-31T21:03:20.155 DEBUG:boto:Headers: {} 2023-05-31T21:03:20.156 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T21:03:20.156 DEBUG:boto:Port: 8001 2023-05-31T21:03:20.157 DEBUG:boto:Params: {} 2023-05-31T21:03:20.157 DEBUG:boto:Token: None 2023-05-31T21:03:20.157 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 21:03:20 GMT /pklxkd-61/obj17 2023-05-31T21:03:20.158 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:FiKAeNXO8w31GpF03WmdvMnNi6U= 2023-05-31T21:03:20.158 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 21:03:20 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:FiKAeNXO8w31GpF03WmdvMnNi6U=', 'Content-Length': '0'} 2023-05-31T21:03:20.160 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:59:44 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a87dc756115f5ae0-006477b618-1f4e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 21:03:20 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:03:20.160 DEBUG:boto:path=/pklxkd-61/obj17 2023-05-31T21:03:20.161 DEBUG:boto:auth_path=/pklxkd-61/obj17 2023-05-31T21:03:20.161 DEBUG:boto:Method: HEAD 2023-05-31T21:03:20.161 DEBUG:boto:Path: /pklxkd-61/obj17 2023-05-31T21:03:20.162 DEBUG:boto:Data: 2023-05-31T21:03:20.162 DEBUG:boto:Headers: {} 2023-05-31T21:03:20.162 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T21:03:20.163 DEBUG:boto:Port: 8000 2023-05-31T21:03:20.163 DEBUG:boto:Params: {} 2023-05-31T21:03:20.164 DEBUG:boto:Token: None 2023-05-31T21:03:20.164 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 21:03:20 GMT /pklxkd-61/obj17 2023-05-31T21:03:20.165 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:FiKAeNXO8w31GpF03WmdvMnNi6U= 2023-05-31T21:03:20.165 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 21:03:20 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:FiKAeNXO8w31GpF03WmdvMnNi6U=', 'Content-Length': '0'} 2023-05-31T21:03:20.167 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:59:44 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000733a710f36c15b6b-006477b618-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 21:03:20 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:03:20.167 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj17 2023-05-31T21:03:20.168 DEBUG:boto:path=/pklxkd-61/obj17 2023-05-31T21:03:20.168 DEBUG:boto:auth_path=/pklxkd-61/obj17 2023-05-31T21:03:20.168 DEBUG:boto:Method: GET 2023-05-31T21:03:20.169 DEBUG:boto:Path: /pklxkd-61/obj17 2023-05-31T21:03:20.169 DEBUG:boto:Data: 2023-05-31T21:03:20.170 DEBUG:boto:Headers: {} 2023-05-31T21:03:20.170 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T21:03:20.170 DEBUG:boto:Port: 8001 2023-05-31T21:03:20.171 DEBUG:boto:Params: {} 2023-05-31T21:03:20.171 DEBUG:boto:Token: None 2023-05-31T21:03:20.171 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 21:03:20 GMT /pklxkd-61/obj17 2023-05-31T21:03:20.172 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:euF9OwNiX+wWSfGyjBZESIrsv8k= 2023-05-31T21:03:20.172 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 21:03:20 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:euF9OwNiX+wWSfGyjBZESIrsv8k=', 'Content-Length': '0'} 2023-05-31T21:03:20.174 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:59:44 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002b9cc15ed244327e-006477b618-1f4e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 21:03:20 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:03:20.174 DEBUG:boto:path=/pklxkd-61/obj17 2023-05-31T21:03:20.175 DEBUG:boto:auth_path=/pklxkd-61/obj17 2023-05-31T21:03:20.175 DEBUG:boto:Method: GET 2023-05-31T21:03:20.176 DEBUG:boto:Path: /pklxkd-61/obj17 2023-05-31T21:03:20.176 DEBUG:boto:Data: 2023-05-31T21:03:20.176 DEBUG:boto:Headers: {} 2023-05-31T21:03:20.177 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T21:03:20.177 DEBUG:boto:Port: 8000 2023-05-31T21:03:20.177 DEBUG:boto:Params: {} 2023-05-31T21:03:20.178 DEBUG:boto:Token: None 2023-05-31T21:03:20.178 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 21:03:20 GMT /pklxkd-61/obj17 2023-05-31T21:03:20.179 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:euF9OwNiX+wWSfGyjBZESIrsv8k= 2023-05-31T21:03:20.179 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 21:03:20 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:euF9OwNiX+wWSfGyjBZESIrsv8k=', 'Content-Length': '0'} 2023-05-31T21:03:20.180 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:59:44 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000521124c3fec6476e-006477b618-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 21:03:20 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:03:20.181 DEBUG:boto:path=/pklxkd-61/ 2023-05-31T21:03:20.181 DEBUG:boto:auth_path=/pklxkd-61/ 2023-05-31T21:03:20.182 DEBUG:boto:path=/pklxkd-61/?prefix=obj17 2023-05-31T21:03:20.182 DEBUG:boto:auth_path=/pklxkd-61/?prefix=obj17 2023-05-31T21:03:20.183 DEBUG:boto:Method: GET 2023-05-31T21:03:20.183 DEBUG:boto:Path: /pklxkd-61/?prefix=obj17 2023-05-31T21:03:20.183 DEBUG:boto:Data: 2023-05-31T21:03:20.184 DEBUG:boto:Headers: {} 2023-05-31T21:03:20.184 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T21:03:20.184 DEBUG:boto:Port: 8001 2023-05-31T21:03:20.185 DEBUG:boto:Params: {} 2023-05-31T21:03:20.185 DEBUG:boto:Token: None 2023-05-31T21:03:20.186 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 21:03:20 GMT /pklxkd-61/ 2023-05-31T21:03:20.186 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:2L5YtQy7Tf5tqdFPiN197mxB4Yg= 2023-05-31T21:03:20.187 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 21:03:20 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:2L5YtQy7Tf5tqdFPiN197mxB4Yg=', 'Content-Length': '0'} 2023-05-31T21:03:20.189 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000005885ae46fed416eb-006477b618-1f4e-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 21:03:20 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:03:20.189 DEBUG:boto:b'pklxkd-61obj171000falseobj172023-05-31T20:59:44.769Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T21:03:20.190 DEBUG:boto:path=/pklxkd-61/ 2023-05-31T21:03:20.190 DEBUG:boto:auth_path=/pklxkd-61/ 2023-05-31T21:03:20.191 DEBUG:boto:path=/pklxkd-61/?prefix=obj17 2023-05-31T21:03:20.191 DEBUG:boto:auth_path=/pklxkd-61/?prefix=obj17 2023-05-31T21:03:20.191 DEBUG:boto:Method: GET 2023-05-31T21:03:20.192 DEBUG:boto:Path: /pklxkd-61/?prefix=obj17 2023-05-31T21:03:20.192 DEBUG:boto:Data: 2023-05-31T21:03:20.193 DEBUG:boto:Headers: {} 2023-05-31T21:03:20.193 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T21:03:20.193 DEBUG:boto:Port: 8000 2023-05-31T21:03:20.194 DEBUG:boto:Params: {} 2023-05-31T21:03:20.194 DEBUG:boto:Token: None 2023-05-31T21:03:20.194 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 21:03:20 GMT /pklxkd-61/ 2023-05-31T21:03:20.195 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:2L5YtQy7Tf5tqdFPiN197mxB4Yg= 2023-05-31T21:03:20.195 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 21:03:20 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:2L5YtQy7Tf5tqdFPiN197mxB4Yg=', 'Content-Length': '0'} 2023-05-31T21:03:20.199 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000bedbc8caa34abca4-006477b618-1630-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 21:03:20 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:03:20.240 DEBUG:boto:b'pklxkd-61obj171000falseobj172023-05-31T20:59:44.769Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T21:03:20.241 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj185 2023-05-31T21:03:20.241 DEBUG:boto:path=/pklxkd-61/obj185 2023-05-31T21:03:20.242 DEBUG:boto:auth_path=/pklxkd-61/obj185 2023-05-31T21:03:20.242 DEBUG:boto:path=/pklxkd-61/obj185?versionId=null 2023-05-31T21:03:20.242 DEBUG:boto:auth_path=/pklxkd-61/obj185?versionId=null 2023-05-31T21:03:20.243 DEBUG:boto:Method: GET 2023-05-31T21:03:20.243 DEBUG:boto:Path: /pklxkd-61/obj185?versionId=null 2023-05-31T21:03:20.244 DEBUG:boto:Data: 2023-05-31T21:03:20.244 DEBUG:boto:Headers: {} 2023-05-31T21:03:20.244 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T21:03:20.245 DEBUG:boto:Port: 8001 2023-05-31T21:03:20.245 DEBUG:boto:Params: {} 2023-05-31T21:03:20.245 DEBUG:boto:Token: None 2023-05-31T21:03:20.246 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 21:03:20 GMT /pklxkd-61/obj185?versionId=null 2023-05-31T21:03:20.246 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:LUDP0Sce5KnKvY4jgU6Cxap7Qus= 2023-05-31T21:03:20.247 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 21:03:20 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:LUDP0Sce5KnKvY4jgU6Cxap7Qus=', 'Content-Length': '0'} 2023-05-31T21:03:20.249 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:59:47 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003e9b2f40820a6da8-006477b618-1f4e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 21:03:20 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:03:20.249 DEBUG:boto:path=/pklxkd-61/obj185 2023-05-31T21:03:20.250 DEBUG:boto:auth_path=/pklxkd-61/obj185 2023-05-31T21:03:20.250 DEBUG:boto:path=/pklxkd-61/obj185?versionId=null 2023-05-31T21:03:20.251 DEBUG:boto:auth_path=/pklxkd-61/obj185?versionId=null 2023-05-31T21:03:20.251 DEBUG:boto:Method: GET 2023-05-31T21:03:20.251 DEBUG:boto:Path: /pklxkd-61/obj185?versionId=null 2023-05-31T21:03:20.252 DEBUG:boto:Data: 2023-05-31T21:03:20.252 DEBUG:boto:Headers: {} 2023-05-31T21:03:20.252 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T21:03:20.253 DEBUG:boto:Port: 8000 2023-05-31T21:03:20.253 DEBUG:boto:Params: {} 2023-05-31T21:03:20.253 DEBUG:boto:Token: None 2023-05-31T21:03:20.254 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 21:03:20 GMT /pklxkd-61/obj185?versionId=null 2023-05-31T21:03:20.254 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:LUDP0Sce5KnKvY4jgU6Cxap7Qus= 2023-05-31T21:03:20.255 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 21:03:20 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:LUDP0Sce5KnKvY4jgU6Cxap7Qus=', 'Content-Length': '0'} 2023-05-31T21:03:20.256 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:59:47 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000039601b1cc3a86b88-006477b618-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 21:03:20 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:03:20.257 DEBUG:boto:path=/pklxkd-61/obj185 2023-05-31T21:03:20.257 DEBUG:boto:auth_path=/pklxkd-61/obj185 2023-05-31T21:03:20.258 DEBUG:boto:path=/pklxkd-61/obj185?versionId=null 2023-05-31T21:03:20.258 DEBUG:boto:auth_path=/pklxkd-61/obj185?versionId=null 2023-05-31T21:03:20.259 DEBUG:boto:Method: HEAD 2023-05-31T21:03:20.259 DEBUG:boto:Path: /pklxkd-61/obj185?versionId=null 2023-05-31T21:03:20.259 DEBUG:boto:Data: 2023-05-31T21:03:20.260 DEBUG:boto:Headers: {} 2023-05-31T21:03:20.260 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T21:03:20.260 DEBUG:boto:Port: 8001 2023-05-31T21:03:20.261 DEBUG:boto:Params: {} 2023-05-31T21:03:20.261 DEBUG:boto:Token: None 2023-05-31T21:03:20.261 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 21:03:20 GMT /pklxkd-61/obj185?versionId=null 2023-05-31T21:03:20.262 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:BOznP+WwMDGg8tVmqOeotYDERNM= 2023-05-31T21:03:20.262 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 21:03:20 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:BOznP+WwMDGg8tVmqOeotYDERNM=', 'Content-Length': '0'} 2023-05-31T21:03:20.264 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:59:47 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000bedc92d10ae67606-006477b618-1f4e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 21:03:20 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:03:20.264 DEBUG:boto:path=/pklxkd-61/obj185 2023-05-31T21:03:20.265 DEBUG:boto:auth_path=/pklxkd-61/obj185 2023-05-31T21:03:20.265 DEBUG:boto:path=/pklxkd-61/obj185?versionId=null 2023-05-31T21:03:20.265 DEBUG:boto:auth_path=/pklxkd-61/obj185?versionId=null 2023-05-31T21:03:20.266 DEBUG:boto:Method: HEAD 2023-05-31T21:03:20.266 DEBUG:boto:Path: /pklxkd-61/obj185?versionId=null 2023-05-31T21:03:20.266 DEBUG:boto:Data: 2023-05-31T21:03:20.267 DEBUG:boto:Headers: {} 2023-05-31T21:03:20.267 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T21:03:20.268 DEBUG:boto:Port: 8000 2023-05-31T21:03:20.268 DEBUG:boto:Params: {} 2023-05-31T21:03:20.269 DEBUG:boto:Token: None 2023-05-31T21:03:20.269 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 21:03:20 GMT /pklxkd-61/obj185?versionId=null 2023-05-31T21:03:20.269 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:BOznP+WwMDGg8tVmqOeotYDERNM= 2023-05-31T21:03:20.270 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 21:03:20 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:BOznP+WwMDGg8tVmqOeotYDERNM=', 'Content-Length': '0'} 2023-05-31T21:03:20.271 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:59:47 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c60a47078154ba4b-006477b618-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 21:03:20 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:03:20.272 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj185 2023-05-31T21:03:20.272 DEBUG:boto:path=/pklxkd-61/obj185 2023-05-31T21:03:20.272 DEBUG:boto:auth_path=/pklxkd-61/obj185 2023-05-31T21:03:20.273 DEBUG:boto:path=/pklxkd-61/obj185?versionId=null 2023-05-31T21:03:20.273 DEBUG:boto:auth_path=/pklxkd-61/obj185?versionId=null 2023-05-31T21:03:20.273 DEBUG:boto:Method: GET 2023-05-31T21:03:20.274 DEBUG:boto:Path: /pklxkd-61/obj185?versionId=null 2023-05-31T21:03:20.274 DEBUG:boto:Data: 2023-05-31T21:03:20.275 DEBUG:boto:Headers: {} 2023-05-31T21:03:20.275 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T21:03:20.275 DEBUG:boto:Port: 8001 2023-05-31T21:03:20.276 DEBUG:boto:Params: {} 2023-05-31T21:03:20.276 DEBUG:boto:Token: None 2023-05-31T21:03:20.277 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 21:03:20 GMT /pklxkd-61/obj185?versionId=null 2023-05-31T21:03:20.277 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:LUDP0Sce5KnKvY4jgU6Cxap7Qus= 2023-05-31T21:03:20.277 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 21:03:20 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:LUDP0Sce5KnKvY4jgU6Cxap7Qus=', 'Content-Length': '0'} 2023-05-31T21:03:20.279 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:59:47 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000233738a86ff4fffa-006477b618-1f4e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 21:03:20 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:03:20.320 DEBUG:boto:path=/pklxkd-61/obj185 2023-05-31T21:03:20.321 DEBUG:boto:auth_path=/pklxkd-61/obj185 2023-05-31T21:03:20.321 DEBUG:boto:path=/pklxkd-61/obj185?versionId=null 2023-05-31T21:03:20.321 DEBUG:boto:auth_path=/pklxkd-61/obj185?versionId=null 2023-05-31T21:03:20.322 DEBUG:boto:Method: GET 2023-05-31T21:03:20.322 DEBUG:boto:Path: /pklxkd-61/obj185?versionId=null 2023-05-31T21:03:20.323 DEBUG:boto:Data: 2023-05-31T21:03:20.323 DEBUG:boto:Headers: {} 2023-05-31T21:03:20.323 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T21:03:20.324 DEBUG:boto:Port: 8000 2023-05-31T21:03:20.324 DEBUG:boto:Params: {} 2023-05-31T21:03:20.324 DEBUG:boto:Token: None 2023-05-31T21:03:20.325 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 21:03:20 GMT /pklxkd-61/obj185?versionId=null 2023-05-31T21:03:20.325 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:LUDP0Sce5KnKvY4jgU6Cxap7Qus= 2023-05-31T21:03:20.326 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 21:03:20 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:LUDP0Sce5KnKvY4jgU6Cxap7Qus=', 'Content-Length': '0'} 2023-05-31T21:03:20.327 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:59:47 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005e10fe1762f0e060-006477b618-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 21:03:20 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:03:20.328 DEBUG:boto:path=/pklxkd-61/ 2023-05-31T21:03:20.328 DEBUG:boto:auth_path=/pklxkd-61/ 2023-05-31T21:03:20.329 DEBUG:boto:path=/pklxkd-61/?prefix=obj185 2023-05-31T21:03:20.329 DEBUG:boto:auth_path=/pklxkd-61/?prefix=obj185 2023-05-31T21:03:20.330 DEBUG:boto:Method: GET 2023-05-31T21:03:20.330 DEBUG:boto:Path: /pklxkd-61/?prefix=obj185 2023-05-31T21:03:20.330 DEBUG:boto:Data: 2023-05-31T21:03:20.331 DEBUG:boto:Headers: {} 2023-05-31T21:03:20.331 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T21:03:20.331 DEBUG:boto:Port: 8001 2023-05-31T21:03:20.332 DEBUG:boto:Params: {} 2023-05-31T21:03:20.332 DEBUG:boto:Token: None 2023-05-31T21:03:20.333 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 21:03:20 GMT /pklxkd-61/ 2023-05-31T21:03:20.333 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:2L5YtQy7Tf5tqdFPiN197mxB4Yg= 2023-05-31T21:03:20.333 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 21:03:20 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:2L5YtQy7Tf5tqdFPiN197mxB4Yg=', 'Content-Length': '0'} 2023-05-31T21:03:20.335 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b8f3abf7a7726897-006477b618-1f4e-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 21:03:20 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:03:20.336 DEBUG:boto:b'pklxkd-61obj1851000falseobj1852023-05-31T20:59:47.031Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T21:03:20.336 DEBUG:boto:path=/pklxkd-61/ 2023-05-31T21:03:20.337 DEBUG:boto:auth_path=/pklxkd-61/ 2023-05-31T21:03:20.337 DEBUG:boto:path=/pklxkd-61/?prefix=obj185 2023-05-31T21:03:20.338 DEBUG:boto:auth_path=/pklxkd-61/?prefix=obj185 2023-05-31T21:03:20.338 DEBUG:boto:Method: GET 2023-05-31T21:03:20.338 DEBUG:boto:Path: /pklxkd-61/?prefix=obj185 2023-05-31T21:03:20.339 DEBUG:boto:Data: 2023-05-31T21:03:20.339 DEBUG:boto:Headers: {} 2023-05-31T21:03:20.339 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T21:03:20.340 DEBUG:boto:Port: 8000 2023-05-31T21:03:20.340 DEBUG:boto:Params: {} 2023-05-31T21:03:20.341 DEBUG:boto:Token: None 2023-05-31T21:03:20.341 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 21:03:20 GMT /pklxkd-61/ 2023-05-31T21:03:20.341 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:2L5YtQy7Tf5tqdFPiN197mxB4Yg= 2023-05-31T21:03:20.342 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 21:03:20 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:2L5YtQy7Tf5tqdFPiN197mxB4Yg=', 'Content-Length': '0'} 2023-05-31T21:03:20.345 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000cafbcd6e06ec0c7e-006477b618-1630-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 21:03:20 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:03:20.346 DEBUG:boto:b'pklxkd-61obj1851000falseobj1852023-05-31T20:59:47.031Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T21:03:20.346 DEBUG:boto:path=/pklxkd-61/obj185 2023-05-31T21:03:20.347 DEBUG:boto:auth_path=/pklxkd-61/obj185 2023-05-31T21:03:20.347 DEBUG:boto:Method: HEAD 2023-05-31T21:03:20.348 DEBUG:boto:Path: /pklxkd-61/obj185 2023-05-31T21:03:20.348 DEBUG:boto:Data: 2023-05-31T21:03:20.349 DEBUG:boto:Headers: {} 2023-05-31T21:03:20.349 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T21:03:20.349 DEBUG:boto:Port: 8001 2023-05-31T21:03:20.350 DEBUG:boto:Params: {} 2023-05-31T21:03:20.351 DEBUG:boto:Token: None 2023-05-31T21:03:20.351 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 21:03:20 GMT /pklxkd-61/obj185 2023-05-31T21:03:20.352 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:gOcO7TeBogCu+sZPobVr/moBhAc= 2023-05-31T21:03:20.353 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 21:03:20 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:gOcO7TeBogCu+sZPobVr/moBhAc=', 'Content-Length': '0'} 2023-05-31T21:03:20.354 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:59:47 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006b779785c31cd390-006477b618-1f4e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 21:03:20 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:03:20.355 DEBUG:boto:path=/pklxkd-61/obj185 2023-05-31T21:03:20.355 DEBUG:boto:auth_path=/pklxkd-61/obj185 2023-05-31T21:03:20.356 DEBUG:boto:Method: HEAD 2023-05-31T21:03:20.356 DEBUG:boto:Path: /pklxkd-61/obj185 2023-05-31T21:03:20.356 DEBUG:boto:Data: 2023-05-31T21:03:20.357 DEBUG:boto:Headers: {} 2023-05-31T21:03:20.357 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T21:03:20.357 DEBUG:boto:Port: 8000 2023-05-31T21:03:20.358 DEBUG:boto:Params: {} 2023-05-31T21:03:20.358 DEBUG:boto:Token: None 2023-05-31T21:03:20.358 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 21:03:20 GMT /pklxkd-61/obj185 2023-05-31T21:03:20.359 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:gOcO7TeBogCu+sZPobVr/moBhAc= 2023-05-31T21:03:20.359 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 21:03:20 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:gOcO7TeBogCu+sZPobVr/moBhAc=', 'Content-Length': '0'} 2023-05-31T21:03:20.361 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:59:47 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000081965d99e75eeb55-006477b618-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 21:03:20 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:03:20.361 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj185 2023-05-31T21:03:20.362 DEBUG:boto:path=/pklxkd-61/obj185 2023-05-31T21:03:20.362 DEBUG:boto:auth_path=/pklxkd-61/obj185 2023-05-31T21:03:20.362 DEBUG:boto:Method: GET 2023-05-31T21:03:20.363 DEBUG:boto:Path: /pklxkd-61/obj185 2023-05-31T21:03:20.363 DEBUG:boto:Data: 2023-05-31T21:03:20.363 DEBUG:boto:Headers: {} 2023-05-31T21:03:20.364 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T21:03:20.364 DEBUG:boto:Port: 8001 2023-05-31T21:03:20.364 DEBUG:boto:Params: {} 2023-05-31T21:03:20.365 DEBUG:boto:Token: None 2023-05-31T21:03:20.365 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 21:03:20 GMT /pklxkd-61/obj185 2023-05-31T21:03:20.366 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:KQRIp2ojgde5qVYK5OnwI3e8QxA= 2023-05-31T21:03:20.366 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 21:03:20 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:KQRIp2ojgde5qVYK5OnwI3e8QxA=', 'Content-Length': '0'} 2023-05-31T21:03:20.368 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:59:47 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a2c47c7d1882a747-006477b618-1f4e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 21:03:20 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:03:20.368 DEBUG:boto:path=/pklxkd-61/obj185 2023-05-31T21:03:20.369 DEBUG:boto:auth_path=/pklxkd-61/obj185 2023-05-31T21:03:20.369 DEBUG:boto:Method: GET 2023-05-31T21:03:20.370 DEBUG:boto:Path: /pklxkd-61/obj185 2023-05-31T21:03:20.370 DEBUG:boto:Data: 2023-05-31T21:03:20.370 DEBUG:boto:Headers: {} 2023-05-31T21:03:20.371 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T21:03:20.371 DEBUG:boto:Port: 8000 2023-05-31T21:03:20.371 DEBUG:boto:Params: {} 2023-05-31T21:03:20.372 DEBUG:boto:Token: None 2023-05-31T21:03:20.372 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 21:03:20 GMT /pklxkd-61/obj185 2023-05-31T21:03:20.373 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:KQRIp2ojgde5qVYK5OnwI3e8QxA= 2023-05-31T21:03:20.373 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 21:03:20 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:KQRIp2ojgde5qVYK5OnwI3e8QxA=', 'Content-Length': '0'} 2023-05-31T21:03:20.375 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:59:47 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007d3e688aa38b21e4-006477b618-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 21:03:20 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:03:20.420 DEBUG:boto:path=/pklxkd-61/ 2023-05-31T21:03:20.421 DEBUG:boto:auth_path=/pklxkd-61/ 2023-05-31T21:03:20.421 DEBUG:boto:path=/pklxkd-61/?prefix=obj185 2023-05-31T21:03:20.421 DEBUG:boto:auth_path=/pklxkd-61/?prefix=obj185 2023-05-31T21:03:20.422 DEBUG:boto:Method: GET 2023-05-31T21:03:20.422 DEBUG:boto:Path: /pklxkd-61/?prefix=obj185 2023-05-31T21:03:20.423 DEBUG:boto:Data: 2023-05-31T21:03:20.423 DEBUG:boto:Headers: {} 2023-05-31T21:03:20.423 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T21:03:20.424 DEBUG:boto:Port: 8001 2023-05-31T21:03:20.424 DEBUG:boto:Params: {} 2023-05-31T21:03:20.425 DEBUG:boto:Token: None 2023-05-31T21:03:20.425 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 21:03:20 GMT /pklxkd-61/ 2023-05-31T21:03:20.426 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:2L5YtQy7Tf5tqdFPiN197mxB4Yg= 2023-05-31T21:03:20.426 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 21:03:20 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:2L5YtQy7Tf5tqdFPiN197mxB4Yg=', 'Content-Length': '0'} 2023-05-31T21:03:20.428 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000000c121a84c0cdc5d4-006477b618-1f4e-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 21:03:20 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:03:20.429 DEBUG:boto:b'pklxkd-61obj1851000falseobj1852023-05-31T20:59:47.031Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T21:03:20.429 DEBUG:boto:path=/pklxkd-61/ 2023-05-31T21:03:20.430 DEBUG:boto:auth_path=/pklxkd-61/ 2023-05-31T21:03:20.430 DEBUG:boto:path=/pklxkd-61/?prefix=obj185 2023-05-31T21:03:20.430 DEBUG:boto:auth_path=/pklxkd-61/?prefix=obj185 2023-05-31T21:03:20.431 DEBUG:boto:Method: GET 2023-05-31T21:03:20.431 DEBUG:boto:Path: /pklxkd-61/?prefix=obj185 2023-05-31T21:03:20.431 DEBUG:boto:Data: 2023-05-31T21:03:20.432 DEBUG:boto:Headers: {} 2023-05-31T21:03:20.432 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T21:03:20.433 DEBUG:boto:Port: 8000 2023-05-31T21:03:20.433 DEBUG:boto:Params: {} 2023-05-31T21:03:20.433 DEBUG:boto:Token: None 2023-05-31T21:03:20.434 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 21:03:20 GMT /pklxkd-61/ 2023-05-31T21:03:20.434 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:2L5YtQy7Tf5tqdFPiN197mxB4Yg= 2023-05-31T21:03:20.435 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 21:03:20 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:2L5YtQy7Tf5tqdFPiN197mxB4Yg=', 'Content-Length': '0'} 2023-05-31T21:03:20.438 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000091976e71934c6230-006477b618-1630-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 21:03:20 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:03:20.439 DEBUG:boto:b'pklxkd-61obj1851000falseobj1852023-05-31T20:59:47.031Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T21:03:20.439 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj19 2023-05-31T21:03:20.440 DEBUG:boto:path=/pklxkd-61/obj19 2023-05-31T21:03:20.440 DEBUG:boto:auth_path=/pklxkd-61/obj19 2023-05-31T21:03:20.440 DEBUG:boto:path=/pklxkd-61/obj19?versionId=null 2023-05-31T21:03:20.441 DEBUG:boto:auth_path=/pklxkd-61/obj19?versionId=null 2023-05-31T21:03:20.441 DEBUG:boto:Method: GET 2023-05-31T21:03:20.442 DEBUG:boto:Path: /pklxkd-61/obj19?versionId=null 2023-05-31T21:03:20.442 DEBUG:boto:Data: 2023-05-31T21:03:20.442 DEBUG:boto:Headers: {} 2023-05-31T21:03:20.443 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T21:03:20.443 DEBUG:boto:Port: 8001 2023-05-31T21:03:20.443 DEBUG:boto:Params: {} 2023-05-31T21:03:20.444 DEBUG:boto:Token: None 2023-05-31T21:03:20.444 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 21:03:20 GMT /pklxkd-61/obj19?versionId=null 2023-05-31T21:03:20.445 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:tQjGM6S6DcXdhY0+75pu1L+ZA2g= 2023-05-31T21:03:20.445 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 21:03:20 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:tQjGM6S6DcXdhY0+75pu1L+ZA2g=', 'Content-Length': '0'} 2023-05-31T21:03:20.447 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:59:45 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a0cd636a9540ee6a-006477b618-1f4e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 21:03:20 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:03:20.447 DEBUG:boto:path=/pklxkd-61/obj19 2023-05-31T21:03:20.448 DEBUG:boto:auth_path=/pklxkd-61/obj19 2023-05-31T21:03:20.448 DEBUG:boto:path=/pklxkd-61/obj19?versionId=null 2023-05-31T21:03:20.449 DEBUG:boto:auth_path=/pklxkd-61/obj19?versionId=null 2023-05-31T21:03:20.449 DEBUG:boto:Method: GET 2023-05-31T21:03:20.449 DEBUG:boto:Path: /pklxkd-61/obj19?versionId=null 2023-05-31T21:03:20.450 DEBUG:boto:Data: 2023-05-31T21:03:20.450 DEBUG:boto:Headers: {} 2023-05-31T21:03:20.450 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T21:03:20.451 DEBUG:boto:Port: 8000 2023-05-31T21:03:20.451 DEBUG:boto:Params: {} 2023-05-31T21:03:20.451 DEBUG:boto:Token: None 2023-05-31T21:03:20.452 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 21:03:20 GMT /pklxkd-61/obj19?versionId=null 2023-05-31T21:03:20.452 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:tQjGM6S6DcXdhY0+75pu1L+ZA2g= 2023-05-31T21:03:20.453 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 21:03:20 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:tQjGM6S6DcXdhY0+75pu1L+ZA2g=', 'Content-Length': '0'} 2023-05-31T21:03:20.455 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:59:45 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f7045ce822d16292-006477b618-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 21:03:20 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:03:20.455 DEBUG:boto:path=/pklxkd-61/obj19 2023-05-31T21:03:20.456 DEBUG:boto:auth_path=/pklxkd-61/obj19 2023-05-31T21:03:20.456 DEBUG:boto:path=/pklxkd-61/obj19?versionId=null 2023-05-31T21:03:20.456 DEBUG:boto:auth_path=/pklxkd-61/obj19?versionId=null 2023-05-31T21:03:20.457 DEBUG:boto:Method: HEAD 2023-05-31T21:03:20.457 DEBUG:boto:Path: /pklxkd-61/obj19?versionId=null 2023-05-31T21:03:20.457 DEBUG:boto:Data: 2023-05-31T21:03:20.458 DEBUG:boto:Headers: {} 2023-05-31T21:03:20.458 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T21:03:20.458 DEBUG:boto:Port: 8001 2023-05-31T21:03:20.459 DEBUG:boto:Params: {} 2023-05-31T21:03:20.459 DEBUG:boto:Token: None 2023-05-31T21:03:20.460 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 21:03:20 GMT /pklxkd-61/obj19?versionId=null 2023-05-31T21:03:20.460 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:LZ0lL2DyqQf7Igpb5IpeDl/mcTQ= 2023-05-31T21:03:20.460 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 21:03:20 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:LZ0lL2DyqQf7Igpb5IpeDl/mcTQ=', 'Content-Length': '0'} 2023-05-31T21:03:20.462 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:59:45 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000722170eb9a74e3e2-006477b618-1f4e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 21:03:20 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:03:20.463 DEBUG:boto:path=/pklxkd-61/obj19 2023-05-31T21:03:20.463 DEBUG:boto:auth_path=/pklxkd-61/obj19 2023-05-31T21:03:20.464 DEBUG:boto:path=/pklxkd-61/obj19?versionId=null 2023-05-31T21:03:20.464 DEBUG:boto:auth_path=/pklxkd-61/obj19?versionId=null 2023-05-31T21:03:20.464 DEBUG:boto:Method: HEAD 2023-05-31T21:03:20.465 DEBUG:boto:Path: /pklxkd-61/obj19?versionId=null 2023-05-31T21:03:20.466 DEBUG:boto:Data: 2023-05-31T21:03:20.466 DEBUG:boto:Headers: {} 2023-05-31T21:03:20.467 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T21:03:20.467 DEBUG:boto:Port: 8000 2023-05-31T21:03:20.468 DEBUG:boto:Params: {} 2023-05-31T21:03:20.468 DEBUG:boto:Token: None 2023-05-31T21:03:20.469 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 21:03:20 GMT /pklxkd-61/obj19?versionId=null 2023-05-31T21:03:20.469 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:LZ0lL2DyqQf7Igpb5IpeDl/mcTQ= 2023-05-31T21:03:20.470 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 21:03:20 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:LZ0lL2DyqQf7Igpb5IpeDl/mcTQ=', 'Content-Length': '0'} 2023-05-31T21:03:20.472 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:59:45 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000057f1b53b9b9d3680-006477b618-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 21:03:20 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:03:20.472 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj19 2023-05-31T21:03:20.473 DEBUG:boto:path=/pklxkd-61/obj19 2023-05-31T21:03:20.473 DEBUG:boto:auth_path=/pklxkd-61/obj19 2023-05-31T21:03:20.474 DEBUG:boto:path=/pklxkd-61/obj19?versionId=null 2023-05-31T21:03:20.474 DEBUG:boto:auth_path=/pklxkd-61/obj19?versionId=null 2023-05-31T21:03:20.474 DEBUG:boto:Method: GET 2023-05-31T21:03:20.475 DEBUG:boto:Path: /pklxkd-61/obj19?versionId=null 2023-05-31T21:03:20.475 DEBUG:boto:Data: 2023-05-31T21:03:20.476 DEBUG:boto:Headers: {} 2023-05-31T21:03:20.476 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T21:03:20.477 DEBUG:boto:Port: 8001 2023-05-31T21:03:20.477 DEBUG:boto:Params: {} 2023-05-31T21:03:20.478 DEBUG:boto:Token: None 2023-05-31T21:03:20.478 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 21:03:20 GMT /pklxkd-61/obj19?versionId=null 2023-05-31T21:03:20.479 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:tQjGM6S6DcXdhY0+75pu1L+ZA2g= 2023-05-31T21:03:20.479 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 21:03:20 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:tQjGM6S6DcXdhY0+75pu1L+ZA2g=', 'Content-Length': '0'} 2023-05-31T21:03:20.481 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:59:45 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f3f39e312e629987-006477b618-1f4e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 21:03:20 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:03:20.482 DEBUG:boto:path=/pklxkd-61/obj19 2023-05-31T21:03:20.482 DEBUG:boto:auth_path=/pklxkd-61/obj19 2023-05-31T21:03:20.482 DEBUG:boto:path=/pklxkd-61/obj19?versionId=null 2023-05-31T21:03:20.483 DEBUG:boto:auth_path=/pklxkd-61/obj19?versionId=null 2023-05-31T21:03:20.483 DEBUG:boto:Method: GET 2023-05-31T21:03:20.483 DEBUG:boto:Path: /pklxkd-61/obj19?versionId=null 2023-05-31T21:03:20.484 DEBUG:boto:Data: 2023-05-31T21:03:20.484 DEBUG:boto:Headers: {} 2023-05-31T21:03:20.485 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T21:03:20.485 DEBUG:boto:Port: 8000 2023-05-31T21:03:20.485 DEBUG:boto:Params: {} 2023-05-31T21:03:20.486 DEBUG:boto:Token: None 2023-05-31T21:03:20.486 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 21:03:20 GMT /pklxkd-61/obj19?versionId=null 2023-05-31T21:03:20.487 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:tQjGM6S6DcXdhY0+75pu1L+ZA2g= 2023-05-31T21:03:20.487 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 21:03:20 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:tQjGM6S6DcXdhY0+75pu1L+ZA2g=', 'Content-Length': '0'} 2023-05-31T21:03:20.489 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:59:45 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a39db41e7ec46155-006477b618-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 21:03:20 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:03:20.489 DEBUG:boto:path=/pklxkd-61/ 2023-05-31T21:03:20.490 DEBUG:boto:auth_path=/pklxkd-61/ 2023-05-31T21:03:20.490 DEBUG:boto:path=/pklxkd-61/?prefix=obj19 2023-05-31T21:03:20.491 DEBUG:boto:auth_path=/pklxkd-61/?prefix=obj19 2023-05-31T21:03:20.491 DEBUG:boto:Method: GET 2023-05-31T21:03:20.491 DEBUG:boto:Path: /pklxkd-61/?prefix=obj19 2023-05-31T21:03:20.492 DEBUG:boto:Data: 2023-05-31T21:03:20.492 DEBUG:boto:Headers: {} 2023-05-31T21:03:20.492 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T21:03:20.493 DEBUG:boto:Port: 8001 2023-05-31T21:03:20.493 DEBUG:boto:Params: {} 2023-05-31T21:03:20.494 DEBUG:boto:Token: None 2023-05-31T21:03:20.494 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 21:03:20 GMT /pklxkd-61/ 2023-05-31T21:03:20.495 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:2L5YtQy7Tf5tqdFPiN197mxB4Yg= 2023-05-31T21:03:20.495 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 21:03:20 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:2L5YtQy7Tf5tqdFPiN197mxB4Yg=', 'Content-Length': '0'} 2023-05-31T21:03:20.497 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000071a907bfd9d53a54-006477b618-1f4e-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 21:03:20 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:03:20.498 DEBUG:boto:b'pklxkd-61obj191000falseobj192023-05-31T20:59:45.164Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T21:03:20.499 DEBUG:boto:path=/pklxkd-61/ 2023-05-31T21:03:20.500 DEBUG:boto:auth_path=/pklxkd-61/ 2023-05-31T21:03:20.500 DEBUG:boto:path=/pklxkd-61/?prefix=obj19 2023-05-31T21:03:20.501 DEBUG:boto:auth_path=/pklxkd-61/?prefix=obj19 2023-05-31T21:03:20.502 DEBUG:boto:Method: GET 2023-05-31T21:03:20.503 DEBUG:boto:Path: /pklxkd-61/?prefix=obj19 2023-05-31T21:03:20.504 DEBUG:boto:Data: 2023-05-31T21:03:20.504 DEBUG:boto:Headers: {} 2023-05-31T21:03:20.505 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T21:03:20.506 DEBUG:boto:Port: 8000 2023-05-31T21:03:20.507 DEBUG:boto:Params: {} 2023-05-31T21:03:20.507 DEBUG:boto:Token: None 2023-05-31T21:03:20.507 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 21:03:20 GMT /pklxkd-61/ 2023-05-31T21:03:20.508 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:2L5YtQy7Tf5tqdFPiN197mxB4Yg= 2023-05-31T21:03:20.508 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 21:03:20 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:2L5YtQy7Tf5tqdFPiN197mxB4Yg=', 'Content-Length': '0'} 2023-05-31T21:03:20.512 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000009f041e2a12a2e670-006477b618-1630-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 21:03:20 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:03:20.552 DEBUG:boto:b'pklxkd-61obj191000falseobj192023-05-31T20:59:45.164Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T21:03:20.553 DEBUG:boto:path=/pklxkd-61/obj19 2023-05-31T21:03:20.553 DEBUG:boto:auth_path=/pklxkd-61/obj19 2023-05-31T21:03:20.554 DEBUG:boto:Method: HEAD 2023-05-31T21:03:20.554 DEBUG:boto:Path: /pklxkd-61/obj19 2023-05-31T21:03:20.554 DEBUG:boto:Data: 2023-05-31T21:03:20.555 DEBUG:boto:Headers: {} 2023-05-31T21:03:20.555 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T21:03:20.555 DEBUG:boto:Port: 8001 2023-05-31T21:03:20.556 DEBUG:boto:Params: {} 2023-05-31T21:03:20.556 DEBUG:boto:Token: None 2023-05-31T21:03:20.557 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 21:03:20 GMT /pklxkd-61/obj19 2023-05-31T21:03:20.557 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:zmvs9aS3luaco4xBj/b3gcLD5as= 2023-05-31T21:03:20.557 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 21:03:20 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:zmvs9aS3luaco4xBj/b3gcLD5as=', 'Content-Length': '0'} 2023-05-31T21:03:20.559 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:59:45 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002bb22d36cea4a55e-006477b618-1f4e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 21:03:20 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:03:20.560 DEBUG:boto:path=/pklxkd-61/obj19 2023-05-31T21:03:20.560 DEBUG:boto:auth_path=/pklxkd-61/obj19 2023-05-31T21:03:20.560 DEBUG:boto:Method: HEAD 2023-05-31T21:03:20.561 DEBUG:boto:Path: /pklxkd-61/obj19 2023-05-31T21:03:20.561 DEBUG:boto:Data: 2023-05-31T21:03:20.561 DEBUG:boto:Headers: {} 2023-05-31T21:03:20.562 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T21:03:20.562 DEBUG:boto:Port: 8000 2023-05-31T21:03:20.562 DEBUG:boto:Params: {} 2023-05-31T21:03:20.563 DEBUG:boto:Token: None 2023-05-31T21:03:20.563 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 21:03:20 GMT /pklxkd-61/obj19 2023-05-31T21:03:20.564 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:zmvs9aS3luaco4xBj/b3gcLD5as= 2023-05-31T21:03:20.564 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 21:03:20 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:zmvs9aS3luaco4xBj/b3gcLD5as=', 'Content-Length': '0'} 2023-05-31T21:03:20.566 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:59:45 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000114ef4e8e82659c1-006477b618-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 21:03:20 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:03:20.566 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj19 2023-05-31T21:03:20.566 DEBUG:boto:path=/pklxkd-61/obj19 2023-05-31T21:03:20.567 DEBUG:boto:auth_path=/pklxkd-61/obj19 2023-05-31T21:03:20.567 DEBUG:boto:Method: GET 2023-05-31T21:03:20.568 DEBUG:boto:Path: /pklxkd-61/obj19 2023-05-31T21:03:20.568 DEBUG:boto:Data: 2023-05-31T21:03:20.569 DEBUG:boto:Headers: {} 2023-05-31T21:03:20.569 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T21:03:20.569 DEBUG:boto:Port: 8001 2023-05-31T21:03:20.570 DEBUG:boto:Params: {} 2023-05-31T21:03:20.570 DEBUG:boto:Token: None 2023-05-31T21:03:20.571 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 21:03:20 GMT /pklxkd-61/obj19 2023-05-31T21:03:20.571 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:6NX5uNe32JSzS547LdJw3Yo1FlA= 2023-05-31T21:03:20.571 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 21:03:20 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:6NX5uNe32JSzS547LdJw3Yo1FlA=', 'Content-Length': '0'} 2023-05-31T21:03:20.573 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:59:45 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000016d3c905f91686cc-006477b618-1f4e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 21:03:20 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:03:20.574 DEBUG:boto:path=/pklxkd-61/obj19 2023-05-31T21:03:20.574 DEBUG:boto:auth_path=/pklxkd-61/obj19 2023-05-31T21:03:20.575 DEBUG:boto:Method: GET 2023-05-31T21:03:20.575 DEBUG:boto:Path: /pklxkd-61/obj19 2023-05-31T21:03:20.575 DEBUG:boto:Data: 2023-05-31T21:03:20.576 DEBUG:boto:Headers: {} 2023-05-31T21:03:20.576 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T21:03:20.577 DEBUG:boto:Port: 8000 2023-05-31T21:03:20.577 DEBUG:boto:Params: {} 2023-05-31T21:03:20.577 DEBUG:boto:Token: None 2023-05-31T21:03:20.578 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 21:03:20 GMT /pklxkd-61/obj19 2023-05-31T21:03:20.578 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:6NX5uNe32JSzS547LdJw3Yo1FlA= 2023-05-31T21:03:20.579 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 21:03:20 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:6NX5uNe32JSzS547LdJw3Yo1FlA=', 'Content-Length': '0'} 2023-05-31T21:03:20.581 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:59:45 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ca70d601c2240112-006477b618-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 21:03:20 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:03:20.582 DEBUG:boto:path=/pklxkd-61/ 2023-05-31T21:03:20.582 DEBUG:boto:auth_path=/pklxkd-61/ 2023-05-31T21:03:20.583 DEBUG:boto:path=/pklxkd-61/?prefix=obj19 2023-05-31T21:03:20.583 DEBUG:boto:auth_path=/pklxkd-61/?prefix=obj19 2023-05-31T21:03:20.584 DEBUG:boto:Method: GET 2023-05-31T21:03:20.585 DEBUG:boto:Path: /pklxkd-61/?prefix=obj19 2023-05-31T21:03:20.585 DEBUG:boto:Data: 2023-05-31T21:03:20.585 DEBUG:boto:Headers: {} 2023-05-31T21:03:20.586 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T21:03:20.586 DEBUG:boto:Port: 8001 2023-05-31T21:03:20.587 DEBUG:boto:Params: {} 2023-05-31T21:03:20.587 DEBUG:boto:Token: None 2023-05-31T21:03:20.588 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 21:03:20 GMT /pklxkd-61/ 2023-05-31T21:03:20.588 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:2L5YtQy7Tf5tqdFPiN197mxB4Yg= 2023-05-31T21:03:20.588 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 21:03:20 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:2L5YtQy7Tf5tqdFPiN197mxB4Yg=', 'Content-Length': '0'} 2023-05-31T21:03:20.591 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000002290ac1185df584f-006477b618-1f4e-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 21:03:20 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:03:20.591 DEBUG:boto:b'pklxkd-61obj191000falseobj192023-05-31T20:59:45.164Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T21:03:20.592 DEBUG:boto:path=/pklxkd-61/ 2023-05-31T21:03:20.592 DEBUG:boto:auth_path=/pklxkd-61/ 2023-05-31T21:03:20.593 DEBUG:boto:path=/pklxkd-61/?prefix=obj19 2023-05-31T21:03:20.593 DEBUG:boto:auth_path=/pklxkd-61/?prefix=obj19 2023-05-31T21:03:20.593 DEBUG:boto:Method: GET 2023-05-31T21:03:20.594 DEBUG:boto:Path: /pklxkd-61/?prefix=obj19 2023-05-31T21:03:20.594 DEBUG:boto:Data: 2023-05-31T21:03:20.594 DEBUG:boto:Headers: {} 2023-05-31T21:03:20.595 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T21:03:20.595 DEBUG:boto:Port: 8000 2023-05-31T21:03:20.596 DEBUG:boto:Params: {} 2023-05-31T21:03:20.596 DEBUG:boto:Token: None 2023-05-31T21:03:20.596 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 21:03:20 GMT /pklxkd-61/ 2023-05-31T21:03:20.597 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:2L5YtQy7Tf5tqdFPiN197mxB4Yg= 2023-05-31T21:03:20.597 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 21:03:20 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:2L5YtQy7Tf5tqdFPiN197mxB4Yg=', 'Content-Length': '0'} 2023-05-31T21:03:20.601 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000003eb797aa13aeda6a-006477b618-1630-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 21:03:20 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:03:20.601 DEBUG:boto:b'pklxkd-61obj191000falseobj192023-05-31T20:59:45.164Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T21:03:20.602 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj22 2023-05-31T21:03:20.602 DEBUG:boto:path=/pklxkd-61/obj22 2023-05-31T21:03:20.603 DEBUG:boto:auth_path=/pklxkd-61/obj22 2023-05-31T21:03:20.603 DEBUG:boto:path=/pklxkd-61/obj22?versionId=null 2023-05-31T21:03:20.603 DEBUG:boto:auth_path=/pklxkd-61/obj22?versionId=null 2023-05-31T21:03:20.604 DEBUG:boto:Method: GET 2023-05-31T21:03:20.604 DEBUG:boto:Path: /pklxkd-61/obj22?versionId=null 2023-05-31T21:03:20.605 DEBUG:boto:Data: 2023-05-31T21:03:20.605 DEBUG:boto:Headers: {} 2023-05-31T21:03:20.605 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T21:03:20.606 DEBUG:boto:Port: 8001 2023-05-31T21:03:20.606 DEBUG:boto:Params: {} 2023-05-31T21:03:20.606 DEBUG:boto:Token: None 2023-05-31T21:03:20.607 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 21:03:20 GMT /pklxkd-61/obj22?versionId=null 2023-05-31T21:03:20.607 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:6FtHkTc06XGmtW2oFa0PYnNVOII= 2023-05-31T21:03:20.608 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 21:03:20 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:6FtHkTc06XGmtW2oFa0PYnNVOII=', 'Content-Length': '0'} 2023-05-31T21:03:20.609 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:59:44 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000493eecc15f58c629-006477b618-1f4e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 21:03:20 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:03:20.610 DEBUG:boto:path=/pklxkd-61/obj22 2023-05-31T21:03:20.610 DEBUG:boto:auth_path=/pklxkd-61/obj22 2023-05-31T21:03:20.611 DEBUG:boto:path=/pklxkd-61/obj22?versionId=null 2023-05-31T21:03:20.611 DEBUG:boto:auth_path=/pklxkd-61/obj22?versionId=null 2023-05-31T21:03:20.611 DEBUG:boto:Method: GET 2023-05-31T21:03:20.612 DEBUG:boto:Path: /pklxkd-61/obj22?versionId=null 2023-05-31T21:03:20.612 DEBUG:boto:Data: 2023-05-31T21:03:20.613 DEBUG:boto:Headers: {} 2023-05-31T21:03:20.613 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T21:03:20.613 DEBUG:boto:Port: 8000 2023-05-31T21:03:20.614 DEBUG:boto:Params: {} 2023-05-31T21:03:20.614 DEBUG:boto:Token: None 2023-05-31T21:03:20.614 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 21:03:20 GMT /pklxkd-61/obj22?versionId=null 2023-05-31T21:03:20.615 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:6FtHkTc06XGmtW2oFa0PYnNVOII= 2023-05-31T21:03:20.615 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 21:03:20 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:6FtHkTc06XGmtW2oFa0PYnNVOII=', 'Content-Length': '0'} 2023-05-31T21:03:20.617 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:59:44 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a45833f624189cdf-006477b618-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 21:03:20 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:03:20.660 DEBUG:boto:path=/pklxkd-61/obj22 2023-05-31T21:03:20.661 DEBUG:boto:auth_path=/pklxkd-61/obj22 2023-05-31T21:03:20.661 DEBUG:boto:path=/pklxkd-61/obj22?versionId=null 2023-05-31T21:03:20.661 DEBUG:boto:auth_path=/pklxkd-61/obj22?versionId=null 2023-05-31T21:03:20.662 DEBUG:boto:Method: HEAD 2023-05-31T21:03:20.662 DEBUG:boto:Path: /pklxkd-61/obj22?versionId=null 2023-05-31T21:03:20.662 DEBUG:boto:Data: 2023-05-31T21:03:20.663 DEBUG:boto:Headers: {} 2023-05-31T21:03:20.663 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T21:03:20.663 DEBUG:boto:Port: 8001 2023-05-31T21:03:20.663 DEBUG:boto:Params: {} 2023-05-31T21:03:20.664 DEBUG:boto:Token: None 2023-05-31T21:03:20.664 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 21:03:20 GMT /pklxkd-61/obj22?versionId=null 2023-05-31T21:03:20.665 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:OsxEuXK2buJfFaxxMvtYvO/oOUk= 2023-05-31T21:03:20.665 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 21:03:20 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:OsxEuXK2buJfFaxxMvtYvO/oOUk=', 'Content-Length': '0'} 2023-05-31T21:03:20.667 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:59:44 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005a7b664e4d072199-006477b618-1f4e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 21:03:20 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:03:20.667 DEBUG:boto:path=/pklxkd-61/obj22 2023-05-31T21:03:20.668 DEBUG:boto:auth_path=/pklxkd-61/obj22 2023-05-31T21:03:20.668 DEBUG:boto:path=/pklxkd-61/obj22?versionId=null 2023-05-31T21:03:20.669 DEBUG:boto:auth_path=/pklxkd-61/obj22?versionId=null 2023-05-31T21:03:20.669 DEBUG:boto:Method: HEAD 2023-05-31T21:03:20.669 DEBUG:boto:Path: /pklxkd-61/obj22?versionId=null 2023-05-31T21:03:20.670 DEBUG:boto:Data: 2023-05-31T21:03:20.670 DEBUG:boto:Headers: {} 2023-05-31T21:03:20.670 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T21:03:20.671 DEBUG:boto:Port: 8000 2023-05-31T21:03:20.671 DEBUG:boto:Params: {} 2023-05-31T21:03:20.671 DEBUG:boto:Token: None 2023-05-31T21:03:20.672 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 21:03:20 GMT /pklxkd-61/obj22?versionId=null 2023-05-31T21:03:20.672 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:OsxEuXK2buJfFaxxMvtYvO/oOUk= 2023-05-31T21:03:20.672 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 21:03:20 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:OsxEuXK2buJfFaxxMvtYvO/oOUk=', 'Content-Length': '0'} 2023-05-31T21:03:20.674 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:59:44 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000fd3723d5e94ab3ba-006477b618-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 21:03:20 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:03:20.675 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj22 2023-05-31T21:03:20.675 DEBUG:boto:path=/pklxkd-61/obj22 2023-05-31T21:03:20.676 DEBUG:boto:auth_path=/pklxkd-61/obj22 2023-05-31T21:03:20.676 DEBUG:boto:path=/pklxkd-61/obj22?versionId=null 2023-05-31T21:03:20.676 DEBUG:boto:auth_path=/pklxkd-61/obj22?versionId=null 2023-05-31T21:03:20.677 DEBUG:boto:Method: GET 2023-05-31T21:03:20.677 DEBUG:boto:Path: /pklxkd-61/obj22?versionId=null 2023-05-31T21:03:20.677 DEBUG:boto:Data: 2023-05-31T21:03:20.678 DEBUG:boto:Headers: {} 2023-05-31T21:03:20.678 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T21:03:20.678 DEBUG:boto:Port: 8001 2023-05-31T21:03:20.679 DEBUG:boto:Params: {} 2023-05-31T21:03:20.679 DEBUG:boto:Token: None 2023-05-31T21:03:20.680 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 21:03:20 GMT /pklxkd-61/obj22?versionId=null 2023-05-31T21:03:20.680 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:6FtHkTc06XGmtW2oFa0PYnNVOII= 2023-05-31T21:03:20.680 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 21:03:20 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:6FtHkTc06XGmtW2oFa0PYnNVOII=', 'Content-Length': '0'} 2023-05-31T21:03:20.682 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:59:44 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000642753e0134b8db9-006477b618-1f4e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 21:03:20 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:03:20.683 DEBUG:boto:path=/pklxkd-61/obj22 2023-05-31T21:03:20.683 DEBUG:boto:auth_path=/pklxkd-61/obj22 2023-05-31T21:03:20.684 DEBUG:boto:path=/pklxkd-61/obj22?versionId=null 2023-05-31T21:03:20.684 DEBUG:boto:auth_path=/pklxkd-61/obj22?versionId=null 2023-05-31T21:03:20.684 DEBUG:boto:Method: GET 2023-05-31T21:03:20.685 DEBUG:boto:Path: /pklxkd-61/obj22?versionId=null 2023-05-31T21:03:20.685 DEBUG:boto:Data: 2023-05-31T21:03:20.686 DEBUG:boto:Headers: {} 2023-05-31T21:03:20.686 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T21:03:20.686 DEBUG:boto:Port: 8000 2023-05-31T21:03:20.687 DEBUG:boto:Params: {} 2023-05-31T21:03:20.687 DEBUG:boto:Token: None 2023-05-31T21:03:20.688 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 21:03:20 GMT /pklxkd-61/obj22?versionId=null 2023-05-31T21:03:20.688 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:6FtHkTc06XGmtW2oFa0PYnNVOII= 2023-05-31T21:03:20.689 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 21:03:20 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:6FtHkTc06XGmtW2oFa0PYnNVOII=', 'Content-Length': '0'} 2023-05-31T21:03:20.690 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:59:44 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000055c5eb8092ba3338-006477b618-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 21:03:20 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:03:20.691 DEBUG:boto:path=/pklxkd-61/ 2023-05-31T21:03:20.691 DEBUG:boto:auth_path=/pklxkd-61/ 2023-05-31T21:03:20.692 DEBUG:boto:path=/pklxkd-61/?prefix=obj22 2023-05-31T21:03:20.692 DEBUG:boto:auth_path=/pklxkd-61/?prefix=obj22 2023-05-31T21:03:20.692 DEBUG:boto:Method: GET 2023-05-31T21:03:20.693 DEBUG:boto:Path: /pklxkd-61/?prefix=obj22 2023-05-31T21:03:20.693 DEBUG:boto:Data: 2023-05-31T21:03:20.694 DEBUG:boto:Headers: {} 2023-05-31T21:03:20.694 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T21:03:20.695 DEBUG:boto:Port: 8001 2023-05-31T21:03:20.695 DEBUG:boto:Params: {} 2023-05-31T21:03:20.696 DEBUG:boto:Token: None 2023-05-31T21:03:20.697 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 21:03:20 GMT /pklxkd-61/ 2023-05-31T21:03:20.697 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:2L5YtQy7Tf5tqdFPiN197mxB4Yg= 2023-05-31T21:03:20.698 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 21:03:20 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:2L5YtQy7Tf5tqdFPiN197mxB4Yg=', 'Content-Length': '0'} 2023-05-31T21:03:20.700 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000006860d61f39917544-006477b618-1f4e-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 21:03:20 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:03:20.701 DEBUG:boto:b'pklxkd-61obj221000falseobj222023-05-31T20:59:44.449Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T21:03:20.701 DEBUG:boto:path=/pklxkd-61/ 2023-05-31T21:03:20.702 DEBUG:boto:auth_path=/pklxkd-61/ 2023-05-31T21:03:20.702 DEBUG:boto:path=/pklxkd-61/?prefix=obj22 2023-05-31T21:03:20.702 DEBUG:boto:auth_path=/pklxkd-61/?prefix=obj22 2023-05-31T21:03:20.703 DEBUG:boto:Method: GET 2023-05-31T21:03:20.703 DEBUG:boto:Path: /pklxkd-61/?prefix=obj22 2023-05-31T21:03:20.703 DEBUG:boto:Data: 2023-05-31T21:03:20.704 DEBUG:boto:Headers: {} 2023-05-31T21:03:20.704 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T21:03:20.705 DEBUG:boto:Port: 8000 2023-05-31T21:03:20.705 DEBUG:boto:Params: {} 2023-05-31T21:03:20.705 DEBUG:boto:Token: None 2023-05-31T21:03:20.706 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 21:03:20 GMT /pklxkd-61/ 2023-05-31T21:03:20.706 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:2L5YtQy7Tf5tqdFPiN197mxB4Yg= 2023-05-31T21:03:20.706 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 21:03:20 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:2L5YtQy7Tf5tqdFPiN197mxB4Yg=', 'Content-Length': '0'} 2023-05-31T21:03:20.709 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d4c2d6dc34485bf7-006477b618-1630-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 21:03:20 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:03:20.710 DEBUG:boto:b'pklxkd-61obj221000falseobj222023-05-31T20:59:44.449Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T21:03:20.711 DEBUG:boto:path=/pklxkd-61/obj22 2023-05-31T21:03:20.711 DEBUG:boto:auth_path=/pklxkd-61/obj22 2023-05-31T21:03:20.711 DEBUG:boto:Method: HEAD 2023-05-31T21:03:20.712 DEBUG:boto:Path: /pklxkd-61/obj22 2023-05-31T21:03:20.712 DEBUG:boto:Data: 2023-05-31T21:03:20.712 DEBUG:boto:Headers: {} 2023-05-31T21:03:20.713 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T21:03:20.713 DEBUG:boto:Port: 8001 2023-05-31T21:03:20.713 DEBUG:boto:Params: {} 2023-05-31T21:03:20.714 DEBUG:boto:Token: None 2023-05-31T21:03:20.714 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 21:03:20 GMT /pklxkd-61/obj22 2023-05-31T21:03:20.714 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:gX6Z7Q/wLPZ9QvayCPgbDepOkVs= 2023-05-31T21:03:20.715 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 21:03:20 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:gX6Z7Q/wLPZ9QvayCPgbDepOkVs=', 'Content-Length': '0'} 2023-05-31T21:03:20.716 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:59:44 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005af316edaf01ceaf-006477b618-1f4e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 21:03:20 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:03:20.717 DEBUG:boto:path=/pklxkd-61/obj22 2023-05-31T21:03:20.717 DEBUG:boto:auth_path=/pklxkd-61/obj22 2023-05-31T21:03:20.718 DEBUG:boto:Method: HEAD 2023-05-31T21:03:20.718 DEBUG:boto:Path: /pklxkd-61/obj22 2023-05-31T21:03:20.718 DEBUG:boto:Data: 2023-05-31T21:03:20.719 DEBUG:boto:Headers: {} 2023-05-31T21:03:20.719 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T21:03:20.719 DEBUG:boto:Port: 8000 2023-05-31T21:03:20.720 DEBUG:boto:Params: {} 2023-05-31T21:03:20.720 DEBUG:boto:Token: None 2023-05-31T21:03:20.720 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 21:03:20 GMT /pklxkd-61/obj22 2023-05-31T21:03:20.721 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:gX6Z7Q/wLPZ9QvayCPgbDepOkVs= 2023-05-31T21:03:20.721 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 21:03:20 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:gX6Z7Q/wLPZ9QvayCPgbDepOkVs=', 'Content-Length': '0'} 2023-05-31T21:03:20.723 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:59:44 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000dc102df9643768d6-006477b618-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 21:03:20 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:03:20.723 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj22 2023-05-31T21:03:20.724 DEBUG:boto:path=/pklxkd-61/obj22 2023-05-31T21:03:20.724 DEBUG:boto:auth_path=/pklxkd-61/obj22 2023-05-31T21:03:20.724 DEBUG:boto:Method: GET 2023-05-31T21:03:20.725 DEBUG:boto:Path: /pklxkd-61/obj22 2023-05-31T21:03:20.725 DEBUG:boto:Data: 2023-05-31T21:03:20.725 DEBUG:boto:Headers: {} 2023-05-31T21:03:20.726 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T21:03:20.726 DEBUG:boto:Port: 8001 2023-05-31T21:03:20.726 DEBUG:boto:Params: {} 2023-05-31T21:03:20.727 DEBUG:boto:Token: None 2023-05-31T21:03:20.727 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 21:03:20 GMT /pklxkd-61/obj22 2023-05-31T21:03:20.728 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:6wHrV+Yx58q9A/onymChZ36ksSU= 2023-05-31T21:03:20.728 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 21:03:20 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:6wHrV+Yx58q9A/onymChZ36ksSU=', 'Content-Length': '0'} 2023-05-31T21:03:20.730 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:59:44 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001abaf900e777da5b-006477b618-1f4e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 21:03:20 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:03:20.772 DEBUG:boto:path=/pklxkd-61/obj22 2023-05-31T21:03:20.773 DEBUG:boto:auth_path=/pklxkd-61/obj22 2023-05-31T21:03:20.773 DEBUG:boto:Method: GET 2023-05-31T21:03:20.773 DEBUG:boto:Path: /pklxkd-61/obj22 2023-05-31T21:03:20.774 DEBUG:boto:Data: 2023-05-31T21:03:20.774 DEBUG:boto:Headers: {} 2023-05-31T21:03:20.775 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T21:03:20.775 DEBUG:boto:Port: 8000 2023-05-31T21:03:20.776 DEBUG:boto:Params: {} 2023-05-31T21:03:20.776 DEBUG:boto:Token: None 2023-05-31T21:03:20.777 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 21:03:20 GMT /pklxkd-61/obj22 2023-05-31T21:03:20.777 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:6wHrV+Yx58q9A/onymChZ36ksSU= 2023-05-31T21:03:20.778 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 21:03:20 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:6wHrV+Yx58q9A/onymChZ36ksSU=', 'Content-Length': '0'} 2023-05-31T21:03:20.779 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:59:44 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000001a553bd93a4709d-006477b618-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 21:03:20 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:03:20.780 DEBUG:boto:path=/pklxkd-61/ 2023-05-31T21:03:20.780 DEBUG:boto:auth_path=/pklxkd-61/ 2023-05-31T21:03:20.781 DEBUG:boto:path=/pklxkd-61/?prefix=obj22 2023-05-31T21:03:20.781 DEBUG:boto:auth_path=/pklxkd-61/?prefix=obj22 2023-05-31T21:03:20.781 DEBUG:boto:Method: GET 2023-05-31T21:03:20.782 DEBUG:boto:Path: /pklxkd-61/?prefix=obj22 2023-05-31T21:03:20.782 DEBUG:boto:Data: 2023-05-31T21:03:20.782 DEBUG:boto:Headers: {} 2023-05-31T21:03:20.783 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T21:03:20.783 DEBUG:boto:Port: 8001 2023-05-31T21:03:20.783 DEBUG:boto:Params: {} 2023-05-31T21:03:20.784 DEBUG:boto:Token: None 2023-05-31T21:03:20.784 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 21:03:20 GMT /pklxkd-61/ 2023-05-31T21:03:20.785 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:2L5YtQy7Tf5tqdFPiN197mxB4Yg= 2023-05-31T21:03:20.785 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 21:03:20 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:2L5YtQy7Tf5tqdFPiN197mxB4Yg=', 'Content-Length': '0'} 2023-05-31T21:03:20.787 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000013189e749e33c069-006477b618-1f4e-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 21:03:20 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:03:20.788 DEBUG:boto:b'pklxkd-61obj221000falseobj222023-05-31T20:59:44.449Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T21:03:20.788 DEBUG:boto:path=/pklxkd-61/ 2023-05-31T21:03:20.789 DEBUG:boto:auth_path=/pklxkd-61/ 2023-05-31T21:03:20.789 DEBUG:boto:path=/pklxkd-61/?prefix=obj22 2023-05-31T21:03:20.789 DEBUG:boto:auth_path=/pklxkd-61/?prefix=obj22 2023-05-31T21:03:20.790 DEBUG:boto:Method: GET 2023-05-31T21:03:20.790 DEBUG:boto:Path: /pklxkd-61/?prefix=obj22 2023-05-31T21:03:20.790 DEBUG:boto:Data: 2023-05-31T21:03:20.791 DEBUG:boto:Headers: {} 2023-05-31T21:03:20.791 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T21:03:20.791 DEBUG:boto:Port: 8000 2023-05-31T21:03:20.792 DEBUG:boto:Params: {} 2023-05-31T21:03:20.792 DEBUG:boto:Token: None 2023-05-31T21:03:20.793 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 21:03:20 GMT /pklxkd-61/ 2023-05-31T21:03:20.793 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:2L5YtQy7Tf5tqdFPiN197mxB4Yg= 2023-05-31T21:03:20.794 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 21:03:20 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:2L5YtQy7Tf5tqdFPiN197mxB4Yg=', 'Content-Length': '0'} 2023-05-31T21:03:20.797 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ac9320dd2c5b5516-006477b618-1630-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 21:03:20 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:03:20.798 DEBUG:boto:b'pklxkd-61obj221000falseobj222023-05-31T20:59:44.449Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T21:03:20.798 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj23 2023-05-31T21:03:20.799 DEBUG:boto:path=/pklxkd-61/obj23 2023-05-31T21:03:20.799 DEBUG:boto:auth_path=/pklxkd-61/obj23 2023-05-31T21:03:20.800 DEBUG:boto:path=/pklxkd-61/obj23?versionId=null 2023-05-31T21:03:20.800 DEBUG:boto:auth_path=/pklxkd-61/obj23?versionId=null 2023-05-31T21:03:20.800 DEBUG:boto:Method: GET 2023-05-31T21:03:20.801 DEBUG:boto:Path: /pklxkd-61/obj23?versionId=null 2023-05-31T21:03:20.801 DEBUG:boto:Data: 2023-05-31T21:03:20.801 DEBUG:boto:Headers: {} 2023-05-31T21:03:20.802 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T21:03:20.802 DEBUG:boto:Port: 8001 2023-05-31T21:03:20.802 DEBUG:boto:Params: {} 2023-05-31T21:03:20.803 DEBUG:boto:Token: None 2023-05-31T21:03:20.803 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 21:03:20 GMT /pklxkd-61/obj23?versionId=null 2023-05-31T21:03:20.804 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:twQUuAlQ67VJdMpMbw7iCNxt0zo= 2023-05-31T21:03:20.804 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 21:03:20 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:twQUuAlQ67VJdMpMbw7iCNxt0zo=', 'Content-Length': '0'} 2023-05-31T21:03:20.806 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:59:45 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000068463a511426019d-006477b618-1f4e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 21:03:20 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:03:20.806 DEBUG:boto:path=/pklxkd-61/obj23 2023-05-31T21:03:20.807 DEBUG:boto:auth_path=/pklxkd-61/obj23 2023-05-31T21:03:20.807 DEBUG:boto:path=/pklxkd-61/obj23?versionId=null 2023-05-31T21:03:20.808 DEBUG:boto:auth_path=/pklxkd-61/obj23?versionId=null 2023-05-31T21:03:20.808 DEBUG:boto:Method: GET 2023-05-31T21:03:20.809 DEBUG:boto:Path: /pklxkd-61/obj23?versionId=null 2023-05-31T21:03:20.809 DEBUG:boto:Data: 2023-05-31T21:03:20.810 DEBUG:boto:Headers: {} 2023-05-31T21:03:20.810 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T21:03:20.811 DEBUG:boto:Port: 8000 2023-05-31T21:03:20.812 DEBUG:boto:Params: {} 2023-05-31T21:03:20.812 DEBUG:boto:Token: None 2023-05-31T21:03:20.813 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 21:03:20 GMT /pklxkd-61/obj23?versionId=null 2023-05-31T21:03:20.813 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:twQUuAlQ67VJdMpMbw7iCNxt0zo= 2023-05-31T21:03:20.814 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 21:03:20 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:twQUuAlQ67VJdMpMbw7iCNxt0zo=', 'Content-Length': '0'} 2023-05-31T21:03:20.816 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:59:45 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007208c1fa71ccff1f-006477b618-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 21:03:20 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:03:20.856 DEBUG:boto:path=/pklxkd-61/obj23 2023-05-31T21:03:20.857 DEBUG:boto:auth_path=/pklxkd-61/obj23 2023-05-31T21:03:20.857 DEBUG:boto:path=/pklxkd-61/obj23?versionId=null 2023-05-31T21:03:20.857 DEBUG:boto:auth_path=/pklxkd-61/obj23?versionId=null 2023-05-31T21:03:20.858 DEBUG:boto:Method: HEAD 2023-05-31T21:03:20.858 DEBUG:boto:Path: /pklxkd-61/obj23?versionId=null 2023-05-31T21:03:20.858 DEBUG:boto:Data: 2023-05-31T21:03:20.859 DEBUG:boto:Headers: {} 2023-05-31T21:03:20.859 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T21:03:20.859 DEBUG:boto:Port: 8001 2023-05-31T21:03:20.860 DEBUG:boto:Params: {} 2023-05-31T21:03:20.860 DEBUG:boto:Token: None 2023-05-31T21:03:20.861 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 21:03:20 GMT /pklxkd-61/obj23?versionId=null 2023-05-31T21:03:20.861 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:kzNQ8yhYMrYA14bGcbY/xaUGAfY= 2023-05-31T21:03:20.861 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 21:03:20 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:kzNQ8yhYMrYA14bGcbY/xaUGAfY=', 'Content-Length': '0'} 2023-05-31T21:03:20.863 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:59:45 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009e03c3821a24b8d9-006477b618-1f4e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 21:03:20 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:03:20.864 DEBUG:boto:path=/pklxkd-61/obj23 2023-05-31T21:03:20.864 DEBUG:boto:auth_path=/pklxkd-61/obj23 2023-05-31T21:03:20.864 DEBUG:boto:path=/pklxkd-61/obj23?versionId=null 2023-05-31T21:03:20.865 DEBUG:boto:auth_path=/pklxkd-61/obj23?versionId=null 2023-05-31T21:03:20.865 DEBUG:boto:Method: HEAD 2023-05-31T21:03:20.865 DEBUG:boto:Path: /pklxkd-61/obj23?versionId=null 2023-05-31T21:03:20.866 DEBUG:boto:Data: 2023-05-31T21:03:20.866 DEBUG:boto:Headers: {} 2023-05-31T21:03:20.866 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T21:03:20.867 DEBUG:boto:Port: 8000 2023-05-31T21:03:20.867 DEBUG:boto:Params: {} 2023-05-31T21:03:20.868 DEBUG:boto:Token: None 2023-05-31T21:03:20.868 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 21:03:20 GMT /pklxkd-61/obj23?versionId=null 2023-05-31T21:03:20.868 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:kzNQ8yhYMrYA14bGcbY/xaUGAfY= 2023-05-31T21:03:20.869 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 21:03:20 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:kzNQ8yhYMrYA14bGcbY/xaUGAfY=', 'Content-Length': '0'} 2023-05-31T21:03:20.870 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:59:45 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d4a5e1825a5c61d5-006477b618-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 21:03:20 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:03:20.871 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj23 2023-05-31T21:03:20.871 DEBUG:boto:path=/pklxkd-61/obj23 2023-05-31T21:03:20.872 DEBUG:boto:auth_path=/pklxkd-61/obj23 2023-05-31T21:03:20.872 DEBUG:boto:path=/pklxkd-61/obj23?versionId=null 2023-05-31T21:03:20.872 DEBUG:boto:auth_path=/pklxkd-61/obj23?versionId=null 2023-05-31T21:03:20.873 DEBUG:boto:Method: GET 2023-05-31T21:03:20.873 DEBUG:boto:Path: /pklxkd-61/obj23?versionId=null 2023-05-31T21:03:20.873 DEBUG:boto:Data: 2023-05-31T21:03:20.874 DEBUG:boto:Headers: {} 2023-05-31T21:03:20.874 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T21:03:20.874 DEBUG:boto:Port: 8001 2023-05-31T21:03:20.875 DEBUG:boto:Params: {} 2023-05-31T21:03:20.875 DEBUG:boto:Token: None 2023-05-31T21:03:20.876 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 21:03:20 GMT /pklxkd-61/obj23?versionId=null 2023-05-31T21:03:20.876 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:twQUuAlQ67VJdMpMbw7iCNxt0zo= 2023-05-31T21:03:20.877 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 21:03:20 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:twQUuAlQ67VJdMpMbw7iCNxt0zo=', 'Content-Length': '0'} 2023-05-31T21:03:20.879 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:59:45 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000574ab9ecf8bf65a4-006477b618-1f4e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 21:03:20 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:03:20.879 DEBUG:boto:path=/pklxkd-61/obj23 2023-05-31T21:03:20.880 DEBUG:boto:auth_path=/pklxkd-61/obj23 2023-05-31T21:03:20.880 DEBUG:boto:path=/pklxkd-61/obj23?versionId=null 2023-05-31T21:03:20.881 DEBUG:boto:auth_path=/pklxkd-61/obj23?versionId=null 2023-05-31T21:03:20.882 DEBUG:boto:Method: GET 2023-05-31T21:03:20.883 DEBUG:boto:Path: /pklxkd-61/obj23?versionId=null 2023-05-31T21:03:20.884 DEBUG:boto:Data: 2023-05-31T21:03:20.884 DEBUG:boto:Headers: {} 2023-05-31T21:03:20.885 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T21:03:20.886 DEBUG:boto:Port: 8000 2023-05-31T21:03:20.886 DEBUG:boto:Params: {} 2023-05-31T21:03:20.887 DEBUG:boto:Token: None 2023-05-31T21:03:20.887 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 21:03:20 GMT /pklxkd-61/obj23?versionId=null 2023-05-31T21:03:20.888 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:twQUuAlQ67VJdMpMbw7iCNxt0zo= 2023-05-31T21:03:20.888 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 21:03:20 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:twQUuAlQ67VJdMpMbw7iCNxt0zo=', 'Content-Length': '0'} 2023-05-31T21:03:20.890 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:59:45 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008c1e60c80c50ad14-006477b618-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 21:03:20 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:03:20.891 DEBUG:boto:path=/pklxkd-61/ 2023-05-31T21:03:20.891 DEBUG:boto:auth_path=/pklxkd-61/ 2023-05-31T21:03:20.891 DEBUG:boto:path=/pklxkd-61/?prefix=obj23 2023-05-31T21:03:20.892 DEBUG:boto:auth_path=/pklxkd-61/?prefix=obj23 2023-05-31T21:03:20.892 DEBUG:boto:Method: GET 2023-05-31T21:03:20.893 DEBUG:boto:Path: /pklxkd-61/?prefix=obj23 2023-05-31T21:03:20.893 DEBUG:boto:Data: 2023-05-31T21:03:20.893 DEBUG:boto:Headers: {} 2023-05-31T21:03:20.894 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T21:03:20.894 DEBUG:boto:Port: 8001 2023-05-31T21:03:20.894 DEBUG:boto:Params: {} 2023-05-31T21:03:20.895 DEBUG:boto:Token: None 2023-05-31T21:03:20.895 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 21:03:20 GMT /pklxkd-61/ 2023-05-31T21:03:20.896 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:2L5YtQy7Tf5tqdFPiN197mxB4Yg= 2023-05-31T21:03:20.896 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 21:03:20 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:2L5YtQy7Tf5tqdFPiN197mxB4Yg=', 'Content-Length': '0'} 2023-05-31T21:03:20.898 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000074841c9648d306c3-006477b618-1f4e-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 21:03:20 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:03:20.899 DEBUG:boto:b'pklxkd-61obj231000falseobj232023-05-31T20:59:45.517Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T21:03:20.900 DEBUG:boto:path=/pklxkd-61/ 2023-05-31T21:03:20.900 DEBUG:boto:auth_path=/pklxkd-61/ 2023-05-31T21:03:20.900 DEBUG:boto:path=/pklxkd-61/?prefix=obj23 2023-05-31T21:03:20.901 DEBUG:boto:auth_path=/pklxkd-61/?prefix=obj23 2023-05-31T21:03:20.901 DEBUG:boto:Method: GET 2023-05-31T21:03:20.901 DEBUG:boto:Path: /pklxkd-61/?prefix=obj23 2023-05-31T21:03:20.902 DEBUG:boto:Data: 2023-05-31T21:03:20.902 DEBUG:boto:Headers: {} 2023-05-31T21:03:20.903 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T21:03:20.903 DEBUG:boto:Port: 8000 2023-05-31T21:03:20.903 DEBUG:boto:Params: {} 2023-05-31T21:03:20.904 DEBUG:boto:Token: None 2023-05-31T21:03:20.904 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 21:03:20 GMT /pklxkd-61/ 2023-05-31T21:03:20.904 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:2L5YtQy7Tf5tqdFPiN197mxB4Yg= 2023-05-31T21:03:20.905 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 21:03:20 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:2L5YtQy7Tf5tqdFPiN197mxB4Yg=', 'Content-Length': '0'} 2023-05-31T21:03:20.908 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c9214391a9345b5a-006477b618-1630-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 21:03:20 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:03:20.909 DEBUG:boto:b'pklxkd-61obj231000falseobj232023-05-31T20:59:45.517Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T21:03:20.909 DEBUG:boto:path=/pklxkd-61/obj23 2023-05-31T21:03:20.910 DEBUG:boto:auth_path=/pklxkd-61/obj23 2023-05-31T21:03:20.910 DEBUG:boto:Method: HEAD 2023-05-31T21:03:20.911 DEBUG:boto:Path: /pklxkd-61/obj23 2023-05-31T21:03:20.911 DEBUG:boto:Data: 2023-05-31T21:03:20.911 DEBUG:boto:Headers: {} 2023-05-31T21:03:20.912 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T21:03:20.912 DEBUG:boto:Port: 8001 2023-05-31T21:03:20.912 DEBUG:boto:Params: {} 2023-05-31T21:03:20.913 DEBUG:boto:Token: None 2023-05-31T21:03:20.913 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 21:03:20 GMT /pklxkd-61/obj23 2023-05-31T21:03:20.913 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:Ch8x0eS37+rD9Hto00EArgEAqws= 2023-05-31T21:03:20.914 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 21:03:20 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:Ch8x0eS37+rD9Hto00EArgEAqws=', 'Content-Length': '0'} 2023-05-31T21:03:20.915 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:59:45 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e64099419d1c8fbf-006477b618-1f4e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 21:03:20 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:03:20.916 DEBUG:boto:path=/pklxkd-61/obj23 2023-05-31T21:03:20.916 DEBUG:boto:auth_path=/pklxkd-61/obj23 2023-05-31T21:03:20.917 DEBUG:boto:Method: HEAD 2023-05-31T21:03:20.917 DEBUG:boto:Path: /pklxkd-61/obj23 2023-05-31T21:03:20.918 DEBUG:boto:Data: 2023-05-31T21:03:20.918 DEBUG:boto:Headers: {} 2023-05-31T21:03:20.919 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T21:03:20.919 DEBUG:boto:Port: 8000 2023-05-31T21:03:20.919 DEBUG:boto:Params: {} 2023-05-31T21:03:20.920 DEBUG:boto:Token: None 2023-05-31T21:03:20.920 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 21:03:20 GMT /pklxkd-61/obj23 2023-05-31T21:03:20.920 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:Ch8x0eS37+rD9Hto00EArgEAqws= 2023-05-31T21:03:20.921 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 21:03:20 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:Ch8x0eS37+rD9Hto00EArgEAqws=', 'Content-Length': '0'} 2023-05-31T21:03:20.924 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:59:45 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000af4bd84d7b9eefe6-006477b618-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 21:03:20 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:03:20.925 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj23 2023-05-31T21:03:20.925 DEBUG:boto:path=/pklxkd-61/obj23 2023-05-31T21:03:20.926 DEBUG:boto:auth_path=/pklxkd-61/obj23 2023-05-31T21:03:20.927 DEBUG:boto:Method: GET 2023-05-31T21:03:20.928 DEBUG:boto:Path: /pklxkd-61/obj23 2023-05-31T21:03:20.928 DEBUG:boto:Data: 2023-05-31T21:03:20.929 DEBUG:boto:Headers: {} 2023-05-31T21:03:20.930 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T21:03:20.930 DEBUG:boto:Port: 8001 2023-05-31T21:03:20.930 DEBUG:boto:Params: {} 2023-05-31T21:03:20.931 DEBUG:boto:Token: None 2023-05-31T21:03:20.931 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 21:03:20 GMT /pklxkd-61/obj23 2023-05-31T21:03:20.932 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:pBiH/Fs6JKjha9j+DovLUAl1xxo= 2023-05-31T21:03:20.932 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 21:03:20 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:pBiH/Fs6JKjha9j+DovLUAl1xxo=', 'Content-Length': '0'} 2023-05-31T21:03:20.934 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:59:45 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000098c6acbfd3d697c0-006477b618-1f4e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 21:03:20 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:03:20.934 DEBUG:boto:path=/pklxkd-61/obj23 2023-05-31T21:03:20.935 DEBUG:boto:auth_path=/pklxkd-61/obj23 2023-05-31T21:03:20.935 DEBUG:boto:Method: GET 2023-05-31T21:03:20.936 DEBUG:boto:Path: /pklxkd-61/obj23 2023-05-31T21:03:20.936 DEBUG:boto:Data: 2023-05-31T21:03:20.936 DEBUG:boto:Headers: {} 2023-05-31T21:03:20.937 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T21:03:20.937 DEBUG:boto:Port: 8000 2023-05-31T21:03:20.937 DEBUG:boto:Params: {} 2023-05-31T21:03:20.938 DEBUG:boto:Token: None 2023-05-31T21:03:20.938 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 21:03:20 GMT /pklxkd-61/obj23 2023-05-31T21:03:20.939 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:pBiH/Fs6JKjha9j+DovLUAl1xxo= 2023-05-31T21:03:20.939 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 21:03:20 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:pBiH/Fs6JKjha9j+DovLUAl1xxo=', 'Content-Length': '0'} 2023-05-31T21:03:20.941 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:59:45 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e8d09dc58d0d6c88-006477b618-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 21:03:20 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:03:20.941 DEBUG:boto:path=/pklxkd-61/ 2023-05-31T21:03:20.942 DEBUG:boto:auth_path=/pklxkd-61/ 2023-05-31T21:03:20.942 DEBUG:boto:path=/pklxkd-61/?prefix=obj23 2023-05-31T21:03:20.943 DEBUG:boto:auth_path=/pklxkd-61/?prefix=obj23 2023-05-31T21:03:20.943 DEBUG:boto:Method: GET 2023-05-31T21:03:20.943 DEBUG:boto:Path: /pklxkd-61/?prefix=obj23 2023-05-31T21:03:20.944 DEBUG:boto:Data: 2023-05-31T21:03:20.944 DEBUG:boto:Headers: {} 2023-05-31T21:03:20.944 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T21:03:20.945 DEBUG:boto:Port: 8001 2023-05-31T21:03:20.945 DEBUG:boto:Params: {} 2023-05-31T21:03:20.946 DEBUG:boto:Token: None 2023-05-31T21:03:20.946 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 21:03:20 GMT /pklxkd-61/ 2023-05-31T21:03:20.946 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:2L5YtQy7Tf5tqdFPiN197mxB4Yg= 2023-05-31T21:03:20.947 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 21:03:20 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:2L5YtQy7Tf5tqdFPiN197mxB4Yg=', 'Content-Length': '0'} 2023-05-31T21:03:20.949 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000db4b86b709c80753-006477b618-1f4e-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 21:03:20 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:03:20.949 DEBUG:boto:b'pklxkd-61obj231000falseobj232023-05-31T20:59:45.517Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T21:03:20.950 DEBUG:boto:path=/pklxkd-61/ 2023-05-31T21:03:20.950 DEBUG:boto:auth_path=/pklxkd-61/ 2023-05-31T21:03:20.951 DEBUG:boto:path=/pklxkd-61/?prefix=obj23 2023-05-31T21:03:20.951 DEBUG:boto:auth_path=/pklxkd-61/?prefix=obj23 2023-05-31T21:03:20.951 DEBUG:boto:Method: GET 2023-05-31T21:03:20.952 DEBUG:boto:Path: /pklxkd-61/?prefix=obj23 2023-05-31T21:03:20.952 DEBUG:boto:Data: 2023-05-31T21:03:20.953 DEBUG:boto:Headers: {} 2023-05-31T21:03:20.953 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T21:03:20.953 DEBUG:boto:Port: 8000 2023-05-31T21:03:20.954 DEBUG:boto:Params: {} 2023-05-31T21:03:20.954 DEBUG:boto:Token: None 2023-05-31T21:03:20.954 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 21:03:20 GMT /pklxkd-61/ 2023-05-31T21:03:20.955 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:2L5YtQy7Tf5tqdFPiN197mxB4Yg= 2023-05-31T21:03:20.955 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 21:03:20 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:2L5YtQy7Tf5tqdFPiN197mxB4Yg=', 'Content-Length': '0'} 2023-05-31T21:03:20.959 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000004be621d7b3d8633d-006477b618-1630-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 21:03:20 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:03:21.000 DEBUG:boto:b'pklxkd-61obj231000falseobj232023-05-31T20:59:45.517Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T21:03:21.001 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj29 2023-05-31T21:03:21.001 DEBUG:boto:path=/pklxkd-61/obj29 2023-05-31T21:03:21.002 DEBUG:boto:auth_path=/pklxkd-61/obj29 2023-05-31T21:03:21.002 DEBUG:boto:path=/pklxkd-61/obj29?versionId=null 2023-05-31T21:03:21.002 DEBUG:boto:auth_path=/pklxkd-61/obj29?versionId=null 2023-05-31T21:03:21.003 DEBUG:boto:Method: GET 2023-05-31T21:03:21.003 DEBUG:boto:Path: /pklxkd-61/obj29?versionId=null 2023-05-31T21:03:21.003 DEBUG:boto:Data: 2023-05-31T21:03:21.004 DEBUG:boto:Headers: {} 2023-05-31T21:03:21.004 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T21:03:21.004 DEBUG:boto:Port: 8001 2023-05-31T21:03:21.005 DEBUG:boto:Params: {} 2023-05-31T21:03:21.005 DEBUG:boto:Token: None 2023-05-31T21:03:21.006 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 21:03:21 GMT /pklxkd-61/obj29?versionId=null 2023-05-31T21:03:21.006 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:qHY/6W8GFg8QF5aAdD7apBLFBzE= 2023-05-31T21:03:21.006 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 21:03:21 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:qHY/6W8GFg8QF5aAdD7apBLFBzE=', 'Content-Length': '0'} 2023-05-31T21:03:21.008 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:59:45 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ee7ede24dc5cec77-006477b619-1f4e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 21:03:21 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:03:21.009 DEBUG:boto:path=/pklxkd-61/obj29 2023-05-31T21:03:21.009 DEBUG:boto:auth_path=/pklxkd-61/obj29 2023-05-31T21:03:21.010 DEBUG:boto:path=/pklxkd-61/obj29?versionId=null 2023-05-31T21:03:21.010 DEBUG:boto:auth_path=/pklxkd-61/obj29?versionId=null 2023-05-31T21:03:21.010 DEBUG:boto:Method: GET 2023-05-31T21:03:21.011 DEBUG:boto:Path: /pklxkd-61/obj29?versionId=null 2023-05-31T21:03:21.011 DEBUG:boto:Data: 2023-05-31T21:03:21.011 DEBUG:boto:Headers: {} 2023-05-31T21:03:21.012 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T21:03:21.012 DEBUG:boto:Port: 8000 2023-05-31T21:03:21.012 DEBUG:boto:Params: {} 2023-05-31T21:03:21.013 DEBUG:boto:Token: None 2023-05-31T21:03:21.013 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 21:03:21 GMT /pklxkd-61/obj29?versionId=null 2023-05-31T21:03:21.014 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:qHY/6W8GFg8QF5aAdD7apBLFBzE= 2023-05-31T21:03:21.014 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 21:03:21 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:qHY/6W8GFg8QF5aAdD7apBLFBzE=', 'Content-Length': '0'} 2023-05-31T21:03:21.016 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:59:45 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002c149080c3e43727-006477b619-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 21:03:21 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:03:21.017 DEBUG:boto:path=/pklxkd-61/obj29 2023-05-31T21:03:21.017 DEBUG:boto:auth_path=/pklxkd-61/obj29 2023-05-31T21:03:21.017 DEBUG:boto:path=/pklxkd-61/obj29?versionId=null 2023-05-31T21:03:21.018 DEBUG:boto:auth_path=/pklxkd-61/obj29?versionId=null 2023-05-31T21:03:21.018 DEBUG:boto:Method: HEAD 2023-05-31T21:03:21.018 DEBUG:boto:Path: /pklxkd-61/obj29?versionId=null 2023-05-31T21:03:21.019 DEBUG:boto:Data: 2023-05-31T21:03:21.019 DEBUG:boto:Headers: {} 2023-05-31T21:03:21.019 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T21:03:21.020 DEBUG:boto:Port: 8001 2023-05-31T21:03:21.020 DEBUG:boto:Params: {} 2023-05-31T21:03:21.020 DEBUG:boto:Token: None 2023-05-31T21:03:21.021 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 21:03:21 GMT /pklxkd-61/obj29?versionId=null 2023-05-31T21:03:21.021 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:37JuPn7NUCEnPg1+IpDKttpZRw8= 2023-05-31T21:03:21.021 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 21:03:21 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:37JuPn7NUCEnPg1+IpDKttpZRw8=', 'Content-Length': '0'} 2023-05-31T21:03:21.023 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:59:45 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000fa37c35add19dc19-006477b619-1f4e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 21:03:21 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:03:21.023 DEBUG:boto:path=/pklxkd-61/obj29 2023-05-31T21:03:21.024 DEBUG:boto:auth_path=/pklxkd-61/obj29 2023-05-31T21:03:21.024 DEBUG:boto:path=/pklxkd-61/obj29?versionId=null 2023-05-31T21:03:21.025 DEBUG:boto:auth_path=/pklxkd-61/obj29?versionId=null 2023-05-31T21:03:21.025 DEBUG:boto:Method: HEAD 2023-05-31T21:03:21.025 DEBUG:boto:Path: /pklxkd-61/obj29?versionId=null 2023-05-31T21:03:21.026 DEBUG:boto:Data: 2023-05-31T21:03:21.026 DEBUG:boto:Headers: {} 2023-05-31T21:03:21.026 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T21:03:21.027 DEBUG:boto:Port: 8000 2023-05-31T21:03:21.027 DEBUG:boto:Params: {} 2023-05-31T21:03:21.027 DEBUG:boto:Token: None 2023-05-31T21:03:21.028 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 21:03:21 GMT /pklxkd-61/obj29?versionId=null 2023-05-31T21:03:21.028 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:37JuPn7NUCEnPg1+IpDKttpZRw8= 2023-05-31T21:03:21.029 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 21:03:21 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:37JuPn7NUCEnPg1+IpDKttpZRw8=', 'Content-Length': '0'} 2023-05-31T21:03:21.030 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:59:45 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c0b52c8bea2d98fa-006477b619-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 21:03:21 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:03:21.031 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj29 2023-05-31T21:03:21.031 DEBUG:boto:path=/pklxkd-61/obj29 2023-05-31T21:03:21.031 DEBUG:boto:auth_path=/pklxkd-61/obj29 2023-05-31T21:03:21.032 DEBUG:boto:path=/pklxkd-61/obj29?versionId=null 2023-05-31T21:03:21.032 DEBUG:boto:auth_path=/pklxkd-61/obj29?versionId=null 2023-05-31T21:03:21.032 DEBUG:boto:Method: GET 2023-05-31T21:03:21.033 DEBUG:boto:Path: /pklxkd-61/obj29?versionId=null 2023-05-31T21:03:21.033 DEBUG:boto:Data: 2023-05-31T21:03:21.033 DEBUG:boto:Headers: {} 2023-05-31T21:03:21.034 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T21:03:21.034 DEBUG:boto:Port: 8001 2023-05-31T21:03:21.035 DEBUG:boto:Params: {} 2023-05-31T21:03:21.035 DEBUG:boto:Token: None 2023-05-31T21:03:21.035 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 21:03:21 GMT /pklxkd-61/obj29?versionId=null 2023-05-31T21:03:21.036 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:qHY/6W8GFg8QF5aAdD7apBLFBzE= 2023-05-31T21:03:21.036 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 21:03:21 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:qHY/6W8GFg8QF5aAdD7apBLFBzE=', 'Content-Length': '0'} 2023-05-31T21:03:21.038 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:59:45 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c18d595847973275-006477b619-1f4e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 21:03:21 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:03:21.038 DEBUG:boto:path=/pklxkd-61/obj29 2023-05-31T21:03:21.039 DEBUG:boto:auth_path=/pklxkd-61/obj29 2023-05-31T21:03:21.039 DEBUG:boto:path=/pklxkd-61/obj29?versionId=null 2023-05-31T21:03:21.039 DEBUG:boto:auth_path=/pklxkd-61/obj29?versionId=null 2023-05-31T21:03:21.040 DEBUG:boto:Method: GET 2023-05-31T21:03:21.040 DEBUG:boto:Path: /pklxkd-61/obj29?versionId=null 2023-05-31T21:03:21.041 DEBUG:boto:Data: 2023-05-31T21:03:21.041 DEBUG:boto:Headers: {} 2023-05-31T21:03:21.041 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T21:03:21.042 DEBUG:boto:Port: 8000 2023-05-31T21:03:21.042 DEBUG:boto:Params: {} 2023-05-31T21:03:21.043 DEBUG:boto:Token: None 2023-05-31T21:03:21.043 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 21:03:21 GMT /pklxkd-61/obj29?versionId=null 2023-05-31T21:03:21.043 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:qHY/6W8GFg8QF5aAdD7apBLFBzE= 2023-05-31T21:03:21.044 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 21:03:21 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:qHY/6W8GFg8QF5aAdD7apBLFBzE=', 'Content-Length': '0'} 2023-05-31T21:03:21.045 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:59:45 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a9c7a5edbda13a7d-006477b619-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 21:03:21 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:03:21.046 DEBUG:boto:path=/pklxkd-61/ 2023-05-31T21:03:21.046 DEBUG:boto:auth_path=/pklxkd-61/ 2023-05-31T21:03:21.047 DEBUG:boto:path=/pklxkd-61/?prefix=obj29 2023-05-31T21:03:21.047 DEBUG:boto:auth_path=/pklxkd-61/?prefix=obj29 2023-05-31T21:03:21.048 DEBUG:boto:Method: GET 2023-05-31T21:03:21.048 DEBUG:boto:Path: /pklxkd-61/?prefix=obj29 2023-05-31T21:03:21.049 DEBUG:boto:Data: 2023-05-31T21:03:21.049 DEBUG:boto:Headers: {} 2023-05-31T21:03:21.049 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T21:03:21.050 DEBUG:boto:Port: 8001 2023-05-31T21:03:21.050 DEBUG:boto:Params: {} 2023-05-31T21:03:21.050 DEBUG:boto:Token: None 2023-05-31T21:03:21.051 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 21:03:21 GMT /pklxkd-61/ 2023-05-31T21:03:21.052 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:H9PQFgrR8ica+qPwTl/4XbpLrOQ= 2023-05-31T21:03:21.053 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 21:03:21 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:H9PQFgrR8ica+qPwTl/4XbpLrOQ=', 'Content-Length': '0'} 2023-05-31T21:03:21.056 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000005c91c4305988ed11-006477b619-1f4e-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 21:03:21 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:03:21.056 DEBUG:boto:b'pklxkd-61obj291000falseobj292023-05-31T20:59:45.888Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T21:03:21.057 DEBUG:boto:path=/pklxkd-61/ 2023-05-31T21:03:21.057 DEBUG:boto:auth_path=/pklxkd-61/ 2023-05-31T21:03:21.058 DEBUG:boto:path=/pklxkd-61/?prefix=obj29 2023-05-31T21:03:21.058 DEBUG:boto:auth_path=/pklxkd-61/?prefix=obj29 2023-05-31T21:03:21.059 DEBUG:boto:Method: GET 2023-05-31T21:03:21.059 DEBUG:boto:Path: /pklxkd-61/?prefix=obj29 2023-05-31T21:03:21.059 DEBUG:boto:Data: 2023-05-31T21:03:21.060 DEBUG:boto:Headers: {} 2023-05-31T21:03:21.060 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T21:03:21.060 DEBUG:boto:Port: 8000 2023-05-31T21:03:21.061 DEBUG:boto:Params: {} 2023-05-31T21:03:21.061 DEBUG:boto:Token: None 2023-05-31T21:03:21.062 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 21:03:21 GMT /pklxkd-61/ 2023-05-31T21:03:21.062 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:H9PQFgrR8ica+qPwTl/4XbpLrOQ= 2023-05-31T21:03:21.062 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 21:03:21 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:H9PQFgrR8ica+qPwTl/4XbpLrOQ=', 'Content-Length': '0'} 2023-05-31T21:03:21.067 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000955405ea2952d85a-006477b619-1630-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 21:03:21 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:03:21.068 DEBUG:boto:b'pklxkd-61obj291000falseobj292023-05-31T20:59:45.888Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T21:03:21.068 DEBUG:boto:path=/pklxkd-61/obj29 2023-05-31T21:03:21.069 DEBUG:boto:auth_path=/pklxkd-61/obj29 2023-05-31T21:03:21.070 DEBUG:boto:Method: HEAD 2023-05-31T21:03:21.071 DEBUG:boto:Path: /pklxkd-61/obj29 2023-05-31T21:03:21.071 DEBUG:boto:Data: 2023-05-31T21:03:21.072 DEBUG:boto:Headers: {} 2023-05-31T21:03:21.073 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T21:03:21.074 DEBUG:boto:Port: 8001 2023-05-31T21:03:21.075 DEBUG:boto:Params: {} 2023-05-31T21:03:21.075 DEBUG:boto:Token: None 2023-05-31T21:03:21.076 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 21:03:21 GMT /pklxkd-61/obj29 2023-05-31T21:03:21.076 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:YQI86fVWpG52E3U+LnQroy3OSzg= 2023-05-31T21:03:21.077 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 21:03:21 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:YQI86fVWpG52E3U+LnQroy3OSzg=', 'Content-Length': '0'} 2023-05-31T21:03:21.079 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:59:45 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f8561dea5a377626-006477b619-1f4e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 21:03:21 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:03:21.079 DEBUG:boto:path=/pklxkd-61/obj29 2023-05-31T21:03:21.080 DEBUG:boto:auth_path=/pklxkd-61/obj29 2023-05-31T21:03:21.080 DEBUG:boto:Method: HEAD 2023-05-31T21:03:21.081 DEBUG:boto:Path: /pklxkd-61/obj29 2023-05-31T21:03:21.081 DEBUG:boto:Data: 2023-05-31T21:03:21.081 DEBUG:boto:Headers: {} 2023-05-31T21:03:21.082 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T21:03:21.082 DEBUG:boto:Port: 8000 2023-05-31T21:03:21.082 DEBUG:boto:Params: {} 2023-05-31T21:03:21.083 DEBUG:boto:Token: None 2023-05-31T21:03:21.083 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 21:03:21 GMT /pklxkd-61/obj29 2023-05-31T21:03:21.083 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:YQI86fVWpG52E3U+LnQroy3OSzg= 2023-05-31T21:03:21.084 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 21:03:21 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:YQI86fVWpG52E3U+LnQroy3OSzg=', 'Content-Length': '0'} 2023-05-31T21:03:21.085 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:59:45 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000aa06be38194c5fc7-006477b619-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 21:03:21 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:03:21.086 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj29 2023-05-31T21:03:21.086 DEBUG:boto:path=/pklxkd-61/obj29 2023-05-31T21:03:21.087 DEBUG:boto:auth_path=/pklxkd-61/obj29 2023-05-31T21:03:21.087 DEBUG:boto:Method: GET 2023-05-31T21:03:21.088 DEBUG:boto:Path: /pklxkd-61/obj29 2023-05-31T21:03:21.088 DEBUG:boto:Data: 2023-05-31T21:03:21.088 DEBUG:boto:Headers: {} 2023-05-31T21:03:21.089 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T21:03:21.089 DEBUG:boto:Port: 8001 2023-05-31T21:03:21.089 DEBUG:boto:Params: {} 2023-05-31T21:03:21.090 DEBUG:boto:Token: None 2023-05-31T21:03:21.090 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 21:03:21 GMT /pklxkd-61/obj29 2023-05-31T21:03:21.091 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:JF1GVwZAtl3UxZr9pbFXy3C0jLA= 2023-05-31T21:03:21.091 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 21:03:21 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:JF1GVwZAtl3UxZr9pbFXy3C0jLA=', 'Content-Length': '0'} 2023-05-31T21:03:21.093 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:59:45 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000086c6862d8c0e6770-006477b619-1f4e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 21:03:21 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:03:21.093 DEBUG:boto:path=/pklxkd-61/obj29 2023-05-31T21:03:21.094 DEBUG:boto:auth_path=/pklxkd-61/obj29 2023-05-31T21:03:21.094 DEBUG:boto:Method: GET 2023-05-31T21:03:21.094 DEBUG:boto:Path: /pklxkd-61/obj29 2023-05-31T21:03:21.095 DEBUG:boto:Data: 2023-05-31T21:03:21.095 DEBUG:boto:Headers: {} 2023-05-31T21:03:21.095 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T21:03:21.096 DEBUG:boto:Port: 8000 2023-05-31T21:03:21.096 DEBUG:boto:Params: {} 2023-05-31T21:03:21.097 DEBUG:boto:Token: None 2023-05-31T21:03:21.097 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 21:03:21 GMT /pklxkd-61/obj29 2023-05-31T21:03:21.098 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:JF1GVwZAtl3UxZr9pbFXy3C0jLA= 2023-05-31T21:03:21.098 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 21:03:21 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:JF1GVwZAtl3UxZr9pbFXy3C0jLA=', 'Content-Length': '0'} 2023-05-31T21:03:21.100 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:59:45 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000098d0d09e871fd19b-006477b619-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 21:03:21 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:03:21.100 DEBUG:boto:path=/pklxkd-61/ 2023-05-31T21:03:21.101 DEBUG:boto:auth_path=/pklxkd-61/ 2023-05-31T21:03:21.101 DEBUG:boto:path=/pklxkd-61/?prefix=obj29 2023-05-31T21:03:21.101 DEBUG:boto:auth_path=/pklxkd-61/?prefix=obj29 2023-05-31T21:03:21.102 DEBUG:boto:Method: GET 2023-05-31T21:03:21.102 DEBUG:boto:Path: /pklxkd-61/?prefix=obj29 2023-05-31T21:03:21.102 DEBUG:boto:Data: 2023-05-31T21:03:21.103 DEBUG:boto:Headers: {} 2023-05-31T21:03:21.103 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T21:03:21.104 DEBUG:boto:Port: 8001 2023-05-31T21:03:21.104 DEBUG:boto:Params: {} 2023-05-31T21:03:21.104 DEBUG:boto:Token: None 2023-05-31T21:03:21.105 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 21:03:21 GMT /pklxkd-61/ 2023-05-31T21:03:21.105 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:H9PQFgrR8ica+qPwTl/4XbpLrOQ= 2023-05-31T21:03:21.106 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 21:03:21 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:H9PQFgrR8ica+qPwTl/4XbpLrOQ=', 'Content-Length': '0'} 2023-05-31T21:03:21.108 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d340f9e9d0ae64d2-006477b619-1f4e-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 21:03:21 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:03:21.152 DEBUG:boto:b'pklxkd-61obj291000falseobj292023-05-31T20:59:45.888Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T21:03:21.153 DEBUG:boto:path=/pklxkd-61/ 2023-05-31T21:03:21.153 DEBUG:boto:auth_path=/pklxkd-61/ 2023-05-31T21:03:21.154 DEBUG:boto:path=/pklxkd-61/?prefix=obj29 2023-05-31T21:03:21.154 DEBUG:boto:auth_path=/pklxkd-61/?prefix=obj29 2023-05-31T21:03:21.154 DEBUG:boto:Method: GET 2023-05-31T21:03:21.155 DEBUG:boto:Path: /pklxkd-61/?prefix=obj29 2023-05-31T21:03:21.155 DEBUG:boto:Data: 2023-05-31T21:03:21.156 DEBUG:boto:Headers: {} 2023-05-31T21:03:21.156 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T21:03:21.156 DEBUG:boto:Port: 8000 2023-05-31T21:03:21.157 DEBUG:boto:Params: {} 2023-05-31T21:03:21.157 DEBUG:boto:Token: None 2023-05-31T21:03:21.158 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 21:03:21 GMT /pklxkd-61/ 2023-05-31T21:03:21.158 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:H9PQFgrR8ica+qPwTl/4XbpLrOQ= 2023-05-31T21:03:21.158 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 21:03:21 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:H9PQFgrR8ica+qPwTl/4XbpLrOQ=', 'Content-Length': '0'} 2023-05-31T21:03:21.162 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000004964b9d242357e94-006477b619-1630-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 21:03:21 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:03:21.163 DEBUG:boto:b'pklxkd-61obj291000falseobj292023-05-31T20:59:45.888Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T21:03:21.163 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj31 2023-05-31T21:03:21.164 DEBUG:boto:path=/pklxkd-61/obj31 2023-05-31T21:03:21.164 DEBUG:boto:auth_path=/pklxkd-61/obj31 2023-05-31T21:03:21.165 DEBUG:boto:path=/pklxkd-61/obj31?versionId=null 2023-05-31T21:03:21.165 DEBUG:boto:auth_path=/pklxkd-61/obj31?versionId=null 2023-05-31T21:03:21.165 DEBUG:boto:Method: GET 2023-05-31T21:03:21.166 DEBUG:boto:Path: /pklxkd-61/obj31?versionId=null 2023-05-31T21:03:21.166 DEBUG:boto:Data: 2023-05-31T21:03:21.167 DEBUG:boto:Headers: {} 2023-05-31T21:03:21.167 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T21:03:21.167 DEBUG:boto:Port: 8001 2023-05-31T21:03:21.168 DEBUG:boto:Params: {} 2023-05-31T21:03:21.168 DEBUG:boto:Token: None 2023-05-31T21:03:21.169 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 21:03:21 GMT /pklxkd-61/obj31?versionId=null 2023-05-31T21:03:21.169 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:CetiXJWQh70gQTis/NjL5wsfs0A= 2023-05-31T21:03:21.169 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 21:03:21 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:CetiXJWQh70gQTis/NjL5wsfs0A=', 'Content-Length': '0'} 2023-05-31T21:03:21.171 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:59:46 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000035a73ced9685df7b-006477b619-1f4e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 21:03:21 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:03:21.172 DEBUG:boto:path=/pklxkd-61/obj31 2023-05-31T21:03:21.172 DEBUG:boto:auth_path=/pklxkd-61/obj31 2023-05-31T21:03:21.172 DEBUG:boto:path=/pklxkd-61/obj31?versionId=null 2023-05-31T21:03:21.173 DEBUG:boto:auth_path=/pklxkd-61/obj31?versionId=null 2023-05-31T21:03:21.173 DEBUG:boto:Method: GET 2023-05-31T21:03:21.174 DEBUG:boto:Path: /pklxkd-61/obj31?versionId=null 2023-05-31T21:03:21.174 DEBUG:boto:Data: 2023-05-31T21:03:21.174 DEBUG:boto:Headers: {} 2023-05-31T21:03:21.175 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T21:03:21.175 DEBUG:boto:Port: 8000 2023-05-31T21:03:21.175 DEBUG:boto:Params: {} 2023-05-31T21:03:21.176 DEBUG:boto:Token: None 2023-05-31T21:03:21.176 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 21:03:21 GMT /pklxkd-61/obj31?versionId=null 2023-05-31T21:03:21.177 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:CetiXJWQh70gQTis/NjL5wsfs0A= 2023-05-31T21:03:21.177 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 21:03:21 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:CetiXJWQh70gQTis/NjL5wsfs0A=', 'Content-Length': '0'} 2023-05-31T21:03:21.179 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:59:46 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004b0776128194ec67-006477b619-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 21:03:21 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:03:21.220 DEBUG:boto:path=/pklxkd-61/obj31 2023-05-31T21:03:21.221 DEBUG:boto:auth_path=/pklxkd-61/obj31 2023-05-31T21:03:21.221 DEBUG:boto:path=/pklxkd-61/obj31?versionId=null 2023-05-31T21:03:21.222 DEBUG:boto:auth_path=/pklxkd-61/obj31?versionId=null 2023-05-31T21:03:21.222 DEBUG:boto:Method: HEAD 2023-05-31T21:03:21.222 DEBUG:boto:Path: /pklxkd-61/obj31?versionId=null 2023-05-31T21:03:21.223 DEBUG:boto:Data: 2023-05-31T21:03:21.223 DEBUG:boto:Headers: {} 2023-05-31T21:03:21.223 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T21:03:21.223 DEBUG:boto:Port: 8001 2023-05-31T21:03:21.224 DEBUG:boto:Params: {} 2023-05-31T21:03:21.224 DEBUG:boto:Token: None 2023-05-31T21:03:21.225 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 21:03:21 GMT /pklxkd-61/obj31?versionId=null 2023-05-31T21:03:21.225 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:C/1mW4GnbCgP9srhlZuj5kjFzx0= 2023-05-31T21:03:21.226 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 21:03:21 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:C/1mW4GnbCgP9srhlZuj5kjFzx0=', 'Content-Length': '0'} 2023-05-31T21:03:21.227 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:59:46 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004f9882238caf2100-006477b619-1f4e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 21:03:21 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:03:21.228 DEBUG:boto:path=/pklxkd-61/obj31 2023-05-31T21:03:21.228 DEBUG:boto:auth_path=/pklxkd-61/obj31 2023-05-31T21:03:21.229 DEBUG:boto:path=/pklxkd-61/obj31?versionId=null 2023-05-31T21:03:21.229 DEBUG:boto:auth_path=/pklxkd-61/obj31?versionId=null 2023-05-31T21:03:21.229 DEBUG:boto:Method: HEAD 2023-05-31T21:03:21.230 DEBUG:boto:Path: /pklxkd-61/obj31?versionId=null 2023-05-31T21:03:21.230 DEBUG:boto:Data: 2023-05-31T21:03:21.230 DEBUG:boto:Headers: {} 2023-05-31T21:03:21.231 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T21:03:21.231 DEBUG:boto:Port: 8000 2023-05-31T21:03:21.231 DEBUG:boto:Params: {} 2023-05-31T21:03:21.232 DEBUG:boto:Token: None 2023-05-31T21:03:21.232 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 21:03:21 GMT /pklxkd-61/obj31?versionId=null 2023-05-31T21:03:21.233 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:C/1mW4GnbCgP9srhlZuj5kjFzx0= 2023-05-31T21:03:21.233 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 21:03:21 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:C/1mW4GnbCgP9srhlZuj5kjFzx0=', 'Content-Length': '0'} 2023-05-31T21:03:21.235 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:59:46 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000adf10ed5a24849d2-006477b619-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 21:03:21 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:03:21.235 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj31 2023-05-31T21:03:21.235 DEBUG:boto:path=/pklxkd-61/obj31 2023-05-31T21:03:21.236 DEBUG:boto:auth_path=/pklxkd-61/obj31 2023-05-31T21:03:21.236 DEBUG:boto:path=/pklxkd-61/obj31?versionId=null 2023-05-31T21:03:21.237 DEBUG:boto:auth_path=/pklxkd-61/obj31?versionId=null 2023-05-31T21:03:21.237 DEBUG:boto:Method: GET 2023-05-31T21:03:21.237 DEBUG:boto:Path: /pklxkd-61/obj31?versionId=null 2023-05-31T21:03:21.238 DEBUG:boto:Data: 2023-05-31T21:03:21.238 DEBUG:boto:Headers: {} 2023-05-31T21:03:21.238 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T21:03:21.239 DEBUG:boto:Port: 8001 2023-05-31T21:03:21.239 DEBUG:boto:Params: {} 2023-05-31T21:03:21.240 DEBUG:boto:Token: None 2023-05-31T21:03:21.240 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 21:03:21 GMT /pklxkd-61/obj31?versionId=null 2023-05-31T21:03:21.240 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:CetiXJWQh70gQTis/NjL5wsfs0A= 2023-05-31T21:03:21.241 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 21:03:21 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:CetiXJWQh70gQTis/NjL5wsfs0A=', 'Content-Length': '0'} 2023-05-31T21:03:21.242 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:59:46 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000674626393fd7e9a6-006477b619-1f4e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 21:03:21 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:03:21.243 DEBUG:boto:path=/pklxkd-61/obj31 2023-05-31T21:03:21.243 DEBUG:boto:auth_path=/pklxkd-61/obj31 2023-05-31T21:03:21.244 DEBUG:boto:path=/pklxkd-61/obj31?versionId=null 2023-05-31T21:03:21.244 DEBUG:boto:auth_path=/pklxkd-61/obj31?versionId=null 2023-05-31T21:03:21.244 DEBUG:boto:Method: GET 2023-05-31T21:03:21.245 DEBUG:boto:Path: /pklxkd-61/obj31?versionId=null 2023-05-31T21:03:21.245 DEBUG:boto:Data: 2023-05-31T21:03:21.246 DEBUG:boto:Headers: {} 2023-05-31T21:03:21.246 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T21:03:21.246 DEBUG:boto:Port: 8000 2023-05-31T21:03:21.247 DEBUG:boto:Params: {} 2023-05-31T21:03:21.247 DEBUG:boto:Token: None 2023-05-31T21:03:21.248 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 21:03:21 GMT /pklxkd-61/obj31?versionId=null 2023-05-31T21:03:21.249 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:CetiXJWQh70gQTis/NjL5wsfs0A= 2023-05-31T21:03:21.250 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 21:03:21 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:CetiXJWQh70gQTis/NjL5wsfs0A=', 'Content-Length': '0'} 2023-05-31T21:03:21.252 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:59:46 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d6b9c573a1c09aa0-006477b619-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 21:03:21 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:03:21.253 DEBUG:boto:path=/pklxkd-61/ 2023-05-31T21:03:21.254 DEBUG:boto:auth_path=/pklxkd-61/ 2023-05-31T21:03:21.254 DEBUG:boto:path=/pklxkd-61/?prefix=obj31 2023-05-31T21:03:21.254 DEBUG:boto:auth_path=/pklxkd-61/?prefix=obj31 2023-05-31T21:03:21.255 DEBUG:boto:Method: GET 2023-05-31T21:03:21.255 DEBUG:boto:Path: /pklxkd-61/?prefix=obj31 2023-05-31T21:03:21.255 DEBUG:boto:Data: 2023-05-31T21:03:21.256 DEBUG:boto:Headers: {} 2023-05-31T21:03:21.256 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T21:03:21.256 DEBUG:boto:Port: 8001 2023-05-31T21:03:21.257 DEBUG:boto:Params: {} 2023-05-31T21:03:21.257 DEBUG:boto:Token: None 2023-05-31T21:03:21.258 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 21:03:21 GMT /pklxkd-61/ 2023-05-31T21:03:21.258 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:H9PQFgrR8ica+qPwTl/4XbpLrOQ= 2023-05-31T21:03:21.259 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 21:03:21 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:H9PQFgrR8ica+qPwTl/4XbpLrOQ=', 'Content-Length': '0'} 2023-05-31T21:03:21.261 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000001dd9325ec58fce6-006477b619-1f4e-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 21:03:21 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:03:21.261 DEBUG:boto:b'pklxkd-61obj311000falseobj312023-05-31T20:59:46.280Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T21:03:21.262 DEBUG:boto:path=/pklxkd-61/ 2023-05-31T21:03:21.262 DEBUG:boto:auth_path=/pklxkd-61/ 2023-05-31T21:03:21.263 DEBUG:boto:path=/pklxkd-61/?prefix=obj31 2023-05-31T21:03:21.263 DEBUG:boto:auth_path=/pklxkd-61/?prefix=obj31 2023-05-31T21:03:21.263 DEBUG:boto:Method: GET 2023-05-31T21:03:21.264 DEBUG:boto:Path: /pklxkd-61/?prefix=obj31 2023-05-31T21:03:21.264 DEBUG:boto:Data: 2023-05-31T21:03:21.264 DEBUG:boto:Headers: {} 2023-05-31T21:03:21.265 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T21:03:21.265 DEBUG:boto:Port: 8000 2023-05-31T21:03:21.266 DEBUG:boto:Params: {} 2023-05-31T21:03:21.266 DEBUG:boto:Token: None 2023-05-31T21:03:21.267 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 21:03:21 GMT /pklxkd-61/ 2023-05-31T21:03:21.267 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:H9PQFgrR8ica+qPwTl/4XbpLrOQ= 2023-05-31T21:03:21.268 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 21:03:21 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:H9PQFgrR8ica+qPwTl/4XbpLrOQ=', 'Content-Length': '0'} 2023-05-31T21:03:21.272 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000006e05c0ba3f4a9cdc-006477b619-1630-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 21:03:21 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:03:21.273 DEBUG:boto:b'pklxkd-61obj311000falseobj312023-05-31T20:59:46.280Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T21:03:21.273 DEBUG:boto:path=/pklxkd-61/obj31 2023-05-31T21:03:21.273 DEBUG:boto:auth_path=/pklxkd-61/obj31 2023-05-31T21:03:21.274 DEBUG:boto:Method: HEAD 2023-05-31T21:03:21.274 DEBUG:boto:Path: /pklxkd-61/obj31 2023-05-31T21:03:21.275 DEBUG:boto:Data: 2023-05-31T21:03:21.275 DEBUG:boto:Headers: {} 2023-05-31T21:03:21.276 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T21:03:21.276 DEBUG:boto:Port: 8001 2023-05-31T21:03:21.276 DEBUG:boto:Params: {} 2023-05-31T21:03:21.277 DEBUG:boto:Token: None 2023-05-31T21:03:21.277 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 21:03:21 GMT /pklxkd-61/obj31 2023-05-31T21:03:21.278 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:0DRbbJjerl9DoXws+eLh0VfJLqo= 2023-05-31T21:03:21.278 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 21:03:21 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:0DRbbJjerl9DoXws+eLh0VfJLqo=', 'Content-Length': '0'} 2023-05-31T21:03:21.280 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:59:46 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000060b9ee741ac5f4fd-006477b619-1f4e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 21:03:21 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:03:21.280 DEBUG:boto:path=/pklxkd-61/obj31 2023-05-31T21:03:21.281 DEBUG:boto:auth_path=/pklxkd-61/obj31 2023-05-31T21:03:21.281 DEBUG:boto:Method: HEAD 2023-05-31T21:03:21.282 DEBUG:boto:Path: /pklxkd-61/obj31 2023-05-31T21:03:21.282 DEBUG:boto:Data: 2023-05-31T21:03:21.282 DEBUG:boto:Headers: {} 2023-05-31T21:03:21.283 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T21:03:21.283 DEBUG:boto:Port: 8000 2023-05-31T21:03:21.283 DEBUG:boto:Params: {} 2023-05-31T21:03:21.284 DEBUG:boto:Token: None 2023-05-31T21:03:21.284 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 21:03:21 GMT /pklxkd-61/obj31 2023-05-31T21:03:21.285 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:0DRbbJjerl9DoXws+eLh0VfJLqo= 2023-05-31T21:03:21.286 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 21:03:21 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:0DRbbJjerl9DoXws+eLh0VfJLqo=', 'Content-Length': '0'} 2023-05-31T21:03:21.288 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:59:46 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001ec5e492292b6b10-006477b619-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 21:03:21 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:03:21.289 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj31 2023-05-31T21:03:21.290 DEBUG:boto:path=/pklxkd-61/obj31 2023-05-31T21:03:21.290 DEBUG:boto:auth_path=/pklxkd-61/obj31 2023-05-31T21:03:21.290 DEBUG:boto:Method: GET 2023-05-31T21:03:21.291 DEBUG:boto:Path: /pklxkd-61/obj31 2023-05-31T21:03:21.291 DEBUG:boto:Data: 2023-05-31T21:03:21.291 DEBUG:boto:Headers: {} 2023-05-31T21:03:21.292 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T21:03:21.292 DEBUG:boto:Port: 8001 2023-05-31T21:03:21.293 DEBUG:boto:Params: {} 2023-05-31T21:03:21.293 DEBUG:boto:Token: None 2023-05-31T21:03:21.293 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 21:03:21 GMT /pklxkd-61/obj31 2023-05-31T21:03:21.294 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:er92vDKgmSksRKMkz7tPOE01yfo= 2023-05-31T21:03:21.294 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 21:03:21 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:er92vDKgmSksRKMkz7tPOE01yfo=', 'Content-Length': '0'} 2023-05-31T21:03:21.296 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:59:46 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000270c2c2a041eed7d-006477b619-1f4e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 21:03:21 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:03:21.296 DEBUG:boto:path=/pklxkd-61/obj31 2023-05-31T21:03:21.297 DEBUG:boto:auth_path=/pklxkd-61/obj31 2023-05-31T21:03:21.297 DEBUG:boto:Method: GET 2023-05-31T21:03:21.298 DEBUG:boto:Path: /pklxkd-61/obj31 2023-05-31T21:03:21.298 DEBUG:boto:Data: 2023-05-31T21:03:21.298 DEBUG:boto:Headers: {} 2023-05-31T21:03:21.299 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T21:03:21.299 DEBUG:boto:Port: 8000 2023-05-31T21:03:21.299 DEBUG:boto:Params: {} 2023-05-31T21:03:21.300 DEBUG:boto:Token: None 2023-05-31T21:03:21.300 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 21:03:21 GMT /pklxkd-61/obj31 2023-05-31T21:03:21.301 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:er92vDKgmSksRKMkz7tPOE01yfo= 2023-05-31T21:03:21.301 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 21:03:21 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:er92vDKgmSksRKMkz7tPOE01yfo=', 'Content-Length': '0'} 2023-05-31T21:03:21.303 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:59:46 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000db711e30d1293ebe-006477b619-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 21:03:21 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:03:21.344 DEBUG:boto:path=/pklxkd-61/ 2023-05-31T21:03:21.345 DEBUG:boto:auth_path=/pklxkd-61/ 2023-05-31T21:03:21.345 DEBUG:boto:path=/pklxkd-61/?prefix=obj31 2023-05-31T21:03:21.345 DEBUG:boto:auth_path=/pklxkd-61/?prefix=obj31 2023-05-31T21:03:21.346 DEBUG:boto:Method: GET 2023-05-31T21:03:21.346 DEBUG:boto:Path: /pklxkd-61/?prefix=obj31 2023-05-31T21:03:21.347 DEBUG:boto:Data: 2023-05-31T21:03:21.347 DEBUG:boto:Headers: {} 2023-05-31T21:03:21.347 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T21:03:21.348 DEBUG:boto:Port: 8001 2023-05-31T21:03:21.348 DEBUG:boto:Params: {} 2023-05-31T21:03:21.348 DEBUG:boto:Token: None 2023-05-31T21:03:21.349 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 21:03:21 GMT /pklxkd-61/ 2023-05-31T21:03:21.349 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:H9PQFgrR8ica+qPwTl/4XbpLrOQ= 2023-05-31T21:03:21.350 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 21:03:21 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:H9PQFgrR8ica+qPwTl/4XbpLrOQ=', 'Content-Length': '0'} 2023-05-31T21:03:21.352 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000058b04ebd5b431cd8-006477b619-1f4e-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 21:03:21 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:03:21.352 DEBUG:boto:b'pklxkd-61obj311000falseobj312023-05-31T20:59:46.280Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T21:03:21.353 DEBUG:boto:path=/pklxkd-61/ 2023-05-31T21:03:21.353 DEBUG:boto:auth_path=/pklxkd-61/ 2023-05-31T21:03:21.354 DEBUG:boto:path=/pklxkd-61/?prefix=obj31 2023-05-31T21:03:21.354 DEBUG:boto:auth_path=/pklxkd-61/?prefix=obj31 2023-05-31T21:03:21.354 DEBUG:boto:Method: GET 2023-05-31T21:03:21.355 DEBUG:boto:Path: /pklxkd-61/?prefix=obj31 2023-05-31T21:03:21.355 DEBUG:boto:Data: 2023-05-31T21:03:21.355 DEBUG:boto:Headers: {} 2023-05-31T21:03:21.356 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T21:03:21.356 DEBUG:boto:Port: 8000 2023-05-31T21:03:21.357 DEBUG:boto:Params: {} 2023-05-31T21:03:21.357 DEBUG:boto:Token: None 2023-05-31T21:03:21.358 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 21:03:21 GMT /pklxkd-61/ 2023-05-31T21:03:21.358 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:H9PQFgrR8ica+qPwTl/4XbpLrOQ= 2023-05-31T21:03:21.359 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 21:03:21 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:H9PQFgrR8ica+qPwTl/4XbpLrOQ=', 'Content-Length': '0'} 2023-05-31T21:03:21.363 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000afb90651f9ba8b23-006477b619-1630-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 21:03:21 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:03:21.363 DEBUG:boto:b'pklxkd-61obj311000falseobj312023-05-31T20:59:46.280Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T21:03:21.364 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj33 2023-05-31T21:03:21.364 DEBUG:boto:path=/pklxkd-61/obj33 2023-05-31T21:03:21.365 DEBUG:boto:auth_path=/pklxkd-61/obj33 2023-05-31T21:03:21.365 DEBUG:boto:path=/pklxkd-61/obj33?versionId=null 2023-05-31T21:03:21.365 DEBUG:boto:auth_path=/pklxkd-61/obj33?versionId=null 2023-05-31T21:03:21.366 DEBUG:boto:Method: GET 2023-05-31T21:03:21.366 DEBUG:boto:Path: /pklxkd-61/obj33?versionId=null 2023-05-31T21:03:21.366 DEBUG:boto:Data: 2023-05-31T21:03:21.367 DEBUG:boto:Headers: {} 2023-05-31T21:03:21.367 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T21:03:21.368 DEBUG:boto:Port: 8001 2023-05-31T21:03:21.368 DEBUG:boto:Params: {} 2023-05-31T21:03:21.368 DEBUG:boto:Token: None 2023-05-31T21:03:21.369 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 21:03:21 GMT /pklxkd-61/obj33?versionId=null 2023-05-31T21:03:21.369 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:NhvP6sdrZXwELgc3/zTy8jYEMNk= 2023-05-31T21:03:21.370 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 21:03:21 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:NhvP6sdrZXwELgc3/zTy8jYEMNk=', 'Content-Length': '0'} 2023-05-31T21:03:21.371 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:59:44 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d90294d28cea2b6e-006477b619-1f4e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 21:03:21 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:03:21.372 DEBUG:boto:path=/pklxkd-61/obj33 2023-05-31T21:03:21.372 DEBUG:boto:auth_path=/pklxkd-61/obj33 2023-05-31T21:03:21.373 DEBUG:boto:path=/pklxkd-61/obj33?versionId=null 2023-05-31T21:03:21.373 DEBUG:boto:auth_path=/pklxkd-61/obj33?versionId=null 2023-05-31T21:03:21.373 DEBUG:boto:Method: GET 2023-05-31T21:03:21.374 DEBUG:boto:Path: /pklxkd-61/obj33?versionId=null 2023-05-31T21:03:21.374 DEBUG:boto:Data: 2023-05-31T21:03:21.375 DEBUG:boto:Headers: {} 2023-05-31T21:03:21.375 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T21:03:21.375 DEBUG:boto:Port: 8000 2023-05-31T21:03:21.376 DEBUG:boto:Params: {} 2023-05-31T21:03:21.376 DEBUG:boto:Token: None 2023-05-31T21:03:21.376 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 21:03:21 GMT /pklxkd-61/obj33?versionId=null 2023-05-31T21:03:21.377 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:NhvP6sdrZXwELgc3/zTy8jYEMNk= 2023-05-31T21:03:21.377 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 21:03:21 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:NhvP6sdrZXwELgc3/zTy8jYEMNk=', 'Content-Length': '0'} 2023-05-31T21:03:21.379 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:59:44 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000047795e92c47b9dc1-006477b619-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 21:03:21 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:03:21.380 DEBUG:boto:path=/pklxkd-61/obj33 2023-05-31T21:03:21.380 DEBUG:boto:auth_path=/pklxkd-61/obj33 2023-05-31T21:03:21.381 DEBUG:boto:path=/pklxkd-61/obj33?versionId=null 2023-05-31T21:03:21.381 DEBUG:boto:auth_path=/pklxkd-61/obj33?versionId=null 2023-05-31T21:03:21.382 DEBUG:boto:Method: HEAD 2023-05-31T21:03:21.382 DEBUG:boto:Path: /pklxkd-61/obj33?versionId=null 2023-05-31T21:03:21.383 DEBUG:boto:Data: 2023-05-31T21:03:21.384 DEBUG:boto:Headers: {} 2023-05-31T21:03:21.384 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T21:03:21.385 DEBUG:boto:Port: 8001 2023-05-31T21:03:21.385 DEBUG:boto:Params: {} 2023-05-31T21:03:21.385 DEBUG:boto:Token: None 2023-05-31T21:03:21.386 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 21:03:21 GMT /pklxkd-61/obj33?versionId=null 2023-05-31T21:03:21.386 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:oriu3loqDYIEcPeEcyweCcI58O8= 2023-05-31T21:03:21.387 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 21:03:21 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:oriu3loqDYIEcPeEcyweCcI58O8=', 'Content-Length': '0'} 2023-05-31T21:03:21.389 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:59:44 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003b2644ec2e55b1d5-006477b619-1f4e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 21:03:21 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:03:21.389 DEBUG:boto:path=/pklxkd-61/obj33 2023-05-31T21:03:21.390 DEBUG:boto:auth_path=/pklxkd-61/obj33 2023-05-31T21:03:21.390 DEBUG:boto:path=/pklxkd-61/obj33?versionId=null 2023-05-31T21:03:21.390 DEBUG:boto:auth_path=/pklxkd-61/obj33?versionId=null 2023-05-31T21:03:21.391 DEBUG:boto:Method: HEAD 2023-05-31T21:03:21.391 DEBUG:boto:Path: /pklxkd-61/obj33?versionId=null 2023-05-31T21:03:21.391 DEBUG:boto:Data: 2023-05-31T21:03:21.392 DEBUG:boto:Headers: {} 2023-05-31T21:03:21.392 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T21:03:21.392 DEBUG:boto:Port: 8000 2023-05-31T21:03:21.393 DEBUG:boto:Params: {} 2023-05-31T21:03:21.393 DEBUG:boto:Token: None 2023-05-31T21:03:21.393 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 21:03:21 GMT /pklxkd-61/obj33?versionId=null 2023-05-31T21:03:21.394 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:oriu3loqDYIEcPeEcyweCcI58O8= 2023-05-31T21:03:21.394 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 21:03:21 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:oriu3loqDYIEcPeEcyweCcI58O8=', 'Content-Length': '0'} 2023-05-31T21:03:21.397 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:59:44 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000eca7fb38de208e5c-006477b619-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 21:03:21 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:03:21.398 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj33 2023-05-31T21:03:21.398 DEBUG:boto:path=/pklxkd-61/obj33 2023-05-31T21:03:21.398 DEBUG:boto:auth_path=/pklxkd-61/obj33 2023-05-31T21:03:21.399 DEBUG:boto:path=/pklxkd-61/obj33?versionId=null 2023-05-31T21:03:21.399 DEBUG:boto:auth_path=/pklxkd-61/obj33?versionId=null 2023-05-31T21:03:21.399 DEBUG:boto:Method: GET 2023-05-31T21:03:21.400 DEBUG:boto:Path: /pklxkd-61/obj33?versionId=null 2023-05-31T21:03:21.400 DEBUG:boto:Data: 2023-05-31T21:03:21.401 DEBUG:boto:Headers: {} 2023-05-31T21:03:21.401 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T21:03:21.401 DEBUG:boto:Port: 8001 2023-05-31T21:03:21.402 DEBUG:boto:Params: {} 2023-05-31T21:03:21.402 DEBUG:boto:Token: None 2023-05-31T21:03:21.402 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 21:03:21 GMT /pklxkd-61/obj33?versionId=null 2023-05-31T21:03:21.403 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:NhvP6sdrZXwELgc3/zTy8jYEMNk= 2023-05-31T21:03:21.403 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 21:03:21 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:NhvP6sdrZXwELgc3/zTy8jYEMNk=', 'Content-Length': '0'} 2023-05-31T21:03:21.405 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:59:44 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d37c113a9eb153d3-006477b619-1f4e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 21:03:21 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:03:21.448 DEBUG:boto:path=/pklxkd-61/obj33 2023-05-31T21:03:21.449 DEBUG:boto:auth_path=/pklxkd-61/obj33 2023-05-31T21:03:21.449 DEBUG:boto:path=/pklxkd-61/obj33?versionId=null 2023-05-31T21:03:21.449 DEBUG:boto:auth_path=/pklxkd-61/obj33?versionId=null 2023-05-31T21:03:21.450 DEBUG:boto:Method: GET 2023-05-31T21:03:21.450 DEBUG:boto:Path: /pklxkd-61/obj33?versionId=null 2023-05-31T21:03:21.450 DEBUG:boto:Data: 2023-05-31T21:03:21.451 DEBUG:boto:Headers: {} 2023-05-31T21:03:21.451 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T21:03:21.451 DEBUG:boto:Port: 8000 2023-05-31T21:03:21.452 DEBUG:boto:Params: {} 2023-05-31T21:03:21.452 DEBUG:boto:Token: None 2023-05-31T21:03:21.453 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 21:03:21 GMT /pklxkd-61/obj33?versionId=null 2023-05-31T21:03:21.453 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:NhvP6sdrZXwELgc3/zTy8jYEMNk= 2023-05-31T21:03:21.453 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 21:03:21 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:NhvP6sdrZXwELgc3/zTy8jYEMNk=', 'Content-Length': '0'} 2023-05-31T21:03:21.455 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:59:44 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000013811ca334fa476b-006477b619-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 21:03:21 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:03:21.456 DEBUG:boto:path=/pklxkd-61/ 2023-05-31T21:03:21.456 DEBUG:boto:auth_path=/pklxkd-61/ 2023-05-31T21:03:21.456 DEBUG:boto:path=/pklxkd-61/?prefix=obj33 2023-05-31T21:03:21.457 DEBUG:boto:auth_path=/pklxkd-61/?prefix=obj33 2023-05-31T21:03:21.457 DEBUG:boto:Method: GET 2023-05-31T21:03:21.458 DEBUG:boto:Path: /pklxkd-61/?prefix=obj33 2023-05-31T21:03:21.458 DEBUG:boto:Data: 2023-05-31T21:03:21.458 DEBUG:boto:Headers: {} 2023-05-31T21:03:21.459 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T21:03:21.459 DEBUG:boto:Port: 8001 2023-05-31T21:03:21.459 DEBUG:boto:Params: {} 2023-05-31T21:03:21.460 DEBUG:boto:Token: None 2023-05-31T21:03:21.460 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 21:03:21 GMT /pklxkd-61/ 2023-05-31T21:03:21.461 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:H9PQFgrR8ica+qPwTl/4XbpLrOQ= 2023-05-31T21:03:21.461 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 21:03:21 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:H9PQFgrR8ica+qPwTl/4XbpLrOQ=', 'Content-Length': '0'} 2023-05-31T21:03:21.463 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000007f3ec565e72a0c3f-006477b619-1f4e-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 21:03:21 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:03:21.464 DEBUG:boto:b'pklxkd-61obj331000falseobj332023-05-31T20:59:44.466Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T21:03:21.464 DEBUG:boto:path=/pklxkd-61/ 2023-05-31T21:03:21.465 DEBUG:boto:auth_path=/pklxkd-61/ 2023-05-31T21:03:21.465 DEBUG:boto:path=/pklxkd-61/?prefix=obj33 2023-05-31T21:03:21.466 DEBUG:boto:auth_path=/pklxkd-61/?prefix=obj33 2023-05-31T21:03:21.466 DEBUG:boto:Method: GET 2023-05-31T21:03:21.466 DEBUG:boto:Path: /pklxkd-61/?prefix=obj33 2023-05-31T21:03:21.467 DEBUG:boto:Data: 2023-05-31T21:03:21.467 DEBUG:boto:Headers: {} 2023-05-31T21:03:21.467 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T21:03:21.468 DEBUG:boto:Port: 8000 2023-05-31T21:03:21.468 DEBUG:boto:Params: {} 2023-05-31T21:03:21.469 DEBUG:boto:Token: None 2023-05-31T21:03:21.469 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 21:03:21 GMT /pklxkd-61/ 2023-05-31T21:03:21.469 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:H9PQFgrR8ica+qPwTl/4XbpLrOQ= 2023-05-31T21:03:21.470 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 21:03:21 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:H9PQFgrR8ica+qPwTl/4XbpLrOQ=', 'Content-Length': '0'} 2023-05-31T21:03:21.473 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000002d6ff632ed59f5a5-006477b619-1630-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 21:03:21 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:03:21.474 DEBUG:boto:b'pklxkd-61obj331000falseobj332023-05-31T20:59:44.466Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T21:03:21.474 DEBUG:boto:path=/pklxkd-61/obj33 2023-05-31T21:03:21.475 DEBUG:boto:auth_path=/pklxkd-61/obj33 2023-05-31T21:03:21.475 DEBUG:boto:Method: HEAD 2023-05-31T21:03:21.476 DEBUG:boto:Path: /pklxkd-61/obj33 2023-05-31T21:03:21.476 DEBUG:boto:Data: 2023-05-31T21:03:21.476 DEBUG:boto:Headers: {} 2023-05-31T21:03:21.477 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T21:03:21.477 DEBUG:boto:Port: 8001 2023-05-31T21:03:21.477 DEBUG:boto:Params: {} 2023-05-31T21:03:21.478 DEBUG:boto:Token: None 2023-05-31T21:03:21.478 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 21:03:21 GMT /pklxkd-61/obj33 2023-05-31T21:03:21.478 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:g18xj6y+a9qnBL/97FtjMwUf2Wg= 2023-05-31T21:03:21.479 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 21:03:21 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:g18xj6y+a9qnBL/97FtjMwUf2Wg=', 'Content-Length': '0'} 2023-05-31T21:03:21.480 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:59:44 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000bb6de631ea92ede5-006477b619-1f4e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 21:03:21 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:03:21.481 DEBUG:boto:path=/pklxkd-61/obj33 2023-05-31T21:03:21.481 DEBUG:boto:auth_path=/pklxkd-61/obj33 2023-05-31T21:03:21.482 DEBUG:boto:Method: HEAD 2023-05-31T21:03:21.482 DEBUG:boto:Path: /pklxkd-61/obj33 2023-05-31T21:03:21.482 DEBUG:boto:Data: 2023-05-31T21:03:21.483 DEBUG:boto:Headers: {} 2023-05-31T21:03:21.483 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T21:03:21.483 DEBUG:boto:Port: 8000 2023-05-31T21:03:21.484 DEBUG:boto:Params: {} 2023-05-31T21:03:21.484 DEBUG:boto:Token: None 2023-05-31T21:03:21.485 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 21:03:21 GMT /pklxkd-61/obj33 2023-05-31T21:03:21.485 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:g18xj6y+a9qnBL/97FtjMwUf2Wg= 2023-05-31T21:03:21.486 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 21:03:21 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:g18xj6y+a9qnBL/97FtjMwUf2Wg=', 'Content-Length': '0'} 2023-05-31T21:03:21.487 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:59:44 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008858bfc653cc18a1-006477b619-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 21:03:21 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:03:21.488 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj33 2023-05-31T21:03:21.488 DEBUG:boto:path=/pklxkd-61/obj33 2023-05-31T21:03:21.489 DEBUG:boto:auth_path=/pklxkd-61/obj33 2023-05-31T21:03:21.489 DEBUG:boto:Method: GET 2023-05-31T21:03:21.489 DEBUG:boto:Path: /pklxkd-61/obj33 2023-05-31T21:03:21.490 DEBUG:boto:Data: 2023-05-31T21:03:21.490 DEBUG:boto:Headers: {} 2023-05-31T21:03:21.491 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T21:03:21.491 DEBUG:boto:Port: 8001 2023-05-31T21:03:21.491 DEBUG:boto:Params: {} 2023-05-31T21:03:21.492 DEBUG:boto:Token: None 2023-05-31T21:03:21.492 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 21:03:21 GMT /pklxkd-61/obj33 2023-05-31T21:03:21.492 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:2Vw2k4mG4fCSe/WgXYlNdHSbheE= 2023-05-31T21:03:21.493 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 21:03:21 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:2Vw2k4mG4fCSe/WgXYlNdHSbheE=', 'Content-Length': '0'} 2023-05-31T21:03:21.494 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:59:44 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000623dd0c187b30972-006477b619-1f4e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 21:03:21 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:03:21.495 DEBUG:boto:path=/pklxkd-61/obj33 2023-05-31T21:03:21.496 DEBUG:boto:auth_path=/pklxkd-61/obj33 2023-05-31T21:03:21.496 DEBUG:boto:Method: GET 2023-05-31T21:03:21.497 DEBUG:boto:Path: /pklxkd-61/obj33 2023-05-31T21:03:21.497 DEBUG:boto:Data: 2023-05-31T21:03:21.498 DEBUG:boto:Headers: {} 2023-05-31T21:03:21.499 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T21:03:21.499 DEBUG:boto:Port: 8000 2023-05-31T21:03:21.500 DEBUG:boto:Params: {} 2023-05-31T21:03:21.500 DEBUG:boto:Token: None 2023-05-31T21:03:21.501 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 21:03:21 GMT /pklxkd-61/obj33 2023-05-31T21:03:21.501 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:2Vw2k4mG4fCSe/WgXYlNdHSbheE= 2023-05-31T21:03:21.502 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 21:03:21 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:2Vw2k4mG4fCSe/WgXYlNdHSbheE=', 'Content-Length': '0'} 2023-05-31T21:03:21.503 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:59:44 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001506a0aaf82249a5-006477b619-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 21:03:21 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:03:21.548 DEBUG:boto:path=/pklxkd-61/ 2023-05-31T21:03:21.549 DEBUG:boto:auth_path=/pklxkd-61/ 2023-05-31T21:03:21.549 DEBUG:boto:path=/pklxkd-61/?prefix=obj33 2023-05-31T21:03:21.550 DEBUG:boto:auth_path=/pklxkd-61/?prefix=obj33 2023-05-31T21:03:21.550 DEBUG:boto:Method: GET 2023-05-31T21:03:21.550 DEBUG:boto:Path: /pklxkd-61/?prefix=obj33 2023-05-31T21:03:21.551 DEBUG:boto:Data: 2023-05-31T21:03:21.551 DEBUG:boto:Headers: {} 2023-05-31T21:03:21.551 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T21:03:21.552 DEBUG:boto:Port: 8001 2023-05-31T21:03:21.552 DEBUG:boto:Params: {} 2023-05-31T21:03:21.553 DEBUG:boto:Token: None 2023-05-31T21:03:21.553 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 21:03:21 GMT /pklxkd-61/ 2023-05-31T21:03:21.553 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:H9PQFgrR8ica+qPwTl/4XbpLrOQ= 2023-05-31T21:03:21.554 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 21:03:21 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:H9PQFgrR8ica+qPwTl/4XbpLrOQ=', 'Content-Length': '0'} 2023-05-31T21:03:21.556 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000002f0a362f52aeabc7-006477b619-1f4e-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 21:03:21 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:03:21.557 DEBUG:boto:b'pklxkd-61obj331000falseobj332023-05-31T20:59:44.466Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T21:03:21.557 DEBUG:boto:path=/pklxkd-61/ 2023-05-31T21:03:21.558 DEBUG:boto:auth_path=/pklxkd-61/ 2023-05-31T21:03:21.558 DEBUG:boto:path=/pklxkd-61/?prefix=obj33 2023-05-31T21:03:21.558 DEBUG:boto:auth_path=/pklxkd-61/?prefix=obj33 2023-05-31T21:03:21.559 DEBUG:boto:Method: GET 2023-05-31T21:03:21.559 DEBUG:boto:Path: /pklxkd-61/?prefix=obj33 2023-05-31T21:03:21.560 DEBUG:boto:Data: 2023-05-31T21:03:21.560 DEBUG:boto:Headers: {} 2023-05-31T21:03:21.560 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T21:03:21.561 DEBUG:boto:Port: 8000 2023-05-31T21:03:21.561 DEBUG:boto:Params: {} 2023-05-31T21:03:21.561 DEBUG:boto:Token: None 2023-05-31T21:03:21.562 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 21:03:21 GMT /pklxkd-61/ 2023-05-31T21:03:21.562 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:H9PQFgrR8ica+qPwTl/4XbpLrOQ= 2023-05-31T21:03:21.563 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 21:03:21 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:H9PQFgrR8ica+qPwTl/4XbpLrOQ=', 'Content-Length': '0'} 2023-05-31T21:03:21.566 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000cc80903c66fc093e-006477b619-1630-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 21:03:21 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:03:21.567 DEBUG:boto:b'pklxkd-61obj331000falseobj332023-05-31T20:59:44.466Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T21:03:21.567 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj34 2023-05-31T21:03:21.568 DEBUG:boto:path=/pklxkd-61/obj34 2023-05-31T21:03:21.568 DEBUG:boto:auth_path=/pklxkd-61/obj34 2023-05-31T21:03:21.569 DEBUG:boto:path=/pklxkd-61/obj34?versionId=null 2023-05-31T21:03:21.569 DEBUG:boto:auth_path=/pklxkd-61/obj34?versionId=null 2023-05-31T21:03:21.569 DEBUG:boto:Method: GET 2023-05-31T21:03:21.570 DEBUG:boto:Path: /pklxkd-61/obj34?versionId=null 2023-05-31T21:03:21.570 DEBUG:boto:Data: 2023-05-31T21:03:21.570 DEBUG:boto:Headers: {} 2023-05-31T21:03:21.571 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T21:03:21.571 DEBUG:boto:Port: 8001 2023-05-31T21:03:21.571 DEBUG:boto:Params: {} 2023-05-31T21:03:21.572 DEBUG:boto:Token: None 2023-05-31T21:03:21.572 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 21:03:21 GMT /pklxkd-61/obj34?versionId=null 2023-05-31T21:03:21.573 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:XbWtQedQf1oQanf2TAbw8wJEtqQ= 2023-05-31T21:03:21.573 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 21:03:21 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:XbWtQedQf1oQanf2TAbw8wJEtqQ=', 'Content-Length': '0'} 2023-05-31T21:03:21.575 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:59:44 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f537dd1ff85118ea-006477b619-1f4e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 21:03:21 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:03:21.616 DEBUG:boto:path=/pklxkd-61/obj34 2023-05-31T21:03:21.617 DEBUG:boto:auth_path=/pklxkd-61/obj34 2023-05-31T21:03:21.617 DEBUG:boto:path=/pklxkd-61/obj34?versionId=null 2023-05-31T21:03:21.618 DEBUG:boto:auth_path=/pklxkd-61/obj34?versionId=null 2023-05-31T21:03:21.618 DEBUG:boto:Method: GET 2023-05-31T21:03:21.618 DEBUG:boto:Path: /pklxkd-61/obj34?versionId=null 2023-05-31T21:03:21.619 DEBUG:boto:Data: 2023-05-31T21:03:21.619 DEBUG:boto:Headers: {} 2023-05-31T21:03:21.619 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T21:03:21.620 DEBUG:boto:Port: 8000 2023-05-31T21:03:21.620 DEBUG:boto:Params: {} 2023-05-31T21:03:21.621 DEBUG:boto:Token: None 2023-05-31T21:03:21.621 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 21:03:21 GMT /pklxkd-61/obj34?versionId=null 2023-05-31T21:03:21.621 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:XbWtQedQf1oQanf2TAbw8wJEtqQ= 2023-05-31T21:03:21.622 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 21:03:21 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:XbWtQedQf1oQanf2TAbw8wJEtqQ=', 'Content-Length': '0'} 2023-05-31T21:03:21.624 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:59:44 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000066031f4915eb16fb-006477b619-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 21:03:21 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:03:21.624 DEBUG:boto:path=/pklxkd-61/obj34 2023-05-31T21:03:21.625 DEBUG:boto:auth_path=/pklxkd-61/obj34 2023-05-31T21:03:21.625 DEBUG:boto:path=/pklxkd-61/obj34?versionId=null 2023-05-31T21:03:21.625 DEBUG:boto:auth_path=/pklxkd-61/obj34?versionId=null 2023-05-31T21:03:21.626 DEBUG:boto:Method: HEAD 2023-05-31T21:03:21.626 DEBUG:boto:Path: /pklxkd-61/obj34?versionId=null 2023-05-31T21:03:21.626 DEBUG:boto:Data: 2023-05-31T21:03:21.627 DEBUG:boto:Headers: {} 2023-05-31T21:03:21.627 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T21:03:21.627 DEBUG:boto:Port: 8001 2023-05-31T21:03:21.628 DEBUG:boto:Params: {} 2023-05-31T21:03:21.628 DEBUG:boto:Token: None 2023-05-31T21:03:21.629 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 21:03:21 GMT /pklxkd-61/obj34?versionId=null 2023-05-31T21:03:21.629 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:OsqZNUmiHcvLrY1/lY7okavUKbE= 2023-05-31T21:03:21.630 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 21:03:21 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:OsqZNUmiHcvLrY1/lY7okavUKbE=', 'Content-Length': '0'} 2023-05-31T21:03:21.631 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:59:44 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000004f50ba653cb96b7-006477b619-1f4e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 21:03:21 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:03:21.632 DEBUG:boto:path=/pklxkd-61/obj34 2023-05-31T21:03:21.632 DEBUG:boto:auth_path=/pklxkd-61/obj34 2023-05-31T21:03:21.632 DEBUG:boto:path=/pklxkd-61/obj34?versionId=null 2023-05-31T21:03:21.633 DEBUG:boto:auth_path=/pklxkd-61/obj34?versionId=null 2023-05-31T21:03:21.633 DEBUG:boto:Method: HEAD 2023-05-31T21:03:21.634 DEBUG:boto:Path: /pklxkd-61/obj34?versionId=null 2023-05-31T21:03:21.634 DEBUG:boto:Data: 2023-05-31T21:03:21.634 DEBUG:boto:Headers: {} 2023-05-31T21:03:21.635 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T21:03:21.635 DEBUG:boto:Port: 8000 2023-05-31T21:03:21.635 DEBUG:boto:Params: {} 2023-05-31T21:03:21.636 DEBUG:boto:Token: None 2023-05-31T21:03:21.636 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 21:03:21 GMT /pklxkd-61/obj34?versionId=null 2023-05-31T21:03:21.636 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:OsqZNUmiHcvLrY1/lY7okavUKbE= 2023-05-31T21:03:21.637 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 21:03:21 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:OsqZNUmiHcvLrY1/lY7okavUKbE=', 'Content-Length': '0'} 2023-05-31T21:03:21.638 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:59:44 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006dfb9aae0ae3795a-006477b619-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 21:03:21 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:03:21.639 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj34 2023-05-31T21:03:21.639 DEBUG:boto:path=/pklxkd-61/obj34 2023-05-31T21:03:21.640 DEBUG:boto:auth_path=/pklxkd-61/obj34 2023-05-31T21:03:21.640 DEBUG:boto:path=/pklxkd-61/obj34?versionId=null 2023-05-31T21:03:21.640 DEBUG:boto:auth_path=/pklxkd-61/obj34?versionId=null 2023-05-31T21:03:21.641 DEBUG:boto:Method: GET 2023-05-31T21:03:21.641 DEBUG:boto:Path: /pklxkd-61/obj34?versionId=null 2023-05-31T21:03:21.642 DEBUG:boto:Data: 2023-05-31T21:03:21.642 DEBUG:boto:Headers: {} 2023-05-31T21:03:21.642 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T21:03:21.643 DEBUG:boto:Port: 8001 2023-05-31T21:03:21.643 DEBUG:boto:Params: {} 2023-05-31T21:03:21.643 DEBUG:boto:Token: None 2023-05-31T21:03:21.644 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 21:03:21 GMT /pklxkd-61/obj34?versionId=null 2023-05-31T21:03:21.644 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:XbWtQedQf1oQanf2TAbw8wJEtqQ= 2023-05-31T21:03:21.645 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 21:03:21 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:XbWtQedQf1oQanf2TAbw8wJEtqQ=', 'Content-Length': '0'} 2023-05-31T21:03:21.646 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:59:44 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f1199b0bcbc1bee9-006477b619-1f4e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 21:03:21 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:03:21.647 DEBUG:boto:path=/pklxkd-61/obj34 2023-05-31T21:03:21.647 DEBUG:boto:auth_path=/pklxkd-61/obj34 2023-05-31T21:03:21.648 DEBUG:boto:path=/pklxkd-61/obj34?versionId=null 2023-05-31T21:03:21.648 DEBUG:boto:auth_path=/pklxkd-61/obj34?versionId=null 2023-05-31T21:03:21.648 DEBUG:boto:Method: GET 2023-05-31T21:03:21.649 DEBUG:boto:Path: /pklxkd-61/obj34?versionId=null 2023-05-31T21:03:21.649 DEBUG:boto:Data: 2023-05-31T21:03:21.649 DEBUG:boto:Headers: {} 2023-05-31T21:03:21.650 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T21:03:21.650 DEBUG:boto:Port: 8000 2023-05-31T21:03:21.651 DEBUG:boto:Params: {} 2023-05-31T21:03:21.651 DEBUG:boto:Token: None 2023-05-31T21:03:21.651 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 21:03:21 GMT /pklxkd-61/obj34?versionId=null 2023-05-31T21:03:21.652 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:XbWtQedQf1oQanf2TAbw8wJEtqQ= 2023-05-31T21:03:21.652 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 21:03:21 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:XbWtQedQf1oQanf2TAbw8wJEtqQ=', 'Content-Length': '0'} 2023-05-31T21:03:21.654 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:59:44 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005ea9b00eb225f908-006477b619-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 21:03:21 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:03:21.655 DEBUG:boto:path=/pklxkd-61/ 2023-05-31T21:03:21.655 DEBUG:boto:auth_path=/pklxkd-61/ 2023-05-31T21:03:21.655 DEBUG:boto:path=/pklxkd-61/?prefix=obj34 2023-05-31T21:03:21.656 DEBUG:boto:auth_path=/pklxkd-61/?prefix=obj34 2023-05-31T21:03:21.656 DEBUG:boto:Method: GET 2023-05-31T21:03:21.656 DEBUG:boto:Path: /pklxkd-61/?prefix=obj34 2023-05-31T21:03:21.657 DEBUG:boto:Data: 2023-05-31T21:03:21.657 DEBUG:boto:Headers: {} 2023-05-31T21:03:21.657 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T21:03:21.658 DEBUG:boto:Port: 8001 2023-05-31T21:03:21.658 DEBUG:boto:Params: {} 2023-05-31T21:03:21.659 DEBUG:boto:Token: None 2023-05-31T21:03:21.659 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 21:03:21 GMT /pklxkd-61/ 2023-05-31T21:03:21.659 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:H9PQFgrR8ica+qPwTl/4XbpLrOQ= 2023-05-31T21:03:21.660 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 21:03:21 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:H9PQFgrR8ica+qPwTl/4XbpLrOQ=', 'Content-Length': '0'} 2023-05-31T21:03:21.662 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000054d045e097f771fc-006477b619-1f4e-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 21:03:21 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:03:21.663 DEBUG:boto:b'pklxkd-61obj341000falseobj342023-05-31T20:59:44.822Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T21:03:21.663 DEBUG:boto:path=/pklxkd-61/ 2023-05-31T21:03:21.664 DEBUG:boto:auth_path=/pklxkd-61/ 2023-05-31T21:03:21.664 DEBUG:boto:path=/pklxkd-61/?prefix=obj34 2023-05-31T21:03:21.665 DEBUG:boto:auth_path=/pklxkd-61/?prefix=obj34 2023-05-31T21:03:21.665 DEBUG:boto:Method: GET 2023-05-31T21:03:21.665 DEBUG:boto:Path: /pklxkd-61/?prefix=obj34 2023-05-31T21:03:21.666 DEBUG:boto:Data: 2023-05-31T21:03:21.666 DEBUG:boto:Headers: {} 2023-05-31T21:03:21.666 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T21:03:21.667 DEBUG:boto:Port: 8000 2023-05-31T21:03:21.667 DEBUG:boto:Params: {} 2023-05-31T21:03:21.668 DEBUG:boto:Token: None 2023-05-31T21:03:21.668 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 21:03:21 GMT /pklxkd-61/ 2023-05-31T21:03:21.668 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:H9PQFgrR8ica+qPwTl/4XbpLrOQ= 2023-05-31T21:03:21.669 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 21:03:21 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:H9PQFgrR8ica+qPwTl/4XbpLrOQ=', 'Content-Length': '0'} 2023-05-31T21:03:21.672 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000066a442a03994d3c6-006477b619-1630-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 21:03:21 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:03:21.673 DEBUG:boto:b'pklxkd-61obj341000falseobj342023-05-31T20:59:44.822Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T21:03:21.673 DEBUG:boto:path=/pklxkd-61/obj34 2023-05-31T21:03:21.674 DEBUG:boto:auth_path=/pklxkd-61/obj34 2023-05-31T21:03:21.674 DEBUG:boto:Method: HEAD 2023-05-31T21:03:21.674 DEBUG:boto:Path: /pklxkd-61/obj34 2023-05-31T21:03:21.675 DEBUG:boto:Data: 2023-05-31T21:03:21.675 DEBUG:boto:Headers: {} 2023-05-31T21:03:21.675 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T21:03:21.676 DEBUG:boto:Port: 8001 2023-05-31T21:03:21.676 DEBUG:boto:Params: {} 2023-05-31T21:03:21.676 DEBUG:boto:Token: None 2023-05-31T21:03:21.677 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 21:03:21 GMT /pklxkd-61/obj34 2023-05-31T21:03:21.677 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:wJ+Flm0C1LUgNxkOqzsiRV79gmg= 2023-05-31T21:03:21.678 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 21:03:21 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:wJ+Flm0C1LUgNxkOqzsiRV79gmg=', 'Content-Length': '0'} 2023-05-31T21:03:21.679 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:59:44 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000093839945fb7b8316-006477b619-1f4e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 21:03:21 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:03:21.680 DEBUG:boto:path=/pklxkd-61/obj34 2023-05-31T21:03:21.680 DEBUG:boto:auth_path=/pklxkd-61/obj34 2023-05-31T21:03:21.681 DEBUG:boto:Method: HEAD 2023-05-31T21:03:21.681 DEBUG:boto:Path: /pklxkd-61/obj34 2023-05-31T21:03:21.681 DEBUG:boto:Data: 2023-05-31T21:03:21.682 DEBUG:boto:Headers: {} 2023-05-31T21:03:21.682 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T21:03:21.682 DEBUG:boto:Port: 8000 2023-05-31T21:03:21.683 DEBUG:boto:Params: {} 2023-05-31T21:03:21.683 DEBUG:boto:Token: None 2023-05-31T21:03:21.683 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 21:03:21 GMT /pklxkd-61/obj34 2023-05-31T21:03:21.684 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:wJ+Flm0C1LUgNxkOqzsiRV79gmg= 2023-05-31T21:03:21.684 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 21:03:21 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:wJ+Flm0C1LUgNxkOqzsiRV79gmg=', 'Content-Length': '0'} 2023-05-31T21:03:21.686 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:59:44 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000cf561b147ec2b696-006477b619-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 21:03:21 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:03:21.686 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj34 2023-05-31T21:03:21.687 DEBUG:boto:path=/pklxkd-61/obj34 2023-05-31T21:03:21.687 DEBUG:boto:auth_path=/pklxkd-61/obj34 2023-05-31T21:03:21.688 DEBUG:boto:Method: GET 2023-05-31T21:03:21.688 DEBUG:boto:Path: /pklxkd-61/obj34 2023-05-31T21:03:21.689 DEBUG:boto:Data: 2023-05-31T21:03:21.689 DEBUG:boto:Headers: {} 2023-05-31T21:03:21.690 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T21:03:21.690 DEBUG:boto:Port: 8001 2023-05-31T21:03:21.691 DEBUG:boto:Params: {} 2023-05-31T21:03:21.691 DEBUG:boto:Token: None 2023-05-31T21:03:21.692 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 21:03:21 GMT /pklxkd-61/obj34 2023-05-31T21:03:21.692 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:/8m7y5EcHtKVuIgl4guR4c+/sVY= 2023-05-31T21:03:21.692 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 21:03:21 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:/8m7y5EcHtKVuIgl4guR4c+/sVY=', 'Content-Length': '0'} 2023-05-31T21:03:21.694 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:59:44 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000aed88016d4238c2f-006477b619-1f4e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 21:03:21 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:03:21.695 DEBUG:boto:path=/pklxkd-61/obj34 2023-05-31T21:03:21.695 DEBUG:boto:auth_path=/pklxkd-61/obj34 2023-05-31T21:03:21.696 DEBUG:boto:Method: GET 2023-05-31T21:03:21.696 DEBUG:boto:Path: /pklxkd-61/obj34 2023-05-31T21:03:21.697 DEBUG:boto:Data: 2023-05-31T21:03:21.697 DEBUG:boto:Headers: {} 2023-05-31T21:03:21.697 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T21:03:21.698 DEBUG:boto:Port: 8000 2023-05-31T21:03:21.698 DEBUG:boto:Params: {} 2023-05-31T21:03:21.698 DEBUG:boto:Token: None 2023-05-31T21:03:21.699 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 21:03:21 GMT /pklxkd-61/obj34 2023-05-31T21:03:21.699 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:/8m7y5EcHtKVuIgl4guR4c+/sVY= 2023-05-31T21:03:21.700 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 21:03:21 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:/8m7y5EcHtKVuIgl4guR4c+/sVY=', 'Content-Length': '0'} 2023-05-31T21:03:21.701 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:59:44 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006e8b905f5f3a53bd-006477b619-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 21:03:21 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:03:21.702 DEBUG:boto:path=/pklxkd-61/ 2023-05-31T21:03:21.702 DEBUG:boto:auth_path=/pklxkd-61/ 2023-05-31T21:03:21.703 DEBUG:boto:path=/pklxkd-61/?prefix=obj34 2023-05-31T21:03:21.703 DEBUG:boto:auth_path=/pklxkd-61/?prefix=obj34 2023-05-31T21:03:21.703 DEBUG:boto:Method: GET 2023-05-31T21:03:21.704 DEBUG:boto:Path: /pklxkd-61/?prefix=obj34 2023-05-31T21:03:21.704 DEBUG:boto:Data: 2023-05-31T21:03:21.704 DEBUG:boto:Headers: {} 2023-05-31T21:03:21.705 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T21:03:21.705 DEBUG:boto:Port: 8001 2023-05-31T21:03:21.706 DEBUG:boto:Params: {} 2023-05-31T21:03:21.706 DEBUG:boto:Token: None 2023-05-31T21:03:21.706 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 21:03:21 GMT /pklxkd-61/ 2023-05-31T21:03:21.707 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:H9PQFgrR8ica+qPwTl/4XbpLrOQ= 2023-05-31T21:03:21.707 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 21:03:21 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:H9PQFgrR8ica+qPwTl/4XbpLrOQ=', 'Content-Length': '0'} 2023-05-31T21:03:21.709 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000003ce1f444b950acb1-006477b619-1f4e-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 21:03:21 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:03:21.710 DEBUG:boto:b'pklxkd-61obj341000falseobj342023-05-31T20:59:44.822Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T21:03:21.711 DEBUG:boto:path=/pklxkd-61/ 2023-05-31T21:03:21.711 DEBUG:boto:auth_path=/pklxkd-61/ 2023-05-31T21:03:21.711 DEBUG:boto:path=/pklxkd-61/?prefix=obj34 2023-05-31T21:03:21.712 DEBUG:boto:auth_path=/pklxkd-61/?prefix=obj34 2023-05-31T21:03:21.712 DEBUG:boto:Method: GET 2023-05-31T21:03:21.712 DEBUG:boto:Path: /pklxkd-61/?prefix=obj34 2023-05-31T21:03:21.713 DEBUG:boto:Data: 2023-05-31T21:03:21.713 DEBUG:boto:Headers: {} 2023-05-31T21:03:21.713 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T21:03:21.714 DEBUG:boto:Port: 8000 2023-05-31T21:03:21.714 DEBUG:boto:Params: {} 2023-05-31T21:03:21.715 DEBUG:boto:Token: None 2023-05-31T21:03:21.715 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 21:03:21 GMT /pklxkd-61/ 2023-05-31T21:03:21.716 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:H9PQFgrR8ica+qPwTl/4XbpLrOQ= 2023-05-31T21:03:21.716 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 21:03:21 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:H9PQFgrR8ica+qPwTl/4XbpLrOQ=', 'Content-Length': '0'} 2023-05-31T21:03:21.720 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000422b399ebefc0dc8-006477b619-1630-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 21:03:21 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:03:21.720 DEBUG:boto:b'pklxkd-61obj341000falseobj342023-05-31T20:59:44.822Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T21:03:21.721 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj37 2023-05-31T21:03:21.721 DEBUG:boto:path=/pklxkd-61/obj37 2023-05-31T21:03:21.722 DEBUG:boto:auth_path=/pklxkd-61/obj37 2023-05-31T21:03:21.722 DEBUG:boto:path=/pklxkd-61/obj37?versionId=null 2023-05-31T21:03:21.723 DEBUG:boto:auth_path=/pklxkd-61/obj37?versionId=null 2023-05-31T21:03:21.723 DEBUG:boto:Method: GET 2023-05-31T21:03:21.724 DEBUG:boto:Path: /pklxkd-61/obj37?versionId=null 2023-05-31T21:03:21.724 DEBUG:boto:Data: 2023-05-31T21:03:21.725 DEBUG:boto:Headers: {} 2023-05-31T21:03:21.726 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T21:03:21.726 DEBUG:boto:Port: 8001 2023-05-31T21:03:21.727 DEBUG:boto:Params: {} 2023-05-31T21:03:21.727 DEBUG:boto:Token: None 2023-05-31T21:03:21.728 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 21:03:21 GMT /pklxkd-61/obj37?versionId=null 2023-05-31T21:03:21.728 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:2X1cF37k2lWvH2bFIDDYBWY+YGE= 2023-05-31T21:03:21.729 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 21:03:21 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:2X1cF37k2lWvH2bFIDDYBWY+YGE=', 'Content-Length': '0'} 2023-05-31T21:03:21.731 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:59:46 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006f2a6c09578dd184-006477b619-1f4e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 21:03:21 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:03:21.732 DEBUG:boto:path=/pklxkd-61/obj37 2023-05-31T21:03:21.732 DEBUG:boto:auth_path=/pklxkd-61/obj37 2023-05-31T21:03:21.733 DEBUG:boto:path=/pklxkd-61/obj37?versionId=null 2023-05-31T21:03:21.733 DEBUG:boto:auth_path=/pklxkd-61/obj37?versionId=null 2023-05-31T21:03:21.733 DEBUG:boto:Method: GET 2023-05-31T21:03:21.734 DEBUG:boto:Path: /pklxkd-61/obj37?versionId=null 2023-05-31T21:03:21.734 DEBUG:boto:Data: 2023-05-31T21:03:21.734 DEBUG:boto:Headers: {} 2023-05-31T21:03:21.735 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T21:03:21.735 DEBUG:boto:Port: 8000 2023-05-31T21:03:21.736 DEBUG:boto:Params: {} 2023-05-31T21:03:21.736 DEBUG:boto:Token: None 2023-05-31T21:03:21.736 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 21:03:21 GMT /pklxkd-61/obj37?versionId=null 2023-05-31T21:03:21.737 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:2X1cF37k2lWvH2bFIDDYBWY+YGE= 2023-05-31T21:03:21.737 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 21:03:21 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:2X1cF37k2lWvH2bFIDDYBWY+YGE=', 'Content-Length': '0'} 2023-05-31T21:03:21.739 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:59:46 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d327d6bc6c69f111-006477b619-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 21:03:21 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:03:21.740 DEBUG:boto:path=/pklxkd-61/obj37 2023-05-31T21:03:21.740 DEBUG:boto:auth_path=/pklxkd-61/obj37 2023-05-31T21:03:21.741 DEBUG:boto:path=/pklxkd-61/obj37?versionId=null 2023-05-31T21:03:21.741 DEBUG:boto:auth_path=/pklxkd-61/obj37?versionId=null 2023-05-31T21:03:21.741 DEBUG:boto:Method: HEAD 2023-05-31T21:03:21.742 DEBUG:boto:Path: /pklxkd-61/obj37?versionId=null 2023-05-31T21:03:21.742 DEBUG:boto:Data: 2023-05-31T21:03:21.742 DEBUG:boto:Headers: {} 2023-05-31T21:03:21.743 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T21:03:21.743 DEBUG:boto:Port: 8001 2023-05-31T21:03:21.743 DEBUG:boto:Params: {} 2023-05-31T21:03:21.744 DEBUG:boto:Token: None 2023-05-31T21:03:21.744 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 21:03:21 GMT /pklxkd-61/obj37?versionId=null 2023-05-31T21:03:21.744 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:rvCahmhRL8+43eOwq5EHHfltg2U= 2023-05-31T21:03:21.745 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 21:03:21 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:rvCahmhRL8+43eOwq5EHHfltg2U=', 'Content-Length': '0'} 2023-05-31T21:03:21.746 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:59:46 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d5bf10750eaa0c8e-006477b619-1f4e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 21:03:21 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:03:21.747 DEBUG:boto:path=/pklxkd-61/obj37 2023-05-31T21:03:21.747 DEBUG:boto:auth_path=/pklxkd-61/obj37 2023-05-31T21:03:21.748 DEBUG:boto:path=/pklxkd-61/obj37?versionId=null 2023-05-31T21:03:21.748 DEBUG:boto:auth_path=/pklxkd-61/obj37?versionId=null 2023-05-31T21:03:21.748 DEBUG:boto:Method: HEAD 2023-05-31T21:03:21.749 DEBUG:boto:Path: /pklxkd-61/obj37?versionId=null 2023-05-31T21:03:21.749 DEBUG:boto:Data: 2023-05-31T21:03:21.749 DEBUG:boto:Headers: {} 2023-05-31T21:03:21.750 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T21:03:21.750 DEBUG:boto:Port: 8000 2023-05-31T21:03:21.750 DEBUG:boto:Params: {} 2023-05-31T21:03:21.751 DEBUG:boto:Token: None 2023-05-31T21:03:21.751 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 21:03:21 GMT /pklxkd-61/obj37?versionId=null 2023-05-31T21:03:21.752 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:rvCahmhRL8+43eOwq5EHHfltg2U= 2023-05-31T21:03:21.752 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 21:03:21 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:rvCahmhRL8+43eOwq5EHHfltg2U=', 'Content-Length': '0'} 2023-05-31T21:03:21.754 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:59:46 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000374aef88543cd2f8-006477b619-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 21:03:21 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:03:21.754 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj37 2023-05-31T21:03:21.755 DEBUG:boto:path=/pklxkd-61/obj37 2023-05-31T21:03:21.755 DEBUG:boto:auth_path=/pklxkd-61/obj37 2023-05-31T21:03:21.755 DEBUG:boto:path=/pklxkd-61/obj37?versionId=null 2023-05-31T21:03:21.756 DEBUG:boto:auth_path=/pklxkd-61/obj37?versionId=null 2023-05-31T21:03:21.756 DEBUG:boto:Method: GET 2023-05-31T21:03:21.756 DEBUG:boto:Path: /pklxkd-61/obj37?versionId=null 2023-05-31T21:03:21.757 DEBUG:boto:Data: 2023-05-31T21:03:21.757 DEBUG:boto:Headers: {} 2023-05-31T21:03:21.757 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T21:03:21.758 DEBUG:boto:Port: 8001 2023-05-31T21:03:21.758 DEBUG:boto:Params: {} 2023-05-31T21:03:21.759 DEBUG:boto:Token: None 2023-05-31T21:03:21.759 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 21:03:21 GMT /pklxkd-61/obj37?versionId=null 2023-05-31T21:03:21.759 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:2X1cF37k2lWvH2bFIDDYBWY+YGE= 2023-05-31T21:03:21.760 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 21:03:21 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:2X1cF37k2lWvH2bFIDDYBWY+YGE=', 'Content-Length': '0'} 2023-05-31T21:03:21.761 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:59:46 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001d2ccb175ae01ade-006477b619-1f4e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 21:03:21 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:03:21.762 DEBUG:boto:path=/pklxkd-61/obj37 2023-05-31T21:03:21.763 DEBUG:boto:auth_path=/pklxkd-61/obj37 2023-05-31T21:03:21.763 DEBUG:boto:path=/pklxkd-61/obj37?versionId=null 2023-05-31T21:03:21.763 DEBUG:boto:auth_path=/pklxkd-61/obj37?versionId=null 2023-05-31T21:03:21.764 DEBUG:boto:Method: GET 2023-05-31T21:03:21.764 DEBUG:boto:Path: /pklxkd-61/obj37?versionId=null 2023-05-31T21:03:21.764 DEBUG:boto:Data: 2023-05-31T21:03:21.765 DEBUG:boto:Headers: {} 2023-05-31T21:03:21.765 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T21:03:21.765 DEBUG:boto:Port: 8000 2023-05-31T21:03:21.766 DEBUG:boto:Params: {} 2023-05-31T21:03:21.766 DEBUG:boto:Token: None 2023-05-31T21:03:21.767 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 21:03:21 GMT /pklxkd-61/obj37?versionId=null 2023-05-31T21:03:21.767 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:2X1cF37k2lWvH2bFIDDYBWY+YGE= 2023-05-31T21:03:21.767 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 21:03:21 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:2X1cF37k2lWvH2bFIDDYBWY+YGE=', 'Content-Length': '0'} 2023-05-31T21:03:21.769 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:59:46 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003ec00fb66b8fb1ea-006477b619-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 21:03:21 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:03:21.770 DEBUG:boto:path=/pklxkd-61/ 2023-05-31T21:03:21.770 DEBUG:boto:auth_path=/pklxkd-61/ 2023-05-31T21:03:21.771 DEBUG:boto:path=/pklxkd-61/?prefix=obj37 2023-05-31T21:03:21.771 DEBUG:boto:auth_path=/pklxkd-61/?prefix=obj37 2023-05-31T21:03:21.771 DEBUG:boto:Method: GET 2023-05-31T21:03:21.772 DEBUG:boto:Path: /pklxkd-61/?prefix=obj37 2023-05-31T21:03:21.772 DEBUG:boto:Data: 2023-05-31T21:03:21.772 DEBUG:boto:Headers: {} 2023-05-31T21:03:21.773 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T21:03:21.773 DEBUG:boto:Port: 8001 2023-05-31T21:03:21.773 DEBUG:boto:Params: {} 2023-05-31T21:03:21.774 DEBUG:boto:Token: None 2023-05-31T21:03:21.774 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 21:03:21 GMT /pklxkd-61/ 2023-05-31T21:03:21.775 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:H9PQFgrR8ica+qPwTl/4XbpLrOQ= 2023-05-31T21:03:21.775 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 21:03:21 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:H9PQFgrR8ica+qPwTl/4XbpLrOQ=', 'Content-Length': '0'} 2023-05-31T21:03:21.777 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a338d9fc45fdab74-006477b619-1f4e-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 21:03:21 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:03:21.778 DEBUG:boto:b'pklxkd-61obj371000falseobj372023-05-31T20:59:46.915Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T21:03:21.779 DEBUG:boto:path=/pklxkd-61/ 2023-05-31T21:03:21.779 DEBUG:boto:auth_path=/pklxkd-61/ 2023-05-31T21:03:21.779 DEBUG:boto:path=/pklxkd-61/?prefix=obj37 2023-05-31T21:03:21.780 DEBUG:boto:auth_path=/pklxkd-61/?prefix=obj37 2023-05-31T21:03:21.780 DEBUG:boto:Method: GET 2023-05-31T21:03:21.781 DEBUG:boto:Path: /pklxkd-61/?prefix=obj37 2023-05-31T21:03:21.781 DEBUG:boto:Data: 2023-05-31T21:03:21.781 DEBUG:boto:Headers: {} 2023-05-31T21:03:21.782 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T21:03:21.782 DEBUG:boto:Port: 8000 2023-05-31T21:03:21.782 DEBUG:boto:Params: {} 2023-05-31T21:03:21.783 DEBUG:boto:Token: None 2023-05-31T21:03:21.783 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 21:03:21 GMT /pklxkd-61/ 2023-05-31T21:03:21.783 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:H9PQFgrR8ica+qPwTl/4XbpLrOQ= 2023-05-31T21:03:21.784 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 21:03:21 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:H9PQFgrR8ica+qPwTl/4XbpLrOQ=', 'Content-Length': '0'} 2023-05-31T21:03:21.788 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000007541613c59a9d181-006477b619-1630-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 21:03:21 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:03:21.788 DEBUG:boto:b'pklxkd-61obj371000falseobj372023-05-31T20:59:46.915Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T21:03:21.789 DEBUG:boto:path=/pklxkd-61/obj37 2023-05-31T21:03:21.789 DEBUG:boto:auth_path=/pklxkd-61/obj37 2023-05-31T21:03:21.789 DEBUG:boto:Method: HEAD 2023-05-31T21:03:21.790 DEBUG:boto:Path: /pklxkd-61/obj37 2023-05-31T21:03:21.790 DEBUG:boto:Data: 2023-05-31T21:03:21.790 DEBUG:boto:Headers: {} 2023-05-31T21:03:21.791 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T21:03:21.791 DEBUG:boto:Port: 8001 2023-05-31T21:03:21.791 DEBUG:boto:Params: {} 2023-05-31T21:03:21.792 DEBUG:boto:Token: None 2023-05-31T21:03:21.792 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 21:03:21 GMT /pklxkd-61/obj37 2023-05-31T21:03:21.793 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:gEVT1aIKDXZ5yJb+I7sKkZUXOGY= 2023-05-31T21:03:21.793 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 21:03:21 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:gEVT1aIKDXZ5yJb+I7sKkZUXOGY=', 'Content-Length': '0'} 2023-05-31T21:03:21.794 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:59:46 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000906ab3acd8cbd6af-006477b619-1f4e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 21:03:21 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:03:21.795 DEBUG:boto:path=/pklxkd-61/obj37 2023-05-31T21:03:21.795 DEBUG:boto:auth_path=/pklxkd-61/obj37 2023-05-31T21:03:21.796 DEBUG:boto:Method: HEAD 2023-05-31T21:03:21.796 DEBUG:boto:Path: /pklxkd-61/obj37 2023-05-31T21:03:21.796 DEBUG:boto:Data: 2023-05-31T21:03:21.797 DEBUG:boto:Headers: {} 2023-05-31T21:03:21.797 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T21:03:21.797 DEBUG:boto:Port: 8000 2023-05-31T21:03:21.798 DEBUG:boto:Params: {} 2023-05-31T21:03:21.798 DEBUG:boto:Token: None 2023-05-31T21:03:21.799 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 21:03:21 GMT /pklxkd-61/obj37 2023-05-31T21:03:21.799 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:gEVT1aIKDXZ5yJb+I7sKkZUXOGY= 2023-05-31T21:03:21.799 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 21:03:21 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:gEVT1aIKDXZ5yJb+I7sKkZUXOGY=', 'Content-Length': '0'} 2023-05-31T21:03:21.801 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:59:46 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000024eb1ede7cc370fb-006477b619-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 21:03:21 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:03:21.802 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj37 2023-05-31T21:03:21.802 DEBUG:boto:path=/pklxkd-61/obj37 2023-05-31T21:03:21.802 DEBUG:boto:auth_path=/pklxkd-61/obj37 2023-05-31T21:03:21.803 DEBUG:boto:Method: GET 2023-05-31T21:03:21.803 DEBUG:boto:Path: /pklxkd-61/obj37 2023-05-31T21:03:21.803 DEBUG:boto:Data: 2023-05-31T21:03:21.804 DEBUG:boto:Headers: {} 2023-05-31T21:03:21.804 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T21:03:21.805 DEBUG:boto:Port: 8001 2023-05-31T21:03:21.805 DEBUG:boto:Params: {} 2023-05-31T21:03:21.805 DEBUG:boto:Token: None 2023-05-31T21:03:21.806 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 21:03:21 GMT /pklxkd-61/obj37 2023-05-31T21:03:21.806 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:KmpO/Agc/tjwYVRjapjop8+mS+c= 2023-05-31T21:03:21.807 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 21:03:21 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:KmpO/Agc/tjwYVRjapjop8+mS+c=', 'Content-Length': '0'} 2023-05-31T21:03:21.808 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:59:46 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001b35fdb7fe70dcbf-006477b619-1f4e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 21:03:21 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:03:21.852 DEBUG:boto:path=/pklxkd-61/obj37 2023-05-31T21:03:21.853 DEBUG:boto:auth_path=/pklxkd-61/obj37 2023-05-31T21:03:21.853 DEBUG:boto:Method: GET 2023-05-31T21:03:21.853 DEBUG:boto:Path: /pklxkd-61/obj37 2023-05-31T21:03:21.854 DEBUG:boto:Data: 2023-05-31T21:03:21.854 DEBUG:boto:Headers: {} 2023-05-31T21:03:21.855 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T21:03:21.855 DEBUG:boto:Port: 8000 2023-05-31T21:03:21.855 DEBUG:boto:Params: {} 2023-05-31T21:03:21.856 DEBUG:boto:Token: None 2023-05-31T21:03:21.856 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 21:03:21 GMT /pklxkd-61/obj37 2023-05-31T21:03:21.857 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:KmpO/Agc/tjwYVRjapjop8+mS+c= 2023-05-31T21:03:21.857 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 21:03:21 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:KmpO/Agc/tjwYVRjapjop8+mS+c=', 'Content-Length': '0'} 2023-05-31T21:03:21.859 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:59:46 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001492d21d93f62bc7-006477b619-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 21:03:21 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:03:21.859 DEBUG:boto:path=/pklxkd-61/ 2023-05-31T21:03:21.860 DEBUG:boto:auth_path=/pklxkd-61/ 2023-05-31T21:03:21.860 DEBUG:boto:path=/pklxkd-61/?prefix=obj37 2023-05-31T21:03:21.860 DEBUG:boto:auth_path=/pklxkd-61/?prefix=obj37 2023-05-31T21:03:21.861 DEBUG:boto:Method: GET 2023-05-31T21:03:21.861 DEBUG:boto:Path: /pklxkd-61/?prefix=obj37 2023-05-31T21:03:21.862 DEBUG:boto:Data: 2023-05-31T21:03:21.862 DEBUG:boto:Headers: {} 2023-05-31T21:03:21.862 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T21:03:21.863 DEBUG:boto:Port: 8001 2023-05-31T21:03:21.863 DEBUG:boto:Params: {} 2023-05-31T21:03:21.863 DEBUG:boto:Token: None 2023-05-31T21:03:21.864 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 21:03:21 GMT /pklxkd-61/ 2023-05-31T21:03:21.864 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:H9PQFgrR8ica+qPwTl/4XbpLrOQ= 2023-05-31T21:03:21.865 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 21:03:21 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:H9PQFgrR8ica+qPwTl/4XbpLrOQ=', 'Content-Length': '0'} 2023-05-31T21:03:21.867 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000943caac0e4ec8262-006477b619-1f4e-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 21:03:21 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:03:21.868 DEBUG:boto:b'pklxkd-61obj371000falseobj372023-05-31T20:59:46.915Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T21:03:21.868 DEBUG:boto:path=/pklxkd-61/ 2023-05-31T21:03:21.869 DEBUG:boto:auth_path=/pklxkd-61/ 2023-05-31T21:03:21.869 DEBUG:boto:path=/pklxkd-61/?prefix=obj37 2023-05-31T21:03:21.869 DEBUG:boto:auth_path=/pklxkd-61/?prefix=obj37 2023-05-31T21:03:21.870 DEBUG:boto:Method: GET 2023-05-31T21:03:21.870 DEBUG:boto:Path: /pklxkd-61/?prefix=obj37 2023-05-31T21:03:21.870 DEBUG:boto:Data: 2023-05-31T21:03:21.871 DEBUG:boto:Headers: {} 2023-05-31T21:03:21.871 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T21:03:21.871 DEBUG:boto:Port: 8000 2023-05-31T21:03:21.872 DEBUG:boto:Params: {} 2023-05-31T21:03:21.872 DEBUG:boto:Token: None 2023-05-31T21:03:21.873 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 21:03:21 GMT /pklxkd-61/ 2023-05-31T21:03:21.873 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:H9PQFgrR8ica+qPwTl/4XbpLrOQ= 2023-05-31T21:03:21.874 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 21:03:21 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:H9PQFgrR8ica+qPwTl/4XbpLrOQ=', 'Content-Length': '0'} 2023-05-31T21:03:21.877 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000bec31e4ffcec83fc-006477b619-1630-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 21:03:21 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:03:21.878 DEBUG:boto:b'pklxkd-61obj371000falseobj372023-05-31T20:59:46.915Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T21:03:21.878 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj38 2023-05-31T21:03:21.879 DEBUG:boto:path=/pklxkd-61/obj38 2023-05-31T21:03:21.879 DEBUG:boto:auth_path=/pklxkd-61/obj38 2023-05-31T21:03:21.879 DEBUG:boto:path=/pklxkd-61/obj38?versionId=null 2023-05-31T21:03:21.880 DEBUG:boto:auth_path=/pklxkd-61/obj38?versionId=null 2023-05-31T21:03:21.880 DEBUG:boto:Method: GET 2023-05-31T21:03:21.880 DEBUG:boto:Path: /pklxkd-61/obj38?versionId=null 2023-05-31T21:03:21.881 DEBUG:boto:Data: 2023-05-31T21:03:21.881 DEBUG:boto:Headers: {} 2023-05-31T21:03:21.882 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T21:03:21.882 DEBUG:boto:Port: 8001 2023-05-31T21:03:21.882 DEBUG:boto:Params: {} 2023-05-31T21:03:21.883 DEBUG:boto:Token: None 2023-05-31T21:03:21.883 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 21:03:21 GMT /pklxkd-61/obj38?versionId=null 2023-05-31T21:03:21.883 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:OLRluCbzQ8wiaJWjJ1WLg34U/KI= 2023-05-31T21:03:21.884 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 21:03:21 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:OLRluCbzQ8wiaJWjJ1WLg34U/KI=', 'Content-Length': '0'} 2023-05-31T21:03:21.886 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:59:45 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007ac4ec60c1e3bd36-006477b619-1f4e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 21:03:21 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:03:21.886 DEBUG:boto:path=/pklxkd-61/obj38 2023-05-31T21:03:21.887 DEBUG:boto:auth_path=/pklxkd-61/obj38 2023-05-31T21:03:21.887 DEBUG:boto:path=/pklxkd-61/obj38?versionId=null 2023-05-31T21:03:21.887 DEBUG:boto:auth_path=/pklxkd-61/obj38?versionId=null 2023-05-31T21:03:21.888 DEBUG:boto:Method: GET 2023-05-31T21:03:21.888 DEBUG:boto:Path: /pklxkd-61/obj38?versionId=null 2023-05-31T21:03:21.888 DEBUG:boto:Data: 2023-05-31T21:03:21.889 DEBUG:boto:Headers: {} 2023-05-31T21:03:21.889 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T21:03:21.890 DEBUG:boto:Port: 8000 2023-05-31T21:03:21.890 DEBUG:boto:Params: {} 2023-05-31T21:03:21.890 DEBUG:boto:Token: None 2023-05-31T21:03:21.891 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 21:03:21 GMT /pklxkd-61/obj38?versionId=null 2023-05-31T21:03:21.891 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:OLRluCbzQ8wiaJWjJ1WLg34U/KI= 2023-05-31T21:03:21.892 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 21:03:21 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:OLRluCbzQ8wiaJWjJ1WLg34U/KI=', 'Content-Length': '0'} 2023-05-31T21:03:21.893 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:59:45 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000097e7b10778926a06-006477b619-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 21:03:21 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:03:21.936 DEBUG:boto:path=/pklxkd-61/obj38 2023-05-31T21:03:21.937 DEBUG:boto:auth_path=/pklxkd-61/obj38 2023-05-31T21:03:21.937 DEBUG:boto:path=/pklxkd-61/obj38?versionId=null 2023-05-31T21:03:21.937 DEBUG:boto:auth_path=/pklxkd-61/obj38?versionId=null 2023-05-31T21:03:21.938 DEBUG:boto:Method: HEAD 2023-05-31T21:03:21.938 DEBUG:boto:Path: /pklxkd-61/obj38?versionId=null 2023-05-31T21:03:21.938 DEBUG:boto:Data: 2023-05-31T21:03:21.939 DEBUG:boto:Headers: {} 2023-05-31T21:03:21.939 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T21:03:21.939 DEBUG:boto:Port: 8001 2023-05-31T21:03:21.940 DEBUG:boto:Params: {} 2023-05-31T21:03:21.940 DEBUG:boto:Token: None 2023-05-31T21:03:21.941 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 21:03:21 GMT /pklxkd-61/obj38?versionId=null 2023-05-31T21:03:21.941 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:2fr8OBFgVyCTpNq0Vm3k8G6QsFo= 2023-05-31T21:03:21.941 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 21:03:21 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:2fr8OBFgVyCTpNq0Vm3k8G6QsFo=', 'Content-Length': '0'} 2023-05-31T21:03:21.943 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:59:45 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000420c86ae808b8449-006477b619-1f4e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 21:03:21 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:03:21.944 DEBUG:boto:path=/pklxkd-61/obj38 2023-05-31T21:03:21.944 DEBUG:boto:auth_path=/pklxkd-61/obj38 2023-05-31T21:03:21.944 DEBUG:boto:path=/pklxkd-61/obj38?versionId=null 2023-05-31T21:03:21.945 DEBUG:boto:auth_path=/pklxkd-61/obj38?versionId=null 2023-05-31T21:03:21.945 DEBUG:boto:Method: HEAD 2023-05-31T21:03:21.946 DEBUG:boto:Path: /pklxkd-61/obj38?versionId=null 2023-05-31T21:03:21.946 DEBUG:boto:Data: 2023-05-31T21:03:21.946 DEBUG:boto:Headers: {} 2023-05-31T21:03:21.947 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T21:03:21.947 DEBUG:boto:Port: 8000 2023-05-31T21:03:21.947 DEBUG:boto:Params: {} 2023-05-31T21:03:21.948 DEBUG:boto:Token: None 2023-05-31T21:03:21.948 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 21:03:21 GMT /pklxkd-61/obj38?versionId=null 2023-05-31T21:03:21.948 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:2fr8OBFgVyCTpNq0Vm3k8G6QsFo= 2023-05-31T21:03:21.949 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 21:03:21 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:2fr8OBFgVyCTpNq0Vm3k8G6QsFo=', 'Content-Length': '0'} 2023-05-31T21:03:21.951 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:59:45 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d58d9106ac25ec90-006477b619-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 21:03:21 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:03:21.951 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj38 2023-05-31T21:03:21.952 DEBUG:boto:path=/pklxkd-61/obj38 2023-05-31T21:03:21.952 DEBUG:boto:auth_path=/pklxkd-61/obj38 2023-05-31T21:03:21.952 DEBUG:boto:path=/pklxkd-61/obj38?versionId=null 2023-05-31T21:03:21.953 DEBUG:boto:auth_path=/pklxkd-61/obj38?versionId=null 2023-05-31T21:03:21.953 DEBUG:boto:Method: GET 2023-05-31T21:03:21.954 DEBUG:boto:Path: /pklxkd-61/obj38?versionId=null 2023-05-31T21:03:21.954 DEBUG:boto:Data: 2023-05-31T21:03:21.954 DEBUG:boto:Headers: {} 2023-05-31T21:03:21.955 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T21:03:21.955 DEBUG:boto:Port: 8001 2023-05-31T21:03:21.955 DEBUG:boto:Params: {} 2023-05-31T21:03:21.956 DEBUG:boto:Token: None 2023-05-31T21:03:21.956 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 21:03:21 GMT /pklxkd-61/obj38?versionId=null 2023-05-31T21:03:21.957 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:OLRluCbzQ8wiaJWjJ1WLg34U/KI= 2023-05-31T21:03:21.957 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 21:03:21 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:OLRluCbzQ8wiaJWjJ1WLg34U/KI=', 'Content-Length': '0'} 2023-05-31T21:03:21.959 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:59:45 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005750b0c487c90675-006477b619-1f4e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 21:03:21 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:03:21.960 DEBUG:boto:path=/pklxkd-61/obj38 2023-05-31T21:03:21.960 DEBUG:boto:auth_path=/pklxkd-61/obj38 2023-05-31T21:03:21.961 DEBUG:boto:path=/pklxkd-61/obj38?versionId=null 2023-05-31T21:03:21.961 DEBUG:boto:auth_path=/pklxkd-61/obj38?versionId=null 2023-05-31T21:03:21.961 DEBUG:boto:Method: GET 2023-05-31T21:03:21.962 DEBUG:boto:Path: /pklxkd-61/obj38?versionId=null 2023-05-31T21:03:21.962 DEBUG:boto:Data: 2023-05-31T21:03:21.963 DEBUG:boto:Headers: {} 2023-05-31T21:03:21.963 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T21:03:21.963 DEBUG:boto:Port: 8000 2023-05-31T21:03:21.964 DEBUG:boto:Params: {} 2023-05-31T21:03:21.964 DEBUG:boto:Token: None 2023-05-31T21:03:21.964 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 21:03:21 GMT /pklxkd-61/obj38?versionId=null 2023-05-31T21:03:21.965 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:OLRluCbzQ8wiaJWjJ1WLg34U/KI= 2023-05-31T21:03:21.965 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 21:03:21 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:OLRluCbzQ8wiaJWjJ1WLg34U/KI=', 'Content-Length': '0'} 2023-05-31T21:03:21.967 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:59:45 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004546a5a1ad2ed2d6-006477b619-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 21:03:21 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:03:21.968 DEBUG:boto:path=/pklxkd-61/ 2023-05-31T21:03:21.968 DEBUG:boto:auth_path=/pklxkd-61/ 2023-05-31T21:03:21.969 DEBUG:boto:path=/pklxkd-61/?prefix=obj38 2023-05-31T21:03:21.969 DEBUG:boto:auth_path=/pklxkd-61/?prefix=obj38 2023-05-31T21:03:21.969 DEBUG:boto:Method: GET 2023-05-31T21:03:21.970 DEBUG:boto:Path: /pklxkd-61/?prefix=obj38 2023-05-31T21:03:21.970 DEBUG:boto:Data: 2023-05-31T21:03:21.970 DEBUG:boto:Headers: {} 2023-05-31T21:03:21.971 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T21:03:21.971 DEBUG:boto:Port: 8001 2023-05-31T21:03:21.971 DEBUG:boto:Params: {} 2023-05-31T21:03:21.972 DEBUG:boto:Token: None 2023-05-31T21:03:21.972 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 21:03:21 GMT /pklxkd-61/ 2023-05-31T21:03:21.973 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:H9PQFgrR8ica+qPwTl/4XbpLrOQ= 2023-05-31T21:03:21.973 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 21:03:21 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:H9PQFgrR8ica+qPwTl/4XbpLrOQ=', 'Content-Length': '0'} 2023-05-31T21:03:21.975 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000006e65cafd804e1f42-006477b619-1f4e-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 21:03:21 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:03:21.976 DEBUG:boto:b'pklxkd-61obj381000falseobj382023-05-31T20:59:45.182Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T21:03:21.976 DEBUG:boto:path=/pklxkd-61/ 2023-05-31T21:03:21.977 DEBUG:boto:auth_path=/pklxkd-61/ 2023-05-31T21:03:21.977 DEBUG:boto:path=/pklxkd-61/?prefix=obj38 2023-05-31T21:03:21.978 DEBUG:boto:auth_path=/pklxkd-61/?prefix=obj38 2023-05-31T21:03:21.978 DEBUG:boto:Method: GET 2023-05-31T21:03:21.978 DEBUG:boto:Path: /pklxkd-61/?prefix=obj38 2023-05-31T21:03:21.979 DEBUG:boto:Data: 2023-05-31T21:03:21.979 DEBUG:boto:Headers: {} 2023-05-31T21:03:21.979 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T21:03:21.980 DEBUG:boto:Port: 8000 2023-05-31T21:03:21.980 DEBUG:boto:Params: {} 2023-05-31T21:03:21.981 DEBUG:boto:Token: None 2023-05-31T21:03:21.981 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 21:03:21 GMT /pklxkd-61/ 2023-05-31T21:03:21.981 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:H9PQFgrR8ica+qPwTl/4XbpLrOQ= 2023-05-31T21:03:21.982 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 21:03:21 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:H9PQFgrR8ica+qPwTl/4XbpLrOQ=', 'Content-Length': '0'} 2023-05-31T21:03:21.986 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000020d6efeffae98eb5-006477b619-1630-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 21:03:21 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:03:21.986 DEBUG:boto:b'pklxkd-61obj381000falseobj382023-05-31T20:59:45.182Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T21:03:21.987 DEBUG:boto:path=/pklxkd-61/obj38 2023-05-31T21:03:21.987 DEBUG:boto:auth_path=/pklxkd-61/obj38 2023-05-31T21:03:21.987 DEBUG:boto:Method: HEAD 2023-05-31T21:03:21.988 DEBUG:boto:Path: /pklxkd-61/obj38 2023-05-31T21:03:21.988 DEBUG:boto:Data: 2023-05-31T21:03:21.988 DEBUG:boto:Headers: {} 2023-05-31T21:03:21.989 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T21:03:21.989 DEBUG:boto:Port: 8001 2023-05-31T21:03:21.989 DEBUG:boto:Params: {} 2023-05-31T21:03:21.990 DEBUG:boto:Token: None 2023-05-31T21:03:21.990 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 21:03:21 GMT /pklxkd-61/obj38 2023-05-31T21:03:21.991 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:SOBuFt2ZE4mHVmZchdpYLRPQwms= 2023-05-31T21:03:21.991 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 21:03:21 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:SOBuFt2ZE4mHVmZchdpYLRPQwms=', 'Content-Length': '0'} 2023-05-31T21:03:21.993 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:59:45 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c3d70c8227937cf2-006477b619-1f4e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 21:03:21 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:03:21.993 DEBUG:boto:path=/pklxkd-61/obj38 2023-05-31T21:03:21.994 DEBUG:boto:auth_path=/pklxkd-61/obj38 2023-05-31T21:03:21.994 DEBUG:boto:Method: HEAD 2023-05-31T21:03:21.995 DEBUG:boto:Path: /pklxkd-61/obj38 2023-05-31T21:03:21.995 DEBUG:boto:Data: 2023-05-31T21:03:21.995 DEBUG:boto:Headers: {} 2023-05-31T21:03:21.996 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T21:03:21.996 DEBUG:boto:Port: 8000 2023-05-31T21:03:21.996 DEBUG:boto:Params: {} 2023-05-31T21:03:21.997 DEBUG:boto:Token: None 2023-05-31T21:03:21.997 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 21:03:21 GMT /pklxkd-61/obj38 2023-05-31T21:03:21.997 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:SOBuFt2ZE4mHVmZchdpYLRPQwms= 2023-05-31T21:03:21.998 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 21:03:21 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:SOBuFt2ZE4mHVmZchdpYLRPQwms=', 'Content-Length': '0'} 2023-05-31T21:03:21.999 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:59:45 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000011e191c598cc8794-006477b619-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 21:03:21 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:03:22.000 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj38 2023-05-31T21:03:22.000 DEBUG:boto:path=/pklxkd-61/obj38 2023-05-31T21:03:22.001 DEBUG:boto:auth_path=/pklxkd-61/obj38 2023-05-31T21:03:22.001 DEBUG:boto:Method: GET 2023-05-31T21:03:22.002 DEBUG:boto:Path: /pklxkd-61/obj38 2023-05-31T21:03:22.002 DEBUG:boto:Data: 2023-05-31T21:03:22.002 DEBUG:boto:Headers: {} 2023-05-31T21:03:22.003 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T21:03:22.003 DEBUG:boto:Port: 8001 2023-05-31T21:03:22.003 DEBUG:boto:Params: {} 2023-05-31T21:03:22.004 DEBUG:boto:Token: None 2023-05-31T21:03:22.004 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 21:03:22 GMT /pklxkd-61/obj38 2023-05-31T21:03:22.005 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:KTMdhsoQO57LA4ckUwTP0Qsh/8g= 2023-05-31T21:03:22.005 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 21:03:22 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:KTMdhsoQO57LA4ckUwTP0Qsh/8g=', 'Content-Length': '0'} 2023-05-31T21:03:22.007 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:59:45 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b30ceaa05dceb8be-006477b61a-1f4e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 21:03:22 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:03:22.008 DEBUG:boto:path=/pklxkd-61/obj38 2023-05-31T21:03:22.008 DEBUG:boto:auth_path=/pklxkd-61/obj38 2023-05-31T21:03:22.008 DEBUG:boto:Method: GET 2023-05-31T21:03:22.009 DEBUG:boto:Path: /pklxkd-61/obj38 2023-05-31T21:03:22.009 DEBUG:boto:Data: 2023-05-31T21:03:22.009 DEBUG:boto:Headers: {} 2023-05-31T21:03:22.010 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T21:03:22.010 DEBUG:boto:Port: 8000 2023-05-31T21:03:22.011 DEBUG:boto:Params: {} 2023-05-31T21:03:22.011 DEBUG:boto:Token: None 2023-05-31T21:03:22.011 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 21:03:22 GMT /pklxkd-61/obj38 2023-05-31T21:03:22.012 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:KTMdhsoQO57LA4ckUwTP0Qsh/8g= 2023-05-31T21:03:22.012 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 21:03:22 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:KTMdhsoQO57LA4ckUwTP0Qsh/8g=', 'Content-Length': '0'} 2023-05-31T21:03:22.014 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:59:45 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002432d6617a66851c-006477b61a-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 21:03:22 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:03:22.014 DEBUG:boto:path=/pklxkd-61/ 2023-05-31T21:03:22.015 DEBUG:boto:auth_path=/pklxkd-61/ 2023-05-31T21:03:22.015 DEBUG:boto:path=/pklxkd-61/?prefix=obj38 2023-05-31T21:03:22.016 DEBUG:boto:auth_path=/pklxkd-61/?prefix=obj38 2023-05-31T21:03:22.016 DEBUG:boto:Method: GET 2023-05-31T21:03:22.016 DEBUG:boto:Path: /pklxkd-61/?prefix=obj38 2023-05-31T21:03:22.017 DEBUG:boto:Data: 2023-05-31T21:03:22.017 DEBUG:boto:Headers: {} 2023-05-31T21:03:22.018 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T21:03:22.018 DEBUG:boto:Port: 8001 2023-05-31T21:03:22.018 DEBUG:boto:Params: {} 2023-05-31T21:03:22.019 DEBUG:boto:Token: None 2023-05-31T21:03:22.019 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 21:03:22 GMT /pklxkd-61/ 2023-05-31T21:03:22.020 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:4/489Jh79nPyUwTqEQ/UrGtMPCY= 2023-05-31T21:03:22.020 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 21:03:22 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:4/489Jh79nPyUwTqEQ/UrGtMPCY=', 'Content-Length': '0'} 2023-05-31T21:03:22.022 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ca001866f0872276-006477b61a-1f4e-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 21:03:22 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:03:22.023 DEBUG:boto:b'pklxkd-61obj381000falseobj382023-05-31T20:59:45.182Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T21:03:22.024 DEBUG:boto:path=/pklxkd-61/ 2023-05-31T21:03:22.024 DEBUG:boto:auth_path=/pklxkd-61/ 2023-05-31T21:03:22.024 DEBUG:boto:path=/pklxkd-61/?prefix=obj38 2023-05-31T21:03:22.025 DEBUG:boto:auth_path=/pklxkd-61/?prefix=obj38 2023-05-31T21:03:22.025 DEBUG:boto:Method: GET 2023-05-31T21:03:22.025 DEBUG:boto:Path: /pklxkd-61/?prefix=obj38 2023-05-31T21:03:22.026 DEBUG:boto:Data: 2023-05-31T21:03:22.026 DEBUG:boto:Headers: {} 2023-05-31T21:03:22.027 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T21:03:22.027 DEBUG:boto:Port: 8000 2023-05-31T21:03:22.027 DEBUG:boto:Params: {} 2023-05-31T21:03:22.028 DEBUG:boto:Token: None 2023-05-31T21:03:22.028 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 21:03:22 GMT /pklxkd-61/ 2023-05-31T21:03:22.029 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:4/489Jh79nPyUwTqEQ/UrGtMPCY= 2023-05-31T21:03:22.029 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 21:03:22 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:4/489Jh79nPyUwTqEQ/UrGtMPCY=', 'Content-Length': '0'} 2023-05-31T21:03:22.032 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000009938e4be67820346-006477b61a-1630-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 21:03:22 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:03:22.033 DEBUG:boto:b'pklxkd-61obj381000falseobj382023-05-31T20:59:45.182Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T21:03:22.034 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj44 2023-05-31T21:03:22.034 DEBUG:boto:path=/pklxkd-61/obj44 2023-05-31T21:03:22.034 DEBUG:boto:auth_path=/pklxkd-61/obj44 2023-05-31T21:03:22.035 DEBUG:boto:path=/pklxkd-61/obj44?versionId=null 2023-05-31T21:03:22.035 DEBUG:boto:auth_path=/pklxkd-61/obj44?versionId=null 2023-05-31T21:03:22.035 DEBUG:boto:Method: GET 2023-05-31T21:03:22.036 DEBUG:boto:Path: /pklxkd-61/obj44?versionId=null 2023-05-31T21:03:22.036 DEBUG:boto:Data: 2023-05-31T21:03:22.037 DEBUG:boto:Headers: {} 2023-05-31T21:03:22.037 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T21:03:22.037 DEBUG:boto:Port: 8001 2023-05-31T21:03:22.038 DEBUG:boto:Params: {} 2023-05-31T21:03:22.038 DEBUG:boto:Token: None 2023-05-31T21:03:22.038 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 21:03:22 GMT /pklxkd-61/obj44?versionId=null 2023-05-31T21:03:22.039 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:2yAu5Bc6A5cwNgFRgbgtymrAHys= 2023-05-31T21:03:22.039 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 21:03:22 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:2yAu5Bc6A5cwNgFRgbgtymrAHys=', 'Content-Length': '0'} 2023-05-31T21:03:22.041 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:59:44 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009eea3c644c9f5119-006477b61a-1f4e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 21:03:22 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:03:22.042 DEBUG:boto:path=/pklxkd-61/obj44 2023-05-31T21:03:22.042 DEBUG:boto:auth_path=/pklxkd-61/obj44 2023-05-31T21:03:22.043 DEBUG:boto:path=/pklxkd-61/obj44?versionId=null 2023-05-31T21:03:22.043 DEBUG:boto:auth_path=/pklxkd-61/obj44?versionId=null 2023-05-31T21:03:22.043 DEBUG:boto:Method: GET 2023-05-31T21:03:22.044 DEBUG:boto:Path: /pklxkd-61/obj44?versionId=null 2023-05-31T21:03:22.044 DEBUG:boto:Data: 2023-05-31T21:03:22.044 DEBUG:boto:Headers: {} 2023-05-31T21:03:22.045 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T21:03:22.045 DEBUG:boto:Port: 8000 2023-05-31T21:03:22.045 DEBUG:boto:Params: {} 2023-05-31T21:03:22.046 DEBUG:boto:Token: None 2023-05-31T21:03:22.046 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 21:03:22 GMT /pklxkd-61/obj44?versionId=null 2023-05-31T21:03:22.047 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:2yAu5Bc6A5cwNgFRgbgtymrAHys= 2023-05-31T21:03:22.047 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 21:03:22 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:2yAu5Bc6A5cwNgFRgbgtymrAHys=', 'Content-Length': '0'} 2023-05-31T21:03:22.049 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:59:44 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000dee1b00a6a55b7a4-006477b61a-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 21:03:22 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:03:22.096 DEBUG:boto:path=/pklxkd-61/obj44 2023-05-31T21:03:22.097 DEBUG:boto:auth_path=/pklxkd-61/obj44 2023-05-31T21:03:22.097 DEBUG:boto:path=/pklxkd-61/obj44?versionId=null 2023-05-31T21:03:22.097 DEBUG:boto:auth_path=/pklxkd-61/obj44?versionId=null 2023-05-31T21:03:22.098 DEBUG:boto:Method: HEAD 2023-05-31T21:03:22.098 DEBUG:boto:Path: /pklxkd-61/obj44?versionId=null 2023-05-31T21:03:22.098 DEBUG:boto:Data: 2023-05-31T21:03:22.099 DEBUG:boto:Headers: {} 2023-05-31T21:03:22.099 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T21:03:22.099 DEBUG:boto:Port: 8001 2023-05-31T21:03:22.100 DEBUG:boto:Params: {} 2023-05-31T21:03:22.100 DEBUG:boto:Token: None 2023-05-31T21:03:22.101 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 21:03:22 GMT /pklxkd-61/obj44?versionId=null 2023-05-31T21:03:22.101 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:JOwDniv14E43i1PidxnrgpkcvJQ= 2023-05-31T21:03:22.101 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 21:03:22 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:JOwDniv14E43i1PidxnrgpkcvJQ=', 'Content-Length': '0'} 2023-05-31T21:03:22.103 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:59:44 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001926de9329222c96-006477b61a-1f4e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 21:03:22 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:03:22.104 DEBUG:boto:path=/pklxkd-61/obj44 2023-05-31T21:03:22.104 DEBUG:boto:auth_path=/pklxkd-61/obj44 2023-05-31T21:03:22.105 DEBUG:boto:path=/pklxkd-61/obj44?versionId=null 2023-05-31T21:03:22.105 DEBUG:boto:auth_path=/pklxkd-61/obj44?versionId=null 2023-05-31T21:03:22.105 DEBUG:boto:Method: HEAD 2023-05-31T21:03:22.106 DEBUG:boto:Path: /pklxkd-61/obj44?versionId=null 2023-05-31T21:03:22.106 DEBUG:boto:Data: 2023-05-31T21:03:22.106 DEBUG:boto:Headers: {} 2023-05-31T21:03:22.107 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T21:03:22.107 DEBUG:boto:Port: 8000 2023-05-31T21:03:22.107 DEBUG:boto:Params: {} 2023-05-31T21:03:22.108 DEBUG:boto:Token: None 2023-05-31T21:03:22.108 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 21:03:22 GMT /pklxkd-61/obj44?versionId=null 2023-05-31T21:03:22.108 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:JOwDniv14E43i1PidxnrgpkcvJQ= 2023-05-31T21:03:22.109 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 21:03:22 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:JOwDniv14E43i1PidxnrgpkcvJQ=', 'Content-Length': '0'} 2023-05-31T21:03:22.111 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:59:44 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008bed96bf06d3ab48-006477b61a-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 21:03:22 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:03:22.112 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj44 2023-05-31T21:03:22.112 DEBUG:boto:path=/pklxkd-61/obj44 2023-05-31T21:03:22.112 DEBUG:boto:auth_path=/pklxkd-61/obj44 2023-05-31T21:03:22.113 DEBUG:boto:path=/pklxkd-61/obj44?versionId=null 2023-05-31T21:03:22.113 DEBUG:boto:auth_path=/pklxkd-61/obj44?versionId=null 2023-05-31T21:03:22.113 DEBUG:boto:Method: GET 2023-05-31T21:03:22.114 DEBUG:boto:Path: /pklxkd-61/obj44?versionId=null 2023-05-31T21:03:22.114 DEBUG:boto:Data: 2023-05-31T21:03:22.115 DEBUG:boto:Headers: {} 2023-05-31T21:03:22.115 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T21:03:22.115 DEBUG:boto:Port: 8001 2023-05-31T21:03:22.116 DEBUG:boto:Params: {} 2023-05-31T21:03:22.116 DEBUG:boto:Token: None 2023-05-31T21:03:22.117 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 21:03:22 GMT /pklxkd-61/obj44?versionId=null 2023-05-31T21:03:22.117 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:2yAu5Bc6A5cwNgFRgbgtymrAHys= 2023-05-31T21:03:22.117 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 21:03:22 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:2yAu5Bc6A5cwNgFRgbgtymrAHys=', 'Content-Length': '0'} 2023-05-31T21:03:22.119 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:59:44 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000eac7bb17127c9c9b-006477b61a-1f4e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 21:03:22 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:03:22.120 DEBUG:boto:path=/pklxkd-61/obj44 2023-05-31T21:03:22.120 DEBUG:boto:auth_path=/pklxkd-61/obj44 2023-05-31T21:03:22.121 DEBUG:boto:path=/pklxkd-61/obj44?versionId=null 2023-05-31T21:03:22.121 DEBUG:boto:auth_path=/pklxkd-61/obj44?versionId=null 2023-05-31T21:03:22.121 DEBUG:boto:Method: GET 2023-05-31T21:03:22.122 DEBUG:boto:Path: /pklxkd-61/obj44?versionId=null 2023-05-31T21:03:22.122 DEBUG:boto:Data: 2023-05-31T21:03:22.122 DEBUG:boto:Headers: {} 2023-05-31T21:03:22.123 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T21:03:22.123 DEBUG:boto:Port: 8000 2023-05-31T21:03:22.123 DEBUG:boto:Params: {} 2023-05-31T21:03:22.124 DEBUG:boto:Token: None 2023-05-31T21:03:22.124 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 21:03:22 GMT /pklxkd-61/obj44?versionId=null 2023-05-31T21:03:22.125 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:2yAu5Bc6A5cwNgFRgbgtymrAHys= 2023-05-31T21:03:22.125 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 21:03:22 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:2yAu5Bc6A5cwNgFRgbgtymrAHys=', 'Content-Length': '0'} 2023-05-31T21:03:22.127 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:59:44 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a2ab7b5145aecc75-006477b61a-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 21:03:22 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:03:22.128 DEBUG:boto:path=/pklxkd-61/ 2023-05-31T21:03:22.128 DEBUG:boto:auth_path=/pklxkd-61/ 2023-05-31T21:03:22.128 DEBUG:boto:path=/pklxkd-61/?prefix=obj44 2023-05-31T21:03:22.129 DEBUG:boto:auth_path=/pklxkd-61/?prefix=obj44 2023-05-31T21:03:22.129 DEBUG:boto:Method: GET 2023-05-31T21:03:22.130 DEBUG:boto:Path: /pklxkd-61/?prefix=obj44 2023-05-31T21:03:22.130 DEBUG:boto:Data: 2023-05-31T21:03:22.130 DEBUG:boto:Headers: {} 2023-05-31T21:03:22.131 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T21:03:22.131 DEBUG:boto:Port: 8001 2023-05-31T21:03:22.131 DEBUG:boto:Params: {} 2023-05-31T21:03:22.132 DEBUG:boto:Token: None 2023-05-31T21:03:22.132 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 21:03:22 GMT /pklxkd-61/ 2023-05-31T21:03:22.133 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:4/489Jh79nPyUwTqEQ/UrGtMPCY= 2023-05-31T21:03:22.133 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 21:03:22 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:4/489Jh79nPyUwTqEQ/UrGtMPCY=', 'Content-Length': '0'} 2023-05-31T21:03:22.135 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000000bbf3bcc40e0faaa-006477b61a-1f4e-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 21:03:22 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:03:22.136 DEBUG:boto:b'pklxkd-61obj441000falseobj442023-05-31T20:59:44.523Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T21:03:22.136 DEBUG:boto:path=/pklxkd-61/ 2023-05-31T21:03:22.137 DEBUG:boto:auth_path=/pklxkd-61/ 2023-05-31T21:03:22.137 DEBUG:boto:path=/pklxkd-61/?prefix=obj44 2023-05-31T21:03:22.137 DEBUG:boto:auth_path=/pklxkd-61/?prefix=obj44 2023-05-31T21:03:22.138 DEBUG:boto:Method: GET 2023-05-31T21:03:22.138 DEBUG:boto:Path: /pklxkd-61/?prefix=obj44 2023-05-31T21:03:22.139 DEBUG:boto:Data: 2023-05-31T21:03:22.139 DEBUG:boto:Headers: {} 2023-05-31T21:03:22.139 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T21:03:22.140 DEBUG:boto:Port: 8000 2023-05-31T21:03:22.140 DEBUG:boto:Params: {} 2023-05-31T21:03:22.140 DEBUG:boto:Token: None 2023-05-31T21:03:22.141 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 21:03:22 GMT /pklxkd-61/ 2023-05-31T21:03:22.141 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:4/489Jh79nPyUwTqEQ/UrGtMPCY= 2023-05-31T21:03:22.142 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 21:03:22 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:4/489Jh79nPyUwTqEQ/UrGtMPCY=', 'Content-Length': '0'} 2023-05-31T21:03:22.145 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000009c136ca80e39cc32-006477b61a-1630-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 21:03:22 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:03:22.146 DEBUG:boto:b'pklxkd-61obj441000falseobj442023-05-31T20:59:44.523Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T21:03:22.146 DEBUG:boto:path=/pklxkd-61/obj44 2023-05-31T21:03:22.147 DEBUG:boto:auth_path=/pklxkd-61/obj44 2023-05-31T21:03:22.147 DEBUG:boto:Method: HEAD 2023-05-31T21:03:22.148 DEBUG:boto:Path: /pklxkd-61/obj44 2023-05-31T21:03:22.148 DEBUG:boto:Data: 2023-05-31T21:03:22.148 DEBUG:boto:Headers: {} 2023-05-31T21:03:22.149 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T21:03:22.149 DEBUG:boto:Port: 8001 2023-05-31T21:03:22.149 DEBUG:boto:Params: {} 2023-05-31T21:03:22.150 DEBUG:boto:Token: None 2023-05-31T21:03:22.150 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 21:03:22 GMT /pklxkd-61/obj44 2023-05-31T21:03:22.150 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:97tr4clMvuziuoIRmiE9YEWtCfg= 2023-05-31T21:03:22.151 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 21:03:22 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:97tr4clMvuziuoIRmiE9YEWtCfg=', 'Content-Length': '0'} 2023-05-31T21:03:22.152 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:59:44 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000d89e50ffa13b629-006477b61a-1f4e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 21:03:22 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:03:22.153 DEBUG:boto:path=/pklxkd-61/obj44 2023-05-31T21:03:22.153 DEBUG:boto:auth_path=/pklxkd-61/obj44 2023-05-31T21:03:22.154 DEBUG:boto:Method: HEAD 2023-05-31T21:03:22.154 DEBUG:boto:Path: /pklxkd-61/obj44 2023-05-31T21:03:22.154 DEBUG:boto:Data: 2023-05-31T21:03:22.155 DEBUG:boto:Headers: {} 2023-05-31T21:03:22.155 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T21:03:22.155 DEBUG:boto:Port: 8000 2023-05-31T21:03:22.156 DEBUG:boto:Params: {} 2023-05-31T21:03:22.156 DEBUG:boto:Token: None 2023-05-31T21:03:22.157 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 21:03:22 GMT /pklxkd-61/obj44 2023-05-31T21:03:22.157 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:97tr4clMvuziuoIRmiE9YEWtCfg= 2023-05-31T21:03:22.157 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 21:03:22 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:97tr4clMvuziuoIRmiE9YEWtCfg=', 'Content-Length': '0'} 2023-05-31T21:03:22.159 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:59:44 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000066b8a7b02f313a7b-006477b61a-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 21:03:22 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:03:22.160 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj44 2023-05-31T21:03:22.160 DEBUG:boto:path=/pklxkd-61/obj44 2023-05-31T21:03:22.160 DEBUG:boto:auth_path=/pklxkd-61/obj44 2023-05-31T21:03:22.161 DEBUG:boto:Method: GET 2023-05-31T21:03:22.161 DEBUG:boto:Path: /pklxkd-61/obj44 2023-05-31T21:03:22.161 DEBUG:boto:Data: 2023-05-31T21:03:22.162 DEBUG:boto:Headers: {} 2023-05-31T21:03:22.162 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T21:03:22.163 DEBUG:boto:Port: 8001 2023-05-31T21:03:22.163 DEBUG:boto:Params: {} 2023-05-31T21:03:22.163 DEBUG:boto:Token: None 2023-05-31T21:03:22.164 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 21:03:22 GMT /pklxkd-61/obj44 2023-05-31T21:03:22.164 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:7Y0L9qXHQsrRvcJgf2cF9/2E1Tk= 2023-05-31T21:03:22.165 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 21:03:22 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:7Y0L9qXHQsrRvcJgf2cF9/2E1Tk=', 'Content-Length': '0'} 2023-05-31T21:03:22.166 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:59:44 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005263d9cae95130c9-006477b61a-1f4e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 21:03:22 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:03:22.208 DEBUG:boto:path=/pklxkd-61/obj44 2023-05-31T21:03:22.209 DEBUG:boto:auth_path=/pklxkd-61/obj44 2023-05-31T21:03:22.209 DEBUG:boto:Method: GET 2023-05-31T21:03:22.209 DEBUG:boto:Path: /pklxkd-61/obj44 2023-05-31T21:03:22.210 DEBUG:boto:Data: 2023-05-31T21:03:22.210 DEBUG:boto:Headers: {} 2023-05-31T21:03:22.210 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T21:03:22.211 DEBUG:boto:Port: 8000 2023-05-31T21:03:22.211 DEBUG:boto:Params: {} 2023-05-31T21:03:22.212 DEBUG:boto:Token: None 2023-05-31T21:03:22.212 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 21:03:22 GMT /pklxkd-61/obj44 2023-05-31T21:03:22.212 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:7Y0L9qXHQsrRvcJgf2cF9/2E1Tk= 2023-05-31T21:03:22.213 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 21:03:22 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:7Y0L9qXHQsrRvcJgf2cF9/2E1Tk=', 'Content-Length': '0'} 2023-05-31T21:03:22.215 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:59:44 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a4ac2c6eb484198a-006477b61a-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 21:03:22 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:03:22.215 DEBUG:boto:path=/pklxkd-61/ 2023-05-31T21:03:22.216 DEBUG:boto:auth_path=/pklxkd-61/ 2023-05-31T21:03:22.216 DEBUG:boto:path=/pklxkd-61/?prefix=obj44 2023-05-31T21:03:22.217 DEBUG:boto:auth_path=/pklxkd-61/?prefix=obj44 2023-05-31T21:03:22.217 DEBUG:boto:Method: GET 2023-05-31T21:03:22.217 DEBUG:boto:Path: /pklxkd-61/?prefix=obj44 2023-05-31T21:03:22.218 DEBUG:boto:Data: 2023-05-31T21:03:22.218 DEBUG:boto:Headers: {} 2023-05-31T21:03:22.218 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T21:03:22.219 DEBUG:boto:Port: 8001 2023-05-31T21:03:22.219 DEBUG:boto:Params: {} 2023-05-31T21:03:22.219 DEBUG:boto:Token: None 2023-05-31T21:03:22.220 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 21:03:22 GMT /pklxkd-61/ 2023-05-31T21:03:22.220 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:4/489Jh79nPyUwTqEQ/UrGtMPCY= 2023-05-31T21:03:22.221 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 21:03:22 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:4/489Jh79nPyUwTqEQ/UrGtMPCY=', 'Content-Length': '0'} 2023-05-31T21:03:22.223 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000007ff0a794a1537b8c-006477b61a-1f4e-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 21:03:22 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:03:22.224 DEBUG:boto:b'pklxkd-61obj441000falseobj442023-05-31T20:59:44.523Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T21:03:22.224 DEBUG:boto:path=/pklxkd-61/ 2023-05-31T21:03:22.225 DEBUG:boto:auth_path=/pklxkd-61/ 2023-05-31T21:03:22.225 DEBUG:boto:path=/pklxkd-61/?prefix=obj44 2023-05-31T21:03:22.225 DEBUG:boto:auth_path=/pklxkd-61/?prefix=obj44 2023-05-31T21:03:22.226 DEBUG:boto:Method: GET 2023-05-31T21:03:22.226 DEBUG:boto:Path: /pklxkd-61/?prefix=obj44 2023-05-31T21:03:22.226 DEBUG:boto:Data: 2023-05-31T21:03:22.227 DEBUG:boto:Headers: {} 2023-05-31T21:03:22.227 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T21:03:22.227 DEBUG:boto:Port: 8000 2023-05-31T21:03:22.228 DEBUG:boto:Params: {} 2023-05-31T21:03:22.228 DEBUG:boto:Token: None 2023-05-31T21:03:22.229 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 21:03:22 GMT /pklxkd-61/ 2023-05-31T21:03:22.229 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:4/489Jh79nPyUwTqEQ/UrGtMPCY= 2023-05-31T21:03:22.229 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 21:03:22 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:4/489Jh79nPyUwTqEQ/UrGtMPCY=', 'Content-Length': '0'} 2023-05-31T21:03:22.233 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000018ace0cf99038570-006477b61a-1630-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 21:03:22 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:03:22.234 DEBUG:boto:b'pklxkd-61obj441000falseobj442023-05-31T20:59:44.523Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T21:03:22.234 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj46 2023-05-31T21:03:22.235 DEBUG:boto:path=/pklxkd-61/obj46 2023-05-31T21:03:22.235 DEBUG:boto:auth_path=/pklxkd-61/obj46 2023-05-31T21:03:22.235 DEBUG:boto:path=/pklxkd-61/obj46?versionId=null 2023-05-31T21:03:22.236 DEBUG:boto:auth_path=/pklxkd-61/obj46?versionId=null 2023-05-31T21:03:22.236 DEBUG:boto:Method: GET 2023-05-31T21:03:22.236 DEBUG:boto:Path: /pklxkd-61/obj46?versionId=null 2023-05-31T21:03:22.237 DEBUG:boto:Data: 2023-05-31T21:03:22.237 DEBUG:boto:Headers: {} 2023-05-31T21:03:22.238 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T21:03:22.238 DEBUG:boto:Port: 8001 2023-05-31T21:03:22.238 DEBUG:boto:Params: {} 2023-05-31T21:03:22.239 DEBUG:boto:Token: None 2023-05-31T21:03:22.239 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 21:03:22 GMT /pklxkd-61/obj46?versionId=null 2023-05-31T21:03:22.239 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:euPGaE5PVYZEkDEFv7kuTzm/3B4= 2023-05-31T21:03:22.240 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 21:03:22 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:euPGaE5PVYZEkDEFv7kuTzm/3B4=', 'Content-Length': '0'} 2023-05-31T21:03:22.242 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:59:45 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000dad347aadec2a960-006477b61a-1f4e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 21:03:22 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:03:22.243 DEBUG:boto:path=/pklxkd-61/obj46 2023-05-31T21:03:22.243 DEBUG:boto:auth_path=/pklxkd-61/obj46 2023-05-31T21:03:22.243 DEBUG:boto:path=/pklxkd-61/obj46?versionId=null 2023-05-31T21:03:22.244 DEBUG:boto:auth_path=/pklxkd-61/obj46?versionId=null 2023-05-31T21:03:22.244 DEBUG:boto:Method: GET 2023-05-31T21:03:22.245 DEBUG:boto:Path: /pklxkd-61/obj46?versionId=null 2023-05-31T21:03:22.245 DEBUG:boto:Data: 2023-05-31T21:03:22.245 DEBUG:boto:Headers: {} 2023-05-31T21:03:22.246 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T21:03:22.246 DEBUG:boto:Port: 8000 2023-05-31T21:03:22.246 DEBUG:boto:Params: {} 2023-05-31T21:03:22.247 DEBUG:boto:Token: None 2023-05-31T21:03:22.247 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 21:03:22 GMT /pklxkd-61/obj46?versionId=null 2023-05-31T21:03:22.248 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:euPGaE5PVYZEkDEFv7kuTzm/3B4= 2023-05-31T21:03:22.248 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 21:03:22 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:euPGaE5PVYZEkDEFv7kuTzm/3B4=', 'Content-Length': '0'} 2023-05-31T21:03:22.250 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:59:45 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c0636e427e2cfae7-006477b61a-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 21:03:22 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:03:22.292 DEBUG:boto:path=/pklxkd-61/obj46 2023-05-31T21:03:22.293 DEBUG:boto:auth_path=/pklxkd-61/obj46 2023-05-31T21:03:22.293 DEBUG:boto:path=/pklxkd-61/obj46?versionId=null 2023-05-31T21:03:22.293 DEBUG:boto:auth_path=/pklxkd-61/obj46?versionId=null 2023-05-31T21:03:22.294 DEBUG:boto:Method: HEAD 2023-05-31T21:03:22.294 DEBUG:boto:Path: /pklxkd-61/obj46?versionId=null 2023-05-31T21:03:22.294 DEBUG:boto:Data: 2023-05-31T21:03:22.295 DEBUG:boto:Headers: {} 2023-05-31T21:03:22.295 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T21:03:22.295 DEBUG:boto:Port: 8001 2023-05-31T21:03:22.296 DEBUG:boto:Params: {} 2023-05-31T21:03:22.296 DEBUG:boto:Token: None 2023-05-31T21:03:22.296 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 21:03:22 GMT /pklxkd-61/obj46?versionId=null 2023-05-31T21:03:22.297 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:amjMUzZTIwd2qGYyKk2bB9Bx6Ds= 2023-05-31T21:03:22.297 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 21:03:22 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:amjMUzZTIwd2qGYyKk2bB9Bx6Ds=', 'Content-Length': '0'} 2023-05-31T21:03:22.299 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:59:45 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000022152db130816e6b-006477b61a-1f4e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 21:03:22 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:03:22.300 DEBUG:boto:path=/pklxkd-61/obj46 2023-05-31T21:03:22.300 DEBUG:boto:auth_path=/pklxkd-61/obj46 2023-05-31T21:03:22.300 DEBUG:boto:path=/pklxkd-61/obj46?versionId=null 2023-05-31T21:03:22.301 DEBUG:boto:auth_path=/pklxkd-61/obj46?versionId=null 2023-05-31T21:03:22.301 DEBUG:boto:Method: HEAD 2023-05-31T21:03:22.301 DEBUG:boto:Path: /pklxkd-61/obj46?versionId=null 2023-05-31T21:03:22.302 DEBUG:boto:Data: 2023-05-31T21:03:22.302 DEBUG:boto:Headers: {} 2023-05-31T21:03:22.302 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T21:03:22.303 DEBUG:boto:Port: 8000 2023-05-31T21:03:22.303 DEBUG:boto:Params: {} 2023-05-31T21:03:22.304 DEBUG:boto:Token: None 2023-05-31T21:03:22.304 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 21:03:22 GMT /pklxkd-61/obj46?versionId=null 2023-05-31T21:03:22.304 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:amjMUzZTIwd2qGYyKk2bB9Bx6Ds= 2023-05-31T21:03:22.305 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 21:03:22 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:amjMUzZTIwd2qGYyKk2bB9Bx6Ds=', 'Content-Length': '0'} 2023-05-31T21:03:22.306 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:59:45 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000295d224e1d7a8245-006477b61a-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 21:03:22 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:03:22.307 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj46 2023-05-31T21:03:22.307 DEBUG:boto:path=/pklxkd-61/obj46 2023-05-31T21:03:22.308 DEBUG:boto:auth_path=/pklxkd-61/obj46 2023-05-31T21:03:22.308 DEBUG:boto:path=/pklxkd-61/obj46?versionId=null 2023-05-31T21:03:22.308 DEBUG:boto:auth_path=/pklxkd-61/obj46?versionId=null 2023-05-31T21:03:22.309 DEBUG:boto:Method: GET 2023-05-31T21:03:22.309 DEBUG:boto:Path: /pklxkd-61/obj46?versionId=null 2023-05-31T21:03:22.309 DEBUG:boto:Data: 2023-05-31T21:03:22.310 DEBUG:boto:Headers: {} 2023-05-31T21:03:22.310 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T21:03:22.311 DEBUG:boto:Port: 8001 2023-05-31T21:03:22.311 DEBUG:boto:Params: {} 2023-05-31T21:03:22.311 DEBUG:boto:Token: None 2023-05-31T21:03:22.312 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 21:03:22 GMT /pklxkd-61/obj46?versionId=null 2023-05-31T21:03:22.312 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:euPGaE5PVYZEkDEFv7kuTzm/3B4= 2023-05-31T21:03:22.313 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 21:03:22 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:euPGaE5PVYZEkDEFv7kuTzm/3B4=', 'Content-Length': '0'} 2023-05-31T21:03:22.314 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:59:45 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000051c897264955bf1d-006477b61a-1f4e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 21:03:22 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:03:22.315 DEBUG:boto:path=/pklxkd-61/obj46 2023-05-31T21:03:22.315 DEBUG:boto:auth_path=/pklxkd-61/obj46 2023-05-31T21:03:22.316 DEBUG:boto:path=/pklxkd-61/obj46?versionId=null 2023-05-31T21:03:22.316 DEBUG:boto:auth_path=/pklxkd-61/obj46?versionId=null 2023-05-31T21:03:22.316 DEBUG:boto:Method: GET 2023-05-31T21:03:22.317 DEBUG:boto:Path: /pklxkd-61/obj46?versionId=null 2023-05-31T21:03:22.317 DEBUG:boto:Data: 2023-05-31T21:03:22.317 DEBUG:boto:Headers: {} 2023-05-31T21:03:22.318 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T21:03:22.318 DEBUG:boto:Port: 8000 2023-05-31T21:03:22.319 DEBUG:boto:Params: {} 2023-05-31T21:03:22.319 DEBUG:boto:Token: None 2023-05-31T21:03:22.319 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 21:03:22 GMT /pklxkd-61/obj46?versionId=null 2023-05-31T21:03:22.320 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:euPGaE5PVYZEkDEFv7kuTzm/3B4= 2023-05-31T21:03:22.320 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 21:03:22 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:euPGaE5PVYZEkDEFv7kuTzm/3B4=', 'Content-Length': '0'} 2023-05-31T21:03:22.322 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:59:45 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000be535653a946c0b-006477b61a-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 21:03:22 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:03:22.323 DEBUG:boto:path=/pklxkd-61/ 2023-05-31T21:03:22.323 DEBUG:boto:auth_path=/pklxkd-61/ 2023-05-31T21:03:22.323 DEBUG:boto:path=/pklxkd-61/?prefix=obj46 2023-05-31T21:03:22.324 DEBUG:boto:auth_path=/pklxkd-61/?prefix=obj46 2023-05-31T21:03:22.324 DEBUG:boto:Method: GET 2023-05-31T21:03:22.325 DEBUG:boto:Path: /pklxkd-61/?prefix=obj46 2023-05-31T21:03:22.325 DEBUG:boto:Data: 2023-05-31T21:03:22.325 DEBUG:boto:Headers: {} 2023-05-31T21:03:22.326 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T21:03:22.326 DEBUG:boto:Port: 8001 2023-05-31T21:03:22.327 DEBUG:boto:Params: {} 2023-05-31T21:03:22.327 DEBUG:boto:Token: None 2023-05-31T21:03:22.328 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 21:03:22 GMT /pklxkd-61/ 2023-05-31T21:03:22.328 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:4/489Jh79nPyUwTqEQ/UrGtMPCY= 2023-05-31T21:03:22.329 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 21:03:22 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:4/489Jh79nPyUwTqEQ/UrGtMPCY=', 'Content-Length': '0'} 2023-05-31T21:03:22.331 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000082f62e27d4c9dfca-006477b61a-1f4e-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 21:03:22 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:03:22.332 DEBUG:boto:b'pklxkd-61obj461000falseobj462023-05-31T20:59:45.539Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T21:03:22.333 DEBUG:boto:path=/pklxkd-61/ 2023-05-31T21:03:22.333 DEBUG:boto:auth_path=/pklxkd-61/ 2023-05-31T21:03:22.334 DEBUG:boto:path=/pklxkd-61/?prefix=obj46 2023-05-31T21:03:22.334 DEBUG:boto:auth_path=/pklxkd-61/?prefix=obj46 2023-05-31T21:03:22.334 DEBUG:boto:Method: GET 2023-05-31T21:03:22.335 DEBUG:boto:Path: /pklxkd-61/?prefix=obj46 2023-05-31T21:03:22.335 DEBUG:boto:Data: 2023-05-31T21:03:22.335 DEBUG:boto:Headers: {} 2023-05-31T21:03:22.336 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T21:03:22.336 DEBUG:boto:Port: 8000 2023-05-31T21:03:22.337 DEBUG:boto:Params: {} 2023-05-31T21:03:22.337 DEBUG:boto:Token: None 2023-05-31T21:03:22.338 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 21:03:22 GMT /pklxkd-61/ 2023-05-31T21:03:22.338 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:4/489Jh79nPyUwTqEQ/UrGtMPCY= 2023-05-31T21:03:22.338 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 21:03:22 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:4/489Jh79nPyUwTqEQ/UrGtMPCY=', 'Content-Length': '0'} 2023-05-31T21:03:22.342 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000313cd12f5aa07c08-006477b61a-1630-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 21:03:22 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:03:22.343 DEBUG:boto:b'pklxkd-61obj461000falseobj462023-05-31T20:59:45.539Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T21:03:22.343 DEBUG:boto:path=/pklxkd-61/obj46 2023-05-31T21:03:22.344 DEBUG:boto:auth_path=/pklxkd-61/obj46 2023-05-31T21:03:22.344 DEBUG:boto:Method: HEAD 2023-05-31T21:03:22.345 DEBUG:boto:Path: /pklxkd-61/obj46 2023-05-31T21:03:22.345 DEBUG:boto:Data: 2023-05-31T21:03:22.345 DEBUG:boto:Headers: {} 2023-05-31T21:03:22.346 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T21:03:22.346 DEBUG:boto:Port: 8001 2023-05-31T21:03:22.346 DEBUG:boto:Params: {} 2023-05-31T21:03:22.347 DEBUG:boto:Token: None 2023-05-31T21:03:22.347 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 21:03:22 GMT /pklxkd-61/obj46 2023-05-31T21:03:22.347 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:xmMmyUUdwgGSJQls6nTzvi+jdSc= 2023-05-31T21:03:22.348 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 21:03:22 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:xmMmyUUdwgGSJQls6nTzvi+jdSc=', 'Content-Length': '0'} 2023-05-31T21:03:22.350 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:59:45 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b209dbc80253a39f-006477b61a-1f4e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 21:03:22 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:03:22.350 DEBUG:boto:path=/pklxkd-61/obj46 2023-05-31T21:03:22.351 DEBUG:boto:auth_path=/pklxkd-61/obj46 2023-05-31T21:03:22.351 DEBUG:boto:Method: HEAD 2023-05-31T21:03:22.351 DEBUG:boto:Path: /pklxkd-61/obj46 2023-05-31T21:03:22.352 DEBUG:boto:Data: 2023-05-31T21:03:22.352 DEBUG:boto:Headers: {} 2023-05-31T21:03:22.352 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T21:03:22.353 DEBUG:boto:Port: 8000 2023-05-31T21:03:22.353 DEBUG:boto:Params: {} 2023-05-31T21:03:22.354 DEBUG:boto:Token: None 2023-05-31T21:03:22.354 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 21:03:22 GMT /pklxkd-61/obj46 2023-05-31T21:03:22.354 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:xmMmyUUdwgGSJQls6nTzvi+jdSc= 2023-05-31T21:03:22.355 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 21:03:22 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:xmMmyUUdwgGSJQls6nTzvi+jdSc=', 'Content-Length': '0'} 2023-05-31T21:03:22.358 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:59:45 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003810c364c8d5e700-006477b61a-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 21:03:22 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:03:22.359 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj46 2023-05-31T21:03:22.359 DEBUG:boto:path=/pklxkd-61/obj46 2023-05-31T21:03:22.360 DEBUG:boto:auth_path=/pklxkd-61/obj46 2023-05-31T21:03:22.360 DEBUG:boto:Method: GET 2023-05-31T21:03:22.360 DEBUG:boto:Path: /pklxkd-61/obj46 2023-05-31T21:03:22.361 DEBUG:boto:Data: 2023-05-31T21:03:22.361 DEBUG:boto:Headers: {} 2023-05-31T21:03:22.362 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T21:03:22.362 DEBUG:boto:Port: 8001 2023-05-31T21:03:22.362 DEBUG:boto:Params: {} 2023-05-31T21:03:22.363 DEBUG:boto:Token: None 2023-05-31T21:03:22.363 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 21:03:22 GMT /pklxkd-61/obj46 2023-05-31T21:03:22.364 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:gekjC+/Z6JAHoDEr/xO31ZgT8o0= 2023-05-31T21:03:22.364 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 21:03:22 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:gekjC+/Z6JAHoDEr/xO31ZgT8o0=', 'Content-Length': '0'} 2023-05-31T21:03:22.366 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:59:45 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d645a1be91e8a8d7-006477b61a-1f4e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 21:03:22 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:03:22.367 DEBUG:boto:path=/pklxkd-61/obj46 2023-05-31T21:03:22.367 DEBUG:boto:auth_path=/pklxkd-61/obj46 2023-05-31T21:03:22.368 DEBUG:boto:Method: GET 2023-05-31T21:03:22.368 DEBUG:boto:Path: /pklxkd-61/obj46 2023-05-31T21:03:22.368 DEBUG:boto:Data: 2023-05-31T21:03:22.369 DEBUG:boto:Headers: {} 2023-05-31T21:03:22.369 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T21:03:22.369 DEBUG:boto:Port: 8000 2023-05-31T21:03:22.370 DEBUG:boto:Params: {} 2023-05-31T21:03:22.370 DEBUG:boto:Token: None 2023-05-31T21:03:22.371 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 21:03:22 GMT /pklxkd-61/obj46 2023-05-31T21:03:22.371 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:gekjC+/Z6JAHoDEr/xO31ZgT8o0= 2023-05-31T21:03:22.371 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 21:03:22 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:gekjC+/Z6JAHoDEr/xO31ZgT8o0=', 'Content-Length': '0'} 2023-05-31T21:03:22.373 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:59:45 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000049f7700a7fdd1500-006477b61a-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 21:03:22 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:03:22.374 DEBUG:boto:path=/pklxkd-61/ 2023-05-31T21:03:22.374 DEBUG:boto:auth_path=/pklxkd-61/ 2023-05-31T21:03:22.375 DEBUG:boto:path=/pklxkd-61/?prefix=obj46 2023-05-31T21:03:22.375 DEBUG:boto:auth_path=/pklxkd-61/?prefix=obj46 2023-05-31T21:03:22.376 DEBUG:boto:Method: GET 2023-05-31T21:03:22.376 DEBUG:boto:Path: /pklxkd-61/?prefix=obj46 2023-05-31T21:03:22.376 DEBUG:boto:Data: 2023-05-31T21:03:22.377 DEBUG:boto:Headers: {} 2023-05-31T21:03:22.377 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T21:03:22.377 DEBUG:boto:Port: 8001 2023-05-31T21:03:22.378 DEBUG:boto:Params: {} 2023-05-31T21:03:22.378 DEBUG:boto:Token: None 2023-05-31T21:03:22.379 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 21:03:22 GMT /pklxkd-61/ 2023-05-31T21:03:22.379 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:4/489Jh79nPyUwTqEQ/UrGtMPCY= 2023-05-31T21:03:22.379 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 21:03:22 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:4/489Jh79nPyUwTqEQ/UrGtMPCY=', 'Content-Length': '0'} 2023-05-31T21:03:22.382 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000004c2c3a2ba68e67cd-006477b61a-1f4e-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 21:03:22 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:03:22.383 DEBUG:boto:b'pklxkd-61obj461000falseobj462023-05-31T20:59:45.539Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T21:03:22.384 DEBUG:boto:path=/pklxkd-61/ 2023-05-31T21:03:22.384 DEBUG:boto:auth_path=/pklxkd-61/ 2023-05-31T21:03:22.385 DEBUG:boto:path=/pklxkd-61/?prefix=obj46 2023-05-31T21:03:22.385 DEBUG:boto:auth_path=/pklxkd-61/?prefix=obj46 2023-05-31T21:03:22.386 DEBUG:boto:Method: GET 2023-05-31T21:03:22.386 DEBUG:boto:Path: /pklxkd-61/?prefix=obj46 2023-05-31T21:03:22.387 DEBUG:boto:Data: 2023-05-31T21:03:22.388 DEBUG:boto:Headers: {} 2023-05-31T21:03:22.388 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T21:03:22.389 DEBUG:boto:Port: 8000 2023-05-31T21:03:22.390 DEBUG:boto:Params: {} 2023-05-31T21:03:22.390 DEBUG:boto:Token: None 2023-05-31T21:03:22.391 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 21:03:22 GMT /pklxkd-61/ 2023-05-31T21:03:22.392 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:4/489Jh79nPyUwTqEQ/UrGtMPCY= 2023-05-31T21:03:22.392 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 21:03:22 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:4/489Jh79nPyUwTqEQ/UrGtMPCY=', 'Content-Length': '0'} 2023-05-31T21:03:22.396 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e92e303c8a0ca84e-006477b61a-1630-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 21:03:22 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:03:22.397 DEBUG:boto:b'pklxkd-61obj461000falseobj462023-05-31T20:59:45.539Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T21:03:22.398 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj51 2023-05-31T21:03:22.398 DEBUG:boto:path=/pklxkd-61/obj51 2023-05-31T21:03:22.399 DEBUG:boto:auth_path=/pklxkd-61/obj51 2023-05-31T21:03:22.399 DEBUG:boto:path=/pklxkd-61/obj51?versionId=null 2023-05-31T21:03:22.399 DEBUG:boto:auth_path=/pklxkd-61/obj51?versionId=null 2023-05-31T21:03:22.400 DEBUG:boto:Method: GET 2023-05-31T21:03:22.400 DEBUG:boto:Path: /pklxkd-61/obj51?versionId=null 2023-05-31T21:03:22.401 DEBUG:boto:Data: 2023-05-31T21:03:22.401 DEBUG:boto:Headers: {} 2023-05-31T21:03:22.401 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T21:03:22.402 DEBUG:boto:Port: 8001 2023-05-31T21:03:22.402 DEBUG:boto:Params: {} 2023-05-31T21:03:22.403 DEBUG:boto:Token: None 2023-05-31T21:03:22.403 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 21:03:22 GMT /pklxkd-61/obj51?versionId=null 2023-05-31T21:03:22.403 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:MFCSWuX41Z5xmgw6W8Y0ToXNsyk= 2023-05-31T21:03:22.404 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 21:03:22 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:MFCSWuX41Z5xmgw6W8Y0ToXNsyk=', 'Content-Length': '0'} 2023-05-31T21:03:22.405 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:59:44 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e86814187062191b-006477b61a-1f4e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 21:03:22 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:03:22.406 DEBUG:boto:path=/pklxkd-61/obj51 2023-05-31T21:03:22.407 DEBUG:boto:auth_path=/pklxkd-61/obj51 2023-05-31T21:03:22.407 DEBUG:boto:path=/pklxkd-61/obj51?versionId=null 2023-05-31T21:03:22.408 DEBUG:boto:auth_path=/pklxkd-61/obj51?versionId=null 2023-05-31T21:03:22.408 DEBUG:boto:Method: GET 2023-05-31T21:03:22.408 DEBUG:boto:Path: /pklxkd-61/obj51?versionId=null 2023-05-31T21:03:22.409 DEBUG:boto:Data: 2023-05-31T21:03:22.409 DEBUG:boto:Headers: {} 2023-05-31T21:03:22.409 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T21:03:22.410 DEBUG:boto:Port: 8000 2023-05-31T21:03:22.410 DEBUG:boto:Params: {} 2023-05-31T21:03:22.411 DEBUG:boto:Token: None 2023-05-31T21:03:22.411 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 21:03:22 GMT /pklxkd-61/obj51?versionId=null 2023-05-31T21:03:22.411 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:MFCSWuX41Z5xmgw6W8Y0ToXNsyk= 2023-05-31T21:03:22.412 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 21:03:22 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:MFCSWuX41Z5xmgw6W8Y0ToXNsyk=', 'Content-Length': '0'} 2023-05-31T21:03:22.414 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:59:44 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a7d3393c195af8ad-006477b61a-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 21:03:22 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:03:22.456 DEBUG:boto:path=/pklxkd-61/obj51 2023-05-31T21:03:22.457 DEBUG:boto:auth_path=/pklxkd-61/obj51 2023-05-31T21:03:22.457 DEBUG:boto:path=/pklxkd-61/obj51?versionId=null 2023-05-31T21:03:22.457 DEBUG:boto:auth_path=/pklxkd-61/obj51?versionId=null 2023-05-31T21:03:22.458 DEBUG:boto:Method: HEAD 2023-05-31T21:03:22.458 DEBUG:boto:Path: /pklxkd-61/obj51?versionId=null 2023-05-31T21:03:22.458 DEBUG:boto:Data: 2023-05-31T21:03:22.459 DEBUG:boto:Headers: {} 2023-05-31T21:03:22.459 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T21:03:22.459 DEBUG:boto:Port: 8001 2023-05-31T21:03:22.460 DEBUG:boto:Params: {} 2023-05-31T21:03:22.460 DEBUG:boto:Token: None 2023-05-31T21:03:22.461 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 21:03:22 GMT /pklxkd-61/obj51?versionId=null 2023-05-31T21:03:22.461 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:YXoK9c/CusrHhVOXfrnknZuoIcU= 2023-05-31T21:03:22.462 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 21:03:22 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:YXoK9c/CusrHhVOXfrnknZuoIcU=', 'Content-Length': '0'} 2023-05-31T21:03:22.463 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:59:44 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000004369371253d0d1-006477b61a-1f4e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 21:03:22 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:03:22.464 DEBUG:boto:path=/pklxkd-61/obj51 2023-05-31T21:03:22.464 DEBUG:boto:auth_path=/pklxkd-61/obj51 2023-05-31T21:03:22.464 DEBUG:boto:path=/pklxkd-61/obj51?versionId=null 2023-05-31T21:03:22.465 DEBUG:boto:auth_path=/pklxkd-61/obj51?versionId=null 2023-05-31T21:03:22.465 DEBUG:boto:Method: HEAD 2023-05-31T21:03:22.465 DEBUG:boto:Path: /pklxkd-61/obj51?versionId=null 2023-05-31T21:03:22.466 DEBUG:boto:Data: 2023-05-31T21:03:22.466 DEBUG:boto:Headers: {} 2023-05-31T21:03:22.466 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T21:03:22.467 DEBUG:boto:Port: 8000 2023-05-31T21:03:22.467 DEBUG:boto:Params: {} 2023-05-31T21:03:22.467 DEBUG:boto:Token: None 2023-05-31T21:03:22.468 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 21:03:22 GMT /pklxkd-61/obj51?versionId=null 2023-05-31T21:03:22.468 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:YXoK9c/CusrHhVOXfrnknZuoIcU= 2023-05-31T21:03:22.469 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 21:03:22 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:YXoK9c/CusrHhVOXfrnknZuoIcU=', 'Content-Length': '0'} 2023-05-31T21:03:22.470 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:59:44 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008abea524299f8243-006477b61a-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 21:03:22 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:03:22.471 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj51 2023-05-31T21:03:22.471 DEBUG:boto:path=/pklxkd-61/obj51 2023-05-31T21:03:22.472 DEBUG:boto:auth_path=/pklxkd-61/obj51 2023-05-31T21:03:22.472 DEBUG:boto:path=/pklxkd-61/obj51?versionId=null 2023-05-31T21:03:22.472 DEBUG:boto:auth_path=/pklxkd-61/obj51?versionId=null 2023-05-31T21:03:22.473 DEBUG:boto:Method: GET 2023-05-31T21:03:22.473 DEBUG:boto:Path: /pklxkd-61/obj51?versionId=null 2023-05-31T21:03:22.473 DEBUG:boto:Data: 2023-05-31T21:03:22.474 DEBUG:boto:Headers: {} 2023-05-31T21:03:22.474 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T21:03:22.474 DEBUG:boto:Port: 8001 2023-05-31T21:03:22.475 DEBUG:boto:Params: {} 2023-05-31T21:03:22.475 DEBUG:boto:Token: None 2023-05-31T21:03:22.476 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 21:03:22 GMT /pklxkd-61/obj51?versionId=null 2023-05-31T21:03:22.476 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:MFCSWuX41Z5xmgw6W8Y0ToXNsyk= 2023-05-31T21:03:22.476 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 21:03:22 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:MFCSWuX41Z5xmgw6W8Y0ToXNsyk=', 'Content-Length': '0'} 2023-05-31T21:03:22.478 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:59:44 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002c04b03f9f8c214d-006477b61a-1f4e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 21:03:22 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:03:22.479 DEBUG:boto:path=/pklxkd-61/obj51 2023-05-31T21:03:22.479 DEBUG:boto:auth_path=/pklxkd-61/obj51 2023-05-31T21:03:22.480 DEBUG:boto:path=/pklxkd-61/obj51?versionId=null 2023-05-31T21:03:22.480 DEBUG:boto:auth_path=/pklxkd-61/obj51?versionId=null 2023-05-31T21:03:22.480 DEBUG:boto:Method: GET 2023-05-31T21:03:22.481 DEBUG:boto:Path: /pklxkd-61/obj51?versionId=null 2023-05-31T21:03:22.481 DEBUG:boto:Data: 2023-05-31T21:03:22.481 DEBUG:boto:Headers: {} 2023-05-31T21:03:22.482 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T21:03:22.482 DEBUG:boto:Port: 8000 2023-05-31T21:03:22.482 DEBUG:boto:Params: {} 2023-05-31T21:03:22.483 DEBUG:boto:Token: None 2023-05-31T21:03:22.483 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 21:03:22 GMT /pklxkd-61/obj51?versionId=null 2023-05-31T21:03:22.484 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:MFCSWuX41Z5xmgw6W8Y0ToXNsyk= 2023-05-31T21:03:22.484 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 21:03:22 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:MFCSWuX41Z5xmgw6W8Y0ToXNsyk=', 'Content-Length': '0'} 2023-05-31T21:03:22.486 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:59:44 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005529cb4301196851-006477b61a-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 21:03:22 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:03:22.487 DEBUG:boto:path=/pklxkd-61/ 2023-05-31T21:03:22.487 DEBUG:boto:auth_path=/pklxkd-61/ 2023-05-31T21:03:22.487 DEBUG:boto:path=/pklxkd-61/?prefix=obj51 2023-05-31T21:03:22.488 DEBUG:boto:auth_path=/pklxkd-61/?prefix=obj51 2023-05-31T21:03:22.488 DEBUG:boto:Method: GET 2023-05-31T21:03:22.488 DEBUG:boto:Path: /pklxkd-61/?prefix=obj51 2023-05-31T21:03:22.489 DEBUG:boto:Data: 2023-05-31T21:03:22.489 DEBUG:boto:Headers: {} 2023-05-31T21:03:22.490 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T21:03:22.490 DEBUG:boto:Port: 8001 2023-05-31T21:03:22.490 DEBUG:boto:Params: {} 2023-05-31T21:03:22.491 DEBUG:boto:Token: None 2023-05-31T21:03:22.491 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 21:03:22 GMT /pklxkd-61/ 2023-05-31T21:03:22.492 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:4/489Jh79nPyUwTqEQ/UrGtMPCY= 2023-05-31T21:03:22.492 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 21:03:22 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:4/489Jh79nPyUwTqEQ/UrGtMPCY=', 'Content-Length': '0'} 2023-05-31T21:03:22.494 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000001597c3bfe70a2d4c-006477b61a-1f4e-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 21:03:22 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:03:22.495 DEBUG:boto:b'pklxkd-61obj511000falseobj512023-05-31T20:59:44.841Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T21:03:22.496 DEBUG:boto:path=/pklxkd-61/ 2023-05-31T21:03:22.496 DEBUG:boto:auth_path=/pklxkd-61/ 2023-05-31T21:03:22.497 DEBUG:boto:path=/pklxkd-61/?prefix=obj51 2023-05-31T21:03:22.497 DEBUG:boto:auth_path=/pklxkd-61/?prefix=obj51 2023-05-31T21:03:22.497 DEBUG:boto:Method: GET 2023-05-31T21:03:22.498 DEBUG:boto:Path: /pklxkd-61/?prefix=obj51 2023-05-31T21:03:22.498 DEBUG:boto:Data: 2023-05-31T21:03:22.498 DEBUG:boto:Headers: {} 2023-05-31T21:03:22.499 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T21:03:22.499 DEBUG:boto:Port: 8000 2023-05-31T21:03:22.500 DEBUG:boto:Params: {} 2023-05-31T21:03:22.500 DEBUG:boto:Token: None 2023-05-31T21:03:22.501 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 21:03:22 GMT /pklxkd-61/ 2023-05-31T21:03:22.501 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:4/489Jh79nPyUwTqEQ/UrGtMPCY= 2023-05-31T21:03:22.502 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 21:03:22 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:4/489Jh79nPyUwTqEQ/UrGtMPCY=', 'Content-Length': '0'} 2023-05-31T21:03:22.505 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000009d935a3773c4e7b7-006477b61a-1630-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 21:03:22 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:03:22.506 DEBUG:boto:b'pklxkd-61obj511000falseobj512023-05-31T20:59:44.841Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T21:03:22.506 DEBUG:boto:path=/pklxkd-61/obj51 2023-05-31T21:03:22.507 DEBUG:boto:auth_path=/pklxkd-61/obj51 2023-05-31T21:03:22.507 DEBUG:boto:Method: HEAD 2023-05-31T21:03:22.507 DEBUG:boto:Path: /pklxkd-61/obj51 2023-05-31T21:03:22.508 DEBUG:boto:Data: 2023-05-31T21:03:22.508 DEBUG:boto:Headers: {} 2023-05-31T21:03:22.508 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T21:03:22.509 DEBUG:boto:Port: 8001 2023-05-31T21:03:22.509 DEBUG:boto:Params: {} 2023-05-31T21:03:22.510 DEBUG:boto:Token: None 2023-05-31T21:03:22.510 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 21:03:22 GMT /pklxkd-61/obj51 2023-05-31T21:03:22.510 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:115AQ3H3pDtST87CX9o5l581+Gc= 2023-05-31T21:03:22.511 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 21:03:22 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:115AQ3H3pDtST87CX9o5l581+Gc=', 'Content-Length': '0'} 2023-05-31T21:03:22.512 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:59:44 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000013d661686fd0f7b4-006477b61a-1f4e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 21:03:22 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:03:22.513 DEBUG:boto:path=/pklxkd-61/obj51 2023-05-31T21:03:22.513 DEBUG:boto:auth_path=/pklxkd-61/obj51 2023-05-31T21:03:22.514 DEBUG:boto:Method: HEAD 2023-05-31T21:03:22.514 DEBUG:boto:Path: /pklxkd-61/obj51 2023-05-31T21:03:22.514 DEBUG:boto:Data: 2023-05-31T21:03:22.515 DEBUG:boto:Headers: {} 2023-05-31T21:03:22.515 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T21:03:22.515 DEBUG:boto:Port: 8000 2023-05-31T21:03:22.516 DEBUG:boto:Params: {} 2023-05-31T21:03:22.516 DEBUG:boto:Token: None 2023-05-31T21:03:22.516 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 21:03:22 GMT /pklxkd-61/obj51 2023-05-31T21:03:22.517 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:115AQ3H3pDtST87CX9o5l581+Gc= 2023-05-31T21:03:22.517 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 21:03:22 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:115AQ3H3pDtST87CX9o5l581+Gc=', 'Content-Length': '0'} 2023-05-31T21:03:22.519 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:59:44 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a7c9676110213e55-006477b61a-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 21:03:22 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:03:22.519 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj51 2023-05-31T21:03:22.520 DEBUG:boto:path=/pklxkd-61/obj51 2023-05-31T21:03:22.520 DEBUG:boto:auth_path=/pklxkd-61/obj51 2023-05-31T21:03:22.521 DEBUG:boto:Method: GET 2023-05-31T21:03:22.521 DEBUG:boto:Path: /pklxkd-61/obj51 2023-05-31T21:03:22.521 DEBUG:boto:Data: 2023-05-31T21:03:22.522 DEBUG:boto:Headers: {} 2023-05-31T21:03:22.522 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T21:03:22.522 DEBUG:boto:Port: 8001 2023-05-31T21:03:22.523 DEBUG:boto:Params: {} 2023-05-31T21:03:22.523 DEBUG:boto:Token: None 2023-05-31T21:03:22.524 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 21:03:22 GMT /pklxkd-61/obj51 2023-05-31T21:03:22.524 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:qgqbHsaESORmDXTPzGxyvgq8E/Y= 2023-05-31T21:03:22.524 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 21:03:22 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:qgqbHsaESORmDXTPzGxyvgq8E/Y=', 'Content-Length': '0'} 2023-05-31T21:03:22.526 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:59:44 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000af6112c65eadc1f6-006477b61a-1f4e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 21:03:22 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:03:22.568 DEBUG:boto:path=/pklxkd-61/obj51 2023-05-31T21:03:22.569 DEBUG:boto:auth_path=/pklxkd-61/obj51 2023-05-31T21:03:22.569 DEBUG:boto:Method: GET 2023-05-31T21:03:22.569 DEBUG:boto:Path: /pklxkd-61/obj51 2023-05-31T21:03:22.570 DEBUG:boto:Data: 2023-05-31T21:03:22.570 DEBUG:boto:Headers: {} 2023-05-31T21:03:22.570 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T21:03:22.571 DEBUG:boto:Port: 8000 2023-05-31T21:03:22.571 DEBUG:boto:Params: {} 2023-05-31T21:03:22.572 DEBUG:boto:Token: None 2023-05-31T21:03:22.572 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 21:03:22 GMT /pklxkd-61/obj51 2023-05-31T21:03:22.573 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:qgqbHsaESORmDXTPzGxyvgq8E/Y= 2023-05-31T21:03:22.573 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 21:03:22 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:qgqbHsaESORmDXTPzGxyvgq8E/Y=', 'Content-Length': '0'} 2023-05-31T21:03:22.575 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:59:44 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006b2e4ed22db9f4e2-006477b61a-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 21:03:22 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:03:22.575 DEBUG:boto:path=/pklxkd-61/ 2023-05-31T21:03:22.576 DEBUG:boto:auth_path=/pklxkd-61/ 2023-05-31T21:03:22.576 DEBUG:boto:path=/pklxkd-61/?prefix=obj51 2023-05-31T21:03:22.577 DEBUG:boto:auth_path=/pklxkd-61/?prefix=obj51 2023-05-31T21:03:22.577 DEBUG:boto:Method: GET 2023-05-31T21:03:22.577 DEBUG:boto:Path: /pklxkd-61/?prefix=obj51 2023-05-31T21:03:22.578 DEBUG:boto:Data: 2023-05-31T21:03:22.578 DEBUG:boto:Headers: {} 2023-05-31T21:03:22.578 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T21:03:22.579 DEBUG:boto:Port: 8001 2023-05-31T21:03:22.579 DEBUG:boto:Params: {} 2023-05-31T21:03:22.580 DEBUG:boto:Token: None 2023-05-31T21:03:22.580 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 21:03:22 GMT /pklxkd-61/ 2023-05-31T21:03:22.580 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:4/489Jh79nPyUwTqEQ/UrGtMPCY= 2023-05-31T21:03:22.581 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 21:03:22 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:4/489Jh79nPyUwTqEQ/UrGtMPCY=', 'Content-Length': '0'} 2023-05-31T21:03:22.583 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d5a281bbe9601ff7-006477b61a-1f4e-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 21:03:22 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:03:22.584 DEBUG:boto:b'pklxkd-61obj511000falseobj512023-05-31T20:59:44.841Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T21:03:22.584 DEBUG:boto:path=/pklxkd-61/ 2023-05-31T21:03:22.585 DEBUG:boto:auth_path=/pklxkd-61/ 2023-05-31T21:03:22.585 DEBUG:boto:path=/pklxkd-61/?prefix=obj51 2023-05-31T21:03:22.585 DEBUG:boto:auth_path=/pklxkd-61/?prefix=obj51 2023-05-31T21:03:22.586 DEBUG:boto:Method: GET 2023-05-31T21:03:22.586 DEBUG:boto:Path: /pklxkd-61/?prefix=obj51 2023-05-31T21:03:22.586 DEBUG:boto:Data: 2023-05-31T21:03:22.587 DEBUG:boto:Headers: {} 2023-05-31T21:03:22.587 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T21:03:22.587 DEBUG:boto:Port: 8000 2023-05-31T21:03:22.588 DEBUG:boto:Params: {} 2023-05-31T21:03:22.588 DEBUG:boto:Token: None 2023-05-31T21:03:22.589 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 21:03:22 GMT /pklxkd-61/ 2023-05-31T21:03:22.589 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:4/489Jh79nPyUwTqEQ/UrGtMPCY= 2023-05-31T21:03:22.589 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 21:03:22 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:4/489Jh79nPyUwTqEQ/UrGtMPCY=', 'Content-Length': '0'} 2023-05-31T21:03:22.593 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000003505b85c40ff59ec-006477b61a-1630-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 21:03:22 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:03:22.594 DEBUG:boto:b'pklxkd-61obj511000falseobj512023-05-31T20:59:44.841Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T21:03:22.594 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj55 2023-05-31T21:03:22.595 DEBUG:boto:path=/pklxkd-61/obj55 2023-05-31T21:03:22.595 DEBUG:boto:auth_path=/pklxkd-61/obj55 2023-05-31T21:03:22.595 DEBUG:boto:path=/pklxkd-61/obj55?versionId=null 2023-05-31T21:03:22.596 DEBUG:boto:auth_path=/pklxkd-61/obj55?versionId=null 2023-05-31T21:03:22.596 DEBUG:boto:Method: GET 2023-05-31T21:03:22.597 DEBUG:boto:Path: /pklxkd-61/obj55?versionId=null 2023-05-31T21:03:22.597 DEBUG:boto:Data: 2023-05-31T21:03:22.597 DEBUG:boto:Headers: {} 2023-05-31T21:03:22.598 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T21:03:22.598 DEBUG:boto:Port: 8001 2023-05-31T21:03:22.598 DEBUG:boto:Params: {} 2023-05-31T21:03:22.599 DEBUG:boto:Token: None 2023-05-31T21:03:22.599 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 21:03:22 GMT /pklxkd-61/obj55?versionId=null 2023-05-31T21:03:22.600 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:rnlzmki8CZMkUeoeZrbvfvjISjI= 2023-05-31T21:03:22.600 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 21:03:22 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:rnlzmki8CZMkUeoeZrbvfvjISjI=', 'Content-Length': '0'} 2023-05-31T21:03:22.602 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:59:44 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000053f68acc6e4c2155-006477b61a-1f4e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 21:03:22 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:03:22.603 DEBUG:boto:path=/pklxkd-61/obj55 2023-05-31T21:03:22.603 DEBUG:boto:auth_path=/pklxkd-61/obj55 2023-05-31T21:03:22.603 DEBUG:boto:path=/pklxkd-61/obj55?versionId=null 2023-05-31T21:03:22.604 DEBUG:boto:auth_path=/pklxkd-61/obj55?versionId=null 2023-05-31T21:03:22.604 DEBUG:boto:Method: GET 2023-05-31T21:03:22.605 DEBUG:boto:Path: /pklxkd-61/obj55?versionId=null 2023-05-31T21:03:22.605 DEBUG:boto:Data: 2023-05-31T21:03:22.605 DEBUG:boto:Headers: {} 2023-05-31T21:03:22.606 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T21:03:22.606 DEBUG:boto:Port: 8000 2023-05-31T21:03:22.606 DEBUG:boto:Params: {} 2023-05-31T21:03:22.607 DEBUG:boto:Token: None 2023-05-31T21:03:22.607 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 21:03:22 GMT /pklxkd-61/obj55?versionId=null 2023-05-31T21:03:22.608 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:rnlzmki8CZMkUeoeZrbvfvjISjI= 2023-05-31T21:03:22.608 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 21:03:22 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:rnlzmki8CZMkUeoeZrbvfvjISjI=', 'Content-Length': '0'} 2023-05-31T21:03:22.610 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:59:44 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b376441c650d0b02-006477b61a-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 21:03:22 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:03:22.652 DEBUG:boto:path=/pklxkd-61/obj55 2023-05-31T21:03:22.653 DEBUG:boto:auth_path=/pklxkd-61/obj55 2023-05-31T21:03:22.654 DEBUG:boto:path=/pklxkd-61/obj55?versionId=null 2023-05-31T21:03:22.655 DEBUG:boto:auth_path=/pklxkd-61/obj55?versionId=null 2023-05-31T21:03:22.655 DEBUG:boto:Method: HEAD 2023-05-31T21:03:22.656 DEBUG:boto:Path: /pklxkd-61/obj55?versionId=null 2023-05-31T21:03:22.657 DEBUG:boto:Data: 2023-05-31T21:03:22.658 DEBUG:boto:Headers: {} 2023-05-31T21:03:22.658 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T21:03:22.658 DEBUG:boto:Port: 8001 2023-05-31T21:03:22.659 DEBUG:boto:Params: {} 2023-05-31T21:03:22.659 DEBUG:boto:Token: None 2023-05-31T21:03:22.660 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 21:03:22 GMT /pklxkd-61/obj55?versionId=null 2023-05-31T21:03:22.661 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:s7naes8CJXkEN0/kFBaUfw0V1h4= 2023-05-31T21:03:22.661 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 21:03:22 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:s7naes8CJXkEN0/kFBaUfw0V1h4=', 'Content-Length': '0'} 2023-05-31T21:03:22.663 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:59:44 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003314a702556f45ea-006477b61a-1f4e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 21:03:22 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:03:22.664 DEBUG:boto:path=/pklxkd-61/obj55 2023-05-31T21:03:22.664 DEBUG:boto:auth_path=/pklxkd-61/obj55 2023-05-31T21:03:22.664 DEBUG:boto:path=/pklxkd-61/obj55?versionId=null 2023-05-31T21:03:22.665 DEBUG:boto:auth_path=/pklxkd-61/obj55?versionId=null 2023-05-31T21:03:22.665 DEBUG:boto:Method: HEAD 2023-05-31T21:03:22.666 DEBUG:boto:Path: /pklxkd-61/obj55?versionId=null 2023-05-31T21:03:22.667 DEBUG:boto:Data: 2023-05-31T21:03:22.667 DEBUG:boto:Headers: {} 2023-05-31T21:03:22.668 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T21:03:22.668 DEBUG:boto:Port: 8000 2023-05-31T21:03:22.669 DEBUG:boto:Params: {} 2023-05-31T21:03:22.669 DEBUG:boto:Token: None 2023-05-31T21:03:22.669 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 21:03:22 GMT /pklxkd-61/obj55?versionId=null 2023-05-31T21:03:22.670 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:s7naes8CJXkEN0/kFBaUfw0V1h4= 2023-05-31T21:03:22.670 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 21:03:22 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:s7naes8CJXkEN0/kFBaUfw0V1h4=', 'Content-Length': '0'} 2023-05-31T21:03:22.672 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:59:44 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d73fce3edeff487d-006477b61a-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 21:03:22 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:03:22.674 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj55 2023-05-31T21:03:22.674 DEBUG:boto:path=/pklxkd-61/obj55 2023-05-31T21:03:22.675 DEBUG:boto:auth_path=/pklxkd-61/obj55 2023-05-31T21:03:22.676 DEBUG:boto:path=/pklxkd-61/obj55?versionId=null 2023-05-31T21:03:22.677 DEBUG:boto:auth_path=/pklxkd-61/obj55?versionId=null 2023-05-31T21:03:22.677 DEBUG:boto:Method: GET 2023-05-31T21:03:22.677 DEBUG:boto:Path: /pklxkd-61/obj55?versionId=null 2023-05-31T21:03:22.678 DEBUG:boto:Data: 2023-05-31T21:03:22.678 DEBUG:boto:Headers: {} 2023-05-31T21:03:22.679 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T21:03:22.679 DEBUG:boto:Port: 8001 2023-05-31T21:03:22.679 DEBUG:boto:Params: {} 2023-05-31T21:03:22.680 DEBUG:boto:Token: None 2023-05-31T21:03:22.680 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 21:03:22 GMT /pklxkd-61/obj55?versionId=null 2023-05-31T21:03:22.681 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:rnlzmki8CZMkUeoeZrbvfvjISjI= 2023-05-31T21:03:22.681 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 21:03:22 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:rnlzmki8CZMkUeoeZrbvfvjISjI=', 'Content-Length': '0'} 2023-05-31T21:03:22.683 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:59:44 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002adc2a098d27b281-006477b61a-1f4e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 21:03:22 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:03:22.684 DEBUG:boto:path=/pklxkd-61/obj55 2023-05-31T21:03:22.684 DEBUG:boto:auth_path=/pklxkd-61/obj55 2023-05-31T21:03:22.685 DEBUG:boto:path=/pklxkd-61/obj55?versionId=null 2023-05-31T21:03:22.685 DEBUG:boto:auth_path=/pklxkd-61/obj55?versionId=null 2023-05-31T21:03:22.685 DEBUG:boto:Method: GET 2023-05-31T21:03:22.686 DEBUG:boto:Path: /pklxkd-61/obj55?versionId=null 2023-05-31T21:03:22.686 DEBUG:boto:Data: 2023-05-31T21:03:22.686 DEBUG:boto:Headers: {} 2023-05-31T21:03:22.687 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T21:03:22.687 DEBUG:boto:Port: 8000 2023-05-31T21:03:22.687 DEBUG:boto:Params: {} 2023-05-31T21:03:22.688 DEBUG:boto:Token: None 2023-05-31T21:03:22.688 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 21:03:22 GMT /pklxkd-61/obj55?versionId=null 2023-05-31T21:03:22.689 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:rnlzmki8CZMkUeoeZrbvfvjISjI= 2023-05-31T21:03:22.689 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 21:03:22 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:rnlzmki8CZMkUeoeZrbvfvjISjI=', 'Content-Length': '0'} 2023-05-31T21:03:22.691 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:59:44 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000616e442396bf88f1-006477b61a-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 21:03:22 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:03:22.691 DEBUG:boto:path=/pklxkd-61/ 2023-05-31T21:03:22.692 DEBUG:boto:auth_path=/pklxkd-61/ 2023-05-31T21:03:22.692 DEBUG:boto:path=/pklxkd-61/?prefix=obj55 2023-05-31T21:03:22.693 DEBUG:boto:auth_path=/pklxkd-61/?prefix=obj55 2023-05-31T21:03:22.693 DEBUG:boto:Method: GET 2023-05-31T21:03:22.694 DEBUG:boto:Path: /pklxkd-61/?prefix=obj55 2023-05-31T21:03:22.694 DEBUG:boto:Data: 2023-05-31T21:03:22.695 DEBUG:boto:Headers: {} 2023-05-31T21:03:22.695 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T21:03:22.695 DEBUG:boto:Port: 8001 2023-05-31T21:03:22.696 DEBUG:boto:Params: {} 2023-05-31T21:03:22.696 DEBUG:boto:Token: None 2023-05-31T21:03:22.697 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 21:03:22 GMT /pklxkd-61/ 2023-05-31T21:03:22.697 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:4/489Jh79nPyUwTqEQ/UrGtMPCY= 2023-05-31T21:03:22.698 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 21:03:22 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:4/489Jh79nPyUwTqEQ/UrGtMPCY=', 'Content-Length': '0'} 2023-05-31T21:03:22.700 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000000cbf36f6086613d5-006477b61a-1f4e-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 21:03:22 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:03:22.701 DEBUG:boto:b'pklxkd-61obj551000falseobj552023-05-31T20:59:44.541Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T21:03:22.702 DEBUG:boto:path=/pklxkd-61/ 2023-05-31T21:03:22.702 DEBUG:boto:auth_path=/pklxkd-61/ 2023-05-31T21:03:22.702 DEBUG:boto:path=/pklxkd-61/?prefix=obj55 2023-05-31T21:03:22.703 DEBUG:boto:auth_path=/pklxkd-61/?prefix=obj55 2023-05-31T21:03:22.703 DEBUG:boto:Method: GET 2023-05-31T21:03:22.704 DEBUG:boto:Path: /pklxkd-61/?prefix=obj55 2023-05-31T21:03:22.704 DEBUG:boto:Data: 2023-05-31T21:03:22.704 DEBUG:boto:Headers: {} 2023-05-31T21:03:22.705 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T21:03:22.705 DEBUG:boto:Port: 8000 2023-05-31T21:03:22.706 DEBUG:boto:Params: {} 2023-05-31T21:03:22.706 DEBUG:boto:Token: None 2023-05-31T21:03:22.706 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 21:03:22 GMT /pklxkd-61/ 2023-05-31T21:03:22.707 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:4/489Jh79nPyUwTqEQ/UrGtMPCY= 2023-05-31T21:03:22.707 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 21:03:22 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:4/489Jh79nPyUwTqEQ/UrGtMPCY=', 'Content-Length': '0'} 2023-05-31T21:03:22.711 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c69c687772aeb541-006477b61a-1630-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 21:03:22 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:03:22.712 DEBUG:boto:b'pklxkd-61obj551000falseobj552023-05-31T20:59:44.541Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T21:03:22.712 DEBUG:boto:path=/pklxkd-61/obj55 2023-05-31T21:03:22.713 DEBUG:boto:auth_path=/pklxkd-61/obj55 2023-05-31T21:03:22.713 DEBUG:boto:Method: HEAD 2023-05-31T21:03:22.713 DEBUG:boto:Path: /pklxkd-61/obj55 2023-05-31T21:03:22.714 DEBUG:boto:Data: 2023-05-31T21:03:22.714 DEBUG:boto:Headers: {} 2023-05-31T21:03:22.714 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T21:03:22.715 DEBUG:boto:Port: 8001 2023-05-31T21:03:22.715 DEBUG:boto:Params: {} 2023-05-31T21:03:22.715 DEBUG:boto:Token: None 2023-05-31T21:03:22.716 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 21:03:22 GMT /pklxkd-61/obj55 2023-05-31T21:03:22.716 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:p4LF4ByTb+0ME04b5o+tk/qYazc= 2023-05-31T21:03:22.717 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 21:03:22 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:p4LF4ByTb+0ME04b5o+tk/qYazc=', 'Content-Length': '0'} 2023-05-31T21:03:22.718 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:59:44 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000007e4db1c6447357c-006477b61a-1f4e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 21:03:22 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:03:22.719 DEBUG:boto:path=/pklxkd-61/obj55 2023-05-31T21:03:22.719 DEBUG:boto:auth_path=/pklxkd-61/obj55 2023-05-31T21:03:22.720 DEBUG:boto:Method: HEAD 2023-05-31T21:03:22.720 DEBUG:boto:Path: /pklxkd-61/obj55 2023-05-31T21:03:22.721 DEBUG:boto:Data: 2023-05-31T21:03:22.721 DEBUG:boto:Headers: {} 2023-05-31T21:03:22.721 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T21:03:22.722 DEBUG:boto:Port: 8000 2023-05-31T21:03:22.722 DEBUG:boto:Params: {} 2023-05-31T21:03:22.722 DEBUG:boto:Token: None 2023-05-31T21:03:22.723 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 21:03:22 GMT /pklxkd-61/obj55 2023-05-31T21:03:22.723 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:p4LF4ByTb+0ME04b5o+tk/qYazc= 2023-05-31T21:03:22.724 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 21:03:22 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:p4LF4ByTb+0ME04b5o+tk/qYazc=', 'Content-Length': '0'} 2023-05-31T21:03:22.726 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:59:44 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008a6ec820b470d46e-006477b61a-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 21:03:22 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:03:22.726 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj55 2023-05-31T21:03:22.727 DEBUG:boto:path=/pklxkd-61/obj55 2023-05-31T21:03:22.727 DEBUG:boto:auth_path=/pklxkd-61/obj55 2023-05-31T21:03:22.727 DEBUG:boto:Method: GET 2023-05-31T21:03:22.728 DEBUG:boto:Path: /pklxkd-61/obj55 2023-05-31T21:03:22.728 DEBUG:boto:Data: 2023-05-31T21:03:22.729 DEBUG:boto:Headers: {} 2023-05-31T21:03:22.729 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T21:03:22.729 DEBUG:boto:Port: 8001 2023-05-31T21:03:22.730 DEBUG:boto:Params: {} 2023-05-31T21:03:22.730 DEBUG:boto:Token: None 2023-05-31T21:03:22.730 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 21:03:22 GMT /pklxkd-61/obj55 2023-05-31T21:03:22.731 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:zh13NGOJCObUdOAPn649ha20bF8= 2023-05-31T21:03:22.731 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 21:03:22 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:zh13NGOJCObUdOAPn649ha20bF8=', 'Content-Length': '0'} 2023-05-31T21:03:22.733 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:59:44 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000000ab4f37db0e8ee-006477b61a-1f4e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 21:03:22 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:03:22.734 DEBUG:boto:path=/pklxkd-61/obj55 2023-05-31T21:03:22.734 DEBUG:boto:auth_path=/pklxkd-61/obj55 2023-05-31T21:03:22.735 DEBUG:boto:Method: GET 2023-05-31T21:03:22.735 DEBUG:boto:Path: /pklxkd-61/obj55 2023-05-31T21:03:22.736 DEBUG:boto:Data: 2023-05-31T21:03:22.736 DEBUG:boto:Headers: {} 2023-05-31T21:03:22.736 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T21:03:22.737 DEBUG:boto:Port: 8000 2023-05-31T21:03:22.737 DEBUG:boto:Params: {} 2023-05-31T21:03:22.738 DEBUG:boto:Token: None 2023-05-31T21:03:22.738 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 21:03:22 GMT /pklxkd-61/obj55 2023-05-31T21:03:22.738 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:zh13NGOJCObUdOAPn649ha20bF8= 2023-05-31T21:03:22.739 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 21:03:22 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:zh13NGOJCObUdOAPn649ha20bF8=', 'Content-Length': '0'} 2023-05-31T21:03:22.741 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:59:44 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000053ebc051e590ea61-006477b61a-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 21:03:22 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:03:22.784 DEBUG:boto:path=/pklxkd-61/ 2023-05-31T21:03:22.785 DEBUG:boto:auth_path=/pklxkd-61/ 2023-05-31T21:03:22.785 DEBUG:boto:path=/pklxkd-61/?prefix=obj55 2023-05-31T21:03:22.785 DEBUG:boto:auth_path=/pklxkd-61/?prefix=obj55 2023-05-31T21:03:22.786 DEBUG:boto:Method: GET 2023-05-31T21:03:22.786 DEBUG:boto:Path: /pklxkd-61/?prefix=obj55 2023-05-31T21:03:22.787 DEBUG:boto:Data: 2023-05-31T21:03:22.787 DEBUG:boto:Headers: {} 2023-05-31T21:03:22.787 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T21:03:22.788 DEBUG:boto:Port: 8001 2023-05-31T21:03:22.788 DEBUG:boto:Params: {} 2023-05-31T21:03:22.788 DEBUG:boto:Token: None 2023-05-31T21:03:22.789 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 21:03:22 GMT /pklxkd-61/ 2023-05-31T21:03:22.789 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:4/489Jh79nPyUwTqEQ/UrGtMPCY= 2023-05-31T21:03:22.790 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 21:03:22 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:4/489Jh79nPyUwTqEQ/UrGtMPCY=', 'Content-Length': '0'} 2023-05-31T21:03:22.792 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000002f6552808c824e8e-006477b61a-1f4e-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 21:03:22 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:03:22.793 DEBUG:boto:b'pklxkd-61obj551000falseobj552023-05-31T20:59:44.541Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T21:03:22.793 DEBUG:boto:path=/pklxkd-61/ 2023-05-31T21:03:22.794 DEBUG:boto:auth_path=/pklxkd-61/ 2023-05-31T21:03:22.794 DEBUG:boto:path=/pklxkd-61/?prefix=obj55 2023-05-31T21:03:22.794 DEBUG:boto:auth_path=/pklxkd-61/?prefix=obj55 2023-05-31T21:03:22.795 DEBUG:boto:Method: GET 2023-05-31T21:03:22.795 DEBUG:boto:Path: /pklxkd-61/?prefix=obj55 2023-05-31T21:03:22.795 DEBUG:boto:Data: 2023-05-31T21:03:22.796 DEBUG:boto:Headers: {} 2023-05-31T21:03:22.796 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T21:03:22.797 DEBUG:boto:Port: 8000 2023-05-31T21:03:22.797 DEBUG:boto:Params: {} 2023-05-31T21:03:22.797 DEBUG:boto:Token: None 2023-05-31T21:03:22.798 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 21:03:22 GMT /pklxkd-61/ 2023-05-31T21:03:22.798 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:4/489Jh79nPyUwTqEQ/UrGtMPCY= 2023-05-31T21:03:22.799 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 21:03:22 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:4/489Jh79nPyUwTqEQ/UrGtMPCY=', 'Content-Length': '0'} 2023-05-31T21:03:22.802 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000100d1ffddb2a8b89-006477b61a-1630-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 21:03:22 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:03:22.803 DEBUG:boto:b'pklxkd-61obj551000falseobj552023-05-31T20:59:44.541Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T21:03:22.803 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj57 2023-05-31T21:03:22.804 DEBUG:boto:path=/pklxkd-61/obj57 2023-05-31T21:03:22.804 DEBUG:boto:auth_path=/pklxkd-61/obj57 2023-05-31T21:03:22.805 DEBUG:boto:path=/pklxkd-61/obj57?versionId=null 2023-05-31T21:03:22.805 DEBUG:boto:auth_path=/pklxkd-61/obj57?versionId=null 2023-05-31T21:03:22.805 DEBUG:boto:Method: GET 2023-05-31T21:03:22.806 DEBUG:boto:Path: /pklxkd-61/obj57?versionId=null 2023-05-31T21:03:22.806 DEBUG:boto:Data: 2023-05-31T21:03:22.807 DEBUG:boto:Headers: {} 2023-05-31T21:03:22.807 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T21:03:22.807 DEBUG:boto:Port: 8001 2023-05-31T21:03:22.808 DEBUG:boto:Params: {} 2023-05-31T21:03:22.808 DEBUG:boto:Token: None 2023-05-31T21:03:22.809 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 21:03:22 GMT /pklxkd-61/obj57?versionId=null 2023-05-31T21:03:22.809 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:O8W+r95+jxqpLeyNhWdFQtjIdo0= 2023-05-31T21:03:22.810 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 21:03:22 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:O8W+r95+jxqpLeyNhWdFQtjIdo0=', 'Content-Length': '0'} 2023-05-31T21:03:22.811 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:59:45 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000056af0a09a0bd22c1-006477b61a-1f4e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 21:03:22 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:03:22.852 DEBUG:boto:path=/pklxkd-61/obj57 2023-05-31T21:03:22.853 DEBUG:boto:auth_path=/pklxkd-61/obj57 2023-05-31T21:03:22.853 DEBUG:boto:path=/pklxkd-61/obj57?versionId=null 2023-05-31T21:03:22.853 DEBUG:boto:auth_path=/pklxkd-61/obj57?versionId=null 2023-05-31T21:03:22.854 DEBUG:boto:Method: GET 2023-05-31T21:03:22.854 DEBUG:boto:Path: /pklxkd-61/obj57?versionId=null 2023-05-31T21:03:22.854 DEBUG:boto:Data: 2023-05-31T21:03:22.855 DEBUG:boto:Headers: {} 2023-05-31T21:03:22.855 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T21:03:22.855 DEBUG:boto:Port: 8000 2023-05-31T21:03:22.856 DEBUG:boto:Params: {} 2023-05-31T21:03:22.856 DEBUG:boto:Token: None 2023-05-31T21:03:22.857 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 21:03:22 GMT /pklxkd-61/obj57?versionId=null 2023-05-31T21:03:22.857 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:O8W+r95+jxqpLeyNhWdFQtjIdo0= 2023-05-31T21:03:22.858 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 21:03:22 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:O8W+r95+jxqpLeyNhWdFQtjIdo0=', 'Content-Length': '0'} 2023-05-31T21:03:22.859 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:59:45 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003db9a529572ebd1b-006477b61a-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 21:03:22 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:03:22.860 DEBUG:boto:path=/pklxkd-61/obj57 2023-05-31T21:03:22.860 DEBUG:boto:auth_path=/pklxkd-61/obj57 2023-05-31T21:03:22.861 DEBUG:boto:path=/pklxkd-61/obj57?versionId=null 2023-05-31T21:03:22.861 DEBUG:boto:auth_path=/pklxkd-61/obj57?versionId=null 2023-05-31T21:03:22.861 DEBUG:boto:Method: HEAD 2023-05-31T21:03:22.862 DEBUG:boto:Path: /pklxkd-61/obj57?versionId=null 2023-05-31T21:03:22.862 DEBUG:boto:Data: 2023-05-31T21:03:22.862 DEBUG:boto:Headers: {} 2023-05-31T21:03:22.863 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T21:03:22.863 DEBUG:boto:Port: 8001 2023-05-31T21:03:22.863 DEBUG:boto:Params: {} 2023-05-31T21:03:22.864 DEBUG:boto:Token: None 2023-05-31T21:03:22.864 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 21:03:22 GMT /pklxkd-61/obj57?versionId=null 2023-05-31T21:03:22.864 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:PBRp/buixxXNT9r63AsLLA42pQA= 2023-05-31T21:03:22.865 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 21:03:22 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:PBRp/buixxXNT9r63AsLLA42pQA=', 'Content-Length': '0'} 2023-05-31T21:03:22.866 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:59:45 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000067e07d16e6e8a071-006477b61a-1f4e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 21:03:22 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:03:22.867 DEBUG:boto:path=/pklxkd-61/obj57 2023-05-31T21:03:22.867 DEBUG:boto:auth_path=/pklxkd-61/obj57 2023-05-31T21:03:22.868 DEBUG:boto:path=/pklxkd-61/obj57?versionId=null 2023-05-31T21:03:22.868 DEBUG:boto:auth_path=/pklxkd-61/obj57?versionId=null 2023-05-31T21:03:22.868 DEBUG:boto:Method: HEAD 2023-05-31T21:03:22.869 DEBUG:boto:Path: /pklxkd-61/obj57?versionId=null 2023-05-31T21:03:22.869 DEBUG:boto:Data: 2023-05-31T21:03:22.869 DEBUG:boto:Headers: {} 2023-05-31T21:03:22.870 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T21:03:22.870 DEBUG:boto:Port: 8000 2023-05-31T21:03:22.870 DEBUG:boto:Params: {} 2023-05-31T21:03:22.871 DEBUG:boto:Token: None 2023-05-31T21:03:22.874 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 21:03:22 GMT /pklxkd-61/obj57?versionId=null 2023-05-31T21:03:22.875 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:PBRp/buixxXNT9r63AsLLA42pQA= 2023-05-31T21:03:22.875 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 21:03:22 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:PBRp/buixxXNT9r63AsLLA42pQA=', 'Content-Length': '0'} 2023-05-31T21:03:22.877 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:59:45 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000072566e87c78d48d6-006477b61a-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 21:03:22 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:03:22.878 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj57 2023-05-31T21:03:22.878 DEBUG:boto:path=/pklxkd-61/obj57 2023-05-31T21:03:22.879 DEBUG:boto:auth_path=/pklxkd-61/obj57 2023-05-31T21:03:22.879 DEBUG:boto:path=/pklxkd-61/obj57?versionId=null 2023-05-31T21:03:22.880 DEBUG:boto:auth_path=/pklxkd-61/obj57?versionId=null 2023-05-31T21:03:22.880 DEBUG:boto:Method: GET 2023-05-31T21:03:22.880 DEBUG:boto:Path: /pklxkd-61/obj57?versionId=null 2023-05-31T21:03:22.881 DEBUG:boto:Data: 2023-05-31T21:03:22.881 DEBUG:boto:Headers: {} 2023-05-31T21:03:22.881 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T21:03:22.882 DEBUG:boto:Port: 8001 2023-05-31T21:03:22.882 DEBUG:boto:Params: {} 2023-05-31T21:03:22.883 DEBUG:boto:Token: None 2023-05-31T21:03:22.883 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 21:03:22 GMT /pklxkd-61/obj57?versionId=null 2023-05-31T21:03:22.883 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:O8W+r95+jxqpLeyNhWdFQtjIdo0= 2023-05-31T21:03:22.884 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 21:03:22 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:O8W+r95+jxqpLeyNhWdFQtjIdo0=', 'Content-Length': '0'} 2023-05-31T21:03:22.886 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:59:45 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009f2b51372d174f5a-006477b61a-1f4e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 21:03:22 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:03:22.887 DEBUG:boto:path=/pklxkd-61/obj57 2023-05-31T21:03:22.887 DEBUG:boto:auth_path=/pklxkd-61/obj57 2023-05-31T21:03:22.887 DEBUG:boto:path=/pklxkd-61/obj57?versionId=null 2023-05-31T21:03:22.888 DEBUG:boto:auth_path=/pklxkd-61/obj57?versionId=null 2023-05-31T21:03:22.888 DEBUG:boto:Method: GET 2023-05-31T21:03:22.888 DEBUG:boto:Path: /pklxkd-61/obj57?versionId=null 2023-05-31T21:03:22.889 DEBUG:boto:Data: 2023-05-31T21:03:22.889 DEBUG:boto:Headers: {} 2023-05-31T21:03:22.890 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T21:03:22.890 DEBUG:boto:Port: 8000 2023-05-31T21:03:22.890 DEBUG:boto:Params: {} 2023-05-31T21:03:22.891 DEBUG:boto:Token: None 2023-05-31T21:03:22.891 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 21:03:22 GMT /pklxkd-61/obj57?versionId=null 2023-05-31T21:03:22.892 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:O8W+r95+jxqpLeyNhWdFQtjIdo0= 2023-05-31T21:03:22.892 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 21:03:22 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:O8W+r95+jxqpLeyNhWdFQtjIdo0=', 'Content-Length': '0'} 2023-05-31T21:03:22.894 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:59:45 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ba069689a4f2ede5-006477b61a-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 21:03:22 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:03:22.895 DEBUG:boto:path=/pklxkd-61/ 2023-05-31T21:03:22.895 DEBUG:boto:auth_path=/pklxkd-61/ 2023-05-31T21:03:22.895 DEBUG:boto:path=/pklxkd-61/?prefix=obj57 2023-05-31T21:03:22.896 DEBUG:boto:auth_path=/pklxkd-61/?prefix=obj57 2023-05-31T21:03:22.896 DEBUG:boto:Method: GET 2023-05-31T21:03:22.897 DEBUG:boto:Path: /pklxkd-61/?prefix=obj57 2023-05-31T21:03:22.897 DEBUG:boto:Data: 2023-05-31T21:03:22.897 DEBUG:boto:Headers: {} 2023-05-31T21:03:22.898 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T21:03:22.898 DEBUG:boto:Port: 8001 2023-05-31T21:03:22.898 DEBUG:boto:Params: {} 2023-05-31T21:03:22.899 DEBUG:boto:Token: None 2023-05-31T21:03:22.899 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 21:03:22 GMT /pklxkd-61/ 2023-05-31T21:03:22.900 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:4/489Jh79nPyUwTqEQ/UrGtMPCY= 2023-05-31T21:03:22.900 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 21:03:22 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:4/489Jh79nPyUwTqEQ/UrGtMPCY=', 'Content-Length': '0'} 2023-05-31T21:03:22.902 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000002a442eec1bb902de-006477b61a-1f4e-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 21:03:22 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:03:22.903 DEBUG:boto:b'pklxkd-61obj571000falseobj572023-05-31T20:59:45.197Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T21:03:22.904 DEBUG:boto:path=/pklxkd-61/ 2023-05-31T21:03:22.904 DEBUG:boto:auth_path=/pklxkd-61/ 2023-05-31T21:03:22.905 DEBUG:boto:path=/pklxkd-61/?prefix=obj57 2023-05-31T21:03:22.905 DEBUG:boto:auth_path=/pklxkd-61/?prefix=obj57 2023-05-31T21:03:22.905 DEBUG:boto:Method: GET 2023-05-31T21:03:22.906 DEBUG:boto:Path: /pklxkd-61/?prefix=obj57 2023-05-31T21:03:22.906 DEBUG:boto:Data: 2023-05-31T21:03:22.906 DEBUG:boto:Headers: {} 2023-05-31T21:03:22.907 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T21:03:22.907 DEBUG:boto:Port: 8000 2023-05-31T21:03:22.907 DEBUG:boto:Params: {} 2023-05-31T21:03:22.908 DEBUG:boto:Token: None 2023-05-31T21:03:22.908 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 21:03:22 GMT /pklxkd-61/ 2023-05-31T21:03:22.909 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:4/489Jh79nPyUwTqEQ/UrGtMPCY= 2023-05-31T21:03:22.909 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 21:03:22 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:4/489Jh79nPyUwTqEQ/UrGtMPCY=', 'Content-Length': '0'} 2023-05-31T21:03:22.913 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000007042271fcd266aa4-006477b61a-1630-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 21:03:22 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:03:22.913 DEBUG:boto:b'pklxkd-61obj571000falseobj572023-05-31T20:59:45.197Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T21:03:22.914 DEBUG:boto:path=/pklxkd-61/obj57 2023-05-31T21:03:22.914 DEBUG:boto:auth_path=/pklxkd-61/obj57 2023-05-31T21:03:22.915 DEBUG:boto:Method: HEAD 2023-05-31T21:03:22.915 DEBUG:boto:Path: /pklxkd-61/obj57 2023-05-31T21:03:22.916 DEBUG:boto:Data: 2023-05-31T21:03:22.916 DEBUG:boto:Headers: {} 2023-05-31T21:03:22.916 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T21:03:22.917 DEBUG:boto:Port: 8001 2023-05-31T21:03:22.917 DEBUG:boto:Params: {} 2023-05-31T21:03:22.917 DEBUG:boto:Token: None 2023-05-31T21:03:22.918 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 21:03:22 GMT /pklxkd-61/obj57 2023-05-31T21:03:22.918 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:k7cPsfIMNc5Y3BGBdL68FqYNwEQ= 2023-05-31T21:03:22.919 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 21:03:22 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:k7cPsfIMNc5Y3BGBdL68FqYNwEQ=', 'Content-Length': '0'} 2023-05-31T21:03:22.920 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:59:45 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f00c977055262062-006477b61a-1f4e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 21:03:22 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:03:22.921 DEBUG:boto:path=/pklxkd-61/obj57 2023-05-31T21:03:22.921 DEBUG:boto:auth_path=/pklxkd-61/obj57 2023-05-31T21:03:22.922 DEBUG:boto:Method: HEAD 2023-05-31T21:03:22.922 DEBUG:boto:Path: /pklxkd-61/obj57 2023-05-31T21:03:22.922 DEBUG:boto:Data: 2023-05-31T21:03:22.923 DEBUG:boto:Headers: {} 2023-05-31T21:03:22.923 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T21:03:22.923 DEBUG:boto:Port: 8000 2023-05-31T21:03:22.924 DEBUG:boto:Params: {} 2023-05-31T21:03:22.924 DEBUG:boto:Token: None 2023-05-31T21:03:22.924 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 21:03:22 GMT /pklxkd-61/obj57 2023-05-31T21:03:22.925 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:k7cPsfIMNc5Y3BGBdL68FqYNwEQ= 2023-05-31T21:03:22.925 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 21:03:22 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:k7cPsfIMNc5Y3BGBdL68FqYNwEQ=', 'Content-Length': '0'} 2023-05-31T21:03:22.927 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:59:45 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a6b1fd17c3f8e777-006477b61a-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 21:03:22 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:03:22.928 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj57 2023-05-31T21:03:22.928 DEBUG:boto:path=/pklxkd-61/obj57 2023-05-31T21:03:22.928 DEBUG:boto:auth_path=/pklxkd-61/obj57 2023-05-31T21:03:22.929 DEBUG:boto:Method: GET 2023-05-31T21:03:22.929 DEBUG:boto:Path: /pklxkd-61/obj57 2023-05-31T21:03:22.930 DEBUG:boto:Data: 2023-05-31T21:03:22.930 DEBUG:boto:Headers: {} 2023-05-31T21:03:22.930 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T21:03:22.931 DEBUG:boto:Port: 8001 2023-05-31T21:03:22.931 DEBUG:boto:Params: {} 2023-05-31T21:03:22.931 DEBUG:boto:Token: None 2023-05-31T21:03:22.932 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 21:03:22 GMT /pklxkd-61/obj57 2023-05-31T21:03:22.932 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:4Lc87BE4phuuZI+I7Nlxf8+QWWI= 2023-05-31T21:03:22.933 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 21:03:22 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:4Lc87BE4phuuZI+I7Nlxf8+QWWI=', 'Content-Length': '0'} 2023-05-31T21:03:22.934 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:59:45 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a1eecad580d49581-006477b61a-1f4e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 21:03:22 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:03:22.935 DEBUG:boto:path=/pklxkd-61/obj57 2023-05-31T21:03:22.936 DEBUG:boto:auth_path=/pklxkd-61/obj57 2023-05-31T21:03:22.936 DEBUG:boto:Method: GET 2023-05-31T21:03:22.936 DEBUG:boto:Path: /pklxkd-61/obj57 2023-05-31T21:03:22.937 DEBUG:boto:Data: 2023-05-31T21:03:22.937 DEBUG:boto:Headers: {} 2023-05-31T21:03:22.937 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T21:03:22.938 DEBUG:boto:Port: 8000 2023-05-31T21:03:22.938 DEBUG:boto:Params: {} 2023-05-31T21:03:22.939 DEBUG:boto:Token: None 2023-05-31T21:03:22.939 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 21:03:22 GMT /pklxkd-61/obj57 2023-05-31T21:03:22.939 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:4Lc87BE4phuuZI+I7Nlxf8+QWWI= 2023-05-31T21:03:22.940 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 21:03:22 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:4Lc87BE4phuuZI+I7Nlxf8+QWWI=', 'Content-Length': '0'} 2023-05-31T21:03:22.942 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:59:45 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000787dd96b3aa5a60-006477b61a-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 21:03:22 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:03:22.984 DEBUG:boto:path=/pklxkd-61/ 2023-05-31T21:03:22.985 DEBUG:boto:auth_path=/pklxkd-61/ 2023-05-31T21:03:22.985 DEBUG:boto:path=/pklxkd-61/?prefix=obj57 2023-05-31T21:03:22.985 DEBUG:boto:auth_path=/pklxkd-61/?prefix=obj57 2023-05-31T21:03:22.986 DEBUG:boto:Method: GET 2023-05-31T21:03:22.986 DEBUG:boto:Path: /pklxkd-61/?prefix=obj57 2023-05-31T21:03:22.986 DEBUG:boto:Data: 2023-05-31T21:03:22.987 DEBUG:boto:Headers: {} 2023-05-31T21:03:22.987 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T21:03:22.988 DEBUG:boto:Port: 8001 2023-05-31T21:03:22.988 DEBUG:boto:Params: {} 2023-05-31T21:03:22.988 DEBUG:boto:Token: None 2023-05-31T21:03:22.989 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 21:03:22 GMT /pklxkd-61/ 2023-05-31T21:03:22.989 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:4/489Jh79nPyUwTqEQ/UrGtMPCY= 2023-05-31T21:03:22.990 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 21:03:22 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:4/489Jh79nPyUwTqEQ/UrGtMPCY=', 'Content-Length': '0'} 2023-05-31T21:03:22.992 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ddfd37703e940c36-006477b61a-1f4e-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 21:03:22 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:03:22.993 DEBUG:boto:b'pklxkd-61obj571000falseobj572023-05-31T20:59:45.197Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T21:03:22.993 DEBUG:boto:path=/pklxkd-61/ 2023-05-31T21:03:22.994 DEBUG:boto:auth_path=/pklxkd-61/ 2023-05-31T21:03:22.994 DEBUG:boto:path=/pklxkd-61/?prefix=obj57 2023-05-31T21:03:22.995 DEBUG:boto:auth_path=/pklxkd-61/?prefix=obj57 2023-05-31T21:03:22.995 DEBUG:boto:Method: GET 2023-05-31T21:03:22.995 DEBUG:boto:Path: /pklxkd-61/?prefix=obj57 2023-05-31T21:03:22.996 DEBUG:boto:Data: 2023-05-31T21:03:22.996 DEBUG:boto:Headers: {} 2023-05-31T21:03:22.996 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T21:03:22.997 DEBUG:boto:Port: 8000 2023-05-31T21:03:22.997 DEBUG:boto:Params: {} 2023-05-31T21:03:22.998 DEBUG:boto:Token: None 2023-05-31T21:03:22.998 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 21:03:22 GMT /pklxkd-61/ 2023-05-31T21:03:22.998 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:4/489Jh79nPyUwTqEQ/UrGtMPCY= 2023-05-31T21:03:22.999 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 21:03:22 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:4/489Jh79nPyUwTqEQ/UrGtMPCY=', 'Content-Length': '0'} 2023-05-31T21:03:23.003 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d3fd1083c69dbc17-006477b61a-1630-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 21:03:22 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:03:23.003 DEBUG:boto:b'pklxkd-61obj571000falseobj572023-05-31T20:59:45.197Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T21:03:23.004 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj58 2023-05-31T21:03:23.004 DEBUG:boto:path=/pklxkd-61/obj58 2023-05-31T21:03:23.005 DEBUG:boto:auth_path=/pklxkd-61/obj58 2023-05-31T21:03:23.005 DEBUG:boto:path=/pklxkd-61/obj58?versionId=null 2023-05-31T21:03:23.006 DEBUG:boto:auth_path=/pklxkd-61/obj58?versionId=null 2023-05-31T21:03:23.006 DEBUG:boto:Method: GET 2023-05-31T21:03:23.006 DEBUG:boto:Path: /pklxkd-61/obj58?versionId=null 2023-05-31T21:03:23.007 DEBUG:boto:Data: 2023-05-31T21:03:23.007 DEBUG:boto:Headers: {} 2023-05-31T21:03:23.007 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T21:03:23.008 DEBUG:boto:Port: 8001 2023-05-31T21:03:23.008 DEBUG:boto:Params: {} 2023-05-31T21:03:23.009 DEBUG:boto:Token: None 2023-05-31T21:03:23.009 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 21:03:23 GMT /pklxkd-61/obj58?versionId=null 2023-05-31T21:03:23.010 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:Azcg0SZ9xp6VOhak93Dxf5WBtus= 2023-05-31T21:03:23.010 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 21:03:23 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:Azcg0SZ9xp6VOhak93Dxf5WBtus=', 'Content-Length': '0'} 2023-05-31T21:03:23.012 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:59:45 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000747fc0c38c9382c0-006477b61b-1f4e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 21:03:23 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:03:23.052 DEBUG:boto:path=/pklxkd-61/obj58 2023-05-31T21:03:23.053 DEBUG:boto:auth_path=/pklxkd-61/obj58 2023-05-31T21:03:23.053 DEBUG:boto:path=/pklxkd-61/obj58?versionId=null 2023-05-31T21:03:23.053 DEBUG:boto:auth_path=/pklxkd-61/obj58?versionId=null 2023-05-31T21:03:23.054 DEBUG:boto:Method: GET 2023-05-31T21:03:23.054 DEBUG:boto:Path: /pklxkd-61/obj58?versionId=null 2023-05-31T21:03:23.055 DEBUG:boto:Data: 2023-05-31T21:03:23.055 DEBUG:boto:Headers: {} 2023-05-31T21:03:23.055 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T21:03:23.056 DEBUG:boto:Port: 8000 2023-05-31T21:03:23.056 DEBUG:boto:Params: {} 2023-05-31T21:03:23.057 DEBUG:boto:Token: None 2023-05-31T21:03:23.057 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 21:03:23 GMT /pklxkd-61/obj58?versionId=null 2023-05-31T21:03:23.057 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:Azcg0SZ9xp6VOhak93Dxf5WBtus= 2023-05-31T21:03:23.058 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 21:03:23 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:Azcg0SZ9xp6VOhak93Dxf5WBtus=', 'Content-Length': '0'} 2023-05-31T21:03:23.060 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:59:45 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008a210d91d781fb6e-006477b61b-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 21:03:23 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:03:23.061 DEBUG:boto:path=/pklxkd-61/obj58 2023-05-31T21:03:23.062 DEBUG:boto:auth_path=/pklxkd-61/obj58 2023-05-31T21:03:23.062 DEBUG:boto:path=/pklxkd-61/obj58?versionId=null 2023-05-31T21:03:23.062 DEBUG:boto:auth_path=/pklxkd-61/obj58?versionId=null 2023-05-31T21:03:23.063 DEBUG:boto:Method: HEAD 2023-05-31T21:03:23.063 DEBUG:boto:Path: /pklxkd-61/obj58?versionId=null 2023-05-31T21:03:23.063 DEBUG:boto:Data: 2023-05-31T21:03:23.064 DEBUG:boto:Headers: {} 2023-05-31T21:03:23.064 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T21:03:23.065 DEBUG:boto:Port: 8001 2023-05-31T21:03:23.065 DEBUG:boto:Params: {} 2023-05-31T21:03:23.065 DEBUG:boto:Token: None 2023-05-31T21:03:23.066 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 21:03:23 GMT /pklxkd-61/obj58?versionId=null 2023-05-31T21:03:23.066 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:bXmm8C7ns396Ccq5ACg4wc6wcIo= 2023-05-31T21:03:23.067 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 21:03:23 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:bXmm8C7ns396Ccq5ACg4wc6wcIo=', 'Content-Length': '0'} 2023-05-31T21:03:23.069 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:59:45 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b93917a6419dd4cf-006477b61b-1f4e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 21:03:23 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:03:23.070 DEBUG:boto:path=/pklxkd-61/obj58 2023-05-31T21:03:23.071 DEBUG:boto:auth_path=/pklxkd-61/obj58 2023-05-31T21:03:23.072 DEBUG:boto:path=/pklxkd-61/obj58?versionId=null 2023-05-31T21:03:23.072 DEBUG:boto:auth_path=/pklxkd-61/obj58?versionId=null 2023-05-31T21:03:23.073 DEBUG:boto:Method: HEAD 2023-05-31T21:03:23.074 DEBUG:boto:Path: /pklxkd-61/obj58?versionId=null 2023-05-31T21:03:23.074 DEBUG:boto:Data: 2023-05-31T21:03:23.074 DEBUG:boto:Headers: {} 2023-05-31T21:03:23.075 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T21:03:23.075 DEBUG:boto:Port: 8000 2023-05-31T21:03:23.075 DEBUG:boto:Params: {} 2023-05-31T21:03:23.076 DEBUG:boto:Token: None 2023-05-31T21:03:23.076 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 21:03:23 GMT /pklxkd-61/obj58?versionId=null 2023-05-31T21:03:23.077 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:bXmm8C7ns396Ccq5ACg4wc6wcIo= 2023-05-31T21:03:23.077 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 21:03:23 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:bXmm8C7ns396Ccq5ACg4wc6wcIo=', 'Content-Length': '0'} 2023-05-31T21:03:23.079 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:59:45 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003bcf425058d91d6a-006477b61b-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 21:03:23 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:03:23.080 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj58 2023-05-31T21:03:23.080 DEBUG:boto:path=/pklxkd-61/obj58 2023-05-31T21:03:23.081 DEBUG:boto:auth_path=/pklxkd-61/obj58 2023-05-31T21:03:23.082 DEBUG:boto:path=/pklxkd-61/obj58?versionId=null 2023-05-31T21:03:23.083 DEBUG:boto:auth_path=/pklxkd-61/obj58?versionId=null 2023-05-31T21:03:23.083 DEBUG:boto:Method: GET 2023-05-31T21:03:23.084 DEBUG:boto:Path: /pklxkd-61/obj58?versionId=null 2023-05-31T21:03:23.084 DEBUG:boto:Data: 2023-05-31T21:03:23.085 DEBUG:boto:Headers: {} 2023-05-31T21:03:23.085 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T21:03:23.086 DEBUG:boto:Port: 8001 2023-05-31T21:03:23.086 DEBUG:boto:Params: {} 2023-05-31T21:03:23.087 DEBUG:boto:Token: None 2023-05-31T21:03:23.087 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 21:03:23 GMT /pklxkd-61/obj58?versionId=null 2023-05-31T21:03:23.088 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:Azcg0SZ9xp6VOhak93Dxf5WBtus= 2023-05-31T21:03:23.088 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 21:03:23 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:Azcg0SZ9xp6VOhak93Dxf5WBtus=', 'Content-Length': '0'} 2023-05-31T21:03:23.090 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:59:45 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a3536b5a660ecabb-006477b61b-1f4e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 21:03:23 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:03:23.091 DEBUG:boto:path=/pklxkd-61/obj58 2023-05-31T21:03:23.091 DEBUG:boto:auth_path=/pklxkd-61/obj58 2023-05-31T21:03:23.091 DEBUG:boto:path=/pklxkd-61/obj58?versionId=null 2023-05-31T21:03:23.092 DEBUG:boto:auth_path=/pklxkd-61/obj58?versionId=null 2023-05-31T21:03:23.092 DEBUG:boto:Method: GET 2023-05-31T21:03:23.093 DEBUG:boto:Path: /pklxkd-61/obj58?versionId=null 2023-05-31T21:03:23.093 DEBUG:boto:Data: 2023-05-31T21:03:23.093 DEBUG:boto:Headers: {} 2023-05-31T21:03:23.094 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T21:03:23.094 DEBUG:boto:Port: 8000 2023-05-31T21:03:23.095 DEBUG:boto:Params: {} 2023-05-31T21:03:23.095 DEBUG:boto:Token: None 2023-05-31T21:03:23.095 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 21:03:23 GMT /pklxkd-61/obj58?versionId=null 2023-05-31T21:03:23.096 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:Azcg0SZ9xp6VOhak93Dxf5WBtus= 2023-05-31T21:03:23.096 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 21:03:23 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:Azcg0SZ9xp6VOhak93Dxf5WBtus=', 'Content-Length': '0'} 2023-05-31T21:03:23.098 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:59:45 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002b88b6fe0b298f5f-006477b61b-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 21:03:23 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:03:23.099 DEBUG:boto:path=/pklxkd-61/ 2023-05-31T21:03:23.099 DEBUG:boto:auth_path=/pklxkd-61/ 2023-05-31T21:03:23.100 DEBUG:boto:path=/pklxkd-61/?prefix=obj58 2023-05-31T21:03:23.100 DEBUG:boto:auth_path=/pklxkd-61/?prefix=obj58 2023-05-31T21:03:23.101 DEBUG:boto:Method: GET 2023-05-31T21:03:23.101 DEBUG:boto:Path: /pklxkd-61/?prefix=obj58 2023-05-31T21:03:23.101 DEBUG:boto:Data: 2023-05-31T21:03:23.102 DEBUG:boto:Headers: {} 2023-05-31T21:03:23.102 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T21:03:23.102 DEBUG:boto:Port: 8001 2023-05-31T21:03:23.103 DEBUG:boto:Params: {} 2023-05-31T21:03:23.104 DEBUG:boto:Token: None 2023-05-31T21:03:23.104 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 21:03:23 GMT /pklxkd-61/ 2023-05-31T21:03:23.104 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:7YkKDTB+91bWua1TWmlOlWHvc2U= 2023-05-31T21:03:23.105 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 21:03:23 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:7YkKDTB+91bWua1TWmlOlWHvc2U=', 'Content-Length': '0'} 2023-05-31T21:03:23.108 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000002c0633876629ec1f-006477b61b-1f4e-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 21:03:23 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:03:23.109 DEBUG:boto:b'pklxkd-61obj581000falseobj582023-05-31T20:59:45.905Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T21:03:23.109 DEBUG:boto:path=/pklxkd-61/ 2023-05-31T21:03:23.110 DEBUG:boto:auth_path=/pklxkd-61/ 2023-05-31T21:03:23.110 DEBUG:boto:path=/pklxkd-61/?prefix=obj58 2023-05-31T21:03:23.111 DEBUG:boto:auth_path=/pklxkd-61/?prefix=obj58 2023-05-31T21:03:23.111 DEBUG:boto:Method: GET 2023-05-31T21:03:23.111 DEBUG:boto:Path: /pklxkd-61/?prefix=obj58 2023-05-31T21:03:23.112 DEBUG:boto:Data: 2023-05-31T21:03:23.112 DEBUG:boto:Headers: {} 2023-05-31T21:03:23.112 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T21:03:23.113 DEBUG:boto:Port: 8000 2023-05-31T21:03:23.113 DEBUG:boto:Params: {} 2023-05-31T21:03:23.114 DEBUG:boto:Token: None 2023-05-31T21:03:23.114 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 21:03:23 GMT /pklxkd-61/ 2023-05-31T21:03:23.115 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:7YkKDTB+91bWua1TWmlOlWHvc2U= 2023-05-31T21:03:23.115 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 21:03:23 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:7YkKDTB+91bWua1TWmlOlWHvc2U=', 'Content-Length': '0'} 2023-05-31T21:03:23.119 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000003f148416bb86b7a6-006477b61b-1630-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 21:03:23 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:03:23.120 DEBUG:boto:b'pklxkd-61obj581000falseobj582023-05-31T20:59:45.905Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T21:03:23.120 DEBUG:boto:path=/pklxkd-61/obj58 2023-05-31T21:03:23.121 DEBUG:boto:auth_path=/pklxkd-61/obj58 2023-05-31T21:03:23.121 DEBUG:boto:Method: HEAD 2023-05-31T21:03:23.121 DEBUG:boto:Path: /pklxkd-61/obj58 2023-05-31T21:03:23.122 DEBUG:boto:Data: 2023-05-31T21:03:23.122 DEBUG:boto:Headers: {} 2023-05-31T21:03:23.122 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T21:03:23.123 DEBUG:boto:Port: 8001 2023-05-31T21:03:23.123 DEBUG:boto:Params: {} 2023-05-31T21:03:23.124 DEBUG:boto:Token: None 2023-05-31T21:03:23.124 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 21:03:23 GMT /pklxkd-61/obj58 2023-05-31T21:03:23.124 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:vcOxZccRpdUOk521lHj2Lna/IN0= 2023-05-31T21:03:23.125 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 21:03:23 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:vcOxZccRpdUOk521lHj2Lna/IN0=', 'Content-Length': '0'} 2023-05-31T21:03:23.127 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:59:45 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000905755b46a1c8346-006477b61b-1f4e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 21:03:23 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:03:23.127 DEBUG:boto:path=/pklxkd-61/obj58 2023-05-31T21:03:23.128 DEBUG:boto:auth_path=/pklxkd-61/obj58 2023-05-31T21:03:23.128 DEBUG:boto:Method: HEAD 2023-05-31T21:03:23.128 DEBUG:boto:Path: /pklxkd-61/obj58 2023-05-31T21:03:23.129 DEBUG:boto:Data: 2023-05-31T21:03:23.129 DEBUG:boto:Headers: {} 2023-05-31T21:03:23.129 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T21:03:23.130 DEBUG:boto:Port: 8000 2023-05-31T21:03:23.130 DEBUG:boto:Params: {} 2023-05-31T21:03:23.130 DEBUG:boto:Token: None 2023-05-31T21:03:23.131 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 21:03:23 GMT /pklxkd-61/obj58 2023-05-31T21:03:23.131 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:vcOxZccRpdUOk521lHj2Lna/IN0= 2023-05-31T21:03:23.132 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 21:03:23 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:vcOxZccRpdUOk521lHj2Lna/IN0=', 'Content-Length': '0'} 2023-05-31T21:03:23.133 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:59:45 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c60b06bfe3a74d7b-006477b61b-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 21:03:23 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:03:23.134 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj58 2023-05-31T21:03:23.134 DEBUG:boto:path=/pklxkd-61/obj58 2023-05-31T21:03:23.135 DEBUG:boto:auth_path=/pklxkd-61/obj58 2023-05-31T21:03:23.135 DEBUG:boto:Method: GET 2023-05-31T21:03:23.135 DEBUG:boto:Path: /pklxkd-61/obj58 2023-05-31T21:03:23.136 DEBUG:boto:Data: 2023-05-31T21:03:23.136 DEBUG:boto:Headers: {} 2023-05-31T21:03:23.136 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T21:03:23.137 DEBUG:boto:Port: 8001 2023-05-31T21:03:23.137 DEBUG:boto:Params: {} 2023-05-31T21:03:23.138 DEBUG:boto:Token: None 2023-05-31T21:03:23.138 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 21:03:23 GMT /pklxkd-61/obj58 2023-05-31T21:03:23.138 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:phV3pIvVSKY7pWryrd8kAiudGQw= 2023-05-31T21:03:23.139 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 21:03:23 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:phV3pIvVSKY7pWryrd8kAiudGQw=', 'Content-Length': '0'} 2023-05-31T21:03:23.140 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:59:45 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009fbe8874ace9c038-006477b61b-1f4e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 21:03:23 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:03:23.184 DEBUG:boto:path=/pklxkd-61/obj58 2023-05-31T21:03:23.185 DEBUG:boto:auth_path=/pklxkd-61/obj58 2023-05-31T21:03:23.185 DEBUG:boto:Method: GET 2023-05-31T21:03:23.186 DEBUG:boto:Path: /pklxkd-61/obj58 2023-05-31T21:03:23.186 DEBUG:boto:Data: 2023-05-31T21:03:23.186 DEBUG:boto:Headers: {} 2023-05-31T21:03:23.187 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T21:03:23.187 DEBUG:boto:Port: 8000 2023-05-31T21:03:23.187 DEBUG:boto:Params: {} 2023-05-31T21:03:23.188 DEBUG:boto:Token: None 2023-05-31T21:03:23.188 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 21:03:23 GMT /pklxkd-61/obj58 2023-05-31T21:03:23.189 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:phV3pIvVSKY7pWryrd8kAiudGQw= 2023-05-31T21:03:23.189 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 21:03:23 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:phV3pIvVSKY7pWryrd8kAiudGQw=', 'Content-Length': '0'} 2023-05-31T21:03:23.191 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:59:45 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f332346dfd5b3d63-006477b61b-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 21:03:23 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:03:23.192 DEBUG:boto:path=/pklxkd-61/ 2023-05-31T21:03:23.192 DEBUG:boto:auth_path=/pklxkd-61/ 2023-05-31T21:03:23.193 DEBUG:boto:path=/pklxkd-61/?prefix=obj58 2023-05-31T21:03:23.193 DEBUG:boto:auth_path=/pklxkd-61/?prefix=obj58 2023-05-31T21:03:23.193 DEBUG:boto:Method: GET 2023-05-31T21:03:23.194 DEBUG:boto:Path: /pklxkd-61/?prefix=obj58 2023-05-31T21:03:23.194 DEBUG:boto:Data: 2023-05-31T21:03:23.194 DEBUG:boto:Headers: {} 2023-05-31T21:03:23.195 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T21:03:23.195 DEBUG:boto:Port: 8001 2023-05-31T21:03:23.195 DEBUG:boto:Params: {} 2023-05-31T21:03:23.196 DEBUG:boto:Token: None 2023-05-31T21:03:23.196 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 21:03:23 GMT /pklxkd-61/ 2023-05-31T21:03:23.197 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:7YkKDTB+91bWua1TWmlOlWHvc2U= 2023-05-31T21:03:23.197 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 21:03:23 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:7YkKDTB+91bWua1TWmlOlWHvc2U=', 'Content-Length': '0'} 2023-05-31T21:03:23.199 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000816ae7519d47e259-006477b61b-1f4e-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 21:03:23 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:03:23.200 DEBUG:boto:b'pklxkd-61obj581000falseobj582023-05-31T20:59:45.905Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T21:03:23.201 DEBUG:boto:path=/pklxkd-61/ 2023-05-31T21:03:23.201 DEBUG:boto:auth_path=/pklxkd-61/ 2023-05-31T21:03:23.201 DEBUG:boto:path=/pklxkd-61/?prefix=obj58 2023-05-31T21:03:23.202 DEBUG:boto:auth_path=/pklxkd-61/?prefix=obj58 2023-05-31T21:03:23.202 DEBUG:boto:Method: GET 2023-05-31T21:03:23.202 DEBUG:boto:Path: /pklxkd-61/?prefix=obj58 2023-05-31T21:03:23.203 DEBUG:boto:Data: 2023-05-31T21:03:23.203 DEBUG:boto:Headers: {} 2023-05-31T21:03:23.203 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T21:03:23.204 DEBUG:boto:Port: 8000 2023-05-31T21:03:23.204 DEBUG:boto:Params: {} 2023-05-31T21:03:23.205 DEBUG:boto:Token: None 2023-05-31T21:03:23.205 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 21:03:23 GMT /pklxkd-61/ 2023-05-31T21:03:23.205 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:7YkKDTB+91bWua1TWmlOlWHvc2U= 2023-05-31T21:03:23.206 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 21:03:23 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:7YkKDTB+91bWua1TWmlOlWHvc2U=', 'Content-Length': '0'} 2023-05-31T21:03:23.210 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000881bdb699956c026-006477b61b-1630-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 21:03:23 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:03:23.210 DEBUG:boto:b'pklxkd-61obj581000falseobj582023-05-31T20:59:45.905Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T21:03:23.211 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj62 2023-05-31T21:03:23.211 DEBUG:boto:path=/pklxkd-61/obj62 2023-05-31T21:03:23.212 DEBUG:boto:auth_path=/pklxkd-61/obj62 2023-05-31T21:03:23.212 DEBUG:boto:path=/pklxkd-61/obj62?versionId=null 2023-05-31T21:03:23.212 DEBUG:boto:auth_path=/pklxkd-61/obj62?versionId=null 2023-05-31T21:03:23.213 DEBUG:boto:Method: GET 2023-05-31T21:03:23.213 DEBUG:boto:Path: /pklxkd-61/obj62?versionId=null 2023-05-31T21:03:23.213 DEBUG:boto:Data: 2023-05-31T21:03:23.214 DEBUG:boto:Headers: {} 2023-05-31T21:03:23.214 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T21:03:23.214 DEBUG:boto:Port: 8001 2023-05-31T21:03:23.215 DEBUG:boto:Params: {} 2023-05-31T21:03:23.215 DEBUG:boto:Token: None 2023-05-31T21:03:23.216 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 21:03:23 GMT /pklxkd-61/obj62?versionId=null 2023-05-31T21:03:23.216 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:06LPDY0a0ZA1Q+gd+eSig3WflZU= 2023-05-31T21:03:23.216 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 21:03:23 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:06LPDY0a0ZA1Q+gd+eSig3WflZU=', 'Content-Length': '0'} 2023-05-31T21:03:23.218 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:59:46 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b09c51570ad1ca7e-006477b61b-1f4e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 21:03:23 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:03:23.219 DEBUG:boto:path=/pklxkd-61/obj62 2023-05-31T21:03:23.219 DEBUG:boto:auth_path=/pklxkd-61/obj62 2023-05-31T21:03:23.220 DEBUG:boto:path=/pklxkd-61/obj62?versionId=null 2023-05-31T21:03:23.220 DEBUG:boto:auth_path=/pklxkd-61/obj62?versionId=null 2023-05-31T21:03:23.220 DEBUG:boto:Method: GET 2023-05-31T21:03:23.221 DEBUG:boto:Path: /pklxkd-61/obj62?versionId=null 2023-05-31T21:03:23.221 DEBUG:boto:Data: 2023-05-31T21:03:23.221 DEBUG:boto:Headers: {} 2023-05-31T21:03:23.222 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T21:03:23.222 DEBUG:boto:Port: 8000 2023-05-31T21:03:23.222 DEBUG:boto:Params: {} 2023-05-31T21:03:23.223 DEBUG:boto:Token: None 2023-05-31T21:03:23.223 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 21:03:23 GMT /pklxkd-61/obj62?versionId=null 2023-05-31T21:03:23.224 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:06LPDY0a0ZA1Q+gd+eSig3WflZU= 2023-05-31T21:03:23.224 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 21:03:23 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:06LPDY0a0ZA1Q+gd+eSig3WflZU=', 'Content-Length': '0'} 2023-05-31T21:03:23.226 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:59:46 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f8326ec70d47ab3e-006477b61b-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 21:03:23 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:03:23.272 DEBUG:boto:path=/pklxkd-61/obj62 2023-05-31T21:03:23.273 DEBUG:boto:auth_path=/pklxkd-61/obj62 2023-05-31T21:03:23.273 DEBUG:boto:path=/pklxkd-61/obj62?versionId=null 2023-05-31T21:03:23.273 DEBUG:boto:auth_path=/pklxkd-61/obj62?versionId=null 2023-05-31T21:03:23.274 DEBUG:boto:Method: HEAD 2023-05-31T21:03:23.274 DEBUG:boto:Path: /pklxkd-61/obj62?versionId=null 2023-05-31T21:03:23.274 DEBUG:boto:Data: 2023-05-31T21:03:23.275 DEBUG:boto:Headers: {} 2023-05-31T21:03:23.275 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T21:03:23.275 DEBUG:boto:Port: 8001 2023-05-31T21:03:23.276 DEBUG:boto:Params: {} 2023-05-31T21:03:23.276 DEBUG:boto:Token: None 2023-05-31T21:03:23.276 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 21:03:23 GMT /pklxkd-61/obj62?versionId=null 2023-05-31T21:03:23.277 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:xAkw0H76VgIK3+r2VtH39nq7fMk= 2023-05-31T21:03:23.277 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 21:03:23 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:xAkw0H76VgIK3+r2VtH39nq7fMk=', 'Content-Length': '0'} 2023-05-31T21:03:23.279 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:59:46 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e5eb0fd63f10d3d4-006477b61b-1f4e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 21:03:23 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:03:23.279 DEBUG:boto:path=/pklxkd-61/obj62 2023-05-31T21:03:23.280 DEBUG:boto:auth_path=/pklxkd-61/obj62 2023-05-31T21:03:23.280 DEBUG:boto:path=/pklxkd-61/obj62?versionId=null 2023-05-31T21:03:23.280 DEBUG:boto:auth_path=/pklxkd-61/obj62?versionId=null 2023-05-31T21:03:23.281 DEBUG:boto:Method: HEAD 2023-05-31T21:03:23.281 DEBUG:boto:Path: /pklxkd-61/obj62?versionId=null 2023-05-31T21:03:23.281 DEBUG:boto:Data: 2023-05-31T21:03:23.282 DEBUG:boto:Headers: {} 2023-05-31T21:03:23.282 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T21:03:23.282 DEBUG:boto:Port: 8000 2023-05-31T21:03:23.283 DEBUG:boto:Params: {} 2023-05-31T21:03:23.283 DEBUG:boto:Token: None 2023-05-31T21:03:23.283 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 21:03:23 GMT /pklxkd-61/obj62?versionId=null 2023-05-31T21:03:23.284 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:xAkw0H76VgIK3+r2VtH39nq7fMk= 2023-05-31T21:03:23.284 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 21:03:23 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:xAkw0H76VgIK3+r2VtH39nq7fMk=', 'Content-Length': '0'} 2023-05-31T21:03:23.286 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:59:46 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005560ca40b57b17f6-006477b61b-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 21:03:23 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:03:23.286 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj62 2023-05-31T21:03:23.287 DEBUG:boto:path=/pklxkd-61/obj62 2023-05-31T21:03:23.287 DEBUG:boto:auth_path=/pklxkd-61/obj62 2023-05-31T21:03:23.288 DEBUG:boto:path=/pklxkd-61/obj62?versionId=null 2023-05-31T21:03:23.288 DEBUG:boto:auth_path=/pklxkd-61/obj62?versionId=null 2023-05-31T21:03:23.288 DEBUG:boto:Method: GET 2023-05-31T21:03:23.289 DEBUG:boto:Path: /pklxkd-61/obj62?versionId=null 2023-05-31T21:03:23.289 DEBUG:boto:Data: 2023-05-31T21:03:23.289 DEBUG:boto:Headers: {} 2023-05-31T21:03:23.290 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T21:03:23.290 DEBUG:boto:Port: 8001 2023-05-31T21:03:23.290 DEBUG:boto:Params: {} 2023-05-31T21:03:23.291 DEBUG:boto:Token: None 2023-05-31T21:03:23.291 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 21:03:23 GMT /pklxkd-61/obj62?versionId=null 2023-05-31T21:03:23.292 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:06LPDY0a0ZA1Q+gd+eSig3WflZU= 2023-05-31T21:03:23.292 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 21:03:23 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:06LPDY0a0ZA1Q+gd+eSig3WflZU=', 'Content-Length': '0'} 2023-05-31T21:03:23.294 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:59:46 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000675f77dbbbf0b0d-006477b61b-1f4e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 21:03:23 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:03:23.295 DEBUG:boto:path=/pklxkd-61/obj62 2023-05-31T21:03:23.295 DEBUG:boto:auth_path=/pklxkd-61/obj62 2023-05-31T21:03:23.295 DEBUG:boto:path=/pklxkd-61/obj62?versionId=null 2023-05-31T21:03:23.296 DEBUG:boto:auth_path=/pklxkd-61/obj62?versionId=null 2023-05-31T21:03:23.296 DEBUG:boto:Method: GET 2023-05-31T21:03:23.296 DEBUG:boto:Path: /pklxkd-61/obj62?versionId=null 2023-05-31T21:03:23.297 DEBUG:boto:Data: 2023-05-31T21:03:23.297 DEBUG:boto:Headers: {} 2023-05-31T21:03:23.297 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T21:03:23.298 DEBUG:boto:Port: 8000 2023-05-31T21:03:23.298 DEBUG:boto:Params: {} 2023-05-31T21:03:23.299 DEBUG:boto:Token: None 2023-05-31T21:03:23.299 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 21:03:23 GMT /pklxkd-61/obj62?versionId=null 2023-05-31T21:03:23.299 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:06LPDY0a0ZA1Q+gd+eSig3WflZU= 2023-05-31T21:03:23.300 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 21:03:23 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:06LPDY0a0ZA1Q+gd+eSig3WflZU=', 'Content-Length': '0'} 2023-05-31T21:03:23.302 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:59:46 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002992412adba0855e-006477b61b-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 21:03:23 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:03:23.302 DEBUG:boto:path=/pklxkd-61/ 2023-05-31T21:03:23.303 DEBUG:boto:auth_path=/pklxkd-61/ 2023-05-31T21:03:23.303 DEBUG:boto:path=/pklxkd-61/?prefix=obj62 2023-05-31T21:03:23.304 DEBUG:boto:auth_path=/pklxkd-61/?prefix=obj62 2023-05-31T21:03:23.304 DEBUG:boto:Method: GET 2023-05-31T21:03:23.304 DEBUG:boto:Path: /pklxkd-61/?prefix=obj62 2023-05-31T21:03:23.305 DEBUG:boto:Data: 2023-05-31T21:03:23.305 DEBUG:boto:Headers: {} 2023-05-31T21:03:23.305 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T21:03:23.306 DEBUG:boto:Port: 8001 2023-05-31T21:03:23.306 DEBUG:boto:Params: {} 2023-05-31T21:03:23.307 DEBUG:boto:Token: None 2023-05-31T21:03:23.307 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 21:03:23 GMT /pklxkd-61/ 2023-05-31T21:03:23.307 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:7YkKDTB+91bWua1TWmlOlWHvc2U= 2023-05-31T21:03:23.308 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 21:03:23 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:7YkKDTB+91bWua1TWmlOlWHvc2U=', 'Content-Length': '0'} 2023-05-31T21:03:23.310 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000004028d98bf3228d3a-006477b61b-1f4e-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 21:03:23 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:03:23.311 DEBUG:boto:b'pklxkd-61obj621000falseobj622023-05-31T20:59:46.340Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T21:03:23.311 DEBUG:boto:path=/pklxkd-61/ 2023-05-31T21:03:23.312 DEBUG:boto:auth_path=/pklxkd-61/ 2023-05-31T21:03:23.312 DEBUG:boto:path=/pklxkd-61/?prefix=obj62 2023-05-31T21:03:23.313 DEBUG:boto:auth_path=/pklxkd-61/?prefix=obj62 2023-05-31T21:03:23.313 DEBUG:boto:Method: GET 2023-05-31T21:03:23.313 DEBUG:boto:Path: /pklxkd-61/?prefix=obj62 2023-05-31T21:03:23.314 DEBUG:boto:Data: 2023-05-31T21:03:23.314 DEBUG:boto:Headers: {} 2023-05-31T21:03:23.314 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T21:03:23.315 DEBUG:boto:Port: 8000 2023-05-31T21:03:23.315 DEBUG:boto:Params: {} 2023-05-31T21:03:23.315 DEBUG:boto:Token: None 2023-05-31T21:03:23.316 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 21:03:23 GMT /pklxkd-61/ 2023-05-31T21:03:23.316 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:7YkKDTB+91bWua1TWmlOlWHvc2U= 2023-05-31T21:03:23.317 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 21:03:23 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:7YkKDTB+91bWua1TWmlOlWHvc2U=', 'Content-Length': '0'} 2023-05-31T21:03:23.321 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000000b2c73d5fa204af8-006477b61b-1630-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 21:03:23 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:03:23.321 DEBUG:boto:b'pklxkd-61obj621000falseobj622023-05-31T20:59:46.340Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T21:03:23.322 DEBUG:boto:path=/pklxkd-61/obj62 2023-05-31T21:03:23.322 DEBUG:boto:auth_path=/pklxkd-61/obj62 2023-05-31T21:03:23.323 DEBUG:boto:Method: HEAD 2023-05-31T21:03:23.323 DEBUG:boto:Path: /pklxkd-61/obj62 2023-05-31T21:03:23.323 DEBUG:boto:Data: 2023-05-31T21:03:23.324 DEBUG:boto:Headers: {} 2023-05-31T21:03:23.324 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T21:03:23.324 DEBUG:boto:Port: 8001 2023-05-31T21:03:23.325 DEBUG:boto:Params: {} 2023-05-31T21:03:23.325 DEBUG:boto:Token: None 2023-05-31T21:03:23.325 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 21:03:23 GMT /pklxkd-61/obj62 2023-05-31T21:03:23.326 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:vqfcI4e43w5EGvXtsdjPnoPD9mc= 2023-05-31T21:03:23.326 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 21:03:23 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:vqfcI4e43w5EGvXtsdjPnoPD9mc=', 'Content-Length': '0'} 2023-05-31T21:03:23.328 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:59:46 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006f274b9e3d0f6fed-006477b61b-1f4e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 21:03:23 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:03:23.328 DEBUG:boto:path=/pklxkd-61/obj62 2023-05-31T21:03:23.329 DEBUG:boto:auth_path=/pklxkd-61/obj62 2023-05-31T21:03:23.329 DEBUG:boto:Method: HEAD 2023-05-31T21:03:23.329 DEBUG:boto:Path: /pklxkd-61/obj62 2023-05-31T21:03:23.330 DEBUG:boto:Data: 2023-05-31T21:03:23.330 DEBUG:boto:Headers: {} 2023-05-31T21:03:23.330 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T21:03:23.331 DEBUG:boto:Port: 8000 2023-05-31T21:03:23.331 DEBUG:boto:Params: {} 2023-05-31T21:03:23.331 DEBUG:boto:Token: None 2023-05-31T21:03:23.332 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 21:03:23 GMT /pklxkd-61/obj62 2023-05-31T21:03:23.332 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:vqfcI4e43w5EGvXtsdjPnoPD9mc= 2023-05-31T21:03:23.333 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 21:03:23 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:vqfcI4e43w5EGvXtsdjPnoPD9mc=', 'Content-Length': '0'} 2023-05-31T21:03:23.334 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:59:46 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005c73f4900fac08b3-006477b61b-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 21:03:23 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:03:23.335 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj62 2023-05-31T21:03:23.335 DEBUG:boto:path=/pklxkd-61/obj62 2023-05-31T21:03:23.336 DEBUG:boto:auth_path=/pklxkd-61/obj62 2023-05-31T21:03:23.336 DEBUG:boto:Method: GET 2023-05-31T21:03:23.336 DEBUG:boto:Path: /pklxkd-61/obj62 2023-05-31T21:03:23.337 DEBUG:boto:Data: 2023-05-31T21:03:23.337 DEBUG:boto:Headers: {} 2023-05-31T21:03:23.338 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T21:03:23.338 DEBUG:boto:Port: 8001 2023-05-31T21:03:23.338 DEBUG:boto:Params: {} 2023-05-31T21:03:23.339 DEBUG:boto:Token: None 2023-05-31T21:03:23.339 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 21:03:23 GMT /pklxkd-61/obj62 2023-05-31T21:03:23.340 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:5RRth3E/AzJYASslniBx2ADQI6g= 2023-05-31T21:03:23.340 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 21:03:23 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:5RRth3E/AzJYASslniBx2ADQI6g=', 'Content-Length': '0'} 2023-05-31T21:03:23.342 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:59:46 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b270f3f5e1802b55-006477b61b-1f4e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 21:03:23 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:03:23.342 DEBUG:boto:path=/pklxkd-61/obj62 2023-05-31T21:03:23.343 DEBUG:boto:auth_path=/pklxkd-61/obj62 2023-05-31T21:03:23.343 DEBUG:boto:Method: GET 2023-05-31T21:03:23.343 DEBUG:boto:Path: /pklxkd-61/obj62 2023-05-31T21:03:23.344 DEBUG:boto:Data: 2023-05-31T21:03:23.344 DEBUG:boto:Headers: {} 2023-05-31T21:03:23.344 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T21:03:23.345 DEBUG:boto:Port: 8000 2023-05-31T21:03:23.345 DEBUG:boto:Params: {} 2023-05-31T21:03:23.346 DEBUG:boto:Token: None 2023-05-31T21:03:23.346 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 21:03:23 GMT /pklxkd-61/obj62 2023-05-31T21:03:23.346 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:5RRth3E/AzJYASslniBx2ADQI6g= 2023-05-31T21:03:23.347 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 21:03:23 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:5RRth3E/AzJYASslniBx2ADQI6g=', 'Content-Length': '0'} 2023-05-31T21:03:23.349 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:59:46 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000064830cc99d6e7e16-006477b61b-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 21:03:23 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:03:23.349 DEBUG:boto:path=/pklxkd-61/ 2023-05-31T21:03:23.350 DEBUG:boto:auth_path=/pklxkd-61/ 2023-05-31T21:03:23.350 DEBUG:boto:path=/pklxkd-61/?prefix=obj62 2023-05-31T21:03:23.350 DEBUG:boto:auth_path=/pklxkd-61/?prefix=obj62 2023-05-31T21:03:23.351 DEBUG:boto:Method: GET 2023-05-31T21:03:23.351 DEBUG:boto:Path: /pklxkd-61/?prefix=obj62 2023-05-31T21:03:23.351 DEBUG:boto:Data: 2023-05-31T21:03:23.352 DEBUG:boto:Headers: {} 2023-05-31T21:03:23.352 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T21:03:23.352 DEBUG:boto:Port: 8001 2023-05-31T21:03:23.353 DEBUG:boto:Params: {} 2023-05-31T21:03:23.353 DEBUG:boto:Token: None 2023-05-31T21:03:23.354 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 21:03:23 GMT /pklxkd-61/ 2023-05-31T21:03:23.354 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:7YkKDTB+91bWua1TWmlOlWHvc2U= 2023-05-31T21:03:23.354 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 21:03:23 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:7YkKDTB+91bWua1TWmlOlWHvc2U=', 'Content-Length': '0'} 2023-05-31T21:03:23.357 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000499eb6067f07437d-006477b61b-1f4e-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 21:03:23 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:03:23.357 DEBUG:boto:b'pklxkd-61obj621000falseobj622023-05-31T20:59:46.340Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T21:03:23.358 DEBUG:boto:path=/pklxkd-61/ 2023-05-31T21:03:23.358 DEBUG:boto:auth_path=/pklxkd-61/ 2023-05-31T21:03:23.359 DEBUG:boto:path=/pklxkd-61/?prefix=obj62 2023-05-31T21:03:23.359 DEBUG:boto:auth_path=/pklxkd-61/?prefix=obj62 2023-05-31T21:03:23.359 DEBUG:boto:Method: GET 2023-05-31T21:03:23.360 DEBUG:boto:Path: /pklxkd-61/?prefix=obj62 2023-05-31T21:03:23.360 DEBUG:boto:Data: 2023-05-31T21:03:23.360 DEBUG:boto:Headers: {} 2023-05-31T21:03:23.361 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T21:03:23.361 DEBUG:boto:Port: 8000 2023-05-31T21:03:23.361 DEBUG:boto:Params: {} 2023-05-31T21:03:23.362 DEBUG:boto:Token: None 2023-05-31T21:03:23.362 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 21:03:23 GMT /pklxkd-61/ 2023-05-31T21:03:23.363 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:7YkKDTB+91bWua1TWmlOlWHvc2U= 2023-05-31T21:03:23.363 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 21:03:23 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:7YkKDTB+91bWua1TWmlOlWHvc2U=', 'Content-Length': '0'} 2023-05-31T21:03:23.367 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c9f2e5892baed43c-006477b61b-1630-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 21:03:23 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:03:23.412 DEBUG:boto:b'pklxkd-61obj621000falseobj622023-05-31T20:59:46.340Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T21:03:23.413 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj68 2023-05-31T21:03:23.414 DEBUG:boto:path=/pklxkd-61/obj68 2023-05-31T21:03:23.414 DEBUG:boto:auth_path=/pklxkd-61/obj68 2023-05-31T21:03:23.414 DEBUG:boto:path=/pklxkd-61/obj68?versionId=null 2023-05-31T21:03:23.415 DEBUG:boto:auth_path=/pklxkd-61/obj68?versionId=null 2023-05-31T21:03:23.415 DEBUG:boto:Method: GET 2023-05-31T21:03:23.415 DEBUG:boto:Path: /pklxkd-61/obj68?versionId=null 2023-05-31T21:03:23.416 DEBUG:boto:Data: 2023-05-31T21:03:23.416 DEBUG:boto:Headers: {} 2023-05-31T21:03:23.416 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T21:03:23.417 DEBUG:boto:Port: 8001 2023-05-31T21:03:23.417 DEBUG:boto:Params: {} 2023-05-31T21:03:23.418 DEBUG:boto:Token: None 2023-05-31T21:03:23.418 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 21:03:23 GMT /pklxkd-61/obj68?versionId=null 2023-05-31T21:03:23.418 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:ieHdN8xZ7ZmWgiAXKUxVHAPx5DY= 2023-05-31T21:03:23.419 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 21:03:23 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:ieHdN8xZ7ZmWgiAXKUxVHAPx5DY=', 'Content-Length': '0'} 2023-05-31T21:03:23.421 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:59:44 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e795d2eb9c413a41-006477b61b-1f4e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 21:03:23 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:03:23.421 DEBUG:boto:path=/pklxkd-61/obj68 2023-05-31T21:03:23.422 DEBUG:boto:auth_path=/pklxkd-61/obj68 2023-05-31T21:03:23.422 DEBUG:boto:path=/pklxkd-61/obj68?versionId=null 2023-05-31T21:03:23.423 DEBUG:boto:auth_path=/pklxkd-61/obj68?versionId=null 2023-05-31T21:03:23.423 DEBUG:boto:Method: GET 2023-05-31T21:03:23.423 DEBUG:boto:Path: /pklxkd-61/obj68?versionId=null 2023-05-31T21:03:23.424 DEBUG:boto:Data: 2023-05-31T21:03:23.424 DEBUG:boto:Headers: {} 2023-05-31T21:03:23.424 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T21:03:23.425 DEBUG:boto:Port: 8000 2023-05-31T21:03:23.425 DEBUG:boto:Params: {} 2023-05-31T21:03:23.425 DEBUG:boto:Token: None 2023-05-31T21:03:23.426 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 21:03:23 GMT /pklxkd-61/obj68?versionId=null 2023-05-31T21:03:23.426 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:ieHdN8xZ7ZmWgiAXKUxVHAPx5DY= 2023-05-31T21:03:23.427 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 21:03:23 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:ieHdN8xZ7ZmWgiAXKUxVHAPx5DY=', 'Content-Length': '0'} 2023-05-31T21:03:23.429 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:59:44 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c16978c4e14ccd2d-006477b61b-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 21:03:23 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:03:23.429 DEBUG:boto:path=/pklxkd-61/obj68 2023-05-31T21:03:23.430 DEBUG:boto:auth_path=/pklxkd-61/obj68 2023-05-31T21:03:23.430 DEBUG:boto:path=/pklxkd-61/obj68?versionId=null 2023-05-31T21:03:23.431 DEBUG:boto:auth_path=/pklxkd-61/obj68?versionId=null 2023-05-31T21:03:23.431 DEBUG:boto:Method: HEAD 2023-05-31T21:03:23.431 DEBUG:boto:Path: /pklxkd-61/obj68?versionId=null 2023-05-31T21:03:23.432 DEBUG:boto:Data: 2023-05-31T21:03:23.432 DEBUG:boto:Headers: {} 2023-05-31T21:03:23.432 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T21:03:23.433 DEBUG:boto:Port: 8001 2023-05-31T21:03:23.433 DEBUG:boto:Params: {} 2023-05-31T21:03:23.433 DEBUG:boto:Token: None 2023-05-31T21:03:23.434 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 21:03:23 GMT /pklxkd-61/obj68?versionId=null 2023-05-31T21:03:23.434 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:IReXI1Z8xCQrhG+sDGikcMWsqps= 2023-05-31T21:03:23.434 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 21:03:23 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:IReXI1Z8xCQrhG+sDGikcMWsqps=', 'Content-Length': '0'} 2023-05-31T21:03:23.436 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:59:44 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000300ea9dae528dee5-006477b61b-1f4e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 21:03:23 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:03:23.437 DEBUG:boto:path=/pklxkd-61/obj68 2023-05-31T21:03:23.437 DEBUG:boto:auth_path=/pklxkd-61/obj68 2023-05-31T21:03:23.437 DEBUG:boto:path=/pklxkd-61/obj68?versionId=null 2023-05-31T21:03:23.438 DEBUG:boto:auth_path=/pklxkd-61/obj68?versionId=null 2023-05-31T21:03:23.438 DEBUG:boto:Method: HEAD 2023-05-31T21:03:23.438 DEBUG:boto:Path: /pklxkd-61/obj68?versionId=null 2023-05-31T21:03:23.439 DEBUG:boto:Data: 2023-05-31T21:03:23.439 DEBUG:boto:Headers: {} 2023-05-31T21:03:23.440 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T21:03:23.440 DEBUG:boto:Port: 8000 2023-05-31T21:03:23.440 DEBUG:boto:Params: {} 2023-05-31T21:03:23.441 DEBUG:boto:Token: None 2023-05-31T21:03:23.441 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 21:03:23 GMT /pklxkd-61/obj68?versionId=null 2023-05-31T21:03:23.441 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:IReXI1Z8xCQrhG+sDGikcMWsqps= 2023-05-31T21:03:23.442 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 21:03:23 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:IReXI1Z8xCQrhG+sDGikcMWsqps=', 'Content-Length': '0'} 2023-05-31T21:03:23.444 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:59:44 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ca82ac0a67d27abf-006477b61b-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 21:03:23 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:03:23.444 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj68 2023-05-31T21:03:23.445 DEBUG:boto:path=/pklxkd-61/obj68 2023-05-31T21:03:23.445 DEBUG:boto:auth_path=/pklxkd-61/obj68 2023-05-31T21:03:23.445 DEBUG:boto:path=/pklxkd-61/obj68?versionId=null 2023-05-31T21:03:23.446 DEBUG:boto:auth_path=/pklxkd-61/obj68?versionId=null 2023-05-31T21:03:23.446 DEBUG:boto:Method: GET 2023-05-31T21:03:23.446 DEBUG:boto:Path: /pklxkd-61/obj68?versionId=null 2023-05-31T21:03:23.447 DEBUG:boto:Data: 2023-05-31T21:03:23.447 DEBUG:boto:Headers: {} 2023-05-31T21:03:23.448 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T21:03:23.448 DEBUG:boto:Port: 8001 2023-05-31T21:03:23.448 DEBUG:boto:Params: {} 2023-05-31T21:03:23.449 DEBUG:boto:Token: None 2023-05-31T21:03:23.449 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 21:03:23 GMT /pklxkd-61/obj68?versionId=null 2023-05-31T21:03:23.450 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:ieHdN8xZ7ZmWgiAXKUxVHAPx5DY= 2023-05-31T21:03:23.450 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 21:03:23 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:ieHdN8xZ7ZmWgiAXKUxVHAPx5DY=', 'Content-Length': '0'} 2023-05-31T21:03:23.452 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:59:44 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000088714ffc6cb1abfc-006477b61b-1f4e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 21:03:23 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:03:23.453 DEBUG:boto:path=/pklxkd-61/obj68 2023-05-31T21:03:23.453 DEBUG:boto:auth_path=/pklxkd-61/obj68 2023-05-31T21:03:23.454 DEBUG:boto:path=/pklxkd-61/obj68?versionId=null 2023-05-31T21:03:23.454 DEBUG:boto:auth_path=/pklxkd-61/obj68?versionId=null 2023-05-31T21:03:23.454 DEBUG:boto:Method: GET 2023-05-31T21:03:23.455 DEBUG:boto:Path: /pklxkd-61/obj68?versionId=null 2023-05-31T21:03:23.455 DEBUG:boto:Data: 2023-05-31T21:03:23.455 DEBUG:boto:Headers: {} 2023-05-31T21:03:23.456 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T21:03:23.456 DEBUG:boto:Port: 8000 2023-05-31T21:03:23.457 DEBUG:boto:Params: {} 2023-05-31T21:03:23.457 DEBUG:boto:Token: None 2023-05-31T21:03:23.458 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 21:03:23 GMT /pklxkd-61/obj68?versionId=null 2023-05-31T21:03:23.458 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:ieHdN8xZ7ZmWgiAXKUxVHAPx5DY= 2023-05-31T21:03:23.459 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 21:03:23 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:ieHdN8xZ7ZmWgiAXKUxVHAPx5DY=', 'Content-Length': '0'} 2023-05-31T21:03:23.460 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:59:44 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000058eb6164a6e23800-006477b61b-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 21:03:23 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:03:23.461 DEBUG:boto:path=/pklxkd-61/ 2023-05-31T21:03:23.461 DEBUG:boto:auth_path=/pklxkd-61/ 2023-05-31T21:03:23.462 DEBUG:boto:path=/pklxkd-61/?prefix=obj68 2023-05-31T21:03:23.462 DEBUG:boto:auth_path=/pklxkd-61/?prefix=obj68 2023-05-31T21:03:23.463 DEBUG:boto:Method: GET 2023-05-31T21:03:23.463 DEBUG:boto:Path: /pklxkd-61/?prefix=obj68 2023-05-31T21:03:23.463 DEBUG:boto:Data: 2023-05-31T21:03:23.464 DEBUG:boto:Headers: {} 2023-05-31T21:03:23.464 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T21:03:23.464 DEBUG:boto:Port: 8001 2023-05-31T21:03:23.465 DEBUG:boto:Params: {} 2023-05-31T21:03:23.465 DEBUG:boto:Token: None 2023-05-31T21:03:23.466 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 21:03:23 GMT /pklxkd-61/ 2023-05-31T21:03:23.466 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:7YkKDTB+91bWua1TWmlOlWHvc2U= 2023-05-31T21:03:23.466 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 21:03:23 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:7YkKDTB+91bWua1TWmlOlWHvc2U=', 'Content-Length': '0'} 2023-05-31T21:03:23.469 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000218643b1e5740b7d-006477b61b-1f4e-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 21:03:23 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:03:23.516 DEBUG:boto:b'pklxkd-61obj681000falseobj682023-05-31T20:59:44.894Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T21:03:23.518 DEBUG:boto:path=/pklxkd-61/ 2023-05-31T21:03:23.518 DEBUG:boto:auth_path=/pklxkd-61/ 2023-05-31T21:03:23.519 DEBUG:boto:path=/pklxkd-61/?prefix=obj68 2023-05-31T21:03:23.519 DEBUG:boto:auth_path=/pklxkd-61/?prefix=obj68 2023-05-31T21:03:23.520 DEBUG:boto:Method: GET 2023-05-31T21:03:23.520 DEBUG:boto:Path: /pklxkd-61/?prefix=obj68 2023-05-31T21:03:23.521 DEBUG:boto:Data: 2023-05-31T21:03:23.521 DEBUG:boto:Headers: {} 2023-05-31T21:03:23.521 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T21:03:23.522 DEBUG:boto:Port: 8000 2023-05-31T21:03:23.522 DEBUG:boto:Params: {} 2023-05-31T21:03:23.523 DEBUG:boto:Token: None 2023-05-31T21:03:23.523 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 21:03:23 GMT /pklxkd-61/ 2023-05-31T21:03:23.523 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:7YkKDTB+91bWua1TWmlOlWHvc2U= 2023-05-31T21:03:23.524 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 21:03:23 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:7YkKDTB+91bWua1TWmlOlWHvc2U=', 'Content-Length': '0'} 2023-05-31T21:03:23.528 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a2581f740594273e-006477b61b-1630-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 21:03:23 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:03:23.529 DEBUG:boto:b'pklxkd-61obj681000falseobj682023-05-31T20:59:44.894Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T21:03:23.529 DEBUG:boto:path=/pklxkd-61/obj68 2023-05-31T21:03:23.530 DEBUG:boto:auth_path=/pklxkd-61/obj68 2023-05-31T21:03:23.530 DEBUG:boto:Method: HEAD 2023-05-31T21:03:23.530 DEBUG:boto:Path: /pklxkd-61/obj68 2023-05-31T21:03:23.531 DEBUG:boto:Data: 2023-05-31T21:03:23.531 DEBUG:boto:Headers: {} 2023-05-31T21:03:23.531 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T21:03:23.532 DEBUG:boto:Port: 8001 2023-05-31T21:03:23.532 DEBUG:boto:Params: {} 2023-05-31T21:03:23.532 DEBUG:boto:Token: None 2023-05-31T21:03:23.533 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 21:03:23 GMT /pklxkd-61/obj68 2023-05-31T21:03:23.533 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:6BSWpscN80dTEQeA9MlzGZe5rAA= 2023-05-31T21:03:23.533 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 21:03:23 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:6BSWpscN80dTEQeA9MlzGZe5rAA=', 'Content-Length': '0'} 2023-05-31T21:03:23.535 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:59:44 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000675b01f0f24e94ec-006477b61b-1f4e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 21:03:23 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:03:23.536 DEBUG:boto:path=/pklxkd-61/obj68 2023-05-31T21:03:23.536 DEBUG:boto:auth_path=/pklxkd-61/obj68 2023-05-31T21:03:23.537 DEBUG:boto:Method: HEAD 2023-05-31T21:03:23.537 DEBUG:boto:Path: /pklxkd-61/obj68 2023-05-31T21:03:23.538 DEBUG:boto:Data: 2023-05-31T21:03:23.538 DEBUG:boto:Headers: {} 2023-05-31T21:03:23.539 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T21:03:23.539 DEBUG:boto:Port: 8000 2023-05-31T21:03:23.539 DEBUG:boto:Params: {} 2023-05-31T21:03:23.540 DEBUG:boto:Token: None 2023-05-31T21:03:23.540 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 21:03:23 GMT /pklxkd-61/obj68 2023-05-31T21:03:23.540 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:6BSWpscN80dTEQeA9MlzGZe5rAA= 2023-05-31T21:03:23.541 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 21:03:23 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:6BSWpscN80dTEQeA9MlzGZe5rAA=', 'Content-Length': '0'} 2023-05-31T21:03:23.542 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:59:44 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006f9ba7ed69eecde0-006477b61b-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 21:03:23 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:03:23.543 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj68 2023-05-31T21:03:23.543 DEBUG:boto:path=/pklxkd-61/obj68 2023-05-31T21:03:23.544 DEBUG:boto:auth_path=/pklxkd-61/obj68 2023-05-31T21:03:23.544 DEBUG:boto:Method: GET 2023-05-31T21:03:23.545 DEBUG:boto:Path: /pklxkd-61/obj68 2023-05-31T21:03:23.545 DEBUG:boto:Data: 2023-05-31T21:03:23.545 DEBUG:boto:Headers: {} 2023-05-31T21:03:23.546 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T21:03:23.546 DEBUG:boto:Port: 8001 2023-05-31T21:03:23.546 DEBUG:boto:Params: {} 2023-05-31T21:03:23.547 DEBUG:boto:Token: None 2023-05-31T21:03:23.547 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 21:03:23 GMT /pklxkd-61/obj68 2023-05-31T21:03:23.548 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:451NaX/zgklifcZ8DcvfJAHvt7M= 2023-05-31T21:03:23.548 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 21:03:23 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:451NaX/zgklifcZ8DcvfJAHvt7M=', 'Content-Length': '0'} 2023-05-31T21:03:23.550 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:59:44 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a8ed2f1cb3adee33-006477b61b-1f4e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 21:03:23 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:03:23.551 DEBUG:boto:path=/pklxkd-61/obj68 2023-05-31T21:03:23.551 DEBUG:boto:auth_path=/pklxkd-61/obj68 2023-05-31T21:03:23.552 DEBUG:boto:Method: GET 2023-05-31T21:03:23.552 DEBUG:boto:Path: /pklxkd-61/obj68 2023-05-31T21:03:23.552 DEBUG:boto:Data: 2023-05-31T21:03:23.553 DEBUG:boto:Headers: {} 2023-05-31T21:03:23.553 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T21:03:23.554 DEBUG:boto:Port: 8000 2023-05-31T21:03:23.554 DEBUG:boto:Params: {} 2023-05-31T21:03:23.554 DEBUG:boto:Token: None 2023-05-31T21:03:23.555 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 21:03:23 GMT /pklxkd-61/obj68 2023-05-31T21:03:23.555 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:451NaX/zgklifcZ8DcvfJAHvt7M= 2023-05-31T21:03:23.555 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 21:03:23 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:451NaX/zgklifcZ8DcvfJAHvt7M=', 'Content-Length': '0'} 2023-05-31T21:03:23.557 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:59:44 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000056950d96cd0b920e-006477b61b-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 21:03:23 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:03:23.600 DEBUG:boto:path=/pklxkd-61/ 2023-05-31T21:03:23.601 DEBUG:boto:auth_path=/pklxkd-61/ 2023-05-31T21:03:23.601 DEBUG:boto:path=/pklxkd-61/?prefix=obj68 2023-05-31T21:03:23.601 DEBUG:boto:auth_path=/pklxkd-61/?prefix=obj68 2023-05-31T21:03:23.602 DEBUG:boto:Method: GET 2023-05-31T21:03:23.602 DEBUG:boto:Path: /pklxkd-61/?prefix=obj68 2023-05-31T21:03:23.602 DEBUG:boto:Data: 2023-05-31T21:03:23.603 DEBUG:boto:Headers: {} 2023-05-31T21:03:23.603 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T21:03:23.603 DEBUG:boto:Port: 8001 2023-05-31T21:03:23.604 DEBUG:boto:Params: {} 2023-05-31T21:03:23.604 DEBUG:boto:Token: None 2023-05-31T21:03:23.605 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 21:03:23 GMT /pklxkd-61/ 2023-05-31T21:03:23.605 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:7YkKDTB+91bWua1TWmlOlWHvc2U= 2023-05-31T21:03:23.605 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 21:03:23 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:7YkKDTB+91bWua1TWmlOlWHvc2U=', 'Content-Length': '0'} 2023-05-31T21:03:23.608 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c82bb23f38db2105-006477b61b-1f4e-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 21:03:23 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:03:23.608 DEBUG:boto:b'pklxkd-61obj681000falseobj682023-05-31T20:59:44.894Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T21:03:23.609 DEBUG:boto:path=/pklxkd-61/ 2023-05-31T21:03:23.609 DEBUG:boto:auth_path=/pklxkd-61/ 2023-05-31T21:03:23.610 DEBUG:boto:path=/pklxkd-61/?prefix=obj68 2023-05-31T21:03:23.610 DEBUG:boto:auth_path=/pklxkd-61/?prefix=obj68 2023-05-31T21:03:23.611 DEBUG:boto:Method: GET 2023-05-31T21:03:23.611 DEBUG:boto:Path: /pklxkd-61/?prefix=obj68 2023-05-31T21:03:23.611 DEBUG:boto:Data: 2023-05-31T21:03:23.612 DEBUG:boto:Headers: {} 2023-05-31T21:03:23.612 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T21:03:23.612 DEBUG:boto:Port: 8000 2023-05-31T21:03:23.613 DEBUG:boto:Params: {} 2023-05-31T21:03:23.613 DEBUG:boto:Token: None 2023-05-31T21:03:23.614 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 21:03:23 GMT /pklxkd-61/ 2023-05-31T21:03:23.614 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:7YkKDTB+91bWua1TWmlOlWHvc2U= 2023-05-31T21:03:23.614 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 21:03:23 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:7YkKDTB+91bWua1TWmlOlWHvc2U=', 'Content-Length': '0'} 2023-05-31T21:03:23.618 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000002e802a7286c58f97-006477b61b-1630-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 21:03:23 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:03:23.619 DEBUG:boto:b'pklxkd-61obj681000falseobj682023-05-31T20:59:44.894Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T21:03:23.619 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj69 2023-05-31T21:03:23.620 DEBUG:boto:path=/pklxkd-61/obj69 2023-05-31T21:03:23.620 DEBUG:boto:auth_path=/pklxkd-61/obj69 2023-05-31T21:03:23.621 DEBUG:boto:path=/pklxkd-61/obj69?versionId=null 2023-05-31T21:03:23.621 DEBUG:boto:auth_path=/pklxkd-61/obj69?versionId=null 2023-05-31T21:03:23.621 DEBUG:boto:Method: GET 2023-05-31T21:03:23.622 DEBUG:boto:Path: /pklxkd-61/obj69?versionId=null 2023-05-31T21:03:23.622 DEBUG:boto:Data: 2023-05-31T21:03:23.622 DEBUG:boto:Headers: {} 2023-05-31T21:03:23.623 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T21:03:23.623 DEBUG:boto:Port: 8001 2023-05-31T21:03:23.623 DEBUG:boto:Params: {} 2023-05-31T21:03:23.624 DEBUG:boto:Token: None 2023-05-31T21:03:23.624 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 21:03:23 GMT /pklxkd-61/obj69?versionId=null 2023-05-31T21:03:23.625 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:9adKwCSq11OxXuYhOFLgf69HGO8= 2023-05-31T21:03:23.625 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 21:03:23 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:9adKwCSq11OxXuYhOFLgf69HGO8=', 'Content-Length': '0'} 2023-05-31T21:03:23.627 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:59:45 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000dbf65785a40250d9-006477b61b-1f4e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 21:03:23 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:03:23.628 DEBUG:boto:path=/pklxkd-61/obj69 2023-05-31T21:03:23.628 DEBUG:boto:auth_path=/pklxkd-61/obj69 2023-05-31T21:03:23.628 DEBUG:boto:path=/pklxkd-61/obj69?versionId=null 2023-05-31T21:03:23.629 DEBUG:boto:auth_path=/pklxkd-61/obj69?versionId=null 2023-05-31T21:03:23.629 DEBUG:boto:Method: GET 2023-05-31T21:03:23.630 DEBUG:boto:Path: /pklxkd-61/obj69?versionId=null 2023-05-31T21:03:23.630 DEBUG:boto:Data: 2023-05-31T21:03:23.630 DEBUG:boto:Headers: {} 2023-05-31T21:03:23.631 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T21:03:23.631 DEBUG:boto:Port: 8000 2023-05-31T21:03:23.631 DEBUG:boto:Params: {} 2023-05-31T21:03:23.632 DEBUG:boto:Token: None 2023-05-31T21:03:23.632 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 21:03:23 GMT /pklxkd-61/obj69?versionId=null 2023-05-31T21:03:23.633 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:9adKwCSq11OxXuYhOFLgf69HGO8= 2023-05-31T21:03:23.633 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 21:03:23 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:9adKwCSq11OxXuYhOFLgf69HGO8=', 'Content-Length': '0'} 2023-05-31T21:03:23.635 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:59:45 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000004c94f7565da1d60-006477b61b-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 21:03:23 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:03:23.636 DEBUG:boto:path=/pklxkd-61/obj69 2023-05-31T21:03:23.636 DEBUG:boto:auth_path=/pklxkd-61/obj69 2023-05-31T21:03:23.636 DEBUG:boto:path=/pklxkd-61/obj69?versionId=null 2023-05-31T21:03:23.637 DEBUG:boto:auth_path=/pklxkd-61/obj69?versionId=null 2023-05-31T21:03:23.637 DEBUG:boto:Method: HEAD 2023-05-31T21:03:23.637 DEBUG:boto:Path: /pklxkd-61/obj69?versionId=null 2023-05-31T21:03:23.638 DEBUG:boto:Data: 2023-05-31T21:03:23.638 DEBUG:boto:Headers: {} 2023-05-31T21:03:23.638 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T21:03:23.639 DEBUG:boto:Port: 8001 2023-05-31T21:03:23.639 DEBUG:boto:Params: {} 2023-05-31T21:03:23.639 DEBUG:boto:Token: None 2023-05-31T21:03:23.640 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 21:03:23 GMT /pklxkd-61/obj69?versionId=null 2023-05-31T21:03:23.640 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:PgumC7X9BuxDhmJQDUEOsDdpdxE= 2023-05-31T21:03:23.641 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 21:03:23 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:PgumC7X9BuxDhmJQDUEOsDdpdxE=', 'Content-Length': '0'} 2023-05-31T21:03:23.642 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:59:45 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f8b6e1c1fa675954-006477b61b-1f4e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 21:03:23 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:03:23.643 DEBUG:boto:path=/pklxkd-61/obj69 2023-05-31T21:03:23.643 DEBUG:boto:auth_path=/pklxkd-61/obj69 2023-05-31T21:03:23.644 DEBUG:boto:path=/pklxkd-61/obj69?versionId=null 2023-05-31T21:03:23.644 DEBUG:boto:auth_path=/pklxkd-61/obj69?versionId=null 2023-05-31T21:03:23.644 DEBUG:boto:Method: HEAD 2023-05-31T21:03:23.645 DEBUG:boto:Path: /pklxkd-61/obj69?versionId=null 2023-05-31T21:03:23.645 DEBUG:boto:Data: 2023-05-31T21:03:23.645 DEBUG:boto:Headers: {} 2023-05-31T21:03:23.646 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T21:03:23.646 DEBUG:boto:Port: 8000 2023-05-31T21:03:23.646 DEBUG:boto:Params: {} 2023-05-31T21:03:23.647 DEBUG:boto:Token: None 2023-05-31T21:03:23.647 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 21:03:23 GMT /pklxkd-61/obj69?versionId=null 2023-05-31T21:03:23.648 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:PgumC7X9BuxDhmJQDUEOsDdpdxE= 2023-05-31T21:03:23.648 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 21:03:23 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:PgumC7X9BuxDhmJQDUEOsDdpdxE=', 'Content-Length': '0'} 2023-05-31T21:03:23.650 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:59:45 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006d60505ee6f990e5-006477b61b-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 21:03:23 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:03:23.650 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj69 2023-05-31T21:03:23.651 DEBUG:boto:path=/pklxkd-61/obj69 2023-05-31T21:03:23.651 DEBUG:boto:auth_path=/pklxkd-61/obj69 2023-05-31T21:03:23.651 DEBUG:boto:path=/pklxkd-61/obj69?versionId=null 2023-05-31T21:03:23.652 DEBUG:boto:auth_path=/pklxkd-61/obj69?versionId=null 2023-05-31T21:03:23.652 DEBUG:boto:Method: GET 2023-05-31T21:03:23.653 DEBUG:boto:Path: /pklxkd-61/obj69?versionId=null 2023-05-31T21:03:23.653 DEBUG:boto:Data: 2023-05-31T21:03:23.653 DEBUG:boto:Headers: {} 2023-05-31T21:03:23.654 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T21:03:23.654 DEBUG:boto:Port: 8001 2023-05-31T21:03:23.654 DEBUG:boto:Params: {} 2023-05-31T21:03:23.655 DEBUG:boto:Token: None 2023-05-31T21:03:23.655 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 21:03:23 GMT /pklxkd-61/obj69?versionId=null 2023-05-31T21:03:23.656 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:9adKwCSq11OxXuYhOFLgf69HGO8= 2023-05-31T21:03:23.656 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 21:03:23 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:9adKwCSq11OxXuYhOFLgf69HGO8=', 'Content-Length': '0'} 2023-05-31T21:03:23.658 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:59:45 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ee08d84cd541b38e-006477b61b-1f4e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 21:03:23 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:03:23.658 DEBUG:boto:path=/pklxkd-61/obj69 2023-05-31T21:03:23.659 DEBUG:boto:auth_path=/pklxkd-61/obj69 2023-05-31T21:03:23.659 DEBUG:boto:path=/pklxkd-61/obj69?versionId=null 2023-05-31T21:03:23.660 DEBUG:boto:auth_path=/pklxkd-61/obj69?versionId=null 2023-05-31T21:03:23.660 DEBUG:boto:Method: GET 2023-05-31T21:03:23.660 DEBUG:boto:Path: /pklxkd-61/obj69?versionId=null 2023-05-31T21:03:23.661 DEBUG:boto:Data: 2023-05-31T21:03:23.661 DEBUG:boto:Headers: {} 2023-05-31T21:03:23.661 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T21:03:23.662 DEBUG:boto:Port: 8000 2023-05-31T21:03:23.662 DEBUG:boto:Params: {} 2023-05-31T21:03:23.663 DEBUG:boto:Token: None 2023-05-31T21:03:23.663 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 21:03:23 GMT /pklxkd-61/obj69?versionId=null 2023-05-31T21:03:23.663 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:9adKwCSq11OxXuYhOFLgf69HGO8= 2023-05-31T21:03:23.664 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 21:03:23 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:9adKwCSq11OxXuYhOFLgf69HGO8=', 'Content-Length': '0'} 2023-05-31T21:03:23.666 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:59:45 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005e4f464a53e3d85e-006477b61b-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 21:03:23 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:03:23.666 DEBUG:boto:path=/pklxkd-61/ 2023-05-31T21:03:23.667 DEBUG:boto:auth_path=/pklxkd-61/ 2023-05-31T21:03:23.667 DEBUG:boto:path=/pklxkd-61/?prefix=obj69 2023-05-31T21:03:23.667 DEBUG:boto:auth_path=/pklxkd-61/?prefix=obj69 2023-05-31T21:03:23.668 DEBUG:boto:Method: GET 2023-05-31T21:03:23.668 DEBUG:boto:Path: /pklxkd-61/?prefix=obj69 2023-05-31T21:03:23.669 DEBUG:boto:Data: 2023-05-31T21:03:23.669 DEBUG:boto:Headers: {} 2023-05-31T21:03:23.669 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T21:03:23.670 DEBUG:boto:Port: 8001 2023-05-31T21:03:23.670 DEBUG:boto:Params: {} 2023-05-31T21:03:23.670 DEBUG:boto:Token: None 2023-05-31T21:03:23.671 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 21:03:23 GMT /pklxkd-61/ 2023-05-31T21:03:23.671 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:7YkKDTB+91bWua1TWmlOlWHvc2U= 2023-05-31T21:03:23.672 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 21:03:23 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:7YkKDTB+91bWua1TWmlOlWHvc2U=', 'Content-Length': '0'} 2023-05-31T21:03:23.674 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000fecac320e87a2206-006477b61b-1f4e-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 21:03:23 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:03:23.675 DEBUG:boto:b'pklxkd-61obj691000falseobj692023-05-31T20:59:45.559Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T21:03:23.675 DEBUG:boto:path=/pklxkd-61/ 2023-05-31T21:03:23.676 DEBUG:boto:auth_path=/pklxkd-61/ 2023-05-31T21:03:23.676 DEBUG:boto:path=/pklxkd-61/?prefix=obj69 2023-05-31T21:03:23.676 DEBUG:boto:auth_path=/pklxkd-61/?prefix=obj69 2023-05-31T21:03:23.677 DEBUG:boto:Method: GET 2023-05-31T21:03:23.677 DEBUG:boto:Path: /pklxkd-61/?prefix=obj69 2023-05-31T21:03:23.678 DEBUG:boto:Data: 2023-05-31T21:03:23.678 DEBUG:boto:Headers: {} 2023-05-31T21:03:23.678 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T21:03:23.679 DEBUG:boto:Port: 8000 2023-05-31T21:03:23.679 DEBUG:boto:Params: {} 2023-05-31T21:03:23.679 DEBUG:boto:Token: None 2023-05-31T21:03:23.680 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 21:03:23 GMT /pklxkd-61/ 2023-05-31T21:03:23.680 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:7YkKDTB+91bWua1TWmlOlWHvc2U= 2023-05-31T21:03:23.681 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 21:03:23 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:7YkKDTB+91bWua1TWmlOlWHvc2U=', 'Content-Length': '0'} 2023-05-31T21:03:23.685 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ca7a4d339cdf0b90-006477b61b-1630-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 21:03:23 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:03:23.685 DEBUG:boto:b'pklxkd-61obj691000falseobj692023-05-31T20:59:45.559Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T21:03:23.686 DEBUG:boto:path=/pklxkd-61/obj69 2023-05-31T21:03:23.686 DEBUG:boto:auth_path=/pklxkd-61/obj69 2023-05-31T21:03:23.687 DEBUG:boto:Method: HEAD 2023-05-31T21:03:23.687 DEBUG:boto:Path: /pklxkd-61/obj69 2023-05-31T21:03:23.687 DEBUG:boto:Data: 2023-05-31T21:03:23.688 DEBUG:boto:Headers: {} 2023-05-31T21:03:23.688 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T21:03:23.688 DEBUG:boto:Port: 8001 2023-05-31T21:03:23.689 DEBUG:boto:Params: {} 2023-05-31T21:03:23.689 DEBUG:boto:Token: None 2023-05-31T21:03:23.689 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 21:03:23 GMT /pklxkd-61/obj69 2023-05-31T21:03:23.690 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:6S8YRRwNeKJnmYGUe+4YawAWmn8= 2023-05-31T21:03:23.690 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 21:03:23 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:6S8YRRwNeKJnmYGUe+4YawAWmn8=', 'Content-Length': '0'} 2023-05-31T21:03:23.692 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:59:45 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b9d1bf968bae30cb-006477b61b-1f4e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 21:03:23 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:03:23.692 DEBUG:boto:path=/pklxkd-61/obj69 2023-05-31T21:03:23.693 DEBUG:boto:auth_path=/pklxkd-61/obj69 2023-05-31T21:03:23.693 DEBUG:boto:Method: HEAD 2023-05-31T21:03:23.694 DEBUG:boto:Path: /pklxkd-61/obj69 2023-05-31T21:03:23.694 DEBUG:boto:Data: 2023-05-31T21:03:23.694 DEBUG:boto:Headers: {} 2023-05-31T21:03:23.695 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T21:03:23.695 DEBUG:boto:Port: 8000 2023-05-31T21:03:23.695 DEBUG:boto:Params: {} 2023-05-31T21:03:23.696 DEBUG:boto:Token: None 2023-05-31T21:03:23.696 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 21:03:23 GMT /pklxkd-61/obj69 2023-05-31T21:03:23.696 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:6S8YRRwNeKJnmYGUe+4YawAWmn8= 2023-05-31T21:03:23.697 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 21:03:23 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:6S8YRRwNeKJnmYGUe+4YawAWmn8=', 'Content-Length': '0'} 2023-05-31T21:03:23.698 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:59:45 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000cf40bb42088c4c27-006477b61b-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 21:03:23 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:03:23.699 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj69 2023-05-31T21:03:23.699 DEBUG:boto:path=/pklxkd-61/obj69 2023-05-31T21:03:23.700 DEBUG:boto:auth_path=/pklxkd-61/obj69 2023-05-31T21:03:23.700 DEBUG:boto:Method: GET 2023-05-31T21:03:23.701 DEBUG:boto:Path: /pklxkd-61/obj69 2023-05-31T21:03:23.701 DEBUG:boto:Data: 2023-05-31T21:03:23.701 DEBUG:boto:Headers: {} 2023-05-31T21:03:23.702 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T21:03:23.702 DEBUG:boto:Port: 8001 2023-05-31T21:03:23.702 DEBUG:boto:Params: {} 2023-05-31T21:03:23.703 DEBUG:boto:Token: None 2023-05-31T21:03:23.703 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 21:03:23 GMT /pklxkd-61/obj69 2023-05-31T21:03:23.704 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:5Sg88jsWQARvnHKppqEPwcaXHiw= 2023-05-31T21:03:23.704 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 21:03:23 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:5Sg88jsWQARvnHKppqEPwcaXHiw=', 'Content-Length': '0'} 2023-05-31T21:03:23.706 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:59:45 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e35ed54ac0928264-006477b61b-1f4e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 21:03:23 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:03:23.706 DEBUG:boto:path=/pklxkd-61/obj69 2023-05-31T21:03:23.707 DEBUG:boto:auth_path=/pklxkd-61/obj69 2023-05-31T21:03:23.707 DEBUG:boto:Method: GET 2023-05-31T21:03:23.708 DEBUG:boto:Path: /pklxkd-61/obj69 2023-05-31T21:03:23.708 DEBUG:boto:Data: 2023-05-31T21:03:23.708 DEBUG:boto:Headers: {} 2023-05-31T21:03:23.709 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T21:03:23.709 DEBUG:boto:Port: 8000 2023-05-31T21:03:23.709 DEBUG:boto:Params: {} 2023-05-31T21:03:23.710 DEBUG:boto:Token: None 2023-05-31T21:03:23.710 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 21:03:23 GMT /pklxkd-61/obj69 2023-05-31T21:03:23.711 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:5Sg88jsWQARvnHKppqEPwcaXHiw= 2023-05-31T21:03:23.711 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 21:03:23 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:5Sg88jsWQARvnHKppqEPwcaXHiw=', 'Content-Length': '0'} 2023-05-31T21:03:23.713 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:59:45 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003c2dfb5eeced077c-006477b61b-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 21:03:23 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:03:23.756 DEBUG:boto:path=/pklxkd-61/ 2023-05-31T21:03:23.757 DEBUG:boto:auth_path=/pklxkd-61/ 2023-05-31T21:03:23.757 DEBUG:boto:path=/pklxkd-61/?prefix=obj69 2023-05-31T21:03:23.758 DEBUG:boto:auth_path=/pklxkd-61/?prefix=obj69 2023-05-31T21:03:23.758 DEBUG:boto:Method: GET 2023-05-31T21:03:23.758 DEBUG:boto:Path: /pklxkd-61/?prefix=obj69 2023-05-31T21:03:23.759 DEBUG:boto:Data: 2023-05-31T21:03:23.759 DEBUG:boto:Headers: {} 2023-05-31T21:03:23.759 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T21:03:23.760 DEBUG:boto:Port: 8001 2023-05-31T21:03:23.760 DEBUG:boto:Params: {} 2023-05-31T21:03:23.761 DEBUG:boto:Token: None 2023-05-31T21:03:23.761 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 21:03:23 GMT /pklxkd-61/ 2023-05-31T21:03:23.761 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:7YkKDTB+91bWua1TWmlOlWHvc2U= 2023-05-31T21:03:23.762 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 21:03:23 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:7YkKDTB+91bWua1TWmlOlWHvc2U=', 'Content-Length': '0'} 2023-05-31T21:03:23.764 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ea19ef0170d09d32-006477b61b-1f4e-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 21:03:23 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:03:23.765 DEBUG:boto:b'pklxkd-61obj691000falseobj692023-05-31T20:59:45.559Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T21:03:23.765 DEBUG:boto:path=/pklxkd-61/ 2023-05-31T21:03:23.766 DEBUG:boto:auth_path=/pklxkd-61/ 2023-05-31T21:03:23.766 DEBUG:boto:path=/pklxkd-61/?prefix=obj69 2023-05-31T21:03:23.767 DEBUG:boto:auth_path=/pklxkd-61/?prefix=obj69 2023-05-31T21:03:23.767 DEBUG:boto:Method: GET 2023-05-31T21:03:23.767 DEBUG:boto:Path: /pklxkd-61/?prefix=obj69 2023-05-31T21:03:23.768 DEBUG:boto:Data: 2023-05-31T21:03:23.768 DEBUG:boto:Headers: {} 2023-05-31T21:03:23.768 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T21:03:23.769 DEBUG:boto:Port: 8000 2023-05-31T21:03:23.769 DEBUG:boto:Params: {} 2023-05-31T21:03:23.770 DEBUG:boto:Token: None 2023-05-31T21:03:23.770 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 21:03:23 GMT /pklxkd-61/ 2023-05-31T21:03:23.771 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:7YkKDTB+91bWua1TWmlOlWHvc2U= 2023-05-31T21:03:23.771 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 21:03:23 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:7YkKDTB+91bWua1TWmlOlWHvc2U=', 'Content-Length': '0'} 2023-05-31T21:03:23.776 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000007d0d304f658a6dfe-006477b61b-1630-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 21:03:23 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:03:23.777 DEBUG:boto:b'pklxkd-61obj691000falseobj692023-05-31T20:59:45.559Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T21:03:23.778 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj74 2023-05-31T21:03:23.779 DEBUG:boto:path=/pklxkd-61/obj74 2023-05-31T21:03:23.780 DEBUG:boto:auth_path=/pklxkd-61/obj74 2023-05-31T21:03:23.781 DEBUG:boto:path=/pklxkd-61/obj74?versionId=null 2023-05-31T21:03:23.781 DEBUG:boto:auth_path=/pklxkd-61/obj74?versionId=null 2023-05-31T21:03:23.782 DEBUG:boto:Method: GET 2023-05-31T21:03:23.782 DEBUG:boto:Path: /pklxkd-61/obj74?versionId=null 2023-05-31T21:03:23.783 DEBUG:boto:Data: 2023-05-31T21:03:23.783 DEBUG:boto:Headers: {} 2023-05-31T21:03:23.784 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T21:03:23.784 DEBUG:boto:Port: 8001 2023-05-31T21:03:23.784 DEBUG:boto:Params: {} 2023-05-31T21:03:23.785 DEBUG:boto:Token: None 2023-05-31T21:03:23.785 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 21:03:23 GMT /pklxkd-61/obj74?versionId=null 2023-05-31T21:03:23.786 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:pCRRTvoOKbLYhDrt/J/4ebRcUnI= 2023-05-31T21:03:23.786 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 21:03:23 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:pCRRTvoOKbLYhDrt/J/4ebRcUnI=', 'Content-Length': '0'} 2023-05-31T21:03:23.788 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:59:46 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b416931a1bcf996d-006477b61b-1f4e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 21:03:23 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:03:23.789 DEBUG:boto:path=/pklxkd-61/obj74 2023-05-31T21:03:23.789 DEBUG:boto:auth_path=/pklxkd-61/obj74 2023-05-31T21:03:23.790 DEBUG:boto:path=/pklxkd-61/obj74?versionId=null 2023-05-31T21:03:23.790 DEBUG:boto:auth_path=/pklxkd-61/obj74?versionId=null 2023-05-31T21:03:23.790 DEBUG:boto:Method: GET 2023-05-31T21:03:23.791 DEBUG:boto:Path: /pklxkd-61/obj74?versionId=null 2023-05-31T21:03:23.791 DEBUG:boto:Data: 2023-05-31T21:03:23.792 DEBUG:boto:Headers: {} 2023-05-31T21:03:23.792 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T21:03:23.792 DEBUG:boto:Port: 8000 2023-05-31T21:03:23.793 DEBUG:boto:Params: {} 2023-05-31T21:03:23.793 DEBUG:boto:Token: None 2023-05-31T21:03:23.794 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 21:03:23 GMT /pklxkd-61/obj74?versionId=null 2023-05-31T21:03:23.794 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:pCRRTvoOKbLYhDrt/J/4ebRcUnI= 2023-05-31T21:03:23.794 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 21:03:23 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:pCRRTvoOKbLYhDrt/J/4ebRcUnI=', 'Content-Length': '0'} 2023-05-31T21:03:23.796 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:59:46 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009274821e74459a15-006477b61b-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 21:03:23 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:03:23.797 DEBUG:boto:path=/pklxkd-61/obj74 2023-05-31T21:03:23.797 DEBUG:boto:auth_path=/pklxkd-61/obj74 2023-05-31T21:03:23.798 DEBUG:boto:path=/pklxkd-61/obj74?versionId=null 2023-05-31T21:03:23.798 DEBUG:boto:auth_path=/pklxkd-61/obj74?versionId=null 2023-05-31T21:03:23.798 DEBUG:boto:Method: HEAD 2023-05-31T21:03:23.799 DEBUG:boto:Path: /pklxkd-61/obj74?versionId=null 2023-05-31T21:03:23.799 DEBUG:boto:Data: 2023-05-31T21:03:23.799 DEBUG:boto:Headers: {} 2023-05-31T21:03:23.800 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T21:03:23.800 DEBUG:boto:Port: 8001 2023-05-31T21:03:23.800 DEBUG:boto:Params: {} 2023-05-31T21:03:23.801 DEBUG:boto:Token: None 2023-05-31T21:03:23.801 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 21:03:23 GMT /pklxkd-61/obj74?versionId=null 2023-05-31T21:03:23.802 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:Z9xk+KR9TxGXQ4PMYlJJznDthlI= 2023-05-31T21:03:23.802 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 21:03:23 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:Z9xk+KR9TxGXQ4PMYlJJznDthlI=', 'Content-Length': '0'} 2023-05-31T21:03:23.804 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:59:46 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000521e49723c582d6d-006477b61b-1f4e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 21:03:23 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:03:23.804 DEBUG:boto:path=/pklxkd-61/obj74 2023-05-31T21:03:23.805 DEBUG:boto:auth_path=/pklxkd-61/obj74 2023-05-31T21:03:23.805 DEBUG:boto:path=/pklxkd-61/obj74?versionId=null 2023-05-31T21:03:23.805 DEBUG:boto:auth_path=/pklxkd-61/obj74?versionId=null 2023-05-31T21:03:23.806 DEBUG:boto:Method: HEAD 2023-05-31T21:03:23.806 DEBUG:boto:Path: /pklxkd-61/obj74?versionId=null 2023-05-31T21:03:23.806 DEBUG:boto:Data: 2023-05-31T21:03:23.807 DEBUG:boto:Headers: {} 2023-05-31T21:03:23.807 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T21:03:23.807 DEBUG:boto:Port: 8000 2023-05-31T21:03:23.808 DEBUG:boto:Params: {} 2023-05-31T21:03:23.809 DEBUG:boto:Token: None 2023-05-31T21:03:23.810 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 21:03:23 GMT /pklxkd-61/obj74?versionId=null 2023-05-31T21:03:23.811 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:Z9xk+KR9TxGXQ4PMYlJJznDthlI= 2023-05-31T21:03:23.812 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 21:03:23 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:Z9xk+KR9TxGXQ4PMYlJJznDthlI=', 'Content-Length': '0'} 2023-05-31T21:03:23.814 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:59:46 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007fd7981dc72b1e0c-006477b61b-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 21:03:23 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:03:23.815 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj74 2023-05-31T21:03:23.816 DEBUG:boto:path=/pklxkd-61/obj74 2023-05-31T21:03:23.817 DEBUG:boto:auth_path=/pklxkd-61/obj74 2023-05-31T21:03:23.818 DEBUG:boto:path=/pklxkd-61/obj74?versionId=null 2023-05-31T21:03:23.818 DEBUG:boto:auth_path=/pklxkd-61/obj74?versionId=null 2023-05-31T21:03:23.819 DEBUG:boto:Method: GET 2023-05-31T21:03:23.819 DEBUG:boto:Path: /pklxkd-61/obj74?versionId=null 2023-05-31T21:03:23.819 DEBUG:boto:Data: 2023-05-31T21:03:23.820 DEBUG:boto:Headers: {} 2023-05-31T21:03:23.820 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T21:03:23.821 DEBUG:boto:Port: 8001 2023-05-31T21:03:23.821 DEBUG:boto:Params: {} 2023-05-31T21:03:23.821 DEBUG:boto:Token: None 2023-05-31T21:03:23.822 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 21:03:23 GMT /pklxkd-61/obj74?versionId=null 2023-05-31T21:03:23.822 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:pCRRTvoOKbLYhDrt/J/4ebRcUnI= 2023-05-31T21:03:23.823 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 21:03:23 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:pCRRTvoOKbLYhDrt/J/4ebRcUnI=', 'Content-Length': '0'} 2023-05-31T21:03:23.824 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:59:46 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000fa2ded0f519947bc-006477b61b-1f4e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 21:03:23 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:03:23.868 DEBUG:boto:path=/pklxkd-61/obj74 2023-05-31T21:03:23.869 DEBUG:boto:auth_path=/pklxkd-61/obj74 2023-05-31T21:03:23.869 DEBUG:boto:path=/pklxkd-61/obj74?versionId=null 2023-05-31T21:03:23.871 DEBUG:boto:auth_path=/pklxkd-61/obj74?versionId=null 2023-05-31T21:03:23.871 DEBUG:boto:Method: GET 2023-05-31T21:03:23.872 DEBUG:boto:Path: /pklxkd-61/obj74?versionId=null 2023-05-31T21:03:23.872 DEBUG:boto:Data: 2023-05-31T21:03:23.873 DEBUG:boto:Headers: {} 2023-05-31T21:03:23.873 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T21:03:23.873 DEBUG:boto:Port: 8000 2023-05-31T21:03:23.874 DEBUG:boto:Params: {} 2023-05-31T21:03:23.874 DEBUG:boto:Token: None 2023-05-31T21:03:23.875 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 21:03:23 GMT /pklxkd-61/obj74?versionId=null 2023-05-31T21:03:23.875 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:pCRRTvoOKbLYhDrt/J/4ebRcUnI= 2023-05-31T21:03:23.876 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 21:03:23 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:pCRRTvoOKbLYhDrt/J/4ebRcUnI=', 'Content-Length': '0'} 2023-05-31T21:03:23.878 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:59:46 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000980c2daaaee61bef-006477b61b-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 21:03:23 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:03:23.878 DEBUG:boto:path=/pklxkd-61/ 2023-05-31T21:03:23.879 DEBUG:boto:auth_path=/pklxkd-61/ 2023-05-31T21:03:23.879 DEBUG:boto:path=/pklxkd-61/?prefix=obj74 2023-05-31T21:03:23.880 DEBUG:boto:auth_path=/pklxkd-61/?prefix=obj74 2023-05-31T21:03:23.880 DEBUG:boto:Method: GET 2023-05-31T21:03:23.880 DEBUG:boto:Path: /pklxkd-61/?prefix=obj74 2023-05-31T21:03:23.881 DEBUG:boto:Data: 2023-05-31T21:03:23.881 DEBUG:boto:Headers: {} 2023-05-31T21:03:23.881 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T21:03:23.882 DEBUG:boto:Port: 8001 2023-05-31T21:03:23.882 DEBUG:boto:Params: {} 2023-05-31T21:03:23.883 DEBUG:boto:Token: None 2023-05-31T21:03:23.883 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 21:03:23 GMT /pklxkd-61/ 2023-05-31T21:03:23.883 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:7YkKDTB+91bWua1TWmlOlWHvc2U= 2023-05-31T21:03:23.884 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 21:03:23 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:7YkKDTB+91bWua1TWmlOlWHvc2U=', 'Content-Length': '0'} 2023-05-31T21:03:23.886 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000005bb44f3ac6ab7cb7-006477b61b-1f4e-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 21:03:23 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:03:23.887 DEBUG:boto:b'pklxkd-61obj741000falseobj742023-05-31T20:59:46.933Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T21:03:23.888 DEBUG:boto:path=/pklxkd-61/ 2023-05-31T21:03:23.888 DEBUG:boto:auth_path=/pklxkd-61/ 2023-05-31T21:03:23.888 DEBUG:boto:path=/pklxkd-61/?prefix=obj74 2023-05-31T21:03:23.889 DEBUG:boto:auth_path=/pklxkd-61/?prefix=obj74 2023-05-31T21:03:23.889 DEBUG:boto:Method: GET 2023-05-31T21:03:23.890 DEBUG:boto:Path: /pklxkd-61/?prefix=obj74 2023-05-31T21:03:23.890 DEBUG:boto:Data: 2023-05-31T21:03:23.890 DEBUG:boto:Headers: {} 2023-05-31T21:03:23.891 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T21:03:23.891 DEBUG:boto:Port: 8000 2023-05-31T21:03:23.892 DEBUG:boto:Params: {} 2023-05-31T21:03:23.892 DEBUG:boto:Token: None 2023-05-31T21:03:23.892 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 21:03:23 GMT /pklxkd-61/ 2023-05-31T21:03:23.893 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:7YkKDTB+91bWua1TWmlOlWHvc2U= 2023-05-31T21:03:23.893 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 21:03:23 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:7YkKDTB+91bWua1TWmlOlWHvc2U=', 'Content-Length': '0'} 2023-05-31T21:03:23.897 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000002efdfef50ee3c551-006477b61b-1630-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 21:03:23 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:03:23.898 DEBUG:boto:b'pklxkd-61obj741000falseobj742023-05-31T20:59:46.933Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T21:03:23.898 DEBUG:boto:path=/pklxkd-61/obj74 2023-05-31T21:03:23.899 DEBUG:boto:auth_path=/pklxkd-61/obj74 2023-05-31T21:03:23.899 DEBUG:boto:Method: HEAD 2023-05-31T21:03:23.899 DEBUG:boto:Path: /pklxkd-61/obj74 2023-05-31T21:03:23.900 DEBUG:boto:Data: 2023-05-31T21:03:23.900 DEBUG:boto:Headers: {} 2023-05-31T21:03:23.900 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T21:03:23.901 DEBUG:boto:Port: 8001 2023-05-31T21:03:23.901 DEBUG:boto:Params: {} 2023-05-31T21:03:23.901 DEBUG:boto:Token: None 2023-05-31T21:03:23.902 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 21:03:23 GMT /pklxkd-61/obj74 2023-05-31T21:03:23.902 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:fKRP+ar0quqE7e9Zxj4gulRVePk= 2023-05-31T21:03:23.903 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 21:03:23 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:fKRP+ar0quqE7e9Zxj4gulRVePk=', 'Content-Length': '0'} 2023-05-31T21:03:23.904 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:59:46 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000025b0b6bc52993a3-006477b61b-1f4e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 21:03:23 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:03:23.905 DEBUG:boto:path=/pklxkd-61/obj74 2023-05-31T21:03:23.905 DEBUG:boto:auth_path=/pklxkd-61/obj74 2023-05-31T21:03:23.906 DEBUG:boto:Method: HEAD 2023-05-31T21:03:23.906 DEBUG:boto:Path: /pklxkd-61/obj74 2023-05-31T21:03:23.907 DEBUG:boto:Data: 2023-05-31T21:03:23.907 DEBUG:boto:Headers: {} 2023-05-31T21:03:23.908 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T21:03:23.908 DEBUG:boto:Port: 8000 2023-05-31T21:03:23.908 DEBUG:boto:Params: {} 2023-05-31T21:03:23.909 DEBUG:boto:Token: None 2023-05-31T21:03:23.909 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 21:03:23 GMT /pklxkd-61/obj74 2023-05-31T21:03:23.910 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:fKRP+ar0quqE7e9Zxj4gulRVePk= 2023-05-31T21:03:23.910 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 21:03:23 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:fKRP+ar0quqE7e9Zxj4gulRVePk=', 'Content-Length': '0'} 2023-05-31T21:03:23.912 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:59:46 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a6e0cc85c5a62773-006477b61b-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 21:03:23 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:03:23.912 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj74 2023-05-31T21:03:23.913 DEBUG:boto:path=/pklxkd-61/obj74 2023-05-31T21:03:23.913 DEBUG:boto:auth_path=/pklxkd-61/obj74 2023-05-31T21:03:23.914 DEBUG:boto:Method: GET 2023-05-31T21:03:23.914 DEBUG:boto:Path: /pklxkd-61/obj74 2023-05-31T21:03:23.914 DEBUG:boto:Data: 2023-05-31T21:03:23.915 DEBUG:boto:Headers: {} 2023-05-31T21:03:23.915 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T21:03:23.915 DEBUG:boto:Port: 8001 2023-05-31T21:03:23.916 DEBUG:boto:Params: {} 2023-05-31T21:03:23.916 DEBUG:boto:Token: None 2023-05-31T21:03:23.917 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 21:03:23 GMT /pklxkd-61/obj74 2023-05-31T21:03:23.917 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:Bb8J72xdij/149cAD8ebdL9b3c8= 2023-05-31T21:03:23.917 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 21:03:23 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:Bb8J72xdij/149cAD8ebdL9b3c8=', 'Content-Length': '0'} 2023-05-31T21:03:23.919 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:59:46 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000077aacab87d1cc5d2-006477b61b-1f4e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 21:03:23 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:03:23.920 DEBUG:boto:path=/pklxkd-61/obj74 2023-05-31T21:03:23.920 DEBUG:boto:auth_path=/pklxkd-61/obj74 2023-05-31T21:03:23.921 DEBUG:boto:Method: GET 2023-05-31T21:03:23.921 DEBUG:boto:Path: /pklxkd-61/obj74 2023-05-31T21:03:23.921 DEBUG:boto:Data: 2023-05-31T21:03:23.922 DEBUG:boto:Headers: {} 2023-05-31T21:03:23.922 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T21:03:23.923 DEBUG:boto:Port: 8000 2023-05-31T21:03:23.923 DEBUG:boto:Params: {} 2023-05-31T21:03:23.923 DEBUG:boto:Token: None 2023-05-31T21:03:23.924 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 21:03:23 GMT /pklxkd-61/obj74 2023-05-31T21:03:23.924 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:Bb8J72xdij/149cAD8ebdL9b3c8= 2023-05-31T21:03:23.925 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 21:03:23 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:Bb8J72xdij/149cAD8ebdL9b3c8=', 'Content-Length': '0'} 2023-05-31T21:03:23.927 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:59:46 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000778044f5ae5f93ee-006477b61b-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 21:03:23 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:03:23.927 DEBUG:boto:path=/pklxkd-61/ 2023-05-31T21:03:23.928 DEBUG:boto:auth_path=/pklxkd-61/ 2023-05-31T21:03:23.928 DEBUG:boto:path=/pklxkd-61/?prefix=obj74 2023-05-31T21:03:23.928 DEBUG:boto:auth_path=/pklxkd-61/?prefix=obj74 2023-05-31T21:03:23.929 DEBUG:boto:Method: GET 2023-05-31T21:03:23.929 DEBUG:boto:Path: /pklxkd-61/?prefix=obj74 2023-05-31T21:03:23.929 DEBUG:boto:Data: 2023-05-31T21:03:23.930 DEBUG:boto:Headers: {} 2023-05-31T21:03:23.930 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T21:03:23.931 DEBUG:boto:Port: 8001 2023-05-31T21:03:23.931 DEBUG:boto:Params: {} 2023-05-31T21:03:23.931 DEBUG:boto:Token: None 2023-05-31T21:03:23.932 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 21:03:23 GMT /pklxkd-61/ 2023-05-31T21:03:23.932 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:7YkKDTB+91bWua1TWmlOlWHvc2U= 2023-05-31T21:03:23.933 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 21:03:23 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:7YkKDTB+91bWua1TWmlOlWHvc2U=', 'Content-Length': '0'} 2023-05-31T21:03:23.935 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a76016848d4f08ac-006477b61b-1f4e-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 21:03:23 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:03:23.935 DEBUG:boto:b'pklxkd-61obj741000falseobj742023-05-31T20:59:46.933Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T21:03:23.936 DEBUG:boto:path=/pklxkd-61/ 2023-05-31T21:03:23.936 DEBUG:boto:auth_path=/pklxkd-61/ 2023-05-31T21:03:23.937 DEBUG:boto:path=/pklxkd-61/?prefix=obj74 2023-05-31T21:03:23.937 DEBUG:boto:auth_path=/pklxkd-61/?prefix=obj74 2023-05-31T21:03:23.937 DEBUG:boto:Method: GET 2023-05-31T21:03:23.938 DEBUG:boto:Path: /pklxkd-61/?prefix=obj74 2023-05-31T21:03:23.938 DEBUG:boto:Data: 2023-05-31T21:03:23.939 DEBUG:boto:Headers: {} 2023-05-31T21:03:23.939 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T21:03:23.939 DEBUG:boto:Port: 8000 2023-05-31T21:03:23.940 DEBUG:boto:Params: {} 2023-05-31T21:03:23.940 DEBUG:boto:Token: None 2023-05-31T21:03:23.940 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 21:03:23 GMT /pklxkd-61/ 2023-05-31T21:03:23.941 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:7YkKDTB+91bWua1TWmlOlWHvc2U= 2023-05-31T21:03:23.941 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 21:03:23 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:7YkKDTB+91bWua1TWmlOlWHvc2U=', 'Content-Length': '0'} 2023-05-31T21:03:23.945 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a7da05b215f274d0-006477b61b-1630-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 21:03:23 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:03:23.946 DEBUG:boto:b'pklxkd-61obj741000falseobj742023-05-31T20:59:46.933Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T21:03:23.947 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj76 2023-05-31T21:03:23.947 DEBUG:boto:path=/pklxkd-61/obj76 2023-05-31T21:03:23.947 DEBUG:boto:auth_path=/pklxkd-61/obj76 2023-05-31T21:03:23.948 DEBUG:boto:path=/pklxkd-61/obj76?versionId=null 2023-05-31T21:03:23.948 DEBUG:boto:auth_path=/pklxkd-61/obj76?versionId=null 2023-05-31T21:03:23.949 DEBUG:boto:Method: GET 2023-05-31T21:03:23.949 DEBUG:boto:Path: /pklxkd-61/obj76?versionId=null 2023-05-31T21:03:23.949 DEBUG:boto:Data: 2023-05-31T21:03:23.950 DEBUG:boto:Headers: {} 2023-05-31T21:03:23.950 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T21:03:23.951 DEBUG:boto:Port: 8001 2023-05-31T21:03:23.951 DEBUG:boto:Params: {} 2023-05-31T21:03:23.951 DEBUG:boto:Token: None 2023-05-31T21:03:23.952 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 21:03:23 GMT /pklxkd-61/obj76?versionId=null 2023-05-31T21:03:23.952 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:md/ANYNXgTBdlrSU9VOcAw4wSUo= 2023-05-31T21:03:23.953 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 21:03:23 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:md/ANYNXgTBdlrSU9VOcAw4wSUo=', 'Content-Length': '0'} 2023-05-31T21:03:23.955 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:59:45 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000cada68bbe6bddae8-006477b61b-1f4e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 21:03:23 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:03:23.955 DEBUG:boto:path=/pklxkd-61/obj76 2023-05-31T21:03:23.956 DEBUG:boto:auth_path=/pklxkd-61/obj76 2023-05-31T21:03:23.956 DEBUG:boto:path=/pklxkd-61/obj76?versionId=null 2023-05-31T21:03:23.957 DEBUG:boto:auth_path=/pklxkd-61/obj76?versionId=null 2023-05-31T21:03:23.957 DEBUG:boto:Method: GET 2023-05-31T21:03:23.957 DEBUG:boto:Path: /pklxkd-61/obj76?versionId=null 2023-05-31T21:03:23.958 DEBUG:boto:Data: 2023-05-31T21:03:23.958 DEBUG:boto:Headers: {} 2023-05-31T21:03:23.959 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T21:03:23.959 DEBUG:boto:Port: 8000 2023-05-31T21:03:23.959 DEBUG:boto:Params: {} 2023-05-31T21:03:23.960 DEBUG:boto:Token: None 2023-05-31T21:03:23.960 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 21:03:23 GMT /pklxkd-61/obj76?versionId=null 2023-05-31T21:03:23.961 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:md/ANYNXgTBdlrSU9VOcAw4wSUo= 2023-05-31T21:03:23.961 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 21:03:23 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:md/ANYNXgTBdlrSU9VOcAw4wSUo=', 'Content-Length': '0'} 2023-05-31T21:03:23.963 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:59:45 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000cf89432fbb121f98-006477b61b-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 21:03:23 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:03:23.963 DEBUG:boto:path=/pklxkd-61/obj76 2023-05-31T21:03:23.964 DEBUG:boto:auth_path=/pklxkd-61/obj76 2023-05-31T21:03:23.964 DEBUG:boto:path=/pklxkd-61/obj76?versionId=null 2023-05-31T21:03:23.964 DEBUG:boto:auth_path=/pklxkd-61/obj76?versionId=null 2023-05-31T21:03:23.965 DEBUG:boto:Method: HEAD 2023-05-31T21:03:23.965 DEBUG:boto:Path: /pklxkd-61/obj76?versionId=null 2023-05-31T21:03:23.965 DEBUG:boto:Data: 2023-05-31T21:03:23.966 DEBUG:boto:Headers: {} 2023-05-31T21:03:23.966 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T21:03:23.966 DEBUG:boto:Port: 8001 2023-05-31T21:03:23.967 DEBUG:boto:Params: {} 2023-05-31T21:03:23.967 DEBUG:boto:Token: None 2023-05-31T21:03:23.968 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 21:03:23 GMT /pklxkd-61/obj76?versionId=null 2023-05-31T21:03:23.968 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:1JdbP9vt3TyGejtkwNYXzeuMEH4= 2023-05-31T21:03:23.968 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 21:03:23 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:1JdbP9vt3TyGejtkwNYXzeuMEH4=', 'Content-Length': '0'} 2023-05-31T21:03:23.971 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:59:45 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000eb2cbbcb5f0b2eb0-006477b61b-1f4e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 21:03:23 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:03:23.971 DEBUG:boto:path=/pklxkd-61/obj76 2023-05-31T21:03:23.972 DEBUG:boto:auth_path=/pklxkd-61/obj76 2023-05-31T21:03:23.972 DEBUG:boto:path=/pklxkd-61/obj76?versionId=null 2023-05-31T21:03:23.973 DEBUG:boto:auth_path=/pklxkd-61/obj76?versionId=null 2023-05-31T21:03:23.973 DEBUG:boto:Method: HEAD 2023-05-31T21:03:23.973 DEBUG:boto:Path: /pklxkd-61/obj76?versionId=null 2023-05-31T21:03:23.974 DEBUG:boto:Data: 2023-05-31T21:03:23.974 DEBUG:boto:Headers: {} 2023-05-31T21:03:23.974 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T21:03:23.975 DEBUG:boto:Port: 8000 2023-05-31T21:03:23.975 DEBUG:boto:Params: {} 2023-05-31T21:03:23.975 DEBUG:boto:Token: None 2023-05-31T21:03:23.976 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 21:03:23 GMT /pklxkd-61/obj76?versionId=null 2023-05-31T21:03:23.976 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:1JdbP9vt3TyGejtkwNYXzeuMEH4= 2023-05-31T21:03:23.977 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 21:03:23 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:1JdbP9vt3TyGejtkwNYXzeuMEH4=', 'Content-Length': '0'} 2023-05-31T21:03:23.978 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:59:45 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000099daf58784f4498-006477b61b-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 21:03:23 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:03:23.979 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj76 2023-05-31T21:03:23.979 DEBUG:boto:path=/pklxkd-61/obj76 2023-05-31T21:03:23.980 DEBUG:boto:auth_path=/pklxkd-61/obj76 2023-05-31T21:03:23.980 DEBUG:boto:path=/pklxkd-61/obj76?versionId=null 2023-05-31T21:03:23.980 DEBUG:boto:auth_path=/pklxkd-61/obj76?versionId=null 2023-05-31T21:03:23.981 DEBUG:boto:Method: GET 2023-05-31T21:03:23.981 DEBUG:boto:Path: /pklxkd-61/obj76?versionId=null 2023-05-31T21:03:23.981 DEBUG:boto:Data: 2023-05-31T21:03:23.982 DEBUG:boto:Headers: {} 2023-05-31T21:03:23.982 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T21:03:23.982 DEBUG:boto:Port: 8001 2023-05-31T21:03:23.983 DEBUG:boto:Params: {} 2023-05-31T21:03:23.983 DEBUG:boto:Token: None 2023-05-31T21:03:23.984 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 21:03:23 GMT /pklxkd-61/obj76?versionId=null 2023-05-31T21:03:23.984 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:md/ANYNXgTBdlrSU9VOcAw4wSUo= 2023-05-31T21:03:23.984 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 21:03:23 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:md/ANYNXgTBdlrSU9VOcAw4wSUo=', 'Content-Length': '0'} 2023-05-31T21:03:23.986 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:59:45 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008046866bab2e2cb1-006477b61b-1f4e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 21:03:23 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:03:23.987 DEBUG:boto:path=/pklxkd-61/obj76 2023-05-31T21:03:23.987 DEBUG:boto:auth_path=/pklxkd-61/obj76 2023-05-31T21:03:23.988 DEBUG:boto:path=/pklxkd-61/obj76?versionId=null 2023-05-31T21:03:23.988 DEBUG:boto:auth_path=/pklxkd-61/obj76?versionId=null 2023-05-31T21:03:23.988 DEBUG:boto:Method: GET 2023-05-31T21:03:23.989 DEBUG:boto:Path: /pklxkd-61/obj76?versionId=null 2023-05-31T21:03:23.989 DEBUG:boto:Data: 2023-05-31T21:03:23.989 DEBUG:boto:Headers: {} 2023-05-31T21:03:23.990 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T21:03:23.990 DEBUG:boto:Port: 8000 2023-05-31T21:03:23.991 DEBUG:boto:Params: {} 2023-05-31T21:03:23.991 DEBUG:boto:Token: None 2023-05-31T21:03:23.991 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 21:03:23 GMT /pklxkd-61/obj76?versionId=null 2023-05-31T21:03:23.992 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:md/ANYNXgTBdlrSU9VOcAw4wSUo= 2023-05-31T21:03:23.992 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 21:03:23 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:md/ANYNXgTBdlrSU9VOcAw4wSUo=', 'Content-Length': '0'} 2023-05-31T21:03:23.994 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:59:45 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004acce1c2f6c15eb0-006477b61b-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 21:03:23 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:03:23.995 DEBUG:boto:path=/pklxkd-61/ 2023-05-31T21:03:23.995 DEBUG:boto:auth_path=/pklxkd-61/ 2023-05-31T21:03:23.996 DEBUG:boto:path=/pklxkd-61/?prefix=obj76 2023-05-31T21:03:23.996 DEBUG:boto:auth_path=/pklxkd-61/?prefix=obj76 2023-05-31T21:03:23.996 DEBUG:boto:Method: GET 2023-05-31T21:03:23.997 DEBUG:boto:Path: /pklxkd-61/?prefix=obj76 2023-05-31T21:03:23.997 DEBUG:boto:Data: 2023-05-31T21:03:23.997 DEBUG:boto:Headers: {} 2023-05-31T21:03:23.998 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T21:03:23.998 DEBUG:boto:Port: 8001 2023-05-31T21:03:23.998 DEBUG:boto:Params: {} 2023-05-31T21:03:23.999 DEBUG:boto:Token: None 2023-05-31T21:03:23.999 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 21:03:23 GMT /pklxkd-61/ 2023-05-31T21:03:24.000 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:7YkKDTB+91bWua1TWmlOlWHvc2U= 2023-05-31T21:03:24.000 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 21:03:23 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:7YkKDTB+91bWua1TWmlOlWHvc2U=', 'Content-Length': '0'} 2023-05-31T21:03:24.002 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d4d7689040c99614-006477b61b-1f4e-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 21:03:23 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:03:24.003 DEBUG:boto:b'pklxkd-61obj761000falseobj762023-05-31T20:59:45.256Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T21:03:24.004 DEBUG:boto:path=/pklxkd-61/ 2023-05-31T21:03:24.004 DEBUG:boto:auth_path=/pklxkd-61/ 2023-05-31T21:03:24.004 DEBUG:boto:path=/pklxkd-61/?prefix=obj76 2023-05-31T21:03:24.005 DEBUG:boto:auth_path=/pklxkd-61/?prefix=obj76 2023-05-31T21:03:24.005 DEBUG:boto:Method: GET 2023-05-31T21:03:24.005 DEBUG:boto:Path: /pklxkd-61/?prefix=obj76 2023-05-31T21:03:24.006 DEBUG:boto:Data: 2023-05-31T21:03:24.006 DEBUG:boto:Headers: {} 2023-05-31T21:03:24.007 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T21:03:24.007 DEBUG:boto:Port: 8000 2023-05-31T21:03:24.007 DEBUG:boto:Params: {} 2023-05-31T21:03:24.008 DEBUG:boto:Token: None 2023-05-31T21:03:24.008 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 21:03:24 GMT /pklxkd-61/ 2023-05-31T21:03:24.009 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:Y+z1tdIY0pnnJpqEt5b8+XUlWS4= 2023-05-31T21:03:24.009 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 21:03:24 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:Y+z1tdIY0pnnJpqEt5b8+XUlWS4=', 'Content-Length': '0'} 2023-05-31T21:03:24.013 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000005a9e7be8cb396fe6-006477b61c-1630-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 21:03:24 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:03:24.014 DEBUG:boto:b'pklxkd-61obj761000falseobj762023-05-31T20:59:45.256Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T21:03:24.014 DEBUG:boto:path=/pklxkd-61/obj76 2023-05-31T21:03:24.015 DEBUG:boto:auth_path=/pklxkd-61/obj76 2023-05-31T21:03:24.015 DEBUG:boto:Method: HEAD 2023-05-31T21:03:24.015 DEBUG:boto:Path: /pklxkd-61/obj76 2023-05-31T21:03:24.016 DEBUG:boto:Data: 2023-05-31T21:03:24.016 DEBUG:boto:Headers: {} 2023-05-31T21:03:24.016 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T21:03:24.017 DEBUG:boto:Port: 8001 2023-05-31T21:03:24.017 DEBUG:boto:Params: {} 2023-05-31T21:03:24.017 DEBUG:boto:Token: None 2023-05-31T21:03:24.018 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 21:03:24 GMT /pklxkd-61/obj76 2023-05-31T21:03:24.018 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:wx4nB8t8EczBFQG9G+XYRkZ0ox0= 2023-05-31T21:03:24.019 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 21:03:24 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:wx4nB8t8EczBFQG9G+XYRkZ0ox0=', 'Content-Length': '0'} 2023-05-31T21:03:24.020 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:59:45 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a2a4cb3857a31a34-006477b61c-1f4e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 21:03:24 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:03:24.021 DEBUG:boto:path=/pklxkd-61/obj76 2023-05-31T21:03:24.021 DEBUG:boto:auth_path=/pklxkd-61/obj76 2023-05-31T21:03:24.022 DEBUG:boto:Method: HEAD 2023-05-31T21:03:24.022 DEBUG:boto:Path: /pklxkd-61/obj76 2023-05-31T21:03:24.022 DEBUG:boto:Data: 2023-05-31T21:03:24.023 DEBUG:boto:Headers: {} 2023-05-31T21:03:24.023 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T21:03:24.023 DEBUG:boto:Port: 8000 2023-05-31T21:03:24.024 DEBUG:boto:Params: {} 2023-05-31T21:03:24.024 DEBUG:boto:Token: None 2023-05-31T21:03:24.024 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 21:03:24 GMT /pklxkd-61/obj76 2023-05-31T21:03:24.025 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:wx4nB8t8EczBFQG9G+XYRkZ0ox0= 2023-05-31T21:03:24.025 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 21:03:24 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:wx4nB8t8EczBFQG9G+XYRkZ0ox0=', 'Content-Length': '0'} 2023-05-31T21:03:24.027 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:59:45 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004f587053310c48f9-006477b61c-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 21:03:24 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:03:24.028 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj76 2023-05-31T21:03:24.028 DEBUG:boto:path=/pklxkd-61/obj76 2023-05-31T21:03:24.028 DEBUG:boto:auth_path=/pklxkd-61/obj76 2023-05-31T21:03:24.029 DEBUG:boto:Method: GET 2023-05-31T21:03:24.029 DEBUG:boto:Path: /pklxkd-61/obj76 2023-05-31T21:03:24.030 DEBUG:boto:Data: 2023-05-31T21:03:24.030 DEBUG:boto:Headers: {} 2023-05-31T21:03:24.030 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T21:03:24.031 DEBUG:boto:Port: 8001 2023-05-31T21:03:24.031 DEBUG:boto:Params: {} 2023-05-31T21:03:24.031 DEBUG:boto:Token: None 2023-05-31T21:03:24.032 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 21:03:24 GMT /pklxkd-61/obj76 2023-05-31T21:03:24.032 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:F4t8G7L0/9Wr4TbUa4U20Rr2Y6A= 2023-05-31T21:03:24.033 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 21:03:24 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:F4t8G7L0/9Wr4TbUa4U20Rr2Y6A=', 'Content-Length': '0'} 2023-05-31T21:03:24.034 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:59:45 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003ef0147e8be3ad91-006477b61c-1f4e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 21:03:24 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:03:24.076 DEBUG:boto:path=/pklxkd-61/obj76 2023-05-31T21:03:24.077 DEBUG:boto:auth_path=/pklxkd-61/obj76 2023-05-31T21:03:24.077 DEBUG:boto:Method: GET 2023-05-31T21:03:24.078 DEBUG:boto:Path: /pklxkd-61/obj76 2023-05-31T21:03:24.078 DEBUG:boto:Data: 2023-05-31T21:03:24.079 DEBUG:boto:Headers: {} 2023-05-31T21:03:24.079 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T21:03:24.079 DEBUG:boto:Port: 8000 2023-05-31T21:03:24.080 DEBUG:boto:Params: {} 2023-05-31T21:03:24.080 DEBUG:boto:Token: None 2023-05-31T21:03:24.081 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 21:03:24 GMT /pklxkd-61/obj76 2023-05-31T21:03:24.081 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:F4t8G7L0/9Wr4TbUa4U20Rr2Y6A= 2023-05-31T21:03:24.082 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 21:03:24 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:F4t8G7L0/9Wr4TbUa4U20Rr2Y6A=', 'Content-Length': '0'} 2023-05-31T21:03:24.084 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:59:45 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000081008bb191a685ac-006477b61c-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 21:03:24 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:03:24.085 DEBUG:boto:path=/pklxkd-61/ 2023-05-31T21:03:24.085 DEBUG:boto:auth_path=/pklxkd-61/ 2023-05-31T21:03:24.086 DEBUG:boto:path=/pklxkd-61/?prefix=obj76 2023-05-31T21:03:24.086 DEBUG:boto:auth_path=/pklxkd-61/?prefix=obj76 2023-05-31T21:03:24.087 DEBUG:boto:Method: GET 2023-05-31T21:03:24.088 DEBUG:boto:Path: /pklxkd-61/?prefix=obj76 2023-05-31T21:03:24.088 DEBUG:boto:Data: 2023-05-31T21:03:24.089 DEBUG:boto:Headers: {} 2023-05-31T21:03:24.089 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T21:03:24.090 DEBUG:boto:Port: 8001 2023-05-31T21:03:24.090 DEBUG:boto:Params: {} 2023-05-31T21:03:24.090 DEBUG:boto:Token: None 2023-05-31T21:03:24.091 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 21:03:24 GMT /pklxkd-61/ 2023-05-31T21:03:24.091 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:Y+z1tdIY0pnnJpqEt5b8+XUlWS4= 2023-05-31T21:03:24.092 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 21:03:24 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:Y+z1tdIY0pnnJpqEt5b8+XUlWS4=', 'Content-Length': '0'} 2023-05-31T21:03:24.094 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000000c25bc3fd23d789c-006477b61c-1f4e-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 21:03:24 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:03:24.095 DEBUG:boto:b'pklxkd-61obj761000falseobj762023-05-31T20:59:45.256Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T21:03:24.095 DEBUG:boto:path=/pklxkd-61/ 2023-05-31T21:03:24.096 DEBUG:boto:auth_path=/pklxkd-61/ 2023-05-31T21:03:24.096 DEBUG:boto:path=/pklxkd-61/?prefix=obj76 2023-05-31T21:03:24.097 DEBUG:boto:auth_path=/pklxkd-61/?prefix=obj76 2023-05-31T21:03:24.097 DEBUG:boto:Method: GET 2023-05-31T21:03:24.097 DEBUG:boto:Path: /pklxkd-61/?prefix=obj76 2023-05-31T21:03:24.098 DEBUG:boto:Data: 2023-05-31T21:03:24.098 DEBUG:boto:Headers: {} 2023-05-31T21:03:24.098 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T21:03:24.099 DEBUG:boto:Port: 8000 2023-05-31T21:03:24.099 DEBUG:boto:Params: {} 2023-05-31T21:03:24.099 DEBUG:boto:Token: None 2023-05-31T21:03:24.100 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 21:03:24 GMT /pklxkd-61/ 2023-05-31T21:03:24.100 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:Y+z1tdIY0pnnJpqEt5b8+XUlWS4= 2023-05-31T21:03:24.101 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 21:03:24 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:Y+z1tdIY0pnnJpqEt5b8+XUlWS4=', 'Content-Length': '0'} 2023-05-31T21:03:24.105 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000dd0db7fd6b215b02-006477b61c-1630-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 21:03:24 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:03:24.106 DEBUG:boto:b'pklxkd-61obj761000falseobj762023-05-31T20:59:45.256Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T21:03:24.106 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj85 2023-05-31T21:03:24.107 DEBUG:boto:path=/pklxkd-61/obj85 2023-05-31T21:03:24.107 DEBUG:boto:auth_path=/pklxkd-61/obj85 2023-05-31T21:03:24.107 DEBUG:boto:path=/pklxkd-61/obj85?versionId=null 2023-05-31T21:03:24.108 DEBUG:boto:auth_path=/pklxkd-61/obj85?versionId=null 2023-05-31T21:03:24.108 DEBUG:boto:Method: GET 2023-05-31T21:03:24.109 DEBUG:boto:Path: /pklxkd-61/obj85?versionId=null 2023-05-31T21:03:24.109 DEBUG:boto:Data: 2023-05-31T21:03:24.109 DEBUG:boto:Headers: {} 2023-05-31T21:03:24.110 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T21:03:24.110 DEBUG:boto:Port: 8001 2023-05-31T21:03:24.110 DEBUG:boto:Params: {} 2023-05-31T21:03:24.111 DEBUG:boto:Token: None 2023-05-31T21:03:24.111 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 21:03:24 GMT /pklxkd-61/obj85?versionId=null 2023-05-31T21:03:24.112 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:bYICaCZmOyA3vjp9OMhnNyFeHXc= 2023-05-31T21:03:24.112 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 21:03:24 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:bYICaCZmOyA3vjp9OMhnNyFeHXc=', 'Content-Length': '0'} 2023-05-31T21:03:24.114 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:59:44 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a7b8c6cc8926d5d7-006477b61c-1f4e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 21:03:24 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:03:24.115 DEBUG:boto:path=/pklxkd-61/obj85 2023-05-31T21:03:24.115 DEBUG:boto:auth_path=/pklxkd-61/obj85 2023-05-31T21:03:24.116 DEBUG:boto:path=/pklxkd-61/obj85?versionId=null 2023-05-31T21:03:24.116 DEBUG:boto:auth_path=/pklxkd-61/obj85?versionId=null 2023-05-31T21:03:24.116 DEBUG:boto:Method: GET 2023-05-31T21:03:24.117 DEBUG:boto:Path: /pklxkd-61/obj85?versionId=null 2023-05-31T21:03:24.117 DEBUG:boto:Data: 2023-05-31T21:03:24.117 DEBUG:boto:Headers: {} 2023-05-31T21:03:24.118 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T21:03:24.118 DEBUG:boto:Port: 8000 2023-05-31T21:03:24.119 DEBUG:boto:Params: {} 2023-05-31T21:03:24.119 DEBUG:boto:Token: None 2023-05-31T21:03:24.119 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 21:03:24 GMT /pklxkd-61/obj85?versionId=null 2023-05-31T21:03:24.120 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:bYICaCZmOyA3vjp9OMhnNyFeHXc= 2023-05-31T21:03:24.120 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 21:03:24 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:bYICaCZmOyA3vjp9OMhnNyFeHXc=', 'Content-Length': '0'} 2023-05-31T21:03:24.123 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:59:44 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d0a73b6663c005a4-006477b61c-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 21:03:24 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:03:24.123 DEBUG:boto:path=/pklxkd-61/obj85 2023-05-31T21:03:24.124 DEBUG:boto:auth_path=/pklxkd-61/obj85 2023-05-31T21:03:24.124 DEBUG:boto:path=/pklxkd-61/obj85?versionId=null 2023-05-31T21:03:24.124 DEBUG:boto:auth_path=/pklxkd-61/obj85?versionId=null 2023-05-31T21:03:24.125 DEBUG:boto:Method: HEAD 2023-05-31T21:03:24.125 DEBUG:boto:Path: /pklxkd-61/obj85?versionId=null 2023-05-31T21:03:24.125 DEBUG:boto:Data: 2023-05-31T21:03:24.126 DEBUG:boto:Headers: {} 2023-05-31T21:03:24.126 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T21:03:24.126 DEBUG:boto:Port: 8001 2023-05-31T21:03:24.127 DEBUG:boto:Params: {} 2023-05-31T21:03:24.127 DEBUG:boto:Token: None 2023-05-31T21:03:24.128 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 21:03:24 GMT /pklxkd-61/obj85?versionId=null 2023-05-31T21:03:24.128 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:IE1IY62tLVm7Qxt3RGJLyOFiBbo= 2023-05-31T21:03:24.128 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 21:03:24 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:IE1IY62tLVm7Qxt3RGJLyOFiBbo=', 'Content-Length': '0'} 2023-05-31T21:03:24.130 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:59:44 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000eb2b92c146b4f225-006477b61c-1f4e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 21:03:24 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:03:24.131 DEBUG:boto:path=/pklxkd-61/obj85 2023-05-31T21:03:24.131 DEBUG:boto:auth_path=/pklxkd-61/obj85 2023-05-31T21:03:24.131 DEBUG:boto:path=/pklxkd-61/obj85?versionId=null 2023-05-31T21:03:24.132 DEBUG:boto:auth_path=/pklxkd-61/obj85?versionId=null 2023-05-31T21:03:24.132 DEBUG:boto:Method: HEAD 2023-05-31T21:03:24.132 DEBUG:boto:Path: /pklxkd-61/obj85?versionId=null 2023-05-31T21:03:24.133 DEBUG:boto:Data: 2023-05-31T21:03:24.133 DEBUG:boto:Headers: {} 2023-05-31T21:03:24.133 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T21:03:24.134 DEBUG:boto:Port: 8000 2023-05-31T21:03:24.134 DEBUG:boto:Params: {} 2023-05-31T21:03:24.135 DEBUG:boto:Token: None 2023-05-31T21:03:24.135 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 21:03:24 GMT /pklxkd-61/obj85?versionId=null 2023-05-31T21:03:24.135 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:IE1IY62tLVm7Qxt3RGJLyOFiBbo= 2023-05-31T21:03:24.136 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 21:03:24 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:IE1IY62tLVm7Qxt3RGJLyOFiBbo=', 'Content-Length': '0'} 2023-05-31T21:03:24.138 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:59:44 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000075cef61af5c5fe2-006477b61c-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 21:03:24 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:03:24.138 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj85 2023-05-31T21:03:24.138 DEBUG:boto:path=/pklxkd-61/obj85 2023-05-31T21:03:24.139 DEBUG:boto:auth_path=/pklxkd-61/obj85 2023-05-31T21:03:24.140 DEBUG:boto:path=/pklxkd-61/obj85?versionId=null 2023-05-31T21:03:24.140 DEBUG:boto:auth_path=/pklxkd-61/obj85?versionId=null 2023-05-31T21:03:24.141 DEBUG:boto:Method: GET 2023-05-31T21:03:24.141 DEBUG:boto:Path: /pklxkd-61/obj85?versionId=null 2023-05-31T21:03:24.141 DEBUG:boto:Data: 2023-05-31T21:03:24.142 DEBUG:boto:Headers: {} 2023-05-31T21:03:24.142 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T21:03:24.142 DEBUG:boto:Port: 8001 2023-05-31T21:03:24.143 DEBUG:boto:Params: {} 2023-05-31T21:03:24.143 DEBUG:boto:Token: None 2023-05-31T21:03:24.144 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 21:03:24 GMT /pklxkd-61/obj85?versionId=null 2023-05-31T21:03:24.144 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:bYICaCZmOyA3vjp9OMhnNyFeHXc= 2023-05-31T21:03:24.144 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 21:03:24 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:bYICaCZmOyA3vjp9OMhnNyFeHXc=', 'Content-Length': '0'} 2023-05-31T21:03:24.146 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:59:44 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000001624d44b3f4a65e-006477b61c-1f4e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 21:03:24 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:03:24.147 DEBUG:boto:path=/pklxkd-61/obj85 2023-05-31T21:03:24.147 DEBUG:boto:auth_path=/pklxkd-61/obj85 2023-05-31T21:03:24.148 DEBUG:boto:path=/pklxkd-61/obj85?versionId=null 2023-05-31T21:03:24.148 DEBUG:boto:auth_path=/pklxkd-61/obj85?versionId=null 2023-05-31T21:03:24.149 DEBUG:boto:Method: GET 2023-05-31T21:03:24.150 DEBUG:boto:Path: /pklxkd-61/obj85?versionId=null 2023-05-31T21:03:24.151 DEBUG:boto:Data: 2023-05-31T21:03:24.152 DEBUG:boto:Headers: {} 2023-05-31T21:03:24.153 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T21:03:24.154 DEBUG:boto:Port: 8000 2023-05-31T21:03:24.155 DEBUG:boto:Params: {} 2023-05-31T21:03:24.155 DEBUG:boto:Token: None 2023-05-31T21:03:24.156 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 21:03:24 GMT /pklxkd-61/obj85?versionId=null 2023-05-31T21:03:24.157 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:bYICaCZmOyA3vjp9OMhnNyFeHXc= 2023-05-31T21:03:24.158 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 21:03:24 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:bYICaCZmOyA3vjp9OMhnNyFeHXc=', 'Content-Length': '0'} 2023-05-31T21:03:24.159 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:59:44 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008abc0e9555d51190-006477b61c-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 21:03:24 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:03:24.160 DEBUG:boto:path=/pklxkd-61/ 2023-05-31T21:03:24.161 DEBUG:boto:auth_path=/pklxkd-61/ 2023-05-31T21:03:24.161 DEBUG:boto:path=/pklxkd-61/?prefix=obj85 2023-05-31T21:03:24.161 DEBUG:boto:auth_path=/pklxkd-61/?prefix=obj85 2023-05-31T21:03:24.162 DEBUG:boto:Method: GET 2023-05-31T21:03:24.162 DEBUG:boto:Path: /pklxkd-61/?prefix=obj85 2023-05-31T21:03:24.163 DEBUG:boto:Data: 2023-05-31T21:03:24.163 DEBUG:boto:Headers: {} 2023-05-31T21:03:24.163 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T21:03:24.164 DEBUG:boto:Port: 8001 2023-05-31T21:03:24.164 DEBUG:boto:Params: {} 2023-05-31T21:03:24.165 DEBUG:boto:Token: None 2023-05-31T21:03:24.165 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 21:03:24 GMT /pklxkd-61/ 2023-05-31T21:03:24.166 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:Y+z1tdIY0pnnJpqEt5b8+XUlWS4= 2023-05-31T21:03:24.166 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 21:03:24 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:Y+z1tdIY0pnnJpqEt5b8+XUlWS4=', 'Content-Length': '0'} 2023-05-31T21:03:24.169 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000da88b6d121157db5-006477b61c-1f4e-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 21:03:24 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:03:24.169 DEBUG:boto:b'pklxkd-61obj851000falseobj852023-05-31T20:59:44.912Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T21:03:24.170 DEBUG:boto:path=/pklxkd-61/ 2023-05-31T21:03:24.170 DEBUG:boto:auth_path=/pklxkd-61/ 2023-05-31T21:03:24.171 DEBUG:boto:path=/pklxkd-61/?prefix=obj85 2023-05-31T21:03:24.171 DEBUG:boto:auth_path=/pklxkd-61/?prefix=obj85 2023-05-31T21:03:24.171 DEBUG:boto:Method: GET 2023-05-31T21:03:24.172 DEBUG:boto:Path: /pklxkd-61/?prefix=obj85 2023-05-31T21:03:24.172 DEBUG:boto:Data: 2023-05-31T21:03:24.172 DEBUG:boto:Headers: {} 2023-05-31T21:03:24.173 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T21:03:24.173 DEBUG:boto:Port: 8000 2023-05-31T21:03:24.173 DEBUG:boto:Params: {} 2023-05-31T21:03:24.174 DEBUG:boto:Token: None 2023-05-31T21:03:24.174 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 21:03:24 GMT /pklxkd-61/ 2023-05-31T21:03:24.175 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:Y+z1tdIY0pnnJpqEt5b8+XUlWS4= 2023-05-31T21:03:24.175 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 21:03:24 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:Y+z1tdIY0pnnJpqEt5b8+XUlWS4=', 'Content-Length': '0'} 2023-05-31T21:03:24.179 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000bb56a7337e9e7cbc-006477b61c-1630-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 21:03:24 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:03:24.179 DEBUG:boto:b'pklxkd-61obj851000falseobj852023-05-31T20:59:44.912Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T21:03:24.180 DEBUG:boto:path=/pklxkd-61/obj85 2023-05-31T21:03:24.181 DEBUG:boto:auth_path=/pklxkd-61/obj85 2023-05-31T21:03:24.181 DEBUG:boto:Method: HEAD 2023-05-31T21:03:24.181 DEBUG:boto:Path: /pklxkd-61/obj85 2023-05-31T21:03:24.182 DEBUG:boto:Data: 2023-05-31T21:03:24.182 DEBUG:boto:Headers: {} 2023-05-31T21:03:24.182 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T21:03:24.183 DEBUG:boto:Port: 8001 2023-05-31T21:03:24.183 DEBUG:boto:Params: {} 2023-05-31T21:03:24.183 DEBUG:boto:Token: None 2023-05-31T21:03:24.184 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 21:03:24 GMT /pklxkd-61/obj85 2023-05-31T21:03:24.184 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:R8C5yv0P56w1mnqk4fkP6mN8/Zg= 2023-05-31T21:03:24.185 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 21:03:24 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:R8C5yv0P56w1mnqk4fkP6mN8/Zg=', 'Content-Length': '0'} 2023-05-31T21:03:24.187 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:59:44 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003a92503b4d5a6313-006477b61c-1f4e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 21:03:24 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:03:24.187 DEBUG:boto:path=/pklxkd-61/obj85 2023-05-31T21:03:24.188 DEBUG:boto:auth_path=/pklxkd-61/obj85 2023-05-31T21:03:24.188 DEBUG:boto:Method: HEAD 2023-05-31T21:03:24.189 DEBUG:boto:Path: /pklxkd-61/obj85 2023-05-31T21:03:24.189 DEBUG:boto:Data: 2023-05-31T21:03:24.189 DEBUG:boto:Headers: {} 2023-05-31T21:03:24.190 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T21:03:24.190 DEBUG:boto:Port: 8000 2023-05-31T21:03:24.190 DEBUG:boto:Params: {} 2023-05-31T21:03:24.191 DEBUG:boto:Token: None 2023-05-31T21:03:24.191 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 21:03:24 GMT /pklxkd-61/obj85 2023-05-31T21:03:24.191 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:R8C5yv0P56w1mnqk4fkP6mN8/Zg= 2023-05-31T21:03:24.192 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 21:03:24 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:R8C5yv0P56w1mnqk4fkP6mN8/Zg=', 'Content-Length': '0'} 2023-05-31T21:03:24.194 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:59:44 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ebc7557a285fe73f-006477b61c-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 21:03:24 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:03:24.194 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj85 2023-05-31T21:03:24.195 DEBUG:boto:path=/pklxkd-61/obj85 2023-05-31T21:03:24.195 DEBUG:boto:auth_path=/pklxkd-61/obj85 2023-05-31T21:03:24.195 DEBUG:boto:Method: GET 2023-05-31T21:03:24.196 DEBUG:boto:Path: /pklxkd-61/obj85 2023-05-31T21:03:24.196 DEBUG:boto:Data: 2023-05-31T21:03:24.196 DEBUG:boto:Headers: {} 2023-05-31T21:03:24.197 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T21:03:24.197 DEBUG:boto:Port: 8001 2023-05-31T21:03:24.198 DEBUG:boto:Params: {} 2023-05-31T21:03:24.198 DEBUG:boto:Token: None 2023-05-31T21:03:24.198 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 21:03:24 GMT /pklxkd-61/obj85 2023-05-31T21:03:24.199 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:9pLgc03W+nU/VKyaiORQYdKPSds= 2023-05-31T21:03:24.199 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 21:03:24 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:9pLgc03W+nU/VKyaiORQYdKPSds=', 'Content-Length': '0'} 2023-05-31T21:03:24.201 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:59:44 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000448ff6117c8bb15-006477b61c-1f4e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 21:03:24 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:03:24.201 DEBUG:boto:path=/pklxkd-61/obj85 2023-05-31T21:03:24.202 DEBUG:boto:auth_path=/pklxkd-61/obj85 2023-05-31T21:03:24.202 DEBUG:boto:Method: GET 2023-05-31T21:03:24.203 DEBUG:boto:Path: /pklxkd-61/obj85 2023-05-31T21:03:24.203 DEBUG:boto:Data: 2023-05-31T21:03:24.203 DEBUG:boto:Headers: {} 2023-05-31T21:03:24.204 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T21:03:24.204 DEBUG:boto:Port: 8000 2023-05-31T21:03:24.204 DEBUG:boto:Params: {} 2023-05-31T21:03:24.205 DEBUG:boto:Token: None 2023-05-31T21:03:24.205 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 21:03:24 GMT /pklxkd-61/obj85 2023-05-31T21:03:24.206 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:9pLgc03W+nU/VKyaiORQYdKPSds= 2023-05-31T21:03:24.206 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 21:03:24 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:9pLgc03W+nU/VKyaiORQYdKPSds=', 'Content-Length': '0'} 2023-05-31T21:03:24.208 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:59:44 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000049a5637f6bba537e-006477b61c-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 21:03:24 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:03:24.209 DEBUG:boto:path=/pklxkd-61/ 2023-05-31T21:03:24.209 DEBUG:boto:auth_path=/pklxkd-61/ 2023-05-31T21:03:24.210 DEBUG:boto:path=/pklxkd-61/?prefix=obj85 2023-05-31T21:03:24.210 DEBUG:boto:auth_path=/pklxkd-61/?prefix=obj85 2023-05-31T21:03:24.210 DEBUG:boto:Method: GET 2023-05-31T21:03:24.211 DEBUG:boto:Path: /pklxkd-61/?prefix=obj85 2023-05-31T21:03:24.211 DEBUG:boto:Data: 2023-05-31T21:03:24.211 DEBUG:boto:Headers: {} 2023-05-31T21:03:24.212 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T21:03:24.212 DEBUG:boto:Port: 8001 2023-05-31T21:03:24.213 DEBUG:boto:Params: {} 2023-05-31T21:03:24.214 DEBUG:boto:Token: None 2023-05-31T21:03:24.215 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 21:03:24 GMT /pklxkd-61/ 2023-05-31T21:03:24.216 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:Y+z1tdIY0pnnJpqEt5b8+XUlWS4= 2023-05-31T21:03:24.217 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 21:03:24 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:Y+z1tdIY0pnnJpqEt5b8+XUlWS4=', 'Content-Length': '0'} 2023-05-31T21:03:24.220 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000684bf2c70331b5bf-006477b61c-1f4e-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 21:03:24 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:03:24.264 DEBUG:boto:b'pklxkd-61obj851000falseobj852023-05-31T20:59:44.912Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T21:03:24.265 DEBUG:boto:path=/pklxkd-61/ 2023-05-31T21:03:24.266 DEBUG:boto:auth_path=/pklxkd-61/ 2023-05-31T21:03:24.266 DEBUG:boto:path=/pklxkd-61/?prefix=obj85 2023-05-31T21:03:24.266 DEBUG:boto:auth_path=/pklxkd-61/?prefix=obj85 2023-05-31T21:03:24.267 DEBUG:boto:Method: GET 2023-05-31T21:03:24.267 DEBUG:boto:Path: /pklxkd-61/?prefix=obj85 2023-05-31T21:03:24.267 DEBUG:boto:Data: 2023-05-31T21:03:24.268 DEBUG:boto:Headers: {} 2023-05-31T21:03:24.268 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T21:03:24.269 DEBUG:boto:Port: 8000 2023-05-31T21:03:24.269 DEBUG:boto:Params: {} 2023-05-31T21:03:24.269 DEBUG:boto:Token: None 2023-05-31T21:03:24.270 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 21:03:24 GMT /pklxkd-61/ 2023-05-31T21:03:24.270 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:Y+z1tdIY0pnnJpqEt5b8+XUlWS4= 2023-05-31T21:03:24.271 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 21:03:24 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:Y+z1tdIY0pnnJpqEt5b8+XUlWS4=', 'Content-Length': '0'} 2023-05-31T21:03:24.275 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000003d0ba2a519987641-006477b61c-1630-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 21:03:24 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:03:24.275 DEBUG:boto:b'pklxkd-61obj851000falseobj852023-05-31T20:59:44.912Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T21:03:24.276 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj87 2023-05-31T21:03:24.276 DEBUG:boto:path=/pklxkd-61/obj87 2023-05-31T21:03:24.277 DEBUG:boto:auth_path=/pklxkd-61/obj87 2023-05-31T21:03:24.277 DEBUG:boto:path=/pklxkd-61/obj87?versionId=null 2023-05-31T21:03:24.277 DEBUG:boto:auth_path=/pklxkd-61/obj87?versionId=null 2023-05-31T21:03:24.278 DEBUG:boto:Method: GET 2023-05-31T21:03:24.278 DEBUG:boto:Path: /pklxkd-61/obj87?versionId=null 2023-05-31T21:03:24.279 DEBUG:boto:Data: 2023-05-31T21:03:24.279 DEBUG:boto:Headers: {} 2023-05-31T21:03:24.279 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T21:03:24.280 DEBUG:boto:Port: 8001 2023-05-31T21:03:24.280 DEBUG:boto:Params: {} 2023-05-31T21:03:24.280 DEBUG:boto:Token: None 2023-05-31T21:03:24.281 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 21:03:24 GMT /pklxkd-61/obj87?versionId=null 2023-05-31T21:03:24.281 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:LCJiIjWN6gtmeW/p3HVG6hDoqT8= 2023-05-31T21:03:24.282 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 21:03:24 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:LCJiIjWN6gtmeW/p3HVG6hDoqT8=', 'Content-Length': '0'} 2023-05-31T21:03:24.284 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:59:45 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b0c109d05fd54a43-006477b61c-1f4e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 21:03:24 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:03:24.285 DEBUG:boto:path=/pklxkd-61/obj87 2023-05-31T21:03:24.285 DEBUG:boto:auth_path=/pklxkd-61/obj87 2023-05-31T21:03:24.286 DEBUG:boto:path=/pklxkd-61/obj87?versionId=null 2023-05-31T21:03:24.286 DEBUG:boto:auth_path=/pklxkd-61/obj87?versionId=null 2023-05-31T21:03:24.287 DEBUG:boto:Method: GET 2023-05-31T21:03:24.287 DEBUG:boto:Path: /pklxkd-61/obj87?versionId=null 2023-05-31T21:03:24.287 DEBUG:boto:Data: 2023-05-31T21:03:24.288 DEBUG:boto:Headers: {} 2023-05-31T21:03:24.288 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T21:03:24.289 DEBUG:boto:Port: 8000 2023-05-31T21:03:24.289 DEBUG:boto:Params: {} 2023-05-31T21:03:24.289 DEBUG:boto:Token: None 2023-05-31T21:03:24.290 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 21:03:24 GMT /pklxkd-61/obj87?versionId=null 2023-05-31T21:03:24.290 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:LCJiIjWN6gtmeW/p3HVG6hDoqT8= 2023-05-31T21:03:24.291 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 21:03:24 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:LCJiIjWN6gtmeW/p3HVG6hDoqT8=', 'Content-Length': '0'} 2023-05-31T21:03:24.292 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:59:45 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009826c59252517b75-006477b61c-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 21:03:24 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:03:24.336 DEBUG:boto:path=/pklxkd-61/obj87 2023-05-31T21:03:24.337 DEBUG:boto:auth_path=/pklxkd-61/obj87 2023-05-31T21:03:24.337 DEBUG:boto:path=/pklxkd-61/obj87?versionId=null 2023-05-31T21:03:24.337 DEBUG:boto:auth_path=/pklxkd-61/obj87?versionId=null 2023-05-31T21:03:24.338 DEBUG:boto:Method: HEAD 2023-05-31T21:03:24.338 DEBUG:boto:Path: /pklxkd-61/obj87?versionId=null 2023-05-31T21:03:24.338 DEBUG:boto:Data: 2023-05-31T21:03:24.339 DEBUG:boto:Headers: {} 2023-05-31T21:03:24.339 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T21:03:24.339 DEBUG:boto:Port: 8001 2023-05-31T21:03:24.340 DEBUG:boto:Params: {} 2023-05-31T21:03:24.340 DEBUG:boto:Token: None 2023-05-31T21:03:24.341 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 21:03:24 GMT /pklxkd-61/obj87?versionId=null 2023-05-31T21:03:24.341 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:hQX9x/9TKutkPztcMvndZYwdJYY= 2023-05-31T21:03:24.341 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 21:03:24 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:hQX9x/9TKutkPztcMvndZYwdJYY=', 'Content-Length': '0'} 2023-05-31T21:03:24.343 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:59:45 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000fb37e901ba693466-006477b61c-1f4e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 21:03:24 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:03:24.344 DEBUG:boto:path=/pklxkd-61/obj87 2023-05-31T21:03:24.344 DEBUG:boto:auth_path=/pklxkd-61/obj87 2023-05-31T21:03:24.345 DEBUG:boto:path=/pklxkd-61/obj87?versionId=null 2023-05-31T21:03:24.345 DEBUG:boto:auth_path=/pklxkd-61/obj87?versionId=null 2023-05-31T21:03:24.345 DEBUG:boto:Method: HEAD 2023-05-31T21:03:24.346 DEBUG:boto:Path: /pklxkd-61/obj87?versionId=null 2023-05-31T21:03:24.346 DEBUG:boto:Data: 2023-05-31T21:03:24.346 DEBUG:boto:Headers: {} 2023-05-31T21:03:24.347 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T21:03:24.347 DEBUG:boto:Port: 8000 2023-05-31T21:03:24.347 DEBUG:boto:Params: {} 2023-05-31T21:03:24.348 DEBUG:boto:Token: None 2023-05-31T21:03:24.348 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 21:03:24 GMT /pklxkd-61/obj87?versionId=null 2023-05-31T21:03:24.349 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:hQX9x/9TKutkPztcMvndZYwdJYY= 2023-05-31T21:03:24.349 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 21:03:24 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:hQX9x/9TKutkPztcMvndZYwdJYY=', 'Content-Length': '0'} 2023-05-31T21:03:24.351 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:59:45 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000038cd6f960b7cc0eb-006477b61c-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 21:03:24 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:03:24.351 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj87 2023-05-31T21:03:24.352 DEBUG:boto:path=/pklxkd-61/obj87 2023-05-31T21:03:24.352 DEBUG:boto:auth_path=/pklxkd-61/obj87 2023-05-31T21:03:24.353 DEBUG:boto:path=/pklxkd-61/obj87?versionId=null 2023-05-31T21:03:24.353 DEBUG:boto:auth_path=/pklxkd-61/obj87?versionId=null 2023-05-31T21:03:24.353 DEBUG:boto:Method: GET 2023-05-31T21:03:24.354 DEBUG:boto:Path: /pklxkd-61/obj87?versionId=null 2023-05-31T21:03:24.354 DEBUG:boto:Data: 2023-05-31T21:03:24.354 DEBUG:boto:Headers: {} 2023-05-31T21:03:24.355 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T21:03:24.355 DEBUG:boto:Port: 8001 2023-05-31T21:03:24.355 DEBUG:boto:Params: {} 2023-05-31T21:03:24.356 DEBUG:boto:Token: None 2023-05-31T21:03:24.356 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 21:03:24 GMT /pklxkd-61/obj87?versionId=null 2023-05-31T21:03:24.357 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:LCJiIjWN6gtmeW/p3HVG6hDoqT8= 2023-05-31T21:03:24.357 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 21:03:24 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:LCJiIjWN6gtmeW/p3HVG6hDoqT8=', 'Content-Length': '0'} 2023-05-31T21:03:24.359 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:59:45 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002551f86e3bd2c35f-006477b61c-1f4e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 21:03:24 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:03:24.359 DEBUG:boto:path=/pklxkd-61/obj87 2023-05-31T21:03:24.360 DEBUG:boto:auth_path=/pklxkd-61/obj87 2023-05-31T21:03:24.360 DEBUG:boto:path=/pklxkd-61/obj87?versionId=null 2023-05-31T21:03:24.360 DEBUG:boto:auth_path=/pklxkd-61/obj87?versionId=null 2023-05-31T21:03:24.361 DEBUG:boto:Method: GET 2023-05-31T21:03:24.361 DEBUG:boto:Path: /pklxkd-61/obj87?versionId=null 2023-05-31T21:03:24.362 DEBUG:boto:Data: 2023-05-31T21:03:24.362 DEBUG:boto:Headers: {} 2023-05-31T21:03:24.362 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T21:03:24.363 DEBUG:boto:Port: 8000 2023-05-31T21:03:24.363 DEBUG:boto:Params: {} 2023-05-31T21:03:24.363 DEBUG:boto:Token: None 2023-05-31T21:03:24.364 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 21:03:24 GMT /pklxkd-61/obj87?versionId=null 2023-05-31T21:03:24.364 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:LCJiIjWN6gtmeW/p3HVG6hDoqT8= 2023-05-31T21:03:24.365 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 21:03:24 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:LCJiIjWN6gtmeW/p3HVG6hDoqT8=', 'Content-Length': '0'} 2023-05-31T21:03:24.367 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:59:45 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d504038820fb084c-006477b61c-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 21:03:24 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:03:24.368 DEBUG:boto:path=/pklxkd-61/ 2023-05-31T21:03:24.368 DEBUG:boto:auth_path=/pklxkd-61/ 2023-05-31T21:03:24.368 DEBUG:boto:path=/pklxkd-61/?prefix=obj87 2023-05-31T21:03:24.369 DEBUG:boto:auth_path=/pklxkd-61/?prefix=obj87 2023-05-31T21:03:24.369 DEBUG:boto:Method: GET 2023-05-31T21:03:24.369 DEBUG:boto:Path: /pklxkd-61/?prefix=obj87 2023-05-31T21:03:24.370 DEBUG:boto:Data: 2023-05-31T21:03:24.370 DEBUG:boto:Headers: {} 2023-05-31T21:03:24.371 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T21:03:24.371 DEBUG:boto:Port: 8001 2023-05-31T21:03:24.371 DEBUG:boto:Params: {} 2023-05-31T21:03:24.372 DEBUG:boto:Token: None 2023-05-31T21:03:24.372 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 21:03:24 GMT /pklxkd-61/ 2023-05-31T21:03:24.372 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:Y+z1tdIY0pnnJpqEt5b8+XUlWS4= 2023-05-31T21:03:24.373 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 21:03:24 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:Y+z1tdIY0pnnJpqEt5b8+XUlWS4=', 'Content-Length': '0'} 2023-05-31T21:03:24.375 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000662fa1bae989221d-006477b61c-1f4e-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 21:03:24 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:03:24.376 DEBUG:boto:b'pklxkd-61obj871000falseobj872023-05-31T20:59:45.959Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T21:03:24.376 DEBUG:boto:path=/pklxkd-61/ 2023-05-31T21:03:24.377 DEBUG:boto:auth_path=/pklxkd-61/ 2023-05-31T21:03:24.377 DEBUG:boto:path=/pklxkd-61/?prefix=obj87 2023-05-31T21:03:24.377 DEBUG:boto:auth_path=/pklxkd-61/?prefix=obj87 2023-05-31T21:03:24.378 DEBUG:boto:Method: GET 2023-05-31T21:03:24.378 DEBUG:boto:Path: /pklxkd-61/?prefix=obj87 2023-05-31T21:03:24.378 DEBUG:boto:Data: 2023-05-31T21:03:24.379 DEBUG:boto:Headers: {} 2023-05-31T21:03:24.379 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T21:03:24.380 DEBUG:boto:Port: 8000 2023-05-31T21:03:24.380 DEBUG:boto:Params: {} 2023-05-31T21:03:24.380 DEBUG:boto:Token: None 2023-05-31T21:03:24.381 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 21:03:24 GMT /pklxkd-61/ 2023-05-31T21:03:24.381 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:Y+z1tdIY0pnnJpqEt5b8+XUlWS4= 2023-05-31T21:03:24.382 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 21:03:24 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:Y+z1tdIY0pnnJpqEt5b8+XUlWS4=', 'Content-Length': '0'} 2023-05-31T21:03:24.385 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e9ada051e53e0f88-006477b61c-1630-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 21:03:24 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:03:24.386 DEBUG:boto:b'pklxkd-61obj871000falseobj872023-05-31T20:59:45.959Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T21:03:24.387 DEBUG:boto:path=/pklxkd-61/obj87 2023-05-31T21:03:24.387 DEBUG:boto:auth_path=/pklxkd-61/obj87 2023-05-31T21:03:24.387 DEBUG:boto:Method: HEAD 2023-05-31T21:03:24.388 DEBUG:boto:Path: /pklxkd-61/obj87 2023-05-31T21:03:24.388 DEBUG:boto:Data: 2023-05-31T21:03:24.388 DEBUG:boto:Headers: {} 2023-05-31T21:03:24.389 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T21:03:24.389 DEBUG:boto:Port: 8001 2023-05-31T21:03:24.389 DEBUG:boto:Params: {} 2023-05-31T21:03:24.390 DEBUG:boto:Token: None 2023-05-31T21:03:24.390 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 21:03:24 GMT /pklxkd-61/obj87 2023-05-31T21:03:24.390 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:wD04wonagHA9O33+scXMzz5CZnY= 2023-05-31T21:03:24.391 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 21:03:24 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:wD04wonagHA9O33+scXMzz5CZnY=', 'Content-Length': '0'} 2023-05-31T21:03:24.393 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:59:45 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000024131abfb4366227-006477b61c-1f4e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 21:03:24 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:03:24.393 DEBUG:boto:path=/pklxkd-61/obj87 2023-05-31T21:03:24.394 DEBUG:boto:auth_path=/pklxkd-61/obj87 2023-05-31T21:03:24.394 DEBUG:boto:Method: HEAD 2023-05-31T21:03:24.394 DEBUG:boto:Path: /pklxkd-61/obj87 2023-05-31T21:03:24.395 DEBUG:boto:Data: 2023-05-31T21:03:24.395 DEBUG:boto:Headers: {} 2023-05-31T21:03:24.395 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T21:03:24.396 DEBUG:boto:Port: 8000 2023-05-31T21:03:24.396 DEBUG:boto:Params: {} 2023-05-31T21:03:24.396 DEBUG:boto:Token: None 2023-05-31T21:03:24.397 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 21:03:24 GMT /pklxkd-61/obj87 2023-05-31T21:03:24.397 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:wD04wonagHA9O33+scXMzz5CZnY= 2023-05-31T21:03:24.397 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 21:03:24 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:wD04wonagHA9O33+scXMzz5CZnY=', 'Content-Length': '0'} 2023-05-31T21:03:24.399 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:59:45 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a2cdca1addeaaf57-006477b61c-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 21:03:24 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:03:24.400 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj87 2023-05-31T21:03:24.400 DEBUG:boto:path=/pklxkd-61/obj87 2023-05-31T21:03:24.400 DEBUG:boto:auth_path=/pklxkd-61/obj87 2023-05-31T21:03:24.401 DEBUG:boto:Method: GET 2023-05-31T21:03:24.401 DEBUG:boto:Path: /pklxkd-61/obj87 2023-05-31T21:03:24.401 DEBUG:boto:Data: 2023-05-31T21:03:24.402 DEBUG:boto:Headers: {} 2023-05-31T21:03:24.402 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T21:03:24.402 DEBUG:boto:Port: 8001 2023-05-31T21:03:24.403 DEBUG:boto:Params: {} 2023-05-31T21:03:24.403 DEBUG:boto:Token: None 2023-05-31T21:03:24.404 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 21:03:24 GMT /pklxkd-61/obj87 2023-05-31T21:03:24.404 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:exNuE/CfD3SJPYll1aboCRayjzM= 2023-05-31T21:03:24.404 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 21:03:24 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:exNuE/CfD3SJPYll1aboCRayjzM=', 'Content-Length': '0'} 2023-05-31T21:03:24.406 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:59:45 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000078fdc2ced40c012f-006477b61c-1f4e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 21:03:24 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:03:24.407 DEBUG:boto:path=/pklxkd-61/obj87 2023-05-31T21:03:24.407 DEBUG:boto:auth_path=/pklxkd-61/obj87 2023-05-31T21:03:24.408 DEBUG:boto:Method: GET 2023-05-31T21:03:24.408 DEBUG:boto:Path: /pklxkd-61/obj87 2023-05-31T21:03:24.408 DEBUG:boto:Data: 2023-05-31T21:03:24.409 DEBUG:boto:Headers: {} 2023-05-31T21:03:24.409 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T21:03:24.409 DEBUG:boto:Port: 8000 2023-05-31T21:03:24.410 DEBUG:boto:Params: {} 2023-05-31T21:03:24.410 DEBUG:boto:Token: None 2023-05-31T21:03:24.411 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 21:03:24 GMT /pklxkd-61/obj87 2023-05-31T21:03:24.411 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:exNuE/CfD3SJPYll1aboCRayjzM= 2023-05-31T21:03:24.411 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 21:03:24 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:exNuE/CfD3SJPYll1aboCRayjzM=', 'Content-Length': '0'} 2023-05-31T21:03:24.413 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:59:45 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000017fa360377f7c1c1-006477b61c-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 21:03:24 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:03:24.414 DEBUG:boto:path=/pklxkd-61/ 2023-05-31T21:03:24.414 DEBUG:boto:auth_path=/pklxkd-61/ 2023-05-31T21:03:24.415 DEBUG:boto:path=/pklxkd-61/?prefix=obj87 2023-05-31T21:03:24.415 DEBUG:boto:auth_path=/pklxkd-61/?prefix=obj87 2023-05-31T21:03:24.415 DEBUG:boto:Method: GET 2023-05-31T21:03:24.416 DEBUG:boto:Path: /pklxkd-61/?prefix=obj87 2023-05-31T21:03:24.416 DEBUG:boto:Data: 2023-05-31T21:03:24.416 DEBUG:boto:Headers: {} 2023-05-31T21:03:24.417 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T21:03:24.417 DEBUG:boto:Port: 8001 2023-05-31T21:03:24.417 DEBUG:boto:Params: {} 2023-05-31T21:03:24.418 DEBUG:boto:Token: None 2023-05-31T21:03:24.418 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 21:03:24 GMT /pklxkd-61/ 2023-05-31T21:03:24.419 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:Y+z1tdIY0pnnJpqEt5b8+XUlWS4= 2023-05-31T21:03:24.419 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 21:03:24 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:Y+z1tdIY0pnnJpqEt5b8+XUlWS4=', 'Content-Length': '0'} 2023-05-31T21:03:24.422 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000019a552d2832c044d-006477b61c-1f4e-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 21:03:24 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:03:24.422 DEBUG:boto:b'pklxkd-61obj871000falseobj872023-05-31T20:59:45.959Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T21:03:24.423 DEBUG:boto:path=/pklxkd-61/ 2023-05-31T21:03:24.423 DEBUG:boto:auth_path=/pklxkd-61/ 2023-05-31T21:03:24.423 DEBUG:boto:path=/pklxkd-61/?prefix=obj87 2023-05-31T21:03:24.424 DEBUG:boto:auth_path=/pklxkd-61/?prefix=obj87 2023-05-31T21:03:24.424 DEBUG:boto:Method: GET 2023-05-31T21:03:24.425 DEBUG:boto:Path: /pklxkd-61/?prefix=obj87 2023-05-31T21:03:24.425 DEBUG:boto:Data: 2023-05-31T21:03:24.425 DEBUG:boto:Headers: {} 2023-05-31T21:03:24.426 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T21:03:24.426 DEBUG:boto:Port: 8000 2023-05-31T21:03:24.426 DEBUG:boto:Params: {} 2023-05-31T21:03:24.427 DEBUG:boto:Token: None 2023-05-31T21:03:24.427 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 21:03:24 GMT /pklxkd-61/ 2023-05-31T21:03:24.427 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:Y+z1tdIY0pnnJpqEt5b8+XUlWS4= 2023-05-31T21:03:24.428 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 21:03:24 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:Y+z1tdIY0pnnJpqEt5b8+XUlWS4=', 'Content-Length': '0'} 2023-05-31T21:03:24.432 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000004616ebd97d51c7dc-006477b61c-1630-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 21:03:24 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:03:24.472 DEBUG:boto:b'pklxkd-61obj871000falseobj872023-05-31T20:59:45.959Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T21:03:24.473 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj92 2023-05-31T21:03:24.473 DEBUG:boto:path=/pklxkd-61/obj92 2023-05-31T21:03:24.473 DEBUG:boto:auth_path=/pklxkd-61/obj92 2023-05-31T21:03:24.474 DEBUG:boto:path=/pklxkd-61/obj92?versionId=null 2023-05-31T21:03:24.474 DEBUG:boto:auth_path=/pklxkd-61/obj92?versionId=null 2023-05-31T21:03:24.475 DEBUG:boto:Method: GET 2023-05-31T21:03:24.476 DEBUG:boto:Path: /pklxkd-61/obj92?versionId=null 2023-05-31T21:03:24.476 DEBUG:boto:Data: 2023-05-31T21:03:24.477 DEBUG:boto:Headers: {} 2023-05-31T21:03:24.478 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T21:03:24.479 DEBUG:boto:Port: 8001 2023-05-31T21:03:24.480 DEBUG:boto:Params: {} 2023-05-31T21:03:24.481 DEBUG:boto:Token: None 2023-05-31T21:03:24.481 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 21:03:24 GMT /pklxkd-61/obj92?versionId=null 2023-05-31T21:03:24.482 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:cTk1DtrcgnvwaN7jCUdyjVMHPv8= 2023-05-31T21:03:24.482 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 21:03:24 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:cTk1DtrcgnvwaN7jCUdyjVMHPv8=', 'Content-Length': '0'} 2023-05-31T21:03:24.484 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:59:45 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000076602d986f23f93f-006477b61c-1f4e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 21:03:24 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:03:24.485 DEBUG:boto:path=/pklxkd-61/obj92 2023-05-31T21:03:24.485 DEBUG:boto:auth_path=/pklxkd-61/obj92 2023-05-31T21:03:24.486 DEBUG:boto:path=/pklxkd-61/obj92?versionId=null 2023-05-31T21:03:24.486 DEBUG:boto:auth_path=/pklxkd-61/obj92?versionId=null 2023-05-31T21:03:24.486 DEBUG:boto:Method: GET 2023-05-31T21:03:24.487 DEBUG:boto:Path: /pklxkd-61/obj92?versionId=null 2023-05-31T21:03:24.487 DEBUG:boto:Data: 2023-05-31T21:03:24.487 DEBUG:boto:Headers: {} 2023-05-31T21:03:24.488 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T21:03:24.488 DEBUG:boto:Port: 8000 2023-05-31T21:03:24.489 DEBUG:boto:Params: {} 2023-05-31T21:03:24.489 DEBUG:boto:Token: None 2023-05-31T21:03:24.489 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 21:03:24 GMT /pklxkd-61/obj92?versionId=null 2023-05-31T21:03:24.490 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:cTk1DtrcgnvwaN7jCUdyjVMHPv8= 2023-05-31T21:03:24.490 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 21:03:24 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:cTk1DtrcgnvwaN7jCUdyjVMHPv8=', 'Content-Length': '0'} 2023-05-31T21:03:24.492 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:59:45 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003b6fa6eed0b34e59-006477b61c-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 21:03:24 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:03:24.493 DEBUG:boto:path=/pklxkd-61/obj92 2023-05-31T21:03:24.493 DEBUG:boto:auth_path=/pklxkd-61/obj92 2023-05-31T21:03:24.493 DEBUG:boto:path=/pklxkd-61/obj92?versionId=null 2023-05-31T21:03:24.494 DEBUG:boto:auth_path=/pklxkd-61/obj92?versionId=null 2023-05-31T21:03:24.494 DEBUG:boto:Method: HEAD 2023-05-31T21:03:24.494 DEBUG:boto:Path: /pklxkd-61/obj92?versionId=null 2023-05-31T21:03:24.495 DEBUG:boto:Data: 2023-05-31T21:03:24.495 DEBUG:boto:Headers: {} 2023-05-31T21:03:24.495 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T21:03:24.496 DEBUG:boto:Port: 8001 2023-05-31T21:03:24.496 DEBUG:boto:Params: {} 2023-05-31T21:03:24.497 DEBUG:boto:Token: None 2023-05-31T21:03:24.497 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 21:03:24 GMT /pklxkd-61/obj92?versionId=null 2023-05-31T21:03:24.497 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:4qiej4rRkipeHH6IDAn7s3Yoois= 2023-05-31T21:03:24.498 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 21:03:24 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:4qiej4rRkipeHH6IDAn7s3Yoois=', 'Content-Length': '0'} 2023-05-31T21:03:24.499 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:59:45 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000cdf664451e8baeb2-006477b61c-1f4e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 21:03:24 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:03:24.500 DEBUG:boto:path=/pklxkd-61/obj92 2023-05-31T21:03:24.500 DEBUG:boto:auth_path=/pklxkd-61/obj92 2023-05-31T21:03:24.501 DEBUG:boto:path=/pklxkd-61/obj92?versionId=null 2023-05-31T21:03:24.501 DEBUG:boto:auth_path=/pklxkd-61/obj92?versionId=null 2023-05-31T21:03:24.501 DEBUG:boto:Method: HEAD 2023-05-31T21:03:24.502 DEBUG:boto:Path: /pklxkd-61/obj92?versionId=null 2023-05-31T21:03:24.502 DEBUG:boto:Data: 2023-05-31T21:03:24.502 DEBUG:boto:Headers: {} 2023-05-31T21:03:24.503 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T21:03:24.503 DEBUG:boto:Port: 8000 2023-05-31T21:03:24.503 DEBUG:boto:Params: {} 2023-05-31T21:03:24.504 DEBUG:boto:Token: None 2023-05-31T21:03:24.504 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 21:03:24 GMT /pklxkd-61/obj92?versionId=null 2023-05-31T21:03:24.505 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:4qiej4rRkipeHH6IDAn7s3Yoois= 2023-05-31T21:03:24.505 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 21:03:24 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:4qiej4rRkipeHH6IDAn7s3Yoois=', 'Content-Length': '0'} 2023-05-31T21:03:24.507 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:59:45 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d9d5797134dd9f3d-006477b61c-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 21:03:24 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:03:24.507 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj92 2023-05-31T21:03:24.508 DEBUG:boto:path=/pklxkd-61/obj92 2023-05-31T21:03:24.508 DEBUG:boto:auth_path=/pklxkd-61/obj92 2023-05-31T21:03:24.508 DEBUG:boto:path=/pklxkd-61/obj92?versionId=null 2023-05-31T21:03:24.509 DEBUG:boto:auth_path=/pklxkd-61/obj92?versionId=null 2023-05-31T21:03:24.509 DEBUG:boto:Method: GET 2023-05-31T21:03:24.509 DEBUG:boto:Path: /pklxkd-61/obj92?versionId=null 2023-05-31T21:03:24.510 DEBUG:boto:Data: 2023-05-31T21:03:24.510 DEBUG:boto:Headers: {} 2023-05-31T21:03:24.511 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T21:03:24.511 DEBUG:boto:Port: 8001 2023-05-31T21:03:24.511 DEBUG:boto:Params: {} 2023-05-31T21:03:24.512 DEBUG:boto:Token: None 2023-05-31T21:03:24.512 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 21:03:24 GMT /pklxkd-61/obj92?versionId=null 2023-05-31T21:03:24.512 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:cTk1DtrcgnvwaN7jCUdyjVMHPv8= 2023-05-31T21:03:24.513 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 21:03:24 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:cTk1DtrcgnvwaN7jCUdyjVMHPv8=', 'Content-Length': '0'} 2023-05-31T21:03:24.515 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:59:45 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004e84858502a2fe94-006477b61c-1f4e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 21:03:24 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:03:24.560 DEBUG:boto:path=/pklxkd-61/obj92 2023-05-31T21:03:24.561 DEBUG:boto:auth_path=/pklxkd-61/obj92 2023-05-31T21:03:24.561 DEBUG:boto:path=/pklxkd-61/obj92?versionId=null 2023-05-31T21:03:24.562 DEBUG:boto:auth_path=/pklxkd-61/obj92?versionId=null 2023-05-31T21:03:24.562 DEBUG:boto:Method: GET 2023-05-31T21:03:24.562 DEBUG:boto:Path: /pklxkd-61/obj92?versionId=null 2023-05-31T21:03:24.563 DEBUG:boto:Data: 2023-05-31T21:03:24.563 DEBUG:boto:Headers: {} 2023-05-31T21:03:24.564 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T21:03:24.564 DEBUG:boto:Port: 8000 2023-05-31T21:03:24.564 DEBUG:boto:Params: {} 2023-05-31T21:03:24.565 DEBUG:boto:Token: None 2023-05-31T21:03:24.565 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 21:03:24 GMT /pklxkd-61/obj92?versionId=null 2023-05-31T21:03:24.566 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:cTk1DtrcgnvwaN7jCUdyjVMHPv8= 2023-05-31T21:03:24.566 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 21:03:24 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:cTk1DtrcgnvwaN7jCUdyjVMHPv8=', 'Content-Length': '0'} 2023-05-31T21:03:24.568 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:59:45 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000232ebc0195b7c451-006477b61c-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 21:03:24 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:03:24.569 DEBUG:boto:path=/pklxkd-61/ 2023-05-31T21:03:24.569 DEBUG:boto:auth_path=/pklxkd-61/ 2023-05-31T21:03:24.570 DEBUG:boto:path=/pklxkd-61/?prefix=obj92 2023-05-31T21:03:24.570 DEBUG:boto:auth_path=/pklxkd-61/?prefix=obj92 2023-05-31T21:03:24.570 DEBUG:boto:Method: GET 2023-05-31T21:03:24.571 DEBUG:boto:Path: /pklxkd-61/?prefix=obj92 2023-05-31T21:03:24.571 DEBUG:boto:Data: 2023-05-31T21:03:24.571 DEBUG:boto:Headers: {} 2023-05-31T21:03:24.572 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T21:03:24.572 DEBUG:boto:Port: 8001 2023-05-31T21:03:24.573 DEBUG:boto:Params: {} 2023-05-31T21:03:24.573 DEBUG:boto:Token: None 2023-05-31T21:03:24.573 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 21:03:24 GMT /pklxkd-61/ 2023-05-31T21:03:24.574 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:Y+z1tdIY0pnnJpqEt5b8+XUlWS4= 2023-05-31T21:03:24.574 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 21:03:24 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:Y+z1tdIY0pnnJpqEt5b8+XUlWS4=', 'Content-Length': '0'} 2023-05-31T21:03:24.577 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000193beeb0f677edb2-006477b61c-1f4e-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 21:03:24 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:03:24.577 DEBUG:boto:b'pklxkd-61obj921000falseobj922023-05-31T20:59:45.616Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T21:03:24.578 DEBUG:boto:path=/pklxkd-61/ 2023-05-31T21:03:24.578 DEBUG:boto:auth_path=/pklxkd-61/ 2023-05-31T21:03:24.579 DEBUG:boto:path=/pklxkd-61/?prefix=obj92 2023-05-31T21:03:24.579 DEBUG:boto:auth_path=/pklxkd-61/?prefix=obj92 2023-05-31T21:03:24.579 DEBUG:boto:Method: GET 2023-05-31T21:03:24.580 DEBUG:boto:Path: /pklxkd-61/?prefix=obj92 2023-05-31T21:03:24.580 DEBUG:boto:Data: 2023-05-31T21:03:24.580 DEBUG:boto:Headers: {} 2023-05-31T21:03:24.581 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T21:03:24.581 DEBUG:boto:Port: 8000 2023-05-31T21:03:24.581 DEBUG:boto:Params: {} 2023-05-31T21:03:24.582 DEBUG:boto:Token: None 2023-05-31T21:03:24.582 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 21:03:24 GMT /pklxkd-61/ 2023-05-31T21:03:24.583 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:Y+z1tdIY0pnnJpqEt5b8+XUlWS4= 2023-05-31T21:03:24.584 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 21:03:24 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:Y+z1tdIY0pnnJpqEt5b8+XUlWS4=', 'Content-Length': '0'} 2023-05-31T21:03:24.589 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e5637c68376eeb7d-006477b61c-1630-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 21:03:24 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:03:24.632 DEBUG:boto:b'pklxkd-61obj921000falseobj922023-05-31T20:59:45.616Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T21:03:24.633 DEBUG:boto:path=/pklxkd-61/obj92 2023-05-31T21:03:24.633 DEBUG:boto:auth_path=/pklxkd-61/obj92 2023-05-31T21:03:24.634 DEBUG:boto:Method: HEAD 2023-05-31T21:03:24.634 DEBUG:boto:Path: /pklxkd-61/obj92 2023-05-31T21:03:24.635 DEBUG:boto:Data: 2023-05-31T21:03:24.635 DEBUG:boto:Headers: {} 2023-05-31T21:03:24.635 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T21:03:24.636 DEBUG:boto:Port: 8001 2023-05-31T21:03:24.636 DEBUG:boto:Params: {} 2023-05-31T21:03:24.636 DEBUG:boto:Token: None 2023-05-31T21:03:24.637 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 21:03:24 GMT /pklxkd-61/obj92 2023-05-31T21:03:24.637 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:Slmj6lQTMj0gUm0JLvKgV39AqLw= 2023-05-31T21:03:24.638 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 21:03:24 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:Slmj6lQTMj0gUm0JLvKgV39AqLw=', 'Content-Length': '0'} 2023-05-31T21:03:24.639 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:59:45 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000095aae8253834b067-006477b61c-1f4e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 21:03:24 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:03:24.640 DEBUG:boto:path=/pklxkd-61/obj92 2023-05-31T21:03:24.640 DEBUG:boto:auth_path=/pklxkd-61/obj92 2023-05-31T21:03:24.641 DEBUG:boto:Method: HEAD 2023-05-31T21:03:24.641 DEBUG:boto:Path: /pklxkd-61/obj92 2023-05-31T21:03:24.641 DEBUG:boto:Data: 2023-05-31T21:03:24.642 DEBUG:boto:Headers: {} 2023-05-31T21:03:24.642 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T21:03:24.642 DEBUG:boto:Port: 8000 2023-05-31T21:03:24.643 DEBUG:boto:Params: {} 2023-05-31T21:03:24.643 DEBUG:boto:Token: None 2023-05-31T21:03:24.644 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 21:03:24 GMT /pklxkd-61/obj92 2023-05-31T21:03:24.644 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:Slmj6lQTMj0gUm0JLvKgV39AqLw= 2023-05-31T21:03:24.644 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 21:03:24 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:Slmj6lQTMj0gUm0JLvKgV39AqLw=', 'Content-Length': '0'} 2023-05-31T21:03:24.646 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:59:45 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001239f381ec5c3dec-006477b61c-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 21:03:24 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:03:24.647 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj92 2023-05-31T21:03:24.647 DEBUG:boto:path=/pklxkd-61/obj92 2023-05-31T21:03:24.648 DEBUG:boto:auth_path=/pklxkd-61/obj92 2023-05-31T21:03:24.648 DEBUG:boto:Method: GET 2023-05-31T21:03:24.648 DEBUG:boto:Path: /pklxkd-61/obj92 2023-05-31T21:03:24.649 DEBUG:boto:Data: 2023-05-31T21:03:24.650 DEBUG:boto:Headers: {} 2023-05-31T21:03:24.650 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T21:03:24.651 DEBUG:boto:Port: 8001 2023-05-31T21:03:24.652 DEBUG:boto:Params: {} 2023-05-31T21:03:24.653 DEBUG:boto:Token: None 2023-05-31T21:03:24.654 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 21:03:24 GMT /pklxkd-61/obj92 2023-05-31T21:03:24.655 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:gVg8+gTrRMhxvhya7ktsvZhMaG8= 2023-05-31T21:03:24.655 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 21:03:24 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:gVg8+gTrRMhxvhya7ktsvZhMaG8=', 'Content-Length': '0'} 2023-05-31T21:03:24.657 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:59:45 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006753ee98515c722a-006477b61c-1f4e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 21:03:24 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:03:24.658 DEBUG:boto:path=/pklxkd-61/obj92 2023-05-31T21:03:24.658 DEBUG:boto:auth_path=/pklxkd-61/obj92 2023-05-31T21:03:24.659 DEBUG:boto:Method: GET 2023-05-31T21:03:24.659 DEBUG:boto:Path: /pklxkd-61/obj92 2023-05-31T21:03:24.660 DEBUG:boto:Data: 2023-05-31T21:03:24.660 DEBUG:boto:Headers: {} 2023-05-31T21:03:24.660 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T21:03:24.661 DEBUG:boto:Port: 8000 2023-05-31T21:03:24.661 DEBUG:boto:Params: {} 2023-05-31T21:03:24.661 DEBUG:boto:Token: None 2023-05-31T21:03:24.662 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 21:03:24 GMT /pklxkd-61/obj92 2023-05-31T21:03:24.662 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:gVg8+gTrRMhxvhya7ktsvZhMaG8= 2023-05-31T21:03:24.663 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 21:03:24 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:gVg8+gTrRMhxvhya7ktsvZhMaG8=', 'Content-Length': '0'} 2023-05-31T21:03:24.664 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:59:45 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007e87af2d01de7409-006477b61c-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 21:03:24 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:03:24.665 DEBUG:boto:path=/pklxkd-61/ 2023-05-31T21:03:24.666 DEBUG:boto:auth_path=/pklxkd-61/ 2023-05-31T21:03:24.666 DEBUG:boto:path=/pklxkd-61/?prefix=obj92 2023-05-31T21:03:24.666 DEBUG:boto:auth_path=/pklxkd-61/?prefix=obj92 2023-05-31T21:03:24.667 DEBUG:boto:Method: GET 2023-05-31T21:03:24.667 DEBUG:boto:Path: /pklxkd-61/?prefix=obj92 2023-05-31T21:03:24.668 DEBUG:boto:Data: 2023-05-31T21:03:24.668 DEBUG:boto:Headers: {} 2023-05-31T21:03:24.668 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T21:03:24.669 DEBUG:boto:Port: 8001 2023-05-31T21:03:24.669 DEBUG:boto:Params: {} 2023-05-31T21:03:24.670 DEBUG:boto:Token: None 2023-05-31T21:03:24.670 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 21:03:24 GMT /pklxkd-61/ 2023-05-31T21:03:24.670 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:Y+z1tdIY0pnnJpqEt5b8+XUlWS4= 2023-05-31T21:03:24.671 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 21:03:24 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:Y+z1tdIY0pnnJpqEt5b8+XUlWS4=', 'Content-Length': '0'} 2023-05-31T21:03:24.673 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000661ba0e0a751135f-006477b61c-1f4e-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 21:03:24 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:03:24.674 DEBUG:boto:b'pklxkd-61obj921000falseobj922023-05-31T20:59:45.616Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T21:03:24.674 DEBUG:boto:path=/pklxkd-61/ 2023-05-31T21:03:24.675 DEBUG:boto:auth_path=/pklxkd-61/ 2023-05-31T21:03:24.675 DEBUG:boto:path=/pklxkd-61/?prefix=obj92 2023-05-31T21:03:24.675 DEBUG:boto:auth_path=/pklxkd-61/?prefix=obj92 2023-05-31T21:03:24.676 DEBUG:boto:Method: GET 2023-05-31T21:03:24.676 DEBUG:boto:Path: /pklxkd-61/?prefix=obj92 2023-05-31T21:03:24.677 DEBUG:boto:Data: 2023-05-31T21:03:24.677 DEBUG:boto:Headers: {} 2023-05-31T21:03:24.677 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T21:03:24.678 DEBUG:boto:Port: 8000 2023-05-31T21:03:24.678 DEBUG:boto:Params: {} 2023-05-31T21:03:24.678 DEBUG:boto:Token: None 2023-05-31T21:03:24.679 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 21:03:24 GMT /pklxkd-61/ 2023-05-31T21:03:24.679 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:Y+z1tdIY0pnnJpqEt5b8+XUlWS4= 2023-05-31T21:03:24.680 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 21:03:24 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:Y+z1tdIY0pnnJpqEt5b8+XUlWS4=', 'Content-Length': '0'} 2023-05-31T21:03:24.683 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c0ec24169abd4861-006477b61c-1630-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 21:03:24 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:03:24.684 DEBUG:boto:b'pklxkd-61obj921000falseobj922023-05-31T20:59:45.616Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T21:03:24.685 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj93 2023-05-31T21:03:24.685 DEBUG:boto:path=/pklxkd-61/obj93 2023-05-31T21:03:24.685 DEBUG:boto:auth_path=/pklxkd-61/obj93 2023-05-31T21:03:24.686 DEBUG:boto:path=/pklxkd-61/obj93?versionId=null 2023-05-31T21:03:24.686 DEBUG:boto:auth_path=/pklxkd-61/obj93?versionId=null 2023-05-31T21:03:24.686 DEBUG:boto:Method: GET 2023-05-31T21:03:24.687 DEBUG:boto:Path: /pklxkd-61/obj93?versionId=null 2023-05-31T21:03:24.687 DEBUG:boto:Data: 2023-05-31T21:03:24.688 DEBUG:boto:Headers: {} 2023-05-31T21:03:24.688 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T21:03:24.688 DEBUG:boto:Port: 8001 2023-05-31T21:03:24.689 DEBUG:boto:Params: {} 2023-05-31T21:03:24.689 DEBUG:boto:Token: None 2023-05-31T21:03:24.689 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 21:03:24 GMT /pklxkd-61/obj93?versionId=null 2023-05-31T21:03:24.690 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:1UKKBqWDUsbgWv+U3TOvaKCtwQs= 2023-05-31T21:03:24.690 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 21:03:24 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:1UKKBqWDUsbgWv+U3TOvaKCtwQs=', 'Content-Length': '0'} 2023-05-31T21:03:24.692 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:59:46 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000780a3aac6efcfebc-006477b61c-1f4e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 21:03:24 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:03:24.693 DEBUG:boto:path=/pklxkd-61/obj93 2023-05-31T21:03:24.693 DEBUG:boto:auth_path=/pklxkd-61/obj93 2023-05-31T21:03:24.694 DEBUG:boto:path=/pklxkd-61/obj93?versionId=null 2023-05-31T21:03:24.694 DEBUG:boto:auth_path=/pklxkd-61/obj93?versionId=null 2023-05-31T21:03:24.694 DEBUG:boto:Method: GET 2023-05-31T21:03:24.695 DEBUG:boto:Path: /pklxkd-61/obj93?versionId=null 2023-05-31T21:03:24.695 DEBUG:boto:Data: 2023-05-31T21:03:24.695 DEBUG:boto:Headers: {} 2023-05-31T21:03:24.696 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T21:03:24.696 DEBUG:boto:Port: 8000 2023-05-31T21:03:24.697 DEBUG:boto:Params: {} 2023-05-31T21:03:24.697 DEBUG:boto:Token: None 2023-05-31T21:03:24.697 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 21:03:24 GMT /pklxkd-61/obj93?versionId=null 2023-05-31T21:03:24.698 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:1UKKBqWDUsbgWv+U3TOvaKCtwQs= 2023-05-31T21:03:24.698 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 21:03:24 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:1UKKBqWDUsbgWv+U3TOvaKCtwQs=', 'Content-Length': '0'} 2023-05-31T21:03:24.700 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:59:46 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000086a89737b21f7235-006477b61c-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 21:03:24 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:03:24.701 DEBUG:boto:path=/pklxkd-61/obj93 2023-05-31T21:03:24.701 DEBUG:boto:auth_path=/pklxkd-61/obj93 2023-05-31T21:03:24.701 DEBUG:boto:path=/pklxkd-61/obj93?versionId=null 2023-05-31T21:03:24.702 DEBUG:boto:auth_path=/pklxkd-61/obj93?versionId=null 2023-05-31T21:03:24.702 DEBUG:boto:Method: HEAD 2023-05-31T21:03:24.703 DEBUG:boto:Path: /pklxkd-61/obj93?versionId=null 2023-05-31T21:03:24.703 DEBUG:boto:Data: 2023-05-31T21:03:24.703 DEBUG:boto:Headers: {} 2023-05-31T21:03:24.704 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T21:03:24.704 DEBUG:boto:Port: 8001 2023-05-31T21:03:24.704 DEBUG:boto:Params: {} 2023-05-31T21:03:24.705 DEBUG:boto:Token: None 2023-05-31T21:03:24.705 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 21:03:24 GMT /pklxkd-61/obj93?versionId=null 2023-05-31T21:03:24.706 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:L9E/MEGr48VHegtZj6S2HPUzepc= 2023-05-31T21:03:24.706 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 21:03:24 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:L9E/MEGr48VHegtZj6S2HPUzepc=', 'Content-Length': '0'} 2023-05-31T21:03:24.708 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:59:46 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ed6395db947c6995-006477b61c-1f4e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 21:03:24 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:03:24.708 DEBUG:boto:path=/pklxkd-61/obj93 2023-05-31T21:03:24.709 DEBUG:boto:auth_path=/pklxkd-61/obj93 2023-05-31T21:03:24.709 DEBUG:boto:path=/pklxkd-61/obj93?versionId=null 2023-05-31T21:03:24.709 DEBUG:boto:auth_path=/pklxkd-61/obj93?versionId=null 2023-05-31T21:03:24.710 DEBUG:boto:Method: HEAD 2023-05-31T21:03:24.710 DEBUG:boto:Path: /pklxkd-61/obj93?versionId=null 2023-05-31T21:03:24.710 DEBUG:boto:Data: 2023-05-31T21:03:24.711 DEBUG:boto:Headers: {} 2023-05-31T21:03:24.711 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T21:03:24.711 DEBUG:boto:Port: 8000 2023-05-31T21:03:24.712 DEBUG:boto:Params: {} 2023-05-31T21:03:24.712 DEBUG:boto:Token: None 2023-05-31T21:03:24.712 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 21:03:24 GMT /pklxkd-61/obj93?versionId=null 2023-05-31T21:03:24.713 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:L9E/MEGr48VHegtZj6S2HPUzepc= 2023-05-31T21:03:24.713 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 21:03:24 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:L9E/MEGr48VHegtZj6S2HPUzepc=', 'Content-Length': '0'} 2023-05-31T21:03:24.715 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:59:46 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003fc2740c6f98b5cf-006477b61c-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 21:03:24 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:03:24.716 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj93 2023-05-31T21:03:24.716 DEBUG:boto:path=/pklxkd-61/obj93 2023-05-31T21:03:24.716 DEBUG:boto:auth_path=/pklxkd-61/obj93 2023-05-31T21:03:24.717 DEBUG:boto:path=/pklxkd-61/obj93?versionId=null 2023-05-31T21:03:24.717 DEBUG:boto:auth_path=/pklxkd-61/obj93?versionId=null 2023-05-31T21:03:24.717 DEBUG:boto:Method: GET 2023-05-31T21:03:24.718 DEBUG:boto:Path: /pklxkd-61/obj93?versionId=null 2023-05-31T21:03:24.718 DEBUG:boto:Data: 2023-05-31T21:03:24.719 DEBUG:boto:Headers: {} 2023-05-31T21:03:24.719 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T21:03:24.719 DEBUG:boto:Port: 8001 2023-05-31T21:03:24.720 DEBUG:boto:Params: {} 2023-05-31T21:03:24.720 DEBUG:boto:Token: None 2023-05-31T21:03:24.721 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 21:03:24 GMT /pklxkd-61/obj93?versionId=null 2023-05-31T21:03:24.721 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:1UKKBqWDUsbgWv+U3TOvaKCtwQs= 2023-05-31T21:03:24.722 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 21:03:24 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:1UKKBqWDUsbgWv+U3TOvaKCtwQs=', 'Content-Length': '0'} 2023-05-31T21:03:24.723 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:59:46 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000614216c96ec51d1a-006477b61c-1f4e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 21:03:24 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:03:24.764 DEBUG:boto:path=/pklxkd-61/obj93 2023-05-31T21:03:24.765 DEBUG:boto:auth_path=/pklxkd-61/obj93 2023-05-31T21:03:24.765 DEBUG:boto:path=/pklxkd-61/obj93?versionId=null 2023-05-31T21:03:24.765 DEBUG:boto:auth_path=/pklxkd-61/obj93?versionId=null 2023-05-31T21:03:24.766 DEBUG:boto:Method: GET 2023-05-31T21:03:24.766 DEBUG:boto:Path: /pklxkd-61/obj93?versionId=null 2023-05-31T21:03:24.766 DEBUG:boto:Data: 2023-05-31T21:03:24.767 DEBUG:boto:Headers: {} 2023-05-31T21:03:24.767 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T21:03:24.767 DEBUG:boto:Port: 8000 2023-05-31T21:03:24.768 DEBUG:boto:Params: {} 2023-05-31T21:03:24.768 DEBUG:boto:Token: None 2023-05-31T21:03:24.769 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 21:03:24 GMT /pklxkd-61/obj93?versionId=null 2023-05-31T21:03:24.769 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:1UKKBqWDUsbgWv+U3TOvaKCtwQs= 2023-05-31T21:03:24.770 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 21:03:24 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:1UKKBqWDUsbgWv+U3TOvaKCtwQs=', 'Content-Length': '0'} 2023-05-31T21:03:24.772 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:59:46 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e8be8ecf4a2cf5b4-006477b61c-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 21:03:24 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:03:24.772 DEBUG:boto:path=/pklxkd-61/ 2023-05-31T21:03:24.773 DEBUG:boto:auth_path=/pklxkd-61/ 2023-05-31T21:03:24.773 DEBUG:boto:path=/pklxkd-61/?prefix=obj93 2023-05-31T21:03:24.773 DEBUG:boto:auth_path=/pklxkd-61/?prefix=obj93 2023-05-31T21:03:24.774 DEBUG:boto:Method: GET 2023-05-31T21:03:24.774 DEBUG:boto:Path: /pklxkd-61/?prefix=obj93 2023-05-31T21:03:24.774 DEBUG:boto:Data: 2023-05-31T21:03:24.775 DEBUG:boto:Headers: {} 2023-05-31T21:03:24.775 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T21:03:24.775 DEBUG:boto:Port: 8001 2023-05-31T21:03:24.776 DEBUG:boto:Params: {} 2023-05-31T21:03:24.776 DEBUG:boto:Token: None 2023-05-31T21:03:24.777 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 21:03:24 GMT /pklxkd-61/ 2023-05-31T21:03:24.777 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:Y+z1tdIY0pnnJpqEt5b8+XUlWS4= 2023-05-31T21:03:24.777 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 21:03:24 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:Y+z1tdIY0pnnJpqEt5b8+XUlWS4=', 'Content-Length': '0'} 2023-05-31T21:03:24.780 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d7e7bceb7eac8c86-006477b61c-1f4e-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 21:03:24 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:03:24.780 DEBUG:boto:b'pklxkd-61obj931000falseobj932023-05-31T20:59:46.358Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T21:03:24.781 DEBUG:boto:path=/pklxkd-61/ 2023-05-31T21:03:24.781 DEBUG:boto:auth_path=/pklxkd-61/ 2023-05-31T21:03:24.782 DEBUG:boto:path=/pklxkd-61/?prefix=obj93 2023-05-31T21:03:24.782 DEBUG:boto:auth_path=/pklxkd-61/?prefix=obj93 2023-05-31T21:03:24.782 DEBUG:boto:Method: GET 2023-05-31T21:03:24.783 DEBUG:boto:Path: /pklxkd-61/?prefix=obj93 2023-05-31T21:03:24.783 DEBUG:boto:Data: 2023-05-31T21:03:24.783 DEBUG:boto:Headers: {} 2023-05-31T21:03:24.784 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T21:03:24.784 DEBUG:boto:Port: 8000 2023-05-31T21:03:24.785 DEBUG:boto:Params: {} 2023-05-31T21:03:24.785 DEBUG:boto:Token: None 2023-05-31T21:03:24.785 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 21:03:24 GMT /pklxkd-61/ 2023-05-31T21:03:24.786 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:Y+z1tdIY0pnnJpqEt5b8+XUlWS4= 2023-05-31T21:03:24.786 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 21:03:24 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:Y+z1tdIY0pnnJpqEt5b8+XUlWS4=', 'Content-Length': '0'} 2023-05-31T21:03:24.790 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e8341c47498c41d6-006477b61c-1630-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 21:03:24 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:03:24.790 DEBUG:boto:b'pklxkd-61obj931000falseobj932023-05-31T20:59:46.358Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T21:03:24.791 DEBUG:boto:path=/pklxkd-61/obj93 2023-05-31T21:03:24.791 DEBUG:boto:auth_path=/pklxkd-61/obj93 2023-05-31T21:03:24.792 DEBUG:boto:Method: HEAD 2023-05-31T21:03:24.792 DEBUG:boto:Path: /pklxkd-61/obj93 2023-05-31T21:03:24.792 DEBUG:boto:Data: 2023-05-31T21:03:24.793 DEBUG:boto:Headers: {} 2023-05-31T21:03:24.793 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T21:03:24.793 DEBUG:boto:Port: 8001 2023-05-31T21:03:24.794 DEBUG:boto:Params: {} 2023-05-31T21:03:24.794 DEBUG:boto:Token: None 2023-05-31T21:03:24.794 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 21:03:24 GMT /pklxkd-61/obj93 2023-05-31T21:03:24.795 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:6f2gnxmaup40pXEHAodIOQar8B4= 2023-05-31T21:03:24.795 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 21:03:24 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:6f2gnxmaup40pXEHAodIOQar8B4=', 'Content-Length': '0'} 2023-05-31T21:03:24.797 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:59:46 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003e5daaeff58cfca4-006477b61c-1f4e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 21:03:24 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:03:24.798 DEBUG:boto:path=/pklxkd-61/obj93 2023-05-31T21:03:24.798 DEBUG:boto:auth_path=/pklxkd-61/obj93 2023-05-31T21:03:24.798 DEBUG:boto:Method: HEAD 2023-05-31T21:03:24.799 DEBUG:boto:Path: /pklxkd-61/obj93 2023-05-31T21:03:24.799 DEBUG:boto:Data: 2023-05-31T21:03:24.799 DEBUG:boto:Headers: {} 2023-05-31T21:03:24.800 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T21:03:24.800 DEBUG:boto:Port: 8000 2023-05-31T21:03:24.800 DEBUG:boto:Params: {} 2023-05-31T21:03:24.801 DEBUG:boto:Token: None 2023-05-31T21:03:24.801 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 21:03:24 GMT /pklxkd-61/obj93 2023-05-31T21:03:24.801 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:6f2gnxmaup40pXEHAodIOQar8B4= 2023-05-31T21:03:24.802 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 21:03:24 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:6f2gnxmaup40pXEHAodIOQar8B4=', 'Content-Length': '0'} 2023-05-31T21:03:24.804 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:59:46 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000faa31c3dea9db725-006477b61c-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 21:03:24 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:03:24.804 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj93 2023-05-31T21:03:24.805 DEBUG:boto:path=/pklxkd-61/obj93 2023-05-31T21:03:24.805 DEBUG:boto:auth_path=/pklxkd-61/obj93 2023-05-31T21:03:24.805 DEBUG:boto:Method: GET 2023-05-31T21:03:24.806 DEBUG:boto:Path: /pklxkd-61/obj93 2023-05-31T21:03:24.806 DEBUG:boto:Data: 2023-05-31T21:03:24.806 DEBUG:boto:Headers: {} 2023-05-31T21:03:24.807 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T21:03:24.807 DEBUG:boto:Port: 8001 2023-05-31T21:03:24.807 DEBUG:boto:Params: {} 2023-05-31T21:03:24.808 DEBUG:boto:Token: None 2023-05-31T21:03:24.808 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 21:03:24 GMT /pklxkd-61/obj93 2023-05-31T21:03:24.809 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:F1l86Gq0N5CCA7ppPA43QK4H8HM= 2023-05-31T21:03:24.809 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 21:03:24 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:F1l86Gq0N5CCA7ppPA43QK4H8HM=', 'Content-Length': '0'} 2023-05-31T21:03:24.811 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:59:46 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000078d29e9334091550-006477b61c-1f4e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 21:03:24 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:03:24.811 DEBUG:boto:path=/pklxkd-61/obj93 2023-05-31T21:03:24.812 DEBUG:boto:auth_path=/pklxkd-61/obj93 2023-05-31T21:03:24.812 DEBUG:boto:Method: GET 2023-05-31T21:03:24.812 DEBUG:boto:Path: /pklxkd-61/obj93 2023-05-31T21:03:24.813 DEBUG:boto:Data: 2023-05-31T21:03:24.813 DEBUG:boto:Headers: {} 2023-05-31T21:03:24.813 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T21:03:24.814 DEBUG:boto:Port: 8000 2023-05-31T21:03:24.814 DEBUG:boto:Params: {} 2023-05-31T21:03:24.815 DEBUG:boto:Token: None 2023-05-31T21:03:24.815 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 21:03:24 GMT /pklxkd-61/obj93 2023-05-31T21:03:24.815 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:F1l86Gq0N5CCA7ppPA43QK4H8HM= 2023-05-31T21:03:24.816 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 21:03:24 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:F1l86Gq0N5CCA7ppPA43QK4H8HM=', 'Content-Length': '0'} 2023-05-31T21:03:24.817 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:59:46 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000060fbc2885d537f08-006477b61c-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 21:03:24 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:03:24.818 DEBUG:boto:path=/pklxkd-61/ 2023-05-31T21:03:24.819 DEBUG:boto:auth_path=/pklxkd-61/ 2023-05-31T21:03:24.819 DEBUG:boto:path=/pklxkd-61/?prefix=obj93 2023-05-31T21:03:24.819 DEBUG:boto:auth_path=/pklxkd-61/?prefix=obj93 2023-05-31T21:03:24.820 DEBUG:boto:Method: GET 2023-05-31T21:03:24.820 DEBUG:boto:Path: /pklxkd-61/?prefix=obj93 2023-05-31T21:03:24.820 DEBUG:boto:Data: 2023-05-31T21:03:24.821 DEBUG:boto:Headers: {} 2023-05-31T21:03:24.821 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T21:03:24.821 DEBUG:boto:Port: 8001 2023-05-31T21:03:24.822 DEBUG:boto:Params: {} 2023-05-31T21:03:24.822 DEBUG:boto:Token: None 2023-05-31T21:03:24.823 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 21:03:24 GMT /pklxkd-61/ 2023-05-31T21:03:24.823 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:Y+z1tdIY0pnnJpqEt5b8+XUlWS4= 2023-05-31T21:03:24.823 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 21:03:24 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:Y+z1tdIY0pnnJpqEt5b8+XUlWS4=', 'Content-Length': '0'} 2023-05-31T21:03:24.826 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000001e14ae8347481d04-006477b61c-1f4e-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 21:03:24 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:03:24.826 DEBUG:boto:b'pklxkd-61obj931000falseobj932023-05-31T20:59:46.358Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T21:03:24.827 DEBUG:boto:path=/pklxkd-61/ 2023-05-31T21:03:24.827 DEBUG:boto:auth_path=/pklxkd-61/ 2023-05-31T21:03:24.828 DEBUG:boto:path=/pklxkd-61/?prefix=obj93 2023-05-31T21:03:24.828 DEBUG:boto:auth_path=/pklxkd-61/?prefix=obj93 2023-05-31T21:03:24.828 DEBUG:boto:Method: GET 2023-05-31T21:03:24.829 DEBUG:boto:Path: /pklxkd-61/?prefix=obj93 2023-05-31T21:03:24.829 DEBUG:boto:Data: 2023-05-31T21:03:24.830 DEBUG:boto:Headers: {} 2023-05-31T21:03:24.830 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T21:03:24.830 DEBUG:boto:Port: 8000 2023-05-31T21:03:24.831 DEBUG:boto:Params: {} 2023-05-31T21:03:24.831 DEBUG:boto:Token: None 2023-05-31T21:03:24.831 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 21:03:24 GMT /pklxkd-61/ 2023-05-31T21:03:24.832 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:Y+z1tdIY0pnnJpqEt5b8+XUlWS4= 2023-05-31T21:03:24.832 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 21:03:24 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:Y+z1tdIY0pnnJpqEt5b8+XUlWS4=', 'Content-Length': '0'} 2023-05-31T21:03:24.836 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f3ac36ab0fc51a82-006477b61c-1630-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 21:03:24 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:03:24.837 DEBUG:boto:b'pklxkd-61obj931000falseobj932023-05-31T20:59:46.358Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T21:03:24.837 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj95 2023-05-31T21:03:24.838 DEBUG:boto:path=/pklxkd-61/obj95 2023-05-31T21:03:24.838 DEBUG:boto:auth_path=/pklxkd-61/obj95 2023-05-31T21:03:24.838 DEBUG:boto:path=/pklxkd-61/obj95?versionId=null 2023-05-31T21:03:24.839 DEBUG:boto:auth_path=/pklxkd-61/obj95?versionId=null 2023-05-31T21:03:24.839 DEBUG:boto:Method: GET 2023-05-31T21:03:24.839 DEBUG:boto:Path: /pklxkd-61/obj95?versionId=null 2023-05-31T21:03:24.840 DEBUG:boto:Data: 2023-05-31T21:03:24.840 DEBUG:boto:Headers: {} 2023-05-31T21:03:24.841 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T21:03:24.841 DEBUG:boto:Port: 8001 2023-05-31T21:03:24.841 DEBUG:boto:Params: {} 2023-05-31T21:03:24.842 DEBUG:boto:Token: None 2023-05-31T21:03:24.842 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 21:03:24 GMT /pklxkd-61/obj95?versionId=null 2023-05-31T21:03:24.843 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:snsyVy2sniLhn51IsqXvWhHRaP4= 2023-05-31T21:03:24.843 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 21:03:24 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:snsyVy2sniLhn51IsqXvWhHRaP4=', 'Content-Length': '0'} 2023-05-31T21:03:24.845 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:59:45 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000085c2eafde899fa85-006477b61c-1f4e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 21:03:24 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:03:24.845 DEBUG:boto:path=/pklxkd-61/obj95 2023-05-31T21:03:24.846 DEBUG:boto:auth_path=/pklxkd-61/obj95 2023-05-31T21:03:24.846 DEBUG:boto:path=/pklxkd-61/obj95?versionId=null 2023-05-31T21:03:24.847 DEBUG:boto:auth_path=/pklxkd-61/obj95?versionId=null 2023-05-31T21:03:24.847 DEBUG:boto:Method: GET 2023-05-31T21:03:24.847 DEBUG:boto:Path: /pklxkd-61/obj95?versionId=null 2023-05-31T21:03:24.848 DEBUG:boto:Data: 2023-05-31T21:03:24.848 DEBUG:boto:Headers: {} 2023-05-31T21:03:24.849 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T21:03:24.850 DEBUG:boto:Port: 8000 2023-05-31T21:03:24.850 DEBUG:boto:Params: {} 2023-05-31T21:03:24.850 DEBUG:boto:Token: None 2023-05-31T21:03:24.851 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 21:03:24 GMT /pklxkd-61/obj95?versionId=null 2023-05-31T21:03:24.851 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:snsyVy2sniLhn51IsqXvWhHRaP4= 2023-05-31T21:03:24.852 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 21:03:24 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:snsyVy2sniLhn51IsqXvWhHRaP4=', 'Content-Length': '0'} 2023-05-31T21:03:24.853 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:59:45 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006c70d7c417096221-006477b61c-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 21:03:24 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:03:24.854 DEBUG:boto:path=/pklxkd-61/obj95 2023-05-31T21:03:24.854 DEBUG:boto:auth_path=/pklxkd-61/obj95 2023-05-31T21:03:24.855 DEBUG:boto:path=/pklxkd-61/obj95?versionId=null 2023-05-31T21:03:24.855 DEBUG:boto:auth_path=/pklxkd-61/obj95?versionId=null 2023-05-31T21:03:24.855 DEBUG:boto:Method: HEAD 2023-05-31T21:03:24.856 DEBUG:boto:Path: /pklxkd-61/obj95?versionId=null 2023-05-31T21:03:24.856 DEBUG:boto:Data: 2023-05-31T21:03:24.857 DEBUG:boto:Headers: {} 2023-05-31T21:03:24.857 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T21:03:24.857 DEBUG:boto:Port: 8001 2023-05-31T21:03:24.857 DEBUG:boto:Params: {} 2023-05-31T21:03:24.858 DEBUG:boto:Token: None 2023-05-31T21:03:24.858 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 21:03:24 GMT /pklxkd-61/obj95?versionId=null 2023-05-31T21:03:24.859 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:JW7arPpr1AHS7mLC2qbr46JLTmc= 2023-05-31T21:03:24.859 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 21:03:24 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:JW7arPpr1AHS7mLC2qbr46JLTmc=', 'Content-Length': '0'} 2023-05-31T21:03:24.861 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:59:45 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000043f0077777359b53-006477b61c-1f4e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 21:03:24 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:03:24.861 DEBUG:boto:path=/pklxkd-61/obj95 2023-05-31T21:03:24.862 DEBUG:boto:auth_path=/pklxkd-61/obj95 2023-05-31T21:03:24.862 DEBUG:boto:path=/pklxkd-61/obj95?versionId=null 2023-05-31T21:03:24.862 DEBUG:boto:auth_path=/pklxkd-61/obj95?versionId=null 2023-05-31T21:03:24.863 DEBUG:boto:Method: HEAD 2023-05-31T21:03:24.863 DEBUG:boto:Path: /pklxkd-61/obj95?versionId=null 2023-05-31T21:03:24.863 DEBUG:boto:Data: 2023-05-31T21:03:24.864 DEBUG:boto:Headers: {} 2023-05-31T21:03:24.864 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T21:03:24.864 DEBUG:boto:Port: 8000 2023-05-31T21:03:24.865 DEBUG:boto:Params: {} 2023-05-31T21:03:24.865 DEBUG:boto:Token: None 2023-05-31T21:03:24.866 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 21:03:24 GMT /pklxkd-61/obj95?versionId=null 2023-05-31T21:03:24.866 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:JW7arPpr1AHS7mLC2qbr46JLTmc= 2023-05-31T21:03:24.867 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 21:03:24 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:JW7arPpr1AHS7mLC2qbr46JLTmc=', 'Content-Length': '0'} 2023-05-31T21:03:24.868 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:59:45 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000059d1e8a060b8c29f-006477b61c-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 21:03:24 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:03:24.869 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj95 2023-05-31T21:03:24.869 DEBUG:boto:path=/pklxkd-61/obj95 2023-05-31T21:03:24.870 DEBUG:boto:auth_path=/pklxkd-61/obj95 2023-05-31T21:03:24.870 DEBUG:boto:path=/pklxkd-61/obj95?versionId=null 2023-05-31T21:03:24.871 DEBUG:boto:auth_path=/pklxkd-61/obj95?versionId=null 2023-05-31T21:03:24.871 DEBUG:boto:Method: GET 2023-05-31T21:03:24.871 DEBUG:boto:Path: /pklxkd-61/obj95?versionId=null 2023-05-31T21:03:24.872 DEBUG:boto:Data: 2023-05-31T21:03:24.872 DEBUG:boto:Headers: {} 2023-05-31T21:03:24.872 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T21:03:24.873 DEBUG:boto:Port: 8001 2023-05-31T21:03:24.873 DEBUG:boto:Params: {} 2023-05-31T21:03:24.873 DEBUG:boto:Token: None 2023-05-31T21:03:24.874 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 21:03:24 GMT /pklxkd-61/obj95?versionId=null 2023-05-31T21:03:24.874 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:snsyVy2sniLhn51IsqXvWhHRaP4= 2023-05-31T21:03:24.875 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 21:03:24 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:snsyVy2sniLhn51IsqXvWhHRaP4=', 'Content-Length': '0'} 2023-05-31T21:03:24.876 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:59:45 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e72264a6e9764c7e-006477b61c-1f4e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 21:03:24 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:03:24.920 DEBUG:boto:path=/pklxkd-61/obj95 2023-05-31T21:03:24.921 DEBUG:boto:auth_path=/pklxkd-61/obj95 2023-05-31T21:03:24.921 DEBUG:boto:path=/pklxkd-61/obj95?versionId=null 2023-05-31T21:03:24.922 DEBUG:boto:auth_path=/pklxkd-61/obj95?versionId=null 2023-05-31T21:03:24.922 DEBUG:boto:Method: GET 2023-05-31T21:03:24.922 DEBUG:boto:Path: /pklxkd-61/obj95?versionId=null 2023-05-31T21:03:24.923 DEBUG:boto:Data: 2023-05-31T21:03:24.923 DEBUG:boto:Headers: {} 2023-05-31T21:03:24.924 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T21:03:24.924 DEBUG:boto:Port: 8000 2023-05-31T21:03:24.924 DEBUG:boto:Params: {} 2023-05-31T21:03:24.925 DEBUG:boto:Token: None 2023-05-31T21:03:24.925 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 21:03:24 GMT /pklxkd-61/obj95?versionId=null 2023-05-31T21:03:24.926 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:snsyVy2sniLhn51IsqXvWhHRaP4= 2023-05-31T21:03:24.926 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 21:03:24 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:snsyVy2sniLhn51IsqXvWhHRaP4=', 'Content-Length': '0'} 2023-05-31T21:03:24.928 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:59:45 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000160603017146fb2c-006477b61c-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 21:03:24 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:03:24.929 DEBUG:boto:path=/pklxkd-61/ 2023-05-31T21:03:24.929 DEBUG:boto:auth_path=/pklxkd-61/ 2023-05-31T21:03:24.929 DEBUG:boto:path=/pklxkd-61/?prefix=obj95 2023-05-31T21:03:24.930 DEBUG:boto:auth_path=/pklxkd-61/?prefix=obj95 2023-05-31T21:03:24.930 DEBUG:boto:Method: GET 2023-05-31T21:03:24.930 DEBUG:boto:Path: /pklxkd-61/?prefix=obj95 2023-05-31T21:03:24.931 DEBUG:boto:Data: 2023-05-31T21:03:24.931 DEBUG:boto:Headers: {} 2023-05-31T21:03:24.932 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T21:03:24.932 DEBUG:boto:Port: 8001 2023-05-31T21:03:24.932 DEBUG:boto:Params: {} 2023-05-31T21:03:24.933 DEBUG:boto:Token: None 2023-05-31T21:03:24.933 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 21:03:24 GMT /pklxkd-61/ 2023-05-31T21:03:24.934 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:Y+z1tdIY0pnnJpqEt5b8+XUlWS4= 2023-05-31T21:03:24.934 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 21:03:24 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:Y+z1tdIY0pnnJpqEt5b8+XUlWS4=', 'Content-Length': '0'} 2023-05-31T21:03:24.936 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e3f24eb3625414b0-006477b61c-1f4e-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 21:03:24 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:03:24.937 DEBUG:boto:b'pklxkd-61obj951000falseobj952023-05-31T20:59:45.274Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T21:03:24.937 DEBUG:boto:path=/pklxkd-61/ 2023-05-31T21:03:24.938 DEBUG:boto:auth_path=/pklxkd-61/ 2023-05-31T21:03:24.938 DEBUG:boto:path=/pklxkd-61/?prefix=obj95 2023-05-31T21:03:24.938 DEBUG:boto:auth_path=/pklxkd-61/?prefix=obj95 2023-05-31T21:03:24.939 DEBUG:boto:Method: GET 2023-05-31T21:03:24.939 DEBUG:boto:Path: /pklxkd-61/?prefix=obj95 2023-05-31T21:03:24.940 DEBUG:boto:Data: 2023-05-31T21:03:24.940 DEBUG:boto:Headers: {} 2023-05-31T21:03:24.940 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T21:03:24.941 DEBUG:boto:Port: 8000 2023-05-31T21:03:24.941 DEBUG:boto:Params: {} 2023-05-31T21:03:24.941 DEBUG:boto:Token: None 2023-05-31T21:03:24.942 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 21:03:24 GMT /pklxkd-61/ 2023-05-31T21:03:24.942 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:Y+z1tdIY0pnnJpqEt5b8+XUlWS4= 2023-05-31T21:03:24.943 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 21:03:24 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:Y+z1tdIY0pnnJpqEt5b8+XUlWS4=', 'Content-Length': '0'} 2023-05-31T21:03:24.946 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000efa6fa74b871cb21-006477b61c-1630-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 21:03:24 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:03:24.947 DEBUG:boto:b'pklxkd-61obj951000falseobj952023-05-31T20:59:45.274Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T21:03:24.948 DEBUG:boto:path=/pklxkd-61/obj95 2023-05-31T21:03:24.948 DEBUG:boto:auth_path=/pklxkd-61/obj95 2023-05-31T21:03:24.948 DEBUG:boto:Method: HEAD 2023-05-31T21:03:24.949 DEBUG:boto:Path: /pklxkd-61/obj95 2023-05-31T21:03:24.949 DEBUG:boto:Data: 2023-05-31T21:03:24.949 DEBUG:boto:Headers: {} 2023-05-31T21:03:24.950 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T21:03:24.950 DEBUG:boto:Port: 8001 2023-05-31T21:03:24.950 DEBUG:boto:Params: {} 2023-05-31T21:03:24.951 DEBUG:boto:Token: None 2023-05-31T21:03:24.951 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 21:03:24 GMT /pklxkd-61/obj95 2023-05-31T21:03:24.952 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:PyapUIIAbyrfewwvV4MtoJbnwZs= 2023-05-31T21:03:24.952 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 21:03:24 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:PyapUIIAbyrfewwvV4MtoJbnwZs=', 'Content-Length': '0'} 2023-05-31T21:03:24.954 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:59:45 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000357887aca2453781-006477b61c-1f4e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 21:03:24 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:03:24.954 DEBUG:boto:path=/pklxkd-61/obj95 2023-05-31T21:03:24.955 DEBUG:boto:auth_path=/pklxkd-61/obj95 2023-05-31T21:03:24.955 DEBUG:boto:Method: HEAD 2023-05-31T21:03:24.955 DEBUG:boto:Path: /pklxkd-61/obj95 2023-05-31T21:03:24.956 DEBUG:boto:Data: 2023-05-31T21:03:24.956 DEBUG:boto:Headers: {} 2023-05-31T21:03:24.957 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T21:03:24.958 DEBUG:boto:Port: 8000 2023-05-31T21:03:24.959 DEBUG:boto:Params: {} 2023-05-31T21:03:24.960 DEBUG:boto:Token: None 2023-05-31T21:03:24.961 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 21:03:24 GMT /pklxkd-61/obj95 2023-05-31T21:03:24.961 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:PyapUIIAbyrfewwvV4MtoJbnwZs= 2023-05-31T21:03:24.962 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 21:03:24 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:PyapUIIAbyrfewwvV4MtoJbnwZs=', 'Content-Length': '0'} 2023-05-31T21:03:24.963 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:59:45 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000410d0fbbfcd509aa-006477b61c-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 21:03:24 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:03:24.964 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj95 2023-05-31T21:03:24.964 DEBUG:boto:path=/pklxkd-61/obj95 2023-05-31T21:03:24.965 DEBUG:boto:auth_path=/pklxkd-61/obj95 2023-05-31T21:03:24.965 DEBUG:boto:Method: GET 2023-05-31T21:03:24.965 DEBUG:boto:Path: /pklxkd-61/obj95 2023-05-31T21:03:24.966 DEBUG:boto:Data: 2023-05-31T21:03:24.966 DEBUG:boto:Headers: {} 2023-05-31T21:03:24.966 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T21:03:24.967 DEBUG:boto:Port: 8001 2023-05-31T21:03:24.967 DEBUG:boto:Params: {} 2023-05-31T21:03:24.968 DEBUG:boto:Token: None 2023-05-31T21:03:24.968 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 21:03:24 GMT /pklxkd-61/obj95 2023-05-31T21:03:24.969 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:HtYR+QfbY2A8w2pYmMm4ZdocFUs= 2023-05-31T21:03:24.969 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 21:03:24 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:HtYR+QfbY2A8w2pYmMm4ZdocFUs=', 'Content-Length': '0'} 2023-05-31T21:03:24.971 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:59:45 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006b6062be5c65bcf3-006477b61c-1f4e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 21:03:24 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:03:24.971 DEBUG:boto:path=/pklxkd-61/obj95 2023-05-31T21:03:24.972 DEBUG:boto:auth_path=/pklxkd-61/obj95 2023-05-31T21:03:24.972 DEBUG:boto:Method: GET 2023-05-31T21:03:24.973 DEBUG:boto:Path: /pklxkd-61/obj95 2023-05-31T21:03:24.973 DEBUG:boto:Data: 2023-05-31T21:03:24.973 DEBUG:boto:Headers: {} 2023-05-31T21:03:24.974 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T21:03:24.974 DEBUG:boto:Port: 8000 2023-05-31T21:03:24.974 DEBUG:boto:Params: {} 2023-05-31T21:03:24.975 DEBUG:boto:Token: None 2023-05-31T21:03:24.975 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 21:03:24 GMT /pklxkd-61/obj95 2023-05-31T21:03:24.976 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:HtYR+QfbY2A8w2pYmMm4ZdocFUs= 2023-05-31T21:03:24.976 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 21:03:24 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:HtYR+QfbY2A8w2pYmMm4ZdocFUs=', 'Content-Length': '0'} 2023-05-31T21:03:24.978 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:59:45 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b788eb191de489ac-006477b61c-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 21:03:24 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:03:25.020 DEBUG:boto:path=/pklxkd-61/ 2023-05-31T21:03:25.021 DEBUG:boto:auth_path=/pklxkd-61/ 2023-05-31T21:03:25.021 DEBUG:boto:path=/pklxkd-61/?prefix=obj95 2023-05-31T21:03:25.022 DEBUG:boto:auth_path=/pklxkd-61/?prefix=obj95 2023-05-31T21:03:25.022 DEBUG:boto:Method: GET 2023-05-31T21:03:25.022 DEBUG:boto:Path: /pklxkd-61/?prefix=obj95 2023-05-31T21:03:25.023 DEBUG:boto:Data: 2023-05-31T21:03:25.023 DEBUG:boto:Headers: {} 2023-05-31T21:03:25.024 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T21:03:25.024 DEBUG:boto:Port: 8001 2023-05-31T21:03:25.024 DEBUG:boto:Params: {} 2023-05-31T21:03:25.025 DEBUG:boto:Token: None 2023-05-31T21:03:25.025 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 21:03:25 GMT /pklxkd-61/ 2023-05-31T21:03:25.026 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:i1ctDNJazKCa873oTl3rwxeLHoQ= 2023-05-31T21:03:25.026 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 21:03:25 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:i1ctDNJazKCa873oTl3rwxeLHoQ=', 'Content-Length': '0'} 2023-05-31T21:03:25.028 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000005500fbbf131cdfff-006477b61d-1f4e-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 21:03:25 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:03:25.029 DEBUG:boto:b'pklxkd-61obj951000falseobj952023-05-31T20:59:45.274Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T21:03:25.030 DEBUG:boto:path=/pklxkd-61/ 2023-05-31T21:03:25.030 DEBUG:boto:auth_path=/pklxkd-61/ 2023-05-31T21:03:25.030 DEBUG:boto:path=/pklxkd-61/?prefix=obj95 2023-05-31T21:03:25.031 DEBUG:boto:auth_path=/pklxkd-61/?prefix=obj95 2023-05-31T21:03:25.031 DEBUG:boto:Method: GET 2023-05-31T21:03:25.032 DEBUG:boto:Path: /pklxkd-61/?prefix=obj95 2023-05-31T21:03:25.032 DEBUG:boto:Data: 2023-05-31T21:03:25.032 DEBUG:boto:Headers: {} 2023-05-31T21:03:25.033 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T21:03:25.033 DEBUG:boto:Port: 8000 2023-05-31T21:03:25.034 DEBUG:boto:Params: {} 2023-05-31T21:03:25.034 DEBUG:boto:Token: None 2023-05-31T21:03:25.035 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 21:03:25 GMT /pklxkd-61/ 2023-05-31T21:03:25.035 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:i1ctDNJazKCa873oTl3rwxeLHoQ= 2023-05-31T21:03:25.035 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 21:03:25 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:i1ctDNJazKCa873oTl3rwxeLHoQ=', 'Content-Length': '0'} 2023-05-31T21:03:25.039 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000008f058c752e7cd710-006477b61d-1630-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 21:03:25 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:03:25.040 DEBUG:boto:b'pklxkd-61obj951000falseobj952023-05-31T20:59:45.274Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T21:03:25.041 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=pklxkd-61 zones={a2, a1} 2023-05-31T21:03:25.041 DEBUG:rgw_multi.tests:data checkpoint: source=a1 target=a2 2023-05-31T21:03:25.041 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['datalog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-31T21:03:25.042 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'datalog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-31T21:03:25.042 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 datalog status --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-05-31T21:03:25.097 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2023-05-31T21:03:25.097 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2023-05-31T21:03:25.177 INFO:teuthology.orchestra.run.smithi099.stdout:[ 2023-05-31T21:03:25.178 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T21:03:25.178 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "00000000000000000000:00000000000000031645", 2023-05-31T21:03:25.178 INFO:teuthology.orchestra.run.smithi099.stdout: "last_update": "2023-05-31T20:59:50.644038Z" 2023-05-31T21:03:25.178 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:03:25.179 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T21:03:25.179 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "00000000000000000000:00000000000000022723", 2023-05-31T21:03:25.179 INFO:teuthology.orchestra.run.smithi099.stdout: "last_update": "2023-05-31T20:59:50.645856Z" 2023-05-31T21:03:25.180 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:03:25.180 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T21:03:25.180 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "00000000000000000000:00000000000000037248", 2023-05-31T21:03:25.181 INFO:teuthology.orchestra.run.smithi099.stdout: "last_update": "2023-05-31T20:59:47.033585Z" 2023-05-31T21:03:25.181 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:03:25.181 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T21:03:25.181 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "00000000000000000000:00000000000000032158", 2023-05-31T21:03:25.182 INFO:teuthology.orchestra.run.smithi099.stdout: "last_update": "2023-05-31T20:59:50.647392Z" 2023-05-31T21:03:25.182 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T21:03:25.182 INFO:teuthology.orchestra.run.smithi099.stdout:] 2023-05-31T21:03:25.184 DEBUG:rgw_multi.tests:data markers for zone=a1 markers={0: '00000000000000000000:00000000000000031645', 1: '00000000000000000000:00000000000000022723', 2: '00000000000000000000:00000000000000037248', 3: '00000000000000000000:00000000000000032158'} 2023-05-31T21:03:25.185 INFO:rgw_multi.tests:starting data checkpoint for target_zone=a2 source_zone=a1 2023-05-31T21:03:25.185 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-31T21:03:25.186 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-31T21:03:25.187 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-05-31T21:03:25.239 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2023-05-31T21:03:25.239 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2023-05-31T21:03:25.286 INFO:teuthology.orchestra.run.smithi099.stdout:{ 2023-05-31T21:03:25.286 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_status": { 2023-05-31T21:03:25.286 INFO:teuthology.orchestra.run.smithi099.stdout: "info": { 2023-05-31T21:03:25.287 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "sync", 2023-05-31T21:03:25.287 INFO:teuthology.orchestra.run.smithi099.stdout: "num_shards": 4, 2023-05-31T21:03:25.287 INFO:teuthology.orchestra.run.smithi099.stdout: "instance_id": 11608385958497787600 2023-05-31T21:03:25.288 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:03:25.288 INFO:teuthology.orchestra.run.smithi099.stdout: "markers": [ 2023-05-31T21:03:25.288 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T21:03:25.289 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 0, 2023-05-31T21:03:25.289 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T21:03:25.289 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "incremental-sync", 2023-05-31T21:03:25.290 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "00000000000000000000:00000000000000031645", 2023-05-31T21:03:25.290 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T21:03:25.290 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2023-05-31T21:03:25.290 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T21:03:25.291 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2023-05-31T20:59:50.644038Z" 2023-05-31T21:03:25.291 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T21:03:25.291 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:03:25.291 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T21:03:25.291 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 1, 2023-05-31T21:03:25.291 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T21:03:25.291 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "incremental-sync", 2023-05-31T21:03:25.292 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "00000000000000000000:00000000000000022723", 2023-05-31T21:03:25.292 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T21:03:25.292 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2023-05-31T21:03:25.292 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T21:03:25.292 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2023-05-31T20:59:50.645856Z" 2023-05-31T21:03:25.292 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T21:03:25.293 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:03:25.293 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T21:03:25.293 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 2, 2023-05-31T21:03:25.293 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T21:03:25.293 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "incremental-sync", 2023-05-31T21:03:25.293 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "00000000000000000000:00000000000000037248", 2023-05-31T21:03:25.293 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T21:03:25.294 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 52, 2023-05-31T21:03:25.294 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T21:03:25.294 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2023-05-31T20:59:47.033585Z" 2023-05-31T21:03:25.294 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T21:03:25.294 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:03:25.294 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T21:03:25.294 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 3, 2023-05-31T21:03:25.295 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T21:03:25.295 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "incremental-sync", 2023-05-31T21:03:25.295 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "00000000000000000000:00000000000000032158", 2023-05-31T21:03:25.295 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T21:03:25.295 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 9, 2023-05-31T21:03:25.295 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T21:03:25.295 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2023-05-31T20:59:50.647392Z" 2023-05-31T21:03:25.296 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T21:03:25.296 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T21:03:25.296 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2023-05-31T21:03:25.296 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:03:25.296 INFO:teuthology.orchestra.run.smithi099.stdout: "full_sync": { 2023-05-31T21:03:25.296 INFO:teuthology.orchestra.run.smithi099.stdout: "total": 61, 2023-05-31T21:03:25.297 INFO:teuthology.orchestra.run.smithi099.stdout: "complete": 61 2023-05-31T21:03:25.297 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:03:25.297 INFO:teuthology.orchestra.run.smithi099.stdout: "current_time": "2023-05-31T21:03:25Z" 2023-05-31T21:03:25.297 INFO:teuthology.orchestra.run.smithi099.stdout:} 2023-05-31T21:03:25.299 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 11608385958497787600 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000031645", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-05-31T20:59:50.644038Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000022723", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-05-31T20:59:50.645856Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000037248", "next_step_marker": "", "total_entries": 52, "pos": 0, "timestamp": "2023-05-31T20:59:47.033585Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000032158", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2023-05-31T20:59:50.647392Z" } } ] }, "full_sync": { "total": 61, "complete": 61 }, "current_time": "2023-05-31T21:03:25Z" } 2023-05-31T21:03:25.300 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000031645', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-05-31T20:59:50.644038Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000022723', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-05-31T20:59:50.645856Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000037248', 'next_step_marker': '', 'total_entries': 52, 'pos': 0, 'timestamp': '2023-05-31T20:59:47.033585Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000032158', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2023-05-31T20:59:50.647392Z'}}] 2023-05-31T21:03:25.301 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000031645', 1: '00000000000000000000:00000000000000022723', 2: '00000000000000000000:00000000000000037248', 3: '00000000000000000000:00000000000000032158'} 2023-05-31T21:03:25.301 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000031645', 1: '00000000000000000000:00000000000000022723', 2: '00000000000000000000:00000000000000037248', 3: '00000000000000000000:00000000000000032158'} 2023-05-31T21:03:25.302 INFO:rgw_multi.tests:finished data checkpoint for target_zone=a2 source_zone=a1 2023-05-31T21:03:25.303 DEBUG:rgw_multi.tests:data checkpoint: source=a2 target=a1 2023-05-31T21:03:25.304 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['datalog', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-31T21:03:25.304 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'datalog', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-31T21:03:25.305 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 datalog status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-05-31T21:03:25.353 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2023-05-31T21:03:25.354 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2023-05-31T21:03:25.418 INFO:teuthology.orchestra.run.smithi099.stdout:[ 2023-05-31T21:03:25.418 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T21:03:25.419 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "00000000000000000000:00000000000000010549", 2023-05-31T21:03:25.419 INFO:teuthology.orchestra.run.smithi099.stdout: "last_update": "2023-05-31T21:01:37.288197Z" 2023-05-31T21:03:25.419 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:03:25.419 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T21:03:25.419 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "00000000000000000000:00000000000000005024", 2023-05-31T21:03:25.419 INFO:teuthology.orchestra.run.smithi099.stdout: "last_update": "2023-05-31T21:01:37.289790Z" 2023-05-31T21:03:25.419 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:03:25.420 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T21:03:25.420 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "00000000000000000000:00000000000000014604", 2023-05-31T21:03:25.420 INFO:teuthology.orchestra.run.smithi099.stdout: "last_update": "2023-05-31T21:01:37.292691Z" 2023-05-31T21:03:25.420 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:03:25.420 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T21:03:25.420 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "00000000000000000000:00000000000000013761", 2023-05-31T21:03:25.421 INFO:teuthology.orchestra.run.smithi099.stdout: "last_update": "2023-05-31T21:01:37.295501Z" 2023-05-31T21:03:25.421 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T21:03:25.421 INFO:teuthology.orchestra.run.smithi099.stdout:] 2023-05-31T21:03:25.422 DEBUG:rgw_multi.tests:data markers for zone=a2 markers={0: '00000000000000000000:00000000000000010549', 1: '00000000000000000000:00000000000000005024', 2: '00000000000000000000:00000000000000014604', 3: '00000000000000000000:00000000000000013761'} 2023-05-31T21:03:25.422 INFO:rgw_multi.tests:starting data checkpoint for target_zone=a1 source_zone=a2 2023-05-31T21:03:25.423 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-31T21:03:25.423 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-31T21:03:25.423 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --source-zone a2 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-05-31T21:03:25.476 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2023-05-31T21:03:25.477 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2023-05-31T21:03:25.557 INFO:teuthology.orchestra.run.smithi099.stdout:{ 2023-05-31T21:03:25.557 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_status": { 2023-05-31T21:03:25.557 INFO:teuthology.orchestra.run.smithi099.stdout: "info": { 2023-05-31T21:03:25.557 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "sync", 2023-05-31T21:03:25.558 INFO:teuthology.orchestra.run.smithi099.stdout: "num_shards": 4, 2023-05-31T21:03:25.558 INFO:teuthology.orchestra.run.smithi099.stdout: "instance_id": 17941554979570305020 2023-05-31T21:03:25.558 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:03:25.558 INFO:teuthology.orchestra.run.smithi099.stdout: "markers": [ 2023-05-31T21:03:25.558 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T21:03:25.558 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 0, 2023-05-31T21:03:25.559 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T21:03:25.559 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "incremental-sync", 2023-05-31T21:03:25.559 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "00000000000000000000:00000000000000010549", 2023-05-31T21:03:25.559 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T21:03:25.559 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2023-05-31T21:03:25.559 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T21:03:25.559 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2023-05-31T21:01:37.288197Z" 2023-05-31T21:03:25.560 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T21:03:25.560 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:03:25.560 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T21:03:25.560 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 1, 2023-05-31T21:03:25.560 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T21:03:25.560 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "incremental-sync", 2023-05-31T21:03:25.561 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "00000000000000000000:00000000000000005024", 2023-05-31T21:03:25.561 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T21:03:25.561 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2023-05-31T21:03:25.561 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T21:03:25.561 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2023-05-31T21:01:37.289790Z" 2023-05-31T21:03:25.561 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T21:03:25.562 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:03:25.562 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T21:03:25.562 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 2, 2023-05-31T21:03:25.562 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T21:03:25.562 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "incremental-sync", 2023-05-31T21:03:25.562 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "00000000000000000000:00000000000000014604", 2023-05-31T21:03:25.562 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T21:03:25.563 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2023-05-31T21:03:25.563 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T21:03:25.563 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2023-05-31T21:01:37.292691Z" 2023-05-31T21:03:25.563 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T21:03:25.563 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:03:25.563 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T21:03:25.563 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 3, 2023-05-31T21:03:25.564 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T21:03:25.564 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "incremental-sync", 2023-05-31T21:03:25.564 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "00000000000000000000:00000000000000013761", 2023-05-31T21:03:25.564 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T21:03:25.564 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2023-05-31T21:03:25.564 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T21:03:25.564 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2023-05-31T21:01:37.295501Z" 2023-05-31T21:03:25.565 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T21:03:25.565 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T21:03:25.565 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2023-05-31T21:03:25.565 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:03:25.565 INFO:teuthology.orchestra.run.smithi099.stdout: "full_sync": { 2023-05-31T21:03:25.565 INFO:teuthology.orchestra.run.smithi099.stdout: "total": 0, 2023-05-31T21:03:25.565 INFO:teuthology.orchestra.run.smithi099.stdout: "complete": 0 2023-05-31T21:03:25.566 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:03:25.566 INFO:teuthology.orchestra.run.smithi099.stdout: "current_time": "2023-05-31T21:03:25Z" 2023-05-31T21:03:25.566 INFO:teuthology.orchestra.run.smithi099.stdout:} 2023-05-31T21:03:25.567 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 17941554979570305020 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000010549", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-05-31T21:01:37.288197Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000005024", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-05-31T21:01:37.289790Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000014604", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-05-31T21:01:37.292691Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000013761", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-05-31T21:01:37.295501Z" } } ] }, "full_sync": { "total": 0, "complete": 0 }, "current_time": "2023-05-31T21:03:25Z" } 2023-05-31T21:03:25.567 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000010549', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-05-31T21:01:37.288197Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000005024', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-05-31T21:01:37.289790Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000014604', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-05-31T21:01:37.292691Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000013761', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-05-31T21:01:37.295501Z'}}] 2023-05-31T21:03:25.568 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000010549', 1: '00000000000000000000:00000000000000005024', 2: '00000000000000000000:00000000000000014604', 3: '00000000000000000000:00000000000000013761'} 2023-05-31T21:03:25.568 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000010549', 1: '00000000000000000000:00000000000000005024', 2: '00000000000000000000:00000000000000014604', 3: '00000000000000000000:00000000000000013761'} 2023-05-31T21:03:25.568 INFO:rgw_multi.tests:finished data checkpoint for target_zone=a1 source_zone=a2 2023-05-31T21:03:25.569 INFO:tasks.rgw_multisite_tests:rgw_multi.tests.test_resharded_bucket_full_sync_after_data_sync_init ... ok 2023-05-31T21:03:25.570 DEBUG:boto:path=/pklxkd-62/ 2023-05-31T21:03:25.571 DEBUG:boto:auth_path=/pklxkd-62/ 2023-05-31T21:03:25.571 DEBUG:boto:Method: PUT 2023-05-31T21:03:25.571 DEBUG:boto:Path: /pklxkd-62/ 2023-05-31T21:03:25.572 DEBUG:boto:Data: 2023-05-31T21:03:25.572 DEBUG:boto:Headers: {} 2023-05-31T21:03:25.572 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T21:03:25.573 DEBUG:boto:Port: 8000 2023-05-31T21:03:25.573 DEBUG:boto:Params: {} 2023-05-31T21:03:25.573 DEBUG:boto:Token: None 2023-05-31T21:03:25.574 DEBUG:boto:StringToSign: PUT Wed, 31 May 2023 21:03:25 GMT /pklxkd-62/ 2023-05-31T21:03:25.574 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:1b9/+BrcgctsHuF6i/kosQQbUCE= 2023-05-31T21:03:25.575 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 21:03:25 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:1b9/+BrcgctsHuF6i/kosQQbUCE=', 'Content-Length': '0'} 2023-05-31T21:03:25.602 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx000006adb6f8fad8663d4-006477b61d-1630-a1'), ('Content-Length', '0'), ('Date', 'Wed, 31 May 2023 21:03:25 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:03:25.603 DEBUG:rgw_multi.tests:created bucket=pklxkd-62 2023-05-31T21:03:25.603 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-31T21:03:25.604 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-31T21:03:25.604 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-05-31T21:03:25.652 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2023-05-31T21:03:25.653 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2023-05-31T21:03:25.689 INFO:teuthology.orchestra.run.smithi099.stderr:No --period given, using current period=2e6123d2-d695-4350-9a01-61a455bc782b 2023-05-31T21:03:25.703 INFO:teuthology.orchestra.run.smithi099.stdout:[ 2023-05-31T21:03:25.703 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T21:03:25.703 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1685566501.757183_260.1", 2023-05-31T21:03:25.703 INFO:teuthology.orchestra.run.smithi099.stdout: "last_update": "2023-05-31T20:55:01.757183Z" 2023-05-31T21:03:25.704 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:03:25.704 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T21:03:25.704 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1685566501.735717_251.1", 2023-05-31T21:03:25.704 INFO:teuthology.orchestra.run.smithi099.stdout: "last_update": "2023-05-31T20:55:01.735717Z" 2023-05-31T21:03:25.704 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:03:25.704 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T21:03:25.704 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1685562970.274572_93.1", 2023-05-31T21:03:25.705 INFO:teuthology.orchestra.run.smithi099.stdout: "last_update": "2023-05-31T19:56:10.274572Z" 2023-05-31T21:03:25.705 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:03:25.705 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T21:03:25.705 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1685567005.597271_572.1", 2023-05-31T21:03:25.705 INFO:teuthology.orchestra.run.smithi099.stdout: "last_update": "2023-05-31T21:03:25.597271Z" 2023-05-31T21:03:25.705 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T21:03:25.705 INFO:teuthology.orchestra.run.smithi099.stdout:] 2023-05-31T21:03:25.706 DEBUG:rgw_multi.tests:master meta markers={0: '1_1685566501.757183_260.1', 1: '1_1685566501.735717_251.1', 2: '1_1685562970.274572_93.1', 3: '1_1685567005.597271_572.1'} 2023-05-31T21:03:25.707 INFO:rgw_multi.tests:starting meta checkpoint for zone=a2 2023-05-31T21:03:25.707 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-31T21:03:25.708 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-31T21:03:25.708 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-05-31T21:03:25.757 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2023-05-31T21:03:25.758 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2023-05-31T21:03:25.803 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T21:03:25.799+0000 7f62f61a9780 20 RGW-SYNC:meta: read sync status 2023-05-31T21:03:25.804 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T21:03:25.799+0000 7f62f61a9780 20 RGW-SYNC:meta: read sync status 2023-05-31T21:03:25.826 INFO:teuthology.orchestra.run.smithi099.stdout:{ 2023-05-31T21:03:25.826 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_status": { 2023-05-31T21:03:25.826 INFO:teuthology.orchestra.run.smithi099.stdout: "info": { 2023-05-31T21:03:25.827 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "sync", 2023-05-31T21:03:25.827 INFO:teuthology.orchestra.run.smithi099.stdout: "num_shards": 4, 2023-05-31T21:03:25.827 INFO:teuthology.orchestra.run.smithi099.stdout: "period": "2e6123d2-d695-4350-9a01-61a455bc782b", 2023-05-31T21:03:25.827 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2023-05-31T21:03:25.827 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:03:25.827 INFO:teuthology.orchestra.run.smithi099.stdout: "markers": [ 2023-05-31T21:03:25.828 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T21:03:25.828 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 0, 2023-05-31T21:03:25.828 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T21:03:25.828 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2023-05-31T21:03:25.828 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1685566501.757183_260.1", 2023-05-31T21:03:25.828 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T21:03:25.828 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 1, 2023-05-31T21:03:25.829 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T21:03:25.829 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2023-05-31T20:55:01.757183Z", 2023-05-31T21:03:25.829 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2023-05-31T21:03:25.829 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T21:03:25.829 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:03:25.829 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T21:03:25.830 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 1, 2023-05-31T21:03:25.830 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T21:03:25.830 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2023-05-31T21:03:25.830 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1685566501.735717_251.1", 2023-05-31T21:03:25.830 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T21:03:25.830 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2023-05-31T21:03:25.830 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T21:03:25.831 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2023-05-31T20:55:01.735717Z", 2023-05-31T21:03:25.831 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2023-05-31T21:03:25.831 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T21:03:25.831 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:03:25.831 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T21:03:25.831 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 2, 2023-05-31T21:03:25.831 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T21:03:25.832 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2023-05-31T21:03:25.832 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1685562970.274572_93.1", 2023-05-31T21:03:25.832 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T21:03:25.832 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2023-05-31T21:03:25.832 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T21:03:25.832 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2023-05-31T19:56:10.274572Z", 2023-05-31T21:03:25.832 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2023-05-31T21:03:25.833 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T21:03:25.833 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:03:25.833 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T21:03:25.833 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 3, 2023-05-31T21:03:25.833 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T21:03:25.833 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2023-05-31T21:03:25.833 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1685567005.582979_569.1", 2023-05-31T21:03:25.834 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T21:03:25.834 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2023-05-31T21:03:25.834 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T21:03:25.834 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2023-05-31T21:03:25.582979Z", 2023-05-31T21:03:25.834 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2023-05-31T21:03:25.834 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T21:03:25.834 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T21:03:25.835 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2023-05-31T21:03:25.835 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:03:25.835 INFO:teuthology.orchestra.run.smithi099.stdout: "full_sync": { 2023-05-31T21:03:25.835 INFO:teuthology.orchestra.run.smithi099.stdout: "total": 1, 2023-05-31T21:03:25.835 INFO:teuthology.orchestra.run.smithi099.stdout: "complete": 1 2023-05-31T21:03:25.835 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:03:25.835 INFO:teuthology.orchestra.run.smithi099.stdout: "current_time": "2023-05-31T21:03:25Z" 2023-05-31T21:03:25.836 INFO:teuthology.orchestra.run.smithi099.stdout:} 2023-05-31T21:03:25.836 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "2e6123d2-d695-4350-9a01-61a455bc782b", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1685566501.757183_260.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-05-31T20:55:01.757183Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1685566501.735717_251.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-05-31T20:55:01.735717Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1685562970.274572_93.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-05-31T19:56:10.274572Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1685567005.582979_569.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-05-31T21:03:25.582979Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-05-31T21:03:25Z" } 2023-05-31T21:03:25.837 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1685566501.757183_260.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-05-31T20:55:01.757183Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1685566501.735717_251.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-05-31T20:55:01.735717Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1685562970.274572_93.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-05-31T19:56:10.274572Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1685567005.582979_569.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-05-31T21:03:25.582979Z', 'realm_epoch': 2}}] 2023-05-31T21:03:25.837 DEBUG:rgw_multi.tests:log_status={0: '1_1685566501.757183_260.1', 1: '1_1685566501.735717_251.1', 2: '1_1685562970.274572_93.1', 3: '1_1685567005.597271_572.1'} 2023-05-31T21:03:25.838 DEBUG:rgw_multi.tests:sync_status={0: '1_1685566501.757183_260.1', 1: '1_1685566501.735717_251.1', 2: '1_1685562970.274572_93.1', 3: '1_1685567005.582979_569.1'} 2023-05-31T21:03:25.838 WARNING:rgw_multi.tests:zone a2 behind master: shard=3 master=1_1685567005.597271_572.1 target=1_1685567005.582979_569.1 2023-05-31T21:03:30.839 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-31T21:03:30.840 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-31T21:03:30.841 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-05-31T21:03:30.898 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2023-05-31T21:03:30.898 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2023-05-31T21:03:30.937 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T21:03:30.932+0000 7fd90db9a780 20 RGW-SYNC:meta: read sync status 2023-05-31T21:03:30.937 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T21:03:30.933+0000 7fd90db9a780 20 RGW-SYNC:meta: read sync status 2023-05-31T21:03:30.955 INFO:teuthology.orchestra.run.smithi099.stdout:{ 2023-05-31T21:03:30.956 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_status": { 2023-05-31T21:03:30.956 INFO:teuthology.orchestra.run.smithi099.stdout: "info": { 2023-05-31T21:03:30.956 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "sync", 2023-05-31T21:03:30.956 INFO:teuthology.orchestra.run.smithi099.stdout: "num_shards": 4, 2023-05-31T21:03:30.957 INFO:teuthology.orchestra.run.smithi099.stdout: "period": "2e6123d2-d695-4350-9a01-61a455bc782b", 2023-05-31T21:03:30.957 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2023-05-31T21:03:30.957 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:03:30.957 INFO:teuthology.orchestra.run.smithi099.stdout: "markers": [ 2023-05-31T21:03:30.957 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T21:03:30.957 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 0, 2023-05-31T21:03:30.958 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T21:03:30.958 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2023-05-31T21:03:30.958 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1685566501.757183_260.1", 2023-05-31T21:03:30.958 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T21:03:30.958 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 1, 2023-05-31T21:03:30.958 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T21:03:30.958 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2023-05-31T20:55:01.757183Z", 2023-05-31T21:03:30.959 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2023-05-31T21:03:30.959 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T21:03:30.959 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:03:30.959 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T21:03:30.959 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 1, 2023-05-31T21:03:30.959 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T21:03:30.960 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2023-05-31T21:03:30.960 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1685566501.735717_251.1", 2023-05-31T21:03:30.960 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T21:03:30.960 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2023-05-31T21:03:30.960 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T21:03:30.960 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2023-05-31T20:55:01.735717Z", 2023-05-31T21:03:30.961 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2023-05-31T21:03:30.961 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T21:03:30.961 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:03:30.961 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T21:03:30.961 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 2, 2023-05-31T21:03:30.961 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T21:03:30.961 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2023-05-31T21:03:30.962 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1685562970.274572_93.1", 2023-05-31T21:03:30.962 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T21:03:30.962 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2023-05-31T21:03:30.962 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T21:03:30.962 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2023-05-31T19:56:10.274572Z", 2023-05-31T21:03:30.962 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2023-05-31T21:03:30.963 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T21:03:30.963 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:03:30.963 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T21:03:30.963 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 3, 2023-05-31T21:03:30.963 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T21:03:30.963 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2023-05-31T21:03:30.964 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1685567005.597271_572.1", 2023-05-31T21:03:30.964 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T21:03:30.964 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2023-05-31T21:03:30.964 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T21:03:30.964 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2023-05-31T21:03:25.597271Z", 2023-05-31T21:03:30.964 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2023-05-31T21:03:30.965 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T21:03:30.965 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T21:03:30.965 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2023-05-31T21:03:30.965 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:03:30.965 INFO:teuthology.orchestra.run.smithi099.stdout: "full_sync": { 2023-05-31T21:03:30.965 INFO:teuthology.orchestra.run.smithi099.stdout: "total": 1, 2023-05-31T21:03:30.966 INFO:teuthology.orchestra.run.smithi099.stdout: "complete": 1 2023-05-31T21:03:30.966 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:03:30.966 INFO:teuthology.orchestra.run.smithi099.stdout: "current_time": "2023-05-31T21:03:30Z" 2023-05-31T21:03:30.966 INFO:teuthology.orchestra.run.smithi099.stdout:} 2023-05-31T21:03:30.967 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "2e6123d2-d695-4350-9a01-61a455bc782b", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1685566501.757183_260.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-05-31T20:55:01.757183Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1685566501.735717_251.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-05-31T20:55:01.735717Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1685562970.274572_93.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-05-31T19:56:10.274572Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1685567005.597271_572.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-05-31T21:03:25.597271Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-05-31T21:03:30Z" } 2023-05-31T21:03:30.968 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1685566501.757183_260.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-05-31T20:55:01.757183Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1685566501.735717_251.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-05-31T20:55:01.735717Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1685562970.274572_93.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-05-31T19:56:10.274572Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1685567005.597271_572.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-05-31T21:03:25.597271Z', 'realm_epoch': 2}}] 2023-05-31T21:03:30.968 DEBUG:rgw_multi.tests:log_status={0: '1_1685566501.757183_260.1', 1: '1_1685566501.735717_251.1', 2: '1_1685562970.274572_93.1', 3: '1_1685567005.597271_572.1'} 2023-05-31T21:03:30.968 DEBUG:rgw_multi.tests:sync_status={0: '1_1685566501.757183_260.1', 1: '1_1685566501.735717_251.1', 2: '1_1685562970.274572_93.1', 3: '1_1685567005.597271_572.1'} 2023-05-31T21:03:30.969 INFO:rgw_multi.tests:finish meta checkpoint for zone=a2 2023-05-31T21:03:30.969 DEBUG:boto:path=/pklxkd-62/ 2023-05-31T21:03:30.969 DEBUG:boto:auth_path=/pklxkd-62/ 2023-05-31T21:03:30.970 DEBUG:boto:Method: HEAD 2023-05-31T21:03:30.970 DEBUG:boto:Path: /pklxkd-62/ 2023-05-31T21:03:30.971 DEBUG:boto:Data: 2023-05-31T21:03:30.971 DEBUG:boto:Headers: {} 2023-05-31T21:03:30.971 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T21:03:30.972 DEBUG:boto:Port: 8000 2023-05-31T21:03:30.972 DEBUG:boto:Params: {} 2023-05-31T21:03:30.972 DEBUG:boto:Token: None 2023-05-31T21:03:30.973 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 21:03:30 GMT /pklxkd-62/ 2023-05-31T21:03:30.973 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:kjPSCVfpt0/8OtxS1gjjChn6glM= 2023-05-31T21:03:30.974 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 21:03:30 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:kjPSCVfpt0/8OtxS1gjjChn6glM=', 'Content-Length': '0'} 2023-05-31T21:03:30.976 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000024c991d4c7e48df4-006477b622-1630-a1'), ('Content-Length', '0'), ('Date', 'Wed, 31 May 2023 21:03:30 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:03:30.977 DEBUG:boto:path=/pklxkd-62/dummy 2023-05-31T21:03:30.977 DEBUG:boto:auth_path=/pklxkd-62/dummy 2023-05-31T21:03:30.977 DEBUG:boto:Method: PUT 2023-05-31T21:03:30.978 DEBUG:boto:Path: /pklxkd-62/dummy 2023-05-31T21:03:30.978 DEBUG:boto:Data: 2023-05-31T21:03:30.979 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-05-31T21:03:30.979 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T21:03:30.979 DEBUG:boto:Port: 8000 2023-05-31T21:03:30.980 DEBUG:boto:Params: {} 2023-05-31T21:03:30.980 DEBUG:boto:Token: None 2023-05-31T21:03:30.980 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Wed, 31 May 2023 21:03:30 GMT /pklxkd-62/dummy 2023-05-31T21:03:30.981 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:ssWN+hdJWEZuouRTotO01JYqkz4= 2023-05-31T21:03:30.981 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Wed, 31 May 2023 21:03:30 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:ssWN+hdJWEZuouRTotO01JYqkz4='} 2023-05-31T21:03:30.988 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000e33476c985f72def-006477b622-1630-a1'), ('Date', 'Wed, 31 May 2023 21:03:30 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:03:30.989 DEBUG:rgw_multi.tests:bucket checkpoint: source=a1 target=a2 bucket=pklxkd-62 2023-05-31T21:03:30.989 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'pklxkd-62', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-05-31T21:03:30.990 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'pklxkd-62', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-05-31T21:03:30.990 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket pklxkd-62 --source-zone a1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2023-05-31T21:03:31.046 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2023-05-31T21:03:31.047 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2023-05-31T21:03:31.101 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T21:03:31.096+0000 7fc33ffb6780 1 waiting to reach incremental sync.. 2023-05-31T21:03:36.102 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T21:03:36.098+0000 7fc33ffb6780 1 waiting to reach incremental sync.. 2023-05-31T21:03:41.102 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T21:03:41.099+0000 7fc33ffb6780 1 waiting to reach incremental sync.. 2023-05-31T21:03:46.106 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T21:03:46.101+0000 7fc33ffb6780 1 bucket sync caught up with source: 2023-05-31T21:03:46.106 INFO:teuthology.orchestra.run.smithi099.stderr: local status: [, , , , , 00000000001.93.6, , , , , ] 2023-05-31T21:03:46.106 INFO:teuthology.orchestra.run.smithi099.stderr: remote markers: [, , , , , 00000000001.93.6, , , , , ] 2023-05-31T21:03:46.106 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T21:03:46.101+0000 7fc33ffb6780 0 bucket checkpoint complete 2023-05-31T21:03:46.124 DEBUG:rgw_multi.tests:bucket checkpoint: source=a2 target=a1 bucket=pklxkd-62 2023-05-31T21:03:46.125 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'pklxkd-62', '--source-zone', 'a2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-05-31T21:03:46.125 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'pklxkd-62', '--source-zone', 'a2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-05-31T21:03:46.126 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket pklxkd-62 --source-zone a2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2023-05-31T21:03:46.187 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2023-05-31T21:03:46.188 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2023-05-31T21:03:46.241 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T21:03:46.237+0000 7f095c51c780 1 waiting to reach incremental sync.. 2023-05-31T21:03:51.242 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T21:03:51.238+0000 7f095c51c780 1 waiting to reach incremental sync.. 2023-05-31T21:03:56.243 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T21:03:56.238+0000 7f095c51c780 1 bucket sync caught up with source: 2023-05-31T21:03:56.244 INFO:teuthology.orchestra.run.smithi099.stderr: local status: [, , , , , 00000000001.804.6, , , , , ] 2023-05-31T21:03:56.244 INFO:teuthology.orchestra.run.smithi099.stderr: remote markers: [, , , , , 00000000001.804.6, , , , , ] 2023-05-31T21:03:56.245 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T21:03:56.238+0000 7f095c51c780 0 bucket checkpoint complete 2023-05-31T21:03:56.261 INFO:tasks.rgw_multi.zone_rados:comparing bucket=pklxkd-62 zones={a2, a1} 2023-05-31T21:03:56.262 DEBUG:boto:path=/pklxkd-62/ 2023-05-31T21:03:56.262 DEBUG:boto:auth_path=/pklxkd-62/ 2023-05-31T21:03:56.263 DEBUG:boto:Method: HEAD 2023-05-31T21:03:56.263 DEBUG:boto:Path: /pklxkd-62/ 2023-05-31T21:03:56.263 DEBUG:boto:Data: 2023-05-31T21:03:56.264 DEBUG:boto:Headers: {} 2023-05-31T21:03:56.264 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T21:03:56.265 DEBUG:boto:Port: 8001 2023-05-31T21:03:56.265 DEBUG:boto:Params: {} 2023-05-31T21:03:56.265 DEBUG:boto:Token: None 2023-05-31T21:03:56.266 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 21:03:56 GMT /pklxkd-62/ 2023-05-31T21:03:56.266 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:6bwW3kQ+GTDHwzV3if/cNaXA9xk= 2023-05-31T21:03:56.267 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 21:03:56 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:6bwW3kQ+GTDHwzV3if/cNaXA9xk=', 'Content-Length': '0'} 2023-05-31T21:03:56.269 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '3'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000e5dacc8f67f00c3d-006477b63c-1f4e-a2'), ('Content-Length', '0'), ('Date', 'Wed, 31 May 2023 21:03:56 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:03:56.270 DEBUG:boto:path=/pklxkd-62/ 2023-05-31T21:03:56.270 DEBUG:boto:auth_path=/pklxkd-62/ 2023-05-31T21:03:56.271 DEBUG:boto:Method: HEAD 2023-05-31T21:03:56.271 DEBUG:boto:Path: /pklxkd-62/ 2023-05-31T21:03:56.271 DEBUG:boto:Data: 2023-05-31T21:03:56.272 DEBUG:boto:Headers: {} 2023-05-31T21:03:56.272 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T21:03:56.273 DEBUG:boto:Port: 8000 2023-05-31T21:03:56.273 DEBUG:boto:Params: {} 2023-05-31T21:03:56.273 DEBUG:boto:Token: None 2023-05-31T21:03:56.274 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 21:03:56 GMT /pklxkd-62/ 2023-05-31T21:03:56.274 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:6bwW3kQ+GTDHwzV3if/cNaXA9xk= 2023-05-31T21:03:56.275 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 21:03:56 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:6bwW3kQ+GTDHwzV3if/cNaXA9xk=', 'Content-Length': '0'} 2023-05-31T21:03:56.277 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '3'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000b572e3a1bdb47a23-006477b63c-1630-a1'), ('Content-Length', '0'), ('Date', 'Wed, 31 May 2023 21:03:56 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:03:56.278 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2023-05-31T21:03:56.278 DEBUG:boto:path=/pklxkd-62/ 2023-05-31T21:03:56.279 DEBUG:boto:auth_path=/pklxkd-62/ 2023-05-31T21:03:56.279 DEBUG:boto:path=/pklxkd-62/?versions&max-keys=999 2023-05-31T21:03:56.279 DEBUG:boto:auth_path=/pklxkd-62/?versions&max-keys=999 2023-05-31T21:03:56.280 DEBUG:boto:Method: GET 2023-05-31T21:03:56.280 DEBUG:boto:Path: /pklxkd-62/?versions&max-keys=999 2023-05-31T21:03:56.281 DEBUG:boto:Data: 2023-05-31T21:03:56.281 DEBUG:boto:Headers: {} 2023-05-31T21:03:56.281 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T21:03:56.282 DEBUG:boto:Port: 8001 2023-05-31T21:03:56.282 DEBUG:boto:Params: {} 2023-05-31T21:03:56.282 DEBUG:boto:Token: None 2023-05-31T21:03:56.283 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 21:03:56 GMT /pklxkd-62/?versions 2023-05-31T21:03:56.283 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:bk8pE3XvRCrXX9DApN+3CxE/mtU= 2023-05-31T21:03:56.284 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 21:03:56 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:bk8pE3XvRCrXX9DApN+3CxE/mtU=', 'Content-Length': '0'} 2023-05-31T21:03:56.286 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000061aee7ea3d3d7414-006477b63c-1f4e-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 21:03:56 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:03:56.328 DEBUG:boto:b'pklxkd-62999falsedummynulltrue2023-05-31T21:03:30.980Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T21:03:56.329 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2023-05-31T21:03:56.330 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2023-05-31T21:03:56.330 DEBUG:boto:path=/pklxkd-62/ 2023-05-31T21:03:56.331 DEBUG:boto:auth_path=/pklxkd-62/ 2023-05-31T21:03:56.331 DEBUG:boto:path=/pklxkd-62/?versions&max-keys=999 2023-05-31T21:03:56.331 DEBUG:boto:auth_path=/pklxkd-62/?versions&max-keys=999 2023-05-31T21:03:56.332 DEBUG:boto:Method: GET 2023-05-31T21:03:56.332 DEBUG:boto:Path: /pklxkd-62/?versions&max-keys=999 2023-05-31T21:03:56.332 DEBUG:boto:Data: 2023-05-31T21:03:56.333 DEBUG:boto:Headers: {} 2023-05-31T21:03:56.333 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T21:03:56.333 DEBUG:boto:Port: 8000 2023-05-31T21:03:56.334 DEBUG:boto:Params: {} 2023-05-31T21:03:56.334 DEBUG:boto:Token: None 2023-05-31T21:03:56.335 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 21:03:56 GMT /pklxkd-62/?versions 2023-05-31T21:03:56.336 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:bk8pE3XvRCrXX9DApN+3CxE/mtU= 2023-05-31T21:03:56.337 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 21:03:56 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:bk8pE3XvRCrXX9DApN+3CxE/mtU=', 'Content-Length': '0'} 2023-05-31T21:03:56.339 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b8c70b43c44fd861-006477b63c-1630-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 21:03:56 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:03:56.340 DEBUG:boto:b'pklxkd-62999falsedummynulltrue2023-05-31T21:03:30.980Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T21:03:56.340 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2023-05-31T21:03:56.341 DEBUG:boto:path=/pklxkd-62/ 2023-05-31T21:03:56.341 DEBUG:boto:auth_path=/pklxkd-62/ 2023-05-31T21:03:56.342 DEBUG:boto:path=/pklxkd-62/?versions&max-keys=999 2023-05-31T21:03:56.342 DEBUG:boto:auth_path=/pklxkd-62/?versions&max-keys=999 2023-05-31T21:03:56.342 DEBUG:boto:Method: GET 2023-05-31T21:03:56.343 DEBUG:boto:Path: /pklxkd-62/?versions&max-keys=999 2023-05-31T21:03:56.343 DEBUG:boto:Data: 2023-05-31T21:03:56.343 DEBUG:boto:Headers: {} 2023-05-31T21:03:56.344 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T21:03:56.344 DEBUG:boto:Port: 8001 2023-05-31T21:03:56.345 DEBUG:boto:Params: {} 2023-05-31T21:03:56.345 DEBUG:boto:Token: None 2023-05-31T21:03:56.345 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 21:03:56 GMT /pklxkd-62/?versions 2023-05-31T21:03:56.346 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:bk8pE3XvRCrXX9DApN+3CxE/mtU= 2023-05-31T21:03:56.346 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 21:03:56 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:bk8pE3XvRCrXX9DApN+3CxE/mtU=', 'Content-Length': '0'} 2023-05-31T21:03:56.349 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000002c0f2a6dff193e49-006477b63c-1f4e-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 21:03:56 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:03:56.349 DEBUG:boto:b'pklxkd-62999falsedummynulltrue2023-05-31T21:03:30.980Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T21:03:56.350 DEBUG:boto:path=/pklxkd-62/ 2023-05-31T21:03:56.350 DEBUG:boto:auth_path=/pklxkd-62/ 2023-05-31T21:03:56.350 DEBUG:boto:path=/pklxkd-62/?versions&max-keys=999 2023-05-31T21:03:56.351 DEBUG:boto:auth_path=/pklxkd-62/?versions&max-keys=999 2023-05-31T21:03:56.351 DEBUG:boto:Method: GET 2023-05-31T21:03:56.352 DEBUG:boto:Path: /pklxkd-62/?versions&max-keys=999 2023-05-31T21:03:56.352 DEBUG:boto:Data: 2023-05-31T21:03:56.352 DEBUG:boto:Headers: {} 2023-05-31T21:03:56.353 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T21:03:56.353 DEBUG:boto:Port: 8000 2023-05-31T21:03:56.353 DEBUG:boto:Params: {} 2023-05-31T21:03:56.354 DEBUG:boto:Token: None 2023-05-31T21:03:56.354 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 21:03:56 GMT /pklxkd-62/?versions 2023-05-31T21:03:56.354 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:bk8pE3XvRCrXX9DApN+3CxE/mtU= 2023-05-31T21:03:56.355 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 21:03:56 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:bk8pE3XvRCrXX9DApN+3CxE/mtU=', 'Content-Length': '0'} 2023-05-31T21:03:56.357 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ee6e63c868fc3829-006477b63c-1630-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 21:03:56 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:03:56.358 DEBUG:boto:b'pklxkd-62999falsedummynulltrue2023-05-31T21:03:30.980Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T21:03:56.358 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2023-05-31T21:03:56.359 DEBUG:boto:path=/pklxkd-62/dummy 2023-05-31T21:03:56.359 DEBUG:boto:auth_path=/pklxkd-62/dummy 2023-05-31T21:03:56.360 DEBUG:boto:path=/pklxkd-62/dummy?versionId=null 2023-05-31T21:03:56.360 DEBUG:boto:auth_path=/pklxkd-62/dummy?versionId=null 2023-05-31T21:03:56.361 DEBUG:boto:Method: GET 2023-05-31T21:03:56.361 DEBUG:boto:Path: /pklxkd-62/dummy?versionId=null 2023-05-31T21:03:56.361 DEBUG:boto:Data: 2023-05-31T21:03:56.362 DEBUG:boto:Headers: {} 2023-05-31T21:03:56.362 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T21:03:56.363 DEBUG:boto:Port: 8001 2023-05-31T21:03:56.363 DEBUG:boto:Params: {} 2023-05-31T21:03:56.364 DEBUG:boto:Token: None 2023-05-31T21:03:56.364 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 21:03:56 GMT /pklxkd-62/dummy?versionId=null 2023-05-31T21:03:56.364 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:oamp3GHjRnnZbTprW13hm6WybkM= 2023-05-31T21:03:56.365 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 21:03:56 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:oamp3GHjRnnZbTprW13hm6WybkM=', 'Content-Length': '0'} 2023-05-31T21:03:56.367 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 21:03:30 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007ae3b8360d18a7e7-006477b63c-1f4e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 21:03:56 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:03:56.368 DEBUG:boto:path=/pklxkd-62/dummy 2023-05-31T21:03:56.368 DEBUG:boto:auth_path=/pklxkd-62/dummy 2023-05-31T21:03:56.369 DEBUG:boto:path=/pklxkd-62/dummy?versionId=null 2023-05-31T21:03:56.369 DEBUG:boto:auth_path=/pklxkd-62/dummy?versionId=null 2023-05-31T21:03:56.369 DEBUG:boto:Method: GET 2023-05-31T21:03:56.370 DEBUG:boto:Path: /pklxkd-62/dummy?versionId=null 2023-05-31T21:03:56.370 DEBUG:boto:Data: 2023-05-31T21:03:56.370 DEBUG:boto:Headers: {} 2023-05-31T21:03:56.371 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T21:03:56.371 DEBUG:boto:Port: 8000 2023-05-31T21:03:56.371 DEBUG:boto:Params: {} 2023-05-31T21:03:56.372 DEBUG:boto:Token: None 2023-05-31T21:03:56.372 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 21:03:56 GMT /pklxkd-62/dummy?versionId=null 2023-05-31T21:03:56.373 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:oamp3GHjRnnZbTprW13hm6WybkM= 2023-05-31T21:03:56.373 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 21:03:56 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:oamp3GHjRnnZbTprW13hm6WybkM=', 'Content-Length': '0'} 2023-05-31T21:03:56.375 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 21:03:30 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000459c57e9a02f33fe-006477b63c-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 21:03:56 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:03:56.376 DEBUG:boto:path=/pklxkd-62/dummy 2023-05-31T21:03:56.377 DEBUG:boto:auth_path=/pklxkd-62/dummy 2023-05-31T21:03:56.377 DEBUG:boto:path=/pklxkd-62/dummy?versionId=null 2023-05-31T21:03:56.377 DEBUG:boto:auth_path=/pklxkd-62/dummy?versionId=null 2023-05-31T21:03:56.378 DEBUG:boto:Method: HEAD 2023-05-31T21:03:56.378 DEBUG:boto:Path: /pklxkd-62/dummy?versionId=null 2023-05-31T21:03:56.378 DEBUG:boto:Data: 2023-05-31T21:03:56.379 DEBUG:boto:Headers: {} 2023-05-31T21:03:56.379 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T21:03:56.379 DEBUG:boto:Port: 8001 2023-05-31T21:03:56.380 DEBUG:boto:Params: {} 2023-05-31T21:03:56.380 DEBUG:boto:Token: None 2023-05-31T21:03:56.381 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 21:03:56 GMT /pklxkd-62/dummy?versionId=null 2023-05-31T21:03:56.381 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:SiCnIBuOxFCPJjRMcBKFlWoRSgE= 2023-05-31T21:03:56.382 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 21:03:56 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:SiCnIBuOxFCPJjRMcBKFlWoRSgE=', 'Content-Length': '0'} 2023-05-31T21:03:56.383 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 21:03:30 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000bd57271f54393d02-006477b63c-1f4e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 21:03:56 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:03:56.384 DEBUG:boto:path=/pklxkd-62/dummy 2023-05-31T21:03:56.384 DEBUG:boto:auth_path=/pklxkd-62/dummy 2023-05-31T21:03:56.385 DEBUG:boto:path=/pklxkd-62/dummy?versionId=null 2023-05-31T21:03:56.385 DEBUG:boto:auth_path=/pklxkd-62/dummy?versionId=null 2023-05-31T21:03:56.385 DEBUG:boto:Method: HEAD 2023-05-31T21:03:56.386 DEBUG:boto:Path: /pklxkd-62/dummy?versionId=null 2023-05-31T21:03:56.386 DEBUG:boto:Data: 2023-05-31T21:03:56.386 DEBUG:boto:Headers: {} 2023-05-31T21:03:56.387 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T21:03:56.387 DEBUG:boto:Port: 8000 2023-05-31T21:03:56.388 DEBUG:boto:Params: {} 2023-05-31T21:03:56.388 DEBUG:boto:Token: None 2023-05-31T21:03:56.389 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 21:03:56 GMT /pklxkd-62/dummy?versionId=null 2023-05-31T21:03:56.389 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:SiCnIBuOxFCPJjRMcBKFlWoRSgE= 2023-05-31T21:03:56.389 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 21:03:56 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:SiCnIBuOxFCPJjRMcBKFlWoRSgE=', 'Content-Length': '0'} 2023-05-31T21:03:56.391 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 21:03:30 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004fa8496a2437cef1-006477b63c-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 21:03:56 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:03:56.392 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2023-05-31T21:03:56.392 DEBUG:boto:path=/pklxkd-62/dummy 2023-05-31T21:03:56.393 DEBUG:boto:auth_path=/pklxkd-62/dummy 2023-05-31T21:03:56.393 DEBUG:boto:path=/pklxkd-62/dummy?versionId=null 2023-05-31T21:03:56.393 DEBUG:boto:auth_path=/pklxkd-62/dummy?versionId=null 2023-05-31T21:03:56.394 DEBUG:boto:Method: GET 2023-05-31T21:03:56.394 DEBUG:boto:Path: /pklxkd-62/dummy?versionId=null 2023-05-31T21:03:56.395 DEBUG:boto:Data: 2023-05-31T21:03:56.395 DEBUG:boto:Headers: {} 2023-05-31T21:03:56.395 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T21:03:56.396 DEBUG:boto:Port: 8001 2023-05-31T21:03:56.396 DEBUG:boto:Params: {} 2023-05-31T21:03:56.396 DEBUG:boto:Token: None 2023-05-31T21:03:56.397 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 21:03:56 GMT /pklxkd-62/dummy?versionId=null 2023-05-31T21:03:56.398 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:oamp3GHjRnnZbTprW13hm6WybkM= 2023-05-31T21:03:56.398 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 21:03:56 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:oamp3GHjRnnZbTprW13hm6WybkM=', 'Content-Length': '0'} 2023-05-31T21:03:56.400 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 21:03:30 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000027020237cdb7b5d9-006477b63c-1f4e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 21:03:56 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:03:56.401 DEBUG:boto:path=/pklxkd-62/dummy 2023-05-31T21:03:56.401 DEBUG:boto:auth_path=/pklxkd-62/dummy 2023-05-31T21:03:56.401 DEBUG:boto:path=/pklxkd-62/dummy?versionId=null 2023-05-31T21:03:56.402 DEBUG:boto:auth_path=/pklxkd-62/dummy?versionId=null 2023-05-31T21:03:56.402 DEBUG:boto:Method: GET 2023-05-31T21:03:56.403 DEBUG:boto:Path: /pklxkd-62/dummy?versionId=null 2023-05-31T21:03:56.403 DEBUG:boto:Data: 2023-05-31T21:03:56.403 DEBUG:boto:Headers: {} 2023-05-31T21:03:56.404 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T21:03:56.404 DEBUG:boto:Port: 8000 2023-05-31T21:03:56.404 DEBUG:boto:Params: {} 2023-05-31T21:03:56.405 DEBUG:boto:Token: None 2023-05-31T21:03:56.405 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 21:03:56 GMT /pklxkd-62/dummy?versionId=null 2023-05-31T21:03:56.406 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:oamp3GHjRnnZbTprW13hm6WybkM= 2023-05-31T21:03:56.406 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 21:03:56 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:oamp3GHjRnnZbTprW13hm6WybkM=', 'Content-Length': '0'} 2023-05-31T21:03:56.408 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 21:03:30 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000044bfd8a98b54aa9c-006477b63c-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 21:03:56 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:03:56.409 DEBUG:boto:path=/pklxkd-62/ 2023-05-31T21:03:56.409 DEBUG:boto:auth_path=/pklxkd-62/ 2023-05-31T21:03:56.410 DEBUG:boto:path=/pklxkd-62/?prefix=dummy 2023-05-31T21:03:56.410 DEBUG:boto:auth_path=/pklxkd-62/?prefix=dummy 2023-05-31T21:03:56.410 DEBUG:boto:Method: GET 2023-05-31T21:03:56.411 DEBUG:boto:Path: /pklxkd-62/?prefix=dummy 2023-05-31T21:03:56.411 DEBUG:boto:Data: 2023-05-31T21:03:56.411 DEBUG:boto:Headers: {} 2023-05-31T21:03:56.412 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T21:03:56.412 DEBUG:boto:Port: 8001 2023-05-31T21:03:56.413 DEBUG:boto:Params: {} 2023-05-31T21:03:56.413 DEBUG:boto:Token: None 2023-05-31T21:03:56.413 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 21:03:56 GMT /pklxkd-62/ 2023-05-31T21:03:56.414 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:EwGU2D20q2RjbJi0OLHhUnnBNzY= 2023-05-31T21:03:56.414 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 21:03:56 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:EwGU2D20q2RjbJi0OLHhUnnBNzY=', 'Content-Length': '0'} 2023-05-31T21:03:56.417 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000005795073693c659af-006477b63c-1f4e-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 21:03:56 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:03:56.417 DEBUG:boto:b'pklxkd-62dummy1000falsedummy2023-05-31T21:03:30.980Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T21:03:56.418 DEBUG:boto:path=/pklxkd-62/ 2023-05-31T21:03:56.418 DEBUG:boto:auth_path=/pklxkd-62/ 2023-05-31T21:03:56.419 DEBUG:boto:path=/pklxkd-62/?prefix=dummy 2023-05-31T21:03:56.419 DEBUG:boto:auth_path=/pklxkd-62/?prefix=dummy 2023-05-31T21:03:56.419 DEBUG:boto:Method: GET 2023-05-31T21:03:56.420 DEBUG:boto:Path: /pklxkd-62/?prefix=dummy 2023-05-31T21:03:56.420 DEBUG:boto:Data: 2023-05-31T21:03:56.421 DEBUG:boto:Headers: {} 2023-05-31T21:03:56.421 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T21:03:56.422 DEBUG:boto:Port: 8000 2023-05-31T21:03:56.422 DEBUG:boto:Params: {} 2023-05-31T21:03:56.422 DEBUG:boto:Token: None 2023-05-31T21:03:56.423 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 21:03:56 GMT /pklxkd-62/ 2023-05-31T21:03:56.423 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:EwGU2D20q2RjbJi0OLHhUnnBNzY= 2023-05-31T21:03:56.424 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 21:03:56 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:EwGU2D20q2RjbJi0OLHhUnnBNzY=', 'Content-Length': '0'} 2023-05-31T21:03:56.426 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000002b43319c63d01c69-006477b63c-1630-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 21:03:56 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:03:56.468 DEBUG:boto:b'pklxkd-62dummy1000falsedummy2023-05-31T21:03:30.980Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T21:03:56.469 DEBUG:boto:path=/pklxkd-62/dummy 2023-05-31T21:03:56.469 DEBUG:boto:auth_path=/pklxkd-62/dummy 2023-05-31T21:03:56.470 DEBUG:boto:Method: HEAD 2023-05-31T21:03:56.470 DEBUG:boto:Path: /pklxkd-62/dummy 2023-05-31T21:03:56.470 DEBUG:boto:Data: 2023-05-31T21:03:56.471 DEBUG:boto:Headers: {} 2023-05-31T21:03:56.471 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T21:03:56.472 DEBUG:boto:Port: 8001 2023-05-31T21:03:56.472 DEBUG:boto:Params: {} 2023-05-31T21:03:56.472 DEBUG:boto:Token: None 2023-05-31T21:03:56.473 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 21:03:56 GMT /pklxkd-62/dummy 2023-05-31T21:03:56.473 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:CuS3fZTSY6h+aWHZu0D+QSS7U1M= 2023-05-31T21:03:56.474 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 21:03:56 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:CuS3fZTSY6h+aWHZu0D+QSS7U1M=', 'Content-Length': '0'} 2023-05-31T21:03:56.475 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 21:03:30 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003b8ecaefc92039c3-006477b63c-1f4e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 21:03:56 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:03:56.476 DEBUG:boto:path=/pklxkd-62/dummy 2023-05-31T21:03:56.476 DEBUG:boto:auth_path=/pklxkd-62/dummy 2023-05-31T21:03:56.476 DEBUG:boto:Method: HEAD 2023-05-31T21:03:56.477 DEBUG:boto:Path: /pklxkd-62/dummy 2023-05-31T21:03:56.477 DEBUG:boto:Data: 2023-05-31T21:03:56.478 DEBUG:boto:Headers: {} 2023-05-31T21:03:56.478 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T21:03:56.478 DEBUG:boto:Port: 8000 2023-05-31T21:03:56.479 DEBUG:boto:Params: {} 2023-05-31T21:03:56.479 DEBUG:boto:Token: None 2023-05-31T21:03:56.479 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 21:03:56 GMT /pklxkd-62/dummy 2023-05-31T21:03:56.480 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:CuS3fZTSY6h+aWHZu0D+QSS7U1M= 2023-05-31T21:03:56.480 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 21:03:56 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:CuS3fZTSY6h+aWHZu0D+QSS7U1M=', 'Content-Length': '0'} 2023-05-31T21:03:56.482 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 21:03:30 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008bd29ba630894af2-006477b63c-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 21:03:56 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:03:56.483 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2023-05-31T21:03:56.483 DEBUG:boto:path=/pklxkd-62/dummy 2023-05-31T21:03:56.484 DEBUG:boto:auth_path=/pklxkd-62/dummy 2023-05-31T21:03:56.484 DEBUG:boto:Method: GET 2023-05-31T21:03:56.485 DEBUG:boto:Path: /pklxkd-62/dummy 2023-05-31T21:03:56.485 DEBUG:boto:Data: 2023-05-31T21:03:56.486 DEBUG:boto:Headers: {} 2023-05-31T21:03:56.486 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T21:03:56.486 DEBUG:boto:Port: 8001 2023-05-31T21:03:56.487 DEBUG:boto:Params: {} 2023-05-31T21:03:56.487 DEBUG:boto:Token: None 2023-05-31T21:03:56.487 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 21:03:56 GMT /pklxkd-62/dummy 2023-05-31T21:03:56.488 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:v1dfklaSUWQ8tuvj/lZWhcUyrbk= 2023-05-31T21:03:56.488 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 21:03:56 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:v1dfklaSUWQ8tuvj/lZWhcUyrbk=', 'Content-Length': '0'} 2023-05-31T21:03:56.491 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 21:03:30 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000fd2d55da620d767d-006477b63c-1f4e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 21:03:56 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:03:56.492 DEBUG:boto:path=/pklxkd-62/dummy 2023-05-31T21:03:56.493 DEBUG:boto:auth_path=/pklxkd-62/dummy 2023-05-31T21:03:56.494 DEBUG:boto:Method: GET 2023-05-31T21:03:56.495 DEBUG:boto:Path: /pklxkd-62/dummy 2023-05-31T21:03:56.496 DEBUG:boto:Data: 2023-05-31T21:03:56.496 DEBUG:boto:Headers: {} 2023-05-31T21:03:56.497 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T21:03:56.498 DEBUG:boto:Port: 8000 2023-05-31T21:03:56.499 DEBUG:boto:Params: {} 2023-05-31T21:03:56.499 DEBUG:boto:Token: None 2023-05-31T21:03:56.500 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 21:03:56 GMT /pklxkd-62/dummy 2023-05-31T21:03:56.500 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:v1dfklaSUWQ8tuvj/lZWhcUyrbk= 2023-05-31T21:03:56.501 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 21:03:56 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:v1dfklaSUWQ8tuvj/lZWhcUyrbk=', 'Content-Length': '0'} 2023-05-31T21:03:56.503 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 21:03:30 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a8ec4236ba2fc1b6-006477b63c-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 21:03:56 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:03:56.504 DEBUG:boto:path=/pklxkd-62/ 2023-05-31T21:03:56.504 DEBUG:boto:auth_path=/pklxkd-62/ 2023-05-31T21:03:56.505 DEBUG:boto:path=/pklxkd-62/?prefix=dummy 2023-05-31T21:03:56.505 DEBUG:boto:auth_path=/pklxkd-62/?prefix=dummy 2023-05-31T21:03:56.505 DEBUG:boto:Method: GET 2023-05-31T21:03:56.506 DEBUG:boto:Path: /pklxkd-62/?prefix=dummy 2023-05-31T21:03:56.506 DEBUG:boto:Data: 2023-05-31T21:03:56.506 DEBUG:boto:Headers: {} 2023-05-31T21:03:56.507 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T21:03:56.507 DEBUG:boto:Port: 8001 2023-05-31T21:03:56.507 DEBUG:boto:Params: {} 2023-05-31T21:03:56.508 DEBUG:boto:Token: None 2023-05-31T21:03:56.508 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 21:03:56 GMT /pklxkd-62/ 2023-05-31T21:03:56.509 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:EwGU2D20q2RjbJi0OLHhUnnBNzY= 2023-05-31T21:03:56.509 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 21:03:56 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:EwGU2D20q2RjbJi0OLHhUnnBNzY=', 'Content-Length': '0'} 2023-05-31T21:03:56.512 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000858df26d3e182f0a-006477b63c-1f4e-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 21:03:56 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:03:56.512 DEBUG:boto:b'pklxkd-62dummy1000falsedummy2023-05-31T21:03:30.980Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T21:03:56.513 DEBUG:boto:path=/pklxkd-62/ 2023-05-31T21:03:56.513 DEBUG:boto:auth_path=/pklxkd-62/ 2023-05-31T21:03:56.514 DEBUG:boto:path=/pklxkd-62/?prefix=dummy 2023-05-31T21:03:56.514 DEBUG:boto:auth_path=/pklxkd-62/?prefix=dummy 2023-05-31T21:03:56.515 DEBUG:boto:Method: GET 2023-05-31T21:03:56.515 DEBUG:boto:Path: /pklxkd-62/?prefix=dummy 2023-05-31T21:03:56.515 DEBUG:boto:Data: 2023-05-31T21:03:56.516 DEBUG:boto:Headers: {} 2023-05-31T21:03:56.516 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T21:03:56.517 DEBUG:boto:Port: 8000 2023-05-31T21:03:56.517 DEBUG:boto:Params: {} 2023-05-31T21:03:56.517 DEBUG:boto:Token: None 2023-05-31T21:03:56.518 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 21:03:56 GMT /pklxkd-62/ 2023-05-31T21:03:56.519 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:EwGU2D20q2RjbJi0OLHhUnnBNzY= 2023-05-31T21:03:56.519 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 21:03:56 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:EwGU2D20q2RjbJi0OLHhUnnBNzY=', 'Content-Length': '0'} 2023-05-31T21:03:56.522 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000061f654535e5c85b7-006477b63c-1630-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 21:03:56 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:03:56.522 DEBUG:boto:b'pklxkd-62dummy1000falsedummy2023-05-31T21:03:30.980Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T21:03:56.523 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=pklxkd-62 zones={a2, a1} 2023-05-31T21:03:56.523 DEBUG:tasks.rgw.c1.client.1:waiting for process to exit 2023-05-31T21:03:56.524 INFO:teuthology.orchestra.run:waiting for 300 2023-05-31T21:03:56.525 INFO:tasks.rgw.c1.client.1.smithi099.stdout:2023-05-31T21:03:56.520+0000 7ff254896700 -1 received signal: Terminated from /usr/bin/python3 /bin/daemon-helper term radosgw --rgw-frontends beast port=8001 -n client.1 --cluster c1 -k /etc/ceph/c1.client.1.keyring --log-file /var/log/ceph/rgw.c1.client.1.log --rgw_ops_log_socket_path /home/ubuntu/cephtest/rgw.opslog.c1.client.1.sock --foreground --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm (PID: 167133) UID: 0 2023-05-31T21:03:56.525 INFO:tasks.rgw.c1.client.1.smithi099.stdout:2023-05-31T21:03:56.520+0000 7ff258aff7c0 -1 shutting down 2023-05-31T21:03:57.126 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.1 is failed for ~0s 2023-05-31T21:04:02.623 INFO:tasks.rgw.c1.client.1:Stopped 2023-05-31T21:04:02.623 DEBUG:boto:path=/pklxkd-62/ 2023-05-31T21:04:02.624 DEBUG:boto:auth_path=/pklxkd-62/ 2023-05-31T21:04:02.624 DEBUG:boto:Method: HEAD 2023-05-31T21:04:02.624 DEBUG:boto:Path: /pklxkd-62/ 2023-05-31T21:04:02.625 DEBUG:boto:Data: 2023-05-31T21:04:02.625 DEBUG:boto:Headers: {} 2023-05-31T21:04:02.625 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T21:04:02.626 DEBUG:boto:Port: 8000 2023-05-31T21:04:02.626 DEBUG:boto:Params: {} 2023-05-31T21:04:02.626 DEBUG:boto:Token: None 2023-05-31T21:04:02.627 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 21:04:02 GMT /pklxkd-62/ 2023-05-31T21:04:02.627 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:7Mwtn8SeWW2qW5CNDpEq4ZDsACU= 2023-05-31T21:04:02.628 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 21:04:02 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:7Mwtn8SeWW2qW5CNDpEq4ZDsACU=', 'Content-Length': '0'} 2023-05-31T21:04:02.630 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '3'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000a7164b71350f5a3d-006477b642-1630-a1'), ('Content-Length', '0'), ('Date', 'Wed, 31 May 2023 21:04:02 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:04:02.631 DEBUG:boto:path=/pklxkd-62/obj11 2023-05-31T21:04:02.631 DEBUG:boto:auth_path=/pklxkd-62/obj11 2023-05-31T21:04:02.632 DEBUG:boto:Method: PUT 2023-05-31T21:04:02.632 DEBUG:boto:Path: /pklxkd-62/obj11 2023-05-31T21:04:02.632 DEBUG:boto:Data: 2023-05-31T21:04:02.633 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-05-31T21:04:02.633 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T21:04:02.633 DEBUG:boto:Port: 8000 2023-05-31T21:04:02.634 DEBUG:boto:Params: {} 2023-05-31T21:04:02.634 DEBUG:boto:Token: None 2023-05-31T21:04:02.634 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Wed, 31 May 2023 21:04:02 GMT /pklxkd-62/obj11 2023-05-31T21:04:02.635 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:YnPX+Ac9wYvx5TiBGzRA7C4bAhc= 2023-05-31T21:04:02.635 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Wed, 31 May 2023 21:04:02 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:YnPX+Ac9wYvx5TiBGzRA7C4bAhc='} 2023-05-31T21:04:02.642 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000048428732ee6b7e75-006477b642-1630-a1'), ('Date', 'Wed, 31 May 2023 21:04:02 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:04:02.643 DEBUG:boto:path=/pklxkd-62/ 2023-05-31T21:04:02.643 DEBUG:boto:auth_path=/pklxkd-62/ 2023-05-31T21:04:02.643 DEBUG:boto:Method: HEAD 2023-05-31T21:04:02.644 DEBUG:boto:Path: /pklxkd-62/ 2023-05-31T21:04:02.644 DEBUG:boto:Data: 2023-05-31T21:04:02.644 DEBUG:boto:Headers: {} 2023-05-31T21:04:02.645 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T21:04:02.645 DEBUG:boto:Port: 8000 2023-05-31T21:04:02.645 DEBUG:boto:Params: {} 2023-05-31T21:04:02.646 DEBUG:boto:Token: None 2023-05-31T21:04:02.646 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 21:04:02 GMT /pklxkd-62/ 2023-05-31T21:04:02.647 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:7Mwtn8SeWW2qW5CNDpEq4ZDsACU= 2023-05-31T21:04:02.647 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 21:04:02 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:7Mwtn8SeWW2qW5CNDpEq4ZDsACU=', 'Content-Length': '0'} 2023-05-31T21:04:02.649 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '2'), ('X-RGW-Bytes-Used', '6'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000fbf4a01702a5e039-006477b642-1630-a1'), ('Content-Length', '0'), ('Date', 'Wed, 31 May 2023 21:04:02 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:04:02.650 DEBUG:boto:path=/pklxkd-62/obj22 2023-05-31T21:04:02.650 DEBUG:boto:auth_path=/pklxkd-62/obj22 2023-05-31T21:04:02.651 DEBUG:boto:Method: PUT 2023-05-31T21:04:02.651 DEBUG:boto:Path: /pklxkd-62/obj22 2023-05-31T21:04:02.651 DEBUG:boto:Data: 2023-05-31T21:04:02.652 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-05-31T21:04:02.652 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T21:04:02.652 DEBUG:boto:Port: 8000 2023-05-31T21:04:02.653 DEBUG:boto:Params: {} 2023-05-31T21:04:02.653 DEBUG:boto:Token: None 2023-05-31T21:04:02.654 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Wed, 31 May 2023 21:04:02 GMT /pklxkd-62/obj22 2023-05-31T21:04:02.654 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:0B67YqUh2wjTQic2g3jCjPKqN0w= 2023-05-31T21:04:02.654 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Wed, 31 May 2023 21:04:02 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:0B67YqUh2wjTQic2g3jCjPKqN0w='} 2023-05-31T21:04:02.696 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000249157a071db1eb3-006477b642-1630-a1'), ('Date', 'Wed, 31 May 2023 21:04:02 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:04:02.697 DEBUG:boto:path=/pklxkd-62/ 2023-05-31T21:04:02.697 DEBUG:boto:auth_path=/pklxkd-62/ 2023-05-31T21:04:02.698 DEBUG:boto:Method: HEAD 2023-05-31T21:04:02.698 DEBUG:boto:Path: /pklxkd-62/ 2023-05-31T21:04:02.698 DEBUG:boto:Data: 2023-05-31T21:04:02.699 DEBUG:boto:Headers: {} 2023-05-31T21:04:02.699 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T21:04:02.699 DEBUG:boto:Port: 8000 2023-05-31T21:04:02.700 DEBUG:boto:Params: {} 2023-05-31T21:04:02.700 DEBUG:boto:Token: None 2023-05-31T21:04:02.701 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 21:04:02 GMT /pklxkd-62/ 2023-05-31T21:04:02.701 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:7Mwtn8SeWW2qW5CNDpEq4ZDsACU= 2023-05-31T21:04:02.701 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 21:04:02 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:7Mwtn8SeWW2qW5CNDpEq4ZDsACU=', 'Content-Length': '0'} 2023-05-31T21:04:02.704 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '3'), ('X-RGW-Bytes-Used', '9'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000005816426fda6fe553-006477b642-1630-a1'), ('Content-Length', '0'), ('Date', 'Wed, 31 May 2023 21:04:02 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:04:02.704 DEBUG:boto:path=/pklxkd-62/obj33 2023-05-31T21:04:02.705 DEBUG:boto:auth_path=/pklxkd-62/obj33 2023-05-31T21:04:02.705 DEBUG:boto:Method: PUT 2023-05-31T21:04:02.705 DEBUG:boto:Path: /pklxkd-62/obj33 2023-05-31T21:04:02.706 DEBUG:boto:Data: 2023-05-31T21:04:02.706 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-05-31T21:04:02.706 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T21:04:02.707 DEBUG:boto:Port: 8000 2023-05-31T21:04:02.707 DEBUG:boto:Params: {} 2023-05-31T21:04:02.708 DEBUG:boto:Token: None 2023-05-31T21:04:02.708 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Wed, 31 May 2023 21:04:02 GMT /pklxkd-62/obj33 2023-05-31T21:04:02.708 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:xQsnTwP57OezxWrScUjsOQqYROM= 2023-05-31T21:04:02.709 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Wed, 31 May 2023 21:04:02 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:xQsnTwP57OezxWrScUjsOQqYROM='} 2023-05-31T21:04:02.719 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000005245e8738e29a6df-006477b642-1630-a1'), ('Date', 'Wed, 31 May 2023 21:04:02 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:04:02.720 DEBUG:boto:path=/pklxkd-62/ 2023-05-31T21:04:02.720 DEBUG:boto:auth_path=/pklxkd-62/ 2023-05-31T21:04:02.721 DEBUG:boto:Method: HEAD 2023-05-31T21:04:02.721 DEBUG:boto:Path: /pklxkd-62/ 2023-05-31T21:04:02.721 DEBUG:boto:Data: 2023-05-31T21:04:02.722 DEBUG:boto:Headers: {} 2023-05-31T21:04:02.722 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T21:04:02.722 DEBUG:boto:Port: 8000 2023-05-31T21:04:02.723 DEBUG:boto:Params: {} 2023-05-31T21:04:02.723 DEBUG:boto:Token: None 2023-05-31T21:04:02.724 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 21:04:02 GMT /pklxkd-62/ 2023-05-31T21:04:02.724 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:7Mwtn8SeWW2qW5CNDpEq4ZDsACU= 2023-05-31T21:04:02.724 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 21:04:02 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:7Mwtn8SeWW2qW5CNDpEq4ZDsACU=', 'Content-Length': '0'} 2023-05-31T21:04:02.727 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '4'), ('X-RGW-Bytes-Used', '12'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000009cad434d73888927-006477b642-1630-a1'), ('Content-Length', '0'), ('Date', 'Wed, 31 May 2023 21:04:02 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:04:02.727 DEBUG:boto:path=/pklxkd-62/obj44 2023-05-31T21:04:02.728 DEBUG:boto:auth_path=/pklxkd-62/obj44 2023-05-31T21:04:02.728 DEBUG:boto:Method: PUT 2023-05-31T21:04:02.728 DEBUG:boto:Path: /pklxkd-62/obj44 2023-05-31T21:04:02.729 DEBUG:boto:Data: 2023-05-31T21:04:02.729 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-05-31T21:04:02.729 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T21:04:02.730 DEBUG:boto:Port: 8000 2023-05-31T21:04:02.730 DEBUG:boto:Params: {} 2023-05-31T21:04:02.730 DEBUG:boto:Token: None 2023-05-31T21:04:02.731 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Wed, 31 May 2023 21:04:02 GMT /pklxkd-62/obj44 2023-05-31T21:04:02.731 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:ULegHNqDOq4T+V9+BYk2/4uVKhM= 2023-05-31T21:04:02.732 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Wed, 31 May 2023 21:04:02 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:ULegHNqDOq4T+V9+BYk2/4uVKhM='} 2023-05-31T21:04:02.736 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.1 has been restored 2023-05-31T21:04:02.740 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000334a902ffb56f5c8-006477b642-1630-a1'), ('Date', 'Wed, 31 May 2023 21:04:02 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:04:02.741 DEBUG:boto:path=/pklxkd-62/ 2023-05-31T21:04:02.741 DEBUG:boto:auth_path=/pklxkd-62/ 2023-05-31T21:04:02.742 DEBUG:boto:Method: HEAD 2023-05-31T21:04:02.742 DEBUG:boto:Path: /pklxkd-62/ 2023-05-31T21:04:02.742 DEBUG:boto:Data: 2023-05-31T21:04:02.743 DEBUG:boto:Headers: {} 2023-05-31T21:04:02.743 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T21:04:02.743 DEBUG:boto:Port: 8000 2023-05-31T21:04:02.744 DEBUG:boto:Params: {} 2023-05-31T21:04:02.744 DEBUG:boto:Token: None 2023-05-31T21:04:02.744 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 21:04:02 GMT /pklxkd-62/ 2023-05-31T21:04:02.745 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:7Mwtn8SeWW2qW5CNDpEq4ZDsACU= 2023-05-31T21:04:02.745 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 21:04:02 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:7Mwtn8SeWW2qW5CNDpEq4ZDsACU=', 'Content-Length': '0'} 2023-05-31T21:04:02.747 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '5'), ('X-RGW-Bytes-Used', '15'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000c40982af08a2ad28-006477b642-1630-a1'), ('Content-Length', '0'), ('Date', 'Wed, 31 May 2023 21:04:02 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:04:02.748 DEBUG:boto:path=/pklxkd-62/obj55 2023-05-31T21:04:02.748 DEBUG:boto:auth_path=/pklxkd-62/obj55 2023-05-31T21:04:02.749 DEBUG:boto:Method: PUT 2023-05-31T21:04:02.749 DEBUG:boto:Path: /pklxkd-62/obj55 2023-05-31T21:04:02.749 DEBUG:boto:Data: 2023-05-31T21:04:02.750 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-05-31T21:04:02.750 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T21:04:02.750 DEBUG:boto:Port: 8000 2023-05-31T21:04:02.751 DEBUG:boto:Params: {} 2023-05-31T21:04:02.751 DEBUG:boto:Token: None 2023-05-31T21:04:02.751 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Wed, 31 May 2023 21:04:02 GMT /pklxkd-62/obj55 2023-05-31T21:04:02.752 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:H+BRM4oDyjfEHeYsZjjtSCZ/2Mk= 2023-05-31T21:04:02.752 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Wed, 31 May 2023 21:04:02 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:H+BRM4oDyjfEHeYsZjjtSCZ/2Mk='} 2023-05-31T21:04:02.796 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000004e1e5c9d1e2c731b-006477b642-1630-a1'), ('Date', 'Wed, 31 May 2023 21:04:02 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:04:02.797 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'init', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-05-31T21:04:02.797 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'init', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-05-31T21:04:02.798 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync init --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 2023-05-31T21:04:02.854 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2023-05-31T21:04:02.855 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2023-05-31T21:04:02.907 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T21:04:02.903+0000 7fce01d79780 5 RGW-SYNC:data: waiting on data sync status lease 2023-05-31T21:04:02.909 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T21:04:02.904+0000 7fce01d79780 5 RGW-SYNC:data: acquired data sync status lease 2023-05-31T21:04:02.950 INFO:tasks.rgw.c1.client.1:Restarting daemon 2023-05-31T21:04:02.951 DEBUG:teuthology.orchestra.run.smithi099:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term radosgw --rgw-frontends 'beast port=8001' -n client.1 --cluster c1 -k /etc/ceph/c1.client.1.keyring --log-file /var/log/ceph/rgw.c1.client.1.log --rgw_ops_log_socket_path /home/ubuntu/cephtest/rgw.opslog.c1.client.1.sock --foreground --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm | sudo tee /var/log/ceph/rgw.c1.client.1.stdout 2>&1 2023-05-31T21:04:02.953 INFO:tasks.rgw.c1.client.1:Started 2023-05-31T21:04:02.953 DEBUG:teuthology.orchestra.run.smithi099:> curl http://smithi099.front.sepia.ceph.com:8001 2023-05-31T21:04:02.975 DEBUG:teuthology.orchestra.run:got remote process result: 7 2023-05-31T21:04:02.976 INFO:teuthology.orchestra.run.smithi099.stderr: % Total % Received % Xferd Average Speed Time Time Time Current 2023-05-31T21:04:02.976 INFO:teuthology.orchestra.run.smithi099.stderr: Dload Upload Total Spent Left Speed 2023-05-31T21:04:02.976 INFO:teuthology.orchestra.run.smithi099.stderr: 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0curl: (7) Failed to connect to smithi099.front.sepia.ceph.com port 8001: Connection refused 2023-05-31T21:04:03.977 DEBUG:teuthology.orchestra.run.smithi099:> curl http://smithi099.front.sepia.ceph.com:8001 2023-05-31T21:04:03.999 INFO:teuthology.orchestra.run.smithi099.stderr: % Total % Received % Xferd Average Speed Time Time Time Current 2023-05-31T21:04:03.999 INFO:teuthology.orchestra.run.smithi099.stderr: Dload Upload Total Spent Left Speed 2023-05-31T21:04:04.001 INFO:teuthology.orchestra.run.smithi099.stderr: 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 100 214 0 214 0 0 208k 0 --:--:-- --:--:-- --:--:-- 208k 2023-05-31T21:04:04.001 INFO:teuthology.orchestra.run.smithi099.stdout:anonymous 2023-05-31T21:04:04.002 DEBUG:rgw_multi.tests:bucket checkpoint: source=a1 target=a2 bucket=pklxkd-62 2023-05-31T21:04:04.002 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'pklxkd-62', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-05-31T21:04:04.003 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'pklxkd-62', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-05-31T21:04:04.004 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket pklxkd-62 --source-zone a1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2023-05-31T21:04:04.100 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2023-05-31T21:04:04.100 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2023-05-31T21:04:04.138 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T21:04:04.133+0000 7f5508628780 1 waiting for incremental sync to catch up: 2023-05-31T21:04:04.138 INFO:teuthology.orchestra.run.smithi099.stderr: local status: [, , , , , 00000000001.93.6, , , , , ] 2023-05-31T21:04:04.138 INFO:teuthology.orchestra.run.smithi099.stderr: remote markers: [, , , , , 00000000001.93.6, , , 00000000001.99.6, 00000000002.123.6, 00000000002.114.6] 2023-05-31T21:04:09.139 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T21:04:09.134+0000 7f5508628780 1 waiting for incremental sync to catch up: 2023-05-31T21:04:09.140 INFO:teuthology.orchestra.run.smithi099.stderr: local status: [, , , , , 00000000001.93.6, , , , , ] 2023-05-31T21:04:09.140 INFO:teuthology.orchestra.run.smithi099.stderr: remote markers: [, , , , , 00000000001.93.6, , , 00000000001.99.6, 00000000002.123.6, 00000000002.114.6] 2023-05-31T21:04:14.141 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T21:04:14.136+0000 7f5508628780 1 waiting for incremental sync to catch up: 2023-05-31T21:04:14.141 INFO:teuthology.orchestra.run.smithi099.stderr: local status: [, , , , , 00000000001.93.6, , , , , ] 2023-05-31T21:04:14.141 INFO:teuthology.orchestra.run.smithi099.stderr: remote markers: [, , , , , 00000000001.93.6, , , 00000000001.99.6, 00000000002.123.6, 00000000002.114.6] 2023-05-31T21:04:19.142 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T21:04:19.138+0000 7f5508628780 1 waiting for incremental sync to catch up: 2023-05-31T21:04:19.142 INFO:teuthology.orchestra.run.smithi099.stderr: local status: [, , , , , 00000000001.93.6, , , , , ] 2023-05-31T21:04:19.142 INFO:teuthology.orchestra.run.smithi099.stderr: remote markers: [, , , , , 00000000001.93.6, , , 00000000001.99.6, 00000000002.123.6, 00000000002.114.6] 2023-05-31T21:04:24.143 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T21:04:24.140+0000 7f5508628780 1 waiting for incremental sync to catch up: 2023-05-31T21:04:24.144 INFO:teuthology.orchestra.run.smithi099.stderr: local status: [, , , , , 00000000001.93.6, , , , , ] 2023-05-31T21:04:24.144 INFO:teuthology.orchestra.run.smithi099.stderr: remote markers: [, , , , , 00000000001.93.6, , , 00000000001.99.6, 00000000002.123.6, 00000000002.114.6] 2023-05-31T21:04:29.144 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T21:04:29.140+0000 7f5508628780 1 waiting for incremental sync to catch up: 2023-05-31T21:04:29.144 INFO:teuthology.orchestra.run.smithi099.stderr: local status: [, , , , , 00000000001.93.6, , , , , ] 2023-05-31T21:04:29.145 INFO:teuthology.orchestra.run.smithi099.stderr: remote markers: [, , , , , 00000000001.93.6, , , 00000000001.99.6, 00000000002.123.6, 00000000002.114.6] 2023-05-31T21:04:34.145 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T21:04:34.141+0000 7f5508628780 1 waiting for incremental sync to catch up: 2023-05-31T21:04:34.145 INFO:teuthology.orchestra.run.smithi099.stderr: local status: [, , , , , 00000000001.93.6, , , , , ] 2023-05-31T21:04:34.145 INFO:teuthology.orchestra.run.smithi099.stderr: remote markers: [, , , , , 00000000001.93.6, , , 00000000001.99.6, 00000000002.123.6, 00000000002.114.6] 2023-05-31T21:04:39.147 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T21:04:39.142+0000 7f5508628780 1 waiting for incremental sync to catch up: 2023-05-31T21:04:39.147 INFO:teuthology.orchestra.run.smithi099.stderr: local status: [, , , , , 00000000001.93.6, , , , , ] 2023-05-31T21:04:39.147 INFO:teuthology.orchestra.run.smithi099.stderr: remote markers: [, , , , , 00000000001.93.6, , , 00000000001.99.6, 00000000002.123.6, 00000000002.114.6] 2023-05-31T21:04:44.148 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T21:04:44.143+0000 7f5508628780 1 waiting for incremental sync to catch up: 2023-05-31T21:04:44.148 INFO:teuthology.orchestra.run.smithi099.stderr: local status: [, , , , , 00000000001.93.6, , , , , ] 2023-05-31T21:04:44.148 INFO:teuthology.orchestra.run.smithi099.stderr: remote markers: [, , , , , 00000000001.93.6, , , 00000000001.99.6, 00000000002.123.6, 00000000002.114.6] 2023-05-31T21:04:49.149 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T21:04:49.144+0000 7f5508628780 1 waiting for incremental sync to catch up: 2023-05-31T21:04:49.150 INFO:teuthology.orchestra.run.smithi099.stderr: local status: [, , , , , 00000000001.93.6, , , , , ] 2023-05-31T21:04:49.150 INFO:teuthology.orchestra.run.smithi099.stderr: remote markers: [, , , , , 00000000001.93.6, , , 00000000001.99.6, 00000000002.123.6, 00000000002.114.6] 2023-05-31T21:04:54.150 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T21:04:54.146+0000 7f5508628780 1 waiting for incremental sync to catch up: 2023-05-31T21:04:54.151 INFO:teuthology.orchestra.run.smithi099.stderr: local status: [, , , , , 00000000001.93.6, , , , , ] 2023-05-31T21:04:54.151 INFO:teuthology.orchestra.run.smithi099.stderr: remote markers: [, , , , , 00000000001.93.6, , , 00000000001.99.6, 00000000002.123.6, 00000000002.114.6] 2023-05-31T21:04:59.152 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T21:04:59.148+0000 7f5508628780 1 waiting for incremental sync to catch up: 2023-05-31T21:04:59.152 INFO:teuthology.orchestra.run.smithi099.stderr: local status: [, , , , , 00000000001.93.6, , , , , ] 2023-05-31T21:04:59.153 INFO:teuthology.orchestra.run.smithi099.stderr: remote markers: [, , , , , 00000000001.93.6, , , 00000000001.99.6, 00000000002.123.6, 00000000002.114.6] 2023-05-31T21:05:04.154 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T21:05:04.150+0000 7f5508628780 1 waiting for incremental sync to catch up: 2023-05-31T21:05:04.154 INFO:teuthology.orchestra.run.smithi099.stderr: local status: [, , , , , 00000000001.93.6, , , , , ] 2023-05-31T21:05:04.154 INFO:teuthology.orchestra.run.smithi099.stderr: remote markers: [, , , , , 00000000001.93.6, , , 00000000001.99.6, 00000000002.123.6, 00000000002.114.6] 2023-05-31T21:05:09.155 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T21:05:09.151+0000 7f5508628780 1 waiting for incremental sync to catch up: 2023-05-31T21:05:09.155 INFO:teuthology.orchestra.run.smithi099.stderr: local status: [, , , , , 00000000001.93.6, , , , , ] 2023-05-31T21:05:09.155 INFO:teuthology.orchestra.run.smithi099.stderr: remote markers: [, , , , , 00000000001.93.6, , , 00000000001.99.6, 00000000002.123.6, 00000000002.114.6] 2023-05-31T21:05:14.156 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T21:05:14.153+0000 7f5508628780 1 waiting for incremental sync to catch up: 2023-05-31T21:05:14.157 INFO:teuthology.orchestra.run.smithi099.stderr: local status: [, , , , , 00000000001.93.6, , , , , ] 2023-05-31T21:05:14.157 INFO:teuthology.orchestra.run.smithi099.stderr: remote markers: [, , , , , 00000000001.93.6, , , 00000000001.99.6, 00000000002.123.6, 00000000002.114.6] 2023-05-31T21:05:19.157 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T21:05:19.153+0000 7f5508628780 1 waiting for incremental sync to catch up: 2023-05-31T21:05:19.158 INFO:teuthology.orchestra.run.smithi099.stderr: local status: [, , , , , 00000000001.93.6, , , , , ] 2023-05-31T21:05:19.158 INFO:teuthology.orchestra.run.smithi099.stderr: remote markers: [, , , , , 00000000001.93.6, , , 00000000001.99.6, 00000000002.123.6, 00000000002.114.6] 2023-05-31T21:05:24.159 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T21:05:24.154+0000 7f5508628780 1 waiting for incremental sync to catch up: 2023-05-31T21:05:24.159 INFO:teuthology.orchestra.run.smithi099.stderr: local status: [, , , , , 00000000001.93.6, , , , , ] 2023-05-31T21:05:24.159 INFO:teuthology.orchestra.run.smithi099.stderr: remote markers: [, , , , , 00000000001.93.6, , , 00000000001.99.6, 00000000002.123.6, 00000000002.114.6] 2023-05-31T21:05:29.160 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T21:05:29.156+0000 7f5508628780 1 waiting for incremental sync to catch up: 2023-05-31T21:05:29.161 INFO:teuthology.orchestra.run.smithi099.stderr: local status: [, , , , , 00000000001.93.6, , , , , ] 2023-05-31T21:05:29.161 INFO:teuthology.orchestra.run.smithi099.stderr: remote markers: [, , , , , 00000000001.93.6, , , 00000000001.99.6, 00000000002.123.6, 00000000002.114.6] 2023-05-31T21:05:34.162 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T21:05:34.157+0000 7f5508628780 1 waiting for incremental sync to catch up: 2023-05-31T21:05:34.162 INFO:teuthology.orchestra.run.smithi099.stderr: local status: [, , , , , 00000000001.93.6, , , , , 00000000001.112.6] 2023-05-31T21:05:34.162 INFO:teuthology.orchestra.run.smithi099.stderr: remote markers: [, , , , , 00000000001.93.6, , , 00000000001.99.6, 00000000002.123.6, 00000000002.114.6] 2023-05-31T21:05:39.163 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T21:05:39.159+0000 7f5508628780 1 waiting for incremental sync to catch up: 2023-05-31T21:05:39.164 INFO:teuthology.orchestra.run.smithi099.stderr: local status: [, , , , , 00000000001.93.6, , , 00000000001.99.6, 00000000002.123.6, 00000000001.112.6] 2023-05-31T21:05:39.164 INFO:teuthology.orchestra.run.smithi099.stderr: remote markers: [, , , , , 00000000001.93.6, , , 00000000001.99.6, 00000000002.123.6, 00000000002.114.6] 2023-05-31T21:05:44.164 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T21:05:44.161+0000 7f5508628780 1 waiting for incremental sync to catch up: 2023-05-31T21:05:44.165 INFO:teuthology.orchestra.run.smithi099.stderr: local status: [, , , , , 00000000001.93.6, , , 00000000001.99.6, 00000000002.123.6, 00000000001.112.6] 2023-05-31T21:05:44.165 INFO:teuthology.orchestra.run.smithi099.stderr: remote markers: [, , , , , 00000000001.93.6, , , 00000000001.99.6, 00000000002.123.6, 00000000002.114.6] 2023-05-31T21:05:49.166 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T21:05:49.162+0000 7f5508628780 1 waiting for incremental sync to catch up: 2023-05-31T21:05:49.166 INFO:teuthology.orchestra.run.smithi099.stderr: local status: [, , , , , 00000000001.93.6, , , 00000000001.99.6, 00000000002.123.6, 00000000001.112.6] 2023-05-31T21:05:49.167 INFO:teuthology.orchestra.run.smithi099.stderr: remote markers: [, , , , , 00000000001.93.6, , , 00000000001.99.6, 00000000002.123.6, 00000000002.114.6] 2023-05-31T21:05:54.167 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T21:05:54.163+0000 7f5508628780 1 waiting for incremental sync to catch up: 2023-05-31T21:05:54.168 INFO:teuthology.orchestra.run.smithi099.stderr: local status: [, , , , , 00000000001.93.6, , , 00000000001.99.6, 00000000002.123.6, 00000000001.112.6] 2023-05-31T21:05:54.168 INFO:teuthology.orchestra.run.smithi099.stderr: remote markers: [, , , , , 00000000001.93.6, , , 00000000001.99.6, 00000000002.123.6, 00000000002.114.6] 2023-05-31T21:05:59.169 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T21:05:59.164+0000 7f5508628780 1 waiting for incremental sync to catch up: 2023-05-31T21:05:59.169 INFO:teuthology.orchestra.run.smithi099.stderr: local status: [, , , , , 00000000001.93.6, , , 00000000001.99.6, 00000000002.123.6, 00000000001.112.6] 2023-05-31T21:05:59.169 INFO:teuthology.orchestra.run.smithi099.stderr: remote markers: [, , , , , 00000000001.93.6, , , 00000000001.99.6, 00000000002.123.6, 00000000002.114.6] 2023-05-31T21:06:04.170 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T21:06:04.165+0000 7f5508628780 1 waiting for incremental sync to catch up: 2023-05-31T21:06:04.170 INFO:teuthology.orchestra.run.smithi099.stderr: local status: [, , , , , 00000000001.93.6, , , 00000000001.99.6, 00000000002.123.6, 00000000001.112.6] 2023-05-31T21:06:04.170 INFO:teuthology.orchestra.run.smithi099.stderr: remote markers: [, , , , , 00000000001.93.6, , , 00000000001.99.6, 00000000002.123.6, 00000000002.114.6] 2023-05-31T21:06:09.171 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T21:06:09.167+0000 7f5508628780 1 waiting for incremental sync to catch up: 2023-05-31T21:06:09.172 INFO:teuthology.orchestra.run.smithi099.stderr: local status: [, , , , , 00000000001.93.6, , , 00000000001.99.6, 00000000002.123.6, 00000000001.112.6] 2023-05-31T21:06:09.172 INFO:teuthology.orchestra.run.smithi099.stderr: remote markers: [, , , , , 00000000001.93.6, , , 00000000001.99.6, 00000000002.123.6, 00000000002.114.6] 2023-05-31T21:06:14.173 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T21:06:14.168+0000 7f5508628780 1 waiting for incremental sync to catch up: 2023-05-31T21:06:14.173 INFO:teuthology.orchestra.run.smithi099.stderr: local status: [, , , , , 00000000001.93.6, , , 00000000001.99.6, 00000000002.123.6, 00000000001.112.6] 2023-05-31T21:06:14.173 INFO:teuthology.orchestra.run.smithi099.stderr: remote markers: [, , , , , 00000000001.93.6, , , 00000000001.99.6, 00000000002.123.6, 00000000002.114.6] 2023-05-31T21:06:19.175 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T21:06:19.170+0000 7f5508628780 1 waiting for incremental sync to catch up: 2023-05-31T21:06:19.175 INFO:teuthology.orchestra.run.smithi099.stderr: local status: [, , , , , 00000000001.93.6, , , 00000000001.99.6, 00000000002.123.6, 00000000001.112.6] 2023-05-31T21:06:19.176 INFO:teuthology.orchestra.run.smithi099.stderr: remote markers: [, , , , , 00000000001.93.6, , , 00000000001.99.6, 00000000002.123.6, 00000000002.114.6] 2023-05-31T21:06:24.176 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T21:06:24.173+0000 7f5508628780 1 waiting for incremental sync to catch up: 2023-05-31T21:06:24.177 INFO:teuthology.orchestra.run.smithi099.stderr: local status: [, , , , , 00000000001.93.6, , , 00000000001.99.6, 00000000002.123.6, 00000000001.112.6] 2023-05-31T21:06:24.177 INFO:teuthology.orchestra.run.smithi099.stderr: remote markers: [, , , , , 00000000001.93.6, , , 00000000001.99.6, 00000000002.123.6, 00000000002.114.6] 2023-05-31T21:06:29.178 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T21:06:29.174+0000 7f5508628780 1 waiting for incremental sync to catch up: 2023-05-31T21:06:29.178 INFO:teuthology.orchestra.run.smithi099.stderr: local status: [, , , , , 00000000001.93.6, , , 00000000001.99.6, 00000000002.123.6, 00000000001.112.6] 2023-05-31T21:06:29.178 INFO:teuthology.orchestra.run.smithi099.stderr: remote markers: [, , , , , 00000000001.93.6, , , 00000000001.99.6, 00000000002.123.6, 00000000002.114.6] 2023-05-31T21:06:34.179 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T21:06:34.175+0000 7f5508628780 1 bucket sync caught up with source: 2023-05-31T21:06:34.179 INFO:teuthology.orchestra.run.smithi099.stderr: local status: [, , , , , 00000000001.93.6, , , 00000000001.99.6, 00000000002.123.6, 00000000002.114.6] 2023-05-31T21:06:34.179 INFO:teuthology.orchestra.run.smithi099.stderr: remote markers: [, , , , , 00000000001.93.6, , , 00000000001.99.6, 00000000002.123.6, 00000000002.114.6] 2023-05-31T21:06:34.180 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T21:06:34.175+0000 7f5508628780 0 bucket checkpoint complete 2023-05-31T21:06:34.198 DEBUG:rgw_multi.tests:bucket checkpoint: source=a2 target=a1 bucket=pklxkd-62 2023-05-31T21:06:34.199 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'pklxkd-62', '--source-zone', 'a2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-05-31T21:06:34.200 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'pklxkd-62', '--source-zone', 'a2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-05-31T21:06:34.200 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket pklxkd-62 --source-zone a2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2023-05-31T21:06:34.262 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2023-05-31T21:06:34.262 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2023-05-31T21:06:34.314 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T21:06:34.310+0000 7f750c66e780 1 waiting for incremental sync to catch up: 2023-05-31T21:06:34.314 INFO:teuthology.orchestra.run.smithi099.stderr: local status: [, , , , , 00000000001.804.6, , , 00000000001.95.6, 00000000002.1362.6, 00000000001.1112.6] 2023-05-31T21:06:34.314 INFO:teuthology.orchestra.run.smithi099.stderr: remote markers: [, , , , , 00000000001.804.6, , , 00000000001.95.6, 00000000002.1362.6, 00000000002.1114.6] 2023-05-31T21:06:39.315 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T21:06:39.312+0000 7f750c66e780 1 bucket sync caught up with source: 2023-05-31T21:06:39.316 INFO:teuthology.orchestra.run.smithi099.stderr: local status: [, , , , , 00000000001.804.6, , , 00000000001.95.6, 00000000002.1362.6, 00000000002.1114.6] 2023-05-31T21:06:39.316 INFO:teuthology.orchestra.run.smithi099.stderr: remote markers: [, , , , , 00000000001.804.6, , , 00000000001.95.6, 00000000002.1362.6, 00000000002.1114.6] 2023-05-31T21:06:39.316 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T21:06:39.312+0000 7f750c66e780 0 bucket checkpoint complete 2023-05-31T21:06:39.337 INFO:tasks.rgw_multi.zone_rados:comparing bucket=pklxkd-62 zones={a2, a1} 2023-05-31T21:06:39.337 DEBUG:boto:path=/pklxkd-62/ 2023-05-31T21:06:39.338 DEBUG:boto:auth_path=/pklxkd-62/ 2023-05-31T21:06:39.338 DEBUG:boto:Method: HEAD 2023-05-31T21:06:39.339 DEBUG:boto:Path: /pklxkd-62/ 2023-05-31T21:06:39.339 DEBUG:boto:Data: 2023-05-31T21:06:39.339 DEBUG:boto:Headers: {} 2023-05-31T21:06:39.340 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T21:06:39.340 DEBUG:boto:Port: 8001 2023-05-31T21:06:39.340 DEBUG:boto:Params: {} 2023-05-31T21:06:39.341 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8001} 2023-05-31T21:06:39.341 DEBUG:boto:Token: None 2023-05-31T21:06:39.342 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 21:06:39 GMT /pklxkd-62/ 2023-05-31T21:06:39.342 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:PiCFDDXPuDSo2oK3myiAITFMO0c= 2023-05-31T21:06:39.343 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 21:06:39 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:PiCFDDXPuDSo2oK3myiAITFMO0c=', 'Content-Length': '0'} 2023-05-31T21:06:39.348 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '6'), ('X-RGW-Bytes-Used', '18'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000074af1d02d678abcc-006477b6df-1fb6-a2'), ('Content-Length', '0'), ('Date', 'Wed, 31 May 2023 21:06:39 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:06:39.348 DEBUG:boto:path=/pklxkd-62/ 2023-05-31T21:06:39.349 DEBUG:boto:auth_path=/pklxkd-62/ 2023-05-31T21:06:39.349 DEBUG:boto:Method: HEAD 2023-05-31T21:06:39.349 DEBUG:boto:Path: /pklxkd-62/ 2023-05-31T21:06:39.350 DEBUG:boto:Data: 2023-05-31T21:06:39.350 DEBUG:boto:Headers: {} 2023-05-31T21:06:39.350 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T21:06:39.351 DEBUG:boto:Port: 8000 2023-05-31T21:06:39.351 DEBUG:boto:Params: {} 2023-05-31T21:06:39.352 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2023-05-31T21:06:39.352 DEBUG:boto:Token: None 2023-05-31T21:06:39.352 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 21:06:39 GMT /pklxkd-62/ 2023-05-31T21:06:39.353 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:PiCFDDXPuDSo2oK3myiAITFMO0c= 2023-05-31T21:06:39.353 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 21:06:39 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:PiCFDDXPuDSo2oK3myiAITFMO0c=', 'Content-Length': '0'} 2023-05-31T21:06:39.356 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '6'), ('X-RGW-Bytes-Used', '18'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000618aeafdc8695212-006477b6df-1630-a1'), ('Content-Length', '0'), ('Date', 'Wed, 31 May 2023 21:06:39 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:06:39.356 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2023-05-31T21:06:39.357 DEBUG:boto:path=/pklxkd-62/ 2023-05-31T21:06:39.357 DEBUG:boto:auth_path=/pklxkd-62/ 2023-05-31T21:06:39.358 DEBUG:boto:path=/pklxkd-62/?versions&max-keys=999 2023-05-31T21:06:39.358 DEBUG:boto:auth_path=/pklxkd-62/?versions&max-keys=999 2023-05-31T21:06:39.358 DEBUG:boto:Method: GET 2023-05-31T21:06:39.359 DEBUG:boto:Path: /pklxkd-62/?versions&max-keys=999 2023-05-31T21:06:39.359 DEBUG:boto:Data: 2023-05-31T21:06:39.359 DEBUG:boto:Headers: {} 2023-05-31T21:06:39.360 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T21:06:39.360 DEBUG:boto:Port: 8001 2023-05-31T21:06:39.360 DEBUG:boto:Params: {} 2023-05-31T21:06:39.361 DEBUG:boto:Token: None 2023-05-31T21:06:39.361 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 21:06:39 GMT /pklxkd-62/?versions 2023-05-31T21:06:39.362 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:hCiKlVCnSYX+7U78frLH0FHV5C8= 2023-05-31T21:06:39.362 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 21:06:39 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:hCiKlVCnSYX+7U78frLH0FHV5C8=', 'Content-Length': '0'} 2023-05-31T21:06:39.364 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c870226452067afd-006477b6df-1fb6-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 21:06:39 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:06:39.365 DEBUG:boto:b'pklxkd-62999falsedummynulltrue2023-05-31T21:03:30.980Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj11nulltrue2023-05-31T21:04:02.634Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj22nulltrue2023-05-31T21:04:02.653Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj33nulltrue2023-05-31T21:04:02.707Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj44nulltrue2023-05-31T21:04:02.730Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj55nulltrue2023-05-31T21:04:02.751Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T21:06:39.366 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2023-05-31T21:06:39.366 DEBUG:tasks.rgw_multi.zone_rados:o=obj11 2023-05-31T21:06:39.366 DEBUG:tasks.rgw_multi.zone_rados:o=obj22 2023-05-31T21:06:39.367 DEBUG:tasks.rgw_multi.zone_rados:o=obj33 2023-05-31T21:06:39.367 DEBUG:tasks.rgw_multi.zone_rados:o=obj44 2023-05-31T21:06:39.367 DEBUG:tasks.rgw_multi.zone_rados:o=obj55 2023-05-31T21:06:39.368 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2023-05-31T21:06:39.368 DEBUG:boto:path=/pklxkd-62/ 2023-05-31T21:06:39.368 DEBUG:boto:auth_path=/pklxkd-62/ 2023-05-31T21:06:39.369 DEBUG:boto:path=/pklxkd-62/?versions&max-keys=999 2023-05-31T21:06:39.369 DEBUG:boto:auth_path=/pklxkd-62/?versions&max-keys=999 2023-05-31T21:06:39.369 DEBUG:boto:Method: GET 2023-05-31T21:06:39.370 DEBUG:boto:Path: /pklxkd-62/?versions&max-keys=999 2023-05-31T21:06:39.370 DEBUG:boto:Data: 2023-05-31T21:06:39.370 DEBUG:boto:Headers: {} 2023-05-31T21:06:39.371 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T21:06:39.371 DEBUG:boto:Port: 8000 2023-05-31T21:06:39.371 DEBUG:boto:Params: {} 2023-05-31T21:06:39.372 DEBUG:boto:Token: None 2023-05-31T21:06:39.372 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 21:06:39 GMT /pklxkd-62/?versions 2023-05-31T21:06:39.373 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:hCiKlVCnSYX+7U78frLH0FHV5C8= 2023-05-31T21:06:39.373 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 21:06:39 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:hCiKlVCnSYX+7U78frLH0FHV5C8=', 'Content-Length': '0'} 2023-05-31T21:06:39.375 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000001811c4cc8cd7e2f3-006477b6df-1630-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 21:06:39 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:06:39.376 DEBUG:boto:b'pklxkd-62999falsedummynulltrue2023-05-31T21:03:30.980Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj11nulltrue2023-05-31T21:04:02.634Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj22nulltrue2023-05-31T21:04:02.653Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj33nulltrue2023-05-31T21:04:02.707Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj44nulltrue2023-05-31T21:04:02.730Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj55nulltrue2023-05-31T21:04:02.751Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T21:06:39.377 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2023-05-31T21:06:39.377 DEBUG:tasks.rgw_multi.zone_rados:o=obj11 2023-05-31T21:06:39.377 DEBUG:tasks.rgw_multi.zone_rados:o=obj22 2023-05-31T21:06:39.378 DEBUG:tasks.rgw_multi.zone_rados:o=obj33 2023-05-31T21:06:39.378 DEBUG:tasks.rgw_multi.zone_rados:o=obj44 2023-05-31T21:06:39.378 DEBUG:tasks.rgw_multi.zone_rados:o=obj55 2023-05-31T21:06:39.379 DEBUG:boto:path=/pklxkd-62/ 2023-05-31T21:06:39.379 DEBUG:boto:auth_path=/pklxkd-62/ 2023-05-31T21:06:39.379 DEBUG:boto:path=/pklxkd-62/?versions&max-keys=999 2023-05-31T21:06:39.380 DEBUG:boto:auth_path=/pklxkd-62/?versions&max-keys=999 2023-05-31T21:06:39.380 DEBUG:boto:Method: GET 2023-05-31T21:06:39.380 DEBUG:boto:Path: /pklxkd-62/?versions&max-keys=999 2023-05-31T21:06:39.381 DEBUG:boto:Data: 2023-05-31T21:06:39.381 DEBUG:boto:Headers: {} 2023-05-31T21:06:39.381 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T21:06:39.382 DEBUG:boto:Port: 8001 2023-05-31T21:06:39.382 DEBUG:boto:Params: {} 2023-05-31T21:06:39.382 DEBUG:boto:Token: None 2023-05-31T21:06:39.383 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 21:06:39 GMT /pklxkd-62/?versions 2023-05-31T21:06:39.383 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:hCiKlVCnSYX+7U78frLH0FHV5C8= 2023-05-31T21:06:39.384 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 21:06:39 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:hCiKlVCnSYX+7U78frLH0FHV5C8=', 'Content-Length': '0'} 2023-05-31T21:06:39.386 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d2d9e2b57201d1bb-006477b6df-1fb6-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 21:06:39 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:06:39.386 DEBUG:boto:b'pklxkd-62999falsedummynulltrue2023-05-31T21:03:30.980Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj11nulltrue2023-05-31T21:04:02.634Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj22nulltrue2023-05-31T21:04:02.653Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj33nulltrue2023-05-31T21:04:02.707Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj44nulltrue2023-05-31T21:04:02.730Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj55nulltrue2023-05-31T21:04:02.751Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T21:06:39.387 DEBUG:boto:path=/pklxkd-62/ 2023-05-31T21:06:39.388 DEBUG:boto:auth_path=/pklxkd-62/ 2023-05-31T21:06:39.388 DEBUG:boto:path=/pklxkd-62/?versions&max-keys=999 2023-05-31T21:06:39.388 DEBUG:boto:auth_path=/pklxkd-62/?versions&max-keys=999 2023-05-31T21:06:39.389 DEBUG:boto:Method: GET 2023-05-31T21:06:39.389 DEBUG:boto:Path: /pklxkd-62/?versions&max-keys=999 2023-05-31T21:06:39.389 DEBUG:boto:Data: 2023-05-31T21:06:39.390 DEBUG:boto:Headers: {} 2023-05-31T21:06:39.390 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T21:06:39.390 DEBUG:boto:Port: 8000 2023-05-31T21:06:39.391 DEBUG:boto:Params: {} 2023-05-31T21:06:39.391 DEBUG:boto:Token: None 2023-05-31T21:06:39.392 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 21:06:39 GMT /pklxkd-62/?versions 2023-05-31T21:06:39.392 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:hCiKlVCnSYX+7U78frLH0FHV5C8= 2023-05-31T21:06:39.392 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 21:06:39 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:hCiKlVCnSYX+7U78frLH0FHV5C8=', 'Content-Length': '0'} 2023-05-31T21:06:39.395 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000535e63c46657c069-006477b6df-1630-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 21:06:39 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:06:39.395 DEBUG:boto:b'pklxkd-62999falsedummynulltrue2023-05-31T21:03:30.980Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj11nulltrue2023-05-31T21:04:02.634Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj22nulltrue2023-05-31T21:04:02.653Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj33nulltrue2023-05-31T21:04:02.707Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj44nulltrue2023-05-31T21:04:02.730Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj55nulltrue2023-05-31T21:04:02.751Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T21:06:39.396 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2023-05-31T21:06:39.397 DEBUG:boto:path=/pklxkd-62/dummy 2023-05-31T21:06:39.397 DEBUG:boto:auth_path=/pklxkd-62/dummy 2023-05-31T21:06:39.397 DEBUG:boto:path=/pklxkd-62/dummy?versionId=null 2023-05-31T21:06:39.398 DEBUG:boto:auth_path=/pklxkd-62/dummy?versionId=null 2023-05-31T21:06:39.398 DEBUG:boto:Method: GET 2023-05-31T21:06:39.398 DEBUG:boto:Path: /pklxkd-62/dummy?versionId=null 2023-05-31T21:06:39.399 DEBUG:boto:Data: 2023-05-31T21:06:39.399 DEBUG:boto:Headers: {} 2023-05-31T21:06:39.399 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T21:06:39.400 DEBUG:boto:Port: 8001 2023-05-31T21:06:39.400 DEBUG:boto:Params: {} 2023-05-31T21:06:39.401 DEBUG:boto:Token: None 2023-05-31T21:06:39.401 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 21:06:39 GMT /pklxkd-62/dummy?versionId=null 2023-05-31T21:06:39.401 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:dANXOSbUVxwzKEEhNIsohVCqF+A= 2023-05-31T21:06:39.402 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 21:06:39 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:dANXOSbUVxwzKEEhNIsohVCqF+A=', 'Content-Length': '0'} 2023-05-31T21:06:39.404 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 21:03:30 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f217ab1417667a3d-006477b6df-1fb6-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 21:06:39 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:06:39.405 DEBUG:boto:path=/pklxkd-62/dummy 2023-05-31T21:06:39.405 DEBUG:boto:auth_path=/pklxkd-62/dummy 2023-05-31T21:06:39.405 DEBUG:boto:path=/pklxkd-62/dummy?versionId=null 2023-05-31T21:06:39.406 DEBUG:boto:auth_path=/pklxkd-62/dummy?versionId=null 2023-05-31T21:06:39.406 DEBUG:boto:Method: GET 2023-05-31T21:06:39.406 DEBUG:boto:Path: /pklxkd-62/dummy?versionId=null 2023-05-31T21:06:39.407 DEBUG:boto:Data: 2023-05-31T21:06:39.407 DEBUG:boto:Headers: {} 2023-05-31T21:06:39.408 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T21:06:39.408 DEBUG:boto:Port: 8000 2023-05-31T21:06:39.408 DEBUG:boto:Params: {} 2023-05-31T21:06:39.409 DEBUG:boto:Token: None 2023-05-31T21:06:39.409 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 21:06:39 GMT /pklxkd-62/dummy?versionId=null 2023-05-31T21:06:39.409 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:dANXOSbUVxwzKEEhNIsohVCqF+A= 2023-05-31T21:06:39.410 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 21:06:39 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:dANXOSbUVxwzKEEhNIsohVCqF+A=', 'Content-Length': '0'} 2023-05-31T21:06:39.412 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 21:03:30 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000138b2fa45f9d48bf-006477b6df-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 21:06:39 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:06:39.412 DEBUG:boto:path=/pklxkd-62/dummy 2023-05-31T21:06:39.413 DEBUG:boto:auth_path=/pklxkd-62/dummy 2023-05-31T21:06:39.413 DEBUG:boto:path=/pklxkd-62/dummy?versionId=null 2023-05-31T21:06:39.413 DEBUG:boto:auth_path=/pklxkd-62/dummy?versionId=null 2023-05-31T21:06:39.414 DEBUG:boto:Method: HEAD 2023-05-31T21:06:39.414 DEBUG:boto:Path: /pklxkd-62/dummy?versionId=null 2023-05-31T21:06:39.415 DEBUG:boto:Data: 2023-05-31T21:06:39.415 DEBUG:boto:Headers: {} 2023-05-31T21:06:39.415 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T21:06:39.415 DEBUG:boto:Port: 8001 2023-05-31T21:06:39.416 DEBUG:boto:Params: {} 2023-05-31T21:06:39.416 DEBUG:boto:Token: None 2023-05-31T21:06:39.417 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 21:06:39 GMT /pklxkd-62/dummy?versionId=null 2023-05-31T21:06:39.417 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:O+1gEGdHCnWwqrRzPiS/tvg/gp0= 2023-05-31T21:06:39.417 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 21:06:39 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:O+1gEGdHCnWwqrRzPiS/tvg/gp0=', 'Content-Length': '0'} 2023-05-31T21:06:39.419 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 21:03:30 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b33f0b7e931dd31d-006477b6df-1fb6-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 21:06:39 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:06:39.420 DEBUG:boto:path=/pklxkd-62/dummy 2023-05-31T21:06:39.420 DEBUG:boto:auth_path=/pklxkd-62/dummy 2023-05-31T21:06:39.420 DEBUG:boto:path=/pklxkd-62/dummy?versionId=null 2023-05-31T21:06:39.421 DEBUG:boto:auth_path=/pklxkd-62/dummy?versionId=null 2023-05-31T21:06:39.421 DEBUG:boto:Method: HEAD 2023-05-31T21:06:39.421 DEBUG:boto:Path: /pklxkd-62/dummy?versionId=null 2023-05-31T21:06:39.422 DEBUG:boto:Data: 2023-05-31T21:06:39.422 DEBUG:boto:Headers: {} 2023-05-31T21:06:39.422 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T21:06:39.423 DEBUG:boto:Port: 8000 2023-05-31T21:06:39.423 DEBUG:boto:Params: {} 2023-05-31T21:06:39.423 DEBUG:boto:Token: None 2023-05-31T21:06:39.424 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 21:06:39 GMT /pklxkd-62/dummy?versionId=null 2023-05-31T21:06:39.424 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:O+1gEGdHCnWwqrRzPiS/tvg/gp0= 2023-05-31T21:06:39.425 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 21:06:39 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:O+1gEGdHCnWwqrRzPiS/tvg/gp0=', 'Content-Length': '0'} 2023-05-31T21:06:39.426 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 21:03:30 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000085b933f7ed234265-006477b6df-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 21:06:39 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:06:39.427 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2023-05-31T21:06:39.427 DEBUG:boto:path=/pklxkd-62/dummy 2023-05-31T21:06:39.428 DEBUG:boto:auth_path=/pklxkd-62/dummy 2023-05-31T21:06:39.428 DEBUG:boto:path=/pklxkd-62/dummy?versionId=null 2023-05-31T21:06:39.428 DEBUG:boto:auth_path=/pklxkd-62/dummy?versionId=null 2023-05-31T21:06:39.429 DEBUG:boto:Method: GET 2023-05-31T21:06:39.429 DEBUG:boto:Path: /pklxkd-62/dummy?versionId=null 2023-05-31T21:06:39.429 DEBUG:boto:Data: 2023-05-31T21:06:39.430 DEBUG:boto:Headers: {} 2023-05-31T21:06:39.430 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T21:06:39.431 DEBUG:boto:Port: 8001 2023-05-31T21:06:39.431 DEBUG:boto:Params: {} 2023-05-31T21:06:39.431 DEBUG:boto:Token: None 2023-05-31T21:06:39.432 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 21:06:39 GMT /pklxkd-62/dummy?versionId=null 2023-05-31T21:06:39.432 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:dANXOSbUVxwzKEEhNIsohVCqF+A= 2023-05-31T21:06:39.433 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 21:06:39 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:dANXOSbUVxwzKEEhNIsohVCqF+A=', 'Content-Length': '0'} 2023-05-31T21:06:39.434 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 21:03:30 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000193389bba8615fd8-006477b6df-1fb6-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 21:06:39 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:06:39.435 DEBUG:boto:path=/pklxkd-62/dummy 2023-05-31T21:06:39.436 DEBUG:boto:auth_path=/pklxkd-62/dummy 2023-05-31T21:06:39.436 DEBUG:boto:path=/pklxkd-62/dummy?versionId=null 2023-05-31T21:06:39.436 DEBUG:boto:auth_path=/pklxkd-62/dummy?versionId=null 2023-05-31T21:06:39.437 DEBUG:boto:Method: GET 2023-05-31T21:06:39.437 DEBUG:boto:Path: /pklxkd-62/dummy?versionId=null 2023-05-31T21:06:39.437 DEBUG:boto:Data: 2023-05-31T21:06:39.438 DEBUG:boto:Headers: {} 2023-05-31T21:06:39.438 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T21:06:39.438 DEBUG:boto:Port: 8000 2023-05-31T21:06:39.439 DEBUG:boto:Params: {} 2023-05-31T21:06:39.439 DEBUG:boto:Token: None 2023-05-31T21:06:39.440 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 21:06:39 GMT /pklxkd-62/dummy?versionId=null 2023-05-31T21:06:39.440 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:dANXOSbUVxwzKEEhNIsohVCqF+A= 2023-05-31T21:06:39.440 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 21:06:39 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:dANXOSbUVxwzKEEhNIsohVCqF+A=', 'Content-Length': '0'} 2023-05-31T21:06:39.442 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 21:03:30 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006652153e949ad0c3-006477b6df-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 21:06:39 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:06:39.443 DEBUG:boto:path=/pklxkd-62/ 2023-05-31T21:06:39.443 DEBUG:boto:auth_path=/pklxkd-62/ 2023-05-31T21:06:39.444 DEBUG:boto:path=/pklxkd-62/?prefix=dummy 2023-05-31T21:06:39.444 DEBUG:boto:auth_path=/pklxkd-62/?prefix=dummy 2023-05-31T21:06:39.444 DEBUG:boto:Method: GET 2023-05-31T21:06:39.445 DEBUG:boto:Path: /pklxkd-62/?prefix=dummy 2023-05-31T21:06:39.445 DEBUG:boto:Data: 2023-05-31T21:06:39.445 DEBUG:boto:Headers: {} 2023-05-31T21:06:39.446 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T21:06:39.446 DEBUG:boto:Port: 8001 2023-05-31T21:06:39.447 DEBUG:boto:Params: {} 2023-05-31T21:06:39.447 DEBUG:boto:Token: None 2023-05-31T21:06:39.447 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 21:06:39 GMT /pklxkd-62/ 2023-05-31T21:06:39.448 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:nATqi8rxF7oCpSuUR6RziAwvAdo= 2023-05-31T21:06:39.448 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 21:06:39 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:nATqi8rxF7oCpSuUR6RziAwvAdo=', 'Content-Length': '0'} 2023-05-31T21:06:39.450 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000075fe5604c18b2db8-006477b6df-1fb6-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 21:06:39 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:06:39.451 DEBUG:boto:b'pklxkd-62dummy1000falsedummy2023-05-31T21:03:30.980Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T21:06:39.452 DEBUG:boto:path=/pklxkd-62/ 2023-05-31T21:06:39.452 DEBUG:boto:auth_path=/pklxkd-62/ 2023-05-31T21:06:39.452 DEBUG:boto:path=/pklxkd-62/?prefix=dummy 2023-05-31T21:06:39.453 DEBUG:boto:auth_path=/pklxkd-62/?prefix=dummy 2023-05-31T21:06:39.453 DEBUG:boto:Method: GET 2023-05-31T21:06:39.453 DEBUG:boto:Path: /pklxkd-62/?prefix=dummy 2023-05-31T21:06:39.454 DEBUG:boto:Data: 2023-05-31T21:06:39.454 DEBUG:boto:Headers: {} 2023-05-31T21:06:39.454 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T21:06:39.455 DEBUG:boto:Port: 8000 2023-05-31T21:06:39.455 DEBUG:boto:Params: {} 2023-05-31T21:06:39.456 DEBUG:boto:Token: None 2023-05-31T21:06:39.456 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 21:06:39 GMT /pklxkd-62/ 2023-05-31T21:06:39.456 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:nATqi8rxF7oCpSuUR6RziAwvAdo= 2023-05-31T21:06:39.457 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 21:06:39 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:nATqi8rxF7oCpSuUR6RziAwvAdo=', 'Content-Length': '0'} 2023-05-31T21:06:39.459 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e7d2cbcc613c4e8a-006477b6df-1630-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 21:06:39 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:06:39.460 DEBUG:boto:b'pklxkd-62dummy1000falsedummy2023-05-31T21:03:30.980Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T21:06:39.460 DEBUG:boto:path=/pklxkd-62/dummy 2023-05-31T21:06:39.461 DEBUG:boto:auth_path=/pklxkd-62/dummy 2023-05-31T21:06:39.461 DEBUG:boto:Method: HEAD 2023-05-31T21:06:39.462 DEBUG:boto:Path: /pklxkd-62/dummy 2023-05-31T21:06:39.462 DEBUG:boto:Data: 2023-05-31T21:06:39.463 DEBUG:boto:Headers: {} 2023-05-31T21:06:39.463 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T21:06:39.463 DEBUG:boto:Port: 8001 2023-05-31T21:06:39.464 DEBUG:boto:Params: {} 2023-05-31T21:06:39.464 DEBUG:boto:Token: None 2023-05-31T21:06:39.464 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 21:06:39 GMT /pklxkd-62/dummy 2023-05-31T21:06:39.465 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:ZwfupeEtmVDR/BmoEffCB359jQA= 2023-05-31T21:06:39.465 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 21:06:39 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:ZwfupeEtmVDR/BmoEffCB359jQA=', 'Content-Length': '0'} 2023-05-31T21:06:39.467 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 21:03:30 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003863202c061e8dda-006477b6df-1fb6-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 21:06:39 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:06:39.468 DEBUG:boto:path=/pklxkd-62/dummy 2023-05-31T21:06:39.468 DEBUG:boto:auth_path=/pklxkd-62/dummy 2023-05-31T21:06:39.469 DEBUG:boto:Method: HEAD 2023-05-31T21:06:39.469 DEBUG:boto:Path: /pklxkd-62/dummy 2023-05-31T21:06:39.469 DEBUG:boto:Data: 2023-05-31T21:06:39.470 DEBUG:boto:Headers: {} 2023-05-31T21:06:39.470 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T21:06:39.470 DEBUG:boto:Port: 8000 2023-05-31T21:06:39.471 DEBUG:boto:Params: {} 2023-05-31T21:06:39.471 DEBUG:boto:Token: None 2023-05-31T21:06:39.471 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 21:06:39 GMT /pklxkd-62/dummy 2023-05-31T21:06:39.472 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:ZwfupeEtmVDR/BmoEffCB359jQA= 2023-05-31T21:06:39.472 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 21:06:39 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:ZwfupeEtmVDR/BmoEffCB359jQA=', 'Content-Length': '0'} 2023-05-31T21:06:39.474 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 21:03:30 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c878a0f0ad3cf523-006477b6df-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 21:06:39 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:06:39.475 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2023-05-31T21:06:39.475 DEBUG:boto:path=/pklxkd-62/dummy 2023-05-31T21:06:39.475 DEBUG:boto:auth_path=/pklxkd-62/dummy 2023-05-31T21:06:39.476 DEBUG:boto:Method: GET 2023-05-31T21:06:39.476 DEBUG:boto:Path: /pklxkd-62/dummy 2023-05-31T21:06:39.477 DEBUG:boto:Data: 2023-05-31T21:06:39.477 DEBUG:boto:Headers: {} 2023-05-31T21:06:39.477 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T21:06:39.478 DEBUG:boto:Port: 8001 2023-05-31T21:06:39.478 DEBUG:boto:Params: {} 2023-05-31T21:06:39.478 DEBUG:boto:Token: None 2023-05-31T21:06:39.479 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 21:06:39 GMT /pklxkd-62/dummy 2023-05-31T21:06:39.479 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:LmknSzQfoO7oI2Oo5pXpuhgeBUw= 2023-05-31T21:06:39.480 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 21:06:39 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:LmknSzQfoO7oI2Oo5pXpuhgeBUw=', 'Content-Length': '0'} 2023-05-31T21:06:39.482 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 21:03:30 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d73c0873efb13acb-006477b6df-1fb6-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 21:06:39 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:06:39.482 DEBUG:boto:path=/pklxkd-62/dummy 2023-05-31T21:06:39.483 DEBUG:boto:auth_path=/pklxkd-62/dummy 2023-05-31T21:06:39.483 DEBUG:boto:Method: GET 2023-05-31T21:06:39.483 DEBUG:boto:Path: /pklxkd-62/dummy 2023-05-31T21:06:39.484 DEBUG:boto:Data: 2023-05-31T21:06:39.484 DEBUG:boto:Headers: {} 2023-05-31T21:06:39.485 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T21:06:39.485 DEBUG:boto:Port: 8000 2023-05-31T21:06:39.486 DEBUG:boto:Params: {} 2023-05-31T21:06:39.486 DEBUG:boto:Token: None 2023-05-31T21:06:39.487 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 21:06:39 GMT /pklxkd-62/dummy 2023-05-31T21:06:39.487 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:LmknSzQfoO7oI2Oo5pXpuhgeBUw= 2023-05-31T21:06:39.488 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 21:06:39 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:LmknSzQfoO7oI2Oo5pXpuhgeBUw=', 'Content-Length': '0'} 2023-05-31T21:06:39.489 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 21:03:30 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005fddedf4c6ec90c0-006477b6df-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 21:06:39 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:06:39.490 DEBUG:boto:path=/pklxkd-62/ 2023-05-31T21:06:39.490 DEBUG:boto:auth_path=/pklxkd-62/ 2023-05-31T21:06:39.491 DEBUG:boto:path=/pklxkd-62/?prefix=dummy 2023-05-31T21:06:39.491 DEBUG:boto:auth_path=/pklxkd-62/?prefix=dummy 2023-05-31T21:06:39.492 DEBUG:boto:Method: GET 2023-05-31T21:06:39.492 DEBUG:boto:Path: /pklxkd-62/?prefix=dummy 2023-05-31T21:06:39.493 DEBUG:boto:Data: 2023-05-31T21:06:39.493 DEBUG:boto:Headers: {} 2023-05-31T21:06:39.493 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T21:06:39.494 DEBUG:boto:Port: 8001 2023-05-31T21:06:39.494 DEBUG:boto:Params: {} 2023-05-31T21:06:39.495 DEBUG:boto:Token: None 2023-05-31T21:06:39.495 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 21:06:39 GMT /pklxkd-62/ 2023-05-31T21:06:39.496 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:nATqi8rxF7oCpSuUR6RziAwvAdo= 2023-05-31T21:06:39.496 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 21:06:39 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:nATqi8rxF7oCpSuUR6RziAwvAdo=', 'Content-Length': '0'} 2023-05-31T21:06:39.498 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000004f4d92d61e914bcc-006477b6df-1fb6-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 21:06:39 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:06:39.499 DEBUG:boto:b'pklxkd-62dummy1000falsedummy2023-05-31T21:03:30.980Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T21:06:39.500 DEBUG:boto:path=/pklxkd-62/ 2023-05-31T21:06:39.500 DEBUG:boto:auth_path=/pklxkd-62/ 2023-05-31T21:06:39.500 DEBUG:boto:path=/pklxkd-62/?prefix=dummy 2023-05-31T21:06:39.501 DEBUG:boto:auth_path=/pklxkd-62/?prefix=dummy 2023-05-31T21:06:39.501 DEBUG:boto:Method: GET 2023-05-31T21:06:39.502 DEBUG:boto:Path: /pklxkd-62/?prefix=dummy 2023-05-31T21:06:39.502 DEBUG:boto:Data: 2023-05-31T21:06:39.502 DEBUG:boto:Headers: {} 2023-05-31T21:06:39.503 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T21:06:39.503 DEBUG:boto:Port: 8000 2023-05-31T21:06:39.503 DEBUG:boto:Params: {} 2023-05-31T21:06:39.504 DEBUG:boto:Token: None 2023-05-31T21:06:39.504 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 21:06:39 GMT /pklxkd-62/ 2023-05-31T21:06:39.505 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:nATqi8rxF7oCpSuUR6RziAwvAdo= 2023-05-31T21:06:39.505 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 21:06:39 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:nATqi8rxF7oCpSuUR6RziAwvAdo=', 'Content-Length': '0'} 2023-05-31T21:06:39.507 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f25cfacaa339e8c5-006477b6df-1630-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 21:06:39 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:06:39.508 DEBUG:boto:b'pklxkd-62dummy1000falsedummy2023-05-31T21:03:30.980Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T21:06:39.508 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj11 2023-05-31T21:06:39.509 DEBUG:boto:path=/pklxkd-62/obj11 2023-05-31T21:06:39.509 DEBUG:boto:auth_path=/pklxkd-62/obj11 2023-05-31T21:06:39.510 DEBUG:boto:path=/pklxkd-62/obj11?versionId=null 2023-05-31T21:06:39.510 DEBUG:boto:auth_path=/pklxkd-62/obj11?versionId=null 2023-05-31T21:06:39.510 DEBUG:boto:Method: GET 2023-05-31T21:06:39.511 DEBUG:boto:Path: /pklxkd-62/obj11?versionId=null 2023-05-31T21:06:39.511 DEBUG:boto:Data: 2023-05-31T21:06:39.511 DEBUG:boto:Headers: {} 2023-05-31T21:06:39.512 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T21:06:39.512 DEBUG:boto:Port: 8001 2023-05-31T21:06:39.512 DEBUG:boto:Params: {} 2023-05-31T21:06:39.513 DEBUG:boto:Token: None 2023-05-31T21:06:39.513 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 21:06:39 GMT /pklxkd-62/obj11?versionId=null 2023-05-31T21:06:39.514 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:sLh+cs9+Ex6Qh13j8j8n4dU3M9U= 2023-05-31T21:06:39.514 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 21:06:39 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:sLh+cs9+Ex6Qh13j8j8n4dU3M9U=', 'Content-Length': '0'} 2023-05-31T21:06:39.516 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 21:04:02 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e25478de26484476-006477b6df-1fb6-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 21:06:39 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:06:39.517 DEBUG:boto:path=/pklxkd-62/obj11 2023-05-31T21:06:39.517 DEBUG:boto:auth_path=/pklxkd-62/obj11 2023-05-31T21:06:39.517 DEBUG:boto:path=/pklxkd-62/obj11?versionId=null 2023-05-31T21:06:39.518 DEBUG:boto:auth_path=/pklxkd-62/obj11?versionId=null 2023-05-31T21:06:39.518 DEBUG:boto:Method: GET 2023-05-31T21:06:39.519 DEBUG:boto:Path: /pklxkd-62/obj11?versionId=null 2023-05-31T21:06:39.519 DEBUG:boto:Data: 2023-05-31T21:06:39.519 DEBUG:boto:Headers: {} 2023-05-31T21:06:39.520 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T21:06:39.520 DEBUG:boto:Port: 8000 2023-05-31T21:06:39.520 DEBUG:boto:Params: {} 2023-05-31T21:06:39.521 DEBUG:boto:Token: None 2023-05-31T21:06:39.521 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 21:06:39 GMT /pklxkd-62/obj11?versionId=null 2023-05-31T21:06:39.522 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:sLh+cs9+Ex6Qh13j8j8n4dU3M9U= 2023-05-31T21:06:39.522 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 21:06:39 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:sLh+cs9+Ex6Qh13j8j8n4dU3M9U=', 'Content-Length': '0'} 2023-05-31T21:06:39.524 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 21:04:02 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004894e98fae914306-006477b6df-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 21:06:39 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:06:39.524 DEBUG:boto:path=/pklxkd-62/obj11 2023-05-31T21:06:39.525 DEBUG:boto:auth_path=/pklxkd-62/obj11 2023-05-31T21:06:39.525 DEBUG:boto:path=/pklxkd-62/obj11?versionId=null 2023-05-31T21:06:39.525 DEBUG:boto:auth_path=/pklxkd-62/obj11?versionId=null 2023-05-31T21:06:39.526 DEBUG:boto:Method: HEAD 2023-05-31T21:06:39.526 DEBUG:boto:Path: /pklxkd-62/obj11?versionId=null 2023-05-31T21:06:39.526 DEBUG:boto:Data: 2023-05-31T21:06:39.527 DEBUG:boto:Headers: {} 2023-05-31T21:06:39.527 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T21:06:39.527 DEBUG:boto:Port: 8001 2023-05-31T21:06:39.528 DEBUG:boto:Params: {} 2023-05-31T21:06:39.528 DEBUG:boto:Token: None 2023-05-31T21:06:39.529 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 21:06:39 GMT /pklxkd-62/obj11?versionId=null 2023-05-31T21:06:39.529 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:SsyNw5pks+xdj/0aR8gWobIlpmM= 2023-05-31T21:06:39.529 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 21:06:39 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:SsyNw5pks+xdj/0aR8gWobIlpmM=', 'Content-Length': '0'} 2023-05-31T21:06:39.531 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 21:04:02 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ee4d7b2d08716894-006477b6df-1fb6-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 21:06:39 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:06:39.531 DEBUG:boto:path=/pklxkd-62/obj11 2023-05-31T21:06:39.532 DEBUG:boto:auth_path=/pklxkd-62/obj11 2023-05-31T21:06:39.532 DEBUG:boto:path=/pklxkd-62/obj11?versionId=null 2023-05-31T21:06:39.533 DEBUG:boto:auth_path=/pklxkd-62/obj11?versionId=null 2023-05-31T21:06:39.533 DEBUG:boto:Method: HEAD 2023-05-31T21:06:39.533 DEBUG:boto:Path: /pklxkd-62/obj11?versionId=null 2023-05-31T21:06:39.534 DEBUG:boto:Data: 2023-05-31T21:06:39.534 DEBUG:boto:Headers: {} 2023-05-31T21:06:39.534 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T21:06:39.535 DEBUG:boto:Port: 8000 2023-05-31T21:06:39.535 DEBUG:boto:Params: {} 2023-05-31T21:06:39.535 DEBUG:boto:Token: None 2023-05-31T21:06:39.536 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 21:06:39 GMT /pklxkd-62/obj11?versionId=null 2023-05-31T21:06:39.536 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:SsyNw5pks+xdj/0aR8gWobIlpmM= 2023-05-31T21:06:39.536 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 21:06:39 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:SsyNw5pks+xdj/0aR8gWobIlpmM=', 'Content-Length': '0'} 2023-05-31T21:06:39.538 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 21:04:02 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002dd37cf2f813298b-006477b6df-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 21:06:39 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:06:39.539 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj11 2023-05-31T21:06:39.539 DEBUG:boto:path=/pklxkd-62/obj11 2023-05-31T21:06:39.540 DEBUG:boto:auth_path=/pklxkd-62/obj11 2023-05-31T21:06:39.540 DEBUG:boto:path=/pklxkd-62/obj11?versionId=null 2023-05-31T21:06:39.540 DEBUG:boto:auth_path=/pklxkd-62/obj11?versionId=null 2023-05-31T21:06:39.541 DEBUG:boto:Method: GET 2023-05-31T21:06:39.541 DEBUG:boto:Path: /pklxkd-62/obj11?versionId=null 2023-05-31T21:06:39.542 DEBUG:boto:Data: 2023-05-31T21:06:39.542 DEBUG:boto:Headers: {} 2023-05-31T21:06:39.542 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T21:06:39.543 DEBUG:boto:Port: 8001 2023-05-31T21:06:39.543 DEBUG:boto:Params: {} 2023-05-31T21:06:39.544 DEBUG:boto:Token: None 2023-05-31T21:06:39.544 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 21:06:39 GMT /pklxkd-62/obj11?versionId=null 2023-05-31T21:06:39.544 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:sLh+cs9+Ex6Qh13j8j8n4dU3M9U= 2023-05-31T21:06:39.545 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 21:06:39 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:sLh+cs9+Ex6Qh13j8j8n4dU3M9U=', 'Content-Length': '0'} 2023-05-31T21:06:39.547 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 21:04:02 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001ea53ffdeb83da81-006477b6df-1fb6-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 21:06:39 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:06:39.547 DEBUG:boto:path=/pklxkd-62/obj11 2023-05-31T21:06:39.548 DEBUG:boto:auth_path=/pklxkd-62/obj11 2023-05-31T21:06:39.548 DEBUG:boto:path=/pklxkd-62/obj11?versionId=null 2023-05-31T21:06:39.549 DEBUG:boto:auth_path=/pklxkd-62/obj11?versionId=null 2023-05-31T21:06:39.549 DEBUG:boto:Method: GET 2023-05-31T21:06:39.550 DEBUG:boto:Path: /pklxkd-62/obj11?versionId=null 2023-05-31T21:06:39.551 DEBUG:boto:Data: 2023-05-31T21:06:39.551 DEBUG:boto:Headers: {} 2023-05-31T21:06:39.552 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T21:06:39.553 DEBUG:boto:Port: 8000 2023-05-31T21:06:39.553 DEBUG:boto:Params: {} 2023-05-31T21:06:39.553 DEBUG:boto:Token: None 2023-05-31T21:06:39.554 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 21:06:39 GMT /pklxkd-62/obj11?versionId=null 2023-05-31T21:06:39.555 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:sLh+cs9+Ex6Qh13j8j8n4dU3M9U= 2023-05-31T21:06:39.555 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 21:06:39 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:sLh+cs9+Ex6Qh13j8j8n4dU3M9U=', 'Content-Length': '0'} 2023-05-31T21:06:39.557 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 21:04:02 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000462babf9a1a1b131-006477b6df-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 21:06:39 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:06:39.557 DEBUG:boto:path=/pklxkd-62/ 2023-05-31T21:06:39.558 DEBUG:boto:auth_path=/pklxkd-62/ 2023-05-31T21:06:39.559 DEBUG:boto:path=/pklxkd-62/?prefix=obj11 2023-05-31T21:06:39.559 DEBUG:boto:auth_path=/pklxkd-62/?prefix=obj11 2023-05-31T21:06:39.559 DEBUG:boto:Method: GET 2023-05-31T21:06:39.560 DEBUG:boto:Path: /pklxkd-62/?prefix=obj11 2023-05-31T21:06:39.560 DEBUG:boto:Data: 2023-05-31T21:06:39.560 DEBUG:boto:Headers: {} 2023-05-31T21:06:39.561 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T21:06:39.561 DEBUG:boto:Port: 8001 2023-05-31T21:06:39.562 DEBUG:boto:Params: {} 2023-05-31T21:06:39.562 DEBUG:boto:Token: None 2023-05-31T21:06:39.562 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 21:06:39 GMT /pklxkd-62/ 2023-05-31T21:06:39.563 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:nATqi8rxF7oCpSuUR6RziAwvAdo= 2023-05-31T21:06:39.563 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 21:06:39 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:nATqi8rxF7oCpSuUR6RziAwvAdo=', 'Content-Length': '0'} 2023-05-31T21:06:39.565 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000be95fb22d6e051cc-006477b6df-1fb6-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 21:06:39 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:06:39.566 DEBUG:boto:b'pklxkd-62obj111000falseobj112023-05-31T21:04:02.634Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T21:06:39.566 DEBUG:boto:path=/pklxkd-62/ 2023-05-31T21:06:39.567 DEBUG:boto:auth_path=/pklxkd-62/ 2023-05-31T21:06:39.567 DEBUG:boto:path=/pklxkd-62/?prefix=obj11 2023-05-31T21:06:39.567 DEBUG:boto:auth_path=/pklxkd-62/?prefix=obj11 2023-05-31T21:06:39.568 DEBUG:boto:Method: GET 2023-05-31T21:06:39.568 DEBUG:boto:Path: /pklxkd-62/?prefix=obj11 2023-05-31T21:06:39.569 DEBUG:boto:Data: 2023-05-31T21:06:39.569 DEBUG:boto:Headers: {} 2023-05-31T21:06:39.569 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T21:06:39.570 DEBUG:boto:Port: 8000 2023-05-31T21:06:39.570 DEBUG:boto:Params: {} 2023-05-31T21:06:39.570 DEBUG:boto:Token: None 2023-05-31T21:06:39.571 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 21:06:39 GMT /pklxkd-62/ 2023-05-31T21:06:39.571 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:nATqi8rxF7oCpSuUR6RziAwvAdo= 2023-05-31T21:06:39.572 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 21:06:39 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:nATqi8rxF7oCpSuUR6RziAwvAdo=', 'Content-Length': '0'} 2023-05-31T21:06:39.574 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000007efcad4c81ba081c-006477b6df-1630-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 21:06:39 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:06:39.574 DEBUG:boto:b'pklxkd-62obj111000falseobj112023-05-31T21:04:02.634Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T21:06:39.575 DEBUG:boto:path=/pklxkd-62/obj11 2023-05-31T21:06:39.575 DEBUG:boto:auth_path=/pklxkd-62/obj11 2023-05-31T21:06:39.576 DEBUG:boto:Method: HEAD 2023-05-31T21:06:39.576 DEBUG:boto:Path: /pklxkd-62/obj11 2023-05-31T21:06:39.576 DEBUG:boto:Data: 2023-05-31T21:06:39.577 DEBUG:boto:Headers: {} 2023-05-31T21:06:39.577 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T21:06:39.577 DEBUG:boto:Port: 8001 2023-05-31T21:06:39.578 DEBUG:boto:Params: {} 2023-05-31T21:06:39.578 DEBUG:boto:Token: None 2023-05-31T21:06:39.579 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 21:06:39 GMT /pklxkd-62/obj11 2023-05-31T21:06:39.579 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:/BD+/mWljXAf7sHjv52BTcqKTCA= 2023-05-31T21:06:39.579 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 21:06:39 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:/BD+/mWljXAf7sHjv52BTcqKTCA=', 'Content-Length': '0'} 2023-05-31T21:06:39.581 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 21:04:02 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000421afe54ebf40c53-006477b6df-1fb6-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 21:06:39 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:06:39.582 DEBUG:boto:path=/pklxkd-62/obj11 2023-05-31T21:06:39.582 DEBUG:boto:auth_path=/pklxkd-62/obj11 2023-05-31T21:06:39.582 DEBUG:boto:Method: HEAD 2023-05-31T21:06:39.583 DEBUG:boto:Path: /pklxkd-62/obj11 2023-05-31T21:06:39.583 DEBUG:boto:Data: 2023-05-31T21:06:39.583 DEBUG:boto:Headers: {} 2023-05-31T21:06:39.584 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T21:06:39.584 DEBUG:boto:Port: 8000 2023-05-31T21:06:39.584 DEBUG:boto:Params: {} 2023-05-31T21:06:39.585 DEBUG:boto:Token: None 2023-05-31T21:06:39.585 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 21:06:39 GMT /pklxkd-62/obj11 2023-05-31T21:06:39.586 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:/BD+/mWljXAf7sHjv52BTcqKTCA= 2023-05-31T21:06:39.586 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 21:06:39 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:/BD+/mWljXAf7sHjv52BTcqKTCA=', 'Content-Length': '0'} 2023-05-31T21:06:39.588 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 21:04:02 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d7530d9cbdfcb3bd-006477b6df-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 21:06:39 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:06:39.588 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj11 2023-05-31T21:06:39.589 DEBUG:boto:path=/pklxkd-62/obj11 2023-05-31T21:06:39.589 DEBUG:boto:auth_path=/pklxkd-62/obj11 2023-05-31T21:06:39.590 DEBUG:boto:Method: GET 2023-05-31T21:06:39.590 DEBUG:boto:Path: /pklxkd-62/obj11 2023-05-31T21:06:39.590 DEBUG:boto:Data: 2023-05-31T21:06:39.591 DEBUG:boto:Headers: {} 2023-05-31T21:06:39.591 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T21:06:39.591 DEBUG:boto:Port: 8001 2023-05-31T21:06:39.592 DEBUG:boto:Params: {} 2023-05-31T21:06:39.592 DEBUG:boto:Token: None 2023-05-31T21:06:39.593 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 21:06:39 GMT /pklxkd-62/obj11 2023-05-31T21:06:39.593 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:/rLg/yOrXQSETZDD5HoT+XJZHBs= 2023-05-31T21:06:39.593 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 21:06:39 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:/rLg/yOrXQSETZDD5HoT+XJZHBs=', 'Content-Length': '0'} 2023-05-31T21:06:39.595 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 21:04:02 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008c4850740e14f8f7-006477b6df-1fb6-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 21:06:39 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:06:39.636 DEBUG:boto:path=/pklxkd-62/obj11 2023-05-31T21:06:39.637 DEBUG:boto:auth_path=/pklxkd-62/obj11 2023-05-31T21:06:39.637 DEBUG:boto:Method: GET 2023-05-31T21:06:39.637 DEBUG:boto:Path: /pklxkd-62/obj11 2023-05-31T21:06:39.638 DEBUG:boto:Data: 2023-05-31T21:06:39.638 DEBUG:boto:Headers: {} 2023-05-31T21:06:39.639 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T21:06:39.639 DEBUG:boto:Port: 8000 2023-05-31T21:06:39.639 DEBUG:boto:Params: {} 2023-05-31T21:06:39.640 DEBUG:boto:Token: None 2023-05-31T21:06:39.640 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 21:06:39 GMT /pklxkd-62/obj11 2023-05-31T21:06:39.641 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:/rLg/yOrXQSETZDD5HoT+XJZHBs= 2023-05-31T21:06:39.641 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 21:06:39 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:/rLg/yOrXQSETZDD5HoT+XJZHBs=', 'Content-Length': '0'} 2023-05-31T21:06:39.643 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 21:04:02 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000468669342ab153f7-006477b6df-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 21:06:39 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:06:39.644 DEBUG:boto:path=/pklxkd-62/ 2023-05-31T21:06:39.644 DEBUG:boto:auth_path=/pklxkd-62/ 2023-05-31T21:06:39.645 DEBUG:boto:path=/pklxkd-62/?prefix=obj11 2023-05-31T21:06:39.645 DEBUG:boto:auth_path=/pklxkd-62/?prefix=obj11 2023-05-31T21:06:39.645 DEBUG:boto:Method: GET 2023-05-31T21:06:39.646 DEBUG:boto:Path: /pklxkd-62/?prefix=obj11 2023-05-31T21:06:39.646 DEBUG:boto:Data: 2023-05-31T21:06:39.646 DEBUG:boto:Headers: {} 2023-05-31T21:06:39.647 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T21:06:39.647 DEBUG:boto:Port: 8001 2023-05-31T21:06:39.647 DEBUG:boto:Params: {} 2023-05-31T21:06:39.648 DEBUG:boto:Token: None 2023-05-31T21:06:39.648 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 21:06:39 GMT /pklxkd-62/ 2023-05-31T21:06:39.649 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:nATqi8rxF7oCpSuUR6RziAwvAdo= 2023-05-31T21:06:39.649 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 21:06:39 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:nATqi8rxF7oCpSuUR6RziAwvAdo=', 'Content-Length': '0'} 2023-05-31T21:06:39.651 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ce07d6d7113782f8-006477b6df-1fb6-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 21:06:39 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:06:39.652 DEBUG:boto:b'pklxkd-62obj111000falseobj112023-05-31T21:04:02.634Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T21:06:39.652 DEBUG:boto:path=/pklxkd-62/ 2023-05-31T21:06:39.653 DEBUG:boto:auth_path=/pklxkd-62/ 2023-05-31T21:06:39.653 DEBUG:boto:path=/pklxkd-62/?prefix=obj11 2023-05-31T21:06:39.654 DEBUG:boto:auth_path=/pklxkd-62/?prefix=obj11 2023-05-31T21:06:39.654 DEBUG:boto:Method: GET 2023-05-31T21:06:39.654 DEBUG:boto:Path: /pklxkd-62/?prefix=obj11 2023-05-31T21:06:39.655 DEBUG:boto:Data: 2023-05-31T21:06:39.655 DEBUG:boto:Headers: {} 2023-05-31T21:06:39.655 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T21:06:39.656 DEBUG:boto:Port: 8000 2023-05-31T21:06:39.656 DEBUG:boto:Params: {} 2023-05-31T21:06:39.657 DEBUG:boto:Token: None 2023-05-31T21:06:39.657 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 21:06:39 GMT /pklxkd-62/ 2023-05-31T21:06:39.657 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:nATqi8rxF7oCpSuUR6RziAwvAdo= 2023-05-31T21:06:39.658 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 21:06:39 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:nATqi8rxF7oCpSuUR6RziAwvAdo=', 'Content-Length': '0'} 2023-05-31T21:06:39.660 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d50b1a5bc3b61d9c-006477b6df-1630-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 21:06:39 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:06:39.661 DEBUG:boto:b'pklxkd-62obj111000falseobj112023-05-31T21:04:02.634Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T21:06:39.661 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj22 2023-05-31T21:06:39.662 DEBUG:boto:path=/pklxkd-62/obj22 2023-05-31T21:06:39.662 DEBUG:boto:auth_path=/pklxkd-62/obj22 2023-05-31T21:06:39.662 DEBUG:boto:path=/pklxkd-62/obj22?versionId=null 2023-05-31T21:06:39.663 DEBUG:boto:auth_path=/pklxkd-62/obj22?versionId=null 2023-05-31T21:06:39.663 DEBUG:boto:Method: GET 2023-05-31T21:06:39.663 DEBUG:boto:Path: /pklxkd-62/obj22?versionId=null 2023-05-31T21:06:39.664 DEBUG:boto:Data: 2023-05-31T21:06:39.665 DEBUG:boto:Headers: {} 2023-05-31T21:06:39.665 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T21:06:39.665 DEBUG:boto:Port: 8001 2023-05-31T21:06:39.666 DEBUG:boto:Params: {} 2023-05-31T21:06:39.667 DEBUG:boto:Token: None 2023-05-31T21:06:39.667 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 21:06:39 GMT /pklxkd-62/obj22?versionId=null 2023-05-31T21:06:39.668 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:yiyTTXpve+uoHcwHEQabzehL7hw= 2023-05-31T21:06:39.669 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 21:06:39 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:yiyTTXpve+uoHcwHEQabzehL7hw=', 'Content-Length': '0'} 2023-05-31T21:06:39.671 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 21:04:02 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000054b681e8613ce432-006477b6df-1fb6-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 21:06:39 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:06:39.672 DEBUG:boto:path=/pklxkd-62/obj22 2023-05-31T21:06:39.672 DEBUG:boto:auth_path=/pklxkd-62/obj22 2023-05-31T21:06:39.672 DEBUG:boto:path=/pklxkd-62/obj22?versionId=null 2023-05-31T21:06:39.673 DEBUG:boto:auth_path=/pklxkd-62/obj22?versionId=null 2023-05-31T21:06:39.673 DEBUG:boto:Method: GET 2023-05-31T21:06:39.673 DEBUG:boto:Path: /pklxkd-62/obj22?versionId=null 2023-05-31T21:06:39.674 DEBUG:boto:Data: 2023-05-31T21:06:39.674 DEBUG:boto:Headers: {} 2023-05-31T21:06:39.675 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T21:06:39.675 DEBUG:boto:Port: 8000 2023-05-31T21:06:39.675 DEBUG:boto:Params: {} 2023-05-31T21:06:39.676 DEBUG:boto:Token: None 2023-05-31T21:06:39.676 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 21:06:39 GMT /pklxkd-62/obj22?versionId=null 2023-05-31T21:06:39.677 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:yiyTTXpve+uoHcwHEQabzehL7hw= 2023-05-31T21:06:39.677 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 21:06:39 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:yiyTTXpve+uoHcwHEQabzehL7hw=', 'Content-Length': '0'} 2023-05-31T21:06:39.679 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 21:04:02 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000da5b5e525fb0ebd0-006477b6df-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 21:06:39 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:06:39.680 DEBUG:boto:path=/pklxkd-62/obj22 2023-05-31T21:06:39.680 DEBUG:boto:auth_path=/pklxkd-62/obj22 2023-05-31T21:06:39.680 DEBUG:boto:path=/pklxkd-62/obj22?versionId=null 2023-05-31T21:06:39.681 DEBUG:boto:auth_path=/pklxkd-62/obj22?versionId=null 2023-05-31T21:06:39.681 DEBUG:boto:Method: HEAD 2023-05-31T21:06:39.681 DEBUG:boto:Path: /pklxkd-62/obj22?versionId=null 2023-05-31T21:06:39.682 DEBUG:boto:Data: 2023-05-31T21:06:39.682 DEBUG:boto:Headers: {} 2023-05-31T21:06:39.682 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T21:06:39.683 DEBUG:boto:Port: 8001 2023-05-31T21:06:39.683 DEBUG:boto:Params: {} 2023-05-31T21:06:39.684 DEBUG:boto:Token: None 2023-05-31T21:06:39.684 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 21:06:39 GMT /pklxkd-62/obj22?versionId=null 2023-05-31T21:06:39.684 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:VEBbnxOFucknm/y11LWqAKA6Cjg= 2023-05-31T21:06:39.685 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 21:06:39 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:VEBbnxOFucknm/y11LWqAKA6Cjg=', 'Content-Length': '0'} 2023-05-31T21:06:39.687 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 21:04:02 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000224ecd6dbde710a3-006477b6df-1fb6-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 21:06:39 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:06:39.687 DEBUG:boto:path=/pklxkd-62/obj22 2023-05-31T21:06:39.688 DEBUG:boto:auth_path=/pklxkd-62/obj22 2023-05-31T21:06:39.688 DEBUG:boto:path=/pklxkd-62/obj22?versionId=null 2023-05-31T21:06:39.689 DEBUG:boto:auth_path=/pklxkd-62/obj22?versionId=null 2023-05-31T21:06:39.689 DEBUG:boto:Method: HEAD 2023-05-31T21:06:39.689 DEBUG:boto:Path: /pklxkd-62/obj22?versionId=null 2023-05-31T21:06:39.690 DEBUG:boto:Data: 2023-05-31T21:06:39.690 DEBUG:boto:Headers: {} 2023-05-31T21:06:39.690 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T21:06:39.691 DEBUG:boto:Port: 8000 2023-05-31T21:06:39.691 DEBUG:boto:Params: {} 2023-05-31T21:06:39.692 DEBUG:boto:Token: None 2023-05-31T21:06:39.692 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 21:06:39 GMT /pklxkd-62/obj22?versionId=null 2023-05-31T21:06:39.692 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:VEBbnxOFucknm/y11LWqAKA6Cjg= 2023-05-31T21:06:39.693 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 21:06:39 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:VEBbnxOFucknm/y11LWqAKA6Cjg=', 'Content-Length': '0'} 2023-05-31T21:06:39.694 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 21:04:02 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009006faa7b44a8a31-006477b6df-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 21:06:39 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:06:39.695 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj22 2023-05-31T21:06:39.695 DEBUG:boto:path=/pklxkd-62/obj22 2023-05-31T21:06:39.696 DEBUG:boto:auth_path=/pklxkd-62/obj22 2023-05-31T21:06:39.696 DEBUG:boto:path=/pklxkd-62/obj22?versionId=null 2023-05-31T21:06:39.696 DEBUG:boto:auth_path=/pklxkd-62/obj22?versionId=null 2023-05-31T21:06:39.697 DEBUG:boto:Method: GET 2023-05-31T21:06:39.697 DEBUG:boto:Path: /pklxkd-62/obj22?versionId=null 2023-05-31T21:06:39.697 DEBUG:boto:Data: 2023-05-31T21:06:39.698 DEBUG:boto:Headers: {} 2023-05-31T21:06:39.698 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T21:06:39.699 DEBUG:boto:Port: 8001 2023-05-31T21:06:39.699 DEBUG:boto:Params: {} 2023-05-31T21:06:39.699 DEBUG:boto:Token: None 2023-05-31T21:06:39.700 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 21:06:39 GMT /pklxkd-62/obj22?versionId=null 2023-05-31T21:06:39.700 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:yiyTTXpve+uoHcwHEQabzehL7hw= 2023-05-31T21:06:39.701 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 21:06:39 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:yiyTTXpve+uoHcwHEQabzehL7hw=', 'Content-Length': '0'} 2023-05-31T21:06:39.702 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 21:04:02 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002078bfedad1ea7c1-006477b6df-1fb6-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 21:06:39 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:06:39.703 DEBUG:boto:path=/pklxkd-62/obj22 2023-05-31T21:06:39.703 DEBUG:boto:auth_path=/pklxkd-62/obj22 2023-05-31T21:06:39.704 DEBUG:boto:path=/pklxkd-62/obj22?versionId=null 2023-05-31T21:06:39.704 DEBUG:boto:auth_path=/pklxkd-62/obj22?versionId=null 2023-05-31T21:06:39.705 DEBUG:boto:Method: GET 2023-05-31T21:06:39.705 DEBUG:boto:Path: /pklxkd-62/obj22?versionId=null 2023-05-31T21:06:39.705 DEBUG:boto:Data: 2023-05-31T21:06:39.706 DEBUG:boto:Headers: {} 2023-05-31T21:06:39.706 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T21:06:39.706 DEBUG:boto:Port: 8000 2023-05-31T21:06:39.707 DEBUG:boto:Params: {} 2023-05-31T21:06:39.707 DEBUG:boto:Token: None 2023-05-31T21:06:39.708 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 21:06:39 GMT /pklxkd-62/obj22?versionId=null 2023-05-31T21:06:39.708 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:yiyTTXpve+uoHcwHEQabzehL7hw= 2023-05-31T21:06:39.709 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 21:06:39 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:yiyTTXpve+uoHcwHEQabzehL7hw=', 'Content-Length': '0'} 2023-05-31T21:06:39.710 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 21:04:02 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000abd81c6443021e5c-006477b6df-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 21:06:39 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:06:39.711 DEBUG:boto:path=/pklxkd-62/ 2023-05-31T21:06:39.712 DEBUG:boto:auth_path=/pklxkd-62/ 2023-05-31T21:06:39.712 DEBUG:boto:path=/pklxkd-62/?prefix=obj22 2023-05-31T21:06:39.712 DEBUG:boto:auth_path=/pklxkd-62/?prefix=obj22 2023-05-31T21:06:39.713 DEBUG:boto:Method: GET 2023-05-31T21:06:39.713 DEBUG:boto:Path: /pklxkd-62/?prefix=obj22 2023-05-31T21:06:39.713 DEBUG:boto:Data: 2023-05-31T21:06:39.714 DEBUG:boto:Headers: {} 2023-05-31T21:06:39.714 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T21:06:39.714 DEBUG:boto:Port: 8001 2023-05-31T21:06:39.715 DEBUG:boto:Params: {} 2023-05-31T21:06:39.715 DEBUG:boto:Token: None 2023-05-31T21:06:39.716 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 21:06:39 GMT /pklxkd-62/ 2023-05-31T21:06:39.716 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:nATqi8rxF7oCpSuUR6RziAwvAdo= 2023-05-31T21:06:39.717 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 21:06:39 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:nATqi8rxF7oCpSuUR6RziAwvAdo=', 'Content-Length': '0'} 2023-05-31T21:06:39.719 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e7bd7b2b72d09cea-006477b6df-1fb6-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 21:06:39 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:06:39.719 DEBUG:boto:b'pklxkd-62obj221000falseobj222023-05-31T21:04:02.653Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T21:06:39.721 DEBUG:boto:path=/pklxkd-62/ 2023-05-31T21:06:39.721 DEBUG:boto:auth_path=/pklxkd-62/ 2023-05-31T21:06:39.721 DEBUG:boto:path=/pklxkd-62/?prefix=obj22 2023-05-31T21:06:39.722 DEBUG:boto:auth_path=/pklxkd-62/?prefix=obj22 2023-05-31T21:06:39.722 DEBUG:boto:Method: GET 2023-05-31T21:06:39.722 DEBUG:boto:Path: /pklxkd-62/?prefix=obj22 2023-05-31T21:06:39.723 DEBUG:boto:Data: 2023-05-31T21:06:39.723 DEBUG:boto:Headers: {} 2023-05-31T21:06:39.724 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T21:06:39.724 DEBUG:boto:Port: 8000 2023-05-31T21:06:39.724 DEBUG:boto:Params: {} 2023-05-31T21:06:39.725 DEBUG:boto:Token: None 2023-05-31T21:06:39.725 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 21:06:39 GMT /pklxkd-62/ 2023-05-31T21:06:39.726 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:nATqi8rxF7oCpSuUR6RziAwvAdo= 2023-05-31T21:06:39.726 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 21:06:39 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:nATqi8rxF7oCpSuUR6RziAwvAdo=', 'Content-Length': '0'} 2023-05-31T21:06:39.728 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000009c339c5a0fefed6d-006477b6df-1630-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 21:06:39 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:06:39.729 DEBUG:boto:b'pklxkd-62obj221000falseobj222023-05-31T21:04:02.653Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T21:06:39.729 DEBUG:boto:path=/pklxkd-62/obj22 2023-05-31T21:06:39.730 DEBUG:boto:auth_path=/pklxkd-62/obj22 2023-05-31T21:06:39.730 DEBUG:boto:Method: HEAD 2023-05-31T21:06:39.731 DEBUG:boto:Path: /pklxkd-62/obj22 2023-05-31T21:06:39.731 DEBUG:boto:Data: 2023-05-31T21:06:39.731 DEBUG:boto:Headers: {} 2023-05-31T21:06:39.732 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T21:06:39.732 DEBUG:boto:Port: 8001 2023-05-31T21:06:39.733 DEBUG:boto:Params: {} 2023-05-31T21:06:39.733 DEBUG:boto:Token: None 2023-05-31T21:06:39.733 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 21:06:39 GMT /pklxkd-62/obj22 2023-05-31T21:06:39.734 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:FOtn7CI815TPFIj6dWMvsoH6gCo= 2023-05-31T21:06:39.734 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 21:06:39 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:FOtn7CI815TPFIj6dWMvsoH6gCo=', 'Content-Length': '0'} 2023-05-31T21:06:39.736 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 21:04:02 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a406a64a5106a4d3-006477b6df-1fb6-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 21:06:39 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:06:39.736 DEBUG:boto:path=/pklxkd-62/obj22 2023-05-31T21:06:39.737 DEBUG:boto:auth_path=/pklxkd-62/obj22 2023-05-31T21:06:39.737 DEBUG:boto:Method: HEAD 2023-05-31T21:06:39.737 DEBUG:boto:Path: /pklxkd-62/obj22 2023-05-31T21:06:39.738 DEBUG:boto:Data: 2023-05-31T21:06:39.738 DEBUG:boto:Headers: {} 2023-05-31T21:06:39.739 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T21:06:39.739 DEBUG:boto:Port: 8000 2023-05-31T21:06:39.739 DEBUG:boto:Params: {} 2023-05-31T21:06:39.740 DEBUG:boto:Token: None 2023-05-31T21:06:39.740 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 21:06:39 GMT /pklxkd-62/obj22 2023-05-31T21:06:39.741 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:FOtn7CI815TPFIj6dWMvsoH6gCo= 2023-05-31T21:06:39.741 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 21:06:39 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:FOtn7CI815TPFIj6dWMvsoH6gCo=', 'Content-Length': '0'} 2023-05-31T21:06:39.743 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 21:04:02 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000142291fa02b17691-006477b6df-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 21:06:39 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:06:39.743 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj22 2023-05-31T21:06:39.744 DEBUG:boto:path=/pklxkd-62/obj22 2023-05-31T21:06:39.744 DEBUG:boto:auth_path=/pklxkd-62/obj22 2023-05-31T21:06:39.745 DEBUG:boto:Method: GET 2023-05-31T21:06:39.745 DEBUG:boto:Path: /pklxkd-62/obj22 2023-05-31T21:06:39.745 DEBUG:boto:Data: 2023-05-31T21:06:39.746 DEBUG:boto:Headers: {} 2023-05-31T21:06:39.746 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T21:06:39.746 DEBUG:boto:Port: 8001 2023-05-31T21:06:39.747 DEBUG:boto:Params: {} 2023-05-31T21:06:39.747 DEBUG:boto:Token: None 2023-05-31T21:06:39.747 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 21:06:39 GMT /pklxkd-62/obj22 2023-05-31T21:06:39.748 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:it4b2JilUkDYpTwG+mGLu84pxgQ= 2023-05-31T21:06:39.749 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 21:06:39 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:it4b2JilUkDYpTwG+mGLu84pxgQ=', 'Content-Length': '0'} 2023-05-31T21:06:39.751 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 21:04:02 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c2b177ba9794639a-006477b6df-1fb6-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 21:06:39 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:06:39.751 DEBUG:boto:path=/pklxkd-62/obj22 2023-05-31T21:06:39.752 DEBUG:boto:auth_path=/pklxkd-62/obj22 2023-05-31T21:06:39.752 DEBUG:boto:Method: GET 2023-05-31T21:06:39.753 DEBUG:boto:Path: /pklxkd-62/obj22 2023-05-31T21:06:39.753 DEBUG:boto:Data: 2023-05-31T21:06:39.754 DEBUG:boto:Headers: {} 2023-05-31T21:06:39.754 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T21:06:39.755 DEBUG:boto:Port: 8000 2023-05-31T21:06:39.755 DEBUG:boto:Params: {} 2023-05-31T21:06:39.755 DEBUG:boto:Token: None 2023-05-31T21:06:39.756 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 21:06:39 GMT /pklxkd-62/obj22 2023-05-31T21:06:39.756 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:it4b2JilUkDYpTwG+mGLu84pxgQ= 2023-05-31T21:06:39.757 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 21:06:39 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:it4b2JilUkDYpTwG+mGLu84pxgQ=', 'Content-Length': '0'} 2023-05-31T21:06:39.758 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 21:04:02 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c786928709e4cd2c-006477b6df-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 21:06:39 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:06:39.800 DEBUG:boto:path=/pklxkd-62/ 2023-05-31T21:06:39.801 DEBUG:boto:auth_path=/pklxkd-62/ 2023-05-31T21:06:39.801 DEBUG:boto:path=/pklxkd-62/?prefix=obj22 2023-05-31T21:06:39.801 DEBUG:boto:auth_path=/pklxkd-62/?prefix=obj22 2023-05-31T21:06:39.802 DEBUG:boto:Method: GET 2023-05-31T21:06:39.802 DEBUG:boto:Path: /pklxkd-62/?prefix=obj22 2023-05-31T21:06:39.802 DEBUG:boto:Data: 2023-05-31T21:06:39.803 DEBUG:boto:Headers: {} 2023-05-31T21:06:39.803 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T21:06:39.804 DEBUG:boto:Port: 8001 2023-05-31T21:06:39.804 DEBUG:boto:Params: {} 2023-05-31T21:06:39.804 DEBUG:boto:Token: None 2023-05-31T21:06:39.805 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 21:06:39 GMT /pklxkd-62/ 2023-05-31T21:06:39.805 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:nATqi8rxF7oCpSuUR6RziAwvAdo= 2023-05-31T21:06:39.806 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 21:06:39 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:nATqi8rxF7oCpSuUR6RziAwvAdo=', 'Content-Length': '0'} 2023-05-31T21:06:39.808 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000992bfc3248c726c1-006477b6df-1fb6-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 21:06:39 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:06:39.809 DEBUG:boto:b'pklxkd-62obj221000falseobj222023-05-31T21:04:02.653Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T21:06:39.809 DEBUG:boto:path=/pklxkd-62/ 2023-05-31T21:06:39.810 DEBUG:boto:auth_path=/pklxkd-62/ 2023-05-31T21:06:39.810 DEBUG:boto:path=/pklxkd-62/?prefix=obj22 2023-05-31T21:06:39.811 DEBUG:boto:auth_path=/pklxkd-62/?prefix=obj22 2023-05-31T21:06:39.811 DEBUG:boto:Method: GET 2023-05-31T21:06:39.811 DEBUG:boto:Path: /pklxkd-62/?prefix=obj22 2023-05-31T21:06:39.812 DEBUG:boto:Data: 2023-05-31T21:06:39.812 DEBUG:boto:Headers: {} 2023-05-31T21:06:39.813 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T21:06:39.813 DEBUG:boto:Port: 8000 2023-05-31T21:06:39.813 DEBUG:boto:Params: {} 2023-05-31T21:06:39.814 DEBUG:boto:Token: None 2023-05-31T21:06:39.814 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 21:06:39 GMT /pklxkd-62/ 2023-05-31T21:06:39.815 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:nATqi8rxF7oCpSuUR6RziAwvAdo= 2023-05-31T21:06:39.815 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 21:06:39 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:nATqi8rxF7oCpSuUR6RziAwvAdo=', 'Content-Length': '0'} 2023-05-31T21:06:39.817 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b0aad514fc6e1979-006477b6df-1630-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 21:06:39 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:06:39.818 DEBUG:boto:b'pklxkd-62obj221000falseobj222023-05-31T21:04:02.653Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T21:06:39.819 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj33 2023-05-31T21:06:39.819 DEBUG:boto:path=/pklxkd-62/obj33 2023-05-31T21:06:39.819 DEBUG:boto:auth_path=/pklxkd-62/obj33 2023-05-31T21:06:39.820 DEBUG:boto:path=/pklxkd-62/obj33?versionId=null 2023-05-31T21:06:39.820 DEBUG:boto:auth_path=/pklxkd-62/obj33?versionId=null 2023-05-31T21:06:39.820 DEBUG:boto:Method: GET 2023-05-31T21:06:39.821 DEBUG:boto:Path: /pklxkd-62/obj33?versionId=null 2023-05-31T21:06:39.821 DEBUG:boto:Data: 2023-05-31T21:06:39.821 DEBUG:boto:Headers: {} 2023-05-31T21:06:39.822 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T21:06:39.822 DEBUG:boto:Port: 8001 2023-05-31T21:06:39.823 DEBUG:boto:Params: {} 2023-05-31T21:06:39.823 DEBUG:boto:Token: None 2023-05-31T21:06:39.823 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 21:06:39 GMT /pklxkd-62/obj33?versionId=null 2023-05-31T21:06:39.824 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:zYp3BDhoHo4MDAy9wKZPM3DEIKU= 2023-05-31T21:06:39.824 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 21:06:39 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:zYp3BDhoHo4MDAy9wKZPM3DEIKU=', 'Content-Length': '0'} 2023-05-31T21:06:39.826 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 21:04:02 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a26d3a8bd88b029e-006477b6df-1fb6-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 21:06:39 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:06:39.827 DEBUG:boto:path=/pklxkd-62/obj33 2023-05-31T21:06:39.827 DEBUG:boto:auth_path=/pklxkd-62/obj33 2023-05-31T21:06:39.828 DEBUG:boto:path=/pklxkd-62/obj33?versionId=null 2023-05-31T21:06:39.828 DEBUG:boto:auth_path=/pklxkd-62/obj33?versionId=null 2023-05-31T21:06:39.828 DEBUG:boto:Method: GET 2023-05-31T21:06:39.829 DEBUG:boto:Path: /pklxkd-62/obj33?versionId=null 2023-05-31T21:06:39.829 DEBUG:boto:Data: 2023-05-31T21:06:39.829 DEBUG:boto:Headers: {} 2023-05-31T21:06:39.830 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T21:06:39.830 DEBUG:boto:Port: 8000 2023-05-31T21:06:39.831 DEBUG:boto:Params: {} 2023-05-31T21:06:39.831 DEBUG:boto:Token: None 2023-05-31T21:06:39.832 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 21:06:39 GMT /pklxkd-62/obj33?versionId=null 2023-05-31T21:06:39.832 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:zYp3BDhoHo4MDAy9wKZPM3DEIKU= 2023-05-31T21:06:39.833 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 21:06:39 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:zYp3BDhoHo4MDAy9wKZPM3DEIKU=', 'Content-Length': '0'} 2023-05-31T21:06:39.835 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 21:04:02 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000076e84f70bfcf8be0-006477b6df-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 21:06:39 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:06:39.835 DEBUG:boto:path=/pklxkd-62/obj33 2023-05-31T21:06:39.836 DEBUG:boto:auth_path=/pklxkd-62/obj33 2023-05-31T21:06:39.836 DEBUG:boto:path=/pklxkd-62/obj33?versionId=null 2023-05-31T21:06:39.836 DEBUG:boto:auth_path=/pklxkd-62/obj33?versionId=null 2023-05-31T21:06:39.837 DEBUG:boto:Method: HEAD 2023-05-31T21:06:39.837 DEBUG:boto:Path: /pklxkd-62/obj33?versionId=null 2023-05-31T21:06:39.837 DEBUG:boto:Data: 2023-05-31T21:06:39.838 DEBUG:boto:Headers: {} 2023-05-31T21:06:39.838 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T21:06:39.838 DEBUG:boto:Port: 8001 2023-05-31T21:06:39.839 DEBUG:boto:Params: {} 2023-05-31T21:06:39.839 DEBUG:boto:Token: None 2023-05-31T21:06:39.840 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 21:06:39 GMT /pklxkd-62/obj33?versionId=null 2023-05-31T21:06:39.840 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:D3ZOBFUZCn0hD86H9QyHTpwR61w= 2023-05-31T21:06:39.840 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 21:06:39 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:D3ZOBFUZCn0hD86H9QyHTpwR61w=', 'Content-Length': '0'} 2023-05-31T21:06:39.842 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 21:04:02 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000fa47bcff8e773336-006477b6df-1fb6-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 21:06:39 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:06:39.843 DEBUG:boto:path=/pklxkd-62/obj33 2023-05-31T21:06:39.843 DEBUG:boto:auth_path=/pklxkd-62/obj33 2023-05-31T21:06:39.844 DEBUG:boto:path=/pklxkd-62/obj33?versionId=null 2023-05-31T21:06:39.844 DEBUG:boto:auth_path=/pklxkd-62/obj33?versionId=null 2023-05-31T21:06:39.844 DEBUG:boto:Method: HEAD 2023-05-31T21:06:39.845 DEBUG:boto:Path: /pklxkd-62/obj33?versionId=null 2023-05-31T21:06:39.845 DEBUG:boto:Data: 2023-05-31T21:06:39.845 DEBUG:boto:Headers: {} 2023-05-31T21:06:39.846 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T21:06:39.846 DEBUG:boto:Port: 8000 2023-05-31T21:06:39.846 DEBUG:boto:Params: {} 2023-05-31T21:06:39.847 DEBUG:boto:Token: None 2023-05-31T21:06:39.847 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 21:06:39 GMT /pklxkd-62/obj33?versionId=null 2023-05-31T21:06:39.848 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:D3ZOBFUZCn0hD86H9QyHTpwR61w= 2023-05-31T21:06:39.848 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 21:06:39 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:D3ZOBFUZCn0hD86H9QyHTpwR61w=', 'Content-Length': '0'} 2023-05-31T21:06:39.850 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 21:04:02 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b5a28c10aeabf76a-006477b6df-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 21:06:39 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:06:39.851 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj33 2023-05-31T21:06:39.851 DEBUG:boto:path=/pklxkd-62/obj33 2023-05-31T21:06:39.851 DEBUG:boto:auth_path=/pklxkd-62/obj33 2023-05-31T21:06:39.852 DEBUG:boto:path=/pklxkd-62/obj33?versionId=null 2023-05-31T21:06:39.852 DEBUG:boto:auth_path=/pklxkd-62/obj33?versionId=null 2023-05-31T21:06:39.853 DEBUG:boto:Method: GET 2023-05-31T21:06:39.853 DEBUG:boto:Path: /pklxkd-62/obj33?versionId=null 2023-05-31T21:06:39.853 DEBUG:boto:Data: 2023-05-31T21:06:39.854 DEBUG:boto:Headers: {} 2023-05-31T21:06:39.854 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T21:06:39.854 DEBUG:boto:Port: 8001 2023-05-31T21:06:39.855 DEBUG:boto:Params: {} 2023-05-31T21:06:39.855 DEBUG:boto:Token: None 2023-05-31T21:06:39.856 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 21:06:39 GMT /pklxkd-62/obj33?versionId=null 2023-05-31T21:06:39.856 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:zYp3BDhoHo4MDAy9wKZPM3DEIKU= 2023-05-31T21:06:39.856 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 21:06:39 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:zYp3BDhoHo4MDAy9wKZPM3DEIKU=', 'Content-Length': '0'} 2023-05-31T21:06:39.858 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 21:04:02 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e8d3401766e04e3f-006477b6df-1fb6-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 21:06:39 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:06:39.859 DEBUG:boto:path=/pklxkd-62/obj33 2023-05-31T21:06:39.859 DEBUG:boto:auth_path=/pklxkd-62/obj33 2023-05-31T21:06:39.860 DEBUG:boto:path=/pklxkd-62/obj33?versionId=null 2023-05-31T21:06:39.860 DEBUG:boto:auth_path=/pklxkd-62/obj33?versionId=null 2023-05-31T21:06:39.860 DEBUG:boto:Method: GET 2023-05-31T21:06:39.861 DEBUG:boto:Path: /pklxkd-62/obj33?versionId=null 2023-05-31T21:06:39.861 DEBUG:boto:Data: 2023-05-31T21:06:39.861 DEBUG:boto:Headers: {} 2023-05-31T21:06:39.862 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T21:06:39.863 DEBUG:boto:Port: 8000 2023-05-31T21:06:39.863 DEBUG:boto:Params: {} 2023-05-31T21:06:39.863 DEBUG:boto:Token: None 2023-05-31T21:06:39.864 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 21:06:39 GMT /pklxkd-62/obj33?versionId=null 2023-05-31T21:06:39.864 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:zYp3BDhoHo4MDAy9wKZPM3DEIKU= 2023-05-31T21:06:39.865 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 21:06:39 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:zYp3BDhoHo4MDAy9wKZPM3DEIKU=', 'Content-Length': '0'} 2023-05-31T21:06:39.866 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 21:04:02 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004f0268c512506f95-006477b6df-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 21:06:39 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:06:39.867 DEBUG:boto:path=/pklxkd-62/ 2023-05-31T21:06:39.868 DEBUG:boto:auth_path=/pklxkd-62/ 2023-05-31T21:06:39.868 DEBUG:boto:path=/pklxkd-62/?prefix=obj33 2023-05-31T21:06:39.868 DEBUG:boto:auth_path=/pklxkd-62/?prefix=obj33 2023-05-31T21:06:39.869 DEBUG:boto:Method: GET 2023-05-31T21:06:39.869 DEBUG:boto:Path: /pklxkd-62/?prefix=obj33 2023-05-31T21:06:39.869 DEBUG:boto:Data: 2023-05-31T21:06:39.870 DEBUG:boto:Headers: {} 2023-05-31T21:06:39.870 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T21:06:39.871 DEBUG:boto:Port: 8001 2023-05-31T21:06:39.871 DEBUG:boto:Params: {} 2023-05-31T21:06:39.871 DEBUG:boto:Token: None 2023-05-31T21:06:39.872 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 21:06:39 GMT /pklxkd-62/ 2023-05-31T21:06:39.872 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:nATqi8rxF7oCpSuUR6RziAwvAdo= 2023-05-31T21:06:39.873 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 21:06:39 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:nATqi8rxF7oCpSuUR6RziAwvAdo=', 'Content-Length': '0'} 2023-05-31T21:06:39.875 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000003fda71f73d7b359-006477b6df-1fb6-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 21:06:39 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:06:39.876 DEBUG:boto:b'pklxkd-62obj331000falseobj332023-05-31T21:04:02.707Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T21:06:39.876 DEBUG:boto:path=/pklxkd-62/ 2023-05-31T21:06:39.877 DEBUG:boto:auth_path=/pklxkd-62/ 2023-05-31T21:06:39.877 DEBUG:boto:path=/pklxkd-62/?prefix=obj33 2023-05-31T21:06:39.877 DEBUG:boto:auth_path=/pklxkd-62/?prefix=obj33 2023-05-31T21:06:39.878 DEBUG:boto:Method: GET 2023-05-31T21:06:39.878 DEBUG:boto:Path: /pklxkd-62/?prefix=obj33 2023-05-31T21:06:39.878 DEBUG:boto:Data: 2023-05-31T21:06:39.879 DEBUG:boto:Headers: {} 2023-05-31T21:06:39.879 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T21:06:39.879 DEBUG:boto:Port: 8000 2023-05-31T21:06:39.880 DEBUG:boto:Params: {} 2023-05-31T21:06:39.880 DEBUG:boto:Token: None 2023-05-31T21:06:39.881 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 21:06:39 GMT /pklxkd-62/ 2023-05-31T21:06:39.881 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:nATqi8rxF7oCpSuUR6RziAwvAdo= 2023-05-31T21:06:39.881 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 21:06:39 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:nATqi8rxF7oCpSuUR6RziAwvAdo=', 'Content-Length': '0'} 2023-05-31T21:06:39.883 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000005ec1eb449f73b8bd-006477b6df-1630-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 21:06:39 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:06:39.884 DEBUG:boto:b'pklxkd-62obj331000falseobj332023-05-31T21:04:02.707Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T21:06:39.885 DEBUG:boto:path=/pklxkd-62/obj33 2023-05-31T21:06:39.885 DEBUG:boto:auth_path=/pklxkd-62/obj33 2023-05-31T21:06:39.886 DEBUG:boto:Method: HEAD 2023-05-31T21:06:39.886 DEBUG:boto:Path: /pklxkd-62/obj33 2023-05-31T21:06:39.886 DEBUG:boto:Data: 2023-05-31T21:06:39.887 DEBUG:boto:Headers: {} 2023-05-31T21:06:39.887 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T21:06:39.887 DEBUG:boto:Port: 8001 2023-05-31T21:06:39.888 DEBUG:boto:Params: {} 2023-05-31T21:06:39.888 DEBUG:boto:Token: None 2023-05-31T21:06:39.889 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 21:06:39 GMT /pklxkd-62/obj33 2023-05-31T21:06:39.889 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:DIx4xZlThDHIOLTVX9CJ1zR+aMw= 2023-05-31T21:06:39.889 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 21:06:39 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:DIx4xZlThDHIOLTVX9CJ1zR+aMw=', 'Content-Length': '0'} 2023-05-31T21:06:39.891 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 21:04:02 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004a66ba1792ed2434-006477b6df-1fb6-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 21:06:39 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:06:39.892 DEBUG:boto:path=/pklxkd-62/obj33 2023-05-31T21:06:39.892 DEBUG:boto:auth_path=/pklxkd-62/obj33 2023-05-31T21:06:39.893 DEBUG:boto:Method: HEAD 2023-05-31T21:06:39.893 DEBUG:boto:Path: /pklxkd-62/obj33 2023-05-31T21:06:39.894 DEBUG:boto:Data: 2023-05-31T21:06:39.894 DEBUG:boto:Headers: {} 2023-05-31T21:06:39.895 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T21:06:39.896 DEBUG:boto:Port: 8000 2023-05-31T21:06:39.896 DEBUG:boto:Params: {} 2023-05-31T21:06:39.896 DEBUG:boto:Token: None 2023-05-31T21:06:39.897 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 21:06:39 GMT /pklxkd-62/obj33 2023-05-31T21:06:39.898 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:DIx4xZlThDHIOLTVX9CJ1zR+aMw= 2023-05-31T21:06:39.898 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 21:06:39 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:DIx4xZlThDHIOLTVX9CJ1zR+aMw=', 'Content-Length': '0'} 2023-05-31T21:06:39.900 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 21:04:02 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000072eaf6faf44f93ac-006477b6df-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 21:06:39 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:06:39.900 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj33 2023-05-31T21:06:39.901 DEBUG:boto:path=/pklxkd-62/obj33 2023-05-31T21:06:39.901 DEBUG:boto:auth_path=/pklxkd-62/obj33 2023-05-31T21:06:39.901 DEBUG:boto:Method: GET 2023-05-31T21:06:39.902 DEBUG:boto:Path: /pklxkd-62/obj33 2023-05-31T21:06:39.902 DEBUG:boto:Data: 2023-05-31T21:06:39.902 DEBUG:boto:Headers: {} 2023-05-31T21:06:39.903 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T21:06:39.903 DEBUG:boto:Port: 8001 2023-05-31T21:06:39.904 DEBUG:boto:Params: {} 2023-05-31T21:06:39.904 DEBUG:boto:Token: None 2023-05-31T21:06:39.904 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 21:06:39 GMT /pklxkd-62/obj33 2023-05-31T21:06:39.905 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:hrZ6SEp674rArhbeCCx7LZ9QuMs= 2023-05-31T21:06:39.905 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 21:06:39 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:hrZ6SEp674rArhbeCCx7LZ9QuMs=', 'Content-Length': '0'} 2023-05-31T21:06:39.907 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 21:04:02 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008ba803e658926eec-006477b6df-1fb6-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 21:06:39 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:06:39.908 DEBUG:boto:path=/pklxkd-62/obj33 2023-05-31T21:06:39.908 DEBUG:boto:auth_path=/pklxkd-62/obj33 2023-05-31T21:06:39.909 DEBUG:boto:Method: GET 2023-05-31T21:06:39.909 DEBUG:boto:Path: /pklxkd-62/obj33 2023-05-31T21:06:39.909 DEBUG:boto:Data: 2023-05-31T21:06:39.910 DEBUG:boto:Headers: {} 2023-05-31T21:06:39.910 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T21:06:39.910 DEBUG:boto:Port: 8000 2023-05-31T21:06:39.911 DEBUG:boto:Params: {} 2023-05-31T21:06:39.911 DEBUG:boto:Token: None 2023-05-31T21:06:39.912 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 21:06:39 GMT /pklxkd-62/obj33 2023-05-31T21:06:39.912 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:hrZ6SEp674rArhbeCCx7LZ9QuMs= 2023-05-31T21:06:39.913 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 21:06:39 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:hrZ6SEp674rArhbeCCx7LZ9QuMs=', 'Content-Length': '0'} 2023-05-31T21:06:39.915 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 21:04:02 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b34da23a8e779984-006477b6df-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 21:06:39 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:06:39.915 DEBUG:boto:path=/pklxkd-62/ 2023-05-31T21:06:39.916 DEBUG:boto:auth_path=/pklxkd-62/ 2023-05-31T21:06:39.916 DEBUG:boto:path=/pklxkd-62/?prefix=obj33 2023-05-31T21:06:39.916 DEBUG:boto:auth_path=/pklxkd-62/?prefix=obj33 2023-05-31T21:06:39.917 DEBUG:boto:Method: GET 2023-05-31T21:06:39.917 DEBUG:boto:Path: /pklxkd-62/?prefix=obj33 2023-05-31T21:06:39.917 DEBUG:boto:Data: 2023-05-31T21:06:39.918 DEBUG:boto:Headers: {} 2023-05-31T21:06:39.918 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T21:06:39.918 DEBUG:boto:Port: 8001 2023-05-31T21:06:39.919 DEBUG:boto:Params: {} 2023-05-31T21:06:39.919 DEBUG:boto:Token: None 2023-05-31T21:06:39.920 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 21:06:39 GMT /pklxkd-62/ 2023-05-31T21:06:39.920 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:nATqi8rxF7oCpSuUR6RziAwvAdo= 2023-05-31T21:06:39.921 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 21:06:39 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:nATqi8rxF7oCpSuUR6RziAwvAdo=', 'Content-Length': '0'} 2023-05-31T21:06:39.923 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000005f292d2e14074326-006477b6df-1fb6-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 21:06:39 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:06:39.923 DEBUG:boto:b'pklxkd-62obj331000falseobj332023-05-31T21:04:02.707Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T21:06:39.924 DEBUG:boto:path=/pklxkd-62/ 2023-05-31T21:06:39.924 DEBUG:boto:auth_path=/pklxkd-62/ 2023-05-31T21:06:39.925 DEBUG:boto:path=/pklxkd-62/?prefix=obj33 2023-05-31T21:06:39.925 DEBUG:boto:auth_path=/pklxkd-62/?prefix=obj33 2023-05-31T21:06:39.925 DEBUG:boto:Method: GET 2023-05-31T21:06:39.926 DEBUG:boto:Path: /pklxkd-62/?prefix=obj33 2023-05-31T21:06:39.926 DEBUG:boto:Data: 2023-05-31T21:06:39.926 DEBUG:boto:Headers: {} 2023-05-31T21:06:39.927 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T21:06:39.927 DEBUG:boto:Port: 8000 2023-05-31T21:06:39.928 DEBUG:boto:Params: {} 2023-05-31T21:06:39.928 DEBUG:boto:Token: None 2023-05-31T21:06:39.928 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 21:06:39 GMT /pklxkd-62/ 2023-05-31T21:06:39.929 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:nATqi8rxF7oCpSuUR6RziAwvAdo= 2023-05-31T21:06:39.929 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 21:06:39 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:nATqi8rxF7oCpSuUR6RziAwvAdo=', 'Content-Length': '0'} 2023-05-31T21:06:39.931 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000af2a78675de68548-006477b6df-1630-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 21:06:39 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:06:39.932 DEBUG:boto:b'pklxkd-62obj331000falseobj332023-05-31T21:04:02.707Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T21:06:39.932 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj44 2023-05-31T21:06:39.933 DEBUG:boto:path=/pklxkd-62/obj44 2023-05-31T21:06:39.933 DEBUG:boto:auth_path=/pklxkd-62/obj44 2023-05-31T21:06:39.934 DEBUG:boto:path=/pklxkd-62/obj44?versionId=null 2023-05-31T21:06:39.934 DEBUG:boto:auth_path=/pklxkd-62/obj44?versionId=null 2023-05-31T21:06:39.934 DEBUG:boto:Method: GET 2023-05-31T21:06:39.935 DEBUG:boto:Path: /pklxkd-62/obj44?versionId=null 2023-05-31T21:06:39.935 DEBUG:boto:Data: 2023-05-31T21:06:39.936 DEBUG:boto:Headers: {} 2023-05-31T21:06:39.936 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T21:06:39.936 DEBUG:boto:Port: 8001 2023-05-31T21:06:39.937 DEBUG:boto:Params: {} 2023-05-31T21:06:39.937 DEBUG:boto:Token: None 2023-05-31T21:06:39.938 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 21:06:39 GMT /pklxkd-62/obj44?versionId=null 2023-05-31T21:06:39.938 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:jN2Lyf0eGotW0NcxDqAxiBL6bEc= 2023-05-31T21:06:39.939 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 21:06:39 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:jN2Lyf0eGotW0NcxDqAxiBL6bEc=', 'Content-Length': '0'} 2023-05-31T21:06:39.941 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 21:04:02 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007847ae2772d85189-006477b6df-1fb6-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 21:06:39 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:06:39.941 DEBUG:boto:path=/pklxkd-62/obj44 2023-05-31T21:06:39.942 DEBUG:boto:auth_path=/pklxkd-62/obj44 2023-05-31T21:06:39.942 DEBUG:boto:path=/pklxkd-62/obj44?versionId=null 2023-05-31T21:06:39.943 DEBUG:boto:auth_path=/pklxkd-62/obj44?versionId=null 2023-05-31T21:06:39.943 DEBUG:boto:Method: GET 2023-05-31T21:06:39.943 DEBUG:boto:Path: /pklxkd-62/obj44?versionId=null 2023-05-31T21:06:39.944 DEBUG:boto:Data: 2023-05-31T21:06:39.944 DEBUG:boto:Headers: {} 2023-05-31T21:06:39.944 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T21:06:39.945 DEBUG:boto:Port: 8000 2023-05-31T21:06:39.945 DEBUG:boto:Params: {} 2023-05-31T21:06:39.946 DEBUG:boto:Token: None 2023-05-31T21:06:39.946 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 21:06:39 GMT /pklxkd-62/obj44?versionId=null 2023-05-31T21:06:39.947 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:jN2Lyf0eGotW0NcxDqAxiBL6bEc= 2023-05-31T21:06:39.947 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 21:06:39 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:jN2Lyf0eGotW0NcxDqAxiBL6bEc=', 'Content-Length': '0'} 2023-05-31T21:06:39.949 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 21:04:02 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008992c0c35d121bab-006477b6df-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 21:06:39 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:06:39.950 DEBUG:boto:path=/pklxkd-62/obj44 2023-05-31T21:06:39.950 DEBUG:boto:auth_path=/pklxkd-62/obj44 2023-05-31T21:06:39.951 DEBUG:boto:path=/pklxkd-62/obj44?versionId=null 2023-05-31T21:06:39.951 DEBUG:boto:auth_path=/pklxkd-62/obj44?versionId=null 2023-05-31T21:06:39.951 DEBUG:boto:Method: HEAD 2023-05-31T21:06:39.952 DEBUG:boto:Path: /pklxkd-62/obj44?versionId=null 2023-05-31T21:06:39.952 DEBUG:boto:Data: 2023-05-31T21:06:39.953 DEBUG:boto:Headers: {} 2023-05-31T21:06:39.953 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T21:06:39.953 DEBUG:boto:Port: 8001 2023-05-31T21:06:39.954 DEBUG:boto:Params: {} 2023-05-31T21:06:39.954 DEBUG:boto:Token: None 2023-05-31T21:06:39.954 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 21:06:39 GMT /pklxkd-62/obj44?versionId=null 2023-05-31T21:06:39.955 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:Fu3a7A9M8HwmiemD2nCJwy0+5ys= 2023-05-31T21:06:39.955 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 21:06:39 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:Fu3a7A9M8HwmiemD2nCJwy0+5ys=', 'Content-Length': '0'} 2023-05-31T21:06:39.957 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 21:04:02 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000dbafc6bf7386e3af-006477b6df-1fb6-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 21:06:39 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:06:39.958 DEBUG:boto:path=/pklxkd-62/obj44 2023-05-31T21:06:39.958 DEBUG:boto:auth_path=/pklxkd-62/obj44 2023-05-31T21:06:39.958 DEBUG:boto:path=/pklxkd-62/obj44?versionId=null 2023-05-31T21:06:39.959 DEBUG:boto:auth_path=/pklxkd-62/obj44?versionId=null 2023-05-31T21:06:39.959 DEBUG:boto:Method: HEAD 2023-05-31T21:06:39.959 DEBUG:boto:Path: /pklxkd-62/obj44?versionId=null 2023-05-31T21:06:39.960 DEBUG:boto:Data: 2023-05-31T21:06:39.960 DEBUG:boto:Headers: {} 2023-05-31T21:06:39.960 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T21:06:39.961 DEBUG:boto:Port: 8000 2023-05-31T21:06:39.961 DEBUG:boto:Params: {} 2023-05-31T21:06:39.961 DEBUG:boto:Token: None 2023-05-31T21:06:39.962 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 21:06:39 GMT /pklxkd-62/obj44?versionId=null 2023-05-31T21:06:39.962 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:Fu3a7A9M8HwmiemD2nCJwy0+5ys= 2023-05-31T21:06:39.963 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 21:06:39 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:Fu3a7A9M8HwmiemD2nCJwy0+5ys=', 'Content-Length': '0'} 2023-05-31T21:06:39.964 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 21:04:02 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000cd676dfbcd312206-006477b6df-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 21:06:39 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:06:39.965 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj44 2023-05-31T21:06:39.965 DEBUG:boto:path=/pklxkd-62/obj44 2023-05-31T21:06:39.966 DEBUG:boto:auth_path=/pklxkd-62/obj44 2023-05-31T21:06:39.966 DEBUG:boto:path=/pklxkd-62/obj44?versionId=null 2023-05-31T21:06:39.966 DEBUG:boto:auth_path=/pklxkd-62/obj44?versionId=null 2023-05-31T21:06:39.967 DEBUG:boto:Method: GET 2023-05-31T21:06:39.967 DEBUG:boto:Path: /pklxkd-62/obj44?versionId=null 2023-05-31T21:06:39.967 DEBUG:boto:Data: 2023-05-31T21:06:39.968 DEBUG:boto:Headers: {} 2023-05-31T21:06:39.968 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T21:06:39.969 DEBUG:boto:Port: 8001 2023-05-31T21:06:39.969 DEBUG:boto:Params: {} 2023-05-31T21:06:39.969 DEBUG:boto:Token: None 2023-05-31T21:06:39.970 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 21:06:39 GMT /pklxkd-62/obj44?versionId=null 2023-05-31T21:06:39.970 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:jN2Lyf0eGotW0NcxDqAxiBL6bEc= 2023-05-31T21:06:39.971 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 21:06:39 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:jN2Lyf0eGotW0NcxDqAxiBL6bEc=', 'Content-Length': '0'} 2023-05-31T21:06:39.972 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 21:04:02 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008d8e860a59cc7b47-006477b6df-1fb6-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 21:06:39 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:06:40.016 DEBUG:boto:path=/pklxkd-62/obj44 2023-05-31T21:06:40.017 DEBUG:boto:auth_path=/pklxkd-62/obj44 2023-05-31T21:06:40.017 DEBUG:boto:path=/pklxkd-62/obj44?versionId=null 2023-05-31T21:06:40.017 DEBUG:boto:auth_path=/pklxkd-62/obj44?versionId=null 2023-05-31T21:06:40.018 DEBUG:boto:Method: GET 2023-05-31T21:06:40.018 DEBUG:boto:Path: /pklxkd-62/obj44?versionId=null 2023-05-31T21:06:40.018 DEBUG:boto:Data: 2023-05-31T21:06:40.019 DEBUG:boto:Headers: {} 2023-05-31T21:06:40.019 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T21:06:40.020 DEBUG:boto:Port: 8000 2023-05-31T21:06:40.020 DEBUG:boto:Params: {} 2023-05-31T21:06:40.021 DEBUG:boto:Token: None 2023-05-31T21:06:40.021 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 21:06:40 GMT /pklxkd-62/obj44?versionId=null 2023-05-31T21:06:40.022 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:MoM8JvRj+Jq9NJr6TolAuHve2LA= 2023-05-31T21:06:40.022 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 21:06:40 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:MoM8JvRj+Jq9NJr6TolAuHve2LA=', 'Content-Length': '0'} 2023-05-31T21:06:40.024 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 21:04:02 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ae019fe7214e2d72-006477b6e0-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 21:06:40 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:06:40.025 DEBUG:boto:path=/pklxkd-62/ 2023-05-31T21:06:40.025 DEBUG:boto:auth_path=/pklxkd-62/ 2023-05-31T21:06:40.026 DEBUG:boto:path=/pklxkd-62/?prefix=obj44 2023-05-31T21:06:40.026 DEBUG:boto:auth_path=/pklxkd-62/?prefix=obj44 2023-05-31T21:06:40.026 DEBUG:boto:Method: GET 2023-05-31T21:06:40.027 DEBUG:boto:Path: /pklxkd-62/?prefix=obj44 2023-05-31T21:06:40.027 DEBUG:boto:Data: 2023-05-31T21:06:40.027 DEBUG:boto:Headers: {} 2023-05-31T21:06:40.028 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T21:06:40.028 DEBUG:boto:Port: 8001 2023-05-31T21:06:40.029 DEBUG:boto:Params: {} 2023-05-31T21:06:40.029 DEBUG:boto:Token: None 2023-05-31T21:06:40.029 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 21:06:40 GMT /pklxkd-62/ 2023-05-31T21:06:40.030 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:Zw9HJEYqh8L1MEOneHxUYI8gQCw= 2023-05-31T21:06:40.030 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 21:06:40 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:Zw9HJEYqh8L1MEOneHxUYI8gQCw=', 'Content-Length': '0'} 2023-05-31T21:06:40.033 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b22e6d4e435d8826-006477b6e0-1fb6-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 21:06:40 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:06:40.033 DEBUG:boto:b'pklxkd-62obj441000falseobj442023-05-31T21:04:02.730Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T21:06:40.034 DEBUG:boto:path=/pklxkd-62/ 2023-05-31T21:06:40.034 DEBUG:boto:auth_path=/pklxkd-62/ 2023-05-31T21:06:40.035 DEBUG:boto:path=/pklxkd-62/?prefix=obj44 2023-05-31T21:06:40.035 DEBUG:boto:auth_path=/pklxkd-62/?prefix=obj44 2023-05-31T21:06:40.035 DEBUG:boto:Method: GET 2023-05-31T21:06:40.036 DEBUG:boto:Path: /pklxkd-62/?prefix=obj44 2023-05-31T21:06:40.036 DEBUG:boto:Data: 2023-05-31T21:06:40.036 DEBUG:boto:Headers: {} 2023-05-31T21:06:40.037 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T21:06:40.037 DEBUG:boto:Port: 8000 2023-05-31T21:06:40.037 DEBUG:boto:Params: {} 2023-05-31T21:06:40.038 DEBUG:boto:Token: None 2023-05-31T21:06:40.038 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 21:06:40 GMT /pklxkd-62/ 2023-05-31T21:06:40.039 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:Zw9HJEYqh8L1MEOneHxUYI8gQCw= 2023-05-31T21:06:40.039 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 21:06:40 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:Zw9HJEYqh8L1MEOneHxUYI8gQCw=', 'Content-Length': '0'} 2023-05-31T21:06:40.042 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000006f6b8f37bf628712-006477b6e0-1630-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 21:06:40 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:06:40.042 DEBUG:boto:b'pklxkd-62obj441000falseobj442023-05-31T21:04:02.730Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T21:06:40.043 DEBUG:boto:path=/pklxkd-62/obj44 2023-05-31T21:06:40.043 DEBUG:boto:auth_path=/pklxkd-62/obj44 2023-05-31T21:06:40.044 DEBUG:boto:Method: HEAD 2023-05-31T21:06:40.044 DEBUG:boto:Path: /pklxkd-62/obj44 2023-05-31T21:06:40.044 DEBUG:boto:Data: 2023-05-31T21:06:40.045 DEBUG:boto:Headers: {} 2023-05-31T21:06:40.045 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T21:06:40.046 DEBUG:boto:Port: 8001 2023-05-31T21:06:40.046 DEBUG:boto:Params: {} 2023-05-31T21:06:40.046 DEBUG:boto:Token: None 2023-05-31T21:06:40.047 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 21:06:40 GMT /pklxkd-62/obj44 2023-05-31T21:06:40.047 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:cf+EXmBrXvcIO7yhSlkD+rh/yvQ= 2023-05-31T21:06:40.047 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 21:06:40 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:cf+EXmBrXvcIO7yhSlkD+rh/yvQ=', 'Content-Length': '0'} 2023-05-31T21:06:40.049 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 21:04:02 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a0c482e787ccff83-006477b6e0-1fb6-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 21:06:40 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:06:40.050 DEBUG:boto:path=/pklxkd-62/obj44 2023-05-31T21:06:40.050 DEBUG:boto:auth_path=/pklxkd-62/obj44 2023-05-31T21:06:40.050 DEBUG:boto:Method: HEAD 2023-05-31T21:06:40.051 DEBUG:boto:Path: /pklxkd-62/obj44 2023-05-31T21:06:40.051 DEBUG:boto:Data: 2023-05-31T21:06:40.052 DEBUG:boto:Headers: {} 2023-05-31T21:06:40.052 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T21:06:40.052 DEBUG:boto:Port: 8000 2023-05-31T21:06:40.053 DEBUG:boto:Params: {} 2023-05-31T21:06:40.053 DEBUG:boto:Token: None 2023-05-31T21:06:40.054 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 21:06:40 GMT /pklxkd-62/obj44 2023-05-31T21:06:40.054 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:cf+EXmBrXvcIO7yhSlkD+rh/yvQ= 2023-05-31T21:06:40.054 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 21:06:40 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:cf+EXmBrXvcIO7yhSlkD+rh/yvQ=', 'Content-Length': '0'} 2023-05-31T21:06:40.056 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 21:04:02 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000573fceea5f552f60-006477b6e0-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 21:06:40 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:06:40.057 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj44 2023-05-31T21:06:40.057 DEBUG:boto:path=/pklxkd-62/obj44 2023-05-31T21:06:40.058 DEBUG:boto:auth_path=/pklxkd-62/obj44 2023-05-31T21:06:40.058 DEBUG:boto:Method: GET 2023-05-31T21:06:40.059 DEBUG:boto:Path: /pklxkd-62/obj44 2023-05-31T21:06:40.059 DEBUG:boto:Data: 2023-05-31T21:06:40.059 DEBUG:boto:Headers: {} 2023-05-31T21:06:40.060 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T21:06:40.060 DEBUG:boto:Port: 8001 2023-05-31T21:06:40.061 DEBUG:boto:Params: {} 2023-05-31T21:06:40.061 DEBUG:boto:Token: None 2023-05-31T21:06:40.061 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 21:06:40 GMT /pklxkd-62/obj44 2023-05-31T21:06:40.062 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:Pqv9glz4P/pka4lcyDcg3l3uBCE= 2023-05-31T21:06:40.062 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 21:06:40 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:Pqv9glz4P/pka4lcyDcg3l3uBCE=', 'Content-Length': '0'} 2023-05-31T21:06:40.064 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 21:04:02 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000092da6ab737c121b8-006477b6e0-1fb6-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 21:06:40 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:06:40.065 DEBUG:boto:path=/pklxkd-62/obj44 2023-05-31T21:06:40.066 DEBUG:boto:auth_path=/pklxkd-62/obj44 2023-05-31T21:06:40.066 DEBUG:boto:Method: GET 2023-05-31T21:06:40.066 DEBUG:boto:Path: /pklxkd-62/obj44 2023-05-31T21:06:40.067 DEBUG:boto:Data: 2023-05-31T21:06:40.067 DEBUG:boto:Headers: {} 2023-05-31T21:06:40.067 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T21:06:40.068 DEBUG:boto:Port: 8000 2023-05-31T21:06:40.068 DEBUG:boto:Params: {} 2023-05-31T21:06:40.069 DEBUG:boto:Token: None 2023-05-31T21:06:40.069 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 21:06:40 GMT /pklxkd-62/obj44 2023-05-31T21:06:40.069 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:Pqv9glz4P/pka4lcyDcg3l3uBCE= 2023-05-31T21:06:40.070 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 21:06:40 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:Pqv9glz4P/pka4lcyDcg3l3uBCE=', 'Content-Length': '0'} 2023-05-31T21:06:40.072 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 21:04:02 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000bb93331e8bb82fe8-006477b6e0-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 21:06:40 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:06:40.072 DEBUG:boto:path=/pklxkd-62/ 2023-05-31T21:06:40.073 DEBUG:boto:auth_path=/pklxkd-62/ 2023-05-31T21:06:40.073 DEBUG:boto:path=/pklxkd-62/?prefix=obj44 2023-05-31T21:06:40.073 DEBUG:boto:auth_path=/pklxkd-62/?prefix=obj44 2023-05-31T21:06:40.074 DEBUG:boto:Method: GET 2023-05-31T21:06:40.074 DEBUG:boto:Path: /pklxkd-62/?prefix=obj44 2023-05-31T21:06:40.074 DEBUG:boto:Data: 2023-05-31T21:06:40.075 DEBUG:boto:Headers: {} 2023-05-31T21:06:40.075 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T21:06:40.076 DEBUG:boto:Port: 8001 2023-05-31T21:06:40.076 DEBUG:boto:Params: {} 2023-05-31T21:06:40.076 DEBUG:boto:Token: None 2023-05-31T21:06:40.077 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 21:06:40 GMT /pklxkd-62/ 2023-05-31T21:06:40.077 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:Zw9HJEYqh8L1MEOneHxUYI8gQCw= 2023-05-31T21:06:40.078 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 21:06:40 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:Zw9HJEYqh8L1MEOneHxUYI8gQCw=', 'Content-Length': '0'} 2023-05-31T21:06:40.080 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000666a6f5211bcd4ff-006477b6e0-1fb6-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 21:06:40 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:06:40.124 DEBUG:boto:b'pklxkd-62obj441000falseobj442023-05-31T21:04:02.730Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T21:06:40.125 DEBUG:boto:path=/pklxkd-62/ 2023-05-31T21:06:40.126 DEBUG:boto:auth_path=/pklxkd-62/ 2023-05-31T21:06:40.126 DEBUG:boto:path=/pklxkd-62/?prefix=obj44 2023-05-31T21:06:40.127 DEBUG:boto:auth_path=/pklxkd-62/?prefix=obj44 2023-05-31T21:06:40.128 DEBUG:boto:Method: GET 2023-05-31T21:06:40.128 DEBUG:boto:Path: /pklxkd-62/?prefix=obj44 2023-05-31T21:06:40.129 DEBUG:boto:Data: 2023-05-31T21:06:40.129 DEBUG:boto:Headers: {} 2023-05-31T21:06:40.130 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T21:06:40.130 DEBUG:boto:Port: 8000 2023-05-31T21:06:40.131 DEBUG:boto:Params: {} 2023-05-31T21:06:40.131 DEBUG:boto:Token: None 2023-05-31T21:06:40.131 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 21:06:40 GMT /pklxkd-62/ 2023-05-31T21:06:40.132 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:Zw9HJEYqh8L1MEOneHxUYI8gQCw= 2023-05-31T21:06:40.132 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 21:06:40 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:Zw9HJEYqh8L1MEOneHxUYI8gQCw=', 'Content-Length': '0'} 2023-05-31T21:06:40.135 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000008b8d74b1b99e746-006477b6e0-1630-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 21:06:40 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:06:40.135 DEBUG:boto:b'pklxkd-62obj441000falseobj442023-05-31T21:04:02.730Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T21:06:40.136 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj55 2023-05-31T21:06:40.136 DEBUG:boto:path=/pklxkd-62/obj55 2023-05-31T21:06:40.137 DEBUG:boto:auth_path=/pklxkd-62/obj55 2023-05-31T21:06:40.137 DEBUG:boto:path=/pklxkd-62/obj55?versionId=null 2023-05-31T21:06:40.137 DEBUG:boto:auth_path=/pklxkd-62/obj55?versionId=null 2023-05-31T21:06:40.138 DEBUG:boto:Method: GET 2023-05-31T21:06:40.138 DEBUG:boto:Path: /pklxkd-62/obj55?versionId=null 2023-05-31T21:06:40.138 DEBUG:boto:Data: 2023-05-31T21:06:40.139 DEBUG:boto:Headers: {} 2023-05-31T21:06:40.139 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T21:06:40.139 DEBUG:boto:Port: 8001 2023-05-31T21:06:40.140 DEBUG:boto:Params: {} 2023-05-31T21:06:40.140 DEBUG:boto:Token: None 2023-05-31T21:06:40.141 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 21:06:40 GMT /pklxkd-62/obj55?versionId=null 2023-05-31T21:06:40.141 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:T8L24sdvcjky0XP9hhEmHa0W2iQ= 2023-05-31T21:06:40.142 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 21:06:40 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:T8L24sdvcjky0XP9hhEmHa0W2iQ=', 'Content-Length': '0'} 2023-05-31T21:06:40.144 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 21:04:02 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009e7e7c268a1a4664-006477b6e0-1fb6-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 21:06:40 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:06:40.144 DEBUG:boto:path=/pklxkd-62/obj55 2023-05-31T21:06:40.145 DEBUG:boto:auth_path=/pklxkd-62/obj55 2023-05-31T21:06:40.145 DEBUG:boto:path=/pklxkd-62/obj55?versionId=null 2023-05-31T21:06:40.146 DEBUG:boto:auth_path=/pklxkd-62/obj55?versionId=null 2023-05-31T21:06:40.146 DEBUG:boto:Method: GET 2023-05-31T21:06:40.146 DEBUG:boto:Path: /pklxkd-62/obj55?versionId=null 2023-05-31T21:06:40.147 DEBUG:boto:Data: 2023-05-31T21:06:40.147 DEBUG:boto:Headers: {} 2023-05-31T21:06:40.147 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T21:06:40.148 DEBUG:boto:Port: 8000 2023-05-31T21:06:40.148 DEBUG:boto:Params: {} 2023-05-31T21:06:40.148 DEBUG:boto:Token: None 2023-05-31T21:06:40.149 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 21:06:40 GMT /pklxkd-62/obj55?versionId=null 2023-05-31T21:06:40.149 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:T8L24sdvcjky0XP9hhEmHa0W2iQ= 2023-05-31T21:06:40.150 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 21:06:40 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:T8L24sdvcjky0XP9hhEmHa0W2iQ=', 'Content-Length': '0'} 2023-05-31T21:06:40.152 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 21:04:02 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000eb3f4a85e371e02c-006477b6e0-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 21:06:40 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:06:40.152 DEBUG:boto:path=/pklxkd-62/obj55 2023-05-31T21:06:40.153 DEBUG:boto:auth_path=/pklxkd-62/obj55 2023-05-31T21:06:40.153 DEBUG:boto:path=/pklxkd-62/obj55?versionId=null 2023-05-31T21:06:40.154 DEBUG:boto:auth_path=/pklxkd-62/obj55?versionId=null 2023-05-31T21:06:40.154 DEBUG:boto:Method: HEAD 2023-05-31T21:06:40.154 DEBUG:boto:Path: /pklxkd-62/obj55?versionId=null 2023-05-31T21:06:40.155 DEBUG:boto:Data: 2023-05-31T21:06:40.155 DEBUG:boto:Headers: {} 2023-05-31T21:06:40.155 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T21:06:40.156 DEBUG:boto:Port: 8001 2023-05-31T21:06:40.156 DEBUG:boto:Params: {} 2023-05-31T21:06:40.156 DEBUG:boto:Token: None 2023-05-31T21:06:40.157 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 21:06:40 GMT /pklxkd-62/obj55?versionId=null 2023-05-31T21:06:40.157 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:CwkJ6322ijGZf+rPtEJ8+ai58sc= 2023-05-31T21:06:40.158 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 21:06:40 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:CwkJ6322ijGZf+rPtEJ8+ai58sc=', 'Content-Length': '0'} 2023-05-31T21:06:40.159 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 21:04:02 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000015f3d9fe210a5596-006477b6e0-1fb6-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 21:06:40 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:06:40.160 DEBUG:boto:path=/pklxkd-62/obj55 2023-05-31T21:06:40.160 DEBUG:boto:auth_path=/pklxkd-62/obj55 2023-05-31T21:06:40.161 DEBUG:boto:path=/pklxkd-62/obj55?versionId=null 2023-05-31T21:06:40.161 DEBUG:boto:auth_path=/pklxkd-62/obj55?versionId=null 2023-05-31T21:06:40.161 DEBUG:boto:Method: HEAD 2023-05-31T21:06:40.162 DEBUG:boto:Path: /pklxkd-62/obj55?versionId=null 2023-05-31T21:06:40.162 DEBUG:boto:Data: 2023-05-31T21:06:40.162 DEBUG:boto:Headers: {} 2023-05-31T21:06:40.163 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T21:06:40.163 DEBUG:boto:Port: 8000 2023-05-31T21:06:40.163 DEBUG:boto:Params: {} 2023-05-31T21:06:40.164 DEBUG:boto:Token: None 2023-05-31T21:06:40.164 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 21:06:40 GMT /pklxkd-62/obj55?versionId=null 2023-05-31T21:06:40.165 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:CwkJ6322ijGZf+rPtEJ8+ai58sc= 2023-05-31T21:06:40.165 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 21:06:40 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:CwkJ6322ijGZf+rPtEJ8+ai58sc=', 'Content-Length': '0'} 2023-05-31T21:06:40.167 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 21:04:02 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e1317dbc6ba94577-006477b6e0-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 21:06:40 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:06:40.168 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj55 2023-05-31T21:06:40.168 DEBUG:boto:path=/pklxkd-62/obj55 2023-05-31T21:06:40.168 DEBUG:boto:auth_path=/pklxkd-62/obj55 2023-05-31T21:06:40.169 DEBUG:boto:path=/pklxkd-62/obj55?versionId=null 2023-05-31T21:06:40.169 DEBUG:boto:auth_path=/pklxkd-62/obj55?versionId=null 2023-05-31T21:06:40.169 DEBUG:boto:Method: GET 2023-05-31T21:06:40.170 DEBUG:boto:Path: /pklxkd-62/obj55?versionId=null 2023-05-31T21:06:40.170 DEBUG:boto:Data: 2023-05-31T21:06:40.171 DEBUG:boto:Headers: {} 2023-05-31T21:06:40.171 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T21:06:40.171 DEBUG:boto:Port: 8001 2023-05-31T21:06:40.172 DEBUG:boto:Params: {} 2023-05-31T21:06:40.172 DEBUG:boto:Token: None 2023-05-31T21:06:40.173 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 21:06:40 GMT /pklxkd-62/obj55?versionId=null 2023-05-31T21:06:40.173 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:T8L24sdvcjky0XP9hhEmHa0W2iQ= 2023-05-31T21:06:40.174 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 21:06:40 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:T8L24sdvcjky0XP9hhEmHa0W2iQ=', 'Content-Length': '0'} 2023-05-31T21:06:40.175 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 21:04:02 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000847cdce9b125ab6f-006477b6e0-1fb6-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 21:06:40 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:06:40.176 DEBUG:boto:path=/pklxkd-62/obj55 2023-05-31T21:06:40.176 DEBUG:boto:auth_path=/pklxkd-62/obj55 2023-05-31T21:06:40.177 DEBUG:boto:path=/pklxkd-62/obj55?versionId=null 2023-05-31T21:06:40.177 DEBUG:boto:auth_path=/pklxkd-62/obj55?versionId=null 2023-05-31T21:06:40.177 DEBUG:boto:Method: GET 2023-05-31T21:06:40.178 DEBUG:boto:Path: /pklxkd-62/obj55?versionId=null 2023-05-31T21:06:40.178 DEBUG:boto:Data: 2023-05-31T21:06:40.179 DEBUG:boto:Headers: {} 2023-05-31T21:06:40.179 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T21:06:40.179 DEBUG:boto:Port: 8000 2023-05-31T21:06:40.180 DEBUG:boto:Params: {} 2023-05-31T21:06:40.180 DEBUG:boto:Token: None 2023-05-31T21:06:40.180 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 21:06:40 GMT /pklxkd-62/obj55?versionId=null 2023-05-31T21:06:40.181 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:T8L24sdvcjky0XP9hhEmHa0W2iQ= 2023-05-31T21:06:40.181 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 21:06:40 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:T8L24sdvcjky0XP9hhEmHa0W2iQ=', 'Content-Length': '0'} 2023-05-31T21:06:40.183 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 21:04:02 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000060a77881a314c551-006477b6e0-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 21:06:40 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:06:40.184 DEBUG:boto:path=/pklxkd-62/ 2023-05-31T21:06:40.184 DEBUG:boto:auth_path=/pklxkd-62/ 2023-05-31T21:06:40.184 DEBUG:boto:path=/pklxkd-62/?prefix=obj55 2023-05-31T21:06:40.185 DEBUG:boto:auth_path=/pklxkd-62/?prefix=obj55 2023-05-31T21:06:40.185 DEBUG:boto:Method: GET 2023-05-31T21:06:40.185 DEBUG:boto:Path: /pklxkd-62/?prefix=obj55 2023-05-31T21:06:40.186 DEBUG:boto:Data: 2023-05-31T21:06:40.186 DEBUG:boto:Headers: {} 2023-05-31T21:06:40.186 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T21:06:40.187 DEBUG:boto:Port: 8001 2023-05-31T21:06:40.187 DEBUG:boto:Params: {} 2023-05-31T21:06:40.188 DEBUG:boto:Token: None 2023-05-31T21:06:40.188 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 21:06:40 GMT /pklxkd-62/ 2023-05-31T21:06:40.188 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:Zw9HJEYqh8L1MEOneHxUYI8gQCw= 2023-05-31T21:06:40.189 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 21:06:40 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:Zw9HJEYqh8L1MEOneHxUYI8gQCw=', 'Content-Length': '0'} 2023-05-31T21:06:40.191 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000005e45ff7b91f9b524-006477b6e0-1fb6-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 21:06:40 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:06:40.193 DEBUG:boto:b'pklxkd-62obj551000falseobj552023-05-31T21:04:02.751Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T21:06:40.194 DEBUG:boto:path=/pklxkd-62/ 2023-05-31T21:06:40.194 DEBUG:boto:auth_path=/pklxkd-62/ 2023-05-31T21:06:40.195 DEBUG:boto:path=/pklxkd-62/?prefix=obj55 2023-05-31T21:06:40.196 DEBUG:boto:auth_path=/pklxkd-62/?prefix=obj55 2023-05-31T21:06:40.197 DEBUG:boto:Method: GET 2023-05-31T21:06:40.197 DEBUG:boto:Path: /pklxkd-62/?prefix=obj55 2023-05-31T21:06:40.198 DEBUG:boto:Data: 2023-05-31T21:06:40.199 DEBUG:boto:Headers: {} 2023-05-31T21:06:40.199 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T21:06:40.200 DEBUG:boto:Port: 8000 2023-05-31T21:06:40.201 DEBUG:boto:Params: {} 2023-05-31T21:06:40.202 DEBUG:boto:Token: None 2023-05-31T21:06:40.202 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 21:06:40 GMT /pklxkd-62/ 2023-05-31T21:06:40.203 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:Zw9HJEYqh8L1MEOneHxUYI8gQCw= 2023-05-31T21:06:40.204 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 21:06:40 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:Zw9HJEYqh8L1MEOneHxUYI8gQCw=', 'Content-Length': '0'} 2023-05-31T21:06:40.207 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e846ab00cdf27008-006477b6e0-1630-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 21:06:40 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:06:40.208 DEBUG:boto:b'pklxkd-62obj551000falseobj552023-05-31T21:04:02.751Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T21:06:40.209 DEBUG:boto:path=/pklxkd-62/obj55 2023-05-31T21:06:40.210 DEBUG:boto:auth_path=/pklxkd-62/obj55 2023-05-31T21:06:40.210 DEBUG:boto:Method: HEAD 2023-05-31T21:06:40.211 DEBUG:boto:Path: /pklxkd-62/obj55 2023-05-31T21:06:40.212 DEBUG:boto:Data: 2023-05-31T21:06:40.212 DEBUG:boto:Headers: {} 2023-05-31T21:06:40.213 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T21:06:40.214 DEBUG:boto:Port: 8001 2023-05-31T21:06:40.214 DEBUG:boto:Params: {} 2023-05-31T21:06:40.215 DEBUG:boto:Token: None 2023-05-31T21:06:40.216 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 21:06:40 GMT /pklxkd-62/obj55 2023-05-31T21:06:40.217 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:BbZOef9Lr/Ws8/6xpL+4LlqnccY= 2023-05-31T21:06:40.217 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 21:06:40 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:BbZOef9Lr/Ws8/6xpL+4LlqnccY=', 'Content-Length': '0'} 2023-05-31T21:06:40.220 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 21:04:02 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004d8a3383a25084a4-006477b6e0-1fb6-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 21:06:40 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:06:40.221 DEBUG:boto:path=/pklxkd-62/obj55 2023-05-31T21:06:40.221 DEBUG:boto:auth_path=/pklxkd-62/obj55 2023-05-31T21:06:40.222 DEBUG:boto:Method: HEAD 2023-05-31T21:06:40.223 DEBUG:boto:Path: /pklxkd-62/obj55 2023-05-31T21:06:40.224 DEBUG:boto:Data: 2023-05-31T21:06:40.224 DEBUG:boto:Headers: {} 2023-05-31T21:06:40.225 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T21:06:40.226 DEBUG:boto:Port: 8000 2023-05-31T21:06:40.226 DEBUG:boto:Params: {} 2023-05-31T21:06:40.227 DEBUG:boto:Token: None 2023-05-31T21:06:40.228 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 21:06:40 GMT /pklxkd-62/obj55 2023-05-31T21:06:40.228 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:BbZOef9Lr/Ws8/6xpL+4LlqnccY= 2023-05-31T21:06:40.229 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 21:06:40 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:BbZOef9Lr/Ws8/6xpL+4LlqnccY=', 'Content-Length': '0'} 2023-05-31T21:06:40.231 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 21:04:02 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009245fc861875f250-006477b6e0-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 21:06:40 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:06:40.232 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj55 2023-05-31T21:06:40.233 DEBUG:boto:path=/pklxkd-62/obj55 2023-05-31T21:06:40.234 DEBUG:boto:auth_path=/pklxkd-62/obj55 2023-05-31T21:06:40.234 DEBUG:boto:Method: GET 2023-05-31T21:06:40.235 DEBUG:boto:Path: /pklxkd-62/obj55 2023-05-31T21:06:40.236 DEBUG:boto:Data: 2023-05-31T21:06:40.237 DEBUG:boto:Headers: {} 2023-05-31T21:06:40.237 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T21:06:40.238 DEBUG:boto:Port: 8001 2023-05-31T21:06:40.239 DEBUG:boto:Params: {} 2023-05-31T21:06:40.239 DEBUG:boto:Token: None 2023-05-31T21:06:40.240 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 21:06:40 GMT /pklxkd-62/obj55 2023-05-31T21:06:40.241 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:YIzxcJFsAjcj73JKPJY4gKd9Mzs= 2023-05-31T21:06:40.242 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 21:06:40 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:YIzxcJFsAjcj73JKPJY4gKd9Mzs=', 'Content-Length': '0'} 2023-05-31T21:06:40.244 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 21:04:02 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000046a88fa2ae3afb8b-006477b6e0-1fb6-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 21:06:40 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:06:40.245 DEBUG:boto:path=/pklxkd-62/obj55 2023-05-31T21:06:40.246 DEBUG:boto:auth_path=/pklxkd-62/obj55 2023-05-31T21:06:40.247 DEBUG:boto:Method: GET 2023-05-31T21:06:40.247 DEBUG:boto:Path: /pklxkd-62/obj55 2023-05-31T21:06:40.248 DEBUG:boto:Data: 2023-05-31T21:06:40.249 DEBUG:boto:Headers: {} 2023-05-31T21:06:40.249 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T21:06:40.250 DEBUG:boto:Port: 8000 2023-05-31T21:06:40.251 DEBUG:boto:Params: {} 2023-05-31T21:06:40.252 DEBUG:boto:Token: None 2023-05-31T21:06:40.252 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 21:06:40 GMT /pklxkd-62/obj55 2023-05-31T21:06:40.252 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:YIzxcJFsAjcj73JKPJY4gKd9Mzs= 2023-05-31T21:06:40.253 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 21:06:40 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:YIzxcJFsAjcj73JKPJY4gKd9Mzs=', 'Content-Length': '0'} 2023-05-31T21:06:40.254 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 21:04:02 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000865c8a283174bcec-006477b6e0-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 21:06:40 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:06:40.255 DEBUG:boto:path=/pklxkd-62/ 2023-05-31T21:06:40.256 DEBUG:boto:auth_path=/pklxkd-62/ 2023-05-31T21:06:40.256 DEBUG:boto:path=/pklxkd-62/?prefix=obj55 2023-05-31T21:06:40.256 DEBUG:boto:auth_path=/pklxkd-62/?prefix=obj55 2023-05-31T21:06:40.257 DEBUG:boto:Method: GET 2023-05-31T21:06:40.257 DEBUG:boto:Path: /pklxkd-62/?prefix=obj55 2023-05-31T21:06:40.257 DEBUG:boto:Data: 2023-05-31T21:06:40.258 DEBUG:boto:Headers: {} 2023-05-31T21:06:40.258 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T21:06:40.258 DEBUG:boto:Port: 8001 2023-05-31T21:06:40.259 DEBUG:boto:Params: {} 2023-05-31T21:06:40.259 DEBUG:boto:Token: None 2023-05-31T21:06:40.260 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 21:06:40 GMT /pklxkd-62/ 2023-05-31T21:06:40.260 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:Zw9HJEYqh8L1MEOneHxUYI8gQCw= 2023-05-31T21:06:40.260 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 21:06:40 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:Zw9HJEYqh8L1MEOneHxUYI8gQCw=', 'Content-Length': '0'} 2023-05-31T21:06:40.263 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a1518eb0dd9a432d-006477b6e0-1fb6-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 21:06:40 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:06:40.263 DEBUG:boto:b'pklxkd-62obj551000falseobj552023-05-31T21:04:02.751Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T21:06:40.264 DEBUG:boto:path=/pklxkd-62/ 2023-05-31T21:06:40.264 DEBUG:boto:auth_path=/pklxkd-62/ 2023-05-31T21:06:40.265 DEBUG:boto:path=/pklxkd-62/?prefix=obj55 2023-05-31T21:06:40.265 DEBUG:boto:auth_path=/pklxkd-62/?prefix=obj55 2023-05-31T21:06:40.265 DEBUG:boto:Method: GET 2023-05-31T21:06:40.266 DEBUG:boto:Path: /pklxkd-62/?prefix=obj55 2023-05-31T21:06:40.266 DEBUG:boto:Data: 2023-05-31T21:06:40.266 DEBUG:boto:Headers: {} 2023-05-31T21:06:40.267 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T21:06:40.267 DEBUG:boto:Port: 8000 2023-05-31T21:06:40.267 DEBUG:boto:Params: {} 2023-05-31T21:06:40.268 DEBUG:boto:Token: None 2023-05-31T21:06:40.268 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 21:06:40 GMT /pklxkd-62/ 2023-05-31T21:06:40.269 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:Zw9HJEYqh8L1MEOneHxUYI8gQCw= 2023-05-31T21:06:40.269 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 21:06:40 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:Zw9HJEYqh8L1MEOneHxUYI8gQCw=', 'Content-Length': '0'} 2023-05-31T21:06:40.271 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ebcd6f0149335d4b-006477b6e0-1630-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 21:06:40 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:06:40.272 DEBUG:boto:b'pklxkd-62obj551000falseobj552023-05-31T21:04:02.751Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T21:06:40.273 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=pklxkd-62 zones={a2, a1} 2023-05-31T21:06:40.273 DEBUG:rgw_multi.tests:data checkpoint: source=a1 target=a2 2023-05-31T21:06:40.273 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['datalog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-31T21:06:40.274 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'datalog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-31T21:06:40.274 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 datalog status --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-05-31T21:06:40.329 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2023-05-31T21:06:40.329 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2023-05-31T21:06:40.385 INFO:teuthology.orchestra.run.smithi099.stdout:[ 2023-05-31T21:06:40.386 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T21:06:40.386 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "00000000000000000000:00000000000000031929", 2023-05-31T21:06:40.386 INFO:teuthology.orchestra.run.smithi099.stdout: "last_update": "2023-05-31T21:04:14.651540Z" 2023-05-31T21:06:40.386 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:06:40.386 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T21:06:40.386 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "00000000000000000000:00000000000000022723", 2023-05-31T21:06:40.387 INFO:teuthology.orchestra.run.smithi099.stdout: "last_update": "2023-05-31T20:59:50.645856Z" 2023-05-31T21:06:40.387 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:06:40.387 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T21:06:40.387 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "00000000000000000000:00000000000000037390", 2023-05-31T21:06:40.387 INFO:teuthology.orchestra.run.smithi099.stdout: "last_update": "2023-05-31T21:04:02.734461Z" 2023-05-31T21:06:40.387 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:06:40.387 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T21:06:40.388 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "00000000000000000000:00000000000000032581", 2023-05-31T21:06:40.388 INFO:teuthology.orchestra.run.smithi099.stdout: "last_update": "2023-05-31T21:04:14.653394Z" 2023-05-31T21:06:40.388 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T21:06:40.388 INFO:teuthology.orchestra.run.smithi099.stdout:] 2023-05-31T21:06:40.389 DEBUG:rgw_multi.tests:data markers for zone=a1 markers={0: '00000000000000000000:00000000000000031929', 1: '00000000000000000000:00000000000000022723', 2: '00000000000000000000:00000000000000037390', 3: '00000000000000000000:00000000000000032581'} 2023-05-31T21:06:40.389 INFO:rgw_multi.tests:starting data checkpoint for target_zone=a2 source_zone=a1 2023-05-31T21:06:40.390 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-31T21:06:40.390 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-31T21:06:40.391 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-05-31T21:06:40.443 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2023-05-31T21:06:40.443 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2023-05-31T21:06:40.505 INFO:teuthology.orchestra.run.smithi099.stdout:{ 2023-05-31T21:06:40.505 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_status": { 2023-05-31T21:06:40.506 INFO:teuthology.orchestra.run.smithi099.stdout: "info": { 2023-05-31T21:06:40.506 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "sync", 2023-05-31T21:06:40.506 INFO:teuthology.orchestra.run.smithi099.stdout: "num_shards": 4, 2023-05-31T21:06:40.506 INFO:teuthology.orchestra.run.smithi099.stdout: "instance_id": 6448074114480490995 2023-05-31T21:06:40.506 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:06:40.506 INFO:teuthology.orchestra.run.smithi099.stdout: "markers": [ 2023-05-31T21:06:40.506 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T21:06:40.507 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 0, 2023-05-31T21:06:40.507 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T21:06:40.507 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "incremental-sync", 2023-05-31T21:06:40.507 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "00000000000000000000:00000000000000031929", 2023-05-31T21:06:40.507 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T21:06:40.507 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2023-05-31T21:06:40.507 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T21:06:40.508 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2023-05-31T21:04:14.651540Z" 2023-05-31T21:06:40.508 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T21:06:40.508 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:06:40.508 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T21:06:40.508 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 1, 2023-05-31T21:06:40.508 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T21:06:40.509 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "incremental-sync", 2023-05-31T21:06:40.509 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "00000000000000000000:00000000000000022723", 2023-05-31T21:06:40.509 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T21:06:40.509 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2023-05-31T21:06:40.509 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T21:06:40.509 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2023-05-31T20:59:50.645856Z" 2023-05-31T21:06:40.509 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T21:06:40.510 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:06:40.510 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T21:06:40.510 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 2, 2023-05-31T21:06:40.510 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T21:06:40.510 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "full-sync", 2023-05-31T21:06:40.510 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "pklxkd-51:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.16:0", 2023-05-31T21:06:40.510 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "00000000000000000000:00000000000000037390", 2023-05-31T21:06:40.511 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 53, 2023-05-31T21:06:40.511 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 42, 2023-05-31T21:06:40.511 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2023-05-31T21:04:02.734461Z" 2023-05-31T21:06:40.511 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T21:06:40.511 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:06:40.511 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T21:06:40.511 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 3, 2023-05-31T21:06:40.512 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T21:06:40.512 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "incremental-sync", 2023-05-31T21:06:40.512 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "00000000000000000000:00000000000000032581", 2023-05-31T21:06:40.512 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T21:06:40.512 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 9, 2023-05-31T21:06:40.512 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T21:06:40.513 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2023-05-31T21:04:14.653394Z" 2023-05-31T21:06:40.513 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T21:06:40.513 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T21:06:40.513 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2023-05-31T21:06:40.513 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:06:40.513 INFO:teuthology.orchestra.run.smithi099.stdout: "full_sync": { 2023-05-31T21:06:40.513 INFO:teuthology.orchestra.run.smithi099.stdout: "total": 62, 2023-05-31T21:06:40.514 INFO:teuthology.orchestra.run.smithi099.stdout: "complete": 51 2023-05-31T21:06:40.514 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:06:40.514 INFO:teuthology.orchestra.run.smithi099.stdout: "current_time": "2023-05-31T21:06:40Z" 2023-05-31T21:06:40.514 INFO:teuthology.orchestra.run.smithi099.stdout:} 2023-05-31T21:06:40.515 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 6448074114480490995 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000031929", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-05-31T21:04:14.651540Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000022723", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-05-31T20:59:50.645856Z" } }, { "key": 2, "val": { "status": "full-sync", "marker": "pklxkd-51:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.16:0", "next_step_marker": "00000000000000000000:00000000000000037390", "total_entries": 53, "pos": 42, "timestamp": "2023-05-31T21:04:02.734461Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000032581", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2023-05-31T21:04:14.653394Z" } } ] }, "full_sync": { "total": 62, "complete": 51 }, "current_time": "2023-05-31T21:06:40Z" } 2023-05-31T21:06:40.515 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000031929', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-05-31T21:04:14.651540Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000022723', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-05-31T20:59:50.645856Z'}}, {'key': 2, 'val': {'status': 'full-sync', 'marker': 'pklxkd-51:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.16:0', 'next_step_marker': '00000000000000000000:00000000000000037390', 'total_entries': 53, 'pos': 42, 'timestamp': '2023-05-31T21:04:02.734461Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000032581', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2023-05-31T21:04:14.653394Z'}}] 2023-05-31T21:06:40.516 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000031929', 1: '00000000000000000000:00000000000000022723', 2: '00000000000000000000:00000000000000037390', 3: '00000000000000000000:00000000000000032581'} 2023-05-31T21:06:40.516 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000031929', 1: '00000000000000000000:00000000000000022723', 2: 'pklxkd-51:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.16:0', 3: '00000000000000000000:00000000000000032581'} 2023-05-31T21:06:40.516 INFO:rgw_multi.tests:finished data checkpoint for target_zone=a2 source_zone=a1 2023-05-31T21:06:40.517 DEBUG:rgw_multi.tests:data checkpoint: source=a2 target=a1 2023-05-31T21:06:40.517 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['datalog', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-31T21:06:40.517 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'datalog', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-31T21:06:40.518 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 datalog status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-05-31T21:06:40.565 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2023-05-31T21:06:40.565 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2023-05-31T21:06:40.634 INFO:teuthology.orchestra.run.smithi099.stdout:[ 2023-05-31T21:06:40.634 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T21:06:40.635 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "00000000000000000000:00000000000000010833", 2023-05-31T21:06:40.635 INFO:teuthology.orchestra.run.smithi099.stdout: "last_update": "2023-05-31T21:06:33.196876Z" 2023-05-31T21:06:40.635 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:06:40.635 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T21:06:40.635 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "00000000000000000000:00000000000000005024", 2023-05-31T21:06:40.635 INFO:teuthology.orchestra.run.smithi099.stdout: "last_update": "2023-05-31T21:01:37.289790Z" 2023-05-31T21:06:40.636 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:06:40.636 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T21:06:40.636 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "00000000000000000000:00000000000000014745", 2023-05-31T21:06:40.636 INFO:teuthology.orchestra.run.smithi099.stdout: "last_update": "2023-05-31T21:05:37.374744Z" 2023-05-31T21:06:40.636 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:06:40.636 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T21:06:40.637 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "00000000000000000000:00000000000000014184", 2023-05-31T21:06:40.637 INFO:teuthology.orchestra.run.smithi099.stdout: "last_update": "2023-05-31T21:05:53.064598Z" 2023-05-31T21:06:40.637 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T21:06:40.637 INFO:teuthology.orchestra.run.smithi099.stdout:] 2023-05-31T21:06:40.638 DEBUG:rgw_multi.tests:data markers for zone=a2 markers={0: '00000000000000000000:00000000000000010833', 1: '00000000000000000000:00000000000000005024', 2: '00000000000000000000:00000000000000014745', 3: '00000000000000000000:00000000000000014184'} 2023-05-31T21:06:40.638 INFO:rgw_multi.tests:starting data checkpoint for target_zone=a1 source_zone=a2 2023-05-31T21:06:40.639 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-31T21:06:40.639 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-31T21:06:40.640 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --source-zone a2 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-05-31T21:06:40.690 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2023-05-31T21:06:40.690 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2023-05-31T21:06:40.745 INFO:teuthology.orchestra.run.smithi099.stdout:{ 2023-05-31T21:06:40.745 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_status": { 2023-05-31T21:06:40.745 INFO:teuthology.orchestra.run.smithi099.stdout: "info": { 2023-05-31T21:06:40.746 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "sync", 2023-05-31T21:06:40.746 INFO:teuthology.orchestra.run.smithi099.stdout: "num_shards": 4, 2023-05-31T21:06:40.746 INFO:teuthology.orchestra.run.smithi099.stdout: "instance_id": 17941554979570305020 2023-05-31T21:06:40.746 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:06:40.746 INFO:teuthology.orchestra.run.smithi099.stdout: "markers": [ 2023-05-31T21:06:40.746 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T21:06:40.747 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 0, 2023-05-31T21:06:40.747 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T21:06:40.747 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "incremental-sync", 2023-05-31T21:06:40.747 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "00000000000000000000:00000000000000010833", 2023-05-31T21:06:40.747 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T21:06:40.747 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2023-05-31T21:06:40.747 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T21:06:40.748 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2023-05-31T21:06:33.196876Z" 2023-05-31T21:06:40.748 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T21:06:40.748 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:06:40.748 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T21:06:40.748 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 1, 2023-05-31T21:06:40.748 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T21:06:40.749 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "incremental-sync", 2023-05-31T21:06:40.749 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "00000000000000000000:00000000000000005024", 2023-05-31T21:06:40.749 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T21:06:40.749 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2023-05-31T21:06:40.749 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T21:06:40.749 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2023-05-31T21:01:37.289790Z" 2023-05-31T21:06:40.749 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T21:06:40.750 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:06:40.750 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T21:06:40.750 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 2, 2023-05-31T21:06:40.750 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T21:06:40.750 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "incremental-sync", 2023-05-31T21:06:40.750 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "00000000000000000000:00000000000000014745", 2023-05-31T21:06:40.751 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T21:06:40.751 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2023-05-31T21:06:40.751 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T21:06:40.751 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2023-05-31T21:05:37.374744Z" 2023-05-31T21:06:40.751 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T21:06:40.751 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:06:40.751 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T21:06:40.752 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 3, 2023-05-31T21:06:40.752 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T21:06:40.752 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "incremental-sync", 2023-05-31T21:06:40.752 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "00000000000000000000:00000000000000014184", 2023-05-31T21:06:40.752 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T21:06:40.752 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2023-05-31T21:06:40.752 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T21:06:40.753 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2023-05-31T21:05:53.064598Z" 2023-05-31T21:06:40.753 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T21:06:40.753 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T21:06:40.753 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2023-05-31T21:06:40.753 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:06:40.753 INFO:teuthology.orchestra.run.smithi099.stdout: "full_sync": { 2023-05-31T21:06:40.753 INFO:teuthology.orchestra.run.smithi099.stdout: "total": 0, 2023-05-31T21:06:40.754 INFO:teuthology.orchestra.run.smithi099.stdout: "complete": 0 2023-05-31T21:06:40.754 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:06:40.754 INFO:teuthology.orchestra.run.smithi099.stdout: "current_time": "2023-05-31T21:06:40Z" 2023-05-31T21:06:40.754 INFO:teuthology.orchestra.run.smithi099.stdout:} 2023-05-31T21:06:40.755 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 17941554979570305020 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000010833", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-05-31T21:06:33.196876Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000005024", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-05-31T21:01:37.289790Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000014745", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-05-31T21:05:37.374744Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000014184", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-05-31T21:05:53.064598Z" } } ] }, "full_sync": { "total": 0, "complete": 0 }, "current_time": "2023-05-31T21:06:40Z" } 2023-05-31T21:06:40.755 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000010833', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-05-31T21:06:33.196876Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000005024', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-05-31T21:01:37.289790Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000014745', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-05-31T21:05:37.374744Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000014184', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-05-31T21:05:53.064598Z'}}] 2023-05-31T21:06:40.756 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000010833', 1: '00000000000000000000:00000000000000005024', 2: '00000000000000000000:00000000000000014745', 3: '00000000000000000000:00000000000000014184'} 2023-05-31T21:06:40.756 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000010833', 1: '00000000000000000000:00000000000000005024', 2: '00000000000000000000:00000000000000014745', 3: '00000000000000000000:00000000000000014184'} 2023-05-31T21:06:40.756 INFO:rgw_multi.tests:finished data checkpoint for target_zone=a1 source_zone=a2 2023-05-31T21:06:40.757 INFO:tasks.rgw_multisite_tests:rgw_multi.tests.test_bucket_incremental_sync_after_data_sync_init ... ok 2023-05-31T21:06:40.758 DEBUG:boto:path=/pklxkd-63/ 2023-05-31T21:06:40.759 DEBUG:boto:auth_path=/pklxkd-63/ 2023-05-31T21:06:40.759 DEBUG:boto:Method: PUT 2023-05-31T21:06:40.759 DEBUG:boto:Path: /pklxkd-63/ 2023-05-31T21:06:40.760 DEBUG:boto:Data: 2023-05-31T21:06:40.760 DEBUG:boto:Headers: {} 2023-05-31T21:06:40.760 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T21:06:40.761 DEBUG:boto:Port: 8000 2023-05-31T21:06:40.761 DEBUG:boto:Params: {} 2023-05-31T21:06:40.761 DEBUG:boto:Token: None 2023-05-31T21:06:40.762 DEBUG:boto:StringToSign: PUT Wed, 31 May 2023 21:06:40 GMT /pklxkd-63/ 2023-05-31T21:06:40.762 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:mTfmj5LSAzRFhEtl8LC5DrYrS74= 2023-05-31T21:06:40.763 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 21:06:40 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:mTfmj5LSAzRFhEtl8LC5DrYrS74=', 'Content-Length': '0'} 2023-05-31T21:06:40.778 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx000003568f58ea5ffa0c5-006477b6e0-1630-a1'), ('Content-Length', '0'), ('Date', 'Wed, 31 May 2023 21:06:40 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:06:40.778 DEBUG:rgw_multi.tests:created bucket=pklxkd-63 2023-05-31T21:06:40.779 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-31T21:06:40.779 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-31T21:06:40.779 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-05-31T21:06:40.827 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2023-05-31T21:06:40.828 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2023-05-31T21:06:40.865 INFO:teuthology.orchestra.run.smithi099.stderr:No --period given, using current period=2e6123d2-d695-4350-9a01-61a455bc782b 2023-05-31T21:06:40.890 INFO:teuthology.orchestra.run.smithi099.stdout:[ 2023-05-31T21:06:40.890 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T21:06:40.891 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1685566501.757183_260.1", 2023-05-31T21:06:40.891 INFO:teuthology.orchestra.run.smithi099.stdout: "last_update": "2023-05-31T20:55:01.757183Z" 2023-05-31T21:06:40.891 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:06:40.891 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T21:06:40.891 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1685566501.735717_251.1", 2023-05-31T21:06:40.892 INFO:teuthology.orchestra.run.smithi099.stdout: "last_update": "2023-05-31T20:55:01.735717Z" 2023-05-31T21:06:40.892 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:06:40.892 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T21:06:40.892 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1685562970.274572_93.1", 2023-05-31T21:06:40.892 INFO:teuthology.orchestra.run.smithi099.stdout: "last_update": "2023-05-31T19:56:10.274572Z" 2023-05-31T21:06:40.892 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:06:40.892 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T21:06:40.893 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1685567200.773333_576.1", 2023-05-31T21:06:40.893 INFO:teuthology.orchestra.run.smithi099.stdout: "last_update": "2023-05-31T21:06:40.773333Z" 2023-05-31T21:06:40.893 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T21:06:40.893 INFO:teuthology.orchestra.run.smithi099.stdout:] 2023-05-31T21:06:40.894 DEBUG:rgw_multi.tests:master meta markers={0: '1_1685566501.757183_260.1', 1: '1_1685566501.735717_251.1', 2: '1_1685562970.274572_93.1', 3: '1_1685567200.773333_576.1'} 2023-05-31T21:06:40.894 INFO:rgw_multi.tests:starting meta checkpoint for zone=a2 2023-05-31T21:06:40.895 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-31T21:06:40.895 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-31T21:06:40.895 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-05-31T21:06:40.945 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2023-05-31T21:06:40.946 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2023-05-31T21:06:41.005 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T21:06:41.001+0000 7fbc594f4780 20 RGW-SYNC:meta: read sync status 2023-05-31T21:06:41.006 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T21:06:41.002+0000 7fbc594f4780 20 RGW-SYNC:meta: read sync status 2023-05-31T21:06:41.025 INFO:teuthology.orchestra.run.smithi099.stdout:{ 2023-05-31T21:06:41.026 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_status": { 2023-05-31T21:06:41.026 INFO:teuthology.orchestra.run.smithi099.stdout: "info": { 2023-05-31T21:06:41.026 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "sync", 2023-05-31T21:06:41.026 INFO:teuthology.orchestra.run.smithi099.stdout: "num_shards": 4, 2023-05-31T21:06:41.026 INFO:teuthology.orchestra.run.smithi099.stdout: "period": "2e6123d2-d695-4350-9a01-61a455bc782b", 2023-05-31T21:06:41.027 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2023-05-31T21:06:41.027 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:06:41.027 INFO:teuthology.orchestra.run.smithi099.stdout: "markers": [ 2023-05-31T21:06:41.027 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T21:06:41.027 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 0, 2023-05-31T21:06:41.027 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T21:06:41.027 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2023-05-31T21:06:41.028 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1685566501.757183_260.1", 2023-05-31T21:06:41.028 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T21:06:41.028 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 1, 2023-05-31T21:06:41.028 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T21:06:41.028 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2023-05-31T20:55:01.757183Z", 2023-05-31T21:06:41.028 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2023-05-31T21:06:41.028 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T21:06:41.029 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:06:41.029 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T21:06:41.029 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 1, 2023-05-31T21:06:41.029 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T21:06:41.029 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2023-05-31T21:06:41.029 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1685566501.735717_251.1", 2023-05-31T21:06:41.030 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T21:06:41.030 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2023-05-31T21:06:41.030 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T21:06:41.030 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2023-05-31T20:55:01.735717Z", 2023-05-31T21:06:41.030 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2023-05-31T21:06:41.030 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T21:06:41.030 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:06:41.031 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T21:06:41.031 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 2, 2023-05-31T21:06:41.031 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T21:06:41.031 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2023-05-31T21:06:41.031 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1685562970.274572_93.1", 2023-05-31T21:06:41.031 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T21:06:41.031 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2023-05-31T21:06:41.032 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T21:06:41.032 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2023-05-31T19:56:10.274572Z", 2023-05-31T21:06:41.032 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2023-05-31T21:06:41.032 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T21:06:41.032 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:06:41.032 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T21:06:41.032 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 3, 2023-05-31T21:06:41.033 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T21:06:41.033 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2023-05-31T21:06:41.033 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1685567200.767436_573.1", 2023-05-31T21:06:41.033 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T21:06:41.033 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2023-05-31T21:06:41.033 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T21:06:41.033 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2023-05-31T21:06:40.767436Z", 2023-05-31T21:06:41.034 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2023-05-31T21:06:41.034 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T21:06:41.034 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T21:06:41.034 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2023-05-31T21:06:41.034 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:06:41.034 INFO:teuthology.orchestra.run.smithi099.stdout: "full_sync": { 2023-05-31T21:06:41.034 INFO:teuthology.orchestra.run.smithi099.stdout: "total": 1, 2023-05-31T21:06:41.035 INFO:teuthology.orchestra.run.smithi099.stdout: "complete": 1 2023-05-31T21:06:41.035 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:06:41.035 INFO:teuthology.orchestra.run.smithi099.stdout: "current_time": "2023-05-31T21:06:41Z" 2023-05-31T21:06:41.035 INFO:teuthology.orchestra.run.smithi099.stdout:} 2023-05-31T21:06:41.036 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "2e6123d2-d695-4350-9a01-61a455bc782b", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1685566501.757183_260.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-05-31T20:55:01.757183Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1685566501.735717_251.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-05-31T20:55:01.735717Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1685562970.274572_93.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-05-31T19:56:10.274572Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1685567200.767436_573.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-05-31T21:06:40.767436Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-05-31T21:06:41Z" } 2023-05-31T21:06:41.036 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1685566501.757183_260.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-05-31T20:55:01.757183Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1685566501.735717_251.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-05-31T20:55:01.735717Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1685562970.274572_93.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-05-31T19:56:10.274572Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1685567200.767436_573.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-05-31T21:06:40.767436Z', 'realm_epoch': 2}}] 2023-05-31T21:06:41.037 DEBUG:rgw_multi.tests:log_status={0: '1_1685566501.757183_260.1', 1: '1_1685566501.735717_251.1', 2: '1_1685562970.274572_93.1', 3: '1_1685567200.773333_576.1'} 2023-05-31T21:06:41.037 DEBUG:rgw_multi.tests:sync_status={0: '1_1685566501.757183_260.1', 1: '1_1685566501.735717_251.1', 2: '1_1685562970.274572_93.1', 3: '1_1685567200.767436_573.1'} 2023-05-31T21:06:41.037 WARNING:rgw_multi.tests:zone a2 behind master: shard=3 master=1_1685567200.773333_576.1 target=1_1685567200.767436_573.1 2023-05-31T21:06:46.039 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-31T21:06:46.039 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-31T21:06:46.040 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-05-31T21:06:46.097 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2023-05-31T21:06:46.098 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2023-05-31T21:06:46.151 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T21:06:46.147+0000 7fe186130780 20 RGW-SYNC:meta: read sync status 2023-05-31T21:06:46.152 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T21:06:46.148+0000 7fe186130780 20 RGW-SYNC:meta: read sync status 2023-05-31T21:06:46.173 INFO:teuthology.orchestra.run.smithi099.stdout:{ 2023-05-31T21:06:46.174 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_status": { 2023-05-31T21:06:46.174 INFO:teuthology.orchestra.run.smithi099.stdout: "info": { 2023-05-31T21:06:46.174 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "sync", 2023-05-31T21:06:46.174 INFO:teuthology.orchestra.run.smithi099.stdout: "num_shards": 4, 2023-05-31T21:06:46.174 INFO:teuthology.orchestra.run.smithi099.stdout: "period": "2e6123d2-d695-4350-9a01-61a455bc782b", 2023-05-31T21:06:46.174 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2023-05-31T21:06:46.175 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:06:46.175 INFO:teuthology.orchestra.run.smithi099.stdout: "markers": [ 2023-05-31T21:06:46.175 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T21:06:46.175 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 0, 2023-05-31T21:06:46.175 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T21:06:46.175 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2023-05-31T21:06:46.175 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1685566501.757183_260.1", 2023-05-31T21:06:46.176 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T21:06:46.176 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 1, 2023-05-31T21:06:46.176 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T21:06:46.176 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2023-05-31T20:55:01.757183Z", 2023-05-31T21:06:46.176 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2023-05-31T21:06:46.176 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T21:06:46.176 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:06:46.177 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T21:06:46.177 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 1, 2023-05-31T21:06:46.177 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T21:06:46.177 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2023-05-31T21:06:46.177 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1685566501.735717_251.1", 2023-05-31T21:06:46.177 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T21:06:46.178 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2023-05-31T21:06:46.178 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T21:06:46.178 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2023-05-31T20:55:01.735717Z", 2023-05-31T21:06:46.178 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2023-05-31T21:06:46.178 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T21:06:46.178 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:06:46.178 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T21:06:46.179 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 2, 2023-05-31T21:06:46.179 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T21:06:46.179 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2023-05-31T21:06:46.179 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1685562970.274572_93.1", 2023-05-31T21:06:46.179 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T21:06:46.179 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2023-05-31T21:06:46.179 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T21:06:46.180 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2023-05-31T19:56:10.274572Z", 2023-05-31T21:06:46.180 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2023-05-31T21:06:46.180 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T21:06:46.180 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:06:46.180 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T21:06:46.180 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 3, 2023-05-31T21:06:46.180 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T21:06:46.181 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2023-05-31T21:06:46.181 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1685567200.773333_576.1", 2023-05-31T21:06:46.181 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T21:06:46.181 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2023-05-31T21:06:46.181 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T21:06:46.181 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2023-05-31T21:06:40.773333Z", 2023-05-31T21:06:46.181 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2023-05-31T21:06:46.182 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T21:06:46.182 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T21:06:46.182 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2023-05-31T21:06:46.182 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:06:46.182 INFO:teuthology.orchestra.run.smithi099.stdout: "full_sync": { 2023-05-31T21:06:46.182 INFO:teuthology.orchestra.run.smithi099.stdout: "total": 1, 2023-05-31T21:06:46.182 INFO:teuthology.orchestra.run.smithi099.stdout: "complete": 1 2023-05-31T21:06:46.183 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:06:46.183 INFO:teuthology.orchestra.run.smithi099.stdout: "current_time": "2023-05-31T21:06:46Z" 2023-05-31T21:06:46.183 INFO:teuthology.orchestra.run.smithi099.stdout:} 2023-05-31T21:06:46.184 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "2e6123d2-d695-4350-9a01-61a455bc782b", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1685566501.757183_260.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-05-31T20:55:01.757183Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1685566501.735717_251.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-05-31T20:55:01.735717Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1685562970.274572_93.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-05-31T19:56:10.274572Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1685567200.773333_576.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-05-31T21:06:40.773333Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-05-31T21:06:46Z" } 2023-05-31T21:06:46.184 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1685566501.757183_260.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-05-31T20:55:01.757183Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1685566501.735717_251.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-05-31T20:55:01.735717Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1685562970.274572_93.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-05-31T19:56:10.274572Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1685567200.773333_576.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-05-31T21:06:40.773333Z', 'realm_epoch': 2}}] 2023-05-31T21:06:46.185 DEBUG:rgw_multi.tests:log_status={0: '1_1685566501.757183_260.1', 1: '1_1685566501.735717_251.1', 2: '1_1685562970.274572_93.1', 3: '1_1685567200.773333_576.1'} 2023-05-31T21:06:46.185 DEBUG:rgw_multi.tests:sync_status={0: '1_1685566501.757183_260.1', 1: '1_1685566501.735717_251.1', 2: '1_1685562970.274572_93.1', 3: '1_1685567200.773333_576.1'} 2023-05-31T21:06:46.185 INFO:rgw_multi.tests:finish meta checkpoint for zone=a2 2023-05-31T21:06:46.185 DEBUG:boto:path=/pklxkd-63/ 2023-05-31T21:06:46.186 DEBUG:boto:auth_path=/pklxkd-63/ 2023-05-31T21:06:46.186 DEBUG:boto:Method: HEAD 2023-05-31T21:06:46.187 DEBUG:boto:Path: /pklxkd-63/ 2023-05-31T21:06:46.187 DEBUG:boto:Data: 2023-05-31T21:06:46.187 DEBUG:boto:Headers: {} 2023-05-31T21:06:46.188 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T21:06:46.188 DEBUG:boto:Port: 8000 2023-05-31T21:06:46.188 DEBUG:boto:Params: {} 2023-05-31T21:06:46.189 DEBUG:boto:Token: None 2023-05-31T21:06:46.189 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 21:06:46 GMT /pklxkd-63/ 2023-05-31T21:06:46.189 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:bcNVcPZSg/FRj1VT8thTtJNmxUQ= 2023-05-31T21:06:46.190 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 21:06:46 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:bcNVcPZSg/FRj1VT8thTtJNmxUQ=', 'Content-Length': '0'} 2023-05-31T21:06:46.192 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000075c5c7474026f237-006477b6e6-1630-a1'), ('Content-Length', '0'), ('Date', 'Wed, 31 May 2023 21:06:46 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:06:46.192 DEBUG:boto:path=/pklxkd-63/obj11 2023-05-31T21:06:46.193 DEBUG:boto:auth_path=/pklxkd-63/obj11 2023-05-31T21:06:46.193 DEBUG:boto:Method: PUT 2023-05-31T21:06:46.194 DEBUG:boto:Path: /pklxkd-63/obj11 2023-05-31T21:06:46.194 DEBUG:boto:Data: 2023-05-31T21:06:46.194 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-05-31T21:06:46.195 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T21:06:46.195 DEBUG:boto:Port: 8000 2023-05-31T21:06:46.195 DEBUG:boto:Params: {} 2023-05-31T21:06:46.196 DEBUG:boto:Token: None 2023-05-31T21:06:46.196 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Wed, 31 May 2023 21:06:46 GMT /pklxkd-63/obj11 2023-05-31T21:06:46.196 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:GfZs+Vp+2YOiMXysi+n702nC2lY= 2023-05-31T21:06:46.197 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Wed, 31 May 2023 21:06:46 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:GfZs+Vp+2YOiMXysi+n702nC2lY='} 2023-05-31T21:06:46.205 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000abd08c606220b2d2-006477b6e6-1630-a1'), ('Date', 'Wed, 31 May 2023 21:06:46 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:06:46.206 DEBUG:boto:path=/pklxkd-63/ 2023-05-31T21:06:46.206 DEBUG:boto:auth_path=/pklxkd-63/ 2023-05-31T21:06:46.207 DEBUG:boto:Method: HEAD 2023-05-31T21:06:46.207 DEBUG:boto:Path: /pklxkd-63/ 2023-05-31T21:06:46.207 DEBUG:boto:Data: 2023-05-31T21:06:46.208 DEBUG:boto:Headers: {} 2023-05-31T21:06:46.208 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T21:06:46.208 DEBUG:boto:Port: 8000 2023-05-31T21:06:46.209 DEBUG:boto:Params: {} 2023-05-31T21:06:46.209 DEBUG:boto:Token: None 2023-05-31T21:06:46.209 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 21:06:46 GMT /pklxkd-63/ 2023-05-31T21:06:46.210 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:bcNVcPZSg/FRj1VT8thTtJNmxUQ= 2023-05-31T21:06:46.210 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 21:06:46 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:bcNVcPZSg/FRj1VT8thTtJNmxUQ=', 'Content-Length': '0'} 2023-05-31T21:06:46.212 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '3'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000008301ae5fbabc288-006477b6e6-1630-a1'), ('Content-Length', '0'), ('Date', 'Wed, 31 May 2023 21:06:46 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:06:46.213 DEBUG:boto:path=/pklxkd-63/obj22 2023-05-31T21:06:46.213 DEBUG:boto:auth_path=/pklxkd-63/obj22 2023-05-31T21:06:46.213 DEBUG:boto:Method: PUT 2023-05-31T21:06:46.214 DEBUG:boto:Path: /pklxkd-63/obj22 2023-05-31T21:06:46.214 DEBUG:boto:Data: 2023-05-31T21:06:46.214 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-05-31T21:06:46.215 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T21:06:46.215 DEBUG:boto:Port: 8000 2023-05-31T21:06:46.215 DEBUG:boto:Params: {} 2023-05-31T21:06:46.216 DEBUG:boto:Token: None 2023-05-31T21:06:46.216 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Wed, 31 May 2023 21:06:46 GMT /pklxkd-63/obj22 2023-05-31T21:06:46.217 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:PbPbXWKg3RYRYp11jInc+oIyhZk= 2023-05-31T21:06:46.217 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Wed, 31 May 2023 21:06:46 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:PbPbXWKg3RYRYp11jInc+oIyhZk='} 2023-05-31T21:06:46.260 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000005b28259f64dce2b4-006477b6e6-1630-a1'), ('Date', 'Wed, 31 May 2023 21:06:46 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:06:46.261 DEBUG:boto:path=/pklxkd-63/ 2023-05-31T21:06:46.261 DEBUG:boto:auth_path=/pklxkd-63/ 2023-05-31T21:06:46.262 DEBUG:boto:Method: HEAD 2023-05-31T21:06:46.262 DEBUG:boto:Path: /pklxkd-63/ 2023-05-31T21:06:46.262 DEBUG:boto:Data: 2023-05-31T21:06:46.263 DEBUG:boto:Headers: {} 2023-05-31T21:06:46.263 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T21:06:46.263 DEBUG:boto:Port: 8000 2023-05-31T21:06:46.263 DEBUG:boto:Params: {} 2023-05-31T21:06:46.264 DEBUG:boto:Token: None 2023-05-31T21:06:46.264 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 21:06:46 GMT /pklxkd-63/ 2023-05-31T21:06:46.265 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:bcNVcPZSg/FRj1VT8thTtJNmxUQ= 2023-05-31T21:06:46.265 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 21:06:46 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:bcNVcPZSg/FRj1VT8thTtJNmxUQ=', 'Content-Length': '0'} 2023-05-31T21:06:46.267 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '2'), ('X-RGW-Bytes-Used', '6'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000f604d18b14a281aa-006477b6e6-1630-a1'), ('Content-Length', '0'), ('Date', 'Wed, 31 May 2023 21:06:46 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:06:46.267 DEBUG:boto:path=/pklxkd-63/obj33 2023-05-31T21:06:46.268 DEBUG:boto:auth_path=/pklxkd-63/obj33 2023-05-31T21:06:46.268 DEBUG:boto:Method: PUT 2023-05-31T21:06:46.269 DEBUG:boto:Path: /pklxkd-63/obj33 2023-05-31T21:06:46.269 DEBUG:boto:Data: 2023-05-31T21:06:46.269 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-05-31T21:06:46.270 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T21:06:46.270 DEBUG:boto:Port: 8000 2023-05-31T21:06:46.270 DEBUG:boto:Params: {} 2023-05-31T21:06:46.271 DEBUG:boto:Token: None 2023-05-31T21:06:46.271 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Wed, 31 May 2023 21:06:46 GMT /pklxkd-63/obj33 2023-05-31T21:06:46.271 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:JMRCIKv5NREN7sNHZgQC2if6BJU= 2023-05-31T21:06:46.272 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Wed, 31 May 2023 21:06:46 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:JMRCIKv5NREN7sNHZgQC2if6BJU='} 2023-05-31T21:06:46.277 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000a4eacbb1521d2a83-006477b6e6-1630-a1'), ('Date', 'Wed, 31 May 2023 21:06:46 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:06:46.277 DEBUG:boto:path=/pklxkd-63/ 2023-05-31T21:06:46.278 DEBUG:boto:auth_path=/pklxkd-63/ 2023-05-31T21:06:46.278 DEBUG:boto:Method: HEAD 2023-05-31T21:06:46.278 DEBUG:boto:Path: /pklxkd-63/ 2023-05-31T21:06:46.279 DEBUG:boto:Data: 2023-05-31T21:06:46.279 DEBUG:boto:Headers: {} 2023-05-31T21:06:46.279 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T21:06:46.280 DEBUG:boto:Port: 8000 2023-05-31T21:06:46.280 DEBUG:boto:Params: {} 2023-05-31T21:06:46.280 DEBUG:boto:Token: None 2023-05-31T21:06:46.281 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 21:06:46 GMT /pklxkd-63/ 2023-05-31T21:06:46.281 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:bcNVcPZSg/FRj1VT8thTtJNmxUQ= 2023-05-31T21:06:46.281 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 21:06:46 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:bcNVcPZSg/FRj1VT8thTtJNmxUQ=', 'Content-Length': '0'} 2023-05-31T21:06:46.283 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '3'), ('X-RGW-Bytes-Used', '9'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000004b75a6d684635180-006477b6e6-1630-a1'), ('Content-Length', '0'), ('Date', 'Wed, 31 May 2023 21:06:46 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:06:46.284 DEBUG:boto:path=/pklxkd-63/obj44 2023-05-31T21:06:46.284 DEBUG:boto:auth_path=/pklxkd-63/obj44 2023-05-31T21:06:46.285 DEBUG:boto:Method: PUT 2023-05-31T21:06:46.285 DEBUG:boto:Path: /pklxkd-63/obj44 2023-05-31T21:06:46.285 DEBUG:boto:Data: 2023-05-31T21:06:46.286 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-05-31T21:06:46.286 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T21:06:46.286 DEBUG:boto:Port: 8000 2023-05-31T21:06:46.287 DEBUG:boto:Params: {} 2023-05-31T21:06:46.287 DEBUG:boto:Token: None 2023-05-31T21:06:46.287 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Wed, 31 May 2023 21:06:46 GMT /pklxkd-63/obj44 2023-05-31T21:06:46.288 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:wJtayVKGup7Jg2BoezAs2Co6FN4= 2023-05-31T21:06:46.288 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Wed, 31 May 2023 21:06:46 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:wJtayVKGup7Jg2BoezAs2Co6FN4='} 2023-05-31T21:06:46.294 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000007298f47ce55460ec-006477b6e6-1630-a1'), ('Date', 'Wed, 31 May 2023 21:06:46 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:06:46.294 DEBUG:boto:path=/pklxkd-63/ 2023-05-31T21:06:46.295 DEBUG:boto:auth_path=/pklxkd-63/ 2023-05-31T21:06:46.295 DEBUG:boto:Method: HEAD 2023-05-31T21:06:46.295 DEBUG:boto:Path: /pklxkd-63/ 2023-05-31T21:06:46.296 DEBUG:boto:Data: 2023-05-31T21:06:46.296 DEBUG:boto:Headers: {} 2023-05-31T21:06:46.296 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T21:06:46.297 DEBUG:boto:Port: 8000 2023-05-31T21:06:46.297 DEBUG:boto:Params: {} 2023-05-31T21:06:46.297 DEBUG:boto:Token: None 2023-05-31T21:06:46.298 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 21:06:46 GMT /pklxkd-63/ 2023-05-31T21:06:46.298 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:bcNVcPZSg/FRj1VT8thTtJNmxUQ= 2023-05-31T21:06:46.298 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 21:06:46 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:bcNVcPZSg/FRj1VT8thTtJNmxUQ=', 'Content-Length': '0'} 2023-05-31T21:06:46.300 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '4'), ('X-RGW-Bytes-Used', '12'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000001133cf367c031c8c-006477b6e6-1630-a1'), ('Content-Length', '0'), ('Date', 'Wed, 31 May 2023 21:06:46 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:06:46.301 DEBUG:boto:path=/pklxkd-63/obj55 2023-05-31T21:06:46.301 DEBUG:boto:auth_path=/pklxkd-63/obj55 2023-05-31T21:06:46.301 DEBUG:boto:Method: PUT 2023-05-31T21:06:46.302 DEBUG:boto:Path: /pklxkd-63/obj55 2023-05-31T21:06:46.302 DEBUG:boto:Data: 2023-05-31T21:06:46.302 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-05-31T21:06:46.303 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T21:06:46.303 DEBUG:boto:Port: 8000 2023-05-31T21:06:46.303 DEBUG:boto:Params: {} 2023-05-31T21:06:46.304 DEBUG:boto:Token: None 2023-05-31T21:06:46.304 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Wed, 31 May 2023 21:06:46 GMT /pklxkd-63/obj55 2023-05-31T21:06:46.305 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:XE45D3xSYM0zhAldN+S/Ukc6sLA= 2023-05-31T21:06:46.305 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Wed, 31 May 2023 21:06:46 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:XE45D3xSYM0zhAldN+S/Ukc6sLA='} 2023-05-31T21:06:46.348 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000003d86c94ac1f85316-006477b6e6-1630-a1'), ('Date', 'Wed, 31 May 2023 21:06:46 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:06:46.349 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'reshard', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'pklxkd-63', '--num-shards', '17', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-05-31T21:06:46.349 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'reshard', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'pklxkd-63', '--num-shards', '17', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-05-31T21:06:46.350 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket reshard --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --bucket pklxkd-63 --num-shards 17 --yes-i-really-mean-it --cluster c1 --debug-rgw 0 --debug-ms 0 2023-05-31T21:06:46.402 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2023-05-31T21:06:46.402 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2023-05-31T21:06:46.446 INFO:teuthology.orchestra.run.smithi099.stdout:tenant: 2023-05-31T21:06:46.446 INFO:teuthology.orchestra.run.smithi099.stdout:bucket name: pklxkd-63 2023-05-31T21:06:46.450 INFO:teuthology.orchestra.run.smithi099.stdout:total entries: 5 2023-05-31T21:06:46.501 DEBUG:boto:path=/pklxkd-63/ 2023-05-31T21:06:46.502 DEBUG:boto:auth_path=/pklxkd-63/ 2023-05-31T21:06:46.502 DEBUG:boto:Method: HEAD 2023-05-31T21:06:46.503 DEBUG:boto:Path: /pklxkd-63/ 2023-05-31T21:06:46.503 DEBUG:boto:Data: 2023-05-31T21:06:46.503 DEBUG:boto:Headers: {} 2023-05-31T21:06:46.504 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T21:06:46.504 DEBUG:boto:Port: 8000 2023-05-31T21:06:46.504 DEBUG:boto:Params: {} 2023-05-31T21:06:46.505 DEBUG:boto:Token: None 2023-05-31T21:06:46.505 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 21:06:46 GMT /pklxkd-63/ 2023-05-31T21:06:46.505 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:bcNVcPZSg/FRj1VT8thTtJNmxUQ= 2023-05-31T21:06:46.506 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 21:06:46 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:bcNVcPZSg/FRj1VT8thTtJNmxUQ=', 'Content-Length': '0'} 2023-05-31T21:06:46.508 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '5'), ('X-RGW-Bytes-Used', '15'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000192ab699a1c9f1fb-006477b6e6-1630-a1'), ('Content-Length', '0'), ('Date', 'Wed, 31 May 2023 21:06:46 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:06:46.509 DEBUG:boto:path=/pklxkd-63/obj17 2023-05-31T21:06:46.509 DEBUG:boto:auth_path=/pklxkd-63/obj17 2023-05-31T21:06:46.510 DEBUG:boto:Method: PUT 2023-05-31T21:06:46.510 DEBUG:boto:Path: /pklxkd-63/obj17 2023-05-31T21:06:46.510 DEBUG:boto:Data: 2023-05-31T21:06:46.511 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-05-31T21:06:46.511 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T21:06:46.511 DEBUG:boto:Port: 8000 2023-05-31T21:06:46.512 DEBUG:boto:Params: {} 2023-05-31T21:06:46.513 DEBUG:boto:Token: None 2023-05-31T21:06:46.513 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Wed, 31 May 2023 21:06:46 GMT /pklxkd-63/obj17 2023-05-31T21:06:46.514 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:OrZ4mwcvMAeL5BQ2sQyjkRnOrAg= 2023-05-31T21:06:46.514 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Wed, 31 May 2023 21:06:46 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:OrZ4mwcvMAeL5BQ2sQyjkRnOrAg='} 2023-05-31T21:06:46.522 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000665ec0c114082c09-006477b6e6-1630-a1'), ('Date', 'Wed, 31 May 2023 21:06:46 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:06:46.522 DEBUG:boto:path=/pklxkd-63/ 2023-05-31T21:06:46.523 DEBUG:boto:auth_path=/pklxkd-63/ 2023-05-31T21:06:46.523 DEBUG:boto:Method: HEAD 2023-05-31T21:06:46.523 DEBUG:boto:Path: /pklxkd-63/ 2023-05-31T21:06:46.524 DEBUG:boto:Data: 2023-05-31T21:06:46.524 DEBUG:boto:Headers: {} 2023-05-31T21:06:46.524 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T21:06:46.525 DEBUG:boto:Port: 8000 2023-05-31T21:06:46.525 DEBUG:boto:Params: {} 2023-05-31T21:06:46.525 DEBUG:boto:Token: None 2023-05-31T21:06:46.526 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 21:06:46 GMT /pklxkd-63/ 2023-05-31T21:06:46.526 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:bcNVcPZSg/FRj1VT8thTtJNmxUQ= 2023-05-31T21:06:46.527 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 21:06:46 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:bcNVcPZSg/FRj1VT8thTtJNmxUQ=', 'Content-Length': '0'} 2023-05-31T21:06:46.529 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '6'), ('X-RGW-Bytes-Used', '18'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000008b4a5b6bd048c9fd-006477b6e6-1630-a1'), ('Content-Length', '0'), ('Date', 'Wed, 31 May 2023 21:06:46 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:06:46.529 DEBUG:boto:path=/pklxkd-63/obj34 2023-05-31T21:06:46.530 DEBUG:boto:auth_path=/pklxkd-63/obj34 2023-05-31T21:06:46.530 DEBUG:boto:Method: PUT 2023-05-31T21:06:46.530 DEBUG:boto:Path: /pklxkd-63/obj34 2023-05-31T21:06:46.531 DEBUG:boto:Data: 2023-05-31T21:06:46.531 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-05-31T21:06:46.531 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T21:06:46.532 DEBUG:boto:Port: 8000 2023-05-31T21:06:46.532 DEBUG:boto:Params: {} 2023-05-31T21:06:46.532 DEBUG:boto:Token: None 2023-05-31T21:06:46.533 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Wed, 31 May 2023 21:06:46 GMT /pklxkd-63/obj34 2023-05-31T21:06:46.533 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:lGZVgG1T2eZnI5etXzC2nXcRIDg= 2023-05-31T21:06:46.534 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Wed, 31 May 2023 21:06:46 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:lGZVgG1T2eZnI5etXzC2nXcRIDg='} 2023-05-31T21:06:46.576 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000006a0f79649399bce-006477b6e6-1630-a1'), ('Date', 'Wed, 31 May 2023 21:06:46 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:06:46.577 DEBUG:boto:path=/pklxkd-63/ 2023-05-31T21:06:46.577 DEBUG:boto:auth_path=/pklxkd-63/ 2023-05-31T21:06:46.578 DEBUG:boto:Method: HEAD 2023-05-31T21:06:46.578 DEBUG:boto:Path: /pklxkd-63/ 2023-05-31T21:06:46.578 DEBUG:boto:Data: 2023-05-31T21:06:46.579 DEBUG:boto:Headers: {} 2023-05-31T21:06:46.579 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T21:06:46.579 DEBUG:boto:Port: 8000 2023-05-31T21:06:46.580 DEBUG:boto:Params: {} 2023-05-31T21:06:46.580 DEBUG:boto:Token: None 2023-05-31T21:06:46.581 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 21:06:46 GMT /pklxkd-63/ 2023-05-31T21:06:46.581 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:bcNVcPZSg/FRj1VT8thTtJNmxUQ= 2023-05-31T21:06:46.581 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 21:06:46 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:bcNVcPZSg/FRj1VT8thTtJNmxUQ=', 'Content-Length': '0'} 2023-05-31T21:06:46.584 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '7'), ('X-RGW-Bytes-Used', '21'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000669d4202a23e0db8-006477b6e6-1630-a1'), ('Content-Length', '0'), ('Date', 'Wed, 31 May 2023 21:06:46 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:06:46.584 DEBUG:boto:path=/pklxkd-63/obj51 2023-05-31T21:06:46.585 DEBUG:boto:auth_path=/pklxkd-63/obj51 2023-05-31T21:06:46.585 DEBUG:boto:Method: PUT 2023-05-31T21:06:46.585 DEBUG:boto:Path: /pklxkd-63/obj51 2023-05-31T21:06:46.586 DEBUG:boto:Data: 2023-05-31T21:06:46.586 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-05-31T21:06:46.586 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T21:06:46.587 DEBUG:boto:Port: 8000 2023-05-31T21:06:46.587 DEBUG:boto:Params: {} 2023-05-31T21:06:46.587 DEBUG:boto:Token: None 2023-05-31T21:06:46.588 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Wed, 31 May 2023 21:06:46 GMT /pklxkd-63/obj51 2023-05-31T21:06:46.588 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:UFPzYWgZXJQ/N83OvcYNYPxz0WM= 2023-05-31T21:06:46.589 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Wed, 31 May 2023 21:06:46 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:UFPzYWgZXJQ/N83OvcYNYPxz0WM='} 2023-05-31T21:06:46.595 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000008900e0650d16e444-006477b6e6-1630-a1'), ('Date', 'Wed, 31 May 2023 21:06:46 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:06:46.596 DEBUG:boto:path=/pklxkd-63/ 2023-05-31T21:06:46.596 DEBUG:boto:auth_path=/pklxkd-63/ 2023-05-31T21:06:46.596 DEBUG:boto:Method: HEAD 2023-05-31T21:06:46.597 DEBUG:boto:Path: /pklxkd-63/ 2023-05-31T21:06:46.597 DEBUG:boto:Data: 2023-05-31T21:06:46.597 DEBUG:boto:Headers: {} 2023-05-31T21:06:46.598 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T21:06:46.598 DEBUG:boto:Port: 8000 2023-05-31T21:06:46.598 DEBUG:boto:Params: {} 2023-05-31T21:06:46.599 DEBUG:boto:Token: None 2023-05-31T21:06:46.599 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 21:06:46 GMT /pklxkd-63/ 2023-05-31T21:06:46.599 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:bcNVcPZSg/FRj1VT8thTtJNmxUQ= 2023-05-31T21:06:46.600 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 21:06:46 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:bcNVcPZSg/FRj1VT8thTtJNmxUQ=', 'Content-Length': '0'} 2023-05-31T21:06:46.602 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '8'), ('X-RGW-Bytes-Used', '24'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000077f9ac6df169e51a-006477b6e6-1630-a1'), ('Content-Length', '0'), ('Date', 'Wed, 31 May 2023 21:06:46 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:06:46.603 DEBUG:boto:path=/pklxkd-63/obj68 2023-05-31T21:06:46.603 DEBUG:boto:auth_path=/pklxkd-63/obj68 2023-05-31T21:06:46.604 DEBUG:boto:Method: PUT 2023-05-31T21:06:46.604 DEBUG:boto:Path: /pklxkd-63/obj68 2023-05-31T21:06:46.604 DEBUG:boto:Data: 2023-05-31T21:06:46.605 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-05-31T21:06:46.605 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T21:06:46.605 DEBUG:boto:Port: 8000 2023-05-31T21:06:46.606 DEBUG:boto:Params: {} 2023-05-31T21:06:46.606 DEBUG:boto:Token: None 2023-05-31T21:06:46.606 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Wed, 31 May 2023 21:06:46 GMT /pklxkd-63/obj68 2023-05-31T21:06:46.607 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:oSGOKxHThne7/WPrFjF94PPQApM= 2023-05-31T21:06:46.607 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Wed, 31 May 2023 21:06:46 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:oSGOKxHThne7/WPrFjF94PPQApM='} 2023-05-31T21:06:46.612 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000009d5571bab3e7fdfb-006477b6e6-1630-a1'), ('Date', 'Wed, 31 May 2023 21:06:46 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:06:46.613 DEBUG:boto:path=/pklxkd-63/ 2023-05-31T21:06:46.613 DEBUG:boto:auth_path=/pklxkd-63/ 2023-05-31T21:06:46.614 DEBUG:boto:Method: HEAD 2023-05-31T21:06:46.614 DEBUG:boto:Path: /pklxkd-63/ 2023-05-31T21:06:46.614 DEBUG:boto:Data: 2023-05-31T21:06:46.615 DEBUG:boto:Headers: {} 2023-05-31T21:06:46.615 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T21:06:46.615 DEBUG:boto:Port: 8000 2023-05-31T21:06:46.616 DEBUG:boto:Params: {} 2023-05-31T21:06:46.616 DEBUG:boto:Token: None 2023-05-31T21:06:46.616 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 21:06:46 GMT /pklxkd-63/ 2023-05-31T21:06:46.617 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:bcNVcPZSg/FRj1VT8thTtJNmxUQ= 2023-05-31T21:06:46.617 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 21:06:46 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:bcNVcPZSg/FRj1VT8thTtJNmxUQ=', 'Content-Length': '0'} 2023-05-31T21:06:46.619 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '9'), ('X-RGW-Bytes-Used', '27'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000ee935ba45e8171e7-006477b6e6-1630-a1'), ('Content-Length', '0'), ('Date', 'Wed, 31 May 2023 21:06:46 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:06:46.620 DEBUG:boto:path=/pklxkd-63/obj85 2023-05-31T21:06:46.620 DEBUG:boto:auth_path=/pklxkd-63/obj85 2023-05-31T21:06:46.621 DEBUG:boto:Method: PUT 2023-05-31T21:06:46.621 DEBUG:boto:Path: /pklxkd-63/obj85 2023-05-31T21:06:46.621 DEBUG:boto:Data: 2023-05-31T21:06:46.622 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-05-31T21:06:46.622 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T21:06:46.622 DEBUG:boto:Port: 8000 2023-05-31T21:06:46.623 DEBUG:boto:Params: {} 2023-05-31T21:06:46.623 DEBUG:boto:Token: None 2023-05-31T21:06:46.623 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Wed, 31 May 2023 21:06:46 GMT /pklxkd-63/obj85 2023-05-31T21:06:46.624 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:hSks8AIl9wGlA5rXTY6VJD/Yx9Q= 2023-05-31T21:06:46.624 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Wed, 31 May 2023 21:06:46 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:hSks8AIl9wGlA5rXTY6VJD/Yx9Q='} 2023-05-31T21:06:46.668 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000a403532f1c89ce02-006477b6e6-1630-a1'), ('Date', 'Wed, 31 May 2023 21:06:46 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:06:46.669 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'reshard', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'pklxkd-63', '--num-shards', '19', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-05-31T21:06:46.669 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'reshard', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'pklxkd-63', '--num-shards', '19', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-05-31T21:06:46.670 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket reshard --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --bucket pklxkd-63 --num-shards 19 --yes-i-really-mean-it --cluster c1 --debug-rgw 0 --debug-ms 0 2023-05-31T21:06:46.721 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2023-05-31T21:06:46.721 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2023-05-31T21:06:46.775 INFO:teuthology.orchestra.run.smithi099.stdout:tenant: 2023-05-31T21:06:46.776 INFO:teuthology.orchestra.run.smithi099.stdout:bucket name: pklxkd-63 2023-05-31T21:06:46.781 INFO:teuthology.orchestra.run.smithi099.stdout:total entries: 10 2023-05-31T21:06:46.831 DEBUG:boto:path=/pklxkd-63/ 2023-05-31T21:06:46.831 DEBUG:boto:auth_path=/pklxkd-63/ 2023-05-31T21:06:46.832 DEBUG:boto:Method: HEAD 2023-05-31T21:06:46.832 DEBUG:boto:Path: /pklxkd-63/ 2023-05-31T21:06:46.832 DEBUG:boto:Data: 2023-05-31T21:06:46.833 DEBUG:boto:Headers: {} 2023-05-31T21:06:46.833 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T21:06:46.833 DEBUG:boto:Port: 8000 2023-05-31T21:06:46.834 DEBUG:boto:Params: {} 2023-05-31T21:06:46.834 DEBUG:boto:Token: None 2023-05-31T21:06:46.834 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 21:06:46 GMT /pklxkd-63/ 2023-05-31T21:06:46.835 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:bcNVcPZSg/FRj1VT8thTtJNmxUQ= 2023-05-31T21:06:46.835 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 21:06:46 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:bcNVcPZSg/FRj1VT8thTtJNmxUQ=', 'Content-Length': '0'} 2023-05-31T21:06:46.838 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '10'), ('X-RGW-Bytes-Used', '30'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000000bb7546a3333c47-006477b6e6-1630-a1'), ('Content-Length', '0'), ('Date', 'Wed, 31 May 2023 21:06:46 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:06:46.838 DEBUG:boto:path=/pklxkd-63/obj19 2023-05-31T21:06:46.839 DEBUG:boto:auth_path=/pklxkd-63/obj19 2023-05-31T21:06:46.839 DEBUG:boto:Method: PUT 2023-05-31T21:06:46.840 DEBUG:boto:Path: /pklxkd-63/obj19 2023-05-31T21:06:46.840 DEBUG:boto:Data: 2023-05-31T21:06:46.840 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-05-31T21:06:46.841 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T21:06:46.841 DEBUG:boto:Port: 8000 2023-05-31T21:06:46.841 DEBUG:boto:Params: {} 2023-05-31T21:06:46.842 DEBUG:boto:Token: None 2023-05-31T21:06:46.842 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Wed, 31 May 2023 21:06:46 GMT /pklxkd-63/obj19 2023-05-31T21:06:46.842 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:15k03jeZBvtfCMWs3WizVJ6VkpE= 2023-05-31T21:06:46.843 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Wed, 31 May 2023 21:06:46 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:15k03jeZBvtfCMWs3WizVJ6VkpE='} 2023-05-31T21:06:46.847 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000020343e4c0afd143-006477b6e6-1630-a1'), ('Date', 'Wed, 31 May 2023 21:06:46 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:06:46.848 DEBUG:boto:path=/pklxkd-63/ 2023-05-31T21:06:46.848 DEBUG:boto:auth_path=/pklxkd-63/ 2023-05-31T21:06:46.849 DEBUG:boto:Method: HEAD 2023-05-31T21:06:46.849 DEBUG:boto:Path: /pklxkd-63/ 2023-05-31T21:06:46.849 DEBUG:boto:Data: 2023-05-31T21:06:46.850 DEBUG:boto:Headers: {} 2023-05-31T21:06:46.850 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T21:06:46.850 DEBUG:boto:Port: 8000 2023-05-31T21:06:46.851 DEBUG:boto:Params: {} 2023-05-31T21:06:46.851 DEBUG:boto:Token: None 2023-05-31T21:06:46.852 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 21:06:46 GMT /pklxkd-63/ 2023-05-31T21:06:46.852 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:bcNVcPZSg/FRj1VT8thTtJNmxUQ= 2023-05-31T21:06:46.852 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 21:06:46 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:bcNVcPZSg/FRj1VT8thTtJNmxUQ=', 'Content-Length': '0'} 2023-05-31T21:06:46.855 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '11'), ('X-RGW-Bytes-Used', '33'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000082d23a8c64bce232-006477b6e6-1630-a1'), ('Content-Length', '0'), ('Date', 'Wed, 31 May 2023 21:06:46 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:06:46.855 DEBUG:boto:path=/pklxkd-63/obj38 2023-05-31T21:06:46.856 DEBUG:boto:auth_path=/pklxkd-63/obj38 2023-05-31T21:06:46.856 DEBUG:boto:Method: PUT 2023-05-31T21:06:46.856 DEBUG:boto:Path: /pklxkd-63/obj38 2023-05-31T21:06:46.857 DEBUG:boto:Data: 2023-05-31T21:06:46.857 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-05-31T21:06:46.857 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T21:06:46.858 DEBUG:boto:Port: 8000 2023-05-31T21:06:46.858 DEBUG:boto:Params: {} 2023-05-31T21:06:46.858 DEBUG:boto:Token: None 2023-05-31T21:06:46.859 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Wed, 31 May 2023 21:06:46 GMT /pklxkd-63/obj38 2023-05-31T21:06:46.859 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:4YeXx5JAAvSZJKDMoLkbmUUEDdg= 2023-05-31T21:06:46.860 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Wed, 31 May 2023 21:06:46 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:4YeXx5JAAvSZJKDMoLkbmUUEDdg='} 2023-05-31T21:06:46.863 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000007e5055c29f1d5d2d-006477b6e6-1630-a1'), ('Date', 'Wed, 31 May 2023 21:06:46 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:06:46.864 DEBUG:boto:path=/pklxkd-63/ 2023-05-31T21:06:46.864 DEBUG:boto:auth_path=/pklxkd-63/ 2023-05-31T21:06:46.864 DEBUG:boto:Method: HEAD 2023-05-31T21:06:46.865 DEBUG:boto:Path: /pklxkd-63/ 2023-05-31T21:06:46.865 DEBUG:boto:Data: 2023-05-31T21:06:46.865 DEBUG:boto:Headers: {} 2023-05-31T21:06:46.866 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T21:06:46.866 DEBUG:boto:Port: 8000 2023-05-31T21:06:46.866 DEBUG:boto:Params: {} 2023-05-31T21:06:46.867 DEBUG:boto:Token: None 2023-05-31T21:06:46.867 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 21:06:46 GMT /pklxkd-63/ 2023-05-31T21:06:46.867 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:bcNVcPZSg/FRj1VT8thTtJNmxUQ= 2023-05-31T21:06:46.868 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 21:06:46 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:bcNVcPZSg/FRj1VT8thTtJNmxUQ=', 'Content-Length': '0'} 2023-05-31T21:06:46.870 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '12'), ('X-RGW-Bytes-Used', '36'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000a6cff33ca196fd48-006477b6e6-1630-a1'), ('Content-Length', '0'), ('Date', 'Wed, 31 May 2023 21:06:46 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:06:46.871 DEBUG:boto:path=/pklxkd-63/obj57 2023-05-31T21:06:46.871 DEBUG:boto:auth_path=/pklxkd-63/obj57 2023-05-31T21:06:46.871 DEBUG:boto:Method: PUT 2023-05-31T21:06:46.872 DEBUG:boto:Path: /pklxkd-63/obj57 2023-05-31T21:06:46.872 DEBUG:boto:Data: 2023-05-31T21:06:46.872 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-05-31T21:06:46.873 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T21:06:46.873 DEBUG:boto:Port: 8000 2023-05-31T21:06:46.873 DEBUG:boto:Params: {} 2023-05-31T21:06:46.874 DEBUG:boto:Token: None 2023-05-31T21:06:46.874 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Wed, 31 May 2023 21:06:46 GMT /pklxkd-63/obj57 2023-05-31T21:06:46.875 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:ZM/wRTD2UVYv4UBQ9XZipWrNwYs= 2023-05-31T21:06:46.875 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Wed, 31 May 2023 21:06:46 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:ZM/wRTD2UVYv4UBQ9XZipWrNwYs='} 2023-05-31T21:06:46.881 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000007fd1d16cd0d31bdc-006477b6e6-1630-a1'), ('Date', 'Wed, 31 May 2023 21:06:46 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:06:46.882 DEBUG:boto:path=/pklxkd-63/ 2023-05-31T21:06:46.882 DEBUG:boto:auth_path=/pklxkd-63/ 2023-05-31T21:06:46.882 DEBUG:boto:Method: HEAD 2023-05-31T21:06:46.883 DEBUG:boto:Path: /pklxkd-63/ 2023-05-31T21:06:46.883 DEBUG:boto:Data: 2023-05-31T21:06:46.883 DEBUG:boto:Headers: {} 2023-05-31T21:06:46.884 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T21:06:46.884 DEBUG:boto:Port: 8000 2023-05-31T21:06:46.884 DEBUG:boto:Params: {} 2023-05-31T21:06:46.885 DEBUG:boto:Token: None 2023-05-31T21:06:46.885 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 21:06:46 GMT /pklxkd-63/ 2023-05-31T21:06:46.885 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:bcNVcPZSg/FRj1VT8thTtJNmxUQ= 2023-05-31T21:06:46.886 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 21:06:46 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:bcNVcPZSg/FRj1VT8thTtJNmxUQ=', 'Content-Length': '0'} 2023-05-31T21:06:46.888 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '13'), ('X-RGW-Bytes-Used', '39'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000cfd3303aa4bd4612-006477b6e6-1630-a1'), ('Content-Length', '0'), ('Date', 'Wed, 31 May 2023 21:06:46 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:06:46.889 DEBUG:boto:path=/pklxkd-63/obj76 2023-05-31T21:06:46.889 DEBUG:boto:auth_path=/pklxkd-63/obj76 2023-05-31T21:06:46.889 DEBUG:boto:Method: PUT 2023-05-31T21:06:46.890 DEBUG:boto:Path: /pklxkd-63/obj76 2023-05-31T21:06:46.890 DEBUG:boto:Data: 2023-05-31T21:06:46.890 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-05-31T21:06:46.891 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T21:06:46.891 DEBUG:boto:Port: 8000 2023-05-31T21:06:46.891 DEBUG:boto:Params: {} 2023-05-31T21:06:46.892 DEBUG:boto:Token: None 2023-05-31T21:06:46.892 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Wed, 31 May 2023 21:06:46 GMT /pklxkd-63/obj76 2023-05-31T21:06:46.892 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:4mtLbF5aErMMDq3GaxJKBI8NuSs= 2023-05-31T21:06:46.893 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Wed, 31 May 2023 21:06:46 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:4mtLbF5aErMMDq3GaxJKBI8NuSs='} 2023-05-31T21:06:46.936 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000054b1b5006afef11d-006477b6e6-1630-a1'), ('Date', 'Wed, 31 May 2023 21:06:46 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:06:46.937 DEBUG:boto:path=/pklxkd-63/ 2023-05-31T21:06:46.937 DEBUG:boto:auth_path=/pklxkd-63/ 2023-05-31T21:06:46.937 DEBUG:boto:Method: HEAD 2023-05-31T21:06:46.938 DEBUG:boto:Path: /pklxkd-63/ 2023-05-31T21:06:46.938 DEBUG:boto:Data: 2023-05-31T21:06:46.938 DEBUG:boto:Headers: {} 2023-05-31T21:06:46.939 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T21:06:46.939 DEBUG:boto:Port: 8000 2023-05-31T21:06:46.939 DEBUG:boto:Params: {} 2023-05-31T21:06:46.940 DEBUG:boto:Token: None 2023-05-31T21:06:46.940 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 21:06:46 GMT /pklxkd-63/ 2023-05-31T21:06:46.940 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:bcNVcPZSg/FRj1VT8thTtJNmxUQ= 2023-05-31T21:06:46.941 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 21:06:46 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:bcNVcPZSg/FRj1VT8thTtJNmxUQ=', 'Content-Length': '0'} 2023-05-31T21:06:46.944 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '14'), ('X-RGW-Bytes-Used', '42'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000037caabbad6771194-006477b6e6-1630-a1'), ('Content-Length', '0'), ('Date', 'Wed, 31 May 2023 21:06:46 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:06:46.944 DEBUG:boto:path=/pklxkd-63/obj95 2023-05-31T21:06:46.945 DEBUG:boto:auth_path=/pklxkd-63/obj95 2023-05-31T21:06:46.945 DEBUG:boto:Method: PUT 2023-05-31T21:06:46.945 DEBUG:boto:Path: /pklxkd-63/obj95 2023-05-31T21:06:46.946 DEBUG:boto:Data: 2023-05-31T21:06:46.946 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-05-31T21:06:46.946 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T21:06:46.947 DEBUG:boto:Port: 8000 2023-05-31T21:06:46.947 DEBUG:boto:Params: {} 2023-05-31T21:06:46.947 DEBUG:boto:Token: None 2023-05-31T21:06:46.948 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Wed, 31 May 2023 21:06:46 GMT /pklxkd-63/obj95 2023-05-31T21:06:46.948 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:K7yxuDgRGtzb7pIl5Z9my/8Hjvk= 2023-05-31T21:06:46.949 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Wed, 31 May 2023 21:06:46 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:K7yxuDgRGtzb7pIl5Z9my/8Hjvk='} 2023-05-31T21:06:46.952 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000047db55aa823e5316-006477b6e6-1630-a1'), ('Date', 'Wed, 31 May 2023 21:06:46 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:06:46.953 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'reshard', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'pklxkd-63', '--num-shards', '23', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-05-31T21:06:46.953 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'reshard', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'pklxkd-63', '--num-shards', '23', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-05-31T21:06:46.954 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket reshard --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --bucket pklxkd-63 --num-shards 23 --yes-i-really-mean-it --cluster c1 --debug-rgw 0 --debug-ms 0 2023-05-31T21:06:47.006 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2023-05-31T21:06:47.006 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2023-05-31T21:06:47.079 INFO:teuthology.orchestra.run.smithi099.stdout:tenant: 2023-05-31T21:06:47.079 INFO:teuthology.orchestra.run.smithi099.stdout:bucket name: pklxkd-63 2023-05-31T21:06:47.085 INFO:teuthology.orchestra.run.smithi099.stdout:total entries: 15 2023-05-31T21:06:47.142 DEBUG:boto:path=/pklxkd-63/ 2023-05-31T21:06:47.142 DEBUG:boto:auth_path=/pklxkd-63/ 2023-05-31T21:06:47.143 DEBUG:boto:Method: HEAD 2023-05-31T21:06:47.143 DEBUG:boto:Path: /pklxkd-63/ 2023-05-31T21:06:47.143 DEBUG:boto:Data: 2023-05-31T21:06:47.144 DEBUG:boto:Headers: {} 2023-05-31T21:06:47.144 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T21:06:47.144 DEBUG:boto:Port: 8000 2023-05-31T21:06:47.145 DEBUG:boto:Params: {} 2023-05-31T21:06:47.145 DEBUG:boto:Token: None 2023-05-31T21:06:47.146 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 21:06:47 GMT /pklxkd-63/ 2023-05-31T21:06:47.146 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:PDWLJ6ztPB6x60LVcukbAfyzcxQ= 2023-05-31T21:06:47.147 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 21:06:47 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:PDWLJ6ztPB6x60LVcukbAfyzcxQ=', 'Content-Length': '0'} 2023-05-31T21:06:47.150 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '15'), ('X-RGW-Bytes-Used', '45'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000002f6aef316d52fee-006477b6e7-1630-a1'), ('Content-Length', '0'), ('Date', 'Wed, 31 May 2023 21:06:47 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:06:47.150 DEBUG:boto:path=/pklxkd-63/obj23 2023-05-31T21:06:47.151 DEBUG:boto:auth_path=/pklxkd-63/obj23 2023-05-31T21:06:47.151 DEBUG:boto:Method: PUT 2023-05-31T21:06:47.152 DEBUG:boto:Path: /pklxkd-63/obj23 2023-05-31T21:06:47.152 DEBUG:boto:Data: 2023-05-31T21:06:47.152 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-05-31T21:06:47.153 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T21:06:47.153 DEBUG:boto:Port: 8000 2023-05-31T21:06:47.153 DEBUG:boto:Params: {} 2023-05-31T21:06:47.154 DEBUG:boto:Token: None 2023-05-31T21:06:47.154 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Wed, 31 May 2023 21:06:47 GMT /pklxkd-63/obj23 2023-05-31T21:06:47.155 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:kLuJ+yf0EPxFUL+nN7YCIf06kSE= 2023-05-31T21:06:47.155 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Wed, 31 May 2023 21:06:47 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:kLuJ+yf0EPxFUL+nN7YCIf06kSE='} 2023-05-31T21:06:47.160 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000ca0c2dfd231121de-006477b6e7-1630-a1'), ('Date', 'Wed, 31 May 2023 21:06:47 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:06:47.161 DEBUG:boto:path=/pklxkd-63/ 2023-05-31T21:06:47.161 DEBUG:boto:auth_path=/pklxkd-63/ 2023-05-31T21:06:47.161 DEBUG:boto:Method: HEAD 2023-05-31T21:06:47.162 DEBUG:boto:Path: /pklxkd-63/ 2023-05-31T21:06:47.162 DEBUG:boto:Data: 2023-05-31T21:06:47.162 DEBUG:boto:Headers: {} 2023-05-31T21:06:47.163 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T21:06:47.163 DEBUG:boto:Port: 8000 2023-05-31T21:06:47.163 DEBUG:boto:Params: {} 2023-05-31T21:06:47.164 DEBUG:boto:Token: None 2023-05-31T21:06:47.164 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 21:06:47 GMT /pklxkd-63/ 2023-05-31T21:06:47.165 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:PDWLJ6ztPB6x60LVcukbAfyzcxQ= 2023-05-31T21:06:47.165 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 21:06:47 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:PDWLJ6ztPB6x60LVcukbAfyzcxQ=', 'Content-Length': '0'} 2023-05-31T21:06:47.168 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '16'), ('X-RGW-Bytes-Used', '48'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000006354af178c15516c-006477b6e7-1630-a1'), ('Content-Length', '0'), ('Date', 'Wed, 31 May 2023 21:06:47 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:06:47.168 DEBUG:boto:path=/pklxkd-63/obj46 2023-05-31T21:06:47.169 DEBUG:boto:auth_path=/pklxkd-63/obj46 2023-05-31T21:06:47.169 DEBUG:boto:Method: PUT 2023-05-31T21:06:47.170 DEBUG:boto:Path: /pklxkd-63/obj46 2023-05-31T21:06:47.170 DEBUG:boto:Data: 2023-05-31T21:06:47.170 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-05-31T21:06:47.171 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T21:06:47.171 DEBUG:boto:Port: 8000 2023-05-31T21:06:47.171 DEBUG:boto:Params: {} 2023-05-31T21:06:47.172 DEBUG:boto:Token: None 2023-05-31T21:06:47.172 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Wed, 31 May 2023 21:06:47 GMT /pklxkd-63/obj46 2023-05-31T21:06:47.172 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:UvYUQ/KnzWwSF2T7VbK9kWrVeIE= 2023-05-31T21:06:47.173 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Wed, 31 May 2023 21:06:47 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:UvYUQ/KnzWwSF2T7VbK9kWrVeIE='} 2023-05-31T21:06:47.216 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000715e05fb623f29ba-006477b6e7-1630-a1'), ('Date', 'Wed, 31 May 2023 21:06:47 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:06:47.217 DEBUG:boto:path=/pklxkd-63/ 2023-05-31T21:06:47.217 DEBUG:boto:auth_path=/pklxkd-63/ 2023-05-31T21:06:47.218 DEBUG:boto:Method: HEAD 2023-05-31T21:06:47.218 DEBUG:boto:Path: /pklxkd-63/ 2023-05-31T21:06:47.218 DEBUG:boto:Data: 2023-05-31T21:06:47.219 DEBUG:boto:Headers: {} 2023-05-31T21:06:47.219 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T21:06:47.219 DEBUG:boto:Port: 8000 2023-05-31T21:06:47.220 DEBUG:boto:Params: {} 2023-05-31T21:06:47.220 DEBUG:boto:Token: None 2023-05-31T21:06:47.221 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 21:06:47 GMT /pklxkd-63/ 2023-05-31T21:06:47.221 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:PDWLJ6ztPB6x60LVcukbAfyzcxQ= 2023-05-31T21:06:47.221 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 21:06:47 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:PDWLJ6ztPB6x60LVcukbAfyzcxQ=', 'Content-Length': '0'} 2023-05-31T21:06:47.224 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '17'), ('X-RGW-Bytes-Used', '51'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000e301ad907a50d841-006477b6e7-1630-a1'), ('Content-Length', '0'), ('Date', 'Wed, 31 May 2023 21:06:47 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:06:47.225 DEBUG:boto:path=/pklxkd-63/obj69 2023-05-31T21:06:47.225 DEBUG:boto:auth_path=/pklxkd-63/obj69 2023-05-31T21:06:47.226 DEBUG:boto:Method: PUT 2023-05-31T21:06:47.226 DEBUG:boto:Path: /pklxkd-63/obj69 2023-05-31T21:06:47.226 DEBUG:boto:Data: 2023-05-31T21:06:47.227 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-05-31T21:06:47.227 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T21:06:47.228 DEBUG:boto:Port: 8000 2023-05-31T21:06:47.228 DEBUG:boto:Params: {} 2023-05-31T21:06:47.229 DEBUG:boto:Token: None 2023-05-31T21:06:47.229 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Wed, 31 May 2023 21:06:47 GMT /pklxkd-63/obj69 2023-05-31T21:06:47.229 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:Gf7J2S29DgA6yqGpUVOv0uz0FcI= 2023-05-31T21:06:47.230 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Wed, 31 May 2023 21:06:47 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:Gf7J2S29DgA6yqGpUVOv0uz0FcI='} 2023-05-31T21:06:47.235 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000002d4da9ddd66065f0-006477b6e7-1630-a1'), ('Date', 'Wed, 31 May 2023 21:06:47 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:06:47.235 DEBUG:boto:path=/pklxkd-63/ 2023-05-31T21:06:47.236 DEBUG:boto:auth_path=/pklxkd-63/ 2023-05-31T21:06:47.236 DEBUG:boto:Method: HEAD 2023-05-31T21:06:47.236 DEBUG:boto:Path: /pklxkd-63/ 2023-05-31T21:06:47.237 DEBUG:boto:Data: 2023-05-31T21:06:47.237 DEBUG:boto:Headers: {} 2023-05-31T21:06:47.237 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T21:06:47.238 DEBUG:boto:Port: 8000 2023-05-31T21:06:47.238 DEBUG:boto:Params: {} 2023-05-31T21:06:47.239 DEBUG:boto:Token: None 2023-05-31T21:06:47.239 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 21:06:47 GMT /pklxkd-63/ 2023-05-31T21:06:47.240 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:PDWLJ6ztPB6x60LVcukbAfyzcxQ= 2023-05-31T21:06:47.240 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 21:06:47 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:PDWLJ6ztPB6x60LVcukbAfyzcxQ=', 'Content-Length': '0'} 2023-05-31T21:06:47.242 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '18'), ('X-RGW-Bytes-Used', '54'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000004aa7d88af8550c5a-006477b6e7-1630-a1'), ('Content-Length', '0'), ('Date', 'Wed, 31 May 2023 21:06:47 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:06:47.243 DEBUG:boto:path=/pklxkd-63/obj92 2023-05-31T21:06:47.243 DEBUG:boto:auth_path=/pklxkd-63/obj92 2023-05-31T21:06:47.244 DEBUG:boto:Method: PUT 2023-05-31T21:06:47.244 DEBUG:boto:Path: /pklxkd-63/obj92 2023-05-31T21:06:47.245 DEBUG:boto:Data: 2023-05-31T21:06:47.245 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-05-31T21:06:47.245 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T21:06:47.246 DEBUG:boto:Port: 8000 2023-05-31T21:06:47.246 DEBUG:boto:Params: {} 2023-05-31T21:06:47.246 DEBUG:boto:Token: None 2023-05-31T21:06:47.247 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Wed, 31 May 2023 21:06:47 GMT /pklxkd-63/obj92 2023-05-31T21:06:47.247 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:Vz62czs73e6agjkfVO37hHgeH40= 2023-05-31T21:06:47.247 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Wed, 31 May 2023 21:06:47 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:Vz62czs73e6agjkfVO37hHgeH40='} 2023-05-31T21:06:47.292 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000e11214b34fd54dbd-006477b6e7-1630-a1'), ('Date', 'Wed, 31 May 2023 21:06:47 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:06:47.293 DEBUG:boto:path=/pklxkd-63/ 2023-05-31T21:06:47.293 DEBUG:boto:auth_path=/pklxkd-63/ 2023-05-31T21:06:47.293 DEBUG:boto:Method: HEAD 2023-05-31T21:06:47.294 DEBUG:boto:Path: /pklxkd-63/ 2023-05-31T21:06:47.294 DEBUG:boto:Data: 2023-05-31T21:06:47.294 DEBUG:boto:Headers: {} 2023-05-31T21:06:47.295 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T21:06:47.295 DEBUG:boto:Port: 8000 2023-05-31T21:06:47.295 DEBUG:boto:Params: {} 2023-05-31T21:06:47.296 DEBUG:boto:Token: None 2023-05-31T21:06:47.296 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 21:06:47 GMT /pklxkd-63/ 2023-05-31T21:06:47.297 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:PDWLJ6ztPB6x60LVcukbAfyzcxQ= 2023-05-31T21:06:47.297 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 21:06:47 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:PDWLJ6ztPB6x60LVcukbAfyzcxQ=', 'Content-Length': '0'} 2023-05-31T21:06:47.300 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '19'), ('X-RGW-Bytes-Used', '57'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000008589207dc545f14a-006477b6e7-1630-a1'), ('Content-Length', '0'), ('Date', 'Wed, 31 May 2023 21:06:47 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:06:47.300 DEBUG:boto:path=/pklxkd-63/obj115 2023-05-31T21:06:47.301 DEBUG:boto:auth_path=/pklxkd-63/obj115 2023-05-31T21:06:47.301 DEBUG:boto:Method: PUT 2023-05-31T21:06:47.301 DEBUG:boto:Path: /pklxkd-63/obj115 2023-05-31T21:06:47.302 DEBUG:boto:Data: 2023-05-31T21:06:47.302 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-05-31T21:06:47.302 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T21:06:47.303 DEBUG:boto:Port: 8000 2023-05-31T21:06:47.303 DEBUG:boto:Params: {} 2023-05-31T21:06:47.303 DEBUG:boto:Token: None 2023-05-31T21:06:47.304 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Wed, 31 May 2023 21:06:47 GMT /pklxkd-63/obj115 2023-05-31T21:06:47.304 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:UnGd12vsbkgnzJ8sUj8XTUlw6Lc= 2023-05-31T21:06:47.305 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Wed, 31 May 2023 21:06:47 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:UnGd12vsbkgnzJ8sUj8XTUlw6Lc='} 2023-05-31T21:06:47.311 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000006ff8e05738bf193e-006477b6e7-1630-a1'), ('Date', 'Wed, 31 May 2023 21:06:47 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:06:47.311 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'reshard', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'pklxkd-63', '--num-shards', '29', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-05-31T21:06:47.312 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'reshard', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'pklxkd-63', '--num-shards', '29', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-05-31T21:06:47.312 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket reshard --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --bucket pklxkd-63 --num-shards 29 --yes-i-really-mean-it --cluster c1 --debug-rgw 0 --debug-ms 0 2023-05-31T21:06:47.364 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2023-05-31T21:06:47.365 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2023-05-31T21:06:47.430 INFO:teuthology.orchestra.run.smithi099.stdout:tenant: 2023-05-31T21:06:47.430 INFO:teuthology.orchestra.run.smithi099.stdout:bucket name: pklxkd-63 2023-05-31T21:06:47.438 INFO:teuthology.orchestra.run.smithi099.stdout:total entries: 20 2023-05-31T21:06:47.517 DEBUG:boto:path=/pklxkd-63/ 2023-05-31T21:06:47.518 DEBUG:boto:auth_path=/pklxkd-63/ 2023-05-31T21:06:47.518 DEBUG:boto:Method: HEAD 2023-05-31T21:06:47.518 DEBUG:boto:Path: /pklxkd-63/ 2023-05-31T21:06:47.519 DEBUG:boto:Data: 2023-05-31T21:06:47.519 DEBUG:boto:Headers: {} 2023-05-31T21:06:47.519 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T21:06:47.520 DEBUG:boto:Port: 8000 2023-05-31T21:06:47.520 DEBUG:boto:Params: {} 2023-05-31T21:06:47.521 DEBUG:boto:Token: None 2023-05-31T21:06:47.521 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 21:06:47 GMT /pklxkd-63/ 2023-05-31T21:06:47.521 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:PDWLJ6ztPB6x60LVcukbAfyzcxQ= 2023-05-31T21:06:47.522 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 21:06:47 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:PDWLJ6ztPB6x60LVcukbAfyzcxQ=', 'Content-Length': '0'} 2023-05-31T21:06:47.525 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '20'), ('X-RGW-Bytes-Used', '60'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000dc77ed229906e7c0-006477b6e7-1630-a1'), ('Content-Length', '0'), ('Date', 'Wed, 31 May 2023 21:06:47 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:06:47.526 DEBUG:boto:path=/pklxkd-63/obj29 2023-05-31T21:06:47.526 DEBUG:boto:auth_path=/pklxkd-63/obj29 2023-05-31T21:06:47.526 DEBUG:boto:Method: PUT 2023-05-31T21:06:47.527 DEBUG:boto:Path: /pklxkd-63/obj29 2023-05-31T21:06:47.527 DEBUG:boto:Data: 2023-05-31T21:06:47.527 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-05-31T21:06:47.528 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T21:06:47.528 DEBUG:boto:Port: 8000 2023-05-31T21:06:47.529 DEBUG:boto:Params: {} 2023-05-31T21:06:47.529 DEBUG:boto:Token: None 2023-05-31T21:06:47.529 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Wed, 31 May 2023 21:06:47 GMT /pklxkd-63/obj29 2023-05-31T21:06:47.530 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:n12DIRivPPm+IGWVHMwiotYrp0s= 2023-05-31T21:06:47.530 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Wed, 31 May 2023 21:06:47 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:n12DIRivPPm+IGWVHMwiotYrp0s='} 2023-05-31T21:06:47.536 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000022fbc0d1f071983c-006477b6e7-1630-a1'), ('Date', 'Wed, 31 May 2023 21:06:47 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:06:47.536 DEBUG:boto:path=/pklxkd-63/ 2023-05-31T21:06:47.537 DEBUG:boto:auth_path=/pklxkd-63/ 2023-05-31T21:06:47.537 DEBUG:boto:Method: HEAD 2023-05-31T21:06:47.538 DEBUG:boto:Path: /pklxkd-63/ 2023-05-31T21:06:47.538 DEBUG:boto:Data: 2023-05-31T21:06:47.538 DEBUG:boto:Headers: {} 2023-05-31T21:06:47.539 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T21:06:47.539 DEBUG:boto:Port: 8000 2023-05-31T21:06:47.539 DEBUG:boto:Params: {} 2023-05-31T21:06:47.540 DEBUG:boto:Token: None 2023-05-31T21:06:47.540 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 21:06:47 GMT /pklxkd-63/ 2023-05-31T21:06:47.540 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:PDWLJ6ztPB6x60LVcukbAfyzcxQ= 2023-05-31T21:06:47.541 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 21:06:47 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:PDWLJ6ztPB6x60LVcukbAfyzcxQ=', 'Content-Length': '0'} 2023-05-31T21:06:47.544 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '21'), ('X-RGW-Bytes-Used', '63'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000006b07585032522d26-006477b6e7-1630-a1'), ('Content-Length', '0'), ('Date', 'Wed, 31 May 2023 21:06:47 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:06:47.544 DEBUG:boto:path=/pklxkd-63/obj58 2023-05-31T21:06:47.545 DEBUG:boto:auth_path=/pklxkd-63/obj58 2023-05-31T21:06:47.545 DEBUG:boto:Method: PUT 2023-05-31T21:06:47.546 DEBUG:boto:Path: /pklxkd-63/obj58 2023-05-31T21:06:47.546 DEBUG:boto:Data: 2023-05-31T21:06:47.546 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-05-31T21:06:47.547 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T21:06:47.547 DEBUG:boto:Port: 8000 2023-05-31T21:06:47.548 DEBUG:boto:Params: {} 2023-05-31T21:06:47.548 DEBUG:boto:Token: None 2023-05-31T21:06:47.549 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Wed, 31 May 2023 21:06:47 GMT /pklxkd-63/obj58 2023-05-31T21:06:47.549 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:Ag7/wh1FO6r0zI2o5UknOYdPlDs= 2023-05-31T21:06:47.549 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Wed, 31 May 2023 21:06:47 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:Ag7/wh1FO6r0zI2o5UknOYdPlDs='} 2023-05-31T21:06:47.592 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000094969ce405d469dc-006477b6e7-1630-a1'), ('Date', 'Wed, 31 May 2023 21:06:47 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:06:47.593 DEBUG:boto:path=/pklxkd-63/ 2023-05-31T21:06:47.593 DEBUG:boto:auth_path=/pklxkd-63/ 2023-05-31T21:06:47.594 DEBUG:boto:Method: HEAD 2023-05-31T21:06:47.594 DEBUG:boto:Path: /pklxkd-63/ 2023-05-31T21:06:47.594 DEBUG:boto:Data: 2023-05-31T21:06:47.595 DEBUG:boto:Headers: {} 2023-05-31T21:06:47.595 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T21:06:47.595 DEBUG:boto:Port: 8000 2023-05-31T21:06:47.596 DEBUG:boto:Params: {} 2023-05-31T21:06:47.596 DEBUG:boto:Token: None 2023-05-31T21:06:47.596 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 21:06:47 GMT /pklxkd-63/ 2023-05-31T21:06:47.597 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:PDWLJ6ztPB6x60LVcukbAfyzcxQ= 2023-05-31T21:06:47.597 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 21:06:47 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:PDWLJ6ztPB6x60LVcukbAfyzcxQ=', 'Content-Length': '0'} 2023-05-31T21:06:47.600 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '22'), ('X-RGW-Bytes-Used', '66'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000032600a2834f61e87-006477b6e7-1630-a1'), ('Content-Length', '0'), ('Date', 'Wed, 31 May 2023 21:06:47 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:06:47.600 DEBUG:boto:path=/pklxkd-63/obj87 2023-05-31T21:06:47.601 DEBUG:boto:auth_path=/pklxkd-63/obj87 2023-05-31T21:06:47.601 DEBUG:boto:Method: PUT 2023-05-31T21:06:47.602 DEBUG:boto:Path: /pklxkd-63/obj87 2023-05-31T21:06:47.602 DEBUG:boto:Data: 2023-05-31T21:06:47.602 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-05-31T21:06:47.603 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T21:06:47.603 DEBUG:boto:Port: 8000 2023-05-31T21:06:47.603 DEBUG:boto:Params: {} 2023-05-31T21:06:47.604 DEBUG:boto:Token: None 2023-05-31T21:06:47.604 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Wed, 31 May 2023 21:06:47 GMT /pklxkd-63/obj87 2023-05-31T21:06:47.604 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:DIqorMv0psvr2Biz6Mzu+2JQ5AU= 2023-05-31T21:06:47.605 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Wed, 31 May 2023 21:06:47 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:DIqorMv0psvr2Biz6Mzu+2JQ5AU='} 2023-05-31T21:06:47.610 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000cb7526c0c2b51653-006477b6e7-1630-a1'), ('Date', 'Wed, 31 May 2023 21:06:47 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:06:47.611 DEBUG:boto:path=/pklxkd-63/ 2023-05-31T21:06:47.611 DEBUG:boto:auth_path=/pklxkd-63/ 2023-05-31T21:06:47.611 DEBUG:boto:Method: HEAD 2023-05-31T21:06:47.612 DEBUG:boto:Path: /pklxkd-63/ 2023-05-31T21:06:47.612 DEBUG:boto:Data: 2023-05-31T21:06:47.612 DEBUG:boto:Headers: {} 2023-05-31T21:06:47.613 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T21:06:47.613 DEBUG:boto:Port: 8000 2023-05-31T21:06:47.613 DEBUG:boto:Params: {} 2023-05-31T21:06:47.614 DEBUG:boto:Token: None 2023-05-31T21:06:47.614 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 21:06:47 GMT /pklxkd-63/ 2023-05-31T21:06:47.615 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:PDWLJ6ztPB6x60LVcukbAfyzcxQ= 2023-05-31T21:06:47.615 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 21:06:47 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:PDWLJ6ztPB6x60LVcukbAfyzcxQ=', 'Content-Length': '0'} 2023-05-31T21:06:47.618 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '23'), ('X-RGW-Bytes-Used', '69'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000f0bf5f76852f04d5-006477b6e7-1630-a1'), ('Content-Length', '0'), ('Date', 'Wed, 31 May 2023 21:06:47 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:06:47.618 DEBUG:boto:path=/pklxkd-63/obj116 2023-05-31T21:06:47.619 DEBUG:boto:auth_path=/pklxkd-63/obj116 2023-05-31T21:06:47.619 DEBUG:boto:Method: PUT 2023-05-31T21:06:47.620 DEBUG:boto:Path: /pklxkd-63/obj116 2023-05-31T21:06:47.620 DEBUG:boto:Data: 2023-05-31T21:06:47.620 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-05-31T21:06:47.621 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T21:06:47.621 DEBUG:boto:Port: 8000 2023-05-31T21:06:47.622 DEBUG:boto:Params: {} 2023-05-31T21:06:47.622 DEBUG:boto:Token: None 2023-05-31T21:06:47.622 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Wed, 31 May 2023 21:06:47 GMT /pklxkd-63/obj116 2023-05-31T21:06:47.623 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:/l1LHH0C3O/ogvD6rYWqc4EHZm8= 2023-05-31T21:06:47.623 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Wed, 31 May 2023 21:06:47 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:/l1LHH0C3O/ogvD6rYWqc4EHZm8='} 2023-05-31T21:06:47.668 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000b47fd3bb164eab63-006477b6e7-1630-a1'), ('Date', 'Wed, 31 May 2023 21:06:47 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:06:47.669 DEBUG:boto:path=/pklxkd-63/ 2023-05-31T21:06:47.669 DEBUG:boto:auth_path=/pklxkd-63/ 2023-05-31T21:06:47.670 DEBUG:boto:Method: HEAD 2023-05-31T21:06:47.670 DEBUG:boto:Path: /pklxkd-63/ 2023-05-31T21:06:47.670 DEBUG:boto:Data: 2023-05-31T21:06:47.671 DEBUG:boto:Headers: {} 2023-05-31T21:06:47.671 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T21:06:47.671 DEBUG:boto:Port: 8000 2023-05-31T21:06:47.672 DEBUG:boto:Params: {} 2023-05-31T21:06:47.672 DEBUG:boto:Token: None 2023-05-31T21:06:47.673 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 21:06:47 GMT /pklxkd-63/ 2023-05-31T21:06:47.673 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:PDWLJ6ztPB6x60LVcukbAfyzcxQ= 2023-05-31T21:06:47.673 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 21:06:47 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:PDWLJ6ztPB6x60LVcukbAfyzcxQ=', 'Content-Length': '0'} 2023-05-31T21:06:47.676 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '24'), ('X-RGW-Bytes-Used', '72'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000007ba0f2e718d47479-006477b6e7-1630-a1'), ('Content-Length', '0'), ('Date', 'Wed, 31 May 2023 21:06:47 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:06:47.677 DEBUG:boto:path=/pklxkd-63/obj145 2023-05-31T21:06:47.677 DEBUG:boto:auth_path=/pklxkd-63/obj145 2023-05-31T21:06:47.678 DEBUG:boto:Method: PUT 2023-05-31T21:06:47.678 DEBUG:boto:Path: /pklxkd-63/obj145 2023-05-31T21:06:47.678 DEBUG:boto:Data: 2023-05-31T21:06:47.679 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-05-31T21:06:47.679 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T21:06:47.679 DEBUG:boto:Port: 8000 2023-05-31T21:06:47.680 DEBUG:boto:Params: {} 2023-05-31T21:06:47.680 DEBUG:boto:Token: None 2023-05-31T21:06:47.681 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Wed, 31 May 2023 21:06:47 GMT /pklxkd-63/obj145 2023-05-31T21:06:47.681 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:v6SA4aeiaMf1fHI0ciY61CF5XPo= 2023-05-31T21:06:47.681 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Wed, 31 May 2023 21:06:47 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:v6SA4aeiaMf1fHI0ciY61CF5XPo='} 2023-05-31T21:06:47.686 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000008cc5a189d2ccabd4-006477b6e7-1630-a1'), ('Date', 'Wed, 31 May 2023 21:06:47 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:06:47.687 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'reshard', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'pklxkd-63', '--num-shards', '31', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-05-31T21:06:47.687 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'reshard', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'pklxkd-63', '--num-shards', '31', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-05-31T21:06:47.688 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket reshard --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --bucket pklxkd-63 --num-shards 31 --yes-i-really-mean-it --cluster c1 --debug-rgw 0 --debug-ms 0 2023-05-31T21:06:47.740 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2023-05-31T21:06:47.740 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2023-05-31T21:06:47.797 INFO:teuthology.orchestra.run.smithi099.stdout:tenant: 2023-05-31T21:06:47.797 INFO:teuthology.orchestra.run.smithi099.stdout:bucket name: pklxkd-63 2023-05-31T21:06:47.807 INFO:teuthology.orchestra.run.smithi099.stdout:total entries: 25 2023-05-31T21:06:47.892 DEBUG:boto:path=/pklxkd-63/ 2023-05-31T21:06:47.892 DEBUG:boto:auth_path=/pklxkd-63/ 2023-05-31T21:06:47.893 DEBUG:boto:Method: HEAD 2023-05-31T21:06:47.893 DEBUG:boto:Path: /pklxkd-63/ 2023-05-31T21:06:47.893 DEBUG:boto:Data: 2023-05-31T21:06:47.894 DEBUG:boto:Headers: {} 2023-05-31T21:06:47.894 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T21:06:47.894 DEBUG:boto:Port: 8000 2023-05-31T21:06:47.895 DEBUG:boto:Params: {} 2023-05-31T21:06:47.895 DEBUG:boto:Token: None 2023-05-31T21:06:47.896 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 21:06:47 GMT /pklxkd-63/ 2023-05-31T21:06:47.896 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:PDWLJ6ztPB6x60LVcukbAfyzcxQ= 2023-05-31T21:06:47.896 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 21:06:47 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:PDWLJ6ztPB6x60LVcukbAfyzcxQ=', 'Content-Length': '0'} 2023-05-31T21:06:47.900 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '25'), ('X-RGW-Bytes-Used', '75'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000001d107802d42c9954-006477b6e7-1630-a1'), ('Content-Length', '0'), ('Date', 'Wed, 31 May 2023 21:06:47 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:06:47.900 DEBUG:boto:path=/pklxkd-63/obj31 2023-05-31T21:06:47.901 DEBUG:boto:auth_path=/pklxkd-63/obj31 2023-05-31T21:06:47.901 DEBUG:boto:Method: PUT 2023-05-31T21:06:47.902 DEBUG:boto:Path: /pklxkd-63/obj31 2023-05-31T21:06:47.902 DEBUG:boto:Data: 2023-05-31T21:06:47.902 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-05-31T21:06:47.903 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T21:06:47.903 DEBUG:boto:Port: 8000 2023-05-31T21:06:47.903 DEBUG:boto:Params: {} 2023-05-31T21:06:47.904 DEBUG:boto:Token: None 2023-05-31T21:06:47.904 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Wed, 31 May 2023 21:06:47 GMT /pklxkd-63/obj31 2023-05-31T21:06:47.904 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:zrhkmI62RpvkHDJqCk77h0wa8hM= 2023-05-31T21:06:47.905 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Wed, 31 May 2023 21:06:47 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:zrhkmI62RpvkHDJqCk77h0wa8hM='} 2023-05-31T21:06:47.911 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000004cea157904a87577-006477b6e7-1630-a1'), ('Date', 'Wed, 31 May 2023 21:06:47 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:06:47.911 DEBUG:boto:path=/pklxkd-63/ 2023-05-31T21:06:47.911 DEBUG:boto:auth_path=/pklxkd-63/ 2023-05-31T21:06:47.912 DEBUG:boto:Method: HEAD 2023-05-31T21:06:47.912 DEBUG:boto:Path: /pklxkd-63/ 2023-05-31T21:06:47.912 DEBUG:boto:Data: 2023-05-31T21:06:47.913 DEBUG:boto:Headers: {} 2023-05-31T21:06:47.913 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T21:06:47.913 DEBUG:boto:Port: 8000 2023-05-31T21:06:47.914 DEBUG:boto:Params: {} 2023-05-31T21:06:47.914 DEBUG:boto:Token: None 2023-05-31T21:06:47.915 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 21:06:47 GMT /pklxkd-63/ 2023-05-31T21:06:47.915 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:PDWLJ6ztPB6x60LVcukbAfyzcxQ= 2023-05-31T21:06:47.915 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 21:06:47 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:PDWLJ6ztPB6x60LVcukbAfyzcxQ=', 'Content-Length': '0'} 2023-05-31T21:06:47.918 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '26'), ('X-RGW-Bytes-Used', '78'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000ed54f60914797f2d-006477b6e7-1630-a1'), ('Content-Length', '0'), ('Date', 'Wed, 31 May 2023 21:06:47 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:06:47.919 DEBUG:boto:path=/pklxkd-63/obj62 2023-05-31T21:06:47.919 DEBUG:boto:auth_path=/pklxkd-63/obj62 2023-05-31T21:06:47.920 DEBUG:boto:Method: PUT 2023-05-31T21:06:47.920 DEBUG:boto:Path: /pklxkd-63/obj62 2023-05-31T21:06:47.920 DEBUG:boto:Data: 2023-05-31T21:06:47.921 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-05-31T21:06:47.921 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T21:06:47.921 DEBUG:boto:Port: 8000 2023-05-31T21:06:47.922 DEBUG:boto:Params: {} 2023-05-31T21:06:47.922 DEBUG:boto:Token: None 2023-05-31T21:06:47.923 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Wed, 31 May 2023 21:06:47 GMT /pklxkd-63/obj62 2023-05-31T21:06:47.923 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:Uv+vO4o+BopQS9/BJ3j56lpa4L8= 2023-05-31T21:06:47.924 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Wed, 31 May 2023 21:06:47 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:Uv+vO4o+BopQS9/BJ3j56lpa4L8='} 2023-05-31T21:06:47.968 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000a8c66f26d43f8eef-006477b6e7-1630-a1'), ('Date', 'Wed, 31 May 2023 21:06:47 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:06:47.969 DEBUG:boto:path=/pklxkd-63/ 2023-05-31T21:06:47.969 DEBUG:boto:auth_path=/pklxkd-63/ 2023-05-31T21:06:47.969 DEBUG:boto:Method: HEAD 2023-05-31T21:06:47.970 DEBUG:boto:Path: /pklxkd-63/ 2023-05-31T21:06:47.970 DEBUG:boto:Data: 2023-05-31T21:06:47.970 DEBUG:boto:Headers: {} 2023-05-31T21:06:47.971 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T21:06:47.971 DEBUG:boto:Port: 8000 2023-05-31T21:06:47.971 DEBUG:boto:Params: {} 2023-05-31T21:06:47.972 DEBUG:boto:Token: None 2023-05-31T21:06:47.972 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 21:06:47 GMT /pklxkd-63/ 2023-05-31T21:06:47.973 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:PDWLJ6ztPB6x60LVcukbAfyzcxQ= 2023-05-31T21:06:47.973 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 21:06:47 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:PDWLJ6ztPB6x60LVcukbAfyzcxQ=', 'Content-Length': '0'} 2023-05-31T21:06:47.976 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '27'), ('X-RGW-Bytes-Used', '81'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000003f12ddc3e2f02aa6-006477b6e7-1630-a1'), ('Content-Length', '0'), ('Date', 'Wed, 31 May 2023 21:06:47 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:06:47.977 DEBUG:boto:path=/pklxkd-63/obj93 2023-05-31T21:06:47.977 DEBUG:boto:auth_path=/pklxkd-63/obj93 2023-05-31T21:06:47.977 DEBUG:boto:Method: PUT 2023-05-31T21:06:47.978 DEBUG:boto:Path: /pklxkd-63/obj93 2023-05-31T21:06:47.978 DEBUG:boto:Data: 2023-05-31T21:06:47.979 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-05-31T21:06:47.979 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T21:06:47.979 DEBUG:boto:Port: 8000 2023-05-31T21:06:47.980 DEBUG:boto:Params: {} 2023-05-31T21:06:47.980 DEBUG:boto:Token: None 2023-05-31T21:06:47.980 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Wed, 31 May 2023 21:06:47 GMT /pklxkd-63/obj93 2023-05-31T21:06:47.981 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:+c2QpvLlf0odtZW94vsbDts/7+0= 2023-05-31T21:06:47.981 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Wed, 31 May 2023 21:06:47 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:+c2QpvLlf0odtZW94vsbDts/7+0='} 2023-05-31T21:06:47.987 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000002c83ed27607e74e0-006477b6e7-1630-a1'), ('Date', 'Wed, 31 May 2023 21:06:47 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:06:47.987 DEBUG:boto:path=/pklxkd-63/ 2023-05-31T21:06:47.988 DEBUG:boto:auth_path=/pklxkd-63/ 2023-05-31T21:06:47.988 DEBUG:boto:Method: HEAD 2023-05-31T21:06:47.988 DEBUG:boto:Path: /pklxkd-63/ 2023-05-31T21:06:47.989 DEBUG:boto:Data: 2023-05-31T21:06:47.989 DEBUG:boto:Headers: {} 2023-05-31T21:06:47.989 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T21:06:47.990 DEBUG:boto:Port: 8000 2023-05-31T21:06:47.990 DEBUG:boto:Params: {} 2023-05-31T21:06:47.990 DEBUG:boto:Token: None 2023-05-31T21:06:47.991 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 21:06:47 GMT /pklxkd-63/ 2023-05-31T21:06:47.991 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:PDWLJ6ztPB6x60LVcukbAfyzcxQ= 2023-05-31T21:06:47.992 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 21:06:47 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:PDWLJ6ztPB6x60LVcukbAfyzcxQ=', 'Content-Length': '0'} 2023-05-31T21:06:47.994 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '28'), ('X-RGW-Bytes-Used', '84'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000009659443923b9493c-006477b6e7-1630-a1'), ('Content-Length', '0'), ('Date', 'Wed, 31 May 2023 21:06:47 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:06:47.995 DEBUG:boto:path=/pklxkd-63/obj124 2023-05-31T21:06:47.995 DEBUG:boto:auth_path=/pklxkd-63/obj124 2023-05-31T21:06:47.996 DEBUG:boto:Method: PUT 2023-05-31T21:06:47.996 DEBUG:boto:Path: /pklxkd-63/obj124 2023-05-31T21:06:47.996 DEBUG:boto:Data: 2023-05-31T21:06:47.997 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-05-31T21:06:47.997 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T21:06:47.997 DEBUG:boto:Port: 8000 2023-05-31T21:06:47.998 DEBUG:boto:Params: {} 2023-05-31T21:06:47.998 DEBUG:boto:Token: None 2023-05-31T21:06:47.999 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Wed, 31 May 2023 21:06:47 GMT /pklxkd-63/obj124 2023-05-31T21:06:47.999 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:SSajoq1ocXBjvYoK3elOWblnKqE= 2023-05-31T21:06:47.999 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Wed, 31 May 2023 21:06:47 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:SSajoq1ocXBjvYoK3elOWblnKqE='} 2023-05-31T21:06:48.005 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000b59fb54ab9f2902b-006477b6e7-1630-a1'), ('Date', 'Wed, 31 May 2023 21:06:48 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:06:48.005 DEBUG:boto:path=/pklxkd-63/ 2023-05-31T21:06:48.005 DEBUG:boto:auth_path=/pklxkd-63/ 2023-05-31T21:06:48.006 DEBUG:boto:Method: HEAD 2023-05-31T21:06:48.006 DEBUG:boto:Path: /pklxkd-63/ 2023-05-31T21:06:48.006 DEBUG:boto:Data: 2023-05-31T21:06:48.007 DEBUG:boto:Headers: {} 2023-05-31T21:06:48.007 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T21:06:48.007 DEBUG:boto:Port: 8000 2023-05-31T21:06:48.008 DEBUG:boto:Params: {} 2023-05-31T21:06:48.008 DEBUG:boto:Token: None 2023-05-31T21:06:48.009 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 21:06:48 GMT /pklxkd-63/ 2023-05-31T21:06:48.009 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:3B2mkaM+ObC4zRGdBhJqL6TchVg= 2023-05-31T21:06:48.009 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 21:06:48 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:3B2mkaM+ObC4zRGdBhJqL6TchVg=', 'Content-Length': '0'} 2023-05-31T21:06:48.012 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '29'), ('X-RGW-Bytes-Used', '87'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000e536e172573e3e33-006477b6e8-1630-a1'), ('Content-Length', '0'), ('Date', 'Wed, 31 May 2023 21:06:48 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:06:48.012 DEBUG:boto:path=/pklxkd-63/obj155 2023-05-31T21:06:48.013 DEBUG:boto:auth_path=/pklxkd-63/obj155 2023-05-31T21:06:48.013 DEBUG:boto:Method: PUT 2023-05-31T21:06:48.014 DEBUG:boto:Path: /pklxkd-63/obj155 2023-05-31T21:06:48.014 DEBUG:boto:Data: 2023-05-31T21:06:48.014 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-05-31T21:06:48.015 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T21:06:48.015 DEBUG:boto:Port: 8000 2023-05-31T21:06:48.015 DEBUG:boto:Params: {} 2023-05-31T21:06:48.016 DEBUG:boto:Token: None 2023-05-31T21:06:48.016 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Wed, 31 May 2023 21:06:48 GMT /pklxkd-63/obj155 2023-05-31T21:06:48.016 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:dcpNYhcIb8JZOiCJOlV6lv0aBkE= 2023-05-31T21:06:48.017 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Wed, 31 May 2023 21:06:48 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:dcpNYhcIb8JZOiCJOlV6lv0aBkE='} 2023-05-31T21:06:48.060 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000dd7ea80d569ef016-006477b6e8-1630-a1'), ('Date', 'Wed, 31 May 2023 21:06:48 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:06:48.061 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'reshard', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'pklxkd-63', '--num-shards', '37', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-05-31T21:06:48.061 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'reshard', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'pklxkd-63', '--num-shards', '37', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-05-31T21:06:48.062 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket reshard --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --bucket pklxkd-63 --num-shards 37 --yes-i-really-mean-it --cluster c1 --debug-rgw 0 --debug-ms 0 2023-05-31T21:06:48.115 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2023-05-31T21:06:48.116 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2023-05-31T21:06:48.177 INFO:teuthology.orchestra.run.smithi099.stdout:tenant: 2023-05-31T21:06:48.178 INFO:teuthology.orchestra.run.smithi099.stdout:bucket name: pklxkd-63 2023-05-31T21:06:48.188 INFO:teuthology.orchestra.run.smithi099.stdout:total entries: 30 2023-05-31T21:06:48.279 DEBUG:boto:path=/pklxkd-63/ 2023-05-31T21:06:48.280 DEBUG:boto:auth_path=/pklxkd-63/ 2023-05-31T21:06:48.280 DEBUG:boto:Method: HEAD 2023-05-31T21:06:48.281 DEBUG:boto:Path: /pklxkd-63/ 2023-05-31T21:06:48.281 DEBUG:boto:Data: 2023-05-31T21:06:48.281 DEBUG:boto:Headers: {} 2023-05-31T21:06:48.282 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T21:06:48.282 DEBUG:boto:Port: 8000 2023-05-31T21:06:48.282 DEBUG:boto:Params: {} 2023-05-31T21:06:48.283 DEBUG:boto:Token: None 2023-05-31T21:06:48.283 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 21:06:48 GMT /pklxkd-63/ 2023-05-31T21:06:48.284 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:3B2mkaM+ObC4zRGdBhJqL6TchVg= 2023-05-31T21:06:48.284 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 21:06:48 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:3B2mkaM+ObC4zRGdBhJqL6TchVg=', 'Content-Length': '0'} 2023-05-31T21:06:48.288 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '30'), ('X-RGW-Bytes-Used', '90'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000e18f9d21b0ef49af-006477b6e8-1630-a1'), ('Content-Length', '0'), ('Date', 'Wed, 31 May 2023 21:06:48 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:06:48.288 DEBUG:boto:path=/pklxkd-63/obj37 2023-05-31T21:06:48.289 DEBUG:boto:auth_path=/pklxkd-63/obj37 2023-05-31T21:06:48.289 DEBUG:boto:Method: PUT 2023-05-31T21:06:48.289 DEBUG:boto:Path: /pklxkd-63/obj37 2023-05-31T21:06:48.290 DEBUG:boto:Data: 2023-05-31T21:06:48.290 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-05-31T21:06:48.290 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T21:06:48.291 DEBUG:boto:Port: 8000 2023-05-31T21:06:48.291 DEBUG:boto:Params: {} 2023-05-31T21:06:48.292 DEBUG:boto:Token: None 2023-05-31T21:06:48.292 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Wed, 31 May 2023 21:06:48 GMT /pklxkd-63/obj37 2023-05-31T21:06:48.292 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:+3UrOJNnZc3yQnfzzTyY32/f1g4= 2023-05-31T21:06:48.293 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Wed, 31 May 2023 21:06:48 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:+3UrOJNnZc3yQnfzzTyY32/f1g4='} 2023-05-31T21:06:48.298 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000ed0eba0a8f78ebd4-006477b6e8-1630-a1'), ('Date', 'Wed, 31 May 2023 21:06:48 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:06:48.299 DEBUG:boto:path=/pklxkd-63/ 2023-05-31T21:06:48.299 DEBUG:boto:auth_path=/pklxkd-63/ 2023-05-31T21:06:48.300 DEBUG:boto:Method: HEAD 2023-05-31T21:06:48.300 DEBUG:boto:Path: /pklxkd-63/ 2023-05-31T21:06:48.300 DEBUG:boto:Data: 2023-05-31T21:06:48.301 DEBUG:boto:Headers: {} 2023-05-31T21:06:48.301 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T21:06:48.301 DEBUG:boto:Port: 8000 2023-05-31T21:06:48.302 DEBUG:boto:Params: {} 2023-05-31T21:06:48.302 DEBUG:boto:Token: None 2023-05-31T21:06:48.302 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 21:06:48 GMT /pklxkd-63/ 2023-05-31T21:06:48.303 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:3B2mkaM+ObC4zRGdBhJqL6TchVg= 2023-05-31T21:06:48.303 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 21:06:48 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:3B2mkaM+ObC4zRGdBhJqL6TchVg=', 'Content-Length': '0'} 2023-05-31T21:06:48.306 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '31'), ('X-RGW-Bytes-Used', '93'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000011ac91a76a6b24b2-006477b6e8-1630-a1'), ('Content-Length', '0'), ('Date', 'Wed, 31 May 2023 21:06:48 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:06:48.307 DEBUG:boto:path=/pklxkd-63/obj74 2023-05-31T21:06:48.307 DEBUG:boto:auth_path=/pklxkd-63/obj74 2023-05-31T21:06:48.308 DEBUG:boto:Method: PUT 2023-05-31T21:06:48.308 DEBUG:boto:Path: /pklxkd-63/obj74 2023-05-31T21:06:48.308 DEBUG:boto:Data: 2023-05-31T21:06:48.309 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-05-31T21:06:48.309 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T21:06:48.309 DEBUG:boto:Port: 8000 2023-05-31T21:06:48.310 DEBUG:boto:Params: {} 2023-05-31T21:06:48.310 DEBUG:boto:Token: None 2023-05-31T21:06:48.310 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Wed, 31 May 2023 21:06:48 GMT /pklxkd-63/obj74 2023-05-31T21:06:48.311 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:pOkmsj/3yR+lv0vPx7yWqoR4EsA= 2023-05-31T21:06:48.311 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Wed, 31 May 2023 21:06:48 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:pOkmsj/3yR+lv0vPx7yWqoR4EsA='} 2023-05-31T21:06:48.316 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000a23e22eeb1d8d80c-006477b6e8-1630-a1'), ('Date', 'Wed, 31 May 2023 21:06:48 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:06:48.317 DEBUG:boto:path=/pklxkd-63/ 2023-05-31T21:06:48.317 DEBUG:boto:auth_path=/pklxkd-63/ 2023-05-31T21:06:48.317 DEBUG:boto:Method: HEAD 2023-05-31T21:06:48.318 DEBUG:boto:Path: /pklxkd-63/ 2023-05-31T21:06:48.318 DEBUG:boto:Data: 2023-05-31T21:06:48.318 DEBUG:boto:Headers: {} 2023-05-31T21:06:48.319 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T21:06:48.319 DEBUG:boto:Port: 8000 2023-05-31T21:06:48.319 DEBUG:boto:Params: {} 2023-05-31T21:06:48.320 DEBUG:boto:Token: None 2023-05-31T21:06:48.320 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 21:06:48 GMT /pklxkd-63/ 2023-05-31T21:06:48.321 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:3B2mkaM+ObC4zRGdBhJqL6TchVg= 2023-05-31T21:06:48.321 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 21:06:48 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:3B2mkaM+ObC4zRGdBhJqL6TchVg=', 'Content-Length': '0'} 2023-05-31T21:06:48.324 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '32'), ('X-RGW-Bytes-Used', '96'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000000f247c948c4c519b-006477b6e8-1630-a1'), ('Content-Length', '0'), ('Date', 'Wed, 31 May 2023 21:06:48 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:06:48.325 DEBUG:boto:path=/pklxkd-63/obj111 2023-05-31T21:06:48.325 DEBUG:boto:auth_path=/pklxkd-63/obj111 2023-05-31T21:06:48.325 DEBUG:boto:Method: PUT 2023-05-31T21:06:48.326 DEBUG:boto:Path: /pklxkd-63/obj111 2023-05-31T21:06:48.326 DEBUG:boto:Data: 2023-05-31T21:06:48.327 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-05-31T21:06:48.327 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T21:06:48.327 DEBUG:boto:Port: 8000 2023-05-31T21:06:48.328 DEBUG:boto:Params: {} 2023-05-31T21:06:48.328 DEBUG:boto:Token: None 2023-05-31T21:06:48.329 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Wed, 31 May 2023 21:06:48 GMT /pklxkd-63/obj111 2023-05-31T21:06:48.329 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:yu9VqeI6cTKyuU+Vh1K/vXT2aDg= 2023-05-31T21:06:48.329 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Wed, 31 May 2023 21:06:48 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:yu9VqeI6cTKyuU+Vh1K/vXT2aDg='} 2023-05-31T21:06:48.372 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000000191aba87ef31b77-006477b6e8-1630-a1'), ('Date', 'Wed, 31 May 2023 21:06:48 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:06:48.373 DEBUG:boto:path=/pklxkd-63/ 2023-05-31T21:06:48.373 DEBUG:boto:auth_path=/pklxkd-63/ 2023-05-31T21:06:48.373 DEBUG:boto:Method: HEAD 2023-05-31T21:06:48.374 DEBUG:boto:Path: /pklxkd-63/ 2023-05-31T21:06:48.374 DEBUG:boto:Data: 2023-05-31T21:06:48.375 DEBUG:boto:Headers: {} 2023-05-31T21:06:48.375 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T21:06:48.375 DEBUG:boto:Port: 8000 2023-05-31T21:06:48.375 DEBUG:boto:Params: {} 2023-05-31T21:06:48.376 DEBUG:boto:Token: None 2023-05-31T21:06:48.376 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 21:06:48 GMT /pklxkd-63/ 2023-05-31T21:06:48.377 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:3B2mkaM+ObC4zRGdBhJqL6TchVg= 2023-05-31T21:06:48.377 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 21:06:48 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:3B2mkaM+ObC4zRGdBhJqL6TchVg=', 'Content-Length': '0'} 2023-05-31T21:06:48.380 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '33'), ('X-RGW-Bytes-Used', '99'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000e9c4c11d41b56b4d-006477b6e8-1630-a1'), ('Content-Length', '0'), ('Date', 'Wed, 31 May 2023 21:06:48 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:06:48.381 DEBUG:boto:path=/pklxkd-63/obj148 2023-05-31T21:06:48.381 DEBUG:boto:auth_path=/pklxkd-63/obj148 2023-05-31T21:06:48.381 DEBUG:boto:Method: PUT 2023-05-31T21:06:48.382 DEBUG:boto:Path: /pklxkd-63/obj148 2023-05-31T21:06:48.382 DEBUG:boto:Data: 2023-05-31T21:06:48.382 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-05-31T21:06:48.383 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T21:06:48.383 DEBUG:boto:Port: 8000 2023-05-31T21:06:48.383 DEBUG:boto:Params: {} 2023-05-31T21:06:48.384 DEBUG:boto:Token: None 2023-05-31T21:06:48.384 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Wed, 31 May 2023 21:06:48 GMT /pklxkd-63/obj148 2023-05-31T21:06:48.385 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:4pS9svMLwTYBqPd7WZVhl3vZBK4= 2023-05-31T21:06:48.385 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Wed, 31 May 2023 21:06:48 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:4pS9svMLwTYBqPd7WZVhl3vZBK4='} 2023-05-31T21:06:48.390 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000d40f8aa5c0746ffe-006477b6e8-1630-a1'), ('Date', 'Wed, 31 May 2023 21:06:48 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:06:48.391 DEBUG:boto:path=/pklxkd-63/ 2023-05-31T21:06:48.391 DEBUG:boto:auth_path=/pklxkd-63/ 2023-05-31T21:06:48.392 DEBUG:boto:Method: HEAD 2023-05-31T21:06:48.392 DEBUG:boto:Path: /pklxkd-63/ 2023-05-31T21:06:48.392 DEBUG:boto:Data: 2023-05-31T21:06:48.393 DEBUG:boto:Headers: {} 2023-05-31T21:06:48.393 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T21:06:48.393 DEBUG:boto:Port: 8000 2023-05-31T21:06:48.394 DEBUG:boto:Params: {} 2023-05-31T21:06:48.394 DEBUG:boto:Token: None 2023-05-31T21:06:48.394 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 21:06:48 GMT /pklxkd-63/ 2023-05-31T21:06:48.395 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:3B2mkaM+ObC4zRGdBhJqL6TchVg= 2023-05-31T21:06:48.395 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 21:06:48 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:3B2mkaM+ObC4zRGdBhJqL6TchVg=', 'Content-Length': '0'} 2023-05-31T21:06:48.398 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '34'), ('X-RGW-Bytes-Used', '102'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000094aac4135bb50c2c-006477b6e8-1630-a1'), ('Content-Length', '0'), ('Date', 'Wed, 31 May 2023 21:06:48 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:06:48.399 DEBUG:boto:path=/pklxkd-63/obj185 2023-05-31T21:06:48.400 DEBUG:boto:auth_path=/pklxkd-63/obj185 2023-05-31T21:06:48.400 DEBUG:boto:Method: PUT 2023-05-31T21:06:48.400 DEBUG:boto:Path: /pklxkd-63/obj185 2023-05-31T21:06:48.401 DEBUG:boto:Data: 2023-05-31T21:06:48.401 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-05-31T21:06:48.401 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T21:06:48.402 DEBUG:boto:Port: 8000 2023-05-31T21:06:48.402 DEBUG:boto:Params: {} 2023-05-31T21:06:48.402 DEBUG:boto:Token: None 2023-05-31T21:06:48.403 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Wed, 31 May 2023 21:06:48 GMT /pklxkd-63/obj185 2023-05-31T21:06:48.403 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:tYJpToOQdPMRyxyyp7JpYZOL5RA= 2023-05-31T21:06:48.404 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Wed, 31 May 2023 21:06:48 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:tYJpToOQdPMRyxyyp7JpYZOL5RA='} 2023-05-31T21:06:48.448 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000000476126fb590a8f4-006477b6e8-1630-a1'), ('Date', 'Wed, 31 May 2023 21:06:48 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:06:48.449 DEBUG:rgw_multi.tests:bucket checkpoint: source=a1 target=a2 bucket=pklxkd-63 2023-05-31T21:06:48.449 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'pklxkd-63', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-05-31T21:06:48.450 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'pklxkd-63', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-05-31T21:06:48.450 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket pklxkd-63 --source-zone a1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2023-05-31T21:06:48.506 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2023-05-31T21:06:48.506 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2023-05-31T21:06:48.558 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T21:06:48.553+0000 7f11993d0780 1 waiting to reach incremental sync.. 2023-05-31T21:06:53.560 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T21:06:53.556+0000 7f11993d0780 1 bucket sync caught up with source: 2023-05-31T21:06:53.561 INFO:teuthology.orchestra.run.smithi099.stderr: local status: [, , , , , , 00000000002.94.6, , , , , , 00000000002.85.6, , , , , 00000000002.85.6, , , , , , , , , , , , , , , 00000000001.110.6, , , , 00000000001.83.6] 2023-05-31T21:06:53.561 INFO:teuthology.orchestra.run.smithi099.stderr: remote markers: [, , , , , , 00000000002.94.6, , , , , , 00000000002.85.6, , , , , 00000000002.85.6, , , , , , , , , , , , , , , 00000000001.110.6, , , , 00000000001.83.6] 2023-05-31T21:06:53.561 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T21:06:53.556+0000 7f11993d0780 0 bucket checkpoint complete 2023-05-31T21:06:53.574 DEBUG:rgw_multi.tests:bucket checkpoint: source=a2 target=a1 bucket=pklxkd-63 2023-05-31T21:06:53.574 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'pklxkd-63', '--source-zone', 'a2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-05-31T21:06:53.575 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'pklxkd-63', '--source-zone', 'a2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-05-31T21:06:53.575 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket pklxkd-63 --source-zone a2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2023-05-31T21:06:53.631 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2023-05-31T21:06:53.631 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2023-05-31T21:06:53.680 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T21:06:53.675+0000 7fcadbe43780 1 waiting to reach incremental sync.. 2023-05-31T21:06:58.681 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T21:06:58.676+0000 7fcadbe43780 1 waiting to reach incremental sync.. 2023-05-31T21:07:03.681 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T21:07:03.677+0000 7fcadbe43780 1 waiting to reach incremental sync.. 2023-05-31T21:07:08.682 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T21:07:08.678+0000 7fcadbe43780 1 waiting to reach incremental sync.. 2023-05-31T21:07:13.683 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T21:07:13.679+0000 7fcadbe43780 1 waiting to reach incremental sync.. 2023-05-31T21:07:18.683 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T21:07:18.679+0000 7fcadbe43780 1 waiting to reach incremental sync.. 2023-05-31T21:07:23.684 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T21:07:23.680+0000 7fcadbe43780 1 waiting to reach incremental sync.. 2023-05-31T21:07:28.684 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T21:07:28.680+0000 7fcadbe43780 1 waiting to reach incremental sync.. 2023-05-31T21:07:33.685 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T21:07:33.680+0000 7fcadbe43780 1 waiting to reach incremental sync.. 2023-05-31T21:07:38.686 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T21:07:38.681+0000 7fcadbe43780 1 waiting to reach incremental sync.. 2023-05-31T21:07:43.686 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T21:07:43.682+0000 7fcadbe43780 1 waiting to reach incremental sync.. 2023-05-31T21:07:48.687 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T21:07:48.683+0000 7fcadbe43780 1 waiting to reach incremental sync.. 2023-05-31T21:07:53.688 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T21:07:53.684+0000 7fcadbe43780 1 waiting to reach incremental sync.. 2023-05-31T21:07:58.688 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T21:07:58.685+0000 7fcadbe43780 1 waiting to reach incremental sync.. 2023-05-31T21:08:03.689 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T21:08:03.685+0000 7fcadbe43780 1 waiting to reach incremental sync.. 2023-05-31T21:08:08.690 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T21:08:08.685+0000 7fcadbe43780 1 waiting to reach incremental sync.. 2023-05-31T21:08:13.690 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T21:08:13.686+0000 7fcadbe43780 1 waiting to reach incremental sync.. 2023-05-31T21:08:18.691 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T21:08:18.687+0000 7fcadbe43780 1 waiting to reach incremental sync.. 2023-05-31T21:08:23.692 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T21:08:23.687+0000 7fcadbe43780 1 waiting to reach incremental sync.. 2023-05-31T21:08:28.693 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T21:08:28.689+0000 7fcadbe43780 1 waiting to reach incremental sync.. 2023-05-31T21:08:33.693 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T21:08:33.690+0000 7fcadbe43780 1 waiting to reach incremental sync.. 2023-05-31T21:08:38.695 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T21:08:38.691+0000 7fcadbe43780 1 waiting to reach incremental sync.. 2023-05-31T21:08:43.695 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T21:08:43.691+0000 7fcadbe43780 1 waiting to reach incremental sync.. 2023-05-31T21:08:48.696 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T21:08:48.691+0000 7fcadbe43780 1 waiting to reach incremental sync.. 2023-05-31T21:08:53.696 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T21:08:53.692+0000 7fcadbe43780 1 waiting to reach incremental sync.. 2023-05-31T21:08:58.697 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T21:08:58.692+0000 7fcadbe43780 1 waiting to reach incremental sync.. 2023-05-31T21:09:03.697 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T21:09:03.693+0000 7fcadbe43780 1 waiting to reach incremental sync.. 2023-05-31T21:09:08.698 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T21:09:08.694+0000 7fcadbe43780 1 waiting to reach incremental sync.. 2023-05-31T21:09:13.699 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T21:09:13.695+0000 7fcadbe43780 1 waiting to reach incremental sync.. 2023-05-31T21:09:18.700 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T21:09:18.696+0000 7fcadbe43780 1 waiting to reach incremental sync.. 2023-05-31T21:09:23.700 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T21:09:23.697+0000 7fcadbe43780 1 waiting to reach incremental sync.. 2023-05-31T21:09:28.701 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T21:09:28.697+0000 7fcadbe43780 1 waiting to reach incremental sync.. 2023-05-31T21:09:33.701 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T21:09:33.697+0000 7fcadbe43780 1 waiting to reach incremental sync.. 2023-05-31T21:09:38.702 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T21:09:38.698+0000 7fcadbe43780 1 waiting to reach incremental sync.. 2023-05-31T21:09:43.703 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T21:09:43.698+0000 7fcadbe43780 1 waiting to reach incremental sync.. 2023-05-31T21:09:48.703 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T21:09:48.699+0000 7fcadbe43780 1 waiting to reach incremental sync.. 2023-05-31T21:09:53.704 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T21:09:53.700+0000 7fcadbe43780 1 waiting to reach incremental sync.. 2023-05-31T21:09:58.704 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T21:09:58.701+0000 7fcadbe43780 1 waiting to reach incremental sync.. 2023-05-31T21:10:03.705 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T21:10:03.702+0000 7fcadbe43780 1 waiting to reach incremental sync.. 2023-05-31T21:10:08.706 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T21:10:08.702+0000 7fcadbe43780 1 waiting to reach incremental sync.. 2023-05-31T21:10:13.706 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T21:10:13.702+0000 7fcadbe43780 1 waiting to reach incremental sync.. 2023-05-31T21:10:18.707 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T21:10:18.702+0000 7fcadbe43780 1 waiting to reach incremental sync.. 2023-05-31T21:10:23.708 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T21:10:23.703+0000 7fcadbe43780 1 waiting to reach incremental sync.. 2023-05-31T21:10:28.708 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T21:10:28.704+0000 7fcadbe43780 1 waiting to reach incremental sync.. 2023-05-31T21:10:33.709 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T21:10:33.705+0000 7fcadbe43780 1 waiting to reach incremental sync.. 2023-05-31T21:10:38.710 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T21:10:38.706+0000 7fcadbe43780 1 waiting to reach incremental sync.. 2023-05-31T21:10:43.710 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T21:10:43.707+0000 7fcadbe43780 1 waiting to reach incremental sync.. 2023-05-31T21:10:48.711 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T21:10:48.707+0000 7fcadbe43780 1 waiting to reach incremental sync.. 2023-05-31T21:10:53.711 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T21:10:53.708+0000 7fcadbe43780 1 waiting to reach incremental sync.. 2023-05-31T21:10:58.712 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T21:10:58.707+0000 7fcadbe43780 1 waiting to reach incremental sync.. 2023-05-31T21:11:03.713 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T21:11:03.708+0000 7fcadbe43780 1 waiting to reach incremental sync.. 2023-05-31T21:11:08.713 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T21:11:08.709+0000 7fcadbe43780 1 waiting to reach incremental sync.. 2023-05-31T21:11:13.714 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T21:11:13.710+0000 7fcadbe43780 1 waiting to reach incremental sync.. 2023-05-31T21:11:18.715 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T21:11:18.711+0000 7fcadbe43780 1 waiting to reach incremental sync.. 2023-05-31T21:11:23.715 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T21:11:23.712+0000 7fcadbe43780 1 waiting to reach incremental sync.. 2023-05-31T21:11:28.716 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T21:11:28.712+0000 7fcadbe43780 1 waiting to reach incremental sync.. 2023-05-31T21:11:33.717 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T21:11:33.713+0000 7fcadbe43780 1 waiting to reach incremental sync.. 2023-05-31T21:11:38.717 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T21:11:38.713+0000 7fcadbe43780 1 waiting to reach incremental sync.. 2023-05-31T21:11:43.718 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T21:11:43.714+0000 7fcadbe43780 1 waiting to reach incremental sync.. 2023-05-31T21:11:48.719 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T21:11:48.714+0000 7fcadbe43780 -1 bucket checkpoint timed out waiting to reach incremental sync 2023-05-31T21:11:48.719 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T21:11:48.714+0000 7fcadbe43780 0 bucket sync checkpoint failed: (110) Connection timed out 2023-05-31T21:11:48.719 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T21:11:48.714+0000 7fcadbe43780 -1 bucket sync checkpoint failed: (110) Connection timed out 2023-05-31T21:11:48.740 DEBUG:teuthology.orchestra.run:got remote process result: 110 2023-05-31T21:11:48.742 INFO:tasks.rgw_multisite_tests:rgw_multi.tests.test_resharded_bucket_incremental_sync_latest_after_data_sync_init ... FAIL 2023-05-31T21:11:48.743 DEBUG:boto:path=/pklxkd-64/ 2023-05-31T21:11:48.744 DEBUG:boto:auth_path=/pklxkd-64/ 2023-05-31T21:11:48.744 DEBUG:boto:Method: PUT 2023-05-31T21:11:48.745 DEBUG:boto:Path: /pklxkd-64/ 2023-05-31T21:11:48.745 DEBUG:boto:Data: 2023-05-31T21:11:48.745 DEBUG:boto:Headers: {} 2023-05-31T21:11:48.746 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T21:11:48.746 DEBUG:boto:Port: 8000 2023-05-31T21:11:48.746 DEBUG:boto:Params: {} 2023-05-31T21:11:48.747 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2023-05-31T21:11:48.748 DEBUG:boto:Token: None 2023-05-31T21:11:48.748 DEBUG:boto:StringToSign: PUT Wed, 31 May 2023 21:11:48 GMT /pklxkd-64/ 2023-05-31T21:11:48.748 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:u3CbdAOAJ5dYMZFBGrKRRaPxL5U= 2023-05-31T21:11:48.749 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 21:11:48 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:u3CbdAOAJ5dYMZFBGrKRRaPxL5U=', 'Content-Length': '0'} 2023-05-31T21:11:48.780 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000ce71a9bd56ba6d86-006477b814-1630-a1'), ('Content-Length', '0'), ('Date', 'Wed, 31 May 2023 21:11:48 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:11:48.780 DEBUG:rgw_multi.tests:created bucket=pklxkd-64 2023-05-31T21:11:48.781 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-31T21:11:48.781 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-31T21:11:48.782 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-05-31T21:11:48.835 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2023-05-31T21:11:48.835 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2023-05-31T21:11:48.871 INFO:teuthology.orchestra.run.smithi099.stderr:No --period given, using current period=2e6123d2-d695-4350-9a01-61a455bc782b 2023-05-31T21:11:48.887 INFO:teuthology.orchestra.run.smithi099.stdout:[ 2023-05-31T21:11:48.887 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T21:11:48.887 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1685566501.757183_260.1", 2023-05-31T21:11:48.887 INFO:teuthology.orchestra.run.smithi099.stdout: "last_update": "2023-05-31T20:55:01.757183Z" 2023-05-31T21:11:48.887 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:11:48.888 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T21:11:48.888 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1685566501.735717_251.1", 2023-05-31T21:11:48.888 INFO:teuthology.orchestra.run.smithi099.stdout: "last_update": "2023-05-31T20:55:01.735717Z" 2023-05-31T21:11:48.888 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:11:48.888 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T21:11:48.888 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1685562970.274572_93.1", 2023-05-31T21:11:48.889 INFO:teuthology.orchestra.run.smithi099.stdout: "last_update": "2023-05-31T19:56:10.274572Z" 2023-05-31T21:11:48.889 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:11:48.889 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T21:11:48.889 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1685567508.772581_734.1", 2023-05-31T21:11:48.889 INFO:teuthology.orchestra.run.smithi099.stdout: "last_update": "2023-05-31T21:11:48.772581Z" 2023-05-31T21:11:48.889 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T21:11:48.890 INFO:teuthology.orchestra.run.smithi099.stdout:] 2023-05-31T21:11:48.890 DEBUG:rgw_multi.tests:master meta markers={0: '1_1685566501.757183_260.1', 1: '1_1685566501.735717_251.1', 2: '1_1685562970.274572_93.1', 3: '1_1685567508.772581_734.1'} 2023-05-31T21:11:48.891 INFO:rgw_multi.tests:starting meta checkpoint for zone=a2 2023-05-31T21:11:48.891 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-31T21:11:48.892 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-31T21:11:48.892 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-05-31T21:11:48.943 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2023-05-31T21:11:48.943 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2023-05-31T21:11:48.991 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T21:11:48.987+0000 7f5598d45780 20 RGW-SYNC:meta: read sync status 2023-05-31T21:11:48.992 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T21:11:48.987+0000 7f5598d45780 20 RGW-SYNC:meta: read sync status 2023-05-31T21:11:49.013 INFO:teuthology.orchestra.run.smithi099.stdout:{ 2023-05-31T21:11:49.013 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_status": { 2023-05-31T21:11:49.013 INFO:teuthology.orchestra.run.smithi099.stdout: "info": { 2023-05-31T21:11:49.013 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "sync", 2023-05-31T21:11:49.013 INFO:teuthology.orchestra.run.smithi099.stdout: "num_shards": 4, 2023-05-31T21:11:49.014 INFO:teuthology.orchestra.run.smithi099.stdout: "period": "2e6123d2-d695-4350-9a01-61a455bc782b", 2023-05-31T21:11:49.014 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2023-05-31T21:11:49.014 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:11:49.014 INFO:teuthology.orchestra.run.smithi099.stdout: "markers": [ 2023-05-31T21:11:49.014 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T21:11:49.014 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 0, 2023-05-31T21:11:49.014 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T21:11:49.015 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2023-05-31T21:11:49.015 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1685566501.757183_260.1", 2023-05-31T21:11:49.015 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T21:11:49.015 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 1, 2023-05-31T21:11:49.015 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T21:11:49.015 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2023-05-31T20:55:01.757183Z", 2023-05-31T21:11:49.015 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2023-05-31T21:11:49.016 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T21:11:49.016 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:11:49.016 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T21:11:49.016 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 1, 2023-05-31T21:11:49.016 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T21:11:49.016 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2023-05-31T21:11:49.017 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1685566501.735717_251.1", 2023-05-31T21:11:49.017 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T21:11:49.017 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2023-05-31T21:11:49.017 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T21:11:49.017 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2023-05-31T20:55:01.735717Z", 2023-05-31T21:11:49.017 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2023-05-31T21:11:49.018 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T21:11:49.018 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:11:49.018 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T21:11:49.018 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 2, 2023-05-31T21:11:49.018 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T21:11:49.018 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2023-05-31T21:11:49.018 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1685562970.274572_93.1", 2023-05-31T21:11:49.019 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T21:11:49.019 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2023-05-31T21:11:49.019 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T21:11:49.019 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2023-05-31T19:56:10.274572Z", 2023-05-31T21:11:49.019 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2023-05-31T21:11:49.019 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T21:11:49.019 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:11:49.020 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T21:11:49.020 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 3, 2023-05-31T21:11:49.020 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T21:11:49.020 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2023-05-31T21:11:49.020 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1685567508.772581_734.1", 2023-05-31T21:11:49.020 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T21:11:49.020 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2023-05-31T21:11:49.021 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T21:11:49.021 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2023-05-31T21:11:48.772581Z", 2023-05-31T21:11:49.021 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2023-05-31T21:11:49.021 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T21:11:49.021 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T21:11:49.021 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2023-05-31T21:11:49.022 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:11:49.022 INFO:teuthology.orchestra.run.smithi099.stdout: "full_sync": { 2023-05-31T21:11:49.022 INFO:teuthology.orchestra.run.smithi099.stdout: "total": 1, 2023-05-31T21:11:49.022 INFO:teuthology.orchestra.run.smithi099.stdout: "complete": 1 2023-05-31T21:11:49.022 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:11:49.022 INFO:teuthology.orchestra.run.smithi099.stdout: "current_time": "2023-05-31T21:11:48Z" 2023-05-31T21:11:49.022 INFO:teuthology.orchestra.run.smithi099.stdout:} 2023-05-31T21:11:49.023 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "2e6123d2-d695-4350-9a01-61a455bc782b", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1685566501.757183_260.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-05-31T20:55:01.757183Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1685566501.735717_251.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-05-31T20:55:01.735717Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1685562970.274572_93.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-05-31T19:56:10.274572Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1685567508.772581_734.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-05-31T21:11:48.772581Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-05-31T21:11:48Z" } 2023-05-31T21:11:49.024 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1685566501.757183_260.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-05-31T20:55:01.757183Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1685566501.735717_251.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-05-31T20:55:01.735717Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1685562970.274572_93.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-05-31T19:56:10.274572Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1685567508.772581_734.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-05-31T21:11:48.772581Z', 'realm_epoch': 2}}] 2023-05-31T21:11:49.024 DEBUG:rgw_multi.tests:log_status={0: '1_1685566501.757183_260.1', 1: '1_1685566501.735717_251.1', 2: '1_1685562970.274572_93.1', 3: '1_1685567508.772581_734.1'} 2023-05-31T21:11:49.025 DEBUG:rgw_multi.tests:sync_status={0: '1_1685566501.757183_260.1', 1: '1_1685566501.735717_251.1', 2: '1_1685562970.274572_93.1', 3: '1_1685567508.772581_734.1'} 2023-05-31T21:11:49.025 INFO:rgw_multi.tests:finish meta checkpoint for zone=a2 2023-05-31T21:11:49.025 DEBUG:boto:path=/pklxkd-64/ 2023-05-31T21:11:49.026 DEBUG:boto:auth_path=/pklxkd-64/ 2023-05-31T21:11:49.026 DEBUG:boto:Method: HEAD 2023-05-31T21:11:49.026 DEBUG:boto:Path: /pklxkd-64/ 2023-05-31T21:11:49.027 DEBUG:boto:Data: 2023-05-31T21:11:49.027 DEBUG:boto:Headers: {} 2023-05-31T21:11:49.027 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T21:11:49.028 DEBUG:boto:Port: 8000 2023-05-31T21:11:49.028 DEBUG:boto:Params: {} 2023-05-31T21:11:49.028 DEBUG:boto:Token: None 2023-05-31T21:11:49.029 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 21:11:49 GMT /pklxkd-64/ 2023-05-31T21:11:49.029 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:WtQF3N8/UJADUP+Xy7BD1Fmkr/U= 2023-05-31T21:11:49.030 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 21:11:49 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:WtQF3N8/UJADUP+Xy7BD1Fmkr/U=', 'Content-Length': '0'} 2023-05-31T21:11:49.032 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000037c1cb85c92b2339-006477b815-1630-a1'), ('Content-Length', '0'), ('Date', 'Wed, 31 May 2023 21:11:49 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:11:49.032 DEBUG:boto:path=/pklxkd-64/obj11 2023-05-31T21:11:49.033 DEBUG:boto:auth_path=/pklxkd-64/obj11 2023-05-31T21:11:49.033 DEBUG:boto:Method: PUT 2023-05-31T21:11:49.034 DEBUG:boto:Path: /pklxkd-64/obj11 2023-05-31T21:11:49.034 DEBUG:boto:Data: 2023-05-31T21:11:49.034 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-05-31T21:11:49.035 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T21:11:49.035 DEBUG:boto:Port: 8000 2023-05-31T21:11:49.035 DEBUG:boto:Params: {} 2023-05-31T21:11:49.036 DEBUG:boto:Token: None 2023-05-31T21:11:49.036 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Wed, 31 May 2023 21:11:49 GMT /pklxkd-64/obj11 2023-05-31T21:11:49.037 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:tHsfNYdEXhvC6oJHxPZcU7ntuvA= 2023-05-31T21:11:49.037 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Wed, 31 May 2023 21:11:49 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:tHsfNYdEXhvC6oJHxPZcU7ntuvA='} 2023-05-31T21:11:49.045 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000c126649dfa9904a2-006477b815-1630-a1'), ('Date', 'Wed, 31 May 2023 21:11:49 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:11:49.046 DEBUG:boto:path=/pklxkd-64/ 2023-05-31T21:11:49.046 DEBUG:boto:auth_path=/pklxkd-64/ 2023-05-31T21:11:49.046 DEBUG:boto:Method: HEAD 2023-05-31T21:11:49.047 DEBUG:boto:Path: /pklxkd-64/ 2023-05-31T21:11:49.047 DEBUG:boto:Data: 2023-05-31T21:11:49.047 DEBUG:boto:Headers: {} 2023-05-31T21:11:49.048 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T21:11:49.048 DEBUG:boto:Port: 8000 2023-05-31T21:11:49.048 DEBUG:boto:Params: {} 2023-05-31T21:11:49.049 DEBUG:boto:Token: None 2023-05-31T21:11:49.049 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 21:11:49 GMT /pklxkd-64/ 2023-05-31T21:11:49.049 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:WtQF3N8/UJADUP+Xy7BD1Fmkr/U= 2023-05-31T21:11:49.050 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 21:11:49 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:WtQF3N8/UJADUP+Xy7BD1Fmkr/U=', 'Content-Length': '0'} 2023-05-31T21:11:49.052 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '3'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000013b39f071c4ce2e1-006477b815-1630-a1'), ('Content-Length', '0'), ('Date', 'Wed, 31 May 2023 21:11:49 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:11:49.052 DEBUG:boto:path=/pklxkd-64/obj22 2023-05-31T21:11:49.053 DEBUG:boto:auth_path=/pklxkd-64/obj22 2023-05-31T21:11:49.053 DEBUG:boto:Method: PUT 2023-05-31T21:11:49.053 DEBUG:boto:Path: /pklxkd-64/obj22 2023-05-31T21:11:49.054 DEBUG:boto:Data: 2023-05-31T21:11:49.054 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-05-31T21:11:49.054 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T21:11:49.055 DEBUG:boto:Port: 8000 2023-05-31T21:11:49.055 DEBUG:boto:Params: {} 2023-05-31T21:11:49.055 DEBUG:boto:Token: None 2023-05-31T21:11:49.056 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Wed, 31 May 2023 21:11:49 GMT /pklxkd-64/obj22 2023-05-31T21:11:49.056 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:kQd2RcTkswxSAzA+gLkxtE1EuHs= 2023-05-31T21:11:49.056 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Wed, 31 May 2023 21:11:49 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:kQd2RcTkswxSAzA+gLkxtE1EuHs='} 2023-05-31T21:11:49.062 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000031eb0e03361e4c5d-006477b815-1630-a1'), ('Date', 'Wed, 31 May 2023 21:11:49 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:11:49.062 DEBUG:boto:path=/pklxkd-64/ 2023-05-31T21:11:49.063 DEBUG:boto:auth_path=/pklxkd-64/ 2023-05-31T21:11:49.063 DEBUG:boto:Method: HEAD 2023-05-31T21:11:49.063 DEBUG:boto:Path: /pklxkd-64/ 2023-05-31T21:11:49.064 DEBUG:boto:Data: 2023-05-31T21:11:49.064 DEBUG:boto:Headers: {} 2023-05-31T21:11:49.064 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T21:11:49.065 DEBUG:boto:Port: 8000 2023-05-31T21:11:49.065 DEBUG:boto:Params: {} 2023-05-31T21:11:49.065 DEBUG:boto:Token: None 2023-05-31T21:11:49.066 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 21:11:49 GMT /pklxkd-64/ 2023-05-31T21:11:49.066 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:WtQF3N8/UJADUP+Xy7BD1Fmkr/U= 2023-05-31T21:11:49.066 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 21:11:49 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:WtQF3N8/UJADUP+Xy7BD1Fmkr/U=', 'Content-Length': '0'} 2023-05-31T21:11:49.068 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '2'), ('X-RGW-Bytes-Used', '6'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000b5ce5a33a5d1350f-006477b815-1630-a1'), ('Content-Length', '0'), ('Date', 'Wed, 31 May 2023 21:11:49 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:11:49.069 DEBUG:boto:path=/pklxkd-64/obj33 2023-05-31T21:11:49.069 DEBUG:boto:auth_path=/pklxkd-64/obj33 2023-05-31T21:11:49.070 DEBUG:boto:Method: PUT 2023-05-31T21:11:49.070 DEBUG:boto:Path: /pklxkd-64/obj33 2023-05-31T21:11:49.070 DEBUG:boto:Data: 2023-05-31T21:11:49.071 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-05-31T21:11:49.071 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T21:11:49.071 DEBUG:boto:Port: 8000 2023-05-31T21:11:49.072 DEBUG:boto:Params: {} 2023-05-31T21:11:49.072 DEBUG:boto:Token: None 2023-05-31T21:11:49.072 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Wed, 31 May 2023 21:11:49 GMT /pklxkd-64/obj33 2023-05-31T21:11:49.073 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:NXz3r2yvS8eyJWDa6bNsUS9Ln3o= 2023-05-31T21:11:49.073 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Wed, 31 May 2023 21:11:49 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:NXz3r2yvS8eyJWDa6bNsUS9Ln3o='} 2023-05-31T21:11:49.116 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000f24787bd2326be48-006477b815-1630-a1'), ('Date', 'Wed, 31 May 2023 21:11:49 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:11:49.117 DEBUG:boto:path=/pklxkd-64/ 2023-05-31T21:11:49.117 DEBUG:boto:auth_path=/pklxkd-64/ 2023-05-31T21:11:49.118 DEBUG:boto:Method: HEAD 2023-05-31T21:11:49.118 DEBUG:boto:Path: /pklxkd-64/ 2023-05-31T21:11:49.118 DEBUG:boto:Data: 2023-05-31T21:11:49.119 DEBUG:boto:Headers: {} 2023-05-31T21:11:49.119 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T21:11:49.119 DEBUG:boto:Port: 8000 2023-05-31T21:11:49.119 DEBUG:boto:Params: {} 2023-05-31T21:11:49.120 DEBUG:boto:Token: None 2023-05-31T21:11:49.120 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 21:11:49 GMT /pklxkd-64/ 2023-05-31T21:11:49.121 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:WtQF3N8/UJADUP+Xy7BD1Fmkr/U= 2023-05-31T21:11:49.121 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 21:11:49 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:WtQF3N8/UJADUP+Xy7BD1Fmkr/U=', 'Content-Length': '0'} 2023-05-31T21:11:49.123 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '3'), ('X-RGW-Bytes-Used', '9'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000093a7e10596af3f16-006477b815-1630-a1'), ('Content-Length', '0'), ('Date', 'Wed, 31 May 2023 21:11:49 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:11:49.124 DEBUG:boto:path=/pklxkd-64/obj44 2023-05-31T21:11:49.124 DEBUG:boto:auth_path=/pklxkd-64/obj44 2023-05-31T21:11:49.124 DEBUG:boto:Method: PUT 2023-05-31T21:11:49.125 DEBUG:boto:Path: /pklxkd-64/obj44 2023-05-31T21:11:49.125 DEBUG:boto:Data: 2023-05-31T21:11:49.125 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-05-31T21:11:49.126 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T21:11:49.126 DEBUG:boto:Port: 8000 2023-05-31T21:11:49.126 DEBUG:boto:Params: {} 2023-05-31T21:11:49.127 DEBUG:boto:Token: None 2023-05-31T21:11:49.127 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Wed, 31 May 2023 21:11:49 GMT /pklxkd-64/obj44 2023-05-31T21:11:49.127 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:o+D0PaoSF0fl0zqZh7AruwVEg0Y= 2023-05-31T21:11:49.128 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Wed, 31 May 2023 21:11:49 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:o+D0PaoSF0fl0zqZh7AruwVEg0Y='} 2023-05-31T21:11:49.135 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000008839e63d01901be8-006477b815-1630-a1'), ('Date', 'Wed, 31 May 2023 21:11:49 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:11:49.135 DEBUG:boto:path=/pklxkd-64/ 2023-05-31T21:11:49.136 DEBUG:boto:auth_path=/pklxkd-64/ 2023-05-31T21:11:49.136 DEBUG:boto:Method: HEAD 2023-05-31T21:11:49.136 DEBUG:boto:Path: /pklxkd-64/ 2023-05-31T21:11:49.137 DEBUG:boto:Data: 2023-05-31T21:11:49.137 DEBUG:boto:Headers: {} 2023-05-31T21:11:49.137 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T21:11:49.138 DEBUG:boto:Port: 8000 2023-05-31T21:11:49.138 DEBUG:boto:Params: {} 2023-05-31T21:11:49.138 DEBUG:boto:Token: None 2023-05-31T21:11:49.139 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 21:11:49 GMT /pklxkd-64/ 2023-05-31T21:11:49.139 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:WtQF3N8/UJADUP+Xy7BD1Fmkr/U= 2023-05-31T21:11:49.140 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 21:11:49 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:WtQF3N8/UJADUP+Xy7BD1Fmkr/U=', 'Content-Length': '0'} 2023-05-31T21:11:49.142 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '4'), ('X-RGW-Bytes-Used', '12'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000bca2f544eaa003a0-006477b815-1630-a1'), ('Content-Length', '0'), ('Date', 'Wed, 31 May 2023 21:11:49 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:11:49.142 DEBUG:boto:path=/pklxkd-64/obj55 2023-05-31T21:11:49.142 DEBUG:boto:auth_path=/pklxkd-64/obj55 2023-05-31T21:11:49.143 DEBUG:boto:Method: PUT 2023-05-31T21:11:49.143 DEBUG:boto:Path: /pklxkd-64/obj55 2023-05-31T21:11:49.143 DEBUG:boto:Data: 2023-05-31T21:11:49.144 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-05-31T21:11:49.144 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T21:11:49.145 DEBUG:boto:Port: 8000 2023-05-31T21:11:49.145 DEBUG:boto:Params: {} 2023-05-31T21:11:49.145 DEBUG:boto:Token: None 2023-05-31T21:11:49.146 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Wed, 31 May 2023 21:11:49 GMT /pklxkd-64/obj55 2023-05-31T21:11:49.146 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:EKkHwPtPXSjrCISLwfCrrx0vGpw= 2023-05-31T21:11:49.146 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Wed, 31 May 2023 21:11:49 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:EKkHwPtPXSjrCISLwfCrrx0vGpw='} 2023-05-31T21:11:49.150 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000087e07f1827f05bda-006477b815-1630-a1'), ('Date', 'Wed, 31 May 2023 21:11:49 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:11:49.150 DEBUG:rgw_multi.tests:bucket checkpoint: source=a1 target=a2 bucket=pklxkd-64 2023-05-31T21:11:49.151 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'pklxkd-64', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-05-31T21:11:49.151 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'pklxkd-64', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-05-31T21:11:49.151 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket pklxkd-64 --source-zone a1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2023-05-31T21:11:49.207 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2023-05-31T21:11:49.208 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2023-05-31T21:11:49.246 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T21:11:49.242+0000 7f1c4cdc1780 1 waiting to reach incremental sync.. 2023-05-31T21:11:54.247 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T21:11:54.243+0000 7f1c4cdc1780 1 waiting to reach incremental sync.. 2023-05-31T21:11:59.249 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T21:11:59.244+0000 7f1c4cdc1780 1 bucket sync caught up with source: 2023-05-31T21:11:59.249 INFO:teuthology.orchestra.run.smithi099.stderr: local status: [, , , , , , , , 00000000001.133.6, 00000000002.139.6, 00000000002.101.6] 2023-05-31T21:11:59.249 INFO:teuthology.orchestra.run.smithi099.stderr: remote markers: [, , , , , , , , 00000000001.133.6, 00000000002.139.6, 00000000002.101.6] 2023-05-31T21:11:59.250 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T21:11:59.244+0000 7f1c4cdc1780 0 bucket checkpoint complete 2023-05-31T21:11:59.272 DEBUG:rgw_multi.tests:bucket checkpoint: source=a2 target=a1 bucket=pklxkd-64 2023-05-31T21:11:59.273 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'pklxkd-64', '--source-zone', 'a2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-05-31T21:11:59.273 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'pklxkd-64', '--source-zone', 'a2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-05-31T21:11:59.274 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket pklxkd-64 --source-zone a2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2023-05-31T21:11:59.336 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2023-05-31T21:11:59.336 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2023-05-31T21:11:59.384 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T21:11:59.380+0000 7f229ef2c780 1 waiting for incremental sync to catch up: 2023-05-31T21:11:59.384 INFO:teuthology.orchestra.run.smithi099.stderr: local status: [, , , , , , , , 00000000001.1365.6, 00000000002.3947.6, 00000000001.2023.6] 2023-05-31T21:11:59.385 INFO:teuthology.orchestra.run.smithi099.stderr: remote markers: [, , , , , , , , 00000000001.1365.6, 00000000002.3947.6, 00000000002.2025.6] 2023-05-31T21:12:04.386 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T21:12:04.382+0000 7f229ef2c780 1 waiting for incremental sync to catch up: 2023-05-31T21:12:04.386 INFO:teuthology.orchestra.run.smithi099.stderr: local status: [, , , , , , , , 00000000001.1365.6, 00000000002.3947.6, 00000000001.2023.6] 2023-05-31T21:12:04.386 INFO:teuthology.orchestra.run.smithi099.stderr: remote markers: [, , , , , , , , 00000000001.1365.6, 00000000002.3947.6, 00000000002.2025.6] 2023-05-31T21:12:09.387 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T21:12:09.383+0000 7f229ef2c780 1 waiting for incremental sync to catch up: 2023-05-31T21:12:09.387 INFO:teuthology.orchestra.run.smithi099.stderr: local status: [, , , , , , , , 00000000001.1365.6, 00000000002.3947.6, 00000000001.2023.6] 2023-05-31T21:12:09.387 INFO:teuthology.orchestra.run.smithi099.stderr: remote markers: [, , , , , , , , 00000000001.1365.6, 00000000002.3947.6, 00000000002.2025.6] 2023-05-31T21:12:14.388 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T21:12:14.384+0000 7f229ef2c780 1 waiting for incremental sync to catch up: 2023-05-31T21:12:14.389 INFO:teuthology.orchestra.run.smithi099.stderr: local status: [, , , , , , , , 00000000001.1365.6, 00000000002.3947.6, 00000000001.2023.6] 2023-05-31T21:12:14.389 INFO:teuthology.orchestra.run.smithi099.stderr: remote markers: [, , , , , , , , 00000000001.1365.6, 00000000002.3947.6, 00000000002.2025.6] 2023-05-31T21:12:19.388 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T21:12:19.385+0000 7f229ef2c780 1 bucket sync caught up with source: 2023-05-31T21:12:19.389 INFO:teuthology.orchestra.run.smithi099.stderr: local status: [, , , , , , , , 00000000001.1365.6, 00000000002.3947.6, 00000000002.2025.6] 2023-05-31T21:12:19.389 INFO:teuthology.orchestra.run.smithi099.stderr: remote markers: [, , , , , , , , 00000000001.1365.6, 00000000002.3947.6, 00000000002.2025.6] 2023-05-31T21:12:19.389 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T21:12:19.385+0000 7f229ef2c780 0 bucket checkpoint complete 2023-05-31T21:12:19.411 INFO:tasks.rgw_multi.zone_rados:comparing bucket=pklxkd-64 zones={a2, a1} 2023-05-31T21:12:19.412 DEBUG:boto:path=/pklxkd-64/ 2023-05-31T21:12:19.412 DEBUG:boto:auth_path=/pklxkd-64/ 2023-05-31T21:12:19.412 DEBUG:boto:Method: HEAD 2023-05-31T21:12:19.413 DEBUG:boto:Path: /pklxkd-64/ 2023-05-31T21:12:19.413 DEBUG:boto:Data: 2023-05-31T21:12:19.413 DEBUG:boto:Headers: {} 2023-05-31T21:12:19.414 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T21:12:19.414 DEBUG:boto:Port: 8001 2023-05-31T21:12:19.414 DEBUG:boto:Params: {} 2023-05-31T21:12:19.415 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8001} 2023-05-31T21:12:19.415 DEBUG:boto:Token: None 2023-05-31T21:12:19.416 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 21:12:19 GMT /pklxkd-64/ 2023-05-31T21:12:19.416 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:KsfBBJmgO4AvTs8461Cw/7QVctI= 2023-05-31T21:12:19.417 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 21:12:19 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:KsfBBJmgO4AvTs8461Cw/7QVctI=', 'Content-Length': '0'} 2023-05-31T21:12:19.420 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '5'), ('X-RGW-Bytes-Used', '15'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000ea515b585d467391-006477b833-1fb6-a2'), ('Content-Length', '0'), ('Date', 'Wed, 31 May 2023 21:12:19 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:12:19.421 DEBUG:boto:path=/pklxkd-64/ 2023-05-31T21:12:19.421 DEBUG:boto:auth_path=/pklxkd-64/ 2023-05-31T21:12:19.421 DEBUG:boto:Method: HEAD 2023-05-31T21:12:19.422 DEBUG:boto:Path: /pklxkd-64/ 2023-05-31T21:12:19.422 DEBUG:boto:Data: 2023-05-31T21:12:19.423 DEBUG:boto:Headers: {} 2023-05-31T21:12:19.423 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T21:12:19.423 DEBUG:boto:Port: 8000 2023-05-31T21:12:19.424 DEBUG:boto:Params: {} 2023-05-31T21:12:19.424 DEBUG:boto:Token: None 2023-05-31T21:12:19.424 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 21:12:19 GMT /pklxkd-64/ 2023-05-31T21:12:19.425 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:KsfBBJmgO4AvTs8461Cw/7QVctI= 2023-05-31T21:12:19.425 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 21:12:19 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:KsfBBJmgO4AvTs8461Cw/7QVctI=', 'Content-Length': '0'} 2023-05-31T21:12:19.428 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '5'), ('X-RGW-Bytes-Used', '15'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000000be941d8ce1f1ff2-006477b833-1630-a1'), ('Content-Length', '0'), ('Date', 'Wed, 31 May 2023 21:12:19 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:12:19.428 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2023-05-31T21:12:19.429 DEBUG:boto:path=/pklxkd-64/ 2023-05-31T21:12:19.429 DEBUG:boto:auth_path=/pklxkd-64/ 2023-05-31T21:12:19.429 DEBUG:boto:path=/pklxkd-64/?versions&max-keys=999 2023-05-31T21:12:19.430 DEBUG:boto:auth_path=/pklxkd-64/?versions&max-keys=999 2023-05-31T21:12:19.430 DEBUG:boto:Method: GET 2023-05-31T21:12:19.430 DEBUG:boto:Path: /pklxkd-64/?versions&max-keys=999 2023-05-31T21:12:19.431 DEBUG:boto:Data: 2023-05-31T21:12:19.431 DEBUG:boto:Headers: {} 2023-05-31T21:12:19.431 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T21:12:19.432 DEBUG:boto:Port: 8001 2023-05-31T21:12:19.432 DEBUG:boto:Params: {} 2023-05-31T21:12:19.433 DEBUG:boto:Token: None 2023-05-31T21:12:19.433 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 21:12:19 GMT /pklxkd-64/?versions 2023-05-31T21:12:19.433 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:Y+FNsnKkhxckLDmOpwOjktX94r0= 2023-05-31T21:12:19.434 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 21:12:19 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:Y+FNsnKkhxckLDmOpwOjktX94r0=', 'Content-Length': '0'} 2023-05-31T21:12:19.436 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000022f6e23df123856f-006477b833-1fb6-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 21:12:19 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:12:19.437 DEBUG:boto:b'pklxkd-64999falseobj11nulltrue2023-05-31T21:11:49.036Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj22nulltrue2023-05-31T21:11:49.055Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj33nulltrue2023-05-31T21:11:49.072Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj44nulltrue2023-05-31T21:11:49.126Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj55nulltrue2023-05-31T21:11:49.145Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T21:12:19.438 DEBUG:tasks.rgw_multi.zone_rados:o=obj11 2023-05-31T21:12:19.438 DEBUG:tasks.rgw_multi.zone_rados:o=obj22 2023-05-31T21:12:19.439 DEBUG:tasks.rgw_multi.zone_rados:o=obj33 2023-05-31T21:12:19.439 DEBUG:tasks.rgw_multi.zone_rados:o=obj44 2023-05-31T21:12:19.439 DEBUG:tasks.rgw_multi.zone_rados:o=obj55 2023-05-31T21:12:19.439 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2023-05-31T21:12:19.440 DEBUG:boto:path=/pklxkd-64/ 2023-05-31T21:12:19.440 DEBUG:boto:auth_path=/pklxkd-64/ 2023-05-31T21:12:19.441 DEBUG:boto:path=/pklxkd-64/?versions&max-keys=999 2023-05-31T21:12:19.441 DEBUG:boto:auth_path=/pklxkd-64/?versions&max-keys=999 2023-05-31T21:12:19.441 DEBUG:boto:Method: GET 2023-05-31T21:12:19.442 DEBUG:boto:Path: /pklxkd-64/?versions&max-keys=999 2023-05-31T21:12:19.442 DEBUG:boto:Data: 2023-05-31T21:12:19.442 DEBUG:boto:Headers: {} 2023-05-31T21:12:19.443 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T21:12:19.443 DEBUG:boto:Port: 8000 2023-05-31T21:12:19.443 DEBUG:boto:Params: {} 2023-05-31T21:12:19.444 DEBUG:boto:Token: None 2023-05-31T21:12:19.444 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 21:12:19 GMT /pklxkd-64/?versions 2023-05-31T21:12:19.445 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:Y+FNsnKkhxckLDmOpwOjktX94r0= 2023-05-31T21:12:19.445 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 21:12:19 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:Y+FNsnKkhxckLDmOpwOjktX94r0=', 'Content-Length': '0'} 2023-05-31T21:12:19.447 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d7548dab02f2e1ca-006477b833-1630-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 21:12:19 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:12:19.448 DEBUG:boto:b'pklxkd-64999falseobj11nulltrue2023-05-31T21:11:49.036Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj22nulltrue2023-05-31T21:11:49.055Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj33nulltrue2023-05-31T21:11:49.072Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj44nulltrue2023-05-31T21:11:49.126Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj55nulltrue2023-05-31T21:11:49.145Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T21:12:19.448 DEBUG:tasks.rgw_multi.zone_rados:o=obj11 2023-05-31T21:12:19.449 DEBUG:tasks.rgw_multi.zone_rados:o=obj22 2023-05-31T21:12:19.449 DEBUG:tasks.rgw_multi.zone_rados:o=obj33 2023-05-31T21:12:19.449 DEBUG:tasks.rgw_multi.zone_rados:o=obj44 2023-05-31T21:12:19.450 DEBUG:tasks.rgw_multi.zone_rados:o=obj55 2023-05-31T21:12:19.450 DEBUG:boto:path=/pklxkd-64/ 2023-05-31T21:12:19.451 DEBUG:boto:auth_path=/pklxkd-64/ 2023-05-31T21:12:19.451 DEBUG:boto:path=/pklxkd-64/?versions&max-keys=999 2023-05-31T21:12:19.451 DEBUG:boto:auth_path=/pklxkd-64/?versions&max-keys=999 2023-05-31T21:12:19.452 DEBUG:boto:Method: GET 2023-05-31T21:12:19.452 DEBUG:boto:Path: /pklxkd-64/?versions&max-keys=999 2023-05-31T21:12:19.452 DEBUG:boto:Data: 2023-05-31T21:12:19.453 DEBUG:boto:Headers: {} 2023-05-31T21:12:19.453 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T21:12:19.453 DEBUG:boto:Port: 8001 2023-05-31T21:12:19.454 DEBUG:boto:Params: {} 2023-05-31T21:12:19.454 DEBUG:boto:Token: None 2023-05-31T21:12:19.454 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 21:12:19 GMT /pklxkd-64/?versions 2023-05-31T21:12:19.455 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:Y+FNsnKkhxckLDmOpwOjktX94r0= 2023-05-31T21:12:19.455 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 21:12:19 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:Y+FNsnKkhxckLDmOpwOjktX94r0=', 'Content-Length': '0'} 2023-05-31T21:12:19.458 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000051318874d126aab3-006477b833-1fb6-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 21:12:19 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:12:19.458 DEBUG:boto:b'pklxkd-64999falseobj11nulltrue2023-05-31T21:11:49.036Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj22nulltrue2023-05-31T21:11:49.055Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj33nulltrue2023-05-31T21:11:49.072Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj44nulltrue2023-05-31T21:11:49.126Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj55nulltrue2023-05-31T21:11:49.145Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T21:12:19.459 DEBUG:boto:path=/pklxkd-64/ 2023-05-31T21:12:19.459 DEBUG:boto:auth_path=/pklxkd-64/ 2023-05-31T21:12:19.460 DEBUG:boto:path=/pklxkd-64/?versions&max-keys=999 2023-05-31T21:12:19.460 DEBUG:boto:auth_path=/pklxkd-64/?versions&max-keys=999 2023-05-31T21:12:19.460 DEBUG:boto:Method: GET 2023-05-31T21:12:19.461 DEBUG:boto:Path: /pklxkd-64/?versions&max-keys=999 2023-05-31T21:12:19.461 DEBUG:boto:Data: 2023-05-31T21:12:19.462 DEBUG:boto:Headers: {} 2023-05-31T21:12:19.462 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T21:12:19.462 DEBUG:boto:Port: 8000 2023-05-31T21:12:19.463 DEBUG:boto:Params: {} 2023-05-31T21:12:19.463 DEBUG:boto:Token: None 2023-05-31T21:12:19.463 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 21:12:19 GMT /pklxkd-64/?versions 2023-05-31T21:12:19.464 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:Y+FNsnKkhxckLDmOpwOjktX94r0= 2023-05-31T21:12:19.464 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 21:12:19 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:Y+FNsnKkhxckLDmOpwOjktX94r0=', 'Content-Length': '0'} 2023-05-31T21:12:19.467 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000082f04cd750ca8cff-006477b833-1630-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 21:12:19 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:12:19.467 DEBUG:boto:b'pklxkd-64999falseobj11nulltrue2023-05-31T21:11:49.036Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj22nulltrue2023-05-31T21:11:49.055Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj33nulltrue2023-05-31T21:11:49.072Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj44nulltrue2023-05-31T21:11:49.126Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj55nulltrue2023-05-31T21:11:49.145Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T21:12:19.468 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj11 2023-05-31T21:12:19.468 DEBUG:boto:path=/pklxkd-64/obj11 2023-05-31T21:12:19.469 DEBUG:boto:auth_path=/pklxkd-64/obj11 2023-05-31T21:12:19.469 DEBUG:boto:path=/pklxkd-64/obj11?versionId=null 2023-05-31T21:12:19.470 DEBUG:boto:auth_path=/pklxkd-64/obj11?versionId=null 2023-05-31T21:12:19.470 DEBUG:boto:Method: GET 2023-05-31T21:12:19.470 DEBUG:boto:Path: /pklxkd-64/obj11?versionId=null 2023-05-31T21:12:19.471 DEBUG:boto:Data: 2023-05-31T21:12:19.471 DEBUG:boto:Headers: {} 2023-05-31T21:12:19.471 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T21:12:19.472 DEBUG:boto:Port: 8001 2023-05-31T21:12:19.472 DEBUG:boto:Params: {} 2023-05-31T21:12:19.473 DEBUG:boto:Token: None 2023-05-31T21:12:19.473 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 21:12:19 GMT /pklxkd-64/obj11?versionId=null 2023-05-31T21:12:19.473 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:JoTqV5gdZipLpDgP6hOcJO5nUWg= 2023-05-31T21:12:19.474 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 21:12:19 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:JoTqV5gdZipLpDgP6hOcJO5nUWg=', 'Content-Length': '0'} 2023-05-31T21:12:19.476 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 21:11:49 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006e0d165b023ac06e-006477b833-1fb6-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 21:12:19 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:12:19.516 DEBUG:boto:path=/pklxkd-64/obj11 2023-05-31T21:12:19.517 DEBUG:boto:auth_path=/pklxkd-64/obj11 2023-05-31T21:12:19.517 DEBUG:boto:path=/pklxkd-64/obj11?versionId=null 2023-05-31T21:12:19.517 DEBUG:boto:auth_path=/pklxkd-64/obj11?versionId=null 2023-05-31T21:12:19.518 DEBUG:boto:Method: GET 2023-05-31T21:12:19.518 DEBUG:boto:Path: /pklxkd-64/obj11?versionId=null 2023-05-31T21:12:19.518 DEBUG:boto:Data: 2023-05-31T21:12:19.519 DEBUG:boto:Headers: {} 2023-05-31T21:12:19.519 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T21:12:19.520 DEBUG:boto:Port: 8000 2023-05-31T21:12:19.520 DEBUG:boto:Params: {} 2023-05-31T21:12:19.520 DEBUG:boto:Token: None 2023-05-31T21:12:19.521 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 21:12:19 GMT /pklxkd-64/obj11?versionId=null 2023-05-31T21:12:19.521 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:JoTqV5gdZipLpDgP6hOcJO5nUWg= 2023-05-31T21:12:19.522 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 21:12:19 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:JoTqV5gdZipLpDgP6hOcJO5nUWg=', 'Content-Length': '0'} 2023-05-31T21:12:19.523 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 21:11:49 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000bd4530429db50bb3-006477b833-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 21:12:19 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:12:19.524 DEBUG:boto:path=/pklxkd-64/obj11 2023-05-31T21:12:19.524 DEBUG:boto:auth_path=/pklxkd-64/obj11 2023-05-31T21:12:19.525 DEBUG:boto:path=/pklxkd-64/obj11?versionId=null 2023-05-31T21:12:19.525 DEBUG:boto:auth_path=/pklxkd-64/obj11?versionId=null 2023-05-31T21:12:19.525 DEBUG:boto:Method: HEAD 2023-05-31T21:12:19.526 DEBUG:boto:Path: /pklxkd-64/obj11?versionId=null 2023-05-31T21:12:19.526 DEBUG:boto:Data: 2023-05-31T21:12:19.526 DEBUG:boto:Headers: {} 2023-05-31T21:12:19.527 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T21:12:19.527 DEBUG:boto:Port: 8001 2023-05-31T21:12:19.527 DEBUG:boto:Params: {} 2023-05-31T21:12:19.528 DEBUG:boto:Token: None 2023-05-31T21:12:19.528 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 21:12:19 GMT /pklxkd-64/obj11?versionId=null 2023-05-31T21:12:19.529 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:ojWc3XEBka9c23WFsR2eI4DvBPE= 2023-05-31T21:12:19.529 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 21:12:19 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:ojWc3XEBka9c23WFsR2eI4DvBPE=', 'Content-Length': '0'} 2023-05-31T21:12:19.531 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 21:11:49 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c3c44ac98906140a-006477b833-1fb6-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 21:12:19 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:12:19.531 DEBUG:boto:path=/pklxkd-64/obj11 2023-05-31T21:12:19.532 DEBUG:boto:auth_path=/pklxkd-64/obj11 2023-05-31T21:12:19.532 DEBUG:boto:path=/pklxkd-64/obj11?versionId=null 2023-05-31T21:12:19.532 DEBUG:boto:auth_path=/pklxkd-64/obj11?versionId=null 2023-05-31T21:12:19.533 DEBUG:boto:Method: HEAD 2023-05-31T21:12:19.533 DEBUG:boto:Path: /pklxkd-64/obj11?versionId=null 2023-05-31T21:12:19.533 DEBUG:boto:Data: 2023-05-31T21:12:19.534 DEBUG:boto:Headers: {} 2023-05-31T21:12:19.534 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T21:12:19.534 DEBUG:boto:Port: 8000 2023-05-31T21:12:19.535 DEBUG:boto:Params: {} 2023-05-31T21:12:19.535 DEBUG:boto:Token: None 2023-05-31T21:12:19.536 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 21:12:19 GMT /pklxkd-64/obj11?versionId=null 2023-05-31T21:12:19.536 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:ojWc3XEBka9c23WFsR2eI4DvBPE= 2023-05-31T21:12:19.536 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 21:12:19 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:ojWc3XEBka9c23WFsR2eI4DvBPE=', 'Content-Length': '0'} 2023-05-31T21:12:19.538 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 21:11:49 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000aeb09e39e0082b24-006477b833-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 21:12:19 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:12:19.539 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj11 2023-05-31T21:12:19.539 DEBUG:boto:path=/pklxkd-64/obj11 2023-05-31T21:12:19.540 DEBUG:boto:auth_path=/pklxkd-64/obj11 2023-05-31T21:12:19.540 DEBUG:boto:path=/pklxkd-64/obj11?versionId=null 2023-05-31T21:12:19.540 DEBUG:boto:auth_path=/pklxkd-64/obj11?versionId=null 2023-05-31T21:12:19.541 DEBUG:boto:Method: GET 2023-05-31T21:12:19.541 DEBUG:boto:Path: /pklxkd-64/obj11?versionId=null 2023-05-31T21:12:19.541 DEBUG:boto:Data: 2023-05-31T21:12:19.542 DEBUG:boto:Headers: {} 2023-05-31T21:12:19.542 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T21:12:19.543 DEBUG:boto:Port: 8001 2023-05-31T21:12:19.543 DEBUG:boto:Params: {} 2023-05-31T21:12:19.543 DEBUG:boto:Token: None 2023-05-31T21:12:19.544 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 21:12:19 GMT /pklxkd-64/obj11?versionId=null 2023-05-31T21:12:19.544 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:JoTqV5gdZipLpDgP6hOcJO5nUWg= 2023-05-31T21:12:19.545 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 21:12:19 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:JoTqV5gdZipLpDgP6hOcJO5nUWg=', 'Content-Length': '0'} 2023-05-31T21:12:19.546 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 21:11:49 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007393a01af39bd799-006477b833-1fb6-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 21:12:19 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:12:19.547 DEBUG:boto:path=/pklxkd-64/obj11 2023-05-31T21:12:19.548 DEBUG:boto:auth_path=/pklxkd-64/obj11 2023-05-31T21:12:19.548 DEBUG:boto:path=/pklxkd-64/obj11?versionId=null 2023-05-31T21:12:19.548 DEBUG:boto:auth_path=/pklxkd-64/obj11?versionId=null 2023-05-31T21:12:19.549 DEBUG:boto:Method: GET 2023-05-31T21:12:19.549 DEBUG:boto:Path: /pklxkd-64/obj11?versionId=null 2023-05-31T21:12:19.549 DEBUG:boto:Data: 2023-05-31T21:12:19.550 DEBUG:boto:Headers: {} 2023-05-31T21:12:19.550 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T21:12:19.550 DEBUG:boto:Port: 8000 2023-05-31T21:12:19.551 DEBUG:boto:Params: {} 2023-05-31T21:12:19.551 DEBUG:boto:Token: None 2023-05-31T21:12:19.552 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 21:12:19 GMT /pklxkd-64/obj11?versionId=null 2023-05-31T21:12:19.552 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:JoTqV5gdZipLpDgP6hOcJO5nUWg= 2023-05-31T21:12:19.553 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 21:12:19 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:JoTqV5gdZipLpDgP6hOcJO5nUWg=', 'Content-Length': '0'} 2023-05-31T21:12:19.554 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 21:11:49 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000930bf0409d66d684-006477b833-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 21:12:19 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:12:19.555 DEBUG:boto:path=/pklxkd-64/ 2023-05-31T21:12:19.556 DEBUG:boto:auth_path=/pklxkd-64/ 2023-05-31T21:12:19.556 DEBUG:boto:path=/pklxkd-64/?prefix=obj11 2023-05-31T21:12:19.556 DEBUG:boto:auth_path=/pklxkd-64/?prefix=obj11 2023-05-31T21:12:19.557 DEBUG:boto:Method: GET 2023-05-31T21:12:19.557 DEBUG:boto:Path: /pklxkd-64/?prefix=obj11 2023-05-31T21:12:19.557 DEBUG:boto:Data: 2023-05-31T21:12:19.558 DEBUG:boto:Headers: {} 2023-05-31T21:12:19.558 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T21:12:19.559 DEBUG:boto:Port: 8001 2023-05-31T21:12:19.559 DEBUG:boto:Params: {} 2023-05-31T21:12:19.559 DEBUG:boto:Token: None 2023-05-31T21:12:19.560 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 21:12:19 GMT /pklxkd-64/ 2023-05-31T21:12:19.560 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:/ACajcjJ+wTjnzFp03OUBb/7FmU= 2023-05-31T21:12:19.561 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 21:12:19 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:/ACajcjJ+wTjnzFp03OUBb/7FmU=', 'Content-Length': '0'} 2023-05-31T21:12:19.563 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000091f9c45b81aff639-006477b833-1fb6-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 21:12:19 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:12:19.563 DEBUG:boto:b'pklxkd-64obj111000falseobj112023-05-31T21:11:49.036Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T21:12:19.564 DEBUG:boto:path=/pklxkd-64/ 2023-05-31T21:12:19.565 DEBUG:boto:auth_path=/pklxkd-64/ 2023-05-31T21:12:19.565 DEBUG:boto:path=/pklxkd-64/?prefix=obj11 2023-05-31T21:12:19.565 DEBUG:boto:auth_path=/pklxkd-64/?prefix=obj11 2023-05-31T21:12:19.566 DEBUG:boto:Method: GET 2023-05-31T21:12:19.566 DEBUG:boto:Path: /pklxkd-64/?prefix=obj11 2023-05-31T21:12:19.566 DEBUG:boto:Data: 2023-05-31T21:12:19.567 DEBUG:boto:Headers: {} 2023-05-31T21:12:19.567 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T21:12:19.567 DEBUG:boto:Port: 8000 2023-05-31T21:12:19.568 DEBUG:boto:Params: {} 2023-05-31T21:12:19.568 DEBUG:boto:Token: None 2023-05-31T21:12:19.569 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 21:12:19 GMT /pklxkd-64/ 2023-05-31T21:12:19.569 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:/ACajcjJ+wTjnzFp03OUBb/7FmU= 2023-05-31T21:12:19.569 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 21:12:19 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:/ACajcjJ+wTjnzFp03OUBb/7FmU=', 'Content-Length': '0'} 2023-05-31T21:12:19.572 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000008424732359768433-006477b833-1630-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 21:12:19 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:12:19.572 DEBUG:boto:b'pklxkd-64obj111000falseobj112023-05-31T21:11:49.036Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T21:12:19.573 DEBUG:boto:path=/pklxkd-64/obj11 2023-05-31T21:12:19.573 DEBUG:boto:auth_path=/pklxkd-64/obj11 2023-05-31T21:12:19.574 DEBUG:boto:Method: HEAD 2023-05-31T21:12:19.574 DEBUG:boto:Path: /pklxkd-64/obj11 2023-05-31T21:12:19.574 DEBUG:boto:Data: 2023-05-31T21:12:19.575 DEBUG:boto:Headers: {} 2023-05-31T21:12:19.575 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T21:12:19.575 DEBUG:boto:Port: 8001 2023-05-31T21:12:19.576 DEBUG:boto:Params: {} 2023-05-31T21:12:19.576 DEBUG:boto:Token: None 2023-05-31T21:12:19.577 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 21:12:19 GMT /pklxkd-64/obj11 2023-05-31T21:12:19.577 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:mEI87qCToWPjfFwbwN26x4EB8q0= 2023-05-31T21:12:19.577 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 21:12:19 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:mEI87qCToWPjfFwbwN26x4EB8q0=', 'Content-Length': '0'} 2023-05-31T21:12:19.579 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 21:11:49 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a9edf60ee3b588d4-006477b833-1fb6-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 21:12:19 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:12:19.580 DEBUG:boto:path=/pklxkd-64/obj11 2023-05-31T21:12:19.580 DEBUG:boto:auth_path=/pklxkd-64/obj11 2023-05-31T21:12:19.580 DEBUG:boto:Method: HEAD 2023-05-31T21:12:19.581 DEBUG:boto:Path: /pklxkd-64/obj11 2023-05-31T21:12:19.581 DEBUG:boto:Data: 2023-05-31T21:12:19.581 DEBUG:boto:Headers: {} 2023-05-31T21:12:19.582 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T21:12:19.582 DEBUG:boto:Port: 8000 2023-05-31T21:12:19.582 DEBUG:boto:Params: {} 2023-05-31T21:12:19.583 DEBUG:boto:Token: None 2023-05-31T21:12:19.583 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 21:12:19 GMT /pklxkd-64/obj11 2023-05-31T21:12:19.584 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:mEI87qCToWPjfFwbwN26x4EB8q0= 2023-05-31T21:12:19.584 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 21:12:19 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:mEI87qCToWPjfFwbwN26x4EB8q0=', 'Content-Length': '0'} 2023-05-31T21:12:19.586 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 21:11:49 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ef2cfb8d27bda539-006477b833-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 21:12:19 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:12:19.586 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj11 2023-05-31T21:12:19.587 DEBUG:boto:path=/pklxkd-64/obj11 2023-05-31T21:12:19.587 DEBUG:boto:auth_path=/pklxkd-64/obj11 2023-05-31T21:12:19.587 DEBUG:boto:Method: GET 2023-05-31T21:12:19.588 DEBUG:boto:Path: /pklxkd-64/obj11 2023-05-31T21:12:19.588 DEBUG:boto:Data: 2023-05-31T21:12:19.589 DEBUG:boto:Headers: {} 2023-05-31T21:12:19.589 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T21:12:19.589 DEBUG:boto:Port: 8001 2023-05-31T21:12:19.590 DEBUG:boto:Params: {} 2023-05-31T21:12:19.590 DEBUG:boto:Token: None 2023-05-31T21:12:19.591 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 21:12:19 GMT /pklxkd-64/obj11 2023-05-31T21:12:19.591 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:iIB+1MW4PDp12ma/FQGl76unqys= 2023-05-31T21:12:19.591 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 21:12:19 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:iIB+1MW4PDp12ma/FQGl76unqys=', 'Content-Length': '0'} 2023-05-31T21:12:19.593 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 21:11:49 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006d6ce2b0d97bd2a6-006477b833-1fb6-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 21:12:19 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:12:19.594 DEBUG:boto:path=/pklxkd-64/obj11 2023-05-31T21:12:19.594 DEBUG:boto:auth_path=/pklxkd-64/obj11 2023-05-31T21:12:19.595 DEBUG:boto:Method: GET 2023-05-31T21:12:19.595 DEBUG:boto:Path: /pklxkd-64/obj11 2023-05-31T21:12:19.596 DEBUG:boto:Data: 2023-05-31T21:12:19.596 DEBUG:boto:Headers: {} 2023-05-31T21:12:19.596 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T21:12:19.597 DEBUG:boto:Port: 8000 2023-05-31T21:12:19.597 DEBUG:boto:Params: {} 2023-05-31T21:12:19.597 DEBUG:boto:Token: None 2023-05-31T21:12:19.598 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 21:12:19 GMT /pklxkd-64/obj11 2023-05-31T21:12:19.598 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:iIB+1MW4PDp12ma/FQGl76unqys= 2023-05-31T21:12:19.599 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 21:12:19 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:iIB+1MW4PDp12ma/FQGl76unqys=', 'Content-Length': '0'} 2023-05-31T21:12:19.600 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 21:11:49 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009904d83b360f5c33-006477b833-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 21:12:19 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:12:19.601 DEBUG:boto:path=/pklxkd-64/ 2023-05-31T21:12:19.602 DEBUG:boto:auth_path=/pklxkd-64/ 2023-05-31T21:12:19.602 DEBUG:boto:path=/pklxkd-64/?prefix=obj11 2023-05-31T21:12:19.602 DEBUG:boto:auth_path=/pklxkd-64/?prefix=obj11 2023-05-31T21:12:19.603 DEBUG:boto:Method: GET 2023-05-31T21:12:19.603 DEBUG:boto:Path: /pklxkd-64/?prefix=obj11 2023-05-31T21:12:19.603 DEBUG:boto:Data: 2023-05-31T21:12:19.604 DEBUG:boto:Headers: {} 2023-05-31T21:12:19.604 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T21:12:19.604 DEBUG:boto:Port: 8001 2023-05-31T21:12:19.605 DEBUG:boto:Params: {} 2023-05-31T21:12:19.605 DEBUG:boto:Token: None 2023-05-31T21:12:19.606 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 21:12:19 GMT /pklxkd-64/ 2023-05-31T21:12:19.606 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:/ACajcjJ+wTjnzFp03OUBb/7FmU= 2023-05-31T21:12:19.607 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 21:12:19 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:/ACajcjJ+wTjnzFp03OUBb/7FmU=', 'Content-Length': '0'} 2023-05-31T21:12:19.609 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000032e3a758b471a239-006477b833-1fb6-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 21:12:19 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:12:19.609 DEBUG:boto:b'pklxkd-64obj111000falseobj112023-05-31T21:11:49.036Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T21:12:19.610 DEBUG:boto:path=/pklxkd-64/ 2023-05-31T21:12:19.610 DEBUG:boto:auth_path=/pklxkd-64/ 2023-05-31T21:12:19.611 DEBUG:boto:path=/pklxkd-64/?prefix=obj11 2023-05-31T21:12:19.611 DEBUG:boto:auth_path=/pklxkd-64/?prefix=obj11 2023-05-31T21:12:19.611 DEBUG:boto:Method: GET 2023-05-31T21:12:19.612 DEBUG:boto:Path: /pklxkd-64/?prefix=obj11 2023-05-31T21:12:19.612 DEBUG:boto:Data: 2023-05-31T21:12:19.612 DEBUG:boto:Headers: {} 2023-05-31T21:12:19.613 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T21:12:19.613 DEBUG:boto:Port: 8000 2023-05-31T21:12:19.614 DEBUG:boto:Params: {} 2023-05-31T21:12:19.614 DEBUG:boto:Token: None 2023-05-31T21:12:19.614 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 21:12:19 GMT /pklxkd-64/ 2023-05-31T21:12:19.615 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:/ACajcjJ+wTjnzFp03OUBb/7FmU= 2023-05-31T21:12:19.615 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 21:12:19 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:/ACajcjJ+wTjnzFp03OUBb/7FmU=', 'Content-Length': '0'} 2023-05-31T21:12:19.617 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000078b8c48d71fefd79-006477b833-1630-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 21:12:19 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:12:19.660 DEBUG:boto:b'pklxkd-64obj111000falseobj112023-05-31T21:11:49.036Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T21:12:19.661 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj22 2023-05-31T21:12:19.661 DEBUG:boto:path=/pklxkd-64/obj22 2023-05-31T21:12:19.662 DEBUG:boto:auth_path=/pklxkd-64/obj22 2023-05-31T21:12:19.662 DEBUG:boto:path=/pklxkd-64/obj22?versionId=null 2023-05-31T21:12:19.662 DEBUG:boto:auth_path=/pklxkd-64/obj22?versionId=null 2023-05-31T21:12:19.663 DEBUG:boto:Method: GET 2023-05-31T21:12:19.663 DEBUG:boto:Path: /pklxkd-64/obj22?versionId=null 2023-05-31T21:12:19.663 DEBUG:boto:Data: 2023-05-31T21:12:19.664 DEBUG:boto:Headers: {} 2023-05-31T21:12:19.664 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T21:12:19.664 DEBUG:boto:Port: 8001 2023-05-31T21:12:19.665 DEBUG:boto:Params: {} 2023-05-31T21:12:19.665 DEBUG:boto:Token: None 2023-05-31T21:12:19.666 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 21:12:19 GMT /pklxkd-64/obj22?versionId=null 2023-05-31T21:12:19.666 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:OZnIiljRkUdTAdQuQcOP60YYcNM= 2023-05-31T21:12:19.667 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 21:12:19 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:OZnIiljRkUdTAdQuQcOP60YYcNM=', 'Content-Length': '0'} 2023-05-31T21:12:19.668 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 21:11:49 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001a2a4c8db5672789-006477b833-1fb6-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 21:12:19 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:12:19.669 DEBUG:boto:path=/pklxkd-64/obj22 2023-05-31T21:12:19.670 DEBUG:boto:auth_path=/pklxkd-64/obj22 2023-05-31T21:12:19.670 DEBUG:boto:path=/pklxkd-64/obj22?versionId=null 2023-05-31T21:12:19.670 DEBUG:boto:auth_path=/pklxkd-64/obj22?versionId=null 2023-05-31T21:12:19.671 DEBUG:boto:Method: GET 2023-05-31T21:12:19.671 DEBUG:boto:Path: /pklxkd-64/obj22?versionId=null 2023-05-31T21:12:19.671 DEBUG:boto:Data: 2023-05-31T21:12:19.672 DEBUG:boto:Headers: {} 2023-05-31T21:12:19.672 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T21:12:19.673 DEBUG:boto:Port: 8000 2023-05-31T21:12:19.673 DEBUG:boto:Params: {} 2023-05-31T21:12:19.673 DEBUG:boto:Token: None 2023-05-31T21:12:19.674 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 21:12:19 GMT /pklxkd-64/obj22?versionId=null 2023-05-31T21:12:19.674 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:OZnIiljRkUdTAdQuQcOP60YYcNM= 2023-05-31T21:12:19.675 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 21:12:19 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:OZnIiljRkUdTAdQuQcOP60YYcNM=', 'Content-Length': '0'} 2023-05-31T21:12:19.676 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 21:11:49 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003456431745452446-006477b833-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 21:12:19 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:12:19.677 DEBUG:boto:path=/pklxkd-64/obj22 2023-05-31T21:12:19.677 DEBUG:boto:auth_path=/pklxkd-64/obj22 2023-05-31T21:12:19.678 DEBUG:boto:path=/pklxkd-64/obj22?versionId=null 2023-05-31T21:12:19.678 DEBUG:boto:auth_path=/pklxkd-64/obj22?versionId=null 2023-05-31T21:12:19.678 DEBUG:boto:Method: HEAD 2023-05-31T21:12:19.679 DEBUG:boto:Path: /pklxkd-64/obj22?versionId=null 2023-05-31T21:12:19.679 DEBUG:boto:Data: 2023-05-31T21:12:19.680 DEBUG:boto:Headers: {} 2023-05-31T21:12:19.680 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T21:12:19.680 DEBUG:boto:Port: 8001 2023-05-31T21:12:19.681 DEBUG:boto:Params: {} 2023-05-31T21:12:19.681 DEBUG:boto:Token: None 2023-05-31T21:12:19.681 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 21:12:19 GMT /pklxkd-64/obj22?versionId=null 2023-05-31T21:12:19.682 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:6GDQIzjluGv0gpA51bCI21tCHEs= 2023-05-31T21:12:19.682 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 21:12:19 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:6GDQIzjluGv0gpA51bCI21tCHEs=', 'Content-Length': '0'} 2023-05-31T21:12:19.684 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 21:11:49 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a8b2fd9edb88258b-006477b833-1fb6-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 21:12:19 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:12:19.684 DEBUG:boto:path=/pklxkd-64/obj22 2023-05-31T21:12:19.685 DEBUG:boto:auth_path=/pklxkd-64/obj22 2023-05-31T21:12:19.685 DEBUG:boto:path=/pklxkd-64/obj22?versionId=null 2023-05-31T21:12:19.685 DEBUG:boto:auth_path=/pklxkd-64/obj22?versionId=null 2023-05-31T21:12:19.686 DEBUG:boto:Method: HEAD 2023-05-31T21:12:19.686 DEBUG:boto:Path: /pklxkd-64/obj22?versionId=null 2023-05-31T21:12:19.687 DEBUG:boto:Data: 2023-05-31T21:12:19.687 DEBUG:boto:Headers: {} 2023-05-31T21:12:19.687 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T21:12:19.688 DEBUG:boto:Port: 8000 2023-05-31T21:12:19.688 DEBUG:boto:Params: {} 2023-05-31T21:12:19.688 DEBUG:boto:Token: None 2023-05-31T21:12:19.689 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 21:12:19 GMT /pklxkd-64/obj22?versionId=null 2023-05-31T21:12:19.689 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:6GDQIzjluGv0gpA51bCI21tCHEs= 2023-05-31T21:12:19.690 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 21:12:19 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:6GDQIzjluGv0gpA51bCI21tCHEs=', 'Content-Length': '0'} 2023-05-31T21:12:19.691 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 21:11:49 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008a6c9877956b9292-006477b833-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 21:12:19 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:12:19.692 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj22 2023-05-31T21:12:19.692 DEBUG:boto:path=/pklxkd-64/obj22 2023-05-31T21:12:19.693 DEBUG:boto:auth_path=/pklxkd-64/obj22 2023-05-31T21:12:19.693 DEBUG:boto:path=/pklxkd-64/obj22?versionId=null 2023-05-31T21:12:19.693 DEBUG:boto:auth_path=/pklxkd-64/obj22?versionId=null 2023-05-31T21:12:19.694 DEBUG:boto:Method: GET 2023-05-31T21:12:19.694 DEBUG:boto:Path: /pklxkd-64/obj22?versionId=null 2023-05-31T21:12:19.694 DEBUG:boto:Data: 2023-05-31T21:12:19.695 DEBUG:boto:Headers: {} 2023-05-31T21:12:19.695 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T21:12:19.696 DEBUG:boto:Port: 8001 2023-05-31T21:12:19.696 DEBUG:boto:Params: {} 2023-05-31T21:12:19.696 DEBUG:boto:Token: None 2023-05-31T21:12:19.697 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 21:12:19 GMT /pklxkd-64/obj22?versionId=null 2023-05-31T21:12:19.697 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:OZnIiljRkUdTAdQuQcOP60YYcNM= 2023-05-31T21:12:19.698 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 21:12:19 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:OZnIiljRkUdTAdQuQcOP60YYcNM=', 'Content-Length': '0'} 2023-05-31T21:12:19.700 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 21:11:49 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002d59a3b67ab06811-006477b833-1fb6-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 21:12:19 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:12:19.700 DEBUG:boto:path=/pklxkd-64/obj22 2023-05-31T21:12:19.701 DEBUG:boto:auth_path=/pklxkd-64/obj22 2023-05-31T21:12:19.701 DEBUG:boto:path=/pklxkd-64/obj22?versionId=null 2023-05-31T21:12:19.702 DEBUG:boto:auth_path=/pklxkd-64/obj22?versionId=null 2023-05-31T21:12:19.702 DEBUG:boto:Method: GET 2023-05-31T21:12:19.702 DEBUG:boto:Path: /pklxkd-64/obj22?versionId=null 2023-05-31T21:12:19.703 DEBUG:boto:Data: 2023-05-31T21:12:19.703 DEBUG:boto:Headers: {} 2023-05-31T21:12:19.703 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T21:12:19.704 DEBUG:boto:Port: 8000 2023-05-31T21:12:19.704 DEBUG:boto:Params: {} 2023-05-31T21:12:19.705 DEBUG:boto:Token: None 2023-05-31T21:12:19.705 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 21:12:19 GMT /pklxkd-64/obj22?versionId=null 2023-05-31T21:12:19.705 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:OZnIiljRkUdTAdQuQcOP60YYcNM= 2023-05-31T21:12:19.706 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 21:12:19 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:OZnIiljRkUdTAdQuQcOP60YYcNM=', 'Content-Length': '0'} 2023-05-31T21:12:19.708 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 21:11:49 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e0ee5c04fe00e14d-006477b833-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 21:12:19 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:12:19.708 DEBUG:boto:path=/pklxkd-64/ 2023-05-31T21:12:19.709 DEBUG:boto:auth_path=/pklxkd-64/ 2023-05-31T21:12:19.709 DEBUG:boto:path=/pklxkd-64/?prefix=obj22 2023-05-31T21:12:19.710 DEBUG:boto:auth_path=/pklxkd-64/?prefix=obj22 2023-05-31T21:12:19.710 DEBUG:boto:Method: GET 2023-05-31T21:12:19.710 DEBUG:boto:Path: /pklxkd-64/?prefix=obj22 2023-05-31T21:12:19.711 DEBUG:boto:Data: 2023-05-31T21:12:19.711 DEBUG:boto:Headers: {} 2023-05-31T21:12:19.711 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T21:12:19.712 DEBUG:boto:Port: 8001 2023-05-31T21:12:19.712 DEBUG:boto:Params: {} 2023-05-31T21:12:19.713 DEBUG:boto:Token: None 2023-05-31T21:12:19.713 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 21:12:19 GMT /pklxkd-64/ 2023-05-31T21:12:19.713 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:/ACajcjJ+wTjnzFp03OUBb/7FmU= 2023-05-31T21:12:19.714 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 21:12:19 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:/ACajcjJ+wTjnzFp03OUBb/7FmU=', 'Content-Length': '0'} 2023-05-31T21:12:19.716 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c9dc0401efa98e1f-006477b833-1fb6-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 21:12:19 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:12:19.717 DEBUG:boto:b'pklxkd-64obj221000falseobj222023-05-31T21:11:49.055Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T21:12:19.718 DEBUG:boto:path=/pklxkd-64/ 2023-05-31T21:12:19.718 DEBUG:boto:auth_path=/pklxkd-64/ 2023-05-31T21:12:19.718 DEBUG:boto:path=/pklxkd-64/?prefix=obj22 2023-05-31T21:12:19.719 DEBUG:boto:auth_path=/pklxkd-64/?prefix=obj22 2023-05-31T21:12:19.719 DEBUG:boto:Method: GET 2023-05-31T21:12:19.719 DEBUG:boto:Path: /pklxkd-64/?prefix=obj22 2023-05-31T21:12:19.720 DEBUG:boto:Data: 2023-05-31T21:12:19.720 DEBUG:boto:Headers: {} 2023-05-31T21:12:19.721 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T21:12:19.721 DEBUG:boto:Port: 8000 2023-05-31T21:12:19.721 DEBUG:boto:Params: {} 2023-05-31T21:12:19.722 DEBUG:boto:Token: None 2023-05-31T21:12:19.722 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 21:12:19 GMT /pklxkd-64/ 2023-05-31T21:12:19.723 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:/ACajcjJ+wTjnzFp03OUBb/7FmU= 2023-05-31T21:12:19.723 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 21:12:19 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:/ACajcjJ+wTjnzFp03OUBb/7FmU=', 'Content-Length': '0'} 2023-05-31T21:12:19.725 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000076226418bf2b7ba5-006477b833-1630-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 21:12:19 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:12:19.726 DEBUG:boto:b'pklxkd-64obj221000falseobj222023-05-31T21:11:49.055Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T21:12:19.726 DEBUG:boto:path=/pklxkd-64/obj22 2023-05-31T21:12:19.727 DEBUG:boto:auth_path=/pklxkd-64/obj22 2023-05-31T21:12:19.727 DEBUG:boto:Method: HEAD 2023-05-31T21:12:19.727 DEBUG:boto:Path: /pklxkd-64/obj22 2023-05-31T21:12:19.728 DEBUG:boto:Data: 2023-05-31T21:12:19.728 DEBUG:boto:Headers: {} 2023-05-31T21:12:19.729 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T21:12:19.729 DEBUG:boto:Port: 8001 2023-05-31T21:12:19.729 DEBUG:boto:Params: {} 2023-05-31T21:12:19.730 DEBUG:boto:Token: None 2023-05-31T21:12:19.730 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 21:12:19 GMT /pklxkd-64/obj22 2023-05-31T21:12:19.730 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:xbUN1gny4Y3f+TfLN6Z/p45Qfa4= 2023-05-31T21:12:19.731 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 21:12:19 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:xbUN1gny4Y3f+TfLN6Z/p45Qfa4=', 'Content-Length': '0'} 2023-05-31T21:12:19.733 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 21:11:49 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000921218d1fbf10413-006477b833-1fb6-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 21:12:19 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:12:19.733 DEBUG:boto:path=/pklxkd-64/obj22 2023-05-31T21:12:19.733 DEBUG:boto:auth_path=/pklxkd-64/obj22 2023-05-31T21:12:19.734 DEBUG:boto:Method: HEAD 2023-05-31T21:12:19.734 DEBUG:boto:Path: /pklxkd-64/obj22 2023-05-31T21:12:19.735 DEBUG:boto:Data: 2023-05-31T21:12:19.735 DEBUG:boto:Headers: {} 2023-05-31T21:12:19.735 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T21:12:19.736 DEBUG:boto:Port: 8000 2023-05-31T21:12:19.736 DEBUG:boto:Params: {} 2023-05-31T21:12:19.736 DEBUG:boto:Token: None 2023-05-31T21:12:19.737 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 21:12:19 GMT /pklxkd-64/obj22 2023-05-31T21:12:19.737 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:xbUN1gny4Y3f+TfLN6Z/p45Qfa4= 2023-05-31T21:12:19.738 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 21:12:19 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:xbUN1gny4Y3f+TfLN6Z/p45Qfa4=', 'Content-Length': '0'} 2023-05-31T21:12:19.739 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 21:11:49 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008dc2e38ab4c8027c-006477b833-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 21:12:19 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:12:19.740 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj22 2023-05-31T21:12:19.740 DEBUG:boto:path=/pklxkd-64/obj22 2023-05-31T21:12:19.741 DEBUG:boto:auth_path=/pklxkd-64/obj22 2023-05-31T21:12:19.741 DEBUG:boto:Method: GET 2023-05-31T21:12:19.741 DEBUG:boto:Path: /pklxkd-64/obj22 2023-05-31T21:12:19.742 DEBUG:boto:Data: 2023-05-31T21:12:19.742 DEBUG:boto:Headers: {} 2023-05-31T21:12:19.742 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T21:12:19.743 DEBUG:boto:Port: 8001 2023-05-31T21:12:19.743 DEBUG:boto:Params: {} 2023-05-31T21:12:19.744 DEBUG:boto:Token: None 2023-05-31T21:12:19.744 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 21:12:19 GMT /pklxkd-64/obj22 2023-05-31T21:12:19.744 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:LqymWWd+I4aNdHpCpypwt512aNY= 2023-05-31T21:12:19.745 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 21:12:19 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:LqymWWd+I4aNdHpCpypwt512aNY=', 'Content-Length': '0'} 2023-05-31T21:12:19.746 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 21:11:49 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000034db1197e1549773-006477b833-1fb6-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 21:12:19 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:12:19.788 DEBUG:boto:path=/pklxkd-64/obj22 2023-05-31T21:12:19.789 DEBUG:boto:auth_path=/pklxkd-64/obj22 2023-05-31T21:12:19.789 DEBUG:boto:Method: GET 2023-05-31T21:12:19.790 DEBUG:boto:Path: /pklxkd-64/obj22 2023-05-31T21:12:19.790 DEBUG:boto:Data: 2023-05-31T21:12:19.790 DEBUG:boto:Headers: {} 2023-05-31T21:12:19.791 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T21:12:19.791 DEBUG:boto:Port: 8000 2023-05-31T21:12:19.791 DEBUG:boto:Params: {} 2023-05-31T21:12:19.792 DEBUG:boto:Token: None 2023-05-31T21:12:19.792 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 21:12:19 GMT /pklxkd-64/obj22 2023-05-31T21:12:19.793 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:LqymWWd+I4aNdHpCpypwt512aNY= 2023-05-31T21:12:19.793 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 21:12:19 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:LqymWWd+I4aNdHpCpypwt512aNY=', 'Content-Length': '0'} 2023-05-31T21:12:19.795 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 21:11:49 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000013d5f4c7b81b6fcf-006477b833-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 21:12:19 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:12:19.796 DEBUG:boto:path=/pklxkd-64/ 2023-05-31T21:12:19.796 DEBUG:boto:auth_path=/pklxkd-64/ 2023-05-31T21:12:19.797 DEBUG:boto:path=/pklxkd-64/?prefix=obj22 2023-05-31T21:12:19.797 DEBUG:boto:auth_path=/pklxkd-64/?prefix=obj22 2023-05-31T21:12:19.797 DEBUG:boto:Method: GET 2023-05-31T21:12:19.798 DEBUG:boto:Path: /pklxkd-64/?prefix=obj22 2023-05-31T21:12:19.798 DEBUG:boto:Data: 2023-05-31T21:12:19.799 DEBUG:boto:Headers: {} 2023-05-31T21:12:19.799 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T21:12:19.799 DEBUG:boto:Port: 8001 2023-05-31T21:12:19.800 DEBUG:boto:Params: {} 2023-05-31T21:12:19.800 DEBUG:boto:Token: None 2023-05-31T21:12:19.801 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 21:12:19 GMT /pklxkd-64/ 2023-05-31T21:12:19.801 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:/ACajcjJ+wTjnzFp03OUBb/7FmU= 2023-05-31T21:12:19.801 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 21:12:19 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:/ACajcjJ+wTjnzFp03OUBb/7FmU=', 'Content-Length': '0'} 2023-05-31T21:12:19.804 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000498eece8668cc6c1-006477b833-1fb6-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 21:12:19 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:12:19.805 DEBUG:boto:b'pklxkd-64obj221000falseobj222023-05-31T21:11:49.055Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T21:12:19.805 DEBUG:boto:path=/pklxkd-64/ 2023-05-31T21:12:19.806 DEBUG:boto:auth_path=/pklxkd-64/ 2023-05-31T21:12:19.806 DEBUG:boto:path=/pklxkd-64/?prefix=obj22 2023-05-31T21:12:19.806 DEBUG:boto:auth_path=/pklxkd-64/?prefix=obj22 2023-05-31T21:12:19.807 DEBUG:boto:Method: GET 2023-05-31T21:12:19.807 DEBUG:boto:Path: /pklxkd-64/?prefix=obj22 2023-05-31T21:12:19.808 DEBUG:boto:Data: 2023-05-31T21:12:19.808 DEBUG:boto:Headers: {} 2023-05-31T21:12:19.808 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T21:12:19.809 DEBUG:boto:Port: 8000 2023-05-31T21:12:19.809 DEBUG:boto:Params: {} 2023-05-31T21:12:19.809 DEBUG:boto:Token: None 2023-05-31T21:12:19.810 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 21:12:19 GMT /pklxkd-64/ 2023-05-31T21:12:19.810 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:/ACajcjJ+wTjnzFp03OUBb/7FmU= 2023-05-31T21:12:19.811 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 21:12:19 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:/ACajcjJ+wTjnzFp03OUBb/7FmU=', 'Content-Length': '0'} 2023-05-31T21:12:19.813 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000009d7b945442a28b01-006477b833-1630-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 21:12:19 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:12:19.814 DEBUG:boto:b'pklxkd-64obj221000falseobj222023-05-31T21:11:49.055Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T21:12:19.814 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj33 2023-05-31T21:12:19.815 DEBUG:boto:path=/pklxkd-64/obj33 2023-05-31T21:12:19.815 DEBUG:boto:auth_path=/pklxkd-64/obj33 2023-05-31T21:12:19.816 DEBUG:boto:path=/pklxkd-64/obj33?versionId=null 2023-05-31T21:12:19.816 DEBUG:boto:auth_path=/pklxkd-64/obj33?versionId=null 2023-05-31T21:12:19.817 DEBUG:boto:Method: GET 2023-05-31T21:12:19.817 DEBUG:boto:Path: /pklxkd-64/obj33?versionId=null 2023-05-31T21:12:19.817 DEBUG:boto:Data: 2023-05-31T21:12:19.818 DEBUG:boto:Headers: {} 2023-05-31T21:12:19.818 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T21:12:19.819 DEBUG:boto:Port: 8001 2023-05-31T21:12:19.819 DEBUG:boto:Params: {} 2023-05-31T21:12:19.819 DEBUG:boto:Token: None 2023-05-31T21:12:19.820 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 21:12:19 GMT /pklxkd-64/obj33?versionId=null 2023-05-31T21:12:19.820 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:iYoF5oE6hcynDhEPAEuGmTfaklQ= 2023-05-31T21:12:19.821 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 21:12:19 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:iYoF5oE6hcynDhEPAEuGmTfaklQ=', 'Content-Length': '0'} 2023-05-31T21:12:19.822 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 21:11:49 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000063cd4400e3239841-006477b833-1fb6-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 21:12:19 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:12:19.823 DEBUG:boto:path=/pklxkd-64/obj33 2023-05-31T21:12:19.823 DEBUG:boto:auth_path=/pklxkd-64/obj33 2023-05-31T21:12:19.824 DEBUG:boto:path=/pklxkd-64/obj33?versionId=null 2023-05-31T21:12:19.824 DEBUG:boto:auth_path=/pklxkd-64/obj33?versionId=null 2023-05-31T21:12:19.825 DEBUG:boto:Method: GET 2023-05-31T21:12:19.825 DEBUG:boto:Path: /pklxkd-64/obj33?versionId=null 2023-05-31T21:12:19.825 DEBUG:boto:Data: 2023-05-31T21:12:19.826 DEBUG:boto:Headers: {} 2023-05-31T21:12:19.826 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T21:12:19.827 DEBUG:boto:Port: 8000 2023-05-31T21:12:19.827 DEBUG:boto:Params: {} 2023-05-31T21:12:19.827 DEBUG:boto:Token: None 2023-05-31T21:12:19.828 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 21:12:19 GMT /pklxkd-64/obj33?versionId=null 2023-05-31T21:12:19.828 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:iYoF5oE6hcynDhEPAEuGmTfaklQ= 2023-05-31T21:12:19.829 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 21:12:19 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:iYoF5oE6hcynDhEPAEuGmTfaklQ=', 'Content-Length': '0'} 2023-05-31T21:12:19.831 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 21:11:49 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007d67214e85dc9510-006477b833-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 21:12:19 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:12:19.831 DEBUG:boto:path=/pklxkd-64/obj33 2023-05-31T21:12:19.832 DEBUG:boto:auth_path=/pklxkd-64/obj33 2023-05-31T21:12:19.832 DEBUG:boto:path=/pklxkd-64/obj33?versionId=null 2023-05-31T21:12:19.832 DEBUG:boto:auth_path=/pklxkd-64/obj33?versionId=null 2023-05-31T21:12:19.833 DEBUG:boto:Method: HEAD 2023-05-31T21:12:19.833 DEBUG:boto:Path: /pklxkd-64/obj33?versionId=null 2023-05-31T21:12:19.833 DEBUG:boto:Data: 2023-05-31T21:12:19.834 DEBUG:boto:Headers: {} 2023-05-31T21:12:19.834 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T21:12:19.834 DEBUG:boto:Port: 8001 2023-05-31T21:12:19.835 DEBUG:boto:Params: {} 2023-05-31T21:12:19.835 DEBUG:boto:Token: None 2023-05-31T21:12:19.835 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 21:12:19 GMT /pklxkd-64/obj33?versionId=null 2023-05-31T21:12:19.836 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:TcU7F0hzqnrh05Rc4sPTJ3X8B6k= 2023-05-31T21:12:19.837 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 21:12:19 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:TcU7F0hzqnrh05Rc4sPTJ3X8B6k=', 'Content-Length': '0'} 2023-05-31T21:12:19.838 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 21:11:49 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009549219113534b95-006477b833-1fb6-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 21:12:19 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:12:19.839 DEBUG:boto:path=/pklxkd-64/obj33 2023-05-31T21:12:19.839 DEBUG:boto:auth_path=/pklxkd-64/obj33 2023-05-31T21:12:19.840 DEBUG:boto:path=/pklxkd-64/obj33?versionId=null 2023-05-31T21:12:19.840 DEBUG:boto:auth_path=/pklxkd-64/obj33?versionId=null 2023-05-31T21:12:19.840 DEBUG:boto:Method: HEAD 2023-05-31T21:12:19.841 DEBUG:boto:Path: /pklxkd-64/obj33?versionId=null 2023-05-31T21:12:19.841 DEBUG:boto:Data: 2023-05-31T21:12:19.841 DEBUG:boto:Headers: {} 2023-05-31T21:12:19.842 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T21:12:19.842 DEBUG:boto:Port: 8000 2023-05-31T21:12:19.842 DEBUG:boto:Params: {} 2023-05-31T21:12:19.843 DEBUG:boto:Token: None 2023-05-31T21:12:19.843 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 21:12:19 GMT /pklxkd-64/obj33?versionId=null 2023-05-31T21:12:19.843 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:TcU7F0hzqnrh05Rc4sPTJ3X8B6k= 2023-05-31T21:12:19.844 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 21:12:19 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:TcU7F0hzqnrh05Rc4sPTJ3X8B6k=', 'Content-Length': '0'} 2023-05-31T21:12:19.845 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 21:11:49 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b39a0f55c0643646-006477b833-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 21:12:19 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:12:19.846 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj33 2023-05-31T21:12:19.846 DEBUG:boto:path=/pklxkd-64/obj33 2023-05-31T21:12:19.847 DEBUG:boto:auth_path=/pklxkd-64/obj33 2023-05-31T21:12:19.847 DEBUG:boto:path=/pklxkd-64/obj33?versionId=null 2023-05-31T21:12:19.847 DEBUG:boto:auth_path=/pklxkd-64/obj33?versionId=null 2023-05-31T21:12:19.848 DEBUG:boto:Method: GET 2023-05-31T21:12:19.848 DEBUG:boto:Path: /pklxkd-64/obj33?versionId=null 2023-05-31T21:12:19.849 DEBUG:boto:Data: 2023-05-31T21:12:19.849 DEBUG:boto:Headers: {} 2023-05-31T21:12:19.849 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T21:12:19.850 DEBUG:boto:Port: 8001 2023-05-31T21:12:19.850 DEBUG:boto:Params: {} 2023-05-31T21:12:19.851 DEBUG:boto:Token: None 2023-05-31T21:12:19.851 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 21:12:19 GMT /pklxkd-64/obj33?versionId=null 2023-05-31T21:12:19.851 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:iYoF5oE6hcynDhEPAEuGmTfaklQ= 2023-05-31T21:12:19.852 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 21:12:19 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:iYoF5oE6hcynDhEPAEuGmTfaklQ=', 'Content-Length': '0'} 2023-05-31T21:12:19.854 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 21:11:49 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000f27337af0ad23bf-006477b833-1fb6-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 21:12:19 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:12:19.854 DEBUG:boto:path=/pklxkd-64/obj33 2023-05-31T21:12:19.855 DEBUG:boto:auth_path=/pklxkd-64/obj33 2023-05-31T21:12:19.855 DEBUG:boto:path=/pklxkd-64/obj33?versionId=null 2023-05-31T21:12:19.855 DEBUG:boto:auth_path=/pklxkd-64/obj33?versionId=null 2023-05-31T21:12:19.856 DEBUG:boto:Method: GET 2023-05-31T21:12:19.856 DEBUG:boto:Path: /pklxkd-64/obj33?versionId=null 2023-05-31T21:12:19.856 DEBUG:boto:Data: 2023-05-31T21:12:19.857 DEBUG:boto:Headers: {} 2023-05-31T21:12:19.857 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T21:12:19.858 DEBUG:boto:Port: 8000 2023-05-31T21:12:19.858 DEBUG:boto:Params: {} 2023-05-31T21:12:19.858 DEBUG:boto:Token: None 2023-05-31T21:12:19.859 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 21:12:19 GMT /pklxkd-64/obj33?versionId=null 2023-05-31T21:12:19.859 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:iYoF5oE6hcynDhEPAEuGmTfaklQ= 2023-05-31T21:12:19.860 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 21:12:19 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:iYoF5oE6hcynDhEPAEuGmTfaklQ=', 'Content-Length': '0'} 2023-05-31T21:12:19.862 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 21:11:49 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004c9e36fe1be1349b-006477b833-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 21:12:19 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:12:19.862 DEBUG:boto:path=/pklxkd-64/ 2023-05-31T21:12:19.863 DEBUG:boto:auth_path=/pklxkd-64/ 2023-05-31T21:12:19.863 DEBUG:boto:path=/pklxkd-64/?prefix=obj33 2023-05-31T21:12:19.863 DEBUG:boto:auth_path=/pklxkd-64/?prefix=obj33 2023-05-31T21:12:19.864 DEBUG:boto:Method: GET 2023-05-31T21:12:19.864 DEBUG:boto:Path: /pklxkd-64/?prefix=obj33 2023-05-31T21:12:19.865 DEBUG:boto:Data: 2023-05-31T21:12:19.865 DEBUG:boto:Headers: {} 2023-05-31T21:12:19.865 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T21:12:19.866 DEBUG:boto:Port: 8001 2023-05-31T21:12:19.866 DEBUG:boto:Params: {} 2023-05-31T21:12:19.866 DEBUG:boto:Token: None 2023-05-31T21:12:19.867 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 21:12:19 GMT /pklxkd-64/ 2023-05-31T21:12:19.867 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:/ACajcjJ+wTjnzFp03OUBb/7FmU= 2023-05-31T21:12:19.868 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 21:12:19 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:/ACajcjJ+wTjnzFp03OUBb/7FmU=', 'Content-Length': '0'} 2023-05-31T21:12:19.870 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c081e3df9d2a930e-006477b833-1fb6-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 21:12:19 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:12:19.870 DEBUG:boto:b'pklxkd-64obj331000falseobj332023-05-31T21:11:49.072Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T21:12:19.871 DEBUG:boto:path=/pklxkd-64/ 2023-05-31T21:12:19.872 DEBUG:boto:auth_path=/pklxkd-64/ 2023-05-31T21:12:19.872 DEBUG:boto:path=/pklxkd-64/?prefix=obj33 2023-05-31T21:12:19.872 DEBUG:boto:auth_path=/pklxkd-64/?prefix=obj33 2023-05-31T21:12:19.873 DEBUG:boto:Method: GET 2023-05-31T21:12:19.873 DEBUG:boto:Path: /pklxkd-64/?prefix=obj33 2023-05-31T21:12:19.873 DEBUG:boto:Data: 2023-05-31T21:12:19.874 DEBUG:boto:Headers: {} 2023-05-31T21:12:19.874 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T21:12:19.874 DEBUG:boto:Port: 8000 2023-05-31T21:12:19.875 DEBUG:boto:Params: {} 2023-05-31T21:12:19.875 DEBUG:boto:Token: None 2023-05-31T21:12:19.876 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 21:12:19 GMT /pklxkd-64/ 2023-05-31T21:12:19.876 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:/ACajcjJ+wTjnzFp03OUBb/7FmU= 2023-05-31T21:12:19.876 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 21:12:19 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:/ACajcjJ+wTjnzFp03OUBb/7FmU=', 'Content-Length': '0'} 2023-05-31T21:12:19.879 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000023995f2146040a59-006477b833-1630-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 21:12:19 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:12:19.880 DEBUG:boto:b'pklxkd-64obj331000falseobj332023-05-31T21:11:49.072Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T21:12:19.880 DEBUG:boto:path=/pklxkd-64/obj33 2023-05-31T21:12:19.881 DEBUG:boto:auth_path=/pklxkd-64/obj33 2023-05-31T21:12:19.881 DEBUG:boto:Method: HEAD 2023-05-31T21:12:19.881 DEBUG:boto:Path: /pklxkd-64/obj33 2023-05-31T21:12:19.882 DEBUG:boto:Data: 2023-05-31T21:12:19.882 DEBUG:boto:Headers: {} 2023-05-31T21:12:19.882 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T21:12:19.883 DEBUG:boto:Port: 8001 2023-05-31T21:12:19.883 DEBUG:boto:Params: {} 2023-05-31T21:12:19.883 DEBUG:boto:Token: None 2023-05-31T21:12:19.884 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 21:12:19 GMT /pklxkd-64/obj33 2023-05-31T21:12:19.884 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:mRCtkdEvAE91nG83ueDl6lKeNdE= 2023-05-31T21:12:19.885 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 21:12:19 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:mRCtkdEvAE91nG83ueDl6lKeNdE=', 'Content-Length': '0'} 2023-05-31T21:12:19.887 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 21:11:49 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006efed62d7d44ebbf-006477b833-1fb6-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 21:12:19 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:12:19.887 DEBUG:boto:path=/pklxkd-64/obj33 2023-05-31T21:12:19.888 DEBUG:boto:auth_path=/pklxkd-64/obj33 2023-05-31T21:12:19.888 DEBUG:boto:Method: HEAD 2023-05-31T21:12:19.888 DEBUG:boto:Path: /pklxkd-64/obj33 2023-05-31T21:12:19.889 DEBUG:boto:Data: 2023-05-31T21:12:19.889 DEBUG:boto:Headers: {} 2023-05-31T21:12:19.889 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T21:12:19.890 DEBUG:boto:Port: 8000 2023-05-31T21:12:19.890 DEBUG:boto:Params: {} 2023-05-31T21:12:19.890 DEBUG:boto:Token: None 2023-05-31T21:12:19.891 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 21:12:19 GMT /pklxkd-64/obj33 2023-05-31T21:12:19.891 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:mRCtkdEvAE91nG83ueDl6lKeNdE= 2023-05-31T21:12:19.892 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 21:12:19 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:mRCtkdEvAE91nG83ueDl6lKeNdE=', 'Content-Length': '0'} 2023-05-31T21:12:19.893 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 21:11:49 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000575ad72a3c5e51fc-006477b833-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 21:12:19 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:12:19.894 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj33 2023-05-31T21:12:19.894 DEBUG:boto:path=/pklxkd-64/obj33 2023-05-31T21:12:19.895 DEBUG:boto:auth_path=/pklxkd-64/obj33 2023-05-31T21:12:19.895 DEBUG:boto:Method: GET 2023-05-31T21:12:19.895 DEBUG:boto:Path: /pklxkd-64/obj33 2023-05-31T21:12:19.896 DEBUG:boto:Data: 2023-05-31T21:12:19.896 DEBUG:boto:Headers: {} 2023-05-31T21:12:19.897 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T21:12:19.897 DEBUG:boto:Port: 8001 2023-05-31T21:12:19.897 DEBUG:boto:Params: {} 2023-05-31T21:12:19.898 DEBUG:boto:Token: None 2023-05-31T21:12:19.898 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 21:12:19 GMT /pklxkd-64/obj33 2023-05-31T21:12:19.898 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:5vzeOAOjDn24xq6OH9FzejuQMMc= 2023-05-31T21:12:19.899 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 21:12:19 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:5vzeOAOjDn24xq6OH9FzejuQMMc=', 'Content-Length': '0'} 2023-05-31T21:12:19.901 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 21:11:49 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000de4e4dfbd48df35c-006477b833-1fb6-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 21:12:19 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:12:19.901 DEBUG:boto:path=/pklxkd-64/obj33 2023-05-31T21:12:19.902 DEBUG:boto:auth_path=/pklxkd-64/obj33 2023-05-31T21:12:19.902 DEBUG:boto:Method: GET 2023-05-31T21:12:19.903 DEBUG:boto:Path: /pklxkd-64/obj33 2023-05-31T21:12:19.903 DEBUG:boto:Data: 2023-05-31T21:12:19.903 DEBUG:boto:Headers: {} 2023-05-31T21:12:19.904 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T21:12:19.904 DEBUG:boto:Port: 8000 2023-05-31T21:12:19.904 DEBUG:boto:Params: {} 2023-05-31T21:12:19.905 DEBUG:boto:Token: None 2023-05-31T21:12:19.905 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 21:12:19 GMT /pklxkd-64/obj33 2023-05-31T21:12:19.906 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:5vzeOAOjDn24xq6OH9FzejuQMMc= 2023-05-31T21:12:19.906 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 21:12:19 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:5vzeOAOjDn24xq6OH9FzejuQMMc=', 'Content-Length': '0'} 2023-05-31T21:12:19.908 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 21:11:49 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a1b8646e1ab404de-006477b833-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 21:12:19 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:12:19.908 DEBUG:boto:path=/pklxkd-64/ 2023-05-31T21:12:19.909 DEBUG:boto:auth_path=/pklxkd-64/ 2023-05-31T21:12:19.909 DEBUG:boto:path=/pklxkd-64/?prefix=obj33 2023-05-31T21:12:19.910 DEBUG:boto:auth_path=/pklxkd-64/?prefix=obj33 2023-05-31T21:12:19.910 DEBUG:boto:Method: GET 2023-05-31T21:12:19.911 DEBUG:boto:Path: /pklxkd-64/?prefix=obj33 2023-05-31T21:12:19.911 DEBUG:boto:Data: 2023-05-31T21:12:19.911 DEBUG:boto:Headers: {} 2023-05-31T21:12:19.912 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T21:12:19.912 DEBUG:boto:Port: 8001 2023-05-31T21:12:19.912 DEBUG:boto:Params: {} 2023-05-31T21:12:19.913 DEBUG:boto:Token: None 2023-05-31T21:12:19.913 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 21:12:19 GMT /pklxkd-64/ 2023-05-31T21:12:19.914 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:/ACajcjJ+wTjnzFp03OUBb/7FmU= 2023-05-31T21:12:19.914 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 21:12:19 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:/ACajcjJ+wTjnzFp03OUBb/7FmU=', 'Content-Length': '0'} 2023-05-31T21:12:19.917 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000001ee9029dabb1fd07-006477b833-1fb6-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 21:12:19 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:12:19.960 DEBUG:boto:b'pklxkd-64obj331000falseobj332023-05-31T21:11:49.072Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T21:12:19.961 DEBUG:boto:path=/pklxkd-64/ 2023-05-31T21:12:19.961 DEBUG:boto:auth_path=/pklxkd-64/ 2023-05-31T21:12:19.962 DEBUG:boto:path=/pklxkd-64/?prefix=obj33 2023-05-31T21:12:19.962 DEBUG:boto:auth_path=/pklxkd-64/?prefix=obj33 2023-05-31T21:12:19.962 DEBUG:boto:Method: GET 2023-05-31T21:12:19.963 DEBUG:boto:Path: /pklxkd-64/?prefix=obj33 2023-05-31T21:12:19.963 DEBUG:boto:Data: 2023-05-31T21:12:19.963 DEBUG:boto:Headers: {} 2023-05-31T21:12:19.964 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T21:12:19.964 DEBUG:boto:Port: 8000 2023-05-31T21:12:19.964 DEBUG:boto:Params: {} 2023-05-31T21:12:19.965 DEBUG:boto:Token: None 2023-05-31T21:12:19.965 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 21:12:19 GMT /pklxkd-64/ 2023-05-31T21:12:19.966 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:/ACajcjJ+wTjnzFp03OUBb/7FmU= 2023-05-31T21:12:19.966 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 21:12:19 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:/ACajcjJ+wTjnzFp03OUBb/7FmU=', 'Content-Length': '0'} 2023-05-31T21:12:19.968 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a43dca39185a5522-006477b833-1630-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 21:12:19 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:12:19.969 DEBUG:boto:b'pklxkd-64obj331000falseobj332023-05-31T21:11:49.072Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T21:12:19.969 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj44 2023-05-31T21:12:19.970 DEBUG:boto:path=/pklxkd-64/obj44 2023-05-31T21:12:19.970 DEBUG:boto:auth_path=/pklxkd-64/obj44 2023-05-31T21:12:19.971 DEBUG:boto:path=/pklxkd-64/obj44?versionId=null 2023-05-31T21:12:19.971 DEBUG:boto:auth_path=/pklxkd-64/obj44?versionId=null 2023-05-31T21:12:19.971 DEBUG:boto:Method: GET 2023-05-31T21:12:19.972 DEBUG:boto:Path: /pklxkd-64/obj44?versionId=null 2023-05-31T21:12:19.972 DEBUG:boto:Data: 2023-05-31T21:12:19.972 DEBUG:boto:Headers: {} 2023-05-31T21:12:19.973 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T21:12:19.973 DEBUG:boto:Port: 8001 2023-05-31T21:12:19.974 DEBUG:boto:Params: {} 2023-05-31T21:12:19.974 DEBUG:boto:Token: None 2023-05-31T21:12:19.974 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 21:12:19 GMT /pklxkd-64/obj44?versionId=null 2023-05-31T21:12:19.975 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:z9fRV0/Yh/KKEef4iAzn+B+wCIQ= 2023-05-31T21:12:19.975 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 21:12:19 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:z9fRV0/Yh/KKEef4iAzn+B+wCIQ=', 'Content-Length': '0'} 2023-05-31T21:12:19.977 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 21:11:49 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003b0401011baccec4-006477b833-1fb6-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 21:12:19 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:12:19.978 DEBUG:boto:path=/pklxkd-64/obj44 2023-05-31T21:12:19.978 DEBUG:boto:auth_path=/pklxkd-64/obj44 2023-05-31T21:12:19.978 DEBUG:boto:path=/pklxkd-64/obj44?versionId=null 2023-05-31T21:12:19.979 DEBUG:boto:auth_path=/pklxkd-64/obj44?versionId=null 2023-05-31T21:12:19.979 DEBUG:boto:Method: GET 2023-05-31T21:12:19.979 DEBUG:boto:Path: /pklxkd-64/obj44?versionId=null 2023-05-31T21:12:19.980 DEBUG:boto:Data: 2023-05-31T21:12:19.980 DEBUG:boto:Headers: {} 2023-05-31T21:12:19.981 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T21:12:19.981 DEBUG:boto:Port: 8000 2023-05-31T21:12:19.981 DEBUG:boto:Params: {} 2023-05-31T21:12:19.982 DEBUG:boto:Token: None 2023-05-31T21:12:19.982 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 21:12:19 GMT /pklxkd-64/obj44?versionId=null 2023-05-31T21:12:19.982 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:z9fRV0/Yh/KKEef4iAzn+B+wCIQ= 2023-05-31T21:12:19.983 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 21:12:19 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:z9fRV0/Yh/KKEef4iAzn+B+wCIQ=', 'Content-Length': '0'} 2023-05-31T21:12:19.988 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 21:11:49 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006b6d1e8ee42fb8c5-006477b833-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 21:12:19 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:12:19.989 DEBUG:boto:path=/pklxkd-64/obj44 2023-05-31T21:12:19.989 DEBUG:boto:auth_path=/pklxkd-64/obj44 2023-05-31T21:12:19.990 DEBUG:boto:path=/pklxkd-64/obj44?versionId=null 2023-05-31T21:12:19.990 DEBUG:boto:auth_path=/pklxkd-64/obj44?versionId=null 2023-05-31T21:12:19.990 DEBUG:boto:Method: HEAD 2023-05-31T21:12:19.991 DEBUG:boto:Path: /pklxkd-64/obj44?versionId=null 2023-05-31T21:12:19.991 DEBUG:boto:Data: 2023-05-31T21:12:19.991 DEBUG:boto:Headers: {} 2023-05-31T21:12:19.992 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T21:12:19.992 DEBUG:boto:Port: 8001 2023-05-31T21:12:19.992 DEBUG:boto:Params: {} 2023-05-31T21:12:19.993 DEBUG:boto:Token: None 2023-05-31T21:12:19.993 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 21:12:19 GMT /pklxkd-64/obj44?versionId=null 2023-05-31T21:12:19.993 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:ibhi57mXKqH+2CYQztxin9SGqWg= 2023-05-31T21:12:19.994 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 21:12:19 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:ibhi57mXKqH+2CYQztxin9SGqWg=', 'Content-Length': '0'} 2023-05-31T21:12:19.995 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 21:11:49 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000059566caf0c9a59d1-006477b833-1fb6-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 21:12:19 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:12:19.996 DEBUG:boto:path=/pklxkd-64/obj44 2023-05-31T21:12:19.996 DEBUG:boto:auth_path=/pklxkd-64/obj44 2023-05-31T21:12:19.997 DEBUG:boto:path=/pklxkd-64/obj44?versionId=null 2023-05-31T21:12:19.997 DEBUG:boto:auth_path=/pklxkd-64/obj44?versionId=null 2023-05-31T21:12:19.997 DEBUG:boto:Method: HEAD 2023-05-31T21:12:19.998 DEBUG:boto:Path: /pklxkd-64/obj44?versionId=null 2023-05-31T21:12:19.998 DEBUG:boto:Data: 2023-05-31T21:12:19.998 DEBUG:boto:Headers: {} 2023-05-31T21:12:19.999 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T21:12:19.999 DEBUG:boto:Port: 8000 2023-05-31T21:12:19.999 DEBUG:boto:Params: {} 2023-05-31T21:12:20.000 DEBUG:boto:Token: None 2023-05-31T21:12:20.000 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 21:12:20 GMT /pklxkd-64/obj44?versionId=null 2023-05-31T21:12:20.001 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:k3nwDZhgaR8EUg57JyJlccWnVeo= 2023-05-31T21:12:20.001 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 21:12:20 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:k3nwDZhgaR8EUg57JyJlccWnVeo=', 'Content-Length': '0'} 2023-05-31T21:12:20.003 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 21:11:49 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007874cd1e16bb6878-006477b833-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 21:12:20 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:12:20.003 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj44 2023-05-31T21:12:20.004 DEBUG:boto:path=/pklxkd-64/obj44 2023-05-31T21:12:20.004 DEBUG:boto:auth_path=/pklxkd-64/obj44 2023-05-31T21:12:20.004 DEBUG:boto:path=/pklxkd-64/obj44?versionId=null 2023-05-31T21:12:20.005 DEBUG:boto:auth_path=/pklxkd-64/obj44?versionId=null 2023-05-31T21:12:20.005 DEBUG:boto:Method: GET 2023-05-31T21:12:20.005 DEBUG:boto:Path: /pklxkd-64/obj44?versionId=null 2023-05-31T21:12:20.006 DEBUG:boto:Data: 2023-05-31T21:12:20.006 DEBUG:boto:Headers: {} 2023-05-31T21:12:20.007 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T21:12:20.007 DEBUG:boto:Port: 8001 2023-05-31T21:12:20.007 DEBUG:boto:Params: {} 2023-05-31T21:12:20.008 DEBUG:boto:Token: None 2023-05-31T21:12:20.008 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 21:12:20 GMT /pklxkd-64/obj44?versionId=null 2023-05-31T21:12:20.009 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:u+nw0+IA9IGHYaKonVjMTqWl3Cw= 2023-05-31T21:12:20.009 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 21:12:20 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:u+nw0+IA9IGHYaKonVjMTqWl3Cw=', 'Content-Length': '0'} 2023-05-31T21:12:20.011 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 21:11:49 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000cecb2855bc60cdd1-006477b834-1fb6-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 21:12:20 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:12:20.012 DEBUG:boto:path=/pklxkd-64/obj44 2023-05-31T21:12:20.012 DEBUG:boto:auth_path=/pklxkd-64/obj44 2023-05-31T21:12:20.012 DEBUG:boto:path=/pklxkd-64/obj44?versionId=null 2023-05-31T21:12:20.013 DEBUG:boto:auth_path=/pklxkd-64/obj44?versionId=null 2023-05-31T21:12:20.013 DEBUG:boto:Method: GET 2023-05-31T21:12:20.013 DEBUG:boto:Path: /pklxkd-64/obj44?versionId=null 2023-05-31T21:12:20.014 DEBUG:boto:Data: 2023-05-31T21:12:20.014 DEBUG:boto:Headers: {} 2023-05-31T21:12:20.014 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T21:12:20.015 DEBUG:boto:Port: 8000 2023-05-31T21:12:20.015 DEBUG:boto:Params: {} 2023-05-31T21:12:20.016 DEBUG:boto:Token: None 2023-05-31T21:12:20.016 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 21:12:20 GMT /pklxkd-64/obj44?versionId=null 2023-05-31T21:12:20.016 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:u+nw0+IA9IGHYaKonVjMTqWl3Cw= 2023-05-31T21:12:20.017 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 21:12:20 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:u+nw0+IA9IGHYaKonVjMTqWl3Cw=', 'Content-Length': '0'} 2023-05-31T21:12:20.019 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 21:11:49 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e88e1f67f32b0cbd-006477b834-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 21:12:20 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:12:20.060 DEBUG:boto:path=/pklxkd-64/ 2023-05-31T21:12:20.061 DEBUG:boto:auth_path=/pklxkd-64/ 2023-05-31T21:12:20.061 DEBUG:boto:path=/pklxkd-64/?prefix=obj44 2023-05-31T21:12:20.061 DEBUG:boto:auth_path=/pklxkd-64/?prefix=obj44 2023-05-31T21:12:20.062 DEBUG:boto:Method: GET 2023-05-31T21:12:20.062 DEBUG:boto:Path: /pklxkd-64/?prefix=obj44 2023-05-31T21:12:20.062 DEBUG:boto:Data: 2023-05-31T21:12:20.063 DEBUG:boto:Headers: {} 2023-05-31T21:12:20.063 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T21:12:20.064 DEBUG:boto:Port: 8001 2023-05-31T21:12:20.064 DEBUG:boto:Params: {} 2023-05-31T21:12:20.064 DEBUG:boto:Token: None 2023-05-31T21:12:20.065 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 21:12:20 GMT /pklxkd-64/ 2023-05-31T21:12:20.065 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:eDdIGyOBSPGk4dT06OEG2smpfgM= 2023-05-31T21:12:20.066 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 21:12:20 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:eDdIGyOBSPGk4dT06OEG2smpfgM=', 'Content-Length': '0'} 2023-05-31T21:12:20.068 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e690366cf18d4124-006477b834-1fb6-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 21:12:20 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:12:20.069 DEBUG:boto:b'pklxkd-64obj441000falseobj442023-05-31T21:11:49.126Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T21:12:20.069 DEBUG:boto:path=/pklxkd-64/ 2023-05-31T21:12:20.070 DEBUG:boto:auth_path=/pklxkd-64/ 2023-05-31T21:12:20.070 DEBUG:boto:path=/pklxkd-64/?prefix=obj44 2023-05-31T21:12:20.070 DEBUG:boto:auth_path=/pklxkd-64/?prefix=obj44 2023-05-31T21:12:20.071 DEBUG:boto:Method: GET 2023-05-31T21:12:20.071 DEBUG:boto:Path: /pklxkd-64/?prefix=obj44 2023-05-31T21:12:20.071 DEBUG:boto:Data: 2023-05-31T21:12:20.072 DEBUG:boto:Headers: {} 2023-05-31T21:12:20.072 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T21:12:20.072 DEBUG:boto:Port: 8000 2023-05-31T21:12:20.073 DEBUG:boto:Params: {} 2023-05-31T21:12:20.073 DEBUG:boto:Token: None 2023-05-31T21:12:20.074 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 21:12:20 GMT /pklxkd-64/ 2023-05-31T21:12:20.074 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:eDdIGyOBSPGk4dT06OEG2smpfgM= 2023-05-31T21:12:20.075 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 21:12:20 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:eDdIGyOBSPGk4dT06OEG2smpfgM=', 'Content-Length': '0'} 2023-05-31T21:12:20.077 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f7e7658991dc86aa-006477b834-1630-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 21:12:20 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:12:20.078 DEBUG:boto:b'pklxkd-64obj441000falseobj442023-05-31T21:11:49.126Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T21:12:20.078 DEBUG:boto:path=/pklxkd-64/obj44 2023-05-31T21:12:20.079 DEBUG:boto:auth_path=/pklxkd-64/obj44 2023-05-31T21:12:20.079 DEBUG:boto:Method: HEAD 2023-05-31T21:12:20.079 DEBUG:boto:Path: /pklxkd-64/obj44 2023-05-31T21:12:20.080 DEBUG:boto:Data: 2023-05-31T21:12:20.080 DEBUG:boto:Headers: {} 2023-05-31T21:12:20.080 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T21:12:20.081 DEBUG:boto:Port: 8001 2023-05-31T21:12:20.081 DEBUG:boto:Params: {} 2023-05-31T21:12:20.081 DEBUG:boto:Token: None 2023-05-31T21:12:20.082 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 21:12:20 GMT /pklxkd-64/obj44 2023-05-31T21:12:20.082 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:elrRkq62ZvBip1R1pOI1SUb3yyg= 2023-05-31T21:12:20.083 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 21:12:20 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:elrRkq62ZvBip1R1pOI1SUb3yyg=', 'Content-Length': '0'} 2023-05-31T21:12:20.084 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 21:11:49 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f6cd579eeb358040-006477b834-1fb6-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 21:12:20 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:12:20.085 DEBUG:boto:path=/pklxkd-64/obj44 2023-05-31T21:12:20.085 DEBUG:boto:auth_path=/pklxkd-64/obj44 2023-05-31T21:12:20.086 DEBUG:boto:Method: HEAD 2023-05-31T21:12:20.086 DEBUG:boto:Path: /pklxkd-64/obj44 2023-05-31T21:12:20.086 DEBUG:boto:Data: 2023-05-31T21:12:20.087 DEBUG:boto:Headers: {} 2023-05-31T21:12:20.087 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T21:12:20.087 DEBUG:boto:Port: 8000 2023-05-31T21:12:20.088 DEBUG:boto:Params: {} 2023-05-31T21:12:20.088 DEBUG:boto:Token: None 2023-05-31T21:12:20.088 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 21:12:20 GMT /pklxkd-64/obj44 2023-05-31T21:12:20.089 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:elrRkq62ZvBip1R1pOI1SUb3yyg= 2023-05-31T21:12:20.089 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 21:12:20 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:elrRkq62ZvBip1R1pOI1SUb3yyg=', 'Content-Length': '0'} 2023-05-31T21:12:20.091 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 21:11:49 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f16745f4a0aa1fed-006477b834-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 21:12:20 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:12:20.091 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj44 2023-05-31T21:12:20.092 DEBUG:boto:path=/pklxkd-64/obj44 2023-05-31T21:12:20.092 DEBUG:boto:auth_path=/pklxkd-64/obj44 2023-05-31T21:12:20.092 DEBUG:boto:Method: GET 2023-05-31T21:12:20.093 DEBUG:boto:Path: /pklxkd-64/obj44 2023-05-31T21:12:20.093 DEBUG:boto:Data: 2023-05-31T21:12:20.094 DEBUG:boto:Headers: {} 2023-05-31T21:12:20.094 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T21:12:20.094 DEBUG:boto:Port: 8001 2023-05-31T21:12:20.095 DEBUG:boto:Params: {} 2023-05-31T21:12:20.095 DEBUG:boto:Token: None 2023-05-31T21:12:20.095 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 21:12:20 GMT /pklxkd-64/obj44 2023-05-31T21:12:20.096 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:lLDDwRw4jQfWrAY9MrH2RMIZY3c= 2023-05-31T21:12:20.096 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 21:12:20 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:lLDDwRw4jQfWrAY9MrH2RMIZY3c=', 'Content-Length': '0'} 2023-05-31T21:12:20.098 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 21:11:49 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ec9f776c8038cfb6-006477b834-1fb6-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 21:12:20 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:12:20.140 DEBUG:boto:path=/pklxkd-64/obj44 2023-05-31T21:12:20.141 DEBUG:boto:auth_path=/pklxkd-64/obj44 2023-05-31T21:12:20.141 DEBUG:boto:Method: GET 2023-05-31T21:12:20.142 DEBUG:boto:Path: /pklxkd-64/obj44 2023-05-31T21:12:20.142 DEBUG:boto:Data: 2023-05-31T21:12:20.142 DEBUG:boto:Headers: {} 2023-05-31T21:12:20.143 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T21:12:20.143 DEBUG:boto:Port: 8000 2023-05-31T21:12:20.143 DEBUG:boto:Params: {} 2023-05-31T21:12:20.144 DEBUG:boto:Token: None 2023-05-31T21:12:20.144 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 21:12:20 GMT /pklxkd-64/obj44 2023-05-31T21:12:20.145 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:lLDDwRw4jQfWrAY9MrH2RMIZY3c= 2023-05-31T21:12:20.145 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 21:12:20 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:lLDDwRw4jQfWrAY9MrH2RMIZY3c=', 'Content-Length': '0'} 2023-05-31T21:12:20.147 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 21:11:49 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000cdc7915a9b14b739-006477b834-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 21:12:20 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:12:20.148 DEBUG:boto:path=/pklxkd-64/ 2023-05-31T21:12:20.149 DEBUG:boto:auth_path=/pklxkd-64/ 2023-05-31T21:12:20.149 DEBUG:boto:path=/pklxkd-64/?prefix=obj44 2023-05-31T21:12:20.150 DEBUG:boto:auth_path=/pklxkd-64/?prefix=obj44 2023-05-31T21:12:20.150 DEBUG:boto:Method: GET 2023-05-31T21:12:20.151 DEBUG:boto:Path: /pklxkd-64/?prefix=obj44 2023-05-31T21:12:20.151 DEBUG:boto:Data: 2023-05-31T21:12:20.152 DEBUG:boto:Headers: {} 2023-05-31T21:12:20.153 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T21:12:20.153 DEBUG:boto:Port: 8001 2023-05-31T21:12:20.154 DEBUG:boto:Params: {} 2023-05-31T21:12:20.154 DEBUG:boto:Token: None 2023-05-31T21:12:20.155 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 21:12:20 GMT /pklxkd-64/ 2023-05-31T21:12:20.156 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:eDdIGyOBSPGk4dT06OEG2smpfgM= 2023-05-31T21:12:20.156 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 21:12:20 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:eDdIGyOBSPGk4dT06OEG2smpfgM=', 'Content-Length': '0'} 2023-05-31T21:12:20.158 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b08e34544de52037-006477b834-1fb6-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 21:12:20 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:12:20.159 DEBUG:boto:b'pklxkd-64obj441000falseobj442023-05-31T21:11:49.126Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T21:12:20.159 DEBUG:boto:path=/pklxkd-64/ 2023-05-31T21:12:20.160 DEBUG:boto:auth_path=/pklxkd-64/ 2023-05-31T21:12:20.160 DEBUG:boto:path=/pklxkd-64/?prefix=obj44 2023-05-31T21:12:20.161 DEBUG:boto:auth_path=/pklxkd-64/?prefix=obj44 2023-05-31T21:12:20.161 DEBUG:boto:Method: GET 2023-05-31T21:12:20.161 DEBUG:boto:Path: /pklxkd-64/?prefix=obj44 2023-05-31T21:12:20.162 DEBUG:boto:Data: 2023-05-31T21:12:20.162 DEBUG:boto:Headers: {} 2023-05-31T21:12:20.162 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T21:12:20.163 DEBUG:boto:Port: 8000 2023-05-31T21:12:20.163 DEBUG:boto:Params: {} 2023-05-31T21:12:20.163 DEBUG:boto:Token: None 2023-05-31T21:12:20.164 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 21:12:20 GMT /pklxkd-64/ 2023-05-31T21:12:20.164 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:eDdIGyOBSPGk4dT06OEG2smpfgM= 2023-05-31T21:12:20.165 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 21:12:20 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:eDdIGyOBSPGk4dT06OEG2smpfgM=', 'Content-Length': '0'} 2023-05-31T21:12:20.167 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000006fc5c4a082cab2c9-006477b834-1630-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 21:12:20 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:12:20.168 DEBUG:boto:b'pklxkd-64obj441000falseobj442023-05-31T21:11:49.126Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T21:12:20.168 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj55 2023-05-31T21:12:20.169 DEBUG:boto:path=/pklxkd-64/obj55 2023-05-31T21:12:20.169 DEBUG:boto:auth_path=/pklxkd-64/obj55 2023-05-31T21:12:20.169 DEBUG:boto:path=/pklxkd-64/obj55?versionId=null 2023-05-31T21:12:20.170 DEBUG:boto:auth_path=/pklxkd-64/obj55?versionId=null 2023-05-31T21:12:20.170 DEBUG:boto:Method: GET 2023-05-31T21:12:20.170 DEBUG:boto:Path: /pklxkd-64/obj55?versionId=null 2023-05-31T21:12:20.171 DEBUG:boto:Data: 2023-05-31T21:12:20.171 DEBUG:boto:Headers: {} 2023-05-31T21:12:20.172 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T21:12:20.172 DEBUG:boto:Port: 8001 2023-05-31T21:12:20.172 DEBUG:boto:Params: {} 2023-05-31T21:12:20.173 DEBUG:boto:Token: None 2023-05-31T21:12:20.173 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 21:12:20 GMT /pklxkd-64/obj55?versionId=null 2023-05-31T21:12:20.174 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:9yjL+gXeHFdgrQ4CunoYeEHYQFg= 2023-05-31T21:12:20.174 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 21:12:20 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:9yjL+gXeHFdgrQ4CunoYeEHYQFg=', 'Content-Length': '0'} 2023-05-31T21:12:20.176 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 21:11:49 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000923ad71e61192bda-006477b834-1fb6-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 21:12:20 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:12:20.177 DEBUG:boto:path=/pklxkd-64/obj55 2023-05-31T21:12:20.177 DEBUG:boto:auth_path=/pklxkd-64/obj55 2023-05-31T21:12:20.177 DEBUG:boto:path=/pklxkd-64/obj55?versionId=null 2023-05-31T21:12:20.178 DEBUG:boto:auth_path=/pklxkd-64/obj55?versionId=null 2023-05-31T21:12:20.178 DEBUG:boto:Method: GET 2023-05-31T21:12:20.178 DEBUG:boto:Path: /pklxkd-64/obj55?versionId=null 2023-05-31T21:12:20.179 DEBUG:boto:Data: 2023-05-31T21:12:20.179 DEBUG:boto:Headers: {} 2023-05-31T21:12:20.179 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T21:12:20.180 DEBUG:boto:Port: 8000 2023-05-31T21:12:20.180 DEBUG:boto:Params: {} 2023-05-31T21:12:20.181 DEBUG:boto:Token: None 2023-05-31T21:12:20.181 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 21:12:20 GMT /pklxkd-64/obj55?versionId=null 2023-05-31T21:12:20.181 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:9yjL+gXeHFdgrQ4CunoYeEHYQFg= 2023-05-31T21:12:20.182 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 21:12:20 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:9yjL+gXeHFdgrQ4CunoYeEHYQFg=', 'Content-Length': '0'} 2023-05-31T21:12:20.184 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 21:11:49 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000aaf54489ecfb2a70-006477b834-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 21:12:20 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:12:20.184 DEBUG:boto:path=/pklxkd-64/obj55 2023-05-31T21:12:20.185 DEBUG:boto:auth_path=/pklxkd-64/obj55 2023-05-31T21:12:20.185 DEBUG:boto:path=/pklxkd-64/obj55?versionId=null 2023-05-31T21:12:20.185 DEBUG:boto:auth_path=/pklxkd-64/obj55?versionId=null 2023-05-31T21:12:20.186 DEBUG:boto:Method: HEAD 2023-05-31T21:12:20.186 DEBUG:boto:Path: /pklxkd-64/obj55?versionId=null 2023-05-31T21:12:20.186 DEBUG:boto:Data: 2023-05-31T21:12:20.187 DEBUG:boto:Headers: {} 2023-05-31T21:12:20.187 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T21:12:20.188 DEBUG:boto:Port: 8001 2023-05-31T21:12:20.188 DEBUG:boto:Params: {} 2023-05-31T21:12:20.189 DEBUG:boto:Token: None 2023-05-31T21:12:20.190 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 21:12:20 GMT /pklxkd-64/obj55?versionId=null 2023-05-31T21:12:20.190 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:l+HBRB7ah6vPUB+HRib6vkbdDZU= 2023-05-31T21:12:20.190 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 21:12:20 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:l+HBRB7ah6vPUB+HRib6vkbdDZU=', 'Content-Length': '0'} 2023-05-31T21:12:20.192 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 21:11:49 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002addf74279b12ffd-006477b834-1fb6-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 21:12:20 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:12:20.193 DEBUG:boto:path=/pklxkd-64/obj55 2023-05-31T21:12:20.193 DEBUG:boto:auth_path=/pklxkd-64/obj55 2023-05-31T21:12:20.193 DEBUG:boto:path=/pklxkd-64/obj55?versionId=null 2023-05-31T21:12:20.194 DEBUG:boto:auth_path=/pklxkd-64/obj55?versionId=null 2023-05-31T21:12:20.194 DEBUG:boto:Method: HEAD 2023-05-31T21:12:20.194 DEBUG:boto:Path: /pklxkd-64/obj55?versionId=null 2023-05-31T21:12:20.195 DEBUG:boto:Data: 2023-05-31T21:12:20.195 DEBUG:boto:Headers: {} 2023-05-31T21:12:20.195 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T21:12:20.196 DEBUG:boto:Port: 8000 2023-05-31T21:12:20.196 DEBUG:boto:Params: {} 2023-05-31T21:12:20.196 DEBUG:boto:Token: None 2023-05-31T21:12:20.197 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 21:12:20 GMT /pklxkd-64/obj55?versionId=null 2023-05-31T21:12:20.197 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:l+HBRB7ah6vPUB+HRib6vkbdDZU= 2023-05-31T21:12:20.197 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 21:12:20 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:l+HBRB7ah6vPUB+HRib6vkbdDZU=', 'Content-Length': '0'} 2023-05-31T21:12:20.199 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 21:11:49 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000080068e285ca5de6c-006477b834-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 21:12:20 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:12:20.200 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj55 2023-05-31T21:12:20.201 DEBUG:boto:path=/pklxkd-64/obj55 2023-05-31T21:12:20.201 DEBUG:boto:auth_path=/pklxkd-64/obj55 2023-05-31T21:12:20.202 DEBUG:boto:path=/pklxkd-64/obj55?versionId=null 2023-05-31T21:12:20.202 DEBUG:boto:auth_path=/pklxkd-64/obj55?versionId=null 2023-05-31T21:12:20.202 DEBUG:boto:Method: GET 2023-05-31T21:12:20.203 DEBUG:boto:Path: /pklxkd-64/obj55?versionId=null 2023-05-31T21:12:20.203 DEBUG:boto:Data: 2023-05-31T21:12:20.203 DEBUG:boto:Headers: {} 2023-05-31T21:12:20.204 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T21:12:20.204 DEBUG:boto:Port: 8001 2023-05-31T21:12:20.205 DEBUG:boto:Params: {} 2023-05-31T21:12:20.205 DEBUG:boto:Token: None 2023-05-31T21:12:20.205 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 21:12:20 GMT /pklxkd-64/obj55?versionId=null 2023-05-31T21:12:20.206 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:9yjL+gXeHFdgrQ4CunoYeEHYQFg= 2023-05-31T21:12:20.206 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 21:12:20 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:9yjL+gXeHFdgrQ4CunoYeEHYQFg=', 'Content-Length': '0'} 2023-05-31T21:12:20.208 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 21:11:49 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b6c0fa901402e645-006477b834-1fb6-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 21:12:20 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:12:20.209 DEBUG:boto:path=/pklxkd-64/obj55 2023-05-31T21:12:20.209 DEBUG:boto:auth_path=/pklxkd-64/obj55 2023-05-31T21:12:20.210 DEBUG:boto:path=/pklxkd-64/obj55?versionId=null 2023-05-31T21:12:20.210 DEBUG:boto:auth_path=/pklxkd-64/obj55?versionId=null 2023-05-31T21:12:20.211 DEBUG:boto:Method: GET 2023-05-31T21:12:20.211 DEBUG:boto:Path: /pklxkd-64/obj55?versionId=null 2023-05-31T21:12:20.211 DEBUG:boto:Data: 2023-05-31T21:12:20.212 DEBUG:boto:Headers: {} 2023-05-31T21:12:20.212 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T21:12:20.212 DEBUG:boto:Port: 8000 2023-05-31T21:12:20.213 DEBUG:boto:Params: {} 2023-05-31T21:12:20.213 DEBUG:boto:Token: None 2023-05-31T21:12:20.214 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 21:12:20 GMT /pklxkd-64/obj55?versionId=null 2023-05-31T21:12:20.214 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:9yjL+gXeHFdgrQ4CunoYeEHYQFg= 2023-05-31T21:12:20.214 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 21:12:20 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:9yjL+gXeHFdgrQ4CunoYeEHYQFg=', 'Content-Length': '0'} 2023-05-31T21:12:20.216 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 21:11:49 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002bb73188614ddb18-006477b834-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 21:12:20 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:12:20.217 DEBUG:boto:path=/pklxkd-64/ 2023-05-31T21:12:20.217 DEBUG:boto:auth_path=/pklxkd-64/ 2023-05-31T21:12:20.218 DEBUG:boto:path=/pklxkd-64/?prefix=obj55 2023-05-31T21:12:20.218 DEBUG:boto:auth_path=/pklxkd-64/?prefix=obj55 2023-05-31T21:12:20.219 DEBUG:boto:Method: GET 2023-05-31T21:12:20.219 DEBUG:boto:Path: /pklxkd-64/?prefix=obj55 2023-05-31T21:12:20.220 DEBUG:boto:Data: 2023-05-31T21:12:20.220 DEBUG:boto:Headers: {} 2023-05-31T21:12:20.220 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T21:12:20.221 DEBUG:boto:Port: 8001 2023-05-31T21:12:20.221 DEBUG:boto:Params: {} 2023-05-31T21:12:20.222 DEBUG:boto:Token: None 2023-05-31T21:12:20.222 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 21:12:20 GMT /pklxkd-64/ 2023-05-31T21:12:20.222 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:eDdIGyOBSPGk4dT06OEG2smpfgM= 2023-05-31T21:12:20.223 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 21:12:20 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:eDdIGyOBSPGk4dT06OEG2smpfgM=', 'Content-Length': '0'} 2023-05-31T21:12:20.225 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000000bbe93386a6cc587-006477b834-1fb6-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 21:12:20 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:12:20.268 DEBUG:boto:b'pklxkd-64obj551000falseobj552023-05-31T21:11:49.145Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T21:12:20.269 DEBUG:boto:path=/pklxkd-64/ 2023-05-31T21:12:20.269 DEBUG:boto:auth_path=/pklxkd-64/ 2023-05-31T21:12:20.270 DEBUG:boto:path=/pklxkd-64/?prefix=obj55 2023-05-31T21:12:20.270 DEBUG:boto:auth_path=/pklxkd-64/?prefix=obj55 2023-05-31T21:12:20.270 DEBUG:boto:Method: GET 2023-05-31T21:12:20.271 DEBUG:boto:Path: /pklxkd-64/?prefix=obj55 2023-05-31T21:12:20.271 DEBUG:boto:Data: 2023-05-31T21:12:20.271 DEBUG:boto:Headers: {} 2023-05-31T21:12:20.272 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T21:12:20.272 DEBUG:boto:Port: 8000 2023-05-31T21:12:20.272 DEBUG:boto:Params: {} 2023-05-31T21:12:20.273 DEBUG:boto:Token: None 2023-05-31T21:12:20.273 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 21:12:20 GMT /pklxkd-64/ 2023-05-31T21:12:20.274 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:eDdIGyOBSPGk4dT06OEG2smpfgM= 2023-05-31T21:12:20.274 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 21:12:20 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:eDdIGyOBSPGk4dT06OEG2smpfgM=', 'Content-Length': '0'} 2023-05-31T21:12:20.277 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000feca934e4ff397ad-006477b834-1630-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 21:12:20 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:12:20.277 DEBUG:boto:b'pklxkd-64obj551000falseobj552023-05-31T21:11:49.145Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T21:12:20.278 DEBUG:boto:path=/pklxkd-64/obj55 2023-05-31T21:12:20.278 DEBUG:boto:auth_path=/pklxkd-64/obj55 2023-05-31T21:12:20.279 DEBUG:boto:Method: HEAD 2023-05-31T21:12:20.279 DEBUG:boto:Path: /pklxkd-64/obj55 2023-05-31T21:12:20.279 DEBUG:boto:Data: 2023-05-31T21:12:20.280 DEBUG:boto:Headers: {} 2023-05-31T21:12:20.280 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T21:12:20.280 DEBUG:boto:Port: 8001 2023-05-31T21:12:20.281 DEBUG:boto:Params: {} 2023-05-31T21:12:20.281 DEBUG:boto:Token: None 2023-05-31T21:12:20.282 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 21:12:20 GMT /pklxkd-64/obj55 2023-05-31T21:12:20.282 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:fTtLSyxSzeH/WJVZLufS0POd02k= 2023-05-31T21:12:20.282 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 21:12:20 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:fTtLSyxSzeH/WJVZLufS0POd02k=', 'Content-Length': '0'} 2023-05-31T21:12:20.284 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 21:11:49 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000360849725bfb8d51-006477b834-1fb6-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 21:12:20 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:12:20.284 DEBUG:boto:path=/pklxkd-64/obj55 2023-05-31T21:12:20.285 DEBUG:boto:auth_path=/pklxkd-64/obj55 2023-05-31T21:12:20.285 DEBUG:boto:Method: HEAD 2023-05-31T21:12:20.286 DEBUG:boto:Path: /pklxkd-64/obj55 2023-05-31T21:12:20.286 DEBUG:boto:Data: 2023-05-31T21:12:20.286 DEBUG:boto:Headers: {} 2023-05-31T21:12:20.287 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T21:12:20.287 DEBUG:boto:Port: 8000 2023-05-31T21:12:20.287 DEBUG:boto:Params: {} 2023-05-31T21:12:20.288 DEBUG:boto:Token: None 2023-05-31T21:12:20.288 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 21:12:20 GMT /pklxkd-64/obj55 2023-05-31T21:12:20.288 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:fTtLSyxSzeH/WJVZLufS0POd02k= 2023-05-31T21:12:20.289 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 21:12:20 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:fTtLSyxSzeH/WJVZLufS0POd02k=', 'Content-Length': '0'} 2023-05-31T21:12:20.290 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 21:11:49 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f81aaf3a5f656709-006477b834-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 21:12:20 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:12:20.291 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj55 2023-05-31T21:12:20.291 DEBUG:boto:path=/pklxkd-64/obj55 2023-05-31T21:12:20.292 DEBUG:boto:auth_path=/pklxkd-64/obj55 2023-05-31T21:12:20.292 DEBUG:boto:Method: GET 2023-05-31T21:12:20.293 DEBUG:boto:Path: /pklxkd-64/obj55 2023-05-31T21:12:20.293 DEBUG:boto:Data: 2023-05-31T21:12:20.293 DEBUG:boto:Headers: {} 2023-05-31T21:12:20.294 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T21:12:20.294 DEBUG:boto:Port: 8001 2023-05-31T21:12:20.294 DEBUG:boto:Params: {} 2023-05-31T21:12:20.295 DEBUG:boto:Token: None 2023-05-31T21:12:20.295 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 21:12:20 GMT /pklxkd-64/obj55 2023-05-31T21:12:20.295 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:Zdlk1/S3LaNKscA8PK/PDYPMsZk= 2023-05-31T21:12:20.296 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 21:12:20 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:Zdlk1/S3LaNKscA8PK/PDYPMsZk=', 'Content-Length': '0'} 2023-05-31T21:12:20.298 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 21:11:49 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004769432281c95f51-006477b834-1fb6-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 21:12:20 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:12:20.298 DEBUG:boto:path=/pklxkd-64/obj55 2023-05-31T21:12:20.299 DEBUG:boto:auth_path=/pklxkd-64/obj55 2023-05-31T21:12:20.299 DEBUG:boto:Method: GET 2023-05-31T21:12:20.300 DEBUG:boto:Path: /pklxkd-64/obj55 2023-05-31T21:12:20.300 DEBUG:boto:Data: 2023-05-31T21:12:20.300 DEBUG:boto:Headers: {} 2023-05-31T21:12:20.301 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T21:12:20.301 DEBUG:boto:Port: 8000 2023-05-31T21:12:20.301 DEBUG:boto:Params: {} 2023-05-31T21:12:20.302 DEBUG:boto:Token: None 2023-05-31T21:12:20.302 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 21:12:20 GMT /pklxkd-64/obj55 2023-05-31T21:12:20.303 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:Zdlk1/S3LaNKscA8PK/PDYPMsZk= 2023-05-31T21:12:20.303 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 21:12:20 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:Zdlk1/S3LaNKscA8PK/PDYPMsZk=', 'Content-Length': '0'} 2023-05-31T21:12:20.305 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 21:11:49 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c58b03dabd2b47c6-006477b834-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 21:12:20 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:12:20.348 DEBUG:boto:path=/pklxkd-64/ 2023-05-31T21:12:20.349 DEBUG:boto:auth_path=/pklxkd-64/ 2023-05-31T21:12:20.349 DEBUG:boto:path=/pklxkd-64/?prefix=obj55 2023-05-31T21:12:20.349 DEBUG:boto:auth_path=/pklxkd-64/?prefix=obj55 2023-05-31T21:12:20.350 DEBUG:boto:Method: GET 2023-05-31T21:12:20.350 DEBUG:boto:Path: /pklxkd-64/?prefix=obj55 2023-05-31T21:12:20.350 DEBUG:boto:Data: 2023-05-31T21:12:20.351 DEBUG:boto:Headers: {} 2023-05-31T21:12:20.351 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T21:12:20.352 DEBUG:boto:Port: 8001 2023-05-31T21:12:20.352 DEBUG:boto:Params: {} 2023-05-31T21:12:20.352 DEBUG:boto:Token: None 2023-05-31T21:12:20.353 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 21:12:20 GMT /pklxkd-64/ 2023-05-31T21:12:20.353 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:eDdIGyOBSPGk4dT06OEG2smpfgM= 2023-05-31T21:12:20.354 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 21:12:20 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:eDdIGyOBSPGk4dT06OEG2smpfgM=', 'Content-Length': '0'} 2023-05-31T21:12:20.356 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000971df19e014e1838-006477b834-1fb6-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 21:12:20 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:12:20.357 DEBUG:boto:b'pklxkd-64obj551000falseobj552023-05-31T21:11:49.145Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T21:12:20.357 DEBUG:boto:path=/pklxkd-64/ 2023-05-31T21:12:20.358 DEBUG:boto:auth_path=/pklxkd-64/ 2023-05-31T21:12:20.358 DEBUG:boto:path=/pklxkd-64/?prefix=obj55 2023-05-31T21:12:20.358 DEBUG:boto:auth_path=/pklxkd-64/?prefix=obj55 2023-05-31T21:12:20.359 DEBUG:boto:Method: GET 2023-05-31T21:12:20.359 DEBUG:boto:Path: /pklxkd-64/?prefix=obj55 2023-05-31T21:12:20.359 DEBUG:boto:Data: 2023-05-31T21:12:20.360 DEBUG:boto:Headers: {} 2023-05-31T21:12:20.360 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T21:12:20.361 DEBUG:boto:Port: 8000 2023-05-31T21:12:20.361 DEBUG:boto:Params: {} 2023-05-31T21:12:20.361 DEBUG:boto:Token: None 2023-05-31T21:12:20.362 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 21:12:20 GMT /pklxkd-64/ 2023-05-31T21:12:20.362 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:eDdIGyOBSPGk4dT06OEG2smpfgM= 2023-05-31T21:12:20.363 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 21:12:20 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:eDdIGyOBSPGk4dT06OEG2smpfgM=', 'Content-Length': '0'} 2023-05-31T21:12:20.365 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000085ae51b5583591fb-006477b834-1630-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 21:12:20 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:12:20.366 DEBUG:boto:b'pklxkd-64obj551000falseobj552023-05-31T21:11:49.145Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T21:12:20.366 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=pklxkd-64 zones={a2, a1} 2023-05-31T21:12:20.367 DEBUG:tasks.rgw.c1.client.1:waiting for process to exit 2023-05-31T21:12:20.367 INFO:teuthology.orchestra.run:waiting for 300 2023-05-31T21:12:20.368 INFO:tasks.rgw.c1.client.1.smithi099.stdout:2023-05-31T21:12:20.364+0000 7f33ee678700 -1 received signal: Terminated from /usr/bin/python3 /bin/daemon-helper term radosgw --rgw-frontends beast port=8001 -n client.1 --cluster c1 -k /etc/ceph/c1.client.1.keyring --log-file /var/log/ceph/rgw.c1.client.1.log --rgw_ops_log_socket_path /home/ubuntu/cephtest/rgw.opslog.c1.client.1.sock --foreground --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm (PID: 169194) UID: 0 2023-05-31T21:12:20.368 INFO:tasks.rgw.c1.client.1.smithi099.stdout:2023-05-31T21:12:20.364+0000 7f33f28e17c0 -1 shutting down 2023-05-31T21:12:24.689 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.1 is failed for ~0s 2023-05-31T21:12:26.466 INFO:tasks.rgw.c1.client.1:Stopped 2023-05-31T21:12:26.467 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'reshard', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'pklxkd-64', '--num-shards', '17', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-05-31T21:12:26.467 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'reshard', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'pklxkd-64', '--num-shards', '17', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-05-31T21:12:26.468 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket reshard --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --bucket pklxkd-64 --num-shards 17 --yes-i-really-mean-it --cluster c1 --debug-rgw 0 --debug-ms 0 2023-05-31T21:12:26.528 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2023-05-31T21:12:26.528 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2023-05-31T21:12:26.590 INFO:teuthology.orchestra.run.smithi099.stdout:tenant: 2023-05-31T21:12:26.591 INFO:teuthology.orchestra.run.smithi099.stdout:bucket name: pklxkd-64 2023-05-31T21:12:26.594 INFO:teuthology.orchestra.run.smithi099.stdout:total entries: 5 2023-05-31T21:12:26.657 DEBUG:boto:path=/pklxkd-64/ 2023-05-31T21:12:26.657 DEBUG:boto:auth_path=/pklxkd-64/ 2023-05-31T21:12:26.658 DEBUG:boto:Method: HEAD 2023-05-31T21:12:26.658 DEBUG:boto:Path: /pklxkd-64/ 2023-05-31T21:12:26.658 DEBUG:boto:Data: 2023-05-31T21:12:26.659 DEBUG:boto:Headers: {} 2023-05-31T21:12:26.659 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T21:12:26.659 DEBUG:boto:Port: 8000 2023-05-31T21:12:26.660 DEBUG:boto:Params: {} 2023-05-31T21:12:26.660 DEBUG:boto:Token: None 2023-05-31T21:12:26.661 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 21:12:26 GMT /pklxkd-64/ 2023-05-31T21:12:26.661 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:hP1uXxRwqPN9evN7pLR4dhogLN8= 2023-05-31T21:12:26.661 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 21:12:26 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:hP1uXxRwqPN9evN7pLR4dhogLN8=', 'Content-Length': '0'} 2023-05-31T21:12:26.664 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '5'), ('X-RGW-Bytes-Used', '15'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000fed9c66a1b7fb1d5-006477b83a-1630-a1'), ('Content-Length', '0'), ('Date', 'Wed, 31 May 2023 21:12:26 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:12:26.665 DEBUG:boto:path=/pklxkd-64/obj17 2023-05-31T21:12:26.665 DEBUG:boto:auth_path=/pklxkd-64/obj17 2023-05-31T21:12:26.665 DEBUG:boto:Method: PUT 2023-05-31T21:12:26.666 DEBUG:boto:Path: /pklxkd-64/obj17 2023-05-31T21:12:26.666 DEBUG:boto:Data: 2023-05-31T21:12:26.667 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-05-31T21:12:26.667 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T21:12:26.667 DEBUG:boto:Port: 8000 2023-05-31T21:12:26.668 DEBUG:boto:Params: {} 2023-05-31T21:12:26.668 DEBUG:boto:Token: None 2023-05-31T21:12:26.669 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Wed, 31 May 2023 21:12:26 GMT /pklxkd-64/obj17 2023-05-31T21:12:26.669 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:q9jUMGlyDGU2cYgC/i0bSOx9c0I= 2023-05-31T21:12:26.670 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Wed, 31 May 2023 21:12:26 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:q9jUMGlyDGU2cYgC/i0bSOx9c0I='} 2023-05-31T21:12:26.675 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000c52b5be32e911359-006477b83a-1630-a1'), ('Date', 'Wed, 31 May 2023 21:12:26 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:12:26.676 DEBUG:boto:path=/pklxkd-64/ 2023-05-31T21:12:26.676 DEBUG:boto:auth_path=/pklxkd-64/ 2023-05-31T21:12:26.676 DEBUG:boto:Method: HEAD 2023-05-31T21:12:26.677 DEBUG:boto:Path: /pklxkd-64/ 2023-05-31T21:12:26.677 DEBUG:boto:Data: 2023-05-31T21:12:26.677 DEBUG:boto:Headers: {} 2023-05-31T21:12:26.678 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T21:12:26.678 DEBUG:boto:Port: 8000 2023-05-31T21:12:26.678 DEBUG:boto:Params: {} 2023-05-31T21:12:26.679 DEBUG:boto:Token: None 2023-05-31T21:12:26.679 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 21:12:26 GMT /pklxkd-64/ 2023-05-31T21:12:26.680 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:hP1uXxRwqPN9evN7pLR4dhogLN8= 2023-05-31T21:12:26.680 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 21:12:26 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:hP1uXxRwqPN9evN7pLR4dhogLN8=', 'Content-Length': '0'} 2023-05-31T21:12:26.682 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '6'), ('X-RGW-Bytes-Used', '18'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000091128a856c40bb70-006477b83a-1630-a1'), ('Content-Length', '0'), ('Date', 'Wed, 31 May 2023 21:12:26 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:12:26.683 DEBUG:boto:path=/pklxkd-64/obj34 2023-05-31T21:12:26.683 DEBUG:boto:auth_path=/pklxkd-64/obj34 2023-05-31T21:12:26.683 DEBUG:boto:Method: PUT 2023-05-31T21:12:26.684 DEBUG:boto:Path: /pklxkd-64/obj34 2023-05-31T21:12:26.684 DEBUG:boto:Data: 2023-05-31T21:12:26.684 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-05-31T21:12:26.685 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T21:12:26.685 DEBUG:boto:Port: 8000 2023-05-31T21:12:26.685 DEBUG:boto:Params: {} 2023-05-31T21:12:26.686 DEBUG:boto:Token: None 2023-05-31T21:12:26.686 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Wed, 31 May 2023 21:12:26 GMT /pklxkd-64/obj34 2023-05-31T21:12:26.687 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:YL1eSekfJ5uppmX6M09RK1igy7M= 2023-05-31T21:12:26.687 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Wed, 31 May 2023 21:12:26 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:YL1eSekfJ5uppmX6M09RK1igy7M='} 2023-05-31T21:12:26.695 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000476c64a49cd694e8-006477b83a-1630-a1'), ('Date', 'Wed, 31 May 2023 21:12:26 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:12:26.695 DEBUG:boto:path=/pklxkd-64/ 2023-05-31T21:12:26.695 DEBUG:boto:auth_path=/pklxkd-64/ 2023-05-31T21:12:26.696 DEBUG:boto:Method: HEAD 2023-05-31T21:12:26.696 DEBUG:boto:Path: /pklxkd-64/ 2023-05-31T21:12:26.696 DEBUG:boto:Data: 2023-05-31T21:12:26.697 DEBUG:boto:Headers: {} 2023-05-31T21:12:26.697 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T21:12:26.697 DEBUG:boto:Port: 8000 2023-05-31T21:12:26.698 DEBUG:boto:Params: {} 2023-05-31T21:12:26.698 DEBUG:boto:Token: None 2023-05-31T21:12:26.699 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 21:12:26 GMT /pklxkd-64/ 2023-05-31T21:12:26.699 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:hP1uXxRwqPN9evN7pLR4dhogLN8= 2023-05-31T21:12:26.699 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 21:12:26 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:hP1uXxRwqPN9evN7pLR4dhogLN8=', 'Content-Length': '0'} 2023-05-31T21:12:26.701 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '7'), ('X-RGW-Bytes-Used', '21'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000004b21c357feb9d319-006477b83a-1630-a1'), ('Content-Length', '0'), ('Date', 'Wed, 31 May 2023 21:12:26 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:12:26.702 DEBUG:boto:path=/pklxkd-64/obj51 2023-05-31T21:12:26.702 DEBUG:boto:auth_path=/pklxkd-64/obj51 2023-05-31T21:12:26.703 DEBUG:boto:Method: PUT 2023-05-31T21:12:26.703 DEBUG:boto:Path: /pklxkd-64/obj51 2023-05-31T21:12:26.703 DEBUG:boto:Data: 2023-05-31T21:12:26.704 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-05-31T21:12:26.704 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T21:12:26.704 DEBUG:boto:Port: 8000 2023-05-31T21:12:26.705 DEBUG:boto:Params: {} 2023-05-31T21:12:26.705 DEBUG:boto:Token: None 2023-05-31T21:12:26.705 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Wed, 31 May 2023 21:12:26 GMT /pklxkd-64/obj51 2023-05-31T21:12:26.706 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:rEjT25Nwzn3abcaOepFslarWnWY= 2023-05-31T21:12:26.706 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Wed, 31 May 2023 21:12:26 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:rEjT25Nwzn3abcaOepFslarWnWY='} 2023-05-31T21:12:26.748 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000858170e1d4305a2a-006477b83a-1630-a1'), ('Date', 'Wed, 31 May 2023 21:12:26 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:12:26.749 DEBUG:boto:path=/pklxkd-64/ 2023-05-31T21:12:26.749 DEBUG:boto:auth_path=/pklxkd-64/ 2023-05-31T21:12:26.749 DEBUG:boto:Method: HEAD 2023-05-31T21:12:26.750 DEBUG:boto:Path: /pklxkd-64/ 2023-05-31T21:12:26.750 DEBUG:boto:Data: 2023-05-31T21:12:26.750 DEBUG:boto:Headers: {} 2023-05-31T21:12:26.751 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T21:12:26.751 DEBUG:boto:Port: 8000 2023-05-31T21:12:26.751 DEBUG:boto:Params: {} 2023-05-31T21:12:26.752 DEBUG:boto:Token: None 2023-05-31T21:12:26.752 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 21:12:26 GMT /pklxkd-64/ 2023-05-31T21:12:26.753 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:hP1uXxRwqPN9evN7pLR4dhogLN8= 2023-05-31T21:12:26.753 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 21:12:26 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:hP1uXxRwqPN9evN7pLR4dhogLN8=', 'Content-Length': '0'} 2023-05-31T21:12:26.756 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '8'), ('X-RGW-Bytes-Used', '24'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000dca3325349f0b971-006477b83a-1630-a1'), ('Content-Length', '0'), ('Date', 'Wed, 31 May 2023 21:12:26 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:12:26.756 DEBUG:boto:path=/pklxkd-64/obj68 2023-05-31T21:12:26.757 DEBUG:boto:auth_path=/pklxkd-64/obj68 2023-05-31T21:12:26.757 DEBUG:boto:Method: PUT 2023-05-31T21:12:26.757 DEBUG:boto:Path: /pklxkd-64/obj68 2023-05-31T21:12:26.758 DEBUG:boto:Data: 2023-05-31T21:12:26.758 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-05-31T21:12:26.758 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T21:12:26.759 DEBUG:boto:Port: 8000 2023-05-31T21:12:26.759 DEBUG:boto:Params: {} 2023-05-31T21:12:26.759 DEBUG:boto:Token: None 2023-05-31T21:12:26.760 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Wed, 31 May 2023 21:12:26 GMT /pklxkd-64/obj68 2023-05-31T21:12:26.760 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:RsEodSXQsrVbxryD3yyUqPyQLvU= 2023-05-31T21:12:26.760 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Wed, 31 May 2023 21:12:26 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:RsEodSXQsrVbxryD3yyUqPyQLvU='} 2023-05-31T21:12:26.769 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000037c652e39bbf1cc4-006477b83a-1630-a1'), ('Date', 'Wed, 31 May 2023 21:12:26 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:12:26.770 DEBUG:boto:path=/pklxkd-64/ 2023-05-31T21:12:26.770 DEBUG:boto:auth_path=/pklxkd-64/ 2023-05-31T21:12:26.771 DEBUG:boto:Method: HEAD 2023-05-31T21:12:26.771 DEBUG:boto:Path: /pklxkd-64/ 2023-05-31T21:12:26.771 DEBUG:boto:Data: 2023-05-31T21:12:26.772 DEBUG:boto:Headers: {} 2023-05-31T21:12:26.772 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T21:12:26.772 DEBUG:boto:Port: 8000 2023-05-31T21:12:26.773 DEBUG:boto:Params: {} 2023-05-31T21:12:26.773 DEBUG:boto:Token: None 2023-05-31T21:12:26.773 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 21:12:26 GMT /pklxkd-64/ 2023-05-31T21:12:26.774 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:hP1uXxRwqPN9evN7pLR4dhogLN8= 2023-05-31T21:12:26.774 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 21:12:26 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:hP1uXxRwqPN9evN7pLR4dhogLN8=', 'Content-Length': '0'} 2023-05-31T21:12:26.776 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '9'), ('X-RGW-Bytes-Used', '27'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000835a50cb2f758df4-006477b83a-1630-a1'), ('Content-Length', '0'), ('Date', 'Wed, 31 May 2023 21:12:26 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:12:26.777 DEBUG:boto:path=/pklxkd-64/obj85 2023-05-31T21:12:26.777 DEBUG:boto:auth_path=/pklxkd-64/obj85 2023-05-31T21:12:26.778 DEBUG:boto:Method: PUT 2023-05-31T21:12:26.778 DEBUG:boto:Path: /pklxkd-64/obj85 2023-05-31T21:12:26.778 DEBUG:boto:Data: 2023-05-31T21:12:26.779 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-05-31T21:12:26.779 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T21:12:26.779 DEBUG:boto:Port: 8000 2023-05-31T21:12:26.780 DEBUG:boto:Params: {} 2023-05-31T21:12:26.780 DEBUG:boto:Token: None 2023-05-31T21:12:26.781 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Wed, 31 May 2023 21:12:26 GMT /pklxkd-64/obj85 2023-05-31T21:12:26.781 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:keSx+iMcDOy4YwcK4PcsPrFnLiY= 2023-05-31T21:12:26.781 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Wed, 31 May 2023 21:12:26 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:keSx+iMcDOy4YwcK4PcsPrFnLiY='} 2023-05-31T21:12:26.825 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000047c73a5bb275012b-006477b83a-1630-a1'), ('Date', 'Wed, 31 May 2023 21:12:26 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:12:26.826 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'reshard', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'pklxkd-64', '--num-shards', '19', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-05-31T21:12:26.826 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'reshard', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'pklxkd-64', '--num-shards', '19', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-05-31T21:12:26.827 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket reshard --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --bucket pklxkd-64 --num-shards 19 --yes-i-really-mean-it --cluster c1 --debug-rgw 0 --debug-ms 0 2023-05-31T21:12:26.880 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2023-05-31T21:12:26.881 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2023-05-31T21:12:26.937 INFO:teuthology.orchestra.run.smithi099.stdout:tenant: 2023-05-31T21:12:26.938 INFO:teuthology.orchestra.run.smithi099.stdout:bucket name: pklxkd-64 2023-05-31T21:12:26.943 INFO:teuthology.orchestra.run.smithi099.stdout:total entries: 10 2023-05-31T21:12:27.017 DEBUG:boto:path=/pklxkd-64/ 2023-05-31T21:12:27.017 DEBUG:boto:auth_path=/pklxkd-64/ 2023-05-31T21:12:27.017 DEBUG:boto:Method: HEAD 2023-05-31T21:12:27.018 DEBUG:boto:Path: /pklxkd-64/ 2023-05-31T21:12:27.018 DEBUG:boto:Data: 2023-05-31T21:12:27.019 DEBUG:boto:Headers: {} 2023-05-31T21:12:27.019 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T21:12:27.020 DEBUG:boto:Port: 8000 2023-05-31T21:12:27.020 DEBUG:boto:Params: {} 2023-05-31T21:12:27.020 DEBUG:boto:Token: None 2023-05-31T21:12:27.021 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 21:12:27 GMT /pklxkd-64/ 2023-05-31T21:12:27.021 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:5X+CZRYkJIw4yCbpLG+ZuT3v7Tg= 2023-05-31T21:12:27.022 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 21:12:27 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:5X+CZRYkJIw4yCbpLG+ZuT3v7Tg=', 'Content-Length': '0'} 2023-05-31T21:12:27.024 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '10'), ('X-RGW-Bytes-Used', '30'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000047597428e64ddea2-006477b83b-1630-a1'), ('Content-Length', '0'), ('Date', 'Wed, 31 May 2023 21:12:27 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:12:27.025 DEBUG:boto:path=/pklxkd-64/obj19 2023-05-31T21:12:27.025 DEBUG:boto:auth_path=/pklxkd-64/obj19 2023-05-31T21:12:27.026 DEBUG:boto:Method: PUT 2023-05-31T21:12:27.026 DEBUG:boto:Path: /pklxkd-64/obj19 2023-05-31T21:12:27.026 DEBUG:boto:Data: 2023-05-31T21:12:27.027 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-05-31T21:12:27.027 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T21:12:27.027 DEBUG:boto:Port: 8000 2023-05-31T21:12:27.028 DEBUG:boto:Params: {} 2023-05-31T21:12:27.028 DEBUG:boto:Token: None 2023-05-31T21:12:27.028 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Wed, 31 May 2023 21:12:27 GMT /pklxkd-64/obj19 2023-05-31T21:12:27.029 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:A2JdQ9/5ABu80cWWI9FSo44ZsjY= 2023-05-31T21:12:27.029 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Wed, 31 May 2023 21:12:27 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:A2JdQ9/5ABu80cWWI9FSo44ZsjY='} 2023-05-31T21:12:27.035 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000090c769d364ce31af-006477b83b-1630-a1'), ('Date', 'Wed, 31 May 2023 21:12:27 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:12:27.036 DEBUG:boto:path=/pklxkd-64/ 2023-05-31T21:12:27.036 DEBUG:boto:auth_path=/pklxkd-64/ 2023-05-31T21:12:27.037 DEBUG:boto:Method: HEAD 2023-05-31T21:12:27.037 DEBUG:boto:Path: /pklxkd-64/ 2023-05-31T21:12:27.037 DEBUG:boto:Data: 2023-05-31T21:12:27.038 DEBUG:boto:Headers: {} 2023-05-31T21:12:27.038 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T21:12:27.038 DEBUG:boto:Port: 8000 2023-05-31T21:12:27.039 DEBUG:boto:Params: {} 2023-05-31T21:12:27.039 DEBUG:boto:Token: None 2023-05-31T21:12:27.039 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 21:12:27 GMT /pklxkd-64/ 2023-05-31T21:12:27.040 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:5X+CZRYkJIw4yCbpLG+ZuT3v7Tg= 2023-05-31T21:12:27.040 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 21:12:27 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:5X+CZRYkJIw4yCbpLG+ZuT3v7Tg=', 'Content-Length': '0'} 2023-05-31T21:12:27.042 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '11'), ('X-RGW-Bytes-Used', '33'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000007c56c81ad97d7096-006477b83b-1630-a1'), ('Content-Length', '0'), ('Date', 'Wed, 31 May 2023 21:12:27 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:12:27.043 DEBUG:boto:path=/pklxkd-64/obj38 2023-05-31T21:12:27.043 DEBUG:boto:auth_path=/pklxkd-64/obj38 2023-05-31T21:12:27.043 DEBUG:boto:Method: PUT 2023-05-31T21:12:27.044 DEBUG:boto:Path: /pklxkd-64/obj38 2023-05-31T21:12:27.044 DEBUG:boto:Data: 2023-05-31T21:12:27.044 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-05-31T21:12:27.045 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T21:12:27.045 DEBUG:boto:Port: 8000 2023-05-31T21:12:27.045 DEBUG:boto:Params: {} 2023-05-31T21:12:27.046 DEBUG:boto:Token: None 2023-05-31T21:12:27.046 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Wed, 31 May 2023 21:12:27 GMT /pklxkd-64/obj38 2023-05-31T21:12:27.047 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:Vb9bEB0EerBFaVWfIMaOQieqPRY= 2023-05-31T21:12:27.047 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Wed, 31 May 2023 21:12:27 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:Vb9bEB0EerBFaVWfIMaOQieqPRY='} 2023-05-31T21:12:27.052 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000b8e4e43c12f7782f-006477b83b-1630-a1'), ('Date', 'Wed, 31 May 2023 21:12:27 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:12:27.053 DEBUG:boto:path=/pklxkd-64/ 2023-05-31T21:12:27.053 DEBUG:boto:auth_path=/pklxkd-64/ 2023-05-31T21:12:27.053 DEBUG:boto:Method: HEAD 2023-05-31T21:12:27.054 DEBUG:boto:Path: /pklxkd-64/ 2023-05-31T21:12:27.054 DEBUG:boto:Data: 2023-05-31T21:12:27.054 DEBUG:boto:Headers: {} 2023-05-31T21:12:27.055 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T21:12:27.055 DEBUG:boto:Port: 8000 2023-05-31T21:12:27.055 DEBUG:boto:Params: {} 2023-05-31T21:12:27.056 DEBUG:boto:Token: None 2023-05-31T21:12:27.056 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 21:12:27 GMT /pklxkd-64/ 2023-05-31T21:12:27.056 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:5X+CZRYkJIw4yCbpLG+ZuT3v7Tg= 2023-05-31T21:12:27.057 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 21:12:27 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:5X+CZRYkJIw4yCbpLG+ZuT3v7Tg=', 'Content-Length': '0'} 2023-05-31T21:12:27.059 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '12'), ('X-RGW-Bytes-Used', '36'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000f530210c80d6c0bf-006477b83b-1630-a1'), ('Content-Length', '0'), ('Date', 'Wed, 31 May 2023 21:12:27 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:12:27.059 DEBUG:boto:path=/pklxkd-64/obj57 2023-05-31T21:12:27.060 DEBUG:boto:auth_path=/pklxkd-64/obj57 2023-05-31T21:12:27.060 DEBUG:boto:Method: PUT 2023-05-31T21:12:27.061 DEBUG:boto:Path: /pklxkd-64/obj57 2023-05-31T21:12:27.061 DEBUG:boto:Data: 2023-05-31T21:12:27.061 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-05-31T21:12:27.062 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T21:12:27.062 DEBUG:boto:Port: 8000 2023-05-31T21:12:27.062 DEBUG:boto:Params: {} 2023-05-31T21:12:27.063 DEBUG:boto:Token: None 2023-05-31T21:12:27.063 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Wed, 31 May 2023 21:12:27 GMT /pklxkd-64/obj57 2023-05-31T21:12:27.063 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:T1Q6dGrai9KIojbw0x4os6xu57E= 2023-05-31T21:12:27.064 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Wed, 31 May 2023 21:12:27 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:T1Q6dGrai9KIojbw0x4os6xu57E='} 2023-05-31T21:12:27.108 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000fcaebd6c75751c2f-006477b83b-1630-a1'), ('Date', 'Wed, 31 May 2023 21:12:27 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:12:27.109 DEBUG:boto:path=/pklxkd-64/ 2023-05-31T21:12:27.109 DEBUG:boto:auth_path=/pklxkd-64/ 2023-05-31T21:12:27.110 DEBUG:boto:Method: HEAD 2023-05-31T21:12:27.110 DEBUG:boto:Path: /pklxkd-64/ 2023-05-31T21:12:27.110 DEBUG:boto:Data: 2023-05-31T21:12:27.111 DEBUG:boto:Headers: {} 2023-05-31T21:12:27.111 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T21:12:27.111 DEBUG:boto:Port: 8000 2023-05-31T21:12:27.111 DEBUG:boto:Params: {} 2023-05-31T21:12:27.112 DEBUG:boto:Token: None 2023-05-31T21:12:27.112 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 21:12:27 GMT /pklxkd-64/ 2023-05-31T21:12:27.113 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:5X+CZRYkJIw4yCbpLG+ZuT3v7Tg= 2023-05-31T21:12:27.113 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 21:12:27 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:5X+CZRYkJIw4yCbpLG+ZuT3v7Tg=', 'Content-Length': '0'} 2023-05-31T21:12:27.116 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '13'), ('X-RGW-Bytes-Used', '39'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000cb8a08ebf87ab97c-006477b83b-1630-a1'), ('Content-Length', '0'), ('Date', 'Wed, 31 May 2023 21:12:27 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:12:27.116 DEBUG:boto:path=/pklxkd-64/obj76 2023-05-31T21:12:27.116 DEBUG:boto:auth_path=/pklxkd-64/obj76 2023-05-31T21:12:27.117 DEBUG:boto:Method: PUT 2023-05-31T21:12:27.117 DEBUG:boto:Path: /pklxkd-64/obj76 2023-05-31T21:12:27.118 DEBUG:boto:Data: 2023-05-31T21:12:27.118 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-05-31T21:12:27.118 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T21:12:27.118 DEBUG:boto:Port: 8000 2023-05-31T21:12:27.119 DEBUG:boto:Params: {} 2023-05-31T21:12:27.119 DEBUG:boto:Token: None 2023-05-31T21:12:27.120 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Wed, 31 May 2023 21:12:27 GMT /pklxkd-64/obj76 2023-05-31T21:12:27.120 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:OnUg3LFa+4PHCkrdCam2Hzgfpd4= 2023-05-31T21:12:27.121 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Wed, 31 May 2023 21:12:27 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:OnUg3LFa+4PHCkrdCam2Hzgfpd4='} 2023-05-31T21:12:27.131 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000fcb4f66aa66bd147-006477b83b-1630-a1'), ('Date', 'Wed, 31 May 2023 21:12:27 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:12:27.132 DEBUG:boto:path=/pklxkd-64/ 2023-05-31T21:12:27.132 DEBUG:boto:auth_path=/pklxkd-64/ 2023-05-31T21:12:27.133 DEBUG:boto:Method: HEAD 2023-05-31T21:12:27.133 DEBUG:boto:Path: /pklxkd-64/ 2023-05-31T21:12:27.133 DEBUG:boto:Data: 2023-05-31T21:12:27.134 DEBUG:boto:Headers: {} 2023-05-31T21:12:27.134 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T21:12:27.134 DEBUG:boto:Port: 8000 2023-05-31T21:12:27.134 DEBUG:boto:Params: {} 2023-05-31T21:12:27.135 DEBUG:boto:Token: None 2023-05-31T21:12:27.135 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 21:12:27 GMT /pklxkd-64/ 2023-05-31T21:12:27.136 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:5X+CZRYkJIw4yCbpLG+ZuT3v7Tg= 2023-05-31T21:12:27.136 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 21:12:27 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:5X+CZRYkJIw4yCbpLG+ZuT3v7Tg=', 'Content-Length': '0'} 2023-05-31T21:12:27.138 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '14'), ('X-RGW-Bytes-Used', '42'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000bc9292d68385041e-006477b83b-1630-a1'), ('Content-Length', '0'), ('Date', 'Wed, 31 May 2023 21:12:27 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:12:27.139 DEBUG:boto:path=/pklxkd-64/obj95 2023-05-31T21:12:27.139 DEBUG:boto:auth_path=/pklxkd-64/obj95 2023-05-31T21:12:27.139 DEBUG:boto:Method: PUT 2023-05-31T21:12:27.140 DEBUG:boto:Path: /pklxkd-64/obj95 2023-05-31T21:12:27.140 DEBUG:boto:Data: 2023-05-31T21:12:27.140 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-05-31T21:12:27.141 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T21:12:27.141 DEBUG:boto:Port: 8000 2023-05-31T21:12:27.141 DEBUG:boto:Params: {} 2023-05-31T21:12:27.142 DEBUG:boto:Token: None 2023-05-31T21:12:27.142 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Wed, 31 May 2023 21:12:27 GMT /pklxkd-64/obj95 2023-05-31T21:12:27.143 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:b0WHs+gaxTAohHBUeJZEhCprsuw= 2023-05-31T21:12:27.143 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Wed, 31 May 2023 21:12:27 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:b0WHs+gaxTAohHBUeJZEhCprsuw='} 2023-05-31T21:12:27.147 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000007be561e131667703-006477b83b-1630-a1'), ('Date', 'Wed, 31 May 2023 21:12:27 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:12:27.147 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'reshard', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'pklxkd-64', '--num-shards', '23', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-05-31T21:12:27.148 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'reshard', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'pklxkd-64', '--num-shards', '23', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-05-31T21:12:27.148 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket reshard --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --bucket pklxkd-64 --num-shards 23 --yes-i-really-mean-it --cluster c1 --debug-rgw 0 --debug-ms 0 2023-05-31T21:12:27.200 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2023-05-31T21:12:27.201 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2023-05-31T21:12:27.251 INFO:teuthology.orchestra.run.smithi099.stdout:tenant: 2023-05-31T21:12:27.252 INFO:teuthology.orchestra.run.smithi099.stdout:bucket name: pklxkd-64 2023-05-31T21:12:27.258 INFO:teuthology.orchestra.run.smithi099.stdout:total entries: 15 2023-05-31T21:12:27.315 DEBUG:boto:path=/pklxkd-64/ 2023-05-31T21:12:27.315 DEBUG:boto:auth_path=/pklxkd-64/ 2023-05-31T21:12:27.316 DEBUG:boto:Method: HEAD 2023-05-31T21:12:27.316 DEBUG:boto:Path: /pklxkd-64/ 2023-05-31T21:12:27.316 DEBUG:boto:Data: 2023-05-31T21:12:27.317 DEBUG:boto:Headers: {} 2023-05-31T21:12:27.317 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T21:12:27.317 DEBUG:boto:Port: 8000 2023-05-31T21:12:27.318 DEBUG:boto:Params: {} 2023-05-31T21:12:27.318 DEBUG:boto:Token: None 2023-05-31T21:12:27.318 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 21:12:27 GMT /pklxkd-64/ 2023-05-31T21:12:27.319 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:5X+CZRYkJIw4yCbpLG+ZuT3v7Tg= 2023-05-31T21:12:27.319 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 21:12:27 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:5X+CZRYkJIw4yCbpLG+ZuT3v7Tg=', 'Content-Length': '0'} 2023-05-31T21:12:27.322 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '15'), ('X-RGW-Bytes-Used', '45'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000d869f571c73db76d-006477b83b-1630-a1'), ('Content-Length', '0'), ('Date', 'Wed, 31 May 2023 21:12:27 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:12:27.323 DEBUG:boto:path=/pklxkd-64/obj23 2023-05-31T21:12:27.323 DEBUG:boto:auth_path=/pklxkd-64/obj23 2023-05-31T21:12:27.323 DEBUG:boto:Method: PUT 2023-05-31T21:12:27.324 DEBUG:boto:Path: /pklxkd-64/obj23 2023-05-31T21:12:27.324 DEBUG:boto:Data: 2023-05-31T21:12:27.325 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-05-31T21:12:27.325 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T21:12:27.325 DEBUG:boto:Port: 8000 2023-05-31T21:12:27.325 DEBUG:boto:Params: {} 2023-05-31T21:12:27.326 DEBUG:boto:Token: None 2023-05-31T21:12:27.326 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Wed, 31 May 2023 21:12:27 GMT /pklxkd-64/obj23 2023-05-31T21:12:27.327 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:W6y6Sx+q5uJUypBgeAdudmkJ1WY= 2023-05-31T21:12:27.327 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Wed, 31 May 2023 21:12:27 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:W6y6Sx+q5uJUypBgeAdudmkJ1WY='} 2023-05-31T21:12:27.372 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000003830f63fcc2aab6f-006477b83b-1630-a1'), ('Date', 'Wed, 31 May 2023 21:12:27 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:12:27.373 DEBUG:boto:path=/pklxkd-64/ 2023-05-31T21:12:27.373 DEBUG:boto:auth_path=/pklxkd-64/ 2023-05-31T21:12:27.373 DEBUG:boto:Method: HEAD 2023-05-31T21:12:27.374 DEBUG:boto:Path: /pklxkd-64/ 2023-05-31T21:12:27.374 DEBUG:boto:Data: 2023-05-31T21:12:27.374 DEBUG:boto:Headers: {} 2023-05-31T21:12:27.375 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T21:12:27.375 DEBUG:boto:Port: 8000 2023-05-31T21:12:27.375 DEBUG:boto:Params: {} 2023-05-31T21:12:27.376 DEBUG:boto:Token: None 2023-05-31T21:12:27.376 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 21:12:27 GMT /pklxkd-64/ 2023-05-31T21:12:27.376 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:5X+CZRYkJIw4yCbpLG+ZuT3v7Tg= 2023-05-31T21:12:27.377 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 21:12:27 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:5X+CZRYkJIw4yCbpLG+ZuT3v7Tg=', 'Content-Length': '0'} 2023-05-31T21:12:27.380 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '16'), ('X-RGW-Bytes-Used', '48'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000030a4be84dd1dd6a3-006477b83b-1630-a1'), ('Content-Length', '0'), ('Date', 'Wed, 31 May 2023 21:12:27 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:12:27.380 DEBUG:boto:path=/pklxkd-64/obj46 2023-05-31T21:12:27.381 DEBUG:boto:auth_path=/pklxkd-64/obj46 2023-05-31T21:12:27.381 DEBUG:boto:Method: PUT 2023-05-31T21:12:27.382 DEBUG:boto:Path: /pklxkd-64/obj46 2023-05-31T21:12:27.382 DEBUG:boto:Data: 2023-05-31T21:12:27.382 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-05-31T21:12:27.383 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T21:12:27.383 DEBUG:boto:Port: 8000 2023-05-31T21:12:27.383 DEBUG:boto:Params: {} 2023-05-31T21:12:27.384 DEBUG:boto:Token: None 2023-05-31T21:12:27.384 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Wed, 31 May 2023 21:12:27 GMT /pklxkd-64/obj46 2023-05-31T21:12:27.384 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:+v8b/mAzsWDWLlfFczRLwaAIU6s= 2023-05-31T21:12:27.385 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Wed, 31 May 2023 21:12:27 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:+v8b/mAzsWDWLlfFczRLwaAIU6s='} 2023-05-31T21:12:27.391 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000861cb24573876db2-006477b83b-1630-a1'), ('Date', 'Wed, 31 May 2023 21:12:27 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:12:27.392 DEBUG:boto:path=/pklxkd-64/ 2023-05-31T21:12:27.392 DEBUG:boto:auth_path=/pklxkd-64/ 2023-05-31T21:12:27.392 DEBUG:boto:Method: HEAD 2023-05-31T21:12:27.393 DEBUG:boto:Path: /pklxkd-64/ 2023-05-31T21:12:27.393 DEBUG:boto:Data: 2023-05-31T21:12:27.393 DEBUG:boto:Headers: {} 2023-05-31T21:12:27.394 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T21:12:27.394 DEBUG:boto:Port: 8000 2023-05-31T21:12:27.394 DEBUG:boto:Params: {} 2023-05-31T21:12:27.395 DEBUG:boto:Token: None 2023-05-31T21:12:27.395 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 21:12:27 GMT /pklxkd-64/ 2023-05-31T21:12:27.395 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:5X+CZRYkJIw4yCbpLG+ZuT3v7Tg= 2023-05-31T21:12:27.396 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 21:12:27 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:5X+CZRYkJIw4yCbpLG+ZuT3v7Tg=', 'Content-Length': '0'} 2023-05-31T21:12:27.398 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '17'), ('X-RGW-Bytes-Used', '51'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000f4eab07ae429c1e9-006477b83b-1630-a1'), ('Content-Length', '0'), ('Date', 'Wed, 31 May 2023 21:12:27 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:12:27.399 DEBUG:boto:path=/pklxkd-64/obj69 2023-05-31T21:12:27.399 DEBUG:boto:auth_path=/pklxkd-64/obj69 2023-05-31T21:12:27.400 DEBUG:boto:Method: PUT 2023-05-31T21:12:27.400 DEBUG:boto:Path: /pklxkd-64/obj69 2023-05-31T21:12:27.400 DEBUG:boto:Data: 2023-05-31T21:12:27.401 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-05-31T21:12:27.401 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T21:12:27.401 DEBUG:boto:Port: 8000 2023-05-31T21:12:27.402 DEBUG:boto:Params: {} 2023-05-31T21:12:27.402 DEBUG:boto:Token: None 2023-05-31T21:12:27.402 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Wed, 31 May 2023 21:12:27 GMT /pklxkd-64/obj69 2023-05-31T21:12:27.403 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:xGQkOVG17u+wkUcSi8jToV8jB88= 2023-05-31T21:12:27.403 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Wed, 31 May 2023 21:12:27 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:xGQkOVG17u+wkUcSi8jToV8jB88='} 2023-05-31T21:12:27.410 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000ba7a4a31fb53d4eb-006477b83b-1630-a1'), ('Date', 'Wed, 31 May 2023 21:12:27 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:12:27.411 DEBUG:boto:path=/pklxkd-64/ 2023-05-31T21:12:27.411 DEBUG:boto:auth_path=/pklxkd-64/ 2023-05-31T21:12:27.412 DEBUG:boto:Method: HEAD 2023-05-31T21:12:27.412 DEBUG:boto:Path: /pklxkd-64/ 2023-05-31T21:12:27.412 DEBUG:boto:Data: 2023-05-31T21:12:27.413 DEBUG:boto:Headers: {} 2023-05-31T21:12:27.413 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T21:12:27.413 DEBUG:boto:Port: 8000 2023-05-31T21:12:27.414 DEBUG:boto:Params: {} 2023-05-31T21:12:27.414 DEBUG:boto:Token: None 2023-05-31T21:12:27.414 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 21:12:27 GMT /pklxkd-64/ 2023-05-31T21:12:27.415 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:5X+CZRYkJIw4yCbpLG+ZuT3v7Tg= 2023-05-31T21:12:27.415 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 21:12:27 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:5X+CZRYkJIw4yCbpLG+ZuT3v7Tg=', 'Content-Length': '0'} 2023-05-31T21:12:27.418 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '18'), ('X-RGW-Bytes-Used', '54'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000957971fdda725249-006477b83b-1630-a1'), ('Content-Length', '0'), ('Date', 'Wed, 31 May 2023 21:12:27 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:12:27.418 DEBUG:boto:path=/pklxkd-64/obj92 2023-05-31T21:12:27.418 DEBUG:boto:auth_path=/pklxkd-64/obj92 2023-05-31T21:12:27.419 DEBUG:boto:Method: PUT 2023-05-31T21:12:27.419 DEBUG:boto:Path: /pklxkd-64/obj92 2023-05-31T21:12:27.420 DEBUG:boto:Data: 2023-05-31T21:12:27.420 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-05-31T21:12:27.420 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T21:12:27.421 DEBUG:boto:Port: 8000 2023-05-31T21:12:27.421 DEBUG:boto:Params: {} 2023-05-31T21:12:27.421 DEBUG:boto:Token: None 2023-05-31T21:12:27.422 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Wed, 31 May 2023 21:12:27 GMT /pklxkd-64/obj92 2023-05-31T21:12:27.422 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:ymh+SiqHVZO5Gjm728vj5qQJ5tI= 2023-05-31T21:12:27.423 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Wed, 31 May 2023 21:12:27 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:ymh+SiqHVZO5Gjm728vj5qQJ5tI='} 2023-05-31T21:12:27.464 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000aff85a7af419da3e-006477b83b-1630-a1'), ('Date', 'Wed, 31 May 2023 21:12:27 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:12:27.465 DEBUG:boto:path=/pklxkd-64/ 2023-05-31T21:12:27.465 DEBUG:boto:auth_path=/pklxkd-64/ 2023-05-31T21:12:27.466 DEBUG:boto:Method: HEAD 2023-05-31T21:12:27.466 DEBUG:boto:Path: /pklxkd-64/ 2023-05-31T21:12:27.466 DEBUG:boto:Data: 2023-05-31T21:12:27.467 DEBUG:boto:Headers: {} 2023-05-31T21:12:27.467 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T21:12:27.467 DEBUG:boto:Port: 8000 2023-05-31T21:12:27.467 DEBUG:boto:Params: {} 2023-05-31T21:12:27.468 DEBUG:boto:Token: None 2023-05-31T21:12:27.468 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 21:12:27 GMT /pklxkd-64/ 2023-05-31T21:12:27.469 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:5X+CZRYkJIw4yCbpLG+ZuT3v7Tg= 2023-05-31T21:12:27.469 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 21:12:27 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:5X+CZRYkJIw4yCbpLG+ZuT3v7Tg=', 'Content-Length': '0'} 2023-05-31T21:12:27.472 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '19'), ('X-RGW-Bytes-Used', '57'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000c07a17f39d8fdcce-006477b83b-1630-a1'), ('Content-Length', '0'), ('Date', 'Wed, 31 May 2023 21:12:27 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:12:27.473 DEBUG:boto:path=/pklxkd-64/obj115 2023-05-31T21:12:27.473 DEBUG:boto:auth_path=/pklxkd-64/obj115 2023-05-31T21:12:27.474 DEBUG:boto:Method: PUT 2023-05-31T21:12:27.474 DEBUG:boto:Path: /pklxkd-64/obj115 2023-05-31T21:12:27.474 DEBUG:boto:Data: 2023-05-31T21:12:27.475 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-05-31T21:12:27.475 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T21:12:27.475 DEBUG:boto:Port: 8000 2023-05-31T21:12:27.476 DEBUG:boto:Params: {} 2023-05-31T21:12:27.476 DEBUG:boto:Token: None 2023-05-31T21:12:27.476 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Wed, 31 May 2023 21:12:27 GMT /pklxkd-64/obj115 2023-05-31T21:12:27.477 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:ooz3ewEud8Vv3By4QWkU3r1snyI= 2023-05-31T21:12:27.477 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Wed, 31 May 2023 21:12:27 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:ooz3ewEud8Vv3By4QWkU3r1snyI='} 2023-05-31T21:12:27.485 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000009b862c5a1d7d8cbb-006477b83b-1630-a1'), ('Date', 'Wed, 31 May 2023 21:12:27 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:12:27.486 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'reshard', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'pklxkd-64', '--num-shards', '29', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-05-31T21:12:27.486 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'reshard', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'pklxkd-64', '--num-shards', '29', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-05-31T21:12:27.487 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket reshard --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --bucket pklxkd-64 --num-shards 29 --yes-i-really-mean-it --cluster c1 --debug-rgw 0 --debug-ms 0 2023-05-31T21:12:27.539 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2023-05-31T21:12:27.540 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2023-05-31T21:12:27.609 INFO:teuthology.orchestra.run.smithi099.stdout:tenant: 2023-05-31T21:12:27.609 INFO:teuthology.orchestra.run.smithi099.stdout:bucket name: pklxkd-64 2023-05-31T21:12:27.618 INFO:teuthology.orchestra.run.smithi099.stdout:total entries: 20 2023-05-31T21:12:27.697 DEBUG:boto:path=/pklxkd-64/ 2023-05-31T21:12:27.697 DEBUG:boto:auth_path=/pklxkd-64/ 2023-05-31T21:12:27.697 DEBUG:boto:Method: HEAD 2023-05-31T21:12:27.698 DEBUG:boto:Path: /pklxkd-64/ 2023-05-31T21:12:27.698 DEBUG:boto:Data: 2023-05-31T21:12:27.698 DEBUG:boto:Headers: {} 2023-05-31T21:12:27.699 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T21:12:27.699 DEBUG:boto:Port: 8000 2023-05-31T21:12:27.699 DEBUG:boto:Params: {} 2023-05-31T21:12:27.700 DEBUG:boto:Token: None 2023-05-31T21:12:27.700 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 21:12:27 GMT /pklxkd-64/ 2023-05-31T21:12:27.701 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:5X+CZRYkJIw4yCbpLG+ZuT3v7Tg= 2023-05-31T21:12:27.701 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 21:12:27 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:5X+CZRYkJIw4yCbpLG+ZuT3v7Tg=', 'Content-Length': '0'} 2023-05-31T21:12:27.705 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '20'), ('X-RGW-Bytes-Used', '60'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000543e72c4cec958f4-006477b83b-1630-a1'), ('Content-Length', '0'), ('Date', 'Wed, 31 May 2023 21:12:27 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:12:27.705 DEBUG:boto:path=/pklxkd-64/obj29 2023-05-31T21:12:27.706 DEBUG:boto:auth_path=/pklxkd-64/obj29 2023-05-31T21:12:27.706 DEBUG:boto:Method: PUT 2023-05-31T21:12:27.706 DEBUG:boto:Path: /pklxkd-64/obj29 2023-05-31T21:12:27.707 DEBUG:boto:Data: 2023-05-31T21:12:27.707 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-05-31T21:12:27.707 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T21:12:27.708 DEBUG:boto:Port: 8000 2023-05-31T21:12:27.708 DEBUG:boto:Params: {} 2023-05-31T21:12:27.708 DEBUG:boto:Token: None 2023-05-31T21:12:27.709 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Wed, 31 May 2023 21:12:27 GMT /pklxkd-64/obj29 2023-05-31T21:12:27.709 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:t7QT1RnKQ1Uo0s5Xjhh5UkMsRUI= 2023-05-31T21:12:27.710 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Wed, 31 May 2023 21:12:27 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:t7QT1RnKQ1Uo0s5Xjhh5UkMsRUI='} 2023-05-31T21:12:27.752 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000073c5576572c5f2d5-006477b83b-1630-a1'), ('Date', 'Wed, 31 May 2023 21:12:27 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:12:27.753 DEBUG:boto:path=/pklxkd-64/ 2023-05-31T21:12:27.753 DEBUG:boto:auth_path=/pklxkd-64/ 2023-05-31T21:12:27.754 DEBUG:boto:Method: HEAD 2023-05-31T21:12:27.754 DEBUG:boto:Path: /pklxkd-64/ 2023-05-31T21:12:27.754 DEBUG:boto:Data: 2023-05-31T21:12:27.755 DEBUG:boto:Headers: {} 2023-05-31T21:12:27.755 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T21:12:27.755 DEBUG:boto:Port: 8000 2023-05-31T21:12:27.755 DEBUG:boto:Params: {} 2023-05-31T21:12:27.756 DEBUG:boto:Token: None 2023-05-31T21:12:27.756 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 21:12:27 GMT /pklxkd-64/ 2023-05-31T21:12:27.757 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:5X+CZRYkJIw4yCbpLG+ZuT3v7Tg= 2023-05-31T21:12:27.757 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 21:12:27 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:5X+CZRYkJIw4yCbpLG+ZuT3v7Tg=', 'Content-Length': '0'} 2023-05-31T21:12:27.760 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '21'), ('X-RGW-Bytes-Used', '63'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000006b0b7911504667f4-006477b83b-1630-a1'), ('Content-Length', '0'), ('Date', 'Wed, 31 May 2023 21:12:27 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:12:27.760 DEBUG:boto:path=/pklxkd-64/obj58 2023-05-31T21:12:27.761 DEBUG:boto:auth_path=/pklxkd-64/obj58 2023-05-31T21:12:27.761 DEBUG:boto:Method: PUT 2023-05-31T21:12:27.762 DEBUG:boto:Path: /pklxkd-64/obj58 2023-05-31T21:12:27.762 DEBUG:boto:Data: 2023-05-31T21:12:27.762 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-05-31T21:12:27.763 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T21:12:27.763 DEBUG:boto:Port: 8000 2023-05-31T21:12:27.763 DEBUG:boto:Params: {} 2023-05-31T21:12:27.764 DEBUG:boto:Token: None 2023-05-31T21:12:27.764 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Wed, 31 May 2023 21:12:27 GMT /pklxkd-64/obj58 2023-05-31T21:12:27.765 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:9I8FljyiHRtiyAl9DFT5eSkTJGI= 2023-05-31T21:12:27.765 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Wed, 31 May 2023 21:12:27 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:9I8FljyiHRtiyAl9DFT5eSkTJGI='} 2023-05-31T21:12:27.770 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000002306688ce75ac555-006477b83b-1630-a1'), ('Date', 'Wed, 31 May 2023 21:12:27 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:12:27.770 DEBUG:boto:path=/pklxkd-64/ 2023-05-31T21:12:27.771 DEBUG:boto:auth_path=/pklxkd-64/ 2023-05-31T21:12:27.771 DEBUG:boto:Method: HEAD 2023-05-31T21:12:27.772 DEBUG:boto:Path: /pklxkd-64/ 2023-05-31T21:12:27.772 DEBUG:boto:Data: 2023-05-31T21:12:27.772 DEBUG:boto:Headers: {} 2023-05-31T21:12:27.773 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T21:12:27.773 DEBUG:boto:Port: 8000 2023-05-31T21:12:27.773 DEBUG:boto:Params: {} 2023-05-31T21:12:27.773 DEBUG:boto:Token: None 2023-05-31T21:12:27.774 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 21:12:27 GMT /pklxkd-64/ 2023-05-31T21:12:27.774 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:5X+CZRYkJIw4yCbpLG+ZuT3v7Tg= 2023-05-31T21:12:27.775 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 21:12:27 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:5X+CZRYkJIw4yCbpLG+ZuT3v7Tg=', 'Content-Length': '0'} 2023-05-31T21:12:27.777 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '22'), ('X-RGW-Bytes-Used', '66'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000fbf3f5a8ab3e526d-006477b83b-1630-a1'), ('Content-Length', '0'), ('Date', 'Wed, 31 May 2023 21:12:27 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:12:27.778 DEBUG:boto:path=/pklxkd-64/obj87 2023-05-31T21:12:27.778 DEBUG:boto:auth_path=/pklxkd-64/obj87 2023-05-31T21:12:27.779 DEBUG:boto:Method: PUT 2023-05-31T21:12:27.779 DEBUG:boto:Path: /pklxkd-64/obj87 2023-05-31T21:12:27.779 DEBUG:boto:Data: 2023-05-31T21:12:27.780 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-05-31T21:12:27.780 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T21:12:27.780 DEBUG:boto:Port: 8000 2023-05-31T21:12:27.781 DEBUG:boto:Params: {} 2023-05-31T21:12:27.781 DEBUG:boto:Token: None 2023-05-31T21:12:27.781 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Wed, 31 May 2023 21:12:27 GMT /pklxkd-64/obj87 2023-05-31T21:12:27.782 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:K5xk9bbxgpZuDo7eiNcISG+mzQA= 2023-05-31T21:12:27.782 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Wed, 31 May 2023 21:12:27 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:K5xk9bbxgpZuDo7eiNcISG+mzQA='} 2023-05-31T21:12:27.825 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000e46a35ebe04ed500-006477b83b-1630-a1'), ('Date', 'Wed, 31 May 2023 21:12:27 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:12:27.826 DEBUG:boto:path=/pklxkd-64/ 2023-05-31T21:12:27.826 DEBUG:boto:auth_path=/pklxkd-64/ 2023-05-31T21:12:27.827 DEBUG:boto:Method: HEAD 2023-05-31T21:12:27.827 DEBUG:boto:Path: /pklxkd-64/ 2023-05-31T21:12:27.827 DEBUG:boto:Data: 2023-05-31T21:12:27.828 DEBUG:boto:Headers: {} 2023-05-31T21:12:27.828 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T21:12:27.829 DEBUG:boto:Port: 8000 2023-05-31T21:12:27.829 DEBUG:boto:Params: {} 2023-05-31T21:12:27.829 DEBUG:boto:Token: None 2023-05-31T21:12:27.830 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 21:12:27 GMT /pklxkd-64/ 2023-05-31T21:12:27.830 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:5X+CZRYkJIw4yCbpLG+ZuT3v7Tg= 2023-05-31T21:12:27.831 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 21:12:27 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:5X+CZRYkJIw4yCbpLG+ZuT3v7Tg=', 'Content-Length': '0'} 2023-05-31T21:12:27.834 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '23'), ('X-RGW-Bytes-Used', '69'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000003644d7c46a50a743-006477b83b-1630-a1'), ('Content-Length', '0'), ('Date', 'Wed, 31 May 2023 21:12:27 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:12:27.834 DEBUG:boto:path=/pklxkd-64/obj116 2023-05-31T21:12:27.835 DEBUG:boto:auth_path=/pklxkd-64/obj116 2023-05-31T21:12:27.835 DEBUG:boto:Method: PUT 2023-05-31T21:12:27.835 DEBUG:boto:Path: /pklxkd-64/obj116 2023-05-31T21:12:27.836 DEBUG:boto:Data: 2023-05-31T21:12:27.836 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-05-31T21:12:27.836 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T21:12:27.837 DEBUG:boto:Port: 8000 2023-05-31T21:12:27.837 DEBUG:boto:Params: {} 2023-05-31T21:12:27.837 DEBUG:boto:Token: None 2023-05-31T21:12:27.838 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Wed, 31 May 2023 21:12:27 GMT /pklxkd-64/obj116 2023-05-31T21:12:27.838 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:0GB6LjmEjDniQHpt1z6yvzOxfqA= 2023-05-31T21:12:27.839 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Wed, 31 May 2023 21:12:27 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:0GB6LjmEjDniQHpt1z6yvzOxfqA='} 2023-05-31T21:12:27.845 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000d02f82117d4df397-006477b83b-1630-a1'), ('Date', 'Wed, 31 May 2023 21:12:27 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:12:27.845 DEBUG:boto:path=/pklxkd-64/ 2023-05-31T21:12:27.846 DEBUG:boto:auth_path=/pklxkd-64/ 2023-05-31T21:12:27.846 DEBUG:boto:Method: HEAD 2023-05-31T21:12:27.847 DEBUG:boto:Path: /pklxkd-64/ 2023-05-31T21:12:27.847 DEBUG:boto:Data: 2023-05-31T21:12:27.847 DEBUG:boto:Headers: {} 2023-05-31T21:12:27.848 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T21:12:27.848 DEBUG:boto:Port: 8000 2023-05-31T21:12:27.848 DEBUG:boto:Params: {} 2023-05-31T21:12:27.849 DEBUG:boto:Token: None 2023-05-31T21:12:27.849 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 21:12:27 GMT /pklxkd-64/ 2023-05-31T21:12:27.850 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:5X+CZRYkJIw4yCbpLG+ZuT3v7Tg= 2023-05-31T21:12:27.850 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 21:12:27 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:5X+CZRYkJIw4yCbpLG+ZuT3v7Tg=', 'Content-Length': '0'} 2023-05-31T21:12:27.852 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '24'), ('X-RGW-Bytes-Used', '72'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000005d279c20b2596ab0-006477b83b-1630-a1'), ('Content-Length', '0'), ('Date', 'Wed, 31 May 2023 21:12:27 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:12:27.853 DEBUG:boto:path=/pklxkd-64/obj145 2023-05-31T21:12:27.853 DEBUG:boto:auth_path=/pklxkd-64/obj145 2023-05-31T21:12:27.854 DEBUG:boto:Method: PUT 2023-05-31T21:12:27.854 DEBUG:boto:Path: /pklxkd-64/obj145 2023-05-31T21:12:27.854 DEBUG:boto:Data: 2023-05-31T21:12:27.855 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-05-31T21:12:27.855 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T21:12:27.855 DEBUG:boto:Port: 8000 2023-05-31T21:12:27.856 DEBUG:boto:Params: {} 2023-05-31T21:12:27.856 DEBUG:boto:Token: None 2023-05-31T21:12:27.857 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Wed, 31 May 2023 21:12:27 GMT /pklxkd-64/obj145 2023-05-31T21:12:27.857 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:Ybsyo2W+0a7PkvpH4cKZS+1fMos= 2023-05-31T21:12:27.857 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Wed, 31 May 2023 21:12:27 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:Ybsyo2W+0a7PkvpH4cKZS+1fMos='} 2023-05-31T21:12:27.900 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000007eadb198801829c3-006477b83b-1630-a1'), ('Date', 'Wed, 31 May 2023 21:12:27 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:12:27.901 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'reshard', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'pklxkd-64', '--num-shards', '31', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-05-31T21:12:27.901 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'reshard', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'pklxkd-64', '--num-shards', '31', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-05-31T21:12:27.902 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket reshard --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --bucket pklxkd-64 --num-shards 31 --yes-i-really-mean-it --cluster c1 --debug-rgw 0 --debug-ms 0 2023-05-31T21:12:27.954 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2023-05-31T21:12:27.954 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2023-05-31T21:12:28.018 INFO:teuthology.orchestra.run.smithi099.stdout:tenant: 2023-05-31T21:12:28.019 INFO:teuthology.orchestra.run.smithi099.stdout:bucket name: pklxkd-64 2023-05-31T21:12:28.029 INFO:teuthology.orchestra.run.smithi099.stdout:total entries: 25 2023-05-31T21:12:28.119 DEBUG:boto:path=/pklxkd-64/ 2023-05-31T21:12:28.120 DEBUG:boto:auth_path=/pklxkd-64/ 2023-05-31T21:12:28.120 DEBUG:boto:Method: HEAD 2023-05-31T21:12:28.121 DEBUG:boto:Path: /pklxkd-64/ 2023-05-31T21:12:28.121 DEBUG:boto:Data: 2023-05-31T21:12:28.121 DEBUG:boto:Headers: {} 2023-05-31T21:12:28.122 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T21:12:28.122 DEBUG:boto:Port: 8000 2023-05-31T21:12:28.122 DEBUG:boto:Params: {} 2023-05-31T21:12:28.123 DEBUG:boto:Token: None 2023-05-31T21:12:28.123 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 21:12:28 GMT /pklxkd-64/ 2023-05-31T21:12:28.124 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:wRm+Nzs6+vQGxm2DWUEut4l9HJU= 2023-05-31T21:12:28.124 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 21:12:28 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:wRm+Nzs6+vQGxm2DWUEut4l9HJU=', 'Content-Length': '0'} 2023-05-31T21:12:28.128 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '25'), ('X-RGW-Bytes-Used', '75'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000026e9078c7e3b079-006477b83c-1630-a1'), ('Content-Length', '0'), ('Date', 'Wed, 31 May 2023 21:12:28 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:12:28.128 DEBUG:boto:path=/pklxkd-64/obj31 2023-05-31T21:12:28.129 DEBUG:boto:auth_path=/pklxkd-64/obj31 2023-05-31T21:12:28.129 DEBUG:boto:Method: PUT 2023-05-31T21:12:28.129 DEBUG:boto:Path: /pklxkd-64/obj31 2023-05-31T21:12:28.130 DEBUG:boto:Data: 2023-05-31T21:12:28.130 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-05-31T21:12:28.130 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T21:12:28.131 DEBUG:boto:Port: 8000 2023-05-31T21:12:28.131 DEBUG:boto:Params: {} 2023-05-31T21:12:28.131 DEBUG:boto:Token: None 2023-05-31T21:12:28.132 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Wed, 31 May 2023 21:12:28 GMT /pklxkd-64/obj31 2023-05-31T21:12:28.132 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:xMsZxJtxILTfdzX032FxNs2P278= 2023-05-31T21:12:28.133 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Wed, 31 May 2023 21:12:28 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:xMsZxJtxILTfdzX032FxNs2P278='} 2023-05-31T21:12:28.138 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000dad4ea32b5673cb2-006477b83c-1630-a1'), ('Date', 'Wed, 31 May 2023 21:12:28 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:12:28.138 DEBUG:boto:path=/pklxkd-64/ 2023-05-31T21:12:28.139 DEBUG:boto:auth_path=/pklxkd-64/ 2023-05-31T21:12:28.139 DEBUG:boto:Method: HEAD 2023-05-31T21:12:28.140 DEBUG:boto:Path: /pklxkd-64/ 2023-05-31T21:12:28.140 DEBUG:boto:Data: 2023-05-31T21:12:28.140 DEBUG:boto:Headers: {} 2023-05-31T21:12:28.141 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T21:12:28.141 DEBUG:boto:Port: 8000 2023-05-31T21:12:28.141 DEBUG:boto:Params: {} 2023-05-31T21:12:28.142 DEBUG:boto:Token: None 2023-05-31T21:12:28.142 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 21:12:28 GMT /pklxkd-64/ 2023-05-31T21:12:28.143 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:wRm+Nzs6+vQGxm2DWUEut4l9HJU= 2023-05-31T21:12:28.143 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 21:12:28 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:wRm+Nzs6+vQGxm2DWUEut4l9HJU=', 'Content-Length': '0'} 2023-05-31T21:12:28.146 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '26'), ('X-RGW-Bytes-Used', '78'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000688681a10b388eff-006477b83c-1630-a1'), ('Content-Length', '0'), ('Date', 'Wed, 31 May 2023 21:12:28 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:12:28.146 DEBUG:boto:path=/pklxkd-64/obj62 2023-05-31T21:12:28.147 DEBUG:boto:auth_path=/pklxkd-64/obj62 2023-05-31T21:12:28.147 DEBUG:boto:Method: PUT 2023-05-31T21:12:28.148 DEBUG:boto:Path: /pklxkd-64/obj62 2023-05-31T21:12:28.148 DEBUG:boto:Data: 2023-05-31T21:12:28.148 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-05-31T21:12:28.149 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T21:12:28.149 DEBUG:boto:Port: 8000 2023-05-31T21:12:28.149 DEBUG:boto:Params: {} 2023-05-31T21:12:28.150 DEBUG:boto:Token: None 2023-05-31T21:12:28.150 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Wed, 31 May 2023 21:12:28 GMT /pklxkd-64/obj62 2023-05-31T21:12:28.150 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:d1DyCtPDHtuQTxUCfMZO0lvjABY= 2023-05-31T21:12:28.151 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Wed, 31 May 2023 21:12:28 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:d1DyCtPDHtuQTxUCfMZO0lvjABY='} 2023-05-31T21:12:28.156 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000cb80b6c2b0bdbcf3-006477b83c-1630-a1'), ('Date', 'Wed, 31 May 2023 21:12:28 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:12:28.157 DEBUG:boto:path=/pklxkd-64/ 2023-05-31T21:12:28.157 DEBUG:boto:auth_path=/pklxkd-64/ 2023-05-31T21:12:28.157 DEBUG:boto:Method: HEAD 2023-05-31T21:12:28.158 DEBUG:boto:Path: /pklxkd-64/ 2023-05-31T21:12:28.158 DEBUG:boto:Data: 2023-05-31T21:12:28.158 DEBUG:boto:Headers: {} 2023-05-31T21:12:28.159 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T21:12:28.159 DEBUG:boto:Port: 8000 2023-05-31T21:12:28.159 DEBUG:boto:Params: {} 2023-05-31T21:12:28.160 DEBUG:boto:Token: None 2023-05-31T21:12:28.160 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 21:12:28 GMT /pklxkd-64/ 2023-05-31T21:12:28.161 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:wRm+Nzs6+vQGxm2DWUEut4l9HJU= 2023-05-31T21:12:28.161 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 21:12:28 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:wRm+Nzs6+vQGxm2DWUEut4l9HJU=', 'Content-Length': '0'} 2023-05-31T21:12:28.164 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '27'), ('X-RGW-Bytes-Used', '81'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000011de4131e59e949-006477b83c-1630-a1'), ('Content-Length', '0'), ('Date', 'Wed, 31 May 2023 21:12:28 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:12:28.165 DEBUG:boto:path=/pklxkd-64/obj93 2023-05-31T21:12:28.165 DEBUG:boto:auth_path=/pklxkd-64/obj93 2023-05-31T21:12:28.165 DEBUG:boto:Method: PUT 2023-05-31T21:12:28.166 DEBUG:boto:Path: /pklxkd-64/obj93 2023-05-31T21:12:28.166 DEBUG:boto:Data: 2023-05-31T21:12:28.167 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-05-31T21:12:28.167 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T21:12:28.167 DEBUG:boto:Port: 8000 2023-05-31T21:12:28.168 DEBUG:boto:Params: {} 2023-05-31T21:12:28.168 DEBUG:boto:Token: None 2023-05-31T21:12:28.168 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Wed, 31 May 2023 21:12:28 GMT /pklxkd-64/obj93 2023-05-31T21:12:28.169 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:4DH1gTpDYbojyofOP8+LM6MvJhQ= 2023-05-31T21:12:28.169 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Wed, 31 May 2023 21:12:28 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:4DH1gTpDYbojyofOP8+LM6MvJhQ='} 2023-05-31T21:12:28.212 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000df6a8753180e4e4b-006477b83c-1630-a1'), ('Date', 'Wed, 31 May 2023 21:12:28 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:12:28.213 DEBUG:boto:path=/pklxkd-64/ 2023-05-31T21:12:28.213 DEBUG:boto:auth_path=/pklxkd-64/ 2023-05-31T21:12:28.214 DEBUG:boto:Method: HEAD 2023-05-31T21:12:28.214 DEBUG:boto:Path: /pklxkd-64/ 2023-05-31T21:12:28.214 DEBUG:boto:Data: 2023-05-31T21:12:28.215 DEBUG:boto:Headers: {} 2023-05-31T21:12:28.215 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T21:12:28.215 DEBUG:boto:Port: 8000 2023-05-31T21:12:28.216 DEBUG:boto:Params: {} 2023-05-31T21:12:28.216 DEBUG:boto:Token: None 2023-05-31T21:12:28.216 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 21:12:28 GMT /pklxkd-64/ 2023-05-31T21:12:28.217 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:wRm+Nzs6+vQGxm2DWUEut4l9HJU= 2023-05-31T21:12:28.217 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 21:12:28 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:wRm+Nzs6+vQGxm2DWUEut4l9HJU=', 'Content-Length': '0'} 2023-05-31T21:12:28.220 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '28'), ('X-RGW-Bytes-Used', '84'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000006498dacbcc2710c0-006477b83c-1630-a1'), ('Content-Length', '0'), ('Date', 'Wed, 31 May 2023 21:12:28 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:12:28.221 DEBUG:boto:path=/pklxkd-64/obj124 2023-05-31T21:12:28.221 DEBUG:boto:auth_path=/pklxkd-64/obj124 2023-05-31T21:12:28.221 DEBUG:boto:Method: PUT 2023-05-31T21:12:28.222 DEBUG:boto:Path: /pklxkd-64/obj124 2023-05-31T21:12:28.222 DEBUG:boto:Data: 2023-05-31T21:12:28.222 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-05-31T21:12:28.223 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T21:12:28.223 DEBUG:boto:Port: 8000 2023-05-31T21:12:28.224 DEBUG:boto:Params: {} 2023-05-31T21:12:28.224 DEBUG:boto:Token: None 2023-05-31T21:12:28.225 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Wed, 31 May 2023 21:12:28 GMT /pklxkd-64/obj124 2023-05-31T21:12:28.225 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:i9QnqSzgpLYMUplJoj3OIyIlFYg= 2023-05-31T21:12:28.225 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Wed, 31 May 2023 21:12:28 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:i9QnqSzgpLYMUplJoj3OIyIlFYg='} 2023-05-31T21:12:28.230 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000f74be47a51de8ab5-006477b83c-1630-a1'), ('Date', 'Wed, 31 May 2023 21:12:28 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:12:28.231 DEBUG:boto:path=/pklxkd-64/ 2023-05-31T21:12:28.231 DEBUG:boto:auth_path=/pklxkd-64/ 2023-05-31T21:12:28.232 DEBUG:boto:Method: HEAD 2023-05-31T21:12:28.232 DEBUG:boto:Path: /pklxkd-64/ 2023-05-31T21:12:28.232 DEBUG:boto:Data: 2023-05-31T21:12:28.233 DEBUG:boto:Headers: {} 2023-05-31T21:12:28.233 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T21:12:28.233 DEBUG:boto:Port: 8000 2023-05-31T21:12:28.234 DEBUG:boto:Params: {} 2023-05-31T21:12:28.234 DEBUG:boto:Token: None 2023-05-31T21:12:28.234 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 21:12:28 GMT /pklxkd-64/ 2023-05-31T21:12:28.235 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:wRm+Nzs6+vQGxm2DWUEut4l9HJU= 2023-05-31T21:12:28.235 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 21:12:28 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:wRm+Nzs6+vQGxm2DWUEut4l9HJU=', 'Content-Length': '0'} 2023-05-31T21:12:28.238 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '29'), ('X-RGW-Bytes-Used', '87'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000d726c76c09455665-006477b83c-1630-a1'), ('Content-Length', '0'), ('Date', 'Wed, 31 May 2023 21:12:28 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:12:28.239 DEBUG:boto:path=/pklxkd-64/obj155 2023-05-31T21:12:28.239 DEBUG:boto:auth_path=/pklxkd-64/obj155 2023-05-31T21:12:28.239 DEBUG:boto:Method: PUT 2023-05-31T21:12:28.240 DEBUG:boto:Path: /pklxkd-64/obj155 2023-05-31T21:12:28.240 DEBUG:boto:Data: 2023-05-31T21:12:28.240 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-05-31T21:12:28.241 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T21:12:28.241 DEBUG:boto:Port: 8000 2023-05-31T21:12:28.242 DEBUG:boto:Params: {} 2023-05-31T21:12:28.242 DEBUG:boto:Token: None 2023-05-31T21:12:28.242 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Wed, 31 May 2023 21:12:28 GMT /pklxkd-64/obj155 2023-05-31T21:12:28.243 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:1rueDycE+KWjtf+YsP6oWag+EL8= 2023-05-31T21:12:28.243 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Wed, 31 May 2023 21:12:28 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:1rueDycE+KWjtf+YsP6oWag+EL8='} 2023-05-31T21:12:28.292 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000035038da524831a5e-006477b83c-1630-a1'), ('Date', 'Wed, 31 May 2023 21:12:28 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:12:28.293 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'reshard', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'pklxkd-64', '--num-shards', '37', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-05-31T21:12:28.293 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'reshard', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'pklxkd-64', '--num-shards', '37', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-05-31T21:12:28.294 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket reshard --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --bucket pklxkd-64 --num-shards 37 --yes-i-really-mean-it --cluster c1 --debug-rgw 0 --debug-ms 0 2023-05-31T21:12:28.346 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2023-05-31T21:12:28.347 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2023-05-31T21:12:28.426 INFO:teuthology.orchestra.run.smithi099.stdout:tenant: 2023-05-31T21:12:28.426 INFO:teuthology.orchestra.run.smithi099.stdout:bucket name: pklxkd-64 2023-05-31T21:12:28.437 INFO:teuthology.orchestra.run.smithi099.stdout:total entries: 30 2023-05-31T21:12:28.514 DEBUG:boto:path=/pklxkd-64/ 2023-05-31T21:12:28.515 DEBUG:boto:auth_path=/pklxkd-64/ 2023-05-31T21:12:28.515 DEBUG:boto:Method: HEAD 2023-05-31T21:12:28.516 DEBUG:boto:Path: /pklxkd-64/ 2023-05-31T21:12:28.516 DEBUG:boto:Data: 2023-05-31T21:12:28.516 DEBUG:boto:Headers: {} 2023-05-31T21:12:28.517 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T21:12:28.517 DEBUG:boto:Port: 8000 2023-05-31T21:12:28.517 DEBUG:boto:Params: {} 2023-05-31T21:12:28.518 DEBUG:boto:Token: None 2023-05-31T21:12:28.518 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 21:12:28 GMT /pklxkd-64/ 2023-05-31T21:12:28.519 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:wRm+Nzs6+vQGxm2DWUEut4l9HJU= 2023-05-31T21:12:28.519 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 21:12:28 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:wRm+Nzs6+vQGxm2DWUEut4l9HJU=', 'Content-Length': '0'} 2023-05-31T21:12:28.523 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '30'), ('X-RGW-Bytes-Used', '90'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000020b7b9eef9416f4b-006477b83c-1630-a1'), ('Content-Length', '0'), ('Date', 'Wed, 31 May 2023 21:12:28 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:12:28.523 DEBUG:boto:path=/pklxkd-64/obj37 2023-05-31T21:12:28.524 DEBUG:boto:auth_path=/pklxkd-64/obj37 2023-05-31T21:12:28.524 DEBUG:boto:Method: PUT 2023-05-31T21:12:28.525 DEBUG:boto:Path: /pklxkd-64/obj37 2023-05-31T21:12:28.525 DEBUG:boto:Data: 2023-05-31T21:12:28.525 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-05-31T21:12:28.526 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T21:12:28.526 DEBUG:boto:Port: 8000 2023-05-31T21:12:28.526 DEBUG:boto:Params: {} 2023-05-31T21:12:28.527 DEBUG:boto:Token: None 2023-05-31T21:12:28.527 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Wed, 31 May 2023 21:12:28 GMT /pklxkd-64/obj37 2023-05-31T21:12:28.528 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:NHXLlWggdAotn2hmkZiWiZKRvN4= 2023-05-31T21:12:28.528 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Wed, 31 May 2023 21:12:28 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:NHXLlWggdAotn2hmkZiWiZKRvN4='} 2023-05-31T21:12:28.534 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000002f914c6c1a1c0f4e-006477b83c-1630-a1'), ('Date', 'Wed, 31 May 2023 21:12:28 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:12:28.534 DEBUG:boto:path=/pklxkd-64/ 2023-05-31T21:12:28.534 DEBUG:boto:auth_path=/pklxkd-64/ 2023-05-31T21:12:28.535 DEBUG:boto:Method: HEAD 2023-05-31T21:12:28.535 DEBUG:boto:Path: /pklxkd-64/ 2023-05-31T21:12:28.535 DEBUG:boto:Data: 2023-05-31T21:12:28.536 DEBUG:boto:Headers: {} 2023-05-31T21:12:28.536 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T21:12:28.536 DEBUG:boto:Port: 8000 2023-05-31T21:12:28.537 DEBUG:boto:Params: {} 2023-05-31T21:12:28.537 DEBUG:boto:Token: None 2023-05-31T21:12:28.538 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 21:12:28 GMT /pklxkd-64/ 2023-05-31T21:12:28.538 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:wRm+Nzs6+vQGxm2DWUEut4l9HJU= 2023-05-31T21:12:28.538 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 21:12:28 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:wRm+Nzs6+vQGxm2DWUEut4l9HJU=', 'Content-Length': '0'} 2023-05-31T21:12:28.542 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '31'), ('X-RGW-Bytes-Used', '93'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000002d96a6b944dd8e88-006477b83c-1630-a1'), ('Content-Length', '0'), ('Date', 'Wed, 31 May 2023 21:12:28 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:12:28.542 DEBUG:boto:path=/pklxkd-64/obj74 2023-05-31T21:12:28.543 DEBUG:boto:auth_path=/pklxkd-64/obj74 2023-05-31T21:12:28.543 DEBUG:boto:Method: PUT 2023-05-31T21:12:28.543 DEBUG:boto:Path: /pklxkd-64/obj74 2023-05-31T21:12:28.544 DEBUG:boto:Data: 2023-05-31T21:12:28.544 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-05-31T21:12:28.545 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T21:12:28.545 DEBUG:boto:Port: 8000 2023-05-31T21:12:28.545 DEBUG:boto:Params: {} 2023-05-31T21:12:28.546 DEBUG:boto:Token: None 2023-05-31T21:12:28.546 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Wed, 31 May 2023 21:12:28 GMT /pklxkd-64/obj74 2023-05-31T21:12:28.546 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:JO1sVYCYBNr10jHyEESI41YkW4c= 2023-05-31T21:12:28.547 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Wed, 31 May 2023 21:12:28 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:JO1sVYCYBNr10jHyEESI41YkW4c='} 2023-05-31T21:12:28.588 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000089d7680e128596fa-006477b83c-1630-a1'), ('Date', 'Wed, 31 May 2023 21:12:28 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:12:28.589 DEBUG:boto:path=/pklxkd-64/ 2023-05-31T21:12:28.589 DEBUG:boto:auth_path=/pklxkd-64/ 2023-05-31T21:12:28.589 DEBUG:boto:Method: HEAD 2023-05-31T21:12:28.590 DEBUG:boto:Path: /pklxkd-64/ 2023-05-31T21:12:28.590 DEBUG:boto:Data: 2023-05-31T21:12:28.590 DEBUG:boto:Headers: {} 2023-05-31T21:12:28.591 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T21:12:28.591 DEBUG:boto:Port: 8000 2023-05-31T21:12:28.592 DEBUG:boto:Params: {} 2023-05-31T21:12:28.592 DEBUG:boto:Token: None 2023-05-31T21:12:28.593 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 21:12:28 GMT /pklxkd-64/ 2023-05-31T21:12:28.593 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:wRm+Nzs6+vQGxm2DWUEut4l9HJU= 2023-05-31T21:12:28.593 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 21:12:28 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:wRm+Nzs6+vQGxm2DWUEut4l9HJU=', 'Content-Length': '0'} 2023-05-31T21:12:28.597 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '32'), ('X-RGW-Bytes-Used', '96'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000177586ae6be493a7-006477b83c-1630-a1'), ('Content-Length', '0'), ('Date', 'Wed, 31 May 2023 21:12:28 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:12:28.597 DEBUG:boto:path=/pklxkd-64/obj111 2023-05-31T21:12:28.598 DEBUG:boto:auth_path=/pklxkd-64/obj111 2023-05-31T21:12:28.598 DEBUG:boto:Method: PUT 2023-05-31T21:12:28.598 DEBUG:boto:Path: /pklxkd-64/obj111 2023-05-31T21:12:28.599 DEBUG:boto:Data: 2023-05-31T21:12:28.599 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-05-31T21:12:28.599 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T21:12:28.600 DEBUG:boto:Port: 8000 2023-05-31T21:12:28.600 DEBUG:boto:Params: {} 2023-05-31T21:12:28.600 DEBUG:boto:Token: None 2023-05-31T21:12:28.601 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Wed, 31 May 2023 21:12:28 GMT /pklxkd-64/obj111 2023-05-31T21:12:28.601 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:PQYRTSWgOHNBCS+sq/d7wkr7tWE= 2023-05-31T21:12:28.602 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Wed, 31 May 2023 21:12:28 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:PQYRTSWgOHNBCS+sq/d7wkr7tWE='} 2023-05-31T21:12:28.608 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000000ab4af260ddb5f9d-006477b83c-1630-a1'), ('Date', 'Wed, 31 May 2023 21:12:28 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:12:28.609 DEBUG:boto:path=/pklxkd-64/ 2023-05-31T21:12:28.609 DEBUG:boto:auth_path=/pklxkd-64/ 2023-05-31T21:12:28.610 DEBUG:boto:Method: HEAD 2023-05-31T21:12:28.610 DEBUG:boto:Path: /pklxkd-64/ 2023-05-31T21:12:28.610 DEBUG:boto:Data: 2023-05-31T21:12:28.611 DEBUG:boto:Headers: {} 2023-05-31T21:12:28.611 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T21:12:28.611 DEBUG:boto:Port: 8000 2023-05-31T21:12:28.612 DEBUG:boto:Params: {} 2023-05-31T21:12:28.612 DEBUG:boto:Token: None 2023-05-31T21:12:28.613 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 21:12:28 GMT /pklxkd-64/ 2023-05-31T21:12:28.613 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:wRm+Nzs6+vQGxm2DWUEut4l9HJU= 2023-05-31T21:12:28.613 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 21:12:28 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:wRm+Nzs6+vQGxm2DWUEut4l9HJU=', 'Content-Length': '0'} 2023-05-31T21:12:28.616 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '33'), ('X-RGW-Bytes-Used', '99'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000c162c2186fe1b9c9-006477b83c-1630-a1'), ('Content-Length', '0'), ('Date', 'Wed, 31 May 2023 21:12:28 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:12:28.617 DEBUG:boto:path=/pklxkd-64/obj148 2023-05-31T21:12:28.617 DEBUG:boto:auth_path=/pklxkd-64/obj148 2023-05-31T21:12:28.618 DEBUG:boto:Method: PUT 2023-05-31T21:12:28.618 DEBUG:boto:Path: /pklxkd-64/obj148 2023-05-31T21:12:28.618 DEBUG:boto:Data: 2023-05-31T21:12:28.619 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-05-31T21:12:28.619 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T21:12:28.619 DEBUG:boto:Port: 8000 2023-05-31T21:12:28.620 DEBUG:boto:Params: {} 2023-05-31T21:12:28.620 DEBUG:boto:Token: None 2023-05-31T21:12:28.620 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Wed, 31 May 2023 21:12:28 GMT /pklxkd-64/obj148 2023-05-31T21:12:28.621 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:h5bJfi8JUN74fr/t6Qjb6Vk4sUM= 2023-05-31T21:12:28.621 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Wed, 31 May 2023 21:12:28 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:h5bJfi8JUN74fr/t6Qjb6Vk4sUM='} 2023-05-31T21:12:28.664 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000a5f187110ff17551-006477b83c-1630-a1'), ('Date', 'Wed, 31 May 2023 21:12:28 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:12:28.665 DEBUG:boto:path=/pklxkd-64/ 2023-05-31T21:12:28.665 DEBUG:boto:auth_path=/pklxkd-64/ 2023-05-31T21:12:28.666 DEBUG:boto:Method: HEAD 2023-05-31T21:12:28.666 DEBUG:boto:Path: /pklxkd-64/ 2023-05-31T21:12:28.666 DEBUG:boto:Data: 2023-05-31T21:12:28.667 DEBUG:boto:Headers: {} 2023-05-31T21:12:28.667 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T21:12:28.667 DEBUG:boto:Port: 8000 2023-05-31T21:12:28.668 DEBUG:boto:Params: {} 2023-05-31T21:12:28.668 DEBUG:boto:Token: None 2023-05-31T21:12:28.668 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 21:12:28 GMT /pklxkd-64/ 2023-05-31T21:12:28.669 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:wRm+Nzs6+vQGxm2DWUEut4l9HJU= 2023-05-31T21:12:28.669 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 21:12:28 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:wRm+Nzs6+vQGxm2DWUEut4l9HJU=', 'Content-Length': '0'} 2023-05-31T21:12:28.672 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '34'), ('X-RGW-Bytes-Used', '102'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000d2da6c89b8b166a9-006477b83c-1630-a1'), ('Content-Length', '0'), ('Date', 'Wed, 31 May 2023 21:12:28 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:12:28.673 DEBUG:boto:path=/pklxkd-64/obj185 2023-05-31T21:12:28.673 DEBUG:boto:auth_path=/pklxkd-64/obj185 2023-05-31T21:12:28.674 DEBUG:boto:Method: PUT 2023-05-31T21:12:28.674 DEBUG:boto:Path: /pklxkd-64/obj185 2023-05-31T21:12:28.674 DEBUG:boto:Data: 2023-05-31T21:12:28.675 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-05-31T21:12:28.675 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T21:12:28.675 DEBUG:boto:Port: 8000 2023-05-31T21:12:28.676 DEBUG:boto:Params: {} 2023-05-31T21:12:28.676 DEBUG:boto:Token: None 2023-05-31T21:12:28.677 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Wed, 31 May 2023 21:12:28 GMT /pklxkd-64/obj185 2023-05-31T21:12:28.677 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:Y2SdKSit78jNOubvBZare6NJX2I= 2023-05-31T21:12:28.678 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Wed, 31 May 2023 21:12:28 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:Y2SdKSit78jNOubvBZare6NJX2I='} 2023-05-31T21:12:28.687 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000f7b93d23e5d8e8a4-006477b83c-1630-a1'), ('Date', 'Wed, 31 May 2023 21:12:28 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:12:28.687 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'init', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-05-31T21:12:28.688 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'init', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-05-31T21:12:28.688 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync init --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 2023-05-31T21:12:28.738 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2023-05-31T21:12:28.739 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2023-05-31T21:12:28.792 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T21:12:28.787+0000 7f823c58b780 5 RGW-SYNC:data: waiting on data sync status lease 2023-05-31T21:12:28.793 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T21:12:28.788+0000 7f823c58b780 5 RGW-SYNC:data: acquired data sync status lease 2023-05-31T21:12:28.834 INFO:tasks.rgw.c1.client.1:Restarting daemon 2023-05-31T21:12:28.835 DEBUG:teuthology.orchestra.run.smithi099:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term radosgw --rgw-frontends 'beast port=8001' -n client.1 --cluster c1 -k /etc/ceph/c1.client.1.keyring --log-file /var/log/ceph/rgw.c1.client.1.log --rgw_ops_log_socket_path /home/ubuntu/cephtest/rgw.opslog.c1.client.1.sock --foreground --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm | sudo tee /var/log/ceph/rgw.c1.client.1.stdout 2>&1 2023-05-31T21:12:28.837 INFO:tasks.rgw.c1.client.1:Started 2023-05-31T21:12:28.837 DEBUG:teuthology.orchestra.run.smithi099:> curl http://smithi099.front.sepia.ceph.com:8001 2023-05-31T21:12:28.857 INFO:teuthology.orchestra.run.smithi099.stderr: % Total % Received % Xferd Average Speed Time Time Time Current 2023-05-31T21:12:28.858 INFO:teuthology.orchestra.run.smithi099.stderr: Dload Upload Total Spent Left Speed 2023-05-31T21:12:28.858 DEBUG:teuthology.orchestra.run:got remote process result: 7 2023-05-31T21:12:28.859 INFO:teuthology.orchestra.run.smithi099.stderr: 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0curl: (7) Failed to connect to smithi099.front.sepia.ceph.com port 8001: Connection refused 2023-05-31T21:12:29.860 DEBUG:teuthology.orchestra.run.smithi099:> curl http://smithi099.front.sepia.ceph.com:8001 2023-05-31T21:12:29.881 INFO:teuthology.orchestra.run.smithi099.stderr: % Total % Received % Xferd Average Speed Time Time Time Current 2023-05-31T21:12:29.882 INFO:teuthology.orchestra.run.smithi099.stderr: Dload Upload Total Spent Left Speed 2023-05-31T21:12:29.882 INFO:teuthology.orchestra.run.smithi099.stderr: 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 100 214 0 214 0 0 208k 0 --:--:-- --:--:-- --:--:-- 208k 2023-05-31T21:12:29.883 INFO:teuthology.orchestra.run.smithi099.stdout:anonymous 2023-05-31T21:12:29.883 DEBUG:rgw_multi.tests:bucket checkpoint: source=a1 target=a2 bucket=pklxkd-64 2023-05-31T21:12:29.884 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'pklxkd-64', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-05-31T21:12:29.884 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'pklxkd-64', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-05-31T21:12:29.885 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket pklxkd-64 --source-zone a1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2023-05-31T21:12:29.979 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2023-05-31T21:12:29.979 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2023-05-31T21:12:30.024 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T21:12:30.020+0000 7f951157b780 1 waiting to reach latest gen 6, on 0.. 2023-05-31T21:12:30.394 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.1 has been restored 2023-05-31T21:12:35.024 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T21:12:35.020+0000 7f951157b780 1 waiting to reach latest gen 6, on 0.. 2023-05-31T21:12:40.025 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T21:12:40.021+0000 7f951157b780 1 waiting to reach latest gen 6, on 0.. 2023-05-31T21:12:45.026 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T21:12:45.022+0000 7f951157b780 1 waiting to reach latest gen 6, on 0.. 2023-05-31T21:12:50.026 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T21:12:50.023+0000 7f951157b780 1 waiting to reach latest gen 6, on 0.. 2023-05-31T21:12:55.027 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T21:12:55.024+0000 7f951157b780 1 waiting to reach latest gen 6, on 0.. 2023-05-31T21:13:00.028 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T21:13:00.024+0000 7f951157b780 1 waiting to reach latest gen 6, on 0.. 2023-05-31T21:13:05.028 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T21:13:05.024+0000 7f951157b780 1 waiting to reach latest gen 6, on 0.. 2023-05-31T21:13:10.029 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T21:13:10.024+0000 7f951157b780 1 waiting to reach latest gen 6, on 0.. 2023-05-31T21:13:15.030 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T21:13:15.025+0000 7f951157b780 1 waiting to reach latest gen 6, on 0.. 2023-05-31T21:13:20.031 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T21:13:20.026+0000 7f951157b780 1 waiting to reach latest gen 6, on 0.. 2023-05-31T21:13:25.031 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T21:13:25.027+0000 7f951157b780 1 waiting to reach latest gen 6, on 0.. 2023-05-31T21:13:30.032 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T21:13:30.028+0000 7f951157b780 1 waiting to reach latest gen 6, on 0.. 2023-05-31T21:13:35.033 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T21:13:35.029+0000 7f951157b780 1 waiting to reach latest gen 6, on 0.. 2023-05-31T21:13:40.033 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T21:13:40.030+0000 7f951157b780 1 waiting to reach latest gen 6, on 0.. 2023-05-31T21:13:45.034 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T21:13:45.030+0000 7f951157b780 1 waiting to reach latest gen 6, on 0.. 2023-05-31T21:13:50.034 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T21:13:50.030+0000 7f951157b780 1 waiting to reach latest gen 6, on 0.. 2023-05-31T21:13:55.035 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T21:13:55.030+0000 7f951157b780 1 waiting to reach latest gen 6, on 0.. 2023-05-31T21:14:00.035 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T21:14:00.031+0000 7f951157b780 1 waiting to reach latest gen 6, on 0.. 2023-05-31T21:14:05.036 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T21:14:05.032+0000 7f951157b780 1 waiting to reach latest gen 6, on 2.. 2023-05-31T21:14:10.037 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T21:14:10.033+0000 7f951157b780 1 waiting to reach latest gen 6, on 2.. 2023-05-31T21:14:15.037 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T21:14:15.034+0000 7f951157b780 1 waiting to reach latest gen 6, on 2.. 2023-05-31T21:14:20.038 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T21:14:20.035+0000 7f951157b780 1 waiting to reach latest gen 6, on 2.. 2023-05-31T21:14:25.039 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T21:14:25.035+0000 7f951157b780 1 waiting to reach latest gen 6, on 2.. 2023-05-31T21:14:30.039 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T21:14:30.035+0000 7f951157b780 1 waiting to reach latest gen 6, on 2.. 2023-05-31T21:14:35.040 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T21:14:35.036+0000 7f951157b780 1 waiting to reach latest gen 6, on 2.. 2023-05-31T21:14:40.041 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T21:14:40.036+0000 7f951157b780 1 waiting to reach latest gen 6, on 2.. 2023-05-31T21:14:45.041 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T21:14:45.037+0000 7f951157b780 1 waiting to reach latest gen 6, on 2.. 2023-05-31T21:14:50.042 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T21:14:50.039+0000 7f951157b780 1 waiting to reach latest gen 6, on 2.. 2023-05-31T21:14:55.043 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T21:14:55.039+0000 7f951157b780 1 waiting to reach latest gen 6, on 2.. 2023-05-31T21:15:00.043 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T21:15:00.040+0000 7f951157b780 1 waiting to reach latest gen 6, on 2.. 2023-05-31T21:15:05.044 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T21:15:05.040+0000 7f951157b780 1 waiting to reach latest gen 6, on 2.. 2023-05-31T21:15:10.044 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T21:15:10.040+0000 7f951157b780 1 waiting to reach latest gen 6, on 2.. 2023-05-31T21:15:15.045 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T21:15:15.040+0000 7f951157b780 1 waiting to reach latest gen 6, on 2.. 2023-05-31T21:15:20.046 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T21:15:20.041+0000 7f951157b780 1 waiting to reach latest gen 6, on 2.. 2023-05-31T21:15:25.047 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T21:15:25.042+0000 7f951157b780 1 waiting to reach latest gen 6, on 2.. 2023-05-31T21:15:30.047 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T21:15:30.043+0000 7f951157b780 1 waiting to reach latest gen 6, on 2.. 2023-05-31T21:15:35.048 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T21:15:35.044+0000 7f951157b780 1 waiting to reach latest gen 6, on 2.. 2023-05-31T21:15:40.048 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T21:15:40.045+0000 7f951157b780 1 waiting to reach latest gen 6, on 2.. 2023-05-31T21:15:45.049 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T21:15:45.046+0000 7f951157b780 1 waiting to reach latest gen 6, on 2.. 2023-05-31T21:15:50.050 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T21:15:50.046+0000 7f951157b780 1 waiting to reach latest gen 6, on 2.. 2023-05-31T21:15:55.050 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T21:15:55.046+0000 7f951157b780 1 waiting to reach latest gen 6, on 2.. 2023-05-31T21:16:00.051 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T21:16:00.047+0000 7f951157b780 1 waiting to reach latest gen 6, on 2.. 2023-05-31T21:16:05.052 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T21:16:05.048+0000 7f951157b780 1 waiting to reach latest gen 6, on 2.. 2023-05-31T21:16:10.053 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T21:16:10.049+0000 7f951157b780 1 waiting to reach latest gen 6, on 2.. 2023-05-31T21:16:15.053 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T21:16:15.050+0000 7f951157b780 1 waiting to reach latest gen 6, on 2.. 2023-05-31T21:16:20.054 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T21:16:20.051+0000 7f951157b780 1 waiting to reach latest gen 6, on 2.. 2023-05-31T21:16:25.055 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T21:16:25.052+0000 7f951157b780 1 waiting to reach latest gen 6, on 2.. 2023-05-31T21:16:30.056 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T21:16:30.052+0000 7f951157b780 1 waiting to reach latest gen 6, on 2.. 2023-05-31T21:16:35.057 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T21:16:35.052+0000 7f951157b780 1 waiting to reach latest gen 6, on 2.. 2023-05-31T21:16:40.057 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T21:16:40.053+0000 7f951157b780 1 waiting to reach latest gen 6, on 2.. 2023-05-31T21:16:45.058 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T21:16:45.054+0000 7f951157b780 1 waiting to reach latest gen 6, on 2.. 2023-05-31T21:16:50.058 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T21:16:50.054+0000 7f951157b780 1 waiting to reach latest gen 6, on 2.. 2023-05-31T21:16:55.059 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T21:16:55.055+0000 7f951157b780 1 waiting to reach latest gen 6, on 2.. 2023-05-31T21:17:00.060 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T21:17:00.056+0000 7f951157b780 1 waiting to reach latest gen 6, on 2.. 2023-05-31T21:17:05.060 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T21:17:05.057+0000 7f951157b780 1 waiting to reach latest gen 6, on 2.. 2023-05-31T21:17:10.061 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T21:17:10.057+0000 7f951157b780 1 waiting to reach latest gen 6, on 2.. 2023-05-31T21:17:15.062 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T21:17:15.058+0000 7f951157b780 1 waiting to reach latest gen 6, on 2.. 2023-05-31T21:17:20.062 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T21:17:20.058+0000 7f951157b780 1 waiting to reach latest gen 6, on 2.. 2023-05-31T21:17:25.063 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T21:17:25.059+0000 7f951157b780 -1 bucket checkpoint timed out waiting to reach latest generation 6 2023-05-31T21:17:25.063 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T21:17:25.059+0000 7f951157b780 0 bucket sync checkpoint failed: (110) Connection timed out 2023-05-31T21:17:25.064 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T21:17:25.059+0000 7f951157b780 -1 bucket sync checkpoint failed: (110) Connection timed out 2023-05-31T21:17:25.084 DEBUG:teuthology.orchestra.run:got remote process result: 110 2023-05-31T21:17:25.087 INFO:tasks.rgw_multisite_tests:rgw_multi.tests.test_resharded_bucket_incremental_sync_oldest_after_data_sync_init ... FAIL 2023-05-31T21:17:25.089 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-31T21:17:25.089 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-31T21:17:25.090 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-05-31T21:17:25.145 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2023-05-31T21:17:25.146 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2023-05-31T21:17:25.207 INFO:teuthology.orchestra.run.smithi099.stderr:No --period given, using current period=2e6123d2-d695-4350-9a01-61a455bc782b 2023-05-31T21:17:25.224 INFO:teuthology.orchestra.run.smithi099.stdout:[ 2023-05-31T21:17:25.224 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T21:17:25.225 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1685566501.757183_260.1", 2023-05-31T21:17:25.225 INFO:teuthology.orchestra.run.smithi099.stdout: "last_update": "2023-05-31T20:55:01.757183Z" 2023-05-31T21:17:25.225 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:17:25.225 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T21:17:25.225 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1685566501.735717_251.1", 2023-05-31T21:17:25.225 INFO:teuthology.orchestra.run.smithi099.stdout: "last_update": "2023-05-31T20:55:01.735717Z" 2023-05-31T21:17:25.225 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:17:25.226 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T21:17:25.226 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1685562970.274572_93.1", 2023-05-31T21:17:25.226 INFO:teuthology.orchestra.run.smithi099.stdout: "last_update": "2023-05-31T19:56:10.274572Z" 2023-05-31T21:17:25.226 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:17:25.226 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T21:17:25.226 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1685567548.445367_776.1", 2023-05-31T21:17:25.226 INFO:teuthology.orchestra.run.smithi099.stdout: "last_update": "2023-05-31T21:12:28.445367Z" 2023-05-31T21:17:25.227 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T21:17:25.227 INFO:teuthology.orchestra.run.smithi099.stdout:] 2023-05-31T21:17:25.228 DEBUG:rgw_multi.tests:master meta markers={0: '1_1685566501.757183_260.1', 1: '1_1685566501.735717_251.1', 2: '1_1685562970.274572_93.1', 3: '1_1685567548.445367_776.1'} 2023-05-31T21:17:25.228 INFO:rgw_multi.tests:starting meta checkpoint for zone=a2 2023-05-31T21:17:25.229 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-31T21:17:25.229 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-31T21:17:25.229 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-05-31T21:17:25.278 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2023-05-31T21:17:25.279 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2023-05-31T21:17:25.310 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T21:17:25.305+0000 7f6977068780 20 RGW-SYNC:meta: read sync status 2023-05-31T21:17:25.310 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T21:17:25.306+0000 7f6977068780 20 RGW-SYNC:meta: read sync status 2023-05-31T21:17:25.324 INFO:teuthology.orchestra.run.smithi099.stdout:{ 2023-05-31T21:17:25.324 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_status": { 2023-05-31T21:17:25.324 INFO:teuthology.orchestra.run.smithi099.stdout: "info": { 2023-05-31T21:17:25.325 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "sync", 2023-05-31T21:17:25.325 INFO:teuthology.orchestra.run.smithi099.stdout: "num_shards": 4, 2023-05-31T21:17:25.325 INFO:teuthology.orchestra.run.smithi099.stdout: "period": "2e6123d2-d695-4350-9a01-61a455bc782b", 2023-05-31T21:17:25.325 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2023-05-31T21:17:25.325 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:17:25.325 INFO:teuthology.orchestra.run.smithi099.stdout: "markers": [ 2023-05-31T21:17:25.326 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T21:17:25.326 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 0, 2023-05-31T21:17:25.326 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T21:17:25.326 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2023-05-31T21:17:25.326 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1685566501.757183_260.1", 2023-05-31T21:17:25.326 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T21:17:25.326 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 1, 2023-05-31T21:17:25.327 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T21:17:25.327 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2023-05-31T20:55:01.757183Z", 2023-05-31T21:17:25.327 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2023-05-31T21:17:25.327 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T21:17:25.327 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:17:25.327 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T21:17:25.327 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 1, 2023-05-31T21:17:25.328 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T21:17:25.328 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2023-05-31T21:17:25.328 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1685566501.735717_251.1", 2023-05-31T21:17:25.328 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T21:17:25.328 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2023-05-31T21:17:25.328 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T21:17:25.329 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2023-05-31T20:55:01.735717Z", 2023-05-31T21:17:25.329 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2023-05-31T21:17:25.329 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T21:17:25.329 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:17:25.329 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T21:17:25.329 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 2, 2023-05-31T21:17:25.329 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T21:17:25.330 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2023-05-31T21:17:25.330 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1685562970.274572_93.1", 2023-05-31T21:17:25.330 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T21:17:25.330 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2023-05-31T21:17:25.330 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T21:17:25.330 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2023-05-31T19:56:10.274572Z", 2023-05-31T21:17:25.330 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2023-05-31T21:17:25.331 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T21:17:25.331 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:17:25.331 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T21:17:25.331 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 3, 2023-05-31T21:17:25.331 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T21:17:25.331 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2023-05-31T21:17:25.331 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1685567548.445367_776.1", 2023-05-31T21:17:25.332 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T21:17:25.332 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2023-05-31T21:17:25.332 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T21:17:25.332 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2023-05-31T21:12:28.445367Z", 2023-05-31T21:17:25.332 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2023-05-31T21:17:25.332 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T21:17:25.333 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T21:17:25.333 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2023-05-31T21:17:25.333 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:17:25.333 INFO:teuthology.orchestra.run.smithi099.stdout: "full_sync": { 2023-05-31T21:17:25.333 INFO:teuthology.orchestra.run.smithi099.stdout: "total": 1, 2023-05-31T21:17:25.333 INFO:teuthology.orchestra.run.smithi099.stdout: "complete": 1 2023-05-31T21:17:25.333 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:17:25.334 INFO:teuthology.orchestra.run.smithi099.stdout: "current_time": "2023-05-31T21:17:25Z" 2023-05-31T21:17:25.334 INFO:teuthology.orchestra.run.smithi099.stdout:} 2023-05-31T21:17:25.334 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "2e6123d2-d695-4350-9a01-61a455bc782b", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1685566501.757183_260.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-05-31T20:55:01.757183Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1685566501.735717_251.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-05-31T20:55:01.735717Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1685562970.274572_93.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-05-31T19:56:10.274572Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1685567548.445367_776.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-05-31T21:12:28.445367Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-05-31T21:17:25Z" } 2023-05-31T21:17:25.335 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1685566501.757183_260.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-05-31T20:55:01.757183Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1685566501.735717_251.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-05-31T20:55:01.735717Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1685562970.274572_93.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-05-31T19:56:10.274572Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1685567548.445367_776.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-05-31T21:12:28.445367Z', 'realm_epoch': 2}}] 2023-05-31T21:17:25.335 DEBUG:rgw_multi.tests:log_status={0: '1_1685566501.757183_260.1', 1: '1_1685566501.735717_251.1', 2: '1_1685562970.274572_93.1', 3: '1_1685567548.445367_776.1'} 2023-05-31T21:17:25.336 DEBUG:rgw_multi.tests:sync_status={0: '1_1685566501.757183_260.1', 1: '1_1685566501.735717_251.1', 2: '1_1685562970.274572_93.1', 3: '1_1685567548.445367_776.1'} 2023-05-31T21:17:25.336 INFO:rgw_multi.tests:finish meta checkpoint for zone=a2 2023-05-31T21:17:25.337 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'policy', 'get', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-05-31T21:17:25.337 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'policy', 'get', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-05-31T21:17:25.338 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync policy get --cluster c1 --debug-rgw 0 --debug-ms 0 2023-05-31T21:17:25.383 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2023-05-31T21:17:25.383 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2023-05-31T21:17:25.448 INFO:teuthology.orchestra.run.smithi099.stdout:{"groups":[]} 2023-05-31T21:17:25.448 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'group', 'create', '--group-id', 'sync-group', '--status', 'allowed', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-05-31T21:17:25.449 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'group', 'create', '--group-id', 'sync-group', '--status', 'allowed', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-05-31T21:17:25.449 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync group create --group-id sync-group --status allowed --cluster c1 --debug-rgw 0 --debug-ms 0 2023-05-31T21:17:25.498 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2023-05-31T21:17:25.498 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2023-05-31T21:17:25.582 INFO:teuthology.orchestra.run.smithi099.stdout:{"groups":[{"id":"sync-group","data_flow":{},"pipes":[],"status":"allowed"}]} 2023-05-31T21:17:25.583 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'group', 'modify', '--group-id', 'sync-group', '--status', 'enabled', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-05-31T21:17:25.583 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'group', 'modify', '--group-id', 'sync-group', '--status', 'enabled', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-05-31T21:17:25.584 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync group modify --group-id sync-group --status enabled --cluster c1 --debug-rgw 0 --debug-ms 0 2023-05-31T21:17:25.633 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2023-05-31T21:17:25.633 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2023-05-31T21:17:25.684 INFO:teuthology.orchestra.run.smithi099.stdout:{"groups":[{"id":"sync-group","data_flow":{},"pipes":[],"status":"enabled"}]} 2023-05-31T21:17:25.685 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'group', 'get', '--group-id', 'sync-group', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-05-31T21:17:25.685 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'group', 'get', '--group-id', 'sync-group', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-05-31T21:17:25.685 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync group get --group-id sync-group --cluster c1 --debug-rgw 0 --debug-ms 0 2023-05-31T21:17:25.733 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2023-05-31T21:17:25.733 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2023-05-31T21:17:25.787 INFO:teuthology.orchestra.run.smithi099.stdout:{"id":"sync-group","data_flow":{},"pipes":[],"status":"enabled"} 2023-05-31T21:17:25.788 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'policy', 'get', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-05-31T21:17:25.788 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'policy', 'get', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-05-31T21:17:25.789 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync policy get --cluster c1 --debug-rgw 0 --debug-ms 0 2023-05-31T21:17:25.834 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2023-05-31T21:17:25.834 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2023-05-31T21:17:25.892 INFO:teuthology.orchestra.run.smithi099.stdout:{"groups":[{"id":"sync-group","data_flow":{},"pipes":[],"status":"enabled"}]} 2023-05-31T21:17:25.893 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'group', 'flow', 'create', '--group-id', 'sync-group', '--flow-id', 'sync-flow1', '--flow-type', 'symmetrical', '--zones=a1,a2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-05-31T21:17:25.893 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'group', 'flow', 'create', '--group-id', 'sync-group', '--flow-id', 'sync-flow1', '--flow-type', 'symmetrical', '--zones=a1,a2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-05-31T21:17:25.894 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync group flow create --group-id sync-group --flow-id sync-flow1 --flow-type symmetrical --zones=a1,a2 --cluster c1 --debug-rgw 0 --debug-ms 0 2023-05-31T21:17:25.947 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2023-05-31T21:17:25.947 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2023-05-31T21:17:26.001 INFO:teuthology.orchestra.run.smithi099.stdout:{"groups":[{"id":"sync-group","data_flow":{"symmetrical":[{"id":"sync-flow1","zones":["a1","a2"]}]},"pipes":[],"status":"enabled"}]} 2023-05-31T21:17:26.001 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'group', 'flow', 'create', '--group-id', 'sync-group', '--flow-id', 'sync-flow2', '--flow-type', 'directional', '--source-zone=a1', '--dest-zone=a2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-05-31T21:17:26.002 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'group', 'flow', 'create', '--group-id', 'sync-group', '--flow-id', 'sync-flow2', '--flow-type', 'directional', '--source-zone=a1', '--dest-zone=a2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-05-31T21:17:26.002 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync group flow create --group-id sync-group --flow-id sync-flow2 --flow-type directional --source-zone=a1 --dest-zone=a2 --cluster c1 --debug-rgw 0 --debug-ms 0 2023-05-31T21:17:26.056 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2023-05-31T21:17:26.057 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2023-05-31T21:17:26.122 INFO:teuthology.orchestra.run.smithi099.stdout:{"groups":[{"id":"sync-group","data_flow":{"symmetrical":[{"id":"sync-flow1","zones":["a1","a2"]}],"directional":[{"source_zone":"a1","dest_zone":"a2"}]},"pipes":[],"status":"enabled"}]} 2023-05-31T21:17:26.122 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'group', 'pipe', 'create', '--group-id', 'sync-group', '--pipe-id', 'sync-pipe', '--source-zones=a1,a2', '--dest-zones=a1,a2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-05-31T21:17:26.123 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'group', 'pipe', 'create', '--group-id', 'sync-group', '--pipe-id', 'sync-pipe', '--source-zones=a1,a2', '--dest-zones=a1,a2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-05-31T21:17:26.123 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync group pipe create --group-id sync-group --pipe-id sync-pipe --source-zones=a1,a2 --dest-zones=a1,a2 --cluster c1 --debug-rgw 0 --debug-ms 0 2023-05-31T21:17:26.176 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2023-05-31T21:17:26.176 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2023-05-31T21:17:26.230 INFO:teuthology.orchestra.run.smithi099.stdout:{"groups":[{"id":"sync-group","data_flow":{"symmetrical":[{"id":"sync-flow1","zones":["a1","a2"]}],"directional":[{"source_zone":"a1","dest_zone":"a2"}]},"pipes":[{"id":"sync-pipe","source":{"bucket":"*","zones":["a1","a2"]},"dest":{"bucket":"*","zones":["a1","a2"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":""}}],"status":"enabled"}]} 2023-05-31T21:17:26.231 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'group', 'get', '--group-id', 'sync-group', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-05-31T21:17:26.231 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'group', 'get', '--group-id', 'sync-group', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-05-31T21:17:26.232 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync group get --group-id sync-group --cluster c1 --debug-rgw 0 --debug-ms 0 2023-05-31T21:17:26.278 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2023-05-31T21:17:26.278 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2023-05-31T21:17:26.336 INFO:teuthology.orchestra.run.smithi099.stdout:{"id":"sync-group","data_flow":{"symmetrical":[{"id":"sync-flow1","zones":["a1","a2"]}],"directional":[{"source_zone":"a1","dest_zone":"a2"}]},"pipes":[{"id":"sync-pipe","source":{"bucket":"*","zones":["a1","a2"]},"dest":{"bucket":"*","zones":["a1","a2"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":""}}],"status":"enabled"} 2023-05-31T21:17:26.336 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['period', 'update', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-05-31T21:17:26.337 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'period', 'update', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-05-31T21:17:26.337 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 period update --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --commit --cluster c1 --debug-rgw 0 --debug-ms 0 2023-05-31T21:17:26.385 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2023-05-31T21:17:26.385 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2023-05-31T21:17:26.455 INFO:teuthology.orchestra.run.smithi099.stdout:{ 2023-05-31T21:17:26.456 INFO:teuthology.orchestra.run.smithi099.stdout: "id": "2e6123d2-d695-4350-9a01-61a455bc782b", 2023-05-31T21:17:26.456 INFO:teuthology.orchestra.run.smithi099.stdout: "epoch": 13, 2023-05-31T21:17:26.456 INFO:teuthology.orchestra.run.smithi099.stdout: "predecessor_uuid": "629b9ced-80af-41ed-8a6c-f2cfc334cbb4", 2023-05-31T21:17:26.456 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_status": [], 2023-05-31T21:17:26.456 INFO:teuthology.orchestra.run.smithi099.stdout: "period_map": { 2023-05-31T21:17:26.456 INFO:teuthology.orchestra.run.smithi099.stdout: "id": "2e6123d2-d695-4350-9a01-61a455bc782b", 2023-05-31T21:17:26.457 INFO:teuthology.orchestra.run.smithi099.stdout: "zonegroups": [ 2023-05-31T21:17:26.457 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T21:17:26.457 INFO:teuthology.orchestra.run.smithi099.stdout: "id": "a", 2023-05-31T21:17:26.457 INFO:teuthology.orchestra.run.smithi099.stdout: "name": "a", 2023-05-31T21:17:26.457 INFO:teuthology.orchestra.run.smithi099.stdout: "api_name": "", 2023-05-31T21:17:26.457 INFO:teuthology.orchestra.run.smithi099.stdout: "is_master": true, 2023-05-31T21:17:26.458 INFO:teuthology.orchestra.run.smithi099.stdout: "endpoints": [ 2023-05-31T21:17:26.458 INFO:teuthology.orchestra.run.smithi099.stdout: "http://smithi099.front.sepia.ceph.com:8000" 2023-05-31T21:17:26.458 INFO:teuthology.orchestra.run.smithi099.stdout: ], 2023-05-31T21:17:26.458 INFO:teuthology.orchestra.run.smithi099.stdout: "hostnames": [], 2023-05-31T21:17:26.458 INFO:teuthology.orchestra.run.smithi099.stdout: "hostnames_s3website": [], 2023-05-31T21:17:26.458 INFO:teuthology.orchestra.run.smithi099.stdout: "master_zone": "3d163b1e-09c7-47e2-ae87-add4aace3f38", 2023-05-31T21:17:26.458 INFO:teuthology.orchestra.run.smithi099.stdout: "zones": [ 2023-05-31T21:17:26.459 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T21:17:26.459 INFO:teuthology.orchestra.run.smithi099.stdout: "id": "3d163b1e-09c7-47e2-ae87-add4aace3f38", 2023-05-31T21:17:26.459 INFO:teuthology.orchestra.run.smithi099.stdout: "name": "a1", 2023-05-31T21:17:26.459 INFO:teuthology.orchestra.run.smithi099.stdout: "endpoints": [ 2023-05-31T21:17:26.459 INFO:teuthology.orchestra.run.smithi099.stdout: "http://smithi099.front.sepia.ceph.com:8000" 2023-05-31T21:17:26.459 INFO:teuthology.orchestra.run.smithi099.stdout: ], 2023-05-31T21:17:26.459 INFO:teuthology.orchestra.run.smithi099.stdout: "log_meta": false, 2023-05-31T21:17:26.460 INFO:teuthology.orchestra.run.smithi099.stdout: "log_data": true, 2023-05-31T21:17:26.460 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket_index_max_shards": 11, 2023-05-31T21:17:26.460 INFO:teuthology.orchestra.run.smithi099.stdout: "read_only": false, 2023-05-31T21:17:26.460 INFO:teuthology.orchestra.run.smithi099.stdout: "tier_type": "", 2023-05-31T21:17:26.460 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_from_all": true, 2023-05-31T21:17:26.460 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_from": [], 2023-05-31T21:17:26.461 INFO:teuthology.orchestra.run.smithi099.stdout: "redirect_zone": "", 2023-05-31T21:17:26.461 INFO:teuthology.orchestra.run.smithi099.stdout: "supported_features": [ 2023-05-31T21:17:26.461 INFO:teuthology.orchestra.run.smithi099.stdout: "resharding" 2023-05-31T21:17:26.461 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2023-05-31T21:17:26.461 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:17:26.461 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T21:17:26.461 INFO:teuthology.orchestra.run.smithi099.stdout: "id": "5d8acec7-7e57-4e56-8b32-e3d6b5036001", 2023-05-31T21:17:26.462 INFO:teuthology.orchestra.run.smithi099.stdout: "name": "a2", 2023-05-31T21:17:26.462 INFO:teuthology.orchestra.run.smithi099.stdout: "endpoints": [ 2023-05-31T21:17:26.462 INFO:teuthology.orchestra.run.smithi099.stdout: "http://smithi099.front.sepia.ceph.com:8001" 2023-05-31T21:17:26.462 INFO:teuthology.orchestra.run.smithi099.stdout: ], 2023-05-31T21:17:26.462 INFO:teuthology.orchestra.run.smithi099.stdout: "log_meta": false, 2023-05-31T21:17:26.462 INFO:teuthology.orchestra.run.smithi099.stdout: "log_data": true, 2023-05-31T21:17:26.462 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket_index_max_shards": 11, 2023-05-31T21:17:26.463 INFO:teuthology.orchestra.run.smithi099.stdout: "read_only": false, 2023-05-31T21:17:26.463 INFO:teuthology.orchestra.run.smithi099.stdout: "tier_type": "", 2023-05-31T21:17:26.463 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_from_all": true, 2023-05-31T21:17:26.463 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_from": [], 2023-05-31T21:17:26.463 INFO:teuthology.orchestra.run.smithi099.stdout: "redirect_zone": "", 2023-05-31T21:17:26.463 INFO:teuthology.orchestra.run.smithi099.stdout: "supported_features": [ 2023-05-31T21:17:26.463 INFO:teuthology.orchestra.run.smithi099.stdout: "resharding" 2023-05-31T21:17:26.464 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2023-05-31T21:17:26.464 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T21:17:26.464 INFO:teuthology.orchestra.run.smithi099.stdout: ], 2023-05-31T21:17:26.464 INFO:teuthology.orchestra.run.smithi099.stdout: "placement_targets": [ 2023-05-31T21:17:26.464 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T21:17:26.464 INFO:teuthology.orchestra.run.smithi099.stdout: "name": "default-placement", 2023-05-31T21:17:26.465 INFO:teuthology.orchestra.run.smithi099.stdout: "tags": [], 2023-05-31T21:17:26.465 INFO:teuthology.orchestra.run.smithi099.stdout: "storage_classes": [ 2023-05-31T21:17:26.465 INFO:teuthology.orchestra.run.smithi099.stdout: "STANDARD" 2023-05-31T21:17:26.465 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2023-05-31T21:17:26.465 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T21:17:26.465 INFO:teuthology.orchestra.run.smithi099.stdout: ], 2023-05-31T21:17:26.465 INFO:teuthology.orchestra.run.smithi099.stdout: "default_placement": "default-placement", 2023-05-31T21:17:26.466 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_id": "aaa33de6-515b-4cac-8f1b-3a1a30452117", 2023-05-31T21:17:26.466 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_policy": { 2023-05-31T21:17:26.466 INFO:teuthology.orchestra.run.smithi099.stdout: "groups": [ 2023-05-31T21:17:26.466 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T21:17:26.466 INFO:teuthology.orchestra.run.smithi099.stdout: "id": "sync-group", 2023-05-31T21:17:26.466 INFO:teuthology.orchestra.run.smithi099.stdout: "data_flow": { 2023-05-31T21:17:26.466 INFO:teuthology.orchestra.run.smithi099.stdout: "symmetrical": [ 2023-05-31T21:17:26.467 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T21:17:26.467 INFO:teuthology.orchestra.run.smithi099.stdout: "id": "sync-flow1", 2023-05-31T21:17:26.467 INFO:teuthology.orchestra.run.smithi099.stdout: "zones": [ 2023-05-31T21:17:26.467 INFO:teuthology.orchestra.run.smithi099.stdout: "3d163b1e-09c7-47e2-ae87-add4aace3f38", 2023-05-31T21:17:26.467 INFO:teuthology.orchestra.run.smithi099.stdout: "5d8acec7-7e57-4e56-8b32-e3d6b5036001" 2023-05-31T21:17:26.467 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2023-05-31T21:17:26.467 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T21:17:26.468 INFO:teuthology.orchestra.run.smithi099.stdout: ], 2023-05-31T21:17:26.468 INFO:teuthology.orchestra.run.smithi099.stdout: "directional": [ 2023-05-31T21:17:26.468 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T21:17:26.468 INFO:teuthology.orchestra.run.smithi099.stdout: "source_zone": "3d163b1e-09c7-47e2-ae87-add4aace3f38", 2023-05-31T21:17:26.468 INFO:teuthology.orchestra.run.smithi099.stdout: "dest_zone": "5d8acec7-7e57-4e56-8b32-e3d6b5036001" 2023-05-31T21:17:26.468 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T21:17:26.469 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2023-05-31T21:17:26.469 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:17:26.469 INFO:teuthology.orchestra.run.smithi099.stdout: "pipes": [ 2023-05-31T21:17:26.469 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T21:17:26.469 INFO:teuthology.orchestra.run.smithi099.stdout: "id": "sync-pipe", 2023-05-31T21:17:26.469 INFO:teuthology.orchestra.run.smithi099.stdout: "source": { 2023-05-31T21:17:26.469 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket": "*", 2023-05-31T21:17:26.470 INFO:teuthology.orchestra.run.smithi099.stdout: "zones": [ 2023-05-31T21:17:26.470 INFO:teuthology.orchestra.run.smithi099.stdout: "3d163b1e-09c7-47e2-ae87-add4aace3f38", 2023-05-31T21:17:26.470 INFO:teuthology.orchestra.run.smithi099.stdout: "5d8acec7-7e57-4e56-8b32-e3d6b5036001" 2023-05-31T21:17:26.470 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2023-05-31T21:17:26.470 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:17:26.470 INFO:teuthology.orchestra.run.smithi099.stdout: "dest": { 2023-05-31T21:17:26.470 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket": "*", 2023-05-31T21:17:26.471 INFO:teuthology.orchestra.run.smithi099.stdout: "zones": [ 2023-05-31T21:17:26.471 INFO:teuthology.orchestra.run.smithi099.stdout: "3d163b1e-09c7-47e2-ae87-add4aace3f38", 2023-05-31T21:17:26.471 INFO:teuthology.orchestra.run.smithi099.stdout: "5d8acec7-7e57-4e56-8b32-e3d6b5036001" 2023-05-31T21:17:26.471 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2023-05-31T21:17:26.471 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:17:26.471 INFO:teuthology.orchestra.run.smithi099.stdout: "params": { 2023-05-31T21:17:26.471 INFO:teuthology.orchestra.run.smithi099.stdout: "source": { 2023-05-31T21:17:26.472 INFO:teuthology.orchestra.run.smithi099.stdout: "filter": { 2023-05-31T21:17:26.472 INFO:teuthology.orchestra.run.smithi099.stdout: "tags": [] 2023-05-31T21:17:26.472 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T21:17:26.472 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:17:26.472 INFO:teuthology.orchestra.run.smithi099.stdout: "dest": {}, 2023-05-31T21:17:26.472 INFO:teuthology.orchestra.run.smithi099.stdout: "priority": 0, 2023-05-31T21:17:26.472 INFO:teuthology.orchestra.run.smithi099.stdout: "mode": "system", 2023-05-31T21:17:26.473 INFO:teuthology.orchestra.run.smithi099.stdout: "user": "" 2023-05-31T21:17:26.473 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T21:17:26.473 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T21:17:26.473 INFO:teuthology.orchestra.run.smithi099.stdout: ], 2023-05-31T21:17:26.473 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "enabled" 2023-05-31T21:17:26.473 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T21:17:26.473 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2023-05-31T21:17:26.474 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:17:26.474 INFO:teuthology.orchestra.run.smithi099.stdout: "enabled_features": [ 2023-05-31T21:17:26.474 INFO:teuthology.orchestra.run.smithi099.stdout: "resharding" 2023-05-31T21:17:26.474 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2023-05-31T21:17:26.474 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:17:26.474 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T21:17:26.474 INFO:teuthology.orchestra.run.smithi099.stdout: "id": "b", 2023-05-31T21:17:26.475 INFO:teuthology.orchestra.run.smithi099.stdout: "name": "b", 2023-05-31T21:17:26.475 INFO:teuthology.orchestra.run.smithi099.stdout: "api_name": "", 2023-05-31T21:17:26.475 INFO:teuthology.orchestra.run.smithi099.stdout: "is_master": false, 2023-05-31T21:17:26.475 INFO:teuthology.orchestra.run.smithi099.stdout: "endpoints": [ 2023-05-31T21:17:26.475 INFO:teuthology.orchestra.run.smithi099.stdout: "http://smithi112.front.sepia.ceph.com:8000" 2023-05-31T21:17:26.475 INFO:teuthology.orchestra.run.smithi099.stdout: ], 2023-05-31T21:17:26.475 INFO:teuthology.orchestra.run.smithi099.stdout: "hostnames": [], 2023-05-31T21:17:26.476 INFO:teuthology.orchestra.run.smithi099.stdout: "hostnames_s3website": [], 2023-05-31T21:17:26.476 INFO:teuthology.orchestra.run.smithi099.stdout: "master_zone": "f85d7236-2f27-4988-af26-9c66ab280b1b", 2023-05-31T21:17:26.476 INFO:teuthology.orchestra.run.smithi099.stdout: "zones": [ 2023-05-31T21:17:26.476 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T21:17:26.476 INFO:teuthology.orchestra.run.smithi099.stdout: "id": "a6a3aab5-bc52-44a4-8543-5c5ea6056acd", 2023-05-31T21:17:26.476 INFO:teuthology.orchestra.run.smithi099.stdout: "name": "b2", 2023-05-31T21:17:26.476 INFO:teuthology.orchestra.run.smithi099.stdout: "endpoints": [ 2023-05-31T21:17:26.477 INFO:teuthology.orchestra.run.smithi099.stdout: "http://smithi112.front.sepia.ceph.com:8001" 2023-05-31T21:17:26.477 INFO:teuthology.orchestra.run.smithi099.stdout: ], 2023-05-31T21:17:26.477 INFO:teuthology.orchestra.run.smithi099.stdout: "log_meta": false, 2023-05-31T21:17:26.477 INFO:teuthology.orchestra.run.smithi099.stdout: "log_data": true, 2023-05-31T21:17:26.477 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket_index_max_shards": 11, 2023-05-31T21:17:26.477 INFO:teuthology.orchestra.run.smithi099.stdout: "read_only": false, 2023-05-31T21:17:26.477 INFO:teuthology.orchestra.run.smithi099.stdout: "tier_type": "", 2023-05-31T21:17:26.478 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_from_all": true, 2023-05-31T21:17:26.478 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_from": [], 2023-05-31T21:17:26.478 INFO:teuthology.orchestra.run.smithi099.stdout: "redirect_zone": "", 2023-05-31T21:17:26.478 INFO:teuthology.orchestra.run.smithi099.stdout: "supported_features": [ 2023-05-31T21:17:26.478 INFO:teuthology.orchestra.run.smithi099.stdout: "resharding" 2023-05-31T21:17:26.478 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2023-05-31T21:17:26.478 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:17:26.479 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T21:17:26.479 INFO:teuthology.orchestra.run.smithi099.stdout: "id": "f85d7236-2f27-4988-af26-9c66ab280b1b", 2023-05-31T21:17:26.479 INFO:teuthology.orchestra.run.smithi099.stdout: "name": "b1", 2023-05-31T21:17:26.479 INFO:teuthology.orchestra.run.smithi099.stdout: "endpoints": [ 2023-05-31T21:17:26.479 INFO:teuthology.orchestra.run.smithi099.stdout: "http://smithi112.front.sepia.ceph.com:8000" 2023-05-31T21:17:26.479 INFO:teuthology.orchestra.run.smithi099.stdout: ], 2023-05-31T21:17:26.479 INFO:teuthology.orchestra.run.smithi099.stdout: "log_meta": false, 2023-05-31T21:17:26.480 INFO:teuthology.orchestra.run.smithi099.stdout: "log_data": true, 2023-05-31T21:17:26.480 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket_index_max_shards": 11, 2023-05-31T21:17:26.480 INFO:teuthology.orchestra.run.smithi099.stdout: "read_only": false, 2023-05-31T21:17:26.480 INFO:teuthology.orchestra.run.smithi099.stdout: "tier_type": "", 2023-05-31T21:17:26.480 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_from_all": true, 2023-05-31T21:17:26.480 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_from": [], 2023-05-31T21:17:26.480 INFO:teuthology.orchestra.run.smithi099.stdout: "redirect_zone": "", 2023-05-31T21:17:26.481 INFO:teuthology.orchestra.run.smithi099.stdout: "supported_features": [ 2023-05-31T21:17:26.481 INFO:teuthology.orchestra.run.smithi099.stdout: "resharding" 2023-05-31T21:17:26.481 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2023-05-31T21:17:26.481 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T21:17:26.481 INFO:teuthology.orchestra.run.smithi099.stdout: ], 2023-05-31T21:17:26.481 INFO:teuthology.orchestra.run.smithi099.stdout: "placement_targets": [ 2023-05-31T21:17:26.481 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T21:17:26.482 INFO:teuthology.orchestra.run.smithi099.stdout: "name": "default-placement", 2023-05-31T21:17:26.482 INFO:teuthology.orchestra.run.smithi099.stdout: "tags": [], 2023-05-31T21:17:26.482 INFO:teuthology.orchestra.run.smithi099.stdout: "storage_classes": [ 2023-05-31T21:17:26.482 INFO:teuthology.orchestra.run.smithi099.stdout: "STANDARD" 2023-05-31T21:17:26.482 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2023-05-31T21:17:26.482 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T21:17:26.482 INFO:teuthology.orchestra.run.smithi099.stdout: ], 2023-05-31T21:17:26.483 INFO:teuthology.orchestra.run.smithi099.stdout: "default_placement": "default-placement", 2023-05-31T21:17:26.483 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_id": "aaa33de6-515b-4cac-8f1b-3a1a30452117", 2023-05-31T21:17:26.483 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_policy": { 2023-05-31T21:17:26.483 INFO:teuthology.orchestra.run.smithi099.stdout: "groups": [] 2023-05-31T21:17:26.483 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:17:26.483 INFO:teuthology.orchestra.run.smithi099.stdout: "enabled_features": [ 2023-05-31T21:17:26.483 INFO:teuthology.orchestra.run.smithi099.stdout: "resharding" 2023-05-31T21:17:26.484 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2023-05-31T21:17:26.484 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T21:17:26.484 INFO:teuthology.orchestra.run.smithi099.stdout: ], 2023-05-31T21:17:26.484 INFO:teuthology.orchestra.run.smithi099.stdout: "short_zone_ids": [ 2023-05-31T21:17:26.484 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T21:17:26.484 INFO:teuthology.orchestra.run.smithi099.stdout: "key": "3d163b1e-09c7-47e2-ae87-add4aace3f38", 2023-05-31T21:17:26.485 INFO:teuthology.orchestra.run.smithi099.stdout: "val": 3101833771 2023-05-31T21:17:26.485 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:17:26.486 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T21:17:26.486 INFO:teuthology.orchestra.run.smithi099.stdout: "key": "5d8acec7-7e57-4e56-8b32-e3d6b5036001", 2023-05-31T21:17:26.486 INFO:teuthology.orchestra.run.smithi099.stdout: "val": 4255530623 2023-05-31T21:17:26.486 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:17:26.486 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T21:17:26.486 INFO:teuthology.orchestra.run.smithi099.stdout: "key": "a6a3aab5-bc52-44a4-8543-5c5ea6056acd", 2023-05-31T21:17:26.487 INFO:teuthology.orchestra.run.smithi099.stdout: "val": 3708073710 2023-05-31T21:17:26.487 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:17:26.487 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T21:17:26.487 INFO:teuthology.orchestra.run.smithi099.stdout: "key": "f85d7236-2f27-4988-af26-9c66ab280b1b", 2023-05-31T21:17:26.487 INFO:teuthology.orchestra.run.smithi099.stdout: "val": 233395200 2023-05-31T21:17:26.487 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T21:17:26.488 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2023-05-31T21:17:26.488 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:17:26.488 INFO:teuthology.orchestra.run.smithi099.stdout: "master_zonegroup": "a", 2023-05-31T21:17:26.488 INFO:teuthology.orchestra.run.smithi099.stdout: "master_zone": "3d163b1e-09c7-47e2-ae87-add4aace3f38", 2023-05-31T21:17:26.488 INFO:teuthology.orchestra.run.smithi099.stdout: "period_config": { 2023-05-31T21:17:26.488 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket_quota": { 2023-05-31T21:17:26.488 INFO:teuthology.orchestra.run.smithi099.stdout: "enabled": false, 2023-05-31T21:17:26.489 INFO:teuthology.orchestra.run.smithi099.stdout: "check_on_raw": false, 2023-05-31T21:17:26.489 INFO:teuthology.orchestra.run.smithi099.stdout: "max_size": -1, 2023-05-31T21:17:26.489 INFO:teuthology.orchestra.run.smithi099.stdout: "max_size_kb": 0, 2023-05-31T21:17:26.489 INFO:teuthology.orchestra.run.smithi099.stdout: "max_objects": -1 2023-05-31T21:17:26.489 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:17:26.489 INFO:teuthology.orchestra.run.smithi099.stdout: "user_quota": { 2023-05-31T21:17:26.489 INFO:teuthology.orchestra.run.smithi099.stdout: "enabled": false, 2023-05-31T21:17:26.490 INFO:teuthology.orchestra.run.smithi099.stdout: "check_on_raw": false, 2023-05-31T21:17:26.490 INFO:teuthology.orchestra.run.smithi099.stdout: "max_size": -1, 2023-05-31T21:17:26.490 INFO:teuthology.orchestra.run.smithi099.stdout: "max_size_kb": 0, 2023-05-31T21:17:26.490 INFO:teuthology.orchestra.run.smithi099.stdout: "max_objects": -1 2023-05-31T21:17:26.490 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:17:26.490 INFO:teuthology.orchestra.run.smithi099.stdout: "user_ratelimit": { 2023-05-31T21:17:26.490 INFO:teuthology.orchestra.run.smithi099.stdout: "max_read_ops": 0, 2023-05-31T21:17:26.491 INFO:teuthology.orchestra.run.smithi099.stdout: "max_write_ops": 0, 2023-05-31T21:17:26.491 INFO:teuthology.orchestra.run.smithi099.stdout: "max_read_bytes": 0, 2023-05-31T21:17:26.491 INFO:teuthology.orchestra.run.smithi099.stdout: "max_write_bytes": 0, 2023-05-31T21:17:26.491 INFO:teuthology.orchestra.run.smithi099.stdout: "enabled": false 2023-05-31T21:17:26.491 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:17:26.491 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket_ratelimit": { 2023-05-31T21:17:26.491 INFO:teuthology.orchestra.run.smithi099.stdout: "max_read_ops": 0, 2023-05-31T21:17:26.492 INFO:teuthology.orchestra.run.smithi099.stdout: "max_write_ops": 0, 2023-05-31T21:17:26.492 INFO:teuthology.orchestra.run.smithi099.stdout: "max_read_bytes": 0, 2023-05-31T21:17:26.492 INFO:teuthology.orchestra.run.smithi099.stdout: "max_write_bytes": 0, 2023-05-31T21:17:26.492 INFO:teuthology.orchestra.run.smithi099.stdout: "enabled": false 2023-05-31T21:17:26.492 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:17:26.492 INFO:teuthology.orchestra.run.smithi099.stdout: "anonymous_ratelimit": { 2023-05-31T21:17:26.492 INFO:teuthology.orchestra.run.smithi099.stdout: "max_read_ops": 0, 2023-05-31T21:17:26.493 INFO:teuthology.orchestra.run.smithi099.stdout: "max_write_ops": 0, 2023-05-31T21:17:26.493 INFO:teuthology.orchestra.run.smithi099.stdout: "max_read_bytes": 0, 2023-05-31T21:17:26.493 INFO:teuthology.orchestra.run.smithi099.stdout: "max_write_bytes": 0, 2023-05-31T21:17:26.493 INFO:teuthology.orchestra.run.smithi099.stdout: "enabled": false 2023-05-31T21:17:26.493 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T21:17:26.493 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:17:26.493 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_id": "aaa33de6-515b-4cac-8f1b-3a1a30452117", 2023-05-31T21:17:26.494 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_name": "test-realm", 2023-05-31T21:17:26.494 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2023-05-31T21:17:26.494 INFO:teuthology.orchestra.run.smithi099.stdout:} 2023-05-31T21:17:26.494 INFO:rgw_multi.tests:create bucket zone=a1 name=pklxkd-65 2023-05-31T21:17:26.495 DEBUG:boto:path=/pklxkd-65/ 2023-05-31T21:17:26.495 DEBUG:boto:auth_path=/pklxkd-65/ 2023-05-31T21:17:26.496 DEBUG:boto:Method: PUT 2023-05-31T21:17:26.496 DEBUG:boto:Path: /pklxkd-65/ 2023-05-31T21:17:26.496 DEBUG:boto:Data: 2023-05-31T21:17:26.497 DEBUG:boto:Headers: {} 2023-05-31T21:17:26.497 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T21:17:26.497 DEBUG:boto:Port: 8000 2023-05-31T21:17:26.498 DEBUG:boto:Params: {} 2023-05-31T21:17:26.498 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2023-05-31T21:17:26.499 DEBUG:boto:Token: None 2023-05-31T21:17:26.499 DEBUG:boto:StringToSign: PUT Wed, 31 May 2023 21:17:26 GMT /pklxkd-65/ 2023-05-31T21:17:26.500 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:+bXCMkl7W8Zd2UB2OfxTnEutXjg= 2023-05-31T21:17:26.500 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 21:17:26 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:+bXCMkl7W8Zd2UB2OfxTnEutXjg=', 'Content-Length': '0'} 2023-05-31T21:17:26.860 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx000008f779e64aea39540-006477b966-2110-a1'), ('Content-Length', '0'), ('Date', 'Wed, 31 May 2023 21:17:26 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:17:26.861 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'group', 'create', '--group-id', 'sync-bucket', '--status', 'allowed', '--bucket', 'pklxkd-65', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-05-31T21:17:26.862 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'group', 'create', '--group-id', 'sync-bucket', '--status', 'allowed', '--bucket', 'pklxkd-65', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-05-31T21:17:26.862 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync group create --group-id sync-bucket --status allowed --bucket pklxkd-65 --cluster c1 --debug-rgw 0 --debug-ms 0 2023-05-31T21:17:26.927 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2023-05-31T21:17:26.927 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2023-05-31T21:17:27.072 INFO:teuthology.orchestra.run.smithi099.stdout:{"groups":[{"id":"sync-bucket","data_flow":{},"pipes":[],"status":"allowed"}]} 2023-05-31T21:17:27.073 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'group', 'modify', '--group-id', 'sync-bucket', '--status', 'enabled', '--bucket', 'pklxkd-65', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-05-31T21:17:27.073 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'group', 'modify', '--group-id', 'sync-bucket', '--status', 'enabled', '--bucket', 'pklxkd-65', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-05-31T21:17:27.074 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync group modify --group-id sync-bucket --status enabled --bucket pklxkd-65 --cluster c1 --debug-rgw 0 --debug-ms 0 2023-05-31T21:17:27.128 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2023-05-31T21:17:27.128 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2023-05-31T21:17:27.251 INFO:teuthology.orchestra.run.smithi099.stdout:{"groups":[{"id":"sync-bucket","data_flow":{},"pipes":[],"status":"enabled"}]} 2023-05-31T21:17:27.252 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'group', 'get', '--group-id', 'sync-bucket', '--bucket', 'pklxkd-65', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-05-31T21:17:27.252 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'group', 'get', '--group-id', 'sync-bucket', '--bucket', 'pklxkd-65', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-05-31T21:17:27.253 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync group get --group-id sync-bucket --bucket pklxkd-65 --cluster c1 --debug-rgw 0 --debug-ms 0 2023-05-31T21:17:27.311 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2023-05-31T21:17:27.311 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2023-05-31T21:17:27.416 INFO:teuthology.orchestra.run.smithi099.stdout:{"id":"sync-bucket","data_flow":{},"pipes":[],"status":"enabled"} 2023-05-31T21:17:27.417 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'policy', 'get', '--bucket', 'pklxkd-65', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-05-31T21:17:27.417 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'policy', 'get', '--bucket', 'pklxkd-65', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-05-31T21:17:27.418 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync policy get --bucket pklxkd-65 --cluster c1 --debug-rgw 0 --debug-ms 0 2023-05-31T21:17:27.470 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2023-05-31T21:17:27.470 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2023-05-31T21:17:27.553 INFO:teuthology.orchestra.run.smithi099.stdout:{"groups":[{"id":"sync-bucket","data_flow":{},"pipes":[],"status":"enabled"}]} 2023-05-31T21:17:27.554 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'group', 'flow', 'create', '--group-id', 'sync-bucket', '--flow-id', 'sync-flow1', '--flow-type', 'symmetrical', '--zones=a1,a2', '--bucket', 'pklxkd-65', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-05-31T21:17:27.554 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'group', 'flow', 'create', '--group-id', 'sync-bucket', '--flow-id', 'sync-flow1', '--flow-type', 'symmetrical', '--zones=a1,a2', '--bucket', 'pklxkd-65', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-05-31T21:17:27.555 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync group flow create --group-id sync-bucket --flow-id sync-flow1 --flow-type symmetrical --zones=a1,a2 --bucket pklxkd-65 --cluster c1 --debug-rgw 0 --debug-ms 0 2023-05-31T21:17:27.619 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2023-05-31T21:17:27.619 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2023-05-31T21:17:27.694 INFO:teuthology.orchestra.run.smithi099.stdout:{"groups":[{"id":"sync-bucket","data_flow":{"symmetrical":[{"id":"sync-flow1","zones":["a1","a2"]}]},"pipes":[],"status":"enabled"}]} 2023-05-31T21:17:27.695 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'group', 'flow', 'create', '--group-id', 'sync-bucket', '--flow-id', 'sync-flow2', '--flow-type', 'directional', '--source-zone=a1', '--dest-zone=a2', '--bucket', 'pklxkd-65', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-05-31T21:17:27.695 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'group', 'flow', 'create', '--group-id', 'sync-bucket', '--flow-id', 'sync-flow2', '--flow-type', 'directional', '--source-zone=a1', '--dest-zone=a2', '--bucket', 'pklxkd-65', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-05-31T21:17:27.695 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync group flow create --group-id sync-bucket --flow-id sync-flow2 --flow-type directional --source-zone=a1 --dest-zone=a2 --bucket pklxkd-65 --cluster c1 --debug-rgw 0 --debug-ms 0 2023-05-31T21:17:27.753 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2023-05-31T21:17:27.753 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2023-05-31T21:17:27.828 INFO:teuthology.orchestra.run.smithi099.stdout:{"groups":[{"id":"sync-bucket","data_flow":{"symmetrical":[{"id":"sync-flow1","zones":["a1","a2"]}],"directional":[{"source_zone":"a1","dest_zone":"a2"}]},"pipes":[],"status":"enabled"}]} 2023-05-31T21:17:27.828 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'group', 'pipe', 'create', '--group-id', 'sync-bucket', '--pipe-id', 'sync-pipe', '--source-zones=a1,a2', '--dest-zones=a1,a2', '--bucket=pklxkd-65', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-05-31T21:17:27.829 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'group', 'pipe', 'create', '--group-id', 'sync-bucket', '--pipe-id', 'sync-pipe', '--source-zones=a1,a2', '--dest-zones=a1,a2', '--bucket=pklxkd-65', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-05-31T21:17:27.829 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync group pipe create --group-id sync-bucket --pipe-id sync-pipe --source-zones=a1,a2 --dest-zones=a1,a2 --bucket=pklxkd-65 --cluster c1 --debug-rgw 0 --debug-ms 0 2023-05-31T21:17:27.884 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2023-05-31T21:17:27.884 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2023-05-31T21:17:27.955 INFO:teuthology.orchestra.run.smithi099.stdout:{"groups":[{"id":"sync-bucket","data_flow":{"symmetrical":[{"id":"sync-flow1","zones":["a1","a2"]}],"directional":[{"source_zone":"a1","dest_zone":"a2"}]},"pipes":[{"id":"sync-pipe","source":{"bucket":"*","zones":["a1","a2"]},"dest":{"bucket":"*","zones":["a1","a2"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":"rgw-multisite-test-user"}}],"status":"enabled"}]} 2023-05-31T21:17:27.955 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'group', 'get', '--group-id', 'sync-bucket', '--bucket', 'pklxkd-65', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-05-31T21:17:27.956 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'group', 'get', '--group-id', 'sync-bucket', '--bucket', 'pklxkd-65', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-05-31T21:17:27.956 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync group get --group-id sync-bucket --bucket pklxkd-65 --cluster c1 --debug-rgw 0 --debug-ms 0 2023-05-31T21:17:28.006 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2023-05-31T21:17:28.007 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2023-05-31T21:17:28.063 INFO:teuthology.orchestra.run.smithi099.stdout:{"id":"sync-bucket","data_flow":{"symmetrical":[{"id":"sync-flow1","zones":["a1","a2"]}],"directional":[{"source_zone":"a1","dest_zone":"a2"}]},"pipes":[{"id":"sync-pipe","source":{"bucket":"*","zones":["a1","a2"]},"dest":{"bucket":"*","zones":["a1","a2"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":"rgw-multisite-test-user"}}],"status":"enabled"} 2023-05-31T21:17:28.064 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-31T21:17:28.064 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-31T21:17:28.064 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-05-31T21:17:28.115 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2023-05-31T21:17:28.115 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2023-05-31T21:17:28.157 INFO:teuthology.orchestra.run.smithi099.stderr:No --period given, using current period=2e6123d2-d695-4350-9a01-61a455bc782b 2023-05-31T21:17:28.172 INFO:teuthology.orchestra.run.smithi099.stdout:[ 2023-05-31T21:17:28.173 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T21:17:28.173 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1685566501.757183_260.1", 2023-05-31T21:17:28.173 INFO:teuthology.orchestra.run.smithi099.stdout: "last_update": "2023-05-31T20:55:01.757183Z" 2023-05-31T21:17:28.173 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:17:28.173 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T21:17:28.174 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1685566501.735717_251.1", 2023-05-31T21:17:28.174 INFO:teuthology.orchestra.run.smithi099.stdout: "last_update": "2023-05-31T20:55:01.735717Z" 2023-05-31T21:17:28.174 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:17:28.174 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T21:17:28.174 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1685562970.274572_93.1", 2023-05-31T21:17:28.174 INFO:teuthology.orchestra.run.smithi099.stdout: "last_update": "2023-05-31T19:56:10.274572Z" 2023-05-31T21:17:28.175 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:17:28.175 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T21:17:28.175 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1685567847.936050_900.1", 2023-05-31T21:17:28.175 INFO:teuthology.orchestra.run.smithi099.stdout: "last_update": "2023-05-31T21:17:27.936050Z" 2023-05-31T21:17:28.175 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T21:17:28.175 INFO:teuthology.orchestra.run.smithi099.stdout:] 2023-05-31T21:17:28.176 DEBUG:rgw_multi.tests:master meta markers={0: '1_1685566501.757183_260.1', 1: '1_1685566501.735717_251.1', 2: '1_1685562970.274572_93.1', 3: '1_1685567847.936050_900.1'} 2023-05-31T21:17:28.177 INFO:rgw_multi.tests:starting meta checkpoint for zone=a2 2023-05-31T21:17:28.177 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-31T21:17:28.177 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-31T21:17:28.178 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-05-31T21:17:28.226 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2023-05-31T21:17:28.227 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2023-05-31T21:17:28.262 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T21:17:28.258+0000 7f447928d780 20 RGW-SYNC:meta: read sync status 2023-05-31T21:17:28.263 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T21:17:28.259+0000 7f447928d780 20 RGW-SYNC:meta: read sync status 2023-05-31T21:17:28.276 INFO:teuthology.orchestra.run.smithi099.stdout:{ 2023-05-31T21:17:28.276 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_status": { 2023-05-31T21:17:28.277 INFO:teuthology.orchestra.run.smithi099.stdout: "info": { 2023-05-31T21:17:28.277 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "sync", 2023-05-31T21:17:28.277 INFO:teuthology.orchestra.run.smithi099.stdout: "num_shards": 4, 2023-05-31T21:17:28.277 INFO:teuthology.orchestra.run.smithi099.stdout: "period": "2e6123d2-d695-4350-9a01-61a455bc782b", 2023-05-31T21:17:28.277 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2023-05-31T21:17:28.277 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:17:28.278 INFO:teuthology.orchestra.run.smithi099.stdout: "markers": [ 2023-05-31T21:17:28.278 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T21:17:28.278 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 0, 2023-05-31T21:17:28.278 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T21:17:28.278 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2023-05-31T21:17:28.278 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1685566501.757183_260.1", 2023-05-31T21:17:28.279 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T21:17:28.279 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 1, 2023-05-31T21:17:28.279 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T21:17:28.279 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2023-05-31T20:55:01.757183Z", 2023-05-31T21:17:28.279 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2023-05-31T21:17:28.279 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T21:17:28.279 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:17:28.280 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T21:17:28.280 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 1, 2023-05-31T21:17:28.280 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T21:17:28.280 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2023-05-31T21:17:28.280 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1685566501.735717_251.1", 2023-05-31T21:17:28.280 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T21:17:28.281 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2023-05-31T21:17:28.281 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T21:17:28.281 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2023-05-31T20:55:01.735717Z", 2023-05-31T21:17:28.281 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2023-05-31T21:17:28.281 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T21:17:28.281 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:17:28.281 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T21:17:28.282 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 2, 2023-05-31T21:17:28.282 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T21:17:28.282 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2023-05-31T21:17:28.282 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1685562970.274572_93.1", 2023-05-31T21:17:28.282 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T21:17:28.282 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2023-05-31T21:17:28.283 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T21:17:28.283 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2023-05-31T19:56:10.274572Z", 2023-05-31T21:17:28.283 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2023-05-31T21:17:28.283 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T21:17:28.283 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:17:28.283 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T21:17:28.283 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 3, 2023-05-31T21:17:28.284 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T21:17:28.284 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2023-05-31T21:17:28.284 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1685567548.445367_776.1", 2023-05-31T21:17:28.284 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T21:17:28.284 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2023-05-31T21:17:28.285 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T21:17:28.285 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2023-05-31T21:12:28.445367Z", 2023-05-31T21:17:28.285 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2023-05-31T21:17:28.285 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T21:17:28.285 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T21:17:28.285 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2023-05-31T21:17:28.285 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:17:28.286 INFO:teuthology.orchestra.run.smithi099.stdout: "full_sync": { 2023-05-31T21:17:28.286 INFO:teuthology.orchestra.run.smithi099.stdout: "total": 1, 2023-05-31T21:17:28.286 INFO:teuthology.orchestra.run.smithi099.stdout: "complete": 1 2023-05-31T21:17:28.286 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:17:28.286 INFO:teuthology.orchestra.run.smithi099.stdout: "current_time": "2023-05-31T21:17:28Z" 2023-05-31T21:17:28.287 INFO:teuthology.orchestra.run.smithi099.stdout:} 2023-05-31T21:17:28.287 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "2e6123d2-d695-4350-9a01-61a455bc782b", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1685566501.757183_260.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-05-31T20:55:01.757183Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1685566501.735717_251.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-05-31T20:55:01.735717Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1685562970.274572_93.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-05-31T19:56:10.274572Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1685567548.445367_776.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-05-31T21:12:28.445367Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-05-31T21:17:28Z" } 2023-05-31T21:17:28.288 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1685566501.757183_260.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-05-31T20:55:01.757183Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1685566501.735717_251.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-05-31T20:55:01.735717Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1685562970.274572_93.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-05-31T19:56:10.274572Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1685567548.445367_776.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-05-31T21:12:28.445367Z', 'realm_epoch': 2}}] 2023-05-31T21:17:28.288 DEBUG:rgw_multi.tests:log_status={0: '1_1685566501.757183_260.1', 1: '1_1685566501.735717_251.1', 2: '1_1685562970.274572_93.1', 3: '1_1685567847.936050_900.1'} 2023-05-31T21:17:28.289 DEBUG:rgw_multi.tests:sync_status={0: '1_1685566501.757183_260.1', 1: '1_1685566501.735717_251.1', 2: '1_1685562970.274572_93.1', 3: '1_1685567548.445367_776.1'} 2023-05-31T21:17:28.289 WARNING:rgw_multi.tests:zone a2 behind master: shard=3 master=1_1685567847.936050_900.1 target=1_1685567548.445367_776.1 2023-05-31T21:17:33.291 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-31T21:17:33.291 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-31T21:17:33.292 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-05-31T21:17:33.348 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2023-05-31T21:17:33.348 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2023-05-31T21:17:33.399 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T21:17:33.395+0000 7f9c2dc98780 20 RGW-SYNC:meta: read sync status 2023-05-31T21:17:33.400 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T21:17:33.396+0000 7f9c2dc98780 20 RGW-SYNC:meta: read sync status 2023-05-31T21:17:33.416 INFO:teuthology.orchestra.run.smithi099.stdout:{ 2023-05-31T21:17:33.417 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_status": { 2023-05-31T21:17:33.417 INFO:teuthology.orchestra.run.smithi099.stdout: "info": { 2023-05-31T21:17:33.417 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "sync", 2023-05-31T21:17:33.417 INFO:teuthology.orchestra.run.smithi099.stdout: "num_shards": 4, 2023-05-31T21:17:33.417 INFO:teuthology.orchestra.run.smithi099.stdout: "period": "2e6123d2-d695-4350-9a01-61a455bc782b", 2023-05-31T21:17:33.418 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2023-05-31T21:17:33.418 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:17:33.418 INFO:teuthology.orchestra.run.smithi099.stdout: "markers": [ 2023-05-31T21:17:33.418 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T21:17:33.418 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 0, 2023-05-31T21:17:33.418 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T21:17:33.418 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2023-05-31T21:17:33.419 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1685566501.757183_260.1", 2023-05-31T21:17:33.419 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T21:17:33.419 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 1, 2023-05-31T21:17:33.419 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T21:17:33.419 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2023-05-31T20:55:01.757183Z", 2023-05-31T21:17:33.419 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2023-05-31T21:17:33.419 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T21:17:33.420 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:17:33.420 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T21:17:33.420 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 1, 2023-05-31T21:17:33.420 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T21:17:33.420 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2023-05-31T21:17:33.420 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1685566501.735717_251.1", 2023-05-31T21:17:33.420 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T21:17:33.421 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2023-05-31T21:17:33.421 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T21:17:33.421 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2023-05-31T20:55:01.735717Z", 2023-05-31T21:17:33.421 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2023-05-31T21:17:33.421 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T21:17:33.421 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:17:33.422 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T21:17:33.422 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 2, 2023-05-31T21:17:33.422 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T21:17:33.422 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2023-05-31T21:17:33.422 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1685562970.274572_93.1", 2023-05-31T21:17:33.422 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T21:17:33.422 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2023-05-31T21:17:33.423 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T21:17:33.423 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2023-05-31T19:56:10.274572Z", 2023-05-31T21:17:33.423 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2023-05-31T21:17:33.423 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T21:17:33.423 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:17:33.423 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T21:17:33.423 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 3, 2023-05-31T21:17:33.424 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T21:17:33.424 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2023-05-31T21:17:33.424 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1685567548.445367_776.1", 2023-05-31T21:17:33.424 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T21:17:33.424 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2023-05-31T21:17:33.424 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T21:17:33.424 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2023-05-31T21:12:28.445367Z", 2023-05-31T21:17:33.425 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2023-05-31T21:17:33.425 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T21:17:33.425 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T21:17:33.425 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2023-05-31T21:17:33.425 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:17:33.425 INFO:teuthology.orchestra.run.smithi099.stdout: "full_sync": { 2023-05-31T21:17:33.425 INFO:teuthology.orchestra.run.smithi099.stdout: "total": 1, 2023-05-31T21:17:33.426 INFO:teuthology.orchestra.run.smithi099.stdout: "complete": 1 2023-05-31T21:17:33.426 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:17:33.426 INFO:teuthology.orchestra.run.smithi099.stdout: "current_time": "2023-05-31T21:17:33Z" 2023-05-31T21:17:33.426 INFO:teuthology.orchestra.run.smithi099.stdout:} 2023-05-31T21:17:33.427 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "2e6123d2-d695-4350-9a01-61a455bc782b", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1685566501.757183_260.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-05-31T20:55:01.757183Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1685566501.735717_251.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-05-31T20:55:01.735717Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1685562970.274572_93.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-05-31T19:56:10.274572Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1685567548.445367_776.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-05-31T21:12:28.445367Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-05-31T21:17:33Z" } 2023-05-31T21:17:33.427 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1685566501.757183_260.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-05-31T20:55:01.757183Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1685566501.735717_251.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-05-31T20:55:01.735717Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1685562970.274572_93.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-05-31T19:56:10.274572Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1685567548.445367_776.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-05-31T21:12:28.445367Z', 'realm_epoch': 2}}] 2023-05-31T21:17:33.428 DEBUG:rgw_multi.tests:log_status={0: '1_1685566501.757183_260.1', 1: '1_1685566501.735717_251.1', 2: '1_1685562970.274572_93.1', 3: '1_1685567847.936050_900.1'} 2023-05-31T21:17:33.428 DEBUG:rgw_multi.tests:sync_status={0: '1_1685566501.757183_260.1', 1: '1_1685566501.735717_251.1', 2: '1_1685562970.274572_93.1', 3: '1_1685567548.445367_776.1'} 2023-05-31T21:17:33.429 WARNING:rgw_multi.tests:zone a2 behind master: shard=3 master=1_1685567847.936050_900.1 target=1_1685567548.445367_776.1 2023-05-31T21:17:38.430 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-31T21:17:38.431 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-31T21:17:38.431 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-05-31T21:17:38.488 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2023-05-31T21:17:38.489 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2023-05-31T21:17:38.528 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T21:17:38.523+0000 7f1fbd370780 20 RGW-SYNC:meta: read sync status 2023-05-31T21:17:38.529 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T21:17:38.524+0000 7f1fbd370780 20 RGW-SYNC:meta: read sync status 2023-05-31T21:17:38.551 INFO:teuthology.orchestra.run.smithi099.stdout:{ 2023-05-31T21:17:38.551 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_status": { 2023-05-31T21:17:38.551 INFO:teuthology.orchestra.run.smithi099.stdout: "info": { 2023-05-31T21:17:38.552 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "sync", 2023-05-31T21:17:38.552 INFO:teuthology.orchestra.run.smithi099.stdout: "num_shards": 4, 2023-05-31T21:17:38.552 INFO:teuthology.orchestra.run.smithi099.stdout: "period": "2e6123d2-d695-4350-9a01-61a455bc782b", 2023-05-31T21:17:38.552 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2023-05-31T21:17:38.552 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:17:38.552 INFO:teuthology.orchestra.run.smithi099.stdout: "markers": [ 2023-05-31T21:17:38.553 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T21:17:38.553 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 0, 2023-05-31T21:17:38.553 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T21:17:38.553 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2023-05-31T21:17:38.553 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1685566501.757183_260.1", 2023-05-31T21:17:38.553 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T21:17:38.553 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 1, 2023-05-31T21:17:38.554 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T21:17:38.554 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2023-05-31T20:55:01.757183Z", 2023-05-31T21:17:38.554 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2023-05-31T21:17:38.554 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T21:17:38.554 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:17:38.554 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T21:17:38.555 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 1, 2023-05-31T21:17:38.555 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T21:17:38.555 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2023-05-31T21:17:38.555 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1685566501.735717_251.1", 2023-05-31T21:17:38.555 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T21:17:38.555 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2023-05-31T21:17:38.556 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T21:17:38.556 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2023-05-31T20:55:01.735717Z", 2023-05-31T21:17:38.556 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2023-05-31T21:17:38.556 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T21:17:38.556 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:17:38.556 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T21:17:38.557 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 2, 2023-05-31T21:17:38.557 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T21:17:38.557 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2023-05-31T21:17:38.557 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1685562970.274572_93.1", 2023-05-31T21:17:38.557 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T21:17:38.557 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2023-05-31T21:17:38.557 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T21:17:38.558 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2023-05-31T19:56:10.274572Z", 2023-05-31T21:17:38.558 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2023-05-31T21:17:38.558 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T21:17:38.558 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:17:38.558 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T21:17:38.558 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 3, 2023-05-31T21:17:38.559 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T21:17:38.559 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2023-05-31T21:17:38.559 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1685567548.445367_776.1", 2023-05-31T21:17:38.559 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T21:17:38.559 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2023-05-31T21:17:38.559 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T21:17:38.559 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2023-05-31T21:12:28.445367Z", 2023-05-31T21:17:38.560 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2023-05-31T21:17:38.560 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T21:17:38.560 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T21:17:38.560 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2023-05-31T21:17:38.560 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:17:38.560 INFO:teuthology.orchestra.run.smithi099.stdout: "full_sync": { 2023-05-31T21:17:38.561 INFO:teuthology.orchestra.run.smithi099.stdout: "total": 1, 2023-05-31T21:17:38.561 INFO:teuthology.orchestra.run.smithi099.stdout: "complete": 1 2023-05-31T21:17:38.561 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:17:38.561 INFO:teuthology.orchestra.run.smithi099.stdout: "current_time": "2023-05-31T21:17:38Z" 2023-05-31T21:17:38.561 INFO:teuthology.orchestra.run.smithi099.stdout:} 2023-05-31T21:17:38.562 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "2e6123d2-d695-4350-9a01-61a455bc782b", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1685566501.757183_260.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-05-31T20:55:01.757183Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1685566501.735717_251.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-05-31T20:55:01.735717Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1685562970.274572_93.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-05-31T19:56:10.274572Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1685567548.445367_776.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-05-31T21:12:28.445367Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-05-31T21:17:38Z" } 2023-05-31T21:17:38.563 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1685566501.757183_260.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-05-31T20:55:01.757183Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1685566501.735717_251.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-05-31T20:55:01.735717Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1685562970.274572_93.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-05-31T19:56:10.274572Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1685567548.445367_776.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-05-31T21:12:28.445367Z', 'realm_epoch': 2}}] 2023-05-31T21:17:38.563 DEBUG:rgw_multi.tests:log_status={0: '1_1685566501.757183_260.1', 1: '1_1685566501.735717_251.1', 2: '1_1685562970.274572_93.1', 3: '1_1685567847.936050_900.1'} 2023-05-31T21:17:38.563 DEBUG:rgw_multi.tests:sync_status={0: '1_1685566501.757183_260.1', 1: '1_1685566501.735717_251.1', 2: '1_1685562970.274572_93.1', 3: '1_1685567548.445367_776.1'} 2023-05-31T21:17:38.564 WARNING:rgw_multi.tests:zone a2 behind master: shard=3 master=1_1685567847.936050_900.1 target=1_1685567548.445367_776.1 2023-05-31T21:17:43.565 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-31T21:17:43.566 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-31T21:17:43.566 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-05-31T21:17:43.623 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2023-05-31T21:17:43.624 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2023-05-31T21:17:43.672 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T21:17:43.668+0000 7ff5d7094780 20 RGW-SYNC:meta: read sync status 2023-05-31T21:17:43.673 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T21:17:43.669+0000 7ff5d7094780 20 RGW-SYNC:meta: read sync status 2023-05-31T21:17:43.688 INFO:teuthology.orchestra.run.smithi099.stdout:{ 2023-05-31T21:17:43.689 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_status": { 2023-05-31T21:17:43.689 INFO:teuthology.orchestra.run.smithi099.stdout: "info": { 2023-05-31T21:17:43.689 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "sync", 2023-05-31T21:17:43.689 INFO:teuthology.orchestra.run.smithi099.stdout: "num_shards": 4, 2023-05-31T21:17:43.689 INFO:teuthology.orchestra.run.smithi099.stdout: "period": "2e6123d2-d695-4350-9a01-61a455bc782b", 2023-05-31T21:17:43.690 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2023-05-31T21:17:43.690 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:17:43.690 INFO:teuthology.orchestra.run.smithi099.stdout: "markers": [ 2023-05-31T21:17:43.690 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T21:17:43.690 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 0, 2023-05-31T21:17:43.690 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T21:17:43.690 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2023-05-31T21:17:43.691 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1685566501.757183_260.1", 2023-05-31T21:17:43.691 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T21:17:43.691 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 1, 2023-05-31T21:17:43.691 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T21:17:43.691 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2023-05-31T20:55:01.757183Z", 2023-05-31T21:17:43.691 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2023-05-31T21:17:43.691 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T21:17:43.692 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:17:43.692 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T21:17:43.692 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 1, 2023-05-31T21:17:43.692 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T21:17:43.692 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2023-05-31T21:17:43.693 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1685566501.735717_251.1", 2023-05-31T21:17:43.693 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T21:17:43.693 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2023-05-31T21:17:43.693 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T21:17:43.693 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2023-05-31T20:55:01.735717Z", 2023-05-31T21:17:43.693 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2023-05-31T21:17:43.693 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T21:17:43.694 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:17:43.694 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T21:17:43.694 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 2, 2023-05-31T21:17:43.694 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T21:17:43.694 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2023-05-31T21:17:43.694 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1685562970.274572_93.1", 2023-05-31T21:17:43.695 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T21:17:43.695 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2023-05-31T21:17:43.695 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T21:17:43.695 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2023-05-31T19:56:10.274572Z", 2023-05-31T21:17:43.695 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2023-05-31T21:17:43.695 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T21:17:43.695 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:17:43.696 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T21:17:43.696 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 3, 2023-05-31T21:17:43.696 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T21:17:43.696 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2023-05-31T21:17:43.696 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1685567548.445367_776.1", 2023-05-31T21:17:43.696 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T21:17:43.696 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2023-05-31T21:17:43.697 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T21:17:43.697 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2023-05-31T21:12:28.445367Z", 2023-05-31T21:17:43.697 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2023-05-31T21:17:43.697 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T21:17:43.697 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T21:17:43.697 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2023-05-31T21:17:43.697 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:17:43.698 INFO:teuthology.orchestra.run.smithi099.stdout: "full_sync": { 2023-05-31T21:17:43.698 INFO:teuthology.orchestra.run.smithi099.stdout: "total": 1, 2023-05-31T21:17:43.698 INFO:teuthology.orchestra.run.smithi099.stdout: "complete": 1 2023-05-31T21:17:43.698 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:17:43.698 INFO:teuthology.orchestra.run.smithi099.stdout: "current_time": "2023-05-31T21:17:43Z" 2023-05-31T21:17:43.698 INFO:teuthology.orchestra.run.smithi099.stdout:} 2023-05-31T21:17:43.699 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "2e6123d2-d695-4350-9a01-61a455bc782b", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1685566501.757183_260.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-05-31T20:55:01.757183Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1685566501.735717_251.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-05-31T20:55:01.735717Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1685562970.274572_93.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-05-31T19:56:10.274572Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1685567548.445367_776.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-05-31T21:12:28.445367Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-05-31T21:17:43Z" } 2023-05-31T21:17:43.700 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1685566501.757183_260.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-05-31T20:55:01.757183Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1685566501.735717_251.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-05-31T20:55:01.735717Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1685562970.274572_93.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-05-31T19:56:10.274572Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1685567548.445367_776.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-05-31T21:12:28.445367Z', 'realm_epoch': 2}}] 2023-05-31T21:17:43.700 DEBUG:rgw_multi.tests:log_status={0: '1_1685566501.757183_260.1', 1: '1_1685566501.735717_251.1', 2: '1_1685562970.274572_93.1', 3: '1_1685567847.936050_900.1'} 2023-05-31T21:17:43.700 DEBUG:rgw_multi.tests:sync_status={0: '1_1685566501.757183_260.1', 1: '1_1685566501.735717_251.1', 2: '1_1685562970.274572_93.1', 3: '1_1685567548.445367_776.1'} 2023-05-31T21:17:43.701 WARNING:rgw_multi.tests:zone a2 behind master: shard=3 master=1_1685567847.936050_900.1 target=1_1685567548.445367_776.1 2023-05-31T21:17:48.702 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-31T21:17:48.703 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-31T21:17:48.704 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-05-31T21:17:48.761 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2023-05-31T21:17:48.761 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2023-05-31T21:17:48.798 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T21:17:48.794+0000 7f21481fc780 20 RGW-SYNC:meta: read sync status 2023-05-31T21:17:48.798 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T21:17:48.795+0000 7f21481fc780 20 RGW-SYNC:meta: read sync status 2023-05-31T21:17:48.812 INFO:teuthology.orchestra.run.smithi099.stdout:{ 2023-05-31T21:17:48.812 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_status": { 2023-05-31T21:17:48.813 INFO:teuthology.orchestra.run.smithi099.stdout: "info": { 2023-05-31T21:17:48.813 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "sync", 2023-05-31T21:17:48.813 INFO:teuthology.orchestra.run.smithi099.stdout: "num_shards": 4, 2023-05-31T21:17:48.813 INFO:teuthology.orchestra.run.smithi099.stdout: "period": "2e6123d2-d695-4350-9a01-61a455bc782b", 2023-05-31T21:17:48.813 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2023-05-31T21:17:48.813 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:17:48.813 INFO:teuthology.orchestra.run.smithi099.stdout: "markers": [ 2023-05-31T21:17:48.814 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T21:17:48.814 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 0, 2023-05-31T21:17:48.814 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T21:17:48.814 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2023-05-31T21:17:48.814 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1685566501.757183_260.1", 2023-05-31T21:17:48.814 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T21:17:48.815 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 1, 2023-05-31T21:17:48.815 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T21:17:48.815 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2023-05-31T20:55:01.757183Z", 2023-05-31T21:17:48.815 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2023-05-31T21:17:48.815 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T21:17:48.815 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:17:48.815 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T21:17:48.816 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 1, 2023-05-31T21:17:48.816 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T21:17:48.816 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2023-05-31T21:17:48.816 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1685566501.735717_251.1", 2023-05-31T21:17:48.816 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T21:17:48.816 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2023-05-31T21:17:48.817 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T21:17:48.817 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2023-05-31T20:55:01.735717Z", 2023-05-31T21:17:48.817 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2023-05-31T21:17:48.817 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T21:17:48.817 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:17:48.817 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T21:17:48.817 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 2, 2023-05-31T21:17:48.818 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T21:17:48.818 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2023-05-31T21:17:48.818 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1685562970.274572_93.1", 2023-05-31T21:17:48.818 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T21:17:48.818 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2023-05-31T21:17:48.818 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T21:17:48.818 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2023-05-31T19:56:10.274572Z", 2023-05-31T21:17:48.819 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2023-05-31T21:17:48.819 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T21:17:48.819 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:17:48.819 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T21:17:48.819 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 3, 2023-05-31T21:17:48.819 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T21:17:48.819 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2023-05-31T21:17:48.820 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1685567548.445367_776.1", 2023-05-31T21:17:48.820 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T21:17:48.820 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2023-05-31T21:17:48.820 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T21:17:48.820 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2023-05-31T21:12:28.445367Z", 2023-05-31T21:17:48.820 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2023-05-31T21:17:48.820 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T21:17:48.821 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T21:17:48.821 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2023-05-31T21:17:48.821 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:17:48.821 INFO:teuthology.orchestra.run.smithi099.stdout: "full_sync": { 2023-05-31T21:17:48.821 INFO:teuthology.orchestra.run.smithi099.stdout: "total": 1, 2023-05-31T21:17:48.821 INFO:teuthology.orchestra.run.smithi099.stdout: "complete": 1 2023-05-31T21:17:48.821 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:17:48.822 INFO:teuthology.orchestra.run.smithi099.stdout: "current_time": "2023-05-31T21:17:48Z" 2023-05-31T21:17:48.822 INFO:teuthology.orchestra.run.smithi099.stdout:} 2023-05-31T21:17:48.823 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "2e6123d2-d695-4350-9a01-61a455bc782b", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1685566501.757183_260.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-05-31T20:55:01.757183Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1685566501.735717_251.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-05-31T20:55:01.735717Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1685562970.274572_93.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-05-31T19:56:10.274572Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1685567548.445367_776.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-05-31T21:12:28.445367Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-05-31T21:17:48Z" } 2023-05-31T21:17:48.823 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1685566501.757183_260.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-05-31T20:55:01.757183Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1685566501.735717_251.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-05-31T20:55:01.735717Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1685562970.274572_93.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-05-31T19:56:10.274572Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1685567548.445367_776.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-05-31T21:12:28.445367Z', 'realm_epoch': 2}}] 2023-05-31T21:17:48.824 DEBUG:rgw_multi.tests:log_status={0: '1_1685566501.757183_260.1', 1: '1_1685566501.735717_251.1', 2: '1_1685562970.274572_93.1', 3: '1_1685567847.936050_900.1'} 2023-05-31T21:17:48.824 DEBUG:rgw_multi.tests:sync_status={0: '1_1685566501.757183_260.1', 1: '1_1685566501.735717_251.1', 2: '1_1685562970.274572_93.1', 3: '1_1685567548.445367_776.1'} 2023-05-31T21:17:48.824 WARNING:rgw_multi.tests:zone a2 behind master: shard=3 master=1_1685567847.936050_900.1 target=1_1685567548.445367_776.1 2023-05-31T21:17:53.825 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-31T21:17:53.826 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-31T21:17:53.826 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-05-31T21:17:53.883 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2023-05-31T21:17:53.883 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2023-05-31T21:17:53.916 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T21:17:53.913+0000 7f2f86d45780 20 RGW-SYNC:meta: read sync status 2023-05-31T21:17:53.917 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T21:17:53.913+0000 7f2f86d45780 20 RGW-SYNC:meta: read sync status 2023-05-31T21:17:53.931 INFO:teuthology.orchestra.run.smithi099.stdout:{ 2023-05-31T21:17:53.931 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_status": { 2023-05-31T21:17:53.932 INFO:teuthology.orchestra.run.smithi099.stdout: "info": { 2023-05-31T21:17:53.932 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "sync", 2023-05-31T21:17:53.932 INFO:teuthology.orchestra.run.smithi099.stdout: "num_shards": 4, 2023-05-31T21:17:53.933 INFO:teuthology.orchestra.run.smithi099.stdout: "period": "2e6123d2-d695-4350-9a01-61a455bc782b", 2023-05-31T21:17:53.933 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2023-05-31T21:17:53.933 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:17:53.933 INFO:teuthology.orchestra.run.smithi099.stdout: "markers": [ 2023-05-31T21:17:53.933 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T21:17:53.933 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 0, 2023-05-31T21:17:53.933 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T21:17:53.934 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2023-05-31T21:17:53.934 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1685566501.757183_260.1", 2023-05-31T21:17:53.934 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T21:17:53.934 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 1, 2023-05-31T21:17:53.934 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T21:17:53.934 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2023-05-31T20:55:01.757183Z", 2023-05-31T21:17:53.935 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2023-05-31T21:17:53.935 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T21:17:53.935 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:17:53.935 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T21:17:53.935 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 1, 2023-05-31T21:17:53.935 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T21:17:53.935 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2023-05-31T21:17:53.936 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1685566501.735717_251.1", 2023-05-31T21:17:53.936 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T21:17:53.936 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2023-05-31T21:17:53.936 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T21:17:53.936 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2023-05-31T20:55:01.735717Z", 2023-05-31T21:17:53.936 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2023-05-31T21:17:53.936 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T21:17:53.937 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:17:53.937 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T21:17:53.937 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 2, 2023-05-31T21:17:53.937 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T21:17:53.937 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2023-05-31T21:17:53.937 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1685562970.274572_93.1", 2023-05-31T21:17:53.938 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T21:17:53.938 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2023-05-31T21:17:53.938 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T21:17:53.938 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2023-05-31T19:56:10.274572Z", 2023-05-31T21:17:53.938 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2023-05-31T21:17:53.938 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T21:17:53.938 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:17:53.939 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T21:17:53.939 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 3, 2023-05-31T21:17:53.939 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T21:17:53.939 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2023-05-31T21:17:53.939 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1685567548.445367_776.1", 2023-05-31T21:17:53.939 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T21:17:53.939 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2023-05-31T21:17:53.940 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T21:17:53.940 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2023-05-31T21:12:28.445367Z", 2023-05-31T21:17:53.940 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2023-05-31T21:17:53.940 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T21:17:53.940 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T21:17:53.940 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2023-05-31T21:17:53.940 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:17:53.941 INFO:teuthology.orchestra.run.smithi099.stdout: "full_sync": { 2023-05-31T21:17:53.941 INFO:teuthology.orchestra.run.smithi099.stdout: "total": 1, 2023-05-31T21:17:53.941 INFO:teuthology.orchestra.run.smithi099.stdout: "complete": 1 2023-05-31T21:17:53.941 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:17:53.941 INFO:teuthology.orchestra.run.smithi099.stdout: "current_time": "2023-05-31T21:17:53Z" 2023-05-31T21:17:53.941 INFO:teuthology.orchestra.run.smithi099.stdout:} 2023-05-31T21:17:53.942 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "2e6123d2-d695-4350-9a01-61a455bc782b", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1685566501.757183_260.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-05-31T20:55:01.757183Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1685566501.735717_251.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-05-31T20:55:01.735717Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1685562970.274572_93.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-05-31T19:56:10.274572Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1685567548.445367_776.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-05-31T21:12:28.445367Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-05-31T21:17:53Z" } 2023-05-31T21:17:53.943 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1685566501.757183_260.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-05-31T20:55:01.757183Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1685566501.735717_251.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-05-31T20:55:01.735717Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1685562970.274572_93.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-05-31T19:56:10.274572Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1685567548.445367_776.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-05-31T21:12:28.445367Z', 'realm_epoch': 2}}] 2023-05-31T21:17:53.943 DEBUG:rgw_multi.tests:log_status={0: '1_1685566501.757183_260.1', 1: '1_1685566501.735717_251.1', 2: '1_1685562970.274572_93.1', 3: '1_1685567847.936050_900.1'} 2023-05-31T21:17:53.944 DEBUG:rgw_multi.tests:sync_status={0: '1_1685566501.757183_260.1', 1: '1_1685566501.735717_251.1', 2: '1_1685562970.274572_93.1', 3: '1_1685567548.445367_776.1'} 2023-05-31T21:17:53.944 WARNING:rgw_multi.tests:zone a2 behind master: shard=3 master=1_1685567847.936050_900.1 target=1_1685567548.445367_776.1 2023-05-31T21:17:58.945 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-31T21:17:58.946 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-31T21:17:58.947 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-05-31T21:17:59.004 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2023-05-31T21:17:59.005 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2023-05-31T21:17:59.045 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T21:17:59.042+0000 7ff611171780 20 RGW-SYNC:meta: read sync status 2023-05-31T21:17:59.047 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T21:17:59.043+0000 7ff611171780 20 RGW-SYNC:meta: read sync status 2023-05-31T21:17:59.065 INFO:teuthology.orchestra.run.smithi099.stdout:{ 2023-05-31T21:17:59.066 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_status": { 2023-05-31T21:17:59.066 INFO:teuthology.orchestra.run.smithi099.stdout: "info": { 2023-05-31T21:17:59.066 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "sync", 2023-05-31T21:17:59.066 INFO:teuthology.orchestra.run.smithi099.stdout: "num_shards": 4, 2023-05-31T21:17:59.066 INFO:teuthology.orchestra.run.smithi099.stdout: "period": "2e6123d2-d695-4350-9a01-61a455bc782b", 2023-05-31T21:17:59.067 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2023-05-31T21:17:59.067 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:17:59.067 INFO:teuthology.orchestra.run.smithi099.stdout: "markers": [ 2023-05-31T21:17:59.067 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T21:17:59.067 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 0, 2023-05-31T21:17:59.067 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T21:17:59.067 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2023-05-31T21:17:59.068 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1685566501.757183_260.1", 2023-05-31T21:17:59.068 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T21:17:59.068 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 1, 2023-05-31T21:17:59.069 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T21:17:59.069 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2023-05-31T20:55:01.757183Z", 2023-05-31T21:17:59.069 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2023-05-31T21:17:59.069 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T21:17:59.070 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:17:59.070 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T21:17:59.070 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 1, 2023-05-31T21:17:59.071 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T21:17:59.071 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2023-05-31T21:17:59.071 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1685566501.735717_251.1", 2023-05-31T21:17:59.072 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T21:17:59.072 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2023-05-31T21:17:59.072 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T21:17:59.073 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2023-05-31T20:55:01.735717Z", 2023-05-31T21:17:59.073 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2023-05-31T21:17:59.073 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T21:17:59.073 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:17:59.074 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T21:17:59.074 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 2, 2023-05-31T21:17:59.074 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T21:17:59.075 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2023-05-31T21:17:59.075 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1685562970.274572_93.1", 2023-05-31T21:17:59.075 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T21:17:59.075 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2023-05-31T21:17:59.075 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T21:17:59.075 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2023-05-31T19:56:10.274572Z", 2023-05-31T21:17:59.076 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2023-05-31T21:17:59.076 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T21:17:59.076 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:17:59.076 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T21:17:59.076 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 3, 2023-05-31T21:17:59.076 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T21:17:59.076 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2023-05-31T21:17:59.077 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1685567548.445367_776.1", 2023-05-31T21:17:59.077 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T21:17:59.077 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2023-05-31T21:17:59.077 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T21:17:59.077 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2023-05-31T21:12:28.445367Z", 2023-05-31T21:17:59.077 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2023-05-31T21:17:59.077 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T21:17:59.078 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T21:17:59.078 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2023-05-31T21:17:59.078 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:17:59.078 INFO:teuthology.orchestra.run.smithi099.stdout: "full_sync": { 2023-05-31T21:17:59.078 INFO:teuthology.orchestra.run.smithi099.stdout: "total": 1, 2023-05-31T21:17:59.078 INFO:teuthology.orchestra.run.smithi099.stdout: "complete": 1 2023-05-31T21:17:59.078 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:17:59.079 INFO:teuthology.orchestra.run.smithi099.stdout: "current_time": "2023-05-31T21:17:59Z" 2023-05-31T21:17:59.079 INFO:teuthology.orchestra.run.smithi099.stdout:} 2023-05-31T21:17:59.080 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "2e6123d2-d695-4350-9a01-61a455bc782b", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1685566501.757183_260.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-05-31T20:55:01.757183Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1685566501.735717_251.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-05-31T20:55:01.735717Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1685562970.274572_93.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-05-31T19:56:10.274572Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1685567548.445367_776.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-05-31T21:12:28.445367Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-05-31T21:17:59Z" } 2023-05-31T21:17:59.080 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1685566501.757183_260.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-05-31T20:55:01.757183Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1685566501.735717_251.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-05-31T20:55:01.735717Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1685562970.274572_93.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-05-31T19:56:10.274572Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1685567548.445367_776.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-05-31T21:12:28.445367Z', 'realm_epoch': 2}}] 2023-05-31T21:17:59.081 DEBUG:rgw_multi.tests:log_status={0: '1_1685566501.757183_260.1', 1: '1_1685566501.735717_251.1', 2: '1_1685562970.274572_93.1', 3: '1_1685567847.936050_900.1'} 2023-05-31T21:17:59.081 DEBUG:rgw_multi.tests:sync_status={0: '1_1685566501.757183_260.1', 1: '1_1685566501.735717_251.1', 2: '1_1685562970.274572_93.1', 3: '1_1685567548.445367_776.1'} 2023-05-31T21:17:59.082 WARNING:rgw_multi.tests:zone a2 behind master: shard=3 master=1_1685567847.936050_900.1 target=1_1685567548.445367_776.1 2023-05-31T21:18:04.084 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-31T21:18:04.085 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-31T21:18:04.085 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-05-31T21:18:04.143 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2023-05-31T21:18:04.144 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2023-05-31T21:18:04.192 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T21:18:04.188+0000 7fe4372da780 20 RGW-SYNC:meta: read sync status 2023-05-31T21:18:04.193 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T21:18:04.189+0000 7fe4372da780 20 RGW-SYNC:meta: read sync status 2023-05-31T21:18:04.211 INFO:teuthology.orchestra.run.smithi099.stdout:{ 2023-05-31T21:18:04.211 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_status": { 2023-05-31T21:18:04.212 INFO:teuthology.orchestra.run.smithi099.stdout: "info": { 2023-05-31T21:18:04.212 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "sync", 2023-05-31T21:18:04.212 INFO:teuthology.orchestra.run.smithi099.stdout: "num_shards": 4, 2023-05-31T21:18:04.212 INFO:teuthology.orchestra.run.smithi099.stdout: "period": "2e6123d2-d695-4350-9a01-61a455bc782b", 2023-05-31T21:18:04.212 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2023-05-31T21:18:04.212 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:18:04.212 INFO:teuthology.orchestra.run.smithi099.stdout: "markers": [ 2023-05-31T21:18:04.213 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T21:18:04.213 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 0, 2023-05-31T21:18:04.213 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T21:18:04.213 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2023-05-31T21:18:04.213 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1685566501.757183_260.1", 2023-05-31T21:18:04.213 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T21:18:04.213 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 1, 2023-05-31T21:18:04.214 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T21:18:04.214 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2023-05-31T20:55:01.757183Z", 2023-05-31T21:18:04.214 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2023-05-31T21:18:04.214 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T21:18:04.214 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:18:04.214 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T21:18:04.215 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 1, 2023-05-31T21:18:04.215 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T21:18:04.215 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2023-05-31T21:18:04.215 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1685566501.735717_251.1", 2023-05-31T21:18:04.215 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T21:18:04.215 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2023-05-31T21:18:04.215 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T21:18:04.216 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2023-05-31T20:55:01.735717Z", 2023-05-31T21:18:04.216 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2023-05-31T21:18:04.216 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T21:18:04.216 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:18:04.216 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T21:18:04.216 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 2, 2023-05-31T21:18:04.216 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T21:18:04.217 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2023-05-31T21:18:04.217 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1685562970.274572_93.1", 2023-05-31T21:18:04.217 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T21:18:04.217 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2023-05-31T21:18:04.217 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T21:18:04.217 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2023-05-31T19:56:10.274572Z", 2023-05-31T21:18:04.217 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2023-05-31T21:18:04.218 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T21:18:04.218 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:18:04.218 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T21:18:04.218 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 3, 2023-05-31T21:18:04.218 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T21:18:04.218 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2023-05-31T21:18:04.218 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1685567548.445367_776.1", 2023-05-31T21:18:04.219 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T21:18:04.219 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2023-05-31T21:18:04.219 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T21:18:04.219 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2023-05-31T21:12:28.445367Z", 2023-05-31T21:18:04.219 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2023-05-31T21:18:04.219 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T21:18:04.219 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T21:18:04.220 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2023-05-31T21:18:04.220 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:18:04.220 INFO:teuthology.orchestra.run.smithi099.stdout: "full_sync": { 2023-05-31T21:18:04.220 INFO:teuthology.orchestra.run.smithi099.stdout: "total": 1, 2023-05-31T21:18:04.220 INFO:teuthology.orchestra.run.smithi099.stdout: "complete": 1 2023-05-31T21:18:04.220 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:18:04.220 INFO:teuthology.orchestra.run.smithi099.stdout: "current_time": "2023-05-31T21:18:04Z" 2023-05-31T21:18:04.221 INFO:teuthology.orchestra.run.smithi099.stdout:} 2023-05-31T21:18:04.222 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "2e6123d2-d695-4350-9a01-61a455bc782b", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1685566501.757183_260.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-05-31T20:55:01.757183Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1685566501.735717_251.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-05-31T20:55:01.735717Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1685562970.274572_93.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-05-31T19:56:10.274572Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1685567548.445367_776.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-05-31T21:12:28.445367Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-05-31T21:18:04Z" } 2023-05-31T21:18:04.222 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1685566501.757183_260.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-05-31T20:55:01.757183Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1685566501.735717_251.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-05-31T20:55:01.735717Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1685562970.274572_93.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-05-31T19:56:10.274572Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1685567548.445367_776.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-05-31T21:12:28.445367Z', 'realm_epoch': 2}}] 2023-05-31T21:18:04.222 DEBUG:rgw_multi.tests:log_status={0: '1_1685566501.757183_260.1', 1: '1_1685566501.735717_251.1', 2: '1_1685562970.274572_93.1', 3: '1_1685567847.936050_900.1'} 2023-05-31T21:18:04.223 DEBUG:rgw_multi.tests:sync_status={0: '1_1685566501.757183_260.1', 1: '1_1685566501.735717_251.1', 2: '1_1685562970.274572_93.1', 3: '1_1685567548.445367_776.1'} 2023-05-31T21:18:04.223 WARNING:rgw_multi.tests:zone a2 behind master: shard=3 master=1_1685567847.936050_900.1 target=1_1685567548.445367_776.1 2023-05-31T21:18:09.224 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-31T21:18:09.225 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-31T21:18:09.226 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-05-31T21:18:09.283 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2023-05-31T21:18:09.283 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2023-05-31T21:18:09.318 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T21:18:09.314+0000 7f72f0901780 20 RGW-SYNC:meta: read sync status 2023-05-31T21:18:09.319 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T21:18:09.315+0000 7f72f0901780 20 RGW-SYNC:meta: read sync status 2023-05-31T21:18:09.336 INFO:teuthology.orchestra.run.smithi099.stdout:{ 2023-05-31T21:18:09.337 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_status": { 2023-05-31T21:18:09.337 INFO:teuthology.orchestra.run.smithi099.stdout: "info": { 2023-05-31T21:18:09.337 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "sync", 2023-05-31T21:18:09.337 INFO:teuthology.orchestra.run.smithi099.stdout: "num_shards": 4, 2023-05-31T21:18:09.337 INFO:teuthology.orchestra.run.smithi099.stdout: "period": "2e6123d2-d695-4350-9a01-61a455bc782b", 2023-05-31T21:18:09.338 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2023-05-31T21:18:09.338 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:18:09.338 INFO:teuthology.orchestra.run.smithi099.stdout: "markers": [ 2023-05-31T21:18:09.338 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T21:18:09.338 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 0, 2023-05-31T21:18:09.338 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T21:18:09.338 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2023-05-31T21:18:09.339 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1685566501.757183_260.1", 2023-05-31T21:18:09.339 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T21:18:09.339 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 1, 2023-05-31T21:18:09.339 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T21:18:09.340 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2023-05-31T20:55:01.757183Z", 2023-05-31T21:18:09.340 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2023-05-31T21:18:09.340 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T21:18:09.340 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:18:09.340 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T21:18:09.340 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 1, 2023-05-31T21:18:09.341 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T21:18:09.341 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2023-05-31T21:18:09.341 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1685566501.735717_251.1", 2023-05-31T21:18:09.341 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T21:18:09.341 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2023-05-31T21:18:09.341 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T21:18:09.341 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2023-05-31T20:55:01.735717Z", 2023-05-31T21:18:09.342 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2023-05-31T21:18:09.342 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T21:18:09.342 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:18:09.342 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T21:18:09.342 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 2, 2023-05-31T21:18:09.342 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T21:18:09.342 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2023-05-31T21:18:09.343 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1685562970.274572_93.1", 2023-05-31T21:18:09.343 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T21:18:09.343 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2023-05-31T21:18:09.343 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T21:18:09.343 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2023-05-31T19:56:10.274572Z", 2023-05-31T21:18:09.343 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2023-05-31T21:18:09.344 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T21:18:09.344 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:18:09.344 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T21:18:09.344 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 3, 2023-05-31T21:18:09.344 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T21:18:09.344 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2023-05-31T21:18:09.344 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1685567548.445367_776.1", 2023-05-31T21:18:09.345 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T21:18:09.345 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2023-05-31T21:18:09.345 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T21:18:09.346 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2023-05-31T21:12:28.445367Z", 2023-05-31T21:18:09.346 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2023-05-31T21:18:09.347 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T21:18:09.347 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T21:18:09.347 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2023-05-31T21:18:09.348 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:18:09.348 INFO:teuthology.orchestra.run.smithi099.stdout: "full_sync": { 2023-05-31T21:18:09.348 INFO:teuthology.orchestra.run.smithi099.stdout: "total": 1, 2023-05-31T21:18:09.349 INFO:teuthology.orchestra.run.smithi099.stdout: "complete": 1 2023-05-31T21:18:09.349 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:18:09.349 INFO:teuthology.orchestra.run.smithi099.stdout: "current_time": "2023-05-31T21:18:09Z" 2023-05-31T21:18:09.350 INFO:teuthology.orchestra.run.smithi099.stdout:} 2023-05-31T21:18:09.351 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "2e6123d2-d695-4350-9a01-61a455bc782b", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1685566501.757183_260.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-05-31T20:55:01.757183Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1685566501.735717_251.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-05-31T20:55:01.735717Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1685562970.274572_93.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-05-31T19:56:10.274572Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1685567548.445367_776.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-05-31T21:12:28.445367Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-05-31T21:18:09Z" } 2023-05-31T21:18:09.352 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1685566501.757183_260.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-05-31T20:55:01.757183Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1685566501.735717_251.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-05-31T20:55:01.735717Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1685562970.274572_93.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-05-31T19:56:10.274572Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1685567548.445367_776.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-05-31T21:12:28.445367Z', 'realm_epoch': 2}}] 2023-05-31T21:18:09.353 DEBUG:rgw_multi.tests:log_status={0: '1_1685566501.757183_260.1', 1: '1_1685566501.735717_251.1', 2: '1_1685562970.274572_93.1', 3: '1_1685567847.936050_900.1'} 2023-05-31T21:18:09.353 DEBUG:rgw_multi.tests:sync_status={0: '1_1685566501.757183_260.1', 1: '1_1685566501.735717_251.1', 2: '1_1685562970.274572_93.1', 3: '1_1685567548.445367_776.1'} 2023-05-31T21:18:09.354 WARNING:rgw_multi.tests:zone a2 behind master: shard=3 master=1_1685567847.936050_900.1 target=1_1685567548.445367_776.1 2023-05-31T21:18:14.355 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-31T21:18:14.355 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-31T21:18:14.356 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-05-31T21:18:14.413 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2023-05-31T21:18:14.414 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2023-05-31T21:18:14.454 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T21:18:14.450+0000 7f016fb1c780 20 RGW-SYNC:meta: read sync status 2023-05-31T21:18:14.455 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T21:18:14.451+0000 7f016fb1c780 20 RGW-SYNC:meta: read sync status 2023-05-31T21:18:14.471 INFO:teuthology.orchestra.run.smithi099.stdout:{ 2023-05-31T21:18:14.472 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_status": { 2023-05-31T21:18:14.472 INFO:teuthology.orchestra.run.smithi099.stdout: "info": { 2023-05-31T21:18:14.472 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "sync", 2023-05-31T21:18:14.472 INFO:teuthology.orchestra.run.smithi099.stdout: "num_shards": 4, 2023-05-31T21:18:14.473 INFO:teuthology.orchestra.run.smithi099.stdout: "period": "2e6123d2-d695-4350-9a01-61a455bc782b", 2023-05-31T21:18:14.473 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2023-05-31T21:18:14.473 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:18:14.473 INFO:teuthology.orchestra.run.smithi099.stdout: "markers": [ 2023-05-31T21:18:14.473 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T21:18:14.473 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 0, 2023-05-31T21:18:14.474 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T21:18:14.474 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2023-05-31T21:18:14.474 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1685566501.757183_260.1", 2023-05-31T21:18:14.474 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T21:18:14.474 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 1, 2023-05-31T21:18:14.474 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T21:18:14.475 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2023-05-31T20:55:01.757183Z", 2023-05-31T21:18:14.475 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2023-05-31T21:18:14.475 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T21:18:14.475 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:18:14.475 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T21:18:14.475 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 1, 2023-05-31T21:18:14.475 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T21:18:14.476 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2023-05-31T21:18:14.476 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1685566501.735717_251.1", 2023-05-31T21:18:14.476 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T21:18:14.476 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2023-05-31T21:18:14.476 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T21:18:14.476 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2023-05-31T20:55:01.735717Z", 2023-05-31T21:18:14.476 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2023-05-31T21:18:14.477 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T21:18:14.477 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:18:14.477 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T21:18:14.477 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 2, 2023-05-31T21:18:14.477 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T21:18:14.477 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2023-05-31T21:18:14.477 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1685562970.274572_93.1", 2023-05-31T21:18:14.478 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T21:18:14.478 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2023-05-31T21:18:14.478 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T21:18:14.478 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2023-05-31T19:56:10.274572Z", 2023-05-31T21:18:14.478 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2023-05-31T21:18:14.478 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T21:18:14.478 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:18:14.479 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T21:18:14.479 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 3, 2023-05-31T21:18:14.479 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T21:18:14.479 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2023-05-31T21:18:14.479 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1685567548.445367_776.1", 2023-05-31T21:18:14.479 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T21:18:14.479 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2023-05-31T21:18:14.480 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T21:18:14.480 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2023-05-31T21:12:28.445367Z", 2023-05-31T21:18:14.480 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2023-05-31T21:18:14.480 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T21:18:14.480 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T21:18:14.480 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2023-05-31T21:18:14.481 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:18:14.481 INFO:teuthology.orchestra.run.smithi099.stdout: "full_sync": { 2023-05-31T21:18:14.481 INFO:teuthology.orchestra.run.smithi099.stdout: "total": 1, 2023-05-31T21:18:14.481 INFO:teuthology.orchestra.run.smithi099.stdout: "complete": 1 2023-05-31T21:18:14.481 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:18:14.481 INFO:teuthology.orchestra.run.smithi099.stdout: "current_time": "2023-05-31T21:18:14Z" 2023-05-31T21:18:14.481 INFO:teuthology.orchestra.run.smithi099.stdout:} 2023-05-31T21:18:14.482 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "2e6123d2-d695-4350-9a01-61a455bc782b", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1685566501.757183_260.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-05-31T20:55:01.757183Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1685566501.735717_251.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-05-31T20:55:01.735717Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1685562970.274572_93.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-05-31T19:56:10.274572Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1685567548.445367_776.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-05-31T21:12:28.445367Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-05-31T21:18:14Z" } 2023-05-31T21:18:14.483 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1685566501.757183_260.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-05-31T20:55:01.757183Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1685566501.735717_251.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-05-31T20:55:01.735717Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1685562970.274572_93.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-05-31T19:56:10.274572Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1685567548.445367_776.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-05-31T21:12:28.445367Z', 'realm_epoch': 2}}] 2023-05-31T21:18:14.483 DEBUG:rgw_multi.tests:log_status={0: '1_1685566501.757183_260.1', 1: '1_1685566501.735717_251.1', 2: '1_1685562970.274572_93.1', 3: '1_1685567847.936050_900.1'} 2023-05-31T21:18:14.484 DEBUG:rgw_multi.tests:sync_status={0: '1_1685566501.757183_260.1', 1: '1_1685566501.735717_251.1', 2: '1_1685562970.274572_93.1', 3: '1_1685567548.445367_776.1'} 2023-05-31T21:18:14.484 WARNING:rgw_multi.tests:zone a2 behind master: shard=3 master=1_1685567847.936050_900.1 target=1_1685567548.445367_776.1 2023-05-31T21:18:19.485 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-31T21:18:19.486 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-31T21:18:19.486 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-05-31T21:18:19.543 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2023-05-31T21:18:19.544 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2023-05-31T21:18:19.578 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T21:18:19.574+0000 7fc06369c780 20 RGW-SYNC:meta: read sync status 2023-05-31T21:18:19.579 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T21:18:19.575+0000 7fc06369c780 20 RGW-SYNC:meta: read sync status 2023-05-31T21:18:19.593 INFO:teuthology.orchestra.run.smithi099.stdout:{ 2023-05-31T21:18:19.594 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_status": { 2023-05-31T21:18:19.594 INFO:teuthology.orchestra.run.smithi099.stdout: "info": { 2023-05-31T21:18:19.594 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "sync", 2023-05-31T21:18:19.594 INFO:teuthology.orchestra.run.smithi099.stdout: "num_shards": 4, 2023-05-31T21:18:19.594 INFO:teuthology.orchestra.run.smithi099.stdout: "period": "2e6123d2-d695-4350-9a01-61a455bc782b", 2023-05-31T21:18:19.595 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2023-05-31T21:18:19.595 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:18:19.595 INFO:teuthology.orchestra.run.smithi099.stdout: "markers": [ 2023-05-31T21:18:19.595 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T21:18:19.595 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 0, 2023-05-31T21:18:19.595 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T21:18:19.595 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2023-05-31T21:18:19.596 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1685566501.757183_260.1", 2023-05-31T21:18:19.596 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T21:18:19.596 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 1, 2023-05-31T21:18:19.596 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T21:18:19.596 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2023-05-31T20:55:01.757183Z", 2023-05-31T21:18:19.596 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2023-05-31T21:18:19.596 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T21:18:19.597 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:18:19.597 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T21:18:19.597 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 1, 2023-05-31T21:18:19.597 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T21:18:19.597 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2023-05-31T21:18:19.597 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1685566501.735717_251.1", 2023-05-31T21:18:19.597 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T21:18:19.598 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2023-05-31T21:18:19.598 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T21:18:19.598 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2023-05-31T20:55:01.735717Z", 2023-05-31T21:18:19.598 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2023-05-31T21:18:19.598 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T21:18:19.598 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:18:19.598 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T21:18:19.599 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 2, 2023-05-31T21:18:19.599 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T21:18:19.599 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2023-05-31T21:18:19.599 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1685562970.274572_93.1", 2023-05-31T21:18:19.599 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T21:18:19.599 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2023-05-31T21:18:19.600 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T21:18:19.600 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2023-05-31T19:56:10.274572Z", 2023-05-31T21:18:19.600 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2023-05-31T21:18:19.600 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T21:18:19.600 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:18:19.600 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T21:18:19.600 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 3, 2023-05-31T21:18:19.601 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T21:18:19.601 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2023-05-31T21:18:19.601 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1685567548.445367_776.1", 2023-05-31T21:18:19.601 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T21:18:19.601 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2023-05-31T21:18:19.602 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T21:18:19.602 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2023-05-31T21:12:28.445367Z", 2023-05-31T21:18:19.602 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2023-05-31T21:18:19.602 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T21:18:19.602 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T21:18:19.602 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2023-05-31T21:18:19.602 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:18:19.603 INFO:teuthology.orchestra.run.smithi099.stdout: "full_sync": { 2023-05-31T21:18:19.603 INFO:teuthology.orchestra.run.smithi099.stdout: "total": 1, 2023-05-31T21:18:19.603 INFO:teuthology.orchestra.run.smithi099.stdout: "complete": 1 2023-05-31T21:18:19.603 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:18:19.603 INFO:teuthology.orchestra.run.smithi099.stdout: "current_time": "2023-05-31T21:18:19Z" 2023-05-31T21:18:19.603 INFO:teuthology.orchestra.run.smithi099.stdout:} 2023-05-31T21:18:19.604 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "2e6123d2-d695-4350-9a01-61a455bc782b", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1685566501.757183_260.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-05-31T20:55:01.757183Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1685566501.735717_251.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-05-31T20:55:01.735717Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1685562970.274572_93.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-05-31T19:56:10.274572Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1685567548.445367_776.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-05-31T21:12:28.445367Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-05-31T21:18:19Z" } 2023-05-31T21:18:19.605 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1685566501.757183_260.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-05-31T20:55:01.757183Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1685566501.735717_251.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-05-31T20:55:01.735717Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1685562970.274572_93.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-05-31T19:56:10.274572Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1685567548.445367_776.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-05-31T21:12:28.445367Z', 'realm_epoch': 2}}] 2023-05-31T21:18:19.605 DEBUG:rgw_multi.tests:log_status={0: '1_1685566501.757183_260.1', 1: '1_1685566501.735717_251.1', 2: '1_1685562970.274572_93.1', 3: '1_1685567847.936050_900.1'} 2023-05-31T21:18:19.605 DEBUG:rgw_multi.tests:sync_status={0: '1_1685566501.757183_260.1', 1: '1_1685566501.735717_251.1', 2: '1_1685562970.274572_93.1', 3: '1_1685567548.445367_776.1'} 2023-05-31T21:18:19.606 WARNING:rgw_multi.tests:zone a2 behind master: shard=3 master=1_1685567847.936050_900.1 target=1_1685567548.445367_776.1 2023-05-31T21:18:24.607 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-31T21:18:24.608 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-31T21:18:24.608 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-05-31T21:18:24.665 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2023-05-31T21:18:24.666 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2023-05-31T21:18:24.716 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T21:18:24.713+0000 7f6c38694780 20 RGW-SYNC:meta: read sync status 2023-05-31T21:18:24.717 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T21:18:24.714+0000 7f6c38694780 20 RGW-SYNC:meta: read sync status 2023-05-31T21:18:24.739 INFO:teuthology.orchestra.run.smithi099.stdout:{ 2023-05-31T21:18:24.739 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_status": { 2023-05-31T21:18:24.739 INFO:teuthology.orchestra.run.smithi099.stdout: "info": { 2023-05-31T21:18:24.739 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "sync", 2023-05-31T21:18:24.740 INFO:teuthology.orchestra.run.smithi099.stdout: "num_shards": 4, 2023-05-31T21:18:24.740 INFO:teuthology.orchestra.run.smithi099.stdout: "period": "2e6123d2-d695-4350-9a01-61a455bc782b", 2023-05-31T21:18:24.740 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2023-05-31T21:18:24.740 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:18:24.740 INFO:teuthology.orchestra.run.smithi099.stdout: "markers": [ 2023-05-31T21:18:24.740 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T21:18:24.740 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 0, 2023-05-31T21:18:24.741 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T21:18:24.741 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2023-05-31T21:18:24.741 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1685566501.757183_260.1", 2023-05-31T21:18:24.741 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T21:18:24.741 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 1, 2023-05-31T21:18:24.741 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T21:18:24.741 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2023-05-31T20:55:01.757183Z", 2023-05-31T21:18:24.742 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2023-05-31T21:18:24.742 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T21:18:24.742 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:18:24.742 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T21:18:24.742 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 1, 2023-05-31T21:18:24.742 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T21:18:24.743 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2023-05-31T21:18:24.743 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1685566501.735717_251.1", 2023-05-31T21:18:24.743 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T21:18:24.743 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2023-05-31T21:18:24.743 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T21:18:24.743 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2023-05-31T20:55:01.735717Z", 2023-05-31T21:18:24.743 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2023-05-31T21:18:24.744 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T21:18:24.744 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:18:24.744 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T21:18:24.744 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 2, 2023-05-31T21:18:24.744 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T21:18:24.744 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2023-05-31T21:18:24.744 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1685562970.274572_93.1", 2023-05-31T21:18:24.745 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T21:18:24.745 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2023-05-31T21:18:24.745 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T21:18:24.745 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2023-05-31T19:56:10.274572Z", 2023-05-31T21:18:24.745 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2023-05-31T21:18:24.745 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T21:18:24.745 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:18:24.746 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T21:18:24.746 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 3, 2023-05-31T21:18:24.746 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T21:18:24.746 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2023-05-31T21:18:24.746 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1685567548.445367_776.1", 2023-05-31T21:18:24.746 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T21:18:24.746 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2023-05-31T21:18:24.747 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T21:18:24.747 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2023-05-31T21:12:28.445367Z", 2023-05-31T21:18:24.747 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2023-05-31T21:18:24.747 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T21:18:24.747 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T21:18:24.747 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2023-05-31T21:18:24.747 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:18:24.748 INFO:teuthology.orchestra.run.smithi099.stdout: "full_sync": { 2023-05-31T21:18:24.748 INFO:teuthology.orchestra.run.smithi099.stdout: "total": 1, 2023-05-31T21:18:24.748 INFO:teuthology.orchestra.run.smithi099.stdout: "complete": 1 2023-05-31T21:18:24.748 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:18:24.748 INFO:teuthology.orchestra.run.smithi099.stdout: "current_time": "2023-05-31T21:18:24Z" 2023-05-31T21:18:24.748 INFO:teuthology.orchestra.run.smithi099.stdout:} 2023-05-31T21:18:24.749 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "2e6123d2-d695-4350-9a01-61a455bc782b", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1685566501.757183_260.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-05-31T20:55:01.757183Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1685566501.735717_251.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-05-31T20:55:01.735717Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1685562970.274572_93.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-05-31T19:56:10.274572Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1685567548.445367_776.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-05-31T21:12:28.445367Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-05-31T21:18:24Z" } 2023-05-31T21:18:24.750 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1685566501.757183_260.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-05-31T20:55:01.757183Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1685566501.735717_251.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-05-31T20:55:01.735717Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1685562970.274572_93.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-05-31T19:56:10.274572Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1685567548.445367_776.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-05-31T21:12:28.445367Z', 'realm_epoch': 2}}] 2023-05-31T21:18:24.750 DEBUG:rgw_multi.tests:log_status={0: '1_1685566501.757183_260.1', 1: '1_1685566501.735717_251.1', 2: '1_1685562970.274572_93.1', 3: '1_1685567847.936050_900.1'} 2023-05-31T21:18:24.751 DEBUG:rgw_multi.tests:sync_status={0: '1_1685566501.757183_260.1', 1: '1_1685566501.735717_251.1', 2: '1_1685562970.274572_93.1', 3: '1_1685567548.445367_776.1'} 2023-05-31T21:18:24.751 WARNING:rgw_multi.tests:zone a2 behind master: shard=3 master=1_1685567847.936050_900.1 target=1_1685567548.445367_776.1 2023-05-31T21:18:29.752 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-31T21:18:29.752 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-31T21:18:29.753 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-05-31T21:18:29.810 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2023-05-31T21:18:29.811 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2023-05-31T21:18:29.845 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T21:18:29.842+0000 7fe04a072780 20 RGW-SYNC:meta: read sync status 2023-05-31T21:18:29.846 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T21:18:29.843+0000 7fe04a072780 20 RGW-SYNC:meta: read sync status 2023-05-31T21:18:29.861 INFO:teuthology.orchestra.run.smithi099.stdout:{ 2023-05-31T21:18:29.861 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_status": { 2023-05-31T21:18:29.861 INFO:teuthology.orchestra.run.smithi099.stdout: "info": { 2023-05-31T21:18:29.862 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "sync", 2023-05-31T21:18:29.862 INFO:teuthology.orchestra.run.smithi099.stdout: "num_shards": 4, 2023-05-31T21:18:29.862 INFO:teuthology.orchestra.run.smithi099.stdout: "period": "2e6123d2-d695-4350-9a01-61a455bc782b", 2023-05-31T21:18:29.862 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2023-05-31T21:18:29.862 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:18:29.862 INFO:teuthology.orchestra.run.smithi099.stdout: "markers": [ 2023-05-31T21:18:29.863 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T21:18:29.863 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 0, 2023-05-31T21:18:29.863 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T21:18:29.863 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2023-05-31T21:18:29.863 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1685566501.757183_260.1", 2023-05-31T21:18:29.863 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T21:18:29.864 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 1, 2023-05-31T21:18:29.864 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T21:18:29.864 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2023-05-31T20:55:01.757183Z", 2023-05-31T21:18:29.864 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2023-05-31T21:18:29.864 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T21:18:29.865 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:18:29.865 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T21:18:29.865 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 1, 2023-05-31T21:18:29.865 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T21:18:29.865 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2023-05-31T21:18:29.865 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1685566501.735717_251.1", 2023-05-31T21:18:29.866 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T21:18:29.866 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2023-05-31T21:18:29.866 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T21:18:29.866 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2023-05-31T20:55:01.735717Z", 2023-05-31T21:18:29.866 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2023-05-31T21:18:29.866 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T21:18:29.867 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:18:29.867 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T21:18:29.867 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 2, 2023-05-31T21:18:29.867 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T21:18:29.867 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2023-05-31T21:18:29.867 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1685562970.274572_93.1", 2023-05-31T21:18:29.868 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T21:18:29.868 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2023-05-31T21:18:29.868 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T21:18:29.868 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2023-05-31T19:56:10.274572Z", 2023-05-31T21:18:29.868 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2023-05-31T21:18:29.868 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T21:18:29.869 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:18:29.869 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T21:18:29.869 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 3, 2023-05-31T21:18:29.869 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T21:18:29.869 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2023-05-31T21:18:29.869 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1685567548.445367_776.1", 2023-05-31T21:18:29.869 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T21:18:29.870 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2023-05-31T21:18:29.870 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T21:18:29.870 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2023-05-31T21:12:28.445367Z", 2023-05-31T21:18:29.870 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2023-05-31T21:18:29.870 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T21:18:29.870 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T21:18:29.871 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2023-05-31T21:18:29.871 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:18:29.871 INFO:teuthology.orchestra.run.smithi099.stdout: "full_sync": { 2023-05-31T21:18:29.871 INFO:teuthology.orchestra.run.smithi099.stdout: "total": 1, 2023-05-31T21:18:29.871 INFO:teuthology.orchestra.run.smithi099.stdout: "complete": 1 2023-05-31T21:18:29.871 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:18:29.871 INFO:teuthology.orchestra.run.smithi099.stdout: "current_time": "2023-05-31T21:18:29Z" 2023-05-31T21:18:29.872 INFO:teuthology.orchestra.run.smithi099.stdout:} 2023-05-31T21:18:29.873 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "2e6123d2-d695-4350-9a01-61a455bc782b", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1685566501.757183_260.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-05-31T20:55:01.757183Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1685566501.735717_251.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-05-31T20:55:01.735717Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1685562970.274572_93.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-05-31T19:56:10.274572Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1685567548.445367_776.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-05-31T21:12:28.445367Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-05-31T21:18:29Z" } 2023-05-31T21:18:29.873 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1685566501.757183_260.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-05-31T20:55:01.757183Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1685566501.735717_251.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-05-31T20:55:01.735717Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1685562970.274572_93.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-05-31T19:56:10.274572Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1685567548.445367_776.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-05-31T21:12:28.445367Z', 'realm_epoch': 2}}] 2023-05-31T21:18:29.874 DEBUG:rgw_multi.tests:log_status={0: '1_1685566501.757183_260.1', 1: '1_1685566501.735717_251.1', 2: '1_1685562970.274572_93.1', 3: '1_1685567847.936050_900.1'} 2023-05-31T21:18:29.874 DEBUG:rgw_multi.tests:sync_status={0: '1_1685566501.757183_260.1', 1: '1_1685566501.735717_251.1', 2: '1_1685562970.274572_93.1', 3: '1_1685567548.445367_776.1'} 2023-05-31T21:18:29.874 WARNING:rgw_multi.tests:zone a2 behind master: shard=3 master=1_1685567847.936050_900.1 target=1_1685567548.445367_776.1 2023-05-31T21:18:34.876 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-31T21:18:34.877 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-31T21:18:34.877 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-05-31T21:18:34.934 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2023-05-31T21:18:34.935 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2023-05-31T21:18:34.983 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T21:18:34.980+0000 7fc1250f4780 20 RGW-SYNC:meta: read sync status 2023-05-31T21:18:34.984 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T21:18:34.981+0000 7fc1250f4780 20 RGW-SYNC:meta: read sync status 2023-05-31T21:18:35.007 INFO:teuthology.orchestra.run.smithi099.stdout:{ 2023-05-31T21:18:35.007 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_status": { 2023-05-31T21:18:35.007 INFO:teuthology.orchestra.run.smithi099.stdout: "info": { 2023-05-31T21:18:35.007 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "sync", 2023-05-31T21:18:35.007 INFO:teuthology.orchestra.run.smithi099.stdout: "num_shards": 4, 2023-05-31T21:18:35.008 INFO:teuthology.orchestra.run.smithi099.stdout: "period": "2e6123d2-d695-4350-9a01-61a455bc782b", 2023-05-31T21:18:35.008 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2023-05-31T21:18:35.008 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:18:35.008 INFO:teuthology.orchestra.run.smithi099.stdout: "markers": [ 2023-05-31T21:18:35.008 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T21:18:35.008 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 0, 2023-05-31T21:18:35.009 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T21:18:35.009 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2023-05-31T21:18:35.009 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1685566501.757183_260.1", 2023-05-31T21:18:35.009 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T21:18:35.009 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 1, 2023-05-31T21:18:35.010 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T21:18:35.010 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2023-05-31T20:55:01.757183Z", 2023-05-31T21:18:35.010 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2023-05-31T21:18:35.010 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T21:18:35.010 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:18:35.010 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T21:18:35.010 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 1, 2023-05-31T21:18:35.011 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T21:18:35.011 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2023-05-31T21:18:35.011 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1685566501.735717_251.1", 2023-05-31T21:18:35.011 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T21:18:35.011 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2023-05-31T21:18:35.011 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T21:18:35.011 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2023-05-31T20:55:01.735717Z", 2023-05-31T21:18:35.012 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2023-05-31T21:18:35.012 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T21:18:35.012 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:18:35.012 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T21:18:35.012 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 2, 2023-05-31T21:18:35.012 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T21:18:35.013 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2023-05-31T21:18:35.013 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1685562970.274572_93.1", 2023-05-31T21:18:35.013 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T21:18:35.013 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2023-05-31T21:18:35.013 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T21:18:35.013 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2023-05-31T19:56:10.274572Z", 2023-05-31T21:18:35.013 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2023-05-31T21:18:35.014 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T21:18:35.014 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:18:35.014 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T21:18:35.014 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 3, 2023-05-31T21:18:35.014 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T21:18:35.014 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2023-05-31T21:18:35.014 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1685567548.445367_776.1", 2023-05-31T21:18:35.015 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T21:18:35.015 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2023-05-31T21:18:35.015 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T21:18:35.015 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2023-05-31T21:12:28.445367Z", 2023-05-31T21:18:35.015 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2023-05-31T21:18:35.015 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T21:18:35.015 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T21:18:35.016 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2023-05-31T21:18:35.016 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:18:35.016 INFO:teuthology.orchestra.run.smithi099.stdout: "full_sync": { 2023-05-31T21:18:35.016 INFO:teuthology.orchestra.run.smithi099.stdout: "total": 1, 2023-05-31T21:18:35.016 INFO:teuthology.orchestra.run.smithi099.stdout: "complete": 1 2023-05-31T21:18:35.016 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:18:35.017 INFO:teuthology.orchestra.run.smithi099.stdout: "current_time": "2023-05-31T21:18:34Z" 2023-05-31T21:18:35.017 INFO:teuthology.orchestra.run.smithi099.stdout:} 2023-05-31T21:18:35.018 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "2e6123d2-d695-4350-9a01-61a455bc782b", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1685566501.757183_260.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-05-31T20:55:01.757183Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1685566501.735717_251.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-05-31T20:55:01.735717Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1685562970.274572_93.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-05-31T19:56:10.274572Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1685567548.445367_776.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-05-31T21:12:28.445367Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-05-31T21:18:34Z" } 2023-05-31T21:18:35.018 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1685566501.757183_260.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-05-31T20:55:01.757183Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1685566501.735717_251.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-05-31T20:55:01.735717Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1685562970.274572_93.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-05-31T19:56:10.274572Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1685567548.445367_776.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-05-31T21:12:28.445367Z', 'realm_epoch': 2}}] 2023-05-31T21:18:35.018 DEBUG:rgw_multi.tests:log_status={0: '1_1685566501.757183_260.1', 1: '1_1685566501.735717_251.1', 2: '1_1685562970.274572_93.1', 3: '1_1685567847.936050_900.1'} 2023-05-31T21:18:35.019 DEBUG:rgw_multi.tests:sync_status={0: '1_1685566501.757183_260.1', 1: '1_1685566501.735717_251.1', 2: '1_1685562970.274572_93.1', 3: '1_1685567548.445367_776.1'} 2023-05-31T21:18:35.019 WARNING:rgw_multi.tests:zone a2 behind master: shard=3 master=1_1685567847.936050_900.1 target=1_1685567548.445367_776.1 2023-05-31T21:18:40.020 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-31T21:18:40.020 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-31T21:18:40.021 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-05-31T21:18:40.079 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2023-05-31T21:18:40.079 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2023-05-31T21:18:40.129 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T21:18:40.126+0000 7f29a3ae6780 20 RGW-SYNC:meta: read sync status 2023-05-31T21:18:40.130 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T21:18:40.127+0000 7f29a3ae6780 20 RGW-SYNC:meta: read sync status 2023-05-31T21:18:40.149 INFO:teuthology.orchestra.run.smithi099.stdout:{ 2023-05-31T21:18:40.149 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_status": { 2023-05-31T21:18:40.149 INFO:teuthology.orchestra.run.smithi099.stdout: "info": { 2023-05-31T21:18:40.149 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "sync", 2023-05-31T21:18:40.149 INFO:teuthology.orchestra.run.smithi099.stdout: "num_shards": 4, 2023-05-31T21:18:40.150 INFO:teuthology.orchestra.run.smithi099.stdout: "period": "2e6123d2-d695-4350-9a01-61a455bc782b", 2023-05-31T21:18:40.150 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2023-05-31T21:18:40.150 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:18:40.150 INFO:teuthology.orchestra.run.smithi099.stdout: "markers": [ 2023-05-31T21:18:40.150 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T21:18:40.150 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 0, 2023-05-31T21:18:40.151 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T21:18:40.151 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2023-05-31T21:18:40.151 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1685566501.757183_260.1", 2023-05-31T21:18:40.151 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T21:18:40.151 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 1, 2023-05-31T21:18:40.151 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T21:18:40.151 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2023-05-31T20:55:01.757183Z", 2023-05-31T21:18:40.152 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2023-05-31T21:18:40.152 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T21:18:40.152 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:18:40.152 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T21:18:40.152 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 1, 2023-05-31T21:18:40.152 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T21:18:40.152 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2023-05-31T21:18:40.153 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1685566501.735717_251.1", 2023-05-31T21:18:40.153 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T21:18:40.153 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2023-05-31T21:18:40.153 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T21:18:40.153 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2023-05-31T20:55:01.735717Z", 2023-05-31T21:18:40.153 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2023-05-31T21:18:40.153 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T21:18:40.154 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:18:40.154 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T21:18:40.154 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 2, 2023-05-31T21:18:40.154 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T21:18:40.154 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2023-05-31T21:18:40.154 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1685562970.274572_93.1", 2023-05-31T21:18:40.155 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T21:18:40.155 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2023-05-31T21:18:40.155 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T21:18:40.155 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2023-05-31T19:56:10.274572Z", 2023-05-31T21:18:40.155 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2023-05-31T21:18:40.155 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T21:18:40.155 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:18:40.156 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T21:18:40.156 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 3, 2023-05-31T21:18:40.156 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T21:18:40.156 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2023-05-31T21:18:40.156 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1685567847.673276_896.1", 2023-05-31T21:18:40.156 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T21:18:40.156 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2023-05-31T21:18:40.157 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T21:18:40.157 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2023-05-31T21:17:27.673276Z", 2023-05-31T21:18:40.157 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2023-05-31T21:18:40.157 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T21:18:40.157 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T21:18:40.157 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2023-05-31T21:18:40.157 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:18:40.158 INFO:teuthology.orchestra.run.smithi099.stdout: "full_sync": { 2023-05-31T21:18:40.158 INFO:teuthology.orchestra.run.smithi099.stdout: "total": 1, 2023-05-31T21:18:40.158 INFO:teuthology.orchestra.run.smithi099.stdout: "complete": 1 2023-05-31T21:18:40.158 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:18:40.158 INFO:teuthology.orchestra.run.smithi099.stdout: "current_time": "2023-05-31T21:18:40Z" 2023-05-31T21:18:40.158 INFO:teuthology.orchestra.run.smithi099.stdout:} 2023-05-31T21:18:40.159 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "2e6123d2-d695-4350-9a01-61a455bc782b", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1685566501.757183_260.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-05-31T20:55:01.757183Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1685566501.735717_251.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-05-31T20:55:01.735717Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1685562970.274572_93.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-05-31T19:56:10.274572Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1685567847.673276_896.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-05-31T21:17:27.673276Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-05-31T21:18:40Z" } 2023-05-31T21:18:40.160 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1685566501.757183_260.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-05-31T20:55:01.757183Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1685566501.735717_251.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-05-31T20:55:01.735717Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1685562970.274572_93.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-05-31T19:56:10.274572Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1685567847.673276_896.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-05-31T21:17:27.673276Z', 'realm_epoch': 2}}] 2023-05-31T21:18:40.160 DEBUG:rgw_multi.tests:log_status={0: '1_1685566501.757183_260.1', 1: '1_1685566501.735717_251.1', 2: '1_1685562970.274572_93.1', 3: '1_1685567847.936050_900.1'} 2023-05-31T21:18:40.160 DEBUG:rgw_multi.tests:sync_status={0: '1_1685566501.757183_260.1', 1: '1_1685566501.735717_251.1', 2: '1_1685562970.274572_93.1', 3: '1_1685567847.673276_896.1'} 2023-05-31T21:18:40.161 WARNING:rgw_multi.tests:zone a2 behind master: shard=3 master=1_1685567847.936050_900.1 target=1_1685567847.673276_896.1 2023-05-31T21:18:45.162 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-31T21:18:45.162 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-31T21:18:45.163 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-05-31T21:18:45.219 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2023-05-31T21:18:45.220 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2023-05-31T21:18:45.264 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T21:18:45.259+0000 7fdf10f8e780 20 RGW-SYNC:meta: read sync status 2023-05-31T21:18:45.264 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T21:18:45.260+0000 7fdf10f8e780 20 RGW-SYNC:meta: read sync status 2023-05-31T21:18:45.282 INFO:teuthology.orchestra.run.smithi099.stdout:{ 2023-05-31T21:18:45.282 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_status": { 2023-05-31T21:18:45.283 INFO:teuthology.orchestra.run.smithi099.stdout: "info": { 2023-05-31T21:18:45.283 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "sync", 2023-05-31T21:18:45.283 INFO:teuthology.orchestra.run.smithi099.stdout: "num_shards": 4, 2023-05-31T21:18:45.283 INFO:teuthology.orchestra.run.smithi099.stdout: "period": "2e6123d2-d695-4350-9a01-61a455bc782b", 2023-05-31T21:18:45.283 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2023-05-31T21:18:45.283 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:18:45.283 INFO:teuthology.orchestra.run.smithi099.stdout: "markers": [ 2023-05-31T21:18:45.284 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T21:18:45.284 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 0, 2023-05-31T21:18:45.284 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T21:18:45.284 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2023-05-31T21:18:45.284 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1685566501.757183_260.1", 2023-05-31T21:18:45.284 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T21:18:45.285 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 1, 2023-05-31T21:18:45.285 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T21:18:45.285 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2023-05-31T20:55:01.757183Z", 2023-05-31T21:18:45.285 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2023-05-31T21:18:45.285 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T21:18:45.285 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:18:45.285 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T21:18:45.286 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 1, 2023-05-31T21:18:45.286 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T21:18:45.286 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2023-05-31T21:18:45.286 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1685566501.735717_251.1", 2023-05-31T21:18:45.286 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T21:18:45.286 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2023-05-31T21:18:45.286 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T21:18:45.287 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2023-05-31T20:55:01.735717Z", 2023-05-31T21:18:45.287 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2023-05-31T21:18:45.287 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T21:18:45.287 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:18:45.287 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T21:18:45.287 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 2, 2023-05-31T21:18:45.287 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T21:18:45.288 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2023-05-31T21:18:45.288 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1685562970.274572_93.1", 2023-05-31T21:18:45.288 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T21:18:45.288 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2023-05-31T21:18:45.288 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T21:18:45.288 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2023-05-31T19:56:10.274572Z", 2023-05-31T21:18:45.288 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2023-05-31T21:18:45.289 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T21:18:45.289 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:18:45.289 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T21:18:45.289 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 3, 2023-05-31T21:18:45.289 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T21:18:45.289 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2023-05-31T21:18:45.289 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1685567847.673276_896.1", 2023-05-31T21:18:45.290 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T21:18:45.290 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2023-05-31T21:18:45.290 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T21:18:45.290 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2023-05-31T21:17:27.673276Z", 2023-05-31T21:18:45.290 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2023-05-31T21:18:45.290 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T21:18:45.291 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T21:18:45.291 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2023-05-31T21:18:45.291 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:18:45.291 INFO:teuthology.orchestra.run.smithi099.stdout: "full_sync": { 2023-05-31T21:18:45.291 INFO:teuthology.orchestra.run.smithi099.stdout: "total": 1, 2023-05-31T21:18:45.291 INFO:teuthology.orchestra.run.smithi099.stdout: "complete": 1 2023-05-31T21:18:45.291 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:18:45.292 INFO:teuthology.orchestra.run.smithi099.stdout: "current_time": "2023-05-31T21:18:45Z" 2023-05-31T21:18:45.292 INFO:teuthology.orchestra.run.smithi099.stdout:} 2023-05-31T21:18:45.292 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "2e6123d2-d695-4350-9a01-61a455bc782b", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1685566501.757183_260.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-05-31T20:55:01.757183Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1685566501.735717_251.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-05-31T20:55:01.735717Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1685562970.274572_93.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-05-31T19:56:10.274572Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1685567847.673276_896.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-05-31T21:17:27.673276Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-05-31T21:18:45Z" } 2023-05-31T21:18:45.293 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1685566501.757183_260.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-05-31T20:55:01.757183Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1685566501.735717_251.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-05-31T20:55:01.735717Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1685562970.274572_93.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-05-31T19:56:10.274572Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1685567847.673276_896.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-05-31T21:17:27.673276Z', 'realm_epoch': 2}}] 2023-05-31T21:18:45.293 DEBUG:rgw_multi.tests:log_status={0: '1_1685566501.757183_260.1', 1: '1_1685566501.735717_251.1', 2: '1_1685562970.274572_93.1', 3: '1_1685567847.936050_900.1'} 2023-05-31T21:18:45.294 DEBUG:rgw_multi.tests:sync_status={0: '1_1685566501.757183_260.1', 1: '1_1685566501.735717_251.1', 2: '1_1685562970.274572_93.1', 3: '1_1685567847.673276_896.1'} 2023-05-31T21:18:45.294 WARNING:rgw_multi.tests:zone a2 behind master: shard=3 master=1_1685567847.936050_900.1 target=1_1685567847.673276_896.1 2023-05-31T21:18:50.295 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-31T21:18:50.295 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-31T21:18:50.296 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-05-31T21:18:50.353 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2023-05-31T21:18:50.354 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2023-05-31T21:18:50.418 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T21:18:50.414+0000 7f25c63ec780 20 RGW-SYNC:meta: read sync status 2023-05-31T21:18:50.419 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T21:18:50.415+0000 7f25c63ec780 20 RGW-SYNC:meta: read sync status 2023-05-31T21:18:50.442 INFO:teuthology.orchestra.run.smithi099.stdout:{ 2023-05-31T21:18:50.442 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_status": { 2023-05-31T21:18:50.443 INFO:teuthology.orchestra.run.smithi099.stdout: "info": { 2023-05-31T21:18:50.443 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "sync", 2023-05-31T21:18:50.443 INFO:teuthology.orchestra.run.smithi099.stdout: "num_shards": 4, 2023-05-31T21:18:50.443 INFO:teuthology.orchestra.run.smithi099.stdout: "period": "2e6123d2-d695-4350-9a01-61a455bc782b", 2023-05-31T21:18:50.443 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2023-05-31T21:18:50.443 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:18:50.444 INFO:teuthology.orchestra.run.smithi099.stdout: "markers": [ 2023-05-31T21:18:50.444 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T21:18:50.444 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 0, 2023-05-31T21:18:50.444 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T21:18:50.444 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2023-05-31T21:18:50.444 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1685566501.757183_260.1", 2023-05-31T21:18:50.444 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T21:18:50.445 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 1, 2023-05-31T21:18:50.445 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T21:18:50.445 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2023-05-31T20:55:01.757183Z", 2023-05-31T21:18:50.445 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2023-05-31T21:18:50.445 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T21:18:50.445 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:18:50.445 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T21:18:50.446 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 1, 2023-05-31T21:18:50.446 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T21:18:50.446 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2023-05-31T21:18:50.446 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1685566501.735717_251.1", 2023-05-31T21:18:50.446 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T21:18:50.446 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2023-05-31T21:18:50.446 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T21:18:50.447 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2023-05-31T20:55:01.735717Z", 2023-05-31T21:18:50.447 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2023-05-31T21:18:50.447 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T21:18:50.447 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:18:50.447 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T21:18:50.447 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 2, 2023-05-31T21:18:50.448 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T21:18:50.448 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2023-05-31T21:18:50.448 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1685562970.274572_93.1", 2023-05-31T21:18:50.448 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T21:18:50.448 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2023-05-31T21:18:50.448 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T21:18:50.449 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2023-05-31T19:56:10.274572Z", 2023-05-31T21:18:50.449 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2023-05-31T21:18:50.449 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T21:18:50.449 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:18:50.449 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T21:18:50.449 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 3, 2023-05-31T21:18:50.449 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T21:18:50.450 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2023-05-31T21:18:50.450 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1685567847.936050_900.1", 2023-05-31T21:18:50.450 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T21:18:50.450 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2023-05-31T21:18:50.450 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T21:18:50.450 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2023-05-31T21:17:27.936050Z", 2023-05-31T21:18:50.450 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2023-05-31T21:18:50.451 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T21:18:50.451 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T21:18:50.451 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2023-05-31T21:18:50.451 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:18:50.451 INFO:teuthology.orchestra.run.smithi099.stdout: "full_sync": { 2023-05-31T21:18:50.451 INFO:teuthology.orchestra.run.smithi099.stdout: "total": 1, 2023-05-31T21:18:50.451 INFO:teuthology.orchestra.run.smithi099.stdout: "complete": 1 2023-05-31T21:18:50.452 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:18:50.452 INFO:teuthology.orchestra.run.smithi099.stdout: "current_time": "2023-05-31T21:18:50Z" 2023-05-31T21:18:50.452 INFO:teuthology.orchestra.run.smithi099.stdout:} 2023-05-31T21:18:50.453 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "2e6123d2-d695-4350-9a01-61a455bc782b", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1685566501.757183_260.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-05-31T20:55:01.757183Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1685566501.735717_251.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-05-31T20:55:01.735717Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1685562970.274572_93.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-05-31T19:56:10.274572Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1685567847.936050_900.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-05-31T21:17:27.936050Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-05-31T21:18:50Z" } 2023-05-31T21:18:50.453 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1685566501.757183_260.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-05-31T20:55:01.757183Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1685566501.735717_251.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-05-31T20:55:01.735717Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1685562970.274572_93.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-05-31T19:56:10.274572Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1685567847.936050_900.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-05-31T21:17:27.936050Z', 'realm_epoch': 2}}] 2023-05-31T21:18:50.454 DEBUG:rgw_multi.tests:log_status={0: '1_1685566501.757183_260.1', 1: '1_1685566501.735717_251.1', 2: '1_1685562970.274572_93.1', 3: '1_1685567847.936050_900.1'} 2023-05-31T21:18:50.454 DEBUG:rgw_multi.tests:sync_status={0: '1_1685566501.757183_260.1', 1: '1_1685566501.735717_251.1', 2: '1_1685562970.274572_93.1', 3: '1_1685567847.936050_900.1'} 2023-05-31T21:18:50.454 INFO:rgw_multi.tests:finish meta checkpoint for zone=a2 2023-05-31T21:18:50.455 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'group', 'pipe', 'remove', '--group-id', 'sync-bucket', '--pipe-id', 'sync-pipe', '--bucket=pklxkd-65', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-05-31T21:18:50.455 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'group', 'pipe', 'remove', '--group-id', 'sync-bucket', '--pipe-id', 'sync-pipe', '--bucket=pklxkd-65', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-05-31T21:18:50.455 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync group pipe remove --group-id sync-bucket --pipe-id sync-pipe --bucket=pklxkd-65 --cluster c1 --debug-rgw 0 --debug-ms 0 2023-05-31T21:18:50.507 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2023-05-31T21:18:50.507 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2023-05-31T21:18:50.569 INFO:teuthology.orchestra.run.smithi099.stdout:{"groups":[{"id":"sync-bucket","data_flow":{"symmetrical":[{"id":"sync-flow1","zones":["a1","a2"]}],"directional":[{"source_zone":"a1","dest_zone":"a2"}]},"pipes":[],"status":"enabled"}]} 2023-05-31T21:18:50.569 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'group', 'flow', 'remove', '--group-id', 'sync-bucket', '--flow-id', 'sync-flow2', '--flow-type', 'directional', '--source-zone=a1', '--dest-zone=a2', '--bucket', 'pklxkd-65', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-05-31T21:18:50.570 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'group', 'flow', 'remove', '--group-id', 'sync-bucket', '--flow-id', 'sync-flow2', '--flow-type', 'directional', '--source-zone=a1', '--dest-zone=a2', '--bucket', 'pklxkd-65', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-05-31T21:18:50.570 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync group flow remove --group-id sync-bucket --flow-id sync-flow2 --flow-type directional --source-zone=a1 --dest-zone=a2 --bucket pklxkd-65 --cluster c1 --debug-rgw 0 --debug-ms 0 2023-05-31T21:18:50.627 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2023-05-31T21:18:50.627 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2023-05-31T21:18:50.690 INFO:teuthology.orchestra.run.smithi099.stdout:{"groups":[{"id":"sync-bucket","data_flow":{"symmetrical":[{"id":"sync-flow1","zones":["a1","a2"]}]},"pipes":[],"status":"enabled"}]} 2023-05-31T21:18:50.691 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'group', 'flow', 'remove', '--group-id', 'sync-bucket', '--flow-id', 'sync-flow1', '--flow-type', 'symmetrical', '--zones=a1,a2', '--bucket', 'pklxkd-65', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-05-31T21:18:50.692 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'group', 'flow', 'remove', '--group-id', 'sync-bucket', '--flow-id', 'sync-flow1', '--flow-type', 'symmetrical', '--zones=a1,a2', '--bucket', 'pklxkd-65', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-05-31T21:18:50.693 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync group flow remove --group-id sync-bucket --flow-id sync-flow1 --flow-type symmetrical --zones=a1,a2 --bucket pklxkd-65 --cluster c1 --debug-rgw 0 --debug-ms 0 2023-05-31T21:18:50.748 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2023-05-31T21:18:50.749 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2023-05-31T21:18:50.803 INFO:teuthology.orchestra.run.smithi099.stdout:{"groups":[{"id":"sync-bucket","data_flow":{},"pipes":[],"status":"enabled"}]} 2023-05-31T21:18:50.804 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'group', 'remove', '--group-id', 'sync-bucket', '--bucket', 'pklxkd-65', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-05-31T21:18:50.804 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'group', 'remove', '--group-id', 'sync-bucket', '--bucket', 'pklxkd-65', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-05-31T21:18:50.805 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync group remove --group-id sync-bucket --bucket pklxkd-65 --cluster c1 --debug-rgw 0 --debug-ms 0 2023-05-31T21:18:50.854 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2023-05-31T21:18:50.854 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2023-05-31T21:18:50.914 INFO:teuthology.orchestra.run.smithi099.stdout:{"sync_policy":{"groups":[]}} 2023-05-31T21:18:50.914 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'policy', 'get', '--bucket', 'pklxkd-65', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-05-31T21:18:50.915 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'policy', 'get', '--bucket', 'pklxkd-65', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-05-31T21:18:50.915 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync policy get --bucket pklxkd-65 --cluster c1 --debug-rgw 0 --debug-ms 0 2023-05-31T21:18:50.962 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2023-05-31T21:18:50.963 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2023-05-31T21:18:51.029 INFO:teuthology.orchestra.run.smithi099.stdout:{"groups":[]} 2023-05-31T21:18:51.029 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-31T21:18:51.030 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-31T21:18:51.030 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-05-31T21:18:51.079 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2023-05-31T21:18:51.080 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2023-05-31T21:18:51.124 INFO:teuthology.orchestra.run.smithi099.stderr:No --period given, using current period=2e6123d2-d695-4350-9a01-61a455bc782b 2023-05-31T21:18:51.139 INFO:teuthology.orchestra.run.smithi099.stdout:[ 2023-05-31T21:18:51.140 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T21:18:51.140 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1685566501.757183_260.1", 2023-05-31T21:18:51.140 INFO:teuthology.orchestra.run.smithi099.stdout: "last_update": "2023-05-31T20:55:01.757183Z" 2023-05-31T21:18:51.140 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:18:51.141 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T21:18:51.141 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1685566501.735717_251.1", 2023-05-31T21:18:51.141 INFO:teuthology.orchestra.run.smithi099.stdout: "last_update": "2023-05-31T20:55:01.735717Z" 2023-05-31T21:18:51.141 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:18:51.141 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T21:18:51.141 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1685562970.274572_93.1", 2023-05-31T21:18:51.141 INFO:teuthology.orchestra.run.smithi099.stdout: "last_update": "2023-05-31T19:56:10.274572Z" 2023-05-31T21:18:51.142 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:18:51.142 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T21:18:51.142 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1685567930.890099_908.1", 2023-05-31T21:18:51.142 INFO:teuthology.orchestra.run.smithi099.stdout: "last_update": "2023-05-31T21:18:50.890099Z" 2023-05-31T21:18:51.142 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T21:18:51.142 INFO:teuthology.orchestra.run.smithi099.stdout:] 2023-05-31T21:18:51.143 DEBUG:rgw_multi.tests:master meta markers={0: '1_1685566501.757183_260.1', 1: '1_1685566501.735717_251.1', 2: '1_1685562970.274572_93.1', 3: '1_1685567930.890099_908.1'} 2023-05-31T21:18:51.144 INFO:rgw_multi.tests:starting meta checkpoint for zone=a2 2023-05-31T21:18:51.144 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-31T21:18:51.145 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-31T21:18:51.145 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-05-31T21:18:51.194 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2023-05-31T21:18:51.195 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2023-05-31T21:18:51.233 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T21:18:51.228+0000 7fb37db29780 20 RGW-SYNC:meta: read sync status 2023-05-31T21:18:51.233 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T21:18:51.229+0000 7fb37db29780 20 RGW-SYNC:meta: read sync status 2023-05-31T21:18:51.250 INFO:teuthology.orchestra.run.smithi099.stdout:{ 2023-05-31T21:18:51.250 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_status": { 2023-05-31T21:18:51.250 INFO:teuthology.orchestra.run.smithi099.stdout: "info": { 2023-05-31T21:18:51.250 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "sync", 2023-05-31T21:18:51.250 INFO:teuthology.orchestra.run.smithi099.stdout: "num_shards": 4, 2023-05-31T21:18:51.251 INFO:teuthology.orchestra.run.smithi099.stdout: "period": "2e6123d2-d695-4350-9a01-61a455bc782b", 2023-05-31T21:18:51.251 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2023-05-31T21:18:51.251 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:18:51.251 INFO:teuthology.orchestra.run.smithi099.stdout: "markers": [ 2023-05-31T21:18:51.251 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T21:18:51.251 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 0, 2023-05-31T21:18:51.251 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T21:18:51.252 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2023-05-31T21:18:51.252 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1685566501.757183_260.1", 2023-05-31T21:18:51.252 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T21:18:51.252 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 1, 2023-05-31T21:18:51.252 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T21:18:51.252 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2023-05-31T20:55:01.757183Z", 2023-05-31T21:18:51.253 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2023-05-31T21:18:51.253 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T21:18:51.253 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:18:51.253 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T21:18:51.253 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 1, 2023-05-31T21:18:51.253 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T21:18:51.254 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2023-05-31T21:18:51.254 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1685566501.735717_251.1", 2023-05-31T21:18:51.254 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T21:18:51.254 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2023-05-31T21:18:51.254 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T21:18:51.254 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2023-05-31T20:55:01.735717Z", 2023-05-31T21:18:51.255 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2023-05-31T21:18:51.255 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T21:18:51.255 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:18:51.255 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T21:18:51.255 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 2, 2023-05-31T21:18:51.255 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T21:18:51.256 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2023-05-31T21:18:51.256 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1685562970.274572_93.1", 2023-05-31T21:18:51.256 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T21:18:51.256 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2023-05-31T21:18:51.256 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T21:18:51.256 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2023-05-31T19:56:10.274572Z", 2023-05-31T21:18:51.257 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2023-05-31T21:18:51.257 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T21:18:51.257 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:18:51.257 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T21:18:51.257 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 3, 2023-05-31T21:18:51.258 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T21:18:51.258 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2023-05-31T21:18:51.258 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1685567847.936050_900.1", 2023-05-31T21:18:51.259 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T21:18:51.259 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2023-05-31T21:18:51.259 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T21:18:51.260 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2023-05-31T21:17:27.936050Z", 2023-05-31T21:18:51.260 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2023-05-31T21:18:51.260 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T21:18:51.261 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T21:18:51.261 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2023-05-31T21:18:51.261 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:18:51.262 INFO:teuthology.orchestra.run.smithi099.stdout: "full_sync": { 2023-05-31T21:18:51.262 INFO:teuthology.orchestra.run.smithi099.stdout: "total": 1, 2023-05-31T21:18:51.262 INFO:teuthology.orchestra.run.smithi099.stdout: "complete": 1 2023-05-31T21:18:51.263 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:18:51.263 INFO:teuthology.orchestra.run.smithi099.stdout: "current_time": "2023-05-31T21:18:51Z" 2023-05-31T21:18:51.263 INFO:teuthology.orchestra.run.smithi099.stdout:} 2023-05-31T21:18:51.265 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "2e6123d2-d695-4350-9a01-61a455bc782b", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1685566501.757183_260.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-05-31T20:55:01.757183Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1685566501.735717_251.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-05-31T20:55:01.735717Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1685562970.274572_93.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-05-31T19:56:10.274572Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1685567847.936050_900.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-05-31T21:17:27.936050Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-05-31T21:18:51Z" } 2023-05-31T21:18:51.265 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1685566501.757183_260.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-05-31T20:55:01.757183Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1685566501.735717_251.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-05-31T20:55:01.735717Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1685562970.274572_93.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-05-31T19:56:10.274572Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1685567847.936050_900.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-05-31T21:17:27.936050Z', 'realm_epoch': 2}}] 2023-05-31T21:18:51.266 DEBUG:rgw_multi.tests:log_status={0: '1_1685566501.757183_260.1', 1: '1_1685566501.735717_251.1', 2: '1_1685562970.274572_93.1', 3: '1_1685567930.890099_908.1'} 2023-05-31T21:18:51.267 DEBUG:rgw_multi.tests:sync_status={0: '1_1685566501.757183_260.1', 1: '1_1685566501.735717_251.1', 2: '1_1685562970.274572_93.1', 3: '1_1685567847.936050_900.1'} 2023-05-31T21:18:51.267 WARNING:rgw_multi.tests:zone a2 behind master: shard=3 master=1_1685567930.890099_908.1 target=1_1685567847.936050_900.1 2023-05-31T21:18:56.268 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-31T21:18:56.269 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-31T21:18:56.269 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-05-31T21:18:56.327 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2023-05-31T21:18:56.327 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2023-05-31T21:18:56.370 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T21:18:56.365+0000 7f0b452f2780 20 RGW-SYNC:meta: read sync status 2023-05-31T21:18:56.371 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T21:18:56.366+0000 7f0b452f2780 20 RGW-SYNC:meta: read sync status 2023-05-31T21:18:56.389 INFO:teuthology.orchestra.run.smithi099.stdout:{ 2023-05-31T21:18:56.390 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_status": { 2023-05-31T21:18:56.390 INFO:teuthology.orchestra.run.smithi099.stdout: "info": { 2023-05-31T21:18:56.390 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "sync", 2023-05-31T21:18:56.391 INFO:teuthology.orchestra.run.smithi099.stdout: "num_shards": 4, 2023-05-31T21:18:56.391 INFO:teuthology.orchestra.run.smithi099.stdout: "period": "2e6123d2-d695-4350-9a01-61a455bc782b", 2023-05-31T21:18:56.391 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2023-05-31T21:18:56.392 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:18:56.392 INFO:teuthology.orchestra.run.smithi099.stdout: "markers": [ 2023-05-31T21:18:56.392 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T21:18:56.393 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 0, 2023-05-31T21:18:56.393 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T21:18:56.393 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2023-05-31T21:18:56.393 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1685566501.757183_260.1", 2023-05-31T21:18:56.393 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T21:18:56.394 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 1, 2023-05-31T21:18:56.394 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T21:18:56.394 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2023-05-31T20:55:01.757183Z", 2023-05-31T21:18:56.394 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2023-05-31T21:18:56.395 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T21:18:56.395 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:18:56.395 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T21:18:56.395 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 1, 2023-05-31T21:18:56.396 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T21:18:56.396 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2023-05-31T21:18:56.396 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1685566501.735717_251.1", 2023-05-31T21:18:56.396 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T21:18:56.396 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2023-05-31T21:18:56.396 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T21:18:56.397 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2023-05-31T20:55:01.735717Z", 2023-05-31T21:18:56.397 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2023-05-31T21:18:56.397 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T21:18:56.397 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:18:56.397 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T21:18:56.397 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 2, 2023-05-31T21:18:56.397 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T21:18:56.398 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2023-05-31T21:18:56.398 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1685562970.274572_93.1", 2023-05-31T21:18:56.398 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T21:18:56.398 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2023-05-31T21:18:56.398 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T21:18:56.398 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2023-05-31T19:56:10.274572Z", 2023-05-31T21:18:56.398 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2023-05-31T21:18:56.399 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T21:18:56.399 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:18:56.399 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T21:18:56.399 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 3, 2023-05-31T21:18:56.399 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T21:18:56.399 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2023-05-31T21:18:56.399 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1685567847.936050_900.1", 2023-05-31T21:18:56.400 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T21:18:56.400 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2023-05-31T21:18:56.400 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T21:18:56.400 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2023-05-31T21:17:27.936050Z", 2023-05-31T21:18:56.400 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2023-05-31T21:18:56.400 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T21:18:56.401 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T21:18:56.401 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2023-05-31T21:18:56.401 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:18:56.401 INFO:teuthology.orchestra.run.smithi099.stdout: "full_sync": { 2023-05-31T21:18:56.401 INFO:teuthology.orchestra.run.smithi099.stdout: "total": 1, 2023-05-31T21:18:56.401 INFO:teuthology.orchestra.run.smithi099.stdout: "complete": 1 2023-05-31T21:18:56.402 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:18:56.402 INFO:teuthology.orchestra.run.smithi099.stdout: "current_time": "2023-05-31T21:18:56Z" 2023-05-31T21:18:56.402 INFO:teuthology.orchestra.run.smithi099.stdout:} 2023-05-31T21:18:56.404 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "2e6123d2-d695-4350-9a01-61a455bc782b", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1685566501.757183_260.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-05-31T20:55:01.757183Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1685566501.735717_251.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-05-31T20:55:01.735717Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1685562970.274572_93.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-05-31T19:56:10.274572Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1685567847.936050_900.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-05-31T21:17:27.936050Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-05-31T21:18:56Z" } 2023-05-31T21:18:56.405 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1685566501.757183_260.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-05-31T20:55:01.757183Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1685566501.735717_251.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-05-31T20:55:01.735717Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1685562970.274572_93.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-05-31T19:56:10.274572Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1685567847.936050_900.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-05-31T21:17:27.936050Z', 'realm_epoch': 2}}] 2023-05-31T21:18:56.406 DEBUG:rgw_multi.tests:log_status={0: '1_1685566501.757183_260.1', 1: '1_1685566501.735717_251.1', 2: '1_1685562970.274572_93.1', 3: '1_1685567930.890099_908.1'} 2023-05-31T21:18:56.406 DEBUG:rgw_multi.tests:sync_status={0: '1_1685566501.757183_260.1', 1: '1_1685566501.735717_251.1', 2: '1_1685562970.274572_93.1', 3: '1_1685567847.936050_900.1'} 2023-05-31T21:18:56.407 WARNING:rgw_multi.tests:zone a2 behind master: shard=3 master=1_1685567930.890099_908.1 target=1_1685567847.936050_900.1 2023-05-31T21:19:01.408 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-31T21:19:01.409 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-31T21:19:01.409 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-05-31T21:19:01.466 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2023-05-31T21:19:01.466 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2023-05-31T21:19:01.514 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T21:19:01.510+0000 7f707d8b0780 20 RGW-SYNC:meta: read sync status 2023-05-31T21:19:01.515 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T21:19:01.510+0000 7f707d8b0780 20 RGW-SYNC:meta: read sync status 2023-05-31T21:19:01.537 INFO:teuthology.orchestra.run.smithi099.stdout:{ 2023-05-31T21:19:01.537 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_status": { 2023-05-31T21:19:01.537 INFO:teuthology.orchestra.run.smithi099.stdout: "info": { 2023-05-31T21:19:01.538 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "sync", 2023-05-31T21:19:01.538 INFO:teuthology.orchestra.run.smithi099.stdout: "num_shards": 4, 2023-05-31T21:19:01.538 INFO:teuthology.orchestra.run.smithi099.stdout: "period": "2e6123d2-d695-4350-9a01-61a455bc782b", 2023-05-31T21:19:01.538 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2023-05-31T21:19:01.538 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:19:01.538 INFO:teuthology.orchestra.run.smithi099.stdout: "markers": [ 2023-05-31T21:19:01.538 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T21:19:01.539 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 0, 2023-05-31T21:19:01.539 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T21:19:01.539 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2023-05-31T21:19:01.539 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1685566501.757183_260.1", 2023-05-31T21:19:01.539 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T21:19:01.539 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 1, 2023-05-31T21:19:01.540 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T21:19:01.540 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2023-05-31T20:55:01.757183Z", 2023-05-31T21:19:01.540 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2023-05-31T21:19:01.540 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T21:19:01.540 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:19:01.540 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T21:19:01.540 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 1, 2023-05-31T21:19:01.541 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T21:19:01.541 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2023-05-31T21:19:01.541 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1685566501.735717_251.1", 2023-05-31T21:19:01.541 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T21:19:01.541 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2023-05-31T21:19:01.541 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T21:19:01.541 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2023-05-31T20:55:01.735717Z", 2023-05-31T21:19:01.542 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2023-05-31T21:19:01.542 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T21:19:01.542 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:19:01.542 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T21:19:01.542 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 2, 2023-05-31T21:19:01.542 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T21:19:01.543 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2023-05-31T21:19:01.543 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1685562970.274572_93.1", 2023-05-31T21:19:01.543 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T21:19:01.543 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2023-05-31T21:19:01.543 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T21:19:01.543 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2023-05-31T19:56:10.274572Z", 2023-05-31T21:19:01.543 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2023-05-31T21:19:01.544 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T21:19:01.544 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:19:01.544 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T21:19:01.544 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 3, 2023-05-31T21:19:01.544 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T21:19:01.544 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2023-05-31T21:19:01.544 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1685567930.890099_908.1", 2023-05-31T21:19:01.545 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T21:19:01.545 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2023-05-31T21:19:01.545 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T21:19:01.545 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2023-05-31T21:18:50.890099Z", 2023-05-31T21:19:01.545 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2023-05-31T21:19:01.545 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T21:19:01.545 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T21:19:01.546 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2023-05-31T21:19:01.546 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:19:01.546 INFO:teuthology.orchestra.run.smithi099.stdout: "full_sync": { 2023-05-31T21:19:01.546 INFO:teuthology.orchestra.run.smithi099.stdout: "total": 1, 2023-05-31T21:19:01.546 INFO:teuthology.orchestra.run.smithi099.stdout: "complete": 1 2023-05-31T21:19:01.546 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:19:01.546 INFO:teuthology.orchestra.run.smithi099.stdout: "current_time": "2023-05-31T21:19:01Z" 2023-05-31T21:19:01.547 INFO:teuthology.orchestra.run.smithi099.stdout:} 2023-05-31T21:19:01.548 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "2e6123d2-d695-4350-9a01-61a455bc782b", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1685566501.757183_260.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-05-31T20:55:01.757183Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1685566501.735717_251.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-05-31T20:55:01.735717Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1685562970.274572_93.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-05-31T19:56:10.274572Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1685567930.890099_908.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-05-31T21:18:50.890099Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-05-31T21:19:01Z" } 2023-05-31T21:19:01.548 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1685566501.757183_260.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-05-31T20:55:01.757183Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1685566501.735717_251.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-05-31T20:55:01.735717Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1685562970.274572_93.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-05-31T19:56:10.274572Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1685567930.890099_908.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-05-31T21:18:50.890099Z', 'realm_epoch': 2}}] 2023-05-31T21:19:01.549 DEBUG:rgw_multi.tests:log_status={0: '1_1685566501.757183_260.1', 1: '1_1685566501.735717_251.1', 2: '1_1685562970.274572_93.1', 3: '1_1685567930.890099_908.1'} 2023-05-31T21:19:01.549 DEBUG:rgw_multi.tests:sync_status={0: '1_1685566501.757183_260.1', 1: '1_1685566501.735717_251.1', 2: '1_1685562970.274572_93.1', 3: '1_1685567930.890099_908.1'} 2023-05-31T21:19:01.549 INFO:rgw_multi.tests:finish meta checkpoint for zone=a2 2023-05-31T21:19:01.549 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'group', 'pipe', 'remove', '--group-id', 'sync-group', '--pipe-id', 'sync-pipe', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-05-31T21:19:01.550 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'group', 'pipe', 'remove', '--group-id', 'sync-group', '--pipe-id', 'sync-pipe', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-05-31T21:19:01.550 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync group pipe remove --group-id sync-group --pipe-id sync-pipe --cluster c1 --debug-rgw 0 --debug-ms 0 2023-05-31T21:19:01.601 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2023-05-31T21:19:01.601 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2023-05-31T21:19:01.677 INFO:teuthology.orchestra.run.smithi099.stdout:{"groups":[{"id":"sync-group","data_flow":{"symmetrical":[{"id":"sync-flow1","zones":["a1","a2"]}],"directional":[{"source_zone":"a1","dest_zone":"a2"}]},"pipes":[],"status":"enabled"}]} 2023-05-31T21:19:01.677 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'group', 'flow', 'remove', '--group-id', 'sync-group', '--flow-id', 'sync-flow2', '--flow-type', 'directional', '--source-zone=a1', '--dest-zone=a2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-05-31T21:19:01.678 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'group', 'flow', 'remove', '--group-id', 'sync-group', '--flow-id', 'sync-flow2', '--flow-type', 'directional', '--source-zone=a1', '--dest-zone=a2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-05-31T21:19:01.678 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync group flow remove --group-id sync-group --flow-id sync-flow2 --flow-type directional --source-zone=a1 --dest-zone=a2 --cluster c1 --debug-rgw 0 --debug-ms 0 2023-05-31T21:19:01.732 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2023-05-31T21:19:01.733 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2023-05-31T21:19:01.781 INFO:teuthology.orchestra.run.smithi099.stdout:{"groups":[{"id":"sync-group","data_flow":{"symmetrical":[{"id":"sync-flow1","zones":["a1","a2"]}]},"pipes":[],"status":"enabled"}]} 2023-05-31T21:19:01.781 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'group', 'flow', 'remove', '--group-id', 'sync-group', '--flow-id', 'sync-flow1', '--flow-type', 'symmetrical', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-05-31T21:19:01.782 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'group', 'flow', 'remove', '--group-id', 'sync-group', '--flow-id', 'sync-flow1', '--flow-type', 'symmetrical', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-05-31T21:19:01.782 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync group flow remove --group-id sync-group --flow-id sync-flow1 --flow-type symmetrical --cluster c1 --debug-rgw 0 --debug-ms 0 2023-05-31T21:19:01.835 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2023-05-31T21:19:01.835 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2023-05-31T21:19:01.900 INFO:teuthology.orchestra.run.smithi099.stdout:{"groups":[{"id":"sync-group","data_flow":{},"pipes":[],"status":"enabled"}]} 2023-05-31T21:19:01.900 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'group', 'remove', '--group-id', 'sync-group', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-05-31T21:19:01.901 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'group', 'remove', '--group-id', 'sync-group', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-05-31T21:19:01.901 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync group remove --group-id sync-group --cluster c1 --debug-rgw 0 --debug-ms 0 2023-05-31T21:19:01.948 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2023-05-31T21:19:01.949 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2023-05-31T21:19:02.023 INFO:teuthology.orchestra.run.smithi099.stdout:{"sync_policy":{"groups":[]}} 2023-05-31T21:19:02.024 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'policy', 'get', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-05-31T21:19:02.024 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'policy', 'get', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-05-31T21:19:02.025 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync policy get --cluster c1 --debug-rgw 0 --debug-ms 0 2023-05-31T21:19:02.070 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2023-05-31T21:19:02.070 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2023-05-31T21:19:02.119 INFO:teuthology.orchestra.run.smithi099.stdout:{"groups":[]} 2023-05-31T21:19:02.119 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['period', 'update', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-05-31T21:19:02.120 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'period', 'update', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-05-31T21:19:02.120 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 period update --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --commit --cluster c1 --debug-rgw 0 --debug-ms 0 2023-05-31T21:19:02.169 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2023-05-31T21:19:02.169 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2023-05-31T21:19:02.240 INFO:teuthology.orchestra.run.smithi099.stdout:{ 2023-05-31T21:19:02.240 INFO:teuthology.orchestra.run.smithi099.stdout: "id": "2e6123d2-d695-4350-9a01-61a455bc782b", 2023-05-31T21:19:02.240 INFO:teuthology.orchestra.run.smithi099.stdout: "epoch": 14, 2023-05-31T21:19:02.240 INFO:teuthology.orchestra.run.smithi099.stdout: "predecessor_uuid": "629b9ced-80af-41ed-8a6c-f2cfc334cbb4", 2023-05-31T21:19:02.241 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_status": [], 2023-05-31T21:19:02.241 INFO:teuthology.orchestra.run.smithi099.stdout: "period_map": { 2023-05-31T21:19:02.241 INFO:teuthology.orchestra.run.smithi099.stdout: "id": "2e6123d2-d695-4350-9a01-61a455bc782b", 2023-05-31T21:19:02.241 INFO:teuthology.orchestra.run.smithi099.stdout: "zonegroups": [ 2023-05-31T21:19:02.241 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T21:19:02.241 INFO:teuthology.orchestra.run.smithi099.stdout: "id": "a", 2023-05-31T21:19:02.241 INFO:teuthology.orchestra.run.smithi099.stdout: "name": "a", 2023-05-31T21:19:02.242 INFO:teuthology.orchestra.run.smithi099.stdout: "api_name": "", 2023-05-31T21:19:02.242 INFO:teuthology.orchestra.run.smithi099.stdout: "is_master": true, 2023-05-31T21:19:02.242 INFO:teuthology.orchestra.run.smithi099.stdout: "endpoints": [ 2023-05-31T21:19:02.242 INFO:teuthology.orchestra.run.smithi099.stdout: "http://smithi099.front.sepia.ceph.com:8000" 2023-05-31T21:19:02.242 INFO:teuthology.orchestra.run.smithi099.stdout: ], 2023-05-31T21:19:02.242 INFO:teuthology.orchestra.run.smithi099.stdout: "hostnames": [], 2023-05-31T21:19:02.242 INFO:teuthology.orchestra.run.smithi099.stdout: "hostnames_s3website": [], 2023-05-31T21:19:02.243 INFO:teuthology.orchestra.run.smithi099.stdout: "master_zone": "3d163b1e-09c7-47e2-ae87-add4aace3f38", 2023-05-31T21:19:02.243 INFO:teuthology.orchestra.run.smithi099.stdout: "zones": [ 2023-05-31T21:19:02.243 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T21:19:02.243 INFO:teuthology.orchestra.run.smithi099.stdout: "id": "3d163b1e-09c7-47e2-ae87-add4aace3f38", 2023-05-31T21:19:02.243 INFO:teuthology.orchestra.run.smithi099.stdout: "name": "a1", 2023-05-31T21:19:02.243 INFO:teuthology.orchestra.run.smithi099.stdout: "endpoints": [ 2023-05-31T21:19:02.243 INFO:teuthology.orchestra.run.smithi099.stdout: "http://smithi099.front.sepia.ceph.com:8000" 2023-05-31T21:19:02.244 INFO:teuthology.orchestra.run.smithi099.stdout: ], 2023-05-31T21:19:02.244 INFO:teuthology.orchestra.run.smithi099.stdout: "log_meta": false, 2023-05-31T21:19:02.244 INFO:teuthology.orchestra.run.smithi099.stdout: "log_data": true, 2023-05-31T21:19:02.244 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket_index_max_shards": 11, 2023-05-31T21:19:02.244 INFO:teuthology.orchestra.run.smithi099.stdout: "read_only": false, 2023-05-31T21:19:02.244 INFO:teuthology.orchestra.run.smithi099.stdout: "tier_type": "", 2023-05-31T21:19:02.245 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_from_all": true, 2023-05-31T21:19:02.245 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_from": [], 2023-05-31T21:19:02.245 INFO:teuthology.orchestra.run.smithi099.stdout: "redirect_zone": "", 2023-05-31T21:19:02.245 INFO:teuthology.orchestra.run.smithi099.stdout: "supported_features": [ 2023-05-31T21:19:02.245 INFO:teuthology.orchestra.run.smithi099.stdout: "resharding" 2023-05-31T21:19:02.245 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2023-05-31T21:19:02.245 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:19:02.246 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T21:19:02.246 INFO:teuthology.orchestra.run.smithi099.stdout: "id": "5d8acec7-7e57-4e56-8b32-e3d6b5036001", 2023-05-31T21:19:02.246 INFO:teuthology.orchestra.run.smithi099.stdout: "name": "a2", 2023-05-31T21:19:02.246 INFO:teuthology.orchestra.run.smithi099.stdout: "endpoints": [ 2023-05-31T21:19:02.247 INFO:teuthology.orchestra.run.smithi099.stdout: "http://smithi099.front.sepia.ceph.com:8001" 2023-05-31T21:19:02.247 INFO:teuthology.orchestra.run.smithi099.stdout: ], 2023-05-31T21:19:02.247 INFO:teuthology.orchestra.run.smithi099.stdout: "log_meta": false, 2023-05-31T21:19:02.247 INFO:teuthology.orchestra.run.smithi099.stdout: "log_data": true, 2023-05-31T21:19:02.247 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket_index_max_shards": 11, 2023-05-31T21:19:02.247 INFO:teuthology.orchestra.run.smithi099.stdout: "read_only": false, 2023-05-31T21:19:02.248 INFO:teuthology.orchestra.run.smithi099.stdout: "tier_type": "", 2023-05-31T21:19:02.248 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_from_all": true, 2023-05-31T21:19:02.248 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_from": [], 2023-05-31T21:19:02.248 INFO:teuthology.orchestra.run.smithi099.stdout: "redirect_zone": "", 2023-05-31T21:19:02.248 INFO:teuthology.orchestra.run.smithi099.stdout: "supported_features": [ 2023-05-31T21:19:02.248 INFO:teuthology.orchestra.run.smithi099.stdout: "resharding" 2023-05-31T21:19:02.249 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2023-05-31T21:19:02.249 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T21:19:02.249 INFO:teuthology.orchestra.run.smithi099.stdout: ], 2023-05-31T21:19:02.249 INFO:teuthology.orchestra.run.smithi099.stdout: "placement_targets": [ 2023-05-31T21:19:02.249 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T21:19:02.249 INFO:teuthology.orchestra.run.smithi099.stdout: "name": "default-placement", 2023-05-31T21:19:02.249 INFO:teuthology.orchestra.run.smithi099.stdout: "tags": [], 2023-05-31T21:19:02.250 INFO:teuthology.orchestra.run.smithi099.stdout: "storage_classes": [ 2023-05-31T21:19:02.250 INFO:teuthology.orchestra.run.smithi099.stdout: "STANDARD" 2023-05-31T21:19:02.250 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2023-05-31T21:19:02.250 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T21:19:02.250 INFO:teuthology.orchestra.run.smithi099.stdout: ], 2023-05-31T21:19:02.250 INFO:teuthology.orchestra.run.smithi099.stdout: "default_placement": "default-placement", 2023-05-31T21:19:02.250 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_id": "aaa33de6-515b-4cac-8f1b-3a1a30452117", 2023-05-31T21:19:02.251 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_policy": { 2023-05-31T21:19:02.251 INFO:teuthology.orchestra.run.smithi099.stdout: "groups": [] 2023-05-31T21:19:02.251 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:19:02.251 INFO:teuthology.orchestra.run.smithi099.stdout: "enabled_features": [ 2023-05-31T21:19:02.251 INFO:teuthology.orchestra.run.smithi099.stdout: "resharding" 2023-05-31T21:19:02.251 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2023-05-31T21:19:02.251 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:19:02.252 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T21:19:02.252 INFO:teuthology.orchestra.run.smithi099.stdout: "id": "b", 2023-05-31T21:19:02.252 INFO:teuthology.orchestra.run.smithi099.stdout: "name": "b", 2023-05-31T21:19:02.252 INFO:teuthology.orchestra.run.smithi099.stdout: "api_name": "", 2023-05-31T21:19:02.252 INFO:teuthology.orchestra.run.smithi099.stdout: "is_master": false, 2023-05-31T21:19:02.252 INFO:teuthology.orchestra.run.smithi099.stdout: "endpoints": [ 2023-05-31T21:19:02.253 INFO:teuthology.orchestra.run.smithi099.stdout: "http://smithi112.front.sepia.ceph.com:8000" 2023-05-31T21:19:02.253 INFO:teuthology.orchestra.run.smithi099.stdout: ], 2023-05-31T21:19:02.253 INFO:teuthology.orchestra.run.smithi099.stdout: "hostnames": [], 2023-05-31T21:19:02.253 INFO:teuthology.orchestra.run.smithi099.stdout: "hostnames_s3website": [], 2023-05-31T21:19:02.253 INFO:teuthology.orchestra.run.smithi099.stdout: "master_zone": "f85d7236-2f27-4988-af26-9c66ab280b1b", 2023-05-31T21:19:02.253 INFO:teuthology.orchestra.run.smithi099.stdout: "zones": [ 2023-05-31T21:19:02.253 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T21:19:02.254 INFO:teuthology.orchestra.run.smithi099.stdout: "id": "a6a3aab5-bc52-44a4-8543-5c5ea6056acd", 2023-05-31T21:19:02.254 INFO:teuthology.orchestra.run.smithi099.stdout: "name": "b2", 2023-05-31T21:19:02.254 INFO:teuthology.orchestra.run.smithi099.stdout: "endpoints": [ 2023-05-31T21:19:02.254 INFO:teuthology.orchestra.run.smithi099.stdout: "http://smithi112.front.sepia.ceph.com:8001" 2023-05-31T21:19:02.254 INFO:teuthology.orchestra.run.smithi099.stdout: ], 2023-05-31T21:19:02.254 INFO:teuthology.orchestra.run.smithi099.stdout: "log_meta": false, 2023-05-31T21:19:02.254 INFO:teuthology.orchestra.run.smithi099.stdout: "log_data": true, 2023-05-31T21:19:02.255 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket_index_max_shards": 11, 2023-05-31T21:19:02.255 INFO:teuthology.orchestra.run.smithi099.stdout: "read_only": false, 2023-05-31T21:19:02.255 INFO:teuthology.orchestra.run.smithi099.stdout: "tier_type": "", 2023-05-31T21:19:02.255 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_from_all": true, 2023-05-31T21:19:02.256 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_from": [], 2023-05-31T21:19:02.256 INFO:teuthology.orchestra.run.smithi099.stdout: "redirect_zone": "", 2023-05-31T21:19:02.256 INFO:teuthology.orchestra.run.smithi099.stdout: "supported_features": [ 2023-05-31T21:19:02.257 INFO:teuthology.orchestra.run.smithi099.stdout: "resharding" 2023-05-31T21:19:02.257 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2023-05-31T21:19:02.257 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:19:02.258 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T21:19:02.258 INFO:teuthology.orchestra.run.smithi099.stdout: "id": "f85d7236-2f27-4988-af26-9c66ab280b1b", 2023-05-31T21:19:02.259 INFO:teuthology.orchestra.run.smithi099.stdout: "name": "b1", 2023-05-31T21:19:02.259 INFO:teuthology.orchestra.run.smithi099.stdout: "endpoints": [ 2023-05-31T21:19:02.259 INFO:teuthology.orchestra.run.smithi099.stdout: "http://smithi112.front.sepia.ceph.com:8000" 2023-05-31T21:19:02.260 INFO:teuthology.orchestra.run.smithi099.stdout: ], 2023-05-31T21:19:02.260 INFO:teuthology.orchestra.run.smithi099.stdout: "log_meta": false, 2023-05-31T21:19:02.260 INFO:teuthology.orchestra.run.smithi099.stdout: "log_data": true, 2023-05-31T21:19:02.260 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket_index_max_shards": 11, 2023-05-31T21:19:02.261 INFO:teuthology.orchestra.run.smithi099.stdout: "read_only": false, 2023-05-31T21:19:02.261 INFO:teuthology.orchestra.run.smithi099.stdout: "tier_type": "", 2023-05-31T21:19:02.261 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_from_all": true, 2023-05-31T21:19:02.262 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_from": [], 2023-05-31T21:19:02.262 INFO:teuthology.orchestra.run.smithi099.stdout: "redirect_zone": "", 2023-05-31T21:19:02.262 INFO:teuthology.orchestra.run.smithi099.stdout: "supported_features": [ 2023-05-31T21:19:02.262 INFO:teuthology.orchestra.run.smithi099.stdout: "resharding" 2023-05-31T21:19:02.262 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2023-05-31T21:19:02.263 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T21:19:02.263 INFO:teuthology.orchestra.run.smithi099.stdout: ], 2023-05-31T21:19:02.263 INFO:teuthology.orchestra.run.smithi099.stdout: "placement_targets": [ 2023-05-31T21:19:02.263 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T21:19:02.263 INFO:teuthology.orchestra.run.smithi099.stdout: "name": "default-placement", 2023-05-31T21:19:02.263 INFO:teuthology.orchestra.run.smithi099.stdout: "tags": [], 2023-05-31T21:19:02.264 INFO:teuthology.orchestra.run.smithi099.stdout: "storage_classes": [ 2023-05-31T21:19:02.264 INFO:teuthology.orchestra.run.smithi099.stdout: "STANDARD" 2023-05-31T21:19:02.264 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2023-05-31T21:19:02.264 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T21:19:02.264 INFO:teuthology.orchestra.run.smithi099.stdout: ], 2023-05-31T21:19:02.264 INFO:teuthology.orchestra.run.smithi099.stdout: "default_placement": "default-placement", 2023-05-31T21:19:02.264 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_id": "aaa33de6-515b-4cac-8f1b-3a1a30452117", 2023-05-31T21:19:02.265 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_policy": { 2023-05-31T21:19:02.265 INFO:teuthology.orchestra.run.smithi099.stdout: "groups": [] 2023-05-31T21:19:02.265 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:19:02.265 INFO:teuthology.orchestra.run.smithi099.stdout: "enabled_features": [ 2023-05-31T21:19:02.265 INFO:teuthology.orchestra.run.smithi099.stdout: "resharding" 2023-05-31T21:19:02.265 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2023-05-31T21:19:02.266 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T21:19:02.266 INFO:teuthology.orchestra.run.smithi099.stdout: ], 2023-05-31T21:19:02.266 INFO:teuthology.orchestra.run.smithi099.stdout: "short_zone_ids": [ 2023-05-31T21:19:02.266 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T21:19:02.266 INFO:teuthology.orchestra.run.smithi099.stdout: "key": "3d163b1e-09c7-47e2-ae87-add4aace3f38", 2023-05-31T21:19:02.266 INFO:teuthology.orchestra.run.smithi099.stdout: "val": 3101833771 2023-05-31T21:19:02.266 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:19:02.267 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T21:19:02.267 INFO:teuthology.orchestra.run.smithi099.stdout: "key": "5d8acec7-7e57-4e56-8b32-e3d6b5036001", 2023-05-31T21:19:02.268 INFO:teuthology.orchestra.run.smithi099.stdout: "val": 4255530623 2023-05-31T21:19:02.268 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:19:02.268 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T21:19:02.269 INFO:teuthology.orchestra.run.smithi099.stdout: "key": "a6a3aab5-bc52-44a4-8543-5c5ea6056acd", 2023-05-31T21:19:02.269 INFO:teuthology.orchestra.run.smithi099.stdout: "val": 3708073710 2023-05-31T21:19:02.269 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:19:02.269 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T21:19:02.269 INFO:teuthology.orchestra.run.smithi099.stdout: "key": "f85d7236-2f27-4988-af26-9c66ab280b1b", 2023-05-31T21:19:02.269 INFO:teuthology.orchestra.run.smithi099.stdout: "val": 233395200 2023-05-31T21:19:02.269 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T21:19:02.270 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2023-05-31T21:19:02.270 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:19:02.270 INFO:teuthology.orchestra.run.smithi099.stdout: "master_zonegroup": "a", 2023-05-31T21:19:02.270 INFO:teuthology.orchestra.run.smithi099.stdout: "master_zone": "3d163b1e-09c7-47e2-ae87-add4aace3f38", 2023-05-31T21:19:02.270 INFO:teuthology.orchestra.run.smithi099.stdout: "period_config": { 2023-05-31T21:19:02.271 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket_quota": { 2023-05-31T21:19:02.271 INFO:teuthology.orchestra.run.smithi099.stdout: "enabled": false, 2023-05-31T21:19:02.271 INFO:teuthology.orchestra.run.smithi099.stdout: "check_on_raw": false, 2023-05-31T21:19:02.271 INFO:teuthology.orchestra.run.smithi099.stdout: "max_size": -1, 2023-05-31T21:19:02.272 INFO:teuthology.orchestra.run.smithi099.stdout: "max_size_kb": 0, 2023-05-31T21:19:02.272 INFO:teuthology.orchestra.run.smithi099.stdout: "max_objects": -1 2023-05-31T21:19:02.272 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:19:02.272 INFO:teuthology.orchestra.run.smithi099.stdout: "user_quota": { 2023-05-31T21:19:02.272 INFO:teuthology.orchestra.run.smithi099.stdout: "enabled": false, 2023-05-31T21:19:02.272 INFO:teuthology.orchestra.run.smithi099.stdout: "check_on_raw": false, 2023-05-31T21:19:02.273 INFO:teuthology.orchestra.run.smithi099.stdout: "max_size": -1, 2023-05-31T21:19:02.273 INFO:teuthology.orchestra.run.smithi099.stdout: "max_size_kb": 0, 2023-05-31T21:19:02.273 INFO:teuthology.orchestra.run.smithi099.stdout: "max_objects": -1 2023-05-31T21:19:02.273 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:19:02.273 INFO:teuthology.orchestra.run.smithi099.stdout: "user_ratelimit": { 2023-05-31T21:19:02.273 INFO:teuthology.orchestra.run.smithi099.stdout: "max_read_ops": 0, 2023-05-31T21:19:02.274 INFO:teuthology.orchestra.run.smithi099.stdout: "max_write_ops": 0, 2023-05-31T21:19:02.274 INFO:teuthology.orchestra.run.smithi099.stdout: "max_read_bytes": 0, 2023-05-31T21:19:02.274 INFO:teuthology.orchestra.run.smithi099.stdout: "max_write_bytes": 0, 2023-05-31T21:19:02.274 INFO:teuthology.orchestra.run.smithi099.stdout: "enabled": false 2023-05-31T21:19:02.274 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:19:02.274 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket_ratelimit": { 2023-05-31T21:19:02.274 INFO:teuthology.orchestra.run.smithi099.stdout: "max_read_ops": 0, 2023-05-31T21:19:02.275 INFO:teuthology.orchestra.run.smithi099.stdout: "max_write_ops": 0, 2023-05-31T21:19:02.275 INFO:teuthology.orchestra.run.smithi099.stdout: "max_read_bytes": 0, 2023-05-31T21:19:02.275 INFO:teuthology.orchestra.run.smithi099.stdout: "max_write_bytes": 0, 2023-05-31T21:19:02.275 INFO:teuthology.orchestra.run.smithi099.stdout: "enabled": false 2023-05-31T21:19:02.275 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:19:02.275 INFO:teuthology.orchestra.run.smithi099.stdout: "anonymous_ratelimit": { 2023-05-31T21:19:02.276 INFO:teuthology.orchestra.run.smithi099.stdout: "max_read_ops": 0, 2023-05-31T21:19:02.276 INFO:teuthology.orchestra.run.smithi099.stdout: "max_write_ops": 0, 2023-05-31T21:19:02.276 INFO:teuthology.orchestra.run.smithi099.stdout: "max_read_bytes": 0, 2023-05-31T21:19:02.276 INFO:teuthology.orchestra.run.smithi099.stdout: "max_write_bytes": 0, 2023-05-31T21:19:02.276 INFO:teuthology.orchestra.run.smithi099.stdout: "enabled": false 2023-05-31T21:19:02.276 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T21:19:02.276 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:19:02.277 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_id": "aaa33de6-515b-4cac-8f1b-3a1a30452117", 2023-05-31T21:19:02.277 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_name": "test-realm", 2023-05-31T21:19:02.277 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2023-05-31T21:19:02.277 INFO:teuthology.orchestra.run.smithi099.stdout:} 2023-05-31T21:19:02.279 INFO:tasks.rgw_multisite_tests:test_sync_policy_config_zonegroup: ... ok 2023-05-31T21:19:02.280 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-31T21:19:02.280 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-31T21:19:02.281 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-05-31T21:19:02.329 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2023-05-31T21:19:02.329 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2023-05-31T21:19:02.364 INFO:teuthology.orchestra.run.smithi099.stderr:No --period given, using current period=2e6123d2-d695-4350-9a01-61a455bc782b 2023-05-31T21:19:02.377 INFO:teuthology.orchestra.run.smithi099.stdout:[ 2023-05-31T21:19:02.378 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T21:19:02.378 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1685566501.757183_260.1", 2023-05-31T21:19:02.378 INFO:teuthology.orchestra.run.smithi099.stdout: "last_update": "2023-05-31T20:55:01.757183Z" 2023-05-31T21:19:02.378 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:19:02.378 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T21:19:02.378 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1685566501.735717_251.1", 2023-05-31T21:19:02.379 INFO:teuthology.orchestra.run.smithi099.stdout: "last_update": "2023-05-31T20:55:01.735717Z" 2023-05-31T21:19:02.379 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:19:02.379 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T21:19:02.379 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1685562970.274572_93.1", 2023-05-31T21:19:02.379 INFO:teuthology.orchestra.run.smithi099.stdout: "last_update": "2023-05-31T19:56:10.274572Z" 2023-05-31T21:19:02.379 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:19:02.380 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T21:19:02.380 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1685567930.890099_908.1", 2023-05-31T21:19:02.380 INFO:teuthology.orchestra.run.smithi099.stdout: "last_update": "2023-05-31T21:18:50.890099Z" 2023-05-31T21:19:02.380 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T21:19:02.380 INFO:teuthology.orchestra.run.smithi099.stdout:] 2023-05-31T21:19:02.381 DEBUG:rgw_multi.tests:master meta markers={0: '1_1685566501.757183_260.1', 1: '1_1685566501.735717_251.1', 2: '1_1685562970.274572_93.1', 3: '1_1685567930.890099_908.1'} 2023-05-31T21:19:02.382 INFO:rgw_multi.tests:starting meta checkpoint for zone=a2 2023-05-31T21:19:02.382 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-31T21:19:02.382 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-31T21:19:02.383 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-05-31T21:19:02.439 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2023-05-31T21:19:02.439 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2023-05-31T21:19:02.501 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T21:19:02.496+0000 7fe56bba0780 20 RGW-SYNC:meta: read sync status 2023-05-31T21:19:02.502 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T21:19:02.498+0000 7fe56bba0780 20 RGW-SYNC:meta: read sync status 2023-05-31T21:19:02.546 INFO:teuthology.orchestra.run.smithi099.stdout:{ 2023-05-31T21:19:02.546 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_status": { 2023-05-31T21:19:02.547 INFO:teuthology.orchestra.run.smithi099.stdout: "info": { 2023-05-31T21:19:02.547 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "sync", 2023-05-31T21:19:02.547 INFO:teuthology.orchestra.run.smithi099.stdout: "num_shards": 4, 2023-05-31T21:19:02.547 INFO:teuthology.orchestra.run.smithi099.stdout: "period": "2e6123d2-d695-4350-9a01-61a455bc782b", 2023-05-31T21:19:02.547 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2023-05-31T21:19:02.547 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:19:02.548 INFO:teuthology.orchestra.run.smithi099.stdout: "markers": [ 2023-05-31T21:19:02.548 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T21:19:02.548 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 0, 2023-05-31T21:19:02.548 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T21:19:02.548 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2023-05-31T21:19:02.548 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1685566501.757183_260.1", 2023-05-31T21:19:02.549 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T21:19:02.549 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 1, 2023-05-31T21:19:02.549 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T21:19:02.549 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2023-05-31T20:55:01.757183Z", 2023-05-31T21:19:02.549 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2023-05-31T21:19:02.549 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T21:19:02.549 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:19:02.550 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T21:19:02.550 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 1, 2023-05-31T21:19:02.550 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T21:19:02.550 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2023-05-31T21:19:02.550 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1685566501.735717_251.1", 2023-05-31T21:19:02.550 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T21:19:02.551 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2023-05-31T21:19:02.551 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T21:19:02.552 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2023-05-31T20:55:01.735717Z", 2023-05-31T21:19:02.552 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2023-05-31T21:19:02.552 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T21:19:02.553 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:19:02.553 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T21:19:02.553 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 2, 2023-05-31T21:19:02.553 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T21:19:02.554 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2023-05-31T21:19:02.554 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1685562970.274572_93.1", 2023-05-31T21:19:02.555 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T21:19:02.555 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2023-05-31T21:19:02.555 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T21:19:02.555 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2023-05-31T19:56:10.274572Z", 2023-05-31T21:19:02.556 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2023-05-31T21:19:02.556 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T21:19:02.556 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:19:02.556 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T21:19:02.557 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 3, 2023-05-31T21:19:02.557 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T21:19:02.557 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2023-05-31T21:19:02.558 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1685567930.890099_908.1", 2023-05-31T21:19:02.558 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T21:19:02.558 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2023-05-31T21:19:02.558 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T21:19:02.558 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2023-05-31T21:18:50.890099Z", 2023-05-31T21:19:02.558 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2023-05-31T21:19:02.559 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T21:19:02.559 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T21:19:02.559 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2023-05-31T21:19:02.559 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:19:02.559 INFO:teuthology.orchestra.run.smithi099.stdout: "full_sync": { 2023-05-31T21:19:02.559 INFO:teuthology.orchestra.run.smithi099.stdout: "total": 1, 2023-05-31T21:19:02.560 INFO:teuthology.orchestra.run.smithi099.stdout: "complete": 1 2023-05-31T21:19:02.560 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:19:02.560 INFO:teuthology.orchestra.run.smithi099.stdout: "current_time": "2023-05-31T21:19:02Z" 2023-05-31T21:19:02.560 INFO:teuthology.orchestra.run.smithi099.stdout:} 2023-05-31T21:19:02.561 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "2e6123d2-d695-4350-9a01-61a455bc782b", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1685566501.757183_260.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-05-31T20:55:01.757183Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1685566501.735717_251.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-05-31T20:55:01.735717Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1685562970.274572_93.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-05-31T19:56:10.274572Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1685567930.890099_908.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-05-31T21:18:50.890099Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-05-31T21:19:02Z" } 2023-05-31T21:19:02.562 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1685566501.757183_260.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-05-31T20:55:01.757183Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1685566501.735717_251.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-05-31T20:55:01.735717Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1685562970.274572_93.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-05-31T19:56:10.274572Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1685567930.890099_908.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-05-31T21:18:50.890099Z', 'realm_epoch': 2}}] 2023-05-31T21:19:02.562 DEBUG:rgw_multi.tests:log_status={0: '1_1685566501.757183_260.1', 1: '1_1685566501.735717_251.1', 2: '1_1685562970.274572_93.1', 3: '1_1685567930.890099_908.1'} 2023-05-31T21:19:02.562 DEBUG:rgw_multi.tests:sync_status={0: '1_1685566501.757183_260.1', 1: '1_1685566501.735717_251.1', 2: '1_1685562970.274572_93.1', 3: '1_1685567930.890099_908.1'} 2023-05-31T21:19:02.563 INFO:rgw_multi.tests:finish meta checkpoint for zone=a2 2023-05-31T21:19:02.563 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'policy', 'get', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-05-31T21:19:02.564 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'policy', 'get', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-05-31T21:19:02.564 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync policy get --cluster c1 --debug-rgw 0 --debug-ms 0 2023-05-31T21:19:02.594 INFO:tasks.rgw.c2.client.1.smithi112.stdout:2023-05-31T21:19:02.590+0000 7fecb0821700 -1 WARNING: The OSD cluster is overloaded and struggling to complete ops. You need more capacity to serve this level of demand. 2023-05-31T21:19:02.622 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2023-05-31T21:19:02.623 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2023-05-31T21:19:02.756 INFO:teuthology.orchestra.run.smithi099.stdout:{"groups":[]} 2023-05-31T21:19:02.757 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'group', 'create', '--group-id', 'sync-group', '--status', 'allowed', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-05-31T21:19:02.757 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'group', 'create', '--group-id', 'sync-group', '--status', 'allowed', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-05-31T21:19:02.758 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync group create --group-id sync-group --status allowed --cluster c1 --debug-rgw 0 --debug-ms 0 2023-05-31T21:19:02.825 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2023-05-31T21:19:02.826 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2023-05-31T21:19:02.954 INFO:teuthology.orchestra.run.smithi099.stdout:{"groups":[{"id":"sync-group","data_flow":{},"pipes":[],"status":"allowed"}]} 2023-05-31T21:19:02.955 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'group', 'flow', 'create', '--group-id', 'sync-group', '--flow-id', 'sync-flow1', '--flow-type', 'symmetrical', '--zones=a1,a2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-05-31T21:19:02.955 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'group', 'flow', 'create', '--group-id', 'sync-group', '--flow-id', 'sync-flow1', '--flow-type', 'symmetrical', '--zones=a1,a2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-05-31T21:19:02.956 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync group flow create --group-id sync-group --flow-id sync-flow1 --flow-type symmetrical --zones=a1,a2 --cluster c1 --debug-rgw 0 --debug-ms 0 2023-05-31T21:19:03.017 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2023-05-31T21:19:03.018 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2023-05-31T21:19:03.108 INFO:teuthology.orchestra.run.smithi099.stdout:{"groups":[{"id":"sync-group","data_flow":{"symmetrical":[{"id":"sync-flow1","zones":["a1","a2"]}]},"pipes":[],"status":"allowed"}]} 2023-05-31T21:19:03.109 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'group', 'pipe', 'create', '--group-id', 'sync-group', '--pipe-id', 'sync-pipe', '--source-zones=a1,a2', '--dest-zones=a1,a2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-05-31T21:19:03.109 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'group', 'pipe', 'create', '--group-id', 'sync-group', '--pipe-id', 'sync-pipe', '--source-zones=a1,a2', '--dest-zones=a1,a2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-05-31T21:19:03.109 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync group pipe create --group-id sync-group --pipe-id sync-pipe --source-zones=a1,a2 --dest-zones=a1,a2 --cluster c1 --debug-rgw 0 --debug-ms 0 2023-05-31T21:19:03.167 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2023-05-31T21:19:03.167 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2023-05-31T21:19:03.251 INFO:teuthology.orchestra.run.smithi099.stdout:{"groups":[{"id":"sync-group","data_flow":{"symmetrical":[{"id":"sync-flow1","zones":["a1","a2"]}]},"pipes":[{"id":"sync-pipe","source":{"bucket":"*","zones":["a1","a2"]},"dest":{"bucket":"*","zones":["a1","a2"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":""}}],"status":"allowed"}]} 2023-05-31T21:19:03.252 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'group', 'modify', '--group-id', 'sync-group', '--status', 'enabled', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-05-31T21:19:03.252 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'group', 'modify', '--group-id', 'sync-group', '--status', 'enabled', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-05-31T21:19:03.253 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync group modify --group-id sync-group --status enabled --cluster c1 --debug-rgw 0 --debug-ms 0 2023-05-31T21:19:03.311 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2023-05-31T21:19:03.312 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2023-05-31T21:19:03.412 INFO:teuthology.orchestra.run.smithi099.stdout:{"groups":[{"id":"sync-group","data_flow":{"symmetrical":[{"id":"sync-flow1","zones":["a1","a2"]}]},"pipes":[{"id":"sync-pipe","source":{"bucket":"*","zones":["a1","a2"]},"dest":{"bucket":"*","zones":["a1","a2"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":""}}],"status":"enabled"}]} 2023-05-31T21:19:03.413 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['period', 'update', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-05-31T21:19:03.413 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'period', 'update', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-05-31T21:19:03.414 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 period update --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --commit --cluster c1 --debug-rgw 0 --debug-ms 0 2023-05-31T21:19:03.464 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2023-05-31T21:19:03.465 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2023-05-31T21:19:03.545 INFO:teuthology.orchestra.run.smithi099.stdout:{ 2023-05-31T21:19:03.545 INFO:teuthology.orchestra.run.smithi099.stdout: "id": "2e6123d2-d695-4350-9a01-61a455bc782b", 2023-05-31T21:19:03.545 INFO:teuthology.orchestra.run.smithi099.stdout: "epoch": 15, 2023-05-31T21:19:03.546 INFO:teuthology.orchestra.run.smithi099.stdout: "predecessor_uuid": "629b9ced-80af-41ed-8a6c-f2cfc334cbb4", 2023-05-31T21:19:03.546 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_status": [], 2023-05-31T21:19:03.546 INFO:teuthology.orchestra.run.smithi099.stdout: "period_map": { 2023-05-31T21:19:03.546 INFO:teuthology.orchestra.run.smithi099.stdout: "id": "2e6123d2-d695-4350-9a01-61a455bc782b", 2023-05-31T21:19:03.546 INFO:teuthology.orchestra.run.smithi099.stdout: "zonegroups": [ 2023-05-31T21:19:03.546 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T21:19:03.547 INFO:teuthology.orchestra.run.smithi099.stdout: "id": "a", 2023-05-31T21:19:03.547 INFO:teuthology.orchestra.run.smithi099.stdout: "name": "a", 2023-05-31T21:19:03.547 INFO:teuthology.orchestra.run.smithi099.stdout: "api_name": "", 2023-05-31T21:19:03.547 INFO:teuthology.orchestra.run.smithi099.stdout: "is_master": true, 2023-05-31T21:19:03.547 INFO:teuthology.orchestra.run.smithi099.stdout: "endpoints": [ 2023-05-31T21:19:03.547 INFO:teuthology.orchestra.run.smithi099.stdout: "http://smithi099.front.sepia.ceph.com:8000" 2023-05-31T21:19:03.547 INFO:teuthology.orchestra.run.smithi099.stdout: ], 2023-05-31T21:19:03.548 INFO:teuthology.orchestra.run.smithi099.stdout: "hostnames": [], 2023-05-31T21:19:03.548 INFO:teuthology.orchestra.run.smithi099.stdout: "hostnames_s3website": [], 2023-05-31T21:19:03.548 INFO:teuthology.orchestra.run.smithi099.stdout: "master_zone": "3d163b1e-09c7-47e2-ae87-add4aace3f38", 2023-05-31T21:19:03.548 INFO:teuthology.orchestra.run.smithi099.stdout: "zones": [ 2023-05-31T21:19:03.548 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T21:19:03.548 INFO:teuthology.orchestra.run.smithi099.stdout: "id": "3d163b1e-09c7-47e2-ae87-add4aace3f38", 2023-05-31T21:19:03.549 INFO:teuthology.orchestra.run.smithi099.stdout: "name": "a1", 2023-05-31T21:19:03.549 INFO:teuthology.orchestra.run.smithi099.stdout: "endpoints": [ 2023-05-31T21:19:03.549 INFO:teuthology.orchestra.run.smithi099.stdout: "http://smithi099.front.sepia.ceph.com:8000" 2023-05-31T21:19:03.549 INFO:teuthology.orchestra.run.smithi099.stdout: ], 2023-05-31T21:19:03.549 INFO:teuthology.orchestra.run.smithi099.stdout: "log_meta": false, 2023-05-31T21:19:03.549 INFO:teuthology.orchestra.run.smithi099.stdout: "log_data": true, 2023-05-31T21:19:03.549 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket_index_max_shards": 11, 2023-05-31T21:19:03.550 INFO:teuthology.orchestra.run.smithi099.stdout: "read_only": false, 2023-05-31T21:19:03.550 INFO:teuthology.orchestra.run.smithi099.stdout: "tier_type": "", 2023-05-31T21:19:03.550 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_from_all": true, 2023-05-31T21:19:03.550 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_from": [], 2023-05-31T21:19:03.550 INFO:teuthology.orchestra.run.smithi099.stdout: "redirect_zone": "", 2023-05-31T21:19:03.550 INFO:teuthology.orchestra.run.smithi099.stdout: "supported_features": [ 2023-05-31T21:19:03.550 INFO:teuthology.orchestra.run.smithi099.stdout: "resharding" 2023-05-31T21:19:03.551 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2023-05-31T21:19:03.551 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:19:03.551 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T21:19:03.551 INFO:teuthology.orchestra.run.smithi099.stdout: "id": "5d8acec7-7e57-4e56-8b32-e3d6b5036001", 2023-05-31T21:19:03.551 INFO:teuthology.orchestra.run.smithi099.stdout: "name": "a2", 2023-05-31T21:19:03.551 INFO:teuthology.orchestra.run.smithi099.stdout: "endpoints": [ 2023-05-31T21:19:03.551 INFO:teuthology.orchestra.run.smithi099.stdout: "http://smithi099.front.sepia.ceph.com:8001" 2023-05-31T21:19:03.552 INFO:teuthology.orchestra.run.smithi099.stdout: ], 2023-05-31T21:19:03.552 INFO:teuthology.orchestra.run.smithi099.stdout: "log_meta": false, 2023-05-31T21:19:03.552 INFO:teuthology.orchestra.run.smithi099.stdout: "log_data": true, 2023-05-31T21:19:03.552 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket_index_max_shards": 11, 2023-05-31T21:19:03.552 INFO:teuthology.orchestra.run.smithi099.stdout: "read_only": false, 2023-05-31T21:19:03.552 INFO:teuthology.orchestra.run.smithi099.stdout: "tier_type": "", 2023-05-31T21:19:03.553 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_from_all": true, 2023-05-31T21:19:03.553 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_from": [], 2023-05-31T21:19:03.553 INFO:teuthology.orchestra.run.smithi099.stdout: "redirect_zone": "", 2023-05-31T21:19:03.553 INFO:teuthology.orchestra.run.smithi099.stdout: "supported_features": [ 2023-05-31T21:19:03.553 INFO:teuthology.orchestra.run.smithi099.stdout: "resharding" 2023-05-31T21:19:03.553 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2023-05-31T21:19:03.553 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T21:19:03.554 INFO:teuthology.orchestra.run.smithi099.stdout: ], 2023-05-31T21:19:03.554 INFO:teuthology.orchestra.run.smithi099.stdout: "placement_targets": [ 2023-05-31T21:19:03.554 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T21:19:03.554 INFO:teuthology.orchestra.run.smithi099.stdout: "name": "default-placement", 2023-05-31T21:19:03.554 INFO:teuthology.orchestra.run.smithi099.stdout: "tags": [], 2023-05-31T21:19:03.554 INFO:teuthology.orchestra.run.smithi099.stdout: "storage_classes": [ 2023-05-31T21:19:03.554 INFO:teuthology.orchestra.run.smithi099.stdout: "STANDARD" 2023-05-31T21:19:03.555 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2023-05-31T21:19:03.555 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T21:19:03.555 INFO:teuthology.orchestra.run.smithi099.stdout: ], 2023-05-31T21:19:03.555 INFO:teuthology.orchestra.run.smithi099.stdout: "default_placement": "default-placement", 2023-05-31T21:19:03.555 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_id": "aaa33de6-515b-4cac-8f1b-3a1a30452117", 2023-05-31T21:19:03.555 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_policy": { 2023-05-31T21:19:03.555 INFO:teuthology.orchestra.run.smithi099.stdout: "groups": [ 2023-05-31T21:19:03.556 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T21:19:03.556 INFO:teuthology.orchestra.run.smithi099.stdout: "id": "sync-group", 2023-05-31T21:19:03.556 INFO:teuthology.orchestra.run.smithi099.stdout: "data_flow": { 2023-05-31T21:19:03.556 INFO:teuthology.orchestra.run.smithi099.stdout: "symmetrical": [ 2023-05-31T21:19:03.556 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T21:19:03.556 INFO:teuthology.orchestra.run.smithi099.stdout: "id": "sync-flow1", 2023-05-31T21:19:03.556 INFO:teuthology.orchestra.run.smithi099.stdout: "zones": [ 2023-05-31T21:19:03.557 INFO:teuthology.orchestra.run.smithi099.stdout: "3d163b1e-09c7-47e2-ae87-add4aace3f38", 2023-05-31T21:19:03.557 INFO:teuthology.orchestra.run.smithi099.stdout: "5d8acec7-7e57-4e56-8b32-e3d6b5036001" 2023-05-31T21:19:03.557 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2023-05-31T21:19:03.557 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T21:19:03.557 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2023-05-31T21:19:03.557 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:19:03.557 INFO:teuthology.orchestra.run.smithi099.stdout: "pipes": [ 2023-05-31T21:19:03.558 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T21:19:03.558 INFO:teuthology.orchestra.run.smithi099.stdout: "id": "sync-pipe", 2023-05-31T21:19:03.558 INFO:teuthology.orchestra.run.smithi099.stdout: "source": { 2023-05-31T21:19:03.558 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket": "*", 2023-05-31T21:19:03.558 INFO:teuthology.orchestra.run.smithi099.stdout: "zones": [ 2023-05-31T21:19:03.558 INFO:teuthology.orchestra.run.smithi099.stdout: "3d163b1e-09c7-47e2-ae87-add4aace3f38", 2023-05-31T21:19:03.558 INFO:teuthology.orchestra.run.smithi099.stdout: "5d8acec7-7e57-4e56-8b32-e3d6b5036001" 2023-05-31T21:19:03.559 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2023-05-31T21:19:03.559 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:19:03.559 INFO:teuthology.orchestra.run.smithi099.stdout: "dest": { 2023-05-31T21:19:03.559 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket": "*", 2023-05-31T21:19:03.559 INFO:teuthology.orchestra.run.smithi099.stdout: "zones": [ 2023-05-31T21:19:03.559 INFO:teuthology.orchestra.run.smithi099.stdout: "3d163b1e-09c7-47e2-ae87-add4aace3f38", 2023-05-31T21:19:03.559 INFO:teuthology.orchestra.run.smithi099.stdout: "5d8acec7-7e57-4e56-8b32-e3d6b5036001" 2023-05-31T21:19:03.560 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2023-05-31T21:19:03.560 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:19:03.560 INFO:teuthology.orchestra.run.smithi099.stdout: "params": { 2023-05-31T21:19:03.560 INFO:teuthology.orchestra.run.smithi099.stdout: "source": { 2023-05-31T21:19:03.560 INFO:teuthology.orchestra.run.smithi099.stdout: "filter": { 2023-05-31T21:19:03.560 INFO:teuthology.orchestra.run.smithi099.stdout: "tags": [] 2023-05-31T21:19:03.561 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T21:19:03.561 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:19:03.561 INFO:teuthology.orchestra.run.smithi099.stdout: "dest": {}, 2023-05-31T21:19:03.561 INFO:teuthology.orchestra.run.smithi099.stdout: "priority": 0, 2023-05-31T21:19:03.561 INFO:teuthology.orchestra.run.smithi099.stdout: "mode": "system", 2023-05-31T21:19:03.561 INFO:teuthology.orchestra.run.smithi099.stdout: "user": "" 2023-05-31T21:19:03.561 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T21:19:03.562 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T21:19:03.562 INFO:teuthology.orchestra.run.smithi099.stdout: ], 2023-05-31T21:19:03.562 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "enabled" 2023-05-31T21:19:03.562 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T21:19:03.562 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2023-05-31T21:19:03.562 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:19:03.562 INFO:teuthology.orchestra.run.smithi099.stdout: "enabled_features": [ 2023-05-31T21:19:03.563 INFO:teuthology.orchestra.run.smithi099.stdout: "resharding" 2023-05-31T21:19:03.563 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2023-05-31T21:19:03.563 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:19:03.563 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T21:19:03.563 INFO:teuthology.orchestra.run.smithi099.stdout: "id": "b", 2023-05-31T21:19:03.563 INFO:teuthology.orchestra.run.smithi099.stdout: "name": "b", 2023-05-31T21:19:03.563 INFO:teuthology.orchestra.run.smithi099.stdout: "api_name": "", 2023-05-31T21:19:03.564 INFO:teuthology.orchestra.run.smithi099.stdout: "is_master": false, 2023-05-31T21:19:03.564 INFO:teuthology.orchestra.run.smithi099.stdout: "endpoints": [ 2023-05-31T21:19:03.564 INFO:teuthology.orchestra.run.smithi099.stdout: "http://smithi112.front.sepia.ceph.com:8000" 2023-05-31T21:19:03.564 INFO:teuthology.orchestra.run.smithi099.stdout: ], 2023-05-31T21:19:03.564 INFO:teuthology.orchestra.run.smithi099.stdout: "hostnames": [], 2023-05-31T21:19:03.564 INFO:teuthology.orchestra.run.smithi099.stdout: "hostnames_s3website": [], 2023-05-31T21:19:03.564 INFO:teuthology.orchestra.run.smithi099.stdout: "master_zone": "f85d7236-2f27-4988-af26-9c66ab280b1b", 2023-05-31T21:19:03.565 INFO:teuthology.orchestra.run.smithi099.stdout: "zones": [ 2023-05-31T21:19:03.565 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T21:19:03.565 INFO:teuthology.orchestra.run.smithi099.stdout: "id": "a6a3aab5-bc52-44a4-8543-5c5ea6056acd", 2023-05-31T21:19:03.565 INFO:teuthology.orchestra.run.smithi099.stdout: "name": "b2", 2023-05-31T21:19:03.565 INFO:teuthology.orchestra.run.smithi099.stdout: "endpoints": [ 2023-05-31T21:19:03.565 INFO:teuthology.orchestra.run.smithi099.stdout: "http://smithi112.front.sepia.ceph.com:8001" 2023-05-31T21:19:03.565 INFO:teuthology.orchestra.run.smithi099.stdout: ], 2023-05-31T21:19:03.566 INFO:teuthology.orchestra.run.smithi099.stdout: "log_meta": false, 2023-05-31T21:19:03.566 INFO:teuthology.orchestra.run.smithi099.stdout: "log_data": true, 2023-05-31T21:19:03.566 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket_index_max_shards": 11, 2023-05-31T21:19:03.566 INFO:teuthology.orchestra.run.smithi099.stdout: "read_only": false, 2023-05-31T21:19:03.566 INFO:teuthology.orchestra.run.smithi099.stdout: "tier_type": "", 2023-05-31T21:19:03.566 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_from_all": true, 2023-05-31T21:19:03.566 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_from": [], 2023-05-31T21:19:03.567 INFO:teuthology.orchestra.run.smithi099.stdout: "redirect_zone": "", 2023-05-31T21:19:03.567 INFO:teuthology.orchestra.run.smithi099.stdout: "supported_features": [ 2023-05-31T21:19:03.567 INFO:teuthology.orchestra.run.smithi099.stdout: "resharding" 2023-05-31T21:19:03.567 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2023-05-31T21:19:03.567 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:19:03.567 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T21:19:03.567 INFO:teuthology.orchestra.run.smithi099.stdout: "id": "f85d7236-2f27-4988-af26-9c66ab280b1b", 2023-05-31T21:19:03.568 INFO:teuthology.orchestra.run.smithi099.stdout: "name": "b1", 2023-05-31T21:19:03.568 INFO:teuthology.orchestra.run.smithi099.stdout: "endpoints": [ 2023-05-31T21:19:03.568 INFO:teuthology.orchestra.run.smithi099.stdout: "http://smithi112.front.sepia.ceph.com:8000" 2023-05-31T21:19:03.568 INFO:teuthology.orchestra.run.smithi099.stdout: ], 2023-05-31T21:19:03.568 INFO:teuthology.orchestra.run.smithi099.stdout: "log_meta": false, 2023-05-31T21:19:03.568 INFO:teuthology.orchestra.run.smithi099.stdout: "log_data": true, 2023-05-31T21:19:03.568 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket_index_max_shards": 11, 2023-05-31T21:19:03.569 INFO:teuthology.orchestra.run.smithi099.stdout: "read_only": false, 2023-05-31T21:19:03.569 INFO:teuthology.orchestra.run.smithi099.stdout: "tier_type": "", 2023-05-31T21:19:03.569 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_from_all": true, 2023-05-31T21:19:03.569 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_from": [], 2023-05-31T21:19:03.569 INFO:teuthology.orchestra.run.smithi099.stdout: "redirect_zone": "", 2023-05-31T21:19:03.569 INFO:teuthology.orchestra.run.smithi099.stdout: "supported_features": [ 2023-05-31T21:19:03.569 INFO:teuthology.orchestra.run.smithi099.stdout: "resharding" 2023-05-31T21:19:03.570 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2023-05-31T21:19:03.570 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T21:19:03.570 INFO:teuthology.orchestra.run.smithi099.stdout: ], 2023-05-31T21:19:03.570 INFO:teuthology.orchestra.run.smithi099.stdout: "placement_targets": [ 2023-05-31T21:19:03.570 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T21:19:03.570 INFO:teuthology.orchestra.run.smithi099.stdout: "name": "default-placement", 2023-05-31T21:19:03.570 INFO:teuthology.orchestra.run.smithi099.stdout: "tags": [], 2023-05-31T21:19:03.571 INFO:teuthology.orchestra.run.smithi099.stdout: "storage_classes": [ 2023-05-31T21:19:03.571 INFO:teuthology.orchestra.run.smithi099.stdout: "STANDARD" 2023-05-31T21:19:03.571 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2023-05-31T21:19:03.571 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T21:19:03.571 INFO:teuthology.orchestra.run.smithi099.stdout: ], 2023-05-31T21:19:03.571 INFO:teuthology.orchestra.run.smithi099.stdout: "default_placement": "default-placement", 2023-05-31T21:19:03.571 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_id": "aaa33de6-515b-4cac-8f1b-3a1a30452117", 2023-05-31T21:19:03.572 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_policy": { 2023-05-31T21:19:03.572 INFO:teuthology.orchestra.run.smithi099.stdout: "groups": [] 2023-05-31T21:19:03.572 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:19:03.572 INFO:teuthology.orchestra.run.smithi099.stdout: "enabled_features": [ 2023-05-31T21:19:03.572 INFO:teuthology.orchestra.run.smithi099.stdout: "resharding" 2023-05-31T21:19:03.572 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2023-05-31T21:19:03.573 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T21:19:03.573 INFO:teuthology.orchestra.run.smithi099.stdout: ], 2023-05-31T21:19:03.573 INFO:teuthology.orchestra.run.smithi099.stdout: "short_zone_ids": [ 2023-05-31T21:19:03.573 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T21:19:03.573 INFO:teuthology.orchestra.run.smithi099.stdout: "key": "3d163b1e-09c7-47e2-ae87-add4aace3f38", 2023-05-31T21:19:03.573 INFO:teuthology.orchestra.run.smithi099.stdout: "val": 3101833771 2023-05-31T21:19:03.573 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:19:03.574 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T21:19:03.574 INFO:teuthology.orchestra.run.smithi099.stdout: "key": "5d8acec7-7e57-4e56-8b32-e3d6b5036001", 2023-05-31T21:19:03.574 INFO:teuthology.orchestra.run.smithi099.stdout: "val": 4255530623 2023-05-31T21:19:03.574 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:19:03.574 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T21:19:03.574 INFO:teuthology.orchestra.run.smithi099.stdout: "key": "a6a3aab5-bc52-44a4-8543-5c5ea6056acd", 2023-05-31T21:19:03.574 INFO:teuthology.orchestra.run.smithi099.stdout: "val": 3708073710 2023-05-31T21:19:03.575 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:19:03.575 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T21:19:03.576 INFO:teuthology.orchestra.run.smithi099.stdout: "key": "f85d7236-2f27-4988-af26-9c66ab280b1b", 2023-05-31T21:19:03.576 INFO:teuthology.orchestra.run.smithi099.stdout: "val": 233395200 2023-05-31T21:19:03.576 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T21:19:03.576 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2023-05-31T21:19:03.576 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:19:03.577 INFO:teuthology.orchestra.run.smithi099.stdout: "master_zonegroup": "a", 2023-05-31T21:19:03.577 INFO:teuthology.orchestra.run.smithi099.stdout: "master_zone": "3d163b1e-09c7-47e2-ae87-add4aace3f38", 2023-05-31T21:19:03.577 INFO:teuthology.orchestra.run.smithi099.stdout: "period_config": { 2023-05-31T21:19:03.577 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket_quota": { 2023-05-31T21:19:03.577 INFO:teuthology.orchestra.run.smithi099.stdout: "enabled": false, 2023-05-31T21:19:03.577 INFO:teuthology.orchestra.run.smithi099.stdout: "check_on_raw": false, 2023-05-31T21:19:03.577 INFO:teuthology.orchestra.run.smithi099.stdout: "max_size": -1, 2023-05-31T21:19:03.578 INFO:teuthology.orchestra.run.smithi099.stdout: "max_size_kb": 0, 2023-05-31T21:19:03.578 INFO:teuthology.orchestra.run.smithi099.stdout: "max_objects": -1 2023-05-31T21:19:03.578 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:19:03.578 INFO:teuthology.orchestra.run.smithi099.stdout: "user_quota": { 2023-05-31T21:19:03.578 INFO:teuthology.orchestra.run.smithi099.stdout: "enabled": false, 2023-05-31T21:19:03.578 INFO:teuthology.orchestra.run.smithi099.stdout: "check_on_raw": false, 2023-05-31T21:19:03.578 INFO:teuthology.orchestra.run.smithi099.stdout: "max_size": -1, 2023-05-31T21:19:03.579 INFO:teuthology.orchestra.run.smithi099.stdout: "max_size_kb": 0, 2023-05-31T21:19:03.579 INFO:teuthology.orchestra.run.smithi099.stdout: "max_objects": -1 2023-05-31T21:19:03.579 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:19:03.579 INFO:teuthology.orchestra.run.smithi099.stdout: "user_ratelimit": { 2023-05-31T21:19:03.579 INFO:teuthology.orchestra.run.smithi099.stdout: "max_read_ops": 0, 2023-05-31T21:19:03.579 INFO:teuthology.orchestra.run.smithi099.stdout: "max_write_ops": 0, 2023-05-31T21:19:03.579 INFO:teuthology.orchestra.run.smithi099.stdout: "max_read_bytes": 0, 2023-05-31T21:19:03.580 INFO:teuthology.orchestra.run.smithi099.stdout: "max_write_bytes": 0, 2023-05-31T21:19:03.580 INFO:teuthology.orchestra.run.smithi099.stdout: "enabled": false 2023-05-31T21:19:03.580 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:19:03.580 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket_ratelimit": { 2023-05-31T21:19:03.580 INFO:teuthology.orchestra.run.smithi099.stdout: "max_read_ops": 0, 2023-05-31T21:19:03.580 INFO:teuthology.orchestra.run.smithi099.stdout: "max_write_ops": 0, 2023-05-31T21:19:03.580 INFO:teuthology.orchestra.run.smithi099.stdout: "max_read_bytes": 0, 2023-05-31T21:19:03.581 INFO:teuthology.orchestra.run.smithi099.stdout: "max_write_bytes": 0, 2023-05-31T21:19:03.581 INFO:teuthology.orchestra.run.smithi099.stdout: "enabled": false 2023-05-31T21:19:03.581 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:19:03.581 INFO:teuthology.orchestra.run.smithi099.stdout: "anonymous_ratelimit": { 2023-05-31T21:19:03.581 INFO:teuthology.orchestra.run.smithi099.stdout: "max_read_ops": 0, 2023-05-31T21:19:03.581 INFO:teuthology.orchestra.run.smithi099.stdout: "max_write_ops": 0, 2023-05-31T21:19:03.581 INFO:teuthology.orchestra.run.smithi099.stdout: "max_read_bytes": 0, 2023-05-31T21:19:03.582 INFO:teuthology.orchestra.run.smithi099.stdout: "max_write_bytes": 0, 2023-05-31T21:19:03.582 INFO:teuthology.orchestra.run.smithi099.stdout: "enabled": false 2023-05-31T21:19:03.582 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T21:19:03.582 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:19:03.582 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_id": "aaa33de6-515b-4cac-8f1b-3a1a30452117", 2023-05-31T21:19:03.582 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_name": "test-realm", 2023-05-31T21:19:03.582 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2023-05-31T21:19:03.583 INFO:teuthology.orchestra.run.smithi099.stdout:} 2023-05-31T21:19:03.583 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'policy', 'get', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-05-31T21:19:03.583 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'policy', 'get', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-05-31T21:19:03.584 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync policy get --cluster c1 --debug-rgw 0 --debug-ms 0 2023-05-31T21:19:03.629 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2023-05-31T21:19:03.630 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2023-05-31T21:19:03.684 INFO:teuthology.orchestra.run.smithi099.stdout:{"groups":[{"id":"sync-group","data_flow":{"symmetrical":[{"id":"sync-flow1","zones":["a1","a2"]}]},"pipes":[{"id":"sync-pipe","source":{"bucket":"*","zones":["a1","a2"]},"dest":{"bucket":"*","zones":["a1","a2"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":""}}],"status":"enabled"}]} 2023-05-31T21:19:03.685 INFO:rgw_multi.tests:create bucket zone=a1 name=pklxkd-66 2023-05-31T21:19:03.685 DEBUG:boto:path=/pklxkd-66/ 2023-05-31T21:19:03.686 DEBUG:boto:auth_path=/pklxkd-66/ 2023-05-31T21:19:03.686 DEBUG:boto:Method: PUT 2023-05-31T21:19:03.686 DEBUG:boto:Path: /pklxkd-66/ 2023-05-31T21:19:03.687 DEBUG:boto:Data: 2023-05-31T21:19:03.687 DEBUG:boto:Headers: {} 2023-05-31T21:19:03.687 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T21:19:03.688 DEBUG:boto:Port: 8000 2023-05-31T21:19:03.688 DEBUG:boto:Params: {} 2023-05-31T21:19:03.689 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2023-05-31T21:19:03.689 DEBUG:boto:Token: None 2023-05-31T21:19:03.690 DEBUG:boto:StringToSign: PUT Wed, 31 May 2023 21:19:03 GMT /pklxkd-66/ 2023-05-31T21:19:03.690 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:sZ36IwxDZ+qnMY1jX2USb/XyDC0= 2023-05-31T21:19:03.690 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 21:19:03 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:sZ36IwxDZ+qnMY1jX2USb/XyDC0=', 'Content-Length': '0'} 2023-05-31T21:19:03.855 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000c92cf16026854497-006477b9c7-22ab-a1'), ('Content-Length', '0'), ('Date', 'Wed, 31 May 2023 21:19:03 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:19:03.856 DEBUG:boto:path=/pklxkd-66/ 2023-05-31T21:19:03.856 DEBUG:boto:auth_path=/pklxkd-66/ 2023-05-31T21:19:03.857 DEBUG:boto:Method: HEAD 2023-05-31T21:19:03.857 DEBUG:boto:Path: /pklxkd-66/ 2023-05-31T21:19:03.857 DEBUG:boto:Data: 2023-05-31T21:19:03.858 DEBUG:boto:Headers: {} 2023-05-31T21:19:03.858 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T21:19:03.858 DEBUG:boto:Port: 8000 2023-05-31T21:19:03.859 DEBUG:boto:Params: {} 2023-05-31T21:19:03.859 DEBUG:boto:Token: None 2023-05-31T21:19:03.859 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 21:19:03 GMT /pklxkd-66/ 2023-05-31T21:19:03.860 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:ZR0JwQAYqStiVoQNO+s2iO1i6X0= 2023-05-31T21:19:03.860 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 21:19:03 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:ZR0JwQAYqStiVoQNO+s2iO1i6X0=', 'Content-Length': '0'} 2023-05-31T21:19:03.863 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000056d6d5a7896fd896-006477b9c7-22ab-a1'), ('Content-Length', '0'), ('Date', 'Wed, 31 May 2023 21:19:03 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:19:03.863 DEBUG:boto:path=/pklxkd-66/obj1 2023-05-31T21:19:03.864 DEBUG:boto:auth_path=/pklxkd-66/obj1 2023-05-31T21:19:03.864 DEBUG:boto:Method: PUT 2023-05-31T21:19:03.864 DEBUG:boto:Path: /pklxkd-66/obj1 2023-05-31T21:19:03.865 DEBUG:boto:Data: 2023-05-31T21:19:03.865 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2023-05-31T21:19:03.865 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T21:19:03.866 DEBUG:boto:Port: 8000 2023-05-31T21:19:03.866 DEBUG:boto:Params: {} 2023-05-31T21:19:03.866 DEBUG:boto:Token: None 2023-05-31T21:19:03.867 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Wed, 31 May 2023 21:19:03 GMT /pklxkd-66/obj1 2023-05-31T21:19:03.867 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:ZjtIhRDTNXVGaH3rYymVgCXHlMU= 2023-05-31T21:19:03.868 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Wed, 31 May 2023 21:19:03 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:ZjtIhRDTNXVGaH3rYymVgCXHlMU='} 2023-05-31T21:19:03.881 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000ca7d6715b4032ce7-006477b9c7-22ab-a1'), ('Date', 'Wed, 31 May 2023 21:19:03 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:19:03.882 INFO:rgw_multi.tests:create bucket zone=a2 name=pklxkd-67 2023-05-31T21:19:03.883 DEBUG:boto:path=/pklxkd-67/ 2023-05-31T21:19:03.884 DEBUG:boto:auth_path=/pklxkd-67/ 2023-05-31T21:19:03.885 DEBUG:boto:Method: PUT 2023-05-31T21:19:03.885 DEBUG:boto:Path: /pklxkd-67/ 2023-05-31T21:19:03.886 DEBUG:boto:Data: 2023-05-31T21:19:03.887 DEBUG:boto:Headers: {} 2023-05-31T21:19:03.887 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T21:19:03.888 DEBUG:boto:Port: 8001 2023-05-31T21:19:03.889 DEBUG:boto:Params: {} 2023-05-31T21:19:03.890 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8001} 2023-05-31T21:19:03.891 DEBUG:boto:Token: None 2023-05-31T21:19:03.892 DEBUG:boto:StringToSign: PUT Wed, 31 May 2023 21:19:03 GMT /pklxkd-67/ 2023-05-31T21:19:03.892 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:eNT2YZ3QBSmKOpRS7yLSPkTvT9w= 2023-05-31T21:19:03.893 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 21:19:03 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:eNT2YZ3QBSmKOpRS7yLSPkTvT9w=', 'Content-Length': '0'} 2023-05-31T21:19:03.985 INFO:tasks.rgw.c1.client.1.smithi099.stdout:2023-05-31T21:19:03.981+0000 7f6c3ba21700 -1 rgw async rados processor: ERROR: _send_request(): get_sync_policy_handler() returned -2 2023-05-31T21:19:04.044 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx000000d5ddd57e56652cc-006477b9c7-22af-a2'), ('Content-Length', '0'), ('Date', 'Wed, 31 May 2023 21:19:04 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:19:04.045 DEBUG:boto:path=/pklxkd-67/ 2023-05-31T21:19:04.045 DEBUG:boto:auth_path=/pklxkd-67/ 2023-05-31T21:19:04.046 DEBUG:boto:Method: HEAD 2023-05-31T21:19:04.046 DEBUG:boto:Path: /pklxkd-67/ 2023-05-31T21:19:04.046 DEBUG:boto:Data: 2023-05-31T21:19:04.047 DEBUG:boto:Headers: {} 2023-05-31T21:19:04.047 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T21:19:04.047 DEBUG:boto:Port: 8001 2023-05-31T21:19:04.048 DEBUG:boto:Params: {} 2023-05-31T21:19:04.048 DEBUG:boto:Token: None 2023-05-31T21:19:04.049 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 21:19:04 GMT /pklxkd-67/ 2023-05-31T21:19:04.049 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:vITwzV27Z1eV5/gK1WlZAOuaMx4= 2023-05-31T21:19:04.050 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 21:19:04 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:vITwzV27Z1eV5/gK1WlZAOuaMx4=', 'Content-Length': '0'} 2023-05-31T21:19:04.053 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000071f7335270b9ca7d-006477b9c8-22af-a2'), ('Content-Length', '0'), ('Date', 'Wed, 31 May 2023 21:19:04 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:19:04.053 DEBUG:boto:path=/pklxkd-67/obj2 2023-05-31T21:19:04.054 DEBUG:boto:auth_path=/pklxkd-67/obj2 2023-05-31T21:19:04.054 DEBUG:boto:Method: PUT 2023-05-31T21:19:04.054 DEBUG:boto:Path: /pklxkd-67/obj2 2023-05-31T21:19:04.055 DEBUG:boto:Data: 2023-05-31T21:19:04.055 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2023-05-31T21:19:04.055 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T21:19:04.056 DEBUG:boto:Port: 8001 2023-05-31T21:19:04.056 DEBUG:boto:Params: {} 2023-05-31T21:19:04.057 DEBUG:boto:Token: None 2023-05-31T21:19:04.057 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Wed, 31 May 2023 21:19:04 GMT /pklxkd-67/obj2 2023-05-31T21:19:04.058 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:mONfomNcbjgtwH6yTobizPLp/0M= 2023-05-31T21:19:04.058 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Wed, 31 May 2023 21:19:04 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:mONfomNcbjgtwH6yTobizPLp/0M='} 2023-05-31T21:19:04.088 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000446be8c8ff5171e1-006477b9c8-22af-a2'), ('Date', 'Wed, 31 May 2023 21:19:04 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:19:04.089 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-31T21:19:04.089 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-31T21:19:04.090 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-05-31T21:19:04.151 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2023-05-31T21:19:04.151 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2023-05-31T21:19:04.219 INFO:teuthology.orchestra.run.smithi099.stderr:No --period given, using current period=2e6123d2-d695-4350-9a01-61a455bc782b 2023-05-31T21:19:04.271 INFO:teuthology.orchestra.run.smithi099.stdout:[ 2023-05-31T21:19:04.271 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T21:19:04.271 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1685566501.757183_260.1", 2023-05-31T21:19:04.271 INFO:teuthology.orchestra.run.smithi099.stdout: "last_update": "2023-05-31T20:55:01.757183Z" 2023-05-31T21:19:04.271 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:19:04.272 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T21:19:04.272 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1685566501.735717_251.1", 2023-05-31T21:19:04.272 INFO:teuthology.orchestra.run.smithi099.stdout: "last_update": "2023-05-31T20:55:01.735717Z" 2023-05-31T21:19:04.272 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:19:04.272 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T21:19:04.272 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1685562970.274572_93.1", 2023-05-31T21:19:04.273 INFO:teuthology.orchestra.run.smithi099.stdout: "last_update": "2023-05-31T19:56:10.274572Z" 2023-05-31T21:19:04.273 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:19:04.273 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T21:19:04.273 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1685567943.999406_931.1", 2023-05-31T21:19:04.273 INFO:teuthology.orchestra.run.smithi099.stdout: "last_update": "2023-05-31T21:19:03.999406Z" 2023-05-31T21:19:04.273 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T21:19:04.273 INFO:teuthology.orchestra.run.smithi099.stdout:] 2023-05-31T21:19:04.274 DEBUG:rgw_multi.tests:master meta markers={0: '1_1685566501.757183_260.1', 1: '1_1685566501.735717_251.1', 2: '1_1685562970.274572_93.1', 3: '1_1685567943.999406_931.1'} 2023-05-31T21:19:04.275 INFO:rgw_multi.tests:starting meta checkpoint for zone=a2 2023-05-31T21:19:04.275 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-31T21:19:04.276 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-31T21:19:04.276 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-05-31T21:19:04.332 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2023-05-31T21:19:04.333 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2023-05-31T21:19:04.393 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T21:19:04.389+0000 7fa20ed72780 20 RGW-SYNC:meta: read sync status 2023-05-31T21:19:04.394 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T21:19:04.390+0000 7fa20ed72780 20 RGW-SYNC:meta: read sync status 2023-05-31T21:19:04.422 INFO:teuthology.orchestra.run.smithi099.stdout:{ 2023-05-31T21:19:04.423 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_status": { 2023-05-31T21:19:04.423 INFO:teuthology.orchestra.run.smithi099.stdout: "info": { 2023-05-31T21:19:04.423 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "sync", 2023-05-31T21:19:04.423 INFO:teuthology.orchestra.run.smithi099.stdout: "num_shards": 4, 2023-05-31T21:19:04.423 INFO:teuthology.orchestra.run.smithi099.stdout: "period": "2e6123d2-d695-4350-9a01-61a455bc782b", 2023-05-31T21:19:04.424 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2023-05-31T21:19:04.424 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:19:04.424 INFO:teuthology.orchestra.run.smithi099.stdout: "markers": [ 2023-05-31T21:19:04.424 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T21:19:04.424 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 0, 2023-05-31T21:19:04.424 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T21:19:04.425 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2023-05-31T21:19:04.425 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1685566501.757183_260.1", 2023-05-31T21:19:04.425 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T21:19:04.425 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 1, 2023-05-31T21:19:04.425 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T21:19:04.425 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2023-05-31T20:55:01.757183Z", 2023-05-31T21:19:04.425 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2023-05-31T21:19:04.426 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T21:19:04.426 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:19:04.426 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T21:19:04.426 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 1, 2023-05-31T21:19:04.426 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T21:19:04.426 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2023-05-31T21:19:04.426 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1685566501.735717_251.1", 2023-05-31T21:19:04.427 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T21:19:04.427 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2023-05-31T21:19:04.427 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T21:19:04.427 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2023-05-31T20:55:01.735717Z", 2023-05-31T21:19:04.427 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2023-05-31T21:19:04.427 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T21:19:04.428 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:19:04.428 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T21:19:04.428 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 2, 2023-05-31T21:19:04.428 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T21:19:04.428 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2023-05-31T21:19:04.428 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1685562970.274572_93.1", 2023-05-31T21:19:04.428 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T21:19:04.429 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2023-05-31T21:19:04.429 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T21:19:04.429 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2023-05-31T19:56:10.274572Z", 2023-05-31T21:19:04.429 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2023-05-31T21:19:04.429 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T21:19:04.429 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:19:04.429 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T21:19:04.430 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 3, 2023-05-31T21:19:04.430 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T21:19:04.430 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2023-05-31T21:19:04.430 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1685567930.890099_908.1", 2023-05-31T21:19:04.430 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T21:19:04.430 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2023-05-31T21:19:04.430 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T21:19:04.431 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2023-05-31T21:18:50.890099Z", 2023-05-31T21:19:04.431 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2023-05-31T21:19:04.431 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T21:19:04.431 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T21:19:04.431 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2023-05-31T21:19:04.431 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:19:04.431 INFO:teuthology.orchestra.run.smithi099.stdout: "full_sync": { 2023-05-31T21:19:04.432 INFO:teuthology.orchestra.run.smithi099.stdout: "total": 1, 2023-05-31T21:19:04.432 INFO:teuthology.orchestra.run.smithi099.stdout: "complete": 1 2023-05-31T21:19:04.432 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:19:04.432 INFO:teuthology.orchestra.run.smithi099.stdout: "current_time": "2023-05-31T21:19:04Z" 2023-05-31T21:19:04.432 INFO:teuthology.orchestra.run.smithi099.stdout:} 2023-05-31T21:19:04.433 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "2e6123d2-d695-4350-9a01-61a455bc782b", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1685566501.757183_260.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-05-31T20:55:01.757183Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1685566501.735717_251.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-05-31T20:55:01.735717Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1685562970.274572_93.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-05-31T19:56:10.274572Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1685567930.890099_908.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-05-31T21:18:50.890099Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-05-31T21:19:04Z" } 2023-05-31T21:19:04.434 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1685566501.757183_260.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-05-31T20:55:01.757183Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1685566501.735717_251.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-05-31T20:55:01.735717Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1685562970.274572_93.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-05-31T19:56:10.274572Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1685567930.890099_908.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-05-31T21:18:50.890099Z', 'realm_epoch': 2}}] 2023-05-31T21:19:04.434 DEBUG:rgw_multi.tests:log_status={0: '1_1685566501.757183_260.1', 1: '1_1685566501.735717_251.1', 2: '1_1685562970.274572_93.1', 3: '1_1685567943.999406_931.1'} 2023-05-31T21:19:04.434 DEBUG:rgw_multi.tests:sync_status={0: '1_1685566501.757183_260.1', 1: '1_1685566501.735717_251.1', 2: '1_1685562970.274572_93.1', 3: '1_1685567930.890099_908.1'} 2023-05-31T21:19:04.435 WARNING:rgw_multi.tests:zone a2 behind master: shard=3 master=1_1685567943.999406_931.1 target=1_1685567930.890099_908.1 2023-05-31T21:19:09.436 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-31T21:19:09.437 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-31T21:19:09.438 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-05-31T21:19:09.495 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2023-05-31T21:19:09.496 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2023-05-31T21:19:09.529 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T21:19:09.526+0000 7fdcc7b34780 20 RGW-SYNC:meta: read sync status 2023-05-31T21:19:09.531 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T21:19:09.528+0000 7fdcc7b34780 20 RGW-SYNC:meta: read sync status 2023-05-31T21:19:09.547 INFO:teuthology.orchestra.run.smithi099.stdout:{ 2023-05-31T21:19:09.548 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_status": { 2023-05-31T21:19:09.548 INFO:teuthology.orchestra.run.smithi099.stdout: "info": { 2023-05-31T21:19:09.548 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "sync", 2023-05-31T21:19:09.548 INFO:teuthology.orchestra.run.smithi099.stdout: "num_shards": 4, 2023-05-31T21:19:09.549 INFO:teuthology.orchestra.run.smithi099.stdout: "period": "2e6123d2-d695-4350-9a01-61a455bc782b", 2023-05-31T21:19:09.549 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2023-05-31T21:19:09.549 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:19:09.549 INFO:teuthology.orchestra.run.smithi099.stdout: "markers": [ 2023-05-31T21:19:09.549 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T21:19:09.549 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 0, 2023-05-31T21:19:09.550 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T21:19:09.550 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2023-05-31T21:19:09.550 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1685566501.757183_260.1", 2023-05-31T21:19:09.550 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T21:19:09.550 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 1, 2023-05-31T21:19:09.550 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T21:19:09.550 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2023-05-31T20:55:01.757183Z", 2023-05-31T21:19:09.551 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2023-05-31T21:19:09.551 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T21:19:09.551 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:19:09.551 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T21:19:09.551 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 1, 2023-05-31T21:19:09.551 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T21:19:09.551 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2023-05-31T21:19:09.552 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1685566501.735717_251.1", 2023-05-31T21:19:09.552 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T21:19:09.552 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2023-05-31T21:19:09.552 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T21:19:09.552 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2023-05-31T20:55:01.735717Z", 2023-05-31T21:19:09.553 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2023-05-31T21:19:09.553 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T21:19:09.553 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:19:09.553 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T21:19:09.553 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 2, 2023-05-31T21:19:09.553 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T21:19:09.553 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2023-05-31T21:19:09.554 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1685562970.274572_93.1", 2023-05-31T21:19:09.554 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T21:19:09.554 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2023-05-31T21:19:09.554 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T21:19:09.554 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2023-05-31T19:56:10.274572Z", 2023-05-31T21:19:09.555 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2023-05-31T21:19:09.555 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T21:19:09.555 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:19:09.555 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T21:19:09.555 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 3, 2023-05-31T21:19:09.555 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T21:19:09.556 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2023-05-31T21:19:09.556 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1685567930.890099_908.1", 2023-05-31T21:19:09.556 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T21:19:09.556 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2023-05-31T21:19:09.556 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T21:19:09.557 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2023-05-31T21:18:50.890099Z", 2023-05-31T21:19:09.557 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2023-05-31T21:19:09.557 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T21:19:09.557 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T21:19:09.557 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2023-05-31T21:19:09.557 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:19:09.557 INFO:teuthology.orchestra.run.smithi099.stdout: "full_sync": { 2023-05-31T21:19:09.558 INFO:teuthology.orchestra.run.smithi099.stdout: "total": 1, 2023-05-31T21:19:09.558 INFO:teuthology.orchestra.run.smithi099.stdout: "complete": 1 2023-05-31T21:19:09.558 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:19:09.558 INFO:teuthology.orchestra.run.smithi099.stdout: "current_time": "2023-05-31T21:19:09Z" 2023-05-31T21:19:09.558 INFO:teuthology.orchestra.run.smithi099.stdout:} 2023-05-31T21:19:09.559 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "2e6123d2-d695-4350-9a01-61a455bc782b", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1685566501.757183_260.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-05-31T20:55:01.757183Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1685566501.735717_251.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-05-31T20:55:01.735717Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1685562970.274572_93.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-05-31T19:56:10.274572Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1685567930.890099_908.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-05-31T21:18:50.890099Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-05-31T21:19:09Z" } 2023-05-31T21:19:09.560 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1685566501.757183_260.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-05-31T20:55:01.757183Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1685566501.735717_251.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-05-31T20:55:01.735717Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1685562970.274572_93.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-05-31T19:56:10.274572Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1685567930.890099_908.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-05-31T21:18:50.890099Z', 'realm_epoch': 2}}] 2023-05-31T21:19:09.560 DEBUG:rgw_multi.tests:log_status={0: '1_1685566501.757183_260.1', 1: '1_1685566501.735717_251.1', 2: '1_1685562970.274572_93.1', 3: '1_1685567943.999406_931.1'} 2023-05-31T21:19:09.561 DEBUG:rgw_multi.tests:sync_status={0: '1_1685566501.757183_260.1', 1: '1_1685566501.735717_251.1', 2: '1_1685562970.274572_93.1', 3: '1_1685567930.890099_908.1'} 2023-05-31T21:19:09.561 WARNING:rgw_multi.tests:zone a2 behind master: shard=3 master=1_1685567943.999406_931.1 target=1_1685567930.890099_908.1 2023-05-31T21:19:14.562 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-31T21:19:14.563 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-31T21:19:14.564 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-05-31T21:19:14.622 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2023-05-31T21:19:14.623 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2023-05-31T21:19:14.661 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T21:19:14.658+0000 7f2556724780 20 RGW-SYNC:meta: read sync status 2023-05-31T21:19:14.662 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T21:19:14.659+0000 7f2556724780 20 RGW-SYNC:meta: read sync status 2023-05-31T21:19:14.676 INFO:teuthology.orchestra.run.smithi099.stdout:{ 2023-05-31T21:19:14.677 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_status": { 2023-05-31T21:19:14.677 INFO:teuthology.orchestra.run.smithi099.stdout: "info": { 2023-05-31T21:19:14.677 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "sync", 2023-05-31T21:19:14.677 INFO:teuthology.orchestra.run.smithi099.stdout: "num_shards": 4, 2023-05-31T21:19:14.677 INFO:teuthology.orchestra.run.smithi099.stdout: "period": "2e6123d2-d695-4350-9a01-61a455bc782b", 2023-05-31T21:19:14.678 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2023-05-31T21:19:14.678 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:19:14.678 INFO:teuthology.orchestra.run.smithi099.stdout: "markers": [ 2023-05-31T21:19:14.678 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T21:19:14.678 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 0, 2023-05-31T21:19:14.678 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T21:19:14.678 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2023-05-31T21:19:14.679 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1685566501.757183_260.1", 2023-05-31T21:19:14.679 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T21:19:14.679 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 1, 2023-05-31T21:19:14.679 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T21:19:14.679 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2023-05-31T20:55:01.757183Z", 2023-05-31T21:19:14.679 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2023-05-31T21:19:14.680 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T21:19:14.680 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:19:14.680 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T21:19:14.680 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 1, 2023-05-31T21:19:14.680 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T21:19:14.680 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2023-05-31T21:19:14.680 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1685566501.735717_251.1", 2023-05-31T21:19:14.681 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T21:19:14.681 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2023-05-31T21:19:14.681 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T21:19:14.682 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2023-05-31T20:55:01.735717Z", 2023-05-31T21:19:14.682 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2023-05-31T21:19:14.682 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T21:19:14.682 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:19:14.682 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T21:19:14.683 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 2, 2023-05-31T21:19:14.683 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T21:19:14.683 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2023-05-31T21:19:14.683 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1685562970.274572_93.1", 2023-05-31T21:19:14.683 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T21:19:14.683 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2023-05-31T21:19:14.683 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T21:19:14.684 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2023-05-31T19:56:10.274572Z", 2023-05-31T21:19:14.684 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2023-05-31T21:19:14.684 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T21:19:14.684 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:19:14.684 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T21:19:14.684 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 3, 2023-05-31T21:19:14.685 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T21:19:14.685 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2023-05-31T21:19:14.685 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1685567930.890099_908.1", 2023-05-31T21:19:14.685 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T21:19:14.685 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2023-05-31T21:19:14.685 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T21:19:14.685 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2023-05-31T21:18:50.890099Z", 2023-05-31T21:19:14.686 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2023-05-31T21:19:14.686 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T21:19:14.686 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T21:19:14.686 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2023-05-31T21:19:14.686 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:19:14.686 INFO:teuthology.orchestra.run.smithi099.stdout: "full_sync": { 2023-05-31T21:19:14.686 INFO:teuthology.orchestra.run.smithi099.stdout: "total": 1, 2023-05-31T21:19:14.687 INFO:teuthology.orchestra.run.smithi099.stdout: "complete": 1 2023-05-31T21:19:14.687 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:19:14.687 INFO:teuthology.orchestra.run.smithi099.stdout: "current_time": "2023-05-31T21:19:14Z" 2023-05-31T21:19:14.687 INFO:teuthology.orchestra.run.smithi099.stdout:} 2023-05-31T21:19:14.688 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "2e6123d2-d695-4350-9a01-61a455bc782b", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1685566501.757183_260.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-05-31T20:55:01.757183Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1685566501.735717_251.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-05-31T20:55:01.735717Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1685562970.274572_93.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-05-31T19:56:10.274572Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1685567930.890099_908.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-05-31T21:18:50.890099Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-05-31T21:19:14Z" } 2023-05-31T21:19:14.688 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1685566501.757183_260.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-05-31T20:55:01.757183Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1685566501.735717_251.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-05-31T20:55:01.735717Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1685562970.274572_93.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-05-31T19:56:10.274572Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1685567930.890099_908.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-05-31T21:18:50.890099Z', 'realm_epoch': 2}}] 2023-05-31T21:19:14.689 DEBUG:rgw_multi.tests:log_status={0: '1_1685566501.757183_260.1', 1: '1_1685566501.735717_251.1', 2: '1_1685562970.274572_93.1', 3: '1_1685567943.999406_931.1'} 2023-05-31T21:19:14.689 DEBUG:rgw_multi.tests:sync_status={0: '1_1685566501.757183_260.1', 1: '1_1685566501.735717_251.1', 2: '1_1685562970.274572_93.1', 3: '1_1685567930.890099_908.1'} 2023-05-31T21:19:14.690 WARNING:rgw_multi.tests:zone a2 behind master: shard=3 master=1_1685567943.999406_931.1 target=1_1685567930.890099_908.1 2023-05-31T21:19:19.691 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-31T21:19:19.691 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-31T21:19:19.692 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-05-31T21:19:19.749 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2023-05-31T21:19:19.749 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2023-05-31T21:19:19.797 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T21:19:19.794+0000 7f4538345780 20 RGW-SYNC:meta: read sync status 2023-05-31T21:19:19.798 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T21:19:19.795+0000 7f4538345780 20 RGW-SYNC:meta: read sync status 2023-05-31T21:19:19.814 INFO:teuthology.orchestra.run.smithi099.stdout:{ 2023-05-31T21:19:19.814 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_status": { 2023-05-31T21:19:19.814 INFO:teuthology.orchestra.run.smithi099.stdout: "info": { 2023-05-31T21:19:19.815 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "sync", 2023-05-31T21:19:19.815 INFO:teuthology.orchestra.run.smithi099.stdout: "num_shards": 4, 2023-05-31T21:19:19.815 INFO:teuthology.orchestra.run.smithi099.stdout: "period": "2e6123d2-d695-4350-9a01-61a455bc782b", 2023-05-31T21:19:19.815 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2023-05-31T21:19:19.815 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:19:19.815 INFO:teuthology.orchestra.run.smithi099.stdout: "markers": [ 2023-05-31T21:19:19.816 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T21:19:19.816 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 0, 2023-05-31T21:19:19.816 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T21:19:19.816 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2023-05-31T21:19:19.816 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1685566501.757183_260.1", 2023-05-31T21:19:19.816 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T21:19:19.816 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 1, 2023-05-31T21:19:19.817 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T21:19:19.817 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2023-05-31T20:55:01.757183Z", 2023-05-31T21:19:19.817 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2023-05-31T21:19:19.817 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T21:19:19.817 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:19:19.817 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T21:19:19.817 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 1, 2023-05-31T21:19:19.818 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T21:19:19.818 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2023-05-31T21:19:19.818 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1685566501.735717_251.1", 2023-05-31T21:19:19.818 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T21:19:19.818 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2023-05-31T21:19:19.818 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T21:19:19.819 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2023-05-31T20:55:01.735717Z", 2023-05-31T21:19:19.819 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2023-05-31T21:19:19.819 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T21:19:19.819 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:19:19.819 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T21:19:19.819 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 2, 2023-05-31T21:19:19.819 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T21:19:19.820 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2023-05-31T21:19:19.820 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1685562970.274572_93.1", 2023-05-31T21:19:19.820 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T21:19:19.820 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2023-05-31T21:19:19.820 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T21:19:19.820 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2023-05-31T19:56:10.274572Z", 2023-05-31T21:19:19.820 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2023-05-31T21:19:19.821 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T21:19:19.821 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:19:19.821 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T21:19:19.821 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 3, 2023-05-31T21:19:19.821 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T21:19:19.821 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2023-05-31T21:19:19.822 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1685567930.890099_908.1", 2023-05-31T21:19:19.822 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T21:19:19.822 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2023-05-31T21:19:19.822 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T21:19:19.822 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2023-05-31T21:18:50.890099Z", 2023-05-31T21:19:19.822 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2023-05-31T21:19:19.822 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T21:19:19.823 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T21:19:19.823 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2023-05-31T21:19:19.823 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:19:19.823 INFO:teuthology.orchestra.run.smithi099.stdout: "full_sync": { 2023-05-31T21:19:19.823 INFO:teuthology.orchestra.run.smithi099.stdout: "total": 1, 2023-05-31T21:19:19.823 INFO:teuthology.orchestra.run.smithi099.stdout: "complete": 1 2023-05-31T21:19:19.823 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:19:19.824 INFO:teuthology.orchestra.run.smithi099.stdout: "current_time": "2023-05-31T21:19:19Z" 2023-05-31T21:19:19.824 INFO:teuthology.orchestra.run.smithi099.stdout:} 2023-05-31T21:19:19.825 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "2e6123d2-d695-4350-9a01-61a455bc782b", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1685566501.757183_260.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-05-31T20:55:01.757183Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1685566501.735717_251.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-05-31T20:55:01.735717Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1685562970.274572_93.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-05-31T19:56:10.274572Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1685567930.890099_908.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-05-31T21:18:50.890099Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-05-31T21:19:19Z" } 2023-05-31T21:19:19.825 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1685566501.757183_260.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-05-31T20:55:01.757183Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1685566501.735717_251.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-05-31T20:55:01.735717Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1685562970.274572_93.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-05-31T19:56:10.274572Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1685567930.890099_908.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-05-31T21:18:50.890099Z', 'realm_epoch': 2}}] 2023-05-31T21:19:19.826 DEBUG:rgw_multi.tests:log_status={0: '1_1685566501.757183_260.1', 1: '1_1685566501.735717_251.1', 2: '1_1685562970.274572_93.1', 3: '1_1685567943.999406_931.1'} 2023-05-31T21:19:19.826 DEBUG:rgw_multi.tests:sync_status={0: '1_1685566501.757183_260.1', 1: '1_1685566501.735717_251.1', 2: '1_1685562970.274572_93.1', 3: '1_1685567930.890099_908.1'} 2023-05-31T21:19:19.826 WARNING:rgw_multi.tests:zone a2 behind master: shard=3 master=1_1685567943.999406_931.1 target=1_1685567930.890099_908.1 2023-05-31T21:19:24.827 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-31T21:19:24.828 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-31T21:19:24.828 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-05-31T21:19:24.886 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2023-05-31T21:19:24.886 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2023-05-31T21:19:24.925 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T21:19:24.920+0000 7fe392dc4780 20 RGW-SYNC:meta: read sync status 2023-05-31T21:19:24.926 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T21:19:24.921+0000 7fe392dc4780 20 RGW-SYNC:meta: read sync status 2023-05-31T21:19:24.947 INFO:teuthology.orchestra.run.smithi099.stdout:{ 2023-05-31T21:19:24.947 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_status": { 2023-05-31T21:19:24.947 INFO:teuthology.orchestra.run.smithi099.stdout: "info": { 2023-05-31T21:19:24.948 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "sync", 2023-05-31T21:19:24.948 INFO:teuthology.orchestra.run.smithi099.stdout: "num_shards": 4, 2023-05-31T21:19:24.948 INFO:teuthology.orchestra.run.smithi099.stdout: "period": "2e6123d2-d695-4350-9a01-61a455bc782b", 2023-05-31T21:19:24.948 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2023-05-31T21:19:24.948 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:19:24.948 INFO:teuthology.orchestra.run.smithi099.stdout: "markers": [ 2023-05-31T21:19:24.949 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T21:19:24.949 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 0, 2023-05-31T21:19:24.949 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T21:19:24.949 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2023-05-31T21:19:24.949 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1685566501.757183_260.1", 2023-05-31T21:19:24.949 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T21:19:24.949 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 1, 2023-05-31T21:19:24.950 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T21:19:24.950 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2023-05-31T20:55:01.757183Z", 2023-05-31T21:19:24.950 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2023-05-31T21:19:24.950 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T21:19:24.950 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:19:24.950 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T21:19:24.951 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 1, 2023-05-31T21:19:24.951 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T21:19:24.951 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2023-05-31T21:19:24.951 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1685566501.735717_251.1", 2023-05-31T21:19:24.951 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T21:19:24.951 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2023-05-31T21:19:24.951 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T21:19:24.952 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2023-05-31T20:55:01.735717Z", 2023-05-31T21:19:24.952 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2023-05-31T21:19:24.952 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T21:19:24.952 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:19:24.952 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T21:19:24.952 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 2, 2023-05-31T21:19:24.953 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T21:19:24.953 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2023-05-31T21:19:24.953 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1685562970.274572_93.1", 2023-05-31T21:19:24.953 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T21:19:24.953 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2023-05-31T21:19:24.953 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T21:19:24.953 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2023-05-31T19:56:10.274572Z", 2023-05-31T21:19:24.954 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2023-05-31T21:19:24.954 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T21:19:24.954 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:19:24.954 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T21:19:24.954 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 3, 2023-05-31T21:19:24.954 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T21:19:24.954 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2023-05-31T21:19:24.955 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1685567930.890099_908.1", 2023-05-31T21:19:24.955 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T21:19:24.955 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2023-05-31T21:19:24.955 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T21:19:24.955 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2023-05-31T21:18:50.890099Z", 2023-05-31T21:19:24.955 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2023-05-31T21:19:24.955 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T21:19:24.956 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T21:19:24.956 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2023-05-31T21:19:24.956 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:19:24.956 INFO:teuthology.orchestra.run.smithi099.stdout: "full_sync": { 2023-05-31T21:19:24.956 INFO:teuthology.orchestra.run.smithi099.stdout: "total": 1, 2023-05-31T21:19:24.956 INFO:teuthology.orchestra.run.smithi099.stdout: "complete": 1 2023-05-31T21:19:24.957 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:19:24.957 INFO:teuthology.orchestra.run.smithi099.stdout: "current_time": "2023-05-31T21:19:24Z" 2023-05-31T21:19:24.957 INFO:teuthology.orchestra.run.smithi099.stdout:} 2023-05-31T21:19:24.958 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "2e6123d2-d695-4350-9a01-61a455bc782b", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1685566501.757183_260.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-05-31T20:55:01.757183Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1685566501.735717_251.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-05-31T20:55:01.735717Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1685562970.274572_93.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-05-31T19:56:10.274572Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1685567930.890099_908.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-05-31T21:18:50.890099Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-05-31T21:19:24Z" } 2023-05-31T21:19:24.958 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1685566501.757183_260.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-05-31T20:55:01.757183Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1685566501.735717_251.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-05-31T20:55:01.735717Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1685562970.274572_93.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-05-31T19:56:10.274572Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1685567930.890099_908.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-05-31T21:18:50.890099Z', 'realm_epoch': 2}}] 2023-05-31T21:19:24.959 DEBUG:rgw_multi.tests:log_status={0: '1_1685566501.757183_260.1', 1: '1_1685566501.735717_251.1', 2: '1_1685562970.274572_93.1', 3: '1_1685567943.999406_931.1'} 2023-05-31T21:19:24.959 DEBUG:rgw_multi.tests:sync_status={0: '1_1685566501.757183_260.1', 1: '1_1685566501.735717_251.1', 2: '1_1685562970.274572_93.1', 3: '1_1685567930.890099_908.1'} 2023-05-31T21:19:24.959 WARNING:rgw_multi.tests:zone a2 behind master: shard=3 master=1_1685567943.999406_931.1 target=1_1685567930.890099_908.1 2023-05-31T21:19:29.960 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-31T21:19:29.960 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-31T21:19:29.961 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-05-31T21:19:30.019 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2023-05-31T21:19:30.020 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2023-05-31T21:19:30.057 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T21:19:30.052+0000 7fb8145cc780 20 RGW-SYNC:meta: read sync status 2023-05-31T21:19:30.057 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T21:19:30.053+0000 7fb8145cc780 20 RGW-SYNC:meta: read sync status 2023-05-31T21:19:30.072 INFO:teuthology.orchestra.run.smithi099.stdout:{ 2023-05-31T21:19:30.072 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_status": { 2023-05-31T21:19:30.072 INFO:teuthology.orchestra.run.smithi099.stdout: "info": { 2023-05-31T21:19:30.072 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "sync", 2023-05-31T21:19:30.073 INFO:teuthology.orchestra.run.smithi099.stdout: "num_shards": 4, 2023-05-31T21:19:30.073 INFO:teuthology.orchestra.run.smithi099.stdout: "period": "2e6123d2-d695-4350-9a01-61a455bc782b", 2023-05-31T21:19:30.073 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2023-05-31T21:19:30.073 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:19:30.073 INFO:teuthology.orchestra.run.smithi099.stdout: "markers": [ 2023-05-31T21:19:30.073 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T21:19:30.073 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 0, 2023-05-31T21:19:30.074 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T21:19:30.074 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2023-05-31T21:19:30.074 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1685566501.757183_260.1", 2023-05-31T21:19:30.074 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T21:19:30.074 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 1, 2023-05-31T21:19:30.074 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T21:19:30.074 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2023-05-31T20:55:01.757183Z", 2023-05-31T21:19:30.075 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2023-05-31T21:19:30.075 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T21:19:30.075 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:19:30.075 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T21:19:30.075 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 1, 2023-05-31T21:19:30.075 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T21:19:30.075 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2023-05-31T21:19:30.076 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1685566501.735717_251.1", 2023-05-31T21:19:30.076 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T21:19:30.076 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2023-05-31T21:19:30.076 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T21:19:30.076 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2023-05-31T20:55:01.735717Z", 2023-05-31T21:19:30.076 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2023-05-31T21:19:30.077 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T21:19:30.077 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:19:30.077 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T21:19:30.077 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 2, 2023-05-31T21:19:30.077 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T21:19:30.077 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2023-05-31T21:19:30.077 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1685562970.274572_93.1", 2023-05-31T21:19:30.078 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T21:19:30.078 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2023-05-31T21:19:30.078 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T21:19:30.078 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2023-05-31T19:56:10.274572Z", 2023-05-31T21:19:30.078 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2023-05-31T21:19:30.078 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T21:19:30.078 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:19:30.079 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T21:19:30.079 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 3, 2023-05-31T21:19:30.079 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T21:19:30.079 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2023-05-31T21:19:30.079 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1685567930.890099_908.1", 2023-05-31T21:19:30.079 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T21:19:30.079 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2023-05-31T21:19:30.080 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T21:19:30.080 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2023-05-31T21:18:50.890099Z", 2023-05-31T21:19:30.080 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2023-05-31T21:19:30.081 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T21:19:30.081 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T21:19:30.081 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2023-05-31T21:19:30.081 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:19:30.081 INFO:teuthology.orchestra.run.smithi099.stdout: "full_sync": { 2023-05-31T21:19:30.081 INFO:teuthology.orchestra.run.smithi099.stdout: "total": 1, 2023-05-31T21:19:30.081 INFO:teuthology.orchestra.run.smithi099.stdout: "complete": 1 2023-05-31T21:19:30.082 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:19:30.082 INFO:teuthology.orchestra.run.smithi099.stdout: "current_time": "2023-05-31T21:19:30Z" 2023-05-31T21:19:30.082 INFO:teuthology.orchestra.run.smithi099.stdout:} 2023-05-31T21:19:30.083 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "2e6123d2-d695-4350-9a01-61a455bc782b", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1685566501.757183_260.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-05-31T20:55:01.757183Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1685566501.735717_251.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-05-31T20:55:01.735717Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1685562970.274572_93.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-05-31T19:56:10.274572Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1685567930.890099_908.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-05-31T21:18:50.890099Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-05-31T21:19:30Z" } 2023-05-31T21:19:30.083 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1685566501.757183_260.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-05-31T20:55:01.757183Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1685566501.735717_251.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-05-31T20:55:01.735717Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1685562970.274572_93.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-05-31T19:56:10.274572Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1685567930.890099_908.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-05-31T21:18:50.890099Z', 'realm_epoch': 2}}] 2023-05-31T21:19:30.084 DEBUG:rgw_multi.tests:log_status={0: '1_1685566501.757183_260.1', 1: '1_1685566501.735717_251.1', 2: '1_1685562970.274572_93.1', 3: '1_1685567943.999406_931.1'} 2023-05-31T21:19:30.085 DEBUG:rgw_multi.tests:sync_status={0: '1_1685566501.757183_260.1', 1: '1_1685566501.735717_251.1', 2: '1_1685562970.274572_93.1', 3: '1_1685567930.890099_908.1'} 2023-05-31T21:19:30.085 WARNING:rgw_multi.tests:zone a2 behind master: shard=3 master=1_1685567943.999406_931.1 target=1_1685567930.890099_908.1 2023-05-31T21:19:35.087 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-31T21:19:35.087 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-31T21:19:35.088 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-05-31T21:19:35.142 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2023-05-31T21:19:35.142 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2023-05-31T21:19:35.196 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T21:19:35.191+0000 7fbc09d1f780 20 RGW-SYNC:meta: read sync status 2023-05-31T21:19:35.196 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T21:19:35.192+0000 7fbc09d1f780 20 RGW-SYNC:meta: read sync status 2023-05-31T21:19:35.210 INFO:teuthology.orchestra.run.smithi099.stdout:{ 2023-05-31T21:19:35.210 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_status": { 2023-05-31T21:19:35.211 INFO:teuthology.orchestra.run.smithi099.stdout: "info": { 2023-05-31T21:19:35.211 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "sync", 2023-05-31T21:19:35.211 INFO:teuthology.orchestra.run.smithi099.stdout: "num_shards": 4, 2023-05-31T21:19:35.211 INFO:teuthology.orchestra.run.smithi099.stdout: "period": "2e6123d2-d695-4350-9a01-61a455bc782b", 2023-05-31T21:19:35.211 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2023-05-31T21:19:35.211 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:19:35.211 INFO:teuthology.orchestra.run.smithi099.stdout: "markers": [ 2023-05-31T21:19:35.212 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T21:19:35.212 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 0, 2023-05-31T21:19:35.212 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T21:19:35.212 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2023-05-31T21:19:35.212 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1685566501.757183_260.1", 2023-05-31T21:19:35.212 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T21:19:35.213 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 1, 2023-05-31T21:19:35.213 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T21:19:35.213 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2023-05-31T20:55:01.757183Z", 2023-05-31T21:19:35.213 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2023-05-31T21:19:35.213 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T21:19:35.213 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:19:35.213 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T21:19:35.214 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 1, 2023-05-31T21:19:35.214 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T21:19:35.214 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2023-05-31T21:19:35.214 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1685566501.735717_251.1", 2023-05-31T21:19:35.214 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T21:19:35.214 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2023-05-31T21:19:35.214 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T21:19:35.215 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2023-05-31T20:55:01.735717Z", 2023-05-31T21:19:35.215 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2023-05-31T21:19:35.215 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T21:19:35.215 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:19:35.215 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T21:19:35.215 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 2, 2023-05-31T21:19:35.215 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T21:19:35.216 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2023-05-31T21:19:35.216 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1685562970.274572_93.1", 2023-05-31T21:19:35.216 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T21:19:35.216 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2023-05-31T21:19:35.216 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T21:19:35.217 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2023-05-31T19:56:10.274572Z", 2023-05-31T21:19:35.217 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2023-05-31T21:19:35.217 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T21:19:35.217 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:19:35.217 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T21:19:35.218 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 3, 2023-05-31T21:19:35.218 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T21:19:35.218 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2023-05-31T21:19:35.218 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1685567930.890099_908.1", 2023-05-31T21:19:35.218 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T21:19:35.219 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2023-05-31T21:19:35.219 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T21:19:35.219 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2023-05-31T21:18:50.890099Z", 2023-05-31T21:19:35.219 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2023-05-31T21:19:35.219 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T21:19:35.219 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T21:19:35.219 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2023-05-31T21:19:35.220 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:19:35.220 INFO:teuthology.orchestra.run.smithi099.stdout: "full_sync": { 2023-05-31T21:19:35.220 INFO:teuthology.orchestra.run.smithi099.stdout: "total": 1, 2023-05-31T21:19:35.220 INFO:teuthology.orchestra.run.smithi099.stdout: "complete": 1 2023-05-31T21:19:35.220 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:19:35.220 INFO:teuthology.orchestra.run.smithi099.stdout: "current_time": "2023-05-31T21:19:35Z" 2023-05-31T21:19:35.220 INFO:teuthology.orchestra.run.smithi099.stdout:} 2023-05-31T21:19:35.221 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "2e6123d2-d695-4350-9a01-61a455bc782b", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1685566501.757183_260.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-05-31T20:55:01.757183Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1685566501.735717_251.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-05-31T20:55:01.735717Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1685562970.274572_93.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-05-31T19:56:10.274572Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1685567930.890099_908.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-05-31T21:18:50.890099Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-05-31T21:19:35Z" } 2023-05-31T21:19:35.222 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1685566501.757183_260.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-05-31T20:55:01.757183Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1685566501.735717_251.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-05-31T20:55:01.735717Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1685562970.274572_93.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-05-31T19:56:10.274572Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1685567930.890099_908.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-05-31T21:18:50.890099Z', 'realm_epoch': 2}}] 2023-05-31T21:19:35.222 DEBUG:rgw_multi.tests:log_status={0: '1_1685566501.757183_260.1', 1: '1_1685566501.735717_251.1', 2: '1_1685562970.274572_93.1', 3: '1_1685567943.999406_931.1'} 2023-05-31T21:19:35.223 DEBUG:rgw_multi.tests:sync_status={0: '1_1685566501.757183_260.1', 1: '1_1685566501.735717_251.1', 2: '1_1685562970.274572_93.1', 3: '1_1685567930.890099_908.1'} 2023-05-31T21:19:35.223 WARNING:rgw_multi.tests:zone a2 behind master: shard=3 master=1_1685567943.999406_931.1 target=1_1685567930.890099_908.1 2023-05-31T21:19:40.224 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-31T21:19:40.224 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-31T21:19:40.225 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-05-31T21:19:40.281 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2023-05-31T21:19:40.282 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2023-05-31T21:19:40.315 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T21:19:40.311+0000 7fd4ff7d0780 20 RGW-SYNC:meta: read sync status 2023-05-31T21:19:40.316 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T21:19:40.312+0000 7fd4ff7d0780 20 RGW-SYNC:meta: read sync status 2023-05-31T21:19:40.331 INFO:teuthology.orchestra.run.smithi099.stdout:{ 2023-05-31T21:19:40.331 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_status": { 2023-05-31T21:19:40.331 INFO:teuthology.orchestra.run.smithi099.stdout: "info": { 2023-05-31T21:19:40.332 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "sync", 2023-05-31T21:19:40.332 INFO:teuthology.orchestra.run.smithi099.stdout: "num_shards": 4, 2023-05-31T21:19:40.332 INFO:teuthology.orchestra.run.smithi099.stdout: "period": "2e6123d2-d695-4350-9a01-61a455bc782b", 2023-05-31T21:19:40.332 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2023-05-31T21:19:40.332 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:19:40.332 INFO:teuthology.orchestra.run.smithi099.stdout: "markers": [ 2023-05-31T21:19:40.333 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T21:19:40.333 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 0, 2023-05-31T21:19:40.333 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T21:19:40.333 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2023-05-31T21:19:40.333 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1685566501.757183_260.1", 2023-05-31T21:19:40.333 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T21:19:40.333 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 1, 2023-05-31T21:19:40.334 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T21:19:40.334 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2023-05-31T20:55:01.757183Z", 2023-05-31T21:19:40.334 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2023-05-31T21:19:40.334 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T21:19:40.334 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:19:40.334 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T21:19:40.334 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 1, 2023-05-31T21:19:40.335 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T21:19:40.335 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2023-05-31T21:19:40.335 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1685566501.735717_251.1", 2023-05-31T21:19:40.335 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T21:19:40.335 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2023-05-31T21:19:40.335 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T21:19:40.335 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2023-05-31T20:55:01.735717Z", 2023-05-31T21:19:40.336 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2023-05-31T21:19:40.336 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T21:19:40.336 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:19:40.336 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T21:19:40.336 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 2, 2023-05-31T21:19:40.336 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T21:19:40.337 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2023-05-31T21:19:40.337 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1685562970.274572_93.1", 2023-05-31T21:19:40.337 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T21:19:40.337 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2023-05-31T21:19:40.337 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T21:19:40.337 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2023-05-31T19:56:10.274572Z", 2023-05-31T21:19:40.337 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2023-05-31T21:19:40.338 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T21:19:40.338 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:19:40.338 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T21:19:40.338 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 3, 2023-05-31T21:19:40.338 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T21:19:40.338 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2023-05-31T21:19:40.338 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1685567930.890099_908.1", 2023-05-31T21:19:40.339 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T21:19:40.339 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2023-05-31T21:19:40.339 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T21:19:40.339 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2023-05-31T21:18:50.890099Z", 2023-05-31T21:19:40.339 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2023-05-31T21:19:40.339 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T21:19:40.339 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T21:19:40.340 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2023-05-31T21:19:40.340 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:19:40.340 INFO:teuthology.orchestra.run.smithi099.stdout: "full_sync": { 2023-05-31T21:19:40.340 INFO:teuthology.orchestra.run.smithi099.stdout: "total": 1, 2023-05-31T21:19:40.340 INFO:teuthology.orchestra.run.smithi099.stdout: "complete": 1 2023-05-31T21:19:40.340 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:19:40.341 INFO:teuthology.orchestra.run.smithi099.stdout: "current_time": "2023-05-31T21:19:40Z" 2023-05-31T21:19:40.341 INFO:teuthology.orchestra.run.smithi099.stdout:} 2023-05-31T21:19:40.341 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "2e6123d2-d695-4350-9a01-61a455bc782b", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1685566501.757183_260.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-05-31T20:55:01.757183Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1685566501.735717_251.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-05-31T20:55:01.735717Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1685562970.274572_93.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-05-31T19:56:10.274572Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1685567930.890099_908.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-05-31T21:18:50.890099Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-05-31T21:19:40Z" } 2023-05-31T21:19:40.342 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1685566501.757183_260.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-05-31T20:55:01.757183Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1685566501.735717_251.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-05-31T20:55:01.735717Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1685562970.274572_93.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-05-31T19:56:10.274572Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1685567930.890099_908.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-05-31T21:18:50.890099Z', 'realm_epoch': 2}}] 2023-05-31T21:19:40.342 DEBUG:rgw_multi.tests:log_status={0: '1_1685566501.757183_260.1', 1: '1_1685566501.735717_251.1', 2: '1_1685562970.274572_93.1', 3: '1_1685567943.999406_931.1'} 2023-05-31T21:19:40.343 DEBUG:rgw_multi.tests:sync_status={0: '1_1685566501.757183_260.1', 1: '1_1685566501.735717_251.1', 2: '1_1685562970.274572_93.1', 3: '1_1685567930.890099_908.1'} 2023-05-31T21:19:40.343 WARNING:rgw_multi.tests:zone a2 behind master: shard=3 master=1_1685567943.999406_931.1 target=1_1685567930.890099_908.1 2023-05-31T21:19:45.345 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-31T21:19:45.345 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-31T21:19:45.346 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-05-31T21:19:45.403 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2023-05-31T21:19:45.403 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2023-05-31T21:19:45.446 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T21:19:45.442+0000 7f1c41307780 20 RGW-SYNC:meta: read sync status 2023-05-31T21:19:45.447 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T21:19:45.443+0000 7f1c41307780 20 RGW-SYNC:meta: read sync status 2023-05-31T21:19:45.465 INFO:teuthology.orchestra.run.smithi099.stdout:{ 2023-05-31T21:19:45.466 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_status": { 2023-05-31T21:19:45.466 INFO:teuthology.orchestra.run.smithi099.stdout: "info": { 2023-05-31T21:19:45.466 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "sync", 2023-05-31T21:19:45.466 INFO:teuthology.orchestra.run.smithi099.stdout: "num_shards": 4, 2023-05-31T21:19:45.466 INFO:teuthology.orchestra.run.smithi099.stdout: "period": "2e6123d2-d695-4350-9a01-61a455bc782b", 2023-05-31T21:19:45.466 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2023-05-31T21:19:45.467 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:19:45.467 INFO:teuthology.orchestra.run.smithi099.stdout: "markers": [ 2023-05-31T21:19:45.467 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T21:19:45.467 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 0, 2023-05-31T21:19:45.467 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T21:19:45.467 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2023-05-31T21:19:45.468 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1685566501.757183_260.1", 2023-05-31T21:19:45.468 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T21:19:45.468 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 1, 2023-05-31T21:19:45.469 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T21:19:45.469 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2023-05-31T20:55:01.757183Z", 2023-05-31T21:19:45.469 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2023-05-31T21:19:45.469 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T21:19:45.470 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:19:45.470 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T21:19:45.470 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 1, 2023-05-31T21:19:45.470 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T21:19:45.470 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2023-05-31T21:19:45.470 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1685566501.735717_251.1", 2023-05-31T21:19:45.471 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T21:19:45.471 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2023-05-31T21:19:45.471 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T21:19:45.471 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2023-05-31T20:55:01.735717Z", 2023-05-31T21:19:45.471 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2023-05-31T21:19:45.471 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T21:19:45.472 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:19:45.472 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T21:19:45.472 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 2, 2023-05-31T21:19:45.472 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T21:19:45.472 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2023-05-31T21:19:45.472 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1685562970.274572_93.1", 2023-05-31T21:19:45.472 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T21:19:45.473 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2023-05-31T21:19:45.473 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T21:19:45.473 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2023-05-31T19:56:10.274572Z", 2023-05-31T21:19:45.473 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2023-05-31T21:19:45.474 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T21:19:45.474 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:19:45.474 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T21:19:45.474 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 3, 2023-05-31T21:19:45.475 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T21:19:45.475 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2023-05-31T21:19:45.475 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1685567930.890099_908.1", 2023-05-31T21:19:45.475 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T21:19:45.476 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2023-05-31T21:19:45.476 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T21:19:45.476 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2023-05-31T21:18:50.890099Z", 2023-05-31T21:19:45.476 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2023-05-31T21:19:45.477 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T21:19:45.477 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T21:19:45.477 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2023-05-31T21:19:45.478 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:19:45.478 INFO:teuthology.orchestra.run.smithi099.stdout: "full_sync": { 2023-05-31T21:19:45.478 INFO:teuthology.orchestra.run.smithi099.stdout: "total": 1, 2023-05-31T21:19:45.479 INFO:teuthology.orchestra.run.smithi099.stdout: "complete": 1 2023-05-31T21:19:45.479 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:19:45.479 INFO:teuthology.orchestra.run.smithi099.stdout: "current_time": "2023-05-31T21:19:45Z" 2023-05-31T21:19:45.480 INFO:teuthology.orchestra.run.smithi099.stdout:} 2023-05-31T21:19:45.481 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "2e6123d2-d695-4350-9a01-61a455bc782b", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1685566501.757183_260.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-05-31T20:55:01.757183Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1685566501.735717_251.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-05-31T20:55:01.735717Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1685562970.274572_93.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-05-31T19:56:10.274572Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1685567930.890099_908.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-05-31T21:18:50.890099Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-05-31T21:19:45Z" } 2023-05-31T21:19:45.481 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1685566501.757183_260.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-05-31T20:55:01.757183Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1685566501.735717_251.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-05-31T20:55:01.735717Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1685562970.274572_93.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-05-31T19:56:10.274572Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1685567930.890099_908.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-05-31T21:18:50.890099Z', 'realm_epoch': 2}}] 2023-05-31T21:19:45.482 DEBUG:rgw_multi.tests:log_status={0: '1_1685566501.757183_260.1', 1: '1_1685566501.735717_251.1', 2: '1_1685562970.274572_93.1', 3: '1_1685567943.999406_931.1'} 2023-05-31T21:19:45.482 DEBUG:rgw_multi.tests:sync_status={0: '1_1685566501.757183_260.1', 1: '1_1685566501.735717_251.1', 2: '1_1685562970.274572_93.1', 3: '1_1685567930.890099_908.1'} 2023-05-31T21:19:45.483 WARNING:rgw_multi.tests:zone a2 behind master: shard=3 master=1_1685567943.999406_931.1 target=1_1685567930.890099_908.1 2023-05-31T21:19:50.484 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-31T21:19:50.485 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-31T21:19:50.485 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-05-31T21:19:50.543 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2023-05-31T21:19:50.543 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2023-05-31T21:19:50.589 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T21:19:50.586+0000 7ff01f885780 20 RGW-SYNC:meta: read sync status 2023-05-31T21:19:50.590 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T21:19:50.587+0000 7ff01f885780 20 RGW-SYNC:meta: read sync status 2023-05-31T21:19:50.605 INFO:teuthology.orchestra.run.smithi099.stdout:{ 2023-05-31T21:19:50.605 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_status": { 2023-05-31T21:19:50.605 INFO:teuthology.orchestra.run.smithi099.stdout: "info": { 2023-05-31T21:19:50.605 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "sync", 2023-05-31T21:19:50.605 INFO:teuthology.orchestra.run.smithi099.stdout: "num_shards": 4, 2023-05-31T21:19:50.606 INFO:teuthology.orchestra.run.smithi099.stdout: "period": "2e6123d2-d695-4350-9a01-61a455bc782b", 2023-05-31T21:19:50.606 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2023-05-31T21:19:50.606 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:19:50.606 INFO:teuthology.orchestra.run.smithi099.stdout: "markers": [ 2023-05-31T21:19:50.606 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T21:19:50.606 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 0, 2023-05-31T21:19:50.607 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T21:19:50.607 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2023-05-31T21:19:50.607 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1685566501.757183_260.1", 2023-05-31T21:19:50.607 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T21:19:50.607 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 1, 2023-05-31T21:19:50.607 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T21:19:50.607 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2023-05-31T20:55:01.757183Z", 2023-05-31T21:19:50.608 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2023-05-31T21:19:50.608 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T21:19:50.608 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:19:50.608 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T21:19:50.608 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 1, 2023-05-31T21:19:50.608 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T21:19:50.608 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2023-05-31T21:19:50.609 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1685566501.735717_251.1", 2023-05-31T21:19:50.609 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T21:19:50.609 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2023-05-31T21:19:50.609 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T21:19:50.609 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2023-05-31T20:55:01.735717Z", 2023-05-31T21:19:50.609 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2023-05-31T21:19:50.610 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T21:19:50.610 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:19:50.610 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T21:19:50.610 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 2, 2023-05-31T21:19:50.610 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T21:19:50.610 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2023-05-31T21:19:50.610 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1685562970.274572_93.1", 2023-05-31T21:19:50.611 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T21:19:50.611 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2023-05-31T21:19:50.611 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T21:19:50.611 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2023-05-31T19:56:10.274572Z", 2023-05-31T21:19:50.611 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2023-05-31T21:19:50.611 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T21:19:50.611 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:19:50.612 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T21:19:50.612 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 3, 2023-05-31T21:19:50.612 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T21:19:50.612 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2023-05-31T21:19:50.612 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1685567930.890099_908.1", 2023-05-31T21:19:50.612 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T21:19:50.612 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2023-05-31T21:19:50.613 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T21:19:50.613 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2023-05-31T21:18:50.890099Z", 2023-05-31T21:19:50.613 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2023-05-31T21:19:50.613 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T21:19:50.613 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T21:19:50.613 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2023-05-31T21:19:50.613 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:19:50.614 INFO:teuthology.orchestra.run.smithi099.stdout: "full_sync": { 2023-05-31T21:19:50.614 INFO:teuthology.orchestra.run.smithi099.stdout: "total": 1, 2023-05-31T21:19:50.614 INFO:teuthology.orchestra.run.smithi099.stdout: "complete": 1 2023-05-31T21:19:50.614 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:19:50.614 INFO:teuthology.orchestra.run.smithi099.stdout: "current_time": "2023-05-31T21:19:50Z" 2023-05-31T21:19:50.614 INFO:teuthology.orchestra.run.smithi099.stdout:} 2023-05-31T21:19:50.615 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "2e6123d2-d695-4350-9a01-61a455bc782b", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1685566501.757183_260.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-05-31T20:55:01.757183Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1685566501.735717_251.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-05-31T20:55:01.735717Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1685562970.274572_93.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-05-31T19:56:10.274572Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1685567930.890099_908.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-05-31T21:18:50.890099Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-05-31T21:19:50Z" } 2023-05-31T21:19:50.616 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1685566501.757183_260.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-05-31T20:55:01.757183Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1685566501.735717_251.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-05-31T20:55:01.735717Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1685562970.274572_93.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-05-31T19:56:10.274572Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1685567930.890099_908.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-05-31T21:18:50.890099Z', 'realm_epoch': 2}}] 2023-05-31T21:19:50.616 DEBUG:rgw_multi.tests:log_status={0: '1_1685566501.757183_260.1', 1: '1_1685566501.735717_251.1', 2: '1_1685562970.274572_93.1', 3: '1_1685567943.999406_931.1'} 2023-05-31T21:19:50.617 DEBUG:rgw_multi.tests:sync_status={0: '1_1685566501.757183_260.1', 1: '1_1685566501.735717_251.1', 2: '1_1685562970.274572_93.1', 3: '1_1685567930.890099_908.1'} 2023-05-31T21:19:50.617 WARNING:rgw_multi.tests:zone a2 behind master: shard=3 master=1_1685567943.999406_931.1 target=1_1685567930.890099_908.1 2023-05-31T21:19:55.618 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-31T21:19:55.618 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-31T21:19:55.619 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-05-31T21:19:55.676 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2023-05-31T21:19:55.677 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2023-05-31T21:19:55.731 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T21:19:55.728+0000 7fb0b26de780 20 RGW-SYNC:meta: read sync status 2023-05-31T21:19:55.732 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T21:19:55.729+0000 7fb0b26de780 20 RGW-SYNC:meta: read sync status 2023-05-31T21:19:55.754 INFO:teuthology.orchestra.run.smithi099.stdout:{ 2023-05-31T21:19:55.754 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_status": { 2023-05-31T21:19:55.754 INFO:teuthology.orchestra.run.smithi099.stdout: "info": { 2023-05-31T21:19:55.754 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "sync", 2023-05-31T21:19:55.755 INFO:teuthology.orchestra.run.smithi099.stdout: "num_shards": 4, 2023-05-31T21:19:55.755 INFO:teuthology.orchestra.run.smithi099.stdout: "period": "2e6123d2-d695-4350-9a01-61a455bc782b", 2023-05-31T21:19:55.755 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2023-05-31T21:19:55.755 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:19:55.755 INFO:teuthology.orchestra.run.smithi099.stdout: "markers": [ 2023-05-31T21:19:55.755 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T21:19:55.755 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 0, 2023-05-31T21:19:55.756 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T21:19:55.756 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2023-05-31T21:19:55.756 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1685566501.757183_260.1", 2023-05-31T21:19:55.756 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T21:19:55.756 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 1, 2023-05-31T21:19:55.757 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T21:19:55.757 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2023-05-31T20:55:01.757183Z", 2023-05-31T21:19:55.757 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2023-05-31T21:19:55.757 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T21:19:55.757 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:19:55.757 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T21:19:55.758 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 1, 2023-05-31T21:19:55.758 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T21:19:55.758 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2023-05-31T21:19:55.758 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1685566501.735717_251.1", 2023-05-31T21:19:55.758 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T21:19:55.758 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2023-05-31T21:19:55.759 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T21:19:55.759 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2023-05-31T20:55:01.735717Z", 2023-05-31T21:19:55.759 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2023-05-31T21:19:55.759 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T21:19:55.759 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:19:55.759 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T21:19:55.760 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 2, 2023-05-31T21:19:55.760 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T21:19:55.760 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2023-05-31T21:19:55.760 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1685562970.274572_93.1", 2023-05-31T21:19:55.760 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T21:19:55.761 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2023-05-31T21:19:55.761 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T21:19:55.761 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2023-05-31T19:56:10.274572Z", 2023-05-31T21:19:55.761 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2023-05-31T21:19:55.761 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T21:19:55.761 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:19:55.762 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T21:19:55.762 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 3, 2023-05-31T21:19:55.762 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T21:19:55.762 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2023-05-31T21:19:55.762 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1685567930.890099_908.1", 2023-05-31T21:19:55.762 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T21:19:55.762 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2023-05-31T21:19:55.763 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T21:19:55.763 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2023-05-31T21:18:50.890099Z", 2023-05-31T21:19:55.763 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2023-05-31T21:19:55.763 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T21:19:55.763 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T21:19:55.763 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2023-05-31T21:19:55.764 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:19:55.764 INFO:teuthology.orchestra.run.smithi099.stdout: "full_sync": { 2023-05-31T21:19:55.764 INFO:teuthology.orchestra.run.smithi099.stdout: "total": 1, 2023-05-31T21:19:55.764 INFO:teuthology.orchestra.run.smithi099.stdout: "complete": 1 2023-05-31T21:19:55.764 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:19:55.764 INFO:teuthology.orchestra.run.smithi099.stdout: "current_time": "2023-05-31T21:19:55Z" 2023-05-31T21:19:55.765 INFO:teuthology.orchestra.run.smithi099.stdout:} 2023-05-31T21:19:55.766 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "2e6123d2-d695-4350-9a01-61a455bc782b", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1685566501.757183_260.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-05-31T20:55:01.757183Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1685566501.735717_251.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-05-31T20:55:01.735717Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1685562970.274572_93.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-05-31T19:56:10.274572Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1685567930.890099_908.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-05-31T21:18:50.890099Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-05-31T21:19:55Z" } 2023-05-31T21:19:55.766 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1685566501.757183_260.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-05-31T20:55:01.757183Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1685566501.735717_251.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-05-31T20:55:01.735717Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1685562970.274572_93.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-05-31T19:56:10.274572Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1685567930.890099_908.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-05-31T21:18:50.890099Z', 'realm_epoch': 2}}] 2023-05-31T21:19:55.767 DEBUG:rgw_multi.tests:log_status={0: '1_1685566501.757183_260.1', 1: '1_1685566501.735717_251.1', 2: '1_1685562970.274572_93.1', 3: '1_1685567943.999406_931.1'} 2023-05-31T21:19:55.767 DEBUG:rgw_multi.tests:sync_status={0: '1_1685566501.757183_260.1', 1: '1_1685566501.735717_251.1', 2: '1_1685562970.274572_93.1', 3: '1_1685567930.890099_908.1'} 2023-05-31T21:19:55.768 WARNING:rgw_multi.tests:zone a2 behind master: shard=3 master=1_1685567943.999406_931.1 target=1_1685567930.890099_908.1 2023-05-31T21:20:00.769 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-31T21:20:00.770 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-31T21:20:00.770 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-05-31T21:20:00.828 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2023-05-31T21:20:00.828 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2023-05-31T21:20:00.876 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T21:20:00.873+0000 7fa4bb910780 20 RGW-SYNC:meta: read sync status 2023-05-31T21:20:00.877 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T21:20:00.874+0000 7fa4bb910780 20 RGW-SYNC:meta: read sync status 2023-05-31T21:20:00.893 INFO:teuthology.orchestra.run.smithi099.stdout:{ 2023-05-31T21:20:00.893 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_status": { 2023-05-31T21:20:00.893 INFO:teuthology.orchestra.run.smithi099.stdout: "info": { 2023-05-31T21:20:00.894 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "sync", 2023-05-31T21:20:00.894 INFO:teuthology.orchestra.run.smithi099.stdout: "num_shards": 4, 2023-05-31T21:20:00.894 INFO:teuthology.orchestra.run.smithi099.stdout: "period": "2e6123d2-d695-4350-9a01-61a455bc782b", 2023-05-31T21:20:00.894 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2023-05-31T21:20:00.894 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:20:00.894 INFO:teuthology.orchestra.run.smithi099.stdout: "markers": [ 2023-05-31T21:20:00.894 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T21:20:00.895 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 0, 2023-05-31T21:20:00.895 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T21:20:00.895 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2023-05-31T21:20:00.895 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1685566501.757183_260.1", 2023-05-31T21:20:00.895 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T21:20:00.896 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 1, 2023-05-31T21:20:00.896 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T21:20:00.896 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2023-05-31T20:55:01.757183Z", 2023-05-31T21:20:00.896 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2023-05-31T21:20:00.896 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T21:20:00.896 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:20:00.897 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T21:20:00.897 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 1, 2023-05-31T21:20:00.897 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T21:20:00.897 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2023-05-31T21:20:00.897 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1685566501.735717_251.1", 2023-05-31T21:20:00.897 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T21:20:00.897 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2023-05-31T21:20:00.898 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T21:20:00.898 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2023-05-31T20:55:01.735717Z", 2023-05-31T21:20:00.899 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2023-05-31T21:20:00.900 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T21:20:00.900 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:20:00.900 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T21:20:00.900 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 2, 2023-05-31T21:20:00.900 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T21:20:00.901 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2023-05-31T21:20:00.901 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1685562970.274572_93.1", 2023-05-31T21:20:00.901 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T21:20:00.901 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2023-05-31T21:20:00.901 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T21:20:00.901 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2023-05-31T19:56:10.274572Z", 2023-05-31T21:20:00.902 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2023-05-31T21:20:00.902 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T21:20:00.902 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:20:00.902 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T21:20:00.902 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 3, 2023-05-31T21:20:00.903 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T21:20:00.903 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2023-05-31T21:20:00.903 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1685567930.890099_908.1", 2023-05-31T21:20:00.903 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T21:20:00.903 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2023-05-31T21:20:00.903 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T21:20:00.904 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2023-05-31T21:18:50.890099Z", 2023-05-31T21:20:00.904 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2023-05-31T21:20:00.904 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T21:20:00.904 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T21:20:00.904 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2023-05-31T21:20:00.905 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:20:00.905 INFO:teuthology.orchestra.run.smithi099.stdout: "full_sync": { 2023-05-31T21:20:00.905 INFO:teuthology.orchestra.run.smithi099.stdout: "total": 1, 2023-05-31T21:20:00.906 INFO:teuthology.orchestra.run.smithi099.stdout: "complete": 1 2023-05-31T21:20:00.906 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:20:00.906 INFO:teuthology.orchestra.run.smithi099.stdout: "current_time": "2023-05-31T21:20:00Z" 2023-05-31T21:20:00.907 INFO:teuthology.orchestra.run.smithi099.stdout:} 2023-05-31T21:20:00.908 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "2e6123d2-d695-4350-9a01-61a455bc782b", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1685566501.757183_260.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-05-31T20:55:01.757183Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1685566501.735717_251.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-05-31T20:55:01.735717Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1685562970.274572_93.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-05-31T19:56:10.274572Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1685567930.890099_908.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-05-31T21:18:50.890099Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-05-31T21:20:00Z" } 2023-05-31T21:20:00.909 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1685566501.757183_260.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-05-31T20:55:01.757183Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1685566501.735717_251.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-05-31T20:55:01.735717Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1685562970.274572_93.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-05-31T19:56:10.274572Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1685567930.890099_908.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-05-31T21:18:50.890099Z', 'realm_epoch': 2}}] 2023-05-31T21:20:00.909 DEBUG:rgw_multi.tests:log_status={0: '1_1685566501.757183_260.1', 1: '1_1685566501.735717_251.1', 2: '1_1685562970.274572_93.1', 3: '1_1685567943.999406_931.1'} 2023-05-31T21:20:00.910 DEBUG:rgw_multi.tests:sync_status={0: '1_1685566501.757183_260.1', 1: '1_1685566501.735717_251.1', 2: '1_1685562970.274572_93.1', 3: '1_1685567930.890099_908.1'} 2023-05-31T21:20:00.910 WARNING:rgw_multi.tests:zone a2 behind master: shard=3 master=1_1685567943.999406_931.1 target=1_1685567930.890099_908.1 2023-05-31T21:20:05.911 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-31T21:20:05.912 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-31T21:20:05.913 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-05-31T21:20:05.970 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2023-05-31T21:20:05.970 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2023-05-31T21:20:06.017 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T21:20:06.013+0000 7f085c0cd780 20 RGW-SYNC:meta: read sync status 2023-05-31T21:20:06.017 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T21:20:06.014+0000 7f085c0cd780 20 RGW-SYNC:meta: read sync status 2023-05-31T21:20:06.034 INFO:teuthology.orchestra.run.smithi099.stdout:{ 2023-05-31T21:20:06.034 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_status": { 2023-05-31T21:20:06.034 INFO:teuthology.orchestra.run.smithi099.stdout: "info": { 2023-05-31T21:20:06.034 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "sync", 2023-05-31T21:20:06.035 INFO:teuthology.orchestra.run.smithi099.stdout: "num_shards": 4, 2023-05-31T21:20:06.035 INFO:teuthology.orchestra.run.smithi099.stdout: "period": "2e6123d2-d695-4350-9a01-61a455bc782b", 2023-05-31T21:20:06.035 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2023-05-31T21:20:06.035 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:20:06.035 INFO:teuthology.orchestra.run.smithi099.stdout: "markers": [ 2023-05-31T21:20:06.035 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T21:20:06.036 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 0, 2023-05-31T21:20:06.036 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T21:20:06.036 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2023-05-31T21:20:06.036 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1685566501.757183_260.1", 2023-05-31T21:20:06.036 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T21:20:06.036 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 1, 2023-05-31T21:20:06.037 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T21:20:06.037 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2023-05-31T20:55:01.757183Z", 2023-05-31T21:20:06.037 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2023-05-31T21:20:06.037 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T21:20:06.037 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:20:06.037 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T21:20:06.038 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 1, 2023-05-31T21:20:06.038 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T21:20:06.038 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2023-05-31T21:20:06.038 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1685566501.735717_251.1", 2023-05-31T21:20:06.038 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T21:20:06.038 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2023-05-31T21:20:06.039 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T21:20:06.039 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2023-05-31T20:55:01.735717Z", 2023-05-31T21:20:06.039 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2023-05-31T21:20:06.039 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T21:20:06.039 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:20:06.039 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T21:20:06.039 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 2, 2023-05-31T21:20:06.040 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T21:20:06.040 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2023-05-31T21:20:06.040 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1685562970.274572_93.1", 2023-05-31T21:20:06.040 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T21:20:06.040 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2023-05-31T21:20:06.040 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T21:20:06.041 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2023-05-31T19:56:10.274572Z", 2023-05-31T21:20:06.041 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2023-05-31T21:20:06.041 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T21:20:06.041 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:20:06.041 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T21:20:06.041 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 3, 2023-05-31T21:20:06.042 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T21:20:06.042 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2023-05-31T21:20:06.042 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1685567930.890099_908.1", 2023-05-31T21:20:06.042 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T21:20:06.042 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2023-05-31T21:20:06.042 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T21:20:06.043 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2023-05-31T21:18:50.890099Z", 2023-05-31T21:20:06.043 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2023-05-31T21:20:06.043 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T21:20:06.043 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T21:20:06.043 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2023-05-31T21:20:06.043 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:20:06.043 INFO:teuthology.orchestra.run.smithi099.stdout: "full_sync": { 2023-05-31T21:20:06.044 INFO:teuthology.orchestra.run.smithi099.stdout: "total": 1, 2023-05-31T21:20:06.044 INFO:teuthology.orchestra.run.smithi099.stdout: "complete": 1 2023-05-31T21:20:06.044 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:20:06.044 INFO:teuthology.orchestra.run.smithi099.stdout: "current_time": "2023-05-31T21:20:06Z" 2023-05-31T21:20:06.045 INFO:teuthology.orchestra.run.smithi099.stdout:} 2023-05-31T21:20:06.045 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "2e6123d2-d695-4350-9a01-61a455bc782b", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1685566501.757183_260.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-05-31T20:55:01.757183Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1685566501.735717_251.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-05-31T20:55:01.735717Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1685562970.274572_93.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-05-31T19:56:10.274572Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1685567930.890099_908.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-05-31T21:18:50.890099Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-05-31T21:20:06Z" } 2023-05-31T21:20:06.046 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1685566501.757183_260.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-05-31T20:55:01.757183Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1685566501.735717_251.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-05-31T20:55:01.735717Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1685562970.274572_93.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-05-31T19:56:10.274572Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1685567930.890099_908.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-05-31T21:18:50.890099Z', 'realm_epoch': 2}}] 2023-05-31T21:20:06.046 DEBUG:rgw_multi.tests:log_status={0: '1_1685566501.757183_260.1', 1: '1_1685566501.735717_251.1', 2: '1_1685562970.274572_93.1', 3: '1_1685567943.999406_931.1'} 2023-05-31T21:20:06.047 DEBUG:rgw_multi.tests:sync_status={0: '1_1685566501.757183_260.1', 1: '1_1685566501.735717_251.1', 2: '1_1685562970.274572_93.1', 3: '1_1685567930.890099_908.1'} 2023-05-31T21:20:06.047 WARNING:rgw_multi.tests:zone a2 behind master: shard=3 master=1_1685567943.999406_931.1 target=1_1685567930.890099_908.1 2023-05-31T21:20:11.048 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-31T21:20:11.049 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-31T21:20:11.049 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-05-31T21:20:11.106 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2023-05-31T21:20:11.107 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2023-05-31T21:20:11.164 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T21:20:11.159+0000 7f823ed62780 20 RGW-SYNC:meta: read sync status 2023-05-31T21:20:11.164 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T21:20:11.160+0000 7f823ed62780 20 RGW-SYNC:meta: read sync status 2023-05-31T21:20:11.186 INFO:teuthology.orchestra.run.smithi099.stdout:{ 2023-05-31T21:20:11.186 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_status": { 2023-05-31T21:20:11.186 INFO:teuthology.orchestra.run.smithi099.stdout: "info": { 2023-05-31T21:20:11.186 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "sync", 2023-05-31T21:20:11.187 INFO:teuthology.orchestra.run.smithi099.stdout: "num_shards": 4, 2023-05-31T21:20:11.187 INFO:teuthology.orchestra.run.smithi099.stdout: "period": "2e6123d2-d695-4350-9a01-61a455bc782b", 2023-05-31T21:20:11.187 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2023-05-31T21:20:11.187 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:20:11.187 INFO:teuthology.orchestra.run.smithi099.stdout: "markers": [ 2023-05-31T21:20:11.187 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T21:20:11.188 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 0, 2023-05-31T21:20:11.188 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T21:20:11.188 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2023-05-31T21:20:11.188 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1685566501.757183_260.1", 2023-05-31T21:20:11.189 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T21:20:11.189 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 1, 2023-05-31T21:20:11.189 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T21:20:11.189 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2023-05-31T20:55:01.757183Z", 2023-05-31T21:20:11.189 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2023-05-31T21:20:11.189 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T21:20:11.190 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:20:11.190 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T21:20:11.190 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 1, 2023-05-31T21:20:11.190 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T21:20:11.190 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2023-05-31T21:20:11.190 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1685566501.735717_251.1", 2023-05-31T21:20:11.190 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T21:20:11.191 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2023-05-31T21:20:11.191 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T21:20:11.191 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2023-05-31T20:55:01.735717Z", 2023-05-31T21:20:11.191 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2023-05-31T21:20:11.191 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T21:20:11.191 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:20:11.191 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T21:20:11.192 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 2, 2023-05-31T21:20:11.192 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T21:20:11.192 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2023-05-31T21:20:11.192 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1685562970.274572_93.1", 2023-05-31T21:20:11.192 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T21:20:11.192 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2023-05-31T21:20:11.193 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T21:20:11.193 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2023-05-31T19:56:10.274572Z", 2023-05-31T21:20:11.193 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2023-05-31T21:20:11.193 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T21:20:11.193 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:20:11.193 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T21:20:11.193 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 3, 2023-05-31T21:20:11.194 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T21:20:11.194 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2023-05-31T21:20:11.194 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1685567930.890099_908.1", 2023-05-31T21:20:11.194 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T21:20:11.194 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2023-05-31T21:20:11.194 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T21:20:11.195 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2023-05-31T21:18:50.890099Z", 2023-05-31T21:20:11.195 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2023-05-31T21:20:11.195 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T21:20:11.195 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T21:20:11.195 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2023-05-31T21:20:11.195 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:20:11.196 INFO:teuthology.orchestra.run.smithi099.stdout: "full_sync": { 2023-05-31T21:20:11.196 INFO:teuthology.orchestra.run.smithi099.stdout: "total": 1, 2023-05-31T21:20:11.196 INFO:teuthology.orchestra.run.smithi099.stdout: "complete": 1 2023-05-31T21:20:11.196 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:20:11.196 INFO:teuthology.orchestra.run.smithi099.stdout: "current_time": "2023-05-31T21:20:11Z" 2023-05-31T21:20:11.196 INFO:teuthology.orchestra.run.smithi099.stdout:} 2023-05-31T21:20:11.197 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "2e6123d2-d695-4350-9a01-61a455bc782b", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1685566501.757183_260.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-05-31T20:55:01.757183Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1685566501.735717_251.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-05-31T20:55:01.735717Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1685562970.274572_93.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-05-31T19:56:10.274572Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1685567930.890099_908.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-05-31T21:18:50.890099Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-05-31T21:20:11Z" } 2023-05-31T21:20:11.198 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1685566501.757183_260.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-05-31T20:55:01.757183Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1685566501.735717_251.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-05-31T20:55:01.735717Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1685562970.274572_93.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-05-31T19:56:10.274572Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1685567930.890099_908.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-05-31T21:18:50.890099Z', 'realm_epoch': 2}}] 2023-05-31T21:20:11.198 DEBUG:rgw_multi.tests:log_status={0: '1_1685566501.757183_260.1', 1: '1_1685566501.735717_251.1', 2: '1_1685562970.274572_93.1', 3: '1_1685567943.999406_931.1'} 2023-05-31T21:20:11.199 DEBUG:rgw_multi.tests:sync_status={0: '1_1685566501.757183_260.1', 1: '1_1685566501.735717_251.1', 2: '1_1685562970.274572_93.1', 3: '1_1685567930.890099_908.1'} 2023-05-31T21:20:11.200 WARNING:rgw_multi.tests:zone a2 behind master: shard=3 master=1_1685567943.999406_931.1 target=1_1685567930.890099_908.1 2023-05-31T21:20:16.202 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-31T21:20:16.203 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-31T21:20:16.204 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-05-31T21:20:16.261 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2023-05-31T21:20:16.262 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2023-05-31T21:20:16.312 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T21:20:16.307+0000 7f09717d7780 20 RGW-SYNC:meta: read sync status 2023-05-31T21:20:16.312 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T21:20:16.308+0000 7f09717d7780 20 RGW-SYNC:meta: read sync status 2023-05-31T21:20:16.327 INFO:teuthology.orchestra.run.smithi099.stdout:{ 2023-05-31T21:20:16.327 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_status": { 2023-05-31T21:20:16.327 INFO:teuthology.orchestra.run.smithi099.stdout: "info": { 2023-05-31T21:20:16.327 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "sync", 2023-05-31T21:20:16.327 INFO:teuthology.orchestra.run.smithi099.stdout: "num_shards": 4, 2023-05-31T21:20:16.327 INFO:teuthology.orchestra.run.smithi099.stdout: "period": "2e6123d2-d695-4350-9a01-61a455bc782b", 2023-05-31T21:20:16.328 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2023-05-31T21:20:16.328 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:20:16.328 INFO:teuthology.orchestra.run.smithi099.stdout: "markers": [ 2023-05-31T21:20:16.328 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T21:20:16.328 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 0, 2023-05-31T21:20:16.328 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T21:20:16.329 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2023-05-31T21:20:16.329 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1685566501.757183_260.1", 2023-05-31T21:20:16.329 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T21:20:16.329 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 1, 2023-05-31T21:20:16.329 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T21:20:16.329 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2023-05-31T20:55:01.757183Z", 2023-05-31T21:20:16.330 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2023-05-31T21:20:16.330 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T21:20:16.330 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:20:16.330 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T21:20:16.330 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 1, 2023-05-31T21:20:16.330 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T21:20:16.330 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2023-05-31T21:20:16.331 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1685566501.735717_251.1", 2023-05-31T21:20:16.331 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T21:20:16.331 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2023-05-31T21:20:16.331 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T21:20:16.331 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2023-05-31T20:55:01.735717Z", 2023-05-31T21:20:16.331 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2023-05-31T21:20:16.331 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T21:20:16.332 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:20:16.332 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T21:20:16.332 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 2, 2023-05-31T21:20:16.332 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T21:20:16.332 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2023-05-31T21:20:16.332 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1685562970.274572_93.1", 2023-05-31T21:20:16.333 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T21:20:16.333 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2023-05-31T21:20:16.333 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T21:20:16.333 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2023-05-31T19:56:10.274572Z", 2023-05-31T21:20:16.333 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2023-05-31T21:20:16.333 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T21:20:16.333 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:20:16.334 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T21:20:16.334 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 3, 2023-05-31T21:20:16.334 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T21:20:16.334 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2023-05-31T21:20:16.334 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1685567930.890099_908.1", 2023-05-31T21:20:16.334 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T21:20:16.334 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2023-05-31T21:20:16.335 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T21:20:16.335 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2023-05-31T21:18:50.890099Z", 2023-05-31T21:20:16.335 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2023-05-31T21:20:16.335 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T21:20:16.335 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T21:20:16.335 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2023-05-31T21:20:16.335 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:20:16.336 INFO:teuthology.orchestra.run.smithi099.stdout: "full_sync": { 2023-05-31T21:20:16.336 INFO:teuthology.orchestra.run.smithi099.stdout: "total": 1, 2023-05-31T21:20:16.336 INFO:teuthology.orchestra.run.smithi099.stdout: "complete": 1 2023-05-31T21:20:16.336 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:20:16.336 INFO:teuthology.orchestra.run.smithi099.stdout: "current_time": "2023-05-31T21:20:16Z" 2023-05-31T21:20:16.336 INFO:teuthology.orchestra.run.smithi099.stdout:} 2023-05-31T21:20:16.337 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "2e6123d2-d695-4350-9a01-61a455bc782b", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1685566501.757183_260.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-05-31T20:55:01.757183Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1685566501.735717_251.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-05-31T20:55:01.735717Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1685562970.274572_93.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-05-31T19:56:10.274572Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1685567930.890099_908.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-05-31T21:18:50.890099Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-05-31T21:20:16Z" } 2023-05-31T21:20:16.338 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1685566501.757183_260.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-05-31T20:55:01.757183Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1685566501.735717_251.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-05-31T20:55:01.735717Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1685562970.274572_93.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-05-31T19:56:10.274572Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1685567930.890099_908.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-05-31T21:18:50.890099Z', 'realm_epoch': 2}}] 2023-05-31T21:20:16.338 DEBUG:rgw_multi.tests:log_status={0: '1_1685566501.757183_260.1', 1: '1_1685566501.735717_251.1', 2: '1_1685562970.274572_93.1', 3: '1_1685567943.999406_931.1'} 2023-05-31T21:20:16.339 DEBUG:rgw_multi.tests:sync_status={0: '1_1685566501.757183_260.1', 1: '1_1685566501.735717_251.1', 2: '1_1685562970.274572_93.1', 3: '1_1685567930.890099_908.1'} 2023-05-31T21:20:16.339 WARNING:rgw_multi.tests:zone a2 behind master: shard=3 master=1_1685567943.999406_931.1 target=1_1685567930.890099_908.1 2023-05-31T21:20:21.340 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-31T21:20:21.341 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-31T21:20:21.341 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-05-31T21:20:21.399 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2023-05-31T21:20:21.399 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2023-05-31T21:20:21.440 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T21:20:21.435+0000 7f895a046780 20 RGW-SYNC:meta: read sync status 2023-05-31T21:20:21.440 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T21:20:21.436+0000 7f895a046780 20 RGW-SYNC:meta: read sync status 2023-05-31T21:20:21.461 INFO:teuthology.orchestra.run.smithi099.stdout:{ 2023-05-31T21:20:21.461 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_status": { 2023-05-31T21:20:21.461 INFO:teuthology.orchestra.run.smithi099.stdout: "info": { 2023-05-31T21:20:21.461 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "sync", 2023-05-31T21:20:21.462 INFO:teuthology.orchestra.run.smithi099.stdout: "num_shards": 4, 2023-05-31T21:20:21.462 INFO:teuthology.orchestra.run.smithi099.stdout: "period": "2e6123d2-d695-4350-9a01-61a455bc782b", 2023-05-31T21:20:21.462 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2023-05-31T21:20:21.462 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:20:21.462 INFO:teuthology.orchestra.run.smithi099.stdout: "markers": [ 2023-05-31T21:20:21.462 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T21:20:21.463 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 0, 2023-05-31T21:20:21.463 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T21:20:21.463 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2023-05-31T21:20:21.463 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1685566501.757183_260.1", 2023-05-31T21:20:21.463 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T21:20:21.463 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 1, 2023-05-31T21:20:21.464 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T21:20:21.464 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2023-05-31T20:55:01.757183Z", 2023-05-31T21:20:21.464 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2023-05-31T21:20:21.464 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T21:20:21.464 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:20:21.464 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T21:20:21.465 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 1, 2023-05-31T21:20:21.465 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T21:20:21.465 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2023-05-31T21:20:21.466 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1685566501.735717_251.1", 2023-05-31T21:20:21.466 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T21:20:21.466 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2023-05-31T21:20:21.467 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T21:20:21.467 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2023-05-31T20:55:01.735717Z", 2023-05-31T21:20:21.467 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2023-05-31T21:20:21.468 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T21:20:21.468 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:20:21.468 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T21:20:21.469 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 2, 2023-05-31T21:20:21.469 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T21:20:21.469 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2023-05-31T21:20:21.469 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1685562970.274572_93.1", 2023-05-31T21:20:21.470 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T21:20:21.470 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2023-05-31T21:20:21.470 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T21:20:21.470 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2023-05-31T19:56:10.274572Z", 2023-05-31T21:20:21.471 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2023-05-31T21:20:21.471 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T21:20:21.471 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:20:21.472 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T21:20:21.472 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 3, 2023-05-31T21:20:21.472 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T21:20:21.472 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2023-05-31T21:20:21.473 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1685567930.890099_908.1", 2023-05-31T21:20:21.473 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T21:20:21.473 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2023-05-31T21:20:21.473 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T21:20:21.473 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2023-05-31T21:18:50.890099Z", 2023-05-31T21:20:21.473 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2023-05-31T21:20:21.474 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T21:20:21.474 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T21:20:21.474 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2023-05-31T21:20:21.474 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:20:21.474 INFO:teuthology.orchestra.run.smithi099.stdout: "full_sync": { 2023-05-31T21:20:21.474 INFO:teuthology.orchestra.run.smithi099.stdout: "total": 1, 2023-05-31T21:20:21.475 INFO:teuthology.orchestra.run.smithi099.stdout: "complete": 1 2023-05-31T21:20:21.475 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:20:21.475 INFO:teuthology.orchestra.run.smithi099.stdout: "current_time": "2023-05-31T21:20:21Z" 2023-05-31T21:20:21.475 INFO:teuthology.orchestra.run.smithi099.stdout:} 2023-05-31T21:20:21.476 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "2e6123d2-d695-4350-9a01-61a455bc782b", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1685566501.757183_260.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-05-31T20:55:01.757183Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1685566501.735717_251.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-05-31T20:55:01.735717Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1685562970.274572_93.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-05-31T19:56:10.274572Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1685567930.890099_908.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-05-31T21:18:50.890099Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-05-31T21:20:21Z" } 2023-05-31T21:20:21.476 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1685566501.757183_260.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-05-31T20:55:01.757183Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1685566501.735717_251.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-05-31T20:55:01.735717Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1685562970.274572_93.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-05-31T19:56:10.274572Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1685567930.890099_908.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-05-31T21:18:50.890099Z', 'realm_epoch': 2}}] 2023-05-31T21:20:21.477 DEBUG:rgw_multi.tests:log_status={0: '1_1685566501.757183_260.1', 1: '1_1685566501.735717_251.1', 2: '1_1685562970.274572_93.1', 3: '1_1685567943.999406_931.1'} 2023-05-31T21:20:21.477 DEBUG:rgw_multi.tests:sync_status={0: '1_1685566501.757183_260.1', 1: '1_1685566501.735717_251.1', 2: '1_1685562970.274572_93.1', 3: '1_1685567930.890099_908.1'} 2023-05-31T21:20:21.478 WARNING:rgw_multi.tests:zone a2 behind master: shard=3 master=1_1685567943.999406_931.1 target=1_1685567930.890099_908.1 2023-05-31T21:20:26.479 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-31T21:20:26.480 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-31T21:20:26.481 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-05-31T21:20:26.538 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2023-05-31T21:20:26.539 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2023-05-31T21:20:26.590 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T21:20:26.586+0000 7f8753c37780 20 RGW-SYNC:meta: read sync status 2023-05-31T21:20:26.591 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T21:20:26.587+0000 7f8753c37780 20 RGW-SYNC:meta: read sync status 2023-05-31T21:20:26.611 INFO:teuthology.orchestra.run.smithi099.stdout:{ 2023-05-31T21:20:26.612 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_status": { 2023-05-31T21:20:26.612 INFO:teuthology.orchestra.run.smithi099.stdout: "info": { 2023-05-31T21:20:26.613 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "sync", 2023-05-31T21:20:26.613 INFO:teuthology.orchestra.run.smithi099.stdout: "num_shards": 4, 2023-05-31T21:20:26.613 INFO:teuthology.orchestra.run.smithi099.stdout: "period": "2e6123d2-d695-4350-9a01-61a455bc782b", 2023-05-31T21:20:26.613 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2023-05-31T21:20:26.614 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:20:26.614 INFO:teuthology.orchestra.run.smithi099.stdout: "markers": [ 2023-05-31T21:20:26.614 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T21:20:26.615 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 0, 2023-05-31T21:20:26.615 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T21:20:26.615 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2023-05-31T21:20:26.616 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1685566501.757183_260.1", 2023-05-31T21:20:26.616 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T21:20:26.616 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 1, 2023-05-31T21:20:26.617 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T21:20:26.617 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2023-05-31T20:55:01.757183Z", 2023-05-31T21:20:26.617 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2023-05-31T21:20:26.618 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T21:20:26.618 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:20:26.618 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T21:20:26.619 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 1, 2023-05-31T21:20:26.619 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T21:20:26.619 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2023-05-31T21:20:26.619 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1685566501.735717_251.1", 2023-05-31T21:20:26.620 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T21:20:26.620 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2023-05-31T21:20:26.620 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T21:20:26.621 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2023-05-31T20:55:01.735717Z", 2023-05-31T21:20:26.621 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2023-05-31T21:20:26.621 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T21:20:26.621 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:20:26.622 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T21:20:26.622 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 2, 2023-05-31T21:20:26.622 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T21:20:26.623 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2023-05-31T21:20:26.623 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1685562970.274572_93.1", 2023-05-31T21:20:26.623 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T21:20:26.624 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2023-05-31T21:20:26.624 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T21:20:26.624 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2023-05-31T19:56:10.274572Z", 2023-05-31T21:20:26.624 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2023-05-31T21:20:26.625 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T21:20:26.625 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:20:26.625 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T21:20:26.625 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 3, 2023-05-31T21:20:26.626 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T21:20:26.626 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2023-05-31T21:20:26.626 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1685567930.890099_908.1", 2023-05-31T21:20:26.626 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T21:20:26.627 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2023-05-31T21:20:26.627 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T21:20:26.627 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2023-05-31T21:18:50.890099Z", 2023-05-31T21:20:26.628 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2023-05-31T21:20:26.628 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T21:20:26.628 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T21:20:26.629 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2023-05-31T21:20:26.629 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:20:26.630 INFO:teuthology.orchestra.run.smithi099.stdout: "full_sync": { 2023-05-31T21:20:26.630 INFO:teuthology.orchestra.run.smithi099.stdout: "total": 1, 2023-05-31T21:20:26.630 INFO:teuthology.orchestra.run.smithi099.stdout: "complete": 1 2023-05-31T21:20:26.631 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:20:26.631 INFO:teuthology.orchestra.run.smithi099.stdout: "current_time": "2023-05-31T21:20:26Z" 2023-05-31T21:20:26.631 INFO:teuthology.orchestra.run.smithi099.stdout:} 2023-05-31T21:20:26.633 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "2e6123d2-d695-4350-9a01-61a455bc782b", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1685566501.757183_260.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-05-31T20:55:01.757183Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1685566501.735717_251.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-05-31T20:55:01.735717Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1685562970.274572_93.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-05-31T19:56:10.274572Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1685567930.890099_908.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-05-31T21:18:50.890099Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-05-31T21:20:26Z" } 2023-05-31T21:20:26.634 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1685566501.757183_260.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-05-31T20:55:01.757183Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1685566501.735717_251.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-05-31T20:55:01.735717Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1685562970.274572_93.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-05-31T19:56:10.274572Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1685567930.890099_908.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-05-31T21:18:50.890099Z', 'realm_epoch': 2}}] 2023-05-31T21:20:26.635 DEBUG:rgw_multi.tests:log_status={0: '1_1685566501.757183_260.1', 1: '1_1685566501.735717_251.1', 2: '1_1685562970.274572_93.1', 3: '1_1685567943.999406_931.1'} 2023-05-31T21:20:26.635 DEBUG:rgw_multi.tests:sync_status={0: '1_1685566501.757183_260.1', 1: '1_1685566501.735717_251.1', 2: '1_1685562970.274572_93.1', 3: '1_1685567930.890099_908.1'} 2023-05-31T21:20:26.636 WARNING:rgw_multi.tests:zone a2 behind master: shard=3 master=1_1685567943.999406_931.1 target=1_1685567930.890099_908.1 2023-05-31T21:20:31.638 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-31T21:20:31.638 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-31T21:20:31.639 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-05-31T21:20:31.696 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2023-05-31T21:20:31.697 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2023-05-31T21:20:31.731 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T21:20:31.728+0000 7fdb76055780 20 RGW-SYNC:meta: read sync status 2023-05-31T21:20:31.732 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T21:20:31.728+0000 7fdb76055780 20 RGW-SYNC:meta: read sync status 2023-05-31T21:20:31.746 INFO:teuthology.orchestra.run.smithi099.stdout:{ 2023-05-31T21:20:31.747 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_status": { 2023-05-31T21:20:31.747 INFO:teuthology.orchestra.run.smithi099.stdout: "info": { 2023-05-31T21:20:31.747 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "sync", 2023-05-31T21:20:31.747 INFO:teuthology.orchestra.run.smithi099.stdout: "num_shards": 4, 2023-05-31T21:20:31.747 INFO:teuthology.orchestra.run.smithi099.stdout: "period": "2e6123d2-d695-4350-9a01-61a455bc782b", 2023-05-31T21:20:31.747 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2023-05-31T21:20:31.748 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:20:31.748 INFO:teuthology.orchestra.run.smithi099.stdout: "markers": [ 2023-05-31T21:20:31.748 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T21:20:31.748 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 0, 2023-05-31T21:20:31.748 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T21:20:31.749 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2023-05-31T21:20:31.749 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1685566501.757183_260.1", 2023-05-31T21:20:31.749 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T21:20:31.749 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 1, 2023-05-31T21:20:31.749 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T21:20:31.749 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2023-05-31T20:55:01.757183Z", 2023-05-31T21:20:31.750 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2023-05-31T21:20:31.750 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T21:20:31.750 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:20:31.750 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T21:20:31.750 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 1, 2023-05-31T21:20:31.750 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T21:20:31.750 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2023-05-31T21:20:31.751 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1685566501.735717_251.1", 2023-05-31T21:20:31.751 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T21:20:31.751 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2023-05-31T21:20:31.751 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T21:20:31.751 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2023-05-31T20:55:01.735717Z", 2023-05-31T21:20:31.751 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2023-05-31T21:20:31.751 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T21:20:31.752 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:20:31.752 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T21:20:31.752 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 2, 2023-05-31T21:20:31.752 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T21:20:31.752 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2023-05-31T21:20:31.752 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1685562970.274572_93.1", 2023-05-31T21:20:31.752 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T21:20:31.753 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2023-05-31T21:20:31.753 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T21:20:31.753 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2023-05-31T19:56:10.274572Z", 2023-05-31T21:20:31.753 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2023-05-31T21:20:31.753 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T21:20:31.753 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:20:31.753 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T21:20:31.754 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 3, 2023-05-31T21:20:31.754 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T21:20:31.754 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2023-05-31T21:20:31.754 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1685567930.890099_908.1", 2023-05-31T21:20:31.754 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T21:20:31.754 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2023-05-31T21:20:31.754 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T21:20:31.755 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2023-05-31T21:18:50.890099Z", 2023-05-31T21:20:31.755 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2023-05-31T21:20:31.755 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T21:20:31.755 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T21:20:31.755 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2023-05-31T21:20:31.755 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:20:31.755 INFO:teuthology.orchestra.run.smithi099.stdout: "full_sync": { 2023-05-31T21:20:31.756 INFO:teuthology.orchestra.run.smithi099.stdout: "total": 1, 2023-05-31T21:20:31.756 INFO:teuthology.orchestra.run.smithi099.stdout: "complete": 1 2023-05-31T21:20:31.756 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:20:31.756 INFO:teuthology.orchestra.run.smithi099.stdout: "current_time": "2023-05-31T21:20:31Z" 2023-05-31T21:20:31.756 INFO:teuthology.orchestra.run.smithi099.stdout:} 2023-05-31T21:20:31.757 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "2e6123d2-d695-4350-9a01-61a455bc782b", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1685566501.757183_260.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-05-31T20:55:01.757183Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1685566501.735717_251.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-05-31T20:55:01.735717Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1685562970.274572_93.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-05-31T19:56:10.274572Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1685567930.890099_908.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-05-31T21:18:50.890099Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-05-31T21:20:31Z" } 2023-05-31T21:20:31.758 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1685566501.757183_260.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-05-31T20:55:01.757183Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1685566501.735717_251.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-05-31T20:55:01.735717Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1685562970.274572_93.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-05-31T19:56:10.274572Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1685567930.890099_908.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-05-31T21:18:50.890099Z', 'realm_epoch': 2}}] 2023-05-31T21:20:31.758 DEBUG:rgw_multi.tests:log_status={0: '1_1685566501.757183_260.1', 1: '1_1685566501.735717_251.1', 2: '1_1685562970.274572_93.1', 3: '1_1685567943.999406_931.1'} 2023-05-31T21:20:31.758 DEBUG:rgw_multi.tests:sync_status={0: '1_1685566501.757183_260.1', 1: '1_1685566501.735717_251.1', 2: '1_1685562970.274572_93.1', 3: '1_1685567930.890099_908.1'} 2023-05-31T21:20:31.759 WARNING:rgw_multi.tests:zone a2 behind master: shard=3 master=1_1685567943.999406_931.1 target=1_1685567930.890099_908.1 2023-05-31T21:20:36.760 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-31T21:20:36.761 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-31T21:20:36.761 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-05-31T21:20:36.819 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2023-05-31T21:20:36.820 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2023-05-31T21:20:36.872 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T21:20:36.868+0000 7f3b8be12780 20 RGW-SYNC:meta: read sync status 2023-05-31T21:20:36.873 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T21:20:36.869+0000 7f3b8be12780 20 RGW-SYNC:meta: read sync status 2023-05-31T21:20:36.893 INFO:teuthology.orchestra.run.smithi099.stdout:{ 2023-05-31T21:20:36.893 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_status": { 2023-05-31T21:20:36.893 INFO:teuthology.orchestra.run.smithi099.stdout: "info": { 2023-05-31T21:20:36.893 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "sync", 2023-05-31T21:20:36.893 INFO:teuthology.orchestra.run.smithi099.stdout: "num_shards": 4, 2023-05-31T21:20:36.894 INFO:teuthology.orchestra.run.smithi099.stdout: "period": "2e6123d2-d695-4350-9a01-61a455bc782b", 2023-05-31T21:20:36.894 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2023-05-31T21:20:36.894 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:20:36.894 INFO:teuthology.orchestra.run.smithi099.stdout: "markers": [ 2023-05-31T21:20:36.894 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T21:20:36.894 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 0, 2023-05-31T21:20:36.894 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T21:20:36.895 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2023-05-31T21:20:36.895 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1685566501.757183_260.1", 2023-05-31T21:20:36.895 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T21:20:36.895 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 1, 2023-05-31T21:20:36.895 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T21:20:36.895 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2023-05-31T20:55:01.757183Z", 2023-05-31T21:20:36.895 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2023-05-31T21:20:36.896 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T21:20:36.896 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:20:36.896 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T21:20:36.896 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 1, 2023-05-31T21:20:36.896 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T21:20:36.896 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2023-05-31T21:20:36.897 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1685566501.735717_251.1", 2023-05-31T21:20:36.897 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T21:20:36.897 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2023-05-31T21:20:36.897 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T21:20:36.897 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2023-05-31T20:55:01.735717Z", 2023-05-31T21:20:36.897 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2023-05-31T21:20:36.897 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T21:20:36.898 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:20:36.898 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T21:20:36.898 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 2, 2023-05-31T21:20:36.898 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T21:20:36.898 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2023-05-31T21:20:36.898 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1685562970.274572_93.1", 2023-05-31T21:20:36.898 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T21:20:36.899 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2023-05-31T21:20:36.899 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T21:20:36.899 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2023-05-31T19:56:10.274572Z", 2023-05-31T21:20:36.899 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2023-05-31T21:20:36.899 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T21:20:36.899 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:20:36.899 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T21:20:36.900 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 3, 2023-05-31T21:20:36.900 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T21:20:36.900 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2023-05-31T21:20:36.900 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1685567930.890099_908.1", 2023-05-31T21:20:36.900 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T21:20:36.900 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2023-05-31T21:20:36.900 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T21:20:36.901 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2023-05-31T21:18:50.890099Z", 2023-05-31T21:20:36.901 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2023-05-31T21:20:36.901 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T21:20:36.901 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T21:20:36.901 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2023-05-31T21:20:36.901 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:20:36.901 INFO:teuthology.orchestra.run.smithi099.stdout: "full_sync": { 2023-05-31T21:20:36.902 INFO:teuthology.orchestra.run.smithi099.stdout: "total": 1, 2023-05-31T21:20:36.902 INFO:teuthology.orchestra.run.smithi099.stdout: "complete": 1 2023-05-31T21:20:36.902 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:20:36.902 INFO:teuthology.orchestra.run.smithi099.stdout: "current_time": "2023-05-31T21:20:36Z" 2023-05-31T21:20:36.902 INFO:teuthology.orchestra.run.smithi099.stdout:} 2023-05-31T21:20:36.903 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "2e6123d2-d695-4350-9a01-61a455bc782b", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1685566501.757183_260.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-05-31T20:55:01.757183Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1685566501.735717_251.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-05-31T20:55:01.735717Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1685562970.274572_93.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-05-31T19:56:10.274572Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1685567930.890099_908.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-05-31T21:18:50.890099Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-05-31T21:20:36Z" } 2023-05-31T21:20:36.904 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1685566501.757183_260.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-05-31T20:55:01.757183Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1685566501.735717_251.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-05-31T20:55:01.735717Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1685562970.274572_93.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-05-31T19:56:10.274572Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1685567930.890099_908.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-05-31T21:18:50.890099Z', 'realm_epoch': 2}}] 2023-05-31T21:20:36.904 DEBUG:rgw_multi.tests:log_status={0: '1_1685566501.757183_260.1', 1: '1_1685566501.735717_251.1', 2: '1_1685562970.274572_93.1', 3: '1_1685567943.999406_931.1'} 2023-05-31T21:20:36.904 DEBUG:rgw_multi.tests:sync_status={0: '1_1685566501.757183_260.1', 1: '1_1685566501.735717_251.1', 2: '1_1685562970.274572_93.1', 3: '1_1685567930.890099_908.1'} 2023-05-31T21:20:36.905 WARNING:rgw_multi.tests:zone a2 behind master: shard=3 master=1_1685567943.999406_931.1 target=1_1685567930.890099_908.1 2023-05-31T21:20:41.906 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-31T21:20:41.907 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-31T21:20:41.907 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-05-31T21:20:41.965 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2023-05-31T21:20:41.966 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2023-05-31T21:20:42.009 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T21:20:42.006+0000 7efdd36fb780 20 RGW-SYNC:meta: read sync status 2023-05-31T21:20:42.010 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T21:20:42.007+0000 7efdd36fb780 20 RGW-SYNC:meta: read sync status 2023-05-31T21:20:42.027 INFO:teuthology.orchestra.run.smithi099.stdout:{ 2023-05-31T21:20:42.028 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_status": { 2023-05-31T21:20:42.029 INFO:teuthology.orchestra.run.smithi099.stdout: "info": { 2023-05-31T21:20:42.029 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "sync", 2023-05-31T21:20:42.029 INFO:teuthology.orchestra.run.smithi099.stdout: "num_shards": 4, 2023-05-31T21:20:42.030 INFO:teuthology.orchestra.run.smithi099.stdout: "period": "2e6123d2-d695-4350-9a01-61a455bc782b", 2023-05-31T21:20:42.030 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2023-05-31T21:20:42.030 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:20:42.031 INFO:teuthology.orchestra.run.smithi099.stdout: "markers": [ 2023-05-31T21:20:42.031 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T21:20:42.031 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 0, 2023-05-31T21:20:42.032 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T21:20:42.032 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2023-05-31T21:20:42.033 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1685566501.757183_260.1", 2023-05-31T21:20:42.033 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T21:20:42.033 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 1, 2023-05-31T21:20:42.034 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T21:20:42.034 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2023-05-31T20:55:01.757183Z", 2023-05-31T21:20:42.034 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2023-05-31T21:20:42.035 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T21:20:42.035 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:20:42.035 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T21:20:42.036 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 1, 2023-05-31T21:20:42.036 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T21:20:42.037 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2023-05-31T21:20:42.037 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1685566501.735717_251.1", 2023-05-31T21:20:42.037 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T21:20:42.038 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2023-05-31T21:20:42.038 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T21:20:42.038 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2023-05-31T20:55:01.735717Z", 2023-05-31T21:20:42.039 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2023-05-31T21:20:42.039 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T21:20:42.039 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:20:42.040 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T21:20:42.040 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 2, 2023-05-31T21:20:42.041 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T21:20:42.041 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2023-05-31T21:20:42.041 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1685562970.274572_93.1", 2023-05-31T21:20:42.042 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T21:20:42.042 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2023-05-31T21:20:42.042 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T21:20:42.043 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2023-05-31T19:56:10.274572Z", 2023-05-31T21:20:42.043 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2023-05-31T21:20:42.043 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T21:20:42.044 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:20:42.044 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T21:20:42.044 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 3, 2023-05-31T21:20:42.045 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T21:20:42.045 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2023-05-31T21:20:42.045 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1685567930.890099_908.1", 2023-05-31T21:20:42.046 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T21:20:42.046 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2023-05-31T21:20:42.047 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T21:20:42.047 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2023-05-31T21:18:50.890099Z", 2023-05-31T21:20:42.047 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2023-05-31T21:20:42.048 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T21:20:42.048 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T21:20:42.048 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2023-05-31T21:20:42.049 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:20:42.049 INFO:teuthology.orchestra.run.smithi099.stdout: "full_sync": { 2023-05-31T21:20:42.049 INFO:teuthology.orchestra.run.smithi099.stdout: "total": 1, 2023-05-31T21:20:42.050 INFO:teuthology.orchestra.run.smithi099.stdout: "complete": 1 2023-05-31T21:20:42.050 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:20:42.050 INFO:teuthology.orchestra.run.smithi099.stdout: "current_time": "2023-05-31T21:20:42Z" 2023-05-31T21:20:42.051 INFO:teuthology.orchestra.run.smithi099.stdout:} 2023-05-31T21:20:42.052 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "2e6123d2-d695-4350-9a01-61a455bc782b", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1685566501.757183_260.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-05-31T20:55:01.757183Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1685566501.735717_251.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-05-31T20:55:01.735717Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1685562970.274572_93.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-05-31T19:56:10.274572Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1685567930.890099_908.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-05-31T21:18:50.890099Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-05-31T21:20:42Z" } 2023-05-31T21:20:42.053 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1685566501.757183_260.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-05-31T20:55:01.757183Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1685566501.735717_251.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-05-31T20:55:01.735717Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1685562970.274572_93.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-05-31T19:56:10.274572Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1685567930.890099_908.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-05-31T21:18:50.890099Z', 'realm_epoch': 2}}] 2023-05-31T21:20:42.054 DEBUG:rgw_multi.tests:log_status={0: '1_1685566501.757183_260.1', 1: '1_1685566501.735717_251.1', 2: '1_1685562970.274572_93.1', 3: '1_1685567943.999406_931.1'} 2023-05-31T21:20:42.055 DEBUG:rgw_multi.tests:sync_status={0: '1_1685566501.757183_260.1', 1: '1_1685566501.735717_251.1', 2: '1_1685562970.274572_93.1', 3: '1_1685567930.890099_908.1'} 2023-05-31T21:20:42.056 WARNING:rgw_multi.tests:zone a2 behind master: shard=3 master=1_1685567943.999406_931.1 target=1_1685567930.890099_908.1 2023-05-31T21:20:47.057 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-31T21:20:47.058 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-31T21:20:47.058 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-05-31T21:20:47.116 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2023-05-31T21:20:47.117 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2023-05-31T21:20:47.151 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T21:20:47.148+0000 7ff922e58780 20 RGW-SYNC:meta: read sync status 2023-05-31T21:20:47.152 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T21:20:47.149+0000 7ff922e58780 20 RGW-SYNC:meta: read sync status 2023-05-31T21:20:47.167 INFO:teuthology.orchestra.run.smithi099.stdout:{ 2023-05-31T21:20:47.167 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_status": { 2023-05-31T21:20:47.167 INFO:teuthology.orchestra.run.smithi099.stdout: "info": { 2023-05-31T21:20:47.167 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "sync", 2023-05-31T21:20:47.168 INFO:teuthology.orchestra.run.smithi099.stdout: "num_shards": 4, 2023-05-31T21:20:47.168 INFO:teuthology.orchestra.run.smithi099.stdout: "period": "2e6123d2-d695-4350-9a01-61a455bc782b", 2023-05-31T21:20:47.168 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2023-05-31T21:20:47.168 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:20:47.168 INFO:teuthology.orchestra.run.smithi099.stdout: "markers": [ 2023-05-31T21:20:47.168 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T21:20:47.169 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 0, 2023-05-31T21:20:47.169 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T21:20:47.169 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2023-05-31T21:20:47.169 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1685566501.757183_260.1", 2023-05-31T21:20:47.169 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T21:20:47.170 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 1, 2023-05-31T21:20:47.170 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T21:20:47.170 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2023-05-31T20:55:01.757183Z", 2023-05-31T21:20:47.170 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2023-05-31T21:20:47.170 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T21:20:47.170 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:20:47.170 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T21:20:47.171 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 1, 2023-05-31T21:20:47.171 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T21:20:47.171 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2023-05-31T21:20:47.171 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1685566501.735717_251.1", 2023-05-31T21:20:47.171 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T21:20:47.171 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2023-05-31T21:20:47.171 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T21:20:47.172 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2023-05-31T20:55:01.735717Z", 2023-05-31T21:20:47.172 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2023-05-31T21:20:47.172 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T21:20:47.172 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:20:47.172 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T21:20:47.172 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 2, 2023-05-31T21:20:47.173 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T21:20:47.173 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2023-05-31T21:20:47.173 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1685562970.274572_93.1", 2023-05-31T21:20:47.173 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T21:20:47.173 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2023-05-31T21:20:47.173 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T21:20:47.173 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2023-05-31T19:56:10.274572Z", 2023-05-31T21:20:47.174 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2023-05-31T21:20:47.174 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T21:20:47.174 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:20:47.174 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T21:20:47.174 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 3, 2023-05-31T21:20:47.174 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T21:20:47.174 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2023-05-31T21:20:47.175 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1685567930.890099_908.1", 2023-05-31T21:20:47.175 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T21:20:47.175 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2023-05-31T21:20:47.175 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T21:20:47.175 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2023-05-31T21:18:50.890099Z", 2023-05-31T21:20:47.176 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2023-05-31T21:20:47.176 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T21:20:47.176 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T21:20:47.176 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2023-05-31T21:20:47.177 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:20:47.177 INFO:teuthology.orchestra.run.smithi099.stdout: "full_sync": { 2023-05-31T21:20:47.177 INFO:teuthology.orchestra.run.smithi099.stdout: "total": 1, 2023-05-31T21:20:47.177 INFO:teuthology.orchestra.run.smithi099.stdout: "complete": 1 2023-05-31T21:20:47.177 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:20:47.178 INFO:teuthology.orchestra.run.smithi099.stdout: "current_time": "2023-05-31T21:20:47Z" 2023-05-31T21:20:47.178 INFO:teuthology.orchestra.run.smithi099.stdout:} 2023-05-31T21:20:47.179 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "2e6123d2-d695-4350-9a01-61a455bc782b", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1685566501.757183_260.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-05-31T20:55:01.757183Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1685566501.735717_251.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-05-31T20:55:01.735717Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1685562970.274572_93.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-05-31T19:56:10.274572Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1685567930.890099_908.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-05-31T21:18:50.890099Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-05-31T21:20:47Z" } 2023-05-31T21:20:47.179 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1685566501.757183_260.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-05-31T20:55:01.757183Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1685566501.735717_251.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-05-31T20:55:01.735717Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1685562970.274572_93.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-05-31T19:56:10.274572Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1685567930.890099_908.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-05-31T21:18:50.890099Z', 'realm_epoch': 2}}] 2023-05-31T21:20:47.180 DEBUG:rgw_multi.tests:log_status={0: '1_1685566501.757183_260.1', 1: '1_1685566501.735717_251.1', 2: '1_1685562970.274572_93.1', 3: '1_1685567943.999406_931.1'} 2023-05-31T21:20:47.180 DEBUG:rgw_multi.tests:sync_status={0: '1_1685566501.757183_260.1', 1: '1_1685566501.735717_251.1', 2: '1_1685562970.274572_93.1', 3: '1_1685567930.890099_908.1'} 2023-05-31T21:20:47.181 WARNING:rgw_multi.tests:zone a2 behind master: shard=3 master=1_1685567943.999406_931.1 target=1_1685567930.890099_908.1 2023-05-31T21:20:52.183 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-31T21:20:52.183 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-31T21:20:52.184 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-05-31T21:20:52.241 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2023-05-31T21:20:52.241 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2023-05-31T21:20:52.291 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T21:20:52.287+0000 7f9ce99de780 20 RGW-SYNC:meta: read sync status 2023-05-31T21:20:52.292 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T21:20:52.288+0000 7f9ce99de780 20 RGW-SYNC:meta: read sync status 2023-05-31T21:20:52.312 INFO:teuthology.orchestra.run.smithi099.stdout:{ 2023-05-31T21:20:52.313 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_status": { 2023-05-31T21:20:52.313 INFO:teuthology.orchestra.run.smithi099.stdout: "info": { 2023-05-31T21:20:52.313 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "sync", 2023-05-31T21:20:52.313 INFO:teuthology.orchestra.run.smithi099.stdout: "num_shards": 4, 2023-05-31T21:20:52.313 INFO:teuthology.orchestra.run.smithi099.stdout: "period": "2e6123d2-d695-4350-9a01-61a455bc782b", 2023-05-31T21:20:52.314 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2023-05-31T21:20:52.314 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:20:52.314 INFO:teuthology.orchestra.run.smithi099.stdout: "markers": [ 2023-05-31T21:20:52.314 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T21:20:52.314 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 0, 2023-05-31T21:20:52.314 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T21:20:52.315 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2023-05-31T21:20:52.315 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1685566501.757183_260.1", 2023-05-31T21:20:52.315 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T21:20:52.315 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 1, 2023-05-31T21:20:52.316 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T21:20:52.316 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2023-05-31T20:55:01.757183Z", 2023-05-31T21:20:52.316 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2023-05-31T21:20:52.316 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T21:20:52.316 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:20:52.316 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T21:20:52.317 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 1, 2023-05-31T21:20:52.317 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T21:20:52.317 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2023-05-31T21:20:52.317 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1685566501.735717_251.1", 2023-05-31T21:20:52.317 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T21:20:52.317 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2023-05-31T21:20:52.317 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T21:20:52.318 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2023-05-31T20:55:01.735717Z", 2023-05-31T21:20:52.318 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2023-05-31T21:20:52.318 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T21:20:52.318 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:20:52.318 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T21:20:52.318 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 2, 2023-05-31T21:20:52.318 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T21:20:52.319 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2023-05-31T21:20:52.319 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1685562970.274572_93.1", 2023-05-31T21:20:52.319 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T21:20:52.319 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2023-05-31T21:20:52.319 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T21:20:52.319 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2023-05-31T19:56:10.274572Z", 2023-05-31T21:20:52.320 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2023-05-31T21:20:52.320 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T21:20:52.320 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:20:52.320 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T21:20:52.320 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 3, 2023-05-31T21:20:52.320 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T21:20:52.320 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2023-05-31T21:20:52.321 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1685567930.890099_908.1", 2023-05-31T21:20:52.321 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T21:20:52.321 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2023-05-31T21:20:52.321 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T21:20:52.321 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2023-05-31T21:18:50.890099Z", 2023-05-31T21:20:52.321 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2023-05-31T21:20:52.321 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T21:20:52.322 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T21:20:52.322 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2023-05-31T21:20:52.322 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:20:52.322 INFO:teuthology.orchestra.run.smithi099.stdout: "full_sync": { 2023-05-31T21:20:52.322 INFO:teuthology.orchestra.run.smithi099.stdout: "total": 1, 2023-05-31T21:20:52.322 INFO:teuthology.orchestra.run.smithi099.stdout: "complete": 1 2023-05-31T21:20:52.322 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:20:52.323 INFO:teuthology.orchestra.run.smithi099.stdout: "current_time": "2023-05-31T21:20:52Z" 2023-05-31T21:20:52.323 INFO:teuthology.orchestra.run.smithi099.stdout:} 2023-05-31T21:20:52.324 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "2e6123d2-d695-4350-9a01-61a455bc782b", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1685566501.757183_260.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-05-31T20:55:01.757183Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1685566501.735717_251.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-05-31T20:55:01.735717Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1685562970.274572_93.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-05-31T19:56:10.274572Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1685567930.890099_908.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-05-31T21:18:50.890099Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-05-31T21:20:52Z" } 2023-05-31T21:20:52.325 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1685566501.757183_260.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-05-31T20:55:01.757183Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1685566501.735717_251.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-05-31T20:55:01.735717Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1685562970.274572_93.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-05-31T19:56:10.274572Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1685567930.890099_908.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-05-31T21:18:50.890099Z', 'realm_epoch': 2}}] 2023-05-31T21:20:52.325 DEBUG:rgw_multi.tests:log_status={0: '1_1685566501.757183_260.1', 1: '1_1685566501.735717_251.1', 2: '1_1685562970.274572_93.1', 3: '1_1685567943.999406_931.1'} 2023-05-31T21:20:52.326 DEBUG:rgw_multi.tests:sync_status={0: '1_1685566501.757183_260.1', 1: '1_1685566501.735717_251.1', 2: '1_1685562970.274572_93.1', 3: '1_1685567930.890099_908.1'} 2023-05-31T21:20:52.327 WARNING:rgw_multi.tests:zone a2 behind master: shard=3 master=1_1685567943.999406_931.1 target=1_1685567930.890099_908.1 2023-05-31T21:20:57.328 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-31T21:20:57.328 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-31T21:20:57.329 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-05-31T21:20:57.386 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2023-05-31T21:20:57.386 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2023-05-31T21:20:57.432 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T21:20:57.427+0000 7f561228b780 20 RGW-SYNC:meta: read sync status 2023-05-31T21:20:57.432 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T21:20:57.428+0000 7f561228b780 20 RGW-SYNC:meta: read sync status 2023-05-31T21:20:57.447 INFO:teuthology.orchestra.run.smithi099.stdout:{ 2023-05-31T21:20:57.447 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_status": { 2023-05-31T21:20:57.447 INFO:teuthology.orchestra.run.smithi099.stdout: "info": { 2023-05-31T21:20:57.448 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "sync", 2023-05-31T21:20:57.448 INFO:teuthology.orchestra.run.smithi099.stdout: "num_shards": 4, 2023-05-31T21:20:57.448 INFO:teuthology.orchestra.run.smithi099.stdout: "period": "2e6123d2-d695-4350-9a01-61a455bc782b", 2023-05-31T21:20:57.448 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2023-05-31T21:20:57.448 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:20:57.448 INFO:teuthology.orchestra.run.smithi099.stdout: "markers": [ 2023-05-31T21:20:57.449 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T21:20:57.449 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 0, 2023-05-31T21:20:57.449 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T21:20:57.449 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2023-05-31T21:20:57.449 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1685566501.757183_260.1", 2023-05-31T21:20:57.449 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T21:20:57.450 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 1, 2023-05-31T21:20:57.450 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T21:20:57.450 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2023-05-31T20:55:01.757183Z", 2023-05-31T21:20:57.450 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2023-05-31T21:20:57.450 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T21:20:57.450 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:20:57.450 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T21:20:57.451 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 1, 2023-05-31T21:20:57.451 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T21:20:57.451 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2023-05-31T21:20:57.451 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1685566501.735717_251.1", 2023-05-31T21:20:57.451 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T21:20:57.451 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2023-05-31T21:20:57.451 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T21:20:57.452 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2023-05-31T20:55:01.735717Z", 2023-05-31T21:20:57.452 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2023-05-31T21:20:57.452 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T21:20:57.452 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:20:57.452 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T21:20:57.452 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 2, 2023-05-31T21:20:57.452 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T21:20:57.453 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2023-05-31T21:20:57.453 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1685562970.274572_93.1", 2023-05-31T21:20:57.453 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T21:20:57.453 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2023-05-31T21:20:57.453 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T21:20:57.453 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2023-05-31T19:56:10.274572Z", 2023-05-31T21:20:57.453 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2023-05-31T21:20:57.454 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T21:20:57.454 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:20:57.454 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T21:20:57.454 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 3, 2023-05-31T21:20:57.454 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T21:20:57.454 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2023-05-31T21:20:57.455 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1685567930.890099_908.1", 2023-05-31T21:20:57.455 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T21:20:57.455 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2023-05-31T21:20:57.455 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T21:20:57.455 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2023-05-31T21:18:50.890099Z", 2023-05-31T21:20:57.455 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2023-05-31T21:20:57.455 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T21:20:57.456 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T21:20:57.456 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2023-05-31T21:20:57.456 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:20:57.456 INFO:teuthology.orchestra.run.smithi099.stdout: "full_sync": { 2023-05-31T21:20:57.456 INFO:teuthology.orchestra.run.smithi099.stdout: "total": 1, 2023-05-31T21:20:57.456 INFO:teuthology.orchestra.run.smithi099.stdout: "complete": 1 2023-05-31T21:20:57.456 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:20:57.457 INFO:teuthology.orchestra.run.smithi099.stdout: "current_time": "2023-05-31T21:20:57Z" 2023-05-31T21:20:57.457 INFO:teuthology.orchestra.run.smithi099.stdout:} 2023-05-31T21:20:57.458 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "2e6123d2-d695-4350-9a01-61a455bc782b", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1685566501.757183_260.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-05-31T20:55:01.757183Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1685566501.735717_251.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-05-31T20:55:01.735717Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1685562970.274572_93.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-05-31T19:56:10.274572Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1685567930.890099_908.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-05-31T21:18:50.890099Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-05-31T21:20:57Z" } 2023-05-31T21:20:57.458 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1685566501.757183_260.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-05-31T20:55:01.757183Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1685566501.735717_251.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-05-31T20:55:01.735717Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1685562970.274572_93.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-05-31T19:56:10.274572Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1685567930.890099_908.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-05-31T21:18:50.890099Z', 'realm_epoch': 2}}] 2023-05-31T21:20:57.459 DEBUG:rgw_multi.tests:log_status={0: '1_1685566501.757183_260.1', 1: '1_1685566501.735717_251.1', 2: '1_1685562970.274572_93.1', 3: '1_1685567943.999406_931.1'} 2023-05-31T21:20:57.459 DEBUG:rgw_multi.tests:sync_status={0: '1_1685566501.757183_260.1', 1: '1_1685566501.735717_251.1', 2: '1_1685562970.274572_93.1', 3: '1_1685567930.890099_908.1'} 2023-05-31T21:20:57.459 WARNING:rgw_multi.tests:zone a2 behind master: shard=3 master=1_1685567943.999406_931.1 target=1_1685567930.890099_908.1 2023-05-31T21:21:02.460 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-31T21:21:02.461 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-31T21:21:02.461 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-05-31T21:21:02.519 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2023-05-31T21:21:02.520 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2023-05-31T21:21:02.573 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T21:21:02.569+0000 7fa4a57ad780 20 RGW-SYNC:meta: read sync status 2023-05-31T21:21:02.574 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T21:21:02.570+0000 7fa4a57ad780 20 RGW-SYNC:meta: read sync status 2023-05-31T21:21:02.592 INFO:teuthology.orchestra.run.smithi099.stdout:{ 2023-05-31T21:21:02.593 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_status": { 2023-05-31T21:21:02.593 INFO:teuthology.orchestra.run.smithi099.stdout: "info": { 2023-05-31T21:21:02.593 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "sync", 2023-05-31T21:21:02.593 INFO:teuthology.orchestra.run.smithi099.stdout: "num_shards": 4, 2023-05-31T21:21:02.594 INFO:teuthology.orchestra.run.smithi099.stdout: "period": "2e6123d2-d695-4350-9a01-61a455bc782b", 2023-05-31T21:21:02.594 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2023-05-31T21:21:02.594 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:21:02.594 INFO:teuthology.orchestra.run.smithi099.stdout: "markers": [ 2023-05-31T21:21:02.594 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T21:21:02.594 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 0, 2023-05-31T21:21:02.594 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T21:21:02.595 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2023-05-31T21:21:02.595 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1685566501.757183_260.1", 2023-05-31T21:21:02.595 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T21:21:02.595 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 1, 2023-05-31T21:21:02.595 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T21:21:02.596 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2023-05-31T20:55:01.757183Z", 2023-05-31T21:21:02.596 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2023-05-31T21:21:02.596 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T21:21:02.596 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:21:02.596 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T21:21:02.596 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 1, 2023-05-31T21:21:02.597 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T21:21:02.597 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2023-05-31T21:21:02.597 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1685566501.735717_251.1", 2023-05-31T21:21:02.597 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T21:21:02.597 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2023-05-31T21:21:02.597 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T21:21:02.597 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2023-05-31T20:55:01.735717Z", 2023-05-31T21:21:02.598 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2023-05-31T21:21:02.598 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T21:21:02.598 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:21:02.598 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T21:21:02.598 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 2, 2023-05-31T21:21:02.598 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T21:21:02.599 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2023-05-31T21:21:02.599 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1685562970.274572_93.1", 2023-05-31T21:21:02.599 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T21:21:02.600 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2023-05-31T21:21:02.600 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T21:21:02.600 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2023-05-31T19:56:10.274572Z", 2023-05-31T21:21:02.600 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2023-05-31T21:21:02.600 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T21:21:02.600 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:21:02.600 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T21:21:02.601 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 3, 2023-05-31T21:21:02.601 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T21:21:02.601 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2023-05-31T21:21:02.601 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1685567930.890099_908.1", 2023-05-31T21:21:02.601 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T21:21:02.601 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2023-05-31T21:21:02.601 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T21:21:02.602 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2023-05-31T21:18:50.890099Z", 2023-05-31T21:21:02.602 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2023-05-31T21:21:02.602 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T21:21:02.602 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T21:21:02.602 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2023-05-31T21:21:02.602 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:21:02.603 INFO:teuthology.orchestra.run.smithi099.stdout: "full_sync": { 2023-05-31T21:21:02.603 INFO:teuthology.orchestra.run.smithi099.stdout: "total": 1, 2023-05-31T21:21:02.603 INFO:teuthology.orchestra.run.smithi099.stdout: "complete": 1 2023-05-31T21:21:02.603 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:21:02.603 INFO:teuthology.orchestra.run.smithi099.stdout: "current_time": "2023-05-31T21:21:02Z" 2023-05-31T21:21:02.603 INFO:teuthology.orchestra.run.smithi099.stdout:} 2023-05-31T21:21:02.604 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "2e6123d2-d695-4350-9a01-61a455bc782b", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1685566501.757183_260.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-05-31T20:55:01.757183Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1685566501.735717_251.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-05-31T20:55:01.735717Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1685562970.274572_93.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-05-31T19:56:10.274572Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1685567930.890099_908.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-05-31T21:18:50.890099Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-05-31T21:21:02Z" } 2023-05-31T21:21:02.605 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1685566501.757183_260.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-05-31T20:55:01.757183Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1685566501.735717_251.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-05-31T20:55:01.735717Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1685562970.274572_93.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-05-31T19:56:10.274572Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1685567930.890099_908.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-05-31T21:18:50.890099Z', 'realm_epoch': 2}}] 2023-05-31T21:21:02.605 DEBUG:rgw_multi.tests:log_status={0: '1_1685566501.757183_260.1', 1: '1_1685566501.735717_251.1', 2: '1_1685562970.274572_93.1', 3: '1_1685567943.999406_931.1'} 2023-05-31T21:21:02.606 DEBUG:rgw_multi.tests:sync_status={0: '1_1685566501.757183_260.1', 1: '1_1685566501.735717_251.1', 2: '1_1685562970.274572_93.1', 3: '1_1685567930.890099_908.1'} 2023-05-31T21:21:02.606 WARNING:rgw_multi.tests:zone a2 behind master: shard=3 master=1_1685567943.999406_931.1 target=1_1685567930.890099_908.1 2023-05-31T21:21:07.607 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-31T21:21:07.608 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-31T21:21:07.608 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-05-31T21:21:07.665 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2023-05-31T21:21:07.666 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2023-05-31T21:21:07.704 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T21:21:07.700+0000 7f548c7c3780 20 RGW-SYNC:meta: read sync status 2023-05-31T21:21:07.705 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T21:21:07.701+0000 7f548c7c3780 20 RGW-SYNC:meta: read sync status 2023-05-31T21:21:07.721 INFO:teuthology.orchestra.run.smithi099.stdout:{ 2023-05-31T21:21:07.722 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_status": { 2023-05-31T21:21:07.722 INFO:teuthology.orchestra.run.smithi099.stdout: "info": { 2023-05-31T21:21:07.722 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "sync", 2023-05-31T21:21:07.722 INFO:teuthology.orchestra.run.smithi099.stdout: "num_shards": 4, 2023-05-31T21:21:07.723 INFO:teuthology.orchestra.run.smithi099.stdout: "period": "2e6123d2-d695-4350-9a01-61a455bc782b", 2023-05-31T21:21:07.723 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2023-05-31T21:21:07.723 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:21:07.724 INFO:teuthology.orchestra.run.smithi099.stdout: "markers": [ 2023-05-31T21:21:07.724 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T21:21:07.724 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 0, 2023-05-31T21:21:07.725 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T21:21:07.725 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2023-05-31T21:21:07.725 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1685566501.757183_260.1", 2023-05-31T21:21:07.725 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T21:21:07.726 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 1, 2023-05-31T21:21:07.726 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T21:21:07.726 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2023-05-31T20:55:01.757183Z", 2023-05-31T21:21:07.726 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2023-05-31T21:21:07.727 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T21:21:07.727 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:21:07.727 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T21:21:07.727 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 1, 2023-05-31T21:21:07.728 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T21:21:07.728 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2023-05-31T21:21:07.728 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1685566501.735717_251.1", 2023-05-31T21:21:07.728 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T21:21:07.729 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2023-05-31T21:21:07.729 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T21:21:07.729 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2023-05-31T20:55:01.735717Z", 2023-05-31T21:21:07.730 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2023-05-31T21:21:07.730 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T21:21:07.730 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:21:07.730 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T21:21:07.731 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 2, 2023-05-31T21:21:07.731 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T21:21:07.734 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2023-05-31T21:21:07.734 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1685562970.274572_93.1", 2023-05-31T21:21:07.734 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T21:21:07.734 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2023-05-31T21:21:07.734 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T21:21:07.735 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2023-05-31T19:56:10.274572Z", 2023-05-31T21:21:07.735 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2023-05-31T21:21:07.735 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T21:21:07.735 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:21:07.735 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T21:21:07.735 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 3, 2023-05-31T21:21:07.736 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T21:21:07.736 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2023-05-31T21:21:07.736 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1685567930.890099_908.1", 2023-05-31T21:21:07.736 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T21:21:07.736 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2023-05-31T21:21:07.737 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T21:21:07.737 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2023-05-31T21:18:50.890099Z", 2023-05-31T21:21:07.737 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2023-05-31T21:21:07.737 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T21:21:07.737 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T21:21:07.738 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2023-05-31T21:21:07.738 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:21:07.738 INFO:teuthology.orchestra.run.smithi099.stdout: "full_sync": { 2023-05-31T21:21:07.738 INFO:teuthology.orchestra.run.smithi099.stdout: "total": 1, 2023-05-31T21:21:07.738 INFO:teuthology.orchestra.run.smithi099.stdout: "complete": 1 2023-05-31T21:21:07.738 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:21:07.738 INFO:teuthology.orchestra.run.smithi099.stdout: "current_time": "2023-05-31T21:21:07Z" 2023-05-31T21:21:07.739 INFO:teuthology.orchestra.run.smithi099.stdout:} 2023-05-31T21:21:07.740 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "2e6123d2-d695-4350-9a01-61a455bc782b", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1685566501.757183_260.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-05-31T20:55:01.757183Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1685566501.735717_251.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-05-31T20:55:01.735717Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1685562970.274572_93.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-05-31T19:56:10.274572Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1685567930.890099_908.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-05-31T21:18:50.890099Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-05-31T21:21:07Z" } 2023-05-31T21:21:07.740 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1685566501.757183_260.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-05-31T20:55:01.757183Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1685566501.735717_251.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-05-31T20:55:01.735717Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1685562970.274572_93.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-05-31T19:56:10.274572Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1685567930.890099_908.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-05-31T21:18:50.890099Z', 'realm_epoch': 2}}] 2023-05-31T21:21:07.741 DEBUG:rgw_multi.tests:log_status={0: '1_1685566501.757183_260.1', 1: '1_1685566501.735717_251.1', 2: '1_1685562970.274572_93.1', 3: '1_1685567943.999406_931.1'} 2023-05-31T21:21:07.742 DEBUG:rgw_multi.tests:sync_status={0: '1_1685566501.757183_260.1', 1: '1_1685566501.735717_251.1', 2: '1_1685562970.274572_93.1', 3: '1_1685567930.890099_908.1'} 2023-05-31T21:21:07.742 WARNING:rgw_multi.tests:zone a2 behind master: shard=3 master=1_1685567943.999406_931.1 target=1_1685567930.890099_908.1 2023-05-31T21:21:12.743 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-31T21:21:12.744 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-31T21:21:12.745 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-05-31T21:21:12.803 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2023-05-31T21:21:12.804 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2023-05-31T21:21:12.847 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T21:21:12.844+0000 7f9f1e5bd780 20 RGW-SYNC:meta: read sync status 2023-05-31T21:21:12.848 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T21:21:12.845+0000 7f9f1e5bd780 20 RGW-SYNC:meta: read sync status 2023-05-31T21:21:12.863 INFO:teuthology.orchestra.run.smithi099.stdout:{ 2023-05-31T21:21:12.863 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_status": { 2023-05-31T21:21:12.863 INFO:teuthology.orchestra.run.smithi099.stdout: "info": { 2023-05-31T21:21:12.864 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "sync", 2023-05-31T21:21:12.864 INFO:teuthology.orchestra.run.smithi099.stdout: "num_shards": 4, 2023-05-31T21:21:12.864 INFO:teuthology.orchestra.run.smithi099.stdout: "period": "2e6123d2-d695-4350-9a01-61a455bc782b", 2023-05-31T21:21:12.864 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2023-05-31T21:21:12.864 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:21:12.864 INFO:teuthology.orchestra.run.smithi099.stdout: "markers": [ 2023-05-31T21:21:12.865 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T21:21:12.865 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 0, 2023-05-31T21:21:12.865 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T21:21:12.865 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2023-05-31T21:21:12.865 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1685566501.757183_260.1", 2023-05-31T21:21:12.865 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T21:21:12.866 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 1, 2023-05-31T21:21:12.866 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T21:21:12.866 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2023-05-31T20:55:01.757183Z", 2023-05-31T21:21:12.866 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2023-05-31T21:21:12.866 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T21:21:12.866 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:21:12.866 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T21:21:12.867 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 1, 2023-05-31T21:21:12.869 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T21:21:12.869 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2023-05-31T21:21:12.870 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1685566501.735717_251.1", 2023-05-31T21:21:12.870 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T21:21:12.870 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2023-05-31T21:21:12.870 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T21:21:12.870 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2023-05-31T20:55:01.735717Z", 2023-05-31T21:21:12.870 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2023-05-31T21:21:12.871 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T21:21:12.871 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:21:12.871 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T21:21:12.871 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 2, 2023-05-31T21:21:12.871 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T21:21:12.871 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2023-05-31T21:21:12.872 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1685562970.274572_93.1", 2023-05-31T21:21:12.872 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T21:21:12.872 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2023-05-31T21:21:12.872 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T21:21:12.872 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2023-05-31T19:56:10.274572Z", 2023-05-31T21:21:12.872 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2023-05-31T21:21:12.872 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T21:21:12.873 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:21:12.873 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T21:21:12.873 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 3, 2023-05-31T21:21:12.873 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T21:21:12.873 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2023-05-31T21:21:12.873 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1685567930.890099_908.1", 2023-05-31T21:21:12.873 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T21:21:12.874 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2023-05-31T21:21:12.874 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T21:21:12.874 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2023-05-31T21:18:50.890099Z", 2023-05-31T21:21:12.874 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2023-05-31T21:21:12.874 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T21:21:12.874 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T21:21:12.874 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2023-05-31T21:21:12.875 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:21:12.875 INFO:teuthology.orchestra.run.smithi099.stdout: "full_sync": { 2023-05-31T21:21:12.875 INFO:teuthology.orchestra.run.smithi099.stdout: "total": 1, 2023-05-31T21:21:12.875 INFO:teuthology.orchestra.run.smithi099.stdout: "complete": 1 2023-05-31T21:21:12.875 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:21:12.875 INFO:teuthology.orchestra.run.smithi099.stdout: "current_time": "2023-05-31T21:21:12Z" 2023-05-31T21:21:12.875 INFO:teuthology.orchestra.run.smithi099.stdout:} 2023-05-31T21:21:12.877 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "2e6123d2-d695-4350-9a01-61a455bc782b", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1685566501.757183_260.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-05-31T20:55:01.757183Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1685566501.735717_251.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-05-31T20:55:01.735717Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1685562970.274572_93.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-05-31T19:56:10.274572Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1685567930.890099_908.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-05-31T21:18:50.890099Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-05-31T21:21:12Z" } 2023-05-31T21:21:12.878 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1685566501.757183_260.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-05-31T20:55:01.757183Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1685566501.735717_251.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-05-31T20:55:01.735717Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1685562970.274572_93.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-05-31T19:56:10.274572Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1685567930.890099_908.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-05-31T21:18:50.890099Z', 'realm_epoch': 2}}] 2023-05-31T21:21:12.878 DEBUG:rgw_multi.tests:log_status={0: '1_1685566501.757183_260.1', 1: '1_1685566501.735717_251.1', 2: '1_1685562970.274572_93.1', 3: '1_1685567943.999406_931.1'} 2023-05-31T21:21:12.878 DEBUG:rgw_multi.tests:sync_status={0: '1_1685566501.757183_260.1', 1: '1_1685566501.735717_251.1', 2: '1_1685562970.274572_93.1', 3: '1_1685567930.890099_908.1'} 2023-05-31T21:21:12.879 WARNING:rgw_multi.tests:zone a2 behind master: shard=3 master=1_1685567943.999406_931.1 target=1_1685567930.890099_908.1 2023-05-31T21:21:17.879 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-31T21:21:17.880 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-31T21:21:17.881 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-05-31T21:21:17.937 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2023-05-31T21:21:17.938 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2023-05-31T21:21:17.988 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T21:21:17.985+0000 7f0c0eda9780 20 RGW-SYNC:meta: read sync status 2023-05-31T21:21:17.990 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T21:21:17.986+0000 7f0c0eda9780 20 RGW-SYNC:meta: read sync status 2023-05-31T21:21:18.005 INFO:teuthology.orchestra.run.smithi099.stdout:{ 2023-05-31T21:21:18.006 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_status": { 2023-05-31T21:21:18.006 INFO:teuthology.orchestra.run.smithi099.stdout: "info": { 2023-05-31T21:21:18.006 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "sync", 2023-05-31T21:21:18.006 INFO:teuthology.orchestra.run.smithi099.stdout: "num_shards": 4, 2023-05-31T21:21:18.006 INFO:teuthology.orchestra.run.smithi099.stdout: "period": "2e6123d2-d695-4350-9a01-61a455bc782b", 2023-05-31T21:21:18.006 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2023-05-31T21:21:18.007 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:21:18.007 INFO:teuthology.orchestra.run.smithi099.stdout: "markers": [ 2023-05-31T21:21:18.007 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T21:21:18.007 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 0, 2023-05-31T21:21:18.007 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T21:21:18.007 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2023-05-31T21:21:18.008 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1685566501.757183_260.1", 2023-05-31T21:21:18.008 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T21:21:18.008 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 1, 2023-05-31T21:21:18.008 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T21:21:18.008 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2023-05-31T20:55:01.757183Z", 2023-05-31T21:21:18.009 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2023-05-31T21:21:18.009 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T21:21:18.009 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:21:18.009 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T21:21:18.009 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 1, 2023-05-31T21:21:18.009 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T21:21:18.009 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2023-05-31T21:21:18.010 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1685566501.735717_251.1", 2023-05-31T21:21:18.010 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T21:21:18.010 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2023-05-31T21:21:18.010 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T21:21:18.010 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2023-05-31T20:55:01.735717Z", 2023-05-31T21:21:18.010 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2023-05-31T21:21:18.011 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T21:21:18.011 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:21:18.011 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T21:21:18.011 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 2, 2023-05-31T21:21:18.011 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T21:21:18.011 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2023-05-31T21:21:18.012 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1685562970.274572_93.1", 2023-05-31T21:21:18.012 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T21:21:18.012 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2023-05-31T21:21:18.012 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T21:21:18.012 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2023-05-31T19:56:10.274572Z", 2023-05-31T21:21:18.012 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2023-05-31T21:21:18.012 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T21:21:18.013 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:21:18.013 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T21:21:18.013 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 3, 2023-05-31T21:21:18.013 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T21:21:18.013 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2023-05-31T21:21:18.014 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1685567930.890099_908.1", 2023-05-31T21:21:18.014 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T21:21:18.014 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2023-05-31T21:21:18.014 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T21:21:18.014 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2023-05-31T21:18:50.890099Z", 2023-05-31T21:21:18.015 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2023-05-31T21:21:18.015 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T21:21:18.015 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T21:21:18.015 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2023-05-31T21:21:18.015 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:21:18.015 INFO:teuthology.orchestra.run.smithi099.stdout: "full_sync": { 2023-05-31T21:21:18.016 INFO:teuthology.orchestra.run.smithi099.stdout: "total": 1, 2023-05-31T21:21:18.016 INFO:teuthology.orchestra.run.smithi099.stdout: "complete": 1 2023-05-31T21:21:18.016 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:21:18.016 INFO:teuthology.orchestra.run.smithi099.stdout: "current_time": "2023-05-31T21:21:17Z" 2023-05-31T21:21:18.016 INFO:teuthology.orchestra.run.smithi099.stdout:} 2023-05-31T21:21:18.017 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "2e6123d2-d695-4350-9a01-61a455bc782b", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1685566501.757183_260.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-05-31T20:55:01.757183Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1685566501.735717_251.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-05-31T20:55:01.735717Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1685562970.274572_93.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-05-31T19:56:10.274572Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1685567930.890099_908.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-05-31T21:18:50.890099Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-05-31T21:21:17Z" } 2023-05-31T21:21:18.018 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1685566501.757183_260.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-05-31T20:55:01.757183Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1685566501.735717_251.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-05-31T20:55:01.735717Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1685562970.274572_93.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-05-31T19:56:10.274572Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1685567930.890099_908.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-05-31T21:18:50.890099Z', 'realm_epoch': 2}}] 2023-05-31T21:21:18.018 DEBUG:rgw_multi.tests:log_status={0: '1_1685566501.757183_260.1', 1: '1_1685566501.735717_251.1', 2: '1_1685562970.274572_93.1', 3: '1_1685567943.999406_931.1'} 2023-05-31T21:21:18.019 DEBUG:rgw_multi.tests:sync_status={0: '1_1685566501.757183_260.1', 1: '1_1685566501.735717_251.1', 2: '1_1685562970.274572_93.1', 3: '1_1685567930.890099_908.1'} 2023-05-31T21:21:18.019 WARNING:rgw_multi.tests:zone a2 behind master: shard=3 master=1_1685567943.999406_931.1 target=1_1685567930.890099_908.1 2023-05-31T21:21:23.021 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-31T21:21:23.022 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-31T21:21:23.023 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-05-31T21:21:23.080 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2023-05-31T21:21:23.081 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2023-05-31T21:21:23.122 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T21:21:23.119+0000 7f866a340780 20 RGW-SYNC:meta: read sync status 2023-05-31T21:21:23.123 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T21:21:23.120+0000 7f866a340780 20 RGW-SYNC:meta: read sync status 2023-05-31T21:21:23.145 INFO:teuthology.orchestra.run.smithi099.stdout:{ 2023-05-31T21:21:23.145 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_status": { 2023-05-31T21:21:23.145 INFO:teuthology.orchestra.run.smithi099.stdout: "info": { 2023-05-31T21:21:23.146 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "sync", 2023-05-31T21:21:23.146 INFO:teuthology.orchestra.run.smithi099.stdout: "num_shards": 4, 2023-05-31T21:21:23.146 INFO:teuthology.orchestra.run.smithi099.stdout: "period": "2e6123d2-d695-4350-9a01-61a455bc782b", 2023-05-31T21:21:23.146 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2023-05-31T21:21:23.146 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:21:23.146 INFO:teuthology.orchestra.run.smithi099.stdout: "markers": [ 2023-05-31T21:21:23.146 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T21:21:23.147 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 0, 2023-05-31T21:21:23.147 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T21:21:23.147 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2023-05-31T21:21:23.147 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1685566501.757183_260.1", 2023-05-31T21:21:23.147 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T21:21:23.147 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 1, 2023-05-31T21:21:23.148 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T21:21:23.148 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2023-05-31T20:55:01.757183Z", 2023-05-31T21:21:23.148 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2023-05-31T21:21:23.148 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T21:21:23.148 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:21:23.148 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T21:21:23.148 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 1, 2023-05-31T21:21:23.149 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T21:21:23.149 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2023-05-31T21:21:23.149 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1685566501.735717_251.1", 2023-05-31T21:21:23.149 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T21:21:23.149 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2023-05-31T21:21:23.149 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T21:21:23.150 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2023-05-31T20:55:01.735717Z", 2023-05-31T21:21:23.150 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2023-05-31T21:21:23.150 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T21:21:23.150 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:21:23.150 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T21:21:23.150 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 2, 2023-05-31T21:21:23.150 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T21:21:23.151 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2023-05-31T21:21:23.151 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1685562970.274572_93.1", 2023-05-31T21:21:23.151 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T21:21:23.151 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2023-05-31T21:21:23.151 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T21:21:23.151 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2023-05-31T19:56:10.274572Z", 2023-05-31T21:21:23.151 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2023-05-31T21:21:23.152 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T21:21:23.152 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:21:23.152 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T21:21:23.152 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 3, 2023-05-31T21:21:23.152 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T21:21:23.152 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2023-05-31T21:21:23.153 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1685567943.999406_931.1", 2023-05-31T21:21:23.153 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T21:21:23.153 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2023-05-31T21:21:23.153 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T21:21:23.153 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2023-05-31T21:19:03.999406Z", 2023-05-31T21:21:23.153 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2023-05-31T21:21:23.153 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T21:21:23.154 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T21:21:23.154 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2023-05-31T21:21:23.154 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:21:23.154 INFO:teuthology.orchestra.run.smithi099.stdout: "full_sync": { 2023-05-31T21:21:23.154 INFO:teuthology.orchestra.run.smithi099.stdout: "total": 1, 2023-05-31T21:21:23.154 INFO:teuthology.orchestra.run.smithi099.stdout: "complete": 1 2023-05-31T21:21:23.154 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:21:23.155 INFO:teuthology.orchestra.run.smithi099.stdout: "current_time": "2023-05-31T21:21:23Z" 2023-05-31T21:21:23.155 INFO:teuthology.orchestra.run.smithi099.stdout:} 2023-05-31T21:21:23.156 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "2e6123d2-d695-4350-9a01-61a455bc782b", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1685566501.757183_260.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-05-31T20:55:01.757183Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1685566501.735717_251.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-05-31T20:55:01.735717Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1685562970.274572_93.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-05-31T19:56:10.274572Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1685567943.999406_931.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-05-31T21:19:03.999406Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-05-31T21:21:23Z" } 2023-05-31T21:21:23.156 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1685566501.757183_260.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-05-31T20:55:01.757183Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1685566501.735717_251.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-05-31T20:55:01.735717Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1685562970.274572_93.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-05-31T19:56:10.274572Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1685567943.999406_931.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-05-31T21:19:03.999406Z', 'realm_epoch': 2}}] 2023-05-31T21:21:23.157 DEBUG:rgw_multi.tests:log_status={0: '1_1685566501.757183_260.1', 1: '1_1685566501.735717_251.1', 2: '1_1685562970.274572_93.1', 3: '1_1685567943.999406_931.1'} 2023-05-31T21:21:23.157 DEBUG:rgw_multi.tests:sync_status={0: '1_1685566501.757183_260.1', 1: '1_1685566501.735717_251.1', 2: '1_1685562970.274572_93.1', 3: '1_1685567943.999406_931.1'} 2023-05-31T21:21:23.157 INFO:rgw_multi.tests:finish meta checkpoint for zone=a2 2023-05-31T21:21:23.158 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['datalog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-31T21:21:23.158 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'datalog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-31T21:21:23.158 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 datalog status --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-05-31T21:21:23.208 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2023-05-31T21:21:23.208 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2023-05-31T21:21:23.285 INFO:teuthology.orchestra.run.smithi099.stdout:[ 2023-05-31T21:21:23.285 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T21:21:23.285 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "00000000000000000000:00000000000000044815", 2023-05-31T21:21:23.286 INFO:teuthology.orchestra.run.smithi099.stdout: "last_update": "2023-05-31T21:21:05.078544Z" 2023-05-31T21:21:23.286 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:21:23.286 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T21:21:23.286 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "00000000000000000000:00000000000000031499", 2023-05-31T21:21:23.286 INFO:teuthology.orchestra.run.smithi099.stdout: "last_update": "2023-05-31T21:12:40.667485Z" 2023-05-31T21:21:23.286 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:21:23.286 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T21:21:23.287 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "00000000000000000000:00000000000000050831", 2023-05-31T21:21:23.287 INFO:teuthology.orchestra.run.smithi099.stdout: "last_update": "2023-05-31T21:19:03.876722Z" 2023-05-31T21:21:23.287 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:21:23.287 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T21:21:23.287 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "00000000000000000000:00000000000000043896", 2023-05-31T21:21:23.288 INFO:teuthology.orchestra.run.smithi099.stdout: "last_update": "2023-05-31T21:12:28.529956Z" 2023-05-31T21:21:23.288 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T21:21:23.288 INFO:teuthology.orchestra.run.smithi099.stdout:] 2023-05-31T21:21:23.289 DEBUG:rgw_multi.tests:data markers for zone=a1 markers={0: '00000000000000000000:00000000000000044815', 1: '00000000000000000000:00000000000000031499', 2: '00000000000000000000:00000000000000050831', 3: '00000000000000000000:00000000000000043896'} 2023-05-31T21:21:23.290 INFO:rgw_multi.tests:starting data checkpoint for target_zone=a2 source_zone=a1 2023-05-31T21:21:23.290 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-31T21:21:23.291 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-31T21:21:23.291 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-05-31T21:21:23.343 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2023-05-31T21:21:23.344 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2023-05-31T21:21:23.393 INFO:teuthology.orchestra.run.smithi099.stdout:{ 2023-05-31T21:21:23.394 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_status": { 2023-05-31T21:21:23.394 INFO:teuthology.orchestra.run.smithi099.stdout: "info": { 2023-05-31T21:21:23.394 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "sync", 2023-05-31T21:21:23.395 INFO:teuthology.orchestra.run.smithi099.stdout: "num_shards": 4, 2023-05-31T21:21:23.395 INFO:teuthology.orchestra.run.smithi099.stdout: "instance_id": 452136274422638824 2023-05-31T21:21:23.395 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:21:23.395 INFO:teuthology.orchestra.run.smithi099.stdout: "markers": [ 2023-05-31T21:21:23.395 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T21:21:23.395 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 0, 2023-05-31T21:21:23.395 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T21:21:23.396 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "incremental-sync", 2023-05-31T21:21:23.396 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "00000000000000000000:00000000000000044674", 2023-05-31T21:21:23.396 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T21:21:23.396 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2023-05-31T21:21:23.397 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T21:21:23.397 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2023-05-31T21:12:28.603419Z" 2023-05-31T21:21:23.397 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T21:21:23.397 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:21:23.398 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T21:21:23.398 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 1, 2023-05-31T21:21:23.398 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T21:21:23.399 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "incremental-sync", 2023-05-31T21:21:23.399 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "00000000000000000000:00000000000000031499", 2023-05-31T21:21:23.399 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T21:21:23.400 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2023-05-31T21:21:23.400 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T21:21:23.400 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2023-05-31T21:12:40.667485Z" 2023-05-31T21:21:23.401 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T21:21:23.401 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:21:23.401 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T21:21:23.402 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 2, 2023-05-31T21:21:23.402 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T21:21:23.402 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "incremental-sync", 2023-05-31T21:21:23.403 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "00000000000000000000:00000000000000050831", 2023-05-31T21:21:23.403 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T21:21:23.404 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 55, 2023-05-31T21:21:23.404 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T21:21:23.405 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2023-05-31T21:19:03.876722Z" 2023-05-31T21:21:23.405 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T21:21:23.406 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:21:23.406 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T21:21:23.406 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 3, 2023-05-31T21:21:23.407 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T21:21:23.407 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "incremental-sync", 2023-05-31T21:21:23.407 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "00000000000000000000:00000000000000043896", 2023-05-31T21:21:23.408 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T21:21:23.408 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 9, 2023-05-31T21:21:23.408 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T21:21:23.409 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2023-05-31T21:12:28.529956Z" 2023-05-31T21:21:23.409 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T21:21:23.409 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T21:21:23.409 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2023-05-31T21:21:23.410 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:21:23.410 INFO:teuthology.orchestra.run.smithi099.stdout: "full_sync": { 2023-05-31T21:21:23.410 INFO:teuthology.orchestra.run.smithi099.stdout: "total": 64, 2023-05-31T21:21:23.411 INFO:teuthology.orchestra.run.smithi099.stdout: "complete": 64 2023-05-31T21:21:23.411 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:21:23.411 INFO:teuthology.orchestra.run.smithi099.stdout: "current_time": "2023-05-31T21:21:23Z" 2023-05-31T21:21:23.412 INFO:teuthology.orchestra.run.smithi099.stdout:} 2023-05-31T21:21:23.413 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 452136274422638824 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000044674", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-05-31T21:12:28.603419Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000031499", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-05-31T21:12:40.667485Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000050831", "next_step_marker": "", "total_entries": 55, "pos": 0, "timestamp": "2023-05-31T21:19:03.876722Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000043896", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2023-05-31T21:12:28.529956Z" } } ] }, "full_sync": { "total": 64, "complete": 64 }, "current_time": "2023-05-31T21:21:23Z" } 2023-05-31T21:21:23.417 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000044674', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-05-31T21:12:28.603419Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000031499', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-05-31T21:12:40.667485Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000050831', 'next_step_marker': '', 'total_entries': 55, 'pos': 0, 'timestamp': '2023-05-31T21:19:03.876722Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000043896', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2023-05-31T21:12:28.529956Z'}}] 2023-05-31T21:21:23.417 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000044815', 1: '00000000000000000000:00000000000000031499', 2: '00000000000000000000:00000000000000050831', 3: '00000000000000000000:00000000000000043896'} 2023-05-31T21:21:23.418 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000044674', 1: '00000000000000000000:00000000000000031499', 2: '00000000000000000000:00000000000000050831', 3: '00000000000000000000:00000000000000043896'} 2023-05-31T21:21:23.418 WARNING:rgw_multi.tests:data of zone a2 behind zone a1: shard=0 master=00000000000000000000:00000000000000044815 target=00000000000000000000:00000000000000044674 2023-05-31T21:21:28.419 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-31T21:21:28.421 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-31T21:21:28.422 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-05-31T21:21:28.482 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2023-05-31T21:21:28.482 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2023-05-31T21:21:28.546 INFO:teuthology.orchestra.run.smithi099.stdout:{ 2023-05-31T21:21:28.547 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_status": { 2023-05-31T21:21:28.547 INFO:teuthology.orchestra.run.smithi099.stdout: "info": { 2023-05-31T21:21:28.547 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "sync", 2023-05-31T21:21:28.547 INFO:teuthology.orchestra.run.smithi099.stdout: "num_shards": 4, 2023-05-31T21:21:28.548 INFO:teuthology.orchestra.run.smithi099.stdout: "instance_id": 452136274422638824 2023-05-31T21:21:28.548 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:21:28.548 INFO:teuthology.orchestra.run.smithi099.stdout: "markers": [ 2023-05-31T21:21:28.548 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T21:21:28.548 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 0, 2023-05-31T21:21:28.548 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T21:21:28.548 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "incremental-sync", 2023-05-31T21:21:28.549 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "00000000000000000000:00000000000000044815", 2023-05-31T21:21:28.549 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T21:21:28.549 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2023-05-31T21:21:28.549 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T21:21:28.549 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2023-05-31T21:21:05.078544Z" 2023-05-31T21:21:28.549 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T21:21:28.549 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:21:28.550 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T21:21:28.550 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 1, 2023-05-31T21:21:28.550 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T21:21:28.550 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "incremental-sync", 2023-05-31T21:21:28.550 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "00000000000000000000:00000000000000031499", 2023-05-31T21:21:28.550 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T21:21:28.551 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2023-05-31T21:21:28.551 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T21:21:28.551 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2023-05-31T21:12:40.667485Z" 2023-05-31T21:21:28.551 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T21:21:28.551 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:21:28.551 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T21:21:28.551 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 2, 2023-05-31T21:21:28.552 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T21:21:28.552 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "incremental-sync", 2023-05-31T21:21:28.552 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "00000000000000000000:00000000000000050831", 2023-05-31T21:21:28.552 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T21:21:28.552 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 55, 2023-05-31T21:21:28.552 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T21:21:28.552 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2023-05-31T21:19:03.876722Z" 2023-05-31T21:21:28.553 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T21:21:28.553 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:21:28.553 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T21:21:28.553 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 3, 2023-05-31T21:21:28.553 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T21:21:28.553 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "incremental-sync", 2023-05-31T21:21:28.553 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "00000000000000000000:00000000000000043896", 2023-05-31T21:21:28.554 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T21:21:28.554 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 9, 2023-05-31T21:21:28.554 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T21:21:28.554 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2023-05-31T21:12:28.529956Z" 2023-05-31T21:21:28.554 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T21:21:28.554 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T21:21:28.555 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2023-05-31T21:21:28.555 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:21:28.555 INFO:teuthology.orchestra.run.smithi099.stdout: "full_sync": { 2023-05-31T21:21:28.555 INFO:teuthology.orchestra.run.smithi099.stdout: "total": 64, 2023-05-31T21:21:28.555 INFO:teuthology.orchestra.run.smithi099.stdout: "complete": 64 2023-05-31T21:21:28.555 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:21:28.555 INFO:teuthology.orchestra.run.smithi099.stdout: "current_time": "2023-05-31T21:21:28Z" 2023-05-31T21:21:28.556 INFO:teuthology.orchestra.run.smithi099.stdout:} 2023-05-31T21:21:28.557 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 452136274422638824 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000044815", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-05-31T21:21:05.078544Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000031499", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-05-31T21:12:40.667485Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000050831", "next_step_marker": "", "total_entries": 55, "pos": 0, "timestamp": "2023-05-31T21:19:03.876722Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000043896", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2023-05-31T21:12:28.529956Z" } } ] }, "full_sync": { "total": 64, "complete": 64 }, "current_time": "2023-05-31T21:21:28Z" } 2023-05-31T21:21:28.557 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000044815', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-05-31T21:21:05.078544Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000031499', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-05-31T21:12:40.667485Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000050831', 'next_step_marker': '', 'total_entries': 55, 'pos': 0, 'timestamp': '2023-05-31T21:19:03.876722Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000043896', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2023-05-31T21:12:28.529956Z'}}] 2023-05-31T21:21:28.558 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000044815', 1: '00000000000000000000:00000000000000031499', 2: '00000000000000000000:00000000000000050831', 3: '00000000000000000000:00000000000000043896'} 2023-05-31T21:21:28.558 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000044815', 1: '00000000000000000000:00000000000000031499', 2: '00000000000000000000:00000000000000050831', 3: '00000000000000000000:00000000000000043896'} 2023-05-31T21:21:28.558 INFO:rgw_multi.tests:finished data checkpoint for target_zone=a2 source_zone=a1 2023-05-31T21:21:28.559 DEBUG:boto:path=/pklxkd-66/ 2023-05-31T21:21:28.559 DEBUG:boto:auth_path=/pklxkd-66/ 2023-05-31T21:21:28.559 DEBUG:boto:Method: HEAD 2023-05-31T21:21:28.560 DEBUG:boto:Path: /pklxkd-66/ 2023-05-31T21:21:28.560 DEBUG:boto:Data: 2023-05-31T21:21:28.560 DEBUG:boto:Headers: {} 2023-05-31T21:21:28.561 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T21:21:28.561 DEBUG:boto:Port: 8001 2023-05-31T21:21:28.561 DEBUG:boto:Params: {} 2023-05-31T21:21:28.563 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8001} 2023-05-31T21:21:28.564 DEBUG:boto:Token: None 2023-05-31T21:21:28.565 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 21:21:28 GMT /pklxkd-66/ 2023-05-31T21:21:28.565 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:9VP0xAfAnl9z2wDrZhu9kxZdeAg= 2023-05-31T21:21:28.566 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 21:21:28 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:9VP0xAfAnl9z2wDrZhu9kxZdeAg=', 'Content-Length': '0'} 2023-05-31T21:21:28.570 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '6'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000eb02cb4cdcdaee74-006477ba58-22af-a2'), ('Content-Length', '0'), ('Date', 'Wed, 31 May 2023 21:21:28 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:21:28.570 DEBUG:boto:path=/pklxkd-66/obj1 2023-05-31T21:21:28.571 DEBUG:boto:auth_path=/pklxkd-66/obj1 2023-05-31T21:21:28.571 DEBUG:boto:Method: HEAD 2023-05-31T21:21:28.572 DEBUG:boto:Path: /pklxkd-66/obj1 2023-05-31T21:21:28.572 DEBUG:boto:Data: 2023-05-31T21:21:28.573 DEBUG:boto:Headers: {} 2023-05-31T21:21:28.573 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T21:21:28.573 DEBUG:boto:Port: 8001 2023-05-31T21:21:28.574 DEBUG:boto:Params: {} 2023-05-31T21:21:28.574 DEBUG:boto:Token: None 2023-05-31T21:21:28.575 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 21:21:28 GMT /pklxkd-66/obj1 2023-05-31T21:21:28.575 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:KT0Ii9lMOfdUolwCCfgo1ZvPeHg= 2023-05-31T21:21:28.576 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 21:21:28 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:KT0Ii9lMOfdUolwCCfgo1ZvPeHg=', 'Content-Length': '0'} 2023-05-31T21:21:28.577 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 21:19:03 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000b5d54099afb64238-006477ba58-22af-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 21:21:28 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:21:28.578 DEBUG:boto:path=/pklxkd-66/obj1 2023-05-31T21:21:28.579 DEBUG:boto:auth_path=/pklxkd-66/obj1 2023-05-31T21:21:28.580 DEBUG:boto:Method: GET 2023-05-31T21:21:28.580 DEBUG:boto:Path: /pklxkd-66/obj1 2023-05-31T21:21:28.581 DEBUG:boto:Data: 2023-05-31T21:21:28.582 DEBUG:boto:Headers: {} 2023-05-31T21:21:28.582 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T21:21:28.582 DEBUG:boto:Port: 8001 2023-05-31T21:21:28.583 DEBUG:boto:Params: {} 2023-05-31T21:21:28.583 DEBUG:boto:Token: None 2023-05-31T21:21:28.583 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 21:21:28 GMT /pklxkd-66/obj1 2023-05-31T21:21:28.584 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:wyJwMlGn3JLsaz5mNmtqrlkx5k4= 2023-05-31T21:21:28.584 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 21:21:28 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:wyJwMlGn3JLsaz5mNmtqrlkx5k4=', 'Content-Length': '0'} 2023-05-31T21:21:28.586 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 21:19:03 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000066a569195099ddc1-006477ba58-22af-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 21:21:28 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:21:28.587 DEBUG:boto:path=/pklxkd-67/ 2023-05-31T21:21:28.588 DEBUG:boto:auth_path=/pklxkd-67/ 2023-05-31T21:21:28.588 DEBUG:boto:Method: HEAD 2023-05-31T21:21:28.589 DEBUG:boto:Path: /pklxkd-67/ 2023-05-31T21:21:28.589 DEBUG:boto:Data: 2023-05-31T21:21:28.589 DEBUG:boto:Headers: {} 2023-05-31T21:21:28.590 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T21:21:28.590 DEBUG:boto:Port: 8000 2023-05-31T21:21:28.590 DEBUG:boto:Params: {} 2023-05-31T21:21:28.591 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2023-05-31T21:21:28.591 DEBUG:boto:Token: None 2023-05-31T21:21:28.592 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 21:21:28 GMT /pklxkd-67/ 2023-05-31T21:21:28.592 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:qBf/kqim0Evwt7dIV1GkBoo5iD8= 2023-05-31T21:21:28.592 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 21:21:28 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:qBf/kqim0Evwt7dIV1GkBoo5iD8=', 'Content-Length': '0'} 2023-05-31T21:21:28.595 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '6'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000009526a9d9e61a0b8c-006477ba58-22ab-a1'), ('Content-Length', '0'), ('Date', 'Wed, 31 May 2023 21:21:28 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:21:28.596 DEBUG:boto:path=/pklxkd-67/obj2 2023-05-31T21:21:28.596 DEBUG:boto:auth_path=/pklxkd-67/obj2 2023-05-31T21:21:28.596 DEBUG:boto:Method: HEAD 2023-05-31T21:21:28.597 DEBUG:boto:Path: /pklxkd-67/obj2 2023-05-31T21:21:28.597 DEBUG:boto:Data: 2023-05-31T21:21:28.597 DEBUG:boto:Headers: {} 2023-05-31T21:21:28.598 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T21:21:28.598 DEBUG:boto:Port: 8000 2023-05-31T21:21:28.598 DEBUG:boto:Params: {} 2023-05-31T21:21:28.599 DEBUG:boto:Token: None 2023-05-31T21:21:28.599 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 21:21:28 GMT /pklxkd-67/obj2 2023-05-31T21:21:28.600 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:kxbJyn5v4kUNWY20NSav2UMP+aE= 2023-05-31T21:21:28.600 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 21:21:28 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:kxbJyn5v4kUNWY20NSav2UMP+aE=', 'Content-Length': '0'} 2023-05-31T21:21:28.601 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 21:19:04 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000007ec362b49ffef197-006477ba58-22ab-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 21:21:28 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:21:28.602 DEBUG:boto:path=/pklxkd-67/obj2 2023-05-31T21:21:28.602 DEBUG:boto:auth_path=/pklxkd-67/obj2 2023-05-31T21:21:28.603 DEBUG:boto:Method: GET 2023-05-31T21:21:28.603 DEBUG:boto:Path: /pklxkd-67/obj2 2023-05-31T21:21:28.604 DEBUG:boto:Data: 2023-05-31T21:21:28.604 DEBUG:boto:Headers: {} 2023-05-31T21:21:28.604 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T21:21:28.605 DEBUG:boto:Port: 8000 2023-05-31T21:21:28.605 DEBUG:boto:Params: {} 2023-05-31T21:21:28.605 DEBUG:boto:Token: None 2023-05-31T21:21:28.606 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 21:21:28 GMT /pklxkd-67/obj2 2023-05-31T21:21:28.606 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:ExpP6nIfuJE/B27b/vvahVhoiVs= 2023-05-31T21:21:28.607 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 21:21:28 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:ExpP6nIfuJE/B27b/vvahVhoiVs=', 'Content-Length': '0'} 2023-05-31T21:21:28.608 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 21:19:04 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000003d81dbe7bb120b73-006477ba58-22ab-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 21:21:28 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:21:28.609 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'group', 'remove', '--group-id', 'sync-group', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-05-31T21:21:28.609 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'group', 'remove', '--group-id', 'sync-group', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-05-31T21:21:28.610 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync group remove --group-id sync-group --cluster c1 --debug-rgw 0 --debug-ms 0 2023-05-31T21:21:28.659 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2023-05-31T21:21:28.659 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2023-05-31T21:21:28.710 INFO:teuthology.orchestra.run.smithi099.stdout:{"sync_policy":{"groups":[]}} 2023-05-31T21:21:28.711 INFO:tasks.rgw_multisite_tests:test_sync_flow_symmetrical_zonegroup_all: ... ok 2023-05-31T21:21:28.712 INFO:tasks.rgw_multisite_tests:test_sync_flow_symmetrical_zonegroup_select: ... SKIP: test_sync_flow_symmetrical_zonegroup_select skipped. Requires 3 or more zones in master zonegroup. 2023-05-31T21:21:28.713 INFO:tasks.rgw_multisite_tests:test_sync_flow_directional_zonegroup_select: ... SKIP: test_sync_flow_symmetrical_zonegroup_select skipped. Requires 3 or more zones in master zonegroup. 2023-05-31T21:21:28.714 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-31T21:21:28.716 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-31T21:21:28.716 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-05-31T21:21:28.765 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2023-05-31T21:21:28.765 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2023-05-31T21:21:28.826 INFO:teuthology.orchestra.run.smithi099.stderr:No --period given, using current period=2e6123d2-d695-4350-9a01-61a455bc782b 2023-05-31T21:21:28.851 INFO:teuthology.orchestra.run.smithi099.stdout:[ 2023-05-31T21:21:28.851 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T21:21:28.851 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1685566501.757183_260.1", 2023-05-31T21:21:28.852 INFO:teuthology.orchestra.run.smithi099.stdout: "last_update": "2023-05-31T20:55:01.757183Z" 2023-05-31T21:21:28.852 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:21:28.852 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T21:21:28.852 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1685566501.735717_251.1", 2023-05-31T21:21:28.852 INFO:teuthology.orchestra.run.smithi099.stdout: "last_update": "2023-05-31T20:55:01.735717Z" 2023-05-31T21:21:28.853 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:21:28.853 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T21:21:28.853 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1685562970.274572_93.1", 2023-05-31T21:21:28.853 INFO:teuthology.orchestra.run.smithi099.stdout: "last_update": "2023-05-31T19:56:10.274572Z" 2023-05-31T21:21:28.853 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:21:28.854 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T21:21:28.854 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1685567943.999406_931.1", 2023-05-31T21:21:28.854 INFO:teuthology.orchestra.run.smithi099.stdout: "last_update": "2023-05-31T21:19:03.999406Z" 2023-05-31T21:21:28.854 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T21:21:28.854 INFO:teuthology.orchestra.run.smithi099.stdout:] 2023-05-31T21:21:28.856 DEBUG:rgw_multi.tests:master meta markers={0: '1_1685566501.757183_260.1', 1: '1_1685566501.735717_251.1', 2: '1_1685562970.274572_93.1', 3: '1_1685567943.999406_931.1'} 2023-05-31T21:21:28.856 INFO:rgw_multi.tests:starting meta checkpoint for zone=a2 2023-05-31T21:21:28.857 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-31T21:21:28.857 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-31T21:21:28.858 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-05-31T21:21:28.907 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2023-05-31T21:21:28.908 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2023-05-31T21:21:28.952 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T21:21:28.949+0000 7efd59a07780 20 RGW-SYNC:meta: read sync status 2023-05-31T21:21:28.953 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T21:21:28.950+0000 7efd59a07780 20 RGW-SYNC:meta: read sync status 2023-05-31T21:21:28.967 INFO:teuthology.orchestra.run.smithi099.stdout:{ 2023-05-31T21:21:28.967 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_status": { 2023-05-31T21:21:28.967 INFO:teuthology.orchestra.run.smithi099.stdout: "info": { 2023-05-31T21:21:28.967 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "sync", 2023-05-31T21:21:28.968 INFO:teuthology.orchestra.run.smithi099.stdout: "num_shards": 4, 2023-05-31T21:21:28.968 INFO:teuthology.orchestra.run.smithi099.stdout: "period": "2e6123d2-d695-4350-9a01-61a455bc782b", 2023-05-31T21:21:28.968 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2023-05-31T21:21:28.968 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:21:28.968 INFO:teuthology.orchestra.run.smithi099.stdout: "markers": [ 2023-05-31T21:21:28.968 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T21:21:28.968 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 0, 2023-05-31T21:21:28.969 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T21:21:28.969 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2023-05-31T21:21:28.969 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1685566501.757183_260.1", 2023-05-31T21:21:28.969 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T21:21:28.969 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 1, 2023-05-31T21:21:28.969 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T21:21:28.970 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2023-05-31T20:55:01.757183Z", 2023-05-31T21:21:28.970 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2023-05-31T21:21:28.970 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T21:21:28.970 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:21:28.970 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T21:21:28.970 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 1, 2023-05-31T21:21:28.970 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T21:21:28.971 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2023-05-31T21:21:28.971 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1685566501.735717_251.1", 2023-05-31T21:21:28.971 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T21:21:28.971 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2023-05-31T21:21:28.971 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T21:21:28.972 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2023-05-31T20:55:01.735717Z", 2023-05-31T21:21:28.972 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2023-05-31T21:21:28.972 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T21:21:28.972 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:21:28.973 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T21:21:28.973 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 2, 2023-05-31T21:21:28.973 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T21:21:28.973 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2023-05-31T21:21:28.973 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1685562970.274572_93.1", 2023-05-31T21:21:28.974 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T21:21:28.974 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2023-05-31T21:21:28.974 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T21:21:28.974 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2023-05-31T19:56:10.274572Z", 2023-05-31T21:21:28.974 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2023-05-31T21:21:28.974 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T21:21:28.975 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:21:28.975 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T21:21:28.975 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 3, 2023-05-31T21:21:28.975 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T21:21:28.975 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2023-05-31T21:21:28.975 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1685567943.999406_931.1", 2023-05-31T21:21:28.975 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T21:21:28.976 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2023-05-31T21:21:28.976 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T21:21:28.976 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2023-05-31T21:19:03.999406Z", 2023-05-31T21:21:28.976 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2023-05-31T21:21:28.976 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T21:21:28.976 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T21:21:28.976 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2023-05-31T21:21:28.977 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:21:28.977 INFO:teuthology.orchestra.run.smithi099.stdout: "full_sync": { 2023-05-31T21:21:28.977 INFO:teuthology.orchestra.run.smithi099.stdout: "total": 1, 2023-05-31T21:21:28.977 INFO:teuthology.orchestra.run.smithi099.stdout: "complete": 1 2023-05-31T21:21:28.977 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:21:28.977 INFO:teuthology.orchestra.run.smithi099.stdout: "current_time": "2023-05-31T21:21:28Z" 2023-05-31T21:21:28.978 INFO:teuthology.orchestra.run.smithi099.stdout:} 2023-05-31T21:21:28.979 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "2e6123d2-d695-4350-9a01-61a455bc782b", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1685566501.757183_260.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-05-31T20:55:01.757183Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1685566501.735717_251.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-05-31T20:55:01.735717Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1685562970.274572_93.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-05-31T19:56:10.274572Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1685567943.999406_931.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-05-31T21:19:03.999406Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-05-31T21:21:28Z" } 2023-05-31T21:21:28.979 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1685566501.757183_260.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-05-31T20:55:01.757183Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1685566501.735717_251.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-05-31T20:55:01.735717Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1685562970.274572_93.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-05-31T19:56:10.274572Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1685567943.999406_931.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-05-31T21:19:03.999406Z', 'realm_epoch': 2}}] 2023-05-31T21:21:28.979 DEBUG:rgw_multi.tests:log_status={0: '1_1685566501.757183_260.1', 1: '1_1685566501.735717_251.1', 2: '1_1685562970.274572_93.1', 3: '1_1685567943.999406_931.1'} 2023-05-31T21:21:28.980 DEBUG:rgw_multi.tests:sync_status={0: '1_1685566501.757183_260.1', 1: '1_1685566501.735717_251.1', 2: '1_1685562970.274572_93.1', 3: '1_1685567943.999406_931.1'} 2023-05-31T21:21:28.980 INFO:rgw_multi.tests:finish meta checkpoint for zone=a2 2023-05-31T21:21:28.981 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'policy', 'get', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-05-31T21:21:28.981 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'policy', 'get', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-05-31T21:21:28.981 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync policy get --cluster c1 --debug-rgw 0 --debug-ms 0 2023-05-31T21:21:29.027 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2023-05-31T21:21:29.028 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2023-05-31T21:21:29.094 INFO:teuthology.orchestra.run.smithi099.stdout:{"groups":[]} 2023-05-31T21:21:29.095 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'policy', 'get', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-05-31T21:21:29.096 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'policy', 'get', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-05-31T21:21:29.097 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync policy get --cluster c1 --debug-rgw 0 --debug-ms 0 2023-05-31T21:21:29.143 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2023-05-31T21:21:29.144 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2023-05-31T21:21:29.221 INFO:teuthology.orchestra.run.smithi099.stdout:{"groups":[]} 2023-05-31T21:21:29.222 INFO:rgw_multi.tests:create bucket zone=a1 name=pklxkd-68 2023-05-31T21:21:29.223 DEBUG:boto:path=/pklxkd-68/ 2023-05-31T21:21:29.224 DEBUG:boto:auth_path=/pklxkd-68/ 2023-05-31T21:21:29.225 DEBUG:boto:Method: PUT 2023-05-31T21:21:29.225 DEBUG:boto:Path: /pklxkd-68/ 2023-05-31T21:21:29.226 DEBUG:boto:Data: 2023-05-31T21:21:29.227 DEBUG:boto:Headers: {} 2023-05-31T21:21:29.227 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T21:21:29.228 DEBUG:boto:Port: 8000 2023-05-31T21:21:29.228 DEBUG:boto:Params: {} 2023-05-31T21:21:29.229 DEBUG:boto:Token: None 2023-05-31T21:21:29.230 DEBUG:boto:StringToSign: PUT Wed, 31 May 2023 21:21:29 GMT /pklxkd-68/ 2023-05-31T21:21:29.230 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:vzOPE0/dSQUrwe/dBHBbvXm2k1c= 2023-05-31T21:21:29.231 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 21:21:29 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:vzOPE0/dSQUrwe/dBHBbvXm2k1c=', 'Content-Length': '0'} 2023-05-31T21:21:29.250 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000b84acbfc8c03f6b6-006477ba59-22ab-a1'), ('Content-Length', '0'), ('Date', 'Wed, 31 May 2023 21:21:29 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:21:29.251 INFO:rgw_multi.tests:create bucket zone=a1 name=pklxkd-69 2023-05-31T21:21:29.251 DEBUG:boto:path=/pklxkd-69/ 2023-05-31T21:21:29.251 DEBUG:boto:auth_path=/pklxkd-69/ 2023-05-31T21:21:29.252 DEBUG:boto:Method: PUT 2023-05-31T21:21:29.252 DEBUG:boto:Path: /pklxkd-69/ 2023-05-31T21:21:29.252 DEBUG:boto:Data: 2023-05-31T21:21:29.253 DEBUG:boto:Headers: {} 2023-05-31T21:21:29.253 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T21:21:29.253 DEBUG:boto:Port: 8000 2023-05-31T21:21:29.254 DEBUG:boto:Params: {} 2023-05-31T21:21:29.254 DEBUG:boto:Token: None 2023-05-31T21:21:29.254 DEBUG:boto:StringToSign: PUT Wed, 31 May 2023 21:21:29 GMT /pklxkd-69/ 2023-05-31T21:21:29.255 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:GfXB6v/3aRB1HwWO9DeWd+HiK/o= 2023-05-31T21:21:29.255 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 21:21:29 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:GfXB6v/3aRB1HwWO9DeWd+HiK/o=', 'Content-Length': '0'} 2023-05-31T21:21:29.273 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx0000005a68c715c02b31b-006477ba59-22ab-a1'), ('Content-Length', '0'), ('Date', 'Wed, 31 May 2023 21:21:29 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:21:29.274 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-31T21:21:29.274 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-31T21:21:29.275 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-05-31T21:21:29.326 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2023-05-31T21:21:29.327 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2023-05-31T21:21:29.396 INFO:teuthology.orchestra.run.smithi099.stderr:No --period given, using current period=2e6123d2-d695-4350-9a01-61a455bc782b 2023-05-31T21:21:29.420 INFO:teuthology.orchestra.run.smithi099.stdout:[ 2023-05-31T21:21:29.421 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T21:21:29.421 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1685566501.757183_260.1", 2023-05-31T21:21:29.421 INFO:teuthology.orchestra.run.smithi099.stdout: "last_update": "2023-05-31T20:55:01.757183Z" 2023-05-31T21:21:29.421 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:21:29.421 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T21:21:29.422 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1685566501.735717_251.1", 2023-05-31T21:21:29.422 INFO:teuthology.orchestra.run.smithi099.stdout: "last_update": "2023-05-31T20:55:01.735717Z" 2023-05-31T21:21:29.422 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:21:29.422 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T21:21:29.422 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1685562970.274572_93.1", 2023-05-31T21:21:29.422 INFO:teuthology.orchestra.run.smithi099.stdout: "last_update": "2023-05-31T19:56:10.274572Z" 2023-05-31T21:21:29.422 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:21:29.423 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T21:21:29.423 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1685568089.268814_960.1", 2023-05-31T21:21:29.423 INFO:teuthology.orchestra.run.smithi099.stdout: "last_update": "2023-05-31T21:21:29.268814Z" 2023-05-31T21:21:29.423 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T21:21:29.423 INFO:teuthology.orchestra.run.smithi099.stdout:] 2023-05-31T21:21:29.424 DEBUG:rgw_multi.tests:master meta markers={0: '1_1685566501.757183_260.1', 1: '1_1685566501.735717_251.1', 2: '1_1685562970.274572_93.1', 3: '1_1685568089.268814_960.1'} 2023-05-31T21:21:29.425 INFO:rgw_multi.tests:starting meta checkpoint for zone=a2 2023-05-31T21:21:29.425 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-31T21:21:29.425 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-31T21:21:29.426 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-05-31T21:21:29.475 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2023-05-31T21:21:29.476 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2023-05-31T21:21:29.508 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T21:21:29.504+0000 7ff1eb11f780 20 RGW-SYNC:meta: read sync status 2023-05-31T21:21:29.509 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T21:21:29.505+0000 7ff1eb11f780 20 RGW-SYNC:meta: read sync status 2023-05-31T21:21:29.522 INFO:teuthology.orchestra.run.smithi099.stdout:{ 2023-05-31T21:21:29.523 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_status": { 2023-05-31T21:21:29.523 INFO:teuthology.orchestra.run.smithi099.stdout: "info": { 2023-05-31T21:21:29.523 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "sync", 2023-05-31T21:21:29.524 INFO:teuthology.orchestra.run.smithi099.stdout: "num_shards": 4, 2023-05-31T21:21:29.524 INFO:teuthology.orchestra.run.smithi099.stdout: "period": "2e6123d2-d695-4350-9a01-61a455bc782b", 2023-05-31T21:21:29.524 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2023-05-31T21:21:29.524 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:21:29.525 INFO:teuthology.orchestra.run.smithi099.stdout: "markers": [ 2023-05-31T21:21:29.525 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T21:21:29.525 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 0, 2023-05-31T21:21:29.526 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T21:21:29.526 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2023-05-31T21:21:29.526 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1685566501.757183_260.1", 2023-05-31T21:21:29.527 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T21:21:29.527 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 1, 2023-05-31T21:21:29.528 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T21:21:29.528 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2023-05-31T20:55:01.757183Z", 2023-05-31T21:21:29.528 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2023-05-31T21:21:29.529 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T21:21:29.529 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:21:29.529 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T21:21:29.530 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 1, 2023-05-31T21:21:29.530 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T21:21:29.530 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2023-05-31T21:21:29.530 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1685566501.735717_251.1", 2023-05-31T21:21:29.531 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T21:21:29.531 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2023-05-31T21:21:29.531 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T21:21:29.532 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2023-05-31T20:55:01.735717Z", 2023-05-31T21:21:29.532 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2023-05-31T21:21:29.532 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T21:21:29.532 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:21:29.533 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T21:21:29.533 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 2, 2023-05-31T21:21:29.533 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T21:21:29.534 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2023-05-31T21:21:29.534 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1685562970.274572_93.1", 2023-05-31T21:21:29.534 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T21:21:29.534 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2023-05-31T21:21:29.535 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T21:21:29.535 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2023-05-31T19:56:10.274572Z", 2023-05-31T21:21:29.536 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2023-05-31T21:21:29.536 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T21:21:29.537 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:21:29.537 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T21:21:29.537 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 3, 2023-05-31T21:21:29.538 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T21:21:29.538 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2023-05-31T21:21:29.538 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1685567943.999406_931.1", 2023-05-31T21:21:29.539 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T21:21:29.539 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2023-05-31T21:21:29.539 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T21:21:29.540 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2023-05-31T21:19:03.999406Z", 2023-05-31T21:21:29.540 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2023-05-31T21:21:29.540 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T21:21:29.541 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T21:21:29.541 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2023-05-31T21:21:29.541 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:21:29.542 INFO:teuthology.orchestra.run.smithi099.stdout: "full_sync": { 2023-05-31T21:21:29.545 INFO:teuthology.orchestra.run.smithi099.stdout: "total": 1, 2023-05-31T21:21:29.545 INFO:teuthology.orchestra.run.smithi099.stdout: "complete": 1 2023-05-31T21:21:29.545 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:21:29.546 INFO:teuthology.orchestra.run.smithi099.stdout: "current_time": "2023-05-31T21:21:29Z" 2023-05-31T21:21:29.546 INFO:teuthology.orchestra.run.smithi099.stdout:} 2023-05-31T21:21:29.548 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "2e6123d2-d695-4350-9a01-61a455bc782b", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1685566501.757183_260.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-05-31T20:55:01.757183Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1685566501.735717_251.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-05-31T20:55:01.735717Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1685562970.274572_93.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-05-31T19:56:10.274572Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1685567943.999406_931.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-05-31T21:19:03.999406Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-05-31T21:21:29Z" } 2023-05-31T21:21:29.548 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1685566501.757183_260.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-05-31T20:55:01.757183Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1685566501.735717_251.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-05-31T20:55:01.735717Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1685562970.274572_93.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-05-31T19:56:10.274572Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1685567943.999406_931.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-05-31T21:19:03.999406Z', 'realm_epoch': 2}}] 2023-05-31T21:21:29.549 DEBUG:rgw_multi.tests:log_status={0: '1_1685566501.757183_260.1', 1: '1_1685566501.735717_251.1', 2: '1_1685562970.274572_93.1', 3: '1_1685568089.268814_960.1'} 2023-05-31T21:21:29.549 DEBUG:rgw_multi.tests:sync_status={0: '1_1685566501.757183_260.1', 1: '1_1685566501.735717_251.1', 2: '1_1685562970.274572_93.1', 3: '1_1685567943.999406_931.1'} 2023-05-31T21:21:29.550 WARNING:rgw_multi.tests:zone a2 behind master: shard=3 master=1_1685568089.268814_960.1 target=1_1685567943.999406_931.1 2023-05-31T21:21:34.550 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-31T21:21:34.551 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-31T21:21:34.552 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-05-31T21:21:34.611 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2023-05-31T21:21:34.611 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2023-05-31T21:21:34.646 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T21:21:34.642+0000 7f4312202780 20 RGW-SYNC:meta: read sync status 2023-05-31T21:21:34.647 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T21:21:34.643+0000 7f4312202780 20 RGW-SYNC:meta: read sync status 2023-05-31T21:21:34.662 INFO:teuthology.orchestra.run.smithi099.stdout:{ 2023-05-31T21:21:34.662 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_status": { 2023-05-31T21:21:34.663 INFO:teuthology.orchestra.run.smithi099.stdout: "info": { 2023-05-31T21:21:34.663 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "sync", 2023-05-31T21:21:34.663 INFO:teuthology.orchestra.run.smithi099.stdout: "num_shards": 4, 2023-05-31T21:21:34.663 INFO:teuthology.orchestra.run.smithi099.stdout: "period": "2e6123d2-d695-4350-9a01-61a455bc782b", 2023-05-31T21:21:34.663 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2023-05-31T21:21:34.663 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:21:34.663 INFO:teuthology.orchestra.run.smithi099.stdout: "markers": [ 2023-05-31T21:21:34.664 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T21:21:34.664 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 0, 2023-05-31T21:21:34.664 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T21:21:34.664 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2023-05-31T21:21:34.664 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1685566501.757183_260.1", 2023-05-31T21:21:34.664 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T21:21:34.665 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 1, 2023-05-31T21:21:34.665 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T21:21:34.665 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2023-05-31T20:55:01.757183Z", 2023-05-31T21:21:34.665 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2023-05-31T21:21:34.665 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T21:21:34.665 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:21:34.666 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T21:21:34.666 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 1, 2023-05-31T21:21:34.666 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T21:21:34.666 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2023-05-31T21:21:34.666 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1685566501.735717_251.1", 2023-05-31T21:21:34.666 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T21:21:34.666 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2023-05-31T21:21:34.667 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T21:21:34.667 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2023-05-31T20:55:01.735717Z", 2023-05-31T21:21:34.667 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2023-05-31T21:21:34.667 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T21:21:34.667 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:21:34.667 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T21:21:34.667 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 2, 2023-05-31T21:21:34.668 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T21:21:34.668 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2023-05-31T21:21:34.668 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1685562970.274572_93.1", 2023-05-31T21:21:34.668 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T21:21:34.668 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2023-05-31T21:21:34.668 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T21:21:34.668 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2023-05-31T19:56:10.274572Z", 2023-05-31T21:21:34.669 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2023-05-31T21:21:34.669 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T21:21:34.669 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:21:34.669 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T21:21:34.669 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 3, 2023-05-31T21:21:34.669 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T21:21:34.669 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2023-05-31T21:21:34.670 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1685568089.268814_960.1", 2023-05-31T21:21:34.670 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T21:21:34.670 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2023-05-31T21:21:34.670 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T21:21:34.670 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2023-05-31T21:21:29.268814Z", 2023-05-31T21:21:34.670 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2023-05-31T21:21:34.670 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T21:21:34.671 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T21:21:34.671 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2023-05-31T21:21:34.671 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:21:34.671 INFO:teuthology.orchestra.run.smithi099.stdout: "full_sync": { 2023-05-31T21:21:34.671 INFO:teuthology.orchestra.run.smithi099.stdout: "total": 1, 2023-05-31T21:21:34.671 INFO:teuthology.orchestra.run.smithi099.stdout: "complete": 1 2023-05-31T21:21:34.672 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:21:34.672 INFO:teuthology.orchestra.run.smithi099.stdout: "current_time": "2023-05-31T21:21:34Z" 2023-05-31T21:21:34.672 INFO:teuthology.orchestra.run.smithi099.stdout:} 2023-05-31T21:21:34.673 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "2e6123d2-d695-4350-9a01-61a455bc782b", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1685566501.757183_260.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-05-31T20:55:01.757183Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1685566501.735717_251.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-05-31T20:55:01.735717Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1685562970.274572_93.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-05-31T19:56:10.274572Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1685568089.268814_960.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-05-31T21:21:29.268814Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-05-31T21:21:34Z" } 2023-05-31T21:21:34.673 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1685566501.757183_260.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-05-31T20:55:01.757183Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1685566501.735717_251.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-05-31T20:55:01.735717Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1685562970.274572_93.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-05-31T19:56:10.274572Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1685568089.268814_960.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-05-31T21:21:29.268814Z', 'realm_epoch': 2}}] 2023-05-31T21:21:34.674 DEBUG:rgw_multi.tests:log_status={0: '1_1685566501.757183_260.1', 1: '1_1685566501.735717_251.1', 2: '1_1685562970.274572_93.1', 3: '1_1685568089.268814_960.1'} 2023-05-31T21:21:34.674 DEBUG:rgw_multi.tests:sync_status={0: '1_1685566501.757183_260.1', 1: '1_1685566501.735717_251.1', 2: '1_1685562970.274572_93.1', 3: '1_1685568089.268814_960.1'} 2023-05-31T21:21:34.675 INFO:rgw_multi.tests:finish meta checkpoint for zone=a2 2023-05-31T21:21:34.675 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'group', 'create', '--group-id', 'sync-group', '--status', 'allowed', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-05-31T21:21:34.675 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'group', 'create', '--group-id', 'sync-group', '--status', 'allowed', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-05-31T21:21:34.676 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync group create --group-id sync-group --status allowed --cluster c1 --debug-rgw 0 --debug-ms 0 2023-05-31T21:21:34.728 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2023-05-31T21:21:34.728 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2023-05-31T21:21:34.783 INFO:teuthology.orchestra.run.smithi099.stdout:{"groups":[{"id":"sync-group","data_flow":{},"pipes":[],"status":"allowed"}]} 2023-05-31T21:21:34.784 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'group', 'flow', 'create', '--group-id', 'sync-group', '--flow-id', 'sync-flow1', '--flow-type', 'symmetrical', '--zones=a1,a2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-05-31T21:21:34.784 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'group', 'flow', 'create', '--group-id', 'sync-group', '--flow-id', 'sync-flow1', '--flow-type', 'symmetrical', '--zones=a1,a2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-05-31T21:21:34.785 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync group flow create --group-id sync-group --flow-id sync-flow1 --flow-type symmetrical --zones=a1,a2 --cluster c1 --debug-rgw 0 --debug-ms 0 2023-05-31T21:21:34.838 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2023-05-31T21:21:34.838 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2023-05-31T21:21:34.913 INFO:teuthology.orchestra.run.smithi099.stdout:{"groups":[{"id":"sync-group","data_flow":{"symmetrical":[{"id":"sync-flow1","zones":["a1","a2"]}]},"pipes":[],"status":"allowed"}]} 2023-05-31T21:21:34.914 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'group', 'pipe', 'create', '--group-id', 'sync-group', '--pipe-id', 'sync-pipe', '--source-zones=a1,a2', '--dest-zones=a1,a2', '--source-bucket=pklxkd-68', '--dest-bucket=pklxkd-68', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-05-31T21:21:34.914 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'group', 'pipe', 'create', '--group-id', 'sync-group', '--pipe-id', 'sync-pipe', '--source-zones=a1,a2', '--dest-zones=a1,a2', '--source-bucket=pklxkd-68', '--dest-bucket=pklxkd-68', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-05-31T21:21:34.915 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync group pipe create --group-id sync-group --pipe-id sync-pipe --source-zones=a1,a2 --dest-zones=a1,a2 --source-bucket=pklxkd-68 --dest-bucket=pklxkd-68 --cluster c1 --debug-rgw 0 --debug-ms 0 2023-05-31T21:21:34.970 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2023-05-31T21:21:34.970 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2023-05-31T21:21:35.054 INFO:teuthology.orchestra.run.smithi099.stdout:{"groups":[{"id":"sync-group","data_flow":{"symmetrical":[{"id":"sync-flow1","zones":["a1","a2"]}]},"pipes":[{"id":"sync-pipe","source":{"bucket":"pklxkd-68","zones":["a1","a2"]},"dest":{"bucket":"pklxkd-68","zones":["a1","a2"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":""}}],"status":"allowed"}]} 2023-05-31T21:21:35.054 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'group', 'modify', '--group-id', 'sync-group', '--status', 'enabled', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-05-31T21:21:35.055 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'group', 'modify', '--group-id', 'sync-group', '--status', 'enabled', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-05-31T21:21:35.055 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync group modify --group-id sync-group --status enabled --cluster c1 --debug-rgw 0 --debug-ms 0 2023-05-31T21:21:35.105 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2023-05-31T21:21:35.105 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2023-05-31T21:21:35.178 INFO:teuthology.orchestra.run.smithi099.stdout:{"groups":[{"id":"sync-group","data_flow":{"symmetrical":[{"id":"sync-flow1","zones":["a1","a2"]}]},"pipes":[{"id":"sync-pipe","source":{"bucket":"pklxkd-68","zones":["a1","a2"]},"dest":{"bucket":"pklxkd-68","zones":["a1","a2"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":""}}],"status":"enabled"}]} 2023-05-31T21:21:35.179 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'policy', 'get', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-05-31T21:21:35.180 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'policy', 'get', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-05-31T21:21:35.180 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync policy get --cluster c1 --debug-rgw 0 --debug-ms 0 2023-05-31T21:21:35.226 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2023-05-31T21:21:35.227 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2023-05-31T21:21:35.275 INFO:teuthology.orchestra.run.smithi099.stdout:{"groups":[{"id":"sync-group","data_flow":{"symmetrical":[{"id":"sync-flow1","zones":["a1","a2"]}]},"pipes":[{"id":"sync-pipe","source":{"bucket":"pklxkd-68","zones":["a1","a2"]},"dest":{"bucket":"pklxkd-68","zones":["a1","a2"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":""}}],"status":"enabled"}]} 2023-05-31T21:21:35.275 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['period', 'update', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-05-31T21:21:35.277 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'period', 'update', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-05-31T21:21:35.278 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 period update --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --commit --cluster c1 --debug-rgw 0 --debug-ms 0 2023-05-31T21:21:35.326 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2023-05-31T21:21:35.326 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2023-05-31T21:21:35.400 INFO:teuthology.orchestra.run.smithi099.stdout:{ 2023-05-31T21:21:35.400 INFO:teuthology.orchestra.run.smithi099.stdout: "id": "2e6123d2-d695-4350-9a01-61a455bc782b", 2023-05-31T21:21:35.401 INFO:teuthology.orchestra.run.smithi099.stdout: "epoch": 16, 2023-05-31T21:21:35.401 INFO:teuthology.orchestra.run.smithi099.stdout: "predecessor_uuid": "629b9ced-80af-41ed-8a6c-f2cfc334cbb4", 2023-05-31T21:21:35.401 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_status": [], 2023-05-31T21:21:35.402 INFO:teuthology.orchestra.run.smithi099.stdout: "period_map": { 2023-05-31T21:21:35.402 INFO:teuthology.orchestra.run.smithi099.stdout: "id": "2e6123d2-d695-4350-9a01-61a455bc782b", 2023-05-31T21:21:35.403 INFO:teuthology.orchestra.run.smithi099.stdout: "zonegroups": [ 2023-05-31T21:21:35.403 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T21:21:35.403 INFO:teuthology.orchestra.run.smithi099.stdout: "id": "a", 2023-05-31T21:21:35.403 INFO:teuthology.orchestra.run.smithi099.stdout: "name": "a", 2023-05-31T21:21:35.403 INFO:teuthology.orchestra.run.smithi099.stdout: "api_name": "", 2023-05-31T21:21:35.403 INFO:teuthology.orchestra.run.smithi099.stdout: "is_master": true, 2023-05-31T21:21:35.404 INFO:teuthology.orchestra.run.smithi099.stdout: "endpoints": [ 2023-05-31T21:21:35.404 INFO:teuthology.orchestra.run.smithi099.stdout: "http://smithi099.front.sepia.ceph.com:8000" 2023-05-31T21:21:35.404 INFO:teuthology.orchestra.run.smithi099.stdout: ], 2023-05-31T21:21:35.404 INFO:teuthology.orchestra.run.smithi099.stdout: "hostnames": [], 2023-05-31T21:21:35.404 INFO:teuthology.orchestra.run.smithi099.stdout: "hostnames_s3website": [], 2023-05-31T21:21:35.404 INFO:teuthology.orchestra.run.smithi099.stdout: "master_zone": "3d163b1e-09c7-47e2-ae87-add4aace3f38", 2023-05-31T21:21:35.405 INFO:teuthology.orchestra.run.smithi099.stdout: "zones": [ 2023-05-31T21:21:35.405 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T21:21:35.405 INFO:teuthology.orchestra.run.smithi099.stdout: "id": "3d163b1e-09c7-47e2-ae87-add4aace3f38", 2023-05-31T21:21:35.405 INFO:teuthology.orchestra.run.smithi099.stdout: "name": "a1", 2023-05-31T21:21:35.405 INFO:teuthology.orchestra.run.smithi099.stdout: "endpoints": [ 2023-05-31T21:21:35.405 INFO:teuthology.orchestra.run.smithi099.stdout: "http://smithi099.front.sepia.ceph.com:8000" 2023-05-31T21:21:35.405 INFO:teuthology.orchestra.run.smithi099.stdout: ], 2023-05-31T21:21:35.406 INFO:teuthology.orchestra.run.smithi099.stdout: "log_meta": false, 2023-05-31T21:21:35.406 INFO:teuthology.orchestra.run.smithi099.stdout: "log_data": true, 2023-05-31T21:21:35.406 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket_index_max_shards": 11, 2023-05-31T21:21:35.406 INFO:teuthology.orchestra.run.smithi099.stdout: "read_only": false, 2023-05-31T21:21:35.406 INFO:teuthology.orchestra.run.smithi099.stdout: "tier_type": "", 2023-05-31T21:21:35.406 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_from_all": true, 2023-05-31T21:21:35.407 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_from": [], 2023-05-31T21:21:35.407 INFO:teuthology.orchestra.run.smithi099.stdout: "redirect_zone": "", 2023-05-31T21:21:35.407 INFO:teuthology.orchestra.run.smithi099.stdout: "supported_features": [ 2023-05-31T21:21:35.407 INFO:teuthology.orchestra.run.smithi099.stdout: "resharding" 2023-05-31T21:21:35.407 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2023-05-31T21:21:35.408 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:21:35.408 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T21:21:35.408 INFO:teuthology.orchestra.run.smithi099.stdout: "id": "5d8acec7-7e57-4e56-8b32-e3d6b5036001", 2023-05-31T21:21:35.408 INFO:teuthology.orchestra.run.smithi099.stdout: "name": "a2", 2023-05-31T21:21:35.408 INFO:teuthology.orchestra.run.smithi099.stdout: "endpoints": [ 2023-05-31T21:21:35.408 INFO:teuthology.orchestra.run.smithi099.stdout: "http://smithi099.front.sepia.ceph.com:8001" 2023-05-31T21:21:35.408 INFO:teuthology.orchestra.run.smithi099.stdout: ], 2023-05-31T21:21:35.409 INFO:teuthology.orchestra.run.smithi099.stdout: "log_meta": false, 2023-05-31T21:21:35.409 INFO:teuthology.orchestra.run.smithi099.stdout: "log_data": true, 2023-05-31T21:21:35.409 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket_index_max_shards": 11, 2023-05-31T21:21:35.409 INFO:teuthology.orchestra.run.smithi099.stdout: "read_only": false, 2023-05-31T21:21:35.409 INFO:teuthology.orchestra.run.smithi099.stdout: "tier_type": "", 2023-05-31T21:21:35.409 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_from_all": true, 2023-05-31T21:21:35.409 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_from": [], 2023-05-31T21:21:35.410 INFO:teuthology.orchestra.run.smithi099.stdout: "redirect_zone": "", 2023-05-31T21:21:35.410 INFO:teuthology.orchestra.run.smithi099.stdout: "supported_features": [ 2023-05-31T21:21:35.410 INFO:teuthology.orchestra.run.smithi099.stdout: "resharding" 2023-05-31T21:21:35.410 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2023-05-31T21:21:35.410 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T21:21:35.410 INFO:teuthology.orchestra.run.smithi099.stdout: ], 2023-05-31T21:21:35.410 INFO:teuthology.orchestra.run.smithi099.stdout: "placement_targets": [ 2023-05-31T21:21:35.411 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T21:21:35.411 INFO:teuthology.orchestra.run.smithi099.stdout: "name": "default-placement", 2023-05-31T21:21:35.411 INFO:teuthology.orchestra.run.smithi099.stdout: "tags": [], 2023-05-31T21:21:35.411 INFO:teuthology.orchestra.run.smithi099.stdout: "storage_classes": [ 2023-05-31T21:21:35.411 INFO:teuthology.orchestra.run.smithi099.stdout: "STANDARD" 2023-05-31T21:21:35.411 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2023-05-31T21:21:35.411 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T21:21:35.412 INFO:teuthology.orchestra.run.smithi099.stdout: ], 2023-05-31T21:21:35.412 INFO:teuthology.orchestra.run.smithi099.stdout: "default_placement": "default-placement", 2023-05-31T21:21:35.412 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_id": "aaa33de6-515b-4cac-8f1b-3a1a30452117", 2023-05-31T21:21:35.412 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_policy": { 2023-05-31T21:21:35.412 INFO:teuthology.orchestra.run.smithi099.stdout: "groups": [ 2023-05-31T21:21:35.412 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T21:21:35.413 INFO:teuthology.orchestra.run.smithi099.stdout: "id": "sync-group", 2023-05-31T21:21:35.413 INFO:teuthology.orchestra.run.smithi099.stdout: "data_flow": { 2023-05-31T21:21:35.413 INFO:teuthology.orchestra.run.smithi099.stdout: "symmetrical": [ 2023-05-31T21:21:35.413 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T21:21:35.413 INFO:teuthology.orchestra.run.smithi099.stdout: "id": "sync-flow1", 2023-05-31T21:21:35.413 INFO:teuthology.orchestra.run.smithi099.stdout: "zones": [ 2023-05-31T21:21:35.413 INFO:teuthology.orchestra.run.smithi099.stdout: "3d163b1e-09c7-47e2-ae87-add4aace3f38", 2023-05-31T21:21:35.414 INFO:teuthology.orchestra.run.smithi099.stdout: "5d8acec7-7e57-4e56-8b32-e3d6b5036001" 2023-05-31T21:21:35.414 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2023-05-31T21:21:35.414 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T21:21:35.414 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2023-05-31T21:21:35.414 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:21:35.414 INFO:teuthology.orchestra.run.smithi099.stdout: "pipes": [ 2023-05-31T21:21:35.414 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T21:21:35.415 INFO:teuthology.orchestra.run.smithi099.stdout: "id": "sync-pipe", 2023-05-31T21:21:35.415 INFO:teuthology.orchestra.run.smithi099.stdout: "source": { 2023-05-31T21:21:35.415 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket": "pklxkd-68", 2023-05-31T21:21:35.415 INFO:teuthology.orchestra.run.smithi099.stdout: "zones": [ 2023-05-31T21:21:35.415 INFO:teuthology.orchestra.run.smithi099.stdout: "3d163b1e-09c7-47e2-ae87-add4aace3f38", 2023-05-31T21:21:35.415 INFO:teuthology.orchestra.run.smithi099.stdout: "5d8acec7-7e57-4e56-8b32-e3d6b5036001" 2023-05-31T21:21:35.416 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2023-05-31T21:21:35.416 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:21:35.416 INFO:teuthology.orchestra.run.smithi099.stdout: "dest": { 2023-05-31T21:21:35.416 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket": "pklxkd-68", 2023-05-31T21:21:35.416 INFO:teuthology.orchestra.run.smithi099.stdout: "zones": [ 2023-05-31T21:21:35.416 INFO:teuthology.orchestra.run.smithi099.stdout: "3d163b1e-09c7-47e2-ae87-add4aace3f38", 2023-05-31T21:21:35.417 INFO:teuthology.orchestra.run.smithi099.stdout: "5d8acec7-7e57-4e56-8b32-e3d6b5036001" 2023-05-31T21:21:35.417 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2023-05-31T21:21:35.417 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:21:35.417 INFO:teuthology.orchestra.run.smithi099.stdout: "params": { 2023-05-31T21:21:35.417 INFO:teuthology.orchestra.run.smithi099.stdout: "source": { 2023-05-31T21:21:35.417 INFO:teuthology.orchestra.run.smithi099.stdout: "filter": { 2023-05-31T21:21:35.418 INFO:teuthology.orchestra.run.smithi099.stdout: "tags": [] 2023-05-31T21:21:35.418 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T21:21:35.418 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:21:35.418 INFO:teuthology.orchestra.run.smithi099.stdout: "dest": {}, 2023-05-31T21:21:35.418 INFO:teuthology.orchestra.run.smithi099.stdout: "priority": 0, 2023-05-31T21:21:35.419 INFO:teuthology.orchestra.run.smithi099.stdout: "mode": "system", 2023-05-31T21:21:35.419 INFO:teuthology.orchestra.run.smithi099.stdout: "user": "" 2023-05-31T21:21:35.419 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T21:21:35.419 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T21:21:35.419 INFO:teuthology.orchestra.run.smithi099.stdout: ], 2023-05-31T21:21:35.420 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "enabled" 2023-05-31T21:21:35.420 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T21:21:35.420 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2023-05-31T21:21:35.420 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:21:35.420 INFO:teuthology.orchestra.run.smithi099.stdout: "enabled_features": [ 2023-05-31T21:21:35.420 INFO:teuthology.orchestra.run.smithi099.stdout: "resharding" 2023-05-31T21:21:35.421 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2023-05-31T21:21:35.421 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:21:35.421 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T21:21:35.421 INFO:teuthology.orchestra.run.smithi099.stdout: "id": "b", 2023-05-31T21:21:35.421 INFO:teuthology.orchestra.run.smithi099.stdout: "name": "b", 2023-05-31T21:21:35.421 INFO:teuthology.orchestra.run.smithi099.stdout: "api_name": "", 2023-05-31T21:21:35.421 INFO:teuthology.orchestra.run.smithi099.stdout: "is_master": false, 2023-05-31T21:21:35.422 INFO:teuthology.orchestra.run.smithi099.stdout: "endpoints": [ 2023-05-31T21:21:35.422 INFO:teuthology.orchestra.run.smithi099.stdout: "http://smithi112.front.sepia.ceph.com:8000" 2023-05-31T21:21:35.422 INFO:teuthology.orchestra.run.smithi099.stdout: ], 2023-05-31T21:21:35.422 INFO:teuthology.orchestra.run.smithi099.stdout: "hostnames": [], 2023-05-31T21:21:35.422 INFO:teuthology.orchestra.run.smithi099.stdout: "hostnames_s3website": [], 2023-05-31T21:21:35.422 INFO:teuthology.orchestra.run.smithi099.stdout: "master_zone": "f85d7236-2f27-4988-af26-9c66ab280b1b", 2023-05-31T21:21:35.422 INFO:teuthology.orchestra.run.smithi099.stdout: "zones": [ 2023-05-31T21:21:35.423 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T21:21:35.423 INFO:teuthology.orchestra.run.smithi099.stdout: "id": "a6a3aab5-bc52-44a4-8543-5c5ea6056acd", 2023-05-31T21:21:35.423 INFO:teuthology.orchestra.run.smithi099.stdout: "name": "b2", 2023-05-31T21:21:35.423 INFO:teuthology.orchestra.run.smithi099.stdout: "endpoints": [ 2023-05-31T21:21:35.423 INFO:teuthology.orchestra.run.smithi099.stdout: "http://smithi112.front.sepia.ceph.com:8001" 2023-05-31T21:21:35.423 INFO:teuthology.orchestra.run.smithi099.stdout: ], 2023-05-31T21:21:35.423 INFO:teuthology.orchestra.run.smithi099.stdout: "log_meta": false, 2023-05-31T21:21:35.424 INFO:teuthology.orchestra.run.smithi099.stdout: "log_data": true, 2023-05-31T21:21:35.424 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket_index_max_shards": 11, 2023-05-31T21:21:35.424 INFO:teuthology.orchestra.run.smithi099.stdout: "read_only": false, 2023-05-31T21:21:35.424 INFO:teuthology.orchestra.run.smithi099.stdout: "tier_type": "", 2023-05-31T21:21:35.424 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_from_all": true, 2023-05-31T21:21:35.424 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_from": [], 2023-05-31T21:21:35.425 INFO:teuthology.orchestra.run.smithi099.stdout: "redirect_zone": "", 2023-05-31T21:21:35.425 INFO:teuthology.orchestra.run.smithi099.stdout: "supported_features": [ 2023-05-31T21:21:35.426 INFO:teuthology.orchestra.run.smithi099.stdout: "resharding" 2023-05-31T21:21:35.426 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2023-05-31T21:21:35.426 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:21:35.427 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T21:21:35.427 INFO:teuthology.orchestra.run.smithi099.stdout: "id": "f85d7236-2f27-4988-af26-9c66ab280b1b", 2023-05-31T21:21:35.427 INFO:teuthology.orchestra.run.smithi099.stdout: "name": "b1", 2023-05-31T21:21:35.427 INFO:teuthology.orchestra.run.smithi099.stdout: "endpoints": [ 2023-05-31T21:21:35.427 INFO:teuthology.orchestra.run.smithi099.stdout: "http://smithi112.front.sepia.ceph.com:8000" 2023-05-31T21:21:35.428 INFO:teuthology.orchestra.run.smithi099.stdout: ], 2023-05-31T21:21:35.428 INFO:teuthology.orchestra.run.smithi099.stdout: "log_meta": false, 2023-05-31T21:21:35.428 INFO:teuthology.orchestra.run.smithi099.stdout: "log_data": true, 2023-05-31T21:21:35.429 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket_index_max_shards": 11, 2023-05-31T21:21:35.429 INFO:teuthology.orchestra.run.smithi099.stdout: "read_only": false, 2023-05-31T21:21:35.429 INFO:teuthology.orchestra.run.smithi099.stdout: "tier_type": "", 2023-05-31T21:21:35.430 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_from_all": true, 2023-05-31T21:21:35.430 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_from": [], 2023-05-31T21:21:35.430 INFO:teuthology.orchestra.run.smithi099.stdout: "redirect_zone": "", 2023-05-31T21:21:35.430 INFO:teuthology.orchestra.run.smithi099.stdout: "supported_features": [ 2023-05-31T21:21:35.430 INFO:teuthology.orchestra.run.smithi099.stdout: "resharding" 2023-05-31T21:21:35.430 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2023-05-31T21:21:35.431 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T21:21:35.431 INFO:teuthology.orchestra.run.smithi099.stdout: ], 2023-05-31T21:21:35.431 INFO:teuthology.orchestra.run.smithi099.stdout: "placement_targets": [ 2023-05-31T21:21:35.431 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T21:21:35.431 INFO:teuthology.orchestra.run.smithi099.stdout: "name": "default-placement", 2023-05-31T21:21:35.431 INFO:teuthology.orchestra.run.smithi099.stdout: "tags": [], 2023-05-31T21:21:35.431 INFO:teuthology.orchestra.run.smithi099.stdout: "storage_classes": [ 2023-05-31T21:21:35.432 INFO:teuthology.orchestra.run.smithi099.stdout: "STANDARD" 2023-05-31T21:21:35.432 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2023-05-31T21:21:35.432 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T21:21:35.432 INFO:teuthology.orchestra.run.smithi099.stdout: ], 2023-05-31T21:21:35.432 INFO:teuthology.orchestra.run.smithi099.stdout: "default_placement": "default-placement", 2023-05-31T21:21:35.432 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_id": "aaa33de6-515b-4cac-8f1b-3a1a30452117", 2023-05-31T21:21:35.433 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_policy": { 2023-05-31T21:21:35.433 INFO:teuthology.orchestra.run.smithi099.stdout: "groups": [] 2023-05-31T21:21:35.433 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:21:35.433 INFO:teuthology.orchestra.run.smithi099.stdout: "enabled_features": [ 2023-05-31T21:21:35.433 INFO:teuthology.orchestra.run.smithi099.stdout: "resharding" 2023-05-31T21:21:35.433 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2023-05-31T21:21:35.433 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T21:21:35.434 INFO:teuthology.orchestra.run.smithi099.stdout: ], 2023-05-31T21:21:35.434 INFO:teuthology.orchestra.run.smithi099.stdout: "short_zone_ids": [ 2023-05-31T21:21:35.434 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T21:21:35.434 INFO:teuthology.orchestra.run.smithi099.stdout: "key": "3d163b1e-09c7-47e2-ae87-add4aace3f38", 2023-05-31T21:21:35.434 INFO:teuthology.orchestra.run.smithi099.stdout: "val": 3101833771 2023-05-31T21:21:35.434 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:21:35.435 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T21:21:35.435 INFO:teuthology.orchestra.run.smithi099.stdout: "key": "5d8acec7-7e57-4e56-8b32-e3d6b5036001", 2023-05-31T21:21:35.435 INFO:teuthology.orchestra.run.smithi099.stdout: "val": 4255530623 2023-05-31T21:21:35.435 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:21:35.436 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T21:21:35.436 INFO:teuthology.orchestra.run.smithi099.stdout: "key": "a6a3aab5-bc52-44a4-8543-5c5ea6056acd", 2023-05-31T21:21:35.438 INFO:teuthology.orchestra.run.smithi099.stdout: "val": 3708073710 2023-05-31T21:21:35.438 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:21:35.438 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T21:21:35.440 INFO:teuthology.orchestra.run.smithi099.stdout: "key": "f85d7236-2f27-4988-af26-9c66ab280b1b", 2023-05-31T21:21:35.440 INFO:teuthology.orchestra.run.smithi099.stdout: "val": 233395200 2023-05-31T21:21:35.441 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T21:21:35.441 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2023-05-31T21:21:35.441 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:21:35.441 INFO:teuthology.orchestra.run.smithi099.stdout: "master_zonegroup": "a", 2023-05-31T21:21:35.441 INFO:teuthology.orchestra.run.smithi099.stdout: "master_zone": "3d163b1e-09c7-47e2-ae87-add4aace3f38", 2023-05-31T21:21:35.441 INFO:teuthology.orchestra.run.smithi099.stdout: "period_config": { 2023-05-31T21:21:35.442 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket_quota": { 2023-05-31T21:21:35.442 INFO:teuthology.orchestra.run.smithi099.stdout: "enabled": false, 2023-05-31T21:21:35.442 INFO:teuthology.orchestra.run.smithi099.stdout: "check_on_raw": false, 2023-05-31T21:21:35.442 INFO:teuthology.orchestra.run.smithi099.stdout: "max_size": -1, 2023-05-31T21:21:35.442 INFO:teuthology.orchestra.run.smithi099.stdout: "max_size_kb": 0, 2023-05-31T21:21:35.442 INFO:teuthology.orchestra.run.smithi099.stdout: "max_objects": -1 2023-05-31T21:21:35.442 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:21:35.443 INFO:teuthology.orchestra.run.smithi099.stdout: "user_quota": { 2023-05-31T21:21:35.443 INFO:teuthology.orchestra.run.smithi099.stdout: "enabled": false, 2023-05-31T21:21:35.443 INFO:teuthology.orchestra.run.smithi099.stdout: "check_on_raw": false, 2023-05-31T21:21:35.443 INFO:teuthology.orchestra.run.smithi099.stdout: "max_size": -1, 2023-05-31T21:21:35.443 INFO:teuthology.orchestra.run.smithi099.stdout: "max_size_kb": 0, 2023-05-31T21:21:35.443 INFO:teuthology.orchestra.run.smithi099.stdout: "max_objects": -1 2023-05-31T21:21:35.443 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:21:35.444 INFO:teuthology.orchestra.run.smithi099.stdout: "user_ratelimit": { 2023-05-31T21:21:35.444 INFO:teuthology.orchestra.run.smithi099.stdout: "max_read_ops": 0, 2023-05-31T21:21:35.444 INFO:teuthology.orchestra.run.smithi099.stdout: "max_write_ops": 0, 2023-05-31T21:21:35.444 INFO:teuthology.orchestra.run.smithi099.stdout: "max_read_bytes": 0, 2023-05-31T21:21:35.445 INFO:teuthology.orchestra.run.smithi099.stdout: "max_write_bytes": 0, 2023-05-31T21:21:35.445 INFO:teuthology.orchestra.run.smithi099.stdout: "enabled": false 2023-05-31T21:21:35.445 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:21:35.446 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket_ratelimit": { 2023-05-31T21:21:35.446 INFO:teuthology.orchestra.run.smithi099.stdout: "max_read_ops": 0, 2023-05-31T21:21:35.446 INFO:teuthology.orchestra.run.smithi099.stdout: "max_write_ops": 0, 2023-05-31T21:21:35.446 INFO:teuthology.orchestra.run.smithi099.stdout: "max_read_bytes": 0, 2023-05-31T21:21:35.447 INFO:teuthology.orchestra.run.smithi099.stdout: "max_write_bytes": 0, 2023-05-31T21:21:35.447 INFO:teuthology.orchestra.run.smithi099.stdout: "enabled": false 2023-05-31T21:21:35.447 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:21:35.447 INFO:teuthology.orchestra.run.smithi099.stdout: "anonymous_ratelimit": { 2023-05-31T21:21:35.447 INFO:teuthology.orchestra.run.smithi099.stdout: "max_read_ops": 0, 2023-05-31T21:21:35.448 INFO:teuthology.orchestra.run.smithi099.stdout: "max_write_ops": 0, 2023-05-31T21:21:35.448 INFO:teuthology.orchestra.run.smithi099.stdout: "max_read_bytes": 0, 2023-05-31T21:21:35.448 INFO:teuthology.orchestra.run.smithi099.stdout: "max_write_bytes": 0, 2023-05-31T21:21:35.448 INFO:teuthology.orchestra.run.smithi099.stdout: "enabled": false 2023-05-31T21:21:35.449 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T21:21:35.449 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:21:35.449 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_id": "aaa33de6-515b-4cac-8f1b-3a1a30452117", 2023-05-31T21:21:35.449 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_name": "test-realm", 2023-05-31T21:21:35.449 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2023-05-31T21:21:35.449 INFO:teuthology.orchestra.run.smithi099.stdout:} 2023-05-31T21:21:35.450 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'info', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-05-31T21:21:35.451 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'info', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-05-31T21:21:35.451 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync info --cluster c1 --debug-rgw 0 --debug-ms 0 2023-05-31T21:21:35.502 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2023-05-31T21:21:35.502 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2023-05-31T21:21:35.611 INFO:teuthology.orchestra.run.smithi099.stdout:{"sources":[{"id":"sync-pipe","source":{"zone":"a2","bucket":"pklxkd-68"},"dest":{"zone":"a1","bucket":"pklxkd-68"},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":""}}],"dests":[{"id":"sync-pipe","source":{"zone":"a1","bucket":"pklxkd-68"},"dest":{"zone":"a2","bucket":"pklxkd-68"},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":""}}],"hints":{"sources":[],"dests":[]},"resolved-hints-1":{"sources":[],"dests":[]},"resolved-hints":{"sources":[],"dests":[]}} 2023-05-31T21:21:35.612 DEBUG:boto:path=/pklxkd-68/ 2023-05-31T21:21:35.613 DEBUG:boto:auth_path=/pklxkd-68/ 2023-05-31T21:21:35.614 DEBUG:boto:Method: HEAD 2023-05-31T21:21:35.615 DEBUG:boto:Path: /pklxkd-68/ 2023-05-31T21:21:35.616 DEBUG:boto:Data: 2023-05-31T21:21:35.617 DEBUG:boto:Headers: {} 2023-05-31T21:21:35.617 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T21:21:35.620 DEBUG:boto:Port: 8000 2023-05-31T21:21:35.621 DEBUG:boto:Params: {} 2023-05-31T21:21:35.621 DEBUG:boto:Token: None 2023-05-31T21:21:35.621 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 21:21:35 GMT /pklxkd-68/ 2023-05-31T21:21:35.622 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:waphac2IPrTsrsEibWZ+aErNyv0= 2023-05-31T21:21:35.622 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 21:21:35 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:waphac2IPrTsrsEibWZ+aErNyv0=', 'Content-Length': '0'} 2023-05-31T21:21:35.738 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000f3a604afe26736ee-006477ba5f-242d-a1'), ('Content-Length', '0'), ('Date', 'Wed, 31 May 2023 21:21:35 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:21:35.739 DEBUG:boto:path=/pklxkd-68/obj1 2023-05-31T21:21:35.739 DEBUG:boto:auth_path=/pklxkd-68/obj1 2023-05-31T21:21:35.739 DEBUG:boto:Method: PUT 2023-05-31T21:21:35.740 DEBUG:boto:Path: /pklxkd-68/obj1 2023-05-31T21:21:35.740 DEBUG:boto:Data: 2023-05-31T21:21:35.741 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2023-05-31T21:21:35.742 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T21:21:35.743 DEBUG:boto:Port: 8000 2023-05-31T21:21:35.744 DEBUG:boto:Params: {} 2023-05-31T21:21:35.745 DEBUG:boto:Token: None 2023-05-31T21:21:35.746 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Wed, 31 May 2023 21:21:35 GMT /pklxkd-68/obj1 2023-05-31T21:21:35.747 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:mVI/r+9qbAxPZXIal0tVAHngjH8= 2023-05-31T21:21:35.748 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Wed, 31 May 2023 21:21:35 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:mVI/r+9qbAxPZXIal0tVAHngjH8='} 2023-05-31T21:21:35.777 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000c2e7b72ae9b90a31-006477ba5f-242d-a1'), ('Date', 'Wed, 31 May 2023 21:21:35 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:21:35.778 DEBUG:boto:path=/pklxkd-68/ 2023-05-31T21:21:35.778 DEBUG:boto:auth_path=/pklxkd-68/ 2023-05-31T21:21:35.779 DEBUG:boto:Method: HEAD 2023-05-31T21:21:35.780 DEBUG:boto:Path: /pklxkd-68/ 2023-05-31T21:21:35.781 DEBUG:boto:Data: 2023-05-31T21:21:35.782 DEBUG:boto:Headers: {} 2023-05-31T21:21:35.783 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T21:21:35.784 DEBUG:boto:Port: 8000 2023-05-31T21:21:35.785 DEBUG:boto:Params: {} 2023-05-31T21:21:35.785 DEBUG:boto:Token: None 2023-05-31T21:21:35.785 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 21:21:35 GMT /pklxkd-68/ 2023-05-31T21:21:35.786 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:waphac2IPrTsrsEibWZ+aErNyv0= 2023-05-31T21:21:35.786 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 21:21:35 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:waphac2IPrTsrsEibWZ+aErNyv0=', 'Content-Length': '0'} 2023-05-31T21:21:35.790 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '6'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000017f85e330601fcd-006477ba5f-242d-a1'), ('Content-Length', '0'), ('Date', 'Wed, 31 May 2023 21:21:35 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:21:35.791 DEBUG:boto:path=/pklxkd-68/obj2 2023-05-31T21:21:35.792 DEBUG:boto:auth_path=/pklxkd-68/obj2 2023-05-31T21:21:35.792 DEBUG:boto:Method: PUT 2023-05-31T21:21:35.792 DEBUG:boto:Path: /pklxkd-68/obj2 2023-05-31T21:21:35.793 DEBUG:boto:Data: 2023-05-31T21:21:35.793 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2023-05-31T21:21:35.793 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T21:21:35.794 DEBUG:boto:Port: 8000 2023-05-31T21:21:35.794 DEBUG:boto:Params: {} 2023-05-31T21:21:35.795 DEBUG:boto:Token: None 2023-05-31T21:21:35.795 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Wed, 31 May 2023 21:21:35 GMT /pklxkd-68/obj2 2023-05-31T21:21:35.795 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:aHKs/QEIand6BqZjck4e1wt+6Bk= 2023-05-31T21:21:35.796 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Wed, 31 May 2023 21:21:35 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:aHKs/QEIand6BqZjck4e1wt+6Bk='} 2023-05-31T21:21:35.840 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000584ac59c48bd95e6-006477ba5f-242d-a1'), ('Date', 'Wed, 31 May 2023 21:21:35 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:21:35.841 DEBUG:boto:path=/pklxkd-68/ 2023-05-31T21:21:35.841 DEBUG:boto:auth_path=/pklxkd-68/ 2023-05-31T21:21:35.842 DEBUG:boto:Method: HEAD 2023-05-31T21:21:35.842 DEBUG:boto:Path: /pklxkd-68/ 2023-05-31T21:21:35.842 DEBUG:boto:Data: 2023-05-31T21:21:35.843 DEBUG:boto:Headers: {} 2023-05-31T21:21:35.843 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T21:21:35.843 DEBUG:boto:Port: 8000 2023-05-31T21:21:35.844 DEBUG:boto:Params: {} 2023-05-31T21:21:35.844 DEBUG:boto:Token: None 2023-05-31T21:21:35.845 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 21:21:35 GMT /pklxkd-68/ 2023-05-31T21:21:35.845 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:waphac2IPrTsrsEibWZ+aErNyv0= 2023-05-31T21:21:35.846 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 21:21:35 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:waphac2IPrTsrsEibWZ+aErNyv0=', 'Content-Length': '0'} 2023-05-31T21:21:35.849 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '2'), ('X-RGW-Bytes-Used', '12'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000aa561205481c805a-006477ba5f-242d-a1'), ('Content-Length', '0'), ('Date', 'Wed, 31 May 2023 21:21:35 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:21:35.850 DEBUG:boto:path=/pklxkd-68/obj3 2023-05-31T21:21:35.850 DEBUG:boto:auth_path=/pklxkd-68/obj3 2023-05-31T21:21:35.851 DEBUG:boto:Method: PUT 2023-05-31T21:21:35.851 DEBUG:boto:Path: /pklxkd-68/obj3 2023-05-31T21:21:35.851 DEBUG:boto:Data: 2023-05-31T21:21:35.852 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2023-05-31T21:21:35.852 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T21:21:35.853 DEBUG:boto:Port: 8000 2023-05-31T21:21:35.853 DEBUG:boto:Params: {} 2023-05-31T21:21:35.853 DEBUG:boto:Token: None 2023-05-31T21:21:35.854 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Wed, 31 May 2023 21:21:35 GMT /pklxkd-68/obj3 2023-05-31T21:21:35.854 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:Bn8vqAG+9LvasOYLKU5JX0URbBk= 2023-05-31T21:21:35.855 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Wed, 31 May 2023 21:21:35 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:Bn8vqAG+9LvasOYLKU5JX0URbBk='} 2023-05-31T21:21:35.874 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000058cc7d734f4fbf91-006477ba5f-242d-a1'), ('Date', 'Wed, 31 May 2023 21:21:35 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:21:35.875 DEBUG:boto:path=/pklxkd-69/ 2023-05-31T21:21:35.875 DEBUG:boto:auth_path=/pklxkd-69/ 2023-05-31T21:21:35.876 DEBUG:boto:Method: HEAD 2023-05-31T21:21:35.876 DEBUG:boto:Path: /pklxkd-69/ 2023-05-31T21:21:35.877 DEBUG:boto:Data: 2023-05-31T21:21:35.877 DEBUG:boto:Headers: {} 2023-05-31T21:21:35.877 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T21:21:35.878 DEBUG:boto:Port: 8000 2023-05-31T21:21:35.878 DEBUG:boto:Params: {} 2023-05-31T21:21:35.878 DEBUG:boto:Token: None 2023-05-31T21:21:35.879 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 21:21:35 GMT /pklxkd-69/ 2023-05-31T21:21:35.879 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:N0bfzuM83NgwBSzsFSC64cfJmzI= 2023-05-31T21:21:35.880 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 21:21:35 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:N0bfzuM83NgwBSzsFSC64cfJmzI=', 'Content-Length': '0'} 2023-05-31T21:21:35.889 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000001f28a495cdf8b6f8-006477ba5f-242d-a1'), ('Content-Length', '0'), ('Date', 'Wed, 31 May 2023 21:21:35 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:21:35.890 DEBUG:boto:path=/pklxkd-69/%5B%27obj1%27%2C%20%27obj2%27%2C%20%27obj3%27%5D 2023-05-31T21:21:35.890 DEBUG:boto:auth_path=/pklxkd-69/%5B%27obj1%27%2C%20%27obj2%27%2C%20%27obj3%27%5D 2023-05-31T21:21:35.891 DEBUG:boto:Method: PUT 2023-05-31T21:21:35.891 DEBUG:boto:Path: /pklxkd-69/%5B%27obj1%27%2C%20%27obj2%27%2C%20%27obj3%27%5D 2023-05-31T21:21:35.891 DEBUG:boto:Data: 2023-05-31T21:21:35.892 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2023-05-31T21:21:35.892 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T21:21:35.892 DEBUG:boto:Port: 8000 2023-05-31T21:21:35.893 DEBUG:boto:Params: {} 2023-05-31T21:21:35.893 DEBUG:boto:Token: None 2023-05-31T21:21:35.893 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Wed, 31 May 2023 21:21:35 GMT /pklxkd-69/%5B%27obj1%27%2C%20%27obj2%27%2C%20%27obj3%27%5D 2023-05-31T21:21:35.894 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:zKqYD/2vHmuBRT+s3/OBgHjg4A8= 2023-05-31T21:21:35.894 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Wed, 31 May 2023 21:21:35 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:zKqYD/2vHmuBRT+s3/OBgHjg4A8='} 2023-05-31T21:21:35.936 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000004f731815dc999073-006477ba5f-242d-a1'), ('Date', 'Wed, 31 May 2023 21:21:35 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:21:35.938 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-31T21:21:35.939 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-31T21:21:35.939 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-05-31T21:21:36.008 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2023-05-31T21:21:36.009 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2023-05-31T21:21:36.109 INFO:teuthology.orchestra.run.smithi099.stderr:No --period given, using current period=2e6123d2-d695-4350-9a01-61a455bc782b 2023-05-31T21:21:36.152 INFO:teuthology.orchestra.run.smithi099.stdout:[ 2023-05-31T21:21:36.153 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T21:21:36.153 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1685566501.757183_260.1", 2023-05-31T21:21:36.153 INFO:teuthology.orchestra.run.smithi099.stdout: "last_update": "2023-05-31T20:55:01.757183Z" 2023-05-31T21:21:36.153 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:21:36.154 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T21:21:36.154 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1685566501.735717_251.1", 2023-05-31T21:21:36.154 INFO:teuthology.orchestra.run.smithi099.stdout: "last_update": "2023-05-31T20:55:01.735717Z" 2023-05-31T21:21:36.154 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:21:36.154 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T21:21:36.154 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1685562970.274572_93.1", 2023-05-31T21:21:36.155 INFO:teuthology.orchestra.run.smithi099.stdout: "last_update": "2023-05-31T19:56:10.274572Z" 2023-05-31T21:21:36.155 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:21:36.155 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T21:21:36.155 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1685568089.268814_960.1", 2023-05-31T21:21:36.156 INFO:teuthology.orchestra.run.smithi099.stdout: "last_update": "2023-05-31T21:21:29.268814Z" 2023-05-31T21:21:36.156 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T21:21:36.156 INFO:teuthology.orchestra.run.smithi099.stdout:] 2023-05-31T21:21:36.157 DEBUG:rgw_multi.tests:master meta markers={0: '1_1685566501.757183_260.1', 1: '1_1685566501.735717_251.1', 2: '1_1685562970.274572_93.1', 3: '1_1685568089.268814_960.1'} 2023-05-31T21:21:36.157 INFO:rgw_multi.tests:starting meta checkpoint for zone=a2 2023-05-31T21:21:36.158 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-31T21:21:36.159 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-31T21:21:36.159 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-05-31T21:21:36.212 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2023-05-31T21:21:36.213 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2023-05-31T21:21:36.266 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T21:21:36.261+0000 7f899ad53780 20 RGW-SYNC:meta: read sync status 2023-05-31T21:21:36.267 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T21:21:36.263+0000 7f899ad53780 20 RGW-SYNC:meta: read sync status 2023-05-31T21:21:36.307 INFO:teuthology.orchestra.run.smithi099.stdout:{ 2023-05-31T21:21:36.308 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_status": { 2023-05-31T21:21:36.308 INFO:teuthology.orchestra.run.smithi099.stdout: "info": { 2023-05-31T21:21:36.308 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "sync", 2023-05-31T21:21:36.308 INFO:teuthology.orchestra.run.smithi099.stdout: "num_shards": 4, 2023-05-31T21:21:36.309 INFO:teuthology.orchestra.run.smithi099.stdout: "period": "2e6123d2-d695-4350-9a01-61a455bc782b", 2023-05-31T21:21:36.309 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2023-05-31T21:21:36.309 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:21:36.309 INFO:teuthology.orchestra.run.smithi099.stdout: "markers": [ 2023-05-31T21:21:36.309 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T21:21:36.309 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 0, 2023-05-31T21:21:36.309 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T21:21:36.310 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2023-05-31T21:21:36.310 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1685566501.757183_260.1", 2023-05-31T21:21:36.310 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T21:21:36.310 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 1, 2023-05-31T21:21:36.310 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T21:21:36.310 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2023-05-31T20:55:01.757183Z", 2023-05-31T21:21:36.311 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2023-05-31T21:21:36.311 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T21:21:36.311 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:21:36.311 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T21:21:36.311 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 1, 2023-05-31T21:21:36.311 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T21:21:36.311 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2023-05-31T21:21:36.312 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1685566501.735717_251.1", 2023-05-31T21:21:36.312 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T21:21:36.312 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2023-05-31T21:21:36.312 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T21:21:36.312 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2023-05-31T20:55:01.735717Z", 2023-05-31T21:21:36.312 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2023-05-31T21:21:36.312 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T21:21:36.313 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:21:36.313 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T21:21:36.313 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 2, 2023-05-31T21:21:36.313 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T21:21:36.313 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2023-05-31T21:21:36.313 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1685562970.274572_93.1", 2023-05-31T21:21:36.313 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T21:21:36.314 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2023-05-31T21:21:36.314 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T21:21:36.314 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2023-05-31T19:56:10.274572Z", 2023-05-31T21:21:36.314 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2023-05-31T21:21:36.314 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T21:21:36.314 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:21:36.314 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T21:21:36.315 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 3, 2023-05-31T21:21:36.315 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T21:21:36.315 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2023-05-31T21:21:36.315 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1685568089.268814_960.1", 2023-05-31T21:21:36.315 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T21:21:36.315 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2023-05-31T21:21:36.316 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T21:21:36.316 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2023-05-31T21:21:29.268814Z", 2023-05-31T21:21:36.316 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2023-05-31T21:21:36.316 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T21:21:36.316 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T21:21:36.316 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2023-05-31T21:21:36.316 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:21:36.317 INFO:teuthology.orchestra.run.smithi099.stdout: "full_sync": { 2023-05-31T21:21:36.317 INFO:teuthology.orchestra.run.smithi099.stdout: "total": 1, 2023-05-31T21:21:36.317 INFO:teuthology.orchestra.run.smithi099.stdout: "complete": 1 2023-05-31T21:21:36.317 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:21:36.317 INFO:teuthology.orchestra.run.smithi099.stdout: "current_time": "2023-05-31T21:21:36Z" 2023-05-31T21:21:36.317 INFO:teuthology.orchestra.run.smithi099.stdout:} 2023-05-31T21:21:36.318 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "2e6123d2-d695-4350-9a01-61a455bc782b", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1685566501.757183_260.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-05-31T20:55:01.757183Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1685566501.735717_251.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-05-31T20:55:01.735717Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1685562970.274572_93.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-05-31T19:56:10.274572Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1685568089.268814_960.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-05-31T21:21:29.268814Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-05-31T21:21:36Z" } 2023-05-31T21:21:36.319 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1685566501.757183_260.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-05-31T20:55:01.757183Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1685566501.735717_251.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-05-31T20:55:01.735717Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1685562970.274572_93.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-05-31T19:56:10.274572Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1685568089.268814_960.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-05-31T21:21:29.268814Z', 'realm_epoch': 2}}] 2023-05-31T21:21:36.319 DEBUG:rgw_multi.tests:log_status={0: '1_1685566501.757183_260.1', 1: '1_1685566501.735717_251.1', 2: '1_1685562970.274572_93.1', 3: '1_1685568089.268814_960.1'} 2023-05-31T21:21:36.319 DEBUG:rgw_multi.tests:sync_status={0: '1_1685566501.757183_260.1', 1: '1_1685566501.735717_251.1', 2: '1_1685562970.274572_93.1', 3: '1_1685568089.268814_960.1'} 2023-05-31T21:21:36.320 INFO:rgw_multi.tests:finish meta checkpoint for zone=a2 2023-05-31T21:21:36.320 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['datalog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-31T21:21:36.321 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'datalog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-31T21:21:36.321 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 datalog status --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-05-31T21:21:36.376 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2023-05-31T21:21:36.376 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2023-05-31T21:21:36.479 INFO:teuthology.orchestra.run.smithi099.stdout:[ 2023-05-31T21:21:36.479 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T21:21:36.479 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "00000000000000000000:00000000000000044955", 2023-05-31T21:21:36.479 INFO:teuthology.orchestra.run.smithi099.stdout: "last_update": "2023-05-31T21:21:35.807414Z" 2023-05-31T21:21:36.479 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:21:36.480 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T21:21:36.480 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "00000000000000000000:00000000000000031499", 2023-05-31T21:21:36.480 INFO:teuthology.orchestra.run.smithi099.stdout: "last_update": "2023-05-31T21:12:40.667485Z" 2023-05-31T21:21:36.480 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:21:36.480 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T21:21:36.481 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "00000000000000000000:00000000000000051111", 2023-05-31T21:21:36.481 INFO:teuthology.orchestra.run.smithi099.stdout: "last_update": "2023-05-31T21:21:35.866368Z" 2023-05-31T21:21:36.481 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:21:36.481 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T21:21:36.482 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "00000000000000000000:00000000000000044038", 2023-05-31T21:21:36.482 INFO:teuthology.orchestra.run.smithi099.stdout: "last_update": "2023-05-31T21:21:35.912562Z" 2023-05-31T21:21:36.482 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T21:21:36.482 INFO:teuthology.orchestra.run.smithi099.stdout:] 2023-05-31T21:21:36.483 DEBUG:rgw_multi.tests:data markers for zone=a1 markers={0: '00000000000000000000:00000000000000044955', 1: '00000000000000000000:00000000000000031499', 2: '00000000000000000000:00000000000000051111', 3: '00000000000000000000:00000000000000044038'} 2023-05-31T21:21:36.483 INFO:rgw_multi.tests:starting data checkpoint for target_zone=a2 source_zone=a1 2023-05-31T21:21:36.484 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-31T21:21:36.484 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-31T21:21:36.485 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-05-31T21:21:36.541 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2023-05-31T21:21:36.541 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2023-05-31T21:21:36.637 INFO:teuthology.orchestra.run.smithi099.stdout:{ 2023-05-31T21:21:36.637 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_status": { 2023-05-31T21:21:36.637 INFO:teuthology.orchestra.run.smithi099.stdout: "info": { 2023-05-31T21:21:36.638 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "sync", 2023-05-31T21:21:36.638 INFO:teuthology.orchestra.run.smithi099.stdout: "num_shards": 4, 2023-05-31T21:21:36.638 INFO:teuthology.orchestra.run.smithi099.stdout: "instance_id": 452136274422638824 2023-05-31T21:21:36.638 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:21:36.638 INFO:teuthology.orchestra.run.smithi099.stdout: "markers": [ 2023-05-31T21:21:36.638 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T21:21:36.638 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 0, 2023-05-31T21:21:36.639 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T21:21:36.639 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "incremental-sync", 2023-05-31T21:21:36.639 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "00000000000000000000:00000000000000044815", 2023-05-31T21:21:36.639 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T21:21:36.639 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2023-05-31T21:21:36.639 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T21:21:36.640 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2023-05-31T21:21:05.078544Z" 2023-05-31T21:21:36.640 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T21:21:36.640 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:21:36.640 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T21:21:36.640 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 1, 2023-05-31T21:21:36.641 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T21:21:36.641 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "incremental-sync", 2023-05-31T21:21:36.641 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "00000000000000000000:00000000000000031499", 2023-05-31T21:21:36.641 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T21:21:36.641 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2023-05-31T21:21:36.641 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T21:21:36.641 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2023-05-31T21:12:40.667485Z" 2023-05-31T21:21:36.642 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T21:21:36.642 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:21:36.642 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T21:21:36.642 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 2, 2023-05-31T21:21:36.642 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T21:21:36.643 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "incremental-sync", 2023-05-31T21:21:36.643 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "00000000000000000000:00000000000000050831", 2023-05-31T21:21:36.643 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T21:21:36.643 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 55, 2023-05-31T21:21:36.643 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T21:21:36.643 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2023-05-31T21:19:03.876722Z" 2023-05-31T21:21:36.644 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T21:21:36.644 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:21:36.644 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T21:21:36.644 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 3, 2023-05-31T21:21:36.644 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T21:21:36.644 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "incremental-sync", 2023-05-31T21:21:36.644 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "00000000000000000000:00000000000000043896", 2023-05-31T21:21:36.645 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T21:21:36.645 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 9, 2023-05-31T21:21:36.645 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T21:21:36.645 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2023-05-31T21:12:28.529956Z" 2023-05-31T21:21:36.645 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T21:21:36.645 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T21:21:36.645 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2023-05-31T21:21:36.646 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:21:36.646 INFO:teuthology.orchestra.run.smithi099.stdout: "full_sync": { 2023-05-31T21:21:36.646 INFO:teuthology.orchestra.run.smithi099.stdout: "total": 64, 2023-05-31T21:21:36.646 INFO:teuthology.orchestra.run.smithi099.stdout: "complete": 64 2023-05-31T21:21:36.646 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:21:36.646 INFO:teuthology.orchestra.run.smithi099.stdout: "current_time": "2023-05-31T21:21:36Z" 2023-05-31T21:21:36.647 INFO:teuthology.orchestra.run.smithi099.stdout:} 2023-05-31T21:21:36.647 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 452136274422638824 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000044815", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-05-31T21:21:05.078544Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000031499", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-05-31T21:12:40.667485Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000050831", "next_step_marker": "", "total_entries": 55, "pos": 0, "timestamp": "2023-05-31T21:19:03.876722Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000043896", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2023-05-31T21:12:28.529956Z" } } ] }, "full_sync": { "total": 64, "complete": 64 }, "current_time": "2023-05-31T21:21:36Z" } 2023-05-31T21:21:36.648 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000044815', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-05-31T21:21:05.078544Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000031499', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-05-31T21:12:40.667485Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000050831', 'next_step_marker': '', 'total_entries': 55, 'pos': 0, 'timestamp': '2023-05-31T21:19:03.876722Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000043896', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2023-05-31T21:12:28.529956Z'}}] 2023-05-31T21:21:36.649 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000044955', 1: '00000000000000000000:00000000000000031499', 2: '00000000000000000000:00000000000000051111', 3: '00000000000000000000:00000000000000044038'} 2023-05-31T21:21:36.649 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000044815', 1: '00000000000000000000:00000000000000031499', 2: '00000000000000000000:00000000000000050831', 3: '00000000000000000000:00000000000000043896'} 2023-05-31T21:21:36.649 WARNING:rgw_multi.tests:data of zone a2 behind zone a1: shard=0 master=00000000000000000000:00000000000000044955 target=00000000000000000000:00000000000000044815, shard=2 master=00000000000000000000:00000000000000051111 target=00000000000000000000:00000000000000050831, shard=3 master=00000000000000000000:00000000000000044038 target=00000000000000000000:00000000000000043896 2023-05-31T21:21:41.650 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-31T21:21:41.651 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-31T21:21:41.651 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-05-31T21:21:41.710 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2023-05-31T21:21:41.711 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2023-05-31T21:21:41.784 INFO:teuthology.orchestra.run.smithi099.stdout:{ 2023-05-31T21:21:41.784 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_status": { 2023-05-31T21:21:41.784 INFO:teuthology.orchestra.run.smithi099.stdout: "info": { 2023-05-31T21:21:41.784 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "sync", 2023-05-31T21:21:41.785 INFO:teuthology.orchestra.run.smithi099.stdout: "num_shards": 4, 2023-05-31T21:21:41.785 INFO:teuthology.orchestra.run.smithi099.stdout: "instance_id": 452136274422638824 2023-05-31T21:21:41.785 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:21:41.785 INFO:teuthology.orchestra.run.smithi099.stdout: "markers": [ 2023-05-31T21:21:41.785 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T21:21:41.786 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 0, 2023-05-31T21:21:41.786 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T21:21:41.786 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "incremental-sync", 2023-05-31T21:21:41.786 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "00000000000000000000:00000000000000044815", 2023-05-31T21:21:41.787 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T21:21:41.787 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2023-05-31T21:21:41.787 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T21:21:41.787 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2023-05-31T21:21:05.078544Z" 2023-05-31T21:21:41.787 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T21:21:41.787 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:21:41.787 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T21:21:41.788 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 1, 2023-05-31T21:21:41.788 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T21:21:41.788 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "incremental-sync", 2023-05-31T21:21:41.788 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "00000000000000000000:00000000000000031499", 2023-05-31T21:21:41.788 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T21:21:41.788 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2023-05-31T21:21:41.789 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T21:21:41.789 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2023-05-31T21:12:40.667485Z" 2023-05-31T21:21:41.789 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T21:21:41.789 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:21:41.789 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T21:21:41.789 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 2, 2023-05-31T21:21:41.790 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T21:21:41.790 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "incremental-sync", 2023-05-31T21:21:41.790 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "00000000000000000000:00000000000000050831", 2023-05-31T21:21:41.790 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T21:21:41.790 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 55, 2023-05-31T21:21:41.790 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T21:21:41.790 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2023-05-31T21:19:03.876722Z" 2023-05-31T21:21:41.791 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T21:21:41.791 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:21:41.791 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T21:21:41.792 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 3, 2023-05-31T21:21:41.792 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T21:21:41.792 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "incremental-sync", 2023-05-31T21:21:41.792 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "00000000000000000000:00000000000000043896", 2023-05-31T21:21:41.792 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T21:21:41.792 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 9, 2023-05-31T21:21:41.792 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T21:21:41.793 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2023-05-31T21:12:28.529956Z" 2023-05-31T21:21:41.793 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T21:21:41.793 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T21:21:41.793 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2023-05-31T21:21:41.793 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:21:41.793 INFO:teuthology.orchestra.run.smithi099.stdout: "full_sync": { 2023-05-31T21:21:41.793 INFO:teuthology.orchestra.run.smithi099.stdout: "total": 64, 2023-05-31T21:21:41.794 INFO:teuthology.orchestra.run.smithi099.stdout: "complete": 64 2023-05-31T21:21:41.794 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:21:41.794 INFO:teuthology.orchestra.run.smithi099.stdout: "current_time": "2023-05-31T21:21:41Z" 2023-05-31T21:21:41.794 INFO:teuthology.orchestra.run.smithi099.stdout:} 2023-05-31T21:21:41.795 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 452136274422638824 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000044815", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-05-31T21:21:05.078544Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000031499", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-05-31T21:12:40.667485Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000050831", "next_step_marker": "", "total_entries": 55, "pos": 0, "timestamp": "2023-05-31T21:19:03.876722Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000043896", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2023-05-31T21:12:28.529956Z" } } ] }, "full_sync": { "total": 64, "complete": 64 }, "current_time": "2023-05-31T21:21:41Z" } 2023-05-31T21:21:41.796 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000044815', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-05-31T21:21:05.078544Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000031499', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-05-31T21:12:40.667485Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000050831', 'next_step_marker': '', 'total_entries': 55, 'pos': 0, 'timestamp': '2023-05-31T21:19:03.876722Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000043896', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2023-05-31T21:12:28.529956Z'}}] 2023-05-31T21:21:41.796 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000044955', 1: '00000000000000000000:00000000000000031499', 2: '00000000000000000000:00000000000000051111', 3: '00000000000000000000:00000000000000044038'} 2023-05-31T21:21:41.796 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000044815', 1: '00000000000000000000:00000000000000031499', 2: '00000000000000000000:00000000000000050831', 3: '00000000000000000000:00000000000000043896'} 2023-05-31T21:21:41.797 WARNING:rgw_multi.tests:data of zone a2 behind zone a1: shard=0 master=00000000000000000000:00000000000000044955 target=00000000000000000000:00000000000000044815, shard=2 master=00000000000000000000:00000000000000051111 target=00000000000000000000:00000000000000050831, shard=3 master=00000000000000000000:00000000000000044038 target=00000000000000000000:00000000000000043896 2023-05-31T21:21:46.798 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-31T21:21:46.799 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-31T21:21:46.799 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-05-31T21:21:46.861 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2023-05-31T21:21:46.862 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2023-05-31T21:21:46.912 INFO:teuthology.orchestra.run.smithi099.stdout:{ 2023-05-31T21:21:46.913 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_status": { 2023-05-31T21:21:46.913 INFO:teuthology.orchestra.run.smithi099.stdout: "info": { 2023-05-31T21:21:46.913 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "sync", 2023-05-31T21:21:46.913 INFO:teuthology.orchestra.run.smithi099.stdout: "num_shards": 4, 2023-05-31T21:21:46.913 INFO:teuthology.orchestra.run.smithi099.stdout: "instance_id": 452136274422638824 2023-05-31T21:21:46.914 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:21:46.914 INFO:teuthology.orchestra.run.smithi099.stdout: "markers": [ 2023-05-31T21:21:46.914 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T21:21:46.914 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 0, 2023-05-31T21:21:46.914 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T21:21:46.915 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "incremental-sync", 2023-05-31T21:21:46.915 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "00000000000000000000:00000000000000044815", 2023-05-31T21:21:46.915 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T21:21:46.915 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2023-05-31T21:21:46.915 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T21:21:46.915 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2023-05-31T21:21:05.078544Z" 2023-05-31T21:21:46.916 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T21:21:46.916 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:21:46.916 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T21:21:46.916 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 1, 2023-05-31T21:21:46.916 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T21:21:46.917 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "incremental-sync", 2023-05-31T21:21:46.917 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "00000000000000000000:00000000000000031499", 2023-05-31T21:21:46.917 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T21:21:46.918 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2023-05-31T21:21:46.918 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T21:21:46.918 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2023-05-31T21:12:40.667485Z" 2023-05-31T21:21:46.919 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T21:21:46.919 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:21:46.919 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T21:21:46.919 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 2, 2023-05-31T21:21:46.920 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T21:21:46.920 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "incremental-sync", 2023-05-31T21:21:46.921 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "00000000000000000000:00000000000000050831", 2023-05-31T21:21:46.921 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T21:21:46.922 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 55, 2023-05-31T21:21:46.922 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T21:21:46.922 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2023-05-31T21:19:03.876722Z" 2023-05-31T21:21:46.922 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T21:21:46.922 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:21:46.922 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T21:21:46.923 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 3, 2023-05-31T21:21:46.923 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T21:21:46.923 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "incremental-sync", 2023-05-31T21:21:46.923 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "00000000000000000000:00000000000000043896", 2023-05-31T21:21:46.923 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T21:21:46.923 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 9, 2023-05-31T21:21:46.923 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T21:21:46.924 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2023-05-31T21:12:28.529956Z" 2023-05-31T21:21:46.924 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T21:21:46.924 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T21:21:46.924 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2023-05-31T21:21:46.924 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:21:46.924 INFO:teuthology.orchestra.run.smithi099.stdout: "full_sync": { 2023-05-31T21:21:46.925 INFO:teuthology.orchestra.run.smithi099.stdout: "total": 64, 2023-05-31T21:21:46.925 INFO:teuthology.orchestra.run.smithi099.stdout: "complete": 64 2023-05-31T21:21:46.925 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:21:46.925 INFO:teuthology.orchestra.run.smithi099.stdout: "current_time": "2023-05-31T21:21:46Z" 2023-05-31T21:21:46.925 INFO:teuthology.orchestra.run.smithi099.stdout:} 2023-05-31T21:21:46.926 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 452136274422638824 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000044815", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-05-31T21:21:05.078544Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000031499", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-05-31T21:12:40.667485Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000050831", "next_step_marker": "", "total_entries": 55, "pos": 0, "timestamp": "2023-05-31T21:19:03.876722Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000043896", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2023-05-31T21:12:28.529956Z" } } ] }, "full_sync": { "total": 64, "complete": 64 }, "current_time": "2023-05-31T21:21:46Z" } 2023-05-31T21:21:46.927 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000044815', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-05-31T21:21:05.078544Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000031499', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-05-31T21:12:40.667485Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000050831', 'next_step_marker': '', 'total_entries': 55, 'pos': 0, 'timestamp': '2023-05-31T21:19:03.876722Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000043896', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2023-05-31T21:12:28.529956Z'}}] 2023-05-31T21:21:46.927 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000044955', 1: '00000000000000000000:00000000000000031499', 2: '00000000000000000000:00000000000000051111', 3: '00000000000000000000:00000000000000044038'} 2023-05-31T21:21:46.928 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000044815', 1: '00000000000000000000:00000000000000031499', 2: '00000000000000000000:00000000000000050831', 3: '00000000000000000000:00000000000000043896'} 2023-05-31T21:21:46.928 WARNING:rgw_multi.tests:data of zone a2 behind zone a1: shard=0 master=00000000000000000000:00000000000000044955 target=00000000000000000000:00000000000000044815, shard=2 master=00000000000000000000:00000000000000051111 target=00000000000000000000:00000000000000050831, shard=3 master=00000000000000000000:00000000000000044038 target=00000000000000000000:00000000000000043896 2023-05-31T21:21:51.930 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-31T21:21:51.931 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-31T21:21:51.932 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-05-31T21:21:51.993 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2023-05-31T21:21:51.993 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2023-05-31T21:21:52.057 INFO:teuthology.orchestra.run.smithi099.stdout:{ 2023-05-31T21:21:52.057 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_status": { 2023-05-31T21:21:52.057 INFO:teuthology.orchestra.run.smithi099.stdout: "info": { 2023-05-31T21:21:52.057 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "sync", 2023-05-31T21:21:52.058 INFO:teuthology.orchestra.run.smithi099.stdout: "num_shards": 4, 2023-05-31T21:21:52.058 INFO:teuthology.orchestra.run.smithi099.stdout: "instance_id": 452136274422638824 2023-05-31T21:21:52.058 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:21:52.058 INFO:teuthology.orchestra.run.smithi099.stdout: "markers": [ 2023-05-31T21:21:52.058 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T21:21:52.058 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 0, 2023-05-31T21:21:52.059 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T21:21:52.059 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "incremental-sync", 2023-05-31T21:21:52.059 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "00000000000000000000:00000000000000044815", 2023-05-31T21:21:52.059 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T21:21:52.059 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2023-05-31T21:21:52.059 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T21:21:52.059 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2023-05-31T21:21:05.078544Z" 2023-05-31T21:21:52.060 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T21:21:52.060 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:21:52.060 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T21:21:52.060 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 1, 2023-05-31T21:21:52.060 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T21:21:52.060 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "incremental-sync", 2023-05-31T21:21:52.061 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "00000000000000000000:00000000000000031499", 2023-05-31T21:21:52.061 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T21:21:52.061 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2023-05-31T21:21:52.061 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T21:21:52.061 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2023-05-31T21:12:40.667485Z" 2023-05-31T21:21:52.061 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T21:21:52.061 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:21:52.062 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T21:21:52.062 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 2, 2023-05-31T21:21:52.062 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T21:21:52.062 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "incremental-sync", 2023-05-31T21:21:52.062 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "00000000000000000000:00000000000000050831", 2023-05-31T21:21:52.062 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T21:21:52.062 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 55, 2023-05-31T21:21:52.063 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T21:21:52.064 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2023-05-31T21:19:03.876722Z" 2023-05-31T21:21:52.064 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T21:21:52.064 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:21:52.064 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T21:21:52.064 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 3, 2023-05-31T21:21:52.065 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T21:21:52.065 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "incremental-sync", 2023-05-31T21:21:52.065 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "00000000000000000000:00000000000000043896", 2023-05-31T21:21:52.065 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T21:21:52.065 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 9, 2023-05-31T21:21:52.066 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T21:21:52.066 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2023-05-31T21:12:28.529956Z" 2023-05-31T21:21:52.066 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T21:21:52.066 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T21:21:52.066 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2023-05-31T21:21:52.066 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:21:52.066 INFO:teuthology.orchestra.run.smithi099.stdout: "full_sync": { 2023-05-31T21:21:52.067 INFO:teuthology.orchestra.run.smithi099.stdout: "total": 64, 2023-05-31T21:21:52.067 INFO:teuthology.orchestra.run.smithi099.stdout: "complete": 64 2023-05-31T21:21:52.067 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:21:52.067 INFO:teuthology.orchestra.run.smithi099.stdout: "current_time": "2023-05-31T21:21:52Z" 2023-05-31T21:21:52.067 INFO:teuthology.orchestra.run.smithi099.stdout:} 2023-05-31T21:21:52.068 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 452136274422638824 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000044815", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-05-31T21:21:05.078544Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000031499", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-05-31T21:12:40.667485Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000050831", "next_step_marker": "", "total_entries": 55, "pos": 0, "timestamp": "2023-05-31T21:19:03.876722Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000043896", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2023-05-31T21:12:28.529956Z" } } ] }, "full_sync": { "total": 64, "complete": 64 }, "current_time": "2023-05-31T21:21:52Z" } 2023-05-31T21:21:52.069 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000044815', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-05-31T21:21:05.078544Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000031499', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-05-31T21:12:40.667485Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000050831', 'next_step_marker': '', 'total_entries': 55, 'pos': 0, 'timestamp': '2023-05-31T21:19:03.876722Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000043896', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2023-05-31T21:12:28.529956Z'}}] 2023-05-31T21:21:52.069 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000044955', 1: '00000000000000000000:00000000000000031499', 2: '00000000000000000000:00000000000000051111', 3: '00000000000000000000:00000000000000044038'} 2023-05-31T21:21:52.069 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000044815', 1: '00000000000000000000:00000000000000031499', 2: '00000000000000000000:00000000000000050831', 3: '00000000000000000000:00000000000000043896'} 2023-05-31T21:21:52.070 WARNING:rgw_multi.tests:data of zone a2 behind zone a1: shard=0 master=00000000000000000000:00000000000000044955 target=00000000000000000000:00000000000000044815, shard=2 master=00000000000000000000:00000000000000051111 target=00000000000000000000:00000000000000050831, shard=3 master=00000000000000000000:00000000000000044038 target=00000000000000000000:00000000000000043896 2023-05-31T21:21:57.071 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-31T21:21:57.072 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-31T21:21:57.072 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-05-31T21:21:57.132 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2023-05-31T21:21:57.132 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2023-05-31T21:21:57.195 INFO:teuthology.orchestra.run.smithi099.stdout:{ 2023-05-31T21:21:57.196 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_status": { 2023-05-31T21:21:57.196 INFO:teuthology.orchestra.run.smithi099.stdout: "info": { 2023-05-31T21:21:57.197 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "sync", 2023-05-31T21:21:57.197 INFO:teuthology.orchestra.run.smithi099.stdout: "num_shards": 4, 2023-05-31T21:21:57.197 INFO:teuthology.orchestra.run.smithi099.stdout: "instance_id": 452136274422638824 2023-05-31T21:21:57.198 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:21:57.198 INFO:teuthology.orchestra.run.smithi099.stdout: "markers": [ 2023-05-31T21:21:57.199 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T21:21:57.199 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 0, 2023-05-31T21:21:57.200 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T21:21:57.200 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "incremental-sync", 2023-05-31T21:21:57.200 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "00000000000000000000:00000000000000044815", 2023-05-31T21:21:57.201 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T21:21:57.201 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2023-05-31T21:21:57.201 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T21:21:57.202 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2023-05-31T21:21:05.078544Z" 2023-05-31T21:21:57.202 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T21:21:57.202 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:21:57.203 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T21:21:57.203 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 1, 2023-05-31T21:21:57.203 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T21:21:57.203 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "incremental-sync", 2023-05-31T21:21:57.204 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "00000000000000000000:00000000000000031499", 2023-05-31T21:21:57.204 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T21:21:57.204 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2023-05-31T21:21:57.205 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T21:21:57.205 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2023-05-31T21:12:40.667485Z" 2023-05-31T21:21:57.205 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T21:21:57.206 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:21:57.206 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T21:21:57.206 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 2, 2023-05-31T21:21:57.207 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T21:21:57.207 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "incremental-sync", 2023-05-31T21:21:57.208 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "00000000000000000000:00000000000000050831", 2023-05-31T21:21:57.208 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T21:21:57.209 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 55, 2023-05-31T21:21:57.209 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T21:21:57.209 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2023-05-31T21:19:03.876722Z" 2023-05-31T21:21:57.209 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T21:21:57.209 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:21:57.210 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T21:21:57.210 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 3, 2023-05-31T21:21:57.210 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T21:21:57.211 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "incremental-sync", 2023-05-31T21:21:57.211 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "00000000000000000000:00000000000000043896", 2023-05-31T21:21:57.211 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T21:21:57.211 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 9, 2023-05-31T21:21:57.211 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T21:21:57.211 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2023-05-31T21:12:28.529956Z" 2023-05-31T21:21:57.211 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T21:21:57.212 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T21:21:57.212 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2023-05-31T21:21:57.212 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:21:57.212 INFO:teuthology.orchestra.run.smithi099.stdout: "full_sync": { 2023-05-31T21:21:57.212 INFO:teuthology.orchestra.run.smithi099.stdout: "total": 64, 2023-05-31T21:21:57.212 INFO:teuthology.orchestra.run.smithi099.stdout: "complete": 64 2023-05-31T21:21:57.213 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:21:57.213 INFO:teuthology.orchestra.run.smithi099.stdout: "current_time": "2023-05-31T21:21:57Z" 2023-05-31T21:21:57.213 INFO:teuthology.orchestra.run.smithi099.stdout:} 2023-05-31T21:21:57.214 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 452136274422638824 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000044815", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-05-31T21:21:05.078544Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000031499", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-05-31T21:12:40.667485Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000050831", "next_step_marker": "", "total_entries": 55, "pos": 0, "timestamp": "2023-05-31T21:19:03.876722Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000043896", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2023-05-31T21:12:28.529956Z" } } ] }, "full_sync": { "total": 64, "complete": 64 }, "current_time": "2023-05-31T21:21:57Z" } 2023-05-31T21:21:57.214 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000044815', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-05-31T21:21:05.078544Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000031499', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-05-31T21:12:40.667485Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000050831', 'next_step_marker': '', 'total_entries': 55, 'pos': 0, 'timestamp': '2023-05-31T21:19:03.876722Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000043896', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2023-05-31T21:12:28.529956Z'}}] 2023-05-31T21:21:57.215 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000044955', 1: '00000000000000000000:00000000000000031499', 2: '00000000000000000000:00000000000000051111', 3: '00000000000000000000:00000000000000044038'} 2023-05-31T21:21:57.215 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000044815', 1: '00000000000000000000:00000000000000031499', 2: '00000000000000000000:00000000000000050831', 3: '00000000000000000000:00000000000000043896'} 2023-05-31T21:21:57.215 WARNING:rgw_multi.tests:data of zone a2 behind zone a1: shard=0 master=00000000000000000000:00000000000000044955 target=00000000000000000000:00000000000000044815, shard=2 master=00000000000000000000:00000000000000051111 target=00000000000000000000:00000000000000050831, shard=3 master=00000000000000000000:00000000000000044038 target=00000000000000000000:00000000000000043896 2023-05-31T21:22:02.217 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-31T21:22:02.217 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-31T21:22:02.218 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-05-31T21:22:02.279 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2023-05-31T21:22:02.279 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2023-05-31T21:22:02.347 INFO:teuthology.orchestra.run.smithi099.stdout:{ 2023-05-31T21:22:02.347 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_status": { 2023-05-31T21:22:02.347 INFO:teuthology.orchestra.run.smithi099.stdout: "info": { 2023-05-31T21:22:02.348 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "sync", 2023-05-31T21:22:02.348 INFO:teuthology.orchestra.run.smithi099.stdout: "num_shards": 4, 2023-05-31T21:22:02.348 INFO:teuthology.orchestra.run.smithi099.stdout: "instance_id": 452136274422638824 2023-05-31T21:22:02.348 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:22:02.348 INFO:teuthology.orchestra.run.smithi099.stdout: "markers": [ 2023-05-31T21:22:02.348 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T21:22:02.349 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 0, 2023-05-31T21:22:02.349 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T21:22:02.349 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "incremental-sync", 2023-05-31T21:22:02.349 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "00000000000000000000:00000000000000044815", 2023-05-31T21:22:02.349 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T21:22:02.349 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2023-05-31T21:22:02.349 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T21:22:02.350 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2023-05-31T21:21:05.078544Z" 2023-05-31T21:22:02.350 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T21:22:02.350 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:22:02.350 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T21:22:02.350 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 1, 2023-05-31T21:22:02.351 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T21:22:02.351 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "incremental-sync", 2023-05-31T21:22:02.351 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "00000000000000000000:00000000000000031499", 2023-05-31T21:22:02.351 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T21:22:02.351 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2023-05-31T21:22:02.351 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T21:22:02.352 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2023-05-31T21:12:40.667485Z" 2023-05-31T21:22:02.352 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T21:22:02.352 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:22:02.352 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T21:22:02.352 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 2, 2023-05-31T21:22:02.352 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T21:22:02.352 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "incremental-sync", 2023-05-31T21:22:02.353 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "00000000000000000000:00000000000000050831", 2023-05-31T21:22:02.353 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T21:22:02.353 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 55, 2023-05-31T21:22:02.353 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T21:22:02.353 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2023-05-31T21:19:03.876722Z" 2023-05-31T21:22:02.353 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T21:22:02.353 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:22:02.354 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T21:22:02.354 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 3, 2023-05-31T21:22:02.354 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T21:22:02.354 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "incremental-sync", 2023-05-31T21:22:02.354 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "00000000000000000000:00000000000000043896", 2023-05-31T21:22:02.354 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T21:22:02.354 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 9, 2023-05-31T21:22:02.355 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T21:22:02.355 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2023-05-31T21:12:28.529956Z" 2023-05-31T21:22:02.355 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T21:22:02.355 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T21:22:02.355 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2023-05-31T21:22:02.355 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:22:02.356 INFO:teuthology.orchestra.run.smithi099.stdout: "full_sync": { 2023-05-31T21:22:02.356 INFO:teuthology.orchestra.run.smithi099.stdout: "total": 64, 2023-05-31T21:22:02.356 INFO:teuthology.orchestra.run.smithi099.stdout: "complete": 64 2023-05-31T21:22:02.356 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:22:02.356 INFO:teuthology.orchestra.run.smithi099.stdout: "current_time": "2023-05-31T21:22:02Z" 2023-05-31T21:22:02.356 INFO:teuthology.orchestra.run.smithi099.stdout:} 2023-05-31T21:22:02.357 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 452136274422638824 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000044815", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-05-31T21:21:05.078544Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000031499", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-05-31T21:12:40.667485Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000050831", "next_step_marker": "", "total_entries": 55, "pos": 0, "timestamp": "2023-05-31T21:19:03.876722Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000043896", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2023-05-31T21:12:28.529956Z" } } ] }, "full_sync": { "total": 64, "complete": 64 }, "current_time": "2023-05-31T21:22:02Z" } 2023-05-31T21:22:02.358 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000044815', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-05-31T21:21:05.078544Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000031499', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-05-31T21:12:40.667485Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000050831', 'next_step_marker': '', 'total_entries': 55, 'pos': 0, 'timestamp': '2023-05-31T21:19:03.876722Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000043896', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2023-05-31T21:12:28.529956Z'}}] 2023-05-31T21:22:02.358 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000044955', 1: '00000000000000000000:00000000000000031499', 2: '00000000000000000000:00000000000000051111', 3: '00000000000000000000:00000000000000044038'} 2023-05-31T21:22:02.358 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000044815', 1: '00000000000000000000:00000000000000031499', 2: '00000000000000000000:00000000000000050831', 3: '00000000000000000000:00000000000000043896'} 2023-05-31T21:22:02.359 WARNING:rgw_multi.tests:data of zone a2 behind zone a1: shard=0 master=00000000000000000000:00000000000000044955 target=00000000000000000000:00000000000000044815, shard=2 master=00000000000000000000:00000000000000051111 target=00000000000000000000:00000000000000050831, shard=3 master=00000000000000000000:00000000000000044038 target=00000000000000000000:00000000000000043896 2023-05-31T21:22:07.360 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-31T21:22:07.361 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-31T21:22:07.361 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-05-31T21:22:07.419 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2023-05-31T21:22:07.420 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2023-05-31T21:22:07.488 INFO:teuthology.orchestra.run.smithi099.stdout:{ 2023-05-31T21:22:07.489 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_status": { 2023-05-31T21:22:07.489 INFO:teuthology.orchestra.run.smithi099.stdout: "info": { 2023-05-31T21:22:07.489 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "sync", 2023-05-31T21:22:07.489 INFO:teuthology.orchestra.run.smithi099.stdout: "num_shards": 4, 2023-05-31T21:22:07.489 INFO:teuthology.orchestra.run.smithi099.stdout: "instance_id": 452136274422638824 2023-05-31T21:22:07.490 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:22:07.490 INFO:teuthology.orchestra.run.smithi099.stdout: "markers": [ 2023-05-31T21:22:07.490 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T21:22:07.490 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 0, 2023-05-31T21:22:07.490 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T21:22:07.490 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "incremental-sync", 2023-05-31T21:22:07.490 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "00000000000000000000:00000000000000044815", 2023-05-31T21:22:07.491 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T21:22:07.491 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2023-05-31T21:22:07.491 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T21:22:07.491 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2023-05-31T21:21:05.078544Z" 2023-05-31T21:22:07.491 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T21:22:07.491 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:22:07.492 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T21:22:07.492 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 1, 2023-05-31T21:22:07.492 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T21:22:07.492 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "incremental-sync", 2023-05-31T21:22:07.492 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "00000000000000000000:00000000000000031499", 2023-05-31T21:22:07.492 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T21:22:07.493 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2023-05-31T21:22:07.493 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T21:22:07.493 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2023-05-31T21:12:40.667485Z" 2023-05-31T21:22:07.493 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T21:22:07.493 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:22:07.493 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T21:22:07.493 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 2, 2023-05-31T21:22:07.494 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T21:22:07.494 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "incremental-sync", 2023-05-31T21:22:07.494 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "00000000000000000000:00000000000000050831", 2023-05-31T21:22:07.494 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T21:22:07.494 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 55, 2023-05-31T21:22:07.494 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T21:22:07.494 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2023-05-31T21:19:03.876722Z" 2023-05-31T21:22:07.495 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T21:22:07.495 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:22:07.495 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T21:22:07.495 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 3, 2023-05-31T21:22:07.495 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T21:22:07.495 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "incremental-sync", 2023-05-31T21:22:07.495 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "00000000000000000000:00000000000000043896", 2023-05-31T21:22:07.496 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T21:22:07.496 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 9, 2023-05-31T21:22:07.496 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T21:22:07.496 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2023-05-31T21:12:28.529956Z" 2023-05-31T21:22:07.496 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T21:22:07.497 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T21:22:07.497 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2023-05-31T21:22:07.497 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:22:07.497 INFO:teuthology.orchestra.run.smithi099.stdout: "full_sync": { 2023-05-31T21:22:07.497 INFO:teuthology.orchestra.run.smithi099.stdout: "total": 64, 2023-05-31T21:22:07.497 INFO:teuthology.orchestra.run.smithi099.stdout: "complete": 64 2023-05-31T21:22:07.498 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:22:07.498 INFO:teuthology.orchestra.run.smithi099.stdout: "current_time": "2023-05-31T21:22:07Z" 2023-05-31T21:22:07.498 INFO:teuthology.orchestra.run.smithi099.stdout:} 2023-05-31T21:22:07.499 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 452136274422638824 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000044815", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-05-31T21:21:05.078544Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000031499", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-05-31T21:12:40.667485Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000050831", "next_step_marker": "", "total_entries": 55, "pos": 0, "timestamp": "2023-05-31T21:19:03.876722Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000043896", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2023-05-31T21:12:28.529956Z" } } ] }, "full_sync": { "total": 64, "complete": 64 }, "current_time": "2023-05-31T21:22:07Z" } 2023-05-31T21:22:07.500 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000044815', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-05-31T21:21:05.078544Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000031499', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-05-31T21:12:40.667485Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000050831', 'next_step_marker': '', 'total_entries': 55, 'pos': 0, 'timestamp': '2023-05-31T21:19:03.876722Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000043896', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2023-05-31T21:12:28.529956Z'}}] 2023-05-31T21:22:07.500 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000044955', 1: '00000000000000000000:00000000000000031499', 2: '00000000000000000000:00000000000000051111', 3: '00000000000000000000:00000000000000044038'} 2023-05-31T21:22:07.500 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000044815', 1: '00000000000000000000:00000000000000031499', 2: '00000000000000000000:00000000000000050831', 3: '00000000000000000000:00000000000000043896'} 2023-05-31T21:22:07.501 WARNING:rgw_multi.tests:data of zone a2 behind zone a1: shard=0 master=00000000000000000000:00000000000000044955 target=00000000000000000000:00000000000000044815, shard=2 master=00000000000000000000:00000000000000051111 target=00000000000000000000:00000000000000050831, shard=3 master=00000000000000000000:00000000000000044038 target=00000000000000000000:00000000000000043896 2023-05-31T21:22:12.502 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-31T21:22:12.503 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-31T21:22:12.504 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-05-31T21:22:12.563 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2023-05-31T21:22:12.563 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2023-05-31T21:22:12.621 INFO:teuthology.orchestra.run.smithi099.stdout:{ 2023-05-31T21:22:12.621 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_status": { 2023-05-31T21:22:12.622 INFO:teuthology.orchestra.run.smithi099.stdout: "info": { 2023-05-31T21:22:12.622 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "sync", 2023-05-31T21:22:12.622 INFO:teuthology.orchestra.run.smithi099.stdout: "num_shards": 4, 2023-05-31T21:22:12.622 INFO:teuthology.orchestra.run.smithi099.stdout: "instance_id": 452136274422638824 2023-05-31T21:22:12.622 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:22:12.622 INFO:teuthology.orchestra.run.smithi099.stdout: "markers": [ 2023-05-31T21:22:12.622 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T21:22:12.623 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 0, 2023-05-31T21:22:12.623 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T21:22:12.623 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "incremental-sync", 2023-05-31T21:22:12.623 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "00000000000000000000:00000000000000044815", 2023-05-31T21:22:12.623 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T21:22:12.623 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2023-05-31T21:22:12.624 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T21:22:12.624 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2023-05-31T21:21:05.078544Z" 2023-05-31T21:22:12.624 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T21:22:12.624 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:22:12.624 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T21:22:12.624 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 1, 2023-05-31T21:22:12.624 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T21:22:12.625 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "incremental-sync", 2023-05-31T21:22:12.625 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "00000000000000000000:00000000000000031499", 2023-05-31T21:22:12.625 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T21:22:12.625 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2023-05-31T21:22:12.625 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T21:22:12.625 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2023-05-31T21:12:40.667485Z" 2023-05-31T21:22:12.626 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T21:22:12.626 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:22:12.626 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T21:22:12.626 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 2, 2023-05-31T21:22:12.626 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T21:22:12.626 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "incremental-sync", 2023-05-31T21:22:12.626 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "00000000000000000000:00000000000000050831", 2023-05-31T21:22:12.627 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T21:22:12.627 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 55, 2023-05-31T21:22:12.627 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T21:22:12.627 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2023-05-31T21:19:03.876722Z" 2023-05-31T21:22:12.627 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T21:22:12.627 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:22:12.627 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T21:22:12.628 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 3, 2023-05-31T21:22:12.628 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T21:22:12.628 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "incremental-sync", 2023-05-31T21:22:12.628 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "00000000000000000000:00000000000000043896", 2023-05-31T21:22:12.628 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T21:22:12.628 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 9, 2023-05-31T21:22:12.629 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T21:22:12.629 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2023-05-31T21:12:28.529956Z" 2023-05-31T21:22:12.629 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T21:22:12.629 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T21:22:12.629 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2023-05-31T21:22:12.629 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:22:12.629 INFO:teuthology.orchestra.run.smithi099.stdout: "full_sync": { 2023-05-31T21:22:12.630 INFO:teuthology.orchestra.run.smithi099.stdout: "total": 64, 2023-05-31T21:22:12.630 INFO:teuthology.orchestra.run.smithi099.stdout: "complete": 64 2023-05-31T21:22:12.630 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:22:12.630 INFO:teuthology.orchestra.run.smithi099.stdout: "current_time": "2023-05-31T21:22:12Z" 2023-05-31T21:22:12.630 INFO:teuthology.orchestra.run.smithi099.stdout:} 2023-05-31T21:22:12.631 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 452136274422638824 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000044815", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-05-31T21:21:05.078544Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000031499", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-05-31T21:12:40.667485Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000050831", "next_step_marker": "", "total_entries": 55, "pos": 0, "timestamp": "2023-05-31T21:19:03.876722Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000043896", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2023-05-31T21:12:28.529956Z" } } ] }, "full_sync": { "total": 64, "complete": 64 }, "current_time": "2023-05-31T21:22:12Z" } 2023-05-31T21:22:12.632 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000044815', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-05-31T21:21:05.078544Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000031499', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-05-31T21:12:40.667485Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000050831', 'next_step_marker': '', 'total_entries': 55, 'pos': 0, 'timestamp': '2023-05-31T21:19:03.876722Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000043896', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2023-05-31T21:12:28.529956Z'}}] 2023-05-31T21:22:12.632 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000044955', 1: '00000000000000000000:00000000000000031499', 2: '00000000000000000000:00000000000000051111', 3: '00000000000000000000:00000000000000044038'} 2023-05-31T21:22:12.632 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000044815', 1: '00000000000000000000:00000000000000031499', 2: '00000000000000000000:00000000000000050831', 3: '00000000000000000000:00000000000000043896'} 2023-05-31T21:22:12.633 WARNING:rgw_multi.tests:data of zone a2 behind zone a1: shard=0 master=00000000000000000000:00000000000000044955 target=00000000000000000000:00000000000000044815, shard=2 master=00000000000000000000:00000000000000051111 target=00000000000000000000:00000000000000050831, shard=3 master=00000000000000000000:00000000000000044038 target=00000000000000000000:00000000000000043896 2023-05-31T21:22:17.634 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-31T21:22:17.635 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-31T21:22:17.635 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-05-31T21:22:17.695 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2023-05-31T21:22:17.695 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2023-05-31T21:22:17.763 INFO:teuthology.orchestra.run.smithi099.stdout:{ 2023-05-31T21:22:17.764 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_status": { 2023-05-31T21:22:17.764 INFO:teuthology.orchestra.run.smithi099.stdout: "info": { 2023-05-31T21:22:17.764 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "sync", 2023-05-31T21:22:17.764 INFO:teuthology.orchestra.run.smithi099.stdout: "num_shards": 4, 2023-05-31T21:22:17.765 INFO:teuthology.orchestra.run.smithi099.stdout: "instance_id": 452136274422638824 2023-05-31T21:22:17.765 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:22:17.765 INFO:teuthology.orchestra.run.smithi099.stdout: "markers": [ 2023-05-31T21:22:17.765 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T21:22:17.765 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 0, 2023-05-31T21:22:17.765 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T21:22:17.765 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "incremental-sync", 2023-05-31T21:22:17.766 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "00000000000000000000:00000000000000044815", 2023-05-31T21:22:17.766 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T21:22:17.766 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2023-05-31T21:22:17.766 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T21:22:17.766 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2023-05-31T21:21:05.078544Z" 2023-05-31T21:22:17.766 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T21:22:17.766 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:22:17.767 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T21:22:17.767 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 1, 2023-05-31T21:22:17.767 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T21:22:17.767 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "incremental-sync", 2023-05-31T21:22:17.767 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "00000000000000000000:00000000000000031499", 2023-05-31T21:22:17.767 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T21:22:17.768 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2023-05-31T21:22:17.768 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T21:22:17.768 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2023-05-31T21:12:40.667485Z" 2023-05-31T21:22:17.768 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T21:22:17.768 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:22:17.768 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T21:22:17.768 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 2, 2023-05-31T21:22:17.769 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T21:22:17.769 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "incremental-sync", 2023-05-31T21:22:17.769 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "00000000000000000000:00000000000000050831", 2023-05-31T21:22:17.769 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T21:22:17.769 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 55, 2023-05-31T21:22:17.769 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T21:22:17.769 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2023-05-31T21:19:03.876722Z" 2023-05-31T21:22:17.770 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T21:22:17.770 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:22:17.770 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T21:22:17.770 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 3, 2023-05-31T21:22:17.770 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T21:22:17.770 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "incremental-sync", 2023-05-31T21:22:17.771 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "00000000000000000000:00000000000000043896", 2023-05-31T21:22:17.771 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T21:22:17.771 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 9, 2023-05-31T21:22:17.771 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T21:22:17.771 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2023-05-31T21:12:28.529956Z" 2023-05-31T21:22:17.771 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T21:22:17.771 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T21:22:17.772 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2023-05-31T21:22:17.772 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:22:17.772 INFO:teuthology.orchestra.run.smithi099.stdout: "full_sync": { 2023-05-31T21:22:17.772 INFO:teuthology.orchestra.run.smithi099.stdout: "total": 64, 2023-05-31T21:22:17.772 INFO:teuthology.orchestra.run.smithi099.stdout: "complete": 64 2023-05-31T21:22:17.772 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:22:17.772 INFO:teuthology.orchestra.run.smithi099.stdout: "current_time": "2023-05-31T21:22:17Z" 2023-05-31T21:22:17.773 INFO:teuthology.orchestra.run.smithi099.stdout:} 2023-05-31T21:22:17.773 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 452136274422638824 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000044815", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-05-31T21:21:05.078544Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000031499", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-05-31T21:12:40.667485Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000050831", "next_step_marker": "", "total_entries": 55, "pos": 0, "timestamp": "2023-05-31T21:19:03.876722Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000043896", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2023-05-31T21:12:28.529956Z" } } ] }, "full_sync": { "total": 64, "complete": 64 }, "current_time": "2023-05-31T21:22:17Z" } 2023-05-31T21:22:17.774 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000044815', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-05-31T21:21:05.078544Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000031499', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-05-31T21:12:40.667485Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000050831', 'next_step_marker': '', 'total_entries': 55, 'pos': 0, 'timestamp': '2023-05-31T21:19:03.876722Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000043896', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2023-05-31T21:12:28.529956Z'}}] 2023-05-31T21:22:17.774 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000044955', 1: '00000000000000000000:00000000000000031499', 2: '00000000000000000000:00000000000000051111', 3: '00000000000000000000:00000000000000044038'} 2023-05-31T21:22:17.775 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000044815', 1: '00000000000000000000:00000000000000031499', 2: '00000000000000000000:00000000000000050831', 3: '00000000000000000000:00000000000000043896'} 2023-05-31T21:22:17.775 WARNING:rgw_multi.tests:data of zone a2 behind zone a1: shard=0 master=00000000000000000000:00000000000000044955 target=00000000000000000000:00000000000000044815, shard=2 master=00000000000000000000:00000000000000051111 target=00000000000000000000:00000000000000050831, shard=3 master=00000000000000000000:00000000000000044038 target=00000000000000000000:00000000000000043896 2023-05-31T21:22:22.776 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-31T21:22:22.777 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-31T21:22:22.777 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-05-31T21:22:22.837 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2023-05-31T21:22:22.837 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2023-05-31T21:22:22.911 INFO:teuthology.orchestra.run.smithi099.stdout:{ 2023-05-31T21:22:22.912 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_status": { 2023-05-31T21:22:22.912 INFO:teuthology.orchestra.run.smithi099.stdout: "info": { 2023-05-31T21:22:22.912 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "sync", 2023-05-31T21:22:22.912 INFO:teuthology.orchestra.run.smithi099.stdout: "num_shards": 4, 2023-05-31T21:22:22.912 INFO:teuthology.orchestra.run.smithi099.stdout: "instance_id": 452136274422638824 2023-05-31T21:22:22.913 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:22:22.913 INFO:teuthology.orchestra.run.smithi099.stdout: "markers": [ 2023-05-31T21:22:22.913 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T21:22:22.913 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 0, 2023-05-31T21:22:22.913 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T21:22:22.913 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "incremental-sync", 2023-05-31T21:22:22.913 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "00000000000000000000:00000000000000044815", 2023-05-31T21:22:22.914 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T21:22:22.914 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2023-05-31T21:22:22.914 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T21:22:22.914 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2023-05-31T21:21:05.078544Z" 2023-05-31T21:22:22.915 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T21:22:22.915 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:22:22.915 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T21:22:22.915 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 1, 2023-05-31T21:22:22.915 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T21:22:22.915 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "incremental-sync", 2023-05-31T21:22:22.916 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "00000000000000000000:00000000000000031499", 2023-05-31T21:22:22.916 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T21:22:22.916 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2023-05-31T21:22:22.916 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T21:22:22.916 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2023-05-31T21:12:40.667485Z" 2023-05-31T21:22:22.916 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T21:22:22.917 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:22:22.917 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T21:22:22.917 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 2, 2023-05-31T21:22:22.917 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T21:22:22.917 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "incremental-sync", 2023-05-31T21:22:22.917 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "00000000000000000000:00000000000000050831", 2023-05-31T21:22:22.918 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T21:22:22.918 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 55, 2023-05-31T21:22:22.918 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T21:22:22.918 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2023-05-31T21:19:03.876722Z" 2023-05-31T21:22:22.918 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T21:22:22.918 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:22:22.918 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T21:22:22.919 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 3, 2023-05-31T21:22:22.919 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T21:22:22.919 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "incremental-sync", 2023-05-31T21:22:22.919 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "00000000000000000000:00000000000000043896", 2023-05-31T21:22:22.919 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T21:22:22.919 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 9, 2023-05-31T21:22:22.919 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T21:22:22.920 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2023-05-31T21:12:28.529956Z" 2023-05-31T21:22:22.920 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T21:22:22.920 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T21:22:22.920 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2023-05-31T21:22:22.920 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:22:22.920 INFO:teuthology.orchestra.run.smithi099.stdout: "full_sync": { 2023-05-31T21:22:22.921 INFO:teuthology.orchestra.run.smithi099.stdout: "total": 64, 2023-05-31T21:22:22.921 INFO:teuthology.orchestra.run.smithi099.stdout: "complete": 64 2023-05-31T21:22:22.921 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:22:22.921 INFO:teuthology.orchestra.run.smithi099.stdout: "current_time": "2023-05-31T21:22:22Z" 2023-05-31T21:22:22.922 INFO:teuthology.orchestra.run.smithi099.stdout:} 2023-05-31T21:22:22.923 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 452136274422638824 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000044815", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-05-31T21:21:05.078544Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000031499", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-05-31T21:12:40.667485Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000050831", "next_step_marker": "", "total_entries": 55, "pos": 0, "timestamp": "2023-05-31T21:19:03.876722Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000043896", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2023-05-31T21:12:28.529956Z" } } ] }, "full_sync": { "total": 64, "complete": 64 }, "current_time": "2023-05-31T21:22:22Z" } 2023-05-31T21:22:22.923 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000044815', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-05-31T21:21:05.078544Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000031499', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-05-31T21:12:40.667485Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000050831', 'next_step_marker': '', 'total_entries': 55, 'pos': 0, 'timestamp': '2023-05-31T21:19:03.876722Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000043896', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2023-05-31T21:12:28.529956Z'}}] 2023-05-31T21:22:22.923 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000044955', 1: '00000000000000000000:00000000000000031499', 2: '00000000000000000000:00000000000000051111', 3: '00000000000000000000:00000000000000044038'} 2023-05-31T21:22:22.924 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000044815', 1: '00000000000000000000:00000000000000031499', 2: '00000000000000000000:00000000000000050831', 3: '00000000000000000000:00000000000000043896'} 2023-05-31T21:22:22.925 WARNING:rgw_multi.tests:data of zone a2 behind zone a1: shard=0 master=00000000000000000000:00000000000000044955 target=00000000000000000000:00000000000000044815, shard=2 master=00000000000000000000:00000000000000051111 target=00000000000000000000:00000000000000050831, shard=3 master=00000000000000000000:00000000000000044038 target=00000000000000000000:00000000000000043896 2023-05-31T21:22:27.925 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-31T21:22:27.926 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-31T21:22:27.927 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-05-31T21:22:27.986 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2023-05-31T21:22:27.987 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2023-05-31T21:22:28.057 INFO:teuthology.orchestra.run.smithi099.stdout:{ 2023-05-31T21:22:28.057 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_status": { 2023-05-31T21:22:28.057 INFO:teuthology.orchestra.run.smithi099.stdout: "info": { 2023-05-31T21:22:28.058 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "sync", 2023-05-31T21:22:28.058 INFO:teuthology.orchestra.run.smithi099.stdout: "num_shards": 4, 2023-05-31T21:22:28.058 INFO:teuthology.orchestra.run.smithi099.stdout: "instance_id": 452136274422638824 2023-05-31T21:22:28.058 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:22:28.058 INFO:teuthology.orchestra.run.smithi099.stdout: "markers": [ 2023-05-31T21:22:28.058 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T21:22:28.060 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 0, 2023-05-31T21:22:28.060 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T21:22:28.060 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "incremental-sync", 2023-05-31T21:22:28.060 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "00000000000000000000:00000000000000044815", 2023-05-31T21:22:28.060 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T21:22:28.061 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2023-05-31T21:22:28.061 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T21:22:28.062 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2023-05-31T21:21:05.078544Z" 2023-05-31T21:22:28.062 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T21:22:28.062 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:22:28.063 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T21:22:28.063 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 1, 2023-05-31T21:22:28.063 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T21:22:28.063 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "incremental-sync", 2023-05-31T21:22:28.063 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "00000000000000000000:00000000000000031499", 2023-05-31T21:22:28.063 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T21:22:28.064 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2023-05-31T21:22:28.064 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T21:22:28.065 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2023-05-31T21:12:40.667485Z" 2023-05-31T21:22:28.065 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T21:22:28.065 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:22:28.065 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T21:22:28.065 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 2, 2023-05-31T21:22:28.065 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T21:22:28.065 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "incremental-sync", 2023-05-31T21:22:28.066 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "00000000000000000000:00000000000000050831", 2023-05-31T21:22:28.066 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T21:22:28.066 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 55, 2023-05-31T21:22:28.066 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T21:22:28.066 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2023-05-31T21:19:03.876722Z" 2023-05-31T21:22:28.066 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T21:22:28.067 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:22:28.067 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T21:22:28.067 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 3, 2023-05-31T21:22:28.067 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T21:22:28.067 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "incremental-sync", 2023-05-31T21:22:28.067 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "00000000000000000000:00000000000000043896", 2023-05-31T21:22:28.067 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T21:22:28.068 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 9, 2023-05-31T21:22:28.068 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T21:22:28.069 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2023-05-31T21:12:28.529956Z" 2023-05-31T21:22:28.069 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T21:22:28.069 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T21:22:28.069 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2023-05-31T21:22:28.069 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:22:28.069 INFO:teuthology.orchestra.run.smithi099.stdout: "full_sync": { 2023-05-31T21:22:28.069 INFO:teuthology.orchestra.run.smithi099.stdout: "total": 64, 2023-05-31T21:22:28.070 INFO:teuthology.orchestra.run.smithi099.stdout: "complete": 64 2023-05-31T21:22:28.070 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:22:28.070 INFO:teuthology.orchestra.run.smithi099.stdout: "current_time": "2023-05-31T21:22:28Z" 2023-05-31T21:22:28.070 INFO:teuthology.orchestra.run.smithi099.stdout:} 2023-05-31T21:22:28.071 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 452136274422638824 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000044815", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-05-31T21:21:05.078544Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000031499", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-05-31T21:12:40.667485Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000050831", "next_step_marker": "", "total_entries": 55, "pos": 0, "timestamp": "2023-05-31T21:19:03.876722Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000043896", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2023-05-31T21:12:28.529956Z" } } ] }, "full_sync": { "total": 64, "complete": 64 }, "current_time": "2023-05-31T21:22:28Z" } 2023-05-31T21:22:28.072 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000044815', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-05-31T21:21:05.078544Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000031499', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-05-31T21:12:40.667485Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000050831', 'next_step_marker': '', 'total_entries': 55, 'pos': 0, 'timestamp': '2023-05-31T21:19:03.876722Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000043896', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2023-05-31T21:12:28.529956Z'}}] 2023-05-31T21:22:28.072 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000044955', 1: '00000000000000000000:00000000000000031499', 2: '00000000000000000000:00000000000000051111', 3: '00000000000000000000:00000000000000044038'} 2023-05-31T21:22:28.072 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000044815', 1: '00000000000000000000:00000000000000031499', 2: '00000000000000000000:00000000000000050831', 3: '00000000000000000000:00000000000000043896'} 2023-05-31T21:22:28.073 WARNING:rgw_multi.tests:data of zone a2 behind zone a1: shard=0 master=00000000000000000000:00000000000000044955 target=00000000000000000000:00000000000000044815, shard=2 master=00000000000000000000:00000000000000051111 target=00000000000000000000:00000000000000050831, shard=3 master=00000000000000000000:00000000000000044038 target=00000000000000000000:00000000000000043896 2023-05-31T21:22:33.074 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-31T21:22:33.075 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-31T21:22:33.076 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-05-31T21:22:33.134 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2023-05-31T21:22:33.135 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2023-05-31T21:22:33.194 INFO:teuthology.orchestra.run.smithi099.stdout:{ 2023-05-31T21:22:33.195 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_status": { 2023-05-31T21:22:33.195 INFO:teuthology.orchestra.run.smithi099.stdout: "info": { 2023-05-31T21:22:33.195 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "sync", 2023-05-31T21:22:33.195 INFO:teuthology.orchestra.run.smithi099.stdout: "num_shards": 4, 2023-05-31T21:22:33.195 INFO:teuthology.orchestra.run.smithi099.stdout: "instance_id": 452136274422638824 2023-05-31T21:22:33.196 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:22:33.196 INFO:teuthology.orchestra.run.smithi099.stdout: "markers": [ 2023-05-31T21:22:33.196 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T21:22:33.196 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 0, 2023-05-31T21:22:33.196 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T21:22:33.196 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "incremental-sync", 2023-05-31T21:22:33.196 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "00000000000000000000:00000000000000044815", 2023-05-31T21:22:33.197 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T21:22:33.197 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2023-05-31T21:22:33.197 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T21:22:33.197 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2023-05-31T21:21:05.078544Z" 2023-05-31T21:22:33.197 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T21:22:33.197 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:22:33.198 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T21:22:33.198 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 1, 2023-05-31T21:22:33.198 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T21:22:33.198 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "incremental-sync", 2023-05-31T21:22:33.198 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "00000000000000000000:00000000000000031499", 2023-05-31T21:22:33.198 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T21:22:33.198 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2023-05-31T21:22:33.199 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T21:22:33.199 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2023-05-31T21:12:40.667485Z" 2023-05-31T21:22:33.199 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T21:22:33.199 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:22:33.199 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T21:22:33.199 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 2, 2023-05-31T21:22:33.199 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T21:22:33.200 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "incremental-sync", 2023-05-31T21:22:33.200 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "00000000000000000000:00000000000000050831", 2023-05-31T21:22:33.200 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T21:22:33.200 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 55, 2023-05-31T21:22:33.200 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T21:22:33.200 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2023-05-31T21:19:03.876722Z" 2023-05-31T21:22:33.200 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T21:22:33.201 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:22:33.201 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T21:22:33.201 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 3, 2023-05-31T21:22:33.201 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T21:22:33.201 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "incremental-sync", 2023-05-31T21:22:33.201 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "00000000000000000000:00000000000000043896", 2023-05-31T21:22:33.202 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T21:22:33.202 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 9, 2023-05-31T21:22:33.202 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T21:22:33.202 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2023-05-31T21:12:28.529956Z" 2023-05-31T21:22:33.202 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T21:22:33.203 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T21:22:33.203 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2023-05-31T21:22:33.203 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:22:33.203 INFO:teuthology.orchestra.run.smithi099.stdout: "full_sync": { 2023-05-31T21:22:33.203 INFO:teuthology.orchestra.run.smithi099.stdout: "total": 64, 2023-05-31T21:22:33.203 INFO:teuthology.orchestra.run.smithi099.stdout: "complete": 64 2023-05-31T21:22:33.203 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:22:33.204 INFO:teuthology.orchestra.run.smithi099.stdout: "current_time": "2023-05-31T21:22:33Z" 2023-05-31T21:22:33.204 INFO:teuthology.orchestra.run.smithi099.stdout:} 2023-05-31T21:22:33.205 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 452136274422638824 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000044815", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-05-31T21:21:05.078544Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000031499", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-05-31T21:12:40.667485Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000050831", "next_step_marker": "", "total_entries": 55, "pos": 0, "timestamp": "2023-05-31T21:19:03.876722Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000043896", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2023-05-31T21:12:28.529956Z" } } ] }, "full_sync": { "total": 64, "complete": 64 }, "current_time": "2023-05-31T21:22:33Z" } 2023-05-31T21:22:33.205 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000044815', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-05-31T21:21:05.078544Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000031499', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-05-31T21:12:40.667485Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000050831', 'next_step_marker': '', 'total_entries': 55, 'pos': 0, 'timestamp': '2023-05-31T21:19:03.876722Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000043896', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2023-05-31T21:12:28.529956Z'}}] 2023-05-31T21:22:33.206 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000044955', 1: '00000000000000000000:00000000000000031499', 2: '00000000000000000000:00000000000000051111', 3: '00000000000000000000:00000000000000044038'} 2023-05-31T21:22:33.206 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000044815', 1: '00000000000000000000:00000000000000031499', 2: '00000000000000000000:00000000000000050831', 3: '00000000000000000000:00000000000000043896'} 2023-05-31T21:22:33.206 WARNING:rgw_multi.tests:data of zone a2 behind zone a1: shard=0 master=00000000000000000000:00000000000000044955 target=00000000000000000000:00000000000000044815, shard=2 master=00000000000000000000:00000000000000051111 target=00000000000000000000:00000000000000050831, shard=3 master=00000000000000000000:00000000000000044038 target=00000000000000000000:00000000000000043896 2023-05-31T21:22:38.208 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-31T21:22:38.208 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-31T21:22:38.209 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-05-31T21:22:38.268 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2023-05-31T21:22:38.269 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2023-05-31T21:22:38.331 INFO:teuthology.orchestra.run.smithi099.stdout:{ 2023-05-31T21:22:38.331 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_status": { 2023-05-31T21:22:38.331 INFO:teuthology.orchestra.run.smithi099.stdout: "info": { 2023-05-31T21:22:38.332 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "sync", 2023-05-31T21:22:38.332 INFO:teuthology.orchestra.run.smithi099.stdout: "num_shards": 4, 2023-05-31T21:22:38.332 INFO:teuthology.orchestra.run.smithi099.stdout: "instance_id": 452136274422638824 2023-05-31T21:22:38.332 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:22:38.332 INFO:teuthology.orchestra.run.smithi099.stdout: "markers": [ 2023-05-31T21:22:38.332 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T21:22:38.333 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 0, 2023-05-31T21:22:38.333 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T21:22:38.333 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "incremental-sync", 2023-05-31T21:22:38.333 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "00000000000000000000:00000000000000044815", 2023-05-31T21:22:38.333 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T21:22:38.333 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2023-05-31T21:22:38.333 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T21:22:38.334 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2023-05-31T21:21:05.078544Z" 2023-05-31T21:22:38.334 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T21:22:38.334 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:22:38.334 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T21:22:38.334 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 1, 2023-05-31T21:22:38.334 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T21:22:38.335 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "incremental-sync", 2023-05-31T21:22:38.335 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "00000000000000000000:00000000000000031499", 2023-05-31T21:22:38.335 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T21:22:38.335 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2023-05-31T21:22:38.335 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T21:22:38.335 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2023-05-31T21:12:40.667485Z" 2023-05-31T21:22:38.335 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T21:22:38.336 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:22:38.336 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T21:22:38.336 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 2, 2023-05-31T21:22:38.336 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T21:22:38.336 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "incremental-sync", 2023-05-31T21:22:38.336 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "00000000000000000000:00000000000000050831", 2023-05-31T21:22:38.336 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T21:22:38.337 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 55, 2023-05-31T21:22:38.337 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T21:22:38.338 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2023-05-31T21:19:03.876722Z" 2023-05-31T21:22:38.338 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T21:22:38.338 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:22:38.338 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T21:22:38.338 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 3, 2023-05-31T21:22:38.339 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T21:22:38.339 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "incremental-sync", 2023-05-31T21:22:38.339 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "00000000000000000000:00000000000000043896", 2023-05-31T21:22:38.339 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T21:22:38.339 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 9, 2023-05-31T21:22:38.339 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T21:22:38.339 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2023-05-31T21:12:28.529956Z" 2023-05-31T21:22:38.340 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T21:22:38.340 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T21:22:38.340 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2023-05-31T21:22:38.341 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:22:38.341 INFO:teuthology.orchestra.run.smithi099.stdout: "full_sync": { 2023-05-31T21:22:38.341 INFO:teuthology.orchestra.run.smithi099.stdout: "total": 64, 2023-05-31T21:22:38.341 INFO:teuthology.orchestra.run.smithi099.stdout: "complete": 64 2023-05-31T21:22:38.342 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:22:38.342 INFO:teuthology.orchestra.run.smithi099.stdout: "current_time": "2023-05-31T21:22:38Z" 2023-05-31T21:22:38.342 INFO:teuthology.orchestra.run.smithi099.stdout:} 2023-05-31T21:22:38.343 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 452136274422638824 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000044815", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-05-31T21:21:05.078544Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000031499", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-05-31T21:12:40.667485Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000050831", "next_step_marker": "", "total_entries": 55, "pos": 0, "timestamp": "2023-05-31T21:19:03.876722Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000043896", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2023-05-31T21:12:28.529956Z" } } ] }, "full_sync": { "total": 64, "complete": 64 }, "current_time": "2023-05-31T21:22:38Z" } 2023-05-31T21:22:38.344 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000044815', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-05-31T21:21:05.078544Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000031499', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-05-31T21:12:40.667485Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000050831', 'next_step_marker': '', 'total_entries': 55, 'pos': 0, 'timestamp': '2023-05-31T21:19:03.876722Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000043896', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2023-05-31T21:12:28.529956Z'}}] 2023-05-31T21:22:38.344 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000044955', 1: '00000000000000000000:00000000000000031499', 2: '00000000000000000000:00000000000000051111', 3: '00000000000000000000:00000000000000044038'} 2023-05-31T21:22:38.344 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000044815', 1: '00000000000000000000:00000000000000031499', 2: '00000000000000000000:00000000000000050831', 3: '00000000000000000000:00000000000000043896'} 2023-05-31T21:22:38.345 WARNING:rgw_multi.tests:data of zone a2 behind zone a1: shard=0 master=00000000000000000000:00000000000000044955 target=00000000000000000000:00000000000000044815, shard=2 master=00000000000000000000:00000000000000051111 target=00000000000000000000:00000000000000050831, shard=3 master=00000000000000000000:00000000000000044038 target=00000000000000000000:00000000000000043896 2023-05-31T21:22:43.346 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-31T21:22:43.346 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-31T21:22:43.347 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-05-31T21:22:43.406 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2023-05-31T21:22:43.406 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2023-05-31T21:22:43.483 INFO:teuthology.orchestra.run.smithi099.stdout:{ 2023-05-31T21:22:43.484 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_status": { 2023-05-31T21:22:43.484 INFO:teuthology.orchestra.run.smithi099.stdout: "info": { 2023-05-31T21:22:43.484 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "sync", 2023-05-31T21:22:43.484 INFO:teuthology.orchestra.run.smithi099.stdout: "num_shards": 4, 2023-05-31T21:22:43.484 INFO:teuthology.orchestra.run.smithi099.stdout: "instance_id": 452136274422638824 2023-05-31T21:22:43.485 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:22:43.486 INFO:teuthology.orchestra.run.smithi099.stdout: "markers": [ 2023-05-31T21:22:43.486 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T21:22:43.486 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 0, 2023-05-31T21:22:43.486 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T21:22:43.487 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "incremental-sync", 2023-05-31T21:22:43.487 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "00000000000000000000:00000000000000044815", 2023-05-31T21:22:43.487 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T21:22:43.487 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2023-05-31T21:22:43.487 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T21:22:43.487 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2023-05-31T21:21:05.078544Z" 2023-05-31T21:22:43.488 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T21:22:43.488 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:22:43.488 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T21:22:43.488 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 1, 2023-05-31T21:22:43.488 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T21:22:43.488 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "incremental-sync", 2023-05-31T21:22:43.489 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "00000000000000000000:00000000000000031499", 2023-05-31T21:22:43.489 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T21:22:43.489 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2023-05-31T21:22:43.489 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T21:22:43.489 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2023-05-31T21:12:40.667485Z" 2023-05-31T21:22:43.489 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T21:22:43.489 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:22:43.490 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T21:22:43.490 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 2, 2023-05-31T21:22:43.490 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T21:22:43.490 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "incremental-sync", 2023-05-31T21:22:43.490 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "00000000000000000000:00000000000000050831", 2023-05-31T21:22:43.490 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T21:22:43.491 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 55, 2023-05-31T21:22:43.491 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T21:22:43.491 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2023-05-31T21:19:03.876722Z" 2023-05-31T21:22:43.491 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T21:22:43.491 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:22:43.491 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T21:22:43.491 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 3, 2023-05-31T21:22:43.492 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T21:22:43.492 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "incremental-sync", 2023-05-31T21:22:43.492 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "00000000000000000000:00000000000000043896", 2023-05-31T21:22:43.492 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T21:22:43.492 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 9, 2023-05-31T21:22:43.492 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T21:22:43.492 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2023-05-31T21:12:28.529956Z" 2023-05-31T21:22:43.493 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T21:22:43.493 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T21:22:43.493 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2023-05-31T21:22:43.493 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:22:43.493 INFO:teuthology.orchestra.run.smithi099.stdout: "full_sync": { 2023-05-31T21:22:43.493 INFO:teuthology.orchestra.run.smithi099.stdout: "total": 64, 2023-05-31T21:22:43.494 INFO:teuthology.orchestra.run.smithi099.stdout: "complete": 64 2023-05-31T21:22:43.494 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:22:43.494 INFO:teuthology.orchestra.run.smithi099.stdout: "current_time": "2023-05-31T21:22:43Z" 2023-05-31T21:22:43.494 INFO:teuthology.orchestra.run.smithi099.stdout:} 2023-05-31T21:22:43.495 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 452136274422638824 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000044815", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-05-31T21:21:05.078544Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000031499", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-05-31T21:12:40.667485Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000050831", "next_step_marker": "", "total_entries": 55, "pos": 0, "timestamp": "2023-05-31T21:19:03.876722Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000043896", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2023-05-31T21:12:28.529956Z" } } ] }, "full_sync": { "total": 64, "complete": 64 }, "current_time": "2023-05-31T21:22:43Z" } 2023-05-31T21:22:43.499 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000044815', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-05-31T21:21:05.078544Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000031499', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-05-31T21:12:40.667485Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000050831', 'next_step_marker': '', 'total_entries': 55, 'pos': 0, 'timestamp': '2023-05-31T21:19:03.876722Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000043896', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2023-05-31T21:12:28.529956Z'}}] 2023-05-31T21:22:43.500 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000044955', 1: '00000000000000000000:00000000000000031499', 2: '00000000000000000000:00000000000000051111', 3: '00000000000000000000:00000000000000044038'} 2023-05-31T21:22:43.500 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000044815', 1: '00000000000000000000:00000000000000031499', 2: '00000000000000000000:00000000000000050831', 3: '00000000000000000000:00000000000000043896'} 2023-05-31T21:22:43.501 WARNING:rgw_multi.tests:data of zone a2 behind zone a1: shard=0 master=00000000000000000000:00000000000000044955 target=00000000000000000000:00000000000000044815, shard=2 master=00000000000000000000:00000000000000051111 target=00000000000000000000:00000000000000050831, shard=3 master=00000000000000000000:00000000000000044038 target=00000000000000000000:00000000000000043896 2023-05-31T21:22:48.501 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-31T21:22:48.502 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-31T21:22:48.503 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-05-31T21:22:48.562 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2023-05-31T21:22:48.563 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2023-05-31T21:22:48.628 INFO:teuthology.orchestra.run.smithi099.stdout:{ 2023-05-31T21:22:48.629 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_status": { 2023-05-31T21:22:48.629 INFO:teuthology.orchestra.run.smithi099.stdout: "info": { 2023-05-31T21:22:48.629 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "sync", 2023-05-31T21:22:48.629 INFO:teuthology.orchestra.run.smithi099.stdout: "num_shards": 4, 2023-05-31T21:22:48.629 INFO:teuthology.orchestra.run.smithi099.stdout: "instance_id": 452136274422638824 2023-05-31T21:22:48.630 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:22:48.630 INFO:teuthology.orchestra.run.smithi099.stdout: "markers": [ 2023-05-31T21:22:48.630 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T21:22:48.630 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 0, 2023-05-31T21:22:48.630 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T21:22:48.630 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "incremental-sync", 2023-05-31T21:22:48.630 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "00000000000000000000:00000000000000044815", 2023-05-31T21:22:48.631 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T21:22:48.631 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2023-05-31T21:22:48.631 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T21:22:48.631 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2023-05-31T21:21:05.078544Z" 2023-05-31T21:22:48.631 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T21:22:48.631 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:22:48.632 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T21:22:48.632 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 1, 2023-05-31T21:22:48.632 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T21:22:48.632 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "incremental-sync", 2023-05-31T21:22:48.632 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "00000000000000000000:00000000000000031499", 2023-05-31T21:22:48.632 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T21:22:48.632 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2023-05-31T21:22:48.633 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T21:22:48.633 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2023-05-31T21:12:40.667485Z" 2023-05-31T21:22:48.633 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T21:22:48.633 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:22:48.633 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T21:22:48.633 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 2, 2023-05-31T21:22:48.634 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T21:22:48.634 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "incremental-sync", 2023-05-31T21:22:48.634 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "00000000000000000000:00000000000000050831", 2023-05-31T21:22:48.634 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T21:22:48.634 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 55, 2023-05-31T21:22:48.634 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T21:22:48.634 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2023-05-31T21:19:03.876722Z" 2023-05-31T21:22:48.635 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T21:22:48.635 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:22:48.635 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T21:22:48.635 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 3, 2023-05-31T21:22:48.635 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T21:22:48.635 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "incremental-sync", 2023-05-31T21:22:48.636 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "00000000000000000000:00000000000000043896", 2023-05-31T21:22:48.636 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T21:22:48.636 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 9, 2023-05-31T21:22:48.636 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T21:22:48.636 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2023-05-31T21:12:28.529956Z" 2023-05-31T21:22:48.636 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T21:22:48.636 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T21:22:48.637 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2023-05-31T21:22:48.637 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:22:48.637 INFO:teuthology.orchestra.run.smithi099.stdout: "full_sync": { 2023-05-31T21:22:48.637 INFO:teuthology.orchestra.run.smithi099.stdout: "total": 64, 2023-05-31T21:22:48.637 INFO:teuthology.orchestra.run.smithi099.stdout: "complete": 64 2023-05-31T21:22:48.637 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:22:48.638 INFO:teuthology.orchestra.run.smithi099.stdout: "current_time": "2023-05-31T21:22:48Z" 2023-05-31T21:22:48.638 INFO:teuthology.orchestra.run.smithi099.stdout:} 2023-05-31T21:22:48.639 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 452136274422638824 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000044815", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-05-31T21:21:05.078544Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000031499", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-05-31T21:12:40.667485Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000050831", "next_step_marker": "", "total_entries": 55, "pos": 0, "timestamp": "2023-05-31T21:19:03.876722Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000043896", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2023-05-31T21:12:28.529956Z" } } ] }, "full_sync": { "total": 64, "complete": 64 }, "current_time": "2023-05-31T21:22:48Z" } 2023-05-31T21:22:48.639 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000044815', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-05-31T21:21:05.078544Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000031499', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-05-31T21:12:40.667485Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000050831', 'next_step_marker': '', 'total_entries': 55, 'pos': 0, 'timestamp': '2023-05-31T21:19:03.876722Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000043896', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2023-05-31T21:12:28.529956Z'}}] 2023-05-31T21:22:48.640 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000044955', 1: '00000000000000000000:00000000000000031499', 2: '00000000000000000000:00000000000000051111', 3: '00000000000000000000:00000000000000044038'} 2023-05-31T21:22:48.640 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000044815', 1: '00000000000000000000:00000000000000031499', 2: '00000000000000000000:00000000000000050831', 3: '00000000000000000000:00000000000000043896'} 2023-05-31T21:22:48.640 WARNING:rgw_multi.tests:data of zone a2 behind zone a1: shard=0 master=00000000000000000000:00000000000000044955 target=00000000000000000000:00000000000000044815, shard=2 master=00000000000000000000:00000000000000051111 target=00000000000000000000:00000000000000050831, shard=3 master=00000000000000000000:00000000000000044038 target=00000000000000000000:00000000000000043896 2023-05-31T21:22:53.642 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-31T21:22:53.643 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-31T21:22:53.644 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-05-31T21:22:53.703 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2023-05-31T21:22:53.704 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2023-05-31T21:22:53.757 INFO:teuthology.orchestra.run.smithi099.stdout:{ 2023-05-31T21:22:53.757 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_status": { 2023-05-31T21:22:53.758 INFO:teuthology.orchestra.run.smithi099.stdout: "info": { 2023-05-31T21:22:53.758 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "sync", 2023-05-31T21:22:53.758 INFO:teuthology.orchestra.run.smithi099.stdout: "num_shards": 4, 2023-05-31T21:22:53.758 INFO:teuthology.orchestra.run.smithi099.stdout: "instance_id": 452136274422638824 2023-05-31T21:22:53.758 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:22:53.758 INFO:teuthology.orchestra.run.smithi099.stdout: "markers": [ 2023-05-31T21:22:53.759 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T21:22:53.759 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 0, 2023-05-31T21:22:53.759 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T21:22:53.759 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "incremental-sync", 2023-05-31T21:22:53.759 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "00000000000000000000:00000000000000044815", 2023-05-31T21:22:53.759 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T21:22:53.759 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2023-05-31T21:22:53.760 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T21:22:53.760 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2023-05-31T21:21:05.078544Z" 2023-05-31T21:22:53.760 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T21:22:53.760 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:22:53.760 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T21:22:53.760 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 1, 2023-05-31T21:22:53.761 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T21:22:53.761 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "incremental-sync", 2023-05-31T21:22:53.761 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "00000000000000000000:00000000000000031499", 2023-05-31T21:22:53.761 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T21:22:53.761 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2023-05-31T21:22:53.761 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T21:22:53.761 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2023-05-31T21:12:40.667485Z" 2023-05-31T21:22:53.762 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T21:22:53.762 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:22:53.762 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T21:22:53.762 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 2, 2023-05-31T21:22:53.762 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T21:22:53.762 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "incremental-sync", 2023-05-31T21:22:53.762 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "00000000000000000000:00000000000000050831", 2023-05-31T21:22:53.763 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T21:22:53.763 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 55, 2023-05-31T21:22:53.763 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T21:22:53.763 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2023-05-31T21:19:03.876722Z" 2023-05-31T21:22:53.763 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T21:22:53.763 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:22:53.763 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T21:22:53.764 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 3, 2023-05-31T21:22:53.764 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T21:22:53.764 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "incremental-sync", 2023-05-31T21:22:53.764 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "00000000000000000000:00000000000000043896", 2023-05-31T21:22:53.764 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T21:22:53.764 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 9, 2023-05-31T21:22:53.764 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T21:22:53.765 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2023-05-31T21:12:28.529956Z" 2023-05-31T21:22:53.765 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T21:22:53.765 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T21:22:53.765 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2023-05-31T21:22:53.765 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:22:53.765 INFO:teuthology.orchestra.run.smithi099.stdout: "full_sync": { 2023-05-31T21:22:53.765 INFO:teuthology.orchestra.run.smithi099.stdout: "total": 64, 2023-05-31T21:22:53.766 INFO:teuthology.orchestra.run.smithi099.stdout: "complete": 64 2023-05-31T21:22:53.766 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:22:53.766 INFO:teuthology.orchestra.run.smithi099.stdout: "current_time": "2023-05-31T21:22:53Z" 2023-05-31T21:22:53.766 INFO:teuthology.orchestra.run.smithi099.stdout:} 2023-05-31T21:22:53.767 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 452136274422638824 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000044815", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-05-31T21:21:05.078544Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000031499", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-05-31T21:12:40.667485Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000050831", "next_step_marker": "", "total_entries": 55, "pos": 0, "timestamp": "2023-05-31T21:19:03.876722Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000043896", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2023-05-31T21:12:28.529956Z" } } ] }, "full_sync": { "total": 64, "complete": 64 }, "current_time": "2023-05-31T21:22:53Z" } 2023-05-31T21:22:53.768 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000044815', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-05-31T21:21:05.078544Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000031499', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-05-31T21:12:40.667485Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000050831', 'next_step_marker': '', 'total_entries': 55, 'pos': 0, 'timestamp': '2023-05-31T21:19:03.876722Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000043896', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2023-05-31T21:12:28.529956Z'}}] 2023-05-31T21:22:53.768 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000044955', 1: '00000000000000000000:00000000000000031499', 2: '00000000000000000000:00000000000000051111', 3: '00000000000000000000:00000000000000044038'} 2023-05-31T21:22:53.768 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000044815', 1: '00000000000000000000:00000000000000031499', 2: '00000000000000000000:00000000000000050831', 3: '00000000000000000000:00000000000000043896'} 2023-05-31T21:22:53.769 WARNING:rgw_multi.tests:data of zone a2 behind zone a1: shard=0 master=00000000000000000000:00000000000000044955 target=00000000000000000000:00000000000000044815, shard=2 master=00000000000000000000:00000000000000051111 target=00000000000000000000:00000000000000050831, shard=3 master=00000000000000000000:00000000000000044038 target=00000000000000000000:00000000000000043896 2023-05-31T21:22:58.770 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-31T21:22:58.771 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-31T21:22:58.771 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-05-31T21:22:58.830 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2023-05-31T21:22:58.831 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2023-05-31T21:22:58.908 INFO:teuthology.orchestra.run.smithi099.stdout:{ 2023-05-31T21:22:58.908 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_status": { 2023-05-31T21:22:58.909 INFO:teuthology.orchestra.run.smithi099.stdout: "info": { 2023-05-31T21:22:58.909 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "sync", 2023-05-31T21:22:58.909 INFO:teuthology.orchestra.run.smithi099.stdout: "num_shards": 4, 2023-05-31T21:22:58.909 INFO:teuthology.orchestra.run.smithi099.stdout: "instance_id": 452136274422638824 2023-05-31T21:22:58.909 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:22:58.909 INFO:teuthology.orchestra.run.smithi099.stdout: "markers": [ 2023-05-31T21:22:58.910 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T21:22:58.910 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 0, 2023-05-31T21:22:58.910 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T21:22:58.910 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "incremental-sync", 2023-05-31T21:22:58.910 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "00000000000000000000:00000000000000044815", 2023-05-31T21:22:58.910 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T21:22:58.910 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2023-05-31T21:22:58.911 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T21:22:58.911 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2023-05-31T21:21:05.078544Z" 2023-05-31T21:22:58.911 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T21:22:58.911 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:22:58.911 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T21:22:58.911 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 1, 2023-05-31T21:22:58.912 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T21:22:58.912 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "incremental-sync", 2023-05-31T21:22:58.912 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "00000000000000000000:00000000000000031499", 2023-05-31T21:22:58.912 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T21:22:58.912 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2023-05-31T21:22:58.912 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T21:22:58.913 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2023-05-31T21:12:40.667485Z" 2023-05-31T21:22:58.913 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T21:22:58.913 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:22:58.913 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T21:22:58.913 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 2, 2023-05-31T21:22:58.913 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T21:22:58.914 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "incremental-sync", 2023-05-31T21:22:58.914 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "00000000000000000000:00000000000000050831", 2023-05-31T21:22:58.914 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T21:22:58.914 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 55, 2023-05-31T21:22:58.914 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T21:22:58.914 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2023-05-31T21:19:03.876722Z" 2023-05-31T21:22:58.915 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T21:22:58.915 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:22:58.915 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T21:22:58.915 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 3, 2023-05-31T21:22:58.915 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T21:22:58.915 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "incremental-sync", 2023-05-31T21:22:58.916 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "00000000000000000000:00000000000000043896", 2023-05-31T21:22:58.916 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T21:22:58.916 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 9, 2023-05-31T21:22:58.917 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T21:22:58.917 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2023-05-31T21:12:28.529956Z" 2023-05-31T21:22:58.917 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T21:22:58.918 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T21:22:58.918 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2023-05-31T21:22:58.918 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:22:58.918 INFO:teuthology.orchestra.run.smithi099.stdout: "full_sync": { 2023-05-31T21:22:58.918 INFO:teuthology.orchestra.run.smithi099.stdout: "total": 64, 2023-05-31T21:22:58.918 INFO:teuthology.orchestra.run.smithi099.stdout: "complete": 64 2023-05-31T21:22:58.918 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:22:58.919 INFO:teuthology.orchestra.run.smithi099.stdout: "current_time": "2023-05-31T21:22:58Z" 2023-05-31T21:22:58.919 INFO:teuthology.orchestra.run.smithi099.stdout:} 2023-05-31T21:22:58.920 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 452136274422638824 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000044815", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-05-31T21:21:05.078544Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000031499", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-05-31T21:12:40.667485Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000050831", "next_step_marker": "", "total_entries": 55, "pos": 0, "timestamp": "2023-05-31T21:19:03.876722Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000043896", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2023-05-31T21:12:28.529956Z" } } ] }, "full_sync": { "total": 64, "complete": 64 }, "current_time": "2023-05-31T21:22:58Z" } 2023-05-31T21:22:58.920 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000044815', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-05-31T21:21:05.078544Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000031499', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-05-31T21:12:40.667485Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000050831', 'next_step_marker': '', 'total_entries': 55, 'pos': 0, 'timestamp': '2023-05-31T21:19:03.876722Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000043896', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2023-05-31T21:12:28.529956Z'}}] 2023-05-31T21:22:58.921 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000044955', 1: '00000000000000000000:00000000000000031499', 2: '00000000000000000000:00000000000000051111', 3: '00000000000000000000:00000000000000044038'} 2023-05-31T21:22:58.921 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000044815', 1: '00000000000000000000:00000000000000031499', 2: '00000000000000000000:00000000000000050831', 3: '00000000000000000000:00000000000000043896'} 2023-05-31T21:22:58.921 WARNING:rgw_multi.tests:data of zone a2 behind zone a1: shard=0 master=00000000000000000000:00000000000000044955 target=00000000000000000000:00000000000000044815, shard=2 master=00000000000000000000:00000000000000051111 target=00000000000000000000:00000000000000050831, shard=3 master=00000000000000000000:00000000000000044038 target=00000000000000000000:00000000000000043896 2023-05-31T21:23:03.923 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-31T21:23:03.924 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-31T21:23:03.925 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-05-31T21:23:03.986 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2023-05-31T21:23:03.987 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2023-05-31T21:23:04.048 INFO:teuthology.orchestra.run.smithi099.stdout:{ 2023-05-31T21:23:04.048 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_status": { 2023-05-31T21:23:04.048 INFO:teuthology.orchestra.run.smithi099.stdout: "info": { 2023-05-31T21:23:04.049 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "sync", 2023-05-31T21:23:04.049 INFO:teuthology.orchestra.run.smithi099.stdout: "num_shards": 4, 2023-05-31T21:23:04.049 INFO:teuthology.orchestra.run.smithi099.stdout: "instance_id": 452136274422638824 2023-05-31T21:23:04.049 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:23:04.049 INFO:teuthology.orchestra.run.smithi099.stdout: "markers": [ 2023-05-31T21:23:04.049 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T21:23:04.050 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 0, 2023-05-31T21:23:04.050 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T21:23:04.050 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "incremental-sync", 2023-05-31T21:23:04.050 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "00000000000000000000:00000000000000044815", 2023-05-31T21:23:04.050 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T21:23:04.050 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2023-05-31T21:23:04.051 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T21:23:04.051 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2023-05-31T21:21:05.078544Z" 2023-05-31T21:23:04.051 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T21:23:04.051 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:23:04.051 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T21:23:04.051 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 1, 2023-05-31T21:23:04.052 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T21:23:04.052 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "incremental-sync", 2023-05-31T21:23:04.052 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "00000000000000000000:00000000000000031499", 2023-05-31T21:23:04.057 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T21:23:04.057 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2023-05-31T21:23:04.057 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T21:23:04.058 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2023-05-31T21:12:40.667485Z" 2023-05-31T21:23:04.058 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T21:23:04.058 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:23:04.059 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T21:23:04.059 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 2, 2023-05-31T21:23:04.059 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T21:23:04.059 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "incremental-sync", 2023-05-31T21:23:04.060 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "00000000000000000000:00000000000000050831", 2023-05-31T21:23:04.060 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T21:23:04.060 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 55, 2023-05-31T21:23:04.061 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T21:23:04.062 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2023-05-31T21:19:03.876722Z" 2023-05-31T21:23:04.062 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T21:23:04.062 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:23:04.062 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T21:23:04.062 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 3, 2023-05-31T21:23:04.063 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T21:23:04.063 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "incremental-sync", 2023-05-31T21:23:04.063 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "00000000000000000000:00000000000000043896", 2023-05-31T21:23:04.063 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T21:23:04.063 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 9, 2023-05-31T21:23:04.063 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T21:23:04.063 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2023-05-31T21:12:28.529956Z" 2023-05-31T21:23:04.064 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T21:23:04.064 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T21:23:04.064 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2023-05-31T21:23:04.064 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:23:04.064 INFO:teuthology.orchestra.run.smithi099.stdout: "full_sync": { 2023-05-31T21:23:04.064 INFO:teuthology.orchestra.run.smithi099.stdout: "total": 64, 2023-05-31T21:23:04.065 INFO:teuthology.orchestra.run.smithi099.stdout: "complete": 64 2023-05-31T21:23:04.065 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:23:04.065 INFO:teuthology.orchestra.run.smithi099.stdout: "current_time": "2023-05-31T21:23:04Z" 2023-05-31T21:23:04.065 INFO:teuthology.orchestra.run.smithi099.stdout:} 2023-05-31T21:23:04.066 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 452136274422638824 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000044815", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-05-31T21:21:05.078544Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000031499", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-05-31T21:12:40.667485Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000050831", "next_step_marker": "", "total_entries": 55, "pos": 0, "timestamp": "2023-05-31T21:19:03.876722Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000043896", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2023-05-31T21:12:28.529956Z" } } ] }, "full_sync": { "total": 64, "complete": 64 }, "current_time": "2023-05-31T21:23:04Z" } 2023-05-31T21:23:04.067 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000044815', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-05-31T21:21:05.078544Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000031499', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-05-31T21:12:40.667485Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000050831', 'next_step_marker': '', 'total_entries': 55, 'pos': 0, 'timestamp': '2023-05-31T21:19:03.876722Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000043896', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2023-05-31T21:12:28.529956Z'}}] 2023-05-31T21:23:04.067 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000044955', 1: '00000000000000000000:00000000000000031499', 2: '00000000000000000000:00000000000000051111', 3: '00000000000000000000:00000000000000044038'} 2023-05-31T21:23:04.067 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000044815', 1: '00000000000000000000:00000000000000031499', 2: '00000000000000000000:00000000000000050831', 3: '00000000000000000000:00000000000000043896'} 2023-05-31T21:23:04.068 WARNING:rgw_multi.tests:data of zone a2 behind zone a1: shard=0 master=00000000000000000000:00000000000000044955 target=00000000000000000000:00000000000000044815, shard=2 master=00000000000000000000:00000000000000051111 target=00000000000000000000:00000000000000050831, shard=3 master=00000000000000000000:00000000000000044038 target=00000000000000000000:00000000000000043896 2023-05-31T21:23:07.843 INFO:tasks.rgw.c2.client.1.smithi112.stdout:2023-05-31T21:23:07.840+0000 7fecb0821700 -1 WARNING: The OSD cluster is overloaded and struggling to complete ops. You need more capacity to serve this level of demand. 2023-05-31T21:23:09.069 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-31T21:23:09.070 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-31T21:23:09.071 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-05-31T21:23:09.131 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2023-05-31T21:23:09.131 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2023-05-31T21:23:09.184 INFO:teuthology.orchestra.run.smithi099.stdout:{ 2023-05-31T21:23:09.184 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_status": { 2023-05-31T21:23:09.185 INFO:teuthology.orchestra.run.smithi099.stdout: "info": { 2023-05-31T21:23:09.185 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "sync", 2023-05-31T21:23:09.185 INFO:teuthology.orchestra.run.smithi099.stdout: "num_shards": 4, 2023-05-31T21:23:09.186 INFO:teuthology.orchestra.run.smithi099.stdout: "instance_id": 452136274422638824 2023-05-31T21:23:09.186 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:23:09.186 INFO:teuthology.orchestra.run.smithi099.stdout: "markers": [ 2023-05-31T21:23:09.187 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T21:23:09.187 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 0, 2023-05-31T21:23:09.187 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T21:23:09.188 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "incremental-sync", 2023-05-31T21:23:09.189 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "00000000000000000000:00000000000000044955", 2023-05-31T21:23:09.189 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T21:23:09.189 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2023-05-31T21:23:09.190 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T21:23:09.190 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2023-05-31T21:21:35.807414Z" 2023-05-31T21:23:09.190 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T21:23:09.191 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:23:09.191 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T21:23:09.191 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 1, 2023-05-31T21:23:09.192 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T21:23:09.192 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "incremental-sync", 2023-05-31T21:23:09.192 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "00000000000000000000:00000000000000031499", 2023-05-31T21:23:09.193 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T21:23:09.193 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2023-05-31T21:23:09.193 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T21:23:09.193 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2023-05-31T21:12:40.667485Z" 2023-05-31T21:23:09.194 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T21:23:09.194 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:23:09.194 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T21:23:09.195 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 2, 2023-05-31T21:23:09.195 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T21:23:09.195 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "incremental-sync", 2023-05-31T21:23:09.196 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "00000000000000000000:00000000000000051111", 2023-05-31T21:23:09.196 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T21:23:09.196 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 55, 2023-05-31T21:23:09.197 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T21:23:09.197 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2023-05-31T21:21:35.866368Z" 2023-05-31T21:23:09.198 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T21:23:09.198 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:23:09.199 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T21:23:09.199 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 3, 2023-05-31T21:23:09.199 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T21:23:09.200 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "incremental-sync", 2023-05-31T21:23:09.200 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "00000000000000000000:00000000000000044038", 2023-05-31T21:23:09.200 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T21:23:09.201 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 9, 2023-05-31T21:23:09.201 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T21:23:09.201 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2023-05-31T21:21:35.912562Z" 2023-05-31T21:23:09.202 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T21:23:09.202 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T21:23:09.202 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2023-05-31T21:23:09.203 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:23:09.203 INFO:teuthology.orchestra.run.smithi099.stdout: "full_sync": { 2023-05-31T21:23:09.203 INFO:teuthology.orchestra.run.smithi099.stdout: "total": 64, 2023-05-31T21:23:09.204 INFO:teuthology.orchestra.run.smithi099.stdout: "complete": 64 2023-05-31T21:23:09.204 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:23:09.205 INFO:teuthology.orchestra.run.smithi099.stdout: "current_time": "2023-05-31T21:23:09Z" 2023-05-31T21:23:09.205 INFO:teuthology.orchestra.run.smithi099.stdout:} 2023-05-31T21:23:09.207 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 452136274422638824 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000044955", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-05-31T21:21:35.807414Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000031499", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-05-31T21:12:40.667485Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000051111", "next_step_marker": "", "total_entries": 55, "pos": 0, "timestamp": "2023-05-31T21:21:35.866368Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000044038", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2023-05-31T21:21:35.912562Z" } } ] }, "full_sync": { "total": 64, "complete": 64 }, "current_time": "2023-05-31T21:23:09Z" } 2023-05-31T21:23:09.207 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000044955', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-05-31T21:21:35.807414Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000031499', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-05-31T21:12:40.667485Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000051111', 'next_step_marker': '', 'total_entries': 55, 'pos': 0, 'timestamp': '2023-05-31T21:21:35.866368Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000044038', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2023-05-31T21:21:35.912562Z'}}] 2023-05-31T21:23:09.208 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000044955', 1: '00000000000000000000:00000000000000031499', 2: '00000000000000000000:00000000000000051111', 3: '00000000000000000000:00000000000000044038'} 2023-05-31T21:23:09.208 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000044955', 1: '00000000000000000000:00000000000000031499', 2: '00000000000000000000:00000000000000051111', 3: '00000000000000000000:00000000000000044038'} 2023-05-31T21:23:09.209 INFO:rgw_multi.tests:finished data checkpoint for target_zone=a2 source_zone=a1 2023-05-31T21:23:09.209 DEBUG:boto:path=/pklxkd-68/ 2023-05-31T21:23:09.210 DEBUG:boto:auth_path=/pklxkd-68/ 2023-05-31T21:23:09.210 DEBUG:boto:Method: HEAD 2023-05-31T21:23:09.210 DEBUG:boto:Path: /pklxkd-68/ 2023-05-31T21:23:09.211 DEBUG:boto:Data: 2023-05-31T21:23:09.211 DEBUG:boto:Headers: {} 2023-05-31T21:23:09.211 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T21:23:09.212 DEBUG:boto:Port: 8001 2023-05-31T21:23:09.212 DEBUG:boto:Params: {} 2023-05-31T21:23:09.213 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8001} 2023-05-31T21:23:09.213 DEBUG:boto:Token: None 2023-05-31T21:23:09.214 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 21:23:09 GMT /pklxkd-68/ 2023-05-31T21:23:09.214 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:VSZdDti3zILUSUUi9mHLWMzr1/w= 2023-05-31T21:23:09.214 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 21:23:09 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:VSZdDti3zILUSUUi9mHLWMzr1/w=', 'Content-Length': '0'} 2023-05-31T21:23:09.219 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '2'), ('X-RGW-Bytes-Used', '12'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000c55aa408ee276086-006477babd-2423-a2'), ('Content-Length', '0'), ('Date', 'Wed, 31 May 2023 21:23:09 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:23:09.220 DEBUG:boto:path=/pklxkd-68/obj1 2023-05-31T21:23:09.220 DEBUG:boto:auth_path=/pklxkd-68/obj1 2023-05-31T21:23:09.221 DEBUG:boto:Method: HEAD 2023-05-31T21:23:09.221 DEBUG:boto:Path: /pklxkd-68/obj1 2023-05-31T21:23:09.222 DEBUG:boto:Data: 2023-05-31T21:23:09.222 DEBUG:boto:Headers: {} 2023-05-31T21:23:09.222 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T21:23:09.223 DEBUG:boto:Port: 8001 2023-05-31T21:23:09.223 DEBUG:boto:Params: {} 2023-05-31T21:23:09.223 DEBUG:boto:Token: None 2023-05-31T21:23:09.224 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 21:23:09 GMT /pklxkd-68/obj1 2023-05-31T21:23:09.224 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:gNfHIUa1ogwkHtXeBd5lOWoE1Cc= 2023-05-31T21:23:09.225 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 21:23:09 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:gNfHIUa1ogwkHtXeBd5lOWoE1Cc=', 'Content-Length': '0'} 2023-05-31T21:23:09.227 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 21:21:35 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000064f451063416ebe-006477babd-2423-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 21:23:09 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:23:09.227 DEBUG:boto:path=/pklxkd-68/obj1 2023-05-31T21:23:09.228 DEBUG:boto:auth_path=/pklxkd-68/obj1 2023-05-31T21:23:09.228 DEBUG:boto:Method: GET 2023-05-31T21:23:09.229 DEBUG:boto:Path: /pklxkd-68/obj1 2023-05-31T21:23:09.229 DEBUG:boto:Data: 2023-05-31T21:23:09.230 DEBUG:boto:Headers: {} 2023-05-31T21:23:09.230 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T21:23:09.231 DEBUG:boto:Port: 8001 2023-05-31T21:23:09.231 DEBUG:boto:Params: {} 2023-05-31T21:23:09.232 DEBUG:boto:Token: None 2023-05-31T21:23:09.233 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 21:23:09 GMT /pklxkd-68/obj1 2023-05-31T21:23:09.234 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:ffZk0/JGsZwpTv7KaXovK4n7aGA= 2023-05-31T21:23:09.235 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 21:23:09 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:ffZk0/JGsZwpTv7KaXovK4n7aGA=', 'Content-Length': '0'} 2023-05-31T21:23:09.237 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 21:21:35 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000bd9595fb58ffdcda-006477babd-2423-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 21:23:09 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:23:09.239 DEBUG:boto:path=/pklxkd-68/obj2 2023-05-31T21:23:09.239 DEBUG:boto:auth_path=/pklxkd-68/obj2 2023-05-31T21:23:09.240 DEBUG:boto:Method: HEAD 2023-05-31T21:23:09.240 DEBUG:boto:Path: /pklxkd-68/obj2 2023-05-31T21:23:09.240 DEBUG:boto:Data: 2023-05-31T21:23:09.241 DEBUG:boto:Headers: {} 2023-05-31T21:23:09.241 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T21:23:09.241 DEBUG:boto:Port: 8001 2023-05-31T21:23:09.242 DEBUG:boto:Params: {} 2023-05-31T21:23:09.242 DEBUG:boto:Token: None 2023-05-31T21:23:09.243 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 21:23:09 GMT /pklxkd-68/obj2 2023-05-31T21:23:09.243 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:lBtr+2sQG/Fk8BzkadV/5iEzqEI= 2023-05-31T21:23:09.244 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 21:23:09 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:lBtr+2sQG/Fk8BzkadV/5iEzqEI=', 'Content-Length': '0'} 2023-05-31T21:23:09.245 DEBUG:boto:Response headers: [('Content-Length', '219'), ('x-amz-request-id', 'tx0000091a16af7b267f2f6-006477babd-2423-a2'), ('Accept-Ranges', 'bytes'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 21:23:09 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:23:09.248 INFO:tasks.rgw_multisite_tests:test_sync_single_bucket: ... FAIL 2023-05-31T21:23:09.249 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-31T21:23:09.249 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-31T21:23:09.250 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-05-31T21:23:09.299 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2023-05-31T21:23:09.300 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2023-05-31T21:23:09.335 INFO:teuthology.orchestra.run.smithi099.stderr:No --period given, using current period=2e6123d2-d695-4350-9a01-61a455bc782b 2023-05-31T21:23:09.349 INFO:teuthology.orchestra.run.smithi099.stdout:[ 2023-05-31T21:23:09.350 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T21:23:09.350 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1685566501.757183_260.1", 2023-05-31T21:23:09.350 INFO:teuthology.orchestra.run.smithi099.stdout: "last_update": "2023-05-31T20:55:01.757183Z" 2023-05-31T21:23:09.350 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:23:09.350 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T21:23:09.351 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1685566501.735717_251.1", 2023-05-31T21:23:09.351 INFO:teuthology.orchestra.run.smithi099.stdout: "last_update": "2023-05-31T20:55:01.735717Z" 2023-05-31T21:23:09.351 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:23:09.351 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T21:23:09.351 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1685562970.274572_93.1", 2023-05-31T21:23:09.351 INFO:teuthology.orchestra.run.smithi099.stdout: "last_update": "2023-05-31T19:56:10.274572Z" 2023-05-31T21:23:09.352 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:23:09.352 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T21:23:09.352 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1685568089.268814_960.1", 2023-05-31T21:23:09.352 INFO:teuthology.orchestra.run.smithi099.stdout: "last_update": "2023-05-31T21:21:29.268814Z" 2023-05-31T21:23:09.352 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T21:23:09.352 INFO:teuthology.orchestra.run.smithi099.stdout:] 2023-05-31T21:23:09.353 DEBUG:rgw_multi.tests:master meta markers={0: '1_1685566501.757183_260.1', 1: '1_1685566501.735717_251.1', 2: '1_1685562970.274572_93.1', 3: '1_1685568089.268814_960.1'} 2023-05-31T21:23:09.354 INFO:rgw_multi.tests:starting meta checkpoint for zone=a2 2023-05-31T21:23:09.354 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-31T21:23:09.355 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-31T21:23:09.355 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-05-31T21:23:09.405 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2023-05-31T21:23:09.405 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2023-05-31T21:23:09.438 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T21:23:09.433+0000 7fd52c934780 20 RGW-SYNC:meta: read sync status 2023-05-31T21:23:09.439 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T21:23:09.434+0000 7fd52c934780 20 RGW-SYNC:meta: read sync status 2023-05-31T21:23:09.453 INFO:teuthology.orchestra.run.smithi099.stdout:{ 2023-05-31T21:23:09.454 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_status": { 2023-05-31T21:23:09.454 INFO:teuthology.orchestra.run.smithi099.stdout: "info": { 2023-05-31T21:23:09.454 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "sync", 2023-05-31T21:23:09.454 INFO:teuthology.orchestra.run.smithi099.stdout: "num_shards": 4, 2023-05-31T21:23:09.454 INFO:teuthology.orchestra.run.smithi099.stdout: "period": "2e6123d2-d695-4350-9a01-61a455bc782b", 2023-05-31T21:23:09.455 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2023-05-31T21:23:09.455 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:23:09.455 INFO:teuthology.orchestra.run.smithi099.stdout: "markers": [ 2023-05-31T21:23:09.455 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T21:23:09.455 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 0, 2023-05-31T21:23:09.455 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T21:23:09.455 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2023-05-31T21:23:09.456 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1685566501.757183_260.1", 2023-05-31T21:23:09.456 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T21:23:09.456 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 1, 2023-05-31T21:23:09.456 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T21:23:09.456 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2023-05-31T20:55:01.757183Z", 2023-05-31T21:23:09.456 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2023-05-31T21:23:09.457 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T21:23:09.457 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:23:09.457 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T21:23:09.457 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 1, 2023-05-31T21:23:09.457 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T21:23:09.457 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2023-05-31T21:23:09.457 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1685566501.735717_251.1", 2023-05-31T21:23:09.458 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T21:23:09.458 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2023-05-31T21:23:09.460 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T21:23:09.460 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2023-05-31T20:55:01.735717Z", 2023-05-31T21:23:09.460 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2023-05-31T21:23:09.461 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T21:23:09.461 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:23:09.461 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T21:23:09.461 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 2, 2023-05-31T21:23:09.461 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T21:23:09.461 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2023-05-31T21:23:09.462 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1685562970.274572_93.1", 2023-05-31T21:23:09.462 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T21:23:09.462 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2023-05-31T21:23:09.463 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T21:23:09.463 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2023-05-31T19:56:10.274572Z", 2023-05-31T21:23:09.463 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2023-05-31T21:23:09.463 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T21:23:09.463 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:23:09.463 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T21:23:09.464 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 3, 2023-05-31T21:23:09.464 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T21:23:09.464 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2023-05-31T21:23:09.464 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1685568089.268814_960.1", 2023-05-31T21:23:09.465 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T21:23:09.465 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2023-05-31T21:23:09.465 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T21:23:09.465 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2023-05-31T21:21:29.268814Z", 2023-05-31T21:23:09.465 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2023-05-31T21:23:09.465 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T21:23:09.466 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T21:23:09.466 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2023-05-31T21:23:09.466 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:23:09.466 INFO:teuthology.orchestra.run.smithi099.stdout: "full_sync": { 2023-05-31T21:23:09.466 INFO:teuthology.orchestra.run.smithi099.stdout: "total": 1, 2023-05-31T21:23:09.467 INFO:teuthology.orchestra.run.smithi099.stdout: "complete": 1 2023-05-31T21:23:09.467 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:23:09.467 INFO:teuthology.orchestra.run.smithi099.stdout: "current_time": "2023-05-31T21:23:09Z" 2023-05-31T21:23:09.467 INFO:teuthology.orchestra.run.smithi099.stdout:} 2023-05-31T21:23:09.468 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "2e6123d2-d695-4350-9a01-61a455bc782b", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1685566501.757183_260.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-05-31T20:55:01.757183Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1685566501.735717_251.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-05-31T20:55:01.735717Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1685562970.274572_93.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-05-31T19:56:10.274572Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1685568089.268814_960.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-05-31T21:21:29.268814Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-05-31T21:23:09Z" } 2023-05-31T21:23:09.469 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1685566501.757183_260.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-05-31T20:55:01.757183Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1685566501.735717_251.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-05-31T20:55:01.735717Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1685562970.274572_93.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-05-31T19:56:10.274572Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1685568089.268814_960.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-05-31T21:21:29.268814Z', 'realm_epoch': 2}}] 2023-05-31T21:23:09.469 DEBUG:rgw_multi.tests:log_status={0: '1_1685566501.757183_260.1', 1: '1_1685566501.735717_251.1', 2: '1_1685562970.274572_93.1', 3: '1_1685568089.268814_960.1'} 2023-05-31T21:23:09.470 DEBUG:rgw_multi.tests:sync_status={0: '1_1685566501.757183_260.1', 1: '1_1685566501.735717_251.1', 2: '1_1685562970.274572_93.1', 3: '1_1685568089.268814_960.1'} 2023-05-31T21:23:09.470 INFO:rgw_multi.tests:finish meta checkpoint for zone=a2 2023-05-31T21:23:09.471 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'policy', 'get', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-05-31T21:23:09.471 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'policy', 'get', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-05-31T21:23:09.472 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync policy get --cluster c1 --debug-rgw 0 --debug-ms 0 2023-05-31T21:23:09.518 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2023-05-31T21:23:09.518 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2023-05-31T21:23:09.584 INFO:teuthology.orchestra.run.smithi099.stdout:{"groups":[{"id":"sync-group","data_flow":{"symmetrical":[{"id":"sync-flow1","zones":["a1","a2"]}]},"pipes":[{"id":"sync-pipe","source":{"bucket":"pklxkd-68","zones":["a1","a2"]},"dest":{"bucket":"pklxkd-68","zones":["a1","a2"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":""}}],"status":"enabled"}]} 2023-05-31T21:23:09.584 INFO:rgw_multi.tests:create bucket zone=a1 name=pklxkd-70 2023-05-31T21:23:09.585 DEBUG:boto:path=/pklxkd-70/ 2023-05-31T21:23:09.585 DEBUG:boto:auth_path=/pklxkd-70/ 2023-05-31T21:23:09.586 DEBUG:boto:Method: PUT 2023-05-31T21:23:09.586 DEBUG:boto:Path: /pklxkd-70/ 2023-05-31T21:23:09.586 DEBUG:boto:Data: 2023-05-31T21:23:09.587 DEBUG:boto:Headers: {} 2023-05-31T21:23:09.587 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T21:23:09.588 DEBUG:boto:Port: 8000 2023-05-31T21:23:09.588 DEBUG:boto:Params: {} 2023-05-31T21:23:09.589 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2023-05-31T21:23:09.589 DEBUG:boto:Token: None 2023-05-31T21:23:09.590 DEBUG:boto:StringToSign: PUT Wed, 31 May 2023 21:23:09 GMT /pklxkd-70/ 2023-05-31T21:23:09.592 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:6Kxvgi+eMcrsuDFk4deKTTDEl3U= 2023-05-31T21:23:09.592 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 21:23:09 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:6Kxvgi+eMcrsuDFk4deKTTDEl3U=', 'Content-Length': '0'} 2023-05-31T21:23:09.623 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000d5b01357decf0b1d-006477babd-242d-a1'), ('Content-Length', '0'), ('Date', 'Wed, 31 May 2023 21:23:09 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:23:09.624 INFO:rgw_multi.tests:create bucket zone=a1 name=pklxkd-71 2023-05-31T21:23:09.624 DEBUG:boto:path=/pklxkd-71/ 2023-05-31T21:23:09.624 DEBUG:boto:auth_path=/pklxkd-71/ 2023-05-31T21:23:09.625 DEBUG:boto:Method: PUT 2023-05-31T21:23:09.626 DEBUG:boto:Path: /pklxkd-71/ 2023-05-31T21:23:09.627 DEBUG:boto:Data: 2023-05-31T21:23:09.628 DEBUG:boto:Headers: {} 2023-05-31T21:23:09.628 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T21:23:09.629 DEBUG:boto:Port: 8000 2023-05-31T21:23:09.629 DEBUG:boto:Params: {} 2023-05-31T21:23:09.630 DEBUG:boto:Token: None 2023-05-31T21:23:09.631 DEBUG:boto:StringToSign: PUT Wed, 31 May 2023 21:23:09 GMT /pklxkd-71/ 2023-05-31T21:23:09.631 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:rvF9HQ3SF9ryZIc6Wua40DVjz+c= 2023-05-31T21:23:09.632 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 21:23:09 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:rvF9HQ3SF9ryZIc6Wua40DVjz+c=', 'Content-Length': '0'} 2023-05-31T21:23:09.661 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000c5f4ed1a8e2474b3-006477babd-242d-a1'), ('Content-Length', '0'), ('Date', 'Wed, 31 May 2023 21:23:09 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:23:09.661 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-31T21:23:09.662 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-31T21:23:09.662 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-05-31T21:23:09.711 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2023-05-31T21:23:09.711 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2023-05-31T21:23:09.750 INFO:teuthology.orchestra.run.smithi099.stderr:No --period given, using current period=2e6123d2-d695-4350-9a01-61a455bc782b 2023-05-31T21:23:09.778 INFO:teuthology.orchestra.run.smithi099.stdout:[ 2023-05-31T21:23:09.778 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T21:23:09.779 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1685566501.757183_260.1", 2023-05-31T21:23:09.779 INFO:teuthology.orchestra.run.smithi099.stdout: "last_update": "2023-05-31T20:55:01.757183Z" 2023-05-31T21:23:09.780 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:23:09.780 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T21:23:09.781 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1685566501.735717_251.1", 2023-05-31T21:23:09.782 INFO:teuthology.orchestra.run.smithi099.stdout: "last_update": "2023-05-31T20:55:01.735717Z" 2023-05-31T21:23:09.782 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:23:09.783 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T21:23:09.783 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1685562970.274572_93.1", 2023-05-31T21:23:09.783 INFO:teuthology.orchestra.run.smithi099.stdout: "last_update": "2023-05-31T19:56:10.274572Z" 2023-05-31T21:23:09.784 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:23:09.784 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T21:23:09.784 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1685568189.652614_974.1", 2023-05-31T21:23:09.784 INFO:teuthology.orchestra.run.smithi099.stdout: "last_update": "2023-05-31T21:23:09.652614Z" 2023-05-31T21:23:09.784 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T21:23:09.784 INFO:teuthology.orchestra.run.smithi099.stdout:] 2023-05-31T21:23:09.785 DEBUG:rgw_multi.tests:master meta markers={0: '1_1685566501.757183_260.1', 1: '1_1685566501.735717_251.1', 2: '1_1685562970.274572_93.1', 3: '1_1685568189.652614_974.1'} 2023-05-31T21:23:09.786 INFO:rgw_multi.tests:starting meta checkpoint for zone=a2 2023-05-31T21:23:09.786 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-31T21:23:09.787 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-31T21:23:09.787 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-05-31T21:23:09.836 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2023-05-31T21:23:09.836 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2023-05-31T21:23:09.887 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T21:23:09.883+0000 7f81add60780 20 RGW-SYNC:meta: read sync status 2023-05-31T21:23:09.888 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T21:23:09.884+0000 7f81add60780 20 RGW-SYNC:meta: read sync status 2023-05-31T21:23:09.909 INFO:teuthology.orchestra.run.smithi099.stdout:{ 2023-05-31T21:23:09.909 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_status": { 2023-05-31T21:23:09.910 INFO:teuthology.orchestra.run.smithi099.stdout: "info": { 2023-05-31T21:23:09.910 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "sync", 2023-05-31T21:23:09.910 INFO:teuthology.orchestra.run.smithi099.stdout: "num_shards": 4, 2023-05-31T21:23:09.910 INFO:teuthology.orchestra.run.smithi099.stdout: "period": "2e6123d2-d695-4350-9a01-61a455bc782b", 2023-05-31T21:23:09.910 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2023-05-31T21:23:09.910 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:23:09.911 INFO:teuthology.orchestra.run.smithi099.stdout: "markers": [ 2023-05-31T21:23:09.911 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T21:23:09.911 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 0, 2023-05-31T21:23:09.911 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T21:23:09.911 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2023-05-31T21:23:09.911 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1685566501.757183_260.1", 2023-05-31T21:23:09.912 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T21:23:09.912 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 1, 2023-05-31T21:23:09.912 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T21:23:09.912 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2023-05-31T20:55:01.757183Z", 2023-05-31T21:23:09.912 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2023-05-31T21:23:09.912 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T21:23:09.913 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:23:09.913 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T21:23:09.913 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 1, 2023-05-31T21:23:09.913 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T21:23:09.913 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2023-05-31T21:23:09.913 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1685566501.735717_251.1", 2023-05-31T21:23:09.913 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T21:23:09.914 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2023-05-31T21:23:09.914 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T21:23:09.914 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2023-05-31T20:55:01.735717Z", 2023-05-31T21:23:09.914 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2023-05-31T21:23:09.914 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T21:23:09.914 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:23:09.915 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T21:23:09.915 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 2, 2023-05-31T21:23:09.915 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T21:23:09.915 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2023-05-31T21:23:09.915 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1685562970.274572_93.1", 2023-05-31T21:23:09.915 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T21:23:09.916 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2023-05-31T21:23:09.916 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T21:23:09.917 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2023-05-31T19:56:10.274572Z", 2023-05-31T21:23:09.917 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2023-05-31T21:23:09.917 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T21:23:09.917 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:23:09.918 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T21:23:09.919 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 3, 2023-05-31T21:23:09.919 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T21:23:09.919 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2023-05-31T21:23:09.919 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1685568089.268814_960.1", 2023-05-31T21:23:09.920 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T21:23:09.920 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2023-05-31T21:23:09.920 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T21:23:09.920 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2023-05-31T21:21:29.268814Z", 2023-05-31T21:23:09.920 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2023-05-31T21:23:09.920 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T21:23:09.921 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T21:23:09.921 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2023-05-31T21:23:09.921 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:23:09.921 INFO:teuthology.orchestra.run.smithi099.stdout: "full_sync": { 2023-05-31T21:23:09.922 INFO:teuthology.orchestra.run.smithi099.stdout: "total": 1, 2023-05-31T21:23:09.922 INFO:teuthology.orchestra.run.smithi099.stdout: "complete": 1 2023-05-31T21:23:09.922 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:23:09.922 INFO:teuthology.orchestra.run.smithi099.stdout: "current_time": "2023-05-31T21:23:09Z" 2023-05-31T21:23:09.922 INFO:teuthology.orchestra.run.smithi099.stdout:} 2023-05-31T21:23:09.923 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "2e6123d2-d695-4350-9a01-61a455bc782b", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1685566501.757183_260.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-05-31T20:55:01.757183Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1685566501.735717_251.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-05-31T20:55:01.735717Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1685562970.274572_93.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-05-31T19:56:10.274572Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1685568089.268814_960.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-05-31T21:21:29.268814Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-05-31T21:23:09Z" } 2023-05-31T21:23:09.924 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1685566501.757183_260.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-05-31T20:55:01.757183Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1685566501.735717_251.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-05-31T20:55:01.735717Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1685562970.274572_93.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-05-31T19:56:10.274572Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1685568089.268814_960.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-05-31T21:21:29.268814Z', 'realm_epoch': 2}}] 2023-05-31T21:23:09.924 DEBUG:rgw_multi.tests:log_status={0: '1_1685566501.757183_260.1', 1: '1_1685566501.735717_251.1', 2: '1_1685562970.274572_93.1', 3: '1_1685568189.652614_974.1'} 2023-05-31T21:23:09.924 DEBUG:rgw_multi.tests:sync_status={0: '1_1685566501.757183_260.1', 1: '1_1685566501.735717_251.1', 2: '1_1685562970.274572_93.1', 3: '1_1685568089.268814_960.1'} 2023-05-31T21:23:09.925 WARNING:rgw_multi.tests:zone a2 behind master: shard=3 master=1_1685568189.652614_974.1 target=1_1685568089.268814_960.1 2023-05-31T21:23:14.926 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-31T21:23:14.926 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-31T21:23:14.927 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-05-31T21:23:14.985 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2023-05-31T21:23:14.985 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2023-05-31T21:23:15.031 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T21:23:15.026+0000 7f94dac64780 20 RGW-SYNC:meta: read sync status 2023-05-31T21:23:15.031 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T21:23:15.027+0000 7f94dac64780 20 RGW-SYNC:meta: read sync status 2023-05-31T21:23:15.048 INFO:teuthology.orchestra.run.smithi099.stdout:{ 2023-05-31T21:23:15.049 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_status": { 2023-05-31T21:23:15.049 INFO:teuthology.orchestra.run.smithi099.stdout: "info": { 2023-05-31T21:23:15.049 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "sync", 2023-05-31T21:23:15.049 INFO:teuthology.orchestra.run.smithi099.stdout: "num_shards": 4, 2023-05-31T21:23:15.049 INFO:teuthology.orchestra.run.smithi099.stdout: "period": "2e6123d2-d695-4350-9a01-61a455bc782b", 2023-05-31T21:23:15.050 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2023-05-31T21:23:15.050 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:23:15.050 INFO:teuthology.orchestra.run.smithi099.stdout: "markers": [ 2023-05-31T21:23:15.050 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T21:23:15.050 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 0, 2023-05-31T21:23:15.050 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T21:23:15.050 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2023-05-31T21:23:15.051 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1685566501.757183_260.1", 2023-05-31T21:23:15.051 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T21:23:15.051 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 1, 2023-05-31T21:23:15.051 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T21:23:15.051 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2023-05-31T20:55:01.757183Z", 2023-05-31T21:23:15.051 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2023-05-31T21:23:15.052 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T21:23:15.052 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:23:15.052 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T21:23:15.052 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 1, 2023-05-31T21:23:15.052 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T21:23:15.052 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2023-05-31T21:23:15.052 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1685566501.735717_251.1", 2023-05-31T21:23:15.053 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T21:23:15.053 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2023-05-31T21:23:15.054 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T21:23:15.054 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2023-05-31T20:55:01.735717Z", 2023-05-31T21:23:15.054 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2023-05-31T21:23:15.054 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T21:23:15.055 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:23:15.055 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T21:23:15.055 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 2, 2023-05-31T21:23:15.055 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T21:23:15.055 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2023-05-31T21:23:15.055 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1685562970.274572_93.1", 2023-05-31T21:23:15.055 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T21:23:15.056 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2023-05-31T21:23:15.056 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T21:23:15.056 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2023-05-31T19:56:10.274572Z", 2023-05-31T21:23:15.057 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2023-05-31T21:23:15.057 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T21:23:15.057 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:23:15.057 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T21:23:15.057 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 3, 2023-05-31T21:23:15.057 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T21:23:15.057 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2023-05-31T21:23:15.058 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1685568089.268814_960.1", 2023-05-31T21:23:15.058 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T21:23:15.058 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2023-05-31T21:23:15.058 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T21:23:15.058 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2023-05-31T21:21:29.268814Z", 2023-05-31T21:23:15.058 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2023-05-31T21:23:15.058 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T21:23:15.059 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T21:23:15.059 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2023-05-31T21:23:15.059 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:23:15.059 INFO:teuthology.orchestra.run.smithi099.stdout: "full_sync": { 2023-05-31T21:23:15.059 INFO:teuthology.orchestra.run.smithi099.stdout: "total": 1, 2023-05-31T21:23:15.059 INFO:teuthology.orchestra.run.smithi099.stdout: "complete": 1 2023-05-31T21:23:15.060 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:23:15.060 INFO:teuthology.orchestra.run.smithi099.stdout: "current_time": "2023-05-31T21:23:15Z" 2023-05-31T21:23:15.060 INFO:teuthology.orchestra.run.smithi099.stdout:} 2023-05-31T21:23:15.061 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "2e6123d2-d695-4350-9a01-61a455bc782b", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1685566501.757183_260.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-05-31T20:55:01.757183Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1685566501.735717_251.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-05-31T20:55:01.735717Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1685562970.274572_93.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-05-31T19:56:10.274572Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1685568089.268814_960.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-05-31T21:21:29.268814Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-05-31T21:23:15Z" } 2023-05-31T21:23:15.062 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1685566501.757183_260.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-05-31T20:55:01.757183Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1685566501.735717_251.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-05-31T20:55:01.735717Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1685562970.274572_93.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-05-31T19:56:10.274572Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1685568089.268814_960.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-05-31T21:21:29.268814Z', 'realm_epoch': 2}}] 2023-05-31T21:23:15.063 DEBUG:rgw_multi.tests:log_status={0: '1_1685566501.757183_260.1', 1: '1_1685566501.735717_251.1', 2: '1_1685562970.274572_93.1', 3: '1_1685568189.652614_974.1'} 2023-05-31T21:23:15.063 DEBUG:rgw_multi.tests:sync_status={0: '1_1685566501.757183_260.1', 1: '1_1685566501.735717_251.1', 2: '1_1685562970.274572_93.1', 3: '1_1685568089.268814_960.1'} 2023-05-31T21:23:15.063 WARNING:rgw_multi.tests:zone a2 behind master: shard=3 master=1_1685568189.652614_974.1 target=1_1685568089.268814_960.1 2023-05-31T21:23:20.065 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-31T21:23:20.065 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-31T21:23:20.066 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-05-31T21:23:20.126 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2023-05-31T21:23:20.126 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2023-05-31T21:23:20.172 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T21:23:20.169+0000 7f67dbfd7780 20 RGW-SYNC:meta: read sync status 2023-05-31T21:23:20.173 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T21:23:20.169+0000 7f67dbfd7780 20 RGW-SYNC:meta: read sync status 2023-05-31T21:23:20.191 INFO:teuthology.orchestra.run.smithi099.stdout:{ 2023-05-31T21:23:20.192 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_status": { 2023-05-31T21:23:20.192 INFO:teuthology.orchestra.run.smithi099.stdout: "info": { 2023-05-31T21:23:20.192 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "sync", 2023-05-31T21:23:20.192 INFO:teuthology.orchestra.run.smithi099.stdout: "num_shards": 4, 2023-05-31T21:23:20.193 INFO:teuthology.orchestra.run.smithi099.stdout: "period": "2e6123d2-d695-4350-9a01-61a455bc782b", 2023-05-31T21:23:20.193 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2023-05-31T21:23:20.193 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:23:20.193 INFO:teuthology.orchestra.run.smithi099.stdout: "markers": [ 2023-05-31T21:23:20.193 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T21:23:20.193 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 0, 2023-05-31T21:23:20.193 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T21:23:20.194 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2023-05-31T21:23:20.194 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1685566501.757183_260.1", 2023-05-31T21:23:20.194 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T21:23:20.194 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 1, 2023-05-31T21:23:20.194 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T21:23:20.194 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2023-05-31T20:55:01.757183Z", 2023-05-31T21:23:20.195 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2023-05-31T21:23:20.195 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T21:23:20.195 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:23:20.195 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T21:23:20.195 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 1, 2023-05-31T21:23:20.195 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T21:23:20.195 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2023-05-31T21:23:20.196 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1685566501.735717_251.1", 2023-05-31T21:23:20.196 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T21:23:20.196 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2023-05-31T21:23:20.196 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T21:23:20.196 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2023-05-31T20:55:01.735717Z", 2023-05-31T21:23:20.196 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2023-05-31T21:23:20.196 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T21:23:20.197 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:23:20.197 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T21:23:20.197 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 2, 2023-05-31T21:23:20.197 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T21:23:20.197 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2023-05-31T21:23:20.197 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1685562970.274572_93.1", 2023-05-31T21:23:20.197 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T21:23:20.198 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2023-05-31T21:23:20.198 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T21:23:20.198 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2023-05-31T19:56:10.274572Z", 2023-05-31T21:23:20.198 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2023-05-31T21:23:20.198 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T21:23:20.198 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:23:20.198 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T21:23:20.199 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 3, 2023-05-31T21:23:20.199 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T21:23:20.199 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2023-05-31T21:23:20.199 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1685568089.268814_960.1", 2023-05-31T21:23:20.199 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T21:23:20.199 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2023-05-31T21:23:20.199 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T21:23:20.200 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2023-05-31T21:21:29.268814Z", 2023-05-31T21:23:20.200 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2023-05-31T21:23:20.200 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T21:23:20.200 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T21:23:20.200 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2023-05-31T21:23:20.200 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:23:20.201 INFO:teuthology.orchestra.run.smithi099.stdout: "full_sync": { 2023-05-31T21:23:20.201 INFO:teuthology.orchestra.run.smithi099.stdout: "total": 1, 2023-05-31T21:23:20.201 INFO:teuthology.orchestra.run.smithi099.stdout: "complete": 1 2023-05-31T21:23:20.201 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:23:20.201 INFO:teuthology.orchestra.run.smithi099.stdout: "current_time": "2023-05-31T21:23:20Z" 2023-05-31T21:23:20.201 INFO:teuthology.orchestra.run.smithi099.stdout:} 2023-05-31T21:23:20.202 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "2e6123d2-d695-4350-9a01-61a455bc782b", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1685566501.757183_260.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-05-31T20:55:01.757183Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1685566501.735717_251.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-05-31T20:55:01.735717Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1685562970.274572_93.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-05-31T19:56:10.274572Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1685568089.268814_960.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-05-31T21:21:29.268814Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-05-31T21:23:20Z" } 2023-05-31T21:23:20.203 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1685566501.757183_260.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-05-31T20:55:01.757183Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1685566501.735717_251.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-05-31T20:55:01.735717Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1685562970.274572_93.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-05-31T19:56:10.274572Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1685568089.268814_960.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-05-31T21:21:29.268814Z', 'realm_epoch': 2}}] 2023-05-31T21:23:20.203 DEBUG:rgw_multi.tests:log_status={0: '1_1685566501.757183_260.1', 1: '1_1685566501.735717_251.1', 2: '1_1685562970.274572_93.1', 3: '1_1685568189.652614_974.1'} 2023-05-31T21:23:20.204 DEBUG:rgw_multi.tests:sync_status={0: '1_1685566501.757183_260.1', 1: '1_1685566501.735717_251.1', 2: '1_1685562970.274572_93.1', 3: '1_1685568089.268814_960.1'} 2023-05-31T21:23:20.204 WARNING:rgw_multi.tests:zone a2 behind master: shard=3 master=1_1685568189.652614_974.1 target=1_1685568089.268814_960.1 2023-05-31T21:23:25.205 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-31T21:23:25.206 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-31T21:23:25.207 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-05-31T21:23:25.265 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2023-05-31T21:23:25.265 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2023-05-31T21:23:25.315 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T21:23:25.312+0000 7f24f8315780 20 RGW-SYNC:meta: read sync status 2023-05-31T21:23:25.316 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T21:23:25.313+0000 7f24f8315780 20 RGW-SYNC:meta: read sync status 2023-05-31T21:23:25.339 INFO:teuthology.orchestra.run.smithi099.stdout:{ 2023-05-31T21:23:25.339 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_status": { 2023-05-31T21:23:25.339 INFO:teuthology.orchestra.run.smithi099.stdout: "info": { 2023-05-31T21:23:25.339 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "sync", 2023-05-31T21:23:25.339 INFO:teuthology.orchestra.run.smithi099.stdout: "num_shards": 4, 2023-05-31T21:23:25.340 INFO:teuthology.orchestra.run.smithi099.stdout: "period": "2e6123d2-d695-4350-9a01-61a455bc782b", 2023-05-31T21:23:25.340 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2023-05-31T21:23:25.340 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:23:25.340 INFO:teuthology.orchestra.run.smithi099.stdout: "markers": [ 2023-05-31T21:23:25.340 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T21:23:25.340 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 0, 2023-05-31T21:23:25.341 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T21:23:25.341 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2023-05-31T21:23:25.341 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1685566501.757183_260.1", 2023-05-31T21:23:25.341 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T21:23:25.341 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 1, 2023-05-31T21:23:25.341 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T21:23:25.342 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2023-05-31T20:55:01.757183Z", 2023-05-31T21:23:25.342 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2023-05-31T21:23:25.342 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T21:23:25.342 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:23:25.342 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T21:23:25.342 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 1, 2023-05-31T21:23:25.342 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T21:23:25.343 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2023-05-31T21:23:25.343 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1685566501.735717_251.1", 2023-05-31T21:23:25.343 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T21:23:25.343 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2023-05-31T21:23:25.343 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T21:23:25.343 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2023-05-31T20:55:01.735717Z", 2023-05-31T21:23:25.343 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2023-05-31T21:23:25.344 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T21:23:25.344 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:23:25.344 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T21:23:25.344 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 2, 2023-05-31T21:23:25.344 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T21:23:25.344 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2023-05-31T21:23:25.344 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1685562970.274572_93.1", 2023-05-31T21:23:25.345 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T21:23:25.345 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2023-05-31T21:23:25.345 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T21:23:25.345 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2023-05-31T19:56:10.274572Z", 2023-05-31T21:23:25.345 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2023-05-31T21:23:25.345 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T21:23:25.346 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:23:25.346 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T21:23:25.346 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 3, 2023-05-31T21:23:25.346 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T21:23:25.346 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2023-05-31T21:23:25.346 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1685568089.268814_960.1", 2023-05-31T21:23:25.346 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T21:23:25.347 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2023-05-31T21:23:25.347 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T21:23:25.347 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2023-05-31T21:21:29.268814Z", 2023-05-31T21:23:25.347 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2023-05-31T21:23:25.347 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T21:23:25.347 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T21:23:25.347 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2023-05-31T21:23:25.348 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:23:25.348 INFO:teuthology.orchestra.run.smithi099.stdout: "full_sync": { 2023-05-31T21:23:25.348 INFO:teuthology.orchestra.run.smithi099.stdout: "total": 1, 2023-05-31T21:23:25.348 INFO:teuthology.orchestra.run.smithi099.stdout: "complete": 1 2023-05-31T21:23:25.348 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:23:25.348 INFO:teuthology.orchestra.run.smithi099.stdout: "current_time": "2023-05-31T21:23:25Z" 2023-05-31T21:23:25.349 INFO:teuthology.orchestra.run.smithi099.stdout:} 2023-05-31T21:23:25.350 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "2e6123d2-d695-4350-9a01-61a455bc782b", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1685566501.757183_260.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-05-31T20:55:01.757183Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1685566501.735717_251.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-05-31T20:55:01.735717Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1685562970.274572_93.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-05-31T19:56:10.274572Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1685568089.268814_960.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-05-31T21:21:29.268814Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-05-31T21:23:25Z" } 2023-05-31T21:23:25.350 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1685566501.757183_260.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-05-31T20:55:01.757183Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1685566501.735717_251.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-05-31T20:55:01.735717Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1685562970.274572_93.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-05-31T19:56:10.274572Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1685568089.268814_960.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-05-31T21:21:29.268814Z', 'realm_epoch': 2}}] 2023-05-31T21:23:25.351 DEBUG:rgw_multi.tests:log_status={0: '1_1685566501.757183_260.1', 1: '1_1685566501.735717_251.1', 2: '1_1685562970.274572_93.1', 3: '1_1685568189.652614_974.1'} 2023-05-31T21:23:25.351 DEBUG:rgw_multi.tests:sync_status={0: '1_1685566501.757183_260.1', 1: '1_1685566501.735717_251.1', 2: '1_1685562970.274572_93.1', 3: '1_1685568089.268814_960.1'} 2023-05-31T21:23:25.351 WARNING:rgw_multi.tests:zone a2 behind master: shard=3 master=1_1685568189.652614_974.1 target=1_1685568089.268814_960.1 2023-05-31T21:23:30.353 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-31T21:23:30.353 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-31T21:23:30.354 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-05-31T21:23:30.411 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2023-05-31T21:23:30.411 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2023-05-31T21:23:30.461 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T21:23:30.458+0000 7fc9d3d24780 20 RGW-SYNC:meta: read sync status 2023-05-31T21:23:30.462 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T21:23:30.459+0000 7fc9d3d24780 20 RGW-SYNC:meta: read sync status 2023-05-31T21:23:30.480 INFO:teuthology.orchestra.run.smithi099.stdout:{ 2023-05-31T21:23:30.480 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_status": { 2023-05-31T21:23:30.480 INFO:teuthology.orchestra.run.smithi099.stdout: "info": { 2023-05-31T21:23:30.480 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "sync", 2023-05-31T21:23:30.480 INFO:teuthology.orchestra.run.smithi099.stdout: "num_shards": 4, 2023-05-31T21:23:30.481 INFO:teuthology.orchestra.run.smithi099.stdout: "period": "2e6123d2-d695-4350-9a01-61a455bc782b", 2023-05-31T21:23:30.481 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2023-05-31T21:23:30.481 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:23:30.481 INFO:teuthology.orchestra.run.smithi099.stdout: "markers": [ 2023-05-31T21:23:30.481 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T21:23:30.481 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 0, 2023-05-31T21:23:30.482 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T21:23:30.482 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2023-05-31T21:23:30.482 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1685566501.757183_260.1", 2023-05-31T21:23:30.482 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T21:23:30.482 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 1, 2023-05-31T21:23:30.482 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T21:23:30.482 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2023-05-31T20:55:01.757183Z", 2023-05-31T21:23:30.483 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2023-05-31T21:23:30.483 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T21:23:30.483 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:23:30.483 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T21:23:30.483 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 1, 2023-05-31T21:23:30.483 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T21:23:30.483 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2023-05-31T21:23:30.484 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1685566501.735717_251.1", 2023-05-31T21:23:30.484 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T21:23:30.484 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2023-05-31T21:23:30.484 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T21:23:30.484 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2023-05-31T20:55:01.735717Z", 2023-05-31T21:23:30.484 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2023-05-31T21:23:30.484 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T21:23:30.485 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:23:30.485 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T21:23:30.485 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 2, 2023-05-31T21:23:30.485 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T21:23:30.485 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2023-05-31T21:23:30.485 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1685562970.274572_93.1", 2023-05-31T21:23:30.485 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T21:23:30.486 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2023-05-31T21:23:30.486 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T21:23:30.486 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2023-05-31T19:56:10.274572Z", 2023-05-31T21:23:30.486 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2023-05-31T21:23:30.486 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T21:23:30.486 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:23:30.486 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T21:23:30.487 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 3, 2023-05-31T21:23:30.487 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T21:23:30.487 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2023-05-31T21:23:30.487 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1685568089.268814_960.1", 2023-05-31T21:23:30.487 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T21:23:30.488 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2023-05-31T21:23:30.488 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T21:23:30.488 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2023-05-31T21:21:29.268814Z", 2023-05-31T21:23:30.488 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2023-05-31T21:23:30.488 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T21:23:30.488 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T21:23:30.489 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2023-05-31T21:23:30.489 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:23:30.489 INFO:teuthology.orchestra.run.smithi099.stdout: "full_sync": { 2023-05-31T21:23:30.489 INFO:teuthology.orchestra.run.smithi099.stdout: "total": 1, 2023-05-31T21:23:30.489 INFO:teuthology.orchestra.run.smithi099.stdout: "complete": 1 2023-05-31T21:23:30.489 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:23:30.489 INFO:teuthology.orchestra.run.smithi099.stdout: "current_time": "2023-05-31T21:23:30Z" 2023-05-31T21:23:30.490 INFO:teuthology.orchestra.run.smithi099.stdout:} 2023-05-31T21:23:30.490 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "2e6123d2-d695-4350-9a01-61a455bc782b", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1685566501.757183_260.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-05-31T20:55:01.757183Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1685566501.735717_251.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-05-31T20:55:01.735717Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1685562970.274572_93.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-05-31T19:56:10.274572Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1685568089.268814_960.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-05-31T21:21:29.268814Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-05-31T21:23:30Z" } 2023-05-31T21:23:30.491 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1685566501.757183_260.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-05-31T20:55:01.757183Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1685566501.735717_251.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-05-31T20:55:01.735717Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1685562970.274572_93.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-05-31T19:56:10.274572Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1685568089.268814_960.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-05-31T21:21:29.268814Z', 'realm_epoch': 2}}] 2023-05-31T21:23:30.491 DEBUG:rgw_multi.tests:log_status={0: '1_1685566501.757183_260.1', 1: '1_1685566501.735717_251.1', 2: '1_1685562970.274572_93.1', 3: '1_1685568189.652614_974.1'} 2023-05-31T21:23:30.492 DEBUG:rgw_multi.tests:sync_status={0: '1_1685566501.757183_260.1', 1: '1_1685566501.735717_251.1', 2: '1_1685562970.274572_93.1', 3: '1_1685568089.268814_960.1'} 2023-05-31T21:23:30.492 WARNING:rgw_multi.tests:zone a2 behind master: shard=3 master=1_1685568189.652614_974.1 target=1_1685568089.268814_960.1 2023-05-31T21:23:35.493 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-31T21:23:35.494 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-31T21:23:35.494 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-05-31T21:23:35.552 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2023-05-31T21:23:35.552 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2023-05-31T21:23:35.603 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T21:23:35.600+0000 7f99168b6780 20 RGW-SYNC:meta: read sync status 2023-05-31T21:23:35.604 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T21:23:35.601+0000 7f99168b6780 20 RGW-SYNC:meta: read sync status 2023-05-31T21:23:35.622 INFO:teuthology.orchestra.run.smithi099.stdout:{ 2023-05-31T21:23:35.623 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_status": { 2023-05-31T21:23:35.623 INFO:teuthology.orchestra.run.smithi099.stdout: "info": { 2023-05-31T21:23:35.623 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "sync", 2023-05-31T21:23:35.623 INFO:teuthology.orchestra.run.smithi099.stdout: "num_shards": 4, 2023-05-31T21:23:35.623 INFO:teuthology.orchestra.run.smithi099.stdout: "period": "2e6123d2-d695-4350-9a01-61a455bc782b", 2023-05-31T21:23:35.623 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2023-05-31T21:23:35.624 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:23:35.624 INFO:teuthology.orchestra.run.smithi099.stdout: "markers": [ 2023-05-31T21:23:35.624 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T21:23:35.624 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 0, 2023-05-31T21:23:35.624 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T21:23:35.624 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2023-05-31T21:23:35.625 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1685566501.757183_260.1", 2023-05-31T21:23:35.625 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T21:23:35.625 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 1, 2023-05-31T21:23:35.625 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T21:23:35.625 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2023-05-31T20:55:01.757183Z", 2023-05-31T21:23:35.625 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2023-05-31T21:23:35.625 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T21:23:35.626 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:23:35.626 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T21:23:35.626 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 1, 2023-05-31T21:23:35.626 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T21:23:35.626 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2023-05-31T21:23:35.626 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1685566501.735717_251.1", 2023-05-31T21:23:35.626 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T21:23:35.627 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2023-05-31T21:23:35.627 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T21:23:35.627 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2023-05-31T20:55:01.735717Z", 2023-05-31T21:23:35.627 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2023-05-31T21:23:35.627 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T21:23:35.627 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:23:35.627 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T21:23:35.628 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 2, 2023-05-31T21:23:35.628 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T21:23:35.628 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2023-05-31T21:23:35.628 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1685562970.274572_93.1", 2023-05-31T21:23:35.628 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T21:23:35.628 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2023-05-31T21:23:35.628 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T21:23:35.629 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2023-05-31T19:56:10.274572Z", 2023-05-31T21:23:35.629 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2023-05-31T21:23:35.629 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T21:23:35.629 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:23:35.629 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T21:23:35.629 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 3, 2023-05-31T21:23:35.629 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T21:23:35.630 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2023-05-31T21:23:35.630 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1685568089.268814_960.1", 2023-05-31T21:23:35.630 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T21:23:35.630 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2023-05-31T21:23:35.630 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T21:23:35.630 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2023-05-31T21:21:29.268814Z", 2023-05-31T21:23:35.630 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2023-05-31T21:23:35.631 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T21:23:35.631 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T21:23:35.631 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2023-05-31T21:23:35.631 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:23:35.631 INFO:teuthology.orchestra.run.smithi099.stdout: "full_sync": { 2023-05-31T21:23:35.631 INFO:teuthology.orchestra.run.smithi099.stdout: "total": 1, 2023-05-31T21:23:35.631 INFO:teuthology.orchestra.run.smithi099.stdout: "complete": 1 2023-05-31T21:23:35.632 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:23:35.632 INFO:teuthology.orchestra.run.smithi099.stdout: "current_time": "2023-05-31T21:23:35Z" 2023-05-31T21:23:35.632 INFO:teuthology.orchestra.run.smithi099.stdout:} 2023-05-31T21:23:35.633 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "2e6123d2-d695-4350-9a01-61a455bc782b", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1685566501.757183_260.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-05-31T20:55:01.757183Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1685566501.735717_251.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-05-31T20:55:01.735717Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1685562970.274572_93.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-05-31T19:56:10.274572Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1685568089.268814_960.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-05-31T21:21:29.268814Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-05-31T21:23:35Z" } 2023-05-31T21:23:35.633 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1685566501.757183_260.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-05-31T20:55:01.757183Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1685566501.735717_251.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-05-31T20:55:01.735717Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1685562970.274572_93.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-05-31T19:56:10.274572Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1685568089.268814_960.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-05-31T21:21:29.268814Z', 'realm_epoch': 2}}] 2023-05-31T21:23:35.634 DEBUG:rgw_multi.tests:log_status={0: '1_1685566501.757183_260.1', 1: '1_1685566501.735717_251.1', 2: '1_1685562970.274572_93.1', 3: '1_1685568189.652614_974.1'} 2023-05-31T21:23:35.634 DEBUG:rgw_multi.tests:sync_status={0: '1_1685566501.757183_260.1', 1: '1_1685566501.735717_251.1', 2: '1_1685562970.274572_93.1', 3: '1_1685568089.268814_960.1'} 2023-05-31T21:23:35.635 WARNING:rgw_multi.tests:zone a2 behind master: shard=3 master=1_1685568189.652614_974.1 target=1_1685568089.268814_960.1 2023-05-31T21:23:40.636 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-31T21:23:40.636 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-31T21:23:40.637 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-05-31T21:23:40.694 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2023-05-31T21:23:40.694 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2023-05-31T21:23:40.730 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T21:23:40.725+0000 7f2d9507c780 20 RGW-SYNC:meta: read sync status 2023-05-31T21:23:40.731 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T21:23:40.727+0000 7f2d9507c780 20 RGW-SYNC:meta: read sync status 2023-05-31T21:23:40.751 INFO:teuthology.orchestra.run.smithi099.stdout:{ 2023-05-31T21:23:40.751 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_status": { 2023-05-31T21:23:40.752 INFO:teuthology.orchestra.run.smithi099.stdout: "info": { 2023-05-31T21:23:40.752 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "sync", 2023-05-31T21:23:40.752 INFO:teuthology.orchestra.run.smithi099.stdout: "num_shards": 4, 2023-05-31T21:23:40.752 INFO:teuthology.orchestra.run.smithi099.stdout: "period": "2e6123d2-d695-4350-9a01-61a455bc782b", 2023-05-31T21:23:40.753 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2023-05-31T21:23:40.753 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:23:40.754 INFO:teuthology.orchestra.run.smithi099.stdout: "markers": [ 2023-05-31T21:23:40.754 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T21:23:40.754 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 0, 2023-05-31T21:23:40.754 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T21:23:40.755 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2023-05-31T21:23:40.755 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1685566501.757183_260.1", 2023-05-31T21:23:40.755 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T21:23:40.755 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 1, 2023-05-31T21:23:40.757 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T21:23:40.757 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2023-05-31T20:55:01.757183Z", 2023-05-31T21:23:40.757 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2023-05-31T21:23:40.758 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T21:23:40.758 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:23:40.758 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T21:23:40.758 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 1, 2023-05-31T21:23:40.758 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T21:23:40.758 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2023-05-31T21:23:40.759 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1685566501.735717_251.1", 2023-05-31T21:23:40.759 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T21:23:40.759 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2023-05-31T21:23:40.759 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T21:23:40.759 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2023-05-31T20:55:01.735717Z", 2023-05-31T21:23:40.759 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2023-05-31T21:23:40.759 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T21:23:40.760 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:23:40.760 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T21:23:40.760 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 2, 2023-05-31T21:23:40.760 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T21:23:40.760 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2023-05-31T21:23:40.760 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1685562970.274572_93.1", 2023-05-31T21:23:40.761 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T21:23:40.761 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2023-05-31T21:23:40.761 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T21:23:40.761 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2023-05-31T19:56:10.274572Z", 2023-05-31T21:23:40.761 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2023-05-31T21:23:40.761 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T21:23:40.761 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:23:40.762 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T21:23:40.762 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 3, 2023-05-31T21:23:40.762 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T21:23:40.762 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2023-05-31T21:23:40.762 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1685568189.652614_974.1", 2023-05-31T21:23:40.762 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T21:23:40.762 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2023-05-31T21:23:40.763 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T21:23:40.763 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2023-05-31T21:23:09.652614Z", 2023-05-31T21:23:40.763 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2023-05-31T21:23:40.763 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T21:23:40.763 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T21:23:40.763 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2023-05-31T21:23:40.763 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:23:40.764 INFO:teuthology.orchestra.run.smithi099.stdout: "full_sync": { 2023-05-31T21:23:40.764 INFO:teuthology.orchestra.run.smithi099.stdout: "total": 1, 2023-05-31T21:23:40.764 INFO:teuthology.orchestra.run.smithi099.stdout: "complete": 1 2023-05-31T21:23:40.764 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:23:40.764 INFO:teuthology.orchestra.run.smithi099.stdout: "current_time": "2023-05-31T21:23:40Z" 2023-05-31T21:23:40.764 INFO:teuthology.orchestra.run.smithi099.stdout:} 2023-05-31T21:23:40.768 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "2e6123d2-d695-4350-9a01-61a455bc782b", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1685566501.757183_260.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-05-31T20:55:01.757183Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1685566501.735717_251.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-05-31T20:55:01.735717Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1685562970.274572_93.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-05-31T19:56:10.274572Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1685568189.652614_974.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-05-31T21:23:09.652614Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-05-31T21:23:40Z" } 2023-05-31T21:23:40.769 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1685566501.757183_260.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-05-31T20:55:01.757183Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1685566501.735717_251.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-05-31T20:55:01.735717Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1685562970.274572_93.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-05-31T19:56:10.274572Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1685568189.652614_974.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-05-31T21:23:09.652614Z', 'realm_epoch': 2}}] 2023-05-31T21:23:40.769 DEBUG:rgw_multi.tests:log_status={0: '1_1685566501.757183_260.1', 1: '1_1685566501.735717_251.1', 2: '1_1685562970.274572_93.1', 3: '1_1685568189.652614_974.1'} 2023-05-31T21:23:40.769 DEBUG:rgw_multi.tests:sync_status={0: '1_1685566501.757183_260.1', 1: '1_1685566501.735717_251.1', 2: '1_1685562970.274572_93.1', 3: '1_1685568189.652614_974.1'} 2023-05-31T21:23:40.770 INFO:rgw_multi.tests:finish meta checkpoint for zone=a2 2023-05-31T21:23:40.770 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'group', 'create', '--group-id', 'sync-group', '--status', 'allowed', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-05-31T21:23:40.771 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'group', 'create', '--group-id', 'sync-group', '--status', 'allowed', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-05-31T21:23:40.771 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync group create --group-id sync-group --status allowed --cluster c1 --debug-rgw 0 --debug-ms 0 2023-05-31T21:23:40.824 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2023-05-31T21:23:40.824 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2023-05-31T21:23:40.906 INFO:teuthology.orchestra.run.smithi099.stdout:{"groups":[{"id":"sync-group","data_flow":{"symmetrical":[{"id":"sync-flow1","zones":["a1","a2"]}]},"pipes":[{"id":"sync-pipe","source":{"bucket":"pklxkd-68","zones":["a1","a2"]},"dest":{"bucket":"pklxkd-68","zones":["a1","a2"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":""}}],"status":"allowed"}]} 2023-05-31T21:23:40.906 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'group', 'flow', 'create', '--group-id', 'sync-group', '--flow-id', 'sync-flow1', '--flow-type', 'symmetrical', '--zones=a1,a2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-05-31T21:23:40.907 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'group', 'flow', 'create', '--group-id', 'sync-group', '--flow-id', 'sync-flow1', '--flow-type', 'symmetrical', '--zones=a1,a2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-05-31T21:23:40.907 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync group flow create --group-id sync-group --flow-id sync-flow1 --flow-type symmetrical --zones=a1,a2 --cluster c1 --debug-rgw 0 --debug-ms 0 2023-05-31T21:23:40.961 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2023-05-31T21:23:40.962 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2023-05-31T21:23:41.029 INFO:teuthology.orchestra.run.smithi099.stdout:{"groups":[{"id":"sync-group","data_flow":{"symmetrical":[{"id":"sync-flow1","zones":["a1","a2"]}]},"pipes":[{"id":"sync-pipe","source":{"bucket":"pklxkd-68","zones":["a1","a2"]},"dest":{"bucket":"pklxkd-68","zones":["a1","a2"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":""}}],"status":"allowed"}]} 2023-05-31T21:23:41.030 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'group', 'pipe', 'create', '--group-id', 'sync-group', '--pipe-id', 'sync-pipe', '--source-zones=a1', '--dest-zones=a2', '--source-bucket=pklxkd-70', '--dest-bucket=pklxkd-71', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-05-31T21:23:41.030 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'group', 'pipe', 'create', '--group-id', 'sync-group', '--pipe-id', 'sync-pipe', '--source-zones=a1', '--dest-zones=a2', '--source-bucket=pklxkd-70', '--dest-bucket=pklxkd-71', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-05-31T21:23:41.031 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync group pipe create --group-id sync-group --pipe-id sync-pipe --source-zones=a1 --dest-zones=a2 --source-bucket=pklxkd-70 --dest-bucket=pklxkd-71 --cluster c1 --debug-rgw 0 --debug-ms 0 2023-05-31T21:23:41.084 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2023-05-31T21:23:41.085 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2023-05-31T21:23:41.135 INFO:teuthology.orchestra.run.smithi099.stdout:{"groups":[{"id":"sync-group","data_flow":{"symmetrical":[{"id":"sync-flow1","zones":["a1","a2"]}]},"pipes":[{"id":"sync-pipe","source":{"bucket":"pklxkd-70","zones":["a1","a2"]},"dest":{"bucket":"pklxkd-71","zones":["a1","a2"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":""}}],"status":"allowed"}]} 2023-05-31T21:23:41.136 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'group', 'modify', '--group-id', 'sync-group', '--status', 'enabled', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-05-31T21:23:41.136 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'group', 'modify', '--group-id', 'sync-group', '--status', 'enabled', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-05-31T21:23:41.137 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync group modify --group-id sync-group --status enabled --cluster c1 --debug-rgw 0 --debug-ms 0 2023-05-31T21:23:41.185 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2023-05-31T21:23:41.186 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2023-05-31T21:23:41.245 INFO:teuthology.orchestra.run.smithi099.stdout:{"groups":[{"id":"sync-group","data_flow":{"symmetrical":[{"id":"sync-flow1","zones":["a1","a2"]}]},"pipes":[{"id":"sync-pipe","source":{"bucket":"pklxkd-70","zones":["a1","a2"]},"dest":{"bucket":"pklxkd-71","zones":["a1","a2"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":""}}],"status":"enabled"}]} 2023-05-31T21:23:41.245 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['period', 'update', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-05-31T21:23:41.246 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'period', 'update', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-05-31T21:23:41.246 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 period update --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --commit --cluster c1 --debug-rgw 0 --debug-ms 0 2023-05-31T21:23:41.294 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2023-05-31T21:23:41.295 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2023-05-31T21:23:41.372 INFO:teuthology.orchestra.run.smithi099.stdout:{ 2023-05-31T21:23:41.373 INFO:teuthology.orchestra.run.smithi099.stdout: "id": "2e6123d2-d695-4350-9a01-61a455bc782b", 2023-05-31T21:23:41.373 INFO:teuthology.orchestra.run.smithi099.stdout: "epoch": 17, 2023-05-31T21:23:41.373 INFO:teuthology.orchestra.run.smithi099.stdout: "predecessor_uuid": "629b9ced-80af-41ed-8a6c-f2cfc334cbb4", 2023-05-31T21:23:41.373 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_status": [], 2023-05-31T21:23:41.373 INFO:teuthology.orchestra.run.smithi099.stdout: "period_map": { 2023-05-31T21:23:41.374 INFO:teuthology.orchestra.run.smithi099.stdout: "id": "2e6123d2-d695-4350-9a01-61a455bc782b", 2023-05-31T21:23:41.374 INFO:teuthology.orchestra.run.smithi099.stdout: "zonegroups": [ 2023-05-31T21:23:41.374 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T21:23:41.374 INFO:teuthology.orchestra.run.smithi099.stdout: "id": "a", 2023-05-31T21:23:41.374 INFO:teuthology.orchestra.run.smithi099.stdout: "name": "a", 2023-05-31T21:23:41.374 INFO:teuthology.orchestra.run.smithi099.stdout: "api_name": "", 2023-05-31T21:23:41.374 INFO:teuthology.orchestra.run.smithi099.stdout: "is_master": true, 2023-05-31T21:23:41.375 INFO:teuthology.orchestra.run.smithi099.stdout: "endpoints": [ 2023-05-31T21:23:41.375 INFO:teuthology.orchestra.run.smithi099.stdout: "http://smithi099.front.sepia.ceph.com:8000" 2023-05-31T21:23:41.375 INFO:teuthology.orchestra.run.smithi099.stdout: ], 2023-05-31T21:23:41.375 INFO:teuthology.orchestra.run.smithi099.stdout: "hostnames": [], 2023-05-31T21:23:41.375 INFO:teuthology.orchestra.run.smithi099.stdout: "hostnames_s3website": [], 2023-05-31T21:23:41.375 INFO:teuthology.orchestra.run.smithi099.stdout: "master_zone": "3d163b1e-09c7-47e2-ae87-add4aace3f38", 2023-05-31T21:23:41.375 INFO:teuthology.orchestra.run.smithi099.stdout: "zones": [ 2023-05-31T21:23:41.376 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T21:23:41.376 INFO:teuthology.orchestra.run.smithi099.stdout: "id": "3d163b1e-09c7-47e2-ae87-add4aace3f38", 2023-05-31T21:23:41.376 INFO:teuthology.orchestra.run.smithi099.stdout: "name": "a1", 2023-05-31T21:23:41.376 INFO:teuthology.orchestra.run.smithi099.stdout: "endpoints": [ 2023-05-31T21:23:41.376 INFO:teuthology.orchestra.run.smithi099.stdout: "http://smithi099.front.sepia.ceph.com:8000" 2023-05-31T21:23:41.376 INFO:teuthology.orchestra.run.smithi099.stdout: ], 2023-05-31T21:23:41.377 INFO:teuthology.orchestra.run.smithi099.stdout: "log_meta": false, 2023-05-31T21:23:41.377 INFO:teuthology.orchestra.run.smithi099.stdout: "log_data": true, 2023-05-31T21:23:41.377 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket_index_max_shards": 11, 2023-05-31T21:23:41.377 INFO:teuthology.orchestra.run.smithi099.stdout: "read_only": false, 2023-05-31T21:23:41.377 INFO:teuthology.orchestra.run.smithi099.stdout: "tier_type": "", 2023-05-31T21:23:41.377 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_from_all": true, 2023-05-31T21:23:41.377 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_from": [], 2023-05-31T21:23:41.378 INFO:teuthology.orchestra.run.smithi099.stdout: "redirect_zone": "", 2023-05-31T21:23:41.378 INFO:teuthology.orchestra.run.smithi099.stdout: "supported_features": [ 2023-05-31T21:23:41.378 INFO:teuthology.orchestra.run.smithi099.stdout: "resharding" 2023-05-31T21:23:41.378 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2023-05-31T21:23:41.378 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:23:41.378 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T21:23:41.378 INFO:teuthology.orchestra.run.smithi099.stdout: "id": "5d8acec7-7e57-4e56-8b32-e3d6b5036001", 2023-05-31T21:23:41.379 INFO:teuthology.orchestra.run.smithi099.stdout: "name": "a2", 2023-05-31T21:23:41.379 INFO:teuthology.orchestra.run.smithi099.stdout: "endpoints": [ 2023-05-31T21:23:41.379 INFO:teuthology.orchestra.run.smithi099.stdout: "http://smithi099.front.sepia.ceph.com:8001" 2023-05-31T21:23:41.379 INFO:teuthology.orchestra.run.smithi099.stdout: ], 2023-05-31T21:23:41.379 INFO:teuthology.orchestra.run.smithi099.stdout: "log_meta": false, 2023-05-31T21:23:41.379 INFO:teuthology.orchestra.run.smithi099.stdout: "log_data": true, 2023-05-31T21:23:41.379 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket_index_max_shards": 11, 2023-05-31T21:23:41.380 INFO:teuthology.orchestra.run.smithi099.stdout: "read_only": false, 2023-05-31T21:23:41.380 INFO:teuthology.orchestra.run.smithi099.stdout: "tier_type": "", 2023-05-31T21:23:41.380 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_from_all": true, 2023-05-31T21:23:41.380 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_from": [], 2023-05-31T21:23:41.380 INFO:teuthology.orchestra.run.smithi099.stdout: "redirect_zone": "", 2023-05-31T21:23:41.380 INFO:teuthology.orchestra.run.smithi099.stdout: "supported_features": [ 2023-05-31T21:23:41.381 INFO:teuthology.orchestra.run.smithi099.stdout: "resharding" 2023-05-31T21:23:41.381 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2023-05-31T21:23:41.381 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T21:23:41.381 INFO:teuthology.orchestra.run.smithi099.stdout: ], 2023-05-31T21:23:41.381 INFO:teuthology.orchestra.run.smithi099.stdout: "placement_targets": [ 2023-05-31T21:23:41.381 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T21:23:41.381 INFO:teuthology.orchestra.run.smithi099.stdout: "name": "default-placement", 2023-05-31T21:23:41.382 INFO:teuthology.orchestra.run.smithi099.stdout: "tags": [], 2023-05-31T21:23:41.382 INFO:teuthology.orchestra.run.smithi099.stdout: "storage_classes": [ 2023-05-31T21:23:41.382 INFO:teuthology.orchestra.run.smithi099.stdout: "STANDARD" 2023-05-31T21:23:41.382 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2023-05-31T21:23:41.382 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T21:23:41.382 INFO:teuthology.orchestra.run.smithi099.stdout: ], 2023-05-31T21:23:41.382 INFO:teuthology.orchestra.run.smithi099.stdout: "default_placement": "default-placement", 2023-05-31T21:23:41.383 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_id": "aaa33de6-515b-4cac-8f1b-3a1a30452117", 2023-05-31T21:23:41.383 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_policy": { 2023-05-31T21:23:41.383 INFO:teuthology.orchestra.run.smithi099.stdout: "groups": [ 2023-05-31T21:23:41.383 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T21:23:41.383 INFO:teuthology.orchestra.run.smithi099.stdout: "id": "sync-group", 2023-05-31T21:23:41.383 INFO:teuthology.orchestra.run.smithi099.stdout: "data_flow": { 2023-05-31T21:23:41.383 INFO:teuthology.orchestra.run.smithi099.stdout: "symmetrical": [ 2023-05-31T21:23:41.384 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T21:23:41.384 INFO:teuthology.orchestra.run.smithi099.stdout: "id": "sync-flow1", 2023-05-31T21:23:41.384 INFO:teuthology.orchestra.run.smithi099.stdout: "zones": [ 2023-05-31T21:23:41.384 INFO:teuthology.orchestra.run.smithi099.stdout: "3d163b1e-09c7-47e2-ae87-add4aace3f38", 2023-05-31T21:23:41.384 INFO:teuthology.orchestra.run.smithi099.stdout: "5d8acec7-7e57-4e56-8b32-e3d6b5036001" 2023-05-31T21:23:41.384 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2023-05-31T21:23:41.384 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T21:23:41.385 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2023-05-31T21:23:41.385 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:23:41.385 INFO:teuthology.orchestra.run.smithi099.stdout: "pipes": [ 2023-05-31T21:23:41.385 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T21:23:41.385 INFO:teuthology.orchestra.run.smithi099.stdout: "id": "sync-pipe", 2023-05-31T21:23:41.385 INFO:teuthology.orchestra.run.smithi099.stdout: "source": { 2023-05-31T21:23:41.385 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket": "pklxkd-70", 2023-05-31T21:23:41.386 INFO:teuthology.orchestra.run.smithi099.stdout: "zones": [ 2023-05-31T21:23:41.386 INFO:teuthology.orchestra.run.smithi099.stdout: "3d163b1e-09c7-47e2-ae87-add4aace3f38", 2023-05-31T21:23:41.386 INFO:teuthology.orchestra.run.smithi099.stdout: "5d8acec7-7e57-4e56-8b32-e3d6b5036001" 2023-05-31T21:23:41.386 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2023-05-31T21:23:41.386 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:23:41.386 INFO:teuthology.orchestra.run.smithi099.stdout: "dest": { 2023-05-31T21:23:41.386 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket": "pklxkd-71", 2023-05-31T21:23:41.387 INFO:teuthology.orchestra.run.smithi099.stdout: "zones": [ 2023-05-31T21:23:41.387 INFO:teuthology.orchestra.run.smithi099.stdout: "3d163b1e-09c7-47e2-ae87-add4aace3f38", 2023-05-31T21:23:41.387 INFO:teuthology.orchestra.run.smithi099.stdout: "5d8acec7-7e57-4e56-8b32-e3d6b5036001" 2023-05-31T21:23:41.387 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2023-05-31T21:23:41.387 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:23:41.387 INFO:teuthology.orchestra.run.smithi099.stdout: "params": { 2023-05-31T21:23:41.387 INFO:teuthology.orchestra.run.smithi099.stdout: "source": { 2023-05-31T21:23:41.388 INFO:teuthology.orchestra.run.smithi099.stdout: "filter": { 2023-05-31T21:23:41.388 INFO:teuthology.orchestra.run.smithi099.stdout: "tags": [] 2023-05-31T21:23:41.388 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T21:23:41.388 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:23:41.388 INFO:teuthology.orchestra.run.smithi099.stdout: "dest": {}, 2023-05-31T21:23:41.388 INFO:teuthology.orchestra.run.smithi099.stdout: "priority": 0, 2023-05-31T21:23:41.388 INFO:teuthology.orchestra.run.smithi099.stdout: "mode": "system", 2023-05-31T21:23:41.389 INFO:teuthology.orchestra.run.smithi099.stdout: "user": "" 2023-05-31T21:23:41.389 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T21:23:41.389 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T21:23:41.389 INFO:teuthology.orchestra.run.smithi099.stdout: ], 2023-05-31T21:23:41.389 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "enabled" 2023-05-31T21:23:41.389 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T21:23:41.389 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2023-05-31T21:23:41.390 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:23:41.390 INFO:teuthology.orchestra.run.smithi099.stdout: "enabled_features": [ 2023-05-31T21:23:41.390 INFO:teuthology.orchestra.run.smithi099.stdout: "resharding" 2023-05-31T21:23:41.390 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2023-05-31T21:23:41.390 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:23:41.390 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T21:23:41.390 INFO:teuthology.orchestra.run.smithi099.stdout: "id": "b", 2023-05-31T21:23:41.391 INFO:teuthology.orchestra.run.smithi099.stdout: "name": "b", 2023-05-31T21:23:41.391 INFO:teuthology.orchestra.run.smithi099.stdout: "api_name": "", 2023-05-31T21:23:41.391 INFO:teuthology.orchestra.run.smithi099.stdout: "is_master": false, 2023-05-31T21:23:41.391 INFO:teuthology.orchestra.run.smithi099.stdout: "endpoints": [ 2023-05-31T21:23:41.391 INFO:teuthology.orchestra.run.smithi099.stdout: "http://smithi112.front.sepia.ceph.com:8000" 2023-05-31T21:23:41.391 INFO:teuthology.orchestra.run.smithi099.stdout: ], 2023-05-31T21:23:41.391 INFO:teuthology.orchestra.run.smithi099.stdout: "hostnames": [], 2023-05-31T21:23:41.392 INFO:teuthology.orchestra.run.smithi099.stdout: "hostnames_s3website": [], 2023-05-31T21:23:41.392 INFO:teuthology.orchestra.run.smithi099.stdout: "master_zone": "f85d7236-2f27-4988-af26-9c66ab280b1b", 2023-05-31T21:23:41.392 INFO:teuthology.orchestra.run.smithi099.stdout: "zones": [ 2023-05-31T21:23:41.392 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T21:23:41.392 INFO:teuthology.orchestra.run.smithi099.stdout: "id": "a6a3aab5-bc52-44a4-8543-5c5ea6056acd", 2023-05-31T21:23:41.392 INFO:teuthology.orchestra.run.smithi099.stdout: "name": "b2", 2023-05-31T21:23:41.393 INFO:teuthology.orchestra.run.smithi099.stdout: "endpoints": [ 2023-05-31T21:23:41.393 INFO:teuthology.orchestra.run.smithi099.stdout: "http://smithi112.front.sepia.ceph.com:8001" 2023-05-31T21:23:41.393 INFO:teuthology.orchestra.run.smithi099.stdout: ], 2023-05-31T21:23:41.396 INFO:teuthology.orchestra.run.smithi099.stdout: "log_meta": false, 2023-05-31T21:23:41.396 INFO:teuthology.orchestra.run.smithi099.stdout: "log_data": true, 2023-05-31T21:23:41.396 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket_index_max_shards": 11, 2023-05-31T21:23:41.396 INFO:teuthology.orchestra.run.smithi099.stdout: "read_only": false, 2023-05-31T21:23:41.396 INFO:teuthology.orchestra.run.smithi099.stdout: "tier_type": "", 2023-05-31T21:23:41.396 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_from_all": true, 2023-05-31T21:23:41.396 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_from": [], 2023-05-31T21:23:41.397 INFO:teuthology.orchestra.run.smithi099.stdout: "redirect_zone": "", 2023-05-31T21:23:41.397 INFO:teuthology.orchestra.run.smithi099.stdout: "supported_features": [ 2023-05-31T21:23:41.397 INFO:teuthology.orchestra.run.smithi099.stdout: "resharding" 2023-05-31T21:23:41.397 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2023-05-31T21:23:41.397 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:23:41.397 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T21:23:41.397 INFO:teuthology.orchestra.run.smithi099.stdout: "id": "f85d7236-2f27-4988-af26-9c66ab280b1b", 2023-05-31T21:23:41.398 INFO:teuthology.orchestra.run.smithi099.stdout: "name": "b1", 2023-05-31T21:23:41.398 INFO:teuthology.orchestra.run.smithi099.stdout: "endpoints": [ 2023-05-31T21:23:41.398 INFO:teuthology.orchestra.run.smithi099.stdout: "http://smithi112.front.sepia.ceph.com:8000" 2023-05-31T21:23:41.398 INFO:teuthology.orchestra.run.smithi099.stdout: ], 2023-05-31T21:23:41.398 INFO:teuthology.orchestra.run.smithi099.stdout: "log_meta": false, 2023-05-31T21:23:41.398 INFO:teuthology.orchestra.run.smithi099.stdout: "log_data": true, 2023-05-31T21:23:41.398 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket_index_max_shards": 11, 2023-05-31T21:23:41.399 INFO:teuthology.orchestra.run.smithi099.stdout: "read_only": false, 2023-05-31T21:23:41.399 INFO:teuthology.orchestra.run.smithi099.stdout: "tier_type": "", 2023-05-31T21:23:41.399 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_from_all": true, 2023-05-31T21:23:41.399 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_from": [], 2023-05-31T21:23:41.399 INFO:teuthology.orchestra.run.smithi099.stdout: "redirect_zone": "", 2023-05-31T21:23:41.399 INFO:teuthology.orchestra.run.smithi099.stdout: "supported_features": [ 2023-05-31T21:23:41.399 INFO:teuthology.orchestra.run.smithi099.stdout: "resharding" 2023-05-31T21:23:41.400 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2023-05-31T21:23:41.400 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T21:23:41.400 INFO:teuthology.orchestra.run.smithi099.stdout: ], 2023-05-31T21:23:41.400 INFO:teuthology.orchestra.run.smithi099.stdout: "placement_targets": [ 2023-05-31T21:23:41.400 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T21:23:41.400 INFO:teuthology.orchestra.run.smithi099.stdout: "name": "default-placement", 2023-05-31T21:23:41.400 INFO:teuthology.orchestra.run.smithi099.stdout: "tags": [], 2023-05-31T21:23:41.401 INFO:teuthology.orchestra.run.smithi099.stdout: "storage_classes": [ 2023-05-31T21:23:41.401 INFO:teuthology.orchestra.run.smithi099.stdout: "STANDARD" 2023-05-31T21:23:41.401 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2023-05-31T21:23:41.401 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T21:23:41.401 INFO:teuthology.orchestra.run.smithi099.stdout: ], 2023-05-31T21:23:41.401 INFO:teuthology.orchestra.run.smithi099.stdout: "default_placement": "default-placement", 2023-05-31T21:23:41.401 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_id": "aaa33de6-515b-4cac-8f1b-3a1a30452117", 2023-05-31T21:23:41.402 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_policy": { 2023-05-31T21:23:41.402 INFO:teuthology.orchestra.run.smithi099.stdout: "groups": [] 2023-05-31T21:23:41.402 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:23:41.402 INFO:teuthology.orchestra.run.smithi099.stdout: "enabled_features": [ 2023-05-31T21:23:41.402 INFO:teuthology.orchestra.run.smithi099.stdout: "resharding" 2023-05-31T21:23:41.402 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2023-05-31T21:23:41.402 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T21:23:41.403 INFO:teuthology.orchestra.run.smithi099.stdout: ], 2023-05-31T21:23:41.403 INFO:teuthology.orchestra.run.smithi099.stdout: "short_zone_ids": [ 2023-05-31T21:23:41.403 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T21:23:41.403 INFO:teuthology.orchestra.run.smithi099.stdout: "key": "3d163b1e-09c7-47e2-ae87-add4aace3f38", 2023-05-31T21:23:41.403 INFO:teuthology.orchestra.run.smithi099.stdout: "val": 3101833771 2023-05-31T21:23:41.403 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:23:41.403 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T21:23:41.404 INFO:teuthology.orchestra.run.smithi099.stdout: "key": "5d8acec7-7e57-4e56-8b32-e3d6b5036001", 2023-05-31T21:23:41.404 INFO:teuthology.orchestra.run.smithi099.stdout: "val": 4255530623 2023-05-31T21:23:41.404 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:23:41.404 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T21:23:41.404 INFO:teuthology.orchestra.run.smithi099.stdout: "key": "a6a3aab5-bc52-44a4-8543-5c5ea6056acd", 2023-05-31T21:23:41.404 INFO:teuthology.orchestra.run.smithi099.stdout: "val": 3708073710 2023-05-31T21:23:41.404 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:23:41.405 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T21:23:41.406 INFO:teuthology.orchestra.run.smithi099.stdout: "key": "f85d7236-2f27-4988-af26-9c66ab280b1b", 2023-05-31T21:23:41.406 INFO:teuthology.orchestra.run.smithi099.stdout: "val": 233395200 2023-05-31T21:23:41.406 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T21:23:41.406 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2023-05-31T21:23:41.406 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:23:41.406 INFO:teuthology.orchestra.run.smithi099.stdout: "master_zonegroup": "a", 2023-05-31T21:23:41.407 INFO:teuthology.orchestra.run.smithi099.stdout: "master_zone": "3d163b1e-09c7-47e2-ae87-add4aace3f38", 2023-05-31T21:23:41.407 INFO:teuthology.orchestra.run.smithi099.stdout: "period_config": { 2023-05-31T21:23:41.407 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket_quota": { 2023-05-31T21:23:41.407 INFO:teuthology.orchestra.run.smithi099.stdout: "enabled": false, 2023-05-31T21:23:41.407 INFO:teuthology.orchestra.run.smithi099.stdout: "check_on_raw": false, 2023-05-31T21:23:41.407 INFO:teuthology.orchestra.run.smithi099.stdout: "max_size": -1, 2023-05-31T21:23:41.408 INFO:teuthology.orchestra.run.smithi099.stdout: "max_size_kb": 0, 2023-05-31T21:23:41.408 INFO:teuthology.orchestra.run.smithi099.stdout: "max_objects": -1 2023-05-31T21:23:41.408 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:23:41.408 INFO:teuthology.orchestra.run.smithi099.stdout: "user_quota": { 2023-05-31T21:23:41.408 INFO:teuthology.orchestra.run.smithi099.stdout: "enabled": false, 2023-05-31T21:23:41.408 INFO:teuthology.orchestra.run.smithi099.stdout: "check_on_raw": false, 2023-05-31T21:23:41.408 INFO:teuthology.orchestra.run.smithi099.stdout: "max_size": -1, 2023-05-31T21:23:41.409 INFO:teuthology.orchestra.run.smithi099.stdout: "max_size_kb": 0, 2023-05-31T21:23:41.409 INFO:teuthology.orchestra.run.smithi099.stdout: "max_objects": -1 2023-05-31T21:23:41.409 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:23:41.409 INFO:teuthology.orchestra.run.smithi099.stdout: "user_ratelimit": { 2023-05-31T21:23:41.409 INFO:teuthology.orchestra.run.smithi099.stdout: "max_read_ops": 0, 2023-05-31T21:23:41.409 INFO:teuthology.orchestra.run.smithi099.stdout: "max_write_ops": 0, 2023-05-31T21:23:41.409 INFO:teuthology.orchestra.run.smithi099.stdout: "max_read_bytes": 0, 2023-05-31T21:23:41.410 INFO:teuthology.orchestra.run.smithi099.stdout: "max_write_bytes": 0, 2023-05-31T21:23:41.410 INFO:teuthology.orchestra.run.smithi099.stdout: "enabled": false 2023-05-31T21:23:41.410 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:23:41.410 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket_ratelimit": { 2023-05-31T21:23:41.410 INFO:teuthology.orchestra.run.smithi099.stdout: "max_read_ops": 0, 2023-05-31T21:23:41.410 INFO:teuthology.orchestra.run.smithi099.stdout: "max_write_ops": 0, 2023-05-31T21:23:41.410 INFO:teuthology.orchestra.run.smithi099.stdout: "max_read_bytes": 0, 2023-05-31T21:23:41.411 INFO:teuthology.orchestra.run.smithi099.stdout: "max_write_bytes": 0, 2023-05-31T21:23:41.411 INFO:teuthology.orchestra.run.smithi099.stdout: "enabled": false 2023-05-31T21:23:41.411 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:23:41.411 INFO:teuthology.orchestra.run.smithi099.stdout: "anonymous_ratelimit": { 2023-05-31T21:23:41.411 INFO:teuthology.orchestra.run.smithi099.stdout: "max_read_ops": 0, 2023-05-31T21:23:41.411 INFO:teuthology.orchestra.run.smithi099.stdout: "max_write_ops": 0, 2023-05-31T21:23:41.411 INFO:teuthology.orchestra.run.smithi099.stdout: "max_read_bytes": 0, 2023-05-31T21:23:41.412 INFO:teuthology.orchestra.run.smithi099.stdout: "max_write_bytes": 0, 2023-05-31T21:23:41.412 INFO:teuthology.orchestra.run.smithi099.stdout: "enabled": false 2023-05-31T21:23:41.412 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T21:23:41.412 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:23:41.412 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_id": "aaa33de6-515b-4cac-8f1b-3a1a30452117", 2023-05-31T21:23:41.412 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_name": "test-realm", 2023-05-31T21:23:41.412 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2023-05-31T21:23:41.413 INFO:teuthology.orchestra.run.smithi099.stdout:} 2023-05-31T21:23:41.413 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'policy', 'get', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-05-31T21:23:41.413 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'policy', 'get', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-05-31T21:23:41.414 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync policy get --cluster c1 --debug-rgw 0 --debug-ms 0 2023-05-31T21:23:41.460 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2023-05-31T21:23:41.460 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2023-05-31T21:23:41.520 INFO:teuthology.orchestra.run.smithi099.stdout:{"groups":[{"id":"sync-group","data_flow":{"symmetrical":[{"id":"sync-flow1","zones":["a1","a2"]}]},"pipes":[{"id":"sync-pipe","source":{"bucket":"pklxkd-70","zones":["a1","a2"]},"dest":{"bucket":"pklxkd-71","zones":["a1","a2"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":""}}],"status":"enabled"}]} 2023-05-31T21:23:41.521 DEBUG:boto:path=/pklxkd-70/ 2023-05-31T21:23:41.521 DEBUG:boto:auth_path=/pklxkd-70/ 2023-05-31T21:23:41.521 DEBUG:boto:Method: HEAD 2023-05-31T21:23:41.522 DEBUG:boto:Path: /pklxkd-70/ 2023-05-31T21:23:41.522 DEBUG:boto:Data: 2023-05-31T21:23:41.522 DEBUG:boto:Headers: {} 2023-05-31T21:23:41.523 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T21:23:41.523 DEBUG:boto:Port: 8000 2023-05-31T21:23:41.523 DEBUG:boto:Params: {} 2023-05-31T21:23:41.524 DEBUG:boto:Token: None 2023-05-31T21:23:41.524 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 21:23:41 GMT /pklxkd-70/ 2023-05-31T21:23:41.525 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:o257PhwKGFd4B7OIGrZBdybKcr0= 2023-05-31T21:23:41.526 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 21:23:41 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:o257PhwKGFd4B7OIGrZBdybKcr0=', 'Content-Length': '0'} 2023-05-31T21:23:41.836 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000388fe8f6fdd079b6-006477badd-256e-a1'), ('Content-Length', '0'), ('Date', 'Wed, 31 May 2023 21:23:41 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:23:41.837 DEBUG:boto:path=/pklxkd-70/obj1 2023-05-31T21:23:41.837 DEBUG:boto:auth_path=/pklxkd-70/obj1 2023-05-31T21:23:41.838 DEBUG:boto:Method: PUT 2023-05-31T21:23:41.838 DEBUG:boto:Path: /pklxkd-70/obj1 2023-05-31T21:23:41.838 DEBUG:boto:Data: 2023-05-31T21:23:41.839 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2023-05-31T21:23:41.839 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T21:23:41.839 DEBUG:boto:Port: 8000 2023-05-31T21:23:41.840 DEBUG:boto:Params: {} 2023-05-31T21:23:41.840 DEBUG:boto:Token: None 2023-05-31T21:23:41.841 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Wed, 31 May 2023 21:23:41 GMT /pklxkd-70/obj1 2023-05-31T21:23:41.842 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:WoM8mAbyyVgomjJiYByyntI6rBM= 2023-05-31T21:23:41.842 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Wed, 31 May 2023 21:23:41 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:WoM8mAbyyVgomjJiYByyntI6rBM='} 2023-05-31T21:23:41.862 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000007ebefd9fdefffa80-006477badd-256e-a1'), ('Date', 'Wed, 31 May 2023 21:23:41 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:23:41.863 DEBUG:boto:path=/pklxkd-70/ 2023-05-31T21:23:41.864 DEBUG:boto:auth_path=/pklxkd-70/ 2023-05-31T21:23:41.864 DEBUG:boto:Method: HEAD 2023-05-31T21:23:41.864 DEBUG:boto:Path: /pklxkd-70/ 2023-05-31T21:23:41.865 DEBUG:boto:Data: 2023-05-31T21:23:41.866 DEBUG:boto:Headers: {} 2023-05-31T21:23:41.866 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T21:23:41.867 DEBUG:boto:Port: 8000 2023-05-31T21:23:41.868 DEBUG:boto:Params: {} 2023-05-31T21:23:41.869 DEBUG:boto:Token: None 2023-05-31T21:23:41.869 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 21:23:41 GMT /pklxkd-70/ 2023-05-31T21:23:41.870 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:o257PhwKGFd4B7OIGrZBdybKcr0= 2023-05-31T21:23:41.870 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 21:23:41 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:o257PhwKGFd4B7OIGrZBdybKcr0=', 'Content-Length': '0'} 2023-05-31T21:23:41.874 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '6'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000066e9a188cd333cc2-006477badd-256e-a1'), ('Content-Length', '0'), ('Date', 'Wed, 31 May 2023 21:23:41 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:23:41.874 DEBUG:boto:path=/pklxkd-70/obj2 2023-05-31T21:23:41.875 DEBUG:boto:auth_path=/pklxkd-70/obj2 2023-05-31T21:23:41.875 DEBUG:boto:Method: PUT 2023-05-31T21:23:41.875 DEBUG:boto:Path: /pklxkd-70/obj2 2023-05-31T21:23:41.876 DEBUG:boto:Data: 2023-05-31T21:23:41.876 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2023-05-31T21:23:41.877 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T21:23:41.877 DEBUG:boto:Port: 8000 2023-05-31T21:23:41.877 DEBUG:boto:Params: {} 2023-05-31T21:23:41.878 DEBUG:boto:Token: None 2023-05-31T21:23:41.878 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Wed, 31 May 2023 21:23:41 GMT /pklxkd-70/obj2 2023-05-31T21:23:41.879 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:1iKRjbRxk6XWpUmorFhiFssS1ik= 2023-05-31T21:23:41.879 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Wed, 31 May 2023 21:23:41 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:1iKRjbRxk6XWpUmorFhiFssS1ik='} 2023-05-31T21:23:41.924 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000003f7935abdf814c1f-006477badd-256e-a1'), ('Date', 'Wed, 31 May 2023 21:23:41 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:23:41.925 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-31T21:23:41.925 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-31T21:23:41.926 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-05-31T21:23:41.991 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2023-05-31T21:23:41.991 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2023-05-31T21:23:42.076 INFO:teuthology.orchestra.run.smithi099.stderr:No --period given, using current period=2e6123d2-d695-4350-9a01-61a455bc782b 2023-05-31T21:23:42.127 INFO:teuthology.orchestra.run.smithi099.stdout:[ 2023-05-31T21:23:42.127 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T21:23:42.127 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1685566501.757183_260.1", 2023-05-31T21:23:42.127 INFO:teuthology.orchestra.run.smithi099.stdout: "last_update": "2023-05-31T20:55:01.757183Z" 2023-05-31T21:23:42.128 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:23:42.128 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T21:23:42.128 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1685566501.735717_251.1", 2023-05-31T21:23:42.128 INFO:teuthology.orchestra.run.smithi099.stdout: "last_update": "2023-05-31T20:55:01.735717Z" 2023-05-31T21:23:42.128 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:23:42.128 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T21:23:42.129 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1685562970.274572_93.1", 2023-05-31T21:23:42.129 INFO:teuthology.orchestra.run.smithi099.stdout: "last_update": "2023-05-31T19:56:10.274572Z" 2023-05-31T21:23:42.129 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:23:42.129 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T21:23:42.129 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1685568189.652614_974.1", 2023-05-31T21:23:42.129 INFO:teuthology.orchestra.run.smithi099.stdout: "last_update": "2023-05-31T21:23:09.652614Z" 2023-05-31T21:23:42.130 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T21:23:42.130 INFO:teuthology.orchestra.run.smithi099.stdout:] 2023-05-31T21:23:42.131 DEBUG:rgw_multi.tests:master meta markers={0: '1_1685566501.757183_260.1', 1: '1_1685566501.735717_251.1', 2: '1_1685562970.274572_93.1', 3: '1_1685568189.652614_974.1'} 2023-05-31T21:23:42.131 INFO:rgw_multi.tests:starting meta checkpoint for zone=a2 2023-05-31T21:23:42.132 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-31T21:23:42.132 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-31T21:23:42.132 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-05-31T21:23:42.199 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2023-05-31T21:23:42.199 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2023-05-31T21:23:42.270 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T21:23:42.265+0000 7f177ed2f780 20 RGW-SYNC:meta: read sync status 2023-05-31T21:23:42.270 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T21:23:42.266+0000 7f177ed2f780 20 RGW-SYNC:meta: read sync status 2023-05-31T21:23:42.293 INFO:teuthology.orchestra.run.smithi099.stdout:{ 2023-05-31T21:23:42.293 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_status": { 2023-05-31T21:23:42.293 INFO:teuthology.orchestra.run.smithi099.stdout: "info": { 2023-05-31T21:23:42.293 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "sync", 2023-05-31T21:23:42.294 INFO:teuthology.orchestra.run.smithi099.stdout: "num_shards": 4, 2023-05-31T21:23:42.294 INFO:teuthology.orchestra.run.smithi099.stdout: "period": "2e6123d2-d695-4350-9a01-61a455bc782b", 2023-05-31T21:23:42.294 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2023-05-31T21:23:42.294 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:23:42.294 INFO:teuthology.orchestra.run.smithi099.stdout: "markers": [ 2023-05-31T21:23:42.294 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T21:23:42.294 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 0, 2023-05-31T21:23:42.295 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T21:23:42.295 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2023-05-31T21:23:42.295 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1685566501.757183_260.1", 2023-05-31T21:23:42.295 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T21:23:42.295 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 1, 2023-05-31T21:23:42.295 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T21:23:42.295 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2023-05-31T20:55:01.757183Z", 2023-05-31T21:23:42.296 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2023-05-31T21:23:42.296 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T21:23:42.296 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:23:42.296 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T21:23:42.296 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 1, 2023-05-31T21:23:42.296 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T21:23:42.297 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2023-05-31T21:23:42.297 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1685566501.735717_251.1", 2023-05-31T21:23:42.297 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T21:23:42.297 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2023-05-31T21:23:42.297 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T21:23:42.297 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2023-05-31T20:55:01.735717Z", 2023-05-31T21:23:42.297 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2023-05-31T21:23:42.298 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T21:23:42.298 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:23:42.298 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T21:23:42.298 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 2, 2023-05-31T21:23:42.298 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T21:23:42.298 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2023-05-31T21:23:42.298 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1685562970.274572_93.1", 2023-05-31T21:23:42.299 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T21:23:42.299 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2023-05-31T21:23:42.299 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T21:23:42.299 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2023-05-31T19:56:10.274572Z", 2023-05-31T21:23:42.299 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2023-05-31T21:23:42.299 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T21:23:42.299 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:23:42.300 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T21:23:42.300 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 3, 2023-05-31T21:23:42.300 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T21:23:42.300 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2023-05-31T21:23:42.301 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1685568189.652614_974.1", 2023-05-31T21:23:42.302 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T21:23:42.302 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2023-05-31T21:23:42.302 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T21:23:42.303 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2023-05-31T21:23:09.652614Z", 2023-05-31T21:23:42.303 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2023-05-31T21:23:42.303 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T21:23:42.304 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T21:23:42.304 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2023-05-31T21:23:42.304 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:23:42.305 INFO:teuthology.orchestra.run.smithi099.stdout: "full_sync": { 2023-05-31T21:23:42.305 INFO:teuthology.orchestra.run.smithi099.stdout: "total": 1, 2023-05-31T21:23:42.305 INFO:teuthology.orchestra.run.smithi099.stdout: "complete": 1 2023-05-31T21:23:42.306 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:23:42.306 INFO:teuthology.orchestra.run.smithi099.stdout: "current_time": "2023-05-31T21:23:42Z" 2023-05-31T21:23:42.306 INFO:teuthology.orchestra.run.smithi099.stdout:} 2023-05-31T21:23:42.307 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "2e6123d2-d695-4350-9a01-61a455bc782b", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1685566501.757183_260.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-05-31T20:55:01.757183Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1685566501.735717_251.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-05-31T20:55:01.735717Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1685562970.274572_93.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-05-31T19:56:10.274572Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1685568189.652614_974.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-05-31T21:23:09.652614Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-05-31T21:23:42Z" } 2023-05-31T21:23:42.308 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1685566501.757183_260.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-05-31T20:55:01.757183Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1685566501.735717_251.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-05-31T20:55:01.735717Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1685562970.274572_93.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-05-31T19:56:10.274572Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1685568189.652614_974.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-05-31T21:23:09.652614Z', 'realm_epoch': 2}}] 2023-05-31T21:23:42.309 DEBUG:rgw_multi.tests:log_status={0: '1_1685566501.757183_260.1', 1: '1_1685566501.735717_251.1', 2: '1_1685562970.274572_93.1', 3: '1_1685568189.652614_974.1'} 2023-05-31T21:23:42.310 DEBUG:rgw_multi.tests:sync_status={0: '1_1685566501.757183_260.1', 1: '1_1685566501.735717_251.1', 2: '1_1685562970.274572_93.1', 3: '1_1685568189.652614_974.1'} 2023-05-31T21:23:42.311 INFO:rgw_multi.tests:finish meta checkpoint for zone=a2 2023-05-31T21:23:42.312 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['datalog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-31T21:23:42.313 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'datalog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-31T21:23:42.314 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 datalog status --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-05-31T21:23:42.367 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2023-05-31T21:23:42.368 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2023-05-31T21:23:42.461 INFO:teuthology.orchestra.run.smithi099.stdout:[ 2023-05-31T21:23:42.461 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T21:23:42.462 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "00000000000000000000:00000000000000045095", 2023-05-31T21:23:42.462 INFO:teuthology.orchestra.run.smithi099.stdout: "last_update": "2023-05-31T21:23:41.891197Z" 2023-05-31T21:23:42.462 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:23:42.462 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T21:23:42.462 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "00000000000000000000:00000000000000031499", 2023-05-31T21:23:42.463 INFO:teuthology.orchestra.run.smithi099.stdout: "last_update": "2023-05-31T21:12:40.667485Z" 2023-05-31T21:23:42.463 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:23:42.463 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T21:23:42.463 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "00000000000000000000:00000000000000051251", 2023-05-31T21:23:42.463 INFO:teuthology.orchestra.run.smithi099.stdout: "last_update": "2023-05-31T21:23:41.857089Z" 2023-05-31T21:23:42.463 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:23:42.464 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T21:23:42.464 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "00000000000000000000:00000000000000044038", 2023-05-31T21:23:42.464 INFO:teuthology.orchestra.run.smithi099.stdout: "last_update": "2023-05-31T21:21:35.912562Z" 2023-05-31T21:23:42.464 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T21:23:42.464 INFO:teuthology.orchestra.run.smithi099.stdout:] 2023-05-31T21:23:42.465 DEBUG:rgw_multi.tests:data markers for zone=a1 markers={0: '00000000000000000000:00000000000000045095', 1: '00000000000000000000:00000000000000031499', 2: '00000000000000000000:00000000000000051251', 3: '00000000000000000000:00000000000000044038'} 2023-05-31T21:23:42.466 INFO:rgw_multi.tests:starting data checkpoint for target_zone=a2 source_zone=a1 2023-05-31T21:23:42.466 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-31T21:23:42.467 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-31T21:23:42.467 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-05-31T21:23:42.524 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2023-05-31T21:23:42.525 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2023-05-31T21:23:42.600 INFO:teuthology.orchestra.run.smithi099.stdout:{ 2023-05-31T21:23:42.601 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_status": { 2023-05-31T21:23:42.601 INFO:teuthology.orchestra.run.smithi099.stdout: "info": { 2023-05-31T21:23:42.601 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "sync", 2023-05-31T21:23:42.601 INFO:teuthology.orchestra.run.smithi099.stdout: "num_shards": 4, 2023-05-31T21:23:42.601 INFO:teuthology.orchestra.run.smithi099.stdout: "instance_id": 452136274422638824 2023-05-31T21:23:42.601 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:23:42.602 INFO:teuthology.orchestra.run.smithi099.stdout: "markers": [ 2023-05-31T21:23:42.602 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T21:23:42.602 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 0, 2023-05-31T21:23:42.602 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T21:23:42.602 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "incremental-sync", 2023-05-31T21:23:42.602 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "00000000000000000000:00000000000000044955", 2023-05-31T21:23:42.603 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T21:23:42.603 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2023-05-31T21:23:42.603 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T21:23:42.603 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2023-05-31T21:21:35.807414Z" 2023-05-31T21:23:42.603 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T21:23:42.603 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:23:42.603 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T21:23:42.604 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 1, 2023-05-31T21:23:42.604 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T21:23:42.604 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "incremental-sync", 2023-05-31T21:23:42.604 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "00000000000000000000:00000000000000031499", 2023-05-31T21:23:42.604 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T21:23:42.604 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2023-05-31T21:23:42.605 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T21:23:42.605 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2023-05-31T21:12:40.667485Z" 2023-05-31T21:23:42.605 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T21:23:42.605 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:23:42.605 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T21:23:42.605 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 2, 2023-05-31T21:23:42.605 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T21:23:42.606 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "incremental-sync", 2023-05-31T21:23:42.606 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "00000000000000000000:00000000000000051111", 2023-05-31T21:23:42.606 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T21:23:42.606 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 55, 2023-05-31T21:23:42.606 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T21:23:42.606 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2023-05-31T21:21:35.866368Z" 2023-05-31T21:23:42.607 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T21:23:42.607 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:23:42.607 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T21:23:42.607 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 3, 2023-05-31T21:23:42.607 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T21:23:42.607 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "incremental-sync", 2023-05-31T21:23:42.607 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "00000000000000000000:00000000000000044038", 2023-05-31T21:23:42.608 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T21:23:42.608 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 9, 2023-05-31T21:23:42.608 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T21:23:42.608 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2023-05-31T21:21:35.912562Z" 2023-05-31T21:23:42.608 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T21:23:42.608 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T21:23:42.608 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2023-05-31T21:23:42.609 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:23:42.609 INFO:teuthology.orchestra.run.smithi099.stdout: "full_sync": { 2023-05-31T21:23:42.609 INFO:teuthology.orchestra.run.smithi099.stdout: "total": 64, 2023-05-31T21:23:42.609 INFO:teuthology.orchestra.run.smithi099.stdout: "complete": 64 2023-05-31T21:23:42.609 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:23:42.609 INFO:teuthology.orchestra.run.smithi099.stdout: "current_time": "2023-05-31T21:23:42Z" 2023-05-31T21:23:42.610 INFO:teuthology.orchestra.run.smithi099.stdout:} 2023-05-31T21:23:42.611 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 452136274422638824 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000044955", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-05-31T21:21:35.807414Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000031499", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-05-31T21:12:40.667485Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000051111", "next_step_marker": "", "total_entries": 55, "pos": 0, "timestamp": "2023-05-31T21:21:35.866368Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000044038", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2023-05-31T21:21:35.912562Z" } } ] }, "full_sync": { "total": 64, "complete": 64 }, "current_time": "2023-05-31T21:23:42Z" } 2023-05-31T21:23:42.611 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000044955', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-05-31T21:21:35.807414Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000031499', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-05-31T21:12:40.667485Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000051111', 'next_step_marker': '', 'total_entries': 55, 'pos': 0, 'timestamp': '2023-05-31T21:21:35.866368Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000044038', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2023-05-31T21:21:35.912562Z'}}] 2023-05-31T21:23:42.611 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000045095', 1: '00000000000000000000:00000000000000031499', 2: '00000000000000000000:00000000000000051251', 3: '00000000000000000000:00000000000000044038'} 2023-05-31T21:23:42.612 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000044955', 1: '00000000000000000000:00000000000000031499', 2: '00000000000000000000:00000000000000051111', 3: '00000000000000000000:00000000000000044038'} 2023-05-31T21:23:42.612 WARNING:rgw_multi.tests:data of zone a2 behind zone a1: shard=0 master=00000000000000000000:00000000000000045095 target=00000000000000000000:00000000000000044955, shard=2 master=00000000000000000000:00000000000000051251 target=00000000000000000000:00000000000000051111 2023-05-31T21:23:47.614 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-31T21:23:47.614 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-31T21:23:47.615 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-05-31T21:23:47.675 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2023-05-31T21:23:47.675 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2023-05-31T21:23:47.730 INFO:teuthology.orchestra.run.smithi099.stdout:{ 2023-05-31T21:23:47.730 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_status": { 2023-05-31T21:23:47.730 INFO:teuthology.orchestra.run.smithi099.stdout: "info": { 2023-05-31T21:23:47.730 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "sync", 2023-05-31T21:23:47.730 INFO:teuthology.orchestra.run.smithi099.stdout: "num_shards": 4, 2023-05-31T21:23:47.731 INFO:teuthology.orchestra.run.smithi099.stdout: "instance_id": 452136274422638824 2023-05-31T21:23:47.731 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:23:47.731 INFO:teuthology.orchestra.run.smithi099.stdout: "markers": [ 2023-05-31T21:23:47.731 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T21:23:47.731 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 0, 2023-05-31T21:23:47.731 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T21:23:47.732 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "incremental-sync", 2023-05-31T21:23:47.732 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "00000000000000000000:00000000000000044955", 2023-05-31T21:23:47.732 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T21:23:47.732 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2023-05-31T21:23:47.732 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T21:23:47.732 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2023-05-31T21:21:35.807414Z" 2023-05-31T21:23:47.732 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T21:23:47.733 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:23:47.733 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T21:23:47.733 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 1, 2023-05-31T21:23:47.733 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T21:23:47.733 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "incremental-sync", 2023-05-31T21:23:47.733 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "00000000000000000000:00000000000000031499", 2023-05-31T21:23:47.733 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T21:23:47.734 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2023-05-31T21:23:47.734 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T21:23:47.734 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2023-05-31T21:12:40.667485Z" 2023-05-31T21:23:47.734 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T21:23:47.734 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:23:47.734 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T21:23:47.735 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 2, 2023-05-31T21:23:47.735 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T21:23:47.735 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "incremental-sync", 2023-05-31T21:23:47.735 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "00000000000000000000:00000000000000051111", 2023-05-31T21:23:47.735 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T21:23:47.735 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 55, 2023-05-31T21:23:47.735 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T21:23:47.736 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2023-05-31T21:21:35.866368Z" 2023-05-31T21:23:47.736 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T21:23:47.737 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:23:47.737 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T21:23:47.737 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 3, 2023-05-31T21:23:47.737 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T21:23:47.737 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "incremental-sync", 2023-05-31T21:23:47.737 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "00000000000000000000:00000000000000044038", 2023-05-31T21:23:47.738 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T21:23:47.738 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 9, 2023-05-31T21:23:47.738 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T21:23:47.738 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2023-05-31T21:21:35.912562Z" 2023-05-31T21:23:47.738 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T21:23:47.738 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T21:23:47.738 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2023-05-31T21:23:47.739 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:23:47.739 INFO:teuthology.orchestra.run.smithi099.stdout: "full_sync": { 2023-05-31T21:23:47.739 INFO:teuthology.orchestra.run.smithi099.stdout: "total": 64, 2023-05-31T21:23:47.739 INFO:teuthology.orchestra.run.smithi099.stdout: "complete": 64 2023-05-31T21:23:47.739 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:23:47.739 INFO:teuthology.orchestra.run.smithi099.stdout: "current_time": "2023-05-31T21:23:47Z" 2023-05-31T21:23:47.739 INFO:teuthology.orchestra.run.smithi099.stdout:} 2023-05-31T21:23:47.740 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 452136274422638824 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000044955", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-05-31T21:21:35.807414Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000031499", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-05-31T21:12:40.667485Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000051111", "next_step_marker": "", "total_entries": 55, "pos": 0, "timestamp": "2023-05-31T21:21:35.866368Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000044038", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2023-05-31T21:21:35.912562Z" } } ] }, "full_sync": { "total": 64, "complete": 64 }, "current_time": "2023-05-31T21:23:47Z" } 2023-05-31T21:23:47.741 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000044955', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-05-31T21:21:35.807414Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000031499', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-05-31T21:12:40.667485Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000051111', 'next_step_marker': '', 'total_entries': 55, 'pos': 0, 'timestamp': '2023-05-31T21:21:35.866368Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000044038', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2023-05-31T21:21:35.912562Z'}}] 2023-05-31T21:23:47.741 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000045095', 1: '00000000000000000000:00000000000000031499', 2: '00000000000000000000:00000000000000051251', 3: '00000000000000000000:00000000000000044038'} 2023-05-31T21:23:47.742 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000044955', 1: '00000000000000000000:00000000000000031499', 2: '00000000000000000000:00000000000000051111', 3: '00000000000000000000:00000000000000044038'} 2023-05-31T21:23:47.742 WARNING:rgw_multi.tests:data of zone a2 behind zone a1: shard=0 master=00000000000000000000:00000000000000045095 target=00000000000000000000:00000000000000044955, shard=2 master=00000000000000000000:00000000000000051251 target=00000000000000000000:00000000000000051111 2023-05-31T21:23:52.743 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-31T21:23:52.744 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-31T21:23:52.745 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-05-31T21:23:52.805 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2023-05-31T21:23:52.806 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2023-05-31T21:23:52.883 INFO:teuthology.orchestra.run.smithi099.stdout:{ 2023-05-31T21:23:52.884 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_status": { 2023-05-31T21:23:52.884 INFO:teuthology.orchestra.run.smithi099.stdout: "info": { 2023-05-31T21:23:52.884 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "sync", 2023-05-31T21:23:52.884 INFO:teuthology.orchestra.run.smithi099.stdout: "num_shards": 4, 2023-05-31T21:23:52.884 INFO:teuthology.orchestra.run.smithi099.stdout: "instance_id": 452136274422638824 2023-05-31T21:23:52.885 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:23:52.885 INFO:teuthology.orchestra.run.smithi099.stdout: "markers": [ 2023-05-31T21:23:52.885 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T21:23:52.885 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 0, 2023-05-31T21:23:52.885 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T21:23:52.885 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "incremental-sync", 2023-05-31T21:23:52.885 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "00000000000000000000:00000000000000044955", 2023-05-31T21:23:52.886 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T21:23:52.886 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2023-05-31T21:23:52.886 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T21:23:52.886 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2023-05-31T21:21:35.807414Z" 2023-05-31T21:23:52.886 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T21:23:52.886 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:23:52.887 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T21:23:52.887 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 1, 2023-05-31T21:23:52.887 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T21:23:52.887 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "incremental-sync", 2023-05-31T21:23:52.887 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "00000000000000000000:00000000000000031499", 2023-05-31T21:23:52.888 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T21:23:52.888 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2023-05-31T21:23:52.888 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T21:23:52.888 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2023-05-31T21:12:40.667485Z" 2023-05-31T21:23:52.888 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T21:23:52.888 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:23:52.888 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T21:23:52.889 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 2, 2023-05-31T21:23:52.889 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T21:23:52.889 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "incremental-sync", 2023-05-31T21:23:52.889 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "00000000000000000000:00000000000000051111", 2023-05-31T21:23:52.889 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T21:23:52.889 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 55, 2023-05-31T21:23:52.889 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T21:23:52.890 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2023-05-31T21:21:35.866368Z" 2023-05-31T21:23:52.890 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T21:23:52.890 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:23:52.890 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T21:23:52.890 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 3, 2023-05-31T21:23:52.890 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T21:23:52.890 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "incremental-sync", 2023-05-31T21:23:52.891 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "00000000000000000000:00000000000000044038", 2023-05-31T21:23:52.891 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T21:23:52.891 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 9, 2023-05-31T21:23:52.891 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T21:23:52.891 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2023-05-31T21:21:35.912562Z" 2023-05-31T21:23:52.891 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T21:23:52.892 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T21:23:52.892 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2023-05-31T21:23:52.892 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:23:52.892 INFO:teuthology.orchestra.run.smithi099.stdout: "full_sync": { 2023-05-31T21:23:52.892 INFO:teuthology.orchestra.run.smithi099.stdout: "total": 64, 2023-05-31T21:23:52.892 INFO:teuthology.orchestra.run.smithi099.stdout: "complete": 64 2023-05-31T21:23:52.893 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:23:52.893 INFO:teuthology.orchestra.run.smithi099.stdout: "current_time": "2023-05-31T21:23:52Z" 2023-05-31T21:23:52.893 INFO:teuthology.orchestra.run.smithi099.stdout:} 2023-05-31T21:23:52.894 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 452136274422638824 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000044955", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-05-31T21:21:35.807414Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000031499", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-05-31T21:12:40.667485Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000051111", "next_step_marker": "", "total_entries": 55, "pos": 0, "timestamp": "2023-05-31T21:21:35.866368Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000044038", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2023-05-31T21:21:35.912562Z" } } ] }, "full_sync": { "total": 64, "complete": 64 }, "current_time": "2023-05-31T21:23:52Z" } 2023-05-31T21:23:52.894 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000044955', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-05-31T21:21:35.807414Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000031499', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-05-31T21:12:40.667485Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000051111', 'next_step_marker': '', 'total_entries': 55, 'pos': 0, 'timestamp': '2023-05-31T21:21:35.866368Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000044038', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2023-05-31T21:21:35.912562Z'}}] 2023-05-31T21:23:52.895 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000045095', 1: '00000000000000000000:00000000000000031499', 2: '00000000000000000000:00000000000000051251', 3: '00000000000000000000:00000000000000044038'} 2023-05-31T21:23:52.895 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000044955', 1: '00000000000000000000:00000000000000031499', 2: '00000000000000000000:00000000000000051111', 3: '00000000000000000000:00000000000000044038'} 2023-05-31T21:23:52.895 WARNING:rgw_multi.tests:data of zone a2 behind zone a1: shard=0 master=00000000000000000000:00000000000000045095 target=00000000000000000000:00000000000000044955, shard=2 master=00000000000000000000:00000000000000051251 target=00000000000000000000:00000000000000051111 2023-05-31T21:23:57.896 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-31T21:23:57.897 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-31T21:23:57.898 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-05-31T21:23:57.956 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2023-05-31T21:23:57.957 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2023-05-31T21:23:58.008 INFO:teuthology.orchestra.run.smithi099.stdout:{ 2023-05-31T21:23:58.009 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_status": { 2023-05-31T21:23:58.009 INFO:teuthology.orchestra.run.smithi099.stdout: "info": { 2023-05-31T21:23:58.009 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "sync", 2023-05-31T21:23:58.009 INFO:teuthology.orchestra.run.smithi099.stdout: "num_shards": 4, 2023-05-31T21:23:58.009 INFO:teuthology.orchestra.run.smithi099.stdout: "instance_id": 452136274422638824 2023-05-31T21:23:58.010 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:23:58.010 INFO:teuthology.orchestra.run.smithi099.stdout: "markers": [ 2023-05-31T21:23:58.010 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T21:23:58.010 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 0, 2023-05-31T21:23:58.010 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T21:23:58.010 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "incremental-sync", 2023-05-31T21:23:58.011 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "00000000000000000000:00000000000000044955", 2023-05-31T21:23:58.011 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T21:23:58.011 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2023-05-31T21:23:58.011 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T21:23:58.011 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2023-05-31T21:21:35.807414Z" 2023-05-31T21:23:58.012 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T21:23:58.012 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:23:58.012 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T21:23:58.012 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 1, 2023-05-31T21:23:58.013 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T21:23:58.013 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "incremental-sync", 2023-05-31T21:23:58.013 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "00000000000000000000:00000000000000031499", 2023-05-31T21:23:58.013 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T21:23:58.013 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2023-05-31T21:23:58.013 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T21:23:58.013 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2023-05-31T21:12:40.667485Z" 2023-05-31T21:23:58.014 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T21:23:58.014 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:23:58.014 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T21:23:58.014 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 2, 2023-05-31T21:23:58.014 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T21:23:58.015 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "incremental-sync", 2023-05-31T21:23:58.015 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "00000000000000000000:00000000000000051111", 2023-05-31T21:23:58.015 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T21:23:58.015 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 55, 2023-05-31T21:23:58.015 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T21:23:58.015 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2023-05-31T21:21:35.866368Z" 2023-05-31T21:23:58.015 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T21:23:58.016 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:23:58.016 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T21:23:58.016 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 3, 2023-05-31T21:23:58.017 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T21:23:58.017 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "incremental-sync", 2023-05-31T21:23:58.017 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "00000000000000000000:00000000000000044038", 2023-05-31T21:23:58.017 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T21:23:58.018 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 9, 2023-05-31T21:23:58.018 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T21:23:58.018 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2023-05-31T21:21:35.912562Z" 2023-05-31T21:23:58.018 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T21:23:58.018 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T21:23:58.019 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2023-05-31T21:23:58.019 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:23:58.019 INFO:teuthology.orchestra.run.smithi099.stdout: "full_sync": { 2023-05-31T21:23:58.019 INFO:teuthology.orchestra.run.smithi099.stdout: "total": 64, 2023-05-31T21:23:58.019 INFO:teuthology.orchestra.run.smithi099.stdout: "complete": 64 2023-05-31T21:23:58.020 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:23:58.020 INFO:teuthology.orchestra.run.smithi099.stdout: "current_time": "2023-05-31T21:23:57Z" 2023-05-31T21:23:58.020 INFO:teuthology.orchestra.run.smithi099.stdout:} 2023-05-31T21:23:58.021 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 452136274422638824 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000044955", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-05-31T21:21:35.807414Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000031499", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-05-31T21:12:40.667485Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000051111", "next_step_marker": "", "total_entries": 55, "pos": 0, "timestamp": "2023-05-31T21:21:35.866368Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000044038", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2023-05-31T21:21:35.912562Z" } } ] }, "full_sync": { "total": 64, "complete": 64 }, "current_time": "2023-05-31T21:23:57Z" } 2023-05-31T21:23:58.021 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000044955', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-05-31T21:21:35.807414Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000031499', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-05-31T21:12:40.667485Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000051111', 'next_step_marker': '', 'total_entries': 55, 'pos': 0, 'timestamp': '2023-05-31T21:21:35.866368Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000044038', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2023-05-31T21:21:35.912562Z'}}] 2023-05-31T21:23:58.022 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000045095', 1: '00000000000000000000:00000000000000031499', 2: '00000000000000000000:00000000000000051251', 3: '00000000000000000000:00000000000000044038'} 2023-05-31T21:23:58.022 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000044955', 1: '00000000000000000000:00000000000000031499', 2: '00000000000000000000:00000000000000051111', 3: '00000000000000000000:00000000000000044038'} 2023-05-31T21:23:58.023 WARNING:rgw_multi.tests:data of zone a2 behind zone a1: shard=0 master=00000000000000000000:00000000000000045095 target=00000000000000000000:00000000000000044955, shard=2 master=00000000000000000000:00000000000000051251 target=00000000000000000000:00000000000000051111 2023-05-31T21:24:03.024 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-31T21:24:03.024 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-31T21:24:03.025 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-05-31T21:24:03.083 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2023-05-31T21:24:03.083 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2023-05-31T21:24:03.151 INFO:teuthology.orchestra.run.smithi099.stdout:{ 2023-05-31T21:24:03.151 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_status": { 2023-05-31T21:24:03.152 INFO:teuthology.orchestra.run.smithi099.stdout: "info": { 2023-05-31T21:24:03.152 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "sync", 2023-05-31T21:24:03.152 INFO:teuthology.orchestra.run.smithi099.stdout: "num_shards": 4, 2023-05-31T21:24:03.152 INFO:teuthology.orchestra.run.smithi099.stdout: "instance_id": 452136274422638824 2023-05-31T21:24:03.152 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:24:03.153 INFO:teuthology.orchestra.run.smithi099.stdout: "markers": [ 2023-05-31T21:24:03.153 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T21:24:03.153 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 0, 2023-05-31T21:24:03.153 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T21:24:03.153 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "incremental-sync", 2023-05-31T21:24:03.154 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "00000000000000000000:00000000000000044955", 2023-05-31T21:24:03.154 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T21:24:03.154 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2023-05-31T21:24:03.154 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T21:24:03.154 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2023-05-31T21:21:35.807414Z" 2023-05-31T21:24:03.155 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T21:24:03.155 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:24:03.155 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T21:24:03.155 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 1, 2023-05-31T21:24:03.155 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T21:24:03.155 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "incremental-sync", 2023-05-31T21:24:03.155 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "00000000000000000000:00000000000000031499", 2023-05-31T21:24:03.156 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T21:24:03.156 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2023-05-31T21:24:03.156 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T21:24:03.157 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2023-05-31T21:12:40.667485Z" 2023-05-31T21:24:03.157 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T21:24:03.157 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:24:03.157 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T21:24:03.157 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 2, 2023-05-31T21:24:03.157 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T21:24:03.158 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "incremental-sync", 2023-05-31T21:24:03.158 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "00000000000000000000:00000000000000051111", 2023-05-31T21:24:03.158 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T21:24:03.158 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 55, 2023-05-31T21:24:03.158 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T21:24:03.158 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2023-05-31T21:21:35.866368Z" 2023-05-31T21:24:03.158 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T21:24:03.159 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:24:03.159 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T21:24:03.159 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 3, 2023-05-31T21:24:03.159 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T21:24:03.159 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "incremental-sync", 2023-05-31T21:24:03.159 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "00000000000000000000:00000000000000044038", 2023-05-31T21:24:03.159 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T21:24:03.160 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 9, 2023-05-31T21:24:03.160 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T21:24:03.160 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2023-05-31T21:21:35.912562Z" 2023-05-31T21:24:03.160 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T21:24:03.160 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T21:24:03.160 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2023-05-31T21:24:03.161 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:24:03.161 INFO:teuthology.orchestra.run.smithi099.stdout: "full_sync": { 2023-05-31T21:24:03.161 INFO:teuthology.orchestra.run.smithi099.stdout: "total": 64, 2023-05-31T21:24:03.161 INFO:teuthology.orchestra.run.smithi099.stdout: "complete": 64 2023-05-31T21:24:03.162 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:24:03.162 INFO:teuthology.orchestra.run.smithi099.stdout: "current_time": "2023-05-31T21:24:03Z" 2023-05-31T21:24:03.162 INFO:teuthology.orchestra.run.smithi099.stdout:} 2023-05-31T21:24:03.164 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 452136274422638824 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000044955", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-05-31T21:21:35.807414Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000031499", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-05-31T21:12:40.667485Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000051111", "next_step_marker": "", "total_entries": 55, "pos": 0, "timestamp": "2023-05-31T21:21:35.866368Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000044038", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2023-05-31T21:21:35.912562Z" } } ] }, "full_sync": { "total": 64, "complete": 64 }, "current_time": "2023-05-31T21:24:03Z" } 2023-05-31T21:24:03.165 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000044955', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-05-31T21:21:35.807414Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000031499', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-05-31T21:12:40.667485Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000051111', 'next_step_marker': '', 'total_entries': 55, 'pos': 0, 'timestamp': '2023-05-31T21:21:35.866368Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000044038', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2023-05-31T21:21:35.912562Z'}}] 2023-05-31T21:24:03.166 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000045095', 1: '00000000000000000000:00000000000000031499', 2: '00000000000000000000:00000000000000051251', 3: '00000000000000000000:00000000000000044038'} 2023-05-31T21:24:03.167 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000044955', 1: '00000000000000000000:00000000000000031499', 2: '00000000000000000000:00000000000000051111', 3: '00000000000000000000:00000000000000044038'} 2023-05-31T21:24:03.168 WARNING:rgw_multi.tests:data of zone a2 behind zone a1: shard=0 master=00000000000000000000:00000000000000045095 target=00000000000000000000:00000000000000044955, shard=2 master=00000000000000000000:00000000000000051251 target=00000000000000000000:00000000000000051111 2023-05-31T21:24:08.169 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-31T21:24:08.170 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-31T21:24:08.171 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-05-31T21:24:08.227 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2023-05-31T21:24:08.227 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2023-05-31T21:24:08.282 INFO:teuthology.orchestra.run.smithi099.stdout:{ 2023-05-31T21:24:08.282 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_status": { 2023-05-31T21:24:08.283 INFO:teuthology.orchestra.run.smithi099.stdout: "info": { 2023-05-31T21:24:08.283 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "sync", 2023-05-31T21:24:08.283 INFO:teuthology.orchestra.run.smithi099.stdout: "num_shards": 4, 2023-05-31T21:24:08.283 INFO:teuthology.orchestra.run.smithi099.stdout: "instance_id": 452136274422638824 2023-05-31T21:24:08.283 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:24:08.284 INFO:teuthology.orchestra.run.smithi099.stdout: "markers": [ 2023-05-31T21:24:08.284 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T21:24:08.284 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 0, 2023-05-31T21:24:08.284 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T21:24:08.284 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "incremental-sync", 2023-05-31T21:24:08.285 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "00000000000000000000:00000000000000044955", 2023-05-31T21:24:08.285 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T21:24:08.285 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2023-05-31T21:24:08.285 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T21:24:08.286 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2023-05-31T21:21:35.807414Z" 2023-05-31T21:24:08.286 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T21:24:08.286 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:24:08.286 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T21:24:08.286 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 1, 2023-05-31T21:24:08.286 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T21:24:08.286 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "incremental-sync", 2023-05-31T21:24:08.287 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "00000000000000000000:00000000000000031499", 2023-05-31T21:24:08.287 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T21:24:08.287 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2023-05-31T21:24:08.287 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T21:24:08.287 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2023-05-31T21:12:40.667485Z" 2023-05-31T21:24:08.288 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T21:24:08.288 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:24:08.288 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T21:24:08.288 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 2, 2023-05-31T21:24:08.289 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T21:24:08.289 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "incremental-sync", 2023-05-31T21:24:08.289 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "00000000000000000000:00000000000000051111", 2023-05-31T21:24:08.290 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T21:24:08.290 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 55, 2023-05-31T21:24:08.290 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T21:24:08.291 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2023-05-31T21:21:35.866368Z" 2023-05-31T21:24:08.291 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T21:24:08.291 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:24:08.291 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T21:24:08.291 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 3, 2023-05-31T21:24:08.292 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T21:24:08.292 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "incremental-sync", 2023-05-31T21:24:08.292 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "00000000000000000000:00000000000000044038", 2023-05-31T21:24:08.292 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T21:24:08.292 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 9, 2023-05-31T21:24:08.292 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T21:24:08.293 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2023-05-31T21:21:35.912562Z" 2023-05-31T21:24:08.293 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T21:24:08.293 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T21:24:08.293 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2023-05-31T21:24:08.293 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:24:08.293 INFO:teuthology.orchestra.run.smithi099.stdout: "full_sync": { 2023-05-31T21:24:08.293 INFO:teuthology.orchestra.run.smithi099.stdout: "total": 64, 2023-05-31T21:24:08.294 INFO:teuthology.orchestra.run.smithi099.stdout: "complete": 64 2023-05-31T21:24:08.294 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:24:08.294 INFO:teuthology.orchestra.run.smithi099.stdout: "current_time": "2023-05-31T21:24:08Z" 2023-05-31T21:24:08.294 INFO:teuthology.orchestra.run.smithi099.stdout:} 2023-05-31T21:24:08.295 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 452136274422638824 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000044955", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-05-31T21:21:35.807414Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000031499", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-05-31T21:12:40.667485Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000051111", "next_step_marker": "", "total_entries": 55, "pos": 0, "timestamp": "2023-05-31T21:21:35.866368Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000044038", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2023-05-31T21:21:35.912562Z" } } ] }, "full_sync": { "total": 64, "complete": 64 }, "current_time": "2023-05-31T21:24:08Z" } 2023-05-31T21:24:08.295 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000044955', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-05-31T21:21:35.807414Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000031499', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-05-31T21:12:40.667485Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000051111', 'next_step_marker': '', 'total_entries': 55, 'pos': 0, 'timestamp': '2023-05-31T21:21:35.866368Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000044038', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2023-05-31T21:21:35.912562Z'}}] 2023-05-31T21:24:08.296 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000045095', 1: '00000000000000000000:00000000000000031499', 2: '00000000000000000000:00000000000000051251', 3: '00000000000000000000:00000000000000044038'} 2023-05-31T21:24:08.296 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000044955', 1: '00000000000000000000:00000000000000031499', 2: '00000000000000000000:00000000000000051111', 3: '00000000000000000000:00000000000000044038'} 2023-05-31T21:24:08.297 WARNING:rgw_multi.tests:data of zone a2 behind zone a1: shard=0 master=00000000000000000000:00000000000000045095 target=00000000000000000000:00000000000000044955, shard=2 master=00000000000000000000:00000000000000051251 target=00000000000000000000:00000000000000051111 2023-05-31T21:24:13.298 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-31T21:24:13.299 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-31T21:24:13.299 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-05-31T21:24:13.360 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2023-05-31T21:24:13.361 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2023-05-31T21:24:13.429 INFO:teuthology.orchestra.run.smithi099.stdout:{ 2023-05-31T21:24:13.430 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_status": { 2023-05-31T21:24:13.430 INFO:teuthology.orchestra.run.smithi099.stdout: "info": { 2023-05-31T21:24:13.430 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "sync", 2023-05-31T21:24:13.430 INFO:teuthology.orchestra.run.smithi099.stdout: "num_shards": 4, 2023-05-31T21:24:13.430 INFO:teuthology.orchestra.run.smithi099.stdout: "instance_id": 452136274422638824 2023-05-31T21:24:13.431 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:24:13.431 INFO:teuthology.orchestra.run.smithi099.stdout: "markers": [ 2023-05-31T21:24:13.431 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T21:24:13.431 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 0, 2023-05-31T21:24:13.431 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T21:24:13.431 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "incremental-sync", 2023-05-31T21:24:13.431 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "00000000000000000000:00000000000000044955", 2023-05-31T21:24:13.432 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T21:24:13.432 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2023-05-31T21:24:13.432 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T21:24:13.432 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2023-05-31T21:21:35.807414Z" 2023-05-31T21:24:13.433 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T21:24:13.433 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:24:13.433 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T21:24:13.433 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 1, 2023-05-31T21:24:13.434 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T21:24:13.434 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "incremental-sync", 2023-05-31T21:24:13.435 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "00000000000000000000:00000000000000031499", 2023-05-31T21:24:13.436 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T21:24:13.436 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2023-05-31T21:24:13.436 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T21:24:13.436 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2023-05-31T21:12:40.667485Z" 2023-05-31T21:24:13.436 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T21:24:13.436 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:24:13.437 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T21:24:13.437 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 2, 2023-05-31T21:24:13.437 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T21:24:13.437 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "incremental-sync", 2023-05-31T21:24:13.437 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "00000000000000000000:00000000000000051111", 2023-05-31T21:24:13.437 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T21:24:13.437 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 55, 2023-05-31T21:24:13.438 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T21:24:13.438 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2023-05-31T21:21:35.866368Z" 2023-05-31T21:24:13.438 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T21:24:13.439 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:24:13.439 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T21:24:13.439 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 3, 2023-05-31T21:24:13.439 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T21:24:13.439 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "incremental-sync", 2023-05-31T21:24:13.439 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "00000000000000000000:00000000000000044038", 2023-05-31T21:24:13.440 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T21:24:13.440 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 9, 2023-05-31T21:24:13.440 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T21:24:13.440 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2023-05-31T21:21:35.912562Z" 2023-05-31T21:24:13.440 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T21:24:13.440 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T21:24:13.441 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2023-05-31T21:24:13.441 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:24:13.441 INFO:teuthology.orchestra.run.smithi099.stdout: "full_sync": { 2023-05-31T21:24:13.441 INFO:teuthology.orchestra.run.smithi099.stdout: "total": 64, 2023-05-31T21:24:13.441 INFO:teuthology.orchestra.run.smithi099.stdout: "complete": 64 2023-05-31T21:24:13.441 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:24:13.441 INFO:teuthology.orchestra.run.smithi099.stdout: "current_time": "2023-05-31T21:24:13Z" 2023-05-31T21:24:13.442 INFO:teuthology.orchestra.run.smithi099.stdout:} 2023-05-31T21:24:13.443 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 452136274422638824 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000044955", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-05-31T21:21:35.807414Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000031499", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-05-31T21:12:40.667485Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000051111", "next_step_marker": "", "total_entries": 55, "pos": 0, "timestamp": "2023-05-31T21:21:35.866368Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000044038", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2023-05-31T21:21:35.912562Z" } } ] }, "full_sync": { "total": 64, "complete": 64 }, "current_time": "2023-05-31T21:24:13Z" } 2023-05-31T21:24:13.443 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000044955', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-05-31T21:21:35.807414Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000031499', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-05-31T21:12:40.667485Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000051111', 'next_step_marker': '', 'total_entries': 55, 'pos': 0, 'timestamp': '2023-05-31T21:21:35.866368Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000044038', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2023-05-31T21:21:35.912562Z'}}] 2023-05-31T21:24:13.444 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000045095', 1: '00000000000000000000:00000000000000031499', 2: '00000000000000000000:00000000000000051251', 3: '00000000000000000000:00000000000000044038'} 2023-05-31T21:24:13.444 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000044955', 1: '00000000000000000000:00000000000000031499', 2: '00000000000000000000:00000000000000051111', 3: '00000000000000000000:00000000000000044038'} 2023-05-31T21:24:13.444 WARNING:rgw_multi.tests:data of zone a2 behind zone a1: shard=0 master=00000000000000000000:00000000000000045095 target=00000000000000000000:00000000000000044955, shard=2 master=00000000000000000000:00000000000000051251 target=00000000000000000000:00000000000000051111 2023-05-31T21:24:18.445 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-31T21:24:18.446 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-31T21:24:18.446 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-05-31T21:24:18.504 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2023-05-31T21:24:18.505 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2023-05-31T21:24:18.567 INFO:teuthology.orchestra.run.smithi099.stdout:{ 2023-05-31T21:24:18.567 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_status": { 2023-05-31T21:24:18.567 INFO:teuthology.orchestra.run.smithi099.stdout: "info": { 2023-05-31T21:24:18.568 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "sync", 2023-05-31T21:24:18.568 INFO:teuthology.orchestra.run.smithi099.stdout: "num_shards": 4, 2023-05-31T21:24:18.568 INFO:teuthology.orchestra.run.smithi099.stdout: "instance_id": 452136274422638824 2023-05-31T21:24:18.568 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:24:18.571 INFO:teuthology.orchestra.run.smithi099.stdout: "markers": [ 2023-05-31T21:24:18.571 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T21:24:18.572 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 0, 2023-05-31T21:24:18.572 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T21:24:18.572 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "incremental-sync", 2023-05-31T21:24:18.572 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "00000000000000000000:00000000000000044955", 2023-05-31T21:24:18.572 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T21:24:18.572 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2023-05-31T21:24:18.572 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T21:24:18.573 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2023-05-31T21:21:35.807414Z" 2023-05-31T21:24:18.573 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T21:24:18.573 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:24:18.573 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T21:24:18.573 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 1, 2023-05-31T21:24:18.573 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T21:24:18.574 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "incremental-sync", 2023-05-31T21:24:18.574 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "00000000000000000000:00000000000000031499", 2023-05-31T21:24:18.574 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T21:24:18.574 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2023-05-31T21:24:18.574 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T21:24:18.575 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2023-05-31T21:12:40.667485Z" 2023-05-31T21:24:18.575 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T21:24:18.575 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:24:18.575 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T21:24:18.575 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 2, 2023-05-31T21:24:18.575 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T21:24:18.575 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "incremental-sync", 2023-05-31T21:24:18.576 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "00000000000000000000:00000000000000051111", 2023-05-31T21:24:18.576 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T21:24:18.576 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 55, 2023-05-31T21:24:18.576 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T21:24:18.576 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2023-05-31T21:21:35.866368Z" 2023-05-31T21:24:18.576 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T21:24:18.577 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:24:18.577 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T21:24:18.577 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 3, 2023-05-31T21:24:18.577 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T21:24:18.577 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "incremental-sync", 2023-05-31T21:24:18.577 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "00000000000000000000:00000000000000044038", 2023-05-31T21:24:18.577 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T21:24:18.578 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 9, 2023-05-31T21:24:18.578 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T21:24:18.578 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2023-05-31T21:21:35.912562Z" 2023-05-31T21:24:18.578 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T21:24:18.578 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T21:24:18.578 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2023-05-31T21:24:18.578 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:24:18.579 INFO:teuthology.orchestra.run.smithi099.stdout: "full_sync": { 2023-05-31T21:24:18.579 INFO:teuthology.orchestra.run.smithi099.stdout: "total": 64, 2023-05-31T21:24:18.579 INFO:teuthology.orchestra.run.smithi099.stdout: "complete": 64 2023-05-31T21:24:18.579 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:24:18.580 INFO:teuthology.orchestra.run.smithi099.stdout: "current_time": "2023-05-31T21:24:18Z" 2023-05-31T21:24:18.580 INFO:teuthology.orchestra.run.smithi099.stdout:} 2023-05-31T21:24:18.581 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 452136274422638824 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000044955", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-05-31T21:21:35.807414Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000031499", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-05-31T21:12:40.667485Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000051111", "next_step_marker": "", "total_entries": 55, "pos": 0, "timestamp": "2023-05-31T21:21:35.866368Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000044038", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2023-05-31T21:21:35.912562Z" } } ] }, "full_sync": { "total": 64, "complete": 64 }, "current_time": "2023-05-31T21:24:18Z" } 2023-05-31T21:24:18.581 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000044955', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-05-31T21:21:35.807414Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000031499', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-05-31T21:12:40.667485Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000051111', 'next_step_marker': '', 'total_entries': 55, 'pos': 0, 'timestamp': '2023-05-31T21:21:35.866368Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000044038', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2023-05-31T21:21:35.912562Z'}}] 2023-05-31T21:24:18.582 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000045095', 1: '00000000000000000000:00000000000000031499', 2: '00000000000000000000:00000000000000051251', 3: '00000000000000000000:00000000000000044038'} 2023-05-31T21:24:18.582 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000044955', 1: '00000000000000000000:00000000000000031499', 2: '00000000000000000000:00000000000000051111', 3: '00000000000000000000:00000000000000044038'} 2023-05-31T21:24:18.582 WARNING:rgw_multi.tests:data of zone a2 behind zone a1: shard=0 master=00000000000000000000:00000000000000045095 target=00000000000000000000:00000000000000044955, shard=2 master=00000000000000000000:00000000000000051251 target=00000000000000000000:00000000000000051111 2023-05-31T21:24:23.583 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-31T21:24:23.584 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-31T21:24:23.584 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-05-31T21:24:23.644 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2023-05-31T21:24:23.645 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2023-05-31T21:24:23.707 INFO:teuthology.orchestra.run.smithi099.stdout:{ 2023-05-31T21:24:23.708 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_status": { 2023-05-31T21:24:23.708 INFO:teuthology.orchestra.run.smithi099.stdout: "info": { 2023-05-31T21:24:23.708 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "sync", 2023-05-31T21:24:23.709 INFO:teuthology.orchestra.run.smithi099.stdout: "num_shards": 4, 2023-05-31T21:24:23.709 INFO:teuthology.orchestra.run.smithi099.stdout: "instance_id": 452136274422638824 2023-05-31T21:24:23.709 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:24:23.709 INFO:teuthology.orchestra.run.smithi099.stdout: "markers": [ 2023-05-31T21:24:23.709 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T21:24:23.709 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 0, 2023-05-31T21:24:23.710 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T21:24:23.710 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "incremental-sync", 2023-05-31T21:24:23.710 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "00000000000000000000:00000000000000044955", 2023-05-31T21:24:23.710 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T21:24:23.710 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2023-05-31T21:24:23.710 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T21:24:23.710 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2023-05-31T21:21:35.807414Z" 2023-05-31T21:24:23.711 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T21:24:23.711 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:24:23.711 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T21:24:23.711 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 1, 2023-05-31T21:24:23.711 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T21:24:23.712 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "incremental-sync", 2023-05-31T21:24:23.712 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "00000000000000000000:00000000000000031499", 2023-05-31T21:24:23.712 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T21:24:23.713 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2023-05-31T21:24:23.713 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T21:24:23.713 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2023-05-31T21:12:40.667485Z" 2023-05-31T21:24:23.713 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T21:24:23.713 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:24:23.713 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T21:24:23.714 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 2, 2023-05-31T21:24:23.714 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T21:24:23.714 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "incremental-sync", 2023-05-31T21:24:23.714 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "00000000000000000000:00000000000000051111", 2023-05-31T21:24:23.714 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T21:24:23.714 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 55, 2023-05-31T21:24:23.715 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T21:24:23.715 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2023-05-31T21:21:35.866368Z" 2023-05-31T21:24:23.715 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T21:24:23.715 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:24:23.716 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T21:24:23.716 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 3, 2023-05-31T21:24:23.716 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T21:24:23.716 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "incremental-sync", 2023-05-31T21:24:23.717 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "00000000000000000000:00000000000000044038", 2023-05-31T21:24:23.717 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T21:24:23.717 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 9, 2023-05-31T21:24:23.717 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T21:24:23.717 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2023-05-31T21:21:35.912562Z" 2023-05-31T21:24:23.717 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T21:24:23.718 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T21:24:23.718 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2023-05-31T21:24:23.718 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:24:23.718 INFO:teuthology.orchestra.run.smithi099.stdout: "full_sync": { 2023-05-31T21:24:23.718 INFO:teuthology.orchestra.run.smithi099.stdout: "total": 64, 2023-05-31T21:24:23.718 INFO:teuthology.orchestra.run.smithi099.stdout: "complete": 64 2023-05-31T21:24:23.719 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:24:23.719 INFO:teuthology.orchestra.run.smithi099.stdout: "current_time": "2023-05-31T21:24:23Z" 2023-05-31T21:24:23.719 INFO:teuthology.orchestra.run.smithi099.stdout:} 2023-05-31T21:24:23.720 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 452136274422638824 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000044955", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-05-31T21:21:35.807414Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000031499", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-05-31T21:12:40.667485Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000051111", "next_step_marker": "", "total_entries": 55, "pos": 0, "timestamp": "2023-05-31T21:21:35.866368Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000044038", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2023-05-31T21:21:35.912562Z" } } ] }, "full_sync": { "total": 64, "complete": 64 }, "current_time": "2023-05-31T21:24:23Z" } 2023-05-31T21:24:23.721 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000044955', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-05-31T21:21:35.807414Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000031499', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-05-31T21:12:40.667485Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000051111', 'next_step_marker': '', 'total_entries': 55, 'pos': 0, 'timestamp': '2023-05-31T21:21:35.866368Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000044038', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2023-05-31T21:21:35.912562Z'}}] 2023-05-31T21:24:23.721 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000045095', 1: '00000000000000000000:00000000000000031499', 2: '00000000000000000000:00000000000000051251', 3: '00000000000000000000:00000000000000044038'} 2023-05-31T21:24:23.722 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000044955', 1: '00000000000000000000:00000000000000031499', 2: '00000000000000000000:00000000000000051111', 3: '00000000000000000000:00000000000000044038'} 2023-05-31T21:24:23.722 WARNING:rgw_multi.tests:data of zone a2 behind zone a1: shard=0 master=00000000000000000000:00000000000000045095 target=00000000000000000000:00000000000000044955, shard=2 master=00000000000000000000:00000000000000051251 target=00000000000000000000:00000000000000051111 2023-05-31T21:24:28.724 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-31T21:24:28.725 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-31T21:24:28.725 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-05-31T21:24:28.785 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2023-05-31T21:24:28.785 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2023-05-31T21:24:28.845 INFO:teuthology.orchestra.run.smithi099.stdout:{ 2023-05-31T21:24:28.845 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_status": { 2023-05-31T21:24:28.845 INFO:teuthology.orchestra.run.smithi099.stdout: "info": { 2023-05-31T21:24:28.845 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "sync", 2023-05-31T21:24:28.846 INFO:teuthology.orchestra.run.smithi099.stdout: "num_shards": 4, 2023-05-31T21:24:28.846 INFO:teuthology.orchestra.run.smithi099.stdout: "instance_id": 452136274422638824 2023-05-31T21:24:28.846 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:24:28.846 INFO:teuthology.orchestra.run.smithi099.stdout: "markers": [ 2023-05-31T21:24:28.846 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T21:24:28.846 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 0, 2023-05-31T21:24:28.846 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T21:24:28.847 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "incremental-sync", 2023-05-31T21:24:28.847 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "00000000000000000000:00000000000000044955", 2023-05-31T21:24:28.847 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T21:24:28.847 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2023-05-31T21:24:28.847 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T21:24:28.847 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2023-05-31T21:21:35.807414Z" 2023-05-31T21:24:28.848 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T21:24:28.848 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:24:28.848 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T21:24:28.848 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 1, 2023-05-31T21:24:28.848 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T21:24:28.848 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "incremental-sync", 2023-05-31T21:24:28.848 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "00000000000000000000:00000000000000031499", 2023-05-31T21:24:28.849 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T21:24:28.849 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2023-05-31T21:24:28.849 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T21:24:28.849 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2023-05-31T21:12:40.667485Z" 2023-05-31T21:24:28.849 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T21:24:28.849 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:24:28.849 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T21:24:28.850 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 2, 2023-05-31T21:24:28.850 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T21:24:28.850 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "incremental-sync", 2023-05-31T21:24:28.850 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "00000000000000000000:00000000000000051111", 2023-05-31T21:24:28.850 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T21:24:28.850 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 55, 2023-05-31T21:24:28.850 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T21:24:28.851 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2023-05-31T21:21:35.866368Z" 2023-05-31T21:24:28.851 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T21:24:28.851 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:24:28.851 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T21:24:28.851 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 3, 2023-05-31T21:24:28.851 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T21:24:28.851 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "incremental-sync", 2023-05-31T21:24:28.852 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "00000000000000000000:00000000000000044038", 2023-05-31T21:24:28.852 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T21:24:28.852 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 9, 2023-05-31T21:24:28.852 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T21:24:28.852 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2023-05-31T21:21:35.912562Z" 2023-05-31T21:24:28.853 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T21:24:28.853 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T21:24:28.853 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2023-05-31T21:24:28.853 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:24:28.853 INFO:teuthology.orchestra.run.smithi099.stdout: "full_sync": { 2023-05-31T21:24:28.854 INFO:teuthology.orchestra.run.smithi099.stdout: "total": 64, 2023-05-31T21:24:28.854 INFO:teuthology.orchestra.run.smithi099.stdout: "complete": 64 2023-05-31T21:24:28.854 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:24:28.854 INFO:teuthology.orchestra.run.smithi099.stdout: "current_time": "2023-05-31T21:24:28Z" 2023-05-31T21:24:28.854 INFO:teuthology.orchestra.run.smithi099.stdout:} 2023-05-31T21:24:28.855 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 452136274422638824 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000044955", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-05-31T21:21:35.807414Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000031499", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-05-31T21:12:40.667485Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000051111", "next_step_marker": "", "total_entries": 55, "pos": 0, "timestamp": "2023-05-31T21:21:35.866368Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000044038", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2023-05-31T21:21:35.912562Z" } } ] }, "full_sync": { "total": 64, "complete": 64 }, "current_time": "2023-05-31T21:24:28Z" } 2023-05-31T21:24:28.856 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000044955', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-05-31T21:21:35.807414Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000031499', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-05-31T21:12:40.667485Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000051111', 'next_step_marker': '', 'total_entries': 55, 'pos': 0, 'timestamp': '2023-05-31T21:21:35.866368Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000044038', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2023-05-31T21:21:35.912562Z'}}] 2023-05-31T21:24:28.856 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000045095', 1: '00000000000000000000:00000000000000031499', 2: '00000000000000000000:00000000000000051251', 3: '00000000000000000000:00000000000000044038'} 2023-05-31T21:24:28.856 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000044955', 1: '00000000000000000000:00000000000000031499', 2: '00000000000000000000:00000000000000051111', 3: '00000000000000000000:00000000000000044038'} 2023-05-31T21:24:28.857 WARNING:rgw_multi.tests:data of zone a2 behind zone a1: shard=0 master=00000000000000000000:00000000000000045095 target=00000000000000000000:00000000000000044955, shard=2 master=00000000000000000000:00000000000000051251 target=00000000000000000000:00000000000000051111 2023-05-31T21:24:33.857 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-31T21:24:33.858 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-31T21:24:33.859 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-05-31T21:24:33.918 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2023-05-31T21:24:33.919 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2023-05-31T21:24:33.989 INFO:teuthology.orchestra.run.smithi099.stdout:{ 2023-05-31T21:24:33.990 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_status": { 2023-05-31T21:24:33.990 INFO:teuthology.orchestra.run.smithi099.stdout: "info": { 2023-05-31T21:24:33.990 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "sync", 2023-05-31T21:24:33.991 INFO:teuthology.orchestra.run.smithi099.stdout: "num_shards": 4, 2023-05-31T21:24:33.991 INFO:teuthology.orchestra.run.smithi099.stdout: "instance_id": 452136274422638824 2023-05-31T21:24:33.991 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:24:33.992 INFO:teuthology.orchestra.run.smithi099.stdout: "markers": [ 2023-05-31T21:24:33.992 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T21:24:33.992 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 0, 2023-05-31T21:24:33.993 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T21:24:33.993 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "incremental-sync", 2023-05-31T21:24:33.993 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "00000000000000000000:00000000000000044955", 2023-05-31T21:24:33.996 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T21:24:33.996 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2023-05-31T21:24:33.996 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T21:24:33.997 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2023-05-31T21:21:35.807414Z" 2023-05-31T21:24:33.997 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T21:24:33.997 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:24:33.997 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T21:24:33.997 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 1, 2023-05-31T21:24:33.997 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T21:24:33.998 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "incremental-sync", 2023-05-31T21:24:33.998 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "00000000000000000000:00000000000000031499", 2023-05-31T21:24:33.998 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T21:24:33.998 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2023-05-31T21:24:33.998 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T21:24:33.998 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2023-05-31T21:12:40.667485Z" 2023-05-31T21:24:33.999 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T21:24:33.999 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:24:33.999 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T21:24:33.999 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 2, 2023-05-31T21:24:33.999 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T21:24:33.999 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "incremental-sync", 2023-05-31T21:24:33.999 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "00000000000000000000:00000000000000051111", 2023-05-31T21:24:34.000 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T21:24:34.000 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 55, 2023-05-31T21:24:34.000 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T21:24:34.000 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2023-05-31T21:21:35.866368Z" 2023-05-31T21:24:34.000 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T21:24:34.000 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:24:34.001 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T21:24:34.004 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 3, 2023-05-31T21:24:34.004 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T21:24:34.005 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "incremental-sync", 2023-05-31T21:24:34.005 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "00000000000000000000:00000000000000044038", 2023-05-31T21:24:34.005 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T21:24:34.005 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 9, 2023-05-31T21:24:34.005 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T21:24:34.005 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2023-05-31T21:21:35.912562Z" 2023-05-31T21:24:34.006 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T21:24:34.006 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T21:24:34.006 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2023-05-31T21:24:34.006 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:24:34.006 INFO:teuthology.orchestra.run.smithi099.stdout: "full_sync": { 2023-05-31T21:24:34.006 INFO:teuthology.orchestra.run.smithi099.stdout: "total": 64, 2023-05-31T21:24:34.006 INFO:teuthology.orchestra.run.smithi099.stdout: "complete": 64 2023-05-31T21:24:34.007 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:24:34.007 INFO:teuthology.orchestra.run.smithi099.stdout: "current_time": "2023-05-31T21:24:33Z" 2023-05-31T21:24:34.007 INFO:teuthology.orchestra.run.smithi099.stdout:} 2023-05-31T21:24:34.008 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 452136274422638824 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000044955", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-05-31T21:21:35.807414Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000031499", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-05-31T21:12:40.667485Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000051111", "next_step_marker": "", "total_entries": 55, "pos": 0, "timestamp": "2023-05-31T21:21:35.866368Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000044038", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2023-05-31T21:21:35.912562Z" } } ] }, "full_sync": { "total": 64, "complete": 64 }, "current_time": "2023-05-31T21:24:33Z" } 2023-05-31T21:24:34.008 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000044955', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-05-31T21:21:35.807414Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000031499', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-05-31T21:12:40.667485Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000051111', 'next_step_marker': '', 'total_entries': 55, 'pos': 0, 'timestamp': '2023-05-31T21:21:35.866368Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000044038', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2023-05-31T21:21:35.912562Z'}}] 2023-05-31T21:24:34.009 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000045095', 1: '00000000000000000000:00000000000000031499', 2: '00000000000000000000:00000000000000051251', 3: '00000000000000000000:00000000000000044038'} 2023-05-31T21:24:34.009 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000044955', 1: '00000000000000000000:00000000000000031499', 2: '00000000000000000000:00000000000000051111', 3: '00000000000000000000:00000000000000044038'} 2023-05-31T21:24:34.009 WARNING:rgw_multi.tests:data of zone a2 behind zone a1: shard=0 master=00000000000000000000:00000000000000045095 target=00000000000000000000:00000000000000044955, shard=2 master=00000000000000000000:00000000000000051251 target=00000000000000000000:00000000000000051111 2023-05-31T21:24:39.011 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-31T21:24:39.012 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-31T21:24:39.014 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-05-31T21:24:39.075 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2023-05-31T21:24:39.075 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2023-05-31T21:24:39.143 INFO:teuthology.orchestra.run.smithi099.stdout:{ 2023-05-31T21:24:39.143 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_status": { 2023-05-31T21:24:39.143 INFO:teuthology.orchestra.run.smithi099.stdout: "info": { 2023-05-31T21:24:39.143 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "sync", 2023-05-31T21:24:39.144 INFO:teuthology.orchestra.run.smithi099.stdout: "num_shards": 4, 2023-05-31T21:24:39.144 INFO:teuthology.orchestra.run.smithi099.stdout: "instance_id": 452136274422638824 2023-05-31T21:24:39.144 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:24:39.144 INFO:teuthology.orchestra.run.smithi099.stdout: "markers": [ 2023-05-31T21:24:39.144 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T21:24:39.144 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 0, 2023-05-31T21:24:39.145 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T21:24:39.145 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "incremental-sync", 2023-05-31T21:24:39.145 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "00000000000000000000:00000000000000044955", 2023-05-31T21:24:39.145 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T21:24:39.145 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2023-05-31T21:24:39.145 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T21:24:39.146 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2023-05-31T21:21:35.807414Z" 2023-05-31T21:24:39.146 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T21:24:39.146 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:24:39.146 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T21:24:39.146 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 1, 2023-05-31T21:24:39.146 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T21:24:39.147 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "incremental-sync", 2023-05-31T21:24:39.147 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "00000000000000000000:00000000000000031499", 2023-05-31T21:24:39.147 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T21:24:39.147 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2023-05-31T21:24:39.147 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T21:24:39.147 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2023-05-31T21:12:40.667485Z" 2023-05-31T21:24:39.147 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T21:24:39.148 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:24:39.148 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T21:24:39.148 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 2, 2023-05-31T21:24:39.148 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T21:24:39.148 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "incremental-sync", 2023-05-31T21:24:39.148 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "00000000000000000000:00000000000000051111", 2023-05-31T21:24:39.149 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T21:24:39.149 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 55, 2023-05-31T21:24:39.149 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T21:24:39.149 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2023-05-31T21:21:35.866368Z" 2023-05-31T21:24:39.150 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T21:24:39.150 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:24:39.150 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T21:24:39.150 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 3, 2023-05-31T21:24:39.151 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T21:24:39.151 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "incremental-sync", 2023-05-31T21:24:39.151 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "00000000000000000000:00000000000000044038", 2023-05-31T21:24:39.151 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T21:24:39.151 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 9, 2023-05-31T21:24:39.151 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T21:24:39.151 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2023-05-31T21:21:35.912562Z" 2023-05-31T21:24:39.152 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T21:24:39.152 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T21:24:39.152 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2023-05-31T21:24:39.152 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:24:39.152 INFO:teuthology.orchestra.run.smithi099.stdout: "full_sync": { 2023-05-31T21:24:39.152 INFO:teuthology.orchestra.run.smithi099.stdout: "total": 64, 2023-05-31T21:24:39.152 INFO:teuthology.orchestra.run.smithi099.stdout: "complete": 64 2023-05-31T21:24:39.153 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:24:39.153 INFO:teuthology.orchestra.run.smithi099.stdout: "current_time": "2023-05-31T21:24:39Z" 2023-05-31T21:24:39.153 INFO:teuthology.orchestra.run.smithi099.stdout:} 2023-05-31T21:24:39.154 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 452136274422638824 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000044955", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-05-31T21:21:35.807414Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000031499", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-05-31T21:12:40.667485Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000051111", "next_step_marker": "", "total_entries": 55, "pos": 0, "timestamp": "2023-05-31T21:21:35.866368Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000044038", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2023-05-31T21:21:35.912562Z" } } ] }, "full_sync": { "total": 64, "complete": 64 }, "current_time": "2023-05-31T21:24:39Z" } 2023-05-31T21:24:39.155 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000044955', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-05-31T21:21:35.807414Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000031499', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-05-31T21:12:40.667485Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000051111', 'next_step_marker': '', 'total_entries': 55, 'pos': 0, 'timestamp': '2023-05-31T21:21:35.866368Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000044038', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2023-05-31T21:21:35.912562Z'}}] 2023-05-31T21:24:39.155 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000045095', 1: '00000000000000000000:00000000000000031499', 2: '00000000000000000000:00000000000000051251', 3: '00000000000000000000:00000000000000044038'} 2023-05-31T21:24:39.155 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000044955', 1: '00000000000000000000:00000000000000031499', 2: '00000000000000000000:00000000000000051111', 3: '00000000000000000000:00000000000000044038'} 2023-05-31T21:24:39.156 WARNING:rgw_multi.tests:data of zone a2 behind zone a1: shard=0 master=00000000000000000000:00000000000000045095 target=00000000000000000000:00000000000000044955, shard=2 master=00000000000000000000:00000000000000051251 target=00000000000000000000:00000000000000051111 2023-05-31T21:24:44.159 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-31T21:24:44.160 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-31T21:24:44.160 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-05-31T21:24:44.221 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2023-05-31T21:24:44.221 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2023-05-31T21:24:44.282 INFO:teuthology.orchestra.run.smithi099.stdout:{ 2023-05-31T21:24:44.282 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_status": { 2023-05-31T21:24:44.282 INFO:teuthology.orchestra.run.smithi099.stdout: "info": { 2023-05-31T21:24:44.283 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "sync", 2023-05-31T21:24:44.283 INFO:teuthology.orchestra.run.smithi099.stdout: "num_shards": 4, 2023-05-31T21:24:44.283 INFO:teuthology.orchestra.run.smithi099.stdout: "instance_id": 452136274422638824 2023-05-31T21:24:44.283 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:24:44.283 INFO:teuthology.orchestra.run.smithi099.stdout: "markers": [ 2023-05-31T21:24:44.283 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T21:24:44.283 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 0, 2023-05-31T21:24:44.284 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T21:24:44.284 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "incremental-sync", 2023-05-31T21:24:44.284 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "00000000000000000000:00000000000000044955", 2023-05-31T21:24:44.284 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T21:24:44.284 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2023-05-31T21:24:44.284 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T21:24:44.285 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2023-05-31T21:21:35.807414Z" 2023-05-31T21:24:44.285 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T21:24:44.285 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:24:44.285 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T21:24:44.285 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 1, 2023-05-31T21:24:44.285 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T21:24:44.286 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "incremental-sync", 2023-05-31T21:24:44.286 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "00000000000000000000:00000000000000031499", 2023-05-31T21:24:44.286 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T21:24:44.286 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2023-05-31T21:24:44.286 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T21:24:44.286 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2023-05-31T21:12:40.667485Z" 2023-05-31T21:24:44.287 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T21:24:44.287 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:24:44.287 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T21:24:44.287 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 2, 2023-05-31T21:24:44.287 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T21:24:44.288 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "incremental-sync", 2023-05-31T21:24:44.288 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "00000000000000000000:00000000000000051111", 2023-05-31T21:24:44.288 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T21:24:44.288 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 55, 2023-05-31T21:24:44.288 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T21:24:44.288 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2023-05-31T21:21:35.866368Z" 2023-05-31T21:24:44.289 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T21:24:44.289 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:24:44.289 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T21:24:44.289 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 3, 2023-05-31T21:24:44.289 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T21:24:44.289 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "incremental-sync", 2023-05-31T21:24:44.289 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "00000000000000000000:00000000000000044038", 2023-05-31T21:24:44.290 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T21:24:44.290 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 9, 2023-05-31T21:24:44.290 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T21:24:44.290 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2023-05-31T21:21:35.912562Z" 2023-05-31T21:24:44.290 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T21:24:44.290 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T21:24:44.290 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2023-05-31T21:24:44.291 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:24:44.291 INFO:teuthology.orchestra.run.smithi099.stdout: "full_sync": { 2023-05-31T21:24:44.291 INFO:teuthology.orchestra.run.smithi099.stdout: "total": 64, 2023-05-31T21:24:44.291 INFO:teuthology.orchestra.run.smithi099.stdout: "complete": 64 2023-05-31T21:24:44.291 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:24:44.291 INFO:teuthology.orchestra.run.smithi099.stdout: "current_time": "2023-05-31T21:24:44Z" 2023-05-31T21:24:44.291 INFO:teuthology.orchestra.run.smithi099.stdout:} 2023-05-31T21:24:44.292 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 452136274422638824 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000044955", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-05-31T21:21:35.807414Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000031499", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-05-31T21:12:40.667485Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000051111", "next_step_marker": "", "total_entries": 55, "pos": 0, "timestamp": "2023-05-31T21:21:35.866368Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000044038", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2023-05-31T21:21:35.912562Z" } } ] }, "full_sync": { "total": 64, "complete": 64 }, "current_time": "2023-05-31T21:24:44Z" } 2023-05-31T21:24:44.293 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000044955', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-05-31T21:21:35.807414Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000031499', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-05-31T21:12:40.667485Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000051111', 'next_step_marker': '', 'total_entries': 55, 'pos': 0, 'timestamp': '2023-05-31T21:21:35.866368Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000044038', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2023-05-31T21:21:35.912562Z'}}] 2023-05-31T21:24:44.294 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000045095', 1: '00000000000000000000:00000000000000031499', 2: '00000000000000000000:00000000000000051251', 3: '00000000000000000000:00000000000000044038'} 2023-05-31T21:24:44.294 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000044955', 1: '00000000000000000000:00000000000000031499', 2: '00000000000000000000:00000000000000051111', 3: '00000000000000000000:00000000000000044038'} 2023-05-31T21:24:44.294 WARNING:rgw_multi.tests:data of zone a2 behind zone a1: shard=0 master=00000000000000000000:00000000000000045095 target=00000000000000000000:00000000000000044955, shard=2 master=00000000000000000000:00000000000000051251 target=00000000000000000000:00000000000000051111 2023-05-31T21:24:49.295 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-31T21:24:49.296 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-31T21:24:49.297 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-05-31T21:24:49.357 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2023-05-31T21:24:49.357 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2023-05-31T21:24:49.420 INFO:teuthology.orchestra.run.smithi099.stdout:{ 2023-05-31T21:24:49.420 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_status": { 2023-05-31T21:24:49.421 INFO:teuthology.orchestra.run.smithi099.stdout: "info": { 2023-05-31T21:24:49.421 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "sync", 2023-05-31T21:24:49.421 INFO:teuthology.orchestra.run.smithi099.stdout: "num_shards": 4, 2023-05-31T21:24:49.421 INFO:teuthology.orchestra.run.smithi099.stdout: "instance_id": 452136274422638824 2023-05-31T21:24:49.421 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:24:49.421 INFO:teuthology.orchestra.run.smithi099.stdout: "markers": [ 2023-05-31T21:24:49.422 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T21:24:49.422 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 0, 2023-05-31T21:24:49.422 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T21:24:49.422 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "incremental-sync", 2023-05-31T21:24:49.422 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "00000000000000000000:00000000000000044955", 2023-05-31T21:24:49.422 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T21:24:49.423 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2023-05-31T21:24:49.423 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T21:24:49.423 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2023-05-31T21:21:35.807414Z" 2023-05-31T21:24:49.423 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T21:24:49.423 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:24:49.423 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T21:24:49.423 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 1, 2023-05-31T21:24:49.424 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T21:24:49.424 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "incremental-sync", 2023-05-31T21:24:49.424 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "00000000000000000000:00000000000000031499", 2023-05-31T21:24:49.424 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T21:24:49.424 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2023-05-31T21:24:49.424 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T21:24:49.424 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2023-05-31T21:12:40.667485Z" 2023-05-31T21:24:49.425 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T21:24:49.425 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:24:49.425 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T21:24:49.425 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 2, 2023-05-31T21:24:49.425 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T21:24:49.425 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "incremental-sync", 2023-05-31T21:24:49.426 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "00000000000000000000:00000000000000051111", 2023-05-31T21:24:49.426 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T21:24:49.426 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 55, 2023-05-31T21:24:49.426 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T21:24:49.426 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2023-05-31T21:21:35.866368Z" 2023-05-31T21:24:49.426 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T21:24:49.426 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:24:49.427 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T21:24:49.427 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 3, 2023-05-31T21:24:49.427 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T21:24:49.427 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "incremental-sync", 2023-05-31T21:24:49.427 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "00000000000000000000:00000000000000044038", 2023-05-31T21:24:49.427 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T21:24:49.427 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 9, 2023-05-31T21:24:49.428 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T21:24:49.428 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2023-05-31T21:21:35.912562Z" 2023-05-31T21:24:49.428 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T21:24:49.428 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T21:24:49.428 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2023-05-31T21:24:49.428 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:24:49.429 INFO:teuthology.orchestra.run.smithi099.stdout: "full_sync": { 2023-05-31T21:24:49.429 INFO:teuthology.orchestra.run.smithi099.stdout: "total": 64, 2023-05-31T21:24:49.429 INFO:teuthology.orchestra.run.smithi099.stdout: "complete": 64 2023-05-31T21:24:49.429 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:24:49.429 INFO:teuthology.orchestra.run.smithi099.stdout: "current_time": "2023-05-31T21:24:49Z" 2023-05-31T21:24:49.429 INFO:teuthology.orchestra.run.smithi099.stdout:} 2023-05-31T21:24:49.430 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 452136274422638824 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000044955", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-05-31T21:21:35.807414Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000031499", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-05-31T21:12:40.667485Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000051111", "next_step_marker": "", "total_entries": 55, "pos": 0, "timestamp": "2023-05-31T21:21:35.866368Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000044038", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2023-05-31T21:21:35.912562Z" } } ] }, "full_sync": { "total": 64, "complete": 64 }, "current_time": "2023-05-31T21:24:49Z" } 2023-05-31T21:24:49.431 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000044955', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-05-31T21:21:35.807414Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000031499', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-05-31T21:12:40.667485Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000051111', 'next_step_marker': '', 'total_entries': 55, 'pos': 0, 'timestamp': '2023-05-31T21:21:35.866368Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000044038', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2023-05-31T21:21:35.912562Z'}}] 2023-05-31T21:24:49.431 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000045095', 1: '00000000000000000000:00000000000000031499', 2: '00000000000000000000:00000000000000051251', 3: '00000000000000000000:00000000000000044038'} 2023-05-31T21:24:49.431 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000044955', 1: '00000000000000000000:00000000000000031499', 2: '00000000000000000000:00000000000000051111', 3: '00000000000000000000:00000000000000044038'} 2023-05-31T21:24:49.432 WARNING:rgw_multi.tests:data of zone a2 behind zone a1: shard=0 master=00000000000000000000:00000000000000045095 target=00000000000000000000:00000000000000044955, shard=2 master=00000000000000000000:00000000000000051251 target=00000000000000000000:00000000000000051111 2023-05-31T21:24:54.433 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-31T21:24:54.433 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-31T21:24:54.434 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-05-31T21:24:54.493 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2023-05-31T21:24:54.493 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2023-05-31T21:24:54.570 INFO:teuthology.orchestra.run.smithi099.stdout:{ 2023-05-31T21:24:54.571 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_status": { 2023-05-31T21:24:54.571 INFO:teuthology.orchestra.run.smithi099.stdout: "info": { 2023-05-31T21:24:54.571 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "sync", 2023-05-31T21:24:54.572 INFO:teuthology.orchestra.run.smithi099.stdout: "num_shards": 4, 2023-05-31T21:24:54.572 INFO:teuthology.orchestra.run.smithi099.stdout: "instance_id": 452136274422638824 2023-05-31T21:24:54.572 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:24:54.573 INFO:teuthology.orchestra.run.smithi099.stdout: "markers": [ 2023-05-31T21:24:54.573 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T21:24:54.573 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 0, 2023-05-31T21:24:54.574 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T21:24:54.574 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "incremental-sync", 2023-05-31T21:24:54.575 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "00000000000000000000:00000000000000044955", 2023-05-31T21:24:54.575 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T21:24:54.575 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2023-05-31T21:24:54.576 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T21:24:54.576 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2023-05-31T21:21:35.807414Z" 2023-05-31T21:24:54.576 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T21:24:54.577 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:24:54.577 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T21:24:54.577 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 1, 2023-05-31T21:24:54.579 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T21:24:54.579 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "incremental-sync", 2023-05-31T21:24:54.579 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "00000000000000000000:00000000000000031499", 2023-05-31T21:24:54.579 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T21:24:54.579 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2023-05-31T21:24:54.579 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T21:24:54.580 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2023-05-31T21:12:40.667485Z" 2023-05-31T21:24:54.580 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T21:24:54.580 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:24:54.580 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T21:24:54.580 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 2, 2023-05-31T21:24:54.580 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T21:24:54.581 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "incremental-sync", 2023-05-31T21:24:54.581 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "00000000000000000000:00000000000000051111", 2023-05-31T21:24:54.581 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T21:24:54.581 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 55, 2023-05-31T21:24:54.581 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T21:24:54.581 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2023-05-31T21:21:35.866368Z" 2023-05-31T21:24:54.581 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T21:24:54.582 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:24:54.582 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T21:24:54.582 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 3, 2023-05-31T21:24:54.582 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T21:24:54.582 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "incremental-sync", 2023-05-31T21:24:54.582 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "00000000000000000000:00000000000000044038", 2023-05-31T21:24:54.582 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T21:24:54.583 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 9, 2023-05-31T21:24:54.583 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T21:24:54.583 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2023-05-31T21:21:35.912562Z" 2023-05-31T21:24:54.583 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T21:24:54.583 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T21:24:54.583 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2023-05-31T21:24:54.583 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:24:54.584 INFO:teuthology.orchestra.run.smithi099.stdout: "full_sync": { 2023-05-31T21:24:54.584 INFO:teuthology.orchestra.run.smithi099.stdout: "total": 64, 2023-05-31T21:24:54.584 INFO:teuthology.orchestra.run.smithi099.stdout: "complete": 64 2023-05-31T21:24:54.584 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:24:54.584 INFO:teuthology.orchestra.run.smithi099.stdout: "current_time": "2023-05-31T21:24:54Z" 2023-05-31T21:24:54.584 INFO:teuthology.orchestra.run.smithi099.stdout:} 2023-05-31T21:24:54.585 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 452136274422638824 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000044955", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-05-31T21:21:35.807414Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000031499", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-05-31T21:12:40.667485Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000051111", "next_step_marker": "", "total_entries": 55, "pos": 0, "timestamp": "2023-05-31T21:21:35.866368Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000044038", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2023-05-31T21:21:35.912562Z" } } ] }, "full_sync": { "total": 64, "complete": 64 }, "current_time": "2023-05-31T21:24:54Z" } 2023-05-31T21:24:54.586 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000044955', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-05-31T21:21:35.807414Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000031499', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-05-31T21:12:40.667485Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000051111', 'next_step_marker': '', 'total_entries': 55, 'pos': 0, 'timestamp': '2023-05-31T21:21:35.866368Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000044038', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2023-05-31T21:21:35.912562Z'}}] 2023-05-31T21:24:54.587 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000045095', 1: '00000000000000000000:00000000000000031499', 2: '00000000000000000000:00000000000000051251', 3: '00000000000000000000:00000000000000044038'} 2023-05-31T21:24:54.587 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000044955', 1: '00000000000000000000:00000000000000031499', 2: '00000000000000000000:00000000000000051111', 3: '00000000000000000000:00000000000000044038'} 2023-05-31T21:24:54.587 WARNING:rgw_multi.tests:data of zone a2 behind zone a1: shard=0 master=00000000000000000000:00000000000000045095 target=00000000000000000000:00000000000000044955, shard=2 master=00000000000000000000:00000000000000051251 target=00000000000000000000:00000000000000051111 2023-05-31T21:24:59.589 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-31T21:24:59.590 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-31T21:24:59.590 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-05-31T21:24:59.649 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2023-05-31T21:24:59.649 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2023-05-31T21:24:59.713 INFO:teuthology.orchestra.run.smithi099.stdout:{ 2023-05-31T21:24:59.713 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_status": { 2023-05-31T21:24:59.713 INFO:teuthology.orchestra.run.smithi099.stdout: "info": { 2023-05-31T21:24:59.713 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "sync", 2023-05-31T21:24:59.713 INFO:teuthology.orchestra.run.smithi099.stdout: "num_shards": 4, 2023-05-31T21:24:59.714 INFO:teuthology.orchestra.run.smithi099.stdout: "instance_id": 452136274422638824 2023-05-31T21:24:59.714 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:24:59.714 INFO:teuthology.orchestra.run.smithi099.stdout: "markers": [ 2023-05-31T21:24:59.714 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T21:24:59.714 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 0, 2023-05-31T21:24:59.714 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T21:24:59.715 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "incremental-sync", 2023-05-31T21:24:59.715 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "00000000000000000000:00000000000000044955", 2023-05-31T21:24:59.715 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T21:24:59.715 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2023-05-31T21:24:59.715 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T21:24:59.715 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2023-05-31T21:21:35.807414Z" 2023-05-31T21:24:59.715 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T21:24:59.716 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:24:59.716 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T21:24:59.716 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 1, 2023-05-31T21:24:59.716 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T21:24:59.716 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "incremental-sync", 2023-05-31T21:24:59.716 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "00000000000000000000:00000000000000031499", 2023-05-31T21:24:59.717 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T21:24:59.717 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2023-05-31T21:24:59.717 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T21:24:59.717 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2023-05-31T21:12:40.667485Z" 2023-05-31T21:24:59.718 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T21:24:59.718 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:24:59.718 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T21:24:59.718 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 2, 2023-05-31T21:24:59.718 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T21:24:59.718 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "incremental-sync", 2023-05-31T21:24:59.718 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "00000000000000000000:00000000000000051111", 2023-05-31T21:24:59.719 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T21:24:59.719 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 55, 2023-05-31T21:24:59.719 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T21:24:59.719 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2023-05-31T21:21:35.866368Z" 2023-05-31T21:24:59.719 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T21:24:59.719 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:24:59.719 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T21:24:59.720 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 3, 2023-05-31T21:24:59.720 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T21:24:59.720 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "incremental-sync", 2023-05-31T21:24:59.720 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "00000000000000000000:00000000000000044038", 2023-05-31T21:24:59.720 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T21:24:59.720 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 9, 2023-05-31T21:24:59.721 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T21:24:59.721 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2023-05-31T21:21:35.912562Z" 2023-05-31T21:24:59.721 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T21:24:59.721 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T21:24:59.721 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2023-05-31T21:24:59.721 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:24:59.721 INFO:teuthology.orchestra.run.smithi099.stdout: "full_sync": { 2023-05-31T21:24:59.722 INFO:teuthology.orchestra.run.smithi099.stdout: "total": 64, 2023-05-31T21:24:59.722 INFO:teuthology.orchestra.run.smithi099.stdout: "complete": 64 2023-05-31T21:24:59.722 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:24:59.722 INFO:teuthology.orchestra.run.smithi099.stdout: "current_time": "2023-05-31T21:24:59Z" 2023-05-31T21:24:59.722 INFO:teuthology.orchestra.run.smithi099.stdout:} 2023-05-31T21:24:59.723 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 452136274422638824 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000044955", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-05-31T21:21:35.807414Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000031499", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-05-31T21:12:40.667485Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000051111", "next_step_marker": "", "total_entries": 55, "pos": 0, "timestamp": "2023-05-31T21:21:35.866368Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000044038", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2023-05-31T21:21:35.912562Z" } } ] }, "full_sync": { "total": 64, "complete": 64 }, "current_time": "2023-05-31T21:24:59Z" } 2023-05-31T21:24:59.724 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000044955', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-05-31T21:21:35.807414Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000031499', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-05-31T21:12:40.667485Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000051111', 'next_step_marker': '', 'total_entries': 55, 'pos': 0, 'timestamp': '2023-05-31T21:21:35.866368Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000044038', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2023-05-31T21:21:35.912562Z'}}] 2023-05-31T21:24:59.724 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000045095', 1: '00000000000000000000:00000000000000031499', 2: '00000000000000000000:00000000000000051251', 3: '00000000000000000000:00000000000000044038'} 2023-05-31T21:24:59.725 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000044955', 1: '00000000000000000000:00000000000000031499', 2: '00000000000000000000:00000000000000051111', 3: '00000000000000000000:00000000000000044038'} 2023-05-31T21:24:59.725 WARNING:rgw_multi.tests:data of zone a2 behind zone a1: shard=0 master=00000000000000000000:00000000000000045095 target=00000000000000000000:00000000000000044955, shard=2 master=00000000000000000000:00000000000000051251 target=00000000000000000000:00000000000000051111 2023-05-31T21:25:04.726 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-31T21:25:04.727 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-31T21:25:04.727 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-05-31T21:25:04.786 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2023-05-31T21:25:04.787 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2023-05-31T21:25:04.849 INFO:teuthology.orchestra.run.smithi099.stdout:{ 2023-05-31T21:25:04.850 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_status": { 2023-05-31T21:25:04.850 INFO:teuthology.orchestra.run.smithi099.stdout: "info": { 2023-05-31T21:25:04.850 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "sync", 2023-05-31T21:25:04.850 INFO:teuthology.orchestra.run.smithi099.stdout: "num_shards": 4, 2023-05-31T21:25:04.850 INFO:teuthology.orchestra.run.smithi099.stdout: "instance_id": 452136274422638824 2023-05-31T21:25:04.851 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:25:04.851 INFO:teuthology.orchestra.run.smithi099.stdout: "markers": [ 2023-05-31T21:25:04.851 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T21:25:04.851 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 0, 2023-05-31T21:25:04.851 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T21:25:04.851 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "incremental-sync", 2023-05-31T21:25:04.851 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "00000000000000000000:00000000000000044955", 2023-05-31T21:25:04.852 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T21:25:04.852 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2023-05-31T21:25:04.852 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T21:25:04.852 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2023-05-31T21:21:35.807414Z" 2023-05-31T21:25:04.852 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T21:25:04.852 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:25:04.853 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T21:25:04.853 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 1, 2023-05-31T21:25:04.853 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T21:25:04.853 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "incremental-sync", 2023-05-31T21:25:04.853 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "00000000000000000000:00000000000000031499", 2023-05-31T21:25:04.853 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T21:25:04.853 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2023-05-31T21:25:04.854 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T21:25:04.854 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2023-05-31T21:12:40.667485Z" 2023-05-31T21:25:04.854 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T21:25:04.854 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:25:04.854 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T21:25:04.854 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 2, 2023-05-31T21:25:04.855 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T21:25:04.855 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "incremental-sync", 2023-05-31T21:25:04.855 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "00000000000000000000:00000000000000051111", 2023-05-31T21:25:04.855 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T21:25:04.855 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 55, 2023-05-31T21:25:04.855 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T21:25:04.855 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2023-05-31T21:21:35.866368Z" 2023-05-31T21:25:04.856 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T21:25:04.856 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:25:04.856 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T21:25:04.856 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 3, 2023-05-31T21:25:04.857 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T21:25:04.857 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "incremental-sync", 2023-05-31T21:25:04.857 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "00000000000000000000:00000000000000044038", 2023-05-31T21:25:04.857 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T21:25:04.857 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 9, 2023-05-31T21:25:04.857 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T21:25:04.857 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2023-05-31T21:21:35.912562Z" 2023-05-31T21:25:04.858 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T21:25:04.858 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T21:25:04.859 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2023-05-31T21:25:04.859 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:25:04.859 INFO:teuthology.orchestra.run.smithi099.stdout: "full_sync": { 2023-05-31T21:25:04.859 INFO:teuthology.orchestra.run.smithi099.stdout: "total": 64, 2023-05-31T21:25:04.859 INFO:teuthology.orchestra.run.smithi099.stdout: "complete": 64 2023-05-31T21:25:04.860 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:25:04.860 INFO:teuthology.orchestra.run.smithi099.stdout: "current_time": "2023-05-31T21:25:04Z" 2023-05-31T21:25:04.860 INFO:teuthology.orchestra.run.smithi099.stdout:} 2023-05-31T21:25:04.861 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 452136274422638824 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000044955", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-05-31T21:21:35.807414Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000031499", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-05-31T21:12:40.667485Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000051111", "next_step_marker": "", "total_entries": 55, "pos": 0, "timestamp": "2023-05-31T21:21:35.866368Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000044038", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2023-05-31T21:21:35.912562Z" } } ] }, "full_sync": { "total": 64, "complete": 64 }, "current_time": "2023-05-31T21:25:04Z" } 2023-05-31T21:25:04.861 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000044955', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-05-31T21:21:35.807414Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000031499', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-05-31T21:12:40.667485Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000051111', 'next_step_marker': '', 'total_entries': 55, 'pos': 0, 'timestamp': '2023-05-31T21:21:35.866368Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000044038', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2023-05-31T21:21:35.912562Z'}}] 2023-05-31T21:25:04.862 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000045095', 1: '00000000000000000000:00000000000000031499', 2: '00000000000000000000:00000000000000051251', 3: '00000000000000000000:00000000000000044038'} 2023-05-31T21:25:04.862 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000044955', 1: '00000000000000000000:00000000000000031499', 2: '00000000000000000000:00000000000000051111', 3: '00000000000000000000:00000000000000044038'} 2023-05-31T21:25:04.862 WARNING:rgw_multi.tests:data of zone a2 behind zone a1: shard=0 master=00000000000000000000:00000000000000045095 target=00000000000000000000:00000000000000044955, shard=2 master=00000000000000000000:00000000000000051251 target=00000000000000000000:00000000000000051111 2023-05-31T21:25:09.864 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-31T21:25:09.865 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-31T21:25:09.865 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-05-31T21:25:09.924 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2023-05-31T21:25:09.924 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2023-05-31T21:25:09.988 INFO:teuthology.orchestra.run.smithi099.stdout:{ 2023-05-31T21:25:09.988 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_status": { 2023-05-31T21:25:09.988 INFO:teuthology.orchestra.run.smithi099.stdout: "info": { 2023-05-31T21:25:09.988 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "sync", 2023-05-31T21:25:09.989 INFO:teuthology.orchestra.run.smithi099.stdout: "num_shards": 4, 2023-05-31T21:25:09.989 INFO:teuthology.orchestra.run.smithi099.stdout: "instance_id": 452136274422638824 2023-05-31T21:25:09.989 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:25:09.989 INFO:teuthology.orchestra.run.smithi099.stdout: "markers": [ 2023-05-31T21:25:09.989 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T21:25:09.989 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 0, 2023-05-31T21:25:09.990 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T21:25:09.990 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "incremental-sync", 2023-05-31T21:25:09.990 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "00000000000000000000:00000000000000044955", 2023-05-31T21:25:09.990 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T21:25:09.990 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2023-05-31T21:25:09.990 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T21:25:09.990 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2023-05-31T21:21:35.807414Z" 2023-05-31T21:25:09.991 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T21:25:09.991 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:25:09.991 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T21:25:09.991 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 1, 2023-05-31T21:25:09.991 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T21:25:09.991 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "incremental-sync", 2023-05-31T21:25:09.991 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "00000000000000000000:00000000000000031499", 2023-05-31T21:25:09.992 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T21:25:09.992 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2023-05-31T21:25:09.992 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T21:25:09.992 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2023-05-31T21:12:40.667485Z" 2023-05-31T21:25:09.992 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T21:25:09.992 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:25:09.992 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T21:25:09.993 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 2, 2023-05-31T21:25:09.993 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T21:25:09.993 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "incremental-sync", 2023-05-31T21:25:09.993 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "00000000000000000000:00000000000000051111", 2023-05-31T21:25:09.993 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T21:25:09.993 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 55, 2023-05-31T21:25:09.993 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T21:25:09.994 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2023-05-31T21:21:35.866368Z" 2023-05-31T21:25:09.994 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T21:25:09.994 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:25:09.994 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T21:25:09.994 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 3, 2023-05-31T21:25:09.994 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T21:25:09.995 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "incremental-sync", 2023-05-31T21:25:09.996 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "00000000000000000000:00000000000000044038", 2023-05-31T21:25:09.996 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T21:25:09.996 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 9, 2023-05-31T21:25:09.996 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T21:25:09.996 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2023-05-31T21:21:35.912562Z" 2023-05-31T21:25:09.996 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T21:25:09.997 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T21:25:09.997 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2023-05-31T21:25:09.997 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:25:09.997 INFO:teuthology.orchestra.run.smithi099.stdout: "full_sync": { 2023-05-31T21:25:09.997 INFO:teuthology.orchestra.run.smithi099.stdout: "total": 64, 2023-05-31T21:25:09.997 INFO:teuthology.orchestra.run.smithi099.stdout: "complete": 64 2023-05-31T21:25:09.997 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:25:09.998 INFO:teuthology.orchestra.run.smithi099.stdout: "current_time": "2023-05-31T21:25:09Z" 2023-05-31T21:25:09.998 INFO:teuthology.orchestra.run.smithi099.stdout:} 2023-05-31T21:25:09.999 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 452136274422638824 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000044955", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-05-31T21:21:35.807414Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000031499", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-05-31T21:12:40.667485Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000051111", "next_step_marker": "", "total_entries": 55, "pos": 0, "timestamp": "2023-05-31T21:21:35.866368Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000044038", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2023-05-31T21:21:35.912562Z" } } ] }, "full_sync": { "total": 64, "complete": 64 }, "current_time": "2023-05-31T21:25:09Z" } 2023-05-31T21:25:09.999 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000044955', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-05-31T21:21:35.807414Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000031499', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-05-31T21:12:40.667485Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000051111', 'next_step_marker': '', 'total_entries': 55, 'pos': 0, 'timestamp': '2023-05-31T21:21:35.866368Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000044038', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2023-05-31T21:21:35.912562Z'}}] 2023-05-31T21:25:10.000 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000045095', 1: '00000000000000000000:00000000000000031499', 2: '00000000000000000000:00000000000000051251', 3: '00000000000000000000:00000000000000044038'} 2023-05-31T21:25:10.000 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000044955', 1: '00000000000000000000:00000000000000031499', 2: '00000000000000000000:00000000000000051111', 3: '00000000000000000000:00000000000000044038'} 2023-05-31T21:25:10.000 WARNING:rgw_multi.tests:data of zone a2 behind zone a1: shard=0 master=00000000000000000000:00000000000000045095 target=00000000000000000000:00000000000000044955, shard=2 master=00000000000000000000:00000000000000051251 target=00000000000000000000:00000000000000051111 2023-05-31T21:25:12.764 INFO:tasks.rgw.c2.client.1.smithi112.stdout:2023-05-31T21:25:12.759+0000 7fecb0821700 -1 WARNING: The OSD cluster is overloaded and struggling to complete ops. You need more capacity to serve this level of demand. 2023-05-31T21:25:15.001 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-31T21:25:15.002 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-31T21:25:15.002 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-05-31T21:25:15.062 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2023-05-31T21:25:15.062 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2023-05-31T21:25:15.117 INFO:teuthology.orchestra.run.smithi099.stdout:{ 2023-05-31T21:25:15.118 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_status": { 2023-05-31T21:25:15.118 INFO:teuthology.orchestra.run.smithi099.stdout: "info": { 2023-05-31T21:25:15.118 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "sync", 2023-05-31T21:25:15.118 INFO:teuthology.orchestra.run.smithi099.stdout: "num_shards": 4, 2023-05-31T21:25:15.118 INFO:teuthology.orchestra.run.smithi099.stdout: "instance_id": 452136274422638824 2023-05-31T21:25:15.118 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:25:15.118 INFO:teuthology.orchestra.run.smithi099.stdout: "markers": [ 2023-05-31T21:25:15.119 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T21:25:15.119 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 0, 2023-05-31T21:25:15.119 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T21:25:15.119 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "incremental-sync", 2023-05-31T21:25:15.119 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "00000000000000000000:00000000000000045095", 2023-05-31T21:25:15.119 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T21:25:15.120 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2023-05-31T21:25:15.120 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T21:25:15.120 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2023-05-31T21:23:41.891197Z" 2023-05-31T21:25:15.120 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T21:25:15.120 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:25:15.120 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T21:25:15.120 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 1, 2023-05-31T21:25:15.121 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T21:25:15.121 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "incremental-sync", 2023-05-31T21:25:15.121 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "00000000000000000000:00000000000000031499", 2023-05-31T21:25:15.121 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T21:25:15.121 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2023-05-31T21:25:15.121 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T21:25:15.121 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2023-05-31T21:12:40.667485Z" 2023-05-31T21:25:15.122 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T21:25:15.122 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:25:15.122 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T21:25:15.122 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 2, 2023-05-31T21:25:15.122 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T21:25:15.122 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "incremental-sync", 2023-05-31T21:25:15.122 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "00000000000000000000:00000000000000051251", 2023-05-31T21:25:15.123 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T21:25:15.123 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 55, 2023-05-31T21:25:15.123 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T21:25:15.123 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2023-05-31T21:23:41.857089Z" 2023-05-31T21:25:15.123 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T21:25:15.123 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:25:15.123 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T21:25:15.124 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 3, 2023-05-31T21:25:15.124 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T21:25:15.124 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "incremental-sync", 2023-05-31T21:25:15.124 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "00000000000000000000:00000000000000044038", 2023-05-31T21:25:15.124 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T21:25:15.124 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 9, 2023-05-31T21:25:15.124 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T21:25:15.125 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2023-05-31T21:21:35.912562Z" 2023-05-31T21:25:15.125 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T21:25:15.125 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T21:25:15.125 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2023-05-31T21:25:15.125 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:25:15.125 INFO:teuthology.orchestra.run.smithi099.stdout: "full_sync": { 2023-05-31T21:25:15.125 INFO:teuthology.orchestra.run.smithi099.stdout: "total": 64, 2023-05-31T21:25:15.126 INFO:teuthology.orchestra.run.smithi099.stdout: "complete": 64 2023-05-31T21:25:15.127 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:25:15.127 INFO:teuthology.orchestra.run.smithi099.stdout: "current_time": "2023-05-31T21:25:15Z" 2023-05-31T21:25:15.127 INFO:teuthology.orchestra.run.smithi099.stdout:} 2023-05-31T21:25:15.128 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 452136274422638824 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000045095", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-05-31T21:23:41.891197Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000031499", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-05-31T21:12:40.667485Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000051251", "next_step_marker": "", "total_entries": 55, "pos": 0, "timestamp": "2023-05-31T21:23:41.857089Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000044038", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2023-05-31T21:21:35.912562Z" } } ] }, "full_sync": { "total": 64, "complete": 64 }, "current_time": "2023-05-31T21:25:15Z" } 2023-05-31T21:25:15.129 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000045095', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-05-31T21:23:41.891197Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000031499', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-05-31T21:12:40.667485Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000051251', 'next_step_marker': '', 'total_entries': 55, 'pos': 0, 'timestamp': '2023-05-31T21:23:41.857089Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000044038', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2023-05-31T21:21:35.912562Z'}}] 2023-05-31T21:25:15.129 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000045095', 1: '00000000000000000000:00000000000000031499', 2: '00000000000000000000:00000000000000051251', 3: '00000000000000000000:00000000000000044038'} 2023-05-31T21:25:15.129 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000045095', 1: '00000000000000000000:00000000000000031499', 2: '00000000000000000000:00000000000000051251', 3: '00000000000000000000:00000000000000044038'} 2023-05-31T21:25:15.130 INFO:rgw_multi.tests:finished data checkpoint for target_zone=a2 source_zone=a1 2023-05-31T21:25:15.130 DEBUG:boto:path=/pklxkd-70/ 2023-05-31T21:25:15.131 DEBUG:boto:auth_path=/pklxkd-70/ 2023-05-31T21:25:15.131 DEBUG:boto:Method: HEAD 2023-05-31T21:25:15.131 DEBUG:boto:Path: /pklxkd-70/ 2023-05-31T21:25:15.132 DEBUG:boto:Data: 2023-05-31T21:25:15.132 DEBUG:boto:Headers: {} 2023-05-31T21:25:15.132 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T21:25:15.133 DEBUG:boto:Port: 8001 2023-05-31T21:25:15.133 DEBUG:boto:Params: {} 2023-05-31T21:25:15.133 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8001} 2023-05-31T21:25:15.134 DEBUG:boto:Token: None 2023-05-31T21:25:15.134 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 21:25:15 GMT /pklxkd-70/ 2023-05-31T21:25:15.135 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:pdfre02ybYLhhZVPsS6z4AHA2eE= 2023-05-31T21:25:15.135 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 21:25:15 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:pdfre02ybYLhhZVPsS6z4AHA2eE=', 'Content-Length': '0'} 2023-05-31T21:25:15.140 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000009244299b29017e5-006477bb3b-2568-a2'), ('Content-Length', '0'), ('Date', 'Wed, 31 May 2023 21:25:15 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:25:15.141 DEBUG:boto:path=/pklxkd-70/obj1 2023-05-31T21:25:15.141 DEBUG:boto:auth_path=/pklxkd-70/obj1 2023-05-31T21:25:15.142 DEBUG:boto:Method: HEAD 2023-05-31T21:25:15.143 DEBUG:boto:Path: /pklxkd-70/obj1 2023-05-31T21:25:15.143 DEBUG:boto:Data: 2023-05-31T21:25:15.144 DEBUG:boto:Headers: {} 2023-05-31T21:25:15.144 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T21:25:15.144 DEBUG:boto:Port: 8001 2023-05-31T21:25:15.145 DEBUG:boto:Params: {} 2023-05-31T21:25:15.146 DEBUG:boto:Token: None 2023-05-31T21:25:15.146 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 21:25:15 GMT /pklxkd-70/obj1 2023-05-31T21:25:15.147 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:dNWxs2rwBRNmX25VIYTS4yh8m7E= 2023-05-31T21:25:15.148 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 21:25:15 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:dNWxs2rwBRNmX25VIYTS4yh8m7E=', 'Content-Length': '0'} 2023-05-31T21:25:15.150 DEBUG:boto:Response headers: [('Content-Length', '219'), ('x-amz-request-id', 'tx000009e90ebaa7e7eccf9-006477bb3b-2568-a2'), ('Accept-Ranges', 'bytes'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 21:25:15 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:25:15.150 DEBUG:boto:path=/pklxkd-70/obj2 2023-05-31T21:25:15.151 DEBUG:boto:auth_path=/pklxkd-70/obj2 2023-05-31T21:25:15.151 DEBUG:boto:Method: HEAD 2023-05-31T21:25:15.151 DEBUG:boto:Path: /pklxkd-70/obj2 2023-05-31T21:25:15.152 DEBUG:boto:Data: 2023-05-31T21:25:15.152 DEBUG:boto:Headers: {} 2023-05-31T21:25:15.153 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T21:25:15.153 DEBUG:boto:Port: 8001 2023-05-31T21:25:15.154 DEBUG:boto:Params: {} 2023-05-31T21:25:15.154 DEBUG:boto:Token: None 2023-05-31T21:25:15.155 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 21:25:15 GMT /pklxkd-70/obj2 2023-05-31T21:25:15.155 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:t4sUupKmptccQm3hxt11XRKlPlY= 2023-05-31T21:25:15.156 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 21:25:15 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:t4sUupKmptccQm3hxt11XRKlPlY=', 'Content-Length': '0'} 2023-05-31T21:25:15.158 DEBUG:boto:Response headers: [('Content-Length', '219'), ('x-amz-request-id', 'tx000004a59cdeef25826f2-006477bb3b-2568-a2'), ('Accept-Ranges', 'bytes'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 21:25:15 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:25:15.159 DEBUG:boto:path=/pklxkd-71/ 2023-05-31T21:25:15.160 DEBUG:boto:auth_path=/pklxkd-71/ 2023-05-31T21:25:15.161 DEBUG:boto:Method: HEAD 2023-05-31T21:25:15.161 DEBUG:boto:Path: /pklxkd-71/ 2023-05-31T21:25:15.162 DEBUG:boto:Data: 2023-05-31T21:25:15.162 DEBUG:boto:Headers: {} 2023-05-31T21:25:15.163 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T21:25:15.164 DEBUG:boto:Port: 8001 2023-05-31T21:25:15.164 DEBUG:boto:Params: {} 2023-05-31T21:25:15.165 DEBUG:boto:Token: None 2023-05-31T21:25:15.165 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 21:25:15 GMT /pklxkd-71/ 2023-05-31T21:25:15.166 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:6uGhApshjrFdPQ8cpLDVFozLAX8= 2023-05-31T21:25:15.166 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 21:25:15 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:6uGhApshjrFdPQ8cpLDVFozLAX8=', 'Content-Length': '0'} 2023-05-31T21:25:15.168 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '2'), ('X-RGW-Bytes-Used', '12'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000005fbf2a52ffec1dcc-006477bb3b-2568-a2'), ('Content-Length', '0'), ('Date', 'Wed, 31 May 2023 21:25:15 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:25:15.169 DEBUG:boto:path=/pklxkd-71/obj1 2023-05-31T21:25:15.169 DEBUG:boto:auth_path=/pklxkd-71/obj1 2023-05-31T21:25:15.170 DEBUG:boto:Method: HEAD 2023-05-31T21:25:15.170 DEBUG:boto:Path: /pklxkd-71/obj1 2023-05-31T21:25:15.171 DEBUG:boto:Data: 2023-05-31T21:25:15.171 DEBUG:boto:Headers: {} 2023-05-31T21:25:15.171 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T21:25:15.172 DEBUG:boto:Port: 8001 2023-05-31T21:25:15.172 DEBUG:boto:Params: {} 2023-05-31T21:25:15.173 DEBUG:boto:Token: None 2023-05-31T21:25:15.173 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 21:25:15 GMT /pklxkd-71/obj1 2023-05-31T21:25:15.173 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:XZH/hK7U0FLGuwD+FFPU8tUp84Q= 2023-05-31T21:25:15.174 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 21:25:15 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:XZH/hK7U0FLGuwD+FFPU8tUp84Q=', 'Content-Length': '0'} 2023-05-31T21:25:15.176 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 21:23:41 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000009ffae562a1ea6742-006477bb3b-2568-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 21:25:15 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:25:15.177 DEBUG:boto:path=/pklxkd-71/obj1 2023-05-31T21:25:15.178 DEBUG:boto:auth_path=/pklxkd-71/obj1 2023-05-31T21:25:15.179 DEBUG:boto:Method: GET 2023-05-31T21:25:15.180 DEBUG:boto:Path: /pklxkd-71/obj1 2023-05-31T21:25:15.181 DEBUG:boto:Data: 2023-05-31T21:25:15.181 DEBUG:boto:Headers: {} 2023-05-31T21:25:15.182 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T21:25:15.183 DEBUG:boto:Port: 8001 2023-05-31T21:25:15.184 DEBUG:boto:Params: {} 2023-05-31T21:25:15.185 DEBUG:boto:Token: None 2023-05-31T21:25:15.186 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 21:25:15 GMT /pklxkd-71/obj1 2023-05-31T21:25:15.186 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:K00XtdQP3hSsSgl41rQI/u46/hw= 2023-05-31T21:25:15.187 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 21:25:15 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:K00XtdQP3hSsSgl41rQI/u46/hw=', 'Content-Length': '0'} 2023-05-31T21:25:15.189 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 21:23:41 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000a81b9ea5ba2d45f6-006477bb3b-2568-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 21:25:15 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:25:15.190 DEBUG:boto:path=/pklxkd-71/obj2 2023-05-31T21:25:15.190 DEBUG:boto:auth_path=/pklxkd-71/obj2 2023-05-31T21:25:15.191 DEBUG:boto:Method: HEAD 2023-05-31T21:25:15.191 DEBUG:boto:Path: /pklxkd-71/obj2 2023-05-31T21:25:15.191 DEBUG:boto:Data: 2023-05-31T21:25:15.192 DEBUG:boto:Headers: {} 2023-05-31T21:25:15.192 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T21:25:15.193 DEBUG:boto:Port: 8001 2023-05-31T21:25:15.193 DEBUG:boto:Params: {} 2023-05-31T21:25:15.194 DEBUG:boto:Token: None 2023-05-31T21:25:15.194 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 21:25:15 GMT /pklxkd-71/obj2 2023-05-31T21:25:15.195 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:sN6S13IET+5xuOAKUJ4sc+vI6e8= 2023-05-31T21:25:15.195 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 21:25:15 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:sN6S13IET+5xuOAKUJ4sc+vI6e8=', 'Content-Length': '0'} 2023-05-31T21:25:15.197 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 21:23:41 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000c9d484b117584dad-006477bb3b-2568-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 21:25:15 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:25:15.197 DEBUG:boto:path=/pklxkd-71/obj2 2023-05-31T21:25:15.198 DEBUG:boto:auth_path=/pklxkd-71/obj2 2023-05-31T21:25:15.198 DEBUG:boto:Method: GET 2023-05-31T21:25:15.199 DEBUG:boto:Path: /pklxkd-71/obj2 2023-05-31T21:25:15.199 DEBUG:boto:Data: 2023-05-31T21:25:15.200 DEBUG:boto:Headers: {} 2023-05-31T21:25:15.200 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T21:25:15.200 DEBUG:boto:Port: 8001 2023-05-31T21:25:15.201 DEBUG:boto:Params: {} 2023-05-31T21:25:15.202 DEBUG:boto:Token: None 2023-05-31T21:25:15.202 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 21:25:15 GMT /pklxkd-71/obj2 2023-05-31T21:25:15.203 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:UCZ1Zm9KAzPWeKDzSRJcSxptrmg= 2023-05-31T21:25:15.204 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 21:25:15 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:UCZ1Zm9KAzPWeKDzSRJcSxptrmg=', 'Content-Length': '0'} 2023-05-31T21:25:15.207 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 21:23:41 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000e6db8b2a93b0b8b8-006477bb3b-2568-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 21:25:15 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:25:15.248 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'group', 'pipe', 'remove', '--group-id', 'sync-group', '--pipe-id', 'sync-pipe', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-05-31T21:25:15.249 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'group', 'pipe', 'remove', '--group-id', 'sync-group', '--pipe-id', 'sync-pipe', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-05-31T21:25:15.249 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync group pipe remove --group-id sync-group --pipe-id sync-pipe --cluster c1 --debug-rgw 0 --debug-ms 0 2023-05-31T21:25:15.300 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2023-05-31T21:25:15.300 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2023-05-31T21:25:15.369 INFO:teuthology.orchestra.run.smithi099.stdout:{"groups":[{"id":"sync-group","data_flow":{"symmetrical":[{"id":"sync-flow1","zones":["a1","a2"]}]},"pipes":[],"status":"enabled"}]} 2023-05-31T21:25:15.370 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'group', 'pipe', 'create', '--group-id', 'sync-group', '--pipe-id', 'sync-pipe', '--source-zones=a1,a2', '--dest-zones=a1,a2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-05-31T21:25:15.370 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'group', 'pipe', 'create', '--group-id', 'sync-group', '--pipe-id', 'sync-pipe', '--source-zones=a1,a2', '--dest-zones=a1,a2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-05-31T21:25:15.371 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync group pipe create --group-id sync-group --pipe-id sync-pipe --source-zones=a1,a2 --dest-zones=a1,a2 --cluster c1 --debug-rgw 0 --debug-ms 0 2023-05-31T21:25:15.424 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2023-05-31T21:25:15.424 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2023-05-31T21:25:15.492 INFO:teuthology.orchestra.run.smithi099.stdout:{"groups":[{"id":"sync-group","data_flow":{"symmetrical":[{"id":"sync-flow1","zones":["a1","a2"]}]},"pipes":[{"id":"sync-pipe","source":{"bucket":"*","zones":["a1","a2"]},"dest":{"bucket":"*","zones":["a1","a2"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":""}}],"status":"enabled"}]} 2023-05-31T21:25:15.492 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'group', 'modify', '--group-id', 'sync-group', '--status', 'allowed', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-05-31T21:25:15.493 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'group', 'modify', '--group-id', 'sync-group', '--status', 'allowed', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-05-31T21:25:15.493 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync group modify --group-id sync-group --status allowed --cluster c1 --debug-rgw 0 --debug-ms 0 2023-05-31T21:25:15.543 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2023-05-31T21:25:15.544 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2023-05-31T21:25:15.607 INFO:teuthology.orchestra.run.smithi099.stdout:{"groups":[{"id":"sync-group","data_flow":{"symmetrical":[{"id":"sync-flow1","zones":["a1","a2"]}]},"pipes":[{"id":"sync-pipe","source":{"bucket":"*","zones":["a1","a2"]},"dest":{"bucket":"*","zones":["a1","a2"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":""}}],"status":"allowed"}]} 2023-05-31T21:25:15.607 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['period', 'update', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-05-31T21:25:15.608 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'period', 'update', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-05-31T21:25:15.608 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 period update --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --commit --cluster c1 --debug-rgw 0 --debug-ms 0 2023-05-31T21:25:15.656 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2023-05-31T21:25:15.657 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2023-05-31T21:25:15.720 INFO:teuthology.orchestra.run.smithi099.stdout:{ 2023-05-31T21:25:15.720 INFO:teuthology.orchestra.run.smithi099.stdout: "id": "2e6123d2-d695-4350-9a01-61a455bc782b", 2023-05-31T21:25:15.720 INFO:teuthology.orchestra.run.smithi099.stdout: "epoch": 18, 2023-05-31T21:25:15.720 INFO:teuthology.orchestra.run.smithi099.stdout: "predecessor_uuid": "629b9ced-80af-41ed-8a6c-f2cfc334cbb4", 2023-05-31T21:25:15.721 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_status": [], 2023-05-31T21:25:15.721 INFO:teuthology.orchestra.run.smithi099.stdout: "period_map": { 2023-05-31T21:25:15.721 INFO:teuthology.orchestra.run.smithi099.stdout: "id": "2e6123d2-d695-4350-9a01-61a455bc782b", 2023-05-31T21:25:15.721 INFO:teuthology.orchestra.run.smithi099.stdout: "zonegroups": [ 2023-05-31T21:25:15.721 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T21:25:15.721 INFO:teuthology.orchestra.run.smithi099.stdout: "id": "a", 2023-05-31T21:25:15.722 INFO:teuthology.orchestra.run.smithi099.stdout: "name": "a", 2023-05-31T21:25:15.722 INFO:teuthology.orchestra.run.smithi099.stdout: "api_name": "", 2023-05-31T21:25:15.722 INFO:teuthology.orchestra.run.smithi099.stdout: "is_master": true, 2023-05-31T21:25:15.722 INFO:teuthology.orchestra.run.smithi099.stdout: "endpoints": [ 2023-05-31T21:25:15.722 INFO:teuthology.orchestra.run.smithi099.stdout: "http://smithi099.front.sepia.ceph.com:8000" 2023-05-31T21:25:15.722 INFO:teuthology.orchestra.run.smithi099.stdout: ], 2023-05-31T21:25:15.723 INFO:teuthology.orchestra.run.smithi099.stdout: "hostnames": [], 2023-05-31T21:25:15.723 INFO:teuthology.orchestra.run.smithi099.stdout: "hostnames_s3website": [], 2023-05-31T21:25:15.723 INFO:teuthology.orchestra.run.smithi099.stdout: "master_zone": "3d163b1e-09c7-47e2-ae87-add4aace3f38", 2023-05-31T21:25:15.723 INFO:teuthology.orchestra.run.smithi099.stdout: "zones": [ 2023-05-31T21:25:15.723 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T21:25:15.723 INFO:teuthology.orchestra.run.smithi099.stdout: "id": "3d163b1e-09c7-47e2-ae87-add4aace3f38", 2023-05-31T21:25:15.723 INFO:teuthology.orchestra.run.smithi099.stdout: "name": "a1", 2023-05-31T21:25:15.724 INFO:teuthology.orchestra.run.smithi099.stdout: "endpoints": [ 2023-05-31T21:25:15.724 INFO:teuthology.orchestra.run.smithi099.stdout: "http://smithi099.front.sepia.ceph.com:8000" 2023-05-31T21:25:15.724 INFO:teuthology.orchestra.run.smithi099.stdout: ], 2023-05-31T21:25:15.724 INFO:teuthology.orchestra.run.smithi099.stdout: "log_meta": false, 2023-05-31T21:25:15.724 INFO:teuthology.orchestra.run.smithi099.stdout: "log_data": true, 2023-05-31T21:25:15.724 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket_index_max_shards": 11, 2023-05-31T21:25:15.725 INFO:teuthology.orchestra.run.smithi099.stdout: "read_only": false, 2023-05-31T21:25:15.725 INFO:teuthology.orchestra.run.smithi099.stdout: "tier_type": "", 2023-05-31T21:25:15.725 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_from_all": true, 2023-05-31T21:25:15.725 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_from": [], 2023-05-31T21:25:15.725 INFO:teuthology.orchestra.run.smithi099.stdout: "redirect_zone": "", 2023-05-31T21:25:15.725 INFO:teuthology.orchestra.run.smithi099.stdout: "supported_features": [ 2023-05-31T21:25:15.725 INFO:teuthology.orchestra.run.smithi099.stdout: "resharding" 2023-05-31T21:25:15.726 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2023-05-31T21:25:15.726 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:25:15.726 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T21:25:15.726 INFO:teuthology.orchestra.run.smithi099.stdout: "id": "5d8acec7-7e57-4e56-8b32-e3d6b5036001", 2023-05-31T21:25:15.726 INFO:teuthology.orchestra.run.smithi099.stdout: "name": "a2", 2023-05-31T21:25:15.726 INFO:teuthology.orchestra.run.smithi099.stdout: "endpoints": [ 2023-05-31T21:25:15.726 INFO:teuthology.orchestra.run.smithi099.stdout: "http://smithi099.front.sepia.ceph.com:8001" 2023-05-31T21:25:15.727 INFO:teuthology.orchestra.run.smithi099.stdout: ], 2023-05-31T21:25:15.727 INFO:teuthology.orchestra.run.smithi099.stdout: "log_meta": false, 2023-05-31T21:25:15.727 INFO:teuthology.orchestra.run.smithi099.stdout: "log_data": true, 2023-05-31T21:25:15.727 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket_index_max_shards": 11, 2023-05-31T21:25:15.727 INFO:teuthology.orchestra.run.smithi099.stdout: "read_only": false, 2023-05-31T21:25:15.727 INFO:teuthology.orchestra.run.smithi099.stdout: "tier_type": "", 2023-05-31T21:25:15.728 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_from_all": true, 2023-05-31T21:25:15.728 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_from": [], 2023-05-31T21:25:15.728 INFO:teuthology.orchestra.run.smithi099.stdout: "redirect_zone": "", 2023-05-31T21:25:15.728 INFO:teuthology.orchestra.run.smithi099.stdout: "supported_features": [ 2023-05-31T21:25:15.728 INFO:teuthology.orchestra.run.smithi099.stdout: "resharding" 2023-05-31T21:25:15.729 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2023-05-31T21:25:15.729 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T21:25:15.729 INFO:teuthology.orchestra.run.smithi099.stdout: ], 2023-05-31T21:25:15.729 INFO:teuthology.orchestra.run.smithi099.stdout: "placement_targets": [ 2023-05-31T21:25:15.729 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T21:25:15.729 INFO:teuthology.orchestra.run.smithi099.stdout: "name": "default-placement", 2023-05-31T21:25:15.729 INFO:teuthology.orchestra.run.smithi099.stdout: "tags": [], 2023-05-31T21:25:15.730 INFO:teuthology.orchestra.run.smithi099.stdout: "storage_classes": [ 2023-05-31T21:25:15.730 INFO:teuthology.orchestra.run.smithi099.stdout: "STANDARD" 2023-05-31T21:25:15.730 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2023-05-31T21:25:15.730 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T21:25:15.730 INFO:teuthology.orchestra.run.smithi099.stdout: ], 2023-05-31T21:25:15.730 INFO:teuthology.orchestra.run.smithi099.stdout: "default_placement": "default-placement", 2023-05-31T21:25:15.730 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_id": "aaa33de6-515b-4cac-8f1b-3a1a30452117", 2023-05-31T21:25:15.731 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_policy": { 2023-05-31T21:25:15.731 INFO:teuthology.orchestra.run.smithi099.stdout: "groups": [ 2023-05-31T21:25:15.731 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T21:25:15.731 INFO:teuthology.orchestra.run.smithi099.stdout: "id": "sync-group", 2023-05-31T21:25:15.731 INFO:teuthology.orchestra.run.smithi099.stdout: "data_flow": { 2023-05-31T21:25:15.731 INFO:teuthology.orchestra.run.smithi099.stdout: "symmetrical": [ 2023-05-31T21:25:15.732 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T21:25:15.732 INFO:teuthology.orchestra.run.smithi099.stdout: "id": "sync-flow1", 2023-05-31T21:25:15.732 INFO:teuthology.orchestra.run.smithi099.stdout: "zones": [ 2023-05-31T21:25:15.732 INFO:teuthology.orchestra.run.smithi099.stdout: "3d163b1e-09c7-47e2-ae87-add4aace3f38", 2023-05-31T21:25:15.732 INFO:teuthology.orchestra.run.smithi099.stdout: "5d8acec7-7e57-4e56-8b32-e3d6b5036001" 2023-05-31T21:25:15.732 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2023-05-31T21:25:15.733 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T21:25:15.733 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2023-05-31T21:25:15.733 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:25:15.733 INFO:teuthology.orchestra.run.smithi099.stdout: "pipes": [ 2023-05-31T21:25:15.733 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T21:25:15.733 INFO:teuthology.orchestra.run.smithi099.stdout: "id": "sync-pipe", 2023-05-31T21:25:15.733 INFO:teuthology.orchestra.run.smithi099.stdout: "source": { 2023-05-31T21:25:15.734 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket": "*", 2023-05-31T21:25:15.734 INFO:teuthology.orchestra.run.smithi099.stdout: "zones": [ 2023-05-31T21:25:15.734 INFO:teuthology.orchestra.run.smithi099.stdout: "3d163b1e-09c7-47e2-ae87-add4aace3f38", 2023-05-31T21:25:15.734 INFO:teuthology.orchestra.run.smithi099.stdout: "5d8acec7-7e57-4e56-8b32-e3d6b5036001" 2023-05-31T21:25:15.734 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2023-05-31T21:25:15.734 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:25:15.735 INFO:teuthology.orchestra.run.smithi099.stdout: "dest": { 2023-05-31T21:25:15.735 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket": "*", 2023-05-31T21:25:15.735 INFO:teuthology.orchestra.run.smithi099.stdout: "zones": [ 2023-05-31T21:25:15.735 INFO:teuthology.orchestra.run.smithi099.stdout: "3d163b1e-09c7-47e2-ae87-add4aace3f38", 2023-05-31T21:25:15.735 INFO:teuthology.orchestra.run.smithi099.stdout: "5d8acec7-7e57-4e56-8b32-e3d6b5036001" 2023-05-31T21:25:15.735 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2023-05-31T21:25:15.736 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:25:15.736 INFO:teuthology.orchestra.run.smithi099.stdout: "params": { 2023-05-31T21:25:15.736 INFO:teuthology.orchestra.run.smithi099.stdout: "source": { 2023-05-31T21:25:15.736 INFO:teuthology.orchestra.run.smithi099.stdout: "filter": { 2023-05-31T21:25:15.736 INFO:teuthology.orchestra.run.smithi099.stdout: "tags": [] 2023-05-31T21:25:15.736 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T21:25:15.737 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:25:15.737 INFO:teuthology.orchestra.run.smithi099.stdout: "dest": {}, 2023-05-31T21:25:15.737 INFO:teuthology.orchestra.run.smithi099.stdout: "priority": 0, 2023-05-31T21:25:15.737 INFO:teuthology.orchestra.run.smithi099.stdout: "mode": "system", 2023-05-31T21:25:15.737 INFO:teuthology.orchestra.run.smithi099.stdout: "user": "" 2023-05-31T21:25:15.737 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T21:25:15.737 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T21:25:15.738 INFO:teuthology.orchestra.run.smithi099.stdout: ], 2023-05-31T21:25:15.738 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "allowed" 2023-05-31T21:25:15.738 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T21:25:15.738 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2023-05-31T21:25:15.738 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:25:15.738 INFO:teuthology.orchestra.run.smithi099.stdout: "enabled_features": [ 2023-05-31T21:25:15.739 INFO:teuthology.orchestra.run.smithi099.stdout: "resharding" 2023-05-31T21:25:15.739 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2023-05-31T21:25:15.739 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:25:15.739 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T21:25:15.739 INFO:teuthology.orchestra.run.smithi099.stdout: "id": "b", 2023-05-31T21:25:15.739 INFO:teuthology.orchestra.run.smithi099.stdout: "name": "b", 2023-05-31T21:25:15.739 INFO:teuthology.orchestra.run.smithi099.stdout: "api_name": "", 2023-05-31T21:25:15.740 INFO:teuthology.orchestra.run.smithi099.stdout: "is_master": false, 2023-05-31T21:25:15.740 INFO:teuthology.orchestra.run.smithi099.stdout: "endpoints": [ 2023-05-31T21:25:15.740 INFO:teuthology.orchestra.run.smithi099.stdout: "http://smithi112.front.sepia.ceph.com:8000" 2023-05-31T21:25:15.740 INFO:teuthology.orchestra.run.smithi099.stdout: ], 2023-05-31T21:25:15.740 INFO:teuthology.orchestra.run.smithi099.stdout: "hostnames": [], 2023-05-31T21:25:15.740 INFO:teuthology.orchestra.run.smithi099.stdout: "hostnames_s3website": [], 2023-05-31T21:25:15.741 INFO:teuthology.orchestra.run.smithi099.stdout: "master_zone": "f85d7236-2f27-4988-af26-9c66ab280b1b", 2023-05-31T21:25:15.741 INFO:teuthology.orchestra.run.smithi099.stdout: "zones": [ 2023-05-31T21:25:15.741 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T21:25:15.741 INFO:teuthology.orchestra.run.smithi099.stdout: "id": "a6a3aab5-bc52-44a4-8543-5c5ea6056acd", 2023-05-31T21:25:15.741 INFO:teuthology.orchestra.run.smithi099.stdout: "name": "b2", 2023-05-31T21:25:15.741 INFO:teuthology.orchestra.run.smithi099.stdout: "endpoints": [ 2023-05-31T21:25:15.741 INFO:teuthology.orchestra.run.smithi099.stdout: "http://smithi112.front.sepia.ceph.com:8001" 2023-05-31T21:25:15.742 INFO:teuthology.orchestra.run.smithi099.stdout: ], 2023-05-31T21:25:15.742 INFO:teuthology.orchestra.run.smithi099.stdout: "log_meta": false, 2023-05-31T21:25:15.742 INFO:teuthology.orchestra.run.smithi099.stdout: "log_data": true, 2023-05-31T21:25:15.742 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket_index_max_shards": 11, 2023-05-31T21:25:15.742 INFO:teuthology.orchestra.run.smithi099.stdout: "read_only": false, 2023-05-31T21:25:15.742 INFO:teuthology.orchestra.run.smithi099.stdout: "tier_type": "", 2023-05-31T21:25:15.742 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_from_all": true, 2023-05-31T21:25:15.743 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_from": [], 2023-05-31T21:25:15.743 INFO:teuthology.orchestra.run.smithi099.stdout: "redirect_zone": "", 2023-05-31T21:25:15.743 INFO:teuthology.orchestra.run.smithi099.stdout: "supported_features": [ 2023-05-31T21:25:15.743 INFO:teuthology.orchestra.run.smithi099.stdout: "resharding" 2023-05-31T21:25:15.743 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2023-05-31T21:25:15.743 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:25:15.743 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T21:25:15.744 INFO:teuthology.orchestra.run.smithi099.stdout: "id": "f85d7236-2f27-4988-af26-9c66ab280b1b", 2023-05-31T21:25:15.744 INFO:teuthology.orchestra.run.smithi099.stdout: "name": "b1", 2023-05-31T21:25:15.744 INFO:teuthology.orchestra.run.smithi099.stdout: "endpoints": [ 2023-05-31T21:25:15.744 INFO:teuthology.orchestra.run.smithi099.stdout: "http://smithi112.front.sepia.ceph.com:8000" 2023-05-31T21:25:15.744 INFO:teuthology.orchestra.run.smithi099.stdout: ], 2023-05-31T21:25:15.744 INFO:teuthology.orchestra.run.smithi099.stdout: "log_meta": false, 2023-05-31T21:25:15.744 INFO:teuthology.orchestra.run.smithi099.stdout: "log_data": true, 2023-05-31T21:25:15.745 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket_index_max_shards": 11, 2023-05-31T21:25:15.745 INFO:teuthology.orchestra.run.smithi099.stdout: "read_only": false, 2023-05-31T21:25:15.745 INFO:teuthology.orchestra.run.smithi099.stdout: "tier_type": "", 2023-05-31T21:25:15.745 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_from_all": true, 2023-05-31T21:25:15.745 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_from": [], 2023-05-31T21:25:15.745 INFO:teuthology.orchestra.run.smithi099.stdout: "redirect_zone": "", 2023-05-31T21:25:15.745 INFO:teuthology.orchestra.run.smithi099.stdout: "supported_features": [ 2023-05-31T21:25:15.746 INFO:teuthology.orchestra.run.smithi099.stdout: "resharding" 2023-05-31T21:25:15.746 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2023-05-31T21:25:15.746 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T21:25:15.746 INFO:teuthology.orchestra.run.smithi099.stdout: ], 2023-05-31T21:25:15.746 INFO:teuthology.orchestra.run.smithi099.stdout: "placement_targets": [ 2023-05-31T21:25:15.746 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T21:25:15.747 INFO:teuthology.orchestra.run.smithi099.stdout: "name": "default-placement", 2023-05-31T21:25:15.747 INFO:teuthology.orchestra.run.smithi099.stdout: "tags": [], 2023-05-31T21:25:15.747 INFO:teuthology.orchestra.run.smithi099.stdout: "storage_classes": [ 2023-05-31T21:25:15.747 INFO:teuthology.orchestra.run.smithi099.stdout: "STANDARD" 2023-05-31T21:25:15.747 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2023-05-31T21:25:15.747 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T21:25:15.748 INFO:teuthology.orchestra.run.smithi099.stdout: ], 2023-05-31T21:25:15.748 INFO:teuthology.orchestra.run.smithi099.stdout: "default_placement": "default-placement", 2023-05-31T21:25:15.748 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_id": "aaa33de6-515b-4cac-8f1b-3a1a30452117", 2023-05-31T21:25:15.748 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_policy": { 2023-05-31T21:25:15.748 INFO:teuthology.orchestra.run.smithi099.stdout: "groups": [] 2023-05-31T21:25:15.748 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:25:15.748 INFO:teuthology.orchestra.run.smithi099.stdout: "enabled_features": [ 2023-05-31T21:25:15.749 INFO:teuthology.orchestra.run.smithi099.stdout: "resharding" 2023-05-31T21:25:15.749 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2023-05-31T21:25:15.749 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T21:25:15.749 INFO:teuthology.orchestra.run.smithi099.stdout: ], 2023-05-31T21:25:15.749 INFO:teuthology.orchestra.run.smithi099.stdout: "short_zone_ids": [ 2023-05-31T21:25:15.749 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T21:25:15.749 INFO:teuthology.orchestra.run.smithi099.stdout: "key": "3d163b1e-09c7-47e2-ae87-add4aace3f38", 2023-05-31T21:25:15.750 INFO:teuthology.orchestra.run.smithi099.stdout: "val": 3101833771 2023-05-31T21:25:15.750 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:25:15.750 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T21:25:15.750 INFO:teuthology.orchestra.run.smithi099.stdout: "key": "5d8acec7-7e57-4e56-8b32-e3d6b5036001", 2023-05-31T21:25:15.750 INFO:teuthology.orchestra.run.smithi099.stdout: "val": 4255530623 2023-05-31T21:25:15.750 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:25:15.750 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T21:25:15.751 INFO:teuthology.orchestra.run.smithi099.stdout: "key": "a6a3aab5-bc52-44a4-8543-5c5ea6056acd", 2023-05-31T21:25:15.751 INFO:teuthology.orchestra.run.smithi099.stdout: "val": 3708073710 2023-05-31T21:25:15.751 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:25:15.751 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T21:25:15.752 INFO:teuthology.orchestra.run.smithi099.stdout: "key": "f85d7236-2f27-4988-af26-9c66ab280b1b", 2023-05-31T21:25:15.752 INFO:teuthology.orchestra.run.smithi099.stdout: "val": 233395200 2023-05-31T21:25:15.753 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T21:25:15.753 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2023-05-31T21:25:15.753 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:25:15.753 INFO:teuthology.orchestra.run.smithi099.stdout: "master_zonegroup": "a", 2023-05-31T21:25:15.753 INFO:teuthology.orchestra.run.smithi099.stdout: "master_zone": "3d163b1e-09c7-47e2-ae87-add4aace3f38", 2023-05-31T21:25:15.753 INFO:teuthology.orchestra.run.smithi099.stdout: "period_config": { 2023-05-31T21:25:15.753 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket_quota": { 2023-05-31T21:25:15.754 INFO:teuthology.orchestra.run.smithi099.stdout: "enabled": false, 2023-05-31T21:25:15.754 INFO:teuthology.orchestra.run.smithi099.stdout: "check_on_raw": false, 2023-05-31T21:25:15.754 INFO:teuthology.orchestra.run.smithi099.stdout: "max_size": -1, 2023-05-31T21:25:15.754 INFO:teuthology.orchestra.run.smithi099.stdout: "max_size_kb": 0, 2023-05-31T21:25:15.754 INFO:teuthology.orchestra.run.smithi099.stdout: "max_objects": -1 2023-05-31T21:25:15.754 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:25:15.754 INFO:teuthology.orchestra.run.smithi099.stdout: "user_quota": { 2023-05-31T21:25:15.755 INFO:teuthology.orchestra.run.smithi099.stdout: "enabled": false, 2023-05-31T21:25:15.755 INFO:teuthology.orchestra.run.smithi099.stdout: "check_on_raw": false, 2023-05-31T21:25:15.755 INFO:teuthology.orchestra.run.smithi099.stdout: "max_size": -1, 2023-05-31T21:25:15.755 INFO:teuthology.orchestra.run.smithi099.stdout: "max_size_kb": 0, 2023-05-31T21:25:15.755 INFO:teuthology.orchestra.run.smithi099.stdout: "max_objects": -1 2023-05-31T21:25:15.755 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:25:15.755 INFO:teuthology.orchestra.run.smithi099.stdout: "user_ratelimit": { 2023-05-31T21:25:15.756 INFO:teuthology.orchestra.run.smithi099.stdout: "max_read_ops": 0, 2023-05-31T21:25:15.756 INFO:teuthology.orchestra.run.smithi099.stdout: "max_write_ops": 0, 2023-05-31T21:25:15.756 INFO:teuthology.orchestra.run.smithi099.stdout: "max_read_bytes": 0, 2023-05-31T21:25:15.756 INFO:teuthology.orchestra.run.smithi099.stdout: "max_write_bytes": 0, 2023-05-31T21:25:15.756 INFO:teuthology.orchestra.run.smithi099.stdout: "enabled": false 2023-05-31T21:25:15.757 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:25:15.757 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket_ratelimit": { 2023-05-31T21:25:15.757 INFO:teuthology.orchestra.run.smithi099.stdout: "max_read_ops": 0, 2023-05-31T21:25:15.757 INFO:teuthology.orchestra.run.smithi099.stdout: "max_write_ops": 0, 2023-05-31T21:25:15.757 INFO:teuthology.orchestra.run.smithi099.stdout: "max_read_bytes": 0, 2023-05-31T21:25:15.757 INFO:teuthology.orchestra.run.smithi099.stdout: "max_write_bytes": 0, 2023-05-31T21:25:15.757 INFO:teuthology.orchestra.run.smithi099.stdout: "enabled": false 2023-05-31T21:25:15.758 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:25:15.758 INFO:teuthology.orchestra.run.smithi099.stdout: "anonymous_ratelimit": { 2023-05-31T21:25:15.758 INFO:teuthology.orchestra.run.smithi099.stdout: "max_read_ops": 0, 2023-05-31T21:25:15.758 INFO:teuthology.orchestra.run.smithi099.stdout: "max_write_ops": 0, 2023-05-31T21:25:15.758 INFO:teuthology.orchestra.run.smithi099.stdout: "max_read_bytes": 0, 2023-05-31T21:25:15.758 INFO:teuthology.orchestra.run.smithi099.stdout: "max_write_bytes": 0, 2023-05-31T21:25:15.758 INFO:teuthology.orchestra.run.smithi099.stdout: "enabled": false 2023-05-31T21:25:15.759 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T21:25:15.759 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:25:15.759 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_id": "aaa33de6-515b-4cac-8f1b-3a1a30452117", 2023-05-31T21:25:15.759 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_name": "test-realm", 2023-05-31T21:25:15.759 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2023-05-31T21:25:15.759 INFO:teuthology.orchestra.run.smithi099.stdout:} 2023-05-31T21:25:15.760 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'policy', 'get', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-05-31T21:25:15.760 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'policy', 'get', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-05-31T21:25:15.761 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync policy get --cluster c1 --debug-rgw 0 --debug-ms 0 2023-05-31T21:25:15.807 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2023-05-31T21:25:15.807 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2023-05-31T21:25:15.884 INFO:teuthology.orchestra.run.smithi099.stdout:{"groups":[{"id":"sync-group","data_flow":{"symmetrical":[{"id":"sync-flow1","zones":["a1","a2"]}]},"pipes":[{"id":"sync-pipe","source":{"bucket":"*","zones":["a1","a2"]},"dest":{"bucket":"*","zones":["a1","a2"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":""}}],"status":"allowed"}]} 2023-05-31T21:25:15.885 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'group', 'create', '--group-id', 'sync-bucket', '--status', 'allowed', '--bucket', 'pklxkd-70', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-05-31T21:25:15.885 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'group', 'create', '--group-id', 'sync-bucket', '--status', 'allowed', '--bucket', 'pklxkd-70', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-05-31T21:25:15.886 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync group create --group-id sync-bucket --status allowed --bucket pklxkd-70 --cluster c1 --debug-rgw 0 --debug-ms 0 2023-05-31T21:25:15.945 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2023-05-31T21:25:15.946 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2023-05-31T21:25:16.087 INFO:teuthology.orchestra.run.smithi099.stdout:{"groups":[{"id":"sync-bucket","data_flow":{},"pipes":[],"status":"allowed"}]} 2023-05-31T21:25:16.088 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'group', 'flow', 'create', '--group-id', 'sync-bucket', '--flow-id', 'sync-flowA', '--flow-type', 'symmetrical', '--zones=a1,a2', '--bucket', 'pklxkd-70', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-05-31T21:25:16.089 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'group', 'flow', 'create', '--group-id', 'sync-bucket', '--flow-id', 'sync-flowA', '--flow-type', 'symmetrical', '--zones=a1,a2', '--bucket', 'pklxkd-70', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-05-31T21:25:16.089 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync group flow create --group-id sync-bucket --flow-id sync-flowA --flow-type symmetrical --zones=a1,a2 --bucket pklxkd-70 --cluster c1 --debug-rgw 0 --debug-ms 0 2023-05-31T21:25:16.162 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2023-05-31T21:25:16.163 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2023-05-31T21:25:16.291 INFO:teuthology.orchestra.run.smithi099.stdout:{"groups":[{"id":"sync-bucket","data_flow":{"symmetrical":[{"id":"sync-flowA","zones":["a1","a2"]}]},"pipes":[],"status":"allowed"}]} 2023-05-31T21:25:16.291 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'group', 'pipe', 'create', '--group-id', 'sync-bucket', '--pipe-id', 'sync-pipeA', '--source-zones=a1,a2', '--dest-zones=a1,a2', '--bucket=pklxkd-70', '--source-bucket=*', '--dest-bucket=pklxkd-71', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-05-31T21:25:16.292 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'group', 'pipe', 'create', '--group-id', 'sync-bucket', '--pipe-id', 'sync-pipeA', '--source-zones=a1,a2', '--dest-zones=a1,a2', '--bucket=pklxkd-70', '--source-bucket=*', '--dest-bucket=pklxkd-71', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-05-31T21:25:16.292 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync group pipe create --group-id sync-bucket --pipe-id sync-pipeA --source-zones=a1,a2 --dest-zones=a1,a2 --bucket=pklxkd-70 '--source-bucket=*' --dest-bucket=pklxkd-71 --cluster c1 --debug-rgw 0 --debug-ms 0 2023-05-31T21:25:16.355 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2023-05-31T21:25:16.355 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2023-05-31T21:25:16.505 INFO:teuthology.orchestra.run.smithi099.stdout:{"groups":[{"id":"sync-bucket","data_flow":{"symmetrical":[{"id":"sync-flowA","zones":["a1","a2"]}]},"pipes":[{"id":"sync-pipeA","source":{"bucket":"*","zones":["a1","a2"]},"dest":{"bucket":"pklxkd-71","zones":["a1","a2"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":"rgw-multisite-test-user"}}],"status":"allowed"}]} 2023-05-31T21:25:16.506 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'group', 'modify', '--group-id', 'sync-bucket', '--status', 'enabled', '--bucket', 'pklxkd-70', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-05-31T21:25:16.506 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'group', 'modify', '--group-id', 'sync-bucket', '--status', 'enabled', '--bucket', 'pklxkd-70', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-05-31T21:25:16.507 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync group modify --group-id sync-bucket --status enabled --bucket pklxkd-70 --cluster c1 --debug-rgw 0 --debug-ms 0 2023-05-31T21:25:16.561 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2023-05-31T21:25:16.561 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2023-05-31T21:25:16.669 INFO:teuthology.orchestra.run.smithi099.stdout:{"groups":[{"id":"sync-bucket","data_flow":{"symmetrical":[{"id":"sync-flowA","zones":["a1","a2"]}]},"pipes":[{"id":"sync-pipeA","source":{"bucket":"*","zones":["a1","a2"]},"dest":{"bucket":"pklxkd-71","zones":["a1","a2"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":"rgw-multisite-test-user"}}],"status":"enabled"}]} 2023-05-31T21:25:16.669 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'policy', 'get', '--bucket', 'pklxkd-70', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-05-31T21:25:16.669 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'policy', 'get', '--bucket', 'pklxkd-70', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-05-31T21:25:16.670 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync policy get --bucket pklxkd-70 --cluster c1 --debug-rgw 0 --debug-ms 0 2023-05-31T21:25:16.719 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2023-05-31T21:25:16.720 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2023-05-31T21:25:16.813 INFO:teuthology.orchestra.run.smithi099.stdout:{"groups":[{"id":"sync-bucket","data_flow":{"symmetrical":[{"id":"sync-flowA","zones":["a1","a2"]}]},"pipes":[{"id":"sync-pipeA","source":{"bucket":"*","zones":["a1","a2"]},"dest":{"bucket":"pklxkd-71","zones":["a1","a2"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":"rgw-multisite-test-user"}}],"status":"enabled"}]} 2023-05-31T21:25:16.814 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-31T21:25:16.815 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-31T21:25:16.815 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-05-31T21:25:16.867 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2023-05-31T21:25:16.867 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2023-05-31T21:25:16.922 INFO:teuthology.orchestra.run.smithi099.stderr:No --period given, using current period=2e6123d2-d695-4350-9a01-61a455bc782b 2023-05-31T21:25:16.965 INFO:teuthology.orchestra.run.smithi099.stdout:[ 2023-05-31T21:25:16.966 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T21:25:16.966 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1685566501.757183_260.1", 2023-05-31T21:25:16.967 INFO:teuthology.orchestra.run.smithi099.stdout: "last_update": "2023-05-31T20:55:01.757183Z" 2023-05-31T21:25:16.967 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:25:16.967 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T21:25:16.967 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1685566501.735717_251.1", 2023-05-31T21:25:16.968 INFO:teuthology.orchestra.run.smithi099.stdout: "last_update": "2023-05-31T20:55:01.735717Z" 2023-05-31T21:25:16.968 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:25:16.968 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T21:25:16.968 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1685562970.274572_93.1", 2023-05-31T21:25:16.969 INFO:teuthology.orchestra.run.smithi099.stdout: "last_update": "2023-05-31T19:56:10.274572Z" 2023-05-31T21:25:16.969 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:25:16.969 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T21:25:16.969 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1685568316.644293_992.1", 2023-05-31T21:25:16.970 INFO:teuthology.orchestra.run.smithi099.stdout: "last_update": "2023-05-31T21:25:16.644293Z" 2023-05-31T21:25:16.970 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T21:25:16.970 INFO:teuthology.orchestra.run.smithi099.stdout:] 2023-05-31T21:25:16.972 DEBUG:rgw_multi.tests:master meta markers={0: '1_1685566501.757183_260.1', 1: '1_1685566501.735717_251.1', 2: '1_1685562970.274572_93.1', 3: '1_1685568316.644293_992.1'} 2023-05-31T21:25:16.973 INFO:rgw_multi.tests:starting meta checkpoint for zone=a2 2023-05-31T21:25:16.973 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-31T21:25:16.974 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-31T21:25:16.975 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-05-31T21:25:17.030 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2023-05-31T21:25:17.031 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2023-05-31T21:25:17.078 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T21:25:17.073+0000 7f08eac0b780 20 RGW-SYNC:meta: read sync status 2023-05-31T21:25:17.079 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T21:25:17.074+0000 7f08eac0b780 20 RGW-SYNC:meta: read sync status 2023-05-31T21:25:17.107 INFO:teuthology.orchestra.run.smithi099.stdout:{ 2023-05-31T21:25:17.108 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_status": { 2023-05-31T21:25:17.108 INFO:teuthology.orchestra.run.smithi099.stdout: "info": { 2023-05-31T21:25:17.108 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "sync", 2023-05-31T21:25:17.108 INFO:teuthology.orchestra.run.smithi099.stdout: "num_shards": 4, 2023-05-31T21:25:17.108 INFO:teuthology.orchestra.run.smithi099.stdout: "period": "2e6123d2-d695-4350-9a01-61a455bc782b", 2023-05-31T21:25:17.108 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2023-05-31T21:25:17.109 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:25:17.109 INFO:teuthology.orchestra.run.smithi099.stdout: "markers": [ 2023-05-31T21:25:17.109 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T21:25:17.109 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 0, 2023-05-31T21:25:17.109 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T21:25:17.109 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2023-05-31T21:25:17.109 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1685566501.757183_260.1", 2023-05-31T21:25:17.110 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T21:25:17.110 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 1, 2023-05-31T21:25:17.110 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T21:25:17.110 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2023-05-31T20:55:01.757183Z", 2023-05-31T21:25:17.110 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2023-05-31T21:25:17.110 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T21:25:17.110 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:25:17.111 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T21:25:17.111 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 1, 2023-05-31T21:25:17.111 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T21:25:17.111 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2023-05-31T21:25:17.112 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1685566501.735717_251.1", 2023-05-31T21:25:17.112 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T21:25:17.112 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2023-05-31T21:25:17.112 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T21:25:17.112 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2023-05-31T20:55:01.735717Z", 2023-05-31T21:25:17.112 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2023-05-31T21:25:17.113 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T21:25:17.113 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:25:17.113 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T21:25:17.113 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 2, 2023-05-31T21:25:17.113 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T21:25:17.113 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2023-05-31T21:25:17.113 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1685562970.274572_93.1", 2023-05-31T21:25:17.114 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T21:25:17.114 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2023-05-31T21:25:17.114 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T21:25:17.114 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2023-05-31T19:56:10.274572Z", 2023-05-31T21:25:17.114 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2023-05-31T21:25:17.114 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T21:25:17.114 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:25:17.115 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T21:25:17.115 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 3, 2023-05-31T21:25:17.115 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T21:25:17.115 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2023-05-31T21:25:17.115 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1685568189.652614_974.1", 2023-05-31T21:25:17.115 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T21:25:17.115 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2023-05-31T21:25:17.116 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T21:25:17.116 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2023-05-31T21:23:09.652614Z", 2023-05-31T21:25:17.116 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2023-05-31T21:25:17.116 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T21:25:17.116 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T21:25:17.117 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2023-05-31T21:25:17.117 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:25:17.118 INFO:teuthology.orchestra.run.smithi099.stdout: "full_sync": { 2023-05-31T21:25:17.118 INFO:teuthology.orchestra.run.smithi099.stdout: "total": 1, 2023-05-31T21:25:17.118 INFO:teuthology.orchestra.run.smithi099.stdout: "complete": 1 2023-05-31T21:25:17.118 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:25:17.118 INFO:teuthology.orchestra.run.smithi099.stdout: "current_time": "2023-05-31T21:25:17Z" 2023-05-31T21:25:17.118 INFO:teuthology.orchestra.run.smithi099.stdout:} 2023-05-31T21:25:17.119 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "2e6123d2-d695-4350-9a01-61a455bc782b", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1685566501.757183_260.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-05-31T20:55:01.757183Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1685566501.735717_251.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-05-31T20:55:01.735717Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1685562970.274572_93.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-05-31T19:56:10.274572Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1685568189.652614_974.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-05-31T21:23:09.652614Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-05-31T21:25:17Z" } 2023-05-31T21:25:17.120 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1685566501.757183_260.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-05-31T20:55:01.757183Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1685566501.735717_251.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-05-31T20:55:01.735717Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1685562970.274572_93.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-05-31T19:56:10.274572Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1685568189.652614_974.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-05-31T21:23:09.652614Z', 'realm_epoch': 2}}] 2023-05-31T21:25:17.120 DEBUG:rgw_multi.tests:log_status={0: '1_1685566501.757183_260.1', 1: '1_1685566501.735717_251.1', 2: '1_1685562970.274572_93.1', 3: '1_1685568316.644293_992.1'} 2023-05-31T21:25:17.121 DEBUG:rgw_multi.tests:sync_status={0: '1_1685566501.757183_260.1', 1: '1_1685566501.735717_251.1', 2: '1_1685562970.274572_93.1', 3: '1_1685568189.652614_974.1'} 2023-05-31T21:25:17.121 WARNING:rgw_multi.tests:zone a2 behind master: shard=3 master=1_1685568316.644293_992.1 target=1_1685568189.652614_974.1 2023-05-31T21:25:22.123 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-31T21:25:22.123 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-31T21:25:22.124 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-05-31T21:25:22.182 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2023-05-31T21:25:22.183 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2023-05-31T21:25:22.236 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T21:25:22.231+0000 7fe873d97780 20 RGW-SYNC:meta: read sync status 2023-05-31T21:25:22.236 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T21:25:22.232+0000 7fe873d97780 20 RGW-SYNC:meta: read sync status 2023-05-31T21:25:22.257 INFO:teuthology.orchestra.run.smithi099.stdout:{ 2023-05-31T21:25:22.258 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_status": { 2023-05-31T21:25:22.258 INFO:teuthology.orchestra.run.smithi099.stdout: "info": { 2023-05-31T21:25:22.258 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "sync", 2023-05-31T21:25:22.258 INFO:teuthology.orchestra.run.smithi099.stdout: "num_shards": 4, 2023-05-31T21:25:22.258 INFO:teuthology.orchestra.run.smithi099.stdout: "period": "2e6123d2-d695-4350-9a01-61a455bc782b", 2023-05-31T21:25:22.258 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2023-05-31T21:25:22.259 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:25:22.259 INFO:teuthology.orchestra.run.smithi099.stdout: "markers": [ 2023-05-31T21:25:22.259 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T21:25:22.259 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 0, 2023-05-31T21:25:22.259 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T21:25:22.259 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2023-05-31T21:25:22.259 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1685566501.757183_260.1", 2023-05-31T21:25:22.260 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T21:25:22.260 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 1, 2023-05-31T21:25:22.260 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T21:25:22.260 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2023-05-31T20:55:01.757183Z", 2023-05-31T21:25:22.260 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2023-05-31T21:25:22.260 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T21:25:22.261 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:25:22.261 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T21:25:22.261 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 1, 2023-05-31T21:25:22.261 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T21:25:22.261 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2023-05-31T21:25:22.261 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1685566501.735717_251.1", 2023-05-31T21:25:22.262 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T21:25:22.262 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2023-05-31T21:25:22.262 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T21:25:22.262 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2023-05-31T20:55:01.735717Z", 2023-05-31T21:25:22.262 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2023-05-31T21:25:22.262 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T21:25:22.262 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:25:22.263 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T21:25:22.263 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 2, 2023-05-31T21:25:22.263 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T21:25:22.264 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2023-05-31T21:25:22.264 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1685562970.274572_93.1", 2023-05-31T21:25:22.264 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T21:25:22.264 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2023-05-31T21:25:22.264 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T21:25:22.264 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2023-05-31T19:56:10.274572Z", 2023-05-31T21:25:22.264 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2023-05-31T21:25:22.265 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T21:25:22.265 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:25:22.265 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T21:25:22.265 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 3, 2023-05-31T21:25:22.265 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T21:25:22.265 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2023-05-31T21:25:22.266 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1685568189.652614_974.1", 2023-05-31T21:25:22.266 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T21:25:22.266 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2023-05-31T21:25:22.266 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T21:25:22.266 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2023-05-31T21:23:09.652614Z", 2023-05-31T21:25:22.266 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2023-05-31T21:25:22.266 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T21:25:22.267 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T21:25:22.267 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2023-05-31T21:25:22.267 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:25:22.267 INFO:teuthology.orchestra.run.smithi099.stdout: "full_sync": { 2023-05-31T21:25:22.267 INFO:teuthology.orchestra.run.smithi099.stdout: "total": 1, 2023-05-31T21:25:22.267 INFO:teuthology.orchestra.run.smithi099.stdout: "complete": 1 2023-05-31T21:25:22.267 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:25:22.268 INFO:teuthology.orchestra.run.smithi099.stdout: "current_time": "2023-05-31T21:25:22Z" 2023-05-31T21:25:22.268 INFO:teuthology.orchestra.run.smithi099.stdout:} 2023-05-31T21:25:22.269 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "2e6123d2-d695-4350-9a01-61a455bc782b", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1685566501.757183_260.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-05-31T20:55:01.757183Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1685566501.735717_251.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-05-31T20:55:01.735717Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1685562970.274572_93.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-05-31T19:56:10.274572Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1685568189.652614_974.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-05-31T21:23:09.652614Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-05-31T21:25:22Z" } 2023-05-31T21:25:22.269 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1685566501.757183_260.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-05-31T20:55:01.757183Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1685566501.735717_251.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-05-31T20:55:01.735717Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1685562970.274572_93.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-05-31T19:56:10.274572Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1685568189.652614_974.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-05-31T21:23:09.652614Z', 'realm_epoch': 2}}] 2023-05-31T21:25:22.270 DEBUG:rgw_multi.tests:log_status={0: '1_1685566501.757183_260.1', 1: '1_1685566501.735717_251.1', 2: '1_1685562970.274572_93.1', 3: '1_1685568316.644293_992.1'} 2023-05-31T21:25:22.270 DEBUG:rgw_multi.tests:sync_status={0: '1_1685566501.757183_260.1', 1: '1_1685566501.735717_251.1', 2: '1_1685562970.274572_93.1', 3: '1_1685568189.652614_974.1'} 2023-05-31T21:25:22.270 WARNING:rgw_multi.tests:zone a2 behind master: shard=3 master=1_1685568316.644293_992.1 target=1_1685568189.652614_974.1 2023-05-31T21:25:27.271 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-31T21:25:27.272 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-31T21:25:27.273 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-05-31T21:25:27.330 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2023-05-31T21:25:27.331 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2023-05-31T21:25:27.371 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T21:25:27.368+0000 7f581ab6c780 20 RGW-SYNC:meta: read sync status 2023-05-31T21:25:27.373 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T21:25:27.369+0000 7f581ab6c780 20 RGW-SYNC:meta: read sync status 2023-05-31T21:25:27.394 INFO:teuthology.orchestra.run.smithi099.stdout:{ 2023-05-31T21:25:27.394 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_status": { 2023-05-31T21:25:27.394 INFO:teuthology.orchestra.run.smithi099.stdout: "info": { 2023-05-31T21:25:27.394 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "sync", 2023-05-31T21:25:27.395 INFO:teuthology.orchestra.run.smithi099.stdout: "num_shards": 4, 2023-05-31T21:25:27.395 INFO:teuthology.orchestra.run.smithi099.stdout: "period": "2e6123d2-d695-4350-9a01-61a455bc782b", 2023-05-31T21:25:27.395 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2023-05-31T21:25:27.395 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:25:27.395 INFO:teuthology.orchestra.run.smithi099.stdout: "markers": [ 2023-05-31T21:25:27.395 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T21:25:27.395 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 0, 2023-05-31T21:25:27.396 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T21:25:27.396 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2023-05-31T21:25:27.396 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1685566501.757183_260.1", 2023-05-31T21:25:27.396 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T21:25:27.396 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 1, 2023-05-31T21:25:27.396 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T21:25:27.397 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2023-05-31T20:55:01.757183Z", 2023-05-31T21:25:27.397 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2023-05-31T21:25:27.397 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T21:25:27.398 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:25:27.398 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T21:25:27.398 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 1, 2023-05-31T21:25:27.398 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T21:25:27.398 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2023-05-31T21:25:27.398 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1685566501.735717_251.1", 2023-05-31T21:25:27.398 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T21:25:27.399 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2023-05-31T21:25:27.399 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T21:25:27.399 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2023-05-31T20:55:01.735717Z", 2023-05-31T21:25:27.399 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2023-05-31T21:25:27.399 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T21:25:27.399 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:25:27.399 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T21:25:27.400 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 2, 2023-05-31T21:25:27.400 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T21:25:27.400 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2023-05-31T21:25:27.400 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1685562970.274572_93.1", 2023-05-31T21:25:27.400 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T21:25:27.400 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2023-05-31T21:25:27.401 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T21:25:27.401 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2023-05-31T19:56:10.274572Z", 2023-05-31T21:25:27.401 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2023-05-31T21:25:27.401 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T21:25:27.401 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:25:27.401 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T21:25:27.401 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 3, 2023-05-31T21:25:27.402 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T21:25:27.402 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2023-05-31T21:25:27.402 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1685568189.652614_974.1", 2023-05-31T21:25:27.402 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T21:25:27.402 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2023-05-31T21:25:27.402 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T21:25:27.402 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2023-05-31T21:23:09.652614Z", 2023-05-31T21:25:27.403 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2023-05-31T21:25:27.403 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T21:25:27.403 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T21:25:27.403 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2023-05-31T21:25:27.403 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:25:27.403 INFO:teuthology.orchestra.run.smithi099.stdout: "full_sync": { 2023-05-31T21:25:27.403 INFO:teuthology.orchestra.run.smithi099.stdout: "total": 1, 2023-05-31T21:25:27.404 INFO:teuthology.orchestra.run.smithi099.stdout: "complete": 1 2023-05-31T21:25:27.404 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:25:27.404 INFO:teuthology.orchestra.run.smithi099.stdout: "current_time": "2023-05-31T21:25:27Z" 2023-05-31T21:25:27.404 INFO:teuthology.orchestra.run.smithi099.stdout:} 2023-05-31T21:25:27.406 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "2e6123d2-d695-4350-9a01-61a455bc782b", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1685566501.757183_260.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-05-31T20:55:01.757183Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1685566501.735717_251.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-05-31T20:55:01.735717Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1685562970.274572_93.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-05-31T19:56:10.274572Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1685568189.652614_974.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-05-31T21:23:09.652614Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-05-31T21:25:27Z" } 2023-05-31T21:25:27.408 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1685566501.757183_260.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-05-31T20:55:01.757183Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1685566501.735717_251.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-05-31T20:55:01.735717Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1685562970.274572_93.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-05-31T19:56:10.274572Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1685568189.652614_974.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-05-31T21:23:09.652614Z', 'realm_epoch': 2}}] 2023-05-31T21:25:27.409 DEBUG:rgw_multi.tests:log_status={0: '1_1685566501.757183_260.1', 1: '1_1685566501.735717_251.1', 2: '1_1685562970.274572_93.1', 3: '1_1685568316.644293_992.1'} 2023-05-31T21:25:27.410 DEBUG:rgw_multi.tests:sync_status={0: '1_1685566501.757183_260.1', 1: '1_1685566501.735717_251.1', 2: '1_1685562970.274572_93.1', 3: '1_1685568189.652614_974.1'} 2023-05-31T21:25:27.410 WARNING:rgw_multi.tests:zone a2 behind master: shard=3 master=1_1685568316.644293_992.1 target=1_1685568189.652614_974.1 2023-05-31T21:25:32.411 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-31T21:25:32.412 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-31T21:25:32.413 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-05-31T21:25:32.470 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2023-05-31T21:25:32.470 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2023-05-31T21:25:32.517 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T21:25:32.514+0000 7f7aa46d7780 20 RGW-SYNC:meta: read sync status 2023-05-31T21:25:32.518 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T21:25:32.515+0000 7f7aa46d7780 20 RGW-SYNC:meta: read sync status 2023-05-31T21:25:32.532 INFO:teuthology.orchestra.run.smithi099.stdout:{ 2023-05-31T21:25:32.533 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_status": { 2023-05-31T21:25:32.533 INFO:teuthology.orchestra.run.smithi099.stdout: "info": { 2023-05-31T21:25:32.533 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "sync", 2023-05-31T21:25:32.533 INFO:teuthology.orchestra.run.smithi099.stdout: "num_shards": 4, 2023-05-31T21:25:32.533 INFO:teuthology.orchestra.run.smithi099.stdout: "period": "2e6123d2-d695-4350-9a01-61a455bc782b", 2023-05-31T21:25:32.533 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2023-05-31T21:25:32.534 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:25:32.534 INFO:teuthology.orchestra.run.smithi099.stdout: "markers": [ 2023-05-31T21:25:32.534 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T21:25:32.534 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 0, 2023-05-31T21:25:32.534 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T21:25:32.534 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2023-05-31T21:25:32.535 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1685566501.757183_260.1", 2023-05-31T21:25:32.535 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T21:25:32.535 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 1, 2023-05-31T21:25:32.535 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T21:25:32.535 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2023-05-31T20:55:01.757183Z", 2023-05-31T21:25:32.535 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2023-05-31T21:25:32.535 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T21:25:32.536 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:25:32.536 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T21:25:32.536 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 1, 2023-05-31T21:25:32.536 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T21:25:32.536 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2023-05-31T21:25:32.536 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1685566501.735717_251.1", 2023-05-31T21:25:32.536 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T21:25:32.537 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2023-05-31T21:25:32.537 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T21:25:32.537 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2023-05-31T20:55:01.735717Z", 2023-05-31T21:25:32.537 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2023-05-31T21:25:32.537 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T21:25:32.537 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:25:32.537 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T21:25:32.538 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 2, 2023-05-31T21:25:32.538 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T21:25:32.538 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2023-05-31T21:25:32.538 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1685562970.274572_93.1", 2023-05-31T21:25:32.538 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T21:25:32.538 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2023-05-31T21:25:32.538 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T21:25:32.539 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2023-05-31T19:56:10.274572Z", 2023-05-31T21:25:32.539 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2023-05-31T21:25:32.539 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T21:25:32.539 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:25:32.539 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T21:25:32.539 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 3, 2023-05-31T21:25:32.539 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T21:25:32.540 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2023-05-31T21:25:32.540 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1685568189.652614_974.1", 2023-05-31T21:25:32.540 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T21:25:32.540 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2023-05-31T21:25:32.540 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T21:25:32.540 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2023-05-31T21:23:09.652614Z", 2023-05-31T21:25:32.540 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2023-05-31T21:25:32.541 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T21:25:32.541 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T21:25:32.541 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2023-05-31T21:25:32.541 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:25:32.541 INFO:teuthology.orchestra.run.smithi099.stdout: "full_sync": { 2023-05-31T21:25:32.541 INFO:teuthology.orchestra.run.smithi099.stdout: "total": 1, 2023-05-31T21:25:32.541 INFO:teuthology.orchestra.run.smithi099.stdout: "complete": 1 2023-05-31T21:25:32.542 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:25:32.542 INFO:teuthology.orchestra.run.smithi099.stdout: "current_time": "2023-05-31T21:25:32Z" 2023-05-31T21:25:32.542 INFO:teuthology.orchestra.run.smithi099.stdout:} 2023-05-31T21:25:32.543 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "2e6123d2-d695-4350-9a01-61a455bc782b", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1685566501.757183_260.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-05-31T20:55:01.757183Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1685566501.735717_251.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-05-31T20:55:01.735717Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1685562970.274572_93.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-05-31T19:56:10.274572Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1685568189.652614_974.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-05-31T21:23:09.652614Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-05-31T21:25:32Z" } 2023-05-31T21:25:32.544 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1685566501.757183_260.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-05-31T20:55:01.757183Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1685566501.735717_251.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-05-31T20:55:01.735717Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1685562970.274572_93.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-05-31T19:56:10.274572Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1685568189.652614_974.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-05-31T21:23:09.652614Z', 'realm_epoch': 2}}] 2023-05-31T21:25:32.544 DEBUG:rgw_multi.tests:log_status={0: '1_1685566501.757183_260.1', 1: '1_1685566501.735717_251.1', 2: '1_1685562970.274572_93.1', 3: '1_1685568316.644293_992.1'} 2023-05-31T21:25:32.544 DEBUG:rgw_multi.tests:sync_status={0: '1_1685566501.757183_260.1', 1: '1_1685566501.735717_251.1', 2: '1_1685562970.274572_93.1', 3: '1_1685568189.652614_974.1'} 2023-05-31T21:25:32.545 WARNING:rgw_multi.tests:zone a2 behind master: shard=3 master=1_1685568316.644293_992.1 target=1_1685568189.652614_974.1 2023-05-31T21:25:37.546 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-31T21:25:37.547 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-31T21:25:37.547 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-05-31T21:25:37.605 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2023-05-31T21:25:37.605 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2023-05-31T21:25:37.656 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T21:25:37.653+0000 7fb03fe07780 20 RGW-SYNC:meta: read sync status 2023-05-31T21:25:37.657 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T21:25:37.654+0000 7fb03fe07780 20 RGW-SYNC:meta: read sync status 2023-05-31T21:25:37.679 INFO:teuthology.orchestra.run.smithi099.stdout:{ 2023-05-31T21:25:37.680 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_status": { 2023-05-31T21:25:37.680 INFO:teuthology.orchestra.run.smithi099.stdout: "info": { 2023-05-31T21:25:37.680 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "sync", 2023-05-31T21:25:37.680 INFO:teuthology.orchestra.run.smithi099.stdout: "num_shards": 4, 2023-05-31T21:25:37.680 INFO:teuthology.orchestra.run.smithi099.stdout: "period": "2e6123d2-d695-4350-9a01-61a455bc782b", 2023-05-31T21:25:37.680 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2023-05-31T21:25:37.681 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:25:37.681 INFO:teuthology.orchestra.run.smithi099.stdout: "markers": [ 2023-05-31T21:25:37.681 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T21:25:37.681 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 0, 2023-05-31T21:25:37.681 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T21:25:37.681 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2023-05-31T21:25:37.682 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1685566501.757183_260.1", 2023-05-31T21:25:37.682 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T21:25:37.682 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 1, 2023-05-31T21:25:37.682 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T21:25:37.682 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2023-05-31T20:55:01.757183Z", 2023-05-31T21:25:37.682 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2023-05-31T21:25:37.682 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T21:25:37.683 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:25:37.683 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T21:25:37.683 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 1, 2023-05-31T21:25:37.683 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T21:25:37.683 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2023-05-31T21:25:37.683 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1685566501.735717_251.1", 2023-05-31T21:25:37.683 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T21:25:37.684 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2023-05-31T21:25:37.684 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T21:25:37.684 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2023-05-31T20:55:01.735717Z", 2023-05-31T21:25:37.684 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2023-05-31T21:25:37.684 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T21:25:37.684 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:25:37.685 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T21:25:37.685 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 2, 2023-05-31T21:25:37.685 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T21:25:37.685 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2023-05-31T21:25:37.685 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1685562970.274572_93.1", 2023-05-31T21:25:37.685 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T21:25:37.686 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2023-05-31T21:25:37.686 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T21:25:37.686 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2023-05-31T19:56:10.274572Z", 2023-05-31T21:25:37.686 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2023-05-31T21:25:37.686 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T21:25:37.686 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:25:37.686 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T21:25:37.687 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 3, 2023-05-31T21:25:37.687 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T21:25:37.687 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2023-05-31T21:25:37.687 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1685568189.652614_974.1", 2023-05-31T21:25:37.687 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T21:25:37.687 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2023-05-31T21:25:37.688 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T21:25:37.688 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2023-05-31T21:23:09.652614Z", 2023-05-31T21:25:37.688 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2023-05-31T21:25:37.688 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T21:25:37.689 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T21:25:37.689 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2023-05-31T21:25:37.689 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:25:37.689 INFO:teuthology.orchestra.run.smithi099.stdout: "full_sync": { 2023-05-31T21:25:37.689 INFO:teuthology.orchestra.run.smithi099.stdout: "total": 1, 2023-05-31T21:25:37.691 INFO:teuthology.orchestra.run.smithi099.stdout: "complete": 1 2023-05-31T21:25:37.691 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:25:37.691 INFO:teuthology.orchestra.run.smithi099.stdout: "current_time": "2023-05-31T21:25:37Z" 2023-05-31T21:25:37.691 INFO:teuthology.orchestra.run.smithi099.stdout:} 2023-05-31T21:25:37.693 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "2e6123d2-d695-4350-9a01-61a455bc782b", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1685566501.757183_260.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-05-31T20:55:01.757183Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1685566501.735717_251.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-05-31T20:55:01.735717Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1685562970.274572_93.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-05-31T19:56:10.274572Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1685568189.652614_974.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-05-31T21:23:09.652614Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-05-31T21:25:37Z" } 2023-05-31T21:25:37.693 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1685566501.757183_260.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-05-31T20:55:01.757183Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1685566501.735717_251.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-05-31T20:55:01.735717Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1685562970.274572_93.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-05-31T19:56:10.274572Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1685568189.652614_974.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-05-31T21:23:09.652614Z', 'realm_epoch': 2}}] 2023-05-31T21:25:37.694 DEBUG:rgw_multi.tests:log_status={0: '1_1685566501.757183_260.1', 1: '1_1685566501.735717_251.1', 2: '1_1685562970.274572_93.1', 3: '1_1685568316.644293_992.1'} 2023-05-31T21:25:37.694 DEBUG:rgw_multi.tests:sync_status={0: '1_1685566501.757183_260.1', 1: '1_1685566501.735717_251.1', 2: '1_1685562970.274572_93.1', 3: '1_1685568189.652614_974.1'} 2023-05-31T21:25:37.695 WARNING:rgw_multi.tests:zone a2 behind master: shard=3 master=1_1685568316.644293_992.1 target=1_1685568189.652614_974.1 2023-05-31T21:25:42.696 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-31T21:25:42.697 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-31T21:25:42.697 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-05-31T21:25:42.755 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2023-05-31T21:25:42.755 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2023-05-31T21:25:42.797 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T21:25:42.793+0000 7fc6b00e9780 20 RGW-SYNC:meta: read sync status 2023-05-31T21:25:42.797 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T21:25:42.794+0000 7fc6b00e9780 20 RGW-SYNC:meta: read sync status 2023-05-31T21:25:42.814 INFO:teuthology.orchestra.run.smithi099.stdout:{ 2023-05-31T21:25:42.814 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_status": { 2023-05-31T21:25:42.815 INFO:teuthology.orchestra.run.smithi099.stdout: "info": { 2023-05-31T21:25:42.815 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "sync", 2023-05-31T21:25:42.815 INFO:teuthology.orchestra.run.smithi099.stdout: "num_shards": 4, 2023-05-31T21:25:42.815 INFO:teuthology.orchestra.run.smithi099.stdout: "period": "2e6123d2-d695-4350-9a01-61a455bc782b", 2023-05-31T21:25:42.815 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2023-05-31T21:25:42.816 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:25:42.816 INFO:teuthology.orchestra.run.smithi099.stdout: "markers": [ 2023-05-31T21:25:42.816 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T21:25:42.816 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 0, 2023-05-31T21:25:42.817 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T21:25:42.817 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2023-05-31T21:25:42.817 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1685566501.757183_260.1", 2023-05-31T21:25:42.817 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T21:25:42.817 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 1, 2023-05-31T21:25:42.818 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T21:25:42.818 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2023-05-31T20:55:01.757183Z", 2023-05-31T21:25:42.818 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2023-05-31T21:25:42.818 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T21:25:42.818 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:25:42.818 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T21:25:42.819 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 1, 2023-05-31T21:25:42.819 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T21:25:42.819 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2023-05-31T21:25:42.819 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1685566501.735717_251.1", 2023-05-31T21:25:42.820 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T21:25:42.820 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2023-05-31T21:25:42.820 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T21:25:42.820 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2023-05-31T20:55:01.735717Z", 2023-05-31T21:25:42.820 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2023-05-31T21:25:42.820 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T21:25:42.821 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:25:42.821 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T21:25:42.821 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 2, 2023-05-31T21:25:42.821 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T21:25:42.821 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2023-05-31T21:25:42.822 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1685562970.274572_93.1", 2023-05-31T21:25:42.822 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T21:25:42.823 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2023-05-31T21:25:42.823 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T21:25:42.823 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2023-05-31T19:56:10.274572Z", 2023-05-31T21:25:42.824 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2023-05-31T21:25:42.824 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T21:25:42.824 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:25:42.825 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T21:25:42.825 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 3, 2023-05-31T21:25:42.826 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T21:25:42.826 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2023-05-31T21:25:42.826 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1685568189.652614_974.1", 2023-05-31T21:25:42.827 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T21:25:42.827 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2023-05-31T21:25:42.827 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T21:25:42.828 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2023-05-31T21:23:09.652614Z", 2023-05-31T21:25:42.828 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2023-05-31T21:25:42.829 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T21:25:42.829 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T21:25:42.829 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2023-05-31T21:25:42.830 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:25:42.830 INFO:teuthology.orchestra.run.smithi099.stdout: "full_sync": { 2023-05-31T21:25:42.830 INFO:teuthology.orchestra.run.smithi099.stdout: "total": 1, 2023-05-31T21:25:42.831 INFO:teuthology.orchestra.run.smithi099.stdout: "complete": 1 2023-05-31T21:25:42.831 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:25:42.831 INFO:teuthology.orchestra.run.smithi099.stdout: "current_time": "2023-05-31T21:25:42Z" 2023-05-31T21:25:42.832 INFO:teuthology.orchestra.run.smithi099.stdout:} 2023-05-31T21:25:42.833 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "2e6123d2-d695-4350-9a01-61a455bc782b", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1685566501.757183_260.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-05-31T20:55:01.757183Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1685566501.735717_251.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-05-31T20:55:01.735717Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1685562970.274572_93.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-05-31T19:56:10.274572Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1685568189.652614_974.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-05-31T21:23:09.652614Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-05-31T21:25:42Z" } 2023-05-31T21:25:42.834 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1685566501.757183_260.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-05-31T20:55:01.757183Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1685566501.735717_251.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-05-31T20:55:01.735717Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1685562970.274572_93.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-05-31T19:56:10.274572Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1685568189.652614_974.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-05-31T21:23:09.652614Z', 'realm_epoch': 2}}] 2023-05-31T21:25:42.835 DEBUG:rgw_multi.tests:log_status={0: '1_1685566501.757183_260.1', 1: '1_1685566501.735717_251.1', 2: '1_1685562970.274572_93.1', 3: '1_1685568316.644293_992.1'} 2023-05-31T21:25:42.835 DEBUG:rgw_multi.tests:sync_status={0: '1_1685566501.757183_260.1', 1: '1_1685566501.735717_251.1', 2: '1_1685562970.274572_93.1', 3: '1_1685568189.652614_974.1'} 2023-05-31T21:25:42.836 WARNING:rgw_multi.tests:zone a2 behind master: shard=3 master=1_1685568316.644293_992.1 target=1_1685568189.652614_974.1 2023-05-31T21:25:47.837 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-31T21:25:47.837 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-31T21:25:47.838 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-05-31T21:25:47.895 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2023-05-31T21:25:47.895 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2023-05-31T21:25:47.935 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T21:25:47.931+0000 7f6b179bc780 20 RGW-SYNC:meta: read sync status 2023-05-31T21:25:47.936 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T21:25:47.932+0000 7f6b179bc780 20 RGW-SYNC:meta: read sync status 2023-05-31T21:25:47.952 INFO:teuthology.orchestra.run.smithi099.stdout:{ 2023-05-31T21:25:47.953 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_status": { 2023-05-31T21:25:47.953 INFO:teuthology.orchestra.run.smithi099.stdout: "info": { 2023-05-31T21:25:47.953 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "sync", 2023-05-31T21:25:47.953 INFO:teuthology.orchestra.run.smithi099.stdout: "num_shards": 4, 2023-05-31T21:25:47.953 INFO:teuthology.orchestra.run.smithi099.stdout: "period": "2e6123d2-d695-4350-9a01-61a455bc782b", 2023-05-31T21:25:47.954 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2023-05-31T21:25:47.954 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:25:47.954 INFO:teuthology.orchestra.run.smithi099.stdout: "markers": [ 2023-05-31T21:25:47.954 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T21:25:47.955 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 0, 2023-05-31T21:25:47.955 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T21:25:47.955 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2023-05-31T21:25:47.956 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1685566501.757183_260.1", 2023-05-31T21:25:47.956 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T21:25:47.956 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 1, 2023-05-31T21:25:47.957 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T21:25:47.957 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2023-05-31T20:55:01.757183Z", 2023-05-31T21:25:47.957 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2023-05-31T21:25:47.958 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T21:25:47.958 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:25:47.958 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T21:25:47.958 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 1, 2023-05-31T21:25:47.959 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T21:25:47.959 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2023-05-31T21:25:47.959 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1685566501.735717_251.1", 2023-05-31T21:25:47.960 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T21:25:47.961 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2023-05-31T21:25:47.961 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T21:25:47.961 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2023-05-31T20:55:01.735717Z", 2023-05-31T21:25:47.961 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2023-05-31T21:25:47.961 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T21:25:47.961 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:25:47.962 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T21:25:47.962 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 2, 2023-05-31T21:25:47.962 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T21:25:47.962 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2023-05-31T21:25:47.963 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1685562970.274572_93.1", 2023-05-31T21:25:47.963 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T21:25:47.963 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2023-05-31T21:25:47.964 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T21:25:47.964 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2023-05-31T19:56:10.274572Z", 2023-05-31T21:25:47.964 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2023-05-31T21:25:47.965 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T21:25:47.965 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:25:47.965 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T21:25:47.966 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 3, 2023-05-31T21:25:47.966 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T21:25:47.966 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2023-05-31T21:25:47.967 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1685568316.022446_981.1", 2023-05-31T21:25:47.967 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T21:25:47.967 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2023-05-31T21:25:47.968 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T21:25:47.968 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2023-05-31T21:25:16.022446Z", 2023-05-31T21:25:47.968 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2023-05-31T21:25:47.969 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T21:25:47.969 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T21:25:47.969 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2023-05-31T21:25:47.970 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:25:47.970 INFO:teuthology.orchestra.run.smithi099.stdout: "full_sync": { 2023-05-31T21:25:47.970 INFO:teuthology.orchestra.run.smithi099.stdout: "total": 1, 2023-05-31T21:25:47.971 INFO:teuthology.orchestra.run.smithi099.stdout: "complete": 1 2023-05-31T21:25:47.971 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:25:47.971 INFO:teuthology.orchestra.run.smithi099.stdout: "current_time": "2023-05-31T21:25:47Z" 2023-05-31T21:25:47.972 INFO:teuthology.orchestra.run.smithi099.stdout:} 2023-05-31T21:25:47.973 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "2e6123d2-d695-4350-9a01-61a455bc782b", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1685566501.757183_260.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-05-31T20:55:01.757183Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1685566501.735717_251.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-05-31T20:55:01.735717Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1685562970.274572_93.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-05-31T19:56:10.274572Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1685568316.022446_981.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-05-31T21:25:16.022446Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-05-31T21:25:47Z" } 2023-05-31T21:25:47.974 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1685566501.757183_260.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-05-31T20:55:01.757183Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1685566501.735717_251.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-05-31T20:55:01.735717Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1685562970.274572_93.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-05-31T19:56:10.274572Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1685568316.022446_981.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-05-31T21:25:16.022446Z', 'realm_epoch': 2}}] 2023-05-31T21:25:47.975 DEBUG:rgw_multi.tests:log_status={0: '1_1685566501.757183_260.1', 1: '1_1685566501.735717_251.1', 2: '1_1685562970.274572_93.1', 3: '1_1685568316.644293_992.1'} 2023-05-31T21:25:47.975 DEBUG:rgw_multi.tests:sync_status={0: '1_1685566501.757183_260.1', 1: '1_1685566501.735717_251.1', 2: '1_1685562970.274572_93.1', 3: '1_1685568316.022446_981.1'} 2023-05-31T21:25:47.976 WARNING:rgw_multi.tests:zone a2 behind master: shard=3 master=1_1685568316.644293_992.1 target=1_1685568316.022446_981.1 2023-05-31T21:25:52.977 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-31T21:25:52.978 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-31T21:25:52.978 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-05-31T21:25:53.035 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2023-05-31T21:25:53.035 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2023-05-31T21:25:53.078 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T21:25:53.074+0000 7feb42635780 20 RGW-SYNC:meta: read sync status 2023-05-31T21:25:53.079 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T21:25:53.075+0000 7feb42635780 20 RGW-SYNC:meta: read sync status 2023-05-31T21:25:53.096 INFO:teuthology.orchestra.run.smithi099.stdout:{ 2023-05-31T21:25:53.097 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_status": { 2023-05-31T21:25:53.097 INFO:teuthology.orchestra.run.smithi099.stdout: "info": { 2023-05-31T21:25:53.097 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "sync", 2023-05-31T21:25:53.097 INFO:teuthology.orchestra.run.smithi099.stdout: "num_shards": 4, 2023-05-31T21:25:53.097 INFO:teuthology.orchestra.run.smithi099.stdout: "period": "2e6123d2-d695-4350-9a01-61a455bc782b", 2023-05-31T21:25:53.098 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2023-05-31T21:25:53.098 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:25:53.098 INFO:teuthology.orchestra.run.smithi099.stdout: "markers": [ 2023-05-31T21:25:53.098 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T21:25:53.098 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 0, 2023-05-31T21:25:53.098 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T21:25:53.098 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2023-05-31T21:25:53.099 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1685566501.757183_260.1", 2023-05-31T21:25:53.099 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T21:25:53.099 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 1, 2023-05-31T21:25:53.099 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T21:25:53.099 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2023-05-31T20:55:01.757183Z", 2023-05-31T21:25:53.099 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2023-05-31T21:25:53.099 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T21:25:53.100 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:25:53.100 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T21:25:53.100 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 1, 2023-05-31T21:25:53.100 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T21:25:53.100 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2023-05-31T21:25:53.100 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1685566501.735717_251.1", 2023-05-31T21:25:53.101 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T21:25:53.101 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2023-05-31T21:25:53.101 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T21:25:53.101 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2023-05-31T20:55:01.735717Z", 2023-05-31T21:25:53.101 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2023-05-31T21:25:53.101 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T21:25:53.101 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:25:53.102 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T21:25:53.102 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 2, 2023-05-31T21:25:53.102 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T21:25:53.102 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2023-05-31T21:25:53.102 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1685562970.274572_93.1", 2023-05-31T21:25:53.103 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T21:25:53.104 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2023-05-31T21:25:53.104 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T21:25:53.105 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2023-05-31T19:56:10.274572Z", 2023-05-31T21:25:53.105 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2023-05-31T21:25:53.105 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T21:25:53.106 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:25:53.106 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T21:25:53.106 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 3, 2023-05-31T21:25:53.107 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T21:25:53.107 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2023-05-31T21:25:53.107 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1685568316.644293_992.1", 2023-05-31T21:25:53.108 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T21:25:53.108 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2023-05-31T21:25:53.109 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T21:25:53.109 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2023-05-31T21:25:16.644293Z", 2023-05-31T21:25:53.109 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2023-05-31T21:25:53.109 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T21:25:53.110 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T21:25:53.110 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2023-05-31T21:25:53.110 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:25:53.111 INFO:teuthology.orchestra.run.smithi099.stdout: "full_sync": { 2023-05-31T21:25:53.111 INFO:teuthology.orchestra.run.smithi099.stdout: "total": 1, 2023-05-31T21:25:53.111 INFO:teuthology.orchestra.run.smithi099.stdout: "complete": 1 2023-05-31T21:25:53.111 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:25:53.111 INFO:teuthology.orchestra.run.smithi099.stdout: "current_time": "2023-05-31T21:25:53Z" 2023-05-31T21:25:53.112 INFO:teuthology.orchestra.run.smithi099.stdout:} 2023-05-31T21:25:53.113 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "2e6123d2-d695-4350-9a01-61a455bc782b", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1685566501.757183_260.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-05-31T20:55:01.757183Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1685566501.735717_251.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-05-31T20:55:01.735717Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1685562970.274572_93.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-05-31T19:56:10.274572Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1685568316.644293_992.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-05-31T21:25:16.644293Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-05-31T21:25:53Z" } 2023-05-31T21:25:53.113 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1685566501.757183_260.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-05-31T20:55:01.757183Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1685566501.735717_251.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-05-31T20:55:01.735717Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1685562970.274572_93.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-05-31T19:56:10.274572Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1685568316.644293_992.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-05-31T21:25:16.644293Z', 'realm_epoch': 2}}] 2023-05-31T21:25:53.114 DEBUG:rgw_multi.tests:log_status={0: '1_1685566501.757183_260.1', 1: '1_1685566501.735717_251.1', 2: '1_1685562970.274572_93.1', 3: '1_1685568316.644293_992.1'} 2023-05-31T21:25:53.114 DEBUG:rgw_multi.tests:sync_status={0: '1_1685566501.757183_260.1', 1: '1_1685566501.735717_251.1', 2: '1_1685562970.274572_93.1', 3: '1_1685568316.644293_992.1'} 2023-05-31T21:25:53.114 INFO:rgw_multi.tests:finish meta checkpoint for zone=a2 2023-05-31T21:25:53.115 DEBUG:boto:path=/pklxkd-70/ 2023-05-31T21:25:53.115 DEBUG:boto:auth_path=/pklxkd-70/ 2023-05-31T21:25:53.116 DEBUG:boto:Method: HEAD 2023-05-31T21:25:53.117 DEBUG:boto:Path: /pklxkd-70/ 2023-05-31T21:25:53.117 DEBUG:boto:Data: 2023-05-31T21:25:53.117 DEBUG:boto:Headers: {} 2023-05-31T21:25:53.118 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T21:25:53.119 DEBUG:boto:Port: 8000 2023-05-31T21:25:53.119 DEBUG:boto:Params: {} 2023-05-31T21:25:53.120 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2023-05-31T21:25:53.121 DEBUG:boto:Token: None 2023-05-31T21:25:53.122 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 21:25:53 GMT /pklxkd-70/ 2023-05-31T21:25:53.124 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:qpkaMI3tdj7B/99ATeOZqZjJnxg= 2023-05-31T21:25:53.124 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 21:25:53 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:qpkaMI3tdj7B/99ATeOZqZjJnxg=', 'Content-Length': '0'} 2023-05-31T21:25:53.129 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '2'), ('X-RGW-Bytes-Used', '12'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000d356265dac8c58ca-006477bb61-264e-a1'), ('Content-Length', '0'), ('Date', 'Wed, 31 May 2023 21:25:53 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:25:53.130 DEBUG:boto:path=/pklxkd-70/obj5 2023-05-31T21:25:53.131 DEBUG:boto:auth_path=/pklxkd-70/obj5 2023-05-31T21:25:53.131 DEBUG:boto:Method: PUT 2023-05-31T21:25:53.132 DEBUG:boto:Path: /pklxkd-70/obj5 2023-05-31T21:25:53.132 DEBUG:boto:Data: 2023-05-31T21:25:53.133 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2023-05-31T21:25:53.134 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T21:25:53.134 DEBUG:boto:Port: 8000 2023-05-31T21:25:53.134 DEBUG:boto:Params: {} 2023-05-31T21:25:53.135 DEBUG:boto:Token: None 2023-05-31T21:25:53.136 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Wed, 31 May 2023 21:25:53 GMT /pklxkd-70/obj5 2023-05-31T21:25:53.136 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:TZANIdCyqeXYn82mrz8Kc4kFosQ= 2023-05-31T21:25:53.137 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Wed, 31 May 2023 21:25:53 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:TZANIdCyqeXYn82mrz8Kc4kFosQ='} 2023-05-31T21:25:53.146 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000070a5cd4ce7fd7b0c-006477bb61-264e-a1'), ('Date', 'Wed, 31 May 2023 21:25:53 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:25:53.147 DEBUG:boto:path=/pklxkd-70/ 2023-05-31T21:25:53.148 DEBUG:boto:auth_path=/pklxkd-70/ 2023-05-31T21:25:53.148 DEBUG:boto:Method: HEAD 2023-05-31T21:25:53.148 DEBUG:boto:Path: /pklxkd-70/ 2023-05-31T21:25:53.149 DEBUG:boto:Data: 2023-05-31T21:25:53.149 DEBUG:boto:Headers: {} 2023-05-31T21:25:53.149 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T21:25:53.150 DEBUG:boto:Port: 8000 2023-05-31T21:25:53.150 DEBUG:boto:Params: {} 2023-05-31T21:25:53.150 DEBUG:boto:Token: None 2023-05-31T21:25:53.151 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 21:25:53 GMT /pklxkd-70/ 2023-05-31T21:25:53.151 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:qpkaMI3tdj7B/99ATeOZqZjJnxg= 2023-05-31T21:25:53.152 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 21:25:53 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:qpkaMI3tdj7B/99ATeOZqZjJnxg=', 'Content-Length': '0'} 2023-05-31T21:25:53.154 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '3'), ('X-RGW-Bytes-Used', '18'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000001db4b0e58cc1349b-006477bb61-264e-a1'), ('Content-Length', '0'), ('Date', 'Wed, 31 May 2023 21:25:53 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:25:53.155 DEBUG:boto:path=/pklxkd-70/obj6 2023-05-31T21:25:53.157 DEBUG:boto:auth_path=/pklxkd-70/obj6 2023-05-31T21:25:53.158 DEBUG:boto:Method: PUT 2023-05-31T21:25:53.160 DEBUG:boto:Path: /pklxkd-70/obj6 2023-05-31T21:25:53.161 DEBUG:boto:Data: 2023-05-31T21:25:53.161 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2023-05-31T21:25:53.162 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T21:25:53.162 DEBUG:boto:Port: 8000 2023-05-31T21:25:53.162 DEBUG:boto:Params: {} 2023-05-31T21:25:53.163 DEBUG:boto:Token: None 2023-05-31T21:25:53.163 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Wed, 31 May 2023 21:25:53 GMT /pklxkd-70/obj6 2023-05-31T21:25:53.164 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:jXahvW8xxYSCryTSu0hY3coXejE= 2023-05-31T21:25:53.164 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Wed, 31 May 2023 21:25:53 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:jXahvW8xxYSCryTSu0hY3coXejE='} 2023-05-31T21:25:53.175 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000008ede50218f46ac45-006477bb61-264e-a1'), ('Date', 'Wed, 31 May 2023 21:25:53 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:25:53.176 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-31T21:25:53.176 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-31T21:25:53.176 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-05-31T21:25:53.228 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2023-05-31T21:25:53.229 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2023-05-31T21:25:53.267 INFO:teuthology.orchestra.run.smithi099.stderr:No --period given, using current period=2e6123d2-d695-4350-9a01-61a455bc782b 2023-05-31T21:25:53.281 INFO:teuthology.orchestra.run.smithi099.stdout:[ 2023-05-31T21:25:53.281 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T21:25:53.281 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1685566501.757183_260.1", 2023-05-31T21:25:53.282 INFO:teuthology.orchestra.run.smithi099.stdout: "last_update": "2023-05-31T20:55:01.757183Z" 2023-05-31T21:25:53.282 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:25:53.282 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T21:25:53.282 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1685566501.735717_251.1", 2023-05-31T21:25:53.282 INFO:teuthology.orchestra.run.smithi099.stdout: "last_update": "2023-05-31T20:55:01.735717Z" 2023-05-31T21:25:53.282 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:25:53.282 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T21:25:53.283 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1685562970.274572_93.1", 2023-05-31T21:25:53.283 INFO:teuthology.orchestra.run.smithi099.stdout: "last_update": "2023-05-31T19:56:10.274572Z" 2023-05-31T21:25:53.283 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:25:53.283 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T21:25:53.283 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1685568316.644293_992.1", 2023-05-31T21:25:53.283 INFO:teuthology.orchestra.run.smithi099.stdout: "last_update": "2023-05-31T21:25:16.644293Z" 2023-05-31T21:25:53.283 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T21:25:53.284 INFO:teuthology.orchestra.run.smithi099.stdout:] 2023-05-31T21:25:53.285 DEBUG:rgw_multi.tests:master meta markers={0: '1_1685566501.757183_260.1', 1: '1_1685566501.735717_251.1', 2: '1_1685562970.274572_93.1', 3: '1_1685568316.644293_992.1'} 2023-05-31T21:25:53.285 INFO:rgw_multi.tests:starting meta checkpoint for zone=a2 2023-05-31T21:25:53.285 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-31T21:25:53.286 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-31T21:25:53.286 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-05-31T21:25:53.336 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2023-05-31T21:25:53.336 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2023-05-31T21:25:53.371 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T21:25:53.367+0000 7f59e1faa780 20 RGW-SYNC:meta: read sync status 2023-05-31T21:25:53.371 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T21:25:53.367+0000 7f59e1faa780 20 RGW-SYNC:meta: read sync status 2023-05-31T21:25:53.395 INFO:teuthology.orchestra.run.smithi099.stdout:{ 2023-05-31T21:25:53.395 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_status": { 2023-05-31T21:25:53.395 INFO:teuthology.orchestra.run.smithi099.stdout: "info": { 2023-05-31T21:25:53.396 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "sync", 2023-05-31T21:25:53.396 INFO:teuthology.orchestra.run.smithi099.stdout: "num_shards": 4, 2023-05-31T21:25:53.396 INFO:teuthology.orchestra.run.smithi099.stdout: "period": "2e6123d2-d695-4350-9a01-61a455bc782b", 2023-05-31T21:25:53.396 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2023-05-31T21:25:53.396 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:25:53.396 INFO:teuthology.orchestra.run.smithi099.stdout: "markers": [ 2023-05-31T21:25:53.397 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T21:25:53.397 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 0, 2023-05-31T21:25:53.397 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T21:25:53.397 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2023-05-31T21:25:53.397 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1685566501.757183_260.1", 2023-05-31T21:25:53.397 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T21:25:53.397 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 1, 2023-05-31T21:25:53.398 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T21:25:53.398 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2023-05-31T20:55:01.757183Z", 2023-05-31T21:25:53.398 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2023-05-31T21:25:53.398 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T21:25:53.398 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:25:53.398 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T21:25:53.398 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 1, 2023-05-31T21:25:53.399 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T21:25:53.399 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2023-05-31T21:25:53.399 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1685566501.735717_251.1", 2023-05-31T21:25:53.399 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T21:25:53.399 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2023-05-31T21:25:53.399 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T21:25:53.399 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2023-05-31T20:55:01.735717Z", 2023-05-31T21:25:53.400 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2023-05-31T21:25:53.400 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T21:25:53.400 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:25:53.400 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T21:25:53.400 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 2, 2023-05-31T21:25:53.400 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T21:25:53.401 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2023-05-31T21:25:53.401 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1685562970.274572_93.1", 2023-05-31T21:25:53.401 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T21:25:53.401 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2023-05-31T21:25:53.401 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T21:25:53.401 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2023-05-31T19:56:10.274572Z", 2023-05-31T21:25:53.401 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2023-05-31T21:25:53.402 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T21:25:53.402 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:25:53.402 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T21:25:53.402 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 3, 2023-05-31T21:25:53.402 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T21:25:53.403 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2023-05-31T21:25:53.403 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1685568316.644293_992.1", 2023-05-31T21:25:53.403 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T21:25:53.403 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2023-05-31T21:25:53.403 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T21:25:53.403 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2023-05-31T21:25:16.644293Z", 2023-05-31T21:25:53.404 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2023-05-31T21:25:53.404 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T21:25:53.404 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T21:25:53.404 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2023-05-31T21:25:53.404 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:25:53.404 INFO:teuthology.orchestra.run.smithi099.stdout: "full_sync": { 2023-05-31T21:25:53.404 INFO:teuthology.orchestra.run.smithi099.stdout: "total": 1, 2023-05-31T21:25:53.405 INFO:teuthology.orchestra.run.smithi099.stdout: "complete": 1 2023-05-31T21:25:53.405 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:25:53.405 INFO:teuthology.orchestra.run.smithi099.stdout: "current_time": "2023-05-31T21:25:53Z" 2023-05-31T21:25:53.405 INFO:teuthology.orchestra.run.smithi099.stdout:} 2023-05-31T21:25:53.405 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "2e6123d2-d695-4350-9a01-61a455bc782b", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1685566501.757183_260.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-05-31T20:55:01.757183Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1685566501.735717_251.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-05-31T20:55:01.735717Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1685562970.274572_93.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-05-31T19:56:10.274572Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1685568316.644293_992.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-05-31T21:25:16.644293Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-05-31T21:25:53Z" } 2023-05-31T21:25:53.406 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1685566501.757183_260.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-05-31T20:55:01.757183Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1685566501.735717_251.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-05-31T20:55:01.735717Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1685562970.274572_93.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-05-31T19:56:10.274572Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1685568316.644293_992.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-05-31T21:25:16.644293Z', 'realm_epoch': 2}}] 2023-05-31T21:25:53.406 DEBUG:rgw_multi.tests:log_status={0: '1_1685566501.757183_260.1', 1: '1_1685566501.735717_251.1', 2: '1_1685562970.274572_93.1', 3: '1_1685568316.644293_992.1'} 2023-05-31T21:25:53.407 DEBUG:rgw_multi.tests:sync_status={0: '1_1685566501.757183_260.1', 1: '1_1685566501.735717_251.1', 2: '1_1685562970.274572_93.1', 3: '1_1685568316.644293_992.1'} 2023-05-31T21:25:53.407 INFO:rgw_multi.tests:finish meta checkpoint for zone=a2 2023-05-31T21:25:53.407 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['datalog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-31T21:25:53.408 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'datalog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-31T21:25:53.408 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 datalog status --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-05-31T21:25:53.456 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2023-05-31T21:25:53.456 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2023-05-31T21:25:53.529 INFO:teuthology.orchestra.run.smithi099.stdout:[ 2023-05-31T21:25:53.530 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T21:25:53.531 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "00000000000000000000:00000000000000045235", 2023-05-31T21:25:53.534 INFO:teuthology.orchestra.run.smithi099.stdout: "last_update": "2023-05-31T21:25:53.171643Z" 2023-05-31T21:25:53.534 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:25:53.534 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T21:25:53.535 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "00000000000000000000:00000000000000031499", 2023-05-31T21:25:53.535 INFO:teuthology.orchestra.run.smithi099.stdout: "last_update": "2023-05-31T21:12:40.667485Z" 2023-05-31T21:25:53.535 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:25:53.536 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T21:25:53.536 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "00000000000000000000:00000000000000051391", 2023-05-31T21:25:53.536 INFO:teuthology.orchestra.run.smithi099.stdout: "last_update": "2023-05-31T21:25:53.141700Z" 2023-05-31T21:25:53.537 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:25:53.537 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T21:25:53.537 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "00000000000000000000:00000000000000044038", 2023-05-31T21:25:53.539 INFO:teuthology.orchestra.run.smithi099.stdout: "last_update": "2023-05-31T21:21:35.912562Z" 2023-05-31T21:25:53.539 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T21:25:53.540 INFO:teuthology.orchestra.run.smithi099.stdout:] 2023-05-31T21:25:53.541 DEBUG:rgw_multi.tests:data markers for zone=a1 markers={0: '00000000000000000000:00000000000000045235', 1: '00000000000000000000:00000000000000031499', 2: '00000000000000000000:00000000000000051391', 3: '00000000000000000000:00000000000000044038'} 2023-05-31T21:25:53.541 INFO:rgw_multi.tests:starting data checkpoint for target_zone=a2 source_zone=a1 2023-05-31T21:25:53.542 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-31T21:25:53.542 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-31T21:25:53.543 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-05-31T21:25:53.594 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2023-05-31T21:25:53.594 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2023-05-31T21:25:53.644 INFO:teuthology.orchestra.run.smithi099.stdout:{ 2023-05-31T21:25:53.644 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_status": { 2023-05-31T21:25:53.644 INFO:teuthology.orchestra.run.smithi099.stdout: "info": { 2023-05-31T21:25:53.645 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "sync", 2023-05-31T21:25:53.645 INFO:teuthology.orchestra.run.smithi099.stdout: "num_shards": 4, 2023-05-31T21:25:53.645 INFO:teuthology.orchestra.run.smithi099.stdout: "instance_id": 452136274422638824 2023-05-31T21:25:53.645 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:25:53.645 INFO:teuthology.orchestra.run.smithi099.stdout: "markers": [ 2023-05-31T21:25:53.645 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T21:25:53.646 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 0, 2023-05-31T21:25:53.646 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T21:25:53.646 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "incremental-sync", 2023-05-31T21:25:53.646 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "00000000000000000000:00000000000000045095", 2023-05-31T21:25:53.646 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T21:25:53.646 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2023-05-31T21:25:53.646 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T21:25:53.647 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2023-05-31T21:23:41.891197Z" 2023-05-31T21:25:53.647 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T21:25:53.647 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:25:53.647 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T21:25:53.647 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 1, 2023-05-31T21:25:53.647 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T21:25:53.647 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "incremental-sync", 2023-05-31T21:25:53.648 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "00000000000000000000:00000000000000031499", 2023-05-31T21:25:53.648 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T21:25:53.648 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2023-05-31T21:25:53.648 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T21:25:53.648 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2023-05-31T21:12:40.667485Z" 2023-05-31T21:25:53.648 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T21:25:53.649 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:25:53.649 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T21:25:53.649 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 2, 2023-05-31T21:25:53.649 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T21:25:53.649 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "incremental-sync", 2023-05-31T21:25:53.649 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "00000000000000000000:00000000000000051251", 2023-05-31T21:25:53.649 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T21:25:53.650 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 55, 2023-05-31T21:25:53.650 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T21:25:53.650 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2023-05-31T21:23:41.857089Z" 2023-05-31T21:25:53.650 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T21:25:53.650 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:25:53.650 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T21:25:53.651 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 3, 2023-05-31T21:25:53.651 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T21:25:53.651 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "incremental-sync", 2023-05-31T21:25:53.651 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "00000000000000000000:00000000000000044038", 2023-05-31T21:25:53.651 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T21:25:53.651 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 9, 2023-05-31T21:25:53.651 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T21:25:53.652 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2023-05-31T21:21:35.912562Z" 2023-05-31T21:25:53.652 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T21:25:53.652 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T21:25:53.652 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2023-05-31T21:25:53.652 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:25:53.653 INFO:teuthology.orchestra.run.smithi099.stdout: "full_sync": { 2023-05-31T21:25:53.653 INFO:teuthology.orchestra.run.smithi099.stdout: "total": 64, 2023-05-31T21:25:53.653 INFO:teuthology.orchestra.run.smithi099.stdout: "complete": 64 2023-05-31T21:25:53.653 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:25:53.653 INFO:teuthology.orchestra.run.smithi099.stdout: "current_time": "2023-05-31T21:25:53Z" 2023-05-31T21:25:53.653 INFO:teuthology.orchestra.run.smithi099.stdout:} 2023-05-31T21:25:53.655 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 452136274422638824 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000045095", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-05-31T21:23:41.891197Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000031499", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-05-31T21:12:40.667485Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000051251", "next_step_marker": "", "total_entries": 55, "pos": 0, "timestamp": "2023-05-31T21:23:41.857089Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000044038", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2023-05-31T21:21:35.912562Z" } } ] }, "full_sync": { "total": 64, "complete": 64 }, "current_time": "2023-05-31T21:25:53Z" } 2023-05-31T21:25:53.655 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000045095', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-05-31T21:23:41.891197Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000031499', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-05-31T21:12:40.667485Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000051251', 'next_step_marker': '', 'total_entries': 55, 'pos': 0, 'timestamp': '2023-05-31T21:23:41.857089Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000044038', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2023-05-31T21:21:35.912562Z'}}] 2023-05-31T21:25:53.656 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000045235', 1: '00000000000000000000:00000000000000031499', 2: '00000000000000000000:00000000000000051391', 3: '00000000000000000000:00000000000000044038'} 2023-05-31T21:25:53.656 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000045095', 1: '00000000000000000000:00000000000000031499', 2: '00000000000000000000:00000000000000051251', 3: '00000000000000000000:00000000000000044038'} 2023-05-31T21:25:53.656 WARNING:rgw_multi.tests:data of zone a2 behind zone a1: shard=0 master=00000000000000000000:00000000000000045235 target=00000000000000000000:00000000000000045095, shard=2 master=00000000000000000000:00000000000000051391 target=00000000000000000000:00000000000000051251 2023-05-31T21:25:58.657 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-31T21:25:58.658 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-31T21:25:58.658 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-05-31T21:25:58.717 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2023-05-31T21:25:58.717 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2023-05-31T21:25:58.795 INFO:teuthology.orchestra.run.smithi099.stdout:{ 2023-05-31T21:25:58.796 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_status": { 2023-05-31T21:25:58.796 INFO:teuthology.orchestra.run.smithi099.stdout: "info": { 2023-05-31T21:25:58.796 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "sync", 2023-05-31T21:25:58.797 INFO:teuthology.orchestra.run.smithi099.stdout: "num_shards": 4, 2023-05-31T21:25:58.797 INFO:teuthology.orchestra.run.smithi099.stdout: "instance_id": 452136274422638824 2023-05-31T21:25:58.797 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:25:58.798 INFO:teuthology.orchestra.run.smithi099.stdout: "markers": [ 2023-05-31T21:25:58.798 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T21:25:58.798 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 0, 2023-05-31T21:25:58.799 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T21:25:58.799 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "incremental-sync", 2023-05-31T21:25:58.799 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "00000000000000000000:00000000000000045095", 2023-05-31T21:25:58.799 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T21:25:58.800 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2023-05-31T21:25:58.800 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T21:25:58.800 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2023-05-31T21:23:41.891197Z" 2023-05-31T21:25:58.800 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T21:25:58.801 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:25:58.801 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T21:25:58.801 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 1, 2023-05-31T21:25:58.802 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T21:25:58.802 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "incremental-sync", 2023-05-31T21:25:58.802 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "00000000000000000000:00000000000000031499", 2023-05-31T21:25:58.802 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T21:25:58.802 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2023-05-31T21:25:58.802 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T21:25:58.803 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2023-05-31T21:12:40.667485Z" 2023-05-31T21:25:58.803 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T21:25:58.803 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:25:58.803 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T21:25:58.803 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 2, 2023-05-31T21:25:58.803 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T21:25:58.804 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "incremental-sync", 2023-05-31T21:25:58.804 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "00000000000000000000:00000000000000051251", 2023-05-31T21:25:58.804 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T21:25:58.804 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 55, 2023-05-31T21:25:58.805 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T21:25:58.805 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2023-05-31T21:23:41.857089Z" 2023-05-31T21:25:58.805 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T21:25:58.805 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:25:58.806 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T21:25:58.806 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 3, 2023-05-31T21:25:58.806 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T21:25:58.807 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "incremental-sync", 2023-05-31T21:25:58.807 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "00000000000000000000:00000000000000044038", 2023-05-31T21:25:58.807 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T21:25:58.808 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 9, 2023-05-31T21:25:58.808 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T21:25:58.808 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2023-05-31T21:21:35.912562Z" 2023-05-31T21:25:58.809 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T21:25:58.809 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T21:25:58.809 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2023-05-31T21:25:58.809 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:25:58.810 INFO:teuthology.orchestra.run.smithi099.stdout: "full_sync": { 2023-05-31T21:25:58.810 INFO:teuthology.orchestra.run.smithi099.stdout: "total": 64, 2023-05-31T21:25:58.810 INFO:teuthology.orchestra.run.smithi099.stdout: "complete": 64 2023-05-31T21:25:58.811 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:25:58.811 INFO:teuthology.orchestra.run.smithi099.stdout: "current_time": "2023-05-31T21:25:58Z" 2023-05-31T21:25:58.811 INFO:teuthology.orchestra.run.smithi099.stdout:} 2023-05-31T21:25:58.812 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 452136274422638824 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000045095", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-05-31T21:23:41.891197Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000031499", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-05-31T21:12:40.667485Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000051251", "next_step_marker": "", "total_entries": 55, "pos": 0, "timestamp": "2023-05-31T21:23:41.857089Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000044038", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2023-05-31T21:21:35.912562Z" } } ] }, "full_sync": { "total": 64, "complete": 64 }, "current_time": "2023-05-31T21:25:58Z" } 2023-05-31T21:25:58.813 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000045095', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-05-31T21:23:41.891197Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000031499', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-05-31T21:12:40.667485Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000051251', 'next_step_marker': '', 'total_entries': 55, 'pos': 0, 'timestamp': '2023-05-31T21:23:41.857089Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000044038', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2023-05-31T21:21:35.912562Z'}}] 2023-05-31T21:25:58.814 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000045235', 1: '00000000000000000000:00000000000000031499', 2: '00000000000000000000:00000000000000051391', 3: '00000000000000000000:00000000000000044038'} 2023-05-31T21:25:58.814 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000045095', 1: '00000000000000000000:00000000000000031499', 2: '00000000000000000000:00000000000000051251', 3: '00000000000000000000:00000000000000044038'} 2023-05-31T21:25:58.815 WARNING:rgw_multi.tests:data of zone a2 behind zone a1: shard=0 master=00000000000000000000:00000000000000045235 target=00000000000000000000:00000000000000045095, shard=2 master=00000000000000000000:00000000000000051391 target=00000000000000000000:00000000000000051251 2023-05-31T21:26:03.816 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-31T21:26:03.817 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-31T21:26:03.817 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-05-31T21:26:03.876 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2023-05-31T21:26:03.876 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2023-05-31T21:26:03.932 INFO:teuthology.orchestra.run.smithi099.stdout:{ 2023-05-31T21:26:03.932 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_status": { 2023-05-31T21:26:03.933 INFO:teuthology.orchestra.run.smithi099.stdout: "info": { 2023-05-31T21:26:03.933 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "sync", 2023-05-31T21:26:03.933 INFO:teuthology.orchestra.run.smithi099.stdout: "num_shards": 4, 2023-05-31T21:26:03.933 INFO:teuthology.orchestra.run.smithi099.stdout: "instance_id": 452136274422638824 2023-05-31T21:26:03.933 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:26:03.933 INFO:teuthology.orchestra.run.smithi099.stdout: "markers": [ 2023-05-31T21:26:03.933 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T21:26:03.934 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 0, 2023-05-31T21:26:03.934 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T21:26:03.934 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "incremental-sync", 2023-05-31T21:26:03.934 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "00000000000000000000:00000000000000045095", 2023-05-31T21:26:03.934 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T21:26:03.934 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2023-05-31T21:26:03.935 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T21:26:03.935 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2023-05-31T21:23:41.891197Z" 2023-05-31T21:26:03.935 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T21:26:03.935 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:26:03.935 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T21:26:03.935 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 1, 2023-05-31T21:26:03.935 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T21:26:03.936 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "incremental-sync", 2023-05-31T21:26:03.936 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "00000000000000000000:00000000000000031499", 2023-05-31T21:26:03.936 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T21:26:03.936 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2023-05-31T21:26:03.936 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T21:26:03.936 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2023-05-31T21:12:40.667485Z" 2023-05-31T21:26:03.937 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T21:26:03.937 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:26:03.937 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T21:26:03.937 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 2, 2023-05-31T21:26:03.937 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T21:26:03.937 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "incremental-sync", 2023-05-31T21:26:03.937 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "00000000000000000000:00000000000000051251", 2023-05-31T21:26:03.938 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T21:26:03.938 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 55, 2023-05-31T21:26:03.938 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T21:26:03.938 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2023-05-31T21:23:41.857089Z" 2023-05-31T21:26:03.938 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T21:26:03.938 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:26:03.938 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T21:26:03.939 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 3, 2023-05-31T21:26:03.939 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T21:26:03.939 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "incremental-sync", 2023-05-31T21:26:03.939 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "00000000000000000000:00000000000000044038", 2023-05-31T21:26:03.939 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T21:26:03.939 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 9, 2023-05-31T21:26:03.939 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T21:26:03.940 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2023-05-31T21:21:35.912562Z" 2023-05-31T21:26:03.940 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T21:26:03.940 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T21:26:03.940 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2023-05-31T21:26:03.940 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:26:03.941 INFO:teuthology.orchestra.run.smithi099.stdout: "full_sync": { 2023-05-31T21:26:03.941 INFO:teuthology.orchestra.run.smithi099.stdout: "total": 64, 2023-05-31T21:26:03.941 INFO:teuthology.orchestra.run.smithi099.stdout: "complete": 64 2023-05-31T21:26:03.941 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:26:03.941 INFO:teuthology.orchestra.run.smithi099.stdout: "current_time": "2023-05-31T21:26:03Z" 2023-05-31T21:26:03.941 INFO:teuthology.orchestra.run.smithi099.stdout:} 2023-05-31T21:26:03.942 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 452136274422638824 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000045095", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-05-31T21:23:41.891197Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000031499", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-05-31T21:12:40.667485Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000051251", "next_step_marker": "", "total_entries": 55, "pos": 0, "timestamp": "2023-05-31T21:23:41.857089Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000044038", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2023-05-31T21:21:35.912562Z" } } ] }, "full_sync": { "total": 64, "complete": 64 }, "current_time": "2023-05-31T21:26:03Z" } 2023-05-31T21:26:03.943 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000045095', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-05-31T21:23:41.891197Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000031499', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-05-31T21:12:40.667485Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000051251', 'next_step_marker': '', 'total_entries': 55, 'pos': 0, 'timestamp': '2023-05-31T21:23:41.857089Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000044038', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2023-05-31T21:21:35.912562Z'}}] 2023-05-31T21:26:03.943 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000045235', 1: '00000000000000000000:00000000000000031499', 2: '00000000000000000000:00000000000000051391', 3: '00000000000000000000:00000000000000044038'} 2023-05-31T21:26:03.943 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000045095', 1: '00000000000000000000:00000000000000031499', 2: '00000000000000000000:00000000000000051251', 3: '00000000000000000000:00000000000000044038'} 2023-05-31T21:26:03.944 WARNING:rgw_multi.tests:data of zone a2 behind zone a1: shard=0 master=00000000000000000000:00000000000000045235 target=00000000000000000000:00000000000000045095, shard=2 master=00000000000000000000:00000000000000051391 target=00000000000000000000:00000000000000051251 2023-05-31T21:26:08.945 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-31T21:26:08.946 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-31T21:26:08.946 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-05-31T21:26:09.006 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2023-05-31T21:26:09.006 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2023-05-31T21:26:09.072 INFO:teuthology.orchestra.run.smithi099.stdout:{ 2023-05-31T21:26:09.073 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_status": { 2023-05-31T21:26:09.073 INFO:teuthology.orchestra.run.smithi099.stdout: "info": { 2023-05-31T21:26:09.073 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "sync", 2023-05-31T21:26:09.073 INFO:teuthology.orchestra.run.smithi099.stdout: "num_shards": 4, 2023-05-31T21:26:09.073 INFO:teuthology.orchestra.run.smithi099.stdout: "instance_id": 452136274422638824 2023-05-31T21:26:09.074 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:26:09.074 INFO:teuthology.orchestra.run.smithi099.stdout: "markers": [ 2023-05-31T21:26:09.074 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T21:26:09.074 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 0, 2023-05-31T21:26:09.074 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T21:26:09.074 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "incremental-sync", 2023-05-31T21:26:09.075 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "00000000000000000000:00000000000000045095", 2023-05-31T21:26:09.075 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T21:26:09.075 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2023-05-31T21:26:09.075 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T21:26:09.075 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2023-05-31T21:23:41.891197Z" 2023-05-31T21:26:09.075 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T21:26:09.075 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:26:09.076 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T21:26:09.076 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 1, 2023-05-31T21:26:09.076 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T21:26:09.076 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "incremental-sync", 2023-05-31T21:26:09.076 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "00000000000000000000:00000000000000031499", 2023-05-31T21:26:09.076 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T21:26:09.077 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2023-05-31T21:26:09.077 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T21:26:09.077 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2023-05-31T21:12:40.667485Z" 2023-05-31T21:26:09.077 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T21:26:09.077 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:26:09.077 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T21:26:09.077 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 2, 2023-05-31T21:26:09.078 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T21:26:09.078 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "incremental-sync", 2023-05-31T21:26:09.078 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "00000000000000000000:00000000000000051251", 2023-05-31T21:26:09.078 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T21:26:09.078 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 55, 2023-05-31T21:26:09.078 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T21:26:09.078 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2023-05-31T21:23:41.857089Z" 2023-05-31T21:26:09.079 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T21:26:09.079 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:26:09.079 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T21:26:09.079 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 3, 2023-05-31T21:26:09.079 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T21:26:09.079 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "incremental-sync", 2023-05-31T21:26:09.079 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "00000000000000000000:00000000000000044038", 2023-05-31T21:26:09.080 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T21:26:09.080 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 9, 2023-05-31T21:26:09.080 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T21:26:09.080 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2023-05-31T21:21:35.912562Z" 2023-05-31T21:26:09.081 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T21:26:09.082 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T21:26:09.082 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2023-05-31T21:26:09.082 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:26:09.082 INFO:teuthology.orchestra.run.smithi099.stdout: "full_sync": { 2023-05-31T21:26:09.082 INFO:teuthology.orchestra.run.smithi099.stdout: "total": 64, 2023-05-31T21:26:09.082 INFO:teuthology.orchestra.run.smithi099.stdout: "complete": 64 2023-05-31T21:26:09.083 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:26:09.083 INFO:teuthology.orchestra.run.smithi099.stdout: "current_time": "2023-05-31T21:26:09Z" 2023-05-31T21:26:09.083 INFO:teuthology.orchestra.run.smithi099.stdout:} 2023-05-31T21:26:09.084 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 452136274422638824 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000045095", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-05-31T21:23:41.891197Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000031499", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-05-31T21:12:40.667485Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000051251", "next_step_marker": "", "total_entries": 55, "pos": 0, "timestamp": "2023-05-31T21:23:41.857089Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000044038", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2023-05-31T21:21:35.912562Z" } } ] }, "full_sync": { "total": 64, "complete": 64 }, "current_time": "2023-05-31T21:26:09Z" } 2023-05-31T21:26:09.085 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000045095', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-05-31T21:23:41.891197Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000031499', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-05-31T21:12:40.667485Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000051251', 'next_step_marker': '', 'total_entries': 55, 'pos': 0, 'timestamp': '2023-05-31T21:23:41.857089Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000044038', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2023-05-31T21:21:35.912562Z'}}] 2023-05-31T21:26:09.085 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000045235', 1: '00000000000000000000:00000000000000031499', 2: '00000000000000000000:00000000000000051391', 3: '00000000000000000000:00000000000000044038'} 2023-05-31T21:26:09.085 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000045095', 1: '00000000000000000000:00000000000000031499', 2: '00000000000000000000:00000000000000051251', 3: '00000000000000000000:00000000000000044038'} 2023-05-31T21:26:09.086 WARNING:rgw_multi.tests:data of zone a2 behind zone a1: shard=0 master=00000000000000000000:00000000000000045235 target=00000000000000000000:00000000000000045095, shard=2 master=00000000000000000000:00000000000000051391 target=00000000000000000000:00000000000000051251 2023-05-31T21:26:14.087 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-31T21:26:14.088 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-31T21:26:14.089 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-05-31T21:26:14.151 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2023-05-31T21:26:14.151 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2023-05-31T21:26:14.204 INFO:teuthology.orchestra.run.smithi099.stdout:{ 2023-05-31T21:26:14.204 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_status": { 2023-05-31T21:26:14.205 INFO:teuthology.orchestra.run.smithi099.stdout: "info": { 2023-05-31T21:26:14.205 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "sync", 2023-05-31T21:26:14.205 INFO:teuthology.orchestra.run.smithi099.stdout: "num_shards": 4, 2023-05-31T21:26:14.205 INFO:teuthology.orchestra.run.smithi099.stdout: "instance_id": 452136274422638824 2023-05-31T21:26:14.205 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:26:14.205 INFO:teuthology.orchestra.run.smithi099.stdout: "markers": [ 2023-05-31T21:26:14.205 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T21:26:14.206 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 0, 2023-05-31T21:26:14.206 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T21:26:14.206 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "incremental-sync", 2023-05-31T21:26:14.206 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "00000000000000000000:00000000000000045095", 2023-05-31T21:26:14.206 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T21:26:14.206 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2023-05-31T21:26:14.206 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T21:26:14.207 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2023-05-31T21:23:41.891197Z" 2023-05-31T21:26:14.207 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T21:26:14.207 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:26:14.207 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T21:26:14.207 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 1, 2023-05-31T21:26:14.207 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T21:26:14.207 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "incremental-sync", 2023-05-31T21:26:14.208 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "00000000000000000000:00000000000000031499", 2023-05-31T21:26:14.208 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T21:26:14.208 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2023-05-31T21:26:14.208 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T21:26:14.208 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2023-05-31T21:12:40.667485Z" 2023-05-31T21:26:14.208 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T21:26:14.209 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:26:14.209 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T21:26:14.209 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 2, 2023-05-31T21:26:14.209 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T21:26:14.209 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "incremental-sync", 2023-05-31T21:26:14.210 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "00000000000000000000:00000000000000051251", 2023-05-31T21:26:14.210 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T21:26:14.210 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 55, 2023-05-31T21:26:14.210 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T21:26:14.210 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2023-05-31T21:23:41.857089Z" 2023-05-31T21:26:14.210 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T21:26:14.210 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:26:14.211 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T21:26:14.211 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 3, 2023-05-31T21:26:14.211 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T21:26:14.211 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "incremental-sync", 2023-05-31T21:26:14.211 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "00000000000000000000:00000000000000044038", 2023-05-31T21:26:14.211 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T21:26:14.211 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 9, 2023-05-31T21:26:14.212 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T21:26:14.212 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2023-05-31T21:21:35.912562Z" 2023-05-31T21:26:14.212 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T21:26:14.212 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T21:26:14.212 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2023-05-31T21:26:14.212 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:26:14.213 INFO:teuthology.orchestra.run.smithi099.stdout: "full_sync": { 2023-05-31T21:26:14.213 INFO:teuthology.orchestra.run.smithi099.stdout: "total": 64, 2023-05-31T21:26:14.213 INFO:teuthology.orchestra.run.smithi099.stdout: "complete": 64 2023-05-31T21:26:14.213 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:26:14.213 INFO:teuthology.orchestra.run.smithi099.stdout: "current_time": "2023-05-31T21:26:14Z" 2023-05-31T21:26:14.213 INFO:teuthology.orchestra.run.smithi099.stdout:} 2023-05-31T21:26:14.214 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 452136274422638824 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000045095", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-05-31T21:23:41.891197Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000031499", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-05-31T21:12:40.667485Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000051251", "next_step_marker": "", "total_entries": 55, "pos": 0, "timestamp": "2023-05-31T21:23:41.857089Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000044038", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2023-05-31T21:21:35.912562Z" } } ] }, "full_sync": { "total": 64, "complete": 64 }, "current_time": "2023-05-31T21:26:14Z" } 2023-05-31T21:26:14.215 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000045095', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-05-31T21:23:41.891197Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000031499', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-05-31T21:12:40.667485Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000051251', 'next_step_marker': '', 'total_entries': 55, 'pos': 0, 'timestamp': '2023-05-31T21:23:41.857089Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000044038', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2023-05-31T21:21:35.912562Z'}}] 2023-05-31T21:26:14.215 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000045235', 1: '00000000000000000000:00000000000000031499', 2: '00000000000000000000:00000000000000051391', 3: '00000000000000000000:00000000000000044038'} 2023-05-31T21:26:14.216 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000045095', 1: '00000000000000000000:00000000000000031499', 2: '00000000000000000000:00000000000000051251', 3: '00000000000000000000:00000000000000044038'} 2023-05-31T21:26:14.216 WARNING:rgw_multi.tests:data of zone a2 behind zone a1: shard=0 master=00000000000000000000:00000000000000045235 target=00000000000000000000:00000000000000045095, shard=2 master=00000000000000000000:00000000000000051391 target=00000000000000000000:00000000000000051251 2023-05-31T21:26:19.217 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-31T21:26:19.218 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-31T21:26:19.218 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-05-31T21:26:19.277 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2023-05-31T21:26:19.278 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2023-05-31T21:26:19.354 INFO:teuthology.orchestra.run.smithi099.stdout:{ 2023-05-31T21:26:19.354 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_status": { 2023-05-31T21:26:19.355 INFO:teuthology.orchestra.run.smithi099.stdout: "info": { 2023-05-31T21:26:19.355 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "sync", 2023-05-31T21:26:19.355 INFO:teuthology.orchestra.run.smithi099.stdout: "num_shards": 4, 2023-05-31T21:26:19.355 INFO:teuthology.orchestra.run.smithi099.stdout: "instance_id": 452136274422638824 2023-05-31T21:26:19.355 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:26:19.356 INFO:teuthology.orchestra.run.smithi099.stdout: "markers": [ 2023-05-31T21:26:19.356 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T21:26:19.356 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 0, 2023-05-31T21:26:19.356 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T21:26:19.356 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "incremental-sync", 2023-05-31T21:26:19.357 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "00000000000000000000:00000000000000045095", 2023-05-31T21:26:19.357 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T21:26:19.357 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2023-05-31T21:26:19.357 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T21:26:19.357 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2023-05-31T21:23:41.891197Z" 2023-05-31T21:26:19.357 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T21:26:19.358 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:26:19.358 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T21:26:19.358 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 1, 2023-05-31T21:26:19.358 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T21:26:19.358 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "incremental-sync", 2023-05-31T21:26:19.358 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "00000000000000000000:00000000000000031499", 2023-05-31T21:26:19.359 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T21:26:19.359 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2023-05-31T21:26:19.359 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T21:26:19.359 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2023-05-31T21:12:40.667485Z" 2023-05-31T21:26:19.359 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T21:26:19.359 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:26:19.360 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T21:26:19.360 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 2, 2023-05-31T21:26:19.360 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T21:26:19.360 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "incremental-sync", 2023-05-31T21:26:19.361 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "00000000000000000000:00000000000000051251", 2023-05-31T21:26:19.361 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T21:26:19.361 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 55, 2023-05-31T21:26:19.361 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T21:26:19.361 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2023-05-31T21:23:41.857089Z" 2023-05-31T21:26:19.361 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T21:26:19.362 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:26:19.362 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T21:26:19.362 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 3, 2023-05-31T21:26:19.362 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T21:26:19.362 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "incremental-sync", 2023-05-31T21:26:19.362 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "00000000000000000000:00000000000000044038", 2023-05-31T21:26:19.362 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T21:26:19.363 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 9, 2023-05-31T21:26:19.363 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T21:26:19.363 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2023-05-31T21:21:35.912562Z" 2023-05-31T21:26:19.363 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T21:26:19.364 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T21:26:19.364 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2023-05-31T21:26:19.364 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:26:19.365 INFO:teuthology.orchestra.run.smithi099.stdout: "full_sync": { 2023-05-31T21:26:19.365 INFO:teuthology.orchestra.run.smithi099.stdout: "total": 64, 2023-05-31T21:26:19.365 INFO:teuthology.orchestra.run.smithi099.stdout: "complete": 64 2023-05-31T21:26:19.366 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:26:19.366 INFO:teuthology.orchestra.run.smithi099.stdout: "current_time": "2023-05-31T21:26:19Z" 2023-05-31T21:26:19.366 INFO:teuthology.orchestra.run.smithi099.stdout:} 2023-05-31T21:26:19.368 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 452136274422638824 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000045095", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-05-31T21:23:41.891197Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000031499", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-05-31T21:12:40.667485Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000051251", "next_step_marker": "", "total_entries": 55, "pos": 0, "timestamp": "2023-05-31T21:23:41.857089Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000044038", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2023-05-31T21:21:35.912562Z" } } ] }, "full_sync": { "total": 64, "complete": 64 }, "current_time": "2023-05-31T21:26:19Z" } 2023-05-31T21:26:19.369 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000045095', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-05-31T21:23:41.891197Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000031499', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-05-31T21:12:40.667485Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000051251', 'next_step_marker': '', 'total_entries': 55, 'pos': 0, 'timestamp': '2023-05-31T21:23:41.857089Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000044038', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2023-05-31T21:21:35.912562Z'}}] 2023-05-31T21:26:19.370 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000045235', 1: '00000000000000000000:00000000000000031499', 2: '00000000000000000000:00000000000000051391', 3: '00000000000000000000:00000000000000044038'} 2023-05-31T21:26:19.370 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000045095', 1: '00000000000000000000:00000000000000031499', 2: '00000000000000000000:00000000000000051251', 3: '00000000000000000000:00000000000000044038'} 2023-05-31T21:26:19.371 WARNING:rgw_multi.tests:data of zone a2 behind zone a1: shard=0 master=00000000000000000000:00000000000000045235 target=00000000000000000000:00000000000000045095, shard=2 master=00000000000000000000:00000000000000051391 target=00000000000000000000:00000000000000051251 2023-05-31T21:26:24.372 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-31T21:26:24.373 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-31T21:26:24.373 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-05-31T21:26:24.433 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2023-05-31T21:26:24.434 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2023-05-31T21:26:24.485 INFO:teuthology.orchestra.run.smithi099.stdout:{ 2023-05-31T21:26:24.486 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_status": { 2023-05-31T21:26:24.486 INFO:teuthology.orchestra.run.smithi099.stdout: "info": { 2023-05-31T21:26:24.486 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "sync", 2023-05-31T21:26:24.486 INFO:teuthology.orchestra.run.smithi099.stdout: "num_shards": 4, 2023-05-31T21:26:24.487 INFO:teuthology.orchestra.run.smithi099.stdout: "instance_id": 452136274422638824 2023-05-31T21:26:24.487 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:26:24.487 INFO:teuthology.orchestra.run.smithi099.stdout: "markers": [ 2023-05-31T21:26:24.487 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T21:26:24.487 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 0, 2023-05-31T21:26:24.487 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T21:26:24.487 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "incremental-sync", 2023-05-31T21:26:24.488 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "00000000000000000000:00000000000000045095", 2023-05-31T21:26:24.488 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T21:26:24.488 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2023-05-31T21:26:24.488 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T21:26:24.488 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2023-05-31T21:23:41.891197Z" 2023-05-31T21:26:24.488 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T21:26:24.489 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:26:24.489 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T21:26:24.489 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 1, 2023-05-31T21:26:24.489 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T21:26:24.489 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "incremental-sync", 2023-05-31T21:26:24.489 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "00000000000000000000:00000000000000031499", 2023-05-31T21:26:24.490 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T21:26:24.490 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2023-05-31T21:26:24.490 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T21:26:24.490 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2023-05-31T21:12:40.667485Z" 2023-05-31T21:26:24.490 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T21:26:24.490 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:26:24.491 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T21:26:24.491 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 2, 2023-05-31T21:26:24.491 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T21:26:24.492 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "incremental-sync", 2023-05-31T21:26:24.492 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "00000000000000000000:00000000000000051251", 2023-05-31T21:26:24.492 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T21:26:24.492 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 55, 2023-05-31T21:26:24.493 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T21:26:24.493 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2023-05-31T21:23:41.857089Z" 2023-05-31T21:26:24.493 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T21:26:24.494 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:26:24.494 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T21:26:24.494 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 3, 2023-05-31T21:26:24.495 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T21:26:24.495 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "incremental-sync", 2023-05-31T21:26:24.495 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "00000000000000000000:00000000000000044038", 2023-05-31T21:26:24.495 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T21:26:24.496 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 9, 2023-05-31T21:26:24.496 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T21:26:24.496 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2023-05-31T21:21:35.912562Z" 2023-05-31T21:26:24.497 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T21:26:24.497 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T21:26:24.497 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2023-05-31T21:26:24.498 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:26:24.498 INFO:teuthology.orchestra.run.smithi099.stdout: "full_sync": { 2023-05-31T21:26:24.498 INFO:teuthology.orchestra.run.smithi099.stdout: "total": 64, 2023-05-31T21:26:24.498 INFO:teuthology.orchestra.run.smithi099.stdout: "complete": 64 2023-05-31T21:26:24.498 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:26:24.498 INFO:teuthology.orchestra.run.smithi099.stdout: "current_time": "2023-05-31T21:26:24Z" 2023-05-31T21:26:24.499 INFO:teuthology.orchestra.run.smithi099.stdout:} 2023-05-31T21:26:24.499 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 452136274422638824 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000045095", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-05-31T21:23:41.891197Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000031499", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-05-31T21:12:40.667485Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000051251", "next_step_marker": "", "total_entries": 55, "pos": 0, "timestamp": "2023-05-31T21:23:41.857089Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000044038", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2023-05-31T21:21:35.912562Z" } } ] }, "full_sync": { "total": 64, "complete": 64 }, "current_time": "2023-05-31T21:26:24Z" } 2023-05-31T21:26:24.500 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000045095', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-05-31T21:23:41.891197Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000031499', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-05-31T21:12:40.667485Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000051251', 'next_step_marker': '', 'total_entries': 55, 'pos': 0, 'timestamp': '2023-05-31T21:23:41.857089Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000044038', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2023-05-31T21:21:35.912562Z'}}] 2023-05-31T21:26:24.501 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000045235', 1: '00000000000000000000:00000000000000031499', 2: '00000000000000000000:00000000000000051391', 3: '00000000000000000000:00000000000000044038'} 2023-05-31T21:26:24.501 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000045095', 1: '00000000000000000000:00000000000000031499', 2: '00000000000000000000:00000000000000051251', 3: '00000000000000000000:00000000000000044038'} 2023-05-31T21:26:24.502 WARNING:rgw_multi.tests:data of zone a2 behind zone a1: shard=0 master=00000000000000000000:00000000000000045235 target=00000000000000000000:00000000000000045095, shard=2 master=00000000000000000000:00000000000000051391 target=00000000000000000000:00000000000000051251 2023-05-31T21:26:29.503 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-31T21:26:29.504 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-31T21:26:29.505 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-05-31T21:26:29.566 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2023-05-31T21:26:29.567 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2023-05-31T21:26:29.640 INFO:teuthology.orchestra.run.smithi099.stdout:{ 2023-05-31T21:26:29.641 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_status": { 2023-05-31T21:26:29.641 INFO:teuthology.orchestra.run.smithi099.stdout: "info": { 2023-05-31T21:26:29.641 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "sync", 2023-05-31T21:26:29.641 INFO:teuthology.orchestra.run.smithi099.stdout: "num_shards": 4, 2023-05-31T21:26:29.641 INFO:teuthology.orchestra.run.smithi099.stdout: "instance_id": 452136274422638824 2023-05-31T21:26:29.641 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:26:29.641 INFO:teuthology.orchestra.run.smithi099.stdout: "markers": [ 2023-05-31T21:26:29.642 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T21:26:29.642 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 0, 2023-05-31T21:26:29.642 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T21:26:29.642 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "incremental-sync", 2023-05-31T21:26:29.642 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "00000000000000000000:00000000000000045095", 2023-05-31T21:26:29.642 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T21:26:29.643 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2023-05-31T21:26:29.643 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T21:26:29.643 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2023-05-31T21:23:41.891197Z" 2023-05-31T21:26:29.643 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T21:26:29.643 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:26:29.643 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T21:26:29.643 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 1, 2023-05-31T21:26:29.644 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T21:26:29.644 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "incremental-sync", 2023-05-31T21:26:29.644 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "00000000000000000000:00000000000000031499", 2023-05-31T21:26:29.644 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T21:26:29.644 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2023-05-31T21:26:29.644 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T21:26:29.644 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2023-05-31T21:12:40.667485Z" 2023-05-31T21:26:29.645 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T21:26:29.645 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:26:29.645 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T21:26:29.645 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 2, 2023-05-31T21:26:29.645 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T21:26:29.645 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "incremental-sync", 2023-05-31T21:26:29.646 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "00000000000000000000:00000000000000051251", 2023-05-31T21:26:29.646 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T21:26:29.646 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 55, 2023-05-31T21:26:29.646 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T21:26:29.646 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2023-05-31T21:23:41.857089Z" 2023-05-31T21:26:29.646 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T21:26:29.646 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:26:29.647 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T21:26:29.647 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 3, 2023-05-31T21:26:29.647 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T21:26:29.647 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "incremental-sync", 2023-05-31T21:26:29.647 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "00000000000000000000:00000000000000044038", 2023-05-31T21:26:29.647 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T21:26:29.647 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 9, 2023-05-31T21:26:29.648 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T21:26:29.648 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2023-05-31T21:21:35.912562Z" 2023-05-31T21:26:29.648 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T21:26:29.648 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T21:26:29.648 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2023-05-31T21:26:29.648 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:26:29.649 INFO:teuthology.orchestra.run.smithi099.stdout: "full_sync": { 2023-05-31T21:26:29.649 INFO:teuthology.orchestra.run.smithi099.stdout: "total": 64, 2023-05-31T21:26:29.649 INFO:teuthology.orchestra.run.smithi099.stdout: "complete": 64 2023-05-31T21:26:29.649 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:26:29.649 INFO:teuthology.orchestra.run.smithi099.stdout: "current_time": "2023-05-31T21:26:29Z" 2023-05-31T21:26:29.649 INFO:teuthology.orchestra.run.smithi099.stdout:} 2023-05-31T21:26:29.651 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 452136274422638824 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000045095", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-05-31T21:23:41.891197Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000031499", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-05-31T21:12:40.667485Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000051251", "next_step_marker": "", "total_entries": 55, "pos": 0, "timestamp": "2023-05-31T21:23:41.857089Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000044038", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2023-05-31T21:21:35.912562Z" } } ] }, "full_sync": { "total": 64, "complete": 64 }, "current_time": "2023-05-31T21:26:29Z" } 2023-05-31T21:26:29.651 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000045095', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-05-31T21:23:41.891197Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000031499', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-05-31T21:12:40.667485Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000051251', 'next_step_marker': '', 'total_entries': 55, 'pos': 0, 'timestamp': '2023-05-31T21:23:41.857089Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000044038', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2023-05-31T21:21:35.912562Z'}}] 2023-05-31T21:26:29.652 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000045235', 1: '00000000000000000000:00000000000000031499', 2: '00000000000000000000:00000000000000051391', 3: '00000000000000000000:00000000000000044038'} 2023-05-31T21:26:29.652 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000045095', 1: '00000000000000000000:00000000000000031499', 2: '00000000000000000000:00000000000000051251', 3: '00000000000000000000:00000000000000044038'} 2023-05-31T21:26:29.652 WARNING:rgw_multi.tests:data of zone a2 behind zone a1: shard=0 master=00000000000000000000:00000000000000045235 target=00000000000000000000:00000000000000045095, shard=2 master=00000000000000000000:00000000000000051391 target=00000000000000000000:00000000000000051251 2023-05-31T21:26:34.654 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-31T21:26:34.654 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-31T21:26:34.655 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-05-31T21:26:34.715 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2023-05-31T21:26:34.715 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2023-05-31T21:26:34.790 INFO:teuthology.orchestra.run.smithi099.stdout:{ 2023-05-31T21:26:34.791 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_status": { 2023-05-31T21:26:34.791 INFO:teuthology.orchestra.run.smithi099.stdout: "info": { 2023-05-31T21:26:34.792 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "sync", 2023-05-31T21:26:34.792 INFO:teuthology.orchestra.run.smithi099.stdout: "num_shards": 4, 2023-05-31T21:26:34.792 INFO:teuthology.orchestra.run.smithi099.stdout: "instance_id": 452136274422638824 2023-05-31T21:26:34.793 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:26:34.793 INFO:teuthology.orchestra.run.smithi099.stdout: "markers": [ 2023-05-31T21:26:34.793 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T21:26:34.794 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 0, 2023-05-31T21:26:34.794 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T21:26:34.794 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "incremental-sync", 2023-05-31T21:26:34.794 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "00000000000000000000:00000000000000045095", 2023-05-31T21:26:34.794 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T21:26:34.795 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2023-05-31T21:26:34.795 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T21:26:34.795 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2023-05-31T21:23:41.891197Z" 2023-05-31T21:26:34.796 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T21:26:34.797 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:26:34.797 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T21:26:34.797 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 1, 2023-05-31T21:26:34.798 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T21:26:34.798 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "incremental-sync", 2023-05-31T21:26:34.798 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "00000000000000000000:00000000000000031499", 2023-05-31T21:26:34.799 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T21:26:34.799 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2023-05-31T21:26:34.800 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T21:26:34.800 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2023-05-31T21:12:40.667485Z" 2023-05-31T21:26:34.800 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T21:26:34.801 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:26:34.801 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T21:26:34.802 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 2, 2023-05-31T21:26:34.802 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T21:26:34.802 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "incremental-sync", 2023-05-31T21:26:34.803 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "00000000000000000000:00000000000000051251", 2023-05-31T21:26:34.803 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T21:26:34.803 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 55, 2023-05-31T21:26:34.803 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T21:26:34.804 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2023-05-31T21:23:41.857089Z" 2023-05-31T21:26:34.804 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T21:26:34.804 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:26:34.804 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T21:26:34.804 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 3, 2023-05-31T21:26:34.804 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T21:26:34.805 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "incremental-sync", 2023-05-31T21:26:34.805 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "00000000000000000000:00000000000000044038", 2023-05-31T21:26:34.805 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T21:26:34.805 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 9, 2023-05-31T21:26:34.805 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T21:26:34.805 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2023-05-31T21:21:35.912562Z" 2023-05-31T21:26:34.805 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T21:26:34.806 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T21:26:34.806 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2023-05-31T21:26:34.806 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:26:34.806 INFO:teuthology.orchestra.run.smithi099.stdout: "full_sync": { 2023-05-31T21:26:34.806 INFO:teuthology.orchestra.run.smithi099.stdout: "total": 64, 2023-05-31T21:26:34.806 INFO:teuthology.orchestra.run.smithi099.stdout: "complete": 64 2023-05-31T21:26:34.807 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:26:34.807 INFO:teuthology.orchestra.run.smithi099.stdout: "current_time": "2023-05-31T21:26:34Z" 2023-05-31T21:26:34.807 INFO:teuthology.orchestra.run.smithi099.stdout:} 2023-05-31T21:26:34.809 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 452136274422638824 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000045095", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-05-31T21:23:41.891197Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000031499", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-05-31T21:12:40.667485Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000051251", "next_step_marker": "", "total_entries": 55, "pos": 0, "timestamp": "2023-05-31T21:23:41.857089Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000044038", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2023-05-31T21:21:35.912562Z" } } ] }, "full_sync": { "total": 64, "complete": 64 }, "current_time": "2023-05-31T21:26:34Z" } 2023-05-31T21:26:34.810 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000045095', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-05-31T21:23:41.891197Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000031499', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-05-31T21:12:40.667485Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000051251', 'next_step_marker': '', 'total_entries': 55, 'pos': 0, 'timestamp': '2023-05-31T21:23:41.857089Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000044038', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2023-05-31T21:21:35.912562Z'}}] 2023-05-31T21:26:34.810 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000045235', 1: '00000000000000000000:00000000000000031499', 2: '00000000000000000000:00000000000000051391', 3: '00000000000000000000:00000000000000044038'} 2023-05-31T21:26:34.811 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000045095', 1: '00000000000000000000:00000000000000031499', 2: '00000000000000000000:00000000000000051251', 3: '00000000000000000000:00000000000000044038'} 2023-05-31T21:26:34.812 WARNING:rgw_multi.tests:data of zone a2 behind zone a1: shard=0 master=00000000000000000000:00000000000000045235 target=00000000000000000000:00000000000000045095, shard=2 master=00000000000000000000:00000000000000051391 target=00000000000000000000:00000000000000051251 2023-05-31T21:26:39.813 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-31T21:26:39.813 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-31T21:26:39.814 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-05-31T21:26:39.874 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2023-05-31T21:26:39.874 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2023-05-31T21:26:39.932 INFO:teuthology.orchestra.run.smithi099.stdout:{ 2023-05-31T21:26:39.932 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_status": { 2023-05-31T21:26:39.933 INFO:teuthology.orchestra.run.smithi099.stdout: "info": { 2023-05-31T21:26:39.933 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "sync", 2023-05-31T21:26:39.933 INFO:teuthology.orchestra.run.smithi099.stdout: "num_shards": 4, 2023-05-31T21:26:39.934 INFO:teuthology.orchestra.run.smithi099.stdout: "instance_id": 452136274422638824 2023-05-31T21:26:39.934 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:26:39.934 INFO:teuthology.orchestra.run.smithi099.stdout: "markers": [ 2023-05-31T21:26:39.935 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T21:26:39.935 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 0, 2023-05-31T21:26:39.935 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T21:26:39.936 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "incremental-sync", 2023-05-31T21:26:39.936 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "00000000000000000000:00000000000000045095", 2023-05-31T21:26:39.936 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T21:26:39.936 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2023-05-31T21:26:39.937 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T21:26:39.937 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2023-05-31T21:23:41.891197Z" 2023-05-31T21:26:39.937 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T21:26:39.938 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:26:39.938 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T21:26:39.938 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 1, 2023-05-31T21:26:39.939 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T21:26:39.939 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "incremental-sync", 2023-05-31T21:26:39.939 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "00000000000000000000:00000000000000031499", 2023-05-31T21:26:39.940 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T21:26:39.942 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2023-05-31T21:26:39.942 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T21:26:39.942 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2023-05-31T21:12:40.667485Z" 2023-05-31T21:26:39.943 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T21:26:39.943 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:26:39.943 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T21:26:39.944 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 2, 2023-05-31T21:26:39.944 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T21:26:39.944 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "incremental-sync", 2023-05-31T21:26:39.945 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "00000000000000000000:00000000000000051251", 2023-05-31T21:26:39.945 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T21:26:39.945 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 55, 2023-05-31T21:26:39.945 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T21:26:39.946 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2023-05-31T21:23:41.857089Z" 2023-05-31T21:26:39.946 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T21:26:39.947 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:26:39.947 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T21:26:39.948 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 3, 2023-05-31T21:26:39.949 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T21:26:39.952 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "incremental-sync", 2023-05-31T21:26:39.952 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "00000000000000000000:00000000000000044038", 2023-05-31T21:26:39.952 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T21:26:39.953 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 9, 2023-05-31T21:26:39.953 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T21:26:39.953 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2023-05-31T21:21:35.912562Z" 2023-05-31T21:26:39.954 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T21:26:39.954 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T21:26:39.954 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2023-05-31T21:26:39.955 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:26:39.955 INFO:teuthology.orchestra.run.smithi099.stdout: "full_sync": { 2023-05-31T21:26:39.955 INFO:teuthology.orchestra.run.smithi099.stdout: "total": 64, 2023-05-31T21:26:39.955 INFO:teuthology.orchestra.run.smithi099.stdout: "complete": 64 2023-05-31T21:26:39.956 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:26:39.956 INFO:teuthology.orchestra.run.smithi099.stdout: "current_time": "2023-05-31T21:26:39Z" 2023-05-31T21:26:39.956 INFO:teuthology.orchestra.run.smithi099.stdout:} 2023-05-31T21:26:39.958 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 452136274422638824 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000045095", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-05-31T21:23:41.891197Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000031499", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-05-31T21:12:40.667485Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000051251", "next_step_marker": "", "total_entries": 55, "pos": 0, "timestamp": "2023-05-31T21:23:41.857089Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000044038", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2023-05-31T21:21:35.912562Z" } } ] }, "full_sync": { "total": 64, "complete": 64 }, "current_time": "2023-05-31T21:26:39Z" } 2023-05-31T21:26:39.959 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000045095', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-05-31T21:23:41.891197Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000031499', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-05-31T21:12:40.667485Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000051251', 'next_step_marker': '', 'total_entries': 55, 'pos': 0, 'timestamp': '2023-05-31T21:23:41.857089Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000044038', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2023-05-31T21:21:35.912562Z'}}] 2023-05-31T21:26:39.960 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000045235', 1: '00000000000000000000:00000000000000031499', 2: '00000000000000000000:00000000000000051391', 3: '00000000000000000000:00000000000000044038'} 2023-05-31T21:26:39.961 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000045095', 1: '00000000000000000000:00000000000000031499', 2: '00000000000000000000:00000000000000051251', 3: '00000000000000000000:00000000000000044038'} 2023-05-31T21:26:39.961 WARNING:rgw_multi.tests:data of zone a2 behind zone a1: shard=0 master=00000000000000000000:00000000000000045235 target=00000000000000000000:00000000000000045095, shard=2 master=00000000000000000000:00000000000000051391 target=00000000000000000000:00000000000000051251 2023-05-31T21:26:44.963 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-31T21:26:44.964 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-31T21:26:44.964 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-05-31T21:26:45.023 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2023-05-31T21:26:45.024 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2023-05-31T21:26:45.086 INFO:teuthology.orchestra.run.smithi099.stdout:{ 2023-05-31T21:26:45.086 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_status": { 2023-05-31T21:26:45.086 INFO:teuthology.orchestra.run.smithi099.stdout: "info": { 2023-05-31T21:26:45.086 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "sync", 2023-05-31T21:26:45.086 INFO:teuthology.orchestra.run.smithi099.stdout: "num_shards": 4, 2023-05-31T21:26:45.087 INFO:teuthology.orchestra.run.smithi099.stdout: "instance_id": 452136274422638824 2023-05-31T21:26:45.087 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:26:45.087 INFO:teuthology.orchestra.run.smithi099.stdout: "markers": [ 2023-05-31T21:26:45.087 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T21:26:45.087 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 0, 2023-05-31T21:26:45.087 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T21:26:45.087 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "incremental-sync", 2023-05-31T21:26:45.088 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "00000000000000000000:00000000000000045095", 2023-05-31T21:26:45.088 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T21:26:45.088 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2023-05-31T21:26:45.088 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T21:26:45.088 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2023-05-31T21:23:41.891197Z" 2023-05-31T21:26:45.088 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T21:26:45.089 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:26:45.089 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T21:26:45.089 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 1, 2023-05-31T21:26:45.089 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T21:26:45.089 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "incremental-sync", 2023-05-31T21:26:45.089 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "00000000000000000000:00000000000000031499", 2023-05-31T21:26:45.090 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T21:26:45.090 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2023-05-31T21:26:45.090 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T21:26:45.090 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2023-05-31T21:12:40.667485Z" 2023-05-31T21:26:45.090 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T21:26:45.090 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:26:45.090 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T21:26:45.091 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 2, 2023-05-31T21:26:45.091 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T21:26:45.091 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "incremental-sync", 2023-05-31T21:26:45.091 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "00000000000000000000:00000000000000051251", 2023-05-31T21:26:45.091 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T21:26:45.091 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 55, 2023-05-31T21:26:45.091 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T21:26:45.092 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2023-05-31T21:23:41.857089Z" 2023-05-31T21:26:45.092 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T21:26:45.092 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:26:45.092 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T21:26:45.092 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 3, 2023-05-31T21:26:45.092 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T21:26:45.092 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "incremental-sync", 2023-05-31T21:26:45.093 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "00000000000000000000:00000000000000044038", 2023-05-31T21:26:45.093 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T21:26:45.093 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 9, 2023-05-31T21:26:45.093 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T21:26:45.093 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2023-05-31T21:21:35.912562Z" 2023-05-31T21:26:45.093 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T21:26:45.093 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T21:26:45.094 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2023-05-31T21:26:45.094 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:26:45.095 INFO:teuthology.orchestra.run.smithi099.stdout: "full_sync": { 2023-05-31T21:26:45.095 INFO:teuthology.orchestra.run.smithi099.stdout: "total": 64, 2023-05-31T21:26:45.095 INFO:teuthology.orchestra.run.smithi099.stdout: "complete": 64 2023-05-31T21:26:45.095 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:26:45.095 INFO:teuthology.orchestra.run.smithi099.stdout: "current_time": "2023-05-31T21:26:45Z" 2023-05-31T21:26:45.096 INFO:teuthology.orchestra.run.smithi099.stdout:} 2023-05-31T21:26:45.097 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 452136274422638824 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000045095", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-05-31T21:23:41.891197Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000031499", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-05-31T21:12:40.667485Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000051251", "next_step_marker": "", "total_entries": 55, "pos": 0, "timestamp": "2023-05-31T21:23:41.857089Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000044038", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2023-05-31T21:21:35.912562Z" } } ] }, "full_sync": { "total": 64, "complete": 64 }, "current_time": "2023-05-31T21:26:45Z" } 2023-05-31T21:26:45.097 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000045095', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-05-31T21:23:41.891197Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000031499', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-05-31T21:12:40.667485Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000051251', 'next_step_marker': '', 'total_entries': 55, 'pos': 0, 'timestamp': '2023-05-31T21:23:41.857089Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000044038', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2023-05-31T21:21:35.912562Z'}}] 2023-05-31T21:26:45.098 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000045235', 1: '00000000000000000000:00000000000000031499', 2: '00000000000000000000:00000000000000051391', 3: '00000000000000000000:00000000000000044038'} 2023-05-31T21:26:45.098 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000045095', 1: '00000000000000000000:00000000000000031499', 2: '00000000000000000000:00000000000000051251', 3: '00000000000000000000:00000000000000044038'} 2023-05-31T21:26:45.098 WARNING:rgw_multi.tests:data of zone a2 behind zone a1: shard=0 master=00000000000000000000:00000000000000045235 target=00000000000000000000:00000000000000045095, shard=2 master=00000000000000000000:00000000000000051391 target=00000000000000000000:00000000000000051251 2023-05-31T21:26:50.099 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-31T21:26:50.100 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-31T21:26:50.101 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-05-31T21:26:50.162 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2023-05-31T21:26:50.163 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2023-05-31T21:26:50.239 INFO:teuthology.orchestra.run.smithi099.stdout:{ 2023-05-31T21:26:50.239 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_status": { 2023-05-31T21:26:50.239 INFO:teuthology.orchestra.run.smithi099.stdout: "info": { 2023-05-31T21:26:50.239 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "sync", 2023-05-31T21:26:50.240 INFO:teuthology.orchestra.run.smithi099.stdout: "num_shards": 4, 2023-05-31T21:26:50.240 INFO:teuthology.orchestra.run.smithi099.stdout: "instance_id": 452136274422638824 2023-05-31T21:26:50.240 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:26:50.240 INFO:teuthology.orchestra.run.smithi099.stdout: "markers": [ 2023-05-31T21:26:50.240 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T21:26:50.240 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 0, 2023-05-31T21:26:50.241 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T21:26:50.241 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "incremental-sync", 2023-05-31T21:26:50.241 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "00000000000000000000:00000000000000045095", 2023-05-31T21:26:50.241 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T21:26:50.241 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2023-05-31T21:26:50.241 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T21:26:50.241 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2023-05-31T21:23:41.891197Z" 2023-05-31T21:26:50.242 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T21:26:50.242 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:26:50.242 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T21:26:50.242 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 1, 2023-05-31T21:26:50.242 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T21:26:50.242 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "incremental-sync", 2023-05-31T21:26:50.242 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "00000000000000000000:00000000000000031499", 2023-05-31T21:26:50.243 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T21:26:50.243 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2023-05-31T21:26:50.243 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T21:26:50.243 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2023-05-31T21:12:40.667485Z" 2023-05-31T21:26:50.243 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T21:26:50.243 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:26:50.243 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T21:26:50.244 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 2, 2023-05-31T21:26:50.244 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T21:26:50.244 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "incremental-sync", 2023-05-31T21:26:50.244 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "00000000000000000000:00000000000000051251", 2023-05-31T21:26:50.244 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T21:26:50.244 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 55, 2023-05-31T21:26:50.245 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T21:26:50.245 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2023-05-31T21:23:41.857089Z" 2023-05-31T21:26:50.245 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T21:26:50.245 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:26:50.245 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T21:26:50.245 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 3, 2023-05-31T21:26:50.245 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T21:26:50.246 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "incremental-sync", 2023-05-31T21:26:50.246 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "00000000000000000000:00000000000000044038", 2023-05-31T21:26:50.246 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T21:26:50.246 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 9, 2023-05-31T21:26:50.246 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T21:26:50.246 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2023-05-31T21:21:35.912562Z" 2023-05-31T21:26:50.247 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T21:26:50.247 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T21:26:50.248 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2023-05-31T21:26:50.248 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:26:50.248 INFO:teuthology.orchestra.run.smithi099.stdout: "full_sync": { 2023-05-31T21:26:50.249 INFO:teuthology.orchestra.run.smithi099.stdout: "total": 64, 2023-05-31T21:26:50.249 INFO:teuthology.orchestra.run.smithi099.stdout: "complete": 64 2023-05-31T21:26:50.249 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:26:50.250 INFO:teuthology.orchestra.run.smithi099.stdout: "current_time": "2023-05-31T21:26:50Z" 2023-05-31T21:26:50.250 INFO:teuthology.orchestra.run.smithi099.stdout:} 2023-05-31T21:26:50.251 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 452136274422638824 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000045095", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-05-31T21:23:41.891197Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000031499", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-05-31T21:12:40.667485Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000051251", "next_step_marker": "", "total_entries": 55, "pos": 0, "timestamp": "2023-05-31T21:23:41.857089Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000044038", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2023-05-31T21:21:35.912562Z" } } ] }, "full_sync": { "total": 64, "complete": 64 }, "current_time": "2023-05-31T21:26:50Z" } 2023-05-31T21:26:50.252 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000045095', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-05-31T21:23:41.891197Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000031499', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-05-31T21:12:40.667485Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000051251', 'next_step_marker': '', 'total_entries': 55, 'pos': 0, 'timestamp': '2023-05-31T21:23:41.857089Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000044038', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2023-05-31T21:21:35.912562Z'}}] 2023-05-31T21:26:50.253 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000045235', 1: '00000000000000000000:00000000000000031499', 2: '00000000000000000000:00000000000000051391', 3: '00000000000000000000:00000000000000044038'} 2023-05-31T21:26:50.253 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000045095', 1: '00000000000000000000:00000000000000031499', 2: '00000000000000000000:00000000000000051251', 3: '00000000000000000000:00000000000000044038'} 2023-05-31T21:26:50.254 WARNING:rgw_multi.tests:data of zone a2 behind zone a1: shard=0 master=00000000000000000000:00000000000000045235 target=00000000000000000000:00000000000000045095, shard=2 master=00000000000000000000:00000000000000051391 target=00000000000000000000:00000000000000051251 2023-05-31T21:26:55.256 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-31T21:26:55.257 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-31T21:26:55.257 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-05-31T21:26:55.319 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2023-05-31T21:26:55.319 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2023-05-31T21:26:55.371 INFO:teuthology.orchestra.run.smithi099.stdout:{ 2023-05-31T21:26:55.372 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_status": { 2023-05-31T21:26:55.372 INFO:teuthology.orchestra.run.smithi099.stdout: "info": { 2023-05-31T21:26:55.372 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "sync", 2023-05-31T21:26:55.372 INFO:teuthology.orchestra.run.smithi099.stdout: "num_shards": 4, 2023-05-31T21:26:55.373 INFO:teuthology.orchestra.run.smithi099.stdout: "instance_id": 452136274422638824 2023-05-31T21:26:55.373 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:26:55.373 INFO:teuthology.orchestra.run.smithi099.stdout: "markers": [ 2023-05-31T21:26:55.373 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T21:26:55.373 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 0, 2023-05-31T21:26:55.373 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T21:26:55.374 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "incremental-sync", 2023-05-31T21:26:55.374 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "00000000000000000000:00000000000000045095", 2023-05-31T21:26:55.374 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T21:26:55.374 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2023-05-31T21:26:55.374 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T21:26:55.374 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2023-05-31T21:23:41.891197Z" 2023-05-31T21:26:55.374 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T21:26:55.375 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:26:55.375 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T21:26:55.375 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 1, 2023-05-31T21:26:55.375 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T21:26:55.375 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "incremental-sync", 2023-05-31T21:26:55.375 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "00000000000000000000:00000000000000031499", 2023-05-31T21:26:55.375 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T21:26:55.376 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2023-05-31T21:26:55.376 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T21:26:55.376 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2023-05-31T21:12:40.667485Z" 2023-05-31T21:26:55.377 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T21:26:55.377 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:26:55.377 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T21:26:55.377 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 2, 2023-05-31T21:26:55.377 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T21:26:55.377 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "incremental-sync", 2023-05-31T21:26:55.378 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "00000000000000000000:00000000000000051251", 2023-05-31T21:26:55.378 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T21:26:55.378 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 55, 2023-05-31T21:26:55.378 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T21:26:55.378 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2023-05-31T21:23:41.857089Z" 2023-05-31T21:26:55.378 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T21:26:55.378 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:26:55.379 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T21:26:55.379 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 3, 2023-05-31T21:26:55.379 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T21:26:55.379 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "incremental-sync", 2023-05-31T21:26:55.379 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "00000000000000000000:00000000000000044038", 2023-05-31T21:26:55.379 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T21:26:55.379 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 9, 2023-05-31T21:26:55.380 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T21:26:55.380 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2023-05-31T21:21:35.912562Z" 2023-05-31T21:26:55.380 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T21:26:55.380 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T21:26:55.380 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2023-05-31T21:26:55.380 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:26:55.381 INFO:teuthology.orchestra.run.smithi099.stdout: "full_sync": { 2023-05-31T21:26:55.381 INFO:teuthology.orchestra.run.smithi099.stdout: "total": 64, 2023-05-31T21:26:55.381 INFO:teuthology.orchestra.run.smithi099.stdout: "complete": 64 2023-05-31T21:26:55.381 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:26:55.382 INFO:teuthology.orchestra.run.smithi099.stdout: "current_time": "2023-05-31T21:26:55Z" 2023-05-31T21:26:55.382 INFO:teuthology.orchestra.run.smithi099.stdout:} 2023-05-31T21:26:55.383 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 452136274422638824 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000045095", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-05-31T21:23:41.891197Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000031499", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-05-31T21:12:40.667485Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000051251", "next_step_marker": "", "total_entries": 55, "pos": 0, "timestamp": "2023-05-31T21:23:41.857089Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000044038", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2023-05-31T21:21:35.912562Z" } } ] }, "full_sync": { "total": 64, "complete": 64 }, "current_time": "2023-05-31T21:26:55Z" } 2023-05-31T21:26:55.383 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000045095', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-05-31T21:23:41.891197Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000031499', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-05-31T21:12:40.667485Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000051251', 'next_step_marker': '', 'total_entries': 55, 'pos': 0, 'timestamp': '2023-05-31T21:23:41.857089Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000044038', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2023-05-31T21:21:35.912562Z'}}] 2023-05-31T21:26:55.384 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000045235', 1: '00000000000000000000:00000000000000031499', 2: '00000000000000000000:00000000000000051391', 3: '00000000000000000000:00000000000000044038'} 2023-05-31T21:26:55.384 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000045095', 1: '00000000000000000000:00000000000000031499', 2: '00000000000000000000:00000000000000051251', 3: '00000000000000000000:00000000000000044038'} 2023-05-31T21:26:55.384 WARNING:rgw_multi.tests:data of zone a2 behind zone a1: shard=0 master=00000000000000000000:00000000000000045235 target=00000000000000000000:00000000000000045095, shard=2 master=00000000000000000000:00000000000000051391 target=00000000000000000000:00000000000000051251 2023-05-31T21:27:00.385 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-31T21:27:00.386 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-31T21:27:00.387 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-05-31T21:27:00.447 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2023-05-31T21:27:00.448 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2023-05-31T21:27:00.520 INFO:teuthology.orchestra.run.smithi099.stdout:{ 2023-05-31T21:27:00.521 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_status": { 2023-05-31T21:27:00.521 INFO:teuthology.orchestra.run.smithi099.stdout: "info": { 2023-05-31T21:27:00.521 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "sync", 2023-05-31T21:27:00.521 INFO:teuthology.orchestra.run.smithi099.stdout: "num_shards": 4, 2023-05-31T21:27:00.521 INFO:teuthology.orchestra.run.smithi099.stdout: "instance_id": 452136274422638824 2023-05-31T21:27:00.522 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:27:00.522 INFO:teuthology.orchestra.run.smithi099.stdout: "markers": [ 2023-05-31T21:27:00.522 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T21:27:00.522 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 0, 2023-05-31T21:27:00.522 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T21:27:00.522 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "incremental-sync", 2023-05-31T21:27:00.523 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "00000000000000000000:00000000000000045095", 2023-05-31T21:27:00.523 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T21:27:00.523 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2023-05-31T21:27:00.523 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T21:27:00.523 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2023-05-31T21:23:41.891197Z" 2023-05-31T21:27:00.523 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T21:27:00.523 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:27:00.524 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T21:27:00.524 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 1, 2023-05-31T21:27:00.524 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T21:27:00.524 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "incremental-sync", 2023-05-31T21:27:00.524 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "00000000000000000000:00000000000000031499", 2023-05-31T21:27:00.524 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T21:27:00.524 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2023-05-31T21:27:00.525 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T21:27:00.525 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2023-05-31T21:12:40.667485Z" 2023-05-31T21:27:00.525 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T21:27:00.525 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:27:00.525 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T21:27:00.525 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 2, 2023-05-31T21:27:00.526 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T21:27:00.526 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "incremental-sync", 2023-05-31T21:27:00.526 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "00000000000000000000:00000000000000051251", 2023-05-31T21:27:00.526 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T21:27:00.526 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 55, 2023-05-31T21:27:00.526 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T21:27:00.526 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2023-05-31T21:23:41.857089Z" 2023-05-31T21:27:00.527 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T21:27:00.527 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:27:00.527 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T21:27:00.527 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 3, 2023-05-31T21:27:00.527 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T21:27:00.527 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "incremental-sync", 2023-05-31T21:27:00.527 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "00000000000000000000:00000000000000044038", 2023-05-31T21:27:00.528 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T21:27:00.528 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 9, 2023-05-31T21:27:00.528 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T21:27:00.528 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2023-05-31T21:21:35.912562Z" 2023-05-31T21:27:00.528 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T21:27:00.528 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T21:27:00.528 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2023-05-31T21:27:00.529 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:27:00.529 INFO:teuthology.orchestra.run.smithi099.stdout: "full_sync": { 2023-05-31T21:27:00.529 INFO:teuthology.orchestra.run.smithi099.stdout: "total": 64, 2023-05-31T21:27:00.529 INFO:teuthology.orchestra.run.smithi099.stdout: "complete": 64 2023-05-31T21:27:00.529 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:27:00.529 INFO:teuthology.orchestra.run.smithi099.stdout: "current_time": "2023-05-31T21:27:00Z" 2023-05-31T21:27:00.529 INFO:teuthology.orchestra.run.smithi099.stdout:} 2023-05-31T21:27:00.530 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 452136274422638824 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000045095", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-05-31T21:23:41.891197Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000031499", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-05-31T21:12:40.667485Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000051251", "next_step_marker": "", "total_entries": 55, "pos": 0, "timestamp": "2023-05-31T21:23:41.857089Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000044038", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2023-05-31T21:21:35.912562Z" } } ] }, "full_sync": { "total": 64, "complete": 64 }, "current_time": "2023-05-31T21:27:00Z" } 2023-05-31T21:27:00.531 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000045095', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-05-31T21:23:41.891197Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000031499', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-05-31T21:12:40.667485Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000051251', 'next_step_marker': '', 'total_entries': 55, 'pos': 0, 'timestamp': '2023-05-31T21:23:41.857089Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000044038', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2023-05-31T21:21:35.912562Z'}}] 2023-05-31T21:27:00.531 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000045235', 1: '00000000000000000000:00000000000000031499', 2: '00000000000000000000:00000000000000051391', 3: '00000000000000000000:00000000000000044038'} 2023-05-31T21:27:00.532 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000045095', 1: '00000000000000000000:00000000000000031499', 2: '00000000000000000000:00000000000000051251', 3: '00000000000000000000:00000000000000044038'} 2023-05-31T21:27:00.532 WARNING:rgw_multi.tests:data of zone a2 behind zone a1: shard=0 master=00000000000000000000:00000000000000045235 target=00000000000000000000:00000000000000045095, shard=2 master=00000000000000000000:00000000000000051391 target=00000000000000000000:00000000000000051251 2023-05-31T21:27:05.533 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-31T21:27:05.534 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-31T21:27:05.535 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-05-31T21:27:05.594 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2023-05-31T21:27:05.594 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2023-05-31T21:27:05.658 INFO:teuthology.orchestra.run.smithi099.stdout:{ 2023-05-31T21:27:05.658 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_status": { 2023-05-31T21:27:05.658 INFO:teuthology.orchestra.run.smithi099.stdout: "info": { 2023-05-31T21:27:05.658 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "sync", 2023-05-31T21:27:05.659 INFO:teuthology.orchestra.run.smithi099.stdout: "num_shards": 4, 2023-05-31T21:27:05.659 INFO:teuthology.orchestra.run.smithi099.stdout: "instance_id": 452136274422638824 2023-05-31T21:27:05.659 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:27:05.659 INFO:teuthology.orchestra.run.smithi099.stdout: "markers": [ 2023-05-31T21:27:05.659 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T21:27:05.659 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 0, 2023-05-31T21:27:05.659 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T21:27:05.660 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "incremental-sync", 2023-05-31T21:27:05.660 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "00000000000000000000:00000000000000045095", 2023-05-31T21:27:05.660 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T21:27:05.660 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2023-05-31T21:27:05.660 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T21:27:05.661 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2023-05-31T21:23:41.891197Z" 2023-05-31T21:27:05.661 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T21:27:05.661 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:27:05.661 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T21:27:05.661 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 1, 2023-05-31T21:27:05.661 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T21:27:05.661 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "incremental-sync", 2023-05-31T21:27:05.662 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "00000000000000000000:00000000000000031499", 2023-05-31T21:27:05.662 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T21:27:05.662 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2023-05-31T21:27:05.662 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T21:27:05.662 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2023-05-31T21:12:40.667485Z" 2023-05-31T21:27:05.662 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T21:27:05.662 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:27:05.663 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T21:27:05.663 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 2, 2023-05-31T21:27:05.663 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T21:27:05.663 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "incremental-sync", 2023-05-31T21:27:05.663 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "00000000000000000000:00000000000000051251", 2023-05-31T21:27:05.663 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T21:27:05.663 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 55, 2023-05-31T21:27:05.664 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T21:27:05.664 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2023-05-31T21:23:41.857089Z" 2023-05-31T21:27:05.664 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T21:27:05.664 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:27:05.664 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T21:27:05.664 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 3, 2023-05-31T21:27:05.665 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T21:27:05.665 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "incremental-sync", 2023-05-31T21:27:05.665 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "00000000000000000000:00000000000000044038", 2023-05-31T21:27:05.665 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T21:27:05.665 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 9, 2023-05-31T21:27:05.665 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T21:27:05.665 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2023-05-31T21:21:35.912562Z" 2023-05-31T21:27:05.666 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T21:27:05.666 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T21:27:05.666 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2023-05-31T21:27:05.666 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:27:05.666 INFO:teuthology.orchestra.run.smithi099.stdout: "full_sync": { 2023-05-31T21:27:05.666 INFO:teuthology.orchestra.run.smithi099.stdout: "total": 64, 2023-05-31T21:27:05.666 INFO:teuthology.orchestra.run.smithi099.stdout: "complete": 64 2023-05-31T21:27:05.667 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:27:05.667 INFO:teuthology.orchestra.run.smithi099.stdout: "current_time": "2023-05-31T21:27:05Z" 2023-05-31T21:27:05.667 INFO:teuthology.orchestra.run.smithi099.stdout:} 2023-05-31T21:27:05.668 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 452136274422638824 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000045095", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-05-31T21:23:41.891197Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000031499", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-05-31T21:12:40.667485Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000051251", "next_step_marker": "", "total_entries": 55, "pos": 0, "timestamp": "2023-05-31T21:23:41.857089Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000044038", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2023-05-31T21:21:35.912562Z" } } ] }, "full_sync": { "total": 64, "complete": 64 }, "current_time": "2023-05-31T21:27:05Z" } 2023-05-31T21:27:05.668 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000045095', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-05-31T21:23:41.891197Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000031499', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-05-31T21:12:40.667485Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000051251', 'next_step_marker': '', 'total_entries': 55, 'pos': 0, 'timestamp': '2023-05-31T21:23:41.857089Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000044038', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2023-05-31T21:21:35.912562Z'}}] 2023-05-31T21:27:05.669 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000045235', 1: '00000000000000000000:00000000000000031499', 2: '00000000000000000000:00000000000000051391', 3: '00000000000000000000:00000000000000044038'} 2023-05-31T21:27:05.669 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000045095', 1: '00000000000000000000:00000000000000031499', 2: '00000000000000000000:00000000000000051251', 3: '00000000000000000000:00000000000000044038'} 2023-05-31T21:27:05.669 WARNING:rgw_multi.tests:data of zone a2 behind zone a1: shard=0 master=00000000000000000000:00000000000000045235 target=00000000000000000000:00000000000000045095, shard=2 master=00000000000000000000:00000000000000051391 target=00000000000000000000:00000000000000051251 2023-05-31T21:27:10.670 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-31T21:27:10.671 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-31T21:27:10.671 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-05-31T21:27:10.730 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2023-05-31T21:27:10.730 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2023-05-31T21:27:10.789 INFO:teuthology.orchestra.run.smithi099.stdout:{ 2023-05-31T21:27:10.789 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_status": { 2023-05-31T21:27:10.790 INFO:teuthology.orchestra.run.smithi099.stdout: "info": { 2023-05-31T21:27:10.790 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "sync", 2023-05-31T21:27:10.790 INFO:teuthology.orchestra.run.smithi099.stdout: "num_shards": 4, 2023-05-31T21:27:10.790 INFO:teuthology.orchestra.run.smithi099.stdout: "instance_id": 452136274422638824 2023-05-31T21:27:10.790 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:27:10.790 INFO:teuthology.orchestra.run.smithi099.stdout: "markers": [ 2023-05-31T21:27:10.791 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T21:27:10.791 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 0, 2023-05-31T21:27:10.791 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T21:27:10.791 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "incremental-sync", 2023-05-31T21:27:10.791 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "00000000000000000000:00000000000000045095", 2023-05-31T21:27:10.791 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T21:27:10.791 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2023-05-31T21:27:10.792 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T21:27:10.794 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2023-05-31T21:23:41.891197Z" 2023-05-31T21:27:10.794 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T21:27:10.794 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:27:10.794 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T21:27:10.794 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 1, 2023-05-31T21:27:10.795 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T21:27:10.795 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "incremental-sync", 2023-05-31T21:27:10.795 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "00000000000000000000:00000000000000031499", 2023-05-31T21:27:10.795 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T21:27:10.795 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2023-05-31T21:27:10.795 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T21:27:10.795 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2023-05-31T21:12:40.667485Z" 2023-05-31T21:27:10.796 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T21:27:10.796 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:27:10.796 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T21:27:10.796 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 2, 2023-05-31T21:27:10.797 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T21:27:10.797 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "incremental-sync", 2023-05-31T21:27:10.797 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "00000000000000000000:00000000000000051251", 2023-05-31T21:27:10.797 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T21:27:10.797 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 55, 2023-05-31T21:27:10.797 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T21:27:10.797 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2023-05-31T21:23:41.857089Z" 2023-05-31T21:27:10.798 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T21:27:10.798 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:27:10.798 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T21:27:10.798 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 3, 2023-05-31T21:27:10.799 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T21:27:10.799 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "incremental-sync", 2023-05-31T21:27:10.799 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "00000000000000000000:00000000000000044038", 2023-05-31T21:27:10.799 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T21:27:10.799 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 9, 2023-05-31T21:27:10.799 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T21:27:10.800 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2023-05-31T21:21:35.912562Z" 2023-05-31T21:27:10.800 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T21:27:10.800 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T21:27:10.800 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2023-05-31T21:27:10.800 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:27:10.800 INFO:teuthology.orchestra.run.smithi099.stdout: "full_sync": { 2023-05-31T21:27:10.801 INFO:teuthology.orchestra.run.smithi099.stdout: "total": 64, 2023-05-31T21:27:10.801 INFO:teuthology.orchestra.run.smithi099.stdout: "complete": 64 2023-05-31T21:27:10.801 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:27:10.801 INFO:teuthology.orchestra.run.smithi099.stdout: "current_time": "2023-05-31T21:27:10Z" 2023-05-31T21:27:10.801 INFO:teuthology.orchestra.run.smithi099.stdout:} 2023-05-31T21:27:10.802 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 452136274422638824 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000045095", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-05-31T21:23:41.891197Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000031499", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-05-31T21:12:40.667485Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000051251", "next_step_marker": "", "total_entries": 55, "pos": 0, "timestamp": "2023-05-31T21:23:41.857089Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000044038", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2023-05-31T21:21:35.912562Z" } } ] }, "full_sync": { "total": 64, "complete": 64 }, "current_time": "2023-05-31T21:27:10Z" } 2023-05-31T21:27:10.803 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000045095', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-05-31T21:23:41.891197Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000031499', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-05-31T21:12:40.667485Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000051251', 'next_step_marker': '', 'total_entries': 55, 'pos': 0, 'timestamp': '2023-05-31T21:23:41.857089Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000044038', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2023-05-31T21:21:35.912562Z'}}] 2023-05-31T21:27:10.803 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000045235', 1: '00000000000000000000:00000000000000031499', 2: '00000000000000000000:00000000000000051391', 3: '00000000000000000000:00000000000000044038'} 2023-05-31T21:27:10.803 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000045095', 1: '00000000000000000000:00000000000000031499', 2: '00000000000000000000:00000000000000051251', 3: '00000000000000000000:00000000000000044038'} 2023-05-31T21:27:10.804 WARNING:rgw_multi.tests:data of zone a2 behind zone a1: shard=0 master=00000000000000000000:00000000000000045235 target=00000000000000000000:00000000000000045095, shard=2 master=00000000000000000000:00000000000000051391 target=00000000000000000000:00000000000000051251 2023-05-31T21:27:15.804 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-31T21:27:15.805 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-31T21:27:15.806 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-05-31T21:27:15.866 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2023-05-31T21:27:15.866 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2023-05-31T21:27:15.939 INFO:teuthology.orchestra.run.smithi099.stdout:{ 2023-05-31T21:27:15.940 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_status": { 2023-05-31T21:27:15.940 INFO:teuthology.orchestra.run.smithi099.stdout: "info": { 2023-05-31T21:27:15.940 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "sync", 2023-05-31T21:27:15.941 INFO:teuthology.orchestra.run.smithi099.stdout: "num_shards": 4, 2023-05-31T21:27:15.941 INFO:teuthology.orchestra.run.smithi099.stdout: "instance_id": 452136274422638824 2023-05-31T21:27:15.941 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:27:15.941 INFO:teuthology.orchestra.run.smithi099.stdout: "markers": [ 2023-05-31T21:27:15.941 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T21:27:15.941 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 0, 2023-05-31T21:27:15.942 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T21:27:15.942 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "incremental-sync", 2023-05-31T21:27:15.942 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "00000000000000000000:00000000000000045095", 2023-05-31T21:27:15.942 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T21:27:15.942 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2023-05-31T21:27:15.942 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T21:27:15.943 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2023-05-31T21:23:41.891197Z" 2023-05-31T21:27:15.943 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T21:27:15.943 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:27:15.943 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T21:27:15.943 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 1, 2023-05-31T21:27:15.943 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T21:27:15.943 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "incremental-sync", 2023-05-31T21:27:15.944 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "00000000000000000000:00000000000000031499", 2023-05-31T21:27:15.944 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T21:27:15.944 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2023-05-31T21:27:15.944 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T21:27:15.944 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2023-05-31T21:12:40.667485Z" 2023-05-31T21:27:15.944 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T21:27:15.945 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:27:15.945 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T21:27:15.945 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 2, 2023-05-31T21:27:15.945 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T21:27:15.945 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "incremental-sync", 2023-05-31T21:27:15.946 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "00000000000000000000:00000000000000051251", 2023-05-31T21:27:15.946 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T21:27:15.946 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 55, 2023-05-31T21:27:15.946 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T21:27:15.946 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2023-05-31T21:23:41.857089Z" 2023-05-31T21:27:15.946 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T21:27:15.947 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:27:15.947 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T21:27:15.947 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 3, 2023-05-31T21:27:15.947 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T21:27:15.947 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "incremental-sync", 2023-05-31T21:27:15.947 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "00000000000000000000:00000000000000044038", 2023-05-31T21:27:15.947 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T21:27:15.948 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 9, 2023-05-31T21:27:15.948 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T21:27:15.948 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2023-05-31T21:21:35.912562Z" 2023-05-31T21:27:15.948 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T21:27:15.948 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T21:27:15.948 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2023-05-31T21:27:15.949 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:27:15.949 INFO:teuthology.orchestra.run.smithi099.stdout: "full_sync": { 2023-05-31T21:27:15.949 INFO:teuthology.orchestra.run.smithi099.stdout: "total": 64, 2023-05-31T21:27:15.949 INFO:teuthology.orchestra.run.smithi099.stdout: "complete": 64 2023-05-31T21:27:15.952 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:27:15.952 INFO:teuthology.orchestra.run.smithi099.stdout: "current_time": "2023-05-31T21:27:15Z" 2023-05-31T21:27:15.953 INFO:teuthology.orchestra.run.smithi099.stdout:} 2023-05-31T21:27:15.954 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 452136274422638824 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000045095", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-05-31T21:23:41.891197Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000031499", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-05-31T21:12:40.667485Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000051251", "next_step_marker": "", "total_entries": 55, "pos": 0, "timestamp": "2023-05-31T21:23:41.857089Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000044038", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2023-05-31T21:21:35.912562Z" } } ] }, "full_sync": { "total": 64, "complete": 64 }, "current_time": "2023-05-31T21:27:15Z" } 2023-05-31T21:27:15.955 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000045095', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-05-31T21:23:41.891197Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000031499', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-05-31T21:12:40.667485Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000051251', 'next_step_marker': '', 'total_entries': 55, 'pos': 0, 'timestamp': '2023-05-31T21:23:41.857089Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000044038', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2023-05-31T21:21:35.912562Z'}}] 2023-05-31T21:27:15.956 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000045235', 1: '00000000000000000000:00000000000000031499', 2: '00000000000000000000:00000000000000051391', 3: '00000000000000000000:00000000000000044038'} 2023-05-31T21:27:15.957 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000045095', 1: '00000000000000000000:00000000000000031499', 2: '00000000000000000000:00000000000000051251', 3: '00000000000000000000:00000000000000044038'} 2023-05-31T21:27:15.957 WARNING:rgw_multi.tests:data of zone a2 behind zone a1: shard=0 master=00000000000000000000:00000000000000045235 target=00000000000000000000:00000000000000045095, shard=2 master=00000000000000000000:00000000000000051391 target=00000000000000000000:00000000000000051251 2023-05-31T21:27:17.172 INFO:tasks.rgw.c2.client.1.smithi112.stdout:2023-05-31T21:27:17.168+0000 7fecb0821700 -1 WARNING: The OSD cluster is overloaded and struggling to complete ops. You need more capacity to serve this level of demand. 2023-05-31T21:27:20.959 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-31T21:27:20.960 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-31T21:27:20.960 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-05-31T21:27:21.020 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2023-05-31T21:27:21.020 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2023-05-31T21:27:21.096 INFO:teuthology.orchestra.run.smithi099.stdout:{ 2023-05-31T21:27:21.097 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_status": { 2023-05-31T21:27:21.097 INFO:teuthology.orchestra.run.smithi099.stdout: "info": { 2023-05-31T21:27:21.097 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "sync", 2023-05-31T21:27:21.098 INFO:teuthology.orchestra.run.smithi099.stdout: "num_shards": 4, 2023-05-31T21:27:21.098 INFO:teuthology.orchestra.run.smithi099.stdout: "instance_id": 452136274422638824 2023-05-31T21:27:21.098 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:27:21.098 INFO:teuthology.orchestra.run.smithi099.stdout: "markers": [ 2023-05-31T21:27:21.099 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T21:27:21.099 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 0, 2023-05-31T21:27:21.099 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T21:27:21.100 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "incremental-sync", 2023-05-31T21:27:21.100 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "00000000000000000000:00000000000000045235", 2023-05-31T21:27:21.100 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T21:27:21.100 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2023-05-31T21:27:21.101 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T21:27:21.101 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2023-05-31T21:25:53.171643Z" 2023-05-31T21:27:21.101 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T21:27:21.101 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:27:21.102 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T21:27:21.102 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 1, 2023-05-31T21:27:21.102 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T21:27:21.103 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "incremental-sync", 2023-05-31T21:27:21.103 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "00000000000000000000:00000000000000031499", 2023-05-31T21:27:21.103 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T21:27:21.103 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2023-05-31T21:27:21.104 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T21:27:21.104 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2023-05-31T21:12:40.667485Z" 2023-05-31T21:27:21.104 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T21:27:21.104 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:27:21.105 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T21:27:21.105 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 2, 2023-05-31T21:27:21.105 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T21:27:21.105 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "incremental-sync", 2023-05-31T21:27:21.106 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "00000000000000000000:00000000000000051391", 2023-05-31T21:27:21.106 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T21:27:21.106 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 55, 2023-05-31T21:27:21.106 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T21:27:21.107 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2023-05-31T21:25:53.141700Z" 2023-05-31T21:27:21.107 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T21:27:21.107 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:27:21.108 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T21:27:21.108 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 3, 2023-05-31T21:27:21.109 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T21:27:21.109 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "incremental-sync", 2023-05-31T21:27:21.109 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "00000000000000000000:00000000000000044038", 2023-05-31T21:27:21.110 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T21:27:21.110 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 9, 2023-05-31T21:27:21.110 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T21:27:21.111 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2023-05-31T21:21:35.912562Z" 2023-05-31T21:27:21.111 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T21:27:21.111 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T21:27:21.111 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2023-05-31T21:27:21.112 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:27:21.112 INFO:teuthology.orchestra.run.smithi099.stdout: "full_sync": { 2023-05-31T21:27:21.112 INFO:teuthology.orchestra.run.smithi099.stdout: "total": 64, 2023-05-31T21:27:21.113 INFO:teuthology.orchestra.run.smithi099.stdout: "complete": 64 2023-05-31T21:27:21.113 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:27:21.113 INFO:teuthology.orchestra.run.smithi099.stdout: "current_time": "2023-05-31T21:27:21Z" 2023-05-31T21:27:21.114 INFO:teuthology.orchestra.run.smithi099.stdout:} 2023-05-31T21:27:21.115 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 452136274422638824 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000045235", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-05-31T21:25:53.171643Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000031499", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-05-31T21:12:40.667485Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000051391", "next_step_marker": "", "total_entries": 55, "pos": 0, "timestamp": "2023-05-31T21:25:53.141700Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000044038", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2023-05-31T21:21:35.912562Z" } } ] }, "full_sync": { "total": 64, "complete": 64 }, "current_time": "2023-05-31T21:27:21Z" } 2023-05-31T21:27:21.116 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000045235', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-05-31T21:25:53.171643Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000031499', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-05-31T21:12:40.667485Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000051391', 'next_step_marker': '', 'total_entries': 55, 'pos': 0, 'timestamp': '2023-05-31T21:25:53.141700Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000044038', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2023-05-31T21:21:35.912562Z'}}] 2023-05-31T21:27:21.117 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000045235', 1: '00000000000000000000:00000000000000031499', 2: '00000000000000000000:00000000000000051391', 3: '00000000000000000000:00000000000000044038'} 2023-05-31T21:27:21.118 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000045235', 1: '00000000000000000000:00000000000000031499', 2: '00000000000000000000:00000000000000051391', 3: '00000000000000000000:00000000000000044038'} 2023-05-31T21:27:21.119 INFO:rgw_multi.tests:finished data checkpoint for target_zone=a2 source_zone=a1 2023-05-31T21:27:21.119 DEBUG:boto:path=/pklxkd-70/ 2023-05-31T21:27:21.120 DEBUG:boto:auth_path=/pklxkd-70/ 2023-05-31T21:27:21.121 DEBUG:boto:Method: HEAD 2023-05-31T21:27:21.121 DEBUG:boto:Path: /pklxkd-70/ 2023-05-31T21:27:21.122 DEBUG:boto:Data: 2023-05-31T21:27:21.123 DEBUG:boto:Headers: {} 2023-05-31T21:27:21.124 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T21:27:21.125 DEBUG:boto:Port: 8001 2023-05-31T21:27:21.125 DEBUG:boto:Params: {} 2023-05-31T21:27:21.126 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8001} 2023-05-31T21:27:21.127 DEBUG:boto:Token: None 2023-05-31T21:27:21.127 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 21:27:21 GMT /pklxkd-70/ 2023-05-31T21:27:21.128 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:G6q8GEacO1rbZG4SoRD1egda5q4= 2023-05-31T21:27:21.128 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 21:27:21 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:G6q8GEacO1rbZG4SoRD1egda5q4=', 'Content-Length': '0'} 2023-05-31T21:27:21.132 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000001f6c533083e2810c-006477bbb9-2648-a2'), ('Content-Length', '0'), ('Date', 'Wed, 31 May 2023 21:27:21 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:27:21.133 DEBUG:boto:path=/pklxkd-70/obj5 2023-05-31T21:27:21.134 DEBUG:boto:auth_path=/pklxkd-70/obj5 2023-05-31T21:27:21.134 DEBUG:boto:Method: HEAD 2023-05-31T21:27:21.134 DEBUG:boto:Path: /pklxkd-70/obj5 2023-05-31T21:27:21.135 DEBUG:boto:Data: 2023-05-31T21:27:21.135 DEBUG:boto:Headers: {} 2023-05-31T21:27:21.135 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T21:27:21.136 DEBUG:boto:Port: 8001 2023-05-31T21:27:21.136 DEBUG:boto:Params: {} 2023-05-31T21:27:21.136 DEBUG:boto:Token: None 2023-05-31T21:27:21.137 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 21:27:21 GMT /pklxkd-70/obj5 2023-05-31T21:27:21.137 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:j9QirMnBUuaSwwQ06Iap60jp3QE= 2023-05-31T21:27:21.138 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 21:27:21 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:j9QirMnBUuaSwwQ06Iap60jp3QE=', 'Content-Length': '0'} 2023-05-31T21:27:21.139 DEBUG:boto:Response headers: [('Content-Length', '219'), ('x-amz-request-id', 'tx00000a9a3547a9d73d61a-006477bbb9-2648-a2'), ('Accept-Ranges', 'bytes'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 21:27:21 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:27:21.140 DEBUG:boto:path=/pklxkd-70/obj6 2023-05-31T21:27:21.140 DEBUG:boto:auth_path=/pklxkd-70/obj6 2023-05-31T21:27:21.140 DEBUG:boto:Method: HEAD 2023-05-31T21:27:21.141 DEBUG:boto:Path: /pklxkd-70/obj6 2023-05-31T21:27:21.141 DEBUG:boto:Data: 2023-05-31T21:27:21.141 DEBUG:boto:Headers: {} 2023-05-31T21:27:21.142 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T21:27:21.142 DEBUG:boto:Port: 8001 2023-05-31T21:27:21.144 DEBUG:boto:Params: {} 2023-05-31T21:27:21.145 DEBUG:boto:Token: None 2023-05-31T21:27:21.145 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 21:27:21 GMT /pklxkd-70/obj6 2023-05-31T21:27:21.145 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:l23jGzMbmXZEloOsu1KU8pWtLMA= 2023-05-31T21:27:21.146 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 21:27:21 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:l23jGzMbmXZEloOsu1KU8pWtLMA=', 'Content-Length': '0'} 2023-05-31T21:27:21.147 DEBUG:boto:Response headers: [('Content-Length', '219'), ('x-amz-request-id', 'tx0000084e6888de2863543-006477bbb9-2648-a2'), ('Accept-Ranges', 'bytes'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 21:27:21 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:27:21.148 DEBUG:boto:path=/pklxkd-71/ 2023-05-31T21:27:21.148 DEBUG:boto:auth_path=/pklxkd-71/ 2023-05-31T21:27:21.148 DEBUG:boto:Method: HEAD 2023-05-31T21:27:21.149 DEBUG:boto:Path: /pklxkd-71/ 2023-05-31T21:27:21.149 DEBUG:boto:Data: 2023-05-31T21:27:21.149 DEBUG:boto:Headers: {} 2023-05-31T21:27:21.150 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T21:27:21.150 DEBUG:boto:Port: 8001 2023-05-31T21:27:21.150 DEBUG:boto:Params: {} 2023-05-31T21:27:21.151 DEBUG:boto:Token: None 2023-05-31T21:27:21.151 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 21:27:21 GMT /pklxkd-71/ 2023-05-31T21:27:21.151 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:PLvxnZTUIsKWW5zLKpBFvKFMRQo= 2023-05-31T21:27:21.152 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 21:27:21 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:PLvxnZTUIsKWW5zLKpBFvKFMRQo=', 'Content-Length': '0'} 2023-05-31T21:27:21.154 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '4'), ('X-RGW-Bytes-Used', '24'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000d5869d0e1b4afb8e-006477bbb9-2648-a2'), ('Content-Length', '0'), ('Date', 'Wed, 31 May 2023 21:27:21 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:27:21.154 DEBUG:boto:path=/pklxkd-71/obj5 2023-05-31T21:27:21.155 DEBUG:boto:auth_path=/pklxkd-71/obj5 2023-05-31T21:27:21.155 DEBUG:boto:Method: HEAD 2023-05-31T21:27:21.155 DEBUG:boto:Path: /pklxkd-71/obj5 2023-05-31T21:27:21.156 DEBUG:boto:Data: 2023-05-31T21:27:21.156 DEBUG:boto:Headers: {} 2023-05-31T21:27:21.156 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T21:27:21.157 DEBUG:boto:Port: 8001 2023-05-31T21:27:21.157 DEBUG:boto:Params: {} 2023-05-31T21:27:21.157 DEBUG:boto:Token: None 2023-05-31T21:27:21.158 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 21:27:21 GMT /pklxkd-71/obj5 2023-05-31T21:27:21.158 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:X9/djewic903SYTxpvw+6C/k6Wg= 2023-05-31T21:27:21.159 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 21:27:21 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:X9/djewic903SYTxpvw+6C/k6Wg=', 'Content-Length': '0'} 2023-05-31T21:27:21.160 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 21:25:53 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000005cc02dffa718b1bc-006477bbb9-2648-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 21:27:21 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:27:21.161 DEBUG:boto:path=/pklxkd-71/obj5 2023-05-31T21:27:21.162 DEBUG:boto:auth_path=/pklxkd-71/obj5 2023-05-31T21:27:21.162 DEBUG:boto:Method: GET 2023-05-31T21:27:21.162 DEBUG:boto:Path: /pklxkd-71/obj5 2023-05-31T21:27:21.163 DEBUG:boto:Data: 2023-05-31T21:27:21.163 DEBUG:boto:Headers: {} 2023-05-31T21:27:21.163 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T21:27:21.164 DEBUG:boto:Port: 8001 2023-05-31T21:27:21.164 DEBUG:boto:Params: {} 2023-05-31T21:27:21.164 DEBUG:boto:Token: None 2023-05-31T21:27:21.165 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 21:27:21 GMT /pklxkd-71/obj5 2023-05-31T21:27:21.165 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:RB75IQcTdWCcS5hnrHK+tnbYK2I= 2023-05-31T21:27:21.166 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 21:27:21 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:RB75IQcTdWCcS5hnrHK+tnbYK2I=', 'Content-Length': '0'} 2023-05-31T21:27:21.168 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 21:25:53 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000009bdc022e22b8d85b-006477bbb9-2648-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 21:27:21 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:27:21.168 DEBUG:boto:path=/pklxkd-71/obj6 2023-05-31T21:27:21.169 DEBUG:boto:auth_path=/pklxkd-71/obj6 2023-05-31T21:27:21.169 DEBUG:boto:Method: HEAD 2023-05-31T21:27:21.169 DEBUG:boto:Path: /pklxkd-71/obj6 2023-05-31T21:27:21.170 DEBUG:boto:Data: 2023-05-31T21:27:21.170 DEBUG:boto:Headers: {} 2023-05-31T21:27:21.170 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T21:27:21.171 DEBUG:boto:Port: 8001 2023-05-31T21:27:21.171 DEBUG:boto:Params: {} 2023-05-31T21:27:21.172 DEBUG:boto:Token: None 2023-05-31T21:27:21.172 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 21:27:21 GMT /pklxkd-71/obj6 2023-05-31T21:27:21.172 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:JKfR5XF4+KfEZTtRXQuzTVAuYfc= 2023-05-31T21:27:21.173 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 21:27:21 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:JKfR5XF4+KfEZTtRXQuzTVAuYfc=', 'Content-Length': '0'} 2023-05-31T21:27:21.174 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 21:25:53 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000005c3d2b9df701dae-006477bbb9-2648-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 21:27:21 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:27:21.175 DEBUG:boto:path=/pklxkd-71/obj6 2023-05-31T21:27:21.175 DEBUG:boto:auth_path=/pklxkd-71/obj6 2023-05-31T21:27:21.176 DEBUG:boto:Method: GET 2023-05-31T21:27:21.176 DEBUG:boto:Path: /pklxkd-71/obj6 2023-05-31T21:27:21.176 DEBUG:boto:Data: 2023-05-31T21:27:21.177 DEBUG:boto:Headers: {} 2023-05-31T21:27:21.177 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T21:27:21.177 DEBUG:boto:Port: 8001 2023-05-31T21:27:21.178 DEBUG:boto:Params: {} 2023-05-31T21:27:21.179 DEBUG:boto:Token: None 2023-05-31T21:27:21.179 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 21:27:21 GMT /pklxkd-71/obj6 2023-05-31T21:27:21.179 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:gFNfMnt9n3CCwxMslsCb2mnZPug= 2023-05-31T21:27:21.180 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 21:27:21 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:gFNfMnt9n3CCwxMslsCb2mnZPug=', 'Content-Length': '0'} 2023-05-31T21:27:21.182 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 21:25:53 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000002ab6a33ace52cd84-006477bbb9-2648-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 21:27:21 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:27:21.182 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'group', 'remove', '--group-id', 'sync-bucket', '--bucket', 'pklxkd-70', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-05-31T21:27:21.183 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'group', 'remove', '--group-id', 'sync-bucket', '--bucket', 'pklxkd-70', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-05-31T21:27:21.183 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync group remove --group-id sync-bucket --bucket pklxkd-70 --cluster c1 --debug-rgw 0 --debug-ms 0 2023-05-31T21:27:21.233 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2023-05-31T21:27:21.234 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2023-05-31T21:27:21.301 INFO:teuthology.orchestra.run.smithi099.stdout:{"sync_policy":{"groups":[]}} 2023-05-31T21:27:21.302 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-31T21:27:21.302 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-31T21:27:21.303 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-05-31T21:27:21.351 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2023-05-31T21:27:21.352 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2023-05-31T21:27:21.385 INFO:teuthology.orchestra.run.smithi099.stderr:No --period given, using current period=2e6123d2-d695-4350-9a01-61a455bc782b 2023-05-31T21:27:21.401 INFO:teuthology.orchestra.run.smithi099.stdout:[ 2023-05-31T21:27:21.402 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T21:27:21.402 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1685566501.757183_260.1", 2023-05-31T21:27:21.402 INFO:teuthology.orchestra.run.smithi099.stdout: "last_update": "2023-05-31T20:55:01.757183Z" 2023-05-31T21:27:21.402 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:27:21.403 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T21:27:21.403 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1685566501.735717_251.1", 2023-05-31T21:27:21.403 INFO:teuthology.orchestra.run.smithi099.stdout: "last_update": "2023-05-31T20:55:01.735717Z" 2023-05-31T21:27:21.403 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:27:21.403 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T21:27:21.403 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1685562970.274572_93.1", 2023-05-31T21:27:21.404 INFO:teuthology.orchestra.run.smithi099.stdout: "last_update": "2023-05-31T19:56:10.274572Z" 2023-05-31T21:27:21.404 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:27:21.404 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T21:27:21.404 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1685568441.280578_996.1", 2023-05-31T21:27:21.404 INFO:teuthology.orchestra.run.smithi099.stdout: "last_update": "2023-05-31T21:27:21.280578Z" 2023-05-31T21:27:21.404 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T21:27:21.405 INFO:teuthology.orchestra.run.smithi099.stdout:] 2023-05-31T21:27:21.407 DEBUG:rgw_multi.tests:master meta markers={0: '1_1685566501.757183_260.1', 1: '1_1685566501.735717_251.1', 2: '1_1685562970.274572_93.1', 3: '1_1685568441.280578_996.1'} 2023-05-31T21:27:21.408 INFO:rgw_multi.tests:starting meta checkpoint for zone=a2 2023-05-31T21:27:21.409 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-31T21:27:21.410 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-31T21:27:21.411 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-05-31T21:27:21.463 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2023-05-31T21:27:21.464 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2023-05-31T21:27:21.516 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T21:27:21.512+0000 7f9635785780 20 RGW-SYNC:meta: read sync status 2023-05-31T21:27:21.517 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T21:27:21.513+0000 7f9635785780 20 RGW-SYNC:meta: read sync status 2023-05-31T21:27:21.538 INFO:teuthology.orchestra.run.smithi099.stdout:{ 2023-05-31T21:27:21.539 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_status": { 2023-05-31T21:27:21.539 INFO:teuthology.orchestra.run.smithi099.stdout: "info": { 2023-05-31T21:27:21.539 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "sync", 2023-05-31T21:27:21.539 INFO:teuthology.orchestra.run.smithi099.stdout: "num_shards": 4, 2023-05-31T21:27:21.539 INFO:teuthology.orchestra.run.smithi099.stdout: "period": "2e6123d2-d695-4350-9a01-61a455bc782b", 2023-05-31T21:27:21.539 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2023-05-31T21:27:21.540 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:27:21.540 INFO:teuthology.orchestra.run.smithi099.stdout: "markers": [ 2023-05-31T21:27:21.540 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T21:27:21.540 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 0, 2023-05-31T21:27:21.540 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T21:27:21.540 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2023-05-31T21:27:21.541 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1685566501.757183_260.1", 2023-05-31T21:27:21.541 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T21:27:21.541 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 1, 2023-05-31T21:27:21.541 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T21:27:21.541 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2023-05-31T20:55:01.757183Z", 2023-05-31T21:27:21.541 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2023-05-31T21:27:21.542 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T21:27:21.542 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:27:21.542 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T21:27:21.542 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 1, 2023-05-31T21:27:21.542 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T21:27:21.542 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2023-05-31T21:27:21.543 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1685566501.735717_251.1", 2023-05-31T21:27:21.543 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T21:27:21.543 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2023-05-31T21:27:21.543 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T21:27:21.543 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2023-05-31T20:55:01.735717Z", 2023-05-31T21:27:21.543 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2023-05-31T21:27:21.544 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T21:27:21.544 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:27:21.544 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T21:27:21.544 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 2, 2023-05-31T21:27:21.544 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T21:27:21.544 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2023-05-31T21:27:21.544 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1685562970.274572_93.1", 2023-05-31T21:27:21.545 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T21:27:21.545 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2023-05-31T21:27:21.545 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T21:27:21.545 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2023-05-31T19:56:10.274572Z", 2023-05-31T21:27:21.545 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2023-05-31T21:27:21.545 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T21:27:21.546 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:27:21.546 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T21:27:21.546 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 3, 2023-05-31T21:27:21.546 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T21:27:21.546 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2023-05-31T21:27:21.546 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1685568316.644293_992.1", 2023-05-31T21:27:21.547 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T21:27:21.547 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2023-05-31T21:27:21.547 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T21:27:21.547 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2023-05-31T21:25:16.644293Z", 2023-05-31T21:27:21.547 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2023-05-31T21:27:21.547 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T21:27:21.547 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T21:27:21.548 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2023-05-31T21:27:21.548 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:27:21.548 INFO:teuthology.orchestra.run.smithi099.stdout: "full_sync": { 2023-05-31T21:27:21.548 INFO:teuthology.orchestra.run.smithi099.stdout: "total": 1, 2023-05-31T21:27:21.548 INFO:teuthology.orchestra.run.smithi099.stdout: "complete": 1 2023-05-31T21:27:21.548 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:27:21.549 INFO:teuthology.orchestra.run.smithi099.stdout: "current_time": "2023-05-31T21:27:21Z" 2023-05-31T21:27:21.549 INFO:teuthology.orchestra.run.smithi099.stdout:} 2023-05-31T21:27:21.550 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "2e6123d2-d695-4350-9a01-61a455bc782b", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1685566501.757183_260.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-05-31T20:55:01.757183Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1685566501.735717_251.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-05-31T20:55:01.735717Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1685562970.274572_93.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-05-31T19:56:10.274572Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1685568316.644293_992.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-05-31T21:25:16.644293Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-05-31T21:27:21Z" } 2023-05-31T21:27:21.550 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1685566501.757183_260.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-05-31T20:55:01.757183Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1685566501.735717_251.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-05-31T20:55:01.735717Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1685562970.274572_93.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-05-31T19:56:10.274572Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1685568316.644293_992.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-05-31T21:25:16.644293Z', 'realm_epoch': 2}}] 2023-05-31T21:27:21.551 DEBUG:rgw_multi.tests:log_status={0: '1_1685566501.757183_260.1', 1: '1_1685566501.735717_251.1', 2: '1_1685562970.274572_93.1', 3: '1_1685568441.280578_996.1'} 2023-05-31T21:27:21.551 DEBUG:rgw_multi.tests:sync_status={0: '1_1685566501.757183_260.1', 1: '1_1685566501.735717_251.1', 2: '1_1685562970.274572_93.1', 3: '1_1685568316.644293_992.1'} 2023-05-31T21:27:21.552 WARNING:rgw_multi.tests:zone a2 behind master: shard=3 master=1_1685568441.280578_996.1 target=1_1685568316.644293_992.1 2023-05-31T21:27:26.552 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-31T21:27:26.553 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-31T21:27:26.554 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-05-31T21:27:26.612 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2023-05-31T21:27:26.613 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2023-05-31T21:27:26.655 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T21:27:26.651+0000 7f5826fcc780 20 RGW-SYNC:meta: read sync status 2023-05-31T21:27:26.656 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T21:27:26.652+0000 7f5826fcc780 20 RGW-SYNC:meta: read sync status 2023-05-31T21:27:26.671 INFO:teuthology.orchestra.run.smithi099.stdout:{ 2023-05-31T21:27:26.671 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_status": { 2023-05-31T21:27:26.671 INFO:teuthology.orchestra.run.smithi099.stdout: "info": { 2023-05-31T21:27:26.672 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "sync", 2023-05-31T21:27:26.672 INFO:teuthology.orchestra.run.smithi099.stdout: "num_shards": 4, 2023-05-31T21:27:26.672 INFO:teuthology.orchestra.run.smithi099.stdout: "period": "2e6123d2-d695-4350-9a01-61a455bc782b", 2023-05-31T21:27:26.672 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2023-05-31T21:27:26.672 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:27:26.672 INFO:teuthology.orchestra.run.smithi099.stdout: "markers": [ 2023-05-31T21:27:26.673 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T21:27:26.673 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 0, 2023-05-31T21:27:26.673 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T21:27:26.673 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2023-05-31T21:27:26.673 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1685566501.757183_260.1", 2023-05-31T21:27:26.673 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T21:27:26.673 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 1, 2023-05-31T21:27:26.674 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T21:27:26.674 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2023-05-31T20:55:01.757183Z", 2023-05-31T21:27:26.674 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2023-05-31T21:27:26.674 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T21:27:26.674 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:27:26.674 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T21:27:26.675 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 1, 2023-05-31T21:27:26.675 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T21:27:26.676 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2023-05-31T21:27:26.676 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1685566501.735717_251.1", 2023-05-31T21:27:26.676 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T21:27:26.677 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2023-05-31T21:27:26.677 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T21:27:26.677 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2023-05-31T20:55:01.735717Z", 2023-05-31T21:27:26.677 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2023-05-31T21:27:26.677 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T21:27:26.677 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:27:26.678 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T21:27:26.678 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 2, 2023-05-31T21:27:26.678 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T21:27:26.678 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2023-05-31T21:27:26.679 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1685562970.274572_93.1", 2023-05-31T21:27:26.679 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T21:27:26.679 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2023-05-31T21:27:26.680 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T21:27:26.680 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2023-05-31T19:56:10.274572Z", 2023-05-31T21:27:26.680 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2023-05-31T21:27:26.681 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T21:27:26.682 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:27:26.682 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T21:27:26.682 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 3, 2023-05-31T21:27:26.682 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T21:27:26.683 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2023-05-31T21:27:26.683 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1685568316.644293_992.1", 2023-05-31T21:27:26.683 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T21:27:26.683 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2023-05-31T21:27:26.683 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T21:27:26.684 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2023-05-31T21:25:16.644293Z", 2023-05-31T21:27:26.684 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2023-05-31T21:27:26.684 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T21:27:26.685 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T21:27:26.685 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2023-05-31T21:27:26.685 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:27:26.685 INFO:teuthology.orchestra.run.smithi099.stdout: "full_sync": { 2023-05-31T21:27:26.686 INFO:teuthology.orchestra.run.smithi099.stdout: "total": 1, 2023-05-31T21:27:26.686 INFO:teuthology.orchestra.run.smithi099.stdout: "complete": 1 2023-05-31T21:27:26.686 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:27:26.686 INFO:teuthology.orchestra.run.smithi099.stdout: "current_time": "2023-05-31T21:27:26Z" 2023-05-31T21:27:26.686 INFO:teuthology.orchestra.run.smithi099.stdout:} 2023-05-31T21:27:26.688 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "2e6123d2-d695-4350-9a01-61a455bc782b", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1685566501.757183_260.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-05-31T20:55:01.757183Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1685566501.735717_251.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-05-31T20:55:01.735717Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1685562970.274572_93.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-05-31T19:56:10.274572Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1685568316.644293_992.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-05-31T21:25:16.644293Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-05-31T21:27:26Z" } 2023-05-31T21:27:26.688 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1685566501.757183_260.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-05-31T20:55:01.757183Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1685566501.735717_251.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-05-31T20:55:01.735717Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1685562970.274572_93.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-05-31T19:56:10.274572Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1685568316.644293_992.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-05-31T21:25:16.644293Z', 'realm_epoch': 2}}] 2023-05-31T21:27:26.689 DEBUG:rgw_multi.tests:log_status={0: '1_1685566501.757183_260.1', 1: '1_1685566501.735717_251.1', 2: '1_1685562970.274572_93.1', 3: '1_1685568441.280578_996.1'} 2023-05-31T21:27:26.690 DEBUG:rgw_multi.tests:sync_status={0: '1_1685566501.757183_260.1', 1: '1_1685566501.735717_251.1', 2: '1_1685562970.274572_93.1', 3: '1_1685568316.644293_992.1'} 2023-05-31T21:27:26.691 WARNING:rgw_multi.tests:zone a2 behind master: shard=3 master=1_1685568441.280578_996.1 target=1_1685568316.644293_992.1 2023-05-31T21:27:31.693 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-31T21:27:31.694 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-31T21:27:31.694 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-05-31T21:27:31.751 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2023-05-31T21:27:31.751 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2023-05-31T21:27:31.812 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T21:27:31.807+0000 7f6db21fd780 20 RGW-SYNC:meta: read sync status 2023-05-31T21:27:31.812 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T21:27:31.808+0000 7f6db21fd780 20 RGW-SYNC:meta: read sync status 2023-05-31T21:27:31.831 INFO:teuthology.orchestra.run.smithi099.stdout:{ 2023-05-31T21:27:31.832 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_status": { 2023-05-31T21:27:31.832 INFO:teuthology.orchestra.run.smithi099.stdout: "info": { 2023-05-31T21:27:31.833 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "sync", 2023-05-31T21:27:31.833 INFO:teuthology.orchestra.run.smithi099.stdout: "num_shards": 4, 2023-05-31T21:27:31.833 INFO:teuthology.orchestra.run.smithi099.stdout: "period": "2e6123d2-d695-4350-9a01-61a455bc782b", 2023-05-31T21:27:31.834 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2023-05-31T21:27:31.834 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:27:31.834 INFO:teuthology.orchestra.run.smithi099.stdout: "markers": [ 2023-05-31T21:27:31.835 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T21:27:31.835 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 0, 2023-05-31T21:27:31.835 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T21:27:31.835 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2023-05-31T21:27:31.836 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1685566501.757183_260.1", 2023-05-31T21:27:31.836 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T21:27:31.836 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 1, 2023-05-31T21:27:31.837 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T21:27:31.837 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2023-05-31T20:55:01.757183Z", 2023-05-31T21:27:31.837 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2023-05-31T21:27:31.838 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T21:27:31.838 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:27:31.838 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T21:27:31.839 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 1, 2023-05-31T21:27:31.839 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T21:27:31.839 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2023-05-31T21:27:31.839 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1685566501.735717_251.1", 2023-05-31T21:27:31.840 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T21:27:31.840 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2023-05-31T21:27:31.840 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T21:27:31.840 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2023-05-31T20:55:01.735717Z", 2023-05-31T21:27:31.840 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2023-05-31T21:27:31.840 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T21:27:31.841 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:27:31.841 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T21:27:31.841 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 2, 2023-05-31T21:27:31.841 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T21:27:31.841 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2023-05-31T21:27:31.841 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1685562970.274572_93.1", 2023-05-31T21:27:31.841 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T21:27:31.842 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2023-05-31T21:27:31.842 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T21:27:31.842 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2023-05-31T19:56:10.274572Z", 2023-05-31T21:27:31.842 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2023-05-31T21:27:31.842 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T21:27:31.843 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:27:31.843 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T21:27:31.843 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 3, 2023-05-31T21:27:31.843 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T21:27:31.843 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2023-05-31T21:27:31.843 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1685568441.280578_996.1", 2023-05-31T21:27:31.844 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T21:27:31.844 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2023-05-31T21:27:31.844 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T21:27:31.844 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2023-05-31T21:27:21.280578Z", 2023-05-31T21:27:31.844 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2023-05-31T21:27:31.845 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T21:27:31.845 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T21:27:31.845 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2023-05-31T21:27:31.846 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:27:31.846 INFO:teuthology.orchestra.run.smithi099.stdout: "full_sync": { 2023-05-31T21:27:31.846 INFO:teuthology.orchestra.run.smithi099.stdout: "total": 1, 2023-05-31T21:27:31.846 INFO:teuthology.orchestra.run.smithi099.stdout: "complete": 1 2023-05-31T21:27:31.847 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:27:31.847 INFO:teuthology.orchestra.run.smithi099.stdout: "current_time": "2023-05-31T21:27:31Z" 2023-05-31T21:27:31.847 INFO:teuthology.orchestra.run.smithi099.stdout:} 2023-05-31T21:27:31.849 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "2e6123d2-d695-4350-9a01-61a455bc782b", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1685566501.757183_260.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-05-31T20:55:01.757183Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1685566501.735717_251.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-05-31T20:55:01.735717Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1685562970.274572_93.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-05-31T19:56:10.274572Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1685568441.280578_996.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-05-31T21:27:21.280578Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-05-31T21:27:31Z" } 2023-05-31T21:27:31.850 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1685566501.757183_260.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-05-31T20:55:01.757183Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1685566501.735717_251.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-05-31T20:55:01.735717Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1685562970.274572_93.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-05-31T19:56:10.274572Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1685568441.280578_996.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-05-31T21:27:21.280578Z', 'realm_epoch': 2}}] 2023-05-31T21:27:31.851 DEBUG:rgw_multi.tests:log_status={0: '1_1685566501.757183_260.1', 1: '1_1685566501.735717_251.1', 2: '1_1685562970.274572_93.1', 3: '1_1685568441.280578_996.1'} 2023-05-31T21:27:31.851 DEBUG:rgw_multi.tests:sync_status={0: '1_1685566501.757183_260.1', 1: '1_1685566501.735717_251.1', 2: '1_1685562970.274572_93.1', 3: '1_1685568441.280578_996.1'} 2023-05-31T21:27:31.852 INFO:rgw_multi.tests:finish meta checkpoint for zone=a2 2023-05-31T21:27:31.853 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'policy', 'get', '--bucket', 'pklxkd-70', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-05-31T21:27:31.853 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'policy', 'get', '--bucket', 'pklxkd-70', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-05-31T21:27:31.854 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync policy get --bucket pklxkd-70 --cluster c1 --debug-rgw 0 --debug-ms 0 2023-05-31T21:27:31.902 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2023-05-31T21:27:31.902 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2023-05-31T21:27:31.952 INFO:teuthology.orchestra.run.smithi099.stdout:{"groups":[]} 2023-05-31T21:27:31.953 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'group', 'create', '--group-id', 'sync-bucket', '--status', 'allowed', '--bucket', 'pklxkd-70', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-05-31T21:27:31.953 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'group', 'create', '--group-id', 'sync-bucket', '--status', 'allowed', '--bucket', 'pklxkd-70', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-05-31T21:27:31.954 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync group create --group-id sync-bucket --status allowed --bucket pklxkd-70 --cluster c1 --debug-rgw 0 --debug-ms 0 2023-05-31T21:27:32.006 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2023-05-31T21:27:32.006 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2023-05-31T21:27:32.092 INFO:teuthology.orchestra.run.smithi099.stdout:{"groups":[{"id":"sync-bucket","data_flow":{},"pipes":[],"status":"allowed"}]} 2023-05-31T21:27:32.093 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'group', 'flow', 'create', '--group-id', 'sync-bucket', '--flow-id', 'sync-flowA', '--flow-type', 'symmetrical', '--zones=a1,a2', '--bucket', 'pklxkd-70', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-05-31T21:27:32.093 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'group', 'flow', 'create', '--group-id', 'sync-bucket', '--flow-id', 'sync-flowA', '--flow-type', 'symmetrical', '--zones=a1,a2', '--bucket', 'pklxkd-70', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-05-31T21:27:32.094 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync group flow create --group-id sync-bucket --flow-id sync-flowA --flow-type symmetrical --zones=a1,a2 --bucket pklxkd-70 --cluster c1 --debug-rgw 0 --debug-ms 0 2023-05-31T21:27:32.150 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2023-05-31T21:27:32.150 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2023-05-31T21:27:32.225 INFO:teuthology.orchestra.run.smithi099.stdout:{"groups":[{"id":"sync-bucket","data_flow":{"symmetrical":[{"id":"sync-flowA","zones":["a1","a2"]}]},"pipes":[],"status":"allowed"}]} 2023-05-31T21:27:32.226 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'group', 'pipe', 'create', '--group-id', 'sync-bucket', '--pipe-id', 'sync-pipe', '--source-zones=a1,a2', '--dest-zones=a1,a2', '--bucket=pklxkd-70', '--source-bucket=pklxkd-71', '--dest-bucket=*', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-05-31T21:27:32.226 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'group', 'pipe', 'create', '--group-id', 'sync-bucket', '--pipe-id', 'sync-pipe', '--source-zones=a1,a2', '--dest-zones=a1,a2', '--bucket=pklxkd-70', '--source-bucket=pklxkd-71', '--dest-bucket=*', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-05-31T21:27:32.227 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync group pipe create --group-id sync-bucket --pipe-id sync-pipe --source-zones=a1,a2 --dest-zones=a1,a2 --bucket=pklxkd-70 --source-bucket=pklxkd-71 '--dest-bucket=*' --cluster c1 --debug-rgw 0 --debug-ms 0 2023-05-31T21:27:32.282 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2023-05-31T21:27:32.282 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2023-05-31T21:27:32.340 INFO:teuthology.orchestra.run.smithi099.stdout:{"groups":[{"id":"sync-bucket","data_flow":{"symmetrical":[{"id":"sync-flowA","zones":["a1","a2"]}]},"pipes":[{"id":"sync-pipe","source":{"bucket":"pklxkd-71","zones":["a1","a2"]},"dest":{"bucket":"*","zones":["a1","a2"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":"rgw-multisite-test-user"}}],"status":"allowed"}]} 2023-05-31T21:27:32.340 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'group', 'modify', '--group-id', 'sync-bucket', '--status', 'enabled', '--bucket', 'pklxkd-70', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-05-31T21:27:32.341 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'group', 'modify', '--group-id', 'sync-bucket', '--status', 'enabled', '--bucket', 'pklxkd-70', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-05-31T21:27:32.342 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync group modify --group-id sync-bucket --status enabled --bucket pklxkd-70 --cluster c1 --debug-rgw 0 --debug-ms 0 2023-05-31T21:27:32.393 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2023-05-31T21:27:32.393 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2023-05-31T21:27:32.453 INFO:teuthology.orchestra.run.smithi099.stdout:{"groups":[{"id":"sync-bucket","data_flow":{"symmetrical":[{"id":"sync-flowA","zones":["a1","a2"]}]},"pipes":[{"id":"sync-pipe","source":{"bucket":"pklxkd-71","zones":["a1","a2"]},"dest":{"bucket":"*","zones":["a1","a2"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":"rgw-multisite-test-user"}}],"status":"enabled"}]} 2023-05-31T21:27:32.454 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'policy', 'get', '--bucket', 'pklxkd-70', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-05-31T21:27:32.455 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'policy', 'get', '--bucket', 'pklxkd-70', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-05-31T21:27:32.456 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync policy get --bucket pklxkd-70 --cluster c1 --debug-rgw 0 --debug-ms 0 2023-05-31T21:27:32.504 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2023-05-31T21:27:32.504 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2023-05-31T21:27:32.563 INFO:teuthology.orchestra.run.smithi099.stdout:{"groups":[{"id":"sync-bucket","data_flow":{"symmetrical":[{"id":"sync-flowA","zones":["a1","a2"]}]},"pipes":[{"id":"sync-pipe","source":{"bucket":"pklxkd-71","zones":["a1","a2"]},"dest":{"bucket":"*","zones":["a1","a2"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":"rgw-multisite-test-user"}}],"status":"enabled"}]} 2023-05-31T21:27:32.564 DEBUG:boto:path=/pklxkd-70/ 2023-05-31T21:27:32.565 DEBUG:boto:auth_path=/pklxkd-70/ 2023-05-31T21:27:32.566 DEBUG:boto:Method: HEAD 2023-05-31T21:27:32.567 DEBUG:boto:Path: /pklxkd-70/ 2023-05-31T21:27:32.567 DEBUG:boto:Data: 2023-05-31T21:27:32.568 DEBUG:boto:Headers: {} 2023-05-31T21:27:32.569 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T21:27:32.569 DEBUG:boto:Port: 8001 2023-05-31T21:27:32.569 DEBUG:boto:Params: {} 2023-05-31T21:27:32.570 DEBUG:boto:Token: None 2023-05-31T21:27:32.570 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 21:27:32 GMT /pklxkd-70/ 2023-05-31T21:27:32.571 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:QUN0+qg/hh4LzygjHqOHp3I3CR8= 2023-05-31T21:27:32.571 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 21:27:32 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:QUN0+qg/hh4LzygjHqOHp3I3CR8=', 'Content-Length': '0'} 2023-05-31T21:27:32.574 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000001f1d02c7d6cf8760-006477bbc4-2648-a2'), ('Content-Length', '0'), ('Date', 'Wed, 31 May 2023 21:27:32 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:27:32.574 DEBUG:boto:path=/pklxkd-70/obj7 2023-05-31T21:27:32.575 DEBUG:boto:auth_path=/pklxkd-70/obj7 2023-05-31T21:27:32.575 DEBUG:boto:Method: PUT 2023-05-31T21:27:32.576 DEBUG:boto:Path: /pklxkd-70/obj7 2023-05-31T21:27:32.576 DEBUG:boto:Data: 2023-05-31T21:27:32.576 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2023-05-31T21:27:32.577 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T21:27:32.577 DEBUG:boto:Port: 8001 2023-05-31T21:27:32.577 DEBUG:boto:Params: {} 2023-05-31T21:27:32.578 DEBUG:boto:Token: None 2023-05-31T21:27:32.578 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Wed, 31 May 2023 21:27:32 GMT /pklxkd-70/obj7 2023-05-31T21:27:32.579 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:FRK1C1DEepL+Eab/6keYFpdbRAo= 2023-05-31T21:27:32.579 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Wed, 31 May 2023 21:27:32 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:FRK1C1DEepL+Eab/6keYFpdbRAo='} 2023-05-31T21:27:32.586 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000009e29e881aa1a295b-006477bbc4-2648-a2'), ('Date', 'Wed, 31 May 2023 21:27:32 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:27:32.586 DEBUG:boto:path=/pklxkd-70/ 2023-05-31T21:27:32.587 DEBUG:boto:auth_path=/pklxkd-70/ 2023-05-31T21:27:32.587 DEBUG:boto:Method: HEAD 2023-05-31T21:27:32.587 DEBUG:boto:Path: /pklxkd-70/ 2023-05-31T21:27:32.588 DEBUG:boto:Data: 2023-05-31T21:27:32.588 DEBUG:boto:Headers: {} 2023-05-31T21:27:32.588 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T21:27:32.589 DEBUG:boto:Port: 8001 2023-05-31T21:27:32.589 DEBUG:boto:Params: {} 2023-05-31T21:27:32.589 DEBUG:boto:Token: None 2023-05-31T21:27:32.590 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 21:27:32 GMT /pklxkd-70/ 2023-05-31T21:27:32.590 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:QUN0+qg/hh4LzygjHqOHp3I3CR8= 2023-05-31T21:27:32.591 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 21:27:32 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:QUN0+qg/hh4LzygjHqOHp3I3CR8=', 'Content-Length': '0'} 2023-05-31T21:27:32.593 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '6'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000008ce70eae774fe603-006477bbc4-2648-a2'), ('Content-Length', '0'), ('Date', 'Wed, 31 May 2023 21:27:32 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:27:32.594 DEBUG:boto:path=/pklxkd-70/obj8 2023-05-31T21:27:32.594 DEBUG:boto:auth_path=/pklxkd-70/obj8 2023-05-31T21:27:32.595 DEBUG:boto:Method: PUT 2023-05-31T21:27:32.595 DEBUG:boto:Path: /pklxkd-70/obj8 2023-05-31T21:27:32.596 DEBUG:boto:Data: 2023-05-31T21:27:32.596 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2023-05-31T21:27:32.597 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T21:27:32.597 DEBUG:boto:Port: 8001 2023-05-31T21:27:32.598 DEBUG:boto:Params: {} 2023-05-31T21:27:32.599 DEBUG:boto:Token: None 2023-05-31T21:27:32.599 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Wed, 31 May 2023 21:27:32 GMT /pklxkd-70/obj8 2023-05-31T21:27:32.600 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:mGuScrnsyxmjZhwO7kQLRpMAkV8= 2023-05-31T21:27:32.601 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Wed, 31 May 2023 21:27:32 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:mGuScrnsyxmjZhwO7kQLRpMAkV8='} 2023-05-31T21:27:32.644 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000056c0729a0e474e18-006477bbc4-2648-a2'), ('Date', 'Wed, 31 May 2023 21:27:32 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:27:32.645 DEBUG:boto:path=/pklxkd-71/ 2023-05-31T21:27:32.646 DEBUG:boto:auth_path=/pklxkd-71/ 2023-05-31T21:27:32.646 DEBUG:boto:Method: HEAD 2023-05-31T21:27:32.646 DEBUG:boto:Path: /pklxkd-71/ 2023-05-31T21:27:32.647 DEBUG:boto:Data: 2023-05-31T21:27:32.647 DEBUG:boto:Headers: {} 2023-05-31T21:27:32.647 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T21:27:32.648 DEBUG:boto:Port: 8001 2023-05-31T21:27:32.648 DEBUG:boto:Params: {} 2023-05-31T21:27:32.649 DEBUG:boto:Token: None 2023-05-31T21:27:32.650 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 21:27:32 GMT /pklxkd-71/ 2023-05-31T21:27:32.650 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:86cj5S7Et38tfsnUoSzY9Frmo50= 2023-05-31T21:27:32.651 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 21:27:32 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:86cj5S7Et38tfsnUoSzY9Frmo50=', 'Content-Length': '0'} 2023-05-31T21:27:32.654 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '4'), ('X-RGW-Bytes-Used', '24'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000fe99e8d55aa66581-006477bbc4-2648-a2'), ('Content-Length', '0'), ('Date', 'Wed, 31 May 2023 21:27:32 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:27:32.655 DEBUG:boto:path=/pklxkd-71/obj9 2023-05-31T21:27:32.655 DEBUG:boto:auth_path=/pklxkd-71/obj9 2023-05-31T21:27:32.656 DEBUG:boto:Method: PUT 2023-05-31T21:27:32.656 DEBUG:boto:Path: /pklxkd-71/obj9 2023-05-31T21:27:32.656 DEBUG:boto:Data: 2023-05-31T21:27:32.657 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2023-05-31T21:27:32.657 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T21:27:32.658 DEBUG:boto:Port: 8001 2023-05-31T21:27:32.658 DEBUG:boto:Params: {} 2023-05-31T21:27:32.658 DEBUG:boto:Token: None 2023-05-31T21:27:32.659 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Wed, 31 May 2023 21:27:32 GMT /pklxkd-71/obj9 2023-05-31T21:27:32.659 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:gK4TZ+/9PBodSO5ovcCFw/tnyVo= 2023-05-31T21:27:32.660 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Wed, 31 May 2023 21:27:32 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:gK4TZ+/9PBodSO5ovcCFw/tnyVo='} 2023-05-31T21:27:32.667 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000d67f49c0a38c3fa6-006477bbc4-2648-a2'), ('Date', 'Wed, 31 May 2023 21:27:32 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:27:32.668 DEBUG:boto:path=/pklxkd-71/ 2023-05-31T21:27:32.668 DEBUG:boto:auth_path=/pklxkd-71/ 2023-05-31T21:27:32.669 DEBUG:boto:Method: HEAD 2023-05-31T21:27:32.669 DEBUG:boto:Path: /pklxkd-71/ 2023-05-31T21:27:32.669 DEBUG:boto:Data: 2023-05-31T21:27:32.670 DEBUG:boto:Headers: {} 2023-05-31T21:27:32.670 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T21:27:32.671 DEBUG:boto:Port: 8001 2023-05-31T21:27:32.671 DEBUG:boto:Params: {} 2023-05-31T21:27:32.671 DEBUG:boto:Token: None 2023-05-31T21:27:32.672 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 21:27:32 GMT /pklxkd-71/ 2023-05-31T21:27:32.672 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:86cj5S7Et38tfsnUoSzY9Frmo50= 2023-05-31T21:27:32.673 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 21:27:32 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:86cj5S7Et38tfsnUoSzY9Frmo50=', 'Content-Length': '0'} 2023-05-31T21:27:32.675 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '5'), ('X-RGW-Bytes-Used', '30'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000003b17441a7c93ff11-006477bbc4-2648-a2'), ('Content-Length', '0'), ('Date', 'Wed, 31 May 2023 21:27:32 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:27:32.675 DEBUG:boto:path=/pklxkd-71/obj10 2023-05-31T21:27:32.676 DEBUG:boto:auth_path=/pklxkd-71/obj10 2023-05-31T21:27:32.676 DEBUG:boto:Method: PUT 2023-05-31T21:27:32.676 DEBUG:boto:Path: /pklxkd-71/obj10 2023-05-31T21:27:32.677 DEBUG:boto:Data: 2023-05-31T21:27:32.677 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2023-05-31T21:27:32.677 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T21:27:32.678 DEBUG:boto:Port: 8001 2023-05-31T21:27:32.678 DEBUG:boto:Params: {} 2023-05-31T21:27:32.679 DEBUG:boto:Token: None 2023-05-31T21:27:32.679 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Wed, 31 May 2023 21:27:32 GMT /pklxkd-71/obj10 2023-05-31T21:27:32.679 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:plsowOsV6HdzlPUzVu5ydjFtKko= 2023-05-31T21:27:32.680 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Wed, 31 May 2023 21:27:32 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:plsowOsV6HdzlPUzVu5ydjFtKko='} 2023-05-31T21:27:32.687 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000b1a9d45cc2c5bd4f-006477bbc4-2648-a2'), ('Date', 'Wed, 31 May 2023 21:27:32 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:27:32.688 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-31T21:27:32.689 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-31T21:27:32.689 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-05-31T21:27:32.738 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2023-05-31T21:27:32.739 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2023-05-31T21:27:32.779 INFO:teuthology.orchestra.run.smithi099.stderr:No --period given, using current period=2e6123d2-d695-4350-9a01-61a455bc782b 2023-05-31T21:27:32.797 INFO:teuthology.orchestra.run.smithi099.stdout:[ 2023-05-31T21:27:32.798 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T21:27:32.798 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1685566501.757183_260.1", 2023-05-31T21:27:32.798 INFO:teuthology.orchestra.run.smithi099.stdout: "last_update": "2023-05-31T20:55:01.757183Z" 2023-05-31T21:27:32.798 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:27:32.798 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T21:27:32.798 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1685566501.735717_251.1", 2023-05-31T21:27:32.799 INFO:teuthology.orchestra.run.smithi099.stdout: "last_update": "2023-05-31T20:55:01.735717Z" 2023-05-31T21:27:32.799 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:27:32.799 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T21:27:32.799 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1685562970.274572_93.1", 2023-05-31T21:27:32.799 INFO:teuthology.orchestra.run.smithi099.stdout: "last_update": "2023-05-31T19:56:10.274572Z" 2023-05-31T21:27:32.799 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:27:32.799 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T21:27:32.800 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1685568452.432716_1004.1", 2023-05-31T21:27:32.800 INFO:teuthology.orchestra.run.smithi099.stdout: "last_update": "2023-05-31T21:27:32.432716Z" 2023-05-31T21:27:32.800 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T21:27:32.800 INFO:teuthology.orchestra.run.smithi099.stdout:] 2023-05-31T21:27:32.802 DEBUG:rgw_multi.tests:master meta markers={0: '1_1685566501.757183_260.1', 1: '1_1685566501.735717_251.1', 2: '1_1685562970.274572_93.1', 3: '1_1685568452.432716_1004.1'} 2023-05-31T21:27:32.802 INFO:rgw_multi.tests:starting meta checkpoint for zone=a2 2023-05-31T21:27:32.802 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-31T21:27:32.803 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-31T21:27:32.804 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-05-31T21:27:32.853 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2023-05-31T21:27:32.854 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2023-05-31T21:27:32.883 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T21:27:32.878+0000 7f9c7caeb780 20 RGW-SYNC:meta: read sync status 2023-05-31T21:27:32.884 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T21:27:32.879+0000 7f9c7caeb780 20 RGW-SYNC:meta: read sync status 2023-05-31T21:27:32.897 INFO:teuthology.orchestra.run.smithi099.stdout:{ 2023-05-31T21:27:32.897 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_status": { 2023-05-31T21:27:32.897 INFO:teuthology.orchestra.run.smithi099.stdout: "info": { 2023-05-31T21:27:32.898 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "sync", 2023-05-31T21:27:32.898 INFO:teuthology.orchestra.run.smithi099.stdout: "num_shards": 4, 2023-05-31T21:27:32.898 INFO:teuthology.orchestra.run.smithi099.stdout: "period": "2e6123d2-d695-4350-9a01-61a455bc782b", 2023-05-31T21:27:32.898 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2023-05-31T21:27:32.899 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:27:32.899 INFO:teuthology.orchestra.run.smithi099.stdout: "markers": [ 2023-05-31T21:27:32.899 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T21:27:32.899 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 0, 2023-05-31T21:27:32.899 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T21:27:32.899 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2023-05-31T21:27:32.900 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1685566501.757183_260.1", 2023-05-31T21:27:32.900 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T21:27:32.900 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 1, 2023-05-31T21:27:32.900 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T21:27:32.900 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2023-05-31T20:55:01.757183Z", 2023-05-31T21:27:32.900 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2023-05-31T21:27:32.901 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T21:27:32.901 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:27:32.901 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T21:27:32.901 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 1, 2023-05-31T21:27:32.901 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T21:27:32.902 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2023-05-31T21:27:32.902 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1685566501.735717_251.1", 2023-05-31T21:27:32.902 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T21:27:32.903 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2023-05-31T21:27:32.903 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T21:27:32.903 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2023-05-31T20:55:01.735717Z", 2023-05-31T21:27:32.904 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2023-05-31T21:27:32.904 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T21:27:32.904 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:27:32.904 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T21:27:32.904 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 2, 2023-05-31T21:27:32.905 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T21:27:32.905 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2023-05-31T21:27:32.905 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1685562970.274572_93.1", 2023-05-31T21:27:32.905 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T21:27:32.906 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2023-05-31T21:27:32.906 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T21:27:32.906 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2023-05-31T19:56:10.274572Z", 2023-05-31T21:27:32.906 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2023-05-31T21:27:32.906 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T21:27:32.907 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:27:32.907 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T21:27:32.907 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 3, 2023-05-31T21:27:32.907 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T21:27:32.907 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2023-05-31T21:27:32.907 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1685568441.280578_996.1", 2023-05-31T21:27:32.908 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T21:27:32.908 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2023-05-31T21:27:32.908 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T21:27:32.908 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2023-05-31T21:27:21.280578Z", 2023-05-31T21:27:32.908 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2023-05-31T21:27:32.908 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T21:27:32.909 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T21:27:32.909 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2023-05-31T21:27:32.909 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:27:32.910 INFO:teuthology.orchestra.run.smithi099.stdout: "full_sync": { 2023-05-31T21:27:32.910 INFO:teuthology.orchestra.run.smithi099.stdout: "total": 1, 2023-05-31T21:27:32.910 INFO:teuthology.orchestra.run.smithi099.stdout: "complete": 1 2023-05-31T21:27:32.910 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:27:32.910 INFO:teuthology.orchestra.run.smithi099.stdout: "current_time": "2023-05-31T21:27:32Z" 2023-05-31T21:27:32.910 INFO:teuthology.orchestra.run.smithi099.stdout:} 2023-05-31T21:27:32.912 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "2e6123d2-d695-4350-9a01-61a455bc782b", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1685566501.757183_260.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-05-31T20:55:01.757183Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1685566501.735717_251.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-05-31T20:55:01.735717Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1685562970.274572_93.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-05-31T19:56:10.274572Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1685568441.280578_996.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-05-31T21:27:21.280578Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-05-31T21:27:32Z" } 2023-05-31T21:27:32.912 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1685566501.757183_260.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-05-31T20:55:01.757183Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1685566501.735717_251.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-05-31T20:55:01.735717Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1685562970.274572_93.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-05-31T19:56:10.274572Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1685568441.280578_996.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-05-31T21:27:21.280578Z', 'realm_epoch': 2}}] 2023-05-31T21:27:32.913 DEBUG:rgw_multi.tests:log_status={0: '1_1685566501.757183_260.1', 1: '1_1685566501.735717_251.1', 2: '1_1685562970.274572_93.1', 3: '1_1685568452.432716_1004.1'} 2023-05-31T21:27:32.913 DEBUG:rgw_multi.tests:sync_status={0: '1_1685566501.757183_260.1', 1: '1_1685566501.735717_251.1', 2: '1_1685562970.274572_93.1', 3: '1_1685568441.280578_996.1'} 2023-05-31T21:27:32.914 WARNING:rgw_multi.tests:zone a2 behind master: shard=3 master=1_1685568452.432716_1004.1 target=1_1685568441.280578_996.1 2023-05-31T21:27:37.915 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-31T21:27:37.916 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-31T21:27:37.917 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-05-31T21:27:37.975 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2023-05-31T21:27:37.975 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2023-05-31T21:27:38.024 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T21:27:38.021+0000 7f2613ca1780 20 RGW-SYNC:meta: read sync status 2023-05-31T21:27:38.025 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T21:27:38.022+0000 7f2613ca1780 20 RGW-SYNC:meta: read sync status 2023-05-31T21:27:38.043 INFO:teuthology.orchestra.run.smithi099.stdout:{ 2023-05-31T21:27:38.044 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_status": { 2023-05-31T21:27:38.044 INFO:teuthology.orchestra.run.smithi099.stdout: "info": { 2023-05-31T21:27:38.044 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "sync", 2023-05-31T21:27:38.044 INFO:teuthology.orchestra.run.smithi099.stdout: "num_shards": 4, 2023-05-31T21:27:38.044 INFO:teuthology.orchestra.run.smithi099.stdout: "period": "2e6123d2-d695-4350-9a01-61a455bc782b", 2023-05-31T21:27:38.045 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2023-05-31T21:27:38.045 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:27:38.045 INFO:teuthology.orchestra.run.smithi099.stdout: "markers": [ 2023-05-31T21:27:38.045 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T21:27:38.045 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 0, 2023-05-31T21:27:38.045 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T21:27:38.045 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2023-05-31T21:27:38.046 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1685566501.757183_260.1", 2023-05-31T21:27:38.046 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T21:27:38.046 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 1, 2023-05-31T21:27:38.046 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T21:27:38.046 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2023-05-31T20:55:01.757183Z", 2023-05-31T21:27:38.046 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2023-05-31T21:27:38.046 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T21:27:38.047 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:27:38.047 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T21:27:38.047 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 1, 2023-05-31T21:27:38.047 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T21:27:38.047 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2023-05-31T21:27:38.047 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1685566501.735717_251.1", 2023-05-31T21:27:38.047 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T21:27:38.048 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2023-05-31T21:27:38.048 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T21:27:38.048 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2023-05-31T20:55:01.735717Z", 2023-05-31T21:27:38.048 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2023-05-31T21:27:38.048 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T21:27:38.049 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:27:38.049 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T21:27:38.049 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 2, 2023-05-31T21:27:38.049 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T21:27:38.049 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2023-05-31T21:27:38.049 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1685562970.274572_93.1", 2023-05-31T21:27:38.049 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T21:27:38.050 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2023-05-31T21:27:38.050 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T21:27:38.050 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2023-05-31T19:56:10.274572Z", 2023-05-31T21:27:38.050 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2023-05-31T21:27:38.050 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T21:27:38.050 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:27:38.051 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T21:27:38.051 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 3, 2023-05-31T21:27:38.051 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T21:27:38.051 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2023-05-31T21:27:38.052 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1685568441.280578_996.1", 2023-05-31T21:27:38.052 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T21:27:38.052 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2023-05-31T21:27:38.052 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T21:27:38.052 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2023-05-31T21:27:21.280578Z", 2023-05-31T21:27:38.052 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2023-05-31T21:27:38.053 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T21:27:38.053 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T21:27:38.053 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2023-05-31T21:27:38.053 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:27:38.053 INFO:teuthology.orchestra.run.smithi099.stdout: "full_sync": { 2023-05-31T21:27:38.053 INFO:teuthology.orchestra.run.smithi099.stdout: "total": 1, 2023-05-31T21:27:38.054 INFO:teuthology.orchestra.run.smithi099.stdout: "complete": 1 2023-05-31T21:27:38.054 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:27:38.054 INFO:teuthology.orchestra.run.smithi099.stdout: "current_time": "2023-05-31T21:27:38Z" 2023-05-31T21:27:38.054 INFO:teuthology.orchestra.run.smithi099.stdout:} 2023-05-31T21:27:38.055 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "2e6123d2-d695-4350-9a01-61a455bc782b", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1685566501.757183_260.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-05-31T20:55:01.757183Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1685566501.735717_251.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-05-31T20:55:01.735717Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1685562970.274572_93.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-05-31T19:56:10.274572Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1685568441.280578_996.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-05-31T21:27:21.280578Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-05-31T21:27:38Z" } 2023-05-31T21:27:38.056 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1685566501.757183_260.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-05-31T20:55:01.757183Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1685566501.735717_251.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-05-31T20:55:01.735717Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1685562970.274572_93.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-05-31T19:56:10.274572Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1685568441.280578_996.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-05-31T21:27:21.280578Z', 'realm_epoch': 2}}] 2023-05-31T21:27:38.056 DEBUG:rgw_multi.tests:log_status={0: '1_1685566501.757183_260.1', 1: '1_1685566501.735717_251.1', 2: '1_1685562970.274572_93.1', 3: '1_1685568452.432716_1004.1'} 2023-05-31T21:27:38.056 DEBUG:rgw_multi.tests:sync_status={0: '1_1685566501.757183_260.1', 1: '1_1685566501.735717_251.1', 2: '1_1685562970.274572_93.1', 3: '1_1685568441.280578_996.1'} 2023-05-31T21:27:38.057 WARNING:rgw_multi.tests:zone a2 behind master: shard=3 master=1_1685568452.432716_1004.1 target=1_1685568441.280578_996.1 2023-05-31T21:27:43.058 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-31T21:27:43.059 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-31T21:27:43.060 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-05-31T21:27:43.117 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2023-05-31T21:27:43.118 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2023-05-31T21:27:43.161 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T21:27:43.158+0000 7fe25412e780 20 RGW-SYNC:meta: read sync status 2023-05-31T21:27:43.162 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T21:27:43.159+0000 7fe25412e780 20 RGW-SYNC:meta: read sync status 2023-05-31T21:27:43.176 INFO:teuthology.orchestra.run.smithi099.stdout:{ 2023-05-31T21:27:43.177 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_status": { 2023-05-31T21:27:43.177 INFO:teuthology.orchestra.run.smithi099.stdout: "info": { 2023-05-31T21:27:43.177 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "sync", 2023-05-31T21:27:43.177 INFO:teuthology.orchestra.run.smithi099.stdout: "num_shards": 4, 2023-05-31T21:27:43.177 INFO:teuthology.orchestra.run.smithi099.stdout: "period": "2e6123d2-d695-4350-9a01-61a455bc782b", 2023-05-31T21:27:43.177 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2023-05-31T21:27:43.178 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:27:43.178 INFO:teuthology.orchestra.run.smithi099.stdout: "markers": [ 2023-05-31T21:27:43.178 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T21:27:43.178 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 0, 2023-05-31T21:27:43.178 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T21:27:43.178 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2023-05-31T21:27:43.178 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1685566501.757183_260.1", 2023-05-31T21:27:43.179 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T21:27:43.179 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 1, 2023-05-31T21:27:43.179 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T21:27:43.179 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2023-05-31T20:55:01.757183Z", 2023-05-31T21:27:43.179 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2023-05-31T21:27:43.179 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T21:27:43.180 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:27:43.180 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T21:27:43.180 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 1, 2023-05-31T21:27:43.180 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T21:27:43.180 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2023-05-31T21:27:43.180 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1685566501.735717_251.1", 2023-05-31T21:27:43.181 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T21:27:43.181 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2023-05-31T21:27:43.181 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T21:27:43.181 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2023-05-31T20:55:01.735717Z", 2023-05-31T21:27:43.181 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2023-05-31T21:27:43.181 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T21:27:43.182 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:27:43.182 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T21:27:43.182 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 2, 2023-05-31T21:27:43.182 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T21:27:43.183 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2023-05-31T21:27:43.183 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1685562970.274572_93.1", 2023-05-31T21:27:43.183 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T21:27:43.183 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2023-05-31T21:27:43.183 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T21:27:43.183 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2023-05-31T19:56:10.274572Z", 2023-05-31T21:27:43.183 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2023-05-31T21:27:43.184 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T21:27:43.184 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:27:43.184 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T21:27:43.184 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 3, 2023-05-31T21:27:43.184 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T21:27:43.184 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2023-05-31T21:27:43.185 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1685568441.280578_996.1", 2023-05-31T21:27:43.185 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T21:27:43.185 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2023-05-31T21:27:43.185 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T21:27:43.186 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2023-05-31T21:27:21.280578Z", 2023-05-31T21:27:43.186 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2023-05-31T21:27:43.187 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T21:27:43.187 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T21:27:43.187 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2023-05-31T21:27:43.188 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:27:43.188 INFO:teuthology.orchestra.run.smithi099.stdout: "full_sync": { 2023-05-31T21:27:43.188 INFO:teuthology.orchestra.run.smithi099.stdout: "total": 1, 2023-05-31T21:27:43.189 INFO:teuthology.orchestra.run.smithi099.stdout: "complete": 1 2023-05-31T21:27:43.189 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:27:43.189 INFO:teuthology.orchestra.run.smithi099.stdout: "current_time": "2023-05-31T21:27:43Z" 2023-05-31T21:27:43.190 INFO:teuthology.orchestra.run.smithi099.stdout:} 2023-05-31T21:27:43.191 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "2e6123d2-d695-4350-9a01-61a455bc782b", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1685566501.757183_260.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-05-31T20:55:01.757183Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1685566501.735717_251.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-05-31T20:55:01.735717Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1685562970.274572_93.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-05-31T19:56:10.274572Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1685568441.280578_996.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-05-31T21:27:21.280578Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-05-31T21:27:43Z" } 2023-05-31T21:27:43.192 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1685566501.757183_260.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-05-31T20:55:01.757183Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1685566501.735717_251.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-05-31T20:55:01.735717Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1685562970.274572_93.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-05-31T19:56:10.274572Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1685568441.280578_996.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-05-31T21:27:21.280578Z', 'realm_epoch': 2}}] 2023-05-31T21:27:43.193 DEBUG:rgw_multi.tests:log_status={0: '1_1685566501.757183_260.1', 1: '1_1685566501.735717_251.1', 2: '1_1685562970.274572_93.1', 3: '1_1685568452.432716_1004.1'} 2023-05-31T21:27:43.194 DEBUG:rgw_multi.tests:sync_status={0: '1_1685566501.757183_260.1', 1: '1_1685566501.735717_251.1', 2: '1_1685562970.274572_93.1', 3: '1_1685568441.280578_996.1'} 2023-05-31T21:27:43.195 WARNING:rgw_multi.tests:zone a2 behind master: shard=3 master=1_1685568452.432716_1004.1 target=1_1685568441.280578_996.1 2023-05-31T21:27:48.196 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-31T21:27:48.197 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-31T21:27:48.197 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-05-31T21:27:48.254 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2023-05-31T21:27:48.254 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2023-05-31T21:27:48.326 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T21:27:48.323+0000 7f0a33920780 20 RGW-SYNC:meta: read sync status 2023-05-31T21:27:48.327 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T21:27:48.324+0000 7f0a33920780 20 RGW-SYNC:meta: read sync status 2023-05-31T21:27:48.359 INFO:teuthology.orchestra.run.smithi099.stdout:{ 2023-05-31T21:27:48.359 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_status": { 2023-05-31T21:27:48.359 INFO:teuthology.orchestra.run.smithi099.stdout: "info": { 2023-05-31T21:27:48.359 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "sync", 2023-05-31T21:27:48.360 INFO:teuthology.orchestra.run.smithi099.stdout: "num_shards": 4, 2023-05-31T21:27:48.360 INFO:teuthology.orchestra.run.smithi099.stdout: "period": "2e6123d2-d695-4350-9a01-61a455bc782b", 2023-05-31T21:27:48.360 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2023-05-31T21:27:48.360 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:27:48.360 INFO:teuthology.orchestra.run.smithi099.stdout: "markers": [ 2023-05-31T21:27:48.360 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T21:27:48.361 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 0, 2023-05-31T21:27:48.361 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T21:27:48.361 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2023-05-31T21:27:48.361 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1685566501.757183_260.1", 2023-05-31T21:27:48.361 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T21:27:48.361 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 1, 2023-05-31T21:27:48.361 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T21:27:48.362 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2023-05-31T20:55:01.757183Z", 2023-05-31T21:27:48.362 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2023-05-31T21:27:48.362 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T21:27:48.362 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:27:48.362 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T21:27:48.362 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 1, 2023-05-31T21:27:48.363 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T21:27:48.363 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2023-05-31T21:27:48.363 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1685566501.735717_251.1", 2023-05-31T21:27:48.363 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T21:27:48.363 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2023-05-31T21:27:48.363 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T21:27:48.363 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2023-05-31T20:55:01.735717Z", 2023-05-31T21:27:48.364 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2023-05-31T21:27:48.364 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T21:27:48.364 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:27:48.364 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T21:27:48.364 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 2, 2023-05-31T21:27:48.364 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T21:27:48.364 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2023-05-31T21:27:48.365 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1685562970.274572_93.1", 2023-05-31T21:27:48.365 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T21:27:48.365 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2023-05-31T21:27:48.365 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T21:27:48.365 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2023-05-31T19:56:10.274572Z", 2023-05-31T21:27:48.365 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2023-05-31T21:27:48.366 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T21:27:48.366 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:27:48.366 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T21:27:48.366 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 3, 2023-05-31T21:27:48.366 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T21:27:48.366 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2023-05-31T21:27:48.366 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1685568452.059513_997.1", 2023-05-31T21:27:48.367 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T21:27:48.367 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2023-05-31T21:27:48.367 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T21:27:48.367 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2023-05-31T21:27:32.059513Z", 2023-05-31T21:27:48.367 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2023-05-31T21:27:48.367 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T21:27:48.368 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T21:27:48.368 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2023-05-31T21:27:48.368 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:27:48.368 INFO:teuthology.orchestra.run.smithi099.stdout: "full_sync": { 2023-05-31T21:27:48.368 INFO:teuthology.orchestra.run.smithi099.stdout: "total": 1, 2023-05-31T21:27:48.368 INFO:teuthology.orchestra.run.smithi099.stdout: "complete": 1 2023-05-31T21:27:48.368 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:27:48.369 INFO:teuthology.orchestra.run.smithi099.stdout: "current_time": "2023-05-31T21:27:48Z" 2023-05-31T21:27:48.369 INFO:teuthology.orchestra.run.smithi099.stdout:} 2023-05-31T21:27:48.370 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "2e6123d2-d695-4350-9a01-61a455bc782b", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1685566501.757183_260.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-05-31T20:55:01.757183Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1685566501.735717_251.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-05-31T20:55:01.735717Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1685562970.274572_93.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-05-31T19:56:10.274572Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1685568452.059513_997.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-05-31T21:27:32.059513Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-05-31T21:27:48Z" } 2023-05-31T21:27:48.370 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1685566501.757183_260.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-05-31T20:55:01.757183Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1685566501.735717_251.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-05-31T20:55:01.735717Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1685562970.274572_93.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-05-31T19:56:10.274572Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1685568452.059513_997.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-05-31T21:27:32.059513Z', 'realm_epoch': 2}}] 2023-05-31T21:27:48.371 DEBUG:rgw_multi.tests:log_status={0: '1_1685566501.757183_260.1', 1: '1_1685566501.735717_251.1', 2: '1_1685562970.274572_93.1', 3: '1_1685568452.432716_1004.1'} 2023-05-31T21:27:48.371 DEBUG:rgw_multi.tests:sync_status={0: '1_1685566501.757183_260.1', 1: '1_1685566501.735717_251.1', 2: '1_1685562970.274572_93.1', 3: '1_1685568452.059513_997.1'} 2023-05-31T21:27:48.371 WARNING:rgw_multi.tests:zone a2 behind master: shard=3 master=1_1685568452.432716_1004.1 target=1_1685568452.059513_997.1 2023-05-31T21:27:53.372 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-31T21:27:53.373 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-31T21:27:53.373 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-05-31T21:27:53.432 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2023-05-31T21:27:53.432 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2023-05-31T21:27:53.477 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T21:27:53.471+0000 7fc57862e780 20 RGW-SYNC:meta: read sync status 2023-05-31T21:27:53.478 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T21:27:53.473+0000 7fc57862e780 20 RGW-SYNC:meta: read sync status 2023-05-31T21:27:53.493 INFO:teuthology.orchestra.run.smithi099.stdout:{ 2023-05-31T21:27:53.493 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_status": { 2023-05-31T21:27:53.494 INFO:teuthology.orchestra.run.smithi099.stdout: "info": { 2023-05-31T21:27:53.494 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "sync", 2023-05-31T21:27:53.494 INFO:teuthology.orchestra.run.smithi099.stdout: "num_shards": 4, 2023-05-31T21:27:53.494 INFO:teuthology.orchestra.run.smithi099.stdout: "period": "2e6123d2-d695-4350-9a01-61a455bc782b", 2023-05-31T21:27:53.494 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2023-05-31T21:27:53.495 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:27:53.495 INFO:teuthology.orchestra.run.smithi099.stdout: "markers": [ 2023-05-31T21:27:53.495 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T21:27:53.495 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 0, 2023-05-31T21:27:53.495 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T21:27:53.496 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2023-05-31T21:27:53.496 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1685566501.757183_260.1", 2023-05-31T21:27:53.496 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T21:27:53.496 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 1, 2023-05-31T21:27:53.496 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T21:27:53.496 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2023-05-31T20:55:01.757183Z", 2023-05-31T21:27:53.497 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2023-05-31T21:27:53.497 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T21:27:53.497 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:27:53.497 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T21:27:53.497 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 1, 2023-05-31T21:27:53.498 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T21:27:53.498 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2023-05-31T21:27:53.498 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1685566501.735717_251.1", 2023-05-31T21:27:53.498 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T21:27:53.498 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2023-05-31T21:27:53.499 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T21:27:53.499 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2023-05-31T20:55:01.735717Z", 2023-05-31T21:27:53.499 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2023-05-31T21:27:53.499 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T21:27:53.499 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:27:53.499 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T21:27:53.499 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 2, 2023-05-31T21:27:53.500 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T21:27:53.500 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2023-05-31T21:27:53.500 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1685562970.274572_93.1", 2023-05-31T21:27:53.501 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T21:27:53.502 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2023-05-31T21:27:53.502 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T21:27:53.502 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2023-05-31T19:56:10.274572Z", 2023-05-31T21:27:53.503 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2023-05-31T21:27:53.503 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T21:27:53.503 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:27:53.503 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T21:27:53.504 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 3, 2023-05-31T21:27:53.504 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T21:27:53.504 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2023-05-31T21:27:53.505 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1685568452.432716_1004.1", 2023-05-31T21:27:53.505 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T21:27:53.505 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2023-05-31T21:27:53.506 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T21:27:53.506 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2023-05-31T21:27:32.432716Z", 2023-05-31T21:27:53.506 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2023-05-31T21:27:53.507 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T21:27:53.507 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T21:27:53.507 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2023-05-31T21:27:53.508 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:27:53.508 INFO:teuthology.orchestra.run.smithi099.stdout: "full_sync": { 2023-05-31T21:27:53.508 INFO:teuthology.orchestra.run.smithi099.stdout: "total": 1, 2023-05-31T21:27:53.509 INFO:teuthology.orchestra.run.smithi099.stdout: "complete": 1 2023-05-31T21:27:53.509 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:27:53.509 INFO:teuthology.orchestra.run.smithi099.stdout: "current_time": "2023-05-31T21:27:53Z" 2023-05-31T21:27:53.510 INFO:teuthology.orchestra.run.smithi099.stdout:} 2023-05-31T21:27:53.511 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "2e6123d2-d695-4350-9a01-61a455bc782b", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1685566501.757183_260.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-05-31T20:55:01.757183Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1685566501.735717_251.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-05-31T20:55:01.735717Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1685562970.274572_93.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-05-31T19:56:10.274572Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1685568452.432716_1004.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-05-31T21:27:32.432716Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-05-31T21:27:53Z" } 2023-05-31T21:27:53.512 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1685566501.757183_260.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-05-31T20:55:01.757183Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1685566501.735717_251.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-05-31T20:55:01.735717Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1685562970.274572_93.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-05-31T19:56:10.274572Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1685568452.432716_1004.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-05-31T21:27:32.432716Z', 'realm_epoch': 2}}] 2023-05-31T21:27:53.512 DEBUG:rgw_multi.tests:log_status={0: '1_1685566501.757183_260.1', 1: '1_1685566501.735717_251.1', 2: '1_1685562970.274572_93.1', 3: '1_1685568452.432716_1004.1'} 2023-05-31T21:27:53.513 DEBUG:rgw_multi.tests:sync_status={0: '1_1685566501.757183_260.1', 1: '1_1685566501.735717_251.1', 2: '1_1685562970.274572_93.1', 3: '1_1685568452.432716_1004.1'} 2023-05-31T21:27:53.513 INFO:rgw_multi.tests:finish meta checkpoint for zone=a2 2023-05-31T21:27:53.513 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['datalog', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-31T21:27:53.514 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'datalog', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-31T21:27:53.514 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 datalog status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-05-31T21:27:53.563 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2023-05-31T21:27:53.563 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2023-05-31T21:27:53.624 INFO:teuthology.orchestra.run.smithi099.stdout:[ 2023-05-31T21:27:53.624 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T21:27:53.624 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "00000000000000000000:00000000000000012529", 2023-05-31T21:27:53.624 INFO:teuthology.orchestra.run.smithi099.stdout: "last_update": "2023-05-31T21:27:17.166209Z" 2023-05-31T21:27:53.625 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:27:53.625 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T21:27:53.625 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "00000000000000000000:00000000000000005869", 2023-05-31T21:27:53.625 INFO:teuthology.orchestra.run.smithi099.stdout: "last_update": "2023-05-31T21:27:32.682331Z" 2023-05-31T21:27:53.625 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:27:53.625 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T21:27:53.625 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "00000000000000000000:00000000000000016432", 2023-05-31T21:27:53.626 INFO:teuthology.orchestra.run.smithi099.stdout: "last_update": "2023-05-31T21:27:32.582372Z" 2023-05-31T21:27:53.626 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:27:53.626 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T21:27:53.626 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "00000000000000000000:00000000000000016158", 2023-05-31T21:27:53.626 INFO:teuthology.orchestra.run.smithi099.stdout: "last_update": "2023-05-31T21:27:32.662152Z" 2023-05-31T21:27:53.626 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T21:27:53.627 INFO:teuthology.orchestra.run.smithi099.stdout:] 2023-05-31T21:27:53.627 DEBUG:rgw_multi.tests:data markers for zone=a2 markers={0: '00000000000000000000:00000000000000012529', 1: '00000000000000000000:00000000000000005869', 2: '00000000000000000000:00000000000000016432', 3: '00000000000000000000:00000000000000016158'} 2023-05-31T21:27:53.628 INFO:rgw_multi.tests:starting data checkpoint for target_zone=a1 source_zone=a2 2023-05-31T21:27:53.628 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-31T21:27:53.629 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-31T21:27:53.629 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --source-zone a2 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-05-31T21:27:53.680 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2023-05-31T21:27:53.681 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2023-05-31T21:27:53.733 INFO:teuthology.orchestra.run.smithi099.stdout:{ 2023-05-31T21:27:53.733 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_status": { 2023-05-31T21:27:53.733 INFO:teuthology.orchestra.run.smithi099.stdout: "info": { 2023-05-31T21:27:53.734 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "sync", 2023-05-31T21:27:53.734 INFO:teuthology.orchestra.run.smithi099.stdout: "num_shards": 4, 2023-05-31T21:27:53.734 INFO:teuthology.orchestra.run.smithi099.stdout: "instance_id": 17941554979570305020 2023-05-31T21:27:53.734 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:27:53.734 INFO:teuthology.orchestra.run.smithi099.stdout: "markers": [ 2023-05-31T21:27:53.734 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T21:27:53.735 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 0, 2023-05-31T21:27:53.735 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T21:27:53.735 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "incremental-sync", 2023-05-31T21:27:53.735 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "00000000000000000000:00000000000000012529", 2023-05-31T21:27:53.735 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T21:27:53.736 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2023-05-31T21:27:53.736 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T21:27:53.736 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2023-05-31T21:27:17.166209Z" 2023-05-31T21:27:53.736 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T21:27:53.737 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:27:53.737 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T21:27:53.739 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 1, 2023-05-31T21:27:53.739 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T21:27:53.739 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "incremental-sync", 2023-05-31T21:27:53.742 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "00000000000000000000:00000000000000005869", 2023-05-31T21:27:53.742 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T21:27:53.742 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2023-05-31T21:27:53.742 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T21:27:53.742 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2023-05-31T21:27:32.682331Z" 2023-05-31T21:27:53.742 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T21:27:53.743 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:27:53.743 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T21:27:53.743 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 2, 2023-05-31T21:27:53.743 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T21:27:53.743 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "incremental-sync", 2023-05-31T21:27:53.743 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "00000000000000000000:00000000000000016432", 2023-05-31T21:27:53.744 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T21:27:53.744 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2023-05-31T21:27:53.744 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T21:27:53.744 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2023-05-31T21:27:32.582372Z" 2023-05-31T21:27:53.744 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T21:27:53.745 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:27:53.745 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T21:27:53.745 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 3, 2023-05-31T21:27:53.745 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T21:27:53.745 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "incremental-sync", 2023-05-31T21:27:53.746 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "00000000000000000000:00000000000000016158", 2023-05-31T21:27:53.746 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T21:27:53.746 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2023-05-31T21:27:53.746 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T21:27:53.746 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2023-05-31T21:27:32.662152Z" 2023-05-31T21:27:53.746 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T21:27:53.746 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T21:27:53.747 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2023-05-31T21:27:53.747 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:27:53.747 INFO:teuthology.orchestra.run.smithi099.stdout: "full_sync": { 2023-05-31T21:27:53.747 INFO:teuthology.orchestra.run.smithi099.stdout: "total": 0, 2023-05-31T21:27:53.748 INFO:teuthology.orchestra.run.smithi099.stdout: "complete": 0 2023-05-31T21:27:53.748 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:27:53.748 INFO:teuthology.orchestra.run.smithi099.stdout: "current_time": "2023-05-31T21:27:53Z" 2023-05-31T21:27:53.748 INFO:teuthology.orchestra.run.smithi099.stdout:} 2023-05-31T21:27:53.749 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 17941554979570305020 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000012529", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-05-31T21:27:17.166209Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000005869", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-05-31T21:27:32.682331Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000016432", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-05-31T21:27:32.582372Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000016158", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-05-31T21:27:32.662152Z" } } ] }, "full_sync": { "total": 0, "complete": 0 }, "current_time": "2023-05-31T21:27:53Z" } 2023-05-31T21:27:53.750 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000012529', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-05-31T21:27:17.166209Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000005869', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-05-31T21:27:32.682331Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000016432', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-05-31T21:27:32.582372Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000016158', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-05-31T21:27:32.662152Z'}}] 2023-05-31T21:27:53.750 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000012529', 1: '00000000000000000000:00000000000000005869', 2: '00000000000000000000:00000000000000016432', 3: '00000000000000000000:00000000000000016158'} 2023-05-31T21:27:53.750 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000012529', 1: '00000000000000000000:00000000000000005869', 2: '00000000000000000000:00000000000000016432', 3: '00000000000000000000:00000000000000016158'} 2023-05-31T21:27:53.751 INFO:rgw_multi.tests:finished data checkpoint for target_zone=a1 source_zone=a2 2023-05-31T21:27:53.751 DEBUG:boto:path=/pklxkd-70/ 2023-05-31T21:27:53.751 DEBUG:boto:auth_path=/pklxkd-70/ 2023-05-31T21:27:53.752 DEBUG:boto:Method: HEAD 2023-05-31T21:27:53.753 DEBUG:boto:Path: /pklxkd-70/ 2023-05-31T21:27:53.753 DEBUG:boto:Data: 2023-05-31T21:27:53.754 DEBUG:boto:Headers: {} 2023-05-31T21:27:53.755 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T21:27:53.755 DEBUG:boto:Port: 8000 2023-05-31T21:27:53.756 DEBUG:boto:Params: {} 2023-05-31T21:27:53.756 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2023-05-31T21:27:53.757 DEBUG:boto:Token: None 2023-05-31T21:27:53.757 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 21:27:53 GMT /pklxkd-70/ 2023-05-31T21:27:53.758 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:kwaFxAtRNPlonX7eFfdh27/+QFc= 2023-05-31T21:27:53.758 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 21:27:53 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:kwaFxAtRNPlonX7eFfdh27/+QFc=', 'Content-Length': '0'} 2023-05-31T21:27:53.761 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '6'), ('X-RGW-Bytes-Used', '36'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000001cbba17351db2502-006477bbd9-264e-a1'), ('Content-Length', '0'), ('Date', 'Wed, 31 May 2023 21:27:53 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:27:53.762 DEBUG:boto:path=/pklxkd-70/obj7 2023-05-31T21:27:53.763 DEBUG:boto:auth_path=/pklxkd-70/obj7 2023-05-31T21:27:53.764 DEBUG:boto:Method: HEAD 2023-05-31T21:27:53.765 DEBUG:boto:Path: /pklxkd-70/obj7 2023-05-31T21:27:53.765 DEBUG:boto:Data: 2023-05-31T21:27:53.766 DEBUG:boto:Headers: {} 2023-05-31T21:27:53.767 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T21:27:53.768 DEBUG:boto:Port: 8000 2023-05-31T21:27:53.768 DEBUG:boto:Params: {} 2023-05-31T21:27:53.768 DEBUG:boto:Token: None 2023-05-31T21:27:53.769 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 21:27:53 GMT /pklxkd-70/obj7 2023-05-31T21:27:53.770 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:ur2Rxhc1IQa+JWZ+BQDHFIqOaAQ= 2023-05-31T21:27:53.770 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 21:27:53 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:ur2Rxhc1IQa+JWZ+BQDHFIqOaAQ=', 'Content-Length': '0'} 2023-05-31T21:27:53.772 DEBUG:boto:Response headers: [('Content-Length', '219'), ('x-amz-request-id', 'tx0000022c0ee03c5913e74-006477bbd9-264e-a1'), ('Accept-Ranges', 'bytes'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 21:27:53 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:27:53.772 DEBUG:boto:path=/pklxkd-70/obj8 2023-05-31T21:27:53.773 DEBUG:boto:auth_path=/pklxkd-70/obj8 2023-05-31T21:27:53.773 DEBUG:boto:Method: HEAD 2023-05-31T21:27:53.774 DEBUG:boto:Path: /pklxkd-70/obj8 2023-05-31T21:27:53.774 DEBUG:boto:Data: 2023-05-31T21:27:53.775 DEBUG:boto:Headers: {} 2023-05-31T21:27:53.775 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T21:27:53.776 DEBUG:boto:Port: 8000 2023-05-31T21:27:53.776 DEBUG:boto:Params: {} 2023-05-31T21:27:53.777 DEBUG:boto:Token: None 2023-05-31T21:27:53.778 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 21:27:53 GMT /pklxkd-70/obj8 2023-05-31T21:27:53.778 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:AkXOqVjZZhWu8qamPZW+Au/q1Ac= 2023-05-31T21:27:53.779 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 21:27:53 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:AkXOqVjZZhWu8qamPZW+Au/q1Ac=', 'Content-Length': '0'} 2023-05-31T21:27:53.781 DEBUG:boto:Response headers: [('Content-Length', '219'), ('x-amz-request-id', 'tx000000a3f8cb4fb5f977b-006477bbd9-264e-a1'), ('Accept-Ranges', 'bytes'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 21:27:53 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:27:53.781 DEBUG:boto:path=/pklxkd-70/obj9 2023-05-31T21:27:53.782 DEBUG:boto:auth_path=/pklxkd-70/obj9 2023-05-31T21:27:53.782 DEBUG:boto:Method: HEAD 2023-05-31T21:27:53.783 DEBUG:boto:Path: /pklxkd-70/obj9 2023-05-31T21:27:53.783 DEBUG:boto:Data: 2023-05-31T21:27:53.783 DEBUG:boto:Headers: {} 2023-05-31T21:27:53.784 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T21:27:53.785 DEBUG:boto:Port: 8000 2023-05-31T21:27:53.785 DEBUG:boto:Params: {} 2023-05-31T21:27:53.785 DEBUG:boto:Token: None 2023-05-31T21:27:53.786 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 21:27:53 GMT /pklxkd-70/obj9 2023-05-31T21:27:53.786 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:lyBwC+Mm+QBoqZaPtwmYwxdX490= 2023-05-31T21:27:53.787 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 21:27:53 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:lyBwC+Mm+QBoqZaPtwmYwxdX490=', 'Content-Length': '0'} 2023-05-31T21:27:53.788 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 21:27:32 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000f1c2bd5709320a44-006477bbd9-264e-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 21:27:53 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:27:53.789 DEBUG:boto:path=/pklxkd-70/obj9 2023-05-31T21:27:53.789 DEBUG:boto:auth_path=/pklxkd-70/obj9 2023-05-31T21:27:53.790 DEBUG:boto:Method: GET 2023-05-31T21:27:53.791 DEBUG:boto:Path: /pklxkd-70/obj9 2023-05-31T21:27:53.792 DEBUG:boto:Data: 2023-05-31T21:27:53.792 DEBUG:boto:Headers: {} 2023-05-31T21:27:53.793 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T21:27:53.793 DEBUG:boto:Port: 8000 2023-05-31T21:27:53.794 DEBUG:boto:Params: {} 2023-05-31T21:27:53.795 DEBUG:boto:Token: None 2023-05-31T21:27:53.796 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 21:27:53 GMT /pklxkd-70/obj9 2023-05-31T21:27:53.797 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:ye/nHgjJLYACDsCARUVstsQWsyM= 2023-05-31T21:27:53.798 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 21:27:53 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:ye/nHgjJLYACDsCARUVstsQWsyM=', 'Content-Length': '0'} 2023-05-31T21:27:53.800 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 21:27:32 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000063502152363845d0-006477bbd9-264e-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 21:27:53 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:27:53.801 DEBUG:boto:path=/pklxkd-70/obj10 2023-05-31T21:27:53.801 DEBUG:boto:auth_path=/pklxkd-70/obj10 2023-05-31T21:27:53.802 DEBUG:boto:Method: HEAD 2023-05-31T21:27:53.802 DEBUG:boto:Path: /pklxkd-70/obj10 2023-05-31T21:27:53.803 DEBUG:boto:Data: 2023-05-31T21:27:53.803 DEBUG:boto:Headers: {} 2023-05-31T21:27:53.804 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T21:27:53.804 DEBUG:boto:Port: 8000 2023-05-31T21:27:53.805 DEBUG:boto:Params: {} 2023-05-31T21:27:53.805 DEBUG:boto:Token: None 2023-05-31T21:27:53.806 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 21:27:53 GMT /pklxkd-70/obj10 2023-05-31T21:27:53.806 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:BTsgrNK3tBO6uDvGiSLigX4rOPw= 2023-05-31T21:27:53.807 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 21:27:53 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:BTsgrNK3tBO6uDvGiSLigX4rOPw=', 'Content-Length': '0'} 2023-05-31T21:27:53.808 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 21:27:32 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000de56fbe2480d2b86-006477bbd9-264e-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 21:27:53 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:27:53.809 DEBUG:boto:path=/pklxkd-70/obj10 2023-05-31T21:27:53.810 DEBUG:boto:auth_path=/pklxkd-70/obj10 2023-05-31T21:27:53.810 DEBUG:boto:Method: GET 2023-05-31T21:27:53.811 DEBUG:boto:Path: /pklxkd-70/obj10 2023-05-31T21:27:53.811 DEBUG:boto:Data: 2023-05-31T21:27:53.811 DEBUG:boto:Headers: {} 2023-05-31T21:27:53.812 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T21:27:53.812 DEBUG:boto:Port: 8000 2023-05-31T21:27:53.812 DEBUG:boto:Params: {} 2023-05-31T21:27:53.813 DEBUG:boto:Token: None 2023-05-31T21:27:53.813 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 21:27:53 GMT /pklxkd-70/obj10 2023-05-31T21:27:53.814 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:cVemj6kWsSBBNm8JKck2dZHfGVI= 2023-05-31T21:27:53.815 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 21:27:53 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:cVemj6kWsSBBNm8JKck2dZHfGVI=', 'Content-Length': '0'} 2023-05-31T21:27:53.817 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 21:27:32 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000b7c5476712932276-006477bbd9-264e-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 21:27:53 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:27:53.864 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'group', 'remove', '--group-id', 'sync-bucket', '--bucket', 'pklxkd-70', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-05-31T21:27:53.865 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'group', 'remove', '--group-id', 'sync-bucket', '--bucket', 'pklxkd-70', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-05-31T21:27:53.866 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync group remove --group-id sync-bucket --bucket pklxkd-70 --cluster c1 --debug-rgw 0 --debug-ms 0 2023-05-31T21:27:53.916 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2023-05-31T21:27:53.917 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2023-05-31T21:27:53.993 INFO:teuthology.orchestra.run.smithi099.stdout:{"sync_policy":{"groups":[]}} 2023-05-31T21:27:53.993 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'group', 'remove', '--group-id', 'sync-group', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-05-31T21:27:53.994 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'group', 'remove', '--group-id', 'sync-group', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-05-31T21:27:53.994 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync group remove --group-id sync-group --cluster c1 --debug-rgw 0 --debug-ms 0 2023-05-31T21:27:54.043 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2023-05-31T21:27:54.043 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2023-05-31T21:27:54.120 INFO:teuthology.orchestra.run.smithi099.stdout:{"sync_policy":{"groups":[]}} 2023-05-31T21:27:54.121 INFO:tasks.rgw_multisite_tests:test_sync_different_buckets: ... ok 2023-05-31T21:27:54.123 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-31T21:27:54.123 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-31T21:27:54.124 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-05-31T21:27:54.173 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2023-05-31T21:27:54.173 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2023-05-31T21:27:54.216 INFO:teuthology.orchestra.run.smithi099.stderr:No --period given, using current period=2e6123d2-d695-4350-9a01-61a455bc782b 2023-05-31T21:27:54.232 INFO:teuthology.orchestra.run.smithi099.stdout:[ 2023-05-31T21:27:54.232 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T21:27:54.233 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1685566501.757183_260.1", 2023-05-31T21:27:54.233 INFO:teuthology.orchestra.run.smithi099.stdout: "last_update": "2023-05-31T20:55:01.757183Z" 2023-05-31T21:27:54.233 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:27:54.234 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T21:27:54.234 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1685566501.735717_251.1", 2023-05-31T21:27:54.234 INFO:teuthology.orchestra.run.smithi099.stdout: "last_update": "2023-05-31T20:55:01.735717Z" 2023-05-31T21:27:54.234 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:27:54.234 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T21:27:54.234 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1685562970.274572_93.1", 2023-05-31T21:27:54.234 INFO:teuthology.orchestra.run.smithi099.stdout: "last_update": "2023-05-31T19:56:10.274572Z" 2023-05-31T21:27:54.235 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:27:54.235 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T21:27:54.235 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1685568473.966785_1006.1", 2023-05-31T21:27:54.235 INFO:teuthology.orchestra.run.smithi099.stdout: "last_update": "2023-05-31T21:27:53.966785Z" 2023-05-31T21:27:54.235 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T21:27:54.236 INFO:teuthology.orchestra.run.smithi099.stdout:] 2023-05-31T21:27:54.237 DEBUG:rgw_multi.tests:master meta markers={0: '1_1685566501.757183_260.1', 1: '1_1685566501.735717_251.1', 2: '1_1685562970.274572_93.1', 3: '1_1685568473.966785_1006.1'} 2023-05-31T21:27:54.238 INFO:rgw_multi.tests:starting meta checkpoint for zone=a2 2023-05-31T21:27:54.238 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-31T21:27:54.239 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-31T21:27:54.239 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-05-31T21:27:54.289 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2023-05-31T21:27:54.289 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2023-05-31T21:27:54.342 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T21:27:54.338+0000 7f3a387d8780 20 RGW-SYNC:meta: read sync status 2023-05-31T21:27:54.342 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T21:27:54.338+0000 7f3a387d8780 20 RGW-SYNC:meta: read sync status 2023-05-31T21:27:54.362 INFO:teuthology.orchestra.run.smithi099.stdout:{ 2023-05-31T21:27:54.362 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_status": { 2023-05-31T21:27:54.362 INFO:teuthology.orchestra.run.smithi099.stdout: "info": { 2023-05-31T21:27:54.363 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "sync", 2023-05-31T21:27:54.363 INFO:teuthology.orchestra.run.smithi099.stdout: "num_shards": 4, 2023-05-31T21:27:54.363 INFO:teuthology.orchestra.run.smithi099.stdout: "period": "2e6123d2-d695-4350-9a01-61a455bc782b", 2023-05-31T21:27:54.363 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2023-05-31T21:27:54.363 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:27:54.363 INFO:teuthology.orchestra.run.smithi099.stdout: "markers": [ 2023-05-31T21:27:54.363 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T21:27:54.364 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 0, 2023-05-31T21:27:54.364 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T21:27:54.364 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2023-05-31T21:27:54.364 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1685566501.757183_260.1", 2023-05-31T21:27:54.364 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T21:27:54.364 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 1, 2023-05-31T21:27:54.365 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T21:27:54.365 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2023-05-31T20:55:01.757183Z", 2023-05-31T21:27:54.365 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2023-05-31T21:27:54.365 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T21:27:54.365 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:27:54.365 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T21:27:54.365 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 1, 2023-05-31T21:27:54.366 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T21:27:54.366 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2023-05-31T21:27:54.366 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1685566501.735717_251.1", 2023-05-31T21:27:54.366 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T21:27:54.366 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2023-05-31T21:27:54.366 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T21:27:54.366 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2023-05-31T20:55:01.735717Z", 2023-05-31T21:27:54.367 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2023-05-31T21:27:54.367 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T21:27:54.367 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:27:54.367 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T21:27:54.367 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 2, 2023-05-31T21:27:54.367 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T21:27:54.367 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2023-05-31T21:27:54.368 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1685562970.274572_93.1", 2023-05-31T21:27:54.368 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T21:27:54.368 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2023-05-31T21:27:54.368 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T21:27:54.368 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2023-05-31T19:56:10.274572Z", 2023-05-31T21:27:54.369 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2023-05-31T21:27:54.369 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T21:27:54.369 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:27:54.369 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T21:27:54.369 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 3, 2023-05-31T21:27:54.369 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T21:27:54.369 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2023-05-31T21:27:54.370 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1685568452.432716_1004.1", 2023-05-31T21:27:54.370 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T21:27:54.370 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2023-05-31T21:27:54.370 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T21:27:54.370 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2023-05-31T21:27:32.432716Z", 2023-05-31T21:27:54.370 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2023-05-31T21:27:54.370 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T21:27:54.371 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T21:27:54.371 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2023-05-31T21:27:54.371 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:27:54.372 INFO:teuthology.orchestra.run.smithi099.stdout: "full_sync": { 2023-05-31T21:27:54.372 INFO:teuthology.orchestra.run.smithi099.stdout: "total": 1, 2023-05-31T21:27:54.372 INFO:teuthology.orchestra.run.smithi099.stdout: "complete": 1 2023-05-31T21:27:54.372 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:27:54.373 INFO:teuthology.orchestra.run.smithi099.stdout: "current_time": "2023-05-31T21:27:54Z" 2023-05-31T21:27:54.373 INFO:teuthology.orchestra.run.smithi099.stdout:} 2023-05-31T21:27:54.374 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "2e6123d2-d695-4350-9a01-61a455bc782b", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1685566501.757183_260.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-05-31T20:55:01.757183Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1685566501.735717_251.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-05-31T20:55:01.735717Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1685562970.274572_93.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-05-31T19:56:10.274572Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1685568452.432716_1004.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-05-31T21:27:32.432716Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-05-31T21:27:54Z" } 2023-05-31T21:27:54.374 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1685566501.757183_260.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-05-31T20:55:01.757183Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1685566501.735717_251.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-05-31T20:55:01.735717Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1685562970.274572_93.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-05-31T19:56:10.274572Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1685568452.432716_1004.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-05-31T21:27:32.432716Z', 'realm_epoch': 2}}] 2023-05-31T21:27:54.374 DEBUG:rgw_multi.tests:log_status={0: '1_1685566501.757183_260.1', 1: '1_1685566501.735717_251.1', 2: '1_1685562970.274572_93.1', 3: '1_1685568473.966785_1006.1'} 2023-05-31T21:27:54.375 DEBUG:rgw_multi.tests:sync_status={0: '1_1685566501.757183_260.1', 1: '1_1685566501.735717_251.1', 2: '1_1685562970.274572_93.1', 3: '1_1685568452.432716_1004.1'} 2023-05-31T21:27:54.375 WARNING:rgw_multi.tests:zone a2 behind master: shard=3 master=1_1685568473.966785_1006.1 target=1_1685568452.432716_1004.1 2023-05-31T21:27:59.377 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-31T21:27:59.378 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-31T21:27:59.379 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-05-31T21:27:59.438 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2023-05-31T21:27:59.439 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2023-05-31T21:27:59.479 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T21:27:59.475+0000 7f71a0c90780 20 RGW-SYNC:meta: read sync status 2023-05-31T21:27:59.480 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T21:27:59.476+0000 7f71a0c90780 20 RGW-SYNC:meta: read sync status 2023-05-31T21:27:59.501 INFO:teuthology.orchestra.run.smithi099.stdout:{ 2023-05-31T21:27:59.502 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_status": { 2023-05-31T21:27:59.502 INFO:teuthology.orchestra.run.smithi099.stdout: "info": { 2023-05-31T21:27:59.503 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "sync", 2023-05-31T21:27:59.503 INFO:teuthology.orchestra.run.smithi099.stdout: "num_shards": 4, 2023-05-31T21:27:59.503 INFO:teuthology.orchestra.run.smithi099.stdout: "period": "2e6123d2-d695-4350-9a01-61a455bc782b", 2023-05-31T21:27:59.504 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2023-05-31T21:27:59.504 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:27:59.504 INFO:teuthology.orchestra.run.smithi099.stdout: "markers": [ 2023-05-31T21:27:59.505 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T21:27:59.505 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 0, 2023-05-31T21:27:59.505 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T21:27:59.505 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2023-05-31T21:27:59.506 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1685566501.757183_260.1", 2023-05-31T21:27:59.506 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T21:27:59.507 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 1, 2023-05-31T21:27:59.507 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T21:27:59.508 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2023-05-31T20:55:01.757183Z", 2023-05-31T21:27:59.508 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2023-05-31T21:27:59.508 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T21:27:59.509 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:27:59.509 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T21:27:59.509 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 1, 2023-05-31T21:27:59.510 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T21:27:59.510 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2023-05-31T21:27:59.510 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1685566501.735717_251.1", 2023-05-31T21:27:59.511 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T21:27:59.511 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2023-05-31T21:27:59.511 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T21:27:59.511 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2023-05-31T20:55:01.735717Z", 2023-05-31T21:27:59.512 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2023-05-31T21:27:59.512 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T21:27:59.512 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:27:59.512 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T21:27:59.513 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 2, 2023-05-31T21:27:59.513 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T21:27:59.513 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2023-05-31T21:27:59.514 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1685562970.274572_93.1", 2023-05-31T21:27:59.514 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T21:27:59.514 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2023-05-31T21:27:59.515 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T21:27:59.515 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2023-05-31T19:56:10.274572Z", 2023-05-31T21:27:59.515 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2023-05-31T21:27:59.515 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T21:27:59.515 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:27:59.515 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T21:27:59.516 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 3, 2023-05-31T21:27:59.516 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T21:27:59.516 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2023-05-31T21:27:59.516 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1685568452.432716_1004.1", 2023-05-31T21:27:59.516 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T21:27:59.516 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2023-05-31T21:27:59.517 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T21:27:59.517 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2023-05-31T21:27:32.432716Z", 2023-05-31T21:27:59.517 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2023-05-31T21:27:59.517 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T21:27:59.518 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T21:27:59.518 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2023-05-31T21:27:59.518 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:27:59.518 INFO:teuthology.orchestra.run.smithi099.stdout: "full_sync": { 2023-05-31T21:27:59.518 INFO:teuthology.orchestra.run.smithi099.stdout: "total": 1, 2023-05-31T21:27:59.518 INFO:teuthology.orchestra.run.smithi099.stdout: "complete": 1 2023-05-31T21:27:59.519 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:27:59.519 INFO:teuthology.orchestra.run.smithi099.stdout: "current_time": "2023-05-31T21:27:59Z" 2023-05-31T21:27:59.519 INFO:teuthology.orchestra.run.smithi099.stdout:} 2023-05-31T21:27:59.520 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "2e6123d2-d695-4350-9a01-61a455bc782b", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1685566501.757183_260.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-05-31T20:55:01.757183Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1685566501.735717_251.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-05-31T20:55:01.735717Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1685562970.274572_93.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-05-31T19:56:10.274572Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1685568452.432716_1004.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-05-31T21:27:32.432716Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-05-31T21:27:59Z" } 2023-05-31T21:27:59.521 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1685566501.757183_260.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-05-31T20:55:01.757183Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1685566501.735717_251.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-05-31T20:55:01.735717Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1685562970.274572_93.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-05-31T19:56:10.274572Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1685568452.432716_1004.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-05-31T21:27:32.432716Z', 'realm_epoch': 2}}] 2023-05-31T21:27:59.521 DEBUG:rgw_multi.tests:log_status={0: '1_1685566501.757183_260.1', 1: '1_1685566501.735717_251.1', 2: '1_1685562970.274572_93.1', 3: '1_1685568473.966785_1006.1'} 2023-05-31T21:27:59.522 DEBUG:rgw_multi.tests:sync_status={0: '1_1685566501.757183_260.1', 1: '1_1685566501.735717_251.1', 2: '1_1685562970.274572_93.1', 3: '1_1685568452.432716_1004.1'} 2023-05-31T21:27:59.522 WARNING:rgw_multi.tests:zone a2 behind master: shard=3 master=1_1685568473.966785_1006.1 target=1_1685568452.432716_1004.1 2023-05-31T21:28:04.523 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-31T21:28:04.524 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-31T21:28:04.525 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-05-31T21:28:04.582 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2023-05-31T21:28:04.583 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2023-05-31T21:28:04.619 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T21:28:04.615+0000 7f81db60d780 20 RGW-SYNC:meta: read sync status 2023-05-31T21:28:04.620 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T21:28:04.616+0000 7f81db60d780 20 RGW-SYNC:meta: read sync status 2023-05-31T21:28:04.634 INFO:teuthology.orchestra.run.smithi099.stdout:{ 2023-05-31T21:28:04.634 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_status": { 2023-05-31T21:28:04.635 INFO:teuthology.orchestra.run.smithi099.stdout: "info": { 2023-05-31T21:28:04.635 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "sync", 2023-05-31T21:28:04.635 INFO:teuthology.orchestra.run.smithi099.stdout: "num_shards": 4, 2023-05-31T21:28:04.635 INFO:teuthology.orchestra.run.smithi099.stdout: "period": "2e6123d2-d695-4350-9a01-61a455bc782b", 2023-05-31T21:28:04.635 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2023-05-31T21:28:04.635 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:28:04.636 INFO:teuthology.orchestra.run.smithi099.stdout: "markers": [ 2023-05-31T21:28:04.636 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T21:28:04.636 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 0, 2023-05-31T21:28:04.636 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T21:28:04.636 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2023-05-31T21:28:04.636 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1685566501.757183_260.1", 2023-05-31T21:28:04.637 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T21:28:04.637 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 1, 2023-05-31T21:28:04.637 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T21:28:04.637 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2023-05-31T20:55:01.757183Z", 2023-05-31T21:28:04.637 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2023-05-31T21:28:04.637 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T21:28:04.637 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:28:04.638 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T21:28:04.638 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 1, 2023-05-31T21:28:04.638 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T21:28:04.638 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2023-05-31T21:28:04.638 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1685566501.735717_251.1", 2023-05-31T21:28:04.638 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T21:28:04.638 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2023-05-31T21:28:04.639 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T21:28:04.639 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2023-05-31T20:55:01.735717Z", 2023-05-31T21:28:04.639 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2023-05-31T21:28:04.639 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T21:28:04.639 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:28:04.639 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T21:28:04.639 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 2, 2023-05-31T21:28:04.640 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T21:28:04.640 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2023-05-31T21:28:04.640 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1685562970.274572_93.1", 2023-05-31T21:28:04.640 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T21:28:04.640 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2023-05-31T21:28:04.640 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T21:28:04.640 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2023-05-31T19:56:10.274572Z", 2023-05-31T21:28:04.641 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2023-05-31T21:28:04.641 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T21:28:04.641 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:28:04.641 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T21:28:04.641 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 3, 2023-05-31T21:28:04.641 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T21:28:04.641 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2023-05-31T21:28:04.642 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1685568452.432716_1004.1", 2023-05-31T21:28:04.642 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T21:28:04.642 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2023-05-31T21:28:04.642 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T21:28:04.642 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2023-05-31T21:27:32.432716Z", 2023-05-31T21:28:04.642 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2023-05-31T21:28:04.642 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T21:28:04.643 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T21:28:04.643 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2023-05-31T21:28:04.643 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:28:04.643 INFO:teuthology.orchestra.run.smithi099.stdout: "full_sync": { 2023-05-31T21:28:04.643 INFO:teuthology.orchestra.run.smithi099.stdout: "total": 1, 2023-05-31T21:28:04.643 INFO:teuthology.orchestra.run.smithi099.stdout: "complete": 1 2023-05-31T21:28:04.643 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:28:04.644 INFO:teuthology.orchestra.run.smithi099.stdout: "current_time": "2023-05-31T21:28:04Z" 2023-05-31T21:28:04.644 INFO:teuthology.orchestra.run.smithi099.stdout:} 2023-05-31T21:28:04.645 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "2e6123d2-d695-4350-9a01-61a455bc782b", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1685566501.757183_260.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-05-31T20:55:01.757183Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1685566501.735717_251.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-05-31T20:55:01.735717Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1685562970.274572_93.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-05-31T19:56:10.274572Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1685568452.432716_1004.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-05-31T21:27:32.432716Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-05-31T21:28:04Z" } 2023-05-31T21:28:04.645 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1685566501.757183_260.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-05-31T20:55:01.757183Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1685566501.735717_251.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-05-31T20:55:01.735717Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1685562970.274572_93.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-05-31T19:56:10.274572Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1685568452.432716_1004.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-05-31T21:27:32.432716Z', 'realm_epoch': 2}}] 2023-05-31T21:28:04.646 DEBUG:rgw_multi.tests:log_status={0: '1_1685566501.757183_260.1', 1: '1_1685566501.735717_251.1', 2: '1_1685562970.274572_93.1', 3: '1_1685568473.966785_1006.1'} 2023-05-31T21:28:04.646 DEBUG:rgw_multi.tests:sync_status={0: '1_1685566501.757183_260.1', 1: '1_1685566501.735717_251.1', 2: '1_1685562970.274572_93.1', 3: '1_1685568452.432716_1004.1'} 2023-05-31T21:28:04.646 WARNING:rgw_multi.tests:zone a2 behind master: shard=3 master=1_1685568473.966785_1006.1 target=1_1685568452.432716_1004.1 2023-05-31T21:28:09.648 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-31T21:28:09.649 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-31T21:28:09.649 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-05-31T21:28:09.705 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2023-05-31T21:28:09.705 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2023-05-31T21:28:09.759 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T21:28:09.755+0000 7fdf7277f780 20 RGW-SYNC:meta: read sync status 2023-05-31T21:28:09.760 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T21:28:09.756+0000 7fdf7277f780 20 RGW-SYNC:meta: read sync status 2023-05-31T21:28:09.776 INFO:teuthology.orchestra.run.smithi099.stdout:{ 2023-05-31T21:28:09.776 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_status": { 2023-05-31T21:28:09.776 INFO:teuthology.orchestra.run.smithi099.stdout: "info": { 2023-05-31T21:28:09.776 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "sync", 2023-05-31T21:28:09.777 INFO:teuthology.orchestra.run.smithi099.stdout: "num_shards": 4, 2023-05-31T21:28:09.777 INFO:teuthology.orchestra.run.smithi099.stdout: "period": "2e6123d2-d695-4350-9a01-61a455bc782b", 2023-05-31T21:28:09.777 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2023-05-31T21:28:09.777 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:28:09.777 INFO:teuthology.orchestra.run.smithi099.stdout: "markers": [ 2023-05-31T21:28:09.777 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T21:28:09.778 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 0, 2023-05-31T21:28:09.778 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T21:28:09.778 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2023-05-31T21:28:09.778 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1685566501.757183_260.1", 2023-05-31T21:28:09.778 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T21:28:09.778 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 1, 2023-05-31T21:28:09.778 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T21:28:09.779 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2023-05-31T20:55:01.757183Z", 2023-05-31T21:28:09.779 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2023-05-31T21:28:09.779 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T21:28:09.779 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:28:09.779 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T21:28:09.779 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 1, 2023-05-31T21:28:09.779 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T21:28:09.780 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2023-05-31T21:28:09.780 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1685566501.735717_251.1", 2023-05-31T21:28:09.780 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T21:28:09.780 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2023-05-31T21:28:09.780 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T21:28:09.780 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2023-05-31T20:55:01.735717Z", 2023-05-31T21:28:09.781 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2023-05-31T21:28:09.781 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T21:28:09.781 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:28:09.781 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T21:28:09.781 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 2, 2023-05-31T21:28:09.781 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T21:28:09.781 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2023-05-31T21:28:09.782 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1685562970.274572_93.1", 2023-05-31T21:28:09.782 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T21:28:09.782 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2023-05-31T21:28:09.782 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T21:28:09.782 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2023-05-31T19:56:10.274572Z", 2023-05-31T21:28:09.782 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2023-05-31T21:28:09.782 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T21:28:09.783 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:28:09.783 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T21:28:09.783 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 3, 2023-05-31T21:28:09.783 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T21:28:09.783 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2023-05-31T21:28:09.783 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1685568473.966785_1006.1", 2023-05-31T21:28:09.783 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T21:28:09.784 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2023-05-31T21:28:09.784 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T21:28:09.784 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2023-05-31T21:27:53.966785Z", 2023-05-31T21:28:09.785 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2023-05-31T21:28:09.785 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T21:28:09.785 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T21:28:09.785 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2023-05-31T21:28:09.785 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:28:09.785 INFO:teuthology.orchestra.run.smithi099.stdout: "full_sync": { 2023-05-31T21:28:09.785 INFO:teuthology.orchestra.run.smithi099.stdout: "total": 1, 2023-05-31T21:28:09.786 INFO:teuthology.orchestra.run.smithi099.stdout: "complete": 1 2023-05-31T21:28:09.786 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:28:09.786 INFO:teuthology.orchestra.run.smithi099.stdout: "current_time": "2023-05-31T21:28:09Z" 2023-05-31T21:28:09.786 INFO:teuthology.orchestra.run.smithi099.stdout:} 2023-05-31T21:28:09.787 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "2e6123d2-d695-4350-9a01-61a455bc782b", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1685566501.757183_260.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-05-31T20:55:01.757183Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1685566501.735717_251.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-05-31T20:55:01.735717Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1685562970.274572_93.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-05-31T19:56:10.274572Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1685568473.966785_1006.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-05-31T21:27:53.966785Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-05-31T21:28:09Z" } 2023-05-31T21:28:09.787 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1685566501.757183_260.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-05-31T20:55:01.757183Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1685566501.735717_251.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-05-31T20:55:01.735717Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1685562970.274572_93.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-05-31T19:56:10.274572Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1685568473.966785_1006.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-05-31T21:27:53.966785Z', 'realm_epoch': 2}}] 2023-05-31T21:28:09.788 DEBUG:rgw_multi.tests:log_status={0: '1_1685566501.757183_260.1', 1: '1_1685566501.735717_251.1', 2: '1_1685562970.274572_93.1', 3: '1_1685568473.966785_1006.1'} 2023-05-31T21:28:09.788 DEBUG:rgw_multi.tests:sync_status={0: '1_1685566501.757183_260.1', 1: '1_1685566501.735717_251.1', 2: '1_1685562970.274572_93.1', 3: '1_1685568473.966785_1006.1'} 2023-05-31T21:28:09.789 INFO:rgw_multi.tests:finish meta checkpoint for zone=a2 2023-05-31T21:28:09.789 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'policy', 'get', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-05-31T21:28:09.789 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'policy', 'get', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-05-31T21:28:09.790 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync policy get --cluster c1 --debug-rgw 0 --debug-ms 0 2023-05-31T21:28:09.836 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2023-05-31T21:28:09.836 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2023-05-31T21:28:09.906 INFO:teuthology.orchestra.run.smithi099.stdout:{"groups":[]} 2023-05-31T21:28:09.907 INFO:rgw_multi.tests:create bucket zone=a1 name=pklxkd-72 2023-05-31T21:28:09.907 DEBUG:boto:path=/pklxkd-72/ 2023-05-31T21:28:09.907 DEBUG:boto:auth_path=/pklxkd-72/ 2023-05-31T21:28:09.908 DEBUG:boto:Method: PUT 2023-05-31T21:28:09.908 DEBUG:boto:Path: /pklxkd-72/ 2023-05-31T21:28:09.908 DEBUG:boto:Data: 2023-05-31T21:28:09.909 DEBUG:boto:Headers: {} 2023-05-31T21:28:09.909 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T21:28:09.909 DEBUG:boto:Port: 8000 2023-05-31T21:28:09.910 DEBUG:boto:Params: {} 2023-05-31T21:28:09.910 DEBUG:boto:Token: None 2023-05-31T21:28:09.911 DEBUG:boto:StringToSign: PUT Wed, 31 May 2023 21:28:09 GMT /pklxkd-72/ 2023-05-31T21:28:09.911 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:Cy1XqspsHBkkNuz52tx5tVjAXPo= 2023-05-31T21:28:09.912 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 21:28:09 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:Cy1XqspsHBkkNuz52tx5tVjAXPo=', 'Content-Length': '0'} 2023-05-31T21:28:09.942 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx000009f02d8f9ebdd14e9-006477bbe9-264e-a1'), ('Content-Length', '0'), ('Date', 'Wed, 31 May 2023 21:28:09 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:28:09.942 INFO:rgw_multi.tests:create bucket zone=a1 name=pklxkd-73 2023-05-31T21:28:09.943 DEBUG:boto:path=/pklxkd-73/ 2023-05-31T21:28:09.943 DEBUG:boto:auth_path=/pklxkd-73/ 2023-05-31T21:28:09.943 DEBUG:boto:Method: PUT 2023-05-31T21:28:09.944 DEBUG:boto:Path: /pklxkd-73/ 2023-05-31T21:28:09.944 DEBUG:boto:Data: 2023-05-31T21:28:09.945 DEBUG:boto:Headers: {} 2023-05-31T21:28:09.945 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T21:28:09.945 DEBUG:boto:Port: 8000 2023-05-31T21:28:09.946 DEBUG:boto:Params: {} 2023-05-31T21:28:09.946 DEBUG:boto:Token: None 2023-05-31T21:28:09.946 DEBUG:boto:StringToSign: PUT Wed, 31 May 2023 21:28:09 GMT /pklxkd-73/ 2023-05-31T21:28:09.947 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:qC8hvvNRTXfWcvcI0mIBxQl74n0= 2023-05-31T21:28:09.947 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 21:28:09 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:qC8hvvNRTXfWcvcI0mIBxQl74n0=', 'Content-Length': '0'} 2023-05-31T21:28:09.981 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000ff187ef5c6106ce8-006477bbe9-264e-a1'), ('Content-Length', '0'), ('Date', 'Wed, 31 May 2023 21:28:09 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:28:09.982 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-31T21:28:09.982 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-31T21:28:09.983 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-05-31T21:28:10.034 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2023-05-31T21:28:10.035 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2023-05-31T21:28:10.072 INFO:teuthology.orchestra.run.smithi099.stderr:No --period given, using current period=2e6123d2-d695-4350-9a01-61a455bc782b 2023-05-31T21:28:10.085 INFO:teuthology.orchestra.run.smithi099.stdout:[ 2023-05-31T21:28:10.086 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T21:28:10.086 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1685566501.757183_260.1", 2023-05-31T21:28:10.086 INFO:teuthology.orchestra.run.smithi099.stdout: "last_update": "2023-05-31T20:55:01.757183Z" 2023-05-31T21:28:10.086 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:28:10.086 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T21:28:10.086 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1685566501.735717_251.1", 2023-05-31T21:28:10.087 INFO:teuthology.orchestra.run.smithi099.stdout: "last_update": "2023-05-31T20:55:01.735717Z" 2023-05-31T21:28:10.087 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:28:10.087 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T21:28:10.087 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1685562970.274572_93.1", 2023-05-31T21:28:10.087 INFO:teuthology.orchestra.run.smithi099.stdout: "last_update": "2023-05-31T19:56:10.274572Z" 2023-05-31T21:28:10.087 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:28:10.087 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T21:28:10.088 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1685568489.974220_1014.1", 2023-05-31T21:28:10.088 INFO:teuthology.orchestra.run.smithi099.stdout: "last_update": "2023-05-31T21:28:09.974220Z" 2023-05-31T21:28:10.088 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T21:28:10.088 INFO:teuthology.orchestra.run.smithi099.stdout:] 2023-05-31T21:28:10.089 DEBUG:rgw_multi.tests:master meta markers={0: '1_1685566501.757183_260.1', 1: '1_1685566501.735717_251.1', 2: '1_1685562970.274572_93.1', 3: '1_1685568489.974220_1014.1'} 2023-05-31T21:28:10.090 INFO:rgw_multi.tests:starting meta checkpoint for zone=a2 2023-05-31T21:28:10.090 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-31T21:28:10.090 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-31T21:28:10.091 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-05-31T21:28:10.141 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2023-05-31T21:28:10.142 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2023-05-31T21:28:10.197 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T21:28:10.193+0000 7fed542cb780 20 RGW-SYNC:meta: read sync status 2023-05-31T21:28:10.197 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T21:28:10.194+0000 7fed542cb780 20 RGW-SYNC:meta: read sync status 2023-05-31T21:28:10.232 INFO:teuthology.orchestra.run.smithi099.stdout:{ 2023-05-31T21:28:10.233 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_status": { 2023-05-31T21:28:10.233 INFO:teuthology.orchestra.run.smithi099.stdout: "info": { 2023-05-31T21:28:10.233 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "sync", 2023-05-31T21:28:10.233 INFO:teuthology.orchestra.run.smithi099.stdout: "num_shards": 4, 2023-05-31T21:28:10.233 INFO:teuthology.orchestra.run.smithi099.stdout: "period": "2e6123d2-d695-4350-9a01-61a455bc782b", 2023-05-31T21:28:10.233 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2023-05-31T21:28:10.234 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:28:10.234 INFO:teuthology.orchestra.run.smithi099.stdout: "markers": [ 2023-05-31T21:28:10.234 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T21:28:10.234 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 0, 2023-05-31T21:28:10.234 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T21:28:10.234 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2023-05-31T21:28:10.234 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1685566501.757183_260.1", 2023-05-31T21:28:10.235 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T21:28:10.235 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 1, 2023-05-31T21:28:10.235 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T21:28:10.235 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2023-05-31T20:55:01.757183Z", 2023-05-31T21:28:10.235 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2023-05-31T21:28:10.235 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T21:28:10.235 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:28:10.236 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T21:28:10.236 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 1, 2023-05-31T21:28:10.236 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T21:28:10.236 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2023-05-31T21:28:10.237 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1685566501.735717_251.1", 2023-05-31T21:28:10.237 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T21:28:10.237 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2023-05-31T21:28:10.237 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T21:28:10.237 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2023-05-31T20:55:01.735717Z", 2023-05-31T21:28:10.237 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2023-05-31T21:28:10.238 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T21:28:10.238 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:28:10.238 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T21:28:10.238 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 2, 2023-05-31T21:28:10.238 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T21:28:10.238 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2023-05-31T21:28:10.239 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1685562970.274572_93.1", 2023-05-31T21:28:10.239 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T21:28:10.239 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2023-05-31T21:28:10.239 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T21:28:10.239 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2023-05-31T19:56:10.274572Z", 2023-05-31T21:28:10.239 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2023-05-31T21:28:10.239 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T21:28:10.240 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:28:10.240 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T21:28:10.240 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 3, 2023-05-31T21:28:10.240 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T21:28:10.240 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2023-05-31T21:28:10.240 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1685568489.956971_1011.1", 2023-05-31T21:28:10.240 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T21:28:10.241 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2023-05-31T21:28:10.241 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T21:28:10.241 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2023-05-31T21:28:09.956971Z", 2023-05-31T21:28:10.241 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2023-05-31T21:28:10.242 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T21:28:10.242 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T21:28:10.242 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2023-05-31T21:28:10.242 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:28:10.242 INFO:teuthology.orchestra.run.smithi099.stdout: "full_sync": { 2023-05-31T21:28:10.243 INFO:teuthology.orchestra.run.smithi099.stdout: "total": 1, 2023-05-31T21:28:10.243 INFO:teuthology.orchestra.run.smithi099.stdout: "complete": 1 2023-05-31T21:28:10.243 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:28:10.243 INFO:teuthology.orchestra.run.smithi099.stdout: "current_time": "2023-05-31T21:28:10Z" 2023-05-31T21:28:10.243 INFO:teuthology.orchestra.run.smithi099.stdout:} 2023-05-31T21:28:10.244 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "2e6123d2-d695-4350-9a01-61a455bc782b", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1685566501.757183_260.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-05-31T20:55:01.757183Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1685566501.735717_251.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-05-31T20:55:01.735717Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1685562970.274572_93.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-05-31T19:56:10.274572Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1685568489.956971_1011.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-05-31T21:28:09.956971Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-05-31T21:28:10Z" } 2023-05-31T21:28:10.245 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1685566501.757183_260.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-05-31T20:55:01.757183Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1685566501.735717_251.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-05-31T20:55:01.735717Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1685562970.274572_93.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-05-31T19:56:10.274572Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1685568489.956971_1011.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-05-31T21:28:09.956971Z', 'realm_epoch': 2}}] 2023-05-31T21:28:10.245 DEBUG:rgw_multi.tests:log_status={0: '1_1685566501.757183_260.1', 1: '1_1685566501.735717_251.1', 2: '1_1685562970.274572_93.1', 3: '1_1685568489.974220_1014.1'} 2023-05-31T21:28:10.245 DEBUG:rgw_multi.tests:sync_status={0: '1_1685566501.757183_260.1', 1: '1_1685566501.735717_251.1', 2: '1_1685562970.274572_93.1', 3: '1_1685568489.956971_1011.1'} 2023-05-31T21:28:10.246 WARNING:rgw_multi.tests:zone a2 behind master: shard=3 master=1_1685568489.974220_1014.1 target=1_1685568489.956971_1011.1 2023-05-31T21:28:15.247 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-31T21:28:15.248 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-31T21:28:15.249 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-05-31T21:28:15.307 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2023-05-31T21:28:15.308 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2023-05-31T21:28:15.343 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T21:28:15.338+0000 7f1eaef38780 20 RGW-SYNC:meta: read sync status 2023-05-31T21:28:15.343 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T21:28:15.339+0000 7f1eaef38780 20 RGW-SYNC:meta: read sync status 2023-05-31T21:28:15.358 INFO:teuthology.orchestra.run.smithi099.stdout:{ 2023-05-31T21:28:15.359 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_status": { 2023-05-31T21:28:15.359 INFO:teuthology.orchestra.run.smithi099.stdout: "info": { 2023-05-31T21:28:15.359 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "sync", 2023-05-31T21:28:15.360 INFO:teuthology.orchestra.run.smithi099.stdout: "num_shards": 4, 2023-05-31T21:28:15.360 INFO:teuthology.orchestra.run.smithi099.stdout: "period": "2e6123d2-d695-4350-9a01-61a455bc782b", 2023-05-31T21:28:15.360 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2023-05-31T21:28:15.361 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:28:15.361 INFO:teuthology.orchestra.run.smithi099.stdout: "markers": [ 2023-05-31T21:28:15.361 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T21:28:15.361 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 0, 2023-05-31T21:28:15.361 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T21:28:15.361 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2023-05-31T21:28:15.362 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1685566501.757183_260.1", 2023-05-31T21:28:15.362 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T21:28:15.362 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 1, 2023-05-31T21:28:15.362 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T21:28:15.363 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2023-05-31T20:55:01.757183Z", 2023-05-31T21:28:15.363 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2023-05-31T21:28:15.363 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T21:28:15.364 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:28:15.364 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T21:28:15.364 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 1, 2023-05-31T21:28:15.364 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T21:28:15.364 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2023-05-31T21:28:15.364 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1685566501.735717_251.1", 2023-05-31T21:28:15.365 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T21:28:15.365 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2023-05-31T21:28:15.365 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T21:28:15.365 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2023-05-31T20:55:01.735717Z", 2023-05-31T21:28:15.365 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2023-05-31T21:28:15.365 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T21:28:15.365 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:28:15.366 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T21:28:15.366 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 2, 2023-05-31T21:28:15.367 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T21:28:15.367 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2023-05-31T21:28:15.367 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1685562970.274572_93.1", 2023-05-31T21:28:15.368 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T21:28:15.368 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2023-05-31T21:28:15.369 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T21:28:15.369 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2023-05-31T19:56:10.274572Z", 2023-05-31T21:28:15.369 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2023-05-31T21:28:15.370 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T21:28:15.370 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:28:15.370 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T21:28:15.371 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 3, 2023-05-31T21:28:15.371 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T21:28:15.371 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2023-05-31T21:28:15.371 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1685568489.974220_1014.1", 2023-05-31T21:28:15.372 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T21:28:15.372 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2023-05-31T21:28:15.372 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T21:28:15.372 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2023-05-31T21:28:09.974220Z", 2023-05-31T21:28:15.372 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2023-05-31T21:28:15.372 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T21:28:15.373 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T21:28:15.373 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2023-05-31T21:28:15.373 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:28:15.373 INFO:teuthology.orchestra.run.smithi099.stdout: "full_sync": { 2023-05-31T21:28:15.373 INFO:teuthology.orchestra.run.smithi099.stdout: "total": 1, 2023-05-31T21:28:15.373 INFO:teuthology.orchestra.run.smithi099.stdout: "complete": 1 2023-05-31T21:28:15.374 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:28:15.374 INFO:teuthology.orchestra.run.smithi099.stdout: "current_time": "2023-05-31T21:28:15Z" 2023-05-31T21:28:15.374 INFO:teuthology.orchestra.run.smithi099.stdout:} 2023-05-31T21:28:15.375 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "2e6123d2-d695-4350-9a01-61a455bc782b", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1685566501.757183_260.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-05-31T20:55:01.757183Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1685566501.735717_251.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-05-31T20:55:01.735717Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1685562970.274572_93.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-05-31T19:56:10.274572Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1685568489.974220_1014.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-05-31T21:28:09.974220Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-05-31T21:28:15Z" } 2023-05-31T21:28:15.376 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1685566501.757183_260.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-05-31T20:55:01.757183Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1685566501.735717_251.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-05-31T20:55:01.735717Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1685562970.274572_93.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-05-31T19:56:10.274572Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1685568489.974220_1014.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-05-31T21:28:09.974220Z', 'realm_epoch': 2}}] 2023-05-31T21:28:15.376 DEBUG:rgw_multi.tests:log_status={0: '1_1685566501.757183_260.1', 1: '1_1685566501.735717_251.1', 2: '1_1685562970.274572_93.1', 3: '1_1685568489.974220_1014.1'} 2023-05-31T21:28:15.377 DEBUG:rgw_multi.tests:sync_status={0: '1_1685566501.757183_260.1', 1: '1_1685566501.735717_251.1', 2: '1_1685562970.274572_93.1', 3: '1_1685568489.974220_1014.1'} 2023-05-31T21:28:15.377 INFO:rgw_multi.tests:finish meta checkpoint for zone=a2 2023-05-31T21:28:15.377 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'group', 'create', '--group-id', 'sync-group', '--status', 'allowed', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-05-31T21:28:15.378 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'group', 'create', '--group-id', 'sync-group', '--status', 'allowed', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-05-31T21:28:15.379 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync group create --group-id sync-group --status allowed --cluster c1 --debug-rgw 0 --debug-ms 0 2023-05-31T21:28:15.430 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2023-05-31T21:28:15.431 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2023-05-31T21:28:15.490 INFO:teuthology.orchestra.run.smithi099.stdout:{"groups":[{"id":"sync-group","data_flow":{},"pipes":[],"status":"allowed"}]} 2023-05-31T21:28:15.490 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'group', 'flow', 'create', '--group-id', 'sync-group', '--flow-id', 'sync-flow', '--flow-type', 'directional', '--source-zone=a1', '--dest-zone=a2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-05-31T21:28:15.491 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'group', 'flow', 'create', '--group-id', 'sync-group', '--flow-id', 'sync-flow', '--flow-type', 'directional', '--source-zone=a1', '--dest-zone=a2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-05-31T21:28:15.491 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync group flow create --group-id sync-group --flow-id sync-flow --flow-type directional --source-zone=a1 --dest-zone=a2 --cluster c1 --debug-rgw 0 --debug-ms 0 2023-05-31T21:28:15.545 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2023-05-31T21:28:15.546 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2023-05-31T21:28:15.618 INFO:teuthology.orchestra.run.smithi099.stdout:{"groups":[{"id":"sync-group","data_flow":{"directional":[{"source_zone":"a1","dest_zone":"a2"}]},"pipes":[],"status":"allowed"}]} 2023-05-31T21:28:15.619 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'group', 'pipe', 'create', '--group-id', 'sync-group', '--pipe-id', 'sync-pipe-pklxkd-72', '--source-zones=a1', '--dest-zones=a2', '--source-bucket=pklxkd-72', '--dest-bucket=pklxkd-73', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-05-31T21:28:15.619 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'group', 'pipe', 'create', '--group-id', 'sync-group', '--pipe-id', 'sync-pipe-pklxkd-72', '--source-zones=a1', '--dest-zones=a2', '--source-bucket=pklxkd-72', '--dest-bucket=pklxkd-73', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-05-31T21:28:15.619 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync group pipe create --group-id sync-group --pipe-id sync-pipe-pklxkd-72 --source-zones=a1 --dest-zones=a2 --source-bucket=pklxkd-72 --dest-bucket=pklxkd-73 --cluster c1 --debug-rgw 0 --debug-ms 0 2023-05-31T21:28:15.674 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2023-05-31T21:28:15.675 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2023-05-31T21:28:15.748 INFO:teuthology.orchestra.run.smithi099.stdout:{"groups":[{"id":"sync-group","data_flow":{"directional":[{"source_zone":"a1","dest_zone":"a2"}]},"pipes":[{"id":"sync-pipe-pklxkd-72","source":{"bucket":"pklxkd-72","zones":["a1"]},"dest":{"bucket":"pklxkd-73","zones":["a2"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":""}}],"status":"allowed"}]} 2023-05-31T21:28:15.749 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'group', 'pipe', 'create', '--group-id', 'sync-group', '--pipe-id', 'sync-pipe-pklxkd-73', '--source-zones=a1', '--dest-zones=a2', '--source-bucket=pklxkd-73', '--dest-bucket=pklxkd-73', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-05-31T21:28:15.749 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'group', 'pipe', 'create', '--group-id', 'sync-group', '--pipe-id', 'sync-pipe-pklxkd-73', '--source-zones=a1', '--dest-zones=a2', '--source-bucket=pklxkd-73', '--dest-bucket=pklxkd-73', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-05-31T21:28:15.750 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync group pipe create --group-id sync-group --pipe-id sync-pipe-pklxkd-73 --source-zones=a1 --dest-zones=a2 --source-bucket=pklxkd-73 --dest-bucket=pklxkd-73 --cluster c1 --debug-rgw 0 --debug-ms 0 2023-05-31T21:28:15.804 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2023-05-31T21:28:15.805 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2023-05-31T21:28:15.859 INFO:teuthology.orchestra.run.smithi099.stdout:{"groups":[{"id":"sync-group","data_flow":{"directional":[{"source_zone":"a1","dest_zone":"a2"}]},"pipes":[{"id":"sync-pipe-pklxkd-72","source":{"bucket":"pklxkd-72","zones":["a1"]},"dest":{"bucket":"pklxkd-73","zones":["a2"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":""}},{"id":"sync-pipe-pklxkd-73","source":{"bucket":"pklxkd-73","zones":["a1"]},"dest":{"bucket":"pklxkd-73","zones":["a2"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":""}}],"status":"allowed"}]} 2023-05-31T21:28:15.860 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'group', 'modify', '--group-id', 'sync-group', '--status', 'enabled', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-05-31T21:28:15.860 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'group', 'modify', '--group-id', 'sync-group', '--status', 'enabled', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-05-31T21:28:15.861 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync group modify --group-id sync-group --status enabled --cluster c1 --debug-rgw 0 --debug-ms 0 2023-05-31T21:28:15.911 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2023-05-31T21:28:15.913 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2023-05-31T21:28:15.974 INFO:teuthology.orchestra.run.smithi099.stdout:{"groups":[{"id":"sync-group","data_flow":{"directional":[{"source_zone":"a1","dest_zone":"a2"}]},"pipes":[{"id":"sync-pipe-pklxkd-72","source":{"bucket":"pklxkd-72","zones":["a1"]},"dest":{"bucket":"pklxkd-73","zones":["a2"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":""}},{"id":"sync-pipe-pklxkd-73","source":{"bucket":"pklxkd-73","zones":["a1"]},"dest":{"bucket":"pklxkd-73","zones":["a2"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":""}}],"status":"enabled"}]} 2023-05-31T21:28:15.975 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['period', 'update', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-05-31T21:28:15.975 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'period', 'update', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-05-31T21:28:15.976 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 period update --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --commit --cluster c1 --debug-rgw 0 --debug-ms 0 2023-05-31T21:28:16.027 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2023-05-31T21:28:16.027 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2023-05-31T21:28:16.110 INFO:teuthology.orchestra.run.smithi099.stdout:{ 2023-05-31T21:28:16.110 INFO:teuthology.orchestra.run.smithi099.stdout: "id": "2e6123d2-d695-4350-9a01-61a455bc782b", 2023-05-31T21:28:16.110 INFO:teuthology.orchestra.run.smithi099.stdout: "epoch": 19, 2023-05-31T21:28:16.111 INFO:teuthology.orchestra.run.smithi099.stdout: "predecessor_uuid": "629b9ced-80af-41ed-8a6c-f2cfc334cbb4", 2023-05-31T21:28:16.111 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_status": [], 2023-05-31T21:28:16.111 INFO:teuthology.orchestra.run.smithi099.stdout: "period_map": { 2023-05-31T21:28:16.111 INFO:teuthology.orchestra.run.smithi099.stdout: "id": "2e6123d2-d695-4350-9a01-61a455bc782b", 2023-05-31T21:28:16.111 INFO:teuthology.orchestra.run.smithi099.stdout: "zonegroups": [ 2023-05-31T21:28:16.111 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T21:28:16.112 INFO:teuthology.orchestra.run.smithi099.stdout: "id": "a", 2023-05-31T21:28:16.112 INFO:teuthology.orchestra.run.smithi099.stdout: "name": "a", 2023-05-31T21:28:16.112 INFO:teuthology.orchestra.run.smithi099.stdout: "api_name": "", 2023-05-31T21:28:16.112 INFO:teuthology.orchestra.run.smithi099.stdout: "is_master": true, 2023-05-31T21:28:16.112 INFO:teuthology.orchestra.run.smithi099.stdout: "endpoints": [ 2023-05-31T21:28:16.113 INFO:teuthology.orchestra.run.smithi099.stdout: "http://smithi099.front.sepia.ceph.com:8000" 2023-05-31T21:28:16.113 INFO:teuthology.orchestra.run.smithi099.stdout: ], 2023-05-31T21:28:16.113 INFO:teuthology.orchestra.run.smithi099.stdout: "hostnames": [], 2023-05-31T21:28:16.113 INFO:teuthology.orchestra.run.smithi099.stdout: "hostnames_s3website": [], 2023-05-31T21:28:16.113 INFO:teuthology.orchestra.run.smithi099.stdout: "master_zone": "3d163b1e-09c7-47e2-ae87-add4aace3f38", 2023-05-31T21:28:16.113 INFO:teuthology.orchestra.run.smithi099.stdout: "zones": [ 2023-05-31T21:28:16.114 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T21:28:16.114 INFO:teuthology.orchestra.run.smithi099.stdout: "id": "3d163b1e-09c7-47e2-ae87-add4aace3f38", 2023-05-31T21:28:16.114 INFO:teuthology.orchestra.run.smithi099.stdout: "name": "a1", 2023-05-31T21:28:16.114 INFO:teuthology.orchestra.run.smithi099.stdout: "endpoints": [ 2023-05-31T21:28:16.114 INFO:teuthology.orchestra.run.smithi099.stdout: "http://smithi099.front.sepia.ceph.com:8000" 2023-05-31T21:28:16.114 INFO:teuthology.orchestra.run.smithi099.stdout: ], 2023-05-31T21:28:16.115 INFO:teuthology.orchestra.run.smithi099.stdout: "log_meta": false, 2023-05-31T21:28:16.115 INFO:teuthology.orchestra.run.smithi099.stdout: "log_data": true, 2023-05-31T21:28:16.115 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket_index_max_shards": 11, 2023-05-31T21:28:16.115 INFO:teuthology.orchestra.run.smithi099.stdout: "read_only": false, 2023-05-31T21:28:16.115 INFO:teuthology.orchestra.run.smithi099.stdout: "tier_type": "", 2023-05-31T21:28:16.116 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_from_all": true, 2023-05-31T21:28:16.116 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_from": [], 2023-05-31T21:28:16.116 INFO:teuthology.orchestra.run.smithi099.stdout: "redirect_zone": "", 2023-05-31T21:28:16.116 INFO:teuthology.orchestra.run.smithi099.stdout: "supported_features": [ 2023-05-31T21:28:16.116 INFO:teuthology.orchestra.run.smithi099.stdout: "resharding" 2023-05-31T21:28:16.117 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2023-05-31T21:28:16.117 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:28:16.117 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T21:28:16.117 INFO:teuthology.orchestra.run.smithi099.stdout: "id": "5d8acec7-7e57-4e56-8b32-e3d6b5036001", 2023-05-31T21:28:16.118 INFO:teuthology.orchestra.run.smithi099.stdout: "name": "a2", 2023-05-31T21:28:16.118 INFO:teuthology.orchestra.run.smithi099.stdout: "endpoints": [ 2023-05-31T21:28:16.118 INFO:teuthology.orchestra.run.smithi099.stdout: "http://smithi099.front.sepia.ceph.com:8001" 2023-05-31T21:28:16.118 INFO:teuthology.orchestra.run.smithi099.stdout: ], 2023-05-31T21:28:16.118 INFO:teuthology.orchestra.run.smithi099.stdout: "log_meta": false, 2023-05-31T21:28:16.119 INFO:teuthology.orchestra.run.smithi099.stdout: "log_data": true, 2023-05-31T21:28:16.119 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket_index_max_shards": 11, 2023-05-31T21:28:16.119 INFO:teuthology.orchestra.run.smithi099.stdout: "read_only": false, 2023-05-31T21:28:16.119 INFO:teuthology.orchestra.run.smithi099.stdout: "tier_type": "", 2023-05-31T21:28:16.119 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_from_all": true, 2023-05-31T21:28:16.119 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_from": [], 2023-05-31T21:28:16.119 INFO:teuthology.orchestra.run.smithi099.stdout: "redirect_zone": "", 2023-05-31T21:28:16.120 INFO:teuthology.orchestra.run.smithi099.stdout: "supported_features": [ 2023-05-31T21:28:16.120 INFO:teuthology.orchestra.run.smithi099.stdout: "resharding" 2023-05-31T21:28:16.120 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2023-05-31T21:28:16.121 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T21:28:16.121 INFO:teuthology.orchestra.run.smithi099.stdout: ], 2023-05-31T21:28:16.121 INFO:teuthology.orchestra.run.smithi099.stdout: "placement_targets": [ 2023-05-31T21:28:16.121 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T21:28:16.122 INFO:teuthology.orchestra.run.smithi099.stdout: "name": "default-placement", 2023-05-31T21:28:16.122 INFO:teuthology.orchestra.run.smithi099.stdout: "tags": [], 2023-05-31T21:28:16.122 INFO:teuthology.orchestra.run.smithi099.stdout: "storage_classes": [ 2023-05-31T21:28:16.122 INFO:teuthology.orchestra.run.smithi099.stdout: "STANDARD" 2023-05-31T21:28:16.122 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2023-05-31T21:28:16.122 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T21:28:16.123 INFO:teuthology.orchestra.run.smithi099.stdout: ], 2023-05-31T21:28:16.123 INFO:teuthology.orchestra.run.smithi099.stdout: "default_placement": "default-placement", 2023-05-31T21:28:16.123 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_id": "aaa33de6-515b-4cac-8f1b-3a1a30452117", 2023-05-31T21:28:16.123 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_policy": { 2023-05-31T21:28:16.123 INFO:teuthology.orchestra.run.smithi099.stdout: "groups": [ 2023-05-31T21:28:16.124 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T21:28:16.124 INFO:teuthology.orchestra.run.smithi099.stdout: "id": "sync-group", 2023-05-31T21:28:16.124 INFO:teuthology.orchestra.run.smithi099.stdout: "data_flow": { 2023-05-31T21:28:16.124 INFO:teuthology.orchestra.run.smithi099.stdout: "directional": [ 2023-05-31T21:28:16.124 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T21:28:16.124 INFO:teuthology.orchestra.run.smithi099.stdout: "source_zone": "3d163b1e-09c7-47e2-ae87-add4aace3f38", 2023-05-31T21:28:16.125 INFO:teuthology.orchestra.run.smithi099.stdout: "dest_zone": "5d8acec7-7e57-4e56-8b32-e3d6b5036001" 2023-05-31T21:28:16.125 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T21:28:16.125 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2023-05-31T21:28:16.125 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:28:16.125 INFO:teuthology.orchestra.run.smithi099.stdout: "pipes": [ 2023-05-31T21:28:16.125 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T21:28:16.126 INFO:teuthology.orchestra.run.smithi099.stdout: "id": "sync-pipe-pklxkd-72", 2023-05-31T21:28:16.126 INFO:teuthology.orchestra.run.smithi099.stdout: "source": { 2023-05-31T21:28:16.126 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket": "pklxkd-72", 2023-05-31T21:28:16.126 INFO:teuthology.orchestra.run.smithi099.stdout: "zones": [ 2023-05-31T21:28:16.126 INFO:teuthology.orchestra.run.smithi099.stdout: "3d163b1e-09c7-47e2-ae87-add4aace3f38" 2023-05-31T21:28:16.127 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2023-05-31T21:28:16.127 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:28:16.127 INFO:teuthology.orchestra.run.smithi099.stdout: "dest": { 2023-05-31T21:28:16.127 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket": "pklxkd-73", 2023-05-31T21:28:16.127 INFO:teuthology.orchestra.run.smithi099.stdout: "zones": [ 2023-05-31T21:28:16.128 INFO:teuthology.orchestra.run.smithi099.stdout: "5d8acec7-7e57-4e56-8b32-e3d6b5036001" 2023-05-31T21:28:16.128 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2023-05-31T21:28:16.129 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:28:16.130 INFO:teuthology.orchestra.run.smithi099.stdout: "params": { 2023-05-31T21:28:16.130 INFO:teuthology.orchestra.run.smithi099.stdout: "source": { 2023-05-31T21:28:16.130 INFO:teuthology.orchestra.run.smithi099.stdout: "filter": { 2023-05-31T21:28:16.131 INFO:teuthology.orchestra.run.smithi099.stdout: "tags": [] 2023-05-31T21:28:16.131 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T21:28:16.131 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:28:16.131 INFO:teuthology.orchestra.run.smithi099.stdout: "dest": {}, 2023-05-31T21:28:16.131 INFO:teuthology.orchestra.run.smithi099.stdout: "priority": 0, 2023-05-31T21:28:16.131 INFO:teuthology.orchestra.run.smithi099.stdout: "mode": "system", 2023-05-31T21:28:16.132 INFO:teuthology.orchestra.run.smithi099.stdout: "user": "" 2023-05-31T21:28:16.132 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T21:28:16.132 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:28:16.132 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T21:28:16.132 INFO:teuthology.orchestra.run.smithi099.stdout: "id": "sync-pipe-pklxkd-73", 2023-05-31T21:28:16.132 INFO:teuthology.orchestra.run.smithi099.stdout: "source": { 2023-05-31T21:28:16.133 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket": "pklxkd-73", 2023-05-31T21:28:16.133 INFO:teuthology.orchestra.run.smithi099.stdout: "zones": [ 2023-05-31T21:28:16.133 INFO:teuthology.orchestra.run.smithi099.stdout: "3d163b1e-09c7-47e2-ae87-add4aace3f38" 2023-05-31T21:28:16.133 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2023-05-31T21:28:16.134 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:28:16.134 INFO:teuthology.orchestra.run.smithi099.stdout: "dest": { 2023-05-31T21:28:16.134 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket": "pklxkd-73", 2023-05-31T21:28:16.134 INFO:teuthology.orchestra.run.smithi099.stdout: "zones": [ 2023-05-31T21:28:16.134 INFO:teuthology.orchestra.run.smithi099.stdout: "5d8acec7-7e57-4e56-8b32-e3d6b5036001" 2023-05-31T21:28:16.135 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2023-05-31T21:28:16.135 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:28:16.135 INFO:teuthology.orchestra.run.smithi099.stdout: "params": { 2023-05-31T21:28:16.135 INFO:teuthology.orchestra.run.smithi099.stdout: "source": { 2023-05-31T21:28:16.135 INFO:teuthology.orchestra.run.smithi099.stdout: "filter": { 2023-05-31T21:28:16.136 INFO:teuthology.orchestra.run.smithi099.stdout: "tags": [] 2023-05-31T21:28:16.136 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T21:28:16.136 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:28:16.136 INFO:teuthology.orchestra.run.smithi099.stdout: "dest": {}, 2023-05-31T21:28:16.136 INFO:teuthology.orchestra.run.smithi099.stdout: "priority": 0, 2023-05-31T21:28:16.136 INFO:teuthology.orchestra.run.smithi099.stdout: "mode": "system", 2023-05-31T21:28:16.137 INFO:teuthology.orchestra.run.smithi099.stdout: "user": "" 2023-05-31T21:28:16.137 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T21:28:16.137 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T21:28:16.137 INFO:teuthology.orchestra.run.smithi099.stdout: ], 2023-05-31T21:28:16.137 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "enabled" 2023-05-31T21:28:16.137 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T21:28:16.138 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2023-05-31T21:28:16.138 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:28:16.138 INFO:teuthology.orchestra.run.smithi099.stdout: "enabled_features": [ 2023-05-31T21:28:16.138 INFO:teuthology.orchestra.run.smithi099.stdout: "resharding" 2023-05-31T21:28:16.138 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2023-05-31T21:28:16.139 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:28:16.139 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T21:28:16.139 INFO:teuthology.orchestra.run.smithi099.stdout: "id": "b", 2023-05-31T21:28:16.139 INFO:teuthology.orchestra.run.smithi099.stdout: "name": "b", 2023-05-31T21:28:16.139 INFO:teuthology.orchestra.run.smithi099.stdout: "api_name": "", 2023-05-31T21:28:16.139 INFO:teuthology.orchestra.run.smithi099.stdout: "is_master": false, 2023-05-31T21:28:16.140 INFO:teuthology.orchestra.run.smithi099.stdout: "endpoints": [ 2023-05-31T21:28:16.140 INFO:teuthology.orchestra.run.smithi099.stdout: "http://smithi112.front.sepia.ceph.com:8000" 2023-05-31T21:28:16.140 INFO:teuthology.orchestra.run.smithi099.stdout: ], 2023-05-31T21:28:16.140 INFO:teuthology.orchestra.run.smithi099.stdout: "hostnames": [], 2023-05-31T21:28:16.141 INFO:teuthology.orchestra.run.smithi099.stdout: "hostnames_s3website": [], 2023-05-31T21:28:16.141 INFO:teuthology.orchestra.run.smithi099.stdout: "master_zone": "f85d7236-2f27-4988-af26-9c66ab280b1b", 2023-05-31T21:28:16.142 INFO:teuthology.orchestra.run.smithi099.stdout: "zones": [ 2023-05-31T21:28:16.142 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T21:28:16.142 INFO:teuthology.orchestra.run.smithi099.stdout: "id": "a6a3aab5-bc52-44a4-8543-5c5ea6056acd", 2023-05-31T21:28:16.143 INFO:teuthology.orchestra.run.smithi099.stdout: "name": "b2", 2023-05-31T21:28:16.143 INFO:teuthology.orchestra.run.smithi099.stdout: "endpoints": [ 2023-05-31T21:28:16.144 INFO:teuthology.orchestra.run.smithi099.stdout: "http://smithi112.front.sepia.ceph.com:8001" 2023-05-31T21:28:16.144 INFO:teuthology.orchestra.run.smithi099.stdout: ], 2023-05-31T21:28:16.145 INFO:teuthology.orchestra.run.smithi099.stdout: "log_meta": false, 2023-05-31T21:28:16.145 INFO:teuthology.orchestra.run.smithi099.stdout: "log_data": true, 2023-05-31T21:28:16.145 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket_index_max_shards": 11, 2023-05-31T21:28:16.146 INFO:teuthology.orchestra.run.smithi099.stdout: "read_only": false, 2023-05-31T21:28:16.146 INFO:teuthology.orchestra.run.smithi099.stdout: "tier_type": "", 2023-05-31T21:28:16.146 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_from_all": true, 2023-05-31T21:28:16.146 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_from": [], 2023-05-31T21:28:16.146 INFO:teuthology.orchestra.run.smithi099.stdout: "redirect_zone": "", 2023-05-31T21:28:16.147 INFO:teuthology.orchestra.run.smithi099.stdout: "supported_features": [ 2023-05-31T21:28:16.147 INFO:teuthology.orchestra.run.smithi099.stdout: "resharding" 2023-05-31T21:28:16.147 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2023-05-31T21:28:16.147 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:28:16.147 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T21:28:16.147 INFO:teuthology.orchestra.run.smithi099.stdout: "id": "f85d7236-2f27-4988-af26-9c66ab280b1b", 2023-05-31T21:28:16.148 INFO:teuthology.orchestra.run.smithi099.stdout: "name": "b1", 2023-05-31T21:28:16.148 INFO:teuthology.orchestra.run.smithi099.stdout: "endpoints": [ 2023-05-31T21:28:16.148 INFO:teuthology.orchestra.run.smithi099.stdout: "http://smithi112.front.sepia.ceph.com:8000" 2023-05-31T21:28:16.148 INFO:teuthology.orchestra.run.smithi099.stdout: ], 2023-05-31T21:28:16.148 INFO:teuthology.orchestra.run.smithi099.stdout: "log_meta": false, 2023-05-31T21:28:16.148 INFO:teuthology.orchestra.run.smithi099.stdout: "log_data": true, 2023-05-31T21:28:16.149 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket_index_max_shards": 11, 2023-05-31T21:28:16.149 INFO:teuthology.orchestra.run.smithi099.stdout: "read_only": false, 2023-05-31T21:28:16.149 INFO:teuthology.orchestra.run.smithi099.stdout: "tier_type": "", 2023-05-31T21:28:16.149 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_from_all": true, 2023-05-31T21:28:16.149 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_from": [], 2023-05-31T21:28:16.149 INFO:teuthology.orchestra.run.smithi099.stdout: "redirect_zone": "", 2023-05-31T21:28:16.149 INFO:teuthology.orchestra.run.smithi099.stdout: "supported_features": [ 2023-05-31T21:28:16.150 INFO:teuthology.orchestra.run.smithi099.stdout: "resharding" 2023-05-31T21:28:16.150 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2023-05-31T21:28:16.150 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T21:28:16.150 INFO:teuthology.orchestra.run.smithi099.stdout: ], 2023-05-31T21:28:16.151 INFO:teuthology.orchestra.run.smithi099.stdout: "placement_targets": [ 2023-05-31T21:28:16.151 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T21:28:16.151 INFO:teuthology.orchestra.run.smithi099.stdout: "name": "default-placement", 2023-05-31T21:28:16.152 INFO:teuthology.orchestra.run.smithi099.stdout: "tags": [], 2023-05-31T21:28:16.152 INFO:teuthology.orchestra.run.smithi099.stdout: "storage_classes": [ 2023-05-31T21:28:16.152 INFO:teuthology.orchestra.run.smithi099.stdout: "STANDARD" 2023-05-31T21:28:16.152 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2023-05-31T21:28:16.152 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T21:28:16.152 INFO:teuthology.orchestra.run.smithi099.stdout: ], 2023-05-31T21:28:16.153 INFO:teuthology.orchestra.run.smithi099.stdout: "default_placement": "default-placement", 2023-05-31T21:28:16.153 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_id": "aaa33de6-515b-4cac-8f1b-3a1a30452117", 2023-05-31T21:28:16.153 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_policy": { 2023-05-31T21:28:16.153 INFO:teuthology.orchestra.run.smithi099.stdout: "groups": [] 2023-05-31T21:28:16.153 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:28:16.153 INFO:teuthology.orchestra.run.smithi099.stdout: "enabled_features": [ 2023-05-31T21:28:16.153 INFO:teuthology.orchestra.run.smithi099.stdout: "resharding" 2023-05-31T21:28:16.154 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2023-05-31T21:28:16.154 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T21:28:16.154 INFO:teuthology.orchestra.run.smithi099.stdout: ], 2023-05-31T21:28:16.154 INFO:teuthology.orchestra.run.smithi099.stdout: "short_zone_ids": [ 2023-05-31T21:28:16.154 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T21:28:16.154 INFO:teuthology.orchestra.run.smithi099.stdout: "key": "3d163b1e-09c7-47e2-ae87-add4aace3f38", 2023-05-31T21:28:16.154 INFO:teuthology.orchestra.run.smithi099.stdout: "val": 3101833771 2023-05-31T21:28:16.155 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:28:16.155 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T21:28:16.155 INFO:teuthology.orchestra.run.smithi099.stdout: "key": "5d8acec7-7e57-4e56-8b32-e3d6b5036001", 2023-05-31T21:28:16.155 INFO:teuthology.orchestra.run.smithi099.stdout: "val": 4255530623 2023-05-31T21:28:16.155 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:28:16.155 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T21:28:16.156 INFO:teuthology.orchestra.run.smithi099.stdout: "key": "a6a3aab5-bc52-44a4-8543-5c5ea6056acd", 2023-05-31T21:28:16.156 INFO:teuthology.orchestra.run.smithi099.stdout: "val": 3708073710 2023-05-31T21:28:16.156 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:28:16.156 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T21:28:16.156 INFO:teuthology.orchestra.run.smithi099.stdout: "key": "f85d7236-2f27-4988-af26-9c66ab280b1b", 2023-05-31T21:28:16.156 INFO:teuthology.orchestra.run.smithi099.stdout: "val": 233395200 2023-05-31T21:28:16.156 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T21:28:16.157 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2023-05-31T21:28:16.157 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:28:16.157 INFO:teuthology.orchestra.run.smithi099.stdout: "master_zonegroup": "a", 2023-05-31T21:28:16.157 INFO:teuthology.orchestra.run.smithi099.stdout: "master_zone": "3d163b1e-09c7-47e2-ae87-add4aace3f38", 2023-05-31T21:28:16.157 INFO:teuthology.orchestra.run.smithi099.stdout: "period_config": { 2023-05-31T21:28:16.157 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket_quota": { 2023-05-31T21:28:16.157 INFO:teuthology.orchestra.run.smithi099.stdout: "enabled": false, 2023-05-31T21:28:16.158 INFO:teuthology.orchestra.run.smithi099.stdout: "check_on_raw": false, 2023-05-31T21:28:16.158 INFO:teuthology.orchestra.run.smithi099.stdout: "max_size": -1, 2023-05-31T21:28:16.158 INFO:teuthology.orchestra.run.smithi099.stdout: "max_size_kb": 0, 2023-05-31T21:28:16.158 INFO:teuthology.orchestra.run.smithi099.stdout: "max_objects": -1 2023-05-31T21:28:16.158 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:28:16.158 INFO:teuthology.orchestra.run.smithi099.stdout: "user_quota": { 2023-05-31T21:28:16.158 INFO:teuthology.orchestra.run.smithi099.stdout: "enabled": false, 2023-05-31T21:28:16.159 INFO:teuthology.orchestra.run.smithi099.stdout: "check_on_raw": false, 2023-05-31T21:28:16.159 INFO:teuthology.orchestra.run.smithi099.stdout: "max_size": -1, 2023-05-31T21:28:16.159 INFO:teuthology.orchestra.run.smithi099.stdout: "max_size_kb": 0, 2023-05-31T21:28:16.159 INFO:teuthology.orchestra.run.smithi099.stdout: "max_objects": -1 2023-05-31T21:28:16.159 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:28:16.159 INFO:teuthology.orchestra.run.smithi099.stdout: "user_ratelimit": { 2023-05-31T21:28:16.159 INFO:teuthology.orchestra.run.smithi099.stdout: "max_read_ops": 0, 2023-05-31T21:28:16.160 INFO:teuthology.orchestra.run.smithi099.stdout: "max_write_ops": 0, 2023-05-31T21:28:16.160 INFO:teuthology.orchestra.run.smithi099.stdout: "max_read_bytes": 0, 2023-05-31T21:28:16.160 INFO:teuthology.orchestra.run.smithi099.stdout: "max_write_bytes": 0, 2023-05-31T21:28:16.160 INFO:teuthology.orchestra.run.smithi099.stdout: "enabled": false 2023-05-31T21:28:16.160 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:28:16.160 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket_ratelimit": { 2023-05-31T21:28:16.160 INFO:teuthology.orchestra.run.smithi099.stdout: "max_read_ops": 0, 2023-05-31T21:28:16.161 INFO:teuthology.orchestra.run.smithi099.stdout: "max_write_ops": 0, 2023-05-31T21:28:16.161 INFO:teuthology.orchestra.run.smithi099.stdout: "max_read_bytes": 0, 2023-05-31T21:28:16.161 INFO:teuthology.orchestra.run.smithi099.stdout: "max_write_bytes": 0, 2023-05-31T21:28:16.161 INFO:teuthology.orchestra.run.smithi099.stdout: "enabled": false 2023-05-31T21:28:16.161 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:28:16.161 INFO:teuthology.orchestra.run.smithi099.stdout: "anonymous_ratelimit": { 2023-05-31T21:28:16.162 INFO:teuthology.orchestra.run.smithi099.stdout: "max_read_ops": 0, 2023-05-31T21:28:16.162 INFO:teuthology.orchestra.run.smithi099.stdout: "max_write_ops": 0, 2023-05-31T21:28:16.162 INFO:teuthology.orchestra.run.smithi099.stdout: "max_read_bytes": 0, 2023-05-31T21:28:16.162 INFO:teuthology.orchestra.run.smithi099.stdout: "max_write_bytes": 0, 2023-05-31T21:28:16.162 INFO:teuthology.orchestra.run.smithi099.stdout: "enabled": false 2023-05-31T21:28:16.162 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T21:28:16.162 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:28:16.163 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_id": "aaa33de6-515b-4cac-8f1b-3a1a30452117", 2023-05-31T21:28:16.163 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_name": "test-realm", 2023-05-31T21:28:16.163 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2023-05-31T21:28:16.163 INFO:teuthology.orchestra.run.smithi099.stdout:} 2023-05-31T21:28:16.163 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'policy', 'get', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-05-31T21:28:16.164 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'policy', 'get', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-05-31T21:28:16.164 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync policy get --cluster c1 --debug-rgw 0 --debug-ms 0 2023-05-31T21:28:16.216 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2023-05-31T21:28:16.217 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2023-05-31T21:28:16.335 INFO:teuthology.orchestra.run.smithi099.stdout:{"groups":[{"id":"sync-group","data_flow":{"directional":[{"source_zone":"a1","dest_zone":"a2"}]},"pipes":[{"id":"sync-pipe-pklxkd-72","source":{"bucket":"pklxkd-72","zones":["a1"]},"dest":{"bucket":"pklxkd-73","zones":["a2"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":""}},{"id":"sync-pipe-pklxkd-73","source":{"bucket":"pklxkd-73","zones":["a1"]},"dest":{"bucket":"pklxkd-73","zones":["a2"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":""}}],"status":"enabled"}]} 2023-05-31T21:28:16.336 DEBUG:boto:path=/pklxkd-72/ 2023-05-31T21:28:16.337 DEBUG:boto:auth_path=/pklxkd-72/ 2023-05-31T21:28:16.337 DEBUG:boto:Method: HEAD 2023-05-31T21:28:16.338 DEBUG:boto:Path: /pklxkd-72/ 2023-05-31T21:28:16.339 DEBUG:boto:Data: 2023-05-31T21:28:16.339 DEBUG:boto:Headers: {} 2023-05-31T21:28:16.340 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T21:28:16.341 DEBUG:boto:Port: 8000 2023-05-31T21:28:16.342 DEBUG:boto:Params: {} 2023-05-31T21:28:16.343 DEBUG:boto:Token: None 2023-05-31T21:28:16.343 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 21:28:16 GMT /pklxkd-72/ 2023-05-31T21:28:16.344 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:9PtYFtThg5hPPy0zS71/BqB2Dq0= 2023-05-31T21:28:16.345 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 21:28:16 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:9PtYFtThg5hPPy0zS71/BqB2Dq0=', 'Content-Length': '0'} 2023-05-31T21:28:16.354 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000010c7bc910f3b07c6-006477bbf0-2889-a1'), ('Content-Length', '0'), ('Date', 'Wed, 31 May 2023 21:28:16 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:28:16.355 DEBUG:boto:path=/pklxkd-72/obj1 2023-05-31T21:28:16.356 DEBUG:boto:auth_path=/pklxkd-72/obj1 2023-05-31T21:28:16.358 DEBUG:boto:Method: PUT 2023-05-31T21:28:16.358 DEBUG:boto:Path: /pklxkd-72/obj1 2023-05-31T21:28:16.359 DEBUG:boto:Data: 2023-05-31T21:28:16.360 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2023-05-31T21:28:16.360 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T21:28:16.361 DEBUG:boto:Port: 8000 2023-05-31T21:28:16.361 DEBUG:boto:Params: {} 2023-05-31T21:28:16.362 DEBUG:boto:Token: None 2023-05-31T21:28:16.363 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Wed, 31 May 2023 21:28:16 GMT /pklxkd-72/obj1 2023-05-31T21:28:16.363 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:FO1bXcUKeUfkhg7Qx9jopHBdlhA= 2023-05-31T21:28:16.364 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Wed, 31 May 2023 21:28:16 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:FO1bXcUKeUfkhg7Qx9jopHBdlhA='} 2023-05-31T21:28:16.390 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000008423f3ae98190f06-006477bbf0-2889-a1'), ('Date', 'Wed, 31 May 2023 21:28:16 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:28:16.391 DEBUG:boto:path=/pklxkd-72/ 2023-05-31T21:28:16.391 DEBUG:boto:auth_path=/pklxkd-72/ 2023-05-31T21:28:16.391 DEBUG:boto:Method: HEAD 2023-05-31T21:28:16.392 DEBUG:boto:Path: /pklxkd-72/ 2023-05-31T21:28:16.392 DEBUG:boto:Data: 2023-05-31T21:28:16.392 DEBUG:boto:Headers: {} 2023-05-31T21:28:16.393 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T21:28:16.393 DEBUG:boto:Port: 8000 2023-05-31T21:28:16.393 DEBUG:boto:Params: {} 2023-05-31T21:28:16.394 DEBUG:boto:Token: None 2023-05-31T21:28:16.394 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 21:28:16 GMT /pklxkd-72/ 2023-05-31T21:28:16.395 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:9PtYFtThg5hPPy0zS71/BqB2Dq0= 2023-05-31T21:28:16.395 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 21:28:16 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:9PtYFtThg5hPPy0zS71/BqB2Dq0=', 'Content-Length': '0'} 2023-05-31T21:28:16.397 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '6'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000012439cbc60c0185f-006477bbf0-2889-a1'), ('Content-Length', '0'), ('Date', 'Wed, 31 May 2023 21:28:16 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:28:16.399 DEBUG:boto:path=/pklxkd-72/obj2 2023-05-31T21:28:16.400 DEBUG:boto:auth_path=/pklxkd-72/obj2 2023-05-31T21:28:16.400 DEBUG:boto:Method: PUT 2023-05-31T21:28:16.401 DEBUG:boto:Path: /pklxkd-72/obj2 2023-05-31T21:28:16.401 DEBUG:boto:Data: 2023-05-31T21:28:16.402 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2023-05-31T21:28:16.402 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T21:28:16.403 DEBUG:boto:Port: 8000 2023-05-31T21:28:16.403 DEBUG:boto:Params: {} 2023-05-31T21:28:16.403 DEBUG:boto:Token: None 2023-05-31T21:28:16.404 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Wed, 31 May 2023 21:28:16 GMT /pklxkd-72/obj2 2023-05-31T21:28:16.404 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:PeN9keHJFLO7RITqn4MZMHXEpAo= 2023-05-31T21:28:16.405 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Wed, 31 May 2023 21:28:16 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:PeN9keHJFLO7RITqn4MZMHXEpAo='} 2023-05-31T21:28:16.448 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000005b9889f488b162b9-006477bbf0-2889-a1'), ('Date', 'Wed, 31 May 2023 21:28:16 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:28:16.449 DEBUG:boto:path=/pklxkd-73/ 2023-05-31T21:28:16.450 DEBUG:boto:auth_path=/pklxkd-73/ 2023-05-31T21:28:16.450 DEBUG:boto:Method: HEAD 2023-05-31T21:28:16.450 DEBUG:boto:Path: /pklxkd-73/ 2023-05-31T21:28:16.451 DEBUG:boto:Data: 2023-05-31T21:28:16.451 DEBUG:boto:Headers: {} 2023-05-31T21:28:16.451 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T21:28:16.452 DEBUG:boto:Port: 8000 2023-05-31T21:28:16.452 DEBUG:boto:Params: {} 2023-05-31T21:28:16.452 DEBUG:boto:Token: None 2023-05-31T21:28:16.453 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 21:28:16 GMT /pklxkd-73/ 2023-05-31T21:28:16.453 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:j+nV5H6YldvybXmh5I9m+woTyoE= 2023-05-31T21:28:16.454 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 21:28:16 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:j+nV5H6YldvybXmh5I9m+woTyoE=', 'Content-Length': '0'} 2023-05-31T21:28:16.464 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000fd0936e3dbc45f97-006477bbf0-2889-a1'), ('Content-Length', '0'), ('Date', 'Wed, 31 May 2023 21:28:16 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:28:16.465 DEBUG:boto:path=/pklxkd-73/obj3 2023-05-31T21:28:16.465 DEBUG:boto:auth_path=/pklxkd-73/obj3 2023-05-31T21:28:16.466 DEBUG:boto:Method: PUT 2023-05-31T21:28:16.466 DEBUG:boto:Path: /pklxkd-73/obj3 2023-05-31T21:28:16.466 DEBUG:boto:Data: 2023-05-31T21:28:16.467 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2023-05-31T21:28:16.467 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T21:28:16.467 DEBUG:boto:Port: 8000 2023-05-31T21:28:16.468 DEBUG:boto:Params: {} 2023-05-31T21:28:16.468 DEBUG:boto:Token: None 2023-05-31T21:28:16.469 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Wed, 31 May 2023 21:28:16 GMT /pklxkd-73/obj3 2023-05-31T21:28:16.469 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:2IMH2+VthMq4o9Rpi0wpKGsElT8= 2023-05-31T21:28:16.470 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Wed, 31 May 2023 21:28:16 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:2IMH2+VthMq4o9Rpi0wpKGsElT8='} 2023-05-31T21:28:16.481 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000cd410b439d942f09-006477bbf0-2889-a1'), ('Date', 'Wed, 31 May 2023 21:28:16 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:28:16.482 DEBUG:boto:path=/pklxkd-73/ 2023-05-31T21:28:16.482 DEBUG:boto:auth_path=/pklxkd-73/ 2023-05-31T21:28:16.482 DEBUG:boto:Method: HEAD 2023-05-31T21:28:16.483 DEBUG:boto:Path: /pklxkd-73/ 2023-05-31T21:28:16.483 DEBUG:boto:Data: 2023-05-31T21:28:16.483 DEBUG:boto:Headers: {} 2023-05-31T21:28:16.484 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T21:28:16.484 DEBUG:boto:Port: 8000 2023-05-31T21:28:16.484 DEBUG:boto:Params: {} 2023-05-31T21:28:16.485 DEBUG:boto:Token: None 2023-05-31T21:28:16.485 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 21:28:16 GMT /pklxkd-73/ 2023-05-31T21:28:16.486 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:j+nV5H6YldvybXmh5I9m+woTyoE= 2023-05-31T21:28:16.486 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 21:28:16 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:j+nV5H6YldvybXmh5I9m+woTyoE=', 'Content-Length': '0'} 2023-05-31T21:28:16.488 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '6'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000046de10161e6d37a5-006477bbf0-2889-a1'), ('Content-Length', '0'), ('Date', 'Wed, 31 May 2023 21:28:16 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:28:16.489 DEBUG:boto:path=/pklxkd-73/obj4 2023-05-31T21:28:16.489 DEBUG:boto:auth_path=/pklxkd-73/obj4 2023-05-31T21:28:16.490 DEBUG:boto:Method: PUT 2023-05-31T21:28:16.490 DEBUG:boto:Path: /pklxkd-73/obj4 2023-05-31T21:28:16.491 DEBUG:boto:Data: 2023-05-31T21:28:16.491 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2023-05-31T21:28:16.491 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T21:28:16.492 DEBUG:boto:Port: 8000 2023-05-31T21:28:16.492 DEBUG:boto:Params: {} 2023-05-31T21:28:16.492 DEBUG:boto:Token: None 2023-05-31T21:28:16.493 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Wed, 31 May 2023 21:28:16 GMT /pklxkd-73/obj4 2023-05-31T21:28:16.493 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:jmIrMmstM8992AAbTygYZtWzN9c= 2023-05-31T21:28:16.494 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Wed, 31 May 2023 21:28:16 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:jmIrMmstM8992AAbTygYZtWzN9c='} 2023-05-31T21:28:16.537 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000c933cad929c90c98-006477bbf0-2889-a1'), ('Date', 'Wed, 31 May 2023 21:28:16 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:28:16.538 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-31T21:28:16.538 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-31T21:28:16.539 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-05-31T21:28:16.599 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2023-05-31T21:28:16.600 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2023-05-31T21:28:16.695 INFO:teuthology.orchestra.run.smithi099.stderr:No --period given, using current period=2e6123d2-d695-4350-9a01-61a455bc782b 2023-05-31T21:28:16.749 INFO:teuthology.orchestra.run.smithi099.stdout:[ 2023-05-31T21:28:16.749 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T21:28:16.750 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1685566501.757183_260.1", 2023-05-31T21:28:16.750 INFO:teuthology.orchestra.run.smithi099.stdout: "last_update": "2023-05-31T20:55:01.757183Z" 2023-05-31T21:28:16.750 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:28:16.750 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T21:28:16.750 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1685566501.735717_251.1", 2023-05-31T21:28:16.751 INFO:teuthology.orchestra.run.smithi099.stdout: "last_update": "2023-05-31T20:55:01.735717Z" 2023-05-31T21:28:16.751 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:28:16.751 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T21:28:16.751 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1685562970.274572_93.1", 2023-05-31T21:28:16.752 INFO:teuthology.orchestra.run.smithi099.stdout: "last_update": "2023-05-31T19:56:10.274572Z" 2023-05-31T21:28:16.752 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:28:16.752 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T21:28:16.752 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1685568489.974220_1014.1", 2023-05-31T21:28:16.752 INFO:teuthology.orchestra.run.smithi099.stdout: "last_update": "2023-05-31T21:28:09.974220Z" 2023-05-31T21:28:16.752 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T21:28:16.753 INFO:teuthology.orchestra.run.smithi099.stdout:] 2023-05-31T21:28:16.754 DEBUG:rgw_multi.tests:master meta markers={0: '1_1685566501.757183_260.1', 1: '1_1685566501.735717_251.1', 2: '1_1685562970.274572_93.1', 3: '1_1685568489.974220_1014.1'} 2023-05-31T21:28:16.754 INFO:rgw_multi.tests:starting meta checkpoint for zone=a2 2023-05-31T21:28:16.754 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-31T21:28:16.755 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-31T21:28:16.755 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-05-31T21:28:16.821 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2023-05-31T21:28:16.822 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2023-05-31T21:28:16.907 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T21:28:16.904+0000 7fc4af060780 20 RGW-SYNC:meta: read sync status 2023-05-31T21:28:16.908 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T21:28:16.905+0000 7fc4af060780 20 RGW-SYNC:meta: read sync status 2023-05-31T21:28:16.937 INFO:teuthology.orchestra.run.smithi099.stdout:{ 2023-05-31T21:28:16.938 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_status": { 2023-05-31T21:28:16.938 INFO:teuthology.orchestra.run.smithi099.stdout: "info": { 2023-05-31T21:28:16.938 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "sync", 2023-05-31T21:28:16.938 INFO:teuthology.orchestra.run.smithi099.stdout: "num_shards": 4, 2023-05-31T21:28:16.939 INFO:teuthology.orchestra.run.smithi099.stdout: "period": "2e6123d2-d695-4350-9a01-61a455bc782b", 2023-05-31T21:28:16.939 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2023-05-31T21:28:16.939 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:28:16.939 INFO:teuthology.orchestra.run.smithi099.stdout: "markers": [ 2023-05-31T21:28:16.939 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T21:28:16.940 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 0, 2023-05-31T21:28:16.940 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T21:28:16.940 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2023-05-31T21:28:16.940 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1685566501.757183_260.1", 2023-05-31T21:28:16.940 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T21:28:16.941 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 1, 2023-05-31T21:28:16.941 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T21:28:16.941 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2023-05-31T20:55:01.757183Z", 2023-05-31T21:28:16.941 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2023-05-31T21:28:16.941 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T21:28:16.941 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:28:16.941 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T21:28:16.942 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 1, 2023-05-31T21:28:16.942 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T21:28:16.942 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2023-05-31T21:28:16.942 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1685566501.735717_251.1", 2023-05-31T21:28:16.942 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T21:28:16.943 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2023-05-31T21:28:16.943 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T21:28:16.943 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2023-05-31T20:55:01.735717Z", 2023-05-31T21:28:16.943 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2023-05-31T21:28:16.943 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T21:28:16.943 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:28:16.943 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T21:28:16.944 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 2, 2023-05-31T21:28:16.944 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T21:28:16.944 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2023-05-31T21:28:16.944 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1685562970.274572_93.1", 2023-05-31T21:28:16.944 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T21:28:16.944 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2023-05-31T21:28:16.945 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T21:28:16.945 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2023-05-31T19:56:10.274572Z", 2023-05-31T21:28:16.945 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2023-05-31T21:28:16.945 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T21:28:16.945 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:28:16.945 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T21:28:16.945 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 3, 2023-05-31T21:28:16.946 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T21:28:16.946 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2023-05-31T21:28:16.946 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1685568489.974220_1014.1", 2023-05-31T21:28:16.946 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T21:28:16.946 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2023-05-31T21:28:16.946 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T21:28:16.946 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2023-05-31T21:28:09.974220Z", 2023-05-31T21:28:16.947 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2023-05-31T21:28:16.947 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T21:28:16.947 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T21:28:16.947 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2023-05-31T21:28:16.947 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:28:16.947 INFO:teuthology.orchestra.run.smithi099.stdout: "full_sync": { 2023-05-31T21:28:16.948 INFO:teuthology.orchestra.run.smithi099.stdout: "total": 1, 2023-05-31T21:28:16.948 INFO:teuthology.orchestra.run.smithi099.stdout: "complete": 1 2023-05-31T21:28:16.948 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:28:16.948 INFO:teuthology.orchestra.run.smithi099.stdout: "current_time": "2023-05-31T21:28:16Z" 2023-05-31T21:28:16.948 INFO:teuthology.orchestra.run.smithi099.stdout:} 2023-05-31T21:28:16.949 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "2e6123d2-d695-4350-9a01-61a455bc782b", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1685566501.757183_260.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-05-31T20:55:01.757183Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1685566501.735717_251.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-05-31T20:55:01.735717Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1685562970.274572_93.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-05-31T19:56:10.274572Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1685568489.974220_1014.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-05-31T21:28:09.974220Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-05-31T21:28:16Z" } 2023-05-31T21:28:16.950 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1685566501.757183_260.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-05-31T20:55:01.757183Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1685566501.735717_251.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-05-31T20:55:01.735717Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1685562970.274572_93.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-05-31T19:56:10.274572Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1685568489.974220_1014.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-05-31T21:28:09.974220Z', 'realm_epoch': 2}}] 2023-05-31T21:28:16.950 DEBUG:rgw_multi.tests:log_status={0: '1_1685566501.757183_260.1', 1: '1_1685566501.735717_251.1', 2: '1_1685562970.274572_93.1', 3: '1_1685568489.974220_1014.1'} 2023-05-31T21:28:16.950 DEBUG:rgw_multi.tests:sync_status={0: '1_1685566501.757183_260.1', 1: '1_1685566501.735717_251.1', 2: '1_1685562970.274572_93.1', 3: '1_1685568489.974220_1014.1'} 2023-05-31T21:28:16.951 INFO:rgw_multi.tests:finish meta checkpoint for zone=a2 2023-05-31T21:28:16.951 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['datalog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-31T21:28:16.952 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'datalog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-31T21:28:16.952 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 datalog status --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-05-31T21:28:17.009 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2023-05-31T21:28:17.009 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2023-05-31T21:28:17.126 INFO:teuthology.orchestra.run.smithi099.stdout:[ 2023-05-31T21:28:17.126 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T21:28:17.127 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "00000000000000000000:00000000000000045375", 2023-05-31T21:28:17.127 INFO:teuthology.orchestra.run.smithi099.stdout: "last_update": "2023-05-31T21:28:16.422581Z" 2023-05-31T21:28:17.127 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:28:17.127 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T21:28:17.127 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "00000000000000000000:00000000000000031780", 2023-05-31T21:28:17.127 INFO:teuthology.orchestra.run.smithi099.stdout: "last_update": "2023-05-31T21:28:16.501293Z" 2023-05-31T21:28:17.128 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:28:17.128 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T21:28:17.128 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "00000000000000000000:00000000000000051671", 2023-05-31T21:28:17.128 INFO:teuthology.orchestra.run.smithi099.stdout: "last_update": "2023-05-31T21:28:16.476487Z" 2023-05-31T21:28:17.128 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:28:17.128 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T21:28:17.128 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "00000000000000000000:00000000000000044178", 2023-05-31T21:28:17.129 INFO:teuthology.orchestra.run.smithi099.stdout: "last_update": "2023-05-31T21:27:37.215905Z" 2023-05-31T21:28:17.129 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T21:28:17.129 INFO:teuthology.orchestra.run.smithi099.stdout:] 2023-05-31T21:28:17.130 DEBUG:rgw_multi.tests:data markers for zone=a1 markers={0: '00000000000000000000:00000000000000045375', 1: '00000000000000000000:00000000000000031780', 2: '00000000000000000000:00000000000000051671', 3: '00000000000000000000:00000000000000044178'} 2023-05-31T21:28:17.130 INFO:rgw_multi.tests:starting data checkpoint for target_zone=a2 source_zone=a1 2023-05-31T21:28:17.131 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-31T21:28:17.131 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-31T21:28:17.131 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-05-31T21:28:17.192 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2023-05-31T21:28:17.193 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2023-05-31T21:28:17.313 INFO:teuthology.orchestra.run.smithi099.stdout:{ 2023-05-31T21:28:17.314 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_status": { 2023-05-31T21:28:17.314 INFO:teuthology.orchestra.run.smithi099.stdout: "info": { 2023-05-31T21:28:17.315 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "sync", 2023-05-31T21:28:17.315 INFO:teuthology.orchestra.run.smithi099.stdout: "num_shards": 4, 2023-05-31T21:28:17.316 INFO:teuthology.orchestra.run.smithi099.stdout: "instance_id": 452136274422638824 2023-05-31T21:28:17.316 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:28:17.316 INFO:teuthology.orchestra.run.smithi099.stdout: "markers": [ 2023-05-31T21:28:17.316 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T21:28:17.316 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 0, 2023-05-31T21:28:17.317 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T21:28:17.317 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "incremental-sync", 2023-05-31T21:28:17.317 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "00000000000000000000:00000000000000045235", 2023-05-31T21:28:17.317 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T21:28:17.317 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2023-05-31T21:28:17.318 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T21:28:17.318 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2023-05-31T21:25:53.171643Z" 2023-05-31T21:28:17.318 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T21:28:17.318 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:28:17.318 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T21:28:17.318 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 1, 2023-05-31T21:28:17.318 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T21:28:17.319 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "incremental-sync", 2023-05-31T21:28:17.319 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "00000000000000000000:00000000000000031640", 2023-05-31T21:28:17.319 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T21:28:17.319 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2023-05-31T21:28:17.319 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T21:28:17.319 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2023-05-31T21:27:37.214586Z" 2023-05-31T21:28:17.319 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T21:28:17.320 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:28:17.320 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T21:28:17.320 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 2, 2023-05-31T21:28:17.320 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T21:28:17.321 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "incremental-sync", 2023-05-31T21:28:17.321 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "00000000000000000000:00000000000000051391", 2023-05-31T21:28:17.321 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T21:28:17.321 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 55, 2023-05-31T21:28:17.322 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T21:28:17.322 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2023-05-31T21:25:53.141700Z" 2023-05-31T21:28:17.322 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T21:28:17.323 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:28:17.323 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T21:28:17.323 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 3, 2023-05-31T21:28:17.324 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T21:28:17.324 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "incremental-sync", 2023-05-31T21:28:17.324 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "00000000000000000000:00000000000000044178", 2023-05-31T21:28:17.325 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T21:28:17.325 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 9, 2023-05-31T21:28:17.325 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T21:28:17.325 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2023-05-31T21:27:37.215905Z" 2023-05-31T21:28:17.326 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T21:28:17.326 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T21:28:17.326 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2023-05-31T21:28:17.327 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:28:17.327 INFO:teuthology.orchestra.run.smithi099.stdout: "full_sync": { 2023-05-31T21:28:17.327 INFO:teuthology.orchestra.run.smithi099.stdout: "total": 64, 2023-05-31T21:28:17.328 INFO:teuthology.orchestra.run.smithi099.stdout: "complete": 64 2023-05-31T21:28:17.328 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:28:17.328 INFO:teuthology.orchestra.run.smithi099.stdout: "current_time": "2023-05-31T21:28:17Z" 2023-05-31T21:28:17.329 INFO:teuthology.orchestra.run.smithi099.stdout:} 2023-05-31T21:28:17.330 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 452136274422638824 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000045235", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-05-31T21:25:53.171643Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000031640", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-05-31T21:27:37.214586Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000051391", "next_step_marker": "", "total_entries": 55, "pos": 0, "timestamp": "2023-05-31T21:25:53.141700Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000044178", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2023-05-31T21:27:37.215905Z" } } ] }, "full_sync": { "total": 64, "complete": 64 }, "current_time": "2023-05-31T21:28:17Z" } 2023-05-31T21:28:17.331 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000045235', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-05-31T21:25:53.171643Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000031640', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-05-31T21:27:37.214586Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000051391', 'next_step_marker': '', 'total_entries': 55, 'pos': 0, 'timestamp': '2023-05-31T21:25:53.141700Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000044178', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2023-05-31T21:27:37.215905Z'}}] 2023-05-31T21:28:17.331 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000045375', 1: '00000000000000000000:00000000000000031780', 2: '00000000000000000000:00000000000000051671', 3: '00000000000000000000:00000000000000044178'} 2023-05-31T21:28:17.332 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000045235', 1: '00000000000000000000:00000000000000031640', 2: '00000000000000000000:00000000000000051391', 3: '00000000000000000000:00000000000000044178'} 2023-05-31T21:28:17.333 WARNING:rgw_multi.tests:data of zone a2 behind zone a1: shard=0 master=00000000000000000000:00000000000000045375 target=00000000000000000000:00000000000000045235, shard=1 master=00000000000000000000:00000000000000031780 target=00000000000000000000:00000000000000031640, shard=2 master=00000000000000000000:00000000000000051671 target=00000000000000000000:00000000000000051391 2023-05-31T21:28:22.334 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-31T21:28:22.336 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-31T21:28:22.336 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-05-31T21:28:22.396 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2023-05-31T21:28:22.396 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2023-05-31T21:28:22.480 INFO:teuthology.orchestra.run.smithi099.stdout:{ 2023-05-31T21:28:22.480 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_status": { 2023-05-31T21:28:22.481 INFO:teuthology.orchestra.run.smithi099.stdout: "info": { 2023-05-31T21:28:22.481 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "sync", 2023-05-31T21:28:22.481 INFO:teuthology.orchestra.run.smithi099.stdout: "num_shards": 4, 2023-05-31T21:28:22.481 INFO:teuthology.orchestra.run.smithi099.stdout: "instance_id": 452136274422638824 2023-05-31T21:28:22.481 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:28:22.481 INFO:teuthology.orchestra.run.smithi099.stdout: "markers": [ 2023-05-31T21:28:22.482 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T21:28:22.482 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 0, 2023-05-31T21:28:22.482 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T21:28:22.482 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "incremental-sync", 2023-05-31T21:28:22.482 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "00000000000000000000:00000000000000045235", 2023-05-31T21:28:22.482 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T21:28:22.482 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2023-05-31T21:28:22.483 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T21:28:22.483 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2023-05-31T21:25:53.171643Z" 2023-05-31T21:28:22.483 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T21:28:22.483 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:28:22.483 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T21:28:22.483 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 1, 2023-05-31T21:28:22.484 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T21:28:22.484 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "incremental-sync", 2023-05-31T21:28:22.484 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "00000000000000000000:00000000000000031640", 2023-05-31T21:28:22.484 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T21:28:22.484 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2023-05-31T21:28:22.484 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T21:28:22.485 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2023-05-31T21:27:37.214586Z" 2023-05-31T21:28:22.485 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T21:28:22.485 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:28:22.485 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T21:28:22.486 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 2, 2023-05-31T21:28:22.486 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T21:28:22.486 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "incremental-sync", 2023-05-31T21:28:22.486 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "00000000000000000000:00000000000000051391", 2023-05-31T21:28:22.486 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T21:28:22.486 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 55, 2023-05-31T21:28:22.487 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T21:28:22.487 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2023-05-31T21:25:53.141700Z" 2023-05-31T21:28:22.487 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T21:28:22.487 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:28:22.488 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T21:28:22.488 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 3, 2023-05-31T21:28:22.488 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T21:28:22.488 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "incremental-sync", 2023-05-31T21:28:22.488 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "00000000000000000000:00000000000000044178", 2023-05-31T21:28:22.488 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T21:28:22.489 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 9, 2023-05-31T21:28:22.489 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T21:28:22.489 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2023-05-31T21:27:37.215905Z" 2023-05-31T21:28:22.490 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T21:28:22.490 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T21:28:22.490 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2023-05-31T21:28:22.490 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:28:22.491 INFO:teuthology.orchestra.run.smithi099.stdout: "full_sync": { 2023-05-31T21:28:22.491 INFO:teuthology.orchestra.run.smithi099.stdout: "total": 64, 2023-05-31T21:28:22.491 INFO:teuthology.orchestra.run.smithi099.stdout: "complete": 64 2023-05-31T21:28:22.491 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:28:22.492 INFO:teuthology.orchestra.run.smithi099.stdout: "current_time": "2023-05-31T21:28:22Z" 2023-05-31T21:28:22.492 INFO:teuthology.orchestra.run.smithi099.stdout:} 2023-05-31T21:28:22.493 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 452136274422638824 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000045235", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-05-31T21:25:53.171643Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000031640", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-05-31T21:27:37.214586Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000051391", "next_step_marker": "", "total_entries": 55, "pos": 0, "timestamp": "2023-05-31T21:25:53.141700Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000044178", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2023-05-31T21:27:37.215905Z" } } ] }, "full_sync": { "total": 64, "complete": 64 }, "current_time": "2023-05-31T21:28:22Z" } 2023-05-31T21:28:22.493 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000045235', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-05-31T21:25:53.171643Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000031640', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-05-31T21:27:37.214586Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000051391', 'next_step_marker': '', 'total_entries': 55, 'pos': 0, 'timestamp': '2023-05-31T21:25:53.141700Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000044178', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2023-05-31T21:27:37.215905Z'}}] 2023-05-31T21:28:22.494 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000045375', 1: '00000000000000000000:00000000000000031780', 2: '00000000000000000000:00000000000000051671', 3: '00000000000000000000:00000000000000044178'} 2023-05-31T21:28:22.494 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000045235', 1: '00000000000000000000:00000000000000031640', 2: '00000000000000000000:00000000000000051391', 3: '00000000000000000000:00000000000000044178'} 2023-05-31T21:28:22.494 WARNING:rgw_multi.tests:data of zone a2 behind zone a1: shard=0 master=00000000000000000000:00000000000000045375 target=00000000000000000000:00000000000000045235, shard=1 master=00000000000000000000:00000000000000031780 target=00000000000000000000:00000000000000031640, shard=2 master=00000000000000000000:00000000000000051671 target=00000000000000000000:00000000000000051391 2023-05-31T21:28:27.495 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-31T21:28:27.496 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-31T21:28:27.497 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-05-31T21:28:27.557 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2023-05-31T21:28:27.558 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2023-05-31T21:28:27.621 INFO:teuthology.orchestra.run.smithi099.stdout:{ 2023-05-31T21:28:27.621 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_status": { 2023-05-31T21:28:27.621 INFO:teuthology.orchestra.run.smithi099.stdout: "info": { 2023-05-31T21:28:27.622 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "sync", 2023-05-31T21:28:27.622 INFO:teuthology.orchestra.run.smithi099.stdout: "num_shards": 4, 2023-05-31T21:28:27.622 INFO:teuthology.orchestra.run.smithi099.stdout: "instance_id": 452136274422638824 2023-05-31T21:28:27.622 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:28:27.622 INFO:teuthology.orchestra.run.smithi099.stdout: "markers": [ 2023-05-31T21:28:27.622 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T21:28:27.622 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 0, 2023-05-31T21:28:27.623 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T21:28:27.623 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "incremental-sync", 2023-05-31T21:28:27.623 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "00000000000000000000:00000000000000045235", 2023-05-31T21:28:27.623 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T21:28:27.623 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2023-05-31T21:28:27.623 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T21:28:27.624 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2023-05-31T21:25:53.171643Z" 2023-05-31T21:28:27.624 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T21:28:27.624 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:28:27.624 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T21:28:27.625 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 1, 2023-05-31T21:28:27.625 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T21:28:27.625 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "incremental-sync", 2023-05-31T21:28:27.626 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "00000000000000000000:00000000000000031640", 2023-05-31T21:28:27.627 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T21:28:27.627 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2023-05-31T21:28:27.627 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T21:28:27.627 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2023-05-31T21:27:37.214586Z" 2023-05-31T21:28:27.627 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T21:28:27.627 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:28:27.627 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T21:28:27.628 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 2, 2023-05-31T21:28:27.628 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T21:28:27.628 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "incremental-sync", 2023-05-31T21:28:27.628 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "00000000000000000000:00000000000000051391", 2023-05-31T21:28:27.628 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T21:28:27.628 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 55, 2023-05-31T21:28:27.629 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T21:28:27.629 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2023-05-31T21:25:53.141700Z" 2023-05-31T21:28:27.629 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T21:28:27.629 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:28:27.629 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T21:28:27.629 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 3, 2023-05-31T21:28:27.629 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T21:28:27.630 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "incremental-sync", 2023-05-31T21:28:27.630 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "00000000000000000000:00000000000000044178", 2023-05-31T21:28:27.630 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T21:28:27.630 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 9, 2023-05-31T21:28:27.630 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T21:28:27.630 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2023-05-31T21:27:37.215905Z" 2023-05-31T21:28:27.631 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T21:28:27.631 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T21:28:27.631 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2023-05-31T21:28:27.631 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:28:27.631 INFO:teuthology.orchestra.run.smithi099.stdout: "full_sync": { 2023-05-31T21:28:27.631 INFO:teuthology.orchestra.run.smithi099.stdout: "total": 64, 2023-05-31T21:28:27.631 INFO:teuthology.orchestra.run.smithi099.stdout: "complete": 64 2023-05-31T21:28:27.632 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:28:27.632 INFO:teuthology.orchestra.run.smithi099.stdout: "current_time": "2023-05-31T21:28:27Z" 2023-05-31T21:28:27.632 INFO:teuthology.orchestra.run.smithi099.stdout:} 2023-05-31T21:28:27.633 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 452136274422638824 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000045235", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-05-31T21:25:53.171643Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000031640", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-05-31T21:27:37.214586Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000051391", "next_step_marker": "", "total_entries": 55, "pos": 0, "timestamp": "2023-05-31T21:25:53.141700Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000044178", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2023-05-31T21:27:37.215905Z" } } ] }, "full_sync": { "total": 64, "complete": 64 }, "current_time": "2023-05-31T21:28:27Z" } 2023-05-31T21:28:27.633 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000045235', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-05-31T21:25:53.171643Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000031640', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-05-31T21:27:37.214586Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000051391', 'next_step_marker': '', 'total_entries': 55, 'pos': 0, 'timestamp': '2023-05-31T21:25:53.141700Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000044178', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2023-05-31T21:27:37.215905Z'}}] 2023-05-31T21:28:27.634 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000045375', 1: '00000000000000000000:00000000000000031780', 2: '00000000000000000000:00000000000000051671', 3: '00000000000000000000:00000000000000044178'} 2023-05-31T21:28:27.634 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000045235', 1: '00000000000000000000:00000000000000031640', 2: '00000000000000000000:00000000000000051391', 3: '00000000000000000000:00000000000000044178'} 2023-05-31T21:28:27.634 WARNING:rgw_multi.tests:data of zone a2 behind zone a1: shard=0 master=00000000000000000000:00000000000000045375 target=00000000000000000000:00000000000000045235, shard=1 master=00000000000000000000:00000000000000031780 target=00000000000000000000:00000000000000031640, shard=2 master=00000000000000000000:00000000000000051671 target=00000000000000000000:00000000000000051391 2023-05-31T21:28:32.635 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-31T21:28:32.636 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-31T21:28:32.637 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-05-31T21:28:32.696 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2023-05-31T21:28:32.697 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2023-05-31T21:28:32.770 INFO:teuthology.orchestra.run.smithi099.stdout:{ 2023-05-31T21:28:32.771 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_status": { 2023-05-31T21:28:32.771 INFO:teuthology.orchestra.run.smithi099.stdout: "info": { 2023-05-31T21:28:32.771 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "sync", 2023-05-31T21:28:32.771 INFO:teuthology.orchestra.run.smithi099.stdout: "num_shards": 4, 2023-05-31T21:28:32.771 INFO:teuthology.orchestra.run.smithi099.stdout: "instance_id": 452136274422638824 2023-05-31T21:28:32.772 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:28:32.772 INFO:teuthology.orchestra.run.smithi099.stdout: "markers": [ 2023-05-31T21:28:32.772 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T21:28:32.772 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 0, 2023-05-31T21:28:32.772 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T21:28:32.772 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "incremental-sync", 2023-05-31T21:28:32.773 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "00000000000000000000:00000000000000045235", 2023-05-31T21:28:32.773 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T21:28:32.773 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2023-05-31T21:28:32.773 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T21:28:32.773 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2023-05-31T21:25:53.171643Z" 2023-05-31T21:28:32.773 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T21:28:32.773 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:28:32.774 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T21:28:32.774 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 1, 2023-05-31T21:28:32.774 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T21:28:32.774 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "incremental-sync", 2023-05-31T21:28:32.774 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "00000000000000000000:00000000000000031640", 2023-05-31T21:28:32.774 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T21:28:32.774 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2023-05-31T21:28:32.775 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T21:28:32.775 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2023-05-31T21:27:37.214586Z" 2023-05-31T21:28:32.775 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T21:28:32.775 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:28:32.775 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T21:28:32.775 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 2, 2023-05-31T21:28:32.776 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T21:28:32.776 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "incremental-sync", 2023-05-31T21:28:32.776 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "00000000000000000000:00000000000000051391", 2023-05-31T21:28:32.776 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T21:28:32.776 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 55, 2023-05-31T21:28:32.776 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T21:28:32.776 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2023-05-31T21:25:53.141700Z" 2023-05-31T21:28:32.777 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T21:28:32.777 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:28:32.777 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T21:28:32.777 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 3, 2023-05-31T21:28:32.777 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T21:28:32.777 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "incremental-sync", 2023-05-31T21:28:32.777 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "00000000000000000000:00000000000000044178", 2023-05-31T21:28:32.778 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T21:28:32.778 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 9, 2023-05-31T21:28:32.778 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T21:28:32.778 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2023-05-31T21:27:37.215905Z" 2023-05-31T21:28:32.778 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T21:28:32.778 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T21:28:32.779 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2023-05-31T21:28:32.779 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:28:32.779 INFO:teuthology.orchestra.run.smithi099.stdout: "full_sync": { 2023-05-31T21:28:32.779 INFO:teuthology.orchestra.run.smithi099.stdout: "total": 64, 2023-05-31T21:28:32.779 INFO:teuthology.orchestra.run.smithi099.stdout: "complete": 64 2023-05-31T21:28:32.779 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:28:32.779 INFO:teuthology.orchestra.run.smithi099.stdout: "current_time": "2023-05-31T21:28:32Z" 2023-05-31T21:28:32.780 INFO:teuthology.orchestra.run.smithi099.stdout:} 2023-05-31T21:28:32.780 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 452136274422638824 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000045235", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-05-31T21:25:53.171643Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000031640", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-05-31T21:27:37.214586Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000051391", "next_step_marker": "", "total_entries": 55, "pos": 0, "timestamp": "2023-05-31T21:25:53.141700Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000044178", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2023-05-31T21:27:37.215905Z" } } ] }, "full_sync": { "total": 64, "complete": 64 }, "current_time": "2023-05-31T21:28:32Z" } 2023-05-31T21:28:32.781 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000045235', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-05-31T21:25:53.171643Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000031640', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-05-31T21:27:37.214586Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000051391', 'next_step_marker': '', 'total_entries': 55, 'pos': 0, 'timestamp': '2023-05-31T21:25:53.141700Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000044178', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2023-05-31T21:27:37.215905Z'}}] 2023-05-31T21:28:32.781 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000045375', 1: '00000000000000000000:00000000000000031780', 2: '00000000000000000000:00000000000000051671', 3: '00000000000000000000:00000000000000044178'} 2023-05-31T21:28:32.782 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000045235', 1: '00000000000000000000:00000000000000031640', 2: '00000000000000000000:00000000000000051391', 3: '00000000000000000000:00000000000000044178'} 2023-05-31T21:28:32.782 WARNING:rgw_multi.tests:data of zone a2 behind zone a1: shard=0 master=00000000000000000000:00000000000000045375 target=00000000000000000000:00000000000000045235, shard=1 master=00000000000000000000:00000000000000031780 target=00000000000000000000:00000000000000031640, shard=2 master=00000000000000000000:00000000000000051671 target=00000000000000000000:00000000000000051391 2023-05-31T21:28:37.783 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-31T21:28:37.784 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-31T21:28:37.784 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-05-31T21:28:37.844 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2023-05-31T21:28:37.844 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2023-05-31T21:28:37.900 INFO:teuthology.orchestra.run.smithi099.stdout:{ 2023-05-31T21:28:37.900 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_status": { 2023-05-31T21:28:37.900 INFO:teuthology.orchestra.run.smithi099.stdout: "info": { 2023-05-31T21:28:37.901 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "sync", 2023-05-31T21:28:37.901 INFO:teuthology.orchestra.run.smithi099.stdout: "num_shards": 4, 2023-05-31T21:28:37.901 INFO:teuthology.orchestra.run.smithi099.stdout: "instance_id": 452136274422638824 2023-05-31T21:28:37.901 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:28:37.901 INFO:teuthology.orchestra.run.smithi099.stdout: "markers": [ 2023-05-31T21:28:37.901 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T21:28:37.902 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 0, 2023-05-31T21:28:37.902 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T21:28:37.902 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "incremental-sync", 2023-05-31T21:28:37.902 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "00000000000000000000:00000000000000045235", 2023-05-31T21:28:37.902 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T21:28:37.902 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2023-05-31T21:28:37.902 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T21:28:37.903 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2023-05-31T21:25:53.171643Z" 2023-05-31T21:28:37.903 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T21:28:37.903 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:28:37.903 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T21:28:37.903 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 1, 2023-05-31T21:28:37.903 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T21:28:37.903 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "incremental-sync", 2023-05-31T21:28:37.904 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "00000000000000000000:00000000000000031640", 2023-05-31T21:28:37.904 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T21:28:37.904 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2023-05-31T21:28:37.904 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T21:28:37.904 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2023-05-31T21:27:37.214586Z" 2023-05-31T21:28:37.904 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T21:28:37.904 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:28:37.905 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T21:28:37.905 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 2, 2023-05-31T21:28:37.905 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T21:28:37.905 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "incremental-sync", 2023-05-31T21:28:37.905 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "00000000000000000000:00000000000000051391", 2023-05-31T21:28:37.905 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T21:28:37.906 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 55, 2023-05-31T21:28:37.906 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T21:28:37.906 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2023-05-31T21:25:53.141700Z" 2023-05-31T21:28:37.906 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T21:28:37.906 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:28:37.906 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T21:28:37.906 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 3, 2023-05-31T21:28:37.907 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T21:28:37.907 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "incremental-sync", 2023-05-31T21:28:37.907 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "00000000000000000000:00000000000000044178", 2023-05-31T21:28:37.907 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T21:28:37.907 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 9, 2023-05-31T21:28:37.907 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T21:28:37.907 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2023-05-31T21:27:37.215905Z" 2023-05-31T21:28:37.908 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T21:28:37.908 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T21:28:37.908 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2023-05-31T21:28:37.908 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:28:37.908 INFO:teuthology.orchestra.run.smithi099.stdout: "full_sync": { 2023-05-31T21:28:37.908 INFO:teuthology.orchestra.run.smithi099.stdout: "total": 64, 2023-05-31T21:28:37.908 INFO:teuthology.orchestra.run.smithi099.stdout: "complete": 64 2023-05-31T21:28:37.909 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:28:37.909 INFO:teuthology.orchestra.run.smithi099.stdout: "current_time": "2023-05-31T21:28:37Z" 2023-05-31T21:28:37.909 INFO:teuthology.orchestra.run.smithi099.stdout:} 2023-05-31T21:28:37.910 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 452136274422638824 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000045235", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-05-31T21:25:53.171643Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000031640", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-05-31T21:27:37.214586Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000051391", "next_step_marker": "", "total_entries": 55, "pos": 0, "timestamp": "2023-05-31T21:25:53.141700Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000044178", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2023-05-31T21:27:37.215905Z" } } ] }, "full_sync": { "total": 64, "complete": 64 }, "current_time": "2023-05-31T21:28:37Z" } 2023-05-31T21:28:37.910 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000045235', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-05-31T21:25:53.171643Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000031640', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-05-31T21:27:37.214586Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000051391', 'next_step_marker': '', 'total_entries': 55, 'pos': 0, 'timestamp': '2023-05-31T21:25:53.141700Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000044178', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2023-05-31T21:27:37.215905Z'}}] 2023-05-31T21:28:37.911 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000045375', 1: '00000000000000000000:00000000000000031780', 2: '00000000000000000000:00000000000000051671', 3: '00000000000000000000:00000000000000044178'} 2023-05-31T21:28:37.911 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000045235', 1: '00000000000000000000:00000000000000031640', 2: '00000000000000000000:00000000000000051391', 3: '00000000000000000000:00000000000000044178'} 2023-05-31T21:28:37.911 WARNING:rgw_multi.tests:data of zone a2 behind zone a1: shard=0 master=00000000000000000000:00000000000000045375 target=00000000000000000000:00000000000000045235, shard=1 master=00000000000000000000:00000000000000031780 target=00000000000000000000:00000000000000031640, shard=2 master=00000000000000000000:00000000000000051671 target=00000000000000000000:00000000000000051391 2023-05-31T21:28:42.912 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-31T21:28:42.913 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-31T21:28:42.914 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-05-31T21:28:42.974 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2023-05-31T21:28:42.974 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2023-05-31T21:28:43.048 INFO:teuthology.orchestra.run.smithi099.stdout:{ 2023-05-31T21:28:43.049 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_status": { 2023-05-31T21:28:43.049 INFO:teuthology.orchestra.run.smithi099.stdout: "info": { 2023-05-31T21:28:43.049 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "sync", 2023-05-31T21:28:43.049 INFO:teuthology.orchestra.run.smithi099.stdout: "num_shards": 4, 2023-05-31T21:28:43.049 INFO:teuthology.orchestra.run.smithi099.stdout: "instance_id": 452136274422638824 2023-05-31T21:28:43.049 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:28:43.050 INFO:teuthology.orchestra.run.smithi099.stdout: "markers": [ 2023-05-31T21:28:43.050 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T21:28:43.050 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 0, 2023-05-31T21:28:43.050 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T21:28:43.050 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "incremental-sync", 2023-05-31T21:28:43.050 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "00000000000000000000:00000000000000045235", 2023-05-31T21:28:43.051 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T21:28:43.051 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2023-05-31T21:28:43.051 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T21:28:43.051 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2023-05-31T21:25:53.171643Z" 2023-05-31T21:28:43.051 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T21:28:43.051 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:28:43.051 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T21:28:43.052 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 1, 2023-05-31T21:28:43.052 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T21:28:43.052 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "incremental-sync", 2023-05-31T21:28:43.052 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "00000000000000000000:00000000000000031640", 2023-05-31T21:28:43.052 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T21:28:43.052 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2023-05-31T21:28:43.053 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T21:28:43.053 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2023-05-31T21:27:37.214586Z" 2023-05-31T21:28:43.053 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T21:28:43.053 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:28:43.053 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T21:28:43.053 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 2, 2023-05-31T21:28:43.053 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T21:28:43.054 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "incremental-sync", 2023-05-31T21:28:43.054 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "00000000000000000000:00000000000000051391", 2023-05-31T21:28:43.054 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T21:28:43.054 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 55, 2023-05-31T21:28:43.054 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T21:28:43.054 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2023-05-31T21:25:53.141700Z" 2023-05-31T21:28:43.054 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T21:28:43.055 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:28:43.055 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T21:28:43.055 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 3, 2023-05-31T21:28:43.055 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T21:28:43.055 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "incremental-sync", 2023-05-31T21:28:43.055 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "00000000000000000000:00000000000000044178", 2023-05-31T21:28:43.055 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T21:28:43.056 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 9, 2023-05-31T21:28:43.056 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T21:28:43.056 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2023-05-31T21:27:37.215905Z" 2023-05-31T21:28:43.056 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T21:28:43.056 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T21:28:43.056 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2023-05-31T21:28:43.056 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:28:43.057 INFO:teuthology.orchestra.run.smithi099.stdout: "full_sync": { 2023-05-31T21:28:43.057 INFO:teuthology.orchestra.run.smithi099.stdout: "total": 64, 2023-05-31T21:28:43.057 INFO:teuthology.orchestra.run.smithi099.stdout: "complete": 64 2023-05-31T21:28:43.057 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:28:43.057 INFO:teuthology.orchestra.run.smithi099.stdout: "current_time": "2023-05-31T21:28:43Z" 2023-05-31T21:28:43.057 INFO:teuthology.orchestra.run.smithi099.stdout:} 2023-05-31T21:28:43.058 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 452136274422638824 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000045235", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-05-31T21:25:53.171643Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000031640", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-05-31T21:27:37.214586Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000051391", "next_step_marker": "", "total_entries": 55, "pos": 0, "timestamp": "2023-05-31T21:25:53.141700Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000044178", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2023-05-31T21:27:37.215905Z" } } ] }, "full_sync": { "total": 64, "complete": 64 }, "current_time": "2023-05-31T21:28:43Z" } 2023-05-31T21:28:43.059 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000045235', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-05-31T21:25:53.171643Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000031640', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-05-31T21:27:37.214586Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000051391', 'next_step_marker': '', 'total_entries': 55, 'pos': 0, 'timestamp': '2023-05-31T21:25:53.141700Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000044178', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2023-05-31T21:27:37.215905Z'}}] 2023-05-31T21:28:43.059 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000045375', 1: '00000000000000000000:00000000000000031780', 2: '00000000000000000000:00000000000000051671', 3: '00000000000000000000:00000000000000044178'} 2023-05-31T21:28:43.060 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000045235', 1: '00000000000000000000:00000000000000031640', 2: '00000000000000000000:00000000000000051391', 3: '00000000000000000000:00000000000000044178'} 2023-05-31T21:28:43.060 WARNING:rgw_multi.tests:data of zone a2 behind zone a1: shard=0 master=00000000000000000000:00000000000000045375 target=00000000000000000000:00000000000000045235, shard=1 master=00000000000000000000:00000000000000031780 target=00000000000000000000:00000000000000031640, shard=2 master=00000000000000000000:00000000000000051671 target=00000000000000000000:00000000000000051391 2023-05-31T21:28:48.061 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-31T21:28:48.062 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-31T21:28:48.062 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-05-31T21:28:48.124 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2023-05-31T21:28:48.124 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2023-05-31T21:28:48.196 INFO:teuthology.orchestra.run.smithi099.stdout:{ 2023-05-31T21:28:48.197 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_status": { 2023-05-31T21:28:48.197 INFO:teuthology.orchestra.run.smithi099.stdout: "info": { 2023-05-31T21:28:48.197 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "sync", 2023-05-31T21:28:48.197 INFO:teuthology.orchestra.run.smithi099.stdout: "num_shards": 4, 2023-05-31T21:28:48.197 INFO:teuthology.orchestra.run.smithi099.stdout: "instance_id": 452136274422638824 2023-05-31T21:28:48.198 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:28:48.198 INFO:teuthology.orchestra.run.smithi099.stdout: "markers": [ 2023-05-31T21:28:48.198 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T21:28:48.198 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 0, 2023-05-31T21:28:48.198 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T21:28:48.198 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "incremental-sync", 2023-05-31T21:28:48.198 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "00000000000000000000:00000000000000045235", 2023-05-31T21:28:48.199 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T21:28:48.199 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2023-05-31T21:28:48.199 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T21:28:48.199 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2023-05-31T21:25:53.171643Z" 2023-05-31T21:28:48.199 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T21:28:48.199 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:28:48.200 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T21:28:48.200 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 1, 2023-05-31T21:28:48.200 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T21:28:48.200 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "incremental-sync", 2023-05-31T21:28:48.200 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "00000000000000000000:00000000000000031640", 2023-05-31T21:28:48.200 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T21:28:48.200 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2023-05-31T21:28:48.201 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T21:28:48.201 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2023-05-31T21:27:37.214586Z" 2023-05-31T21:28:48.201 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T21:28:48.201 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:28:48.201 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T21:28:48.201 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 2, 2023-05-31T21:28:48.202 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T21:28:48.202 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "incremental-sync", 2023-05-31T21:28:48.202 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "00000000000000000000:00000000000000051391", 2023-05-31T21:28:48.202 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T21:28:48.202 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 55, 2023-05-31T21:28:48.202 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T21:28:48.202 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2023-05-31T21:25:53.141700Z" 2023-05-31T21:28:48.203 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T21:28:48.203 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:28:48.203 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T21:28:48.203 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 3, 2023-05-31T21:28:48.203 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T21:28:48.203 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "incremental-sync", 2023-05-31T21:28:48.203 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "00000000000000000000:00000000000000044178", 2023-05-31T21:28:48.204 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T21:28:48.204 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 9, 2023-05-31T21:28:48.204 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T21:28:48.204 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2023-05-31T21:27:37.215905Z" 2023-05-31T21:28:48.204 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T21:28:48.204 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T21:28:48.205 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2023-05-31T21:28:48.205 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:28:48.205 INFO:teuthology.orchestra.run.smithi099.stdout: "full_sync": { 2023-05-31T21:28:48.205 INFO:teuthology.orchestra.run.smithi099.stdout: "total": 64, 2023-05-31T21:28:48.205 INFO:teuthology.orchestra.run.smithi099.stdout: "complete": 64 2023-05-31T21:28:48.205 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:28:48.205 INFO:teuthology.orchestra.run.smithi099.stdout: "current_time": "2023-05-31T21:28:48Z" 2023-05-31T21:28:48.206 INFO:teuthology.orchestra.run.smithi099.stdout:} 2023-05-31T21:28:48.207 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 452136274422638824 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000045235", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-05-31T21:25:53.171643Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000031640", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-05-31T21:27:37.214586Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000051391", "next_step_marker": "", "total_entries": 55, "pos": 0, "timestamp": "2023-05-31T21:25:53.141700Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000044178", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2023-05-31T21:27:37.215905Z" } } ] }, "full_sync": { "total": 64, "complete": 64 }, "current_time": "2023-05-31T21:28:48Z" } 2023-05-31T21:28:48.207 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000045235', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-05-31T21:25:53.171643Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000031640', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-05-31T21:27:37.214586Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000051391', 'next_step_marker': '', 'total_entries': 55, 'pos': 0, 'timestamp': '2023-05-31T21:25:53.141700Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000044178', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2023-05-31T21:27:37.215905Z'}}] 2023-05-31T21:28:48.208 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000045375', 1: '00000000000000000000:00000000000000031780', 2: '00000000000000000000:00000000000000051671', 3: '00000000000000000000:00000000000000044178'} 2023-05-31T21:28:48.208 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000045235', 1: '00000000000000000000:00000000000000031640', 2: '00000000000000000000:00000000000000051391', 3: '00000000000000000000:00000000000000044178'} 2023-05-31T21:28:48.208 WARNING:rgw_multi.tests:data of zone a2 behind zone a1: shard=0 master=00000000000000000000:00000000000000045375 target=00000000000000000000:00000000000000045235, shard=1 master=00000000000000000000:00000000000000031780 target=00000000000000000000:00000000000000031640, shard=2 master=00000000000000000000:00000000000000051671 target=00000000000000000000:00000000000000051391 2023-05-31T21:28:53.209 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-31T21:28:53.210 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-31T21:28:53.210 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-05-31T21:28:53.270 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2023-05-31T21:28:53.270 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2023-05-31T21:28:53.321 INFO:teuthology.orchestra.run.smithi099.stdout:{ 2023-05-31T21:28:53.322 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_status": { 2023-05-31T21:28:53.322 INFO:teuthology.orchestra.run.smithi099.stdout: "info": { 2023-05-31T21:28:53.322 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "sync", 2023-05-31T21:28:53.322 INFO:teuthology.orchestra.run.smithi099.stdout: "num_shards": 4, 2023-05-31T21:28:53.322 INFO:teuthology.orchestra.run.smithi099.stdout: "instance_id": 452136274422638824 2023-05-31T21:28:53.323 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:28:53.323 INFO:teuthology.orchestra.run.smithi099.stdout: "markers": [ 2023-05-31T21:28:53.323 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T21:28:53.323 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 0, 2023-05-31T21:28:53.323 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T21:28:53.323 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "incremental-sync", 2023-05-31T21:28:53.324 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "00000000000000000000:00000000000000045235", 2023-05-31T21:28:53.324 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T21:28:53.324 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2023-05-31T21:28:53.324 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T21:28:53.324 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2023-05-31T21:25:53.171643Z" 2023-05-31T21:28:53.324 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T21:28:53.325 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:28:53.325 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T21:28:53.325 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 1, 2023-05-31T21:28:53.325 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T21:28:53.325 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "incremental-sync", 2023-05-31T21:28:53.325 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "00000000000000000000:00000000000000031640", 2023-05-31T21:28:53.326 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T21:28:53.326 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2023-05-31T21:28:53.326 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T21:28:53.326 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2023-05-31T21:27:37.214586Z" 2023-05-31T21:28:53.327 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T21:28:53.327 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:28:53.327 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T21:28:53.327 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 2, 2023-05-31T21:28:53.327 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T21:28:53.328 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "incremental-sync", 2023-05-31T21:28:53.328 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "00000000000000000000:00000000000000051391", 2023-05-31T21:28:53.328 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T21:28:53.328 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 55, 2023-05-31T21:28:53.328 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T21:28:53.328 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2023-05-31T21:25:53.141700Z" 2023-05-31T21:28:53.329 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T21:28:53.329 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:28:53.329 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T21:28:53.329 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 3, 2023-05-31T21:28:53.329 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T21:28:53.329 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "incremental-sync", 2023-05-31T21:28:53.330 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "00000000000000000000:00000000000000044178", 2023-05-31T21:28:53.330 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T21:28:53.330 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 9, 2023-05-31T21:28:53.331 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T21:28:53.331 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2023-05-31T21:27:37.215905Z" 2023-05-31T21:28:53.331 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T21:28:53.331 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T21:28:53.331 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2023-05-31T21:28:53.331 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:28:53.332 INFO:teuthology.orchestra.run.smithi099.stdout: "full_sync": { 2023-05-31T21:28:53.332 INFO:teuthology.orchestra.run.smithi099.stdout: "total": 64, 2023-05-31T21:28:53.332 INFO:teuthology.orchestra.run.smithi099.stdout: "complete": 64 2023-05-31T21:28:53.332 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:28:53.332 INFO:teuthology.orchestra.run.smithi099.stdout: "current_time": "2023-05-31T21:28:53Z" 2023-05-31T21:28:53.332 INFO:teuthology.orchestra.run.smithi099.stdout:} 2023-05-31T21:28:53.333 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 452136274422638824 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000045235", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-05-31T21:25:53.171643Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000031640", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-05-31T21:27:37.214586Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000051391", "next_step_marker": "", "total_entries": 55, "pos": 0, "timestamp": "2023-05-31T21:25:53.141700Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000044178", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2023-05-31T21:27:37.215905Z" } } ] }, "full_sync": { "total": 64, "complete": 64 }, "current_time": "2023-05-31T21:28:53Z" } 2023-05-31T21:28:53.334 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000045235', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-05-31T21:25:53.171643Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000031640', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-05-31T21:27:37.214586Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000051391', 'next_step_marker': '', 'total_entries': 55, 'pos': 0, 'timestamp': '2023-05-31T21:25:53.141700Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000044178', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2023-05-31T21:27:37.215905Z'}}] 2023-05-31T21:28:53.334 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000045375', 1: '00000000000000000000:00000000000000031780', 2: '00000000000000000000:00000000000000051671', 3: '00000000000000000000:00000000000000044178'} 2023-05-31T21:28:53.335 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000045235', 1: '00000000000000000000:00000000000000031640', 2: '00000000000000000000:00000000000000051391', 3: '00000000000000000000:00000000000000044178'} 2023-05-31T21:28:53.335 WARNING:rgw_multi.tests:data of zone a2 behind zone a1: shard=0 master=00000000000000000000:00000000000000045375 target=00000000000000000000:00000000000000045235, shard=1 master=00000000000000000000:00000000000000031780 target=00000000000000000000:00000000000000031640, shard=2 master=00000000000000000000:00000000000000051671 target=00000000000000000000:00000000000000051391 2023-05-31T21:28:58.336 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-31T21:28:58.337 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-31T21:28:58.338 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-05-31T21:28:58.396 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2023-05-31T21:28:58.397 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2023-05-31T21:28:58.468 INFO:teuthology.orchestra.run.smithi099.stdout:{ 2023-05-31T21:28:58.468 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_status": { 2023-05-31T21:28:58.468 INFO:teuthology.orchestra.run.smithi099.stdout: "info": { 2023-05-31T21:28:58.468 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "sync", 2023-05-31T21:28:58.469 INFO:teuthology.orchestra.run.smithi099.stdout: "num_shards": 4, 2023-05-31T21:28:58.469 INFO:teuthology.orchestra.run.smithi099.stdout: "instance_id": 452136274422638824 2023-05-31T21:28:58.469 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:28:58.469 INFO:teuthology.orchestra.run.smithi099.stdout: "markers": [ 2023-05-31T21:28:58.469 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T21:28:58.469 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 0, 2023-05-31T21:28:58.470 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T21:28:58.470 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "incremental-sync", 2023-05-31T21:28:58.470 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "00000000000000000000:00000000000000045235", 2023-05-31T21:28:58.470 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T21:28:58.470 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2023-05-31T21:28:58.470 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T21:28:58.470 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2023-05-31T21:25:53.171643Z" 2023-05-31T21:28:58.471 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T21:28:58.471 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:28:58.471 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T21:28:58.471 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 1, 2023-05-31T21:28:58.471 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T21:28:58.471 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "incremental-sync", 2023-05-31T21:28:58.471 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "00000000000000000000:00000000000000031640", 2023-05-31T21:28:58.472 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T21:28:58.472 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2023-05-31T21:28:58.472 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T21:28:58.472 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2023-05-31T21:27:37.214586Z" 2023-05-31T21:28:58.472 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T21:28:58.472 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:28:58.473 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T21:28:58.473 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 2, 2023-05-31T21:28:58.473 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T21:28:58.473 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "incremental-sync", 2023-05-31T21:28:58.473 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "00000000000000000000:00000000000000051391", 2023-05-31T21:28:58.473 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T21:28:58.473 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 55, 2023-05-31T21:28:58.474 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T21:28:58.474 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2023-05-31T21:25:53.141700Z" 2023-05-31T21:28:58.474 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T21:28:58.474 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:28:58.474 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T21:28:58.474 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 3, 2023-05-31T21:28:58.474 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T21:28:58.475 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "incremental-sync", 2023-05-31T21:28:58.475 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "00000000000000000000:00000000000000044178", 2023-05-31T21:28:58.475 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T21:28:58.475 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 9, 2023-05-31T21:28:58.475 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T21:28:58.475 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2023-05-31T21:27:37.215905Z" 2023-05-31T21:28:58.475 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T21:28:58.476 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T21:28:58.476 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2023-05-31T21:28:58.476 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:28:58.476 INFO:teuthology.orchestra.run.smithi099.stdout: "full_sync": { 2023-05-31T21:28:58.476 INFO:teuthology.orchestra.run.smithi099.stdout: "total": 64, 2023-05-31T21:28:58.476 INFO:teuthology.orchestra.run.smithi099.stdout: "complete": 64 2023-05-31T21:28:58.476 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:28:58.477 INFO:teuthology.orchestra.run.smithi099.stdout: "current_time": "2023-05-31T21:28:58Z" 2023-05-31T21:28:58.477 INFO:teuthology.orchestra.run.smithi099.stdout:} 2023-05-31T21:28:58.478 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 452136274422638824 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000045235", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-05-31T21:25:53.171643Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000031640", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-05-31T21:27:37.214586Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000051391", "next_step_marker": "", "total_entries": 55, "pos": 0, "timestamp": "2023-05-31T21:25:53.141700Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000044178", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2023-05-31T21:27:37.215905Z" } } ] }, "full_sync": { "total": 64, "complete": 64 }, "current_time": "2023-05-31T21:28:58Z" } 2023-05-31T21:28:58.478 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000045235', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-05-31T21:25:53.171643Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000031640', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-05-31T21:27:37.214586Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000051391', 'next_step_marker': '', 'total_entries': 55, 'pos': 0, 'timestamp': '2023-05-31T21:25:53.141700Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000044178', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2023-05-31T21:27:37.215905Z'}}] 2023-05-31T21:28:58.479 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000045375', 1: '00000000000000000000:00000000000000031780', 2: '00000000000000000000:00000000000000051671', 3: '00000000000000000000:00000000000000044178'} 2023-05-31T21:28:58.479 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000045235', 1: '00000000000000000000:00000000000000031640', 2: '00000000000000000000:00000000000000051391', 3: '00000000000000000000:00000000000000044178'} 2023-05-31T21:28:58.479 WARNING:rgw_multi.tests:data of zone a2 behind zone a1: shard=0 master=00000000000000000000:00000000000000045375 target=00000000000000000000:00000000000000045235, shard=1 master=00000000000000000000:00000000000000031780 target=00000000000000000000:00000000000000031640, shard=2 master=00000000000000000000:00000000000000051671 target=00000000000000000000:00000000000000051391 2023-05-31T21:29:03.481 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-31T21:29:03.481 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-31T21:29:03.482 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-05-31T21:29:03.541 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2023-05-31T21:29:03.541 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2023-05-31T21:29:03.598 INFO:teuthology.orchestra.run.smithi099.stdout:{ 2023-05-31T21:29:03.599 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_status": { 2023-05-31T21:29:03.599 INFO:teuthology.orchestra.run.smithi099.stdout: "info": { 2023-05-31T21:29:03.599 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "sync", 2023-05-31T21:29:03.599 INFO:teuthology.orchestra.run.smithi099.stdout: "num_shards": 4, 2023-05-31T21:29:03.599 INFO:teuthology.orchestra.run.smithi099.stdout: "instance_id": 452136274422638824 2023-05-31T21:29:03.600 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:29:03.600 INFO:teuthology.orchestra.run.smithi099.stdout: "markers": [ 2023-05-31T21:29:03.600 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T21:29:03.600 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 0, 2023-05-31T21:29:03.600 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T21:29:03.600 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "incremental-sync", 2023-05-31T21:29:03.601 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "00000000000000000000:00000000000000045235", 2023-05-31T21:29:03.601 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T21:29:03.601 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2023-05-31T21:29:03.601 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T21:29:03.601 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2023-05-31T21:25:53.171643Z" 2023-05-31T21:29:03.601 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T21:29:03.602 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:29:03.602 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T21:29:03.602 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 1, 2023-05-31T21:29:03.602 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T21:29:03.602 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "incremental-sync", 2023-05-31T21:29:03.602 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "00000000000000000000:00000000000000031640", 2023-05-31T21:29:03.603 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T21:29:03.603 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2023-05-31T21:29:03.603 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T21:29:03.603 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2023-05-31T21:27:37.214586Z" 2023-05-31T21:29:03.603 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T21:29:03.603 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:29:03.604 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T21:29:03.604 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 2, 2023-05-31T21:29:03.604 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T21:29:03.604 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "incremental-sync", 2023-05-31T21:29:03.604 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "00000000000000000000:00000000000000051391", 2023-05-31T21:29:03.604 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T21:29:03.605 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 55, 2023-05-31T21:29:03.605 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T21:29:03.605 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2023-05-31T21:25:53.141700Z" 2023-05-31T21:29:03.605 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T21:29:03.605 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:29:03.605 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T21:29:03.606 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 3, 2023-05-31T21:29:03.606 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T21:29:03.606 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "incremental-sync", 2023-05-31T21:29:03.607 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "00000000000000000000:00000000000000044178", 2023-05-31T21:29:03.607 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T21:29:03.607 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 9, 2023-05-31T21:29:03.608 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T21:29:03.608 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2023-05-31T21:27:37.215905Z" 2023-05-31T21:29:03.608 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T21:29:03.609 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T21:29:03.609 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2023-05-31T21:29:03.609 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:29:03.609 INFO:teuthology.orchestra.run.smithi099.stdout: "full_sync": { 2023-05-31T21:29:03.610 INFO:teuthology.orchestra.run.smithi099.stdout: "total": 64, 2023-05-31T21:29:03.610 INFO:teuthology.orchestra.run.smithi099.stdout: "complete": 64 2023-05-31T21:29:03.610 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:29:03.611 INFO:teuthology.orchestra.run.smithi099.stdout: "current_time": "2023-05-31T21:29:03Z" 2023-05-31T21:29:03.611 INFO:teuthology.orchestra.run.smithi099.stdout:} 2023-05-31T21:29:03.613 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 452136274422638824 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000045235", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-05-31T21:25:53.171643Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000031640", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-05-31T21:27:37.214586Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000051391", "next_step_marker": "", "total_entries": 55, "pos": 0, "timestamp": "2023-05-31T21:25:53.141700Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000044178", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2023-05-31T21:27:37.215905Z" } } ] }, "full_sync": { "total": 64, "complete": 64 }, "current_time": "2023-05-31T21:29:03Z" } 2023-05-31T21:29:03.614 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000045235', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-05-31T21:25:53.171643Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000031640', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-05-31T21:27:37.214586Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000051391', 'next_step_marker': '', 'total_entries': 55, 'pos': 0, 'timestamp': '2023-05-31T21:25:53.141700Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000044178', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2023-05-31T21:27:37.215905Z'}}] 2023-05-31T21:29:03.614 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000045375', 1: '00000000000000000000:00000000000000031780', 2: '00000000000000000000:00000000000000051671', 3: '00000000000000000000:00000000000000044178'} 2023-05-31T21:29:03.615 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000045235', 1: '00000000000000000000:00000000000000031640', 2: '00000000000000000000:00000000000000051391', 3: '00000000000000000000:00000000000000044178'} 2023-05-31T21:29:03.615 WARNING:rgw_multi.tests:data of zone a2 behind zone a1: shard=0 master=00000000000000000000:00000000000000045375 target=00000000000000000000:00000000000000045235, shard=1 master=00000000000000000000:00000000000000031780 target=00000000000000000000:00000000000000031640, shard=2 master=00000000000000000000:00000000000000051671 target=00000000000000000000:00000000000000051391 2023-05-31T21:29:08.617 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-31T21:29:08.617 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-31T21:29:08.618 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-05-31T21:29:08.676 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2023-05-31T21:29:08.677 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2023-05-31T21:29:08.753 INFO:teuthology.orchestra.run.smithi099.stdout:{ 2023-05-31T21:29:08.754 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_status": { 2023-05-31T21:29:08.754 INFO:teuthology.orchestra.run.smithi099.stdout: "info": { 2023-05-31T21:29:08.754 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "sync", 2023-05-31T21:29:08.754 INFO:teuthology.orchestra.run.smithi099.stdout: "num_shards": 4, 2023-05-31T21:29:08.754 INFO:teuthology.orchestra.run.smithi099.stdout: "instance_id": 452136274422638824 2023-05-31T21:29:08.755 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:29:08.755 INFO:teuthology.orchestra.run.smithi099.stdout: "markers": [ 2023-05-31T21:29:08.755 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T21:29:08.755 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 0, 2023-05-31T21:29:08.755 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T21:29:08.755 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "incremental-sync", 2023-05-31T21:29:08.755 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "00000000000000000000:00000000000000045235", 2023-05-31T21:29:08.756 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T21:29:08.756 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2023-05-31T21:29:08.756 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T21:29:08.756 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2023-05-31T21:25:53.171643Z" 2023-05-31T21:29:08.756 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T21:29:08.756 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:29:08.757 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T21:29:08.757 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 1, 2023-05-31T21:29:08.757 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T21:29:08.757 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "incremental-sync", 2023-05-31T21:29:08.757 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "00000000000000000000:00000000000000031640", 2023-05-31T21:29:08.757 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T21:29:08.757 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2023-05-31T21:29:08.758 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T21:29:08.758 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2023-05-31T21:27:37.214586Z" 2023-05-31T21:29:08.758 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T21:29:08.758 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:29:08.758 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T21:29:08.758 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 2, 2023-05-31T21:29:08.759 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T21:29:08.759 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "incremental-sync", 2023-05-31T21:29:08.759 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "00000000000000000000:00000000000000051391", 2023-05-31T21:29:08.759 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T21:29:08.759 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 55, 2023-05-31T21:29:08.759 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T21:29:08.759 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2023-05-31T21:25:53.141700Z" 2023-05-31T21:29:08.760 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T21:29:08.760 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:29:08.760 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T21:29:08.760 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 3, 2023-05-31T21:29:08.760 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T21:29:08.760 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "incremental-sync", 2023-05-31T21:29:08.760 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "00000000000000000000:00000000000000044178", 2023-05-31T21:29:08.761 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T21:29:08.761 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 9, 2023-05-31T21:29:08.761 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T21:29:08.761 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2023-05-31T21:27:37.215905Z" 2023-05-31T21:29:08.761 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T21:29:08.761 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T21:29:08.761 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2023-05-31T21:29:08.762 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:29:08.762 INFO:teuthology.orchestra.run.smithi099.stdout: "full_sync": { 2023-05-31T21:29:08.762 INFO:teuthology.orchestra.run.smithi099.stdout: "total": 64, 2023-05-31T21:29:08.762 INFO:teuthology.orchestra.run.smithi099.stdout: "complete": 64 2023-05-31T21:29:08.762 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:29:08.762 INFO:teuthology.orchestra.run.smithi099.stdout: "current_time": "2023-05-31T21:29:08Z" 2023-05-31T21:29:08.762 INFO:teuthology.orchestra.run.smithi099.stdout:} 2023-05-31T21:29:08.763 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 452136274422638824 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000045235", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-05-31T21:25:53.171643Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000031640", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-05-31T21:27:37.214586Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000051391", "next_step_marker": "", "total_entries": 55, "pos": 0, "timestamp": "2023-05-31T21:25:53.141700Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000044178", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2023-05-31T21:27:37.215905Z" } } ] }, "full_sync": { "total": 64, "complete": 64 }, "current_time": "2023-05-31T21:29:08Z" } 2023-05-31T21:29:08.764 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000045235', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-05-31T21:25:53.171643Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000031640', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-05-31T21:27:37.214586Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000051391', 'next_step_marker': '', 'total_entries': 55, 'pos': 0, 'timestamp': '2023-05-31T21:25:53.141700Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000044178', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2023-05-31T21:27:37.215905Z'}}] 2023-05-31T21:29:08.764 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000045375', 1: '00000000000000000000:00000000000000031780', 2: '00000000000000000000:00000000000000051671', 3: '00000000000000000000:00000000000000044178'} 2023-05-31T21:29:08.765 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000045235', 1: '00000000000000000000:00000000000000031640', 2: '00000000000000000000:00000000000000051391', 3: '00000000000000000000:00000000000000044178'} 2023-05-31T21:29:08.765 WARNING:rgw_multi.tests:data of zone a2 behind zone a1: shard=0 master=00000000000000000000:00000000000000045375 target=00000000000000000000:00000000000000045235, shard=1 master=00000000000000000000:00000000000000031780 target=00000000000000000000:00000000000000031640, shard=2 master=00000000000000000000:00000000000000051671 target=00000000000000000000:00000000000000051391 2023-05-31T21:29:13.766 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-31T21:29:13.767 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-31T21:29:13.767 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-05-31T21:29:13.827 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2023-05-31T21:29:13.827 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2023-05-31T21:29:13.897 INFO:teuthology.orchestra.run.smithi099.stdout:{ 2023-05-31T21:29:13.897 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_status": { 2023-05-31T21:29:13.898 INFO:teuthology.orchestra.run.smithi099.stdout: "info": { 2023-05-31T21:29:13.898 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "sync", 2023-05-31T21:29:13.898 INFO:teuthology.orchestra.run.smithi099.stdout: "num_shards": 4, 2023-05-31T21:29:13.898 INFO:teuthology.orchestra.run.smithi099.stdout: "instance_id": 452136274422638824 2023-05-31T21:29:13.899 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:29:13.899 INFO:teuthology.orchestra.run.smithi099.stdout: "markers": [ 2023-05-31T21:29:13.899 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T21:29:13.899 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 0, 2023-05-31T21:29:13.899 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T21:29:13.899 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "incremental-sync", 2023-05-31T21:29:13.900 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "00000000000000000000:00000000000000045235", 2023-05-31T21:29:13.900 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T21:29:13.900 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2023-05-31T21:29:13.900 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T21:29:13.901 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2023-05-31T21:25:53.171643Z" 2023-05-31T21:29:13.901 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T21:29:13.901 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:29:13.901 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T21:29:13.901 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 1, 2023-05-31T21:29:13.902 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T21:29:13.902 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "incremental-sync", 2023-05-31T21:29:13.902 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "00000000000000000000:00000000000000031640", 2023-05-31T21:29:13.902 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T21:29:13.902 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2023-05-31T21:29:13.902 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T21:29:13.902 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2023-05-31T21:27:37.214586Z" 2023-05-31T21:29:13.903 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T21:29:13.903 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:29:13.904 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T21:29:13.904 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 2, 2023-05-31T21:29:13.904 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T21:29:13.904 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "incremental-sync", 2023-05-31T21:29:13.904 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "00000000000000000000:00000000000000051391", 2023-05-31T21:29:13.905 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T21:29:13.905 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 55, 2023-05-31T21:29:13.905 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T21:29:13.905 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2023-05-31T21:25:53.141700Z" 2023-05-31T21:29:13.905 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T21:29:13.905 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:29:13.905 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T21:29:13.906 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 3, 2023-05-31T21:29:13.906 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T21:29:13.906 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "incremental-sync", 2023-05-31T21:29:13.906 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "00000000000000000000:00000000000000044178", 2023-05-31T21:29:13.906 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T21:29:13.906 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 9, 2023-05-31T21:29:13.907 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T21:29:13.907 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2023-05-31T21:27:37.215905Z" 2023-05-31T21:29:13.907 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T21:29:13.907 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T21:29:13.907 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2023-05-31T21:29:13.907 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:29:13.907 INFO:teuthology.orchestra.run.smithi099.stdout: "full_sync": { 2023-05-31T21:29:13.908 INFO:teuthology.orchestra.run.smithi099.stdout: "total": 64, 2023-05-31T21:29:13.908 INFO:teuthology.orchestra.run.smithi099.stdout: "complete": 64 2023-05-31T21:29:13.908 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:29:13.908 INFO:teuthology.orchestra.run.smithi099.stdout: "current_time": "2023-05-31T21:29:13Z" 2023-05-31T21:29:13.908 INFO:teuthology.orchestra.run.smithi099.stdout:} 2023-05-31T21:29:13.909 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 452136274422638824 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000045235", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-05-31T21:25:53.171643Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000031640", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-05-31T21:27:37.214586Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000051391", "next_step_marker": "", "total_entries": 55, "pos": 0, "timestamp": "2023-05-31T21:25:53.141700Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000044178", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2023-05-31T21:27:37.215905Z" } } ] }, "full_sync": { "total": 64, "complete": 64 }, "current_time": "2023-05-31T21:29:13Z" } 2023-05-31T21:29:13.910 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000045235', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-05-31T21:25:53.171643Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000031640', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-05-31T21:27:37.214586Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000051391', 'next_step_marker': '', 'total_entries': 55, 'pos': 0, 'timestamp': '2023-05-31T21:25:53.141700Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000044178', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2023-05-31T21:27:37.215905Z'}}] 2023-05-31T21:29:13.911 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000045375', 1: '00000000000000000000:00000000000000031780', 2: '00000000000000000000:00000000000000051671', 3: '00000000000000000000:00000000000000044178'} 2023-05-31T21:29:13.911 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000045235', 1: '00000000000000000000:00000000000000031640', 2: '00000000000000000000:00000000000000051391', 3: '00000000000000000000:00000000000000044178'} 2023-05-31T21:29:13.911 WARNING:rgw_multi.tests:data of zone a2 behind zone a1: shard=0 master=00000000000000000000:00000000000000045375 target=00000000000000000000:00000000000000045235, shard=1 master=00000000000000000000:00000000000000031780 target=00000000000000000000:00000000000000031640, shard=2 master=00000000000000000000:00000000000000051671 target=00000000000000000000:00000000000000051391 2023-05-31T21:29:17.718 INFO:tasks.rgw.c2.client.1.smithi112.stdout:2023-05-31T21:29:17.714+0000 7fecb1022700 -1 WARNING: The OSD cluster is overloaded and struggling to complete ops. You need more capacity to serve this level of demand. 2023-05-31T21:29:18.913 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-31T21:29:18.913 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-31T21:29:18.914 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-05-31T21:29:18.972 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2023-05-31T21:29:18.973 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2023-05-31T21:29:19.060 INFO:teuthology.orchestra.run.smithi099.stdout:{ 2023-05-31T21:29:19.061 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_status": { 2023-05-31T21:29:19.061 INFO:teuthology.orchestra.run.smithi099.stdout: "info": { 2023-05-31T21:29:19.061 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "sync", 2023-05-31T21:29:19.061 INFO:teuthology.orchestra.run.smithi099.stdout: "num_shards": 4, 2023-05-31T21:29:19.061 INFO:teuthology.orchestra.run.smithi099.stdout: "instance_id": 452136274422638824 2023-05-31T21:29:19.061 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:29:19.062 INFO:teuthology.orchestra.run.smithi099.stdout: "markers": [ 2023-05-31T21:29:19.062 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T21:29:19.062 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 0, 2023-05-31T21:29:19.062 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T21:29:19.062 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "incremental-sync", 2023-05-31T21:29:19.062 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "00000000000000000000:00000000000000045375", 2023-05-31T21:29:19.063 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T21:29:19.063 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2023-05-31T21:29:19.063 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T21:29:19.063 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2023-05-31T21:28:16.422581Z" 2023-05-31T21:29:19.063 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T21:29:19.063 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:29:19.063 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T21:29:19.064 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 1, 2023-05-31T21:29:19.064 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T21:29:19.064 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "incremental-sync", 2023-05-31T21:29:19.064 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "00000000000000000000:00000000000000031780", 2023-05-31T21:29:19.064 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T21:29:19.064 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2023-05-31T21:29:19.064 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T21:29:19.065 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2023-05-31T21:28:16.501293Z" 2023-05-31T21:29:19.065 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T21:29:19.065 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:29:19.065 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T21:29:19.065 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 2, 2023-05-31T21:29:19.065 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T21:29:19.065 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "incremental-sync", 2023-05-31T21:29:19.066 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "00000000000000000000:00000000000000051671", 2023-05-31T21:29:19.066 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T21:29:19.066 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 55, 2023-05-31T21:29:19.066 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T21:29:19.066 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2023-05-31T21:28:16.476487Z" 2023-05-31T21:29:19.066 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T21:29:19.067 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:29:19.067 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T21:29:19.067 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 3, 2023-05-31T21:29:19.067 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T21:29:19.067 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "incremental-sync", 2023-05-31T21:29:19.067 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "00000000000000000000:00000000000000044178", 2023-05-31T21:29:19.067 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T21:29:19.068 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 9, 2023-05-31T21:29:19.068 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T21:29:19.068 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2023-05-31T21:27:37.215905Z" 2023-05-31T21:29:19.068 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T21:29:19.068 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T21:29:19.068 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2023-05-31T21:29:19.068 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:29:19.069 INFO:teuthology.orchestra.run.smithi099.stdout: "full_sync": { 2023-05-31T21:29:19.069 INFO:teuthology.orchestra.run.smithi099.stdout: "total": 64, 2023-05-31T21:29:19.069 INFO:teuthology.orchestra.run.smithi099.stdout: "complete": 64 2023-05-31T21:29:19.069 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:29:19.069 INFO:teuthology.orchestra.run.smithi099.stdout: "current_time": "2023-05-31T21:29:19Z" 2023-05-31T21:29:19.069 INFO:teuthology.orchestra.run.smithi099.stdout:} 2023-05-31T21:29:19.070 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 452136274422638824 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000045375", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-05-31T21:28:16.422581Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000031780", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-05-31T21:28:16.501293Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000051671", "next_step_marker": "", "total_entries": 55, "pos": 0, "timestamp": "2023-05-31T21:28:16.476487Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000044178", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2023-05-31T21:27:37.215905Z" } } ] }, "full_sync": { "total": 64, "complete": 64 }, "current_time": "2023-05-31T21:29:19Z" } 2023-05-31T21:29:19.071 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000045375', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-05-31T21:28:16.422581Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000031780', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-05-31T21:28:16.501293Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000051671', 'next_step_marker': '', 'total_entries': 55, 'pos': 0, 'timestamp': '2023-05-31T21:28:16.476487Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000044178', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2023-05-31T21:27:37.215905Z'}}] 2023-05-31T21:29:19.071 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000045375', 1: '00000000000000000000:00000000000000031780', 2: '00000000000000000000:00000000000000051671', 3: '00000000000000000000:00000000000000044178'} 2023-05-31T21:29:19.071 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000045375', 1: '00000000000000000000:00000000000000031780', 2: '00000000000000000000:00000000000000051671', 3: '00000000000000000000:00000000000000044178'} 2023-05-31T21:29:19.072 INFO:rgw_multi.tests:finished data checkpoint for target_zone=a2 source_zone=a1 2023-05-31T21:29:19.072 DEBUG:boto:path=/pklxkd-72/ 2023-05-31T21:29:19.073 DEBUG:boto:auth_path=/pklxkd-72/ 2023-05-31T21:29:19.073 DEBUG:boto:Method: HEAD 2023-05-31T21:29:19.073 DEBUG:boto:Path: /pklxkd-72/ 2023-05-31T21:29:19.074 DEBUG:boto:Data: 2023-05-31T21:29:19.074 DEBUG:boto:Headers: {} 2023-05-31T21:29:19.074 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T21:29:19.075 DEBUG:boto:Port: 8001 2023-05-31T21:29:19.075 DEBUG:boto:Params: {} 2023-05-31T21:29:19.075 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8001} 2023-05-31T21:29:19.076 DEBUG:boto:Token: None 2023-05-31T21:29:19.076 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 21:29:19 GMT /pklxkd-72/ 2023-05-31T21:29:19.077 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:BFJJo4xAtyS1PNeIId1usN44Ywg= 2023-05-31T21:29:19.077 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 21:29:19 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:BFJJo4xAtyS1PNeIId1usN44Ywg=', 'Content-Length': '0'} 2023-05-31T21:29:19.081 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000005b7b42cf84d2e2e9-006477bc2f-2895-a2'), ('Content-Length', '0'), ('Date', 'Wed, 31 May 2023 21:29:19 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:29:19.082 DEBUG:boto:path=/pklxkd-72/obj1 2023-05-31T21:29:19.082 DEBUG:boto:auth_path=/pklxkd-72/obj1 2023-05-31T21:29:19.082 DEBUG:boto:Method: HEAD 2023-05-31T21:29:19.083 DEBUG:boto:Path: /pklxkd-72/obj1 2023-05-31T21:29:19.083 DEBUG:boto:Data: 2023-05-31T21:29:19.083 DEBUG:boto:Headers: {} 2023-05-31T21:29:19.084 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T21:29:19.084 DEBUG:boto:Port: 8001 2023-05-31T21:29:19.084 DEBUG:boto:Params: {} 2023-05-31T21:29:19.085 DEBUG:boto:Token: None 2023-05-31T21:29:19.085 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 21:29:19 GMT /pklxkd-72/obj1 2023-05-31T21:29:19.086 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:r+A6pzN1ALBdR8XWTn8ykL4OXHk= 2023-05-31T21:29:19.086 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 21:29:19 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:r+A6pzN1ALBdR8XWTn8ykL4OXHk=', 'Content-Length': '0'} 2023-05-31T21:29:19.087 DEBUG:boto:Response headers: [('Content-Length', '219'), ('x-amz-request-id', 'tx00000181d0e0a5695c8eb-006477bc2f-2895-a2'), ('Accept-Ranges', 'bytes'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 21:29:19 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:29:19.088 DEBUG:boto:path=/pklxkd-72/obj2 2023-05-31T21:29:19.088 DEBUG:boto:auth_path=/pklxkd-72/obj2 2023-05-31T21:29:19.089 DEBUG:boto:Method: HEAD 2023-05-31T21:29:19.089 DEBUG:boto:Path: /pklxkd-72/obj2 2023-05-31T21:29:19.089 DEBUG:boto:Data: 2023-05-31T21:29:19.090 DEBUG:boto:Headers: {} 2023-05-31T21:29:19.090 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T21:29:19.090 DEBUG:boto:Port: 8001 2023-05-31T21:29:19.091 DEBUG:boto:Params: {} 2023-05-31T21:29:19.091 DEBUG:boto:Token: None 2023-05-31T21:29:19.091 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 21:29:19 GMT /pklxkd-72/obj2 2023-05-31T21:29:19.092 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:P4IzF3tySTbOEpWmEoDYBOjVvIQ= 2023-05-31T21:29:19.092 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 21:29:19 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:P4IzF3tySTbOEpWmEoDYBOjVvIQ=', 'Content-Length': '0'} 2023-05-31T21:29:19.094 DEBUG:boto:Response headers: [('Content-Length', '219'), ('x-amz-request-id', 'tx000004d0aec115bf6872e-006477bc2f-2895-a2'), ('Accept-Ranges', 'bytes'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 21:29:19 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:29:19.094 DEBUG:boto:path=/pklxkd-72/obj3 2023-05-31T21:29:19.095 DEBUG:boto:auth_path=/pklxkd-72/obj3 2023-05-31T21:29:19.095 DEBUG:boto:Method: HEAD 2023-05-31T21:29:19.095 DEBUG:boto:Path: /pklxkd-72/obj3 2023-05-31T21:29:19.096 DEBUG:boto:Data: 2023-05-31T21:29:19.096 DEBUG:boto:Headers: {} 2023-05-31T21:29:19.096 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T21:29:19.097 DEBUG:boto:Port: 8001 2023-05-31T21:29:19.097 DEBUG:boto:Params: {} 2023-05-31T21:29:19.097 DEBUG:boto:Token: None 2023-05-31T21:29:19.098 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 21:29:19 GMT /pklxkd-72/obj3 2023-05-31T21:29:19.098 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:S1TArTQ8AaqFEWTbMqpujSd91X8= 2023-05-31T21:29:19.098 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 21:29:19 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:S1TArTQ8AaqFEWTbMqpujSd91X8=', 'Content-Length': '0'} 2023-05-31T21:29:19.100 DEBUG:boto:Response headers: [('Content-Length', '219'), ('x-amz-request-id', 'tx00000e6950a6c6b6e4784-006477bc2f-2895-a2'), ('Accept-Ranges', 'bytes'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 21:29:19 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:29:19.100 DEBUG:boto:path=/pklxkd-72/obj4 2023-05-31T21:29:19.101 DEBUG:boto:auth_path=/pklxkd-72/obj4 2023-05-31T21:29:19.101 DEBUG:boto:Method: HEAD 2023-05-31T21:29:19.101 DEBUG:boto:Path: /pklxkd-72/obj4 2023-05-31T21:29:19.102 DEBUG:boto:Data: 2023-05-31T21:29:19.102 DEBUG:boto:Headers: {} 2023-05-31T21:29:19.102 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T21:29:19.103 DEBUG:boto:Port: 8001 2023-05-31T21:29:19.103 DEBUG:boto:Params: {} 2023-05-31T21:29:19.103 DEBUG:boto:Token: None 2023-05-31T21:29:19.104 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 21:29:19 GMT /pklxkd-72/obj4 2023-05-31T21:29:19.104 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:zUGPQgsgcD1bptrIsSIh2SkGxjE= 2023-05-31T21:29:19.104 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 21:29:19 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:zUGPQgsgcD1bptrIsSIh2SkGxjE=', 'Content-Length': '0'} 2023-05-31T21:29:19.106 DEBUG:boto:Response headers: [('Content-Length', '219'), ('x-amz-request-id', 'tx0000079c98e2558020c81-006477bc2f-2895-a2'), ('Accept-Ranges', 'bytes'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 21:29:19 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:29:19.106 DEBUG:boto:path=/pklxkd-73/ 2023-05-31T21:29:19.107 DEBUG:boto:auth_path=/pklxkd-73/ 2023-05-31T21:29:19.107 DEBUG:boto:Method: HEAD 2023-05-31T21:29:19.107 DEBUG:boto:Path: /pklxkd-73/ 2023-05-31T21:29:19.108 DEBUG:boto:Data: 2023-05-31T21:29:19.108 DEBUG:boto:Headers: {} 2023-05-31T21:29:19.108 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T21:29:19.109 DEBUG:boto:Port: 8001 2023-05-31T21:29:19.109 DEBUG:boto:Params: {} 2023-05-31T21:29:19.109 DEBUG:boto:Token: None 2023-05-31T21:29:19.110 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 21:29:19 GMT /pklxkd-73/ 2023-05-31T21:29:19.110 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:sJckmyz/H1LJxlU3uaTHyo7egJI= 2023-05-31T21:29:19.110 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 21:29:19 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:sJckmyz/H1LJxlU3uaTHyo7egJI=', 'Content-Length': '0'} 2023-05-31T21:29:19.113 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '4'), ('X-RGW-Bytes-Used', '24'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000004c67f428c7fea22c-006477bc2f-2895-a2'), ('Content-Length', '0'), ('Date', 'Wed, 31 May 2023 21:29:19 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:29:19.113 DEBUG:boto:path=/pklxkd-73/obj1 2023-05-31T21:29:19.113 DEBUG:boto:auth_path=/pklxkd-73/obj1 2023-05-31T21:29:19.114 DEBUG:boto:Method: HEAD 2023-05-31T21:29:19.114 DEBUG:boto:Path: /pklxkd-73/obj1 2023-05-31T21:29:19.114 DEBUG:boto:Data: 2023-05-31T21:29:19.115 DEBUG:boto:Headers: {} 2023-05-31T21:29:19.115 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T21:29:19.115 DEBUG:boto:Port: 8001 2023-05-31T21:29:19.116 DEBUG:boto:Params: {} 2023-05-31T21:29:19.116 DEBUG:boto:Token: None 2023-05-31T21:29:19.116 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 21:29:19 GMT /pklxkd-73/obj1 2023-05-31T21:29:19.117 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:phEb9kqw3ArZ6zfq4zWwft5SFQ0= 2023-05-31T21:29:19.117 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 21:29:19 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:phEb9kqw3ArZ6zfq4zWwft5SFQ0=', 'Content-Length': '0'} 2023-05-31T21:29:19.119 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 21:28:16 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000009557cea09a20409f-006477bc2f-2895-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 21:29:19 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:29:19.120 DEBUG:boto:path=/pklxkd-73/obj1 2023-05-31T21:29:19.120 DEBUG:boto:auth_path=/pklxkd-73/obj1 2023-05-31T21:29:19.120 DEBUG:boto:Method: GET 2023-05-31T21:29:19.121 DEBUG:boto:Path: /pklxkd-73/obj1 2023-05-31T21:29:19.121 DEBUG:boto:Data: 2023-05-31T21:29:19.121 DEBUG:boto:Headers: {} 2023-05-31T21:29:19.122 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T21:29:19.122 DEBUG:boto:Port: 8001 2023-05-31T21:29:19.122 DEBUG:boto:Params: {} 2023-05-31T21:29:19.123 DEBUG:boto:Token: None 2023-05-31T21:29:19.123 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 21:29:19 GMT /pklxkd-73/obj1 2023-05-31T21:29:19.124 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:qYzcv4wNIUPtK73on70Hg1uRCiI= 2023-05-31T21:29:19.124 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 21:29:19 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:qYzcv4wNIUPtK73on70Hg1uRCiI=', 'Content-Length': '0'} 2023-05-31T21:29:19.126 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 21:28:16 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000af3dca5f466e1f0c-006477bc2f-2895-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 21:29:19 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:29:19.126 DEBUG:boto:path=/pklxkd-73/obj2 2023-05-31T21:29:19.127 DEBUG:boto:auth_path=/pklxkd-73/obj2 2023-05-31T21:29:19.127 DEBUG:boto:Method: HEAD 2023-05-31T21:29:19.127 DEBUG:boto:Path: /pklxkd-73/obj2 2023-05-31T21:29:19.128 DEBUG:boto:Data: 2023-05-31T21:29:19.128 DEBUG:boto:Headers: {} 2023-05-31T21:29:19.129 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T21:29:19.129 DEBUG:boto:Port: 8001 2023-05-31T21:29:19.129 DEBUG:boto:Params: {} 2023-05-31T21:29:19.130 DEBUG:boto:Token: None 2023-05-31T21:29:19.130 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 21:29:19 GMT /pklxkd-73/obj2 2023-05-31T21:29:19.131 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:ItvQ3HbDpzjecotJU0Jbr2FsyyI= 2023-05-31T21:29:19.131 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 21:29:19 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:ItvQ3HbDpzjecotJU0Jbr2FsyyI=', 'Content-Length': '0'} 2023-05-31T21:29:19.132 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 21:28:16 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000040dc51eefd9e4536-006477bc2f-2895-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 21:29:19 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:29:19.133 DEBUG:boto:path=/pklxkd-73/obj2 2023-05-31T21:29:19.133 DEBUG:boto:auth_path=/pklxkd-73/obj2 2023-05-31T21:29:19.134 DEBUG:boto:Method: GET 2023-05-31T21:29:19.134 DEBUG:boto:Path: /pklxkd-73/obj2 2023-05-31T21:29:19.134 DEBUG:boto:Data: 2023-05-31T21:29:19.135 DEBUG:boto:Headers: {} 2023-05-31T21:29:19.135 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T21:29:19.135 DEBUG:boto:Port: 8001 2023-05-31T21:29:19.136 DEBUG:boto:Params: {} 2023-05-31T21:29:19.136 DEBUG:boto:Token: None 2023-05-31T21:29:19.137 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 21:29:19 GMT /pklxkd-73/obj2 2023-05-31T21:29:19.137 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:2XfO5WHUyQj4hijiDmsxKcAN9Q0= 2023-05-31T21:29:19.137 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 21:29:19 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:2XfO5WHUyQj4hijiDmsxKcAN9Q0=', 'Content-Length': '0'} 2023-05-31T21:29:19.139 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 21:28:16 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000025fac1bc4b9fde98-006477bc2f-2895-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 21:29:19 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:29:19.180 DEBUG:boto:path=/pklxkd-73/obj3 2023-05-31T21:29:19.181 DEBUG:boto:auth_path=/pklxkd-73/obj3 2023-05-31T21:29:19.181 DEBUG:boto:Method: HEAD 2023-05-31T21:29:19.182 DEBUG:boto:Path: /pklxkd-73/obj3 2023-05-31T21:29:19.182 DEBUG:boto:Data: 2023-05-31T21:29:19.182 DEBUG:boto:Headers: {} 2023-05-31T21:29:19.183 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T21:29:19.183 DEBUG:boto:Port: 8001 2023-05-31T21:29:19.183 DEBUG:boto:Params: {} 2023-05-31T21:29:19.184 DEBUG:boto:Token: None 2023-05-31T21:29:19.184 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 21:29:19 GMT /pklxkd-73/obj3 2023-05-31T21:29:19.184 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:o7cXevH62cF5/L8I7QK4EnEwUWw= 2023-05-31T21:29:19.185 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 21:29:19 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:o7cXevH62cF5/L8I7QK4EnEwUWw=', 'Content-Length': '0'} 2023-05-31T21:29:19.186 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 21:28:16 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000fe882a74c119fc0a-006477bc2f-2895-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 21:29:19 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:29:19.187 DEBUG:boto:path=/pklxkd-73/obj3 2023-05-31T21:29:19.187 DEBUG:boto:auth_path=/pklxkd-73/obj3 2023-05-31T21:29:19.188 DEBUG:boto:Method: GET 2023-05-31T21:29:19.188 DEBUG:boto:Path: /pklxkd-73/obj3 2023-05-31T21:29:19.189 DEBUG:boto:Data: 2023-05-31T21:29:19.189 DEBUG:boto:Headers: {} 2023-05-31T21:29:19.189 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T21:29:19.190 DEBUG:boto:Port: 8001 2023-05-31T21:29:19.190 DEBUG:boto:Params: {} 2023-05-31T21:29:19.190 DEBUG:boto:Token: None 2023-05-31T21:29:19.191 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 21:29:19 GMT /pklxkd-73/obj3 2023-05-31T21:29:19.191 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:1GAZM9EFqP0vtRu3pnD4RZOM9sw= 2023-05-31T21:29:19.192 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 21:29:19 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:1GAZM9EFqP0vtRu3pnD4RZOM9sw=', 'Content-Length': '0'} 2023-05-31T21:29:19.194 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 21:28:16 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000191f35953b715922-006477bc2f-2895-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 21:29:19 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:29:19.194 DEBUG:boto:path=/pklxkd-73/obj4 2023-05-31T21:29:19.195 DEBUG:boto:auth_path=/pklxkd-73/obj4 2023-05-31T21:29:19.195 DEBUG:boto:Method: HEAD 2023-05-31T21:29:19.195 DEBUG:boto:Path: /pklxkd-73/obj4 2023-05-31T21:29:19.196 DEBUG:boto:Data: 2023-05-31T21:29:19.196 DEBUG:boto:Headers: {} 2023-05-31T21:29:19.196 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T21:29:19.197 DEBUG:boto:Port: 8001 2023-05-31T21:29:19.197 DEBUG:boto:Params: {} 2023-05-31T21:29:19.197 DEBUG:boto:Token: None 2023-05-31T21:29:19.198 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 21:29:19 GMT /pklxkd-73/obj4 2023-05-31T21:29:19.198 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:iiUaAKfJIaMhzvz7bs9BGIRJU4I= 2023-05-31T21:29:19.199 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 21:29:19 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:iiUaAKfJIaMhzvz7bs9BGIRJU4I=', 'Content-Length': '0'} 2023-05-31T21:29:19.200 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 21:28:16 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000e73a54e5b051ba27-006477bc2f-2895-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 21:29:19 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:29:19.201 DEBUG:boto:path=/pklxkd-73/obj4 2023-05-31T21:29:19.201 DEBUG:boto:auth_path=/pklxkd-73/obj4 2023-05-31T21:29:19.202 DEBUG:boto:Method: GET 2023-05-31T21:29:19.202 DEBUG:boto:Path: /pklxkd-73/obj4 2023-05-31T21:29:19.202 DEBUG:boto:Data: 2023-05-31T21:29:19.203 DEBUG:boto:Headers: {} 2023-05-31T21:29:19.203 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T21:29:19.203 DEBUG:boto:Port: 8001 2023-05-31T21:29:19.204 DEBUG:boto:Params: {} 2023-05-31T21:29:19.204 DEBUG:boto:Token: None 2023-05-31T21:29:19.205 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 21:29:19 GMT /pklxkd-73/obj4 2023-05-31T21:29:19.205 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:ylYFT2rgy+tBVS+mJt8Wl5FOJKY= 2023-05-31T21:29:19.205 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 21:29:19 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:ylYFT2rgy+tBVS+mJt8Wl5FOJKY=', 'Content-Length': '0'} 2023-05-31T21:29:19.207 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 21:28:16 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000c9e70b9197b82c25-006477bc2f-2895-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 21:29:19 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:29:19.208 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'group', 'pipe', 'remove', '--group-id', 'sync-group', '--pipe-id', 'sync-pipe-pklxkd-72', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-05-31T21:29:19.209 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'group', 'pipe', 'remove', '--group-id', 'sync-group', '--pipe-id', 'sync-pipe-pklxkd-72', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-05-31T21:29:19.209 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync group pipe remove --group-id sync-group --pipe-id sync-pipe-pklxkd-72 --cluster c1 --debug-rgw 0 --debug-ms 0 2023-05-31T21:29:19.259 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2023-05-31T21:29:19.260 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2023-05-31T21:29:19.315 INFO:teuthology.orchestra.run.smithi099.stdout:{"groups":[{"id":"sync-group","data_flow":{"directional":[{"source_zone":"a1","dest_zone":"a2"}]},"pipes":[{"id":"sync-pipe-pklxkd-73","source":{"bucket":"pklxkd-73","zones":["a1"]},"dest":{"bucket":"pklxkd-73","zones":["a2"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":""}}],"status":"enabled"}]} 2023-05-31T21:29:19.315 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'group', 'pipe', 'remove', '--group-id', 'sync-group', '--pipe-id', 'sync-pipe-pklxkd-73', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-05-31T21:29:19.316 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'group', 'pipe', 'remove', '--group-id', 'sync-group', '--pipe-id', 'sync-pipe-pklxkd-73', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-05-31T21:29:19.316 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync group pipe remove --group-id sync-group --pipe-id sync-pipe-pklxkd-73 --cluster c1 --debug-rgw 0 --debug-ms 0 2023-05-31T21:29:19.367 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2023-05-31T21:29:19.367 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2023-05-31T21:29:19.431 INFO:teuthology.orchestra.run.smithi099.stdout:{"groups":[{"id":"sync-group","data_flow":{"directional":[{"source_zone":"a1","dest_zone":"a2"}]},"pipes":[],"status":"enabled"}]} 2023-05-31T21:29:19.432 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'group', 'flow', 'remove', '--group-id', 'sync-group', '--flow-id', 'sync-flow', '--flow-type', 'directional', '--source-zone=a1', '--dest-zone=a2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-05-31T21:29:19.432 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'group', 'flow', 'remove', '--group-id', 'sync-group', '--flow-id', 'sync-flow', '--flow-type', 'directional', '--source-zone=a1', '--dest-zone=a2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-05-31T21:29:19.433 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync group flow remove --group-id sync-group --flow-id sync-flow --flow-type directional --source-zone=a1 --dest-zone=a2 --cluster c1 --debug-rgw 0 --debug-ms 0 2023-05-31T21:29:19.487 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2023-05-31T21:29:19.487 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2023-05-31T21:29:19.565 INFO:teuthology.orchestra.run.smithi099.stdout:{"groups":[{"id":"sync-group","data_flow":{},"pipes":[],"status":"enabled"}]} 2023-05-31T21:29:19.565 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'group', 'flow', 'create', '--group-id', 'sync-group', '--flow-id', 'sync-flow1', '--flow-type', 'symmetrical', '--zones=a1,a2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-05-31T21:29:19.566 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'group', 'flow', 'create', '--group-id', 'sync-group', '--flow-id', 'sync-flow1', '--flow-type', 'symmetrical', '--zones=a1,a2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-05-31T21:29:19.567 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync group flow create --group-id sync-group --flow-id sync-flow1 --flow-type symmetrical --zones=a1,a2 --cluster c1 --debug-rgw 0 --debug-ms 0 2023-05-31T21:29:19.620 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2023-05-31T21:29:19.620 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2023-05-31T21:29:19.680 INFO:teuthology.orchestra.run.smithi099.stdout:{"groups":[{"id":"sync-group","data_flow":{"symmetrical":[{"id":"sync-flow1","zones":["a1","a2"]}]},"pipes":[],"status":"enabled"}]} 2023-05-31T21:29:19.681 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'group', 'pipe', 'create', '--group-id', 'sync-group', '--pipe-id', 'sync-pipe', '--source-zones=a1,a2', '--dest-zones=a1,a2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-05-31T21:29:19.681 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'group', 'pipe', 'create', '--group-id', 'sync-group', '--pipe-id', 'sync-pipe', '--source-zones=a1,a2', '--dest-zones=a1,a2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-05-31T21:29:19.682 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync group pipe create --group-id sync-group --pipe-id sync-pipe --source-zones=a1,a2 --dest-zones=a1,a2 --cluster c1 --debug-rgw 0 --debug-ms 0 2023-05-31T21:29:19.734 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2023-05-31T21:29:19.734 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2023-05-31T21:29:19.801 INFO:teuthology.orchestra.run.smithi099.stdout:{"groups":[{"id":"sync-group","data_flow":{"symmetrical":[{"id":"sync-flow1","zones":["a1","a2"]}]},"pipes":[{"id":"sync-pipe","source":{"bucket":"*","zones":["a1","a2"]},"dest":{"bucket":"*","zones":["a1","a2"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":""}}],"status":"enabled"}]} 2023-05-31T21:29:19.801 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'group', 'modify', '--group-id', 'sync-group', '--status', 'allowed', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-05-31T21:29:19.802 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'group', 'modify', '--group-id', 'sync-group', '--status', 'allowed', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-05-31T21:29:19.803 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync group modify --group-id sync-group --status allowed --cluster c1 --debug-rgw 0 --debug-ms 0 2023-05-31T21:29:19.851 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2023-05-31T21:29:19.852 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2023-05-31T21:29:19.931 INFO:teuthology.orchestra.run.smithi099.stdout:{"groups":[{"id":"sync-group","data_flow":{"symmetrical":[{"id":"sync-flow1","zones":["a1","a2"]}]},"pipes":[{"id":"sync-pipe","source":{"bucket":"*","zones":["a1","a2"]},"dest":{"bucket":"*","zones":["a1","a2"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":""}}],"status":"allowed"}]} 2023-05-31T21:29:19.931 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['period', 'update', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-05-31T21:29:19.932 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'period', 'update', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-05-31T21:29:19.932 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 period update --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --commit --cluster c1 --debug-rgw 0 --debug-ms 0 2023-05-31T21:29:19.980 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2023-05-31T21:29:19.980 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2023-05-31T21:29:20.039 INFO:teuthology.orchestra.run.smithi099.stdout:{ 2023-05-31T21:29:20.039 INFO:teuthology.orchestra.run.smithi099.stdout: "id": "2e6123d2-d695-4350-9a01-61a455bc782b", 2023-05-31T21:29:20.040 INFO:teuthology.orchestra.run.smithi099.stdout: "epoch": 20, 2023-05-31T21:29:20.040 INFO:teuthology.orchestra.run.smithi099.stdout: "predecessor_uuid": "629b9ced-80af-41ed-8a6c-f2cfc334cbb4", 2023-05-31T21:29:20.040 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_status": [], 2023-05-31T21:29:20.040 INFO:teuthology.orchestra.run.smithi099.stdout: "period_map": { 2023-05-31T21:29:20.040 INFO:teuthology.orchestra.run.smithi099.stdout: "id": "2e6123d2-d695-4350-9a01-61a455bc782b", 2023-05-31T21:29:20.040 INFO:teuthology.orchestra.run.smithi099.stdout: "zonegroups": [ 2023-05-31T21:29:20.041 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T21:29:20.041 INFO:teuthology.orchestra.run.smithi099.stdout: "id": "a", 2023-05-31T21:29:20.041 INFO:teuthology.orchestra.run.smithi099.stdout: "name": "a", 2023-05-31T21:29:20.041 INFO:teuthology.orchestra.run.smithi099.stdout: "api_name": "", 2023-05-31T21:29:20.041 INFO:teuthology.orchestra.run.smithi099.stdout: "is_master": true, 2023-05-31T21:29:20.041 INFO:teuthology.orchestra.run.smithi099.stdout: "endpoints": [ 2023-05-31T21:29:20.041 INFO:teuthology.orchestra.run.smithi099.stdout: "http://smithi099.front.sepia.ceph.com:8000" 2023-05-31T21:29:20.042 INFO:teuthology.orchestra.run.smithi099.stdout: ], 2023-05-31T21:29:20.042 INFO:teuthology.orchestra.run.smithi099.stdout: "hostnames": [], 2023-05-31T21:29:20.042 INFO:teuthology.orchestra.run.smithi099.stdout: "hostnames_s3website": [], 2023-05-31T21:29:20.042 INFO:teuthology.orchestra.run.smithi099.stdout: "master_zone": "3d163b1e-09c7-47e2-ae87-add4aace3f38", 2023-05-31T21:29:20.042 INFO:teuthology.orchestra.run.smithi099.stdout: "zones": [ 2023-05-31T21:29:20.042 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T21:29:20.043 INFO:teuthology.orchestra.run.smithi099.stdout: "id": "3d163b1e-09c7-47e2-ae87-add4aace3f38", 2023-05-31T21:29:20.043 INFO:teuthology.orchestra.run.smithi099.stdout: "name": "a1", 2023-05-31T21:29:20.043 INFO:teuthology.orchestra.run.smithi099.stdout: "endpoints": [ 2023-05-31T21:29:20.043 INFO:teuthology.orchestra.run.smithi099.stdout: "http://smithi099.front.sepia.ceph.com:8000" 2023-05-31T21:29:20.043 INFO:teuthology.orchestra.run.smithi099.stdout: ], 2023-05-31T21:29:20.043 INFO:teuthology.orchestra.run.smithi099.stdout: "log_meta": false, 2023-05-31T21:29:20.044 INFO:teuthology.orchestra.run.smithi099.stdout: "log_data": true, 2023-05-31T21:29:20.044 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket_index_max_shards": 11, 2023-05-31T21:29:20.044 INFO:teuthology.orchestra.run.smithi099.stdout: "read_only": false, 2023-05-31T21:29:20.044 INFO:teuthology.orchestra.run.smithi099.stdout: "tier_type": "", 2023-05-31T21:29:20.044 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_from_all": true, 2023-05-31T21:29:20.044 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_from": [], 2023-05-31T21:29:20.044 INFO:teuthology.orchestra.run.smithi099.stdout: "redirect_zone": "", 2023-05-31T21:29:20.045 INFO:teuthology.orchestra.run.smithi099.stdout: "supported_features": [ 2023-05-31T21:29:20.045 INFO:teuthology.orchestra.run.smithi099.stdout: "resharding" 2023-05-31T21:29:20.045 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2023-05-31T21:29:20.045 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:29:20.045 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T21:29:20.045 INFO:teuthology.orchestra.run.smithi099.stdout: "id": "5d8acec7-7e57-4e56-8b32-e3d6b5036001", 2023-05-31T21:29:20.045 INFO:teuthology.orchestra.run.smithi099.stdout: "name": "a2", 2023-05-31T21:29:20.046 INFO:teuthology.orchestra.run.smithi099.stdout: "endpoints": [ 2023-05-31T21:29:20.046 INFO:teuthology.orchestra.run.smithi099.stdout: "http://smithi099.front.sepia.ceph.com:8001" 2023-05-31T21:29:20.046 INFO:teuthology.orchestra.run.smithi099.stdout: ], 2023-05-31T21:29:20.046 INFO:teuthology.orchestra.run.smithi099.stdout: "log_meta": false, 2023-05-31T21:29:20.046 INFO:teuthology.orchestra.run.smithi099.stdout: "log_data": true, 2023-05-31T21:29:20.046 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket_index_max_shards": 11, 2023-05-31T21:29:20.047 INFO:teuthology.orchestra.run.smithi099.stdout: "read_only": false, 2023-05-31T21:29:20.047 INFO:teuthology.orchestra.run.smithi099.stdout: "tier_type": "", 2023-05-31T21:29:20.047 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_from_all": true, 2023-05-31T21:29:20.047 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_from": [], 2023-05-31T21:29:20.047 INFO:teuthology.orchestra.run.smithi099.stdout: "redirect_zone": "", 2023-05-31T21:29:20.047 INFO:teuthology.orchestra.run.smithi099.stdout: "supported_features": [ 2023-05-31T21:29:20.047 INFO:teuthology.orchestra.run.smithi099.stdout: "resharding" 2023-05-31T21:29:20.048 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2023-05-31T21:29:20.048 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T21:29:20.048 INFO:teuthology.orchestra.run.smithi099.stdout: ], 2023-05-31T21:29:20.048 INFO:teuthology.orchestra.run.smithi099.stdout: "placement_targets": [ 2023-05-31T21:29:20.048 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T21:29:20.048 INFO:teuthology.orchestra.run.smithi099.stdout: "name": "default-placement", 2023-05-31T21:29:20.048 INFO:teuthology.orchestra.run.smithi099.stdout: "tags": [], 2023-05-31T21:29:20.049 INFO:teuthology.orchestra.run.smithi099.stdout: "storage_classes": [ 2023-05-31T21:29:20.049 INFO:teuthology.orchestra.run.smithi099.stdout: "STANDARD" 2023-05-31T21:29:20.049 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2023-05-31T21:29:20.049 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T21:29:20.049 INFO:teuthology.orchestra.run.smithi099.stdout: ], 2023-05-31T21:29:20.049 INFO:teuthology.orchestra.run.smithi099.stdout: "default_placement": "default-placement", 2023-05-31T21:29:20.049 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_id": "aaa33de6-515b-4cac-8f1b-3a1a30452117", 2023-05-31T21:29:20.050 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_policy": { 2023-05-31T21:29:20.050 INFO:teuthology.orchestra.run.smithi099.stdout: "groups": [ 2023-05-31T21:29:20.050 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T21:29:20.050 INFO:teuthology.orchestra.run.smithi099.stdout: "id": "sync-group", 2023-05-31T21:29:20.050 INFO:teuthology.orchestra.run.smithi099.stdout: "data_flow": { 2023-05-31T21:29:20.050 INFO:teuthology.orchestra.run.smithi099.stdout: "symmetrical": [ 2023-05-31T21:29:20.051 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T21:29:20.051 INFO:teuthology.orchestra.run.smithi099.stdout: "id": "sync-flow1", 2023-05-31T21:29:20.051 INFO:teuthology.orchestra.run.smithi099.stdout: "zones": [ 2023-05-31T21:29:20.051 INFO:teuthology.orchestra.run.smithi099.stdout: "3d163b1e-09c7-47e2-ae87-add4aace3f38", 2023-05-31T21:29:20.051 INFO:teuthology.orchestra.run.smithi099.stdout: "5d8acec7-7e57-4e56-8b32-e3d6b5036001" 2023-05-31T21:29:20.051 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2023-05-31T21:29:20.052 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T21:29:20.052 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2023-05-31T21:29:20.052 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:29:20.052 INFO:teuthology.orchestra.run.smithi099.stdout: "pipes": [ 2023-05-31T21:29:20.052 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T21:29:20.052 INFO:teuthology.orchestra.run.smithi099.stdout: "id": "sync-pipe", 2023-05-31T21:29:20.052 INFO:teuthology.orchestra.run.smithi099.stdout: "source": { 2023-05-31T21:29:20.053 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket": "*", 2023-05-31T21:29:20.053 INFO:teuthology.orchestra.run.smithi099.stdout: "zones": [ 2023-05-31T21:29:20.053 INFO:teuthology.orchestra.run.smithi099.stdout: "3d163b1e-09c7-47e2-ae87-add4aace3f38", 2023-05-31T21:29:20.053 INFO:teuthology.orchestra.run.smithi099.stdout: "5d8acec7-7e57-4e56-8b32-e3d6b5036001" 2023-05-31T21:29:20.053 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2023-05-31T21:29:20.053 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:29:20.053 INFO:teuthology.orchestra.run.smithi099.stdout: "dest": { 2023-05-31T21:29:20.054 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket": "*", 2023-05-31T21:29:20.054 INFO:teuthology.orchestra.run.smithi099.stdout: "zones": [ 2023-05-31T21:29:20.054 INFO:teuthology.orchestra.run.smithi099.stdout: "3d163b1e-09c7-47e2-ae87-add4aace3f38", 2023-05-31T21:29:20.054 INFO:teuthology.orchestra.run.smithi099.stdout: "5d8acec7-7e57-4e56-8b32-e3d6b5036001" 2023-05-31T21:29:20.054 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2023-05-31T21:29:20.054 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:29:20.054 INFO:teuthology.orchestra.run.smithi099.stdout: "params": { 2023-05-31T21:29:20.055 INFO:teuthology.orchestra.run.smithi099.stdout: "source": { 2023-05-31T21:29:20.055 INFO:teuthology.orchestra.run.smithi099.stdout: "filter": { 2023-05-31T21:29:20.055 INFO:teuthology.orchestra.run.smithi099.stdout: "tags": [] 2023-05-31T21:29:20.055 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T21:29:20.055 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:29:20.055 INFO:teuthology.orchestra.run.smithi099.stdout: "dest": {}, 2023-05-31T21:29:20.055 INFO:teuthology.orchestra.run.smithi099.stdout: "priority": 0, 2023-05-31T21:29:20.056 INFO:teuthology.orchestra.run.smithi099.stdout: "mode": "system", 2023-05-31T21:29:20.056 INFO:teuthology.orchestra.run.smithi099.stdout: "user": "" 2023-05-31T21:29:20.056 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T21:29:20.056 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T21:29:20.056 INFO:teuthology.orchestra.run.smithi099.stdout: ], 2023-05-31T21:29:20.056 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "allowed" 2023-05-31T21:29:20.056 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T21:29:20.057 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2023-05-31T21:29:20.057 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:29:20.057 INFO:teuthology.orchestra.run.smithi099.stdout: "enabled_features": [ 2023-05-31T21:29:20.057 INFO:teuthology.orchestra.run.smithi099.stdout: "resharding" 2023-05-31T21:29:20.057 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2023-05-31T21:29:20.057 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:29:20.057 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T21:29:20.058 INFO:teuthology.orchestra.run.smithi099.stdout: "id": "b", 2023-05-31T21:29:20.058 INFO:teuthology.orchestra.run.smithi099.stdout: "name": "b", 2023-05-31T21:29:20.058 INFO:teuthology.orchestra.run.smithi099.stdout: "api_name": "", 2023-05-31T21:29:20.058 INFO:teuthology.orchestra.run.smithi099.stdout: "is_master": false, 2023-05-31T21:29:20.058 INFO:teuthology.orchestra.run.smithi099.stdout: "endpoints": [ 2023-05-31T21:29:20.058 INFO:teuthology.orchestra.run.smithi099.stdout: "http://smithi112.front.sepia.ceph.com:8000" 2023-05-31T21:29:20.059 INFO:teuthology.orchestra.run.smithi099.stdout: ], 2023-05-31T21:29:20.059 INFO:teuthology.orchestra.run.smithi099.stdout: "hostnames": [], 2023-05-31T21:29:20.059 INFO:teuthology.orchestra.run.smithi099.stdout: "hostnames_s3website": [], 2023-05-31T21:29:20.059 INFO:teuthology.orchestra.run.smithi099.stdout: "master_zone": "f85d7236-2f27-4988-af26-9c66ab280b1b", 2023-05-31T21:29:20.059 INFO:teuthology.orchestra.run.smithi099.stdout: "zones": [ 2023-05-31T21:29:20.059 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T21:29:20.059 INFO:teuthology.orchestra.run.smithi099.stdout: "id": "a6a3aab5-bc52-44a4-8543-5c5ea6056acd", 2023-05-31T21:29:20.060 INFO:teuthology.orchestra.run.smithi099.stdout: "name": "b2", 2023-05-31T21:29:20.060 INFO:teuthology.orchestra.run.smithi099.stdout: "endpoints": [ 2023-05-31T21:29:20.060 INFO:teuthology.orchestra.run.smithi099.stdout: "http://smithi112.front.sepia.ceph.com:8001" 2023-05-31T21:29:20.060 INFO:teuthology.orchestra.run.smithi099.stdout: ], 2023-05-31T21:29:20.060 INFO:teuthology.orchestra.run.smithi099.stdout: "log_meta": false, 2023-05-31T21:29:20.060 INFO:teuthology.orchestra.run.smithi099.stdout: "log_data": true, 2023-05-31T21:29:20.060 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket_index_max_shards": 11, 2023-05-31T21:29:20.061 INFO:teuthology.orchestra.run.smithi099.stdout: "read_only": false, 2023-05-31T21:29:20.061 INFO:teuthology.orchestra.run.smithi099.stdout: "tier_type": "", 2023-05-31T21:29:20.061 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_from_all": true, 2023-05-31T21:29:20.061 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_from": [], 2023-05-31T21:29:20.061 INFO:teuthology.orchestra.run.smithi099.stdout: "redirect_zone": "", 2023-05-31T21:29:20.061 INFO:teuthology.orchestra.run.smithi099.stdout: "supported_features": [ 2023-05-31T21:29:20.061 INFO:teuthology.orchestra.run.smithi099.stdout: "resharding" 2023-05-31T21:29:20.062 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2023-05-31T21:29:20.062 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:29:20.062 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T21:29:20.062 INFO:teuthology.orchestra.run.smithi099.stdout: "id": "f85d7236-2f27-4988-af26-9c66ab280b1b", 2023-05-31T21:29:20.062 INFO:teuthology.orchestra.run.smithi099.stdout: "name": "b1", 2023-05-31T21:29:20.062 INFO:teuthology.orchestra.run.smithi099.stdout: "endpoints": [ 2023-05-31T21:29:20.062 INFO:teuthology.orchestra.run.smithi099.stdout: "http://smithi112.front.sepia.ceph.com:8000" 2023-05-31T21:29:20.063 INFO:teuthology.orchestra.run.smithi099.stdout: ], 2023-05-31T21:29:20.063 INFO:teuthology.orchestra.run.smithi099.stdout: "log_meta": false, 2023-05-31T21:29:20.063 INFO:teuthology.orchestra.run.smithi099.stdout: "log_data": true, 2023-05-31T21:29:20.063 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket_index_max_shards": 11, 2023-05-31T21:29:20.063 INFO:teuthology.orchestra.run.smithi099.stdout: "read_only": false, 2023-05-31T21:29:20.063 INFO:teuthology.orchestra.run.smithi099.stdout: "tier_type": "", 2023-05-31T21:29:20.063 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_from_all": true, 2023-05-31T21:29:20.064 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_from": [], 2023-05-31T21:29:20.064 INFO:teuthology.orchestra.run.smithi099.stdout: "redirect_zone": "", 2023-05-31T21:29:20.064 INFO:teuthology.orchestra.run.smithi099.stdout: "supported_features": [ 2023-05-31T21:29:20.064 INFO:teuthology.orchestra.run.smithi099.stdout: "resharding" 2023-05-31T21:29:20.064 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2023-05-31T21:29:20.064 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T21:29:20.064 INFO:teuthology.orchestra.run.smithi099.stdout: ], 2023-05-31T21:29:20.065 INFO:teuthology.orchestra.run.smithi099.stdout: "placement_targets": [ 2023-05-31T21:29:20.065 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T21:29:20.065 INFO:teuthology.orchestra.run.smithi099.stdout: "name": "default-placement", 2023-05-31T21:29:20.065 INFO:teuthology.orchestra.run.smithi099.stdout: "tags": [], 2023-05-31T21:29:20.065 INFO:teuthology.orchestra.run.smithi099.stdout: "storage_classes": [ 2023-05-31T21:29:20.065 INFO:teuthology.orchestra.run.smithi099.stdout: "STANDARD" 2023-05-31T21:29:20.065 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2023-05-31T21:29:20.066 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T21:29:20.066 INFO:teuthology.orchestra.run.smithi099.stdout: ], 2023-05-31T21:29:20.066 INFO:teuthology.orchestra.run.smithi099.stdout: "default_placement": "default-placement", 2023-05-31T21:29:20.066 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_id": "aaa33de6-515b-4cac-8f1b-3a1a30452117", 2023-05-31T21:29:20.066 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_policy": { 2023-05-31T21:29:20.066 INFO:teuthology.orchestra.run.smithi099.stdout: "groups": [] 2023-05-31T21:29:20.066 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:29:20.067 INFO:teuthology.orchestra.run.smithi099.stdout: "enabled_features": [ 2023-05-31T21:29:20.067 INFO:teuthology.orchestra.run.smithi099.stdout: "resharding" 2023-05-31T21:29:20.067 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2023-05-31T21:29:20.067 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T21:29:20.067 INFO:teuthology.orchestra.run.smithi099.stdout: ], 2023-05-31T21:29:20.067 INFO:teuthology.orchestra.run.smithi099.stdout: "short_zone_ids": [ 2023-05-31T21:29:20.068 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T21:29:20.068 INFO:teuthology.orchestra.run.smithi099.stdout: "key": "3d163b1e-09c7-47e2-ae87-add4aace3f38", 2023-05-31T21:29:20.068 INFO:teuthology.orchestra.run.smithi099.stdout: "val": 3101833771 2023-05-31T21:29:20.068 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:29:20.068 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T21:29:20.068 INFO:teuthology.orchestra.run.smithi099.stdout: "key": "5d8acec7-7e57-4e56-8b32-e3d6b5036001", 2023-05-31T21:29:20.068 INFO:teuthology.orchestra.run.smithi099.stdout: "val": 4255530623 2023-05-31T21:29:20.069 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:29:20.069 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T21:29:20.069 INFO:teuthology.orchestra.run.smithi099.stdout: "key": "a6a3aab5-bc52-44a4-8543-5c5ea6056acd", 2023-05-31T21:29:20.069 INFO:teuthology.orchestra.run.smithi099.stdout: "val": 3708073710 2023-05-31T21:29:20.069 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:29:20.069 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T21:29:20.070 INFO:teuthology.orchestra.run.smithi099.stdout: "key": "f85d7236-2f27-4988-af26-9c66ab280b1b", 2023-05-31T21:29:20.070 INFO:teuthology.orchestra.run.smithi099.stdout: "val": 233395200 2023-05-31T21:29:20.071 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T21:29:20.071 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2023-05-31T21:29:20.071 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:29:20.071 INFO:teuthology.orchestra.run.smithi099.stdout: "master_zonegroup": "a", 2023-05-31T21:29:20.071 INFO:teuthology.orchestra.run.smithi099.stdout: "master_zone": "3d163b1e-09c7-47e2-ae87-add4aace3f38", 2023-05-31T21:29:20.071 INFO:teuthology.orchestra.run.smithi099.stdout: "period_config": { 2023-05-31T21:29:20.072 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket_quota": { 2023-05-31T21:29:20.072 INFO:teuthology.orchestra.run.smithi099.stdout: "enabled": false, 2023-05-31T21:29:20.072 INFO:teuthology.orchestra.run.smithi099.stdout: "check_on_raw": false, 2023-05-31T21:29:20.072 INFO:teuthology.orchestra.run.smithi099.stdout: "max_size": -1, 2023-05-31T21:29:20.072 INFO:teuthology.orchestra.run.smithi099.stdout: "max_size_kb": 0, 2023-05-31T21:29:20.072 INFO:teuthology.orchestra.run.smithi099.stdout: "max_objects": -1 2023-05-31T21:29:20.072 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:29:20.073 INFO:teuthology.orchestra.run.smithi099.stdout: "user_quota": { 2023-05-31T21:29:20.073 INFO:teuthology.orchestra.run.smithi099.stdout: "enabled": false, 2023-05-31T21:29:20.073 INFO:teuthology.orchestra.run.smithi099.stdout: "check_on_raw": false, 2023-05-31T21:29:20.073 INFO:teuthology.orchestra.run.smithi099.stdout: "max_size": -1, 2023-05-31T21:29:20.073 INFO:teuthology.orchestra.run.smithi099.stdout: "max_size_kb": 0, 2023-05-31T21:29:20.073 INFO:teuthology.orchestra.run.smithi099.stdout: "max_objects": -1 2023-05-31T21:29:20.073 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:29:20.074 INFO:teuthology.orchestra.run.smithi099.stdout: "user_ratelimit": { 2023-05-31T21:29:20.074 INFO:teuthology.orchestra.run.smithi099.stdout: "max_read_ops": 0, 2023-05-31T21:29:20.074 INFO:teuthology.orchestra.run.smithi099.stdout: "max_write_ops": 0, 2023-05-31T21:29:20.074 INFO:teuthology.orchestra.run.smithi099.stdout: "max_read_bytes": 0, 2023-05-31T21:29:20.074 INFO:teuthology.orchestra.run.smithi099.stdout: "max_write_bytes": 0, 2023-05-31T21:29:20.074 INFO:teuthology.orchestra.run.smithi099.stdout: "enabled": false 2023-05-31T21:29:20.075 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:29:20.075 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket_ratelimit": { 2023-05-31T21:29:20.075 INFO:teuthology.orchestra.run.smithi099.stdout: "max_read_ops": 0, 2023-05-31T21:29:20.075 INFO:teuthology.orchestra.run.smithi099.stdout: "max_write_ops": 0, 2023-05-31T21:29:20.075 INFO:teuthology.orchestra.run.smithi099.stdout: "max_read_bytes": 0, 2023-05-31T21:29:20.075 INFO:teuthology.orchestra.run.smithi099.stdout: "max_write_bytes": 0, 2023-05-31T21:29:20.075 INFO:teuthology.orchestra.run.smithi099.stdout: "enabled": false 2023-05-31T21:29:20.076 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:29:20.076 INFO:teuthology.orchestra.run.smithi099.stdout: "anonymous_ratelimit": { 2023-05-31T21:29:20.076 INFO:teuthology.orchestra.run.smithi099.stdout: "max_read_ops": 0, 2023-05-31T21:29:20.076 INFO:teuthology.orchestra.run.smithi099.stdout: "max_write_ops": 0, 2023-05-31T21:29:20.076 INFO:teuthology.orchestra.run.smithi099.stdout: "max_read_bytes": 0, 2023-05-31T21:29:20.076 INFO:teuthology.orchestra.run.smithi099.stdout: "max_write_bytes": 0, 2023-05-31T21:29:20.076 INFO:teuthology.orchestra.run.smithi099.stdout: "enabled": false 2023-05-31T21:29:20.077 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T21:29:20.077 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:29:20.077 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_id": "aaa33de6-515b-4cac-8f1b-3a1a30452117", 2023-05-31T21:29:20.077 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_name": "test-realm", 2023-05-31T21:29:20.077 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2023-05-31T21:29:20.077 INFO:teuthology.orchestra.run.smithi099.stdout:} 2023-05-31T21:29:20.078 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'policy', 'get', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-05-31T21:29:20.078 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'policy', 'get', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-05-31T21:29:20.079 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync policy get --cluster c1 --debug-rgw 0 --debug-ms 0 2023-05-31T21:29:20.125 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2023-05-31T21:29:20.126 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2023-05-31T21:29:20.216 INFO:teuthology.orchestra.run.smithi099.stdout:{"groups":[{"id":"sync-group","data_flow":{"symmetrical":[{"id":"sync-flow1","zones":["a1","a2"]}]},"pipes":[{"id":"sync-pipe","source":{"bucket":"*","zones":["a1","a2"]},"dest":{"bucket":"*","zones":["a1","a2"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":""}}],"status":"allowed"}]} 2023-05-31T21:29:20.217 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'group', 'create', '--group-id', 'sync-bucket', '--status', 'allowed', '--bucket', 'pklxkd-72', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-05-31T21:29:20.217 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'group', 'create', '--group-id', 'sync-bucket', '--status', 'allowed', '--bucket', 'pklxkd-72', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-05-31T21:29:20.218 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync group create --group-id sync-bucket --status allowed --bucket pklxkd-72 --cluster c1 --debug-rgw 0 --debug-ms 0 2023-05-31T21:29:20.276 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2023-05-31T21:29:20.276 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2023-05-31T21:29:20.419 INFO:teuthology.orchestra.run.smithi099.stdout:{"groups":[{"id":"sync-bucket","data_flow":{},"pipes":[],"status":"allowed"}]} 2023-05-31T21:29:20.419 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'group', 'flow', 'create', '--group-id', 'sync-bucket', '--flow-id', 'sync-flowA', '--flow-type', 'symmetrical', '--zones=a1,a2', '--bucket', 'pklxkd-72', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-05-31T21:29:20.420 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'group', 'flow', 'create', '--group-id', 'sync-bucket', '--flow-id', 'sync-flowA', '--flow-type', 'symmetrical', '--zones=a1,a2', '--bucket', 'pklxkd-72', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-05-31T21:29:20.420 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync group flow create --group-id sync-bucket --flow-id sync-flowA --flow-type symmetrical --zones=a1,a2 --bucket pklxkd-72 --cluster c1 --debug-rgw 0 --debug-ms 0 2023-05-31T21:29:20.488 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2023-05-31T21:29:20.489 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2023-05-31T21:29:20.624 INFO:teuthology.orchestra.run.smithi099.stdout:{"groups":[{"id":"sync-bucket","data_flow":{"symmetrical":[{"id":"sync-flowA","zones":["a1","a2"]}]},"pipes":[],"status":"allowed"}]} 2023-05-31T21:29:20.625 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'group', 'pipe', 'create', '--group-id', 'sync-bucket', '--pipe-id', 'sync-pipe-pklxkd-72', '--source-zones=a1', '--dest-zones=a2', '--bucket=pklxkd-72', '--source-bucket=pklxkd-72', '--dest-bucket=*', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-05-31T21:29:20.625 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'group', 'pipe', 'create', '--group-id', 'sync-bucket', '--pipe-id', 'sync-pipe-pklxkd-72', '--source-zones=a1', '--dest-zones=a2', '--bucket=pklxkd-72', '--source-bucket=pklxkd-72', '--dest-bucket=*', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-05-31T21:29:20.626 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync group pipe create --group-id sync-bucket --pipe-id sync-pipe-pklxkd-72 --source-zones=a1 --dest-zones=a2 --bucket=pklxkd-72 --source-bucket=pklxkd-72 '--dest-bucket=*' --cluster c1 --debug-rgw 0 --debug-ms 0 2023-05-31T21:29:20.690 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2023-05-31T21:29:20.691 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2023-05-31T21:29:20.798 INFO:teuthology.orchestra.run.smithi099.stdout:{"groups":[{"id":"sync-bucket","data_flow":{"symmetrical":[{"id":"sync-flowA","zones":["a1","a2"]}]},"pipes":[{"id":"sync-pipe-pklxkd-72","source":{"bucket":"pklxkd-72","zones":["a1"]},"dest":{"bucket":"*","zones":["a2"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":"rgw-multisite-test-user"}}],"status":"allowed"}]} 2023-05-31T21:29:20.799 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'group', 'pipe', 'create', '--group-id', 'sync-bucket', '--pipe-id', 'sync-pipe-pklxkd-73', '--source-zones=a1', '--dest-zones=a2', '--bucket=pklxkd-72', '--source-bucket=pklxkd-73', '--dest-bucket=*', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-05-31T21:29:20.799 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'group', 'pipe', 'create', '--group-id', 'sync-bucket', '--pipe-id', 'sync-pipe-pklxkd-73', '--source-zones=a1', '--dest-zones=a2', '--bucket=pklxkd-72', '--source-bucket=pklxkd-73', '--dest-bucket=*', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-05-31T21:29:20.800 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync group pipe create --group-id sync-bucket --pipe-id sync-pipe-pklxkd-73 --source-zones=a1 --dest-zones=a2 --bucket=pklxkd-72 --source-bucket=pklxkd-73 '--dest-bucket=*' --cluster c1 --debug-rgw 0 --debug-ms 0 2023-05-31T21:29:20.865 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2023-05-31T21:29:20.865 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2023-05-31T21:29:20.995 INFO:teuthology.orchestra.run.smithi099.stdout:{"groups":[{"id":"sync-bucket","data_flow":{"symmetrical":[{"id":"sync-flowA","zones":["a1","a2"]}]},"pipes":[{"id":"sync-pipe-pklxkd-72","source":{"bucket":"pklxkd-72","zones":["a1"]},"dest":{"bucket":"*","zones":["a2"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":"rgw-multisite-test-user"}},{"id":"sync-pipe-pklxkd-73","source":{"bucket":"pklxkd-73","zones":["a1"]},"dest":{"bucket":"*","zones":["a2"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":"rgw-multisite-test-user"}}],"status":"allowed"}]} 2023-05-31T21:29:20.996 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'group', 'modify', '--group-id', 'sync-bucket', '--status', 'enabled', '--bucket', 'pklxkd-72', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-05-31T21:29:20.996 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'group', 'modify', '--group-id', 'sync-bucket', '--status', 'enabled', '--bucket', 'pklxkd-72', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-05-31T21:29:20.997 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync group modify --group-id sync-bucket --status enabled --bucket pklxkd-72 --cluster c1 --debug-rgw 0 --debug-ms 0 2023-05-31T21:29:21.052 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2023-05-31T21:29:21.053 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2023-05-31T21:29:21.144 INFO:teuthology.orchestra.run.smithi099.stdout:{"groups":[{"id":"sync-bucket","data_flow":{"symmetrical":[{"id":"sync-flowA","zones":["a1","a2"]}]},"pipes":[{"id":"sync-pipe-pklxkd-72","source":{"bucket":"pklxkd-72","zones":["a1"]},"dest":{"bucket":"*","zones":["a2"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":"rgw-multisite-test-user"}},{"id":"sync-pipe-pklxkd-73","source":{"bucket":"pklxkd-73","zones":["a1"]},"dest":{"bucket":"*","zones":["a2"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":"rgw-multisite-test-user"}}],"status":"enabled"}]} 2023-05-31T21:29:21.144 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'policy', 'get', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-05-31T21:29:21.145 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'policy', 'get', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-05-31T21:29:21.145 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync policy get --cluster c1 --debug-rgw 0 --debug-ms 0 2023-05-31T21:29:21.193 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2023-05-31T21:29:21.193 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2023-05-31T21:29:21.252 INFO:teuthology.orchestra.run.smithi099.stdout:{"groups":[{"id":"sync-group","data_flow":{"symmetrical":[{"id":"sync-flow1","zones":["a1","a2"]}]},"pipes":[{"id":"sync-pipe","source":{"bucket":"*","zones":["a1","a2"]},"dest":{"bucket":"*","zones":["a1","a2"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":""}}],"status":"allowed"}]} 2023-05-31T21:29:21.252 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-31T21:29:21.253 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-31T21:29:21.253 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-05-31T21:29:21.301 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2023-05-31T21:29:21.302 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2023-05-31T21:29:21.335 INFO:teuthology.orchestra.run.smithi099.stderr:No --period given, using current period=2e6123d2-d695-4350-9a01-61a455bc782b 2023-05-31T21:29:21.350 INFO:teuthology.orchestra.run.smithi099.stdout:[ 2023-05-31T21:29:21.351 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T21:29:21.351 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1685566501.757183_260.1", 2023-05-31T21:29:21.351 INFO:teuthology.orchestra.run.smithi099.stdout: "last_update": "2023-05-31T20:55:01.757183Z" 2023-05-31T21:29:21.351 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:29:21.351 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T21:29:21.351 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1685566501.735717_251.1", 2023-05-31T21:29:21.352 INFO:teuthology.orchestra.run.smithi099.stdout: "last_update": "2023-05-31T20:55:01.735717Z" 2023-05-31T21:29:21.352 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:29:21.352 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T21:29:21.352 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1685562970.274572_93.1", 2023-05-31T21:29:21.352 INFO:teuthology.orchestra.run.smithi099.stdout: "last_update": "2023-05-31T19:56:10.274572Z" 2023-05-31T21:29:21.352 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:29:21.353 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T21:29:21.353 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1685568561.112279_1036.1", 2023-05-31T21:29:21.353 INFO:teuthology.orchestra.run.smithi099.stdout: "last_update": "2023-05-31T21:29:21.112279Z" 2023-05-31T21:29:21.353 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T21:29:21.353 INFO:teuthology.orchestra.run.smithi099.stdout:] 2023-05-31T21:29:21.354 DEBUG:rgw_multi.tests:master meta markers={0: '1_1685566501.757183_260.1', 1: '1_1685566501.735717_251.1', 2: '1_1685562970.274572_93.1', 3: '1_1685568561.112279_1036.1'} 2023-05-31T21:29:21.355 INFO:rgw_multi.tests:starting meta checkpoint for zone=a2 2023-05-31T21:29:21.355 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-31T21:29:21.355 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-31T21:29:21.356 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-05-31T21:29:21.412 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2023-05-31T21:29:21.413 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2023-05-31T21:29:21.453 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T21:29:21.449+0000 7f8f0cd91780 20 RGW-SYNC:meta: read sync status 2023-05-31T21:29:21.454 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T21:29:21.450+0000 7f8f0cd91780 20 RGW-SYNC:meta: read sync status 2023-05-31T21:29:21.475 INFO:teuthology.orchestra.run.smithi099.stdout:{ 2023-05-31T21:29:21.475 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_status": { 2023-05-31T21:29:21.475 INFO:teuthology.orchestra.run.smithi099.stdout: "info": { 2023-05-31T21:29:21.475 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "sync", 2023-05-31T21:29:21.476 INFO:teuthology.orchestra.run.smithi099.stdout: "num_shards": 4, 2023-05-31T21:29:21.476 INFO:teuthology.orchestra.run.smithi099.stdout: "period": "2e6123d2-d695-4350-9a01-61a455bc782b", 2023-05-31T21:29:21.476 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2023-05-31T21:29:21.476 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:29:21.476 INFO:teuthology.orchestra.run.smithi099.stdout: "markers": [ 2023-05-31T21:29:21.476 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T21:29:21.477 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 0, 2023-05-31T21:29:21.477 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T21:29:21.477 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2023-05-31T21:29:21.477 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1685566501.757183_260.1", 2023-05-31T21:29:21.477 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T21:29:21.477 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 1, 2023-05-31T21:29:21.477 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T21:29:21.478 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2023-05-31T20:55:01.757183Z", 2023-05-31T21:29:21.478 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2023-05-31T21:29:21.478 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T21:29:21.478 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:29:21.478 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T21:29:21.478 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 1, 2023-05-31T21:29:21.478 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T21:29:21.479 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2023-05-31T21:29:21.479 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1685566501.735717_251.1", 2023-05-31T21:29:21.479 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T21:29:21.479 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2023-05-31T21:29:21.479 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T21:29:21.479 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2023-05-31T20:55:01.735717Z", 2023-05-31T21:29:21.480 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2023-05-31T21:29:21.480 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T21:29:21.480 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:29:21.480 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T21:29:21.481 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 2, 2023-05-31T21:29:21.481 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T21:29:21.481 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2023-05-31T21:29:21.481 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1685562970.274572_93.1", 2023-05-31T21:29:21.481 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T21:29:21.481 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2023-05-31T21:29:21.481 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T21:29:21.482 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2023-05-31T19:56:10.274572Z", 2023-05-31T21:29:21.482 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2023-05-31T21:29:21.482 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T21:29:21.482 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:29:21.482 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T21:29:21.482 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 3, 2023-05-31T21:29:21.482 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T21:29:21.483 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2023-05-31T21:29:21.483 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1685568489.974220_1014.1", 2023-05-31T21:29:21.483 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T21:29:21.483 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2023-05-31T21:29:21.483 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T21:29:21.483 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2023-05-31T21:28:09.974220Z", 2023-05-31T21:29:21.484 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2023-05-31T21:29:21.484 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T21:29:21.484 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T21:29:21.484 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2023-05-31T21:29:21.484 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:29:21.484 INFO:teuthology.orchestra.run.smithi099.stdout: "full_sync": { 2023-05-31T21:29:21.484 INFO:teuthology.orchestra.run.smithi099.stdout: "total": 1, 2023-05-31T21:29:21.485 INFO:teuthology.orchestra.run.smithi099.stdout: "complete": 1 2023-05-31T21:29:21.485 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:29:21.485 INFO:teuthology.orchestra.run.smithi099.stdout: "current_time": "2023-05-31T21:29:21Z" 2023-05-31T21:29:21.485 INFO:teuthology.orchestra.run.smithi099.stdout:} 2023-05-31T21:29:21.486 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "2e6123d2-d695-4350-9a01-61a455bc782b", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1685566501.757183_260.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-05-31T20:55:01.757183Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1685566501.735717_251.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-05-31T20:55:01.735717Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1685562970.274572_93.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-05-31T19:56:10.274572Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1685568489.974220_1014.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-05-31T21:28:09.974220Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-05-31T21:29:21Z" } 2023-05-31T21:29:21.487 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1685566501.757183_260.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-05-31T20:55:01.757183Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1685566501.735717_251.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-05-31T20:55:01.735717Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1685562970.274572_93.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-05-31T19:56:10.274572Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1685568489.974220_1014.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-05-31T21:28:09.974220Z', 'realm_epoch': 2}}] 2023-05-31T21:29:21.487 DEBUG:rgw_multi.tests:log_status={0: '1_1685566501.757183_260.1', 1: '1_1685566501.735717_251.1', 2: '1_1685562970.274572_93.1', 3: '1_1685568561.112279_1036.1'} 2023-05-31T21:29:21.488 DEBUG:rgw_multi.tests:sync_status={0: '1_1685566501.757183_260.1', 1: '1_1685566501.735717_251.1', 2: '1_1685562970.274572_93.1', 3: '1_1685568489.974220_1014.1'} 2023-05-31T21:29:21.488 WARNING:rgw_multi.tests:zone a2 behind master: shard=3 master=1_1685568561.112279_1036.1 target=1_1685568489.974220_1014.1 2023-05-31T21:29:26.489 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-31T21:29:26.489 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-31T21:29:26.490 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-05-31T21:29:26.547 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2023-05-31T21:29:26.548 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2023-05-31T21:29:26.581 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T21:29:26.577+0000 7f3d9e706780 20 RGW-SYNC:meta: read sync status 2023-05-31T21:29:26.582 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T21:29:26.578+0000 7f3d9e706780 20 RGW-SYNC:meta: read sync status 2023-05-31T21:29:26.596 INFO:teuthology.orchestra.run.smithi099.stdout:{ 2023-05-31T21:29:26.596 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_status": { 2023-05-31T21:29:26.596 INFO:teuthology.orchestra.run.smithi099.stdout: "info": { 2023-05-31T21:29:26.596 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "sync", 2023-05-31T21:29:26.597 INFO:teuthology.orchestra.run.smithi099.stdout: "num_shards": 4, 2023-05-31T21:29:26.597 INFO:teuthology.orchestra.run.smithi099.stdout: "period": "2e6123d2-d695-4350-9a01-61a455bc782b", 2023-05-31T21:29:26.597 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2023-05-31T21:29:26.597 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:29:26.597 INFO:teuthology.orchestra.run.smithi099.stdout: "markers": [ 2023-05-31T21:29:26.597 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T21:29:26.598 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 0, 2023-05-31T21:29:26.598 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T21:29:26.598 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2023-05-31T21:29:26.598 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1685566501.757183_260.1", 2023-05-31T21:29:26.598 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T21:29:26.598 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 1, 2023-05-31T21:29:26.598 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T21:29:26.599 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2023-05-31T20:55:01.757183Z", 2023-05-31T21:29:26.599 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2023-05-31T21:29:26.599 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T21:29:26.599 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:29:26.599 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T21:29:26.599 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 1, 2023-05-31T21:29:26.599 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T21:29:26.600 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2023-05-31T21:29:26.600 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1685566501.735717_251.1", 2023-05-31T21:29:26.600 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T21:29:26.600 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2023-05-31T21:29:26.600 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T21:29:26.600 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2023-05-31T20:55:01.735717Z", 2023-05-31T21:29:26.601 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2023-05-31T21:29:26.601 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T21:29:26.601 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:29:26.601 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T21:29:26.601 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 2, 2023-05-31T21:29:26.601 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T21:29:26.601 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2023-05-31T21:29:26.602 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1685562970.274572_93.1", 2023-05-31T21:29:26.602 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T21:29:26.602 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2023-05-31T21:29:26.602 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T21:29:26.602 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2023-05-31T19:56:10.274572Z", 2023-05-31T21:29:26.602 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2023-05-31T21:29:26.602 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T21:29:26.603 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:29:26.603 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T21:29:26.603 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 3, 2023-05-31T21:29:26.603 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T21:29:26.603 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2023-05-31T21:29:26.603 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1685568489.974220_1014.1", 2023-05-31T21:29:26.603 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T21:29:26.604 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2023-05-31T21:29:26.604 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T21:29:26.604 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2023-05-31T21:28:09.974220Z", 2023-05-31T21:29:26.604 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2023-05-31T21:29:26.604 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T21:29:26.604 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T21:29:26.604 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2023-05-31T21:29:26.605 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:29:26.605 INFO:teuthology.orchestra.run.smithi099.stdout: "full_sync": { 2023-05-31T21:29:26.605 INFO:teuthology.orchestra.run.smithi099.stdout: "total": 1, 2023-05-31T21:29:26.605 INFO:teuthology.orchestra.run.smithi099.stdout: "complete": 1 2023-05-31T21:29:26.605 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:29:26.605 INFO:teuthology.orchestra.run.smithi099.stdout: "current_time": "2023-05-31T21:29:26Z" 2023-05-31T21:29:26.605 INFO:teuthology.orchestra.run.smithi099.stdout:} 2023-05-31T21:29:26.606 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "2e6123d2-d695-4350-9a01-61a455bc782b", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1685566501.757183_260.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-05-31T20:55:01.757183Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1685566501.735717_251.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-05-31T20:55:01.735717Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1685562970.274572_93.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-05-31T19:56:10.274572Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1685568489.974220_1014.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-05-31T21:28:09.974220Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-05-31T21:29:26Z" } 2023-05-31T21:29:26.607 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1685566501.757183_260.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-05-31T20:55:01.757183Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1685566501.735717_251.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-05-31T20:55:01.735717Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1685562970.274572_93.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-05-31T19:56:10.274572Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1685568489.974220_1014.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-05-31T21:28:09.974220Z', 'realm_epoch': 2}}] 2023-05-31T21:29:26.607 DEBUG:rgw_multi.tests:log_status={0: '1_1685566501.757183_260.1', 1: '1_1685566501.735717_251.1', 2: '1_1685562970.274572_93.1', 3: '1_1685568561.112279_1036.1'} 2023-05-31T21:29:26.607 DEBUG:rgw_multi.tests:sync_status={0: '1_1685566501.757183_260.1', 1: '1_1685566501.735717_251.1', 2: '1_1685562970.274572_93.1', 3: '1_1685568489.974220_1014.1'} 2023-05-31T21:29:26.608 WARNING:rgw_multi.tests:zone a2 behind master: shard=3 master=1_1685568561.112279_1036.1 target=1_1685568489.974220_1014.1 2023-05-31T21:29:31.609 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-31T21:29:31.610 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-31T21:29:31.610 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-05-31T21:29:31.668 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2023-05-31T21:29:31.668 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2023-05-31T21:29:31.710 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T21:29:31.706+0000 7f66ace9a780 20 RGW-SYNC:meta: read sync status 2023-05-31T21:29:31.711 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T21:29:31.707+0000 7f66ace9a780 20 RGW-SYNC:meta: read sync status 2023-05-31T21:29:31.731 INFO:teuthology.orchestra.run.smithi099.stdout:{ 2023-05-31T21:29:31.732 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_status": { 2023-05-31T21:29:31.732 INFO:teuthology.orchestra.run.smithi099.stdout: "info": { 2023-05-31T21:29:31.732 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "sync", 2023-05-31T21:29:31.733 INFO:teuthology.orchestra.run.smithi099.stdout: "num_shards": 4, 2023-05-31T21:29:31.734 INFO:teuthology.orchestra.run.smithi099.stdout: "period": "2e6123d2-d695-4350-9a01-61a455bc782b", 2023-05-31T21:29:31.734 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2023-05-31T21:29:31.735 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:29:31.735 INFO:teuthology.orchestra.run.smithi099.stdout: "markers": [ 2023-05-31T21:29:31.735 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T21:29:31.736 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 0, 2023-05-31T21:29:31.736 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T21:29:31.736 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2023-05-31T21:29:31.737 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1685566501.757183_260.1", 2023-05-31T21:29:31.737 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T21:29:31.737 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 1, 2023-05-31T21:29:31.738 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T21:29:31.738 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2023-05-31T20:55:01.757183Z", 2023-05-31T21:29:31.738 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2023-05-31T21:29:31.739 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T21:29:31.739 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:29:31.739 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T21:29:31.739 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 1, 2023-05-31T21:29:31.740 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T21:29:31.740 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2023-05-31T21:29:31.740 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1685566501.735717_251.1", 2023-05-31T21:29:31.741 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T21:29:31.741 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2023-05-31T21:29:31.741 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T21:29:31.742 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2023-05-31T20:55:01.735717Z", 2023-05-31T21:29:31.742 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2023-05-31T21:29:31.742 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T21:29:31.742 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:29:31.743 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T21:29:31.743 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 2, 2023-05-31T21:29:31.743 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T21:29:31.744 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2023-05-31T21:29:31.744 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1685562970.274572_93.1", 2023-05-31T21:29:31.744 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T21:29:31.746 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2023-05-31T21:29:31.746 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T21:29:31.746 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2023-05-31T19:56:10.274572Z", 2023-05-31T21:29:31.746 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2023-05-31T21:29:31.747 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T21:29:31.747 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:29:31.747 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T21:29:31.748 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 3, 2023-05-31T21:29:31.748 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T21:29:31.748 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2023-05-31T21:29:31.749 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1685568489.974220_1014.1", 2023-05-31T21:29:31.749 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T21:29:31.749 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2023-05-31T21:29:31.749 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T21:29:31.750 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2023-05-31T21:28:09.974220Z", 2023-05-31T21:29:31.750 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2023-05-31T21:29:31.750 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T21:29:31.751 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T21:29:31.751 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2023-05-31T21:29:31.751 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:29:31.752 INFO:teuthology.orchestra.run.smithi099.stdout: "full_sync": { 2023-05-31T21:29:31.752 INFO:teuthology.orchestra.run.smithi099.stdout: "total": 1, 2023-05-31T21:29:31.752 INFO:teuthology.orchestra.run.smithi099.stdout: "complete": 1 2023-05-31T21:29:31.752 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:29:31.752 INFO:teuthology.orchestra.run.smithi099.stdout: "current_time": "2023-05-31T21:29:31Z" 2023-05-31T21:29:31.753 INFO:teuthology.orchestra.run.smithi099.stdout:} 2023-05-31T21:29:31.756 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "2e6123d2-d695-4350-9a01-61a455bc782b", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1685566501.757183_260.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-05-31T20:55:01.757183Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1685566501.735717_251.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-05-31T20:55:01.735717Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1685562970.274572_93.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-05-31T19:56:10.274572Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1685568489.974220_1014.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-05-31T21:28:09.974220Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-05-31T21:29:31Z" } 2023-05-31T21:29:31.757 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1685566501.757183_260.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-05-31T20:55:01.757183Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1685566501.735717_251.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-05-31T20:55:01.735717Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1685562970.274572_93.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-05-31T19:56:10.274572Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1685568489.974220_1014.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-05-31T21:28:09.974220Z', 'realm_epoch': 2}}] 2023-05-31T21:29:31.758 DEBUG:rgw_multi.tests:log_status={0: '1_1685566501.757183_260.1', 1: '1_1685566501.735717_251.1', 2: '1_1685562970.274572_93.1', 3: '1_1685568561.112279_1036.1'} 2023-05-31T21:29:31.759 DEBUG:rgw_multi.tests:sync_status={0: '1_1685566501.757183_260.1', 1: '1_1685566501.735717_251.1', 2: '1_1685562970.274572_93.1', 3: '1_1685568489.974220_1014.1'} 2023-05-31T21:29:31.759 WARNING:rgw_multi.tests:zone a2 behind master: shard=3 master=1_1685568561.112279_1036.1 target=1_1685568489.974220_1014.1 2023-05-31T21:29:36.759 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-31T21:29:36.760 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-31T21:29:36.761 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-05-31T21:29:36.819 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2023-05-31T21:29:36.820 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2023-05-31T21:29:36.868 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T21:29:36.864+0000 7f2380a93780 20 RGW-SYNC:meta: read sync status 2023-05-31T21:29:36.869 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T21:29:36.865+0000 7f2380a93780 20 RGW-SYNC:meta: read sync status 2023-05-31T21:29:36.886 INFO:teuthology.orchestra.run.smithi099.stdout:{ 2023-05-31T21:29:36.886 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_status": { 2023-05-31T21:29:36.886 INFO:teuthology.orchestra.run.smithi099.stdout: "info": { 2023-05-31T21:29:36.886 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "sync", 2023-05-31T21:29:36.887 INFO:teuthology.orchestra.run.smithi099.stdout: "num_shards": 4, 2023-05-31T21:29:36.887 INFO:teuthology.orchestra.run.smithi099.stdout: "period": "2e6123d2-d695-4350-9a01-61a455bc782b", 2023-05-31T21:29:36.887 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2023-05-31T21:29:36.887 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:29:36.888 INFO:teuthology.orchestra.run.smithi099.stdout: "markers": [ 2023-05-31T21:29:36.888 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T21:29:36.888 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 0, 2023-05-31T21:29:36.888 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T21:29:36.888 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2023-05-31T21:29:36.889 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1685566501.757183_260.1", 2023-05-31T21:29:36.889 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T21:29:36.889 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 1, 2023-05-31T21:29:36.889 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T21:29:36.889 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2023-05-31T20:55:01.757183Z", 2023-05-31T21:29:36.889 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2023-05-31T21:29:36.889 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T21:29:36.890 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:29:36.890 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T21:29:36.890 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 1, 2023-05-31T21:29:36.890 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T21:29:36.890 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2023-05-31T21:29:36.890 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1685566501.735717_251.1", 2023-05-31T21:29:36.891 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T21:29:36.891 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2023-05-31T21:29:36.891 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T21:29:36.891 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2023-05-31T20:55:01.735717Z", 2023-05-31T21:29:36.891 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2023-05-31T21:29:36.891 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T21:29:36.891 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:29:36.892 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T21:29:36.892 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 2, 2023-05-31T21:29:36.893 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T21:29:36.893 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2023-05-31T21:29:36.893 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1685562970.274572_93.1", 2023-05-31T21:29:36.893 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T21:29:36.893 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2023-05-31T21:29:36.894 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T21:29:36.894 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2023-05-31T19:56:10.274572Z", 2023-05-31T21:29:36.894 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2023-05-31T21:29:36.894 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T21:29:36.894 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:29:36.894 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T21:29:36.894 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 3, 2023-05-31T21:29:36.895 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T21:29:36.895 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2023-05-31T21:29:36.895 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1685568489.974220_1014.1", 2023-05-31T21:29:36.895 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T21:29:36.895 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2023-05-31T21:29:36.896 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T21:29:36.896 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2023-05-31T21:28:09.974220Z", 2023-05-31T21:29:36.896 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2023-05-31T21:29:36.896 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T21:29:36.896 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T21:29:36.897 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2023-05-31T21:29:36.897 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:29:36.897 INFO:teuthology.orchestra.run.smithi099.stdout: "full_sync": { 2023-05-31T21:29:36.897 INFO:teuthology.orchestra.run.smithi099.stdout: "total": 1, 2023-05-31T21:29:36.897 INFO:teuthology.orchestra.run.smithi099.stdout: "complete": 1 2023-05-31T21:29:36.897 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:29:36.898 INFO:teuthology.orchestra.run.smithi099.stdout: "current_time": "2023-05-31T21:29:36Z" 2023-05-31T21:29:36.898 INFO:teuthology.orchestra.run.smithi099.stdout:} 2023-05-31T21:29:36.899 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "2e6123d2-d695-4350-9a01-61a455bc782b", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1685566501.757183_260.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-05-31T20:55:01.757183Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1685566501.735717_251.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-05-31T20:55:01.735717Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1685562970.274572_93.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-05-31T19:56:10.274572Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1685568489.974220_1014.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-05-31T21:28:09.974220Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-05-31T21:29:36Z" } 2023-05-31T21:29:36.899 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1685566501.757183_260.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-05-31T20:55:01.757183Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1685566501.735717_251.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-05-31T20:55:01.735717Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1685562970.274572_93.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-05-31T19:56:10.274572Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1685568489.974220_1014.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-05-31T21:28:09.974220Z', 'realm_epoch': 2}}] 2023-05-31T21:29:36.900 DEBUG:rgw_multi.tests:log_status={0: '1_1685566501.757183_260.1', 1: '1_1685566501.735717_251.1', 2: '1_1685562970.274572_93.1', 3: '1_1685568561.112279_1036.1'} 2023-05-31T21:29:36.900 DEBUG:rgw_multi.tests:sync_status={0: '1_1685566501.757183_260.1', 1: '1_1685566501.735717_251.1', 2: '1_1685562970.274572_93.1', 3: '1_1685568489.974220_1014.1'} 2023-05-31T21:29:36.900 WARNING:rgw_multi.tests:zone a2 behind master: shard=3 master=1_1685568561.112279_1036.1 target=1_1685568489.974220_1014.1 2023-05-31T21:29:41.902 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-31T21:29:41.902 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-31T21:29:41.903 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-05-31T21:29:41.960 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2023-05-31T21:29:41.960 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2023-05-31T21:29:41.995 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T21:29:41.992+0000 7f5637efe780 20 RGW-SYNC:meta: read sync status 2023-05-31T21:29:41.996 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T21:29:41.993+0000 7f5637efe780 20 RGW-SYNC:meta: read sync status 2023-05-31T21:29:42.012 INFO:teuthology.orchestra.run.smithi099.stdout:{ 2023-05-31T21:29:42.012 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_status": { 2023-05-31T21:29:42.012 INFO:teuthology.orchestra.run.smithi099.stdout: "info": { 2023-05-31T21:29:42.012 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "sync", 2023-05-31T21:29:42.013 INFO:teuthology.orchestra.run.smithi099.stdout: "num_shards": 4, 2023-05-31T21:29:42.013 INFO:teuthology.orchestra.run.smithi099.stdout: "period": "2e6123d2-d695-4350-9a01-61a455bc782b", 2023-05-31T21:29:42.013 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2023-05-31T21:29:42.013 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:29:42.013 INFO:teuthology.orchestra.run.smithi099.stdout: "markers": [ 2023-05-31T21:29:42.013 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T21:29:42.014 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 0, 2023-05-31T21:29:42.014 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T21:29:42.014 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2023-05-31T21:29:42.014 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1685566501.757183_260.1", 2023-05-31T21:29:42.014 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T21:29:42.014 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 1, 2023-05-31T21:29:42.014 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T21:29:42.015 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2023-05-31T20:55:01.757183Z", 2023-05-31T21:29:42.015 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2023-05-31T21:29:42.015 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T21:29:42.015 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:29:42.015 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T21:29:42.015 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 1, 2023-05-31T21:29:42.016 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T21:29:42.016 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2023-05-31T21:29:42.016 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1685566501.735717_251.1", 2023-05-31T21:29:42.016 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T21:29:42.016 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2023-05-31T21:29:42.016 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T21:29:42.017 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2023-05-31T20:55:01.735717Z", 2023-05-31T21:29:42.017 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2023-05-31T21:29:42.017 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T21:29:42.017 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:29:42.017 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T21:29:42.017 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 2, 2023-05-31T21:29:42.017 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T21:29:42.018 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2023-05-31T21:29:42.018 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1685562970.274572_93.1", 2023-05-31T21:29:42.018 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T21:29:42.018 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2023-05-31T21:29:42.018 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T21:29:42.018 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2023-05-31T19:56:10.274572Z", 2023-05-31T21:29:42.019 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2023-05-31T21:29:42.019 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T21:29:42.019 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:29:42.019 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T21:29:42.019 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 3, 2023-05-31T21:29:42.019 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T21:29:42.020 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2023-05-31T21:29:42.020 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1685568489.974220_1014.1", 2023-05-31T21:29:42.020 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T21:29:42.020 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2023-05-31T21:29:42.020 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T21:29:42.020 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2023-05-31T21:28:09.974220Z", 2023-05-31T21:29:42.021 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2023-05-31T21:29:42.021 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T21:29:42.021 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T21:29:42.021 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2023-05-31T21:29:42.021 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:29:42.021 INFO:teuthology.orchestra.run.smithi099.stdout: "full_sync": { 2023-05-31T21:29:42.022 INFO:teuthology.orchestra.run.smithi099.stdout: "total": 1, 2023-05-31T21:29:42.022 INFO:teuthology.orchestra.run.smithi099.stdout: "complete": 1 2023-05-31T21:29:42.022 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:29:42.022 INFO:teuthology.orchestra.run.smithi099.stdout: "current_time": "2023-05-31T21:29:41Z" 2023-05-31T21:29:42.022 INFO:teuthology.orchestra.run.smithi099.stdout:} 2023-05-31T21:29:42.023 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "2e6123d2-d695-4350-9a01-61a455bc782b", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1685566501.757183_260.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-05-31T20:55:01.757183Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1685566501.735717_251.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-05-31T20:55:01.735717Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1685562970.274572_93.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-05-31T19:56:10.274572Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1685568489.974220_1014.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-05-31T21:28:09.974220Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-05-31T21:29:41Z" } 2023-05-31T21:29:42.024 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1685566501.757183_260.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-05-31T20:55:01.757183Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1685566501.735717_251.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-05-31T20:55:01.735717Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1685562970.274572_93.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-05-31T19:56:10.274572Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1685568489.974220_1014.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-05-31T21:28:09.974220Z', 'realm_epoch': 2}}] 2023-05-31T21:29:42.024 DEBUG:rgw_multi.tests:log_status={0: '1_1685566501.757183_260.1', 1: '1_1685566501.735717_251.1', 2: '1_1685562970.274572_93.1', 3: '1_1685568561.112279_1036.1'} 2023-05-31T21:29:42.025 DEBUG:rgw_multi.tests:sync_status={0: '1_1685566501.757183_260.1', 1: '1_1685566501.735717_251.1', 2: '1_1685562970.274572_93.1', 3: '1_1685568489.974220_1014.1'} 2023-05-31T21:29:42.025 WARNING:rgw_multi.tests:zone a2 behind master: shard=3 master=1_1685568561.112279_1036.1 target=1_1685568489.974220_1014.1 2023-05-31T21:29:47.026 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-31T21:29:47.027 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-31T21:29:47.027 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-05-31T21:29:47.085 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2023-05-31T21:29:47.085 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2023-05-31T21:29:47.135 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T21:29:47.132+0000 7f5934670780 20 RGW-SYNC:meta: read sync status 2023-05-31T21:29:47.136 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T21:29:47.133+0000 7f5934670780 20 RGW-SYNC:meta: read sync status 2023-05-31T21:29:47.158 INFO:teuthology.orchestra.run.smithi099.stdout:{ 2023-05-31T21:29:47.158 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_status": { 2023-05-31T21:29:47.158 INFO:teuthology.orchestra.run.smithi099.stdout: "info": { 2023-05-31T21:29:47.158 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "sync", 2023-05-31T21:29:47.158 INFO:teuthology.orchestra.run.smithi099.stdout: "num_shards": 4, 2023-05-31T21:29:47.158 INFO:teuthology.orchestra.run.smithi099.stdout: "period": "2e6123d2-d695-4350-9a01-61a455bc782b", 2023-05-31T21:29:47.159 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2023-05-31T21:29:47.159 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:29:47.159 INFO:teuthology.orchestra.run.smithi099.stdout: "markers": [ 2023-05-31T21:29:47.159 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T21:29:47.159 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 0, 2023-05-31T21:29:47.159 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T21:29:47.160 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2023-05-31T21:29:47.160 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1685566501.757183_260.1", 2023-05-31T21:29:47.160 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T21:29:47.160 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 1, 2023-05-31T21:29:47.160 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T21:29:47.160 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2023-05-31T20:55:01.757183Z", 2023-05-31T21:29:47.160 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2023-05-31T21:29:47.161 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T21:29:47.161 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:29:47.161 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T21:29:47.161 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 1, 2023-05-31T21:29:47.161 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T21:29:47.161 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2023-05-31T21:29:47.161 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1685566501.735717_251.1", 2023-05-31T21:29:47.162 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T21:29:47.162 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2023-05-31T21:29:47.162 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T21:29:47.162 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2023-05-31T20:55:01.735717Z", 2023-05-31T21:29:47.162 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2023-05-31T21:29:47.162 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T21:29:47.162 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:29:47.163 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T21:29:47.163 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 2, 2023-05-31T21:29:47.163 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T21:29:47.163 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2023-05-31T21:29:47.163 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1685562970.274572_93.1", 2023-05-31T21:29:47.163 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T21:29:47.163 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2023-05-31T21:29:47.164 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T21:29:47.164 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2023-05-31T19:56:10.274572Z", 2023-05-31T21:29:47.164 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2023-05-31T21:29:47.164 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T21:29:47.164 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:29:47.164 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T21:29:47.165 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 3, 2023-05-31T21:29:47.165 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T21:29:47.165 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2023-05-31T21:29:47.165 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1685568489.974220_1014.1", 2023-05-31T21:29:47.165 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T21:29:47.165 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2023-05-31T21:29:47.165 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T21:29:47.166 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2023-05-31T21:28:09.974220Z", 2023-05-31T21:29:47.166 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2023-05-31T21:29:47.166 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T21:29:47.166 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T21:29:47.166 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2023-05-31T21:29:47.166 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:29:47.166 INFO:teuthology.orchestra.run.smithi099.stdout: "full_sync": { 2023-05-31T21:29:47.167 INFO:teuthology.orchestra.run.smithi099.stdout: "total": 1, 2023-05-31T21:29:47.167 INFO:teuthology.orchestra.run.smithi099.stdout: "complete": 1 2023-05-31T21:29:47.167 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:29:47.167 INFO:teuthology.orchestra.run.smithi099.stdout: "current_time": "2023-05-31T21:29:47Z" 2023-05-31T21:29:47.167 INFO:teuthology.orchestra.run.smithi099.stdout:} 2023-05-31T21:29:47.168 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "2e6123d2-d695-4350-9a01-61a455bc782b", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1685566501.757183_260.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-05-31T20:55:01.757183Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1685566501.735717_251.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-05-31T20:55:01.735717Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1685562970.274572_93.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-05-31T19:56:10.274572Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1685568489.974220_1014.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-05-31T21:28:09.974220Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-05-31T21:29:47Z" } 2023-05-31T21:29:47.168 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1685566501.757183_260.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-05-31T20:55:01.757183Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1685566501.735717_251.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-05-31T20:55:01.735717Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1685562970.274572_93.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-05-31T19:56:10.274572Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1685568489.974220_1014.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-05-31T21:28:09.974220Z', 'realm_epoch': 2}}] 2023-05-31T21:29:47.169 DEBUG:rgw_multi.tests:log_status={0: '1_1685566501.757183_260.1', 1: '1_1685566501.735717_251.1', 2: '1_1685562970.274572_93.1', 3: '1_1685568561.112279_1036.1'} 2023-05-31T21:29:47.169 DEBUG:rgw_multi.tests:sync_status={0: '1_1685566501.757183_260.1', 1: '1_1685566501.735717_251.1', 2: '1_1685562970.274572_93.1', 3: '1_1685568489.974220_1014.1'} 2023-05-31T21:29:47.170 WARNING:rgw_multi.tests:zone a2 behind master: shard=3 master=1_1685568561.112279_1036.1 target=1_1685568489.974220_1014.1 2023-05-31T21:29:52.170 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-31T21:29:52.171 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-31T21:29:52.172 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-05-31T21:29:52.227 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2023-05-31T21:29:52.228 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2023-05-31T21:29:52.275 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T21:29:52.272+0000 7ff01e920780 20 RGW-SYNC:meta: read sync status 2023-05-31T21:29:52.276 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T21:29:52.273+0000 7ff01e920780 20 RGW-SYNC:meta: read sync status 2023-05-31T21:29:52.297 INFO:teuthology.orchestra.run.smithi099.stdout:{ 2023-05-31T21:29:52.297 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_status": { 2023-05-31T21:29:52.297 INFO:teuthology.orchestra.run.smithi099.stdout: "info": { 2023-05-31T21:29:52.297 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "sync", 2023-05-31T21:29:52.297 INFO:teuthology.orchestra.run.smithi099.stdout: "num_shards": 4, 2023-05-31T21:29:52.298 INFO:teuthology.orchestra.run.smithi099.stdout: "period": "2e6123d2-d695-4350-9a01-61a455bc782b", 2023-05-31T21:29:52.298 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2023-05-31T21:29:52.298 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:29:52.298 INFO:teuthology.orchestra.run.smithi099.stdout: "markers": [ 2023-05-31T21:29:52.298 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T21:29:52.298 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 0, 2023-05-31T21:29:52.299 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T21:29:52.299 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2023-05-31T21:29:52.299 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1685566501.757183_260.1", 2023-05-31T21:29:52.299 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T21:29:52.299 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 1, 2023-05-31T21:29:52.299 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T21:29:52.299 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2023-05-31T20:55:01.757183Z", 2023-05-31T21:29:52.300 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2023-05-31T21:29:52.300 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T21:29:52.300 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:29:52.300 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T21:29:52.300 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 1, 2023-05-31T21:29:52.300 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T21:29:52.300 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2023-05-31T21:29:52.301 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1685566501.735717_251.1", 2023-05-31T21:29:52.301 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T21:29:52.301 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2023-05-31T21:29:52.301 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T21:29:52.301 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2023-05-31T20:55:01.735717Z", 2023-05-31T21:29:52.301 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2023-05-31T21:29:52.301 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T21:29:52.302 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:29:52.302 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T21:29:52.302 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 2, 2023-05-31T21:29:52.302 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T21:29:52.302 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2023-05-31T21:29:52.302 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1685562970.274572_93.1", 2023-05-31T21:29:52.303 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T21:29:52.303 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2023-05-31T21:29:52.303 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T21:29:52.303 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2023-05-31T19:56:10.274572Z", 2023-05-31T21:29:52.303 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2023-05-31T21:29:52.303 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T21:29:52.303 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:29:52.304 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T21:29:52.304 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 3, 2023-05-31T21:29:52.304 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T21:29:52.304 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2023-05-31T21:29:52.304 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1685568560.368414_1023.1", 2023-05-31T21:29:52.304 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T21:29:52.305 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2023-05-31T21:29:52.305 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T21:29:52.305 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2023-05-31T21:29:20.368414Z", 2023-05-31T21:29:52.305 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2023-05-31T21:29:52.305 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T21:29:52.305 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T21:29:52.305 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2023-05-31T21:29:52.306 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:29:52.306 INFO:teuthology.orchestra.run.smithi099.stdout: "full_sync": { 2023-05-31T21:29:52.306 INFO:teuthology.orchestra.run.smithi099.stdout: "total": 1, 2023-05-31T21:29:52.306 INFO:teuthology.orchestra.run.smithi099.stdout: "complete": 1 2023-05-31T21:29:52.306 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:29:52.306 INFO:teuthology.orchestra.run.smithi099.stdout: "current_time": "2023-05-31T21:29:52Z" 2023-05-31T21:29:52.306 INFO:teuthology.orchestra.run.smithi099.stdout:} 2023-05-31T21:29:52.307 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "2e6123d2-d695-4350-9a01-61a455bc782b", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1685566501.757183_260.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-05-31T20:55:01.757183Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1685566501.735717_251.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-05-31T20:55:01.735717Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1685562970.274572_93.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-05-31T19:56:10.274572Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1685568560.368414_1023.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-05-31T21:29:20.368414Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-05-31T21:29:52Z" } 2023-05-31T21:29:52.308 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1685566501.757183_260.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-05-31T20:55:01.757183Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1685566501.735717_251.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-05-31T20:55:01.735717Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1685562970.274572_93.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-05-31T19:56:10.274572Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1685568560.368414_1023.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-05-31T21:29:20.368414Z', 'realm_epoch': 2}}] 2023-05-31T21:29:52.308 DEBUG:rgw_multi.tests:log_status={0: '1_1685566501.757183_260.1', 1: '1_1685566501.735717_251.1', 2: '1_1685562970.274572_93.1', 3: '1_1685568561.112279_1036.1'} 2023-05-31T21:29:52.309 DEBUG:rgw_multi.tests:sync_status={0: '1_1685566501.757183_260.1', 1: '1_1685566501.735717_251.1', 2: '1_1685562970.274572_93.1', 3: '1_1685568560.368414_1023.1'} 2023-05-31T21:29:52.309 WARNING:rgw_multi.tests:zone a2 behind master: shard=3 master=1_1685568561.112279_1036.1 target=1_1685568560.368414_1023.1 2023-05-31T21:29:57.310 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-31T21:29:57.311 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-31T21:29:57.311 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-05-31T21:29:57.369 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2023-05-31T21:29:57.369 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2023-05-31T21:29:57.422 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T21:29:57.418+0000 7f93752f2780 20 RGW-SYNC:meta: read sync status 2023-05-31T21:29:57.422 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T21:29:57.419+0000 7f93752f2780 20 RGW-SYNC:meta: read sync status 2023-05-31T21:29:57.445 INFO:teuthology.orchestra.run.smithi099.stdout:{ 2023-05-31T21:29:57.446 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_status": { 2023-05-31T21:29:57.446 INFO:teuthology.orchestra.run.smithi099.stdout: "info": { 2023-05-31T21:29:57.446 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "sync", 2023-05-31T21:29:57.446 INFO:teuthology.orchestra.run.smithi099.stdout: "num_shards": 4, 2023-05-31T21:29:57.447 INFO:teuthology.orchestra.run.smithi099.stdout: "period": "2e6123d2-d695-4350-9a01-61a455bc782b", 2023-05-31T21:29:57.447 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2023-05-31T21:29:57.447 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:29:57.447 INFO:teuthology.orchestra.run.smithi099.stdout: "markers": [ 2023-05-31T21:29:57.447 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T21:29:57.447 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 0, 2023-05-31T21:29:57.448 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T21:29:57.448 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2023-05-31T21:29:57.448 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1685566501.757183_260.1", 2023-05-31T21:29:57.448 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T21:29:57.448 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 1, 2023-05-31T21:29:57.448 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T21:29:57.449 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2023-05-31T20:55:01.757183Z", 2023-05-31T21:29:57.449 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2023-05-31T21:29:57.449 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T21:29:57.449 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:29:57.449 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T21:29:57.449 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 1, 2023-05-31T21:29:57.449 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T21:29:57.450 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2023-05-31T21:29:57.450 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1685566501.735717_251.1", 2023-05-31T21:29:57.450 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T21:29:57.450 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2023-05-31T21:29:57.450 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T21:29:57.450 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2023-05-31T20:55:01.735717Z", 2023-05-31T21:29:57.451 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2023-05-31T21:29:57.451 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T21:29:57.451 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:29:57.451 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T21:29:57.451 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 2, 2023-05-31T21:29:57.451 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T21:29:57.452 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2023-05-31T21:29:57.452 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1685562970.274572_93.1", 2023-05-31T21:29:57.452 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T21:29:57.452 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2023-05-31T21:29:57.452 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T21:29:57.452 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2023-05-31T19:56:10.274572Z", 2023-05-31T21:29:57.453 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2023-05-31T21:29:57.453 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T21:29:57.453 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:29:57.453 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T21:29:57.453 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 3, 2023-05-31T21:29:57.453 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T21:29:57.453 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2023-05-31T21:29:57.454 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1685568561.112279_1036.1", 2023-05-31T21:29:57.454 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T21:29:57.454 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2023-05-31T21:29:57.454 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T21:29:57.454 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2023-05-31T21:29:21.112279Z", 2023-05-31T21:29:57.454 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2023-05-31T21:29:57.455 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T21:29:57.455 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T21:29:57.455 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2023-05-31T21:29:57.455 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:29:57.455 INFO:teuthology.orchestra.run.smithi099.stdout: "full_sync": { 2023-05-31T21:29:57.455 INFO:teuthology.orchestra.run.smithi099.stdout: "total": 1, 2023-05-31T21:29:57.455 INFO:teuthology.orchestra.run.smithi099.stdout: "complete": 1 2023-05-31T21:29:57.456 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:29:57.456 INFO:teuthology.orchestra.run.smithi099.stdout: "current_time": "2023-05-31T21:29:57Z" 2023-05-31T21:29:57.456 INFO:teuthology.orchestra.run.smithi099.stdout:} 2023-05-31T21:29:57.457 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "2e6123d2-d695-4350-9a01-61a455bc782b", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1685566501.757183_260.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-05-31T20:55:01.757183Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1685566501.735717_251.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-05-31T20:55:01.735717Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1685562970.274572_93.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-05-31T19:56:10.274572Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1685568561.112279_1036.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-05-31T21:29:21.112279Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-05-31T21:29:57Z" } 2023-05-31T21:29:57.457 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1685566501.757183_260.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-05-31T20:55:01.757183Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1685566501.735717_251.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-05-31T20:55:01.735717Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1685562970.274572_93.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-05-31T19:56:10.274572Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1685568561.112279_1036.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-05-31T21:29:21.112279Z', 'realm_epoch': 2}}] 2023-05-31T21:29:57.458 DEBUG:rgw_multi.tests:log_status={0: '1_1685566501.757183_260.1', 1: '1_1685566501.735717_251.1', 2: '1_1685562970.274572_93.1', 3: '1_1685568561.112279_1036.1'} 2023-05-31T21:29:57.458 DEBUG:rgw_multi.tests:sync_status={0: '1_1685566501.757183_260.1', 1: '1_1685566501.735717_251.1', 2: '1_1685562970.274572_93.1', 3: '1_1685568561.112279_1036.1'} 2023-05-31T21:29:57.459 INFO:rgw_multi.tests:finish meta checkpoint for zone=a2 2023-05-31T21:29:57.459 DEBUG:boto:path=/pklxkd-72/ 2023-05-31T21:29:57.459 DEBUG:boto:auth_path=/pklxkd-72/ 2023-05-31T21:29:57.460 DEBUG:boto:Method: HEAD 2023-05-31T21:29:57.460 DEBUG:boto:Path: /pklxkd-72/ 2023-05-31T21:29:57.460 DEBUG:boto:Data: 2023-05-31T21:29:57.461 DEBUG:boto:Headers: {} 2023-05-31T21:29:57.461 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T21:29:57.461 DEBUG:boto:Port: 8000 2023-05-31T21:29:57.462 DEBUG:boto:Params: {} 2023-05-31T21:29:57.462 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2023-05-31T21:29:57.463 DEBUG:boto:Token: None 2023-05-31T21:29:57.463 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 21:29:57 GMT /pklxkd-72/ 2023-05-31T21:29:57.464 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:F21Pds4gJbdmbGiZ1XnS+wcZrLw= 2023-05-31T21:29:57.464 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 21:29:57 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:F21Pds4gJbdmbGiZ1XnS+wcZrLw=', 'Content-Length': '0'} 2023-05-31T21:29:57.468 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '2'), ('X-RGW-Bytes-Used', '12'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000083c10bed5eea0c31-006477bc55-2958-a1'), ('Content-Length', '0'), ('Date', 'Wed, 31 May 2023 21:29:57 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:29:57.469 DEBUG:boto:path=/pklxkd-72/obj5 2023-05-31T21:29:57.469 DEBUG:boto:auth_path=/pklxkd-72/obj5 2023-05-31T21:29:57.470 DEBUG:boto:Method: PUT 2023-05-31T21:29:57.470 DEBUG:boto:Path: /pklxkd-72/obj5 2023-05-31T21:29:57.470 DEBUG:boto:Data: 2023-05-31T21:29:57.471 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2023-05-31T21:29:57.471 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T21:29:57.472 DEBUG:boto:Port: 8000 2023-05-31T21:29:57.472 DEBUG:boto:Params: {} 2023-05-31T21:29:57.472 DEBUG:boto:Token: None 2023-05-31T21:29:57.473 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Wed, 31 May 2023 21:29:57 GMT /pklxkd-72/obj5 2023-05-31T21:29:57.473 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:rXaM+mUYb/nryX39GzigpcwXV84= 2023-05-31T21:29:57.474 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Wed, 31 May 2023 21:29:57 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:rXaM+mUYb/nryX39GzigpcwXV84='} 2023-05-31T21:29:57.487 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000815722a9f9ef11b4-006477bc55-2958-a1'), ('Date', 'Wed, 31 May 2023 21:29:57 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:29:57.488 DEBUG:boto:path=/pklxkd-72/ 2023-05-31T21:29:57.488 DEBUG:boto:auth_path=/pklxkd-72/ 2023-05-31T21:29:57.489 DEBUG:boto:Method: HEAD 2023-05-31T21:29:57.489 DEBUG:boto:Path: /pklxkd-72/ 2023-05-31T21:29:57.489 DEBUG:boto:Data: 2023-05-31T21:29:57.490 DEBUG:boto:Headers: {} 2023-05-31T21:29:57.490 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T21:29:57.491 DEBUG:boto:Port: 8000 2023-05-31T21:29:57.491 DEBUG:boto:Params: {} 2023-05-31T21:29:57.491 DEBUG:boto:Token: None 2023-05-31T21:29:57.492 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 21:29:57 GMT /pklxkd-72/ 2023-05-31T21:29:57.492 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:F21Pds4gJbdmbGiZ1XnS+wcZrLw= 2023-05-31T21:29:57.493 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 21:29:57 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:F21Pds4gJbdmbGiZ1XnS+wcZrLw=', 'Content-Length': '0'} 2023-05-31T21:29:57.495 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '3'), ('X-RGW-Bytes-Used', '18'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000e7dba73aec154804-006477bc55-2958-a1'), ('Content-Length', '0'), ('Date', 'Wed, 31 May 2023 21:29:57 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:29:57.495 DEBUG:boto:path=/pklxkd-72/obj6 2023-05-31T21:29:57.496 DEBUG:boto:auth_path=/pklxkd-72/obj6 2023-05-31T21:29:57.496 DEBUG:boto:Method: PUT 2023-05-31T21:29:57.496 DEBUG:boto:Path: /pklxkd-72/obj6 2023-05-31T21:29:57.497 DEBUG:boto:Data: 2023-05-31T21:29:57.497 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2023-05-31T21:29:57.497 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T21:29:57.498 DEBUG:boto:Port: 8000 2023-05-31T21:29:57.498 DEBUG:boto:Params: {} 2023-05-31T21:29:57.498 DEBUG:boto:Token: None 2023-05-31T21:29:57.499 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Wed, 31 May 2023 21:29:57 GMT /pklxkd-72/obj6 2023-05-31T21:29:57.499 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:MPGBl8jm7DzCPiqXtoDDJMIO98s= 2023-05-31T21:29:57.500 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Wed, 31 May 2023 21:29:57 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:MPGBl8jm7DzCPiqXtoDDJMIO98s='} 2023-05-31T21:29:57.510 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000a6f82f98c41b8409-006477bc55-2958-a1'), ('Date', 'Wed, 31 May 2023 21:29:57 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:29:57.510 DEBUG:boto:path=/pklxkd-73/ 2023-05-31T21:29:57.511 DEBUG:boto:auth_path=/pklxkd-73/ 2023-05-31T21:29:57.511 DEBUG:boto:Method: HEAD 2023-05-31T21:29:57.511 DEBUG:boto:Path: /pklxkd-73/ 2023-05-31T21:29:57.512 DEBUG:boto:Data: 2023-05-31T21:29:57.512 DEBUG:boto:Headers: {} 2023-05-31T21:29:57.512 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T21:29:57.513 DEBUG:boto:Port: 8000 2023-05-31T21:29:57.513 DEBUG:boto:Params: {} 2023-05-31T21:29:57.514 DEBUG:boto:Token: None 2023-05-31T21:29:57.514 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 21:29:57 GMT /pklxkd-73/ 2023-05-31T21:29:57.514 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:5+tH0aHlFpzrhDHewn/fGUEnCaY= 2023-05-31T21:29:57.515 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 21:29:57 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:5+tH0aHlFpzrhDHewn/fGUEnCaY=', 'Content-Length': '0'} 2023-05-31T21:29:57.517 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '2'), ('X-RGW-Bytes-Used', '12'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000da9fadc96819c97c-006477bc55-2958-a1'), ('Content-Length', '0'), ('Date', 'Wed, 31 May 2023 21:29:57 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:29:57.517 DEBUG:boto:path=/pklxkd-73/obj7 2023-05-31T21:29:57.518 DEBUG:boto:auth_path=/pklxkd-73/obj7 2023-05-31T21:29:57.518 DEBUG:boto:Method: PUT 2023-05-31T21:29:57.519 DEBUG:boto:Path: /pklxkd-73/obj7 2023-05-31T21:29:57.519 DEBUG:boto:Data: 2023-05-31T21:29:57.519 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2023-05-31T21:29:57.520 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T21:29:57.520 DEBUG:boto:Port: 8000 2023-05-31T21:29:57.520 DEBUG:boto:Params: {} 2023-05-31T21:29:57.521 DEBUG:boto:Token: None 2023-05-31T21:29:57.521 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Wed, 31 May 2023 21:29:57 GMT /pklxkd-73/obj7 2023-05-31T21:29:57.522 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:9M7Z2y8F+6b09LscCSHaB8ticgY= 2023-05-31T21:29:57.522 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Wed, 31 May 2023 21:29:57 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:9M7Z2y8F+6b09LscCSHaB8ticgY='} 2023-05-31T21:29:57.564 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000c32e435997d99d10-006477bc55-2958-a1'), ('Date', 'Wed, 31 May 2023 21:29:57 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:29:57.565 DEBUG:boto:path=/pklxkd-73/ 2023-05-31T21:29:57.565 DEBUG:boto:auth_path=/pklxkd-73/ 2023-05-31T21:29:57.566 DEBUG:boto:Method: HEAD 2023-05-31T21:29:57.566 DEBUG:boto:Path: /pklxkd-73/ 2023-05-31T21:29:57.566 DEBUG:boto:Data: 2023-05-31T21:29:57.567 DEBUG:boto:Headers: {} 2023-05-31T21:29:57.567 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T21:29:57.567 DEBUG:boto:Port: 8000 2023-05-31T21:29:57.568 DEBUG:boto:Params: {} 2023-05-31T21:29:57.568 DEBUG:boto:Token: None 2023-05-31T21:29:57.569 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 21:29:57 GMT /pklxkd-73/ 2023-05-31T21:29:57.569 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:5+tH0aHlFpzrhDHewn/fGUEnCaY= 2023-05-31T21:29:57.569 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 21:29:57 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:5+tH0aHlFpzrhDHewn/fGUEnCaY=', 'Content-Length': '0'} 2023-05-31T21:29:57.571 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '3'), ('X-RGW-Bytes-Used', '18'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000595b708d9f08954d-006477bc55-2958-a1'), ('Content-Length', '0'), ('Date', 'Wed, 31 May 2023 21:29:57 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:29:57.572 DEBUG:boto:path=/pklxkd-73/obj8 2023-05-31T21:29:57.572 DEBUG:boto:auth_path=/pklxkd-73/obj8 2023-05-31T21:29:57.573 DEBUG:boto:Method: PUT 2023-05-31T21:29:57.573 DEBUG:boto:Path: /pklxkd-73/obj8 2023-05-31T21:29:57.574 DEBUG:boto:Data: 2023-05-31T21:29:57.574 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2023-05-31T21:29:57.574 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T21:29:57.575 DEBUG:boto:Port: 8000 2023-05-31T21:29:57.575 DEBUG:boto:Params: {} 2023-05-31T21:29:57.575 DEBUG:boto:Token: None 2023-05-31T21:29:57.576 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Wed, 31 May 2023 21:29:57 GMT /pklxkd-73/obj8 2023-05-31T21:29:57.576 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:LHqecPLTHuDIH4WYPs0xOKo39S8= 2023-05-31T21:29:57.577 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Wed, 31 May 2023 21:29:57 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:LHqecPLTHuDIH4WYPs0xOKo39S8='} 2023-05-31T21:29:57.586 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000048692692bd253a4-006477bc55-2958-a1'), ('Date', 'Wed, 31 May 2023 21:29:57 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:29:57.587 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-31T21:29:57.587 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-31T21:29:57.588 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-05-31T21:29:57.636 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2023-05-31T21:29:57.637 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2023-05-31T21:29:57.674 INFO:teuthology.orchestra.run.smithi099.stderr:No --period given, using current period=2e6123d2-d695-4350-9a01-61a455bc782b 2023-05-31T21:29:57.688 INFO:teuthology.orchestra.run.smithi099.stdout:[ 2023-05-31T21:29:57.689 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T21:29:57.689 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1685566501.757183_260.1", 2023-05-31T21:29:57.689 INFO:teuthology.orchestra.run.smithi099.stdout: "last_update": "2023-05-31T20:55:01.757183Z" 2023-05-31T21:29:57.689 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:29:57.689 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T21:29:57.689 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1685566501.735717_251.1", 2023-05-31T21:29:57.690 INFO:teuthology.orchestra.run.smithi099.stdout: "last_update": "2023-05-31T20:55:01.735717Z" 2023-05-31T21:29:57.690 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:29:57.690 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T21:29:57.690 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1685562970.274572_93.1", 2023-05-31T21:29:57.690 INFO:teuthology.orchestra.run.smithi099.stdout: "last_update": "2023-05-31T19:56:10.274572Z" 2023-05-31T21:29:57.690 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:29:57.691 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T21:29:57.691 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1685568561.112279_1036.1", 2023-05-31T21:29:57.691 INFO:teuthology.orchestra.run.smithi099.stdout: "last_update": "2023-05-31T21:29:21.112279Z" 2023-05-31T21:29:57.691 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T21:29:57.692 INFO:teuthology.orchestra.run.smithi099.stdout:] 2023-05-31T21:29:57.693 DEBUG:rgw_multi.tests:master meta markers={0: '1_1685566501.757183_260.1', 1: '1_1685566501.735717_251.1', 2: '1_1685562970.274572_93.1', 3: '1_1685568561.112279_1036.1'} 2023-05-31T21:29:57.693 INFO:rgw_multi.tests:starting meta checkpoint for zone=a2 2023-05-31T21:29:57.693 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-31T21:29:57.694 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-31T21:29:57.694 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-05-31T21:29:57.743 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2023-05-31T21:29:57.743 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2023-05-31T21:29:57.779 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T21:29:57.776+0000 7fa2bb2e1780 20 RGW-SYNC:meta: read sync status 2023-05-31T21:29:57.780 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T21:29:57.777+0000 7fa2bb2e1780 20 RGW-SYNC:meta: read sync status 2023-05-31T21:29:57.797 INFO:teuthology.orchestra.run.smithi099.stdout:{ 2023-05-31T21:29:57.797 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_status": { 2023-05-31T21:29:57.798 INFO:teuthology.orchestra.run.smithi099.stdout: "info": { 2023-05-31T21:29:57.798 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "sync", 2023-05-31T21:29:57.798 INFO:teuthology.orchestra.run.smithi099.stdout: "num_shards": 4, 2023-05-31T21:29:57.798 INFO:teuthology.orchestra.run.smithi099.stdout: "period": "2e6123d2-d695-4350-9a01-61a455bc782b", 2023-05-31T21:29:57.798 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2023-05-31T21:29:57.798 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:29:57.799 INFO:teuthology.orchestra.run.smithi099.stdout: "markers": [ 2023-05-31T21:29:57.799 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T21:29:57.799 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 0, 2023-05-31T21:29:57.799 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T21:29:57.799 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2023-05-31T21:29:57.799 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1685566501.757183_260.1", 2023-05-31T21:29:57.799 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T21:29:57.800 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 1, 2023-05-31T21:29:57.800 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T21:29:57.800 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2023-05-31T20:55:01.757183Z", 2023-05-31T21:29:57.800 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2023-05-31T21:29:57.800 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T21:29:57.800 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:29:57.801 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T21:29:57.801 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 1, 2023-05-31T21:29:57.801 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T21:29:57.801 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2023-05-31T21:29:57.801 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1685566501.735717_251.1", 2023-05-31T21:29:57.801 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T21:29:57.801 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2023-05-31T21:29:57.802 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T21:29:57.802 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2023-05-31T20:55:01.735717Z", 2023-05-31T21:29:57.802 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2023-05-31T21:29:57.802 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T21:29:57.802 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:29:57.802 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T21:29:57.802 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 2, 2023-05-31T21:29:57.803 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T21:29:57.803 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2023-05-31T21:29:57.803 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1685562970.274572_93.1", 2023-05-31T21:29:57.803 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T21:29:57.803 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2023-05-31T21:29:57.803 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T21:29:57.803 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2023-05-31T19:56:10.274572Z", 2023-05-31T21:29:57.804 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2023-05-31T21:29:57.804 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T21:29:57.804 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:29:57.804 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T21:29:57.804 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 3, 2023-05-31T21:29:57.804 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T21:29:57.805 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2023-05-31T21:29:57.805 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1685568561.112279_1036.1", 2023-05-31T21:29:57.805 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T21:29:57.805 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2023-05-31T21:29:57.805 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T21:29:57.805 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2023-05-31T21:29:21.112279Z", 2023-05-31T21:29:57.805 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2023-05-31T21:29:57.806 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T21:29:57.806 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T21:29:57.806 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2023-05-31T21:29:57.806 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:29:57.806 INFO:teuthology.orchestra.run.smithi099.stdout: "full_sync": { 2023-05-31T21:29:57.806 INFO:teuthology.orchestra.run.smithi099.stdout: "total": 1, 2023-05-31T21:29:57.806 INFO:teuthology.orchestra.run.smithi099.stdout: "complete": 1 2023-05-31T21:29:57.807 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:29:57.807 INFO:teuthology.orchestra.run.smithi099.stdout: "current_time": "2023-05-31T21:29:57Z" 2023-05-31T21:29:57.807 INFO:teuthology.orchestra.run.smithi099.stdout:} 2023-05-31T21:29:57.808 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "2e6123d2-d695-4350-9a01-61a455bc782b", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1685566501.757183_260.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-05-31T20:55:01.757183Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1685566501.735717_251.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-05-31T20:55:01.735717Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1685562970.274572_93.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-05-31T19:56:10.274572Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1685568561.112279_1036.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-05-31T21:29:21.112279Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-05-31T21:29:57Z" } 2023-05-31T21:29:57.809 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1685566501.757183_260.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-05-31T20:55:01.757183Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1685566501.735717_251.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-05-31T20:55:01.735717Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1685562970.274572_93.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-05-31T19:56:10.274572Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1685568561.112279_1036.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-05-31T21:29:21.112279Z', 'realm_epoch': 2}}] 2023-05-31T21:29:57.810 DEBUG:rgw_multi.tests:log_status={0: '1_1685566501.757183_260.1', 1: '1_1685566501.735717_251.1', 2: '1_1685562970.274572_93.1', 3: '1_1685568561.112279_1036.1'} 2023-05-31T21:29:57.811 DEBUG:rgw_multi.tests:sync_status={0: '1_1685566501.757183_260.1', 1: '1_1685566501.735717_251.1', 2: '1_1685562970.274572_93.1', 3: '1_1685568561.112279_1036.1'} 2023-05-31T21:29:57.811 INFO:rgw_multi.tests:finish meta checkpoint for zone=a2 2023-05-31T21:29:57.811 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['datalog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-31T21:29:57.812 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'datalog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-31T21:29:57.812 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 datalog status --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-05-31T21:29:57.860 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2023-05-31T21:29:57.861 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2023-05-31T21:29:57.933 INFO:teuthology.orchestra.run.smithi099.stdout:[ 2023-05-31T21:29:57.933 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T21:29:57.933 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "00000000000000000000:00000000000000045515", 2023-05-31T21:29:57.933 INFO:teuthology.orchestra.run.smithi099.stdout: "last_update": "2023-05-31T21:29:57.506834Z" 2023-05-31T21:29:57.934 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:29:57.934 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T21:29:57.934 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "00000000000000000000:00000000000000031920", 2023-05-31T21:29:57.934 INFO:teuthology.orchestra.run.smithi099.stdout: "last_update": "2023-05-31T21:29:57.583636Z" 2023-05-31T21:29:57.934 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:29:57.934 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T21:29:57.934 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "00000000000000000000:00000000000000051951", 2023-05-31T21:29:57.935 INFO:teuthology.orchestra.run.smithi099.stdout: "last_update": "2023-05-31T21:29:57.526899Z" 2023-05-31T21:29:57.935 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:29:57.935 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T21:29:57.935 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "00000000000000000000:00000000000000044178", 2023-05-31T21:29:57.935 INFO:teuthology.orchestra.run.smithi099.stdout: "last_update": "2023-05-31T21:27:37.215905Z" 2023-05-31T21:29:57.935 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T21:29:57.935 INFO:teuthology.orchestra.run.smithi099.stdout:] 2023-05-31T21:29:57.937 DEBUG:rgw_multi.tests:data markers for zone=a1 markers={0: '00000000000000000000:00000000000000045515', 1: '00000000000000000000:00000000000000031920', 2: '00000000000000000000:00000000000000051951', 3: '00000000000000000000:00000000000000044178'} 2023-05-31T21:29:57.937 INFO:rgw_multi.tests:starting data checkpoint for target_zone=a2 source_zone=a1 2023-05-31T21:29:57.938 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-31T21:29:57.938 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-31T21:29:57.938 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-05-31T21:29:57.989 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2023-05-31T21:29:57.990 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2023-05-31T21:29:58.065 INFO:teuthology.orchestra.run.smithi099.stdout:{ 2023-05-31T21:29:58.065 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_status": { 2023-05-31T21:29:58.065 INFO:teuthology.orchestra.run.smithi099.stdout: "info": { 2023-05-31T21:29:58.066 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "sync", 2023-05-31T21:29:58.066 INFO:teuthology.orchestra.run.smithi099.stdout: "num_shards": 4, 2023-05-31T21:29:58.066 INFO:teuthology.orchestra.run.smithi099.stdout: "instance_id": 452136274422638824 2023-05-31T21:29:58.066 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:29:58.066 INFO:teuthology.orchestra.run.smithi099.stdout: "markers": [ 2023-05-31T21:29:58.066 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T21:29:58.067 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 0, 2023-05-31T21:29:58.067 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T21:29:58.067 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "incremental-sync", 2023-05-31T21:29:58.067 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "00000000000000000000:00000000000000045375", 2023-05-31T21:29:58.067 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T21:29:58.067 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2023-05-31T21:29:58.067 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T21:29:58.068 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2023-05-31T21:28:16.422581Z" 2023-05-31T21:29:58.068 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T21:29:58.068 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:29:58.068 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T21:29:58.068 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 1, 2023-05-31T21:29:58.069 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T21:29:58.069 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "incremental-sync", 2023-05-31T21:29:58.069 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "00000000000000000000:00000000000000031780", 2023-05-31T21:29:58.069 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T21:29:58.069 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2023-05-31T21:29:58.069 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T21:29:58.069 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2023-05-31T21:28:16.501293Z" 2023-05-31T21:29:58.070 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T21:29:58.070 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:29:58.070 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T21:29:58.070 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 2, 2023-05-31T21:29:58.070 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T21:29:58.071 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "incremental-sync", 2023-05-31T21:29:58.071 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "00000000000000000000:00000000000000051671", 2023-05-31T21:29:58.071 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T21:29:58.071 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 55, 2023-05-31T21:29:58.071 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T21:29:58.071 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2023-05-31T21:28:16.476487Z" 2023-05-31T21:29:58.071 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T21:29:58.072 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:29:58.072 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T21:29:58.072 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 3, 2023-05-31T21:29:58.072 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T21:29:58.072 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "incremental-sync", 2023-05-31T21:29:58.072 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "00000000000000000000:00000000000000044178", 2023-05-31T21:29:58.072 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T21:29:58.073 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 9, 2023-05-31T21:29:58.073 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T21:29:58.073 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2023-05-31T21:27:37.215905Z" 2023-05-31T21:29:58.073 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T21:29:58.073 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T21:29:58.073 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2023-05-31T21:29:58.074 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:29:58.074 INFO:teuthology.orchestra.run.smithi099.stdout: "full_sync": { 2023-05-31T21:29:58.074 INFO:teuthology.orchestra.run.smithi099.stdout: "total": 64, 2023-05-31T21:29:58.074 INFO:teuthology.orchestra.run.smithi099.stdout: "complete": 64 2023-05-31T21:29:58.074 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:29:58.075 INFO:teuthology.orchestra.run.smithi099.stdout: "current_time": "2023-05-31T21:29:58Z" 2023-05-31T21:29:58.075 INFO:teuthology.orchestra.run.smithi099.stdout:} 2023-05-31T21:29:58.076 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 452136274422638824 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000045375", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-05-31T21:28:16.422581Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000031780", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-05-31T21:28:16.501293Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000051671", "next_step_marker": "", "total_entries": 55, "pos": 0, "timestamp": "2023-05-31T21:28:16.476487Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000044178", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2023-05-31T21:27:37.215905Z" } } ] }, "full_sync": { "total": 64, "complete": 64 }, "current_time": "2023-05-31T21:29:58Z" } 2023-05-31T21:29:58.076 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000045375', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-05-31T21:28:16.422581Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000031780', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-05-31T21:28:16.501293Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000051671', 'next_step_marker': '', 'total_entries': 55, 'pos': 0, 'timestamp': '2023-05-31T21:28:16.476487Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000044178', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2023-05-31T21:27:37.215905Z'}}] 2023-05-31T21:29:58.077 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000045515', 1: '00000000000000000000:00000000000000031920', 2: '00000000000000000000:00000000000000051951', 3: '00000000000000000000:00000000000000044178'} 2023-05-31T21:29:58.077 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000045375', 1: '00000000000000000000:00000000000000031780', 2: '00000000000000000000:00000000000000051671', 3: '00000000000000000000:00000000000000044178'} 2023-05-31T21:29:58.078 WARNING:rgw_multi.tests:data of zone a2 behind zone a1: shard=0 master=00000000000000000000:00000000000000045515 target=00000000000000000000:00000000000000045375, shard=1 master=00000000000000000000:00000000000000031920 target=00000000000000000000:00000000000000031780, shard=2 master=00000000000000000000:00000000000000051951 target=00000000000000000000:00000000000000051671 2023-05-31T21:30:03.079 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-31T21:30:03.080 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-31T21:30:03.081 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-05-31T21:30:03.141 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2023-05-31T21:30:03.142 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2023-05-31T21:30:03.198 INFO:teuthology.orchestra.run.smithi099.stdout:{ 2023-05-31T21:30:03.199 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_status": { 2023-05-31T21:30:03.199 INFO:teuthology.orchestra.run.smithi099.stdout: "info": { 2023-05-31T21:30:03.199 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "sync", 2023-05-31T21:30:03.199 INFO:teuthology.orchestra.run.smithi099.stdout: "num_shards": 4, 2023-05-31T21:30:03.199 INFO:teuthology.orchestra.run.smithi099.stdout: "instance_id": 452136274422638824 2023-05-31T21:30:03.199 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:30:03.200 INFO:teuthology.orchestra.run.smithi099.stdout: "markers": [ 2023-05-31T21:30:03.200 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T21:30:03.200 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 0, 2023-05-31T21:30:03.200 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T21:30:03.200 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "incremental-sync", 2023-05-31T21:30:03.200 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "00000000000000000000:00000000000000045375", 2023-05-31T21:30:03.200 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T21:30:03.201 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2023-05-31T21:30:03.201 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T21:30:03.201 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2023-05-31T21:28:16.422581Z" 2023-05-31T21:30:03.201 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T21:30:03.201 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:30:03.201 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T21:30:03.201 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 1, 2023-05-31T21:30:03.202 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T21:30:03.202 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "incremental-sync", 2023-05-31T21:30:03.202 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "00000000000000000000:00000000000000031780", 2023-05-31T21:30:03.202 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T21:30:03.202 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2023-05-31T21:30:03.202 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T21:30:03.202 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2023-05-31T21:28:16.501293Z" 2023-05-31T21:30:03.203 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T21:30:03.203 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:30:03.203 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T21:30:03.203 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 2, 2023-05-31T21:30:03.203 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T21:30:03.203 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "incremental-sync", 2023-05-31T21:30:03.203 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "00000000000000000000:00000000000000051671", 2023-05-31T21:30:03.204 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T21:30:03.204 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 55, 2023-05-31T21:30:03.204 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T21:30:03.204 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2023-05-31T21:28:16.476487Z" 2023-05-31T21:30:03.204 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T21:30:03.204 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:30:03.205 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T21:30:03.205 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 3, 2023-05-31T21:30:03.205 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T21:30:03.205 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "incremental-sync", 2023-05-31T21:30:03.205 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "00000000000000000000:00000000000000044178", 2023-05-31T21:30:03.205 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T21:30:03.205 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 9, 2023-05-31T21:30:03.206 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T21:30:03.206 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2023-05-31T21:27:37.215905Z" 2023-05-31T21:30:03.206 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T21:30:03.206 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T21:30:03.206 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2023-05-31T21:30:03.206 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:30:03.206 INFO:teuthology.orchestra.run.smithi099.stdout: "full_sync": { 2023-05-31T21:30:03.207 INFO:teuthology.orchestra.run.smithi099.stdout: "total": 64, 2023-05-31T21:30:03.207 INFO:teuthology.orchestra.run.smithi099.stdout: "complete": 64 2023-05-31T21:30:03.207 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:30:03.207 INFO:teuthology.orchestra.run.smithi099.stdout: "current_time": "2023-05-31T21:30:03Z" 2023-05-31T21:30:03.207 INFO:teuthology.orchestra.run.smithi099.stdout:} 2023-05-31T21:30:03.208 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 452136274422638824 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000045375", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-05-31T21:28:16.422581Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000031780", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-05-31T21:28:16.501293Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000051671", "next_step_marker": "", "total_entries": 55, "pos": 0, "timestamp": "2023-05-31T21:28:16.476487Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000044178", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2023-05-31T21:27:37.215905Z" } } ] }, "full_sync": { "total": 64, "complete": 64 }, "current_time": "2023-05-31T21:30:03Z" } 2023-05-31T21:30:03.209 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000045375', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-05-31T21:28:16.422581Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000031780', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-05-31T21:28:16.501293Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000051671', 'next_step_marker': '', 'total_entries': 55, 'pos': 0, 'timestamp': '2023-05-31T21:28:16.476487Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000044178', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2023-05-31T21:27:37.215905Z'}}] 2023-05-31T21:30:03.209 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000045515', 1: '00000000000000000000:00000000000000031920', 2: '00000000000000000000:00000000000000051951', 3: '00000000000000000000:00000000000000044178'} 2023-05-31T21:30:03.209 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000045375', 1: '00000000000000000000:00000000000000031780', 2: '00000000000000000000:00000000000000051671', 3: '00000000000000000000:00000000000000044178'} 2023-05-31T21:30:03.210 WARNING:rgw_multi.tests:data of zone a2 behind zone a1: shard=0 master=00000000000000000000:00000000000000045515 target=00000000000000000000:00000000000000045375, shard=1 master=00000000000000000000:00000000000000031920 target=00000000000000000000:00000000000000031780, shard=2 master=00000000000000000000:00000000000000051951 target=00000000000000000000:00000000000000051671 2023-05-31T21:30:08.211 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-31T21:30:08.211 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-31T21:30:08.212 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-05-31T21:30:08.271 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2023-05-31T21:30:08.271 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2023-05-31T21:30:08.329 INFO:teuthology.orchestra.run.smithi099.stdout:{ 2023-05-31T21:30:08.329 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_status": { 2023-05-31T21:30:08.329 INFO:teuthology.orchestra.run.smithi099.stdout: "info": { 2023-05-31T21:30:08.330 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "sync", 2023-05-31T21:30:08.330 INFO:teuthology.orchestra.run.smithi099.stdout: "num_shards": 4, 2023-05-31T21:30:08.330 INFO:teuthology.orchestra.run.smithi099.stdout: "instance_id": 452136274422638824 2023-05-31T21:30:08.330 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:30:08.330 INFO:teuthology.orchestra.run.smithi099.stdout: "markers": [ 2023-05-31T21:30:08.330 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T21:30:08.331 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 0, 2023-05-31T21:30:08.331 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T21:30:08.331 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "incremental-sync", 2023-05-31T21:30:08.331 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "00000000000000000000:00000000000000045375", 2023-05-31T21:30:08.331 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T21:30:08.331 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2023-05-31T21:30:08.331 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T21:30:08.332 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2023-05-31T21:28:16.422581Z" 2023-05-31T21:30:08.332 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T21:30:08.332 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:30:08.332 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T21:30:08.332 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 1, 2023-05-31T21:30:08.332 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T21:30:08.333 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "incremental-sync", 2023-05-31T21:30:08.333 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "00000000000000000000:00000000000000031780", 2023-05-31T21:30:08.333 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T21:30:08.333 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2023-05-31T21:30:08.333 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T21:30:08.333 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2023-05-31T21:28:16.501293Z" 2023-05-31T21:30:08.333 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T21:30:08.334 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:30:08.334 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T21:30:08.334 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 2, 2023-05-31T21:30:08.334 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T21:30:08.334 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "incremental-sync", 2023-05-31T21:30:08.334 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "00000000000000000000:00000000000000051671", 2023-05-31T21:30:08.334 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T21:30:08.335 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 55, 2023-05-31T21:30:08.335 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T21:30:08.335 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2023-05-31T21:28:16.476487Z" 2023-05-31T21:30:08.335 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T21:30:08.335 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:30:08.335 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T21:30:08.335 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 3, 2023-05-31T21:30:08.336 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T21:30:08.336 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "incremental-sync", 2023-05-31T21:30:08.336 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "00000000000000000000:00000000000000044178", 2023-05-31T21:30:08.336 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T21:30:08.336 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 9, 2023-05-31T21:30:08.336 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T21:30:08.337 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2023-05-31T21:27:37.215905Z" 2023-05-31T21:30:08.337 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T21:30:08.337 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T21:30:08.337 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2023-05-31T21:30:08.337 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:30:08.337 INFO:teuthology.orchestra.run.smithi099.stdout: "full_sync": { 2023-05-31T21:30:08.337 INFO:teuthology.orchestra.run.smithi099.stdout: "total": 64, 2023-05-31T21:30:08.338 INFO:teuthology.orchestra.run.smithi099.stdout: "complete": 64 2023-05-31T21:30:08.338 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:30:08.338 INFO:teuthology.orchestra.run.smithi099.stdout: "current_time": "2023-05-31T21:30:08Z" 2023-05-31T21:30:08.338 INFO:teuthology.orchestra.run.smithi099.stdout:} 2023-05-31T21:30:08.339 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 452136274422638824 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000045375", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-05-31T21:28:16.422581Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000031780", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-05-31T21:28:16.501293Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000051671", "next_step_marker": "", "total_entries": 55, "pos": 0, "timestamp": "2023-05-31T21:28:16.476487Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000044178", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2023-05-31T21:27:37.215905Z" } } ] }, "full_sync": { "total": 64, "complete": 64 }, "current_time": "2023-05-31T21:30:08Z" } 2023-05-31T21:30:08.339 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000045375', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-05-31T21:28:16.422581Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000031780', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-05-31T21:28:16.501293Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000051671', 'next_step_marker': '', 'total_entries': 55, 'pos': 0, 'timestamp': '2023-05-31T21:28:16.476487Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000044178', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2023-05-31T21:27:37.215905Z'}}] 2023-05-31T21:30:08.340 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000045515', 1: '00000000000000000000:00000000000000031920', 2: '00000000000000000000:00000000000000051951', 3: '00000000000000000000:00000000000000044178'} 2023-05-31T21:30:08.340 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000045375', 1: '00000000000000000000:00000000000000031780', 2: '00000000000000000000:00000000000000051671', 3: '00000000000000000000:00000000000000044178'} 2023-05-31T21:30:08.341 WARNING:rgw_multi.tests:data of zone a2 behind zone a1: shard=0 master=00000000000000000000:00000000000000045515 target=00000000000000000000:00000000000000045375, shard=1 master=00000000000000000000:00000000000000031920 target=00000000000000000000:00000000000000031780, shard=2 master=00000000000000000000:00000000000000051951 target=00000000000000000000:00000000000000051671 2023-05-31T21:30:13.342 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-31T21:30:13.342 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-31T21:30:13.343 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-05-31T21:30:13.402 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2023-05-31T21:30:13.403 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2023-05-31T21:30:13.471 INFO:teuthology.orchestra.run.smithi099.stdout:{ 2023-05-31T21:30:13.471 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_status": { 2023-05-31T21:30:13.471 INFO:teuthology.orchestra.run.smithi099.stdout: "info": { 2023-05-31T21:30:13.471 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "sync", 2023-05-31T21:30:13.472 INFO:teuthology.orchestra.run.smithi099.stdout: "num_shards": 4, 2023-05-31T21:30:13.472 INFO:teuthology.orchestra.run.smithi099.stdout: "instance_id": 452136274422638824 2023-05-31T21:30:13.472 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:30:13.472 INFO:teuthology.orchestra.run.smithi099.stdout: "markers": [ 2023-05-31T21:30:13.472 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T21:30:13.472 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 0, 2023-05-31T21:30:13.472 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T21:30:13.473 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "incremental-sync", 2023-05-31T21:30:13.473 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "00000000000000000000:00000000000000045375", 2023-05-31T21:30:13.473 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T21:30:13.473 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2023-05-31T21:30:13.473 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T21:30:13.473 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2023-05-31T21:28:16.422581Z" 2023-05-31T21:30:13.473 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T21:30:13.474 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:30:13.474 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T21:30:13.474 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 1, 2023-05-31T21:30:13.474 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T21:30:13.474 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "incremental-sync", 2023-05-31T21:30:13.474 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "00000000000000000000:00000000000000031780", 2023-05-31T21:30:13.474 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T21:30:13.475 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2023-05-31T21:30:13.475 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T21:30:13.475 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2023-05-31T21:28:16.501293Z" 2023-05-31T21:30:13.475 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T21:30:13.475 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:30:13.475 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T21:30:13.475 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 2, 2023-05-31T21:30:13.476 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T21:30:13.476 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "incremental-sync", 2023-05-31T21:30:13.476 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "00000000000000000000:00000000000000051671", 2023-05-31T21:30:13.476 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T21:30:13.476 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 55, 2023-05-31T21:30:13.476 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T21:30:13.477 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2023-05-31T21:28:16.476487Z" 2023-05-31T21:30:13.477 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T21:30:13.477 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:30:13.477 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T21:30:13.477 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 3, 2023-05-31T21:30:13.477 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T21:30:13.477 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "incremental-sync", 2023-05-31T21:30:13.478 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "00000000000000000000:00000000000000044178", 2023-05-31T21:30:13.478 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T21:30:13.478 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 9, 2023-05-31T21:30:13.478 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T21:30:13.478 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2023-05-31T21:27:37.215905Z" 2023-05-31T21:30:13.478 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T21:30:13.478 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T21:30:13.479 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2023-05-31T21:30:13.479 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:30:13.479 INFO:teuthology.orchestra.run.smithi099.stdout: "full_sync": { 2023-05-31T21:30:13.479 INFO:teuthology.orchestra.run.smithi099.stdout: "total": 64, 2023-05-31T21:30:13.479 INFO:teuthology.orchestra.run.smithi099.stdout: "complete": 64 2023-05-31T21:30:13.479 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:30:13.479 INFO:teuthology.orchestra.run.smithi099.stdout: "current_time": "2023-05-31T21:30:13Z" 2023-05-31T21:30:13.480 INFO:teuthology.orchestra.run.smithi099.stdout:} 2023-05-31T21:30:13.480 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 452136274422638824 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000045375", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-05-31T21:28:16.422581Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000031780", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-05-31T21:28:16.501293Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000051671", "next_step_marker": "", "total_entries": 55, "pos": 0, "timestamp": "2023-05-31T21:28:16.476487Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000044178", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2023-05-31T21:27:37.215905Z" } } ] }, "full_sync": { "total": 64, "complete": 64 }, "current_time": "2023-05-31T21:30:13Z" } 2023-05-31T21:30:13.481 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000045375', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-05-31T21:28:16.422581Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000031780', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-05-31T21:28:16.501293Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000051671', 'next_step_marker': '', 'total_entries': 55, 'pos': 0, 'timestamp': '2023-05-31T21:28:16.476487Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000044178', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2023-05-31T21:27:37.215905Z'}}] 2023-05-31T21:30:13.481 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000045515', 1: '00000000000000000000:00000000000000031920', 2: '00000000000000000000:00000000000000051951', 3: '00000000000000000000:00000000000000044178'} 2023-05-31T21:30:13.482 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000045375', 1: '00000000000000000000:00000000000000031780', 2: '00000000000000000000:00000000000000051671', 3: '00000000000000000000:00000000000000044178'} 2023-05-31T21:30:13.482 WARNING:rgw_multi.tests:data of zone a2 behind zone a1: shard=0 master=00000000000000000000:00000000000000045515 target=00000000000000000000:00000000000000045375, shard=1 master=00000000000000000000:00000000000000031920 target=00000000000000000000:00000000000000031780, shard=2 master=00000000000000000000:00000000000000051951 target=00000000000000000000:00000000000000051671 2023-05-31T21:30:18.483 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-31T21:30:18.484 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-31T21:30:18.484 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-05-31T21:30:18.543 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2023-05-31T21:30:18.543 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2023-05-31T21:30:18.596 INFO:teuthology.orchestra.run.smithi099.stdout:{ 2023-05-31T21:30:18.597 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_status": { 2023-05-31T21:30:18.597 INFO:teuthology.orchestra.run.smithi099.stdout: "info": { 2023-05-31T21:30:18.597 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "sync", 2023-05-31T21:30:18.597 INFO:teuthology.orchestra.run.smithi099.stdout: "num_shards": 4, 2023-05-31T21:30:18.598 INFO:teuthology.orchestra.run.smithi099.stdout: "instance_id": 452136274422638824 2023-05-31T21:30:18.598 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:30:18.598 INFO:teuthology.orchestra.run.smithi099.stdout: "markers": [ 2023-05-31T21:30:18.598 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T21:30:18.598 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 0, 2023-05-31T21:30:18.598 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T21:30:18.599 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "incremental-sync", 2023-05-31T21:30:18.599 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "00000000000000000000:00000000000000045375", 2023-05-31T21:30:18.599 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T21:30:18.599 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2023-05-31T21:30:18.599 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T21:30:18.599 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2023-05-31T21:28:16.422581Z" 2023-05-31T21:30:18.600 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T21:30:18.600 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:30:18.600 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T21:30:18.600 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 1, 2023-05-31T21:30:18.600 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T21:30:18.601 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "incremental-sync", 2023-05-31T21:30:18.601 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "00000000000000000000:00000000000000031780", 2023-05-31T21:30:18.601 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T21:30:18.601 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2023-05-31T21:30:18.601 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T21:30:18.601 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2023-05-31T21:28:16.501293Z" 2023-05-31T21:30:18.602 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T21:30:18.602 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:30:18.602 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T21:30:18.602 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 2, 2023-05-31T21:30:18.602 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T21:30:18.602 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "incremental-sync", 2023-05-31T21:30:18.603 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "00000000000000000000:00000000000000051671", 2023-05-31T21:30:18.603 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T21:30:18.603 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 55, 2023-05-31T21:30:18.603 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T21:30:18.603 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2023-05-31T21:28:16.476487Z" 2023-05-31T21:30:18.603 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T21:30:18.603 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:30:18.604 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T21:30:18.604 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 3, 2023-05-31T21:30:18.604 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T21:30:18.604 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "incremental-sync", 2023-05-31T21:30:18.604 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "00000000000000000000:00000000000000044178", 2023-05-31T21:30:18.604 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T21:30:18.605 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 9, 2023-05-31T21:30:18.605 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T21:30:18.605 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2023-05-31T21:27:37.215905Z" 2023-05-31T21:30:18.605 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T21:30:18.605 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T21:30:18.605 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2023-05-31T21:30:18.606 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:30:18.606 INFO:teuthology.orchestra.run.smithi099.stdout: "full_sync": { 2023-05-31T21:30:18.606 INFO:teuthology.orchestra.run.smithi099.stdout: "total": 64, 2023-05-31T21:30:18.606 INFO:teuthology.orchestra.run.smithi099.stdout: "complete": 64 2023-05-31T21:30:18.606 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:30:18.606 INFO:teuthology.orchestra.run.smithi099.stdout: "current_time": "2023-05-31T21:30:18Z" 2023-05-31T21:30:18.606 INFO:teuthology.orchestra.run.smithi099.stdout:} 2023-05-31T21:30:18.607 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 452136274422638824 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000045375", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-05-31T21:28:16.422581Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000031780", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-05-31T21:28:16.501293Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000051671", "next_step_marker": "", "total_entries": 55, "pos": 0, "timestamp": "2023-05-31T21:28:16.476487Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000044178", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2023-05-31T21:27:37.215905Z" } } ] }, "full_sync": { "total": 64, "complete": 64 }, "current_time": "2023-05-31T21:30:18Z" } 2023-05-31T21:30:18.608 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000045375', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-05-31T21:28:16.422581Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000031780', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-05-31T21:28:16.501293Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000051671', 'next_step_marker': '', 'total_entries': 55, 'pos': 0, 'timestamp': '2023-05-31T21:28:16.476487Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000044178', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2023-05-31T21:27:37.215905Z'}}] 2023-05-31T21:30:18.608 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000045515', 1: '00000000000000000000:00000000000000031920', 2: '00000000000000000000:00000000000000051951', 3: '00000000000000000000:00000000000000044178'} 2023-05-31T21:30:18.609 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000045375', 1: '00000000000000000000:00000000000000031780', 2: '00000000000000000000:00000000000000051671', 3: '00000000000000000000:00000000000000044178'} 2023-05-31T21:30:18.609 WARNING:rgw_multi.tests:data of zone a2 behind zone a1: shard=0 master=00000000000000000000:00000000000000045515 target=00000000000000000000:00000000000000045375, shard=1 master=00000000000000000000:00000000000000031920 target=00000000000000000000:00000000000000031780, shard=2 master=00000000000000000000:00000000000000051951 target=00000000000000000000:00000000000000051671 2023-05-31T21:30:23.610 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-31T21:30:23.611 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-31T21:30:23.611 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-05-31T21:30:23.670 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2023-05-31T21:30:23.671 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2023-05-31T21:30:23.745 INFO:teuthology.orchestra.run.smithi099.stdout:{ 2023-05-31T21:30:23.745 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_status": { 2023-05-31T21:30:23.745 INFO:teuthology.orchestra.run.smithi099.stdout: "info": { 2023-05-31T21:30:23.745 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "sync", 2023-05-31T21:30:23.746 INFO:teuthology.orchestra.run.smithi099.stdout: "num_shards": 4, 2023-05-31T21:30:23.746 INFO:teuthology.orchestra.run.smithi099.stdout: "instance_id": 452136274422638824 2023-05-31T21:30:23.746 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:30:23.746 INFO:teuthology.orchestra.run.smithi099.stdout: "markers": [ 2023-05-31T21:30:23.747 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T21:30:23.747 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 0, 2023-05-31T21:30:23.747 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T21:30:23.747 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "incremental-sync", 2023-05-31T21:30:23.747 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "00000000000000000000:00000000000000045375", 2023-05-31T21:30:23.748 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T21:30:23.748 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2023-05-31T21:30:23.748 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T21:30:23.748 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2023-05-31T21:28:16.422581Z" 2023-05-31T21:30:23.748 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T21:30:23.748 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:30:23.749 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T21:30:23.749 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 1, 2023-05-31T21:30:23.749 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T21:30:23.749 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "incremental-sync", 2023-05-31T21:30:23.749 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "00000000000000000000:00000000000000031780", 2023-05-31T21:30:23.749 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T21:30:23.749 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2023-05-31T21:30:23.750 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T21:30:23.750 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2023-05-31T21:28:16.501293Z" 2023-05-31T21:30:23.750 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T21:30:23.750 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:30:23.750 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T21:30:23.750 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 2, 2023-05-31T21:30:23.750 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T21:30:23.751 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "incremental-sync", 2023-05-31T21:30:23.751 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "00000000000000000000:00000000000000051671", 2023-05-31T21:30:23.751 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T21:30:23.751 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 55, 2023-05-31T21:30:23.751 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T21:30:23.751 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2023-05-31T21:28:16.476487Z" 2023-05-31T21:30:23.752 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T21:30:23.752 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:30:23.752 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T21:30:23.752 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 3, 2023-05-31T21:30:23.752 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T21:30:23.752 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "incremental-sync", 2023-05-31T21:30:23.753 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "00000000000000000000:00000000000000044178", 2023-05-31T21:30:23.753 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T21:30:23.753 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 9, 2023-05-31T21:30:23.753 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T21:30:23.753 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2023-05-31T21:27:37.215905Z" 2023-05-31T21:30:23.753 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T21:30:23.753 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T21:30:23.754 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2023-05-31T21:30:23.754 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:30:23.754 INFO:teuthology.orchestra.run.smithi099.stdout: "full_sync": { 2023-05-31T21:30:23.754 INFO:teuthology.orchestra.run.smithi099.stdout: "total": 64, 2023-05-31T21:30:23.754 INFO:teuthology.orchestra.run.smithi099.stdout: "complete": 64 2023-05-31T21:30:23.754 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:30:23.754 INFO:teuthology.orchestra.run.smithi099.stdout: "current_time": "2023-05-31T21:30:23Z" 2023-05-31T21:30:23.755 INFO:teuthology.orchestra.run.smithi099.stdout:} 2023-05-31T21:30:23.756 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 452136274422638824 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000045375", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-05-31T21:28:16.422581Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000031780", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-05-31T21:28:16.501293Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000051671", "next_step_marker": "", "total_entries": 55, "pos": 0, "timestamp": "2023-05-31T21:28:16.476487Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000044178", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2023-05-31T21:27:37.215905Z" } } ] }, "full_sync": { "total": 64, "complete": 64 }, "current_time": "2023-05-31T21:30:23Z" } 2023-05-31T21:30:23.756 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000045375', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-05-31T21:28:16.422581Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000031780', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-05-31T21:28:16.501293Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000051671', 'next_step_marker': '', 'total_entries': 55, 'pos': 0, 'timestamp': '2023-05-31T21:28:16.476487Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000044178', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2023-05-31T21:27:37.215905Z'}}] 2023-05-31T21:30:23.757 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000045515', 1: '00000000000000000000:00000000000000031920', 2: '00000000000000000000:00000000000000051951', 3: '00000000000000000000:00000000000000044178'} 2023-05-31T21:30:23.757 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000045375', 1: '00000000000000000000:00000000000000031780', 2: '00000000000000000000:00000000000000051671', 3: '00000000000000000000:00000000000000044178'} 2023-05-31T21:30:23.757 WARNING:rgw_multi.tests:data of zone a2 behind zone a1: shard=0 master=00000000000000000000:00000000000000045515 target=00000000000000000000:00000000000000045375, shard=1 master=00000000000000000000:00000000000000031920 target=00000000000000000000:00000000000000031780, shard=2 master=00000000000000000000:00000000000000051951 target=00000000000000000000:00000000000000051671 2023-05-31T21:30:28.758 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-31T21:30:28.759 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-31T21:30:28.759 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-05-31T21:30:28.818 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2023-05-31T21:30:28.818 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2023-05-31T21:30:28.869 INFO:teuthology.orchestra.run.smithi099.stdout:{ 2023-05-31T21:30:28.869 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_status": { 2023-05-31T21:30:28.870 INFO:teuthology.orchestra.run.smithi099.stdout: "info": { 2023-05-31T21:30:28.870 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "sync", 2023-05-31T21:30:28.870 INFO:teuthology.orchestra.run.smithi099.stdout: "num_shards": 4, 2023-05-31T21:30:28.870 INFO:teuthology.orchestra.run.smithi099.stdout: "instance_id": 452136274422638824 2023-05-31T21:30:28.870 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:30:28.870 INFO:teuthology.orchestra.run.smithi099.stdout: "markers": [ 2023-05-31T21:30:28.871 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T21:30:28.871 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 0, 2023-05-31T21:30:28.871 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T21:30:28.871 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "incremental-sync", 2023-05-31T21:30:28.871 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "00000000000000000000:00000000000000045375", 2023-05-31T21:30:28.871 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T21:30:28.871 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2023-05-31T21:30:28.872 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T21:30:28.872 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2023-05-31T21:28:16.422581Z" 2023-05-31T21:30:28.872 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T21:30:28.872 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:30:28.872 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T21:30:28.872 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 1, 2023-05-31T21:30:28.872 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T21:30:28.873 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "incremental-sync", 2023-05-31T21:30:28.873 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "00000000000000000000:00000000000000031780", 2023-05-31T21:30:28.873 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T21:30:28.873 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2023-05-31T21:30:28.873 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T21:30:28.873 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2023-05-31T21:28:16.501293Z" 2023-05-31T21:30:28.873 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T21:30:28.874 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:30:28.874 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T21:30:28.874 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 2, 2023-05-31T21:30:28.874 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T21:30:28.874 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "incremental-sync", 2023-05-31T21:30:28.874 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "00000000000000000000:00000000000000051671", 2023-05-31T21:30:28.874 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T21:30:28.875 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 55, 2023-05-31T21:30:28.875 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T21:30:28.875 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2023-05-31T21:28:16.476487Z" 2023-05-31T21:30:28.875 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T21:30:28.875 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:30:28.876 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T21:30:28.876 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 3, 2023-05-31T21:30:28.876 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T21:30:28.876 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "incremental-sync", 2023-05-31T21:30:28.876 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "00000000000000000000:00000000000000044178", 2023-05-31T21:30:28.876 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T21:30:28.876 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 9, 2023-05-31T21:30:28.877 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T21:30:28.877 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2023-05-31T21:27:37.215905Z" 2023-05-31T21:30:28.877 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T21:30:28.877 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T21:30:28.877 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2023-05-31T21:30:28.877 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:30:28.877 INFO:teuthology.orchestra.run.smithi099.stdout: "full_sync": { 2023-05-31T21:30:28.878 INFO:teuthology.orchestra.run.smithi099.stdout: "total": 64, 2023-05-31T21:30:28.878 INFO:teuthology.orchestra.run.smithi099.stdout: "complete": 64 2023-05-31T21:30:28.878 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:30:28.878 INFO:teuthology.orchestra.run.smithi099.stdout: "current_time": "2023-05-31T21:30:28Z" 2023-05-31T21:30:28.878 INFO:teuthology.orchestra.run.smithi099.stdout:} 2023-05-31T21:30:28.879 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 452136274422638824 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000045375", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-05-31T21:28:16.422581Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000031780", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-05-31T21:28:16.501293Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000051671", "next_step_marker": "", "total_entries": 55, "pos": 0, "timestamp": "2023-05-31T21:28:16.476487Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000044178", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2023-05-31T21:27:37.215905Z" } } ] }, "full_sync": { "total": 64, "complete": 64 }, "current_time": "2023-05-31T21:30:28Z" } 2023-05-31T21:30:28.879 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000045375', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-05-31T21:28:16.422581Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000031780', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-05-31T21:28:16.501293Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000051671', 'next_step_marker': '', 'total_entries': 55, 'pos': 0, 'timestamp': '2023-05-31T21:28:16.476487Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000044178', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2023-05-31T21:27:37.215905Z'}}] 2023-05-31T21:30:28.880 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000045515', 1: '00000000000000000000:00000000000000031920', 2: '00000000000000000000:00000000000000051951', 3: '00000000000000000000:00000000000000044178'} 2023-05-31T21:30:28.880 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000045375', 1: '00000000000000000000:00000000000000031780', 2: '00000000000000000000:00000000000000051671', 3: '00000000000000000000:00000000000000044178'} 2023-05-31T21:30:28.881 WARNING:rgw_multi.tests:data of zone a2 behind zone a1: shard=0 master=00000000000000000000:00000000000000045515 target=00000000000000000000:00000000000000045375, shard=1 master=00000000000000000000:00000000000000031920 target=00000000000000000000:00000000000000031780, shard=2 master=00000000000000000000:00000000000000051951 target=00000000000000000000:00000000000000051671 2023-05-31T21:30:33.882 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-31T21:30:33.882 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-31T21:30:33.883 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-05-31T21:30:33.943 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2023-05-31T21:30:33.943 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2023-05-31T21:30:34.009 INFO:teuthology.orchestra.run.smithi099.stdout:{ 2023-05-31T21:30:34.009 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_status": { 2023-05-31T21:30:34.010 INFO:teuthology.orchestra.run.smithi099.stdout: "info": { 2023-05-31T21:30:34.010 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "sync", 2023-05-31T21:30:34.010 INFO:teuthology.orchestra.run.smithi099.stdout: "num_shards": 4, 2023-05-31T21:30:34.010 INFO:teuthology.orchestra.run.smithi099.stdout: "instance_id": 452136274422638824 2023-05-31T21:30:34.010 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:30:34.010 INFO:teuthology.orchestra.run.smithi099.stdout: "markers": [ 2023-05-31T21:30:34.010 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T21:30:34.011 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 0, 2023-05-31T21:30:34.011 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T21:30:34.011 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "incremental-sync", 2023-05-31T21:30:34.011 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "00000000000000000000:00000000000000045375", 2023-05-31T21:30:34.011 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T21:30:34.011 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2023-05-31T21:30:34.011 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T21:30:34.012 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2023-05-31T21:28:16.422581Z" 2023-05-31T21:30:34.012 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T21:30:34.012 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:30:34.012 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T21:30:34.013 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 1, 2023-05-31T21:30:34.013 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T21:30:34.013 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "incremental-sync", 2023-05-31T21:30:34.013 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "00000000000000000000:00000000000000031780", 2023-05-31T21:30:34.013 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T21:30:34.013 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2023-05-31T21:30:34.014 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T21:30:34.014 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2023-05-31T21:28:16.501293Z" 2023-05-31T21:30:34.014 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T21:30:34.014 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:30:34.014 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T21:30:34.014 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 2, 2023-05-31T21:30:34.015 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T21:30:34.015 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "incremental-sync", 2023-05-31T21:30:34.015 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "00000000000000000000:00000000000000051671", 2023-05-31T21:30:34.015 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T21:30:34.015 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 55, 2023-05-31T21:30:34.015 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T21:30:34.015 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2023-05-31T21:28:16.476487Z" 2023-05-31T21:30:34.016 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T21:30:34.016 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:30:34.016 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T21:30:34.016 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 3, 2023-05-31T21:30:34.016 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T21:30:34.016 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "incremental-sync", 2023-05-31T21:30:34.017 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "00000000000000000000:00000000000000044178", 2023-05-31T21:30:34.017 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T21:30:34.017 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 9, 2023-05-31T21:30:34.017 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T21:30:34.017 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2023-05-31T21:27:37.215905Z" 2023-05-31T21:30:34.017 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T21:30:34.017 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T21:30:34.018 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2023-05-31T21:30:34.018 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:30:34.018 INFO:teuthology.orchestra.run.smithi099.stdout: "full_sync": { 2023-05-31T21:30:34.018 INFO:teuthology.orchestra.run.smithi099.stdout: "total": 64, 2023-05-31T21:30:34.018 INFO:teuthology.orchestra.run.smithi099.stdout: "complete": 64 2023-05-31T21:30:34.018 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:30:34.018 INFO:teuthology.orchestra.run.smithi099.stdout: "current_time": "2023-05-31T21:30:33Z" 2023-05-31T21:30:34.019 INFO:teuthology.orchestra.run.smithi099.stdout:} 2023-05-31T21:30:34.019 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 452136274422638824 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000045375", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-05-31T21:28:16.422581Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000031780", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-05-31T21:28:16.501293Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000051671", "next_step_marker": "", "total_entries": 55, "pos": 0, "timestamp": "2023-05-31T21:28:16.476487Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000044178", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2023-05-31T21:27:37.215905Z" } } ] }, "full_sync": { "total": 64, "complete": 64 }, "current_time": "2023-05-31T21:30:33Z" } 2023-05-31T21:30:34.020 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000045375', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-05-31T21:28:16.422581Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000031780', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-05-31T21:28:16.501293Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000051671', 'next_step_marker': '', 'total_entries': 55, 'pos': 0, 'timestamp': '2023-05-31T21:28:16.476487Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000044178', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2023-05-31T21:27:37.215905Z'}}] 2023-05-31T21:30:34.020 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000045515', 1: '00000000000000000000:00000000000000031920', 2: '00000000000000000000:00000000000000051951', 3: '00000000000000000000:00000000000000044178'} 2023-05-31T21:30:34.021 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000045375', 1: '00000000000000000000:00000000000000031780', 2: '00000000000000000000:00000000000000051671', 3: '00000000000000000000:00000000000000044178'} 2023-05-31T21:30:34.021 WARNING:rgw_multi.tests:data of zone a2 behind zone a1: shard=0 master=00000000000000000000:00000000000000045515 target=00000000000000000000:00000000000000045375, shard=1 master=00000000000000000000:00000000000000031920 target=00000000000000000000:00000000000000031780, shard=2 master=00000000000000000000:00000000000000051951 target=00000000000000000000:00000000000000051671 2023-05-31T21:30:39.022 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-31T21:30:39.022 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-31T21:30:39.023 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-05-31T21:30:39.083 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2023-05-31T21:30:39.083 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2023-05-31T21:30:39.138 INFO:teuthology.orchestra.run.smithi099.stdout:{ 2023-05-31T21:30:39.138 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_status": { 2023-05-31T21:30:39.138 INFO:teuthology.orchestra.run.smithi099.stdout: "info": { 2023-05-31T21:30:39.139 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "sync", 2023-05-31T21:30:39.139 INFO:teuthology.orchestra.run.smithi099.stdout: "num_shards": 4, 2023-05-31T21:30:39.139 INFO:teuthology.orchestra.run.smithi099.stdout: "instance_id": 452136274422638824 2023-05-31T21:30:39.139 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:30:39.139 INFO:teuthology.orchestra.run.smithi099.stdout: "markers": [ 2023-05-31T21:30:39.139 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T21:30:39.139 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 0, 2023-05-31T21:30:39.140 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T21:30:39.140 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "incremental-sync", 2023-05-31T21:30:39.140 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "00000000000000000000:00000000000000045375", 2023-05-31T21:30:39.140 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T21:30:39.140 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2023-05-31T21:30:39.140 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T21:30:39.141 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2023-05-31T21:28:16.422581Z" 2023-05-31T21:30:39.141 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T21:30:39.141 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:30:39.141 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T21:30:39.141 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 1, 2023-05-31T21:30:39.141 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T21:30:39.142 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "incremental-sync", 2023-05-31T21:30:39.142 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "00000000000000000000:00000000000000031780", 2023-05-31T21:30:39.142 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T21:30:39.142 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2023-05-31T21:30:39.142 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T21:30:39.142 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2023-05-31T21:28:16.501293Z" 2023-05-31T21:30:39.143 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T21:30:39.143 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:30:39.143 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T21:30:39.143 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 2, 2023-05-31T21:30:39.143 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T21:30:39.143 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "incremental-sync", 2023-05-31T21:30:39.143 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "00000000000000000000:00000000000000051671", 2023-05-31T21:30:39.144 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T21:30:39.144 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 55, 2023-05-31T21:30:39.144 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T21:30:39.144 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2023-05-31T21:28:16.476487Z" 2023-05-31T21:30:39.144 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T21:30:39.144 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:30:39.145 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T21:30:39.145 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 3, 2023-05-31T21:30:39.145 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T21:30:39.145 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "incremental-sync", 2023-05-31T21:30:39.145 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "00000000000000000000:00000000000000044178", 2023-05-31T21:30:39.145 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T21:30:39.146 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 9, 2023-05-31T21:30:39.146 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T21:30:39.146 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2023-05-31T21:27:37.215905Z" 2023-05-31T21:30:39.146 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T21:30:39.146 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T21:30:39.146 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2023-05-31T21:30:39.146 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:30:39.147 INFO:teuthology.orchestra.run.smithi099.stdout: "full_sync": { 2023-05-31T21:30:39.147 INFO:teuthology.orchestra.run.smithi099.stdout: "total": 64, 2023-05-31T21:30:39.147 INFO:teuthology.orchestra.run.smithi099.stdout: "complete": 64 2023-05-31T21:30:39.147 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:30:39.147 INFO:teuthology.orchestra.run.smithi099.stdout: "current_time": "2023-05-31T21:30:39Z" 2023-05-31T21:30:39.147 INFO:teuthology.orchestra.run.smithi099.stdout:} 2023-05-31T21:30:39.148 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 452136274422638824 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000045375", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-05-31T21:28:16.422581Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000031780", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-05-31T21:28:16.501293Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000051671", "next_step_marker": "", "total_entries": 55, "pos": 0, "timestamp": "2023-05-31T21:28:16.476487Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000044178", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2023-05-31T21:27:37.215905Z" } } ] }, "full_sync": { "total": 64, "complete": 64 }, "current_time": "2023-05-31T21:30:39Z" } 2023-05-31T21:30:39.149 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000045375', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-05-31T21:28:16.422581Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000031780', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-05-31T21:28:16.501293Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000051671', 'next_step_marker': '', 'total_entries': 55, 'pos': 0, 'timestamp': '2023-05-31T21:28:16.476487Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000044178', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2023-05-31T21:27:37.215905Z'}}] 2023-05-31T21:30:39.149 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000045515', 1: '00000000000000000000:00000000000000031920', 2: '00000000000000000000:00000000000000051951', 3: '00000000000000000000:00000000000000044178'} 2023-05-31T21:30:39.150 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000045375', 1: '00000000000000000000:00000000000000031780', 2: '00000000000000000000:00000000000000051671', 3: '00000000000000000000:00000000000000044178'} 2023-05-31T21:30:39.150 WARNING:rgw_multi.tests:data of zone a2 behind zone a1: shard=0 master=00000000000000000000:00000000000000045515 target=00000000000000000000:00000000000000045375, shard=1 master=00000000000000000000:00000000000000031920 target=00000000000000000000:00000000000000031780, shard=2 master=00000000000000000000:00000000000000051951 target=00000000000000000000:00000000000000051671 2023-05-31T21:30:44.151 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-31T21:30:44.151 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-31T21:30:44.152 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-05-31T21:30:44.212 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2023-05-31T21:30:44.212 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2023-05-31T21:30:44.290 INFO:teuthology.orchestra.run.smithi099.stdout:{ 2023-05-31T21:30:44.290 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_status": { 2023-05-31T21:30:44.290 INFO:teuthology.orchestra.run.smithi099.stdout: "info": { 2023-05-31T21:30:44.290 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "sync", 2023-05-31T21:30:44.290 INFO:teuthology.orchestra.run.smithi099.stdout: "num_shards": 4, 2023-05-31T21:30:44.291 INFO:teuthology.orchestra.run.smithi099.stdout: "instance_id": 452136274422638824 2023-05-31T21:30:44.291 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:30:44.291 INFO:teuthology.orchestra.run.smithi099.stdout: "markers": [ 2023-05-31T21:30:44.291 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T21:30:44.291 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 0, 2023-05-31T21:30:44.291 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T21:30:44.291 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "incremental-sync", 2023-05-31T21:30:44.292 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "00000000000000000000:00000000000000045375", 2023-05-31T21:30:44.292 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T21:30:44.292 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2023-05-31T21:30:44.292 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T21:30:44.292 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2023-05-31T21:28:16.422581Z" 2023-05-31T21:30:44.292 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T21:30:44.293 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:30:44.293 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T21:30:44.293 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 1, 2023-05-31T21:30:44.293 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T21:30:44.293 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "incremental-sync", 2023-05-31T21:30:44.293 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "00000000000000000000:00000000000000031780", 2023-05-31T21:30:44.293 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T21:30:44.294 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2023-05-31T21:30:44.294 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T21:30:44.294 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2023-05-31T21:28:16.501293Z" 2023-05-31T21:30:44.294 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T21:30:44.294 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:30:44.294 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T21:30:44.294 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 2, 2023-05-31T21:30:44.295 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T21:30:44.295 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "incremental-sync", 2023-05-31T21:30:44.295 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "00000000000000000000:00000000000000051671", 2023-05-31T21:30:44.295 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T21:30:44.295 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 55, 2023-05-31T21:30:44.295 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T21:30:44.295 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2023-05-31T21:28:16.476487Z" 2023-05-31T21:30:44.296 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T21:30:44.296 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:30:44.296 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T21:30:44.296 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 3, 2023-05-31T21:30:44.296 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T21:30:44.296 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "incremental-sync", 2023-05-31T21:30:44.297 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "00000000000000000000:00000000000000044178", 2023-05-31T21:30:44.297 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T21:30:44.297 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 9, 2023-05-31T21:30:44.297 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T21:30:44.297 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2023-05-31T21:27:37.215905Z" 2023-05-31T21:30:44.297 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T21:30:44.298 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T21:30:44.298 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2023-05-31T21:30:44.298 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:30:44.298 INFO:teuthology.orchestra.run.smithi099.stdout: "full_sync": { 2023-05-31T21:30:44.298 INFO:teuthology.orchestra.run.smithi099.stdout: "total": 64, 2023-05-31T21:30:44.298 INFO:teuthology.orchestra.run.smithi099.stdout: "complete": 64 2023-05-31T21:30:44.298 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:30:44.299 INFO:teuthology.orchestra.run.smithi099.stdout: "current_time": "2023-05-31T21:30:44Z" 2023-05-31T21:30:44.299 INFO:teuthology.orchestra.run.smithi099.stdout:} 2023-05-31T21:30:44.300 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 452136274422638824 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000045375", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-05-31T21:28:16.422581Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000031780", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-05-31T21:28:16.501293Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000051671", "next_step_marker": "", "total_entries": 55, "pos": 0, "timestamp": "2023-05-31T21:28:16.476487Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000044178", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2023-05-31T21:27:37.215905Z" } } ] }, "full_sync": { "total": 64, "complete": 64 }, "current_time": "2023-05-31T21:30:44Z" } 2023-05-31T21:30:44.300 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000045375', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-05-31T21:28:16.422581Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000031780', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-05-31T21:28:16.501293Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000051671', 'next_step_marker': '', 'total_entries': 55, 'pos': 0, 'timestamp': '2023-05-31T21:28:16.476487Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000044178', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2023-05-31T21:27:37.215905Z'}}] 2023-05-31T21:30:44.301 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000045515', 1: '00000000000000000000:00000000000000031920', 2: '00000000000000000000:00000000000000051951', 3: '00000000000000000000:00000000000000044178'} 2023-05-31T21:30:44.301 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000045375', 1: '00000000000000000000:00000000000000031780', 2: '00000000000000000000:00000000000000051671', 3: '00000000000000000000:00000000000000044178'} 2023-05-31T21:30:44.301 WARNING:rgw_multi.tests:data of zone a2 behind zone a1: shard=0 master=00000000000000000000:00000000000000045515 target=00000000000000000000:00000000000000045375, shard=1 master=00000000000000000000:00000000000000031920 target=00000000000000000000:00000000000000031780, shard=2 master=00000000000000000000:00000000000000051951 target=00000000000000000000:00000000000000051671 2023-05-31T21:30:49.302 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-31T21:30:49.303 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-31T21:30:49.303 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-05-31T21:30:49.364 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2023-05-31T21:30:49.364 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2023-05-31T21:30:49.436 INFO:teuthology.orchestra.run.smithi099.stdout:{ 2023-05-31T21:30:49.436 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_status": { 2023-05-31T21:30:49.436 INFO:teuthology.orchestra.run.smithi099.stdout: "info": { 2023-05-31T21:30:49.436 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "sync", 2023-05-31T21:30:49.437 INFO:teuthology.orchestra.run.smithi099.stdout: "num_shards": 4, 2023-05-31T21:30:49.437 INFO:teuthology.orchestra.run.smithi099.stdout: "instance_id": 452136274422638824 2023-05-31T21:30:49.437 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:30:49.437 INFO:teuthology.orchestra.run.smithi099.stdout: "markers": [ 2023-05-31T21:30:49.437 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T21:30:49.437 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 0, 2023-05-31T21:30:49.438 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T21:30:49.438 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "incremental-sync", 2023-05-31T21:30:49.438 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "00000000000000000000:00000000000000045375", 2023-05-31T21:30:49.438 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T21:30:49.438 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2023-05-31T21:30:49.438 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T21:30:49.438 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2023-05-31T21:28:16.422581Z" 2023-05-31T21:30:49.439 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T21:30:49.439 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:30:49.439 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T21:30:49.439 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 1, 2023-05-31T21:30:49.439 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T21:30:49.439 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "incremental-sync", 2023-05-31T21:30:49.439 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "00000000000000000000:00000000000000031780", 2023-05-31T21:30:49.440 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T21:30:49.440 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2023-05-31T21:30:49.440 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T21:30:49.440 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2023-05-31T21:28:16.501293Z" 2023-05-31T21:30:49.440 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T21:30:49.440 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:30:49.441 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T21:30:49.441 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 2, 2023-05-31T21:30:49.441 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T21:30:49.441 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "incremental-sync", 2023-05-31T21:30:49.441 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "00000000000000000000:00000000000000051671", 2023-05-31T21:30:49.442 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T21:30:49.442 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 55, 2023-05-31T21:30:49.442 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T21:30:49.442 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2023-05-31T21:28:16.476487Z" 2023-05-31T21:30:49.442 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T21:30:49.442 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:30:49.443 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T21:30:49.443 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 3, 2023-05-31T21:30:49.443 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T21:30:49.443 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "incremental-sync", 2023-05-31T21:30:49.443 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "00000000000000000000:00000000000000044178", 2023-05-31T21:30:49.443 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T21:30:49.443 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 9, 2023-05-31T21:30:49.444 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T21:30:49.444 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2023-05-31T21:27:37.215905Z" 2023-05-31T21:30:49.444 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T21:30:49.444 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T21:30:49.444 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2023-05-31T21:30:49.444 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:30:49.444 INFO:teuthology.orchestra.run.smithi099.stdout: "full_sync": { 2023-05-31T21:30:49.445 INFO:teuthology.orchestra.run.smithi099.stdout: "total": 64, 2023-05-31T21:30:49.445 INFO:teuthology.orchestra.run.smithi099.stdout: "complete": 64 2023-05-31T21:30:49.445 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:30:49.445 INFO:teuthology.orchestra.run.smithi099.stdout: "current_time": "2023-05-31T21:30:49Z" 2023-05-31T21:30:49.445 INFO:teuthology.orchestra.run.smithi099.stdout:} 2023-05-31T21:30:49.447 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 452136274422638824 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000045375", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-05-31T21:28:16.422581Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000031780", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-05-31T21:28:16.501293Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000051671", "next_step_marker": "", "total_entries": 55, "pos": 0, "timestamp": "2023-05-31T21:28:16.476487Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000044178", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2023-05-31T21:27:37.215905Z" } } ] }, "full_sync": { "total": 64, "complete": 64 }, "current_time": "2023-05-31T21:30:49Z" } 2023-05-31T21:30:49.447 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000045375', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-05-31T21:28:16.422581Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000031780', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-05-31T21:28:16.501293Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000051671', 'next_step_marker': '', 'total_entries': 55, 'pos': 0, 'timestamp': '2023-05-31T21:28:16.476487Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000044178', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2023-05-31T21:27:37.215905Z'}}] 2023-05-31T21:30:49.448 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000045515', 1: '00000000000000000000:00000000000000031920', 2: '00000000000000000000:00000000000000051951', 3: '00000000000000000000:00000000000000044178'} 2023-05-31T21:30:49.448 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000045375', 1: '00000000000000000000:00000000000000031780', 2: '00000000000000000000:00000000000000051671', 3: '00000000000000000000:00000000000000044178'} 2023-05-31T21:30:49.448 WARNING:rgw_multi.tests:data of zone a2 behind zone a1: shard=0 master=00000000000000000000:00000000000000045515 target=00000000000000000000:00000000000000045375, shard=1 master=00000000000000000000:00000000000000031920 target=00000000000000000000:00000000000000031780, shard=2 master=00000000000000000000:00000000000000051951 target=00000000000000000000:00000000000000051671 2023-05-31T21:30:54.450 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-31T21:30:54.450 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-31T21:30:54.451 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-05-31T21:30:54.510 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2023-05-31T21:30:54.510 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2023-05-31T21:30:54.579 INFO:teuthology.orchestra.run.smithi099.stdout:{ 2023-05-31T21:30:54.580 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_status": { 2023-05-31T21:30:54.580 INFO:teuthology.orchestra.run.smithi099.stdout: "info": { 2023-05-31T21:30:54.580 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "sync", 2023-05-31T21:30:54.580 INFO:teuthology.orchestra.run.smithi099.stdout: "num_shards": 4, 2023-05-31T21:30:54.580 INFO:teuthology.orchestra.run.smithi099.stdout: "instance_id": 452136274422638824 2023-05-31T21:30:54.581 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:30:54.581 INFO:teuthology.orchestra.run.smithi099.stdout: "markers": [ 2023-05-31T21:30:54.581 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T21:30:54.581 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 0, 2023-05-31T21:30:54.581 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T21:30:54.581 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "incremental-sync", 2023-05-31T21:30:54.581 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "00000000000000000000:00000000000000045375", 2023-05-31T21:30:54.582 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T21:30:54.582 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2023-05-31T21:30:54.582 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T21:30:54.582 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2023-05-31T21:28:16.422581Z" 2023-05-31T21:30:54.582 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T21:30:54.582 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:30:54.582 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T21:30:54.583 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 1, 2023-05-31T21:30:54.583 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T21:30:54.583 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "incremental-sync", 2023-05-31T21:30:54.583 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "00000000000000000000:00000000000000031780", 2023-05-31T21:30:54.583 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T21:30:54.583 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2023-05-31T21:30:54.583 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T21:30:54.584 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2023-05-31T21:28:16.501293Z" 2023-05-31T21:30:54.584 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T21:30:54.584 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:30:54.584 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T21:30:54.584 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 2, 2023-05-31T21:30:54.584 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T21:30:54.585 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "incremental-sync", 2023-05-31T21:30:54.585 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "00000000000000000000:00000000000000051671", 2023-05-31T21:30:54.585 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T21:30:54.585 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 55, 2023-05-31T21:30:54.585 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T21:30:54.585 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2023-05-31T21:28:16.476487Z" 2023-05-31T21:30:54.585 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T21:30:54.586 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:30:54.586 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T21:30:54.586 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 3, 2023-05-31T21:30:54.586 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T21:30:54.586 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "incremental-sync", 2023-05-31T21:30:54.586 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "00000000000000000000:00000000000000044178", 2023-05-31T21:30:54.586 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T21:30:54.587 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 9, 2023-05-31T21:30:54.587 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T21:30:54.587 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2023-05-31T21:27:37.215905Z" 2023-05-31T21:30:54.587 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T21:30:54.587 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T21:30:54.587 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2023-05-31T21:30:54.587 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:30:54.588 INFO:teuthology.orchestra.run.smithi099.stdout: "full_sync": { 2023-05-31T21:30:54.588 INFO:teuthology.orchestra.run.smithi099.stdout: "total": 64, 2023-05-31T21:30:54.588 INFO:teuthology.orchestra.run.smithi099.stdout: "complete": 64 2023-05-31T21:30:54.588 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:30:54.588 INFO:teuthology.orchestra.run.smithi099.stdout: "current_time": "2023-05-31T21:30:54Z" 2023-05-31T21:30:54.588 INFO:teuthology.orchestra.run.smithi099.stdout:} 2023-05-31T21:30:54.589 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 452136274422638824 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000045375", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-05-31T21:28:16.422581Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000031780", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-05-31T21:28:16.501293Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000051671", "next_step_marker": "", "total_entries": 55, "pos": 0, "timestamp": "2023-05-31T21:28:16.476487Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000044178", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2023-05-31T21:27:37.215905Z" } } ] }, "full_sync": { "total": 64, "complete": 64 }, "current_time": "2023-05-31T21:30:54Z" } 2023-05-31T21:30:54.590 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000045375', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-05-31T21:28:16.422581Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000031780', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-05-31T21:28:16.501293Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000051671', 'next_step_marker': '', 'total_entries': 55, 'pos': 0, 'timestamp': '2023-05-31T21:28:16.476487Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000044178', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2023-05-31T21:27:37.215905Z'}}] 2023-05-31T21:30:54.590 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000045515', 1: '00000000000000000000:00000000000000031920', 2: '00000000000000000000:00000000000000051951', 3: '00000000000000000000:00000000000000044178'} 2023-05-31T21:30:54.590 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000045375', 1: '00000000000000000000:00000000000000031780', 2: '00000000000000000000:00000000000000051671', 3: '00000000000000000000:00000000000000044178'} 2023-05-31T21:30:54.591 WARNING:rgw_multi.tests:data of zone a2 behind zone a1: shard=0 master=00000000000000000000:00000000000000045515 target=00000000000000000000:00000000000000045375, shard=1 master=00000000000000000000:00000000000000031920 target=00000000000000000000:00000000000000031780, shard=2 master=00000000000000000000:00000000000000051951 target=00000000000000000000:00000000000000051671 2023-05-31T21:30:59.591 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-31T21:30:59.592 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-31T21:30:59.593 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-05-31T21:30:59.652 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2023-05-31T21:30:59.652 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2023-05-31T21:30:59.732 INFO:teuthology.orchestra.run.smithi099.stdout:{ 2023-05-31T21:30:59.732 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_status": { 2023-05-31T21:30:59.732 INFO:teuthology.orchestra.run.smithi099.stdout: "info": { 2023-05-31T21:30:59.732 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "sync", 2023-05-31T21:30:59.733 INFO:teuthology.orchestra.run.smithi099.stdout: "num_shards": 4, 2023-05-31T21:30:59.733 INFO:teuthology.orchestra.run.smithi099.stdout: "instance_id": 452136274422638824 2023-05-31T21:30:59.733 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:30:59.733 INFO:teuthology.orchestra.run.smithi099.stdout: "markers": [ 2023-05-31T21:30:59.733 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T21:30:59.733 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 0, 2023-05-31T21:30:59.733 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T21:30:59.734 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "incremental-sync", 2023-05-31T21:30:59.734 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "00000000000000000000:00000000000000045375", 2023-05-31T21:30:59.734 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T21:30:59.734 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2023-05-31T21:30:59.734 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T21:30:59.734 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2023-05-31T21:28:16.422581Z" 2023-05-31T21:30:59.735 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T21:30:59.735 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:30:59.735 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T21:30:59.735 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 1, 2023-05-31T21:30:59.735 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T21:30:59.735 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "incremental-sync", 2023-05-31T21:30:59.735 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "00000000000000000000:00000000000000031780", 2023-05-31T21:30:59.736 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T21:30:59.736 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2023-05-31T21:30:59.736 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T21:30:59.736 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2023-05-31T21:28:16.501293Z" 2023-05-31T21:30:59.736 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T21:30:59.736 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:30:59.736 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T21:30:59.737 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 2, 2023-05-31T21:30:59.737 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T21:30:59.737 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "incremental-sync", 2023-05-31T21:30:59.737 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "00000000000000000000:00000000000000051671", 2023-05-31T21:30:59.737 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T21:30:59.737 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 55, 2023-05-31T21:30:59.737 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T21:30:59.738 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2023-05-31T21:28:16.476487Z" 2023-05-31T21:30:59.738 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T21:30:59.738 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:30:59.738 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T21:30:59.738 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 3, 2023-05-31T21:30:59.738 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T21:30:59.738 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "incremental-sync", 2023-05-31T21:30:59.739 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "00000000000000000000:00000000000000044178", 2023-05-31T21:30:59.739 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T21:30:59.739 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 9, 2023-05-31T21:30:59.739 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T21:30:59.739 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2023-05-31T21:27:37.215905Z" 2023-05-31T21:30:59.739 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T21:30:59.739 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T21:30:59.740 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2023-05-31T21:30:59.740 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:30:59.740 INFO:teuthology.orchestra.run.smithi099.stdout: "full_sync": { 2023-05-31T21:30:59.740 INFO:teuthology.orchestra.run.smithi099.stdout: "total": 64, 2023-05-31T21:30:59.740 INFO:teuthology.orchestra.run.smithi099.stdout: "complete": 64 2023-05-31T21:30:59.740 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:30:59.741 INFO:teuthology.orchestra.run.smithi099.stdout: "current_time": "2023-05-31T21:30:59Z" 2023-05-31T21:30:59.741 INFO:teuthology.orchestra.run.smithi099.stdout:} 2023-05-31T21:30:59.742 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 452136274422638824 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000045375", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-05-31T21:28:16.422581Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000031780", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-05-31T21:28:16.501293Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000051671", "next_step_marker": "", "total_entries": 55, "pos": 0, "timestamp": "2023-05-31T21:28:16.476487Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000044178", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2023-05-31T21:27:37.215905Z" } } ] }, "full_sync": { "total": 64, "complete": 64 }, "current_time": "2023-05-31T21:30:59Z" } 2023-05-31T21:30:59.742 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000045375', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-05-31T21:28:16.422581Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000031780', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-05-31T21:28:16.501293Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000051671', 'next_step_marker': '', 'total_entries': 55, 'pos': 0, 'timestamp': '2023-05-31T21:28:16.476487Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000044178', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2023-05-31T21:27:37.215905Z'}}] 2023-05-31T21:30:59.743 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000045515', 1: '00000000000000000000:00000000000000031920', 2: '00000000000000000000:00000000000000051951', 3: '00000000000000000000:00000000000000044178'} 2023-05-31T21:30:59.743 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000045375', 1: '00000000000000000000:00000000000000031780', 2: '00000000000000000000:00000000000000051671', 3: '00000000000000000000:00000000000000044178'} 2023-05-31T21:30:59.743 WARNING:rgw_multi.tests:data of zone a2 behind zone a1: shard=0 master=00000000000000000000:00000000000000045515 target=00000000000000000000:00000000000000045375, shard=1 master=00000000000000000000:00000000000000031920 target=00000000000000000000:00000000000000031780, shard=2 master=00000000000000000000:00000000000000051951 target=00000000000000000000:00000000000000051671 2023-05-31T21:31:04.745 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-31T21:31:04.745 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-31T21:31:04.746 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-05-31T21:31:04.806 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2023-05-31T21:31:04.806 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2023-05-31T21:31:04.859 INFO:teuthology.orchestra.run.smithi099.stdout:{ 2023-05-31T21:31:04.859 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_status": { 2023-05-31T21:31:04.860 INFO:teuthology.orchestra.run.smithi099.stdout: "info": { 2023-05-31T21:31:04.860 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "sync", 2023-05-31T21:31:04.860 INFO:teuthology.orchestra.run.smithi099.stdout: "num_shards": 4, 2023-05-31T21:31:04.860 INFO:teuthology.orchestra.run.smithi099.stdout: "instance_id": 452136274422638824 2023-05-31T21:31:04.860 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:31:04.860 INFO:teuthology.orchestra.run.smithi099.stdout: "markers": [ 2023-05-31T21:31:04.861 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T21:31:04.861 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 0, 2023-05-31T21:31:04.861 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T21:31:04.861 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "incremental-sync", 2023-05-31T21:31:04.861 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "00000000000000000000:00000000000000045375", 2023-05-31T21:31:04.861 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T21:31:04.861 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2023-05-31T21:31:04.862 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T21:31:04.862 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2023-05-31T21:28:16.422581Z" 2023-05-31T21:31:04.862 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T21:31:04.862 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:31:04.862 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T21:31:04.862 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 1, 2023-05-31T21:31:04.862 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T21:31:04.863 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "incremental-sync", 2023-05-31T21:31:04.863 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "00000000000000000000:00000000000000031780", 2023-05-31T21:31:04.863 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T21:31:04.863 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2023-05-31T21:31:04.863 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T21:31:04.863 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2023-05-31T21:28:16.501293Z" 2023-05-31T21:31:04.863 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T21:31:04.864 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:31:04.864 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T21:31:04.864 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 2, 2023-05-31T21:31:04.864 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T21:31:04.864 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "incremental-sync", 2023-05-31T21:31:04.864 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "00000000000000000000:00000000000000051671", 2023-05-31T21:31:04.865 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T21:31:04.865 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 55, 2023-05-31T21:31:04.865 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T21:31:04.865 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2023-05-31T21:28:16.476487Z" 2023-05-31T21:31:04.865 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T21:31:04.865 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:31:04.865 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T21:31:04.866 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 3, 2023-05-31T21:31:04.866 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T21:31:04.866 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "incremental-sync", 2023-05-31T21:31:04.866 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "00000000000000000000:00000000000000044178", 2023-05-31T21:31:04.866 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T21:31:04.866 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 9, 2023-05-31T21:31:04.866 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T21:31:04.867 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2023-05-31T21:27:37.215905Z" 2023-05-31T21:31:04.867 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T21:31:04.867 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T21:31:04.867 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2023-05-31T21:31:04.867 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:31:04.867 INFO:teuthology.orchestra.run.smithi099.stdout: "full_sync": { 2023-05-31T21:31:04.867 INFO:teuthology.orchestra.run.smithi099.stdout: "total": 64, 2023-05-31T21:31:04.868 INFO:teuthology.orchestra.run.smithi099.stdout: "complete": 64 2023-05-31T21:31:04.868 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:31:04.868 INFO:teuthology.orchestra.run.smithi099.stdout: "current_time": "2023-05-31T21:31:04Z" 2023-05-31T21:31:04.868 INFO:teuthology.orchestra.run.smithi099.stdout:} 2023-05-31T21:31:04.869 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 452136274422638824 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000045375", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-05-31T21:28:16.422581Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000031780", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-05-31T21:28:16.501293Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000051671", "next_step_marker": "", "total_entries": 55, "pos": 0, "timestamp": "2023-05-31T21:28:16.476487Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000044178", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2023-05-31T21:27:37.215905Z" } } ] }, "full_sync": { "total": 64, "complete": 64 }, "current_time": "2023-05-31T21:31:04Z" } 2023-05-31T21:31:04.869 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000045375', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-05-31T21:28:16.422581Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000031780', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-05-31T21:28:16.501293Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000051671', 'next_step_marker': '', 'total_entries': 55, 'pos': 0, 'timestamp': '2023-05-31T21:28:16.476487Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000044178', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2023-05-31T21:27:37.215905Z'}}] 2023-05-31T21:31:04.870 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000045515', 1: '00000000000000000000:00000000000000031920', 2: '00000000000000000000:00000000000000051951', 3: '00000000000000000000:00000000000000044178'} 2023-05-31T21:31:04.870 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000045375', 1: '00000000000000000000:00000000000000031780', 2: '00000000000000000000:00000000000000051671', 3: '00000000000000000000:00000000000000044178'} 2023-05-31T21:31:04.870 WARNING:rgw_multi.tests:data of zone a2 behind zone a1: shard=0 master=00000000000000000000:00000000000000045515 target=00000000000000000000:00000000000000045375, shard=1 master=00000000000000000000:00000000000000031920 target=00000000000000000000:00000000000000031780, shard=2 master=00000000000000000000:00000000000000051951 target=00000000000000000000:00000000000000051671 2023-05-31T21:31:09.872 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-31T21:31:09.872 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-31T21:31:09.873 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-05-31T21:31:09.932 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2023-05-31T21:31:09.932 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2023-05-31T21:31:10.010 INFO:teuthology.orchestra.run.smithi099.stdout:{ 2023-05-31T21:31:10.010 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_status": { 2023-05-31T21:31:10.010 INFO:teuthology.orchestra.run.smithi099.stdout: "info": { 2023-05-31T21:31:10.010 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "sync", 2023-05-31T21:31:10.011 INFO:teuthology.orchestra.run.smithi099.stdout: "num_shards": 4, 2023-05-31T21:31:10.011 INFO:teuthology.orchestra.run.smithi099.stdout: "instance_id": 452136274422638824 2023-05-31T21:31:10.011 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:31:10.011 INFO:teuthology.orchestra.run.smithi099.stdout: "markers": [ 2023-05-31T21:31:10.011 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T21:31:10.011 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 0, 2023-05-31T21:31:10.012 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T21:31:10.012 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "incremental-sync", 2023-05-31T21:31:10.012 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "00000000000000000000:00000000000000045375", 2023-05-31T21:31:10.012 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T21:31:10.012 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2023-05-31T21:31:10.012 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T21:31:10.012 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2023-05-31T21:28:16.422581Z" 2023-05-31T21:31:10.013 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T21:31:10.013 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:31:10.013 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T21:31:10.013 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 1, 2023-05-31T21:31:10.013 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T21:31:10.013 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "incremental-sync", 2023-05-31T21:31:10.013 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "00000000000000000000:00000000000000031780", 2023-05-31T21:31:10.014 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T21:31:10.014 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2023-05-31T21:31:10.014 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T21:31:10.014 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2023-05-31T21:28:16.501293Z" 2023-05-31T21:31:10.014 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T21:31:10.014 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:31:10.014 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T21:31:10.015 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 2, 2023-05-31T21:31:10.015 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T21:31:10.015 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "incremental-sync", 2023-05-31T21:31:10.015 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "00000000000000000000:00000000000000051671", 2023-05-31T21:31:10.015 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T21:31:10.015 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 55, 2023-05-31T21:31:10.015 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T21:31:10.016 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2023-05-31T21:28:16.476487Z" 2023-05-31T21:31:10.016 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T21:31:10.016 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:31:10.016 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T21:31:10.016 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 3, 2023-05-31T21:31:10.016 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T21:31:10.016 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "incremental-sync", 2023-05-31T21:31:10.017 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "00000000000000000000:00000000000000044178", 2023-05-31T21:31:10.017 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T21:31:10.017 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 9, 2023-05-31T21:31:10.017 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T21:31:10.017 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2023-05-31T21:27:37.215905Z" 2023-05-31T21:31:10.017 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T21:31:10.017 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T21:31:10.018 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2023-05-31T21:31:10.018 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:31:10.018 INFO:teuthology.orchestra.run.smithi099.stdout: "full_sync": { 2023-05-31T21:31:10.018 INFO:teuthology.orchestra.run.smithi099.stdout: "total": 64, 2023-05-31T21:31:10.018 INFO:teuthology.orchestra.run.smithi099.stdout: "complete": 64 2023-05-31T21:31:10.018 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:31:10.019 INFO:teuthology.orchestra.run.smithi099.stdout: "current_time": "2023-05-31T21:31:09Z" 2023-05-31T21:31:10.019 INFO:teuthology.orchestra.run.smithi099.stdout:} 2023-05-31T21:31:10.020 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 452136274422638824 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000045375", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-05-31T21:28:16.422581Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000031780", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-05-31T21:28:16.501293Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000051671", "next_step_marker": "", "total_entries": 55, "pos": 0, "timestamp": "2023-05-31T21:28:16.476487Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000044178", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2023-05-31T21:27:37.215905Z" } } ] }, "full_sync": { "total": 64, "complete": 64 }, "current_time": "2023-05-31T21:31:09Z" } 2023-05-31T21:31:10.020 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000045375', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-05-31T21:28:16.422581Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000031780', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-05-31T21:28:16.501293Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000051671', 'next_step_marker': '', 'total_entries': 55, 'pos': 0, 'timestamp': '2023-05-31T21:28:16.476487Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000044178', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2023-05-31T21:27:37.215905Z'}}] 2023-05-31T21:31:10.020 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000045515', 1: '00000000000000000000:00000000000000031920', 2: '00000000000000000000:00000000000000051951', 3: '00000000000000000000:00000000000000044178'} 2023-05-31T21:31:10.021 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000045375', 1: '00000000000000000000:00000000000000031780', 2: '00000000000000000000:00000000000000051671', 3: '00000000000000000000:00000000000000044178'} 2023-05-31T21:31:10.021 WARNING:rgw_multi.tests:data of zone a2 behind zone a1: shard=0 master=00000000000000000000:00000000000000045515 target=00000000000000000000:00000000000000045375, shard=1 master=00000000000000000000:00000000000000031920 target=00000000000000000000:00000000000000031780, shard=2 master=00000000000000000000:00000000000000051951 target=00000000000000000000:00000000000000051671 2023-05-31T21:31:15.022 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-31T21:31:15.023 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-31T21:31:15.024 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-05-31T21:31:15.083 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2023-05-31T21:31:15.084 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2023-05-31T21:31:15.158 INFO:teuthology.orchestra.run.smithi099.stdout:{ 2023-05-31T21:31:15.158 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_status": { 2023-05-31T21:31:15.158 INFO:teuthology.orchestra.run.smithi099.stdout: "info": { 2023-05-31T21:31:15.158 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "sync", 2023-05-31T21:31:15.159 INFO:teuthology.orchestra.run.smithi099.stdout: "num_shards": 4, 2023-05-31T21:31:15.159 INFO:teuthology.orchestra.run.smithi099.stdout: "instance_id": 452136274422638824 2023-05-31T21:31:15.159 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:31:15.159 INFO:teuthology.orchestra.run.smithi099.stdout: "markers": [ 2023-05-31T21:31:15.159 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T21:31:15.159 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 0, 2023-05-31T21:31:15.159 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T21:31:15.160 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "incremental-sync", 2023-05-31T21:31:15.160 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "00000000000000000000:00000000000000045375", 2023-05-31T21:31:15.160 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T21:31:15.160 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2023-05-31T21:31:15.160 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T21:31:15.160 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2023-05-31T21:28:16.422581Z" 2023-05-31T21:31:15.161 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T21:31:15.161 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:31:15.161 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T21:31:15.161 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 1, 2023-05-31T21:31:15.161 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T21:31:15.161 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "incremental-sync", 2023-05-31T21:31:15.162 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "00000000000000000000:00000000000000031780", 2023-05-31T21:31:15.162 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T21:31:15.162 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2023-05-31T21:31:15.162 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T21:31:15.162 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2023-05-31T21:28:16.501293Z" 2023-05-31T21:31:15.162 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T21:31:15.162 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:31:15.163 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T21:31:15.163 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 2, 2023-05-31T21:31:15.163 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T21:31:15.163 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "incremental-sync", 2023-05-31T21:31:15.163 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "00000000000000000000:00000000000000051671", 2023-05-31T21:31:15.163 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T21:31:15.163 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 55, 2023-05-31T21:31:15.164 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T21:31:15.164 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2023-05-31T21:28:16.476487Z" 2023-05-31T21:31:15.164 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T21:31:15.164 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:31:15.164 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T21:31:15.165 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 3, 2023-05-31T21:31:15.165 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T21:31:15.165 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "incremental-sync", 2023-05-31T21:31:15.165 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "00000000000000000000:00000000000000044178", 2023-05-31T21:31:15.165 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T21:31:15.165 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 9, 2023-05-31T21:31:15.165 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T21:31:15.166 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2023-05-31T21:27:37.215905Z" 2023-05-31T21:31:15.166 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T21:31:15.166 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T21:31:15.166 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2023-05-31T21:31:15.166 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:31:15.166 INFO:teuthology.orchestra.run.smithi099.stdout: "full_sync": { 2023-05-31T21:31:15.166 INFO:teuthology.orchestra.run.smithi099.stdout: "total": 64, 2023-05-31T21:31:15.167 INFO:teuthology.orchestra.run.smithi099.stdout: "complete": 64 2023-05-31T21:31:15.167 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:31:15.167 INFO:teuthology.orchestra.run.smithi099.stdout: "current_time": "2023-05-31T21:31:15Z" 2023-05-31T21:31:15.167 INFO:teuthology.orchestra.run.smithi099.stdout:} 2023-05-31T21:31:15.168 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 452136274422638824 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000045375", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-05-31T21:28:16.422581Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000031780", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-05-31T21:28:16.501293Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000051671", "next_step_marker": "", "total_entries": 55, "pos": 0, "timestamp": "2023-05-31T21:28:16.476487Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000044178", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2023-05-31T21:27:37.215905Z" } } ] }, "full_sync": { "total": 64, "complete": 64 }, "current_time": "2023-05-31T21:31:15Z" } 2023-05-31T21:31:15.168 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000045375', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-05-31T21:28:16.422581Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000031780', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-05-31T21:28:16.501293Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000051671', 'next_step_marker': '', 'total_entries': 55, 'pos': 0, 'timestamp': '2023-05-31T21:28:16.476487Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000044178', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2023-05-31T21:27:37.215905Z'}}] 2023-05-31T21:31:15.169 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000045515', 1: '00000000000000000000:00000000000000031920', 2: '00000000000000000000:00000000000000051951', 3: '00000000000000000000:00000000000000044178'} 2023-05-31T21:31:15.169 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000045375', 1: '00000000000000000000:00000000000000031780', 2: '00000000000000000000:00000000000000051671', 3: '00000000000000000000:00000000000000044178'} 2023-05-31T21:31:15.170 WARNING:rgw_multi.tests:data of zone a2 behind zone a1: shard=0 master=00000000000000000000:00000000000000045515 target=00000000000000000000:00000000000000045375, shard=1 master=00000000000000000000:00000000000000031920 target=00000000000000000000:00000000000000031780, shard=2 master=00000000000000000000:00000000000000051951 target=00000000000000000000:00000000000000051671 2023-05-31T21:31:20.170 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-31T21:31:20.171 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-31T21:31:20.171 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-05-31T21:31:20.231 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2023-05-31T21:31:20.231 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2023-05-31T21:31:20.302 INFO:teuthology.orchestra.run.smithi099.stdout:{ 2023-05-31T21:31:20.302 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_status": { 2023-05-31T21:31:20.302 INFO:teuthology.orchestra.run.smithi099.stdout: "info": { 2023-05-31T21:31:20.302 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "sync", 2023-05-31T21:31:20.303 INFO:teuthology.orchestra.run.smithi099.stdout: "num_shards": 4, 2023-05-31T21:31:20.303 INFO:teuthology.orchestra.run.smithi099.stdout: "instance_id": 452136274422638824 2023-05-31T21:31:20.303 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:31:20.303 INFO:teuthology.orchestra.run.smithi099.stdout: "markers": [ 2023-05-31T21:31:20.303 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T21:31:20.303 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 0, 2023-05-31T21:31:20.303 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T21:31:20.304 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "incremental-sync", 2023-05-31T21:31:20.304 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "00000000000000000000:00000000000000045375", 2023-05-31T21:31:20.304 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T21:31:20.304 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2023-05-31T21:31:20.304 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T21:31:20.304 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2023-05-31T21:28:16.422581Z" 2023-05-31T21:31:20.305 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T21:31:20.305 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:31:20.305 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T21:31:20.305 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 1, 2023-05-31T21:31:20.305 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T21:31:20.305 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "incremental-sync", 2023-05-31T21:31:20.305 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "00000000000000000000:00000000000000031780", 2023-05-31T21:31:20.306 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T21:31:20.306 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2023-05-31T21:31:20.306 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T21:31:20.306 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2023-05-31T21:28:16.501293Z" 2023-05-31T21:31:20.306 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T21:31:20.307 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:31:20.307 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T21:31:20.307 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 2, 2023-05-31T21:31:20.307 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T21:31:20.307 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "incremental-sync", 2023-05-31T21:31:20.308 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "00000000000000000000:00000000000000051671", 2023-05-31T21:31:20.308 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T21:31:20.308 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 55, 2023-05-31T21:31:20.308 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T21:31:20.308 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2023-05-31T21:28:16.476487Z" 2023-05-31T21:31:20.309 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T21:31:20.309 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:31:20.309 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T21:31:20.309 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 3, 2023-05-31T21:31:20.309 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T21:31:20.309 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "incremental-sync", 2023-05-31T21:31:20.309 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "00000000000000000000:00000000000000044178", 2023-05-31T21:31:20.310 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T21:31:20.310 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 9, 2023-05-31T21:31:20.310 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T21:31:20.310 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2023-05-31T21:27:37.215905Z" 2023-05-31T21:31:20.310 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T21:31:20.310 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T21:31:20.310 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2023-05-31T21:31:20.311 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:31:20.311 INFO:teuthology.orchestra.run.smithi099.stdout: "full_sync": { 2023-05-31T21:31:20.311 INFO:teuthology.orchestra.run.smithi099.stdout: "total": 64, 2023-05-31T21:31:20.311 INFO:teuthology.orchestra.run.smithi099.stdout: "complete": 64 2023-05-31T21:31:20.311 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:31:20.312 INFO:teuthology.orchestra.run.smithi099.stdout: "current_time": "2023-05-31T21:31:20Z" 2023-05-31T21:31:20.312 INFO:teuthology.orchestra.run.smithi099.stdout:} 2023-05-31T21:31:20.313 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 452136274422638824 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000045375", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-05-31T21:28:16.422581Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000031780", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-05-31T21:28:16.501293Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000051671", "next_step_marker": "", "total_entries": 55, "pos": 0, "timestamp": "2023-05-31T21:28:16.476487Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000044178", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2023-05-31T21:27:37.215905Z" } } ] }, "full_sync": { "total": 64, "complete": 64 }, "current_time": "2023-05-31T21:31:20Z" } 2023-05-31T21:31:20.313 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000045375', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-05-31T21:28:16.422581Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000031780', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-05-31T21:28:16.501293Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000051671', 'next_step_marker': '', 'total_entries': 55, 'pos': 0, 'timestamp': '2023-05-31T21:28:16.476487Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000044178', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2023-05-31T21:27:37.215905Z'}}] 2023-05-31T21:31:20.314 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000045515', 1: '00000000000000000000:00000000000000031920', 2: '00000000000000000000:00000000000000051951', 3: '00000000000000000000:00000000000000044178'} 2023-05-31T21:31:20.314 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000045375', 1: '00000000000000000000:00000000000000031780', 2: '00000000000000000000:00000000000000051671', 3: '00000000000000000000:00000000000000044178'} 2023-05-31T21:31:20.314 WARNING:rgw_multi.tests:data of zone a2 behind zone a1: shard=0 master=00000000000000000000:00000000000000045515 target=00000000000000000000:00000000000000045375, shard=1 master=00000000000000000000:00000000000000031920 target=00000000000000000000:00000000000000031780, shard=2 master=00000000000000000000:00000000000000051951 target=00000000000000000000:00000000000000051671 2023-05-31T21:31:25.315 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-31T21:31:25.316 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-31T21:31:25.316 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-05-31T21:31:25.375 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2023-05-31T21:31:25.375 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2023-05-31T21:31:25.449 INFO:teuthology.orchestra.run.smithi099.stdout:{ 2023-05-31T21:31:25.450 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_status": { 2023-05-31T21:31:25.450 INFO:teuthology.orchestra.run.smithi099.stdout: "info": { 2023-05-31T21:31:25.450 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "sync", 2023-05-31T21:31:25.450 INFO:teuthology.orchestra.run.smithi099.stdout: "num_shards": 4, 2023-05-31T21:31:25.450 INFO:teuthology.orchestra.run.smithi099.stdout: "instance_id": 452136274422638824 2023-05-31T21:31:25.450 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:31:25.451 INFO:teuthology.orchestra.run.smithi099.stdout: "markers": [ 2023-05-31T21:31:25.451 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T21:31:25.451 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 0, 2023-05-31T21:31:25.451 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T21:31:25.451 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "incremental-sync", 2023-05-31T21:31:25.451 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "00000000000000000000:00000000000000045515", 2023-05-31T21:31:25.451 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T21:31:25.452 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2023-05-31T21:31:25.452 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T21:31:25.452 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2023-05-31T21:29:57.506834Z" 2023-05-31T21:31:25.452 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T21:31:25.452 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:31:25.452 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T21:31:25.453 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 1, 2023-05-31T21:31:25.453 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T21:31:25.453 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "incremental-sync", 2023-05-31T21:31:25.453 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "00000000000000000000:00000000000000031920", 2023-05-31T21:31:25.453 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T21:31:25.453 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2023-05-31T21:31:25.453 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T21:31:25.454 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2023-05-31T21:29:57.583636Z" 2023-05-31T21:31:25.454 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T21:31:25.454 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:31:25.454 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T21:31:25.454 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 2, 2023-05-31T21:31:25.454 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T21:31:25.454 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "incremental-sync", 2023-05-31T21:31:25.455 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "00000000000000000000:00000000000000051951", 2023-05-31T21:31:25.455 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T21:31:25.455 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 55, 2023-05-31T21:31:25.455 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T21:31:25.455 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2023-05-31T21:29:57.526899Z" 2023-05-31T21:31:25.455 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T21:31:25.455 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:31:25.456 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T21:31:25.456 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 3, 2023-05-31T21:31:25.456 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T21:31:25.456 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "incremental-sync", 2023-05-31T21:31:25.456 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "00000000000000000000:00000000000000044178", 2023-05-31T21:31:25.456 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T21:31:25.457 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 9, 2023-05-31T21:31:25.457 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T21:31:25.457 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2023-05-31T21:27:37.215905Z" 2023-05-31T21:31:25.457 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T21:31:25.457 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T21:31:25.457 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2023-05-31T21:31:25.457 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:31:25.458 INFO:teuthology.orchestra.run.smithi099.stdout: "full_sync": { 2023-05-31T21:31:25.458 INFO:teuthology.orchestra.run.smithi099.stdout: "total": 64, 2023-05-31T21:31:25.458 INFO:teuthology.orchestra.run.smithi099.stdout: "complete": 64 2023-05-31T21:31:25.458 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:31:25.458 INFO:teuthology.orchestra.run.smithi099.stdout: "current_time": "2023-05-31T21:31:25Z" 2023-05-31T21:31:25.458 INFO:teuthology.orchestra.run.smithi099.stdout:} 2023-05-31T21:31:25.459 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 452136274422638824 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000045515", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-05-31T21:29:57.506834Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000031920", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-05-31T21:29:57.583636Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000051951", "next_step_marker": "", "total_entries": 55, "pos": 0, "timestamp": "2023-05-31T21:29:57.526899Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000044178", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2023-05-31T21:27:37.215905Z" } } ] }, "full_sync": { "total": 64, "complete": 64 }, "current_time": "2023-05-31T21:31:25Z" } 2023-05-31T21:31:25.460 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000045515', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-05-31T21:29:57.506834Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000031920', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-05-31T21:29:57.583636Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000051951', 'next_step_marker': '', 'total_entries': 55, 'pos': 0, 'timestamp': '2023-05-31T21:29:57.526899Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000044178', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2023-05-31T21:27:37.215905Z'}}] 2023-05-31T21:31:25.460 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000045515', 1: '00000000000000000000:00000000000000031920', 2: '00000000000000000000:00000000000000051951', 3: '00000000000000000000:00000000000000044178'} 2023-05-31T21:31:25.460 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000045515', 1: '00000000000000000000:00000000000000031920', 2: '00000000000000000000:00000000000000051951', 3: '00000000000000000000:00000000000000044178'} 2023-05-31T21:31:25.461 INFO:rgw_multi.tests:finished data checkpoint for target_zone=a2 source_zone=a1 2023-05-31T21:31:25.461 DEBUG:boto:path=/pklxkd-73/ 2023-05-31T21:31:25.462 DEBUG:boto:auth_path=/pklxkd-73/ 2023-05-31T21:31:25.462 DEBUG:boto:Method: HEAD 2023-05-31T21:31:25.462 DEBUG:boto:Path: /pklxkd-73/ 2023-05-31T21:31:25.463 DEBUG:boto:Data: 2023-05-31T21:31:25.463 DEBUG:boto:Headers: {} 2023-05-31T21:31:25.463 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T21:31:25.463 DEBUG:boto:Port: 8001 2023-05-31T21:31:25.464 DEBUG:boto:Params: {} 2023-05-31T21:31:25.464 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8001} 2023-05-31T21:31:25.465 DEBUG:boto:Token: None 2023-05-31T21:31:25.465 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 21:31:25 GMT /pklxkd-73/ 2023-05-31T21:31:25.466 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:cYK3RC61Sar3CdxyPNRXpbeWjvE= 2023-05-31T21:31:25.466 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 21:31:25 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:cYK3RC61Sar3CdxyPNRXpbeWjvE=', 'Content-Length': '0'} 2023-05-31T21:31:25.470 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '4'), ('X-RGW-Bytes-Used', '24'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000bc957d57825cd06b-006477bcad-295a-a2'), ('Content-Length', '0'), ('Date', 'Wed, 31 May 2023 21:31:25 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:31:25.471 DEBUG:boto:path=/pklxkd-73/obj5 2023-05-31T21:31:25.471 DEBUG:boto:auth_path=/pklxkd-73/obj5 2023-05-31T21:31:25.472 DEBUG:boto:Method: HEAD 2023-05-31T21:31:25.472 DEBUG:boto:Path: /pklxkd-73/obj5 2023-05-31T21:31:25.472 DEBUG:boto:Data: 2023-05-31T21:31:25.473 DEBUG:boto:Headers: {} 2023-05-31T21:31:25.473 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T21:31:25.474 DEBUG:boto:Port: 8001 2023-05-31T21:31:25.474 DEBUG:boto:Params: {} 2023-05-31T21:31:25.474 DEBUG:boto:Token: None 2023-05-31T21:31:25.475 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 21:31:25 GMT /pklxkd-73/obj5 2023-05-31T21:31:25.475 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:N6FKCXiG2bbfLpMfh4xP9yHtt1M= 2023-05-31T21:31:25.476 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 21:31:25 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:N6FKCXiG2bbfLpMfh4xP9yHtt1M=', 'Content-Length': '0'} 2023-05-31T21:31:25.477 DEBUG:boto:Response headers: [('Content-Length', '219'), ('x-amz-request-id', 'tx000002a03875ef34b6cda-006477bcad-295a-a2'), ('Accept-Ranges', 'bytes'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 21:31:25 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:31:25.477 DEBUG:boto:path=/pklxkd-73/obj6 2023-05-31T21:31:25.478 DEBUG:boto:auth_path=/pklxkd-73/obj6 2023-05-31T21:31:25.478 DEBUG:boto:Method: HEAD 2023-05-31T21:31:25.479 DEBUG:boto:Path: /pklxkd-73/obj6 2023-05-31T21:31:25.479 DEBUG:boto:Data: 2023-05-31T21:31:25.479 DEBUG:boto:Headers: {} 2023-05-31T21:31:25.480 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T21:31:25.480 DEBUG:boto:Port: 8001 2023-05-31T21:31:25.480 DEBUG:boto:Params: {} 2023-05-31T21:31:25.481 DEBUG:boto:Token: None 2023-05-31T21:31:25.481 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 21:31:25 GMT /pklxkd-73/obj6 2023-05-31T21:31:25.482 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:WgJD09mmQ3RaO79aaziMNsPqJ4s= 2023-05-31T21:31:25.482 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 21:31:25 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:WgJD09mmQ3RaO79aaziMNsPqJ4s=', 'Content-Length': '0'} 2023-05-31T21:31:25.483 DEBUG:boto:Response headers: [('Content-Length', '219'), ('x-amz-request-id', 'tx00000f681d5f2b207003b-006477bcad-295a-a2'), ('Accept-Ranges', 'bytes'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 21:31:25 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:31:25.484 DEBUG:boto:path=/pklxkd-73/obj7 2023-05-31T21:31:25.484 DEBUG:boto:auth_path=/pklxkd-73/obj7 2023-05-31T21:31:25.485 DEBUG:boto:Method: HEAD 2023-05-31T21:31:25.485 DEBUG:boto:Path: /pklxkd-73/obj7 2023-05-31T21:31:25.485 DEBUG:boto:Data: 2023-05-31T21:31:25.486 DEBUG:boto:Headers: {} 2023-05-31T21:31:25.486 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T21:31:25.486 DEBUG:boto:Port: 8001 2023-05-31T21:31:25.487 DEBUG:boto:Params: {} 2023-05-31T21:31:25.487 DEBUG:boto:Token: None 2023-05-31T21:31:25.487 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 21:31:25 GMT /pklxkd-73/obj7 2023-05-31T21:31:25.488 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:ooZUcfP99buMy0+sJJzFhS8cZ2U= 2023-05-31T21:31:25.488 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 21:31:25 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:ooZUcfP99buMy0+sJJzFhS8cZ2U=', 'Content-Length': '0'} 2023-05-31T21:31:25.490 DEBUG:boto:Response headers: [('Content-Length', '219'), ('x-amz-request-id', 'tx00000ad3c678afc74ce10-006477bcad-295a-a2'), ('Accept-Ranges', 'bytes'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 21:31:25 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:31:25.490 DEBUG:boto:path=/pklxkd-73/obj8 2023-05-31T21:31:25.491 DEBUG:boto:auth_path=/pklxkd-73/obj8 2023-05-31T21:31:25.491 DEBUG:boto:Method: HEAD 2023-05-31T21:31:25.491 DEBUG:boto:Path: /pklxkd-73/obj8 2023-05-31T21:31:25.492 DEBUG:boto:Data: 2023-05-31T21:31:25.492 DEBUG:boto:Headers: {} 2023-05-31T21:31:25.492 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T21:31:25.493 DEBUG:boto:Port: 8001 2023-05-31T21:31:25.493 DEBUG:boto:Params: {} 2023-05-31T21:31:25.493 DEBUG:boto:Token: None 2023-05-31T21:31:25.494 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 21:31:25 GMT /pklxkd-73/obj8 2023-05-31T21:31:25.495 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:Lk9j/QesqAD3pRQVIyHpc6MdWXo= 2023-05-31T21:31:25.495 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 21:31:25 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:Lk9j/QesqAD3pRQVIyHpc6MdWXo=', 'Content-Length': '0'} 2023-05-31T21:31:25.496 DEBUG:boto:Response headers: [('Content-Length', '219'), ('x-amz-request-id', 'tx0000069f425a68fd8ca8d-006477bcad-295a-a2'), ('Accept-Ranges', 'bytes'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 21:31:25 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:31:25.497 DEBUG:boto:path=/pklxkd-72/ 2023-05-31T21:31:25.498 DEBUG:boto:auth_path=/pklxkd-72/ 2023-05-31T21:31:25.498 DEBUG:boto:Method: HEAD 2023-05-31T21:31:25.499 DEBUG:boto:Path: /pklxkd-72/ 2023-05-31T21:31:25.500 DEBUG:boto:Data: 2023-05-31T21:31:25.500 DEBUG:boto:Headers: {} 2023-05-31T21:31:25.501 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T21:31:25.502 DEBUG:boto:Port: 8001 2023-05-31T21:31:25.502 DEBUG:boto:Params: {} 2023-05-31T21:31:25.503 DEBUG:boto:Token: None 2023-05-31T21:31:25.504 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 21:31:25 GMT /pklxkd-72/ 2023-05-31T21:31:25.505 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:Xe9FH9UqMdZbBaujXAKWuxKWVTE= 2023-05-31T21:31:25.505 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 21:31:25 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:Xe9FH9UqMdZbBaujXAKWuxKWVTE=', 'Content-Length': '0'} 2023-05-31T21:31:25.507 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '7'), ('X-RGW-Bytes-Used', '42'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000006c7d2125e423be82-006477bcad-295a-a2'), ('Content-Length', '0'), ('Date', 'Wed, 31 May 2023 21:31:25 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:31:25.508 DEBUG:boto:path=/pklxkd-72/obj7 2023-05-31T21:31:25.508 DEBUG:boto:auth_path=/pklxkd-72/obj7 2023-05-31T21:31:25.509 DEBUG:boto:Method: HEAD 2023-05-31T21:31:25.509 DEBUG:boto:Path: /pklxkd-72/obj7 2023-05-31T21:31:25.509 DEBUG:boto:Data: 2023-05-31T21:31:25.510 DEBUG:boto:Headers: {} 2023-05-31T21:31:25.510 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T21:31:25.510 DEBUG:boto:Port: 8001 2023-05-31T21:31:25.511 DEBUG:boto:Params: {} 2023-05-31T21:31:25.511 DEBUG:boto:Token: None 2023-05-31T21:31:25.511 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 21:31:25 GMT /pklxkd-72/obj7 2023-05-31T21:31:25.512 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:Z68UmYLQOCkNnU1WTAXLIMzn71g= 2023-05-31T21:31:25.512 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 21:31:25 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:Z68UmYLQOCkNnU1WTAXLIMzn71g=', 'Content-Length': '0'} 2023-05-31T21:31:25.514 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 21:29:57 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000cecdf4b02bc22f7d-006477bcad-295a-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 21:31:25 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:31:25.515 DEBUG:boto:path=/pklxkd-72/obj7 2023-05-31T21:31:25.515 DEBUG:boto:auth_path=/pklxkd-72/obj7 2023-05-31T21:31:25.516 DEBUG:boto:Method: GET 2023-05-31T21:31:25.516 DEBUG:boto:Path: /pklxkd-72/obj7 2023-05-31T21:31:25.517 DEBUG:boto:Data: 2023-05-31T21:31:25.517 DEBUG:boto:Headers: {} 2023-05-31T21:31:25.517 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T21:31:25.518 DEBUG:boto:Port: 8001 2023-05-31T21:31:25.518 DEBUG:boto:Params: {} 2023-05-31T21:31:25.519 DEBUG:boto:Token: None 2023-05-31T21:31:25.519 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 21:31:25 GMT /pklxkd-72/obj7 2023-05-31T21:31:25.520 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:Cl7zD9Lr0agN3NIzRfSnFdJitNk= 2023-05-31T21:31:25.520 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 21:31:25 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:Cl7zD9Lr0agN3NIzRfSnFdJitNk=', 'Content-Length': '0'} 2023-05-31T21:31:25.523 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 21:29:57 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000004095f79069098877-006477bcad-295a-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 21:31:25 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:31:25.523 DEBUG:boto:path=/pklxkd-72/obj8 2023-05-31T21:31:25.524 DEBUG:boto:auth_path=/pklxkd-72/obj8 2023-05-31T21:31:25.524 DEBUG:boto:Method: HEAD 2023-05-31T21:31:25.524 DEBUG:boto:Path: /pklxkd-72/obj8 2023-05-31T21:31:25.525 DEBUG:boto:Data: 2023-05-31T21:31:25.525 DEBUG:boto:Headers: {} 2023-05-31T21:31:25.525 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T21:31:25.526 DEBUG:boto:Port: 8001 2023-05-31T21:31:25.526 DEBUG:boto:Params: {} 2023-05-31T21:31:25.526 DEBUG:boto:Token: None 2023-05-31T21:31:25.527 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 21:31:25 GMT /pklxkd-72/obj8 2023-05-31T21:31:25.527 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:9gkSEfJQ+mw6AjehtS35Kk9cpxo= 2023-05-31T21:31:25.528 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 21:31:25 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:9gkSEfJQ+mw6AjehtS35Kk9cpxo=', 'Content-Length': '0'} 2023-05-31T21:31:25.529 DEBUG:boto:Response headers: [('Content-Length', '219'), ('x-amz-request-id', 'tx000009a9277fda1c01384-006477bcad-295a-a2'), ('Accept-Ranges', 'bytes'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 21:31:25 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:31:25.535 INFO:tasks.rgw_multisite_tests:test_sync_multiple_buckets_to_single: ... FAIL 2023-05-31T21:31:25.537 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-31T21:31:25.538 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-31T21:31:25.538 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-05-31T21:31:25.587 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2023-05-31T21:31:25.587 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2023-05-31T21:31:25.643 INFO:teuthology.orchestra.run.smithi099.stderr:No --period given, using current period=2e6123d2-d695-4350-9a01-61a455bc782b 2023-05-31T21:31:25.661 INFO:teuthology.orchestra.run.smithi099.stdout:[ 2023-05-31T21:31:25.661 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T21:31:25.662 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1685566501.757183_260.1", 2023-05-31T21:31:25.662 INFO:teuthology.orchestra.run.smithi099.stdout: "last_update": "2023-05-31T20:55:01.757183Z" 2023-05-31T21:31:25.662 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:31:25.662 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T21:31:25.662 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1685566501.735717_251.1", 2023-05-31T21:31:25.662 INFO:teuthology.orchestra.run.smithi099.stdout: "last_update": "2023-05-31T20:55:01.735717Z" 2023-05-31T21:31:25.662 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:31:25.663 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T21:31:25.663 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1685562970.274572_93.1", 2023-05-31T21:31:25.663 INFO:teuthology.orchestra.run.smithi099.stdout: "last_update": "2023-05-31T19:56:10.274572Z" 2023-05-31T21:31:25.663 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:31:25.663 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T21:31:25.663 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1685568561.112279_1036.1", 2023-05-31T21:31:25.664 INFO:teuthology.orchestra.run.smithi099.stdout: "last_update": "2023-05-31T21:29:21.112279Z" 2023-05-31T21:31:25.664 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T21:31:25.664 INFO:teuthology.orchestra.run.smithi099.stdout:] 2023-05-31T21:31:25.665 DEBUG:rgw_multi.tests:master meta markers={0: '1_1685566501.757183_260.1', 1: '1_1685566501.735717_251.1', 2: '1_1685562970.274572_93.1', 3: '1_1685568561.112279_1036.1'} 2023-05-31T21:31:25.665 INFO:rgw_multi.tests:starting meta checkpoint for zone=a2 2023-05-31T21:31:25.666 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-31T21:31:25.666 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-31T21:31:25.666 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-05-31T21:31:25.716 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2023-05-31T21:31:25.717 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2023-05-31T21:31:25.769 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T21:31:25.765+0000 7fdac2848780 20 RGW-SYNC:meta: read sync status 2023-05-31T21:31:25.770 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T21:31:25.766+0000 7fdac2848780 20 RGW-SYNC:meta: read sync status 2023-05-31T21:31:25.790 INFO:teuthology.orchestra.run.smithi099.stdout:{ 2023-05-31T21:31:25.791 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_status": { 2023-05-31T21:31:25.791 INFO:teuthology.orchestra.run.smithi099.stdout: "info": { 2023-05-31T21:31:25.791 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "sync", 2023-05-31T21:31:25.791 INFO:teuthology.orchestra.run.smithi099.stdout: "num_shards": 4, 2023-05-31T21:31:25.791 INFO:teuthology.orchestra.run.smithi099.stdout: "period": "2e6123d2-d695-4350-9a01-61a455bc782b", 2023-05-31T21:31:25.792 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2023-05-31T21:31:25.792 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:31:25.792 INFO:teuthology.orchestra.run.smithi099.stdout: "markers": [ 2023-05-31T21:31:25.792 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T21:31:25.792 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 0, 2023-05-31T21:31:25.792 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T21:31:25.793 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2023-05-31T21:31:25.793 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1685566501.757183_260.1", 2023-05-31T21:31:25.793 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T21:31:25.793 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 1, 2023-05-31T21:31:25.794 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T21:31:25.794 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2023-05-31T20:55:01.757183Z", 2023-05-31T21:31:25.794 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2023-05-31T21:31:25.794 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T21:31:25.794 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:31:25.794 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T21:31:25.795 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 1, 2023-05-31T21:31:25.795 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T21:31:25.795 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2023-05-31T21:31:25.795 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1685566501.735717_251.1", 2023-05-31T21:31:25.795 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T21:31:25.795 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2023-05-31T21:31:25.796 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T21:31:25.796 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2023-05-31T20:55:01.735717Z", 2023-05-31T21:31:25.796 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2023-05-31T21:31:25.796 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T21:31:25.796 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:31:25.797 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T21:31:25.797 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 2, 2023-05-31T21:31:25.797 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T21:31:25.797 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2023-05-31T21:31:25.797 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1685562970.274572_93.1", 2023-05-31T21:31:25.797 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T21:31:25.798 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2023-05-31T21:31:25.798 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T21:31:25.798 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2023-05-31T19:56:10.274572Z", 2023-05-31T21:31:25.798 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2023-05-31T21:31:25.798 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T21:31:25.798 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:31:25.798 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T21:31:25.799 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 3, 2023-05-31T21:31:25.799 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T21:31:25.799 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2023-05-31T21:31:25.799 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1685568561.112279_1036.1", 2023-05-31T21:31:25.799 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T21:31:25.799 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2023-05-31T21:31:25.799 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T21:31:25.800 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2023-05-31T21:29:21.112279Z", 2023-05-31T21:31:25.800 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2023-05-31T21:31:25.800 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T21:31:25.800 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T21:31:25.800 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2023-05-31T21:31:25.800 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:31:25.801 INFO:teuthology.orchestra.run.smithi099.stdout: "full_sync": { 2023-05-31T21:31:25.801 INFO:teuthology.orchestra.run.smithi099.stdout: "total": 1, 2023-05-31T21:31:25.801 INFO:teuthology.orchestra.run.smithi099.stdout: "complete": 1 2023-05-31T21:31:25.801 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:31:25.801 INFO:teuthology.orchestra.run.smithi099.stdout: "current_time": "2023-05-31T21:31:25Z" 2023-05-31T21:31:25.801 INFO:teuthology.orchestra.run.smithi099.stdout:} 2023-05-31T21:31:25.802 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "2e6123d2-d695-4350-9a01-61a455bc782b", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1685566501.757183_260.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-05-31T20:55:01.757183Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1685566501.735717_251.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-05-31T20:55:01.735717Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1685562970.274572_93.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-05-31T19:56:10.274572Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1685568561.112279_1036.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-05-31T21:29:21.112279Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-05-31T21:31:25Z" } 2023-05-31T21:31:25.803 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1685566501.757183_260.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-05-31T20:55:01.757183Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1685566501.735717_251.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-05-31T20:55:01.735717Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1685562970.274572_93.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-05-31T19:56:10.274572Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1685568561.112279_1036.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-05-31T21:29:21.112279Z', 'realm_epoch': 2}}] 2023-05-31T21:31:25.803 DEBUG:rgw_multi.tests:log_status={0: '1_1685566501.757183_260.1', 1: '1_1685566501.735717_251.1', 2: '1_1685562970.274572_93.1', 3: '1_1685568561.112279_1036.1'} 2023-05-31T21:31:25.804 DEBUG:rgw_multi.tests:sync_status={0: '1_1685566501.757183_260.1', 1: '1_1685566501.735717_251.1', 2: '1_1685562970.274572_93.1', 3: '1_1685568561.112279_1036.1'} 2023-05-31T21:31:25.804 INFO:rgw_multi.tests:finish meta checkpoint for zone=a2 2023-05-31T21:31:25.805 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'policy', 'get', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-05-31T21:31:25.805 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'policy', 'get', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-05-31T21:31:25.805 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync policy get --cluster c1 --debug-rgw 0 --debug-ms 0 2023-05-31T21:31:25.851 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2023-05-31T21:31:25.851 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2023-05-31T21:31:25.925 INFO:teuthology.orchestra.run.smithi099.stdout:{"groups":[{"id":"sync-group","data_flow":{"symmetrical":[{"id":"sync-flow1","zones":["a1","a2"]}]},"pipes":[{"id":"sync-pipe","source":{"bucket":"*","zones":["a1","a2"]},"dest":{"bucket":"*","zones":["a1","a2"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":""}}],"status":"allowed"}]} 2023-05-31T21:31:25.926 INFO:rgw_multi.tests:create bucket zone=a1 name=pklxkd-74 2023-05-31T21:31:25.926 DEBUG:boto:path=/pklxkd-74/ 2023-05-31T21:31:25.927 DEBUG:boto:auth_path=/pklxkd-74/ 2023-05-31T21:31:25.927 DEBUG:boto:Method: PUT 2023-05-31T21:31:25.927 DEBUG:boto:Path: /pklxkd-74/ 2023-05-31T21:31:25.928 DEBUG:boto:Data: 2023-05-31T21:31:25.928 DEBUG:boto:Headers: {} 2023-05-31T21:31:25.928 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T21:31:25.929 DEBUG:boto:Port: 8000 2023-05-31T21:31:25.929 DEBUG:boto:Params: {} 2023-05-31T21:31:25.929 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2023-05-31T21:31:25.930 DEBUG:boto:Token: None 2023-05-31T21:31:25.930 DEBUG:boto:StringToSign: PUT Wed, 31 May 2023 21:31:25 GMT /pklxkd-74/ 2023-05-31T21:31:25.931 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:ErHI9ZZXVd0Ip93+949dXhj94es= 2023-05-31T21:31:25.931 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 21:31:25 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:ErHI9ZZXVd0Ip93+949dXhj94es=', 'Content-Length': '0'} 2023-05-31T21:31:25.949 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx000007a17c20473282847-006477bcad-2958-a1'), ('Content-Length', '0'), ('Date', 'Wed, 31 May 2023 21:31:25 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:31:25.949 INFO:rgw_multi.tests:create bucket zone=a1 name=pklxkd-75 2023-05-31T21:31:25.950 DEBUG:boto:path=/pklxkd-75/ 2023-05-31T21:31:25.950 DEBUG:boto:auth_path=/pklxkd-75/ 2023-05-31T21:31:25.950 DEBUG:boto:Method: PUT 2023-05-31T21:31:25.951 DEBUG:boto:Path: /pklxkd-75/ 2023-05-31T21:31:25.951 DEBUG:boto:Data: 2023-05-31T21:31:25.951 DEBUG:boto:Headers: {} 2023-05-31T21:31:25.952 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T21:31:25.952 DEBUG:boto:Port: 8000 2023-05-31T21:31:25.952 DEBUG:boto:Params: {} 2023-05-31T21:31:25.953 DEBUG:boto:Token: None 2023-05-31T21:31:25.953 DEBUG:boto:StringToSign: PUT Wed, 31 May 2023 21:31:25 GMT /pklxkd-75/ 2023-05-31T21:31:25.953 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:bflNiThWF94dPZRTqvOoDJdmAxM= 2023-05-31T21:31:25.954 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 21:31:25 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:bflNiThWF94dPZRTqvOoDJdmAxM=', 'Content-Length': '0'} 2023-05-31T21:31:25.971 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx000002577e4eeaecee22b-006477bcad-2958-a1'), ('Content-Length', '0'), ('Date', 'Wed, 31 May 2023 21:31:25 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:31:25.971 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-31T21:31:25.972 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-31T21:31:25.972 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-05-31T21:31:26.023 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2023-05-31T21:31:26.024 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2023-05-31T21:31:26.088 INFO:teuthology.orchestra.run.smithi099.stderr:No --period given, using current period=2e6123d2-d695-4350-9a01-61a455bc782b 2023-05-31T21:31:26.111 INFO:teuthology.orchestra.run.smithi099.stdout:[ 2023-05-31T21:31:26.111 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T21:31:26.111 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1685566501.757183_260.1", 2023-05-31T21:31:26.112 INFO:teuthology.orchestra.run.smithi099.stdout: "last_update": "2023-05-31T20:55:01.757183Z" 2023-05-31T21:31:26.112 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:31:26.112 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T21:31:26.112 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1685566501.735717_251.1", 2023-05-31T21:31:26.112 INFO:teuthology.orchestra.run.smithi099.stdout: "last_update": "2023-05-31T20:55:01.735717Z" 2023-05-31T21:31:26.112 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:31:26.113 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T21:31:26.113 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1685562970.274572_93.1", 2023-05-31T21:31:26.113 INFO:teuthology.orchestra.run.smithi099.stdout: "last_update": "2023-05-31T19:56:10.274572Z" 2023-05-31T21:31:26.113 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:31:26.113 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T21:31:26.113 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1685568685.966725_1044.1", 2023-05-31T21:31:26.114 INFO:teuthology.orchestra.run.smithi099.stdout: "last_update": "2023-05-31T21:31:25.966725Z" 2023-05-31T21:31:26.114 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T21:31:26.114 INFO:teuthology.orchestra.run.smithi099.stdout:] 2023-05-31T21:31:26.115 DEBUG:rgw_multi.tests:master meta markers={0: '1_1685566501.757183_260.1', 1: '1_1685566501.735717_251.1', 2: '1_1685562970.274572_93.1', 3: '1_1685568685.966725_1044.1'} 2023-05-31T21:31:26.115 INFO:rgw_multi.tests:starting meta checkpoint for zone=a2 2023-05-31T21:31:26.115 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-31T21:31:26.116 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-31T21:31:26.116 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-05-31T21:31:26.165 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2023-05-31T21:31:26.166 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2023-05-31T21:31:26.206 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T21:31:26.202+0000 7f304b855780 20 RGW-SYNC:meta: read sync status 2023-05-31T21:31:26.246 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T21:31:26.203+0000 7f304b855780 20 RGW-SYNC:meta: read sync status 2023-05-31T21:31:26.247 INFO:teuthology.orchestra.run.smithi099.stdout:{ 2023-05-31T21:31:26.247 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_status": { 2023-05-31T21:31:26.247 INFO:teuthology.orchestra.run.smithi099.stdout: "info": { 2023-05-31T21:31:26.247 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "sync", 2023-05-31T21:31:26.248 INFO:teuthology.orchestra.run.smithi099.stdout: "num_shards": 4, 2023-05-31T21:31:26.248 INFO:teuthology.orchestra.run.smithi099.stdout: "period": "2e6123d2-d695-4350-9a01-61a455bc782b", 2023-05-31T21:31:26.248 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2023-05-31T21:31:26.248 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:31:26.248 INFO:teuthology.orchestra.run.smithi099.stdout: "markers": [ 2023-05-31T21:31:26.248 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T21:31:26.249 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 0, 2023-05-31T21:31:26.249 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T21:31:26.249 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2023-05-31T21:31:26.249 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1685566501.757183_260.1", 2023-05-31T21:31:26.249 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T21:31:26.249 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 1, 2023-05-31T21:31:26.249 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T21:31:26.250 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2023-05-31T20:55:01.757183Z", 2023-05-31T21:31:26.250 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2023-05-31T21:31:26.250 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T21:31:26.250 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:31:26.250 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T21:31:26.250 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 1, 2023-05-31T21:31:26.250 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T21:31:26.251 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2023-05-31T21:31:26.251 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1685566501.735717_251.1", 2023-05-31T21:31:26.251 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T21:31:26.251 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2023-05-31T21:31:26.251 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T21:31:26.251 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2023-05-31T20:55:01.735717Z", 2023-05-31T21:31:26.251 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2023-05-31T21:31:26.252 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T21:31:26.252 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:31:26.252 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T21:31:26.252 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 2, 2023-05-31T21:31:26.252 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T21:31:26.252 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2023-05-31T21:31:26.253 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1685562970.274572_93.1", 2023-05-31T21:31:26.253 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T21:31:26.253 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2023-05-31T21:31:26.253 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T21:31:26.253 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2023-05-31T19:56:10.274572Z", 2023-05-31T21:31:26.253 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2023-05-31T21:31:26.253 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T21:31:26.254 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:31:26.254 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T21:31:26.254 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 3, 2023-05-31T21:31:26.254 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T21:31:26.254 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2023-05-31T21:31:26.254 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1685568685.966725_1044.1", 2023-05-31T21:31:26.254 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T21:31:26.255 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2023-05-31T21:31:26.255 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T21:31:26.255 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2023-05-31T21:31:25.966725Z", 2023-05-31T21:31:26.255 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2023-05-31T21:31:26.255 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T21:31:26.255 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T21:31:26.255 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2023-05-31T21:31:26.256 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:31:26.256 INFO:teuthology.orchestra.run.smithi099.stdout: "full_sync": { 2023-05-31T21:31:26.256 INFO:teuthology.orchestra.run.smithi099.stdout: "total": 1, 2023-05-31T21:31:26.256 INFO:teuthology.orchestra.run.smithi099.stdout: "complete": 1 2023-05-31T21:31:26.256 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:31:26.256 INFO:teuthology.orchestra.run.smithi099.stdout: "current_time": "2023-05-31T21:31:26Z" 2023-05-31T21:31:26.257 INFO:teuthology.orchestra.run.smithi099.stdout:} 2023-05-31T21:31:26.257 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "2e6123d2-d695-4350-9a01-61a455bc782b", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1685566501.757183_260.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-05-31T20:55:01.757183Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1685566501.735717_251.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-05-31T20:55:01.735717Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1685562970.274572_93.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-05-31T19:56:10.274572Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1685568685.966725_1044.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-05-31T21:31:25.966725Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-05-31T21:31:26Z" } 2023-05-31T21:31:26.257 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1685566501.757183_260.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-05-31T20:55:01.757183Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1685566501.735717_251.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-05-31T20:55:01.735717Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1685562970.274572_93.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-05-31T19:56:10.274572Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1685568685.966725_1044.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-05-31T21:31:25.966725Z', 'realm_epoch': 2}}] 2023-05-31T21:31:26.258 DEBUG:rgw_multi.tests:log_status={0: '1_1685566501.757183_260.1', 1: '1_1685566501.735717_251.1', 2: '1_1685562970.274572_93.1', 3: '1_1685568685.966725_1044.1'} 2023-05-31T21:31:26.258 DEBUG:rgw_multi.tests:sync_status={0: '1_1685566501.757183_260.1', 1: '1_1685566501.735717_251.1', 2: '1_1685562970.274572_93.1', 3: '1_1685568685.966725_1044.1'} 2023-05-31T21:31:26.258 INFO:rgw_multi.tests:finish meta checkpoint for zone=a2 2023-05-31T21:31:26.259 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'group', 'create', '--group-id', 'sync-group', '--status', 'allowed', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-05-31T21:31:26.259 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'group', 'create', '--group-id', 'sync-group', '--status', 'allowed', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-05-31T21:31:26.259 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync group create --group-id sync-group --status allowed --cluster c1 --debug-rgw 0 --debug-ms 0 2023-05-31T21:31:26.309 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2023-05-31T21:31:26.309 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2023-05-31T21:31:26.380 INFO:teuthology.orchestra.run.smithi099.stdout:{"groups":[{"id":"sync-group","data_flow":{"symmetrical":[{"id":"sync-flow1","zones":["a1","a2"]}]},"pipes":[{"id":"sync-pipe","source":{"bucket":"*","zones":["a1","a2"]},"dest":{"bucket":"*","zones":["a1","a2"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":""}}],"status":"allowed"}]} 2023-05-31T21:31:26.381 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'group', 'flow', 'create', '--group-id', 'sync-group', '--flow-id', 'sync-flow1', '--flow-type', 'symmetrical', '--zones=a1,a2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-05-31T21:31:26.381 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'group', 'flow', 'create', '--group-id', 'sync-group', '--flow-id', 'sync-flow1', '--flow-type', 'symmetrical', '--zones=a1,a2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-05-31T21:31:26.382 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync group flow create --group-id sync-group --flow-id sync-flow1 --flow-type symmetrical --zones=a1,a2 --cluster c1 --debug-rgw 0 --debug-ms 0 2023-05-31T21:31:26.435 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2023-05-31T21:31:26.435 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2023-05-31T21:31:26.518 INFO:teuthology.orchestra.run.smithi099.stdout:{"groups":[{"id":"sync-group","data_flow":{"symmetrical":[{"id":"sync-flow1","zones":["a1","a2"]}]},"pipes":[{"id":"sync-pipe","source":{"bucket":"*","zones":["a1","a2"]},"dest":{"bucket":"*","zones":["a1","a2"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":""}}],"status":"allowed"}]} 2023-05-31T21:31:26.518 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'group', 'pipe', 'create', '--group-id', 'sync-group', '--pipe-id', 'sync-pipe-pklxkd-74', '--source-zones=a1', '--dest-zones=a2', '--source-bucket=pklxkd-74', '--dest-bucket=pklxkd-74', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-05-31T21:31:26.518 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'group', 'pipe', 'create', '--group-id', 'sync-group', '--pipe-id', 'sync-pipe-pklxkd-74', '--source-zones=a1', '--dest-zones=a2', '--source-bucket=pklxkd-74', '--dest-bucket=pklxkd-74', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-05-31T21:31:26.519 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync group pipe create --group-id sync-group --pipe-id sync-pipe-pklxkd-74 --source-zones=a1 --dest-zones=a2 --source-bucket=pklxkd-74 --dest-bucket=pklxkd-74 --cluster c1 --debug-rgw 0 --debug-ms 0 2023-05-31T21:31:26.573 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2023-05-31T21:31:26.573 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2023-05-31T21:31:26.624 INFO:teuthology.orchestra.run.smithi099.stdout:{"groups":[{"id":"sync-group","data_flow":{"symmetrical":[{"id":"sync-flow1","zones":["a1","a2"]}]},"pipes":[{"id":"sync-pipe","source":{"bucket":"*","zones":["a1","a2"]},"dest":{"bucket":"*","zones":["a1","a2"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":""}},{"id":"sync-pipe-pklxkd-74","source":{"bucket":"pklxkd-74","zones":["a1"]},"dest":{"bucket":"pklxkd-74","zones":["a2"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":""}}],"status":"allowed"}]} 2023-05-31T21:31:26.625 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'group', 'pipe', 'create', '--group-id', 'sync-group', '--pipe-id', 'sync-pipe-pklxkd-75', '--source-zones=a1', '--dest-zones=a2', '--source-bucket=pklxkd-74', '--dest-bucket=pklxkd-75', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-05-31T21:31:26.625 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'group', 'pipe', 'create', '--group-id', 'sync-group', '--pipe-id', 'sync-pipe-pklxkd-75', '--source-zones=a1', '--dest-zones=a2', '--source-bucket=pklxkd-74', '--dest-bucket=pklxkd-75', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-05-31T21:31:26.626 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync group pipe create --group-id sync-group --pipe-id sync-pipe-pklxkd-75 --source-zones=a1 --dest-zones=a2 --source-bucket=pklxkd-74 --dest-bucket=pklxkd-75 --cluster c1 --debug-rgw 0 --debug-ms 0 2023-05-31T21:31:26.679 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2023-05-31T21:31:26.679 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2023-05-31T21:31:26.727 INFO:teuthology.orchestra.run.smithi099.stdout:{"groups":[{"id":"sync-group","data_flow":{"symmetrical":[{"id":"sync-flow1","zones":["a1","a2"]}]},"pipes":[{"id":"sync-pipe","source":{"bucket":"*","zones":["a1","a2"]},"dest":{"bucket":"*","zones":["a1","a2"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":""}},{"id":"sync-pipe-pklxkd-74","source":{"bucket":"pklxkd-74","zones":["a1"]},"dest":{"bucket":"pklxkd-74","zones":["a2"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":""}},{"id":"sync-pipe-pklxkd-75","source":{"bucket":"pklxkd-74","zones":["a1"]},"dest":{"bucket":"pklxkd-75","zones":["a2"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":""}}],"status":"allowed"}]} 2023-05-31T21:31:26.727 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'group', 'pipe', 'create', '--group-id', 'sync-group', '--pipe-id', 'sync-pipe', '--source-zones=a1', '--dest-zones=a2', '--source-bucket=pklxkd-74', '--dest-bucket=pklxkd-75', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-05-31T21:31:26.727 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'group', 'pipe', 'create', '--group-id', 'sync-group', '--pipe-id', 'sync-pipe', '--source-zones=a1', '--dest-zones=a2', '--source-bucket=pklxkd-74', '--dest-bucket=pklxkd-75', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-05-31T21:31:26.728 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync group pipe create --group-id sync-group --pipe-id sync-pipe --source-zones=a1 --dest-zones=a2 --source-bucket=pklxkd-74 --dest-bucket=pklxkd-75 --cluster c1 --debug-rgw 0 --debug-ms 0 2023-05-31T21:31:26.782 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2023-05-31T21:31:26.782 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2023-05-31T21:31:26.848 INFO:teuthology.orchestra.run.smithi099.stdout:{"groups":[{"id":"sync-group","data_flow":{"symmetrical":[{"id":"sync-flow1","zones":["a1","a2"]}]},"pipes":[{"id":"sync-pipe","source":{"bucket":"pklxkd-74","zones":["a1","a2"]},"dest":{"bucket":"pklxkd-75","zones":["a1","a2"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":""}},{"id":"sync-pipe-pklxkd-74","source":{"bucket":"pklxkd-74","zones":["a1"]},"dest":{"bucket":"pklxkd-74","zones":["a2"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":""}},{"id":"sync-pipe-pklxkd-75","source":{"bucket":"pklxkd-74","zones":["a1"]},"dest":{"bucket":"pklxkd-75","zones":["a2"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":""}}],"status":"allowed"}]} 2023-05-31T21:31:26.848 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'group', 'modify', '--group-id', 'sync-group', '--status', 'enabled', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-05-31T21:31:26.849 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'group', 'modify', '--group-id', 'sync-group', '--status', 'enabled', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-05-31T21:31:26.849 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync group modify --group-id sync-group --status enabled --cluster c1 --debug-rgw 0 --debug-ms 0 2023-05-31T21:31:26.898 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2023-05-31T21:31:26.898 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2023-05-31T21:31:26.961 INFO:teuthology.orchestra.run.smithi099.stdout:{"groups":[{"id":"sync-group","data_flow":{"symmetrical":[{"id":"sync-flow1","zones":["a1","a2"]}]},"pipes":[{"id":"sync-pipe","source":{"bucket":"pklxkd-74","zones":["a1","a2"]},"dest":{"bucket":"pklxkd-75","zones":["a1","a2"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":""}},{"id":"sync-pipe-pklxkd-74","source":{"bucket":"pklxkd-74","zones":["a1"]},"dest":{"bucket":"pklxkd-74","zones":["a2"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":""}},{"id":"sync-pipe-pklxkd-75","source":{"bucket":"pklxkd-74","zones":["a1"]},"dest":{"bucket":"pklxkd-75","zones":["a2"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":""}}],"status":"enabled"}]} 2023-05-31T21:31:26.961 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['period', 'update', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-05-31T21:31:26.961 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'period', 'update', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-05-31T21:31:26.962 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 period update --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --commit --cluster c1 --debug-rgw 0 --debug-ms 0 2023-05-31T21:31:27.010 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2023-05-31T21:31:27.010 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2023-05-31T21:31:27.068 INFO:teuthology.orchestra.run.smithi099.stdout:{ 2023-05-31T21:31:27.068 INFO:teuthology.orchestra.run.smithi099.stdout: "id": "2e6123d2-d695-4350-9a01-61a455bc782b", 2023-05-31T21:31:27.069 INFO:teuthology.orchestra.run.smithi099.stdout: "epoch": 21, 2023-05-31T21:31:27.069 INFO:teuthology.orchestra.run.smithi099.stdout: "predecessor_uuid": "629b9ced-80af-41ed-8a6c-f2cfc334cbb4", 2023-05-31T21:31:27.069 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_status": [], 2023-05-31T21:31:27.069 INFO:teuthology.orchestra.run.smithi099.stdout: "period_map": { 2023-05-31T21:31:27.069 INFO:teuthology.orchestra.run.smithi099.stdout: "id": "2e6123d2-d695-4350-9a01-61a455bc782b", 2023-05-31T21:31:27.069 INFO:teuthology.orchestra.run.smithi099.stdout: "zonegroups": [ 2023-05-31T21:31:27.070 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T21:31:27.070 INFO:teuthology.orchestra.run.smithi099.stdout: "id": "a", 2023-05-31T21:31:27.070 INFO:teuthology.orchestra.run.smithi099.stdout: "name": "a", 2023-05-31T21:31:27.070 INFO:teuthology.orchestra.run.smithi099.stdout: "api_name": "", 2023-05-31T21:31:27.070 INFO:teuthology.orchestra.run.smithi099.stdout: "is_master": true, 2023-05-31T21:31:27.070 INFO:teuthology.orchestra.run.smithi099.stdout: "endpoints": [ 2023-05-31T21:31:27.070 INFO:teuthology.orchestra.run.smithi099.stdout: "http://smithi099.front.sepia.ceph.com:8000" 2023-05-31T21:31:27.071 INFO:teuthology.orchestra.run.smithi099.stdout: ], 2023-05-31T21:31:27.071 INFO:teuthology.orchestra.run.smithi099.stdout: "hostnames": [], 2023-05-31T21:31:27.071 INFO:teuthology.orchestra.run.smithi099.stdout: "hostnames_s3website": [], 2023-05-31T21:31:27.071 INFO:teuthology.orchestra.run.smithi099.stdout: "master_zone": "3d163b1e-09c7-47e2-ae87-add4aace3f38", 2023-05-31T21:31:27.071 INFO:teuthology.orchestra.run.smithi099.stdout: "zones": [ 2023-05-31T21:31:27.071 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T21:31:27.071 INFO:teuthology.orchestra.run.smithi099.stdout: "id": "3d163b1e-09c7-47e2-ae87-add4aace3f38", 2023-05-31T21:31:27.072 INFO:teuthology.orchestra.run.smithi099.stdout: "name": "a1", 2023-05-31T21:31:27.072 INFO:teuthology.orchestra.run.smithi099.stdout: "endpoints": [ 2023-05-31T21:31:27.072 INFO:teuthology.orchestra.run.smithi099.stdout: "http://smithi099.front.sepia.ceph.com:8000" 2023-05-31T21:31:27.072 INFO:teuthology.orchestra.run.smithi099.stdout: ], 2023-05-31T21:31:27.072 INFO:teuthology.orchestra.run.smithi099.stdout: "log_meta": false, 2023-05-31T21:31:27.072 INFO:teuthology.orchestra.run.smithi099.stdout: "log_data": true, 2023-05-31T21:31:27.073 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket_index_max_shards": 11, 2023-05-31T21:31:27.073 INFO:teuthology.orchestra.run.smithi099.stdout: "read_only": false, 2023-05-31T21:31:27.073 INFO:teuthology.orchestra.run.smithi099.stdout: "tier_type": "", 2023-05-31T21:31:27.073 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_from_all": true, 2023-05-31T21:31:27.073 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_from": [], 2023-05-31T21:31:27.073 INFO:teuthology.orchestra.run.smithi099.stdout: "redirect_zone": "", 2023-05-31T21:31:27.073 INFO:teuthology.orchestra.run.smithi099.stdout: "supported_features": [ 2023-05-31T21:31:27.074 INFO:teuthology.orchestra.run.smithi099.stdout: "resharding" 2023-05-31T21:31:27.074 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2023-05-31T21:31:27.074 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:31:27.074 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T21:31:27.074 INFO:teuthology.orchestra.run.smithi099.stdout: "id": "5d8acec7-7e57-4e56-8b32-e3d6b5036001", 2023-05-31T21:31:27.074 INFO:teuthology.orchestra.run.smithi099.stdout: "name": "a2", 2023-05-31T21:31:27.074 INFO:teuthology.orchestra.run.smithi099.stdout: "endpoints": [ 2023-05-31T21:31:27.075 INFO:teuthology.orchestra.run.smithi099.stdout: "http://smithi099.front.sepia.ceph.com:8001" 2023-05-31T21:31:27.075 INFO:teuthology.orchestra.run.smithi099.stdout: ], 2023-05-31T21:31:27.075 INFO:teuthology.orchestra.run.smithi099.stdout: "log_meta": false, 2023-05-31T21:31:27.075 INFO:teuthology.orchestra.run.smithi099.stdout: "log_data": true, 2023-05-31T21:31:27.075 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket_index_max_shards": 11, 2023-05-31T21:31:27.075 INFO:teuthology.orchestra.run.smithi099.stdout: "read_only": false, 2023-05-31T21:31:27.075 INFO:teuthology.orchestra.run.smithi099.stdout: "tier_type": "", 2023-05-31T21:31:27.076 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_from_all": true, 2023-05-31T21:31:27.076 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_from": [], 2023-05-31T21:31:27.076 INFO:teuthology.orchestra.run.smithi099.stdout: "redirect_zone": "", 2023-05-31T21:31:27.076 INFO:teuthology.orchestra.run.smithi099.stdout: "supported_features": [ 2023-05-31T21:31:27.076 INFO:teuthology.orchestra.run.smithi099.stdout: "resharding" 2023-05-31T21:31:27.076 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2023-05-31T21:31:27.076 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T21:31:27.077 INFO:teuthology.orchestra.run.smithi099.stdout: ], 2023-05-31T21:31:27.077 INFO:teuthology.orchestra.run.smithi099.stdout: "placement_targets": [ 2023-05-31T21:31:27.077 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T21:31:27.077 INFO:teuthology.orchestra.run.smithi099.stdout: "name": "default-placement", 2023-05-31T21:31:27.077 INFO:teuthology.orchestra.run.smithi099.stdout: "tags": [], 2023-05-31T21:31:27.077 INFO:teuthology.orchestra.run.smithi099.stdout: "storage_classes": [ 2023-05-31T21:31:27.077 INFO:teuthology.orchestra.run.smithi099.stdout: "STANDARD" 2023-05-31T21:31:27.078 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2023-05-31T21:31:27.078 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T21:31:27.078 INFO:teuthology.orchestra.run.smithi099.stdout: ], 2023-05-31T21:31:27.078 INFO:teuthology.orchestra.run.smithi099.stdout: "default_placement": "default-placement", 2023-05-31T21:31:27.078 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_id": "aaa33de6-515b-4cac-8f1b-3a1a30452117", 2023-05-31T21:31:27.078 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_policy": { 2023-05-31T21:31:27.078 INFO:teuthology.orchestra.run.smithi099.stdout: "groups": [ 2023-05-31T21:31:27.079 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T21:31:27.079 INFO:teuthology.orchestra.run.smithi099.stdout: "id": "sync-group", 2023-05-31T21:31:27.079 INFO:teuthology.orchestra.run.smithi099.stdout: "data_flow": { 2023-05-31T21:31:27.079 INFO:teuthology.orchestra.run.smithi099.stdout: "symmetrical": [ 2023-05-31T21:31:27.079 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T21:31:27.079 INFO:teuthology.orchestra.run.smithi099.stdout: "id": "sync-flow1", 2023-05-31T21:31:27.079 INFO:teuthology.orchestra.run.smithi099.stdout: "zones": [ 2023-05-31T21:31:27.080 INFO:teuthology.orchestra.run.smithi099.stdout: "3d163b1e-09c7-47e2-ae87-add4aace3f38", 2023-05-31T21:31:27.080 INFO:teuthology.orchestra.run.smithi099.stdout: "5d8acec7-7e57-4e56-8b32-e3d6b5036001" 2023-05-31T21:31:27.080 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2023-05-31T21:31:27.080 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T21:31:27.080 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2023-05-31T21:31:27.080 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:31:27.081 INFO:teuthology.orchestra.run.smithi099.stdout: "pipes": [ 2023-05-31T21:31:27.081 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T21:31:27.081 INFO:teuthology.orchestra.run.smithi099.stdout: "id": "sync-pipe", 2023-05-31T21:31:27.081 INFO:teuthology.orchestra.run.smithi099.stdout: "source": { 2023-05-31T21:31:27.081 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket": "pklxkd-74", 2023-05-31T21:31:27.081 INFO:teuthology.orchestra.run.smithi099.stdout: "zones": [ 2023-05-31T21:31:27.081 INFO:teuthology.orchestra.run.smithi099.stdout: "3d163b1e-09c7-47e2-ae87-add4aace3f38", 2023-05-31T21:31:27.082 INFO:teuthology.orchestra.run.smithi099.stdout: "5d8acec7-7e57-4e56-8b32-e3d6b5036001" 2023-05-31T21:31:27.082 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2023-05-31T21:31:27.082 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:31:27.082 INFO:teuthology.orchestra.run.smithi099.stdout: "dest": { 2023-05-31T21:31:27.082 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket": "pklxkd-75", 2023-05-31T21:31:27.082 INFO:teuthology.orchestra.run.smithi099.stdout: "zones": [ 2023-05-31T21:31:27.082 INFO:teuthology.orchestra.run.smithi099.stdout: "3d163b1e-09c7-47e2-ae87-add4aace3f38", 2023-05-31T21:31:27.083 INFO:teuthology.orchestra.run.smithi099.stdout: "5d8acec7-7e57-4e56-8b32-e3d6b5036001" 2023-05-31T21:31:27.083 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2023-05-31T21:31:27.083 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:31:27.083 INFO:teuthology.orchestra.run.smithi099.stdout: "params": { 2023-05-31T21:31:27.083 INFO:teuthology.orchestra.run.smithi099.stdout: "source": { 2023-05-31T21:31:27.083 INFO:teuthology.orchestra.run.smithi099.stdout: "filter": { 2023-05-31T21:31:27.083 INFO:teuthology.orchestra.run.smithi099.stdout: "tags": [] 2023-05-31T21:31:27.084 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T21:31:27.084 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:31:27.084 INFO:teuthology.orchestra.run.smithi099.stdout: "dest": {}, 2023-05-31T21:31:27.084 INFO:teuthology.orchestra.run.smithi099.stdout: "priority": 0, 2023-05-31T21:31:27.084 INFO:teuthology.orchestra.run.smithi099.stdout: "mode": "system", 2023-05-31T21:31:27.084 INFO:teuthology.orchestra.run.smithi099.stdout: "user": "" 2023-05-31T21:31:27.084 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T21:31:27.085 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:31:27.085 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T21:31:27.085 INFO:teuthology.orchestra.run.smithi099.stdout: "id": "sync-pipe-pklxkd-74", 2023-05-31T21:31:27.085 INFO:teuthology.orchestra.run.smithi099.stdout: "source": { 2023-05-31T21:31:27.085 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket": "pklxkd-74", 2023-05-31T21:31:27.085 INFO:teuthology.orchestra.run.smithi099.stdout: "zones": [ 2023-05-31T21:31:27.085 INFO:teuthology.orchestra.run.smithi099.stdout: "3d163b1e-09c7-47e2-ae87-add4aace3f38" 2023-05-31T21:31:27.086 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2023-05-31T21:31:27.086 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:31:27.086 INFO:teuthology.orchestra.run.smithi099.stdout: "dest": { 2023-05-31T21:31:27.086 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket": "pklxkd-74", 2023-05-31T21:31:27.086 INFO:teuthology.orchestra.run.smithi099.stdout: "zones": [ 2023-05-31T21:31:27.086 INFO:teuthology.orchestra.run.smithi099.stdout: "5d8acec7-7e57-4e56-8b32-e3d6b5036001" 2023-05-31T21:31:27.086 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2023-05-31T21:31:27.087 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:31:27.087 INFO:teuthology.orchestra.run.smithi099.stdout: "params": { 2023-05-31T21:31:27.087 INFO:teuthology.orchestra.run.smithi099.stdout: "source": { 2023-05-31T21:31:27.087 INFO:teuthology.orchestra.run.smithi099.stdout: "filter": { 2023-05-31T21:31:27.087 INFO:teuthology.orchestra.run.smithi099.stdout: "tags": [] 2023-05-31T21:31:27.087 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T21:31:27.087 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:31:27.088 INFO:teuthology.orchestra.run.smithi099.stdout: "dest": {}, 2023-05-31T21:31:27.088 INFO:teuthology.orchestra.run.smithi099.stdout: "priority": 0, 2023-05-31T21:31:27.088 INFO:teuthology.orchestra.run.smithi099.stdout: "mode": "system", 2023-05-31T21:31:27.088 INFO:teuthology.orchestra.run.smithi099.stdout: "user": "" 2023-05-31T21:31:27.088 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T21:31:27.088 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:31:27.088 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T21:31:27.089 INFO:teuthology.orchestra.run.smithi099.stdout: "id": "sync-pipe-pklxkd-75", 2023-05-31T21:31:27.089 INFO:teuthology.orchestra.run.smithi099.stdout: "source": { 2023-05-31T21:31:27.089 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket": "pklxkd-74", 2023-05-31T21:31:27.089 INFO:teuthology.orchestra.run.smithi099.stdout: "zones": [ 2023-05-31T21:31:27.089 INFO:teuthology.orchestra.run.smithi099.stdout: "3d163b1e-09c7-47e2-ae87-add4aace3f38" 2023-05-31T21:31:27.089 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2023-05-31T21:31:27.089 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:31:27.090 INFO:teuthology.orchestra.run.smithi099.stdout: "dest": { 2023-05-31T21:31:27.090 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket": "pklxkd-75", 2023-05-31T21:31:27.090 INFO:teuthology.orchestra.run.smithi099.stdout: "zones": [ 2023-05-31T21:31:27.090 INFO:teuthology.orchestra.run.smithi099.stdout: "5d8acec7-7e57-4e56-8b32-e3d6b5036001" 2023-05-31T21:31:27.090 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2023-05-31T21:31:27.090 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:31:27.090 INFO:teuthology.orchestra.run.smithi099.stdout: "params": { 2023-05-31T21:31:27.091 INFO:teuthology.orchestra.run.smithi099.stdout: "source": { 2023-05-31T21:31:27.091 INFO:teuthology.orchestra.run.smithi099.stdout: "filter": { 2023-05-31T21:31:27.091 INFO:teuthology.orchestra.run.smithi099.stdout: "tags": [] 2023-05-31T21:31:27.091 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T21:31:27.091 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:31:27.091 INFO:teuthology.orchestra.run.smithi099.stdout: "dest": {}, 2023-05-31T21:31:27.091 INFO:teuthology.orchestra.run.smithi099.stdout: "priority": 0, 2023-05-31T21:31:27.092 INFO:teuthology.orchestra.run.smithi099.stdout: "mode": "system", 2023-05-31T21:31:27.092 INFO:teuthology.orchestra.run.smithi099.stdout: "user": "" 2023-05-31T21:31:27.092 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T21:31:27.092 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T21:31:27.092 INFO:teuthology.orchestra.run.smithi099.stdout: ], 2023-05-31T21:31:27.092 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "enabled" 2023-05-31T21:31:27.093 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T21:31:27.093 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2023-05-31T21:31:27.093 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:31:27.093 INFO:teuthology.orchestra.run.smithi099.stdout: "enabled_features": [ 2023-05-31T21:31:27.094 INFO:teuthology.orchestra.run.smithi099.stdout: "resharding" 2023-05-31T21:31:27.094 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2023-05-31T21:31:27.094 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:31:27.094 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T21:31:27.094 INFO:teuthology.orchestra.run.smithi099.stdout: "id": "b", 2023-05-31T21:31:27.094 INFO:teuthology.orchestra.run.smithi099.stdout: "name": "b", 2023-05-31T21:31:27.095 INFO:teuthology.orchestra.run.smithi099.stdout: "api_name": "", 2023-05-31T21:31:27.095 INFO:teuthology.orchestra.run.smithi099.stdout: "is_master": false, 2023-05-31T21:31:27.095 INFO:teuthology.orchestra.run.smithi099.stdout: "endpoints": [ 2023-05-31T21:31:27.095 INFO:teuthology.orchestra.run.smithi099.stdout: "http://smithi112.front.sepia.ceph.com:8000" 2023-05-31T21:31:27.095 INFO:teuthology.orchestra.run.smithi099.stdout: ], 2023-05-31T21:31:27.095 INFO:teuthology.orchestra.run.smithi099.stdout: "hostnames": [], 2023-05-31T21:31:27.095 INFO:teuthology.orchestra.run.smithi099.stdout: "hostnames_s3website": [], 2023-05-31T21:31:27.097 INFO:teuthology.orchestra.run.smithi099.stdout: "master_zone": "f85d7236-2f27-4988-af26-9c66ab280b1b", 2023-05-31T21:31:27.097 INFO:teuthology.orchestra.run.smithi099.stdout: "zones": [ 2023-05-31T21:31:27.097 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T21:31:27.097 INFO:teuthology.orchestra.run.smithi099.stdout: "id": "a6a3aab5-bc52-44a4-8543-5c5ea6056acd", 2023-05-31T21:31:27.097 INFO:teuthology.orchestra.run.smithi099.stdout: "name": "b2", 2023-05-31T21:31:27.097 INFO:teuthology.orchestra.run.smithi099.stdout: "endpoints": [ 2023-05-31T21:31:27.098 INFO:teuthology.orchestra.run.smithi099.stdout: "http://smithi112.front.sepia.ceph.com:8001" 2023-05-31T21:31:27.098 INFO:teuthology.orchestra.run.smithi099.stdout: ], 2023-05-31T21:31:27.098 INFO:teuthology.orchestra.run.smithi099.stdout: "log_meta": false, 2023-05-31T21:31:27.098 INFO:teuthology.orchestra.run.smithi099.stdout: "log_data": true, 2023-05-31T21:31:27.098 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket_index_max_shards": 11, 2023-05-31T21:31:27.098 INFO:teuthology.orchestra.run.smithi099.stdout: "read_only": false, 2023-05-31T21:31:27.099 INFO:teuthology.orchestra.run.smithi099.stdout: "tier_type": "", 2023-05-31T21:31:27.099 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_from_all": true, 2023-05-31T21:31:27.099 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_from": [], 2023-05-31T21:31:27.099 INFO:teuthology.orchestra.run.smithi099.stdout: "redirect_zone": "", 2023-05-31T21:31:27.099 INFO:teuthology.orchestra.run.smithi099.stdout: "supported_features": [ 2023-05-31T21:31:27.099 INFO:teuthology.orchestra.run.smithi099.stdout: "resharding" 2023-05-31T21:31:27.099 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2023-05-31T21:31:27.100 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:31:27.100 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T21:31:27.100 INFO:teuthology.orchestra.run.smithi099.stdout: "id": "f85d7236-2f27-4988-af26-9c66ab280b1b", 2023-05-31T21:31:27.100 INFO:teuthology.orchestra.run.smithi099.stdout: "name": "b1", 2023-05-31T21:31:27.100 INFO:teuthology.orchestra.run.smithi099.stdout: "endpoints": [ 2023-05-31T21:31:27.100 INFO:teuthology.orchestra.run.smithi099.stdout: "http://smithi112.front.sepia.ceph.com:8000" 2023-05-31T21:31:27.100 INFO:teuthology.orchestra.run.smithi099.stdout: ], 2023-05-31T21:31:27.101 INFO:teuthology.orchestra.run.smithi099.stdout: "log_meta": false, 2023-05-31T21:31:27.101 INFO:teuthology.orchestra.run.smithi099.stdout: "log_data": true, 2023-05-31T21:31:27.101 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket_index_max_shards": 11, 2023-05-31T21:31:27.101 INFO:teuthology.orchestra.run.smithi099.stdout: "read_only": false, 2023-05-31T21:31:27.101 INFO:teuthology.orchestra.run.smithi099.stdout: "tier_type": "", 2023-05-31T21:31:27.101 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_from_all": true, 2023-05-31T21:31:27.101 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_from": [], 2023-05-31T21:31:27.102 INFO:teuthology.orchestra.run.smithi099.stdout: "redirect_zone": "", 2023-05-31T21:31:27.102 INFO:teuthology.orchestra.run.smithi099.stdout: "supported_features": [ 2023-05-31T21:31:27.102 INFO:teuthology.orchestra.run.smithi099.stdout: "resharding" 2023-05-31T21:31:27.102 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2023-05-31T21:31:27.102 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T21:31:27.102 INFO:teuthology.orchestra.run.smithi099.stdout: ], 2023-05-31T21:31:27.102 INFO:teuthology.orchestra.run.smithi099.stdout: "placement_targets": [ 2023-05-31T21:31:27.103 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T21:31:27.103 INFO:teuthology.orchestra.run.smithi099.stdout: "name": "default-placement", 2023-05-31T21:31:27.103 INFO:teuthology.orchestra.run.smithi099.stdout: "tags": [], 2023-05-31T21:31:27.103 INFO:teuthology.orchestra.run.smithi099.stdout: "storage_classes": [ 2023-05-31T21:31:27.103 INFO:teuthology.orchestra.run.smithi099.stdout: "STANDARD" 2023-05-31T21:31:27.103 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2023-05-31T21:31:27.103 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T21:31:27.104 INFO:teuthology.orchestra.run.smithi099.stdout: ], 2023-05-31T21:31:27.104 INFO:teuthology.orchestra.run.smithi099.stdout: "default_placement": "default-placement", 2023-05-31T21:31:27.104 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_id": "aaa33de6-515b-4cac-8f1b-3a1a30452117", 2023-05-31T21:31:27.104 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_policy": { 2023-05-31T21:31:27.104 INFO:teuthology.orchestra.run.smithi099.stdout: "groups": [] 2023-05-31T21:31:27.104 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:31:27.105 INFO:teuthology.orchestra.run.smithi099.stdout: "enabled_features": [ 2023-05-31T21:31:27.105 INFO:teuthology.orchestra.run.smithi099.stdout: "resharding" 2023-05-31T21:31:27.105 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2023-05-31T21:31:27.105 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T21:31:27.105 INFO:teuthology.orchestra.run.smithi099.stdout: ], 2023-05-31T21:31:27.105 INFO:teuthology.orchestra.run.smithi099.stdout: "short_zone_ids": [ 2023-05-31T21:31:27.105 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T21:31:27.106 INFO:teuthology.orchestra.run.smithi099.stdout: "key": "3d163b1e-09c7-47e2-ae87-add4aace3f38", 2023-05-31T21:31:27.106 INFO:teuthology.orchestra.run.smithi099.stdout: "val": 3101833771 2023-05-31T21:31:27.106 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:31:27.106 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T21:31:27.106 INFO:teuthology.orchestra.run.smithi099.stdout: "key": "5d8acec7-7e57-4e56-8b32-e3d6b5036001", 2023-05-31T21:31:27.106 INFO:teuthology.orchestra.run.smithi099.stdout: "val": 4255530623 2023-05-31T21:31:27.106 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:31:27.107 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T21:31:27.107 INFO:teuthology.orchestra.run.smithi099.stdout: "key": "a6a3aab5-bc52-44a4-8543-5c5ea6056acd", 2023-05-31T21:31:27.107 INFO:teuthology.orchestra.run.smithi099.stdout: "val": 3708073710 2023-05-31T21:31:27.107 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:31:27.107 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T21:31:27.107 INFO:teuthology.orchestra.run.smithi099.stdout: "key": "f85d7236-2f27-4988-af26-9c66ab280b1b", 2023-05-31T21:31:27.107 INFO:teuthology.orchestra.run.smithi099.stdout: "val": 233395200 2023-05-31T21:31:27.108 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T21:31:27.108 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2023-05-31T21:31:27.108 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:31:27.108 INFO:teuthology.orchestra.run.smithi099.stdout: "master_zonegroup": "a", 2023-05-31T21:31:27.108 INFO:teuthology.orchestra.run.smithi099.stdout: "master_zone": "3d163b1e-09c7-47e2-ae87-add4aace3f38", 2023-05-31T21:31:27.108 INFO:teuthology.orchestra.run.smithi099.stdout: "period_config": { 2023-05-31T21:31:27.108 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket_quota": { 2023-05-31T21:31:27.109 INFO:teuthology.orchestra.run.smithi099.stdout: "enabled": false, 2023-05-31T21:31:27.109 INFO:teuthology.orchestra.run.smithi099.stdout: "check_on_raw": false, 2023-05-31T21:31:27.109 INFO:teuthology.orchestra.run.smithi099.stdout: "max_size": -1, 2023-05-31T21:31:27.109 INFO:teuthology.orchestra.run.smithi099.stdout: "max_size_kb": 0, 2023-05-31T21:31:27.109 INFO:teuthology.orchestra.run.smithi099.stdout: "max_objects": -1 2023-05-31T21:31:27.109 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:31:27.109 INFO:teuthology.orchestra.run.smithi099.stdout: "user_quota": { 2023-05-31T21:31:27.110 INFO:teuthology.orchestra.run.smithi099.stdout: "enabled": false, 2023-05-31T21:31:27.110 INFO:teuthology.orchestra.run.smithi099.stdout: "check_on_raw": false, 2023-05-31T21:31:27.110 INFO:teuthology.orchestra.run.smithi099.stdout: "max_size": -1, 2023-05-31T21:31:27.110 INFO:teuthology.orchestra.run.smithi099.stdout: "max_size_kb": 0, 2023-05-31T21:31:27.110 INFO:teuthology.orchestra.run.smithi099.stdout: "max_objects": -1 2023-05-31T21:31:27.110 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:31:27.110 INFO:teuthology.orchestra.run.smithi099.stdout: "user_ratelimit": { 2023-05-31T21:31:27.111 INFO:teuthology.orchestra.run.smithi099.stdout: "max_read_ops": 0, 2023-05-31T21:31:27.111 INFO:teuthology.orchestra.run.smithi099.stdout: "max_write_ops": 0, 2023-05-31T21:31:27.111 INFO:teuthology.orchestra.run.smithi099.stdout: "max_read_bytes": 0, 2023-05-31T21:31:27.111 INFO:teuthology.orchestra.run.smithi099.stdout: "max_write_bytes": 0, 2023-05-31T21:31:27.111 INFO:teuthology.orchestra.run.smithi099.stdout: "enabled": false 2023-05-31T21:31:27.111 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:31:27.111 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket_ratelimit": { 2023-05-31T21:31:27.112 INFO:teuthology.orchestra.run.smithi099.stdout: "max_read_ops": 0, 2023-05-31T21:31:27.112 INFO:teuthology.orchestra.run.smithi099.stdout: "max_write_ops": 0, 2023-05-31T21:31:27.112 INFO:teuthology.orchestra.run.smithi099.stdout: "max_read_bytes": 0, 2023-05-31T21:31:27.112 INFO:teuthology.orchestra.run.smithi099.stdout: "max_write_bytes": 0, 2023-05-31T21:31:27.112 INFO:teuthology.orchestra.run.smithi099.stdout: "enabled": false 2023-05-31T21:31:27.112 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:31:27.112 INFO:teuthology.orchestra.run.smithi099.stdout: "anonymous_ratelimit": { 2023-05-31T21:31:27.113 INFO:teuthology.orchestra.run.smithi099.stdout: "max_read_ops": 0, 2023-05-31T21:31:27.113 INFO:teuthology.orchestra.run.smithi099.stdout: "max_write_ops": 0, 2023-05-31T21:31:27.113 INFO:teuthology.orchestra.run.smithi099.stdout: "max_read_bytes": 0, 2023-05-31T21:31:27.113 INFO:teuthology.orchestra.run.smithi099.stdout: "max_write_bytes": 0, 2023-05-31T21:31:27.113 INFO:teuthology.orchestra.run.smithi099.stdout: "enabled": false 2023-05-31T21:31:27.113 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T21:31:27.113 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:31:27.114 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_id": "aaa33de6-515b-4cac-8f1b-3a1a30452117", 2023-05-31T21:31:27.114 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_name": "test-realm", 2023-05-31T21:31:27.114 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2023-05-31T21:31:27.114 INFO:teuthology.orchestra.run.smithi099.stdout:} 2023-05-31T21:31:27.115 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'policy', 'get', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-05-31T21:31:27.115 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'policy', 'get', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-05-31T21:31:27.116 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync policy get --cluster c1 --debug-rgw 0 --debug-ms 0 2023-05-31T21:31:27.161 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2023-05-31T21:31:27.162 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2023-05-31T21:31:27.250 INFO:teuthology.orchestra.run.smithi099.stdout:{"groups":[{"id":"sync-group","data_flow":{"symmetrical":[{"id":"sync-flow1","zones":["a1","a2"]}]},"pipes":[{"id":"sync-pipe","source":{"bucket":"pklxkd-74","zones":["a1","a2"]},"dest":{"bucket":"pklxkd-75","zones":["a1","a2"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":""}},{"id":"sync-pipe-pklxkd-74","source":{"bucket":"pklxkd-74","zones":["a1"]},"dest":{"bucket":"pklxkd-74","zones":["a2"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":""}},{"id":"sync-pipe-pklxkd-75","source":{"bucket":"pklxkd-74","zones":["a1"]},"dest":{"bucket":"pklxkd-75","zones":["a2"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":""}}],"status":"enabled"}]} 2023-05-31T21:31:27.251 DEBUG:boto:path=/pklxkd-74/ 2023-05-31T21:31:27.251 DEBUG:boto:auth_path=/pklxkd-74/ 2023-05-31T21:31:27.252 DEBUG:boto:Method: HEAD 2023-05-31T21:31:27.252 DEBUG:boto:Path: /pklxkd-74/ 2023-05-31T21:31:27.252 DEBUG:boto:Data: 2023-05-31T21:31:27.253 DEBUG:boto:Headers: {} 2023-05-31T21:31:27.253 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T21:31:27.253 DEBUG:boto:Port: 8000 2023-05-31T21:31:27.254 DEBUG:boto:Params: {} 2023-05-31T21:31:27.254 DEBUG:boto:Token: None 2023-05-31T21:31:27.255 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 21:31:27 GMT /pklxkd-74/ 2023-05-31T21:31:27.255 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:yfZyB8rxTsFCohn/0NqfV7OMCeg= 2023-05-31T21:31:27.255 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 21:31:27 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:yfZyB8rxTsFCohn/0NqfV7OMCeg=', 'Content-Length': '0'} 2023-05-31T21:31:27.364 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000081c645fd7923446f-006477bcaf-2ae8-a1'), ('Content-Length', '0'), ('Date', 'Wed, 31 May 2023 21:31:27 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:31:27.365 DEBUG:boto:path=/pklxkd-74/obj1 2023-05-31T21:31:27.365 DEBUG:boto:auth_path=/pklxkd-74/obj1 2023-05-31T21:31:27.365 DEBUG:boto:Method: PUT 2023-05-31T21:31:27.366 DEBUG:boto:Path: /pklxkd-74/obj1 2023-05-31T21:31:27.366 DEBUG:boto:Data: 2023-05-31T21:31:27.366 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2023-05-31T21:31:27.367 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T21:31:27.367 DEBUG:boto:Port: 8000 2023-05-31T21:31:27.367 DEBUG:boto:Params: {} 2023-05-31T21:31:27.368 DEBUG:boto:Token: None 2023-05-31T21:31:27.368 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Wed, 31 May 2023 21:31:27 GMT /pklxkd-74/obj1 2023-05-31T21:31:27.369 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:HB5yYXAk5BuhTAdk272wLsxD6T0= 2023-05-31T21:31:27.369 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Wed, 31 May 2023 21:31:27 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:HB5yYXAk5BuhTAdk272wLsxD6T0='} 2023-05-31T21:31:27.393 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000003f43520b3c725e03-006477bcaf-2ae8-a1'), ('Date', 'Wed, 31 May 2023 21:31:27 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:31:27.393 DEBUG:boto:path=/pklxkd-74/ 2023-05-31T21:31:27.393 DEBUG:boto:auth_path=/pklxkd-74/ 2023-05-31T21:31:27.394 DEBUG:boto:Method: HEAD 2023-05-31T21:31:27.394 DEBUG:boto:Path: /pklxkd-74/ 2023-05-31T21:31:27.394 DEBUG:boto:Data: 2023-05-31T21:31:27.395 DEBUG:boto:Headers: {} 2023-05-31T21:31:27.395 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T21:31:27.395 DEBUG:boto:Port: 8000 2023-05-31T21:31:27.396 DEBUG:boto:Params: {} 2023-05-31T21:31:27.396 DEBUG:boto:Token: None 2023-05-31T21:31:27.397 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 21:31:27 GMT /pklxkd-74/ 2023-05-31T21:31:27.397 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:yfZyB8rxTsFCohn/0NqfV7OMCeg= 2023-05-31T21:31:27.397 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 21:31:27 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:yfZyB8rxTsFCohn/0NqfV7OMCeg=', 'Content-Length': '0'} 2023-05-31T21:31:27.400 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '6'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000002e97d0b1a7addae5-006477bcaf-2ae8-a1'), ('Content-Length', '0'), ('Date', 'Wed, 31 May 2023 21:31:27 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:31:27.400 DEBUG:boto:path=/pklxkd-74/obj2 2023-05-31T21:31:27.401 DEBUG:boto:auth_path=/pklxkd-74/obj2 2023-05-31T21:31:27.401 DEBUG:boto:Method: PUT 2023-05-31T21:31:27.402 DEBUG:boto:Path: /pklxkd-74/obj2 2023-05-31T21:31:27.402 DEBUG:boto:Data: 2023-05-31T21:31:27.402 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2023-05-31T21:31:27.403 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T21:31:27.403 DEBUG:boto:Port: 8000 2023-05-31T21:31:27.403 DEBUG:boto:Params: {} 2023-05-31T21:31:27.404 DEBUG:boto:Token: None 2023-05-31T21:31:27.404 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Wed, 31 May 2023 21:31:27 GMT /pklxkd-74/obj2 2023-05-31T21:31:27.404 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:YntFmY7Vgd1nHRKPpGS4CCkkKfI= 2023-05-31T21:31:27.405 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Wed, 31 May 2023 21:31:27 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:YntFmY7Vgd1nHRKPpGS4CCkkKfI='} 2023-05-31T21:31:27.449 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000d0ad0a785e6416de-006477bcaf-2ae8-a1'), ('Date', 'Wed, 31 May 2023 21:31:27 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:31:27.449 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-31T21:31:27.450 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-31T21:31:27.450 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-05-31T21:31:27.516 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2023-05-31T21:31:27.517 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2023-05-31T21:31:27.602 INFO:teuthology.orchestra.run.smithi099.stderr:No --period given, using current period=2e6123d2-d695-4350-9a01-61a455bc782b 2023-05-31T21:31:27.663 INFO:teuthology.orchestra.run.smithi099.stdout:[ 2023-05-31T21:31:27.663 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T21:31:27.663 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1685566501.757183_260.1", 2023-05-31T21:31:27.663 INFO:teuthology.orchestra.run.smithi099.stdout: "last_update": "2023-05-31T20:55:01.757183Z" 2023-05-31T21:31:27.664 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:31:27.664 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T21:31:27.664 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1685566501.735717_251.1", 2023-05-31T21:31:27.664 INFO:teuthology.orchestra.run.smithi099.stdout: "last_update": "2023-05-31T20:55:01.735717Z" 2023-05-31T21:31:27.664 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:31:27.664 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T21:31:27.665 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1685562970.274572_93.1", 2023-05-31T21:31:27.665 INFO:teuthology.orchestra.run.smithi099.stdout: "last_update": "2023-05-31T19:56:10.274572Z" 2023-05-31T21:31:27.665 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:31:27.665 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T21:31:27.665 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1685568685.966725_1044.1", 2023-05-31T21:31:27.665 INFO:teuthology.orchestra.run.smithi099.stdout: "last_update": "2023-05-31T21:31:25.966725Z" 2023-05-31T21:31:27.665 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T21:31:27.666 INFO:teuthology.orchestra.run.smithi099.stdout:] 2023-05-31T21:31:27.666 DEBUG:rgw_multi.tests:master meta markers={0: '1_1685566501.757183_260.1', 1: '1_1685566501.735717_251.1', 2: '1_1685562970.274572_93.1', 3: '1_1685568685.966725_1044.1'} 2023-05-31T21:31:27.667 INFO:rgw_multi.tests:starting meta checkpoint for zone=a2 2023-05-31T21:31:27.667 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-31T21:31:27.668 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-31T21:31:27.668 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-05-31T21:31:27.738 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2023-05-31T21:31:27.738 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2023-05-31T21:31:27.839 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T21:31:27.835+0000 7f411ae0b780 20 RGW-SYNC:meta: read sync status 2023-05-31T21:31:27.840 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T21:31:27.837+0000 7f411ae0b780 20 RGW-SYNC:meta: read sync status 2023-05-31T21:31:27.870 INFO:teuthology.orchestra.run.smithi099.stdout:{ 2023-05-31T21:31:27.870 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_status": { 2023-05-31T21:31:27.871 INFO:teuthology.orchestra.run.smithi099.stdout: "info": { 2023-05-31T21:31:27.871 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "sync", 2023-05-31T21:31:27.871 INFO:teuthology.orchestra.run.smithi099.stdout: "num_shards": 4, 2023-05-31T21:31:27.871 INFO:teuthology.orchestra.run.smithi099.stdout: "period": "2e6123d2-d695-4350-9a01-61a455bc782b", 2023-05-31T21:31:27.871 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2023-05-31T21:31:27.871 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:31:27.872 INFO:teuthology.orchestra.run.smithi099.stdout: "markers": [ 2023-05-31T21:31:27.872 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T21:31:27.872 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 0, 2023-05-31T21:31:27.872 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T21:31:27.872 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2023-05-31T21:31:27.872 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1685566501.757183_260.1", 2023-05-31T21:31:27.873 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T21:31:27.873 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 1, 2023-05-31T21:31:27.873 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T21:31:27.873 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2023-05-31T20:55:01.757183Z", 2023-05-31T21:31:27.873 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2023-05-31T21:31:27.873 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T21:31:27.873 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:31:27.874 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T21:31:27.874 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 1, 2023-05-31T21:31:27.874 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T21:31:27.874 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2023-05-31T21:31:27.874 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1685566501.735717_251.1", 2023-05-31T21:31:27.874 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T21:31:27.874 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2023-05-31T21:31:27.875 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T21:31:27.875 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2023-05-31T20:55:01.735717Z", 2023-05-31T21:31:27.875 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2023-05-31T21:31:27.875 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T21:31:27.875 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:31:27.875 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T21:31:27.875 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 2, 2023-05-31T21:31:27.876 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T21:31:27.876 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2023-05-31T21:31:27.876 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1685562970.274572_93.1", 2023-05-31T21:31:27.876 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T21:31:27.876 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2023-05-31T21:31:27.876 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T21:31:27.876 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2023-05-31T19:56:10.274572Z", 2023-05-31T21:31:27.877 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2023-05-31T21:31:27.877 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T21:31:27.877 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:31:27.877 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T21:31:27.877 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 3, 2023-05-31T21:31:27.877 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T21:31:27.877 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2023-05-31T21:31:27.878 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1685568685.966725_1044.1", 2023-05-31T21:31:27.878 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T21:31:27.878 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2023-05-31T21:31:27.878 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T21:31:27.878 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2023-05-31T21:31:25.966725Z", 2023-05-31T21:31:27.878 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2023-05-31T21:31:27.878 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T21:31:27.879 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T21:31:27.879 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2023-05-31T21:31:27.879 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:31:27.879 INFO:teuthology.orchestra.run.smithi099.stdout: "full_sync": { 2023-05-31T21:31:27.879 INFO:teuthology.orchestra.run.smithi099.stdout: "total": 1, 2023-05-31T21:31:27.879 INFO:teuthology.orchestra.run.smithi099.stdout: "complete": 1 2023-05-31T21:31:27.879 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:31:27.880 INFO:teuthology.orchestra.run.smithi099.stdout: "current_time": "2023-05-31T21:31:27Z" 2023-05-31T21:31:27.880 INFO:teuthology.orchestra.run.smithi099.stdout:} 2023-05-31T21:31:27.881 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "2e6123d2-d695-4350-9a01-61a455bc782b", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1685566501.757183_260.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-05-31T20:55:01.757183Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1685566501.735717_251.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-05-31T20:55:01.735717Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1685562970.274572_93.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-05-31T19:56:10.274572Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1685568685.966725_1044.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-05-31T21:31:25.966725Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-05-31T21:31:27Z" } 2023-05-31T21:31:27.881 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1685566501.757183_260.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-05-31T20:55:01.757183Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1685566501.735717_251.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-05-31T20:55:01.735717Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1685562970.274572_93.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-05-31T19:56:10.274572Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1685568685.966725_1044.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-05-31T21:31:25.966725Z', 'realm_epoch': 2}}] 2023-05-31T21:31:27.882 DEBUG:rgw_multi.tests:log_status={0: '1_1685566501.757183_260.1', 1: '1_1685566501.735717_251.1', 2: '1_1685562970.274572_93.1', 3: '1_1685568685.966725_1044.1'} 2023-05-31T21:31:27.882 DEBUG:rgw_multi.tests:sync_status={0: '1_1685566501.757183_260.1', 1: '1_1685566501.735717_251.1', 2: '1_1685562970.274572_93.1', 3: '1_1685568685.966725_1044.1'} 2023-05-31T21:31:27.882 INFO:rgw_multi.tests:finish meta checkpoint for zone=a2 2023-05-31T21:31:27.883 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['datalog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-31T21:31:27.883 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'datalog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-31T21:31:27.883 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 datalog status --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-05-31T21:31:27.940 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2023-05-31T21:31:27.940 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2023-05-31T21:31:28.056 INFO:teuthology.orchestra.run.smithi099.stdout:[ 2023-05-31T21:31:28.057 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T21:31:28.057 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "00000000000000000000:00000000000000045655", 2023-05-31T21:31:28.057 INFO:teuthology.orchestra.run.smithi099.stdout: "last_update": "2023-05-31T21:31:27.421078Z" 2023-05-31T21:31:28.057 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:31:28.057 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T21:31:28.058 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "00000000000000000000:00000000000000031920", 2023-05-31T21:31:28.058 INFO:teuthology.orchestra.run.smithi099.stdout: "last_update": "2023-05-31T21:29:57.583636Z" 2023-05-31T21:31:28.058 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:31:28.058 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T21:31:28.058 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "00000000000000000000:00000000000000052091", 2023-05-31T21:31:28.058 INFO:teuthology.orchestra.run.smithi099.stdout: "last_update": "2023-05-31T21:31:27.385416Z" 2023-05-31T21:31:28.058 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:31:28.059 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T21:31:28.059 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "00000000000000000000:00000000000000044178", 2023-05-31T21:31:28.059 INFO:teuthology.orchestra.run.smithi099.stdout: "last_update": "2023-05-31T21:27:37.215905Z" 2023-05-31T21:31:28.059 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T21:31:28.059 INFO:teuthology.orchestra.run.smithi099.stdout:] 2023-05-31T21:31:28.060 DEBUG:rgw_multi.tests:data markers for zone=a1 markers={0: '00000000000000000000:00000000000000045655', 1: '00000000000000000000:00000000000000031920', 2: '00000000000000000000:00000000000000052091', 3: '00000000000000000000:00000000000000044178'} 2023-05-31T21:31:28.061 INFO:rgw_multi.tests:starting data checkpoint for target_zone=a2 source_zone=a1 2023-05-31T21:31:28.061 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-31T21:31:28.061 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-31T21:31:28.062 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-05-31T21:31:28.119 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2023-05-31T21:31:28.120 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2023-05-31T21:31:28.200 INFO:teuthology.orchestra.run.smithi099.stdout:{ 2023-05-31T21:31:28.201 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_status": { 2023-05-31T21:31:28.201 INFO:teuthology.orchestra.run.smithi099.stdout: "info": { 2023-05-31T21:31:28.201 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "sync", 2023-05-31T21:31:28.201 INFO:teuthology.orchestra.run.smithi099.stdout: "num_shards": 4, 2023-05-31T21:31:28.201 INFO:teuthology.orchestra.run.smithi099.stdout: "instance_id": 452136274422638824 2023-05-31T21:31:28.201 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:31:28.202 INFO:teuthology.orchestra.run.smithi099.stdout: "markers": [ 2023-05-31T21:31:28.202 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T21:31:28.202 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 0, 2023-05-31T21:31:28.202 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T21:31:28.202 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "incremental-sync", 2023-05-31T21:31:28.202 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "00000000000000000000:00000000000000045515", 2023-05-31T21:31:28.203 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T21:31:28.203 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2023-05-31T21:31:28.203 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T21:31:28.203 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2023-05-31T21:29:57.506834Z" 2023-05-31T21:31:28.203 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T21:31:28.203 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:31:28.204 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T21:31:28.204 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 1, 2023-05-31T21:31:28.204 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T21:31:28.204 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "incremental-sync", 2023-05-31T21:31:28.204 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "00000000000000000000:00000000000000031920", 2023-05-31T21:31:28.204 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T21:31:28.204 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2023-05-31T21:31:28.205 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T21:31:28.205 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2023-05-31T21:29:57.583636Z" 2023-05-31T21:31:28.205 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T21:31:28.205 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:31:28.205 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T21:31:28.205 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 2, 2023-05-31T21:31:28.206 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T21:31:28.206 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "incremental-sync", 2023-05-31T21:31:28.206 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "00000000000000000000:00000000000000051951", 2023-05-31T21:31:28.206 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T21:31:28.206 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 55, 2023-05-31T21:31:28.207 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T21:31:28.207 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2023-05-31T21:29:57.526899Z" 2023-05-31T21:31:28.207 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T21:31:28.207 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:31:28.207 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T21:31:28.208 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 3, 2023-05-31T21:31:28.208 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T21:31:28.208 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "incremental-sync", 2023-05-31T21:31:28.208 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "00000000000000000000:00000000000000044178", 2023-05-31T21:31:28.208 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T21:31:28.208 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 9, 2023-05-31T21:31:28.208 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T21:31:28.209 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2023-05-31T21:27:37.215905Z" 2023-05-31T21:31:28.209 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T21:31:28.209 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T21:31:28.209 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2023-05-31T21:31:28.209 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:31:28.209 INFO:teuthology.orchestra.run.smithi099.stdout: "full_sync": { 2023-05-31T21:31:28.210 INFO:teuthology.orchestra.run.smithi099.stdout: "total": 64, 2023-05-31T21:31:28.210 INFO:teuthology.orchestra.run.smithi099.stdout: "complete": 64 2023-05-31T21:31:28.210 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:31:28.210 INFO:teuthology.orchestra.run.smithi099.stdout: "current_time": "2023-05-31T21:31:28Z" 2023-05-31T21:31:28.210 INFO:teuthology.orchestra.run.smithi099.stdout:} 2023-05-31T21:31:28.211 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 452136274422638824 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000045515", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-05-31T21:29:57.506834Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000031920", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-05-31T21:29:57.583636Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000051951", "next_step_marker": "", "total_entries": 55, "pos": 0, "timestamp": "2023-05-31T21:29:57.526899Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000044178", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2023-05-31T21:27:37.215905Z" } } ] }, "full_sync": { "total": 64, "complete": 64 }, "current_time": "2023-05-31T21:31:28Z" } 2023-05-31T21:31:28.212 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000045515', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-05-31T21:29:57.506834Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000031920', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-05-31T21:29:57.583636Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000051951', 'next_step_marker': '', 'total_entries': 55, 'pos': 0, 'timestamp': '2023-05-31T21:29:57.526899Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000044178', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2023-05-31T21:27:37.215905Z'}}] 2023-05-31T21:31:28.212 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000045655', 1: '00000000000000000000:00000000000000031920', 2: '00000000000000000000:00000000000000052091', 3: '00000000000000000000:00000000000000044178'} 2023-05-31T21:31:28.212 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000045515', 1: '00000000000000000000:00000000000000031920', 2: '00000000000000000000:00000000000000051951', 3: '00000000000000000000:00000000000000044178'} 2023-05-31T21:31:28.213 WARNING:rgw_multi.tests:data of zone a2 behind zone a1: shard=0 master=00000000000000000000:00000000000000045655 target=00000000000000000000:00000000000000045515, shard=2 master=00000000000000000000:00000000000000052091 target=00000000000000000000:00000000000000051951 2023-05-31T21:31:33.214 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-31T21:31:33.215 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-31T21:31:33.216 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-05-31T21:31:33.275 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2023-05-31T21:31:33.276 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2023-05-31T21:31:33.332 INFO:teuthology.orchestra.run.smithi099.stdout:{ 2023-05-31T21:31:33.333 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_status": { 2023-05-31T21:31:33.333 INFO:teuthology.orchestra.run.smithi099.stdout: "info": { 2023-05-31T21:31:33.333 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "sync", 2023-05-31T21:31:33.334 INFO:teuthology.orchestra.run.smithi099.stdout: "num_shards": 4, 2023-05-31T21:31:33.334 INFO:teuthology.orchestra.run.smithi099.stdout: "instance_id": 452136274422638824 2023-05-31T21:31:33.334 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:31:33.334 INFO:teuthology.orchestra.run.smithi099.stdout: "markers": [ 2023-05-31T21:31:33.335 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T21:31:33.335 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 0, 2023-05-31T21:31:33.335 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T21:31:33.335 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "incremental-sync", 2023-05-31T21:31:33.335 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "00000000000000000000:00000000000000045515", 2023-05-31T21:31:33.335 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T21:31:33.335 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2023-05-31T21:31:33.336 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T21:31:33.336 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2023-05-31T21:29:57.506834Z" 2023-05-31T21:31:33.336 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T21:31:33.336 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:31:33.336 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T21:31:33.336 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 1, 2023-05-31T21:31:33.337 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T21:31:33.337 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "incremental-sync", 2023-05-31T21:31:33.337 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "00000000000000000000:00000000000000031920", 2023-05-31T21:31:33.337 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T21:31:33.337 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2023-05-31T21:31:33.337 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T21:31:33.337 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2023-05-31T21:29:57.583636Z" 2023-05-31T21:31:33.338 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T21:31:33.338 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:31:33.338 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T21:31:33.338 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 2, 2023-05-31T21:31:33.338 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T21:31:33.338 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "incremental-sync", 2023-05-31T21:31:33.338 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "00000000000000000000:00000000000000051951", 2023-05-31T21:31:33.339 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T21:31:33.339 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 55, 2023-05-31T21:31:33.339 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T21:31:33.339 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2023-05-31T21:29:57.526899Z" 2023-05-31T21:31:33.339 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T21:31:33.339 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:31:33.339 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T21:31:33.340 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 3, 2023-05-31T21:31:33.340 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T21:31:33.340 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "incremental-sync", 2023-05-31T21:31:33.340 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "00000000000000000000:00000000000000044178", 2023-05-31T21:31:33.340 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T21:31:33.340 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 9, 2023-05-31T21:31:33.340 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T21:31:33.341 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2023-05-31T21:27:37.215905Z" 2023-05-31T21:31:33.341 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T21:31:33.341 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T21:31:33.341 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2023-05-31T21:31:33.341 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:31:33.341 INFO:teuthology.orchestra.run.smithi099.stdout: "full_sync": { 2023-05-31T21:31:33.341 INFO:teuthology.orchestra.run.smithi099.stdout: "total": 64, 2023-05-31T21:31:33.342 INFO:teuthology.orchestra.run.smithi099.stdout: "complete": 64 2023-05-31T21:31:33.342 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:31:33.342 INFO:teuthology.orchestra.run.smithi099.stdout: "current_time": "2023-05-31T21:31:33Z" 2023-05-31T21:31:33.342 INFO:teuthology.orchestra.run.smithi099.stdout:} 2023-05-31T21:31:33.343 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 452136274422638824 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000045515", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-05-31T21:29:57.506834Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000031920", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-05-31T21:29:57.583636Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000051951", "next_step_marker": "", "total_entries": 55, "pos": 0, "timestamp": "2023-05-31T21:29:57.526899Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000044178", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2023-05-31T21:27:37.215905Z" } } ] }, "full_sync": { "total": 64, "complete": 64 }, "current_time": "2023-05-31T21:31:33Z" } 2023-05-31T21:31:33.343 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000045515', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-05-31T21:29:57.506834Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000031920', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-05-31T21:29:57.583636Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000051951', 'next_step_marker': '', 'total_entries': 55, 'pos': 0, 'timestamp': '2023-05-31T21:29:57.526899Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000044178', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2023-05-31T21:27:37.215905Z'}}] 2023-05-31T21:31:33.344 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000045655', 1: '00000000000000000000:00000000000000031920', 2: '00000000000000000000:00000000000000052091', 3: '00000000000000000000:00000000000000044178'} 2023-05-31T21:31:33.344 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000045515', 1: '00000000000000000000:00000000000000031920', 2: '00000000000000000000:00000000000000051951', 3: '00000000000000000000:00000000000000044178'} 2023-05-31T21:31:33.344 WARNING:rgw_multi.tests:data of zone a2 behind zone a1: shard=0 master=00000000000000000000:00000000000000045655 target=00000000000000000000:00000000000000045515, shard=2 master=00000000000000000000:00000000000000052091 target=00000000000000000000:00000000000000051951 2023-05-31T21:31:38.346 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-31T21:31:38.346 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-31T21:31:38.347 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-05-31T21:31:38.405 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2023-05-31T21:31:38.406 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2023-05-31T21:31:38.481 INFO:teuthology.orchestra.run.smithi099.stdout:{ 2023-05-31T21:31:38.481 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_status": { 2023-05-31T21:31:38.481 INFO:teuthology.orchestra.run.smithi099.stdout: "info": { 2023-05-31T21:31:38.482 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "sync", 2023-05-31T21:31:38.482 INFO:teuthology.orchestra.run.smithi099.stdout: "num_shards": 4, 2023-05-31T21:31:38.482 INFO:teuthology.orchestra.run.smithi099.stdout: "instance_id": 452136274422638824 2023-05-31T21:31:38.482 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:31:38.482 INFO:teuthology.orchestra.run.smithi099.stdout: "markers": [ 2023-05-31T21:31:38.482 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T21:31:38.483 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 0, 2023-05-31T21:31:38.483 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T21:31:38.483 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "incremental-sync", 2023-05-31T21:31:38.483 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "00000000000000000000:00000000000000045515", 2023-05-31T21:31:38.483 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T21:31:38.483 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2023-05-31T21:31:38.484 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T21:31:38.484 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2023-05-31T21:29:57.506834Z" 2023-05-31T21:31:38.484 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T21:31:38.484 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:31:38.484 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T21:31:38.484 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 1, 2023-05-31T21:31:38.484 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T21:31:38.485 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "incremental-sync", 2023-05-31T21:31:38.485 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "00000000000000000000:00000000000000031920", 2023-05-31T21:31:38.485 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T21:31:38.485 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2023-05-31T21:31:38.485 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T21:31:38.485 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2023-05-31T21:29:57.583636Z" 2023-05-31T21:31:38.485 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T21:31:38.486 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:31:38.486 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T21:31:38.486 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 2, 2023-05-31T21:31:38.486 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T21:31:38.486 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "incremental-sync", 2023-05-31T21:31:38.486 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "00000000000000000000:00000000000000051951", 2023-05-31T21:31:38.486 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T21:31:38.487 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 55, 2023-05-31T21:31:38.487 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T21:31:38.487 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2023-05-31T21:29:57.526899Z" 2023-05-31T21:31:38.487 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T21:31:38.487 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:31:38.487 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T21:31:38.487 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 3, 2023-05-31T21:31:38.488 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T21:31:38.488 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "incremental-sync", 2023-05-31T21:31:38.488 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "00000000000000000000:00000000000000044178", 2023-05-31T21:31:38.488 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T21:31:38.488 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 9, 2023-05-31T21:31:38.488 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T21:31:38.489 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2023-05-31T21:27:37.215905Z" 2023-05-31T21:31:38.489 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T21:31:38.489 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T21:31:38.489 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2023-05-31T21:31:38.489 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:31:38.489 INFO:teuthology.orchestra.run.smithi099.stdout: "full_sync": { 2023-05-31T21:31:38.489 INFO:teuthology.orchestra.run.smithi099.stdout: "total": 64, 2023-05-31T21:31:38.490 INFO:teuthology.orchestra.run.smithi099.stdout: "complete": 64 2023-05-31T21:31:38.490 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:31:38.490 INFO:teuthology.orchestra.run.smithi099.stdout: "current_time": "2023-05-31T21:31:38Z" 2023-05-31T21:31:38.490 INFO:teuthology.orchestra.run.smithi099.stdout:} 2023-05-31T21:31:38.491 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 452136274422638824 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000045515", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-05-31T21:29:57.506834Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000031920", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-05-31T21:29:57.583636Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000051951", "next_step_marker": "", "total_entries": 55, "pos": 0, "timestamp": "2023-05-31T21:29:57.526899Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000044178", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2023-05-31T21:27:37.215905Z" } } ] }, "full_sync": { "total": 64, "complete": 64 }, "current_time": "2023-05-31T21:31:38Z" } 2023-05-31T21:31:38.491 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000045515', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-05-31T21:29:57.506834Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000031920', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-05-31T21:29:57.583636Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000051951', 'next_step_marker': '', 'total_entries': 55, 'pos': 0, 'timestamp': '2023-05-31T21:29:57.526899Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000044178', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2023-05-31T21:27:37.215905Z'}}] 2023-05-31T21:31:38.492 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000045655', 1: '00000000000000000000:00000000000000031920', 2: '00000000000000000000:00000000000000052091', 3: '00000000000000000000:00000000000000044178'} 2023-05-31T21:31:38.492 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000045515', 1: '00000000000000000000:00000000000000031920', 2: '00000000000000000000:00000000000000051951', 3: '00000000000000000000:00000000000000044178'} 2023-05-31T21:31:38.492 WARNING:rgw_multi.tests:data of zone a2 behind zone a1: shard=0 master=00000000000000000000:00000000000000045655 target=00000000000000000000:00000000000000045515, shard=2 master=00000000000000000000:00000000000000052091 target=00000000000000000000:00000000000000051951 2023-05-31T21:31:43.493 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-31T21:31:43.494 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-31T21:31:43.494 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-05-31T21:31:43.553 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2023-05-31T21:31:43.554 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2023-05-31T21:31:43.616 INFO:teuthology.orchestra.run.smithi099.stdout:{ 2023-05-31T21:31:43.616 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_status": { 2023-05-31T21:31:43.616 INFO:teuthology.orchestra.run.smithi099.stdout: "info": { 2023-05-31T21:31:43.617 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "sync", 2023-05-31T21:31:43.617 INFO:teuthology.orchestra.run.smithi099.stdout: "num_shards": 4, 2023-05-31T21:31:43.617 INFO:teuthology.orchestra.run.smithi099.stdout: "instance_id": 452136274422638824 2023-05-31T21:31:43.617 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:31:43.617 INFO:teuthology.orchestra.run.smithi099.stdout: "markers": [ 2023-05-31T21:31:43.617 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T21:31:43.618 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 0, 2023-05-31T21:31:43.618 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T21:31:43.618 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "incremental-sync", 2023-05-31T21:31:43.618 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "00000000000000000000:00000000000000045515", 2023-05-31T21:31:43.618 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T21:31:43.618 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2023-05-31T21:31:43.618 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T21:31:43.619 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2023-05-31T21:29:57.506834Z" 2023-05-31T21:31:43.619 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T21:31:43.619 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:31:43.619 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T21:31:43.619 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 1, 2023-05-31T21:31:43.619 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T21:31:43.620 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "incremental-sync", 2023-05-31T21:31:43.620 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "00000000000000000000:00000000000000031920", 2023-05-31T21:31:43.620 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T21:31:43.620 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2023-05-31T21:31:43.620 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T21:31:43.620 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2023-05-31T21:29:57.583636Z" 2023-05-31T21:31:43.620 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T21:31:43.621 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:31:43.621 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T21:31:43.621 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 2, 2023-05-31T21:31:43.621 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T21:31:43.621 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "incremental-sync", 2023-05-31T21:31:43.621 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "00000000000000000000:00000000000000051951", 2023-05-31T21:31:43.621 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T21:31:43.622 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 55, 2023-05-31T21:31:43.622 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T21:31:43.622 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2023-05-31T21:29:57.526899Z" 2023-05-31T21:31:43.622 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T21:31:43.622 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:31:43.622 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T21:31:43.623 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 3, 2023-05-31T21:31:43.623 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T21:31:43.623 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "incremental-sync", 2023-05-31T21:31:43.623 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "00000000000000000000:00000000000000044178", 2023-05-31T21:31:43.623 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T21:31:43.623 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 9, 2023-05-31T21:31:43.623 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T21:31:43.624 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2023-05-31T21:27:37.215905Z" 2023-05-31T21:31:43.624 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T21:31:43.624 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T21:31:43.624 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2023-05-31T21:31:43.624 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:31:43.624 INFO:teuthology.orchestra.run.smithi099.stdout: "full_sync": { 2023-05-31T21:31:43.624 INFO:teuthology.orchestra.run.smithi099.stdout: "total": 64, 2023-05-31T21:31:43.625 INFO:teuthology.orchestra.run.smithi099.stdout: "complete": 64 2023-05-31T21:31:43.625 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:31:43.625 INFO:teuthology.orchestra.run.smithi099.stdout: "current_time": "2023-05-31T21:31:43Z" 2023-05-31T21:31:43.625 INFO:teuthology.orchestra.run.smithi099.stdout:} 2023-05-31T21:31:43.626 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 452136274422638824 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000045515", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-05-31T21:29:57.506834Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000031920", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-05-31T21:29:57.583636Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000051951", "next_step_marker": "", "total_entries": 55, "pos": 0, "timestamp": "2023-05-31T21:29:57.526899Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000044178", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2023-05-31T21:27:37.215905Z" } } ] }, "full_sync": { "total": 64, "complete": 64 }, "current_time": "2023-05-31T21:31:43Z" } 2023-05-31T21:31:43.626 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000045515', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-05-31T21:29:57.506834Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000031920', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-05-31T21:29:57.583636Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000051951', 'next_step_marker': '', 'total_entries': 55, 'pos': 0, 'timestamp': '2023-05-31T21:29:57.526899Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000044178', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2023-05-31T21:27:37.215905Z'}}] 2023-05-31T21:31:43.627 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000045655', 1: '00000000000000000000:00000000000000031920', 2: '00000000000000000000:00000000000000052091', 3: '00000000000000000000:00000000000000044178'} 2023-05-31T21:31:43.627 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000045515', 1: '00000000000000000000:00000000000000031920', 2: '00000000000000000000:00000000000000051951', 3: '00000000000000000000:00000000000000044178'} 2023-05-31T21:31:43.627 WARNING:rgw_multi.tests:data of zone a2 behind zone a1: shard=0 master=00000000000000000000:00000000000000045655 target=00000000000000000000:00000000000000045515, shard=2 master=00000000000000000000:00000000000000052091 target=00000000000000000000:00000000000000051951 2023-05-31T21:31:48.628 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-31T21:31:48.629 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-31T21:31:48.630 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-05-31T21:31:48.689 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2023-05-31T21:31:48.689 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2023-05-31T21:31:48.742 INFO:teuthology.orchestra.run.smithi099.stdout:{ 2023-05-31T21:31:48.743 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_status": { 2023-05-31T21:31:48.743 INFO:teuthology.orchestra.run.smithi099.stdout: "info": { 2023-05-31T21:31:48.743 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "sync", 2023-05-31T21:31:48.743 INFO:teuthology.orchestra.run.smithi099.stdout: "num_shards": 4, 2023-05-31T21:31:48.743 INFO:teuthology.orchestra.run.smithi099.stdout: "instance_id": 452136274422638824 2023-05-31T21:31:48.744 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:31:48.744 INFO:teuthology.orchestra.run.smithi099.stdout: "markers": [ 2023-05-31T21:31:48.744 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T21:31:48.744 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 0, 2023-05-31T21:31:48.744 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T21:31:48.744 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "incremental-sync", 2023-05-31T21:31:48.745 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "00000000000000000000:00000000000000045515", 2023-05-31T21:31:48.745 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T21:31:48.745 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2023-05-31T21:31:48.745 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T21:31:48.745 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2023-05-31T21:29:57.506834Z" 2023-05-31T21:31:48.745 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T21:31:48.746 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:31:48.746 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T21:31:48.746 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 1, 2023-05-31T21:31:48.746 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T21:31:48.746 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "incremental-sync", 2023-05-31T21:31:48.747 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "00000000000000000000:00000000000000031920", 2023-05-31T21:31:48.747 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T21:31:48.747 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2023-05-31T21:31:48.747 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T21:31:48.748 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2023-05-31T21:29:57.583636Z" 2023-05-31T21:31:48.748 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T21:31:48.748 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:31:48.748 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T21:31:48.748 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 2, 2023-05-31T21:31:48.748 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T21:31:48.748 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "incremental-sync", 2023-05-31T21:31:48.749 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "00000000000000000000:00000000000000051951", 2023-05-31T21:31:48.749 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T21:31:48.749 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 55, 2023-05-31T21:31:48.749 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T21:31:48.749 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2023-05-31T21:29:57.526899Z" 2023-05-31T21:31:48.749 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T21:31:48.750 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:31:48.750 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T21:31:48.750 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 3, 2023-05-31T21:31:48.750 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T21:31:48.751 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "incremental-sync", 2023-05-31T21:31:48.751 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "00000000000000000000:00000000000000044178", 2023-05-31T21:31:48.751 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T21:31:48.752 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 9, 2023-05-31T21:31:48.752 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T21:31:48.753 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2023-05-31T21:27:37.215905Z" 2023-05-31T21:31:48.753 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T21:31:48.753 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T21:31:48.754 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2023-05-31T21:31:48.754 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:31:48.754 INFO:teuthology.orchestra.run.smithi099.stdout: "full_sync": { 2023-05-31T21:31:48.755 INFO:teuthology.orchestra.run.smithi099.stdout: "total": 64, 2023-05-31T21:31:48.755 INFO:teuthology.orchestra.run.smithi099.stdout: "complete": 64 2023-05-31T21:31:48.756 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:31:48.756 INFO:teuthology.orchestra.run.smithi099.stdout: "current_time": "2023-05-31T21:31:48Z" 2023-05-31T21:31:48.756 INFO:teuthology.orchestra.run.smithi099.stdout:} 2023-05-31T21:31:48.758 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 452136274422638824 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000045515", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-05-31T21:29:57.506834Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000031920", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-05-31T21:29:57.583636Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000051951", "next_step_marker": "", "total_entries": 55, "pos": 0, "timestamp": "2023-05-31T21:29:57.526899Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000044178", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2023-05-31T21:27:37.215905Z" } } ] }, "full_sync": { "total": 64, "complete": 64 }, "current_time": "2023-05-31T21:31:48Z" } 2023-05-31T21:31:48.759 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000045515', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-05-31T21:29:57.506834Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000031920', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-05-31T21:29:57.583636Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000051951', 'next_step_marker': '', 'total_entries': 55, 'pos': 0, 'timestamp': '2023-05-31T21:29:57.526899Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000044178', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2023-05-31T21:27:37.215905Z'}}] 2023-05-31T21:31:48.759 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000045655', 1: '00000000000000000000:00000000000000031920', 2: '00000000000000000000:00000000000000052091', 3: '00000000000000000000:00000000000000044178'} 2023-05-31T21:31:48.760 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000045515', 1: '00000000000000000000:00000000000000031920', 2: '00000000000000000000:00000000000000051951', 3: '00000000000000000000:00000000000000044178'} 2023-05-31T21:31:48.760 WARNING:rgw_multi.tests:data of zone a2 behind zone a1: shard=0 master=00000000000000000000:00000000000000045655 target=00000000000000000000:00000000000000045515, shard=2 master=00000000000000000000:00000000000000052091 target=00000000000000000000:00000000000000051951 2023-05-31T21:31:53.762 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-31T21:31:53.762 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-31T21:31:53.763 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-05-31T21:31:53.822 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2023-05-31T21:31:53.822 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2023-05-31T21:31:53.891 INFO:teuthology.orchestra.run.smithi099.stdout:{ 2023-05-31T21:31:53.892 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_status": { 2023-05-31T21:31:53.892 INFO:teuthology.orchestra.run.smithi099.stdout: "info": { 2023-05-31T21:31:53.892 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "sync", 2023-05-31T21:31:53.892 INFO:teuthology.orchestra.run.smithi099.stdout: "num_shards": 4, 2023-05-31T21:31:53.893 INFO:teuthology.orchestra.run.smithi099.stdout: "instance_id": 452136274422638824 2023-05-31T21:31:53.893 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:31:53.893 INFO:teuthology.orchestra.run.smithi099.stdout: "markers": [ 2023-05-31T21:31:53.893 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T21:31:53.893 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 0, 2023-05-31T21:31:53.893 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T21:31:53.893 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "incremental-sync", 2023-05-31T21:31:53.894 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "00000000000000000000:00000000000000045515", 2023-05-31T21:31:53.894 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T21:31:53.894 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2023-05-31T21:31:53.894 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T21:31:53.894 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2023-05-31T21:29:57.506834Z" 2023-05-31T21:31:53.894 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T21:31:53.895 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:31:53.895 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T21:31:53.895 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 1, 2023-05-31T21:31:53.895 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T21:31:53.895 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "incremental-sync", 2023-05-31T21:31:53.895 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "00000000000000000000:00000000000000031920", 2023-05-31T21:31:53.895 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T21:31:53.896 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2023-05-31T21:31:53.896 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T21:31:53.896 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2023-05-31T21:29:57.583636Z" 2023-05-31T21:31:53.896 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T21:31:53.896 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:31:53.896 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T21:31:53.896 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 2, 2023-05-31T21:31:53.897 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T21:31:53.897 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "incremental-sync", 2023-05-31T21:31:53.897 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "00000000000000000000:00000000000000051951", 2023-05-31T21:31:53.897 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T21:31:53.897 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 55, 2023-05-31T21:31:53.897 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T21:31:53.897 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2023-05-31T21:29:57.526899Z" 2023-05-31T21:31:53.898 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T21:31:53.898 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:31:53.898 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T21:31:53.898 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 3, 2023-05-31T21:31:53.898 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T21:31:53.898 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "incremental-sync", 2023-05-31T21:31:53.898 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "00000000000000000000:00000000000000044178", 2023-05-31T21:31:53.899 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T21:31:53.899 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 9, 2023-05-31T21:31:53.899 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T21:31:53.899 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2023-05-31T21:27:37.215905Z" 2023-05-31T21:31:53.899 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T21:31:53.899 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T21:31:53.900 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2023-05-31T21:31:53.900 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:31:53.900 INFO:teuthology.orchestra.run.smithi099.stdout: "full_sync": { 2023-05-31T21:31:53.900 INFO:teuthology.orchestra.run.smithi099.stdout: "total": 64, 2023-05-31T21:31:53.900 INFO:teuthology.orchestra.run.smithi099.stdout: "complete": 64 2023-05-31T21:31:53.900 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:31:53.900 INFO:teuthology.orchestra.run.smithi099.stdout: "current_time": "2023-05-31T21:31:53Z" 2023-05-31T21:31:53.901 INFO:teuthology.orchestra.run.smithi099.stdout:} 2023-05-31T21:31:53.902 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 452136274422638824 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000045515", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-05-31T21:29:57.506834Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000031920", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-05-31T21:29:57.583636Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000051951", "next_step_marker": "", "total_entries": 55, "pos": 0, "timestamp": "2023-05-31T21:29:57.526899Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000044178", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2023-05-31T21:27:37.215905Z" } } ] }, "full_sync": { "total": 64, "complete": 64 }, "current_time": "2023-05-31T21:31:53Z" } 2023-05-31T21:31:53.902 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000045515', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-05-31T21:29:57.506834Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000031920', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-05-31T21:29:57.583636Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000051951', 'next_step_marker': '', 'total_entries': 55, 'pos': 0, 'timestamp': '2023-05-31T21:29:57.526899Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000044178', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2023-05-31T21:27:37.215905Z'}}] 2023-05-31T21:31:53.902 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000045655', 1: '00000000000000000000:00000000000000031920', 2: '00000000000000000000:00000000000000052091', 3: '00000000000000000000:00000000000000044178'} 2023-05-31T21:31:53.903 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000045515', 1: '00000000000000000000:00000000000000031920', 2: '00000000000000000000:00000000000000051951', 3: '00000000000000000000:00000000000000044178'} 2023-05-31T21:31:53.903 WARNING:rgw_multi.tests:data of zone a2 behind zone a1: shard=0 master=00000000000000000000:00000000000000045655 target=00000000000000000000:00000000000000045515, shard=2 master=00000000000000000000:00000000000000052091 target=00000000000000000000:00000000000000051951 2023-05-31T21:31:58.904 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-31T21:31:58.905 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-31T21:31:58.905 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-05-31T21:31:58.964 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2023-05-31T21:31:58.965 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2023-05-31T21:31:59.018 INFO:teuthology.orchestra.run.smithi099.stdout:{ 2023-05-31T21:31:59.018 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_status": { 2023-05-31T21:31:59.018 INFO:teuthology.orchestra.run.smithi099.stdout: "info": { 2023-05-31T21:31:59.018 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "sync", 2023-05-31T21:31:59.019 INFO:teuthology.orchestra.run.smithi099.stdout: "num_shards": 4, 2023-05-31T21:31:59.019 INFO:teuthology.orchestra.run.smithi099.stdout: "instance_id": 452136274422638824 2023-05-31T21:31:59.019 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:31:59.019 INFO:teuthology.orchestra.run.smithi099.stdout: "markers": [ 2023-05-31T21:31:59.019 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T21:31:59.019 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 0, 2023-05-31T21:31:59.019 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T21:31:59.020 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "incremental-sync", 2023-05-31T21:31:59.020 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "00000000000000000000:00000000000000045515", 2023-05-31T21:31:59.020 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T21:31:59.020 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2023-05-31T21:31:59.020 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T21:31:59.020 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2023-05-31T21:29:57.506834Z" 2023-05-31T21:31:59.021 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T21:31:59.021 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:31:59.021 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T21:31:59.021 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 1, 2023-05-31T21:31:59.021 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T21:31:59.021 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "incremental-sync", 2023-05-31T21:31:59.021 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "00000000000000000000:00000000000000031920", 2023-05-31T21:31:59.022 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T21:31:59.022 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2023-05-31T21:31:59.022 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T21:31:59.022 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2023-05-31T21:29:57.583636Z" 2023-05-31T21:31:59.022 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T21:31:59.022 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:31:59.022 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T21:31:59.023 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 2, 2023-05-31T21:31:59.023 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T21:31:59.023 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "incremental-sync", 2023-05-31T21:31:59.023 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "00000000000000000000:00000000000000051951", 2023-05-31T21:31:59.023 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T21:31:59.023 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 55, 2023-05-31T21:31:59.023 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T21:31:59.024 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2023-05-31T21:29:57.526899Z" 2023-05-31T21:31:59.024 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T21:31:59.024 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:31:59.024 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T21:31:59.024 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 3, 2023-05-31T21:31:59.024 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T21:31:59.025 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "incremental-sync", 2023-05-31T21:31:59.025 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "00000000000000000000:00000000000000044178", 2023-05-31T21:31:59.025 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T21:31:59.025 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 9, 2023-05-31T21:31:59.025 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T21:31:59.025 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2023-05-31T21:27:37.215905Z" 2023-05-31T21:31:59.025 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T21:31:59.026 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T21:31:59.026 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2023-05-31T21:31:59.026 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:31:59.026 INFO:teuthology.orchestra.run.smithi099.stdout: "full_sync": { 2023-05-31T21:31:59.026 INFO:teuthology.orchestra.run.smithi099.stdout: "total": 64, 2023-05-31T21:31:59.026 INFO:teuthology.orchestra.run.smithi099.stdout: "complete": 64 2023-05-31T21:31:59.026 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:31:59.027 INFO:teuthology.orchestra.run.smithi099.stdout: "current_time": "2023-05-31T21:31:59Z" 2023-05-31T21:31:59.027 INFO:teuthology.orchestra.run.smithi099.stdout:} 2023-05-31T21:31:59.027 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 452136274422638824 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000045515", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-05-31T21:29:57.506834Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000031920", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-05-31T21:29:57.583636Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000051951", "next_step_marker": "", "total_entries": 55, "pos": 0, "timestamp": "2023-05-31T21:29:57.526899Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000044178", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2023-05-31T21:27:37.215905Z" } } ] }, "full_sync": { "total": 64, "complete": 64 }, "current_time": "2023-05-31T21:31:59Z" } 2023-05-31T21:31:59.028 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000045515', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-05-31T21:29:57.506834Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000031920', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-05-31T21:29:57.583636Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000051951', 'next_step_marker': '', 'total_entries': 55, 'pos': 0, 'timestamp': '2023-05-31T21:29:57.526899Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000044178', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2023-05-31T21:27:37.215905Z'}}] 2023-05-31T21:31:59.028 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000045655', 1: '00000000000000000000:00000000000000031920', 2: '00000000000000000000:00000000000000052091', 3: '00000000000000000000:00000000000000044178'} 2023-05-31T21:31:59.029 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000045515', 1: '00000000000000000000:00000000000000031920', 2: '00000000000000000000:00000000000000051951', 3: '00000000000000000000:00000000000000044178'} 2023-05-31T21:31:59.029 WARNING:rgw_multi.tests:data of zone a2 behind zone a1: shard=0 master=00000000000000000000:00000000000000045655 target=00000000000000000000:00000000000000045515, shard=2 master=00000000000000000000:00000000000000052091 target=00000000000000000000:00000000000000051951 2023-05-31T21:32:04.030 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-31T21:32:04.031 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-31T21:32:04.031 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-05-31T21:32:04.091 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2023-05-31T21:32:04.092 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2023-05-31T21:32:04.155 INFO:teuthology.orchestra.run.smithi099.stdout:{ 2023-05-31T21:32:04.156 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_status": { 2023-05-31T21:32:04.156 INFO:teuthology.orchestra.run.smithi099.stdout: "info": { 2023-05-31T21:32:04.156 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "sync", 2023-05-31T21:32:04.156 INFO:teuthology.orchestra.run.smithi099.stdout: "num_shards": 4, 2023-05-31T21:32:04.157 INFO:teuthology.orchestra.run.smithi099.stdout: "instance_id": 452136274422638824 2023-05-31T21:32:04.157 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:32:04.157 INFO:teuthology.orchestra.run.smithi099.stdout: "markers": [ 2023-05-31T21:32:04.157 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T21:32:04.157 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 0, 2023-05-31T21:32:04.157 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T21:32:04.157 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "incremental-sync", 2023-05-31T21:32:04.158 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "00000000000000000000:00000000000000045515", 2023-05-31T21:32:04.158 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T21:32:04.158 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2023-05-31T21:32:04.158 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T21:32:04.158 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2023-05-31T21:29:57.506834Z" 2023-05-31T21:32:04.158 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T21:32:04.158 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:32:04.159 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T21:32:04.159 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 1, 2023-05-31T21:32:04.159 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T21:32:04.159 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "incremental-sync", 2023-05-31T21:32:04.159 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "00000000000000000000:00000000000000031920", 2023-05-31T21:32:04.159 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T21:32:04.159 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2023-05-31T21:32:04.160 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T21:32:04.160 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2023-05-31T21:29:57.583636Z" 2023-05-31T21:32:04.160 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T21:32:04.160 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:32:04.160 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T21:32:04.160 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 2, 2023-05-31T21:32:04.161 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T21:32:04.161 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "incremental-sync", 2023-05-31T21:32:04.161 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "00000000000000000000:00000000000000051951", 2023-05-31T21:32:04.161 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T21:32:04.161 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 55, 2023-05-31T21:32:04.161 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T21:32:04.161 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2023-05-31T21:29:57.526899Z" 2023-05-31T21:32:04.162 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T21:32:04.162 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:32:04.162 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T21:32:04.162 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 3, 2023-05-31T21:32:04.162 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T21:32:04.162 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "incremental-sync", 2023-05-31T21:32:04.162 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "00000000000000000000:00000000000000044178", 2023-05-31T21:32:04.163 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T21:32:04.163 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 9, 2023-05-31T21:32:04.163 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T21:32:04.163 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2023-05-31T21:27:37.215905Z" 2023-05-31T21:32:04.163 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T21:32:04.163 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T21:32:04.163 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2023-05-31T21:32:04.164 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:32:04.164 INFO:teuthology.orchestra.run.smithi099.stdout: "full_sync": { 2023-05-31T21:32:04.164 INFO:teuthology.orchestra.run.smithi099.stdout: "total": 64, 2023-05-31T21:32:04.164 INFO:teuthology.orchestra.run.smithi099.stdout: "complete": 64 2023-05-31T21:32:04.164 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:32:04.164 INFO:teuthology.orchestra.run.smithi099.stdout: "current_time": "2023-05-31T21:32:04Z" 2023-05-31T21:32:04.164 INFO:teuthology.orchestra.run.smithi099.stdout:} 2023-05-31T21:32:04.165 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 452136274422638824 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000045515", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-05-31T21:29:57.506834Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000031920", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-05-31T21:29:57.583636Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000051951", "next_step_marker": "", "total_entries": 55, "pos": 0, "timestamp": "2023-05-31T21:29:57.526899Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000044178", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2023-05-31T21:27:37.215905Z" } } ] }, "full_sync": { "total": 64, "complete": 64 }, "current_time": "2023-05-31T21:32:04Z" } 2023-05-31T21:32:04.166 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000045515', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-05-31T21:29:57.506834Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000031920', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-05-31T21:29:57.583636Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000051951', 'next_step_marker': '', 'total_entries': 55, 'pos': 0, 'timestamp': '2023-05-31T21:29:57.526899Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000044178', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2023-05-31T21:27:37.215905Z'}}] 2023-05-31T21:32:04.166 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000045655', 1: '00000000000000000000:00000000000000031920', 2: '00000000000000000000:00000000000000052091', 3: '00000000000000000000:00000000000000044178'} 2023-05-31T21:32:04.167 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000045515', 1: '00000000000000000000:00000000000000031920', 2: '00000000000000000000:00000000000000051951', 3: '00000000000000000000:00000000000000044178'} 2023-05-31T21:32:04.167 WARNING:rgw_multi.tests:data of zone a2 behind zone a1: shard=0 master=00000000000000000000:00000000000000045655 target=00000000000000000000:00000000000000045515, shard=2 master=00000000000000000000:00000000000000052091 target=00000000000000000000:00000000000000051951 2023-05-31T21:32:09.168 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-31T21:32:09.168 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-31T21:32:09.169 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-05-31T21:32:09.228 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2023-05-31T21:32:09.228 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2023-05-31T21:32:09.291 INFO:teuthology.orchestra.run.smithi099.stdout:{ 2023-05-31T21:32:09.292 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_status": { 2023-05-31T21:32:09.292 INFO:teuthology.orchestra.run.smithi099.stdout: "info": { 2023-05-31T21:32:09.292 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "sync", 2023-05-31T21:32:09.292 INFO:teuthology.orchestra.run.smithi099.stdout: "num_shards": 4, 2023-05-31T21:32:09.292 INFO:teuthology.orchestra.run.smithi099.stdout: "instance_id": 452136274422638824 2023-05-31T21:32:09.293 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:32:09.293 INFO:teuthology.orchestra.run.smithi099.stdout: "markers": [ 2023-05-31T21:32:09.293 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T21:32:09.293 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 0, 2023-05-31T21:32:09.293 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T21:32:09.293 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "incremental-sync", 2023-05-31T21:32:09.294 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "00000000000000000000:00000000000000045515", 2023-05-31T21:32:09.294 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T21:32:09.294 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2023-05-31T21:32:09.294 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T21:32:09.294 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2023-05-31T21:29:57.506834Z" 2023-05-31T21:32:09.294 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T21:32:09.294 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:32:09.295 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T21:32:09.295 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 1, 2023-05-31T21:32:09.295 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T21:32:09.295 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "incremental-sync", 2023-05-31T21:32:09.295 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "00000000000000000000:00000000000000031920", 2023-05-31T21:32:09.295 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T21:32:09.295 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2023-05-31T21:32:09.296 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T21:32:09.296 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2023-05-31T21:29:57.583636Z" 2023-05-31T21:32:09.296 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T21:32:09.296 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:32:09.296 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T21:32:09.296 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 2, 2023-05-31T21:32:09.296 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T21:32:09.297 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "incremental-sync", 2023-05-31T21:32:09.297 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "00000000000000000000:00000000000000051951", 2023-05-31T21:32:09.297 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T21:32:09.297 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 55, 2023-05-31T21:32:09.297 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T21:32:09.297 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2023-05-31T21:29:57.526899Z" 2023-05-31T21:32:09.297 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T21:32:09.298 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:32:09.298 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T21:32:09.298 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 3, 2023-05-31T21:32:09.298 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T21:32:09.298 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "incremental-sync", 2023-05-31T21:32:09.298 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "00000000000000000000:00000000000000044178", 2023-05-31T21:32:09.299 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T21:32:09.299 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 9, 2023-05-31T21:32:09.299 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T21:32:09.299 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2023-05-31T21:27:37.215905Z" 2023-05-31T21:32:09.299 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T21:32:09.299 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T21:32:09.299 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2023-05-31T21:32:09.300 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:32:09.300 INFO:teuthology.orchestra.run.smithi099.stdout: "full_sync": { 2023-05-31T21:32:09.300 INFO:teuthology.orchestra.run.smithi099.stdout: "total": 64, 2023-05-31T21:32:09.300 INFO:teuthology.orchestra.run.smithi099.stdout: "complete": 64 2023-05-31T21:32:09.300 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:32:09.300 INFO:teuthology.orchestra.run.smithi099.stdout: "current_time": "2023-05-31T21:32:09Z" 2023-05-31T21:32:09.300 INFO:teuthology.orchestra.run.smithi099.stdout:} 2023-05-31T21:32:09.301 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 452136274422638824 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000045515", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-05-31T21:29:57.506834Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000031920", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-05-31T21:29:57.583636Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000051951", "next_step_marker": "", "total_entries": 55, "pos": 0, "timestamp": "2023-05-31T21:29:57.526899Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000044178", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2023-05-31T21:27:37.215905Z" } } ] }, "full_sync": { "total": 64, "complete": 64 }, "current_time": "2023-05-31T21:32:09Z" } 2023-05-31T21:32:09.302 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000045515', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-05-31T21:29:57.506834Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000031920', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-05-31T21:29:57.583636Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000051951', 'next_step_marker': '', 'total_entries': 55, 'pos': 0, 'timestamp': '2023-05-31T21:29:57.526899Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000044178', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2023-05-31T21:27:37.215905Z'}}] 2023-05-31T21:32:09.302 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000045655', 1: '00000000000000000000:00000000000000031920', 2: '00000000000000000000:00000000000000052091', 3: '00000000000000000000:00000000000000044178'} 2023-05-31T21:32:09.303 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000045515', 1: '00000000000000000000:00000000000000031920', 2: '00000000000000000000:00000000000000051951', 3: '00000000000000000000:00000000000000044178'} 2023-05-31T21:32:09.303 WARNING:rgw_multi.tests:data of zone a2 behind zone a1: shard=0 master=00000000000000000000:00000000000000045655 target=00000000000000000000:00000000000000045515, shard=2 master=00000000000000000000:00000000000000052091 target=00000000000000000000:00000000000000051951 2023-05-31T21:32:14.304 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-31T21:32:14.304 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-31T21:32:14.305 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-05-31T21:32:14.363 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2023-05-31T21:32:14.364 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2023-05-31T21:32:14.423 INFO:teuthology.orchestra.run.smithi099.stdout:{ 2023-05-31T21:32:14.424 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_status": { 2023-05-31T21:32:14.424 INFO:teuthology.orchestra.run.smithi099.stdout: "info": { 2023-05-31T21:32:14.424 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "sync", 2023-05-31T21:32:14.424 INFO:teuthology.orchestra.run.smithi099.stdout: "num_shards": 4, 2023-05-31T21:32:14.425 INFO:teuthology.orchestra.run.smithi099.stdout: "instance_id": 452136274422638824 2023-05-31T21:32:14.425 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:32:14.425 INFO:teuthology.orchestra.run.smithi099.stdout: "markers": [ 2023-05-31T21:32:14.425 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T21:32:14.425 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 0, 2023-05-31T21:32:14.425 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T21:32:14.425 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "incremental-sync", 2023-05-31T21:32:14.426 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "00000000000000000000:00000000000000045515", 2023-05-31T21:32:14.426 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T21:32:14.426 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2023-05-31T21:32:14.426 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T21:32:14.426 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2023-05-31T21:29:57.506834Z" 2023-05-31T21:32:14.426 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T21:32:14.426 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:32:14.427 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T21:32:14.427 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 1, 2023-05-31T21:32:14.427 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T21:32:14.427 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "incremental-sync", 2023-05-31T21:32:14.427 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "00000000000000000000:00000000000000031920", 2023-05-31T21:32:14.427 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T21:32:14.427 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2023-05-31T21:32:14.428 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T21:32:14.428 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2023-05-31T21:29:57.583636Z" 2023-05-31T21:32:14.428 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T21:32:14.428 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:32:14.428 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T21:32:14.428 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 2, 2023-05-31T21:32:14.429 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T21:32:14.429 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "incremental-sync", 2023-05-31T21:32:14.429 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "00000000000000000000:00000000000000051951", 2023-05-31T21:32:14.429 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T21:32:14.429 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 55, 2023-05-31T21:32:14.429 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T21:32:14.429 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2023-05-31T21:29:57.526899Z" 2023-05-31T21:32:14.430 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T21:32:14.430 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:32:14.430 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T21:32:14.430 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 3, 2023-05-31T21:32:14.430 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T21:32:14.430 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "incremental-sync", 2023-05-31T21:32:14.430 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "00000000000000000000:00000000000000044178", 2023-05-31T21:32:14.431 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T21:32:14.431 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 9, 2023-05-31T21:32:14.431 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T21:32:14.431 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2023-05-31T21:27:37.215905Z" 2023-05-31T21:32:14.431 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T21:32:14.431 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T21:32:14.431 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2023-05-31T21:32:14.432 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:32:14.432 INFO:teuthology.orchestra.run.smithi099.stdout: "full_sync": { 2023-05-31T21:32:14.432 INFO:teuthology.orchestra.run.smithi099.stdout: "total": 64, 2023-05-31T21:32:14.432 INFO:teuthology.orchestra.run.smithi099.stdout: "complete": 64 2023-05-31T21:32:14.432 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:32:14.432 INFO:teuthology.orchestra.run.smithi099.stdout: "current_time": "2023-05-31T21:32:14Z" 2023-05-31T21:32:14.433 INFO:teuthology.orchestra.run.smithi099.stdout:} 2023-05-31T21:32:14.433 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 452136274422638824 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000045515", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-05-31T21:29:57.506834Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000031920", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-05-31T21:29:57.583636Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000051951", "next_step_marker": "", "total_entries": 55, "pos": 0, "timestamp": "2023-05-31T21:29:57.526899Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000044178", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2023-05-31T21:27:37.215905Z" } } ] }, "full_sync": { "total": 64, "complete": 64 }, "current_time": "2023-05-31T21:32:14Z" } 2023-05-31T21:32:14.434 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000045515', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-05-31T21:29:57.506834Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000031920', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-05-31T21:29:57.583636Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000051951', 'next_step_marker': '', 'total_entries': 55, 'pos': 0, 'timestamp': '2023-05-31T21:29:57.526899Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000044178', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2023-05-31T21:27:37.215905Z'}}] 2023-05-31T21:32:14.434 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000045655', 1: '00000000000000000000:00000000000000031920', 2: '00000000000000000000:00000000000000052091', 3: '00000000000000000000:00000000000000044178'} 2023-05-31T21:32:14.435 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000045515', 1: '00000000000000000000:00000000000000031920', 2: '00000000000000000000:00000000000000051951', 3: '00000000000000000000:00000000000000044178'} 2023-05-31T21:32:14.435 WARNING:rgw_multi.tests:data of zone a2 behind zone a1: shard=0 master=00000000000000000000:00000000000000045655 target=00000000000000000000:00000000000000045515, shard=2 master=00000000000000000000:00000000000000052091 target=00000000000000000000:00000000000000051951 2023-05-31T21:32:19.436 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-31T21:32:19.437 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-31T21:32:19.437 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-05-31T21:32:19.495 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2023-05-31T21:32:19.496 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2023-05-31T21:32:19.571 INFO:teuthology.orchestra.run.smithi099.stdout:{ 2023-05-31T21:32:19.571 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_status": { 2023-05-31T21:32:19.572 INFO:teuthology.orchestra.run.smithi099.stdout: "info": { 2023-05-31T21:32:19.572 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "sync", 2023-05-31T21:32:19.572 INFO:teuthology.orchestra.run.smithi099.stdout: "num_shards": 4, 2023-05-31T21:32:19.572 INFO:teuthology.orchestra.run.smithi099.stdout: "instance_id": 452136274422638824 2023-05-31T21:32:19.572 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:32:19.572 INFO:teuthology.orchestra.run.smithi099.stdout: "markers": [ 2023-05-31T21:32:19.573 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T21:32:19.573 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 0, 2023-05-31T21:32:19.573 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T21:32:19.573 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "incremental-sync", 2023-05-31T21:32:19.573 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "00000000000000000000:00000000000000045515", 2023-05-31T21:32:19.573 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T21:32:19.573 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2023-05-31T21:32:19.574 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T21:32:19.574 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2023-05-31T21:29:57.506834Z" 2023-05-31T21:32:19.574 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T21:32:19.574 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:32:19.574 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T21:32:19.574 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 1, 2023-05-31T21:32:19.574 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T21:32:19.575 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "incremental-sync", 2023-05-31T21:32:19.575 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "00000000000000000000:00000000000000031920", 2023-05-31T21:32:19.575 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T21:32:19.575 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2023-05-31T21:32:19.575 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T21:32:19.575 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2023-05-31T21:29:57.583636Z" 2023-05-31T21:32:19.575 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T21:32:19.576 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:32:19.576 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T21:32:19.576 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 2, 2023-05-31T21:32:19.576 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T21:32:19.576 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "incremental-sync", 2023-05-31T21:32:19.576 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "00000000000000000000:00000000000000051951", 2023-05-31T21:32:19.577 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T21:32:19.577 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 55, 2023-05-31T21:32:19.577 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T21:32:19.577 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2023-05-31T21:29:57.526899Z" 2023-05-31T21:32:19.577 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T21:32:19.577 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:32:19.577 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T21:32:19.578 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 3, 2023-05-31T21:32:19.578 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T21:32:19.578 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "incremental-sync", 2023-05-31T21:32:19.578 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "00000000000000000000:00000000000000044178", 2023-05-31T21:32:19.578 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T21:32:19.578 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 9, 2023-05-31T21:32:19.578 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T21:32:19.579 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2023-05-31T21:27:37.215905Z" 2023-05-31T21:32:19.579 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T21:32:19.579 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T21:32:19.579 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2023-05-31T21:32:19.579 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:32:19.579 INFO:teuthology.orchestra.run.smithi099.stdout: "full_sync": { 2023-05-31T21:32:19.579 INFO:teuthology.orchestra.run.smithi099.stdout: "total": 64, 2023-05-31T21:32:19.580 INFO:teuthology.orchestra.run.smithi099.stdout: "complete": 64 2023-05-31T21:32:19.580 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:32:19.580 INFO:teuthology.orchestra.run.smithi099.stdout: "current_time": "2023-05-31T21:32:19Z" 2023-05-31T21:32:19.580 INFO:teuthology.orchestra.run.smithi099.stdout:} 2023-05-31T21:32:19.581 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 452136274422638824 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000045515", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-05-31T21:29:57.506834Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000031920", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-05-31T21:29:57.583636Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000051951", "next_step_marker": "", "total_entries": 55, "pos": 0, "timestamp": "2023-05-31T21:29:57.526899Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000044178", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2023-05-31T21:27:37.215905Z" } } ] }, "full_sync": { "total": 64, "complete": 64 }, "current_time": "2023-05-31T21:32:19Z" } 2023-05-31T21:32:19.581 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000045515', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-05-31T21:29:57.506834Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000031920', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-05-31T21:29:57.583636Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000051951', 'next_step_marker': '', 'total_entries': 55, 'pos': 0, 'timestamp': '2023-05-31T21:29:57.526899Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000044178', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2023-05-31T21:27:37.215905Z'}}] 2023-05-31T21:32:19.582 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000045655', 1: '00000000000000000000:00000000000000031920', 2: '00000000000000000000:00000000000000052091', 3: '00000000000000000000:00000000000000044178'} 2023-05-31T21:32:19.582 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000045515', 1: '00000000000000000000:00000000000000031920', 2: '00000000000000000000:00000000000000051951', 3: '00000000000000000000:00000000000000044178'} 2023-05-31T21:32:19.583 WARNING:rgw_multi.tests:data of zone a2 behind zone a1: shard=0 master=00000000000000000000:00000000000000045655 target=00000000000000000000:00000000000000045515, shard=2 master=00000000000000000000:00000000000000052091 target=00000000000000000000:00000000000000051951 2023-05-31T21:32:24.584 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-31T21:32:24.584 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-31T21:32:24.585 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-05-31T21:32:24.644 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2023-05-31T21:32:24.644 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2023-05-31T21:32:24.695 INFO:teuthology.orchestra.run.smithi099.stdout:{ 2023-05-31T21:32:24.695 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_status": { 2023-05-31T21:32:24.695 INFO:teuthology.orchestra.run.smithi099.stdout: "info": { 2023-05-31T21:32:24.696 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "sync", 2023-05-31T21:32:24.696 INFO:teuthology.orchestra.run.smithi099.stdout: "num_shards": 4, 2023-05-31T21:32:24.696 INFO:teuthology.orchestra.run.smithi099.stdout: "instance_id": 452136274422638824 2023-05-31T21:32:24.696 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:32:24.696 INFO:teuthology.orchestra.run.smithi099.stdout: "markers": [ 2023-05-31T21:32:24.696 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T21:32:24.697 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 0, 2023-05-31T21:32:24.697 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T21:32:24.697 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "incremental-sync", 2023-05-31T21:32:24.697 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "00000000000000000000:00000000000000045515", 2023-05-31T21:32:24.697 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T21:32:24.697 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2023-05-31T21:32:24.697 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T21:32:24.698 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2023-05-31T21:29:57.506834Z" 2023-05-31T21:32:24.698 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T21:32:24.698 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:32:24.698 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T21:32:24.698 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 1, 2023-05-31T21:32:24.698 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T21:32:24.698 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "incremental-sync", 2023-05-31T21:32:24.699 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "00000000000000000000:00000000000000031920", 2023-05-31T21:32:24.699 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T21:32:24.699 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2023-05-31T21:32:24.699 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T21:32:24.699 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2023-05-31T21:29:57.583636Z" 2023-05-31T21:32:24.699 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T21:32:24.699 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:32:24.700 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T21:32:24.700 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 2, 2023-05-31T21:32:24.700 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T21:32:24.700 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "incremental-sync", 2023-05-31T21:32:24.700 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "00000000000000000000:00000000000000051951", 2023-05-31T21:32:24.700 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T21:32:24.700 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 55, 2023-05-31T21:32:24.701 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T21:32:24.701 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2023-05-31T21:29:57.526899Z" 2023-05-31T21:32:24.701 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T21:32:24.701 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:32:24.701 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T21:32:24.701 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 3, 2023-05-31T21:32:24.701 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T21:32:24.702 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "incremental-sync", 2023-05-31T21:32:24.702 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "00000000000000000000:00000000000000044178", 2023-05-31T21:32:24.702 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T21:32:24.702 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 9, 2023-05-31T21:32:24.702 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T21:32:24.702 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2023-05-31T21:27:37.215905Z" 2023-05-31T21:32:24.702 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T21:32:24.703 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T21:32:24.703 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2023-05-31T21:32:24.703 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:32:24.703 INFO:teuthology.orchestra.run.smithi099.stdout: "full_sync": { 2023-05-31T21:32:24.703 INFO:teuthology.orchestra.run.smithi099.stdout: "total": 64, 2023-05-31T21:32:24.703 INFO:teuthology.orchestra.run.smithi099.stdout: "complete": 64 2023-05-31T21:32:24.703 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:32:24.704 INFO:teuthology.orchestra.run.smithi099.stdout: "current_time": "2023-05-31T21:32:24Z" 2023-05-31T21:32:24.704 INFO:teuthology.orchestra.run.smithi099.stdout:} 2023-05-31T21:32:24.705 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 452136274422638824 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000045515", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-05-31T21:29:57.506834Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000031920", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-05-31T21:29:57.583636Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000051951", "next_step_marker": "", "total_entries": 55, "pos": 0, "timestamp": "2023-05-31T21:29:57.526899Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000044178", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2023-05-31T21:27:37.215905Z" } } ] }, "full_sync": { "total": 64, "complete": 64 }, "current_time": "2023-05-31T21:32:24Z" } 2023-05-31T21:32:24.705 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000045515', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-05-31T21:29:57.506834Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000031920', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-05-31T21:29:57.583636Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000051951', 'next_step_marker': '', 'total_entries': 55, 'pos': 0, 'timestamp': '2023-05-31T21:29:57.526899Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000044178', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2023-05-31T21:27:37.215905Z'}}] 2023-05-31T21:32:24.706 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000045655', 1: '00000000000000000000:00000000000000031920', 2: '00000000000000000000:00000000000000052091', 3: '00000000000000000000:00000000000000044178'} 2023-05-31T21:32:24.706 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000045515', 1: '00000000000000000000:00000000000000031920', 2: '00000000000000000000:00000000000000051951', 3: '00000000000000000000:00000000000000044178'} 2023-05-31T21:32:24.706 WARNING:rgw_multi.tests:data of zone a2 behind zone a1: shard=0 master=00000000000000000000:00000000000000045655 target=00000000000000000000:00000000000000045515, shard=2 master=00000000000000000000:00000000000000052091 target=00000000000000000000:00000000000000051951 2023-05-31T21:32:29.707 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-31T21:32:29.708 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-31T21:32:29.708 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-05-31T21:32:29.768 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2023-05-31T21:32:29.769 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2023-05-31T21:32:29.851 INFO:teuthology.orchestra.run.smithi099.stdout:{ 2023-05-31T21:32:29.851 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_status": { 2023-05-31T21:32:29.851 INFO:teuthology.orchestra.run.smithi099.stdout: "info": { 2023-05-31T21:32:29.852 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "sync", 2023-05-31T21:32:29.852 INFO:teuthology.orchestra.run.smithi099.stdout: "num_shards": 4, 2023-05-31T21:32:29.852 INFO:teuthology.orchestra.run.smithi099.stdout: "instance_id": 452136274422638824 2023-05-31T21:32:29.852 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:32:29.852 INFO:teuthology.orchestra.run.smithi099.stdout: "markers": [ 2023-05-31T21:32:29.852 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T21:32:29.853 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 0, 2023-05-31T21:32:29.853 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T21:32:29.853 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "incremental-sync", 2023-05-31T21:32:29.853 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "00000000000000000000:00000000000000045515", 2023-05-31T21:32:29.854 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T21:32:29.854 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2023-05-31T21:32:29.854 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T21:32:29.854 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2023-05-31T21:29:57.506834Z" 2023-05-31T21:32:29.854 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T21:32:29.855 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:32:29.855 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T21:32:29.855 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 1, 2023-05-31T21:32:29.855 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T21:32:29.855 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "incremental-sync", 2023-05-31T21:32:29.855 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "00000000000000000000:00000000000000031920", 2023-05-31T21:32:29.856 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T21:32:29.856 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2023-05-31T21:32:29.856 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T21:32:29.856 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2023-05-31T21:29:57.583636Z" 2023-05-31T21:32:29.856 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T21:32:29.856 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:32:29.857 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T21:32:29.857 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 2, 2023-05-31T21:32:29.857 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T21:32:29.857 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "incremental-sync", 2023-05-31T21:32:29.857 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "00000000000000000000:00000000000000051951", 2023-05-31T21:32:29.857 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T21:32:29.857 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 55, 2023-05-31T21:32:29.858 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T21:32:29.858 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2023-05-31T21:29:57.526899Z" 2023-05-31T21:32:29.858 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T21:32:29.858 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:32:29.858 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T21:32:29.858 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 3, 2023-05-31T21:32:29.858 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T21:32:29.859 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "incremental-sync", 2023-05-31T21:32:29.859 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "00000000000000000000:00000000000000044178", 2023-05-31T21:32:29.859 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T21:32:29.859 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 9, 2023-05-31T21:32:29.859 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T21:32:29.859 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2023-05-31T21:27:37.215905Z" 2023-05-31T21:32:29.859 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T21:32:29.860 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T21:32:29.860 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2023-05-31T21:32:29.860 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:32:29.860 INFO:teuthology.orchestra.run.smithi099.stdout: "full_sync": { 2023-05-31T21:32:29.860 INFO:teuthology.orchestra.run.smithi099.stdout: "total": 64, 2023-05-31T21:32:29.860 INFO:teuthology.orchestra.run.smithi099.stdout: "complete": 64 2023-05-31T21:32:29.861 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:32:29.861 INFO:teuthology.orchestra.run.smithi099.stdout: "current_time": "2023-05-31T21:32:29Z" 2023-05-31T21:32:29.861 INFO:teuthology.orchestra.run.smithi099.stdout:} 2023-05-31T21:32:29.862 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 452136274422638824 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000045515", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-05-31T21:29:57.506834Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000031920", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-05-31T21:29:57.583636Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000051951", "next_step_marker": "", "total_entries": 55, "pos": 0, "timestamp": "2023-05-31T21:29:57.526899Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000044178", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2023-05-31T21:27:37.215905Z" } } ] }, "full_sync": { "total": 64, "complete": 64 }, "current_time": "2023-05-31T21:32:29Z" } 2023-05-31T21:32:29.862 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000045515', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-05-31T21:29:57.506834Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000031920', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-05-31T21:29:57.583636Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000051951', 'next_step_marker': '', 'total_entries': 55, 'pos': 0, 'timestamp': '2023-05-31T21:29:57.526899Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000044178', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2023-05-31T21:27:37.215905Z'}}] 2023-05-31T21:32:29.863 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000045655', 1: '00000000000000000000:00000000000000031920', 2: '00000000000000000000:00000000000000052091', 3: '00000000000000000000:00000000000000044178'} 2023-05-31T21:32:29.863 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000045515', 1: '00000000000000000000:00000000000000031920', 2: '00000000000000000000:00000000000000051951', 3: '00000000000000000000:00000000000000044178'} 2023-05-31T21:32:29.863 WARNING:rgw_multi.tests:data of zone a2 behind zone a1: shard=0 master=00000000000000000000:00000000000000045655 target=00000000000000000000:00000000000000045515, shard=2 master=00000000000000000000:00000000000000052091 target=00000000000000000000:00000000000000051951 2023-05-31T21:32:34.864 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-31T21:32:34.865 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-31T21:32:34.865 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-05-31T21:32:34.923 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2023-05-31T21:32:34.923 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2023-05-31T21:32:34.980 INFO:teuthology.orchestra.run.smithi099.stdout:{ 2023-05-31T21:32:34.980 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_status": { 2023-05-31T21:32:34.981 INFO:teuthology.orchestra.run.smithi099.stdout: "info": { 2023-05-31T21:32:34.981 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "sync", 2023-05-31T21:32:34.981 INFO:teuthology.orchestra.run.smithi099.stdout: "num_shards": 4, 2023-05-31T21:32:34.981 INFO:teuthology.orchestra.run.smithi099.stdout: "instance_id": 452136274422638824 2023-05-31T21:32:34.981 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:32:34.981 INFO:teuthology.orchestra.run.smithi099.stdout: "markers": [ 2023-05-31T21:32:34.982 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T21:32:34.982 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 0, 2023-05-31T21:32:34.982 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T21:32:34.982 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "incremental-sync", 2023-05-31T21:32:34.982 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "00000000000000000000:00000000000000045515", 2023-05-31T21:32:34.982 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T21:32:34.983 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2023-05-31T21:32:34.983 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T21:32:34.983 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2023-05-31T21:29:57.506834Z" 2023-05-31T21:32:34.983 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T21:32:34.983 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:32:34.983 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T21:32:34.983 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 1, 2023-05-31T21:32:34.984 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T21:32:34.984 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "incremental-sync", 2023-05-31T21:32:34.984 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "00000000000000000000:00000000000000031920", 2023-05-31T21:32:34.984 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T21:32:34.985 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2023-05-31T21:32:34.985 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T21:32:34.985 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2023-05-31T21:29:57.583636Z" 2023-05-31T21:32:34.985 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T21:32:34.985 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:32:34.986 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T21:32:34.986 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 2, 2023-05-31T21:32:34.986 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T21:32:34.987 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "incremental-sync", 2023-05-31T21:32:34.987 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "00000000000000000000:00000000000000051951", 2023-05-31T21:32:34.987 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T21:32:34.987 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 55, 2023-05-31T21:32:34.988 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T21:32:34.988 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2023-05-31T21:29:57.526899Z" 2023-05-31T21:32:34.988 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T21:32:34.988 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:32:34.988 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T21:32:34.989 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 3, 2023-05-31T21:32:34.989 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T21:32:34.989 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "incremental-sync", 2023-05-31T21:32:34.989 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "00000000000000000000:00000000000000044178", 2023-05-31T21:32:34.989 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T21:32:34.989 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 9, 2023-05-31T21:32:34.989 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T21:32:34.990 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2023-05-31T21:27:37.215905Z" 2023-05-31T21:32:34.990 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T21:32:34.990 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T21:32:34.990 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2023-05-31T21:32:34.990 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:32:34.990 INFO:teuthology.orchestra.run.smithi099.stdout: "full_sync": { 2023-05-31T21:32:34.991 INFO:teuthology.orchestra.run.smithi099.stdout: "total": 64, 2023-05-31T21:32:34.991 INFO:teuthology.orchestra.run.smithi099.stdout: "complete": 64 2023-05-31T21:32:34.991 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:32:34.991 INFO:teuthology.orchestra.run.smithi099.stdout: "current_time": "2023-05-31T21:32:34Z" 2023-05-31T21:32:34.991 INFO:teuthology.orchestra.run.smithi099.stdout:} 2023-05-31T21:32:34.992 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 452136274422638824 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000045515", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-05-31T21:29:57.506834Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000031920", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-05-31T21:29:57.583636Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000051951", "next_step_marker": "", "total_entries": 55, "pos": 0, "timestamp": "2023-05-31T21:29:57.526899Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000044178", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2023-05-31T21:27:37.215905Z" } } ] }, "full_sync": { "total": 64, "complete": 64 }, "current_time": "2023-05-31T21:32:34Z" } 2023-05-31T21:32:34.993 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000045515', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-05-31T21:29:57.506834Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000031920', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-05-31T21:29:57.583636Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000051951', 'next_step_marker': '', 'total_entries': 55, 'pos': 0, 'timestamp': '2023-05-31T21:29:57.526899Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000044178', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2023-05-31T21:27:37.215905Z'}}] 2023-05-31T21:32:34.993 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000045655', 1: '00000000000000000000:00000000000000031920', 2: '00000000000000000000:00000000000000052091', 3: '00000000000000000000:00000000000000044178'} 2023-05-31T21:32:34.994 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000045515', 1: '00000000000000000000:00000000000000031920', 2: '00000000000000000000:00000000000000051951', 3: '00000000000000000000:00000000000000044178'} 2023-05-31T21:32:34.994 WARNING:rgw_multi.tests:data of zone a2 behind zone a1: shard=0 master=00000000000000000000:00000000000000045655 target=00000000000000000000:00000000000000045515, shard=2 master=00000000000000000000:00000000000000052091 target=00000000000000000000:00000000000000051951 2023-05-31T21:32:39.995 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-31T21:32:39.996 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-31T21:32:39.996 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-05-31T21:32:40.058 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2023-05-31T21:32:40.058 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2023-05-31T21:32:40.111 INFO:teuthology.orchestra.run.smithi099.stdout:{ 2023-05-31T21:32:40.111 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_status": { 2023-05-31T21:32:40.111 INFO:teuthology.orchestra.run.smithi099.stdout: "info": { 2023-05-31T21:32:40.111 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "sync", 2023-05-31T21:32:40.111 INFO:teuthology.orchestra.run.smithi099.stdout: "num_shards": 4, 2023-05-31T21:32:40.111 INFO:teuthology.orchestra.run.smithi099.stdout: "instance_id": 452136274422638824 2023-05-31T21:32:40.112 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:32:40.112 INFO:teuthology.orchestra.run.smithi099.stdout: "markers": [ 2023-05-31T21:32:40.112 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T21:32:40.112 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 0, 2023-05-31T21:32:40.112 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T21:32:40.112 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "incremental-sync", 2023-05-31T21:32:40.113 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "00000000000000000000:00000000000000045515", 2023-05-31T21:32:40.113 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T21:32:40.113 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2023-05-31T21:32:40.113 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T21:32:40.113 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2023-05-31T21:29:57.506834Z" 2023-05-31T21:32:40.113 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T21:32:40.113 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:32:40.114 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T21:32:40.114 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 1, 2023-05-31T21:32:40.114 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T21:32:40.114 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "incremental-sync", 2023-05-31T21:32:40.114 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "00000000000000000000:00000000000000031920", 2023-05-31T21:32:40.114 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T21:32:40.114 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2023-05-31T21:32:40.115 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T21:32:40.115 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2023-05-31T21:29:57.583636Z" 2023-05-31T21:32:40.115 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T21:32:40.115 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:32:40.115 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T21:32:40.115 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 2, 2023-05-31T21:32:40.115 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T21:32:40.116 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "incremental-sync", 2023-05-31T21:32:40.116 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "00000000000000000000:00000000000000051951", 2023-05-31T21:32:40.116 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T21:32:40.116 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 55, 2023-05-31T21:32:40.116 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T21:32:40.116 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2023-05-31T21:29:57.526899Z" 2023-05-31T21:32:40.116 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T21:32:40.117 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:32:40.117 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T21:32:40.117 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 3, 2023-05-31T21:32:40.117 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T21:32:40.117 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "incremental-sync", 2023-05-31T21:32:40.117 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "00000000000000000000:00000000000000044178", 2023-05-31T21:32:40.118 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T21:32:40.118 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 9, 2023-05-31T21:32:40.118 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T21:32:40.118 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2023-05-31T21:27:37.215905Z" 2023-05-31T21:32:40.118 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T21:32:40.118 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T21:32:40.118 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2023-05-31T21:32:40.119 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:32:40.119 INFO:teuthology.orchestra.run.smithi099.stdout: "full_sync": { 2023-05-31T21:32:40.119 INFO:teuthology.orchestra.run.smithi099.stdout: "total": 64, 2023-05-31T21:32:40.119 INFO:teuthology.orchestra.run.smithi099.stdout: "complete": 64 2023-05-31T21:32:40.119 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:32:40.119 INFO:teuthology.orchestra.run.smithi099.stdout: "current_time": "2023-05-31T21:32:40Z" 2023-05-31T21:32:40.120 INFO:teuthology.orchestra.run.smithi099.stdout:} 2023-05-31T21:32:40.121 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 452136274422638824 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000045515", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-05-31T21:29:57.506834Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000031920", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-05-31T21:29:57.583636Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000051951", "next_step_marker": "", "total_entries": 55, "pos": 0, "timestamp": "2023-05-31T21:29:57.526899Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000044178", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2023-05-31T21:27:37.215905Z" } } ] }, "full_sync": { "total": 64, "complete": 64 }, "current_time": "2023-05-31T21:32:40Z" } 2023-05-31T21:32:40.121 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000045515', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-05-31T21:29:57.506834Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000031920', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-05-31T21:29:57.583636Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000051951', 'next_step_marker': '', 'total_entries': 55, 'pos': 0, 'timestamp': '2023-05-31T21:29:57.526899Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000044178', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2023-05-31T21:27:37.215905Z'}}] 2023-05-31T21:32:40.122 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000045655', 1: '00000000000000000000:00000000000000031920', 2: '00000000000000000000:00000000000000052091', 3: '00000000000000000000:00000000000000044178'} 2023-05-31T21:32:40.122 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000045515', 1: '00000000000000000000:00000000000000031920', 2: '00000000000000000000:00000000000000051951', 3: '00000000000000000000:00000000000000044178'} 2023-05-31T21:32:40.122 WARNING:rgw_multi.tests:data of zone a2 behind zone a1: shard=0 master=00000000000000000000:00000000000000045655 target=00000000000000000000:00000000000000045515, shard=2 master=00000000000000000000:00000000000000052091 target=00000000000000000000:00000000000000051951 2023-05-31T21:32:45.123 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-31T21:32:45.124 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-31T21:32:45.124 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-05-31T21:32:45.183 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2023-05-31T21:32:45.183 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2023-05-31T21:32:45.264 INFO:teuthology.orchestra.run.smithi099.stdout:{ 2023-05-31T21:32:45.264 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_status": { 2023-05-31T21:32:45.264 INFO:teuthology.orchestra.run.smithi099.stdout: "info": { 2023-05-31T21:32:45.265 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "sync", 2023-05-31T21:32:45.265 INFO:teuthology.orchestra.run.smithi099.stdout: "num_shards": 4, 2023-05-31T21:32:45.265 INFO:teuthology.orchestra.run.smithi099.stdout: "instance_id": 452136274422638824 2023-05-31T21:32:45.265 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:32:45.265 INFO:teuthology.orchestra.run.smithi099.stdout: "markers": [ 2023-05-31T21:32:45.265 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T21:32:45.265 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 0, 2023-05-31T21:32:45.266 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T21:32:45.266 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "incremental-sync", 2023-05-31T21:32:45.266 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "00000000000000000000:00000000000000045515", 2023-05-31T21:32:45.266 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T21:32:45.266 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2023-05-31T21:32:45.266 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T21:32:45.266 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2023-05-31T21:29:57.506834Z" 2023-05-31T21:32:45.267 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T21:32:45.267 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:32:45.267 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T21:32:45.267 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 1, 2023-05-31T21:32:45.267 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T21:32:45.267 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "incremental-sync", 2023-05-31T21:32:45.268 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "00000000000000000000:00000000000000031920", 2023-05-31T21:32:45.268 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T21:32:45.268 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2023-05-31T21:32:45.268 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T21:32:45.268 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2023-05-31T21:29:57.583636Z" 2023-05-31T21:32:45.268 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T21:32:45.268 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:32:45.269 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T21:32:45.269 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 2, 2023-05-31T21:32:45.269 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T21:32:45.269 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "incremental-sync", 2023-05-31T21:32:45.269 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "00000000000000000000:00000000000000051951", 2023-05-31T21:32:45.269 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T21:32:45.269 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 55, 2023-05-31T21:32:45.270 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T21:32:45.270 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2023-05-31T21:29:57.526899Z" 2023-05-31T21:32:45.270 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T21:32:45.270 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:32:45.270 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T21:32:45.270 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 3, 2023-05-31T21:32:45.270 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T21:32:45.271 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "incremental-sync", 2023-05-31T21:32:45.271 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "00000000000000000000:00000000000000044178", 2023-05-31T21:32:45.271 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T21:32:45.271 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 9, 2023-05-31T21:32:45.271 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T21:32:45.271 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2023-05-31T21:27:37.215905Z" 2023-05-31T21:32:45.272 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T21:32:45.272 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T21:32:45.272 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2023-05-31T21:32:45.272 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:32:45.272 INFO:teuthology.orchestra.run.smithi099.stdout: "full_sync": { 2023-05-31T21:32:45.272 INFO:teuthology.orchestra.run.smithi099.stdout: "total": 64, 2023-05-31T21:32:45.272 INFO:teuthology.orchestra.run.smithi099.stdout: "complete": 64 2023-05-31T21:32:45.273 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:32:45.273 INFO:teuthology.orchestra.run.smithi099.stdout: "current_time": "2023-05-31T21:32:45Z" 2023-05-31T21:32:45.273 INFO:teuthology.orchestra.run.smithi099.stdout:} 2023-05-31T21:32:45.274 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 452136274422638824 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000045515", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-05-31T21:29:57.506834Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000031920", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-05-31T21:29:57.583636Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000051951", "next_step_marker": "", "total_entries": 55, "pos": 0, "timestamp": "2023-05-31T21:29:57.526899Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000044178", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2023-05-31T21:27:37.215905Z" } } ] }, "full_sync": { "total": 64, "complete": 64 }, "current_time": "2023-05-31T21:32:45Z" } 2023-05-31T21:32:45.274 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000045515', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-05-31T21:29:57.506834Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000031920', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-05-31T21:29:57.583636Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000051951', 'next_step_marker': '', 'total_entries': 55, 'pos': 0, 'timestamp': '2023-05-31T21:29:57.526899Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000044178', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2023-05-31T21:27:37.215905Z'}}] 2023-05-31T21:32:45.275 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000045655', 1: '00000000000000000000:00000000000000031920', 2: '00000000000000000000:00000000000000052091', 3: '00000000000000000000:00000000000000044178'} 2023-05-31T21:32:45.275 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000045515', 1: '00000000000000000000:00000000000000031920', 2: '00000000000000000000:00000000000000051951', 3: '00000000000000000000:00000000000000044178'} 2023-05-31T21:32:45.275 WARNING:rgw_multi.tests:data of zone a2 behind zone a1: shard=0 master=00000000000000000000:00000000000000045655 target=00000000000000000000:00000000000000045515, shard=2 master=00000000000000000000:00000000000000052091 target=00000000000000000000:00000000000000051951 2023-05-31T21:32:50.276 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-31T21:32:50.277 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-31T21:32:50.278 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-05-31T21:32:50.336 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2023-05-31T21:32:50.337 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2023-05-31T21:32:50.392 INFO:teuthology.orchestra.run.smithi099.stdout:{ 2023-05-31T21:32:50.392 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_status": { 2023-05-31T21:32:50.392 INFO:teuthology.orchestra.run.smithi099.stdout: "info": { 2023-05-31T21:32:50.392 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "sync", 2023-05-31T21:32:50.393 INFO:teuthology.orchestra.run.smithi099.stdout: "num_shards": 4, 2023-05-31T21:32:50.393 INFO:teuthology.orchestra.run.smithi099.stdout: "instance_id": 452136274422638824 2023-05-31T21:32:50.393 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:32:50.393 INFO:teuthology.orchestra.run.smithi099.stdout: "markers": [ 2023-05-31T21:32:50.393 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T21:32:50.393 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 0, 2023-05-31T21:32:50.393 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T21:32:50.394 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "incremental-sync", 2023-05-31T21:32:50.394 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "00000000000000000000:00000000000000045515", 2023-05-31T21:32:50.394 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T21:32:50.394 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2023-05-31T21:32:50.394 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T21:32:50.394 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2023-05-31T21:29:57.506834Z" 2023-05-31T21:32:50.394 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T21:32:50.395 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:32:50.395 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T21:32:50.395 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 1, 2023-05-31T21:32:50.395 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T21:32:50.395 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "incremental-sync", 2023-05-31T21:32:50.395 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "00000000000000000000:00000000000000031920", 2023-05-31T21:32:50.395 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T21:32:50.396 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2023-05-31T21:32:50.396 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T21:32:50.396 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2023-05-31T21:29:57.583636Z" 2023-05-31T21:32:50.396 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T21:32:50.396 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:32:50.396 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T21:32:50.397 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 2, 2023-05-31T21:32:50.397 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T21:32:50.397 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "incremental-sync", 2023-05-31T21:32:50.397 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "00000000000000000000:00000000000000051951", 2023-05-31T21:32:50.397 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T21:32:50.397 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 55, 2023-05-31T21:32:50.397 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T21:32:50.398 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2023-05-31T21:29:57.526899Z" 2023-05-31T21:32:50.398 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T21:32:50.398 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:32:50.398 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T21:32:50.398 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 3, 2023-05-31T21:32:50.398 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T21:32:50.398 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "incremental-sync", 2023-05-31T21:32:50.399 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "00000000000000000000:00000000000000044178", 2023-05-31T21:32:50.399 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T21:32:50.399 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 9, 2023-05-31T21:32:50.399 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T21:32:50.399 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2023-05-31T21:27:37.215905Z" 2023-05-31T21:32:50.399 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T21:32:50.399 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T21:32:50.400 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2023-05-31T21:32:50.400 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:32:50.400 INFO:teuthology.orchestra.run.smithi099.stdout: "full_sync": { 2023-05-31T21:32:50.400 INFO:teuthology.orchestra.run.smithi099.stdout: "total": 64, 2023-05-31T21:32:50.400 INFO:teuthology.orchestra.run.smithi099.stdout: "complete": 64 2023-05-31T21:32:50.401 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:32:50.401 INFO:teuthology.orchestra.run.smithi099.stdout: "current_time": "2023-05-31T21:32:50Z" 2023-05-31T21:32:50.401 INFO:teuthology.orchestra.run.smithi099.stdout:} 2023-05-31T21:32:50.402 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 452136274422638824 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000045515", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-05-31T21:29:57.506834Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000031920", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-05-31T21:29:57.583636Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000051951", "next_step_marker": "", "total_entries": 55, "pos": 0, "timestamp": "2023-05-31T21:29:57.526899Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000044178", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2023-05-31T21:27:37.215905Z" } } ] }, "full_sync": { "total": 64, "complete": 64 }, "current_time": "2023-05-31T21:32:50Z" } 2023-05-31T21:32:50.402 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000045515', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-05-31T21:29:57.506834Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000031920', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-05-31T21:29:57.583636Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000051951', 'next_step_marker': '', 'total_entries': 55, 'pos': 0, 'timestamp': '2023-05-31T21:29:57.526899Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000044178', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2023-05-31T21:27:37.215905Z'}}] 2023-05-31T21:32:50.403 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000045655', 1: '00000000000000000000:00000000000000031920', 2: '00000000000000000000:00000000000000052091', 3: '00000000000000000000:00000000000000044178'} 2023-05-31T21:32:50.403 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000045515', 1: '00000000000000000000:00000000000000031920', 2: '00000000000000000000:00000000000000051951', 3: '00000000000000000000:00000000000000044178'} 2023-05-31T21:32:50.403 WARNING:rgw_multi.tests:data of zone a2 behind zone a1: shard=0 master=00000000000000000000:00000000000000045655 target=00000000000000000000:00000000000000045515, shard=2 master=00000000000000000000:00000000000000052091 target=00000000000000000000:00000000000000051951 2023-05-31T21:32:55.404 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-31T21:32:55.405 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-31T21:32:55.406 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-05-31T21:32:55.464 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2023-05-31T21:32:55.464 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2023-05-31T21:32:55.517 INFO:teuthology.orchestra.run.smithi099.stdout:{ 2023-05-31T21:32:55.517 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_status": { 2023-05-31T21:32:55.517 INFO:teuthology.orchestra.run.smithi099.stdout: "info": { 2023-05-31T21:32:55.517 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "sync", 2023-05-31T21:32:55.518 INFO:teuthology.orchestra.run.smithi099.stdout: "num_shards": 4, 2023-05-31T21:32:55.518 INFO:teuthology.orchestra.run.smithi099.stdout: "instance_id": 452136274422638824 2023-05-31T21:32:55.518 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:32:55.518 INFO:teuthology.orchestra.run.smithi099.stdout: "markers": [ 2023-05-31T21:32:55.518 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T21:32:55.519 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 0, 2023-05-31T21:32:55.519 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T21:32:55.519 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "incremental-sync", 2023-05-31T21:32:55.519 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "00000000000000000000:00000000000000045515", 2023-05-31T21:32:55.519 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T21:32:55.519 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2023-05-31T21:32:55.520 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T21:32:55.520 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2023-05-31T21:29:57.506834Z" 2023-05-31T21:32:55.520 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T21:32:55.520 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:32:55.520 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T21:32:55.520 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 1, 2023-05-31T21:32:55.520 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T21:32:55.521 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "incremental-sync", 2023-05-31T21:32:55.521 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "00000000000000000000:00000000000000031920", 2023-05-31T21:32:55.521 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T21:32:55.521 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2023-05-31T21:32:55.521 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T21:32:55.521 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2023-05-31T21:29:57.583636Z" 2023-05-31T21:32:55.521 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T21:32:55.522 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:32:55.522 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T21:32:55.522 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 2, 2023-05-31T21:32:55.522 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T21:32:55.522 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "incremental-sync", 2023-05-31T21:32:55.522 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "00000000000000000000:00000000000000051951", 2023-05-31T21:32:55.523 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T21:32:55.523 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 55, 2023-05-31T21:32:55.523 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T21:32:55.523 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2023-05-31T21:29:57.526899Z" 2023-05-31T21:32:55.523 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T21:32:55.523 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:32:55.523 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T21:32:55.524 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 3, 2023-05-31T21:32:55.524 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T21:32:55.524 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "incremental-sync", 2023-05-31T21:32:55.524 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "00000000000000000000:00000000000000044178", 2023-05-31T21:32:55.524 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T21:32:55.524 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 9, 2023-05-31T21:32:55.524 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T21:32:55.525 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2023-05-31T21:27:37.215905Z" 2023-05-31T21:32:55.525 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T21:32:55.525 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T21:32:55.525 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2023-05-31T21:32:55.525 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:32:55.525 INFO:teuthology.orchestra.run.smithi099.stdout: "full_sync": { 2023-05-31T21:32:55.525 INFO:teuthology.orchestra.run.smithi099.stdout: "total": 64, 2023-05-31T21:32:55.526 INFO:teuthology.orchestra.run.smithi099.stdout: "complete": 64 2023-05-31T21:32:55.526 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:32:55.526 INFO:teuthology.orchestra.run.smithi099.stdout: "current_time": "2023-05-31T21:32:55Z" 2023-05-31T21:32:55.526 INFO:teuthology.orchestra.run.smithi099.stdout:} 2023-05-31T21:32:55.527 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 452136274422638824 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000045515", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-05-31T21:29:57.506834Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000031920", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-05-31T21:29:57.583636Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000051951", "next_step_marker": "", "total_entries": 55, "pos": 0, "timestamp": "2023-05-31T21:29:57.526899Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000044178", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2023-05-31T21:27:37.215905Z" } } ] }, "full_sync": { "total": 64, "complete": 64 }, "current_time": "2023-05-31T21:32:55Z" } 2023-05-31T21:32:55.527 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000045515', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-05-31T21:29:57.506834Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000031920', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-05-31T21:29:57.583636Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000051951', 'next_step_marker': '', 'total_entries': 55, 'pos': 0, 'timestamp': '2023-05-31T21:29:57.526899Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000044178', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2023-05-31T21:27:37.215905Z'}}] 2023-05-31T21:32:55.528 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000045655', 1: '00000000000000000000:00000000000000031920', 2: '00000000000000000000:00000000000000052091', 3: '00000000000000000000:00000000000000044178'} 2023-05-31T21:32:55.528 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000045515', 1: '00000000000000000000:00000000000000031920', 2: '00000000000000000000:00000000000000051951', 3: '00000000000000000000:00000000000000044178'} 2023-05-31T21:32:55.529 WARNING:rgw_multi.tests:data of zone a2 behind zone a1: shard=0 master=00000000000000000000:00000000000000045655 target=00000000000000000000:00000000000000045515, shard=2 master=00000000000000000000:00000000000000052091 target=00000000000000000000:00000000000000051951 2023-05-31T21:33:00.529 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-31T21:33:00.530 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-31T21:33:00.530 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-05-31T21:33:00.588 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2023-05-31T21:33:00.589 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2023-05-31T21:33:00.665 INFO:teuthology.orchestra.run.smithi099.stdout:{ 2023-05-31T21:33:00.665 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_status": { 2023-05-31T21:33:00.665 INFO:teuthology.orchestra.run.smithi099.stdout: "info": { 2023-05-31T21:33:00.665 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "sync", 2023-05-31T21:33:00.666 INFO:teuthology.orchestra.run.smithi099.stdout: "num_shards": 4, 2023-05-31T21:33:00.666 INFO:teuthology.orchestra.run.smithi099.stdout: "instance_id": 452136274422638824 2023-05-31T21:33:00.666 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:33:00.666 INFO:teuthology.orchestra.run.smithi099.stdout: "markers": [ 2023-05-31T21:33:00.666 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T21:33:00.666 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 0, 2023-05-31T21:33:00.666 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T21:33:00.667 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "incremental-sync", 2023-05-31T21:33:00.667 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "00000000000000000000:00000000000000045515", 2023-05-31T21:33:00.667 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T21:33:00.667 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2023-05-31T21:33:00.667 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T21:33:00.667 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2023-05-31T21:29:57.506834Z" 2023-05-31T21:33:00.667 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T21:33:00.668 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:33:00.668 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T21:33:00.668 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 1, 2023-05-31T21:33:00.668 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T21:33:00.668 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "incremental-sync", 2023-05-31T21:33:00.668 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "00000000000000000000:00000000000000031920", 2023-05-31T21:33:00.669 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T21:33:00.669 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2023-05-31T21:33:00.669 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T21:33:00.669 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2023-05-31T21:29:57.583636Z" 2023-05-31T21:33:00.669 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T21:33:00.669 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:33:00.669 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T21:33:00.670 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 2, 2023-05-31T21:33:00.670 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T21:33:00.670 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "incremental-sync", 2023-05-31T21:33:00.670 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "00000000000000000000:00000000000000051951", 2023-05-31T21:33:00.670 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T21:33:00.670 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 55, 2023-05-31T21:33:00.670 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T21:33:00.671 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2023-05-31T21:29:57.526899Z" 2023-05-31T21:33:00.671 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T21:33:00.671 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:33:00.671 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T21:33:00.671 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 3, 2023-05-31T21:33:00.671 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T21:33:00.671 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "incremental-sync", 2023-05-31T21:33:00.672 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "00000000000000000000:00000000000000044178", 2023-05-31T21:33:00.672 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T21:33:00.672 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 9, 2023-05-31T21:33:00.672 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T21:33:00.672 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2023-05-31T21:27:37.215905Z" 2023-05-31T21:33:00.672 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T21:33:00.673 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T21:33:00.673 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2023-05-31T21:33:00.673 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:33:00.673 INFO:teuthology.orchestra.run.smithi099.stdout: "full_sync": { 2023-05-31T21:33:00.673 INFO:teuthology.orchestra.run.smithi099.stdout: "total": 64, 2023-05-31T21:33:00.673 INFO:teuthology.orchestra.run.smithi099.stdout: "complete": 64 2023-05-31T21:33:00.673 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:33:00.674 INFO:teuthology.orchestra.run.smithi099.stdout: "current_time": "2023-05-31T21:33:00Z" 2023-05-31T21:33:00.674 INFO:teuthology.orchestra.run.smithi099.stdout:} 2023-05-31T21:33:00.674 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 452136274422638824 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000045515", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-05-31T21:29:57.506834Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000031920", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-05-31T21:29:57.583636Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000051951", "next_step_marker": "", "total_entries": 55, "pos": 0, "timestamp": "2023-05-31T21:29:57.526899Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000044178", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2023-05-31T21:27:37.215905Z" } } ] }, "full_sync": { "total": 64, "complete": 64 }, "current_time": "2023-05-31T21:33:00Z" } 2023-05-31T21:33:00.675 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000045515', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-05-31T21:29:57.506834Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000031920', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-05-31T21:29:57.583636Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000051951', 'next_step_marker': '', 'total_entries': 55, 'pos': 0, 'timestamp': '2023-05-31T21:29:57.526899Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000044178', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2023-05-31T21:27:37.215905Z'}}] 2023-05-31T21:33:00.675 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000045655', 1: '00000000000000000000:00000000000000031920', 2: '00000000000000000000:00000000000000052091', 3: '00000000000000000000:00000000000000044178'} 2023-05-31T21:33:00.676 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000045515', 1: '00000000000000000000:00000000000000031920', 2: '00000000000000000000:00000000000000051951', 3: '00000000000000000000:00000000000000044178'} 2023-05-31T21:33:00.676 WARNING:rgw_multi.tests:data of zone a2 behind zone a1: shard=0 master=00000000000000000000:00000000000000045655 target=00000000000000000000:00000000000000045515, shard=2 master=00000000000000000000:00000000000000052091 target=00000000000000000000:00000000000000051951 2023-05-31T21:33:05.677 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-31T21:33:05.678 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-31T21:33:05.678 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-05-31T21:33:05.736 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2023-05-31T21:33:05.737 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2023-05-31T21:33:05.793 INFO:teuthology.orchestra.run.smithi099.stdout:{ 2023-05-31T21:33:05.794 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_status": { 2023-05-31T21:33:05.794 INFO:teuthology.orchestra.run.smithi099.stdout: "info": { 2023-05-31T21:33:05.794 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "sync", 2023-05-31T21:33:05.794 INFO:teuthology.orchestra.run.smithi099.stdout: "num_shards": 4, 2023-05-31T21:33:05.794 INFO:teuthology.orchestra.run.smithi099.stdout: "instance_id": 452136274422638824 2023-05-31T21:33:05.795 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:33:05.795 INFO:teuthology.orchestra.run.smithi099.stdout: "markers": [ 2023-05-31T21:33:05.795 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T21:33:05.795 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 0, 2023-05-31T21:33:05.795 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T21:33:05.795 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "incremental-sync", 2023-05-31T21:33:05.795 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "00000000000000000000:00000000000000045515", 2023-05-31T21:33:05.796 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T21:33:05.796 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2023-05-31T21:33:05.796 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T21:33:05.796 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2023-05-31T21:29:57.506834Z" 2023-05-31T21:33:05.796 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T21:33:05.796 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:33:05.797 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T21:33:05.797 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 1, 2023-05-31T21:33:05.797 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T21:33:05.797 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "incremental-sync", 2023-05-31T21:33:05.797 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "00000000000000000000:00000000000000031920", 2023-05-31T21:33:05.797 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T21:33:05.798 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2023-05-31T21:33:05.798 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T21:33:05.798 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2023-05-31T21:29:57.583636Z" 2023-05-31T21:33:05.798 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T21:33:05.798 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:33:05.798 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T21:33:05.799 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 2, 2023-05-31T21:33:05.799 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T21:33:05.799 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "incremental-sync", 2023-05-31T21:33:05.799 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "00000000000000000000:00000000000000051951", 2023-05-31T21:33:05.799 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T21:33:05.799 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 55, 2023-05-31T21:33:05.799 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T21:33:05.800 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2023-05-31T21:29:57.526899Z" 2023-05-31T21:33:05.800 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T21:33:05.800 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:33:05.800 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T21:33:05.800 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 3, 2023-05-31T21:33:05.800 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T21:33:05.800 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "incremental-sync", 2023-05-31T21:33:05.801 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "00000000000000000000:00000000000000044178", 2023-05-31T21:33:05.801 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T21:33:05.801 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 9, 2023-05-31T21:33:05.801 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T21:33:05.801 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2023-05-31T21:27:37.215905Z" 2023-05-31T21:33:05.801 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T21:33:05.801 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T21:33:05.802 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2023-05-31T21:33:05.802 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:33:05.802 INFO:teuthology.orchestra.run.smithi099.stdout: "full_sync": { 2023-05-31T21:33:05.802 INFO:teuthology.orchestra.run.smithi099.stdout: "total": 64, 2023-05-31T21:33:05.802 INFO:teuthology.orchestra.run.smithi099.stdout: "complete": 64 2023-05-31T21:33:05.802 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:33:05.802 INFO:teuthology.orchestra.run.smithi099.stdout: "current_time": "2023-05-31T21:33:05Z" 2023-05-31T21:33:05.803 INFO:teuthology.orchestra.run.smithi099.stdout:} 2023-05-31T21:33:05.803 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 452136274422638824 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000045515", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-05-31T21:29:57.506834Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000031920", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-05-31T21:29:57.583636Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000051951", "next_step_marker": "", "total_entries": 55, "pos": 0, "timestamp": "2023-05-31T21:29:57.526899Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000044178", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2023-05-31T21:27:37.215905Z" } } ] }, "full_sync": { "total": 64, "complete": 64 }, "current_time": "2023-05-31T21:33:05Z" } 2023-05-31T21:33:05.804 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000045515', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-05-31T21:29:57.506834Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000031920', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-05-31T21:29:57.583636Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000051951', 'next_step_marker': '', 'total_entries': 55, 'pos': 0, 'timestamp': '2023-05-31T21:29:57.526899Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000044178', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2023-05-31T21:27:37.215905Z'}}] 2023-05-31T21:33:05.804 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000045655', 1: '00000000000000000000:00000000000000031920', 2: '00000000000000000000:00000000000000052091', 3: '00000000000000000000:00000000000000044178'} 2023-05-31T21:33:05.805 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000045515', 1: '00000000000000000000:00000000000000031920', 2: '00000000000000000000:00000000000000051951', 3: '00000000000000000000:00000000000000044178'} 2023-05-31T21:33:05.805 WARNING:rgw_multi.tests:data of zone a2 behind zone a1: shard=0 master=00000000000000000000:00000000000000045655 target=00000000000000000000:00000000000000045515, shard=2 master=00000000000000000000:00000000000000052091 target=00000000000000000000:00000000000000051951 2023-05-31T21:33:10.806 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-31T21:33:10.807 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-31T21:33:10.808 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-05-31T21:33:10.866 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2023-05-31T21:33:10.867 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2023-05-31T21:33:10.944 INFO:teuthology.orchestra.run.smithi099.stdout:{ 2023-05-31T21:33:10.945 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_status": { 2023-05-31T21:33:10.945 INFO:teuthology.orchestra.run.smithi099.stdout: "info": { 2023-05-31T21:33:10.945 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "sync", 2023-05-31T21:33:10.945 INFO:teuthology.orchestra.run.smithi099.stdout: "num_shards": 4, 2023-05-31T21:33:10.945 INFO:teuthology.orchestra.run.smithi099.stdout: "instance_id": 452136274422638824 2023-05-31T21:33:10.946 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:33:10.946 INFO:teuthology.orchestra.run.smithi099.stdout: "markers": [ 2023-05-31T21:33:10.946 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T21:33:10.946 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 0, 2023-05-31T21:33:10.946 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T21:33:10.946 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "incremental-sync", 2023-05-31T21:33:10.946 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "00000000000000000000:00000000000000045515", 2023-05-31T21:33:10.947 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T21:33:10.947 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2023-05-31T21:33:10.947 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T21:33:10.947 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2023-05-31T21:29:57.506834Z" 2023-05-31T21:33:10.947 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T21:33:10.947 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:33:10.947 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T21:33:10.948 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 1, 2023-05-31T21:33:10.948 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T21:33:10.948 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "incremental-sync", 2023-05-31T21:33:10.948 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "00000000000000000000:00000000000000031920", 2023-05-31T21:33:10.948 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T21:33:10.948 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2023-05-31T21:33:10.949 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T21:33:10.949 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2023-05-31T21:29:57.583636Z" 2023-05-31T21:33:10.949 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T21:33:10.949 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:33:10.949 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T21:33:10.949 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 2, 2023-05-31T21:33:10.949 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T21:33:10.950 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "incremental-sync", 2023-05-31T21:33:10.950 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "00000000000000000000:00000000000000051951", 2023-05-31T21:33:10.950 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T21:33:10.950 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 55, 2023-05-31T21:33:10.950 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T21:33:10.950 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2023-05-31T21:29:57.526899Z" 2023-05-31T21:33:10.950 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T21:33:10.951 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:33:10.951 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T21:33:10.951 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 3, 2023-05-31T21:33:10.951 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T21:33:10.951 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "incremental-sync", 2023-05-31T21:33:10.951 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "00000000000000000000:00000000000000044178", 2023-05-31T21:33:10.951 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T21:33:10.952 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 9, 2023-05-31T21:33:10.952 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T21:33:10.952 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2023-05-31T21:27:37.215905Z" 2023-05-31T21:33:10.952 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T21:33:10.952 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T21:33:10.952 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2023-05-31T21:33:10.952 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:33:10.953 INFO:teuthology.orchestra.run.smithi099.stdout: "full_sync": { 2023-05-31T21:33:10.953 INFO:teuthology.orchestra.run.smithi099.stdout: "total": 64, 2023-05-31T21:33:10.953 INFO:teuthology.orchestra.run.smithi099.stdout: "complete": 64 2023-05-31T21:33:10.953 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:33:10.953 INFO:teuthology.orchestra.run.smithi099.stdout: "current_time": "2023-05-31T21:33:10Z" 2023-05-31T21:33:10.953 INFO:teuthology.orchestra.run.smithi099.stdout:} 2023-05-31T21:33:10.954 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 452136274422638824 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000045515", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-05-31T21:29:57.506834Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000031920", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-05-31T21:29:57.583636Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000051951", "next_step_marker": "", "total_entries": 55, "pos": 0, "timestamp": "2023-05-31T21:29:57.526899Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000044178", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2023-05-31T21:27:37.215905Z" } } ] }, "full_sync": { "total": 64, "complete": 64 }, "current_time": "2023-05-31T21:33:10Z" } 2023-05-31T21:33:10.955 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000045515', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-05-31T21:29:57.506834Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000031920', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-05-31T21:29:57.583636Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000051951', 'next_step_marker': '', 'total_entries': 55, 'pos': 0, 'timestamp': '2023-05-31T21:29:57.526899Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000044178', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2023-05-31T21:27:37.215905Z'}}] 2023-05-31T21:33:10.955 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000045655', 1: '00000000000000000000:00000000000000031920', 2: '00000000000000000000:00000000000000052091', 3: '00000000000000000000:00000000000000044178'} 2023-05-31T21:33:10.955 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000045515', 1: '00000000000000000000:00000000000000031920', 2: '00000000000000000000:00000000000000051951', 3: '00000000000000000000:00000000000000044178'} 2023-05-31T21:33:10.956 WARNING:rgw_multi.tests:data of zone a2 behind zone a1: shard=0 master=00000000000000000000:00000000000000045655 target=00000000000000000000:00000000000000045515, shard=2 master=00000000000000000000:00000000000000052091 target=00000000000000000000:00000000000000051951 2023-05-31T21:33:15.959 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-31T21:33:15.960 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-31T21:33:15.961 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-05-31T21:33:16.020 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2023-05-31T21:33:16.020 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2023-05-31T21:33:16.075 INFO:teuthology.orchestra.run.smithi099.stdout:{ 2023-05-31T21:33:16.075 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_status": { 2023-05-31T21:33:16.075 INFO:teuthology.orchestra.run.smithi099.stdout: "info": { 2023-05-31T21:33:16.075 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "sync", 2023-05-31T21:33:16.075 INFO:teuthology.orchestra.run.smithi099.stdout: "num_shards": 4, 2023-05-31T21:33:16.075 INFO:teuthology.orchestra.run.smithi099.stdout: "instance_id": 452136274422638824 2023-05-31T21:33:16.076 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:33:16.076 INFO:teuthology.orchestra.run.smithi099.stdout: "markers": [ 2023-05-31T21:33:16.076 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T21:33:16.076 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 0, 2023-05-31T21:33:16.076 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T21:33:16.076 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "incremental-sync", 2023-05-31T21:33:16.077 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "00000000000000000000:00000000000000045515", 2023-05-31T21:33:16.077 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T21:33:16.077 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2023-05-31T21:33:16.077 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T21:33:16.077 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2023-05-31T21:29:57.506834Z" 2023-05-31T21:33:16.077 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T21:33:16.077 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:33:16.078 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T21:33:16.078 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 1, 2023-05-31T21:33:16.078 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T21:33:16.078 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "incremental-sync", 2023-05-31T21:33:16.078 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "00000000000000000000:00000000000000031920", 2023-05-31T21:33:16.078 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T21:33:16.078 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2023-05-31T21:33:16.079 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T21:33:16.079 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2023-05-31T21:29:57.583636Z" 2023-05-31T21:33:16.079 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T21:33:16.079 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:33:16.079 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T21:33:16.080 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 2, 2023-05-31T21:33:16.080 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T21:33:16.080 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "incremental-sync", 2023-05-31T21:33:16.080 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "00000000000000000000:00000000000000051951", 2023-05-31T21:33:16.080 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T21:33:16.081 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 55, 2023-05-31T21:33:16.081 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T21:33:16.081 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2023-05-31T21:29:57.526899Z" 2023-05-31T21:33:16.081 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T21:33:16.081 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:33:16.081 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T21:33:16.081 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 3, 2023-05-31T21:33:16.082 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T21:33:16.082 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "incremental-sync", 2023-05-31T21:33:16.082 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "00000000000000000000:00000000000000044178", 2023-05-31T21:33:16.082 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T21:33:16.082 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 9, 2023-05-31T21:33:16.082 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T21:33:16.082 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2023-05-31T21:27:37.215905Z" 2023-05-31T21:33:16.083 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T21:33:16.083 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T21:33:16.083 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2023-05-31T21:33:16.083 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:33:16.083 INFO:teuthology.orchestra.run.smithi099.stdout: "full_sync": { 2023-05-31T21:33:16.083 INFO:teuthology.orchestra.run.smithi099.stdout: "total": 64, 2023-05-31T21:33:16.083 INFO:teuthology.orchestra.run.smithi099.stdout: "complete": 64 2023-05-31T21:33:16.084 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:33:16.084 INFO:teuthology.orchestra.run.smithi099.stdout: "current_time": "2023-05-31T21:33:16Z" 2023-05-31T21:33:16.084 INFO:teuthology.orchestra.run.smithi099.stdout:} 2023-05-31T21:33:16.085 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 452136274422638824 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000045515", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-05-31T21:29:57.506834Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000031920", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-05-31T21:29:57.583636Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000051951", "next_step_marker": "", "total_entries": 55, "pos": 0, "timestamp": "2023-05-31T21:29:57.526899Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000044178", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2023-05-31T21:27:37.215905Z" } } ] }, "full_sync": { "total": 64, "complete": 64 }, "current_time": "2023-05-31T21:33:16Z" } 2023-05-31T21:33:16.085 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000045515', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-05-31T21:29:57.506834Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000031920', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-05-31T21:29:57.583636Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000051951', 'next_step_marker': '', 'total_entries': 55, 'pos': 0, 'timestamp': '2023-05-31T21:29:57.526899Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000044178', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2023-05-31T21:27:37.215905Z'}}] 2023-05-31T21:33:16.086 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000045655', 1: '00000000000000000000:00000000000000031920', 2: '00000000000000000000:00000000000000052091', 3: '00000000000000000000:00000000000000044178'} 2023-05-31T21:33:16.086 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000045515', 1: '00000000000000000000:00000000000000031920', 2: '00000000000000000000:00000000000000051951', 3: '00000000000000000000:00000000000000044178'} 2023-05-31T21:33:16.086 WARNING:rgw_multi.tests:data of zone a2 behind zone a1: shard=0 master=00000000000000000000:00000000000000045655 target=00000000000000000000:00000000000000045515, shard=2 master=00000000000000000000:00000000000000052091 target=00000000000000000000:00000000000000051951 2023-05-31T21:33:21.087 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-31T21:33:21.088 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-31T21:33:21.088 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-05-31T21:33:21.148 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2023-05-31T21:33:21.148 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2023-05-31T21:33:21.225 INFO:teuthology.orchestra.run.smithi099.stdout:{ 2023-05-31T21:33:21.225 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_status": { 2023-05-31T21:33:21.225 INFO:teuthology.orchestra.run.smithi099.stdout: "info": { 2023-05-31T21:33:21.226 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "sync", 2023-05-31T21:33:21.226 INFO:teuthology.orchestra.run.smithi099.stdout: "num_shards": 4, 2023-05-31T21:33:21.226 INFO:teuthology.orchestra.run.smithi099.stdout: "instance_id": 452136274422638824 2023-05-31T21:33:21.226 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:33:21.226 INFO:teuthology.orchestra.run.smithi099.stdout: "markers": [ 2023-05-31T21:33:21.226 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T21:33:21.227 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 0, 2023-05-31T21:33:21.227 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T21:33:21.227 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "incremental-sync", 2023-05-31T21:33:21.227 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "00000000000000000000:00000000000000045515", 2023-05-31T21:33:21.227 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T21:33:21.227 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2023-05-31T21:33:21.227 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T21:33:21.228 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2023-05-31T21:29:57.506834Z" 2023-05-31T21:33:21.228 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T21:33:21.228 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:33:21.228 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T21:33:21.228 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 1, 2023-05-31T21:33:21.229 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T21:33:21.229 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "incremental-sync", 2023-05-31T21:33:21.229 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "00000000000000000000:00000000000000031920", 2023-05-31T21:33:21.229 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T21:33:21.229 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2023-05-31T21:33:21.230 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T21:33:21.230 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2023-05-31T21:29:57.583636Z" 2023-05-31T21:33:21.230 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T21:33:21.230 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:33:21.230 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T21:33:21.230 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 2, 2023-05-31T21:33:21.231 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T21:33:21.231 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "incremental-sync", 2023-05-31T21:33:21.231 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "00000000000000000000:00000000000000051951", 2023-05-31T21:33:21.232 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T21:33:21.232 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 55, 2023-05-31T21:33:21.232 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T21:33:21.233 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2023-05-31T21:29:57.526899Z" 2023-05-31T21:33:21.233 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T21:33:21.233 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:33:21.233 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T21:33:21.234 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 3, 2023-05-31T21:33:21.234 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T21:33:21.234 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "incremental-sync", 2023-05-31T21:33:21.234 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "00000000000000000000:00000000000000044178", 2023-05-31T21:33:21.234 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T21:33:21.235 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 9, 2023-05-31T21:33:21.235 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T21:33:21.235 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2023-05-31T21:27:37.215905Z" 2023-05-31T21:33:21.235 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T21:33:21.235 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T21:33:21.236 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2023-05-31T21:33:21.236 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:33:21.236 INFO:teuthology.orchestra.run.smithi099.stdout: "full_sync": { 2023-05-31T21:33:21.236 INFO:teuthology.orchestra.run.smithi099.stdout: "total": 64, 2023-05-31T21:33:21.236 INFO:teuthology.orchestra.run.smithi099.stdout: "complete": 64 2023-05-31T21:33:21.236 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:33:21.236 INFO:teuthology.orchestra.run.smithi099.stdout: "current_time": "2023-05-31T21:33:21Z" 2023-05-31T21:33:21.237 INFO:teuthology.orchestra.run.smithi099.stdout:} 2023-05-31T21:33:21.237 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 452136274422638824 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000045515", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-05-31T21:29:57.506834Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000031920", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-05-31T21:29:57.583636Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000051951", "next_step_marker": "", "total_entries": 55, "pos": 0, "timestamp": "2023-05-31T21:29:57.526899Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000044178", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2023-05-31T21:27:37.215905Z" } } ] }, "full_sync": { "total": 64, "complete": 64 }, "current_time": "2023-05-31T21:33:21Z" } 2023-05-31T21:33:21.238 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000045515', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-05-31T21:29:57.506834Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000031920', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-05-31T21:29:57.583636Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000051951', 'next_step_marker': '', 'total_entries': 55, 'pos': 0, 'timestamp': '2023-05-31T21:29:57.526899Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000044178', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2023-05-31T21:27:37.215905Z'}}] 2023-05-31T21:33:21.238 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000045655', 1: '00000000000000000000:00000000000000031920', 2: '00000000000000000000:00000000000000052091', 3: '00000000000000000000:00000000000000044178'} 2023-05-31T21:33:21.239 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000045515', 1: '00000000000000000000:00000000000000031920', 2: '00000000000000000000:00000000000000051951', 3: '00000000000000000000:00000000000000044178'} 2023-05-31T21:33:21.239 WARNING:rgw_multi.tests:data of zone a2 behind zone a1: shard=0 master=00000000000000000000:00000000000000045655 target=00000000000000000000:00000000000000045515, shard=2 master=00000000000000000000:00000000000000052091 target=00000000000000000000:00000000000000051951 2023-05-31T21:33:26.240 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-31T21:33:26.241 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-31T21:33:26.241 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-05-31T21:33:26.300 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2023-05-31T21:33:26.301 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2023-05-31T21:33:26.365 INFO:teuthology.orchestra.run.smithi099.stdout:{ 2023-05-31T21:33:26.365 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_status": { 2023-05-31T21:33:26.366 INFO:teuthology.orchestra.run.smithi099.stdout: "info": { 2023-05-31T21:33:26.366 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "sync", 2023-05-31T21:33:26.366 INFO:teuthology.orchestra.run.smithi099.stdout: "num_shards": 4, 2023-05-31T21:33:26.366 INFO:teuthology.orchestra.run.smithi099.stdout: "instance_id": 452136274422638824 2023-05-31T21:33:26.366 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:33:26.366 INFO:teuthology.orchestra.run.smithi099.stdout: "markers": [ 2023-05-31T21:33:26.367 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T21:33:26.367 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 0, 2023-05-31T21:33:26.367 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T21:33:26.367 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "incremental-sync", 2023-05-31T21:33:26.367 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "00000000000000000000:00000000000000045515", 2023-05-31T21:33:26.367 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T21:33:26.367 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2023-05-31T21:33:26.368 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T21:33:26.368 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2023-05-31T21:29:57.506834Z" 2023-05-31T21:33:26.368 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T21:33:26.368 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:33:26.368 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T21:33:26.368 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 1, 2023-05-31T21:33:26.369 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T21:33:26.369 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "incremental-sync", 2023-05-31T21:33:26.369 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "00000000000000000000:00000000000000031920", 2023-05-31T21:33:26.369 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T21:33:26.369 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2023-05-31T21:33:26.369 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T21:33:26.369 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2023-05-31T21:29:57.583636Z" 2023-05-31T21:33:26.370 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T21:33:26.370 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:33:26.370 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T21:33:26.370 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 2, 2023-05-31T21:33:26.370 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T21:33:26.370 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "incremental-sync", 2023-05-31T21:33:26.370 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "00000000000000000000:00000000000000051951", 2023-05-31T21:33:26.371 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T21:33:26.371 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 55, 2023-05-31T21:33:26.371 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T21:33:26.371 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2023-05-31T21:29:57.526899Z" 2023-05-31T21:33:26.371 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T21:33:26.371 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:33:26.371 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T21:33:26.372 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 3, 2023-05-31T21:33:26.372 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T21:33:26.372 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "incremental-sync", 2023-05-31T21:33:26.372 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "00000000000000000000:00000000000000044178", 2023-05-31T21:33:26.372 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T21:33:26.372 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 9, 2023-05-31T21:33:26.373 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T21:33:26.373 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2023-05-31T21:27:37.215905Z" 2023-05-31T21:33:26.373 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T21:33:26.373 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T21:33:26.373 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2023-05-31T21:33:26.373 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:33:26.373 INFO:teuthology.orchestra.run.smithi099.stdout: "full_sync": { 2023-05-31T21:33:26.374 INFO:teuthology.orchestra.run.smithi099.stdout: "total": 64, 2023-05-31T21:33:26.374 INFO:teuthology.orchestra.run.smithi099.stdout: "complete": 64 2023-05-31T21:33:26.374 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:33:26.374 INFO:teuthology.orchestra.run.smithi099.stdout: "current_time": "2023-05-31T21:33:26Z" 2023-05-31T21:33:26.374 INFO:teuthology.orchestra.run.smithi099.stdout:} 2023-05-31T21:33:26.375 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 452136274422638824 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000045515", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-05-31T21:29:57.506834Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000031920", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-05-31T21:29:57.583636Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000051951", "next_step_marker": "", "total_entries": 55, "pos": 0, "timestamp": "2023-05-31T21:29:57.526899Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000044178", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2023-05-31T21:27:37.215905Z" } } ] }, "full_sync": { "total": 64, "complete": 64 }, "current_time": "2023-05-31T21:33:26Z" } 2023-05-31T21:33:26.375 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000045515', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-05-31T21:29:57.506834Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000031920', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-05-31T21:29:57.583636Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000051951', 'next_step_marker': '', 'total_entries': 55, 'pos': 0, 'timestamp': '2023-05-31T21:29:57.526899Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000044178', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2023-05-31T21:27:37.215905Z'}}] 2023-05-31T21:33:26.376 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000045655', 1: '00000000000000000000:00000000000000031920', 2: '00000000000000000000:00000000000000052091', 3: '00000000000000000000:00000000000000044178'} 2023-05-31T21:33:26.376 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000045515', 1: '00000000000000000000:00000000000000031920', 2: '00000000000000000000:00000000000000051951', 3: '00000000000000000000:00000000000000044178'} 2023-05-31T21:33:26.376 WARNING:rgw_multi.tests:data of zone a2 behind zone a1: shard=0 master=00000000000000000000:00000000000000045655 target=00000000000000000000:00000000000000045515, shard=2 master=00000000000000000000:00000000000000052091 target=00000000000000000000:00000000000000051951 2023-05-31T21:33:31.377 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-31T21:33:31.378 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-31T21:33:31.379 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-05-31T21:33:31.437 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2023-05-31T21:33:31.437 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2023-05-31T21:33:31.505 INFO:teuthology.orchestra.run.smithi099.stdout:{ 2023-05-31T21:33:31.505 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_status": { 2023-05-31T21:33:31.505 INFO:teuthology.orchestra.run.smithi099.stdout: "info": { 2023-05-31T21:33:31.506 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "sync", 2023-05-31T21:33:31.506 INFO:teuthology.orchestra.run.smithi099.stdout: "num_shards": 4, 2023-05-31T21:33:31.506 INFO:teuthology.orchestra.run.smithi099.stdout: "instance_id": 452136274422638824 2023-05-31T21:33:31.506 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:33:31.506 INFO:teuthology.orchestra.run.smithi099.stdout: "markers": [ 2023-05-31T21:33:31.506 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T21:33:31.507 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 0, 2023-05-31T21:33:31.507 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T21:33:31.507 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "incremental-sync", 2023-05-31T21:33:31.507 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "00000000000000000000:00000000000000045655", 2023-05-31T21:33:31.507 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T21:33:31.507 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2023-05-31T21:33:31.507 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T21:33:31.508 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2023-05-31T21:31:27.421078Z" 2023-05-31T21:33:31.508 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T21:33:31.508 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:33:31.508 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T21:33:31.508 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 1, 2023-05-31T21:33:31.508 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T21:33:31.508 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "incremental-sync", 2023-05-31T21:33:31.509 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "00000000000000000000:00000000000000031920", 2023-05-31T21:33:31.509 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T21:33:31.509 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2023-05-31T21:33:31.509 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T21:33:31.509 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2023-05-31T21:29:57.583636Z" 2023-05-31T21:33:31.509 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T21:33:31.510 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:33:31.510 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T21:33:31.510 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 2, 2023-05-31T21:33:31.510 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T21:33:31.510 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "incremental-sync", 2023-05-31T21:33:31.510 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "00000000000000000000:00000000000000052091", 2023-05-31T21:33:31.510 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T21:33:31.511 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 55, 2023-05-31T21:33:31.511 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T21:33:31.511 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2023-05-31T21:31:27.385416Z" 2023-05-31T21:33:31.511 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T21:33:31.511 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:33:31.511 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T21:33:31.511 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 3, 2023-05-31T21:33:31.512 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T21:33:31.512 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "incremental-sync", 2023-05-31T21:33:31.512 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "00000000000000000000:00000000000000044178", 2023-05-31T21:33:31.512 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T21:33:31.512 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 9, 2023-05-31T21:33:31.512 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T21:33:31.512 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2023-05-31T21:27:37.215905Z" 2023-05-31T21:33:31.513 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T21:33:31.513 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T21:33:31.513 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2023-05-31T21:33:31.513 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:33:31.513 INFO:teuthology.orchestra.run.smithi099.stdout: "full_sync": { 2023-05-31T21:33:31.513 INFO:teuthology.orchestra.run.smithi099.stdout: "total": 64, 2023-05-31T21:33:31.513 INFO:teuthology.orchestra.run.smithi099.stdout: "complete": 64 2023-05-31T21:33:31.514 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:33:31.514 INFO:teuthology.orchestra.run.smithi099.stdout: "current_time": "2023-05-31T21:33:31Z" 2023-05-31T21:33:31.514 INFO:teuthology.orchestra.run.smithi099.stdout:} 2023-05-31T21:33:31.515 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 452136274422638824 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000045655", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-05-31T21:31:27.421078Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000031920", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-05-31T21:29:57.583636Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000052091", "next_step_marker": "", "total_entries": 55, "pos": 0, "timestamp": "2023-05-31T21:31:27.385416Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000044178", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2023-05-31T21:27:37.215905Z" } } ] }, "full_sync": { "total": 64, "complete": 64 }, "current_time": "2023-05-31T21:33:31Z" } 2023-05-31T21:33:31.515 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000045655', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-05-31T21:31:27.421078Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000031920', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-05-31T21:29:57.583636Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000052091', 'next_step_marker': '', 'total_entries': 55, 'pos': 0, 'timestamp': '2023-05-31T21:31:27.385416Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000044178', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2023-05-31T21:27:37.215905Z'}}] 2023-05-31T21:33:31.516 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000045655', 1: '00000000000000000000:00000000000000031920', 2: '00000000000000000000:00000000000000052091', 3: '00000000000000000000:00000000000000044178'} 2023-05-31T21:33:31.516 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000045655', 1: '00000000000000000000:00000000000000031920', 2: '00000000000000000000:00000000000000052091', 3: '00000000000000000000:00000000000000044178'} 2023-05-31T21:33:31.516 INFO:rgw_multi.tests:finished data checkpoint for target_zone=a2 source_zone=a1 2023-05-31T21:33:31.517 DEBUG:boto:path=/pklxkd-74/ 2023-05-31T21:33:31.517 DEBUG:boto:auth_path=/pklxkd-74/ 2023-05-31T21:33:31.517 DEBUG:boto:Method: HEAD 2023-05-31T21:33:31.518 DEBUG:boto:Path: /pklxkd-74/ 2023-05-31T21:33:31.518 DEBUG:boto:Data: 2023-05-31T21:33:31.518 DEBUG:boto:Headers: {} 2023-05-31T21:33:31.519 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T21:33:31.519 DEBUG:boto:Port: 8001 2023-05-31T21:33:31.519 DEBUG:boto:Params: {} 2023-05-31T21:33:31.520 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8001} 2023-05-31T21:33:31.520 DEBUG:boto:Token: None 2023-05-31T21:33:31.521 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 21:33:31 GMT /pklxkd-74/ 2023-05-31T21:33:31.521 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:l4H+iXRR3CIa5MoEA49Q1YMqK94= 2023-05-31T21:33:31.522 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 21:33:31 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:l4H+iXRR3CIa5MoEA49Q1YMqK94=', 'Content-Length': '0'} 2023-05-31T21:33:31.526 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '2'), ('X-RGW-Bytes-Used', '12'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000aae46c50c9773d3f-006477bd2b-2aea-a2'), ('Content-Length', '0'), ('Date', 'Wed, 31 May 2023 21:33:31 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:33:31.527 DEBUG:boto:path=/pklxkd-74/obj1 2023-05-31T21:33:31.527 DEBUG:boto:auth_path=/pklxkd-74/obj1 2023-05-31T21:33:31.527 DEBUG:boto:Method: HEAD 2023-05-31T21:33:31.528 DEBUG:boto:Path: /pklxkd-74/obj1 2023-05-31T21:33:31.528 DEBUG:boto:Data: 2023-05-31T21:33:31.528 DEBUG:boto:Headers: {} 2023-05-31T21:33:31.529 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T21:33:31.529 DEBUG:boto:Port: 8001 2023-05-31T21:33:31.529 DEBUG:boto:Params: {} 2023-05-31T21:33:31.530 DEBUG:boto:Token: None 2023-05-31T21:33:31.530 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 21:33:31 GMT /pklxkd-74/obj1 2023-05-31T21:33:31.531 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:Q34w0WYsXBHPwabV+cLtl0yQehk= 2023-05-31T21:33:31.531 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 21:33:31 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:Q34w0WYsXBHPwabV+cLtl0yQehk=', 'Content-Length': '0'} 2023-05-31T21:33:31.533 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 21:31:27 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000e91124af24bde8c9-006477bd2b-2aea-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 21:33:31 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:33:31.534 DEBUG:boto:path=/pklxkd-74/obj1 2023-05-31T21:33:31.534 DEBUG:boto:auth_path=/pklxkd-74/obj1 2023-05-31T21:33:31.534 DEBUG:boto:Method: GET 2023-05-31T21:33:31.535 DEBUG:boto:Path: /pklxkd-74/obj1 2023-05-31T21:33:31.535 DEBUG:boto:Data: 2023-05-31T21:33:31.535 DEBUG:boto:Headers: {} 2023-05-31T21:33:31.536 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T21:33:31.536 DEBUG:boto:Port: 8001 2023-05-31T21:33:31.537 DEBUG:boto:Params: {} 2023-05-31T21:33:31.537 DEBUG:boto:Token: None 2023-05-31T21:33:31.537 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 21:33:31 GMT /pklxkd-74/obj1 2023-05-31T21:33:31.538 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:vBMeHYD2jsI6hyoS6O50RD7YJTk= 2023-05-31T21:33:31.538 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 21:33:31 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:vBMeHYD2jsI6hyoS6O50RD7YJTk=', 'Content-Length': '0'} 2023-05-31T21:33:31.540 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 21:31:27 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000006d634ce2ef0e1b10-006477bd2b-2aea-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 21:33:31 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:33:31.540 DEBUG:boto:path=/pklxkd-74/obj2 2023-05-31T21:33:31.541 DEBUG:boto:auth_path=/pklxkd-74/obj2 2023-05-31T21:33:31.541 DEBUG:boto:Method: HEAD 2023-05-31T21:33:31.542 DEBUG:boto:Path: /pklxkd-74/obj2 2023-05-31T21:33:31.542 DEBUG:boto:Data: 2023-05-31T21:33:31.542 DEBUG:boto:Headers: {} 2023-05-31T21:33:31.543 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T21:33:31.543 DEBUG:boto:Port: 8001 2023-05-31T21:33:31.543 DEBUG:boto:Params: {} 2023-05-31T21:33:31.544 DEBUG:boto:Token: None 2023-05-31T21:33:31.544 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 21:33:31 GMT /pklxkd-74/obj2 2023-05-31T21:33:31.544 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:eC9B0GT7UaHtfuJa1i0lI7QH4S8= 2023-05-31T21:33:31.545 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 21:33:31 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:eC9B0GT7UaHtfuJa1i0lI7QH4S8=', 'Content-Length': '0'} 2023-05-31T21:33:31.546 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 21:31:27 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000075867fb44c3c0878-006477bd2b-2aea-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 21:33:31 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:33:31.547 DEBUG:boto:path=/pklxkd-74/obj2 2023-05-31T21:33:31.547 DEBUG:boto:auth_path=/pklxkd-74/obj2 2023-05-31T21:33:31.548 DEBUG:boto:Method: GET 2023-05-31T21:33:31.548 DEBUG:boto:Path: /pklxkd-74/obj2 2023-05-31T21:33:31.548 DEBUG:boto:Data: 2023-05-31T21:33:31.549 DEBUG:boto:Headers: {} 2023-05-31T21:33:31.549 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T21:33:31.549 DEBUG:boto:Port: 8001 2023-05-31T21:33:31.550 DEBUG:boto:Params: {} 2023-05-31T21:33:31.550 DEBUG:boto:Token: None 2023-05-31T21:33:31.550 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 21:33:31 GMT /pklxkd-74/obj2 2023-05-31T21:33:31.551 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:enUIyJDvTR6qFhv2XsgTMZC/r1Y= 2023-05-31T21:33:31.551 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 21:33:31 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:enUIyJDvTR6qFhv2XsgTMZC/r1Y=', 'Content-Length': '0'} 2023-05-31T21:33:31.553 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 21:31:27 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000015fdf4564f713cb6-006477bd2b-2aea-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 21:33:31 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:33:31.553 DEBUG:boto:path=/pklxkd-75/ 2023-05-31T21:33:31.554 DEBUG:boto:auth_path=/pklxkd-75/ 2023-05-31T21:33:31.554 DEBUG:boto:Method: HEAD 2023-05-31T21:33:31.554 DEBUG:boto:Path: /pklxkd-75/ 2023-05-31T21:33:31.555 DEBUG:boto:Data: 2023-05-31T21:33:31.555 DEBUG:boto:Headers: {} 2023-05-31T21:33:31.555 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T21:33:31.556 DEBUG:boto:Port: 8001 2023-05-31T21:33:31.556 DEBUG:boto:Params: {} 2023-05-31T21:33:31.556 DEBUG:boto:Token: None 2023-05-31T21:33:31.557 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 21:33:31 GMT /pklxkd-75/ 2023-05-31T21:33:31.557 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:wVh/X6mUpo+vg8OQ22z4maKFJ4w= 2023-05-31T21:33:31.558 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 21:33:31 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:wVh/X6mUpo+vg8OQ22z4maKFJ4w=', 'Content-Length': '0'} 2023-05-31T21:33:31.560 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '2'), ('X-RGW-Bytes-Used', '12'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000006a6c3e89274be401-006477bd2b-2aea-a2'), ('Content-Length', '0'), ('Date', 'Wed, 31 May 2023 21:33:31 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:33:31.560 DEBUG:boto:path=/pklxkd-75/obj1 2023-05-31T21:33:31.561 DEBUG:boto:auth_path=/pklxkd-75/obj1 2023-05-31T21:33:31.561 DEBUG:boto:Method: HEAD 2023-05-31T21:33:31.561 DEBUG:boto:Path: /pklxkd-75/obj1 2023-05-31T21:33:31.562 DEBUG:boto:Data: 2023-05-31T21:33:31.562 DEBUG:boto:Headers: {} 2023-05-31T21:33:31.562 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T21:33:31.562 DEBUG:boto:Port: 8001 2023-05-31T21:33:31.563 DEBUG:boto:Params: {} 2023-05-31T21:33:31.563 DEBUG:boto:Token: None 2023-05-31T21:33:31.564 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 21:33:31 GMT /pklxkd-75/obj1 2023-05-31T21:33:31.564 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:ilCosBy5zL7BywSjM8Ggi/vj4S4= 2023-05-31T21:33:31.564 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 21:33:31 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:ilCosBy5zL7BywSjM8Ggi/vj4S4=', 'Content-Length': '0'} 2023-05-31T21:33:31.566 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 21:31:27 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000c4ba7b734b16e285-006477bd2b-2aea-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 21:33:31 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:33:31.567 DEBUG:boto:path=/pklxkd-75/obj1 2023-05-31T21:33:31.567 DEBUG:boto:auth_path=/pklxkd-75/obj1 2023-05-31T21:33:31.567 DEBUG:boto:Method: GET 2023-05-31T21:33:31.568 DEBUG:boto:Path: /pklxkd-75/obj1 2023-05-31T21:33:31.568 DEBUG:boto:Data: 2023-05-31T21:33:31.568 DEBUG:boto:Headers: {} 2023-05-31T21:33:31.569 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T21:33:31.569 DEBUG:boto:Port: 8001 2023-05-31T21:33:31.569 DEBUG:boto:Params: {} 2023-05-31T21:33:31.570 DEBUG:boto:Token: None 2023-05-31T21:33:31.570 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 21:33:31 GMT /pklxkd-75/obj1 2023-05-31T21:33:31.571 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:6nOxf4XqjKzsGYqFP3QLXewAdHY= 2023-05-31T21:33:31.571 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 21:33:31 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:6nOxf4XqjKzsGYqFP3QLXewAdHY=', 'Content-Length': '0'} 2023-05-31T21:33:31.573 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 21:31:27 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000077945532787b3aff-006477bd2b-2aea-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 21:33:31 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:33:31.616 DEBUG:boto:path=/pklxkd-75/obj2 2023-05-31T21:33:31.616 DEBUG:boto:auth_path=/pklxkd-75/obj2 2023-05-31T21:33:31.617 DEBUG:boto:Method: HEAD 2023-05-31T21:33:31.617 DEBUG:boto:Path: /pklxkd-75/obj2 2023-05-31T21:33:31.618 DEBUG:boto:Data: 2023-05-31T21:33:31.618 DEBUG:boto:Headers: {} 2023-05-31T21:33:31.618 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T21:33:31.618 DEBUG:boto:Port: 8001 2023-05-31T21:33:31.619 DEBUG:boto:Params: {} 2023-05-31T21:33:31.619 DEBUG:boto:Token: None 2023-05-31T21:33:31.620 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 21:33:31 GMT /pklxkd-75/obj2 2023-05-31T21:33:31.620 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:+4YKPyG7r3HM4pb2w66VEPTTOl4= 2023-05-31T21:33:31.620 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 21:33:31 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:+4YKPyG7r3HM4pb2w66VEPTTOl4=', 'Content-Length': '0'} 2023-05-31T21:33:31.622 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 21:31:27 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000c3416d056bb66b36-006477bd2b-2aea-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 21:33:31 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:33:31.623 DEBUG:boto:path=/pklxkd-75/obj2 2023-05-31T21:33:31.623 DEBUG:boto:auth_path=/pklxkd-75/obj2 2023-05-31T21:33:31.623 DEBUG:boto:Method: GET 2023-05-31T21:33:31.624 DEBUG:boto:Path: /pklxkd-75/obj2 2023-05-31T21:33:31.624 DEBUG:boto:Data: 2023-05-31T21:33:31.624 DEBUG:boto:Headers: {} 2023-05-31T21:33:31.625 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T21:33:31.625 DEBUG:boto:Port: 8001 2023-05-31T21:33:31.625 DEBUG:boto:Params: {} 2023-05-31T21:33:31.626 DEBUG:boto:Token: None 2023-05-31T21:33:31.626 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 21:33:31 GMT /pklxkd-75/obj2 2023-05-31T21:33:31.627 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:bxodPhaGdHfeOLsbEGccGASZ67c= 2023-05-31T21:33:31.627 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 21:33:31 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:bxodPhaGdHfeOLsbEGccGASZ67c=', 'Content-Length': '0'} 2023-05-31T21:33:31.629 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 21:31:27 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000154b67f93e01121d-006477bd2b-2aea-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 21:33:31 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:33:31.629 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'group', 'pipe', 'remove', '--group-id', 'sync-group', '--pipe-id', 'sync-pipe', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-05-31T21:33:31.630 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'group', 'pipe', 'remove', '--group-id', 'sync-group', '--pipe-id', 'sync-pipe', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-05-31T21:33:31.630 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync group pipe remove --group-id sync-group --pipe-id sync-pipe --cluster c1 --debug-rgw 0 --debug-ms 0 2023-05-31T21:33:31.681 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2023-05-31T21:33:31.681 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2023-05-31T21:33:31.749 INFO:teuthology.orchestra.run.smithi099.stdout:{"groups":[{"id":"sync-group","data_flow":{"symmetrical":[{"id":"sync-flow1","zones":["a1","a2"]}]},"pipes":[{"id":"sync-pipe-pklxkd-74","source":{"bucket":"pklxkd-74","zones":["a1"]},"dest":{"bucket":"pklxkd-74","zones":["a2"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":""}},{"id":"sync-pipe-pklxkd-75","source":{"bucket":"pklxkd-74","zones":["a1"]},"dest":{"bucket":"pklxkd-75","zones":["a2"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":""}}],"status":"enabled"}]} 2023-05-31T21:33:31.750 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'group', 'pipe', 'create', '--group-id', 'sync-group', '--pipe-id', 'sync-pipe', '--source-zones=*', '--dest-zones=*', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-05-31T21:33:31.750 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'group', 'pipe', 'create', '--group-id', 'sync-group', '--pipe-id', 'sync-pipe', '--source-zones=*', '--dest-zones=*', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-05-31T21:33:31.751 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync group pipe create --group-id sync-group --pipe-id sync-pipe '--source-zones=*' '--dest-zones=*' --cluster c1 --debug-rgw 0 --debug-ms 0 2023-05-31T21:33:31.802 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2023-05-31T21:33:31.803 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2023-05-31T21:33:31.861 INFO:teuthology.orchestra.run.smithi099.stderr:WARNING: cannot find source zone id for name=* 2023-05-31T21:33:31.861 INFO:teuthology.orchestra.run.smithi099.stderr:WARNING: cannot find source zone id for name=* 2023-05-31T21:33:31.888 INFO:teuthology.orchestra.run.smithi099.stdout:{"groups":[{"id":"sync-group","data_flow":{"symmetrical":[{"id":"sync-flow1","zones":["a1","a2"]}]},"pipes":[{"id":"sync-pipe-pklxkd-74","source":{"bucket":"pklxkd-74","zones":["a1"]},"dest":{"bucket":"pklxkd-74","zones":["a2"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":""}},{"id":"sync-pipe-pklxkd-75","source":{"bucket":"pklxkd-74","zones":["a1"]},"dest":{"bucket":"pklxkd-75","zones":["a2"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":""}},{"id":"sync-pipe","source":{"bucket":"*","zones":["*"]},"dest":{"bucket":"*","zones":["*"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":""}}],"status":"enabled"}]} 2023-05-31T21:33:31.888 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'group', 'modify', '--group-id', 'sync-group', '--status', 'allowed', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-05-31T21:33:31.889 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'group', 'modify', '--group-id', 'sync-group', '--status', 'allowed', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-05-31T21:33:31.889 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync group modify --group-id sync-group --status allowed --cluster c1 --debug-rgw 0 --debug-ms 0 2023-05-31T21:33:31.937 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2023-05-31T21:33:31.938 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2023-05-31T21:33:31.999 INFO:teuthology.orchestra.run.smithi099.stdout:{"groups":[{"id":"sync-group","data_flow":{"symmetrical":[{"id":"sync-flow1","zones":["a1","a2"]}]},"pipes":[{"id":"sync-pipe-pklxkd-74","source":{"bucket":"pklxkd-74","zones":["a1"]},"dest":{"bucket":"pklxkd-74","zones":["a2"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":""}},{"id":"sync-pipe-pklxkd-75","source":{"bucket":"pklxkd-74","zones":["a1"]},"dest":{"bucket":"pklxkd-75","zones":["a2"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":""}},{"id":"sync-pipe","source":{"bucket":"*","zones":["*"]},"dest":{"bucket":"*","zones":["*"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":""}}],"status":"allowed"}]} 2023-05-31T21:33:31.999 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['period', 'update', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-05-31T21:33:32.000 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'period', 'update', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-05-31T21:33:32.000 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 period update --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --commit --cluster c1 --debug-rgw 0 --debug-ms 0 2023-05-31T21:33:32.048 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2023-05-31T21:33:32.049 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2023-05-31T21:33:32.129 INFO:teuthology.orchestra.run.smithi099.stdout:{ 2023-05-31T21:33:32.129 INFO:teuthology.orchestra.run.smithi099.stdout: "id": "2e6123d2-d695-4350-9a01-61a455bc782b", 2023-05-31T21:33:32.129 INFO:teuthology.orchestra.run.smithi099.stdout: "epoch": 22, 2023-05-31T21:33:32.129 INFO:teuthology.orchestra.run.smithi099.stdout: "predecessor_uuid": "629b9ced-80af-41ed-8a6c-f2cfc334cbb4", 2023-05-31T21:33:32.129 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_status": [], 2023-05-31T21:33:32.130 INFO:teuthology.orchestra.run.smithi099.stdout: "period_map": { 2023-05-31T21:33:32.130 INFO:teuthology.orchestra.run.smithi099.stdout: "id": "2e6123d2-d695-4350-9a01-61a455bc782b", 2023-05-31T21:33:32.130 INFO:teuthology.orchestra.run.smithi099.stdout: "zonegroups": [ 2023-05-31T21:33:32.130 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T21:33:32.130 INFO:teuthology.orchestra.run.smithi099.stdout: "id": "a", 2023-05-31T21:33:32.130 INFO:teuthology.orchestra.run.smithi099.stdout: "name": "a", 2023-05-31T21:33:32.130 INFO:teuthology.orchestra.run.smithi099.stdout: "api_name": "", 2023-05-31T21:33:32.131 INFO:teuthology.orchestra.run.smithi099.stdout: "is_master": true, 2023-05-31T21:33:32.131 INFO:teuthology.orchestra.run.smithi099.stdout: "endpoints": [ 2023-05-31T21:33:32.131 INFO:teuthology.orchestra.run.smithi099.stdout: "http://smithi099.front.sepia.ceph.com:8000" 2023-05-31T21:33:32.131 INFO:teuthology.orchestra.run.smithi099.stdout: ], 2023-05-31T21:33:32.131 INFO:teuthology.orchestra.run.smithi099.stdout: "hostnames": [], 2023-05-31T21:33:32.131 INFO:teuthology.orchestra.run.smithi099.stdout: "hostnames_s3website": [], 2023-05-31T21:33:32.131 INFO:teuthology.orchestra.run.smithi099.stdout: "master_zone": "3d163b1e-09c7-47e2-ae87-add4aace3f38", 2023-05-31T21:33:32.132 INFO:teuthology.orchestra.run.smithi099.stdout: "zones": [ 2023-05-31T21:33:32.132 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T21:33:32.132 INFO:teuthology.orchestra.run.smithi099.stdout: "id": "3d163b1e-09c7-47e2-ae87-add4aace3f38", 2023-05-31T21:33:32.132 INFO:teuthology.orchestra.run.smithi099.stdout: "name": "a1", 2023-05-31T21:33:32.132 INFO:teuthology.orchestra.run.smithi099.stdout: "endpoints": [ 2023-05-31T21:33:32.132 INFO:teuthology.orchestra.run.smithi099.stdout: "http://smithi099.front.sepia.ceph.com:8000" 2023-05-31T21:33:32.132 INFO:teuthology.orchestra.run.smithi099.stdout: ], 2023-05-31T21:33:32.133 INFO:teuthology.orchestra.run.smithi099.stdout: "log_meta": false, 2023-05-31T21:33:32.133 INFO:teuthology.orchestra.run.smithi099.stdout: "log_data": true, 2023-05-31T21:33:32.133 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket_index_max_shards": 11, 2023-05-31T21:33:32.133 INFO:teuthology.orchestra.run.smithi099.stdout: "read_only": false, 2023-05-31T21:33:32.133 INFO:teuthology.orchestra.run.smithi099.stdout: "tier_type": "", 2023-05-31T21:33:32.133 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_from_all": true, 2023-05-31T21:33:32.133 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_from": [], 2023-05-31T21:33:32.134 INFO:teuthology.orchestra.run.smithi099.stdout: "redirect_zone": "", 2023-05-31T21:33:32.134 INFO:teuthology.orchestra.run.smithi099.stdout: "supported_features": [ 2023-05-31T21:33:32.134 INFO:teuthology.orchestra.run.smithi099.stdout: "resharding" 2023-05-31T21:33:32.134 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2023-05-31T21:33:32.134 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:33:32.134 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T21:33:32.134 INFO:teuthology.orchestra.run.smithi099.stdout: "id": "5d8acec7-7e57-4e56-8b32-e3d6b5036001", 2023-05-31T21:33:32.135 INFO:teuthology.orchestra.run.smithi099.stdout: "name": "a2", 2023-05-31T21:33:32.135 INFO:teuthology.orchestra.run.smithi099.stdout: "endpoints": [ 2023-05-31T21:33:32.135 INFO:teuthology.orchestra.run.smithi099.stdout: "http://smithi099.front.sepia.ceph.com:8001" 2023-05-31T21:33:32.135 INFO:teuthology.orchestra.run.smithi099.stdout: ], 2023-05-31T21:33:32.135 INFO:teuthology.orchestra.run.smithi099.stdout: "log_meta": false, 2023-05-31T21:33:32.135 INFO:teuthology.orchestra.run.smithi099.stdout: "log_data": true, 2023-05-31T21:33:32.135 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket_index_max_shards": 11, 2023-05-31T21:33:32.136 INFO:teuthology.orchestra.run.smithi099.stdout: "read_only": false, 2023-05-31T21:33:32.136 INFO:teuthology.orchestra.run.smithi099.stdout: "tier_type": "", 2023-05-31T21:33:32.136 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_from_all": true, 2023-05-31T21:33:32.136 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_from": [], 2023-05-31T21:33:32.136 INFO:teuthology.orchestra.run.smithi099.stdout: "redirect_zone": "", 2023-05-31T21:33:32.136 INFO:teuthology.orchestra.run.smithi099.stdout: "supported_features": [ 2023-05-31T21:33:32.136 INFO:teuthology.orchestra.run.smithi099.stdout: "resharding" 2023-05-31T21:33:32.137 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2023-05-31T21:33:32.137 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T21:33:32.137 INFO:teuthology.orchestra.run.smithi099.stdout: ], 2023-05-31T21:33:32.137 INFO:teuthology.orchestra.run.smithi099.stdout: "placement_targets": [ 2023-05-31T21:33:32.137 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T21:33:32.137 INFO:teuthology.orchestra.run.smithi099.stdout: "name": "default-placement", 2023-05-31T21:33:32.137 INFO:teuthology.orchestra.run.smithi099.stdout: "tags": [], 2023-05-31T21:33:32.138 INFO:teuthology.orchestra.run.smithi099.stdout: "storage_classes": [ 2023-05-31T21:33:32.138 INFO:teuthology.orchestra.run.smithi099.stdout: "STANDARD" 2023-05-31T21:33:32.138 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2023-05-31T21:33:32.138 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T21:33:32.138 INFO:teuthology.orchestra.run.smithi099.stdout: ], 2023-05-31T21:33:32.138 INFO:teuthology.orchestra.run.smithi099.stdout: "default_placement": "default-placement", 2023-05-31T21:33:32.138 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_id": "aaa33de6-515b-4cac-8f1b-3a1a30452117", 2023-05-31T21:33:32.139 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_policy": { 2023-05-31T21:33:32.139 INFO:teuthology.orchestra.run.smithi099.stdout: "groups": [ 2023-05-31T21:33:32.139 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T21:33:32.139 INFO:teuthology.orchestra.run.smithi099.stdout: "id": "sync-group", 2023-05-31T21:33:32.139 INFO:teuthology.orchestra.run.smithi099.stdout: "data_flow": { 2023-05-31T21:33:32.139 INFO:teuthology.orchestra.run.smithi099.stdout: "symmetrical": [ 2023-05-31T21:33:32.139 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T21:33:32.140 INFO:teuthology.orchestra.run.smithi099.stdout: "id": "sync-flow1", 2023-05-31T21:33:32.140 INFO:teuthology.orchestra.run.smithi099.stdout: "zones": [ 2023-05-31T21:33:32.140 INFO:teuthology.orchestra.run.smithi099.stdout: "3d163b1e-09c7-47e2-ae87-add4aace3f38", 2023-05-31T21:33:32.140 INFO:teuthology.orchestra.run.smithi099.stdout: "5d8acec7-7e57-4e56-8b32-e3d6b5036001" 2023-05-31T21:33:32.140 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2023-05-31T21:33:32.140 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T21:33:32.140 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2023-05-31T21:33:32.141 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:33:32.141 INFO:teuthology.orchestra.run.smithi099.stdout: "pipes": [ 2023-05-31T21:33:32.141 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T21:33:32.141 INFO:teuthology.orchestra.run.smithi099.stdout: "id": "sync-pipe-pklxkd-74", 2023-05-31T21:33:32.141 INFO:teuthology.orchestra.run.smithi099.stdout: "source": { 2023-05-31T21:33:32.141 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket": "pklxkd-74", 2023-05-31T21:33:32.141 INFO:teuthology.orchestra.run.smithi099.stdout: "zones": [ 2023-05-31T21:33:32.142 INFO:teuthology.orchestra.run.smithi099.stdout: "3d163b1e-09c7-47e2-ae87-add4aace3f38" 2023-05-31T21:33:32.142 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2023-05-31T21:33:32.142 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:33:32.142 INFO:teuthology.orchestra.run.smithi099.stdout: "dest": { 2023-05-31T21:33:32.142 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket": "pklxkd-74", 2023-05-31T21:33:32.142 INFO:teuthology.orchestra.run.smithi099.stdout: "zones": [ 2023-05-31T21:33:32.142 INFO:teuthology.orchestra.run.smithi099.stdout: "5d8acec7-7e57-4e56-8b32-e3d6b5036001" 2023-05-31T21:33:32.143 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2023-05-31T21:33:32.143 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:33:32.143 INFO:teuthology.orchestra.run.smithi099.stdout: "params": { 2023-05-31T21:33:32.143 INFO:teuthology.orchestra.run.smithi099.stdout: "source": { 2023-05-31T21:33:32.143 INFO:teuthology.orchestra.run.smithi099.stdout: "filter": { 2023-05-31T21:33:32.143 INFO:teuthology.orchestra.run.smithi099.stdout: "tags": [] 2023-05-31T21:33:32.143 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T21:33:32.144 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:33:32.144 INFO:teuthology.orchestra.run.smithi099.stdout: "dest": {}, 2023-05-31T21:33:32.144 INFO:teuthology.orchestra.run.smithi099.stdout: "priority": 0, 2023-05-31T21:33:32.144 INFO:teuthology.orchestra.run.smithi099.stdout: "mode": "system", 2023-05-31T21:33:32.144 INFO:teuthology.orchestra.run.smithi099.stdout: "user": "" 2023-05-31T21:33:32.144 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T21:33:32.145 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:33:32.145 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T21:33:32.145 INFO:teuthology.orchestra.run.smithi099.stdout: "id": "sync-pipe-pklxkd-75", 2023-05-31T21:33:32.145 INFO:teuthology.orchestra.run.smithi099.stdout: "source": { 2023-05-31T21:33:32.145 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket": "pklxkd-74", 2023-05-31T21:33:32.145 INFO:teuthology.orchestra.run.smithi099.stdout: "zones": [ 2023-05-31T21:33:32.145 INFO:teuthology.orchestra.run.smithi099.stdout: "3d163b1e-09c7-47e2-ae87-add4aace3f38" 2023-05-31T21:33:32.146 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2023-05-31T21:33:32.146 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:33:32.146 INFO:teuthology.orchestra.run.smithi099.stdout: "dest": { 2023-05-31T21:33:32.146 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket": "pklxkd-75", 2023-05-31T21:33:32.146 INFO:teuthology.orchestra.run.smithi099.stdout: "zones": [ 2023-05-31T21:33:32.146 INFO:teuthology.orchestra.run.smithi099.stdout: "5d8acec7-7e57-4e56-8b32-e3d6b5036001" 2023-05-31T21:33:32.146 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2023-05-31T21:33:32.146 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:33:32.147 INFO:teuthology.orchestra.run.smithi099.stdout: "params": { 2023-05-31T21:33:32.147 INFO:teuthology.orchestra.run.smithi099.stdout: "source": { 2023-05-31T21:33:32.147 INFO:teuthology.orchestra.run.smithi099.stdout: "filter": { 2023-05-31T21:33:32.147 INFO:teuthology.orchestra.run.smithi099.stdout: "tags": [] 2023-05-31T21:33:32.147 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T21:33:32.147 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:33:32.147 INFO:teuthology.orchestra.run.smithi099.stdout: "dest": {}, 2023-05-31T21:33:32.148 INFO:teuthology.orchestra.run.smithi099.stdout: "priority": 0, 2023-05-31T21:33:32.148 INFO:teuthology.orchestra.run.smithi099.stdout: "mode": "system", 2023-05-31T21:33:32.148 INFO:teuthology.orchestra.run.smithi099.stdout: "user": "" 2023-05-31T21:33:32.148 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T21:33:32.148 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:33:32.148 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T21:33:32.148 INFO:teuthology.orchestra.run.smithi099.stdout: "id": "sync-pipe", 2023-05-31T21:33:32.149 INFO:teuthology.orchestra.run.smithi099.stdout: "source": { 2023-05-31T21:33:32.149 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket": "*", 2023-05-31T21:33:32.149 INFO:teuthology.orchestra.run.smithi099.stdout: "zones": [ 2023-05-31T21:33:32.149 INFO:teuthology.orchestra.run.smithi099.stdout: "*" 2023-05-31T21:33:32.149 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2023-05-31T21:33:32.149 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:33:32.149 INFO:teuthology.orchestra.run.smithi099.stdout: "dest": { 2023-05-31T21:33:32.150 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket": "*", 2023-05-31T21:33:32.150 INFO:teuthology.orchestra.run.smithi099.stdout: "zones": [ 2023-05-31T21:33:32.150 INFO:teuthology.orchestra.run.smithi099.stdout: "*" 2023-05-31T21:33:32.150 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2023-05-31T21:33:32.150 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:33:32.150 INFO:teuthology.orchestra.run.smithi099.stdout: "params": { 2023-05-31T21:33:32.150 INFO:teuthology.orchestra.run.smithi099.stdout: "source": { 2023-05-31T21:33:32.151 INFO:teuthology.orchestra.run.smithi099.stdout: "filter": { 2023-05-31T21:33:32.151 INFO:teuthology.orchestra.run.smithi099.stdout: "tags": [] 2023-05-31T21:33:32.151 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T21:33:32.151 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:33:32.151 INFO:teuthology.orchestra.run.smithi099.stdout: "dest": {}, 2023-05-31T21:33:32.151 INFO:teuthology.orchestra.run.smithi099.stdout: "priority": 0, 2023-05-31T21:33:32.151 INFO:teuthology.orchestra.run.smithi099.stdout: "mode": "system", 2023-05-31T21:33:32.152 INFO:teuthology.orchestra.run.smithi099.stdout: "user": "" 2023-05-31T21:33:32.152 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T21:33:32.152 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T21:33:32.152 INFO:teuthology.orchestra.run.smithi099.stdout: ], 2023-05-31T21:33:32.152 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "allowed" 2023-05-31T21:33:32.152 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T21:33:32.153 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2023-05-31T21:33:32.153 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:33:32.153 INFO:teuthology.orchestra.run.smithi099.stdout: "enabled_features": [ 2023-05-31T21:33:32.153 INFO:teuthology.orchestra.run.smithi099.stdout: "resharding" 2023-05-31T21:33:32.153 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2023-05-31T21:33:32.153 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:33:32.153 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T21:33:32.154 INFO:teuthology.orchestra.run.smithi099.stdout: "id": "b", 2023-05-31T21:33:32.154 INFO:teuthology.orchestra.run.smithi099.stdout: "name": "b", 2023-05-31T21:33:32.154 INFO:teuthology.orchestra.run.smithi099.stdout: "api_name": "", 2023-05-31T21:33:32.154 INFO:teuthology.orchestra.run.smithi099.stdout: "is_master": false, 2023-05-31T21:33:32.154 INFO:teuthology.orchestra.run.smithi099.stdout: "endpoints": [ 2023-05-31T21:33:32.154 INFO:teuthology.orchestra.run.smithi099.stdout: "http://smithi112.front.sepia.ceph.com:8000" 2023-05-31T21:33:32.154 INFO:teuthology.orchestra.run.smithi099.stdout: ], 2023-05-31T21:33:32.155 INFO:teuthology.orchestra.run.smithi099.stdout: "hostnames": [], 2023-05-31T21:33:32.155 INFO:teuthology.orchestra.run.smithi099.stdout: "hostnames_s3website": [], 2023-05-31T21:33:32.155 INFO:teuthology.orchestra.run.smithi099.stdout: "master_zone": "f85d7236-2f27-4988-af26-9c66ab280b1b", 2023-05-31T21:33:32.155 INFO:teuthology.orchestra.run.smithi099.stdout: "zones": [ 2023-05-31T21:33:32.155 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T21:33:32.155 INFO:teuthology.orchestra.run.smithi099.stdout: "id": "a6a3aab5-bc52-44a4-8543-5c5ea6056acd", 2023-05-31T21:33:32.155 INFO:teuthology.orchestra.run.smithi099.stdout: "name": "b2", 2023-05-31T21:33:32.156 INFO:teuthology.orchestra.run.smithi099.stdout: "endpoints": [ 2023-05-31T21:33:32.157 INFO:teuthology.orchestra.run.smithi099.stdout: "http://smithi112.front.sepia.ceph.com:8001" 2023-05-31T21:33:32.158 INFO:teuthology.orchestra.run.smithi099.stdout: ], 2023-05-31T21:33:32.158 INFO:teuthology.orchestra.run.smithi099.stdout: "log_meta": false, 2023-05-31T21:33:32.158 INFO:teuthology.orchestra.run.smithi099.stdout: "log_data": true, 2023-05-31T21:33:32.158 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket_index_max_shards": 11, 2023-05-31T21:33:32.158 INFO:teuthology.orchestra.run.smithi099.stdout: "read_only": false, 2023-05-31T21:33:32.158 INFO:teuthology.orchestra.run.smithi099.stdout: "tier_type": "", 2023-05-31T21:33:32.158 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_from_all": true, 2023-05-31T21:33:32.159 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_from": [], 2023-05-31T21:33:32.159 INFO:teuthology.orchestra.run.smithi099.stdout: "redirect_zone": "", 2023-05-31T21:33:32.159 INFO:teuthology.orchestra.run.smithi099.stdout: "supported_features": [ 2023-05-31T21:33:32.159 INFO:teuthology.orchestra.run.smithi099.stdout: "resharding" 2023-05-31T21:33:32.159 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2023-05-31T21:33:32.159 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:33:32.159 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T21:33:32.160 INFO:teuthology.orchestra.run.smithi099.stdout: "id": "f85d7236-2f27-4988-af26-9c66ab280b1b", 2023-05-31T21:33:32.160 INFO:teuthology.orchestra.run.smithi099.stdout: "name": "b1", 2023-05-31T21:33:32.160 INFO:teuthology.orchestra.run.smithi099.stdout: "endpoints": [ 2023-05-31T21:33:32.160 INFO:teuthology.orchestra.run.smithi099.stdout: "http://smithi112.front.sepia.ceph.com:8000" 2023-05-31T21:33:32.160 INFO:teuthology.orchestra.run.smithi099.stdout: ], 2023-05-31T21:33:32.160 INFO:teuthology.orchestra.run.smithi099.stdout: "log_meta": false, 2023-05-31T21:33:32.160 INFO:teuthology.orchestra.run.smithi099.stdout: "log_data": true, 2023-05-31T21:33:32.161 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket_index_max_shards": 11, 2023-05-31T21:33:32.161 INFO:teuthology.orchestra.run.smithi099.stdout: "read_only": false, 2023-05-31T21:33:32.161 INFO:teuthology.orchestra.run.smithi099.stdout: "tier_type": "", 2023-05-31T21:33:32.161 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_from_all": true, 2023-05-31T21:33:32.161 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_from": [], 2023-05-31T21:33:32.161 INFO:teuthology.orchestra.run.smithi099.stdout: "redirect_zone": "", 2023-05-31T21:33:32.162 INFO:teuthology.orchestra.run.smithi099.stdout: "supported_features": [ 2023-05-31T21:33:32.162 INFO:teuthology.orchestra.run.smithi099.stdout: "resharding" 2023-05-31T21:33:32.162 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2023-05-31T21:33:32.162 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T21:33:32.162 INFO:teuthology.orchestra.run.smithi099.stdout: ], 2023-05-31T21:33:32.162 INFO:teuthology.orchestra.run.smithi099.stdout: "placement_targets": [ 2023-05-31T21:33:32.162 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T21:33:32.163 INFO:teuthology.orchestra.run.smithi099.stdout: "name": "default-placement", 2023-05-31T21:33:32.163 INFO:teuthology.orchestra.run.smithi099.stdout: "tags": [], 2023-05-31T21:33:32.163 INFO:teuthology.orchestra.run.smithi099.stdout: "storage_classes": [ 2023-05-31T21:33:32.163 INFO:teuthology.orchestra.run.smithi099.stdout: "STANDARD" 2023-05-31T21:33:32.163 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2023-05-31T21:33:32.164 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T21:33:32.164 INFO:teuthology.orchestra.run.smithi099.stdout: ], 2023-05-31T21:33:32.164 INFO:teuthology.orchestra.run.smithi099.stdout: "default_placement": "default-placement", 2023-05-31T21:33:32.164 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_id": "aaa33de6-515b-4cac-8f1b-3a1a30452117", 2023-05-31T21:33:32.164 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_policy": { 2023-05-31T21:33:32.164 INFO:teuthology.orchestra.run.smithi099.stdout: "groups": [] 2023-05-31T21:33:32.165 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:33:32.165 INFO:teuthology.orchestra.run.smithi099.stdout: "enabled_features": [ 2023-05-31T21:33:32.165 INFO:teuthology.orchestra.run.smithi099.stdout: "resharding" 2023-05-31T21:33:32.165 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2023-05-31T21:33:32.165 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T21:33:32.165 INFO:teuthology.orchestra.run.smithi099.stdout: ], 2023-05-31T21:33:32.165 INFO:teuthology.orchestra.run.smithi099.stdout: "short_zone_ids": [ 2023-05-31T21:33:32.166 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T21:33:32.166 INFO:teuthology.orchestra.run.smithi099.stdout: "key": "3d163b1e-09c7-47e2-ae87-add4aace3f38", 2023-05-31T21:33:32.166 INFO:teuthology.orchestra.run.smithi099.stdout: "val": 3101833771 2023-05-31T21:33:32.166 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:33:32.166 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T21:33:32.166 INFO:teuthology.orchestra.run.smithi099.stdout: "key": "5d8acec7-7e57-4e56-8b32-e3d6b5036001", 2023-05-31T21:33:32.166 INFO:teuthology.orchestra.run.smithi099.stdout: "val": 4255530623 2023-05-31T21:33:32.167 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:33:32.167 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T21:33:32.167 INFO:teuthology.orchestra.run.smithi099.stdout: "key": "a6a3aab5-bc52-44a4-8543-5c5ea6056acd", 2023-05-31T21:33:32.167 INFO:teuthology.orchestra.run.smithi099.stdout: "val": 3708073710 2023-05-31T21:33:32.167 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:33:32.167 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T21:33:32.167 INFO:teuthology.orchestra.run.smithi099.stdout: "key": "f85d7236-2f27-4988-af26-9c66ab280b1b", 2023-05-31T21:33:32.168 INFO:teuthology.orchestra.run.smithi099.stdout: "val": 233395200 2023-05-31T21:33:32.168 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T21:33:32.168 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2023-05-31T21:33:32.168 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:33:32.168 INFO:teuthology.orchestra.run.smithi099.stdout: "master_zonegroup": "a", 2023-05-31T21:33:32.168 INFO:teuthology.orchestra.run.smithi099.stdout: "master_zone": "3d163b1e-09c7-47e2-ae87-add4aace3f38", 2023-05-31T21:33:32.169 INFO:teuthology.orchestra.run.smithi099.stdout: "period_config": { 2023-05-31T21:33:32.169 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket_quota": { 2023-05-31T21:33:32.169 INFO:teuthology.orchestra.run.smithi099.stdout: "enabled": false, 2023-05-31T21:33:32.169 INFO:teuthology.orchestra.run.smithi099.stdout: "check_on_raw": false, 2023-05-31T21:33:32.169 INFO:teuthology.orchestra.run.smithi099.stdout: "max_size": -1, 2023-05-31T21:33:32.169 INFO:teuthology.orchestra.run.smithi099.stdout: "max_size_kb": 0, 2023-05-31T21:33:32.169 INFO:teuthology.orchestra.run.smithi099.stdout: "max_objects": -1 2023-05-31T21:33:32.170 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:33:32.170 INFO:teuthology.orchestra.run.smithi099.stdout: "user_quota": { 2023-05-31T21:33:32.170 INFO:teuthology.orchestra.run.smithi099.stdout: "enabled": false, 2023-05-31T21:33:32.170 INFO:teuthology.orchestra.run.smithi099.stdout: "check_on_raw": false, 2023-05-31T21:33:32.170 INFO:teuthology.orchestra.run.smithi099.stdout: "max_size": -1, 2023-05-31T21:33:32.170 INFO:teuthology.orchestra.run.smithi099.stdout: "max_size_kb": 0, 2023-05-31T21:33:32.170 INFO:teuthology.orchestra.run.smithi099.stdout: "max_objects": -1 2023-05-31T21:33:32.171 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:33:32.171 INFO:teuthology.orchestra.run.smithi099.stdout: "user_ratelimit": { 2023-05-31T21:33:32.171 INFO:teuthology.orchestra.run.smithi099.stdout: "max_read_ops": 0, 2023-05-31T21:33:32.171 INFO:teuthology.orchestra.run.smithi099.stdout: "max_write_ops": 0, 2023-05-31T21:33:32.171 INFO:teuthology.orchestra.run.smithi099.stdout: "max_read_bytes": 0, 2023-05-31T21:33:32.171 INFO:teuthology.orchestra.run.smithi099.stdout: "max_write_bytes": 0, 2023-05-31T21:33:32.171 INFO:teuthology.orchestra.run.smithi099.stdout: "enabled": false 2023-05-31T21:33:32.171 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:33:32.172 INFO:teuthology.orchestra.run.smithi099.stdout: "bucket_ratelimit": { 2023-05-31T21:33:32.172 INFO:teuthology.orchestra.run.smithi099.stdout: "max_read_ops": 0, 2023-05-31T21:33:32.172 INFO:teuthology.orchestra.run.smithi099.stdout: "max_write_ops": 0, 2023-05-31T21:33:32.172 INFO:teuthology.orchestra.run.smithi099.stdout: "max_read_bytes": 0, 2023-05-31T21:33:32.172 INFO:teuthology.orchestra.run.smithi099.stdout: "max_write_bytes": 0, 2023-05-31T21:33:32.172 INFO:teuthology.orchestra.run.smithi099.stdout: "enabled": false 2023-05-31T21:33:32.173 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:33:32.173 INFO:teuthology.orchestra.run.smithi099.stdout: "anonymous_ratelimit": { 2023-05-31T21:33:32.173 INFO:teuthology.orchestra.run.smithi099.stdout: "max_read_ops": 0, 2023-05-31T21:33:32.173 INFO:teuthology.orchestra.run.smithi099.stdout: "max_write_ops": 0, 2023-05-31T21:33:32.173 INFO:teuthology.orchestra.run.smithi099.stdout: "max_read_bytes": 0, 2023-05-31T21:33:32.173 INFO:teuthology.orchestra.run.smithi099.stdout: "max_write_bytes": 0, 2023-05-31T21:33:32.173 INFO:teuthology.orchestra.run.smithi099.stdout: "enabled": false 2023-05-31T21:33:32.173 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T21:33:32.174 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:33:32.174 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_id": "aaa33de6-515b-4cac-8f1b-3a1a30452117", 2023-05-31T21:33:32.174 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_name": "test-realm", 2023-05-31T21:33:32.174 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2023-05-31T21:33:32.174 INFO:teuthology.orchestra.run.smithi099.stdout:} 2023-05-31T21:33:32.175 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'policy', 'get', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-05-31T21:33:32.176 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'policy', 'get', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-05-31T21:33:32.176 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync policy get --cluster c1 --debug-rgw 0 --debug-ms 0 2023-05-31T21:33:32.224 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2023-05-31T21:33:32.224 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2023-05-31T21:33:32.320 INFO:teuthology.orchestra.run.smithi099.stdout:{"groups":[{"id":"sync-group","data_flow":{"symmetrical":[{"id":"sync-flow1","zones":["a1","a2"]}]},"pipes":[{"id":"sync-pipe-pklxkd-74","source":{"bucket":"pklxkd-74","zones":["a1"]},"dest":{"bucket":"pklxkd-74","zones":["a2"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":""}},{"id":"sync-pipe-pklxkd-75","source":{"bucket":"pklxkd-74","zones":["a1"]},"dest":{"bucket":"pklxkd-75","zones":["a2"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":""}},{"id":"sync-pipe","source":{"bucket":"*","zones":["*"]},"dest":{"bucket":"*","zones":["*"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":""}}],"status":"allowed"}]} 2023-05-31T21:33:32.320 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'group', 'create', '--group-id', 'sync-bucket', '--status', 'allowed', '--bucket', 'pklxkd-74', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-05-31T21:33:32.321 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'group', 'create', '--group-id', 'sync-bucket', '--status', 'allowed', '--bucket', 'pklxkd-74', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-05-31T21:33:32.321 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync group create --group-id sync-bucket --status allowed --bucket pklxkd-74 --cluster c1 --debug-rgw 0 --debug-ms 0 2023-05-31T21:33:32.380 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2023-05-31T21:33:32.380 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2023-05-31T21:33:32.486 INFO:teuthology.orchestra.run.smithi099.stdout:{"groups":[{"id":"sync-bucket","data_flow":{},"pipes":[],"status":"allowed"}]} 2023-05-31T21:33:32.486 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'group', 'flow', 'create', '--group-id', 'sync-bucket', '--flow-id', 'sync-flowA', '--flow-type', 'symmetrical', '--zones=a1,a2', '--bucket', 'pklxkd-74', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-05-31T21:33:32.487 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'group', 'flow', 'create', '--group-id', 'sync-bucket', '--flow-id', 'sync-flowA', '--flow-type', 'symmetrical', '--zones=a1,a2', '--bucket', 'pklxkd-74', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-05-31T21:33:32.487 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync group flow create --group-id sync-bucket --flow-id sync-flowA --flow-type symmetrical --zones=a1,a2 --bucket pklxkd-74 --cluster c1 --debug-rgw 0 --debug-ms 0 2023-05-31T21:33:32.548 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2023-05-31T21:33:32.549 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2023-05-31T21:33:32.695 INFO:teuthology.orchestra.run.smithi099.stdout:{"groups":[{"id":"sync-bucket","data_flow":{"symmetrical":[{"id":"sync-flowA","zones":["a1","a2"]}]},"pipes":[],"status":"allowed"}]} 2023-05-31T21:33:32.696 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'group', 'pipe', 'create', '--group-id', 'sync-bucket', '--pipe-id', 'sync-pipe-pklxkd-74', '--source-zones=a1', '--dest-zones=a2', '--bucket=pklxkd-74', '--source-bucket=*', '--dest-bucket=pklxkd-74', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-05-31T21:33:32.696 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'group', 'pipe', 'create', '--group-id', 'sync-bucket', '--pipe-id', 'sync-pipe-pklxkd-74', '--source-zones=a1', '--dest-zones=a2', '--bucket=pklxkd-74', '--source-bucket=*', '--dest-bucket=pklxkd-74', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-05-31T21:33:32.697 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync group pipe create --group-id sync-bucket --pipe-id sync-pipe-pklxkd-74 --source-zones=a1 --dest-zones=a2 --bucket=pklxkd-74 '--source-bucket=*' --dest-bucket=pklxkd-74 --cluster c1 --debug-rgw 0 --debug-ms 0 2023-05-31T21:33:32.760 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2023-05-31T21:33:32.760 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2023-05-31T21:33:32.881 INFO:teuthology.orchestra.run.smithi099.stdout:{"groups":[{"id":"sync-bucket","data_flow":{"symmetrical":[{"id":"sync-flowA","zones":["a1","a2"]}]},"pipes":[{"id":"sync-pipe-pklxkd-74","source":{"bucket":"*","zones":["a1"]},"dest":{"bucket":"pklxkd-74","zones":["a2"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":"rgw-multisite-test-user"}}],"status":"allowed"}]} 2023-05-31T21:33:32.882 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'group', 'pipe', 'create', '--group-id', 'sync-bucket', '--pipe-id', 'sync-pipe-pklxkd-75', '--source-zones=a1', '--dest-zones=a2', '--bucket=pklxkd-74', '--source-bucket=*', '--dest-bucket=pklxkd-75', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-05-31T21:33:32.882 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'group', 'pipe', 'create', '--group-id', 'sync-bucket', '--pipe-id', 'sync-pipe-pklxkd-75', '--source-zones=a1', '--dest-zones=a2', '--bucket=pklxkd-74', '--source-bucket=*', '--dest-bucket=pklxkd-75', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-05-31T21:33:32.883 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync group pipe create --group-id sync-bucket --pipe-id sync-pipe-pklxkd-75 --source-zones=a1 --dest-zones=a2 --bucket=pklxkd-74 '--source-bucket=*' --dest-bucket=pklxkd-75 --cluster c1 --debug-rgw 0 --debug-ms 0 2023-05-31T21:33:32.943 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2023-05-31T21:33:32.943 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2023-05-31T21:33:33.070 INFO:teuthology.orchestra.run.smithi099.stdout:{"groups":[{"id":"sync-bucket","data_flow":{"symmetrical":[{"id":"sync-flowA","zones":["a1","a2"]}]},"pipes":[{"id":"sync-pipe-pklxkd-74","source":{"bucket":"*","zones":["a1"]},"dest":{"bucket":"pklxkd-74","zones":["a2"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":"rgw-multisite-test-user"}},{"id":"sync-pipe-pklxkd-75","source":{"bucket":"*","zones":["a1"]},"dest":{"bucket":"pklxkd-75","zones":["a2"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":"rgw-multisite-test-user"}}],"status":"allowed"}]} 2023-05-31T21:33:33.070 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'group', 'modify', '--group-id', 'sync-bucket', '--status', 'enabled', '--bucket', 'pklxkd-74', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-05-31T21:33:33.071 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'group', 'modify', '--group-id', 'sync-bucket', '--status', 'enabled', '--bucket', 'pklxkd-74', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-05-31T21:33:33.071 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync group modify --group-id sync-bucket --status enabled --bucket pklxkd-74 --cluster c1 --debug-rgw 0 --debug-ms 0 2023-05-31T21:33:33.128 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2023-05-31T21:33:33.128 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2023-05-31T21:33:33.253 INFO:teuthology.orchestra.run.smithi099.stdout:{"groups":[{"id":"sync-bucket","data_flow":{"symmetrical":[{"id":"sync-flowA","zones":["a1","a2"]}]},"pipes":[{"id":"sync-pipe-pklxkd-74","source":{"bucket":"*","zones":["a1"]},"dest":{"bucket":"pklxkd-74","zones":["a2"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":"rgw-multisite-test-user"}},{"id":"sync-pipe-pklxkd-75","source":{"bucket":"*","zones":["a1"]},"dest":{"bucket":"pklxkd-75","zones":["a2"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":"rgw-multisite-test-user"}}],"status":"enabled"}]} 2023-05-31T21:33:33.253 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'policy', 'get', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-05-31T21:33:33.254 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'policy', 'get', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-05-31T21:33:33.254 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync policy get --cluster c1 --debug-rgw 0 --debug-ms 0 2023-05-31T21:33:33.304 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2023-05-31T21:33:33.305 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2023-05-31T21:33:33.411 INFO:teuthology.orchestra.run.smithi099.stdout:{"groups":[{"id":"sync-group","data_flow":{"symmetrical":[{"id":"sync-flow1","zones":["a1","a2"]}]},"pipes":[{"id":"sync-pipe-pklxkd-74","source":{"bucket":"pklxkd-74","zones":["a1"]},"dest":{"bucket":"pklxkd-74","zones":["a2"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":""}},{"id":"sync-pipe-pklxkd-75","source":{"bucket":"pklxkd-74","zones":["a1"]},"dest":{"bucket":"pklxkd-75","zones":["a2"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":""}},{"id":"sync-pipe","source":{"bucket":"*","zones":["*"]},"dest":{"bucket":"*","zones":["*"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":""}}],"status":"allowed"}]} 2023-05-31T21:33:33.411 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-31T21:33:33.412 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-31T21:33:33.412 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-05-31T21:33:33.464 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2023-05-31T21:33:33.464 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2023-05-31T21:33:33.508 INFO:teuthology.orchestra.run.smithi099.stderr:No --period given, using current period=2e6123d2-d695-4350-9a01-61a455bc782b 2023-05-31T21:33:33.522 INFO:teuthology.orchestra.run.smithi099.stdout:[ 2023-05-31T21:33:33.522 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T21:33:33.523 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1685566501.757183_260.1", 2023-05-31T21:33:33.523 INFO:teuthology.orchestra.run.smithi099.stdout: "last_update": "2023-05-31T20:55:01.757183Z" 2023-05-31T21:33:33.523 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:33:33.523 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T21:33:33.523 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1685566501.735717_251.1", 2023-05-31T21:33:33.523 INFO:teuthology.orchestra.run.smithi099.stdout: "last_update": "2023-05-31T20:55:01.735717Z" 2023-05-31T21:33:33.524 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:33:33.524 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T21:33:33.524 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1685562970.274572_93.1", 2023-05-31T21:33:33.524 INFO:teuthology.orchestra.run.smithi099.stdout: "last_update": "2023-05-31T19:56:10.274572Z" 2023-05-31T21:33:33.524 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:33:33.524 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T21:33:33.524 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1685568813.199723_1066.1", 2023-05-31T21:33:33.525 INFO:teuthology.orchestra.run.smithi099.stdout: "last_update": "2023-05-31T21:33:33.199723Z" 2023-05-31T21:33:33.525 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T21:33:33.525 INFO:teuthology.orchestra.run.smithi099.stdout:] 2023-05-31T21:33:33.526 DEBUG:rgw_multi.tests:master meta markers={0: '1_1685566501.757183_260.1', 1: '1_1685566501.735717_251.1', 2: '1_1685562970.274572_93.1', 3: '1_1685568813.199723_1066.1'} 2023-05-31T21:33:33.526 INFO:rgw_multi.tests:starting meta checkpoint for zone=a2 2023-05-31T21:33:33.526 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-31T21:33:33.527 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-31T21:33:33.527 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-05-31T21:33:33.576 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2023-05-31T21:33:33.577 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2023-05-31T21:33:33.620 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T21:33:33.616+0000 7f8558c1d780 20 RGW-SYNC:meta: read sync status 2023-05-31T21:33:33.621 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T21:33:33.617+0000 7f8558c1d780 20 RGW-SYNC:meta: read sync status 2023-05-31T21:33:33.634 INFO:teuthology.orchestra.run.smithi099.stdout:{ 2023-05-31T21:33:33.634 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_status": { 2023-05-31T21:33:33.635 INFO:teuthology.orchestra.run.smithi099.stdout: "info": { 2023-05-31T21:33:33.635 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "sync", 2023-05-31T21:33:33.635 INFO:teuthology.orchestra.run.smithi099.stdout: "num_shards": 4, 2023-05-31T21:33:33.635 INFO:teuthology.orchestra.run.smithi099.stdout: "period": "2e6123d2-d695-4350-9a01-61a455bc782b", 2023-05-31T21:33:33.635 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2023-05-31T21:33:33.635 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:33:33.636 INFO:teuthology.orchestra.run.smithi099.stdout: "markers": [ 2023-05-31T21:33:33.636 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T21:33:33.636 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 0, 2023-05-31T21:33:33.636 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T21:33:33.636 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2023-05-31T21:33:33.636 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1685566501.757183_260.1", 2023-05-31T21:33:33.636 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T21:33:33.637 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 1, 2023-05-31T21:33:33.637 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T21:33:33.637 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2023-05-31T20:55:01.757183Z", 2023-05-31T21:33:33.637 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2023-05-31T21:33:33.637 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T21:33:33.637 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:33:33.637 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T21:33:33.638 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 1, 2023-05-31T21:33:33.638 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T21:33:33.638 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2023-05-31T21:33:33.638 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1685566501.735717_251.1", 2023-05-31T21:33:33.638 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T21:33:33.638 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2023-05-31T21:33:33.638 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T21:33:33.639 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2023-05-31T20:55:01.735717Z", 2023-05-31T21:33:33.639 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2023-05-31T21:33:33.639 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T21:33:33.639 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:33:33.639 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T21:33:33.639 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 2, 2023-05-31T21:33:33.639 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T21:33:33.640 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2023-05-31T21:33:33.640 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1685562970.274572_93.1", 2023-05-31T21:33:33.640 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T21:33:33.640 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2023-05-31T21:33:33.640 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T21:33:33.640 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2023-05-31T19:56:10.274572Z", 2023-05-31T21:33:33.640 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2023-05-31T21:33:33.641 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T21:33:33.641 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:33:33.641 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T21:33:33.641 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 3, 2023-05-31T21:33:33.641 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T21:33:33.641 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2023-05-31T21:33:33.642 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1685568685.966725_1044.1", 2023-05-31T21:33:33.642 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T21:33:33.642 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2023-05-31T21:33:33.642 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T21:33:33.642 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2023-05-31T21:31:25.966725Z", 2023-05-31T21:33:33.642 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2023-05-31T21:33:33.642 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T21:33:33.642 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T21:33:33.643 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2023-05-31T21:33:33.643 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:33:33.643 INFO:teuthology.orchestra.run.smithi099.stdout: "full_sync": { 2023-05-31T21:33:33.643 INFO:teuthology.orchestra.run.smithi099.stdout: "total": 1, 2023-05-31T21:33:33.643 INFO:teuthology.orchestra.run.smithi099.stdout: "complete": 1 2023-05-31T21:33:33.643 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:33:33.643 INFO:teuthology.orchestra.run.smithi099.stdout: "current_time": "2023-05-31T21:33:33Z" 2023-05-31T21:33:33.644 INFO:teuthology.orchestra.run.smithi099.stdout:} 2023-05-31T21:33:33.644 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "2e6123d2-d695-4350-9a01-61a455bc782b", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1685566501.757183_260.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-05-31T20:55:01.757183Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1685566501.735717_251.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-05-31T20:55:01.735717Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1685562970.274572_93.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-05-31T19:56:10.274572Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1685568685.966725_1044.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-05-31T21:31:25.966725Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-05-31T21:33:33Z" } 2023-05-31T21:33:33.645 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1685566501.757183_260.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-05-31T20:55:01.757183Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1685566501.735717_251.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-05-31T20:55:01.735717Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1685562970.274572_93.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-05-31T19:56:10.274572Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1685568685.966725_1044.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-05-31T21:31:25.966725Z', 'realm_epoch': 2}}] 2023-05-31T21:33:33.645 DEBUG:rgw_multi.tests:log_status={0: '1_1685566501.757183_260.1', 1: '1_1685566501.735717_251.1', 2: '1_1685562970.274572_93.1', 3: '1_1685568813.199723_1066.1'} 2023-05-31T21:33:33.646 DEBUG:rgw_multi.tests:sync_status={0: '1_1685566501.757183_260.1', 1: '1_1685566501.735717_251.1', 2: '1_1685562970.274572_93.1', 3: '1_1685568685.966725_1044.1'} 2023-05-31T21:33:33.646 WARNING:rgw_multi.tests:zone a2 behind master: shard=3 master=1_1685568813.199723_1066.1 target=1_1685568685.966725_1044.1 2023-05-31T21:33:38.647 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-31T21:33:38.647 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-31T21:33:38.648 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-05-31T21:33:38.705 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2023-05-31T21:33:38.705 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2023-05-31T21:33:38.760 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T21:33:38.756+0000 7f6c4ede9780 20 RGW-SYNC:meta: read sync status 2023-05-31T21:33:38.761 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T21:33:38.757+0000 7f6c4ede9780 20 RGW-SYNC:meta: read sync status 2023-05-31T21:33:38.784 INFO:teuthology.orchestra.run.smithi099.stdout:{ 2023-05-31T21:33:38.784 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_status": { 2023-05-31T21:33:38.784 INFO:teuthology.orchestra.run.smithi099.stdout: "info": { 2023-05-31T21:33:38.784 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "sync", 2023-05-31T21:33:38.784 INFO:teuthology.orchestra.run.smithi099.stdout: "num_shards": 4, 2023-05-31T21:33:38.785 INFO:teuthology.orchestra.run.smithi099.stdout: "period": "2e6123d2-d695-4350-9a01-61a455bc782b", 2023-05-31T21:33:38.785 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2023-05-31T21:33:38.785 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:33:38.785 INFO:teuthology.orchestra.run.smithi099.stdout: "markers": [ 2023-05-31T21:33:38.785 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T21:33:38.785 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 0, 2023-05-31T21:33:38.785 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T21:33:38.786 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2023-05-31T21:33:38.786 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1685566501.757183_260.1", 2023-05-31T21:33:38.786 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T21:33:38.786 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 1, 2023-05-31T21:33:38.786 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T21:33:38.786 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2023-05-31T20:55:01.757183Z", 2023-05-31T21:33:38.786 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2023-05-31T21:33:38.787 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T21:33:38.787 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:33:38.787 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T21:33:38.787 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 1, 2023-05-31T21:33:38.787 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T21:33:38.787 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2023-05-31T21:33:38.788 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1685566501.735717_251.1", 2023-05-31T21:33:38.788 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T21:33:38.788 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2023-05-31T21:33:38.788 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T21:33:38.788 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2023-05-31T20:55:01.735717Z", 2023-05-31T21:33:38.788 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2023-05-31T21:33:38.788 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T21:33:38.789 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:33:38.789 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T21:33:38.789 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 2, 2023-05-31T21:33:38.789 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T21:33:38.789 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2023-05-31T21:33:38.789 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1685562970.274572_93.1", 2023-05-31T21:33:38.790 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T21:33:38.790 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2023-05-31T21:33:38.790 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T21:33:38.790 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2023-05-31T19:56:10.274572Z", 2023-05-31T21:33:38.790 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2023-05-31T21:33:38.790 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T21:33:38.790 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:33:38.791 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T21:33:38.791 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 3, 2023-05-31T21:33:38.791 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T21:33:38.791 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2023-05-31T21:33:38.791 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1685568685.966725_1044.1", 2023-05-31T21:33:38.791 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T21:33:38.791 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2023-05-31T21:33:38.792 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T21:33:38.792 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2023-05-31T21:31:25.966725Z", 2023-05-31T21:33:38.792 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2023-05-31T21:33:38.792 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T21:33:38.792 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T21:33:38.793 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2023-05-31T21:33:38.793 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:33:38.793 INFO:teuthology.orchestra.run.smithi099.stdout: "full_sync": { 2023-05-31T21:33:38.793 INFO:teuthology.orchestra.run.smithi099.stdout: "total": 1, 2023-05-31T21:33:38.793 INFO:teuthology.orchestra.run.smithi099.stdout: "complete": 1 2023-05-31T21:33:38.793 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:33:38.793 INFO:teuthology.orchestra.run.smithi099.stdout: "current_time": "2023-05-31T21:33:38Z" 2023-05-31T21:33:38.794 INFO:teuthology.orchestra.run.smithi099.stdout:} 2023-05-31T21:33:38.794 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "2e6123d2-d695-4350-9a01-61a455bc782b", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1685566501.757183_260.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-05-31T20:55:01.757183Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1685566501.735717_251.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-05-31T20:55:01.735717Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1685562970.274572_93.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-05-31T19:56:10.274572Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1685568685.966725_1044.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-05-31T21:31:25.966725Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-05-31T21:33:38Z" } 2023-05-31T21:33:38.795 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1685566501.757183_260.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-05-31T20:55:01.757183Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1685566501.735717_251.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-05-31T20:55:01.735717Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1685562970.274572_93.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-05-31T19:56:10.274572Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1685568685.966725_1044.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-05-31T21:31:25.966725Z', 'realm_epoch': 2}}] 2023-05-31T21:33:38.796 DEBUG:rgw_multi.tests:log_status={0: '1_1685566501.757183_260.1', 1: '1_1685566501.735717_251.1', 2: '1_1685562970.274572_93.1', 3: '1_1685568813.199723_1066.1'} 2023-05-31T21:33:38.796 DEBUG:rgw_multi.tests:sync_status={0: '1_1685566501.757183_260.1', 1: '1_1685566501.735717_251.1', 2: '1_1685562970.274572_93.1', 3: '1_1685568685.966725_1044.1'} 2023-05-31T21:33:38.796 WARNING:rgw_multi.tests:zone a2 behind master: shard=3 master=1_1685568813.199723_1066.1 target=1_1685568685.966725_1044.1 2023-05-31T21:33:43.797 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-31T21:33:43.798 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-31T21:33:43.798 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-05-31T21:33:43.855 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2023-05-31T21:33:43.855 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2023-05-31T21:33:43.890 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T21:33:43.886+0000 7f6be00a0780 20 RGW-SYNC:meta: read sync status 2023-05-31T21:33:43.890 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T21:33:43.887+0000 7f6be00a0780 20 RGW-SYNC:meta: read sync status 2023-05-31T21:33:43.905 INFO:teuthology.orchestra.run.smithi099.stdout:{ 2023-05-31T21:33:43.905 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_status": { 2023-05-31T21:33:43.905 INFO:teuthology.orchestra.run.smithi099.stdout: "info": { 2023-05-31T21:33:43.905 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "sync", 2023-05-31T21:33:43.905 INFO:teuthology.orchestra.run.smithi099.stdout: "num_shards": 4, 2023-05-31T21:33:43.906 INFO:teuthology.orchestra.run.smithi099.stdout: "period": "2e6123d2-d695-4350-9a01-61a455bc782b", 2023-05-31T21:33:43.906 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2023-05-31T21:33:43.906 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:33:43.906 INFO:teuthology.orchestra.run.smithi099.stdout: "markers": [ 2023-05-31T21:33:43.906 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T21:33:43.906 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 0, 2023-05-31T21:33:43.906 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T21:33:43.907 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2023-05-31T21:33:43.907 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1685566501.757183_260.1", 2023-05-31T21:33:43.907 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T21:33:43.907 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 1, 2023-05-31T21:33:43.907 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T21:33:43.907 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2023-05-31T20:55:01.757183Z", 2023-05-31T21:33:43.907 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2023-05-31T21:33:43.908 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T21:33:43.908 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:33:43.908 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T21:33:43.908 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 1, 2023-05-31T21:33:43.908 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T21:33:43.908 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2023-05-31T21:33:43.909 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1685566501.735717_251.1", 2023-05-31T21:33:43.909 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T21:33:43.909 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2023-05-31T21:33:43.909 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T21:33:43.909 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2023-05-31T20:55:01.735717Z", 2023-05-31T21:33:43.909 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2023-05-31T21:33:43.909 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T21:33:43.910 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:33:43.910 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T21:33:43.910 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 2, 2023-05-31T21:33:43.910 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T21:33:43.910 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2023-05-31T21:33:43.910 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1685562970.274572_93.1", 2023-05-31T21:33:43.911 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T21:33:43.911 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2023-05-31T21:33:43.911 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T21:33:43.911 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2023-05-31T19:56:10.274572Z", 2023-05-31T21:33:43.911 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2023-05-31T21:33:43.911 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T21:33:43.911 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:33:43.912 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T21:33:43.912 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 3, 2023-05-31T21:33:43.912 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T21:33:43.912 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2023-05-31T21:33:43.912 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1685568685.966725_1044.1", 2023-05-31T21:33:43.912 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T21:33:43.912 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2023-05-31T21:33:43.913 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T21:33:43.913 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2023-05-31T21:31:25.966725Z", 2023-05-31T21:33:43.913 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2023-05-31T21:33:43.913 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T21:33:43.913 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T21:33:43.913 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2023-05-31T21:33:43.913 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:33:43.914 INFO:teuthology.orchestra.run.smithi099.stdout: "full_sync": { 2023-05-31T21:33:43.914 INFO:teuthology.orchestra.run.smithi099.stdout: "total": 1, 2023-05-31T21:33:43.914 INFO:teuthology.orchestra.run.smithi099.stdout: "complete": 1 2023-05-31T21:33:43.914 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:33:43.914 INFO:teuthology.orchestra.run.smithi099.stdout: "current_time": "2023-05-31T21:33:43Z" 2023-05-31T21:33:43.914 INFO:teuthology.orchestra.run.smithi099.stdout:} 2023-05-31T21:33:43.915 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "2e6123d2-d695-4350-9a01-61a455bc782b", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1685566501.757183_260.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-05-31T20:55:01.757183Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1685566501.735717_251.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-05-31T20:55:01.735717Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1685562970.274572_93.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-05-31T19:56:10.274572Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1685568685.966725_1044.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-05-31T21:31:25.966725Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-05-31T21:33:43Z" } 2023-05-31T21:33:43.916 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1685566501.757183_260.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-05-31T20:55:01.757183Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1685566501.735717_251.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-05-31T20:55:01.735717Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1685562970.274572_93.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-05-31T19:56:10.274572Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1685568685.966725_1044.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-05-31T21:31:25.966725Z', 'realm_epoch': 2}}] 2023-05-31T21:33:43.916 DEBUG:rgw_multi.tests:log_status={0: '1_1685566501.757183_260.1', 1: '1_1685566501.735717_251.1', 2: '1_1685562970.274572_93.1', 3: '1_1685568813.199723_1066.1'} 2023-05-31T21:33:43.916 DEBUG:rgw_multi.tests:sync_status={0: '1_1685566501.757183_260.1', 1: '1_1685566501.735717_251.1', 2: '1_1685562970.274572_93.1', 3: '1_1685568685.966725_1044.1'} 2023-05-31T21:33:43.917 WARNING:rgw_multi.tests:zone a2 behind master: shard=3 master=1_1685568813.199723_1066.1 target=1_1685568685.966725_1044.1 2023-05-31T21:33:48.917 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-31T21:33:48.918 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-31T21:33:48.918 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-05-31T21:33:48.976 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2023-05-31T21:33:48.976 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2023-05-31T21:33:49.028 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T21:33:49.024+0000 7f1dfbf97780 20 RGW-SYNC:meta: read sync status 2023-05-31T21:33:49.029 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T21:33:49.025+0000 7f1dfbf97780 20 RGW-SYNC:meta: read sync status 2023-05-31T21:33:49.046 INFO:teuthology.orchestra.run.smithi099.stdout:{ 2023-05-31T21:33:49.046 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_status": { 2023-05-31T21:33:49.047 INFO:teuthology.orchestra.run.smithi099.stdout: "info": { 2023-05-31T21:33:49.047 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "sync", 2023-05-31T21:33:49.047 INFO:teuthology.orchestra.run.smithi099.stdout: "num_shards": 4, 2023-05-31T21:33:49.047 INFO:teuthology.orchestra.run.smithi099.stdout: "period": "2e6123d2-d695-4350-9a01-61a455bc782b", 2023-05-31T21:33:49.048 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2023-05-31T21:33:49.048 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:33:49.048 INFO:teuthology.orchestra.run.smithi099.stdout: "markers": [ 2023-05-31T21:33:49.048 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T21:33:49.048 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 0, 2023-05-31T21:33:49.048 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T21:33:49.049 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2023-05-31T21:33:49.049 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1685566501.757183_260.1", 2023-05-31T21:33:49.049 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T21:33:49.049 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 1, 2023-05-31T21:33:49.049 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T21:33:49.049 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2023-05-31T20:55:01.757183Z", 2023-05-31T21:33:49.049 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2023-05-31T21:33:49.050 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T21:33:49.050 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:33:49.050 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T21:33:49.050 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 1, 2023-05-31T21:33:49.050 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T21:33:49.050 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2023-05-31T21:33:49.050 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1685566501.735717_251.1", 2023-05-31T21:33:49.051 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T21:33:49.051 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2023-05-31T21:33:49.051 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T21:33:49.051 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2023-05-31T20:55:01.735717Z", 2023-05-31T21:33:49.051 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2023-05-31T21:33:49.051 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T21:33:49.051 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:33:49.052 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T21:33:49.052 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 2, 2023-05-31T21:33:49.052 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T21:33:49.052 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2023-05-31T21:33:49.052 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1685562970.274572_93.1", 2023-05-31T21:33:49.053 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T21:33:49.053 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2023-05-31T21:33:49.053 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T21:33:49.053 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2023-05-31T19:56:10.274572Z", 2023-05-31T21:33:49.053 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2023-05-31T21:33:49.053 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T21:33:49.053 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:33:49.054 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T21:33:49.054 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 3, 2023-05-31T21:33:49.054 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T21:33:49.054 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2023-05-31T21:33:49.054 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1685568685.966725_1044.1", 2023-05-31T21:33:49.054 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T21:33:49.054 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2023-05-31T21:33:49.055 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T21:33:49.055 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2023-05-31T21:31:25.966725Z", 2023-05-31T21:33:49.055 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2023-05-31T21:33:49.055 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T21:33:49.055 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T21:33:49.055 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2023-05-31T21:33:49.055 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:33:49.056 INFO:teuthology.orchestra.run.smithi099.stdout: "full_sync": { 2023-05-31T21:33:49.056 INFO:teuthology.orchestra.run.smithi099.stdout: "total": 1, 2023-05-31T21:33:49.056 INFO:teuthology.orchestra.run.smithi099.stdout: "complete": 1 2023-05-31T21:33:49.056 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:33:49.056 INFO:teuthology.orchestra.run.smithi099.stdout: "current_time": "2023-05-31T21:33:49Z" 2023-05-31T21:33:49.056 INFO:teuthology.orchestra.run.smithi099.stdout:} 2023-05-31T21:33:49.057 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "2e6123d2-d695-4350-9a01-61a455bc782b", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1685566501.757183_260.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-05-31T20:55:01.757183Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1685566501.735717_251.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-05-31T20:55:01.735717Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1685562970.274572_93.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-05-31T19:56:10.274572Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1685568685.966725_1044.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-05-31T21:31:25.966725Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-05-31T21:33:49Z" } 2023-05-31T21:33:49.058 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1685566501.757183_260.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-05-31T20:55:01.757183Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1685566501.735717_251.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-05-31T20:55:01.735717Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1685562970.274572_93.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-05-31T19:56:10.274572Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1685568685.966725_1044.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-05-31T21:31:25.966725Z', 'realm_epoch': 2}}] 2023-05-31T21:33:49.058 DEBUG:rgw_multi.tests:log_status={0: '1_1685566501.757183_260.1', 1: '1_1685566501.735717_251.1', 2: '1_1685562970.274572_93.1', 3: '1_1685568813.199723_1066.1'} 2023-05-31T21:33:49.058 DEBUG:rgw_multi.tests:sync_status={0: '1_1685566501.757183_260.1', 1: '1_1685566501.735717_251.1', 2: '1_1685562970.274572_93.1', 3: '1_1685568685.966725_1044.1'} 2023-05-31T21:33:49.059 WARNING:rgw_multi.tests:zone a2 behind master: shard=3 master=1_1685568813.199723_1066.1 target=1_1685568685.966725_1044.1 2023-05-31T21:33:54.060 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-31T21:33:54.061 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-31T21:33:54.061 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-05-31T21:33:54.118 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2023-05-31T21:33:54.118 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2023-05-31T21:33:54.161 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T21:33:54.158+0000 7fae3bfda780 20 RGW-SYNC:meta: read sync status 2023-05-31T21:33:54.162 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T21:33:54.159+0000 7fae3bfda780 20 RGW-SYNC:meta: read sync status 2023-05-31T21:33:54.179 INFO:teuthology.orchestra.run.smithi099.stdout:{ 2023-05-31T21:33:54.179 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_status": { 2023-05-31T21:33:54.180 INFO:teuthology.orchestra.run.smithi099.stdout: "info": { 2023-05-31T21:33:54.180 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "sync", 2023-05-31T21:33:54.180 INFO:teuthology.orchestra.run.smithi099.stdout: "num_shards": 4, 2023-05-31T21:33:54.180 INFO:teuthology.orchestra.run.smithi099.stdout: "period": "2e6123d2-d695-4350-9a01-61a455bc782b", 2023-05-31T21:33:54.180 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2023-05-31T21:33:54.180 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:33:54.181 INFO:teuthology.orchestra.run.smithi099.stdout: "markers": [ 2023-05-31T21:33:54.181 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T21:33:54.181 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 0, 2023-05-31T21:33:54.181 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T21:33:54.181 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2023-05-31T21:33:54.181 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1685566501.757183_260.1", 2023-05-31T21:33:54.181 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T21:33:54.182 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 1, 2023-05-31T21:33:54.182 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T21:33:54.182 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2023-05-31T20:55:01.757183Z", 2023-05-31T21:33:54.182 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2023-05-31T21:33:54.182 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T21:33:54.182 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:33:54.183 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T21:33:54.183 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 1, 2023-05-31T21:33:54.183 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T21:33:54.183 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2023-05-31T21:33:54.183 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1685566501.735717_251.1", 2023-05-31T21:33:54.183 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T21:33:54.183 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2023-05-31T21:33:54.184 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T21:33:54.184 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2023-05-31T20:55:01.735717Z", 2023-05-31T21:33:54.184 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2023-05-31T21:33:54.184 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T21:33:54.184 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:33:54.184 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T21:33:54.185 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 2, 2023-05-31T21:33:54.185 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T21:33:54.185 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2023-05-31T21:33:54.185 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1685562970.274572_93.1", 2023-05-31T21:33:54.185 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T21:33:54.185 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2023-05-31T21:33:54.185 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T21:33:54.186 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2023-05-31T19:56:10.274572Z", 2023-05-31T21:33:54.186 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2023-05-31T21:33:54.186 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T21:33:54.186 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:33:54.186 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T21:33:54.186 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 3, 2023-05-31T21:33:54.186 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T21:33:54.187 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2023-05-31T21:33:54.187 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1685568685.966725_1044.1", 2023-05-31T21:33:54.187 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T21:33:54.187 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2023-05-31T21:33:54.187 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T21:33:54.187 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2023-05-31T21:31:25.966725Z", 2023-05-31T21:33:54.187 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2023-05-31T21:33:54.188 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T21:33:54.188 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T21:33:54.188 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2023-05-31T21:33:54.188 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:33:54.188 INFO:teuthology.orchestra.run.smithi099.stdout: "full_sync": { 2023-05-31T21:33:54.188 INFO:teuthology.orchestra.run.smithi099.stdout: "total": 1, 2023-05-31T21:33:54.188 INFO:teuthology.orchestra.run.smithi099.stdout: "complete": 1 2023-05-31T21:33:54.189 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:33:54.189 INFO:teuthology.orchestra.run.smithi099.stdout: "current_time": "2023-05-31T21:33:54Z" 2023-05-31T21:33:54.189 INFO:teuthology.orchestra.run.smithi099.stdout:} 2023-05-31T21:33:54.190 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "2e6123d2-d695-4350-9a01-61a455bc782b", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1685566501.757183_260.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-05-31T20:55:01.757183Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1685566501.735717_251.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-05-31T20:55:01.735717Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1685562970.274572_93.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-05-31T19:56:10.274572Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1685568685.966725_1044.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-05-31T21:31:25.966725Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-05-31T21:33:54Z" } 2023-05-31T21:33:54.190 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1685566501.757183_260.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-05-31T20:55:01.757183Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1685566501.735717_251.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-05-31T20:55:01.735717Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1685562970.274572_93.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-05-31T19:56:10.274572Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1685568685.966725_1044.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-05-31T21:31:25.966725Z', 'realm_epoch': 2}}] 2023-05-31T21:33:54.191 DEBUG:rgw_multi.tests:log_status={0: '1_1685566501.757183_260.1', 1: '1_1685566501.735717_251.1', 2: '1_1685562970.274572_93.1', 3: '1_1685568813.199723_1066.1'} 2023-05-31T21:33:54.191 DEBUG:rgw_multi.tests:sync_status={0: '1_1685566501.757183_260.1', 1: '1_1685566501.735717_251.1', 2: '1_1685562970.274572_93.1', 3: '1_1685568685.966725_1044.1'} 2023-05-31T21:33:54.191 WARNING:rgw_multi.tests:zone a2 behind master: shard=3 master=1_1685568813.199723_1066.1 target=1_1685568685.966725_1044.1 2023-05-31T21:33:59.192 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-31T21:33:59.193 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-31T21:33:59.193 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-05-31T21:33:59.251 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2023-05-31T21:33:59.251 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2023-05-31T21:33:59.304 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T21:33:59.301+0000 7f6eeeb09780 20 RGW-SYNC:meta: read sync status 2023-05-31T21:33:59.305 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T21:33:59.302+0000 7f6eeeb09780 20 RGW-SYNC:meta: read sync status 2023-05-31T21:33:59.325 INFO:teuthology.orchestra.run.smithi099.stdout:{ 2023-05-31T21:33:59.325 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_status": { 2023-05-31T21:33:59.325 INFO:teuthology.orchestra.run.smithi099.stdout: "info": { 2023-05-31T21:33:59.325 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "sync", 2023-05-31T21:33:59.325 INFO:teuthology.orchestra.run.smithi099.stdout: "num_shards": 4, 2023-05-31T21:33:59.326 INFO:teuthology.orchestra.run.smithi099.stdout: "period": "2e6123d2-d695-4350-9a01-61a455bc782b", 2023-05-31T21:33:59.326 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2023-05-31T21:33:59.326 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:33:59.326 INFO:teuthology.orchestra.run.smithi099.stdout: "markers": [ 2023-05-31T21:33:59.326 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T21:33:59.326 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 0, 2023-05-31T21:33:59.326 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T21:33:59.327 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2023-05-31T21:33:59.327 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1685566501.757183_260.1", 2023-05-31T21:33:59.327 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T21:33:59.327 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 1, 2023-05-31T21:33:59.327 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T21:33:59.327 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2023-05-31T20:55:01.757183Z", 2023-05-31T21:33:59.328 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2023-05-31T21:33:59.328 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T21:33:59.328 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:33:59.328 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T21:33:59.328 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 1, 2023-05-31T21:33:59.328 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T21:33:59.329 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2023-05-31T21:33:59.329 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1685566501.735717_251.1", 2023-05-31T21:33:59.329 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T21:33:59.329 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2023-05-31T21:33:59.329 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T21:33:59.329 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2023-05-31T20:55:01.735717Z", 2023-05-31T21:33:59.330 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2023-05-31T21:33:59.330 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T21:33:59.330 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:33:59.330 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T21:33:59.330 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 2, 2023-05-31T21:33:59.330 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T21:33:59.330 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2023-05-31T21:33:59.331 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1685562970.274572_93.1", 2023-05-31T21:33:59.331 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T21:33:59.331 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2023-05-31T21:33:59.331 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T21:33:59.331 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2023-05-31T19:56:10.274572Z", 2023-05-31T21:33:59.331 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2023-05-31T21:33:59.332 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T21:33:59.332 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:33:59.332 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T21:33:59.332 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 3, 2023-05-31T21:33:59.332 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T21:33:59.332 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2023-05-31T21:33:59.332 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1685568685.966725_1044.1", 2023-05-31T21:33:59.333 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T21:33:59.333 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2023-05-31T21:33:59.333 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T21:33:59.333 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2023-05-31T21:31:25.966725Z", 2023-05-31T21:33:59.333 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2023-05-31T21:33:59.333 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T21:33:59.334 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T21:33:59.334 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2023-05-31T21:33:59.334 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:33:59.334 INFO:teuthology.orchestra.run.smithi099.stdout: "full_sync": { 2023-05-31T21:33:59.334 INFO:teuthology.orchestra.run.smithi099.stdout: "total": 1, 2023-05-31T21:33:59.334 INFO:teuthology.orchestra.run.smithi099.stdout: "complete": 1 2023-05-31T21:33:59.334 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:33:59.335 INFO:teuthology.orchestra.run.smithi099.stdout: "current_time": "2023-05-31T21:33:59Z" 2023-05-31T21:33:59.335 INFO:teuthology.orchestra.run.smithi099.stdout:} 2023-05-31T21:33:59.335 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "2e6123d2-d695-4350-9a01-61a455bc782b", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1685566501.757183_260.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-05-31T20:55:01.757183Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1685566501.735717_251.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-05-31T20:55:01.735717Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1685562970.274572_93.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-05-31T19:56:10.274572Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1685568685.966725_1044.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-05-31T21:31:25.966725Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-05-31T21:33:59Z" } 2023-05-31T21:33:59.336 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1685566501.757183_260.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-05-31T20:55:01.757183Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1685566501.735717_251.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-05-31T20:55:01.735717Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1685562970.274572_93.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-05-31T19:56:10.274572Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1685568685.966725_1044.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-05-31T21:31:25.966725Z', 'realm_epoch': 2}}] 2023-05-31T21:33:59.336 DEBUG:rgw_multi.tests:log_status={0: '1_1685566501.757183_260.1', 1: '1_1685566501.735717_251.1', 2: '1_1685562970.274572_93.1', 3: '1_1685568813.199723_1066.1'} 2023-05-31T21:33:59.337 DEBUG:rgw_multi.tests:sync_status={0: '1_1685566501.757183_260.1', 1: '1_1685566501.735717_251.1', 2: '1_1685562970.274572_93.1', 3: '1_1685568685.966725_1044.1'} 2023-05-31T21:33:59.337 WARNING:rgw_multi.tests:zone a2 behind master: shard=3 master=1_1685568813.199723_1066.1 target=1_1685568685.966725_1044.1 2023-05-31T21:34:04.339 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-31T21:34:04.340 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-31T21:34:04.340 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-05-31T21:34:04.397 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2023-05-31T21:34:04.397 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2023-05-31T21:34:04.449 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T21:34:04.446+0000 7fed4fd10780 20 RGW-SYNC:meta: read sync status 2023-05-31T21:34:04.449 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T21:34:04.447+0000 7fed4fd10780 20 RGW-SYNC:meta: read sync status 2023-05-31T21:34:04.468 INFO:teuthology.orchestra.run.smithi099.stdout:{ 2023-05-31T21:34:04.469 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_status": { 2023-05-31T21:34:04.469 INFO:teuthology.orchestra.run.smithi099.stdout: "info": { 2023-05-31T21:34:04.469 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "sync", 2023-05-31T21:34:04.469 INFO:teuthology.orchestra.run.smithi099.stdout: "num_shards": 4, 2023-05-31T21:34:04.469 INFO:teuthology.orchestra.run.smithi099.stdout: "period": "2e6123d2-d695-4350-9a01-61a455bc782b", 2023-05-31T21:34:04.469 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2023-05-31T21:34:04.469 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:34:04.470 INFO:teuthology.orchestra.run.smithi099.stdout: "markers": [ 2023-05-31T21:34:04.470 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T21:34:04.470 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 0, 2023-05-31T21:34:04.470 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T21:34:04.470 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2023-05-31T21:34:04.470 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1685566501.757183_260.1", 2023-05-31T21:34:04.471 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T21:34:04.471 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 1, 2023-05-31T21:34:04.471 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T21:34:04.471 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2023-05-31T20:55:01.757183Z", 2023-05-31T21:34:04.471 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2023-05-31T21:34:04.471 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T21:34:04.471 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:34:04.472 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T21:34:04.472 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 1, 2023-05-31T21:34:04.472 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T21:34:04.472 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2023-05-31T21:34:04.472 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1685566501.735717_251.1", 2023-05-31T21:34:04.472 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T21:34:04.472 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2023-05-31T21:34:04.473 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T21:34:04.473 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2023-05-31T20:55:01.735717Z", 2023-05-31T21:34:04.473 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2023-05-31T21:34:04.473 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T21:34:04.473 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:34:04.473 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T21:34:04.473 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 2, 2023-05-31T21:34:04.474 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T21:34:04.474 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2023-05-31T21:34:04.474 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1685562970.274572_93.1", 2023-05-31T21:34:04.474 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T21:34:04.474 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2023-05-31T21:34:04.474 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T21:34:04.474 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2023-05-31T19:56:10.274572Z", 2023-05-31T21:34:04.475 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2023-05-31T21:34:04.475 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T21:34:04.475 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:34:04.475 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T21:34:04.475 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 3, 2023-05-31T21:34:04.475 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T21:34:04.475 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2023-05-31T21:34:04.476 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1685568685.966725_1044.1", 2023-05-31T21:34:04.476 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T21:34:04.476 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2023-05-31T21:34:04.476 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T21:34:04.476 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2023-05-31T21:31:25.966725Z", 2023-05-31T21:34:04.476 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2023-05-31T21:34:04.476 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T21:34:04.477 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T21:34:04.477 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2023-05-31T21:34:04.477 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:34:04.477 INFO:teuthology.orchestra.run.smithi099.stdout: "full_sync": { 2023-05-31T21:34:04.477 INFO:teuthology.orchestra.run.smithi099.stdout: "total": 1, 2023-05-31T21:34:04.477 INFO:teuthology.orchestra.run.smithi099.stdout: "complete": 1 2023-05-31T21:34:04.477 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:34:04.478 INFO:teuthology.orchestra.run.smithi099.stdout: "current_time": "2023-05-31T21:34:04Z" 2023-05-31T21:34:04.478 INFO:teuthology.orchestra.run.smithi099.stdout:} 2023-05-31T21:34:04.478 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "2e6123d2-d695-4350-9a01-61a455bc782b", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1685566501.757183_260.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-05-31T20:55:01.757183Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1685566501.735717_251.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-05-31T20:55:01.735717Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1685562970.274572_93.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-05-31T19:56:10.274572Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1685568685.966725_1044.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-05-31T21:31:25.966725Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-05-31T21:34:04Z" } 2023-05-31T21:34:04.479 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1685566501.757183_260.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-05-31T20:55:01.757183Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1685566501.735717_251.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-05-31T20:55:01.735717Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1685562970.274572_93.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-05-31T19:56:10.274572Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1685568685.966725_1044.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-05-31T21:31:25.966725Z', 'realm_epoch': 2}}] 2023-05-31T21:34:04.479 DEBUG:rgw_multi.tests:log_status={0: '1_1685566501.757183_260.1', 1: '1_1685566501.735717_251.1', 2: '1_1685562970.274572_93.1', 3: '1_1685568813.199723_1066.1'} 2023-05-31T21:34:04.480 DEBUG:rgw_multi.tests:sync_status={0: '1_1685566501.757183_260.1', 1: '1_1685566501.735717_251.1', 2: '1_1685562970.274572_93.1', 3: '1_1685568685.966725_1044.1'} 2023-05-31T21:34:04.480 WARNING:rgw_multi.tests:zone a2 behind master: shard=3 master=1_1685568813.199723_1066.1 target=1_1685568685.966725_1044.1 2023-05-31T21:34:09.481 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-31T21:34:09.482 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-31T21:34:09.482 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-05-31T21:34:09.540 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2023-05-31T21:34:09.540 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2023-05-31T21:34:09.586 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T21:34:09.583+0000 7fde7dce0780 20 RGW-SYNC:meta: read sync status 2023-05-31T21:34:09.586 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T21:34:09.584+0000 7fde7dce0780 20 RGW-SYNC:meta: read sync status 2023-05-31T21:34:09.607 INFO:teuthology.orchestra.run.smithi099.stdout:{ 2023-05-31T21:34:09.607 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_status": { 2023-05-31T21:34:09.607 INFO:teuthology.orchestra.run.smithi099.stdout: "info": { 2023-05-31T21:34:09.607 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "sync", 2023-05-31T21:34:09.608 INFO:teuthology.orchestra.run.smithi099.stdout: "num_shards": 4, 2023-05-31T21:34:09.608 INFO:teuthology.orchestra.run.smithi099.stdout: "period": "2e6123d2-d695-4350-9a01-61a455bc782b", 2023-05-31T21:34:09.608 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2023-05-31T21:34:09.608 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:34:09.608 INFO:teuthology.orchestra.run.smithi099.stdout: "markers": [ 2023-05-31T21:34:09.608 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T21:34:09.608 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 0, 2023-05-31T21:34:09.609 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T21:34:09.609 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2023-05-31T21:34:09.609 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1685566501.757183_260.1", 2023-05-31T21:34:09.609 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T21:34:09.609 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 1, 2023-05-31T21:34:09.609 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T21:34:09.609 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2023-05-31T20:55:01.757183Z", 2023-05-31T21:34:09.610 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2023-05-31T21:34:09.610 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T21:34:09.610 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:34:09.610 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T21:34:09.610 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 1, 2023-05-31T21:34:09.610 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T21:34:09.610 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2023-05-31T21:34:09.611 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1685566501.735717_251.1", 2023-05-31T21:34:09.611 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T21:34:09.611 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2023-05-31T21:34:09.611 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T21:34:09.611 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2023-05-31T20:55:01.735717Z", 2023-05-31T21:34:09.611 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2023-05-31T21:34:09.611 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T21:34:09.612 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:34:09.612 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T21:34:09.612 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 2, 2023-05-31T21:34:09.612 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T21:34:09.612 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2023-05-31T21:34:09.612 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1685562970.274572_93.1", 2023-05-31T21:34:09.612 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T21:34:09.613 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2023-05-31T21:34:09.613 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T21:34:09.613 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2023-05-31T19:56:10.274572Z", 2023-05-31T21:34:09.613 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2023-05-31T21:34:09.613 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T21:34:09.613 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:34:09.613 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T21:34:09.614 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 3, 2023-05-31T21:34:09.614 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T21:34:09.614 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2023-05-31T21:34:09.614 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1685568685.966725_1044.1", 2023-05-31T21:34:09.614 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T21:34:09.614 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2023-05-31T21:34:09.614 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T21:34:09.615 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2023-05-31T21:31:25.966725Z", 2023-05-31T21:34:09.615 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2023-05-31T21:34:09.615 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T21:34:09.615 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T21:34:09.615 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2023-05-31T21:34:09.615 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:34:09.615 INFO:teuthology.orchestra.run.smithi099.stdout: "full_sync": { 2023-05-31T21:34:09.616 INFO:teuthology.orchestra.run.smithi099.stdout: "total": 1, 2023-05-31T21:34:09.616 INFO:teuthology.orchestra.run.smithi099.stdout: "complete": 1 2023-05-31T21:34:09.616 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:34:09.616 INFO:teuthology.orchestra.run.smithi099.stdout: "current_time": "2023-05-31T21:34:09Z" 2023-05-31T21:34:09.616 INFO:teuthology.orchestra.run.smithi099.stdout:} 2023-05-31T21:34:09.617 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "2e6123d2-d695-4350-9a01-61a455bc782b", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1685566501.757183_260.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-05-31T20:55:01.757183Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1685566501.735717_251.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-05-31T20:55:01.735717Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1685562970.274572_93.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-05-31T19:56:10.274572Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1685568685.966725_1044.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-05-31T21:31:25.966725Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-05-31T21:34:09Z" } 2023-05-31T21:34:09.617 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1685566501.757183_260.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-05-31T20:55:01.757183Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1685566501.735717_251.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-05-31T20:55:01.735717Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1685562970.274572_93.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-05-31T19:56:10.274572Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1685568685.966725_1044.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-05-31T21:31:25.966725Z', 'realm_epoch': 2}}] 2023-05-31T21:34:09.618 DEBUG:rgw_multi.tests:log_status={0: '1_1685566501.757183_260.1', 1: '1_1685566501.735717_251.1', 2: '1_1685562970.274572_93.1', 3: '1_1685568813.199723_1066.1'} 2023-05-31T21:34:09.618 DEBUG:rgw_multi.tests:sync_status={0: '1_1685566501.757183_260.1', 1: '1_1685566501.735717_251.1', 2: '1_1685562970.274572_93.1', 3: '1_1685568685.966725_1044.1'} 2023-05-31T21:34:09.619 WARNING:rgw_multi.tests:zone a2 behind master: shard=3 master=1_1685568813.199723_1066.1 target=1_1685568685.966725_1044.1 2023-05-31T21:34:14.619 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-31T21:34:14.620 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-31T21:34:14.621 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-05-31T21:34:14.677 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2023-05-31T21:34:14.678 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2023-05-31T21:34:14.725 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T21:34:14.721+0000 7f4a0c2f3780 20 RGW-SYNC:meta: read sync status 2023-05-31T21:34:14.726 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T21:34:14.722+0000 7f4a0c2f3780 20 RGW-SYNC:meta: read sync status 2023-05-31T21:34:14.744 INFO:teuthology.orchestra.run.smithi099.stdout:{ 2023-05-31T21:34:14.744 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_status": { 2023-05-31T21:34:14.744 INFO:teuthology.orchestra.run.smithi099.stdout: "info": { 2023-05-31T21:34:14.745 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "sync", 2023-05-31T21:34:14.745 INFO:teuthology.orchestra.run.smithi099.stdout: "num_shards": 4, 2023-05-31T21:34:14.745 INFO:teuthology.orchestra.run.smithi099.stdout: "period": "2e6123d2-d695-4350-9a01-61a455bc782b", 2023-05-31T21:34:14.745 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2023-05-31T21:34:14.745 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:34:14.745 INFO:teuthology.orchestra.run.smithi099.stdout: "markers": [ 2023-05-31T21:34:14.746 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T21:34:14.746 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 0, 2023-05-31T21:34:14.746 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T21:34:14.746 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2023-05-31T21:34:14.746 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1685566501.757183_260.1", 2023-05-31T21:34:14.746 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T21:34:14.747 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 1, 2023-05-31T21:34:14.747 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T21:34:14.747 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2023-05-31T20:55:01.757183Z", 2023-05-31T21:34:14.747 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2023-05-31T21:34:14.747 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T21:34:14.747 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:34:14.748 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T21:34:14.748 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 1, 2023-05-31T21:34:14.748 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T21:34:14.748 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2023-05-31T21:34:14.748 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1685566501.735717_251.1", 2023-05-31T21:34:14.748 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T21:34:14.748 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2023-05-31T21:34:14.749 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T21:34:14.749 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2023-05-31T20:55:01.735717Z", 2023-05-31T21:34:14.749 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2023-05-31T21:34:14.749 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T21:34:14.749 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:34:14.749 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T21:34:14.749 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 2, 2023-05-31T21:34:14.750 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T21:34:14.750 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2023-05-31T21:34:14.750 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1685562970.274572_93.1", 2023-05-31T21:34:14.750 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T21:34:14.750 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2023-05-31T21:34:14.750 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T21:34:14.750 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2023-05-31T19:56:10.274572Z", 2023-05-31T21:34:14.751 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2023-05-31T21:34:14.751 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T21:34:14.751 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:34:14.751 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T21:34:14.751 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 3, 2023-05-31T21:34:14.751 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T21:34:14.751 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2023-05-31T21:34:14.752 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1685568685.966725_1044.1", 2023-05-31T21:34:14.752 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T21:34:14.752 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2023-05-31T21:34:14.752 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T21:34:14.752 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2023-05-31T21:31:25.966725Z", 2023-05-31T21:34:14.752 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2023-05-31T21:34:14.752 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T21:34:14.753 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T21:34:14.753 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2023-05-31T21:34:14.753 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:34:14.753 INFO:teuthology.orchestra.run.smithi099.stdout: "full_sync": { 2023-05-31T21:34:14.753 INFO:teuthology.orchestra.run.smithi099.stdout: "total": 1, 2023-05-31T21:34:14.753 INFO:teuthology.orchestra.run.smithi099.stdout: "complete": 1 2023-05-31T21:34:14.753 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:34:14.754 INFO:teuthology.orchestra.run.smithi099.stdout: "current_time": "2023-05-31T21:34:14Z" 2023-05-31T21:34:14.754 INFO:teuthology.orchestra.run.smithi099.stdout:} 2023-05-31T21:34:14.754 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "2e6123d2-d695-4350-9a01-61a455bc782b", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1685566501.757183_260.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-05-31T20:55:01.757183Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1685566501.735717_251.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-05-31T20:55:01.735717Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1685562970.274572_93.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-05-31T19:56:10.274572Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1685568685.966725_1044.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-05-31T21:31:25.966725Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-05-31T21:34:14Z" } 2023-05-31T21:34:14.755 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1685566501.757183_260.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-05-31T20:55:01.757183Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1685566501.735717_251.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-05-31T20:55:01.735717Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1685562970.274572_93.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-05-31T19:56:10.274572Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1685568685.966725_1044.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-05-31T21:31:25.966725Z', 'realm_epoch': 2}}] 2023-05-31T21:34:14.755 DEBUG:rgw_multi.tests:log_status={0: '1_1685566501.757183_260.1', 1: '1_1685566501.735717_251.1', 2: '1_1685562970.274572_93.1', 3: '1_1685568813.199723_1066.1'} 2023-05-31T21:34:14.756 DEBUG:rgw_multi.tests:sync_status={0: '1_1685566501.757183_260.1', 1: '1_1685566501.735717_251.1', 2: '1_1685562970.274572_93.1', 3: '1_1685568685.966725_1044.1'} 2023-05-31T21:34:14.756 WARNING:rgw_multi.tests:zone a2 behind master: shard=3 master=1_1685568813.199723_1066.1 target=1_1685568685.966725_1044.1 2023-05-31T21:34:19.758 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-31T21:34:19.758 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-31T21:34:19.759 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-05-31T21:34:19.815 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2023-05-31T21:34:19.815 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2023-05-31T21:34:19.865 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T21:34:19.861+0000 7f2bda39c780 20 RGW-SYNC:meta: read sync status 2023-05-31T21:34:19.866 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T21:34:19.863+0000 7f2bda39c780 20 RGW-SYNC:meta: read sync status 2023-05-31T21:34:19.887 INFO:teuthology.orchestra.run.smithi099.stdout:{ 2023-05-31T21:34:19.887 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_status": { 2023-05-31T21:34:19.887 INFO:teuthology.orchestra.run.smithi099.stdout: "info": { 2023-05-31T21:34:19.887 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "sync", 2023-05-31T21:34:19.887 INFO:teuthology.orchestra.run.smithi099.stdout: "num_shards": 4, 2023-05-31T21:34:19.888 INFO:teuthology.orchestra.run.smithi099.stdout: "period": "2e6123d2-d695-4350-9a01-61a455bc782b", 2023-05-31T21:34:19.888 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2023-05-31T21:34:19.888 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:34:19.888 INFO:teuthology.orchestra.run.smithi099.stdout: "markers": [ 2023-05-31T21:34:19.888 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T21:34:19.889 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 0, 2023-05-31T21:34:19.889 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T21:34:19.889 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2023-05-31T21:34:19.889 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1685566501.757183_260.1", 2023-05-31T21:34:19.889 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T21:34:19.889 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 1, 2023-05-31T21:34:19.889 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T21:34:19.890 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2023-05-31T20:55:01.757183Z", 2023-05-31T21:34:19.890 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2023-05-31T21:34:19.890 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T21:34:19.890 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:34:19.890 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T21:34:19.890 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 1, 2023-05-31T21:34:19.890 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T21:34:19.891 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2023-05-31T21:34:19.891 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1685566501.735717_251.1", 2023-05-31T21:34:19.891 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T21:34:19.891 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2023-05-31T21:34:19.891 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T21:34:19.891 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2023-05-31T20:55:01.735717Z", 2023-05-31T21:34:19.891 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2023-05-31T21:34:19.892 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T21:34:19.892 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:34:19.892 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T21:34:19.892 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 2, 2023-05-31T21:34:19.892 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T21:34:19.892 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2023-05-31T21:34:19.892 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1685562970.274572_93.1", 2023-05-31T21:34:19.893 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T21:34:19.893 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2023-05-31T21:34:19.893 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T21:34:19.893 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2023-05-31T19:56:10.274572Z", 2023-05-31T21:34:19.893 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2023-05-31T21:34:19.893 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T21:34:19.894 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:34:19.894 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T21:34:19.894 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 3, 2023-05-31T21:34:19.894 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T21:34:19.894 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2023-05-31T21:34:19.894 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1685568685.966725_1044.1", 2023-05-31T21:34:19.894 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T21:34:19.895 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2023-05-31T21:34:19.895 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T21:34:19.895 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2023-05-31T21:31:25.966725Z", 2023-05-31T21:34:19.895 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2023-05-31T21:34:19.895 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T21:34:19.895 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T21:34:19.895 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2023-05-31T21:34:19.896 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:34:19.896 INFO:teuthology.orchestra.run.smithi099.stdout: "full_sync": { 2023-05-31T21:34:19.896 INFO:teuthology.orchestra.run.smithi099.stdout: "total": 1, 2023-05-31T21:34:19.896 INFO:teuthology.orchestra.run.smithi099.stdout: "complete": 1 2023-05-31T21:34:19.896 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:34:19.896 INFO:teuthology.orchestra.run.smithi099.stdout: "current_time": "2023-05-31T21:34:19Z" 2023-05-31T21:34:19.896 INFO:teuthology.orchestra.run.smithi099.stdout:} 2023-05-31T21:34:19.897 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "2e6123d2-d695-4350-9a01-61a455bc782b", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1685566501.757183_260.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-05-31T20:55:01.757183Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1685566501.735717_251.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-05-31T20:55:01.735717Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1685562970.274572_93.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-05-31T19:56:10.274572Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1685568685.966725_1044.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-05-31T21:31:25.966725Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-05-31T21:34:19Z" } 2023-05-31T21:34:19.898 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1685566501.757183_260.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-05-31T20:55:01.757183Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1685566501.735717_251.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-05-31T20:55:01.735717Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1685562970.274572_93.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-05-31T19:56:10.274572Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1685568685.966725_1044.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-05-31T21:31:25.966725Z', 'realm_epoch': 2}}] 2023-05-31T21:34:19.898 DEBUG:rgw_multi.tests:log_status={0: '1_1685566501.757183_260.1', 1: '1_1685566501.735717_251.1', 2: '1_1685562970.274572_93.1', 3: '1_1685568813.199723_1066.1'} 2023-05-31T21:34:19.898 DEBUG:rgw_multi.tests:sync_status={0: '1_1685566501.757183_260.1', 1: '1_1685566501.735717_251.1', 2: '1_1685562970.274572_93.1', 3: '1_1685568685.966725_1044.1'} 2023-05-31T21:34:19.899 WARNING:rgw_multi.tests:zone a2 behind master: shard=3 master=1_1685568813.199723_1066.1 target=1_1685568685.966725_1044.1 2023-05-31T21:34:24.900 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-31T21:34:24.900 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-31T21:34:24.901 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-05-31T21:34:24.958 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2023-05-31T21:34:24.958 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2023-05-31T21:34:24.991 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T21:34:24.987+0000 7f497b7e0780 20 RGW-SYNC:meta: read sync status 2023-05-31T21:34:24.992 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T21:34:24.988+0000 7f497b7e0780 20 RGW-SYNC:meta: read sync status 2023-05-31T21:34:25.005 INFO:teuthology.orchestra.run.smithi099.stdout:{ 2023-05-31T21:34:25.006 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_status": { 2023-05-31T21:34:25.006 INFO:teuthology.orchestra.run.smithi099.stdout: "info": { 2023-05-31T21:34:25.006 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "sync", 2023-05-31T21:34:25.006 INFO:teuthology.orchestra.run.smithi099.stdout: "num_shards": 4, 2023-05-31T21:34:25.006 INFO:teuthology.orchestra.run.smithi099.stdout: "period": "2e6123d2-d695-4350-9a01-61a455bc782b", 2023-05-31T21:34:25.007 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2023-05-31T21:34:25.007 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:34:25.007 INFO:teuthology.orchestra.run.smithi099.stdout: "markers": [ 2023-05-31T21:34:25.007 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T21:34:25.007 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 0, 2023-05-31T21:34:25.007 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T21:34:25.007 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2023-05-31T21:34:25.008 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1685566501.757183_260.1", 2023-05-31T21:34:25.008 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T21:34:25.008 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 1, 2023-05-31T21:34:25.008 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T21:34:25.008 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2023-05-31T20:55:01.757183Z", 2023-05-31T21:34:25.008 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2023-05-31T21:34:25.009 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T21:34:25.009 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:34:25.009 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T21:34:25.009 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 1, 2023-05-31T21:34:25.009 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T21:34:25.009 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2023-05-31T21:34:25.009 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1685566501.735717_251.1", 2023-05-31T21:34:25.010 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T21:34:25.010 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2023-05-31T21:34:25.010 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T21:34:25.010 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2023-05-31T20:55:01.735717Z", 2023-05-31T21:34:25.010 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2023-05-31T21:34:25.010 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T21:34:25.010 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:34:25.011 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T21:34:25.011 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 2, 2023-05-31T21:34:25.011 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T21:34:25.011 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2023-05-31T21:34:25.011 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1685562970.274572_93.1", 2023-05-31T21:34:25.011 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T21:34:25.011 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2023-05-31T21:34:25.012 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T21:34:25.012 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2023-05-31T19:56:10.274572Z", 2023-05-31T21:34:25.012 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2023-05-31T21:34:25.012 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T21:34:25.012 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:34:25.012 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T21:34:25.012 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 3, 2023-05-31T21:34:25.013 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T21:34:25.013 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2023-05-31T21:34:25.013 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1685568685.966725_1044.1", 2023-05-31T21:34:25.013 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T21:34:25.013 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2023-05-31T21:34:25.013 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T21:34:25.013 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2023-05-31T21:31:25.966725Z", 2023-05-31T21:34:25.014 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2023-05-31T21:34:25.014 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T21:34:25.014 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T21:34:25.014 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2023-05-31T21:34:25.014 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:34:25.014 INFO:teuthology.orchestra.run.smithi099.stdout: "full_sync": { 2023-05-31T21:34:25.014 INFO:teuthology.orchestra.run.smithi099.stdout: "total": 1, 2023-05-31T21:34:25.015 INFO:teuthology.orchestra.run.smithi099.stdout: "complete": 1 2023-05-31T21:34:25.015 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:34:25.015 INFO:teuthology.orchestra.run.smithi099.stdout: "current_time": "2023-05-31T21:34:24Z" 2023-05-31T21:34:25.015 INFO:teuthology.orchestra.run.smithi099.stdout:} 2023-05-31T21:34:25.016 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "2e6123d2-d695-4350-9a01-61a455bc782b", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1685566501.757183_260.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-05-31T20:55:01.757183Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1685566501.735717_251.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-05-31T20:55:01.735717Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1685562970.274572_93.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-05-31T19:56:10.274572Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1685568685.966725_1044.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-05-31T21:31:25.966725Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-05-31T21:34:24Z" } 2023-05-31T21:34:25.016 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1685566501.757183_260.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-05-31T20:55:01.757183Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1685566501.735717_251.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-05-31T20:55:01.735717Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1685562970.274572_93.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-05-31T19:56:10.274572Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1685568685.966725_1044.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-05-31T21:31:25.966725Z', 'realm_epoch': 2}}] 2023-05-31T21:34:25.017 DEBUG:rgw_multi.tests:log_status={0: '1_1685566501.757183_260.1', 1: '1_1685566501.735717_251.1', 2: '1_1685562970.274572_93.1', 3: '1_1685568813.199723_1066.1'} 2023-05-31T21:34:25.017 DEBUG:rgw_multi.tests:sync_status={0: '1_1685566501.757183_260.1', 1: '1_1685566501.735717_251.1', 2: '1_1685562970.274572_93.1', 3: '1_1685568685.966725_1044.1'} 2023-05-31T21:34:25.017 WARNING:rgw_multi.tests:zone a2 behind master: shard=3 master=1_1685568813.199723_1066.1 target=1_1685568685.966725_1044.1 2023-05-31T21:34:30.019 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-31T21:34:30.019 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-31T21:34:30.020 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-05-31T21:34:30.077 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2023-05-31T21:34:30.078 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2023-05-31T21:34:30.131 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T21:34:30.127+0000 7f8ec8608780 20 RGW-SYNC:meta: read sync status 2023-05-31T21:34:30.132 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T21:34:30.128+0000 7f8ec8608780 20 RGW-SYNC:meta: read sync status 2023-05-31T21:34:30.155 INFO:teuthology.orchestra.run.smithi099.stdout:{ 2023-05-31T21:34:30.155 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_status": { 2023-05-31T21:34:30.155 INFO:teuthology.orchestra.run.smithi099.stdout: "info": { 2023-05-31T21:34:30.156 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "sync", 2023-05-31T21:34:30.156 INFO:teuthology.orchestra.run.smithi099.stdout: "num_shards": 4, 2023-05-31T21:34:30.156 INFO:teuthology.orchestra.run.smithi099.stdout: "period": "2e6123d2-d695-4350-9a01-61a455bc782b", 2023-05-31T21:34:30.156 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2023-05-31T21:34:30.156 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:34:30.156 INFO:teuthology.orchestra.run.smithi099.stdout: "markers": [ 2023-05-31T21:34:30.157 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T21:34:30.157 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 0, 2023-05-31T21:34:30.157 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T21:34:30.157 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2023-05-31T21:34:30.157 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1685566501.757183_260.1", 2023-05-31T21:34:30.157 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T21:34:30.157 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 1, 2023-05-31T21:34:30.158 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T21:34:30.158 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2023-05-31T20:55:01.757183Z", 2023-05-31T21:34:30.158 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2023-05-31T21:34:30.158 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T21:34:30.158 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:34:30.158 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T21:34:30.158 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 1, 2023-05-31T21:34:30.159 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T21:34:30.159 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2023-05-31T21:34:30.159 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1685566501.735717_251.1", 2023-05-31T21:34:30.159 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T21:34:30.159 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2023-05-31T21:34:30.159 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T21:34:30.159 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2023-05-31T20:55:01.735717Z", 2023-05-31T21:34:30.160 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2023-05-31T21:34:30.160 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T21:34:30.160 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:34:30.160 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T21:34:30.160 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 2, 2023-05-31T21:34:30.160 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T21:34:30.161 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2023-05-31T21:34:30.161 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1685562970.274572_93.1", 2023-05-31T21:34:30.161 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T21:34:30.161 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2023-05-31T21:34:30.161 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T21:34:30.161 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2023-05-31T19:56:10.274572Z", 2023-05-31T21:34:30.161 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2023-05-31T21:34:30.162 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T21:34:30.162 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:34:30.162 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T21:34:30.162 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 3, 2023-05-31T21:34:30.162 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T21:34:30.162 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2023-05-31T21:34:30.162 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1685568685.966725_1044.1", 2023-05-31T21:34:30.163 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T21:34:30.163 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2023-05-31T21:34:30.163 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T21:34:30.163 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2023-05-31T21:31:25.966725Z", 2023-05-31T21:34:30.163 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2023-05-31T21:34:30.163 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T21:34:30.163 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T21:34:30.164 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2023-05-31T21:34:30.164 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:34:30.164 INFO:teuthology.orchestra.run.smithi099.stdout: "full_sync": { 2023-05-31T21:34:30.164 INFO:teuthology.orchestra.run.smithi099.stdout: "total": 1, 2023-05-31T21:34:30.164 INFO:teuthology.orchestra.run.smithi099.stdout: "complete": 1 2023-05-31T21:34:30.164 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:34:30.164 INFO:teuthology.orchestra.run.smithi099.stdout: "current_time": "2023-05-31T21:34:30Z" 2023-05-31T21:34:30.165 INFO:teuthology.orchestra.run.smithi099.stdout:} 2023-05-31T21:34:30.165 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "2e6123d2-d695-4350-9a01-61a455bc782b", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1685566501.757183_260.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-05-31T20:55:01.757183Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1685566501.735717_251.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-05-31T20:55:01.735717Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1685562970.274572_93.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-05-31T19:56:10.274572Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1685568685.966725_1044.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-05-31T21:31:25.966725Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-05-31T21:34:30Z" } 2023-05-31T21:34:30.166 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1685566501.757183_260.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-05-31T20:55:01.757183Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1685566501.735717_251.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-05-31T20:55:01.735717Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1685562970.274572_93.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-05-31T19:56:10.274572Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1685568685.966725_1044.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-05-31T21:31:25.966725Z', 'realm_epoch': 2}}] 2023-05-31T21:34:30.166 DEBUG:rgw_multi.tests:log_status={0: '1_1685566501.757183_260.1', 1: '1_1685566501.735717_251.1', 2: '1_1685562970.274572_93.1', 3: '1_1685568813.199723_1066.1'} 2023-05-31T21:34:30.167 DEBUG:rgw_multi.tests:sync_status={0: '1_1685566501.757183_260.1', 1: '1_1685566501.735717_251.1', 2: '1_1685562970.274572_93.1', 3: '1_1685568685.966725_1044.1'} 2023-05-31T21:34:30.167 WARNING:rgw_multi.tests:zone a2 behind master: shard=3 master=1_1685568813.199723_1066.1 target=1_1685568685.966725_1044.1 2023-05-31T21:34:35.168 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-31T21:34:35.168 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-31T21:34:35.169 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-05-31T21:34:35.226 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2023-05-31T21:34:35.226 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2023-05-31T21:34:35.262 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T21:34:35.259+0000 7f83c81b3780 20 RGW-SYNC:meta: read sync status 2023-05-31T21:34:35.263 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T21:34:35.260+0000 7f83c81b3780 20 RGW-SYNC:meta: read sync status 2023-05-31T21:34:35.278 INFO:teuthology.orchestra.run.smithi099.stdout:{ 2023-05-31T21:34:35.278 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_status": { 2023-05-31T21:34:35.278 INFO:teuthology.orchestra.run.smithi099.stdout: "info": { 2023-05-31T21:34:35.278 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "sync", 2023-05-31T21:34:35.278 INFO:teuthology.orchestra.run.smithi099.stdout: "num_shards": 4, 2023-05-31T21:34:35.279 INFO:teuthology.orchestra.run.smithi099.stdout: "period": "2e6123d2-d695-4350-9a01-61a455bc782b", 2023-05-31T21:34:35.279 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2023-05-31T21:34:35.279 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:34:35.279 INFO:teuthology.orchestra.run.smithi099.stdout: "markers": [ 2023-05-31T21:34:35.279 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T21:34:35.279 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 0, 2023-05-31T21:34:35.279 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T21:34:35.280 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2023-05-31T21:34:35.280 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1685566501.757183_260.1", 2023-05-31T21:34:35.280 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T21:34:35.280 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 1, 2023-05-31T21:34:35.280 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T21:34:35.280 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2023-05-31T20:55:01.757183Z", 2023-05-31T21:34:35.281 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2023-05-31T21:34:35.281 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T21:34:35.281 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:34:35.281 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T21:34:35.281 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 1, 2023-05-31T21:34:35.281 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T21:34:35.281 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2023-05-31T21:34:35.282 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1685566501.735717_251.1", 2023-05-31T21:34:35.282 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T21:34:35.282 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2023-05-31T21:34:35.282 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T21:34:35.282 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2023-05-31T20:55:01.735717Z", 2023-05-31T21:34:35.282 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2023-05-31T21:34:35.282 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T21:34:35.283 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:34:35.283 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T21:34:35.283 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 2, 2023-05-31T21:34:35.283 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T21:34:35.283 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2023-05-31T21:34:35.283 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1685562970.274572_93.1", 2023-05-31T21:34:35.283 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T21:34:35.284 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2023-05-31T21:34:35.284 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T21:34:35.284 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2023-05-31T19:56:10.274572Z", 2023-05-31T21:34:35.284 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2023-05-31T21:34:35.284 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T21:34:35.284 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:34:35.284 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T21:34:35.285 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 3, 2023-05-31T21:34:35.285 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T21:34:35.285 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2023-05-31T21:34:35.285 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1685568685.966725_1044.1", 2023-05-31T21:34:35.285 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T21:34:35.285 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2023-05-31T21:34:35.286 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T21:34:35.286 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2023-05-31T21:31:25.966725Z", 2023-05-31T21:34:35.286 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2023-05-31T21:34:35.286 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T21:34:35.286 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T21:34:35.286 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2023-05-31T21:34:35.286 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:34:35.287 INFO:teuthology.orchestra.run.smithi099.stdout: "full_sync": { 2023-05-31T21:34:35.287 INFO:teuthology.orchestra.run.smithi099.stdout: "total": 1, 2023-05-31T21:34:35.287 INFO:teuthology.orchestra.run.smithi099.stdout: "complete": 1 2023-05-31T21:34:35.287 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:34:35.287 INFO:teuthology.orchestra.run.smithi099.stdout: "current_time": "2023-05-31T21:34:35Z" 2023-05-31T21:34:35.287 INFO:teuthology.orchestra.run.smithi099.stdout:} 2023-05-31T21:34:35.288 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "2e6123d2-d695-4350-9a01-61a455bc782b", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1685566501.757183_260.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-05-31T20:55:01.757183Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1685566501.735717_251.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-05-31T20:55:01.735717Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1685562970.274572_93.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-05-31T19:56:10.274572Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1685568685.966725_1044.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-05-31T21:31:25.966725Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-05-31T21:34:35Z" } 2023-05-31T21:34:35.289 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1685566501.757183_260.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-05-31T20:55:01.757183Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1685566501.735717_251.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-05-31T20:55:01.735717Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1685562970.274572_93.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-05-31T19:56:10.274572Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1685568685.966725_1044.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-05-31T21:31:25.966725Z', 'realm_epoch': 2}}] 2023-05-31T21:34:35.289 DEBUG:rgw_multi.tests:log_status={0: '1_1685566501.757183_260.1', 1: '1_1685566501.735717_251.1', 2: '1_1685562970.274572_93.1', 3: '1_1685568813.199723_1066.1'} 2023-05-31T21:34:35.289 DEBUG:rgw_multi.tests:sync_status={0: '1_1685566501.757183_260.1', 1: '1_1685566501.735717_251.1', 2: '1_1685562970.274572_93.1', 3: '1_1685568685.966725_1044.1'} 2023-05-31T21:34:35.290 WARNING:rgw_multi.tests:zone a2 behind master: shard=3 master=1_1685568813.199723_1066.1 target=1_1685568685.966725_1044.1 2023-05-31T21:34:40.291 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-31T21:34:40.292 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-31T21:34:40.293 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-05-31T21:34:40.350 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2023-05-31T21:34:40.350 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2023-05-31T21:34:40.398 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T21:34:40.395+0000 7f18374c2780 20 RGW-SYNC:meta: read sync status 2023-05-31T21:34:40.398 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T21:34:40.396+0000 7f18374c2780 20 RGW-SYNC:meta: read sync status 2023-05-31T21:34:40.419 INFO:teuthology.orchestra.run.smithi099.stdout:{ 2023-05-31T21:34:40.419 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_status": { 2023-05-31T21:34:40.420 INFO:teuthology.orchestra.run.smithi099.stdout: "info": { 2023-05-31T21:34:40.420 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "sync", 2023-05-31T21:34:40.420 INFO:teuthology.orchestra.run.smithi099.stdout: "num_shards": 4, 2023-05-31T21:34:40.420 INFO:teuthology.orchestra.run.smithi099.stdout: "period": "2e6123d2-d695-4350-9a01-61a455bc782b", 2023-05-31T21:34:40.421 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2023-05-31T21:34:40.421 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:34:40.421 INFO:teuthology.orchestra.run.smithi099.stdout: "markers": [ 2023-05-31T21:34:40.421 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T21:34:40.421 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 0, 2023-05-31T21:34:40.422 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T21:34:40.422 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2023-05-31T21:34:40.422 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1685566501.757183_260.1", 2023-05-31T21:34:40.422 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T21:34:40.423 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 1, 2023-05-31T21:34:40.423 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T21:34:40.423 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2023-05-31T20:55:01.757183Z", 2023-05-31T21:34:40.424 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2023-05-31T21:34:40.424 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T21:34:40.424 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:34:40.424 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T21:34:40.424 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 1, 2023-05-31T21:34:40.425 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T21:34:40.425 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2023-05-31T21:34:40.425 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1685566501.735717_251.1", 2023-05-31T21:34:40.425 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T21:34:40.425 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2023-05-31T21:34:40.425 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T21:34:40.426 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2023-05-31T20:55:01.735717Z", 2023-05-31T21:34:40.426 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2023-05-31T21:34:40.426 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T21:34:40.426 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:34:40.426 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T21:34:40.427 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 2, 2023-05-31T21:34:40.427 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T21:34:40.427 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2023-05-31T21:34:40.427 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1685562970.274572_93.1", 2023-05-31T21:34:40.428 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T21:34:40.428 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2023-05-31T21:34:40.428 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T21:34:40.428 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2023-05-31T19:56:10.274572Z", 2023-05-31T21:34:40.429 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2023-05-31T21:34:40.429 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T21:34:40.429 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:34:40.429 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T21:34:40.429 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 3, 2023-05-31T21:34:40.429 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T21:34:40.429 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2023-05-31T21:34:40.430 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1685568685.966725_1044.1", 2023-05-31T21:34:40.430 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T21:34:40.430 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2023-05-31T21:34:40.430 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T21:34:40.431 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2023-05-31T21:31:25.966725Z", 2023-05-31T21:34:40.431 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2023-05-31T21:34:40.431 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T21:34:40.431 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T21:34:40.432 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2023-05-31T21:34:40.432 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:34:40.432 INFO:teuthology.orchestra.run.smithi099.stdout: "full_sync": { 2023-05-31T21:34:40.432 INFO:teuthology.orchestra.run.smithi099.stdout: "total": 1, 2023-05-31T21:34:40.432 INFO:teuthology.orchestra.run.smithi099.stdout: "complete": 1 2023-05-31T21:34:40.433 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:34:40.433 INFO:teuthology.orchestra.run.smithi099.stdout: "current_time": "2023-05-31T21:34:40Z" 2023-05-31T21:34:40.433 INFO:teuthology.orchestra.run.smithi099.stdout:} 2023-05-31T21:34:40.434 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "2e6123d2-d695-4350-9a01-61a455bc782b", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1685566501.757183_260.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-05-31T20:55:01.757183Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1685566501.735717_251.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-05-31T20:55:01.735717Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1685562970.274572_93.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-05-31T19:56:10.274572Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1685568685.966725_1044.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-05-31T21:31:25.966725Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-05-31T21:34:40Z" } 2023-05-31T21:34:40.434 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1685566501.757183_260.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-05-31T20:55:01.757183Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1685566501.735717_251.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-05-31T20:55:01.735717Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1685562970.274572_93.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-05-31T19:56:10.274572Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1685568685.966725_1044.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-05-31T21:31:25.966725Z', 'realm_epoch': 2}}] 2023-05-31T21:34:40.435 DEBUG:rgw_multi.tests:log_status={0: '1_1685566501.757183_260.1', 1: '1_1685566501.735717_251.1', 2: '1_1685562970.274572_93.1', 3: '1_1685568813.199723_1066.1'} 2023-05-31T21:34:40.435 DEBUG:rgw_multi.tests:sync_status={0: '1_1685566501.757183_260.1', 1: '1_1685566501.735717_251.1', 2: '1_1685562970.274572_93.1', 3: '1_1685568685.966725_1044.1'} 2023-05-31T21:34:40.436 WARNING:rgw_multi.tests:zone a2 behind master: shard=3 master=1_1685568813.199723_1066.1 target=1_1685568685.966725_1044.1 2023-05-31T21:34:45.437 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-31T21:34:45.437 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-31T21:34:45.438 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-05-31T21:34:45.494 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2023-05-31T21:34:45.495 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2023-05-31T21:34:45.547 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T21:34:45.544+0000 7fbc92d3b780 20 RGW-SYNC:meta: read sync status 2023-05-31T21:34:45.548 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T21:34:45.545+0000 7fbc92d3b780 20 RGW-SYNC:meta: read sync status 2023-05-31T21:34:45.562 INFO:teuthology.orchestra.run.smithi099.stdout:{ 2023-05-31T21:34:45.562 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_status": { 2023-05-31T21:34:45.562 INFO:teuthology.orchestra.run.smithi099.stdout: "info": { 2023-05-31T21:34:45.563 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "sync", 2023-05-31T21:34:45.563 INFO:teuthology.orchestra.run.smithi099.stdout: "num_shards": 4, 2023-05-31T21:34:45.563 INFO:teuthology.orchestra.run.smithi099.stdout: "period": "2e6123d2-d695-4350-9a01-61a455bc782b", 2023-05-31T21:34:45.563 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2023-05-31T21:34:45.563 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:34:45.563 INFO:teuthology.orchestra.run.smithi099.stdout: "markers": [ 2023-05-31T21:34:45.563 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T21:34:45.564 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 0, 2023-05-31T21:34:45.564 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T21:34:45.564 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2023-05-31T21:34:45.564 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1685566501.757183_260.1", 2023-05-31T21:34:45.564 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T21:34:45.564 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 1, 2023-05-31T21:34:45.565 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T21:34:45.565 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2023-05-31T20:55:01.757183Z", 2023-05-31T21:34:45.565 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2023-05-31T21:34:45.565 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T21:34:45.565 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:34:45.565 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T21:34:45.565 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 1, 2023-05-31T21:34:45.566 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T21:34:45.566 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2023-05-31T21:34:45.566 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1685566501.735717_251.1", 2023-05-31T21:34:45.566 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T21:34:45.566 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2023-05-31T21:34:45.566 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T21:34:45.566 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2023-05-31T20:55:01.735717Z", 2023-05-31T21:34:45.567 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2023-05-31T21:34:45.567 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T21:34:45.567 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:34:45.567 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T21:34:45.567 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 2, 2023-05-31T21:34:45.567 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T21:34:45.567 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2023-05-31T21:34:45.568 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1685562970.274572_93.1", 2023-05-31T21:34:45.568 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T21:34:45.568 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2023-05-31T21:34:45.568 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T21:34:45.568 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2023-05-31T19:56:10.274572Z", 2023-05-31T21:34:45.568 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2023-05-31T21:34:45.568 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T21:34:45.569 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:34:45.569 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T21:34:45.569 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 3, 2023-05-31T21:34:45.569 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T21:34:45.569 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2023-05-31T21:34:45.569 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1685568685.966725_1044.1", 2023-05-31T21:34:45.569 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T21:34:45.570 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2023-05-31T21:34:45.570 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T21:34:45.570 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2023-05-31T21:31:25.966725Z", 2023-05-31T21:34:45.570 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2023-05-31T21:34:45.570 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T21:34:45.570 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T21:34:45.570 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2023-05-31T21:34:45.571 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:34:45.571 INFO:teuthology.orchestra.run.smithi099.stdout: "full_sync": { 2023-05-31T21:34:45.571 INFO:teuthology.orchestra.run.smithi099.stdout: "total": 1, 2023-05-31T21:34:45.571 INFO:teuthology.orchestra.run.smithi099.stdout: "complete": 1 2023-05-31T21:34:45.571 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:34:45.571 INFO:teuthology.orchestra.run.smithi099.stdout: "current_time": "2023-05-31T21:34:45Z" 2023-05-31T21:34:45.571 INFO:teuthology.orchestra.run.smithi099.stdout:} 2023-05-31T21:34:45.572 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "2e6123d2-d695-4350-9a01-61a455bc782b", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1685566501.757183_260.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-05-31T20:55:01.757183Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1685566501.735717_251.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-05-31T20:55:01.735717Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1685562970.274572_93.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-05-31T19:56:10.274572Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1685568685.966725_1044.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-05-31T21:31:25.966725Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-05-31T21:34:45Z" } 2023-05-31T21:34:45.573 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1685566501.757183_260.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-05-31T20:55:01.757183Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1685566501.735717_251.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-05-31T20:55:01.735717Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1685562970.274572_93.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-05-31T19:56:10.274572Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1685568685.966725_1044.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-05-31T21:31:25.966725Z', 'realm_epoch': 2}}] 2023-05-31T21:34:45.573 DEBUG:rgw_multi.tests:log_status={0: '1_1685566501.757183_260.1', 1: '1_1685566501.735717_251.1', 2: '1_1685562970.274572_93.1', 3: '1_1685568813.199723_1066.1'} 2023-05-31T21:34:45.573 DEBUG:rgw_multi.tests:sync_status={0: '1_1685566501.757183_260.1', 1: '1_1685566501.735717_251.1', 2: '1_1685562970.274572_93.1', 3: '1_1685568685.966725_1044.1'} 2023-05-31T21:34:45.574 WARNING:rgw_multi.tests:zone a2 behind master: shard=3 master=1_1685568813.199723_1066.1 target=1_1685568685.966725_1044.1 2023-05-31T21:34:50.575 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-31T21:34:50.575 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-31T21:34:50.576 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-05-31T21:34:50.632 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2023-05-31T21:34:50.632 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2023-05-31T21:34:50.667 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T21:34:50.664+0000 7f549b2a0780 20 RGW-SYNC:meta: read sync status 2023-05-31T21:34:50.668 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T21:34:50.665+0000 7f549b2a0780 20 RGW-SYNC:meta: read sync status 2023-05-31T21:34:50.686 INFO:teuthology.orchestra.run.smithi099.stdout:{ 2023-05-31T21:34:50.686 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_status": { 2023-05-31T21:34:50.686 INFO:teuthology.orchestra.run.smithi099.stdout: "info": { 2023-05-31T21:34:50.686 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "sync", 2023-05-31T21:34:50.686 INFO:teuthology.orchestra.run.smithi099.stdout: "num_shards": 4, 2023-05-31T21:34:50.686 INFO:teuthology.orchestra.run.smithi099.stdout: "period": "2e6123d2-d695-4350-9a01-61a455bc782b", 2023-05-31T21:34:50.687 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2023-05-31T21:34:50.687 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:34:50.687 INFO:teuthology.orchestra.run.smithi099.stdout: "markers": [ 2023-05-31T21:34:50.687 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T21:34:50.687 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 0, 2023-05-31T21:34:50.687 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T21:34:50.688 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2023-05-31T21:34:50.688 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1685566501.757183_260.1", 2023-05-31T21:34:50.688 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T21:34:50.688 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 1, 2023-05-31T21:34:50.688 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T21:34:50.688 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2023-05-31T20:55:01.757183Z", 2023-05-31T21:34:50.688 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2023-05-31T21:34:50.689 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T21:34:50.689 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:34:50.689 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T21:34:50.689 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 1, 2023-05-31T21:34:50.689 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T21:34:50.689 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2023-05-31T21:34:50.690 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1685566501.735717_251.1", 2023-05-31T21:34:50.690 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T21:34:50.690 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2023-05-31T21:34:50.690 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T21:34:50.690 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2023-05-31T20:55:01.735717Z", 2023-05-31T21:34:50.690 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2023-05-31T21:34:50.690 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T21:34:50.691 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:34:50.691 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T21:34:50.691 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 2, 2023-05-31T21:34:50.691 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T21:34:50.691 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2023-05-31T21:34:50.691 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1685562970.274572_93.1", 2023-05-31T21:34:50.691 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T21:34:50.692 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2023-05-31T21:34:50.692 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T21:34:50.692 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2023-05-31T19:56:10.274572Z", 2023-05-31T21:34:50.692 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2023-05-31T21:34:50.692 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T21:34:50.692 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:34:50.692 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T21:34:50.693 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 3, 2023-05-31T21:34:50.693 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T21:34:50.693 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2023-05-31T21:34:50.693 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1685568685.966725_1044.1", 2023-05-31T21:34:50.693 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T21:34:50.693 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2023-05-31T21:34:50.693 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T21:34:50.694 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2023-05-31T21:31:25.966725Z", 2023-05-31T21:34:50.694 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2023-05-31T21:34:50.694 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T21:34:50.694 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T21:34:50.694 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2023-05-31T21:34:50.694 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:34:50.694 INFO:teuthology.orchestra.run.smithi099.stdout: "full_sync": { 2023-05-31T21:34:50.695 INFO:teuthology.orchestra.run.smithi099.stdout: "total": 1, 2023-05-31T21:34:50.695 INFO:teuthology.orchestra.run.smithi099.stdout: "complete": 1 2023-05-31T21:34:50.695 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:34:50.695 INFO:teuthology.orchestra.run.smithi099.stdout: "current_time": "2023-05-31T21:34:50Z" 2023-05-31T21:34:50.695 INFO:teuthology.orchestra.run.smithi099.stdout:} 2023-05-31T21:34:50.696 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "2e6123d2-d695-4350-9a01-61a455bc782b", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1685566501.757183_260.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-05-31T20:55:01.757183Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1685566501.735717_251.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-05-31T20:55:01.735717Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1685562970.274572_93.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-05-31T19:56:10.274572Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1685568685.966725_1044.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-05-31T21:31:25.966725Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-05-31T21:34:50Z" } 2023-05-31T21:34:50.696 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1685566501.757183_260.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-05-31T20:55:01.757183Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1685566501.735717_251.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-05-31T20:55:01.735717Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1685562970.274572_93.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-05-31T19:56:10.274572Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1685568685.966725_1044.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-05-31T21:31:25.966725Z', 'realm_epoch': 2}}] 2023-05-31T21:34:50.697 DEBUG:rgw_multi.tests:log_status={0: '1_1685566501.757183_260.1', 1: '1_1685566501.735717_251.1', 2: '1_1685562970.274572_93.1', 3: '1_1685568813.199723_1066.1'} 2023-05-31T21:34:50.697 DEBUG:rgw_multi.tests:sync_status={0: '1_1685566501.757183_260.1', 1: '1_1685566501.735717_251.1', 2: '1_1685562970.274572_93.1', 3: '1_1685568685.966725_1044.1'} 2023-05-31T21:34:50.698 WARNING:rgw_multi.tests:zone a2 behind master: shard=3 master=1_1685568813.199723_1066.1 target=1_1685568685.966725_1044.1 2023-05-31T21:34:55.699 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-31T21:34:55.699 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-31T21:34:55.700 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-05-31T21:34:55.756 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2023-05-31T21:34:55.756 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2023-05-31T21:34:55.811 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T21:34:55.807+0000 7fce6fd7b780 20 RGW-SYNC:meta: read sync status 2023-05-31T21:34:55.812 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T21:34:55.808+0000 7fce6fd7b780 20 RGW-SYNC:meta: read sync status 2023-05-31T21:34:55.833 INFO:teuthology.orchestra.run.smithi099.stdout:{ 2023-05-31T21:34:55.833 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_status": { 2023-05-31T21:34:55.833 INFO:teuthology.orchestra.run.smithi099.stdout: "info": { 2023-05-31T21:34:55.834 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "sync", 2023-05-31T21:34:55.834 INFO:teuthology.orchestra.run.smithi099.stdout: "num_shards": 4, 2023-05-31T21:34:55.834 INFO:teuthology.orchestra.run.smithi099.stdout: "period": "2e6123d2-d695-4350-9a01-61a455bc782b", 2023-05-31T21:34:55.834 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2023-05-31T21:34:55.834 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:34:55.834 INFO:teuthology.orchestra.run.smithi099.stdout: "markers": [ 2023-05-31T21:34:55.835 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T21:34:55.835 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 0, 2023-05-31T21:34:55.835 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T21:34:55.835 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2023-05-31T21:34:55.835 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1685566501.757183_260.1", 2023-05-31T21:34:55.835 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T21:34:55.835 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 1, 2023-05-31T21:34:55.836 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T21:34:55.836 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2023-05-31T20:55:01.757183Z", 2023-05-31T21:34:55.836 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2023-05-31T21:34:55.836 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T21:34:55.836 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:34:55.836 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T21:34:55.836 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 1, 2023-05-31T21:34:55.837 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T21:34:55.837 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2023-05-31T21:34:55.837 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1685566501.735717_251.1", 2023-05-31T21:34:55.837 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T21:34:55.837 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2023-05-31T21:34:55.837 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T21:34:55.837 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2023-05-31T20:55:01.735717Z", 2023-05-31T21:34:55.838 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2023-05-31T21:34:55.838 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T21:34:55.838 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:34:55.838 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T21:34:55.838 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 2, 2023-05-31T21:34:55.838 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T21:34:55.838 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2023-05-31T21:34:55.839 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1685562970.274572_93.1", 2023-05-31T21:34:55.839 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T21:34:55.839 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2023-05-31T21:34:55.839 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T21:34:55.839 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2023-05-31T19:56:10.274572Z", 2023-05-31T21:34:55.839 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2023-05-31T21:34:55.839 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T21:34:55.840 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:34:55.840 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T21:34:55.840 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 3, 2023-05-31T21:34:55.840 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T21:34:55.840 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2023-05-31T21:34:55.840 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1685568685.966725_1044.1", 2023-05-31T21:34:55.841 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T21:34:55.841 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2023-05-31T21:34:55.841 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T21:34:55.841 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2023-05-31T21:31:25.966725Z", 2023-05-31T21:34:55.841 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2023-05-31T21:34:55.841 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T21:34:55.841 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T21:34:55.842 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2023-05-31T21:34:55.842 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:34:55.842 INFO:teuthology.orchestra.run.smithi099.stdout: "full_sync": { 2023-05-31T21:34:55.842 INFO:teuthology.orchestra.run.smithi099.stdout: "total": 1, 2023-05-31T21:34:55.842 INFO:teuthology.orchestra.run.smithi099.stdout: "complete": 1 2023-05-31T21:34:55.842 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:34:55.842 INFO:teuthology.orchestra.run.smithi099.stdout: "current_time": "2023-05-31T21:34:55Z" 2023-05-31T21:34:55.843 INFO:teuthology.orchestra.run.smithi099.stdout:} 2023-05-31T21:34:55.843 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "2e6123d2-d695-4350-9a01-61a455bc782b", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1685566501.757183_260.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-05-31T20:55:01.757183Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1685566501.735717_251.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-05-31T20:55:01.735717Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1685562970.274572_93.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-05-31T19:56:10.274572Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1685568685.966725_1044.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-05-31T21:31:25.966725Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-05-31T21:34:55Z" } 2023-05-31T21:34:55.844 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1685566501.757183_260.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-05-31T20:55:01.757183Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1685566501.735717_251.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-05-31T20:55:01.735717Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1685562970.274572_93.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-05-31T19:56:10.274572Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1685568685.966725_1044.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-05-31T21:31:25.966725Z', 'realm_epoch': 2}}] 2023-05-31T21:34:55.844 DEBUG:rgw_multi.tests:log_status={0: '1_1685566501.757183_260.1', 1: '1_1685566501.735717_251.1', 2: '1_1685562970.274572_93.1', 3: '1_1685568813.199723_1066.1'} 2023-05-31T21:34:55.845 DEBUG:rgw_multi.tests:sync_status={0: '1_1685566501.757183_260.1', 1: '1_1685566501.735717_251.1', 2: '1_1685562970.274572_93.1', 3: '1_1685568685.966725_1044.1'} 2023-05-31T21:34:55.845 WARNING:rgw_multi.tests:zone a2 behind master: shard=3 master=1_1685568813.199723_1066.1 target=1_1685568685.966725_1044.1 2023-05-31T21:35:00.845 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-31T21:35:00.846 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-31T21:35:00.847 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-05-31T21:35:00.904 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2023-05-31T21:35:00.904 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2023-05-31T21:35:00.944 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T21:35:00.940+0000 7f107d2bd780 20 RGW-SYNC:meta: read sync status 2023-05-31T21:35:00.945 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T21:35:00.941+0000 7f107d2bd780 20 RGW-SYNC:meta: read sync status 2023-05-31T21:35:00.964 INFO:teuthology.orchestra.run.smithi099.stdout:{ 2023-05-31T21:35:00.965 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_status": { 2023-05-31T21:35:00.965 INFO:teuthology.orchestra.run.smithi099.stdout: "info": { 2023-05-31T21:35:00.965 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "sync", 2023-05-31T21:35:00.965 INFO:teuthology.orchestra.run.smithi099.stdout: "num_shards": 4, 2023-05-31T21:35:00.965 INFO:teuthology.orchestra.run.smithi099.stdout: "period": "2e6123d2-d695-4350-9a01-61a455bc782b", 2023-05-31T21:35:00.966 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2023-05-31T21:35:00.966 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:35:00.966 INFO:teuthology.orchestra.run.smithi099.stdout: "markers": [ 2023-05-31T21:35:00.966 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T21:35:00.966 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 0, 2023-05-31T21:35:00.966 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T21:35:00.966 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2023-05-31T21:35:00.967 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1685566501.757183_260.1", 2023-05-31T21:35:00.967 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T21:35:00.967 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 1, 2023-05-31T21:35:00.967 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T21:35:00.967 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2023-05-31T20:55:01.757183Z", 2023-05-31T21:35:00.967 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2023-05-31T21:35:00.967 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T21:35:00.968 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:35:00.968 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T21:35:00.968 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 1, 2023-05-31T21:35:00.968 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T21:35:00.968 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2023-05-31T21:35:00.968 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1685566501.735717_251.1", 2023-05-31T21:35:00.968 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T21:35:00.969 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2023-05-31T21:35:00.969 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T21:35:00.969 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2023-05-31T20:55:01.735717Z", 2023-05-31T21:35:00.969 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2023-05-31T21:35:00.969 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T21:35:00.969 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:35:00.970 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T21:35:00.970 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 2, 2023-05-31T21:35:00.970 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T21:35:00.970 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2023-05-31T21:35:00.970 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1685562970.274572_93.1", 2023-05-31T21:35:00.970 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T21:35:00.970 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2023-05-31T21:35:00.971 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T21:35:00.971 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2023-05-31T19:56:10.274572Z", 2023-05-31T21:35:00.971 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2023-05-31T21:35:00.971 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T21:35:00.971 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:35:00.971 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T21:35:00.971 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 3, 2023-05-31T21:35:00.972 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T21:35:00.972 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2023-05-31T21:35:00.972 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1685568685.966725_1044.1", 2023-05-31T21:35:00.972 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T21:35:00.972 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2023-05-31T21:35:00.972 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T21:35:00.972 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2023-05-31T21:31:25.966725Z", 2023-05-31T21:35:00.973 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2023-05-31T21:35:00.973 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T21:35:00.973 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T21:35:00.973 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2023-05-31T21:35:00.973 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:35:00.973 INFO:teuthology.orchestra.run.smithi099.stdout: "full_sync": { 2023-05-31T21:35:00.973 INFO:teuthology.orchestra.run.smithi099.stdout: "total": 1, 2023-05-31T21:35:00.974 INFO:teuthology.orchestra.run.smithi099.stdout: "complete": 1 2023-05-31T21:35:00.974 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:35:00.974 INFO:teuthology.orchestra.run.smithi099.stdout: "current_time": "2023-05-31T21:35:00Z" 2023-05-31T21:35:00.974 INFO:teuthology.orchestra.run.smithi099.stdout:} 2023-05-31T21:35:00.975 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "2e6123d2-d695-4350-9a01-61a455bc782b", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1685566501.757183_260.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-05-31T20:55:01.757183Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1685566501.735717_251.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-05-31T20:55:01.735717Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1685562970.274572_93.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-05-31T19:56:10.274572Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1685568685.966725_1044.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-05-31T21:31:25.966725Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-05-31T21:35:00Z" } 2023-05-31T21:35:00.975 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1685566501.757183_260.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-05-31T20:55:01.757183Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1685566501.735717_251.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-05-31T20:55:01.735717Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1685562970.274572_93.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-05-31T19:56:10.274572Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1685568685.966725_1044.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-05-31T21:31:25.966725Z', 'realm_epoch': 2}}] 2023-05-31T21:35:00.976 DEBUG:rgw_multi.tests:log_status={0: '1_1685566501.757183_260.1', 1: '1_1685566501.735717_251.1', 2: '1_1685562970.274572_93.1', 3: '1_1685568813.199723_1066.1'} 2023-05-31T21:35:00.976 DEBUG:rgw_multi.tests:sync_status={0: '1_1685566501.757183_260.1', 1: '1_1685566501.735717_251.1', 2: '1_1685562970.274572_93.1', 3: '1_1685568685.966725_1044.1'} 2023-05-31T21:35:00.976 WARNING:rgw_multi.tests:zone a2 behind master: shard=3 master=1_1685568813.199723_1066.1 target=1_1685568685.966725_1044.1 2023-05-31T21:35:05.977 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-31T21:35:05.978 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-31T21:35:05.979 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-05-31T21:35:06.038 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2023-05-31T21:35:06.038 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2023-05-31T21:35:06.090 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T21:35:06.086+0000 7f14987fe780 20 RGW-SYNC:meta: read sync status 2023-05-31T21:35:06.091 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T21:35:06.087+0000 7f14987fe780 20 RGW-SYNC:meta: read sync status 2023-05-31T21:35:06.113 INFO:teuthology.orchestra.run.smithi099.stdout:{ 2023-05-31T21:35:06.113 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_status": { 2023-05-31T21:35:06.113 INFO:teuthology.orchestra.run.smithi099.stdout: "info": { 2023-05-31T21:35:06.114 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "sync", 2023-05-31T21:35:06.114 INFO:teuthology.orchestra.run.smithi099.stdout: "num_shards": 4, 2023-05-31T21:35:06.114 INFO:teuthology.orchestra.run.smithi099.stdout: "period": "2e6123d2-d695-4350-9a01-61a455bc782b", 2023-05-31T21:35:06.114 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2023-05-31T21:35:06.114 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:35:06.114 INFO:teuthology.orchestra.run.smithi099.stdout: "markers": [ 2023-05-31T21:35:06.114 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T21:35:06.115 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 0, 2023-05-31T21:35:06.115 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T21:35:06.115 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2023-05-31T21:35:06.115 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1685566501.757183_260.1", 2023-05-31T21:35:06.115 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T21:35:06.115 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 1, 2023-05-31T21:35:06.116 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T21:35:06.116 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2023-05-31T20:55:01.757183Z", 2023-05-31T21:35:06.116 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2023-05-31T21:35:06.116 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T21:35:06.116 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:35:06.116 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T21:35:06.116 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 1, 2023-05-31T21:35:06.117 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T21:35:06.117 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2023-05-31T21:35:06.117 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1685566501.735717_251.1", 2023-05-31T21:35:06.117 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T21:35:06.117 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2023-05-31T21:35:06.117 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T21:35:06.117 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2023-05-31T20:55:01.735717Z", 2023-05-31T21:35:06.118 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2023-05-31T21:35:06.118 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T21:35:06.118 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:35:06.118 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T21:35:06.118 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 2, 2023-05-31T21:35:06.118 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T21:35:06.118 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2023-05-31T21:35:06.119 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1685562970.274572_93.1", 2023-05-31T21:35:06.119 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T21:35:06.119 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2023-05-31T21:35:06.119 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T21:35:06.119 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2023-05-31T19:56:10.274572Z", 2023-05-31T21:35:06.119 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2023-05-31T21:35:06.119 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T21:35:06.120 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:35:06.120 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T21:35:06.120 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 3, 2023-05-31T21:35:06.120 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T21:35:06.120 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2023-05-31T21:35:06.120 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1685568813.199723_1066.1", 2023-05-31T21:35:06.120 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T21:35:06.121 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2023-05-31T21:35:06.121 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T21:35:06.121 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2023-05-31T21:33:33.199723Z", 2023-05-31T21:35:06.121 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2023-05-31T21:35:06.121 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T21:35:06.121 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T21:35:06.121 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2023-05-31T21:35:06.122 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:35:06.122 INFO:teuthology.orchestra.run.smithi099.stdout: "full_sync": { 2023-05-31T21:35:06.122 INFO:teuthology.orchestra.run.smithi099.stdout: "total": 1, 2023-05-31T21:35:06.122 INFO:teuthology.orchestra.run.smithi099.stdout: "complete": 1 2023-05-31T21:35:06.122 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:35:06.122 INFO:teuthology.orchestra.run.smithi099.stdout: "current_time": "2023-05-31T21:35:06Z" 2023-05-31T21:35:06.122 INFO:teuthology.orchestra.run.smithi099.stdout:} 2023-05-31T21:35:06.123 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "2e6123d2-d695-4350-9a01-61a455bc782b", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1685566501.757183_260.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-05-31T20:55:01.757183Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1685566501.735717_251.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-05-31T20:55:01.735717Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1685562970.274572_93.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-05-31T19:56:10.274572Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1685568813.199723_1066.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-05-31T21:33:33.199723Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-05-31T21:35:06Z" } 2023-05-31T21:35:06.124 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1685566501.757183_260.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-05-31T20:55:01.757183Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1685566501.735717_251.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-05-31T20:55:01.735717Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1685562970.274572_93.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-05-31T19:56:10.274572Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1685568813.199723_1066.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-05-31T21:33:33.199723Z', 'realm_epoch': 2}}] 2023-05-31T21:35:06.124 DEBUG:rgw_multi.tests:log_status={0: '1_1685566501.757183_260.1', 1: '1_1685566501.735717_251.1', 2: '1_1685562970.274572_93.1', 3: '1_1685568813.199723_1066.1'} 2023-05-31T21:35:06.124 DEBUG:rgw_multi.tests:sync_status={0: '1_1685566501.757183_260.1', 1: '1_1685566501.735717_251.1', 2: '1_1685562970.274572_93.1', 3: '1_1685568813.199723_1066.1'} 2023-05-31T21:35:06.125 INFO:rgw_multi.tests:finish meta checkpoint for zone=a2 2023-05-31T21:35:06.125 DEBUG:boto:path=/pklxkd-74/ 2023-05-31T21:35:06.125 DEBUG:boto:auth_path=/pklxkd-74/ 2023-05-31T21:35:06.126 DEBUG:boto:Method: HEAD 2023-05-31T21:35:06.126 DEBUG:boto:Path: /pklxkd-74/ 2023-05-31T21:35:06.127 DEBUG:boto:Data: 2023-05-31T21:35:06.127 DEBUG:boto:Headers: {} 2023-05-31T21:35:06.127 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T21:35:06.128 DEBUG:boto:Port: 8000 2023-05-31T21:35:06.128 DEBUG:boto:Params: {} 2023-05-31T21:35:06.128 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2023-05-31T21:35:06.129 DEBUG:boto:Token: None 2023-05-31T21:35:06.129 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 21:35:06 GMT /pklxkd-74/ 2023-05-31T21:35:06.130 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:HAIoFFYmFEytK8o/dxB3D6fBvCI= 2023-05-31T21:35:06.130 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 21:35:06 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:HAIoFFYmFEytK8o/dxB3D6fBvCI=', 'Content-Length': '0'} 2023-05-31T21:35:06.134 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '2'), ('X-RGW-Bytes-Used', '12'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000003cf4c9fc621f5a4f-006477bd8a-2bf5-a1'), ('Content-Length', '0'), ('Date', 'Wed, 31 May 2023 21:35:06 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:35:06.135 DEBUG:boto:path=/pklxkd-74/obj3 2023-05-31T21:35:06.135 DEBUG:boto:auth_path=/pklxkd-74/obj3 2023-05-31T21:35:06.136 DEBUG:boto:Method: PUT 2023-05-31T21:35:06.136 DEBUG:boto:Path: /pklxkd-74/obj3 2023-05-31T21:35:06.136 DEBUG:boto:Data: 2023-05-31T21:35:06.137 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2023-05-31T21:35:06.137 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T21:35:06.137 DEBUG:boto:Port: 8000 2023-05-31T21:35:06.138 DEBUG:boto:Params: {} 2023-05-31T21:35:06.138 DEBUG:boto:Token: None 2023-05-31T21:35:06.139 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Wed, 31 May 2023 21:35:06 GMT /pklxkd-74/obj3 2023-05-31T21:35:06.139 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:QChTXSB5DmAALG578+V3Nl/1iIY= 2023-05-31T21:35:06.139 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Wed, 31 May 2023 21:35:06 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:QChTXSB5DmAALG578+V3Nl/1iIY='} 2023-05-31T21:35:06.149 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000032577cac5217064c-006477bd8a-2bf5-a1'), ('Date', 'Wed, 31 May 2023 21:35:06 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:35:06.150 DEBUG:boto:path=/pklxkd-74/ 2023-05-31T21:35:06.150 DEBUG:boto:auth_path=/pklxkd-74/ 2023-05-31T21:35:06.151 DEBUG:boto:Method: HEAD 2023-05-31T21:35:06.151 DEBUG:boto:Path: /pklxkd-74/ 2023-05-31T21:35:06.151 DEBUG:boto:Data: 2023-05-31T21:35:06.152 DEBUG:boto:Headers: {} 2023-05-31T21:35:06.152 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T21:35:06.152 DEBUG:boto:Port: 8000 2023-05-31T21:35:06.153 DEBUG:boto:Params: {} 2023-05-31T21:35:06.153 DEBUG:boto:Token: None 2023-05-31T21:35:06.153 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 21:35:06 GMT /pklxkd-74/ 2023-05-31T21:35:06.154 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:HAIoFFYmFEytK8o/dxB3D6fBvCI= 2023-05-31T21:35:06.154 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 21:35:06 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:HAIoFFYmFEytK8o/dxB3D6fBvCI=', 'Content-Length': '0'} 2023-05-31T21:35:06.156 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '3'), ('X-RGW-Bytes-Used', '18'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000073cca8c733889f31-006477bd8a-2bf5-a1'), ('Content-Length', '0'), ('Date', 'Wed, 31 May 2023 21:35:06 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:35:06.157 DEBUG:boto:path=/pklxkd-74/obj4 2023-05-31T21:35:06.157 DEBUG:boto:auth_path=/pklxkd-74/obj4 2023-05-31T21:35:06.158 DEBUG:boto:Method: PUT 2023-05-31T21:35:06.158 DEBUG:boto:Path: /pklxkd-74/obj4 2023-05-31T21:35:06.158 DEBUG:boto:Data: 2023-05-31T21:35:06.159 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2023-05-31T21:35:06.159 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T21:35:06.159 DEBUG:boto:Port: 8000 2023-05-31T21:35:06.160 DEBUG:boto:Params: {} 2023-05-31T21:35:06.160 DEBUG:boto:Token: None 2023-05-31T21:35:06.161 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Wed, 31 May 2023 21:35:06 GMT /pklxkd-74/obj4 2023-05-31T21:35:06.161 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:c7ppOheQsQuZohnZIv8w05F4GeE= 2023-05-31T21:35:06.161 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Wed, 31 May 2023 21:35:06 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:c7ppOheQsQuZohnZIv8w05F4GeE='} 2023-05-31T21:35:06.204 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000000a725ec48b8e3eaa-006477bd8a-2bf5-a1'), ('Date', 'Wed, 31 May 2023 21:35:06 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:35:06.205 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-31T21:35:06.205 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-31T21:35:06.206 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-05-31T21:35:06.255 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2023-05-31T21:35:06.255 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2023-05-31T21:35:06.298 INFO:teuthology.orchestra.run.smithi099.stderr:No --period given, using current period=2e6123d2-d695-4350-9a01-61a455bc782b 2023-05-31T21:35:06.315 INFO:teuthology.orchestra.run.smithi099.stdout:[ 2023-05-31T21:35:06.315 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T21:35:06.315 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1685566501.757183_260.1", 2023-05-31T21:35:06.315 INFO:teuthology.orchestra.run.smithi099.stdout: "last_update": "2023-05-31T20:55:01.757183Z" 2023-05-31T21:35:06.315 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:35:06.316 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T21:35:06.316 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1685566501.735717_251.1", 2023-05-31T21:35:06.316 INFO:teuthology.orchestra.run.smithi099.stdout: "last_update": "2023-05-31T20:55:01.735717Z" 2023-05-31T21:35:06.316 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:35:06.316 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T21:35:06.316 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1685562970.274572_93.1", 2023-05-31T21:35:06.317 INFO:teuthology.orchestra.run.smithi099.stdout: "last_update": "2023-05-31T19:56:10.274572Z" 2023-05-31T21:35:06.317 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:35:06.317 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T21:35:06.317 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1685568813.199723_1066.1", 2023-05-31T21:35:06.317 INFO:teuthology.orchestra.run.smithi099.stdout: "last_update": "2023-05-31T21:33:33.199723Z" 2023-05-31T21:35:06.317 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T21:35:06.317 INFO:teuthology.orchestra.run.smithi099.stdout:] 2023-05-31T21:35:06.318 DEBUG:rgw_multi.tests:master meta markers={0: '1_1685566501.757183_260.1', 1: '1_1685566501.735717_251.1', 2: '1_1685562970.274572_93.1', 3: '1_1685568813.199723_1066.1'} 2023-05-31T21:35:06.319 INFO:rgw_multi.tests:starting meta checkpoint for zone=a2 2023-05-31T21:35:06.319 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-31T21:35:06.319 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-31T21:35:06.320 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-05-31T21:35:06.368 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2023-05-31T21:35:06.369 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2023-05-31T21:35:06.421 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T21:35:06.417+0000 7f0e0d98b780 20 RGW-SYNC:meta: read sync status 2023-05-31T21:35:06.422 INFO:teuthology.orchestra.run.smithi099.stderr:2023-05-31T21:35:06.418+0000 7f0e0d98b780 20 RGW-SYNC:meta: read sync status 2023-05-31T21:35:06.437 INFO:teuthology.orchestra.run.smithi099.stdout:{ 2023-05-31T21:35:06.437 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_status": { 2023-05-31T21:35:06.437 INFO:teuthology.orchestra.run.smithi099.stdout: "info": { 2023-05-31T21:35:06.438 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "sync", 2023-05-31T21:35:06.438 INFO:teuthology.orchestra.run.smithi099.stdout: "num_shards": 4, 2023-05-31T21:35:06.438 INFO:teuthology.orchestra.run.smithi099.stdout: "period": "2e6123d2-d695-4350-9a01-61a455bc782b", 2023-05-31T21:35:06.438 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2023-05-31T21:35:06.438 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:35:06.438 INFO:teuthology.orchestra.run.smithi099.stdout: "markers": [ 2023-05-31T21:35:06.439 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T21:35:06.439 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 0, 2023-05-31T21:35:06.439 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T21:35:06.439 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2023-05-31T21:35:06.439 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1685566501.757183_260.1", 2023-05-31T21:35:06.439 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T21:35:06.439 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 1, 2023-05-31T21:35:06.440 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T21:35:06.440 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2023-05-31T20:55:01.757183Z", 2023-05-31T21:35:06.440 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2023-05-31T21:35:06.440 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T21:35:06.440 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:35:06.440 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T21:35:06.440 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 1, 2023-05-31T21:35:06.441 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T21:35:06.441 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2023-05-31T21:35:06.441 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1685566501.735717_251.1", 2023-05-31T21:35:06.441 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T21:35:06.441 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2023-05-31T21:35:06.441 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T21:35:06.441 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2023-05-31T20:55:01.735717Z", 2023-05-31T21:35:06.442 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2023-05-31T21:35:06.442 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T21:35:06.442 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:35:06.442 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T21:35:06.442 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 2, 2023-05-31T21:35:06.442 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T21:35:06.442 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2023-05-31T21:35:06.443 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1685562970.274572_93.1", 2023-05-31T21:35:06.443 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T21:35:06.443 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2023-05-31T21:35:06.443 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T21:35:06.443 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2023-05-31T19:56:10.274572Z", 2023-05-31T21:35:06.443 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2023-05-31T21:35:06.443 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T21:35:06.444 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:35:06.444 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T21:35:06.444 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 3, 2023-05-31T21:35:06.444 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T21:35:06.444 INFO:teuthology.orchestra.run.smithi099.stdout: "state": 1, 2023-05-31T21:35:06.444 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "1_1685568813.199723_1066.1", 2023-05-31T21:35:06.444 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T21:35:06.445 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2023-05-31T21:35:06.445 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T21:35:06.445 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2023-05-31T21:33:33.199723Z", 2023-05-31T21:35:06.445 INFO:teuthology.orchestra.run.smithi099.stdout: "realm_epoch": 2 2023-05-31T21:35:06.445 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T21:35:06.445 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T21:35:06.445 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2023-05-31T21:35:06.446 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:35:06.446 INFO:teuthology.orchestra.run.smithi099.stdout: "full_sync": { 2023-05-31T21:35:06.446 INFO:teuthology.orchestra.run.smithi099.stdout: "total": 1, 2023-05-31T21:35:06.446 INFO:teuthology.orchestra.run.smithi099.stdout: "complete": 1 2023-05-31T21:35:06.446 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:35:06.446 INFO:teuthology.orchestra.run.smithi099.stdout: "current_time": "2023-05-31T21:35:06Z" 2023-05-31T21:35:06.446 INFO:teuthology.orchestra.run.smithi099.stdout:} 2023-05-31T21:35:06.447 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "2e6123d2-d695-4350-9a01-61a455bc782b", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1685566501.757183_260.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-05-31T20:55:01.757183Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1685566501.735717_251.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-05-31T20:55:01.735717Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1685562970.274572_93.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-05-31T19:56:10.274572Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1685568813.199723_1066.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-05-31T21:33:33.199723Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-05-31T21:35:06Z" } 2023-05-31T21:35:06.448 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1685566501.757183_260.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-05-31T20:55:01.757183Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1685566501.735717_251.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-05-31T20:55:01.735717Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1685562970.274572_93.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-05-31T19:56:10.274572Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1685568813.199723_1066.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-05-31T21:33:33.199723Z', 'realm_epoch': 2}}] 2023-05-31T21:35:06.448 DEBUG:rgw_multi.tests:log_status={0: '1_1685566501.757183_260.1', 1: '1_1685566501.735717_251.1', 2: '1_1685562970.274572_93.1', 3: '1_1685568813.199723_1066.1'} 2023-05-31T21:35:06.448 DEBUG:rgw_multi.tests:sync_status={0: '1_1685566501.757183_260.1', 1: '1_1685566501.735717_251.1', 2: '1_1685562970.274572_93.1', 3: '1_1685568813.199723_1066.1'} 2023-05-31T21:35:06.449 INFO:rgw_multi.tests:finish meta checkpoint for zone=a2 2023-05-31T21:35:06.449 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['datalog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-31T21:35:06.449 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'datalog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-31T21:35:06.450 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 datalog status --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-05-31T21:35:06.497 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2023-05-31T21:35:06.498 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2023-05-31T21:35:06.558 INFO:teuthology.orchestra.run.smithi099.stdout:[ 2023-05-31T21:35:06.558 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T21:35:06.558 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "00000000000000000000:00000000000000045655", 2023-05-31T21:35:06.558 INFO:teuthology.orchestra.run.smithi099.stdout: "last_update": "2023-05-31T21:31:27.421078Z" 2023-05-31T21:35:06.558 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:35:06.559 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T21:35:06.559 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "00000000000000000000:00000000000000032060", 2023-05-31T21:35:06.559 INFO:teuthology.orchestra.run.smithi099.stdout: "last_update": "2023-05-31T21:35:06.166353Z" 2023-05-31T21:35:06.559 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:35:06.559 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T21:35:06.559 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "00000000000000000000:00000000000000052232", 2023-05-31T21:35:06.560 INFO:teuthology.orchestra.run.smithi099.stdout: "last_update": "2023-05-31T21:35:06.146660Z" 2023-05-31T21:35:06.560 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:35:06.560 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T21:35:06.560 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "00000000000000000000:00000000000000044178", 2023-05-31T21:35:06.560 INFO:teuthology.orchestra.run.smithi099.stdout: "last_update": "2023-05-31T21:27:37.215905Z" 2023-05-31T21:35:06.560 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T21:35:06.560 INFO:teuthology.orchestra.run.smithi099.stdout:] 2023-05-31T21:35:06.561 DEBUG:rgw_multi.tests:data markers for zone=a1 markers={0: '00000000000000000000:00000000000000045655', 1: '00000000000000000000:00000000000000032060', 2: '00000000000000000000:00000000000000052232', 3: '00000000000000000000:00000000000000044178'} 2023-05-31T21:35:06.562 INFO:rgw_multi.tests:starting data checkpoint for target_zone=a2 source_zone=a1 2023-05-31T21:35:06.562 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-31T21:35:06.562 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-31T21:35:06.563 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-05-31T21:35:06.613 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2023-05-31T21:35:06.614 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2023-05-31T21:35:06.671 INFO:teuthology.orchestra.run.smithi099.stdout:{ 2023-05-31T21:35:06.671 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_status": { 2023-05-31T21:35:06.671 INFO:teuthology.orchestra.run.smithi099.stdout: "info": { 2023-05-31T21:35:06.672 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "sync", 2023-05-31T21:35:06.672 INFO:teuthology.orchestra.run.smithi099.stdout: "num_shards": 4, 2023-05-31T21:35:06.672 INFO:teuthology.orchestra.run.smithi099.stdout: "instance_id": 452136274422638824 2023-05-31T21:35:06.672 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:35:06.672 INFO:teuthology.orchestra.run.smithi099.stdout: "markers": [ 2023-05-31T21:35:06.672 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T21:35:06.673 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 0, 2023-05-31T21:35:06.673 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T21:35:06.673 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "incremental-sync", 2023-05-31T21:35:06.673 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "00000000000000000000:00000000000000045655", 2023-05-31T21:35:06.673 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T21:35:06.673 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2023-05-31T21:35:06.673 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T21:35:06.674 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2023-05-31T21:31:27.421078Z" 2023-05-31T21:35:06.674 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T21:35:06.674 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:35:06.674 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T21:35:06.674 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 1, 2023-05-31T21:35:06.674 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T21:35:06.675 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "incremental-sync", 2023-05-31T21:35:06.675 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "00000000000000000000:00000000000000031920", 2023-05-31T21:35:06.675 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T21:35:06.675 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2023-05-31T21:35:06.675 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T21:35:06.675 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2023-05-31T21:29:57.583636Z" 2023-05-31T21:35:06.675 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T21:35:06.676 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:35:06.676 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T21:35:06.676 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 2, 2023-05-31T21:35:06.676 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T21:35:06.676 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "incremental-sync", 2023-05-31T21:35:06.676 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "00000000000000000000:00000000000000052091", 2023-05-31T21:35:06.676 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T21:35:06.677 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 55, 2023-05-31T21:35:06.677 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T21:35:06.677 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2023-05-31T21:31:27.385416Z" 2023-05-31T21:35:06.677 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T21:35:06.677 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:35:06.677 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T21:35:06.677 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 3, 2023-05-31T21:35:06.678 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T21:35:06.678 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "incremental-sync", 2023-05-31T21:35:06.678 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "00000000000000000000:00000000000000044178", 2023-05-31T21:35:06.678 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T21:35:06.678 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 9, 2023-05-31T21:35:06.678 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T21:35:06.678 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2023-05-31T21:27:37.215905Z" 2023-05-31T21:35:06.679 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T21:35:06.679 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T21:35:06.679 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2023-05-31T21:35:06.679 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:35:06.679 INFO:teuthology.orchestra.run.smithi099.stdout: "full_sync": { 2023-05-31T21:35:06.679 INFO:teuthology.orchestra.run.smithi099.stdout: "total": 64, 2023-05-31T21:35:06.679 INFO:teuthology.orchestra.run.smithi099.stdout: "complete": 64 2023-05-31T21:35:06.680 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:35:06.680 INFO:teuthology.orchestra.run.smithi099.stdout: "current_time": "2023-05-31T21:35:06Z" 2023-05-31T21:35:06.680 INFO:teuthology.orchestra.run.smithi099.stdout:} 2023-05-31T21:35:06.681 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 452136274422638824 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000045655", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-05-31T21:31:27.421078Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000031920", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-05-31T21:29:57.583636Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000052091", "next_step_marker": "", "total_entries": 55, "pos": 0, "timestamp": "2023-05-31T21:31:27.385416Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000044178", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2023-05-31T21:27:37.215905Z" } } ] }, "full_sync": { "total": 64, "complete": 64 }, "current_time": "2023-05-31T21:35:06Z" } 2023-05-31T21:35:06.681 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000045655', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-05-31T21:31:27.421078Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000031920', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-05-31T21:29:57.583636Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000052091', 'next_step_marker': '', 'total_entries': 55, 'pos': 0, 'timestamp': '2023-05-31T21:31:27.385416Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000044178', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2023-05-31T21:27:37.215905Z'}}] 2023-05-31T21:35:06.682 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000045655', 1: '00000000000000000000:00000000000000032060', 2: '00000000000000000000:00000000000000052232', 3: '00000000000000000000:00000000000000044178'} 2023-05-31T21:35:06.682 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000045655', 1: '00000000000000000000:00000000000000031920', 2: '00000000000000000000:00000000000000052091', 3: '00000000000000000000:00000000000000044178'} 2023-05-31T21:35:06.682 WARNING:rgw_multi.tests:data of zone a2 behind zone a1: shard=1 master=00000000000000000000:00000000000000032060 target=00000000000000000000:00000000000000031920, shard=2 master=00000000000000000000:00000000000000052232 target=00000000000000000000:00000000000000052091 2023-05-31T21:35:11.683 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-31T21:35:11.684 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-31T21:35:11.685 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-05-31T21:35:11.744 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2023-05-31T21:35:11.745 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2023-05-31T21:35:11.824 INFO:teuthology.orchestra.run.smithi099.stdout:{ 2023-05-31T21:35:11.825 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_status": { 2023-05-31T21:35:11.825 INFO:teuthology.orchestra.run.smithi099.stdout: "info": { 2023-05-31T21:35:11.825 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "sync", 2023-05-31T21:35:11.825 INFO:teuthology.orchestra.run.smithi099.stdout: "num_shards": 4, 2023-05-31T21:35:11.825 INFO:teuthology.orchestra.run.smithi099.stdout: "instance_id": 452136274422638824 2023-05-31T21:35:11.825 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:35:11.826 INFO:teuthology.orchestra.run.smithi099.stdout: "markers": [ 2023-05-31T21:35:11.826 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T21:35:11.826 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 0, 2023-05-31T21:35:11.826 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T21:35:11.826 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "incremental-sync", 2023-05-31T21:35:11.826 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "00000000000000000000:00000000000000045655", 2023-05-31T21:35:11.826 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T21:35:11.827 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2023-05-31T21:35:11.827 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T21:35:11.827 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2023-05-31T21:31:27.421078Z" 2023-05-31T21:35:11.827 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T21:35:11.827 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:35:11.827 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T21:35:11.827 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 1, 2023-05-31T21:35:11.828 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T21:35:11.828 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "incremental-sync", 2023-05-31T21:35:11.828 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "00000000000000000000:00000000000000031920", 2023-05-31T21:35:11.828 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T21:35:11.828 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2023-05-31T21:35:11.828 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T21:35:11.829 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2023-05-31T21:29:57.583636Z" 2023-05-31T21:35:11.829 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T21:35:11.829 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:35:11.829 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T21:35:11.829 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 2, 2023-05-31T21:35:11.829 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T21:35:11.829 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "incremental-sync", 2023-05-31T21:35:11.830 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "00000000000000000000:00000000000000052091", 2023-05-31T21:35:11.830 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T21:35:11.830 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 55, 2023-05-31T21:35:11.830 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T21:35:11.830 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2023-05-31T21:31:27.385416Z" 2023-05-31T21:35:11.830 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T21:35:11.830 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:35:11.831 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T21:35:11.831 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 3, 2023-05-31T21:35:11.831 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T21:35:11.831 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "incremental-sync", 2023-05-31T21:35:11.831 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "00000000000000000000:00000000000000044178", 2023-05-31T21:35:11.831 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T21:35:11.831 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 9, 2023-05-31T21:35:11.832 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T21:35:11.832 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2023-05-31T21:27:37.215905Z" 2023-05-31T21:35:11.832 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T21:35:11.832 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T21:35:11.832 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2023-05-31T21:35:11.832 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:35:11.833 INFO:teuthology.orchestra.run.smithi099.stdout: "full_sync": { 2023-05-31T21:35:11.833 INFO:teuthology.orchestra.run.smithi099.stdout: "total": 64, 2023-05-31T21:35:11.833 INFO:teuthology.orchestra.run.smithi099.stdout: "complete": 64 2023-05-31T21:35:11.833 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:35:11.833 INFO:teuthology.orchestra.run.smithi099.stdout: "current_time": "2023-05-31T21:35:11Z" 2023-05-31T21:35:11.833 INFO:teuthology.orchestra.run.smithi099.stdout:} 2023-05-31T21:35:11.834 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 452136274422638824 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000045655", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-05-31T21:31:27.421078Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000031920", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-05-31T21:29:57.583636Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000052091", "next_step_marker": "", "total_entries": 55, "pos": 0, "timestamp": "2023-05-31T21:31:27.385416Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000044178", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2023-05-31T21:27:37.215905Z" } } ] }, "full_sync": { "total": 64, "complete": 64 }, "current_time": "2023-05-31T21:35:11Z" } 2023-05-31T21:35:11.835 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000045655', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-05-31T21:31:27.421078Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000031920', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-05-31T21:29:57.583636Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000052091', 'next_step_marker': '', 'total_entries': 55, 'pos': 0, 'timestamp': '2023-05-31T21:31:27.385416Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000044178', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2023-05-31T21:27:37.215905Z'}}] 2023-05-31T21:35:11.835 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000045655', 1: '00000000000000000000:00000000000000032060', 2: '00000000000000000000:00000000000000052232', 3: '00000000000000000000:00000000000000044178'} 2023-05-31T21:35:11.835 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000045655', 1: '00000000000000000000:00000000000000031920', 2: '00000000000000000000:00000000000000052091', 3: '00000000000000000000:00000000000000044178'} 2023-05-31T21:35:11.836 WARNING:rgw_multi.tests:data of zone a2 behind zone a1: shard=1 master=00000000000000000000:00000000000000032060 target=00000000000000000000:00000000000000031920, shard=2 master=00000000000000000000:00000000000000052232 target=00000000000000000000:00000000000000052091 2023-05-31T21:35:16.837 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-31T21:35:16.837 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-31T21:35:16.838 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-05-31T21:35:16.897 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2023-05-31T21:35:16.897 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2023-05-31T21:35:16.948 INFO:teuthology.orchestra.run.smithi099.stdout:{ 2023-05-31T21:35:16.949 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_status": { 2023-05-31T21:35:16.949 INFO:teuthology.orchestra.run.smithi099.stdout: "info": { 2023-05-31T21:35:16.949 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "sync", 2023-05-31T21:35:16.949 INFO:teuthology.orchestra.run.smithi099.stdout: "num_shards": 4, 2023-05-31T21:35:16.950 INFO:teuthology.orchestra.run.smithi099.stdout: "instance_id": 452136274422638824 2023-05-31T21:35:16.950 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:35:16.950 INFO:teuthology.orchestra.run.smithi099.stdout: "markers": [ 2023-05-31T21:35:16.950 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T21:35:16.950 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 0, 2023-05-31T21:35:16.950 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T21:35:16.950 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "incremental-sync", 2023-05-31T21:35:16.951 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "00000000000000000000:00000000000000045655", 2023-05-31T21:35:16.951 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T21:35:16.951 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2023-05-31T21:35:16.951 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T21:35:16.951 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2023-05-31T21:31:27.421078Z" 2023-05-31T21:35:16.951 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T21:35:16.952 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:35:16.952 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T21:35:16.952 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 1, 2023-05-31T21:35:16.952 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T21:35:16.952 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "incremental-sync", 2023-05-31T21:35:16.953 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "00000000000000000000:00000000000000031920", 2023-05-31T21:35:16.953 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T21:35:16.953 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2023-05-31T21:35:16.953 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T21:35:16.953 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2023-05-31T21:29:57.583636Z" 2023-05-31T21:35:16.953 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T21:35:16.954 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:35:16.954 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T21:35:16.954 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 2, 2023-05-31T21:35:16.954 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T21:35:16.954 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "incremental-sync", 2023-05-31T21:35:16.954 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "00000000000000000000:00000000000000052091", 2023-05-31T21:35:16.954 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T21:35:16.955 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 55, 2023-05-31T21:35:16.955 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T21:35:16.955 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2023-05-31T21:31:27.385416Z" 2023-05-31T21:35:16.955 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T21:35:16.955 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:35:16.955 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T21:35:16.955 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 3, 2023-05-31T21:35:16.956 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T21:35:16.956 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "incremental-sync", 2023-05-31T21:35:16.956 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "00000000000000000000:00000000000000044178", 2023-05-31T21:35:16.956 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T21:35:16.956 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 9, 2023-05-31T21:35:16.956 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T21:35:16.957 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2023-05-31T21:27:37.215905Z" 2023-05-31T21:35:16.957 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T21:35:16.957 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T21:35:16.957 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2023-05-31T21:35:16.958 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:35:16.958 INFO:teuthology.orchestra.run.smithi099.stdout: "full_sync": { 2023-05-31T21:35:16.958 INFO:teuthology.orchestra.run.smithi099.stdout: "total": 64, 2023-05-31T21:35:16.958 INFO:teuthology.orchestra.run.smithi099.stdout: "complete": 64 2023-05-31T21:35:16.958 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:35:16.958 INFO:teuthology.orchestra.run.smithi099.stdout: "current_time": "2023-05-31T21:35:16Z" 2023-05-31T21:35:16.958 INFO:teuthology.orchestra.run.smithi099.stdout:} 2023-05-31T21:35:16.959 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 452136274422638824 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000045655", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-05-31T21:31:27.421078Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000031920", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-05-31T21:29:57.583636Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000052091", "next_step_marker": "", "total_entries": 55, "pos": 0, "timestamp": "2023-05-31T21:31:27.385416Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000044178", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2023-05-31T21:27:37.215905Z" } } ] }, "full_sync": { "total": 64, "complete": 64 }, "current_time": "2023-05-31T21:35:16Z" } 2023-05-31T21:35:16.960 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000045655', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-05-31T21:31:27.421078Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000031920', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-05-31T21:29:57.583636Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000052091', 'next_step_marker': '', 'total_entries': 55, 'pos': 0, 'timestamp': '2023-05-31T21:31:27.385416Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000044178', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2023-05-31T21:27:37.215905Z'}}] 2023-05-31T21:35:16.961 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000045655', 1: '00000000000000000000:00000000000000032060', 2: '00000000000000000000:00000000000000052232', 3: '00000000000000000000:00000000000000044178'} 2023-05-31T21:35:16.961 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000045655', 1: '00000000000000000000:00000000000000031920', 2: '00000000000000000000:00000000000000052091', 3: '00000000000000000000:00000000000000044178'} 2023-05-31T21:35:16.961 WARNING:rgw_multi.tests:data of zone a2 behind zone a1: shard=1 master=00000000000000000000:00000000000000032060 target=00000000000000000000:00000000000000031920, shard=2 master=00000000000000000000:00000000000000052232 target=00000000000000000000:00000000000000052091 2023-05-31T21:35:21.962 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-31T21:35:21.963 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-31T21:35:21.963 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-05-31T21:35:22.022 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2023-05-31T21:35:22.023 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2023-05-31T21:35:22.093 INFO:teuthology.orchestra.run.smithi099.stdout:{ 2023-05-31T21:35:22.093 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_status": { 2023-05-31T21:35:22.093 INFO:teuthology.orchestra.run.smithi099.stdout: "info": { 2023-05-31T21:35:22.093 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "sync", 2023-05-31T21:35:22.094 INFO:teuthology.orchestra.run.smithi099.stdout: "num_shards": 4, 2023-05-31T21:35:22.094 INFO:teuthology.orchestra.run.smithi099.stdout: "instance_id": 452136274422638824 2023-05-31T21:35:22.094 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:35:22.094 INFO:teuthology.orchestra.run.smithi099.stdout: "markers": [ 2023-05-31T21:35:22.094 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T21:35:22.094 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 0, 2023-05-31T21:35:22.095 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T21:35:22.095 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "incremental-sync", 2023-05-31T21:35:22.095 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "00000000000000000000:00000000000000045655", 2023-05-31T21:35:22.095 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T21:35:22.095 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2023-05-31T21:35:22.095 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T21:35:22.096 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2023-05-31T21:31:27.421078Z" 2023-05-31T21:35:22.096 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T21:35:22.096 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:35:22.096 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T21:35:22.096 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 1, 2023-05-31T21:35:22.096 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T21:35:22.097 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "incremental-sync", 2023-05-31T21:35:22.097 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "00000000000000000000:00000000000000031920", 2023-05-31T21:35:22.097 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T21:35:22.097 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2023-05-31T21:35:22.097 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T21:35:22.097 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2023-05-31T21:29:57.583636Z" 2023-05-31T21:35:22.098 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T21:35:22.098 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:35:22.098 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T21:35:22.098 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 2, 2023-05-31T21:35:22.098 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T21:35:22.098 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "incremental-sync", 2023-05-31T21:35:22.098 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "00000000000000000000:00000000000000052091", 2023-05-31T21:35:22.099 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T21:35:22.099 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 55, 2023-05-31T21:35:22.099 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T21:35:22.099 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2023-05-31T21:31:27.385416Z" 2023-05-31T21:35:22.099 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T21:35:22.099 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:35:22.100 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T21:35:22.100 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 3, 2023-05-31T21:35:22.100 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T21:35:22.100 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "incremental-sync", 2023-05-31T21:35:22.100 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "00000000000000000000:00000000000000044178", 2023-05-31T21:35:22.100 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T21:35:22.101 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 9, 2023-05-31T21:35:22.101 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T21:35:22.101 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2023-05-31T21:27:37.215905Z" 2023-05-31T21:35:22.101 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T21:35:22.101 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T21:35:22.101 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2023-05-31T21:35:22.101 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:35:22.102 INFO:teuthology.orchestra.run.smithi099.stdout: "full_sync": { 2023-05-31T21:35:22.102 INFO:teuthology.orchestra.run.smithi099.stdout: "total": 64, 2023-05-31T21:35:22.102 INFO:teuthology.orchestra.run.smithi099.stdout: "complete": 64 2023-05-31T21:35:22.102 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:35:22.102 INFO:teuthology.orchestra.run.smithi099.stdout: "current_time": "2023-05-31T21:35:22Z" 2023-05-31T21:35:22.102 INFO:teuthology.orchestra.run.smithi099.stdout:} 2023-05-31T21:35:22.103 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 452136274422638824 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000045655", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-05-31T21:31:27.421078Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000031920", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-05-31T21:29:57.583636Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000052091", "next_step_marker": "", "total_entries": 55, "pos": 0, "timestamp": "2023-05-31T21:31:27.385416Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000044178", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2023-05-31T21:27:37.215905Z" } } ] }, "full_sync": { "total": 64, "complete": 64 }, "current_time": "2023-05-31T21:35:22Z" } 2023-05-31T21:35:22.104 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000045655', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-05-31T21:31:27.421078Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000031920', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-05-31T21:29:57.583636Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000052091', 'next_step_marker': '', 'total_entries': 55, 'pos': 0, 'timestamp': '2023-05-31T21:31:27.385416Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000044178', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2023-05-31T21:27:37.215905Z'}}] 2023-05-31T21:35:22.104 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000045655', 1: '00000000000000000000:00000000000000032060', 2: '00000000000000000000:00000000000000052232', 3: '00000000000000000000:00000000000000044178'} 2023-05-31T21:35:22.105 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000045655', 1: '00000000000000000000:00000000000000031920', 2: '00000000000000000000:00000000000000052091', 3: '00000000000000000000:00000000000000044178'} 2023-05-31T21:35:22.105 WARNING:rgw_multi.tests:data of zone a2 behind zone a1: shard=1 master=00000000000000000000:00000000000000032060 target=00000000000000000000:00000000000000031920, shard=2 master=00000000000000000000:00000000000000052232 target=00000000000000000000:00000000000000052091 2023-05-31T21:35:27.106 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-31T21:35:27.106 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-31T21:35:27.107 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-05-31T21:35:27.166 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2023-05-31T21:35:27.166 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2023-05-31T21:35:27.234 INFO:teuthology.orchestra.run.smithi099.stdout:{ 2023-05-31T21:35:27.235 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_status": { 2023-05-31T21:35:27.235 INFO:teuthology.orchestra.run.smithi099.stdout: "info": { 2023-05-31T21:35:27.235 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "sync", 2023-05-31T21:35:27.235 INFO:teuthology.orchestra.run.smithi099.stdout: "num_shards": 4, 2023-05-31T21:35:27.235 INFO:teuthology.orchestra.run.smithi099.stdout: "instance_id": 452136274422638824 2023-05-31T21:35:27.235 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:35:27.235 INFO:teuthology.orchestra.run.smithi099.stdout: "markers": [ 2023-05-31T21:35:27.236 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T21:35:27.236 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 0, 2023-05-31T21:35:27.236 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T21:35:27.236 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "incremental-sync", 2023-05-31T21:35:27.236 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "00000000000000000000:00000000000000045655", 2023-05-31T21:35:27.236 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T21:35:27.237 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2023-05-31T21:35:27.237 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T21:35:27.237 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2023-05-31T21:31:27.421078Z" 2023-05-31T21:35:27.237 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T21:35:27.237 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:35:27.237 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T21:35:27.237 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 1, 2023-05-31T21:35:27.238 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T21:35:27.238 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "incremental-sync", 2023-05-31T21:35:27.238 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "00000000000000000000:00000000000000031920", 2023-05-31T21:35:27.238 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T21:35:27.238 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2023-05-31T21:35:27.238 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T21:35:27.238 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2023-05-31T21:29:57.583636Z" 2023-05-31T21:35:27.239 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T21:35:27.239 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:35:27.239 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T21:35:27.239 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 2, 2023-05-31T21:35:27.239 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T21:35:27.239 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "incremental-sync", 2023-05-31T21:35:27.239 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "00000000000000000000:00000000000000052091", 2023-05-31T21:35:27.240 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T21:35:27.240 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 55, 2023-05-31T21:35:27.240 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T21:35:27.240 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2023-05-31T21:31:27.385416Z" 2023-05-31T21:35:27.240 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T21:35:27.240 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:35:27.240 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T21:35:27.241 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 3, 2023-05-31T21:35:27.241 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T21:35:27.241 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "incremental-sync", 2023-05-31T21:35:27.241 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "00000000000000000000:00000000000000044178", 2023-05-31T21:35:27.241 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T21:35:27.241 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 9, 2023-05-31T21:35:27.241 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T21:35:27.242 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2023-05-31T21:27:37.215905Z" 2023-05-31T21:35:27.242 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T21:35:27.242 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T21:35:27.242 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2023-05-31T21:35:27.242 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:35:27.242 INFO:teuthology.orchestra.run.smithi099.stdout: "full_sync": { 2023-05-31T21:35:27.242 INFO:teuthology.orchestra.run.smithi099.stdout: "total": 64, 2023-05-31T21:35:27.243 INFO:teuthology.orchestra.run.smithi099.stdout: "complete": 64 2023-05-31T21:35:27.243 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:35:27.243 INFO:teuthology.orchestra.run.smithi099.stdout: "current_time": "2023-05-31T21:35:27Z" 2023-05-31T21:35:27.243 INFO:teuthology.orchestra.run.smithi099.stdout:} 2023-05-31T21:35:27.244 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 452136274422638824 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000045655", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-05-31T21:31:27.421078Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000031920", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-05-31T21:29:57.583636Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000052091", "next_step_marker": "", "total_entries": 55, "pos": 0, "timestamp": "2023-05-31T21:31:27.385416Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000044178", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2023-05-31T21:27:37.215905Z" } } ] }, "full_sync": { "total": 64, "complete": 64 }, "current_time": "2023-05-31T21:35:27Z" } 2023-05-31T21:35:27.244 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000045655', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-05-31T21:31:27.421078Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000031920', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-05-31T21:29:57.583636Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000052091', 'next_step_marker': '', 'total_entries': 55, 'pos': 0, 'timestamp': '2023-05-31T21:31:27.385416Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000044178', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2023-05-31T21:27:37.215905Z'}}] 2023-05-31T21:35:27.245 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000045655', 1: '00000000000000000000:00000000000000032060', 2: '00000000000000000000:00000000000000052232', 3: '00000000000000000000:00000000000000044178'} 2023-05-31T21:35:27.245 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000045655', 1: '00000000000000000000:00000000000000031920', 2: '00000000000000000000:00000000000000052091', 3: '00000000000000000000:00000000000000044178'} 2023-05-31T21:35:27.246 WARNING:rgw_multi.tests:data of zone a2 behind zone a1: shard=1 master=00000000000000000000:00000000000000032060 target=00000000000000000000:00000000000000031920, shard=2 master=00000000000000000000:00000000000000052232 target=00000000000000000000:00000000000000052091 2023-05-31T21:35:32.246 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-31T21:35:32.247 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-31T21:35:32.248 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-05-31T21:35:32.307 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2023-05-31T21:35:32.307 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2023-05-31T21:35:32.370 INFO:teuthology.orchestra.run.smithi099.stdout:{ 2023-05-31T21:35:32.371 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_status": { 2023-05-31T21:35:32.371 INFO:teuthology.orchestra.run.smithi099.stdout: "info": { 2023-05-31T21:35:32.371 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "sync", 2023-05-31T21:35:32.371 INFO:teuthology.orchestra.run.smithi099.stdout: "num_shards": 4, 2023-05-31T21:35:32.371 INFO:teuthology.orchestra.run.smithi099.stdout: "instance_id": 452136274422638824 2023-05-31T21:35:32.371 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:35:32.372 INFO:teuthology.orchestra.run.smithi099.stdout: "markers": [ 2023-05-31T21:35:32.372 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T21:35:32.372 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 0, 2023-05-31T21:35:32.372 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T21:35:32.372 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "incremental-sync", 2023-05-31T21:35:32.372 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "00000000000000000000:00000000000000045655", 2023-05-31T21:35:32.373 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T21:35:32.373 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2023-05-31T21:35:32.373 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T21:35:32.373 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2023-05-31T21:31:27.421078Z" 2023-05-31T21:35:32.373 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T21:35:32.373 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:35:32.373 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T21:35:32.374 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 1, 2023-05-31T21:35:32.374 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T21:35:32.374 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "incremental-sync", 2023-05-31T21:35:32.374 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "00000000000000000000:00000000000000031920", 2023-05-31T21:35:32.374 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T21:35:32.374 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2023-05-31T21:35:32.374 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T21:35:32.375 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2023-05-31T21:29:57.583636Z" 2023-05-31T21:35:32.375 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T21:35:32.375 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:35:32.375 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T21:35:32.375 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 2, 2023-05-31T21:35:32.375 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T21:35:32.375 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "incremental-sync", 2023-05-31T21:35:32.376 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "00000000000000000000:00000000000000052091", 2023-05-31T21:35:32.376 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T21:35:32.376 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 55, 2023-05-31T21:35:32.376 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T21:35:32.376 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2023-05-31T21:31:27.385416Z" 2023-05-31T21:35:32.376 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T21:35:32.376 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:35:32.377 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T21:35:32.377 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 3, 2023-05-31T21:35:32.377 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T21:35:32.377 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "incremental-sync", 2023-05-31T21:35:32.377 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "00000000000000000000:00000000000000044178", 2023-05-31T21:35:32.377 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T21:35:32.377 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 9, 2023-05-31T21:35:32.378 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T21:35:32.378 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2023-05-31T21:27:37.215905Z" 2023-05-31T21:35:32.378 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T21:35:32.378 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T21:35:32.378 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2023-05-31T21:35:32.378 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:35:32.378 INFO:teuthology.orchestra.run.smithi099.stdout: "full_sync": { 2023-05-31T21:35:32.379 INFO:teuthology.orchestra.run.smithi099.stdout: "total": 64, 2023-05-31T21:35:32.379 INFO:teuthology.orchestra.run.smithi099.stdout: "complete": 64 2023-05-31T21:35:32.379 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:35:32.379 INFO:teuthology.orchestra.run.smithi099.stdout: "current_time": "2023-05-31T21:35:32Z" 2023-05-31T21:35:32.379 INFO:teuthology.orchestra.run.smithi099.stdout:} 2023-05-31T21:35:32.380 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 452136274422638824 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000045655", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-05-31T21:31:27.421078Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000031920", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-05-31T21:29:57.583636Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000052091", "next_step_marker": "", "total_entries": 55, "pos": 0, "timestamp": "2023-05-31T21:31:27.385416Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000044178", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2023-05-31T21:27:37.215905Z" } } ] }, "full_sync": { "total": 64, "complete": 64 }, "current_time": "2023-05-31T21:35:32Z" } 2023-05-31T21:35:32.380 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000045655', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-05-31T21:31:27.421078Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000031920', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-05-31T21:29:57.583636Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000052091', 'next_step_marker': '', 'total_entries': 55, 'pos': 0, 'timestamp': '2023-05-31T21:31:27.385416Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000044178', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2023-05-31T21:27:37.215905Z'}}] 2023-05-31T21:35:32.381 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000045655', 1: '00000000000000000000:00000000000000032060', 2: '00000000000000000000:00000000000000052232', 3: '00000000000000000000:00000000000000044178'} 2023-05-31T21:35:32.381 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000045655', 1: '00000000000000000000:00000000000000031920', 2: '00000000000000000000:00000000000000052091', 3: '00000000000000000000:00000000000000044178'} 2023-05-31T21:35:32.381 WARNING:rgw_multi.tests:data of zone a2 behind zone a1: shard=1 master=00000000000000000000:00000000000000032060 target=00000000000000000000:00000000000000031920, shard=2 master=00000000000000000000:00000000000000052232 target=00000000000000000000:00000000000000052091 2023-05-31T21:35:37.382 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-31T21:35:37.383 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-31T21:35:37.383 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-05-31T21:35:37.442 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2023-05-31T21:35:37.442 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2023-05-31T21:35:37.502 INFO:teuthology.orchestra.run.smithi099.stdout:{ 2023-05-31T21:35:37.502 INFO:teuthology.orchestra.run.smithi099.stdout: "sync_status": { 2023-05-31T21:35:37.502 INFO:teuthology.orchestra.run.smithi099.stdout: "info": { 2023-05-31T21:35:37.502 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "sync", 2023-05-31T21:35:37.502 INFO:teuthology.orchestra.run.smithi099.stdout: "num_shards": 4, 2023-05-31T21:35:37.503 INFO:teuthology.orchestra.run.smithi099.stdout: "instance_id": 452136274422638824 2023-05-31T21:35:37.503 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:35:37.503 INFO:teuthology.orchestra.run.smithi099.stdout: "markers": [ 2023-05-31T21:35:37.503 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T21:35:37.503 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 0, 2023-05-31T21:35:37.503 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T21:35:37.504 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "incremental-sync", 2023-05-31T21:35:37.504 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "00000000000000000000:00000000000000045655", 2023-05-31T21:35:37.504 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T21:35:37.504 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2023-05-31T21:35:37.504 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T21:35:37.504 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2023-05-31T21:31:27.421078Z" 2023-05-31T21:35:37.505 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T21:35:37.505 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:35:37.505 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T21:35:37.505 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 1, 2023-05-31T21:35:37.505 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T21:35:37.505 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "incremental-sync", 2023-05-31T21:35:37.505 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "00000000000000000000:00000000000000032060", 2023-05-31T21:35:37.506 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T21:35:37.506 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 0, 2023-05-31T21:35:37.506 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T21:35:37.506 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2023-05-31T21:35:06.166353Z" 2023-05-31T21:35:37.506 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T21:35:37.506 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:35:37.506 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T21:35:37.507 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 2, 2023-05-31T21:35:37.507 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T21:35:37.507 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "incremental-sync", 2023-05-31T21:35:37.507 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "00000000000000000000:00000000000000052232", 2023-05-31T21:35:37.507 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T21:35:37.507 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 55, 2023-05-31T21:35:37.507 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T21:35:37.508 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2023-05-31T21:35:06.146660Z" 2023-05-31T21:35:37.508 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T21:35:37.508 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:35:37.508 INFO:teuthology.orchestra.run.smithi099.stdout: { 2023-05-31T21:35:37.508 INFO:teuthology.orchestra.run.smithi099.stdout: "key": 3, 2023-05-31T21:35:37.508 INFO:teuthology.orchestra.run.smithi099.stdout: "val": { 2023-05-31T21:35:37.509 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "incremental-sync", 2023-05-31T21:35:37.509 INFO:teuthology.orchestra.run.smithi099.stdout: "marker": "00000000000000000000:00000000000000044178", 2023-05-31T21:35:37.509 INFO:teuthology.orchestra.run.smithi099.stdout: "next_step_marker": "", 2023-05-31T21:35:37.509 INFO:teuthology.orchestra.run.smithi099.stdout: "total_entries": 9, 2023-05-31T21:35:37.509 INFO:teuthology.orchestra.run.smithi099.stdout: "pos": 0, 2023-05-31T21:35:37.510 INFO:teuthology.orchestra.run.smithi099.stdout: "timestamp": "2023-05-31T21:27:37.215905Z" 2023-05-31T21:35:37.510 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T21:35:37.510 INFO:teuthology.orchestra.run.smithi099.stdout: } 2023-05-31T21:35:37.510 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2023-05-31T21:35:37.510 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:35:37.510 INFO:teuthology.orchestra.run.smithi099.stdout: "full_sync": { 2023-05-31T21:35:37.511 INFO:teuthology.orchestra.run.smithi099.stdout: "total": 64, 2023-05-31T21:35:37.511 INFO:teuthology.orchestra.run.smithi099.stdout: "complete": 64 2023-05-31T21:35:37.511 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2023-05-31T21:35:37.511 INFO:teuthology.orchestra.run.smithi099.stdout: "current_time": "2023-05-31T21:35:37Z" 2023-05-31T21:35:37.511 INFO:teuthology.orchestra.run.smithi099.stdout:} 2023-05-31T21:35:37.512 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 452136274422638824 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000045655", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-05-31T21:31:27.421078Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000032060", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-05-31T21:35:06.166353Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000052232", "next_step_marker": "", "total_entries": 55, "pos": 0, "timestamp": "2023-05-31T21:35:06.146660Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000044178", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2023-05-31T21:27:37.215905Z" } } ] }, "full_sync": { "total": 64, "complete": 64 }, "current_time": "2023-05-31T21:35:37Z" } 2023-05-31T21:35:37.512 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000045655', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-05-31T21:31:27.421078Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000032060', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-05-31T21:35:06.166353Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000052232', 'next_step_marker': '', 'total_entries': 55, 'pos': 0, 'timestamp': '2023-05-31T21:35:06.146660Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000044178', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2023-05-31T21:27:37.215905Z'}}] 2023-05-31T21:35:37.513 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000045655', 1: '00000000000000000000:00000000000000032060', 2: '00000000000000000000:00000000000000052232', 3: '00000000000000000000:00000000000000044178'} 2023-05-31T21:35:37.513 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000045655', 1: '00000000000000000000:00000000000000032060', 2: '00000000000000000000:00000000000000052232', 3: '00000000000000000000:00000000000000044178'} 2023-05-31T21:35:37.513 INFO:rgw_multi.tests:finished data checkpoint for target_zone=a2 source_zone=a1 2023-05-31T21:35:37.514 DEBUG:boto:path=/pklxkd-74/ 2023-05-31T21:35:37.514 DEBUG:boto:auth_path=/pklxkd-74/ 2023-05-31T21:35:37.515 DEBUG:boto:Method: HEAD 2023-05-31T21:35:37.515 DEBUG:boto:Path: /pklxkd-74/ 2023-05-31T21:35:37.515 DEBUG:boto:Data: 2023-05-31T21:35:37.515 DEBUG:boto:Headers: {} 2023-05-31T21:35:37.516 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T21:35:37.516 DEBUG:boto:Port: 8001 2023-05-31T21:35:37.517 DEBUG:boto:Params: {} 2023-05-31T21:35:37.517 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8001} 2023-05-31T21:35:37.517 DEBUG:boto:Token: None 2023-05-31T21:35:37.518 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 21:35:37 GMT /pklxkd-74/ 2023-05-31T21:35:37.518 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:LYYUApE5mTkeZea34f7PvgPfOVw= 2023-05-31T21:35:37.519 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 21:35:37 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:LYYUApE5mTkeZea34f7PvgPfOVw=', 'Content-Length': '0'} 2023-05-31T21:35:37.522 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '4'), ('X-RGW-Bytes-Used', '24'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000004bf73900f7a351ce-006477bda9-2bfc-a2'), ('Content-Length', '0'), ('Date', 'Wed, 31 May 2023 21:35:37 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:35:37.523 DEBUG:boto:path=/pklxkd-74/obj3 2023-05-31T21:35:37.523 DEBUG:boto:auth_path=/pklxkd-74/obj3 2023-05-31T21:35:37.524 DEBUG:boto:Method: HEAD 2023-05-31T21:35:37.524 DEBUG:boto:Path: /pklxkd-74/obj3 2023-05-31T21:35:37.524 DEBUG:boto:Data: 2023-05-31T21:35:37.525 DEBUG:boto:Headers: {} 2023-05-31T21:35:37.525 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T21:35:37.525 DEBUG:boto:Port: 8001 2023-05-31T21:35:37.526 DEBUG:boto:Params: {} 2023-05-31T21:35:37.526 DEBUG:boto:Token: None 2023-05-31T21:35:37.527 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 21:35:37 GMT /pklxkd-74/obj3 2023-05-31T21:35:37.527 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:5bvoAPkk8A90l7ANHSPEbz6sTtE= 2023-05-31T21:35:37.527 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 21:35:37 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:5bvoAPkk8A90l7ANHSPEbz6sTtE=', 'Content-Length': '0'} 2023-05-31T21:35:37.529 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 21:35:06 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000030585567ad3d5b4d-006477bda9-2bfc-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 21:35:37 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:35:37.530 DEBUG:boto:path=/pklxkd-74/obj3 2023-05-31T21:35:37.530 DEBUG:boto:auth_path=/pklxkd-74/obj3 2023-05-31T21:35:37.531 DEBUG:boto:Method: GET 2023-05-31T21:35:37.531 DEBUG:boto:Path: /pklxkd-74/obj3 2023-05-31T21:35:37.531 DEBUG:boto:Data: 2023-05-31T21:35:37.532 DEBUG:boto:Headers: {} 2023-05-31T21:35:37.532 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T21:35:37.532 DEBUG:boto:Port: 8001 2023-05-31T21:35:37.533 DEBUG:boto:Params: {} 2023-05-31T21:35:37.533 DEBUG:boto:Token: None 2023-05-31T21:35:37.534 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 21:35:37 GMT /pklxkd-74/obj3 2023-05-31T21:35:37.534 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:AjLSjJxNob46+lcZkyeKRPLJx00= 2023-05-31T21:35:37.534 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 21:35:37 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:AjLSjJxNob46+lcZkyeKRPLJx00=', 'Content-Length': '0'} 2023-05-31T21:35:37.536 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 21:35:06 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000302ff04975d1c678-006477bda9-2bfc-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 21:35:37 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:35:37.537 DEBUG:boto:path=/pklxkd-74/obj4 2023-05-31T21:35:37.537 DEBUG:boto:auth_path=/pklxkd-74/obj4 2023-05-31T21:35:37.538 DEBUG:boto:Method: HEAD 2023-05-31T21:35:37.538 DEBUG:boto:Path: /pklxkd-74/obj4 2023-05-31T21:35:37.538 DEBUG:boto:Data: 2023-05-31T21:35:37.539 DEBUG:boto:Headers: {} 2023-05-31T21:35:37.539 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T21:35:37.539 DEBUG:boto:Port: 8001 2023-05-31T21:35:37.540 DEBUG:boto:Params: {} 2023-05-31T21:35:37.540 DEBUG:boto:Token: None 2023-05-31T21:35:37.540 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 21:35:37 GMT /pklxkd-74/obj4 2023-05-31T21:35:37.541 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:BMNHda3/WTKxK+etrGCfRnKHa88= 2023-05-31T21:35:37.541 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 21:35:37 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:BMNHda3/WTKxK+etrGCfRnKHa88=', 'Content-Length': '0'} 2023-05-31T21:35:37.543 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 21:35:06 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000094e9e8043d59c317-006477bda9-2bfc-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 21:35:37 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:35:37.543 DEBUG:boto:path=/pklxkd-74/obj4 2023-05-31T21:35:37.544 DEBUG:boto:auth_path=/pklxkd-74/obj4 2023-05-31T21:35:37.544 DEBUG:boto:Method: GET 2023-05-31T21:35:37.544 DEBUG:boto:Path: /pklxkd-74/obj4 2023-05-31T21:35:37.545 DEBUG:boto:Data: 2023-05-31T21:35:37.545 DEBUG:boto:Headers: {} 2023-05-31T21:35:37.545 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T21:35:37.546 DEBUG:boto:Port: 8001 2023-05-31T21:35:37.546 DEBUG:boto:Params: {} 2023-05-31T21:35:37.546 DEBUG:boto:Token: None 2023-05-31T21:35:37.547 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 21:35:37 GMT /pklxkd-74/obj4 2023-05-31T21:35:37.547 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:dG9g2evP5ph5hlAsHQ1dhRhaTTM= 2023-05-31T21:35:37.548 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 21:35:37 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:dG9g2evP5ph5hlAsHQ1dhRhaTTM=', 'Content-Length': '0'} 2023-05-31T21:35:37.549 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 21:35:06 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000493edfa3589e5fd7-006477bda9-2bfc-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 21:35:37 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:35:37.550 DEBUG:boto:path=/pklxkd-75/ 2023-05-31T21:35:37.550 DEBUG:boto:auth_path=/pklxkd-75/ 2023-05-31T21:35:37.551 DEBUG:boto:Method: HEAD 2023-05-31T21:35:37.551 DEBUG:boto:Path: /pklxkd-75/ 2023-05-31T21:35:37.551 DEBUG:boto:Data: 2023-05-31T21:35:37.552 DEBUG:boto:Headers: {} 2023-05-31T21:35:37.552 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T21:35:37.552 DEBUG:boto:Port: 8001 2023-05-31T21:35:37.553 DEBUG:boto:Params: {} 2023-05-31T21:35:37.553 DEBUG:boto:Token: None 2023-05-31T21:35:37.553 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 21:35:37 GMT /pklxkd-75/ 2023-05-31T21:35:37.554 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:qqQFJYYxdBtB/8zUkVN2oE4QpNk= 2023-05-31T21:35:37.554 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 21:35:37 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:qqQFJYYxdBtB/8zUkVN2oE4QpNk=', 'Content-Length': '0'} 2023-05-31T21:35:37.556 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '4'), ('X-RGW-Bytes-Used', '24'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000008e5bd7eba60c4a23-006477bda9-2bfc-a2'), ('Content-Length', '0'), ('Date', 'Wed, 31 May 2023 21:35:37 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:35:37.556 DEBUG:boto:path=/pklxkd-75/obj3 2023-05-31T21:35:37.557 DEBUG:boto:auth_path=/pklxkd-75/obj3 2023-05-31T21:35:37.557 DEBUG:boto:Method: HEAD 2023-05-31T21:35:37.558 DEBUG:boto:Path: /pklxkd-75/obj3 2023-05-31T21:35:37.558 DEBUG:boto:Data: 2023-05-31T21:35:37.558 DEBUG:boto:Headers: {} 2023-05-31T21:35:37.559 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T21:35:37.559 DEBUG:boto:Port: 8001 2023-05-31T21:35:37.559 DEBUG:boto:Params: {} 2023-05-31T21:35:37.560 DEBUG:boto:Token: None 2023-05-31T21:35:37.560 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 21:35:37 GMT /pklxkd-75/obj3 2023-05-31T21:35:37.560 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:4UNUX9zBOiia+rPwg+oeZKIiAmo= 2023-05-31T21:35:37.561 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 21:35:37 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:4UNUX9zBOiia+rPwg+oeZKIiAmo=', 'Content-Length': '0'} 2023-05-31T21:35:37.562 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 21:35:06 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000044969d97662f1f98-006477bda9-2bfc-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 21:35:37 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:35:37.563 DEBUG:boto:path=/pklxkd-75/obj3 2023-05-31T21:35:37.563 DEBUG:boto:auth_path=/pklxkd-75/obj3 2023-05-31T21:35:37.563 DEBUG:boto:Method: GET 2023-05-31T21:35:37.564 DEBUG:boto:Path: /pklxkd-75/obj3 2023-05-31T21:35:37.564 DEBUG:boto:Data: 2023-05-31T21:35:37.564 DEBUG:boto:Headers: {} 2023-05-31T21:35:37.565 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T21:35:37.565 DEBUG:boto:Port: 8001 2023-05-31T21:35:37.566 DEBUG:boto:Params: {} 2023-05-31T21:35:37.566 DEBUG:boto:Token: None 2023-05-31T21:35:37.566 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 21:35:37 GMT /pklxkd-75/obj3 2023-05-31T21:35:37.567 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:atygdA2GLsaaPqHKl1DAqfmNuY4= 2023-05-31T21:35:37.567 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 21:35:37 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:atygdA2GLsaaPqHKl1DAqfmNuY4=', 'Content-Length': '0'} 2023-05-31T21:35:37.569 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 21:35:06 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000cc0761b6c1260405-006477bda9-2bfc-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 21:35:37 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:35:37.569 DEBUG:boto:path=/pklxkd-75/obj4 2023-05-31T21:35:37.570 DEBUG:boto:auth_path=/pklxkd-75/obj4 2023-05-31T21:35:37.570 DEBUG:boto:Method: HEAD 2023-05-31T21:35:37.570 DEBUG:boto:Path: /pklxkd-75/obj4 2023-05-31T21:35:37.571 DEBUG:boto:Data: 2023-05-31T21:35:37.571 DEBUG:boto:Headers: {} 2023-05-31T21:35:37.571 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T21:35:37.572 DEBUG:boto:Port: 8001 2023-05-31T21:35:37.572 DEBUG:boto:Params: {} 2023-05-31T21:35:37.572 DEBUG:boto:Token: None 2023-05-31T21:35:37.573 DEBUG:boto:StringToSign: HEAD Wed, 31 May 2023 21:35:37 GMT /pklxkd-75/obj4 2023-05-31T21:35:37.573 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:vTIZU7LQdl+Mz1QbbuhFxevvVD4= 2023-05-31T21:35:37.573 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 21:35:37 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:vTIZU7LQdl+Mz1QbbuhFxevvVD4=', 'Content-Length': '0'} 2023-05-31T21:35:37.575 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 21:35:06 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000075b72903c0f0a825-006477bda9-2bfc-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 21:35:37 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:35:37.575 DEBUG:boto:path=/pklxkd-75/obj4 2023-05-31T21:35:37.576 DEBUG:boto:auth_path=/pklxkd-75/obj4 2023-05-31T21:35:37.576 DEBUG:boto:Method: GET 2023-05-31T21:35:37.577 DEBUG:boto:Path: /pklxkd-75/obj4 2023-05-31T21:35:37.577 DEBUG:boto:Data: 2023-05-31T21:35:37.577 DEBUG:boto:Headers: {} 2023-05-31T21:35:37.578 DEBUG:boto:Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T21:35:37.578 DEBUG:boto:Port: 8001 2023-05-31T21:35:37.578 DEBUG:boto:Params: {} 2023-05-31T21:35:37.579 DEBUG:boto:Token: None 2023-05-31T21:35:37.579 DEBUG:boto:StringToSign: GET Wed, 31 May 2023 21:35:37 GMT /pklxkd-75/obj4 2023-05-31T21:35:37.580 DEBUG:boto:Signature: AWS Q80H11OLNU1S48PPYL26:XK5ei+wlwXlaTpCPjaJDth5z3to= 2023-05-31T21:35:37.580 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 21:35:37 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:XK5ei+wlwXlaTpCPjaJDth5z3to=', 'Content-Length': '0'} 2023-05-31T21:35:37.581 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 21:35:06 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000aba01a09899d207a-006477bda9-2bfc-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 21:35:37 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:35:37.624 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'group', 'remove', '--group-id', 'sync-bucket', '--bucket', 'pklxkd-74', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-05-31T21:35:37.625 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'group', 'remove', '--group-id', 'sync-bucket', '--bucket', 'pklxkd-74', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-05-31T21:35:37.625 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync group remove --group-id sync-bucket --bucket pklxkd-74 --cluster c1 --debug-rgw 0 --debug-ms 0 2023-05-31T21:35:37.676 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2023-05-31T21:35:37.676 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2023-05-31T21:35:37.772 INFO:teuthology.orchestra.run.smithi099.stdout:{"sync_policy":{"groups":[]}} 2023-05-31T21:35:37.773 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'group', 'remove', '--group-id', 'sync-group', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-05-31T21:35:37.774 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'group', 'remove', '--group-id', 'sync-group', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-05-31T21:35:37.775 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync group remove --group-id sync-group --cluster c1 --debug-rgw 0 --debug-ms 0 2023-05-31T21:35:37.823 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setuser ceph since I am not root 2023-05-31T21:35:37.823 INFO:teuthology.orchestra.run.smithi099.stderr:ignoring --setgroup ceph since I am not root 2023-05-31T21:35:37.871 INFO:teuthology.orchestra.run.smithi099.stdout:{"sync_policy":{"groups":[]}} 2023-05-31T21:35:37.872 INFO:tasks.rgw_multisite_tests:test_sync_single_bucket_to_multiple: ... ok 2023-05-31T21:35:37.873 INFO:tasks.rgw_multisite_tests: 2023-05-31T21:35:37.873 INFO:tasks.rgw_multisite_tests:====================================================================== 2023-05-31T21:35:37.874 INFO:tasks.rgw_multisite_tests:FAIL: rgw_multi.tests.test_bucket_sync_disable 2023-05-31T21:35:37.874 INFO:tasks.rgw_multisite_tests:---------------------------------------------------------------------- 2023-05-31T21:35:37.876 INFO:tasks.rgw_multisite_tests:Traceback (most recent call last): 2023-05-31T21:35:37.876 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_teuthology_fa97ddd17d12e6d74b49196948f66e54fa009130/virtualenv/lib/python3.8/site-packages/nose/case.py", line 198, in runTest 2023-05-31T21:35:37.876 INFO:tasks.rgw_multisite_tests: self.test(*self.arg) 2023-05-31T21:35:37.877 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/github.com_ceph_ceph-c_2870bf456d3115490c153abce9d47ac5c043b65e/qa/../src/test/rgw/rgw_multi/tests.py", line 1140, in test_bucket_sync_disable 2023-05-31T21:35:37.877 INFO:tasks.rgw_multisite_tests: zonegroup_meta_checkpoint(zonegroup) 2023-05-31T21:35:37.877 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/github.com_ceph_ceph-c_2870bf456d3115490c153abce9d47ac5c043b65e/qa/../src/test/rgw/rgw_multi/tests.py", line 208, in zonegroup_meta_checkpoint 2023-05-31T21:35:37.877 INFO:tasks.rgw_multisite_tests: zone_meta_checkpoint(zone, meta_master_zone, master_status) 2023-05-31T21:35:37.878 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/github.com_ceph_ceph-c_2870bf456d3115490c153abce9d47ac5c043b65e/qa/../src/test/rgw/rgw_multi/tests.py", line 197, in zone_meta_checkpoint 2023-05-31T21:35:37.878 INFO:tasks.rgw_multisite_tests: assert False, 'failed meta checkpoint for zone=%s' % zone.name 2023-05-31T21:35:37.878 INFO:tasks.rgw_multisite_tests:AssertionError: failed meta checkpoint for zone=a2 2023-05-31T21:35:37.878 INFO:tasks.rgw_multisite_tests:-------------------- >> begin captured logging << -------------------- 2023-05-31T21:35:37.879 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: INFO: create bucket zone=a1 name=pklxkd-44 2023-05-31T21:35:37.879 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/pklxkd-44/ 2023-05-31T21:35:37.879 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/pklxkd-44/ 2023-05-31T21:35:37.879 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2023-05-31T21:35:37.879 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /pklxkd-44/ 2023-05-31T21:35:37.880 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-05-31T21:35:37.880 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-05-31T21:35:37.880 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T21:35:37.880 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-05-31T21:35:37.881 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-05-31T21:35:37.881 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-05-31T21:35:37.881 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-05-31T21:35:37.881 INFO:tasks.rgw_multisite_tests:PUT 2023-05-31T21:35:37.882 INFO:tasks.rgw_multisite_tests: 2023-05-31T21:35:37.882 INFO:tasks.rgw_multisite_tests: 2023-05-31T21:35:37.882 INFO:tasks.rgw_multisite_tests:Wed, 31 May 2023 20:10:02 GMT 2023-05-31T21:35:37.882 INFO:tasks.rgw_multisite_tests:/pklxkd-44/ 2023-05-31T21:35:37.883 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-05-31T21:35:37.883 INFO:tasks.rgw_multisite_tests:AWS Q80H11OLNU1S48PPYL26:tplv2S6oje6AFjLCtWLkuSyBmvg= 2023-05-31T21:35:37.883 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:10:02 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:tplv2S6oje6AFjLCtWLkuSyBmvg=', 'Content-Length': '0'} 2023-05-31T21:35:37.883 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('x-amz-request-id', 'tx000002280b60cd65dabcc-006477a99a-1630-a1'), ('Content-Length', '0'), ('Date', 'Wed, 31 May 2023 20:10:02 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:35:37.883 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: INFO: create bucket zone=a2 name=pklxkd-45 2023-05-31T21:35:37.884 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/pklxkd-45/ 2023-05-31T21:35:37.884 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/pklxkd-45/ 2023-05-31T21:35:37.884 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2023-05-31T21:35:37.884 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /pklxkd-45/ 2023-05-31T21:35:37.885 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-05-31T21:35:37.885 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-05-31T21:35:37.885 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T21:35:37.885 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-05-31T21:35:37.886 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-05-31T21:35:37.886 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-05-31T21:35:37.886 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-05-31T21:35:37.886 INFO:tasks.rgw_multisite_tests:PUT 2023-05-31T21:35:37.886 INFO:tasks.rgw_multisite_tests: 2023-05-31T21:35:37.887 INFO:tasks.rgw_multisite_tests: 2023-05-31T21:35:37.887 INFO:tasks.rgw_multisite_tests:Wed, 31 May 2023 20:10:02 GMT 2023-05-31T21:35:37.887 INFO:tasks.rgw_multisite_tests:/pklxkd-45/ 2023-05-31T21:35:37.887 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-05-31T21:35:37.888 INFO:tasks.rgw_multisite_tests:AWS Q80H11OLNU1S48PPYL26:PByA4jdQn+Mx1Qjcwt4TX9Y2iEQ= 2023-05-31T21:35:37.888 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:10:02 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:PByA4jdQn+Mx1Qjcwt4TX9Y2iEQ=', 'Content-Length': '0'} 2023-05-31T21:35:37.888 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('x-amz-request-id', 'tx000004328383d7e6905c2-006477a99a-162e-a2'), ('Content-Length', '0'), ('Date', 'Wed, 31 May 2023 20:10:02 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:35:37.888 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-31T21:35:37.889 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-31T21:35:37.889 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-05-31T21:35:37.889 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: ignoring --setuser ceph since I am not root 2023-05-31T21:35:37.889 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-05-31T21:35:37.890 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: No --period given, using current period=2e6123d2-d695-4350-9a01-61a455bc782b 2023-05-31T21:35:37.890 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: [ 2023-05-31T21:35:37.890 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2023-05-31T21:35:37.890 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "marker": "1_1685562631.516089_59.1", 2023-05-31T21:35:37.891 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "last_update": "2023-05-31T19:50:31.516089Z" 2023-05-31T21:35:37.891 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2023-05-31T21:35:37.891 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2023-05-31T21:35:37.891 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "marker": "", 2023-05-31T21:35:37.891 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "last_update": "0.000000" 2023-05-31T21:35:37.892 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2023-05-31T21:35:37.892 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2023-05-31T21:35:37.892 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "marker": "1_1685562970.274572_93.1", 2023-05-31T21:35:37.892 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "last_update": "2023-05-31T19:56:10.274572Z" 2023-05-31T21:35:37.893 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2023-05-31T21:35:37.893 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2023-05-31T21:35:37.893 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "marker": "1_1685563802.468126_296.1", 2023-05-31T21:35:37.893 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "last_update": "2023-05-31T20:10:02.468126Z" 2023-05-31T21:35:37.894 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2023-05-31T21:35:37.894 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: ] 2023-05-31T21:35:37.894 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: master meta markers={0: '1_1685562631.516089_59.1', 1: '', 2: '1_1685562970.274572_93.1', 3: '1_1685563802.468126_296.1'} 2023-05-31T21:35:37.894 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: INFO: starting meta checkpoint for zone=a2 2023-05-31T21:35:37.894 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-31T21:35:37.895 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-31T21:35:37.895 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-05-31T21:35:37.895 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: ignoring --setuser ceph since I am not root 2023-05-31T21:35:37.896 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-05-31T21:35:37.896 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: 2023-05-31T20:10:02.674+0000 7fe79bb04780 20 RGW-SYNC:meta: read sync status 2023-05-31T21:35:37.896 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: 2023-05-31T20:10:02.675+0000 7fe79bb04780 20 RGW-SYNC:meta: read sync status 2023-05-31T21:35:37.896 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2023-05-31T21:35:37.897 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "sync_status": { 2023-05-31T21:35:37.897 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "info": { 2023-05-31T21:35:37.897 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "status": "sync", 2023-05-31T21:35:37.897 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "num_shards": 4, 2023-05-31T21:35:37.898 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "period": "2e6123d2-d695-4350-9a01-61a455bc782b", 2023-05-31T21:35:37.898 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "realm_epoch": 2 2023-05-31T21:35:37.898 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2023-05-31T21:35:37.898 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "markers": [ 2023-05-31T21:35:37.899 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2023-05-31T21:35:37.899 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "key": 0, 2023-05-31T21:35:37.899 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "val": { 2023-05-31T21:35:37.899 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "state": 1, 2023-05-31T21:35:37.900 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "marker": "1_1685562631.516089_59.1", 2023-05-31T21:35:37.900 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "next_step_marker": "", 2023-05-31T21:35:37.900 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "total_entries": 1, 2023-05-31T21:35:37.900 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "pos": 0, 2023-05-31T21:35:37.901 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "timestamp": "2023-05-31T19:50:31.516089Z", 2023-05-31T21:35:37.901 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "realm_epoch": 2 2023-05-31T21:35:37.901 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2023-05-31T21:35:37.901 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2023-05-31T21:35:37.901 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2023-05-31T21:35:37.902 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "key": 1, 2023-05-31T21:35:37.902 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "val": { 2023-05-31T21:35:37.902 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "state": 1, 2023-05-31T21:35:37.902 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "marker": "", 2023-05-31T21:35:37.903 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "next_step_marker": "", 2023-05-31T21:35:37.903 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "total_entries": 0, 2023-05-31T21:35:37.903 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "pos": 0, 2023-05-31T21:35:37.903 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "timestamp": "0.000000", 2023-05-31T21:35:37.904 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "realm_epoch": 2 2023-05-31T21:35:37.904 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2023-05-31T21:35:37.904 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2023-05-31T21:35:37.904 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2023-05-31T21:35:37.904 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "key": 2, 2023-05-31T21:35:37.905 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "val": { 2023-05-31T21:35:37.905 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "state": 1, 2023-05-31T21:35:37.905 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "marker": "1_1685562970.274572_93.1", 2023-05-31T21:35:37.905 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "next_step_marker": "", 2023-05-31T21:35:37.906 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "total_entries": 0, 2023-05-31T21:35:37.906 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "pos": 0, 2023-05-31T21:35:37.906 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "timestamp": "2023-05-31T19:56:10.274572Z", 2023-05-31T21:35:37.906 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "realm_epoch": 2 2023-05-31T21:35:37.906 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2023-05-31T21:35:37.907 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2023-05-31T21:35:37.907 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2023-05-31T21:35:37.907 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "key": 3, 2023-05-31T21:35:37.907 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "val": { 2023-05-31T21:35:37.908 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "state": 1, 2023-05-31T21:35:37.908 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "marker": "1_1685563571.700188_212.1", 2023-05-31T21:35:37.908 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "next_step_marker": "", 2023-05-31T21:35:37.909 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "total_entries": 0, 2023-05-31T21:35:37.909 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "pos": 0, 2023-05-31T21:35:37.909 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "timestamp": "2023-05-31T20:06:11.700188Z", 2023-05-31T21:35:37.909 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "realm_epoch": 2 2023-05-31T21:35:37.910 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2023-05-31T21:35:37.910 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2023-05-31T21:35:37.910 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: ] 2023-05-31T21:35:37.910 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2023-05-31T21:35:37.910 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "full_sync": { 2023-05-31T21:35:37.911 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "total": 1, 2023-05-31T21:35:37.911 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "complete": 1 2023-05-31T21:35:37.911 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2023-05-31T21:35:37.911 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "current_time": "2023-05-31T20:10:02Z" 2023-05-31T21:35:37.912 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2023-05-31T21:35:37.912 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2023-05-31T21:35:37.912 INFO:tasks.rgw_multisite_tests: "sync_status": { 2023-05-31T21:35:37.912 INFO:tasks.rgw_multisite_tests: "info": { 2023-05-31T21:35:37.913 INFO:tasks.rgw_multisite_tests: "status": "sync", 2023-05-31T21:35:37.913 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2023-05-31T21:35:37.913 INFO:tasks.rgw_multisite_tests: "period": "2e6123d2-d695-4350-9a01-61a455bc782b", 2023-05-31T21:35:37.913 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-05-31T21:35:37.913 INFO:tasks.rgw_multisite_tests: }, 2023-05-31T21:35:37.914 INFO:tasks.rgw_multisite_tests: "markers": [ 2023-05-31T21:35:37.914 INFO:tasks.rgw_multisite_tests: { 2023-05-31T21:35:37.914 INFO:tasks.rgw_multisite_tests: "key": 0, 2023-05-31T21:35:37.914 INFO:tasks.rgw_multisite_tests: "val": { 2023-05-31T21:35:37.915 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-05-31T21:35:37.915 INFO:tasks.rgw_multisite_tests: "marker": "1_1685562631.516089_59.1", 2023-05-31T21:35:37.915 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-05-31T21:35:37.915 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2023-05-31T21:35:37.915 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-05-31T21:35:37.916 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-05-31T19:50:31.516089Z", 2023-05-31T21:35:37.916 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-05-31T21:35:37.916 INFO:tasks.rgw_multisite_tests: } 2023-05-31T21:35:37.916 INFO:tasks.rgw_multisite_tests: }, 2023-05-31T21:35:37.917 INFO:tasks.rgw_multisite_tests: { 2023-05-31T21:35:37.917 INFO:tasks.rgw_multisite_tests: "key": 1, 2023-05-31T21:35:37.917 INFO:tasks.rgw_multisite_tests: "val": { 2023-05-31T21:35:37.917 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-05-31T21:35:37.918 INFO:tasks.rgw_multisite_tests: "marker": "", 2023-05-31T21:35:37.918 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-05-31T21:35:37.918 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-05-31T21:35:37.918 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-05-31T21:35:37.919 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2023-05-31T21:35:37.919 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-05-31T21:35:37.919 INFO:tasks.rgw_multisite_tests: } 2023-05-31T21:35:37.919 INFO:tasks.rgw_multisite_tests: }, 2023-05-31T21:35:37.919 INFO:tasks.rgw_multisite_tests: { 2023-05-31T21:35:37.920 INFO:tasks.rgw_multisite_tests: "key": 2, 2023-05-31T21:35:37.920 INFO:tasks.rgw_multisite_tests: "val": { 2023-05-31T21:35:37.920 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-05-31T21:35:37.920 INFO:tasks.rgw_multisite_tests: "marker": "1_1685562970.274572_93.1", 2023-05-31T21:35:37.921 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-05-31T21:35:37.921 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-05-31T21:35:37.921 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-05-31T21:35:37.921 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-05-31T19:56:10.274572Z", 2023-05-31T21:35:37.922 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-05-31T21:35:37.922 INFO:tasks.rgw_multisite_tests: } 2023-05-31T21:35:37.922 INFO:tasks.rgw_multisite_tests: }, 2023-05-31T21:35:37.922 INFO:tasks.rgw_multisite_tests: { 2023-05-31T21:35:37.923 INFO:tasks.rgw_multisite_tests: "key": 3, 2023-05-31T21:35:37.923 INFO:tasks.rgw_multisite_tests: "val": { 2023-05-31T21:35:37.923 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-05-31T21:35:37.923 INFO:tasks.rgw_multisite_tests: "marker": "1_1685563571.700188_212.1", 2023-05-31T21:35:37.923 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-05-31T21:35:37.924 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-05-31T21:35:37.924 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-05-31T21:35:37.924 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-05-31T20:06:11.700188Z", 2023-05-31T21:35:37.924 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-05-31T21:35:37.925 INFO:tasks.rgw_multisite_tests: } 2023-05-31T21:35:37.925 INFO:tasks.rgw_multisite_tests: } 2023-05-31T21:35:37.925 INFO:tasks.rgw_multisite_tests: ] 2023-05-31T21:35:37.925 INFO:tasks.rgw_multisite_tests: }, 2023-05-31T21:35:37.925 INFO:tasks.rgw_multisite_tests: "full_sync": { 2023-05-31T21:35:37.926 INFO:tasks.rgw_multisite_tests: "total": 1, 2023-05-31T21:35:37.926 INFO:tasks.rgw_multisite_tests: "complete": 1 2023-05-31T21:35:37.926 INFO:tasks.rgw_multisite_tests: }, 2023-05-31T21:35:37.926 INFO:tasks.rgw_multisite_tests: "current_time": "2023-05-31T20:10:02Z" 2023-05-31T21:35:37.927 INFO:tasks.rgw_multisite_tests:} 2023-05-31T21:35:37.927 INFO:tasks.rgw_multisite_tests: 2023-05-31T21:35:37.927 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1685562631.516089_59.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-05-31T19:50:31.516089Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1685562970.274572_93.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-05-31T19:56:10.274572Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1685563571.700188_212.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-05-31T20:06:11.700188Z', 'realm_epoch': 2}}] 2023-05-31T21:35:37.927 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1685562631.516089_59.1', 1: '', 2: '1_1685562970.274572_93.1', 3: '1_1685563802.468126_296.1'} 2023-05-31T21:35:37.928 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1685562631.516089_59.1', 1: '', 2: '1_1685562970.274572_93.1', 3: '1_1685563571.700188_212.1'} 2023-05-31T21:35:37.928 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone a2 behind master: shard=3 master=1_1685563802.468126_296.1 target=1_1685563571.700188_212.1 2023-05-31T21:35:37.928 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-31T21:35:37.928 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-31T21:35:37.929 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-05-31T21:35:37.929 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: ignoring --setuser ceph since I am not root 2023-05-31T21:35:37.929 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-05-31T21:35:37.929 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: 2023-05-31T20:10:07.826+0000 7fd59bd4d780 20 RGW-SYNC:meta: read sync status 2023-05-31T21:35:37.930 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: 2023-05-31T20:10:07.830+0000 7fd59bd4d780 20 RGW-SYNC:meta: read sync status 2023-05-31T21:35:37.930 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2023-05-31T21:35:37.930 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "sync_status": { 2023-05-31T21:35:37.930 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "info": { 2023-05-31T21:35:37.930 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "status": "sync", 2023-05-31T21:35:37.931 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "num_shards": 4, 2023-05-31T21:35:37.931 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "period": "2e6123d2-d695-4350-9a01-61a455bc782b", 2023-05-31T21:35:37.931 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "realm_epoch": 2 2023-05-31T21:35:37.931 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2023-05-31T21:35:37.932 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "markers": [ 2023-05-31T21:35:37.932 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2023-05-31T21:35:37.932 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "key": 0, 2023-05-31T21:35:37.932 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "val": { 2023-05-31T21:35:37.933 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "state": 1, 2023-05-31T21:35:37.933 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "marker": "1_1685562631.516089_59.1", 2023-05-31T21:35:37.933 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "next_step_marker": "", 2023-05-31T21:35:37.933 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "total_entries": 1, 2023-05-31T21:35:37.933 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "pos": 0, 2023-05-31T21:35:37.934 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "timestamp": "2023-05-31T19:50:31.516089Z", 2023-05-31T21:35:37.934 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "realm_epoch": 2 2023-05-31T21:35:37.934 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2023-05-31T21:35:37.934 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2023-05-31T21:35:37.935 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2023-05-31T21:35:37.935 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "key": 1, 2023-05-31T21:35:37.935 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "val": { 2023-05-31T21:35:37.935 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "state": 1, 2023-05-31T21:35:37.935 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "marker": "", 2023-05-31T21:35:37.936 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "next_step_marker": "", 2023-05-31T21:35:37.936 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "total_entries": 0, 2023-05-31T21:35:37.936 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "pos": 0, 2023-05-31T21:35:37.936 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "timestamp": "0.000000", 2023-05-31T21:35:37.937 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "realm_epoch": 2 2023-05-31T21:35:37.937 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2023-05-31T21:35:37.937 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2023-05-31T21:35:37.937 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2023-05-31T21:35:37.938 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "key": 2, 2023-05-31T21:35:37.938 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "val": { 2023-05-31T21:35:37.938 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "state": 1, 2023-05-31T21:35:37.938 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "marker": "1_1685562970.274572_93.1", 2023-05-31T21:35:37.939 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "next_step_marker": "", 2023-05-31T21:35:37.939 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "total_entries": 0, 2023-05-31T21:35:37.939 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "pos": 0, 2023-05-31T21:35:37.939 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "timestamp": "2023-05-31T19:56:10.274572Z", 2023-05-31T21:35:37.940 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "realm_epoch": 2 2023-05-31T21:35:37.940 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2023-05-31T21:35:37.940 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2023-05-31T21:35:37.940 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2023-05-31T21:35:37.940 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "key": 3, 2023-05-31T21:35:37.941 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "val": { 2023-05-31T21:35:37.941 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "state": 1, 2023-05-31T21:35:37.941 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "marker": "1_1685563571.700188_212.1", 2023-05-31T21:35:37.941 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "next_step_marker": "", 2023-05-31T21:35:37.942 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "total_entries": 0, 2023-05-31T21:35:37.942 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "pos": 0, 2023-05-31T21:35:37.942 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "timestamp": "2023-05-31T20:06:11.700188Z", 2023-05-31T21:35:37.942 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "realm_epoch": 2 2023-05-31T21:35:37.943 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2023-05-31T21:35:37.943 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2023-05-31T21:35:37.943 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: ] 2023-05-31T21:35:37.943 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2023-05-31T21:35:37.943 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "full_sync": { 2023-05-31T21:35:37.944 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "total": 1, 2023-05-31T21:35:37.944 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "complete": 1 2023-05-31T21:35:37.944 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2023-05-31T21:35:37.944 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "current_time": "2023-05-31T20:10:07Z" 2023-05-31T21:35:37.945 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2023-05-31T21:35:37.945 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2023-05-31T21:35:37.945 INFO:tasks.rgw_multisite_tests: "sync_status": { 2023-05-31T21:35:37.945 INFO:tasks.rgw_multisite_tests: "info": { 2023-05-31T21:35:37.946 INFO:tasks.rgw_multisite_tests: "status": "sync", 2023-05-31T21:35:37.946 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2023-05-31T21:35:37.946 INFO:tasks.rgw_multisite_tests: "period": "2e6123d2-d695-4350-9a01-61a455bc782b", 2023-05-31T21:35:37.946 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-05-31T21:35:37.946 INFO:tasks.rgw_multisite_tests: }, 2023-05-31T21:35:37.947 INFO:tasks.rgw_multisite_tests: "markers": [ 2023-05-31T21:35:37.947 INFO:tasks.rgw_multisite_tests: { 2023-05-31T21:35:37.947 INFO:tasks.rgw_multisite_tests: "key": 0, 2023-05-31T21:35:37.947 INFO:tasks.rgw_multisite_tests: "val": { 2023-05-31T21:35:37.948 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-05-31T21:35:37.948 INFO:tasks.rgw_multisite_tests: "marker": "1_1685562631.516089_59.1", 2023-05-31T21:35:37.948 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-05-31T21:35:37.948 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2023-05-31T21:35:37.949 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-05-31T21:35:37.949 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-05-31T19:50:31.516089Z", 2023-05-31T21:35:37.949 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-05-31T21:35:37.949 INFO:tasks.rgw_multisite_tests: } 2023-05-31T21:35:37.949 INFO:tasks.rgw_multisite_tests: }, 2023-05-31T21:35:37.950 INFO:tasks.rgw_multisite_tests: { 2023-05-31T21:35:37.950 INFO:tasks.rgw_multisite_tests: "key": 1, 2023-05-31T21:35:37.950 INFO:tasks.rgw_multisite_tests: "val": { 2023-05-31T21:35:37.950 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-05-31T21:35:37.951 INFO:tasks.rgw_multisite_tests: "marker": "", 2023-05-31T21:35:37.951 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-05-31T21:35:37.951 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-05-31T21:35:37.951 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-05-31T21:35:37.952 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2023-05-31T21:35:37.952 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-05-31T21:35:37.952 INFO:tasks.rgw_multisite_tests: } 2023-05-31T21:35:37.952 INFO:tasks.rgw_multisite_tests: }, 2023-05-31T21:35:37.952 INFO:tasks.rgw_multisite_tests: { 2023-05-31T21:35:37.953 INFO:tasks.rgw_multisite_tests: "key": 2, 2023-05-31T21:35:37.953 INFO:tasks.rgw_multisite_tests: "val": { 2023-05-31T21:35:37.953 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-05-31T21:35:37.953 INFO:tasks.rgw_multisite_tests: "marker": "1_1685562970.274572_93.1", 2023-05-31T21:35:37.954 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-05-31T21:35:37.954 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-05-31T21:35:37.954 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-05-31T21:35:37.954 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-05-31T19:56:10.274572Z", 2023-05-31T21:35:37.955 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-05-31T21:35:37.955 INFO:tasks.rgw_multisite_tests: } 2023-05-31T21:35:37.955 INFO:tasks.rgw_multisite_tests: }, 2023-05-31T21:35:37.955 INFO:tasks.rgw_multisite_tests: { 2023-05-31T21:35:37.955 INFO:tasks.rgw_multisite_tests: "key": 3, 2023-05-31T21:35:37.956 INFO:tasks.rgw_multisite_tests: "val": { 2023-05-31T21:35:37.956 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-05-31T21:35:37.956 INFO:tasks.rgw_multisite_tests: "marker": "1_1685563571.700188_212.1", 2023-05-31T21:35:37.956 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-05-31T21:35:37.957 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-05-31T21:35:37.957 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-05-31T21:35:37.957 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-05-31T20:06:11.700188Z", 2023-05-31T21:35:37.957 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-05-31T21:35:37.958 INFO:tasks.rgw_multisite_tests: } 2023-05-31T21:35:37.958 INFO:tasks.rgw_multisite_tests: } 2023-05-31T21:35:37.958 INFO:tasks.rgw_multisite_tests: ] 2023-05-31T21:35:37.958 INFO:tasks.rgw_multisite_tests: }, 2023-05-31T21:35:37.959 INFO:tasks.rgw_multisite_tests: "full_sync": { 2023-05-31T21:35:37.959 INFO:tasks.rgw_multisite_tests: "total": 1, 2023-05-31T21:35:37.959 INFO:tasks.rgw_multisite_tests: "complete": 1 2023-05-31T21:35:37.959 INFO:tasks.rgw_multisite_tests: }, 2023-05-31T21:35:37.960 INFO:tasks.rgw_multisite_tests: "current_time": "2023-05-31T20:10:07Z" 2023-05-31T21:35:37.960 INFO:tasks.rgw_multisite_tests:} 2023-05-31T21:35:37.960 INFO:tasks.rgw_multisite_tests: 2023-05-31T21:35:37.960 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1685562631.516089_59.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-05-31T19:50:31.516089Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1685562970.274572_93.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-05-31T19:56:10.274572Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1685563571.700188_212.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-05-31T20:06:11.700188Z', 'realm_epoch': 2}}] 2023-05-31T21:35:37.960 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1685562631.516089_59.1', 1: '', 2: '1_1685562970.274572_93.1', 3: '1_1685563802.468126_296.1'} 2023-05-31T21:35:37.961 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1685562631.516089_59.1', 1: '', 2: '1_1685562970.274572_93.1', 3: '1_1685563571.700188_212.1'} 2023-05-31T21:35:37.961 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone a2 behind master: shard=3 master=1_1685563802.468126_296.1 target=1_1685563571.700188_212.1 2023-05-31T21:35:37.961 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-31T21:35:37.962 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-31T21:35:37.962 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-05-31T21:35:37.962 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: ignoring --setuser ceph since I am not root 2023-05-31T21:35:37.962 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-05-31T21:35:37.963 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: 2023-05-31T20:10:12.983+0000 7f53462a4780 20 RGW-SYNC:meta: read sync status 2023-05-31T21:35:37.963 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: 2023-05-31T20:10:12.984+0000 7f53462a4780 20 RGW-SYNC:meta: read sync status 2023-05-31T21:35:37.963 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2023-05-31T21:35:37.963 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "sync_status": { 2023-05-31T21:35:37.963 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "info": { 2023-05-31T21:35:37.964 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "status": "sync", 2023-05-31T21:35:37.964 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "num_shards": 4, 2023-05-31T21:35:37.964 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "period": "2e6123d2-d695-4350-9a01-61a455bc782b", 2023-05-31T21:35:37.964 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "realm_epoch": 2 2023-05-31T21:35:37.965 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2023-05-31T21:35:37.965 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "markers": [ 2023-05-31T21:35:37.965 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2023-05-31T21:35:37.965 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "key": 0, 2023-05-31T21:35:37.966 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "val": { 2023-05-31T21:35:37.966 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "state": 1, 2023-05-31T21:35:37.966 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "marker": "1_1685562631.516089_59.1", 2023-05-31T21:35:37.966 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "next_step_marker": "", 2023-05-31T21:35:37.966 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "total_entries": 1, 2023-05-31T21:35:37.967 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "pos": 0, 2023-05-31T21:35:37.967 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "timestamp": "2023-05-31T19:50:31.516089Z", 2023-05-31T21:35:37.967 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "realm_epoch": 2 2023-05-31T21:35:37.967 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2023-05-31T21:35:37.968 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2023-05-31T21:35:37.968 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2023-05-31T21:35:37.968 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "key": 1, 2023-05-31T21:35:37.968 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "val": { 2023-05-31T21:35:37.969 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "state": 1, 2023-05-31T21:35:37.969 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "marker": "", 2023-05-31T21:35:37.969 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "next_step_marker": "", 2023-05-31T21:35:37.969 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "total_entries": 0, 2023-05-31T21:35:37.970 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "pos": 0, 2023-05-31T21:35:37.970 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "timestamp": "0.000000", 2023-05-31T21:35:37.970 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "realm_epoch": 2 2023-05-31T21:35:37.970 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2023-05-31T21:35:37.970 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2023-05-31T21:35:37.971 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2023-05-31T21:35:37.971 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "key": 2, 2023-05-31T21:35:37.971 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "val": { 2023-05-31T21:35:37.971 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "state": 1, 2023-05-31T21:35:37.972 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "marker": "1_1685562970.274572_93.1", 2023-05-31T21:35:37.972 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "next_step_marker": "", 2023-05-31T21:35:37.972 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "total_entries": 0, 2023-05-31T21:35:37.973 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "pos": 0, 2023-05-31T21:35:37.973 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "timestamp": "2023-05-31T19:56:10.274572Z", 2023-05-31T21:35:37.973 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "realm_epoch": 2 2023-05-31T21:35:37.973 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2023-05-31T21:35:37.973 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2023-05-31T21:35:37.974 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2023-05-31T21:35:37.974 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "key": 3, 2023-05-31T21:35:37.974 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "val": { 2023-05-31T21:35:37.974 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "state": 1, 2023-05-31T21:35:37.975 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "marker": "1_1685563571.700188_212.1", 2023-05-31T21:35:37.975 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "next_step_marker": "", 2023-05-31T21:35:37.975 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "total_entries": 0, 2023-05-31T21:35:37.975 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "pos": 0, 2023-05-31T21:35:37.976 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "timestamp": "2023-05-31T20:06:11.700188Z", 2023-05-31T21:35:37.976 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "realm_epoch": 2 2023-05-31T21:35:37.976 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2023-05-31T21:35:37.976 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2023-05-31T21:35:37.977 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: ] 2023-05-31T21:35:37.977 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2023-05-31T21:35:37.977 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "full_sync": { 2023-05-31T21:35:37.977 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "total": 1, 2023-05-31T21:35:37.977 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "complete": 1 2023-05-31T21:35:37.978 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2023-05-31T21:35:37.978 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "current_time": "2023-05-31T20:10:12Z" 2023-05-31T21:35:37.978 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2023-05-31T21:35:37.978 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2023-05-31T21:35:37.979 INFO:tasks.rgw_multisite_tests: "sync_status": { 2023-05-31T21:35:37.979 INFO:tasks.rgw_multisite_tests: "info": { 2023-05-31T21:35:37.979 INFO:tasks.rgw_multisite_tests: "status": "sync", 2023-05-31T21:35:37.979 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2023-05-31T21:35:37.980 INFO:tasks.rgw_multisite_tests: "period": "2e6123d2-d695-4350-9a01-61a455bc782b", 2023-05-31T21:35:37.980 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-05-31T21:35:37.980 INFO:tasks.rgw_multisite_tests: }, 2023-05-31T21:35:37.980 INFO:tasks.rgw_multisite_tests: "markers": [ 2023-05-31T21:35:37.981 INFO:tasks.rgw_multisite_tests: { 2023-05-31T21:35:37.981 INFO:tasks.rgw_multisite_tests: "key": 0, 2023-05-31T21:35:37.981 INFO:tasks.rgw_multisite_tests: "val": { 2023-05-31T21:35:37.981 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-05-31T21:35:37.981 INFO:tasks.rgw_multisite_tests: "marker": "1_1685562631.516089_59.1", 2023-05-31T21:35:37.982 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-05-31T21:35:37.982 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2023-05-31T21:35:37.982 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-05-31T21:35:37.982 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-05-31T19:50:31.516089Z", 2023-05-31T21:35:37.983 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-05-31T21:35:37.983 INFO:tasks.rgw_multisite_tests: } 2023-05-31T21:35:37.983 INFO:tasks.rgw_multisite_tests: }, 2023-05-31T21:35:37.983 INFO:tasks.rgw_multisite_tests: { 2023-05-31T21:35:37.984 INFO:tasks.rgw_multisite_tests: "key": 1, 2023-05-31T21:35:37.984 INFO:tasks.rgw_multisite_tests: "val": { 2023-05-31T21:35:37.984 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-05-31T21:35:37.984 INFO:tasks.rgw_multisite_tests: "marker": "", 2023-05-31T21:35:37.984 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-05-31T21:35:37.985 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-05-31T21:35:37.985 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-05-31T21:35:37.985 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2023-05-31T21:35:37.985 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-05-31T21:35:37.986 INFO:tasks.rgw_multisite_tests: } 2023-05-31T21:35:37.986 INFO:tasks.rgw_multisite_tests: }, 2023-05-31T21:35:37.986 INFO:tasks.rgw_multisite_tests: { 2023-05-31T21:35:37.986 INFO:tasks.rgw_multisite_tests: "key": 2, 2023-05-31T21:35:37.987 INFO:tasks.rgw_multisite_tests: "val": { 2023-05-31T21:35:37.987 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-05-31T21:35:37.987 INFO:tasks.rgw_multisite_tests: "marker": "1_1685562970.274572_93.1", 2023-05-31T21:35:37.987 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-05-31T21:35:37.988 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-05-31T21:35:37.988 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-05-31T21:35:37.988 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-05-31T19:56:10.274572Z", 2023-05-31T21:35:37.988 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-05-31T21:35:37.988 INFO:tasks.rgw_multisite_tests: } 2023-05-31T21:35:37.989 INFO:tasks.rgw_multisite_tests: }, 2023-05-31T21:35:37.989 INFO:tasks.rgw_multisite_tests: { 2023-05-31T21:35:37.989 INFO:tasks.rgw_multisite_tests: "key": 3, 2023-05-31T21:35:37.989 INFO:tasks.rgw_multisite_tests: "val": { 2023-05-31T21:35:37.990 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-05-31T21:35:37.990 INFO:tasks.rgw_multisite_tests: "marker": "1_1685563571.700188_212.1", 2023-05-31T21:35:37.990 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-05-31T21:35:37.990 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-05-31T21:35:37.991 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-05-31T21:35:37.991 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-05-31T20:06:11.700188Z", 2023-05-31T21:35:37.991 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-05-31T21:35:37.991 INFO:tasks.rgw_multisite_tests: } 2023-05-31T21:35:37.991 INFO:tasks.rgw_multisite_tests: } 2023-05-31T21:35:37.992 INFO:tasks.rgw_multisite_tests: ] 2023-05-31T21:35:37.992 INFO:tasks.rgw_multisite_tests: }, 2023-05-31T21:35:37.992 INFO:tasks.rgw_multisite_tests: "full_sync": { 2023-05-31T21:35:37.992 INFO:tasks.rgw_multisite_tests: "total": 1, 2023-05-31T21:35:37.993 INFO:tasks.rgw_multisite_tests: "complete": 1 2023-05-31T21:35:37.993 INFO:tasks.rgw_multisite_tests: }, 2023-05-31T21:35:37.993 INFO:tasks.rgw_multisite_tests: "current_time": "2023-05-31T20:10:12Z" 2023-05-31T21:35:37.993 INFO:tasks.rgw_multisite_tests:} 2023-05-31T21:35:37.994 INFO:tasks.rgw_multisite_tests: 2023-05-31T21:35:37.994 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1685562631.516089_59.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-05-31T19:50:31.516089Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1685562970.274572_93.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-05-31T19:56:10.274572Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1685563571.700188_212.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-05-31T20:06:11.700188Z', 'realm_epoch': 2}}] 2023-05-31T21:35:37.994 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1685562631.516089_59.1', 1: '', 2: '1_1685562970.274572_93.1', 3: '1_1685563802.468126_296.1'} 2023-05-31T21:35:37.994 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1685562631.516089_59.1', 1: '', 2: '1_1685562970.274572_93.1', 3: '1_1685563571.700188_212.1'} 2023-05-31T21:35:37.994 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone a2 behind master: shard=3 master=1_1685563802.468126_296.1 target=1_1685563571.700188_212.1 2023-05-31T21:35:37.995 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-31T21:35:37.995 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-31T21:35:37.995 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-05-31T21:35:37.996 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: ignoring --setuser ceph since I am not root 2023-05-31T21:35:37.996 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-05-31T21:35:37.996 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: 2023-05-31T20:10:18.125+0000 7fcee7df8780 20 RGW-SYNC:meta: read sync status 2023-05-31T21:35:37.996 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: 2023-05-31T20:10:18.126+0000 7fcee7df8780 20 RGW-SYNC:meta: read sync status 2023-05-31T21:35:37.997 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2023-05-31T21:35:37.997 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "sync_status": { 2023-05-31T21:35:37.997 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "info": { 2023-05-31T21:35:37.997 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "status": "sync", 2023-05-31T21:35:37.998 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "num_shards": 4, 2023-05-31T21:35:37.998 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "period": "2e6123d2-d695-4350-9a01-61a455bc782b", 2023-05-31T21:35:37.998 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "realm_epoch": 2 2023-05-31T21:35:37.998 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2023-05-31T21:35:37.999 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "markers": [ 2023-05-31T21:35:37.999 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2023-05-31T21:35:37.999 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "key": 0, 2023-05-31T21:35:37.999 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "val": { 2023-05-31T21:35:38.000 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "state": 1, 2023-05-31T21:35:38.000 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "marker": "1_1685562631.516089_59.1", 2023-05-31T21:35:38.000 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "next_step_marker": "", 2023-05-31T21:35:38.000 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "total_entries": 1, 2023-05-31T21:35:38.000 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "pos": 0, 2023-05-31T21:35:38.001 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "timestamp": "2023-05-31T19:50:31.516089Z", 2023-05-31T21:35:38.001 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "realm_epoch": 2 2023-05-31T21:35:38.001 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2023-05-31T21:35:38.001 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2023-05-31T21:35:38.002 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2023-05-31T21:35:38.002 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "key": 1, 2023-05-31T21:35:38.002 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "val": { 2023-05-31T21:35:38.002 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "state": 1, 2023-05-31T21:35:38.003 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "marker": "", 2023-05-31T21:35:38.003 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "next_step_marker": "", 2023-05-31T21:35:38.003 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "total_entries": 0, 2023-05-31T21:35:38.003 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "pos": 0, 2023-05-31T21:35:38.004 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "timestamp": "0.000000", 2023-05-31T21:35:38.004 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "realm_epoch": 2 2023-05-31T21:35:38.004 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2023-05-31T21:35:38.004 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2023-05-31T21:35:38.004 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2023-05-31T21:35:38.005 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "key": 2, 2023-05-31T21:35:38.005 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "val": { 2023-05-31T21:35:38.005 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "state": 1, 2023-05-31T21:35:38.005 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "marker": "1_1685562970.274572_93.1", 2023-05-31T21:35:38.006 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "next_step_marker": "", 2023-05-31T21:35:38.006 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "total_entries": 0, 2023-05-31T21:35:38.006 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "pos": 0, 2023-05-31T21:35:38.006 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "timestamp": "2023-05-31T19:56:10.274572Z", 2023-05-31T21:35:38.006 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "realm_epoch": 2 2023-05-31T21:35:38.007 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2023-05-31T21:35:38.007 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2023-05-31T21:35:38.007 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2023-05-31T21:35:38.007 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "key": 3, 2023-05-31T21:35:38.008 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "val": { 2023-05-31T21:35:38.008 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "state": 1, 2023-05-31T21:35:38.008 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "marker": "1_1685563571.700188_212.1", 2023-05-31T21:35:38.009 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "next_step_marker": "", 2023-05-31T21:35:38.009 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "total_entries": 0, 2023-05-31T21:35:38.009 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "pos": 0, 2023-05-31T21:35:38.009 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "timestamp": "2023-05-31T20:06:11.700188Z", 2023-05-31T21:35:38.010 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "realm_epoch": 2 2023-05-31T21:35:38.010 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2023-05-31T21:35:38.010 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2023-05-31T21:35:38.010 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: ] 2023-05-31T21:35:38.010 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2023-05-31T21:35:38.011 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "full_sync": { 2023-05-31T21:35:38.011 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "total": 1, 2023-05-31T21:35:38.011 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "complete": 1 2023-05-31T21:35:38.011 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2023-05-31T21:35:38.012 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "current_time": "2023-05-31T20:10:18Z" 2023-05-31T21:35:38.012 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2023-05-31T21:35:38.012 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2023-05-31T21:35:38.012 INFO:tasks.rgw_multisite_tests: "sync_status": { 2023-05-31T21:35:38.013 INFO:tasks.rgw_multisite_tests: "info": { 2023-05-31T21:35:38.013 INFO:tasks.rgw_multisite_tests: "status": "sync", 2023-05-31T21:35:38.013 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2023-05-31T21:35:38.013 INFO:tasks.rgw_multisite_tests: "period": "2e6123d2-d695-4350-9a01-61a455bc782b", 2023-05-31T21:35:38.013 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-05-31T21:35:38.014 INFO:tasks.rgw_multisite_tests: }, 2023-05-31T21:35:38.014 INFO:tasks.rgw_multisite_tests: "markers": [ 2023-05-31T21:35:38.014 INFO:tasks.rgw_multisite_tests: { 2023-05-31T21:35:38.014 INFO:tasks.rgw_multisite_tests: "key": 0, 2023-05-31T21:35:38.015 INFO:tasks.rgw_multisite_tests: "val": { 2023-05-31T21:35:38.015 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-05-31T21:35:38.015 INFO:tasks.rgw_multisite_tests: "marker": "1_1685562631.516089_59.1", 2023-05-31T21:35:38.015 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-05-31T21:35:38.016 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2023-05-31T21:35:38.016 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-05-31T21:35:38.016 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-05-31T19:50:31.516089Z", 2023-05-31T21:35:38.016 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-05-31T21:35:38.016 INFO:tasks.rgw_multisite_tests: } 2023-05-31T21:35:38.017 INFO:tasks.rgw_multisite_tests: }, 2023-05-31T21:35:38.017 INFO:tasks.rgw_multisite_tests: { 2023-05-31T21:35:38.017 INFO:tasks.rgw_multisite_tests: "key": 1, 2023-05-31T21:35:38.017 INFO:tasks.rgw_multisite_tests: "val": { 2023-05-31T21:35:38.018 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-05-31T21:35:38.018 INFO:tasks.rgw_multisite_tests: "marker": "", 2023-05-31T21:35:38.018 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-05-31T21:35:38.018 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-05-31T21:35:38.019 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-05-31T21:35:38.019 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2023-05-31T21:35:38.019 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-05-31T21:35:38.019 INFO:tasks.rgw_multisite_tests: } 2023-05-31T21:35:38.019 INFO:tasks.rgw_multisite_tests: }, 2023-05-31T21:35:38.020 INFO:tasks.rgw_multisite_tests: { 2023-05-31T21:35:38.020 INFO:tasks.rgw_multisite_tests: "key": 2, 2023-05-31T21:35:38.020 INFO:tasks.rgw_multisite_tests: "val": { 2023-05-31T21:35:38.020 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-05-31T21:35:38.021 INFO:tasks.rgw_multisite_tests: "marker": "1_1685562970.274572_93.1", 2023-05-31T21:35:38.021 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-05-31T21:35:38.021 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-05-31T21:35:38.021 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-05-31T21:35:38.022 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-05-31T19:56:10.274572Z", 2023-05-31T21:35:38.022 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-05-31T21:35:38.022 INFO:tasks.rgw_multisite_tests: } 2023-05-31T21:35:38.022 INFO:tasks.rgw_multisite_tests: }, 2023-05-31T21:35:38.022 INFO:tasks.rgw_multisite_tests: { 2023-05-31T21:35:38.023 INFO:tasks.rgw_multisite_tests: "key": 3, 2023-05-31T21:35:38.023 INFO:tasks.rgw_multisite_tests: "val": { 2023-05-31T21:35:38.023 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-05-31T21:35:38.023 INFO:tasks.rgw_multisite_tests: "marker": "1_1685563571.700188_212.1", 2023-05-31T21:35:38.024 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-05-31T21:35:38.024 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-05-31T21:35:38.024 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-05-31T21:35:38.024 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-05-31T20:06:11.700188Z", 2023-05-31T21:35:38.025 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-05-31T21:35:38.025 INFO:tasks.rgw_multisite_tests: } 2023-05-31T21:35:38.025 INFO:tasks.rgw_multisite_tests: } 2023-05-31T21:35:38.025 INFO:tasks.rgw_multisite_tests: ] 2023-05-31T21:35:38.025 INFO:tasks.rgw_multisite_tests: }, 2023-05-31T21:35:38.026 INFO:tasks.rgw_multisite_tests: "full_sync": { 2023-05-31T21:35:38.026 INFO:tasks.rgw_multisite_tests: "total": 1, 2023-05-31T21:35:38.026 INFO:tasks.rgw_multisite_tests: "complete": 1 2023-05-31T21:35:38.026 INFO:tasks.rgw_multisite_tests: }, 2023-05-31T21:35:38.027 INFO:tasks.rgw_multisite_tests: "current_time": "2023-05-31T20:10:18Z" 2023-05-31T21:35:38.027 INFO:tasks.rgw_multisite_tests:} 2023-05-31T21:35:38.027 INFO:tasks.rgw_multisite_tests: 2023-05-31T21:35:38.027 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1685562631.516089_59.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-05-31T19:50:31.516089Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1685562970.274572_93.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-05-31T19:56:10.274572Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1685563571.700188_212.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-05-31T20:06:11.700188Z', 'realm_epoch': 2}}] 2023-05-31T21:35:38.028 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1685562631.516089_59.1', 1: '', 2: '1_1685562970.274572_93.1', 3: '1_1685563802.468126_296.1'} 2023-05-31T21:35:38.028 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1685562631.516089_59.1', 1: '', 2: '1_1685562970.274572_93.1', 3: '1_1685563571.700188_212.1'} 2023-05-31T21:35:38.028 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone a2 behind master: shard=3 master=1_1685563802.468126_296.1 target=1_1685563571.700188_212.1 2023-05-31T21:35:38.028 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-31T21:35:38.029 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-31T21:35:38.029 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-05-31T21:35:38.029 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: ignoring --setuser ceph since I am not root 2023-05-31T21:35:38.029 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-05-31T21:35:38.029 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: 2023-05-31T20:10:23.285+0000 7f341a52d780 20 RGW-SYNC:meta: read sync status 2023-05-31T21:35:38.030 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: 2023-05-31T20:10:23.286+0000 7f341a52d780 20 RGW-SYNC:meta: read sync status 2023-05-31T21:35:38.030 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2023-05-31T21:35:38.030 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "sync_status": { 2023-05-31T21:35:38.030 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "info": { 2023-05-31T21:35:38.031 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "status": "sync", 2023-05-31T21:35:38.031 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "num_shards": 4, 2023-05-31T21:35:38.031 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "period": "2e6123d2-d695-4350-9a01-61a455bc782b", 2023-05-31T21:35:38.031 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "realm_epoch": 2 2023-05-31T21:35:38.032 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2023-05-31T21:35:38.032 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "markers": [ 2023-05-31T21:35:38.032 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2023-05-31T21:35:38.032 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "key": 0, 2023-05-31T21:35:38.032 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "val": { 2023-05-31T21:35:38.033 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "state": 1, 2023-05-31T21:35:38.033 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "marker": "1_1685562631.516089_59.1", 2023-05-31T21:35:38.033 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "next_step_marker": "", 2023-05-31T21:35:38.033 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "total_entries": 1, 2023-05-31T21:35:38.034 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "pos": 0, 2023-05-31T21:35:38.034 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "timestamp": "2023-05-31T19:50:31.516089Z", 2023-05-31T21:35:38.034 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "realm_epoch": 2 2023-05-31T21:35:38.034 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2023-05-31T21:35:38.035 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2023-05-31T21:35:38.035 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2023-05-31T21:35:38.035 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "key": 1, 2023-05-31T21:35:38.035 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "val": { 2023-05-31T21:35:38.035 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "state": 1, 2023-05-31T21:35:38.036 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "marker": "", 2023-05-31T21:35:38.036 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "next_step_marker": "", 2023-05-31T21:35:38.036 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "total_entries": 0, 2023-05-31T21:35:38.036 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "pos": 0, 2023-05-31T21:35:38.037 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "timestamp": "0.000000", 2023-05-31T21:35:38.037 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "realm_epoch": 2 2023-05-31T21:35:38.037 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2023-05-31T21:35:38.037 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2023-05-31T21:35:38.038 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2023-05-31T21:35:38.038 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "key": 2, 2023-05-31T21:35:38.038 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "val": { 2023-05-31T21:35:38.038 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "state": 1, 2023-05-31T21:35:38.038 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "marker": "1_1685562970.274572_93.1", 2023-05-31T21:35:38.039 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "next_step_marker": "", 2023-05-31T21:35:38.039 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "total_entries": 0, 2023-05-31T21:35:38.039 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "pos": 0, 2023-05-31T21:35:38.039 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "timestamp": "2023-05-31T19:56:10.274572Z", 2023-05-31T21:35:38.040 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "realm_epoch": 2 2023-05-31T21:35:38.040 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2023-05-31T21:35:38.040 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2023-05-31T21:35:38.040 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2023-05-31T21:35:38.041 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "key": 3, 2023-05-31T21:35:38.041 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "val": { 2023-05-31T21:35:38.041 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "state": 1, 2023-05-31T21:35:38.041 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "marker": "1_1685563571.700188_212.1", 2023-05-31T21:35:38.041 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "next_step_marker": "", 2023-05-31T21:35:38.042 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "total_entries": 0, 2023-05-31T21:35:38.042 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "pos": 0, 2023-05-31T21:35:38.042 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "timestamp": "2023-05-31T20:06:11.700188Z", 2023-05-31T21:35:38.042 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "realm_epoch": 2 2023-05-31T21:35:38.043 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2023-05-31T21:35:38.043 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2023-05-31T21:35:38.043 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: ] 2023-05-31T21:35:38.043 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2023-05-31T21:35:38.044 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "full_sync": { 2023-05-31T21:35:38.044 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "total": 1, 2023-05-31T21:35:38.044 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "complete": 1 2023-05-31T21:35:38.044 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2023-05-31T21:35:38.044 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "current_time": "2023-05-31T20:10:23Z" 2023-05-31T21:35:38.045 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2023-05-31T21:35:38.045 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2023-05-31T21:35:38.045 INFO:tasks.rgw_multisite_tests: "sync_status": { 2023-05-31T21:35:38.045 INFO:tasks.rgw_multisite_tests: "info": { 2023-05-31T21:35:38.046 INFO:tasks.rgw_multisite_tests: "status": "sync", 2023-05-31T21:35:38.046 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2023-05-31T21:35:38.046 INFO:tasks.rgw_multisite_tests: "period": "2e6123d2-d695-4350-9a01-61a455bc782b", 2023-05-31T21:35:38.046 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-05-31T21:35:38.047 INFO:tasks.rgw_multisite_tests: }, 2023-05-31T21:35:38.047 INFO:tasks.rgw_multisite_tests: "markers": [ 2023-05-31T21:35:38.047 INFO:tasks.rgw_multisite_tests: { 2023-05-31T21:35:38.047 INFO:tasks.rgw_multisite_tests: "key": 0, 2023-05-31T21:35:38.047 INFO:tasks.rgw_multisite_tests: "val": { 2023-05-31T21:35:38.048 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-05-31T21:35:38.048 INFO:tasks.rgw_multisite_tests: "marker": "1_1685562631.516089_59.1", 2023-05-31T21:35:38.048 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-05-31T21:35:38.048 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2023-05-31T21:35:38.049 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-05-31T21:35:38.049 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-05-31T19:50:31.516089Z", 2023-05-31T21:35:38.049 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-05-31T21:35:38.049 INFO:tasks.rgw_multisite_tests: } 2023-05-31T21:35:38.050 INFO:tasks.rgw_multisite_tests: }, 2023-05-31T21:35:38.050 INFO:tasks.rgw_multisite_tests: { 2023-05-31T21:35:38.050 INFO:tasks.rgw_multisite_tests: "key": 1, 2023-05-31T21:35:38.050 INFO:tasks.rgw_multisite_tests: "val": { 2023-05-31T21:35:38.050 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-05-31T21:35:38.051 INFO:tasks.rgw_multisite_tests: "marker": "", 2023-05-31T21:35:38.051 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-05-31T21:35:38.051 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-05-31T21:35:38.051 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-05-31T21:35:38.052 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2023-05-31T21:35:38.052 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-05-31T21:35:38.052 INFO:tasks.rgw_multisite_tests: } 2023-05-31T21:35:38.052 INFO:tasks.rgw_multisite_tests: }, 2023-05-31T21:35:38.053 INFO:tasks.rgw_multisite_tests: { 2023-05-31T21:35:38.053 INFO:tasks.rgw_multisite_tests: "key": 2, 2023-05-31T21:35:38.053 INFO:tasks.rgw_multisite_tests: "val": { 2023-05-31T21:35:38.053 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-05-31T21:35:38.053 INFO:tasks.rgw_multisite_tests: "marker": "1_1685562970.274572_93.1", 2023-05-31T21:35:38.054 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-05-31T21:35:38.054 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-05-31T21:35:38.054 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-05-31T21:35:38.054 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-05-31T19:56:10.274572Z", 2023-05-31T21:35:38.055 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-05-31T21:35:38.055 INFO:tasks.rgw_multisite_tests: } 2023-05-31T21:35:38.055 INFO:tasks.rgw_multisite_tests: }, 2023-05-31T21:35:38.055 INFO:tasks.rgw_multisite_tests: { 2023-05-31T21:35:38.056 INFO:tasks.rgw_multisite_tests: "key": 3, 2023-05-31T21:35:38.056 INFO:tasks.rgw_multisite_tests: "val": { 2023-05-31T21:35:38.056 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-05-31T21:35:38.056 INFO:tasks.rgw_multisite_tests: "marker": "1_1685563571.700188_212.1", 2023-05-31T21:35:38.056 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-05-31T21:35:38.057 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-05-31T21:35:38.057 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-05-31T21:35:38.057 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-05-31T20:06:11.700188Z", 2023-05-31T21:35:38.057 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-05-31T21:35:38.058 INFO:tasks.rgw_multisite_tests: } 2023-05-31T21:35:38.058 INFO:tasks.rgw_multisite_tests: } 2023-05-31T21:35:38.058 INFO:tasks.rgw_multisite_tests: ] 2023-05-31T21:35:38.058 INFO:tasks.rgw_multisite_tests: }, 2023-05-31T21:35:38.059 INFO:tasks.rgw_multisite_tests: "full_sync": { 2023-05-31T21:35:38.059 INFO:tasks.rgw_multisite_tests: "total": 1, 2023-05-31T21:35:38.059 INFO:tasks.rgw_multisite_tests: "complete": 1 2023-05-31T21:35:38.059 INFO:tasks.rgw_multisite_tests: }, 2023-05-31T21:35:38.060 INFO:tasks.rgw_multisite_tests: "current_time": "2023-05-31T20:10:23Z" 2023-05-31T21:35:38.060 INFO:tasks.rgw_multisite_tests:} 2023-05-31T21:35:38.060 INFO:tasks.rgw_multisite_tests: 2023-05-31T21:35:38.060 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1685562631.516089_59.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-05-31T19:50:31.516089Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1685562970.274572_93.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-05-31T19:56:10.274572Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1685563571.700188_212.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-05-31T20:06:11.700188Z', 'realm_epoch': 2}}] 2023-05-31T21:35:38.061 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1685562631.516089_59.1', 1: '', 2: '1_1685562970.274572_93.1', 3: '1_1685563802.468126_296.1'} 2023-05-31T21:35:38.061 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1685562631.516089_59.1', 1: '', 2: '1_1685562970.274572_93.1', 3: '1_1685563571.700188_212.1'} 2023-05-31T21:35:38.061 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone a2 behind master: shard=3 master=1_1685563802.468126_296.1 target=1_1685563571.700188_212.1 2023-05-31T21:35:38.061 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-31T21:35:38.062 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-31T21:35:38.062 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-05-31T21:35:38.062 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: ignoring --setuser ceph since I am not root 2023-05-31T21:35:38.062 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-05-31T21:35:38.063 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: 2023-05-31T20:10:28.418+0000 7f124688b780 20 RGW-SYNC:meta: read sync status 2023-05-31T21:35:38.063 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: 2023-05-31T20:10:28.419+0000 7f124688b780 20 RGW-SYNC:meta: read sync status 2023-05-31T21:35:38.063 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2023-05-31T21:35:38.063 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "sync_status": { 2023-05-31T21:35:38.063 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "info": { 2023-05-31T21:35:38.064 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "status": "sync", 2023-05-31T21:35:38.064 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "num_shards": 4, 2023-05-31T21:35:38.064 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "period": "2e6123d2-d695-4350-9a01-61a455bc782b", 2023-05-31T21:35:38.064 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "realm_epoch": 2 2023-05-31T21:35:38.065 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2023-05-31T21:35:38.065 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "markers": [ 2023-05-31T21:35:38.065 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2023-05-31T21:35:38.065 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "key": 0, 2023-05-31T21:35:38.066 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "val": { 2023-05-31T21:35:38.066 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "state": 1, 2023-05-31T21:35:38.066 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "marker": "1_1685562631.516089_59.1", 2023-05-31T21:35:38.066 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "next_step_marker": "", 2023-05-31T21:35:38.066 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "total_entries": 1, 2023-05-31T21:35:38.067 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "pos": 0, 2023-05-31T21:35:38.067 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "timestamp": "2023-05-31T19:50:31.516089Z", 2023-05-31T21:35:38.067 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "realm_epoch": 2 2023-05-31T21:35:38.067 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2023-05-31T21:35:38.068 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2023-05-31T21:35:38.068 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2023-05-31T21:35:38.068 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "key": 1, 2023-05-31T21:35:38.068 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "val": { 2023-05-31T21:35:38.069 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "state": 1, 2023-05-31T21:35:38.069 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "marker": "", 2023-05-31T21:35:38.069 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "next_step_marker": "", 2023-05-31T21:35:38.069 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "total_entries": 0, 2023-05-31T21:35:38.070 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "pos": 0, 2023-05-31T21:35:38.070 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "timestamp": "0.000000", 2023-05-31T21:35:38.070 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "realm_epoch": 2 2023-05-31T21:35:38.070 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2023-05-31T21:35:38.070 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2023-05-31T21:35:38.071 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2023-05-31T21:35:38.071 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "key": 2, 2023-05-31T21:35:38.071 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "val": { 2023-05-31T21:35:38.071 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "state": 1, 2023-05-31T21:35:38.072 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "marker": "1_1685562970.274572_93.1", 2023-05-31T21:35:38.072 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "next_step_marker": "", 2023-05-31T21:35:38.072 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "total_entries": 0, 2023-05-31T21:35:38.072 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "pos": 0, 2023-05-31T21:35:38.073 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "timestamp": "2023-05-31T19:56:10.274572Z", 2023-05-31T21:35:38.073 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "realm_epoch": 2 2023-05-31T21:35:38.073 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2023-05-31T21:35:38.073 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2023-05-31T21:35:38.074 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2023-05-31T21:35:38.074 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "key": 3, 2023-05-31T21:35:38.074 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "val": { 2023-05-31T21:35:38.074 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "state": 1, 2023-05-31T21:35:38.075 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "marker": "1_1685563571.700188_212.1", 2023-05-31T21:35:38.075 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "next_step_marker": "", 2023-05-31T21:35:38.075 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "total_entries": 0, 2023-05-31T21:35:38.075 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "pos": 0, 2023-05-31T21:35:38.075 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "timestamp": "2023-05-31T20:06:11.700188Z", 2023-05-31T21:35:38.076 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "realm_epoch": 2 2023-05-31T21:35:38.076 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2023-05-31T21:35:38.076 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2023-05-31T21:35:38.076 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: ] 2023-05-31T21:35:38.077 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2023-05-31T21:35:38.077 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "full_sync": { 2023-05-31T21:35:38.077 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "total": 1, 2023-05-31T21:35:38.077 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "complete": 1 2023-05-31T21:35:38.078 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2023-05-31T21:35:38.078 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "current_time": "2023-05-31T20:10:28Z" 2023-05-31T21:35:38.078 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2023-05-31T21:35:38.078 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2023-05-31T21:35:38.079 INFO:tasks.rgw_multisite_tests: "sync_status": { 2023-05-31T21:35:38.079 INFO:tasks.rgw_multisite_tests: "info": { 2023-05-31T21:35:38.079 INFO:tasks.rgw_multisite_tests: "status": "sync", 2023-05-31T21:35:38.079 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2023-05-31T21:35:38.079 INFO:tasks.rgw_multisite_tests: "period": "2e6123d2-d695-4350-9a01-61a455bc782b", 2023-05-31T21:35:38.080 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-05-31T21:35:38.080 INFO:tasks.rgw_multisite_tests: }, 2023-05-31T21:35:38.080 INFO:tasks.rgw_multisite_tests: "markers": [ 2023-05-31T21:35:38.080 INFO:tasks.rgw_multisite_tests: { 2023-05-31T21:35:38.081 INFO:tasks.rgw_multisite_tests: "key": 0, 2023-05-31T21:35:38.081 INFO:tasks.rgw_multisite_tests: "val": { 2023-05-31T21:35:38.081 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-05-31T21:35:38.081 INFO:tasks.rgw_multisite_tests: "marker": "1_1685562631.516089_59.1", 2023-05-31T21:35:38.082 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-05-31T21:35:38.082 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2023-05-31T21:35:38.082 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-05-31T21:35:38.082 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-05-31T19:50:31.516089Z", 2023-05-31T21:35:38.082 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-05-31T21:35:38.083 INFO:tasks.rgw_multisite_tests: } 2023-05-31T21:35:38.083 INFO:tasks.rgw_multisite_tests: }, 2023-05-31T21:35:38.083 INFO:tasks.rgw_multisite_tests: { 2023-05-31T21:35:38.083 INFO:tasks.rgw_multisite_tests: "key": 1, 2023-05-31T21:35:38.084 INFO:tasks.rgw_multisite_tests: "val": { 2023-05-31T21:35:38.084 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-05-31T21:35:38.084 INFO:tasks.rgw_multisite_tests: "marker": "", 2023-05-31T21:35:38.084 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-05-31T21:35:38.085 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-05-31T21:35:38.085 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-05-31T21:35:38.085 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2023-05-31T21:35:38.085 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-05-31T21:35:38.086 INFO:tasks.rgw_multisite_tests: } 2023-05-31T21:35:38.086 INFO:tasks.rgw_multisite_tests: }, 2023-05-31T21:35:38.086 INFO:tasks.rgw_multisite_tests: { 2023-05-31T21:35:38.086 INFO:tasks.rgw_multisite_tests: "key": 2, 2023-05-31T21:35:38.086 INFO:tasks.rgw_multisite_tests: "val": { 2023-05-31T21:35:38.087 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-05-31T21:35:38.087 INFO:tasks.rgw_multisite_tests: "marker": "1_1685562970.274572_93.1", 2023-05-31T21:35:38.087 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-05-31T21:35:38.087 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-05-31T21:35:38.088 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-05-31T21:35:38.088 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-05-31T19:56:10.274572Z", 2023-05-31T21:35:38.088 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-05-31T21:35:38.088 INFO:tasks.rgw_multisite_tests: } 2023-05-31T21:35:38.089 INFO:tasks.rgw_multisite_tests: }, 2023-05-31T21:35:38.089 INFO:tasks.rgw_multisite_tests: { 2023-05-31T21:35:38.089 INFO:tasks.rgw_multisite_tests: "key": 3, 2023-05-31T21:35:38.089 INFO:tasks.rgw_multisite_tests: "val": { 2023-05-31T21:35:38.090 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-05-31T21:35:38.090 INFO:tasks.rgw_multisite_tests: "marker": "1_1685563571.700188_212.1", 2023-05-31T21:35:38.090 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-05-31T21:35:38.090 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-05-31T21:35:38.090 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-05-31T21:35:38.091 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-05-31T20:06:11.700188Z", 2023-05-31T21:35:38.091 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-05-31T21:35:38.091 INFO:tasks.rgw_multisite_tests: } 2023-05-31T21:35:38.091 INFO:tasks.rgw_multisite_tests: } 2023-05-31T21:35:38.092 INFO:tasks.rgw_multisite_tests: ] 2023-05-31T21:35:38.092 INFO:tasks.rgw_multisite_tests: }, 2023-05-31T21:35:38.092 INFO:tasks.rgw_multisite_tests: "full_sync": { 2023-05-31T21:35:38.092 INFO:tasks.rgw_multisite_tests: "total": 1, 2023-05-31T21:35:38.093 INFO:tasks.rgw_multisite_tests: "complete": 1 2023-05-31T21:35:38.093 INFO:tasks.rgw_multisite_tests: }, 2023-05-31T21:35:38.093 INFO:tasks.rgw_multisite_tests: "current_time": "2023-05-31T20:10:28Z" 2023-05-31T21:35:38.093 INFO:tasks.rgw_multisite_tests:} 2023-05-31T21:35:38.093 INFO:tasks.rgw_multisite_tests: 2023-05-31T21:35:38.094 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1685562631.516089_59.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-05-31T19:50:31.516089Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1685562970.274572_93.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-05-31T19:56:10.274572Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1685563571.700188_212.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-05-31T20:06:11.700188Z', 'realm_epoch': 2}}] 2023-05-31T21:35:38.094 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1685562631.516089_59.1', 1: '', 2: '1_1685562970.274572_93.1', 3: '1_1685563802.468126_296.1'} 2023-05-31T21:35:38.094 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1685562631.516089_59.1', 1: '', 2: '1_1685562970.274572_93.1', 3: '1_1685563571.700188_212.1'} 2023-05-31T21:35:38.094 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone a2 behind master: shard=3 master=1_1685563802.468126_296.1 target=1_1685563571.700188_212.1 2023-05-31T21:35:38.095 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-31T21:35:38.095 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-31T21:35:38.095 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-05-31T21:35:38.095 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: ignoring --setuser ceph since I am not root 2023-05-31T21:35:38.096 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-05-31T21:35:38.096 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: 2023-05-31T20:10:33.557+0000 7f31b188c780 20 RGW-SYNC:meta: read sync status 2023-05-31T21:35:38.096 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: 2023-05-31T20:10:33.558+0000 7f31b188c780 20 RGW-SYNC:meta: read sync status 2023-05-31T21:35:38.096 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2023-05-31T21:35:38.097 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "sync_status": { 2023-05-31T21:35:38.097 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "info": { 2023-05-31T21:35:38.097 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "status": "sync", 2023-05-31T21:35:38.098 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "num_shards": 4, 2023-05-31T21:35:38.098 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "period": "2e6123d2-d695-4350-9a01-61a455bc782b", 2023-05-31T21:35:38.098 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "realm_epoch": 2 2023-05-31T21:35:38.098 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2023-05-31T21:35:38.098 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "markers": [ 2023-05-31T21:35:38.099 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2023-05-31T21:35:38.099 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "key": 0, 2023-05-31T21:35:38.099 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "val": { 2023-05-31T21:35:38.099 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "state": 1, 2023-05-31T21:35:38.100 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "marker": "1_1685562631.516089_59.1", 2023-05-31T21:35:38.100 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "next_step_marker": "", 2023-05-31T21:35:38.100 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "total_entries": 1, 2023-05-31T21:35:38.100 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "pos": 0, 2023-05-31T21:35:38.101 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "timestamp": "2023-05-31T19:50:31.516089Z", 2023-05-31T21:35:38.101 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "realm_epoch": 2 2023-05-31T21:35:38.101 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2023-05-31T21:35:38.101 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2023-05-31T21:35:38.102 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2023-05-31T21:35:38.102 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "key": 1, 2023-05-31T21:35:38.102 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "val": { 2023-05-31T21:35:38.102 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "state": 1, 2023-05-31T21:35:38.102 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "marker": "", 2023-05-31T21:35:38.103 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "next_step_marker": "", 2023-05-31T21:35:38.103 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "total_entries": 0, 2023-05-31T21:35:38.103 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "pos": 0, 2023-05-31T21:35:38.103 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "timestamp": "0.000000", 2023-05-31T21:35:38.104 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "realm_epoch": 2 2023-05-31T21:35:38.104 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2023-05-31T21:35:38.104 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2023-05-31T21:35:38.104 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2023-05-31T21:35:38.105 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "key": 2, 2023-05-31T21:35:38.105 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "val": { 2023-05-31T21:35:38.105 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "state": 1, 2023-05-31T21:35:38.105 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "marker": "1_1685562970.274572_93.1", 2023-05-31T21:35:38.105 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "next_step_marker": "", 2023-05-31T21:35:38.106 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "total_entries": 0, 2023-05-31T21:35:38.106 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "pos": 0, 2023-05-31T21:35:38.106 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "timestamp": "2023-05-31T19:56:10.274572Z", 2023-05-31T21:35:38.106 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "realm_epoch": 2 2023-05-31T21:35:38.107 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2023-05-31T21:35:38.107 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2023-05-31T21:35:38.107 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2023-05-31T21:35:38.107 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "key": 3, 2023-05-31T21:35:38.107 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "val": { 2023-05-31T21:35:38.108 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "state": 1, 2023-05-31T21:35:38.108 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "marker": "1_1685563571.700188_212.1", 2023-05-31T21:35:38.108 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "next_step_marker": "", 2023-05-31T21:35:38.108 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "total_entries": 0, 2023-05-31T21:35:38.109 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "pos": 0, 2023-05-31T21:35:38.109 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "timestamp": "2023-05-31T20:06:11.700188Z", 2023-05-31T21:35:38.109 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "realm_epoch": 2 2023-05-31T21:35:38.110 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2023-05-31T21:35:38.110 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2023-05-31T21:35:38.110 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: ] 2023-05-31T21:35:38.110 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2023-05-31T21:35:38.110 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "full_sync": { 2023-05-31T21:35:38.111 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "total": 1, 2023-05-31T21:35:38.111 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "complete": 1 2023-05-31T21:35:38.111 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2023-05-31T21:35:38.111 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "current_time": "2023-05-31T20:10:33Z" 2023-05-31T21:35:38.112 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2023-05-31T21:35:38.112 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2023-05-31T21:35:38.112 INFO:tasks.rgw_multisite_tests: "sync_status": { 2023-05-31T21:35:38.112 INFO:tasks.rgw_multisite_tests: "info": { 2023-05-31T21:35:38.113 INFO:tasks.rgw_multisite_tests: "status": "sync", 2023-05-31T21:35:38.113 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2023-05-31T21:35:38.113 INFO:tasks.rgw_multisite_tests: "period": "2e6123d2-d695-4350-9a01-61a455bc782b", 2023-05-31T21:35:38.113 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-05-31T21:35:38.113 INFO:tasks.rgw_multisite_tests: }, 2023-05-31T21:35:38.114 INFO:tasks.rgw_multisite_tests: "markers": [ 2023-05-31T21:35:38.114 INFO:tasks.rgw_multisite_tests: { 2023-05-31T21:35:38.114 INFO:tasks.rgw_multisite_tests: "key": 0, 2023-05-31T21:35:38.114 INFO:tasks.rgw_multisite_tests: "val": { 2023-05-31T21:35:38.115 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-05-31T21:35:38.115 INFO:tasks.rgw_multisite_tests: "marker": "1_1685562631.516089_59.1", 2023-05-31T21:35:38.115 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-05-31T21:35:38.115 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2023-05-31T21:35:38.116 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-05-31T21:35:38.116 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-05-31T19:50:31.516089Z", 2023-05-31T21:35:38.116 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-05-31T21:35:38.116 INFO:tasks.rgw_multisite_tests: } 2023-05-31T21:35:38.117 INFO:tasks.rgw_multisite_tests: }, 2023-05-31T21:35:38.117 INFO:tasks.rgw_multisite_tests: { 2023-05-31T21:35:38.117 INFO:tasks.rgw_multisite_tests: "key": 1, 2023-05-31T21:35:38.117 INFO:tasks.rgw_multisite_tests: "val": { 2023-05-31T21:35:38.117 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-05-31T21:35:38.118 INFO:tasks.rgw_multisite_tests: "marker": "", 2023-05-31T21:35:38.118 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-05-31T21:35:38.118 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-05-31T21:35:38.118 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-05-31T21:35:38.119 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2023-05-31T21:35:38.119 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-05-31T21:35:38.119 INFO:tasks.rgw_multisite_tests: } 2023-05-31T21:35:38.119 INFO:tasks.rgw_multisite_tests: }, 2023-05-31T21:35:38.120 INFO:tasks.rgw_multisite_tests: { 2023-05-31T21:35:38.120 INFO:tasks.rgw_multisite_tests: "key": 2, 2023-05-31T21:35:38.120 INFO:tasks.rgw_multisite_tests: "val": { 2023-05-31T21:35:38.120 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-05-31T21:35:38.120 INFO:tasks.rgw_multisite_tests: "marker": "1_1685562970.274572_93.1", 2023-05-31T21:35:38.121 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-05-31T21:35:38.121 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-05-31T21:35:38.121 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-05-31T21:35:38.121 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-05-31T19:56:10.274572Z", 2023-05-31T21:35:38.122 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-05-31T21:35:38.122 INFO:tasks.rgw_multisite_tests: } 2023-05-31T21:35:38.122 INFO:tasks.rgw_multisite_tests: }, 2023-05-31T21:35:38.122 INFO:tasks.rgw_multisite_tests: { 2023-05-31T21:35:38.123 INFO:tasks.rgw_multisite_tests: "key": 3, 2023-05-31T21:35:38.123 INFO:tasks.rgw_multisite_tests: "val": { 2023-05-31T21:35:38.123 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-05-31T21:35:38.123 INFO:tasks.rgw_multisite_tests: "marker": "1_1685563571.700188_212.1", 2023-05-31T21:35:38.123 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-05-31T21:35:38.124 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-05-31T21:35:38.124 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-05-31T21:35:38.124 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-05-31T20:06:11.700188Z", 2023-05-31T21:35:38.124 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-05-31T21:35:38.125 INFO:tasks.rgw_multisite_tests: } 2023-05-31T21:35:38.125 INFO:tasks.rgw_multisite_tests: } 2023-05-31T21:35:38.125 INFO:tasks.rgw_multisite_tests: ] 2023-05-31T21:35:38.125 INFO:tasks.rgw_multisite_tests: }, 2023-05-31T21:35:38.126 INFO:tasks.rgw_multisite_tests: "full_sync": { 2023-05-31T21:35:38.126 INFO:tasks.rgw_multisite_tests: "total": 1, 2023-05-31T21:35:38.126 INFO:tasks.rgw_multisite_tests: "complete": 1 2023-05-31T21:35:38.126 INFO:tasks.rgw_multisite_tests: }, 2023-05-31T21:35:38.126 INFO:tasks.rgw_multisite_tests: "current_time": "2023-05-31T20:10:33Z" 2023-05-31T21:35:38.127 INFO:tasks.rgw_multisite_tests:} 2023-05-31T21:35:38.127 INFO:tasks.rgw_multisite_tests: 2023-05-31T21:35:38.127 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1685562631.516089_59.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-05-31T19:50:31.516089Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1685562970.274572_93.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-05-31T19:56:10.274572Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1685563571.700188_212.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-05-31T20:06:11.700188Z', 'realm_epoch': 2}}] 2023-05-31T21:35:38.127 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1685562631.516089_59.1', 1: '', 2: '1_1685562970.274572_93.1', 3: '1_1685563802.468126_296.1'} 2023-05-31T21:35:38.128 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1685562631.516089_59.1', 1: '', 2: '1_1685562970.274572_93.1', 3: '1_1685563571.700188_212.1'} 2023-05-31T21:35:38.128 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone a2 behind master: shard=3 master=1_1685563802.468126_296.1 target=1_1685563571.700188_212.1 2023-05-31T21:35:38.128 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-31T21:35:38.128 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-31T21:35:38.129 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-05-31T21:35:38.129 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: ignoring --setuser ceph since I am not root 2023-05-31T21:35:38.129 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-05-31T21:35:38.129 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: 2023-05-31T20:10:38.688+0000 7f54dbcab780 20 RGW-SYNC:meta: read sync status 2023-05-31T21:35:38.130 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: 2023-05-31T20:10:38.689+0000 7f54dbcab780 20 RGW-SYNC:meta: read sync status 2023-05-31T21:35:38.130 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2023-05-31T21:35:38.130 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "sync_status": { 2023-05-31T21:35:38.130 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "info": { 2023-05-31T21:35:38.131 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "status": "sync", 2023-05-31T21:35:38.131 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "num_shards": 4, 2023-05-31T21:35:38.131 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "period": "2e6123d2-d695-4350-9a01-61a455bc782b", 2023-05-31T21:35:38.131 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "realm_epoch": 2 2023-05-31T21:35:38.131 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2023-05-31T21:35:38.132 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "markers": [ 2023-05-31T21:35:38.132 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2023-05-31T21:35:38.132 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "key": 0, 2023-05-31T21:35:38.132 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "val": { 2023-05-31T21:35:38.133 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "state": 1, 2023-05-31T21:35:38.133 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "marker": "1_1685562631.516089_59.1", 2023-05-31T21:35:38.133 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "next_step_marker": "", 2023-05-31T21:35:38.133 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "total_entries": 1, 2023-05-31T21:35:38.134 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "pos": 0, 2023-05-31T21:35:38.134 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "timestamp": "2023-05-31T19:50:31.516089Z", 2023-05-31T21:35:38.134 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "realm_epoch": 2 2023-05-31T21:35:38.134 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2023-05-31T21:35:38.135 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2023-05-31T21:35:38.135 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2023-05-31T21:35:38.135 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "key": 1, 2023-05-31T21:35:38.135 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "val": { 2023-05-31T21:35:38.135 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "state": 1, 2023-05-31T21:35:38.136 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "marker": "", 2023-05-31T21:35:38.136 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "next_step_marker": "", 2023-05-31T21:35:38.136 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "total_entries": 0, 2023-05-31T21:35:38.136 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "pos": 0, 2023-05-31T21:35:38.137 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "timestamp": "0.000000", 2023-05-31T21:35:38.137 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "realm_epoch": 2 2023-05-31T21:35:38.137 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2023-05-31T21:35:38.137 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2023-05-31T21:35:38.138 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2023-05-31T21:35:38.138 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "key": 2, 2023-05-31T21:35:38.138 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "val": { 2023-05-31T21:35:38.138 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "state": 1, 2023-05-31T21:35:38.139 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "marker": "1_1685562970.274572_93.1", 2023-05-31T21:35:38.139 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "next_step_marker": "", 2023-05-31T21:35:38.139 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "total_entries": 0, 2023-05-31T21:35:38.139 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "pos": 0, 2023-05-31T21:35:38.140 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "timestamp": "2023-05-31T19:56:10.274572Z", 2023-05-31T21:35:38.140 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "realm_epoch": 2 2023-05-31T21:35:38.140 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2023-05-31T21:35:38.140 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2023-05-31T21:35:38.141 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2023-05-31T21:35:38.141 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "key": 3, 2023-05-31T21:35:38.141 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "val": { 2023-05-31T21:35:38.141 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "state": 1, 2023-05-31T21:35:38.141 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "marker": "1_1685563571.700188_212.1", 2023-05-31T21:35:38.142 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "next_step_marker": "", 2023-05-31T21:35:38.142 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "total_entries": 0, 2023-05-31T21:35:38.142 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "pos": 0, 2023-05-31T21:35:38.142 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "timestamp": "2023-05-31T20:06:11.700188Z", 2023-05-31T21:35:38.143 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "realm_epoch": 2 2023-05-31T21:35:38.143 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2023-05-31T21:35:38.143 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2023-05-31T21:35:38.143 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: ] 2023-05-31T21:35:38.144 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2023-05-31T21:35:38.144 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "full_sync": { 2023-05-31T21:35:38.144 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "total": 1, 2023-05-31T21:35:38.144 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "complete": 1 2023-05-31T21:35:38.144 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2023-05-31T21:35:38.145 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "current_time": "2023-05-31T20:10:38Z" 2023-05-31T21:35:38.145 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2023-05-31T21:35:38.145 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2023-05-31T21:35:38.145 INFO:tasks.rgw_multisite_tests: "sync_status": { 2023-05-31T21:35:38.146 INFO:tasks.rgw_multisite_tests: "info": { 2023-05-31T21:35:38.146 INFO:tasks.rgw_multisite_tests: "status": "sync", 2023-05-31T21:35:38.146 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2023-05-31T21:35:38.146 INFO:tasks.rgw_multisite_tests: "period": "2e6123d2-d695-4350-9a01-61a455bc782b", 2023-05-31T21:35:38.147 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-05-31T21:35:38.147 INFO:tasks.rgw_multisite_tests: }, 2023-05-31T21:35:38.147 INFO:tasks.rgw_multisite_tests: "markers": [ 2023-05-31T21:35:38.147 INFO:tasks.rgw_multisite_tests: { 2023-05-31T21:35:38.148 INFO:tasks.rgw_multisite_tests: "key": 0, 2023-05-31T21:35:38.148 INFO:tasks.rgw_multisite_tests: "val": { 2023-05-31T21:35:38.148 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-05-31T21:35:38.148 INFO:tasks.rgw_multisite_tests: "marker": "1_1685562631.516089_59.1", 2023-05-31T21:35:38.148 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-05-31T21:35:38.149 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2023-05-31T21:35:38.149 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-05-31T21:35:38.149 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-05-31T19:50:31.516089Z", 2023-05-31T21:35:38.149 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-05-31T21:35:38.150 INFO:tasks.rgw_multisite_tests: } 2023-05-31T21:35:38.150 INFO:tasks.rgw_multisite_tests: }, 2023-05-31T21:35:38.150 INFO:tasks.rgw_multisite_tests: { 2023-05-31T21:35:38.150 INFO:tasks.rgw_multisite_tests: "key": 1, 2023-05-31T21:35:38.151 INFO:tasks.rgw_multisite_tests: "val": { 2023-05-31T21:35:38.151 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-05-31T21:35:38.151 INFO:tasks.rgw_multisite_tests: "marker": "", 2023-05-31T21:35:38.151 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-05-31T21:35:38.151 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-05-31T21:35:38.152 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-05-31T21:35:38.152 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2023-05-31T21:35:38.152 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-05-31T21:35:38.152 INFO:tasks.rgw_multisite_tests: } 2023-05-31T21:35:38.153 INFO:tasks.rgw_multisite_tests: }, 2023-05-31T21:35:38.153 INFO:tasks.rgw_multisite_tests: { 2023-05-31T21:35:38.153 INFO:tasks.rgw_multisite_tests: "key": 2, 2023-05-31T21:35:38.153 INFO:tasks.rgw_multisite_tests: "val": { 2023-05-31T21:35:38.154 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-05-31T21:35:38.154 INFO:tasks.rgw_multisite_tests: "marker": "1_1685562970.274572_93.1", 2023-05-31T21:35:38.154 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-05-31T21:35:38.154 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-05-31T21:35:38.154 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-05-31T21:35:38.155 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-05-31T19:56:10.274572Z", 2023-05-31T21:35:38.155 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-05-31T21:35:38.155 INFO:tasks.rgw_multisite_tests: } 2023-05-31T21:35:38.155 INFO:tasks.rgw_multisite_tests: }, 2023-05-31T21:35:38.156 INFO:tasks.rgw_multisite_tests: { 2023-05-31T21:35:38.156 INFO:tasks.rgw_multisite_tests: "key": 3, 2023-05-31T21:35:38.156 INFO:tasks.rgw_multisite_tests: "val": { 2023-05-31T21:35:38.156 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-05-31T21:35:38.157 INFO:tasks.rgw_multisite_tests: "marker": "1_1685563571.700188_212.1", 2023-05-31T21:35:38.157 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-05-31T21:35:38.157 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-05-31T21:35:38.157 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-05-31T21:35:38.158 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-05-31T20:06:11.700188Z", 2023-05-31T21:35:38.158 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-05-31T21:35:38.158 INFO:tasks.rgw_multisite_tests: } 2023-05-31T21:35:38.158 INFO:tasks.rgw_multisite_tests: } 2023-05-31T21:35:38.159 INFO:tasks.rgw_multisite_tests: ] 2023-05-31T21:35:38.159 INFO:tasks.rgw_multisite_tests: }, 2023-05-31T21:35:38.159 INFO:tasks.rgw_multisite_tests: "full_sync": { 2023-05-31T21:35:38.159 INFO:tasks.rgw_multisite_tests: "total": 1, 2023-05-31T21:35:38.160 INFO:tasks.rgw_multisite_tests: "complete": 1 2023-05-31T21:35:38.160 INFO:tasks.rgw_multisite_tests: }, 2023-05-31T21:35:38.160 INFO:tasks.rgw_multisite_tests: "current_time": "2023-05-31T20:10:38Z" 2023-05-31T21:35:38.160 INFO:tasks.rgw_multisite_tests:} 2023-05-31T21:35:38.160 INFO:tasks.rgw_multisite_tests: 2023-05-31T21:35:38.161 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1685562631.516089_59.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-05-31T19:50:31.516089Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1685562970.274572_93.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-05-31T19:56:10.274572Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1685563571.700188_212.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-05-31T20:06:11.700188Z', 'realm_epoch': 2}}] 2023-05-31T21:35:38.161 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1685562631.516089_59.1', 1: '', 2: '1_1685562970.274572_93.1', 3: '1_1685563802.468126_296.1'} 2023-05-31T21:35:38.161 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1685562631.516089_59.1', 1: '', 2: '1_1685562970.274572_93.1', 3: '1_1685563571.700188_212.1'} 2023-05-31T21:35:38.161 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone a2 behind master: shard=3 master=1_1685563802.468126_296.1 target=1_1685563571.700188_212.1 2023-05-31T21:35:38.162 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-31T21:35:38.162 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-31T21:35:38.162 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-05-31T21:35:38.162 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: ignoring --setuser ceph since I am not root 2023-05-31T21:35:38.163 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-05-31T21:35:38.163 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: 2023-05-31T20:10:43.829+0000 7f02164ea780 20 RGW-SYNC:meta: read sync status 2023-05-31T21:35:38.163 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: 2023-05-31T20:10:43.830+0000 7f02164ea780 20 RGW-SYNC:meta: read sync status 2023-05-31T21:35:38.163 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2023-05-31T21:35:38.164 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "sync_status": { 2023-05-31T21:35:38.164 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "info": { 2023-05-31T21:35:38.164 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "status": "sync", 2023-05-31T21:35:38.164 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "num_shards": 4, 2023-05-31T21:35:38.165 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "period": "2e6123d2-d695-4350-9a01-61a455bc782b", 2023-05-31T21:35:38.165 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "realm_epoch": 2 2023-05-31T21:35:38.165 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2023-05-31T21:35:38.165 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "markers": [ 2023-05-31T21:35:38.166 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2023-05-31T21:35:38.166 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "key": 0, 2023-05-31T21:35:38.166 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "val": { 2023-05-31T21:35:38.166 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "state": 1, 2023-05-31T21:35:38.166 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "marker": "1_1685562631.516089_59.1", 2023-05-31T21:35:38.167 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "next_step_marker": "", 2023-05-31T21:35:38.167 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "total_entries": 1, 2023-05-31T21:35:38.167 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "pos": 0, 2023-05-31T21:35:38.167 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "timestamp": "2023-05-31T19:50:31.516089Z", 2023-05-31T21:35:38.168 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "realm_epoch": 2 2023-05-31T21:35:38.168 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2023-05-31T21:35:38.168 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2023-05-31T21:35:38.168 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2023-05-31T21:35:38.169 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "key": 1, 2023-05-31T21:35:38.169 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "val": { 2023-05-31T21:35:38.169 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "state": 1, 2023-05-31T21:35:38.169 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "marker": "", 2023-05-31T21:35:38.170 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "next_step_marker": "", 2023-05-31T21:35:38.170 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "total_entries": 0, 2023-05-31T21:35:38.170 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "pos": 0, 2023-05-31T21:35:38.170 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "timestamp": "0.000000", 2023-05-31T21:35:38.171 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "realm_epoch": 2 2023-05-31T21:35:38.171 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2023-05-31T21:35:38.171 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2023-05-31T21:35:38.171 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2023-05-31T21:35:38.171 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "key": 2, 2023-05-31T21:35:38.172 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "val": { 2023-05-31T21:35:38.172 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "state": 1, 2023-05-31T21:35:38.172 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "marker": "1_1685562970.274572_93.1", 2023-05-31T21:35:38.172 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "next_step_marker": "", 2023-05-31T21:35:38.173 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "total_entries": 0, 2023-05-31T21:35:38.173 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "pos": 0, 2023-05-31T21:35:38.173 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "timestamp": "2023-05-31T19:56:10.274572Z", 2023-05-31T21:35:38.174 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "realm_epoch": 2 2023-05-31T21:35:38.174 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2023-05-31T21:35:38.174 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2023-05-31T21:35:38.174 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2023-05-31T21:35:38.175 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "key": 3, 2023-05-31T21:35:38.175 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "val": { 2023-05-31T21:35:38.175 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "state": 1, 2023-05-31T21:35:38.175 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "marker": "1_1685563571.700188_212.1", 2023-05-31T21:35:38.175 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "next_step_marker": "", 2023-05-31T21:35:38.176 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "total_entries": 0, 2023-05-31T21:35:38.176 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "pos": 0, 2023-05-31T21:35:38.176 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "timestamp": "2023-05-31T20:06:11.700188Z", 2023-05-31T21:35:38.176 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "realm_epoch": 2 2023-05-31T21:35:38.177 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2023-05-31T21:35:38.177 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2023-05-31T21:35:38.177 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: ] 2023-05-31T21:35:38.177 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2023-05-31T21:35:38.178 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "full_sync": { 2023-05-31T21:35:38.178 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "total": 1, 2023-05-31T21:35:38.178 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "complete": 1 2023-05-31T21:35:38.178 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2023-05-31T21:35:38.179 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "current_time": "2023-05-31T20:10:43Z" 2023-05-31T21:35:38.179 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2023-05-31T21:35:38.179 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2023-05-31T21:35:38.179 INFO:tasks.rgw_multisite_tests: "sync_status": { 2023-05-31T21:35:38.179 INFO:tasks.rgw_multisite_tests: "info": { 2023-05-31T21:35:38.180 INFO:tasks.rgw_multisite_tests: "status": "sync", 2023-05-31T21:35:38.180 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2023-05-31T21:35:38.180 INFO:tasks.rgw_multisite_tests: "period": "2e6123d2-d695-4350-9a01-61a455bc782b", 2023-05-31T21:35:38.180 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-05-31T21:35:38.181 INFO:tasks.rgw_multisite_tests: }, 2023-05-31T21:35:38.181 INFO:tasks.rgw_multisite_tests: "markers": [ 2023-05-31T21:35:38.181 INFO:tasks.rgw_multisite_tests: { 2023-05-31T21:35:38.181 INFO:tasks.rgw_multisite_tests: "key": 0, 2023-05-31T21:35:38.182 INFO:tasks.rgw_multisite_tests: "val": { 2023-05-31T21:35:38.182 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-05-31T21:35:38.182 INFO:tasks.rgw_multisite_tests: "marker": "1_1685562631.516089_59.1", 2023-05-31T21:35:38.182 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-05-31T21:35:38.182 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2023-05-31T21:35:38.183 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-05-31T21:35:38.183 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-05-31T19:50:31.516089Z", 2023-05-31T21:35:38.183 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-05-31T21:35:38.183 INFO:tasks.rgw_multisite_tests: } 2023-05-31T21:35:38.184 INFO:tasks.rgw_multisite_tests: }, 2023-05-31T21:35:38.184 INFO:tasks.rgw_multisite_tests: { 2023-05-31T21:35:38.184 INFO:tasks.rgw_multisite_tests: "key": 1, 2023-05-31T21:35:38.184 INFO:tasks.rgw_multisite_tests: "val": { 2023-05-31T21:35:38.185 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-05-31T21:35:38.185 INFO:tasks.rgw_multisite_tests: "marker": "", 2023-05-31T21:35:38.185 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-05-31T21:35:38.185 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-05-31T21:35:38.185 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-05-31T21:35:38.186 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2023-05-31T21:35:38.186 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-05-31T21:35:38.186 INFO:tasks.rgw_multisite_tests: } 2023-05-31T21:35:38.186 INFO:tasks.rgw_multisite_tests: }, 2023-05-31T21:35:38.187 INFO:tasks.rgw_multisite_tests: { 2023-05-31T21:35:38.187 INFO:tasks.rgw_multisite_tests: "key": 2, 2023-05-31T21:35:38.187 INFO:tasks.rgw_multisite_tests: "val": { 2023-05-31T21:35:38.187 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-05-31T21:35:38.188 INFO:tasks.rgw_multisite_tests: "marker": "1_1685562970.274572_93.1", 2023-05-31T21:35:38.188 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-05-31T21:35:38.188 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-05-31T21:35:38.188 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-05-31T21:35:38.189 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-05-31T19:56:10.274572Z", 2023-05-31T21:35:38.189 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-05-31T21:35:38.189 INFO:tasks.rgw_multisite_tests: } 2023-05-31T21:35:38.189 INFO:tasks.rgw_multisite_tests: }, 2023-05-31T21:35:38.189 INFO:tasks.rgw_multisite_tests: { 2023-05-31T21:35:38.190 INFO:tasks.rgw_multisite_tests: "key": 3, 2023-05-31T21:35:38.190 INFO:tasks.rgw_multisite_tests: "val": { 2023-05-31T21:35:38.190 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-05-31T21:35:38.190 INFO:tasks.rgw_multisite_tests: "marker": "1_1685563571.700188_212.1", 2023-05-31T21:35:38.191 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-05-31T21:35:38.191 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-05-31T21:35:38.191 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-05-31T21:35:38.191 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-05-31T20:06:11.700188Z", 2023-05-31T21:35:38.192 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-05-31T21:35:38.192 INFO:tasks.rgw_multisite_tests: } 2023-05-31T21:35:38.192 INFO:tasks.rgw_multisite_tests: } 2023-05-31T21:35:38.192 INFO:tasks.rgw_multisite_tests: ] 2023-05-31T21:35:38.192 INFO:tasks.rgw_multisite_tests: }, 2023-05-31T21:35:38.193 INFO:tasks.rgw_multisite_tests: "full_sync": { 2023-05-31T21:35:38.193 INFO:tasks.rgw_multisite_tests: "total": 1, 2023-05-31T21:35:38.193 INFO:tasks.rgw_multisite_tests: "complete": 1 2023-05-31T21:35:38.193 INFO:tasks.rgw_multisite_tests: }, 2023-05-31T21:35:38.194 INFO:tasks.rgw_multisite_tests: "current_time": "2023-05-31T20:10:43Z" 2023-05-31T21:35:38.194 INFO:tasks.rgw_multisite_tests:} 2023-05-31T21:35:38.194 INFO:tasks.rgw_multisite_tests: 2023-05-31T21:35:38.194 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1685562631.516089_59.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-05-31T19:50:31.516089Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1685562970.274572_93.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-05-31T19:56:10.274572Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1685563571.700188_212.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-05-31T20:06:11.700188Z', 'realm_epoch': 2}}] 2023-05-31T21:35:38.195 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1685562631.516089_59.1', 1: '', 2: '1_1685562970.274572_93.1', 3: '1_1685563802.468126_296.1'} 2023-05-31T21:35:38.195 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1685562631.516089_59.1', 1: '', 2: '1_1685562970.274572_93.1', 3: '1_1685563571.700188_212.1'} 2023-05-31T21:35:38.195 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone a2 behind master: shard=3 master=1_1685563802.468126_296.1 target=1_1685563571.700188_212.1 2023-05-31T21:35:38.195 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-31T21:35:38.195 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-31T21:35:38.196 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-05-31T21:35:38.196 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: ignoring --setuser ceph since I am not root 2023-05-31T21:35:38.196 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-05-31T21:35:38.197 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: 2023-05-31T20:10:48.977+0000 7f7b91c2b780 20 RGW-SYNC:meta: read sync status 2023-05-31T21:35:38.197 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: 2023-05-31T20:10:48.978+0000 7f7b91c2b780 20 RGW-SYNC:meta: read sync status 2023-05-31T21:35:38.197 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2023-05-31T21:35:38.197 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "sync_status": { 2023-05-31T21:35:38.198 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "info": { 2023-05-31T21:35:38.198 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "status": "sync", 2023-05-31T21:35:38.198 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "num_shards": 4, 2023-05-31T21:35:38.198 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "period": "2e6123d2-d695-4350-9a01-61a455bc782b", 2023-05-31T21:35:38.199 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "realm_epoch": 2 2023-05-31T21:35:38.199 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2023-05-31T21:35:38.199 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "markers": [ 2023-05-31T21:35:38.199 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2023-05-31T21:35:38.200 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "key": 0, 2023-05-31T21:35:38.200 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "val": { 2023-05-31T21:35:38.200 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "state": 1, 2023-05-31T21:35:38.200 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "marker": "1_1685562631.516089_59.1", 2023-05-31T21:35:38.201 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "next_step_marker": "", 2023-05-31T21:35:38.201 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "total_entries": 1, 2023-05-31T21:35:38.201 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "pos": 0, 2023-05-31T21:35:38.201 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "timestamp": "2023-05-31T19:50:31.516089Z", 2023-05-31T21:35:38.201 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "realm_epoch": 2 2023-05-31T21:35:38.202 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2023-05-31T21:35:38.202 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2023-05-31T21:35:38.202 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2023-05-31T21:35:38.202 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "key": 1, 2023-05-31T21:35:38.203 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "val": { 2023-05-31T21:35:38.203 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "state": 1, 2023-05-31T21:35:38.203 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "marker": "", 2023-05-31T21:35:38.203 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "next_step_marker": "", 2023-05-31T21:35:38.204 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "total_entries": 0, 2023-05-31T21:35:38.204 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "pos": 0, 2023-05-31T21:35:38.204 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "timestamp": "0.000000", 2023-05-31T21:35:38.204 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "realm_epoch": 2 2023-05-31T21:35:38.204 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2023-05-31T21:35:38.205 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2023-05-31T21:35:38.205 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2023-05-31T21:35:38.205 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "key": 2, 2023-05-31T21:35:38.205 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "val": { 2023-05-31T21:35:38.206 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "state": 1, 2023-05-31T21:35:38.206 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "marker": "1_1685562970.274572_93.1", 2023-05-31T21:35:38.206 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "next_step_marker": "", 2023-05-31T21:35:38.206 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "total_entries": 0, 2023-05-31T21:35:38.206 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "pos": 0, 2023-05-31T21:35:38.207 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "timestamp": "2023-05-31T19:56:10.274572Z", 2023-05-31T21:35:38.207 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "realm_epoch": 2 2023-05-31T21:35:38.207 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2023-05-31T21:35:38.207 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2023-05-31T21:35:38.208 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2023-05-31T21:35:38.208 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "key": 3, 2023-05-31T21:35:38.208 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "val": { 2023-05-31T21:35:38.208 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "state": 1, 2023-05-31T21:35:38.209 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "marker": "1_1685563571.700188_212.1", 2023-05-31T21:35:38.209 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "next_step_marker": "", 2023-05-31T21:35:38.209 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "total_entries": 0, 2023-05-31T21:35:38.210 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "pos": 0, 2023-05-31T21:35:38.210 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "timestamp": "2023-05-31T20:06:11.700188Z", 2023-05-31T21:35:38.210 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "realm_epoch": 2 2023-05-31T21:35:38.210 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2023-05-31T21:35:38.210 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2023-05-31T21:35:38.211 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: ] 2023-05-31T21:35:38.211 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2023-05-31T21:35:38.211 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "full_sync": { 2023-05-31T21:35:38.211 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "total": 1, 2023-05-31T21:35:38.212 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "complete": 1 2023-05-31T21:35:38.212 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2023-05-31T21:35:38.212 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "current_time": "2023-05-31T20:10:48Z" 2023-05-31T21:35:38.212 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2023-05-31T21:35:38.213 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2023-05-31T21:35:38.213 INFO:tasks.rgw_multisite_tests: "sync_status": { 2023-05-31T21:35:38.213 INFO:tasks.rgw_multisite_tests: "info": { 2023-05-31T21:35:38.213 INFO:tasks.rgw_multisite_tests: "status": "sync", 2023-05-31T21:35:38.213 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2023-05-31T21:35:38.214 INFO:tasks.rgw_multisite_tests: "period": "2e6123d2-d695-4350-9a01-61a455bc782b", 2023-05-31T21:35:38.214 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-05-31T21:35:38.214 INFO:tasks.rgw_multisite_tests: }, 2023-05-31T21:35:38.214 INFO:tasks.rgw_multisite_tests: "markers": [ 2023-05-31T21:35:38.215 INFO:tasks.rgw_multisite_tests: { 2023-05-31T21:35:38.215 INFO:tasks.rgw_multisite_tests: "key": 0, 2023-05-31T21:35:38.215 INFO:tasks.rgw_multisite_tests: "val": { 2023-05-31T21:35:38.215 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-05-31T21:35:38.216 INFO:tasks.rgw_multisite_tests: "marker": "1_1685562631.516089_59.1", 2023-05-31T21:35:38.216 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-05-31T21:35:38.216 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2023-05-31T21:35:38.216 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-05-31T21:35:38.216 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-05-31T19:50:31.516089Z", 2023-05-31T21:35:38.217 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-05-31T21:35:38.217 INFO:tasks.rgw_multisite_tests: } 2023-05-31T21:35:38.217 INFO:tasks.rgw_multisite_tests: }, 2023-05-31T21:35:38.217 INFO:tasks.rgw_multisite_tests: { 2023-05-31T21:35:38.218 INFO:tasks.rgw_multisite_tests: "key": 1, 2023-05-31T21:35:38.218 INFO:tasks.rgw_multisite_tests: "val": { 2023-05-31T21:35:38.218 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-05-31T21:35:38.218 INFO:tasks.rgw_multisite_tests: "marker": "", 2023-05-31T21:35:38.219 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-05-31T21:35:38.219 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-05-31T21:35:38.219 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-05-31T21:35:38.219 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2023-05-31T21:35:38.220 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-05-31T21:35:38.220 INFO:tasks.rgw_multisite_tests: } 2023-05-31T21:35:38.220 INFO:tasks.rgw_multisite_tests: }, 2023-05-31T21:35:38.220 INFO:tasks.rgw_multisite_tests: { 2023-05-31T21:35:38.220 INFO:tasks.rgw_multisite_tests: "key": 2, 2023-05-31T21:35:38.221 INFO:tasks.rgw_multisite_tests: "val": { 2023-05-31T21:35:38.221 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-05-31T21:35:38.221 INFO:tasks.rgw_multisite_tests: "marker": "1_1685562970.274572_93.1", 2023-05-31T21:35:38.221 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-05-31T21:35:38.222 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-05-31T21:35:38.222 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-05-31T21:35:38.222 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-05-31T19:56:10.274572Z", 2023-05-31T21:35:38.222 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-05-31T21:35:38.222 INFO:tasks.rgw_multisite_tests: } 2023-05-31T21:35:38.223 INFO:tasks.rgw_multisite_tests: }, 2023-05-31T21:35:38.223 INFO:tasks.rgw_multisite_tests: { 2023-05-31T21:35:38.223 INFO:tasks.rgw_multisite_tests: "key": 3, 2023-05-31T21:35:38.223 INFO:tasks.rgw_multisite_tests: "val": { 2023-05-31T21:35:38.224 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-05-31T21:35:38.224 INFO:tasks.rgw_multisite_tests: "marker": "1_1685563571.700188_212.1", 2023-05-31T21:35:38.224 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-05-31T21:35:38.224 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-05-31T21:35:38.225 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-05-31T21:35:38.225 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-05-31T20:06:11.700188Z", 2023-05-31T21:35:38.225 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-05-31T21:35:38.225 INFO:tasks.rgw_multisite_tests: } 2023-05-31T21:35:38.225 INFO:tasks.rgw_multisite_tests: } 2023-05-31T21:35:38.226 INFO:tasks.rgw_multisite_tests: ] 2023-05-31T21:35:38.226 INFO:tasks.rgw_multisite_tests: }, 2023-05-31T21:35:38.226 INFO:tasks.rgw_multisite_tests: "full_sync": { 2023-05-31T21:35:38.226 INFO:tasks.rgw_multisite_tests: "total": 1, 2023-05-31T21:35:38.227 INFO:tasks.rgw_multisite_tests: "complete": 1 2023-05-31T21:35:38.227 INFO:tasks.rgw_multisite_tests: }, 2023-05-31T21:35:38.227 INFO:tasks.rgw_multisite_tests: "current_time": "2023-05-31T20:10:48Z" 2023-05-31T21:35:38.227 INFO:tasks.rgw_multisite_tests:} 2023-05-31T21:35:38.228 INFO:tasks.rgw_multisite_tests: 2023-05-31T21:35:38.228 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1685562631.516089_59.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-05-31T19:50:31.516089Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1685562970.274572_93.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-05-31T19:56:10.274572Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1685563571.700188_212.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-05-31T20:06:11.700188Z', 'realm_epoch': 2}}] 2023-05-31T21:35:38.228 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1685562631.516089_59.1', 1: '', 2: '1_1685562970.274572_93.1', 3: '1_1685563802.468126_296.1'} 2023-05-31T21:35:38.228 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1685562631.516089_59.1', 1: '', 2: '1_1685562970.274572_93.1', 3: '1_1685563571.700188_212.1'} 2023-05-31T21:35:38.229 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone a2 behind master: shard=3 master=1_1685563802.468126_296.1 target=1_1685563571.700188_212.1 2023-05-31T21:35:38.229 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-31T21:35:38.229 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-31T21:35:38.229 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-05-31T21:35:38.229 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: ignoring --setuser ceph since I am not root 2023-05-31T21:35:38.230 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-05-31T21:35:38.230 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: 2023-05-31T20:10:54.114+0000 7f01ff776780 20 RGW-SYNC:meta: read sync status 2023-05-31T21:35:38.230 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: 2023-05-31T20:10:54.115+0000 7f01ff776780 20 RGW-SYNC:meta: read sync status 2023-05-31T21:35:38.230 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2023-05-31T21:35:38.231 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "sync_status": { 2023-05-31T21:35:38.231 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "info": { 2023-05-31T21:35:38.231 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "status": "sync", 2023-05-31T21:35:38.231 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "num_shards": 4, 2023-05-31T21:35:38.232 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "period": "2e6123d2-d695-4350-9a01-61a455bc782b", 2023-05-31T21:35:38.232 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "realm_epoch": 2 2023-05-31T21:35:38.232 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2023-05-31T21:35:38.232 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "markers": [ 2023-05-31T21:35:38.232 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2023-05-31T21:35:38.233 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "key": 0, 2023-05-31T21:35:38.233 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "val": { 2023-05-31T21:35:38.233 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "state": 1, 2023-05-31T21:35:38.233 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "marker": "1_1685562631.516089_59.1", 2023-05-31T21:35:38.234 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "next_step_marker": "", 2023-05-31T21:35:38.234 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "total_entries": 1, 2023-05-31T21:35:38.234 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "pos": 0, 2023-05-31T21:35:38.234 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "timestamp": "2023-05-31T19:50:31.516089Z", 2023-05-31T21:35:38.235 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "realm_epoch": 2 2023-05-31T21:35:38.235 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2023-05-31T21:35:38.235 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2023-05-31T21:35:38.235 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2023-05-31T21:35:38.235 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "key": 1, 2023-05-31T21:35:38.236 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "val": { 2023-05-31T21:35:38.236 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "state": 1, 2023-05-31T21:35:38.236 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "marker": "", 2023-05-31T21:35:38.236 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "next_step_marker": "", 2023-05-31T21:35:38.237 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "total_entries": 0, 2023-05-31T21:35:38.237 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "pos": 0, 2023-05-31T21:35:38.237 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "timestamp": "0.000000", 2023-05-31T21:35:38.237 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "realm_epoch": 2 2023-05-31T21:35:38.238 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2023-05-31T21:35:38.238 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2023-05-31T21:35:38.238 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2023-05-31T21:35:38.238 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "key": 2, 2023-05-31T21:35:38.239 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "val": { 2023-05-31T21:35:38.239 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "state": 1, 2023-05-31T21:35:38.239 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "marker": "1_1685562970.274572_93.1", 2023-05-31T21:35:38.239 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "next_step_marker": "", 2023-05-31T21:35:38.239 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "total_entries": 0, 2023-05-31T21:35:38.240 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "pos": 0, 2023-05-31T21:35:38.240 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "timestamp": "2023-05-31T19:56:10.274572Z", 2023-05-31T21:35:38.240 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "realm_epoch": 2 2023-05-31T21:35:38.240 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2023-05-31T21:35:38.241 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2023-05-31T21:35:38.241 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2023-05-31T21:35:38.241 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "key": 3, 2023-05-31T21:35:38.241 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "val": { 2023-05-31T21:35:38.242 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "state": 1, 2023-05-31T21:35:38.242 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "marker": "1_1685563571.700188_212.1", 2023-05-31T21:35:38.242 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "next_step_marker": "", 2023-05-31T21:35:38.242 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "total_entries": 0, 2023-05-31T21:35:38.242 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "pos": 0, 2023-05-31T21:35:38.243 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "timestamp": "2023-05-31T20:06:11.700188Z", 2023-05-31T21:35:38.243 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "realm_epoch": 2 2023-05-31T21:35:38.243 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2023-05-31T21:35:38.243 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2023-05-31T21:35:38.244 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: ] 2023-05-31T21:35:38.244 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2023-05-31T21:35:38.244 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "full_sync": { 2023-05-31T21:35:38.244 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "total": 1, 2023-05-31T21:35:38.245 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "complete": 1 2023-05-31T21:35:38.245 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2023-05-31T21:35:38.245 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "current_time": "2023-05-31T20:10:54Z" 2023-05-31T21:35:38.245 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2023-05-31T21:35:38.245 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2023-05-31T21:35:38.246 INFO:tasks.rgw_multisite_tests: "sync_status": { 2023-05-31T21:35:38.246 INFO:tasks.rgw_multisite_tests: "info": { 2023-05-31T21:35:38.246 INFO:tasks.rgw_multisite_tests: "status": "sync", 2023-05-31T21:35:38.246 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2023-05-31T21:35:38.247 INFO:tasks.rgw_multisite_tests: "period": "2e6123d2-d695-4350-9a01-61a455bc782b", 2023-05-31T21:35:38.247 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-05-31T21:35:38.247 INFO:tasks.rgw_multisite_tests: }, 2023-05-31T21:35:38.247 INFO:tasks.rgw_multisite_tests: "markers": [ 2023-05-31T21:35:38.248 INFO:tasks.rgw_multisite_tests: { 2023-05-31T21:35:38.248 INFO:tasks.rgw_multisite_tests: "key": 0, 2023-05-31T21:35:38.248 INFO:tasks.rgw_multisite_tests: "val": { 2023-05-31T21:35:38.248 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-05-31T21:35:38.249 INFO:tasks.rgw_multisite_tests: "marker": "1_1685562631.516089_59.1", 2023-05-31T21:35:38.249 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-05-31T21:35:38.249 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2023-05-31T21:35:38.249 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-05-31T21:35:38.250 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-05-31T19:50:31.516089Z", 2023-05-31T21:35:38.250 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-05-31T21:35:38.250 INFO:tasks.rgw_multisite_tests: } 2023-05-31T21:35:38.250 INFO:tasks.rgw_multisite_tests: }, 2023-05-31T21:35:38.250 INFO:tasks.rgw_multisite_tests: { 2023-05-31T21:35:38.251 INFO:tasks.rgw_multisite_tests: "key": 1, 2023-05-31T21:35:38.251 INFO:tasks.rgw_multisite_tests: "val": { 2023-05-31T21:35:38.251 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-05-31T21:35:38.251 INFO:tasks.rgw_multisite_tests: "marker": "", 2023-05-31T21:35:38.252 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-05-31T21:35:38.252 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-05-31T21:35:38.252 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-05-31T21:35:38.252 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2023-05-31T21:35:38.253 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-05-31T21:35:38.253 INFO:tasks.rgw_multisite_tests: } 2023-05-31T21:35:38.253 INFO:tasks.rgw_multisite_tests: }, 2023-05-31T21:35:38.253 INFO:tasks.rgw_multisite_tests: { 2023-05-31T21:35:38.253 INFO:tasks.rgw_multisite_tests: "key": 2, 2023-05-31T21:35:38.254 INFO:tasks.rgw_multisite_tests: "val": { 2023-05-31T21:35:38.254 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-05-31T21:35:38.254 INFO:tasks.rgw_multisite_tests: "marker": "1_1685562970.274572_93.1", 2023-05-31T21:35:38.254 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-05-31T21:35:38.255 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-05-31T21:35:38.255 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-05-31T21:35:38.256 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-05-31T19:56:10.274572Z", 2023-05-31T21:35:38.256 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-05-31T21:35:38.256 INFO:tasks.rgw_multisite_tests: } 2023-05-31T21:35:38.257 INFO:tasks.rgw_multisite_tests: }, 2023-05-31T21:35:38.257 INFO:tasks.rgw_multisite_tests: { 2023-05-31T21:35:38.258 INFO:tasks.rgw_multisite_tests: "key": 3, 2023-05-31T21:35:38.258 INFO:tasks.rgw_multisite_tests: "val": { 2023-05-31T21:35:38.259 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-05-31T21:35:38.259 INFO:tasks.rgw_multisite_tests: "marker": "1_1685563571.700188_212.1", 2023-05-31T21:35:38.260 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-05-31T21:35:38.260 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-05-31T21:35:38.261 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-05-31T21:35:38.261 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-05-31T20:06:11.700188Z", 2023-05-31T21:35:38.261 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-05-31T21:35:38.261 INFO:tasks.rgw_multisite_tests: } 2023-05-31T21:35:38.262 INFO:tasks.rgw_multisite_tests: } 2023-05-31T21:35:38.262 INFO:tasks.rgw_multisite_tests: ] 2023-05-31T21:35:38.263 INFO:tasks.rgw_multisite_tests: }, 2023-05-31T21:35:38.263 INFO:tasks.rgw_multisite_tests: "full_sync": { 2023-05-31T21:35:38.263 INFO:tasks.rgw_multisite_tests: "total": 1, 2023-05-31T21:35:38.263 INFO:tasks.rgw_multisite_tests: "complete": 1 2023-05-31T21:35:38.264 INFO:tasks.rgw_multisite_tests: }, 2023-05-31T21:35:38.264 INFO:tasks.rgw_multisite_tests: "current_time": "2023-05-31T20:10:54Z" 2023-05-31T21:35:38.264 INFO:tasks.rgw_multisite_tests:} 2023-05-31T21:35:38.264 INFO:tasks.rgw_multisite_tests: 2023-05-31T21:35:38.265 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1685562631.516089_59.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-05-31T19:50:31.516089Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1685562970.274572_93.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-05-31T19:56:10.274572Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1685563571.700188_212.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-05-31T20:06:11.700188Z', 'realm_epoch': 2}}] 2023-05-31T21:35:38.265 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1685562631.516089_59.1', 1: '', 2: '1_1685562970.274572_93.1', 3: '1_1685563802.468126_296.1'} 2023-05-31T21:35:38.265 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1685562631.516089_59.1', 1: '', 2: '1_1685562970.274572_93.1', 3: '1_1685563571.700188_212.1'} 2023-05-31T21:35:38.265 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone a2 behind master: shard=3 master=1_1685563802.468126_296.1 target=1_1685563571.700188_212.1 2023-05-31T21:35:38.265 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-31T21:35:38.266 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-31T21:35:38.266 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-05-31T21:35:38.266 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: ignoring --setuser ceph since I am not root 2023-05-31T21:35:38.266 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-05-31T21:35:38.267 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: 2023-05-31T20:10:59.266+0000 7f98ef4a8780 20 RGW-SYNC:meta: read sync status 2023-05-31T21:35:38.267 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: 2023-05-31T20:10:59.267+0000 7f98ef4a8780 20 RGW-SYNC:meta: read sync status 2023-05-31T21:35:38.267 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2023-05-31T21:35:38.267 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "sync_status": { 2023-05-31T21:35:38.268 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "info": { 2023-05-31T21:35:38.268 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "status": "sync", 2023-05-31T21:35:38.268 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "num_shards": 4, 2023-05-31T21:35:38.268 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "period": "2e6123d2-d695-4350-9a01-61a455bc782b", 2023-05-31T21:35:38.269 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "realm_epoch": 2 2023-05-31T21:35:38.269 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2023-05-31T21:35:38.269 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "markers": [ 2023-05-31T21:35:38.269 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2023-05-31T21:35:38.270 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "key": 0, 2023-05-31T21:35:38.270 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "val": { 2023-05-31T21:35:38.270 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "state": 1, 2023-05-31T21:35:38.270 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "marker": "1_1685562631.516089_59.1", 2023-05-31T21:35:38.270 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "next_step_marker": "", 2023-05-31T21:35:38.271 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "total_entries": 1, 2023-05-31T21:35:38.271 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "pos": 0, 2023-05-31T21:35:38.271 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "timestamp": "2023-05-31T19:50:31.516089Z", 2023-05-31T21:35:38.271 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "realm_epoch": 2 2023-05-31T21:35:38.272 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2023-05-31T21:35:38.272 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2023-05-31T21:35:38.272 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2023-05-31T21:35:38.272 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "key": 1, 2023-05-31T21:35:38.273 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "val": { 2023-05-31T21:35:38.273 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "state": 1, 2023-05-31T21:35:38.273 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "marker": "", 2023-05-31T21:35:38.273 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "next_step_marker": "", 2023-05-31T21:35:38.274 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "total_entries": 0, 2023-05-31T21:35:38.274 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "pos": 0, 2023-05-31T21:35:38.274 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "timestamp": "0.000000", 2023-05-31T21:35:38.274 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "realm_epoch": 2 2023-05-31T21:35:38.275 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2023-05-31T21:35:38.275 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2023-05-31T21:35:38.275 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2023-05-31T21:35:38.275 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "key": 2, 2023-05-31T21:35:38.275 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "val": { 2023-05-31T21:35:38.276 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "state": 1, 2023-05-31T21:35:38.276 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "marker": "1_1685562970.274572_93.1", 2023-05-31T21:35:38.276 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "next_step_marker": "", 2023-05-31T21:35:38.276 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "total_entries": 0, 2023-05-31T21:35:38.277 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "pos": 0, 2023-05-31T21:35:38.277 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "timestamp": "2023-05-31T19:56:10.274572Z", 2023-05-31T21:35:38.277 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "realm_epoch": 2 2023-05-31T21:35:38.277 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2023-05-31T21:35:38.278 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2023-05-31T21:35:38.278 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2023-05-31T21:35:38.278 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "key": 3, 2023-05-31T21:35:38.278 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "val": { 2023-05-31T21:35:38.279 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "state": 1, 2023-05-31T21:35:38.279 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "marker": "1_1685563571.700188_212.1", 2023-05-31T21:35:38.279 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "next_step_marker": "", 2023-05-31T21:35:38.279 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "total_entries": 0, 2023-05-31T21:35:38.279 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "pos": 0, 2023-05-31T21:35:38.280 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "timestamp": "2023-05-31T20:06:11.700188Z", 2023-05-31T21:35:38.280 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "realm_epoch": 2 2023-05-31T21:35:38.280 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2023-05-31T21:35:38.280 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2023-05-31T21:35:38.281 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: ] 2023-05-31T21:35:38.281 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2023-05-31T21:35:38.281 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "full_sync": { 2023-05-31T21:35:38.281 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "total": 1, 2023-05-31T21:35:38.282 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "complete": 1 2023-05-31T21:35:38.282 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2023-05-31T21:35:38.282 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "current_time": "2023-05-31T20:10:59Z" 2023-05-31T21:35:38.282 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2023-05-31T21:35:38.282 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2023-05-31T21:35:38.283 INFO:tasks.rgw_multisite_tests: "sync_status": { 2023-05-31T21:35:38.283 INFO:tasks.rgw_multisite_tests: "info": { 2023-05-31T21:35:38.283 INFO:tasks.rgw_multisite_tests: "status": "sync", 2023-05-31T21:35:38.283 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2023-05-31T21:35:38.284 INFO:tasks.rgw_multisite_tests: "period": "2e6123d2-d695-4350-9a01-61a455bc782b", 2023-05-31T21:35:38.284 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-05-31T21:35:38.284 INFO:tasks.rgw_multisite_tests: }, 2023-05-31T21:35:38.284 INFO:tasks.rgw_multisite_tests: "markers": [ 2023-05-31T21:35:38.285 INFO:tasks.rgw_multisite_tests: { 2023-05-31T21:35:38.285 INFO:tasks.rgw_multisite_tests: "key": 0, 2023-05-31T21:35:38.285 INFO:tasks.rgw_multisite_tests: "val": { 2023-05-31T21:35:38.285 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-05-31T21:35:38.285 INFO:tasks.rgw_multisite_tests: "marker": "1_1685562631.516089_59.1", 2023-05-31T21:35:38.286 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-05-31T21:35:38.286 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2023-05-31T21:35:38.286 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-05-31T21:35:38.286 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-05-31T19:50:31.516089Z", 2023-05-31T21:35:38.287 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-05-31T21:35:38.287 INFO:tasks.rgw_multisite_tests: } 2023-05-31T21:35:38.287 INFO:tasks.rgw_multisite_tests: }, 2023-05-31T21:35:38.287 INFO:tasks.rgw_multisite_tests: { 2023-05-31T21:35:38.288 INFO:tasks.rgw_multisite_tests: "key": 1, 2023-05-31T21:35:38.288 INFO:tasks.rgw_multisite_tests: "val": { 2023-05-31T21:35:38.288 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-05-31T21:35:38.288 INFO:tasks.rgw_multisite_tests: "marker": "", 2023-05-31T21:35:38.289 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-05-31T21:35:38.289 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-05-31T21:35:38.289 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-05-31T21:35:38.289 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2023-05-31T21:35:38.290 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-05-31T21:35:38.290 INFO:tasks.rgw_multisite_tests: } 2023-05-31T21:35:38.290 INFO:tasks.rgw_multisite_tests: }, 2023-05-31T21:35:38.290 INFO:tasks.rgw_multisite_tests: { 2023-05-31T21:35:38.290 INFO:tasks.rgw_multisite_tests: "key": 2, 2023-05-31T21:35:38.291 INFO:tasks.rgw_multisite_tests: "val": { 2023-05-31T21:35:38.291 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-05-31T21:35:38.291 INFO:tasks.rgw_multisite_tests: "marker": "1_1685562970.274572_93.1", 2023-05-31T21:35:38.291 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-05-31T21:35:38.292 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-05-31T21:35:38.292 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-05-31T21:35:38.292 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-05-31T19:56:10.274572Z", 2023-05-31T21:35:38.292 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-05-31T21:35:38.293 INFO:tasks.rgw_multisite_tests: } 2023-05-31T21:35:38.293 INFO:tasks.rgw_multisite_tests: }, 2023-05-31T21:35:38.293 INFO:tasks.rgw_multisite_tests: { 2023-05-31T21:35:38.293 INFO:tasks.rgw_multisite_tests: "key": 3, 2023-05-31T21:35:38.293 INFO:tasks.rgw_multisite_tests: "val": { 2023-05-31T21:35:38.294 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-05-31T21:35:38.294 INFO:tasks.rgw_multisite_tests: "marker": "1_1685563571.700188_212.1", 2023-05-31T21:35:38.294 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-05-31T21:35:38.294 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-05-31T21:35:38.295 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-05-31T21:35:38.295 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-05-31T20:06:11.700188Z", 2023-05-31T21:35:38.295 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-05-31T21:35:38.295 INFO:tasks.rgw_multisite_tests: } 2023-05-31T21:35:38.296 INFO:tasks.rgw_multisite_tests: } 2023-05-31T21:35:38.296 INFO:tasks.rgw_multisite_tests: ] 2023-05-31T21:35:38.296 INFO:tasks.rgw_multisite_tests: }, 2023-05-31T21:35:38.296 INFO:tasks.rgw_multisite_tests: "full_sync": { 2023-05-31T21:35:38.296 INFO:tasks.rgw_multisite_tests: "total": 1, 2023-05-31T21:35:38.297 INFO:tasks.rgw_multisite_tests: "complete": 1 2023-05-31T21:35:38.297 INFO:tasks.rgw_multisite_tests: }, 2023-05-31T21:35:38.297 INFO:tasks.rgw_multisite_tests: "current_time": "2023-05-31T20:10:59Z" 2023-05-31T21:35:38.298 INFO:tasks.rgw_multisite_tests:} 2023-05-31T21:35:38.298 INFO:tasks.rgw_multisite_tests: 2023-05-31T21:35:38.298 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1685562631.516089_59.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-05-31T19:50:31.516089Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1685562970.274572_93.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-05-31T19:56:10.274572Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1685563571.700188_212.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-05-31T20:06:11.700188Z', 'realm_epoch': 2}}] 2023-05-31T21:35:38.298 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1685562631.516089_59.1', 1: '', 2: '1_1685562970.274572_93.1', 3: '1_1685563802.468126_296.1'} 2023-05-31T21:35:38.299 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1685562631.516089_59.1', 1: '', 2: '1_1685562970.274572_93.1', 3: '1_1685563571.700188_212.1'} 2023-05-31T21:35:38.299 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone a2 behind master: shard=3 master=1_1685563802.468126_296.1 target=1_1685563571.700188_212.1 2023-05-31T21:35:38.299 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-31T21:35:38.299 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-31T21:35:38.299 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-05-31T21:35:38.300 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: ignoring --setuser ceph since I am not root 2023-05-31T21:35:38.300 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-05-31T21:35:38.300 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: 2023-05-31T20:11:04.389+0000 7f92045d6780 20 RGW-SYNC:meta: read sync status 2023-05-31T21:35:38.300 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: 2023-05-31T20:11:04.390+0000 7f92045d6780 20 RGW-SYNC:meta: read sync status 2023-05-31T21:35:38.301 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2023-05-31T21:35:38.301 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "sync_status": { 2023-05-31T21:35:38.301 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "info": { 2023-05-31T21:35:38.301 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "status": "sync", 2023-05-31T21:35:38.302 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "num_shards": 4, 2023-05-31T21:35:38.302 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "period": "2e6123d2-d695-4350-9a01-61a455bc782b", 2023-05-31T21:35:38.302 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "realm_epoch": 2 2023-05-31T21:35:38.302 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2023-05-31T21:35:38.302 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "markers": [ 2023-05-31T21:35:38.303 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2023-05-31T21:35:38.303 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "key": 0, 2023-05-31T21:35:38.303 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "val": { 2023-05-31T21:35:38.303 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "state": 1, 2023-05-31T21:35:38.304 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "marker": "1_1685562631.516089_59.1", 2023-05-31T21:35:38.304 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "next_step_marker": "", 2023-05-31T21:35:38.304 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "total_entries": 1, 2023-05-31T21:35:38.304 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "pos": 0, 2023-05-31T21:35:38.305 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "timestamp": "2023-05-31T19:50:31.516089Z", 2023-05-31T21:35:38.305 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "realm_epoch": 2 2023-05-31T21:35:38.305 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2023-05-31T21:35:38.305 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2023-05-31T21:35:38.305 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2023-05-31T21:35:38.306 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "key": 1, 2023-05-31T21:35:38.306 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "val": { 2023-05-31T21:35:38.306 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "state": 1, 2023-05-31T21:35:38.306 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "marker": "", 2023-05-31T21:35:38.307 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "next_step_marker": "", 2023-05-31T21:35:38.307 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "total_entries": 0, 2023-05-31T21:35:38.307 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "pos": 0, 2023-05-31T21:35:38.307 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "timestamp": "0.000000", 2023-05-31T21:35:38.308 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "realm_epoch": 2 2023-05-31T21:35:38.308 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2023-05-31T21:35:38.308 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2023-05-31T21:35:38.308 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2023-05-31T21:35:38.309 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "key": 2, 2023-05-31T21:35:38.309 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "val": { 2023-05-31T21:35:38.309 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "state": 1, 2023-05-31T21:35:38.309 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "marker": "1_1685562970.274572_93.1", 2023-05-31T21:35:38.310 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "next_step_marker": "", 2023-05-31T21:35:38.310 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "total_entries": 0, 2023-05-31T21:35:38.310 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "pos": 0, 2023-05-31T21:35:38.310 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "timestamp": "2023-05-31T19:56:10.274572Z", 2023-05-31T21:35:38.311 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "realm_epoch": 2 2023-05-31T21:35:38.311 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2023-05-31T21:35:38.311 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2023-05-31T21:35:38.311 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2023-05-31T21:35:38.311 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "key": 3, 2023-05-31T21:35:38.312 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "val": { 2023-05-31T21:35:38.312 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "state": 1, 2023-05-31T21:35:38.312 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "marker": "1_1685563571.700188_212.1", 2023-05-31T21:35:38.312 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "next_step_marker": "", 2023-05-31T21:35:38.313 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "total_entries": 0, 2023-05-31T21:35:38.313 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "pos": 0, 2023-05-31T21:35:38.313 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "timestamp": "2023-05-31T20:06:11.700188Z", 2023-05-31T21:35:38.313 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "realm_epoch": 2 2023-05-31T21:35:38.314 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2023-05-31T21:35:38.314 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2023-05-31T21:35:38.314 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: ] 2023-05-31T21:35:38.314 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2023-05-31T21:35:38.314 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "full_sync": { 2023-05-31T21:35:38.315 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "total": 1, 2023-05-31T21:35:38.315 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "complete": 1 2023-05-31T21:35:38.315 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2023-05-31T21:35:38.315 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "current_time": "2023-05-31T20:11:04Z" 2023-05-31T21:35:38.316 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2023-05-31T21:35:38.316 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2023-05-31T21:35:38.316 INFO:tasks.rgw_multisite_tests: "sync_status": { 2023-05-31T21:35:38.316 INFO:tasks.rgw_multisite_tests: "info": { 2023-05-31T21:35:38.317 INFO:tasks.rgw_multisite_tests: "status": "sync", 2023-05-31T21:35:38.317 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2023-05-31T21:35:38.317 INFO:tasks.rgw_multisite_tests: "period": "2e6123d2-d695-4350-9a01-61a455bc782b", 2023-05-31T21:35:38.317 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-05-31T21:35:38.317 INFO:tasks.rgw_multisite_tests: }, 2023-05-31T21:35:38.318 INFO:tasks.rgw_multisite_tests: "markers": [ 2023-05-31T21:35:38.318 INFO:tasks.rgw_multisite_tests: { 2023-05-31T21:35:38.318 INFO:tasks.rgw_multisite_tests: "key": 0, 2023-05-31T21:35:38.318 INFO:tasks.rgw_multisite_tests: "val": { 2023-05-31T21:35:38.319 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-05-31T21:35:38.319 INFO:tasks.rgw_multisite_tests: "marker": "1_1685562631.516089_59.1", 2023-05-31T21:35:38.319 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-05-31T21:35:38.319 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2023-05-31T21:35:38.320 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-05-31T21:35:38.320 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-05-31T19:50:31.516089Z", 2023-05-31T21:35:38.320 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-05-31T21:35:38.321 INFO:tasks.rgw_multisite_tests: } 2023-05-31T21:35:38.321 INFO:tasks.rgw_multisite_tests: }, 2023-05-31T21:35:38.322 INFO:tasks.rgw_multisite_tests: { 2023-05-31T21:35:38.322 INFO:tasks.rgw_multisite_tests: "key": 1, 2023-05-31T21:35:38.322 INFO:tasks.rgw_multisite_tests: "val": { 2023-05-31T21:35:38.323 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-05-31T21:35:38.323 INFO:tasks.rgw_multisite_tests: "marker": "", 2023-05-31T21:35:38.323 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-05-31T21:35:38.324 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-05-31T21:35:38.324 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-05-31T21:35:38.324 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2023-05-31T21:35:38.325 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-05-31T21:35:38.325 INFO:tasks.rgw_multisite_tests: } 2023-05-31T21:35:38.325 INFO:tasks.rgw_multisite_tests: }, 2023-05-31T21:35:38.325 INFO:tasks.rgw_multisite_tests: { 2023-05-31T21:35:38.326 INFO:tasks.rgw_multisite_tests: "key": 2, 2023-05-31T21:35:38.326 INFO:tasks.rgw_multisite_tests: "val": { 2023-05-31T21:35:38.326 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-05-31T21:35:38.326 INFO:tasks.rgw_multisite_tests: "marker": "1_1685562970.274572_93.1", 2023-05-31T21:35:38.326 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-05-31T21:35:38.327 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-05-31T21:35:38.327 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-05-31T21:35:38.327 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-05-31T19:56:10.274572Z", 2023-05-31T21:35:38.327 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-05-31T21:35:38.328 INFO:tasks.rgw_multisite_tests: } 2023-05-31T21:35:38.328 INFO:tasks.rgw_multisite_tests: }, 2023-05-31T21:35:38.328 INFO:tasks.rgw_multisite_tests: { 2023-05-31T21:35:38.328 INFO:tasks.rgw_multisite_tests: "key": 3, 2023-05-31T21:35:38.329 INFO:tasks.rgw_multisite_tests: "val": { 2023-05-31T21:35:38.329 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-05-31T21:35:38.329 INFO:tasks.rgw_multisite_tests: "marker": "1_1685563571.700188_212.1", 2023-05-31T21:35:38.329 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-05-31T21:35:38.330 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-05-31T21:35:38.330 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-05-31T21:35:38.330 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-05-31T20:06:11.700188Z", 2023-05-31T21:35:38.330 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-05-31T21:35:38.330 INFO:tasks.rgw_multisite_tests: } 2023-05-31T21:35:38.331 INFO:tasks.rgw_multisite_tests: } 2023-05-31T21:35:38.331 INFO:tasks.rgw_multisite_tests: ] 2023-05-31T21:35:38.331 INFO:tasks.rgw_multisite_tests: }, 2023-05-31T21:35:38.331 INFO:tasks.rgw_multisite_tests: "full_sync": { 2023-05-31T21:35:38.332 INFO:tasks.rgw_multisite_tests: "total": 1, 2023-05-31T21:35:38.332 INFO:tasks.rgw_multisite_tests: "complete": 1 2023-05-31T21:35:38.333 INFO:tasks.rgw_multisite_tests: }, 2023-05-31T21:35:38.333 INFO:tasks.rgw_multisite_tests: "current_time": "2023-05-31T20:11:04Z" 2023-05-31T21:35:38.334 INFO:tasks.rgw_multisite_tests:} 2023-05-31T21:35:38.334 INFO:tasks.rgw_multisite_tests: 2023-05-31T21:35:38.335 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1685562631.516089_59.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-05-31T19:50:31.516089Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1685562970.274572_93.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-05-31T19:56:10.274572Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1685563571.700188_212.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-05-31T20:06:11.700188Z', 'realm_epoch': 2}}] 2023-05-31T21:35:38.335 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1685562631.516089_59.1', 1: '', 2: '1_1685562970.274572_93.1', 3: '1_1685563802.468126_296.1'} 2023-05-31T21:35:38.335 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1685562631.516089_59.1', 1: '', 2: '1_1685562970.274572_93.1', 3: '1_1685563571.700188_212.1'} 2023-05-31T21:35:38.336 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone a2 behind master: shard=3 master=1_1685563802.468126_296.1 target=1_1685563571.700188_212.1 2023-05-31T21:35:38.336 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-31T21:35:38.336 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-31T21:35:38.337 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-05-31T21:35:38.337 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: ignoring --setuser ceph since I am not root 2023-05-31T21:35:38.337 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-05-31T21:35:38.338 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: 2023-05-31T20:11:09.526+0000 7f7e4818f780 20 RGW-SYNC:meta: read sync status 2023-05-31T21:35:38.338 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: 2023-05-31T20:11:09.527+0000 7f7e4818f780 20 RGW-SYNC:meta: read sync status 2023-05-31T21:35:38.338 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2023-05-31T21:35:38.338 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "sync_status": { 2023-05-31T21:35:38.338 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "info": { 2023-05-31T21:35:38.339 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "status": "sync", 2023-05-31T21:35:38.339 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "num_shards": 4, 2023-05-31T21:35:38.339 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "period": "2e6123d2-d695-4350-9a01-61a455bc782b", 2023-05-31T21:35:38.339 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "realm_epoch": 2 2023-05-31T21:35:38.340 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2023-05-31T21:35:38.340 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "markers": [ 2023-05-31T21:35:38.340 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2023-05-31T21:35:38.340 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "key": 0, 2023-05-31T21:35:38.341 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "val": { 2023-05-31T21:35:38.341 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "state": 1, 2023-05-31T21:35:38.341 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "marker": "1_1685562631.516089_59.1", 2023-05-31T21:35:38.341 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "next_step_marker": "", 2023-05-31T21:35:38.341 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "total_entries": 1, 2023-05-31T21:35:38.342 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "pos": 0, 2023-05-31T21:35:38.342 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "timestamp": "2023-05-31T19:50:31.516089Z", 2023-05-31T21:35:38.342 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "realm_epoch": 2 2023-05-31T21:35:38.342 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2023-05-31T21:35:38.343 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2023-05-31T21:35:38.343 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2023-05-31T21:35:38.343 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "key": 1, 2023-05-31T21:35:38.343 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "val": { 2023-05-31T21:35:38.344 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "state": 1, 2023-05-31T21:35:38.344 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "marker": "", 2023-05-31T21:35:38.344 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "next_step_marker": "", 2023-05-31T21:35:38.344 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "total_entries": 0, 2023-05-31T21:35:38.344 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "pos": 0, 2023-05-31T21:35:38.345 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "timestamp": "0.000000", 2023-05-31T21:35:38.345 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "realm_epoch": 2 2023-05-31T21:35:38.345 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2023-05-31T21:35:38.345 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2023-05-31T21:35:38.346 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2023-05-31T21:35:38.346 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "key": 2, 2023-05-31T21:35:38.347 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "val": { 2023-05-31T21:35:38.347 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "state": 1, 2023-05-31T21:35:38.348 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "marker": "1_1685562970.274572_93.1", 2023-05-31T21:35:38.348 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "next_step_marker": "", 2023-05-31T21:35:38.348 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "total_entries": 0, 2023-05-31T21:35:38.349 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "pos": 0, 2023-05-31T21:35:38.349 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "timestamp": "2023-05-31T19:56:10.274572Z", 2023-05-31T21:35:38.349 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "realm_epoch": 2 2023-05-31T21:35:38.349 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2023-05-31T21:35:38.350 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2023-05-31T21:35:38.350 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2023-05-31T21:35:38.350 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "key": 3, 2023-05-31T21:35:38.350 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "val": { 2023-05-31T21:35:38.351 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "state": 1, 2023-05-31T21:35:38.351 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "marker": "1_1685563571.700188_212.1", 2023-05-31T21:35:38.351 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "next_step_marker": "", 2023-05-31T21:35:38.351 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "total_entries": 0, 2023-05-31T21:35:38.352 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "pos": 0, 2023-05-31T21:35:38.352 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "timestamp": "2023-05-31T20:06:11.700188Z", 2023-05-31T21:35:38.352 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "realm_epoch": 2 2023-05-31T21:35:38.352 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2023-05-31T21:35:38.353 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2023-05-31T21:35:38.353 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: ] 2023-05-31T21:35:38.353 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2023-05-31T21:35:38.353 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "full_sync": { 2023-05-31T21:35:38.353 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "total": 1, 2023-05-31T21:35:38.354 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "complete": 1 2023-05-31T21:35:38.354 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2023-05-31T21:35:38.354 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "current_time": "2023-05-31T20:11:09Z" 2023-05-31T21:35:38.354 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2023-05-31T21:35:38.355 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2023-05-31T21:35:38.355 INFO:tasks.rgw_multisite_tests: "sync_status": { 2023-05-31T21:35:38.355 INFO:tasks.rgw_multisite_tests: "info": { 2023-05-31T21:35:38.355 INFO:tasks.rgw_multisite_tests: "status": "sync", 2023-05-31T21:35:38.355 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2023-05-31T21:35:38.356 INFO:tasks.rgw_multisite_tests: "period": "2e6123d2-d695-4350-9a01-61a455bc782b", 2023-05-31T21:35:38.356 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-05-31T21:35:38.356 INFO:tasks.rgw_multisite_tests: }, 2023-05-31T21:35:38.356 INFO:tasks.rgw_multisite_tests: "markers": [ 2023-05-31T21:35:38.357 INFO:tasks.rgw_multisite_tests: { 2023-05-31T21:35:38.357 INFO:tasks.rgw_multisite_tests: "key": 0, 2023-05-31T21:35:38.357 INFO:tasks.rgw_multisite_tests: "val": { 2023-05-31T21:35:38.357 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-05-31T21:35:38.358 INFO:tasks.rgw_multisite_tests: "marker": "1_1685562631.516089_59.1", 2023-05-31T21:35:38.358 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-05-31T21:35:38.358 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2023-05-31T21:35:38.358 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-05-31T21:35:38.359 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-05-31T19:50:31.516089Z", 2023-05-31T21:35:38.359 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-05-31T21:35:38.359 INFO:tasks.rgw_multisite_tests: } 2023-05-31T21:35:38.359 INFO:tasks.rgw_multisite_tests: }, 2023-05-31T21:35:38.360 INFO:tasks.rgw_multisite_tests: { 2023-05-31T21:35:38.360 INFO:tasks.rgw_multisite_tests: "key": 1, 2023-05-31T21:35:38.360 INFO:tasks.rgw_multisite_tests: "val": { 2023-05-31T21:35:38.360 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-05-31T21:35:38.361 INFO:tasks.rgw_multisite_tests: "marker": "", 2023-05-31T21:35:38.361 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-05-31T21:35:38.361 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-05-31T21:35:38.361 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-05-31T21:35:38.362 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2023-05-31T21:35:38.362 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-05-31T21:35:38.362 INFO:tasks.rgw_multisite_tests: } 2023-05-31T21:35:38.362 INFO:tasks.rgw_multisite_tests: }, 2023-05-31T21:35:38.362 INFO:tasks.rgw_multisite_tests: { 2023-05-31T21:35:38.363 INFO:tasks.rgw_multisite_tests: "key": 2, 2023-05-31T21:35:38.363 INFO:tasks.rgw_multisite_tests: "val": { 2023-05-31T21:35:38.363 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-05-31T21:35:38.364 INFO:tasks.rgw_multisite_tests: "marker": "1_1685562970.274572_93.1", 2023-05-31T21:35:38.364 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-05-31T21:35:38.364 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-05-31T21:35:38.364 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-05-31T21:35:38.364 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-05-31T19:56:10.274572Z", 2023-05-31T21:35:38.365 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-05-31T21:35:38.365 INFO:tasks.rgw_multisite_tests: } 2023-05-31T21:35:38.365 INFO:tasks.rgw_multisite_tests: }, 2023-05-31T21:35:38.365 INFO:tasks.rgw_multisite_tests: { 2023-05-31T21:35:38.366 INFO:tasks.rgw_multisite_tests: "key": 3, 2023-05-31T21:35:38.366 INFO:tasks.rgw_multisite_tests: "val": { 2023-05-31T21:35:38.366 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-05-31T21:35:38.366 INFO:tasks.rgw_multisite_tests: "marker": "1_1685563571.700188_212.1", 2023-05-31T21:35:38.367 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-05-31T21:35:38.367 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-05-31T21:35:38.367 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-05-31T21:35:38.367 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-05-31T20:06:11.700188Z", 2023-05-31T21:35:38.367 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-05-31T21:35:38.368 INFO:tasks.rgw_multisite_tests: } 2023-05-31T21:35:38.368 INFO:tasks.rgw_multisite_tests: } 2023-05-31T21:35:38.368 INFO:tasks.rgw_multisite_tests: ] 2023-05-31T21:35:38.368 INFO:tasks.rgw_multisite_tests: }, 2023-05-31T21:35:38.369 INFO:tasks.rgw_multisite_tests: "full_sync": { 2023-05-31T21:35:38.369 INFO:tasks.rgw_multisite_tests: "total": 1, 2023-05-31T21:35:38.369 INFO:tasks.rgw_multisite_tests: "complete": 1 2023-05-31T21:35:38.369 INFO:tasks.rgw_multisite_tests: }, 2023-05-31T21:35:38.370 INFO:tasks.rgw_multisite_tests: "current_time": "2023-05-31T20:11:09Z" 2023-05-31T21:35:38.370 INFO:tasks.rgw_multisite_tests:} 2023-05-31T21:35:38.370 INFO:tasks.rgw_multisite_tests: 2023-05-31T21:35:38.370 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1685562631.516089_59.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-05-31T19:50:31.516089Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1685562970.274572_93.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-05-31T19:56:10.274572Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1685563571.700188_212.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-05-31T20:06:11.700188Z', 'realm_epoch': 2}}] 2023-05-31T21:35:38.371 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1685562631.516089_59.1', 1: '', 2: '1_1685562970.274572_93.1', 3: '1_1685563802.468126_296.1'} 2023-05-31T21:35:38.371 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1685562631.516089_59.1', 1: '', 2: '1_1685562970.274572_93.1', 3: '1_1685563571.700188_212.1'} 2023-05-31T21:35:38.371 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone a2 behind master: shard=3 master=1_1685563802.468126_296.1 target=1_1685563571.700188_212.1 2023-05-31T21:35:38.371 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-31T21:35:38.371 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-31T21:35:38.372 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-05-31T21:35:38.372 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: ignoring --setuser ceph since I am not root 2023-05-31T21:35:38.372 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-05-31T21:35:38.372 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: 2023-05-31T20:11:14.666+0000 7f91656c0780 20 RGW-SYNC:meta: read sync status 2023-05-31T21:35:38.373 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: 2023-05-31T20:11:14.667+0000 7f91656c0780 20 RGW-SYNC:meta: read sync status 2023-05-31T21:35:38.373 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2023-05-31T21:35:38.373 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "sync_status": { 2023-05-31T21:35:38.373 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "info": { 2023-05-31T21:35:38.374 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "status": "sync", 2023-05-31T21:35:38.374 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "num_shards": 4, 2023-05-31T21:35:38.374 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "period": "2e6123d2-d695-4350-9a01-61a455bc782b", 2023-05-31T21:35:38.374 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "realm_epoch": 2 2023-05-31T21:35:38.375 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2023-05-31T21:35:38.375 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "markers": [ 2023-05-31T21:35:38.375 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2023-05-31T21:35:38.375 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "key": 0, 2023-05-31T21:35:38.376 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "val": { 2023-05-31T21:35:38.376 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "state": 1, 2023-05-31T21:35:38.376 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "marker": "1_1685562631.516089_59.1", 2023-05-31T21:35:38.376 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "next_step_marker": "", 2023-05-31T21:35:38.377 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "total_entries": 1, 2023-05-31T21:35:38.377 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "pos": 0, 2023-05-31T21:35:38.377 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "timestamp": "2023-05-31T19:50:31.516089Z", 2023-05-31T21:35:38.377 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "realm_epoch": 2 2023-05-31T21:35:38.378 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2023-05-31T21:35:38.378 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2023-05-31T21:35:38.378 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2023-05-31T21:35:38.378 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "key": 1, 2023-05-31T21:35:38.378 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "val": { 2023-05-31T21:35:38.379 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "state": 1, 2023-05-31T21:35:38.379 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "marker": "", 2023-05-31T21:35:38.379 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "next_step_marker": "", 2023-05-31T21:35:38.379 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "total_entries": 0, 2023-05-31T21:35:38.380 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "pos": 0, 2023-05-31T21:35:38.380 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "timestamp": "0.000000", 2023-05-31T21:35:38.380 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "realm_epoch": 2 2023-05-31T21:35:38.380 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2023-05-31T21:35:38.381 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2023-05-31T21:35:38.381 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2023-05-31T21:35:38.381 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "key": 2, 2023-05-31T21:35:38.381 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "val": { 2023-05-31T21:35:38.381 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "state": 1, 2023-05-31T21:35:38.382 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "marker": "1_1685562970.274572_93.1", 2023-05-31T21:35:38.382 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "next_step_marker": "", 2023-05-31T21:35:38.382 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "total_entries": 0, 2023-05-31T21:35:38.382 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "pos": 0, 2023-05-31T21:35:38.383 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "timestamp": "2023-05-31T19:56:10.274572Z", 2023-05-31T21:35:38.383 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "realm_epoch": 2 2023-05-31T21:35:38.383 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2023-05-31T21:35:38.383 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2023-05-31T21:35:38.384 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2023-05-31T21:35:38.384 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "key": 3, 2023-05-31T21:35:38.384 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "val": { 2023-05-31T21:35:38.384 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "state": 1, 2023-05-31T21:35:38.385 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "marker": "1_1685563571.700188_212.1", 2023-05-31T21:35:38.385 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "next_step_marker": "", 2023-05-31T21:35:38.385 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "total_entries": 0, 2023-05-31T21:35:38.385 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "pos": 0, 2023-05-31T21:35:38.385 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "timestamp": "2023-05-31T20:06:11.700188Z", 2023-05-31T21:35:38.386 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "realm_epoch": 2 2023-05-31T21:35:38.386 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2023-05-31T21:35:38.386 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2023-05-31T21:35:38.386 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: ] 2023-05-31T21:35:38.387 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2023-05-31T21:35:38.387 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "full_sync": { 2023-05-31T21:35:38.387 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "total": 1, 2023-05-31T21:35:38.387 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "complete": 1 2023-05-31T21:35:38.388 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2023-05-31T21:35:38.388 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "current_time": "2023-05-31T20:11:14Z" 2023-05-31T21:35:38.388 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2023-05-31T21:35:38.388 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2023-05-31T21:35:38.389 INFO:tasks.rgw_multisite_tests: "sync_status": { 2023-05-31T21:35:38.389 INFO:tasks.rgw_multisite_tests: "info": { 2023-05-31T21:35:38.389 INFO:tasks.rgw_multisite_tests: "status": "sync", 2023-05-31T21:35:38.390 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2023-05-31T21:35:38.390 INFO:tasks.rgw_multisite_tests: "period": "2e6123d2-d695-4350-9a01-61a455bc782b", 2023-05-31T21:35:38.390 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-05-31T21:35:38.391 INFO:tasks.rgw_multisite_tests: }, 2023-05-31T21:35:38.391 INFO:tasks.rgw_multisite_tests: "markers": [ 2023-05-31T21:35:38.391 INFO:tasks.rgw_multisite_tests: { 2023-05-31T21:35:38.391 INFO:tasks.rgw_multisite_tests: "key": 0, 2023-05-31T21:35:38.392 INFO:tasks.rgw_multisite_tests: "val": { 2023-05-31T21:35:38.392 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-05-31T21:35:38.392 INFO:tasks.rgw_multisite_tests: "marker": "1_1685562631.516089_59.1", 2023-05-31T21:35:38.392 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-05-31T21:35:38.393 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2023-05-31T21:35:38.393 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-05-31T21:35:38.393 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-05-31T19:50:31.516089Z", 2023-05-31T21:35:38.393 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-05-31T21:35:38.393 INFO:tasks.rgw_multisite_tests: } 2023-05-31T21:35:38.394 INFO:tasks.rgw_multisite_tests: }, 2023-05-31T21:35:38.394 INFO:tasks.rgw_multisite_tests: { 2023-05-31T21:35:38.394 INFO:tasks.rgw_multisite_tests: "key": 1, 2023-05-31T21:35:38.394 INFO:tasks.rgw_multisite_tests: "val": { 2023-05-31T21:35:38.395 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-05-31T21:35:38.395 INFO:tasks.rgw_multisite_tests: "marker": "", 2023-05-31T21:35:38.395 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-05-31T21:35:38.395 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-05-31T21:35:38.396 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-05-31T21:35:38.396 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2023-05-31T21:35:38.396 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-05-31T21:35:38.396 INFO:tasks.rgw_multisite_tests: } 2023-05-31T21:35:38.397 INFO:tasks.rgw_multisite_tests: }, 2023-05-31T21:35:38.397 INFO:tasks.rgw_multisite_tests: { 2023-05-31T21:35:38.397 INFO:tasks.rgw_multisite_tests: "key": 2, 2023-05-31T21:35:38.397 INFO:tasks.rgw_multisite_tests: "val": { 2023-05-31T21:35:38.397 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-05-31T21:35:38.398 INFO:tasks.rgw_multisite_tests: "marker": "1_1685562970.274572_93.1", 2023-05-31T21:35:38.398 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-05-31T21:35:38.398 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-05-31T21:35:38.399 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-05-31T21:35:38.399 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-05-31T19:56:10.274572Z", 2023-05-31T21:35:38.399 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-05-31T21:35:38.399 INFO:tasks.rgw_multisite_tests: } 2023-05-31T21:35:38.399 INFO:tasks.rgw_multisite_tests: }, 2023-05-31T21:35:38.400 INFO:tasks.rgw_multisite_tests: { 2023-05-31T21:35:38.400 INFO:tasks.rgw_multisite_tests: "key": 3, 2023-05-31T21:35:38.400 INFO:tasks.rgw_multisite_tests: "val": { 2023-05-31T21:35:38.400 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-05-31T21:35:38.401 INFO:tasks.rgw_multisite_tests: "marker": "1_1685563571.700188_212.1", 2023-05-31T21:35:38.401 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-05-31T21:35:38.401 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-05-31T21:35:38.401 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-05-31T21:35:38.402 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-05-31T20:06:11.700188Z", 2023-05-31T21:35:38.402 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-05-31T21:35:38.402 INFO:tasks.rgw_multisite_tests: } 2023-05-31T21:35:38.402 INFO:tasks.rgw_multisite_tests: } 2023-05-31T21:35:38.403 INFO:tasks.rgw_multisite_tests: ] 2023-05-31T21:35:38.403 INFO:tasks.rgw_multisite_tests: }, 2023-05-31T21:35:38.403 INFO:tasks.rgw_multisite_tests: "full_sync": { 2023-05-31T21:35:38.403 INFO:tasks.rgw_multisite_tests: "total": 1, 2023-05-31T21:35:38.403 INFO:tasks.rgw_multisite_tests: "complete": 1 2023-05-31T21:35:38.404 INFO:tasks.rgw_multisite_tests: }, 2023-05-31T21:35:38.404 INFO:tasks.rgw_multisite_tests: "current_time": "2023-05-31T20:11:14Z" 2023-05-31T21:35:38.404 INFO:tasks.rgw_multisite_tests:} 2023-05-31T21:35:38.404 INFO:tasks.rgw_multisite_tests: 2023-05-31T21:35:38.405 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1685562631.516089_59.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-05-31T19:50:31.516089Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1685562970.274572_93.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-05-31T19:56:10.274572Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1685563571.700188_212.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-05-31T20:06:11.700188Z', 'realm_epoch': 2}}] 2023-05-31T21:35:38.405 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1685562631.516089_59.1', 1: '', 2: '1_1685562970.274572_93.1', 3: '1_1685563802.468126_296.1'} 2023-05-31T21:35:38.405 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1685562631.516089_59.1', 1: '', 2: '1_1685562970.274572_93.1', 3: '1_1685563571.700188_212.1'} 2023-05-31T21:35:38.405 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone a2 behind master: shard=3 master=1_1685563802.468126_296.1 target=1_1685563571.700188_212.1 2023-05-31T21:35:38.406 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-31T21:35:38.406 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-31T21:35:38.406 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-05-31T21:35:38.406 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: ignoring --setuser ceph since I am not root 2023-05-31T21:35:38.406 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-05-31T21:35:38.407 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: 2023-05-31T20:11:19.807+0000 7f078c578780 20 RGW-SYNC:meta: read sync status 2023-05-31T21:35:38.407 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: 2023-05-31T20:11:19.808+0000 7f078c578780 20 RGW-SYNC:meta: read sync status 2023-05-31T21:35:38.407 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2023-05-31T21:35:38.407 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "sync_status": { 2023-05-31T21:35:38.408 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "info": { 2023-05-31T21:35:38.408 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "status": "sync", 2023-05-31T21:35:38.408 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "num_shards": 4, 2023-05-31T21:35:38.408 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "period": "2e6123d2-d695-4350-9a01-61a455bc782b", 2023-05-31T21:35:38.409 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "realm_epoch": 2 2023-05-31T21:35:38.409 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2023-05-31T21:35:38.409 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "markers": [ 2023-05-31T21:35:38.409 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2023-05-31T21:35:38.410 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "key": 0, 2023-05-31T21:35:38.410 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "val": { 2023-05-31T21:35:38.410 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "state": 1, 2023-05-31T21:35:38.410 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "marker": "1_1685562631.516089_59.1", 2023-05-31T21:35:38.411 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "next_step_marker": "", 2023-05-31T21:35:38.411 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "total_entries": 1, 2023-05-31T21:35:38.411 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "pos": 0, 2023-05-31T21:35:38.411 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "timestamp": "2023-05-31T19:50:31.516089Z", 2023-05-31T21:35:38.412 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "realm_epoch": 2 2023-05-31T21:35:38.412 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2023-05-31T21:35:38.412 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2023-05-31T21:35:38.412 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2023-05-31T21:35:38.413 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "key": 1, 2023-05-31T21:35:38.413 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "val": { 2023-05-31T21:35:38.413 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "state": 1, 2023-05-31T21:35:38.413 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "marker": "", 2023-05-31T21:35:38.413 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "next_step_marker": "", 2023-05-31T21:35:38.414 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "total_entries": 0, 2023-05-31T21:35:38.414 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "pos": 0, 2023-05-31T21:35:38.414 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "timestamp": "0.000000", 2023-05-31T21:35:38.414 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "realm_epoch": 2 2023-05-31T21:35:38.415 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2023-05-31T21:35:38.415 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2023-05-31T21:35:38.415 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2023-05-31T21:35:38.415 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "key": 2, 2023-05-31T21:35:38.415 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "val": { 2023-05-31T21:35:38.416 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "state": 1, 2023-05-31T21:35:38.416 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "marker": "1_1685562970.274572_93.1", 2023-05-31T21:35:38.416 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "next_step_marker": "", 2023-05-31T21:35:38.416 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "total_entries": 0, 2023-05-31T21:35:38.417 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "pos": 0, 2023-05-31T21:35:38.417 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "timestamp": "2023-05-31T19:56:10.274572Z", 2023-05-31T21:35:38.417 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "realm_epoch": 2 2023-05-31T21:35:38.417 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2023-05-31T21:35:38.418 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2023-05-31T21:35:38.418 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2023-05-31T21:35:38.418 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "key": 3, 2023-05-31T21:35:38.418 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "val": { 2023-05-31T21:35:38.419 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "state": 1, 2023-05-31T21:35:38.419 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "marker": "1_1685563571.700188_212.1", 2023-05-31T21:35:38.419 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "next_step_marker": "", 2023-05-31T21:35:38.419 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "total_entries": 0, 2023-05-31T21:35:38.420 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "pos": 0, 2023-05-31T21:35:38.420 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "timestamp": "2023-05-31T20:06:11.700188Z", 2023-05-31T21:35:38.420 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "realm_epoch": 2 2023-05-31T21:35:38.420 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2023-05-31T21:35:38.420 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2023-05-31T21:35:38.421 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: ] 2023-05-31T21:35:38.421 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2023-05-31T21:35:38.421 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "full_sync": { 2023-05-31T21:35:38.421 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "total": 1, 2023-05-31T21:35:38.422 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "complete": 1 2023-05-31T21:35:38.422 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2023-05-31T21:35:38.422 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "current_time": "2023-05-31T20:11:19Z" 2023-05-31T21:35:38.422 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2023-05-31T21:35:38.423 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2023-05-31T21:35:38.423 INFO:tasks.rgw_multisite_tests: "sync_status": { 2023-05-31T21:35:38.423 INFO:tasks.rgw_multisite_tests: "info": { 2023-05-31T21:35:38.423 INFO:tasks.rgw_multisite_tests: "status": "sync", 2023-05-31T21:35:38.423 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2023-05-31T21:35:38.424 INFO:tasks.rgw_multisite_tests: "period": "2e6123d2-d695-4350-9a01-61a455bc782b", 2023-05-31T21:35:38.424 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-05-31T21:35:38.424 INFO:tasks.rgw_multisite_tests: }, 2023-05-31T21:35:38.424 INFO:tasks.rgw_multisite_tests: "markers": [ 2023-05-31T21:35:38.425 INFO:tasks.rgw_multisite_tests: { 2023-05-31T21:35:38.425 INFO:tasks.rgw_multisite_tests: "key": 0, 2023-05-31T21:35:38.425 INFO:tasks.rgw_multisite_tests: "val": { 2023-05-31T21:35:38.425 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-05-31T21:35:38.425 INFO:tasks.rgw_multisite_tests: "marker": "1_1685562631.516089_59.1", 2023-05-31T21:35:38.426 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-05-31T21:35:38.426 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2023-05-31T21:35:38.426 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-05-31T21:35:38.426 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-05-31T19:50:31.516089Z", 2023-05-31T21:35:38.427 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-05-31T21:35:38.427 INFO:tasks.rgw_multisite_tests: } 2023-05-31T21:35:38.427 INFO:tasks.rgw_multisite_tests: }, 2023-05-31T21:35:38.427 INFO:tasks.rgw_multisite_tests: { 2023-05-31T21:35:38.428 INFO:tasks.rgw_multisite_tests: "key": 1, 2023-05-31T21:35:38.428 INFO:tasks.rgw_multisite_tests: "val": { 2023-05-31T21:35:38.428 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-05-31T21:35:38.428 INFO:tasks.rgw_multisite_tests: "marker": "", 2023-05-31T21:35:38.429 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-05-31T21:35:38.429 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-05-31T21:35:38.429 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-05-31T21:35:38.429 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2023-05-31T21:35:38.429 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-05-31T21:35:38.430 INFO:tasks.rgw_multisite_tests: } 2023-05-31T21:35:38.430 INFO:tasks.rgw_multisite_tests: }, 2023-05-31T21:35:38.430 INFO:tasks.rgw_multisite_tests: { 2023-05-31T21:35:38.430 INFO:tasks.rgw_multisite_tests: "key": 2, 2023-05-31T21:35:38.431 INFO:tasks.rgw_multisite_tests: "val": { 2023-05-31T21:35:38.431 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-05-31T21:35:38.431 INFO:tasks.rgw_multisite_tests: "marker": "1_1685562970.274572_93.1", 2023-05-31T21:35:38.431 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-05-31T21:35:38.432 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-05-31T21:35:38.432 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-05-31T21:35:38.432 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-05-31T19:56:10.274572Z", 2023-05-31T21:35:38.432 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-05-31T21:35:38.432 INFO:tasks.rgw_multisite_tests: } 2023-05-31T21:35:38.433 INFO:tasks.rgw_multisite_tests: }, 2023-05-31T21:35:38.433 INFO:tasks.rgw_multisite_tests: { 2023-05-31T21:35:38.433 INFO:tasks.rgw_multisite_tests: "key": 3, 2023-05-31T21:35:38.433 INFO:tasks.rgw_multisite_tests: "val": { 2023-05-31T21:35:38.434 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-05-31T21:35:38.434 INFO:tasks.rgw_multisite_tests: "marker": "1_1685563571.700188_212.1", 2023-05-31T21:35:38.434 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-05-31T21:35:38.434 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-05-31T21:35:38.435 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-05-31T21:35:38.435 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-05-31T20:06:11.700188Z", 2023-05-31T21:35:38.435 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-05-31T21:35:38.435 INFO:tasks.rgw_multisite_tests: } 2023-05-31T21:35:38.435 INFO:tasks.rgw_multisite_tests: } 2023-05-31T21:35:38.436 INFO:tasks.rgw_multisite_tests: ] 2023-05-31T21:35:38.436 INFO:tasks.rgw_multisite_tests: }, 2023-05-31T21:35:38.436 INFO:tasks.rgw_multisite_tests: "full_sync": { 2023-05-31T21:35:38.436 INFO:tasks.rgw_multisite_tests: "total": 1, 2023-05-31T21:35:38.437 INFO:tasks.rgw_multisite_tests: "complete": 1 2023-05-31T21:35:38.437 INFO:tasks.rgw_multisite_tests: }, 2023-05-31T21:35:38.437 INFO:tasks.rgw_multisite_tests: "current_time": "2023-05-31T20:11:19Z" 2023-05-31T21:35:38.437 INFO:tasks.rgw_multisite_tests:} 2023-05-31T21:35:38.438 INFO:tasks.rgw_multisite_tests: 2023-05-31T21:35:38.438 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1685562631.516089_59.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-05-31T19:50:31.516089Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1685562970.274572_93.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-05-31T19:56:10.274572Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1685563571.700188_212.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-05-31T20:06:11.700188Z', 'realm_epoch': 2}}] 2023-05-31T21:35:38.438 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1685562631.516089_59.1', 1: '', 2: '1_1685562970.274572_93.1', 3: '1_1685563802.468126_296.1'} 2023-05-31T21:35:38.438 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1685562631.516089_59.1', 1: '', 2: '1_1685562970.274572_93.1', 3: '1_1685563571.700188_212.1'} 2023-05-31T21:35:38.438 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone a2 behind master: shard=3 master=1_1685563802.468126_296.1 target=1_1685563571.700188_212.1 2023-05-31T21:35:38.439 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-31T21:35:38.439 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-31T21:35:38.439 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-05-31T21:35:38.439 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: ignoring --setuser ceph since I am not root 2023-05-31T21:35:38.440 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-05-31T21:35:38.440 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: 2023-05-31T20:11:24.964+0000 7f78077af780 20 RGW-SYNC:meta: read sync status 2023-05-31T21:35:38.440 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: 2023-05-31T20:11:24.965+0000 7f78077af780 20 RGW-SYNC:meta: read sync status 2023-05-31T21:35:38.440 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2023-05-31T21:35:38.441 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "sync_status": { 2023-05-31T21:35:38.441 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "info": { 2023-05-31T21:35:38.441 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "status": "sync", 2023-05-31T21:35:38.441 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "num_shards": 4, 2023-05-31T21:35:38.442 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "period": "2e6123d2-d695-4350-9a01-61a455bc782b", 2023-05-31T21:35:38.442 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "realm_epoch": 2 2023-05-31T21:35:38.442 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2023-05-31T21:35:38.442 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "markers": [ 2023-05-31T21:35:38.442 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2023-05-31T21:35:38.443 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "key": 0, 2023-05-31T21:35:38.443 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "val": { 2023-05-31T21:35:38.443 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "state": 1, 2023-05-31T21:35:38.443 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "marker": "1_1685562631.516089_59.1", 2023-05-31T21:35:38.444 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "next_step_marker": "", 2023-05-31T21:35:38.444 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "total_entries": 1, 2023-05-31T21:35:38.444 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "pos": 0, 2023-05-31T21:35:38.444 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "timestamp": "2023-05-31T19:50:31.516089Z", 2023-05-31T21:35:38.444 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "realm_epoch": 2 2023-05-31T21:35:38.445 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2023-05-31T21:35:38.445 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2023-05-31T21:35:38.445 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2023-05-31T21:35:38.445 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "key": 1, 2023-05-31T21:35:38.446 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "val": { 2023-05-31T21:35:38.446 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "state": 1, 2023-05-31T21:35:38.446 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "marker": "", 2023-05-31T21:35:38.446 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "next_step_marker": "", 2023-05-31T21:35:38.447 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "total_entries": 0, 2023-05-31T21:35:38.447 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "pos": 0, 2023-05-31T21:35:38.447 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "timestamp": "0.000000", 2023-05-31T21:35:38.447 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "realm_epoch": 2 2023-05-31T21:35:38.447 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2023-05-31T21:35:38.448 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2023-05-31T21:35:38.448 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2023-05-31T21:35:38.448 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "key": 2, 2023-05-31T21:35:38.448 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "val": { 2023-05-31T21:35:38.449 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "state": 1, 2023-05-31T21:35:38.449 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "marker": "1_1685562970.274572_93.1", 2023-05-31T21:35:38.449 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "next_step_marker": "", 2023-05-31T21:35:38.449 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "total_entries": 0, 2023-05-31T21:35:38.450 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "pos": 0, 2023-05-31T21:35:38.450 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "timestamp": "2023-05-31T19:56:10.274572Z", 2023-05-31T21:35:38.450 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "realm_epoch": 2 2023-05-31T21:35:38.450 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2023-05-31T21:35:38.450 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2023-05-31T21:35:38.451 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2023-05-31T21:35:38.451 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "key": 3, 2023-05-31T21:35:38.451 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "val": { 2023-05-31T21:35:38.451 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "state": 1, 2023-05-31T21:35:38.452 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "marker": "1_1685563571.700188_212.1", 2023-05-31T21:35:38.452 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "next_step_marker": "", 2023-05-31T21:35:38.452 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "total_entries": 0, 2023-05-31T21:35:38.452 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "pos": 0, 2023-05-31T21:35:38.453 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "timestamp": "2023-05-31T20:06:11.700188Z", 2023-05-31T21:35:38.453 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "realm_epoch": 2 2023-05-31T21:35:38.453 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2023-05-31T21:35:38.453 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2023-05-31T21:35:38.453 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: ] 2023-05-31T21:35:38.454 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2023-05-31T21:35:38.454 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "full_sync": { 2023-05-31T21:35:38.454 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "total": 1, 2023-05-31T21:35:38.454 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "complete": 1 2023-05-31T21:35:38.455 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2023-05-31T21:35:38.455 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "current_time": "2023-05-31T20:11:24Z" 2023-05-31T21:35:38.455 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2023-05-31T21:35:38.455 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2023-05-31T21:35:38.456 INFO:tasks.rgw_multisite_tests: "sync_status": { 2023-05-31T21:35:38.456 INFO:tasks.rgw_multisite_tests: "info": { 2023-05-31T21:35:38.456 INFO:tasks.rgw_multisite_tests: "status": "sync", 2023-05-31T21:35:38.456 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2023-05-31T21:35:38.456 INFO:tasks.rgw_multisite_tests: "period": "2e6123d2-d695-4350-9a01-61a455bc782b", 2023-05-31T21:35:38.457 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-05-31T21:35:38.457 INFO:tasks.rgw_multisite_tests: }, 2023-05-31T21:35:38.457 INFO:tasks.rgw_multisite_tests: "markers": [ 2023-05-31T21:35:38.457 INFO:tasks.rgw_multisite_tests: { 2023-05-31T21:35:38.458 INFO:tasks.rgw_multisite_tests: "key": 0, 2023-05-31T21:35:38.458 INFO:tasks.rgw_multisite_tests: "val": { 2023-05-31T21:35:38.458 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-05-31T21:35:38.458 INFO:tasks.rgw_multisite_tests: "marker": "1_1685562631.516089_59.1", 2023-05-31T21:35:38.459 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-05-31T21:35:38.459 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2023-05-31T21:35:38.459 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-05-31T21:35:38.459 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-05-31T19:50:31.516089Z", 2023-05-31T21:35:38.460 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-05-31T21:35:38.460 INFO:tasks.rgw_multisite_tests: } 2023-05-31T21:35:38.460 INFO:tasks.rgw_multisite_tests: }, 2023-05-31T21:35:38.460 INFO:tasks.rgw_multisite_tests: { 2023-05-31T21:35:38.461 INFO:tasks.rgw_multisite_tests: "key": 1, 2023-05-31T21:35:38.461 INFO:tasks.rgw_multisite_tests: "val": { 2023-05-31T21:35:38.461 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-05-31T21:35:38.461 INFO:tasks.rgw_multisite_tests: "marker": "", 2023-05-31T21:35:38.462 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-05-31T21:35:38.462 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-05-31T21:35:38.462 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-05-31T21:35:38.462 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2023-05-31T21:35:38.462 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-05-31T21:35:38.463 INFO:tasks.rgw_multisite_tests: } 2023-05-31T21:35:38.463 INFO:tasks.rgw_multisite_tests: }, 2023-05-31T21:35:38.463 INFO:tasks.rgw_multisite_tests: { 2023-05-31T21:35:38.463 INFO:tasks.rgw_multisite_tests: "key": 2, 2023-05-31T21:35:38.464 INFO:tasks.rgw_multisite_tests: "val": { 2023-05-31T21:35:38.464 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-05-31T21:35:38.464 INFO:tasks.rgw_multisite_tests: "marker": "1_1685562970.274572_93.1", 2023-05-31T21:35:38.464 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-05-31T21:35:38.465 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-05-31T21:35:38.465 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-05-31T21:35:38.465 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-05-31T19:56:10.274572Z", 2023-05-31T21:35:38.465 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-05-31T21:35:38.466 INFO:tasks.rgw_multisite_tests: } 2023-05-31T21:35:38.466 INFO:tasks.rgw_multisite_tests: }, 2023-05-31T21:35:38.466 INFO:tasks.rgw_multisite_tests: { 2023-05-31T21:35:38.466 INFO:tasks.rgw_multisite_tests: "key": 3, 2023-05-31T21:35:38.467 INFO:tasks.rgw_multisite_tests: "val": { 2023-05-31T21:35:38.467 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-05-31T21:35:38.467 INFO:tasks.rgw_multisite_tests: "marker": "1_1685563571.700188_212.1", 2023-05-31T21:35:38.467 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-05-31T21:35:38.467 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-05-31T21:35:38.468 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-05-31T21:35:38.468 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-05-31T20:06:11.700188Z", 2023-05-31T21:35:38.468 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-05-31T21:35:38.468 INFO:tasks.rgw_multisite_tests: } 2023-05-31T21:35:38.469 INFO:tasks.rgw_multisite_tests: } 2023-05-31T21:35:38.469 INFO:tasks.rgw_multisite_tests: ] 2023-05-31T21:35:38.469 INFO:tasks.rgw_multisite_tests: }, 2023-05-31T21:35:38.469 INFO:tasks.rgw_multisite_tests: "full_sync": { 2023-05-31T21:35:38.470 INFO:tasks.rgw_multisite_tests: "total": 1, 2023-05-31T21:35:38.470 INFO:tasks.rgw_multisite_tests: "complete": 1 2023-05-31T21:35:38.470 INFO:tasks.rgw_multisite_tests: }, 2023-05-31T21:35:38.470 INFO:tasks.rgw_multisite_tests: "current_time": "2023-05-31T20:11:24Z" 2023-05-31T21:35:38.471 INFO:tasks.rgw_multisite_tests:} 2023-05-31T21:35:38.471 INFO:tasks.rgw_multisite_tests: 2023-05-31T21:35:38.471 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1685562631.516089_59.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-05-31T19:50:31.516089Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1685562970.274572_93.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-05-31T19:56:10.274572Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1685563571.700188_212.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-05-31T20:06:11.700188Z', 'realm_epoch': 2}}] 2023-05-31T21:35:38.471 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1685562631.516089_59.1', 1: '', 2: '1_1685562970.274572_93.1', 3: '1_1685563802.468126_296.1'} 2023-05-31T21:35:38.471 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1685562631.516089_59.1', 1: '', 2: '1_1685562970.274572_93.1', 3: '1_1685563571.700188_212.1'} 2023-05-31T21:35:38.472 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone a2 behind master: shard=3 master=1_1685563802.468126_296.1 target=1_1685563571.700188_212.1 2023-05-31T21:35:38.472 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-31T21:35:38.472 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-31T21:35:38.472 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-05-31T21:35:38.473 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: ignoring --setuser ceph since I am not root 2023-05-31T21:35:38.473 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-05-31T21:35:38.473 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: 2023-05-31T20:11:30.091+0000 7f983a1f7780 20 RGW-SYNC:meta: read sync status 2023-05-31T21:35:38.473 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: 2023-05-31T20:11:30.092+0000 7f983a1f7780 20 RGW-SYNC:meta: read sync status 2023-05-31T21:35:38.474 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2023-05-31T21:35:38.474 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "sync_status": { 2023-05-31T21:35:38.474 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "info": { 2023-05-31T21:35:38.474 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "status": "sync", 2023-05-31T21:35:38.474 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "num_shards": 4, 2023-05-31T21:35:38.475 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "period": "2e6123d2-d695-4350-9a01-61a455bc782b", 2023-05-31T21:35:38.475 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "realm_epoch": 2 2023-05-31T21:35:38.475 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2023-05-31T21:35:38.476 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "markers": [ 2023-05-31T21:35:38.476 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2023-05-31T21:35:38.476 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "key": 0, 2023-05-31T21:35:38.476 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "val": { 2023-05-31T21:35:38.477 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "state": 1, 2023-05-31T21:35:38.477 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "marker": "1_1685562631.516089_59.1", 2023-05-31T21:35:38.477 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "next_step_marker": "", 2023-05-31T21:35:38.477 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "total_entries": 1, 2023-05-31T21:35:38.478 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "pos": 0, 2023-05-31T21:35:38.478 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "timestamp": "2023-05-31T19:50:31.516089Z", 2023-05-31T21:35:38.478 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "realm_epoch": 2 2023-05-31T21:35:38.478 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2023-05-31T21:35:38.478 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2023-05-31T21:35:38.479 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2023-05-31T21:35:38.479 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "key": 1, 2023-05-31T21:35:38.479 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "val": { 2023-05-31T21:35:38.479 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "state": 1, 2023-05-31T21:35:38.480 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "marker": "", 2023-05-31T21:35:38.480 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "next_step_marker": "", 2023-05-31T21:35:38.480 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "total_entries": 0, 2023-05-31T21:35:38.480 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "pos": 0, 2023-05-31T21:35:38.481 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "timestamp": "0.000000", 2023-05-31T21:35:38.481 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "realm_epoch": 2 2023-05-31T21:35:38.481 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2023-05-31T21:35:38.481 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2023-05-31T21:35:38.482 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2023-05-31T21:35:38.482 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "key": 2, 2023-05-31T21:35:38.482 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "val": { 2023-05-31T21:35:38.482 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "state": 1, 2023-05-31T21:35:38.482 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "marker": "1_1685562970.274572_93.1", 2023-05-31T21:35:38.483 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "next_step_marker": "", 2023-05-31T21:35:38.483 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "total_entries": 0, 2023-05-31T21:35:38.483 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "pos": 0, 2023-05-31T21:35:38.483 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "timestamp": "2023-05-31T19:56:10.274572Z", 2023-05-31T21:35:38.484 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "realm_epoch": 2 2023-05-31T21:35:38.484 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2023-05-31T21:35:38.484 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2023-05-31T21:35:38.484 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2023-05-31T21:35:38.485 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "key": 3, 2023-05-31T21:35:38.485 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "val": { 2023-05-31T21:35:38.485 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "state": 1, 2023-05-31T21:35:38.485 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "marker": "1_1685563571.700188_212.1", 2023-05-31T21:35:38.485 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "next_step_marker": "", 2023-05-31T21:35:38.486 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "total_entries": 0, 2023-05-31T21:35:38.486 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "pos": 0, 2023-05-31T21:35:38.486 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "timestamp": "2023-05-31T20:06:11.700188Z", 2023-05-31T21:35:38.486 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "realm_epoch": 2 2023-05-31T21:35:38.487 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2023-05-31T21:35:38.487 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2023-05-31T21:35:38.487 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: ] 2023-05-31T21:35:38.487 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2023-05-31T21:35:38.488 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "full_sync": { 2023-05-31T21:35:38.488 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "total": 1, 2023-05-31T21:35:38.488 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "complete": 1 2023-05-31T21:35:38.488 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2023-05-31T21:35:38.488 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "current_time": "2023-05-31T20:11:30Z" 2023-05-31T21:35:38.489 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2023-05-31T21:35:38.489 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2023-05-31T21:35:38.489 INFO:tasks.rgw_multisite_tests: "sync_status": { 2023-05-31T21:35:38.489 INFO:tasks.rgw_multisite_tests: "info": { 2023-05-31T21:35:38.490 INFO:tasks.rgw_multisite_tests: "status": "sync", 2023-05-31T21:35:38.490 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2023-05-31T21:35:38.490 INFO:tasks.rgw_multisite_tests: "period": "2e6123d2-d695-4350-9a01-61a455bc782b", 2023-05-31T21:35:38.490 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-05-31T21:35:38.491 INFO:tasks.rgw_multisite_tests: }, 2023-05-31T21:35:38.491 INFO:tasks.rgw_multisite_tests: "markers": [ 2023-05-31T21:35:38.491 INFO:tasks.rgw_multisite_tests: { 2023-05-31T21:35:38.491 INFO:tasks.rgw_multisite_tests: "key": 0, 2023-05-31T21:35:38.492 INFO:tasks.rgw_multisite_tests: "val": { 2023-05-31T21:35:38.492 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-05-31T21:35:38.492 INFO:tasks.rgw_multisite_tests: "marker": "1_1685562631.516089_59.1", 2023-05-31T21:35:38.492 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-05-31T21:35:38.493 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2023-05-31T21:35:38.493 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-05-31T21:35:38.493 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-05-31T19:50:31.516089Z", 2023-05-31T21:35:38.493 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-05-31T21:35:38.493 INFO:tasks.rgw_multisite_tests: } 2023-05-31T21:35:38.494 INFO:tasks.rgw_multisite_tests: }, 2023-05-31T21:35:38.494 INFO:tasks.rgw_multisite_tests: { 2023-05-31T21:35:38.494 INFO:tasks.rgw_multisite_tests: "key": 1, 2023-05-31T21:35:38.494 INFO:tasks.rgw_multisite_tests: "val": { 2023-05-31T21:35:38.495 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-05-31T21:35:38.495 INFO:tasks.rgw_multisite_tests: "marker": "", 2023-05-31T21:35:38.495 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-05-31T21:35:38.495 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-05-31T21:35:38.496 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-05-31T21:35:38.496 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2023-05-31T21:35:38.496 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-05-31T21:35:38.496 INFO:tasks.rgw_multisite_tests: } 2023-05-31T21:35:38.496 INFO:tasks.rgw_multisite_tests: }, 2023-05-31T21:35:38.497 INFO:tasks.rgw_multisite_tests: { 2023-05-31T21:35:38.497 INFO:tasks.rgw_multisite_tests: "key": 2, 2023-05-31T21:35:38.497 INFO:tasks.rgw_multisite_tests: "val": { 2023-05-31T21:35:38.497 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-05-31T21:35:38.498 INFO:tasks.rgw_multisite_tests: "marker": "1_1685562970.274572_93.1", 2023-05-31T21:35:38.498 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-05-31T21:35:38.498 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-05-31T21:35:38.499 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-05-31T21:35:38.499 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-05-31T19:56:10.274572Z", 2023-05-31T21:35:38.499 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-05-31T21:35:38.499 INFO:tasks.rgw_multisite_tests: } 2023-05-31T21:35:38.499 INFO:tasks.rgw_multisite_tests: }, 2023-05-31T21:35:38.500 INFO:tasks.rgw_multisite_tests: { 2023-05-31T21:35:38.500 INFO:tasks.rgw_multisite_tests: "key": 3, 2023-05-31T21:35:38.500 INFO:tasks.rgw_multisite_tests: "val": { 2023-05-31T21:35:38.500 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-05-31T21:35:38.501 INFO:tasks.rgw_multisite_tests: "marker": "1_1685563571.700188_212.1", 2023-05-31T21:35:38.501 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-05-31T21:35:38.501 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-05-31T21:35:38.501 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-05-31T21:35:38.502 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-05-31T20:06:11.700188Z", 2023-05-31T21:35:38.502 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-05-31T21:35:38.502 INFO:tasks.rgw_multisite_tests: } 2023-05-31T21:35:38.502 INFO:tasks.rgw_multisite_tests: } 2023-05-31T21:35:38.503 INFO:tasks.rgw_multisite_tests: ] 2023-05-31T21:35:38.503 INFO:tasks.rgw_multisite_tests: }, 2023-05-31T21:35:38.503 INFO:tasks.rgw_multisite_tests: "full_sync": { 2023-05-31T21:35:38.503 INFO:tasks.rgw_multisite_tests: "total": 1, 2023-05-31T21:35:38.503 INFO:tasks.rgw_multisite_tests: "complete": 1 2023-05-31T21:35:38.504 INFO:tasks.rgw_multisite_tests: }, 2023-05-31T21:35:38.504 INFO:tasks.rgw_multisite_tests: "current_time": "2023-05-31T20:11:30Z" 2023-05-31T21:35:38.504 INFO:tasks.rgw_multisite_tests:} 2023-05-31T21:35:38.504 INFO:tasks.rgw_multisite_tests: 2023-05-31T21:35:38.505 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1685562631.516089_59.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-05-31T19:50:31.516089Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1685562970.274572_93.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-05-31T19:56:10.274572Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1685563571.700188_212.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-05-31T20:06:11.700188Z', 'realm_epoch': 2}}] 2023-05-31T21:35:38.505 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1685562631.516089_59.1', 1: '', 2: '1_1685562970.274572_93.1', 3: '1_1685563802.468126_296.1'} 2023-05-31T21:35:38.505 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1685562631.516089_59.1', 1: '', 2: '1_1685562970.274572_93.1', 3: '1_1685563571.700188_212.1'} 2023-05-31T21:35:38.505 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone a2 behind master: shard=3 master=1_1685563802.468126_296.1 target=1_1685563571.700188_212.1 2023-05-31T21:35:38.506 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-31T21:35:38.506 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-31T21:35:38.506 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-05-31T21:35:38.506 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: ignoring --setuser ceph since I am not root 2023-05-31T21:35:38.507 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-05-31T21:35:38.507 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: 2023-05-31T20:11:35.240+0000 7f58836f9780 20 RGW-SYNC:meta: read sync status 2023-05-31T21:35:38.507 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: 2023-05-31T20:11:35.241+0000 7f58836f9780 20 RGW-SYNC:meta: read sync status 2023-05-31T21:35:38.507 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2023-05-31T21:35:38.508 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "sync_status": { 2023-05-31T21:35:38.508 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "info": { 2023-05-31T21:35:38.508 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "status": "sync", 2023-05-31T21:35:38.508 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "num_shards": 4, 2023-05-31T21:35:38.509 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "period": "2e6123d2-d695-4350-9a01-61a455bc782b", 2023-05-31T21:35:38.509 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "realm_epoch": 2 2023-05-31T21:35:38.509 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2023-05-31T21:35:38.509 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "markers": [ 2023-05-31T21:35:38.510 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2023-05-31T21:35:38.510 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "key": 0, 2023-05-31T21:35:38.510 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "val": { 2023-05-31T21:35:38.511 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "state": 1, 2023-05-31T21:35:38.511 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "marker": "1_1685562631.516089_59.1", 2023-05-31T21:35:38.511 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "next_step_marker": "", 2023-05-31T21:35:38.511 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "total_entries": 1, 2023-05-31T21:35:38.511 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "pos": 0, 2023-05-31T21:35:38.512 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "timestamp": "2023-05-31T19:50:31.516089Z", 2023-05-31T21:35:38.512 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "realm_epoch": 2 2023-05-31T21:35:38.512 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2023-05-31T21:35:38.512 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2023-05-31T21:35:38.513 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2023-05-31T21:35:38.513 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "key": 1, 2023-05-31T21:35:38.513 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "val": { 2023-05-31T21:35:38.513 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "state": 1, 2023-05-31T21:35:38.514 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "marker": "", 2023-05-31T21:35:38.514 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "next_step_marker": "", 2023-05-31T21:35:38.514 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "total_entries": 0, 2023-05-31T21:35:38.514 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "pos": 0, 2023-05-31T21:35:38.515 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "timestamp": "0.000000", 2023-05-31T21:35:38.515 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "realm_epoch": 2 2023-05-31T21:35:38.515 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2023-05-31T21:35:38.515 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2023-05-31T21:35:38.515 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2023-05-31T21:35:38.516 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "key": 2, 2023-05-31T21:35:38.516 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "val": { 2023-05-31T21:35:38.516 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "state": 1, 2023-05-31T21:35:38.516 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "marker": "1_1685562970.274572_93.1", 2023-05-31T21:35:38.517 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "next_step_marker": "", 2023-05-31T21:35:38.517 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "total_entries": 0, 2023-05-31T21:35:38.517 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "pos": 0, 2023-05-31T21:35:38.517 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "timestamp": "2023-05-31T19:56:10.274572Z", 2023-05-31T21:35:38.518 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "realm_epoch": 2 2023-05-31T21:35:38.518 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2023-05-31T21:35:38.518 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2023-05-31T21:35:38.518 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2023-05-31T21:35:38.519 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "key": 3, 2023-05-31T21:35:38.519 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "val": { 2023-05-31T21:35:38.519 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "state": 1, 2023-05-31T21:35:38.519 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "marker": "1_1685563571.700188_212.1", 2023-05-31T21:35:38.519 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "next_step_marker": "", 2023-05-31T21:35:38.520 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "total_entries": 0, 2023-05-31T21:35:38.520 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "pos": 0, 2023-05-31T21:35:38.520 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "timestamp": "2023-05-31T20:06:11.700188Z", 2023-05-31T21:35:38.520 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "realm_epoch": 2 2023-05-31T21:35:38.521 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2023-05-31T21:35:38.521 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2023-05-31T21:35:38.521 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: ] 2023-05-31T21:35:38.521 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2023-05-31T21:35:38.522 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "full_sync": { 2023-05-31T21:35:38.522 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "total": 1, 2023-05-31T21:35:38.522 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "complete": 1 2023-05-31T21:35:38.522 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2023-05-31T21:35:38.522 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "current_time": "2023-05-31T20:11:35Z" 2023-05-31T21:35:38.523 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2023-05-31T21:35:38.523 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2023-05-31T21:35:38.523 INFO:tasks.rgw_multisite_tests: "sync_status": { 2023-05-31T21:35:38.523 INFO:tasks.rgw_multisite_tests: "info": { 2023-05-31T21:35:38.524 INFO:tasks.rgw_multisite_tests: "status": "sync", 2023-05-31T21:35:38.524 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2023-05-31T21:35:38.524 INFO:tasks.rgw_multisite_tests: "period": "2e6123d2-d695-4350-9a01-61a455bc782b", 2023-05-31T21:35:38.524 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-05-31T21:35:38.525 INFO:tasks.rgw_multisite_tests: }, 2023-05-31T21:35:38.525 INFO:tasks.rgw_multisite_tests: "markers": [ 2023-05-31T21:35:38.525 INFO:tasks.rgw_multisite_tests: { 2023-05-31T21:35:38.525 INFO:tasks.rgw_multisite_tests: "key": 0, 2023-05-31T21:35:38.525 INFO:tasks.rgw_multisite_tests: "val": { 2023-05-31T21:35:38.526 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-05-31T21:35:38.526 INFO:tasks.rgw_multisite_tests: "marker": "1_1685562631.516089_59.1", 2023-05-31T21:35:38.526 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-05-31T21:35:38.526 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2023-05-31T21:35:38.527 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-05-31T21:35:38.527 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-05-31T19:50:31.516089Z", 2023-05-31T21:35:38.527 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-05-31T21:35:38.527 INFO:tasks.rgw_multisite_tests: } 2023-05-31T21:35:38.528 INFO:tasks.rgw_multisite_tests: }, 2023-05-31T21:35:38.528 INFO:tasks.rgw_multisite_tests: { 2023-05-31T21:35:38.528 INFO:tasks.rgw_multisite_tests: "key": 1, 2023-05-31T21:35:38.528 INFO:tasks.rgw_multisite_tests: "val": { 2023-05-31T21:35:38.529 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-05-31T21:35:38.529 INFO:tasks.rgw_multisite_tests: "marker": "", 2023-05-31T21:35:38.529 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-05-31T21:35:38.529 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-05-31T21:35:38.529 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-05-31T21:35:38.530 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2023-05-31T21:35:38.530 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-05-31T21:35:38.530 INFO:tasks.rgw_multisite_tests: } 2023-05-31T21:35:38.530 INFO:tasks.rgw_multisite_tests: }, 2023-05-31T21:35:38.531 INFO:tasks.rgw_multisite_tests: { 2023-05-31T21:35:38.531 INFO:tasks.rgw_multisite_tests: "key": 2, 2023-05-31T21:35:38.531 INFO:tasks.rgw_multisite_tests: "val": { 2023-05-31T21:35:38.531 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-05-31T21:35:38.532 INFO:tasks.rgw_multisite_tests: "marker": "1_1685562970.274572_93.1", 2023-05-31T21:35:38.532 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-05-31T21:35:38.532 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-05-31T21:35:38.532 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-05-31T21:35:38.532 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-05-31T19:56:10.274572Z", 2023-05-31T21:35:38.533 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-05-31T21:35:38.533 INFO:tasks.rgw_multisite_tests: } 2023-05-31T21:35:38.533 INFO:tasks.rgw_multisite_tests: }, 2023-05-31T21:35:38.533 INFO:tasks.rgw_multisite_tests: { 2023-05-31T21:35:38.534 INFO:tasks.rgw_multisite_tests: "key": 3, 2023-05-31T21:35:38.534 INFO:tasks.rgw_multisite_tests: "val": { 2023-05-31T21:35:38.534 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-05-31T21:35:38.534 INFO:tasks.rgw_multisite_tests: "marker": "1_1685563571.700188_212.1", 2023-05-31T21:35:38.535 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-05-31T21:35:38.535 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-05-31T21:35:38.535 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-05-31T21:35:38.535 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-05-31T20:06:11.700188Z", 2023-05-31T21:35:38.535 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-05-31T21:35:38.536 INFO:tasks.rgw_multisite_tests: } 2023-05-31T21:35:38.536 INFO:tasks.rgw_multisite_tests: } 2023-05-31T21:35:38.536 INFO:tasks.rgw_multisite_tests: ] 2023-05-31T21:35:38.536 INFO:tasks.rgw_multisite_tests: }, 2023-05-31T21:35:38.537 INFO:tasks.rgw_multisite_tests: "full_sync": { 2023-05-31T21:35:38.537 INFO:tasks.rgw_multisite_tests: "total": 1, 2023-05-31T21:35:38.537 INFO:tasks.rgw_multisite_tests: "complete": 1 2023-05-31T21:35:38.537 INFO:tasks.rgw_multisite_tests: }, 2023-05-31T21:35:38.538 INFO:tasks.rgw_multisite_tests: "current_time": "2023-05-31T20:11:35Z" 2023-05-31T21:35:38.538 INFO:tasks.rgw_multisite_tests:} 2023-05-31T21:35:38.538 INFO:tasks.rgw_multisite_tests: 2023-05-31T21:35:38.538 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1685562631.516089_59.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-05-31T19:50:31.516089Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1685562970.274572_93.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-05-31T19:56:10.274572Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1685563571.700188_212.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-05-31T20:06:11.700188Z', 'realm_epoch': 2}}] 2023-05-31T21:35:38.539 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1685562631.516089_59.1', 1: '', 2: '1_1685562970.274572_93.1', 3: '1_1685563802.468126_296.1'} 2023-05-31T21:35:38.539 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1685562631.516089_59.1', 1: '', 2: '1_1685562970.274572_93.1', 3: '1_1685563571.700188_212.1'} 2023-05-31T21:35:38.539 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone a2 behind master: shard=3 master=1_1685563802.468126_296.1 target=1_1685563571.700188_212.1 2023-05-31T21:35:38.539 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-31T21:35:38.539 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-31T21:35:38.540 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-05-31T21:35:38.540 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: ignoring --setuser ceph since I am not root 2023-05-31T21:35:38.540 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-05-31T21:35:38.540 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: 2023-05-31T20:11:40.366+0000 7f1cd83d2780 20 RGW-SYNC:meta: read sync status 2023-05-31T21:35:38.541 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: 2023-05-31T20:11:40.367+0000 7f1cd83d2780 20 RGW-SYNC:meta: read sync status 2023-05-31T21:35:38.541 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2023-05-31T21:35:38.541 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "sync_status": { 2023-05-31T21:35:38.541 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "info": { 2023-05-31T21:35:38.542 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "status": "sync", 2023-05-31T21:35:38.542 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "num_shards": 4, 2023-05-31T21:35:38.542 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "period": "2e6123d2-d695-4350-9a01-61a455bc782b", 2023-05-31T21:35:38.542 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "realm_epoch": 2 2023-05-31T21:35:38.542 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2023-05-31T21:35:38.543 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "markers": [ 2023-05-31T21:35:38.543 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2023-05-31T21:35:38.543 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "key": 0, 2023-05-31T21:35:38.543 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "val": { 2023-05-31T21:35:38.544 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "state": 1, 2023-05-31T21:35:38.544 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "marker": "1_1685562631.516089_59.1", 2023-05-31T21:35:38.544 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "next_step_marker": "", 2023-05-31T21:35:38.544 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "total_entries": 1, 2023-05-31T21:35:38.545 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "pos": 0, 2023-05-31T21:35:38.545 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "timestamp": "2023-05-31T19:50:31.516089Z", 2023-05-31T21:35:38.545 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "realm_epoch": 2 2023-05-31T21:35:38.545 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2023-05-31T21:35:38.545 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2023-05-31T21:35:38.546 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2023-05-31T21:35:38.546 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "key": 1, 2023-05-31T21:35:38.546 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "val": { 2023-05-31T21:35:38.546 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "state": 1, 2023-05-31T21:35:38.547 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "marker": "", 2023-05-31T21:35:38.547 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "next_step_marker": "", 2023-05-31T21:35:38.547 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "total_entries": 0, 2023-05-31T21:35:38.547 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "pos": 0, 2023-05-31T21:35:38.548 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "timestamp": "0.000000", 2023-05-31T21:35:38.548 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "realm_epoch": 2 2023-05-31T21:35:38.548 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2023-05-31T21:35:38.548 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2023-05-31T21:35:38.549 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2023-05-31T21:35:38.549 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "key": 2, 2023-05-31T21:35:38.549 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "val": { 2023-05-31T21:35:38.549 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "state": 1, 2023-05-31T21:35:38.550 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "marker": "1_1685562970.274572_93.1", 2023-05-31T21:35:38.550 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "next_step_marker": "", 2023-05-31T21:35:38.550 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "total_entries": 0, 2023-05-31T21:35:38.550 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "pos": 0, 2023-05-31T21:35:38.550 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "timestamp": "2023-05-31T19:56:10.274572Z", 2023-05-31T21:35:38.551 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "realm_epoch": 2 2023-05-31T21:35:38.551 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2023-05-31T21:35:38.551 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2023-05-31T21:35:38.551 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2023-05-31T21:35:38.552 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "key": 3, 2023-05-31T21:35:38.552 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "val": { 2023-05-31T21:35:38.552 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "state": 1, 2023-05-31T21:35:38.552 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "marker": "1_1685563571.700188_212.1", 2023-05-31T21:35:38.553 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "next_step_marker": "", 2023-05-31T21:35:38.553 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "total_entries": 0, 2023-05-31T21:35:38.553 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "pos": 0, 2023-05-31T21:35:38.553 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "timestamp": "2023-05-31T20:06:11.700188Z", 2023-05-31T21:35:38.553 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "realm_epoch": 2 2023-05-31T21:35:38.554 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2023-05-31T21:35:38.554 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2023-05-31T21:35:38.554 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: ] 2023-05-31T21:35:38.555 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2023-05-31T21:35:38.555 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "full_sync": { 2023-05-31T21:35:38.555 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "total": 1, 2023-05-31T21:35:38.555 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "complete": 1 2023-05-31T21:35:38.555 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2023-05-31T21:35:38.556 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "current_time": "2023-05-31T20:11:40Z" 2023-05-31T21:35:38.556 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2023-05-31T21:35:38.556 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2023-05-31T21:35:38.556 INFO:tasks.rgw_multisite_tests: "sync_status": { 2023-05-31T21:35:38.557 INFO:tasks.rgw_multisite_tests: "info": { 2023-05-31T21:35:38.557 INFO:tasks.rgw_multisite_tests: "status": "sync", 2023-05-31T21:35:38.557 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2023-05-31T21:35:38.557 INFO:tasks.rgw_multisite_tests: "period": "2e6123d2-d695-4350-9a01-61a455bc782b", 2023-05-31T21:35:38.558 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-05-31T21:35:38.558 INFO:tasks.rgw_multisite_tests: }, 2023-05-31T21:35:38.558 INFO:tasks.rgw_multisite_tests: "markers": [ 2023-05-31T21:35:38.558 INFO:tasks.rgw_multisite_tests: { 2023-05-31T21:35:38.559 INFO:tasks.rgw_multisite_tests: "key": 0, 2023-05-31T21:35:38.559 INFO:tasks.rgw_multisite_tests: "val": { 2023-05-31T21:35:38.559 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-05-31T21:35:38.559 INFO:tasks.rgw_multisite_tests: "marker": "1_1685562631.516089_59.1", 2023-05-31T21:35:38.560 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-05-31T21:35:38.560 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2023-05-31T21:35:38.560 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-05-31T21:35:38.560 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-05-31T19:50:31.516089Z", 2023-05-31T21:35:38.561 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-05-31T21:35:38.561 INFO:tasks.rgw_multisite_tests: } 2023-05-31T21:35:38.561 INFO:tasks.rgw_multisite_tests: }, 2023-05-31T21:35:38.561 INFO:tasks.rgw_multisite_tests: { 2023-05-31T21:35:38.561 INFO:tasks.rgw_multisite_tests: "key": 1, 2023-05-31T21:35:38.562 INFO:tasks.rgw_multisite_tests: "val": { 2023-05-31T21:35:38.562 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-05-31T21:35:38.562 INFO:tasks.rgw_multisite_tests: "marker": "", 2023-05-31T21:35:38.562 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-05-31T21:35:38.563 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-05-31T21:35:38.563 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-05-31T21:35:38.563 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2023-05-31T21:35:38.563 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-05-31T21:35:38.564 INFO:tasks.rgw_multisite_tests: } 2023-05-31T21:35:38.564 INFO:tasks.rgw_multisite_tests: }, 2023-05-31T21:35:38.564 INFO:tasks.rgw_multisite_tests: { 2023-05-31T21:35:38.564 INFO:tasks.rgw_multisite_tests: "key": 2, 2023-05-31T21:35:38.565 INFO:tasks.rgw_multisite_tests: "val": { 2023-05-31T21:35:38.565 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-05-31T21:35:38.565 INFO:tasks.rgw_multisite_tests: "marker": "1_1685562970.274572_93.1", 2023-05-31T21:35:38.565 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-05-31T21:35:38.566 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-05-31T21:35:38.566 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-05-31T21:35:38.566 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-05-31T19:56:10.274572Z", 2023-05-31T21:35:38.566 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-05-31T21:35:38.566 INFO:tasks.rgw_multisite_tests: } 2023-05-31T21:35:38.567 INFO:tasks.rgw_multisite_tests: }, 2023-05-31T21:35:38.567 INFO:tasks.rgw_multisite_tests: { 2023-05-31T21:35:38.567 INFO:tasks.rgw_multisite_tests: "key": 3, 2023-05-31T21:35:38.567 INFO:tasks.rgw_multisite_tests: "val": { 2023-05-31T21:35:38.568 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-05-31T21:35:38.568 INFO:tasks.rgw_multisite_tests: "marker": "1_1685563571.700188_212.1", 2023-05-31T21:35:38.568 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-05-31T21:35:38.568 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-05-31T21:35:38.569 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-05-31T21:35:38.569 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-05-31T20:06:11.700188Z", 2023-05-31T21:35:38.569 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-05-31T21:35:38.569 INFO:tasks.rgw_multisite_tests: } 2023-05-31T21:35:38.570 INFO:tasks.rgw_multisite_tests: } 2023-05-31T21:35:38.570 INFO:tasks.rgw_multisite_tests: ] 2023-05-31T21:35:38.570 INFO:tasks.rgw_multisite_tests: }, 2023-05-31T21:35:38.570 INFO:tasks.rgw_multisite_tests: "full_sync": { 2023-05-31T21:35:38.570 INFO:tasks.rgw_multisite_tests: "total": 1, 2023-05-31T21:35:38.571 INFO:tasks.rgw_multisite_tests: "complete": 1 2023-05-31T21:35:38.571 INFO:tasks.rgw_multisite_tests: }, 2023-05-31T21:35:38.571 INFO:tasks.rgw_multisite_tests: "current_time": "2023-05-31T20:11:40Z" 2023-05-31T21:35:38.571 INFO:tasks.rgw_multisite_tests:} 2023-05-31T21:35:38.572 INFO:tasks.rgw_multisite_tests: 2023-05-31T21:35:38.572 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1685562631.516089_59.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-05-31T19:50:31.516089Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1685562970.274572_93.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-05-31T19:56:10.274572Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1685563571.700188_212.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-05-31T20:06:11.700188Z', 'realm_epoch': 2}}] 2023-05-31T21:35:38.572 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1685562631.516089_59.1', 1: '', 2: '1_1685562970.274572_93.1', 3: '1_1685563802.468126_296.1'} 2023-05-31T21:35:38.572 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1685562631.516089_59.1', 1: '', 2: '1_1685562970.274572_93.1', 3: '1_1685563571.700188_212.1'} 2023-05-31T21:35:38.573 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone a2 behind master: shard=3 master=1_1685563802.468126_296.1 target=1_1685563571.700188_212.1 2023-05-31T21:35:38.573 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-31T21:35:38.573 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-31T21:35:38.573 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-05-31T21:35:38.574 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: ignoring --setuser ceph since I am not root 2023-05-31T21:35:38.574 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-05-31T21:35:38.574 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: 2023-05-31T20:11:45.502+0000 7f923f3a9780 20 RGW-SYNC:meta: read sync status 2023-05-31T21:35:38.574 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: 2023-05-31T20:11:45.503+0000 7f923f3a9780 20 RGW-SYNC:meta: read sync status 2023-05-31T21:35:38.574 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2023-05-31T21:35:38.575 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "sync_status": { 2023-05-31T21:35:38.575 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "info": { 2023-05-31T21:35:38.575 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "status": "sync", 2023-05-31T21:35:38.575 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "num_shards": 4, 2023-05-31T21:35:38.576 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "period": "2e6123d2-d695-4350-9a01-61a455bc782b", 2023-05-31T21:35:38.576 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "realm_epoch": 2 2023-05-31T21:35:38.576 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2023-05-31T21:35:38.576 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "markers": [ 2023-05-31T21:35:38.577 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2023-05-31T21:35:38.577 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "key": 0, 2023-05-31T21:35:38.577 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "val": { 2023-05-31T21:35:38.577 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "state": 1, 2023-05-31T21:35:38.578 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "marker": "1_1685562631.516089_59.1", 2023-05-31T21:35:38.578 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "next_step_marker": "", 2023-05-31T21:35:38.578 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "total_entries": 1, 2023-05-31T21:35:38.578 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "pos": 0, 2023-05-31T21:35:38.579 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "timestamp": "2023-05-31T19:50:31.516089Z", 2023-05-31T21:35:38.579 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "realm_epoch": 2 2023-05-31T21:35:38.579 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2023-05-31T21:35:38.579 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2023-05-31T21:35:38.580 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2023-05-31T21:35:38.580 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "key": 1, 2023-05-31T21:35:38.580 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "val": { 2023-05-31T21:35:38.580 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "state": 1, 2023-05-31T21:35:38.581 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "marker": "", 2023-05-31T21:35:38.581 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "next_step_marker": "", 2023-05-31T21:35:38.581 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "total_entries": 0, 2023-05-31T21:35:38.581 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "pos": 0, 2023-05-31T21:35:38.581 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "timestamp": "0.000000", 2023-05-31T21:35:38.582 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "realm_epoch": 2 2023-05-31T21:35:38.582 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2023-05-31T21:35:38.582 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2023-05-31T21:35:38.582 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2023-05-31T21:35:38.583 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "key": 2, 2023-05-31T21:35:38.583 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "val": { 2023-05-31T21:35:38.583 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "state": 1, 2023-05-31T21:35:38.583 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "marker": "1_1685562970.274572_93.1", 2023-05-31T21:35:38.584 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "next_step_marker": "", 2023-05-31T21:35:38.584 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "total_entries": 0, 2023-05-31T21:35:38.584 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "pos": 0, 2023-05-31T21:35:38.584 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "timestamp": "2023-05-31T19:56:10.274572Z", 2023-05-31T21:35:38.584 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "realm_epoch": 2 2023-05-31T21:35:38.585 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2023-05-31T21:35:38.585 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2023-05-31T21:35:38.585 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2023-05-31T21:35:38.585 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "key": 3, 2023-05-31T21:35:38.586 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "val": { 2023-05-31T21:35:38.586 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "state": 1, 2023-05-31T21:35:38.586 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "marker": "1_1685563571.700188_212.1", 2023-05-31T21:35:38.586 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "next_step_marker": "", 2023-05-31T21:35:38.587 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "total_entries": 0, 2023-05-31T21:35:38.587 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "pos": 0, 2023-05-31T21:35:38.587 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "timestamp": "2023-05-31T20:06:11.700188Z", 2023-05-31T21:35:38.587 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "realm_epoch": 2 2023-05-31T21:35:38.587 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2023-05-31T21:35:38.588 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2023-05-31T21:35:38.588 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: ] 2023-05-31T21:35:38.588 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2023-05-31T21:35:38.588 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "full_sync": { 2023-05-31T21:35:38.589 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "total": 1, 2023-05-31T21:35:38.589 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "complete": 1 2023-05-31T21:35:38.589 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2023-05-31T21:35:38.589 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "current_time": "2023-05-31T20:11:45Z" 2023-05-31T21:35:38.590 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2023-05-31T21:35:38.590 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2023-05-31T21:35:38.590 INFO:tasks.rgw_multisite_tests: "sync_status": { 2023-05-31T21:35:38.590 INFO:tasks.rgw_multisite_tests: "info": { 2023-05-31T21:35:38.591 INFO:tasks.rgw_multisite_tests: "status": "sync", 2023-05-31T21:35:38.591 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2023-05-31T21:35:38.591 INFO:tasks.rgw_multisite_tests: "period": "2e6123d2-d695-4350-9a01-61a455bc782b", 2023-05-31T21:35:38.591 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-05-31T21:35:38.592 INFO:tasks.rgw_multisite_tests: }, 2023-05-31T21:35:38.592 INFO:tasks.rgw_multisite_tests: "markers": [ 2023-05-31T21:35:38.592 INFO:tasks.rgw_multisite_tests: { 2023-05-31T21:35:38.592 INFO:tasks.rgw_multisite_tests: "key": 0, 2023-05-31T21:35:38.593 INFO:tasks.rgw_multisite_tests: "val": { 2023-05-31T21:35:38.593 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-05-31T21:35:38.593 INFO:tasks.rgw_multisite_tests: "marker": "1_1685562631.516089_59.1", 2023-05-31T21:35:38.593 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-05-31T21:35:38.593 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2023-05-31T21:35:38.594 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-05-31T21:35:38.594 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-05-31T19:50:31.516089Z", 2023-05-31T21:35:38.594 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-05-31T21:35:38.594 INFO:tasks.rgw_multisite_tests: } 2023-05-31T21:35:38.595 INFO:tasks.rgw_multisite_tests: }, 2023-05-31T21:35:38.595 INFO:tasks.rgw_multisite_tests: { 2023-05-31T21:35:38.595 INFO:tasks.rgw_multisite_tests: "key": 1, 2023-05-31T21:35:38.595 INFO:tasks.rgw_multisite_tests: "val": { 2023-05-31T21:35:38.596 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-05-31T21:35:38.596 INFO:tasks.rgw_multisite_tests: "marker": "", 2023-05-31T21:35:38.596 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-05-31T21:35:38.596 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-05-31T21:35:38.596 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-05-31T21:35:38.597 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2023-05-31T21:35:38.597 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-05-31T21:35:38.597 INFO:tasks.rgw_multisite_tests: } 2023-05-31T21:35:38.597 INFO:tasks.rgw_multisite_tests: }, 2023-05-31T21:35:38.598 INFO:tasks.rgw_multisite_tests: { 2023-05-31T21:35:38.598 INFO:tasks.rgw_multisite_tests: "key": 2, 2023-05-31T21:35:38.598 INFO:tasks.rgw_multisite_tests: "val": { 2023-05-31T21:35:38.598 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-05-31T21:35:38.599 INFO:tasks.rgw_multisite_tests: "marker": "1_1685562970.274572_93.1", 2023-05-31T21:35:38.599 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-05-31T21:35:38.599 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-05-31T21:35:38.599 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-05-31T21:35:38.600 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-05-31T19:56:10.274572Z", 2023-05-31T21:35:38.600 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-05-31T21:35:38.600 INFO:tasks.rgw_multisite_tests: } 2023-05-31T21:35:38.600 INFO:tasks.rgw_multisite_tests: }, 2023-05-31T21:35:38.601 INFO:tasks.rgw_multisite_tests: { 2023-05-31T21:35:38.601 INFO:tasks.rgw_multisite_tests: "key": 3, 2023-05-31T21:35:38.601 INFO:tasks.rgw_multisite_tests: "val": { 2023-05-31T21:35:38.601 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-05-31T21:35:38.602 INFO:tasks.rgw_multisite_tests: "marker": "1_1685563571.700188_212.1", 2023-05-31T21:35:38.602 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-05-31T21:35:38.602 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-05-31T21:35:38.602 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-05-31T21:35:38.602 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-05-31T20:06:11.700188Z", 2023-05-31T21:35:38.603 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-05-31T21:35:38.603 INFO:tasks.rgw_multisite_tests: } 2023-05-31T21:35:38.603 INFO:tasks.rgw_multisite_tests: } 2023-05-31T21:35:38.603 INFO:tasks.rgw_multisite_tests: ] 2023-05-31T21:35:38.604 INFO:tasks.rgw_multisite_tests: }, 2023-05-31T21:35:38.604 INFO:tasks.rgw_multisite_tests: "full_sync": { 2023-05-31T21:35:38.604 INFO:tasks.rgw_multisite_tests: "total": 1, 2023-05-31T21:35:38.604 INFO:tasks.rgw_multisite_tests: "complete": 1 2023-05-31T21:35:38.605 INFO:tasks.rgw_multisite_tests: }, 2023-05-31T21:35:38.605 INFO:tasks.rgw_multisite_tests: "current_time": "2023-05-31T20:11:45Z" 2023-05-31T21:35:38.605 INFO:tasks.rgw_multisite_tests:} 2023-05-31T21:35:38.605 INFO:tasks.rgw_multisite_tests: 2023-05-31T21:35:38.605 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1685562631.516089_59.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-05-31T19:50:31.516089Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1685562970.274572_93.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-05-31T19:56:10.274572Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1685563571.700188_212.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-05-31T20:06:11.700188Z', 'realm_epoch': 2}}] 2023-05-31T21:35:38.606 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1685562631.516089_59.1', 1: '', 2: '1_1685562970.274572_93.1', 3: '1_1685563802.468126_296.1'} 2023-05-31T21:35:38.606 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1685562631.516089_59.1', 1: '', 2: '1_1685562970.274572_93.1', 3: '1_1685563571.700188_212.1'} 2023-05-31T21:35:38.606 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone a2 behind master: shard=3 master=1_1685563802.468126_296.1 target=1_1685563571.700188_212.1 2023-05-31T21:35:38.606 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-31T21:35:38.607 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-31T21:35:38.607 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-05-31T21:35:38.607 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: ignoring --setuser ceph since I am not root 2023-05-31T21:35:38.607 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-05-31T21:35:38.608 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: 2023-05-31T20:11:50.632+0000 7fdc7493c780 20 RGW-SYNC:meta: read sync status 2023-05-31T21:35:38.608 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: 2023-05-31T20:11:50.633+0000 7fdc7493c780 20 RGW-SYNC:meta: read sync status 2023-05-31T21:35:38.608 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2023-05-31T21:35:38.608 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "sync_status": { 2023-05-31T21:35:38.609 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "info": { 2023-05-31T21:35:38.609 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "status": "sync", 2023-05-31T21:35:38.609 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "num_shards": 4, 2023-05-31T21:35:38.609 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "period": "2e6123d2-d695-4350-9a01-61a455bc782b", 2023-05-31T21:35:38.610 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "realm_epoch": 2 2023-05-31T21:35:38.610 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2023-05-31T21:35:38.610 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "markers": [ 2023-05-31T21:35:38.610 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2023-05-31T21:35:38.611 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "key": 0, 2023-05-31T21:35:38.611 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "val": { 2023-05-31T21:35:38.611 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "state": 1, 2023-05-31T21:35:38.611 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "marker": "1_1685562631.516089_59.1", 2023-05-31T21:35:38.612 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "next_step_marker": "", 2023-05-31T21:35:38.612 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "total_entries": 1, 2023-05-31T21:35:38.612 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "pos": 0, 2023-05-31T21:35:38.612 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "timestamp": "2023-05-31T19:50:31.516089Z", 2023-05-31T21:35:38.612 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "realm_epoch": 2 2023-05-31T21:35:38.613 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2023-05-31T21:35:38.613 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2023-05-31T21:35:38.613 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2023-05-31T21:35:38.613 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "key": 1, 2023-05-31T21:35:38.614 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "val": { 2023-05-31T21:35:38.614 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "state": 1, 2023-05-31T21:35:38.614 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "marker": "", 2023-05-31T21:35:38.614 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "next_step_marker": "", 2023-05-31T21:35:38.615 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "total_entries": 0, 2023-05-31T21:35:38.615 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "pos": 0, 2023-05-31T21:35:38.615 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "timestamp": "0.000000", 2023-05-31T21:35:38.615 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "realm_epoch": 2 2023-05-31T21:35:38.615 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2023-05-31T21:35:38.616 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2023-05-31T21:35:38.616 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2023-05-31T21:35:38.616 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "key": 2, 2023-05-31T21:35:38.616 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "val": { 2023-05-31T21:35:38.617 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "state": 1, 2023-05-31T21:35:38.617 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "marker": "1_1685562970.274572_93.1", 2023-05-31T21:35:38.617 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "next_step_marker": "", 2023-05-31T21:35:38.617 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "total_entries": 0, 2023-05-31T21:35:38.618 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "pos": 0, 2023-05-31T21:35:38.618 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "timestamp": "2023-05-31T19:56:10.274572Z", 2023-05-31T21:35:38.618 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "realm_epoch": 2 2023-05-31T21:35:38.618 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2023-05-31T21:35:38.619 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2023-05-31T21:35:38.619 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2023-05-31T21:35:38.619 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "key": 3, 2023-05-31T21:35:38.619 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "val": { 2023-05-31T21:35:38.619 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "state": 1, 2023-05-31T21:35:38.620 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "marker": "1_1685563571.700188_212.1", 2023-05-31T21:35:38.620 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "next_step_marker": "", 2023-05-31T21:35:38.620 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "total_entries": 0, 2023-05-31T21:35:38.620 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "pos": 0, 2023-05-31T21:35:38.621 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "timestamp": "2023-05-31T20:06:11.700188Z", 2023-05-31T21:35:38.621 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "realm_epoch": 2 2023-05-31T21:35:38.621 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2023-05-31T21:35:38.621 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2023-05-31T21:35:38.622 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: ] 2023-05-31T21:35:38.622 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2023-05-31T21:35:38.622 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "full_sync": { 2023-05-31T21:35:38.622 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "total": 1, 2023-05-31T21:35:38.622 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "complete": 1 2023-05-31T21:35:38.623 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2023-05-31T21:35:38.623 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "current_time": "2023-05-31T20:11:50Z" 2023-05-31T21:35:38.623 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2023-05-31T21:35:38.623 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2023-05-31T21:35:38.624 INFO:tasks.rgw_multisite_tests: "sync_status": { 2023-05-31T21:35:38.624 INFO:tasks.rgw_multisite_tests: "info": { 2023-05-31T21:35:38.624 INFO:tasks.rgw_multisite_tests: "status": "sync", 2023-05-31T21:35:38.624 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2023-05-31T21:35:38.625 INFO:tasks.rgw_multisite_tests: "period": "2e6123d2-d695-4350-9a01-61a455bc782b", 2023-05-31T21:35:38.625 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-05-31T21:35:38.625 INFO:tasks.rgw_multisite_tests: }, 2023-05-31T21:35:38.625 INFO:tasks.rgw_multisite_tests: "markers": [ 2023-05-31T21:35:38.625 INFO:tasks.rgw_multisite_tests: { 2023-05-31T21:35:38.626 INFO:tasks.rgw_multisite_tests: "key": 0, 2023-05-31T21:35:38.626 INFO:tasks.rgw_multisite_tests: "val": { 2023-05-31T21:35:38.626 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-05-31T21:35:38.626 INFO:tasks.rgw_multisite_tests: "marker": "1_1685562631.516089_59.1", 2023-05-31T21:35:38.627 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-05-31T21:35:38.627 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2023-05-31T21:35:38.627 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-05-31T21:35:38.627 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-05-31T19:50:31.516089Z", 2023-05-31T21:35:38.628 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-05-31T21:35:38.628 INFO:tasks.rgw_multisite_tests: } 2023-05-31T21:35:38.628 INFO:tasks.rgw_multisite_tests: }, 2023-05-31T21:35:38.628 INFO:tasks.rgw_multisite_tests: { 2023-05-31T21:35:38.628 INFO:tasks.rgw_multisite_tests: "key": 1, 2023-05-31T21:35:38.629 INFO:tasks.rgw_multisite_tests: "val": { 2023-05-31T21:35:38.629 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-05-31T21:35:38.629 INFO:tasks.rgw_multisite_tests: "marker": "", 2023-05-31T21:35:38.629 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-05-31T21:35:38.630 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-05-31T21:35:38.630 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-05-31T21:35:38.630 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2023-05-31T21:35:38.630 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-05-31T21:35:38.631 INFO:tasks.rgw_multisite_tests: } 2023-05-31T21:35:38.631 INFO:tasks.rgw_multisite_tests: }, 2023-05-31T21:35:38.631 INFO:tasks.rgw_multisite_tests: { 2023-05-31T21:35:38.631 INFO:tasks.rgw_multisite_tests: "key": 2, 2023-05-31T21:35:38.631 INFO:tasks.rgw_multisite_tests: "val": { 2023-05-31T21:35:38.632 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-05-31T21:35:38.632 INFO:tasks.rgw_multisite_tests: "marker": "1_1685562970.274572_93.1", 2023-05-31T21:35:38.632 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-05-31T21:35:38.632 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-05-31T21:35:38.633 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-05-31T21:35:38.633 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-05-31T19:56:10.274572Z", 2023-05-31T21:35:38.633 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-05-31T21:35:38.633 INFO:tasks.rgw_multisite_tests: } 2023-05-31T21:35:38.633 INFO:tasks.rgw_multisite_tests: }, 2023-05-31T21:35:38.634 INFO:tasks.rgw_multisite_tests: { 2023-05-31T21:35:38.634 INFO:tasks.rgw_multisite_tests: "key": 3, 2023-05-31T21:35:38.634 INFO:tasks.rgw_multisite_tests: "val": { 2023-05-31T21:35:38.634 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-05-31T21:35:38.635 INFO:tasks.rgw_multisite_tests: "marker": "1_1685563571.700188_212.1", 2023-05-31T21:35:38.635 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-05-31T21:35:38.635 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-05-31T21:35:38.635 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-05-31T21:35:38.636 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-05-31T20:06:11.700188Z", 2023-05-31T21:35:38.636 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-05-31T21:35:38.636 INFO:tasks.rgw_multisite_tests: } 2023-05-31T21:35:38.636 INFO:tasks.rgw_multisite_tests: } 2023-05-31T21:35:38.636 INFO:tasks.rgw_multisite_tests: ] 2023-05-31T21:35:38.637 INFO:tasks.rgw_multisite_tests: }, 2023-05-31T21:35:38.637 INFO:tasks.rgw_multisite_tests: "full_sync": { 2023-05-31T21:35:38.637 INFO:tasks.rgw_multisite_tests: "total": 1, 2023-05-31T21:35:38.637 INFO:tasks.rgw_multisite_tests: "complete": 1 2023-05-31T21:35:38.638 INFO:tasks.rgw_multisite_tests: }, 2023-05-31T21:35:38.638 INFO:tasks.rgw_multisite_tests: "current_time": "2023-05-31T20:11:50Z" 2023-05-31T21:35:38.638 INFO:tasks.rgw_multisite_tests:} 2023-05-31T21:35:38.638 INFO:tasks.rgw_multisite_tests: 2023-05-31T21:35:38.639 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1685562631.516089_59.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-05-31T19:50:31.516089Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1685562970.274572_93.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-05-31T19:56:10.274572Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1685563571.700188_212.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-05-31T20:06:11.700188Z', 'realm_epoch': 2}}] 2023-05-31T21:35:38.639 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1685562631.516089_59.1', 1: '', 2: '1_1685562970.274572_93.1', 3: '1_1685563802.468126_296.1'} 2023-05-31T21:35:38.639 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1685562631.516089_59.1', 1: '', 2: '1_1685562970.274572_93.1', 3: '1_1685563571.700188_212.1'} 2023-05-31T21:35:38.639 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone a2 behind master: shard=3 master=1_1685563802.468126_296.1 target=1_1685563571.700188_212.1 2023-05-31T21:35:38.640 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-31T21:35:38.640 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-31T21:35:38.640 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-05-31T21:35:38.640 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: ignoring --setuser ceph since I am not root 2023-05-31T21:35:38.641 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-05-31T21:35:38.641 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: 2023-05-31T20:11:55.767+0000 7f20eeed3780 20 RGW-SYNC:meta: read sync status 2023-05-31T21:35:38.641 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: 2023-05-31T20:11:55.768+0000 7f20eeed3780 20 RGW-SYNC:meta: read sync status 2023-05-31T21:35:38.641 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2023-05-31T21:35:38.641 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "sync_status": { 2023-05-31T21:35:38.642 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "info": { 2023-05-31T21:35:38.642 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "status": "sync", 2023-05-31T21:35:38.642 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "num_shards": 4, 2023-05-31T21:35:38.642 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "period": "2e6123d2-d695-4350-9a01-61a455bc782b", 2023-05-31T21:35:38.643 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "realm_epoch": 2 2023-05-31T21:35:38.643 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2023-05-31T21:35:38.643 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "markers": [ 2023-05-31T21:35:38.643 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2023-05-31T21:35:38.644 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "key": 0, 2023-05-31T21:35:38.644 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "val": { 2023-05-31T21:35:38.644 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "state": 1, 2023-05-31T21:35:38.644 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "marker": "1_1685562631.516089_59.1", 2023-05-31T21:35:38.644 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "next_step_marker": "", 2023-05-31T21:35:38.645 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "total_entries": 1, 2023-05-31T21:35:38.645 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "pos": 0, 2023-05-31T21:35:38.645 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "timestamp": "2023-05-31T19:50:31.516089Z", 2023-05-31T21:35:38.645 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "realm_epoch": 2 2023-05-31T21:35:38.646 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2023-05-31T21:35:38.646 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2023-05-31T21:35:38.646 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2023-05-31T21:35:38.646 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "key": 1, 2023-05-31T21:35:38.646 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "val": { 2023-05-31T21:35:38.647 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "state": 1, 2023-05-31T21:35:38.647 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "marker": "", 2023-05-31T21:35:38.647 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "next_step_marker": "", 2023-05-31T21:35:38.647 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "total_entries": 0, 2023-05-31T21:35:38.648 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "pos": 0, 2023-05-31T21:35:38.648 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "timestamp": "0.000000", 2023-05-31T21:35:38.648 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "realm_epoch": 2 2023-05-31T21:35:38.648 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2023-05-31T21:35:38.649 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2023-05-31T21:35:38.649 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2023-05-31T21:35:38.649 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "key": 2, 2023-05-31T21:35:38.649 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "val": { 2023-05-31T21:35:38.650 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "state": 1, 2023-05-31T21:35:38.650 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "marker": "1_1685562970.274572_93.1", 2023-05-31T21:35:38.650 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "next_step_marker": "", 2023-05-31T21:35:38.650 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "total_entries": 0, 2023-05-31T21:35:38.650 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "pos": 0, 2023-05-31T21:35:38.651 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "timestamp": "2023-05-31T19:56:10.274572Z", 2023-05-31T21:35:38.651 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "realm_epoch": 2 2023-05-31T21:35:38.651 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2023-05-31T21:35:38.651 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2023-05-31T21:35:38.652 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2023-05-31T21:35:38.652 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "key": 3, 2023-05-31T21:35:38.652 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "val": { 2023-05-31T21:35:38.652 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "state": 1, 2023-05-31T21:35:38.653 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "marker": "1_1685563571.700188_212.1", 2023-05-31T21:35:38.653 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "next_step_marker": "", 2023-05-31T21:35:38.653 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "total_entries": 0, 2023-05-31T21:35:38.653 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "pos": 0, 2023-05-31T21:35:38.653 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "timestamp": "2023-05-31T20:06:11.700188Z", 2023-05-31T21:35:38.654 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "realm_epoch": 2 2023-05-31T21:35:38.654 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2023-05-31T21:35:38.654 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2023-05-31T21:35:38.654 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: ] 2023-05-31T21:35:38.655 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2023-05-31T21:35:38.655 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "full_sync": { 2023-05-31T21:35:38.655 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "total": 1, 2023-05-31T21:35:38.655 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "complete": 1 2023-05-31T21:35:38.656 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2023-05-31T21:35:38.656 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "current_time": "2023-05-31T20:11:55Z" 2023-05-31T21:35:38.656 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2023-05-31T21:35:38.656 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2023-05-31T21:35:38.657 INFO:tasks.rgw_multisite_tests: "sync_status": { 2023-05-31T21:35:38.657 INFO:tasks.rgw_multisite_tests: "info": { 2023-05-31T21:35:38.657 INFO:tasks.rgw_multisite_tests: "status": "sync", 2023-05-31T21:35:38.657 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2023-05-31T21:35:38.658 INFO:tasks.rgw_multisite_tests: "period": "2e6123d2-d695-4350-9a01-61a455bc782b", 2023-05-31T21:35:38.658 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-05-31T21:35:38.658 INFO:tasks.rgw_multisite_tests: }, 2023-05-31T21:35:38.658 INFO:tasks.rgw_multisite_tests: "markers": [ 2023-05-31T21:35:38.659 INFO:tasks.rgw_multisite_tests: { 2023-05-31T21:35:38.659 INFO:tasks.rgw_multisite_tests: "key": 0, 2023-05-31T21:35:38.659 INFO:tasks.rgw_multisite_tests: "val": { 2023-05-31T21:35:38.659 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-05-31T21:35:38.660 INFO:tasks.rgw_multisite_tests: "marker": "1_1685562631.516089_59.1", 2023-05-31T21:35:38.660 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-05-31T21:35:38.660 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2023-05-31T21:35:38.660 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-05-31T21:35:38.661 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-05-31T19:50:31.516089Z", 2023-05-31T21:35:38.661 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-05-31T21:35:38.661 INFO:tasks.rgw_multisite_tests: } 2023-05-31T21:35:38.661 INFO:tasks.rgw_multisite_tests: }, 2023-05-31T21:35:38.661 INFO:tasks.rgw_multisite_tests: { 2023-05-31T21:35:38.662 INFO:tasks.rgw_multisite_tests: "key": 1, 2023-05-31T21:35:38.662 INFO:tasks.rgw_multisite_tests: "val": { 2023-05-31T21:35:38.662 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-05-31T21:35:38.662 INFO:tasks.rgw_multisite_tests: "marker": "", 2023-05-31T21:35:38.663 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-05-31T21:35:38.663 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-05-31T21:35:38.663 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-05-31T21:35:38.663 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2023-05-31T21:35:38.663 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-05-31T21:35:38.664 INFO:tasks.rgw_multisite_tests: } 2023-05-31T21:35:38.664 INFO:tasks.rgw_multisite_tests: }, 2023-05-31T21:35:38.664 INFO:tasks.rgw_multisite_tests: { 2023-05-31T21:35:38.665 INFO:tasks.rgw_multisite_tests: "key": 2, 2023-05-31T21:35:38.665 INFO:tasks.rgw_multisite_tests: "val": { 2023-05-31T21:35:38.665 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-05-31T21:35:38.665 INFO:tasks.rgw_multisite_tests: "marker": "1_1685562970.274572_93.1", 2023-05-31T21:35:38.665 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-05-31T21:35:38.666 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-05-31T21:35:38.666 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-05-31T21:35:38.666 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-05-31T19:56:10.274572Z", 2023-05-31T21:35:38.666 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-05-31T21:35:38.667 INFO:tasks.rgw_multisite_tests: } 2023-05-31T21:35:38.667 INFO:tasks.rgw_multisite_tests: }, 2023-05-31T21:35:38.667 INFO:tasks.rgw_multisite_tests: { 2023-05-31T21:35:38.667 INFO:tasks.rgw_multisite_tests: "key": 3, 2023-05-31T21:35:38.668 INFO:tasks.rgw_multisite_tests: "val": { 2023-05-31T21:35:38.668 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-05-31T21:35:38.668 INFO:tasks.rgw_multisite_tests: "marker": "1_1685563571.700188_212.1", 2023-05-31T21:35:38.668 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-05-31T21:35:38.669 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-05-31T21:35:38.669 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-05-31T21:35:38.669 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-05-31T20:06:11.700188Z", 2023-05-31T21:35:38.669 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-05-31T21:35:38.669 INFO:tasks.rgw_multisite_tests: } 2023-05-31T21:35:38.670 INFO:tasks.rgw_multisite_tests: } 2023-05-31T21:35:38.670 INFO:tasks.rgw_multisite_tests: ] 2023-05-31T21:35:38.670 INFO:tasks.rgw_multisite_tests: }, 2023-05-31T21:35:38.670 INFO:tasks.rgw_multisite_tests: "full_sync": { 2023-05-31T21:35:38.671 INFO:tasks.rgw_multisite_tests: "total": 1, 2023-05-31T21:35:38.671 INFO:tasks.rgw_multisite_tests: "complete": 1 2023-05-31T21:35:38.671 INFO:tasks.rgw_multisite_tests: }, 2023-05-31T21:35:38.671 INFO:tasks.rgw_multisite_tests: "current_time": "2023-05-31T20:11:55Z" 2023-05-31T21:35:38.672 INFO:tasks.rgw_multisite_tests:} 2023-05-31T21:35:38.672 INFO:tasks.rgw_multisite_tests: 2023-05-31T21:35:38.672 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1685562631.516089_59.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-05-31T19:50:31.516089Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1685562970.274572_93.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-05-31T19:56:10.274572Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1685563571.700188_212.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-05-31T20:06:11.700188Z', 'realm_epoch': 2}}] 2023-05-31T21:35:38.672 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1685562631.516089_59.1', 1: '', 2: '1_1685562970.274572_93.1', 3: '1_1685563802.468126_296.1'} 2023-05-31T21:35:38.673 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1685562631.516089_59.1', 1: '', 2: '1_1685562970.274572_93.1', 3: '1_1685563571.700188_212.1'} 2023-05-31T21:35:38.673 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone a2 behind master: shard=3 master=1_1685563802.468126_296.1 target=1_1685563571.700188_212.1 2023-05-31T21:35:38.673 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-31T21:35:38.673 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-31T21:35:38.673 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-05-31T21:35:38.674 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: ignoring --setuser ceph since I am not root 2023-05-31T21:35:38.674 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-05-31T21:35:38.674 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: 2023-05-31T20:12:00.905+0000 7feb68131780 20 RGW-SYNC:meta: read sync status 2023-05-31T21:35:38.674 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: 2023-05-31T20:12:00.906+0000 7feb68131780 20 RGW-SYNC:meta: read sync status 2023-05-31T21:35:38.675 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2023-05-31T21:35:38.675 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "sync_status": { 2023-05-31T21:35:38.675 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "info": { 2023-05-31T21:35:38.675 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "status": "sync", 2023-05-31T21:35:38.676 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "num_shards": 4, 2023-05-31T21:35:38.676 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "period": "2e6123d2-d695-4350-9a01-61a455bc782b", 2023-05-31T21:35:38.676 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "realm_epoch": 2 2023-05-31T21:35:38.676 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2023-05-31T21:35:38.677 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "markers": [ 2023-05-31T21:35:38.677 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2023-05-31T21:35:38.677 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "key": 0, 2023-05-31T21:35:38.677 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "val": { 2023-05-31T21:35:38.678 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "state": 1, 2023-05-31T21:35:38.678 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "marker": "1_1685562631.516089_59.1", 2023-05-31T21:35:38.678 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "next_step_marker": "", 2023-05-31T21:35:38.678 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "total_entries": 1, 2023-05-31T21:35:38.679 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "pos": 0, 2023-05-31T21:35:38.679 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "timestamp": "2023-05-31T19:50:31.516089Z", 2023-05-31T21:35:38.679 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "realm_epoch": 2 2023-05-31T21:35:38.679 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2023-05-31T21:35:38.680 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2023-05-31T21:35:38.680 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2023-05-31T21:35:38.680 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "key": 1, 2023-05-31T21:35:38.680 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "val": { 2023-05-31T21:35:38.680 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "state": 1, 2023-05-31T21:35:38.681 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "marker": "", 2023-05-31T21:35:38.681 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "next_step_marker": "", 2023-05-31T21:35:38.681 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "total_entries": 0, 2023-05-31T21:35:38.681 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "pos": 0, 2023-05-31T21:35:38.682 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "timestamp": "0.000000", 2023-05-31T21:35:38.682 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "realm_epoch": 2 2023-05-31T21:35:38.682 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2023-05-31T21:35:38.682 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2023-05-31T21:35:38.683 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2023-05-31T21:35:38.683 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "key": 2, 2023-05-31T21:35:38.683 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "val": { 2023-05-31T21:35:38.683 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "state": 1, 2023-05-31T21:35:38.683 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "marker": "1_1685562970.274572_93.1", 2023-05-31T21:35:38.684 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "next_step_marker": "", 2023-05-31T21:35:38.684 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "total_entries": 0, 2023-05-31T21:35:38.684 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "pos": 0, 2023-05-31T21:35:38.684 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "timestamp": "2023-05-31T19:56:10.274572Z", 2023-05-31T21:35:38.685 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "realm_epoch": 2 2023-05-31T21:35:38.685 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2023-05-31T21:35:38.685 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2023-05-31T21:35:38.685 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2023-05-31T21:35:38.685 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "key": 3, 2023-05-31T21:35:38.686 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "val": { 2023-05-31T21:35:38.686 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "state": 1, 2023-05-31T21:35:38.686 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "marker": "1_1685563798.744769_245.1", 2023-05-31T21:35:38.686 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "next_step_marker": "", 2023-05-31T21:35:38.687 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "total_entries": 0, 2023-05-31T21:35:38.687 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "pos": 0, 2023-05-31T21:35:38.687 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "timestamp": "2023-05-31T20:09:58.744769Z", 2023-05-31T21:35:38.687 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "realm_epoch": 2 2023-05-31T21:35:38.688 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2023-05-31T21:35:38.688 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2023-05-31T21:35:38.688 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: ] 2023-05-31T21:35:38.688 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2023-05-31T21:35:38.689 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "full_sync": { 2023-05-31T21:35:38.689 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "total": 1, 2023-05-31T21:35:38.689 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "complete": 1 2023-05-31T21:35:38.689 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2023-05-31T21:35:38.689 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "current_time": "2023-05-31T20:12:00Z" 2023-05-31T21:35:38.690 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2023-05-31T21:35:38.690 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2023-05-31T21:35:38.690 INFO:tasks.rgw_multisite_tests: "sync_status": { 2023-05-31T21:35:38.690 INFO:tasks.rgw_multisite_tests: "info": { 2023-05-31T21:35:38.691 INFO:tasks.rgw_multisite_tests: "status": "sync", 2023-05-31T21:35:38.691 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2023-05-31T21:35:38.691 INFO:tasks.rgw_multisite_tests: "period": "2e6123d2-d695-4350-9a01-61a455bc782b", 2023-05-31T21:35:38.691 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-05-31T21:35:38.692 INFO:tasks.rgw_multisite_tests: }, 2023-05-31T21:35:38.692 INFO:tasks.rgw_multisite_tests: "markers": [ 2023-05-31T21:35:38.692 INFO:tasks.rgw_multisite_tests: { 2023-05-31T21:35:38.692 INFO:tasks.rgw_multisite_tests: "key": 0, 2023-05-31T21:35:38.693 INFO:tasks.rgw_multisite_tests: "val": { 2023-05-31T21:35:38.693 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-05-31T21:35:38.693 INFO:tasks.rgw_multisite_tests: "marker": "1_1685562631.516089_59.1", 2023-05-31T21:35:38.693 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-05-31T21:35:38.694 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2023-05-31T21:35:38.694 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-05-31T21:35:38.694 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-05-31T19:50:31.516089Z", 2023-05-31T21:35:38.694 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-05-31T21:35:38.695 INFO:tasks.rgw_multisite_tests: } 2023-05-31T21:35:38.695 INFO:tasks.rgw_multisite_tests: }, 2023-05-31T21:35:38.695 INFO:tasks.rgw_multisite_tests: { 2023-05-31T21:35:38.695 INFO:tasks.rgw_multisite_tests: "key": 1, 2023-05-31T21:35:38.695 INFO:tasks.rgw_multisite_tests: "val": { 2023-05-31T21:35:38.696 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-05-31T21:35:38.696 INFO:tasks.rgw_multisite_tests: "marker": "", 2023-05-31T21:35:38.696 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-05-31T21:35:38.696 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-05-31T21:35:38.697 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-05-31T21:35:38.697 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2023-05-31T21:35:38.697 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-05-31T21:35:38.697 INFO:tasks.rgw_multisite_tests: } 2023-05-31T21:35:38.698 INFO:tasks.rgw_multisite_tests: }, 2023-05-31T21:35:38.698 INFO:tasks.rgw_multisite_tests: { 2023-05-31T21:35:38.698 INFO:tasks.rgw_multisite_tests: "key": 2, 2023-05-31T21:35:38.698 INFO:tasks.rgw_multisite_tests: "val": { 2023-05-31T21:35:38.699 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-05-31T21:35:38.699 INFO:tasks.rgw_multisite_tests: "marker": "1_1685562970.274572_93.1", 2023-05-31T21:35:38.699 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-05-31T21:35:38.699 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-05-31T21:35:38.700 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-05-31T21:35:38.700 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-05-31T19:56:10.274572Z", 2023-05-31T21:35:38.700 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-05-31T21:35:38.700 INFO:tasks.rgw_multisite_tests: } 2023-05-31T21:35:38.701 INFO:tasks.rgw_multisite_tests: }, 2023-05-31T21:35:38.701 INFO:tasks.rgw_multisite_tests: { 2023-05-31T21:35:38.701 INFO:tasks.rgw_multisite_tests: "key": 3, 2023-05-31T21:35:38.701 INFO:tasks.rgw_multisite_tests: "val": { 2023-05-31T21:35:38.702 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-05-31T21:35:38.702 INFO:tasks.rgw_multisite_tests: "marker": "1_1685563798.744769_245.1", 2023-05-31T21:35:38.702 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-05-31T21:35:38.702 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-05-31T21:35:38.702 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-05-31T21:35:38.703 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-05-31T20:09:58.744769Z", 2023-05-31T21:35:38.703 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-05-31T21:35:38.703 INFO:tasks.rgw_multisite_tests: } 2023-05-31T21:35:38.703 INFO:tasks.rgw_multisite_tests: } 2023-05-31T21:35:38.704 INFO:tasks.rgw_multisite_tests: ] 2023-05-31T21:35:38.704 INFO:tasks.rgw_multisite_tests: }, 2023-05-31T21:35:38.704 INFO:tasks.rgw_multisite_tests: "full_sync": { 2023-05-31T21:35:38.704 INFO:tasks.rgw_multisite_tests: "total": 1, 2023-05-31T21:35:38.705 INFO:tasks.rgw_multisite_tests: "complete": 1 2023-05-31T21:35:38.705 INFO:tasks.rgw_multisite_tests: }, 2023-05-31T21:35:38.705 INFO:tasks.rgw_multisite_tests: "current_time": "2023-05-31T20:12:00Z" 2023-05-31T21:35:38.705 INFO:tasks.rgw_multisite_tests:} 2023-05-31T21:35:38.705 INFO:tasks.rgw_multisite_tests: 2023-05-31T21:35:38.706 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1685562631.516089_59.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-05-31T19:50:31.516089Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1685562970.274572_93.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-05-31T19:56:10.274572Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1685563798.744769_245.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-05-31T20:09:58.744769Z', 'realm_epoch': 2}}] 2023-05-31T21:35:38.706 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1685562631.516089_59.1', 1: '', 2: '1_1685562970.274572_93.1', 3: '1_1685563802.468126_296.1'} 2023-05-31T21:35:38.706 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1685562631.516089_59.1', 1: '', 2: '1_1685562970.274572_93.1', 3: '1_1685563798.744769_245.1'} 2023-05-31T21:35:38.706 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone a2 behind master: shard=3 master=1_1685563802.468126_296.1 target=1_1685563798.744769_245.1 2023-05-31T21:35:38.707 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-31T21:35:38.707 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-31T21:35:38.707 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-05-31T21:35:38.707 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: ignoring --setuser ceph since I am not root 2023-05-31T21:35:38.708 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-05-31T21:35:38.708 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: 2023-05-31T20:12:06.033+0000 7f7aa0265780 20 RGW-SYNC:meta: read sync status 2023-05-31T21:35:38.708 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: 2023-05-31T20:12:06.034+0000 7f7aa0265780 20 RGW-SYNC:meta: read sync status 2023-05-31T21:35:38.708 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2023-05-31T21:35:38.709 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "sync_status": { 2023-05-31T21:35:38.709 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "info": { 2023-05-31T21:35:38.709 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "status": "sync", 2023-05-31T21:35:38.709 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "num_shards": 4, 2023-05-31T21:35:38.709 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "period": "2e6123d2-d695-4350-9a01-61a455bc782b", 2023-05-31T21:35:38.710 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "realm_epoch": 2 2023-05-31T21:35:38.710 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2023-05-31T21:35:38.710 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "markers": [ 2023-05-31T21:35:38.710 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2023-05-31T21:35:38.711 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "key": 0, 2023-05-31T21:35:38.711 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "val": { 2023-05-31T21:35:38.711 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "state": 1, 2023-05-31T21:35:38.711 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "marker": "1_1685562631.516089_59.1", 2023-05-31T21:35:38.712 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "next_step_marker": "", 2023-05-31T21:35:38.712 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "total_entries": 1, 2023-05-31T21:35:38.712 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "pos": 0, 2023-05-31T21:35:38.712 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "timestamp": "2023-05-31T19:50:31.516089Z", 2023-05-31T21:35:38.713 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "realm_epoch": 2 2023-05-31T21:35:38.713 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2023-05-31T21:35:38.713 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2023-05-31T21:35:38.713 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2023-05-31T21:35:38.714 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "key": 1, 2023-05-31T21:35:38.714 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "val": { 2023-05-31T21:35:38.714 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "state": 1, 2023-05-31T21:35:38.714 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "marker": "", 2023-05-31T21:35:38.714 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "next_step_marker": "", 2023-05-31T21:35:38.715 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "total_entries": 0, 2023-05-31T21:35:38.715 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "pos": 0, 2023-05-31T21:35:38.715 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "timestamp": "0.000000", 2023-05-31T21:35:38.715 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "realm_epoch": 2 2023-05-31T21:35:38.716 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2023-05-31T21:35:38.716 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2023-05-31T21:35:38.716 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2023-05-31T21:35:38.716 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "key": 2, 2023-05-31T21:35:38.717 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "val": { 2023-05-31T21:35:38.717 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "state": 1, 2023-05-31T21:35:38.717 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "marker": "1_1685562970.274572_93.1", 2023-05-31T21:35:38.717 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "next_step_marker": "", 2023-05-31T21:35:38.717 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "total_entries": 0, 2023-05-31T21:35:38.718 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "pos": 0, 2023-05-31T21:35:38.718 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "timestamp": "2023-05-31T19:56:10.274572Z", 2023-05-31T21:35:38.718 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "realm_epoch": 2 2023-05-31T21:35:38.718 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2023-05-31T21:35:38.719 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2023-05-31T21:35:38.719 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2023-05-31T21:35:38.719 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "key": 3, 2023-05-31T21:35:38.719 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "val": { 2023-05-31T21:35:38.720 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "state": 1, 2023-05-31T21:35:38.720 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "marker": "1_1685563798.970303_257.1", 2023-05-31T21:35:38.720 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "next_step_marker": "", 2023-05-31T21:35:38.720 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "total_entries": 0, 2023-05-31T21:35:38.721 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "pos": 0, 2023-05-31T21:35:38.721 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "timestamp": "2023-05-31T20:09:58.970303Z", 2023-05-31T21:35:38.721 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "realm_epoch": 2 2023-05-31T21:35:38.721 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2023-05-31T21:35:38.721 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2023-05-31T21:35:38.722 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: ] 2023-05-31T21:35:38.722 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2023-05-31T21:35:38.722 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "full_sync": { 2023-05-31T21:35:38.722 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "total": 1, 2023-05-31T21:35:38.723 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "complete": 1 2023-05-31T21:35:38.723 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2023-05-31T21:35:38.723 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "current_time": "2023-05-31T20:12:06Z" 2023-05-31T21:35:38.723 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2023-05-31T21:35:38.724 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2023-05-31T21:35:38.724 INFO:tasks.rgw_multisite_tests: "sync_status": { 2023-05-31T21:35:38.724 INFO:tasks.rgw_multisite_tests: "info": { 2023-05-31T21:35:38.724 INFO:tasks.rgw_multisite_tests: "status": "sync", 2023-05-31T21:35:38.724 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2023-05-31T21:35:38.725 INFO:tasks.rgw_multisite_tests: "period": "2e6123d2-d695-4350-9a01-61a455bc782b", 2023-05-31T21:35:38.725 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-05-31T21:35:38.725 INFO:tasks.rgw_multisite_tests: }, 2023-05-31T21:35:38.725 INFO:tasks.rgw_multisite_tests: "markers": [ 2023-05-31T21:35:38.726 INFO:tasks.rgw_multisite_tests: { 2023-05-31T21:35:38.726 INFO:tasks.rgw_multisite_tests: "key": 0, 2023-05-31T21:35:38.726 INFO:tasks.rgw_multisite_tests: "val": { 2023-05-31T21:35:38.726 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-05-31T21:35:38.727 INFO:tasks.rgw_multisite_tests: "marker": "1_1685562631.516089_59.1", 2023-05-31T21:35:38.727 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-05-31T21:35:38.727 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2023-05-31T21:35:38.727 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-05-31T21:35:38.728 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-05-31T19:50:31.516089Z", 2023-05-31T21:35:38.728 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-05-31T21:35:38.728 INFO:tasks.rgw_multisite_tests: } 2023-05-31T21:35:38.728 INFO:tasks.rgw_multisite_tests: }, 2023-05-31T21:35:38.728 INFO:tasks.rgw_multisite_tests: { 2023-05-31T21:35:38.729 INFO:tasks.rgw_multisite_tests: "key": 1, 2023-05-31T21:35:38.729 INFO:tasks.rgw_multisite_tests: "val": { 2023-05-31T21:35:38.729 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-05-31T21:35:38.729 INFO:tasks.rgw_multisite_tests: "marker": "", 2023-05-31T21:35:38.730 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-05-31T21:35:38.730 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-05-31T21:35:38.730 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-05-31T21:35:38.730 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2023-05-31T21:35:38.730 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-05-31T21:35:38.731 INFO:tasks.rgw_multisite_tests: } 2023-05-31T21:35:38.731 INFO:tasks.rgw_multisite_tests: }, 2023-05-31T21:35:38.731 INFO:tasks.rgw_multisite_tests: { 2023-05-31T21:35:38.731 INFO:tasks.rgw_multisite_tests: "key": 2, 2023-05-31T21:35:38.732 INFO:tasks.rgw_multisite_tests: "val": { 2023-05-31T21:35:38.732 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-05-31T21:35:38.732 INFO:tasks.rgw_multisite_tests: "marker": "1_1685562970.274572_93.1", 2023-05-31T21:35:38.732 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-05-31T21:35:38.733 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-05-31T21:35:38.733 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-05-31T21:35:38.733 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-05-31T19:56:10.274572Z", 2023-05-31T21:35:38.733 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-05-31T21:35:38.733 INFO:tasks.rgw_multisite_tests: } 2023-05-31T21:35:38.734 INFO:tasks.rgw_multisite_tests: }, 2023-05-31T21:35:38.734 INFO:tasks.rgw_multisite_tests: { 2023-05-31T21:35:38.734 INFO:tasks.rgw_multisite_tests: "key": 3, 2023-05-31T21:35:38.734 INFO:tasks.rgw_multisite_tests: "val": { 2023-05-31T21:35:38.735 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-05-31T21:35:38.735 INFO:tasks.rgw_multisite_tests: "marker": "1_1685563798.970303_257.1", 2023-05-31T21:35:38.735 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-05-31T21:35:38.735 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-05-31T21:35:38.736 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-05-31T21:35:38.736 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-05-31T20:09:58.970303Z", 2023-05-31T21:35:38.736 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-05-31T21:35:38.736 INFO:tasks.rgw_multisite_tests: } 2023-05-31T21:35:38.736 INFO:tasks.rgw_multisite_tests: } 2023-05-31T21:35:38.737 INFO:tasks.rgw_multisite_tests: ] 2023-05-31T21:35:38.737 INFO:tasks.rgw_multisite_tests: }, 2023-05-31T21:35:38.737 INFO:tasks.rgw_multisite_tests: "full_sync": { 2023-05-31T21:35:38.737 INFO:tasks.rgw_multisite_tests: "total": 1, 2023-05-31T21:35:38.738 INFO:tasks.rgw_multisite_tests: "complete": 1 2023-05-31T21:35:38.738 INFO:tasks.rgw_multisite_tests: }, 2023-05-31T21:35:38.738 INFO:tasks.rgw_multisite_tests: "current_time": "2023-05-31T20:12:06Z" 2023-05-31T21:35:38.738 INFO:tasks.rgw_multisite_tests:} 2023-05-31T21:35:38.739 INFO:tasks.rgw_multisite_tests: 2023-05-31T21:35:38.739 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1685562631.516089_59.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-05-31T19:50:31.516089Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1685562970.274572_93.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-05-31T19:56:10.274572Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1685563798.970303_257.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-05-31T20:09:58.970303Z', 'realm_epoch': 2}}] 2023-05-31T21:35:38.739 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1685562631.516089_59.1', 1: '', 2: '1_1685562970.274572_93.1', 3: '1_1685563802.468126_296.1'} 2023-05-31T21:35:38.739 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1685562631.516089_59.1', 1: '', 2: '1_1685562970.274572_93.1', 3: '1_1685563798.970303_257.1'} 2023-05-31T21:35:38.740 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone a2 behind master: shard=3 master=1_1685563802.468126_296.1 target=1_1685563798.970303_257.1 2023-05-31T21:35:38.740 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-31T21:35:38.740 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-31T21:35:38.740 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-05-31T21:35:38.741 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: ignoring --setuser ceph since I am not root 2023-05-31T21:35:38.741 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-05-31T21:35:38.741 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: 2023-05-31T20:12:11.158+0000 7fd83d63a780 20 RGW-SYNC:meta: read sync status 2023-05-31T21:35:38.741 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: 2023-05-31T20:12:11.159+0000 7fd83d63a780 20 RGW-SYNC:meta: read sync status 2023-05-31T21:35:38.741 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2023-05-31T21:35:38.742 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "sync_status": { 2023-05-31T21:35:38.742 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "info": { 2023-05-31T21:35:38.742 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "status": "sync", 2023-05-31T21:35:38.742 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "num_shards": 4, 2023-05-31T21:35:38.743 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "period": "2e6123d2-d695-4350-9a01-61a455bc782b", 2023-05-31T21:35:38.743 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "realm_epoch": 2 2023-05-31T21:35:38.743 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2023-05-31T21:35:38.743 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "markers": [ 2023-05-31T21:35:38.744 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2023-05-31T21:35:38.744 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "key": 0, 2023-05-31T21:35:38.744 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "val": { 2023-05-31T21:35:38.744 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "state": 1, 2023-05-31T21:35:38.745 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "marker": "1_1685562631.516089_59.1", 2023-05-31T21:35:38.745 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "next_step_marker": "", 2023-05-31T21:35:38.745 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "total_entries": 1, 2023-05-31T21:35:38.745 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "pos": 0, 2023-05-31T21:35:38.745 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "timestamp": "2023-05-31T19:50:31.516089Z", 2023-05-31T21:35:38.746 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "realm_epoch": 2 2023-05-31T21:35:38.746 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2023-05-31T21:35:38.746 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2023-05-31T21:35:38.746 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2023-05-31T21:35:38.747 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "key": 1, 2023-05-31T21:35:38.747 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "val": { 2023-05-31T21:35:38.747 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "state": 1, 2023-05-31T21:35:38.747 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "marker": "", 2023-05-31T21:35:38.748 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "next_step_marker": "", 2023-05-31T21:35:38.748 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "total_entries": 0, 2023-05-31T21:35:38.748 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "pos": 0, 2023-05-31T21:35:38.748 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "timestamp": "0.000000", 2023-05-31T21:35:38.749 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "realm_epoch": 2 2023-05-31T21:35:38.749 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2023-05-31T21:35:38.749 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2023-05-31T21:35:38.749 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2023-05-31T21:35:38.749 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "key": 2, 2023-05-31T21:35:38.750 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "val": { 2023-05-31T21:35:38.750 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "state": 1, 2023-05-31T21:35:38.750 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "marker": "1_1685562970.274572_93.1", 2023-05-31T21:35:38.750 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "next_step_marker": "", 2023-05-31T21:35:38.751 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "total_entries": 0, 2023-05-31T21:35:38.751 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "pos": 0, 2023-05-31T21:35:38.751 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "timestamp": "2023-05-31T19:56:10.274572Z", 2023-05-31T21:35:38.751 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "realm_epoch": 2 2023-05-31T21:35:38.752 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2023-05-31T21:35:38.752 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2023-05-31T21:35:38.752 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2023-05-31T21:35:38.752 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "key": 3, 2023-05-31T21:35:38.752 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "val": { 2023-05-31T21:35:38.753 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "state": 1, 2023-05-31T21:35:38.753 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "marker": "1_1685563798.885419_253.1", 2023-05-31T21:35:38.753 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "next_step_marker": "", 2023-05-31T21:35:38.753 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "total_entries": 0, 2023-05-31T21:35:38.754 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "pos": 0, 2023-05-31T21:35:38.754 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "timestamp": "2023-05-31T20:09:58.885419Z", 2023-05-31T21:35:38.754 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "realm_epoch": 2 2023-05-31T21:35:38.754 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2023-05-31T21:35:38.755 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2023-05-31T21:35:38.755 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: ] 2023-05-31T21:35:38.755 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2023-05-31T21:35:38.755 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "full_sync": { 2023-05-31T21:35:38.756 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "total": 1, 2023-05-31T21:35:38.756 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "complete": 1 2023-05-31T21:35:38.756 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2023-05-31T21:35:38.756 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "current_time": "2023-05-31T20:12:11Z" 2023-05-31T21:35:38.757 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2023-05-31T21:35:38.757 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2023-05-31T21:35:38.757 INFO:tasks.rgw_multisite_tests: "sync_status": { 2023-05-31T21:35:38.757 INFO:tasks.rgw_multisite_tests: "info": { 2023-05-31T21:35:38.758 INFO:tasks.rgw_multisite_tests: "status": "sync", 2023-05-31T21:35:38.758 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2023-05-31T21:35:38.758 INFO:tasks.rgw_multisite_tests: "period": "2e6123d2-d695-4350-9a01-61a455bc782b", 2023-05-31T21:35:38.758 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-05-31T21:35:38.759 INFO:tasks.rgw_multisite_tests: }, 2023-05-31T21:35:38.759 INFO:tasks.rgw_multisite_tests: "markers": [ 2023-05-31T21:35:38.759 INFO:tasks.rgw_multisite_tests: { 2023-05-31T21:35:38.759 INFO:tasks.rgw_multisite_tests: "key": 0, 2023-05-31T21:35:38.760 INFO:tasks.rgw_multisite_tests: "val": { 2023-05-31T21:35:38.760 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-05-31T21:35:38.760 INFO:tasks.rgw_multisite_tests: "marker": "1_1685562631.516089_59.1", 2023-05-31T21:35:38.760 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-05-31T21:35:38.760 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2023-05-31T21:35:38.761 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-05-31T21:35:38.761 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-05-31T19:50:31.516089Z", 2023-05-31T21:35:38.761 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-05-31T21:35:38.761 INFO:tasks.rgw_multisite_tests: } 2023-05-31T21:35:38.762 INFO:tasks.rgw_multisite_tests: }, 2023-05-31T21:35:38.762 INFO:tasks.rgw_multisite_tests: { 2023-05-31T21:35:38.762 INFO:tasks.rgw_multisite_tests: "key": 1, 2023-05-31T21:35:38.762 INFO:tasks.rgw_multisite_tests: "val": { 2023-05-31T21:35:38.763 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-05-31T21:35:38.763 INFO:tasks.rgw_multisite_tests: "marker": "", 2023-05-31T21:35:38.763 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-05-31T21:35:38.763 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-05-31T21:35:38.763 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-05-31T21:35:38.764 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2023-05-31T21:35:38.764 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-05-31T21:35:38.764 INFO:tasks.rgw_multisite_tests: } 2023-05-31T21:35:38.764 INFO:tasks.rgw_multisite_tests: }, 2023-05-31T21:35:38.765 INFO:tasks.rgw_multisite_tests: { 2023-05-31T21:35:38.765 INFO:tasks.rgw_multisite_tests: "key": 2, 2023-05-31T21:35:38.765 INFO:tasks.rgw_multisite_tests: "val": { 2023-05-31T21:35:38.765 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-05-31T21:35:38.766 INFO:tasks.rgw_multisite_tests: "marker": "1_1685562970.274572_93.1", 2023-05-31T21:35:38.766 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-05-31T21:35:38.766 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-05-31T21:35:38.766 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-05-31T21:35:38.767 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-05-31T19:56:10.274572Z", 2023-05-31T21:35:38.767 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-05-31T21:35:38.767 INFO:tasks.rgw_multisite_tests: } 2023-05-31T21:35:38.767 INFO:tasks.rgw_multisite_tests: }, 2023-05-31T21:35:38.767 INFO:tasks.rgw_multisite_tests: { 2023-05-31T21:35:38.768 INFO:tasks.rgw_multisite_tests: "key": 3, 2023-05-31T21:35:38.768 INFO:tasks.rgw_multisite_tests: "val": { 2023-05-31T21:35:38.768 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-05-31T21:35:38.769 INFO:tasks.rgw_multisite_tests: "marker": "1_1685563798.885419_253.1", 2023-05-31T21:35:38.769 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-05-31T21:35:38.769 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-05-31T21:35:38.769 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-05-31T21:35:38.770 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-05-31T20:09:58.885419Z", 2023-05-31T21:35:38.770 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-05-31T21:35:38.770 INFO:tasks.rgw_multisite_tests: } 2023-05-31T21:35:38.770 INFO:tasks.rgw_multisite_tests: } 2023-05-31T21:35:38.771 INFO:tasks.rgw_multisite_tests: ] 2023-05-31T21:35:38.771 INFO:tasks.rgw_multisite_tests: }, 2023-05-31T21:35:38.771 INFO:tasks.rgw_multisite_tests: "full_sync": { 2023-05-31T21:35:38.771 INFO:tasks.rgw_multisite_tests: "total": 1, 2023-05-31T21:35:38.771 INFO:tasks.rgw_multisite_tests: "complete": 1 2023-05-31T21:35:38.772 INFO:tasks.rgw_multisite_tests: }, 2023-05-31T21:35:38.772 INFO:tasks.rgw_multisite_tests: "current_time": "2023-05-31T20:12:11Z" 2023-05-31T21:35:38.772 INFO:tasks.rgw_multisite_tests:} 2023-05-31T21:35:38.772 INFO:tasks.rgw_multisite_tests: 2023-05-31T21:35:38.773 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1685562631.516089_59.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-05-31T19:50:31.516089Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1685562970.274572_93.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-05-31T19:56:10.274572Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1685563798.885419_253.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-05-31T20:09:58.885419Z', 'realm_epoch': 2}}] 2023-05-31T21:35:38.773 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1685562631.516089_59.1', 1: '', 2: '1_1685562970.274572_93.1', 3: '1_1685563802.468126_296.1'} 2023-05-31T21:35:38.773 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1685562631.516089_59.1', 1: '', 2: '1_1685562970.274572_93.1', 3: '1_1685563798.885419_253.1'} 2023-05-31T21:35:38.773 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone a2 behind master: shard=3 master=1_1685563802.468126_296.1 target=1_1685563798.885419_253.1 2023-05-31T21:35:38.774 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-31T21:35:38.774 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-31T21:35:38.774 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-05-31T21:35:38.774 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: ignoring --setuser ceph since I am not root 2023-05-31T21:35:38.774 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-05-31T21:35:38.775 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: 2023-05-31T20:12:16.312+0000 7f46d13db780 20 RGW-SYNC:meta: read sync status 2023-05-31T21:35:38.775 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: 2023-05-31T20:12:16.312+0000 7f46d13db780 20 RGW-SYNC:meta: read sync status 2023-05-31T21:35:38.775 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2023-05-31T21:35:38.775 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "sync_status": { 2023-05-31T21:35:38.776 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "info": { 2023-05-31T21:35:38.776 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "status": "sync", 2023-05-31T21:35:38.777 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "num_shards": 4, 2023-05-31T21:35:38.777 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "period": "2e6123d2-d695-4350-9a01-61a455bc782b", 2023-05-31T21:35:38.777 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "realm_epoch": 2 2023-05-31T21:35:38.778 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2023-05-31T21:35:38.778 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "markers": [ 2023-05-31T21:35:38.778 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2023-05-31T21:35:38.779 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "key": 0, 2023-05-31T21:35:38.779 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "val": { 2023-05-31T21:35:38.780 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "state": 1, 2023-05-31T21:35:38.780 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "marker": "1_1685562631.516089_59.1", 2023-05-31T21:35:38.781 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "next_step_marker": "", 2023-05-31T21:35:38.781 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "total_entries": 1, 2023-05-31T21:35:38.782 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "pos": 0, 2023-05-31T21:35:38.782 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "timestamp": "2023-05-31T19:50:31.516089Z", 2023-05-31T21:35:38.782 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "realm_epoch": 2 2023-05-31T21:35:38.782 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2023-05-31T21:35:38.783 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2023-05-31T21:35:38.783 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2023-05-31T21:35:38.783 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "key": 1, 2023-05-31T21:35:38.783 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "val": { 2023-05-31T21:35:38.783 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "state": 1, 2023-05-31T21:35:38.784 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "marker": "", 2023-05-31T21:35:38.784 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "next_step_marker": "", 2023-05-31T21:35:38.784 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "total_entries": 0, 2023-05-31T21:35:38.784 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "pos": 0, 2023-05-31T21:35:38.785 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "timestamp": "0.000000", 2023-05-31T21:35:38.785 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "realm_epoch": 2 2023-05-31T21:35:38.785 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2023-05-31T21:35:38.785 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2023-05-31T21:35:38.786 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2023-05-31T21:35:38.786 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "key": 2, 2023-05-31T21:35:38.786 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "val": { 2023-05-31T21:35:38.786 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "state": 1, 2023-05-31T21:35:38.787 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "marker": "1_1685562970.274572_93.1", 2023-05-31T21:35:38.787 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "next_step_marker": "", 2023-05-31T21:35:38.787 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "total_entries": 0, 2023-05-31T21:35:38.787 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "pos": 0, 2023-05-31T21:35:38.788 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "timestamp": "2023-05-31T19:56:10.274572Z", 2023-05-31T21:35:38.788 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "realm_epoch": 2 2023-05-31T21:35:38.788 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2023-05-31T21:35:38.788 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2023-05-31T21:35:38.789 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2023-05-31T21:35:38.789 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "key": 3, 2023-05-31T21:35:38.789 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "val": { 2023-05-31T21:35:38.790 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "state": 1, 2023-05-31T21:35:38.790 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "marker": "1_1685563798.885419_253.1", 2023-05-31T21:35:38.790 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "next_step_marker": "", 2023-05-31T21:35:38.790 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "total_entries": 0, 2023-05-31T21:35:38.791 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "pos": 0, 2023-05-31T21:35:38.791 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "timestamp": "2023-05-31T20:09:58.885419Z", 2023-05-31T21:35:38.791 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "realm_epoch": 2 2023-05-31T21:35:38.791 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2023-05-31T21:35:38.792 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2023-05-31T21:35:38.792 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: ] 2023-05-31T21:35:38.792 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2023-05-31T21:35:38.792 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "full_sync": { 2023-05-31T21:35:38.793 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "total": 1, 2023-05-31T21:35:38.793 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "complete": 1 2023-05-31T21:35:38.793 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2023-05-31T21:35:38.793 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "current_time": "2023-05-31T20:12:16Z" 2023-05-31T21:35:38.794 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2023-05-31T21:35:38.794 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2023-05-31T21:35:38.794 INFO:tasks.rgw_multisite_tests: "sync_status": { 2023-05-31T21:35:38.795 INFO:tasks.rgw_multisite_tests: "info": { 2023-05-31T21:35:38.795 INFO:tasks.rgw_multisite_tests: "status": "sync", 2023-05-31T21:35:38.795 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2023-05-31T21:35:38.795 INFO:tasks.rgw_multisite_tests: "period": "2e6123d2-d695-4350-9a01-61a455bc782b", 2023-05-31T21:35:38.795 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-05-31T21:35:38.796 INFO:tasks.rgw_multisite_tests: }, 2023-05-31T21:35:38.796 INFO:tasks.rgw_multisite_tests: "markers": [ 2023-05-31T21:35:38.796 INFO:tasks.rgw_multisite_tests: { 2023-05-31T21:35:38.797 INFO:tasks.rgw_multisite_tests: "key": 0, 2023-05-31T21:35:38.797 INFO:tasks.rgw_multisite_tests: "val": { 2023-05-31T21:35:38.797 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-05-31T21:35:38.797 INFO:tasks.rgw_multisite_tests: "marker": "1_1685562631.516089_59.1", 2023-05-31T21:35:38.797 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-05-31T21:35:38.798 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2023-05-31T21:35:38.798 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-05-31T21:35:38.798 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-05-31T19:50:31.516089Z", 2023-05-31T21:35:38.799 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-05-31T21:35:38.799 INFO:tasks.rgw_multisite_tests: } 2023-05-31T21:35:38.799 INFO:tasks.rgw_multisite_tests: }, 2023-05-31T21:35:38.799 INFO:tasks.rgw_multisite_tests: { 2023-05-31T21:35:38.799 INFO:tasks.rgw_multisite_tests: "key": 1, 2023-05-31T21:35:38.800 INFO:tasks.rgw_multisite_tests: "val": { 2023-05-31T21:35:38.800 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-05-31T21:35:38.800 INFO:tasks.rgw_multisite_tests: "marker": "", 2023-05-31T21:35:38.801 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-05-31T21:35:38.801 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-05-31T21:35:38.801 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-05-31T21:35:38.801 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2023-05-31T21:35:38.802 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-05-31T21:35:38.802 INFO:tasks.rgw_multisite_tests: } 2023-05-31T21:35:38.802 INFO:tasks.rgw_multisite_tests: }, 2023-05-31T21:35:38.802 INFO:tasks.rgw_multisite_tests: { 2023-05-31T21:35:38.802 INFO:tasks.rgw_multisite_tests: "key": 2, 2023-05-31T21:35:38.803 INFO:tasks.rgw_multisite_tests: "val": { 2023-05-31T21:35:38.803 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-05-31T21:35:38.803 INFO:tasks.rgw_multisite_tests: "marker": "1_1685562970.274572_93.1", 2023-05-31T21:35:38.803 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-05-31T21:35:38.804 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-05-31T21:35:38.804 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-05-31T21:35:38.804 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-05-31T19:56:10.274572Z", 2023-05-31T21:35:38.804 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-05-31T21:35:38.805 INFO:tasks.rgw_multisite_tests: } 2023-05-31T21:35:38.805 INFO:tasks.rgw_multisite_tests: }, 2023-05-31T21:35:38.805 INFO:tasks.rgw_multisite_tests: { 2023-05-31T21:35:38.805 INFO:tasks.rgw_multisite_tests: "key": 3, 2023-05-31T21:35:38.806 INFO:tasks.rgw_multisite_tests: "val": { 2023-05-31T21:35:38.806 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-05-31T21:35:38.806 INFO:tasks.rgw_multisite_tests: "marker": "1_1685563798.885419_253.1", 2023-05-31T21:35:38.806 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-05-31T21:35:38.806 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-05-31T21:35:38.807 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-05-31T21:35:38.807 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-05-31T20:09:58.885419Z", 2023-05-31T21:35:38.807 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-05-31T21:35:38.807 INFO:tasks.rgw_multisite_tests: } 2023-05-31T21:35:38.808 INFO:tasks.rgw_multisite_tests: } 2023-05-31T21:35:38.808 INFO:tasks.rgw_multisite_tests: ] 2023-05-31T21:35:38.808 INFO:tasks.rgw_multisite_tests: }, 2023-05-31T21:35:38.808 INFO:tasks.rgw_multisite_tests: "full_sync": { 2023-05-31T21:35:38.809 INFO:tasks.rgw_multisite_tests: "total": 1, 2023-05-31T21:35:38.809 INFO:tasks.rgw_multisite_tests: "complete": 1 2023-05-31T21:35:38.809 INFO:tasks.rgw_multisite_tests: }, 2023-05-31T21:35:38.809 INFO:tasks.rgw_multisite_tests: "current_time": "2023-05-31T20:12:16Z" 2023-05-31T21:35:38.810 INFO:tasks.rgw_multisite_tests:} 2023-05-31T21:35:38.810 INFO:tasks.rgw_multisite_tests: 2023-05-31T21:35:38.810 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1685562631.516089_59.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-05-31T19:50:31.516089Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1685562970.274572_93.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-05-31T19:56:10.274572Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1685563798.885419_253.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-05-31T20:09:58.885419Z', 'realm_epoch': 2}}] 2023-05-31T21:35:38.810 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1685562631.516089_59.1', 1: '', 2: '1_1685562970.274572_93.1', 3: '1_1685563802.468126_296.1'} 2023-05-31T21:35:38.811 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1685562631.516089_59.1', 1: '', 2: '1_1685562970.274572_93.1', 3: '1_1685563798.885419_253.1'} 2023-05-31T21:35:38.811 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone a2 behind master: shard=3 master=1_1685563802.468126_296.1 target=1_1685563798.885419_253.1 2023-05-31T21:35:38.811 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-31T21:35:38.811 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-31T21:35:38.812 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-05-31T21:35:38.812 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: ignoring --setuser ceph since I am not root 2023-05-31T21:35:38.812 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-05-31T21:35:38.813 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: 2023-05-31T20:12:21.461+0000 7fe19c197780 20 RGW-SYNC:meta: read sync status 2023-05-31T21:35:38.813 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: 2023-05-31T20:12:21.462+0000 7fe19c197780 20 RGW-SYNC:meta: read sync status 2023-05-31T21:35:38.813 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2023-05-31T21:35:38.813 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "sync_status": { 2023-05-31T21:35:38.813 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "info": { 2023-05-31T21:35:38.814 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "status": "sync", 2023-05-31T21:35:38.814 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "num_shards": 4, 2023-05-31T21:35:38.814 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "period": "2e6123d2-d695-4350-9a01-61a455bc782b", 2023-05-31T21:35:38.814 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "realm_epoch": 2 2023-05-31T21:35:38.815 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2023-05-31T21:35:38.815 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "markers": [ 2023-05-31T21:35:38.815 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2023-05-31T21:35:38.815 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "key": 0, 2023-05-31T21:35:38.816 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "val": { 2023-05-31T21:35:38.816 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "state": 1, 2023-05-31T21:35:38.816 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "marker": "1_1685562631.516089_59.1", 2023-05-31T21:35:38.816 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "next_step_marker": "", 2023-05-31T21:35:38.817 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "total_entries": 1, 2023-05-31T21:35:38.817 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "pos": 0, 2023-05-31T21:35:38.817 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "timestamp": "2023-05-31T19:50:31.516089Z", 2023-05-31T21:35:38.817 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "realm_epoch": 2 2023-05-31T21:35:38.818 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2023-05-31T21:35:38.818 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2023-05-31T21:35:38.818 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2023-05-31T21:35:38.818 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "key": 1, 2023-05-31T21:35:38.818 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "val": { 2023-05-31T21:35:38.819 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "state": 1, 2023-05-31T21:35:38.819 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "marker": "", 2023-05-31T21:35:38.819 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "next_step_marker": "", 2023-05-31T21:35:38.819 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "total_entries": 0, 2023-05-31T21:35:38.820 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "pos": 0, 2023-05-31T21:35:38.820 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "timestamp": "0.000000", 2023-05-31T21:35:38.820 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "realm_epoch": 2 2023-05-31T21:35:38.820 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2023-05-31T21:35:38.821 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2023-05-31T21:35:38.821 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2023-05-31T21:35:38.821 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "key": 2, 2023-05-31T21:35:38.821 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "val": { 2023-05-31T21:35:38.822 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "state": 1, 2023-05-31T21:35:38.822 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "marker": "1_1685562970.274572_93.1", 2023-05-31T21:35:38.822 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "next_step_marker": "", 2023-05-31T21:35:38.822 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "total_entries": 0, 2023-05-31T21:35:38.822 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "pos": 0, 2023-05-31T21:35:38.823 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "timestamp": "2023-05-31T19:56:10.274572Z", 2023-05-31T21:35:38.823 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "realm_epoch": 2 2023-05-31T21:35:38.823 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2023-05-31T21:35:38.823 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2023-05-31T21:35:38.824 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2023-05-31T21:35:38.824 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "key": 3, 2023-05-31T21:35:38.824 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "val": { 2023-05-31T21:35:38.824 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "state": 1, 2023-05-31T21:35:38.825 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "marker": "1_1685563798.815084_250.1", 2023-05-31T21:35:38.825 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "next_step_marker": "", 2023-05-31T21:35:38.825 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "total_entries": 0, 2023-05-31T21:35:38.825 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "pos": 0, 2023-05-31T21:35:38.826 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "timestamp": "2023-05-31T20:09:58.815084Z", 2023-05-31T21:35:38.826 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "realm_epoch": 2 2023-05-31T21:35:38.826 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2023-05-31T21:35:38.826 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2023-05-31T21:35:38.826 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: ] 2023-05-31T21:35:38.827 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2023-05-31T21:35:38.827 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "full_sync": { 2023-05-31T21:35:38.827 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "total": 1, 2023-05-31T21:35:38.827 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "complete": 1 2023-05-31T21:35:38.828 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2023-05-31T21:35:38.828 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "current_time": "2023-05-31T20:12:21Z" 2023-05-31T21:35:38.828 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2023-05-31T21:35:38.828 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2023-05-31T21:35:38.829 INFO:tasks.rgw_multisite_tests: "sync_status": { 2023-05-31T21:35:38.829 INFO:tasks.rgw_multisite_tests: "info": { 2023-05-31T21:35:38.829 INFO:tasks.rgw_multisite_tests: "status": "sync", 2023-05-31T21:35:38.829 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2023-05-31T21:35:38.830 INFO:tasks.rgw_multisite_tests: "period": "2e6123d2-d695-4350-9a01-61a455bc782b", 2023-05-31T21:35:38.830 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-05-31T21:35:38.830 INFO:tasks.rgw_multisite_tests: }, 2023-05-31T21:35:38.830 INFO:tasks.rgw_multisite_tests: "markers": [ 2023-05-31T21:35:38.830 INFO:tasks.rgw_multisite_tests: { 2023-05-31T21:35:38.831 INFO:tasks.rgw_multisite_tests: "key": 0, 2023-05-31T21:35:38.831 INFO:tasks.rgw_multisite_tests: "val": { 2023-05-31T21:35:38.831 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-05-31T21:35:38.831 INFO:tasks.rgw_multisite_tests: "marker": "1_1685562631.516089_59.1", 2023-05-31T21:35:38.832 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-05-31T21:35:38.832 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2023-05-31T21:35:38.832 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-05-31T21:35:38.832 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-05-31T19:50:31.516089Z", 2023-05-31T21:35:38.833 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-05-31T21:35:38.833 INFO:tasks.rgw_multisite_tests: } 2023-05-31T21:35:38.833 INFO:tasks.rgw_multisite_tests: }, 2023-05-31T21:35:38.833 INFO:tasks.rgw_multisite_tests: { 2023-05-31T21:35:38.833 INFO:tasks.rgw_multisite_tests: "key": 1, 2023-05-31T21:35:38.834 INFO:tasks.rgw_multisite_tests: "val": { 2023-05-31T21:35:38.834 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-05-31T21:35:38.834 INFO:tasks.rgw_multisite_tests: "marker": "", 2023-05-31T21:35:38.834 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-05-31T21:35:38.835 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-05-31T21:35:38.835 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-05-31T21:35:38.835 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2023-05-31T21:35:38.835 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-05-31T21:35:38.836 INFO:tasks.rgw_multisite_tests: } 2023-05-31T21:35:38.836 INFO:tasks.rgw_multisite_tests: }, 2023-05-31T21:35:38.836 INFO:tasks.rgw_multisite_tests: { 2023-05-31T21:35:38.836 INFO:tasks.rgw_multisite_tests: "key": 2, 2023-05-31T21:35:38.837 INFO:tasks.rgw_multisite_tests: "val": { 2023-05-31T21:35:38.837 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-05-31T21:35:38.837 INFO:tasks.rgw_multisite_tests: "marker": "1_1685562970.274572_93.1", 2023-05-31T21:35:38.837 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-05-31T21:35:38.837 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-05-31T21:35:38.838 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-05-31T21:35:38.838 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-05-31T19:56:10.274572Z", 2023-05-31T21:35:38.838 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-05-31T21:35:38.838 INFO:tasks.rgw_multisite_tests: } 2023-05-31T21:35:38.839 INFO:tasks.rgw_multisite_tests: }, 2023-05-31T21:35:38.839 INFO:tasks.rgw_multisite_tests: { 2023-05-31T21:35:38.839 INFO:tasks.rgw_multisite_tests: "key": 3, 2023-05-31T21:35:38.839 INFO:tasks.rgw_multisite_tests: "val": { 2023-05-31T21:35:38.840 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-05-31T21:35:38.840 INFO:tasks.rgw_multisite_tests: "marker": "1_1685563798.815084_250.1", 2023-05-31T21:35:38.840 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-05-31T21:35:38.840 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-05-31T21:35:38.841 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-05-31T21:35:38.841 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-05-31T20:09:58.815084Z", 2023-05-31T21:35:38.841 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-05-31T21:35:38.841 INFO:tasks.rgw_multisite_tests: } 2023-05-31T21:35:38.842 INFO:tasks.rgw_multisite_tests: } 2023-05-31T21:35:38.842 INFO:tasks.rgw_multisite_tests: ] 2023-05-31T21:35:38.842 INFO:tasks.rgw_multisite_tests: }, 2023-05-31T21:35:38.842 INFO:tasks.rgw_multisite_tests: "full_sync": { 2023-05-31T21:35:38.842 INFO:tasks.rgw_multisite_tests: "total": 1, 2023-05-31T21:35:38.843 INFO:tasks.rgw_multisite_tests: "complete": 1 2023-05-31T21:35:38.843 INFO:tasks.rgw_multisite_tests: }, 2023-05-31T21:35:38.843 INFO:tasks.rgw_multisite_tests: "current_time": "2023-05-31T20:12:21Z" 2023-05-31T21:35:38.843 INFO:tasks.rgw_multisite_tests:} 2023-05-31T21:35:38.844 INFO:tasks.rgw_multisite_tests: 2023-05-31T21:35:38.844 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1685562631.516089_59.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-05-31T19:50:31.516089Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1685562970.274572_93.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-05-31T19:56:10.274572Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1685563798.815084_250.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-05-31T20:09:58.815084Z', 'realm_epoch': 2}}] 2023-05-31T21:35:38.844 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1685562631.516089_59.1', 1: '', 2: '1_1685562970.274572_93.1', 3: '1_1685563802.468126_296.1'} 2023-05-31T21:35:38.844 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1685562631.516089_59.1', 1: '', 2: '1_1685562970.274572_93.1', 3: '1_1685563798.815084_250.1'} 2023-05-31T21:35:38.845 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone a2 behind master: shard=3 master=1_1685563802.468126_296.1 target=1_1685563798.815084_250.1 2023-05-31T21:35:38.845 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-31T21:35:38.845 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-31T21:35:38.845 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-05-31T21:35:38.846 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: ignoring --setuser ceph since I am not root 2023-05-31T21:35:38.846 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-05-31T21:35:38.846 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: 2023-05-31T20:12:26.594+0000 7fbad1a63780 20 RGW-SYNC:meta: read sync status 2023-05-31T21:35:38.846 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: 2023-05-31T20:12:26.596+0000 7fbad1a63780 20 RGW-SYNC:meta: read sync status 2023-05-31T21:35:38.846 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2023-05-31T21:35:38.847 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "sync_status": { 2023-05-31T21:35:38.847 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "info": { 2023-05-31T21:35:38.847 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "status": "sync", 2023-05-31T21:35:38.847 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "num_shards": 4, 2023-05-31T21:35:38.848 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "period": "2e6123d2-d695-4350-9a01-61a455bc782b", 2023-05-31T21:35:38.848 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "realm_epoch": 2 2023-05-31T21:35:38.848 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2023-05-31T21:35:38.848 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "markers": [ 2023-05-31T21:35:38.849 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2023-05-31T21:35:38.849 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "key": 0, 2023-05-31T21:35:38.849 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "val": { 2023-05-31T21:35:38.849 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "state": 1, 2023-05-31T21:35:38.850 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "marker": "1_1685562631.516089_59.1", 2023-05-31T21:35:38.850 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "next_step_marker": "", 2023-05-31T21:35:38.850 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "total_entries": 1, 2023-05-31T21:35:38.850 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "pos": 0, 2023-05-31T21:35:38.850 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "timestamp": "2023-05-31T19:50:31.516089Z", 2023-05-31T21:35:38.851 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "realm_epoch": 2 2023-05-31T21:35:38.851 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2023-05-31T21:35:38.851 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2023-05-31T21:35:38.851 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2023-05-31T21:35:38.852 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "key": 1, 2023-05-31T21:35:38.852 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "val": { 2023-05-31T21:35:38.852 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "state": 1, 2023-05-31T21:35:38.852 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "marker": "", 2023-05-31T21:35:38.853 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "next_step_marker": "", 2023-05-31T21:35:38.853 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "total_entries": 0, 2023-05-31T21:35:38.853 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "pos": 0, 2023-05-31T21:35:38.853 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "timestamp": "0.000000", 2023-05-31T21:35:38.854 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "realm_epoch": 2 2023-05-31T21:35:38.854 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2023-05-31T21:35:38.854 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2023-05-31T21:35:38.854 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2023-05-31T21:35:38.854 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "key": 2, 2023-05-31T21:35:38.855 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "val": { 2023-05-31T21:35:38.855 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "state": 1, 2023-05-31T21:35:38.855 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "marker": "1_1685562970.274572_93.1", 2023-05-31T21:35:38.855 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "next_step_marker": "", 2023-05-31T21:35:38.856 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "total_entries": 0, 2023-05-31T21:35:38.856 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "pos": 0, 2023-05-31T21:35:38.856 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "timestamp": "2023-05-31T19:56:10.274572Z", 2023-05-31T21:35:38.856 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "realm_epoch": 2 2023-05-31T21:35:38.857 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2023-05-31T21:35:38.857 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2023-05-31T21:35:38.857 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2023-05-31T21:35:38.858 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "key": 3, 2023-05-31T21:35:38.858 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "val": { 2023-05-31T21:35:38.858 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "state": 1, 2023-05-31T21:35:38.858 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "marker": "1_1685563799.098535_267.1", 2023-05-31T21:35:38.858 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "next_step_marker": "", 2023-05-31T21:35:38.859 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "total_entries": 0, 2023-05-31T21:35:38.859 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "pos": 0, 2023-05-31T21:35:38.859 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "timestamp": "2023-05-31T20:09:59.098535Z", 2023-05-31T21:35:38.859 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "realm_epoch": 2 2023-05-31T21:35:38.860 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2023-05-31T21:35:38.860 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2023-05-31T21:35:38.860 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: ] 2023-05-31T21:35:38.860 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2023-05-31T21:35:38.861 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "full_sync": { 2023-05-31T21:35:38.861 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "total": 1, 2023-05-31T21:35:38.861 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "complete": 1 2023-05-31T21:35:38.861 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2023-05-31T21:35:38.862 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "current_time": "2023-05-31T20:12:26Z" 2023-05-31T21:35:38.862 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2023-05-31T21:35:38.862 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2023-05-31T21:35:38.862 INFO:tasks.rgw_multisite_tests: "sync_status": { 2023-05-31T21:35:38.862 INFO:tasks.rgw_multisite_tests: "info": { 2023-05-31T21:35:38.863 INFO:tasks.rgw_multisite_tests: "status": "sync", 2023-05-31T21:35:38.863 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2023-05-31T21:35:38.863 INFO:tasks.rgw_multisite_tests: "period": "2e6123d2-d695-4350-9a01-61a455bc782b", 2023-05-31T21:35:38.863 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-05-31T21:35:38.864 INFO:tasks.rgw_multisite_tests: }, 2023-05-31T21:35:38.864 INFO:tasks.rgw_multisite_tests: "markers": [ 2023-05-31T21:35:38.864 INFO:tasks.rgw_multisite_tests: { 2023-05-31T21:35:38.864 INFO:tasks.rgw_multisite_tests: "key": 0, 2023-05-31T21:35:38.865 INFO:tasks.rgw_multisite_tests: "val": { 2023-05-31T21:35:38.865 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-05-31T21:35:38.865 INFO:tasks.rgw_multisite_tests: "marker": "1_1685562631.516089_59.1", 2023-05-31T21:35:38.865 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-05-31T21:35:38.866 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2023-05-31T21:35:38.866 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-05-31T21:35:38.866 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-05-31T19:50:31.516089Z", 2023-05-31T21:35:38.866 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-05-31T21:35:38.867 INFO:tasks.rgw_multisite_tests: } 2023-05-31T21:35:38.867 INFO:tasks.rgw_multisite_tests: }, 2023-05-31T21:35:38.867 INFO:tasks.rgw_multisite_tests: { 2023-05-31T21:35:38.867 INFO:tasks.rgw_multisite_tests: "key": 1, 2023-05-31T21:35:38.868 INFO:tasks.rgw_multisite_tests: "val": { 2023-05-31T21:35:38.868 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-05-31T21:35:38.868 INFO:tasks.rgw_multisite_tests: "marker": "", 2023-05-31T21:35:38.868 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-05-31T21:35:38.868 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-05-31T21:35:38.869 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-05-31T21:35:38.869 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2023-05-31T21:35:38.869 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-05-31T21:35:38.870 INFO:tasks.rgw_multisite_tests: } 2023-05-31T21:35:38.870 INFO:tasks.rgw_multisite_tests: }, 2023-05-31T21:35:38.870 INFO:tasks.rgw_multisite_tests: { 2023-05-31T21:35:38.870 INFO:tasks.rgw_multisite_tests: "key": 2, 2023-05-31T21:35:38.870 INFO:tasks.rgw_multisite_tests: "val": { 2023-05-31T21:35:38.871 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-05-31T21:35:38.871 INFO:tasks.rgw_multisite_tests: "marker": "1_1685562970.274572_93.1", 2023-05-31T21:35:38.871 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-05-31T21:35:38.871 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-05-31T21:35:38.872 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-05-31T21:35:38.872 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-05-31T19:56:10.274572Z", 2023-05-31T21:35:38.872 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-05-31T21:35:38.872 INFO:tasks.rgw_multisite_tests: } 2023-05-31T21:35:38.873 INFO:tasks.rgw_multisite_tests: }, 2023-05-31T21:35:38.873 INFO:tasks.rgw_multisite_tests: { 2023-05-31T21:35:38.873 INFO:tasks.rgw_multisite_tests: "key": 3, 2023-05-31T21:35:38.873 INFO:tasks.rgw_multisite_tests: "val": { 2023-05-31T21:35:38.874 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-05-31T21:35:38.874 INFO:tasks.rgw_multisite_tests: "marker": "1_1685563799.098535_267.1", 2023-05-31T21:35:38.874 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-05-31T21:35:38.874 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-05-31T21:35:38.874 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-05-31T21:35:38.875 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-05-31T20:09:59.098535Z", 2023-05-31T21:35:38.875 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-05-31T21:35:38.875 INFO:tasks.rgw_multisite_tests: } 2023-05-31T21:35:38.875 INFO:tasks.rgw_multisite_tests: } 2023-05-31T21:35:38.876 INFO:tasks.rgw_multisite_tests: ] 2023-05-31T21:35:38.876 INFO:tasks.rgw_multisite_tests: }, 2023-05-31T21:35:38.876 INFO:tasks.rgw_multisite_tests: "full_sync": { 2023-05-31T21:35:38.876 INFO:tasks.rgw_multisite_tests: "total": 1, 2023-05-31T21:35:38.876 INFO:tasks.rgw_multisite_tests: "complete": 1 2023-05-31T21:35:38.877 INFO:tasks.rgw_multisite_tests: }, 2023-05-31T21:35:38.877 INFO:tasks.rgw_multisite_tests: "current_time": "2023-05-31T20:12:26Z" 2023-05-31T21:35:38.877 INFO:tasks.rgw_multisite_tests:} 2023-05-31T21:35:38.878 INFO:tasks.rgw_multisite_tests: 2023-05-31T21:35:38.878 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1685562631.516089_59.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-05-31T19:50:31.516089Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1685562970.274572_93.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-05-31T19:56:10.274572Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1685563799.098535_267.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-05-31T20:09:59.098535Z', 'realm_epoch': 2}}] 2023-05-31T21:35:38.878 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1685562631.516089_59.1', 1: '', 2: '1_1685562970.274572_93.1', 3: '1_1685563802.468126_296.1'} 2023-05-31T21:35:38.878 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1685562631.516089_59.1', 1: '', 2: '1_1685562970.274572_93.1', 3: '1_1685563799.098535_267.1'} 2023-05-31T21:35:38.879 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone a2 behind master: shard=3 master=1_1685563802.468126_296.1 target=1_1685563799.098535_267.1 2023-05-31T21:35:38.879 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-31T21:35:38.879 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-31T21:35:38.879 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-05-31T21:35:38.879 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: ignoring --setuser ceph since I am not root 2023-05-31T21:35:38.880 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-05-31T21:35:38.880 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: 2023-05-31T20:12:31.713+0000 7f1795161780 20 RGW-SYNC:meta: read sync status 2023-05-31T21:35:38.880 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: 2023-05-31T20:12:31.714+0000 7f1795161780 20 RGW-SYNC:meta: read sync status 2023-05-31T21:35:38.880 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2023-05-31T21:35:38.881 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "sync_status": { 2023-05-31T21:35:38.881 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "info": { 2023-05-31T21:35:38.882 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "status": "sync", 2023-05-31T21:35:38.882 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "num_shards": 4, 2023-05-31T21:35:38.882 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "period": "2e6123d2-d695-4350-9a01-61a455bc782b", 2023-05-31T21:35:38.883 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "realm_epoch": 2 2023-05-31T21:35:38.883 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2023-05-31T21:35:38.884 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "markers": [ 2023-05-31T21:35:38.884 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2023-05-31T21:35:38.885 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "key": 0, 2023-05-31T21:35:38.885 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "val": { 2023-05-31T21:35:38.886 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "state": 1, 2023-05-31T21:35:38.886 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "marker": "1_1685562631.516089_59.1", 2023-05-31T21:35:38.887 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "next_step_marker": "", 2023-05-31T21:35:38.887 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "total_entries": 1, 2023-05-31T21:35:38.887 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "pos": 0, 2023-05-31T21:35:38.888 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "timestamp": "2023-05-31T19:50:31.516089Z", 2023-05-31T21:35:38.888 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "realm_epoch": 2 2023-05-31T21:35:38.888 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2023-05-31T21:35:38.889 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2023-05-31T21:35:38.889 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2023-05-31T21:35:38.889 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "key": 1, 2023-05-31T21:35:38.889 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "val": { 2023-05-31T21:35:38.890 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "state": 1, 2023-05-31T21:35:38.890 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "marker": "", 2023-05-31T21:35:38.890 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "next_step_marker": "", 2023-05-31T21:35:38.890 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "total_entries": 0, 2023-05-31T21:35:38.890 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "pos": 0, 2023-05-31T21:35:38.891 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "timestamp": "0.000000", 2023-05-31T21:35:38.891 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "realm_epoch": 2 2023-05-31T21:35:38.891 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2023-05-31T21:35:38.891 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2023-05-31T21:35:38.892 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2023-05-31T21:35:38.892 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "key": 2, 2023-05-31T21:35:38.892 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "val": { 2023-05-31T21:35:38.892 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "state": 1, 2023-05-31T21:35:38.893 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "marker": "1_1685562970.274572_93.1", 2023-05-31T21:35:38.893 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "next_step_marker": "", 2023-05-31T21:35:38.893 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "total_entries": 0, 2023-05-31T21:35:38.893 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "pos": 0, 2023-05-31T21:35:38.894 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "timestamp": "2023-05-31T19:56:10.274572Z", 2023-05-31T21:35:38.894 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "realm_epoch": 2 2023-05-31T21:35:38.894 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2023-05-31T21:35:38.894 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2023-05-31T21:35:38.894 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2023-05-31T21:35:38.895 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "key": 3, 2023-05-31T21:35:38.895 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "val": { 2023-05-31T21:35:38.895 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "state": 1, 2023-05-31T21:35:38.895 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "marker": "1_1685563798.970303_257.1", 2023-05-31T21:35:38.896 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "next_step_marker": "", 2023-05-31T21:35:38.896 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "total_entries": 0, 2023-05-31T21:35:38.896 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "pos": 0, 2023-05-31T21:35:38.896 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "timestamp": "2023-05-31T20:09:58.970303Z", 2023-05-31T21:35:38.897 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "realm_epoch": 2 2023-05-31T21:35:38.897 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2023-05-31T21:35:38.897 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2023-05-31T21:35:38.897 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: ] 2023-05-31T21:35:38.897 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2023-05-31T21:35:38.898 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "full_sync": { 2023-05-31T21:35:38.898 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "total": 1, 2023-05-31T21:35:38.898 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "complete": 1 2023-05-31T21:35:38.898 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2023-05-31T21:35:38.899 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "current_time": "2023-05-31T20:12:31Z" 2023-05-31T21:35:38.899 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2023-05-31T21:35:38.899 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2023-05-31T21:35:38.899 INFO:tasks.rgw_multisite_tests: "sync_status": { 2023-05-31T21:35:38.900 INFO:tasks.rgw_multisite_tests: "info": { 2023-05-31T21:35:38.900 INFO:tasks.rgw_multisite_tests: "status": "sync", 2023-05-31T21:35:38.900 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2023-05-31T21:35:38.900 INFO:tasks.rgw_multisite_tests: "period": "2e6123d2-d695-4350-9a01-61a455bc782b", 2023-05-31T21:35:38.901 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-05-31T21:35:38.901 INFO:tasks.rgw_multisite_tests: }, 2023-05-31T21:35:38.901 INFO:tasks.rgw_multisite_tests: "markers": [ 2023-05-31T21:35:38.901 INFO:tasks.rgw_multisite_tests: { 2023-05-31T21:35:38.902 INFO:tasks.rgw_multisite_tests: "key": 0, 2023-05-31T21:35:38.902 INFO:tasks.rgw_multisite_tests: "val": { 2023-05-31T21:35:38.902 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-05-31T21:35:38.902 INFO:tasks.rgw_multisite_tests: "marker": "1_1685562631.516089_59.1", 2023-05-31T21:35:38.903 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-05-31T21:35:38.903 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2023-05-31T21:35:38.903 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-05-31T21:35:38.903 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-05-31T19:50:31.516089Z", 2023-05-31T21:35:38.903 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-05-31T21:35:38.904 INFO:tasks.rgw_multisite_tests: } 2023-05-31T21:35:38.904 INFO:tasks.rgw_multisite_tests: }, 2023-05-31T21:35:38.904 INFO:tasks.rgw_multisite_tests: { 2023-05-31T21:35:38.904 INFO:tasks.rgw_multisite_tests: "key": 1, 2023-05-31T21:35:38.905 INFO:tasks.rgw_multisite_tests: "val": { 2023-05-31T21:35:38.905 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-05-31T21:35:38.905 INFO:tasks.rgw_multisite_tests: "marker": "", 2023-05-31T21:35:38.905 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-05-31T21:35:38.906 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-05-31T21:35:38.906 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-05-31T21:35:38.906 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2023-05-31T21:35:38.906 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-05-31T21:35:38.906 INFO:tasks.rgw_multisite_tests: } 2023-05-31T21:35:38.907 INFO:tasks.rgw_multisite_tests: }, 2023-05-31T21:35:38.907 INFO:tasks.rgw_multisite_tests: { 2023-05-31T21:35:38.907 INFO:tasks.rgw_multisite_tests: "key": 2, 2023-05-31T21:35:38.907 INFO:tasks.rgw_multisite_tests: "val": { 2023-05-31T21:35:38.908 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-05-31T21:35:38.908 INFO:tasks.rgw_multisite_tests: "marker": "1_1685562970.274572_93.1", 2023-05-31T21:35:38.908 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-05-31T21:35:38.908 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-05-31T21:35:38.909 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-05-31T21:35:38.909 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-05-31T19:56:10.274572Z", 2023-05-31T21:35:38.909 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-05-31T21:35:38.909 INFO:tasks.rgw_multisite_tests: } 2023-05-31T21:35:38.910 INFO:tasks.rgw_multisite_tests: }, 2023-05-31T21:35:38.910 INFO:tasks.rgw_multisite_tests: { 2023-05-31T21:35:38.910 INFO:tasks.rgw_multisite_tests: "key": 3, 2023-05-31T21:35:38.910 INFO:tasks.rgw_multisite_tests: "val": { 2023-05-31T21:35:38.911 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-05-31T21:35:38.911 INFO:tasks.rgw_multisite_tests: "marker": "1_1685563798.970303_257.1", 2023-05-31T21:35:38.911 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-05-31T21:35:38.911 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-05-31T21:35:38.912 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-05-31T21:35:38.912 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-05-31T20:09:58.970303Z", 2023-05-31T21:35:38.912 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-05-31T21:35:38.912 INFO:tasks.rgw_multisite_tests: } 2023-05-31T21:35:38.913 INFO:tasks.rgw_multisite_tests: } 2023-05-31T21:35:38.913 INFO:tasks.rgw_multisite_tests: ] 2023-05-31T21:35:38.913 INFO:tasks.rgw_multisite_tests: }, 2023-05-31T21:35:38.913 INFO:tasks.rgw_multisite_tests: "full_sync": { 2023-05-31T21:35:38.913 INFO:tasks.rgw_multisite_tests: "total": 1, 2023-05-31T21:35:38.914 INFO:tasks.rgw_multisite_tests: "complete": 1 2023-05-31T21:35:38.914 INFO:tasks.rgw_multisite_tests: }, 2023-05-31T21:35:38.914 INFO:tasks.rgw_multisite_tests: "current_time": "2023-05-31T20:12:31Z" 2023-05-31T21:35:38.914 INFO:tasks.rgw_multisite_tests:} 2023-05-31T21:35:38.915 INFO:tasks.rgw_multisite_tests: 2023-05-31T21:35:38.915 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1685562631.516089_59.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-05-31T19:50:31.516089Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1685562970.274572_93.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-05-31T19:56:10.274572Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1685563798.970303_257.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-05-31T20:09:58.970303Z', 'realm_epoch': 2}}] 2023-05-31T21:35:38.915 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1685562631.516089_59.1', 1: '', 2: '1_1685562970.274572_93.1', 3: '1_1685563802.468126_296.1'} 2023-05-31T21:35:38.915 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1685562631.516089_59.1', 1: '', 2: '1_1685562970.274572_93.1', 3: '1_1685563798.970303_257.1'} 2023-05-31T21:35:38.916 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone a2 behind master: shard=3 master=1_1685563802.468126_296.1 target=1_1685563798.970303_257.1 2023-05-31T21:35:38.916 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-31T21:35:38.916 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-31T21:35:38.916 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-05-31T21:35:38.917 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: ignoring --setuser ceph since I am not root 2023-05-31T21:35:38.917 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-05-31T21:35:38.917 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: 2023-05-31T20:12:36.858+0000 7f8a1ea16780 20 RGW-SYNC:meta: read sync status 2023-05-31T21:35:38.917 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: 2023-05-31T20:12:36.859+0000 7f8a1ea16780 20 RGW-SYNC:meta: read sync status 2023-05-31T21:35:38.918 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2023-05-31T21:35:38.918 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "sync_status": { 2023-05-31T21:35:38.918 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "info": { 2023-05-31T21:35:38.918 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "status": "sync", 2023-05-31T21:35:38.918 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "num_shards": 4, 2023-05-31T21:35:38.919 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "period": "2e6123d2-d695-4350-9a01-61a455bc782b", 2023-05-31T21:35:38.919 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "realm_epoch": 2 2023-05-31T21:35:38.919 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2023-05-31T21:35:38.919 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "markers": [ 2023-05-31T21:35:38.920 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2023-05-31T21:35:38.920 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "key": 0, 2023-05-31T21:35:38.920 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "val": { 2023-05-31T21:35:38.920 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "state": 1, 2023-05-31T21:35:38.921 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "marker": "1_1685562631.516089_59.1", 2023-05-31T21:35:38.921 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "next_step_marker": "", 2023-05-31T21:35:38.921 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "total_entries": 1, 2023-05-31T21:35:38.921 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "pos": 0, 2023-05-31T21:35:38.921 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "timestamp": "2023-05-31T19:50:31.516089Z", 2023-05-31T21:35:38.922 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "realm_epoch": 2 2023-05-31T21:35:38.922 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2023-05-31T21:35:38.922 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2023-05-31T21:35:38.922 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2023-05-31T21:35:38.923 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "key": 1, 2023-05-31T21:35:38.923 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "val": { 2023-05-31T21:35:38.923 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "state": 1, 2023-05-31T21:35:38.923 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "marker": "", 2023-05-31T21:35:38.924 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "next_step_marker": "", 2023-05-31T21:35:38.924 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "total_entries": 0, 2023-05-31T21:35:38.924 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "pos": 0, 2023-05-31T21:35:38.924 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "timestamp": "0.000000", 2023-05-31T21:35:38.924 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "realm_epoch": 2 2023-05-31T21:35:38.925 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2023-05-31T21:35:38.925 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2023-05-31T21:35:38.925 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2023-05-31T21:35:38.925 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "key": 2, 2023-05-31T21:35:38.926 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "val": { 2023-05-31T21:35:38.926 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "state": 1, 2023-05-31T21:35:38.926 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "marker": "1_1685562970.274572_93.1", 2023-05-31T21:35:38.926 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "next_step_marker": "", 2023-05-31T21:35:38.926 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "total_entries": 0, 2023-05-31T21:35:38.927 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "pos": 0, 2023-05-31T21:35:38.927 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "timestamp": "2023-05-31T19:56:10.274572Z", 2023-05-31T21:35:38.927 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "realm_epoch": 2 2023-05-31T21:35:38.927 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2023-05-31T21:35:38.928 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2023-05-31T21:35:38.928 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2023-05-31T21:35:38.928 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "key": 3, 2023-05-31T21:35:38.928 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "val": { 2023-05-31T21:35:38.929 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "state": 1, 2023-05-31T21:35:38.929 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "marker": "1_1685563798.885419_253.1", 2023-05-31T21:35:38.929 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "next_step_marker": "", 2023-05-31T21:35:38.929 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "total_entries": 0, 2023-05-31T21:35:38.930 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "pos": 0, 2023-05-31T21:35:38.930 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "timestamp": "2023-05-31T20:09:58.885419Z", 2023-05-31T21:35:38.930 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "realm_epoch": 2 2023-05-31T21:35:38.930 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2023-05-31T21:35:38.930 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2023-05-31T21:35:38.931 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: ] 2023-05-31T21:35:38.931 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2023-05-31T21:35:38.931 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "full_sync": { 2023-05-31T21:35:38.931 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "total": 1, 2023-05-31T21:35:38.932 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "complete": 1 2023-05-31T21:35:38.932 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2023-05-31T21:35:38.932 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "current_time": "2023-05-31T20:12:36Z" 2023-05-31T21:35:38.932 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2023-05-31T21:35:38.933 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2023-05-31T21:35:38.933 INFO:tasks.rgw_multisite_tests: "sync_status": { 2023-05-31T21:35:38.933 INFO:tasks.rgw_multisite_tests: "info": { 2023-05-31T21:35:38.933 INFO:tasks.rgw_multisite_tests: "status": "sync", 2023-05-31T21:35:38.933 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2023-05-31T21:35:38.934 INFO:tasks.rgw_multisite_tests: "period": "2e6123d2-d695-4350-9a01-61a455bc782b", 2023-05-31T21:35:38.934 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-05-31T21:35:38.934 INFO:tasks.rgw_multisite_tests: }, 2023-05-31T21:35:38.934 INFO:tasks.rgw_multisite_tests: "markers": [ 2023-05-31T21:35:38.935 INFO:tasks.rgw_multisite_tests: { 2023-05-31T21:35:38.935 INFO:tasks.rgw_multisite_tests: "key": 0, 2023-05-31T21:35:38.935 INFO:tasks.rgw_multisite_tests: "val": { 2023-05-31T21:35:38.935 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-05-31T21:35:38.936 INFO:tasks.rgw_multisite_tests: "marker": "1_1685562631.516089_59.1", 2023-05-31T21:35:38.936 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-05-31T21:35:38.936 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2023-05-31T21:35:38.936 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-05-31T21:35:38.936 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-05-31T19:50:31.516089Z", 2023-05-31T21:35:38.937 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-05-31T21:35:38.937 INFO:tasks.rgw_multisite_tests: } 2023-05-31T21:35:38.937 INFO:tasks.rgw_multisite_tests: }, 2023-05-31T21:35:38.937 INFO:tasks.rgw_multisite_tests: { 2023-05-31T21:35:38.938 INFO:tasks.rgw_multisite_tests: "key": 1, 2023-05-31T21:35:38.938 INFO:tasks.rgw_multisite_tests: "val": { 2023-05-31T21:35:38.938 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-05-31T21:35:38.938 INFO:tasks.rgw_multisite_tests: "marker": "", 2023-05-31T21:35:38.939 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-05-31T21:35:38.939 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-05-31T21:35:38.939 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-05-31T21:35:38.939 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2023-05-31T21:35:38.939 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-05-31T21:35:38.940 INFO:tasks.rgw_multisite_tests: } 2023-05-31T21:35:38.940 INFO:tasks.rgw_multisite_tests: }, 2023-05-31T21:35:38.940 INFO:tasks.rgw_multisite_tests: { 2023-05-31T21:35:38.940 INFO:tasks.rgw_multisite_tests: "key": 2, 2023-05-31T21:35:38.941 INFO:tasks.rgw_multisite_tests: "val": { 2023-05-31T21:35:38.941 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-05-31T21:35:38.941 INFO:tasks.rgw_multisite_tests: "marker": "1_1685562970.274572_93.1", 2023-05-31T21:35:38.941 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-05-31T21:35:38.942 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-05-31T21:35:38.942 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-05-31T21:35:38.942 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-05-31T19:56:10.274572Z", 2023-05-31T21:35:38.942 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-05-31T21:35:38.942 INFO:tasks.rgw_multisite_tests: } 2023-05-31T21:35:38.943 INFO:tasks.rgw_multisite_tests: }, 2023-05-31T21:35:38.943 INFO:tasks.rgw_multisite_tests: { 2023-05-31T21:35:38.943 INFO:tasks.rgw_multisite_tests: "key": 3, 2023-05-31T21:35:38.943 INFO:tasks.rgw_multisite_tests: "val": { 2023-05-31T21:35:38.944 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-05-31T21:35:38.944 INFO:tasks.rgw_multisite_tests: "marker": "1_1685563798.885419_253.1", 2023-05-31T21:35:38.944 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-05-31T21:35:38.944 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-05-31T21:35:38.944 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-05-31T21:35:38.945 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-05-31T20:09:58.885419Z", 2023-05-31T21:35:38.945 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-05-31T21:35:38.945 INFO:tasks.rgw_multisite_tests: } 2023-05-31T21:35:38.945 INFO:tasks.rgw_multisite_tests: } 2023-05-31T21:35:38.946 INFO:tasks.rgw_multisite_tests: ] 2023-05-31T21:35:38.946 INFO:tasks.rgw_multisite_tests: }, 2023-05-31T21:35:38.946 INFO:tasks.rgw_multisite_tests: "full_sync": { 2023-05-31T21:35:38.946 INFO:tasks.rgw_multisite_tests: "total": 1, 2023-05-31T21:35:38.947 INFO:tasks.rgw_multisite_tests: "complete": 1 2023-05-31T21:35:38.947 INFO:tasks.rgw_multisite_tests: }, 2023-05-31T21:35:38.947 INFO:tasks.rgw_multisite_tests: "current_time": "2023-05-31T20:12:36Z" 2023-05-31T21:35:38.947 INFO:tasks.rgw_multisite_tests:} 2023-05-31T21:35:38.947 INFO:tasks.rgw_multisite_tests: 2023-05-31T21:35:38.948 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1685562631.516089_59.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-05-31T19:50:31.516089Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1685562970.274572_93.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-05-31T19:56:10.274572Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1685563798.885419_253.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-05-31T20:09:58.885419Z', 'realm_epoch': 2}}] 2023-05-31T21:35:38.948 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1685562631.516089_59.1', 1: '', 2: '1_1685562970.274572_93.1', 3: '1_1685563802.468126_296.1'} 2023-05-31T21:35:38.948 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1685562631.516089_59.1', 1: '', 2: '1_1685562970.274572_93.1', 3: '1_1685563798.885419_253.1'} 2023-05-31T21:35:38.948 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone a2 behind master: shard=3 master=1_1685563802.468126_296.1 target=1_1685563798.885419_253.1 2023-05-31T21:35:38.949 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-31T21:35:38.949 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-31T21:35:38.949 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-05-31T21:35:38.949 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: ignoring --setuser ceph since I am not root 2023-05-31T21:35:38.950 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-05-31T21:35:38.950 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: 2023-05-31T20:12:41.996+0000 7f0995824780 20 RGW-SYNC:meta: read sync status 2023-05-31T21:35:38.950 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: 2023-05-31T20:12:41.998+0000 7f0995824780 20 RGW-SYNC:meta: read sync status 2023-05-31T21:35:38.950 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2023-05-31T21:35:38.951 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "sync_status": { 2023-05-31T21:35:38.951 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "info": { 2023-05-31T21:35:38.951 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "status": "sync", 2023-05-31T21:35:38.951 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "num_shards": 4, 2023-05-31T21:35:38.951 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "period": "2e6123d2-d695-4350-9a01-61a455bc782b", 2023-05-31T21:35:38.952 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "realm_epoch": 2 2023-05-31T21:35:38.952 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2023-05-31T21:35:38.952 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "markers": [ 2023-05-31T21:35:38.952 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2023-05-31T21:35:38.953 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "key": 0, 2023-05-31T21:35:38.953 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "val": { 2023-05-31T21:35:38.953 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "state": 1, 2023-05-31T21:35:38.953 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "marker": "1_1685562631.516089_59.1", 2023-05-31T21:35:38.954 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "next_step_marker": "", 2023-05-31T21:35:38.954 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "total_entries": 1, 2023-05-31T21:35:38.954 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "pos": 0, 2023-05-31T21:35:38.954 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "timestamp": "2023-05-31T19:50:31.516089Z", 2023-05-31T21:35:38.954 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "realm_epoch": 2 2023-05-31T21:35:38.955 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2023-05-31T21:35:38.955 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2023-05-31T21:35:38.956 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2023-05-31T21:35:38.956 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "key": 1, 2023-05-31T21:35:38.957 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "val": { 2023-05-31T21:35:38.957 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "state": 1, 2023-05-31T21:35:38.957 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "marker": "", 2023-05-31T21:35:38.958 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "next_step_marker": "", 2023-05-31T21:35:38.958 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "total_entries": 0, 2023-05-31T21:35:38.958 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "pos": 0, 2023-05-31T21:35:38.959 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "timestamp": "0.000000", 2023-05-31T21:35:38.959 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "realm_epoch": 2 2023-05-31T21:35:38.959 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2023-05-31T21:35:38.959 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2023-05-31T21:35:38.960 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2023-05-31T21:35:38.960 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "key": 2, 2023-05-31T21:35:38.960 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "val": { 2023-05-31T21:35:38.960 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "state": 1, 2023-05-31T21:35:38.961 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "marker": "1_1685562970.274572_93.1", 2023-05-31T21:35:38.961 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "next_step_marker": "", 2023-05-31T21:35:38.961 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "total_entries": 0, 2023-05-31T21:35:38.961 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "pos": 0, 2023-05-31T21:35:38.962 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "timestamp": "2023-05-31T19:56:10.274572Z", 2023-05-31T21:35:38.962 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "realm_epoch": 2 2023-05-31T21:35:38.962 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2023-05-31T21:35:38.962 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2023-05-31T21:35:38.963 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2023-05-31T21:35:38.963 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "key": 3, 2023-05-31T21:35:38.963 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "val": { 2023-05-31T21:35:38.963 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "state": 1, 2023-05-31T21:35:38.963 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "marker": "1_1685563798.807987_249.1", 2023-05-31T21:35:38.964 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "next_step_marker": "", 2023-05-31T21:35:38.964 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "total_entries": 0, 2023-05-31T21:35:38.964 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "pos": 0, 2023-05-31T21:35:38.964 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "timestamp": "2023-05-31T20:09:58.807987Z", 2023-05-31T21:35:38.965 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "realm_epoch": 2 2023-05-31T21:35:38.965 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2023-05-31T21:35:38.965 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2023-05-31T21:35:38.965 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: ] 2023-05-31T21:35:38.966 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2023-05-31T21:35:38.966 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "full_sync": { 2023-05-31T21:35:38.966 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "total": 1, 2023-05-31T21:35:38.966 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "complete": 1 2023-05-31T21:35:38.967 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2023-05-31T21:35:38.967 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "current_time": "2023-05-31T20:12:41Z" 2023-05-31T21:35:38.967 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2023-05-31T21:35:38.967 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2023-05-31T21:35:38.968 INFO:tasks.rgw_multisite_tests: "sync_status": { 2023-05-31T21:35:38.968 INFO:tasks.rgw_multisite_tests: "info": { 2023-05-31T21:35:38.969 INFO:tasks.rgw_multisite_tests: "status": "sync", 2023-05-31T21:35:38.969 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2023-05-31T21:35:38.970 INFO:tasks.rgw_multisite_tests: "period": "2e6123d2-d695-4350-9a01-61a455bc782b", 2023-05-31T21:35:38.970 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-05-31T21:35:38.971 INFO:tasks.rgw_multisite_tests: }, 2023-05-31T21:35:38.971 INFO:tasks.rgw_multisite_tests: "markers": [ 2023-05-31T21:35:38.971 INFO:tasks.rgw_multisite_tests: { 2023-05-31T21:35:38.971 INFO:tasks.rgw_multisite_tests: "key": 0, 2023-05-31T21:35:38.971 INFO:tasks.rgw_multisite_tests: "val": { 2023-05-31T21:35:38.972 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-05-31T21:35:38.972 INFO:tasks.rgw_multisite_tests: "marker": "1_1685562631.516089_59.1", 2023-05-31T21:35:38.972 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-05-31T21:35:38.972 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2023-05-31T21:35:38.973 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-05-31T21:35:38.973 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-05-31T19:50:31.516089Z", 2023-05-31T21:35:38.973 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-05-31T21:35:38.973 INFO:tasks.rgw_multisite_tests: } 2023-05-31T21:35:38.974 INFO:tasks.rgw_multisite_tests: }, 2023-05-31T21:35:38.974 INFO:tasks.rgw_multisite_tests: { 2023-05-31T21:35:38.974 INFO:tasks.rgw_multisite_tests: "key": 1, 2023-05-31T21:35:38.974 INFO:tasks.rgw_multisite_tests: "val": { 2023-05-31T21:35:38.975 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-05-31T21:35:38.975 INFO:tasks.rgw_multisite_tests: "marker": "", 2023-05-31T21:35:38.975 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-05-31T21:35:38.975 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-05-31T21:35:38.975 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-05-31T21:35:38.976 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2023-05-31T21:35:38.976 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-05-31T21:35:38.976 INFO:tasks.rgw_multisite_tests: } 2023-05-31T21:35:38.976 INFO:tasks.rgw_multisite_tests: }, 2023-05-31T21:35:38.977 INFO:tasks.rgw_multisite_tests: { 2023-05-31T21:35:38.977 INFO:tasks.rgw_multisite_tests: "key": 2, 2023-05-31T21:35:38.977 INFO:tasks.rgw_multisite_tests: "val": { 2023-05-31T21:35:38.977 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-05-31T21:35:38.978 INFO:tasks.rgw_multisite_tests: "marker": "1_1685562970.274572_93.1", 2023-05-31T21:35:38.978 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-05-31T21:35:38.978 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-05-31T21:35:38.978 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-05-31T21:35:38.979 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-05-31T19:56:10.274572Z", 2023-05-31T21:35:38.979 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-05-31T21:35:38.979 INFO:tasks.rgw_multisite_tests: } 2023-05-31T21:35:38.979 INFO:tasks.rgw_multisite_tests: }, 2023-05-31T21:35:38.980 INFO:tasks.rgw_multisite_tests: { 2023-05-31T21:35:38.980 INFO:tasks.rgw_multisite_tests: "key": 3, 2023-05-31T21:35:38.980 INFO:tasks.rgw_multisite_tests: "val": { 2023-05-31T21:35:38.980 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-05-31T21:35:38.981 INFO:tasks.rgw_multisite_tests: "marker": "1_1685563798.807987_249.1", 2023-05-31T21:35:38.981 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-05-31T21:35:38.981 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-05-31T21:35:38.981 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-05-31T21:35:38.981 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-05-31T20:09:58.807987Z", 2023-05-31T21:35:38.982 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-05-31T21:35:38.982 INFO:tasks.rgw_multisite_tests: } 2023-05-31T21:35:38.982 INFO:tasks.rgw_multisite_tests: } 2023-05-31T21:35:38.982 INFO:tasks.rgw_multisite_tests: ] 2023-05-31T21:35:38.983 INFO:tasks.rgw_multisite_tests: }, 2023-05-31T21:35:38.983 INFO:tasks.rgw_multisite_tests: "full_sync": { 2023-05-31T21:35:38.983 INFO:tasks.rgw_multisite_tests: "total": 1, 2023-05-31T21:35:38.983 INFO:tasks.rgw_multisite_tests: "complete": 1 2023-05-31T21:35:38.983 INFO:tasks.rgw_multisite_tests: }, 2023-05-31T21:35:38.984 INFO:tasks.rgw_multisite_tests: "current_time": "2023-05-31T20:12:41Z" 2023-05-31T21:35:38.984 INFO:tasks.rgw_multisite_tests:} 2023-05-31T21:35:38.984 INFO:tasks.rgw_multisite_tests: 2023-05-31T21:35:38.984 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1685562631.516089_59.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-05-31T19:50:31.516089Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1685562970.274572_93.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-05-31T19:56:10.274572Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1685563798.807987_249.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-05-31T20:09:58.807987Z', 'realm_epoch': 2}}] 2023-05-31T21:35:38.985 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1685562631.516089_59.1', 1: '', 2: '1_1685562970.274572_93.1', 3: '1_1685563802.468126_296.1'} 2023-05-31T21:35:38.985 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1685562631.516089_59.1', 1: '', 2: '1_1685562970.274572_93.1', 3: '1_1685563798.807987_249.1'} 2023-05-31T21:35:38.985 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone a2 behind master: shard=3 master=1_1685563802.468126_296.1 target=1_1685563798.807987_249.1 2023-05-31T21:35:38.985 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-31T21:35:38.986 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-31T21:35:38.986 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-05-31T21:35:38.986 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: ignoring --setuser ceph since I am not root 2023-05-31T21:35:38.986 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-05-31T21:35:38.987 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: 2023-05-31T20:12:47.127+0000 7f20dd552780 20 RGW-SYNC:meta: read sync status 2023-05-31T21:35:38.987 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: 2023-05-31T20:12:47.128+0000 7f20dd552780 20 RGW-SYNC:meta: read sync status 2023-05-31T21:35:38.987 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2023-05-31T21:35:38.987 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "sync_status": { 2023-05-31T21:35:38.987 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "info": { 2023-05-31T21:35:38.988 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "status": "sync", 2023-05-31T21:35:38.988 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "num_shards": 4, 2023-05-31T21:35:38.988 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "period": "2e6123d2-d695-4350-9a01-61a455bc782b", 2023-05-31T21:35:38.988 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "realm_epoch": 2 2023-05-31T21:35:38.989 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2023-05-31T21:35:38.989 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "markers": [ 2023-05-31T21:35:38.989 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2023-05-31T21:35:38.989 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "key": 0, 2023-05-31T21:35:38.990 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "val": { 2023-05-31T21:35:38.990 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "state": 1, 2023-05-31T21:35:38.990 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "marker": "1_1685562631.516089_59.1", 2023-05-31T21:35:38.990 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "next_step_marker": "", 2023-05-31T21:35:38.991 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "total_entries": 1, 2023-05-31T21:35:38.991 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "pos": 0, 2023-05-31T21:35:38.991 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "timestamp": "2023-05-31T19:50:31.516089Z", 2023-05-31T21:35:38.991 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "realm_epoch": 2 2023-05-31T21:35:38.991 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2023-05-31T21:35:38.992 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2023-05-31T21:35:38.992 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2023-05-31T21:35:38.992 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "key": 1, 2023-05-31T21:35:38.992 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "val": { 2023-05-31T21:35:38.993 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "state": 1, 2023-05-31T21:35:38.993 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "marker": "", 2023-05-31T21:35:38.993 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "next_step_marker": "", 2023-05-31T21:35:38.993 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "total_entries": 0, 2023-05-31T21:35:38.994 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "pos": 0, 2023-05-31T21:35:38.994 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "timestamp": "0.000000", 2023-05-31T21:35:38.994 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "realm_epoch": 2 2023-05-31T21:35:38.994 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2023-05-31T21:35:38.994 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2023-05-31T21:35:38.995 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2023-05-31T21:35:38.995 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "key": 2, 2023-05-31T21:35:38.995 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "val": { 2023-05-31T21:35:38.995 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "state": 1, 2023-05-31T21:35:38.996 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "marker": "1_1685562970.274572_93.1", 2023-05-31T21:35:38.996 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "next_step_marker": "", 2023-05-31T21:35:38.996 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "total_entries": 0, 2023-05-31T21:35:38.996 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "pos": 0, 2023-05-31T21:35:38.997 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "timestamp": "2023-05-31T19:56:10.274572Z", 2023-05-31T21:35:38.997 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "realm_epoch": 2 2023-05-31T21:35:38.997 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2023-05-31T21:35:38.997 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2023-05-31T21:35:38.997 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2023-05-31T21:35:38.998 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "key": 3, 2023-05-31T21:35:38.998 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "val": { 2023-05-31T21:35:38.998 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "state": 1, 2023-05-31T21:35:38.998 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "marker": "1_1685563798.772373_248.1", 2023-05-31T21:35:38.999 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "next_step_marker": "", 2023-05-31T21:35:38.999 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "total_entries": 0, 2023-05-31T21:35:38.999 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "pos": 0, 2023-05-31T21:35:38.999 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "timestamp": "2023-05-31T20:09:58.772373Z", 2023-05-31T21:35:39.000 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "realm_epoch": 2 2023-05-31T21:35:39.000 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2023-05-31T21:35:39.000 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2023-05-31T21:35:39.000 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: ] 2023-05-31T21:35:39.001 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2023-05-31T21:35:39.001 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "full_sync": { 2023-05-31T21:35:39.001 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "total": 1, 2023-05-31T21:35:39.001 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "complete": 1 2023-05-31T21:35:39.001 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2023-05-31T21:35:39.002 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "current_time": "2023-05-31T20:12:47Z" 2023-05-31T21:35:39.002 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2023-05-31T21:35:39.002 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2023-05-31T21:35:39.002 INFO:tasks.rgw_multisite_tests: "sync_status": { 2023-05-31T21:35:39.003 INFO:tasks.rgw_multisite_tests: "info": { 2023-05-31T21:35:39.003 INFO:tasks.rgw_multisite_tests: "status": "sync", 2023-05-31T21:35:39.003 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2023-05-31T21:35:39.003 INFO:tasks.rgw_multisite_tests: "period": "2e6123d2-d695-4350-9a01-61a455bc782b", 2023-05-31T21:35:39.004 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-05-31T21:35:39.004 INFO:tasks.rgw_multisite_tests: }, 2023-05-31T21:35:39.004 INFO:tasks.rgw_multisite_tests: "markers": [ 2023-05-31T21:35:39.004 INFO:tasks.rgw_multisite_tests: { 2023-05-31T21:35:39.005 INFO:tasks.rgw_multisite_tests: "key": 0, 2023-05-31T21:35:39.005 INFO:tasks.rgw_multisite_tests: "val": { 2023-05-31T21:35:39.005 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-05-31T21:35:39.005 INFO:tasks.rgw_multisite_tests: "marker": "1_1685562631.516089_59.1", 2023-05-31T21:35:39.005 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-05-31T21:35:39.006 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2023-05-31T21:35:39.006 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-05-31T21:35:39.006 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-05-31T19:50:31.516089Z", 2023-05-31T21:35:39.006 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-05-31T21:35:39.007 INFO:tasks.rgw_multisite_tests: } 2023-05-31T21:35:39.007 INFO:tasks.rgw_multisite_tests: }, 2023-05-31T21:35:39.007 INFO:tasks.rgw_multisite_tests: { 2023-05-31T21:35:39.007 INFO:tasks.rgw_multisite_tests: "key": 1, 2023-05-31T21:35:39.007 INFO:tasks.rgw_multisite_tests: "val": { 2023-05-31T21:35:39.008 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-05-31T21:35:39.008 INFO:tasks.rgw_multisite_tests: "marker": "", 2023-05-31T21:35:39.008 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-05-31T21:35:39.008 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-05-31T21:35:39.009 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-05-31T21:35:39.009 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2023-05-31T21:35:39.009 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-05-31T21:35:39.009 INFO:tasks.rgw_multisite_tests: } 2023-05-31T21:35:39.010 INFO:tasks.rgw_multisite_tests: }, 2023-05-31T21:35:39.010 INFO:tasks.rgw_multisite_tests: { 2023-05-31T21:35:39.010 INFO:tasks.rgw_multisite_tests: "key": 2, 2023-05-31T21:35:39.010 INFO:tasks.rgw_multisite_tests: "val": { 2023-05-31T21:35:39.011 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-05-31T21:35:39.011 INFO:tasks.rgw_multisite_tests: "marker": "1_1685562970.274572_93.1", 2023-05-31T21:35:39.011 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-05-31T21:35:39.011 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-05-31T21:35:39.011 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-05-31T21:35:39.012 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-05-31T19:56:10.274572Z", 2023-05-31T21:35:39.012 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-05-31T21:35:39.012 INFO:tasks.rgw_multisite_tests: } 2023-05-31T21:35:39.013 INFO:tasks.rgw_multisite_tests: }, 2023-05-31T21:35:39.013 INFO:tasks.rgw_multisite_tests: { 2023-05-31T21:35:39.013 INFO:tasks.rgw_multisite_tests: "key": 3, 2023-05-31T21:35:39.013 INFO:tasks.rgw_multisite_tests: "val": { 2023-05-31T21:35:39.014 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-05-31T21:35:39.014 INFO:tasks.rgw_multisite_tests: "marker": "1_1685563798.772373_248.1", 2023-05-31T21:35:39.015 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-05-31T21:35:39.015 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-05-31T21:35:39.015 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-05-31T21:35:39.015 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-05-31T20:09:58.772373Z", 2023-05-31T21:35:39.016 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-05-31T21:35:39.016 INFO:tasks.rgw_multisite_tests: } 2023-05-31T21:35:39.016 INFO:tasks.rgw_multisite_tests: } 2023-05-31T21:35:39.016 INFO:tasks.rgw_multisite_tests: ] 2023-05-31T21:35:39.017 INFO:tasks.rgw_multisite_tests: }, 2023-05-31T21:35:39.017 INFO:tasks.rgw_multisite_tests: "full_sync": { 2023-05-31T21:35:39.017 INFO:tasks.rgw_multisite_tests: "total": 1, 2023-05-31T21:35:39.017 INFO:tasks.rgw_multisite_tests: "complete": 1 2023-05-31T21:35:39.018 INFO:tasks.rgw_multisite_tests: }, 2023-05-31T21:35:39.018 INFO:tasks.rgw_multisite_tests: "current_time": "2023-05-31T20:12:47Z" 2023-05-31T21:35:39.018 INFO:tasks.rgw_multisite_tests:} 2023-05-31T21:35:39.018 INFO:tasks.rgw_multisite_tests: 2023-05-31T21:35:39.019 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1685562631.516089_59.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-05-31T19:50:31.516089Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1685562970.274572_93.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-05-31T19:56:10.274572Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1685563798.772373_248.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-05-31T20:09:58.772373Z', 'realm_epoch': 2}}] 2023-05-31T21:35:39.019 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1685562631.516089_59.1', 1: '', 2: '1_1685562970.274572_93.1', 3: '1_1685563802.468126_296.1'} 2023-05-31T21:35:39.019 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1685562631.516089_59.1', 1: '', 2: '1_1685562970.274572_93.1', 3: '1_1685563798.772373_248.1'} 2023-05-31T21:35:39.019 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone a2 behind master: shard=3 master=1_1685563802.468126_296.1 target=1_1685563798.772373_248.1 2023-05-31T21:35:39.019 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-31T21:35:39.020 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-31T21:35:39.020 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-05-31T21:35:39.020 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: ignoring --setuser ceph since I am not root 2023-05-31T21:35:39.020 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-05-31T21:35:39.021 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: 2023-05-31T20:12:52.281+0000 7f860d200780 20 RGW-SYNC:meta: read sync status 2023-05-31T21:35:39.021 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: 2023-05-31T20:12:52.282+0000 7f860d200780 20 RGW-SYNC:meta: read sync status 2023-05-31T21:35:39.021 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2023-05-31T21:35:39.021 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "sync_status": { 2023-05-31T21:35:39.022 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "info": { 2023-05-31T21:35:39.022 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "status": "sync", 2023-05-31T21:35:39.022 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "num_shards": 4, 2023-05-31T21:35:39.022 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "period": "2e6123d2-d695-4350-9a01-61a455bc782b", 2023-05-31T21:35:39.023 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "realm_epoch": 2 2023-05-31T21:35:39.023 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2023-05-31T21:35:39.023 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "markers": [ 2023-05-31T21:35:39.023 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2023-05-31T21:35:39.023 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "key": 0, 2023-05-31T21:35:39.024 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "val": { 2023-05-31T21:35:39.024 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "state": 1, 2023-05-31T21:35:39.024 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "marker": "1_1685562631.516089_59.1", 2023-05-31T21:35:39.024 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "next_step_marker": "", 2023-05-31T21:35:39.025 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "total_entries": 1, 2023-05-31T21:35:39.025 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "pos": 0, 2023-05-31T21:35:39.025 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "timestamp": "2023-05-31T19:50:31.516089Z", 2023-05-31T21:35:39.025 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "realm_epoch": 2 2023-05-31T21:35:39.026 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2023-05-31T21:35:39.026 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2023-05-31T21:35:39.026 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2023-05-31T21:35:39.026 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "key": 1, 2023-05-31T21:35:39.027 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "val": { 2023-05-31T21:35:39.027 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "state": 1, 2023-05-31T21:35:39.027 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "marker": "", 2023-05-31T21:35:39.027 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "next_step_marker": "", 2023-05-31T21:35:39.028 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "total_entries": 0, 2023-05-31T21:35:39.028 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "pos": 0, 2023-05-31T21:35:39.028 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "timestamp": "0.000000", 2023-05-31T21:35:39.028 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "realm_epoch": 2 2023-05-31T21:35:39.028 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2023-05-31T21:35:39.029 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2023-05-31T21:35:39.029 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2023-05-31T21:35:39.029 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "key": 2, 2023-05-31T21:35:39.029 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "val": { 2023-05-31T21:35:39.030 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "state": 1, 2023-05-31T21:35:39.030 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "marker": "1_1685562970.274572_93.1", 2023-05-31T21:35:39.030 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "next_step_marker": "", 2023-05-31T21:35:39.030 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "total_entries": 0, 2023-05-31T21:35:39.031 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "pos": 0, 2023-05-31T21:35:39.031 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "timestamp": "2023-05-31T19:56:10.274572Z", 2023-05-31T21:35:39.031 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "realm_epoch": 2 2023-05-31T21:35:39.031 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2023-05-31T21:35:39.031 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2023-05-31T21:35:39.032 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2023-05-31T21:35:39.032 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "key": 3, 2023-05-31T21:35:39.032 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "val": { 2023-05-31T21:35:39.032 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "state": 1, 2023-05-31T21:35:39.033 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "marker": "1_1685563798.885419_253.1", 2023-05-31T21:35:39.033 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "next_step_marker": "", 2023-05-31T21:35:39.033 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "total_entries": 0, 2023-05-31T21:35:39.033 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "pos": 0, 2023-05-31T21:35:39.034 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "timestamp": "2023-05-31T20:09:58.885419Z", 2023-05-31T21:35:39.034 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "realm_epoch": 2 2023-05-31T21:35:39.034 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2023-05-31T21:35:39.034 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2023-05-31T21:35:39.034 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: ] 2023-05-31T21:35:39.035 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2023-05-31T21:35:39.035 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "full_sync": { 2023-05-31T21:35:39.035 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "total": 1, 2023-05-31T21:35:39.035 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "complete": 1 2023-05-31T21:35:39.036 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2023-05-31T21:35:39.036 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "current_time": "2023-05-31T20:12:52Z" 2023-05-31T21:35:39.036 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2023-05-31T21:35:39.036 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2023-05-31T21:35:39.037 INFO:tasks.rgw_multisite_tests: "sync_status": { 2023-05-31T21:35:39.037 INFO:tasks.rgw_multisite_tests: "info": { 2023-05-31T21:35:39.037 INFO:tasks.rgw_multisite_tests: "status": "sync", 2023-05-31T21:35:39.037 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2023-05-31T21:35:39.038 INFO:tasks.rgw_multisite_tests: "period": "2e6123d2-d695-4350-9a01-61a455bc782b", 2023-05-31T21:35:39.038 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-05-31T21:35:39.038 INFO:tasks.rgw_multisite_tests: }, 2023-05-31T21:35:39.038 INFO:tasks.rgw_multisite_tests: "markers": [ 2023-05-31T21:35:39.039 INFO:tasks.rgw_multisite_tests: { 2023-05-31T21:35:39.039 INFO:tasks.rgw_multisite_tests: "key": 0, 2023-05-31T21:35:39.039 INFO:tasks.rgw_multisite_tests: "val": { 2023-05-31T21:35:39.039 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-05-31T21:35:39.039 INFO:tasks.rgw_multisite_tests: "marker": "1_1685562631.516089_59.1", 2023-05-31T21:35:39.040 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-05-31T21:35:39.040 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2023-05-31T21:35:39.040 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-05-31T21:35:39.040 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-05-31T19:50:31.516089Z", 2023-05-31T21:35:39.041 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-05-31T21:35:39.041 INFO:tasks.rgw_multisite_tests: } 2023-05-31T21:35:39.041 INFO:tasks.rgw_multisite_tests: }, 2023-05-31T21:35:39.041 INFO:tasks.rgw_multisite_tests: { 2023-05-31T21:35:39.042 INFO:tasks.rgw_multisite_tests: "key": 1, 2023-05-31T21:35:39.042 INFO:tasks.rgw_multisite_tests: "val": { 2023-05-31T21:35:39.042 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-05-31T21:35:39.042 INFO:tasks.rgw_multisite_tests: "marker": "", 2023-05-31T21:35:39.042 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-05-31T21:35:39.043 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-05-31T21:35:39.043 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-05-31T21:35:39.043 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2023-05-31T21:35:39.043 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-05-31T21:35:39.044 INFO:tasks.rgw_multisite_tests: } 2023-05-31T21:35:39.044 INFO:tasks.rgw_multisite_tests: }, 2023-05-31T21:35:39.044 INFO:tasks.rgw_multisite_tests: { 2023-05-31T21:35:39.044 INFO:tasks.rgw_multisite_tests: "key": 2, 2023-05-31T21:35:39.045 INFO:tasks.rgw_multisite_tests: "val": { 2023-05-31T21:35:39.045 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-05-31T21:35:39.045 INFO:tasks.rgw_multisite_tests: "marker": "1_1685562970.274572_93.1", 2023-05-31T21:35:39.045 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-05-31T21:35:39.045 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-05-31T21:35:39.046 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-05-31T21:35:39.046 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-05-31T19:56:10.274572Z", 2023-05-31T21:35:39.046 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-05-31T21:35:39.046 INFO:tasks.rgw_multisite_tests: } 2023-05-31T21:35:39.047 INFO:tasks.rgw_multisite_tests: }, 2023-05-31T21:35:39.047 INFO:tasks.rgw_multisite_tests: { 2023-05-31T21:35:39.047 INFO:tasks.rgw_multisite_tests: "key": 3, 2023-05-31T21:35:39.047 INFO:tasks.rgw_multisite_tests: "val": { 2023-05-31T21:35:39.047 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-05-31T21:35:39.048 INFO:tasks.rgw_multisite_tests: "marker": "1_1685563798.885419_253.1", 2023-05-31T21:35:39.048 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-05-31T21:35:39.048 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-05-31T21:35:39.048 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-05-31T21:35:39.049 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-05-31T20:09:58.885419Z", 2023-05-31T21:35:39.049 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-05-31T21:35:39.049 INFO:tasks.rgw_multisite_tests: } 2023-05-31T21:35:39.049 INFO:tasks.rgw_multisite_tests: } 2023-05-31T21:35:39.050 INFO:tasks.rgw_multisite_tests: ] 2023-05-31T21:35:39.050 INFO:tasks.rgw_multisite_tests: }, 2023-05-31T21:35:39.050 INFO:tasks.rgw_multisite_tests: "full_sync": { 2023-05-31T21:35:39.050 INFO:tasks.rgw_multisite_tests: "total": 1, 2023-05-31T21:35:39.051 INFO:tasks.rgw_multisite_tests: "complete": 1 2023-05-31T21:35:39.051 INFO:tasks.rgw_multisite_tests: }, 2023-05-31T21:35:39.051 INFO:tasks.rgw_multisite_tests: "current_time": "2023-05-31T20:12:52Z" 2023-05-31T21:35:39.051 INFO:tasks.rgw_multisite_tests:} 2023-05-31T21:35:39.051 INFO:tasks.rgw_multisite_tests: 2023-05-31T21:35:39.052 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1685562631.516089_59.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-05-31T19:50:31.516089Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1685562970.274572_93.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-05-31T19:56:10.274572Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1685563798.885419_253.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-05-31T20:09:58.885419Z', 'realm_epoch': 2}}] 2023-05-31T21:35:39.052 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1685562631.516089_59.1', 1: '', 2: '1_1685562970.274572_93.1', 3: '1_1685563802.468126_296.1'} 2023-05-31T21:35:39.052 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1685562631.516089_59.1', 1: '', 2: '1_1685562970.274572_93.1', 3: '1_1685563798.885419_253.1'} 2023-05-31T21:35:39.052 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone a2 behind master: shard=3 master=1_1685563802.468126_296.1 target=1_1685563798.885419_253.1 2023-05-31T21:35:39.053 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-31T21:35:39.053 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-31T21:35:39.053 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-05-31T21:35:39.053 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: ignoring --setuser ceph since I am not root 2023-05-31T21:35:39.054 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-05-31T21:35:39.054 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: 2023-05-31T20:12:57.406+0000 7f7753179780 20 RGW-SYNC:meta: read sync status 2023-05-31T21:35:39.054 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: 2023-05-31T20:12:57.407+0000 7f7753179780 20 RGW-SYNC:meta: read sync status 2023-05-31T21:35:39.054 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2023-05-31T21:35:39.054 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "sync_status": { 2023-05-31T21:35:39.055 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "info": { 2023-05-31T21:35:39.055 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "status": "sync", 2023-05-31T21:35:39.055 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "num_shards": 4, 2023-05-31T21:35:39.055 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "period": "2e6123d2-d695-4350-9a01-61a455bc782b", 2023-05-31T21:35:39.056 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "realm_epoch": 2 2023-05-31T21:35:39.056 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2023-05-31T21:35:39.056 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "markers": [ 2023-05-31T21:35:39.056 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2023-05-31T21:35:39.057 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "key": 0, 2023-05-31T21:35:39.057 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "val": { 2023-05-31T21:35:39.057 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "state": 1, 2023-05-31T21:35:39.057 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "marker": "1_1685562631.516089_59.1", 2023-05-31T21:35:39.058 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "next_step_marker": "", 2023-05-31T21:35:39.058 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "total_entries": 1, 2023-05-31T21:35:39.058 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "pos": 0, 2023-05-31T21:35:39.058 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "timestamp": "2023-05-31T19:50:31.516089Z", 2023-05-31T21:35:39.059 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "realm_epoch": 2 2023-05-31T21:35:39.059 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2023-05-31T21:35:39.059 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2023-05-31T21:35:39.059 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2023-05-31T21:35:39.059 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "key": 1, 2023-05-31T21:35:39.060 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "val": { 2023-05-31T21:35:39.060 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "state": 1, 2023-05-31T21:35:39.060 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "marker": "", 2023-05-31T21:35:39.060 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "next_step_marker": "", 2023-05-31T21:35:39.061 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "total_entries": 0, 2023-05-31T21:35:39.061 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "pos": 0, 2023-05-31T21:35:39.061 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "timestamp": "0.000000", 2023-05-31T21:35:39.061 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "realm_epoch": 2 2023-05-31T21:35:39.062 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2023-05-31T21:35:39.062 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2023-05-31T21:35:39.062 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2023-05-31T21:35:39.062 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "key": 2, 2023-05-31T21:35:39.062 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "val": { 2023-05-31T21:35:39.063 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "state": 1, 2023-05-31T21:35:39.063 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "marker": "1_1685562970.274572_93.1", 2023-05-31T21:35:39.063 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "next_step_marker": "", 2023-05-31T21:35:39.063 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "total_entries": 0, 2023-05-31T21:35:39.064 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "pos": 0, 2023-05-31T21:35:39.064 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "timestamp": "2023-05-31T19:56:10.274572Z", 2023-05-31T21:35:39.064 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "realm_epoch": 2 2023-05-31T21:35:39.064 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2023-05-31T21:35:39.065 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2023-05-31T21:35:39.065 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2023-05-31T21:35:39.065 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "key": 3, 2023-05-31T21:35:39.065 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "val": { 2023-05-31T21:35:39.066 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "state": 1, 2023-05-31T21:35:39.066 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "marker": "1_1685563799.051608_261.1", 2023-05-31T21:35:39.066 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "next_step_marker": "", 2023-05-31T21:35:39.066 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "total_entries": 0, 2023-05-31T21:35:39.066 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "pos": 0, 2023-05-31T21:35:39.067 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "timestamp": "2023-05-31T20:09:59.051608Z", 2023-05-31T21:35:39.067 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "realm_epoch": 2 2023-05-31T21:35:39.067 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2023-05-31T21:35:39.067 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2023-05-31T21:35:39.068 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: ] 2023-05-31T21:35:39.068 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2023-05-31T21:35:39.068 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "full_sync": { 2023-05-31T21:35:39.068 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "total": 1, 2023-05-31T21:35:39.069 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "complete": 1 2023-05-31T21:35:39.069 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2023-05-31T21:35:39.069 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "current_time": "2023-05-31T20:12:57Z" 2023-05-31T21:35:39.069 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2023-05-31T21:35:39.070 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2023-05-31T21:35:39.070 INFO:tasks.rgw_multisite_tests: "sync_status": { 2023-05-31T21:35:39.070 INFO:tasks.rgw_multisite_tests: "info": { 2023-05-31T21:35:39.070 INFO:tasks.rgw_multisite_tests: "status": "sync", 2023-05-31T21:35:39.070 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2023-05-31T21:35:39.071 INFO:tasks.rgw_multisite_tests: "period": "2e6123d2-d695-4350-9a01-61a455bc782b", 2023-05-31T21:35:39.071 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-05-31T21:35:39.071 INFO:tasks.rgw_multisite_tests: }, 2023-05-31T21:35:39.071 INFO:tasks.rgw_multisite_tests: "markers": [ 2023-05-31T21:35:39.072 INFO:tasks.rgw_multisite_tests: { 2023-05-31T21:35:39.072 INFO:tasks.rgw_multisite_tests: "key": 0, 2023-05-31T21:35:39.072 INFO:tasks.rgw_multisite_tests: "val": { 2023-05-31T21:35:39.072 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-05-31T21:35:39.073 INFO:tasks.rgw_multisite_tests: "marker": "1_1685562631.516089_59.1", 2023-05-31T21:35:39.073 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-05-31T21:35:39.073 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2023-05-31T21:35:39.073 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-05-31T21:35:39.073 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-05-31T19:50:31.516089Z", 2023-05-31T21:35:39.074 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-05-31T21:35:39.074 INFO:tasks.rgw_multisite_tests: } 2023-05-31T21:35:39.074 INFO:tasks.rgw_multisite_tests: }, 2023-05-31T21:35:39.074 INFO:tasks.rgw_multisite_tests: { 2023-05-31T21:35:39.075 INFO:tasks.rgw_multisite_tests: "key": 1, 2023-05-31T21:35:39.075 INFO:tasks.rgw_multisite_tests: "val": { 2023-05-31T21:35:39.075 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-05-31T21:35:39.075 INFO:tasks.rgw_multisite_tests: "marker": "", 2023-05-31T21:35:39.075 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-05-31T21:35:39.076 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-05-31T21:35:39.076 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-05-31T21:35:39.076 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2023-05-31T21:35:39.076 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-05-31T21:35:39.077 INFO:tasks.rgw_multisite_tests: } 2023-05-31T21:35:39.077 INFO:tasks.rgw_multisite_tests: }, 2023-05-31T21:35:39.077 INFO:tasks.rgw_multisite_tests: { 2023-05-31T21:35:39.077 INFO:tasks.rgw_multisite_tests: "key": 2, 2023-05-31T21:35:39.078 INFO:tasks.rgw_multisite_tests: "val": { 2023-05-31T21:35:39.078 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-05-31T21:35:39.078 INFO:tasks.rgw_multisite_tests: "marker": "1_1685562970.274572_93.1", 2023-05-31T21:35:39.078 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-05-31T21:35:39.079 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-05-31T21:35:39.079 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-05-31T21:35:39.079 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-05-31T19:56:10.274572Z", 2023-05-31T21:35:39.079 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-05-31T21:35:39.080 INFO:tasks.rgw_multisite_tests: } 2023-05-31T21:35:39.080 INFO:tasks.rgw_multisite_tests: }, 2023-05-31T21:35:39.080 INFO:tasks.rgw_multisite_tests: { 2023-05-31T21:35:39.080 INFO:tasks.rgw_multisite_tests: "key": 3, 2023-05-31T21:35:39.081 INFO:tasks.rgw_multisite_tests: "val": { 2023-05-31T21:35:39.081 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-05-31T21:35:39.081 INFO:tasks.rgw_multisite_tests: "marker": "1_1685563799.051608_261.1", 2023-05-31T21:35:39.081 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-05-31T21:35:39.082 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-05-31T21:35:39.082 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-05-31T21:35:39.082 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-05-31T20:09:59.051608Z", 2023-05-31T21:35:39.082 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-05-31T21:35:39.082 INFO:tasks.rgw_multisite_tests: } 2023-05-31T21:35:39.083 INFO:tasks.rgw_multisite_tests: } 2023-05-31T21:35:39.083 INFO:tasks.rgw_multisite_tests: ] 2023-05-31T21:35:39.083 INFO:tasks.rgw_multisite_tests: }, 2023-05-31T21:35:39.083 INFO:tasks.rgw_multisite_tests: "full_sync": { 2023-05-31T21:35:39.084 INFO:tasks.rgw_multisite_tests: "total": 1, 2023-05-31T21:35:39.084 INFO:tasks.rgw_multisite_tests: "complete": 1 2023-05-31T21:35:39.084 INFO:tasks.rgw_multisite_tests: }, 2023-05-31T21:35:39.084 INFO:tasks.rgw_multisite_tests: "current_time": "2023-05-31T20:12:57Z" 2023-05-31T21:35:39.084 INFO:tasks.rgw_multisite_tests:} 2023-05-31T21:35:39.085 INFO:tasks.rgw_multisite_tests: 2023-05-31T21:35:39.085 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1685562631.516089_59.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-05-31T19:50:31.516089Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1685562970.274572_93.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-05-31T19:56:10.274572Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1685563799.051608_261.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-05-31T20:09:59.051608Z', 'realm_epoch': 2}}] 2023-05-31T21:35:39.085 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1685562631.516089_59.1', 1: '', 2: '1_1685562970.274572_93.1', 3: '1_1685563802.468126_296.1'} 2023-05-31T21:35:39.085 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1685562631.516089_59.1', 1: '', 2: '1_1685562970.274572_93.1', 3: '1_1685563799.051608_261.1'} 2023-05-31T21:35:39.086 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone a2 behind master: shard=3 master=1_1685563802.468126_296.1 target=1_1685563799.051608_261.1 2023-05-31T21:35:39.086 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-31T21:35:39.086 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-31T21:35:39.086 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-05-31T21:35:39.087 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: ignoring --setuser ceph since I am not root 2023-05-31T21:35:39.087 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-05-31T21:35:39.087 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: 2023-05-31T20:13:02.578+0000 7f6618cde780 20 RGW-SYNC:meta: read sync status 2023-05-31T21:35:39.087 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: 2023-05-31T20:13:02.579+0000 7f6618cde780 20 RGW-SYNC:meta: read sync status 2023-05-31T21:35:39.088 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2023-05-31T21:35:39.088 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "sync_status": { 2023-05-31T21:35:39.088 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "info": { 2023-05-31T21:35:39.088 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "status": "sync", 2023-05-31T21:35:39.089 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "num_shards": 4, 2023-05-31T21:35:39.089 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "period": "2e6123d2-d695-4350-9a01-61a455bc782b", 2023-05-31T21:35:39.089 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "realm_epoch": 2 2023-05-31T21:35:39.089 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2023-05-31T21:35:39.089 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "markers": [ 2023-05-31T21:35:39.090 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2023-05-31T21:35:39.090 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "key": 0, 2023-05-31T21:35:39.090 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "val": { 2023-05-31T21:35:39.090 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "state": 1, 2023-05-31T21:35:39.091 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "marker": "1_1685562631.516089_59.1", 2023-05-31T21:35:39.091 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "next_step_marker": "", 2023-05-31T21:35:39.092 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "total_entries": 1, 2023-05-31T21:35:39.092 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "pos": 0, 2023-05-31T21:35:39.092 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "timestamp": "2023-05-31T19:50:31.516089Z", 2023-05-31T21:35:39.093 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "realm_epoch": 2 2023-05-31T21:35:39.093 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2023-05-31T21:35:39.093 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2023-05-31T21:35:39.093 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2023-05-31T21:35:39.094 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "key": 1, 2023-05-31T21:35:39.094 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "val": { 2023-05-31T21:35:39.094 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "state": 1, 2023-05-31T21:35:39.094 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "marker": "", 2023-05-31T21:35:39.095 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "next_step_marker": "", 2023-05-31T21:35:39.095 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "total_entries": 0, 2023-05-31T21:35:39.095 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "pos": 0, 2023-05-31T21:35:39.095 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "timestamp": "0.000000", 2023-05-31T21:35:39.096 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "realm_epoch": 2 2023-05-31T21:35:39.096 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2023-05-31T21:35:39.096 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2023-05-31T21:35:39.096 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2023-05-31T21:35:39.097 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "key": 2, 2023-05-31T21:35:39.097 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "val": { 2023-05-31T21:35:39.097 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "state": 1, 2023-05-31T21:35:39.097 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "marker": "1_1685562970.274572_93.1", 2023-05-31T21:35:39.097 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "next_step_marker": "", 2023-05-31T21:35:39.098 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "total_entries": 0, 2023-05-31T21:35:39.098 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "pos": 0, 2023-05-31T21:35:39.098 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "timestamp": "2023-05-31T19:56:10.274572Z", 2023-05-31T21:35:39.098 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "realm_epoch": 2 2023-05-31T21:35:39.099 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2023-05-31T21:35:39.099 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2023-05-31T21:35:39.099 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2023-05-31T21:35:39.099 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "key": 3, 2023-05-31T21:35:39.100 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "val": { 2023-05-31T21:35:39.100 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "state": 1, 2023-05-31T21:35:39.100 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "marker": "1_1685563798.756239_246.1", 2023-05-31T21:35:39.100 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "next_step_marker": "", 2023-05-31T21:35:39.101 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "total_entries": 0, 2023-05-31T21:35:39.101 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "pos": 0, 2023-05-31T21:35:39.101 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "timestamp": "2023-05-31T20:09:58.756239Z", 2023-05-31T21:35:39.101 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "realm_epoch": 2 2023-05-31T21:35:39.101 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2023-05-31T21:35:39.102 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2023-05-31T21:35:39.102 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: ] 2023-05-31T21:35:39.102 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2023-05-31T21:35:39.102 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "full_sync": { 2023-05-31T21:35:39.103 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "total": 1, 2023-05-31T21:35:39.103 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "complete": 1 2023-05-31T21:35:39.103 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2023-05-31T21:35:39.103 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "current_time": "2023-05-31T20:13:02Z" 2023-05-31T21:35:39.104 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2023-05-31T21:35:39.104 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2023-05-31T21:35:39.104 INFO:tasks.rgw_multisite_tests: "sync_status": { 2023-05-31T21:35:39.104 INFO:tasks.rgw_multisite_tests: "info": { 2023-05-31T21:35:39.105 INFO:tasks.rgw_multisite_tests: "status": "sync", 2023-05-31T21:35:39.105 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2023-05-31T21:35:39.105 INFO:tasks.rgw_multisite_tests: "period": "2e6123d2-d695-4350-9a01-61a455bc782b", 2023-05-31T21:35:39.105 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-05-31T21:35:39.105 INFO:tasks.rgw_multisite_tests: }, 2023-05-31T21:35:39.106 INFO:tasks.rgw_multisite_tests: "markers": [ 2023-05-31T21:35:39.106 INFO:tasks.rgw_multisite_tests: { 2023-05-31T21:35:39.106 INFO:tasks.rgw_multisite_tests: "key": 0, 2023-05-31T21:35:39.106 INFO:tasks.rgw_multisite_tests: "val": { 2023-05-31T21:35:39.107 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-05-31T21:35:39.107 INFO:tasks.rgw_multisite_tests: "marker": "1_1685562631.516089_59.1", 2023-05-31T21:35:39.107 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-05-31T21:35:39.107 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2023-05-31T21:35:39.108 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-05-31T21:35:39.108 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-05-31T19:50:31.516089Z", 2023-05-31T21:35:39.108 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-05-31T21:35:39.108 INFO:tasks.rgw_multisite_tests: } 2023-05-31T21:35:39.108 INFO:tasks.rgw_multisite_tests: }, 2023-05-31T21:35:39.109 INFO:tasks.rgw_multisite_tests: { 2023-05-31T21:35:39.109 INFO:tasks.rgw_multisite_tests: "key": 1, 2023-05-31T21:35:39.109 INFO:tasks.rgw_multisite_tests: "val": { 2023-05-31T21:35:39.109 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-05-31T21:35:39.110 INFO:tasks.rgw_multisite_tests: "marker": "", 2023-05-31T21:35:39.110 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-05-31T21:35:39.110 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-05-31T21:35:39.110 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-05-31T21:35:39.111 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2023-05-31T21:35:39.111 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-05-31T21:35:39.111 INFO:tasks.rgw_multisite_tests: } 2023-05-31T21:35:39.111 INFO:tasks.rgw_multisite_tests: }, 2023-05-31T21:35:39.111 INFO:tasks.rgw_multisite_tests: { 2023-05-31T21:35:39.112 INFO:tasks.rgw_multisite_tests: "key": 2, 2023-05-31T21:35:39.112 INFO:tasks.rgw_multisite_tests: "val": { 2023-05-31T21:35:39.112 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-05-31T21:35:39.112 INFO:tasks.rgw_multisite_tests: "marker": "1_1685562970.274572_93.1", 2023-05-31T21:35:39.113 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-05-31T21:35:39.113 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-05-31T21:35:39.113 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-05-31T21:35:39.113 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-05-31T19:56:10.274572Z", 2023-05-31T21:35:39.114 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-05-31T21:35:39.114 INFO:tasks.rgw_multisite_tests: } 2023-05-31T21:35:39.114 INFO:tasks.rgw_multisite_tests: }, 2023-05-31T21:35:39.114 INFO:tasks.rgw_multisite_tests: { 2023-05-31T21:35:39.114 INFO:tasks.rgw_multisite_tests: "key": 3, 2023-05-31T21:35:39.115 INFO:tasks.rgw_multisite_tests: "val": { 2023-05-31T21:35:39.115 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-05-31T21:35:39.115 INFO:tasks.rgw_multisite_tests: "marker": "1_1685563798.756239_246.1", 2023-05-31T21:35:39.115 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-05-31T21:35:39.116 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-05-31T21:35:39.116 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-05-31T21:35:39.116 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-05-31T20:09:58.756239Z", 2023-05-31T21:35:39.116 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-05-31T21:35:39.117 INFO:tasks.rgw_multisite_tests: } 2023-05-31T21:35:39.117 INFO:tasks.rgw_multisite_tests: } 2023-05-31T21:35:39.117 INFO:tasks.rgw_multisite_tests: ] 2023-05-31T21:35:39.117 INFO:tasks.rgw_multisite_tests: }, 2023-05-31T21:35:39.117 INFO:tasks.rgw_multisite_tests: "full_sync": { 2023-05-31T21:35:39.118 INFO:tasks.rgw_multisite_tests: "total": 1, 2023-05-31T21:35:39.118 INFO:tasks.rgw_multisite_tests: "complete": 1 2023-05-31T21:35:39.118 INFO:tasks.rgw_multisite_tests: }, 2023-05-31T21:35:39.118 INFO:tasks.rgw_multisite_tests: "current_time": "2023-05-31T20:13:02Z" 2023-05-31T21:35:39.119 INFO:tasks.rgw_multisite_tests:} 2023-05-31T21:35:39.119 INFO:tasks.rgw_multisite_tests: 2023-05-31T21:35:39.119 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1685562631.516089_59.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-05-31T19:50:31.516089Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1685562970.274572_93.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-05-31T19:56:10.274572Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1685563798.756239_246.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-05-31T20:09:58.756239Z', 'realm_epoch': 2}}] 2023-05-31T21:35:39.119 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1685562631.516089_59.1', 1: '', 2: '1_1685562970.274572_93.1', 3: '1_1685563802.468126_296.1'} 2023-05-31T21:35:39.119 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1685562631.516089_59.1', 1: '', 2: '1_1685562970.274572_93.1', 3: '1_1685563798.756239_246.1'} 2023-05-31T21:35:39.120 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone a2 behind master: shard=3 master=1_1685563802.468126_296.1 target=1_1685563798.756239_246.1 2023-05-31T21:35:39.120 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-31T21:35:39.120 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-31T21:35:39.120 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-05-31T21:35:39.121 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: ignoring --setuser ceph since I am not root 2023-05-31T21:35:39.121 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-05-31T21:35:39.121 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: 2023-05-31T20:13:07.739+0000 7f4fd7d70780 20 RGW-SYNC:meta: read sync status 2023-05-31T21:35:39.121 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: 2023-05-31T20:13:07.740+0000 7f4fd7d70780 20 RGW-SYNC:meta: read sync status 2023-05-31T21:35:39.122 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2023-05-31T21:35:39.122 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "sync_status": { 2023-05-31T21:35:39.122 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "info": { 2023-05-31T21:35:39.122 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "status": "sync", 2023-05-31T21:35:39.123 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "num_shards": 4, 2023-05-31T21:35:39.123 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "period": "2e6123d2-d695-4350-9a01-61a455bc782b", 2023-05-31T21:35:39.123 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "realm_epoch": 2 2023-05-31T21:35:39.123 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2023-05-31T21:35:39.123 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "markers": [ 2023-05-31T21:35:39.124 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2023-05-31T21:35:39.124 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "key": 0, 2023-05-31T21:35:39.124 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "val": { 2023-05-31T21:35:39.124 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "state": 1, 2023-05-31T21:35:39.125 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "marker": "1_1685562631.516089_59.1", 2023-05-31T21:35:39.125 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "next_step_marker": "", 2023-05-31T21:35:39.125 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "total_entries": 1, 2023-05-31T21:35:39.125 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "pos": 0, 2023-05-31T21:35:39.126 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "timestamp": "2023-05-31T19:50:31.516089Z", 2023-05-31T21:35:39.126 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "realm_epoch": 2 2023-05-31T21:35:39.126 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2023-05-31T21:35:39.126 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2023-05-31T21:35:39.126 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2023-05-31T21:35:39.127 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "key": 1, 2023-05-31T21:35:39.127 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "val": { 2023-05-31T21:35:39.127 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "state": 1, 2023-05-31T21:35:39.127 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "marker": "", 2023-05-31T21:35:39.128 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "next_step_marker": "", 2023-05-31T21:35:39.128 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "total_entries": 0, 2023-05-31T21:35:39.128 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "pos": 0, 2023-05-31T21:35:39.128 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "timestamp": "0.000000", 2023-05-31T21:35:39.129 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "realm_epoch": 2 2023-05-31T21:35:39.129 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2023-05-31T21:35:39.129 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2023-05-31T21:35:39.129 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2023-05-31T21:35:39.129 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "key": 2, 2023-05-31T21:35:39.130 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "val": { 2023-05-31T21:35:39.130 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "state": 1, 2023-05-31T21:35:39.130 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "marker": "1_1685562970.274572_93.1", 2023-05-31T21:35:39.130 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "next_step_marker": "", 2023-05-31T21:35:39.131 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "total_entries": 0, 2023-05-31T21:35:39.131 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "pos": 0, 2023-05-31T21:35:39.131 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "timestamp": "2023-05-31T19:56:10.274572Z", 2023-05-31T21:35:39.131 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "realm_epoch": 2 2023-05-31T21:35:39.132 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2023-05-31T21:35:39.132 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2023-05-31T21:35:39.132 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2023-05-31T21:35:39.132 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "key": 3, 2023-05-31T21:35:39.132 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "val": { 2023-05-31T21:35:39.133 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "state": 1, 2023-05-31T21:35:39.133 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "marker": "1_1685563798.901482_255.1", 2023-05-31T21:35:39.133 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "next_step_marker": "", 2023-05-31T21:35:39.133 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "total_entries": 0, 2023-05-31T21:35:39.134 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "pos": 0, 2023-05-31T21:35:39.134 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "timestamp": "2023-05-31T20:09:58.901482Z", 2023-05-31T21:35:39.134 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "realm_epoch": 2 2023-05-31T21:35:39.134 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2023-05-31T21:35:39.135 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2023-05-31T21:35:39.135 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: ] 2023-05-31T21:35:39.135 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2023-05-31T21:35:39.135 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "full_sync": { 2023-05-31T21:35:39.135 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "total": 1, 2023-05-31T21:35:39.136 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "complete": 1 2023-05-31T21:35:39.136 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2023-05-31T21:35:39.136 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "current_time": "2023-05-31T20:13:07Z" 2023-05-31T21:35:39.136 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2023-05-31T21:35:39.137 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2023-05-31T21:35:39.137 INFO:tasks.rgw_multisite_tests: "sync_status": { 2023-05-31T21:35:39.137 INFO:tasks.rgw_multisite_tests: "info": { 2023-05-31T21:35:39.137 INFO:tasks.rgw_multisite_tests: "status": "sync", 2023-05-31T21:35:39.137 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2023-05-31T21:35:39.138 INFO:tasks.rgw_multisite_tests: "period": "2e6123d2-d695-4350-9a01-61a455bc782b", 2023-05-31T21:35:39.138 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-05-31T21:35:39.138 INFO:tasks.rgw_multisite_tests: }, 2023-05-31T21:35:39.138 INFO:tasks.rgw_multisite_tests: "markers": [ 2023-05-31T21:35:39.139 INFO:tasks.rgw_multisite_tests: { 2023-05-31T21:35:39.139 INFO:tasks.rgw_multisite_tests: "key": 0, 2023-05-31T21:35:39.139 INFO:tasks.rgw_multisite_tests: "val": { 2023-05-31T21:35:39.139 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-05-31T21:35:39.140 INFO:tasks.rgw_multisite_tests: "marker": "1_1685562631.516089_59.1", 2023-05-31T21:35:39.140 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-05-31T21:35:39.140 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2023-05-31T21:35:39.140 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-05-31T21:35:39.140 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-05-31T19:50:31.516089Z", 2023-05-31T21:35:39.141 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-05-31T21:35:39.141 INFO:tasks.rgw_multisite_tests: } 2023-05-31T21:35:39.141 INFO:tasks.rgw_multisite_tests: }, 2023-05-31T21:35:39.141 INFO:tasks.rgw_multisite_tests: { 2023-05-31T21:35:39.142 INFO:tasks.rgw_multisite_tests: "key": 1, 2023-05-31T21:35:39.142 INFO:tasks.rgw_multisite_tests: "val": { 2023-05-31T21:35:39.142 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-05-31T21:35:39.142 INFO:tasks.rgw_multisite_tests: "marker": "", 2023-05-31T21:35:39.143 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-05-31T21:35:39.143 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-05-31T21:35:39.143 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-05-31T21:35:39.143 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2023-05-31T21:35:39.143 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-05-31T21:35:39.144 INFO:tasks.rgw_multisite_tests: } 2023-05-31T21:35:39.144 INFO:tasks.rgw_multisite_tests: }, 2023-05-31T21:35:39.144 INFO:tasks.rgw_multisite_tests: { 2023-05-31T21:35:39.144 INFO:tasks.rgw_multisite_tests: "key": 2, 2023-05-31T21:35:39.145 INFO:tasks.rgw_multisite_tests: "val": { 2023-05-31T21:35:39.145 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-05-31T21:35:39.145 INFO:tasks.rgw_multisite_tests: "marker": "1_1685562970.274572_93.1", 2023-05-31T21:35:39.145 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-05-31T21:35:39.146 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-05-31T21:35:39.146 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-05-31T21:35:39.146 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-05-31T19:56:10.274572Z", 2023-05-31T21:35:39.146 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-05-31T21:35:39.146 INFO:tasks.rgw_multisite_tests: } 2023-05-31T21:35:39.147 INFO:tasks.rgw_multisite_tests: }, 2023-05-31T21:35:39.147 INFO:tasks.rgw_multisite_tests: { 2023-05-31T21:35:39.147 INFO:tasks.rgw_multisite_tests: "key": 3, 2023-05-31T21:35:39.147 INFO:tasks.rgw_multisite_tests: "val": { 2023-05-31T21:35:39.148 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-05-31T21:35:39.148 INFO:tasks.rgw_multisite_tests: "marker": "1_1685563798.901482_255.1", 2023-05-31T21:35:39.148 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-05-31T21:35:39.148 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-05-31T21:35:39.149 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-05-31T21:35:39.149 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-05-31T20:09:58.901482Z", 2023-05-31T21:35:39.149 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-05-31T21:35:39.149 INFO:tasks.rgw_multisite_tests: } 2023-05-31T21:35:39.149 INFO:tasks.rgw_multisite_tests: } 2023-05-31T21:35:39.150 INFO:tasks.rgw_multisite_tests: ] 2023-05-31T21:35:39.150 INFO:tasks.rgw_multisite_tests: }, 2023-05-31T21:35:39.150 INFO:tasks.rgw_multisite_tests: "full_sync": { 2023-05-31T21:35:39.150 INFO:tasks.rgw_multisite_tests: "total": 1, 2023-05-31T21:35:39.151 INFO:tasks.rgw_multisite_tests: "complete": 1 2023-05-31T21:35:39.151 INFO:tasks.rgw_multisite_tests: }, 2023-05-31T21:35:39.151 INFO:tasks.rgw_multisite_tests: "current_time": "2023-05-31T20:13:07Z" 2023-05-31T21:35:39.151 INFO:tasks.rgw_multisite_tests:} 2023-05-31T21:35:39.151 INFO:tasks.rgw_multisite_tests: 2023-05-31T21:35:39.152 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1685562631.516089_59.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-05-31T19:50:31.516089Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1685562970.274572_93.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-05-31T19:56:10.274572Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1685563798.901482_255.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-05-31T20:09:58.901482Z', 'realm_epoch': 2}}] 2023-05-31T21:35:39.152 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1685562631.516089_59.1', 1: '', 2: '1_1685562970.274572_93.1', 3: '1_1685563802.468126_296.1'} 2023-05-31T21:35:39.152 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1685562631.516089_59.1', 1: '', 2: '1_1685562970.274572_93.1', 3: '1_1685563798.901482_255.1'} 2023-05-31T21:35:39.152 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone a2 behind master: shard=3 master=1_1685563802.468126_296.1 target=1_1685563798.901482_255.1 2023-05-31T21:35:39.153 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-31T21:35:39.153 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-31T21:35:39.153 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-05-31T21:35:39.153 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: ignoring --setuser ceph since I am not root 2023-05-31T21:35:39.154 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-05-31T21:35:39.154 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: 2023-05-31T20:13:12.880+0000 7f610134f780 20 RGW-SYNC:meta: read sync status 2023-05-31T21:35:39.154 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: 2023-05-31T20:13:12.881+0000 7f610134f780 20 RGW-SYNC:meta: read sync status 2023-05-31T21:35:39.154 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2023-05-31T21:35:39.155 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "sync_status": { 2023-05-31T21:35:39.155 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "info": { 2023-05-31T21:35:39.155 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "status": "sync", 2023-05-31T21:35:39.155 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "num_shards": 4, 2023-05-31T21:35:39.155 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "period": "2e6123d2-d695-4350-9a01-61a455bc782b", 2023-05-31T21:35:39.156 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "realm_epoch": 2 2023-05-31T21:35:39.156 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2023-05-31T21:35:39.156 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "markers": [ 2023-05-31T21:35:39.156 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2023-05-31T21:35:39.157 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "key": 0, 2023-05-31T21:35:39.157 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "val": { 2023-05-31T21:35:39.157 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "state": 1, 2023-05-31T21:35:39.157 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "marker": "1_1685562631.516089_59.1", 2023-05-31T21:35:39.158 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "next_step_marker": "", 2023-05-31T21:35:39.158 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "total_entries": 1, 2023-05-31T21:35:39.158 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "pos": 0, 2023-05-31T21:35:39.158 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "timestamp": "2023-05-31T19:50:31.516089Z", 2023-05-31T21:35:39.159 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "realm_epoch": 2 2023-05-31T21:35:39.159 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2023-05-31T21:35:39.159 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2023-05-31T21:35:39.159 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2023-05-31T21:35:39.159 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "key": 1, 2023-05-31T21:35:39.160 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "val": { 2023-05-31T21:35:39.160 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "state": 1, 2023-05-31T21:35:39.160 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "marker": "", 2023-05-31T21:35:39.160 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "next_step_marker": "", 2023-05-31T21:35:39.161 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "total_entries": 0, 2023-05-31T21:35:39.161 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "pos": 0, 2023-05-31T21:35:39.161 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "timestamp": "0.000000", 2023-05-31T21:35:39.161 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "realm_epoch": 2 2023-05-31T21:35:39.161 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2023-05-31T21:35:39.162 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2023-05-31T21:35:39.162 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2023-05-31T21:35:39.162 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "key": 2, 2023-05-31T21:35:39.162 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "val": { 2023-05-31T21:35:39.163 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "state": 1, 2023-05-31T21:35:39.163 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "marker": "1_1685562970.274572_93.1", 2023-05-31T21:35:39.163 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "next_step_marker": "", 2023-05-31T21:35:39.163 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "total_entries": 0, 2023-05-31T21:35:39.164 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "pos": 0, 2023-05-31T21:35:39.164 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "timestamp": "2023-05-31T19:56:10.274572Z", 2023-05-31T21:35:39.164 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "realm_epoch": 2 2023-05-31T21:35:39.164 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2023-05-31T21:35:39.164 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2023-05-31T21:35:39.165 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2023-05-31T21:35:39.165 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "key": 3, 2023-05-31T21:35:39.165 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "val": { 2023-05-31T21:35:39.165 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "state": 1, 2023-05-31T21:35:39.166 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "marker": "1_1685563798.772373_248.1", 2023-05-31T21:35:39.166 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "next_step_marker": "", 2023-05-31T21:35:39.166 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "total_entries": 0, 2023-05-31T21:35:39.166 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "pos": 0, 2023-05-31T21:35:39.167 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "timestamp": "2023-05-31T20:09:58.772373Z", 2023-05-31T21:35:39.167 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "realm_epoch": 2 2023-05-31T21:35:39.167 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2023-05-31T21:35:39.167 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2023-05-31T21:35:39.167 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: ] 2023-05-31T21:35:39.168 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2023-05-31T21:35:39.168 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "full_sync": { 2023-05-31T21:35:39.168 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "total": 1, 2023-05-31T21:35:39.168 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "complete": 1 2023-05-31T21:35:39.169 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2023-05-31T21:35:39.169 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "current_time": "2023-05-31T20:13:12Z" 2023-05-31T21:35:39.169 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2023-05-31T21:35:39.169 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2023-05-31T21:35:39.170 INFO:tasks.rgw_multisite_tests: "sync_status": { 2023-05-31T21:35:39.170 INFO:tasks.rgw_multisite_tests: "info": { 2023-05-31T21:35:39.170 INFO:tasks.rgw_multisite_tests: "status": "sync", 2023-05-31T21:35:39.170 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2023-05-31T21:35:39.170 INFO:tasks.rgw_multisite_tests: "period": "2e6123d2-d695-4350-9a01-61a455bc782b", 2023-05-31T21:35:39.171 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-05-31T21:35:39.171 INFO:tasks.rgw_multisite_tests: }, 2023-05-31T21:35:39.171 INFO:tasks.rgw_multisite_tests: "markers": [ 2023-05-31T21:35:39.171 INFO:tasks.rgw_multisite_tests: { 2023-05-31T21:35:39.172 INFO:tasks.rgw_multisite_tests: "key": 0, 2023-05-31T21:35:39.172 INFO:tasks.rgw_multisite_tests: "val": { 2023-05-31T21:35:39.172 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-05-31T21:35:39.172 INFO:tasks.rgw_multisite_tests: "marker": "1_1685562631.516089_59.1", 2023-05-31T21:35:39.173 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-05-31T21:35:39.173 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2023-05-31T21:35:39.173 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-05-31T21:35:39.173 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-05-31T19:50:31.516089Z", 2023-05-31T21:35:39.174 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-05-31T21:35:39.174 INFO:tasks.rgw_multisite_tests: } 2023-05-31T21:35:39.174 INFO:tasks.rgw_multisite_tests: }, 2023-05-31T21:35:39.174 INFO:tasks.rgw_multisite_tests: { 2023-05-31T21:35:39.174 INFO:tasks.rgw_multisite_tests: "key": 1, 2023-05-31T21:35:39.175 INFO:tasks.rgw_multisite_tests: "val": { 2023-05-31T21:35:39.175 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-05-31T21:35:39.175 INFO:tasks.rgw_multisite_tests: "marker": "", 2023-05-31T21:35:39.175 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-05-31T21:35:39.176 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-05-31T21:35:39.176 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-05-31T21:35:39.176 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2023-05-31T21:35:39.176 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-05-31T21:35:39.176 INFO:tasks.rgw_multisite_tests: } 2023-05-31T21:35:39.177 INFO:tasks.rgw_multisite_tests: }, 2023-05-31T21:35:39.177 INFO:tasks.rgw_multisite_tests: { 2023-05-31T21:35:39.177 INFO:tasks.rgw_multisite_tests: "key": 2, 2023-05-31T21:35:39.177 INFO:tasks.rgw_multisite_tests: "val": { 2023-05-31T21:35:39.178 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-05-31T21:35:39.178 INFO:tasks.rgw_multisite_tests: "marker": "1_1685562970.274572_93.1", 2023-05-31T21:35:39.178 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-05-31T21:35:39.178 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-05-31T21:35:39.179 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-05-31T21:35:39.179 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-05-31T19:56:10.274572Z", 2023-05-31T21:35:39.179 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-05-31T21:35:39.179 INFO:tasks.rgw_multisite_tests: } 2023-05-31T21:35:39.179 INFO:tasks.rgw_multisite_tests: }, 2023-05-31T21:35:39.180 INFO:tasks.rgw_multisite_tests: { 2023-05-31T21:35:39.180 INFO:tasks.rgw_multisite_tests: "key": 3, 2023-05-31T21:35:39.180 INFO:tasks.rgw_multisite_tests: "val": { 2023-05-31T21:35:39.180 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-05-31T21:35:39.181 INFO:tasks.rgw_multisite_tests: "marker": "1_1685563798.772373_248.1", 2023-05-31T21:35:39.181 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-05-31T21:35:39.181 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-05-31T21:35:39.181 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-05-31T21:35:39.182 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-05-31T20:09:58.772373Z", 2023-05-31T21:35:39.182 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-05-31T21:35:39.182 INFO:tasks.rgw_multisite_tests: } 2023-05-31T21:35:39.182 INFO:tasks.rgw_multisite_tests: } 2023-05-31T21:35:39.183 INFO:tasks.rgw_multisite_tests: ] 2023-05-31T21:35:39.183 INFO:tasks.rgw_multisite_tests: }, 2023-05-31T21:35:39.183 INFO:tasks.rgw_multisite_tests: "full_sync": { 2023-05-31T21:35:39.183 INFO:tasks.rgw_multisite_tests: "total": 1, 2023-05-31T21:35:39.183 INFO:tasks.rgw_multisite_tests: "complete": 1 2023-05-31T21:35:39.184 INFO:tasks.rgw_multisite_tests: }, 2023-05-31T21:35:39.184 INFO:tasks.rgw_multisite_tests: "current_time": "2023-05-31T20:13:12Z" 2023-05-31T21:35:39.184 INFO:tasks.rgw_multisite_tests:} 2023-05-31T21:35:39.184 INFO:tasks.rgw_multisite_tests: 2023-05-31T21:35:39.185 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1685562631.516089_59.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-05-31T19:50:31.516089Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1685562970.274572_93.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-05-31T19:56:10.274572Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1685563798.772373_248.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-05-31T20:09:58.772373Z', 'realm_epoch': 2}}] 2023-05-31T21:35:39.185 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1685562631.516089_59.1', 1: '', 2: '1_1685562970.274572_93.1', 3: '1_1685563802.468126_296.1'} 2023-05-31T21:35:39.185 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1685562631.516089_59.1', 1: '', 2: '1_1685562970.274572_93.1', 3: '1_1685563798.772373_248.1'} 2023-05-31T21:35:39.185 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone a2 behind master: shard=3 master=1_1685563802.468126_296.1 target=1_1685563798.772373_248.1 2023-05-31T21:35:39.186 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-31T21:35:39.186 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-31T21:35:39.186 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-05-31T21:35:39.186 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: ignoring --setuser ceph since I am not root 2023-05-31T21:35:39.187 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-05-31T21:35:39.187 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: 2023-05-31T20:13:18.019+0000 7f41f1ed0780 20 RGW-SYNC:meta: read sync status 2023-05-31T21:35:39.187 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: 2023-05-31T20:13:18.021+0000 7f41f1ed0780 20 RGW-SYNC:meta: read sync status 2023-05-31T21:35:39.187 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2023-05-31T21:35:39.188 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "sync_status": { 2023-05-31T21:35:39.188 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "info": { 2023-05-31T21:35:39.188 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "status": "sync", 2023-05-31T21:35:39.188 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "num_shards": 4, 2023-05-31T21:35:39.188 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "period": "2e6123d2-d695-4350-9a01-61a455bc782b", 2023-05-31T21:35:39.189 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "realm_epoch": 2 2023-05-31T21:35:39.189 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2023-05-31T21:35:39.189 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "markers": [ 2023-05-31T21:35:39.189 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2023-05-31T21:35:39.190 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "key": 0, 2023-05-31T21:35:39.190 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "val": { 2023-05-31T21:35:39.190 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "state": 1, 2023-05-31T21:35:39.190 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "marker": "1_1685562631.516089_59.1", 2023-05-31T21:35:39.191 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "next_step_marker": "", 2023-05-31T21:35:39.191 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "total_entries": 1, 2023-05-31T21:35:39.191 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "pos": 0, 2023-05-31T21:35:39.191 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "timestamp": "2023-05-31T19:50:31.516089Z", 2023-05-31T21:35:39.191 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "realm_epoch": 2 2023-05-31T21:35:39.192 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2023-05-31T21:35:39.192 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2023-05-31T21:35:39.192 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2023-05-31T21:35:39.192 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "key": 1, 2023-05-31T21:35:39.193 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "val": { 2023-05-31T21:35:39.193 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "state": 1, 2023-05-31T21:35:39.193 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "marker": "", 2023-05-31T21:35:39.193 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "next_step_marker": "", 2023-05-31T21:35:39.194 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "total_entries": 0, 2023-05-31T21:35:39.194 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "pos": 0, 2023-05-31T21:35:39.194 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "timestamp": "0.000000", 2023-05-31T21:35:39.194 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "realm_epoch": 2 2023-05-31T21:35:39.194 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2023-05-31T21:35:39.195 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2023-05-31T21:35:39.195 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2023-05-31T21:35:39.195 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "key": 2, 2023-05-31T21:35:39.195 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "val": { 2023-05-31T21:35:39.196 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "state": 1, 2023-05-31T21:35:39.196 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "marker": "1_1685562970.274572_93.1", 2023-05-31T21:35:39.196 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "next_step_marker": "", 2023-05-31T21:35:39.196 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "total_entries": 0, 2023-05-31T21:35:39.197 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "pos": 0, 2023-05-31T21:35:39.197 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "timestamp": "2023-05-31T19:56:10.274572Z", 2023-05-31T21:35:39.197 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "realm_epoch": 2 2023-05-31T21:35:39.197 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2023-05-31T21:35:39.197 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2023-05-31T21:35:39.198 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2023-05-31T21:35:39.198 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "key": 3, 2023-05-31T21:35:39.198 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "val": { 2023-05-31T21:35:39.198 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "state": 1, 2023-05-31T21:35:39.199 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "marker": "1_1685563799.058314_262.1", 2023-05-31T21:35:39.199 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "next_step_marker": "", 2023-05-31T21:35:39.199 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "total_entries": 0, 2023-05-31T21:35:39.199 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "pos": 0, 2023-05-31T21:35:39.200 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "timestamp": "2023-05-31T20:09:59.058314Z", 2023-05-31T21:35:39.200 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "realm_epoch": 2 2023-05-31T21:35:39.200 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2023-05-31T21:35:39.200 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2023-05-31T21:35:39.200 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: ] 2023-05-31T21:35:39.201 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2023-05-31T21:35:39.201 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "full_sync": { 2023-05-31T21:35:39.201 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "total": 1, 2023-05-31T21:35:39.201 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "complete": 1 2023-05-31T21:35:39.202 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2023-05-31T21:35:39.202 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "current_time": "2023-05-31T20:13:18Z" 2023-05-31T21:35:39.202 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2023-05-31T21:35:39.202 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2023-05-31T21:35:39.203 INFO:tasks.rgw_multisite_tests: "sync_status": { 2023-05-31T21:35:39.203 INFO:tasks.rgw_multisite_tests: "info": { 2023-05-31T21:35:39.203 INFO:tasks.rgw_multisite_tests: "status": "sync", 2023-05-31T21:35:39.203 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2023-05-31T21:35:39.204 INFO:tasks.rgw_multisite_tests: "period": "2e6123d2-d695-4350-9a01-61a455bc782b", 2023-05-31T21:35:39.204 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-05-31T21:35:39.204 INFO:tasks.rgw_multisite_tests: }, 2023-05-31T21:35:39.204 INFO:tasks.rgw_multisite_tests: "markers": [ 2023-05-31T21:35:39.204 INFO:tasks.rgw_multisite_tests: { 2023-05-31T21:35:39.205 INFO:tasks.rgw_multisite_tests: "key": 0, 2023-05-31T21:35:39.205 INFO:tasks.rgw_multisite_tests: "val": { 2023-05-31T21:35:39.205 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-05-31T21:35:39.205 INFO:tasks.rgw_multisite_tests: "marker": "1_1685562631.516089_59.1", 2023-05-31T21:35:39.206 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-05-31T21:35:39.206 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2023-05-31T21:35:39.206 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-05-31T21:35:39.206 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-05-31T19:50:31.516089Z", 2023-05-31T21:35:39.207 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-05-31T21:35:39.207 INFO:tasks.rgw_multisite_tests: } 2023-05-31T21:35:39.207 INFO:tasks.rgw_multisite_tests: }, 2023-05-31T21:35:39.207 INFO:tasks.rgw_multisite_tests: { 2023-05-31T21:35:39.207 INFO:tasks.rgw_multisite_tests: "key": 1, 2023-05-31T21:35:39.208 INFO:tasks.rgw_multisite_tests: "val": { 2023-05-31T21:35:39.208 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-05-31T21:35:39.208 INFO:tasks.rgw_multisite_tests: "marker": "", 2023-05-31T21:35:39.208 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-05-31T21:35:39.209 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-05-31T21:35:39.209 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-05-31T21:35:39.209 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2023-05-31T21:35:39.209 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-05-31T21:35:39.210 INFO:tasks.rgw_multisite_tests: } 2023-05-31T21:35:39.210 INFO:tasks.rgw_multisite_tests: }, 2023-05-31T21:35:39.210 INFO:tasks.rgw_multisite_tests: { 2023-05-31T21:35:39.210 INFO:tasks.rgw_multisite_tests: "key": 2, 2023-05-31T21:35:39.210 INFO:tasks.rgw_multisite_tests: "val": { 2023-05-31T21:35:39.211 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-05-31T21:35:39.211 INFO:tasks.rgw_multisite_tests: "marker": "1_1685562970.274572_93.1", 2023-05-31T21:35:39.211 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-05-31T21:35:39.211 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-05-31T21:35:39.212 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-05-31T21:35:39.212 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-05-31T19:56:10.274572Z", 2023-05-31T21:35:39.212 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-05-31T21:35:39.212 INFO:tasks.rgw_multisite_tests: } 2023-05-31T21:35:39.213 INFO:tasks.rgw_multisite_tests: }, 2023-05-31T21:35:39.213 INFO:tasks.rgw_multisite_tests: { 2023-05-31T21:35:39.213 INFO:tasks.rgw_multisite_tests: "key": 3, 2023-05-31T21:35:39.213 INFO:tasks.rgw_multisite_tests: "val": { 2023-05-31T21:35:39.213 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-05-31T21:35:39.214 INFO:tasks.rgw_multisite_tests: "marker": "1_1685563799.058314_262.1", 2023-05-31T21:35:39.214 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-05-31T21:35:39.214 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-05-31T21:35:39.214 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-05-31T21:35:39.215 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-05-31T20:09:59.058314Z", 2023-05-31T21:35:39.215 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-05-31T21:35:39.215 INFO:tasks.rgw_multisite_tests: } 2023-05-31T21:35:39.215 INFO:tasks.rgw_multisite_tests: } 2023-05-31T21:35:39.215 INFO:tasks.rgw_multisite_tests: ] 2023-05-31T21:35:39.216 INFO:tasks.rgw_multisite_tests: }, 2023-05-31T21:35:39.216 INFO:tasks.rgw_multisite_tests: "full_sync": { 2023-05-31T21:35:39.216 INFO:tasks.rgw_multisite_tests: "total": 1, 2023-05-31T21:35:39.216 INFO:tasks.rgw_multisite_tests: "complete": 1 2023-05-31T21:35:39.217 INFO:tasks.rgw_multisite_tests: }, 2023-05-31T21:35:39.217 INFO:tasks.rgw_multisite_tests: "current_time": "2023-05-31T20:13:18Z" 2023-05-31T21:35:39.217 INFO:tasks.rgw_multisite_tests:} 2023-05-31T21:35:39.217 INFO:tasks.rgw_multisite_tests: 2023-05-31T21:35:39.218 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1685562631.516089_59.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-05-31T19:50:31.516089Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1685562970.274572_93.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-05-31T19:56:10.274572Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1685563799.058314_262.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-05-31T20:09:59.058314Z', 'realm_epoch': 2}}] 2023-05-31T21:35:39.218 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1685562631.516089_59.1', 1: '', 2: '1_1685562970.274572_93.1', 3: '1_1685563802.468126_296.1'} 2023-05-31T21:35:39.218 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1685562631.516089_59.1', 1: '', 2: '1_1685562970.274572_93.1', 3: '1_1685563799.058314_262.1'} 2023-05-31T21:35:39.218 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone a2 behind master: shard=3 master=1_1685563802.468126_296.1 target=1_1685563799.058314_262.1 2023-05-31T21:35:39.219 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-31T21:35:39.219 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-31T21:35:39.219 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-05-31T21:35:39.219 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: ignoring --setuser ceph since I am not root 2023-05-31T21:35:39.219 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-05-31T21:35:39.220 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: 2023-05-31T20:13:23.140+0000 7f93f83a3780 20 RGW-SYNC:meta: read sync status 2023-05-31T21:35:39.220 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: 2023-05-31T20:13:23.141+0000 7f93f83a3780 20 RGW-SYNC:meta: read sync status 2023-05-31T21:35:39.220 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2023-05-31T21:35:39.220 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "sync_status": { 2023-05-31T21:35:39.221 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "info": { 2023-05-31T21:35:39.221 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "status": "sync", 2023-05-31T21:35:39.221 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "num_shards": 4, 2023-05-31T21:35:39.221 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "period": "2e6123d2-d695-4350-9a01-61a455bc782b", 2023-05-31T21:35:39.222 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "realm_epoch": 2 2023-05-31T21:35:39.222 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2023-05-31T21:35:39.222 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "markers": [ 2023-05-31T21:35:39.222 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2023-05-31T21:35:39.223 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "key": 0, 2023-05-31T21:35:39.223 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "val": { 2023-05-31T21:35:39.223 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "state": 1, 2023-05-31T21:35:39.223 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "marker": "1_1685562631.516089_59.1", 2023-05-31T21:35:39.223 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "next_step_marker": "", 2023-05-31T21:35:39.224 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "total_entries": 1, 2023-05-31T21:35:39.224 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "pos": 0, 2023-05-31T21:35:39.224 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "timestamp": "2023-05-31T19:50:31.516089Z", 2023-05-31T21:35:39.224 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "realm_epoch": 2 2023-05-31T21:35:39.225 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2023-05-31T21:35:39.225 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2023-05-31T21:35:39.225 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2023-05-31T21:35:39.225 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "key": 1, 2023-05-31T21:35:39.226 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "val": { 2023-05-31T21:35:39.226 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "state": 1, 2023-05-31T21:35:39.226 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "marker": "", 2023-05-31T21:35:39.226 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "next_step_marker": "", 2023-05-31T21:35:39.226 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "total_entries": 0, 2023-05-31T21:35:39.227 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "pos": 0, 2023-05-31T21:35:39.227 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "timestamp": "0.000000", 2023-05-31T21:35:39.227 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "realm_epoch": 2 2023-05-31T21:35:39.227 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2023-05-31T21:35:39.228 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2023-05-31T21:35:39.228 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2023-05-31T21:35:39.228 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "key": 2, 2023-05-31T21:35:39.228 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "val": { 2023-05-31T21:35:39.229 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "state": 1, 2023-05-31T21:35:39.229 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "marker": "1_1685562970.274572_93.1", 2023-05-31T21:35:39.229 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "next_step_marker": "", 2023-05-31T21:35:39.229 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "total_entries": 0, 2023-05-31T21:35:39.229 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "pos": 0, 2023-05-31T21:35:39.230 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "timestamp": "2023-05-31T19:56:10.274572Z", 2023-05-31T21:35:39.230 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "realm_epoch": 2 2023-05-31T21:35:39.230 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2023-05-31T21:35:39.230 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2023-05-31T21:35:39.231 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2023-05-31T21:35:39.231 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "key": 3, 2023-05-31T21:35:39.231 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "val": { 2023-05-31T21:35:39.231 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "state": 1, 2023-05-31T21:35:39.232 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "marker": "1_1685563798.772373_248.1", 2023-05-31T21:35:39.232 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "next_step_marker": "", 2023-05-31T21:35:39.232 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "total_entries": 0, 2023-05-31T21:35:39.232 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "pos": 0, 2023-05-31T21:35:39.233 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "timestamp": "2023-05-31T20:09:58.772373Z", 2023-05-31T21:35:39.233 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "realm_epoch": 2 2023-05-31T21:35:39.233 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2023-05-31T21:35:39.233 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2023-05-31T21:35:39.233 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: ] 2023-05-31T21:35:39.234 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2023-05-31T21:35:39.234 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "full_sync": { 2023-05-31T21:35:39.234 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "total": 1, 2023-05-31T21:35:39.234 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "complete": 1 2023-05-31T21:35:39.235 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2023-05-31T21:35:39.235 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "current_time": "2023-05-31T20:13:23Z" 2023-05-31T21:35:39.235 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2023-05-31T21:35:39.235 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2023-05-31T21:35:39.236 INFO:tasks.rgw_multisite_tests: "sync_status": { 2023-05-31T21:35:39.236 INFO:tasks.rgw_multisite_tests: "info": { 2023-05-31T21:35:39.236 INFO:tasks.rgw_multisite_tests: "status": "sync", 2023-05-31T21:35:39.236 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2023-05-31T21:35:39.236 INFO:tasks.rgw_multisite_tests: "period": "2e6123d2-d695-4350-9a01-61a455bc782b", 2023-05-31T21:35:39.237 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-05-31T21:35:39.237 INFO:tasks.rgw_multisite_tests: }, 2023-05-31T21:35:39.237 INFO:tasks.rgw_multisite_tests: "markers": [ 2023-05-31T21:35:39.237 INFO:tasks.rgw_multisite_tests: { 2023-05-31T21:35:39.238 INFO:tasks.rgw_multisite_tests: "key": 0, 2023-05-31T21:35:39.238 INFO:tasks.rgw_multisite_tests: "val": { 2023-05-31T21:35:39.238 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-05-31T21:35:39.238 INFO:tasks.rgw_multisite_tests: "marker": "1_1685562631.516089_59.1", 2023-05-31T21:35:39.239 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-05-31T21:35:39.239 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2023-05-31T21:35:39.239 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-05-31T21:35:39.239 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-05-31T19:50:31.516089Z", 2023-05-31T21:35:39.239 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-05-31T21:35:39.240 INFO:tasks.rgw_multisite_tests: } 2023-05-31T21:35:39.240 INFO:tasks.rgw_multisite_tests: }, 2023-05-31T21:35:39.240 INFO:tasks.rgw_multisite_tests: { 2023-05-31T21:35:39.240 INFO:tasks.rgw_multisite_tests: "key": 1, 2023-05-31T21:35:39.241 INFO:tasks.rgw_multisite_tests: "val": { 2023-05-31T21:35:39.241 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-05-31T21:35:39.241 INFO:tasks.rgw_multisite_tests: "marker": "", 2023-05-31T21:35:39.241 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-05-31T21:35:39.242 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-05-31T21:35:39.242 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-05-31T21:35:39.242 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2023-05-31T21:35:39.242 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-05-31T21:35:39.242 INFO:tasks.rgw_multisite_tests: } 2023-05-31T21:35:39.243 INFO:tasks.rgw_multisite_tests: }, 2023-05-31T21:35:39.243 INFO:tasks.rgw_multisite_tests: { 2023-05-31T21:35:39.243 INFO:tasks.rgw_multisite_tests: "key": 2, 2023-05-31T21:35:39.243 INFO:tasks.rgw_multisite_tests: "val": { 2023-05-31T21:35:39.244 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-05-31T21:35:39.244 INFO:tasks.rgw_multisite_tests: "marker": "1_1685562970.274572_93.1", 2023-05-31T21:35:39.244 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-05-31T21:35:39.244 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-05-31T21:35:39.244 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-05-31T21:35:39.245 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-05-31T19:56:10.274572Z", 2023-05-31T21:35:39.245 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-05-31T21:35:39.245 INFO:tasks.rgw_multisite_tests: } 2023-05-31T21:35:39.245 INFO:tasks.rgw_multisite_tests: }, 2023-05-31T21:35:39.246 INFO:tasks.rgw_multisite_tests: { 2023-05-31T21:35:39.246 INFO:tasks.rgw_multisite_tests: "key": 3, 2023-05-31T21:35:39.246 INFO:tasks.rgw_multisite_tests: "val": { 2023-05-31T21:35:39.246 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-05-31T21:35:39.247 INFO:tasks.rgw_multisite_tests: "marker": "1_1685563798.772373_248.1", 2023-05-31T21:35:39.247 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-05-31T21:35:39.247 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-05-31T21:35:39.247 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-05-31T21:35:39.247 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-05-31T20:09:58.772373Z", 2023-05-31T21:35:39.248 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-05-31T21:35:39.248 INFO:tasks.rgw_multisite_tests: } 2023-05-31T21:35:39.248 INFO:tasks.rgw_multisite_tests: } 2023-05-31T21:35:39.248 INFO:tasks.rgw_multisite_tests: ] 2023-05-31T21:35:39.249 INFO:tasks.rgw_multisite_tests: }, 2023-05-31T21:35:39.249 INFO:tasks.rgw_multisite_tests: "full_sync": { 2023-05-31T21:35:39.249 INFO:tasks.rgw_multisite_tests: "total": 1, 2023-05-31T21:35:39.249 INFO:tasks.rgw_multisite_tests: "complete": 1 2023-05-31T21:35:39.250 INFO:tasks.rgw_multisite_tests: }, 2023-05-31T21:35:39.250 INFO:tasks.rgw_multisite_tests: "current_time": "2023-05-31T20:13:23Z" 2023-05-31T21:35:39.250 INFO:tasks.rgw_multisite_tests:} 2023-05-31T21:35:39.250 INFO:tasks.rgw_multisite_tests: 2023-05-31T21:35:39.250 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1685562631.516089_59.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-05-31T19:50:31.516089Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1685562970.274572_93.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-05-31T19:56:10.274572Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1685563798.772373_248.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-05-31T20:09:58.772373Z', 'realm_epoch': 2}}] 2023-05-31T21:35:39.251 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1685562631.516089_59.1', 1: '', 2: '1_1685562970.274572_93.1', 3: '1_1685563802.468126_296.1'} 2023-05-31T21:35:39.251 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1685562631.516089_59.1', 1: '', 2: '1_1685562970.274572_93.1', 3: '1_1685563798.772373_248.1'} 2023-05-31T21:35:39.251 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone a2 behind master: shard=3 master=1_1685563802.468126_296.1 target=1_1685563798.772373_248.1 2023-05-31T21:35:39.251 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-31T21:35:39.252 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-31T21:35:39.252 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-05-31T21:35:39.252 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: ignoring --setuser ceph since I am not root 2023-05-31T21:35:39.252 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-05-31T21:35:39.253 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: 2023-05-31T20:13:28.252+0000 7fe5bf30e780 20 RGW-SYNC:meta: read sync status 2023-05-31T21:35:39.253 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: 2023-05-31T20:13:28.253+0000 7fe5bf30e780 20 RGW-SYNC:meta: read sync status 2023-05-31T21:35:39.253 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2023-05-31T21:35:39.253 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "sync_status": { 2023-05-31T21:35:39.253 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "info": { 2023-05-31T21:35:39.254 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "status": "sync", 2023-05-31T21:35:39.254 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "num_shards": 4, 2023-05-31T21:35:39.254 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "period": "2e6123d2-d695-4350-9a01-61a455bc782b", 2023-05-31T21:35:39.254 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "realm_epoch": 2 2023-05-31T21:35:39.255 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2023-05-31T21:35:39.255 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "markers": [ 2023-05-31T21:35:39.255 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2023-05-31T21:35:39.256 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "key": 0, 2023-05-31T21:35:39.256 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "val": { 2023-05-31T21:35:39.256 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "state": 1, 2023-05-31T21:35:39.256 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "marker": "1_1685562631.516089_59.1", 2023-05-31T21:35:39.257 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "next_step_marker": "", 2023-05-31T21:35:39.257 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "total_entries": 1, 2023-05-31T21:35:39.257 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "pos": 0, 2023-05-31T21:35:39.257 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "timestamp": "2023-05-31T19:50:31.516089Z", 2023-05-31T21:35:39.258 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "realm_epoch": 2 2023-05-31T21:35:39.258 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2023-05-31T21:35:39.258 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2023-05-31T21:35:39.258 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2023-05-31T21:35:39.259 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "key": 1, 2023-05-31T21:35:39.259 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "val": { 2023-05-31T21:35:39.259 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "state": 1, 2023-05-31T21:35:39.259 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "marker": "", 2023-05-31T21:35:39.259 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "next_step_marker": "", 2023-05-31T21:35:39.260 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "total_entries": 0, 2023-05-31T21:35:39.260 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "pos": 0, 2023-05-31T21:35:39.260 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "timestamp": "0.000000", 2023-05-31T21:35:39.260 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "realm_epoch": 2 2023-05-31T21:35:39.261 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2023-05-31T21:35:39.261 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2023-05-31T21:35:39.261 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2023-05-31T21:35:39.261 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "key": 2, 2023-05-31T21:35:39.262 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "val": { 2023-05-31T21:35:39.262 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "state": 1, 2023-05-31T21:35:39.262 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "marker": "1_1685562970.274572_93.1", 2023-05-31T21:35:39.262 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "next_step_marker": "", 2023-05-31T21:35:39.263 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "total_entries": 0, 2023-05-31T21:35:39.263 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "pos": 0, 2023-05-31T21:35:39.263 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "timestamp": "2023-05-31T19:56:10.274572Z", 2023-05-31T21:35:39.263 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "realm_epoch": 2 2023-05-31T21:35:39.263 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2023-05-31T21:35:39.264 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2023-05-31T21:35:39.264 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2023-05-31T21:35:39.264 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "key": 3, 2023-05-31T21:35:39.264 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "val": { 2023-05-31T21:35:39.265 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "state": 1, 2023-05-31T21:35:39.265 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "marker": "1_1685563798.817516_251.1", 2023-05-31T21:35:39.265 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "next_step_marker": "", 2023-05-31T21:35:39.265 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "total_entries": 0, 2023-05-31T21:35:39.266 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "pos": 0, 2023-05-31T21:35:39.266 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "timestamp": "2023-05-31T20:09:58.817516Z", 2023-05-31T21:35:39.266 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "realm_epoch": 2 2023-05-31T21:35:39.266 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2023-05-31T21:35:39.267 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2023-05-31T21:35:39.267 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: ] 2023-05-31T21:35:39.267 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2023-05-31T21:35:39.267 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "full_sync": { 2023-05-31T21:35:39.268 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "total": 1, 2023-05-31T21:35:39.268 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "complete": 1 2023-05-31T21:35:39.268 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2023-05-31T21:35:39.268 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "current_time": "2023-05-31T20:13:28Z" 2023-05-31T21:35:39.268 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2023-05-31T21:35:39.269 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2023-05-31T21:35:39.269 INFO:tasks.rgw_multisite_tests: "sync_status": { 2023-05-31T21:35:39.269 INFO:tasks.rgw_multisite_tests: "info": { 2023-05-31T21:35:39.269 INFO:tasks.rgw_multisite_tests: "status": "sync", 2023-05-31T21:35:39.270 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2023-05-31T21:35:39.270 INFO:tasks.rgw_multisite_tests: "period": "2e6123d2-d695-4350-9a01-61a455bc782b", 2023-05-31T21:35:39.270 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-05-31T21:35:39.270 INFO:tasks.rgw_multisite_tests: }, 2023-05-31T21:35:39.271 INFO:tasks.rgw_multisite_tests: "markers": [ 2023-05-31T21:35:39.271 INFO:tasks.rgw_multisite_tests: { 2023-05-31T21:35:39.271 INFO:tasks.rgw_multisite_tests: "key": 0, 2023-05-31T21:35:39.271 INFO:tasks.rgw_multisite_tests: "val": { 2023-05-31T21:35:39.271 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-05-31T21:35:39.272 INFO:tasks.rgw_multisite_tests: "marker": "1_1685562631.516089_59.1", 2023-05-31T21:35:39.272 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-05-31T21:35:39.272 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2023-05-31T21:35:39.272 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-05-31T21:35:39.273 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-05-31T19:50:31.516089Z", 2023-05-31T21:35:39.273 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-05-31T21:35:39.273 INFO:tasks.rgw_multisite_tests: } 2023-05-31T21:35:39.273 INFO:tasks.rgw_multisite_tests: }, 2023-05-31T21:35:39.274 INFO:tasks.rgw_multisite_tests: { 2023-05-31T21:35:39.274 INFO:tasks.rgw_multisite_tests: "key": 1, 2023-05-31T21:35:39.274 INFO:tasks.rgw_multisite_tests: "val": { 2023-05-31T21:35:39.274 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-05-31T21:35:39.274 INFO:tasks.rgw_multisite_tests: "marker": "", 2023-05-31T21:35:39.275 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-05-31T21:35:39.275 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-05-31T21:35:39.275 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-05-31T21:35:39.275 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2023-05-31T21:35:39.276 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-05-31T21:35:39.276 INFO:tasks.rgw_multisite_tests: } 2023-05-31T21:35:39.276 INFO:tasks.rgw_multisite_tests: }, 2023-05-31T21:35:39.276 INFO:tasks.rgw_multisite_tests: { 2023-05-31T21:35:39.277 INFO:tasks.rgw_multisite_tests: "key": 2, 2023-05-31T21:35:39.277 INFO:tasks.rgw_multisite_tests: "val": { 2023-05-31T21:35:39.277 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-05-31T21:35:39.277 INFO:tasks.rgw_multisite_tests: "marker": "1_1685562970.274572_93.1", 2023-05-31T21:35:39.278 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-05-31T21:35:39.278 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-05-31T21:35:39.278 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-05-31T21:35:39.278 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-05-31T19:56:10.274572Z", 2023-05-31T21:35:39.278 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-05-31T21:35:39.279 INFO:tasks.rgw_multisite_tests: } 2023-05-31T21:35:39.279 INFO:tasks.rgw_multisite_tests: }, 2023-05-31T21:35:39.279 INFO:tasks.rgw_multisite_tests: { 2023-05-31T21:35:39.279 INFO:tasks.rgw_multisite_tests: "key": 3, 2023-05-31T21:35:39.280 INFO:tasks.rgw_multisite_tests: "val": { 2023-05-31T21:35:39.280 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-05-31T21:35:39.280 INFO:tasks.rgw_multisite_tests: "marker": "1_1685563798.817516_251.1", 2023-05-31T21:35:39.280 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-05-31T21:35:39.281 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-05-31T21:35:39.281 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-05-31T21:35:39.281 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-05-31T20:09:58.817516Z", 2023-05-31T21:35:39.281 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-05-31T21:35:39.281 INFO:tasks.rgw_multisite_tests: } 2023-05-31T21:35:39.282 INFO:tasks.rgw_multisite_tests: } 2023-05-31T21:35:39.282 INFO:tasks.rgw_multisite_tests: ] 2023-05-31T21:35:39.282 INFO:tasks.rgw_multisite_tests: }, 2023-05-31T21:35:39.282 INFO:tasks.rgw_multisite_tests: "full_sync": { 2023-05-31T21:35:39.283 INFO:tasks.rgw_multisite_tests: "total": 1, 2023-05-31T21:35:39.283 INFO:tasks.rgw_multisite_tests: "complete": 1 2023-05-31T21:35:39.283 INFO:tasks.rgw_multisite_tests: }, 2023-05-31T21:35:39.283 INFO:tasks.rgw_multisite_tests: "current_time": "2023-05-31T20:13:28Z" 2023-05-31T21:35:39.284 INFO:tasks.rgw_multisite_tests:} 2023-05-31T21:35:39.284 INFO:tasks.rgw_multisite_tests: 2023-05-31T21:35:39.284 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1685562631.516089_59.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-05-31T19:50:31.516089Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1685562970.274572_93.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-05-31T19:56:10.274572Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1685563798.817516_251.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-05-31T20:09:58.817516Z', 'realm_epoch': 2}}] 2023-05-31T21:35:39.284 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1685562631.516089_59.1', 1: '', 2: '1_1685562970.274572_93.1', 3: '1_1685563802.468126_296.1'} 2023-05-31T21:35:39.284 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1685562631.516089_59.1', 1: '', 2: '1_1685562970.274572_93.1', 3: '1_1685563798.817516_251.1'} 2023-05-31T21:35:39.285 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone a2 behind master: shard=3 master=1_1685563802.468126_296.1 target=1_1685563798.817516_251.1 2023-05-31T21:35:39.285 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-31T21:35:39.285 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-31T21:35:39.285 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-05-31T21:35:39.286 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: ignoring --setuser ceph since I am not root 2023-05-31T21:35:39.286 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-05-31T21:35:39.286 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: 2023-05-31T20:13:33.363+0000 7f06758be780 20 RGW-SYNC:meta: read sync status 2023-05-31T21:35:39.286 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: 2023-05-31T20:13:33.364+0000 7f06758be780 20 RGW-SYNC:meta: read sync status 2023-05-31T21:35:39.287 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2023-05-31T21:35:39.287 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "sync_status": { 2023-05-31T21:35:39.287 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "info": { 2023-05-31T21:35:39.287 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "status": "sync", 2023-05-31T21:35:39.288 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "num_shards": 4, 2023-05-31T21:35:39.288 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "period": "2e6123d2-d695-4350-9a01-61a455bc782b", 2023-05-31T21:35:39.288 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "realm_epoch": 2 2023-05-31T21:35:39.288 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2023-05-31T21:35:39.288 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "markers": [ 2023-05-31T21:35:39.289 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2023-05-31T21:35:39.289 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "key": 0, 2023-05-31T21:35:39.289 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "val": { 2023-05-31T21:35:39.289 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "state": 1, 2023-05-31T21:35:39.290 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "marker": "1_1685562631.516089_59.1", 2023-05-31T21:35:39.290 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "next_step_marker": "", 2023-05-31T21:35:39.290 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "total_entries": 1, 2023-05-31T21:35:39.290 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "pos": 0, 2023-05-31T21:35:39.291 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "timestamp": "2023-05-31T19:50:31.516089Z", 2023-05-31T21:35:39.291 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "realm_epoch": 2 2023-05-31T21:35:39.291 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2023-05-31T21:35:39.291 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2023-05-31T21:35:39.292 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2023-05-31T21:35:39.292 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "key": 1, 2023-05-31T21:35:39.292 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "val": { 2023-05-31T21:35:39.292 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "state": 1, 2023-05-31T21:35:39.292 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "marker": "", 2023-05-31T21:35:39.293 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "next_step_marker": "", 2023-05-31T21:35:39.293 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "total_entries": 0, 2023-05-31T21:35:39.293 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "pos": 0, 2023-05-31T21:35:39.293 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "timestamp": "0.000000", 2023-05-31T21:35:39.294 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "realm_epoch": 2 2023-05-31T21:35:39.294 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2023-05-31T21:35:39.294 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2023-05-31T21:35:39.294 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2023-05-31T21:35:39.294 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "key": 2, 2023-05-31T21:35:39.295 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "val": { 2023-05-31T21:35:39.295 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "state": 1, 2023-05-31T21:35:39.295 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "marker": "1_1685562970.274572_93.1", 2023-05-31T21:35:39.295 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "next_step_marker": "", 2023-05-31T21:35:39.296 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "total_entries": 0, 2023-05-31T21:35:39.296 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "pos": 0, 2023-05-31T21:35:39.296 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "timestamp": "2023-05-31T19:56:10.274572Z", 2023-05-31T21:35:39.296 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "realm_epoch": 2 2023-05-31T21:35:39.297 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2023-05-31T21:35:39.297 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2023-05-31T21:35:39.297 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2023-05-31T21:35:39.297 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "key": 3, 2023-05-31T21:35:39.298 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "val": { 2023-05-31T21:35:39.298 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "state": 1, 2023-05-31T21:35:39.298 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "marker": "1_1685563798.970303_257.1", 2023-05-31T21:35:39.298 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "next_step_marker": "", 2023-05-31T21:35:39.298 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "total_entries": 0, 2023-05-31T21:35:39.299 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "pos": 0, 2023-05-31T21:35:39.299 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "timestamp": "2023-05-31T20:09:58.970303Z", 2023-05-31T21:35:39.299 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "realm_epoch": 2 2023-05-31T21:35:39.299 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2023-05-31T21:35:39.300 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2023-05-31T21:35:39.300 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: ] 2023-05-31T21:35:39.300 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2023-05-31T21:35:39.300 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "full_sync": { 2023-05-31T21:35:39.300 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "total": 1, 2023-05-31T21:35:39.301 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "complete": 1 2023-05-31T21:35:39.301 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2023-05-31T21:35:39.301 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "current_time": "2023-05-31T20:13:33Z" 2023-05-31T21:35:39.301 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2023-05-31T21:35:39.302 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2023-05-31T21:35:39.302 INFO:tasks.rgw_multisite_tests: "sync_status": { 2023-05-31T21:35:39.302 INFO:tasks.rgw_multisite_tests: "info": { 2023-05-31T21:35:39.302 INFO:tasks.rgw_multisite_tests: "status": "sync", 2023-05-31T21:35:39.303 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2023-05-31T21:35:39.303 INFO:tasks.rgw_multisite_tests: "period": "2e6123d2-d695-4350-9a01-61a455bc782b", 2023-05-31T21:35:39.303 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-05-31T21:35:39.303 INFO:tasks.rgw_multisite_tests: }, 2023-05-31T21:35:39.303 INFO:tasks.rgw_multisite_tests: "markers": [ 2023-05-31T21:35:39.304 INFO:tasks.rgw_multisite_tests: { 2023-05-31T21:35:39.304 INFO:tasks.rgw_multisite_tests: "key": 0, 2023-05-31T21:35:39.304 INFO:tasks.rgw_multisite_tests: "val": { 2023-05-31T21:35:39.304 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-05-31T21:35:39.305 INFO:tasks.rgw_multisite_tests: "marker": "1_1685562631.516089_59.1", 2023-05-31T21:35:39.305 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-05-31T21:35:39.305 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2023-05-31T21:35:39.305 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-05-31T21:35:39.306 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-05-31T19:50:31.516089Z", 2023-05-31T21:35:39.306 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-05-31T21:35:39.306 INFO:tasks.rgw_multisite_tests: } 2023-05-31T21:35:39.306 INFO:tasks.rgw_multisite_tests: }, 2023-05-31T21:35:39.306 INFO:tasks.rgw_multisite_tests: { 2023-05-31T21:35:39.307 INFO:tasks.rgw_multisite_tests: "key": 1, 2023-05-31T21:35:39.307 INFO:tasks.rgw_multisite_tests: "val": { 2023-05-31T21:35:39.307 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-05-31T21:35:39.307 INFO:tasks.rgw_multisite_tests: "marker": "", 2023-05-31T21:35:39.308 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-05-31T21:35:39.308 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-05-31T21:35:39.308 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-05-31T21:35:39.308 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2023-05-31T21:35:39.309 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-05-31T21:35:39.309 INFO:tasks.rgw_multisite_tests: } 2023-05-31T21:35:39.309 INFO:tasks.rgw_multisite_tests: }, 2023-05-31T21:35:39.309 INFO:tasks.rgw_multisite_tests: { 2023-05-31T21:35:39.310 INFO:tasks.rgw_multisite_tests: "key": 2, 2023-05-31T21:35:39.310 INFO:tasks.rgw_multisite_tests: "val": { 2023-05-31T21:35:39.310 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-05-31T21:35:39.310 INFO:tasks.rgw_multisite_tests: "marker": "1_1685562970.274572_93.1", 2023-05-31T21:35:39.310 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-05-31T21:35:39.311 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-05-31T21:35:39.311 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-05-31T21:35:39.311 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-05-31T19:56:10.274572Z", 2023-05-31T21:35:39.311 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-05-31T21:35:39.312 INFO:tasks.rgw_multisite_tests: } 2023-05-31T21:35:39.312 INFO:tasks.rgw_multisite_tests: }, 2023-05-31T21:35:39.312 INFO:tasks.rgw_multisite_tests: { 2023-05-31T21:35:39.312 INFO:tasks.rgw_multisite_tests: "key": 3, 2023-05-31T21:35:39.313 INFO:tasks.rgw_multisite_tests: "val": { 2023-05-31T21:35:39.313 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-05-31T21:35:39.313 INFO:tasks.rgw_multisite_tests: "marker": "1_1685563798.970303_257.1", 2023-05-31T21:35:39.313 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-05-31T21:35:39.313 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-05-31T21:35:39.314 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-05-31T21:35:39.314 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-05-31T20:09:58.970303Z", 2023-05-31T21:35:39.314 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-05-31T21:35:39.314 INFO:tasks.rgw_multisite_tests: } 2023-05-31T21:35:39.315 INFO:tasks.rgw_multisite_tests: } 2023-05-31T21:35:39.315 INFO:tasks.rgw_multisite_tests: ] 2023-05-31T21:35:39.315 INFO:tasks.rgw_multisite_tests: }, 2023-05-31T21:35:39.315 INFO:tasks.rgw_multisite_tests: "full_sync": { 2023-05-31T21:35:39.316 INFO:tasks.rgw_multisite_tests: "total": 1, 2023-05-31T21:35:39.316 INFO:tasks.rgw_multisite_tests: "complete": 1 2023-05-31T21:35:39.316 INFO:tasks.rgw_multisite_tests: }, 2023-05-31T21:35:39.316 INFO:tasks.rgw_multisite_tests: "current_time": "2023-05-31T20:13:33Z" 2023-05-31T21:35:39.316 INFO:tasks.rgw_multisite_tests:} 2023-05-31T21:35:39.317 INFO:tasks.rgw_multisite_tests: 2023-05-31T21:35:39.317 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1685562631.516089_59.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-05-31T19:50:31.516089Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1685562970.274572_93.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-05-31T19:56:10.274572Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1685563798.970303_257.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-05-31T20:09:58.970303Z', 'realm_epoch': 2}}] 2023-05-31T21:35:39.317 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1685562631.516089_59.1', 1: '', 2: '1_1685562970.274572_93.1', 3: '1_1685563802.468126_296.1'} 2023-05-31T21:35:39.317 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1685562631.516089_59.1', 1: '', 2: '1_1685562970.274572_93.1', 3: '1_1685563798.970303_257.1'} 2023-05-31T21:35:39.318 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone a2 behind master: shard=3 master=1_1685563802.468126_296.1 target=1_1685563798.970303_257.1 2023-05-31T21:35:39.318 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-31T21:35:39.318 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-31T21:35:39.318 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-05-31T21:35:39.319 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: ignoring --setuser ceph since I am not root 2023-05-31T21:35:39.319 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-05-31T21:35:39.319 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: 2023-05-31T20:13:38.532+0000 7fb7258f9780 20 RGW-SYNC:meta: read sync status 2023-05-31T21:35:39.319 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: 2023-05-31T20:13:38.532+0000 7fb7258f9780 20 RGW-SYNC:meta: read sync status 2023-05-31T21:35:39.319 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2023-05-31T21:35:39.320 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "sync_status": { 2023-05-31T21:35:39.320 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "info": { 2023-05-31T21:35:39.320 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "status": "sync", 2023-05-31T21:35:39.320 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "num_shards": 4, 2023-05-31T21:35:39.321 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "period": "2e6123d2-d695-4350-9a01-61a455bc782b", 2023-05-31T21:35:39.321 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "realm_epoch": 2 2023-05-31T21:35:39.321 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2023-05-31T21:35:39.321 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "markers": [ 2023-05-31T21:35:39.322 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2023-05-31T21:35:39.322 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "key": 0, 2023-05-31T21:35:39.322 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "val": { 2023-05-31T21:35:39.322 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "state": 1, 2023-05-31T21:35:39.322 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "marker": "1_1685562631.516089_59.1", 2023-05-31T21:35:39.323 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "next_step_marker": "", 2023-05-31T21:35:39.323 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "total_entries": 1, 2023-05-31T21:35:39.323 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "pos": 0, 2023-05-31T21:35:39.323 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "timestamp": "2023-05-31T19:50:31.516089Z", 2023-05-31T21:35:39.324 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "realm_epoch": 2 2023-05-31T21:35:39.324 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2023-05-31T21:35:39.324 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2023-05-31T21:35:39.324 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2023-05-31T21:35:39.325 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "key": 1, 2023-05-31T21:35:39.325 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "val": { 2023-05-31T21:35:39.325 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "state": 1, 2023-05-31T21:35:39.325 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "marker": "", 2023-05-31T21:35:39.325 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "next_step_marker": "", 2023-05-31T21:35:39.326 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "total_entries": 0, 2023-05-31T21:35:39.326 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "pos": 0, 2023-05-31T21:35:39.326 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "timestamp": "0.000000", 2023-05-31T21:35:39.326 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "realm_epoch": 2 2023-05-31T21:35:39.327 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2023-05-31T21:35:39.327 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2023-05-31T21:35:39.327 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2023-05-31T21:35:39.327 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "key": 2, 2023-05-31T21:35:39.328 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "val": { 2023-05-31T21:35:39.328 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "state": 1, 2023-05-31T21:35:39.328 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "marker": "1_1685562970.274572_93.1", 2023-05-31T21:35:39.328 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "next_step_marker": "", 2023-05-31T21:35:39.328 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "total_entries": 0, 2023-05-31T21:35:39.329 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "pos": 0, 2023-05-31T21:35:39.329 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "timestamp": "2023-05-31T19:56:10.274572Z", 2023-05-31T21:35:39.329 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "realm_epoch": 2 2023-05-31T21:35:39.329 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2023-05-31T21:35:39.330 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2023-05-31T21:35:39.330 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2023-05-31T21:35:39.330 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "key": 3, 2023-05-31T21:35:39.330 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "val": { 2023-05-31T21:35:39.330 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "state": 1, 2023-05-31T21:35:39.331 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "marker": "1_1685563798.885419_253.1", 2023-05-31T21:35:39.331 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "next_step_marker": "", 2023-05-31T21:35:39.331 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "total_entries": 0, 2023-05-31T21:35:39.331 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "pos": 0, 2023-05-31T21:35:39.332 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "timestamp": "2023-05-31T20:09:58.885419Z", 2023-05-31T21:35:39.332 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "realm_epoch": 2 2023-05-31T21:35:39.332 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2023-05-31T21:35:39.332 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2023-05-31T21:35:39.333 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: ] 2023-05-31T21:35:39.333 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2023-05-31T21:35:39.333 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "full_sync": { 2023-05-31T21:35:39.333 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "total": 1, 2023-05-31T21:35:39.333 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "complete": 1 2023-05-31T21:35:39.334 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2023-05-31T21:35:39.334 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "current_time": "2023-05-31T20:13:38Z" 2023-05-31T21:35:39.334 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2023-05-31T21:35:39.334 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2023-05-31T21:35:39.335 INFO:tasks.rgw_multisite_tests: "sync_status": { 2023-05-31T21:35:39.335 INFO:tasks.rgw_multisite_tests: "info": { 2023-05-31T21:35:39.335 INFO:tasks.rgw_multisite_tests: "status": "sync", 2023-05-31T21:35:39.335 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2023-05-31T21:35:39.336 INFO:tasks.rgw_multisite_tests: "period": "2e6123d2-d695-4350-9a01-61a455bc782b", 2023-05-31T21:35:39.336 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-05-31T21:35:39.336 INFO:tasks.rgw_multisite_tests: }, 2023-05-31T21:35:39.336 INFO:tasks.rgw_multisite_tests: "markers": [ 2023-05-31T21:35:39.337 INFO:tasks.rgw_multisite_tests: { 2023-05-31T21:35:39.337 INFO:tasks.rgw_multisite_tests: "key": 0, 2023-05-31T21:35:39.337 INFO:tasks.rgw_multisite_tests: "val": { 2023-05-31T21:35:39.337 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-05-31T21:35:39.337 INFO:tasks.rgw_multisite_tests: "marker": "1_1685562631.516089_59.1", 2023-05-31T21:35:39.338 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-05-31T21:35:39.338 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2023-05-31T21:35:39.338 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-05-31T21:35:39.338 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-05-31T19:50:31.516089Z", 2023-05-31T21:35:39.339 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-05-31T21:35:39.339 INFO:tasks.rgw_multisite_tests: } 2023-05-31T21:35:39.339 INFO:tasks.rgw_multisite_tests: }, 2023-05-31T21:35:39.339 INFO:tasks.rgw_multisite_tests: { 2023-05-31T21:35:39.339 INFO:tasks.rgw_multisite_tests: "key": 1, 2023-05-31T21:35:39.340 INFO:tasks.rgw_multisite_tests: "val": { 2023-05-31T21:35:39.340 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-05-31T21:35:39.340 INFO:tasks.rgw_multisite_tests: "marker": "", 2023-05-31T21:35:39.340 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-05-31T21:35:39.341 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-05-31T21:35:39.341 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-05-31T21:35:39.341 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2023-05-31T21:35:39.341 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-05-31T21:35:39.342 INFO:tasks.rgw_multisite_tests: } 2023-05-31T21:35:39.342 INFO:tasks.rgw_multisite_tests: }, 2023-05-31T21:35:39.342 INFO:tasks.rgw_multisite_tests: { 2023-05-31T21:35:39.342 INFO:tasks.rgw_multisite_tests: "key": 2, 2023-05-31T21:35:39.342 INFO:tasks.rgw_multisite_tests: "val": { 2023-05-31T21:35:39.343 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-05-31T21:35:39.343 INFO:tasks.rgw_multisite_tests: "marker": "1_1685562970.274572_93.1", 2023-05-31T21:35:39.343 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-05-31T21:35:39.343 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-05-31T21:35:39.344 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-05-31T21:35:39.344 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-05-31T19:56:10.274572Z", 2023-05-31T21:35:39.344 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-05-31T21:35:39.344 INFO:tasks.rgw_multisite_tests: } 2023-05-31T21:35:39.344 INFO:tasks.rgw_multisite_tests: }, 2023-05-31T21:35:39.345 INFO:tasks.rgw_multisite_tests: { 2023-05-31T21:35:39.345 INFO:tasks.rgw_multisite_tests: "key": 3, 2023-05-31T21:35:39.345 INFO:tasks.rgw_multisite_tests: "val": { 2023-05-31T21:35:39.345 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-05-31T21:35:39.346 INFO:tasks.rgw_multisite_tests: "marker": "1_1685563798.885419_253.1", 2023-05-31T21:35:39.346 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-05-31T21:35:39.346 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-05-31T21:35:39.346 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-05-31T21:35:39.347 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-05-31T20:09:58.885419Z", 2023-05-31T21:35:39.347 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-05-31T21:35:39.347 INFO:tasks.rgw_multisite_tests: } 2023-05-31T21:35:39.347 INFO:tasks.rgw_multisite_tests: } 2023-05-31T21:35:39.347 INFO:tasks.rgw_multisite_tests: ] 2023-05-31T21:35:39.348 INFO:tasks.rgw_multisite_tests: }, 2023-05-31T21:35:39.348 INFO:tasks.rgw_multisite_tests: "full_sync": { 2023-05-31T21:35:39.348 INFO:tasks.rgw_multisite_tests: "total": 1, 2023-05-31T21:35:39.348 INFO:tasks.rgw_multisite_tests: "complete": 1 2023-05-31T21:35:39.349 INFO:tasks.rgw_multisite_tests: }, 2023-05-31T21:35:39.349 INFO:tasks.rgw_multisite_tests: "current_time": "2023-05-31T20:13:38Z" 2023-05-31T21:35:39.349 INFO:tasks.rgw_multisite_tests:} 2023-05-31T21:35:39.349 INFO:tasks.rgw_multisite_tests: 2023-05-31T21:35:39.350 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1685562631.516089_59.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-05-31T19:50:31.516089Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1685562970.274572_93.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-05-31T19:56:10.274572Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1685563798.885419_253.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-05-31T20:09:58.885419Z', 'realm_epoch': 2}}] 2023-05-31T21:35:39.350 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1685562631.516089_59.1', 1: '', 2: '1_1685562970.274572_93.1', 3: '1_1685563802.468126_296.1'} 2023-05-31T21:35:39.350 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1685562631.516089_59.1', 1: '', 2: '1_1685562970.274572_93.1', 3: '1_1685563798.885419_253.1'} 2023-05-31T21:35:39.350 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone a2 behind master: shard=3 master=1_1685563802.468126_296.1 target=1_1685563798.885419_253.1 2023-05-31T21:35:39.350 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-31T21:35:39.351 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-31T21:35:39.351 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-05-31T21:35:39.351 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: ignoring --setuser ceph since I am not root 2023-05-31T21:35:39.351 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-05-31T21:35:39.352 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: 2023-05-31T20:13:43.676+0000 7fd7735c1780 20 RGW-SYNC:meta: read sync status 2023-05-31T21:35:39.352 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: 2023-05-31T20:13:43.676+0000 7fd7735c1780 20 RGW-SYNC:meta: read sync status 2023-05-31T21:35:39.352 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2023-05-31T21:35:39.352 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "sync_status": { 2023-05-31T21:35:39.353 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "info": { 2023-05-31T21:35:39.353 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "status": "sync", 2023-05-31T21:35:39.353 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "num_shards": 4, 2023-05-31T21:35:39.353 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "period": "2e6123d2-d695-4350-9a01-61a455bc782b", 2023-05-31T21:35:39.353 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "realm_epoch": 2 2023-05-31T21:35:39.354 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2023-05-31T21:35:39.354 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "markers": [ 2023-05-31T21:35:39.354 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2023-05-31T21:35:39.354 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "key": 0, 2023-05-31T21:35:39.355 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "val": { 2023-05-31T21:35:39.355 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "state": 1, 2023-05-31T21:35:39.355 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "marker": "1_1685562631.516089_59.1", 2023-05-31T21:35:39.355 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "next_step_marker": "", 2023-05-31T21:35:39.356 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "total_entries": 1, 2023-05-31T21:35:39.356 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "pos": 0, 2023-05-31T21:35:39.356 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "timestamp": "2023-05-31T19:50:31.516089Z", 2023-05-31T21:35:39.356 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "realm_epoch": 2 2023-05-31T21:35:39.356 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2023-05-31T21:35:39.357 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2023-05-31T21:35:39.357 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2023-05-31T21:35:39.357 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "key": 1, 2023-05-31T21:35:39.357 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "val": { 2023-05-31T21:35:39.358 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "state": 1, 2023-05-31T21:35:39.358 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "marker": "", 2023-05-31T21:35:39.358 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "next_step_marker": "", 2023-05-31T21:35:39.358 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "total_entries": 0, 2023-05-31T21:35:39.358 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "pos": 0, 2023-05-31T21:35:39.359 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "timestamp": "0.000000", 2023-05-31T21:35:39.359 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "realm_epoch": 2 2023-05-31T21:35:39.359 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2023-05-31T21:35:39.359 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2023-05-31T21:35:39.360 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2023-05-31T21:35:39.360 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "key": 2, 2023-05-31T21:35:39.360 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "val": { 2023-05-31T21:35:39.360 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "state": 1, 2023-05-31T21:35:39.361 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "marker": "1_1685562970.274572_93.1", 2023-05-31T21:35:39.361 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "next_step_marker": "", 2023-05-31T21:35:39.361 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "total_entries": 0, 2023-05-31T21:35:39.361 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "pos": 0, 2023-05-31T21:35:39.361 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "timestamp": "2023-05-31T19:56:10.274572Z", 2023-05-31T21:35:39.362 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "realm_epoch": 2 2023-05-31T21:35:39.362 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2023-05-31T21:35:39.362 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2023-05-31T21:35:39.362 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2023-05-31T21:35:39.363 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "key": 3, 2023-05-31T21:35:39.363 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "val": { 2023-05-31T21:35:39.363 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "state": 1, 2023-05-31T21:35:39.363 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "marker": "1_1685563798.807987_249.1", 2023-05-31T21:35:39.363 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "next_step_marker": "", 2023-05-31T21:35:39.364 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "total_entries": 0, 2023-05-31T21:35:39.364 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "pos": 0, 2023-05-31T21:35:39.364 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "timestamp": "2023-05-31T20:09:58.807987Z", 2023-05-31T21:35:39.364 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "realm_epoch": 2 2023-05-31T21:35:39.365 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2023-05-31T21:35:39.365 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2023-05-31T21:35:39.365 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: ] 2023-05-31T21:35:39.365 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2023-05-31T21:35:39.366 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "full_sync": { 2023-05-31T21:35:39.366 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "total": 1, 2023-05-31T21:35:39.366 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "complete": 1 2023-05-31T21:35:39.366 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2023-05-31T21:35:39.367 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "current_time": "2023-05-31T20:13:43Z" 2023-05-31T21:35:39.367 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2023-05-31T21:35:39.367 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2023-05-31T21:35:39.367 INFO:tasks.rgw_multisite_tests: "sync_status": { 2023-05-31T21:35:39.367 INFO:tasks.rgw_multisite_tests: "info": { 2023-05-31T21:35:39.368 INFO:tasks.rgw_multisite_tests: "status": "sync", 2023-05-31T21:35:39.368 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2023-05-31T21:35:39.368 INFO:tasks.rgw_multisite_tests: "period": "2e6123d2-d695-4350-9a01-61a455bc782b", 2023-05-31T21:35:39.368 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-05-31T21:35:39.369 INFO:tasks.rgw_multisite_tests: }, 2023-05-31T21:35:39.369 INFO:tasks.rgw_multisite_tests: "markers": [ 2023-05-31T21:35:39.369 INFO:tasks.rgw_multisite_tests: { 2023-05-31T21:35:39.369 INFO:tasks.rgw_multisite_tests: "key": 0, 2023-05-31T21:35:39.370 INFO:tasks.rgw_multisite_tests: "val": { 2023-05-31T21:35:39.370 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-05-31T21:35:39.370 INFO:tasks.rgw_multisite_tests: "marker": "1_1685562631.516089_59.1", 2023-05-31T21:35:39.370 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-05-31T21:35:39.370 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2023-05-31T21:35:39.371 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-05-31T21:35:39.371 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-05-31T19:50:31.516089Z", 2023-05-31T21:35:39.371 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-05-31T21:35:39.371 INFO:tasks.rgw_multisite_tests: } 2023-05-31T21:35:39.372 INFO:tasks.rgw_multisite_tests: }, 2023-05-31T21:35:39.372 INFO:tasks.rgw_multisite_tests: { 2023-05-31T21:35:39.372 INFO:tasks.rgw_multisite_tests: "key": 1, 2023-05-31T21:35:39.372 INFO:tasks.rgw_multisite_tests: "val": { 2023-05-31T21:35:39.372 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-05-31T21:35:39.373 INFO:tasks.rgw_multisite_tests: "marker": "", 2023-05-31T21:35:39.373 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-05-31T21:35:39.373 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-05-31T21:35:39.373 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-05-31T21:35:39.374 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2023-05-31T21:35:39.374 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-05-31T21:35:39.374 INFO:tasks.rgw_multisite_tests: } 2023-05-31T21:35:39.374 INFO:tasks.rgw_multisite_tests: }, 2023-05-31T21:35:39.375 INFO:tasks.rgw_multisite_tests: { 2023-05-31T21:35:39.375 INFO:tasks.rgw_multisite_tests: "key": 2, 2023-05-31T21:35:39.375 INFO:tasks.rgw_multisite_tests: "val": { 2023-05-31T21:35:39.375 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-05-31T21:35:39.375 INFO:tasks.rgw_multisite_tests: "marker": "1_1685562970.274572_93.1", 2023-05-31T21:35:39.376 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-05-31T21:35:39.376 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-05-31T21:35:39.376 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-05-31T21:35:39.376 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-05-31T19:56:10.274572Z", 2023-05-31T21:35:39.377 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-05-31T21:35:39.377 INFO:tasks.rgw_multisite_tests: } 2023-05-31T21:35:39.377 INFO:tasks.rgw_multisite_tests: }, 2023-05-31T21:35:39.377 INFO:tasks.rgw_multisite_tests: { 2023-05-31T21:35:39.378 INFO:tasks.rgw_multisite_tests: "key": 3, 2023-05-31T21:35:39.378 INFO:tasks.rgw_multisite_tests: "val": { 2023-05-31T21:35:39.378 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-05-31T21:35:39.378 INFO:tasks.rgw_multisite_tests: "marker": "1_1685563798.807987_249.1", 2023-05-31T21:35:39.378 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-05-31T21:35:39.379 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-05-31T21:35:39.379 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-05-31T21:35:39.379 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-05-31T20:09:58.807987Z", 2023-05-31T21:35:39.379 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-05-31T21:35:39.380 INFO:tasks.rgw_multisite_tests: } 2023-05-31T21:35:39.380 INFO:tasks.rgw_multisite_tests: } 2023-05-31T21:35:39.380 INFO:tasks.rgw_multisite_tests: ] 2023-05-31T21:35:39.380 INFO:tasks.rgw_multisite_tests: }, 2023-05-31T21:35:39.381 INFO:tasks.rgw_multisite_tests: "full_sync": { 2023-05-31T21:35:39.381 INFO:tasks.rgw_multisite_tests: "total": 1, 2023-05-31T21:35:39.381 INFO:tasks.rgw_multisite_tests: "complete": 1 2023-05-31T21:35:39.381 INFO:tasks.rgw_multisite_tests: }, 2023-05-31T21:35:39.381 INFO:tasks.rgw_multisite_tests: "current_time": "2023-05-31T20:13:43Z" 2023-05-31T21:35:39.382 INFO:tasks.rgw_multisite_tests:} 2023-05-31T21:35:39.382 INFO:tasks.rgw_multisite_tests: 2023-05-31T21:35:39.382 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1685562631.516089_59.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-05-31T19:50:31.516089Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1685562970.274572_93.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-05-31T19:56:10.274572Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1685563798.807987_249.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-05-31T20:09:58.807987Z', 'realm_epoch': 2}}] 2023-05-31T21:35:39.382 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1685562631.516089_59.1', 1: '', 2: '1_1685562970.274572_93.1', 3: '1_1685563802.468126_296.1'} 2023-05-31T21:35:39.383 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1685562631.516089_59.1', 1: '', 2: '1_1685562970.274572_93.1', 3: '1_1685563798.807987_249.1'} 2023-05-31T21:35:39.383 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone a2 behind master: shard=3 master=1_1685563802.468126_296.1 target=1_1685563798.807987_249.1 2023-05-31T21:35:39.383 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-31T21:35:39.383 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-31T21:35:39.384 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-05-31T21:35:39.384 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: ignoring --setuser ceph since I am not root 2023-05-31T21:35:39.384 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-05-31T21:35:39.384 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: 2023-05-31T20:13:48.851+0000 7f0917f83780 20 RGW-SYNC:meta: read sync status 2023-05-31T21:35:39.385 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: 2023-05-31T20:13:48.853+0000 7f0917f83780 20 RGW-SYNC:meta: read sync status 2023-05-31T21:35:39.385 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2023-05-31T21:35:39.385 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "sync_status": { 2023-05-31T21:35:39.385 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "info": { 2023-05-31T21:35:39.385 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "status": "sync", 2023-05-31T21:35:39.386 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "num_shards": 4, 2023-05-31T21:35:39.386 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "period": "2e6123d2-d695-4350-9a01-61a455bc782b", 2023-05-31T21:35:39.386 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "realm_epoch": 2 2023-05-31T21:35:39.386 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2023-05-31T21:35:39.387 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "markers": [ 2023-05-31T21:35:39.387 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2023-05-31T21:35:39.387 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "key": 0, 2023-05-31T21:35:39.387 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "val": { 2023-05-31T21:35:39.387 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "state": 1, 2023-05-31T21:35:39.388 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "marker": "1_1685562631.516089_59.1", 2023-05-31T21:35:39.388 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "next_step_marker": "", 2023-05-31T21:35:39.388 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "total_entries": 1, 2023-05-31T21:35:39.388 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "pos": 0, 2023-05-31T21:35:39.389 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "timestamp": "2023-05-31T19:50:31.516089Z", 2023-05-31T21:35:39.389 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "realm_epoch": 2 2023-05-31T21:35:39.389 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2023-05-31T21:35:39.389 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2023-05-31T21:35:39.390 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2023-05-31T21:35:39.390 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "key": 1, 2023-05-31T21:35:39.390 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "val": { 2023-05-31T21:35:39.390 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "state": 1, 2023-05-31T21:35:39.391 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "marker": "", 2023-05-31T21:35:39.391 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "next_step_marker": "", 2023-05-31T21:35:39.391 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "total_entries": 0, 2023-05-31T21:35:39.391 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "pos": 0, 2023-05-31T21:35:39.391 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "timestamp": "0.000000", 2023-05-31T21:35:39.392 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "realm_epoch": 2 2023-05-31T21:35:39.392 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2023-05-31T21:35:39.392 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2023-05-31T21:35:39.392 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2023-05-31T21:35:39.393 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "key": 2, 2023-05-31T21:35:39.393 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "val": { 2023-05-31T21:35:39.393 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "state": 1, 2023-05-31T21:35:39.393 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "marker": "1_1685562970.274572_93.1", 2023-05-31T21:35:39.393 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "next_step_marker": "", 2023-05-31T21:35:39.394 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "total_entries": 0, 2023-05-31T21:35:39.394 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "pos": 0, 2023-05-31T21:35:39.394 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "timestamp": "2023-05-31T19:56:10.274572Z", 2023-05-31T21:35:39.394 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "realm_epoch": 2 2023-05-31T21:35:39.395 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2023-05-31T21:35:39.395 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2023-05-31T21:35:39.395 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2023-05-31T21:35:39.395 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "key": 3, 2023-05-31T21:35:39.396 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "val": { 2023-05-31T21:35:39.396 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "state": 1, 2023-05-31T21:35:39.396 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "marker": "1_1685563798.807987_249.1", 2023-05-31T21:35:39.396 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "next_step_marker": "", 2023-05-31T21:35:39.397 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "total_entries": 0, 2023-05-31T21:35:39.397 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "pos": 0, 2023-05-31T21:35:39.397 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "timestamp": "2023-05-31T20:09:58.807987Z", 2023-05-31T21:35:39.397 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "realm_epoch": 2 2023-05-31T21:35:39.397 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2023-05-31T21:35:39.398 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2023-05-31T21:35:39.398 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: ] 2023-05-31T21:35:39.398 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2023-05-31T21:35:39.398 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "full_sync": { 2023-05-31T21:35:39.399 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "total": 1, 2023-05-31T21:35:39.399 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "complete": 1 2023-05-31T21:35:39.399 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2023-05-31T21:35:39.399 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "current_time": "2023-05-31T20:13:48Z" 2023-05-31T21:35:39.399 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2023-05-31T21:35:39.400 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2023-05-31T21:35:39.400 INFO:tasks.rgw_multisite_tests: "sync_status": { 2023-05-31T21:35:39.400 INFO:tasks.rgw_multisite_tests: "info": { 2023-05-31T21:35:39.400 INFO:tasks.rgw_multisite_tests: "status": "sync", 2023-05-31T21:35:39.401 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2023-05-31T21:35:39.401 INFO:tasks.rgw_multisite_tests: "period": "2e6123d2-d695-4350-9a01-61a455bc782b", 2023-05-31T21:35:39.401 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-05-31T21:35:39.401 INFO:tasks.rgw_multisite_tests: }, 2023-05-31T21:35:39.402 INFO:tasks.rgw_multisite_tests: "markers": [ 2023-05-31T21:35:39.402 INFO:tasks.rgw_multisite_tests: { 2023-05-31T21:35:39.402 INFO:tasks.rgw_multisite_tests: "key": 0, 2023-05-31T21:35:39.402 INFO:tasks.rgw_multisite_tests: "val": { 2023-05-31T21:35:39.402 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-05-31T21:35:39.403 INFO:tasks.rgw_multisite_tests: "marker": "1_1685562631.516089_59.1", 2023-05-31T21:35:39.403 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-05-31T21:35:39.403 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2023-05-31T21:35:39.403 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-05-31T21:35:39.404 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-05-31T19:50:31.516089Z", 2023-05-31T21:35:39.404 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-05-31T21:35:39.404 INFO:tasks.rgw_multisite_tests: } 2023-05-31T21:35:39.404 INFO:tasks.rgw_multisite_tests: }, 2023-05-31T21:35:39.405 INFO:tasks.rgw_multisite_tests: { 2023-05-31T21:35:39.405 INFO:tasks.rgw_multisite_tests: "key": 1, 2023-05-31T21:35:39.405 INFO:tasks.rgw_multisite_tests: "val": { 2023-05-31T21:35:39.405 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-05-31T21:35:39.405 INFO:tasks.rgw_multisite_tests: "marker": "", 2023-05-31T21:35:39.406 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-05-31T21:35:39.406 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-05-31T21:35:39.406 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-05-31T21:35:39.406 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2023-05-31T21:35:39.407 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-05-31T21:35:39.407 INFO:tasks.rgw_multisite_tests: } 2023-05-31T21:35:39.407 INFO:tasks.rgw_multisite_tests: }, 2023-05-31T21:35:39.407 INFO:tasks.rgw_multisite_tests: { 2023-05-31T21:35:39.407 INFO:tasks.rgw_multisite_tests: "key": 2, 2023-05-31T21:35:39.408 INFO:tasks.rgw_multisite_tests: "val": { 2023-05-31T21:35:39.408 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-05-31T21:35:39.408 INFO:tasks.rgw_multisite_tests: "marker": "1_1685562970.274572_93.1", 2023-05-31T21:35:39.408 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-05-31T21:35:39.409 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-05-31T21:35:39.409 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-05-31T21:35:39.409 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-05-31T19:56:10.274572Z", 2023-05-31T21:35:39.409 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-05-31T21:35:39.410 INFO:tasks.rgw_multisite_tests: } 2023-05-31T21:35:39.410 INFO:tasks.rgw_multisite_tests: }, 2023-05-31T21:35:39.410 INFO:tasks.rgw_multisite_tests: { 2023-05-31T21:35:39.410 INFO:tasks.rgw_multisite_tests: "key": 3, 2023-05-31T21:35:39.410 INFO:tasks.rgw_multisite_tests: "val": { 2023-05-31T21:35:39.411 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-05-31T21:35:39.411 INFO:tasks.rgw_multisite_tests: "marker": "1_1685563798.807987_249.1", 2023-05-31T21:35:39.411 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-05-31T21:35:39.411 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-05-31T21:35:39.412 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-05-31T21:35:39.412 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-05-31T20:09:58.807987Z", 2023-05-31T21:35:39.412 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-05-31T21:35:39.412 INFO:tasks.rgw_multisite_tests: } 2023-05-31T21:35:39.413 INFO:tasks.rgw_multisite_tests: } 2023-05-31T21:35:39.413 INFO:tasks.rgw_multisite_tests: ] 2023-05-31T21:35:39.413 INFO:tasks.rgw_multisite_tests: }, 2023-05-31T21:35:39.413 INFO:tasks.rgw_multisite_tests: "full_sync": { 2023-05-31T21:35:39.413 INFO:tasks.rgw_multisite_tests: "total": 1, 2023-05-31T21:35:39.414 INFO:tasks.rgw_multisite_tests: "complete": 1 2023-05-31T21:35:39.414 INFO:tasks.rgw_multisite_tests: }, 2023-05-31T21:35:39.414 INFO:tasks.rgw_multisite_tests: "current_time": "2023-05-31T20:13:48Z" 2023-05-31T21:35:39.414 INFO:tasks.rgw_multisite_tests:} 2023-05-31T21:35:39.415 INFO:tasks.rgw_multisite_tests: 2023-05-31T21:35:39.415 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1685562631.516089_59.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-05-31T19:50:31.516089Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1685562970.274572_93.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-05-31T19:56:10.274572Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1685563798.807987_249.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-05-31T20:09:58.807987Z', 'realm_epoch': 2}}] 2023-05-31T21:35:39.415 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1685562631.516089_59.1', 1: '', 2: '1_1685562970.274572_93.1', 3: '1_1685563802.468126_296.1'} 2023-05-31T21:35:39.415 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1685562631.516089_59.1', 1: '', 2: '1_1685562970.274572_93.1', 3: '1_1685563798.807987_249.1'} 2023-05-31T21:35:39.416 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone a2 behind master: shard=3 master=1_1685563802.468126_296.1 target=1_1685563798.807987_249.1 2023-05-31T21:35:39.416 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-31T21:35:39.416 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-31T21:35:39.416 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-05-31T21:35:39.417 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: ignoring --setuser ceph since I am not root 2023-05-31T21:35:39.417 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-05-31T21:35:39.417 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: 2023-05-31T20:13:54.016+0000 7fa090498780 20 RGW-SYNC:meta: read sync status 2023-05-31T21:35:39.417 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: 2023-05-31T20:13:54.017+0000 7fa090498780 20 RGW-SYNC:meta: read sync status 2023-05-31T21:35:39.417 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2023-05-31T21:35:39.418 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "sync_status": { 2023-05-31T21:35:39.418 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "info": { 2023-05-31T21:35:39.418 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "status": "sync", 2023-05-31T21:35:39.418 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "num_shards": 4, 2023-05-31T21:35:39.419 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "period": "2e6123d2-d695-4350-9a01-61a455bc782b", 2023-05-31T21:35:39.419 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "realm_epoch": 2 2023-05-31T21:35:39.419 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2023-05-31T21:35:39.419 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "markers": [ 2023-05-31T21:35:39.419 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2023-05-31T21:35:39.420 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "key": 0, 2023-05-31T21:35:39.420 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "val": { 2023-05-31T21:35:39.420 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "state": 1, 2023-05-31T21:35:39.420 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "marker": "1_1685562631.516089_59.1", 2023-05-31T21:35:39.421 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "next_step_marker": "", 2023-05-31T21:35:39.421 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "total_entries": 1, 2023-05-31T21:35:39.421 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "pos": 0, 2023-05-31T21:35:39.421 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "timestamp": "2023-05-31T19:50:31.516089Z", 2023-05-31T21:35:39.422 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "realm_epoch": 2 2023-05-31T21:35:39.422 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2023-05-31T21:35:39.422 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2023-05-31T21:35:39.422 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2023-05-31T21:35:39.423 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "key": 1, 2023-05-31T21:35:39.423 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "val": { 2023-05-31T21:35:39.423 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "state": 1, 2023-05-31T21:35:39.423 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "marker": "", 2023-05-31T21:35:39.423 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "next_step_marker": "", 2023-05-31T21:35:39.424 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "total_entries": 0, 2023-05-31T21:35:39.424 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "pos": 0, 2023-05-31T21:35:39.424 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "timestamp": "0.000000", 2023-05-31T21:35:39.424 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "realm_epoch": 2 2023-05-31T21:35:39.425 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2023-05-31T21:35:39.425 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2023-05-31T21:35:39.425 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2023-05-31T21:35:39.425 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "key": 2, 2023-05-31T21:35:39.426 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "val": { 2023-05-31T21:35:39.426 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "state": 1, 2023-05-31T21:35:39.426 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "marker": "1_1685562970.274572_93.1", 2023-05-31T21:35:39.426 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "next_step_marker": "", 2023-05-31T21:35:39.426 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "total_entries": 0, 2023-05-31T21:35:39.427 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "pos": 0, 2023-05-31T21:35:39.427 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "timestamp": "2023-05-31T19:56:10.274572Z", 2023-05-31T21:35:39.427 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "realm_epoch": 2 2023-05-31T21:35:39.427 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2023-05-31T21:35:39.428 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2023-05-31T21:35:39.428 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2023-05-31T21:35:39.428 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "key": 3, 2023-05-31T21:35:39.428 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "val": { 2023-05-31T21:35:39.429 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "state": 1, 2023-05-31T21:35:39.429 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "marker": "1_1685563798.970303_257.1", 2023-05-31T21:35:39.429 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "next_step_marker": "", 2023-05-31T21:35:39.429 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "total_entries": 0, 2023-05-31T21:35:39.430 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "pos": 0, 2023-05-31T21:35:39.430 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "timestamp": "2023-05-31T20:09:58.970303Z", 2023-05-31T21:35:39.430 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "realm_epoch": 2 2023-05-31T21:35:39.430 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2023-05-31T21:35:39.430 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2023-05-31T21:35:39.431 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: ] 2023-05-31T21:35:39.431 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2023-05-31T21:35:39.431 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "full_sync": { 2023-05-31T21:35:39.431 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "total": 1, 2023-05-31T21:35:39.432 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "complete": 1 2023-05-31T21:35:39.432 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2023-05-31T21:35:39.432 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "current_time": "2023-05-31T20:13:54Z" 2023-05-31T21:35:39.432 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2023-05-31T21:35:39.433 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2023-05-31T21:35:39.433 INFO:tasks.rgw_multisite_tests: "sync_status": { 2023-05-31T21:35:39.433 INFO:tasks.rgw_multisite_tests: "info": { 2023-05-31T21:35:39.433 INFO:tasks.rgw_multisite_tests: "status": "sync", 2023-05-31T21:35:39.434 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2023-05-31T21:35:39.434 INFO:tasks.rgw_multisite_tests: "period": "2e6123d2-d695-4350-9a01-61a455bc782b", 2023-05-31T21:35:39.434 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-05-31T21:35:39.434 INFO:tasks.rgw_multisite_tests: }, 2023-05-31T21:35:39.434 INFO:tasks.rgw_multisite_tests: "markers": [ 2023-05-31T21:35:39.435 INFO:tasks.rgw_multisite_tests: { 2023-05-31T21:35:39.435 INFO:tasks.rgw_multisite_tests: "key": 0, 2023-05-31T21:35:39.435 INFO:tasks.rgw_multisite_tests: "val": { 2023-05-31T21:35:39.435 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-05-31T21:35:39.436 INFO:tasks.rgw_multisite_tests: "marker": "1_1685562631.516089_59.1", 2023-05-31T21:35:39.436 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-05-31T21:35:39.436 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2023-05-31T21:35:39.436 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-05-31T21:35:39.437 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-05-31T19:50:31.516089Z", 2023-05-31T21:35:39.437 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-05-31T21:35:39.437 INFO:tasks.rgw_multisite_tests: } 2023-05-31T21:35:39.437 INFO:tasks.rgw_multisite_tests: }, 2023-05-31T21:35:39.437 INFO:tasks.rgw_multisite_tests: { 2023-05-31T21:35:39.438 INFO:tasks.rgw_multisite_tests: "key": 1, 2023-05-31T21:35:39.438 INFO:tasks.rgw_multisite_tests: "val": { 2023-05-31T21:35:39.438 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-05-31T21:35:39.438 INFO:tasks.rgw_multisite_tests: "marker": "", 2023-05-31T21:35:39.439 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-05-31T21:35:39.439 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-05-31T21:35:39.439 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-05-31T21:35:39.439 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2023-05-31T21:35:39.440 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-05-31T21:35:39.440 INFO:tasks.rgw_multisite_tests: } 2023-05-31T21:35:39.440 INFO:tasks.rgw_multisite_tests: }, 2023-05-31T21:35:39.440 INFO:tasks.rgw_multisite_tests: { 2023-05-31T21:35:39.440 INFO:tasks.rgw_multisite_tests: "key": 2, 2023-05-31T21:35:39.441 INFO:tasks.rgw_multisite_tests: "val": { 2023-05-31T21:35:39.441 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-05-31T21:35:39.441 INFO:tasks.rgw_multisite_tests: "marker": "1_1685562970.274572_93.1", 2023-05-31T21:35:39.441 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-05-31T21:35:39.442 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-05-31T21:35:39.442 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-05-31T21:35:39.442 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-05-31T19:56:10.274572Z", 2023-05-31T21:35:39.442 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-05-31T21:35:39.442 INFO:tasks.rgw_multisite_tests: } 2023-05-31T21:35:39.443 INFO:tasks.rgw_multisite_tests: }, 2023-05-31T21:35:39.443 INFO:tasks.rgw_multisite_tests: { 2023-05-31T21:35:39.443 INFO:tasks.rgw_multisite_tests: "key": 3, 2023-05-31T21:35:39.443 INFO:tasks.rgw_multisite_tests: "val": { 2023-05-31T21:35:39.444 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-05-31T21:35:39.444 INFO:tasks.rgw_multisite_tests: "marker": "1_1685563798.970303_257.1", 2023-05-31T21:35:39.444 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-05-31T21:35:39.444 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-05-31T21:35:39.445 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-05-31T21:35:39.445 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-05-31T20:09:58.970303Z", 2023-05-31T21:35:39.445 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-05-31T21:35:39.445 INFO:tasks.rgw_multisite_tests: } 2023-05-31T21:35:39.446 INFO:tasks.rgw_multisite_tests: } 2023-05-31T21:35:39.446 INFO:tasks.rgw_multisite_tests: ] 2023-05-31T21:35:39.446 INFO:tasks.rgw_multisite_tests: }, 2023-05-31T21:35:39.446 INFO:tasks.rgw_multisite_tests: "full_sync": { 2023-05-31T21:35:39.446 INFO:tasks.rgw_multisite_tests: "total": 1, 2023-05-31T21:35:39.447 INFO:tasks.rgw_multisite_tests: "complete": 1 2023-05-31T21:35:39.447 INFO:tasks.rgw_multisite_tests: }, 2023-05-31T21:35:39.447 INFO:tasks.rgw_multisite_tests: "current_time": "2023-05-31T20:13:54Z" 2023-05-31T21:35:39.447 INFO:tasks.rgw_multisite_tests:} 2023-05-31T21:35:39.448 INFO:tasks.rgw_multisite_tests: 2023-05-31T21:35:39.448 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1685562631.516089_59.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-05-31T19:50:31.516089Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1685562970.274572_93.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-05-31T19:56:10.274572Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1685563798.970303_257.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-05-31T20:09:58.970303Z', 'realm_epoch': 2}}] 2023-05-31T21:35:39.448 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1685562631.516089_59.1', 1: '', 2: '1_1685562970.274572_93.1', 3: '1_1685563802.468126_296.1'} 2023-05-31T21:35:39.448 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1685562631.516089_59.1', 1: '', 2: '1_1685562970.274572_93.1', 3: '1_1685563798.970303_257.1'} 2023-05-31T21:35:39.449 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone a2 behind master: shard=3 master=1_1685563802.468126_296.1 target=1_1685563798.970303_257.1 2023-05-31T21:35:39.449 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-31T21:35:39.449 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-31T21:35:39.449 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-05-31T21:35:39.449 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: ignoring --setuser ceph since I am not root 2023-05-31T21:35:39.450 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-05-31T21:35:39.450 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: 2023-05-31T20:13:59.155+0000 7f22fff34780 20 RGW-SYNC:meta: read sync status 2023-05-31T21:35:39.450 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: 2023-05-31T20:13:59.156+0000 7f22fff34780 20 RGW-SYNC:meta: read sync status 2023-05-31T21:35:39.450 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2023-05-31T21:35:39.451 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "sync_status": { 2023-05-31T21:35:39.451 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "info": { 2023-05-31T21:35:39.451 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "status": "sync", 2023-05-31T21:35:39.451 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "num_shards": 4, 2023-05-31T21:35:39.452 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "period": "2e6123d2-d695-4350-9a01-61a455bc782b", 2023-05-31T21:35:39.452 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "realm_epoch": 2 2023-05-31T21:35:39.452 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2023-05-31T21:35:39.452 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "markers": [ 2023-05-31T21:35:39.453 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2023-05-31T21:35:39.453 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "key": 0, 2023-05-31T21:35:39.453 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "val": { 2023-05-31T21:35:39.453 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "state": 1, 2023-05-31T21:35:39.453 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "marker": "1_1685562631.516089_59.1", 2023-05-31T21:35:39.454 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "next_step_marker": "", 2023-05-31T21:35:39.454 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "total_entries": 1, 2023-05-31T21:35:39.454 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "pos": 0, 2023-05-31T21:35:39.454 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "timestamp": "2023-05-31T19:50:31.516089Z", 2023-05-31T21:35:39.455 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "realm_epoch": 2 2023-05-31T21:35:39.455 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2023-05-31T21:35:39.455 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2023-05-31T21:35:39.455 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2023-05-31T21:35:39.456 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "key": 1, 2023-05-31T21:35:39.456 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "val": { 2023-05-31T21:35:39.456 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "state": 1, 2023-05-31T21:35:39.456 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "marker": "", 2023-05-31T21:35:39.457 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "next_step_marker": "", 2023-05-31T21:35:39.457 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "total_entries": 0, 2023-05-31T21:35:39.457 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "pos": 0, 2023-05-31T21:35:39.457 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "timestamp": "0.000000", 2023-05-31T21:35:39.457 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "realm_epoch": 2 2023-05-31T21:35:39.458 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2023-05-31T21:35:39.458 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2023-05-31T21:35:39.458 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2023-05-31T21:35:39.458 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "key": 2, 2023-05-31T21:35:39.459 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "val": { 2023-05-31T21:35:39.459 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "state": 1, 2023-05-31T21:35:39.459 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "marker": "1_1685562970.274572_93.1", 2023-05-31T21:35:39.460 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "next_step_marker": "", 2023-05-31T21:35:39.460 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "total_entries": 0, 2023-05-31T21:35:39.460 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "pos": 0, 2023-05-31T21:35:39.460 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "timestamp": "2023-05-31T19:56:10.274572Z", 2023-05-31T21:35:39.460 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "realm_epoch": 2 2023-05-31T21:35:39.461 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2023-05-31T21:35:39.461 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2023-05-31T21:35:39.461 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2023-05-31T21:35:39.461 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "key": 3, 2023-05-31T21:35:39.462 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "val": { 2023-05-31T21:35:39.462 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "state": 1, 2023-05-31T21:35:39.462 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "marker": "1_1685563798.760350_247.1", 2023-05-31T21:35:39.462 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "next_step_marker": "", 2023-05-31T21:35:39.463 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "total_entries": 0, 2023-05-31T21:35:39.463 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "pos": 0, 2023-05-31T21:35:39.463 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "timestamp": "2023-05-31T20:09:58.760350Z", 2023-05-31T21:35:39.463 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "realm_epoch": 2 2023-05-31T21:35:39.463 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2023-05-31T21:35:39.464 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2023-05-31T21:35:39.464 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: ] 2023-05-31T21:35:39.464 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2023-05-31T21:35:39.464 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "full_sync": { 2023-05-31T21:35:39.465 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "total": 1, 2023-05-31T21:35:39.465 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "complete": 1 2023-05-31T21:35:39.465 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2023-05-31T21:35:39.465 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "current_time": "2023-05-31T20:13:59Z" 2023-05-31T21:35:39.466 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2023-05-31T21:35:39.466 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2023-05-31T21:35:39.466 INFO:tasks.rgw_multisite_tests: "sync_status": { 2023-05-31T21:35:39.466 INFO:tasks.rgw_multisite_tests: "info": { 2023-05-31T21:35:39.467 INFO:tasks.rgw_multisite_tests: "status": "sync", 2023-05-31T21:35:39.467 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2023-05-31T21:35:39.467 INFO:tasks.rgw_multisite_tests: "period": "2e6123d2-d695-4350-9a01-61a455bc782b", 2023-05-31T21:35:39.467 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-05-31T21:35:39.468 INFO:tasks.rgw_multisite_tests: }, 2023-05-31T21:35:39.468 INFO:tasks.rgw_multisite_tests: "markers": [ 2023-05-31T21:35:39.468 INFO:tasks.rgw_multisite_tests: { 2023-05-31T21:35:39.468 INFO:tasks.rgw_multisite_tests: "key": 0, 2023-05-31T21:35:39.468 INFO:tasks.rgw_multisite_tests: "val": { 2023-05-31T21:35:39.469 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-05-31T21:35:39.469 INFO:tasks.rgw_multisite_tests: "marker": "1_1685562631.516089_59.1", 2023-05-31T21:35:39.469 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-05-31T21:35:39.469 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2023-05-31T21:35:39.470 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-05-31T21:35:39.470 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-05-31T19:50:31.516089Z", 2023-05-31T21:35:39.470 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-05-31T21:35:39.470 INFO:tasks.rgw_multisite_tests: } 2023-05-31T21:35:39.471 INFO:tasks.rgw_multisite_tests: }, 2023-05-31T21:35:39.471 INFO:tasks.rgw_multisite_tests: { 2023-05-31T21:35:39.471 INFO:tasks.rgw_multisite_tests: "key": 1, 2023-05-31T21:35:39.471 INFO:tasks.rgw_multisite_tests: "val": { 2023-05-31T21:35:39.471 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-05-31T21:35:39.472 INFO:tasks.rgw_multisite_tests: "marker": "", 2023-05-31T21:35:39.472 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-05-31T21:35:39.472 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-05-31T21:35:39.472 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-05-31T21:35:39.473 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2023-05-31T21:35:39.473 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-05-31T21:35:39.473 INFO:tasks.rgw_multisite_tests: } 2023-05-31T21:35:39.473 INFO:tasks.rgw_multisite_tests: }, 2023-05-31T21:35:39.474 INFO:tasks.rgw_multisite_tests: { 2023-05-31T21:35:39.474 INFO:tasks.rgw_multisite_tests: "key": 2, 2023-05-31T21:35:39.474 INFO:tasks.rgw_multisite_tests: "val": { 2023-05-31T21:35:39.474 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-05-31T21:35:39.475 INFO:tasks.rgw_multisite_tests: "marker": "1_1685562970.274572_93.1", 2023-05-31T21:35:39.475 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-05-31T21:35:39.475 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-05-31T21:35:39.475 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-05-31T21:35:39.475 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-05-31T19:56:10.274572Z", 2023-05-31T21:35:39.476 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-05-31T21:35:39.476 INFO:tasks.rgw_multisite_tests: } 2023-05-31T21:35:39.476 INFO:tasks.rgw_multisite_tests: }, 2023-05-31T21:35:39.476 INFO:tasks.rgw_multisite_tests: { 2023-05-31T21:35:39.477 INFO:tasks.rgw_multisite_tests: "key": 3, 2023-05-31T21:35:39.477 INFO:tasks.rgw_multisite_tests: "val": { 2023-05-31T21:35:39.477 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-05-31T21:35:39.477 INFO:tasks.rgw_multisite_tests: "marker": "1_1685563798.760350_247.1", 2023-05-31T21:35:39.478 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-05-31T21:35:39.478 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-05-31T21:35:39.478 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-05-31T21:35:39.478 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-05-31T20:09:58.760350Z", 2023-05-31T21:35:39.478 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-05-31T21:35:39.479 INFO:tasks.rgw_multisite_tests: } 2023-05-31T21:35:39.479 INFO:tasks.rgw_multisite_tests: } 2023-05-31T21:35:39.479 INFO:tasks.rgw_multisite_tests: ] 2023-05-31T21:35:39.479 INFO:tasks.rgw_multisite_tests: }, 2023-05-31T21:35:39.480 INFO:tasks.rgw_multisite_tests: "full_sync": { 2023-05-31T21:35:39.480 INFO:tasks.rgw_multisite_tests: "total": 1, 2023-05-31T21:35:39.480 INFO:tasks.rgw_multisite_tests: "complete": 1 2023-05-31T21:35:39.480 INFO:tasks.rgw_multisite_tests: }, 2023-05-31T21:35:39.481 INFO:tasks.rgw_multisite_tests: "current_time": "2023-05-31T20:13:59Z" 2023-05-31T21:35:39.481 INFO:tasks.rgw_multisite_tests:} 2023-05-31T21:35:39.481 INFO:tasks.rgw_multisite_tests: 2023-05-31T21:35:39.481 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1685562631.516089_59.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-05-31T19:50:31.516089Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1685562970.274572_93.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-05-31T19:56:10.274572Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1685563798.760350_247.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-05-31T20:09:58.760350Z', 'realm_epoch': 2}}] 2023-05-31T21:35:39.482 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1685562631.516089_59.1', 1: '', 2: '1_1685562970.274572_93.1', 3: '1_1685563802.468126_296.1'} 2023-05-31T21:35:39.482 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1685562631.516089_59.1', 1: '', 2: '1_1685562970.274572_93.1', 3: '1_1685563798.760350_247.1'} 2023-05-31T21:35:39.482 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone a2 behind master: shard=3 master=1_1685563802.468126_296.1 target=1_1685563798.760350_247.1 2023-05-31T21:35:39.482 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-31T21:35:39.482 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-31T21:35:39.483 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-05-31T21:35:39.483 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: ignoring --setuser ceph since I am not root 2023-05-31T21:35:39.483 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-05-31T21:35:39.483 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: 2023-05-31T20:14:04.299+0000 7f652309b780 20 RGW-SYNC:meta: read sync status 2023-05-31T21:35:39.484 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: 2023-05-31T20:14:04.300+0000 7f652309b780 20 RGW-SYNC:meta: read sync status 2023-05-31T21:35:39.484 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2023-05-31T21:35:39.484 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "sync_status": { 2023-05-31T21:35:39.484 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "info": { 2023-05-31T21:35:39.485 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "status": "sync", 2023-05-31T21:35:39.485 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "num_shards": 4, 2023-05-31T21:35:39.485 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "period": "2e6123d2-d695-4350-9a01-61a455bc782b", 2023-05-31T21:35:39.485 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "realm_epoch": 2 2023-05-31T21:35:39.486 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2023-05-31T21:35:39.486 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "markers": [ 2023-05-31T21:35:39.486 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2023-05-31T21:35:39.486 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "key": 0, 2023-05-31T21:35:39.487 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "val": { 2023-05-31T21:35:39.487 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "state": 1, 2023-05-31T21:35:39.487 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "marker": "1_1685562631.516089_59.1", 2023-05-31T21:35:39.487 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "next_step_marker": "", 2023-05-31T21:35:39.487 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "total_entries": 1, 2023-05-31T21:35:39.488 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "pos": 0, 2023-05-31T21:35:39.488 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "timestamp": "2023-05-31T19:50:31.516089Z", 2023-05-31T21:35:39.488 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "realm_epoch": 2 2023-05-31T21:35:39.488 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2023-05-31T21:35:39.489 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2023-05-31T21:35:39.489 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2023-05-31T21:35:39.489 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "key": 1, 2023-05-31T21:35:39.489 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "val": { 2023-05-31T21:35:39.490 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "state": 1, 2023-05-31T21:35:39.490 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "marker": "", 2023-05-31T21:35:39.490 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "next_step_marker": "", 2023-05-31T21:35:39.490 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "total_entries": 0, 2023-05-31T21:35:39.490 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "pos": 0, 2023-05-31T21:35:39.491 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "timestamp": "0.000000", 2023-05-31T21:35:39.491 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "realm_epoch": 2 2023-05-31T21:35:39.491 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2023-05-31T21:35:39.491 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2023-05-31T21:35:39.492 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2023-05-31T21:35:39.492 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "key": 2, 2023-05-31T21:35:39.492 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "val": { 2023-05-31T21:35:39.492 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "state": 1, 2023-05-31T21:35:39.492 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "marker": "1_1685562970.274572_93.1", 2023-05-31T21:35:39.493 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "next_step_marker": "", 2023-05-31T21:35:39.493 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "total_entries": 0, 2023-05-31T21:35:39.493 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "pos": 0, 2023-05-31T21:35:39.493 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "timestamp": "2023-05-31T19:56:10.274572Z", 2023-05-31T21:35:39.494 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "realm_epoch": 2 2023-05-31T21:35:39.494 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2023-05-31T21:35:39.494 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2023-05-31T21:35:39.494 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2023-05-31T21:35:39.495 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "key": 3, 2023-05-31T21:35:39.495 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "val": { 2023-05-31T21:35:39.495 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "state": 1, 2023-05-31T21:35:39.495 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "marker": "1_1685563798.772373_248.1", 2023-05-31T21:35:39.496 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "next_step_marker": "", 2023-05-31T21:35:39.496 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "total_entries": 0, 2023-05-31T21:35:39.496 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "pos": 0, 2023-05-31T21:35:39.496 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "timestamp": "2023-05-31T20:09:58.772373Z", 2023-05-31T21:35:39.497 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "realm_epoch": 2 2023-05-31T21:35:39.497 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2023-05-31T21:35:39.497 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2023-05-31T21:35:39.497 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: ] 2023-05-31T21:35:39.497 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2023-05-31T21:35:39.498 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "full_sync": { 2023-05-31T21:35:39.498 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "total": 1, 2023-05-31T21:35:39.498 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "complete": 1 2023-05-31T21:35:39.498 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2023-05-31T21:35:39.499 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "current_time": "2023-05-31T20:14:04Z" 2023-05-31T21:35:39.499 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2023-05-31T21:35:39.499 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2023-05-31T21:35:39.499 INFO:tasks.rgw_multisite_tests: "sync_status": { 2023-05-31T21:35:39.499 INFO:tasks.rgw_multisite_tests: "info": { 2023-05-31T21:35:39.500 INFO:tasks.rgw_multisite_tests: "status": "sync", 2023-05-31T21:35:39.500 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2023-05-31T21:35:39.500 INFO:tasks.rgw_multisite_tests: "period": "2e6123d2-d695-4350-9a01-61a455bc782b", 2023-05-31T21:35:39.500 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-05-31T21:35:39.501 INFO:tasks.rgw_multisite_tests: }, 2023-05-31T21:35:39.501 INFO:tasks.rgw_multisite_tests: "markers": [ 2023-05-31T21:35:39.501 INFO:tasks.rgw_multisite_tests: { 2023-05-31T21:35:39.501 INFO:tasks.rgw_multisite_tests: "key": 0, 2023-05-31T21:35:39.502 INFO:tasks.rgw_multisite_tests: "val": { 2023-05-31T21:35:39.502 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-05-31T21:35:39.502 INFO:tasks.rgw_multisite_tests: "marker": "1_1685562631.516089_59.1", 2023-05-31T21:35:39.502 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-05-31T21:35:39.502 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2023-05-31T21:35:39.503 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-05-31T21:35:39.503 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-05-31T19:50:31.516089Z", 2023-05-31T21:35:39.503 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-05-31T21:35:39.503 INFO:tasks.rgw_multisite_tests: } 2023-05-31T21:35:39.504 INFO:tasks.rgw_multisite_tests: }, 2023-05-31T21:35:39.504 INFO:tasks.rgw_multisite_tests: { 2023-05-31T21:35:39.504 INFO:tasks.rgw_multisite_tests: "key": 1, 2023-05-31T21:35:39.504 INFO:tasks.rgw_multisite_tests: "val": { 2023-05-31T21:35:39.505 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-05-31T21:35:39.505 INFO:tasks.rgw_multisite_tests: "marker": "", 2023-05-31T21:35:39.505 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-05-31T21:35:39.505 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-05-31T21:35:39.506 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-05-31T21:35:39.506 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2023-05-31T21:35:39.506 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-05-31T21:35:39.506 INFO:tasks.rgw_multisite_tests: } 2023-05-31T21:35:39.506 INFO:tasks.rgw_multisite_tests: }, 2023-05-31T21:35:39.507 INFO:tasks.rgw_multisite_tests: { 2023-05-31T21:35:39.507 INFO:tasks.rgw_multisite_tests: "key": 2, 2023-05-31T21:35:39.507 INFO:tasks.rgw_multisite_tests: "val": { 2023-05-31T21:35:39.507 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-05-31T21:35:39.508 INFO:tasks.rgw_multisite_tests: "marker": "1_1685562970.274572_93.1", 2023-05-31T21:35:39.508 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-05-31T21:35:39.508 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-05-31T21:35:39.508 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-05-31T21:35:39.509 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-05-31T19:56:10.274572Z", 2023-05-31T21:35:39.509 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-05-31T21:35:39.509 INFO:tasks.rgw_multisite_tests: } 2023-05-31T21:35:39.509 INFO:tasks.rgw_multisite_tests: }, 2023-05-31T21:35:39.510 INFO:tasks.rgw_multisite_tests: { 2023-05-31T21:35:39.510 INFO:tasks.rgw_multisite_tests: "key": 3, 2023-05-31T21:35:39.510 INFO:tasks.rgw_multisite_tests: "val": { 2023-05-31T21:35:39.510 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-05-31T21:35:39.510 INFO:tasks.rgw_multisite_tests: "marker": "1_1685563798.772373_248.1", 2023-05-31T21:35:39.511 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-05-31T21:35:39.511 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-05-31T21:35:39.511 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-05-31T21:35:39.511 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-05-31T20:09:58.772373Z", 2023-05-31T21:35:39.512 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-05-31T21:35:39.512 INFO:tasks.rgw_multisite_tests: } 2023-05-31T21:35:39.512 INFO:tasks.rgw_multisite_tests: } 2023-05-31T21:35:39.512 INFO:tasks.rgw_multisite_tests: ] 2023-05-31T21:35:39.513 INFO:tasks.rgw_multisite_tests: }, 2023-05-31T21:35:39.513 INFO:tasks.rgw_multisite_tests: "full_sync": { 2023-05-31T21:35:39.513 INFO:tasks.rgw_multisite_tests: "total": 1, 2023-05-31T21:35:39.513 INFO:tasks.rgw_multisite_tests: "complete": 1 2023-05-31T21:35:39.514 INFO:tasks.rgw_multisite_tests: }, 2023-05-31T21:35:39.514 INFO:tasks.rgw_multisite_tests: "current_time": "2023-05-31T20:14:04Z" 2023-05-31T21:35:39.514 INFO:tasks.rgw_multisite_tests:} 2023-05-31T21:35:39.514 INFO:tasks.rgw_multisite_tests: 2023-05-31T21:35:39.514 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1685562631.516089_59.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-05-31T19:50:31.516089Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1685562970.274572_93.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-05-31T19:56:10.274572Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1685563798.772373_248.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-05-31T20:09:58.772373Z', 'realm_epoch': 2}}] 2023-05-31T21:35:39.515 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1685562631.516089_59.1', 1: '', 2: '1_1685562970.274572_93.1', 3: '1_1685563802.468126_296.1'} 2023-05-31T21:35:39.515 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1685562631.516089_59.1', 1: '', 2: '1_1685562970.274572_93.1', 3: '1_1685563798.772373_248.1'} 2023-05-31T21:35:39.515 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone a2 behind master: shard=3 master=1_1685563802.468126_296.1 target=1_1685563798.772373_248.1 2023-05-31T21:35:39.515 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-31T21:35:39.516 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-31T21:35:39.516 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-05-31T21:35:39.516 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: ignoring --setuser ceph since I am not root 2023-05-31T21:35:39.516 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-05-31T21:35:39.517 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: 2023-05-31T20:14:09.434+0000 7fbc96cf3780 20 RGW-SYNC:meta: read sync status 2023-05-31T21:35:39.517 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: 2023-05-31T20:14:09.435+0000 7fbc96cf3780 20 RGW-SYNC:meta: read sync status 2023-05-31T21:35:39.517 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2023-05-31T21:35:39.517 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "sync_status": { 2023-05-31T21:35:39.517 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "info": { 2023-05-31T21:35:39.518 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "status": "sync", 2023-05-31T21:35:39.518 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "num_shards": 4, 2023-05-31T21:35:39.518 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "period": "2e6123d2-d695-4350-9a01-61a455bc782b", 2023-05-31T21:35:39.518 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "realm_epoch": 2 2023-05-31T21:35:39.519 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2023-05-31T21:35:39.519 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "markers": [ 2023-05-31T21:35:39.519 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2023-05-31T21:35:39.519 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "key": 0, 2023-05-31T21:35:39.520 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "val": { 2023-05-31T21:35:39.520 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "state": 1, 2023-05-31T21:35:39.520 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "marker": "1_1685562631.516089_59.1", 2023-05-31T21:35:39.520 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "next_step_marker": "", 2023-05-31T21:35:39.520 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "total_entries": 1, 2023-05-31T21:35:39.521 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "pos": 0, 2023-05-31T21:35:39.521 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "timestamp": "2023-05-31T19:50:31.516089Z", 2023-05-31T21:35:39.521 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "realm_epoch": 2 2023-05-31T21:35:39.521 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2023-05-31T21:35:39.522 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2023-05-31T21:35:39.522 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2023-05-31T21:35:39.522 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "key": 1, 2023-05-31T21:35:39.522 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "val": { 2023-05-31T21:35:39.523 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "state": 1, 2023-05-31T21:35:39.523 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "marker": "", 2023-05-31T21:35:39.523 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "next_step_marker": "", 2023-05-31T21:35:39.523 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "total_entries": 0, 2023-05-31T21:35:39.523 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "pos": 0, 2023-05-31T21:35:39.524 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "timestamp": "0.000000", 2023-05-31T21:35:39.524 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "realm_epoch": 2 2023-05-31T21:35:39.524 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2023-05-31T21:35:39.524 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2023-05-31T21:35:39.525 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2023-05-31T21:35:39.525 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "key": 2, 2023-05-31T21:35:39.525 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "val": { 2023-05-31T21:35:39.525 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "state": 1, 2023-05-31T21:35:39.526 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "marker": "1_1685562970.274572_93.1", 2023-05-31T21:35:39.526 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "next_step_marker": "", 2023-05-31T21:35:39.526 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "total_entries": 0, 2023-05-31T21:35:39.526 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "pos": 0, 2023-05-31T21:35:39.526 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "timestamp": "2023-05-31T19:56:10.274572Z", 2023-05-31T21:35:39.527 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "realm_epoch": 2 2023-05-31T21:35:39.527 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2023-05-31T21:35:39.527 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2023-05-31T21:35:39.527 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2023-05-31T21:35:39.528 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "key": 3, 2023-05-31T21:35:39.528 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "val": { 2023-05-31T21:35:39.528 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "state": 1, 2023-05-31T21:35:39.528 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "marker": "1_1685563798.756239_246.1", 2023-05-31T21:35:39.529 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "next_step_marker": "", 2023-05-31T21:35:39.529 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "total_entries": 0, 2023-05-31T21:35:39.529 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "pos": 0, 2023-05-31T21:35:39.529 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "timestamp": "2023-05-31T20:09:58.756239Z", 2023-05-31T21:35:39.529 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "realm_epoch": 2 2023-05-31T21:35:39.530 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2023-05-31T21:35:39.530 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2023-05-31T21:35:39.530 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: ] 2023-05-31T21:35:39.530 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2023-05-31T21:35:39.531 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "full_sync": { 2023-05-31T21:35:39.531 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "total": 1, 2023-05-31T21:35:39.531 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "complete": 1 2023-05-31T21:35:39.531 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2023-05-31T21:35:39.531 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "current_time": "2023-05-31T20:14:09Z" 2023-05-31T21:35:39.532 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2023-05-31T21:35:39.532 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2023-05-31T21:35:39.532 INFO:tasks.rgw_multisite_tests: "sync_status": { 2023-05-31T21:35:39.532 INFO:tasks.rgw_multisite_tests: "info": { 2023-05-31T21:35:39.533 INFO:tasks.rgw_multisite_tests: "status": "sync", 2023-05-31T21:35:39.533 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2023-05-31T21:35:39.533 INFO:tasks.rgw_multisite_tests: "period": "2e6123d2-d695-4350-9a01-61a455bc782b", 2023-05-31T21:35:39.533 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-05-31T21:35:39.534 INFO:tasks.rgw_multisite_tests: }, 2023-05-31T21:35:39.534 INFO:tasks.rgw_multisite_tests: "markers": [ 2023-05-31T21:35:39.534 INFO:tasks.rgw_multisite_tests: { 2023-05-31T21:35:39.534 INFO:tasks.rgw_multisite_tests: "key": 0, 2023-05-31T21:35:39.534 INFO:tasks.rgw_multisite_tests: "val": { 2023-05-31T21:35:39.535 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-05-31T21:35:39.535 INFO:tasks.rgw_multisite_tests: "marker": "1_1685562631.516089_59.1", 2023-05-31T21:35:39.535 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-05-31T21:35:39.535 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2023-05-31T21:35:39.536 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-05-31T21:35:39.536 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-05-31T19:50:31.516089Z", 2023-05-31T21:35:39.536 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-05-31T21:35:39.536 INFO:tasks.rgw_multisite_tests: } 2023-05-31T21:35:39.536 INFO:tasks.rgw_multisite_tests: }, 2023-05-31T21:35:39.537 INFO:tasks.rgw_multisite_tests: { 2023-05-31T21:35:39.537 INFO:tasks.rgw_multisite_tests: "key": 1, 2023-05-31T21:35:39.537 INFO:tasks.rgw_multisite_tests: "val": { 2023-05-31T21:35:39.537 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-05-31T21:35:39.538 INFO:tasks.rgw_multisite_tests: "marker": "", 2023-05-31T21:35:39.538 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-05-31T21:35:39.538 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-05-31T21:35:39.538 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-05-31T21:35:39.539 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2023-05-31T21:35:39.539 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-05-31T21:35:39.539 INFO:tasks.rgw_multisite_tests: } 2023-05-31T21:35:39.539 INFO:tasks.rgw_multisite_tests: }, 2023-05-31T21:35:39.539 INFO:tasks.rgw_multisite_tests: { 2023-05-31T21:35:39.540 INFO:tasks.rgw_multisite_tests: "key": 2, 2023-05-31T21:35:39.540 INFO:tasks.rgw_multisite_tests: "val": { 2023-05-31T21:35:39.540 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-05-31T21:35:39.540 INFO:tasks.rgw_multisite_tests: "marker": "1_1685562970.274572_93.1", 2023-05-31T21:35:39.541 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-05-31T21:35:39.541 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-05-31T21:35:39.541 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-05-31T21:35:39.541 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-05-31T19:56:10.274572Z", 2023-05-31T21:35:39.541 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-05-31T21:35:39.542 INFO:tasks.rgw_multisite_tests: } 2023-05-31T21:35:39.542 INFO:tasks.rgw_multisite_tests: }, 2023-05-31T21:35:39.542 INFO:tasks.rgw_multisite_tests: { 2023-05-31T21:35:39.542 INFO:tasks.rgw_multisite_tests: "key": 3, 2023-05-31T21:35:39.543 INFO:tasks.rgw_multisite_tests: "val": { 2023-05-31T21:35:39.543 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-05-31T21:35:39.543 INFO:tasks.rgw_multisite_tests: "marker": "1_1685563798.756239_246.1", 2023-05-31T21:35:39.543 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-05-31T21:35:39.544 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-05-31T21:35:39.544 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-05-31T21:35:39.544 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-05-31T20:09:58.756239Z", 2023-05-31T21:35:39.544 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-05-31T21:35:39.544 INFO:tasks.rgw_multisite_tests: } 2023-05-31T21:35:39.545 INFO:tasks.rgw_multisite_tests: } 2023-05-31T21:35:39.545 INFO:tasks.rgw_multisite_tests: ] 2023-05-31T21:35:39.545 INFO:tasks.rgw_multisite_tests: }, 2023-05-31T21:35:39.545 INFO:tasks.rgw_multisite_tests: "full_sync": { 2023-05-31T21:35:39.546 INFO:tasks.rgw_multisite_tests: "total": 1, 2023-05-31T21:35:39.546 INFO:tasks.rgw_multisite_tests: "complete": 1 2023-05-31T21:35:39.546 INFO:tasks.rgw_multisite_tests: }, 2023-05-31T21:35:39.546 INFO:tasks.rgw_multisite_tests: "current_time": "2023-05-31T20:14:09Z" 2023-05-31T21:35:39.547 INFO:tasks.rgw_multisite_tests:} 2023-05-31T21:35:39.547 INFO:tasks.rgw_multisite_tests: 2023-05-31T21:35:39.547 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1685562631.516089_59.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-05-31T19:50:31.516089Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1685562970.274572_93.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-05-31T19:56:10.274572Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1685563798.756239_246.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-05-31T20:09:58.756239Z', 'realm_epoch': 2}}] 2023-05-31T21:35:39.547 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1685562631.516089_59.1', 1: '', 2: '1_1685562970.274572_93.1', 3: '1_1685563802.468126_296.1'} 2023-05-31T21:35:39.547 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1685562631.516089_59.1', 1: '', 2: '1_1685562970.274572_93.1', 3: '1_1685563798.756239_246.1'} 2023-05-31T21:35:39.548 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone a2 behind master: shard=3 master=1_1685563802.468126_296.1 target=1_1685563798.756239_246.1 2023-05-31T21:35:39.548 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-31T21:35:39.548 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-31T21:35:39.548 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-05-31T21:35:39.549 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: ignoring --setuser ceph since I am not root 2023-05-31T21:35:39.549 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-05-31T21:35:39.549 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: 2023-05-31T20:14:14.578+0000 7f6839109780 20 RGW-SYNC:meta: read sync status 2023-05-31T21:35:39.549 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: 2023-05-31T20:14:14.579+0000 7f6839109780 20 RGW-SYNC:meta: read sync status 2023-05-31T21:35:39.550 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2023-05-31T21:35:39.550 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "sync_status": { 2023-05-31T21:35:39.550 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "info": { 2023-05-31T21:35:39.550 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "status": "sync", 2023-05-31T21:35:39.550 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "num_shards": 4, 2023-05-31T21:35:39.551 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "period": "2e6123d2-d695-4350-9a01-61a455bc782b", 2023-05-31T21:35:39.551 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "realm_epoch": 2 2023-05-31T21:35:39.551 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2023-05-31T21:35:39.551 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "markers": [ 2023-05-31T21:35:39.552 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2023-05-31T21:35:39.552 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "key": 0, 2023-05-31T21:35:39.552 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "val": { 2023-05-31T21:35:39.552 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "state": 1, 2023-05-31T21:35:39.553 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "marker": "1_1685562631.516089_59.1", 2023-05-31T21:35:39.553 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "next_step_marker": "", 2023-05-31T21:35:39.553 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "total_entries": 1, 2023-05-31T21:35:39.553 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "pos": 0, 2023-05-31T21:35:39.553 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "timestamp": "2023-05-31T19:50:31.516089Z", 2023-05-31T21:35:39.554 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "realm_epoch": 2 2023-05-31T21:35:39.554 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2023-05-31T21:35:39.554 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2023-05-31T21:35:39.554 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2023-05-31T21:35:39.555 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "key": 1, 2023-05-31T21:35:39.555 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "val": { 2023-05-31T21:35:39.555 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "state": 1, 2023-05-31T21:35:39.555 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "marker": "", 2023-05-31T21:35:39.556 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "next_step_marker": "", 2023-05-31T21:35:39.556 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "total_entries": 0, 2023-05-31T21:35:39.556 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "pos": 0, 2023-05-31T21:35:39.556 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "timestamp": "0.000000", 2023-05-31T21:35:39.556 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "realm_epoch": 2 2023-05-31T21:35:39.557 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2023-05-31T21:35:39.557 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2023-05-31T21:35:39.557 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2023-05-31T21:35:39.557 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "key": 2, 2023-05-31T21:35:39.558 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "val": { 2023-05-31T21:35:39.558 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "state": 1, 2023-05-31T21:35:39.558 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "marker": "1_1685562970.274572_93.1", 2023-05-31T21:35:39.558 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "next_step_marker": "", 2023-05-31T21:35:39.559 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "total_entries": 0, 2023-05-31T21:35:39.559 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "pos": 0, 2023-05-31T21:35:39.559 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "timestamp": "2023-05-31T19:56:10.274572Z", 2023-05-31T21:35:39.559 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "realm_epoch": 2 2023-05-31T21:35:39.559 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2023-05-31T21:35:39.560 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2023-05-31T21:35:39.560 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2023-05-31T21:35:39.560 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "key": 3, 2023-05-31T21:35:39.560 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "val": { 2023-05-31T21:35:39.561 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "state": 1, 2023-05-31T21:35:39.561 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "marker": "1_1685563798.885419_253.1", 2023-05-31T21:35:39.561 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "next_step_marker": "", 2023-05-31T21:35:39.561 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "total_entries": 0, 2023-05-31T21:35:39.562 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "pos": 0, 2023-05-31T21:35:39.562 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "timestamp": "2023-05-31T20:09:58.885419Z", 2023-05-31T21:35:39.562 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "realm_epoch": 2 2023-05-31T21:35:39.562 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2023-05-31T21:35:39.562 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2023-05-31T21:35:39.563 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: ] 2023-05-31T21:35:39.563 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2023-05-31T21:35:39.563 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "full_sync": { 2023-05-31T21:35:39.563 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "total": 1, 2023-05-31T21:35:39.564 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "complete": 1 2023-05-31T21:35:39.564 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2023-05-31T21:35:39.564 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "current_time": "2023-05-31T20:14:14Z" 2023-05-31T21:35:39.564 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2023-05-31T21:35:39.564 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2023-05-31T21:35:39.565 INFO:tasks.rgw_multisite_tests: "sync_status": { 2023-05-31T21:35:39.565 INFO:tasks.rgw_multisite_tests: "info": { 2023-05-31T21:35:39.565 INFO:tasks.rgw_multisite_tests: "status": "sync", 2023-05-31T21:35:39.565 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2023-05-31T21:35:39.566 INFO:tasks.rgw_multisite_tests: "period": "2e6123d2-d695-4350-9a01-61a455bc782b", 2023-05-31T21:35:39.566 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-05-31T21:35:39.566 INFO:tasks.rgw_multisite_tests: }, 2023-05-31T21:35:39.566 INFO:tasks.rgw_multisite_tests: "markers": [ 2023-05-31T21:35:39.567 INFO:tasks.rgw_multisite_tests: { 2023-05-31T21:35:39.567 INFO:tasks.rgw_multisite_tests: "key": 0, 2023-05-31T21:35:39.567 INFO:tasks.rgw_multisite_tests: "val": { 2023-05-31T21:35:39.567 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-05-31T21:35:39.567 INFO:tasks.rgw_multisite_tests: "marker": "1_1685562631.516089_59.1", 2023-05-31T21:35:39.568 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-05-31T21:35:39.568 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2023-05-31T21:35:39.568 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-05-31T21:35:39.568 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-05-31T19:50:31.516089Z", 2023-05-31T21:35:39.569 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-05-31T21:35:39.569 INFO:tasks.rgw_multisite_tests: } 2023-05-31T21:35:39.569 INFO:tasks.rgw_multisite_tests: }, 2023-05-31T21:35:39.569 INFO:tasks.rgw_multisite_tests: { 2023-05-31T21:35:39.570 INFO:tasks.rgw_multisite_tests: "key": 1, 2023-05-31T21:35:39.570 INFO:tasks.rgw_multisite_tests: "val": { 2023-05-31T21:35:39.570 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-05-31T21:35:39.570 INFO:tasks.rgw_multisite_tests: "marker": "", 2023-05-31T21:35:39.571 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-05-31T21:35:39.571 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-05-31T21:35:39.571 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-05-31T21:35:39.571 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2023-05-31T21:35:39.571 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-05-31T21:35:39.572 INFO:tasks.rgw_multisite_tests: } 2023-05-31T21:35:39.572 INFO:tasks.rgw_multisite_tests: }, 2023-05-31T21:35:39.572 INFO:tasks.rgw_multisite_tests: { 2023-05-31T21:35:39.572 INFO:tasks.rgw_multisite_tests: "key": 2, 2023-05-31T21:35:39.573 INFO:tasks.rgw_multisite_tests: "val": { 2023-05-31T21:35:39.573 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-05-31T21:35:39.573 INFO:tasks.rgw_multisite_tests: "marker": "1_1685562970.274572_93.1", 2023-05-31T21:35:39.573 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-05-31T21:35:39.574 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-05-31T21:35:39.574 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-05-31T21:35:39.574 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-05-31T19:56:10.274572Z", 2023-05-31T21:35:39.574 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-05-31T21:35:39.574 INFO:tasks.rgw_multisite_tests: } 2023-05-31T21:35:39.575 INFO:tasks.rgw_multisite_tests: }, 2023-05-31T21:35:39.575 INFO:tasks.rgw_multisite_tests: { 2023-05-31T21:35:39.575 INFO:tasks.rgw_multisite_tests: "key": 3, 2023-05-31T21:35:39.575 INFO:tasks.rgw_multisite_tests: "val": { 2023-05-31T21:35:39.576 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-05-31T21:35:39.576 INFO:tasks.rgw_multisite_tests: "marker": "1_1685563798.885419_253.1", 2023-05-31T21:35:39.576 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-05-31T21:35:39.576 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-05-31T21:35:39.577 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-05-31T21:35:39.577 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-05-31T20:09:58.885419Z", 2023-05-31T21:35:39.577 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-05-31T21:35:39.577 INFO:tasks.rgw_multisite_tests: } 2023-05-31T21:35:39.578 INFO:tasks.rgw_multisite_tests: } 2023-05-31T21:35:39.578 INFO:tasks.rgw_multisite_tests: ] 2023-05-31T21:35:39.578 INFO:tasks.rgw_multisite_tests: }, 2023-05-31T21:35:39.578 INFO:tasks.rgw_multisite_tests: "full_sync": { 2023-05-31T21:35:39.578 INFO:tasks.rgw_multisite_tests: "total": 1, 2023-05-31T21:35:39.579 INFO:tasks.rgw_multisite_tests: "complete": 1 2023-05-31T21:35:39.579 INFO:tasks.rgw_multisite_tests: }, 2023-05-31T21:35:39.579 INFO:tasks.rgw_multisite_tests: "current_time": "2023-05-31T20:14:14Z" 2023-05-31T21:35:39.579 INFO:tasks.rgw_multisite_tests:} 2023-05-31T21:35:39.580 INFO:tasks.rgw_multisite_tests: 2023-05-31T21:35:39.580 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1685562631.516089_59.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-05-31T19:50:31.516089Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1685562970.274572_93.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-05-31T19:56:10.274572Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1685563798.885419_253.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-05-31T20:09:58.885419Z', 'realm_epoch': 2}}] 2023-05-31T21:35:39.580 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1685562631.516089_59.1', 1: '', 2: '1_1685562970.274572_93.1', 3: '1_1685563802.468126_296.1'} 2023-05-31T21:35:39.580 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1685562631.516089_59.1', 1: '', 2: '1_1685562970.274572_93.1', 3: '1_1685563798.885419_253.1'} 2023-05-31T21:35:39.581 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone a2 behind master: shard=3 master=1_1685563802.468126_296.1 target=1_1685563798.885419_253.1 2023-05-31T21:35:39.581 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-31T21:35:39.581 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-31T21:35:39.581 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-05-31T21:35:39.581 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: ignoring --setuser ceph since I am not root 2023-05-31T21:35:39.582 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-05-31T21:35:39.582 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: 2023-05-31T20:14:19.727+0000 7f60548f0780 20 RGW-SYNC:meta: read sync status 2023-05-31T21:35:39.582 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: 2023-05-31T20:14:19.728+0000 7f60548f0780 20 RGW-SYNC:meta: read sync status 2023-05-31T21:35:39.582 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2023-05-31T21:35:39.583 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "sync_status": { 2023-05-31T21:35:39.583 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "info": { 2023-05-31T21:35:39.583 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "status": "sync", 2023-05-31T21:35:39.583 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "num_shards": 4, 2023-05-31T21:35:39.584 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "period": "2e6123d2-d695-4350-9a01-61a455bc782b", 2023-05-31T21:35:39.584 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "realm_epoch": 2 2023-05-31T21:35:39.584 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2023-05-31T21:35:39.584 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "markers": [ 2023-05-31T21:35:39.585 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2023-05-31T21:35:39.585 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "key": 0, 2023-05-31T21:35:39.585 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "val": { 2023-05-31T21:35:39.585 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "state": 1, 2023-05-31T21:35:39.585 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "marker": "1_1685562631.516089_59.1", 2023-05-31T21:35:39.586 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "next_step_marker": "", 2023-05-31T21:35:39.586 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "total_entries": 1, 2023-05-31T21:35:39.586 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "pos": 0, 2023-05-31T21:35:39.586 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "timestamp": "2023-05-31T19:50:31.516089Z", 2023-05-31T21:35:39.587 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "realm_epoch": 2 2023-05-31T21:35:39.587 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2023-05-31T21:35:39.587 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2023-05-31T21:35:39.587 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2023-05-31T21:35:39.587 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "key": 1, 2023-05-31T21:35:39.588 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "val": { 2023-05-31T21:35:39.588 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "state": 1, 2023-05-31T21:35:39.588 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "marker": "", 2023-05-31T21:35:39.588 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "next_step_marker": "", 2023-05-31T21:35:39.589 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "total_entries": 0, 2023-05-31T21:35:39.589 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "pos": 0, 2023-05-31T21:35:39.589 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "timestamp": "0.000000", 2023-05-31T21:35:39.589 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "realm_epoch": 2 2023-05-31T21:35:39.590 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2023-05-31T21:35:39.590 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2023-05-31T21:35:39.590 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2023-05-31T21:35:39.590 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "key": 2, 2023-05-31T21:35:39.590 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "val": { 2023-05-31T21:35:39.591 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "state": 1, 2023-05-31T21:35:39.591 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "marker": "1_1685562970.274572_93.1", 2023-05-31T21:35:39.591 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "next_step_marker": "", 2023-05-31T21:35:39.591 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "total_entries": 0, 2023-05-31T21:35:39.592 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "pos": 0, 2023-05-31T21:35:39.592 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "timestamp": "2023-05-31T19:56:10.274572Z", 2023-05-31T21:35:39.592 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "realm_epoch": 2 2023-05-31T21:35:39.592 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2023-05-31T21:35:39.593 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2023-05-31T21:35:39.593 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2023-05-31T21:35:39.593 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "key": 3, 2023-05-31T21:35:39.593 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "val": { 2023-05-31T21:35:39.593 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "state": 1, 2023-05-31T21:35:39.594 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "marker": "1_1685563798.896727_254.1", 2023-05-31T21:35:39.594 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "next_step_marker": "", 2023-05-31T21:35:39.594 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "total_entries": 0, 2023-05-31T21:35:39.594 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "pos": 0, 2023-05-31T21:35:39.595 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "timestamp": "2023-05-31T20:09:58.896727Z", 2023-05-31T21:35:39.595 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "realm_epoch": 2 2023-05-31T21:35:39.595 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2023-05-31T21:35:39.595 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2023-05-31T21:35:39.595 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: ] 2023-05-31T21:35:39.596 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2023-05-31T21:35:39.596 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "full_sync": { 2023-05-31T21:35:39.596 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "total": 1, 2023-05-31T21:35:39.596 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "complete": 1 2023-05-31T21:35:39.597 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2023-05-31T21:35:39.597 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "current_time": "2023-05-31T20:14:19Z" 2023-05-31T21:35:39.597 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2023-05-31T21:35:39.597 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2023-05-31T21:35:39.598 INFO:tasks.rgw_multisite_tests: "sync_status": { 2023-05-31T21:35:39.598 INFO:tasks.rgw_multisite_tests: "info": { 2023-05-31T21:35:39.598 INFO:tasks.rgw_multisite_tests: "status": "sync", 2023-05-31T21:35:39.598 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2023-05-31T21:35:39.598 INFO:tasks.rgw_multisite_tests: "period": "2e6123d2-d695-4350-9a01-61a455bc782b", 2023-05-31T21:35:39.599 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-05-31T21:35:39.599 INFO:tasks.rgw_multisite_tests: }, 2023-05-31T21:35:39.599 INFO:tasks.rgw_multisite_tests: "markers": [ 2023-05-31T21:35:39.599 INFO:tasks.rgw_multisite_tests: { 2023-05-31T21:35:39.600 INFO:tasks.rgw_multisite_tests: "key": 0, 2023-05-31T21:35:39.600 INFO:tasks.rgw_multisite_tests: "val": { 2023-05-31T21:35:39.600 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-05-31T21:35:39.600 INFO:tasks.rgw_multisite_tests: "marker": "1_1685562631.516089_59.1", 2023-05-31T21:35:39.600 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-05-31T21:35:39.601 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2023-05-31T21:35:39.601 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-05-31T21:35:39.601 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-05-31T19:50:31.516089Z", 2023-05-31T21:35:39.601 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-05-31T21:35:39.602 INFO:tasks.rgw_multisite_tests: } 2023-05-31T21:35:39.602 INFO:tasks.rgw_multisite_tests: }, 2023-05-31T21:35:39.602 INFO:tasks.rgw_multisite_tests: { 2023-05-31T21:35:39.602 INFO:tasks.rgw_multisite_tests: "key": 1, 2023-05-31T21:35:39.603 INFO:tasks.rgw_multisite_tests: "val": { 2023-05-31T21:35:39.603 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-05-31T21:35:39.603 INFO:tasks.rgw_multisite_tests: "marker": "", 2023-05-31T21:35:39.603 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-05-31T21:35:39.603 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-05-31T21:35:39.604 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-05-31T21:35:39.604 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2023-05-31T21:35:39.604 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-05-31T21:35:39.604 INFO:tasks.rgw_multisite_tests: } 2023-05-31T21:35:39.605 INFO:tasks.rgw_multisite_tests: }, 2023-05-31T21:35:39.605 INFO:tasks.rgw_multisite_tests: { 2023-05-31T21:35:39.605 INFO:tasks.rgw_multisite_tests: "key": 2, 2023-05-31T21:35:39.605 INFO:tasks.rgw_multisite_tests: "val": { 2023-05-31T21:35:39.606 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-05-31T21:35:39.606 INFO:tasks.rgw_multisite_tests: "marker": "1_1685562970.274572_93.1", 2023-05-31T21:35:39.606 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-05-31T21:35:39.606 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-05-31T21:35:39.607 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-05-31T21:35:39.607 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-05-31T19:56:10.274572Z", 2023-05-31T21:35:39.607 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-05-31T21:35:39.607 INFO:tasks.rgw_multisite_tests: } 2023-05-31T21:35:39.607 INFO:tasks.rgw_multisite_tests: }, 2023-05-31T21:35:39.608 INFO:tasks.rgw_multisite_tests: { 2023-05-31T21:35:39.608 INFO:tasks.rgw_multisite_tests: "key": 3, 2023-05-31T21:35:39.608 INFO:tasks.rgw_multisite_tests: "val": { 2023-05-31T21:35:39.608 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-05-31T21:35:39.609 INFO:tasks.rgw_multisite_tests: "marker": "1_1685563798.896727_254.1", 2023-05-31T21:35:39.609 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-05-31T21:35:39.609 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-05-31T21:35:39.609 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-05-31T21:35:39.610 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-05-31T20:09:58.896727Z", 2023-05-31T21:35:39.610 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-05-31T21:35:39.610 INFO:tasks.rgw_multisite_tests: } 2023-05-31T21:35:39.610 INFO:tasks.rgw_multisite_tests: } 2023-05-31T21:35:39.611 INFO:tasks.rgw_multisite_tests: ] 2023-05-31T21:35:39.611 INFO:tasks.rgw_multisite_tests: }, 2023-05-31T21:35:39.611 INFO:tasks.rgw_multisite_tests: "full_sync": { 2023-05-31T21:35:39.611 INFO:tasks.rgw_multisite_tests: "total": 1, 2023-05-31T21:35:39.611 INFO:tasks.rgw_multisite_tests: "complete": 1 2023-05-31T21:35:39.612 INFO:tasks.rgw_multisite_tests: }, 2023-05-31T21:35:39.612 INFO:tasks.rgw_multisite_tests: "current_time": "2023-05-31T20:14:19Z" 2023-05-31T21:35:39.612 INFO:tasks.rgw_multisite_tests:} 2023-05-31T21:35:39.612 INFO:tasks.rgw_multisite_tests: 2023-05-31T21:35:39.613 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1685562631.516089_59.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-05-31T19:50:31.516089Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1685562970.274572_93.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-05-31T19:56:10.274572Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1685563798.896727_254.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-05-31T20:09:58.896727Z', 'realm_epoch': 2}}] 2023-05-31T21:35:39.613 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1685562631.516089_59.1', 1: '', 2: '1_1685562970.274572_93.1', 3: '1_1685563802.468126_296.1'} 2023-05-31T21:35:39.613 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1685562631.516089_59.1', 1: '', 2: '1_1685562970.274572_93.1', 3: '1_1685563798.896727_254.1'} 2023-05-31T21:35:39.613 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone a2 behind master: shard=3 master=1_1685563802.468126_296.1 target=1_1685563798.896727_254.1 2023-05-31T21:35:39.614 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-31T21:35:39.614 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-31T21:35:39.614 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-05-31T21:35:39.614 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: ignoring --setuser ceph since I am not root 2023-05-31T21:35:39.614 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-05-31T21:35:39.615 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: 2023-05-31T20:14:24.846+0000 7fc5f1475780 20 RGW-SYNC:meta: read sync status 2023-05-31T21:35:39.615 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: 2023-05-31T20:14:24.847+0000 7fc5f1475780 20 RGW-SYNC:meta: read sync status 2023-05-31T21:35:39.615 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2023-05-31T21:35:39.615 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "sync_status": { 2023-05-31T21:35:39.616 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "info": { 2023-05-31T21:35:39.616 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "status": "sync", 2023-05-31T21:35:39.616 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "num_shards": 4, 2023-05-31T21:35:39.616 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "period": "2e6123d2-d695-4350-9a01-61a455bc782b", 2023-05-31T21:35:39.617 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "realm_epoch": 2 2023-05-31T21:35:39.617 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2023-05-31T21:35:39.617 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "markers": [ 2023-05-31T21:35:39.617 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2023-05-31T21:35:39.618 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "key": 0, 2023-05-31T21:35:39.618 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "val": { 2023-05-31T21:35:39.618 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "state": 1, 2023-05-31T21:35:39.618 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "marker": "1_1685562631.516089_59.1", 2023-05-31T21:35:39.618 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "next_step_marker": "", 2023-05-31T21:35:39.619 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "total_entries": 1, 2023-05-31T21:35:39.619 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "pos": 0, 2023-05-31T21:35:39.619 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "timestamp": "2023-05-31T19:50:31.516089Z", 2023-05-31T21:35:39.619 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "realm_epoch": 2 2023-05-31T21:35:39.620 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2023-05-31T21:35:39.620 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2023-05-31T21:35:39.620 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2023-05-31T21:35:39.620 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "key": 1, 2023-05-31T21:35:39.620 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "val": { 2023-05-31T21:35:39.621 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "state": 1, 2023-05-31T21:35:39.621 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "marker": "", 2023-05-31T21:35:39.621 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "next_step_marker": "", 2023-05-31T21:35:39.621 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "total_entries": 0, 2023-05-31T21:35:39.622 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "pos": 0, 2023-05-31T21:35:39.622 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "timestamp": "0.000000", 2023-05-31T21:35:39.622 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "realm_epoch": 2 2023-05-31T21:35:39.622 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2023-05-31T21:35:39.623 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2023-05-31T21:35:39.623 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2023-05-31T21:35:39.623 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "key": 2, 2023-05-31T21:35:39.623 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "val": { 2023-05-31T21:35:39.624 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "state": 1, 2023-05-31T21:35:39.624 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "marker": "1_1685562970.274572_93.1", 2023-05-31T21:35:39.624 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "next_step_marker": "", 2023-05-31T21:35:39.624 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "total_entries": 0, 2023-05-31T21:35:39.625 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "pos": 0, 2023-05-31T21:35:39.625 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "timestamp": "2023-05-31T19:56:10.274572Z", 2023-05-31T21:35:39.625 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "realm_epoch": 2 2023-05-31T21:35:39.625 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2023-05-31T21:35:39.625 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2023-05-31T21:35:39.626 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2023-05-31T21:35:39.626 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "key": 3, 2023-05-31T21:35:39.626 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "val": { 2023-05-31T21:35:39.626 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "state": 1, 2023-05-31T21:35:39.627 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "marker": "1_1685563798.756239_246.1", 2023-05-31T21:35:39.627 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "next_step_marker": "", 2023-05-31T21:35:39.627 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "total_entries": 0, 2023-05-31T21:35:39.627 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "pos": 0, 2023-05-31T21:35:39.628 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "timestamp": "2023-05-31T20:09:58.756239Z", 2023-05-31T21:35:39.628 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "realm_epoch": 2 2023-05-31T21:35:39.628 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2023-05-31T21:35:39.628 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2023-05-31T21:35:39.628 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: ] 2023-05-31T21:35:39.629 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2023-05-31T21:35:39.629 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "full_sync": { 2023-05-31T21:35:39.629 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "total": 1, 2023-05-31T21:35:39.629 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "complete": 1 2023-05-31T21:35:39.630 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2023-05-31T21:35:39.630 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "current_time": "2023-05-31T20:14:24Z" 2023-05-31T21:35:39.630 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2023-05-31T21:35:39.630 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2023-05-31T21:35:39.631 INFO:tasks.rgw_multisite_tests: "sync_status": { 2023-05-31T21:35:39.631 INFO:tasks.rgw_multisite_tests: "info": { 2023-05-31T21:35:39.631 INFO:tasks.rgw_multisite_tests: "status": "sync", 2023-05-31T21:35:39.631 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2023-05-31T21:35:39.631 INFO:tasks.rgw_multisite_tests: "period": "2e6123d2-d695-4350-9a01-61a455bc782b", 2023-05-31T21:35:39.632 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-05-31T21:35:39.632 INFO:tasks.rgw_multisite_tests: }, 2023-05-31T21:35:39.632 INFO:tasks.rgw_multisite_tests: "markers": [ 2023-05-31T21:35:39.632 INFO:tasks.rgw_multisite_tests: { 2023-05-31T21:35:39.633 INFO:tasks.rgw_multisite_tests: "key": 0, 2023-05-31T21:35:39.633 INFO:tasks.rgw_multisite_tests: "val": { 2023-05-31T21:35:39.633 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-05-31T21:35:39.633 INFO:tasks.rgw_multisite_tests: "marker": "1_1685562631.516089_59.1", 2023-05-31T21:35:39.634 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-05-31T21:35:39.634 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2023-05-31T21:35:39.634 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-05-31T21:35:39.634 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-05-31T19:50:31.516089Z", 2023-05-31T21:35:39.634 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-05-31T21:35:39.635 INFO:tasks.rgw_multisite_tests: } 2023-05-31T21:35:39.635 INFO:tasks.rgw_multisite_tests: }, 2023-05-31T21:35:39.635 INFO:tasks.rgw_multisite_tests: { 2023-05-31T21:35:39.635 INFO:tasks.rgw_multisite_tests: "key": 1, 2023-05-31T21:35:39.636 INFO:tasks.rgw_multisite_tests: "val": { 2023-05-31T21:35:39.636 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-05-31T21:35:39.636 INFO:tasks.rgw_multisite_tests: "marker": "", 2023-05-31T21:35:39.636 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-05-31T21:35:39.637 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-05-31T21:35:39.637 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-05-31T21:35:39.637 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2023-05-31T21:35:39.637 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-05-31T21:35:39.638 INFO:tasks.rgw_multisite_tests: } 2023-05-31T21:35:39.638 INFO:tasks.rgw_multisite_tests: }, 2023-05-31T21:35:39.638 INFO:tasks.rgw_multisite_tests: { 2023-05-31T21:35:39.638 INFO:tasks.rgw_multisite_tests: "key": 2, 2023-05-31T21:35:39.638 INFO:tasks.rgw_multisite_tests: "val": { 2023-05-31T21:35:39.639 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-05-31T21:35:39.639 INFO:tasks.rgw_multisite_tests: "marker": "1_1685562970.274572_93.1", 2023-05-31T21:35:39.639 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-05-31T21:35:39.639 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-05-31T21:35:39.640 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-05-31T21:35:39.640 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-05-31T19:56:10.274572Z", 2023-05-31T21:35:39.640 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-05-31T21:35:39.640 INFO:tasks.rgw_multisite_tests: } 2023-05-31T21:35:39.640 INFO:tasks.rgw_multisite_tests: }, 2023-05-31T21:35:39.641 INFO:tasks.rgw_multisite_tests: { 2023-05-31T21:35:39.641 INFO:tasks.rgw_multisite_tests: "key": 3, 2023-05-31T21:35:39.641 INFO:tasks.rgw_multisite_tests: "val": { 2023-05-31T21:35:39.641 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-05-31T21:35:39.642 INFO:tasks.rgw_multisite_tests: "marker": "1_1685563798.756239_246.1", 2023-05-31T21:35:39.642 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-05-31T21:35:39.642 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-05-31T21:35:39.642 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-05-31T21:35:39.643 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-05-31T20:09:58.756239Z", 2023-05-31T21:35:39.643 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-05-31T21:35:39.643 INFO:tasks.rgw_multisite_tests: } 2023-05-31T21:35:39.643 INFO:tasks.rgw_multisite_tests: } 2023-05-31T21:35:39.643 INFO:tasks.rgw_multisite_tests: ] 2023-05-31T21:35:39.644 INFO:tasks.rgw_multisite_tests: }, 2023-05-31T21:35:39.644 INFO:tasks.rgw_multisite_tests: "full_sync": { 2023-05-31T21:35:39.644 INFO:tasks.rgw_multisite_tests: "total": 1, 2023-05-31T21:35:39.644 INFO:tasks.rgw_multisite_tests: "complete": 1 2023-05-31T21:35:39.645 INFO:tasks.rgw_multisite_tests: }, 2023-05-31T21:35:39.645 INFO:tasks.rgw_multisite_tests: "current_time": "2023-05-31T20:14:24Z" 2023-05-31T21:35:39.645 INFO:tasks.rgw_multisite_tests:} 2023-05-31T21:35:39.645 INFO:tasks.rgw_multisite_tests: 2023-05-31T21:35:39.646 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1685562631.516089_59.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-05-31T19:50:31.516089Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1685562970.274572_93.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-05-31T19:56:10.274572Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1685563798.756239_246.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-05-31T20:09:58.756239Z', 'realm_epoch': 2}}] 2023-05-31T21:35:39.646 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1685562631.516089_59.1', 1: '', 2: '1_1685562970.274572_93.1', 3: '1_1685563802.468126_296.1'} 2023-05-31T21:35:39.646 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1685562631.516089_59.1', 1: '', 2: '1_1685562970.274572_93.1', 3: '1_1685563798.756239_246.1'} 2023-05-31T21:35:39.646 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone a2 behind master: shard=3 master=1_1685563802.468126_296.1 target=1_1685563798.756239_246.1 2023-05-31T21:35:39.646 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-31T21:35:39.647 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-31T21:35:39.647 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-05-31T21:35:39.647 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: ignoring --setuser ceph since I am not root 2023-05-31T21:35:39.647 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-05-31T21:35:39.648 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: 2023-05-31T20:14:29.968+0000 7fc62da3a780 20 RGW-SYNC:meta: read sync status 2023-05-31T21:35:39.648 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: 2023-05-31T20:14:29.969+0000 7fc62da3a780 20 RGW-SYNC:meta: read sync status 2023-05-31T21:35:39.648 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2023-05-31T21:35:39.648 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "sync_status": { 2023-05-31T21:35:39.649 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "info": { 2023-05-31T21:35:39.649 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "status": "sync", 2023-05-31T21:35:39.649 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "num_shards": 4, 2023-05-31T21:35:39.649 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "period": "2e6123d2-d695-4350-9a01-61a455bc782b", 2023-05-31T21:35:39.649 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "realm_epoch": 2 2023-05-31T21:35:39.650 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2023-05-31T21:35:39.650 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "markers": [ 2023-05-31T21:35:39.650 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2023-05-31T21:35:39.650 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "key": 0, 2023-05-31T21:35:39.651 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "val": { 2023-05-31T21:35:39.651 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "state": 1, 2023-05-31T21:35:39.651 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "marker": "1_1685562631.516089_59.1", 2023-05-31T21:35:39.651 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "next_step_marker": "", 2023-05-31T21:35:39.652 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "total_entries": 1, 2023-05-31T21:35:39.652 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "pos": 0, 2023-05-31T21:35:39.652 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "timestamp": "2023-05-31T19:50:31.516089Z", 2023-05-31T21:35:39.652 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "realm_epoch": 2 2023-05-31T21:35:39.652 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2023-05-31T21:35:39.653 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2023-05-31T21:35:39.653 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2023-05-31T21:35:39.653 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "key": 1, 2023-05-31T21:35:39.653 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "val": { 2023-05-31T21:35:39.654 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "state": 1, 2023-05-31T21:35:39.654 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "marker": "", 2023-05-31T21:35:39.654 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "next_step_marker": "", 2023-05-31T21:35:39.654 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "total_entries": 0, 2023-05-31T21:35:39.654 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "pos": 0, 2023-05-31T21:35:39.655 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "timestamp": "0.000000", 2023-05-31T21:35:39.655 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "realm_epoch": 2 2023-05-31T21:35:39.655 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2023-05-31T21:35:39.655 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2023-05-31T21:35:39.656 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2023-05-31T21:35:39.656 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "key": 2, 2023-05-31T21:35:39.656 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "val": { 2023-05-31T21:35:39.656 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "state": 1, 2023-05-31T21:35:39.657 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "marker": "1_1685562970.274572_93.1", 2023-05-31T21:35:39.657 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "next_step_marker": "", 2023-05-31T21:35:39.657 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "total_entries": 0, 2023-05-31T21:35:39.657 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "pos": 0, 2023-05-31T21:35:39.657 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "timestamp": "2023-05-31T19:56:10.274572Z", 2023-05-31T21:35:39.658 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "realm_epoch": 2 2023-05-31T21:35:39.658 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2023-05-31T21:35:39.658 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2023-05-31T21:35:39.658 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2023-05-31T21:35:39.659 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "key": 3, 2023-05-31T21:35:39.659 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "val": { 2023-05-31T21:35:39.659 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "state": 1, 2023-05-31T21:35:39.659 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "marker": "1_1685563798.807987_249.1", 2023-05-31T21:35:39.660 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "next_step_marker": "", 2023-05-31T21:35:39.660 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "total_entries": 0, 2023-05-31T21:35:39.660 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "pos": 0, 2023-05-31T21:35:39.660 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "timestamp": "2023-05-31T20:09:58.807987Z", 2023-05-31T21:35:39.660 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "realm_epoch": 2 2023-05-31T21:35:39.661 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2023-05-31T21:35:39.661 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2023-05-31T21:35:39.661 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: ] 2023-05-31T21:35:39.661 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2023-05-31T21:35:39.662 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "full_sync": { 2023-05-31T21:35:39.662 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "total": 1, 2023-05-31T21:35:39.662 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "complete": 1 2023-05-31T21:35:39.662 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2023-05-31T21:35:39.663 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "current_time": "2023-05-31T20:14:29Z" 2023-05-31T21:35:39.663 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2023-05-31T21:35:39.663 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2023-05-31T21:35:39.663 INFO:tasks.rgw_multisite_tests: "sync_status": { 2023-05-31T21:35:39.663 INFO:tasks.rgw_multisite_tests: "info": { 2023-05-31T21:35:39.664 INFO:tasks.rgw_multisite_tests: "status": "sync", 2023-05-31T21:35:39.664 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2023-05-31T21:35:39.664 INFO:tasks.rgw_multisite_tests: "period": "2e6123d2-d695-4350-9a01-61a455bc782b", 2023-05-31T21:35:39.664 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-05-31T21:35:39.665 INFO:tasks.rgw_multisite_tests: }, 2023-05-31T21:35:39.665 INFO:tasks.rgw_multisite_tests: "markers": [ 2023-05-31T21:35:39.665 INFO:tasks.rgw_multisite_tests: { 2023-05-31T21:35:39.665 INFO:tasks.rgw_multisite_tests: "key": 0, 2023-05-31T21:35:39.666 INFO:tasks.rgw_multisite_tests: "val": { 2023-05-31T21:35:39.666 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-05-31T21:35:39.666 INFO:tasks.rgw_multisite_tests: "marker": "1_1685562631.516089_59.1", 2023-05-31T21:35:39.666 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-05-31T21:35:39.666 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2023-05-31T21:35:39.667 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-05-31T21:35:39.667 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-05-31T19:50:31.516089Z", 2023-05-31T21:35:39.667 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-05-31T21:35:39.667 INFO:tasks.rgw_multisite_tests: } 2023-05-31T21:35:39.668 INFO:tasks.rgw_multisite_tests: }, 2023-05-31T21:35:39.668 INFO:tasks.rgw_multisite_tests: { 2023-05-31T21:35:39.668 INFO:tasks.rgw_multisite_tests: "key": 1, 2023-05-31T21:35:39.668 INFO:tasks.rgw_multisite_tests: "val": { 2023-05-31T21:35:39.669 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-05-31T21:35:39.669 INFO:tasks.rgw_multisite_tests: "marker": "", 2023-05-31T21:35:39.669 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-05-31T21:35:39.669 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-05-31T21:35:39.669 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-05-31T21:35:39.670 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2023-05-31T21:35:39.670 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-05-31T21:35:39.670 INFO:tasks.rgw_multisite_tests: } 2023-05-31T21:35:39.670 INFO:tasks.rgw_multisite_tests: }, 2023-05-31T21:35:39.671 INFO:tasks.rgw_multisite_tests: { 2023-05-31T21:35:39.671 INFO:tasks.rgw_multisite_tests: "key": 2, 2023-05-31T21:35:39.671 INFO:tasks.rgw_multisite_tests: "val": { 2023-05-31T21:35:39.671 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-05-31T21:35:39.672 INFO:tasks.rgw_multisite_tests: "marker": "1_1685562970.274572_93.1", 2023-05-31T21:35:39.672 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-05-31T21:35:39.672 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-05-31T21:35:39.672 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-05-31T21:35:39.672 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-05-31T19:56:10.274572Z", 2023-05-31T21:35:39.673 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-05-31T21:35:39.673 INFO:tasks.rgw_multisite_tests: } 2023-05-31T21:35:39.673 INFO:tasks.rgw_multisite_tests: }, 2023-05-31T21:35:39.673 INFO:tasks.rgw_multisite_tests: { 2023-05-31T21:35:39.674 INFO:tasks.rgw_multisite_tests: "key": 3, 2023-05-31T21:35:39.674 INFO:tasks.rgw_multisite_tests: "val": { 2023-05-31T21:35:39.674 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-05-31T21:35:39.674 INFO:tasks.rgw_multisite_tests: "marker": "1_1685563798.807987_249.1", 2023-05-31T21:35:39.675 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-05-31T21:35:39.675 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-05-31T21:35:39.675 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-05-31T21:35:39.675 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-05-31T20:09:58.807987Z", 2023-05-31T21:35:39.676 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-05-31T21:35:39.676 INFO:tasks.rgw_multisite_tests: } 2023-05-31T21:35:39.676 INFO:tasks.rgw_multisite_tests: } 2023-05-31T21:35:39.676 INFO:tasks.rgw_multisite_tests: ] 2023-05-31T21:35:39.676 INFO:tasks.rgw_multisite_tests: }, 2023-05-31T21:35:39.677 INFO:tasks.rgw_multisite_tests: "full_sync": { 2023-05-31T21:35:39.677 INFO:tasks.rgw_multisite_tests: "total": 1, 2023-05-31T21:35:39.677 INFO:tasks.rgw_multisite_tests: "complete": 1 2023-05-31T21:35:39.677 INFO:tasks.rgw_multisite_tests: }, 2023-05-31T21:35:39.678 INFO:tasks.rgw_multisite_tests: "current_time": "2023-05-31T20:14:29Z" 2023-05-31T21:35:39.678 INFO:tasks.rgw_multisite_tests:} 2023-05-31T21:35:39.678 INFO:tasks.rgw_multisite_tests: 2023-05-31T21:35:39.678 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1685562631.516089_59.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-05-31T19:50:31.516089Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1685562970.274572_93.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-05-31T19:56:10.274572Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1685563798.807987_249.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-05-31T20:09:58.807987Z', 'realm_epoch': 2}}] 2023-05-31T21:35:39.679 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1685562631.516089_59.1', 1: '', 2: '1_1685562970.274572_93.1', 3: '1_1685563802.468126_296.1'} 2023-05-31T21:35:39.679 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1685562631.516089_59.1', 1: '', 2: '1_1685562970.274572_93.1', 3: '1_1685563798.807987_249.1'} 2023-05-31T21:35:39.679 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone a2 behind master: shard=3 master=1_1685563802.468126_296.1 target=1_1685563798.807987_249.1 2023-05-31T21:35:39.679 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-31T21:35:39.679 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-31T21:35:39.680 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-05-31T21:35:39.680 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: ignoring --setuser ceph since I am not root 2023-05-31T21:35:39.680 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-05-31T21:35:39.680 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: 2023-05-31T20:14:35.097+0000 7f04fa476780 20 RGW-SYNC:meta: read sync status 2023-05-31T21:35:39.681 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: 2023-05-31T20:14:35.098+0000 7f04fa476780 20 RGW-SYNC:meta: read sync status 2023-05-31T21:35:39.681 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2023-05-31T21:35:39.681 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "sync_status": { 2023-05-31T21:35:39.681 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "info": { 2023-05-31T21:35:39.682 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "status": "sync", 2023-05-31T21:35:39.682 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "num_shards": 4, 2023-05-31T21:35:39.682 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "period": "2e6123d2-d695-4350-9a01-61a455bc782b", 2023-05-31T21:35:39.682 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "realm_epoch": 2 2023-05-31T21:35:39.682 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2023-05-31T21:35:39.683 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "markers": [ 2023-05-31T21:35:39.683 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2023-05-31T21:35:39.683 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "key": 0, 2023-05-31T21:35:39.683 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "val": { 2023-05-31T21:35:39.684 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "state": 1, 2023-05-31T21:35:39.684 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "marker": "1_1685562631.516089_59.1", 2023-05-31T21:35:39.684 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "next_step_marker": "", 2023-05-31T21:35:39.684 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "total_entries": 1, 2023-05-31T21:35:39.685 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "pos": 0, 2023-05-31T21:35:39.685 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "timestamp": "2023-05-31T19:50:31.516089Z", 2023-05-31T21:35:39.685 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "realm_epoch": 2 2023-05-31T21:35:39.685 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2023-05-31T21:35:39.686 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2023-05-31T21:35:39.686 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2023-05-31T21:35:39.686 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "key": 1, 2023-05-31T21:35:39.686 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "val": { 2023-05-31T21:35:39.686 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "state": 1, 2023-05-31T21:35:39.687 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "marker": "", 2023-05-31T21:35:39.687 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "next_step_marker": "", 2023-05-31T21:35:39.687 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "total_entries": 0, 2023-05-31T21:35:39.687 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "pos": 0, 2023-05-31T21:35:39.688 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "timestamp": "0.000000", 2023-05-31T21:35:39.688 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "realm_epoch": 2 2023-05-31T21:35:39.688 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2023-05-31T21:35:39.688 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2023-05-31T21:35:39.688 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2023-05-31T21:35:39.689 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "key": 2, 2023-05-31T21:35:39.689 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "val": { 2023-05-31T21:35:39.689 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "state": 1, 2023-05-31T21:35:39.689 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "marker": "1_1685562970.274572_93.1", 2023-05-31T21:35:39.690 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "next_step_marker": "", 2023-05-31T21:35:39.690 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "total_entries": 0, 2023-05-31T21:35:39.690 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "pos": 0, 2023-05-31T21:35:39.690 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "timestamp": "2023-05-31T19:56:10.274572Z", 2023-05-31T21:35:39.691 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "realm_epoch": 2 2023-05-31T21:35:39.691 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2023-05-31T21:35:39.691 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2023-05-31T21:35:39.691 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2023-05-31T21:35:39.691 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "key": 3, 2023-05-31T21:35:39.692 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "val": { 2023-05-31T21:35:39.692 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "state": 1, 2023-05-31T21:35:39.692 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "marker": "1_1685563798.756239_246.1", 2023-05-31T21:35:39.692 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "next_step_marker": "", 2023-05-31T21:35:39.693 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "total_entries": 0, 2023-05-31T21:35:39.693 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "pos": 0, 2023-05-31T21:35:39.693 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "timestamp": "2023-05-31T20:09:58.756239Z", 2023-05-31T21:35:39.693 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "realm_epoch": 2 2023-05-31T21:35:39.694 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2023-05-31T21:35:39.694 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2023-05-31T21:35:39.694 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: ] 2023-05-31T21:35:39.694 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2023-05-31T21:35:39.694 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "full_sync": { 2023-05-31T21:35:39.695 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "total": 1, 2023-05-31T21:35:39.695 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "complete": 1 2023-05-31T21:35:39.695 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2023-05-31T21:35:39.695 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "current_time": "2023-05-31T20:14:35Z" 2023-05-31T21:35:39.696 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2023-05-31T21:35:39.696 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2023-05-31T21:35:39.696 INFO:tasks.rgw_multisite_tests: "sync_status": { 2023-05-31T21:35:39.696 INFO:tasks.rgw_multisite_tests: "info": { 2023-05-31T21:35:39.697 INFO:tasks.rgw_multisite_tests: "status": "sync", 2023-05-31T21:35:39.697 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2023-05-31T21:35:39.697 INFO:tasks.rgw_multisite_tests: "period": "2e6123d2-d695-4350-9a01-61a455bc782b", 2023-05-31T21:35:39.697 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-05-31T21:35:39.698 INFO:tasks.rgw_multisite_tests: }, 2023-05-31T21:35:39.698 INFO:tasks.rgw_multisite_tests: "markers": [ 2023-05-31T21:35:39.698 INFO:tasks.rgw_multisite_tests: { 2023-05-31T21:35:39.698 INFO:tasks.rgw_multisite_tests: "key": 0, 2023-05-31T21:35:39.698 INFO:tasks.rgw_multisite_tests: "val": { 2023-05-31T21:35:39.699 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-05-31T21:35:39.699 INFO:tasks.rgw_multisite_tests: "marker": "1_1685562631.516089_59.1", 2023-05-31T21:35:39.699 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-05-31T21:35:39.699 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2023-05-31T21:35:39.700 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-05-31T21:35:39.700 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-05-31T19:50:31.516089Z", 2023-05-31T21:35:39.700 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-05-31T21:35:39.700 INFO:tasks.rgw_multisite_tests: } 2023-05-31T21:35:39.701 INFO:tasks.rgw_multisite_tests: }, 2023-05-31T21:35:39.701 INFO:tasks.rgw_multisite_tests: { 2023-05-31T21:35:39.701 INFO:tasks.rgw_multisite_tests: "key": 1, 2023-05-31T21:35:39.701 INFO:tasks.rgw_multisite_tests: "val": { 2023-05-31T21:35:39.701 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-05-31T21:35:39.702 INFO:tasks.rgw_multisite_tests: "marker": "", 2023-05-31T21:35:39.702 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-05-31T21:35:39.702 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-05-31T21:35:39.702 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-05-31T21:35:39.703 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2023-05-31T21:35:39.703 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-05-31T21:35:39.703 INFO:tasks.rgw_multisite_tests: } 2023-05-31T21:35:39.703 INFO:tasks.rgw_multisite_tests: }, 2023-05-31T21:35:39.704 INFO:tasks.rgw_multisite_tests: { 2023-05-31T21:35:39.704 INFO:tasks.rgw_multisite_tests: "key": 2, 2023-05-31T21:35:39.704 INFO:tasks.rgw_multisite_tests: "val": { 2023-05-31T21:35:39.704 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-05-31T21:35:39.704 INFO:tasks.rgw_multisite_tests: "marker": "1_1685562970.274572_93.1", 2023-05-31T21:35:39.705 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-05-31T21:35:39.705 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-05-31T21:35:39.705 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-05-31T21:35:39.705 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-05-31T19:56:10.274572Z", 2023-05-31T21:35:39.706 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-05-31T21:35:39.706 INFO:tasks.rgw_multisite_tests: } 2023-05-31T21:35:39.706 INFO:tasks.rgw_multisite_tests: }, 2023-05-31T21:35:39.706 INFO:tasks.rgw_multisite_tests: { 2023-05-31T21:35:39.707 INFO:tasks.rgw_multisite_tests: "key": 3, 2023-05-31T21:35:39.707 INFO:tasks.rgw_multisite_tests: "val": { 2023-05-31T21:35:39.707 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-05-31T21:35:39.707 INFO:tasks.rgw_multisite_tests: "marker": "1_1685563798.756239_246.1", 2023-05-31T21:35:39.707 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-05-31T21:35:39.708 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-05-31T21:35:39.708 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-05-31T21:35:39.708 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-05-31T20:09:58.756239Z", 2023-05-31T21:35:39.708 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-05-31T21:35:39.709 INFO:tasks.rgw_multisite_tests: } 2023-05-31T21:35:39.709 INFO:tasks.rgw_multisite_tests: } 2023-05-31T21:35:39.709 INFO:tasks.rgw_multisite_tests: ] 2023-05-31T21:35:39.709 INFO:tasks.rgw_multisite_tests: }, 2023-05-31T21:35:39.710 INFO:tasks.rgw_multisite_tests: "full_sync": { 2023-05-31T21:35:39.710 INFO:tasks.rgw_multisite_tests: "total": 1, 2023-05-31T21:35:39.710 INFO:tasks.rgw_multisite_tests: "complete": 1 2023-05-31T21:35:39.710 INFO:tasks.rgw_multisite_tests: }, 2023-05-31T21:35:39.710 INFO:tasks.rgw_multisite_tests: "current_time": "2023-05-31T20:14:35Z" 2023-05-31T21:35:39.711 INFO:tasks.rgw_multisite_tests:} 2023-05-31T21:35:39.711 INFO:tasks.rgw_multisite_tests: 2023-05-31T21:35:39.711 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1685562631.516089_59.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-05-31T19:50:31.516089Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1685562970.274572_93.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-05-31T19:56:10.274572Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1685563798.756239_246.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-05-31T20:09:58.756239Z', 'realm_epoch': 2}}] 2023-05-31T21:35:39.711 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1685562631.516089_59.1', 1: '', 2: '1_1685562970.274572_93.1', 3: '1_1685563802.468126_296.1'} 2023-05-31T21:35:39.712 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1685562631.516089_59.1', 1: '', 2: '1_1685562970.274572_93.1', 3: '1_1685563798.756239_246.1'} 2023-05-31T21:35:39.712 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone a2 behind master: shard=3 master=1_1685563802.468126_296.1 target=1_1685563798.756239_246.1 2023-05-31T21:35:39.712 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-31T21:35:39.712 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-31T21:35:39.713 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-05-31T21:35:39.713 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: ignoring --setuser ceph since I am not root 2023-05-31T21:35:39.713 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-05-31T21:35:39.713 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: 2023-05-31T20:14:40.234+0000 7f4b0c479780 20 RGW-SYNC:meta: read sync status 2023-05-31T21:35:39.714 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: 2023-05-31T20:14:40.235+0000 7f4b0c479780 20 RGW-SYNC:meta: read sync status 2023-05-31T21:35:39.714 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2023-05-31T21:35:39.714 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "sync_status": { 2023-05-31T21:35:39.714 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "info": { 2023-05-31T21:35:39.714 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "status": "sync", 2023-05-31T21:35:39.715 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "num_shards": 4, 2023-05-31T21:35:39.715 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "period": "2e6123d2-d695-4350-9a01-61a455bc782b", 2023-05-31T21:35:39.715 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "realm_epoch": 2 2023-05-31T21:35:39.715 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2023-05-31T21:35:39.716 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "markers": [ 2023-05-31T21:35:39.716 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2023-05-31T21:35:39.716 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "key": 0, 2023-05-31T21:35:39.716 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "val": { 2023-05-31T21:35:39.716 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "state": 1, 2023-05-31T21:35:39.717 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "marker": "1_1685562631.516089_59.1", 2023-05-31T21:35:39.717 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "next_step_marker": "", 2023-05-31T21:35:39.717 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "total_entries": 1, 2023-05-31T21:35:39.717 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "pos": 0, 2023-05-31T21:35:39.718 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "timestamp": "2023-05-31T19:50:31.516089Z", 2023-05-31T21:35:39.718 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "realm_epoch": 2 2023-05-31T21:35:39.718 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2023-05-31T21:35:39.718 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2023-05-31T21:35:39.719 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2023-05-31T21:35:39.719 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "key": 1, 2023-05-31T21:35:39.719 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "val": { 2023-05-31T21:35:39.719 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "state": 1, 2023-05-31T21:35:39.719 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "marker": "", 2023-05-31T21:35:39.720 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "next_step_marker": "", 2023-05-31T21:35:39.720 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "total_entries": 0, 2023-05-31T21:35:39.720 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "pos": 0, 2023-05-31T21:35:39.720 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "timestamp": "0.000000", 2023-05-31T21:35:39.721 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "realm_epoch": 2 2023-05-31T21:35:39.721 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2023-05-31T21:35:39.721 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2023-05-31T21:35:39.721 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2023-05-31T21:35:39.722 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "key": 2, 2023-05-31T21:35:39.722 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "val": { 2023-05-31T21:35:39.722 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "state": 1, 2023-05-31T21:35:39.722 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "marker": "1_1685562970.274572_93.1", 2023-05-31T21:35:39.722 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "next_step_marker": "", 2023-05-31T21:35:39.723 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "total_entries": 0, 2023-05-31T21:35:39.723 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "pos": 0, 2023-05-31T21:35:39.723 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "timestamp": "2023-05-31T19:56:10.274572Z", 2023-05-31T21:35:39.723 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "realm_epoch": 2 2023-05-31T21:35:39.724 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2023-05-31T21:35:39.724 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2023-05-31T21:35:39.724 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2023-05-31T21:35:39.724 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "key": 3, 2023-05-31T21:35:39.725 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "val": { 2023-05-31T21:35:39.725 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "state": 1, 2023-05-31T21:35:39.725 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "marker": "1_1685563798.885419_253.1", 2023-05-31T21:35:39.725 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "next_step_marker": "", 2023-05-31T21:35:39.726 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "total_entries": 0, 2023-05-31T21:35:39.726 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "pos": 0, 2023-05-31T21:35:39.726 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "timestamp": "2023-05-31T20:09:58.885419Z", 2023-05-31T21:35:39.726 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "realm_epoch": 2 2023-05-31T21:35:39.726 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2023-05-31T21:35:39.727 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2023-05-31T21:35:39.727 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: ] 2023-05-31T21:35:39.727 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2023-05-31T21:35:39.727 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "full_sync": { 2023-05-31T21:35:39.728 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "total": 1, 2023-05-31T21:35:39.728 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "complete": 1 2023-05-31T21:35:39.728 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2023-05-31T21:35:39.728 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "current_time": "2023-05-31T20:14:40Z" 2023-05-31T21:35:39.729 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2023-05-31T21:35:39.729 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2023-05-31T21:35:39.729 INFO:tasks.rgw_multisite_tests: "sync_status": { 2023-05-31T21:35:39.729 INFO:tasks.rgw_multisite_tests: "info": { 2023-05-31T21:35:39.730 INFO:tasks.rgw_multisite_tests: "status": "sync", 2023-05-31T21:35:39.730 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2023-05-31T21:35:39.730 INFO:tasks.rgw_multisite_tests: "period": "2e6123d2-d695-4350-9a01-61a455bc782b", 2023-05-31T21:35:39.730 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-05-31T21:35:39.730 INFO:tasks.rgw_multisite_tests: }, 2023-05-31T21:35:39.731 INFO:tasks.rgw_multisite_tests: "markers": [ 2023-05-31T21:35:39.731 INFO:tasks.rgw_multisite_tests: { 2023-05-31T21:35:39.731 INFO:tasks.rgw_multisite_tests: "key": 0, 2023-05-31T21:35:39.731 INFO:tasks.rgw_multisite_tests: "val": { 2023-05-31T21:35:39.732 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-05-31T21:35:39.732 INFO:tasks.rgw_multisite_tests: "marker": "1_1685562631.516089_59.1", 2023-05-31T21:35:39.732 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-05-31T21:35:39.732 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2023-05-31T21:35:39.733 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-05-31T21:35:39.733 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-05-31T19:50:31.516089Z", 2023-05-31T21:35:39.733 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-05-31T21:35:39.733 INFO:tasks.rgw_multisite_tests: } 2023-05-31T21:35:39.733 INFO:tasks.rgw_multisite_tests: }, 2023-05-31T21:35:39.734 INFO:tasks.rgw_multisite_tests: { 2023-05-31T21:35:39.734 INFO:tasks.rgw_multisite_tests: "key": 1, 2023-05-31T21:35:39.734 INFO:tasks.rgw_multisite_tests: "val": { 2023-05-31T21:35:39.734 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-05-31T21:35:39.735 INFO:tasks.rgw_multisite_tests: "marker": "", 2023-05-31T21:35:39.735 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-05-31T21:35:39.735 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-05-31T21:35:39.735 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-05-31T21:35:39.735 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2023-05-31T21:35:39.736 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-05-31T21:35:39.736 INFO:tasks.rgw_multisite_tests: } 2023-05-31T21:35:39.736 INFO:tasks.rgw_multisite_tests: }, 2023-05-31T21:35:39.736 INFO:tasks.rgw_multisite_tests: { 2023-05-31T21:35:39.737 INFO:tasks.rgw_multisite_tests: "key": 2, 2023-05-31T21:35:39.737 INFO:tasks.rgw_multisite_tests: "val": { 2023-05-31T21:35:39.737 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-05-31T21:35:39.737 INFO:tasks.rgw_multisite_tests: "marker": "1_1685562970.274572_93.1", 2023-05-31T21:35:39.738 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-05-31T21:35:39.738 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-05-31T21:35:39.738 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-05-31T21:35:39.738 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-05-31T19:56:10.274572Z", 2023-05-31T21:35:39.738 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-05-31T21:35:39.739 INFO:tasks.rgw_multisite_tests: } 2023-05-31T21:35:39.739 INFO:tasks.rgw_multisite_tests: }, 2023-05-31T21:35:39.739 INFO:tasks.rgw_multisite_tests: { 2023-05-31T21:35:39.739 INFO:tasks.rgw_multisite_tests: "key": 3, 2023-05-31T21:35:39.740 INFO:tasks.rgw_multisite_tests: "val": { 2023-05-31T21:35:39.740 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-05-31T21:35:39.740 INFO:tasks.rgw_multisite_tests: "marker": "1_1685563798.885419_253.1", 2023-05-31T21:35:39.740 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-05-31T21:35:39.741 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-05-31T21:35:39.741 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-05-31T21:35:39.741 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-05-31T20:09:58.885419Z", 2023-05-31T21:35:39.741 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-05-31T21:35:39.742 INFO:tasks.rgw_multisite_tests: } 2023-05-31T21:35:39.742 INFO:tasks.rgw_multisite_tests: } 2023-05-31T21:35:39.742 INFO:tasks.rgw_multisite_tests: ] 2023-05-31T21:35:39.742 INFO:tasks.rgw_multisite_tests: }, 2023-05-31T21:35:39.742 INFO:tasks.rgw_multisite_tests: "full_sync": { 2023-05-31T21:35:39.743 INFO:tasks.rgw_multisite_tests: "total": 1, 2023-05-31T21:35:39.743 INFO:tasks.rgw_multisite_tests: "complete": 1 2023-05-31T21:35:39.743 INFO:tasks.rgw_multisite_tests: }, 2023-05-31T21:35:39.743 INFO:tasks.rgw_multisite_tests: "current_time": "2023-05-31T20:14:40Z" 2023-05-31T21:35:39.744 INFO:tasks.rgw_multisite_tests:} 2023-05-31T21:35:39.744 INFO:tasks.rgw_multisite_tests: 2023-05-31T21:35:39.744 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1685562631.516089_59.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-05-31T19:50:31.516089Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1685562970.274572_93.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-05-31T19:56:10.274572Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1685563798.885419_253.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-05-31T20:09:58.885419Z', 'realm_epoch': 2}}] 2023-05-31T21:35:39.744 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1685562631.516089_59.1', 1: '', 2: '1_1685562970.274572_93.1', 3: '1_1685563802.468126_296.1'} 2023-05-31T21:35:39.745 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1685562631.516089_59.1', 1: '', 2: '1_1685562970.274572_93.1', 3: '1_1685563798.885419_253.1'} 2023-05-31T21:35:39.745 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone a2 behind master: shard=3 master=1_1685563802.468126_296.1 target=1_1685563798.885419_253.1 2023-05-31T21:35:39.745 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-31T21:35:39.745 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-31T21:35:39.745 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-05-31T21:35:39.746 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: ignoring --setuser ceph since I am not root 2023-05-31T21:35:39.746 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-05-31T21:35:39.746 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: 2023-05-31T20:14:45.418+0000 7f7f52405780 20 RGW-SYNC:meta: read sync status 2023-05-31T21:35:39.746 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: 2023-05-31T20:14:45.419+0000 7f7f52405780 20 RGW-SYNC:meta: read sync status 2023-05-31T21:35:39.747 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2023-05-31T21:35:39.747 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "sync_status": { 2023-05-31T21:35:39.747 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "info": { 2023-05-31T21:35:39.747 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "status": "sync", 2023-05-31T21:35:39.748 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "num_shards": 4, 2023-05-31T21:35:39.748 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "period": "2e6123d2-d695-4350-9a01-61a455bc782b", 2023-05-31T21:35:39.748 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "realm_epoch": 2 2023-05-31T21:35:39.748 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2023-05-31T21:35:39.748 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "markers": [ 2023-05-31T21:35:39.749 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2023-05-31T21:35:39.749 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "key": 0, 2023-05-31T21:35:39.749 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "val": { 2023-05-31T21:35:39.749 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "state": 1, 2023-05-31T21:35:39.750 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "marker": "1_1685562631.516089_59.1", 2023-05-31T21:35:39.750 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "next_step_marker": "", 2023-05-31T21:35:39.750 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "total_entries": 1, 2023-05-31T21:35:39.750 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "pos": 0, 2023-05-31T21:35:39.751 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "timestamp": "2023-05-31T19:50:31.516089Z", 2023-05-31T21:35:39.751 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "realm_epoch": 2 2023-05-31T21:35:39.751 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2023-05-31T21:35:39.751 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2023-05-31T21:35:39.751 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2023-05-31T21:35:39.752 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "key": 1, 2023-05-31T21:35:39.752 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "val": { 2023-05-31T21:35:39.752 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "state": 1, 2023-05-31T21:35:39.752 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "marker": "", 2023-05-31T21:35:39.753 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "next_step_marker": "", 2023-05-31T21:35:39.753 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "total_entries": 0, 2023-05-31T21:35:39.753 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "pos": 0, 2023-05-31T21:35:39.753 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "timestamp": "0.000000", 2023-05-31T21:35:39.754 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "realm_epoch": 2 2023-05-31T21:35:39.754 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2023-05-31T21:35:39.754 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2023-05-31T21:35:39.754 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2023-05-31T21:35:39.754 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "key": 2, 2023-05-31T21:35:39.755 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "val": { 2023-05-31T21:35:39.755 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "state": 1, 2023-05-31T21:35:39.755 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "marker": "1_1685562970.274572_93.1", 2023-05-31T21:35:39.755 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "next_step_marker": "", 2023-05-31T21:35:39.756 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "total_entries": 0, 2023-05-31T21:35:39.756 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "pos": 0, 2023-05-31T21:35:39.756 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "timestamp": "2023-05-31T19:56:10.274572Z", 2023-05-31T21:35:39.756 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "realm_epoch": 2 2023-05-31T21:35:39.757 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2023-05-31T21:35:39.757 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2023-05-31T21:35:39.757 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2023-05-31T21:35:39.757 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "key": 3, 2023-05-31T21:35:39.757 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "val": { 2023-05-31T21:35:39.758 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "state": 1, 2023-05-31T21:35:39.758 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "marker": "1_1685563798.827280_252.1", 2023-05-31T21:35:39.758 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "next_step_marker": "", 2023-05-31T21:35:39.758 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "total_entries": 0, 2023-05-31T21:35:39.759 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "pos": 0, 2023-05-31T21:35:39.759 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "timestamp": "2023-05-31T20:09:58.827280Z", 2023-05-31T21:35:39.759 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "realm_epoch": 2 2023-05-31T21:35:39.760 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2023-05-31T21:35:39.760 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2023-05-31T21:35:39.760 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: ] 2023-05-31T21:35:39.760 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2023-05-31T21:35:39.760 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "full_sync": { 2023-05-31T21:35:39.761 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "total": 1, 2023-05-31T21:35:39.761 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "complete": 1 2023-05-31T21:35:39.761 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2023-05-31T21:35:39.761 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "current_time": "2023-05-31T20:14:45Z" 2023-05-31T21:35:39.762 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2023-05-31T21:35:39.762 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2023-05-31T21:35:39.762 INFO:tasks.rgw_multisite_tests: "sync_status": { 2023-05-31T21:35:39.762 INFO:tasks.rgw_multisite_tests: "info": { 2023-05-31T21:35:39.763 INFO:tasks.rgw_multisite_tests: "status": "sync", 2023-05-31T21:35:39.763 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2023-05-31T21:35:39.763 INFO:tasks.rgw_multisite_tests: "period": "2e6123d2-d695-4350-9a01-61a455bc782b", 2023-05-31T21:35:39.763 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-05-31T21:35:39.764 INFO:tasks.rgw_multisite_tests: }, 2023-05-31T21:35:39.764 INFO:tasks.rgw_multisite_tests: "markers": [ 2023-05-31T21:35:39.764 INFO:tasks.rgw_multisite_tests: { 2023-05-31T21:35:39.764 INFO:tasks.rgw_multisite_tests: "key": 0, 2023-05-31T21:35:39.765 INFO:tasks.rgw_multisite_tests: "val": { 2023-05-31T21:35:39.765 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-05-31T21:35:39.765 INFO:tasks.rgw_multisite_tests: "marker": "1_1685562631.516089_59.1", 2023-05-31T21:35:39.765 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-05-31T21:35:39.765 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2023-05-31T21:35:39.766 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-05-31T21:35:39.766 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-05-31T19:50:31.516089Z", 2023-05-31T21:35:39.766 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-05-31T21:35:39.766 INFO:tasks.rgw_multisite_tests: } 2023-05-31T21:35:39.767 INFO:tasks.rgw_multisite_tests: }, 2023-05-31T21:35:39.767 INFO:tasks.rgw_multisite_tests: { 2023-05-31T21:35:39.767 INFO:tasks.rgw_multisite_tests: "key": 1, 2023-05-31T21:35:39.767 INFO:tasks.rgw_multisite_tests: "val": { 2023-05-31T21:35:39.767 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-05-31T21:35:39.768 INFO:tasks.rgw_multisite_tests: "marker": "", 2023-05-31T21:35:39.768 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-05-31T21:35:39.768 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-05-31T21:35:39.768 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-05-31T21:35:39.769 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2023-05-31T21:35:39.769 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-05-31T21:35:39.769 INFO:tasks.rgw_multisite_tests: } 2023-05-31T21:35:39.769 INFO:tasks.rgw_multisite_tests: }, 2023-05-31T21:35:39.770 INFO:tasks.rgw_multisite_tests: { 2023-05-31T21:35:39.770 INFO:tasks.rgw_multisite_tests: "key": 2, 2023-05-31T21:35:39.770 INFO:tasks.rgw_multisite_tests: "val": { 2023-05-31T21:35:39.770 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-05-31T21:35:39.771 INFO:tasks.rgw_multisite_tests: "marker": "1_1685562970.274572_93.1", 2023-05-31T21:35:39.771 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-05-31T21:35:39.771 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-05-31T21:35:39.771 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-05-31T21:35:39.771 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-05-31T19:56:10.274572Z", 2023-05-31T21:35:39.772 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-05-31T21:35:39.772 INFO:tasks.rgw_multisite_tests: } 2023-05-31T21:35:39.772 INFO:tasks.rgw_multisite_tests: }, 2023-05-31T21:35:39.772 INFO:tasks.rgw_multisite_tests: { 2023-05-31T21:35:39.773 INFO:tasks.rgw_multisite_tests: "key": 3, 2023-05-31T21:35:39.773 INFO:tasks.rgw_multisite_tests: "val": { 2023-05-31T21:35:39.773 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-05-31T21:35:39.773 INFO:tasks.rgw_multisite_tests: "marker": "1_1685563798.827280_252.1", 2023-05-31T21:35:39.774 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-05-31T21:35:39.774 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-05-31T21:35:39.774 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-05-31T21:35:39.774 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-05-31T20:09:58.827280Z", 2023-05-31T21:35:39.775 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-05-31T21:35:39.775 INFO:tasks.rgw_multisite_tests: } 2023-05-31T21:35:39.775 INFO:tasks.rgw_multisite_tests: } 2023-05-31T21:35:39.775 INFO:tasks.rgw_multisite_tests: ] 2023-05-31T21:35:39.775 INFO:tasks.rgw_multisite_tests: }, 2023-05-31T21:35:39.776 INFO:tasks.rgw_multisite_tests: "full_sync": { 2023-05-31T21:35:39.776 INFO:tasks.rgw_multisite_tests: "total": 1, 2023-05-31T21:35:39.776 INFO:tasks.rgw_multisite_tests: "complete": 1 2023-05-31T21:35:39.777 INFO:tasks.rgw_multisite_tests: }, 2023-05-31T21:35:39.777 INFO:tasks.rgw_multisite_tests: "current_time": "2023-05-31T20:14:45Z" 2023-05-31T21:35:39.777 INFO:tasks.rgw_multisite_tests:} 2023-05-31T21:35:39.777 INFO:tasks.rgw_multisite_tests: 2023-05-31T21:35:39.778 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1685562631.516089_59.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-05-31T19:50:31.516089Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1685562970.274572_93.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-05-31T19:56:10.274572Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1685563798.827280_252.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-05-31T20:09:58.827280Z', 'realm_epoch': 2}}] 2023-05-31T21:35:39.778 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1685562631.516089_59.1', 1: '', 2: '1_1685562970.274572_93.1', 3: '1_1685563802.468126_296.1'} 2023-05-31T21:35:39.778 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1685562631.516089_59.1', 1: '', 2: '1_1685562970.274572_93.1', 3: '1_1685563798.827280_252.1'} 2023-05-31T21:35:39.778 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone a2 behind master: shard=3 master=1_1685563802.468126_296.1 target=1_1685563798.827280_252.1 2023-05-31T21:35:39.779 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-31T21:35:39.779 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-31T21:35:39.779 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-05-31T21:35:39.779 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: ignoring --setuser ceph since I am not root 2023-05-31T21:35:39.780 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-05-31T21:35:39.780 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: 2023-05-31T20:14:50.579+0000 7fa08b133780 20 RGW-SYNC:meta: read sync status 2023-05-31T21:35:39.780 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: 2023-05-31T20:14:50.580+0000 7fa08b133780 20 RGW-SYNC:meta: read sync status 2023-05-31T21:35:39.780 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2023-05-31T21:35:39.780 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "sync_status": { 2023-05-31T21:35:39.781 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "info": { 2023-05-31T21:35:39.781 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "status": "sync", 2023-05-31T21:35:39.781 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "num_shards": 4, 2023-05-31T21:35:39.781 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "period": "2e6123d2-d695-4350-9a01-61a455bc782b", 2023-05-31T21:35:39.782 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "realm_epoch": 2 2023-05-31T21:35:39.782 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2023-05-31T21:35:39.782 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "markers": [ 2023-05-31T21:35:39.782 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2023-05-31T21:35:39.783 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "key": 0, 2023-05-31T21:35:39.783 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "val": { 2023-05-31T21:35:39.783 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "state": 1, 2023-05-31T21:35:39.783 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "marker": "1_1685562631.516089_59.1", 2023-05-31T21:35:39.783 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "next_step_marker": "", 2023-05-31T21:35:39.784 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "total_entries": 1, 2023-05-31T21:35:39.784 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "pos": 0, 2023-05-31T21:35:39.784 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "timestamp": "2023-05-31T19:50:31.516089Z", 2023-05-31T21:35:39.785 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "realm_epoch": 2 2023-05-31T21:35:39.785 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2023-05-31T21:35:39.785 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2023-05-31T21:35:39.785 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2023-05-31T21:35:39.785 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "key": 1, 2023-05-31T21:35:39.786 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "val": { 2023-05-31T21:35:39.786 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "state": 1, 2023-05-31T21:35:39.786 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "marker": "", 2023-05-31T21:35:39.786 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "next_step_marker": "", 2023-05-31T21:35:39.787 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "total_entries": 0, 2023-05-31T21:35:39.787 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "pos": 0, 2023-05-31T21:35:39.787 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "timestamp": "0.000000", 2023-05-31T21:35:39.787 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "realm_epoch": 2 2023-05-31T21:35:39.788 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2023-05-31T21:35:39.788 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2023-05-31T21:35:39.788 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2023-05-31T21:35:39.788 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "key": 2, 2023-05-31T21:35:39.789 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "val": { 2023-05-31T21:35:39.789 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "state": 1, 2023-05-31T21:35:39.789 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "marker": "1_1685562970.274572_93.1", 2023-05-31T21:35:39.789 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "next_step_marker": "", 2023-05-31T21:35:39.789 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "total_entries": 0, 2023-05-31T21:35:39.790 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "pos": 0, 2023-05-31T21:35:39.790 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "timestamp": "2023-05-31T19:56:10.274572Z", 2023-05-31T21:35:39.790 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "realm_epoch": 2 2023-05-31T21:35:39.790 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2023-05-31T21:35:39.791 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2023-05-31T21:35:39.791 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2023-05-31T21:35:39.791 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "key": 3, 2023-05-31T21:35:39.791 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "val": { 2023-05-31T21:35:39.792 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "state": 1, 2023-05-31T21:35:39.792 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "marker": "1_1685563799.098535_267.1", 2023-05-31T21:35:39.792 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "next_step_marker": "", 2023-05-31T21:35:39.792 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "total_entries": 0, 2023-05-31T21:35:39.793 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "pos": 0, 2023-05-31T21:35:39.793 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "timestamp": "2023-05-31T20:09:59.098535Z", 2023-05-31T21:35:39.793 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "realm_epoch": 2 2023-05-31T21:35:39.793 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2023-05-31T21:35:39.793 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2023-05-31T21:35:39.794 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: ] 2023-05-31T21:35:39.794 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2023-05-31T21:35:39.794 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "full_sync": { 2023-05-31T21:35:39.794 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "total": 1, 2023-05-31T21:35:39.795 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "complete": 1 2023-05-31T21:35:39.795 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2023-05-31T21:35:39.795 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "current_time": "2023-05-31T20:14:50Z" 2023-05-31T21:35:39.795 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2023-05-31T21:35:39.795 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2023-05-31T21:35:39.796 INFO:tasks.rgw_multisite_tests: "sync_status": { 2023-05-31T21:35:39.796 INFO:tasks.rgw_multisite_tests: "info": { 2023-05-31T21:35:39.796 INFO:tasks.rgw_multisite_tests: "status": "sync", 2023-05-31T21:35:39.796 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2023-05-31T21:35:39.797 INFO:tasks.rgw_multisite_tests: "period": "2e6123d2-d695-4350-9a01-61a455bc782b", 2023-05-31T21:35:39.797 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-05-31T21:35:39.797 INFO:tasks.rgw_multisite_tests: }, 2023-05-31T21:35:39.797 INFO:tasks.rgw_multisite_tests: "markers": [ 2023-05-31T21:35:39.798 INFO:tasks.rgw_multisite_tests: { 2023-05-31T21:35:39.798 INFO:tasks.rgw_multisite_tests: "key": 0, 2023-05-31T21:35:39.798 INFO:tasks.rgw_multisite_tests: "val": { 2023-05-31T21:35:39.798 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-05-31T21:35:39.798 INFO:tasks.rgw_multisite_tests: "marker": "1_1685562631.516089_59.1", 2023-05-31T21:35:39.799 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-05-31T21:35:39.799 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2023-05-31T21:35:39.799 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-05-31T21:35:39.799 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-05-31T19:50:31.516089Z", 2023-05-31T21:35:39.800 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-05-31T21:35:39.800 INFO:tasks.rgw_multisite_tests: } 2023-05-31T21:35:39.800 INFO:tasks.rgw_multisite_tests: }, 2023-05-31T21:35:39.800 INFO:tasks.rgw_multisite_tests: { 2023-05-31T21:35:39.801 INFO:tasks.rgw_multisite_tests: "key": 1, 2023-05-31T21:35:39.801 INFO:tasks.rgw_multisite_tests: "val": { 2023-05-31T21:35:39.801 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-05-31T21:35:39.801 INFO:tasks.rgw_multisite_tests: "marker": "", 2023-05-31T21:35:39.802 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-05-31T21:35:39.802 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-05-31T21:35:39.802 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-05-31T21:35:39.802 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2023-05-31T21:35:39.802 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-05-31T21:35:39.803 INFO:tasks.rgw_multisite_tests: } 2023-05-31T21:35:39.803 INFO:tasks.rgw_multisite_tests: }, 2023-05-31T21:35:39.803 INFO:tasks.rgw_multisite_tests: { 2023-05-31T21:35:39.803 INFO:tasks.rgw_multisite_tests: "key": 2, 2023-05-31T21:35:39.804 INFO:tasks.rgw_multisite_tests: "val": { 2023-05-31T21:35:39.804 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-05-31T21:35:39.804 INFO:tasks.rgw_multisite_tests: "marker": "1_1685562970.274572_93.1", 2023-05-31T21:35:39.804 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-05-31T21:35:39.805 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-05-31T21:35:39.805 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-05-31T21:35:39.805 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-05-31T19:56:10.274572Z", 2023-05-31T21:35:39.805 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-05-31T21:35:39.806 INFO:tasks.rgw_multisite_tests: } 2023-05-31T21:35:39.806 INFO:tasks.rgw_multisite_tests: }, 2023-05-31T21:35:39.806 INFO:tasks.rgw_multisite_tests: { 2023-05-31T21:35:39.806 INFO:tasks.rgw_multisite_tests: "key": 3, 2023-05-31T21:35:39.806 INFO:tasks.rgw_multisite_tests: "val": { 2023-05-31T21:35:39.807 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-05-31T21:35:39.807 INFO:tasks.rgw_multisite_tests: "marker": "1_1685563799.098535_267.1", 2023-05-31T21:35:39.807 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-05-31T21:35:39.807 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-05-31T21:35:39.808 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-05-31T21:35:39.808 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-05-31T20:09:59.098535Z", 2023-05-31T21:35:39.808 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-05-31T21:35:39.808 INFO:tasks.rgw_multisite_tests: } 2023-05-31T21:35:39.809 INFO:tasks.rgw_multisite_tests: } 2023-05-31T21:35:39.809 INFO:tasks.rgw_multisite_tests: ] 2023-05-31T21:35:39.809 INFO:tasks.rgw_multisite_tests: }, 2023-05-31T21:35:39.809 INFO:tasks.rgw_multisite_tests: "full_sync": { 2023-05-31T21:35:39.809 INFO:tasks.rgw_multisite_tests: "total": 1, 2023-05-31T21:35:39.810 INFO:tasks.rgw_multisite_tests: "complete": 1 2023-05-31T21:35:39.810 INFO:tasks.rgw_multisite_tests: }, 2023-05-31T21:35:39.810 INFO:tasks.rgw_multisite_tests: "current_time": "2023-05-31T20:14:50Z" 2023-05-31T21:35:39.810 INFO:tasks.rgw_multisite_tests:} 2023-05-31T21:35:39.811 INFO:tasks.rgw_multisite_tests: 2023-05-31T21:35:39.811 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1685562631.516089_59.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-05-31T19:50:31.516089Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1685562970.274572_93.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-05-31T19:56:10.274572Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1685563799.098535_267.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-05-31T20:09:59.098535Z', 'realm_epoch': 2}}] 2023-05-31T21:35:39.811 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1685562631.516089_59.1', 1: '', 2: '1_1685562970.274572_93.1', 3: '1_1685563802.468126_296.1'} 2023-05-31T21:35:39.811 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1685562631.516089_59.1', 1: '', 2: '1_1685562970.274572_93.1', 3: '1_1685563799.098535_267.1'} 2023-05-31T21:35:39.812 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone a2 behind master: shard=3 master=1_1685563802.468126_296.1 target=1_1685563799.098535_267.1 2023-05-31T21:35:39.812 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-31T21:35:39.812 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-31T21:35:39.812 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-05-31T21:35:39.813 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: ignoring --setuser ceph since I am not root 2023-05-31T21:35:39.813 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-05-31T21:35:39.813 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: 2023-05-31T20:14:55.726+0000 7fd5e28f5780 20 RGW-SYNC:meta: read sync status 2023-05-31T21:35:39.813 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: 2023-05-31T20:14:55.727+0000 7fd5e28f5780 20 RGW-SYNC:meta: read sync status 2023-05-31T21:35:39.813 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2023-05-31T21:35:39.814 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "sync_status": { 2023-05-31T21:35:39.814 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "info": { 2023-05-31T21:35:39.814 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "status": "sync", 2023-05-31T21:35:39.814 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "num_shards": 4, 2023-05-31T21:35:39.815 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "period": "2e6123d2-d695-4350-9a01-61a455bc782b", 2023-05-31T21:35:39.815 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "realm_epoch": 2 2023-05-31T21:35:39.815 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2023-05-31T21:35:39.815 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "markers": [ 2023-05-31T21:35:39.815 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2023-05-31T21:35:39.816 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "key": 0, 2023-05-31T21:35:39.816 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "val": { 2023-05-31T21:35:39.816 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "state": 1, 2023-05-31T21:35:39.816 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "marker": "1_1685562631.516089_59.1", 2023-05-31T21:35:39.817 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "next_step_marker": "", 2023-05-31T21:35:39.817 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "total_entries": 1, 2023-05-31T21:35:39.817 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "pos": 0, 2023-05-31T21:35:39.817 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "timestamp": "2023-05-31T19:50:31.516089Z", 2023-05-31T21:35:39.818 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "realm_epoch": 2 2023-05-31T21:35:39.818 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2023-05-31T21:35:39.818 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2023-05-31T21:35:39.818 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2023-05-31T21:35:39.819 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "key": 1, 2023-05-31T21:35:39.819 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "val": { 2023-05-31T21:35:39.819 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "state": 1, 2023-05-31T21:35:39.819 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "marker": "", 2023-05-31T21:35:39.819 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "next_step_marker": "", 2023-05-31T21:35:39.820 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "total_entries": 0, 2023-05-31T21:35:39.820 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "pos": 0, 2023-05-31T21:35:39.820 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "timestamp": "0.000000", 2023-05-31T21:35:39.820 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "realm_epoch": 2 2023-05-31T21:35:39.821 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2023-05-31T21:35:39.821 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2023-05-31T21:35:39.821 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2023-05-31T21:35:39.821 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "key": 2, 2023-05-31T21:35:39.822 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "val": { 2023-05-31T21:35:39.822 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "state": 1, 2023-05-31T21:35:39.822 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "marker": "1_1685562970.274572_93.1", 2023-05-31T21:35:39.822 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "next_step_marker": "", 2023-05-31T21:35:39.823 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "total_entries": 0, 2023-05-31T21:35:39.823 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "pos": 0, 2023-05-31T21:35:39.823 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "timestamp": "2023-05-31T19:56:10.274572Z", 2023-05-31T21:35:39.823 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "realm_epoch": 2 2023-05-31T21:35:39.823 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2023-05-31T21:35:39.824 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2023-05-31T21:35:39.824 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2023-05-31T21:35:39.824 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "key": 3, 2023-05-31T21:35:39.824 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "val": { 2023-05-31T21:35:39.825 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "state": 1, 2023-05-31T21:35:39.825 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "marker": "1_1685563798.815084_250.1", 2023-05-31T21:35:39.825 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "next_step_marker": "", 2023-05-31T21:35:39.825 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "total_entries": 0, 2023-05-31T21:35:39.826 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "pos": 0, 2023-05-31T21:35:39.826 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "timestamp": "2023-05-31T20:09:58.815084Z", 2023-05-31T21:35:39.826 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "realm_epoch": 2 2023-05-31T21:35:39.826 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2023-05-31T21:35:39.827 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2023-05-31T21:35:39.827 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: ] 2023-05-31T21:35:39.827 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2023-05-31T21:35:39.827 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "full_sync": { 2023-05-31T21:35:39.827 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "total": 1, 2023-05-31T21:35:39.828 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "complete": 1 2023-05-31T21:35:39.828 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2023-05-31T21:35:39.828 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "current_time": "2023-05-31T20:14:55Z" 2023-05-31T21:35:39.828 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2023-05-31T21:35:39.829 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2023-05-31T21:35:39.829 INFO:tasks.rgw_multisite_tests: "sync_status": { 2023-05-31T21:35:39.829 INFO:tasks.rgw_multisite_tests: "info": { 2023-05-31T21:35:39.829 INFO:tasks.rgw_multisite_tests: "status": "sync", 2023-05-31T21:35:39.829 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2023-05-31T21:35:39.830 INFO:tasks.rgw_multisite_tests: "period": "2e6123d2-d695-4350-9a01-61a455bc782b", 2023-05-31T21:35:39.830 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-05-31T21:35:39.830 INFO:tasks.rgw_multisite_tests: }, 2023-05-31T21:35:39.830 INFO:tasks.rgw_multisite_tests: "markers": [ 2023-05-31T21:35:39.831 INFO:tasks.rgw_multisite_tests: { 2023-05-31T21:35:39.831 INFO:tasks.rgw_multisite_tests: "key": 0, 2023-05-31T21:35:39.831 INFO:tasks.rgw_multisite_tests: "val": { 2023-05-31T21:35:39.831 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-05-31T21:35:39.832 INFO:tasks.rgw_multisite_tests: "marker": "1_1685562631.516089_59.1", 2023-05-31T21:35:39.832 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-05-31T21:35:39.832 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2023-05-31T21:35:39.832 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-05-31T21:35:39.833 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-05-31T19:50:31.516089Z", 2023-05-31T21:35:39.833 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-05-31T21:35:39.833 INFO:tasks.rgw_multisite_tests: } 2023-05-31T21:35:39.833 INFO:tasks.rgw_multisite_tests: }, 2023-05-31T21:35:39.833 INFO:tasks.rgw_multisite_tests: { 2023-05-31T21:35:39.834 INFO:tasks.rgw_multisite_tests: "key": 1, 2023-05-31T21:35:39.834 INFO:tasks.rgw_multisite_tests: "val": { 2023-05-31T21:35:39.834 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-05-31T21:35:39.834 INFO:tasks.rgw_multisite_tests: "marker": "", 2023-05-31T21:35:39.835 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-05-31T21:35:39.835 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-05-31T21:35:39.835 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-05-31T21:35:39.835 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2023-05-31T21:35:39.836 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-05-31T21:35:39.836 INFO:tasks.rgw_multisite_tests: } 2023-05-31T21:35:39.836 INFO:tasks.rgw_multisite_tests: }, 2023-05-31T21:35:39.836 INFO:tasks.rgw_multisite_tests: { 2023-05-31T21:35:39.836 INFO:tasks.rgw_multisite_tests: "key": 2, 2023-05-31T21:35:39.837 INFO:tasks.rgw_multisite_tests: "val": { 2023-05-31T21:35:39.837 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-05-31T21:35:39.837 INFO:tasks.rgw_multisite_tests: "marker": "1_1685562970.274572_93.1", 2023-05-31T21:35:39.837 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-05-31T21:35:39.838 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-05-31T21:35:39.838 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-05-31T21:35:39.838 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-05-31T19:56:10.274572Z", 2023-05-31T21:35:39.838 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-05-31T21:35:39.839 INFO:tasks.rgw_multisite_tests: } 2023-05-31T21:35:39.839 INFO:tasks.rgw_multisite_tests: }, 2023-05-31T21:35:39.839 INFO:tasks.rgw_multisite_tests: { 2023-05-31T21:35:39.839 INFO:tasks.rgw_multisite_tests: "key": 3, 2023-05-31T21:35:39.839 INFO:tasks.rgw_multisite_tests: "val": { 2023-05-31T21:35:39.840 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-05-31T21:35:39.840 INFO:tasks.rgw_multisite_tests: "marker": "1_1685563798.815084_250.1", 2023-05-31T21:35:39.840 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-05-31T21:35:39.840 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-05-31T21:35:39.841 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-05-31T21:35:39.841 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-05-31T20:09:58.815084Z", 2023-05-31T21:35:39.841 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-05-31T21:35:39.841 INFO:tasks.rgw_multisite_tests: } 2023-05-31T21:35:39.842 INFO:tasks.rgw_multisite_tests: } 2023-05-31T21:35:39.842 INFO:tasks.rgw_multisite_tests: ] 2023-05-31T21:35:39.842 INFO:tasks.rgw_multisite_tests: }, 2023-05-31T21:35:39.842 INFO:tasks.rgw_multisite_tests: "full_sync": { 2023-05-31T21:35:39.842 INFO:tasks.rgw_multisite_tests: "total": 1, 2023-05-31T21:35:39.843 INFO:tasks.rgw_multisite_tests: "complete": 1 2023-05-31T21:35:39.843 INFO:tasks.rgw_multisite_tests: }, 2023-05-31T21:35:39.843 INFO:tasks.rgw_multisite_tests: "current_time": "2023-05-31T20:14:55Z" 2023-05-31T21:35:39.843 INFO:tasks.rgw_multisite_tests:} 2023-05-31T21:35:39.844 INFO:tasks.rgw_multisite_tests: 2023-05-31T21:35:39.844 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1685562631.516089_59.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-05-31T19:50:31.516089Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1685562970.274572_93.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-05-31T19:56:10.274572Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1685563798.815084_250.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-05-31T20:09:58.815084Z', 'realm_epoch': 2}}] 2023-05-31T21:35:39.844 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1685562631.516089_59.1', 1: '', 2: '1_1685562970.274572_93.1', 3: '1_1685563802.468126_296.1'} 2023-05-31T21:35:39.844 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1685562631.516089_59.1', 1: '', 2: '1_1685562970.274572_93.1', 3: '1_1685563798.815084_250.1'} 2023-05-31T21:35:39.845 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone a2 behind master: shard=3 master=1_1685563802.468126_296.1 target=1_1685563798.815084_250.1 2023-05-31T21:35:39.845 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-31T21:35:39.845 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-31T21:35:39.845 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-05-31T21:35:39.846 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: ignoring --setuser ceph since I am not root 2023-05-31T21:35:39.846 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-05-31T21:35:39.846 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: 2023-05-31T20:15:00.874+0000 7fb53b89f780 20 RGW-SYNC:meta: read sync status 2023-05-31T21:35:39.846 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: 2023-05-31T20:15:00.875+0000 7fb53b89f780 20 RGW-SYNC:meta: read sync status 2023-05-31T21:35:39.846 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2023-05-31T21:35:39.847 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "sync_status": { 2023-05-31T21:35:39.847 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "info": { 2023-05-31T21:35:39.847 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "status": "sync", 2023-05-31T21:35:39.847 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "num_shards": 4, 2023-05-31T21:35:39.848 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "period": "2e6123d2-d695-4350-9a01-61a455bc782b", 2023-05-31T21:35:39.848 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "realm_epoch": 2 2023-05-31T21:35:39.848 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2023-05-31T21:35:39.848 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "markers": [ 2023-05-31T21:35:39.849 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2023-05-31T21:35:39.849 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "key": 0, 2023-05-31T21:35:39.849 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "val": { 2023-05-31T21:35:39.849 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "state": 1, 2023-05-31T21:35:39.849 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "marker": "1_1685562631.516089_59.1", 2023-05-31T21:35:39.850 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "next_step_marker": "", 2023-05-31T21:35:39.850 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "total_entries": 1, 2023-05-31T21:35:39.850 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "pos": 0, 2023-05-31T21:35:39.850 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "timestamp": "2023-05-31T19:50:31.516089Z", 2023-05-31T21:35:39.851 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "realm_epoch": 2 2023-05-31T21:35:39.851 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2023-05-31T21:35:39.851 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2023-05-31T21:35:39.851 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2023-05-31T21:35:39.852 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "key": 1, 2023-05-31T21:35:39.852 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "val": { 2023-05-31T21:35:39.852 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "state": 1, 2023-05-31T21:35:39.852 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "marker": "", 2023-05-31T21:35:39.852 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "next_step_marker": "", 2023-05-31T21:35:39.853 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "total_entries": 0, 2023-05-31T21:35:39.853 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "pos": 0, 2023-05-31T21:35:39.853 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "timestamp": "0.000000", 2023-05-31T21:35:39.853 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "realm_epoch": 2 2023-05-31T21:35:39.854 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2023-05-31T21:35:39.854 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2023-05-31T21:35:39.854 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2023-05-31T21:35:39.854 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "key": 2, 2023-05-31T21:35:39.855 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "val": { 2023-05-31T21:35:39.855 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "state": 1, 2023-05-31T21:35:39.855 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "marker": "1_1685562970.274572_93.1", 2023-05-31T21:35:39.855 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "next_step_marker": "", 2023-05-31T21:35:39.855 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "total_entries": 0, 2023-05-31T21:35:39.856 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "pos": 0, 2023-05-31T21:35:39.856 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "timestamp": "2023-05-31T19:56:10.274572Z", 2023-05-31T21:35:39.856 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "realm_epoch": 2 2023-05-31T21:35:39.856 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2023-05-31T21:35:39.857 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2023-05-31T21:35:39.857 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2023-05-31T21:35:39.857 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "key": 3, 2023-05-31T21:35:39.857 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "val": { 2023-05-31T21:35:39.857 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "state": 1, 2023-05-31T21:35:39.858 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "marker": "1_1685563798.827280_252.1", 2023-05-31T21:35:39.858 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "next_step_marker": "", 2023-05-31T21:35:39.858 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "total_entries": 0, 2023-05-31T21:35:39.858 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "pos": 0, 2023-05-31T21:35:39.859 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "timestamp": "2023-05-31T20:09:58.827280Z", 2023-05-31T21:35:39.859 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "realm_epoch": 2 2023-05-31T21:35:39.859 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2023-05-31T21:35:39.859 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2023-05-31T21:35:39.860 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: ] 2023-05-31T21:35:39.860 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2023-05-31T21:35:39.860 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "full_sync": { 2023-05-31T21:35:39.860 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "total": 1, 2023-05-31T21:35:39.860 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "complete": 1 2023-05-31T21:35:39.861 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2023-05-31T21:35:39.861 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "current_time": "2023-05-31T20:15:00Z" 2023-05-31T21:35:39.861 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2023-05-31T21:35:39.861 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2023-05-31T21:35:39.862 INFO:tasks.rgw_multisite_tests: "sync_status": { 2023-05-31T21:35:39.862 INFO:tasks.rgw_multisite_tests: "info": { 2023-05-31T21:35:39.862 INFO:tasks.rgw_multisite_tests: "status": "sync", 2023-05-31T21:35:39.862 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2023-05-31T21:35:39.863 INFO:tasks.rgw_multisite_tests: "period": "2e6123d2-d695-4350-9a01-61a455bc782b", 2023-05-31T21:35:39.863 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-05-31T21:35:39.863 INFO:tasks.rgw_multisite_tests: }, 2023-05-31T21:35:39.863 INFO:tasks.rgw_multisite_tests: "markers": [ 2023-05-31T21:35:39.863 INFO:tasks.rgw_multisite_tests: { 2023-05-31T21:35:39.864 INFO:tasks.rgw_multisite_tests: "key": 0, 2023-05-31T21:35:39.864 INFO:tasks.rgw_multisite_tests: "val": { 2023-05-31T21:35:39.864 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-05-31T21:35:39.864 INFO:tasks.rgw_multisite_tests: "marker": "1_1685562631.516089_59.1", 2023-05-31T21:35:39.865 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-05-31T21:35:39.865 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2023-05-31T21:35:39.865 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-05-31T21:35:39.865 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-05-31T19:50:31.516089Z", 2023-05-31T21:35:39.866 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-05-31T21:35:39.866 INFO:tasks.rgw_multisite_tests: } 2023-05-31T21:35:39.866 INFO:tasks.rgw_multisite_tests: }, 2023-05-31T21:35:39.866 INFO:tasks.rgw_multisite_tests: { 2023-05-31T21:35:39.866 INFO:tasks.rgw_multisite_tests: "key": 1, 2023-05-31T21:35:39.867 INFO:tasks.rgw_multisite_tests: "val": { 2023-05-31T21:35:39.867 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-05-31T21:35:39.867 INFO:tasks.rgw_multisite_tests: "marker": "", 2023-05-31T21:35:39.867 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-05-31T21:35:39.868 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-05-31T21:35:39.868 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-05-31T21:35:39.868 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2023-05-31T21:35:39.868 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-05-31T21:35:39.868 INFO:tasks.rgw_multisite_tests: } 2023-05-31T21:35:39.869 INFO:tasks.rgw_multisite_tests: }, 2023-05-31T21:35:39.869 INFO:tasks.rgw_multisite_tests: { 2023-05-31T21:35:39.869 INFO:tasks.rgw_multisite_tests: "key": 2, 2023-05-31T21:35:39.869 INFO:tasks.rgw_multisite_tests: "val": { 2023-05-31T21:35:39.870 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-05-31T21:35:39.870 INFO:tasks.rgw_multisite_tests: "marker": "1_1685562970.274572_93.1", 2023-05-31T21:35:39.870 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-05-31T21:35:39.870 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-05-31T21:35:39.871 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-05-31T21:35:39.871 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-05-31T19:56:10.274572Z", 2023-05-31T21:35:39.871 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-05-31T21:35:39.871 INFO:tasks.rgw_multisite_tests: } 2023-05-31T21:35:39.871 INFO:tasks.rgw_multisite_tests: }, 2023-05-31T21:35:39.872 INFO:tasks.rgw_multisite_tests: { 2023-05-31T21:35:39.872 INFO:tasks.rgw_multisite_tests: "key": 3, 2023-05-31T21:35:39.872 INFO:tasks.rgw_multisite_tests: "val": { 2023-05-31T21:35:39.872 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-05-31T21:35:39.873 INFO:tasks.rgw_multisite_tests: "marker": "1_1685563798.827280_252.1", 2023-05-31T21:35:39.873 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-05-31T21:35:39.873 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-05-31T21:35:39.873 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-05-31T21:35:39.873 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-05-31T20:09:58.827280Z", 2023-05-31T21:35:39.874 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-05-31T21:35:39.874 INFO:tasks.rgw_multisite_tests: } 2023-05-31T21:35:39.874 INFO:tasks.rgw_multisite_tests: } 2023-05-31T21:35:39.875 INFO:tasks.rgw_multisite_tests: ] 2023-05-31T21:35:39.875 INFO:tasks.rgw_multisite_tests: }, 2023-05-31T21:35:39.875 INFO:tasks.rgw_multisite_tests: "full_sync": { 2023-05-31T21:35:39.875 INFO:tasks.rgw_multisite_tests: "total": 1, 2023-05-31T21:35:39.876 INFO:tasks.rgw_multisite_tests: "complete": 1 2023-05-31T21:35:39.876 INFO:tasks.rgw_multisite_tests: }, 2023-05-31T21:35:39.876 INFO:tasks.rgw_multisite_tests: "current_time": "2023-05-31T20:15:00Z" 2023-05-31T21:35:39.876 INFO:tasks.rgw_multisite_tests:} 2023-05-31T21:35:39.876 INFO:tasks.rgw_multisite_tests: 2023-05-31T21:35:39.877 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1685562631.516089_59.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-05-31T19:50:31.516089Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1685562970.274572_93.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-05-31T19:56:10.274572Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1685563798.827280_252.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-05-31T20:09:58.827280Z', 'realm_epoch': 2}}] 2023-05-31T21:35:39.877 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1685562631.516089_59.1', 1: '', 2: '1_1685562970.274572_93.1', 3: '1_1685563802.468126_296.1'} 2023-05-31T21:35:39.877 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1685562631.516089_59.1', 1: '', 2: '1_1685562970.274572_93.1', 3: '1_1685563798.827280_252.1'} 2023-05-31T21:35:39.877 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone a2 behind master: shard=3 master=1_1685563802.468126_296.1 target=1_1685563798.827280_252.1 2023-05-31T21:35:39.878 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-31T21:35:39.878 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-31T21:35:39.878 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-05-31T21:35:39.878 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: ignoring --setuser ceph since I am not root 2023-05-31T21:35:39.879 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-05-31T21:35:39.879 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: 2023-05-31T20:15:06.006+0000 7fd05fd93780 20 RGW-SYNC:meta: read sync status 2023-05-31T21:35:39.879 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: 2023-05-31T20:15:06.007+0000 7fd05fd93780 20 RGW-SYNC:meta: read sync status 2023-05-31T21:35:39.879 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2023-05-31T21:35:39.880 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "sync_status": { 2023-05-31T21:35:39.880 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "info": { 2023-05-31T21:35:39.880 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "status": "sync", 2023-05-31T21:35:39.880 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "num_shards": 4, 2023-05-31T21:35:39.880 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "period": "2e6123d2-d695-4350-9a01-61a455bc782b", 2023-05-31T21:35:39.881 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "realm_epoch": 2 2023-05-31T21:35:39.881 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2023-05-31T21:35:39.881 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "markers": [ 2023-05-31T21:35:39.881 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2023-05-31T21:35:39.882 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "key": 0, 2023-05-31T21:35:39.882 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "val": { 2023-05-31T21:35:39.882 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "state": 1, 2023-05-31T21:35:39.882 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "marker": "1_1685562631.516089_59.1", 2023-05-31T21:35:39.882 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "next_step_marker": "", 2023-05-31T21:35:39.883 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "total_entries": 1, 2023-05-31T21:35:39.883 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "pos": 0, 2023-05-31T21:35:39.883 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "timestamp": "2023-05-31T19:50:31.516089Z", 2023-05-31T21:35:39.883 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "realm_epoch": 2 2023-05-31T21:35:39.884 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2023-05-31T21:35:39.884 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2023-05-31T21:35:39.884 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2023-05-31T21:35:39.884 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "key": 1, 2023-05-31T21:35:39.885 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "val": { 2023-05-31T21:35:39.885 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "state": 1, 2023-05-31T21:35:39.885 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "marker": "", 2023-05-31T21:35:39.885 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "next_step_marker": "", 2023-05-31T21:35:39.886 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "total_entries": 0, 2023-05-31T21:35:39.886 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "pos": 0, 2023-05-31T21:35:39.886 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "timestamp": "0.000000", 2023-05-31T21:35:39.886 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "realm_epoch": 2 2023-05-31T21:35:39.886 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2023-05-31T21:35:39.887 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2023-05-31T21:35:39.887 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2023-05-31T21:35:39.887 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "key": 2, 2023-05-31T21:35:39.887 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "val": { 2023-05-31T21:35:39.888 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "state": 1, 2023-05-31T21:35:39.888 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "marker": "1_1685562970.274572_93.1", 2023-05-31T21:35:39.888 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "next_step_marker": "", 2023-05-31T21:35:39.888 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "total_entries": 0, 2023-05-31T21:35:39.889 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "pos": 0, 2023-05-31T21:35:39.889 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "timestamp": "2023-05-31T19:56:10.274572Z", 2023-05-31T21:35:39.889 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "realm_epoch": 2 2023-05-31T21:35:39.889 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2023-05-31T21:35:39.889 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2023-05-31T21:35:39.890 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2023-05-31T21:35:39.890 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "key": 3, 2023-05-31T21:35:39.890 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "val": { 2023-05-31T21:35:39.890 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "state": 1, 2023-05-31T21:35:39.891 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "marker": "1_1685563798.760350_247.1", 2023-05-31T21:35:39.891 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "next_step_marker": "", 2023-05-31T21:35:39.891 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "total_entries": 0, 2023-05-31T21:35:39.891 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "pos": 0, 2023-05-31T21:35:39.891 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "timestamp": "2023-05-31T20:09:58.760350Z", 2023-05-31T21:35:39.892 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "realm_epoch": 2 2023-05-31T21:35:39.892 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2023-05-31T21:35:39.892 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2023-05-31T21:35:39.892 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: ] 2023-05-31T21:35:39.893 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2023-05-31T21:35:39.893 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "full_sync": { 2023-05-31T21:35:39.893 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "total": 1, 2023-05-31T21:35:39.893 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "complete": 1 2023-05-31T21:35:39.894 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2023-05-31T21:35:39.894 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "current_time": "2023-05-31T20:15:06Z" 2023-05-31T21:35:39.894 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2023-05-31T21:35:39.894 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2023-05-31T21:35:39.894 INFO:tasks.rgw_multisite_tests: "sync_status": { 2023-05-31T21:35:39.895 INFO:tasks.rgw_multisite_tests: "info": { 2023-05-31T21:35:39.895 INFO:tasks.rgw_multisite_tests: "status": "sync", 2023-05-31T21:35:39.895 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2023-05-31T21:35:39.895 INFO:tasks.rgw_multisite_tests: "period": "2e6123d2-d695-4350-9a01-61a455bc782b", 2023-05-31T21:35:39.896 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-05-31T21:35:39.896 INFO:tasks.rgw_multisite_tests: }, 2023-05-31T21:35:39.896 INFO:tasks.rgw_multisite_tests: "markers": [ 2023-05-31T21:35:39.896 INFO:tasks.rgw_multisite_tests: { 2023-05-31T21:35:39.897 INFO:tasks.rgw_multisite_tests: "key": 0, 2023-05-31T21:35:39.897 INFO:tasks.rgw_multisite_tests: "val": { 2023-05-31T21:35:39.897 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-05-31T21:35:39.897 INFO:tasks.rgw_multisite_tests: "marker": "1_1685562631.516089_59.1", 2023-05-31T21:35:39.897 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-05-31T21:35:39.898 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2023-05-31T21:35:39.898 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-05-31T21:35:39.898 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-05-31T19:50:31.516089Z", 2023-05-31T21:35:39.898 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-05-31T21:35:39.899 INFO:tasks.rgw_multisite_tests: } 2023-05-31T21:35:39.899 INFO:tasks.rgw_multisite_tests: }, 2023-05-31T21:35:39.899 INFO:tasks.rgw_multisite_tests: { 2023-05-31T21:35:39.899 INFO:tasks.rgw_multisite_tests: "key": 1, 2023-05-31T21:35:39.900 INFO:tasks.rgw_multisite_tests: "val": { 2023-05-31T21:35:39.900 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-05-31T21:35:39.900 INFO:tasks.rgw_multisite_tests: "marker": "", 2023-05-31T21:35:39.900 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-05-31T21:35:39.900 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-05-31T21:35:39.901 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-05-31T21:35:39.901 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2023-05-31T21:35:39.901 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-05-31T21:35:39.901 INFO:tasks.rgw_multisite_tests: } 2023-05-31T21:35:39.902 INFO:tasks.rgw_multisite_tests: }, 2023-05-31T21:35:39.902 INFO:tasks.rgw_multisite_tests: { 2023-05-31T21:35:39.902 INFO:tasks.rgw_multisite_tests: "key": 2, 2023-05-31T21:35:39.902 INFO:tasks.rgw_multisite_tests: "val": { 2023-05-31T21:35:39.903 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-05-31T21:35:39.903 INFO:tasks.rgw_multisite_tests: "marker": "1_1685562970.274572_93.1", 2023-05-31T21:35:39.903 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-05-31T21:35:39.903 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-05-31T21:35:39.904 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-05-31T21:35:39.904 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-05-31T19:56:10.274572Z", 2023-05-31T21:35:39.904 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-05-31T21:35:39.904 INFO:tasks.rgw_multisite_tests: } 2023-05-31T21:35:39.905 INFO:tasks.rgw_multisite_tests: }, 2023-05-31T21:35:39.905 INFO:tasks.rgw_multisite_tests: { 2023-05-31T21:35:39.905 INFO:tasks.rgw_multisite_tests: "key": 3, 2023-05-31T21:35:39.905 INFO:tasks.rgw_multisite_tests: "val": { 2023-05-31T21:35:39.905 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-05-31T21:35:39.906 INFO:tasks.rgw_multisite_tests: "marker": "1_1685563798.760350_247.1", 2023-05-31T21:35:39.906 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-05-31T21:35:39.906 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-05-31T21:35:39.906 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-05-31T21:35:39.907 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-05-31T20:09:58.760350Z", 2023-05-31T21:35:39.907 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-05-31T21:35:39.907 INFO:tasks.rgw_multisite_tests: } 2023-05-31T21:35:39.907 INFO:tasks.rgw_multisite_tests: } 2023-05-31T21:35:39.908 INFO:tasks.rgw_multisite_tests: ] 2023-05-31T21:35:39.908 INFO:tasks.rgw_multisite_tests: }, 2023-05-31T21:35:39.908 INFO:tasks.rgw_multisite_tests: "full_sync": { 2023-05-31T21:35:39.908 INFO:tasks.rgw_multisite_tests: "total": 1, 2023-05-31T21:35:39.909 INFO:tasks.rgw_multisite_tests: "complete": 1 2023-05-31T21:35:39.909 INFO:tasks.rgw_multisite_tests: }, 2023-05-31T21:35:39.909 INFO:tasks.rgw_multisite_tests: "current_time": "2023-05-31T20:15:06Z" 2023-05-31T21:35:39.909 INFO:tasks.rgw_multisite_tests:} 2023-05-31T21:35:39.909 INFO:tasks.rgw_multisite_tests: 2023-05-31T21:35:39.910 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1685562631.516089_59.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-05-31T19:50:31.516089Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1685562970.274572_93.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-05-31T19:56:10.274572Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1685563798.760350_247.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-05-31T20:09:58.760350Z', 'realm_epoch': 2}}] 2023-05-31T21:35:39.910 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1685562631.516089_59.1', 1: '', 2: '1_1685562970.274572_93.1', 3: '1_1685563802.468126_296.1'} 2023-05-31T21:35:39.910 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1685562631.516089_59.1', 1: '', 2: '1_1685562970.274572_93.1', 3: '1_1685563798.760350_247.1'} 2023-05-31T21:35:39.910 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone a2 behind master: shard=3 master=1_1685563802.468126_296.1 target=1_1685563798.760350_247.1 2023-05-31T21:35:39.911 INFO:tasks.rgw_multisite_tests:--------------------- >> end captured logging << --------------------- 2023-05-31T21:35:39.911 INFO:tasks.rgw_multisite_tests: 2023-05-31T21:35:39.912 INFO:tasks.rgw_multisite_tests:====================================================================== 2023-05-31T21:35:39.912 INFO:tasks.rgw_multisite_tests:FAIL: rgw_multi.tests.test_encrypted_object_sync 2023-05-31T21:35:39.912 INFO:tasks.rgw_multisite_tests:---------------------------------------------------------------------- 2023-05-31T21:35:39.912 INFO:tasks.rgw_multisite_tests:Traceback (most recent call last): 2023-05-31T21:35:39.913 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_teuthology_fa97ddd17d12e6d74b49196948f66e54fa009130/virtualenv/lib/python3.8/site-packages/nose/case.py", line 198, in runTest 2023-05-31T21:35:39.913 INFO:tasks.rgw_multisite_tests: self.test(*self.arg) 2023-05-31T21:35:39.913 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/github.com_ceph_ceph-c_2870bf456d3115490c153abce9d47ac5c043b65e/qa/../src/test/rgw/rgw_multi/tests.py", line 1281, in test_encrypted_object_sync 2023-05-31T21:35:39.914 INFO:tasks.rgw_multisite_tests: zone_bucket_checkpoint(zone2.zone, zone1.zone, bucket_name) 2023-05-31T21:35:39.914 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/github.com_ceph_ceph-c_2870bf456d3115490c153abce9d47ac5c043b65e/qa/../src/test/rgw/rgw_multi/tests.py", line 387, in zone_bucket_checkpoint 2023-05-31T21:35:39.914 INFO:tasks.rgw_multisite_tests: target_zone.cluster.admin(cmd, debug_rgw=1) 2023-05-31T21:35:39.914 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/github.com_ceph_ceph-c_2870bf456d3115490c153abce9d47ac5c043b65e/qa/tasks/rgw_multisite.py", line 197, in admin 2023-05-31T21:35:39.914 INFO:tasks.rgw_multisite_tests: assert r == 0 2023-05-31T21:35:39.915 INFO:tasks.rgw_multisite_tests:AssertionError: 2023-05-31T21:35:39.915 INFO:tasks.rgw_multisite_tests:-------------------- >> begin captured logging << -------------------- 2023-05-31T21:35:39.915 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: INFO: create bucket zone=a1 name=pklxkd-52 2023-05-31T21:35:39.915 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/pklxkd-52/ 2023-05-31T21:35:39.916 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/pklxkd-52/ 2023-05-31T21:35:39.916 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2023-05-31T21:35:39.916 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /pklxkd-52/ 2023-05-31T21:35:39.916 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-05-31T21:35:39.917 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-05-31T21:35:39.917 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T21:35:39.917 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-05-31T21:35:39.917 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-05-31T21:35:39.918 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-05-31T21:35:39.918 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-05-31T21:35:39.918 INFO:tasks.rgw_multisite_tests:PUT 2023-05-31T21:35:39.918 INFO:tasks.rgw_multisite_tests: 2023-05-31T21:35:39.918 INFO:tasks.rgw_multisite_tests: 2023-05-31T21:35:39.919 INFO:tasks.rgw_multisite_tests:Wed, 31 May 2023 20:21:31 GMT 2023-05-31T21:35:39.919 INFO:tasks.rgw_multisite_tests:/pklxkd-52/ 2023-05-31T21:35:39.919 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-05-31T21:35:39.919 INFO:tasks.rgw_multisite_tests:AWS Q80H11OLNU1S48PPYL26:KkRxGeOTtByY3V16EXCkKQ+Z6E4= 2023-05-31T21:35:39.920 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:21:31 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:KkRxGeOTtByY3V16EXCkKQ+Z6E4=', 'Content-Length': '0'} 2023-05-31T21:35:39.920 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('x-amz-request-id', 'tx000003ad4692bb338aa5c-006477ac4b-1630-a1'), ('Content-Length', '0'), ('Date', 'Wed, 31 May 2023 20:21:31 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:35:39.920 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/pklxkd-52/testobj-sse-c 2023-05-31T21:35:39.920 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/pklxkd-52/testobj-sse-c 2023-05-31T21:35:39.921 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2023-05-31T21:35:39.921 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /pklxkd-52/testobj-sse-c 2023-05-31T21:35:39.921 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-05-31T21:35:39.921 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'x-amz-server-side-encryption-customer-algorithm': 'AES256', 'x-amz-server-side-encryption-customer-key': 'pO3upElrwuEXSoFwCfnZPdSsmt/xWeFa0N9KgDijwVs=', 'x-amz-server-side-encryption-customer-key-md5': 'DWygnHRtgiJ77HCm+1rvHw==', 'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': '3FCGuEcom6i4veFJuDiBdQ==', 'Content-Length': '512', 'Expect': '100-Continue'} 2023-05-31T21:35:39.922 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T21:35:39.922 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-05-31T21:35:39.922 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-05-31T21:35:39.922 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-05-31T21:35:39.922 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-05-31T21:35:39.923 INFO:tasks.rgw_multisite_tests:PUT 2023-05-31T21:35:39.923 INFO:tasks.rgw_multisite_tests:3FCGuEcom6i4veFJuDiBdQ== 2023-05-31T21:35:39.923 INFO:tasks.rgw_multisite_tests:application/octet-stream 2023-05-31T21:35:39.923 INFO:tasks.rgw_multisite_tests:Wed, 31 May 2023 20:21:31 GMT 2023-05-31T21:35:39.924 INFO:tasks.rgw_multisite_tests:x-amz-server-side-encryption-customer-algorithm:AES256 2023-05-31T21:35:39.924 INFO:tasks.rgw_multisite_tests:x-amz-server-side-encryption-customer-key:pO3upElrwuEXSoFwCfnZPdSsmt/xWeFa0N9KgDijwVs= 2023-05-31T21:35:39.924 INFO:tasks.rgw_multisite_tests:x-amz-server-side-encryption-customer-key-md5:DWygnHRtgiJ77HCm+1rvHw== 2023-05-31T21:35:39.924 INFO:tasks.rgw_multisite_tests:/pklxkd-52/testobj-sse-c 2023-05-31T21:35:39.925 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-05-31T21:35:39.925 INFO:tasks.rgw_multisite_tests:AWS Q80H11OLNU1S48PPYL26:riRVW3zbKUmgoMIOt87852QFw6A= 2023-05-31T21:35:39.925 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'x-amz-server-side-encryption-customer-algorithm': 'AES256', 'x-amz-server-side-encryption-customer-key': 'pO3upElrwuEXSoFwCfnZPdSsmt/xWeFa0N9KgDijwVs=', 'x-amz-server-side-encryption-customer-key-md5': 'DWygnHRtgiJ77HCm+1rvHw==', 'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': '3FCGuEcom6i4veFJuDiBdQ==', 'Content-Length': '512', 'Expect': '100-Continue', 'Date': 'Wed, 31 May 2023 20:21:31 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:riRVW3zbKUmgoMIOt87852QFw6A='} 2023-05-31T21:35:39.925 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"dc5086b847289ba8b8bde149b8388175"'), ('Accept-Ranges', 'bytes'), ('x-amz-server-side-encryption-customer-algorithm', 'AES256'), ('x-amz-server-side-encryption-customer-key-MD5', 'DWygnHRtgiJ77HCm+1rvHw=='), ('x-amz-request-id', 'tx00000d9be278d3f38473d-006477ac4b-1630-a1'), ('Date', 'Wed, 31 May 2023 20:21:31 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:35:39.925 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/pklxkd-52/testobj-sse-kms 2023-05-31T21:35:39.926 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/pklxkd-52/testobj-sse-kms 2023-05-31T21:35:39.926 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2023-05-31T21:35:39.926 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /pklxkd-52/testobj-sse-kms 2023-05-31T21:35:39.926 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-05-31T21:35:39.927 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'x-amz-server-side-encryption': 'aws:kms', 'x-amz-server-side-encryption-aws-kms-key-id': 'testkey-1', 'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': '3FCGuEcom6i4veFJuDiBdQ==', 'Content-Length': '512', 'Expect': '100-Continue'} 2023-05-31T21:35:39.927 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T21:35:39.927 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-05-31T21:35:39.927 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-05-31T21:35:39.928 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-05-31T21:35:39.928 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-05-31T21:35:39.928 INFO:tasks.rgw_multisite_tests:PUT 2023-05-31T21:35:39.928 INFO:tasks.rgw_multisite_tests:3FCGuEcom6i4veFJuDiBdQ== 2023-05-31T21:35:39.929 INFO:tasks.rgw_multisite_tests:application/octet-stream 2023-05-31T21:35:39.929 INFO:tasks.rgw_multisite_tests:Wed, 31 May 2023 20:21:32 GMT 2023-05-31T21:35:39.929 INFO:tasks.rgw_multisite_tests:x-amz-server-side-encryption:aws:kms 2023-05-31T21:35:39.929 INFO:tasks.rgw_multisite_tests:x-amz-server-side-encryption-aws-kms-key-id:testkey-1 2023-05-31T21:35:39.929 INFO:tasks.rgw_multisite_tests:/pklxkd-52/testobj-sse-kms 2023-05-31T21:35:39.930 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-05-31T21:35:39.930 INFO:tasks.rgw_multisite_tests:AWS Q80H11OLNU1S48PPYL26:9Pnqeur1PMlgPoKnLxG3axpxSFQ= 2023-05-31T21:35:39.930 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'x-amz-server-side-encryption': 'aws:kms', 'x-amz-server-side-encryption-aws-kms-key-id': 'testkey-1', 'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': '3FCGuEcom6i4veFJuDiBdQ==', 'Content-Length': '512', 'Expect': '100-Continue', 'Date': 'Wed, 31 May 2023 20:21:32 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:9Pnqeur1PMlgPoKnLxG3axpxSFQ='} 2023-05-31T21:35:39.930 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"dc5086b847289ba8b8bde149b8388175"'), ('Accept-Ranges', 'bytes'), ('x-amz-server-side-encryption', 'aws:kms'), ('x-amz-server-side-encryption-aws-kms-key-id', 'testkey-1'), ('x-amz-server-side-encryption-context', 'eyJhd3M6czM6YXJuIjoiYXJuOmF3czpzMzo6cmd3LW11bHRpc2l0ZS10ZXN0LXVzZXI6cGtseGtkLTUyL3Rlc3RvYmotc3NlLWttcyJ9'), ('x-amz-request-id', 'tx00000d0ac52938014bfab-006477ac4c-1630-a1'), ('Date', 'Wed, 31 May 2023 20:21:32 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:35:39.931 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-31T21:35:39.931 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-31T21:35:39.931 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-05-31T21:35:39.931 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: ignoring --setuser ceph since I am not root 2023-05-31T21:35:39.932 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-05-31T21:35:39.932 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: No --period given, using current period=2e6123d2-d695-4350-9a01-61a455bc782b 2023-05-31T21:35:39.932 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: [ 2023-05-31T21:35:39.932 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2023-05-31T21:35:39.932 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "marker": "1_1685562631.516089_59.1", 2023-05-31T21:35:39.933 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "last_update": "2023-05-31T19:50:31.516089Z" 2023-05-31T21:35:39.933 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2023-05-31T21:35:39.933 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2023-05-31T21:35:39.933 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "marker": "", 2023-05-31T21:35:39.934 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "last_update": "0.000000" 2023-05-31T21:35:39.934 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2023-05-31T21:35:39.934 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2023-05-31T21:35:39.934 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "marker": "1_1685562970.274572_93.1", 2023-05-31T21:35:39.935 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "last_update": "2023-05-31T19:56:10.274572Z" 2023-05-31T21:35:39.935 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2023-05-31T21:35:39.935 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2023-05-31T21:35:39.935 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "marker": "1_1685564491.970984_346.1", 2023-05-31T21:35:39.935 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "last_update": "2023-05-31T20:21:31.970984Z" 2023-05-31T21:35:39.936 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2023-05-31T21:35:39.936 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: ] 2023-05-31T21:35:39.936 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: master meta markers={0: '1_1685562631.516089_59.1', 1: '', 2: '1_1685562970.274572_93.1', 3: '1_1685564491.970984_346.1'} 2023-05-31T21:35:39.936 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: INFO: starting meta checkpoint for zone=a2 2023-05-31T21:35:39.937 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-31T21:35:39.937 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-31T21:35:39.937 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-05-31T21:35:39.937 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: ignoring --setuser ceph since I am not root 2023-05-31T21:35:39.938 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-05-31T21:35:39.938 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: 2023-05-31T20:21:32.265+0000 7fbcfed50780 20 RGW-SYNC:meta: read sync status 2023-05-31T21:35:39.938 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: 2023-05-31T20:21:32.266+0000 7fbcfed50780 20 RGW-SYNC:meta: read sync status 2023-05-31T21:35:39.938 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2023-05-31T21:35:39.939 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "sync_status": { 2023-05-31T21:35:39.939 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "info": { 2023-05-31T21:35:39.939 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "status": "sync", 2023-05-31T21:35:39.939 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "num_shards": 4, 2023-05-31T21:35:39.939 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "period": "2e6123d2-d695-4350-9a01-61a455bc782b", 2023-05-31T21:35:39.940 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "realm_epoch": 2 2023-05-31T21:35:39.940 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2023-05-31T21:35:39.940 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "markers": [ 2023-05-31T21:35:39.940 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2023-05-31T21:35:39.941 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "key": 0, 2023-05-31T21:35:39.941 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "val": { 2023-05-31T21:35:39.941 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "state": 1, 2023-05-31T21:35:39.941 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "marker": "1_1685562631.516089_59.1", 2023-05-31T21:35:39.942 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "next_step_marker": "", 2023-05-31T21:35:39.942 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "total_entries": 1, 2023-05-31T21:35:39.942 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "pos": 0, 2023-05-31T21:35:39.942 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "timestamp": "2023-05-31T19:50:31.516089Z", 2023-05-31T21:35:39.943 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "realm_epoch": 2 2023-05-31T21:35:39.943 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2023-05-31T21:35:39.943 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2023-05-31T21:35:39.943 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2023-05-31T21:35:39.943 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "key": 1, 2023-05-31T21:35:39.944 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "val": { 2023-05-31T21:35:39.944 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "state": 1, 2023-05-31T21:35:39.944 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "marker": "", 2023-05-31T21:35:39.944 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "next_step_marker": "", 2023-05-31T21:35:39.945 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "total_entries": 0, 2023-05-31T21:35:39.945 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "pos": 0, 2023-05-31T21:35:39.945 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "timestamp": "0.000000", 2023-05-31T21:35:39.945 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "realm_epoch": 2 2023-05-31T21:35:39.946 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2023-05-31T21:35:39.946 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2023-05-31T21:35:39.946 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2023-05-31T21:35:39.946 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "key": 2, 2023-05-31T21:35:39.946 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "val": { 2023-05-31T21:35:39.947 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "state": 1, 2023-05-31T21:35:39.947 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "marker": "1_1685562970.274572_93.1", 2023-05-31T21:35:39.947 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "next_step_marker": "", 2023-05-31T21:35:39.947 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "total_entries": 0, 2023-05-31T21:35:39.948 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "pos": 0, 2023-05-31T21:35:39.948 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "timestamp": "2023-05-31T19:56:10.274572Z", 2023-05-31T21:35:39.948 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "realm_epoch": 2 2023-05-31T21:35:39.948 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2023-05-31T21:35:39.949 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2023-05-31T21:35:39.949 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2023-05-31T21:35:39.949 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "key": 3, 2023-05-31T21:35:39.949 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "val": { 2023-05-31T21:35:39.950 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "state": 1, 2023-05-31T21:35:39.950 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "marker": "1_1685564491.970984_346.1", 2023-05-31T21:35:39.950 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "next_step_marker": "", 2023-05-31T21:35:39.950 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "total_entries": 0, 2023-05-31T21:35:39.950 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "pos": 0, 2023-05-31T21:35:39.951 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "timestamp": "2023-05-31T20:21:31.970984Z", 2023-05-31T21:35:39.951 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "realm_epoch": 2 2023-05-31T21:35:39.951 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2023-05-31T21:35:39.951 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2023-05-31T21:35:39.952 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: ] 2023-05-31T21:35:39.952 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2023-05-31T21:35:39.952 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "full_sync": { 2023-05-31T21:35:39.952 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "total": 1, 2023-05-31T21:35:39.953 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "complete": 1 2023-05-31T21:35:39.953 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2023-05-31T21:35:39.953 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "current_time": "2023-05-31T20:21:32Z" 2023-05-31T21:35:39.953 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2023-05-31T21:35:39.954 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2023-05-31T21:35:39.954 INFO:tasks.rgw_multisite_tests: "sync_status": { 2023-05-31T21:35:39.954 INFO:tasks.rgw_multisite_tests: "info": { 2023-05-31T21:35:39.954 INFO:tasks.rgw_multisite_tests: "status": "sync", 2023-05-31T21:35:39.954 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2023-05-31T21:35:39.955 INFO:tasks.rgw_multisite_tests: "period": "2e6123d2-d695-4350-9a01-61a455bc782b", 2023-05-31T21:35:39.955 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-05-31T21:35:39.955 INFO:tasks.rgw_multisite_tests: }, 2023-05-31T21:35:39.955 INFO:tasks.rgw_multisite_tests: "markers": [ 2023-05-31T21:35:39.956 INFO:tasks.rgw_multisite_tests: { 2023-05-31T21:35:39.956 INFO:tasks.rgw_multisite_tests: "key": 0, 2023-05-31T21:35:39.956 INFO:tasks.rgw_multisite_tests: "val": { 2023-05-31T21:35:39.956 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-05-31T21:35:39.957 INFO:tasks.rgw_multisite_tests: "marker": "1_1685562631.516089_59.1", 2023-05-31T21:35:39.957 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-05-31T21:35:39.957 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2023-05-31T21:35:39.957 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-05-31T21:35:39.957 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-05-31T19:50:31.516089Z", 2023-05-31T21:35:39.958 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-05-31T21:35:39.958 INFO:tasks.rgw_multisite_tests: } 2023-05-31T21:35:39.958 INFO:tasks.rgw_multisite_tests: }, 2023-05-31T21:35:39.958 INFO:tasks.rgw_multisite_tests: { 2023-05-31T21:35:39.959 INFO:tasks.rgw_multisite_tests: "key": 1, 2023-05-31T21:35:39.959 INFO:tasks.rgw_multisite_tests: "val": { 2023-05-31T21:35:39.959 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-05-31T21:35:39.959 INFO:tasks.rgw_multisite_tests: "marker": "", 2023-05-31T21:35:39.960 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-05-31T21:35:39.960 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-05-31T21:35:39.960 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-05-31T21:35:39.960 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2023-05-31T21:35:39.960 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-05-31T21:35:39.961 INFO:tasks.rgw_multisite_tests: } 2023-05-31T21:35:39.961 INFO:tasks.rgw_multisite_tests: }, 2023-05-31T21:35:39.961 INFO:tasks.rgw_multisite_tests: { 2023-05-31T21:35:39.961 INFO:tasks.rgw_multisite_tests: "key": 2, 2023-05-31T21:35:39.962 INFO:tasks.rgw_multisite_tests: "val": { 2023-05-31T21:35:39.962 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-05-31T21:35:39.962 INFO:tasks.rgw_multisite_tests: "marker": "1_1685562970.274572_93.1", 2023-05-31T21:35:39.962 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-05-31T21:35:39.963 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-05-31T21:35:39.963 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-05-31T21:35:39.963 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-05-31T19:56:10.274572Z", 2023-05-31T21:35:39.963 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-05-31T21:35:39.963 INFO:tasks.rgw_multisite_tests: } 2023-05-31T21:35:39.964 INFO:tasks.rgw_multisite_tests: }, 2023-05-31T21:35:39.964 INFO:tasks.rgw_multisite_tests: { 2023-05-31T21:35:39.964 INFO:tasks.rgw_multisite_tests: "key": 3, 2023-05-31T21:35:39.964 INFO:tasks.rgw_multisite_tests: "val": { 2023-05-31T21:35:39.965 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-05-31T21:35:39.965 INFO:tasks.rgw_multisite_tests: "marker": "1_1685564491.970984_346.1", 2023-05-31T21:35:39.965 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-05-31T21:35:39.965 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-05-31T21:35:39.965 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-05-31T21:35:39.966 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-05-31T20:21:31.970984Z", 2023-05-31T21:35:39.966 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-05-31T21:35:39.966 INFO:tasks.rgw_multisite_tests: } 2023-05-31T21:35:39.966 INFO:tasks.rgw_multisite_tests: } 2023-05-31T21:35:39.967 INFO:tasks.rgw_multisite_tests: ] 2023-05-31T21:35:39.967 INFO:tasks.rgw_multisite_tests: }, 2023-05-31T21:35:39.967 INFO:tasks.rgw_multisite_tests: "full_sync": { 2023-05-31T21:35:39.967 INFO:tasks.rgw_multisite_tests: "total": 1, 2023-05-31T21:35:39.968 INFO:tasks.rgw_multisite_tests: "complete": 1 2023-05-31T21:35:39.968 INFO:tasks.rgw_multisite_tests: }, 2023-05-31T21:35:39.968 INFO:tasks.rgw_multisite_tests: "current_time": "2023-05-31T20:21:32Z" 2023-05-31T21:35:39.968 INFO:tasks.rgw_multisite_tests:} 2023-05-31T21:35:39.969 INFO:tasks.rgw_multisite_tests: 2023-05-31T21:35:39.969 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1685562631.516089_59.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-05-31T19:50:31.516089Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1685562970.274572_93.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-05-31T19:56:10.274572Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1685564491.970984_346.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-05-31T20:21:31.970984Z', 'realm_epoch': 2}}] 2023-05-31T21:35:39.969 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1685562631.516089_59.1', 1: '', 2: '1_1685562970.274572_93.1', 3: '1_1685564491.970984_346.1'} 2023-05-31T21:35:39.969 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1685562631.516089_59.1', 1: '', 2: '1_1685562970.274572_93.1', 3: '1_1685564491.970984_346.1'} 2023-05-31T21:35:39.969 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: INFO: finish meta checkpoint for zone=a2 2023-05-31T21:35:39.970 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'pklxkd-52', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-05-31T21:35:39.970 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'pklxkd-52', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-05-31T21:35:39.970 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket pklxkd-52 --source-zone a1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2023-05-31T21:35:39.970 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: ignoring --setuser ceph since I am not root 2023-05-31T21:35:39.971 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-05-31T21:35:39.971 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: 2023-05-31T20:21:32.397+0000 7f0d5e99f780 1 waiting to reach incremental sync.. 2023-05-31T21:35:39.971 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: 2023-05-31T20:21:37.398+0000 7f0d5e99f780 1 waiting to reach incremental sync.. 2023-05-31T21:35:39.971 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: 2023-05-31T20:21:42.399+0000 7f0d5e99f780 1 waiting to reach incremental sync.. 2023-05-31T21:35:39.972 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: 2023-05-31T20:21:47.399+0000 7f0d5e99f780 1 waiting to reach incremental sync.. 2023-05-31T21:35:39.972 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: 2023-05-31T20:21:52.399+0000 7f0d5e99f780 1 waiting to reach incremental sync.. 2023-05-31T21:35:39.972 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: 2023-05-31T20:21:57.400+0000 7f0d5e99f780 1 waiting to reach incremental sync.. 2023-05-31T21:35:39.972 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: 2023-05-31T20:22:02.401+0000 7f0d5e99f780 1 waiting to reach incremental sync.. 2023-05-31T21:35:39.973 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: 2023-05-31T20:22:07.400+0000 7f0d5e99f780 1 waiting to reach incremental sync.. 2023-05-31T21:35:39.973 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: 2023-05-31T20:22:12.401+0000 7f0d5e99f780 1 waiting to reach incremental sync.. 2023-05-31T21:35:39.973 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: 2023-05-31T20:22:17.402+0000 7f0d5e99f780 1 waiting to reach incremental sync.. 2023-05-31T21:35:39.973 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: 2023-05-31T20:22:22.403+0000 7f0d5e99f780 1 waiting to reach incremental sync.. 2023-05-31T21:35:39.973 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: 2023-05-31T20:22:27.404+0000 7f0d5e99f780 1 waiting to reach incremental sync.. 2023-05-31T21:35:39.974 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: 2023-05-31T20:22:32.404+0000 7f0d5e99f780 1 waiting to reach incremental sync.. 2023-05-31T21:35:39.974 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: 2023-05-31T20:22:37.405+0000 7f0d5e99f780 1 waiting to reach incremental sync.. 2023-05-31T21:35:39.974 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: 2023-05-31T20:22:42.405+0000 7f0d5e99f780 1 waiting to reach incremental sync.. 2023-05-31T21:35:39.974 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: 2023-05-31T20:22:47.405+0000 7f0d5e99f780 1 waiting to reach incremental sync.. 2023-05-31T21:35:39.975 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: 2023-05-31T20:22:52.406+0000 7f0d5e99f780 1 waiting to reach incremental sync.. 2023-05-31T21:35:39.975 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: 2023-05-31T20:22:57.406+0000 7f0d5e99f780 1 waiting to reach incremental sync.. 2023-05-31T21:35:39.975 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: 2023-05-31T20:23:02.407+0000 7f0d5e99f780 1 waiting to reach incremental sync.. 2023-05-31T21:35:39.976 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: 2023-05-31T20:23:07.408+0000 7f0d5e99f780 1 waiting to reach incremental sync.. 2023-05-31T21:35:39.976 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: 2023-05-31T20:23:12.409+0000 7f0d5e99f780 1 waiting to reach incremental sync.. 2023-05-31T21:35:39.976 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: 2023-05-31T20:23:17.410+0000 7f0d5e99f780 1 waiting to reach incremental sync.. 2023-05-31T21:35:39.976 INFO:tasks.rgw_multisite_tests:tasks.rgw.c1.client.0.smithi099.stdout: INFO: tcmalloc: large alloc 1164263424 bytes == 0x55a082e18000 @ 0x7f3b20c7d760 0x7f3b20c9f464 0x7f3b20c9f808 0x7f3b247760fa 0x7f3b24776287 0x7f3b247762d7 0x7f3b2477632e 0x7f39e1dbfff2 0x55a077d38029 0x55a0780073e4 0x55a077fe7a46 0x55a0780296f6 0x55a078029f58 0x55a077dcf83b 0x55a077bc15cc 0x55a077bc4ce4 0x55a077b00516 0x55a077b011e4 0x55a07894e53f 2023-05-31T21:35:39.976 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: 2023-05-31T20:23:22.411+0000 7f0d5e99f780 1 waiting to reach incremental sync.. 2023-05-31T21:35:39.977 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: 2023-05-31T20:23:27.411+0000 7f0d5e99f780 1 waiting to reach incremental sync.. 2023-05-31T21:35:39.977 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: 2023-05-31T20:23:32.411+0000 7f0d5e99f780 1 waiting to reach incremental sync.. 2023-05-31T21:35:39.977 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: 2023-05-31T20:23:37.412+0000 7f0d5e99f780 1 waiting to reach incremental sync.. 2023-05-31T21:35:39.977 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: 2023-05-31T20:23:42.412+0000 7f0d5e99f780 1 waiting to reach incremental sync.. 2023-05-31T21:35:39.978 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: 2023-05-31T20:23:47.413+0000 7f0d5e99f780 1 waiting to reach incremental sync.. 2023-05-31T21:35:39.978 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: 2023-05-31T20:23:52.415+0000 7f0d5e99f780 1 waiting to reach incremental sync.. 2023-05-31T21:35:39.978 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: 2023-05-31T20:23:57.416+0000 7f0d5e99f780 1 waiting to reach incremental sync.. 2023-05-31T21:35:39.978 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: 2023-05-31T20:24:02.416+0000 7f0d5e99f780 1 waiting to reach incremental sync.. 2023-05-31T21:35:39.979 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: 2023-05-31T20:24:07.417+0000 7f0d5e99f780 1 waiting to reach incremental sync.. 2023-05-31T21:35:39.979 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: 2023-05-31T20:24:12.416+0000 7f0d5e99f780 1 waiting to reach incremental sync.. 2023-05-31T21:35:39.979 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: 2023-05-31T20:24:17.417+0000 7f0d5e99f780 1 waiting to reach incremental sync.. 2023-05-31T21:35:39.979 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: 2023-05-31T20:24:22.418+0000 7f0d5e99f780 1 waiting to reach incremental sync.. 2023-05-31T21:35:39.979 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: 2023-05-31T20:24:27.418+0000 7f0d5e99f780 1 waiting to reach incremental sync.. 2023-05-31T21:35:39.980 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: 2023-05-31T20:24:32.419+0000 7f0d5e99f780 1 waiting to reach incremental sync.. 2023-05-31T21:35:39.980 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: 2023-05-31T20:24:37.420+0000 7f0d5e99f780 1 waiting to reach incremental sync.. 2023-05-31T21:35:39.980 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: 2023-05-31T20:24:42.421+0000 7f0d5e99f780 1 waiting to reach incremental sync.. 2023-05-31T21:35:39.980 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: 2023-05-31T20:24:47.421+0000 7f0d5e99f780 1 waiting to reach incremental sync.. 2023-05-31T21:35:39.981 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: 2023-05-31T20:24:52.422+0000 7f0d5e99f780 1 waiting to reach incremental sync.. 2023-05-31T21:35:39.981 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: 2023-05-31T20:24:57.422+0000 7f0d5e99f780 1 waiting to reach incremental sync.. 2023-05-31T21:35:39.981 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: 2023-05-31T20:25:02.423+0000 7f0d5e99f780 1 waiting to reach incremental sync.. 2023-05-31T21:35:39.981 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: 2023-05-31T20:25:07.424+0000 7f0d5e99f780 1 waiting to reach incremental sync.. 2023-05-31T21:35:39.982 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: 2023-05-31T20:25:12.424+0000 7f0d5e99f780 1 waiting to reach incremental sync.. 2023-05-31T21:35:39.982 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: 2023-05-31T20:25:17.425+0000 7f0d5e99f780 1 waiting to reach incremental sync.. 2023-05-31T21:35:39.982 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: 2023-05-31T20:25:22.426+0000 7f0d5e99f780 1 waiting to reach incremental sync.. 2023-05-31T21:35:39.982 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: 2023-05-31T20:25:27.427+0000 7f0d5e99f780 1 waiting to reach incremental sync.. 2023-05-31T21:35:39.982 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: 2023-05-31T20:25:32.427+0000 7f0d5e99f780 1 waiting to reach incremental sync.. 2023-05-31T21:35:39.983 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: 2023-05-31T20:25:37.428+0000 7f0d5e99f780 1 waiting to reach incremental sync.. 2023-05-31T21:35:39.983 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: 2023-05-31T20:25:42.428+0000 7f0d5e99f780 1 waiting to reach incremental sync.. 2023-05-31T21:35:39.983 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: 2023-05-31T20:25:47.428+0000 7f0d5e99f780 1 waiting to reach incremental sync.. 2023-05-31T21:35:39.983 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: 2023-05-31T20:25:52.429+0000 7f0d5e99f780 1 waiting to reach incremental sync.. 2023-05-31T21:35:39.984 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: 2023-05-31T20:25:57.430+0000 7f0d5e99f780 1 waiting to reach incremental sync.. 2023-05-31T21:35:39.984 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: 2023-05-31T20:26:02.431+0000 7f0d5e99f780 1 waiting to reach incremental sync.. 2023-05-31T21:35:39.984 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: 2023-05-31T20:26:07.432+0000 7f0d5e99f780 1 waiting to reach incremental sync.. 2023-05-31T21:35:39.984 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: 2023-05-31T20:26:12.432+0000 7f0d5e99f780 1 waiting to reach incremental sync.. 2023-05-31T21:35:39.984 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: 2023-05-31T20:26:17.433+0000 7f0d5e99f780 1 waiting to reach incremental sync.. 2023-05-31T21:35:39.985 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: 2023-05-31T20:26:22.433+0000 7f0d5e99f780 1 waiting to reach incremental sync.. 2023-05-31T21:35:39.985 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: 2023-05-31T20:26:27.434+0000 7f0d5e99f780 -1 bucket checkpoint timed out waiting to reach incremental sync 2023-05-31T21:35:39.985 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: 2023-05-31T20:26:27.434+0000 7f0d5e99f780 0 bucket sync checkpoint failed: (110) Connection timed out 2023-05-31T21:35:39.985 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: 2023-05-31T20:26:27.434+0000 7f0d5e99f780 -1 bucket sync checkpoint failed: (110) Connection timed out 2023-05-31T21:35:39.986 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run: DEBUG: got remote process result: 110 2023-05-31T21:35:39.986 INFO:tasks.rgw_multisite_tests:--------------------- >> end captured logging << --------------------- 2023-05-31T21:35:39.986 INFO:tasks.rgw_multisite_tests: 2023-05-31T21:35:39.986 INFO:tasks.rgw_multisite_tests:====================================================================== 2023-05-31T21:35:39.987 INFO:tasks.rgw_multisite_tests:FAIL: rgw_multi.tests.test_bucket_reshard_index_log_trim 2023-05-31T21:35:39.987 INFO:tasks.rgw_multisite_tests:---------------------------------------------------------------------- 2023-05-31T21:35:39.988 INFO:tasks.rgw_multisite_tests:Traceback (most recent call last): 2023-05-31T21:35:39.989 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_teuthology_fa97ddd17d12e6d74b49196948f66e54fa009130/virtualenv/lib/python3.8/site-packages/nose/case.py", line 198, in runTest 2023-05-31T21:35:39.989 INFO:tasks.rgw_multisite_tests: self.test(*self.arg) 2023-05-31T21:35:39.989 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/github.com_ceph_ceph-c_2870bf456d3115490c153abce9d47ac5c043b65e/qa/../src/test/rgw/rgw_multi/tests.py", line 1418, in test_bucket_reshard_index_log_trim 2023-05-31T21:35:39.989 INFO:tasks.rgw_multisite_tests: assert(len(json_obj_5['layout']['logs']) == 1) 2023-05-31T21:35:39.990 INFO:tasks.rgw_multisite_tests:AssertionError: 2023-05-31T21:35:39.990 INFO:tasks.rgw_multisite_tests:-------------------- >> begin captured logging << -------------------- 2023-05-31T21:35:39.990 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: INFO: create bucket zone=a1 name=pklxkd-55 2023-05-31T21:35:39.990 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/pklxkd-55/ 2023-05-31T21:35:39.990 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/pklxkd-55/ 2023-05-31T21:35:39.991 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2023-05-31T21:35:39.991 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /pklxkd-55/ 2023-05-31T21:35:39.991 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-05-31T21:35:39.991 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-05-31T21:35:39.992 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T21:35:39.992 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-05-31T21:35:39.992 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-05-31T21:35:39.992 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-05-31T21:35:39.993 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-05-31T21:35:39.993 INFO:tasks.rgw_multisite_tests:PUT 2023-05-31T21:35:39.993 INFO:tasks.rgw_multisite_tests: 2023-05-31T21:35:39.993 INFO:tasks.rgw_multisite_tests: 2023-05-31T21:35:39.993 INFO:tasks.rgw_multisite_tests:Wed, 31 May 2023 20:27:16 GMT 2023-05-31T21:35:39.994 INFO:tasks.rgw_multisite_tests:/pklxkd-55/ 2023-05-31T21:35:39.994 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-05-31T21:35:39.994 INFO:tasks.rgw_multisite_tests:AWS Q80H11OLNU1S48PPYL26:hPHfvLf6VvV0UWrup7p4tutMd7A= 2023-05-31T21:35:39.994 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:27:16 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:hPHfvLf6VvV0UWrup7p4tutMd7A=', 'Content-Length': '0'} 2023-05-31T21:35:39.995 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('x-amz-request-id', 'tx0000033a9144774412b74-006477ada4-1630-a1'), ('Content-Length', '0'), ('Date', 'Wed, 31 May 2023 20:27:16 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:35:39.995 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/pklxkd-55/ 2023-05-31T21:35:39.995 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/pklxkd-55/ 2023-05-31T21:35:39.995 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-05-31T21:35:39.996 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /pklxkd-55/ 2023-05-31T21:35:39.996 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-05-31T21:35:39.996 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-05-31T21:35:39.996 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T21:35:39.997 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-05-31T21:35:39.997 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-05-31T21:35:39.997 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-05-31T21:35:39.997 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-05-31T21:35:39.997 INFO:tasks.rgw_multisite_tests:HEAD 2023-05-31T21:35:39.998 INFO:tasks.rgw_multisite_tests: 2023-05-31T21:35:39.998 INFO:tasks.rgw_multisite_tests: 2023-05-31T21:35:39.998 INFO:tasks.rgw_multisite_tests:Wed, 31 May 2023 20:27:16 GMT 2023-05-31T21:35:39.998 INFO:tasks.rgw_multisite_tests:/pklxkd-55/ 2023-05-31T21:35:39.999 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-05-31T21:35:39.999 INFO:tasks.rgw_multisite_tests:AWS Q80H11OLNU1S48PPYL26:/gdTTojnB6Hl7PzQ8paUIVDSxl0= 2023-05-31T21:35:39.999 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:27:16 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:/gdTTojnB6Hl7PzQ8paUIVDSxl0=', 'Content-Length': '0'} 2023-05-31T21:35:39.999 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000c7721aa60aa2030d-006477ada4-1630-a1'), ('Content-Length', '0'), ('Date', 'Wed, 31 May 2023 20:27:16 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:35:39.999 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/pklxkd-55/a 2023-05-31T21:35:40.000 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/pklxkd-55/a 2023-05-31T21:35:40.000 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2023-05-31T21:35:40.000 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /pklxkd-55/a 2023-05-31T21:35:40.000 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-05-31T21:35:40.001 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-05-31T21:35:40.001 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T21:35:40.001 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-05-31T21:35:40.001 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-05-31T21:35:40.002 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-05-31T21:35:40.002 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-05-31T21:35:40.002 INFO:tasks.rgw_multisite_tests:PUT 2023-05-31T21:35:40.002 INFO:tasks.rgw_multisite_tests:rL0Y20zC+Fzt72VPzMSk2A== 2023-05-31T21:35:40.003 INFO:tasks.rgw_multisite_tests:application/octet-stream 2023-05-31T21:35:40.003 INFO:tasks.rgw_multisite_tests:Wed, 31 May 2023 20:27:16 GMT 2023-05-31T21:35:40.003 INFO:tasks.rgw_multisite_tests:/pklxkd-55/a 2023-05-31T21:35:40.003 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-05-31T21:35:40.003 INFO:tasks.rgw_multisite_tests:AWS Q80H11OLNU1S48PPYL26:Jd87CxZPf/jayeGlTBAOqgrkFT8= 2023-05-31T21:35:40.004 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Wed, 31 May 2023 20:27:16 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:Jd87CxZPf/jayeGlTBAOqgrkFT8='} 2023-05-31T21:35:40.004 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000000caa443b4a2238b1-006477ada4-1630-a1'), ('Date', 'Wed, 31 May 2023 20:27:16 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:35:40.004 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/pklxkd-55/ 2023-05-31T21:35:40.004 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/pklxkd-55/ 2023-05-31T21:35:40.005 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-05-31T21:35:40.005 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /pklxkd-55/ 2023-05-31T21:35:40.005 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-05-31T21:35:40.005 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-05-31T21:35:40.006 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T21:35:40.006 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-05-31T21:35:40.006 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-05-31T21:35:40.006 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-05-31T21:35:40.006 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-05-31T21:35:40.007 INFO:tasks.rgw_multisite_tests:HEAD 2023-05-31T21:35:40.007 INFO:tasks.rgw_multisite_tests: 2023-05-31T21:35:40.007 INFO:tasks.rgw_multisite_tests: 2023-05-31T21:35:40.007 INFO:tasks.rgw_multisite_tests:Wed, 31 May 2023 20:27:16 GMT 2023-05-31T21:35:40.008 INFO:tasks.rgw_multisite_tests:/pklxkd-55/ 2023-05-31T21:35:40.008 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-05-31T21:35:40.008 INFO:tasks.rgw_multisite_tests:AWS Q80H11OLNU1S48PPYL26:/gdTTojnB6Hl7PzQ8paUIVDSxl0= 2023-05-31T21:35:40.008 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:27:16 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:/gdTTojnB6Hl7PzQ8paUIVDSxl0=', 'Content-Length': '0'} 2023-05-31T21:35:40.008 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '3'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000002c12c4e858309536-006477ada4-1630-a1'), ('Content-Length', '0'), ('Date', 'Wed, 31 May 2023 20:27:16 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:35:40.009 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/pklxkd-55/b 2023-05-31T21:35:40.009 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/pklxkd-55/b 2023-05-31T21:35:40.009 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2023-05-31T21:35:40.009 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /pklxkd-55/b 2023-05-31T21:35:40.010 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-05-31T21:35:40.010 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-05-31T21:35:40.010 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T21:35:40.010 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-05-31T21:35:40.011 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-05-31T21:35:40.011 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-05-31T21:35:40.011 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-05-31T21:35:40.011 INFO:tasks.rgw_multisite_tests:PUT 2023-05-31T21:35:40.011 INFO:tasks.rgw_multisite_tests:rL0Y20zC+Fzt72VPzMSk2A== 2023-05-31T21:35:40.012 INFO:tasks.rgw_multisite_tests:application/octet-stream 2023-05-31T21:35:40.012 INFO:tasks.rgw_multisite_tests:Wed, 31 May 2023 20:27:16 GMT 2023-05-31T21:35:40.012 INFO:tasks.rgw_multisite_tests:/pklxkd-55/b 2023-05-31T21:35:40.012 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-05-31T21:35:40.013 INFO:tasks.rgw_multisite_tests:AWS Q80H11OLNU1S48PPYL26:YNnzYpVdPiyIPZf+ywza8hMU4f4= 2023-05-31T21:35:40.013 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Wed, 31 May 2023 20:27:16 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:YNnzYpVdPiyIPZf+ywza8hMU4f4='} 2023-05-31T21:35:40.013 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000004188f2160578862d-006477ada4-1630-a1'), ('Date', 'Wed, 31 May 2023 20:27:16 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:35:40.013 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/pklxkd-55/ 2023-05-31T21:35:40.014 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/pklxkd-55/ 2023-05-31T21:35:40.014 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-05-31T21:35:40.014 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /pklxkd-55/ 2023-05-31T21:35:40.014 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-05-31T21:35:40.014 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-05-31T21:35:40.015 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T21:35:40.015 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-05-31T21:35:40.015 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-05-31T21:35:40.015 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-05-31T21:35:40.016 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-05-31T21:35:40.016 INFO:tasks.rgw_multisite_tests:HEAD 2023-05-31T21:35:40.016 INFO:tasks.rgw_multisite_tests: 2023-05-31T21:35:40.016 INFO:tasks.rgw_multisite_tests: 2023-05-31T21:35:40.017 INFO:tasks.rgw_multisite_tests:Wed, 31 May 2023 20:27:16 GMT 2023-05-31T21:35:40.017 INFO:tasks.rgw_multisite_tests:/pklxkd-55/ 2023-05-31T21:35:40.017 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-05-31T21:35:40.017 INFO:tasks.rgw_multisite_tests:AWS Q80H11OLNU1S48PPYL26:/gdTTojnB6Hl7PzQ8paUIVDSxl0= 2023-05-31T21:35:40.017 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:27:16 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:/gdTTojnB6Hl7PzQ8paUIVDSxl0=', 'Content-Length': '0'} 2023-05-31T21:35:40.018 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '2'), ('X-RGW-Bytes-Used', '6'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000089987bce4d708c39-006477ada4-1630-a1'), ('Content-Length', '0'), ('Date', 'Wed, 31 May 2023 20:27:16 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:35:40.018 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/pklxkd-55/c 2023-05-31T21:35:40.018 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/pklxkd-55/c 2023-05-31T21:35:40.018 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2023-05-31T21:35:40.019 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /pklxkd-55/c 2023-05-31T21:35:40.019 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-05-31T21:35:40.019 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-05-31T21:35:40.019 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T21:35:40.020 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-05-31T21:35:40.020 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-05-31T21:35:40.020 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-05-31T21:35:40.020 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-05-31T21:35:40.020 INFO:tasks.rgw_multisite_tests:PUT 2023-05-31T21:35:40.021 INFO:tasks.rgw_multisite_tests:rL0Y20zC+Fzt72VPzMSk2A== 2023-05-31T21:35:40.021 INFO:tasks.rgw_multisite_tests:application/octet-stream 2023-05-31T21:35:40.021 INFO:tasks.rgw_multisite_tests:Wed, 31 May 2023 20:27:16 GMT 2023-05-31T21:35:40.021 INFO:tasks.rgw_multisite_tests:/pklxkd-55/c 2023-05-31T21:35:40.022 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-05-31T21:35:40.022 INFO:tasks.rgw_multisite_tests:AWS Q80H11OLNU1S48PPYL26:BsuxGLmplBlHp3VyZZR3UMY8x3M= 2023-05-31T21:35:40.022 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Wed, 31 May 2023 20:27:16 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:BsuxGLmplBlHp3VyZZR3UMY8x3M='} 2023-05-31T21:35:40.022 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000f45a79d36e823e6a-006477ada4-1630-a1'), ('Date', 'Wed, 31 May 2023 20:27:16 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:35:40.023 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/pklxkd-55/ 2023-05-31T21:35:40.023 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/pklxkd-55/ 2023-05-31T21:35:40.023 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-05-31T21:35:40.023 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /pklxkd-55/ 2023-05-31T21:35:40.024 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-05-31T21:35:40.024 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-05-31T21:35:40.024 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T21:35:40.024 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-05-31T21:35:40.025 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-05-31T21:35:40.025 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-05-31T21:35:40.025 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-05-31T21:35:40.025 INFO:tasks.rgw_multisite_tests:HEAD 2023-05-31T21:35:40.026 INFO:tasks.rgw_multisite_tests: 2023-05-31T21:35:40.026 INFO:tasks.rgw_multisite_tests: 2023-05-31T21:35:40.026 INFO:tasks.rgw_multisite_tests:Wed, 31 May 2023 20:27:16 GMT 2023-05-31T21:35:40.026 INFO:tasks.rgw_multisite_tests:/pklxkd-55/ 2023-05-31T21:35:40.026 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-05-31T21:35:40.027 INFO:tasks.rgw_multisite_tests:AWS Q80H11OLNU1S48PPYL26:/gdTTojnB6Hl7PzQ8paUIVDSxl0= 2023-05-31T21:35:40.027 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:27:16 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:/gdTTojnB6Hl7PzQ8paUIVDSxl0=', 'Content-Length': '0'} 2023-05-31T21:35:40.027 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '3'), ('X-RGW-Bytes-Used', '9'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000931f7c1e239f9d1f-006477ada4-1630-a1'), ('Content-Length', '0'), ('Date', 'Wed, 31 May 2023 20:27:16 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:35:40.027 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/pklxkd-55/d 2023-05-31T21:35:40.028 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/pklxkd-55/d 2023-05-31T21:35:40.028 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2023-05-31T21:35:40.028 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /pklxkd-55/d 2023-05-31T21:35:40.028 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-05-31T21:35:40.029 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-05-31T21:35:40.029 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T21:35:40.029 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-05-31T21:35:40.029 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-05-31T21:35:40.030 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-05-31T21:35:40.030 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-05-31T21:35:40.030 INFO:tasks.rgw_multisite_tests:PUT 2023-05-31T21:35:40.030 INFO:tasks.rgw_multisite_tests:rL0Y20zC+Fzt72VPzMSk2A== 2023-05-31T21:35:40.030 INFO:tasks.rgw_multisite_tests:application/octet-stream 2023-05-31T21:35:40.031 INFO:tasks.rgw_multisite_tests:Wed, 31 May 2023 20:27:16 GMT 2023-05-31T21:35:40.031 INFO:tasks.rgw_multisite_tests:/pklxkd-55/d 2023-05-31T21:35:40.031 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-05-31T21:35:40.031 INFO:tasks.rgw_multisite_tests:AWS Q80H11OLNU1S48PPYL26:hmT5C+THGJVcaahZJaHyH2ViBO8= 2023-05-31T21:35:40.032 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Wed, 31 May 2023 20:27:16 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:hmT5C+THGJVcaahZJaHyH2ViBO8='} 2023-05-31T21:35:40.032 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000d36174b3f8e74157-006477ada4-1630-a1'), ('Date', 'Wed, 31 May 2023 20:27:16 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:35:40.032 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-31T21:35:40.032 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-31T21:35:40.033 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-05-31T21:35:40.033 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: ignoring --setuser ceph since I am not root 2023-05-31T21:35:40.033 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-05-31T21:35:40.033 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: No --period given, using current period=2e6123d2-d695-4350-9a01-61a455bc782b 2023-05-31T21:35:40.034 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: [ 2023-05-31T21:35:40.034 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2023-05-31T21:35:40.034 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "marker": "1_1685562631.516089_59.1", 2023-05-31T21:35:40.034 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "last_update": "2023-05-31T19:50:31.516089Z" 2023-05-31T21:35:40.034 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2023-05-31T21:35:40.035 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2023-05-31T21:35:40.035 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "marker": "", 2023-05-31T21:35:40.035 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "last_update": "0.000000" 2023-05-31T21:35:40.035 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2023-05-31T21:35:40.036 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2023-05-31T21:35:40.036 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "marker": "1_1685562970.274572_93.1", 2023-05-31T21:35:40.036 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "last_update": "2023-05-31T19:56:10.274572Z" 2023-05-31T21:35:40.036 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2023-05-31T21:35:40.037 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2023-05-31T21:35:40.037 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "marker": "1_1685564836.773548_358.1", 2023-05-31T21:35:40.037 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "last_update": "2023-05-31T20:27:16.773548Z" 2023-05-31T21:35:40.037 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2023-05-31T21:35:40.037 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: ] 2023-05-31T21:35:40.038 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: master meta markers={0: '1_1685562631.516089_59.1', 1: '', 2: '1_1685562970.274572_93.1', 3: '1_1685564836.773548_358.1'} 2023-05-31T21:35:40.038 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: INFO: starting meta checkpoint for zone=a2 2023-05-31T21:35:40.038 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-31T21:35:40.038 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-31T21:35:40.039 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-05-31T21:35:40.039 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: ignoring --setuser ceph since I am not root 2023-05-31T21:35:40.039 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-05-31T21:35:40.039 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: 2023-05-31T20:27:17.133+0000 7fa3fc26d780 20 RGW-SYNC:meta: read sync status 2023-05-31T21:35:40.040 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: 2023-05-31T20:27:17.134+0000 7fa3fc26d780 20 RGW-SYNC:meta: read sync status 2023-05-31T21:35:40.040 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2023-05-31T21:35:40.040 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "sync_status": { 2023-05-31T21:35:40.040 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "info": { 2023-05-31T21:35:40.040 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "status": "sync", 2023-05-31T21:35:40.041 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "num_shards": 4, 2023-05-31T21:35:40.041 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "period": "2e6123d2-d695-4350-9a01-61a455bc782b", 2023-05-31T21:35:40.041 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "realm_epoch": 2 2023-05-31T21:35:40.041 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2023-05-31T21:35:40.042 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "markers": [ 2023-05-31T21:35:40.042 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2023-05-31T21:35:40.042 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "key": 0, 2023-05-31T21:35:40.042 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "val": { 2023-05-31T21:35:40.043 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "state": 1, 2023-05-31T21:35:40.043 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "marker": "1_1685562631.516089_59.1", 2023-05-31T21:35:40.043 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "next_step_marker": "", 2023-05-31T21:35:40.043 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "total_entries": 1, 2023-05-31T21:35:40.043 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "pos": 0, 2023-05-31T21:35:40.044 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "timestamp": "2023-05-31T19:50:31.516089Z", 2023-05-31T21:35:40.044 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "realm_epoch": 2 2023-05-31T21:35:40.044 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2023-05-31T21:35:40.044 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2023-05-31T21:35:40.045 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2023-05-31T21:35:40.045 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "key": 1, 2023-05-31T21:35:40.045 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "val": { 2023-05-31T21:35:40.045 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "state": 1, 2023-05-31T21:35:40.046 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "marker": "", 2023-05-31T21:35:40.046 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "next_step_marker": "", 2023-05-31T21:35:40.046 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "total_entries": 0, 2023-05-31T21:35:40.046 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "pos": 0, 2023-05-31T21:35:40.046 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "timestamp": "0.000000", 2023-05-31T21:35:40.047 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "realm_epoch": 2 2023-05-31T21:35:40.047 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2023-05-31T21:35:40.047 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2023-05-31T21:35:40.047 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2023-05-31T21:35:40.048 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "key": 2, 2023-05-31T21:35:40.048 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "val": { 2023-05-31T21:35:40.048 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "state": 1, 2023-05-31T21:35:40.048 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "marker": "1_1685562970.274572_93.1", 2023-05-31T21:35:40.049 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "next_step_marker": "", 2023-05-31T21:35:40.049 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "total_entries": 0, 2023-05-31T21:35:40.049 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "pos": 0, 2023-05-31T21:35:40.049 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "timestamp": "2023-05-31T19:56:10.274572Z", 2023-05-31T21:35:40.049 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "realm_epoch": 2 2023-05-31T21:35:40.050 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2023-05-31T21:35:40.050 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2023-05-31T21:35:40.050 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2023-05-31T21:35:40.050 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "key": 3, 2023-05-31T21:35:40.051 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "val": { 2023-05-31T21:35:40.051 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "state": 1, 2023-05-31T21:35:40.051 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "marker": "1_1685564836.773548_358.1", 2023-05-31T21:35:40.051 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "next_step_marker": "", 2023-05-31T21:35:40.051 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "total_entries": 0, 2023-05-31T21:35:40.052 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "pos": 0, 2023-05-31T21:35:40.052 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "timestamp": "2023-05-31T20:27:16.773548Z", 2023-05-31T21:35:40.052 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "realm_epoch": 2 2023-05-31T21:35:40.052 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2023-05-31T21:35:40.053 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2023-05-31T21:35:40.053 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: ] 2023-05-31T21:35:40.053 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2023-05-31T21:35:40.053 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "full_sync": { 2023-05-31T21:35:40.054 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "total": 1, 2023-05-31T21:35:40.054 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "complete": 1 2023-05-31T21:35:40.054 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2023-05-31T21:35:40.054 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "current_time": "2023-05-31T20:27:17Z" 2023-05-31T21:35:40.054 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2023-05-31T21:35:40.055 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2023-05-31T21:35:40.055 INFO:tasks.rgw_multisite_tests: "sync_status": { 2023-05-31T21:35:40.055 INFO:tasks.rgw_multisite_tests: "info": { 2023-05-31T21:35:40.055 INFO:tasks.rgw_multisite_tests: "status": "sync", 2023-05-31T21:35:40.056 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2023-05-31T21:35:40.056 INFO:tasks.rgw_multisite_tests: "period": "2e6123d2-d695-4350-9a01-61a455bc782b", 2023-05-31T21:35:40.056 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-05-31T21:35:40.056 INFO:tasks.rgw_multisite_tests: }, 2023-05-31T21:35:40.057 INFO:tasks.rgw_multisite_tests: "markers": [ 2023-05-31T21:35:40.057 INFO:tasks.rgw_multisite_tests: { 2023-05-31T21:35:40.057 INFO:tasks.rgw_multisite_tests: "key": 0, 2023-05-31T21:35:40.057 INFO:tasks.rgw_multisite_tests: "val": { 2023-05-31T21:35:40.058 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-05-31T21:35:40.058 INFO:tasks.rgw_multisite_tests: "marker": "1_1685562631.516089_59.1", 2023-05-31T21:35:40.058 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-05-31T21:35:40.058 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2023-05-31T21:35:40.058 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-05-31T21:35:40.059 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-05-31T19:50:31.516089Z", 2023-05-31T21:35:40.059 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-05-31T21:35:40.059 INFO:tasks.rgw_multisite_tests: } 2023-05-31T21:35:40.059 INFO:tasks.rgw_multisite_tests: }, 2023-05-31T21:35:40.060 INFO:tasks.rgw_multisite_tests: { 2023-05-31T21:35:40.060 INFO:tasks.rgw_multisite_tests: "key": 1, 2023-05-31T21:35:40.060 INFO:tasks.rgw_multisite_tests: "val": { 2023-05-31T21:35:40.060 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-05-31T21:35:40.061 INFO:tasks.rgw_multisite_tests: "marker": "", 2023-05-31T21:35:40.061 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-05-31T21:35:40.061 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-05-31T21:35:40.061 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-05-31T21:35:40.061 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2023-05-31T21:35:40.062 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-05-31T21:35:40.062 INFO:tasks.rgw_multisite_tests: } 2023-05-31T21:35:40.062 INFO:tasks.rgw_multisite_tests: }, 2023-05-31T21:35:40.062 INFO:tasks.rgw_multisite_tests: { 2023-05-31T21:35:40.063 INFO:tasks.rgw_multisite_tests: "key": 2, 2023-05-31T21:35:40.063 INFO:tasks.rgw_multisite_tests: "val": { 2023-05-31T21:35:40.063 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-05-31T21:35:40.063 INFO:tasks.rgw_multisite_tests: "marker": "1_1685562970.274572_93.1", 2023-05-31T21:35:40.063 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-05-31T21:35:40.064 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-05-31T21:35:40.064 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-05-31T21:35:40.064 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-05-31T19:56:10.274572Z", 2023-05-31T21:35:40.064 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-05-31T21:35:40.065 INFO:tasks.rgw_multisite_tests: } 2023-05-31T21:35:40.065 INFO:tasks.rgw_multisite_tests: }, 2023-05-31T21:35:40.065 INFO:tasks.rgw_multisite_tests: { 2023-05-31T21:35:40.065 INFO:tasks.rgw_multisite_tests: "key": 3, 2023-05-31T21:35:40.066 INFO:tasks.rgw_multisite_tests: "val": { 2023-05-31T21:35:40.066 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-05-31T21:35:40.066 INFO:tasks.rgw_multisite_tests: "marker": "1_1685564836.773548_358.1", 2023-05-31T21:35:40.066 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-05-31T21:35:40.066 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-05-31T21:35:40.067 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-05-31T21:35:40.067 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-05-31T20:27:16.773548Z", 2023-05-31T21:35:40.067 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-05-31T21:35:40.067 INFO:tasks.rgw_multisite_tests: } 2023-05-31T21:35:40.068 INFO:tasks.rgw_multisite_tests: } 2023-05-31T21:35:40.068 INFO:tasks.rgw_multisite_tests: ] 2023-05-31T21:35:40.068 INFO:tasks.rgw_multisite_tests: }, 2023-05-31T21:35:40.068 INFO:tasks.rgw_multisite_tests: "full_sync": { 2023-05-31T21:35:40.069 INFO:tasks.rgw_multisite_tests: "total": 1, 2023-05-31T21:35:40.069 INFO:tasks.rgw_multisite_tests: "complete": 1 2023-05-31T21:35:40.069 INFO:tasks.rgw_multisite_tests: }, 2023-05-31T21:35:40.069 INFO:tasks.rgw_multisite_tests: "current_time": "2023-05-31T20:27:17Z" 2023-05-31T21:35:40.069 INFO:tasks.rgw_multisite_tests:} 2023-05-31T21:35:40.070 INFO:tasks.rgw_multisite_tests: 2023-05-31T21:35:40.070 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1685562631.516089_59.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-05-31T19:50:31.516089Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1685562970.274572_93.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-05-31T19:56:10.274572Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1685564836.773548_358.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-05-31T20:27:16.773548Z', 'realm_epoch': 2}}] 2023-05-31T21:35:40.070 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1685562631.516089_59.1', 1: '', 2: '1_1685562970.274572_93.1', 3: '1_1685564836.773548_358.1'} 2023-05-31T21:35:40.070 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1685562631.516089_59.1', 1: '', 2: '1_1685562970.274572_93.1', 3: '1_1685564836.773548_358.1'} 2023-05-31T21:35:40.071 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: INFO: finish meta checkpoint for zone=a2 2023-05-31T21:35:40.071 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: bucket checkpoint: source=a1 target=a2 bucket=pklxkd-55 2023-05-31T21:35:40.071 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'pklxkd-55', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-05-31T21:35:40.071 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'pklxkd-55', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-05-31T21:35:40.072 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket pklxkd-55 --source-zone a1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2023-05-31T21:35:40.072 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: ignoring --setuser ceph since I am not root 2023-05-31T21:35:40.072 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-05-31T21:35:40.072 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: 2023-05-31T20:27:17.287+0000 7fd2fdd08780 1 waiting to reach incremental sync.. 2023-05-31T21:35:40.072 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: 2023-05-31T20:27:22.289+0000 7fd2fdd08780 1 bucket sync caught up with source: 2023-05-31T21:35:40.073 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: local status: [, 00000000001.28.6, , , , 00000000001.45.6, , , 00000000001.42.6, , 00000000001.108.6] 2023-05-31T21:35:40.073 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: remote markers: [, 00000000001.28.6, , , , 00000000001.45.6, , , 00000000001.42.6, , 00000000001.108.6] 2023-05-31T21:35:40.073 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: 2023-05-31T20:27:22.289+0000 7fd2fdd08780 0 bucket checkpoint complete 2023-05-31T21:35:40.073 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: bucket checkpoint: source=a2 target=a1 bucket=pklxkd-55 2023-05-31T21:35:40.074 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'pklxkd-55', '--source-zone', 'a2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-05-31T21:35:40.074 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'pklxkd-55', '--source-zone', 'a2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-05-31T21:35:40.074 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket pklxkd-55 --source-zone a2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2023-05-31T21:35:40.074 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: ignoring --setuser ceph since I am not root 2023-05-31T21:35:40.075 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-05-31T21:35:40.075 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: 2023-05-31T20:27:22.413+0000 7f80f47ca780 1 bucket sync caught up with source: 2023-05-31T21:35:40.075 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: local status: [, 00000000001.1059.6, , , , 00000000001.1340.6, , , 00000000001.3554.6, , 00000000001.2404.6] 2023-05-31T21:35:40.075 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: remote markers: [, 00000000001.1059.6, , , , 00000000001.1340.6, , , 00000000001.3554.6, , 00000000001.2404.6] 2023-05-31T21:35:40.076 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: 2023-05-31T20:27:22.413+0000 7f80f47ca780 0 bucket checkpoint complete 2023-05-31T21:35:40.076 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: INFO: comparing bucket=pklxkd-55 zones={a2, a1} 2023-05-31T21:35:40.076 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/pklxkd-55/ 2023-05-31T21:35:40.076 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/pklxkd-55/ 2023-05-31T21:35:40.077 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-05-31T21:35:40.077 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /pklxkd-55/ 2023-05-31T21:35:40.077 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-05-31T21:35:40.077 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-05-31T21:35:40.078 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T21:35:40.078 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-05-31T21:35:40.078 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-05-31T21:35:40.078 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-05-31T21:35:40.078 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-05-31T21:35:40.079 INFO:tasks.rgw_multisite_tests:HEAD 2023-05-31T21:35:40.079 INFO:tasks.rgw_multisite_tests: 2023-05-31T21:35:40.079 INFO:tasks.rgw_multisite_tests: 2023-05-31T21:35:40.079 INFO:tasks.rgw_multisite_tests:Wed, 31 May 2023 20:27:22 GMT 2023-05-31T21:35:40.080 INFO:tasks.rgw_multisite_tests:/pklxkd-55/ 2023-05-31T21:35:40.080 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-05-31T21:35:40.080 INFO:tasks.rgw_multisite_tests:AWS Q80H11OLNU1S48PPYL26:YPYI0xrKQ20HAiP/MqnrYwIJejM= 2023-05-31T21:35:40.080 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:27:22 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:YPYI0xrKQ20HAiP/MqnrYwIJejM=', 'Content-Length': '0'} 2023-05-31T21:35:40.081 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '4'), ('X-RGW-Bytes-Used', '12'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000086949aba72247eff-006477adaa-162e-a2'), ('Content-Length', '0'), ('Date', 'Wed, 31 May 2023 20:27:22 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:35:40.081 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/pklxkd-55/ 2023-05-31T21:35:40.081 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/pklxkd-55/ 2023-05-31T21:35:40.081 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-05-31T21:35:40.081 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /pklxkd-55/ 2023-05-31T21:35:40.082 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-05-31T21:35:40.082 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-05-31T21:35:40.082 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T21:35:40.082 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-05-31T21:35:40.083 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-05-31T21:35:40.083 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-05-31T21:35:40.083 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-05-31T21:35:40.083 INFO:tasks.rgw_multisite_tests:HEAD 2023-05-31T21:35:40.084 INFO:tasks.rgw_multisite_tests: 2023-05-31T21:35:40.084 INFO:tasks.rgw_multisite_tests: 2023-05-31T21:35:40.084 INFO:tasks.rgw_multisite_tests:Wed, 31 May 2023 20:27:22 GMT 2023-05-31T21:35:40.084 INFO:tasks.rgw_multisite_tests:/pklxkd-55/ 2023-05-31T21:35:40.084 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-05-31T21:35:40.085 INFO:tasks.rgw_multisite_tests:AWS Q80H11OLNU1S48PPYL26:YPYI0xrKQ20HAiP/MqnrYwIJejM= 2023-05-31T21:35:40.085 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:27:22 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:YPYI0xrKQ20HAiP/MqnrYwIJejM=', 'Content-Length': '0'} 2023-05-31T21:35:40.085 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '4'), ('X-RGW-Bytes-Used', '12'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000fe55b73ee1900c3a-006477adaa-1630-a1'), ('Content-Length', '0'), ('Date', 'Wed, 31 May 2023 20:27:22 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:35:40.085 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: bucket1 objects: 2023-05-31T21:35:40.086 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/pklxkd-55/ 2023-05-31T21:35:40.086 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/pklxkd-55/ 2023-05-31T21:35:40.086 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/pklxkd-55/?versions&max-keys=999 2023-05-31T21:35:40.086 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/pklxkd-55/?versions&max-keys=999 2023-05-31T21:35:40.087 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-05-31T21:35:40.087 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /pklxkd-55/?versions&max-keys=999 2023-05-31T21:35:40.087 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-05-31T21:35:40.087 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-05-31T21:35:40.088 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T21:35:40.088 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-05-31T21:35:40.088 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-05-31T21:35:40.088 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-05-31T21:35:40.089 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-05-31T21:35:40.089 INFO:tasks.rgw_multisite_tests:GET 2023-05-31T21:35:40.089 INFO:tasks.rgw_multisite_tests: 2023-05-31T21:35:40.089 INFO:tasks.rgw_multisite_tests: 2023-05-31T21:35:40.089 INFO:tasks.rgw_multisite_tests:Wed, 31 May 2023 20:27:22 GMT 2023-05-31T21:35:40.090 INFO:tasks.rgw_multisite_tests:/pklxkd-55/?versions 2023-05-31T21:35:40.090 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-05-31T21:35:40.090 INFO:tasks.rgw_multisite_tests:AWS Q80H11OLNU1S48PPYL26:rH1yLHJbUp2yS0ZgWD3zz80IhC4= 2023-05-31T21:35:40.090 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:27:22 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:rH1yLHJbUp2yS0ZgWD3zz80IhC4=', 'Content-Length': '0'} 2023-05-31T21:35:40.091 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ede5ee216d55c057-006477adaa-162e-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:27:22 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:35:40.091 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'pklxkd-55999falseanulltrue2023-05-31T20:27:16.792Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalbnulltrue2023-05-31T20:27:16.850Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalcnulltrue2023-05-31T20:27:16.869Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormaldnulltrue2023-05-31T20:27:16.893Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T21:35:40.091 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: o=a 2023-05-31T21:35:40.091 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: o=b 2023-05-31T21:35:40.092 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: o=c 2023-05-31T21:35:40.092 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: o=d 2023-05-31T21:35:40.092 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: bucket2 objects: 2023-05-31T21:35:40.092 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/pklxkd-55/ 2023-05-31T21:35:40.092 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/pklxkd-55/ 2023-05-31T21:35:40.093 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/pklxkd-55/?versions&max-keys=999 2023-05-31T21:35:40.093 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/pklxkd-55/?versions&max-keys=999 2023-05-31T21:35:40.093 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-05-31T21:35:40.093 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /pklxkd-55/?versions&max-keys=999 2023-05-31T21:35:40.094 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-05-31T21:35:40.094 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-05-31T21:35:40.094 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T21:35:40.094 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-05-31T21:35:40.095 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-05-31T21:35:40.095 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-05-31T21:35:40.095 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-05-31T21:35:40.095 INFO:tasks.rgw_multisite_tests:GET 2023-05-31T21:35:40.095 INFO:tasks.rgw_multisite_tests: 2023-05-31T21:35:40.096 INFO:tasks.rgw_multisite_tests: 2023-05-31T21:35:40.096 INFO:tasks.rgw_multisite_tests:Wed, 31 May 2023 20:27:22 GMT 2023-05-31T21:35:40.096 INFO:tasks.rgw_multisite_tests:/pklxkd-55/?versions 2023-05-31T21:35:40.096 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-05-31T21:35:40.097 INFO:tasks.rgw_multisite_tests:AWS Q80H11OLNU1S48PPYL26:rH1yLHJbUp2yS0ZgWD3zz80IhC4= 2023-05-31T21:35:40.097 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:27:22 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:rH1yLHJbUp2yS0ZgWD3zz80IhC4=', 'Content-Length': '0'} 2023-05-31T21:35:40.097 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000005ac610fc17f38232-006477adaa-1630-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:27:22 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:35:40.097 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'pklxkd-55999falseanulltrue2023-05-31T20:27:16.792Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalbnulltrue2023-05-31T20:27:16.850Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalcnulltrue2023-05-31T20:27:16.869Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormaldnulltrue2023-05-31T20:27:16.893Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T21:35:40.098 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: o=a 2023-05-31T21:35:40.098 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: o=b 2023-05-31T21:35:40.098 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: o=c 2023-05-31T21:35:40.098 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: o=d 2023-05-31T21:35:40.098 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/pklxkd-55/ 2023-05-31T21:35:40.099 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/pklxkd-55/ 2023-05-31T21:35:40.099 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/pklxkd-55/?versions&max-keys=999 2023-05-31T21:35:40.099 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/pklxkd-55/?versions&max-keys=999 2023-05-31T21:35:40.099 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-05-31T21:35:40.100 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /pklxkd-55/?versions&max-keys=999 2023-05-31T21:35:40.100 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-05-31T21:35:40.100 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-05-31T21:35:40.100 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T21:35:40.101 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-05-31T21:35:40.101 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-05-31T21:35:40.101 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-05-31T21:35:40.101 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-05-31T21:35:40.102 INFO:tasks.rgw_multisite_tests:GET 2023-05-31T21:35:40.102 INFO:tasks.rgw_multisite_tests: 2023-05-31T21:35:40.102 INFO:tasks.rgw_multisite_tests: 2023-05-31T21:35:40.102 INFO:tasks.rgw_multisite_tests:Wed, 31 May 2023 20:27:22 GMT 2023-05-31T21:35:40.102 INFO:tasks.rgw_multisite_tests:/pklxkd-55/?versions 2023-05-31T21:35:40.103 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-05-31T21:35:40.103 INFO:tasks.rgw_multisite_tests:AWS Q80H11OLNU1S48PPYL26:rH1yLHJbUp2yS0ZgWD3zz80IhC4= 2023-05-31T21:35:40.103 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:27:22 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:rH1yLHJbUp2yS0ZgWD3zz80IhC4=', 'Content-Length': '0'} 2023-05-31T21:35:40.103 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f2655bd4ef4e5557-006477adaa-162e-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:27:22 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:35:40.104 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'pklxkd-55999falseanulltrue2023-05-31T20:27:16.792Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalbnulltrue2023-05-31T20:27:16.850Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalcnulltrue2023-05-31T20:27:16.869Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormaldnulltrue2023-05-31T20:27:16.893Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T21:35:40.104 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/pklxkd-55/ 2023-05-31T21:35:40.104 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/pklxkd-55/ 2023-05-31T21:35:40.104 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/pklxkd-55/?versions&max-keys=999 2023-05-31T21:35:40.105 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/pklxkd-55/?versions&max-keys=999 2023-05-31T21:35:40.105 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-05-31T21:35:40.105 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /pklxkd-55/?versions&max-keys=999 2023-05-31T21:35:40.105 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-05-31T21:35:40.106 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-05-31T21:35:40.106 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T21:35:40.106 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-05-31T21:35:40.106 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-05-31T21:35:40.106 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-05-31T21:35:40.107 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-05-31T21:35:40.107 INFO:tasks.rgw_multisite_tests:GET 2023-05-31T21:35:40.107 INFO:tasks.rgw_multisite_tests: 2023-05-31T21:35:40.107 INFO:tasks.rgw_multisite_tests: 2023-05-31T21:35:40.108 INFO:tasks.rgw_multisite_tests:Wed, 31 May 2023 20:27:22 GMT 2023-05-31T21:35:40.108 INFO:tasks.rgw_multisite_tests:/pklxkd-55/?versions 2023-05-31T21:35:40.108 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-05-31T21:35:40.108 INFO:tasks.rgw_multisite_tests:AWS Q80H11OLNU1S48PPYL26:rH1yLHJbUp2yS0ZgWD3zz80IhC4= 2023-05-31T21:35:40.109 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:27:22 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:rH1yLHJbUp2yS0ZgWD3zz80IhC4=', 'Content-Length': '0'} 2023-05-31T21:35:40.109 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ad2e9e00ea56ee1e-006477adaa-1630-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:27:22 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:35:40.109 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'pklxkd-55999falseanulltrue2023-05-31T20:27:16.792Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalbnulltrue2023-05-31T20:27:16.850Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalcnulltrue2023-05-31T20:27:16.869Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormaldnulltrue2023-05-31T20:27:16.893Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T21:35:40.109 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: comparing key name=a 2023-05-31T21:35:40.109 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/pklxkd-55/a 2023-05-31T21:35:40.110 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/pklxkd-55/a 2023-05-31T21:35:40.110 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/pklxkd-55/a?versionId=null 2023-05-31T21:35:40.110 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/pklxkd-55/a?versionId=null 2023-05-31T21:35:40.110 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-05-31T21:35:40.111 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /pklxkd-55/a?versionId=null 2023-05-31T21:35:40.111 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-05-31T21:35:40.111 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-05-31T21:35:40.111 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T21:35:40.112 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-05-31T21:35:40.112 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-05-31T21:35:40.112 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-05-31T21:35:40.112 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-05-31T21:35:40.112 INFO:tasks.rgw_multisite_tests:GET 2023-05-31T21:35:40.113 INFO:tasks.rgw_multisite_tests: 2023-05-31T21:35:40.113 INFO:tasks.rgw_multisite_tests: 2023-05-31T21:35:40.113 INFO:tasks.rgw_multisite_tests:Wed, 31 May 2023 20:27:22 GMT 2023-05-31T21:35:40.113 INFO:tasks.rgw_multisite_tests:/pklxkd-55/a?versionId=null 2023-05-31T21:35:40.114 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-05-31T21:35:40.114 INFO:tasks.rgw_multisite_tests:AWS Q80H11OLNU1S48PPYL26:HjVcheEugGytioIwJHyG6X2nsjE= 2023-05-31T21:35:40.114 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:27:22 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:HjVcheEugGytioIwJHyG6X2nsjE=', 'Content-Length': '0'} 2023-05-31T21:35:40.114 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:27:16 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000504e4be1c777ee4c-006477adaa-162e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:27:22 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:35:40.115 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/pklxkd-55/a 2023-05-31T21:35:40.115 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/pklxkd-55/a 2023-05-31T21:35:40.115 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/pklxkd-55/a?versionId=null 2023-05-31T21:35:40.115 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/pklxkd-55/a?versionId=null 2023-05-31T21:35:40.115 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-05-31T21:35:40.116 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /pklxkd-55/a?versionId=null 2023-05-31T21:35:40.116 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-05-31T21:35:40.116 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-05-31T21:35:40.116 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T21:35:40.117 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-05-31T21:35:40.117 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-05-31T21:35:40.117 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-05-31T21:35:40.117 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-05-31T21:35:40.118 INFO:tasks.rgw_multisite_tests:GET 2023-05-31T21:35:40.118 INFO:tasks.rgw_multisite_tests: 2023-05-31T21:35:40.118 INFO:tasks.rgw_multisite_tests: 2023-05-31T21:35:40.118 INFO:tasks.rgw_multisite_tests:Wed, 31 May 2023 20:27:22 GMT 2023-05-31T21:35:40.118 INFO:tasks.rgw_multisite_tests:/pklxkd-55/a?versionId=null 2023-05-31T21:35:40.119 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-05-31T21:35:40.119 INFO:tasks.rgw_multisite_tests:AWS Q80H11OLNU1S48PPYL26:HjVcheEugGytioIwJHyG6X2nsjE= 2023-05-31T21:35:40.119 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:27:22 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:HjVcheEugGytioIwJHyG6X2nsjE=', 'Content-Length': '0'} 2023-05-31T21:35:40.119 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:27:16 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002502ea0eb09dbcf3-006477adaa-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:27:22 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:35:40.120 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/pklxkd-55/a 2023-05-31T21:35:40.120 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/pklxkd-55/a 2023-05-31T21:35:40.120 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/pklxkd-55/a?versionId=null 2023-05-31T21:35:40.120 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/pklxkd-55/a?versionId=null 2023-05-31T21:35:40.121 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-05-31T21:35:40.121 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /pklxkd-55/a?versionId=null 2023-05-31T21:35:40.121 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-05-31T21:35:40.121 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-05-31T21:35:40.121 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T21:35:40.122 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-05-31T21:35:40.122 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-05-31T21:35:40.122 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-05-31T21:35:40.122 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-05-31T21:35:40.123 INFO:tasks.rgw_multisite_tests:HEAD 2023-05-31T21:35:40.123 INFO:tasks.rgw_multisite_tests: 2023-05-31T21:35:40.123 INFO:tasks.rgw_multisite_tests: 2023-05-31T21:35:40.123 INFO:tasks.rgw_multisite_tests:Wed, 31 May 2023 20:27:22 GMT 2023-05-31T21:35:40.124 INFO:tasks.rgw_multisite_tests:/pklxkd-55/a?versionId=null 2023-05-31T21:35:40.124 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-05-31T21:35:40.124 INFO:tasks.rgw_multisite_tests:AWS Q80H11OLNU1S48PPYL26:WX/Aeemy7lpWwgAxqKo0HzEntEI= 2023-05-31T21:35:40.124 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:27:22 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:WX/Aeemy7lpWwgAxqKo0HzEntEI=', 'Content-Length': '0'} 2023-05-31T21:35:40.124 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:27:16 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000059dcd5da7b46036e-006477adaa-162e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:27:22 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:35:40.125 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/pklxkd-55/a 2023-05-31T21:35:40.125 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/pklxkd-55/a 2023-05-31T21:35:40.125 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/pklxkd-55/a?versionId=null 2023-05-31T21:35:40.125 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/pklxkd-55/a?versionId=null 2023-05-31T21:35:40.126 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-05-31T21:35:40.126 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /pklxkd-55/a?versionId=null 2023-05-31T21:35:40.126 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-05-31T21:35:40.126 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-05-31T21:35:40.127 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T21:35:40.127 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-05-31T21:35:40.127 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-05-31T21:35:40.127 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-05-31T21:35:40.127 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-05-31T21:35:40.128 INFO:tasks.rgw_multisite_tests:HEAD 2023-05-31T21:35:40.128 INFO:tasks.rgw_multisite_tests: 2023-05-31T21:35:40.128 INFO:tasks.rgw_multisite_tests: 2023-05-31T21:35:40.128 INFO:tasks.rgw_multisite_tests:Wed, 31 May 2023 20:27:22 GMT 2023-05-31T21:35:40.129 INFO:tasks.rgw_multisite_tests:/pklxkd-55/a?versionId=null 2023-05-31T21:35:40.129 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-05-31T21:35:40.129 INFO:tasks.rgw_multisite_tests:AWS Q80H11OLNU1S48PPYL26:WX/Aeemy7lpWwgAxqKo0HzEntEI= 2023-05-31T21:35:40.129 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:27:22 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:WX/Aeemy7lpWwgAxqKo0HzEntEI=', 'Content-Length': '0'} 2023-05-31T21:35:40.130 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:27:16 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006f493aaa2431ad88-006477adaa-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:27:22 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:35:40.130 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: comparing key name=a 2023-05-31T21:35:40.130 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/pklxkd-55/a 2023-05-31T21:35:40.130 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/pklxkd-55/a 2023-05-31T21:35:40.130 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/pklxkd-55/a?versionId=null 2023-05-31T21:35:40.131 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/pklxkd-55/a?versionId=null 2023-05-31T21:35:40.131 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-05-31T21:35:40.131 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /pklxkd-55/a?versionId=null 2023-05-31T21:35:40.131 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-05-31T21:35:40.132 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-05-31T21:35:40.132 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T21:35:40.132 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-05-31T21:35:40.132 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-05-31T21:35:40.133 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-05-31T21:35:40.133 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-05-31T21:35:40.133 INFO:tasks.rgw_multisite_tests:GET 2023-05-31T21:35:40.133 INFO:tasks.rgw_multisite_tests: 2023-05-31T21:35:40.133 INFO:tasks.rgw_multisite_tests: 2023-05-31T21:35:40.134 INFO:tasks.rgw_multisite_tests:Wed, 31 May 2023 20:27:22 GMT 2023-05-31T21:35:40.134 INFO:tasks.rgw_multisite_tests:/pklxkd-55/a?versionId=null 2023-05-31T21:35:40.134 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-05-31T21:35:40.134 INFO:tasks.rgw_multisite_tests:AWS Q80H11OLNU1S48PPYL26:HjVcheEugGytioIwJHyG6X2nsjE= 2023-05-31T21:35:40.135 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:27:22 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:HjVcheEugGytioIwJHyG6X2nsjE=', 'Content-Length': '0'} 2023-05-31T21:35:40.135 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:27:16 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b60516f9383d1884-006477adaa-162e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:27:22 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:35:40.135 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/pklxkd-55/a 2023-05-31T21:35:40.135 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/pklxkd-55/a 2023-05-31T21:35:40.136 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/pklxkd-55/a?versionId=null 2023-05-31T21:35:40.136 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/pklxkd-55/a?versionId=null 2023-05-31T21:35:40.136 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-05-31T21:35:40.136 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /pklxkd-55/a?versionId=null 2023-05-31T21:35:40.136 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-05-31T21:35:40.137 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-05-31T21:35:40.137 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T21:35:40.137 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-05-31T21:35:40.137 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-05-31T21:35:40.138 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-05-31T21:35:40.138 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-05-31T21:35:40.138 INFO:tasks.rgw_multisite_tests:GET 2023-05-31T21:35:40.138 INFO:tasks.rgw_multisite_tests: 2023-05-31T21:35:40.138 INFO:tasks.rgw_multisite_tests: 2023-05-31T21:35:40.139 INFO:tasks.rgw_multisite_tests:Wed, 31 May 2023 20:27:22 GMT 2023-05-31T21:35:40.139 INFO:tasks.rgw_multisite_tests:/pklxkd-55/a?versionId=null 2023-05-31T21:35:40.139 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-05-31T21:35:40.139 INFO:tasks.rgw_multisite_tests:AWS Q80H11OLNU1S48PPYL26:HjVcheEugGytioIwJHyG6X2nsjE= 2023-05-31T21:35:40.140 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:27:22 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:HjVcheEugGytioIwJHyG6X2nsjE=', 'Content-Length': '0'} 2023-05-31T21:35:40.140 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:27:16 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000dd64f19f26d5ea30-006477adaa-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:27:22 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:35:40.140 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/pklxkd-55/ 2023-05-31T21:35:40.140 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/pklxkd-55/ 2023-05-31T21:35:40.141 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/pklxkd-55/?prefix=a 2023-05-31T21:35:40.141 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/pklxkd-55/?prefix=a 2023-05-31T21:35:40.141 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-05-31T21:35:40.141 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /pklxkd-55/?prefix=a 2023-05-31T21:35:40.142 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-05-31T21:35:40.142 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-05-31T21:35:40.142 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T21:35:40.142 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-05-31T21:35:40.142 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-05-31T21:35:40.143 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-05-31T21:35:40.143 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-05-31T21:35:40.143 INFO:tasks.rgw_multisite_tests:GET 2023-05-31T21:35:40.143 INFO:tasks.rgw_multisite_tests: 2023-05-31T21:35:40.144 INFO:tasks.rgw_multisite_tests: 2023-05-31T21:35:40.144 INFO:tasks.rgw_multisite_tests:Wed, 31 May 2023 20:27:22 GMT 2023-05-31T21:35:40.144 INFO:tasks.rgw_multisite_tests:/pklxkd-55/ 2023-05-31T21:35:40.144 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-05-31T21:35:40.144 INFO:tasks.rgw_multisite_tests:AWS Q80H11OLNU1S48PPYL26:p2p+8G1ZdPifmzf+JcgpB78h0o8= 2023-05-31T21:35:40.145 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:27:22 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:p2p+8G1ZdPifmzf+JcgpB78h0o8=', 'Content-Length': '0'} 2023-05-31T21:35:40.145 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000004baa82331feb0ff3-006477adaa-162e-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:27:22 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:35:40.145 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'pklxkd-55a1000falsea2023-05-31T20:27:16.792Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T21:35:40.145 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/pklxkd-55/ 2023-05-31T21:35:40.146 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/pklxkd-55/ 2023-05-31T21:35:40.146 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/pklxkd-55/?prefix=a 2023-05-31T21:35:40.146 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/pklxkd-55/?prefix=a 2023-05-31T21:35:40.146 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-05-31T21:35:40.147 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /pklxkd-55/?prefix=a 2023-05-31T21:35:40.147 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-05-31T21:35:40.147 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-05-31T21:35:40.147 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T21:35:40.148 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-05-31T21:35:40.148 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-05-31T21:35:40.148 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-05-31T21:35:40.148 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-05-31T21:35:40.148 INFO:tasks.rgw_multisite_tests:GET 2023-05-31T21:35:40.149 INFO:tasks.rgw_multisite_tests: 2023-05-31T21:35:40.149 INFO:tasks.rgw_multisite_tests: 2023-05-31T21:35:40.149 INFO:tasks.rgw_multisite_tests:Wed, 31 May 2023 20:27:22 GMT 2023-05-31T21:35:40.149 INFO:tasks.rgw_multisite_tests:/pklxkd-55/ 2023-05-31T21:35:40.150 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-05-31T21:35:40.150 INFO:tasks.rgw_multisite_tests:AWS Q80H11OLNU1S48PPYL26:p2p+8G1ZdPifmzf+JcgpB78h0o8= 2023-05-31T21:35:40.150 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:27:22 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:p2p+8G1ZdPifmzf+JcgpB78h0o8=', 'Content-Length': '0'} 2023-05-31T21:35:40.150 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000009e9d72b6eb393c23-006477adaa-1630-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:27:22 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:35:40.150 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'pklxkd-55a1000falsea2023-05-31T20:27:16.792Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T21:35:40.151 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/pklxkd-55/a 2023-05-31T21:35:40.151 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/pklxkd-55/a 2023-05-31T21:35:40.151 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-05-31T21:35:40.151 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /pklxkd-55/a 2023-05-31T21:35:40.152 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-05-31T21:35:40.152 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-05-31T21:35:40.152 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T21:35:40.152 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-05-31T21:35:40.153 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-05-31T21:35:40.153 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-05-31T21:35:40.153 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-05-31T21:35:40.153 INFO:tasks.rgw_multisite_tests:HEAD 2023-05-31T21:35:40.153 INFO:tasks.rgw_multisite_tests: 2023-05-31T21:35:40.154 INFO:tasks.rgw_multisite_tests: 2023-05-31T21:35:40.154 INFO:tasks.rgw_multisite_tests:Wed, 31 May 2023 20:27:22 GMT 2023-05-31T21:35:40.154 INFO:tasks.rgw_multisite_tests:/pklxkd-55/a 2023-05-31T21:35:40.154 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-05-31T21:35:40.155 INFO:tasks.rgw_multisite_tests:AWS Q80H11OLNU1S48PPYL26:FN5t9rLsnpx97Hrm+q+piTRkijU= 2023-05-31T21:35:40.155 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:27:22 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:FN5t9rLsnpx97Hrm+q+piTRkijU=', 'Content-Length': '0'} 2023-05-31T21:35:40.155 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:27:16 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000103b42fd84696813-006477adaa-162e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:27:22 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:35:40.155 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/pklxkd-55/a 2023-05-31T21:35:40.156 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/pklxkd-55/a 2023-05-31T21:35:40.156 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-05-31T21:35:40.156 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /pklxkd-55/a 2023-05-31T21:35:40.156 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-05-31T21:35:40.156 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-05-31T21:35:40.157 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T21:35:40.157 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-05-31T21:35:40.157 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-05-31T21:35:40.157 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-05-31T21:35:40.158 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-05-31T21:35:40.158 INFO:tasks.rgw_multisite_tests:HEAD 2023-05-31T21:35:40.158 INFO:tasks.rgw_multisite_tests: 2023-05-31T21:35:40.158 INFO:tasks.rgw_multisite_tests: 2023-05-31T21:35:40.159 INFO:tasks.rgw_multisite_tests:Wed, 31 May 2023 20:27:22 GMT 2023-05-31T21:35:40.159 INFO:tasks.rgw_multisite_tests:/pklxkd-55/a 2023-05-31T21:35:40.159 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-05-31T21:35:40.159 INFO:tasks.rgw_multisite_tests:AWS Q80H11OLNU1S48PPYL26:FN5t9rLsnpx97Hrm+q+piTRkijU= 2023-05-31T21:35:40.159 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:27:22 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:FN5t9rLsnpx97Hrm+q+piTRkijU=', 'Content-Length': '0'} 2023-05-31T21:35:40.160 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:27:16 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003d0ec68bb5dfca65-006477adaa-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:27:22 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:35:40.160 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: comparing key name=a 2023-05-31T21:35:40.160 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/pklxkd-55/a 2023-05-31T21:35:40.160 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/pklxkd-55/a 2023-05-31T21:35:40.161 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-05-31T21:35:40.161 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /pklxkd-55/a 2023-05-31T21:35:40.161 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-05-31T21:35:40.161 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-05-31T21:35:40.162 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T21:35:40.162 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-05-31T21:35:40.162 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-05-31T21:35:40.162 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-05-31T21:35:40.162 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-05-31T21:35:40.163 INFO:tasks.rgw_multisite_tests:GET 2023-05-31T21:35:40.163 INFO:tasks.rgw_multisite_tests: 2023-05-31T21:35:40.163 INFO:tasks.rgw_multisite_tests: 2023-05-31T21:35:40.163 INFO:tasks.rgw_multisite_tests:Wed, 31 May 2023 20:27:22 GMT 2023-05-31T21:35:40.164 INFO:tasks.rgw_multisite_tests:/pklxkd-55/a 2023-05-31T21:35:40.164 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-05-31T21:35:40.164 INFO:tasks.rgw_multisite_tests:AWS Q80H11OLNU1S48PPYL26:QfBq7c12kIVH21Pp/9FHihChbNU= 2023-05-31T21:35:40.164 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:27:22 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:QfBq7c12kIVH21Pp/9FHihChbNU=', 'Content-Length': '0'} 2023-05-31T21:35:40.165 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:27:16 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f0440d4046779ebb-006477adaa-162e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:27:22 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:35:40.165 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/pklxkd-55/a 2023-05-31T21:35:40.165 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/pklxkd-55/a 2023-05-31T21:35:40.165 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-05-31T21:35:40.165 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /pklxkd-55/a 2023-05-31T21:35:40.166 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-05-31T21:35:40.166 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-05-31T21:35:40.166 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T21:35:40.166 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-05-31T21:35:40.167 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-05-31T21:35:40.167 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-05-31T21:35:40.167 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-05-31T21:35:40.167 INFO:tasks.rgw_multisite_tests:GET 2023-05-31T21:35:40.168 INFO:tasks.rgw_multisite_tests: 2023-05-31T21:35:40.168 INFO:tasks.rgw_multisite_tests: 2023-05-31T21:35:40.168 INFO:tasks.rgw_multisite_tests:Wed, 31 May 2023 20:27:22 GMT 2023-05-31T21:35:40.168 INFO:tasks.rgw_multisite_tests:/pklxkd-55/a 2023-05-31T21:35:40.168 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-05-31T21:35:40.169 INFO:tasks.rgw_multisite_tests:AWS Q80H11OLNU1S48PPYL26:QfBq7c12kIVH21Pp/9FHihChbNU= 2023-05-31T21:35:40.169 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:27:22 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:QfBq7c12kIVH21Pp/9FHihChbNU=', 'Content-Length': '0'} 2023-05-31T21:35:40.169 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:27:16 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000084daad4fcde81b4b-006477adaa-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:27:22 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:35:40.169 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/pklxkd-55/ 2023-05-31T21:35:40.170 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/pklxkd-55/ 2023-05-31T21:35:40.170 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/pklxkd-55/?prefix=a 2023-05-31T21:35:40.170 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/pklxkd-55/?prefix=a 2023-05-31T21:35:40.170 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-05-31T21:35:40.170 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /pklxkd-55/?prefix=a 2023-05-31T21:35:40.171 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-05-31T21:35:40.171 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-05-31T21:35:40.171 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T21:35:40.171 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-05-31T21:35:40.172 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-05-31T21:35:40.172 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-05-31T21:35:40.172 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-05-31T21:35:40.172 INFO:tasks.rgw_multisite_tests:GET 2023-05-31T21:35:40.173 INFO:tasks.rgw_multisite_tests: 2023-05-31T21:35:40.173 INFO:tasks.rgw_multisite_tests: 2023-05-31T21:35:40.173 INFO:tasks.rgw_multisite_tests:Wed, 31 May 2023 20:27:22 GMT 2023-05-31T21:35:40.173 INFO:tasks.rgw_multisite_tests:/pklxkd-55/ 2023-05-31T21:35:40.174 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-05-31T21:35:40.174 INFO:tasks.rgw_multisite_tests:AWS Q80H11OLNU1S48PPYL26:p2p+8G1ZdPifmzf+JcgpB78h0o8= 2023-05-31T21:35:40.174 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:27:22 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:p2p+8G1ZdPifmzf+JcgpB78h0o8=', 'Content-Length': '0'} 2023-05-31T21:35:40.174 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000017d0e613c81f17dc-006477adaa-162e-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:27:22 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:35:40.175 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'pklxkd-55a1000falsea2023-05-31T20:27:16.792Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T21:35:40.175 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/pklxkd-55/ 2023-05-31T21:35:40.175 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/pklxkd-55/ 2023-05-31T21:35:40.175 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/pklxkd-55/?prefix=a 2023-05-31T21:35:40.176 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/pklxkd-55/?prefix=a 2023-05-31T21:35:40.176 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-05-31T21:35:40.176 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /pklxkd-55/?prefix=a 2023-05-31T21:35:40.176 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-05-31T21:35:40.177 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-05-31T21:35:40.177 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T21:35:40.177 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-05-31T21:35:40.177 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-05-31T21:35:40.178 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-05-31T21:35:40.178 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-05-31T21:35:40.178 INFO:tasks.rgw_multisite_tests:GET 2023-05-31T21:35:40.178 INFO:tasks.rgw_multisite_tests: 2023-05-31T21:35:40.178 INFO:tasks.rgw_multisite_tests: 2023-05-31T21:35:40.179 INFO:tasks.rgw_multisite_tests:Wed, 31 May 2023 20:27:22 GMT 2023-05-31T21:35:40.179 INFO:tasks.rgw_multisite_tests:/pklxkd-55/ 2023-05-31T21:35:40.179 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-05-31T21:35:40.179 INFO:tasks.rgw_multisite_tests:AWS Q80H11OLNU1S48PPYL26:p2p+8G1ZdPifmzf+JcgpB78h0o8= 2023-05-31T21:35:40.180 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:27:22 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:p2p+8G1ZdPifmzf+JcgpB78h0o8=', 'Content-Length': '0'} 2023-05-31T21:35:40.180 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000009827cdc059cd62c2-006477adaa-1630-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:27:22 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:35:40.180 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'pklxkd-55a1000falsea2023-05-31T20:27:16.792Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T21:35:40.180 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: comparing key name=b 2023-05-31T21:35:40.181 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/pklxkd-55/b 2023-05-31T21:35:40.181 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/pklxkd-55/b 2023-05-31T21:35:40.181 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/pklxkd-55/b?versionId=null 2023-05-31T21:35:40.181 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/pklxkd-55/b?versionId=null 2023-05-31T21:35:40.182 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-05-31T21:35:40.182 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /pklxkd-55/b?versionId=null 2023-05-31T21:35:40.182 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-05-31T21:35:40.182 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-05-31T21:35:40.182 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T21:35:40.183 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-05-31T21:35:40.183 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-05-31T21:35:40.183 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-05-31T21:35:40.183 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-05-31T21:35:40.184 INFO:tasks.rgw_multisite_tests:GET 2023-05-31T21:35:40.184 INFO:tasks.rgw_multisite_tests: 2023-05-31T21:35:40.184 INFO:tasks.rgw_multisite_tests: 2023-05-31T21:35:40.184 INFO:tasks.rgw_multisite_tests:Wed, 31 May 2023 20:27:22 GMT 2023-05-31T21:35:40.185 INFO:tasks.rgw_multisite_tests:/pklxkd-55/b?versionId=null 2023-05-31T21:35:40.185 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-05-31T21:35:40.185 INFO:tasks.rgw_multisite_tests:AWS Q80H11OLNU1S48PPYL26:QHiV5puDrYlTI9MGI70ZYfdDQxw= 2023-05-31T21:35:40.185 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:27:22 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:QHiV5puDrYlTI9MGI70ZYfdDQxw=', 'Content-Length': '0'} 2023-05-31T21:35:40.185 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:27:16 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003cdf3c8bd483e634-006477adaa-162e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:27:22 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:35:40.186 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/pklxkd-55/b 2023-05-31T21:35:40.186 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/pklxkd-55/b 2023-05-31T21:35:40.186 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/pklxkd-55/b?versionId=null 2023-05-31T21:35:40.186 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/pklxkd-55/b?versionId=null 2023-05-31T21:35:40.187 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-05-31T21:35:40.187 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /pklxkd-55/b?versionId=null 2023-05-31T21:35:40.187 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-05-31T21:35:40.187 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-05-31T21:35:40.188 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T21:35:40.188 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-05-31T21:35:40.188 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-05-31T21:35:40.188 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-05-31T21:35:40.189 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-05-31T21:35:40.189 INFO:tasks.rgw_multisite_tests:GET 2023-05-31T21:35:40.189 INFO:tasks.rgw_multisite_tests: 2023-05-31T21:35:40.189 INFO:tasks.rgw_multisite_tests: 2023-05-31T21:35:40.189 INFO:tasks.rgw_multisite_tests:Wed, 31 May 2023 20:27:22 GMT 2023-05-31T21:35:40.190 INFO:tasks.rgw_multisite_tests:/pklxkd-55/b?versionId=null 2023-05-31T21:35:40.190 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-05-31T21:35:40.190 INFO:tasks.rgw_multisite_tests:AWS Q80H11OLNU1S48PPYL26:QHiV5puDrYlTI9MGI70ZYfdDQxw= 2023-05-31T21:35:40.190 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:27:22 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:QHiV5puDrYlTI9MGI70ZYfdDQxw=', 'Content-Length': '0'} 2023-05-31T21:35:40.191 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:27:16 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000be0c6f3f4c8ac5d-006477adaa-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:27:22 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:35:40.191 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/pklxkd-55/b 2023-05-31T21:35:40.191 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/pklxkd-55/b 2023-05-31T21:35:40.191 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/pklxkd-55/b?versionId=null 2023-05-31T21:35:40.192 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/pklxkd-55/b?versionId=null 2023-05-31T21:35:40.192 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-05-31T21:35:40.192 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /pklxkd-55/b?versionId=null 2023-05-31T21:35:40.192 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-05-31T21:35:40.193 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-05-31T21:35:40.193 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T21:35:40.193 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-05-31T21:35:40.193 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-05-31T21:35:40.193 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-05-31T21:35:40.194 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-05-31T21:35:40.194 INFO:tasks.rgw_multisite_tests:HEAD 2023-05-31T21:35:40.194 INFO:tasks.rgw_multisite_tests: 2023-05-31T21:35:40.194 INFO:tasks.rgw_multisite_tests: 2023-05-31T21:35:40.195 INFO:tasks.rgw_multisite_tests:Wed, 31 May 2023 20:27:22 GMT 2023-05-31T21:35:40.195 INFO:tasks.rgw_multisite_tests:/pklxkd-55/b?versionId=null 2023-05-31T21:35:40.195 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-05-31T21:35:40.195 INFO:tasks.rgw_multisite_tests:AWS Q80H11OLNU1S48PPYL26:IpxKqJi7IvrKxtKS7HSWE8HZNqk= 2023-05-31T21:35:40.195 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:27:22 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:IpxKqJi7IvrKxtKS7HSWE8HZNqk=', 'Content-Length': '0'} 2023-05-31T21:35:40.196 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:27:16 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000070b7595de7bcab08-006477adaa-162e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:27:22 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:35:40.196 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/pklxkd-55/b 2023-05-31T21:35:40.196 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/pklxkd-55/b 2023-05-31T21:35:40.196 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/pklxkd-55/b?versionId=null 2023-05-31T21:35:40.197 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/pklxkd-55/b?versionId=null 2023-05-31T21:35:40.197 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-05-31T21:35:40.197 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /pklxkd-55/b?versionId=null 2023-05-31T21:35:40.197 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-05-31T21:35:40.198 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-05-31T21:35:40.198 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T21:35:40.198 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-05-31T21:35:40.198 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-05-31T21:35:40.199 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-05-31T21:35:40.199 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-05-31T21:35:40.199 INFO:tasks.rgw_multisite_tests:HEAD 2023-05-31T21:35:40.199 INFO:tasks.rgw_multisite_tests: 2023-05-31T21:35:40.199 INFO:tasks.rgw_multisite_tests: 2023-05-31T21:35:40.200 INFO:tasks.rgw_multisite_tests:Wed, 31 May 2023 20:27:22 GMT 2023-05-31T21:35:40.200 INFO:tasks.rgw_multisite_tests:/pklxkd-55/b?versionId=null 2023-05-31T21:35:40.200 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-05-31T21:35:40.200 INFO:tasks.rgw_multisite_tests:AWS Q80H11OLNU1S48PPYL26:IpxKqJi7IvrKxtKS7HSWE8HZNqk= 2023-05-31T21:35:40.201 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:27:22 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:IpxKqJi7IvrKxtKS7HSWE8HZNqk=', 'Content-Length': '0'} 2023-05-31T21:35:40.201 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:27:16 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002a4bda418a9d4a1d-006477adaa-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:27:22 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:35:40.201 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: comparing key name=b 2023-05-31T21:35:40.201 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/pklxkd-55/b 2023-05-31T21:35:40.202 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/pklxkd-55/b 2023-05-31T21:35:40.202 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/pklxkd-55/b?versionId=null 2023-05-31T21:35:40.202 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/pklxkd-55/b?versionId=null 2023-05-31T21:35:40.202 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-05-31T21:35:40.202 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /pklxkd-55/b?versionId=null 2023-05-31T21:35:40.203 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-05-31T21:35:40.203 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-05-31T21:35:40.203 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T21:35:40.203 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-05-31T21:35:40.204 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-05-31T21:35:40.204 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-05-31T21:35:40.204 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-05-31T21:35:40.204 INFO:tasks.rgw_multisite_tests:GET 2023-05-31T21:35:40.204 INFO:tasks.rgw_multisite_tests: 2023-05-31T21:35:40.205 INFO:tasks.rgw_multisite_tests: 2023-05-31T21:35:40.205 INFO:tasks.rgw_multisite_tests:Wed, 31 May 2023 20:27:22 GMT 2023-05-31T21:35:40.205 INFO:tasks.rgw_multisite_tests:/pklxkd-55/b?versionId=null 2023-05-31T21:35:40.205 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-05-31T21:35:40.206 INFO:tasks.rgw_multisite_tests:AWS Q80H11OLNU1S48PPYL26:QHiV5puDrYlTI9MGI70ZYfdDQxw= 2023-05-31T21:35:40.206 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:27:22 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:QHiV5puDrYlTI9MGI70ZYfdDQxw=', 'Content-Length': '0'} 2023-05-31T21:35:40.206 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:27:16 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000350e9545cc9ab1f4-006477adaa-162e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:27:22 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:35:40.206 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/pklxkd-55/b 2023-05-31T21:35:40.207 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/pklxkd-55/b 2023-05-31T21:35:40.207 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/pklxkd-55/b?versionId=null 2023-05-31T21:35:40.207 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/pklxkd-55/b?versionId=null 2023-05-31T21:35:40.207 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-05-31T21:35:40.208 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /pklxkd-55/b?versionId=null 2023-05-31T21:35:40.208 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-05-31T21:35:40.208 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-05-31T21:35:40.208 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T21:35:40.208 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-05-31T21:35:40.209 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-05-31T21:35:40.209 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-05-31T21:35:40.209 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-05-31T21:35:40.209 INFO:tasks.rgw_multisite_tests:GET 2023-05-31T21:35:40.210 INFO:tasks.rgw_multisite_tests: 2023-05-31T21:35:40.210 INFO:tasks.rgw_multisite_tests: 2023-05-31T21:35:40.210 INFO:tasks.rgw_multisite_tests:Wed, 31 May 2023 20:27:22 GMT 2023-05-31T21:35:40.210 INFO:tasks.rgw_multisite_tests:/pklxkd-55/b?versionId=null 2023-05-31T21:35:40.210 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-05-31T21:35:40.211 INFO:tasks.rgw_multisite_tests:AWS Q80H11OLNU1S48PPYL26:QHiV5puDrYlTI9MGI70ZYfdDQxw= 2023-05-31T21:35:40.211 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:27:22 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:QHiV5puDrYlTI9MGI70ZYfdDQxw=', 'Content-Length': '0'} 2023-05-31T21:35:40.211 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:27:16 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008bb112c27dcb4e8d-006477adaa-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:27:22 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:35:40.211 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/pklxkd-55/ 2023-05-31T21:35:40.212 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/pklxkd-55/ 2023-05-31T21:35:40.212 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/pklxkd-55/?prefix=b 2023-05-31T21:35:40.212 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/pklxkd-55/?prefix=b 2023-05-31T21:35:40.212 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-05-31T21:35:40.213 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /pklxkd-55/?prefix=b 2023-05-31T21:35:40.213 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-05-31T21:35:40.213 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-05-31T21:35:40.213 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T21:35:40.214 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-05-31T21:35:40.214 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-05-31T21:35:40.214 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-05-31T21:35:40.214 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-05-31T21:35:40.214 INFO:tasks.rgw_multisite_tests:GET 2023-05-31T21:35:40.215 INFO:tasks.rgw_multisite_tests: 2023-05-31T21:35:40.215 INFO:tasks.rgw_multisite_tests: 2023-05-31T21:35:40.215 INFO:tasks.rgw_multisite_tests:Wed, 31 May 2023 20:27:22 GMT 2023-05-31T21:35:40.215 INFO:tasks.rgw_multisite_tests:/pklxkd-55/ 2023-05-31T21:35:40.216 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-05-31T21:35:40.216 INFO:tasks.rgw_multisite_tests:AWS Q80H11OLNU1S48PPYL26:p2p+8G1ZdPifmzf+JcgpB78h0o8= 2023-05-31T21:35:40.216 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:27:22 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:p2p+8G1ZdPifmzf+JcgpB78h0o8=', 'Content-Length': '0'} 2023-05-31T21:35:40.216 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000067d0b26bcda2484d-006477adaa-162e-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:27:22 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:35:40.216 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'pklxkd-55b1000falseb2023-05-31T20:27:16.850Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T21:35:40.217 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/pklxkd-55/ 2023-05-31T21:35:40.217 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/pklxkd-55/ 2023-05-31T21:35:40.217 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/pklxkd-55/?prefix=b 2023-05-31T21:35:40.217 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/pklxkd-55/?prefix=b 2023-05-31T21:35:40.218 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-05-31T21:35:40.218 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /pklxkd-55/?prefix=b 2023-05-31T21:35:40.218 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-05-31T21:35:40.218 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-05-31T21:35:40.219 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T21:35:40.219 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-05-31T21:35:40.219 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-05-31T21:35:40.219 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-05-31T21:35:40.219 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-05-31T21:35:40.220 INFO:tasks.rgw_multisite_tests:GET 2023-05-31T21:35:40.220 INFO:tasks.rgw_multisite_tests: 2023-05-31T21:35:40.220 INFO:tasks.rgw_multisite_tests: 2023-05-31T21:35:40.220 INFO:tasks.rgw_multisite_tests:Wed, 31 May 2023 20:27:22 GMT 2023-05-31T21:35:40.221 INFO:tasks.rgw_multisite_tests:/pklxkd-55/ 2023-05-31T21:35:40.221 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-05-31T21:35:40.221 INFO:tasks.rgw_multisite_tests:AWS Q80H11OLNU1S48PPYL26:p2p+8G1ZdPifmzf+JcgpB78h0o8= 2023-05-31T21:35:40.221 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:27:22 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:p2p+8G1ZdPifmzf+JcgpB78h0o8=', 'Content-Length': '0'} 2023-05-31T21:35:40.222 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000005511508bf33a4a52-006477adaa-1630-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:27:22 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:35:40.222 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'pklxkd-55b1000falseb2023-05-31T20:27:16.850Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T21:35:40.222 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/pklxkd-55/b 2023-05-31T21:35:40.222 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/pklxkd-55/b 2023-05-31T21:35:40.223 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-05-31T21:35:40.223 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /pklxkd-55/b 2023-05-31T21:35:40.223 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-05-31T21:35:40.223 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-05-31T21:35:40.224 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T21:35:40.224 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-05-31T21:35:40.224 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-05-31T21:35:40.224 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-05-31T21:35:40.225 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-05-31T21:35:40.225 INFO:tasks.rgw_multisite_tests:HEAD 2023-05-31T21:35:40.225 INFO:tasks.rgw_multisite_tests: 2023-05-31T21:35:40.225 INFO:tasks.rgw_multisite_tests: 2023-05-31T21:35:40.225 INFO:tasks.rgw_multisite_tests:Wed, 31 May 2023 20:27:22 GMT 2023-05-31T21:35:40.226 INFO:tasks.rgw_multisite_tests:/pklxkd-55/b 2023-05-31T21:35:40.226 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-05-31T21:35:40.226 INFO:tasks.rgw_multisite_tests:AWS Q80H11OLNU1S48PPYL26:DNPJ7OqoWxtHRIezb8NkCfZP7/U= 2023-05-31T21:35:40.226 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:27:22 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:DNPJ7OqoWxtHRIezb8NkCfZP7/U=', 'Content-Length': '0'} 2023-05-31T21:35:40.227 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:27:16 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ee2ade6ec83adf85-006477adaa-162e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:27:22 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:35:40.227 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/pklxkd-55/b 2023-05-31T21:35:40.227 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/pklxkd-55/b 2023-05-31T21:35:40.227 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-05-31T21:35:40.228 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /pklxkd-55/b 2023-05-31T21:35:40.228 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-05-31T21:35:40.228 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-05-31T21:35:40.228 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T21:35:40.228 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-05-31T21:35:40.229 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-05-31T21:35:40.229 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-05-31T21:35:40.229 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-05-31T21:35:40.229 INFO:tasks.rgw_multisite_tests:HEAD 2023-05-31T21:35:40.230 INFO:tasks.rgw_multisite_tests: 2023-05-31T21:35:40.230 INFO:tasks.rgw_multisite_tests: 2023-05-31T21:35:40.230 INFO:tasks.rgw_multisite_tests:Wed, 31 May 2023 20:27:22 GMT 2023-05-31T21:35:40.230 INFO:tasks.rgw_multisite_tests:/pklxkd-55/b 2023-05-31T21:35:40.231 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-05-31T21:35:40.231 INFO:tasks.rgw_multisite_tests:AWS Q80H11OLNU1S48PPYL26:DNPJ7OqoWxtHRIezb8NkCfZP7/U= 2023-05-31T21:35:40.231 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:27:22 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:DNPJ7OqoWxtHRIezb8NkCfZP7/U=', 'Content-Length': '0'} 2023-05-31T21:35:40.231 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:27:16 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000246302392ec58be-006477adaa-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:27:22 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:35:40.231 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: comparing key name=b 2023-05-31T21:35:40.232 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/pklxkd-55/b 2023-05-31T21:35:40.232 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/pklxkd-55/b 2023-05-31T21:35:40.232 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-05-31T21:35:40.232 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /pklxkd-55/b 2023-05-31T21:35:40.233 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-05-31T21:35:40.233 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-05-31T21:35:40.233 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T21:35:40.233 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-05-31T21:35:40.234 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-05-31T21:35:40.234 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-05-31T21:35:40.234 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-05-31T21:35:40.234 INFO:tasks.rgw_multisite_tests:GET 2023-05-31T21:35:40.235 INFO:tasks.rgw_multisite_tests: 2023-05-31T21:35:40.235 INFO:tasks.rgw_multisite_tests: 2023-05-31T21:35:40.235 INFO:tasks.rgw_multisite_tests:Wed, 31 May 2023 20:27:22 GMT 2023-05-31T21:35:40.235 INFO:tasks.rgw_multisite_tests:/pklxkd-55/b 2023-05-31T21:35:40.235 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-05-31T21:35:40.236 INFO:tasks.rgw_multisite_tests:AWS Q80H11OLNU1S48PPYL26:kwn5VbL4fPr6/U8LpUeyCZxJ52A= 2023-05-31T21:35:40.236 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:27:22 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:kwn5VbL4fPr6/U8LpUeyCZxJ52A=', 'Content-Length': '0'} 2023-05-31T21:35:40.236 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:27:16 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000dd7409e7649fc70f-006477adaa-162e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:27:22 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:35:40.236 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/pklxkd-55/b 2023-05-31T21:35:40.237 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/pklxkd-55/b 2023-05-31T21:35:40.237 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-05-31T21:35:40.237 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /pklxkd-55/b 2023-05-31T21:35:40.237 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-05-31T21:35:40.238 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-05-31T21:35:40.238 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T21:35:40.238 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-05-31T21:35:40.238 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-05-31T21:35:40.239 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-05-31T21:35:40.239 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-05-31T21:35:40.239 INFO:tasks.rgw_multisite_tests:GET 2023-05-31T21:35:40.239 INFO:tasks.rgw_multisite_tests: 2023-05-31T21:35:40.239 INFO:tasks.rgw_multisite_tests: 2023-05-31T21:35:40.240 INFO:tasks.rgw_multisite_tests:Wed, 31 May 2023 20:27:22 GMT 2023-05-31T21:35:40.240 INFO:tasks.rgw_multisite_tests:/pklxkd-55/b 2023-05-31T21:35:40.240 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-05-31T21:35:40.240 INFO:tasks.rgw_multisite_tests:AWS Q80H11OLNU1S48PPYL26:kwn5VbL4fPr6/U8LpUeyCZxJ52A= 2023-05-31T21:35:40.241 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:27:22 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:kwn5VbL4fPr6/U8LpUeyCZxJ52A=', 'Content-Length': '0'} 2023-05-31T21:35:40.241 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:27:16 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000761c1a6885a4dd19-006477adaa-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:27:22 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:35:40.241 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/pklxkd-55/ 2023-05-31T21:35:40.241 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/pklxkd-55/ 2023-05-31T21:35:40.242 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/pklxkd-55/?prefix=b 2023-05-31T21:35:40.242 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/pklxkd-55/?prefix=b 2023-05-31T21:35:40.242 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-05-31T21:35:40.242 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /pklxkd-55/?prefix=b 2023-05-31T21:35:40.242 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-05-31T21:35:40.243 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-05-31T21:35:40.243 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T21:35:40.243 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-05-31T21:35:40.243 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-05-31T21:35:40.244 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-05-31T21:35:40.244 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-05-31T21:35:40.244 INFO:tasks.rgw_multisite_tests:GET 2023-05-31T21:35:40.244 INFO:tasks.rgw_multisite_tests: 2023-05-31T21:35:40.245 INFO:tasks.rgw_multisite_tests: 2023-05-31T21:35:40.245 INFO:tasks.rgw_multisite_tests:Wed, 31 May 2023 20:27:22 GMT 2023-05-31T21:35:40.245 INFO:tasks.rgw_multisite_tests:/pklxkd-55/ 2023-05-31T21:35:40.245 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-05-31T21:35:40.245 INFO:tasks.rgw_multisite_tests:AWS Q80H11OLNU1S48PPYL26:p2p+8G1ZdPifmzf+JcgpB78h0o8= 2023-05-31T21:35:40.246 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:27:22 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:p2p+8G1ZdPifmzf+JcgpB78h0o8=', 'Content-Length': '0'} 2023-05-31T21:35:40.246 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000003e99522e8d609218-006477adaa-162e-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:27:22 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:35:40.246 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'pklxkd-55b1000falseb2023-05-31T20:27:16.850Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T21:35:40.246 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/pklxkd-55/ 2023-05-31T21:35:40.247 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/pklxkd-55/ 2023-05-31T21:35:40.247 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/pklxkd-55/?prefix=b 2023-05-31T21:35:40.247 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/pklxkd-55/?prefix=b 2023-05-31T21:35:40.247 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-05-31T21:35:40.248 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /pklxkd-55/?prefix=b 2023-05-31T21:35:40.248 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-05-31T21:35:40.248 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-05-31T21:35:40.248 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T21:35:40.249 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-05-31T21:35:40.249 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-05-31T21:35:40.249 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-05-31T21:35:40.249 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-05-31T21:35:40.249 INFO:tasks.rgw_multisite_tests:GET 2023-05-31T21:35:40.250 INFO:tasks.rgw_multisite_tests: 2023-05-31T21:35:40.250 INFO:tasks.rgw_multisite_tests: 2023-05-31T21:35:40.250 INFO:tasks.rgw_multisite_tests:Wed, 31 May 2023 20:27:22 GMT 2023-05-31T21:35:40.250 INFO:tasks.rgw_multisite_tests:/pklxkd-55/ 2023-05-31T21:35:40.251 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-05-31T21:35:40.251 INFO:tasks.rgw_multisite_tests:AWS Q80H11OLNU1S48PPYL26:p2p+8G1ZdPifmzf+JcgpB78h0o8= 2023-05-31T21:35:40.251 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:27:22 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:p2p+8G1ZdPifmzf+JcgpB78h0o8=', 'Content-Length': '0'} 2023-05-31T21:35:40.251 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000001a3d66db016228de-006477adaa-1630-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:27:22 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:35:40.251 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'pklxkd-55b1000falseb2023-05-31T20:27:16.850Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T21:35:40.252 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: comparing key name=c 2023-05-31T21:35:40.252 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/pklxkd-55/c 2023-05-31T21:35:40.252 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/pklxkd-55/c 2023-05-31T21:35:40.252 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/pklxkd-55/c?versionId=null 2023-05-31T21:35:40.253 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/pklxkd-55/c?versionId=null 2023-05-31T21:35:40.253 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-05-31T21:35:40.253 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /pklxkd-55/c?versionId=null 2023-05-31T21:35:40.253 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-05-31T21:35:40.254 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-05-31T21:35:40.254 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T21:35:40.254 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-05-31T21:35:40.254 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-05-31T21:35:40.254 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-05-31T21:35:40.255 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-05-31T21:35:40.255 INFO:tasks.rgw_multisite_tests:GET 2023-05-31T21:35:40.255 INFO:tasks.rgw_multisite_tests: 2023-05-31T21:35:40.255 INFO:tasks.rgw_multisite_tests: 2023-05-31T21:35:40.256 INFO:tasks.rgw_multisite_tests:Wed, 31 May 2023 20:27:22 GMT 2023-05-31T21:35:40.256 INFO:tasks.rgw_multisite_tests:/pklxkd-55/c?versionId=null 2023-05-31T21:35:40.256 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-05-31T21:35:40.256 INFO:tasks.rgw_multisite_tests:AWS Q80H11OLNU1S48PPYL26:Axcde+P5vBM6RZWi3eIHC7wJjlI= 2023-05-31T21:35:40.257 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:27:22 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:Axcde+P5vBM6RZWi3eIHC7wJjlI=', 'Content-Length': '0'} 2023-05-31T21:35:40.257 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:27:16 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000390541f92c487c68-006477adaa-162e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:27:22 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:35:40.257 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/pklxkd-55/c 2023-05-31T21:35:40.257 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/pklxkd-55/c 2023-05-31T21:35:40.257 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/pklxkd-55/c?versionId=null 2023-05-31T21:35:40.258 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/pklxkd-55/c?versionId=null 2023-05-31T21:35:40.258 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-05-31T21:35:40.258 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /pklxkd-55/c?versionId=null 2023-05-31T21:35:40.258 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-05-31T21:35:40.259 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-05-31T21:35:40.259 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T21:35:40.259 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-05-31T21:35:40.259 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-05-31T21:35:40.260 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-05-31T21:35:40.260 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-05-31T21:35:40.260 INFO:tasks.rgw_multisite_tests:GET 2023-05-31T21:35:40.260 INFO:tasks.rgw_multisite_tests: 2023-05-31T21:35:40.260 INFO:tasks.rgw_multisite_tests: 2023-05-31T21:35:40.261 INFO:tasks.rgw_multisite_tests:Wed, 31 May 2023 20:27:22 GMT 2023-05-31T21:35:40.261 INFO:tasks.rgw_multisite_tests:/pklxkd-55/c?versionId=null 2023-05-31T21:35:40.261 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-05-31T21:35:40.261 INFO:tasks.rgw_multisite_tests:AWS Q80H11OLNU1S48PPYL26:Axcde+P5vBM6RZWi3eIHC7wJjlI= 2023-05-31T21:35:40.262 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:27:22 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:Axcde+P5vBM6RZWi3eIHC7wJjlI=', 'Content-Length': '0'} 2023-05-31T21:35:40.262 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:27:16 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006054613406236015-006477adaa-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:27:22 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:35:40.262 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/pklxkd-55/c 2023-05-31T21:35:40.262 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/pklxkd-55/c 2023-05-31T21:35:40.263 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/pklxkd-55/c?versionId=null 2023-05-31T21:35:40.263 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/pklxkd-55/c?versionId=null 2023-05-31T21:35:40.263 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-05-31T21:35:40.263 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /pklxkd-55/c?versionId=null 2023-05-31T21:35:40.263 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-05-31T21:35:40.264 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-05-31T21:35:40.264 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T21:35:40.264 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-05-31T21:35:40.264 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-05-31T21:35:40.265 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-05-31T21:35:40.265 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-05-31T21:35:40.265 INFO:tasks.rgw_multisite_tests:HEAD 2023-05-31T21:35:40.265 INFO:tasks.rgw_multisite_tests: 2023-05-31T21:35:40.266 INFO:tasks.rgw_multisite_tests: 2023-05-31T21:35:40.266 INFO:tasks.rgw_multisite_tests:Wed, 31 May 2023 20:27:22 GMT 2023-05-31T21:35:40.266 INFO:tasks.rgw_multisite_tests:/pklxkd-55/c?versionId=null 2023-05-31T21:35:40.266 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-05-31T21:35:40.266 INFO:tasks.rgw_multisite_tests:AWS Q80H11OLNU1S48PPYL26:c1OaLJXgCJtqsCxdKj3F4riZ4YE= 2023-05-31T21:35:40.267 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:27:22 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:c1OaLJXgCJtqsCxdKj3F4riZ4YE=', 'Content-Length': '0'} 2023-05-31T21:35:40.267 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:27:16 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004665bdfd69a2f295-006477adaa-162e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:27:22 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:35:40.267 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/pklxkd-55/c 2023-05-31T21:35:40.267 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/pklxkd-55/c 2023-05-31T21:35:40.268 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/pklxkd-55/c?versionId=null 2023-05-31T21:35:40.268 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/pklxkd-55/c?versionId=null 2023-05-31T21:35:40.268 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-05-31T21:35:40.268 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /pklxkd-55/c?versionId=null 2023-05-31T21:35:40.269 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-05-31T21:35:40.269 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-05-31T21:35:40.269 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T21:35:40.269 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-05-31T21:35:40.269 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-05-31T21:35:40.270 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-05-31T21:35:40.270 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-05-31T21:35:40.270 INFO:tasks.rgw_multisite_tests:HEAD 2023-05-31T21:35:40.270 INFO:tasks.rgw_multisite_tests: 2023-05-31T21:35:40.271 INFO:tasks.rgw_multisite_tests: 2023-05-31T21:35:40.271 INFO:tasks.rgw_multisite_tests:Wed, 31 May 2023 20:27:22 GMT 2023-05-31T21:35:40.271 INFO:tasks.rgw_multisite_tests:/pklxkd-55/c?versionId=null 2023-05-31T21:35:40.271 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-05-31T21:35:40.272 INFO:tasks.rgw_multisite_tests:AWS Q80H11OLNU1S48PPYL26:c1OaLJXgCJtqsCxdKj3F4riZ4YE= 2023-05-31T21:35:40.272 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:27:22 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:c1OaLJXgCJtqsCxdKj3F4riZ4YE=', 'Content-Length': '0'} 2023-05-31T21:35:40.272 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:27:16 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000052cf79f09b60ac8e-006477adaa-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:27:22 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:35:40.272 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: comparing key name=c 2023-05-31T21:35:40.272 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/pklxkd-55/c 2023-05-31T21:35:40.273 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/pklxkd-55/c 2023-05-31T21:35:40.273 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/pklxkd-55/c?versionId=null 2023-05-31T21:35:40.273 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/pklxkd-55/c?versionId=null 2023-05-31T21:35:40.273 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-05-31T21:35:40.274 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /pklxkd-55/c?versionId=null 2023-05-31T21:35:40.274 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-05-31T21:35:40.274 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-05-31T21:35:40.274 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T21:35:40.275 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-05-31T21:35:40.275 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-05-31T21:35:40.275 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-05-31T21:35:40.275 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-05-31T21:35:40.276 INFO:tasks.rgw_multisite_tests:GET 2023-05-31T21:35:40.276 INFO:tasks.rgw_multisite_tests: 2023-05-31T21:35:40.276 INFO:tasks.rgw_multisite_tests: 2023-05-31T21:35:40.276 INFO:tasks.rgw_multisite_tests:Wed, 31 May 2023 20:27:22 GMT 2023-05-31T21:35:40.277 INFO:tasks.rgw_multisite_tests:/pklxkd-55/c?versionId=null 2023-05-31T21:35:40.277 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-05-31T21:35:40.277 INFO:tasks.rgw_multisite_tests:AWS Q80H11OLNU1S48PPYL26:Axcde+P5vBM6RZWi3eIHC7wJjlI= 2023-05-31T21:35:40.277 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:27:22 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:Axcde+P5vBM6RZWi3eIHC7wJjlI=', 'Content-Length': '0'} 2023-05-31T21:35:40.278 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:27:16 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005cd40993d632e698-006477adaa-162e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:27:22 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:35:40.278 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/pklxkd-55/c 2023-05-31T21:35:40.278 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/pklxkd-55/c 2023-05-31T21:35:40.278 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/pklxkd-55/c?versionId=null 2023-05-31T21:35:40.278 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/pklxkd-55/c?versionId=null 2023-05-31T21:35:40.279 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-05-31T21:35:40.279 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /pklxkd-55/c?versionId=null 2023-05-31T21:35:40.279 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-05-31T21:35:40.279 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-05-31T21:35:40.280 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T21:35:40.280 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-05-31T21:35:40.280 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-05-31T21:35:40.280 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-05-31T21:35:40.281 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-05-31T21:35:40.281 INFO:tasks.rgw_multisite_tests:GET 2023-05-31T21:35:40.281 INFO:tasks.rgw_multisite_tests: 2023-05-31T21:35:40.281 INFO:tasks.rgw_multisite_tests: 2023-05-31T21:35:40.281 INFO:tasks.rgw_multisite_tests:Wed, 31 May 2023 20:27:22 GMT 2023-05-31T21:35:40.282 INFO:tasks.rgw_multisite_tests:/pklxkd-55/c?versionId=null 2023-05-31T21:35:40.282 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-05-31T21:35:40.282 INFO:tasks.rgw_multisite_tests:AWS Q80H11OLNU1S48PPYL26:Axcde+P5vBM6RZWi3eIHC7wJjlI= 2023-05-31T21:35:40.282 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:27:22 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:Axcde+P5vBM6RZWi3eIHC7wJjlI=', 'Content-Length': '0'} 2023-05-31T21:35:40.283 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:27:16 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000094e82baf5979d383-006477adaa-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:27:22 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:35:40.283 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/pklxkd-55/ 2023-05-31T21:35:40.283 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/pklxkd-55/ 2023-05-31T21:35:40.283 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/pklxkd-55/?prefix=c 2023-05-31T21:35:40.284 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/pklxkd-55/?prefix=c 2023-05-31T21:35:40.284 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-05-31T21:35:40.284 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /pklxkd-55/?prefix=c 2023-05-31T21:35:40.284 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-05-31T21:35:40.285 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-05-31T21:35:40.285 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T21:35:40.285 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-05-31T21:35:40.285 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-05-31T21:35:40.285 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-05-31T21:35:40.286 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-05-31T21:35:40.286 INFO:tasks.rgw_multisite_tests:GET 2023-05-31T21:35:40.286 INFO:tasks.rgw_multisite_tests: 2023-05-31T21:35:40.286 INFO:tasks.rgw_multisite_tests: 2023-05-31T21:35:40.287 INFO:tasks.rgw_multisite_tests:Wed, 31 May 2023 20:27:22 GMT 2023-05-31T21:35:40.287 INFO:tasks.rgw_multisite_tests:/pklxkd-55/ 2023-05-31T21:35:40.287 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-05-31T21:35:40.287 INFO:tasks.rgw_multisite_tests:AWS Q80H11OLNU1S48PPYL26:p2p+8G1ZdPifmzf+JcgpB78h0o8= 2023-05-31T21:35:40.288 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:27:22 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:p2p+8G1ZdPifmzf+JcgpB78h0o8=', 'Content-Length': '0'} 2023-05-31T21:35:40.288 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000075a35bcf99357963-006477adaa-162e-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:27:22 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:35:40.288 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'pklxkd-55c1000falsec2023-05-31T20:27:16.869Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T21:35:40.288 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/pklxkd-55/ 2023-05-31T21:35:40.289 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/pklxkd-55/ 2023-05-31T21:35:40.289 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/pklxkd-55/?prefix=c 2023-05-31T21:35:40.289 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/pklxkd-55/?prefix=c 2023-05-31T21:35:40.289 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-05-31T21:35:40.289 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /pklxkd-55/?prefix=c 2023-05-31T21:35:40.290 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-05-31T21:35:40.290 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-05-31T21:35:40.290 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T21:35:40.290 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-05-31T21:35:40.291 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-05-31T21:35:40.291 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-05-31T21:35:40.291 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-05-31T21:35:40.291 INFO:tasks.rgw_multisite_tests:GET 2023-05-31T21:35:40.292 INFO:tasks.rgw_multisite_tests: 2023-05-31T21:35:40.292 INFO:tasks.rgw_multisite_tests: 2023-05-31T21:35:40.292 INFO:tasks.rgw_multisite_tests:Wed, 31 May 2023 20:27:22 GMT 2023-05-31T21:35:40.292 INFO:tasks.rgw_multisite_tests:/pklxkd-55/ 2023-05-31T21:35:40.292 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-05-31T21:35:40.293 INFO:tasks.rgw_multisite_tests:AWS Q80H11OLNU1S48PPYL26:p2p+8G1ZdPifmzf+JcgpB78h0o8= 2023-05-31T21:35:40.293 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:27:22 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:p2p+8G1ZdPifmzf+JcgpB78h0o8=', 'Content-Length': '0'} 2023-05-31T21:35:40.293 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000004ec23211a3c2c4f0-006477adaa-1630-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:27:22 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:35:40.293 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'pklxkd-55c1000falsec2023-05-31T20:27:16.869Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T21:35:40.294 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/pklxkd-55/c 2023-05-31T21:35:40.294 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/pklxkd-55/c 2023-05-31T21:35:40.294 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-05-31T21:35:40.294 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /pklxkd-55/c 2023-05-31T21:35:40.295 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-05-31T21:35:40.295 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-05-31T21:35:40.295 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T21:35:40.295 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-05-31T21:35:40.295 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-05-31T21:35:40.296 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-05-31T21:35:40.296 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-05-31T21:35:40.296 INFO:tasks.rgw_multisite_tests:HEAD 2023-05-31T21:35:40.296 INFO:tasks.rgw_multisite_tests: 2023-05-31T21:35:40.297 INFO:tasks.rgw_multisite_tests: 2023-05-31T21:35:40.297 INFO:tasks.rgw_multisite_tests:Wed, 31 May 2023 20:27:23 GMT 2023-05-31T21:35:40.297 INFO:tasks.rgw_multisite_tests:/pklxkd-55/c 2023-05-31T21:35:40.297 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-05-31T21:35:40.298 INFO:tasks.rgw_multisite_tests:AWS Q80H11OLNU1S48PPYL26:hE/j1mXzS//fRGUA9sEWdmEaUWA= 2023-05-31T21:35:40.298 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:27:23 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:hE/j1mXzS//fRGUA9sEWdmEaUWA=', 'Content-Length': '0'} 2023-05-31T21:35:40.298 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:27:16 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d2f715476b53260d-006477adaa-162e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:27:22 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:35:40.298 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/pklxkd-55/c 2023-05-31T21:35:40.299 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/pklxkd-55/c 2023-05-31T21:35:40.299 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-05-31T21:35:40.299 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /pklxkd-55/c 2023-05-31T21:35:40.299 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-05-31T21:35:40.299 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-05-31T21:35:40.300 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T21:35:40.300 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-05-31T21:35:40.300 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-05-31T21:35:40.300 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-05-31T21:35:40.301 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-05-31T21:35:40.301 INFO:tasks.rgw_multisite_tests:HEAD 2023-05-31T21:35:40.301 INFO:tasks.rgw_multisite_tests: 2023-05-31T21:35:40.301 INFO:tasks.rgw_multisite_tests: 2023-05-31T21:35:40.302 INFO:tasks.rgw_multisite_tests:Wed, 31 May 2023 20:27:23 GMT 2023-05-31T21:35:40.302 INFO:tasks.rgw_multisite_tests:/pklxkd-55/c 2023-05-31T21:35:40.302 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-05-31T21:35:40.302 INFO:tasks.rgw_multisite_tests:AWS Q80H11OLNU1S48PPYL26:hE/j1mXzS//fRGUA9sEWdmEaUWA= 2023-05-31T21:35:40.302 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:27:23 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:hE/j1mXzS//fRGUA9sEWdmEaUWA=', 'Content-Length': '0'} 2023-05-31T21:35:40.303 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:27:16 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000298f60bfe1f4e56d-006477adab-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:27:23 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:35:40.303 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: comparing key name=c 2023-05-31T21:35:40.303 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/pklxkd-55/c 2023-05-31T21:35:40.303 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/pklxkd-55/c 2023-05-31T21:35:40.304 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-05-31T21:35:40.304 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /pklxkd-55/c 2023-05-31T21:35:40.304 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-05-31T21:35:40.304 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-05-31T21:35:40.305 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T21:35:40.305 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-05-31T21:35:40.305 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-05-31T21:35:40.305 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-05-31T21:35:40.306 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-05-31T21:35:40.306 INFO:tasks.rgw_multisite_tests:GET 2023-05-31T21:35:40.306 INFO:tasks.rgw_multisite_tests: 2023-05-31T21:35:40.306 INFO:tasks.rgw_multisite_tests: 2023-05-31T21:35:40.306 INFO:tasks.rgw_multisite_tests:Wed, 31 May 2023 20:27:23 GMT 2023-05-31T21:35:40.307 INFO:tasks.rgw_multisite_tests:/pklxkd-55/c 2023-05-31T21:35:40.307 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-05-31T21:35:40.307 INFO:tasks.rgw_multisite_tests:AWS Q80H11OLNU1S48PPYL26:OLZbK6QwcjGMgpIZWtkjzEEZDJ4= 2023-05-31T21:35:40.307 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:27:23 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:OLZbK6QwcjGMgpIZWtkjzEEZDJ4=', 'Content-Length': '0'} 2023-05-31T21:35:40.308 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:27:16 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000015b3e9865681ac4a-006477adab-162e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:27:23 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:35:40.308 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/pklxkd-55/c 2023-05-31T21:35:40.308 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/pklxkd-55/c 2023-05-31T21:35:40.308 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-05-31T21:35:40.309 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /pklxkd-55/c 2023-05-31T21:35:40.309 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-05-31T21:35:40.309 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-05-31T21:35:40.309 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T21:35:40.310 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-05-31T21:35:40.310 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-05-31T21:35:40.310 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-05-31T21:35:40.310 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-05-31T21:35:40.310 INFO:tasks.rgw_multisite_tests:GET 2023-05-31T21:35:40.311 INFO:tasks.rgw_multisite_tests: 2023-05-31T21:35:40.311 INFO:tasks.rgw_multisite_tests: 2023-05-31T21:35:40.311 INFO:tasks.rgw_multisite_tests:Wed, 31 May 2023 20:27:23 GMT 2023-05-31T21:35:40.311 INFO:tasks.rgw_multisite_tests:/pklxkd-55/c 2023-05-31T21:35:40.312 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-05-31T21:35:40.312 INFO:tasks.rgw_multisite_tests:AWS Q80H11OLNU1S48PPYL26:OLZbK6QwcjGMgpIZWtkjzEEZDJ4= 2023-05-31T21:35:40.312 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:27:23 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:OLZbK6QwcjGMgpIZWtkjzEEZDJ4=', 'Content-Length': '0'} 2023-05-31T21:35:40.312 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:27:16 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000dce80df79a3d64c5-006477adab-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:27:23 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:35:40.313 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/pklxkd-55/ 2023-05-31T21:35:40.313 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/pklxkd-55/ 2023-05-31T21:35:40.313 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/pklxkd-55/?prefix=c 2023-05-31T21:35:40.313 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/pklxkd-55/?prefix=c 2023-05-31T21:35:40.313 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-05-31T21:35:40.314 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /pklxkd-55/?prefix=c 2023-05-31T21:35:40.314 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-05-31T21:35:40.314 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-05-31T21:35:40.314 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T21:35:40.315 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-05-31T21:35:40.315 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-05-31T21:35:40.315 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-05-31T21:35:40.315 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-05-31T21:35:40.316 INFO:tasks.rgw_multisite_tests:GET 2023-05-31T21:35:40.316 INFO:tasks.rgw_multisite_tests: 2023-05-31T21:35:40.316 INFO:tasks.rgw_multisite_tests: 2023-05-31T21:35:40.316 INFO:tasks.rgw_multisite_tests:Wed, 31 May 2023 20:27:23 GMT 2023-05-31T21:35:40.316 INFO:tasks.rgw_multisite_tests:/pklxkd-55/ 2023-05-31T21:35:40.317 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-05-31T21:35:40.317 INFO:tasks.rgw_multisite_tests:AWS Q80H11OLNU1S48PPYL26:h67TlTI878JFBrep/coHQ2+SsZE= 2023-05-31T21:35:40.317 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:27:23 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:h67TlTI878JFBrep/coHQ2+SsZE=', 'Content-Length': '0'} 2023-05-31T21:35:40.317 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c1c337f5e81424b3-006477adab-162e-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:27:23 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:35:40.318 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'pklxkd-55c1000falsec2023-05-31T20:27:16.869Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T21:35:40.318 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/pklxkd-55/ 2023-05-31T21:35:40.318 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/pklxkd-55/ 2023-05-31T21:35:40.318 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/pklxkd-55/?prefix=c 2023-05-31T21:35:40.319 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/pklxkd-55/?prefix=c 2023-05-31T21:35:40.319 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-05-31T21:35:40.319 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /pklxkd-55/?prefix=c 2023-05-31T21:35:40.319 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-05-31T21:35:40.319 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-05-31T21:35:40.320 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T21:35:40.320 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-05-31T21:35:40.320 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-05-31T21:35:40.320 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-05-31T21:35:40.321 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-05-31T21:35:40.321 INFO:tasks.rgw_multisite_tests:GET 2023-05-31T21:35:40.321 INFO:tasks.rgw_multisite_tests: 2023-05-31T21:35:40.321 INFO:tasks.rgw_multisite_tests: 2023-05-31T21:35:40.322 INFO:tasks.rgw_multisite_tests:Wed, 31 May 2023 20:27:23 GMT 2023-05-31T21:35:40.322 INFO:tasks.rgw_multisite_tests:/pklxkd-55/ 2023-05-31T21:35:40.322 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-05-31T21:35:40.322 INFO:tasks.rgw_multisite_tests:AWS Q80H11OLNU1S48PPYL26:h67TlTI878JFBrep/coHQ2+SsZE= 2023-05-31T21:35:40.322 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:27:23 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:h67TlTI878JFBrep/coHQ2+SsZE=', 'Content-Length': '0'} 2023-05-31T21:35:40.323 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000006c91edda8ee74844-006477adab-1630-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:27:23 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:35:40.323 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'pklxkd-55c1000falsec2023-05-31T20:27:16.869Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T21:35:40.323 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: comparing key name=d 2023-05-31T21:35:40.323 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/pklxkd-55/d 2023-05-31T21:35:40.324 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/pklxkd-55/d 2023-05-31T21:35:40.324 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/pklxkd-55/d?versionId=null 2023-05-31T21:35:40.324 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/pklxkd-55/d?versionId=null 2023-05-31T21:35:40.324 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-05-31T21:35:40.325 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /pklxkd-55/d?versionId=null 2023-05-31T21:35:40.325 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-05-31T21:35:40.325 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-05-31T21:35:40.325 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T21:35:40.325 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-05-31T21:35:40.326 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-05-31T21:35:40.326 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-05-31T21:35:40.326 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-05-31T21:35:40.326 INFO:tasks.rgw_multisite_tests:GET 2023-05-31T21:35:40.327 INFO:tasks.rgw_multisite_tests: 2023-05-31T21:35:40.327 INFO:tasks.rgw_multisite_tests: 2023-05-31T21:35:40.327 INFO:tasks.rgw_multisite_tests:Wed, 31 May 2023 20:27:23 GMT 2023-05-31T21:35:40.327 INFO:tasks.rgw_multisite_tests:/pklxkd-55/d?versionId=null 2023-05-31T21:35:40.328 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-05-31T21:35:40.328 INFO:tasks.rgw_multisite_tests:AWS Q80H11OLNU1S48PPYL26:twzFSNx7ajueQ8o8KAKeg8K/Oa4= 2023-05-31T21:35:40.328 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:27:23 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:twzFSNx7ajueQ8o8KAKeg8K/Oa4=', 'Content-Length': '0'} 2023-05-31T21:35:40.328 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:27:16 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e401a2203e72f942-006477adab-162e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:27:23 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:35:40.329 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/pklxkd-55/d 2023-05-31T21:35:40.329 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/pklxkd-55/d 2023-05-31T21:35:40.329 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/pklxkd-55/d?versionId=null 2023-05-31T21:35:40.329 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/pklxkd-55/d?versionId=null 2023-05-31T21:35:40.330 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-05-31T21:35:40.330 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /pklxkd-55/d?versionId=null 2023-05-31T21:35:40.330 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-05-31T21:35:40.330 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-05-31T21:35:40.330 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T21:35:40.331 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-05-31T21:35:40.331 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-05-31T21:35:40.331 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-05-31T21:35:40.331 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-05-31T21:35:40.332 INFO:tasks.rgw_multisite_tests:GET 2023-05-31T21:35:40.332 INFO:tasks.rgw_multisite_tests: 2023-05-31T21:35:40.332 INFO:tasks.rgw_multisite_tests: 2023-05-31T21:35:40.332 INFO:tasks.rgw_multisite_tests:Wed, 31 May 2023 20:27:23 GMT 2023-05-31T21:35:40.333 INFO:tasks.rgw_multisite_tests:/pklxkd-55/d?versionId=null 2023-05-31T21:35:40.333 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-05-31T21:35:40.333 INFO:tasks.rgw_multisite_tests:AWS Q80H11OLNU1S48PPYL26:twzFSNx7ajueQ8o8KAKeg8K/Oa4= 2023-05-31T21:35:40.333 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:27:23 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:twzFSNx7ajueQ8o8KAKeg8K/Oa4=', 'Content-Length': '0'} 2023-05-31T21:35:40.333 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:27:16 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000067f445248bff5cec-006477adab-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:27:23 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:35:40.334 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/pklxkd-55/d 2023-05-31T21:35:40.334 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/pklxkd-55/d 2023-05-31T21:35:40.334 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/pklxkd-55/d?versionId=null 2023-05-31T21:35:40.334 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/pklxkd-55/d?versionId=null 2023-05-31T21:35:40.335 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-05-31T21:35:40.335 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /pklxkd-55/d?versionId=null 2023-05-31T21:35:40.335 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-05-31T21:35:40.335 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-05-31T21:35:40.336 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T21:35:40.336 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-05-31T21:35:40.336 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-05-31T21:35:40.336 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-05-31T21:35:40.337 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-05-31T21:35:40.337 INFO:tasks.rgw_multisite_tests:HEAD 2023-05-31T21:35:40.337 INFO:tasks.rgw_multisite_tests: 2023-05-31T21:35:40.337 INFO:tasks.rgw_multisite_tests: 2023-05-31T21:35:40.337 INFO:tasks.rgw_multisite_tests:Wed, 31 May 2023 20:27:23 GMT 2023-05-31T21:35:40.338 INFO:tasks.rgw_multisite_tests:/pklxkd-55/d?versionId=null 2023-05-31T21:35:40.338 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-05-31T21:35:40.338 INFO:tasks.rgw_multisite_tests:AWS Q80H11OLNU1S48PPYL26:cA98pb1KdBFha1CdV/G2h7uc3nw= 2023-05-31T21:35:40.338 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:27:23 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:cA98pb1KdBFha1CdV/G2h7uc3nw=', 'Content-Length': '0'} 2023-05-31T21:35:40.339 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:27:16 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c7c0b4df5a45c2d7-006477adab-162e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:27:23 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:35:40.339 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/pklxkd-55/d 2023-05-31T21:35:40.339 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/pklxkd-55/d 2023-05-31T21:35:40.339 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/pklxkd-55/d?versionId=null 2023-05-31T21:35:40.340 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/pklxkd-55/d?versionId=null 2023-05-31T21:35:40.340 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-05-31T21:35:40.340 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /pklxkd-55/d?versionId=null 2023-05-31T21:35:40.340 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-05-31T21:35:40.340 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-05-31T21:35:40.341 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T21:35:40.341 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-05-31T21:35:40.341 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-05-31T21:35:40.341 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-05-31T21:35:40.342 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-05-31T21:35:40.342 INFO:tasks.rgw_multisite_tests:HEAD 2023-05-31T21:35:40.342 INFO:tasks.rgw_multisite_tests: 2023-05-31T21:35:40.342 INFO:tasks.rgw_multisite_tests: 2023-05-31T21:35:40.343 INFO:tasks.rgw_multisite_tests:Wed, 31 May 2023 20:27:23 GMT 2023-05-31T21:35:40.343 INFO:tasks.rgw_multisite_tests:/pklxkd-55/d?versionId=null 2023-05-31T21:35:40.343 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-05-31T21:35:40.343 INFO:tasks.rgw_multisite_tests:AWS Q80H11OLNU1S48PPYL26:cA98pb1KdBFha1CdV/G2h7uc3nw= 2023-05-31T21:35:40.343 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:27:23 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:cA98pb1KdBFha1CdV/G2h7uc3nw=', 'Content-Length': '0'} 2023-05-31T21:35:40.344 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:27:16 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ed938085a76b3cea-006477adab-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:27:23 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:35:40.344 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: comparing key name=d 2023-05-31T21:35:40.344 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/pklxkd-55/d 2023-05-31T21:35:40.344 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/pklxkd-55/d 2023-05-31T21:35:40.345 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/pklxkd-55/d?versionId=null 2023-05-31T21:35:40.345 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/pklxkd-55/d?versionId=null 2023-05-31T21:35:40.345 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-05-31T21:35:40.345 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /pklxkd-55/d?versionId=null 2023-05-31T21:35:40.346 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-05-31T21:35:40.346 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-05-31T21:35:40.346 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T21:35:40.346 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-05-31T21:35:40.346 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-05-31T21:35:40.347 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-05-31T21:35:40.347 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-05-31T21:35:40.347 INFO:tasks.rgw_multisite_tests:GET 2023-05-31T21:35:40.347 INFO:tasks.rgw_multisite_tests: 2023-05-31T21:35:40.348 INFO:tasks.rgw_multisite_tests: 2023-05-31T21:35:40.348 INFO:tasks.rgw_multisite_tests:Wed, 31 May 2023 20:27:23 GMT 2023-05-31T21:35:40.348 INFO:tasks.rgw_multisite_tests:/pklxkd-55/d?versionId=null 2023-05-31T21:35:40.348 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-05-31T21:35:40.349 INFO:tasks.rgw_multisite_tests:AWS Q80H11OLNU1S48PPYL26:twzFSNx7ajueQ8o8KAKeg8K/Oa4= 2023-05-31T21:35:40.349 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:27:23 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:twzFSNx7ajueQ8o8KAKeg8K/Oa4=', 'Content-Length': '0'} 2023-05-31T21:35:40.349 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:27:16 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006a74cbfd8a276537-006477adab-162e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:27:23 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:35:40.349 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/pklxkd-55/d 2023-05-31T21:35:40.349 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/pklxkd-55/d 2023-05-31T21:35:40.350 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/pklxkd-55/d?versionId=null 2023-05-31T21:35:40.350 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/pklxkd-55/d?versionId=null 2023-05-31T21:35:40.350 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-05-31T21:35:40.350 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /pklxkd-55/d?versionId=null 2023-05-31T21:35:40.351 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-05-31T21:35:40.351 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-05-31T21:35:40.351 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T21:35:40.351 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-05-31T21:35:40.352 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-05-31T21:35:40.352 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-05-31T21:35:40.352 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-05-31T21:35:40.352 INFO:tasks.rgw_multisite_tests:GET 2023-05-31T21:35:40.352 INFO:tasks.rgw_multisite_tests: 2023-05-31T21:35:40.353 INFO:tasks.rgw_multisite_tests: 2023-05-31T21:35:40.353 INFO:tasks.rgw_multisite_tests:Wed, 31 May 2023 20:27:23 GMT 2023-05-31T21:35:40.353 INFO:tasks.rgw_multisite_tests:/pklxkd-55/d?versionId=null 2023-05-31T21:35:40.353 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-05-31T21:35:40.354 INFO:tasks.rgw_multisite_tests:AWS Q80H11OLNU1S48PPYL26:twzFSNx7ajueQ8o8KAKeg8K/Oa4= 2023-05-31T21:35:40.354 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:27:23 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:twzFSNx7ajueQ8o8KAKeg8K/Oa4=', 'Content-Length': '0'} 2023-05-31T21:35:40.354 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:27:16 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006c78464ce6bf70c3-006477adab-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:27:23 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:35:40.354 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/pklxkd-55/ 2023-05-31T21:35:40.355 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/pklxkd-55/ 2023-05-31T21:35:40.355 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/pklxkd-55/?prefix=d 2023-05-31T21:35:40.355 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/pklxkd-55/?prefix=d 2023-05-31T21:35:40.355 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-05-31T21:35:40.355 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /pklxkd-55/?prefix=d 2023-05-31T21:35:40.356 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-05-31T21:35:40.356 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-05-31T21:35:40.356 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T21:35:40.356 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-05-31T21:35:40.357 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-05-31T21:35:40.357 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-05-31T21:35:40.357 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-05-31T21:35:40.357 INFO:tasks.rgw_multisite_tests:GET 2023-05-31T21:35:40.358 INFO:tasks.rgw_multisite_tests: 2023-05-31T21:35:40.358 INFO:tasks.rgw_multisite_tests: 2023-05-31T21:35:40.358 INFO:tasks.rgw_multisite_tests:Wed, 31 May 2023 20:27:23 GMT 2023-05-31T21:35:40.358 INFO:tasks.rgw_multisite_tests:/pklxkd-55/ 2023-05-31T21:35:40.359 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-05-31T21:35:40.359 INFO:tasks.rgw_multisite_tests:AWS Q80H11OLNU1S48PPYL26:h67TlTI878JFBrep/coHQ2+SsZE= 2023-05-31T21:35:40.359 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:27:23 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:h67TlTI878JFBrep/coHQ2+SsZE=', 'Content-Length': '0'} 2023-05-31T21:35:40.359 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e29636335d2b7e4b-006477adab-162e-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:27:23 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:35:40.359 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'pklxkd-55d1000falsed2023-05-31T20:27:16.893Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T21:35:40.360 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/pklxkd-55/ 2023-05-31T21:35:40.360 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/pklxkd-55/ 2023-05-31T21:35:40.360 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/pklxkd-55/?prefix=d 2023-05-31T21:35:40.360 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/pklxkd-55/?prefix=d 2023-05-31T21:35:40.361 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-05-31T21:35:40.361 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /pklxkd-55/?prefix=d 2023-05-31T21:35:40.361 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-05-31T21:35:40.361 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-05-31T21:35:40.362 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T21:35:40.362 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-05-31T21:35:40.362 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-05-31T21:35:40.362 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-05-31T21:35:40.362 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-05-31T21:35:40.363 INFO:tasks.rgw_multisite_tests:GET 2023-05-31T21:35:40.363 INFO:tasks.rgw_multisite_tests: 2023-05-31T21:35:40.363 INFO:tasks.rgw_multisite_tests: 2023-05-31T21:35:40.363 INFO:tasks.rgw_multisite_tests:Wed, 31 May 2023 20:27:23 GMT 2023-05-31T21:35:40.364 INFO:tasks.rgw_multisite_tests:/pklxkd-55/ 2023-05-31T21:35:40.364 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-05-31T21:35:40.364 INFO:tasks.rgw_multisite_tests:AWS Q80H11OLNU1S48PPYL26:h67TlTI878JFBrep/coHQ2+SsZE= 2023-05-31T21:35:40.364 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:27:23 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:h67TlTI878JFBrep/coHQ2+SsZE=', 'Content-Length': '0'} 2023-05-31T21:35:40.365 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f2333e093b9295d1-006477adab-1630-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:27:23 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:35:40.365 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'pklxkd-55d1000falsed2023-05-31T20:27:16.893Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T21:35:40.365 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/pklxkd-55/d 2023-05-31T21:35:40.365 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/pklxkd-55/d 2023-05-31T21:35:40.365 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-05-31T21:35:40.366 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /pklxkd-55/d 2023-05-31T21:35:40.366 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-05-31T21:35:40.366 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-05-31T21:35:40.366 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T21:35:40.367 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-05-31T21:35:40.367 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-05-31T21:35:40.367 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-05-31T21:35:40.367 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-05-31T21:35:40.368 INFO:tasks.rgw_multisite_tests:HEAD 2023-05-31T21:35:40.368 INFO:tasks.rgw_multisite_tests: 2023-05-31T21:35:40.368 INFO:tasks.rgw_multisite_tests: 2023-05-31T21:35:40.368 INFO:tasks.rgw_multisite_tests:Wed, 31 May 2023 20:27:23 GMT 2023-05-31T21:35:40.368 INFO:tasks.rgw_multisite_tests:/pklxkd-55/d 2023-05-31T21:35:40.369 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-05-31T21:35:40.369 INFO:tasks.rgw_multisite_tests:AWS Q80H11OLNU1S48PPYL26:ixht+ZZRHyFfTSXd83JA6A3PTzA= 2023-05-31T21:35:40.369 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:27:23 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:ixht+ZZRHyFfTSXd83JA6A3PTzA=', 'Content-Length': '0'} 2023-05-31T21:35:40.369 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:27:16 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000cb540719c5ef7aeb-006477adab-162e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:27:23 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:35:40.370 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/pklxkd-55/d 2023-05-31T21:35:40.370 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/pklxkd-55/d 2023-05-31T21:35:40.370 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-05-31T21:35:40.370 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /pklxkd-55/d 2023-05-31T21:35:40.371 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-05-31T21:35:40.371 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-05-31T21:35:40.371 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T21:35:40.371 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-05-31T21:35:40.371 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-05-31T21:35:40.372 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-05-31T21:35:40.372 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-05-31T21:35:40.372 INFO:tasks.rgw_multisite_tests:HEAD 2023-05-31T21:35:40.372 INFO:tasks.rgw_multisite_tests: 2023-05-31T21:35:40.373 INFO:tasks.rgw_multisite_tests: 2023-05-31T21:35:40.373 INFO:tasks.rgw_multisite_tests:Wed, 31 May 2023 20:27:23 GMT 2023-05-31T21:35:40.373 INFO:tasks.rgw_multisite_tests:/pklxkd-55/d 2023-05-31T21:35:40.373 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-05-31T21:35:40.374 INFO:tasks.rgw_multisite_tests:AWS Q80H11OLNU1S48PPYL26:ixht+ZZRHyFfTSXd83JA6A3PTzA= 2023-05-31T21:35:40.374 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:27:23 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:ixht+ZZRHyFfTSXd83JA6A3PTzA=', 'Content-Length': '0'} 2023-05-31T21:35:40.374 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:27:16 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006ec1bf2928d4f6b3-006477adab-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:27:23 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:35:40.374 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: comparing key name=d 2023-05-31T21:35:40.374 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/pklxkd-55/d 2023-05-31T21:35:40.375 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/pklxkd-55/d 2023-05-31T21:35:40.375 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-05-31T21:35:40.375 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /pklxkd-55/d 2023-05-31T21:35:40.375 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-05-31T21:35:40.376 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-05-31T21:35:40.376 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T21:35:40.376 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-05-31T21:35:40.377 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-05-31T21:35:40.377 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-05-31T21:35:40.377 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-05-31T21:35:40.377 INFO:tasks.rgw_multisite_tests:GET 2023-05-31T21:35:40.377 INFO:tasks.rgw_multisite_tests: 2023-05-31T21:35:40.378 INFO:tasks.rgw_multisite_tests: 2023-05-31T21:35:40.378 INFO:tasks.rgw_multisite_tests:Wed, 31 May 2023 20:27:23 GMT 2023-05-31T21:35:40.378 INFO:tasks.rgw_multisite_tests:/pklxkd-55/d 2023-05-31T21:35:40.378 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-05-31T21:35:40.379 INFO:tasks.rgw_multisite_tests:AWS Q80H11OLNU1S48PPYL26:DjRBoypgizW522NCxCdGLAVZDyI= 2023-05-31T21:35:40.379 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:27:23 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:DjRBoypgizW522NCxCdGLAVZDyI=', 'Content-Length': '0'} 2023-05-31T21:35:40.379 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:27:16 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a537be9948a819e2-006477adab-162e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:27:23 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:35:40.379 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/pklxkd-55/d 2023-05-31T21:35:40.380 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/pklxkd-55/d 2023-05-31T21:35:40.380 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-05-31T21:35:40.380 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /pklxkd-55/d 2023-05-31T21:35:40.380 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-05-31T21:35:40.381 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-05-31T21:35:40.381 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T21:35:40.381 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-05-31T21:35:40.381 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-05-31T21:35:40.381 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-05-31T21:35:40.382 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-05-31T21:35:40.382 INFO:tasks.rgw_multisite_tests:GET 2023-05-31T21:35:40.382 INFO:tasks.rgw_multisite_tests: 2023-05-31T21:35:40.382 INFO:tasks.rgw_multisite_tests: 2023-05-31T21:35:40.383 INFO:tasks.rgw_multisite_tests:Wed, 31 May 2023 20:27:23 GMT 2023-05-31T21:35:40.383 INFO:tasks.rgw_multisite_tests:/pklxkd-55/d 2023-05-31T21:35:40.383 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-05-31T21:35:40.383 INFO:tasks.rgw_multisite_tests:AWS Q80H11OLNU1S48PPYL26:DjRBoypgizW522NCxCdGLAVZDyI= 2023-05-31T21:35:40.383 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:27:23 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:DjRBoypgizW522NCxCdGLAVZDyI=', 'Content-Length': '0'} 2023-05-31T21:35:40.384 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:27:16 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000946627bd2bdf9764-006477adab-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:27:23 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:35:40.384 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/pklxkd-55/ 2023-05-31T21:35:40.384 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/pklxkd-55/ 2023-05-31T21:35:40.384 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/pklxkd-55/?prefix=d 2023-05-31T21:35:40.385 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/pklxkd-55/?prefix=d 2023-05-31T21:35:40.385 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-05-31T21:35:40.385 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /pklxkd-55/?prefix=d 2023-05-31T21:35:40.385 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-05-31T21:35:40.386 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-05-31T21:35:40.386 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T21:35:40.386 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-05-31T21:35:40.386 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-05-31T21:35:40.387 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-05-31T21:35:40.387 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-05-31T21:35:40.387 INFO:tasks.rgw_multisite_tests:GET 2023-05-31T21:35:40.387 INFO:tasks.rgw_multisite_tests: 2023-05-31T21:35:40.387 INFO:tasks.rgw_multisite_tests: 2023-05-31T21:35:40.388 INFO:tasks.rgw_multisite_tests:Wed, 31 May 2023 20:27:23 GMT 2023-05-31T21:35:40.388 INFO:tasks.rgw_multisite_tests:/pklxkd-55/ 2023-05-31T21:35:40.388 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-05-31T21:35:40.388 INFO:tasks.rgw_multisite_tests:AWS Q80H11OLNU1S48PPYL26:h67TlTI878JFBrep/coHQ2+SsZE= 2023-05-31T21:35:40.389 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:27:23 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:h67TlTI878JFBrep/coHQ2+SsZE=', 'Content-Length': '0'} 2023-05-31T21:35:40.389 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000003b622a3165f84704-006477adab-162e-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:27:23 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:35:40.389 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'pklxkd-55d1000falsed2023-05-31T20:27:16.893Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T21:35:40.389 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/pklxkd-55/ 2023-05-31T21:35:40.390 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/pklxkd-55/ 2023-05-31T21:35:40.390 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/pklxkd-55/?prefix=d 2023-05-31T21:35:40.390 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/pklxkd-55/?prefix=d 2023-05-31T21:35:40.390 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-05-31T21:35:40.391 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /pklxkd-55/?prefix=d 2023-05-31T21:35:40.391 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-05-31T21:35:40.391 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-05-31T21:35:40.391 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T21:35:40.391 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-05-31T21:35:40.392 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-05-31T21:35:40.392 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-05-31T21:35:40.392 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-05-31T21:35:40.392 INFO:tasks.rgw_multisite_tests:GET 2023-05-31T21:35:40.393 INFO:tasks.rgw_multisite_tests: 2023-05-31T21:35:40.393 INFO:tasks.rgw_multisite_tests: 2023-05-31T21:35:40.393 INFO:tasks.rgw_multisite_tests:Wed, 31 May 2023 20:27:23 GMT 2023-05-31T21:35:40.393 INFO:tasks.rgw_multisite_tests:/pklxkd-55/ 2023-05-31T21:35:40.394 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-05-31T21:35:40.394 INFO:tasks.rgw_multisite_tests:AWS Q80H11OLNU1S48PPYL26:h67TlTI878JFBrep/coHQ2+SsZE= 2023-05-31T21:35:40.394 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:27:23 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:h67TlTI878JFBrep/coHQ2+SsZE=', 'Content-Length': '0'} 2023-05-31T21:35:40.394 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000fe3a7becea55f06a-006477adab-1630-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:27:23 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:35:40.394 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'pklxkd-55d1000falsed2023-05-31T20:27:16.893Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T21:35:40.395 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: INFO: success, bucket identical: bucket=pklxkd-55 zones={a2, a1} 2023-05-31T21:35:40.395 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['bucket', 'layout', '--bucket', 'pklxkd-55', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-05-31T21:35:40.395 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'layout', '--bucket', 'pklxkd-55', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-05-31T21:35:40.395 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket layout --bucket pklxkd-55 --cluster c1 --debug-rgw 0 --debug-ms 0 2023-05-31T21:35:40.396 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: ignoring --setuser ceph since I am not root 2023-05-31T21:35:40.396 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-05-31T21:35:40.396 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2023-05-31T21:35:40.396 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "layout": { 2023-05-31T21:35:40.397 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "resharding": "None", 2023-05-31T21:35:40.397 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "current_index": { 2023-05-31T21:35:40.397 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "gen": 0, 2023-05-31T21:35:40.397 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "layout": { 2023-05-31T21:35:40.397 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "type": "Normal", 2023-05-31T21:35:40.398 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "normal": { 2023-05-31T21:35:40.398 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "num_shards": 11, 2023-05-31T21:35:40.398 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "hash_type": "Mod" 2023-05-31T21:35:40.398 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2023-05-31T21:35:40.399 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2023-05-31T21:35:40.399 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2023-05-31T21:35:40.399 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "logs": [ 2023-05-31T21:35:40.399 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2023-05-31T21:35:40.400 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "gen": 0, 2023-05-31T21:35:40.400 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "layout": { 2023-05-31T21:35:40.400 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "type": "InIndex", 2023-05-31T21:35:40.400 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "in_index": { 2023-05-31T21:35:40.400 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "gen": 0, 2023-05-31T21:35:40.401 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "layout": { 2023-05-31T21:35:40.401 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "num_shards": 11, 2023-05-31T21:35:40.401 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "hash_type": "Mod" 2023-05-31T21:35:40.401 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2023-05-31T21:35:40.402 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2023-05-31T21:35:40.402 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2023-05-31T21:35:40.402 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2023-05-31T21:35:40.402 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: ] 2023-05-31T21:35:40.403 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2023-05-31T21:35:40.403 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2023-05-31T21:35:40.403 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['bilog', 'list', '--bucket', 'pklxkd-55', '--gen', '0', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-31T21:35:40.403 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'list', '--bucket', 'pklxkd-55', '--gen', '0', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-31T21:35:40.403 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog list --bucket pklxkd-55 --gen 0 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-05-31T21:35:40.404 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: ignoring --setuser ceph since I am not root 2023-05-31T21:35:40.404 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-05-31T21:35:40.404 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: [ 2023-05-31T21:35:40.404 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2023-05-31T21:35:40.405 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "op_id": "1#00000000001.28.6", 2023-05-31T21:35:40.405 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "op_tag": "3d163b1e-09c7-47e2-ae87-add4aace3f38.5680.912616895933462705", 2023-05-31T21:35:40.405 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "op": "write", 2023-05-31T21:35:40.405 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "object": "a", 2023-05-31T21:35:40.406 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "instance": "", 2023-05-31T21:35:40.406 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "state": "complete", 2023-05-31T21:35:40.406 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "index_ver": 1, 2023-05-31T21:35:40.406 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "timestamp": "2023-05-31T20:27:16.792147057Z", 2023-05-31T21:35:40.406 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "ver": { 2023-05-31T21:35:40.407 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "pool": 9, 2023-05-31T21:35:40.407 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "epoch": 1 2023-05-31T21:35:40.407 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2023-05-31T21:35:40.407 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "bilog_flags": 0, 2023-05-31T21:35:40.408 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "versioned": false, 2023-05-31T21:35:40.408 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "owner": "", 2023-05-31T21:35:40.408 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "owner_display_name": "", 2023-05-31T21:35:40.408 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "zones_trace": [ 2023-05-31T21:35:40.409 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2023-05-31T21:35:40.409 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "entry": "3d163b1e-09c7-47e2-ae87-add4aace3f38:pklxkd-55:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.20" 2023-05-31T21:35:40.409 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2023-05-31T21:35:40.409 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: ] 2023-05-31T21:35:40.409 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2023-05-31T21:35:40.410 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2023-05-31T21:35:40.410 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "op_id": "5#00000000001.45.6", 2023-05-31T21:35:40.410 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "op_tag": "3d163b1e-09c7-47e2-ae87-add4aace3f38.5680.15231583731067601239", 2023-05-31T21:35:40.410 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "op": "write", 2023-05-31T21:35:40.411 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "object": "d", 2023-05-31T21:35:40.411 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "instance": "", 2023-05-31T21:35:40.411 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "state": "complete", 2023-05-31T21:35:40.411 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "index_ver": 1, 2023-05-31T21:35:40.412 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "timestamp": "2023-05-31T20:27:16.893760052Z", 2023-05-31T21:35:40.412 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "ver": { 2023-05-31T21:35:40.412 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "pool": 9, 2023-05-31T21:35:40.412 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "epoch": 3 2023-05-31T21:35:40.412 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2023-05-31T21:35:40.413 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "bilog_flags": 0, 2023-05-31T21:35:40.413 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "versioned": false, 2023-05-31T21:35:40.413 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "owner": "", 2023-05-31T21:35:40.413 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "owner_display_name": "", 2023-05-31T21:35:40.414 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "zones_trace": [ 2023-05-31T21:35:40.414 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2023-05-31T21:35:40.414 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "entry": "3d163b1e-09c7-47e2-ae87-add4aace3f38:pklxkd-55:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.20" 2023-05-31T21:35:40.414 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2023-05-31T21:35:40.415 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: ] 2023-05-31T21:35:40.415 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2023-05-31T21:35:40.415 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2023-05-31T21:35:40.415 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "op_id": "8#00000000001.42.6", 2023-05-31T21:35:40.415 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "op_tag": "3d163b1e-09c7-47e2-ae87-add4aace3f38.5680.17607519642157465194", 2023-05-31T21:35:40.416 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "op": "write", 2023-05-31T21:35:40.416 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "object": "c", 2023-05-31T21:35:40.416 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "instance": "", 2023-05-31T21:35:40.416 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "state": "complete", 2023-05-31T21:35:40.417 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "index_ver": 1, 2023-05-31T21:35:40.417 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "timestamp": "2023-05-31T20:27:16.869625413Z", 2023-05-31T21:35:40.417 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "ver": { 2023-05-31T21:35:40.417 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "pool": 9, 2023-05-31T21:35:40.418 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "epoch": 4 2023-05-31T21:35:40.418 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2023-05-31T21:35:40.418 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "bilog_flags": 0, 2023-05-31T21:35:40.418 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "versioned": false, 2023-05-31T21:35:40.418 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "owner": "", 2023-05-31T21:35:40.419 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "owner_display_name": "", 2023-05-31T21:35:40.419 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "zones_trace": [ 2023-05-31T21:35:40.419 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2023-05-31T21:35:40.419 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "entry": "3d163b1e-09c7-47e2-ae87-add4aace3f38:pklxkd-55:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.20" 2023-05-31T21:35:40.420 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2023-05-31T21:35:40.420 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: ] 2023-05-31T21:35:40.420 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2023-05-31T21:35:40.420 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2023-05-31T21:35:40.421 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "op_id": "10#00000000001.108.6", 2023-05-31T21:35:40.421 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "op_tag": "3d163b1e-09c7-47e2-ae87-add4aace3f38.5680.4722290385692952109", 2023-05-31T21:35:40.421 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "op": "write", 2023-05-31T21:35:40.421 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "object": "b", 2023-05-31T21:35:40.422 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "instance": "", 2023-05-31T21:35:40.422 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "state": "complete", 2023-05-31T21:35:40.422 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "index_ver": 1, 2023-05-31T21:35:40.422 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "timestamp": "2023-05-31T20:27:16.850404114Z", 2023-05-31T21:35:40.423 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "ver": { 2023-05-31T21:35:40.423 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "pool": 9, 2023-05-31T21:35:40.423 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "epoch": 2 2023-05-31T21:35:40.423 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2023-05-31T21:35:40.424 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "bilog_flags": 0, 2023-05-31T21:35:40.424 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "versioned": false, 2023-05-31T21:35:40.424 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "owner": "", 2023-05-31T21:35:40.424 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "owner_display_name": "", 2023-05-31T21:35:40.424 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "zones_trace": [ 2023-05-31T21:35:40.425 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2023-05-31T21:35:40.425 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "entry": "3d163b1e-09c7-47e2-ae87-add4aace3f38:pklxkd-55:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.20" 2023-05-31T21:35:40.425 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2023-05-31T21:35:40.425 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: ] 2023-05-31T21:35:40.426 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2023-05-31T21:35:40.426 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: ] 2023-05-31T21:35:40.426 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['bucket', 'reshard', '--bucket', 'pklxkd-55', '--num-shards', '3', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-05-31T21:35:40.426 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'reshard', '--bucket', 'pklxkd-55', '--num-shards', '3', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-05-31T21:35:40.427 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket reshard --bucket pklxkd-55 --num-shards 3 --yes-i-really-mean-it --cluster c1 --debug-rgw 0 --debug-ms 0 2023-05-31T21:35:40.427 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: ignoring --setuser ceph since I am not root 2023-05-31T21:35:40.427 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-05-31T21:35:40.427 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: tenant: 2023-05-31T21:35:40.427 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: bucket name: pklxkd-55 2023-05-31T21:35:40.428 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: total entries: 4 2023-05-31T21:35:40.428 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['bucket', 'layout', '--bucket', 'pklxkd-55', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-05-31T21:35:40.428 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'layout', '--bucket', 'pklxkd-55', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-05-31T21:35:40.428 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket layout --bucket pklxkd-55 --cluster c1 --debug-rgw 0 --debug-ms 0 2023-05-31T21:35:40.429 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: ignoring --setuser ceph since I am not root 2023-05-31T21:35:40.429 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-05-31T21:35:40.429 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2023-05-31T21:35:40.429 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "layout": { 2023-05-31T21:35:40.430 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "resharding": "None", 2023-05-31T21:35:40.430 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "current_index": { 2023-05-31T21:35:40.430 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "gen": 1, 2023-05-31T21:35:40.430 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "layout": { 2023-05-31T21:35:40.431 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "type": "Normal", 2023-05-31T21:35:40.431 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "normal": { 2023-05-31T21:35:40.431 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "num_shards": 3, 2023-05-31T21:35:40.431 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "hash_type": "Mod" 2023-05-31T21:35:40.431 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2023-05-31T21:35:40.432 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2023-05-31T21:35:40.432 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2023-05-31T21:35:40.432 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "logs": [ 2023-05-31T21:35:40.432 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2023-05-31T21:35:40.433 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "gen": 0, 2023-05-31T21:35:40.433 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "layout": { 2023-05-31T21:35:40.433 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "type": "InIndex", 2023-05-31T21:35:40.433 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "in_index": { 2023-05-31T21:35:40.434 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "gen": 0, 2023-05-31T21:35:40.434 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "layout": { 2023-05-31T21:35:40.434 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "num_shards": 11, 2023-05-31T21:35:40.434 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "hash_type": "Mod" 2023-05-31T21:35:40.435 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2023-05-31T21:35:40.435 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2023-05-31T21:35:40.435 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2023-05-31T21:35:40.435 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2023-05-31T21:35:40.436 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2023-05-31T21:35:40.436 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "gen": 1, 2023-05-31T21:35:40.436 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "layout": { 2023-05-31T21:35:40.436 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "type": "InIndex", 2023-05-31T21:35:40.437 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "in_index": { 2023-05-31T21:35:40.437 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "gen": 1, 2023-05-31T21:35:40.437 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "layout": { 2023-05-31T21:35:40.437 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "num_shards": 3, 2023-05-31T21:35:40.437 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "hash_type": "Mod" 2023-05-31T21:35:40.438 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2023-05-31T21:35:40.438 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2023-05-31T21:35:40.438 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2023-05-31T21:35:40.438 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2023-05-31T21:35:40.439 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: ] 2023-05-31T21:35:40.439 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2023-05-31T21:35:40.439 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2023-05-31T21:35:40.439 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['bilog', 'list', '--bucket', 'pklxkd-55', '--gen', '1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-31T21:35:40.440 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'list', '--bucket', 'pklxkd-55', '--gen', '1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-31T21:35:40.440 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog list --bucket pklxkd-55 --gen 1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-05-31T21:35:40.440 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: ignoring --setuser ceph since I am not root 2023-05-31T21:35:40.440 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-05-31T21:35:40.440 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: [] 2023-05-31T21:35:40.441 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/pklxkd-55/ 2023-05-31T21:35:40.441 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/pklxkd-55/ 2023-05-31T21:35:40.441 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-05-31T21:35:40.441 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /pklxkd-55/ 2023-05-31T21:35:40.442 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-05-31T21:35:40.442 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-05-31T21:35:40.442 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T21:35:40.442 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-05-31T21:35:40.443 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-05-31T21:35:40.443 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-05-31T21:35:40.443 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-05-31T21:35:40.443 INFO:tasks.rgw_multisite_tests:HEAD 2023-05-31T21:35:40.443 INFO:tasks.rgw_multisite_tests: 2023-05-31T21:35:40.444 INFO:tasks.rgw_multisite_tests: 2023-05-31T21:35:40.444 INFO:tasks.rgw_multisite_tests:Wed, 31 May 2023 20:27:23 GMT 2023-05-31T21:35:40.444 INFO:tasks.rgw_multisite_tests:/pklxkd-55/ 2023-05-31T21:35:40.444 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-05-31T21:35:40.445 INFO:tasks.rgw_multisite_tests:AWS Q80H11OLNU1S48PPYL26:CEgAiulrYDLboHO703TUJQSDrdU= 2023-05-31T21:35:40.445 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:27:23 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:CEgAiulrYDLboHO703TUJQSDrdU=', 'Content-Length': '0'} 2023-05-31T21:35:40.445 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '4'), ('X-RGW-Bytes-Used', '12'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000003602e9329a2cfacb-006477adab-1630-a1'), ('Content-Length', '0'), ('Date', 'Wed, 31 May 2023 20:27:23 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:35:40.445 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/pklxkd-55/e 2023-05-31T21:35:40.446 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/pklxkd-55/e 2023-05-31T21:35:40.446 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2023-05-31T21:35:40.446 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /pklxkd-55/e 2023-05-31T21:35:40.446 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-05-31T21:35:40.447 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-05-31T21:35:40.447 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T21:35:40.447 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-05-31T21:35:40.447 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-05-31T21:35:40.447 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-05-31T21:35:40.448 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-05-31T21:35:40.448 INFO:tasks.rgw_multisite_tests:PUT 2023-05-31T21:35:40.448 INFO:tasks.rgw_multisite_tests:rL0Y20zC+Fzt72VPzMSk2A== 2023-05-31T21:35:40.448 INFO:tasks.rgw_multisite_tests:application/octet-stream 2023-05-31T21:35:40.449 INFO:tasks.rgw_multisite_tests:Wed, 31 May 2023 20:27:23 GMT 2023-05-31T21:35:40.449 INFO:tasks.rgw_multisite_tests:/pklxkd-55/e 2023-05-31T21:35:40.449 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-05-31T21:35:40.449 INFO:tasks.rgw_multisite_tests:AWS Q80H11OLNU1S48PPYL26:YG9ufmH3Qho9/nfJWt8bHk8kHpo= 2023-05-31T21:35:40.450 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Wed, 31 May 2023 20:27:23 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:YG9ufmH3Qho9/nfJWt8bHk8kHpo='} 2023-05-31T21:35:40.450 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000c9b76a37aa6279f0-006477adab-1630-a1'), ('Date', 'Wed, 31 May 2023 20:27:23 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:35:40.450 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/pklxkd-55/ 2023-05-31T21:35:40.450 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/pklxkd-55/ 2023-05-31T21:35:40.450 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-05-31T21:35:40.451 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /pklxkd-55/ 2023-05-31T21:35:40.451 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-05-31T21:35:40.451 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-05-31T21:35:40.451 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T21:35:40.452 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-05-31T21:35:40.452 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-05-31T21:35:40.452 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-05-31T21:35:40.452 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-05-31T21:35:40.453 INFO:tasks.rgw_multisite_tests:HEAD 2023-05-31T21:35:40.453 INFO:tasks.rgw_multisite_tests: 2023-05-31T21:35:40.453 INFO:tasks.rgw_multisite_tests: 2023-05-31T21:35:40.453 INFO:tasks.rgw_multisite_tests:Wed, 31 May 2023 20:27:23 GMT 2023-05-31T21:35:40.453 INFO:tasks.rgw_multisite_tests:/pklxkd-55/ 2023-05-31T21:35:40.454 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-05-31T21:35:40.454 INFO:tasks.rgw_multisite_tests:AWS Q80H11OLNU1S48PPYL26:CEgAiulrYDLboHO703TUJQSDrdU= 2023-05-31T21:35:40.454 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:27:23 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:CEgAiulrYDLboHO703TUJQSDrdU=', 'Content-Length': '0'} 2023-05-31T21:35:40.454 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '5'), ('X-RGW-Bytes-Used', '15'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000c47b2a42d23d98bf-006477adab-1630-a1'), ('Content-Length', '0'), ('Date', 'Wed, 31 May 2023 20:27:23 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:35:40.455 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/pklxkd-55/f 2023-05-31T21:35:40.455 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/pklxkd-55/f 2023-05-31T21:35:40.455 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2023-05-31T21:35:40.455 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /pklxkd-55/f 2023-05-31T21:35:40.456 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-05-31T21:35:40.456 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-05-31T21:35:40.456 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T21:35:40.456 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-05-31T21:35:40.457 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-05-31T21:35:40.457 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-05-31T21:35:40.457 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-05-31T21:35:40.457 INFO:tasks.rgw_multisite_tests:PUT 2023-05-31T21:35:40.457 INFO:tasks.rgw_multisite_tests:rL0Y20zC+Fzt72VPzMSk2A== 2023-05-31T21:35:40.458 INFO:tasks.rgw_multisite_tests:application/octet-stream 2023-05-31T21:35:40.458 INFO:tasks.rgw_multisite_tests:Wed, 31 May 2023 20:27:23 GMT 2023-05-31T21:35:40.458 INFO:tasks.rgw_multisite_tests:/pklxkd-55/f 2023-05-31T21:35:40.458 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-05-31T21:35:40.459 INFO:tasks.rgw_multisite_tests:AWS Q80H11OLNU1S48PPYL26:a1pWMm25Au1uR3xgcR7gQfQNXjU= 2023-05-31T21:35:40.459 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Wed, 31 May 2023 20:27:23 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:a1pWMm25Au1uR3xgcR7gQfQNXjU='} 2023-05-31T21:35:40.459 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000510ec301fc0fbe51-006477adab-1630-a1'), ('Date', 'Wed, 31 May 2023 20:27:23 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:35:40.459 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/pklxkd-55/ 2023-05-31T21:35:40.460 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/pklxkd-55/ 2023-05-31T21:35:40.460 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-05-31T21:35:40.460 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /pklxkd-55/ 2023-05-31T21:35:40.460 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-05-31T21:35:40.461 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-05-31T21:35:40.461 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T21:35:40.461 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-05-31T21:35:40.461 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-05-31T21:35:40.461 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-05-31T21:35:40.462 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-05-31T21:35:40.462 INFO:tasks.rgw_multisite_tests:HEAD 2023-05-31T21:35:40.462 INFO:tasks.rgw_multisite_tests: 2023-05-31T21:35:40.462 INFO:tasks.rgw_multisite_tests: 2023-05-31T21:35:40.463 INFO:tasks.rgw_multisite_tests:Wed, 31 May 2023 20:27:24 GMT 2023-05-31T21:35:40.463 INFO:tasks.rgw_multisite_tests:/pklxkd-55/ 2023-05-31T21:35:40.463 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-05-31T21:35:40.463 INFO:tasks.rgw_multisite_tests:AWS Q80H11OLNU1S48PPYL26:Ar+db5NHPQANb45BVa+KAir9ejw= 2023-05-31T21:35:40.464 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:27:24 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:Ar+db5NHPQANb45BVa+KAir9ejw=', 'Content-Length': '0'} 2023-05-31T21:35:40.464 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '6'), ('X-RGW-Bytes-Used', '18'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000ca0be5ef7e1d6577-006477adac-1630-a1'), ('Content-Length', '0'), ('Date', 'Wed, 31 May 2023 20:27:24 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:35:40.464 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/pklxkd-55/g 2023-05-31T21:35:40.464 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/pklxkd-55/g 2023-05-31T21:35:40.464 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2023-05-31T21:35:40.465 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /pklxkd-55/g 2023-05-31T21:35:40.465 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-05-31T21:35:40.465 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-05-31T21:35:40.465 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T21:35:40.466 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-05-31T21:35:40.466 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-05-31T21:35:40.466 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-05-31T21:35:40.466 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-05-31T21:35:40.467 INFO:tasks.rgw_multisite_tests:PUT 2023-05-31T21:35:40.467 INFO:tasks.rgw_multisite_tests:rL0Y20zC+Fzt72VPzMSk2A== 2023-05-31T21:35:40.467 INFO:tasks.rgw_multisite_tests:application/octet-stream 2023-05-31T21:35:40.467 INFO:tasks.rgw_multisite_tests:Wed, 31 May 2023 20:27:24 GMT 2023-05-31T21:35:40.467 INFO:tasks.rgw_multisite_tests:/pklxkd-55/g 2023-05-31T21:35:40.468 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-05-31T21:35:40.468 INFO:tasks.rgw_multisite_tests:AWS Q80H11OLNU1S48PPYL26:3qgXKe/W/MZUJUlYldMKkKa7YJ0= 2023-05-31T21:35:40.468 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Wed, 31 May 2023 20:27:24 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:3qgXKe/W/MZUJUlYldMKkKa7YJ0='} 2023-05-31T21:35:40.468 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000fcc73590820242b0-006477adac-1630-a1'), ('Date', 'Wed, 31 May 2023 20:27:24 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:35:40.469 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/pklxkd-55/ 2023-05-31T21:35:40.469 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/pklxkd-55/ 2023-05-31T21:35:40.469 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-05-31T21:35:40.469 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /pklxkd-55/ 2023-05-31T21:35:40.470 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-05-31T21:35:40.470 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-05-31T21:35:40.470 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T21:35:40.470 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-05-31T21:35:40.470 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-05-31T21:35:40.471 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-05-31T21:35:40.471 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-05-31T21:35:40.471 INFO:tasks.rgw_multisite_tests:HEAD 2023-05-31T21:35:40.471 INFO:tasks.rgw_multisite_tests: 2023-05-31T21:35:40.472 INFO:tasks.rgw_multisite_tests: 2023-05-31T21:35:40.472 INFO:tasks.rgw_multisite_tests:Wed, 31 May 2023 20:27:24 GMT 2023-05-31T21:35:40.472 INFO:tasks.rgw_multisite_tests:/pklxkd-55/ 2023-05-31T21:35:40.472 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-05-31T21:35:40.473 INFO:tasks.rgw_multisite_tests:AWS Q80H11OLNU1S48PPYL26:Ar+db5NHPQANb45BVa+KAir9ejw= 2023-05-31T21:35:40.473 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:27:24 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:Ar+db5NHPQANb45BVa+KAir9ejw=', 'Content-Length': '0'} 2023-05-31T21:35:40.473 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '7'), ('X-RGW-Bytes-Used', '21'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000096f61cb293d6097c-006477adac-1630-a1'), ('Content-Length', '0'), ('Date', 'Wed, 31 May 2023 20:27:24 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:35:40.473 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/pklxkd-55/h 2023-05-31T21:35:40.473 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/pklxkd-55/h 2023-05-31T21:35:40.474 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2023-05-31T21:35:40.474 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /pklxkd-55/h 2023-05-31T21:35:40.474 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-05-31T21:35:40.474 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-05-31T21:35:40.475 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T21:35:40.475 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-05-31T21:35:40.475 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-05-31T21:35:40.475 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-05-31T21:35:40.475 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-05-31T21:35:40.476 INFO:tasks.rgw_multisite_tests:PUT 2023-05-31T21:35:40.476 INFO:tasks.rgw_multisite_tests:rL0Y20zC+Fzt72VPzMSk2A== 2023-05-31T21:35:40.476 INFO:tasks.rgw_multisite_tests:application/octet-stream 2023-05-31T21:35:40.477 INFO:tasks.rgw_multisite_tests:Wed, 31 May 2023 20:27:24 GMT 2023-05-31T21:35:40.477 INFO:tasks.rgw_multisite_tests:/pklxkd-55/h 2023-05-31T21:35:40.477 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-05-31T21:35:40.477 INFO:tasks.rgw_multisite_tests:AWS Q80H11OLNU1S48PPYL26:+TJhWtofpmkyn8DPjTGL0Vvt3WI= 2023-05-31T21:35:40.478 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Wed, 31 May 2023 20:27:24 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:+TJhWtofpmkyn8DPjTGL0Vvt3WI='} 2023-05-31T21:35:40.478 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000cfe08ff100955461-006477adac-1630-a1'), ('Date', 'Wed, 31 May 2023 20:27:24 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:35:40.478 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: bucket checkpoint: source=a1 target=a2 bucket=pklxkd-55 2023-05-31T21:35:40.478 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'pklxkd-55', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-05-31T21:35:40.478 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'pklxkd-55', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-05-31T21:35:40.479 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket pklxkd-55 --source-zone a1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2023-05-31T21:35:40.479 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: ignoring --setuser ceph since I am not root 2023-05-31T21:35:40.479 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-05-31T21:35:40.479 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: 2023-05-31T20:27:24.167+0000 7f0d821a6780 1 waiting to reach latest gen 1, on 0.. 2023-05-31T21:35:40.480 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: 2023-05-31T20:27:29.169+0000 7f0d821a6780 1 waiting to reach latest gen 1, on 0.. 2023-05-31T21:35:40.480 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: 2023-05-31T20:27:34.170+0000 7f0d821a6780 1 waiting to reach latest gen 1, on 0.. 2023-05-31T21:35:40.480 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: 2023-05-31T20:27:39.170+0000 7f0d821a6780 1 waiting to reach latest gen 1, on 0.. 2023-05-31T21:35:40.480 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: 2023-05-31T20:27:44.171+0000 7f0d821a6780 1 waiting for incremental sync to catch up: 2023-05-31T21:35:40.481 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: local status: [, , ] 2023-05-31T21:35:40.481 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: remote markers: [00000000003.52.6, 00000000002.15.6, 00000000002.32.6] 2023-05-31T21:35:40.481 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: 2023-05-31T20:27:49.171+0000 7f0d821a6780 1 waiting for incremental sync to catch up: 2023-05-31T21:35:40.481 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: local status: [, , ] 2023-05-31T21:35:40.482 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: remote markers: [00000000003.52.6, 00000000002.15.6, 00000000002.32.6] 2023-05-31T21:35:40.482 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: 2023-05-31T20:27:54.172+0000 7f0d821a6780 1 waiting for incremental sync to catch up: 2023-05-31T21:35:40.482 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: local status: [, , ] 2023-05-31T21:35:40.482 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: remote markers: [00000000003.52.6, 00000000002.15.6, 00000000002.32.6] 2023-05-31T21:35:40.482 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: 2023-05-31T20:27:59.173+0000 7f0d821a6780 1 waiting for incremental sync to catch up: 2023-05-31T21:35:40.483 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: local status: [, , ] 2023-05-31T21:35:40.483 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: remote markers: [00000000003.52.6, 00000000002.15.6, 00000000002.32.6] 2023-05-31T21:35:40.483 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: 2023-05-31T20:28:04.173+0000 7f0d821a6780 1 waiting for incremental sync to catch up: 2023-05-31T21:35:40.483 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: local status: [, , ] 2023-05-31T21:35:40.484 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: remote markers: [00000000003.52.6, 00000000002.15.6, 00000000002.32.6] 2023-05-31T21:35:40.484 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: 2023-05-31T20:28:09.175+0000 7f0d821a6780 1 waiting for incremental sync to catch up: 2023-05-31T21:35:40.484 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: local status: [, , ] 2023-05-31T21:35:40.484 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: remote markers: [00000000003.52.6, 00000000002.15.6, 00000000002.32.6] 2023-05-31T21:35:40.485 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: 2023-05-31T20:28:14.176+0000 7f0d821a6780 1 waiting for incremental sync to catch up: 2023-05-31T21:35:40.485 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: local status: [, , ] 2023-05-31T21:35:40.485 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: remote markers: [00000000003.52.6, 00000000002.15.6, 00000000002.32.6] 2023-05-31T21:35:40.485 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: 2023-05-31T20:28:19.177+0000 7f0d821a6780 1 waiting for incremental sync to catch up: 2023-05-31T21:35:40.485 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: local status: [, , ] 2023-05-31T21:35:40.486 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: remote markers: [00000000003.52.6, 00000000002.15.6, 00000000002.32.6] 2023-05-31T21:35:40.486 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: 2023-05-31T20:28:24.178+0000 7f0d821a6780 1 waiting for incremental sync to catch up: 2023-05-31T21:35:40.486 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: local status: [, , ] 2023-05-31T21:35:40.486 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: remote markers: [00000000003.52.6, 00000000002.15.6, 00000000002.32.6] 2023-05-31T21:35:40.487 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: 2023-05-31T20:28:29.177+0000 7f0d821a6780 1 waiting for incremental sync to catch up: 2023-05-31T21:35:40.487 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: local status: [, 00000000002.15.6, ] 2023-05-31T21:35:40.487 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: remote markers: [00000000003.52.6, 00000000002.15.6, 00000000002.32.6] 2023-05-31T21:35:40.487 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: 2023-05-31T20:28:34.177+0000 7f0d821a6780 1 waiting for incremental sync to catch up: 2023-05-31T21:35:40.488 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: local status: [, 00000000002.15.6, ] 2023-05-31T21:35:40.488 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: remote markers: [00000000003.52.6, 00000000002.15.6, 00000000002.32.6] 2023-05-31T21:35:40.488 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: 2023-05-31T20:28:39.178+0000 7f0d821a6780 1 waiting for incremental sync to catch up: 2023-05-31T21:35:40.488 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: local status: [, 00000000002.15.6, ] 2023-05-31T21:35:40.489 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: remote markers: [00000000003.52.6, 00000000002.15.6, 00000000002.32.6] 2023-05-31T21:35:40.489 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: 2023-05-31T20:28:44.179+0000 7f0d821a6780 1 waiting for incremental sync to catch up: 2023-05-31T21:35:40.489 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: local status: [, 00000000002.15.6, 00000000002.32.6] 2023-05-31T21:35:40.489 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: remote markers: [00000000003.52.6, 00000000002.15.6, 00000000002.32.6] 2023-05-31T21:35:40.489 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: 2023-05-31T20:28:49.180+0000 7f0d821a6780 1 waiting for incremental sync to catch up: 2023-05-31T21:35:40.490 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: local status: [, 00000000002.15.6, 00000000002.32.6] 2023-05-31T21:35:40.490 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: remote markers: [00000000003.52.6, 00000000002.15.6, 00000000002.32.6] 2023-05-31T21:35:40.490 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: 2023-05-31T20:28:54.181+0000 7f0d821a6780 1 waiting for incremental sync to catch up: 2023-05-31T21:35:40.490 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: local status: [, 00000000002.15.6, 00000000002.32.6] 2023-05-31T21:35:40.491 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: remote markers: [00000000003.52.6, 00000000002.15.6, 00000000002.32.6] 2023-05-31T21:35:40.491 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: 2023-05-31T20:28:59.182+0000 7f0d821a6780 1 waiting for incremental sync to catch up: 2023-05-31T21:35:40.491 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: local status: [, 00000000002.15.6, 00000000002.32.6] 2023-05-31T21:35:40.491 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: remote markers: [00000000003.52.6, 00000000002.15.6, 00000000002.32.6] 2023-05-31T21:35:40.492 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: 2023-05-31T20:29:04.183+0000 7f0d821a6780 1 bucket sync caught up with source: 2023-05-31T21:35:40.492 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: local status: [00000000003.52.6, 00000000002.15.6, 00000000002.32.6] 2023-05-31T21:35:40.492 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: remote markers: [00000000003.52.6, 00000000002.15.6, 00000000002.32.6] 2023-05-31T21:35:40.492 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: 2023-05-31T20:29:04.183+0000 7f0d821a6780 0 bucket checkpoint complete 2023-05-31T21:35:40.492 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: bucket checkpoint: source=a2 target=a1 bucket=pklxkd-55 2023-05-31T21:35:40.493 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'pklxkd-55', '--source-zone', 'a2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-05-31T21:35:40.493 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'pklxkd-55', '--source-zone', 'a2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-05-31T21:35:40.493 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket pklxkd-55 --source-zone a2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2023-05-31T21:35:40.493 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: ignoring --setuser ceph since I am not root 2023-05-31T21:35:40.494 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-05-31T21:35:40.494 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: 2023-05-31T20:29:04.320+0000 7f5722510780 1 waiting for incremental sync to catch up: 2023-05-31T21:35:40.494 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: local status: [, 00000000002.1063.6, , , , 00000000001.1340.6, , , 00000000002.3558.6, , 00000000001.2404.6] 2023-05-31T21:35:40.494 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: remote markers: [, 00000000002.1063.6, , , 00000000001.1666.6, 00000000001.1340.6, , , 00000000002.3558.6, , 00000000002.2408.6] 2023-05-31T21:35:40.495 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: 2023-05-31T20:29:09.322+0000 7f5722510780 1 waiting for incremental sync to catch up: 2023-05-31T21:35:40.495 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: local status: [, 00000000002.1063.6, , , , 00000000001.1340.6, , , 00000000002.3558.6, , 00000000001.2404.6] 2023-05-31T21:35:40.495 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: remote markers: [, 00000000002.1063.6, , , 00000000001.1666.6, 00000000001.1340.6, , , 00000000002.3558.6, , 00000000002.2408.6] 2023-05-31T21:35:40.495 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: 2023-05-31T20:29:14.322+0000 7f5722510780 1 waiting for incremental sync to catch up: 2023-05-31T21:35:40.495 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: local status: [, 00000000002.1063.6, , , , 00000000001.1340.6, , , 00000000002.3558.6, , 00000000001.2404.6] 2023-05-31T21:35:40.496 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: remote markers: [, 00000000002.1063.6, , , 00000000001.1666.6, 00000000001.1340.6, , , 00000000002.3558.6, , 00000000002.2408.6] 2023-05-31T21:35:40.496 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: 2023-05-31T20:29:19.323+0000 7f5722510780 1 waiting for incremental sync to catch up: 2023-05-31T21:35:40.496 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: local status: [, 00000000002.1063.6, , , , 00000000001.1340.6, , , 00000000002.3558.6, , 00000000001.2404.6] 2023-05-31T21:35:40.496 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: remote markers: [, 00000000002.1063.6, , , 00000000001.1666.6, 00000000001.1340.6, , , 00000000002.3558.6, , 00000000002.2408.6] 2023-05-31T21:35:40.497 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: 2023-05-31T20:29:24.324+0000 7f5722510780 1 bucket sync caught up with source: 2023-05-31T21:35:40.497 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: local status: [, 00000000002.1063.6, , , 00000000001.1666.6, 00000000001.1340.6, , , 00000000002.3558.6, , 00000000002.2408.6] 2023-05-31T21:35:40.497 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: remote markers: [, 00000000002.1063.6, , , 00000000001.1666.6, 00000000001.1340.6, , , 00000000002.3558.6, , 00000000002.2408.6] 2023-05-31T21:35:40.497 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: 2023-05-31T20:29:24.324+0000 7f5722510780 0 bucket checkpoint complete 2023-05-31T21:35:40.498 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: INFO: comparing bucket=pklxkd-55 zones={a2, a1} 2023-05-31T21:35:40.498 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/pklxkd-55/ 2023-05-31T21:35:40.498 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/pklxkd-55/ 2023-05-31T21:35:40.498 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-05-31T21:35:40.498 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /pklxkd-55/ 2023-05-31T21:35:40.499 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-05-31T21:35:40.499 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-05-31T21:35:40.499 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T21:35:40.499 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-05-31T21:35:40.500 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-05-31T21:35:40.500 INFO:tasks.rgw_multisite_tests:boto: DEBUG: establishing HTTP connection: kwargs={'timeout': 70, 'port': 8001} 2023-05-31T21:35:40.500 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-05-31T21:35:40.500 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-05-31T21:35:40.501 INFO:tasks.rgw_multisite_tests:HEAD 2023-05-31T21:35:40.501 INFO:tasks.rgw_multisite_tests: 2023-05-31T21:35:40.501 INFO:tasks.rgw_multisite_tests: 2023-05-31T21:35:40.501 INFO:tasks.rgw_multisite_tests:Wed, 31 May 2023 20:29:24 GMT 2023-05-31T21:35:40.501 INFO:tasks.rgw_multisite_tests:/pklxkd-55/ 2023-05-31T21:35:40.502 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-05-31T21:35:40.502 INFO:tasks.rgw_multisite_tests:AWS Q80H11OLNU1S48PPYL26:i72WzALuHR7KCaSfjkxAnzAjKlE= 2023-05-31T21:35:40.502 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:29:24 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:i72WzALuHR7KCaSfjkxAnzAjKlE=', 'Content-Length': '0'} 2023-05-31T21:35:40.502 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '8'), ('X-RGW-Bytes-Used', '24'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000868b23d2978f67e5-006477ae24-162e-a2'), ('Content-Length', '0'), ('Date', 'Wed, 31 May 2023 20:29:24 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:35:40.503 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/pklxkd-55/ 2023-05-31T21:35:40.503 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/pklxkd-55/ 2023-05-31T21:35:40.503 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-05-31T21:35:40.503 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /pklxkd-55/ 2023-05-31T21:35:40.504 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-05-31T21:35:40.504 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-05-31T21:35:40.504 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T21:35:40.504 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-05-31T21:35:40.504 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-05-31T21:35:40.505 INFO:tasks.rgw_multisite_tests:boto: DEBUG: establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2023-05-31T21:35:40.505 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-05-31T21:35:40.505 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-05-31T21:35:40.505 INFO:tasks.rgw_multisite_tests:HEAD 2023-05-31T21:35:40.506 INFO:tasks.rgw_multisite_tests: 2023-05-31T21:35:40.506 INFO:tasks.rgw_multisite_tests: 2023-05-31T21:35:40.506 INFO:tasks.rgw_multisite_tests:Wed, 31 May 2023 20:29:24 GMT 2023-05-31T21:35:40.506 INFO:tasks.rgw_multisite_tests:/pklxkd-55/ 2023-05-31T21:35:40.507 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-05-31T21:35:40.507 INFO:tasks.rgw_multisite_tests:AWS Q80H11OLNU1S48PPYL26:i72WzALuHR7KCaSfjkxAnzAjKlE= 2023-05-31T21:35:40.507 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:29:24 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:i72WzALuHR7KCaSfjkxAnzAjKlE=', 'Content-Length': '0'} 2023-05-31T21:35:40.507 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '8'), ('X-RGW-Bytes-Used', '24'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000074383f9a60d1bb8-006477ae24-1630-a1'), ('Content-Length', '0'), ('Date', 'Wed, 31 May 2023 20:29:24 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:35:40.507 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: bucket1 objects: 2023-05-31T21:35:40.508 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/pklxkd-55/ 2023-05-31T21:35:40.508 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/pklxkd-55/ 2023-05-31T21:35:40.508 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/pklxkd-55/?versions&max-keys=999 2023-05-31T21:35:40.508 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/pklxkd-55/?versions&max-keys=999 2023-05-31T21:35:40.509 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-05-31T21:35:40.509 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /pklxkd-55/?versions&max-keys=999 2023-05-31T21:35:40.509 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-05-31T21:35:40.509 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-05-31T21:35:40.510 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T21:35:40.510 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-05-31T21:35:40.510 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-05-31T21:35:40.510 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-05-31T21:35:40.510 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-05-31T21:35:40.511 INFO:tasks.rgw_multisite_tests:GET 2023-05-31T21:35:40.511 INFO:tasks.rgw_multisite_tests: 2023-05-31T21:35:40.511 INFO:tasks.rgw_multisite_tests: 2023-05-31T21:35:40.511 INFO:tasks.rgw_multisite_tests:Wed, 31 May 2023 20:29:24 GMT 2023-05-31T21:35:40.512 INFO:tasks.rgw_multisite_tests:/pklxkd-55/?versions 2023-05-31T21:35:40.512 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-05-31T21:35:40.512 INFO:tasks.rgw_multisite_tests:AWS Q80H11OLNU1S48PPYL26:+BJqTaR5KWPhPglRdf1Mjt4E5lw= 2023-05-31T21:35:40.512 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:29:24 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:+BJqTaR5KWPhPglRdf1Mjt4E5lw=', 'Content-Length': '0'} 2023-05-31T21:35:40.513 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000000bc1344cc39c9151-006477ae24-162e-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:29:24 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:35:40.513 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'pklxkd-55999falseanulltrue2023-05-31T20:27:16.792Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalbnulltrue2023-05-31T20:27:16.850Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalcnulltrue2023-05-31T20:27:16.869Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormaldnulltrue2023-05-31T20:27:16.893Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalenulltrue2023-05-31T20:27:23.955Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalfnulltrue2023-05-31T20:27:23.977Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalgnulltrue2023-05-31T20:27:24.035Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalhnulltrue2023-05-31T20:27:24.053Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T21:35:40.513 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: o=a 2023-05-31T21:35:40.513 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: o=b 2023-05-31T21:35:40.514 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: o=c 2023-05-31T21:35:40.514 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: o=d 2023-05-31T21:35:40.514 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: o=e 2023-05-31T21:35:40.514 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: o=f 2023-05-31T21:35:40.514 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: o=g 2023-05-31T21:35:40.515 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: o=h 2023-05-31T21:35:40.515 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: bucket2 objects: 2023-05-31T21:35:40.515 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/pklxkd-55/ 2023-05-31T21:35:40.515 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/pklxkd-55/ 2023-05-31T21:35:40.516 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/pklxkd-55/?versions&max-keys=999 2023-05-31T21:35:40.516 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/pklxkd-55/?versions&max-keys=999 2023-05-31T21:35:40.516 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-05-31T21:35:40.516 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /pklxkd-55/?versions&max-keys=999 2023-05-31T21:35:40.517 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-05-31T21:35:40.517 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-05-31T21:35:40.517 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T21:35:40.517 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-05-31T21:35:40.517 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-05-31T21:35:40.518 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-05-31T21:35:40.518 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-05-31T21:35:40.518 INFO:tasks.rgw_multisite_tests:GET 2023-05-31T21:35:40.518 INFO:tasks.rgw_multisite_tests: 2023-05-31T21:35:40.519 INFO:tasks.rgw_multisite_tests: 2023-05-31T21:35:40.519 INFO:tasks.rgw_multisite_tests:Wed, 31 May 2023 20:29:24 GMT 2023-05-31T21:35:40.519 INFO:tasks.rgw_multisite_tests:/pklxkd-55/?versions 2023-05-31T21:35:40.519 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-05-31T21:35:40.519 INFO:tasks.rgw_multisite_tests:AWS Q80H11OLNU1S48PPYL26:+BJqTaR5KWPhPglRdf1Mjt4E5lw= 2023-05-31T21:35:40.520 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:29:24 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:+BJqTaR5KWPhPglRdf1Mjt4E5lw=', 'Content-Length': '0'} 2023-05-31T21:35:40.520 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ddf9e2b1894011a2-006477ae24-1630-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:29:24 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:35:40.520 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'pklxkd-55999falseanulltrue2023-05-31T20:27:16.792Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalbnulltrue2023-05-31T20:27:16.850Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalcnulltrue2023-05-31T20:27:16.869Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormaldnulltrue2023-05-31T20:27:16.893Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalenulltrue2023-05-31T20:27:23.955Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalfnulltrue2023-05-31T20:27:23.977Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalgnulltrue2023-05-31T20:27:24.035Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalhnulltrue2023-05-31T20:27:24.053Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T21:35:40.520 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: o=a 2023-05-31T21:35:40.521 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: o=b 2023-05-31T21:35:40.521 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: o=c 2023-05-31T21:35:40.521 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: o=d 2023-05-31T21:35:40.521 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: o=e 2023-05-31T21:35:40.522 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: o=f 2023-05-31T21:35:40.522 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: o=g 2023-05-31T21:35:40.522 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: o=h 2023-05-31T21:35:40.522 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/pklxkd-55/ 2023-05-31T21:35:40.523 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/pklxkd-55/ 2023-05-31T21:35:40.523 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/pklxkd-55/?versions&max-keys=999 2023-05-31T21:35:40.523 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/pklxkd-55/?versions&max-keys=999 2023-05-31T21:35:40.523 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-05-31T21:35:40.523 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /pklxkd-55/?versions&max-keys=999 2023-05-31T21:35:40.524 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-05-31T21:35:40.524 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-05-31T21:35:40.524 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T21:35:40.524 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-05-31T21:35:40.525 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-05-31T21:35:40.525 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-05-31T21:35:40.525 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-05-31T21:35:40.525 INFO:tasks.rgw_multisite_tests:GET 2023-05-31T21:35:40.526 INFO:tasks.rgw_multisite_tests: 2023-05-31T21:35:40.526 INFO:tasks.rgw_multisite_tests: 2023-05-31T21:35:40.526 INFO:tasks.rgw_multisite_tests:Wed, 31 May 2023 20:29:24 GMT 2023-05-31T21:35:40.526 INFO:tasks.rgw_multisite_tests:/pklxkd-55/?versions 2023-05-31T21:35:40.526 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-05-31T21:35:40.527 INFO:tasks.rgw_multisite_tests:AWS Q80H11OLNU1S48PPYL26:+BJqTaR5KWPhPglRdf1Mjt4E5lw= 2023-05-31T21:35:40.527 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:29:24 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:+BJqTaR5KWPhPglRdf1Mjt4E5lw=', 'Content-Length': '0'} 2023-05-31T21:35:40.527 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ae7ee0fb5dd07b60-006477ae24-162e-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:29:24 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:35:40.527 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'pklxkd-55999falseanulltrue2023-05-31T20:27:16.792Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalbnulltrue2023-05-31T20:27:16.850Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalcnulltrue2023-05-31T20:27:16.869Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormaldnulltrue2023-05-31T20:27:16.893Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalenulltrue2023-05-31T20:27:23.955Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalfnulltrue2023-05-31T20:27:23.977Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalgnulltrue2023-05-31T20:27:24.035Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalhnulltrue2023-05-31T20:27:24.053Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T21:35:40.528 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/pklxkd-55/ 2023-05-31T21:35:40.528 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/pklxkd-55/ 2023-05-31T21:35:40.528 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/pklxkd-55/?versions&max-keys=999 2023-05-31T21:35:40.528 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/pklxkd-55/?versions&max-keys=999 2023-05-31T21:35:40.529 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-05-31T21:35:40.529 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /pklxkd-55/?versions&max-keys=999 2023-05-31T21:35:40.529 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-05-31T21:35:40.529 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-05-31T21:35:40.529 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T21:35:40.530 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-05-31T21:35:40.530 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-05-31T21:35:40.530 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-05-31T21:35:40.530 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-05-31T21:35:40.531 INFO:tasks.rgw_multisite_tests:GET 2023-05-31T21:35:40.531 INFO:tasks.rgw_multisite_tests: 2023-05-31T21:35:40.531 INFO:tasks.rgw_multisite_tests: 2023-05-31T21:35:40.531 INFO:tasks.rgw_multisite_tests:Wed, 31 May 2023 20:29:24 GMT 2023-05-31T21:35:40.532 INFO:tasks.rgw_multisite_tests:/pklxkd-55/?versions 2023-05-31T21:35:40.532 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-05-31T21:35:40.532 INFO:tasks.rgw_multisite_tests:AWS Q80H11OLNU1S48PPYL26:+BJqTaR5KWPhPglRdf1Mjt4E5lw= 2023-05-31T21:35:40.532 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:29:24 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:+BJqTaR5KWPhPglRdf1Mjt4E5lw=', 'Content-Length': '0'} 2023-05-31T21:35:40.532 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d92380a36f1a48fa-006477ae24-1630-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:29:24 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:35:40.533 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'pklxkd-55999falseanulltrue2023-05-31T20:27:16.792Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalbnulltrue2023-05-31T20:27:16.850Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalcnulltrue2023-05-31T20:27:16.869Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormaldnulltrue2023-05-31T20:27:16.893Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalenulltrue2023-05-31T20:27:23.955Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalfnulltrue2023-05-31T20:27:23.977Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalgnulltrue2023-05-31T20:27:24.035Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalhnulltrue2023-05-31T20:27:24.053Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T21:35:40.533 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: comparing key name=a 2023-05-31T21:35:40.533 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/pklxkd-55/a 2023-05-31T21:35:40.533 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/pklxkd-55/a 2023-05-31T21:35:40.534 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/pklxkd-55/a?versionId=null 2023-05-31T21:35:40.534 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/pklxkd-55/a?versionId=null 2023-05-31T21:35:40.534 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-05-31T21:35:40.534 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /pklxkd-55/a?versionId=null 2023-05-31T21:35:40.535 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-05-31T21:35:40.535 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-05-31T21:35:40.535 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T21:35:40.535 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-05-31T21:35:40.536 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-05-31T21:35:40.536 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-05-31T21:35:40.536 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-05-31T21:35:40.536 INFO:tasks.rgw_multisite_tests:GET 2023-05-31T21:35:40.536 INFO:tasks.rgw_multisite_tests: 2023-05-31T21:35:40.537 INFO:tasks.rgw_multisite_tests: 2023-05-31T21:35:40.537 INFO:tasks.rgw_multisite_tests:Wed, 31 May 2023 20:29:24 GMT 2023-05-31T21:35:40.537 INFO:tasks.rgw_multisite_tests:/pklxkd-55/a?versionId=null 2023-05-31T21:35:40.537 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-05-31T21:35:40.538 INFO:tasks.rgw_multisite_tests:AWS Q80H11OLNU1S48PPYL26:+X92hov5hckyw41lnTNPdKmNFYQ= 2023-05-31T21:35:40.538 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:29:24 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:+X92hov5hckyw41lnTNPdKmNFYQ=', 'Content-Length': '0'} 2023-05-31T21:35:40.538 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:27:16 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e40ceace07e7059a-006477ae24-162e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:29:24 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:35:40.538 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/pklxkd-55/a 2023-05-31T21:35:40.539 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/pklxkd-55/a 2023-05-31T21:35:40.539 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/pklxkd-55/a?versionId=null 2023-05-31T21:35:40.539 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/pklxkd-55/a?versionId=null 2023-05-31T21:35:40.539 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-05-31T21:35:40.539 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /pklxkd-55/a?versionId=null 2023-05-31T21:35:40.540 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-05-31T21:35:40.540 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-05-31T21:35:40.540 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T21:35:40.540 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-05-31T21:35:40.541 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-05-31T21:35:40.541 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-05-31T21:35:40.541 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-05-31T21:35:40.541 INFO:tasks.rgw_multisite_tests:GET 2023-05-31T21:35:40.542 INFO:tasks.rgw_multisite_tests: 2023-05-31T21:35:40.542 INFO:tasks.rgw_multisite_tests: 2023-05-31T21:35:40.542 INFO:tasks.rgw_multisite_tests:Wed, 31 May 2023 20:29:24 GMT 2023-05-31T21:35:40.542 INFO:tasks.rgw_multisite_tests:/pklxkd-55/a?versionId=null 2023-05-31T21:35:40.542 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-05-31T21:35:40.543 INFO:tasks.rgw_multisite_tests:AWS Q80H11OLNU1S48PPYL26:+X92hov5hckyw41lnTNPdKmNFYQ= 2023-05-31T21:35:40.543 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:29:24 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:+X92hov5hckyw41lnTNPdKmNFYQ=', 'Content-Length': '0'} 2023-05-31T21:35:40.543 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:27:16 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009763aa4e42f7cd05-006477ae24-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:29:24 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:35:40.543 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/pklxkd-55/a 2023-05-31T21:35:40.544 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/pklxkd-55/a 2023-05-31T21:35:40.544 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/pklxkd-55/a?versionId=null 2023-05-31T21:35:40.544 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/pklxkd-55/a?versionId=null 2023-05-31T21:35:40.544 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-05-31T21:35:40.545 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /pklxkd-55/a?versionId=null 2023-05-31T21:35:40.545 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-05-31T21:35:40.545 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-05-31T21:35:40.545 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T21:35:40.545 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-05-31T21:35:40.546 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-05-31T21:35:40.546 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-05-31T21:35:40.546 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-05-31T21:35:40.546 INFO:tasks.rgw_multisite_tests:HEAD 2023-05-31T21:35:40.547 INFO:tasks.rgw_multisite_tests: 2023-05-31T21:35:40.547 INFO:tasks.rgw_multisite_tests: 2023-05-31T21:35:40.547 INFO:tasks.rgw_multisite_tests:Wed, 31 May 2023 20:29:24 GMT 2023-05-31T21:35:40.547 INFO:tasks.rgw_multisite_tests:/pklxkd-55/a?versionId=null 2023-05-31T21:35:40.547 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-05-31T21:35:40.548 INFO:tasks.rgw_multisite_tests:AWS Q80H11OLNU1S48PPYL26:IU6+aQzj/zu/ThVGrkWnaMaj3/Y= 2023-05-31T21:35:40.548 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:29:24 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:IU6+aQzj/zu/ThVGrkWnaMaj3/Y=', 'Content-Length': '0'} 2023-05-31T21:35:40.548 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:27:16 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000053a87f26e7c712ae-006477ae24-162e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:29:24 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:35:40.548 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/pklxkd-55/a 2023-05-31T21:35:40.549 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/pklxkd-55/a 2023-05-31T21:35:40.549 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/pklxkd-55/a?versionId=null 2023-05-31T21:35:40.549 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/pklxkd-55/a?versionId=null 2023-05-31T21:35:40.549 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-05-31T21:35:40.550 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /pklxkd-55/a?versionId=null 2023-05-31T21:35:40.550 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-05-31T21:35:40.550 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-05-31T21:35:40.550 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T21:35:40.550 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-05-31T21:35:40.551 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-05-31T21:35:40.551 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-05-31T21:35:40.551 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-05-31T21:35:40.551 INFO:tasks.rgw_multisite_tests:HEAD 2023-05-31T21:35:40.552 INFO:tasks.rgw_multisite_tests: 2023-05-31T21:35:40.552 INFO:tasks.rgw_multisite_tests: 2023-05-31T21:35:40.552 INFO:tasks.rgw_multisite_tests:Wed, 31 May 2023 20:29:24 GMT 2023-05-31T21:35:40.552 INFO:tasks.rgw_multisite_tests:/pklxkd-55/a?versionId=null 2023-05-31T21:35:40.553 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-05-31T21:35:40.553 INFO:tasks.rgw_multisite_tests:AWS Q80H11OLNU1S48PPYL26:IU6+aQzj/zu/ThVGrkWnaMaj3/Y= 2023-05-31T21:35:40.553 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:29:24 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:IU6+aQzj/zu/ThVGrkWnaMaj3/Y=', 'Content-Length': '0'} 2023-05-31T21:35:40.553 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:27:16 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000098ccbb35eefd27e2-006477ae24-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:29:24 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:35:40.553 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: comparing key name=a 2023-05-31T21:35:40.554 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/pklxkd-55/a 2023-05-31T21:35:40.554 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/pklxkd-55/a 2023-05-31T21:35:40.554 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/pklxkd-55/a?versionId=null 2023-05-31T21:35:40.554 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/pklxkd-55/a?versionId=null 2023-05-31T21:35:40.555 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-05-31T21:35:40.555 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /pklxkd-55/a?versionId=null 2023-05-31T21:35:40.555 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-05-31T21:35:40.555 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-05-31T21:35:40.556 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T21:35:40.556 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-05-31T21:35:40.556 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-05-31T21:35:40.556 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-05-31T21:35:40.556 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-05-31T21:35:40.557 INFO:tasks.rgw_multisite_tests:GET 2023-05-31T21:35:40.557 INFO:tasks.rgw_multisite_tests: 2023-05-31T21:35:40.557 INFO:tasks.rgw_multisite_tests: 2023-05-31T21:35:40.557 INFO:tasks.rgw_multisite_tests:Wed, 31 May 2023 20:29:24 GMT 2023-05-31T21:35:40.558 INFO:tasks.rgw_multisite_tests:/pklxkd-55/a?versionId=null 2023-05-31T21:35:40.558 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-05-31T21:35:40.558 INFO:tasks.rgw_multisite_tests:AWS Q80H11OLNU1S48PPYL26:+X92hov5hckyw41lnTNPdKmNFYQ= 2023-05-31T21:35:40.558 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:29:24 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:+X92hov5hckyw41lnTNPdKmNFYQ=', 'Content-Length': '0'} 2023-05-31T21:35:40.558 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:27:16 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001bd9c2affb92b517-006477ae24-162e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:29:24 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:35:40.559 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/pklxkd-55/a 2023-05-31T21:35:40.559 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/pklxkd-55/a 2023-05-31T21:35:40.559 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/pklxkd-55/a?versionId=null 2023-05-31T21:35:40.559 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/pklxkd-55/a?versionId=null 2023-05-31T21:35:40.560 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-05-31T21:35:40.560 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /pklxkd-55/a?versionId=null 2023-05-31T21:35:40.560 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-05-31T21:35:40.560 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-05-31T21:35:40.561 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T21:35:40.561 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-05-31T21:35:40.561 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-05-31T21:35:40.561 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-05-31T21:35:40.561 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-05-31T21:35:40.562 INFO:tasks.rgw_multisite_tests:GET 2023-05-31T21:35:40.562 INFO:tasks.rgw_multisite_tests: 2023-05-31T21:35:40.562 INFO:tasks.rgw_multisite_tests: 2023-05-31T21:35:40.562 INFO:tasks.rgw_multisite_tests:Wed, 31 May 2023 20:29:24 GMT 2023-05-31T21:35:40.563 INFO:tasks.rgw_multisite_tests:/pklxkd-55/a?versionId=null 2023-05-31T21:35:40.563 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-05-31T21:35:40.563 INFO:tasks.rgw_multisite_tests:AWS Q80H11OLNU1S48PPYL26:+X92hov5hckyw41lnTNPdKmNFYQ= 2023-05-31T21:35:40.563 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:29:24 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:+X92hov5hckyw41lnTNPdKmNFYQ=', 'Content-Length': '0'} 2023-05-31T21:35:40.564 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:27:16 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003f5b375ffdb4e608-006477ae24-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:29:24 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:35:40.564 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/pklxkd-55/ 2023-05-31T21:35:40.564 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/pklxkd-55/ 2023-05-31T21:35:40.564 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/pklxkd-55/?prefix=a 2023-05-31T21:35:40.564 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/pklxkd-55/?prefix=a 2023-05-31T21:35:40.565 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-05-31T21:35:40.565 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /pklxkd-55/?prefix=a 2023-05-31T21:35:40.565 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-05-31T21:35:40.565 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-05-31T21:35:40.566 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T21:35:40.566 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-05-31T21:35:40.566 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-05-31T21:35:40.566 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-05-31T21:35:40.567 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-05-31T21:35:40.567 INFO:tasks.rgw_multisite_tests:GET 2023-05-31T21:35:40.567 INFO:tasks.rgw_multisite_tests: 2023-05-31T21:35:40.567 INFO:tasks.rgw_multisite_tests: 2023-05-31T21:35:40.567 INFO:tasks.rgw_multisite_tests:Wed, 31 May 2023 20:29:24 GMT 2023-05-31T21:35:40.568 INFO:tasks.rgw_multisite_tests:/pklxkd-55/ 2023-05-31T21:35:40.568 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-05-31T21:35:40.568 INFO:tasks.rgw_multisite_tests:AWS Q80H11OLNU1S48PPYL26:xCRwQ8iFa3tX64tuk2Bv1zIjyBc= 2023-05-31T21:35:40.568 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:29:24 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:xCRwQ8iFa3tX64tuk2Bv1zIjyBc=', 'Content-Length': '0'} 2023-05-31T21:35:40.569 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000002679f89489768330-006477ae24-162e-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:29:24 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:35:40.569 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'pklxkd-55a1000falsea2023-05-31T20:27:16.792Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T21:35:40.569 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/pklxkd-55/ 2023-05-31T21:35:40.569 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/pklxkd-55/ 2023-05-31T21:35:40.570 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/pklxkd-55/?prefix=a 2023-05-31T21:35:40.570 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/pklxkd-55/?prefix=a 2023-05-31T21:35:40.570 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-05-31T21:35:40.570 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /pklxkd-55/?prefix=a 2023-05-31T21:35:40.570 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-05-31T21:35:40.571 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-05-31T21:35:40.571 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T21:35:40.571 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-05-31T21:35:40.571 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-05-31T21:35:40.572 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-05-31T21:35:40.572 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-05-31T21:35:40.572 INFO:tasks.rgw_multisite_tests:GET 2023-05-31T21:35:40.572 INFO:tasks.rgw_multisite_tests: 2023-05-31T21:35:40.573 INFO:tasks.rgw_multisite_tests: 2023-05-31T21:35:40.573 INFO:tasks.rgw_multisite_tests:Wed, 31 May 2023 20:29:24 GMT 2023-05-31T21:35:40.573 INFO:tasks.rgw_multisite_tests:/pklxkd-55/ 2023-05-31T21:35:40.573 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-05-31T21:35:40.573 INFO:tasks.rgw_multisite_tests:AWS Q80H11OLNU1S48PPYL26:xCRwQ8iFa3tX64tuk2Bv1zIjyBc= 2023-05-31T21:35:40.574 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:29:24 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:xCRwQ8iFa3tX64tuk2Bv1zIjyBc=', 'Content-Length': '0'} 2023-05-31T21:35:40.574 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d3ccdf260f6c5cb9-006477ae24-1630-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:29:24 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:35:40.574 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'pklxkd-55a1000falsea2023-05-31T20:27:16.792Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T21:35:40.574 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/pklxkd-55/a 2023-05-31T21:35:40.575 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/pklxkd-55/a 2023-05-31T21:35:40.575 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-05-31T21:35:40.575 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /pklxkd-55/a 2023-05-31T21:35:40.575 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-05-31T21:35:40.575 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-05-31T21:35:40.576 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T21:35:40.576 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-05-31T21:35:40.576 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-05-31T21:35:40.576 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-05-31T21:35:40.577 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-05-31T21:35:40.577 INFO:tasks.rgw_multisite_tests:HEAD 2023-05-31T21:35:40.577 INFO:tasks.rgw_multisite_tests: 2023-05-31T21:35:40.577 INFO:tasks.rgw_multisite_tests: 2023-05-31T21:35:40.578 INFO:tasks.rgw_multisite_tests:Wed, 31 May 2023 20:29:24 GMT 2023-05-31T21:35:40.578 INFO:tasks.rgw_multisite_tests:/pklxkd-55/a 2023-05-31T21:35:40.578 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-05-31T21:35:40.578 INFO:tasks.rgw_multisite_tests:AWS Q80H11OLNU1S48PPYL26:z2RC3sR/xfZmAcFYCxjPlVzZ6Q4= 2023-05-31T21:35:40.579 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:29:24 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:z2RC3sR/xfZmAcFYCxjPlVzZ6Q4=', 'Content-Length': '0'} 2023-05-31T21:35:40.579 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:27:16 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005d208bb192e5dd93-006477ae24-162e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:29:24 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:35:40.579 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/pklxkd-55/a 2023-05-31T21:35:40.579 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/pklxkd-55/a 2023-05-31T21:35:40.580 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-05-31T21:35:40.580 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /pklxkd-55/a 2023-05-31T21:35:40.580 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-05-31T21:35:40.580 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-05-31T21:35:40.581 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T21:35:40.581 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-05-31T21:35:40.581 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-05-31T21:35:40.581 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-05-31T21:35:40.581 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-05-31T21:35:40.582 INFO:tasks.rgw_multisite_tests:HEAD 2023-05-31T21:35:40.582 INFO:tasks.rgw_multisite_tests: 2023-05-31T21:35:40.582 INFO:tasks.rgw_multisite_tests: 2023-05-31T21:35:40.582 INFO:tasks.rgw_multisite_tests:Wed, 31 May 2023 20:29:24 GMT 2023-05-31T21:35:40.583 INFO:tasks.rgw_multisite_tests:/pklxkd-55/a 2023-05-31T21:35:40.583 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-05-31T21:35:40.583 INFO:tasks.rgw_multisite_tests:AWS Q80H11OLNU1S48PPYL26:z2RC3sR/xfZmAcFYCxjPlVzZ6Q4= 2023-05-31T21:35:40.583 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:29:24 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:z2RC3sR/xfZmAcFYCxjPlVzZ6Q4=', 'Content-Length': '0'} 2023-05-31T21:35:40.584 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:27:16 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000205edb67e5b206ea-006477ae24-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:29:24 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:35:40.584 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: comparing key name=a 2023-05-31T21:35:40.584 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/pklxkd-55/a 2023-05-31T21:35:40.584 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/pklxkd-55/a 2023-05-31T21:35:40.584 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-05-31T21:35:40.585 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /pklxkd-55/a 2023-05-31T21:35:40.585 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-05-31T21:35:40.585 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-05-31T21:35:40.585 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T21:35:40.586 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-05-31T21:35:40.586 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-05-31T21:35:40.586 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-05-31T21:35:40.586 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-05-31T21:35:40.587 INFO:tasks.rgw_multisite_tests:GET 2023-05-31T21:35:40.587 INFO:tasks.rgw_multisite_tests: 2023-05-31T21:35:40.587 INFO:tasks.rgw_multisite_tests: 2023-05-31T21:35:40.587 INFO:tasks.rgw_multisite_tests:Wed, 31 May 2023 20:29:24 GMT 2023-05-31T21:35:40.588 INFO:tasks.rgw_multisite_tests:/pklxkd-55/a 2023-05-31T21:35:40.588 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-05-31T21:35:40.588 INFO:tasks.rgw_multisite_tests:AWS Q80H11OLNU1S48PPYL26:vtnGMPFvk2E8cTyk3oTWDuCLguU= 2023-05-31T21:35:40.588 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:29:24 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:vtnGMPFvk2E8cTyk3oTWDuCLguU=', 'Content-Length': '0'} 2023-05-31T21:35:40.588 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:27:16 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005fc24de09c060edc-006477ae24-162e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:29:24 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:35:40.589 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/pklxkd-55/a 2023-05-31T21:35:40.589 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/pklxkd-55/a 2023-05-31T21:35:40.589 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-05-31T21:35:40.589 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /pklxkd-55/a 2023-05-31T21:35:40.590 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-05-31T21:35:40.590 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-05-31T21:35:40.590 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T21:35:40.590 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-05-31T21:35:40.591 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-05-31T21:35:40.591 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-05-31T21:35:40.591 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-05-31T21:35:40.591 INFO:tasks.rgw_multisite_tests:GET 2023-05-31T21:35:40.592 INFO:tasks.rgw_multisite_tests: 2023-05-31T21:35:40.592 INFO:tasks.rgw_multisite_tests: 2023-05-31T21:35:40.592 INFO:tasks.rgw_multisite_tests:Wed, 31 May 2023 20:29:24 GMT 2023-05-31T21:35:40.592 INFO:tasks.rgw_multisite_tests:/pklxkd-55/a 2023-05-31T21:35:40.592 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-05-31T21:35:40.593 INFO:tasks.rgw_multisite_tests:AWS Q80H11OLNU1S48PPYL26:vtnGMPFvk2E8cTyk3oTWDuCLguU= 2023-05-31T21:35:40.593 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:29:24 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:vtnGMPFvk2E8cTyk3oTWDuCLguU=', 'Content-Length': '0'} 2023-05-31T21:35:40.593 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:27:16 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ce6371bb8bf536e3-006477ae24-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:29:24 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:35:40.593 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/pklxkd-55/ 2023-05-31T21:35:40.594 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/pklxkd-55/ 2023-05-31T21:35:40.594 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/pklxkd-55/?prefix=a 2023-05-31T21:35:40.594 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/pklxkd-55/?prefix=a 2023-05-31T21:35:40.594 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-05-31T21:35:40.595 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /pklxkd-55/?prefix=a 2023-05-31T21:35:40.595 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-05-31T21:35:40.595 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-05-31T21:35:40.595 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T21:35:40.595 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-05-31T21:35:40.596 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-05-31T21:35:40.596 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-05-31T21:35:40.596 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-05-31T21:35:40.596 INFO:tasks.rgw_multisite_tests:GET 2023-05-31T21:35:40.597 INFO:tasks.rgw_multisite_tests: 2023-05-31T21:35:40.597 INFO:tasks.rgw_multisite_tests: 2023-05-31T21:35:40.597 INFO:tasks.rgw_multisite_tests:Wed, 31 May 2023 20:29:24 GMT 2023-05-31T21:35:40.597 INFO:tasks.rgw_multisite_tests:/pklxkd-55/ 2023-05-31T21:35:40.598 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-05-31T21:35:40.598 INFO:tasks.rgw_multisite_tests:AWS Q80H11OLNU1S48PPYL26:xCRwQ8iFa3tX64tuk2Bv1zIjyBc= 2023-05-31T21:35:40.598 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:29:24 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:xCRwQ8iFa3tX64tuk2Bv1zIjyBc=', 'Content-Length': '0'} 2023-05-31T21:35:40.598 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000eb2fead78109ec2b-006477ae24-162e-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:29:24 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:35:40.598 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'pklxkd-55a1000falsea2023-05-31T20:27:16.792Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T21:35:40.599 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/pklxkd-55/ 2023-05-31T21:35:40.599 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/pklxkd-55/ 2023-05-31T21:35:40.599 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/pklxkd-55/?prefix=a 2023-05-31T21:35:40.599 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/pklxkd-55/?prefix=a 2023-05-31T21:35:40.600 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-05-31T21:35:40.600 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /pklxkd-55/?prefix=a 2023-05-31T21:35:40.600 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-05-31T21:35:40.600 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-05-31T21:35:40.601 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T21:35:40.601 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-05-31T21:35:40.601 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-05-31T21:35:40.601 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-05-31T21:35:40.601 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-05-31T21:35:40.602 INFO:tasks.rgw_multisite_tests:GET 2023-05-31T21:35:40.602 INFO:tasks.rgw_multisite_tests: 2023-05-31T21:35:40.602 INFO:tasks.rgw_multisite_tests: 2023-05-31T21:35:40.602 INFO:tasks.rgw_multisite_tests:Wed, 31 May 2023 20:29:24 GMT 2023-05-31T21:35:40.603 INFO:tasks.rgw_multisite_tests:/pklxkd-55/ 2023-05-31T21:35:40.603 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-05-31T21:35:40.603 INFO:tasks.rgw_multisite_tests:AWS Q80H11OLNU1S48PPYL26:xCRwQ8iFa3tX64tuk2Bv1zIjyBc= 2023-05-31T21:35:40.603 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:29:24 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:xCRwQ8iFa3tX64tuk2Bv1zIjyBc=', 'Content-Length': '0'} 2023-05-31T21:35:40.603 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000004e9910af8bc9f872-006477ae24-1630-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:29:24 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:35:40.604 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'pklxkd-55a1000falsea2023-05-31T20:27:16.792Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T21:35:40.604 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: comparing key name=b 2023-05-31T21:35:40.604 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/pklxkd-55/b 2023-05-31T21:35:40.604 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/pklxkd-55/b 2023-05-31T21:35:40.605 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/pklxkd-55/b?versionId=null 2023-05-31T21:35:40.605 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/pklxkd-55/b?versionId=null 2023-05-31T21:35:40.605 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-05-31T21:35:40.605 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /pklxkd-55/b?versionId=null 2023-05-31T21:35:40.606 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-05-31T21:35:40.606 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-05-31T21:35:40.606 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T21:35:40.606 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-05-31T21:35:40.606 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-05-31T21:35:40.607 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-05-31T21:35:40.607 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-05-31T21:35:40.607 INFO:tasks.rgw_multisite_tests:GET 2023-05-31T21:35:40.607 INFO:tasks.rgw_multisite_tests: 2023-05-31T21:35:40.608 INFO:tasks.rgw_multisite_tests: 2023-05-31T21:35:40.608 INFO:tasks.rgw_multisite_tests:Wed, 31 May 2023 20:29:24 GMT 2023-05-31T21:35:40.608 INFO:tasks.rgw_multisite_tests:/pklxkd-55/b?versionId=null 2023-05-31T21:35:40.608 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-05-31T21:35:40.609 INFO:tasks.rgw_multisite_tests:AWS Q80H11OLNU1S48PPYL26:ju3OtKkKU11EzgC45iZF4622yUc= 2023-05-31T21:35:40.609 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:29:24 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:ju3OtKkKU11EzgC45iZF4622yUc=', 'Content-Length': '0'} 2023-05-31T21:35:40.609 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:27:16 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000700d36d0a6e044f2-006477ae24-162e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:29:24 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:35:40.609 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/pklxkd-55/b 2023-05-31T21:35:40.609 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/pklxkd-55/b 2023-05-31T21:35:40.610 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/pklxkd-55/b?versionId=null 2023-05-31T21:35:40.610 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/pklxkd-55/b?versionId=null 2023-05-31T21:35:40.610 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-05-31T21:35:40.610 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /pklxkd-55/b?versionId=null 2023-05-31T21:35:40.611 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-05-31T21:35:40.611 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-05-31T21:35:40.611 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T21:35:40.611 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-05-31T21:35:40.612 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-05-31T21:35:40.612 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-05-31T21:35:40.612 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-05-31T21:35:40.612 INFO:tasks.rgw_multisite_tests:GET 2023-05-31T21:35:40.613 INFO:tasks.rgw_multisite_tests: 2023-05-31T21:35:40.613 INFO:tasks.rgw_multisite_tests: 2023-05-31T21:35:40.613 INFO:tasks.rgw_multisite_tests:Wed, 31 May 2023 20:29:24 GMT 2023-05-31T21:35:40.613 INFO:tasks.rgw_multisite_tests:/pklxkd-55/b?versionId=null 2023-05-31T21:35:40.613 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-05-31T21:35:40.614 INFO:tasks.rgw_multisite_tests:AWS Q80H11OLNU1S48PPYL26:ju3OtKkKU11EzgC45iZF4622yUc= 2023-05-31T21:35:40.614 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:29:24 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:ju3OtKkKU11EzgC45iZF4622yUc=', 'Content-Length': '0'} 2023-05-31T21:35:40.614 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:27:16 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003192e70000ac7c25-006477ae24-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:29:24 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:35:40.614 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/pklxkd-55/b 2023-05-31T21:35:40.615 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/pklxkd-55/b 2023-05-31T21:35:40.615 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/pklxkd-55/b?versionId=null 2023-05-31T21:35:40.615 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/pklxkd-55/b?versionId=null 2023-05-31T21:35:40.615 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-05-31T21:35:40.615 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /pklxkd-55/b?versionId=null 2023-05-31T21:35:40.616 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-05-31T21:35:40.616 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-05-31T21:35:40.616 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T21:35:40.616 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-05-31T21:35:40.617 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-05-31T21:35:40.617 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-05-31T21:35:40.617 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-05-31T21:35:40.617 INFO:tasks.rgw_multisite_tests:HEAD 2023-05-31T21:35:40.618 INFO:tasks.rgw_multisite_tests: 2023-05-31T21:35:40.618 INFO:tasks.rgw_multisite_tests: 2023-05-31T21:35:40.618 INFO:tasks.rgw_multisite_tests:Wed, 31 May 2023 20:29:24 GMT 2023-05-31T21:35:40.618 INFO:tasks.rgw_multisite_tests:/pklxkd-55/b?versionId=null 2023-05-31T21:35:40.618 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-05-31T21:35:40.619 INFO:tasks.rgw_multisite_tests:AWS Q80H11OLNU1S48PPYL26:wjpFMkgIyOzOucNXWIg1exaYuVQ= 2023-05-31T21:35:40.619 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:29:24 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:wjpFMkgIyOzOucNXWIg1exaYuVQ=', 'Content-Length': '0'} 2023-05-31T21:35:40.619 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:27:16 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d44028c8af6e0f4e-006477ae24-162e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:29:24 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:35:40.619 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/pklxkd-55/b 2023-05-31T21:35:40.620 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/pklxkd-55/b 2023-05-31T21:35:40.620 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/pklxkd-55/b?versionId=null 2023-05-31T21:35:40.620 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/pklxkd-55/b?versionId=null 2023-05-31T21:35:40.620 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-05-31T21:35:40.621 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /pklxkd-55/b?versionId=null 2023-05-31T21:35:40.621 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-05-31T21:35:40.621 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-05-31T21:35:40.621 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T21:35:40.621 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-05-31T21:35:40.622 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-05-31T21:35:40.622 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-05-31T21:35:40.622 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-05-31T21:35:40.623 INFO:tasks.rgw_multisite_tests:HEAD 2023-05-31T21:35:40.623 INFO:tasks.rgw_multisite_tests: 2023-05-31T21:35:40.623 INFO:tasks.rgw_multisite_tests: 2023-05-31T21:35:40.623 INFO:tasks.rgw_multisite_tests:Wed, 31 May 2023 20:29:24 GMT 2023-05-31T21:35:40.624 INFO:tasks.rgw_multisite_tests:/pklxkd-55/b?versionId=null 2023-05-31T21:35:40.624 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-05-31T21:35:40.624 INFO:tasks.rgw_multisite_tests:AWS Q80H11OLNU1S48PPYL26:wjpFMkgIyOzOucNXWIg1exaYuVQ= 2023-05-31T21:35:40.624 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:29:24 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:wjpFMkgIyOzOucNXWIg1exaYuVQ=', 'Content-Length': '0'} 2023-05-31T21:35:40.624 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:27:16 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007787216d332c7500-006477ae24-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:29:24 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:35:40.625 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: comparing key name=b 2023-05-31T21:35:40.625 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/pklxkd-55/b 2023-05-31T21:35:40.625 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/pklxkd-55/b 2023-05-31T21:35:40.625 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/pklxkd-55/b?versionId=null 2023-05-31T21:35:40.626 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/pklxkd-55/b?versionId=null 2023-05-31T21:35:40.626 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-05-31T21:35:40.626 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /pklxkd-55/b?versionId=null 2023-05-31T21:35:40.626 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-05-31T21:35:40.627 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-05-31T21:35:40.627 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T21:35:40.627 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-05-31T21:35:40.627 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-05-31T21:35:40.628 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-05-31T21:35:40.628 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-05-31T21:35:40.628 INFO:tasks.rgw_multisite_tests:GET 2023-05-31T21:35:40.628 INFO:tasks.rgw_multisite_tests: 2023-05-31T21:35:40.628 INFO:tasks.rgw_multisite_tests: 2023-05-31T21:35:40.629 INFO:tasks.rgw_multisite_tests:Wed, 31 May 2023 20:29:24 GMT 2023-05-31T21:35:40.629 INFO:tasks.rgw_multisite_tests:/pklxkd-55/b?versionId=null 2023-05-31T21:35:40.629 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-05-31T21:35:40.629 INFO:tasks.rgw_multisite_tests:AWS Q80H11OLNU1S48PPYL26:ju3OtKkKU11EzgC45iZF4622yUc= 2023-05-31T21:35:40.630 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:29:24 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:ju3OtKkKU11EzgC45iZF4622yUc=', 'Content-Length': '0'} 2023-05-31T21:35:40.630 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:27:16 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000293b444a3f1448fa-006477ae24-162e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:29:24 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:35:40.630 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/pklxkd-55/b 2023-05-31T21:35:40.630 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/pklxkd-55/b 2023-05-31T21:35:40.631 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/pklxkd-55/b?versionId=null 2023-05-31T21:35:40.631 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/pklxkd-55/b?versionId=null 2023-05-31T21:35:40.631 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-05-31T21:35:40.631 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /pklxkd-55/b?versionId=null 2023-05-31T21:35:40.631 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-05-31T21:35:40.632 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-05-31T21:35:40.632 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T21:35:40.632 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-05-31T21:35:40.632 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-05-31T21:35:40.633 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-05-31T21:35:40.633 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-05-31T21:35:40.633 INFO:tasks.rgw_multisite_tests:GET 2023-05-31T21:35:40.633 INFO:tasks.rgw_multisite_tests: 2023-05-31T21:35:40.634 INFO:tasks.rgw_multisite_tests: 2023-05-31T21:35:40.634 INFO:tasks.rgw_multisite_tests:Wed, 31 May 2023 20:29:24 GMT 2023-05-31T21:35:40.634 INFO:tasks.rgw_multisite_tests:/pklxkd-55/b?versionId=null 2023-05-31T21:35:40.634 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-05-31T21:35:40.635 INFO:tasks.rgw_multisite_tests:AWS Q80H11OLNU1S48PPYL26:ju3OtKkKU11EzgC45iZF4622yUc= 2023-05-31T21:35:40.635 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:29:24 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:ju3OtKkKU11EzgC45iZF4622yUc=', 'Content-Length': '0'} 2023-05-31T21:35:40.635 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:27:16 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001a139aa291cc36de-006477ae24-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:29:24 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:35:40.635 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/pklxkd-55/ 2023-05-31T21:35:40.635 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/pklxkd-55/ 2023-05-31T21:35:40.636 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/pklxkd-55/?prefix=b 2023-05-31T21:35:40.636 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/pklxkd-55/?prefix=b 2023-05-31T21:35:40.636 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-05-31T21:35:40.636 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /pklxkd-55/?prefix=b 2023-05-31T21:35:40.637 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-05-31T21:35:40.637 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-05-31T21:35:40.637 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T21:35:40.637 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-05-31T21:35:40.638 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-05-31T21:35:40.638 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-05-31T21:35:40.638 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-05-31T21:35:40.638 INFO:tasks.rgw_multisite_tests:GET 2023-05-31T21:35:40.638 INFO:tasks.rgw_multisite_tests: 2023-05-31T21:35:40.639 INFO:tasks.rgw_multisite_tests: 2023-05-31T21:35:40.639 INFO:tasks.rgw_multisite_tests:Wed, 31 May 2023 20:29:24 GMT 2023-05-31T21:35:40.639 INFO:tasks.rgw_multisite_tests:/pklxkd-55/ 2023-05-31T21:35:40.639 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-05-31T21:35:40.640 INFO:tasks.rgw_multisite_tests:AWS Q80H11OLNU1S48PPYL26:xCRwQ8iFa3tX64tuk2Bv1zIjyBc= 2023-05-31T21:35:40.640 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:29:24 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:xCRwQ8iFa3tX64tuk2Bv1zIjyBc=', 'Content-Length': '0'} 2023-05-31T21:35:40.640 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000047697f32a096ebf1-006477ae24-162e-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:29:24 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:35:40.640 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'pklxkd-55b1000falseb2023-05-31T20:27:16.850Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T21:35:40.641 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/pklxkd-55/ 2023-05-31T21:35:40.641 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/pklxkd-55/ 2023-05-31T21:35:40.641 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/pklxkd-55/?prefix=b 2023-05-31T21:35:40.641 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/pklxkd-55/?prefix=b 2023-05-31T21:35:40.641 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-05-31T21:35:40.642 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /pklxkd-55/?prefix=b 2023-05-31T21:35:40.642 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-05-31T21:35:40.642 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-05-31T21:35:40.642 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T21:35:40.643 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-05-31T21:35:40.643 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-05-31T21:35:40.643 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-05-31T21:35:40.643 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-05-31T21:35:40.644 INFO:tasks.rgw_multisite_tests:GET 2023-05-31T21:35:40.644 INFO:tasks.rgw_multisite_tests: 2023-05-31T21:35:40.644 INFO:tasks.rgw_multisite_tests: 2023-05-31T21:35:40.644 INFO:tasks.rgw_multisite_tests:Wed, 31 May 2023 20:29:24 GMT 2023-05-31T21:35:40.644 INFO:tasks.rgw_multisite_tests:/pklxkd-55/ 2023-05-31T21:35:40.645 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-05-31T21:35:40.645 INFO:tasks.rgw_multisite_tests:AWS Q80H11OLNU1S48PPYL26:xCRwQ8iFa3tX64tuk2Bv1zIjyBc= 2023-05-31T21:35:40.645 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:29:24 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:xCRwQ8iFa3tX64tuk2Bv1zIjyBc=', 'Content-Length': '0'} 2023-05-31T21:35:40.645 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b36c382bbe0985f3-006477ae24-1630-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:29:24 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:35:40.646 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'pklxkd-55b1000falseb2023-05-31T20:27:16.850Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T21:35:40.646 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/pklxkd-55/b 2023-05-31T21:35:40.646 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/pklxkd-55/b 2023-05-31T21:35:40.646 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-05-31T21:35:40.647 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /pklxkd-55/b 2023-05-31T21:35:40.647 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-05-31T21:35:40.647 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-05-31T21:35:40.647 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T21:35:40.648 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-05-31T21:35:40.648 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-05-31T21:35:40.648 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-05-31T21:35:40.648 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-05-31T21:35:40.648 INFO:tasks.rgw_multisite_tests:HEAD 2023-05-31T21:35:40.649 INFO:tasks.rgw_multisite_tests: 2023-05-31T21:35:40.649 INFO:tasks.rgw_multisite_tests: 2023-05-31T21:35:40.649 INFO:tasks.rgw_multisite_tests:Wed, 31 May 2023 20:29:24 GMT 2023-05-31T21:35:40.649 INFO:tasks.rgw_multisite_tests:/pklxkd-55/b 2023-05-31T21:35:40.650 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-05-31T21:35:40.650 INFO:tasks.rgw_multisite_tests:AWS Q80H11OLNU1S48PPYL26:1aG6/dzrbbfZto0Qb8afOlzk78c= 2023-05-31T21:35:40.650 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:29:24 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:1aG6/dzrbbfZto0Qb8afOlzk78c=', 'Content-Length': '0'} 2023-05-31T21:35:40.650 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:27:16 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d3f6d06111fd48f2-006477ae24-162e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:29:24 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:35:40.651 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/pklxkd-55/b 2023-05-31T21:35:40.651 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/pklxkd-55/b 2023-05-31T21:35:40.651 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-05-31T21:35:40.651 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /pklxkd-55/b 2023-05-31T21:35:40.652 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-05-31T21:35:40.652 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-05-31T21:35:40.652 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T21:35:40.652 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-05-31T21:35:40.652 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-05-31T21:35:40.653 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-05-31T21:35:40.653 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-05-31T21:35:40.653 INFO:tasks.rgw_multisite_tests:HEAD 2023-05-31T21:35:40.653 INFO:tasks.rgw_multisite_tests: 2023-05-31T21:35:40.654 INFO:tasks.rgw_multisite_tests: 2023-05-31T21:35:40.654 INFO:tasks.rgw_multisite_tests:Wed, 31 May 2023 20:29:24 GMT 2023-05-31T21:35:40.654 INFO:tasks.rgw_multisite_tests:/pklxkd-55/b 2023-05-31T21:35:40.654 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-05-31T21:35:40.655 INFO:tasks.rgw_multisite_tests:AWS Q80H11OLNU1S48PPYL26:1aG6/dzrbbfZto0Qb8afOlzk78c= 2023-05-31T21:35:40.655 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:29:24 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:1aG6/dzrbbfZto0Qb8afOlzk78c=', 'Content-Length': '0'} 2023-05-31T21:35:40.655 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:27:16 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006094b18781d552fa-006477ae24-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:29:24 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:35:40.655 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: comparing key name=b 2023-05-31T21:35:40.655 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/pklxkd-55/b 2023-05-31T21:35:40.656 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/pklxkd-55/b 2023-05-31T21:35:40.656 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-05-31T21:35:40.656 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /pklxkd-55/b 2023-05-31T21:35:40.656 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-05-31T21:35:40.657 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-05-31T21:35:40.657 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T21:35:40.657 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-05-31T21:35:40.657 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-05-31T21:35:40.658 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-05-31T21:35:40.658 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-05-31T21:35:40.658 INFO:tasks.rgw_multisite_tests:GET 2023-05-31T21:35:40.658 INFO:tasks.rgw_multisite_tests: 2023-05-31T21:35:40.658 INFO:tasks.rgw_multisite_tests: 2023-05-31T21:35:40.659 INFO:tasks.rgw_multisite_tests:Wed, 31 May 2023 20:29:24 GMT 2023-05-31T21:35:40.659 INFO:tasks.rgw_multisite_tests:/pklxkd-55/b 2023-05-31T21:35:40.659 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-05-31T21:35:40.659 INFO:tasks.rgw_multisite_tests:AWS Q80H11OLNU1S48PPYL26:nRgLIEteHGm2wkSOz/c6odfckCQ= 2023-05-31T21:35:40.660 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:29:24 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:nRgLIEteHGm2wkSOz/c6odfckCQ=', 'Content-Length': '0'} 2023-05-31T21:35:40.660 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:27:16 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000cfcd4cb81f9dbad3-006477ae24-162e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:29:24 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:35:40.660 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/pklxkd-55/b 2023-05-31T21:35:40.660 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/pklxkd-55/b 2023-05-31T21:35:40.661 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-05-31T21:35:40.661 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /pklxkd-55/b 2023-05-31T21:35:40.661 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-05-31T21:35:40.661 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-05-31T21:35:40.661 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T21:35:40.662 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-05-31T21:35:40.662 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-05-31T21:35:40.662 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-05-31T21:35:40.662 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-05-31T21:35:40.663 INFO:tasks.rgw_multisite_tests:GET 2023-05-31T21:35:40.663 INFO:tasks.rgw_multisite_tests: 2023-05-31T21:35:40.663 INFO:tasks.rgw_multisite_tests: 2023-05-31T21:35:40.663 INFO:tasks.rgw_multisite_tests:Wed, 31 May 2023 20:29:24 GMT 2023-05-31T21:35:40.664 INFO:tasks.rgw_multisite_tests:/pklxkd-55/b 2023-05-31T21:35:40.664 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-05-31T21:35:40.664 INFO:tasks.rgw_multisite_tests:AWS Q80H11OLNU1S48PPYL26:nRgLIEteHGm2wkSOz/c6odfckCQ= 2023-05-31T21:35:40.664 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:29:24 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:nRgLIEteHGm2wkSOz/c6odfckCQ=', 'Content-Length': '0'} 2023-05-31T21:35:40.664 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:27:16 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000081dba989f142aefc-006477ae24-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:29:24 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:35:40.665 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/pklxkd-55/ 2023-05-31T21:35:40.665 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/pklxkd-55/ 2023-05-31T21:35:40.665 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/pklxkd-55/?prefix=b 2023-05-31T21:35:40.665 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/pklxkd-55/?prefix=b 2023-05-31T21:35:40.666 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-05-31T21:35:40.666 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /pklxkd-55/?prefix=b 2023-05-31T21:35:40.666 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-05-31T21:35:40.666 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-05-31T21:35:40.667 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T21:35:40.667 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-05-31T21:35:40.667 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-05-31T21:35:40.667 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-05-31T21:35:40.667 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-05-31T21:35:40.668 INFO:tasks.rgw_multisite_tests:GET 2023-05-31T21:35:40.668 INFO:tasks.rgw_multisite_tests: 2023-05-31T21:35:40.668 INFO:tasks.rgw_multisite_tests: 2023-05-31T21:35:40.668 INFO:tasks.rgw_multisite_tests:Wed, 31 May 2023 20:29:24 GMT 2023-05-31T21:35:40.669 INFO:tasks.rgw_multisite_tests:/pklxkd-55/ 2023-05-31T21:35:40.669 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-05-31T21:35:40.669 INFO:tasks.rgw_multisite_tests:AWS Q80H11OLNU1S48PPYL26:xCRwQ8iFa3tX64tuk2Bv1zIjyBc= 2023-05-31T21:35:40.669 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:29:24 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:xCRwQ8iFa3tX64tuk2Bv1zIjyBc=', 'Content-Length': '0'} 2023-05-31T21:35:40.670 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000054dfa196f856584d-006477ae24-162e-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:29:24 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:35:40.670 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'pklxkd-55b1000falseb2023-05-31T20:27:16.850Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T21:35:40.670 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/pklxkd-55/ 2023-05-31T21:35:40.670 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/pklxkd-55/ 2023-05-31T21:35:40.670 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/pklxkd-55/?prefix=b 2023-05-31T21:35:40.671 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/pklxkd-55/?prefix=b 2023-05-31T21:35:40.671 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-05-31T21:35:40.671 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /pklxkd-55/?prefix=b 2023-05-31T21:35:40.671 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-05-31T21:35:40.672 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-05-31T21:35:40.672 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T21:35:40.672 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-05-31T21:35:40.672 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-05-31T21:35:40.673 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-05-31T21:35:40.673 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-05-31T21:35:40.673 INFO:tasks.rgw_multisite_tests:GET 2023-05-31T21:35:40.673 INFO:tasks.rgw_multisite_tests: 2023-05-31T21:35:40.673 INFO:tasks.rgw_multisite_tests: 2023-05-31T21:35:40.674 INFO:tasks.rgw_multisite_tests:Wed, 31 May 2023 20:29:24 GMT 2023-05-31T21:35:40.674 INFO:tasks.rgw_multisite_tests:/pklxkd-55/ 2023-05-31T21:35:40.674 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-05-31T21:35:40.674 INFO:tasks.rgw_multisite_tests:AWS Q80H11OLNU1S48PPYL26:xCRwQ8iFa3tX64tuk2Bv1zIjyBc= 2023-05-31T21:35:40.675 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:29:24 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:xCRwQ8iFa3tX64tuk2Bv1zIjyBc=', 'Content-Length': '0'} 2023-05-31T21:35:40.675 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000013dad2d637250b85-006477ae24-1630-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:29:24 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:35:40.675 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'pklxkd-55b1000falseb2023-05-31T20:27:16.850Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T21:35:40.675 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: comparing key name=c 2023-05-31T21:35:40.675 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/pklxkd-55/c 2023-05-31T21:35:40.676 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/pklxkd-55/c 2023-05-31T21:35:40.676 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/pklxkd-55/c?versionId=null 2023-05-31T21:35:40.676 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/pklxkd-55/c?versionId=null 2023-05-31T21:35:40.676 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-05-31T21:35:40.677 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /pklxkd-55/c?versionId=null 2023-05-31T21:35:40.677 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-05-31T21:35:40.677 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-05-31T21:35:40.677 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T21:35:40.678 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-05-31T21:35:40.678 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-05-31T21:35:40.678 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-05-31T21:35:40.678 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-05-31T21:35:40.679 INFO:tasks.rgw_multisite_tests:GET 2023-05-31T21:35:40.679 INFO:tasks.rgw_multisite_tests: 2023-05-31T21:35:40.679 INFO:tasks.rgw_multisite_tests: 2023-05-31T21:35:40.679 INFO:tasks.rgw_multisite_tests:Wed, 31 May 2023 20:29:24 GMT 2023-05-31T21:35:40.680 INFO:tasks.rgw_multisite_tests:/pklxkd-55/c?versionId=null 2023-05-31T21:35:40.680 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-05-31T21:35:40.680 INFO:tasks.rgw_multisite_tests:AWS Q80H11OLNU1S48PPYL26:dufJB3ZebW9SblWR6TqGAFspARc= 2023-05-31T21:35:40.680 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:29:24 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:dufJB3ZebW9SblWR6TqGAFspARc=', 'Content-Length': '0'} 2023-05-31T21:35:40.680 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:27:16 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000016770863e7b3f199-006477ae24-162e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:29:24 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:35:40.681 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/pklxkd-55/c 2023-05-31T21:35:40.681 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/pklxkd-55/c 2023-05-31T21:35:40.681 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/pklxkd-55/c?versionId=null 2023-05-31T21:35:40.681 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/pklxkd-55/c?versionId=null 2023-05-31T21:35:40.682 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-05-31T21:35:40.682 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /pklxkd-55/c?versionId=null 2023-05-31T21:35:40.682 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-05-31T21:35:40.682 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-05-31T21:35:40.683 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T21:35:40.683 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-05-31T21:35:40.683 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-05-31T21:35:40.683 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-05-31T21:35:40.683 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-05-31T21:35:40.684 INFO:tasks.rgw_multisite_tests:GET 2023-05-31T21:35:40.684 INFO:tasks.rgw_multisite_tests: 2023-05-31T21:35:40.684 INFO:tasks.rgw_multisite_tests: 2023-05-31T21:35:40.684 INFO:tasks.rgw_multisite_tests:Wed, 31 May 2023 20:29:24 GMT 2023-05-31T21:35:40.685 INFO:tasks.rgw_multisite_tests:/pklxkd-55/c?versionId=null 2023-05-31T21:35:40.685 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-05-31T21:35:40.685 INFO:tasks.rgw_multisite_tests:AWS Q80H11OLNU1S48PPYL26:dufJB3ZebW9SblWR6TqGAFspARc= 2023-05-31T21:35:40.685 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:29:24 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:dufJB3ZebW9SblWR6TqGAFspARc=', 'Content-Length': '0'} 2023-05-31T21:35:40.686 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:27:16 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005a7d3062b7f40908-006477ae24-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:29:24 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:35:40.686 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/pklxkd-55/c 2023-05-31T21:35:40.686 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/pklxkd-55/c 2023-05-31T21:35:40.686 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/pklxkd-55/c?versionId=null 2023-05-31T21:35:40.687 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/pklxkd-55/c?versionId=null 2023-05-31T21:35:40.687 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-05-31T21:35:40.687 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /pklxkd-55/c?versionId=null 2023-05-31T21:35:40.687 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-05-31T21:35:40.687 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-05-31T21:35:40.688 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T21:35:40.688 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-05-31T21:35:40.688 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-05-31T21:35:40.688 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-05-31T21:35:40.689 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-05-31T21:35:40.689 INFO:tasks.rgw_multisite_tests:HEAD 2023-05-31T21:35:40.689 INFO:tasks.rgw_multisite_tests: 2023-05-31T21:35:40.689 INFO:tasks.rgw_multisite_tests: 2023-05-31T21:35:40.690 INFO:tasks.rgw_multisite_tests:Wed, 31 May 2023 20:29:24 GMT 2023-05-31T21:35:40.690 INFO:tasks.rgw_multisite_tests:/pklxkd-55/c?versionId=null 2023-05-31T21:35:40.690 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-05-31T21:35:40.690 INFO:tasks.rgw_multisite_tests:AWS Q80H11OLNU1S48PPYL26:JDvQi560BJAmCLq/cVNOrKkPhDc= 2023-05-31T21:35:40.691 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:29:24 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:JDvQi560BJAmCLq/cVNOrKkPhDc=', 'Content-Length': '0'} 2023-05-31T21:35:40.691 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:27:16 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f4585029d29076e9-006477ae24-162e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:29:24 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:35:40.691 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/pklxkd-55/c 2023-05-31T21:35:40.691 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/pklxkd-55/c 2023-05-31T21:35:40.691 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/pklxkd-55/c?versionId=null 2023-05-31T21:35:40.692 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/pklxkd-55/c?versionId=null 2023-05-31T21:35:40.692 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-05-31T21:35:40.692 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /pklxkd-55/c?versionId=null 2023-05-31T21:35:40.692 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-05-31T21:35:40.693 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-05-31T21:35:40.693 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T21:35:40.693 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-05-31T21:35:40.693 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-05-31T21:35:40.694 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-05-31T21:35:40.694 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-05-31T21:35:40.694 INFO:tasks.rgw_multisite_tests:HEAD 2023-05-31T21:35:40.694 INFO:tasks.rgw_multisite_tests: 2023-05-31T21:35:40.694 INFO:tasks.rgw_multisite_tests: 2023-05-31T21:35:40.695 INFO:tasks.rgw_multisite_tests:Wed, 31 May 2023 20:29:24 GMT 2023-05-31T21:35:40.695 INFO:tasks.rgw_multisite_tests:/pklxkd-55/c?versionId=null 2023-05-31T21:35:40.695 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-05-31T21:35:40.695 INFO:tasks.rgw_multisite_tests:AWS Q80H11OLNU1S48PPYL26:JDvQi560BJAmCLq/cVNOrKkPhDc= 2023-05-31T21:35:40.696 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:29:24 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:JDvQi560BJAmCLq/cVNOrKkPhDc=', 'Content-Length': '0'} 2023-05-31T21:35:40.696 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:27:16 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009df90906199d5cf6-006477ae24-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:29:24 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:35:40.696 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: comparing key name=c 2023-05-31T21:35:40.696 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/pklxkd-55/c 2023-05-31T21:35:40.697 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/pklxkd-55/c 2023-05-31T21:35:40.697 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/pklxkd-55/c?versionId=null 2023-05-31T21:35:40.697 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/pklxkd-55/c?versionId=null 2023-05-31T21:35:40.697 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-05-31T21:35:40.697 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /pklxkd-55/c?versionId=null 2023-05-31T21:35:40.698 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-05-31T21:35:40.698 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-05-31T21:35:40.698 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T21:35:40.698 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-05-31T21:35:40.699 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-05-31T21:35:40.699 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-05-31T21:35:40.699 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-05-31T21:35:40.699 INFO:tasks.rgw_multisite_tests:GET 2023-05-31T21:35:40.700 INFO:tasks.rgw_multisite_tests: 2023-05-31T21:35:40.700 INFO:tasks.rgw_multisite_tests: 2023-05-31T21:35:40.700 INFO:tasks.rgw_multisite_tests:Wed, 31 May 2023 20:29:24 GMT 2023-05-31T21:35:40.700 INFO:tasks.rgw_multisite_tests:/pklxkd-55/c?versionId=null 2023-05-31T21:35:40.700 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-05-31T21:35:40.701 INFO:tasks.rgw_multisite_tests:AWS Q80H11OLNU1S48PPYL26:dufJB3ZebW9SblWR6TqGAFspARc= 2023-05-31T21:35:40.701 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:29:24 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:dufJB3ZebW9SblWR6TqGAFspARc=', 'Content-Length': '0'} 2023-05-31T21:35:40.701 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:27:16 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006df2c29781c17f50-006477ae24-162e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:29:24 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:35:40.701 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/pklxkd-55/c 2023-05-31T21:35:40.702 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/pklxkd-55/c 2023-05-31T21:35:40.702 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/pklxkd-55/c?versionId=null 2023-05-31T21:35:40.702 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/pklxkd-55/c?versionId=null 2023-05-31T21:35:40.702 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-05-31T21:35:40.703 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /pklxkd-55/c?versionId=null 2023-05-31T21:35:40.703 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-05-31T21:35:40.703 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-05-31T21:35:40.703 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T21:35:40.703 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-05-31T21:35:40.704 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-05-31T21:35:40.704 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-05-31T21:35:40.704 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-05-31T21:35:40.704 INFO:tasks.rgw_multisite_tests:GET 2023-05-31T21:35:40.705 INFO:tasks.rgw_multisite_tests: 2023-05-31T21:35:40.705 INFO:tasks.rgw_multisite_tests: 2023-05-31T21:35:40.705 INFO:tasks.rgw_multisite_tests:Wed, 31 May 2023 20:29:24 GMT 2023-05-31T21:35:40.705 INFO:tasks.rgw_multisite_tests:/pklxkd-55/c?versionId=null 2023-05-31T21:35:40.706 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-05-31T21:35:40.706 INFO:tasks.rgw_multisite_tests:AWS Q80H11OLNU1S48PPYL26:dufJB3ZebW9SblWR6TqGAFspARc= 2023-05-31T21:35:40.706 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:29:24 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:dufJB3ZebW9SblWR6TqGAFspARc=', 'Content-Length': '0'} 2023-05-31T21:35:40.706 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:27:16 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007cd28dea537dcd90-006477ae24-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:29:24 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:35:40.707 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/pklxkd-55/ 2023-05-31T21:35:40.707 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/pklxkd-55/ 2023-05-31T21:35:40.707 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/pklxkd-55/?prefix=c 2023-05-31T21:35:40.707 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/pklxkd-55/?prefix=c 2023-05-31T21:35:40.707 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-05-31T21:35:40.708 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /pklxkd-55/?prefix=c 2023-05-31T21:35:40.708 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-05-31T21:35:40.708 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-05-31T21:35:40.708 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T21:35:40.709 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-05-31T21:35:40.709 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-05-31T21:35:40.709 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-05-31T21:35:40.709 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-05-31T21:35:40.710 INFO:tasks.rgw_multisite_tests:GET 2023-05-31T21:35:40.710 INFO:tasks.rgw_multisite_tests: 2023-05-31T21:35:40.710 INFO:tasks.rgw_multisite_tests: 2023-05-31T21:35:40.710 INFO:tasks.rgw_multisite_tests:Wed, 31 May 2023 20:29:25 GMT 2023-05-31T21:35:40.710 INFO:tasks.rgw_multisite_tests:/pklxkd-55/ 2023-05-31T21:35:40.711 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-05-31T21:35:40.711 INFO:tasks.rgw_multisite_tests:AWS Q80H11OLNU1S48PPYL26:a0/qPp0bnZeH62hZJf6OA1a38dw= 2023-05-31T21:35:40.711 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:29:25 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:a0/qPp0bnZeH62hZJf6OA1a38dw=', 'Content-Length': '0'} 2023-05-31T21:35:40.711 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a952419d095d0940-006477ae25-162e-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:29:25 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:35:40.712 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'pklxkd-55c1000falsec2023-05-31T20:27:16.869Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T21:35:40.712 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/pklxkd-55/ 2023-05-31T21:35:40.712 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/pklxkd-55/ 2023-05-31T21:35:40.712 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/pklxkd-55/?prefix=c 2023-05-31T21:35:40.713 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/pklxkd-55/?prefix=c 2023-05-31T21:35:40.713 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-05-31T21:35:40.713 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /pklxkd-55/?prefix=c 2023-05-31T21:35:40.713 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-05-31T21:35:40.713 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-05-31T21:35:40.714 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T21:35:40.714 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-05-31T21:35:40.714 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-05-31T21:35:40.714 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-05-31T21:35:40.715 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-05-31T21:35:40.715 INFO:tasks.rgw_multisite_tests:GET 2023-05-31T21:35:40.715 INFO:tasks.rgw_multisite_tests: 2023-05-31T21:35:40.715 INFO:tasks.rgw_multisite_tests: 2023-05-31T21:35:40.716 INFO:tasks.rgw_multisite_tests:Wed, 31 May 2023 20:29:25 GMT 2023-05-31T21:35:40.716 INFO:tasks.rgw_multisite_tests:/pklxkd-55/ 2023-05-31T21:35:40.716 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-05-31T21:35:40.716 INFO:tasks.rgw_multisite_tests:AWS Q80H11OLNU1S48PPYL26:a0/qPp0bnZeH62hZJf6OA1a38dw= 2023-05-31T21:35:40.717 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:29:25 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:a0/qPp0bnZeH62hZJf6OA1a38dw=', 'Content-Length': '0'} 2023-05-31T21:35:40.717 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a62928d117cb770a-006477ae25-1630-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:29:25 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:35:40.717 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'pklxkd-55c1000falsec2023-05-31T20:27:16.869Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T21:35:40.717 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/pklxkd-55/c 2023-05-31T21:35:40.718 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/pklxkd-55/c 2023-05-31T21:35:40.718 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-05-31T21:35:40.718 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /pklxkd-55/c 2023-05-31T21:35:40.718 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-05-31T21:35:40.718 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-05-31T21:35:40.719 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T21:35:40.719 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-05-31T21:35:40.719 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-05-31T21:35:40.719 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-05-31T21:35:40.720 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-05-31T21:35:40.720 INFO:tasks.rgw_multisite_tests:HEAD 2023-05-31T21:35:40.720 INFO:tasks.rgw_multisite_tests: 2023-05-31T21:35:40.720 INFO:tasks.rgw_multisite_tests: 2023-05-31T21:35:40.721 INFO:tasks.rgw_multisite_tests:Wed, 31 May 2023 20:29:25 GMT 2023-05-31T21:35:40.721 INFO:tasks.rgw_multisite_tests:/pklxkd-55/c 2023-05-31T21:35:40.721 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-05-31T21:35:40.721 INFO:tasks.rgw_multisite_tests:AWS Q80H11OLNU1S48PPYL26:qyeZ0BO5kT8gimi9XwzG/GXY2sA= 2023-05-31T21:35:40.721 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:29:25 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:qyeZ0BO5kT8gimi9XwzG/GXY2sA=', 'Content-Length': '0'} 2023-05-31T21:35:40.722 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:27:16 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c2d7afcb31510a06-006477ae25-162e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:29:25 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:35:40.722 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/pklxkd-55/c 2023-05-31T21:35:40.722 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/pklxkd-55/c 2023-05-31T21:35:40.722 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-05-31T21:35:40.723 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /pklxkd-55/c 2023-05-31T21:35:40.723 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-05-31T21:35:40.723 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-05-31T21:35:40.723 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T21:35:40.724 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-05-31T21:35:40.724 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-05-31T21:35:40.724 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-05-31T21:35:40.724 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-05-31T21:35:40.724 INFO:tasks.rgw_multisite_tests:HEAD 2023-05-31T21:35:40.725 INFO:tasks.rgw_multisite_tests: 2023-05-31T21:35:40.725 INFO:tasks.rgw_multisite_tests: 2023-05-31T21:35:40.725 INFO:tasks.rgw_multisite_tests:Wed, 31 May 2023 20:29:25 GMT 2023-05-31T21:35:40.725 INFO:tasks.rgw_multisite_tests:/pklxkd-55/c 2023-05-31T21:35:40.726 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-05-31T21:35:40.726 INFO:tasks.rgw_multisite_tests:AWS Q80H11OLNU1S48PPYL26:qyeZ0BO5kT8gimi9XwzG/GXY2sA= 2023-05-31T21:35:40.726 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:29:25 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:qyeZ0BO5kT8gimi9XwzG/GXY2sA=', 'Content-Length': '0'} 2023-05-31T21:35:40.726 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:27:16 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003f1903bc8a258fa7-006477ae25-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:29:25 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:35:40.727 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: comparing key name=c 2023-05-31T21:35:40.727 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/pklxkd-55/c 2023-05-31T21:35:40.727 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/pklxkd-55/c 2023-05-31T21:35:40.727 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-05-31T21:35:40.727 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /pklxkd-55/c 2023-05-31T21:35:40.728 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-05-31T21:35:40.728 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-05-31T21:35:40.728 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T21:35:40.728 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-05-31T21:35:40.729 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-05-31T21:35:40.729 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-05-31T21:35:40.729 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-05-31T21:35:40.729 INFO:tasks.rgw_multisite_tests:GET 2023-05-31T21:35:40.730 INFO:tasks.rgw_multisite_tests: 2023-05-31T21:35:40.730 INFO:tasks.rgw_multisite_tests: 2023-05-31T21:35:40.730 INFO:tasks.rgw_multisite_tests:Wed, 31 May 2023 20:29:25 GMT 2023-05-31T21:35:40.730 INFO:tasks.rgw_multisite_tests:/pklxkd-55/c 2023-05-31T21:35:40.730 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-05-31T21:35:40.731 INFO:tasks.rgw_multisite_tests:AWS Q80H11OLNU1S48PPYL26:ZwWUW/OF/WaE6lo6AuNjD64oDWo= 2023-05-31T21:35:40.731 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:29:25 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:ZwWUW/OF/WaE6lo6AuNjD64oDWo=', 'Content-Length': '0'} 2023-05-31T21:35:40.731 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:27:16 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c2ef3fe246dc75c4-006477ae25-162e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:29:25 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:35:40.731 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/pklxkd-55/c 2023-05-31T21:35:40.732 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/pklxkd-55/c 2023-05-31T21:35:40.732 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-05-31T21:35:40.732 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /pklxkd-55/c 2023-05-31T21:35:40.732 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-05-31T21:35:40.733 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-05-31T21:35:40.733 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T21:35:40.733 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-05-31T21:35:40.733 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-05-31T21:35:40.733 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-05-31T21:35:40.734 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-05-31T21:35:40.734 INFO:tasks.rgw_multisite_tests:GET 2023-05-31T21:35:40.734 INFO:tasks.rgw_multisite_tests: 2023-05-31T21:35:40.734 INFO:tasks.rgw_multisite_tests: 2023-05-31T21:35:40.735 INFO:tasks.rgw_multisite_tests:Wed, 31 May 2023 20:29:25 GMT 2023-05-31T21:35:40.735 INFO:tasks.rgw_multisite_tests:/pklxkd-55/c 2023-05-31T21:35:40.735 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-05-31T21:35:40.735 INFO:tasks.rgw_multisite_tests:AWS Q80H11OLNU1S48PPYL26:ZwWUW/OF/WaE6lo6AuNjD64oDWo= 2023-05-31T21:35:40.736 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:29:25 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:ZwWUW/OF/WaE6lo6AuNjD64oDWo=', 'Content-Length': '0'} 2023-05-31T21:35:40.736 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:27:16 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000032e76ae3a29807c5-006477ae25-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:29:25 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:35:40.736 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/pklxkd-55/ 2023-05-31T21:35:40.736 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/pklxkd-55/ 2023-05-31T21:35:40.736 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/pklxkd-55/?prefix=c 2023-05-31T21:35:40.737 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/pklxkd-55/?prefix=c 2023-05-31T21:35:40.737 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-05-31T21:35:40.737 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /pklxkd-55/?prefix=c 2023-05-31T21:35:40.737 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-05-31T21:35:40.738 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-05-31T21:35:40.738 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T21:35:40.738 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-05-31T21:35:40.738 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-05-31T21:35:40.739 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-05-31T21:35:40.739 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-05-31T21:35:40.739 INFO:tasks.rgw_multisite_tests:GET 2023-05-31T21:35:40.739 INFO:tasks.rgw_multisite_tests: 2023-05-31T21:35:40.739 INFO:tasks.rgw_multisite_tests: 2023-05-31T21:35:40.740 INFO:tasks.rgw_multisite_tests:Wed, 31 May 2023 20:29:25 GMT 2023-05-31T21:35:40.740 INFO:tasks.rgw_multisite_tests:/pklxkd-55/ 2023-05-31T21:35:40.740 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-05-31T21:35:40.740 INFO:tasks.rgw_multisite_tests:AWS Q80H11OLNU1S48PPYL26:a0/qPp0bnZeH62hZJf6OA1a38dw= 2023-05-31T21:35:40.741 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:29:25 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:a0/qPp0bnZeH62hZJf6OA1a38dw=', 'Content-Length': '0'} 2023-05-31T21:35:40.741 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000004d93bd42868f12e4-006477ae25-162e-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:29:25 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:35:40.741 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'pklxkd-55c1000falsec2023-05-31T20:27:16.869Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T21:35:40.741 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/pklxkd-55/ 2023-05-31T21:35:40.742 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/pklxkd-55/ 2023-05-31T21:35:40.742 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/pklxkd-55/?prefix=c 2023-05-31T21:35:40.742 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/pklxkd-55/?prefix=c 2023-05-31T21:35:40.742 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-05-31T21:35:40.742 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /pklxkd-55/?prefix=c 2023-05-31T21:35:40.743 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-05-31T21:35:40.743 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-05-31T21:35:40.743 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T21:35:40.743 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-05-31T21:35:40.744 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-05-31T21:35:40.744 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-05-31T21:35:40.744 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-05-31T21:35:40.744 INFO:tasks.rgw_multisite_tests:GET 2023-05-31T21:35:40.744 INFO:tasks.rgw_multisite_tests: 2023-05-31T21:35:40.745 INFO:tasks.rgw_multisite_tests: 2023-05-31T21:35:40.745 INFO:tasks.rgw_multisite_tests:Wed, 31 May 2023 20:29:25 GMT 2023-05-31T21:35:40.745 INFO:tasks.rgw_multisite_tests:/pklxkd-55/ 2023-05-31T21:35:40.745 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-05-31T21:35:40.746 INFO:tasks.rgw_multisite_tests:AWS Q80H11OLNU1S48PPYL26:a0/qPp0bnZeH62hZJf6OA1a38dw= 2023-05-31T21:35:40.746 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:29:25 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:a0/qPp0bnZeH62hZJf6OA1a38dw=', 'Content-Length': '0'} 2023-05-31T21:35:40.746 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000002e277aeb8b50bd36-006477ae25-1630-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:29:25 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:35:40.746 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'pklxkd-55c1000falsec2023-05-31T20:27:16.869Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T21:35:40.747 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: comparing key name=d 2023-05-31T21:35:40.747 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/pklxkd-55/d 2023-05-31T21:35:40.747 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/pklxkd-55/d 2023-05-31T21:35:40.747 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/pklxkd-55/d?versionId=null 2023-05-31T21:35:40.748 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/pklxkd-55/d?versionId=null 2023-05-31T21:35:40.748 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-05-31T21:35:40.748 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /pklxkd-55/d?versionId=null 2023-05-31T21:35:40.748 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-05-31T21:35:40.748 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-05-31T21:35:40.749 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T21:35:40.749 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-05-31T21:35:40.749 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-05-31T21:35:40.749 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-05-31T21:35:40.750 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-05-31T21:35:40.750 INFO:tasks.rgw_multisite_tests:GET 2023-05-31T21:35:40.750 INFO:tasks.rgw_multisite_tests: 2023-05-31T21:35:40.750 INFO:tasks.rgw_multisite_tests: 2023-05-31T21:35:40.751 INFO:tasks.rgw_multisite_tests:Wed, 31 May 2023 20:29:25 GMT 2023-05-31T21:35:40.751 INFO:tasks.rgw_multisite_tests:/pklxkd-55/d?versionId=null 2023-05-31T21:35:40.751 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-05-31T21:35:40.751 INFO:tasks.rgw_multisite_tests:AWS Q80H11OLNU1S48PPYL26:UTL6x5PDnkv/ObW9lagIN65WP+4= 2023-05-31T21:35:40.751 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:29:25 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:UTL6x5PDnkv/ObW9lagIN65WP+4=', 'Content-Length': '0'} 2023-05-31T21:35:40.752 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:27:16 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008ca3940a5c0c8656-006477ae25-162e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:29:25 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:35:40.752 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/pklxkd-55/d 2023-05-31T21:35:40.752 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/pklxkd-55/d 2023-05-31T21:35:40.752 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/pklxkd-55/d?versionId=null 2023-05-31T21:35:40.753 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/pklxkd-55/d?versionId=null 2023-05-31T21:35:40.753 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-05-31T21:35:40.753 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /pklxkd-55/d?versionId=null 2023-05-31T21:35:40.753 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-05-31T21:35:40.754 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-05-31T21:35:40.754 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T21:35:40.754 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-05-31T21:35:40.754 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-05-31T21:35:40.754 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-05-31T21:35:40.755 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-05-31T21:35:40.755 INFO:tasks.rgw_multisite_tests:GET 2023-05-31T21:35:40.755 INFO:tasks.rgw_multisite_tests: 2023-05-31T21:35:40.755 INFO:tasks.rgw_multisite_tests: 2023-05-31T21:35:40.756 INFO:tasks.rgw_multisite_tests:Wed, 31 May 2023 20:29:25 GMT 2023-05-31T21:35:40.756 INFO:tasks.rgw_multisite_tests:/pklxkd-55/d?versionId=null 2023-05-31T21:35:40.756 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-05-31T21:35:40.756 INFO:tasks.rgw_multisite_tests:AWS Q80H11OLNU1S48PPYL26:UTL6x5PDnkv/ObW9lagIN65WP+4= 2023-05-31T21:35:40.757 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:29:25 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:UTL6x5PDnkv/ObW9lagIN65WP+4=', 'Content-Length': '0'} 2023-05-31T21:35:40.757 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:27:16 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d622e37ec0832bd9-006477ae25-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:29:25 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:35:40.757 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/pklxkd-55/d 2023-05-31T21:35:40.757 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/pklxkd-55/d 2023-05-31T21:35:40.757 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/pklxkd-55/d?versionId=null 2023-05-31T21:35:40.758 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/pklxkd-55/d?versionId=null 2023-05-31T21:35:40.758 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-05-31T21:35:40.758 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /pklxkd-55/d?versionId=null 2023-05-31T21:35:40.758 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-05-31T21:35:40.759 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-05-31T21:35:40.759 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T21:35:40.759 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-05-31T21:35:40.759 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-05-31T21:35:40.759 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-05-31T21:35:40.760 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-05-31T21:35:40.760 INFO:tasks.rgw_multisite_tests:HEAD 2023-05-31T21:35:40.760 INFO:tasks.rgw_multisite_tests: 2023-05-31T21:35:40.760 INFO:tasks.rgw_multisite_tests: 2023-05-31T21:35:40.761 INFO:tasks.rgw_multisite_tests:Wed, 31 May 2023 20:29:25 GMT 2023-05-31T21:35:40.761 INFO:tasks.rgw_multisite_tests:/pklxkd-55/d?versionId=null 2023-05-31T21:35:40.761 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-05-31T21:35:40.761 INFO:tasks.rgw_multisite_tests:AWS Q80H11OLNU1S48PPYL26:ppfLEA4iI3ZziGNx+2cYKWodLG8= 2023-05-31T21:35:40.762 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:29:25 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:ppfLEA4iI3ZziGNx+2cYKWodLG8=', 'Content-Length': '0'} 2023-05-31T21:35:40.762 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:27:16 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d9ddef9e8bc8cf47-006477ae25-162e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:29:25 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:35:40.762 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/pklxkd-55/d 2023-05-31T21:35:40.762 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/pklxkd-55/d 2023-05-31T21:35:40.763 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/pklxkd-55/d?versionId=null 2023-05-31T21:35:40.763 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/pklxkd-55/d?versionId=null 2023-05-31T21:35:40.763 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-05-31T21:35:40.763 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /pklxkd-55/d?versionId=null 2023-05-31T21:35:40.763 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-05-31T21:35:40.764 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-05-31T21:35:40.764 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T21:35:40.764 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-05-31T21:35:40.764 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-05-31T21:35:40.765 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-05-31T21:35:40.765 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-05-31T21:35:40.765 INFO:tasks.rgw_multisite_tests:HEAD 2023-05-31T21:35:40.765 INFO:tasks.rgw_multisite_tests: 2023-05-31T21:35:40.765 INFO:tasks.rgw_multisite_tests: 2023-05-31T21:35:40.766 INFO:tasks.rgw_multisite_tests:Wed, 31 May 2023 20:29:25 GMT 2023-05-31T21:35:40.766 INFO:tasks.rgw_multisite_tests:/pklxkd-55/d?versionId=null 2023-05-31T21:35:40.766 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-05-31T21:35:40.766 INFO:tasks.rgw_multisite_tests:AWS Q80H11OLNU1S48PPYL26:ppfLEA4iI3ZziGNx+2cYKWodLG8= 2023-05-31T21:35:40.767 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:29:25 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:ppfLEA4iI3ZziGNx+2cYKWodLG8=', 'Content-Length': '0'} 2023-05-31T21:35:40.767 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:27:16 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b7613e7245136de3-006477ae25-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:29:25 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:35:40.767 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: comparing key name=d 2023-05-31T21:35:40.767 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/pklxkd-55/d 2023-05-31T21:35:40.768 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/pklxkd-55/d 2023-05-31T21:35:40.768 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/pklxkd-55/d?versionId=null 2023-05-31T21:35:40.768 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/pklxkd-55/d?versionId=null 2023-05-31T21:35:40.768 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-05-31T21:35:40.768 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /pklxkd-55/d?versionId=null 2023-05-31T21:35:40.769 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-05-31T21:35:40.769 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-05-31T21:35:40.769 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T21:35:40.769 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-05-31T21:35:40.770 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-05-31T21:35:40.770 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-05-31T21:35:40.770 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-05-31T21:35:40.770 INFO:tasks.rgw_multisite_tests:GET 2023-05-31T21:35:40.771 INFO:tasks.rgw_multisite_tests: 2023-05-31T21:35:40.771 INFO:tasks.rgw_multisite_tests: 2023-05-31T21:35:40.771 INFO:tasks.rgw_multisite_tests:Wed, 31 May 2023 20:29:25 GMT 2023-05-31T21:35:40.771 INFO:tasks.rgw_multisite_tests:/pklxkd-55/d?versionId=null 2023-05-31T21:35:40.771 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-05-31T21:35:40.772 INFO:tasks.rgw_multisite_tests:AWS Q80H11OLNU1S48PPYL26:UTL6x5PDnkv/ObW9lagIN65WP+4= 2023-05-31T21:35:40.772 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:29:25 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:UTL6x5PDnkv/ObW9lagIN65WP+4=', 'Content-Length': '0'} 2023-05-31T21:35:40.772 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:27:16 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000bd4caeab758f0f6f-006477ae25-162e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:29:25 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:35:40.772 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/pklxkd-55/d 2023-05-31T21:35:40.773 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/pklxkd-55/d 2023-05-31T21:35:40.773 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/pklxkd-55/d?versionId=null 2023-05-31T21:35:40.773 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/pklxkd-55/d?versionId=null 2023-05-31T21:35:40.773 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-05-31T21:35:40.773 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /pklxkd-55/d?versionId=null 2023-05-31T21:35:40.774 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-05-31T21:35:40.774 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-05-31T21:35:40.774 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T21:35:40.774 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-05-31T21:35:40.775 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-05-31T21:35:40.775 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-05-31T21:35:40.775 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-05-31T21:35:40.775 INFO:tasks.rgw_multisite_tests:GET 2023-05-31T21:35:40.776 INFO:tasks.rgw_multisite_tests: 2023-05-31T21:35:40.776 INFO:tasks.rgw_multisite_tests: 2023-05-31T21:35:40.776 INFO:tasks.rgw_multisite_tests:Wed, 31 May 2023 20:29:25 GMT 2023-05-31T21:35:40.776 INFO:tasks.rgw_multisite_tests:/pklxkd-55/d?versionId=null 2023-05-31T21:35:40.776 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-05-31T21:35:40.777 INFO:tasks.rgw_multisite_tests:AWS Q80H11OLNU1S48PPYL26:UTL6x5PDnkv/ObW9lagIN65WP+4= 2023-05-31T21:35:40.777 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:29:25 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:UTL6x5PDnkv/ObW9lagIN65WP+4=', 'Content-Length': '0'} 2023-05-31T21:35:40.777 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:27:16 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000456554cb69afb174-006477ae25-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:29:25 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:35:40.778 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/pklxkd-55/ 2023-05-31T21:35:40.778 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/pklxkd-55/ 2023-05-31T21:35:40.778 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/pklxkd-55/?prefix=d 2023-05-31T21:35:40.778 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/pklxkd-55/?prefix=d 2023-05-31T21:35:40.779 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-05-31T21:35:40.779 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /pklxkd-55/?prefix=d 2023-05-31T21:35:40.779 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-05-31T21:35:40.779 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-05-31T21:35:40.780 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T21:35:40.780 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-05-31T21:35:40.780 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-05-31T21:35:40.780 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-05-31T21:35:40.780 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-05-31T21:35:40.781 INFO:tasks.rgw_multisite_tests:GET 2023-05-31T21:35:40.781 INFO:tasks.rgw_multisite_tests: 2023-05-31T21:35:40.781 INFO:tasks.rgw_multisite_tests: 2023-05-31T21:35:40.781 INFO:tasks.rgw_multisite_tests:Wed, 31 May 2023 20:29:25 GMT 2023-05-31T21:35:40.782 INFO:tasks.rgw_multisite_tests:/pklxkd-55/ 2023-05-31T21:35:40.782 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-05-31T21:35:40.782 INFO:tasks.rgw_multisite_tests:AWS Q80H11OLNU1S48PPYL26:a0/qPp0bnZeH62hZJf6OA1a38dw= 2023-05-31T21:35:40.782 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:29:25 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:a0/qPp0bnZeH62hZJf6OA1a38dw=', 'Content-Length': '0'} 2023-05-31T21:35:40.783 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000fc0d4ecdab300639-006477ae25-162e-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:29:25 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:35:40.783 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'pklxkd-55d1000falsed2023-05-31T20:27:16.893Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T21:35:40.783 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/pklxkd-55/ 2023-05-31T21:35:40.783 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/pklxkd-55/ 2023-05-31T21:35:40.783 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/pklxkd-55/?prefix=d 2023-05-31T21:35:40.784 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/pklxkd-55/?prefix=d 2023-05-31T21:35:40.784 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-05-31T21:35:40.784 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /pklxkd-55/?prefix=d 2023-05-31T21:35:40.784 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-05-31T21:35:40.785 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-05-31T21:35:40.785 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T21:35:40.785 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-05-31T21:35:40.785 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-05-31T21:35:40.786 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-05-31T21:35:40.786 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-05-31T21:35:40.786 INFO:tasks.rgw_multisite_tests:GET 2023-05-31T21:35:40.786 INFO:tasks.rgw_multisite_tests: 2023-05-31T21:35:40.787 INFO:tasks.rgw_multisite_tests: 2023-05-31T21:35:40.787 INFO:tasks.rgw_multisite_tests:Wed, 31 May 2023 20:29:25 GMT 2023-05-31T21:35:40.787 INFO:tasks.rgw_multisite_tests:/pklxkd-55/ 2023-05-31T21:35:40.787 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-05-31T21:35:40.787 INFO:tasks.rgw_multisite_tests:AWS Q80H11OLNU1S48PPYL26:a0/qPp0bnZeH62hZJf6OA1a38dw= 2023-05-31T21:35:40.788 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:29:25 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:a0/qPp0bnZeH62hZJf6OA1a38dw=', 'Content-Length': '0'} 2023-05-31T21:35:40.788 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000475f353ad9c3a236-006477ae25-1630-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:29:25 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:35:40.788 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'pklxkd-55d1000falsed2023-05-31T20:27:16.893Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T21:35:40.788 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/pklxkd-55/d 2023-05-31T21:35:40.789 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/pklxkd-55/d 2023-05-31T21:35:40.789 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-05-31T21:35:40.789 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /pklxkd-55/d 2023-05-31T21:35:40.789 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-05-31T21:35:40.790 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-05-31T21:35:40.790 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T21:35:40.790 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-05-31T21:35:40.790 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-05-31T21:35:40.791 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-05-31T21:35:40.791 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-05-31T21:35:40.791 INFO:tasks.rgw_multisite_tests:HEAD 2023-05-31T21:35:40.791 INFO:tasks.rgw_multisite_tests: 2023-05-31T21:35:40.791 INFO:tasks.rgw_multisite_tests: 2023-05-31T21:35:40.792 INFO:tasks.rgw_multisite_tests:Wed, 31 May 2023 20:29:25 GMT 2023-05-31T21:35:40.792 INFO:tasks.rgw_multisite_tests:/pklxkd-55/d 2023-05-31T21:35:40.792 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-05-31T21:35:40.792 INFO:tasks.rgw_multisite_tests:AWS Q80H11OLNU1S48PPYL26:/+mjzkh9GSY0ENrh4t2bca2ZSiI= 2023-05-31T21:35:40.793 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:29:25 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:/+mjzkh9GSY0ENrh4t2bca2ZSiI=', 'Content-Length': '0'} 2023-05-31T21:35:40.793 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:27:16 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003fc58f1d87587d62-006477ae25-162e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:29:25 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:35:40.793 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/pklxkd-55/d 2023-05-31T21:35:40.793 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/pklxkd-55/d 2023-05-31T21:35:40.794 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-05-31T21:35:40.794 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /pklxkd-55/d 2023-05-31T21:35:40.794 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-05-31T21:35:40.794 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-05-31T21:35:40.794 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T21:35:40.795 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-05-31T21:35:40.795 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-05-31T21:35:40.795 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-05-31T21:35:40.795 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-05-31T21:35:40.796 INFO:tasks.rgw_multisite_tests:HEAD 2023-05-31T21:35:40.796 INFO:tasks.rgw_multisite_tests: 2023-05-31T21:35:40.796 INFO:tasks.rgw_multisite_tests: 2023-05-31T21:35:40.796 INFO:tasks.rgw_multisite_tests:Wed, 31 May 2023 20:29:25 GMT 2023-05-31T21:35:40.797 INFO:tasks.rgw_multisite_tests:/pklxkd-55/d 2023-05-31T21:35:40.797 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-05-31T21:35:40.797 INFO:tasks.rgw_multisite_tests:AWS Q80H11OLNU1S48PPYL26:/+mjzkh9GSY0ENrh4t2bca2ZSiI= 2023-05-31T21:35:40.797 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:29:25 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:/+mjzkh9GSY0ENrh4t2bca2ZSiI=', 'Content-Length': '0'} 2023-05-31T21:35:40.797 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:27:16 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e180f725adc35ac8-006477ae25-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:29:25 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:35:40.798 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: comparing key name=d 2023-05-31T21:35:40.798 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/pklxkd-55/d 2023-05-31T21:35:40.798 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/pklxkd-55/d 2023-05-31T21:35:40.798 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-05-31T21:35:40.799 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /pklxkd-55/d 2023-05-31T21:35:40.799 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-05-31T21:35:40.799 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-05-31T21:35:40.799 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T21:35:40.800 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-05-31T21:35:40.800 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-05-31T21:35:40.800 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-05-31T21:35:40.800 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-05-31T21:35:40.800 INFO:tasks.rgw_multisite_tests:GET 2023-05-31T21:35:40.801 INFO:tasks.rgw_multisite_tests: 2023-05-31T21:35:40.801 INFO:tasks.rgw_multisite_tests: 2023-05-31T21:35:40.801 INFO:tasks.rgw_multisite_tests:Wed, 31 May 2023 20:29:25 GMT 2023-05-31T21:35:40.801 INFO:tasks.rgw_multisite_tests:/pklxkd-55/d 2023-05-31T21:35:40.802 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-05-31T21:35:40.802 INFO:tasks.rgw_multisite_tests:AWS Q80H11OLNU1S48PPYL26:n/qR9T6ixiWdO9ktyq98t+xoXn8= 2023-05-31T21:35:40.802 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:29:25 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:n/qR9T6ixiWdO9ktyq98t+xoXn8=', 'Content-Length': '0'} 2023-05-31T21:35:40.802 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:27:16 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b4d026ddc8119cd4-006477ae25-162e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:29:25 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:35:40.803 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/pklxkd-55/d 2023-05-31T21:35:40.803 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/pklxkd-55/d 2023-05-31T21:35:40.803 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-05-31T21:35:40.803 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /pklxkd-55/d 2023-05-31T21:35:40.803 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-05-31T21:35:40.804 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-05-31T21:35:40.804 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T21:35:40.804 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-05-31T21:35:40.804 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-05-31T21:35:40.805 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-05-31T21:35:40.805 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-05-31T21:35:40.805 INFO:tasks.rgw_multisite_tests:GET 2023-05-31T21:35:40.805 INFO:tasks.rgw_multisite_tests: 2023-05-31T21:35:40.806 INFO:tasks.rgw_multisite_tests: 2023-05-31T21:35:40.806 INFO:tasks.rgw_multisite_tests:Wed, 31 May 2023 20:29:25 GMT 2023-05-31T21:35:40.806 INFO:tasks.rgw_multisite_tests:/pklxkd-55/d 2023-05-31T21:35:40.806 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-05-31T21:35:40.806 INFO:tasks.rgw_multisite_tests:AWS Q80H11OLNU1S48PPYL26:n/qR9T6ixiWdO9ktyq98t+xoXn8= 2023-05-31T21:35:40.807 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:29:25 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:n/qR9T6ixiWdO9ktyq98t+xoXn8=', 'Content-Length': '0'} 2023-05-31T21:35:40.807 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:27:16 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000423e6993961249ae-006477ae25-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:29:25 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:35:40.807 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/pklxkd-55/ 2023-05-31T21:35:40.807 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/pklxkd-55/ 2023-05-31T21:35:40.808 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/pklxkd-55/?prefix=d 2023-05-31T21:35:40.808 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/pklxkd-55/?prefix=d 2023-05-31T21:35:40.808 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-05-31T21:35:40.808 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /pklxkd-55/?prefix=d 2023-05-31T21:35:40.809 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-05-31T21:35:40.809 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-05-31T21:35:40.809 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T21:35:40.809 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-05-31T21:35:40.810 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-05-31T21:35:40.810 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-05-31T21:35:40.810 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-05-31T21:35:40.810 INFO:tasks.rgw_multisite_tests:GET 2023-05-31T21:35:40.810 INFO:tasks.rgw_multisite_tests: 2023-05-31T21:35:40.811 INFO:tasks.rgw_multisite_tests: 2023-05-31T21:35:40.811 INFO:tasks.rgw_multisite_tests:Wed, 31 May 2023 20:29:25 GMT 2023-05-31T21:35:40.811 INFO:tasks.rgw_multisite_tests:/pklxkd-55/ 2023-05-31T21:35:40.811 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-05-31T21:35:40.812 INFO:tasks.rgw_multisite_tests:AWS Q80H11OLNU1S48PPYL26:a0/qPp0bnZeH62hZJf6OA1a38dw= 2023-05-31T21:35:40.812 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:29:25 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:a0/qPp0bnZeH62hZJf6OA1a38dw=', 'Content-Length': '0'} 2023-05-31T21:35:40.812 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000007adba8616aaa56f2-006477ae25-162e-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:29:25 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:35:40.812 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'pklxkd-55d1000falsed2023-05-31T20:27:16.893Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T21:35:40.813 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/pklxkd-55/ 2023-05-31T21:35:40.813 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/pklxkd-55/ 2023-05-31T21:35:40.813 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/pklxkd-55/?prefix=d 2023-05-31T21:35:40.813 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/pklxkd-55/?prefix=d 2023-05-31T21:35:40.813 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-05-31T21:35:40.814 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /pklxkd-55/?prefix=d 2023-05-31T21:35:40.814 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-05-31T21:35:40.814 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-05-31T21:35:40.814 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T21:35:40.815 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-05-31T21:35:40.815 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-05-31T21:35:40.815 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-05-31T21:35:40.815 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-05-31T21:35:40.816 INFO:tasks.rgw_multisite_tests:GET 2023-05-31T21:35:40.816 INFO:tasks.rgw_multisite_tests: 2023-05-31T21:35:40.816 INFO:tasks.rgw_multisite_tests: 2023-05-31T21:35:40.816 INFO:tasks.rgw_multisite_tests:Wed, 31 May 2023 20:29:25 GMT 2023-05-31T21:35:40.816 INFO:tasks.rgw_multisite_tests:/pklxkd-55/ 2023-05-31T21:35:40.817 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-05-31T21:35:40.817 INFO:tasks.rgw_multisite_tests:AWS Q80H11OLNU1S48PPYL26:a0/qPp0bnZeH62hZJf6OA1a38dw= 2023-05-31T21:35:40.817 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:29:25 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:a0/qPp0bnZeH62hZJf6OA1a38dw=', 'Content-Length': '0'} 2023-05-31T21:35:40.817 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000fec8473da25d2966-006477ae25-1630-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:29:25 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:35:40.818 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'pklxkd-55d1000falsed2023-05-31T20:27:16.893Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T21:35:40.818 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: comparing key name=e 2023-05-31T21:35:40.818 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/pklxkd-55/e 2023-05-31T21:35:40.818 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/pklxkd-55/e 2023-05-31T21:35:40.819 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/pklxkd-55/e?versionId=null 2023-05-31T21:35:40.819 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/pklxkd-55/e?versionId=null 2023-05-31T21:35:40.819 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-05-31T21:35:40.819 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /pklxkd-55/e?versionId=null 2023-05-31T21:35:40.819 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-05-31T21:35:40.820 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-05-31T21:35:40.820 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T21:35:40.820 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-05-31T21:35:40.820 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-05-31T21:35:40.821 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-05-31T21:35:40.821 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-05-31T21:35:40.821 INFO:tasks.rgw_multisite_tests:GET 2023-05-31T21:35:40.821 INFO:tasks.rgw_multisite_tests: 2023-05-31T21:35:40.822 INFO:tasks.rgw_multisite_tests: 2023-05-31T21:35:40.822 INFO:tasks.rgw_multisite_tests:Wed, 31 May 2023 20:29:25 GMT 2023-05-31T21:35:40.822 INFO:tasks.rgw_multisite_tests:/pklxkd-55/e?versionId=null 2023-05-31T21:35:40.822 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-05-31T21:35:40.823 INFO:tasks.rgw_multisite_tests:AWS Q80H11OLNU1S48PPYL26:+YJyBGr8MemE+vt1MoE2zrx05v0= 2023-05-31T21:35:40.823 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:29:25 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:+YJyBGr8MemE+vt1MoE2zrx05v0=', 'Content-Length': '0'} 2023-05-31T21:35:40.823 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:27:23 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001ba4ae6a42db484a-006477ae25-162e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:29:25 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:35:40.823 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/pklxkd-55/e 2023-05-31T21:35:40.823 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/pklxkd-55/e 2023-05-31T21:35:40.824 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/pklxkd-55/e?versionId=null 2023-05-31T21:35:40.824 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/pklxkd-55/e?versionId=null 2023-05-31T21:35:40.824 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-05-31T21:35:40.824 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /pklxkd-55/e?versionId=null 2023-05-31T21:35:40.825 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-05-31T21:35:40.825 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-05-31T21:35:40.825 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T21:35:40.825 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-05-31T21:35:40.826 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-05-31T21:35:40.826 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-05-31T21:35:40.826 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-05-31T21:35:40.826 INFO:tasks.rgw_multisite_tests:GET 2023-05-31T21:35:40.827 INFO:tasks.rgw_multisite_tests: 2023-05-31T21:35:40.827 INFO:tasks.rgw_multisite_tests: 2023-05-31T21:35:40.827 INFO:tasks.rgw_multisite_tests:Wed, 31 May 2023 20:29:25 GMT 2023-05-31T21:35:40.827 INFO:tasks.rgw_multisite_tests:/pklxkd-55/e?versionId=null 2023-05-31T21:35:40.827 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-05-31T21:35:40.828 INFO:tasks.rgw_multisite_tests:AWS Q80H11OLNU1S48PPYL26:+YJyBGr8MemE+vt1MoE2zrx05v0= 2023-05-31T21:35:40.828 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:29:25 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:+YJyBGr8MemE+vt1MoE2zrx05v0=', 'Content-Length': '0'} 2023-05-31T21:35:40.828 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:27:23 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000195d1e3efb6556e0-006477ae25-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:29:25 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:35:40.828 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/pklxkd-55/e 2023-05-31T21:35:40.829 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/pklxkd-55/e 2023-05-31T21:35:40.829 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/pklxkd-55/e?versionId=null 2023-05-31T21:35:40.829 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/pklxkd-55/e?versionId=null 2023-05-31T21:35:40.829 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-05-31T21:35:40.830 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /pklxkd-55/e?versionId=null 2023-05-31T21:35:40.830 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-05-31T21:35:40.830 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-05-31T21:35:40.830 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T21:35:40.830 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-05-31T21:35:40.831 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-05-31T21:35:40.831 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-05-31T21:35:40.831 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-05-31T21:35:40.831 INFO:tasks.rgw_multisite_tests:HEAD 2023-05-31T21:35:40.832 INFO:tasks.rgw_multisite_tests: 2023-05-31T21:35:40.832 INFO:tasks.rgw_multisite_tests: 2023-05-31T21:35:40.832 INFO:tasks.rgw_multisite_tests:Wed, 31 May 2023 20:29:25 GMT 2023-05-31T21:35:40.832 INFO:tasks.rgw_multisite_tests:/pklxkd-55/e?versionId=null 2023-05-31T21:35:40.833 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-05-31T21:35:40.833 INFO:tasks.rgw_multisite_tests:AWS Q80H11OLNU1S48PPYL26:kQUQG6QGzVq9qxPrQO457i+pbjI= 2023-05-31T21:35:40.833 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:29:25 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:kQUQG6QGzVq9qxPrQO457i+pbjI=', 'Content-Length': '0'} 2023-05-31T21:35:40.833 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:27:23 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000003c2ae36b56c84c2-006477ae25-162e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:29:25 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:35:40.834 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/pklxkd-55/e 2023-05-31T21:35:40.834 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/pklxkd-55/e 2023-05-31T21:35:40.834 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/pklxkd-55/e?versionId=null 2023-05-31T21:35:40.834 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/pklxkd-55/e?versionId=null 2023-05-31T21:35:40.834 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-05-31T21:35:40.835 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /pklxkd-55/e?versionId=null 2023-05-31T21:35:40.835 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-05-31T21:35:40.835 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-05-31T21:35:40.835 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T21:35:40.836 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-05-31T21:35:40.836 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-05-31T21:35:40.836 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-05-31T21:35:40.836 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-05-31T21:35:40.837 INFO:tasks.rgw_multisite_tests:HEAD 2023-05-31T21:35:40.837 INFO:tasks.rgw_multisite_tests: 2023-05-31T21:35:40.837 INFO:tasks.rgw_multisite_tests: 2023-05-31T21:35:40.837 INFO:tasks.rgw_multisite_tests:Wed, 31 May 2023 20:29:25 GMT 2023-05-31T21:35:40.837 INFO:tasks.rgw_multisite_tests:/pklxkd-55/e?versionId=null 2023-05-31T21:35:40.838 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-05-31T21:35:40.838 INFO:tasks.rgw_multisite_tests:AWS Q80H11OLNU1S48PPYL26:kQUQG6QGzVq9qxPrQO457i+pbjI= 2023-05-31T21:35:40.838 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:29:25 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:kQUQG6QGzVq9qxPrQO457i+pbjI=', 'Content-Length': '0'} 2023-05-31T21:35:40.838 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:27:23 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007e9c679b223c4f90-006477ae25-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:29:25 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:35:40.839 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: comparing key name=e 2023-05-31T21:35:40.839 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/pklxkd-55/e 2023-05-31T21:35:40.839 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/pklxkd-55/e 2023-05-31T21:35:40.839 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/pklxkd-55/e?versionId=null 2023-05-31T21:35:40.840 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/pklxkd-55/e?versionId=null 2023-05-31T21:35:40.840 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-05-31T21:35:40.840 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /pklxkd-55/e?versionId=null 2023-05-31T21:35:40.840 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-05-31T21:35:40.840 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-05-31T21:35:40.841 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T21:35:40.841 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-05-31T21:35:40.841 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-05-31T21:35:40.841 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-05-31T21:35:40.842 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-05-31T21:35:40.842 INFO:tasks.rgw_multisite_tests:GET 2023-05-31T21:35:40.842 INFO:tasks.rgw_multisite_tests: 2023-05-31T21:35:40.842 INFO:tasks.rgw_multisite_tests: 2023-05-31T21:35:40.843 INFO:tasks.rgw_multisite_tests:Wed, 31 May 2023 20:29:25 GMT 2023-05-31T21:35:40.843 INFO:tasks.rgw_multisite_tests:/pklxkd-55/e?versionId=null 2023-05-31T21:35:40.843 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-05-31T21:35:40.843 INFO:tasks.rgw_multisite_tests:AWS Q80H11OLNU1S48PPYL26:+YJyBGr8MemE+vt1MoE2zrx05v0= 2023-05-31T21:35:40.843 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:29:25 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:+YJyBGr8MemE+vt1MoE2zrx05v0=', 'Content-Length': '0'} 2023-05-31T21:35:40.844 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:27:23 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000030de495e5fcd463d-006477ae25-162e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:29:25 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:35:40.844 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/pklxkd-55/e 2023-05-31T21:35:40.844 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/pklxkd-55/e 2023-05-31T21:35:40.844 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/pklxkd-55/e?versionId=null 2023-05-31T21:35:40.845 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/pklxkd-55/e?versionId=null 2023-05-31T21:35:40.845 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-05-31T21:35:40.845 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /pklxkd-55/e?versionId=null 2023-05-31T21:35:40.845 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-05-31T21:35:40.846 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-05-31T21:35:40.846 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T21:35:40.846 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-05-31T21:35:40.846 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-05-31T21:35:40.846 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-05-31T21:35:40.847 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-05-31T21:35:40.847 INFO:tasks.rgw_multisite_tests:GET 2023-05-31T21:35:40.847 INFO:tasks.rgw_multisite_tests: 2023-05-31T21:35:40.847 INFO:tasks.rgw_multisite_tests: 2023-05-31T21:35:40.848 INFO:tasks.rgw_multisite_tests:Wed, 31 May 2023 20:29:25 GMT 2023-05-31T21:35:40.848 INFO:tasks.rgw_multisite_tests:/pklxkd-55/e?versionId=null 2023-05-31T21:35:40.848 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-05-31T21:35:40.848 INFO:tasks.rgw_multisite_tests:AWS Q80H11OLNU1S48PPYL26:+YJyBGr8MemE+vt1MoE2zrx05v0= 2023-05-31T21:35:40.849 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:29:25 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:+YJyBGr8MemE+vt1MoE2zrx05v0=', 'Content-Length': '0'} 2023-05-31T21:35:40.849 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:27:23 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d7da4ffc9c0cc262-006477ae25-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:29:25 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:35:40.849 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/pklxkd-55/ 2023-05-31T21:35:40.849 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/pklxkd-55/ 2023-05-31T21:35:40.849 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/pklxkd-55/?prefix=e 2023-05-31T21:35:40.850 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/pklxkd-55/?prefix=e 2023-05-31T21:35:40.850 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-05-31T21:35:40.850 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /pklxkd-55/?prefix=e 2023-05-31T21:35:40.850 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-05-31T21:35:40.851 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-05-31T21:35:40.851 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T21:35:40.851 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-05-31T21:35:40.851 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-05-31T21:35:40.852 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-05-31T21:35:40.852 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-05-31T21:35:40.852 INFO:tasks.rgw_multisite_tests:GET 2023-05-31T21:35:40.852 INFO:tasks.rgw_multisite_tests: 2023-05-31T21:35:40.852 INFO:tasks.rgw_multisite_tests: 2023-05-31T21:35:40.853 INFO:tasks.rgw_multisite_tests:Wed, 31 May 2023 20:29:25 GMT 2023-05-31T21:35:40.853 INFO:tasks.rgw_multisite_tests:/pklxkd-55/ 2023-05-31T21:35:40.853 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-05-31T21:35:40.853 INFO:tasks.rgw_multisite_tests:AWS Q80H11OLNU1S48PPYL26:a0/qPp0bnZeH62hZJf6OA1a38dw= 2023-05-31T21:35:40.854 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:29:25 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:a0/qPp0bnZeH62hZJf6OA1a38dw=', 'Content-Length': '0'} 2023-05-31T21:35:40.854 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b1450d5593cb5e3c-006477ae25-162e-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:29:25 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:35:40.854 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'pklxkd-55e1000falsee2023-05-31T20:27:23.955Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T21:35:40.854 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/pklxkd-55/ 2023-05-31T21:35:40.855 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/pklxkd-55/ 2023-05-31T21:35:40.855 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/pklxkd-55/?prefix=e 2023-05-31T21:35:40.855 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/pklxkd-55/?prefix=e 2023-05-31T21:35:40.855 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-05-31T21:35:40.855 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /pklxkd-55/?prefix=e 2023-05-31T21:35:40.856 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-05-31T21:35:40.856 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-05-31T21:35:40.856 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T21:35:40.856 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-05-31T21:35:40.857 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-05-31T21:35:40.857 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-05-31T21:35:40.857 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-05-31T21:35:40.857 INFO:tasks.rgw_multisite_tests:GET 2023-05-31T21:35:40.858 INFO:tasks.rgw_multisite_tests: 2023-05-31T21:35:40.858 INFO:tasks.rgw_multisite_tests: 2023-05-31T21:35:40.858 INFO:tasks.rgw_multisite_tests:Wed, 31 May 2023 20:29:25 GMT 2023-05-31T21:35:40.858 INFO:tasks.rgw_multisite_tests:/pklxkd-55/ 2023-05-31T21:35:40.859 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-05-31T21:35:40.859 INFO:tasks.rgw_multisite_tests:AWS Q80H11OLNU1S48PPYL26:a0/qPp0bnZeH62hZJf6OA1a38dw= 2023-05-31T21:35:40.859 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:29:25 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:a0/qPp0bnZeH62hZJf6OA1a38dw=', 'Content-Length': '0'} 2023-05-31T21:35:40.859 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000040a5747d5a70ec09-006477ae25-1630-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:29:25 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:35:40.859 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'pklxkd-55e1000falsee2023-05-31T20:27:23.955Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T21:35:40.860 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/pklxkd-55/e 2023-05-31T21:35:40.860 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/pklxkd-55/e 2023-05-31T21:35:40.860 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-05-31T21:35:40.860 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /pklxkd-55/e 2023-05-31T21:35:40.861 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-05-31T21:35:40.861 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-05-31T21:35:40.861 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T21:35:40.861 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-05-31T21:35:40.862 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-05-31T21:35:40.862 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-05-31T21:35:40.862 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-05-31T21:35:40.862 INFO:tasks.rgw_multisite_tests:HEAD 2023-05-31T21:35:40.863 INFO:tasks.rgw_multisite_tests: 2023-05-31T21:35:40.863 INFO:tasks.rgw_multisite_tests: 2023-05-31T21:35:40.863 INFO:tasks.rgw_multisite_tests:Wed, 31 May 2023 20:29:25 GMT 2023-05-31T21:35:40.863 INFO:tasks.rgw_multisite_tests:/pklxkd-55/e 2023-05-31T21:35:40.863 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-05-31T21:35:40.864 INFO:tasks.rgw_multisite_tests:AWS Q80H11OLNU1S48PPYL26:WI/JKAsiEmMNazlf+o8dalAWUSE= 2023-05-31T21:35:40.864 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:29:25 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:WI/JKAsiEmMNazlf+o8dalAWUSE=', 'Content-Length': '0'} 2023-05-31T21:35:40.864 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:27:23 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007fe39288ebc64206-006477ae25-162e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:29:25 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:35:40.864 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/pklxkd-55/e 2023-05-31T21:35:40.865 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/pklxkd-55/e 2023-05-31T21:35:40.865 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-05-31T21:35:40.865 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /pklxkd-55/e 2023-05-31T21:35:40.865 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-05-31T21:35:40.866 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-05-31T21:35:40.866 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T21:35:40.866 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-05-31T21:35:40.866 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-05-31T21:35:40.867 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-05-31T21:35:40.867 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-05-31T21:35:40.867 INFO:tasks.rgw_multisite_tests:HEAD 2023-05-31T21:35:40.867 INFO:tasks.rgw_multisite_tests: 2023-05-31T21:35:40.867 INFO:tasks.rgw_multisite_tests: 2023-05-31T21:35:40.868 INFO:tasks.rgw_multisite_tests:Wed, 31 May 2023 20:29:25 GMT 2023-05-31T21:35:40.868 INFO:tasks.rgw_multisite_tests:/pklxkd-55/e 2023-05-31T21:35:40.868 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-05-31T21:35:40.868 INFO:tasks.rgw_multisite_tests:AWS Q80H11OLNU1S48PPYL26:WI/JKAsiEmMNazlf+o8dalAWUSE= 2023-05-31T21:35:40.869 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:29:25 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:WI/JKAsiEmMNazlf+o8dalAWUSE=', 'Content-Length': '0'} 2023-05-31T21:35:40.869 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:27:23 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000068dc75829fdd7306-006477ae25-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:29:25 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:35:40.869 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: comparing key name=e 2023-05-31T21:35:40.869 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/pklxkd-55/e 2023-05-31T21:35:40.870 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/pklxkd-55/e 2023-05-31T21:35:40.870 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-05-31T21:35:40.870 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /pklxkd-55/e 2023-05-31T21:35:40.870 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-05-31T21:35:40.870 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-05-31T21:35:40.871 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T21:35:40.871 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-05-31T21:35:40.871 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-05-31T21:35:40.871 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-05-31T21:35:40.872 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-05-31T21:35:40.872 INFO:tasks.rgw_multisite_tests:GET 2023-05-31T21:35:40.872 INFO:tasks.rgw_multisite_tests: 2023-05-31T21:35:40.872 INFO:tasks.rgw_multisite_tests: 2023-05-31T21:35:40.873 INFO:tasks.rgw_multisite_tests:Wed, 31 May 2023 20:29:25 GMT 2023-05-31T21:35:40.873 INFO:tasks.rgw_multisite_tests:/pklxkd-55/e 2023-05-31T21:35:40.873 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-05-31T21:35:40.873 INFO:tasks.rgw_multisite_tests:AWS Q80H11OLNU1S48PPYL26:MSyipDFZixhTHZOKGu7tlz+s3lQ= 2023-05-31T21:35:40.873 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:29:25 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:MSyipDFZixhTHZOKGu7tlz+s3lQ=', 'Content-Length': '0'} 2023-05-31T21:35:40.874 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:27:23 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000ac76b8d29ae8a76-006477ae25-162e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:29:25 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:35:40.874 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/pklxkd-55/e 2023-05-31T21:35:40.874 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/pklxkd-55/e 2023-05-31T21:35:40.874 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-05-31T21:35:40.875 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /pklxkd-55/e 2023-05-31T21:35:40.875 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-05-31T21:35:40.875 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-05-31T21:35:40.875 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T21:35:40.876 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-05-31T21:35:40.876 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-05-31T21:35:40.876 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-05-31T21:35:40.876 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-05-31T21:35:40.876 INFO:tasks.rgw_multisite_tests:GET 2023-05-31T21:35:40.877 INFO:tasks.rgw_multisite_tests: 2023-05-31T21:35:40.877 INFO:tasks.rgw_multisite_tests: 2023-05-31T21:35:40.877 INFO:tasks.rgw_multisite_tests:Wed, 31 May 2023 20:29:25 GMT 2023-05-31T21:35:40.877 INFO:tasks.rgw_multisite_tests:/pklxkd-55/e 2023-05-31T21:35:40.878 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-05-31T21:35:40.878 INFO:tasks.rgw_multisite_tests:AWS Q80H11OLNU1S48PPYL26:MSyipDFZixhTHZOKGu7tlz+s3lQ= 2023-05-31T21:35:40.878 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:29:25 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:MSyipDFZixhTHZOKGu7tlz+s3lQ=', 'Content-Length': '0'} 2023-05-31T21:35:40.878 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:27:23 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000da86fe62bf35a242-006477ae25-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:29:25 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:35:40.879 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/pklxkd-55/ 2023-05-31T21:35:40.879 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/pklxkd-55/ 2023-05-31T21:35:40.879 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/pklxkd-55/?prefix=e 2023-05-31T21:35:40.879 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/pklxkd-55/?prefix=e 2023-05-31T21:35:40.880 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-05-31T21:35:40.880 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /pklxkd-55/?prefix=e 2023-05-31T21:35:40.880 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-05-31T21:35:40.880 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-05-31T21:35:40.881 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T21:35:40.881 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-05-31T21:35:40.881 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-05-31T21:35:40.881 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-05-31T21:35:40.881 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-05-31T21:35:40.882 INFO:tasks.rgw_multisite_tests:GET 2023-05-31T21:35:40.882 INFO:tasks.rgw_multisite_tests: 2023-05-31T21:35:40.882 INFO:tasks.rgw_multisite_tests: 2023-05-31T21:35:40.882 INFO:tasks.rgw_multisite_tests:Wed, 31 May 2023 20:29:25 GMT 2023-05-31T21:35:40.883 INFO:tasks.rgw_multisite_tests:/pklxkd-55/ 2023-05-31T21:35:40.883 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-05-31T21:35:40.883 INFO:tasks.rgw_multisite_tests:AWS Q80H11OLNU1S48PPYL26:a0/qPp0bnZeH62hZJf6OA1a38dw= 2023-05-31T21:35:40.883 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:29:25 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:a0/qPp0bnZeH62hZJf6OA1a38dw=', 'Content-Length': '0'} 2023-05-31T21:35:40.884 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000051b82ab65d1916ac-006477ae25-162e-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:29:25 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:35:40.884 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'pklxkd-55e1000falsee2023-05-31T20:27:23.955Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T21:35:40.884 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/pklxkd-55/ 2023-05-31T21:35:40.884 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/pklxkd-55/ 2023-05-31T21:35:40.885 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/pklxkd-55/?prefix=e 2023-05-31T21:35:40.885 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/pklxkd-55/?prefix=e 2023-05-31T21:35:40.885 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-05-31T21:35:40.885 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /pklxkd-55/?prefix=e 2023-05-31T21:35:40.885 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-05-31T21:35:40.886 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-05-31T21:35:40.886 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T21:35:40.886 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-05-31T21:35:40.886 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-05-31T21:35:40.887 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-05-31T21:35:40.887 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-05-31T21:35:40.887 INFO:tasks.rgw_multisite_tests:GET 2023-05-31T21:35:40.887 INFO:tasks.rgw_multisite_tests: 2023-05-31T21:35:40.887 INFO:tasks.rgw_multisite_tests: 2023-05-31T21:35:40.888 INFO:tasks.rgw_multisite_tests:Wed, 31 May 2023 20:29:25 GMT 2023-05-31T21:35:40.888 INFO:tasks.rgw_multisite_tests:/pklxkd-55/ 2023-05-31T21:35:40.888 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-05-31T21:35:40.888 INFO:tasks.rgw_multisite_tests:AWS Q80H11OLNU1S48PPYL26:a0/qPp0bnZeH62hZJf6OA1a38dw= 2023-05-31T21:35:40.889 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:29:25 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:a0/qPp0bnZeH62hZJf6OA1a38dw=', 'Content-Length': '0'} 2023-05-31T21:35:40.889 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000002a61b1ba3e303c88-006477ae25-1630-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:29:25 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:35:40.889 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'pklxkd-55e1000falsee2023-05-31T20:27:23.955Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T21:35:40.889 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: comparing key name=f 2023-05-31T21:35:40.890 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/pklxkd-55/f 2023-05-31T21:35:40.890 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/pklxkd-55/f 2023-05-31T21:35:40.890 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/pklxkd-55/f?versionId=null 2023-05-31T21:35:40.890 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/pklxkd-55/f?versionId=null 2023-05-31T21:35:40.891 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-05-31T21:35:40.891 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /pklxkd-55/f?versionId=null 2023-05-31T21:35:40.891 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-05-31T21:35:40.891 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-05-31T21:35:40.891 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T21:35:40.892 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-05-31T21:35:40.892 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-05-31T21:35:40.892 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-05-31T21:35:40.892 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-05-31T21:35:40.893 INFO:tasks.rgw_multisite_tests:GET 2023-05-31T21:35:40.893 INFO:tasks.rgw_multisite_tests: 2023-05-31T21:35:40.893 INFO:tasks.rgw_multisite_tests: 2023-05-31T21:35:40.893 INFO:tasks.rgw_multisite_tests:Wed, 31 May 2023 20:29:25 GMT 2023-05-31T21:35:40.894 INFO:tasks.rgw_multisite_tests:/pklxkd-55/f?versionId=null 2023-05-31T21:35:40.894 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-05-31T21:35:40.894 INFO:tasks.rgw_multisite_tests:AWS Q80H11OLNU1S48PPYL26:7DmZ2WsueqJzxqvgmHkyBb7pLCA= 2023-05-31T21:35:40.894 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:29:25 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:7DmZ2WsueqJzxqvgmHkyBb7pLCA=', 'Content-Length': '0'} 2023-05-31T21:35:40.894 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:27:23 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000052dca59ddc891543-006477ae25-162e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:29:25 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:35:40.895 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/pklxkd-55/f 2023-05-31T21:35:40.895 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/pklxkd-55/f 2023-05-31T21:35:40.895 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/pklxkd-55/f?versionId=null 2023-05-31T21:35:40.895 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/pklxkd-55/f?versionId=null 2023-05-31T21:35:40.896 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-05-31T21:35:40.896 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /pklxkd-55/f?versionId=null 2023-05-31T21:35:40.896 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-05-31T21:35:40.896 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-05-31T21:35:40.897 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T21:35:40.897 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-05-31T21:35:40.897 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-05-31T21:35:40.897 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-05-31T21:35:40.897 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-05-31T21:35:40.898 INFO:tasks.rgw_multisite_tests:GET 2023-05-31T21:35:40.898 INFO:tasks.rgw_multisite_tests: 2023-05-31T21:35:40.898 INFO:tasks.rgw_multisite_tests: 2023-05-31T21:35:40.898 INFO:tasks.rgw_multisite_tests:Wed, 31 May 2023 20:29:25 GMT 2023-05-31T21:35:40.899 INFO:tasks.rgw_multisite_tests:/pklxkd-55/f?versionId=null 2023-05-31T21:35:40.899 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-05-31T21:35:40.899 INFO:tasks.rgw_multisite_tests:AWS Q80H11OLNU1S48PPYL26:7DmZ2WsueqJzxqvgmHkyBb7pLCA= 2023-05-31T21:35:40.899 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:29:25 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:7DmZ2WsueqJzxqvgmHkyBb7pLCA=', 'Content-Length': '0'} 2023-05-31T21:35:40.900 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:27:23 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009dbe235e65120229-006477ae25-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:29:25 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:35:40.900 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/pklxkd-55/f 2023-05-31T21:35:40.900 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/pklxkd-55/f 2023-05-31T21:35:40.900 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/pklxkd-55/f?versionId=null 2023-05-31T21:35:40.901 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/pklxkd-55/f?versionId=null 2023-05-31T21:35:40.901 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-05-31T21:35:40.901 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /pklxkd-55/f?versionId=null 2023-05-31T21:35:40.901 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-05-31T21:35:40.901 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-05-31T21:35:40.902 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T21:35:40.902 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-05-31T21:35:40.902 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-05-31T21:35:40.902 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-05-31T21:35:40.903 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-05-31T21:35:40.903 INFO:tasks.rgw_multisite_tests:HEAD 2023-05-31T21:35:40.903 INFO:tasks.rgw_multisite_tests: 2023-05-31T21:35:40.903 INFO:tasks.rgw_multisite_tests: 2023-05-31T21:35:40.903 INFO:tasks.rgw_multisite_tests:Wed, 31 May 2023 20:29:25 GMT 2023-05-31T21:35:40.904 INFO:tasks.rgw_multisite_tests:/pklxkd-55/f?versionId=null 2023-05-31T21:35:40.904 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-05-31T21:35:40.904 INFO:tasks.rgw_multisite_tests:AWS Q80H11OLNU1S48PPYL26:pVjZQX3/jyGJnRQeh0D4CRVFdPg= 2023-05-31T21:35:40.904 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:29:25 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:pVjZQX3/jyGJnRQeh0D4CRVFdPg=', 'Content-Length': '0'} 2023-05-31T21:35:40.905 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:27:23 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c2cbbc0990208f8b-006477ae25-162e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:29:25 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:35:40.905 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/pklxkd-55/f 2023-05-31T21:35:40.905 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/pklxkd-55/f 2023-05-31T21:35:40.905 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/pklxkd-55/f?versionId=null 2023-05-31T21:35:40.906 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/pklxkd-55/f?versionId=null 2023-05-31T21:35:40.906 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-05-31T21:35:40.906 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /pklxkd-55/f?versionId=null 2023-05-31T21:35:40.906 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-05-31T21:35:40.906 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-05-31T21:35:40.907 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T21:35:40.907 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-05-31T21:35:40.907 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-05-31T21:35:40.907 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-05-31T21:35:40.908 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-05-31T21:35:40.908 INFO:tasks.rgw_multisite_tests:HEAD 2023-05-31T21:35:40.908 INFO:tasks.rgw_multisite_tests: 2023-05-31T21:35:40.908 INFO:tasks.rgw_multisite_tests: 2023-05-31T21:35:40.909 INFO:tasks.rgw_multisite_tests:Wed, 31 May 2023 20:29:25 GMT 2023-05-31T21:35:40.909 INFO:tasks.rgw_multisite_tests:/pklxkd-55/f?versionId=null 2023-05-31T21:35:40.909 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-05-31T21:35:40.909 INFO:tasks.rgw_multisite_tests:AWS Q80H11OLNU1S48PPYL26:pVjZQX3/jyGJnRQeh0D4CRVFdPg= 2023-05-31T21:35:40.909 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:29:25 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:pVjZQX3/jyGJnRQeh0D4CRVFdPg=', 'Content-Length': '0'} 2023-05-31T21:35:40.910 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:27:23 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f89d6e9fed21ab66-006477ae25-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:29:25 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:35:40.910 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: comparing key name=f 2023-05-31T21:35:40.910 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/pklxkd-55/f 2023-05-31T21:35:40.910 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/pklxkd-55/f 2023-05-31T21:35:40.911 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/pklxkd-55/f?versionId=null 2023-05-31T21:35:40.911 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/pklxkd-55/f?versionId=null 2023-05-31T21:35:40.911 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-05-31T21:35:40.911 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /pklxkd-55/f?versionId=null 2023-05-31T21:35:40.912 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-05-31T21:35:40.912 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-05-31T21:35:40.912 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T21:35:40.912 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-05-31T21:35:40.912 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-05-31T21:35:40.913 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-05-31T21:35:40.913 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-05-31T21:35:40.913 INFO:tasks.rgw_multisite_tests:GET 2023-05-31T21:35:40.913 INFO:tasks.rgw_multisite_tests: 2023-05-31T21:35:40.914 INFO:tasks.rgw_multisite_tests: 2023-05-31T21:35:40.914 INFO:tasks.rgw_multisite_tests:Wed, 31 May 2023 20:29:25 GMT 2023-05-31T21:35:40.914 INFO:tasks.rgw_multisite_tests:/pklxkd-55/f?versionId=null 2023-05-31T21:35:40.914 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-05-31T21:35:40.915 INFO:tasks.rgw_multisite_tests:AWS Q80H11OLNU1S48PPYL26:7DmZ2WsueqJzxqvgmHkyBb7pLCA= 2023-05-31T21:35:40.915 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:29:25 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:7DmZ2WsueqJzxqvgmHkyBb7pLCA=', 'Content-Length': '0'} 2023-05-31T21:35:40.915 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:27:23 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001afa07c63e413033-006477ae25-162e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:29:25 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:35:40.915 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/pklxkd-55/f 2023-05-31T21:35:40.915 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/pklxkd-55/f 2023-05-31T21:35:40.916 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/pklxkd-55/f?versionId=null 2023-05-31T21:35:40.916 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/pklxkd-55/f?versionId=null 2023-05-31T21:35:40.916 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-05-31T21:35:40.916 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /pklxkd-55/f?versionId=null 2023-05-31T21:35:40.917 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-05-31T21:35:40.917 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-05-31T21:35:40.917 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T21:35:40.917 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-05-31T21:35:40.918 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-05-31T21:35:40.918 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-05-31T21:35:40.918 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-05-31T21:35:40.918 INFO:tasks.rgw_multisite_tests:GET 2023-05-31T21:35:40.918 INFO:tasks.rgw_multisite_tests: 2023-05-31T21:35:40.919 INFO:tasks.rgw_multisite_tests: 2023-05-31T21:35:40.919 INFO:tasks.rgw_multisite_tests:Wed, 31 May 2023 20:29:25 GMT 2023-05-31T21:35:40.919 INFO:tasks.rgw_multisite_tests:/pklxkd-55/f?versionId=null 2023-05-31T21:35:40.919 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-05-31T21:35:40.920 INFO:tasks.rgw_multisite_tests:AWS Q80H11OLNU1S48PPYL26:7DmZ2WsueqJzxqvgmHkyBb7pLCA= 2023-05-31T21:35:40.920 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:29:25 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:7DmZ2WsueqJzxqvgmHkyBb7pLCA=', 'Content-Length': '0'} 2023-05-31T21:35:40.920 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:27:23 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000fe28f17f6b6de431-006477ae25-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:29:25 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:35:40.920 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/pklxkd-55/ 2023-05-31T21:35:40.921 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/pklxkd-55/ 2023-05-31T21:35:40.921 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/pklxkd-55/?prefix=f 2023-05-31T21:35:40.921 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/pklxkd-55/?prefix=f 2023-05-31T21:35:40.921 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-05-31T21:35:40.922 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /pklxkd-55/?prefix=f 2023-05-31T21:35:40.922 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-05-31T21:35:40.922 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-05-31T21:35:40.922 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T21:35:40.922 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-05-31T21:35:40.923 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-05-31T21:35:40.923 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-05-31T21:35:40.923 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-05-31T21:35:40.923 INFO:tasks.rgw_multisite_tests:GET 2023-05-31T21:35:40.924 INFO:tasks.rgw_multisite_tests: 2023-05-31T21:35:40.924 INFO:tasks.rgw_multisite_tests: 2023-05-31T21:35:40.924 INFO:tasks.rgw_multisite_tests:Wed, 31 May 2023 20:29:25 GMT 2023-05-31T21:35:40.924 INFO:tasks.rgw_multisite_tests:/pklxkd-55/ 2023-05-31T21:35:40.925 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-05-31T21:35:40.925 INFO:tasks.rgw_multisite_tests:AWS Q80H11OLNU1S48PPYL26:a0/qPp0bnZeH62hZJf6OA1a38dw= 2023-05-31T21:35:40.925 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:29:25 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:a0/qPp0bnZeH62hZJf6OA1a38dw=', 'Content-Length': '0'} 2023-05-31T21:35:40.925 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000001f0f7f30e89138ac-006477ae25-162e-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:29:25 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:35:40.926 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'pklxkd-55f1000falsef2023-05-31T20:27:23.977Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T21:35:40.926 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/pklxkd-55/ 2023-05-31T21:35:40.926 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/pklxkd-55/ 2023-05-31T21:35:40.926 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/pklxkd-55/?prefix=f 2023-05-31T21:35:40.927 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/pklxkd-55/?prefix=f 2023-05-31T21:35:40.927 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-05-31T21:35:40.927 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /pklxkd-55/?prefix=f 2023-05-31T21:35:40.927 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-05-31T21:35:40.927 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-05-31T21:35:40.928 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T21:35:40.928 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-05-31T21:35:40.928 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-05-31T21:35:40.928 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-05-31T21:35:40.929 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-05-31T21:35:40.929 INFO:tasks.rgw_multisite_tests:GET 2023-05-31T21:35:40.929 INFO:tasks.rgw_multisite_tests: 2023-05-31T21:35:40.929 INFO:tasks.rgw_multisite_tests: 2023-05-31T21:35:40.930 INFO:tasks.rgw_multisite_tests:Wed, 31 May 2023 20:29:25 GMT 2023-05-31T21:35:40.930 INFO:tasks.rgw_multisite_tests:/pklxkd-55/ 2023-05-31T21:35:40.930 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-05-31T21:35:40.930 INFO:tasks.rgw_multisite_tests:AWS Q80H11OLNU1S48PPYL26:a0/qPp0bnZeH62hZJf6OA1a38dw= 2023-05-31T21:35:40.930 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:29:25 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:a0/qPp0bnZeH62hZJf6OA1a38dw=', 'Content-Length': '0'} 2023-05-31T21:35:40.931 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000089745409c2a37fc7-006477ae25-1630-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:29:25 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:35:40.931 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'pklxkd-55f1000falsef2023-05-31T20:27:23.977Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T21:35:40.931 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/pklxkd-55/f 2023-05-31T21:35:40.931 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/pklxkd-55/f 2023-05-31T21:35:40.932 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-05-31T21:35:40.932 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /pklxkd-55/f 2023-05-31T21:35:40.932 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-05-31T21:35:40.932 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-05-31T21:35:40.933 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T21:35:40.933 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-05-31T21:35:40.933 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-05-31T21:35:40.933 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-05-31T21:35:40.933 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-05-31T21:35:40.934 INFO:tasks.rgw_multisite_tests:HEAD 2023-05-31T21:35:40.934 INFO:tasks.rgw_multisite_tests: 2023-05-31T21:35:40.934 INFO:tasks.rgw_multisite_tests: 2023-05-31T21:35:40.934 INFO:tasks.rgw_multisite_tests:Wed, 31 May 2023 20:29:25 GMT 2023-05-31T21:35:40.935 INFO:tasks.rgw_multisite_tests:/pklxkd-55/f 2023-05-31T21:35:40.935 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-05-31T21:35:40.935 INFO:tasks.rgw_multisite_tests:AWS Q80H11OLNU1S48PPYL26:EWNP/WSX9HAk9qDpuzjBhYIpfL0= 2023-05-31T21:35:40.935 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:29:25 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:EWNP/WSX9HAk9qDpuzjBhYIpfL0=', 'Content-Length': '0'} 2023-05-31T21:35:40.936 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:27:23 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000773bbf118a0cf21a-006477ae25-162e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:29:25 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:35:40.936 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/pklxkd-55/f 2023-05-31T21:35:40.936 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/pklxkd-55/f 2023-05-31T21:35:40.936 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-05-31T21:35:40.936 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /pklxkd-55/f 2023-05-31T21:35:40.937 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-05-31T21:35:40.937 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-05-31T21:35:40.937 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T21:35:40.937 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-05-31T21:35:40.938 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-05-31T21:35:40.938 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-05-31T21:35:40.938 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-05-31T21:35:40.938 INFO:tasks.rgw_multisite_tests:HEAD 2023-05-31T21:35:40.939 INFO:tasks.rgw_multisite_tests: 2023-05-31T21:35:40.939 INFO:tasks.rgw_multisite_tests: 2023-05-31T21:35:40.939 INFO:tasks.rgw_multisite_tests:Wed, 31 May 2023 20:29:25 GMT 2023-05-31T21:35:40.939 INFO:tasks.rgw_multisite_tests:/pklxkd-55/f 2023-05-31T21:35:40.939 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-05-31T21:35:40.940 INFO:tasks.rgw_multisite_tests:AWS Q80H11OLNU1S48PPYL26:EWNP/WSX9HAk9qDpuzjBhYIpfL0= 2023-05-31T21:35:40.940 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:29:25 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:EWNP/WSX9HAk9qDpuzjBhYIpfL0=', 'Content-Length': '0'} 2023-05-31T21:35:40.940 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:27:23 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003a6f026eb2564052-006477ae25-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:29:25 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:35:40.940 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: comparing key name=f 2023-05-31T21:35:40.941 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/pklxkd-55/f 2023-05-31T21:35:40.941 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/pklxkd-55/f 2023-05-31T21:35:40.941 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-05-31T21:35:40.941 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /pklxkd-55/f 2023-05-31T21:35:40.942 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-05-31T21:35:40.942 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-05-31T21:35:40.942 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T21:35:40.942 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-05-31T21:35:40.942 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-05-31T21:35:40.943 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-05-31T21:35:40.943 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-05-31T21:35:40.943 INFO:tasks.rgw_multisite_tests:GET 2023-05-31T21:35:40.943 INFO:tasks.rgw_multisite_tests: 2023-05-31T21:35:40.944 INFO:tasks.rgw_multisite_tests: 2023-05-31T21:35:40.944 INFO:tasks.rgw_multisite_tests:Wed, 31 May 2023 20:29:25 GMT 2023-05-31T21:35:40.944 INFO:tasks.rgw_multisite_tests:/pklxkd-55/f 2023-05-31T21:35:40.944 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-05-31T21:35:40.945 INFO:tasks.rgw_multisite_tests:AWS Q80H11OLNU1S48PPYL26:fS1d9jC0vfS2r1wK4J1RO/REDy4= 2023-05-31T21:35:40.945 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:29:25 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:fS1d9jC0vfS2r1wK4J1RO/REDy4=', 'Content-Length': '0'} 2023-05-31T21:35:40.945 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:27:23 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000007dee78956ea5252-006477ae25-162e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:29:25 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:35:40.945 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/pklxkd-55/f 2023-05-31T21:35:40.945 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/pklxkd-55/f 2023-05-31T21:35:40.946 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-05-31T21:35:40.946 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /pklxkd-55/f 2023-05-31T21:35:40.946 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-05-31T21:35:40.946 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-05-31T21:35:40.947 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T21:35:40.947 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-05-31T21:35:40.947 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-05-31T21:35:40.947 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-05-31T21:35:40.948 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-05-31T21:35:40.948 INFO:tasks.rgw_multisite_tests:GET 2023-05-31T21:35:40.948 INFO:tasks.rgw_multisite_tests: 2023-05-31T21:35:40.948 INFO:tasks.rgw_multisite_tests: 2023-05-31T21:35:40.948 INFO:tasks.rgw_multisite_tests:Wed, 31 May 2023 20:29:25 GMT 2023-05-31T21:35:40.949 INFO:tasks.rgw_multisite_tests:/pklxkd-55/f 2023-05-31T21:35:40.949 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-05-31T21:35:40.949 INFO:tasks.rgw_multisite_tests:AWS Q80H11OLNU1S48PPYL26:fS1d9jC0vfS2r1wK4J1RO/REDy4= 2023-05-31T21:35:40.949 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:29:25 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:fS1d9jC0vfS2r1wK4J1RO/REDy4=', 'Content-Length': '0'} 2023-05-31T21:35:40.950 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:27:23 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000092557c5a0b5a61a8-006477ae25-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:29:25 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:35:40.950 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/pklxkd-55/ 2023-05-31T21:35:40.950 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/pklxkd-55/ 2023-05-31T21:35:40.950 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/pklxkd-55/?prefix=f 2023-05-31T21:35:40.951 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/pklxkd-55/?prefix=f 2023-05-31T21:35:40.951 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-05-31T21:35:40.951 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /pklxkd-55/?prefix=f 2023-05-31T21:35:40.951 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-05-31T21:35:40.951 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-05-31T21:35:40.952 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T21:35:40.952 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-05-31T21:35:40.952 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-05-31T21:35:40.952 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-05-31T21:35:40.953 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-05-31T21:35:40.953 INFO:tasks.rgw_multisite_tests:GET 2023-05-31T21:35:40.953 INFO:tasks.rgw_multisite_tests: 2023-05-31T21:35:40.953 INFO:tasks.rgw_multisite_tests: 2023-05-31T21:35:40.954 INFO:tasks.rgw_multisite_tests:Wed, 31 May 2023 20:29:25 GMT 2023-05-31T21:35:40.954 INFO:tasks.rgw_multisite_tests:/pklxkd-55/ 2023-05-31T21:35:40.954 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-05-31T21:35:40.954 INFO:tasks.rgw_multisite_tests:AWS Q80H11OLNU1S48PPYL26:a0/qPp0bnZeH62hZJf6OA1a38dw= 2023-05-31T21:35:40.954 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:29:25 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:a0/qPp0bnZeH62hZJf6OA1a38dw=', 'Content-Length': '0'} 2023-05-31T21:35:40.955 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000835cae37e5f314d3-006477ae25-162e-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:29:25 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:35:40.955 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'pklxkd-55f1000falsef2023-05-31T20:27:23.977Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T21:35:40.955 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/pklxkd-55/ 2023-05-31T21:35:40.955 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/pklxkd-55/ 2023-05-31T21:35:40.956 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/pklxkd-55/?prefix=f 2023-05-31T21:35:40.956 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/pklxkd-55/?prefix=f 2023-05-31T21:35:40.956 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-05-31T21:35:40.956 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /pklxkd-55/?prefix=f 2023-05-31T21:35:40.957 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-05-31T21:35:40.957 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-05-31T21:35:40.957 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T21:35:40.957 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-05-31T21:35:40.958 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-05-31T21:35:40.958 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-05-31T21:35:40.958 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-05-31T21:35:40.958 INFO:tasks.rgw_multisite_tests:GET 2023-05-31T21:35:40.958 INFO:tasks.rgw_multisite_tests: 2023-05-31T21:35:40.959 INFO:tasks.rgw_multisite_tests: 2023-05-31T21:35:40.959 INFO:tasks.rgw_multisite_tests:Wed, 31 May 2023 20:29:25 GMT 2023-05-31T21:35:40.959 INFO:tasks.rgw_multisite_tests:/pklxkd-55/ 2023-05-31T21:35:40.959 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-05-31T21:35:40.960 INFO:tasks.rgw_multisite_tests:AWS Q80H11OLNU1S48PPYL26:a0/qPp0bnZeH62hZJf6OA1a38dw= 2023-05-31T21:35:40.960 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:29:25 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:a0/qPp0bnZeH62hZJf6OA1a38dw=', 'Content-Length': '0'} 2023-05-31T21:35:40.960 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e5af12247ae7a9c8-006477ae25-1630-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:29:25 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:35:40.960 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'pklxkd-55f1000falsef2023-05-31T20:27:23.977Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T21:35:40.961 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: comparing key name=g 2023-05-31T21:35:40.961 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/pklxkd-55/g 2023-05-31T21:35:40.961 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/pklxkd-55/g 2023-05-31T21:35:40.961 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/pklxkd-55/g?versionId=null 2023-05-31T21:35:40.961 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/pklxkd-55/g?versionId=null 2023-05-31T21:35:40.962 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-05-31T21:35:40.962 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /pklxkd-55/g?versionId=null 2023-05-31T21:35:40.962 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-05-31T21:35:40.962 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-05-31T21:35:40.963 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T21:35:40.963 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-05-31T21:35:40.963 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-05-31T21:35:40.963 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-05-31T21:35:40.964 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-05-31T21:35:40.964 INFO:tasks.rgw_multisite_tests:GET 2023-05-31T21:35:40.964 INFO:tasks.rgw_multisite_tests: 2023-05-31T21:35:40.964 INFO:tasks.rgw_multisite_tests: 2023-05-31T21:35:40.964 INFO:tasks.rgw_multisite_tests:Wed, 31 May 2023 20:29:25 GMT 2023-05-31T21:35:40.965 INFO:tasks.rgw_multisite_tests:/pklxkd-55/g?versionId=null 2023-05-31T21:35:40.965 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-05-31T21:35:40.965 INFO:tasks.rgw_multisite_tests:AWS Q80H11OLNU1S48PPYL26:qsnSCM4kslUB64GjrB2v0KYUJLc= 2023-05-31T21:35:40.965 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:29:25 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:qsnSCM4kslUB64GjrB2v0KYUJLc=', 'Content-Length': '0'} 2023-05-31T21:35:40.966 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:27:24 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002dd250dede2e2fc2-006477ae25-162e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:29:25 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:35:40.966 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/pklxkd-55/g 2023-05-31T21:35:40.966 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/pklxkd-55/g 2023-05-31T21:35:40.966 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/pklxkd-55/g?versionId=null 2023-05-31T21:35:40.967 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/pklxkd-55/g?versionId=null 2023-05-31T21:35:40.967 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-05-31T21:35:40.967 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /pklxkd-55/g?versionId=null 2023-05-31T21:35:40.967 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-05-31T21:35:40.967 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-05-31T21:35:40.968 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T21:35:40.968 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-05-31T21:35:40.968 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-05-31T21:35:40.968 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-05-31T21:35:40.969 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-05-31T21:35:40.969 INFO:tasks.rgw_multisite_tests:GET 2023-05-31T21:35:40.969 INFO:tasks.rgw_multisite_tests: 2023-05-31T21:35:40.969 INFO:tasks.rgw_multisite_tests: 2023-05-31T21:35:40.970 INFO:tasks.rgw_multisite_tests:Wed, 31 May 2023 20:29:25 GMT 2023-05-31T21:35:40.970 INFO:tasks.rgw_multisite_tests:/pklxkd-55/g?versionId=null 2023-05-31T21:35:40.970 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-05-31T21:35:40.970 INFO:tasks.rgw_multisite_tests:AWS Q80H11OLNU1S48PPYL26:qsnSCM4kslUB64GjrB2v0KYUJLc= 2023-05-31T21:35:40.970 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:29:25 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:qsnSCM4kslUB64GjrB2v0KYUJLc=', 'Content-Length': '0'} 2023-05-31T21:35:40.971 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:27:24 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000043cf1be21524017a-006477ae25-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:29:25 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:35:40.971 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/pklxkd-55/g 2023-05-31T21:35:40.971 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/pklxkd-55/g 2023-05-31T21:35:40.971 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/pklxkd-55/g?versionId=null 2023-05-31T21:35:40.972 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/pklxkd-55/g?versionId=null 2023-05-31T21:35:40.972 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-05-31T21:35:40.972 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /pklxkd-55/g?versionId=null 2023-05-31T21:35:40.972 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-05-31T21:35:40.973 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-05-31T21:35:40.973 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T21:35:40.973 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-05-31T21:35:40.973 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-05-31T21:35:40.973 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-05-31T21:35:40.974 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-05-31T21:35:40.974 INFO:tasks.rgw_multisite_tests:HEAD 2023-05-31T21:35:40.974 INFO:tasks.rgw_multisite_tests: 2023-05-31T21:35:40.974 INFO:tasks.rgw_multisite_tests: 2023-05-31T21:35:40.975 INFO:tasks.rgw_multisite_tests:Wed, 31 May 2023 20:29:25 GMT 2023-05-31T21:35:40.975 INFO:tasks.rgw_multisite_tests:/pklxkd-55/g?versionId=null 2023-05-31T21:35:40.975 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-05-31T21:35:40.975 INFO:tasks.rgw_multisite_tests:AWS Q80H11OLNU1S48PPYL26:KE9s1g24EIbZhx5Yr+qs+NeHbco= 2023-05-31T21:35:40.976 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:29:25 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:KE9s1g24EIbZhx5Yr+qs+NeHbco=', 'Content-Length': '0'} 2023-05-31T21:35:40.976 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:27:24 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a64ebb15dc80cc7d-006477ae25-162e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:29:25 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:35:40.976 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/pklxkd-55/g 2023-05-31T21:35:40.976 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/pklxkd-55/g 2023-05-31T21:35:40.976 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/pklxkd-55/g?versionId=null 2023-05-31T21:35:40.977 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/pklxkd-55/g?versionId=null 2023-05-31T21:35:40.977 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-05-31T21:35:40.977 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /pklxkd-55/g?versionId=null 2023-05-31T21:35:40.977 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-05-31T21:35:40.978 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-05-31T21:35:40.978 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T21:35:40.978 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-05-31T21:35:40.978 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-05-31T21:35:40.979 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-05-31T21:35:40.979 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-05-31T21:35:40.979 INFO:tasks.rgw_multisite_tests:HEAD 2023-05-31T21:35:40.979 INFO:tasks.rgw_multisite_tests: 2023-05-31T21:35:40.980 INFO:tasks.rgw_multisite_tests: 2023-05-31T21:35:40.980 INFO:tasks.rgw_multisite_tests:Wed, 31 May 2023 20:29:25 GMT 2023-05-31T21:35:40.980 INFO:tasks.rgw_multisite_tests:/pklxkd-55/g?versionId=null 2023-05-31T21:35:40.980 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-05-31T21:35:40.981 INFO:tasks.rgw_multisite_tests:AWS Q80H11OLNU1S48PPYL26:KE9s1g24EIbZhx5Yr+qs+NeHbco= 2023-05-31T21:35:40.981 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:29:25 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:KE9s1g24EIbZhx5Yr+qs+NeHbco=', 'Content-Length': '0'} 2023-05-31T21:35:40.981 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:27:24 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000098106713d97f6f4d-006477ae25-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:29:25 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:35:40.981 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: comparing key name=g 2023-05-31T21:35:40.982 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/pklxkd-55/g 2023-05-31T21:35:40.982 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/pklxkd-55/g 2023-05-31T21:35:40.982 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/pklxkd-55/g?versionId=null 2023-05-31T21:35:40.982 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/pklxkd-55/g?versionId=null 2023-05-31T21:35:40.983 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-05-31T21:35:40.983 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /pklxkd-55/g?versionId=null 2023-05-31T21:35:40.983 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-05-31T21:35:40.983 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-05-31T21:35:40.983 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T21:35:40.984 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-05-31T21:35:40.984 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-05-31T21:35:40.984 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-05-31T21:35:40.984 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-05-31T21:35:40.985 INFO:tasks.rgw_multisite_tests:GET 2023-05-31T21:35:40.985 INFO:tasks.rgw_multisite_tests: 2023-05-31T21:35:40.985 INFO:tasks.rgw_multisite_tests: 2023-05-31T21:35:40.985 INFO:tasks.rgw_multisite_tests:Wed, 31 May 2023 20:29:25 GMT 2023-05-31T21:35:40.986 INFO:tasks.rgw_multisite_tests:/pklxkd-55/g?versionId=null 2023-05-31T21:35:40.986 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-05-31T21:35:40.986 INFO:tasks.rgw_multisite_tests:AWS Q80H11OLNU1S48PPYL26:qsnSCM4kslUB64GjrB2v0KYUJLc= 2023-05-31T21:35:40.986 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:29:25 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:qsnSCM4kslUB64GjrB2v0KYUJLc=', 'Content-Length': '0'} 2023-05-31T21:35:40.986 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:27:24 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a02e02d558c2820f-006477ae25-162e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:29:25 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:35:40.987 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/pklxkd-55/g 2023-05-31T21:35:40.987 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/pklxkd-55/g 2023-05-31T21:35:40.987 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/pklxkd-55/g?versionId=null 2023-05-31T21:35:40.987 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/pklxkd-55/g?versionId=null 2023-05-31T21:35:40.988 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-05-31T21:35:40.988 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /pklxkd-55/g?versionId=null 2023-05-31T21:35:40.988 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-05-31T21:35:40.988 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-05-31T21:35:40.989 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T21:35:40.989 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-05-31T21:35:40.989 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-05-31T21:35:40.989 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-05-31T21:35:40.990 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-05-31T21:35:40.990 INFO:tasks.rgw_multisite_tests:GET 2023-05-31T21:35:40.990 INFO:tasks.rgw_multisite_tests: 2023-05-31T21:35:40.990 INFO:tasks.rgw_multisite_tests: 2023-05-31T21:35:40.990 INFO:tasks.rgw_multisite_tests:Wed, 31 May 2023 20:29:25 GMT 2023-05-31T21:35:40.991 INFO:tasks.rgw_multisite_tests:/pklxkd-55/g?versionId=null 2023-05-31T21:35:40.991 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-05-31T21:35:40.991 INFO:tasks.rgw_multisite_tests:AWS Q80H11OLNU1S48PPYL26:qsnSCM4kslUB64GjrB2v0KYUJLc= 2023-05-31T21:35:40.991 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:29:25 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:qsnSCM4kslUB64GjrB2v0KYUJLc=', 'Content-Length': '0'} 2023-05-31T21:35:40.992 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:27:24 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ece4a52bfaa107d8-006477ae25-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:29:25 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:35:40.992 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/pklxkd-55/ 2023-05-31T21:35:40.992 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/pklxkd-55/ 2023-05-31T21:35:40.992 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/pklxkd-55/?prefix=g 2023-05-31T21:35:40.993 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/pklxkd-55/?prefix=g 2023-05-31T21:35:40.993 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-05-31T21:35:40.993 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /pklxkd-55/?prefix=g 2023-05-31T21:35:40.993 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-05-31T21:35:40.993 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-05-31T21:35:40.994 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T21:35:40.994 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-05-31T21:35:40.994 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-05-31T21:35:40.994 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-05-31T21:35:40.995 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-05-31T21:35:40.995 INFO:tasks.rgw_multisite_tests:GET 2023-05-31T21:35:40.995 INFO:tasks.rgw_multisite_tests: 2023-05-31T21:35:40.995 INFO:tasks.rgw_multisite_tests: 2023-05-31T21:35:40.996 INFO:tasks.rgw_multisite_tests:Wed, 31 May 2023 20:29:25 GMT 2023-05-31T21:35:40.996 INFO:tasks.rgw_multisite_tests:/pklxkd-55/ 2023-05-31T21:35:40.996 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-05-31T21:35:40.996 INFO:tasks.rgw_multisite_tests:AWS Q80H11OLNU1S48PPYL26:a0/qPp0bnZeH62hZJf6OA1a38dw= 2023-05-31T21:35:40.996 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:29:25 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:a0/qPp0bnZeH62hZJf6OA1a38dw=', 'Content-Length': '0'} 2023-05-31T21:35:40.997 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000213cfb4550359655-006477ae25-162e-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:29:25 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:35:40.997 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'pklxkd-55g1000falseg2023-05-31T20:27:24.035Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T21:35:40.997 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/pklxkd-55/ 2023-05-31T21:35:40.997 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/pklxkd-55/ 2023-05-31T21:35:40.998 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/pklxkd-55/?prefix=g 2023-05-31T21:35:40.998 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/pklxkd-55/?prefix=g 2023-05-31T21:35:40.998 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-05-31T21:35:40.998 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /pklxkd-55/?prefix=g 2023-05-31T21:35:40.999 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-05-31T21:35:40.999 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-05-31T21:35:40.999 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T21:35:40.999 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-05-31T21:35:40.999 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-05-31T21:35:41.000 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-05-31T21:35:41.000 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-05-31T21:35:41.000 INFO:tasks.rgw_multisite_tests:GET 2023-05-31T21:35:41.000 INFO:tasks.rgw_multisite_tests: 2023-05-31T21:35:41.001 INFO:tasks.rgw_multisite_tests: 2023-05-31T21:35:41.001 INFO:tasks.rgw_multisite_tests:Wed, 31 May 2023 20:29:25 GMT 2023-05-31T21:35:41.001 INFO:tasks.rgw_multisite_tests:/pklxkd-55/ 2023-05-31T21:35:41.001 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-05-31T21:35:41.002 INFO:tasks.rgw_multisite_tests:AWS Q80H11OLNU1S48PPYL26:a0/qPp0bnZeH62hZJf6OA1a38dw= 2023-05-31T21:35:41.002 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:29:25 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:a0/qPp0bnZeH62hZJf6OA1a38dw=', 'Content-Length': '0'} 2023-05-31T21:35:41.002 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000090a5490a6429dc72-006477ae25-1630-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:29:25 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:35:41.002 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'pklxkd-55g1000falseg2023-05-31T20:27:24.035Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T21:35:41.002 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/pklxkd-55/g 2023-05-31T21:35:41.003 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/pklxkd-55/g 2023-05-31T21:35:41.003 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-05-31T21:35:41.003 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /pklxkd-55/g 2023-05-31T21:35:41.003 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-05-31T21:35:41.004 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-05-31T21:35:41.004 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T21:35:41.004 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-05-31T21:35:41.004 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-05-31T21:35:41.005 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-05-31T21:35:41.005 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-05-31T21:35:41.005 INFO:tasks.rgw_multisite_tests:HEAD 2023-05-31T21:35:41.005 INFO:tasks.rgw_multisite_tests: 2023-05-31T21:35:41.005 INFO:tasks.rgw_multisite_tests: 2023-05-31T21:35:41.006 INFO:tasks.rgw_multisite_tests:Wed, 31 May 2023 20:29:25 GMT 2023-05-31T21:35:41.006 INFO:tasks.rgw_multisite_tests:/pklxkd-55/g 2023-05-31T21:35:41.006 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-05-31T21:35:41.006 INFO:tasks.rgw_multisite_tests:AWS Q80H11OLNU1S48PPYL26:98XyMXADg1VnamqbYsz7p83l8vU= 2023-05-31T21:35:41.007 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:29:25 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:98XyMXADg1VnamqbYsz7p83l8vU=', 'Content-Length': '0'} 2023-05-31T21:35:41.007 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:27:24 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002a747944dfba0bca-006477ae25-162e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:29:25 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:35:41.007 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/pklxkd-55/g 2023-05-31T21:35:41.007 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/pklxkd-55/g 2023-05-31T21:35:41.008 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-05-31T21:35:41.008 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /pklxkd-55/g 2023-05-31T21:35:41.008 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-05-31T21:35:41.008 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-05-31T21:35:41.008 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T21:35:41.009 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-05-31T21:35:41.009 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-05-31T21:35:41.009 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-05-31T21:35:41.009 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-05-31T21:35:41.010 INFO:tasks.rgw_multisite_tests:HEAD 2023-05-31T21:35:41.010 INFO:tasks.rgw_multisite_tests: 2023-05-31T21:35:41.010 INFO:tasks.rgw_multisite_tests: 2023-05-31T21:35:41.010 INFO:tasks.rgw_multisite_tests:Wed, 31 May 2023 20:29:25 GMT 2023-05-31T21:35:41.011 INFO:tasks.rgw_multisite_tests:/pklxkd-55/g 2023-05-31T21:35:41.011 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-05-31T21:35:41.011 INFO:tasks.rgw_multisite_tests:AWS Q80H11OLNU1S48PPYL26:98XyMXADg1VnamqbYsz7p83l8vU= 2023-05-31T21:35:41.011 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:29:25 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:98XyMXADg1VnamqbYsz7p83l8vU=', 'Content-Length': '0'} 2023-05-31T21:35:41.011 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:27:24 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000410c8de93a4089ab-006477ae25-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:29:25 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:35:41.012 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: comparing key name=g 2023-05-31T21:35:41.012 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/pklxkd-55/g 2023-05-31T21:35:41.012 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/pklxkd-55/g 2023-05-31T21:35:41.012 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-05-31T21:35:41.013 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /pklxkd-55/g 2023-05-31T21:35:41.013 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-05-31T21:35:41.013 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-05-31T21:35:41.013 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T21:35:41.013 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-05-31T21:35:41.014 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-05-31T21:35:41.014 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-05-31T21:35:41.014 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-05-31T21:35:41.014 INFO:tasks.rgw_multisite_tests:GET 2023-05-31T21:35:41.015 INFO:tasks.rgw_multisite_tests: 2023-05-31T21:35:41.015 INFO:tasks.rgw_multisite_tests: 2023-05-31T21:35:41.015 INFO:tasks.rgw_multisite_tests:Wed, 31 May 2023 20:29:25 GMT 2023-05-31T21:35:41.015 INFO:tasks.rgw_multisite_tests:/pklxkd-55/g 2023-05-31T21:35:41.016 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-05-31T21:35:41.016 INFO:tasks.rgw_multisite_tests:AWS Q80H11OLNU1S48PPYL26:9a77YCWe+i1di4u03C/zuiH2Td8= 2023-05-31T21:35:41.016 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:29:25 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:9a77YCWe+i1di4u03C/zuiH2Td8=', 'Content-Length': '0'} 2023-05-31T21:35:41.016 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:27:24 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001a8147bcd114d078-006477ae25-162e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:29:25 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:35:41.017 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/pklxkd-55/g 2023-05-31T21:35:41.017 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/pklxkd-55/g 2023-05-31T21:35:41.017 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-05-31T21:35:41.017 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /pklxkd-55/g 2023-05-31T21:35:41.017 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-05-31T21:35:41.018 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-05-31T21:35:41.018 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T21:35:41.018 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-05-31T21:35:41.018 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-05-31T21:35:41.019 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-05-31T21:35:41.019 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-05-31T21:35:41.019 INFO:tasks.rgw_multisite_tests:GET 2023-05-31T21:35:41.019 INFO:tasks.rgw_multisite_tests: 2023-05-31T21:35:41.019 INFO:tasks.rgw_multisite_tests: 2023-05-31T21:35:41.020 INFO:tasks.rgw_multisite_tests:Wed, 31 May 2023 20:29:25 GMT 2023-05-31T21:35:41.020 INFO:tasks.rgw_multisite_tests:/pklxkd-55/g 2023-05-31T21:35:41.020 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-05-31T21:35:41.020 INFO:tasks.rgw_multisite_tests:AWS Q80H11OLNU1S48PPYL26:9a77YCWe+i1di4u03C/zuiH2Td8= 2023-05-31T21:35:41.021 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:29:25 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:9a77YCWe+i1di4u03C/zuiH2Td8=', 'Content-Length': '0'} 2023-05-31T21:35:41.021 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:27:24 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000044cf4c2b77e79440-006477ae25-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:29:25 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:35:41.021 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/pklxkd-55/ 2023-05-31T21:35:41.021 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/pklxkd-55/ 2023-05-31T21:35:41.022 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/pklxkd-55/?prefix=g 2023-05-31T21:35:41.022 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/pklxkd-55/?prefix=g 2023-05-31T21:35:41.022 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-05-31T21:35:41.022 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /pklxkd-55/?prefix=g 2023-05-31T21:35:41.023 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-05-31T21:35:41.023 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-05-31T21:35:41.023 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T21:35:41.023 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-05-31T21:35:41.024 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-05-31T21:35:41.024 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-05-31T21:35:41.024 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-05-31T21:35:41.024 INFO:tasks.rgw_multisite_tests:GET 2023-05-31T21:35:41.024 INFO:tasks.rgw_multisite_tests: 2023-05-31T21:35:41.025 INFO:tasks.rgw_multisite_tests: 2023-05-31T21:35:41.025 INFO:tasks.rgw_multisite_tests:Wed, 31 May 2023 20:29:25 GMT 2023-05-31T21:35:41.025 INFO:tasks.rgw_multisite_tests:/pklxkd-55/ 2023-05-31T21:35:41.025 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-05-31T21:35:41.026 INFO:tasks.rgw_multisite_tests:AWS Q80H11OLNU1S48PPYL26:a0/qPp0bnZeH62hZJf6OA1a38dw= 2023-05-31T21:35:41.026 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:29:25 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:a0/qPp0bnZeH62hZJf6OA1a38dw=', 'Content-Length': '0'} 2023-05-31T21:35:41.026 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000092555a0ff9c49a44-006477ae25-162e-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:29:25 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:35:41.026 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'pklxkd-55g1000falseg2023-05-31T20:27:24.035Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T21:35:41.027 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/pklxkd-55/ 2023-05-31T21:35:41.027 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/pklxkd-55/ 2023-05-31T21:35:41.027 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/pklxkd-55/?prefix=g 2023-05-31T21:35:41.027 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/pklxkd-55/?prefix=g 2023-05-31T21:35:41.027 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-05-31T21:35:41.028 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /pklxkd-55/?prefix=g 2023-05-31T21:35:41.028 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-05-31T21:35:41.028 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-05-31T21:35:41.028 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T21:35:41.029 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-05-31T21:35:41.029 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-05-31T21:35:41.029 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-05-31T21:35:41.029 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-05-31T21:35:41.030 INFO:tasks.rgw_multisite_tests:GET 2023-05-31T21:35:41.030 INFO:tasks.rgw_multisite_tests: 2023-05-31T21:35:41.030 INFO:tasks.rgw_multisite_tests: 2023-05-31T21:35:41.030 INFO:tasks.rgw_multisite_tests:Wed, 31 May 2023 20:29:25 GMT 2023-05-31T21:35:41.030 INFO:tasks.rgw_multisite_tests:/pklxkd-55/ 2023-05-31T21:35:41.031 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-05-31T21:35:41.031 INFO:tasks.rgw_multisite_tests:AWS Q80H11OLNU1S48PPYL26:a0/qPp0bnZeH62hZJf6OA1a38dw= 2023-05-31T21:35:41.031 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:29:25 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:a0/qPp0bnZeH62hZJf6OA1a38dw=', 'Content-Length': '0'} 2023-05-31T21:35:41.031 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000004dd3d6fce640f28f-006477ae25-1630-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:29:25 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:35:41.032 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'pklxkd-55g1000falseg2023-05-31T20:27:24.035Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T21:35:41.032 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: comparing key name=h 2023-05-31T21:35:41.032 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/pklxkd-55/h 2023-05-31T21:35:41.032 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/pklxkd-55/h 2023-05-31T21:35:41.033 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/pklxkd-55/h?versionId=null 2023-05-31T21:35:41.033 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/pklxkd-55/h?versionId=null 2023-05-31T21:35:41.033 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-05-31T21:35:41.033 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /pklxkd-55/h?versionId=null 2023-05-31T21:35:41.034 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-05-31T21:35:41.034 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-05-31T21:35:41.034 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T21:35:41.034 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-05-31T21:35:41.035 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-05-31T21:35:41.035 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-05-31T21:35:41.035 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-05-31T21:35:41.035 INFO:tasks.rgw_multisite_tests:GET 2023-05-31T21:35:41.035 INFO:tasks.rgw_multisite_tests: 2023-05-31T21:35:41.036 INFO:tasks.rgw_multisite_tests: 2023-05-31T21:35:41.036 INFO:tasks.rgw_multisite_tests:Wed, 31 May 2023 20:29:25 GMT 2023-05-31T21:35:41.036 INFO:tasks.rgw_multisite_tests:/pklxkd-55/h?versionId=null 2023-05-31T21:35:41.036 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-05-31T21:35:41.037 INFO:tasks.rgw_multisite_tests:AWS Q80H11OLNU1S48PPYL26:4Y/8L8lDnhHI55GrOpuluMNFBUo= 2023-05-31T21:35:41.037 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:29:25 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:4Y/8L8lDnhHI55GrOpuluMNFBUo=', 'Content-Length': '0'} 2023-05-31T21:35:41.037 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:27:24 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000068bb8c8ebf25405c-006477ae25-162e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:29:25 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:35:41.037 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/pklxkd-55/h 2023-05-31T21:35:41.038 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/pklxkd-55/h 2023-05-31T21:35:41.038 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/pklxkd-55/h?versionId=null 2023-05-31T21:35:41.038 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/pklxkd-55/h?versionId=null 2023-05-31T21:35:41.038 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-05-31T21:35:41.038 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /pklxkd-55/h?versionId=null 2023-05-31T21:35:41.039 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-05-31T21:35:41.039 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-05-31T21:35:41.039 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T21:35:41.039 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-05-31T21:35:41.040 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-05-31T21:35:41.040 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-05-31T21:35:41.040 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-05-31T21:35:41.040 INFO:tasks.rgw_multisite_tests:GET 2023-05-31T21:35:41.041 INFO:tasks.rgw_multisite_tests: 2023-05-31T21:35:41.041 INFO:tasks.rgw_multisite_tests: 2023-05-31T21:35:41.041 INFO:tasks.rgw_multisite_tests:Wed, 31 May 2023 20:29:25 GMT 2023-05-31T21:35:41.041 INFO:tasks.rgw_multisite_tests:/pklxkd-55/h?versionId=null 2023-05-31T21:35:41.041 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-05-31T21:35:41.042 INFO:tasks.rgw_multisite_tests:AWS Q80H11OLNU1S48PPYL26:4Y/8L8lDnhHI55GrOpuluMNFBUo= 2023-05-31T21:35:41.042 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:29:25 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:4Y/8L8lDnhHI55GrOpuluMNFBUo=', 'Content-Length': '0'} 2023-05-31T21:35:41.042 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:27:24 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ec20a36b751303ea-006477ae25-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:29:25 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:35:41.042 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/pklxkd-55/h 2023-05-31T21:35:41.043 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/pklxkd-55/h 2023-05-31T21:35:41.043 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/pklxkd-55/h?versionId=null 2023-05-31T21:35:41.043 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/pklxkd-55/h?versionId=null 2023-05-31T21:35:41.043 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-05-31T21:35:41.044 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /pklxkd-55/h?versionId=null 2023-05-31T21:35:41.044 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-05-31T21:35:41.044 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-05-31T21:35:41.044 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T21:35:41.044 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-05-31T21:35:41.045 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-05-31T21:35:41.045 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-05-31T21:35:41.045 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-05-31T21:35:41.045 INFO:tasks.rgw_multisite_tests:HEAD 2023-05-31T21:35:41.046 INFO:tasks.rgw_multisite_tests: 2023-05-31T21:35:41.046 INFO:tasks.rgw_multisite_tests: 2023-05-31T21:35:41.046 INFO:tasks.rgw_multisite_tests:Wed, 31 May 2023 20:29:25 GMT 2023-05-31T21:35:41.046 INFO:tasks.rgw_multisite_tests:/pklxkd-55/h?versionId=null 2023-05-31T21:35:41.047 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-05-31T21:35:41.047 INFO:tasks.rgw_multisite_tests:AWS Q80H11OLNU1S48PPYL26:4efssCLMCr2E9Yl7sdCyB23wbtw= 2023-05-31T21:35:41.047 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:29:25 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:4efssCLMCr2E9Yl7sdCyB23wbtw=', 'Content-Length': '0'} 2023-05-31T21:35:41.047 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:27:24 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000033b93ee33ebb509d-006477ae25-162e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:29:25 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:35:41.048 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/pklxkd-55/h 2023-05-31T21:35:41.048 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/pklxkd-55/h 2023-05-31T21:35:41.048 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/pklxkd-55/h?versionId=null 2023-05-31T21:35:41.048 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/pklxkd-55/h?versionId=null 2023-05-31T21:35:41.048 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-05-31T21:35:41.049 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /pklxkd-55/h?versionId=null 2023-05-31T21:35:41.049 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-05-31T21:35:41.049 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-05-31T21:35:41.049 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T21:35:41.050 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-05-31T21:35:41.050 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-05-31T21:35:41.050 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-05-31T21:35:41.050 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-05-31T21:35:41.051 INFO:tasks.rgw_multisite_tests:HEAD 2023-05-31T21:35:41.051 INFO:tasks.rgw_multisite_tests: 2023-05-31T21:35:41.051 INFO:tasks.rgw_multisite_tests: 2023-05-31T21:35:41.051 INFO:tasks.rgw_multisite_tests:Wed, 31 May 2023 20:29:25 GMT 2023-05-31T21:35:41.051 INFO:tasks.rgw_multisite_tests:/pklxkd-55/h?versionId=null 2023-05-31T21:35:41.052 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-05-31T21:35:41.052 INFO:tasks.rgw_multisite_tests:AWS Q80H11OLNU1S48PPYL26:4efssCLMCr2E9Yl7sdCyB23wbtw= 2023-05-31T21:35:41.052 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:29:25 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:4efssCLMCr2E9Yl7sdCyB23wbtw=', 'Content-Length': '0'} 2023-05-31T21:35:41.052 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:27:24 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002b74576e185f3faa-006477ae25-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:29:25 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:35:41.053 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: comparing key name=h 2023-05-31T21:35:41.053 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/pklxkd-55/h 2023-05-31T21:35:41.053 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/pklxkd-55/h 2023-05-31T21:35:41.053 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/pklxkd-55/h?versionId=null 2023-05-31T21:35:41.054 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/pklxkd-55/h?versionId=null 2023-05-31T21:35:41.054 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-05-31T21:35:41.054 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /pklxkd-55/h?versionId=null 2023-05-31T21:35:41.054 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-05-31T21:35:41.054 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-05-31T21:35:41.055 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T21:35:41.055 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-05-31T21:35:41.055 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-05-31T21:35:41.055 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-05-31T21:35:41.056 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-05-31T21:35:41.056 INFO:tasks.rgw_multisite_tests:GET 2023-05-31T21:35:41.056 INFO:tasks.rgw_multisite_tests: 2023-05-31T21:35:41.056 INFO:tasks.rgw_multisite_tests: 2023-05-31T21:35:41.057 INFO:tasks.rgw_multisite_tests:Wed, 31 May 2023 20:29:25 GMT 2023-05-31T21:35:41.057 INFO:tasks.rgw_multisite_tests:/pklxkd-55/h?versionId=null 2023-05-31T21:35:41.057 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-05-31T21:35:41.057 INFO:tasks.rgw_multisite_tests:AWS Q80H11OLNU1S48PPYL26:4Y/8L8lDnhHI55GrOpuluMNFBUo= 2023-05-31T21:35:41.057 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:29:25 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:4Y/8L8lDnhHI55GrOpuluMNFBUo=', 'Content-Length': '0'} 2023-05-31T21:35:41.058 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:27:24 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000023c0be56f71e321e-006477ae25-162e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:29:25 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:35:41.058 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/pklxkd-55/h 2023-05-31T21:35:41.058 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/pklxkd-55/h 2023-05-31T21:35:41.058 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/pklxkd-55/h?versionId=null 2023-05-31T21:35:41.059 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/pklxkd-55/h?versionId=null 2023-05-31T21:35:41.059 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-05-31T21:35:41.059 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /pklxkd-55/h?versionId=null 2023-05-31T21:35:41.059 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-05-31T21:35:41.060 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-05-31T21:35:41.060 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T21:35:41.060 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-05-31T21:35:41.060 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-05-31T21:35:41.060 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-05-31T21:35:41.061 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-05-31T21:35:41.061 INFO:tasks.rgw_multisite_tests:GET 2023-05-31T21:35:41.061 INFO:tasks.rgw_multisite_tests: 2023-05-31T21:35:41.061 INFO:tasks.rgw_multisite_tests: 2023-05-31T21:35:41.062 INFO:tasks.rgw_multisite_tests:Wed, 31 May 2023 20:29:25 GMT 2023-05-31T21:35:41.062 INFO:tasks.rgw_multisite_tests:/pklxkd-55/h?versionId=null 2023-05-31T21:35:41.062 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-05-31T21:35:41.062 INFO:tasks.rgw_multisite_tests:AWS Q80H11OLNU1S48PPYL26:4Y/8L8lDnhHI55GrOpuluMNFBUo= 2023-05-31T21:35:41.062 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:29:25 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:4Y/8L8lDnhHI55GrOpuluMNFBUo=', 'Content-Length': '0'} 2023-05-31T21:35:41.063 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:27:24 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007cafc6647d568eb0-006477ae25-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:29:25 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:35:41.063 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/pklxkd-55/ 2023-05-31T21:35:41.063 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/pklxkd-55/ 2023-05-31T21:35:41.063 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/pklxkd-55/?prefix=h 2023-05-31T21:35:41.064 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/pklxkd-55/?prefix=h 2023-05-31T21:35:41.064 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-05-31T21:35:41.064 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /pklxkd-55/?prefix=h 2023-05-31T21:35:41.064 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-05-31T21:35:41.065 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-05-31T21:35:41.065 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T21:35:41.065 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-05-31T21:35:41.065 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-05-31T21:35:41.066 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-05-31T21:35:41.066 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-05-31T21:35:41.066 INFO:tasks.rgw_multisite_tests:GET 2023-05-31T21:35:41.066 INFO:tasks.rgw_multisite_tests: 2023-05-31T21:35:41.066 INFO:tasks.rgw_multisite_tests: 2023-05-31T21:35:41.067 INFO:tasks.rgw_multisite_tests:Wed, 31 May 2023 20:29:25 GMT 2023-05-31T21:35:41.067 INFO:tasks.rgw_multisite_tests:/pklxkd-55/ 2023-05-31T21:35:41.067 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-05-31T21:35:41.067 INFO:tasks.rgw_multisite_tests:AWS Q80H11OLNU1S48PPYL26:a0/qPp0bnZeH62hZJf6OA1a38dw= 2023-05-31T21:35:41.068 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:29:25 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:a0/qPp0bnZeH62hZJf6OA1a38dw=', 'Content-Length': '0'} 2023-05-31T21:35:41.068 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000edbfa582cb1ba6df-006477ae25-162e-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:29:25 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:35:41.068 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'pklxkd-55h1000falseh2023-05-31T20:27:24.053Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T21:35:41.068 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/pklxkd-55/ 2023-05-31T21:35:41.069 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/pklxkd-55/ 2023-05-31T21:35:41.069 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/pklxkd-55/?prefix=h 2023-05-31T21:35:41.069 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/pklxkd-55/?prefix=h 2023-05-31T21:35:41.069 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-05-31T21:35:41.070 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /pklxkd-55/?prefix=h 2023-05-31T21:35:41.070 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-05-31T21:35:41.070 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-05-31T21:35:41.070 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T21:35:41.070 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-05-31T21:35:41.071 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-05-31T21:35:41.071 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-05-31T21:35:41.071 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-05-31T21:35:41.071 INFO:tasks.rgw_multisite_tests:GET 2023-05-31T21:35:41.072 INFO:tasks.rgw_multisite_tests: 2023-05-31T21:35:41.072 INFO:tasks.rgw_multisite_tests: 2023-05-31T21:35:41.072 INFO:tasks.rgw_multisite_tests:Wed, 31 May 2023 20:29:25 GMT 2023-05-31T21:35:41.072 INFO:tasks.rgw_multisite_tests:/pklxkd-55/ 2023-05-31T21:35:41.072 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-05-31T21:35:41.073 INFO:tasks.rgw_multisite_tests:AWS Q80H11OLNU1S48PPYL26:a0/qPp0bnZeH62hZJf6OA1a38dw= 2023-05-31T21:35:41.073 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:29:25 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:a0/qPp0bnZeH62hZJf6OA1a38dw=', 'Content-Length': '0'} 2023-05-31T21:35:41.073 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d56fab84570fcd38-006477ae25-1630-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:29:25 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:35:41.073 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'pklxkd-55h1000falseh2023-05-31T20:27:24.053Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T21:35:41.074 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/pklxkd-55/h 2023-05-31T21:35:41.074 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/pklxkd-55/h 2023-05-31T21:35:41.074 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-05-31T21:35:41.074 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /pklxkd-55/h 2023-05-31T21:35:41.075 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-05-31T21:35:41.075 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-05-31T21:35:41.075 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T21:35:41.075 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-05-31T21:35:41.075 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-05-31T21:35:41.076 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-05-31T21:35:41.076 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-05-31T21:35:41.076 INFO:tasks.rgw_multisite_tests:HEAD 2023-05-31T21:35:41.076 INFO:tasks.rgw_multisite_tests: 2023-05-31T21:35:41.077 INFO:tasks.rgw_multisite_tests: 2023-05-31T21:35:41.077 INFO:tasks.rgw_multisite_tests:Wed, 31 May 2023 20:29:25 GMT 2023-05-31T21:35:41.077 INFO:tasks.rgw_multisite_tests:/pklxkd-55/h 2023-05-31T21:35:41.077 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-05-31T21:35:41.078 INFO:tasks.rgw_multisite_tests:AWS Q80H11OLNU1S48PPYL26:EJL0qHt0IGOrq/lgbMCgfQTfo7U= 2023-05-31T21:35:41.078 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:29:25 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:EJL0qHt0IGOrq/lgbMCgfQTfo7U=', 'Content-Length': '0'} 2023-05-31T21:35:41.078 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:27:24 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000221909ccab4120e2-006477ae25-162e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:29:25 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:35:41.078 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/pklxkd-55/h 2023-05-31T21:35:41.079 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/pklxkd-55/h 2023-05-31T21:35:41.079 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-05-31T21:35:41.079 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /pklxkd-55/h 2023-05-31T21:35:41.079 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-05-31T21:35:41.080 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-05-31T21:35:41.080 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T21:35:41.080 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-05-31T21:35:41.080 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-05-31T21:35:41.081 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-05-31T21:35:41.081 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-05-31T21:35:41.081 INFO:tasks.rgw_multisite_tests:HEAD 2023-05-31T21:35:41.081 INFO:tasks.rgw_multisite_tests: 2023-05-31T21:35:41.082 INFO:tasks.rgw_multisite_tests: 2023-05-31T21:35:41.082 INFO:tasks.rgw_multisite_tests:Wed, 31 May 2023 20:29:25 GMT 2023-05-31T21:35:41.082 INFO:tasks.rgw_multisite_tests:/pklxkd-55/h 2023-05-31T21:35:41.082 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-05-31T21:35:41.082 INFO:tasks.rgw_multisite_tests:AWS Q80H11OLNU1S48PPYL26:EJL0qHt0IGOrq/lgbMCgfQTfo7U= 2023-05-31T21:35:41.083 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:29:25 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:EJL0qHt0IGOrq/lgbMCgfQTfo7U=', 'Content-Length': '0'} 2023-05-31T21:35:41.083 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:27:24 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000869729a74204d4de-006477ae25-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:29:25 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:35:41.083 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: comparing key name=h 2023-05-31T21:35:41.083 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/pklxkd-55/h 2023-05-31T21:35:41.084 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/pklxkd-55/h 2023-05-31T21:35:41.084 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-05-31T21:35:41.084 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /pklxkd-55/h 2023-05-31T21:35:41.084 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-05-31T21:35:41.085 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-05-31T21:35:41.085 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T21:35:41.085 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-05-31T21:35:41.085 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-05-31T21:35:41.085 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-05-31T21:35:41.086 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-05-31T21:35:41.086 INFO:tasks.rgw_multisite_tests:GET 2023-05-31T21:35:41.086 INFO:tasks.rgw_multisite_tests: 2023-05-31T21:35:41.086 INFO:tasks.rgw_multisite_tests: 2023-05-31T21:35:41.087 INFO:tasks.rgw_multisite_tests:Wed, 31 May 2023 20:29:25 GMT 2023-05-31T21:35:41.087 INFO:tasks.rgw_multisite_tests:/pklxkd-55/h 2023-05-31T21:35:41.087 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-05-31T21:35:41.087 INFO:tasks.rgw_multisite_tests:AWS Q80H11OLNU1S48PPYL26:Nr+uUYQjACsxuRi6Tmzu4v68dVo= 2023-05-31T21:35:41.087 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:29:25 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:Nr+uUYQjACsxuRi6Tmzu4v68dVo=', 'Content-Length': '0'} 2023-05-31T21:35:41.088 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:27:24 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000ca95da153df03d2-006477ae25-162e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:29:25 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:35:41.088 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/pklxkd-55/h 2023-05-31T21:35:41.088 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/pklxkd-55/h 2023-05-31T21:35:41.088 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-05-31T21:35:41.089 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /pklxkd-55/h 2023-05-31T21:35:41.089 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-05-31T21:35:41.089 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-05-31T21:35:41.089 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T21:35:41.090 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-05-31T21:35:41.090 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-05-31T21:35:41.090 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-05-31T21:35:41.090 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-05-31T21:35:41.091 INFO:tasks.rgw_multisite_tests:GET 2023-05-31T21:35:41.091 INFO:tasks.rgw_multisite_tests: 2023-05-31T21:35:41.091 INFO:tasks.rgw_multisite_tests: 2023-05-31T21:35:41.091 INFO:tasks.rgw_multisite_tests:Wed, 31 May 2023 20:29:25 GMT 2023-05-31T21:35:41.091 INFO:tasks.rgw_multisite_tests:/pklxkd-55/h 2023-05-31T21:35:41.092 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-05-31T21:35:41.092 INFO:tasks.rgw_multisite_tests:AWS Q80H11OLNU1S48PPYL26:Nr+uUYQjACsxuRi6Tmzu4v68dVo= 2023-05-31T21:35:41.092 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:29:25 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:Nr+uUYQjACsxuRi6Tmzu4v68dVo=', 'Content-Length': '0'} 2023-05-31T21:35:41.092 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:27:24 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d849d8934f2f837b-006477ae25-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:29:25 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:35:41.093 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/pklxkd-55/ 2023-05-31T21:35:41.093 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/pklxkd-55/ 2023-05-31T21:35:41.093 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/pklxkd-55/?prefix=h 2023-05-31T21:35:41.093 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/pklxkd-55/?prefix=h 2023-05-31T21:35:41.094 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-05-31T21:35:41.094 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /pklxkd-55/?prefix=h 2023-05-31T21:35:41.094 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-05-31T21:35:41.094 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-05-31T21:35:41.095 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T21:35:41.095 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-05-31T21:35:41.095 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-05-31T21:35:41.095 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-05-31T21:35:41.096 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-05-31T21:35:41.096 INFO:tasks.rgw_multisite_tests:GET 2023-05-31T21:35:41.096 INFO:tasks.rgw_multisite_tests: 2023-05-31T21:35:41.096 INFO:tasks.rgw_multisite_tests: 2023-05-31T21:35:41.097 INFO:tasks.rgw_multisite_tests:Wed, 31 May 2023 20:29:25 GMT 2023-05-31T21:35:41.097 INFO:tasks.rgw_multisite_tests:/pklxkd-55/ 2023-05-31T21:35:41.097 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-05-31T21:35:41.097 INFO:tasks.rgw_multisite_tests:AWS Q80H11OLNU1S48PPYL26:a0/qPp0bnZeH62hZJf6OA1a38dw= 2023-05-31T21:35:41.097 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:29:25 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:a0/qPp0bnZeH62hZJf6OA1a38dw=', 'Content-Length': '0'} 2023-05-31T21:35:41.098 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e73cffee0e9c841f-006477ae25-162e-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:29:25 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:35:41.098 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'pklxkd-55h1000falseh2023-05-31T20:27:24.053Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T21:35:41.098 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/pklxkd-55/ 2023-05-31T21:35:41.098 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/pklxkd-55/ 2023-05-31T21:35:41.099 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/pklxkd-55/?prefix=h 2023-05-31T21:35:41.099 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/pklxkd-55/?prefix=h 2023-05-31T21:35:41.099 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-05-31T21:35:41.099 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /pklxkd-55/?prefix=h 2023-05-31T21:35:41.100 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-05-31T21:35:41.100 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-05-31T21:35:41.100 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T21:35:41.100 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-05-31T21:35:41.100 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-05-31T21:35:41.101 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-05-31T21:35:41.101 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-05-31T21:35:41.101 INFO:tasks.rgw_multisite_tests:GET 2023-05-31T21:35:41.101 INFO:tasks.rgw_multisite_tests: 2023-05-31T21:35:41.102 INFO:tasks.rgw_multisite_tests: 2023-05-31T21:35:41.102 INFO:tasks.rgw_multisite_tests:Wed, 31 May 2023 20:29:25 GMT 2023-05-31T21:35:41.102 INFO:tasks.rgw_multisite_tests:/pklxkd-55/ 2023-05-31T21:35:41.102 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-05-31T21:35:41.103 INFO:tasks.rgw_multisite_tests:AWS Q80H11OLNU1S48PPYL26:a0/qPp0bnZeH62hZJf6OA1a38dw= 2023-05-31T21:35:41.103 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:29:25 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:a0/qPp0bnZeH62hZJf6OA1a38dw=', 'Content-Length': '0'} 2023-05-31T21:35:41.103 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000eb1c80318bda261b-006477ae25-1630-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:29:25 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:35:41.103 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'pklxkd-55h1000falseh2023-05-31T20:27:24.053Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T21:35:41.104 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: INFO: success, bucket identical: bucket=pklxkd-55 zones={a2, a1} 2023-05-31T21:35:41.104 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['bucket', 'reshard', '--bucket', 'pklxkd-55', '--num-shards', '3', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-05-31T21:35:41.104 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'reshard', '--bucket', 'pklxkd-55', '--num-shards', '3', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-05-31T21:35:41.104 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket reshard --bucket pklxkd-55 --num-shards 3 --yes-i-really-mean-it --cluster c1 --debug-rgw 0 --debug-ms 0 2023-05-31T21:35:41.105 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: ignoring --setuser ceph since I am not root 2023-05-31T21:35:41.105 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-05-31T21:35:41.105 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: tenant: 2023-05-31T21:35:41.105 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: bucket name: pklxkd-55 2023-05-31T21:35:41.105 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: total entries: 8 2023-05-31T21:35:41.106 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['bucket', 'layout', '--bucket', 'pklxkd-55', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-05-31T21:35:41.106 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'layout', '--bucket', 'pklxkd-55', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-05-31T21:35:41.106 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket layout --bucket pklxkd-55 --cluster c1 --debug-rgw 0 --debug-ms 0 2023-05-31T21:35:41.106 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: ignoring --setuser ceph since I am not root 2023-05-31T21:35:41.107 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-05-31T21:35:41.107 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2023-05-31T21:35:41.107 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "layout": { 2023-05-31T21:35:41.107 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "resharding": "None", 2023-05-31T21:35:41.108 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "current_index": { 2023-05-31T21:35:41.108 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "gen": 2, 2023-05-31T21:35:41.108 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "layout": { 2023-05-31T21:35:41.108 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "type": "Normal", 2023-05-31T21:35:41.109 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "normal": { 2023-05-31T21:35:41.109 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "num_shards": 3, 2023-05-31T21:35:41.109 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "hash_type": "Mod" 2023-05-31T21:35:41.109 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2023-05-31T21:35:41.110 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2023-05-31T21:35:41.110 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2023-05-31T21:35:41.110 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "logs": [ 2023-05-31T21:35:41.110 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2023-05-31T21:35:41.110 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "gen": 0, 2023-05-31T21:35:41.111 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "layout": { 2023-05-31T21:35:41.111 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "type": "InIndex", 2023-05-31T21:35:41.111 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "in_index": { 2023-05-31T21:35:41.111 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "gen": 0, 2023-05-31T21:35:41.112 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "layout": { 2023-05-31T21:35:41.112 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "num_shards": 11, 2023-05-31T21:35:41.112 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "hash_type": "Mod" 2023-05-31T21:35:41.112 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2023-05-31T21:35:41.112 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2023-05-31T21:35:41.113 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2023-05-31T21:35:41.113 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2023-05-31T21:35:41.113 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2023-05-31T21:35:41.113 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "gen": 1, 2023-05-31T21:35:41.114 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "layout": { 2023-05-31T21:35:41.114 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "type": "InIndex", 2023-05-31T21:35:41.114 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "in_index": { 2023-05-31T21:35:41.114 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "gen": 1, 2023-05-31T21:35:41.115 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "layout": { 2023-05-31T21:35:41.115 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "num_shards": 3, 2023-05-31T21:35:41.115 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "hash_type": "Mod" 2023-05-31T21:35:41.115 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2023-05-31T21:35:41.115 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2023-05-31T21:35:41.116 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2023-05-31T21:35:41.116 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2023-05-31T21:35:41.116 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2023-05-31T21:35:41.116 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "gen": 2, 2023-05-31T21:35:41.117 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "layout": { 2023-05-31T21:35:41.117 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "type": "InIndex", 2023-05-31T21:35:41.117 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "in_index": { 2023-05-31T21:35:41.117 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "gen": 2, 2023-05-31T21:35:41.118 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "layout": { 2023-05-31T21:35:41.118 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "num_shards": 3, 2023-05-31T21:35:41.118 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "hash_type": "Mod" 2023-05-31T21:35:41.118 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2023-05-31T21:35:41.118 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2023-05-31T21:35:41.119 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2023-05-31T21:35:41.119 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2023-05-31T21:35:41.119 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: ] 2023-05-31T21:35:41.119 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2023-05-31T21:35:41.120 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2023-05-31T21:35:41.120 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: bucket checkpoint: source=a1 target=a2 bucket=pklxkd-55 2023-05-31T21:35:41.120 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'pklxkd-55', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-05-31T21:35:41.120 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'pklxkd-55', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-05-31T21:35:41.121 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket pklxkd-55 --source-zone a1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2023-05-31T21:35:41.121 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: ignoring --setuser ceph since I am not root 2023-05-31T21:35:41.121 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-05-31T21:35:41.121 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: 2023-05-31T20:29:26.417+0000 7f6086eab780 1 waiting to reach latest gen 2, on 1.. 2023-05-31T21:35:41.121 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: 2023-05-31T20:29:31.418+0000 7f6086eab780 1 waiting to reach latest gen 2, on 1.. 2023-05-31T21:35:41.122 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: 2023-05-31T20:29:36.419+0000 7f6086eab780 1 waiting to reach latest gen 2, on 1.. 2023-05-31T21:35:41.122 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: 2023-05-31T20:29:41.420+0000 7f6086eab780 1 waiting to reach latest gen 2, on 1.. 2023-05-31T21:35:41.122 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: 2023-05-31T20:29:46.420+0000 7f6086eab780 1 bucket sync caught up with empty source 2023-05-31T21:35:41.122 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: 2023-05-31T20:29:46.420+0000 7f6086eab780 0 bucket checkpoint complete 2023-05-31T21:35:41.123 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: bucket checkpoint: source=a2 target=a1 bucket=pklxkd-55 2023-05-31T21:35:41.123 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'pklxkd-55', '--source-zone', 'a2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-05-31T21:35:41.123 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'pklxkd-55', '--source-zone', 'a2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-05-31T21:35:41.123 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket pklxkd-55 --source-zone a2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2023-05-31T21:35:41.124 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: ignoring --setuser ceph since I am not root 2023-05-31T21:35:41.124 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-05-31T21:35:41.124 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: 2023-05-31T20:29:46.554+0000 7f5c73079780 1 bucket sync caught up with source: 2023-05-31T21:35:41.124 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: local status: [, 00000000002.1063.6, , , 00000000001.1666.6, 00000000001.1340.6, , , 00000000002.3558.6, , 00000000002.2408.6] 2023-05-31T21:35:41.124 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: remote markers: [, 00000000002.1063.6, , , 00000000001.1666.6, 00000000001.1340.6, , , 00000000002.3558.6, , 00000000002.2408.6] 2023-05-31T21:35:41.125 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: 2023-05-31T20:29:46.554+0000 7f5c73079780 0 bucket checkpoint complete 2023-05-31T21:35:41.125 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: INFO: comparing bucket=pklxkd-55 zones={a2, a1} 2023-05-31T21:35:41.125 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/pklxkd-55/ 2023-05-31T21:35:41.125 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/pklxkd-55/ 2023-05-31T21:35:41.126 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-05-31T21:35:41.126 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /pklxkd-55/ 2023-05-31T21:35:41.126 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-05-31T21:35:41.126 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-05-31T21:35:41.127 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T21:35:41.127 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-05-31T21:35:41.127 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-05-31T21:35:41.127 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-05-31T21:35:41.128 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-05-31T21:35:41.128 INFO:tasks.rgw_multisite_tests:HEAD 2023-05-31T21:35:41.128 INFO:tasks.rgw_multisite_tests: 2023-05-31T21:35:41.128 INFO:tasks.rgw_multisite_tests: 2023-05-31T21:35:41.128 INFO:tasks.rgw_multisite_tests:Wed, 31 May 2023 20:29:46 GMT 2023-05-31T21:35:41.129 INFO:tasks.rgw_multisite_tests:/pklxkd-55/ 2023-05-31T21:35:41.129 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-05-31T21:35:41.129 INFO:tasks.rgw_multisite_tests:AWS Q80H11OLNU1S48PPYL26:n/yoparVfMjq9ET3vWekmT7kbq0= 2023-05-31T21:35:41.129 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:29:46 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:n/yoparVfMjq9ET3vWekmT7kbq0=', 'Content-Length': '0'} 2023-05-31T21:35:41.130 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '8'), ('X-RGW-Bytes-Used', '24'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000c2632836ee40db21-006477ae3a-162e-a2'), ('Content-Length', '0'), ('Date', 'Wed, 31 May 2023 20:29:46 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:35:41.130 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/pklxkd-55/ 2023-05-31T21:35:41.130 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/pklxkd-55/ 2023-05-31T21:35:41.130 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-05-31T21:35:41.131 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /pklxkd-55/ 2023-05-31T21:35:41.131 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-05-31T21:35:41.131 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-05-31T21:35:41.131 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T21:35:41.131 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-05-31T21:35:41.132 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-05-31T21:35:41.132 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-05-31T21:35:41.132 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-05-31T21:35:41.132 INFO:tasks.rgw_multisite_tests:HEAD 2023-05-31T21:35:41.133 INFO:tasks.rgw_multisite_tests: 2023-05-31T21:35:41.133 INFO:tasks.rgw_multisite_tests: 2023-05-31T21:35:41.133 INFO:tasks.rgw_multisite_tests:Wed, 31 May 2023 20:29:46 GMT 2023-05-31T21:35:41.133 INFO:tasks.rgw_multisite_tests:/pklxkd-55/ 2023-05-31T21:35:41.133 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-05-31T21:35:41.134 INFO:tasks.rgw_multisite_tests:AWS Q80H11OLNU1S48PPYL26:n/yoparVfMjq9ET3vWekmT7kbq0= 2023-05-31T21:35:41.134 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:29:46 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:n/yoparVfMjq9ET3vWekmT7kbq0=', 'Content-Length': '0'} 2023-05-31T21:35:41.134 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '8'), ('X-RGW-Bytes-Used', '24'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000079d1e41e1e695184-006477ae3a-1630-a1'), ('Content-Length', '0'), ('Date', 'Wed, 31 May 2023 20:29:46 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:35:41.134 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: bucket1 objects: 2023-05-31T21:35:41.135 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/pklxkd-55/ 2023-05-31T21:35:41.135 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/pklxkd-55/ 2023-05-31T21:35:41.135 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/pklxkd-55/?versions&max-keys=999 2023-05-31T21:35:41.135 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/pklxkd-55/?versions&max-keys=999 2023-05-31T21:35:41.136 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-05-31T21:35:41.136 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /pklxkd-55/?versions&max-keys=999 2023-05-31T21:35:41.136 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-05-31T21:35:41.136 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-05-31T21:35:41.137 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T21:35:41.137 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-05-31T21:35:41.137 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-05-31T21:35:41.137 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-05-31T21:35:41.138 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-05-31T21:35:41.138 INFO:tasks.rgw_multisite_tests:GET 2023-05-31T21:35:41.138 INFO:tasks.rgw_multisite_tests: 2023-05-31T21:35:41.138 INFO:tasks.rgw_multisite_tests: 2023-05-31T21:35:41.138 INFO:tasks.rgw_multisite_tests:Wed, 31 May 2023 20:29:46 GMT 2023-05-31T21:35:41.139 INFO:tasks.rgw_multisite_tests:/pklxkd-55/?versions 2023-05-31T21:35:41.139 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-05-31T21:35:41.139 INFO:tasks.rgw_multisite_tests:AWS Q80H11OLNU1S48PPYL26:rGmKTw6GqSba2RfYy5rKgh3wg8w= 2023-05-31T21:35:41.139 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:29:46 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:rGmKTw6GqSba2RfYy5rKgh3wg8w=', 'Content-Length': '0'} 2023-05-31T21:35:41.140 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000093dd22a1e6a74c0a-006477ae3a-162e-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:29:46 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:35:41.140 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'pklxkd-55999falseanulltrue2023-05-31T20:27:16.792Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalbnulltrue2023-05-31T20:27:16.850Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalcnulltrue2023-05-31T20:27:16.869Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormaldnulltrue2023-05-31T20:27:16.893Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalenulltrue2023-05-31T20:27:23.955Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalfnulltrue2023-05-31T20:27:23.977Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalgnulltrue2023-05-31T20:27:24.035Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalhnulltrue2023-05-31T20:27:24.053Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T21:35:41.140 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: o=a 2023-05-31T21:35:41.140 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: o=b 2023-05-31T21:35:41.141 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: o=c 2023-05-31T21:35:41.141 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: o=d 2023-05-31T21:35:41.141 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: o=e 2023-05-31T21:35:41.141 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: o=f 2023-05-31T21:35:41.142 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: o=g 2023-05-31T21:35:41.142 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: o=h 2023-05-31T21:35:41.142 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: bucket2 objects: 2023-05-31T21:35:41.142 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/pklxkd-55/ 2023-05-31T21:35:41.142 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/pklxkd-55/ 2023-05-31T21:35:41.143 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/pklxkd-55/?versions&max-keys=999 2023-05-31T21:35:41.143 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/pklxkd-55/?versions&max-keys=999 2023-05-31T21:35:41.143 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-05-31T21:35:41.143 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /pklxkd-55/?versions&max-keys=999 2023-05-31T21:35:41.144 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-05-31T21:35:41.144 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-05-31T21:35:41.144 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T21:35:41.144 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-05-31T21:35:41.144 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-05-31T21:35:41.145 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-05-31T21:35:41.145 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-05-31T21:35:41.145 INFO:tasks.rgw_multisite_tests:GET 2023-05-31T21:35:41.145 INFO:tasks.rgw_multisite_tests: 2023-05-31T21:35:41.146 INFO:tasks.rgw_multisite_tests: 2023-05-31T21:35:41.146 INFO:tasks.rgw_multisite_tests:Wed, 31 May 2023 20:29:46 GMT 2023-05-31T21:35:41.146 INFO:tasks.rgw_multisite_tests:/pklxkd-55/?versions 2023-05-31T21:35:41.146 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-05-31T21:35:41.147 INFO:tasks.rgw_multisite_tests:AWS Q80H11OLNU1S48PPYL26:rGmKTw6GqSba2RfYy5rKgh3wg8w= 2023-05-31T21:35:41.147 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:29:46 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:rGmKTw6GqSba2RfYy5rKgh3wg8w=', 'Content-Length': '0'} 2023-05-31T21:35:41.147 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000085dbbd5b236ca226-006477ae3a-1630-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:29:46 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:35:41.147 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'pklxkd-55999falseanulltrue2023-05-31T20:27:16.792Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalbnulltrue2023-05-31T20:27:16.850Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalcnulltrue2023-05-31T20:27:16.869Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormaldnulltrue2023-05-31T20:27:16.893Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalenulltrue2023-05-31T20:27:23.955Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalfnulltrue2023-05-31T20:27:23.977Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalgnulltrue2023-05-31T20:27:24.035Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalhnulltrue2023-05-31T20:27:24.053Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T21:35:41.147 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: o=a 2023-05-31T21:35:41.148 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: o=b 2023-05-31T21:35:41.148 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: o=c 2023-05-31T21:35:41.148 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: o=d 2023-05-31T21:35:41.148 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: o=e 2023-05-31T21:35:41.149 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: o=f 2023-05-31T21:35:41.149 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: o=g 2023-05-31T21:35:41.149 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: o=h 2023-05-31T21:35:41.149 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/pklxkd-55/ 2023-05-31T21:35:41.150 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/pklxkd-55/ 2023-05-31T21:35:41.150 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/pklxkd-55/?versions&max-keys=999 2023-05-31T21:35:41.150 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/pklxkd-55/?versions&max-keys=999 2023-05-31T21:35:41.150 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-05-31T21:35:41.150 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /pklxkd-55/?versions&max-keys=999 2023-05-31T21:35:41.151 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-05-31T21:35:41.151 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-05-31T21:35:41.151 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T21:35:41.151 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-05-31T21:35:41.152 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-05-31T21:35:41.152 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-05-31T21:35:41.152 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-05-31T21:35:41.152 INFO:tasks.rgw_multisite_tests:GET 2023-05-31T21:35:41.153 INFO:tasks.rgw_multisite_tests: 2023-05-31T21:35:41.153 INFO:tasks.rgw_multisite_tests: 2023-05-31T21:35:41.153 INFO:tasks.rgw_multisite_tests:Wed, 31 May 2023 20:29:46 GMT 2023-05-31T21:35:41.153 INFO:tasks.rgw_multisite_tests:/pklxkd-55/?versions 2023-05-31T21:35:41.153 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-05-31T21:35:41.154 INFO:tasks.rgw_multisite_tests:AWS Q80H11OLNU1S48PPYL26:rGmKTw6GqSba2RfYy5rKgh3wg8w= 2023-05-31T21:35:41.154 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:29:46 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:rGmKTw6GqSba2RfYy5rKgh3wg8w=', 'Content-Length': '0'} 2023-05-31T21:35:41.154 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000899bdab387f96b14-006477ae3a-162e-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:29:46 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:35:41.154 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'pklxkd-55999falseanulltrue2023-05-31T20:27:16.792Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalbnulltrue2023-05-31T20:27:16.850Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalcnulltrue2023-05-31T20:27:16.869Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormaldnulltrue2023-05-31T20:27:16.893Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalenulltrue2023-05-31T20:27:23.955Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalfnulltrue2023-05-31T20:27:23.977Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalgnulltrue2023-05-31T20:27:24.035Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalhnulltrue2023-05-31T20:27:24.053Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T21:35:41.155 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/pklxkd-55/ 2023-05-31T21:35:41.155 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/pklxkd-55/ 2023-05-31T21:35:41.155 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/pklxkd-55/?versions&max-keys=999 2023-05-31T21:35:41.155 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/pklxkd-55/?versions&max-keys=999 2023-05-31T21:35:41.156 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-05-31T21:35:41.156 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /pklxkd-55/?versions&max-keys=999 2023-05-31T21:35:41.156 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-05-31T21:35:41.156 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-05-31T21:35:41.156 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T21:35:41.157 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-05-31T21:35:41.157 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-05-31T21:35:41.157 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-05-31T21:35:41.157 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-05-31T21:35:41.158 INFO:tasks.rgw_multisite_tests:GET 2023-05-31T21:35:41.158 INFO:tasks.rgw_multisite_tests: 2023-05-31T21:35:41.158 INFO:tasks.rgw_multisite_tests: 2023-05-31T21:35:41.158 INFO:tasks.rgw_multisite_tests:Wed, 31 May 2023 20:29:46 GMT 2023-05-31T21:35:41.158 INFO:tasks.rgw_multisite_tests:/pklxkd-55/?versions 2023-05-31T21:35:41.159 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-05-31T21:35:41.159 INFO:tasks.rgw_multisite_tests:AWS Q80H11OLNU1S48PPYL26:rGmKTw6GqSba2RfYy5rKgh3wg8w= 2023-05-31T21:35:41.159 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:29:46 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:rGmKTw6GqSba2RfYy5rKgh3wg8w=', 'Content-Length': '0'} 2023-05-31T21:35:41.159 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000afa4376b4dd6c8ed-006477ae3a-1630-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:29:46 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:35:41.160 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'pklxkd-55999falseanulltrue2023-05-31T20:27:16.792Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalbnulltrue2023-05-31T20:27:16.850Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalcnulltrue2023-05-31T20:27:16.869Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormaldnulltrue2023-05-31T20:27:16.893Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalenulltrue2023-05-31T20:27:23.955Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalfnulltrue2023-05-31T20:27:23.977Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalgnulltrue2023-05-31T20:27:24.035Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalhnulltrue2023-05-31T20:27:24.053Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T21:35:41.160 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: comparing key name=a 2023-05-31T21:35:41.160 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/pklxkd-55/a 2023-05-31T21:35:41.160 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/pklxkd-55/a 2023-05-31T21:35:41.161 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/pklxkd-55/a?versionId=null 2023-05-31T21:35:41.161 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/pklxkd-55/a?versionId=null 2023-05-31T21:35:41.161 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-05-31T21:35:41.161 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /pklxkd-55/a?versionId=null 2023-05-31T21:35:41.161 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-05-31T21:35:41.162 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-05-31T21:35:41.162 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T21:35:41.162 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-05-31T21:35:41.162 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-05-31T21:35:41.163 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-05-31T21:35:41.163 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-05-31T21:35:41.163 INFO:tasks.rgw_multisite_tests:GET 2023-05-31T21:35:41.163 INFO:tasks.rgw_multisite_tests: 2023-05-31T21:35:41.164 INFO:tasks.rgw_multisite_tests: 2023-05-31T21:35:41.164 INFO:tasks.rgw_multisite_tests:Wed, 31 May 2023 20:29:46 GMT 2023-05-31T21:35:41.164 INFO:tasks.rgw_multisite_tests:/pklxkd-55/a?versionId=null 2023-05-31T21:35:41.164 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-05-31T21:35:41.164 INFO:tasks.rgw_multisite_tests:AWS Q80H11OLNU1S48PPYL26:c4UKUgPjYepRbriw4uwPbqdCCx8= 2023-05-31T21:35:41.165 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:29:46 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:c4UKUgPjYepRbriw4uwPbqdCCx8=', 'Content-Length': '0'} 2023-05-31T21:35:41.165 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:27:16 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007dcd78de02405374-006477ae3a-162e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:29:46 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:35:41.165 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/pklxkd-55/a 2023-05-31T21:35:41.165 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/pklxkd-55/a 2023-05-31T21:35:41.166 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/pklxkd-55/a?versionId=null 2023-05-31T21:35:41.166 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/pklxkd-55/a?versionId=null 2023-05-31T21:35:41.166 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-05-31T21:35:41.166 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /pklxkd-55/a?versionId=null 2023-05-31T21:35:41.167 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-05-31T21:35:41.167 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-05-31T21:35:41.167 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T21:35:41.167 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-05-31T21:35:41.168 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-05-31T21:35:41.168 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-05-31T21:35:41.168 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-05-31T21:35:41.168 INFO:tasks.rgw_multisite_tests:GET 2023-05-31T21:35:41.168 INFO:tasks.rgw_multisite_tests: 2023-05-31T21:35:41.169 INFO:tasks.rgw_multisite_tests: 2023-05-31T21:35:41.169 INFO:tasks.rgw_multisite_tests:Wed, 31 May 2023 20:29:46 GMT 2023-05-31T21:35:41.169 INFO:tasks.rgw_multisite_tests:/pklxkd-55/a?versionId=null 2023-05-31T21:35:41.169 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-05-31T21:35:41.170 INFO:tasks.rgw_multisite_tests:AWS Q80H11OLNU1S48PPYL26:c4UKUgPjYepRbriw4uwPbqdCCx8= 2023-05-31T21:35:41.170 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:29:46 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:c4UKUgPjYepRbriw4uwPbqdCCx8=', 'Content-Length': '0'} 2023-05-31T21:35:41.170 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:27:16 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000080e5d1e45cd8a7a9-006477ae3a-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:29:46 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:35:41.170 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/pklxkd-55/a 2023-05-31T21:35:41.170 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/pklxkd-55/a 2023-05-31T21:35:41.171 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/pklxkd-55/a?versionId=null 2023-05-31T21:35:41.171 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/pklxkd-55/a?versionId=null 2023-05-31T21:35:41.171 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-05-31T21:35:41.171 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /pklxkd-55/a?versionId=null 2023-05-31T21:35:41.172 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-05-31T21:35:41.172 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-05-31T21:35:41.172 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T21:35:41.172 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-05-31T21:35:41.173 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-05-31T21:35:41.173 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-05-31T21:35:41.173 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-05-31T21:35:41.173 INFO:tasks.rgw_multisite_tests:HEAD 2023-05-31T21:35:41.174 INFO:tasks.rgw_multisite_tests: 2023-05-31T21:35:41.174 INFO:tasks.rgw_multisite_tests: 2023-05-31T21:35:41.174 INFO:tasks.rgw_multisite_tests:Wed, 31 May 2023 20:29:46 GMT 2023-05-31T21:35:41.174 INFO:tasks.rgw_multisite_tests:/pklxkd-55/a?versionId=null 2023-05-31T21:35:41.175 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-05-31T21:35:41.175 INFO:tasks.rgw_multisite_tests:AWS Q80H11OLNU1S48PPYL26:FZdFuuGnv5YAJEQP8rxZw81dAN8= 2023-05-31T21:35:41.175 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:29:46 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:FZdFuuGnv5YAJEQP8rxZw81dAN8=', 'Content-Length': '0'} 2023-05-31T21:35:41.175 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:27:16 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000920f8d56add47fdd-006477ae3a-162e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:29:46 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:35:41.176 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/pklxkd-55/a 2023-05-31T21:35:41.176 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/pklxkd-55/a 2023-05-31T21:35:41.176 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/pklxkd-55/a?versionId=null 2023-05-31T21:35:41.176 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/pklxkd-55/a?versionId=null 2023-05-31T21:35:41.177 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-05-31T21:35:41.177 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /pklxkd-55/a?versionId=null 2023-05-31T21:35:41.177 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-05-31T21:35:41.177 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-05-31T21:35:41.178 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T21:35:41.178 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-05-31T21:35:41.178 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-05-31T21:35:41.178 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-05-31T21:35:41.178 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-05-31T21:35:41.179 INFO:tasks.rgw_multisite_tests:HEAD 2023-05-31T21:35:41.179 INFO:tasks.rgw_multisite_tests: 2023-05-31T21:35:41.179 INFO:tasks.rgw_multisite_tests: 2023-05-31T21:35:41.179 INFO:tasks.rgw_multisite_tests:Wed, 31 May 2023 20:29:46 GMT 2023-05-31T21:35:41.180 INFO:tasks.rgw_multisite_tests:/pklxkd-55/a?versionId=null 2023-05-31T21:35:41.180 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-05-31T21:35:41.180 INFO:tasks.rgw_multisite_tests:AWS Q80H11OLNU1S48PPYL26:FZdFuuGnv5YAJEQP8rxZw81dAN8= 2023-05-31T21:35:41.181 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:29:46 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:FZdFuuGnv5YAJEQP8rxZw81dAN8=', 'Content-Length': '0'} 2023-05-31T21:35:41.181 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:27:16 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000349696cec2b43e13-006477ae3a-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:29:46 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:35:41.181 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: comparing key name=a 2023-05-31T21:35:41.181 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/pklxkd-55/a 2023-05-31T21:35:41.181 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/pklxkd-55/a 2023-05-31T21:35:41.182 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/pklxkd-55/a?versionId=null 2023-05-31T21:35:41.182 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/pklxkd-55/a?versionId=null 2023-05-31T21:35:41.182 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-05-31T21:35:41.182 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /pklxkd-55/a?versionId=null 2023-05-31T21:35:41.183 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-05-31T21:35:41.183 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-05-31T21:35:41.183 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T21:35:41.183 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-05-31T21:35:41.184 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-05-31T21:35:41.184 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-05-31T21:35:41.184 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-05-31T21:35:41.184 INFO:tasks.rgw_multisite_tests:GET 2023-05-31T21:35:41.184 INFO:tasks.rgw_multisite_tests: 2023-05-31T21:35:41.185 INFO:tasks.rgw_multisite_tests: 2023-05-31T21:35:41.185 INFO:tasks.rgw_multisite_tests:Wed, 31 May 2023 20:29:46 GMT 2023-05-31T21:35:41.185 INFO:tasks.rgw_multisite_tests:/pklxkd-55/a?versionId=null 2023-05-31T21:35:41.185 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-05-31T21:35:41.186 INFO:tasks.rgw_multisite_tests:AWS Q80H11OLNU1S48PPYL26:c4UKUgPjYepRbriw4uwPbqdCCx8= 2023-05-31T21:35:41.186 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:29:46 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:c4UKUgPjYepRbriw4uwPbqdCCx8=', 'Content-Length': '0'} 2023-05-31T21:35:41.186 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:27:16 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004c61cca21ffd61c5-006477ae3a-162e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:29:46 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:35:41.186 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/pklxkd-55/a 2023-05-31T21:35:41.187 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/pklxkd-55/a 2023-05-31T21:35:41.187 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/pklxkd-55/a?versionId=null 2023-05-31T21:35:41.187 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/pklxkd-55/a?versionId=null 2023-05-31T21:35:41.187 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-05-31T21:35:41.187 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /pklxkd-55/a?versionId=null 2023-05-31T21:35:41.188 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-05-31T21:35:41.188 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-05-31T21:35:41.188 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T21:35:41.188 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-05-31T21:35:41.189 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-05-31T21:35:41.189 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-05-31T21:35:41.189 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-05-31T21:35:41.189 INFO:tasks.rgw_multisite_tests:GET 2023-05-31T21:35:41.190 INFO:tasks.rgw_multisite_tests: 2023-05-31T21:35:41.190 INFO:tasks.rgw_multisite_tests: 2023-05-31T21:35:41.190 INFO:tasks.rgw_multisite_tests:Wed, 31 May 2023 20:29:46 GMT 2023-05-31T21:35:41.190 INFO:tasks.rgw_multisite_tests:/pklxkd-55/a?versionId=null 2023-05-31T21:35:41.191 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-05-31T21:35:41.191 INFO:tasks.rgw_multisite_tests:AWS Q80H11OLNU1S48PPYL26:c4UKUgPjYepRbriw4uwPbqdCCx8= 2023-05-31T21:35:41.191 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:29:46 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:c4UKUgPjYepRbriw4uwPbqdCCx8=', 'Content-Length': '0'} 2023-05-31T21:35:41.191 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:27:16 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c8dfbb690d9645e2-006477ae3a-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:29:46 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:35:41.191 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/pklxkd-55/ 2023-05-31T21:35:41.192 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/pklxkd-55/ 2023-05-31T21:35:41.192 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/pklxkd-55/?prefix=a 2023-05-31T21:35:41.192 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/pklxkd-55/?prefix=a 2023-05-31T21:35:41.192 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-05-31T21:35:41.193 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /pklxkd-55/?prefix=a 2023-05-31T21:35:41.193 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-05-31T21:35:41.193 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-05-31T21:35:41.193 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T21:35:41.194 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-05-31T21:35:41.194 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-05-31T21:35:41.194 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-05-31T21:35:41.194 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-05-31T21:35:41.194 INFO:tasks.rgw_multisite_tests:GET 2023-05-31T21:35:41.195 INFO:tasks.rgw_multisite_tests: 2023-05-31T21:35:41.195 INFO:tasks.rgw_multisite_tests: 2023-05-31T21:35:41.195 INFO:tasks.rgw_multisite_tests:Wed, 31 May 2023 20:29:46 GMT 2023-05-31T21:35:41.195 INFO:tasks.rgw_multisite_tests:/pklxkd-55/ 2023-05-31T21:35:41.196 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-05-31T21:35:41.196 INFO:tasks.rgw_multisite_tests:AWS Q80H11OLNU1S48PPYL26:XZ8tDAn1QJf1wBVwJgGN94Jp4hE= 2023-05-31T21:35:41.196 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:29:46 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:XZ8tDAn1QJf1wBVwJgGN94Jp4hE=', 'Content-Length': '0'} 2023-05-31T21:35:41.196 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a9ce75aabbe6fe26-006477ae3a-162e-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:29:46 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:35:41.197 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'pklxkd-55a1000falsea2023-05-31T20:27:16.792Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T21:35:41.197 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/pklxkd-55/ 2023-05-31T21:35:41.197 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/pklxkd-55/ 2023-05-31T21:35:41.197 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/pklxkd-55/?prefix=a 2023-05-31T21:35:41.198 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/pklxkd-55/?prefix=a 2023-05-31T21:35:41.198 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-05-31T21:35:41.198 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /pklxkd-55/?prefix=a 2023-05-31T21:35:41.198 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-05-31T21:35:41.198 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-05-31T21:35:41.199 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T21:35:41.199 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-05-31T21:35:41.199 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-05-31T21:35:41.199 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-05-31T21:35:41.200 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-05-31T21:35:41.200 INFO:tasks.rgw_multisite_tests:GET 2023-05-31T21:35:41.200 INFO:tasks.rgw_multisite_tests: 2023-05-31T21:35:41.200 INFO:tasks.rgw_multisite_tests: 2023-05-31T21:35:41.200 INFO:tasks.rgw_multisite_tests:Wed, 31 May 2023 20:29:46 GMT 2023-05-31T21:35:41.201 INFO:tasks.rgw_multisite_tests:/pklxkd-55/ 2023-05-31T21:35:41.201 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-05-31T21:35:41.201 INFO:tasks.rgw_multisite_tests:AWS Q80H11OLNU1S48PPYL26:XZ8tDAn1QJf1wBVwJgGN94Jp4hE= 2023-05-31T21:35:41.201 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:29:46 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:XZ8tDAn1QJf1wBVwJgGN94Jp4hE=', 'Content-Length': '0'} 2023-05-31T21:35:41.202 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ed98b74f53f67154-006477ae3a-1630-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:29:46 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:35:41.202 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'pklxkd-55a1000falsea2023-05-31T20:27:16.792Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T21:35:41.202 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/pklxkd-55/a 2023-05-31T21:35:41.202 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/pklxkd-55/a 2023-05-31T21:35:41.203 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-05-31T21:35:41.203 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /pklxkd-55/a 2023-05-31T21:35:41.203 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-05-31T21:35:41.203 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-05-31T21:35:41.203 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T21:35:41.204 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-05-31T21:35:41.204 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-05-31T21:35:41.204 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-05-31T21:35:41.204 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-05-31T21:35:41.205 INFO:tasks.rgw_multisite_tests:HEAD 2023-05-31T21:35:41.205 INFO:tasks.rgw_multisite_tests: 2023-05-31T21:35:41.205 INFO:tasks.rgw_multisite_tests: 2023-05-31T21:35:41.205 INFO:tasks.rgw_multisite_tests:Wed, 31 May 2023 20:29:46 GMT 2023-05-31T21:35:41.206 INFO:tasks.rgw_multisite_tests:/pklxkd-55/a 2023-05-31T21:35:41.206 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-05-31T21:35:41.206 INFO:tasks.rgw_multisite_tests:AWS Q80H11OLNU1S48PPYL26:oU2fckqlUEqdKJ120jVqGHKOOWg= 2023-05-31T21:35:41.206 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:29:46 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:oU2fckqlUEqdKJ120jVqGHKOOWg=', 'Content-Length': '0'} 2023-05-31T21:35:41.206 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:27:16 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b0fa36fa327894fc-006477ae3a-162e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:29:46 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:35:41.207 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/pklxkd-55/a 2023-05-31T21:35:41.207 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/pklxkd-55/a 2023-05-31T21:35:41.207 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-05-31T21:35:41.207 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /pklxkd-55/a 2023-05-31T21:35:41.208 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-05-31T21:35:41.208 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-05-31T21:35:41.208 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T21:35:41.208 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-05-31T21:35:41.209 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-05-31T21:35:41.209 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-05-31T21:35:41.209 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-05-31T21:35:41.209 INFO:tasks.rgw_multisite_tests:HEAD 2023-05-31T21:35:41.209 INFO:tasks.rgw_multisite_tests: 2023-05-31T21:35:41.210 INFO:tasks.rgw_multisite_tests: 2023-05-31T21:35:41.210 INFO:tasks.rgw_multisite_tests:Wed, 31 May 2023 20:29:46 GMT 2023-05-31T21:35:41.210 INFO:tasks.rgw_multisite_tests:/pklxkd-55/a 2023-05-31T21:35:41.210 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-05-31T21:35:41.211 INFO:tasks.rgw_multisite_tests:AWS Q80H11OLNU1S48PPYL26:oU2fckqlUEqdKJ120jVqGHKOOWg= 2023-05-31T21:35:41.211 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:29:46 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:oU2fckqlUEqdKJ120jVqGHKOOWg=', 'Content-Length': '0'} 2023-05-31T21:35:41.211 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:27:16 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f4d186947c72b1ae-006477ae3a-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:29:46 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:35:41.211 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: comparing key name=a 2023-05-31T21:35:41.211 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/pklxkd-55/a 2023-05-31T21:35:41.212 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/pklxkd-55/a 2023-05-31T21:35:41.212 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-05-31T21:35:41.212 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /pklxkd-55/a 2023-05-31T21:35:41.212 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-05-31T21:35:41.213 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-05-31T21:35:41.213 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T21:35:41.213 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-05-31T21:35:41.213 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-05-31T21:35:41.214 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-05-31T21:35:41.214 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-05-31T21:35:41.214 INFO:tasks.rgw_multisite_tests:GET 2023-05-31T21:35:41.214 INFO:tasks.rgw_multisite_tests: 2023-05-31T21:35:41.215 INFO:tasks.rgw_multisite_tests: 2023-05-31T21:35:41.215 INFO:tasks.rgw_multisite_tests:Wed, 31 May 2023 20:29:46 GMT 2023-05-31T21:35:41.215 INFO:tasks.rgw_multisite_tests:/pklxkd-55/a 2023-05-31T21:35:41.215 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-05-31T21:35:41.216 INFO:tasks.rgw_multisite_tests:AWS Q80H11OLNU1S48PPYL26:zVsdgueAZ78g4GjCy3kgQ/VWQ6Y= 2023-05-31T21:35:41.216 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:29:46 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:zVsdgueAZ78g4GjCy3kgQ/VWQ6Y=', 'Content-Length': '0'} 2023-05-31T21:35:41.216 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:27:16 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001be3eeafb3076d22-006477ae3a-162e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:29:46 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:35:41.216 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/pklxkd-55/a 2023-05-31T21:35:41.216 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/pklxkd-55/a 2023-05-31T21:35:41.217 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-05-31T21:35:41.217 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /pklxkd-55/a 2023-05-31T21:35:41.217 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-05-31T21:35:41.217 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-05-31T21:35:41.218 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T21:35:41.218 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-05-31T21:35:41.218 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-05-31T21:35:41.218 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-05-31T21:35:41.219 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-05-31T21:35:41.219 INFO:tasks.rgw_multisite_tests:GET 2023-05-31T21:35:41.219 INFO:tasks.rgw_multisite_tests: 2023-05-31T21:35:41.219 INFO:tasks.rgw_multisite_tests: 2023-05-31T21:35:41.219 INFO:tasks.rgw_multisite_tests:Wed, 31 May 2023 20:29:46 GMT 2023-05-31T21:35:41.220 INFO:tasks.rgw_multisite_tests:/pklxkd-55/a 2023-05-31T21:35:41.220 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-05-31T21:35:41.220 INFO:tasks.rgw_multisite_tests:AWS Q80H11OLNU1S48PPYL26:zVsdgueAZ78g4GjCy3kgQ/VWQ6Y= 2023-05-31T21:35:41.220 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:29:46 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:zVsdgueAZ78g4GjCy3kgQ/VWQ6Y=', 'Content-Length': '0'} 2023-05-31T21:35:41.221 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:27:16 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b639e34b4c0e1e6e-006477ae3a-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:29:46 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:35:41.221 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/pklxkd-55/ 2023-05-31T21:35:41.221 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/pklxkd-55/ 2023-05-31T21:35:41.221 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/pklxkd-55/?prefix=a 2023-05-31T21:35:41.222 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/pklxkd-55/?prefix=a 2023-05-31T21:35:41.222 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-05-31T21:35:41.222 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /pklxkd-55/?prefix=a 2023-05-31T21:35:41.222 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-05-31T21:35:41.223 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-05-31T21:35:41.223 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T21:35:41.223 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-05-31T21:35:41.223 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-05-31T21:35:41.223 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-05-31T21:35:41.224 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-05-31T21:35:41.224 INFO:tasks.rgw_multisite_tests:GET 2023-05-31T21:35:41.224 INFO:tasks.rgw_multisite_tests: 2023-05-31T21:35:41.224 INFO:tasks.rgw_multisite_tests: 2023-05-31T21:35:41.225 INFO:tasks.rgw_multisite_tests:Wed, 31 May 2023 20:29:46 GMT 2023-05-31T21:35:41.225 INFO:tasks.rgw_multisite_tests:/pklxkd-55/ 2023-05-31T21:35:41.225 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-05-31T21:35:41.225 INFO:tasks.rgw_multisite_tests:AWS Q80H11OLNU1S48PPYL26:XZ8tDAn1QJf1wBVwJgGN94Jp4hE= 2023-05-31T21:35:41.226 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:29:46 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:XZ8tDAn1QJf1wBVwJgGN94Jp4hE=', 'Content-Length': '0'} 2023-05-31T21:35:41.226 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f7440c964c2076c5-006477ae3a-162e-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:29:46 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:35:41.226 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'pklxkd-55a1000falsea2023-05-31T20:27:16.792Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T21:35:41.226 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/pklxkd-55/ 2023-05-31T21:35:41.227 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/pklxkd-55/ 2023-05-31T21:35:41.227 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/pklxkd-55/?prefix=a 2023-05-31T21:35:41.227 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/pklxkd-55/?prefix=a 2023-05-31T21:35:41.227 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-05-31T21:35:41.227 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /pklxkd-55/?prefix=a 2023-05-31T21:35:41.228 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-05-31T21:35:41.228 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-05-31T21:35:41.228 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T21:35:41.228 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-05-31T21:35:41.229 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-05-31T21:35:41.229 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-05-31T21:35:41.229 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-05-31T21:35:41.229 INFO:tasks.rgw_multisite_tests:GET 2023-05-31T21:35:41.230 INFO:tasks.rgw_multisite_tests: 2023-05-31T21:35:41.230 INFO:tasks.rgw_multisite_tests: 2023-05-31T21:35:41.230 INFO:tasks.rgw_multisite_tests:Wed, 31 May 2023 20:29:46 GMT 2023-05-31T21:35:41.230 INFO:tasks.rgw_multisite_tests:/pklxkd-55/ 2023-05-31T21:35:41.230 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-05-31T21:35:41.231 INFO:tasks.rgw_multisite_tests:AWS Q80H11OLNU1S48PPYL26:XZ8tDAn1QJf1wBVwJgGN94Jp4hE= 2023-05-31T21:35:41.231 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:29:46 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:XZ8tDAn1QJf1wBVwJgGN94Jp4hE=', 'Content-Length': '0'} 2023-05-31T21:35:41.231 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000008d349f11be9e4bf0-006477ae3a-1630-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:29:46 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:35:41.231 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'pklxkd-55a1000falsea2023-05-31T20:27:16.792Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T21:35:41.232 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: comparing key name=b 2023-05-31T21:35:41.232 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/pklxkd-55/b 2023-05-31T21:35:41.232 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/pklxkd-55/b 2023-05-31T21:35:41.232 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/pklxkd-55/b?versionId=null 2023-05-31T21:35:41.233 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/pklxkd-55/b?versionId=null 2023-05-31T21:35:41.233 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-05-31T21:35:41.233 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /pklxkd-55/b?versionId=null 2023-05-31T21:35:41.233 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-05-31T21:35:41.234 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-05-31T21:35:41.234 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T21:35:41.234 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-05-31T21:35:41.234 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-05-31T21:35:41.234 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-05-31T21:35:41.235 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-05-31T21:35:41.235 INFO:tasks.rgw_multisite_tests:GET 2023-05-31T21:35:41.235 INFO:tasks.rgw_multisite_tests: 2023-05-31T21:35:41.235 INFO:tasks.rgw_multisite_tests: 2023-05-31T21:35:41.236 INFO:tasks.rgw_multisite_tests:Wed, 31 May 2023 20:29:46 GMT 2023-05-31T21:35:41.236 INFO:tasks.rgw_multisite_tests:/pklxkd-55/b?versionId=null 2023-05-31T21:35:41.236 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-05-31T21:35:41.236 INFO:tasks.rgw_multisite_tests:AWS Q80H11OLNU1S48PPYL26:dohegYhb5mm2QkrqEQNUwnnKqu4= 2023-05-31T21:35:41.237 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:29:46 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:dohegYhb5mm2QkrqEQNUwnnKqu4=', 'Content-Length': '0'} 2023-05-31T21:35:41.237 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:27:16 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000049894b68834797bb-006477ae3a-162e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:29:46 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:35:41.237 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/pklxkd-55/b 2023-05-31T21:35:41.237 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/pklxkd-55/b 2023-05-31T21:35:41.237 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/pklxkd-55/b?versionId=null 2023-05-31T21:35:41.238 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/pklxkd-55/b?versionId=null 2023-05-31T21:35:41.238 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-05-31T21:35:41.238 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /pklxkd-55/b?versionId=null 2023-05-31T21:35:41.238 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-05-31T21:35:41.239 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-05-31T21:35:41.239 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T21:35:41.239 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-05-31T21:35:41.239 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-05-31T21:35:41.240 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-05-31T21:35:41.240 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-05-31T21:35:41.240 INFO:tasks.rgw_multisite_tests:GET 2023-05-31T21:35:41.240 INFO:tasks.rgw_multisite_tests: 2023-05-31T21:35:41.240 INFO:tasks.rgw_multisite_tests: 2023-05-31T21:35:41.241 INFO:tasks.rgw_multisite_tests:Wed, 31 May 2023 20:29:46 GMT 2023-05-31T21:35:41.241 INFO:tasks.rgw_multisite_tests:/pklxkd-55/b?versionId=null 2023-05-31T21:35:41.241 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-05-31T21:35:41.241 INFO:tasks.rgw_multisite_tests:AWS Q80H11OLNU1S48PPYL26:dohegYhb5mm2QkrqEQNUwnnKqu4= 2023-05-31T21:35:41.242 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:29:46 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:dohegYhb5mm2QkrqEQNUwnnKqu4=', 'Content-Length': '0'} 2023-05-31T21:35:41.242 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:27:16 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c3dfe8c7b2dab992-006477ae3a-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:29:46 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:35:41.242 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/pklxkd-55/b 2023-05-31T21:35:41.242 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/pklxkd-55/b 2023-05-31T21:35:41.243 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/pklxkd-55/b?versionId=null 2023-05-31T21:35:41.243 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/pklxkd-55/b?versionId=null 2023-05-31T21:35:41.243 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-05-31T21:35:41.243 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /pklxkd-55/b?versionId=null 2023-05-31T21:35:41.243 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-05-31T21:35:41.244 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-05-31T21:35:41.244 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T21:35:41.244 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-05-31T21:35:41.244 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-05-31T21:35:41.245 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-05-31T21:35:41.245 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-05-31T21:35:41.245 INFO:tasks.rgw_multisite_tests:HEAD 2023-05-31T21:35:41.245 INFO:tasks.rgw_multisite_tests: 2023-05-31T21:35:41.246 INFO:tasks.rgw_multisite_tests: 2023-05-31T21:35:41.246 INFO:tasks.rgw_multisite_tests:Wed, 31 May 2023 20:29:46 GMT 2023-05-31T21:35:41.246 INFO:tasks.rgw_multisite_tests:/pklxkd-55/b?versionId=null 2023-05-31T21:35:41.246 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-05-31T21:35:41.246 INFO:tasks.rgw_multisite_tests:AWS Q80H11OLNU1S48PPYL26:N8vDB2XHi9UTYW1fpiRhScLt+MU= 2023-05-31T21:35:41.247 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:29:46 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:N8vDB2XHi9UTYW1fpiRhScLt+MU=', 'Content-Length': '0'} 2023-05-31T21:35:41.247 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:27:16 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000089774d4fc910f08-006477ae3a-162e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:29:46 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:35:41.247 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/pklxkd-55/b 2023-05-31T21:35:41.247 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/pklxkd-55/b 2023-05-31T21:35:41.248 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/pklxkd-55/b?versionId=null 2023-05-31T21:35:41.248 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/pklxkd-55/b?versionId=null 2023-05-31T21:35:41.248 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-05-31T21:35:41.248 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /pklxkd-55/b?versionId=null 2023-05-31T21:35:41.249 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-05-31T21:35:41.249 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-05-31T21:35:41.249 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T21:35:41.249 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-05-31T21:35:41.249 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-05-31T21:35:41.250 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-05-31T21:35:41.250 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-05-31T21:35:41.250 INFO:tasks.rgw_multisite_tests:HEAD 2023-05-31T21:35:41.250 INFO:tasks.rgw_multisite_tests: 2023-05-31T21:35:41.251 INFO:tasks.rgw_multisite_tests: 2023-05-31T21:35:41.251 INFO:tasks.rgw_multisite_tests:Wed, 31 May 2023 20:29:46 GMT 2023-05-31T21:35:41.251 INFO:tasks.rgw_multisite_tests:/pklxkd-55/b?versionId=null 2023-05-31T21:35:41.251 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-05-31T21:35:41.251 INFO:tasks.rgw_multisite_tests:AWS Q80H11OLNU1S48PPYL26:N8vDB2XHi9UTYW1fpiRhScLt+MU= 2023-05-31T21:35:41.252 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:29:46 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:N8vDB2XHi9UTYW1fpiRhScLt+MU=', 'Content-Length': '0'} 2023-05-31T21:35:41.252 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:27:16 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d3007d24be31849e-006477ae3a-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:29:46 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:35:41.252 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: comparing key name=b 2023-05-31T21:35:41.252 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/pklxkd-55/b 2023-05-31T21:35:41.253 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/pklxkd-55/b 2023-05-31T21:35:41.253 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/pklxkd-55/b?versionId=null 2023-05-31T21:35:41.253 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/pklxkd-55/b?versionId=null 2023-05-31T21:35:41.253 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-05-31T21:35:41.254 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /pklxkd-55/b?versionId=null 2023-05-31T21:35:41.254 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-05-31T21:35:41.254 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-05-31T21:35:41.254 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T21:35:41.255 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-05-31T21:35:41.255 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-05-31T21:35:41.255 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-05-31T21:35:41.255 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-05-31T21:35:41.255 INFO:tasks.rgw_multisite_tests:GET 2023-05-31T21:35:41.256 INFO:tasks.rgw_multisite_tests: 2023-05-31T21:35:41.256 INFO:tasks.rgw_multisite_tests: 2023-05-31T21:35:41.256 INFO:tasks.rgw_multisite_tests:Wed, 31 May 2023 20:29:46 GMT 2023-05-31T21:35:41.256 INFO:tasks.rgw_multisite_tests:/pklxkd-55/b?versionId=null 2023-05-31T21:35:41.257 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-05-31T21:35:41.257 INFO:tasks.rgw_multisite_tests:AWS Q80H11OLNU1S48PPYL26:dohegYhb5mm2QkrqEQNUwnnKqu4= 2023-05-31T21:35:41.257 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:29:46 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:dohegYhb5mm2QkrqEQNUwnnKqu4=', 'Content-Length': '0'} 2023-05-31T21:35:41.257 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:27:16 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b91b038f54a4410a-006477ae3a-162e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:29:46 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:35:41.258 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/pklxkd-55/b 2023-05-31T21:35:41.258 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/pklxkd-55/b 2023-05-31T21:35:41.258 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/pklxkd-55/b?versionId=null 2023-05-31T21:35:41.258 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/pklxkd-55/b?versionId=null 2023-05-31T21:35:41.258 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-05-31T21:35:41.259 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /pklxkd-55/b?versionId=null 2023-05-31T21:35:41.259 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-05-31T21:35:41.259 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-05-31T21:35:41.259 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T21:35:41.260 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-05-31T21:35:41.260 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-05-31T21:35:41.260 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-05-31T21:35:41.260 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-05-31T21:35:41.261 INFO:tasks.rgw_multisite_tests:GET 2023-05-31T21:35:41.261 INFO:tasks.rgw_multisite_tests: 2023-05-31T21:35:41.261 INFO:tasks.rgw_multisite_tests: 2023-05-31T21:35:41.261 INFO:tasks.rgw_multisite_tests:Wed, 31 May 2023 20:29:46 GMT 2023-05-31T21:35:41.261 INFO:tasks.rgw_multisite_tests:/pklxkd-55/b?versionId=null 2023-05-31T21:35:41.262 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-05-31T21:35:41.262 INFO:tasks.rgw_multisite_tests:AWS Q80H11OLNU1S48PPYL26:dohegYhb5mm2QkrqEQNUwnnKqu4= 2023-05-31T21:35:41.262 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:29:46 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:dohegYhb5mm2QkrqEQNUwnnKqu4=', 'Content-Length': '0'} 2023-05-31T21:35:41.262 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:27:16 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000548aac9ce66b99b2-006477ae3a-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:29:46 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:35:41.263 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/pklxkd-55/ 2023-05-31T21:35:41.263 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/pklxkd-55/ 2023-05-31T21:35:41.263 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/pklxkd-55/?prefix=b 2023-05-31T21:35:41.263 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/pklxkd-55/?prefix=b 2023-05-31T21:35:41.264 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-05-31T21:35:41.264 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /pklxkd-55/?prefix=b 2023-05-31T21:35:41.264 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-05-31T21:35:41.264 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-05-31T21:35:41.264 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T21:35:41.265 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-05-31T21:35:41.265 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-05-31T21:35:41.265 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-05-31T21:35:41.265 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-05-31T21:35:41.266 INFO:tasks.rgw_multisite_tests:GET 2023-05-31T21:35:41.266 INFO:tasks.rgw_multisite_tests: 2023-05-31T21:35:41.266 INFO:tasks.rgw_multisite_tests: 2023-05-31T21:35:41.266 INFO:tasks.rgw_multisite_tests:Wed, 31 May 2023 20:29:46 GMT 2023-05-31T21:35:41.266 INFO:tasks.rgw_multisite_tests:/pklxkd-55/ 2023-05-31T21:35:41.267 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-05-31T21:35:41.267 INFO:tasks.rgw_multisite_tests:AWS Q80H11OLNU1S48PPYL26:XZ8tDAn1QJf1wBVwJgGN94Jp4hE= 2023-05-31T21:35:41.267 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:29:46 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:XZ8tDAn1QJf1wBVwJgGN94Jp4hE=', 'Content-Length': '0'} 2023-05-31T21:35:41.267 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000089edab617d68c7b4-006477ae3a-162e-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:29:46 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:35:41.268 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'pklxkd-55b1000falseb2023-05-31T20:27:16.850Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T21:35:41.268 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/pklxkd-55/ 2023-05-31T21:35:41.268 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/pklxkd-55/ 2023-05-31T21:35:41.268 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/pklxkd-55/?prefix=b 2023-05-31T21:35:41.269 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/pklxkd-55/?prefix=b 2023-05-31T21:35:41.269 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-05-31T21:35:41.269 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /pklxkd-55/?prefix=b 2023-05-31T21:35:41.269 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-05-31T21:35:41.269 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-05-31T21:35:41.270 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T21:35:41.270 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-05-31T21:35:41.270 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-05-31T21:35:41.270 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-05-31T21:35:41.271 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-05-31T21:35:41.271 INFO:tasks.rgw_multisite_tests:GET 2023-05-31T21:35:41.271 INFO:tasks.rgw_multisite_tests: 2023-05-31T21:35:41.271 INFO:tasks.rgw_multisite_tests: 2023-05-31T21:35:41.272 INFO:tasks.rgw_multisite_tests:Wed, 31 May 2023 20:29:46 GMT 2023-05-31T21:35:41.272 INFO:tasks.rgw_multisite_tests:/pklxkd-55/ 2023-05-31T21:35:41.272 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-05-31T21:35:41.272 INFO:tasks.rgw_multisite_tests:AWS Q80H11OLNU1S48PPYL26:XZ8tDAn1QJf1wBVwJgGN94Jp4hE= 2023-05-31T21:35:41.272 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:29:46 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:XZ8tDAn1QJf1wBVwJgGN94Jp4hE=', 'Content-Length': '0'} 2023-05-31T21:35:41.273 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000008085c4ec1a71ff97-006477ae3a-1630-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:29:46 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:35:41.273 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'pklxkd-55b1000falseb2023-05-31T20:27:16.850Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T21:35:41.273 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/pklxkd-55/b 2023-05-31T21:35:41.273 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/pklxkd-55/b 2023-05-31T21:35:41.274 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-05-31T21:35:41.274 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /pklxkd-55/b 2023-05-31T21:35:41.274 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-05-31T21:35:41.274 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-05-31T21:35:41.275 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T21:35:41.275 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-05-31T21:35:41.275 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-05-31T21:35:41.275 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-05-31T21:35:41.275 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-05-31T21:35:41.276 INFO:tasks.rgw_multisite_tests:HEAD 2023-05-31T21:35:41.276 INFO:tasks.rgw_multisite_tests: 2023-05-31T21:35:41.276 INFO:tasks.rgw_multisite_tests: 2023-05-31T21:35:41.276 INFO:tasks.rgw_multisite_tests:Wed, 31 May 2023 20:29:46 GMT 2023-05-31T21:35:41.277 INFO:tasks.rgw_multisite_tests:/pklxkd-55/b 2023-05-31T21:35:41.277 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-05-31T21:35:41.277 INFO:tasks.rgw_multisite_tests:AWS Q80H11OLNU1S48PPYL26:n5uR90dPfv0f03vlYT/EOZ2QVAo= 2023-05-31T21:35:41.277 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:29:46 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:n5uR90dPfv0f03vlYT/EOZ2QVAo=', 'Content-Length': '0'} 2023-05-31T21:35:41.277 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:27:16 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000001dd3f9bc1813902-006477ae3a-162e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:29:46 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:35:41.278 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/pklxkd-55/b 2023-05-31T21:35:41.278 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/pklxkd-55/b 2023-05-31T21:35:41.278 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-05-31T21:35:41.278 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /pklxkd-55/b 2023-05-31T21:35:41.279 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-05-31T21:35:41.279 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-05-31T21:35:41.279 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T21:35:41.279 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-05-31T21:35:41.280 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-05-31T21:35:41.280 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-05-31T21:35:41.280 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-05-31T21:35:41.280 INFO:tasks.rgw_multisite_tests:HEAD 2023-05-31T21:35:41.281 INFO:tasks.rgw_multisite_tests: 2023-05-31T21:35:41.281 INFO:tasks.rgw_multisite_tests: 2023-05-31T21:35:41.281 INFO:tasks.rgw_multisite_tests:Wed, 31 May 2023 20:29:46 GMT 2023-05-31T21:35:41.281 INFO:tasks.rgw_multisite_tests:/pklxkd-55/b 2023-05-31T21:35:41.282 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-05-31T21:35:41.282 INFO:tasks.rgw_multisite_tests:AWS Q80H11OLNU1S48PPYL26:n5uR90dPfv0f03vlYT/EOZ2QVAo= 2023-05-31T21:35:41.282 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:29:46 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:n5uR90dPfv0f03vlYT/EOZ2QVAo=', 'Content-Length': '0'} 2023-05-31T21:35:41.282 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:27:16 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000f9608d2c7e2d7a1-006477ae3a-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:29:46 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:35:41.282 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: comparing key name=b 2023-05-31T21:35:41.283 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/pklxkd-55/b 2023-05-31T21:35:41.283 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/pklxkd-55/b 2023-05-31T21:35:41.283 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-05-31T21:35:41.283 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /pklxkd-55/b 2023-05-31T21:35:41.284 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-05-31T21:35:41.284 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-05-31T21:35:41.284 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T21:35:41.284 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-05-31T21:35:41.285 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-05-31T21:35:41.285 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-05-31T21:35:41.285 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-05-31T21:35:41.285 INFO:tasks.rgw_multisite_tests:GET 2023-05-31T21:35:41.285 INFO:tasks.rgw_multisite_tests: 2023-05-31T21:35:41.286 INFO:tasks.rgw_multisite_tests: 2023-05-31T21:35:41.286 INFO:tasks.rgw_multisite_tests:Wed, 31 May 2023 20:29:46 GMT 2023-05-31T21:35:41.286 INFO:tasks.rgw_multisite_tests:/pklxkd-55/b 2023-05-31T21:35:41.286 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-05-31T21:35:41.287 INFO:tasks.rgw_multisite_tests:AWS Q80H11OLNU1S48PPYL26:wuXKUJQ/mLdFY/inCZsMAtJ0qfU= 2023-05-31T21:35:41.287 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:29:46 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:wuXKUJQ/mLdFY/inCZsMAtJ0qfU=', 'Content-Length': '0'} 2023-05-31T21:35:41.287 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:27:16 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000843d8ac137cd31f9-006477ae3a-162e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:29:46 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:35:41.287 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/pklxkd-55/b 2023-05-31T21:35:41.288 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/pklxkd-55/b 2023-05-31T21:35:41.288 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-05-31T21:35:41.288 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /pklxkd-55/b 2023-05-31T21:35:41.288 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-05-31T21:35:41.288 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-05-31T21:35:41.289 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T21:35:41.289 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-05-31T21:35:41.289 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-05-31T21:35:41.289 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-05-31T21:35:41.290 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-05-31T21:35:41.290 INFO:tasks.rgw_multisite_tests:GET 2023-05-31T21:35:41.290 INFO:tasks.rgw_multisite_tests: 2023-05-31T21:35:41.290 INFO:tasks.rgw_multisite_tests: 2023-05-31T21:35:41.291 INFO:tasks.rgw_multisite_tests:Wed, 31 May 2023 20:29:46 GMT 2023-05-31T21:35:41.291 INFO:tasks.rgw_multisite_tests:/pklxkd-55/b 2023-05-31T21:35:41.291 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-05-31T21:35:41.291 INFO:tasks.rgw_multisite_tests:AWS Q80H11OLNU1S48PPYL26:wuXKUJQ/mLdFY/inCZsMAtJ0qfU= 2023-05-31T21:35:41.292 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:29:46 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:wuXKUJQ/mLdFY/inCZsMAtJ0qfU=', 'Content-Length': '0'} 2023-05-31T21:35:41.292 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:27:16 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009aeb102391373f35-006477ae3a-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:29:46 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:35:41.292 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/pklxkd-55/ 2023-05-31T21:35:41.292 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/pklxkd-55/ 2023-05-31T21:35:41.293 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/pklxkd-55/?prefix=b 2023-05-31T21:35:41.293 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/pklxkd-55/?prefix=b 2023-05-31T21:35:41.293 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-05-31T21:35:41.293 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /pklxkd-55/?prefix=b 2023-05-31T21:35:41.293 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-05-31T21:35:41.294 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-05-31T21:35:41.294 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T21:35:41.294 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-05-31T21:35:41.294 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-05-31T21:35:41.295 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-05-31T21:35:41.295 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-05-31T21:35:41.295 INFO:tasks.rgw_multisite_tests:GET 2023-05-31T21:35:41.295 INFO:tasks.rgw_multisite_tests: 2023-05-31T21:35:41.295 INFO:tasks.rgw_multisite_tests: 2023-05-31T21:35:41.296 INFO:tasks.rgw_multisite_tests:Wed, 31 May 2023 20:29:46 GMT 2023-05-31T21:35:41.296 INFO:tasks.rgw_multisite_tests:/pklxkd-55/ 2023-05-31T21:35:41.296 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-05-31T21:35:41.296 INFO:tasks.rgw_multisite_tests:AWS Q80H11OLNU1S48PPYL26:XZ8tDAn1QJf1wBVwJgGN94Jp4hE= 2023-05-31T21:35:41.297 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:29:46 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:XZ8tDAn1QJf1wBVwJgGN94Jp4hE=', 'Content-Length': '0'} 2023-05-31T21:35:41.297 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000094264aa1eb44dc55-006477ae3a-162e-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:29:46 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:35:41.297 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'pklxkd-55b1000falseb2023-05-31T20:27:16.850Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T21:35:41.297 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/pklxkd-55/ 2023-05-31T21:35:41.298 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/pklxkd-55/ 2023-05-31T21:35:41.298 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/pklxkd-55/?prefix=b 2023-05-31T21:35:41.298 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/pklxkd-55/?prefix=b 2023-05-31T21:35:41.298 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-05-31T21:35:41.298 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /pklxkd-55/?prefix=b 2023-05-31T21:35:41.299 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-05-31T21:35:41.299 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-05-31T21:35:41.299 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T21:35:41.299 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-05-31T21:35:41.300 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-05-31T21:35:41.300 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-05-31T21:35:41.300 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-05-31T21:35:41.300 INFO:tasks.rgw_multisite_tests:GET 2023-05-31T21:35:41.301 INFO:tasks.rgw_multisite_tests: 2023-05-31T21:35:41.301 INFO:tasks.rgw_multisite_tests: 2023-05-31T21:35:41.301 INFO:tasks.rgw_multisite_tests:Wed, 31 May 2023 20:29:46 GMT 2023-05-31T21:35:41.301 INFO:tasks.rgw_multisite_tests:/pklxkd-55/ 2023-05-31T21:35:41.301 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-05-31T21:35:41.302 INFO:tasks.rgw_multisite_tests:AWS Q80H11OLNU1S48PPYL26:XZ8tDAn1QJf1wBVwJgGN94Jp4hE= 2023-05-31T21:35:41.302 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:29:46 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:XZ8tDAn1QJf1wBVwJgGN94Jp4hE=', 'Content-Length': '0'} 2023-05-31T21:35:41.302 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000313a9ec53a3169bc-006477ae3a-1630-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:29:46 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:35:41.302 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'pklxkd-55b1000falseb2023-05-31T20:27:16.850Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T21:35:41.303 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: comparing key name=c 2023-05-31T21:35:41.303 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/pklxkd-55/c 2023-05-31T21:35:41.303 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/pklxkd-55/c 2023-05-31T21:35:41.303 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/pklxkd-55/c?versionId=null 2023-05-31T21:35:41.304 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/pklxkd-55/c?versionId=null 2023-05-31T21:35:41.304 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-05-31T21:35:41.304 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /pklxkd-55/c?versionId=null 2023-05-31T21:35:41.304 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-05-31T21:35:41.304 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-05-31T21:35:41.305 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T21:35:41.305 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-05-31T21:35:41.305 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-05-31T21:35:41.305 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-05-31T21:35:41.306 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-05-31T21:35:41.306 INFO:tasks.rgw_multisite_tests:GET 2023-05-31T21:35:41.306 INFO:tasks.rgw_multisite_tests: 2023-05-31T21:35:41.306 INFO:tasks.rgw_multisite_tests: 2023-05-31T21:35:41.307 INFO:tasks.rgw_multisite_tests:Wed, 31 May 2023 20:29:46 GMT 2023-05-31T21:35:41.307 INFO:tasks.rgw_multisite_tests:/pklxkd-55/c?versionId=null 2023-05-31T21:35:41.307 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-05-31T21:35:41.307 INFO:tasks.rgw_multisite_tests:AWS Q80H11OLNU1S48PPYL26:V8bcEaCXTJmgXE/9Lo6ni/7nA3s= 2023-05-31T21:35:41.307 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:29:46 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:V8bcEaCXTJmgXE/9Lo6ni/7nA3s=', 'Content-Length': '0'} 2023-05-31T21:35:41.308 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:27:16 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e70e8edbbd535d16-006477ae3a-162e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:29:46 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:35:41.308 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/pklxkd-55/c 2023-05-31T21:35:41.308 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/pklxkd-55/c 2023-05-31T21:35:41.308 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/pklxkd-55/c?versionId=null 2023-05-31T21:35:41.309 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/pklxkd-55/c?versionId=null 2023-05-31T21:35:41.309 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-05-31T21:35:41.309 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /pklxkd-55/c?versionId=null 2023-05-31T21:35:41.309 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-05-31T21:35:41.310 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-05-31T21:35:41.310 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T21:35:41.310 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-05-31T21:35:41.310 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-05-31T21:35:41.310 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-05-31T21:35:41.311 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-05-31T21:35:41.311 INFO:tasks.rgw_multisite_tests:GET 2023-05-31T21:35:41.311 INFO:tasks.rgw_multisite_tests: 2023-05-31T21:35:41.311 INFO:tasks.rgw_multisite_tests: 2023-05-31T21:35:41.312 INFO:tasks.rgw_multisite_tests:Wed, 31 May 2023 20:29:46 GMT 2023-05-31T21:35:41.312 INFO:tasks.rgw_multisite_tests:/pklxkd-55/c?versionId=null 2023-05-31T21:35:41.312 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-05-31T21:35:41.312 INFO:tasks.rgw_multisite_tests:AWS Q80H11OLNU1S48PPYL26:V8bcEaCXTJmgXE/9Lo6ni/7nA3s= 2023-05-31T21:35:41.312 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:29:46 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:V8bcEaCXTJmgXE/9Lo6ni/7nA3s=', 'Content-Length': '0'} 2023-05-31T21:35:41.313 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:27:16 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000adfc976e8c841118-006477ae3a-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:29:46 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:35:41.313 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/pklxkd-55/c 2023-05-31T21:35:41.313 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/pklxkd-55/c 2023-05-31T21:35:41.313 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/pklxkd-55/c?versionId=null 2023-05-31T21:35:41.314 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/pklxkd-55/c?versionId=null 2023-05-31T21:35:41.314 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-05-31T21:35:41.314 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /pklxkd-55/c?versionId=null 2023-05-31T21:35:41.314 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-05-31T21:35:41.315 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-05-31T21:35:41.315 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T21:35:41.315 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-05-31T21:35:41.315 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-05-31T21:35:41.315 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-05-31T21:35:41.316 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-05-31T21:35:41.316 INFO:tasks.rgw_multisite_tests:HEAD 2023-05-31T21:35:41.316 INFO:tasks.rgw_multisite_tests: 2023-05-31T21:35:41.316 INFO:tasks.rgw_multisite_tests: 2023-05-31T21:35:41.317 INFO:tasks.rgw_multisite_tests:Wed, 31 May 2023 20:29:46 GMT 2023-05-31T21:35:41.317 INFO:tasks.rgw_multisite_tests:/pklxkd-55/c?versionId=null 2023-05-31T21:35:41.317 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-05-31T21:35:41.317 INFO:tasks.rgw_multisite_tests:AWS Q80H11OLNU1S48PPYL26:AqMFlyYVuhYdkuTQgQwv/WdMYzM= 2023-05-31T21:35:41.318 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:29:46 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:AqMFlyYVuhYdkuTQgQwv/WdMYzM=', 'Content-Length': '0'} 2023-05-31T21:35:41.318 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:27:16 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f38eddbe82c86087-006477ae3a-162e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:29:46 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:35:41.318 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/pklxkd-55/c 2023-05-31T21:35:41.318 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/pklxkd-55/c 2023-05-31T21:35:41.318 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/pklxkd-55/c?versionId=null 2023-05-31T21:35:41.319 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/pklxkd-55/c?versionId=null 2023-05-31T21:35:41.319 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-05-31T21:35:41.319 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /pklxkd-55/c?versionId=null 2023-05-31T21:35:41.319 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-05-31T21:35:41.320 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-05-31T21:35:41.320 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T21:35:41.320 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-05-31T21:35:41.320 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-05-31T21:35:41.321 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-05-31T21:35:41.321 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-05-31T21:35:41.321 INFO:tasks.rgw_multisite_tests:HEAD 2023-05-31T21:35:41.321 INFO:tasks.rgw_multisite_tests: 2023-05-31T21:35:41.321 INFO:tasks.rgw_multisite_tests: 2023-05-31T21:35:41.322 INFO:tasks.rgw_multisite_tests:Wed, 31 May 2023 20:29:46 GMT 2023-05-31T21:35:41.322 INFO:tasks.rgw_multisite_tests:/pklxkd-55/c?versionId=null 2023-05-31T21:35:41.322 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-05-31T21:35:41.322 INFO:tasks.rgw_multisite_tests:AWS Q80H11OLNU1S48PPYL26:AqMFlyYVuhYdkuTQgQwv/WdMYzM= 2023-05-31T21:35:41.323 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:29:46 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:AqMFlyYVuhYdkuTQgQwv/WdMYzM=', 'Content-Length': '0'} 2023-05-31T21:35:41.323 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:27:16 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007e437de488f05cf4-006477ae3a-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:29:46 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:35:41.323 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: comparing key name=c 2023-05-31T21:35:41.323 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/pklxkd-55/c 2023-05-31T21:35:41.324 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/pklxkd-55/c 2023-05-31T21:35:41.324 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/pklxkd-55/c?versionId=null 2023-05-31T21:35:41.324 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/pklxkd-55/c?versionId=null 2023-05-31T21:35:41.324 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-05-31T21:35:41.324 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /pklxkd-55/c?versionId=null 2023-05-31T21:35:41.325 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-05-31T21:35:41.325 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-05-31T21:35:41.325 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T21:35:41.325 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-05-31T21:35:41.326 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-05-31T21:35:41.326 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-05-31T21:35:41.326 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-05-31T21:35:41.326 INFO:tasks.rgw_multisite_tests:GET 2023-05-31T21:35:41.327 INFO:tasks.rgw_multisite_tests: 2023-05-31T21:35:41.327 INFO:tasks.rgw_multisite_tests: 2023-05-31T21:35:41.327 INFO:tasks.rgw_multisite_tests:Wed, 31 May 2023 20:29:46 GMT 2023-05-31T21:35:41.327 INFO:tasks.rgw_multisite_tests:/pklxkd-55/c?versionId=null 2023-05-31T21:35:41.327 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-05-31T21:35:41.328 INFO:tasks.rgw_multisite_tests:AWS Q80H11OLNU1S48PPYL26:V8bcEaCXTJmgXE/9Lo6ni/7nA3s= 2023-05-31T21:35:41.328 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:29:46 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:V8bcEaCXTJmgXE/9Lo6ni/7nA3s=', 'Content-Length': '0'} 2023-05-31T21:35:41.328 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:27:16 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000071f9186ae9818d14-006477ae3a-162e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:29:46 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:35:41.328 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/pklxkd-55/c 2023-05-31T21:35:41.329 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/pklxkd-55/c 2023-05-31T21:35:41.329 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/pklxkd-55/c?versionId=null 2023-05-31T21:35:41.329 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/pklxkd-55/c?versionId=null 2023-05-31T21:35:41.329 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-05-31T21:35:41.330 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /pklxkd-55/c?versionId=null 2023-05-31T21:35:41.330 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-05-31T21:35:41.330 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-05-31T21:35:41.330 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T21:35:41.331 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-05-31T21:35:41.331 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-05-31T21:35:41.331 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-05-31T21:35:41.331 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-05-31T21:35:41.332 INFO:tasks.rgw_multisite_tests:GET 2023-05-31T21:35:41.332 INFO:tasks.rgw_multisite_tests: 2023-05-31T21:35:41.332 INFO:tasks.rgw_multisite_tests: 2023-05-31T21:35:41.332 INFO:tasks.rgw_multisite_tests:Wed, 31 May 2023 20:29:47 GMT 2023-05-31T21:35:41.332 INFO:tasks.rgw_multisite_tests:/pklxkd-55/c?versionId=null 2023-05-31T21:35:41.333 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-05-31T21:35:41.333 INFO:tasks.rgw_multisite_tests:AWS Q80H11OLNU1S48PPYL26:P1aXLMUH6zOhTLbQHnvGsASl6uk= 2023-05-31T21:35:41.333 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:29:47 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:P1aXLMUH6zOhTLbQHnvGsASl6uk=', 'Content-Length': '0'} 2023-05-31T21:35:41.333 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:27:16 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006ba618b21ca7f01d-006477ae3b-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:29:47 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:35:41.334 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/pklxkd-55/ 2023-05-31T21:35:41.334 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/pklxkd-55/ 2023-05-31T21:35:41.334 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/pklxkd-55/?prefix=c 2023-05-31T21:35:41.334 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/pklxkd-55/?prefix=c 2023-05-31T21:35:41.335 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-05-31T21:35:41.335 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /pklxkd-55/?prefix=c 2023-05-31T21:35:41.335 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-05-31T21:35:41.335 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-05-31T21:35:41.335 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T21:35:41.336 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-05-31T21:35:41.336 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-05-31T21:35:41.336 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-05-31T21:35:41.336 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-05-31T21:35:41.337 INFO:tasks.rgw_multisite_tests:GET 2023-05-31T21:35:41.337 INFO:tasks.rgw_multisite_tests: 2023-05-31T21:35:41.337 INFO:tasks.rgw_multisite_tests: 2023-05-31T21:35:41.337 INFO:tasks.rgw_multisite_tests:Wed, 31 May 2023 20:29:47 GMT 2023-05-31T21:35:41.338 INFO:tasks.rgw_multisite_tests:/pklxkd-55/ 2023-05-31T21:35:41.338 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-05-31T21:35:41.338 INFO:tasks.rgw_multisite_tests:AWS Q80H11OLNU1S48PPYL26:IBdSyRVlh8qg/aH9XxJ5KZfq+k4= 2023-05-31T21:35:41.338 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:29:47 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:IBdSyRVlh8qg/aH9XxJ5KZfq+k4=', 'Content-Length': '0'} 2023-05-31T21:35:41.338 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000001921d21d924c3074-006477ae3b-162e-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:29:47 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:35:41.339 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'pklxkd-55c1000falsec2023-05-31T20:27:16.869Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T21:35:41.339 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/pklxkd-55/ 2023-05-31T21:35:41.339 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/pklxkd-55/ 2023-05-31T21:35:41.339 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/pklxkd-55/?prefix=c 2023-05-31T21:35:41.340 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/pklxkd-55/?prefix=c 2023-05-31T21:35:41.340 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-05-31T21:35:41.340 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /pklxkd-55/?prefix=c 2023-05-31T21:35:41.340 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-05-31T21:35:41.341 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-05-31T21:35:41.341 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T21:35:41.341 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-05-31T21:35:41.341 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-05-31T21:35:41.341 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-05-31T21:35:41.342 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-05-31T21:35:41.342 INFO:tasks.rgw_multisite_tests:GET 2023-05-31T21:35:41.342 INFO:tasks.rgw_multisite_tests: 2023-05-31T21:35:41.342 INFO:tasks.rgw_multisite_tests: 2023-05-31T21:35:41.343 INFO:tasks.rgw_multisite_tests:Wed, 31 May 2023 20:29:47 GMT 2023-05-31T21:35:41.343 INFO:tasks.rgw_multisite_tests:/pklxkd-55/ 2023-05-31T21:35:41.343 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-05-31T21:35:41.343 INFO:tasks.rgw_multisite_tests:AWS Q80H11OLNU1S48PPYL26:IBdSyRVlh8qg/aH9XxJ5KZfq+k4= 2023-05-31T21:35:41.343 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:29:47 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:IBdSyRVlh8qg/aH9XxJ5KZfq+k4=', 'Content-Length': '0'} 2023-05-31T21:35:41.344 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c3c76626a0ec93ef-006477ae3b-1630-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:29:47 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:35:41.344 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'pklxkd-55c1000falsec2023-05-31T20:27:16.869Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T21:35:41.344 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/pklxkd-55/c 2023-05-31T21:35:41.344 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/pklxkd-55/c 2023-05-31T21:35:41.345 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-05-31T21:35:41.345 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /pklxkd-55/c 2023-05-31T21:35:41.345 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-05-31T21:35:41.345 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-05-31T21:35:41.346 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T21:35:41.346 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-05-31T21:35:41.346 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-05-31T21:35:41.346 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-05-31T21:35:41.347 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-05-31T21:35:41.347 INFO:tasks.rgw_multisite_tests:HEAD 2023-05-31T21:35:41.347 INFO:tasks.rgw_multisite_tests: 2023-05-31T21:35:41.347 INFO:tasks.rgw_multisite_tests: 2023-05-31T21:35:41.347 INFO:tasks.rgw_multisite_tests:Wed, 31 May 2023 20:29:47 GMT 2023-05-31T21:35:41.348 INFO:tasks.rgw_multisite_tests:/pklxkd-55/c 2023-05-31T21:35:41.348 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-05-31T21:35:41.348 INFO:tasks.rgw_multisite_tests:AWS Q80H11OLNU1S48PPYL26:vjS8H3RSqPpmqDEM9WRL9MuHsJQ= 2023-05-31T21:35:41.348 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:29:47 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:vjS8H3RSqPpmqDEM9WRL9MuHsJQ=', 'Content-Length': '0'} 2023-05-31T21:35:41.349 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:27:16 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000849f06dc367ce32b-006477ae3b-162e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:29:47 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:35:41.349 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/pklxkd-55/c 2023-05-31T21:35:41.349 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/pklxkd-55/c 2023-05-31T21:35:41.349 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-05-31T21:35:41.350 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /pklxkd-55/c 2023-05-31T21:35:41.350 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-05-31T21:35:41.350 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-05-31T21:35:41.350 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T21:35:41.350 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-05-31T21:35:41.351 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-05-31T21:35:41.351 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-05-31T21:35:41.351 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-05-31T21:35:41.351 INFO:tasks.rgw_multisite_tests:HEAD 2023-05-31T21:35:41.352 INFO:tasks.rgw_multisite_tests: 2023-05-31T21:35:41.352 INFO:tasks.rgw_multisite_tests: 2023-05-31T21:35:41.352 INFO:tasks.rgw_multisite_tests:Wed, 31 May 2023 20:29:47 GMT 2023-05-31T21:35:41.352 INFO:tasks.rgw_multisite_tests:/pklxkd-55/c 2023-05-31T21:35:41.353 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-05-31T21:35:41.353 INFO:tasks.rgw_multisite_tests:AWS Q80H11OLNU1S48PPYL26:vjS8H3RSqPpmqDEM9WRL9MuHsJQ= 2023-05-31T21:35:41.353 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:29:47 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:vjS8H3RSqPpmqDEM9WRL9MuHsJQ=', 'Content-Length': '0'} 2023-05-31T21:35:41.353 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:27:16 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f939ed76af465b2d-006477ae3b-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:29:47 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:35:41.353 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: comparing key name=c 2023-05-31T21:35:41.354 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/pklxkd-55/c 2023-05-31T21:35:41.354 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/pklxkd-55/c 2023-05-31T21:35:41.354 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-05-31T21:35:41.354 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /pklxkd-55/c 2023-05-31T21:35:41.355 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-05-31T21:35:41.355 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-05-31T21:35:41.355 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T21:35:41.355 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-05-31T21:35:41.355 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-05-31T21:35:41.356 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-05-31T21:35:41.356 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-05-31T21:35:41.356 INFO:tasks.rgw_multisite_tests:GET 2023-05-31T21:35:41.356 INFO:tasks.rgw_multisite_tests: 2023-05-31T21:35:41.357 INFO:tasks.rgw_multisite_tests: 2023-05-31T21:35:41.357 INFO:tasks.rgw_multisite_tests:Wed, 31 May 2023 20:29:47 GMT 2023-05-31T21:35:41.357 INFO:tasks.rgw_multisite_tests:/pklxkd-55/c 2023-05-31T21:35:41.357 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-05-31T21:35:41.358 INFO:tasks.rgw_multisite_tests:AWS Q80H11OLNU1S48PPYL26:KCsebAKjHoJs41UM5GosrCgfQCs= 2023-05-31T21:35:41.358 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:29:47 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:KCsebAKjHoJs41UM5GosrCgfQCs=', 'Content-Length': '0'} 2023-05-31T21:35:41.358 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:27:16 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000019df14a7220e0b1b-006477ae3b-162e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:29:47 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:35:41.358 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/pklxkd-55/c 2023-05-31T21:35:41.358 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/pklxkd-55/c 2023-05-31T21:35:41.359 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-05-31T21:35:41.359 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /pklxkd-55/c 2023-05-31T21:35:41.359 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-05-31T21:35:41.359 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-05-31T21:35:41.360 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T21:35:41.360 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-05-31T21:35:41.360 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-05-31T21:35:41.360 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-05-31T21:35:41.361 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-05-31T21:35:41.361 INFO:tasks.rgw_multisite_tests:GET 2023-05-31T21:35:41.361 INFO:tasks.rgw_multisite_tests: 2023-05-31T21:35:41.361 INFO:tasks.rgw_multisite_tests: 2023-05-31T21:35:41.361 INFO:tasks.rgw_multisite_tests:Wed, 31 May 2023 20:29:47 GMT 2023-05-31T21:35:41.362 INFO:tasks.rgw_multisite_tests:/pklxkd-55/c 2023-05-31T21:35:41.362 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-05-31T21:35:41.362 INFO:tasks.rgw_multisite_tests:AWS Q80H11OLNU1S48PPYL26:KCsebAKjHoJs41UM5GosrCgfQCs= 2023-05-31T21:35:41.362 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:29:47 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:KCsebAKjHoJs41UM5GosrCgfQCs=', 'Content-Length': '0'} 2023-05-31T21:35:41.363 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:27:16 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000008809774c2acaf8b-006477ae3b-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:29:47 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:35:41.363 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/pklxkd-55/ 2023-05-31T21:35:41.363 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/pklxkd-55/ 2023-05-31T21:35:41.363 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/pklxkd-55/?prefix=c 2023-05-31T21:35:41.364 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/pklxkd-55/?prefix=c 2023-05-31T21:35:41.364 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-05-31T21:35:41.364 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /pklxkd-55/?prefix=c 2023-05-31T21:35:41.364 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-05-31T21:35:41.364 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-05-31T21:35:41.365 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T21:35:41.365 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-05-31T21:35:41.365 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-05-31T21:35:41.365 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-05-31T21:35:41.366 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-05-31T21:35:41.366 INFO:tasks.rgw_multisite_tests:GET 2023-05-31T21:35:41.366 INFO:tasks.rgw_multisite_tests: 2023-05-31T21:35:41.366 INFO:tasks.rgw_multisite_tests: 2023-05-31T21:35:41.366 INFO:tasks.rgw_multisite_tests:Wed, 31 May 2023 20:29:47 GMT 2023-05-31T21:35:41.367 INFO:tasks.rgw_multisite_tests:/pklxkd-55/ 2023-05-31T21:35:41.367 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-05-31T21:35:41.367 INFO:tasks.rgw_multisite_tests:AWS Q80H11OLNU1S48PPYL26:IBdSyRVlh8qg/aH9XxJ5KZfq+k4= 2023-05-31T21:35:41.367 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:29:47 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:IBdSyRVlh8qg/aH9XxJ5KZfq+k4=', 'Content-Length': '0'} 2023-05-31T21:35:41.368 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000debb8a925a227acd-006477ae3b-162e-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:29:47 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:35:41.368 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'pklxkd-55c1000falsec2023-05-31T20:27:16.869Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T21:35:41.368 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/pklxkd-55/ 2023-05-31T21:35:41.368 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/pklxkd-55/ 2023-05-31T21:35:41.369 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/pklxkd-55/?prefix=c 2023-05-31T21:35:41.369 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/pklxkd-55/?prefix=c 2023-05-31T21:35:41.369 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-05-31T21:35:41.369 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /pklxkd-55/?prefix=c 2023-05-31T21:35:41.369 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-05-31T21:35:41.370 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-05-31T21:35:41.370 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T21:35:41.370 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-05-31T21:35:41.370 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-05-31T21:35:41.371 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-05-31T21:35:41.371 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-05-31T21:35:41.371 INFO:tasks.rgw_multisite_tests:GET 2023-05-31T21:35:41.371 INFO:tasks.rgw_multisite_tests: 2023-05-31T21:35:41.372 INFO:tasks.rgw_multisite_tests: 2023-05-31T21:35:41.372 INFO:tasks.rgw_multisite_tests:Wed, 31 May 2023 20:29:47 GMT 2023-05-31T21:35:41.372 INFO:tasks.rgw_multisite_tests:/pklxkd-55/ 2023-05-31T21:35:41.372 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-05-31T21:35:41.373 INFO:tasks.rgw_multisite_tests:AWS Q80H11OLNU1S48PPYL26:IBdSyRVlh8qg/aH9XxJ5KZfq+k4= 2023-05-31T21:35:41.373 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:29:47 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:IBdSyRVlh8qg/aH9XxJ5KZfq+k4=', 'Content-Length': '0'} 2023-05-31T21:35:41.373 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a1f4a69a81fbeb79-006477ae3b-1630-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:29:47 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:35:41.373 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'pklxkd-55c1000falsec2023-05-31T20:27:16.869Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T21:35:41.373 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: comparing key name=d 2023-05-31T21:35:41.374 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/pklxkd-55/d 2023-05-31T21:35:41.374 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/pklxkd-55/d 2023-05-31T21:35:41.374 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/pklxkd-55/d?versionId=null 2023-05-31T21:35:41.374 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/pklxkd-55/d?versionId=null 2023-05-31T21:35:41.375 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-05-31T21:35:41.375 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /pklxkd-55/d?versionId=null 2023-05-31T21:35:41.375 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-05-31T21:35:41.375 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-05-31T21:35:41.375 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T21:35:41.376 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-05-31T21:35:41.376 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-05-31T21:35:41.376 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-05-31T21:35:41.376 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-05-31T21:35:41.377 INFO:tasks.rgw_multisite_tests:GET 2023-05-31T21:35:41.377 INFO:tasks.rgw_multisite_tests: 2023-05-31T21:35:41.377 INFO:tasks.rgw_multisite_tests: 2023-05-31T21:35:41.377 INFO:tasks.rgw_multisite_tests:Wed, 31 May 2023 20:29:47 GMT 2023-05-31T21:35:41.378 INFO:tasks.rgw_multisite_tests:/pklxkd-55/d?versionId=null 2023-05-31T21:35:41.378 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-05-31T21:35:41.378 INFO:tasks.rgw_multisite_tests:AWS Q80H11OLNU1S48PPYL26:2iEJWxnjGeNLda7D6Nei7TwGGLE= 2023-05-31T21:35:41.378 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:29:47 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:2iEJWxnjGeNLda7D6Nei7TwGGLE=', 'Content-Length': '0'} 2023-05-31T21:35:41.379 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:27:16 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000091d5d692949f5efa-006477ae3b-162e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:29:47 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:35:41.379 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/pklxkd-55/d 2023-05-31T21:35:41.379 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/pklxkd-55/d 2023-05-31T21:35:41.379 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/pklxkd-55/d?versionId=null 2023-05-31T21:35:41.379 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/pklxkd-55/d?versionId=null 2023-05-31T21:35:41.380 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-05-31T21:35:41.380 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /pklxkd-55/d?versionId=null 2023-05-31T21:35:41.380 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-05-31T21:35:41.380 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-05-31T21:35:41.381 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T21:35:41.381 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-05-31T21:35:41.381 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-05-31T21:35:41.381 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-05-31T21:35:41.382 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-05-31T21:35:41.382 INFO:tasks.rgw_multisite_tests:GET 2023-05-31T21:35:41.382 INFO:tasks.rgw_multisite_tests: 2023-05-31T21:35:41.382 INFO:tasks.rgw_multisite_tests: 2023-05-31T21:35:41.383 INFO:tasks.rgw_multisite_tests:Wed, 31 May 2023 20:29:47 GMT 2023-05-31T21:35:41.383 INFO:tasks.rgw_multisite_tests:/pklxkd-55/d?versionId=null 2023-05-31T21:35:41.383 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-05-31T21:35:41.383 INFO:tasks.rgw_multisite_tests:AWS Q80H11OLNU1S48PPYL26:2iEJWxnjGeNLda7D6Nei7TwGGLE= 2023-05-31T21:35:41.383 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:29:47 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:2iEJWxnjGeNLda7D6Nei7TwGGLE=', 'Content-Length': '0'} 2023-05-31T21:35:41.384 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:27:16 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000bc20baa0a0f923b3-006477ae3b-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:29:47 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:35:41.384 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/pklxkd-55/d 2023-05-31T21:35:41.384 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/pklxkd-55/d 2023-05-31T21:35:41.384 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/pklxkd-55/d?versionId=null 2023-05-31T21:35:41.385 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/pklxkd-55/d?versionId=null 2023-05-31T21:35:41.385 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-05-31T21:35:41.385 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /pklxkd-55/d?versionId=null 2023-05-31T21:35:41.385 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-05-31T21:35:41.386 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-05-31T21:35:41.386 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T21:35:41.386 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-05-31T21:35:41.386 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-05-31T21:35:41.387 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-05-31T21:35:41.387 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-05-31T21:35:41.387 INFO:tasks.rgw_multisite_tests:HEAD 2023-05-31T21:35:41.387 INFO:tasks.rgw_multisite_tests: 2023-05-31T21:35:41.387 INFO:tasks.rgw_multisite_tests: 2023-05-31T21:35:41.388 INFO:tasks.rgw_multisite_tests:Wed, 31 May 2023 20:29:47 GMT 2023-05-31T21:35:41.388 INFO:tasks.rgw_multisite_tests:/pklxkd-55/d?versionId=null 2023-05-31T21:35:41.388 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-05-31T21:35:41.388 INFO:tasks.rgw_multisite_tests:AWS Q80H11OLNU1S48PPYL26:harZR6uuoONBLYwl9WLoBi2jBDY= 2023-05-31T21:35:41.389 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:29:47 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:harZR6uuoONBLYwl9WLoBi2jBDY=', 'Content-Length': '0'} 2023-05-31T21:35:41.389 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:27:16 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004d43da428e4e7492-006477ae3b-162e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:29:47 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:35:41.389 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/pklxkd-55/d 2023-05-31T21:35:41.389 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/pklxkd-55/d 2023-05-31T21:35:41.390 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/pklxkd-55/d?versionId=null 2023-05-31T21:35:41.390 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/pklxkd-55/d?versionId=null 2023-05-31T21:35:41.390 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-05-31T21:35:41.390 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /pklxkd-55/d?versionId=null 2023-05-31T21:35:41.391 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-05-31T21:35:41.391 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-05-31T21:35:41.391 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T21:35:41.391 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-05-31T21:35:41.392 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-05-31T21:35:41.392 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-05-31T21:35:41.392 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-05-31T21:35:41.392 INFO:tasks.rgw_multisite_tests:HEAD 2023-05-31T21:35:41.393 INFO:tasks.rgw_multisite_tests: 2023-05-31T21:35:41.393 INFO:tasks.rgw_multisite_tests: 2023-05-31T21:35:41.393 INFO:tasks.rgw_multisite_tests:Wed, 31 May 2023 20:29:47 GMT 2023-05-31T21:35:41.393 INFO:tasks.rgw_multisite_tests:/pklxkd-55/d?versionId=null 2023-05-31T21:35:41.393 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-05-31T21:35:41.394 INFO:tasks.rgw_multisite_tests:AWS Q80H11OLNU1S48PPYL26:harZR6uuoONBLYwl9WLoBi2jBDY= 2023-05-31T21:35:41.394 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:29:47 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:harZR6uuoONBLYwl9WLoBi2jBDY=', 'Content-Length': '0'} 2023-05-31T21:35:41.394 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:27:16 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000420c48f5e0e401ce-006477ae3b-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:29:47 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:35:41.394 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: comparing key name=d 2023-05-31T21:35:41.395 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/pklxkd-55/d 2023-05-31T21:35:41.395 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/pklxkd-55/d 2023-05-31T21:35:41.395 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/pklxkd-55/d?versionId=null 2023-05-31T21:35:41.395 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/pklxkd-55/d?versionId=null 2023-05-31T21:35:41.396 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-05-31T21:35:41.396 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /pklxkd-55/d?versionId=null 2023-05-31T21:35:41.396 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-05-31T21:35:41.396 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-05-31T21:35:41.396 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T21:35:41.397 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-05-31T21:35:41.397 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-05-31T21:35:41.397 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-05-31T21:35:41.397 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-05-31T21:35:41.398 INFO:tasks.rgw_multisite_tests:GET 2023-05-31T21:35:41.398 INFO:tasks.rgw_multisite_tests: 2023-05-31T21:35:41.398 INFO:tasks.rgw_multisite_tests: 2023-05-31T21:35:41.398 INFO:tasks.rgw_multisite_tests:Wed, 31 May 2023 20:29:47 GMT 2023-05-31T21:35:41.399 INFO:tasks.rgw_multisite_tests:/pklxkd-55/d?versionId=null 2023-05-31T21:35:41.399 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-05-31T21:35:41.399 INFO:tasks.rgw_multisite_tests:AWS Q80H11OLNU1S48PPYL26:2iEJWxnjGeNLda7D6Nei7TwGGLE= 2023-05-31T21:35:41.399 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:29:47 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:2iEJWxnjGeNLda7D6Nei7TwGGLE=', 'Content-Length': '0'} 2023-05-31T21:35:41.399 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:27:16 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ed00b7dce04f021f-006477ae3b-162e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:29:47 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:35:41.400 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/pklxkd-55/d 2023-05-31T21:35:41.400 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/pklxkd-55/d 2023-05-31T21:35:41.400 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/pklxkd-55/d?versionId=null 2023-05-31T21:35:41.400 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/pklxkd-55/d?versionId=null 2023-05-31T21:35:41.401 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-05-31T21:35:41.401 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /pklxkd-55/d?versionId=null 2023-05-31T21:35:41.401 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-05-31T21:35:41.401 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-05-31T21:35:41.402 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T21:35:41.402 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-05-31T21:35:41.402 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-05-31T21:35:41.402 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-05-31T21:35:41.402 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-05-31T21:35:41.403 INFO:tasks.rgw_multisite_tests:GET 2023-05-31T21:35:41.403 INFO:tasks.rgw_multisite_tests: 2023-05-31T21:35:41.403 INFO:tasks.rgw_multisite_tests: 2023-05-31T21:35:41.403 INFO:tasks.rgw_multisite_tests:Wed, 31 May 2023 20:29:47 GMT 2023-05-31T21:35:41.404 INFO:tasks.rgw_multisite_tests:/pklxkd-55/d?versionId=null 2023-05-31T21:35:41.404 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-05-31T21:35:41.404 INFO:tasks.rgw_multisite_tests:AWS Q80H11OLNU1S48PPYL26:2iEJWxnjGeNLda7D6Nei7TwGGLE= 2023-05-31T21:35:41.404 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:29:47 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:2iEJWxnjGeNLda7D6Nei7TwGGLE=', 'Content-Length': '0'} 2023-05-31T21:35:41.405 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:27:16 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009b6945514ae92903-006477ae3b-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:29:47 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:35:41.405 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/pklxkd-55/ 2023-05-31T21:35:41.405 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/pklxkd-55/ 2023-05-31T21:35:41.405 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/pklxkd-55/?prefix=d 2023-05-31T21:35:41.405 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/pklxkd-55/?prefix=d 2023-05-31T21:35:41.406 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-05-31T21:35:41.406 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /pklxkd-55/?prefix=d 2023-05-31T21:35:41.406 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-05-31T21:35:41.406 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-05-31T21:35:41.407 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T21:35:41.407 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-05-31T21:35:41.407 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-05-31T21:35:41.407 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-05-31T21:35:41.408 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-05-31T21:35:41.408 INFO:tasks.rgw_multisite_tests:GET 2023-05-31T21:35:41.408 INFO:tasks.rgw_multisite_tests: 2023-05-31T21:35:41.408 INFO:tasks.rgw_multisite_tests: 2023-05-31T21:35:41.408 INFO:tasks.rgw_multisite_tests:Wed, 31 May 2023 20:29:47 GMT 2023-05-31T21:35:41.409 INFO:tasks.rgw_multisite_tests:/pklxkd-55/ 2023-05-31T21:35:41.409 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-05-31T21:35:41.409 INFO:tasks.rgw_multisite_tests:AWS Q80H11OLNU1S48PPYL26:IBdSyRVlh8qg/aH9XxJ5KZfq+k4= 2023-05-31T21:35:41.409 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:29:47 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:IBdSyRVlh8qg/aH9XxJ5KZfq+k4=', 'Content-Length': '0'} 2023-05-31T21:35:41.410 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000009e5b2d166e730dc-006477ae3b-162e-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:29:47 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:35:41.410 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'pklxkd-55d1000falsed2023-05-31T20:27:16.893Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T21:35:41.410 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/pklxkd-55/ 2023-05-31T21:35:41.410 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/pklxkd-55/ 2023-05-31T21:35:41.411 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/pklxkd-55/?prefix=d 2023-05-31T21:35:41.411 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/pklxkd-55/?prefix=d 2023-05-31T21:35:41.411 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-05-31T21:35:41.411 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /pklxkd-55/?prefix=d 2023-05-31T21:35:41.411 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-05-31T21:35:41.412 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-05-31T21:35:41.412 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T21:35:41.412 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-05-31T21:35:41.412 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-05-31T21:35:41.413 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-05-31T21:35:41.413 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-05-31T21:35:41.413 INFO:tasks.rgw_multisite_tests:GET 2023-05-31T21:35:41.413 INFO:tasks.rgw_multisite_tests: 2023-05-31T21:35:41.414 INFO:tasks.rgw_multisite_tests: 2023-05-31T21:35:41.414 INFO:tasks.rgw_multisite_tests:Wed, 31 May 2023 20:29:47 GMT 2023-05-31T21:35:41.414 INFO:tasks.rgw_multisite_tests:/pklxkd-55/ 2023-05-31T21:35:41.414 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-05-31T21:35:41.414 INFO:tasks.rgw_multisite_tests:AWS Q80H11OLNU1S48PPYL26:IBdSyRVlh8qg/aH9XxJ5KZfq+k4= 2023-05-31T21:35:41.415 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:29:47 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:IBdSyRVlh8qg/aH9XxJ5KZfq+k4=', 'Content-Length': '0'} 2023-05-31T21:35:41.415 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000003ae87ca4fefd366a-006477ae3b-1630-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:29:47 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:35:41.415 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'pklxkd-55d1000falsed2023-05-31T20:27:16.893Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T21:35:41.415 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/pklxkd-55/d 2023-05-31T21:35:41.416 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/pklxkd-55/d 2023-05-31T21:35:41.416 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-05-31T21:35:41.416 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /pklxkd-55/d 2023-05-31T21:35:41.416 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-05-31T21:35:41.417 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-05-31T21:35:41.417 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T21:35:41.417 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-05-31T21:35:41.417 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-05-31T21:35:41.417 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-05-31T21:35:41.418 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-05-31T21:35:41.418 INFO:tasks.rgw_multisite_tests:HEAD 2023-05-31T21:35:41.418 INFO:tasks.rgw_multisite_tests: 2023-05-31T21:35:41.418 INFO:tasks.rgw_multisite_tests: 2023-05-31T21:35:41.419 INFO:tasks.rgw_multisite_tests:Wed, 31 May 2023 20:29:47 GMT 2023-05-31T21:35:41.419 INFO:tasks.rgw_multisite_tests:/pklxkd-55/d 2023-05-31T21:35:41.419 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-05-31T21:35:41.419 INFO:tasks.rgw_multisite_tests:AWS Q80H11OLNU1S48PPYL26:X+9cyPTt250zU6XVmvtAhT2OEYY= 2023-05-31T21:35:41.419 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:29:47 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:X+9cyPTt250zU6XVmvtAhT2OEYY=', 'Content-Length': '0'} 2023-05-31T21:35:41.420 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:27:16 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000083015d841141e613-006477ae3b-162e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:29:47 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:35:41.420 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/pklxkd-55/d 2023-05-31T21:35:41.420 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/pklxkd-55/d 2023-05-31T21:35:41.420 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-05-31T21:35:41.421 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /pklxkd-55/d 2023-05-31T21:35:41.421 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-05-31T21:35:41.421 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-05-31T21:35:41.421 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T21:35:41.422 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-05-31T21:35:41.422 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-05-31T21:35:41.422 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-05-31T21:35:41.422 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-05-31T21:35:41.423 INFO:tasks.rgw_multisite_tests:HEAD 2023-05-31T21:35:41.423 INFO:tasks.rgw_multisite_tests: 2023-05-31T21:35:41.423 INFO:tasks.rgw_multisite_tests: 2023-05-31T21:35:41.423 INFO:tasks.rgw_multisite_tests:Wed, 31 May 2023 20:29:47 GMT 2023-05-31T21:35:41.423 INFO:tasks.rgw_multisite_tests:/pklxkd-55/d 2023-05-31T21:35:41.424 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-05-31T21:35:41.424 INFO:tasks.rgw_multisite_tests:AWS Q80H11OLNU1S48PPYL26:X+9cyPTt250zU6XVmvtAhT2OEYY= 2023-05-31T21:35:41.424 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:29:47 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:X+9cyPTt250zU6XVmvtAhT2OEYY=', 'Content-Length': '0'} 2023-05-31T21:35:41.424 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:27:16 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000700134a3fe4ade89-006477ae3b-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:29:47 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:35:41.425 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: comparing key name=d 2023-05-31T21:35:41.425 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/pklxkd-55/d 2023-05-31T21:35:41.425 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/pklxkd-55/d 2023-05-31T21:35:41.425 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-05-31T21:35:41.426 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /pklxkd-55/d 2023-05-31T21:35:41.426 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-05-31T21:35:41.426 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-05-31T21:35:41.426 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T21:35:41.427 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-05-31T21:35:41.427 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-05-31T21:35:41.427 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-05-31T21:35:41.427 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-05-31T21:35:41.427 INFO:tasks.rgw_multisite_tests:GET 2023-05-31T21:35:41.428 INFO:tasks.rgw_multisite_tests: 2023-05-31T21:35:41.428 INFO:tasks.rgw_multisite_tests: 2023-05-31T21:35:41.428 INFO:tasks.rgw_multisite_tests:Wed, 31 May 2023 20:29:47 GMT 2023-05-31T21:35:41.428 INFO:tasks.rgw_multisite_tests:/pklxkd-55/d 2023-05-31T21:35:41.429 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-05-31T21:35:41.429 INFO:tasks.rgw_multisite_tests:AWS Q80H11OLNU1S48PPYL26:7NERFesywT7Ehxd1WNnCZkSPCw8= 2023-05-31T21:35:41.429 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:29:47 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:7NERFesywT7Ehxd1WNnCZkSPCw8=', 'Content-Length': '0'} 2023-05-31T21:35:41.429 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:27:16 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000013e2afc8bf7b0126-006477ae3b-162e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:29:47 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:35:41.430 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/pklxkd-55/d 2023-05-31T21:35:41.430 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/pklxkd-55/d 2023-05-31T21:35:41.430 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-05-31T21:35:41.430 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /pklxkd-55/d 2023-05-31T21:35:41.430 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-05-31T21:35:41.431 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-05-31T21:35:41.431 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T21:35:41.431 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-05-31T21:35:41.431 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-05-31T21:35:41.432 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-05-31T21:35:41.432 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-05-31T21:35:41.432 INFO:tasks.rgw_multisite_tests:GET 2023-05-31T21:35:41.432 INFO:tasks.rgw_multisite_tests: 2023-05-31T21:35:41.433 INFO:tasks.rgw_multisite_tests: 2023-05-31T21:35:41.433 INFO:tasks.rgw_multisite_tests:Wed, 31 May 2023 20:29:47 GMT 2023-05-31T21:35:41.433 INFO:tasks.rgw_multisite_tests:/pklxkd-55/d 2023-05-31T21:35:41.433 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-05-31T21:35:41.433 INFO:tasks.rgw_multisite_tests:AWS Q80H11OLNU1S48PPYL26:7NERFesywT7Ehxd1WNnCZkSPCw8= 2023-05-31T21:35:41.434 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:29:47 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:7NERFesywT7Ehxd1WNnCZkSPCw8=', 'Content-Length': '0'} 2023-05-31T21:35:41.434 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:27:16 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000bbec8960dcaf26fd-006477ae3b-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:29:47 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:35:41.434 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/pklxkd-55/ 2023-05-31T21:35:41.435 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/pklxkd-55/ 2023-05-31T21:35:41.435 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/pklxkd-55/?prefix=d 2023-05-31T21:35:41.435 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/pklxkd-55/?prefix=d 2023-05-31T21:35:41.435 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-05-31T21:35:41.436 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /pklxkd-55/?prefix=d 2023-05-31T21:35:41.436 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-05-31T21:35:41.436 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-05-31T21:35:41.436 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T21:35:41.437 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-05-31T21:35:41.437 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-05-31T21:35:41.437 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-05-31T21:35:41.437 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-05-31T21:35:41.437 INFO:tasks.rgw_multisite_tests:GET 2023-05-31T21:35:41.438 INFO:tasks.rgw_multisite_tests: 2023-05-31T21:35:41.438 INFO:tasks.rgw_multisite_tests: 2023-05-31T21:35:41.438 INFO:tasks.rgw_multisite_tests:Wed, 31 May 2023 20:29:47 GMT 2023-05-31T21:35:41.438 INFO:tasks.rgw_multisite_tests:/pklxkd-55/ 2023-05-31T21:35:41.439 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-05-31T21:35:41.439 INFO:tasks.rgw_multisite_tests:AWS Q80H11OLNU1S48PPYL26:IBdSyRVlh8qg/aH9XxJ5KZfq+k4= 2023-05-31T21:35:41.439 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:29:47 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:IBdSyRVlh8qg/aH9XxJ5KZfq+k4=', 'Content-Length': '0'} 2023-05-31T21:35:41.439 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000053ecf5a10431c038-006477ae3b-162e-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:29:47 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:35:41.440 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'pklxkd-55d1000falsed2023-05-31T20:27:16.893Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T21:35:41.440 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/pklxkd-55/ 2023-05-31T21:35:41.440 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/pklxkd-55/ 2023-05-31T21:35:41.440 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/pklxkd-55/?prefix=d 2023-05-31T21:35:41.440 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/pklxkd-55/?prefix=d 2023-05-31T21:35:41.441 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-05-31T21:35:41.441 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /pklxkd-55/?prefix=d 2023-05-31T21:35:41.441 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-05-31T21:35:41.441 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-05-31T21:35:41.442 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T21:35:41.442 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-05-31T21:35:41.442 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-05-31T21:35:41.442 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-05-31T21:35:41.443 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-05-31T21:35:41.443 INFO:tasks.rgw_multisite_tests:GET 2023-05-31T21:35:41.443 INFO:tasks.rgw_multisite_tests: 2023-05-31T21:35:41.443 INFO:tasks.rgw_multisite_tests: 2023-05-31T21:35:41.443 INFO:tasks.rgw_multisite_tests:Wed, 31 May 2023 20:29:47 GMT 2023-05-31T21:35:41.444 INFO:tasks.rgw_multisite_tests:/pklxkd-55/ 2023-05-31T21:35:41.444 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-05-31T21:35:41.444 INFO:tasks.rgw_multisite_tests:AWS Q80H11OLNU1S48PPYL26:IBdSyRVlh8qg/aH9XxJ5KZfq+k4= 2023-05-31T21:35:41.444 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:29:47 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:IBdSyRVlh8qg/aH9XxJ5KZfq+k4=', 'Content-Length': '0'} 2023-05-31T21:35:41.445 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c25f6a88831bfcd2-006477ae3b-1630-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:29:47 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:35:41.445 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'pklxkd-55d1000falsed2023-05-31T20:27:16.893Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T21:35:41.445 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: comparing key name=e 2023-05-31T21:35:41.445 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/pklxkd-55/e 2023-05-31T21:35:41.446 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/pklxkd-55/e 2023-05-31T21:35:41.446 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/pklxkd-55/e?versionId=null 2023-05-31T21:35:41.446 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/pklxkd-55/e?versionId=null 2023-05-31T21:35:41.446 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-05-31T21:35:41.447 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /pklxkd-55/e?versionId=null 2023-05-31T21:35:41.447 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-05-31T21:35:41.447 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-05-31T21:35:41.447 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T21:35:41.448 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-05-31T21:35:41.448 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-05-31T21:35:41.448 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-05-31T21:35:41.448 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-05-31T21:35:41.448 INFO:tasks.rgw_multisite_tests:GET 2023-05-31T21:35:41.449 INFO:tasks.rgw_multisite_tests: 2023-05-31T21:35:41.449 INFO:tasks.rgw_multisite_tests: 2023-05-31T21:35:41.449 INFO:tasks.rgw_multisite_tests:Wed, 31 May 2023 20:29:47 GMT 2023-05-31T21:35:41.449 INFO:tasks.rgw_multisite_tests:/pklxkd-55/e?versionId=null 2023-05-31T21:35:41.450 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-05-31T21:35:41.450 INFO:tasks.rgw_multisite_tests:AWS Q80H11OLNU1S48PPYL26:dbj3RRF7g23SIKkaPNS7dnpciek= 2023-05-31T21:35:41.450 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:29:47 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:dbj3RRF7g23SIKkaPNS7dnpciek=', 'Content-Length': '0'} 2023-05-31T21:35:41.450 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:27:23 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000094d68dcb50968389-006477ae3b-162e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:29:47 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:35:41.451 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/pklxkd-55/e 2023-05-31T21:35:41.451 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/pklxkd-55/e 2023-05-31T21:35:41.451 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/pklxkd-55/e?versionId=null 2023-05-31T21:35:41.451 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/pklxkd-55/e?versionId=null 2023-05-31T21:35:41.451 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-05-31T21:35:41.452 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /pklxkd-55/e?versionId=null 2023-05-31T21:35:41.452 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-05-31T21:35:41.452 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-05-31T21:35:41.452 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T21:35:41.453 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-05-31T21:35:41.453 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-05-31T21:35:41.453 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-05-31T21:35:41.453 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-05-31T21:35:41.454 INFO:tasks.rgw_multisite_tests:GET 2023-05-31T21:35:41.454 INFO:tasks.rgw_multisite_tests: 2023-05-31T21:35:41.454 INFO:tasks.rgw_multisite_tests: 2023-05-31T21:35:41.454 INFO:tasks.rgw_multisite_tests:Wed, 31 May 2023 20:29:47 GMT 2023-05-31T21:35:41.454 INFO:tasks.rgw_multisite_tests:/pklxkd-55/e?versionId=null 2023-05-31T21:35:41.455 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-05-31T21:35:41.455 INFO:tasks.rgw_multisite_tests:AWS Q80H11OLNU1S48PPYL26:dbj3RRF7g23SIKkaPNS7dnpciek= 2023-05-31T21:35:41.455 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:29:47 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:dbj3RRF7g23SIKkaPNS7dnpciek=', 'Content-Length': '0'} 2023-05-31T21:35:41.455 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:27:23 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000225c84ef46c560a-006477ae3b-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:29:47 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:35:41.456 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/pklxkd-55/e 2023-05-31T21:35:41.456 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/pklxkd-55/e 2023-05-31T21:35:41.456 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/pklxkd-55/e?versionId=null 2023-05-31T21:35:41.456 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/pklxkd-55/e?versionId=null 2023-05-31T21:35:41.457 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-05-31T21:35:41.457 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /pklxkd-55/e?versionId=null 2023-05-31T21:35:41.457 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-05-31T21:35:41.457 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-05-31T21:35:41.458 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T21:35:41.458 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-05-31T21:35:41.458 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-05-31T21:35:41.458 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-05-31T21:35:41.458 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-05-31T21:35:41.459 INFO:tasks.rgw_multisite_tests:HEAD 2023-05-31T21:35:41.459 INFO:tasks.rgw_multisite_tests: 2023-05-31T21:35:41.459 INFO:tasks.rgw_multisite_tests: 2023-05-31T21:35:41.459 INFO:tasks.rgw_multisite_tests:Wed, 31 May 2023 20:29:47 GMT 2023-05-31T21:35:41.460 INFO:tasks.rgw_multisite_tests:/pklxkd-55/e?versionId=null 2023-05-31T21:35:41.460 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-05-31T21:35:41.460 INFO:tasks.rgw_multisite_tests:AWS Q80H11OLNU1S48PPYL26:EMisUW3qdqzH3u7tNtCSn9vJCek= 2023-05-31T21:35:41.460 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:29:47 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:EMisUW3qdqzH3u7tNtCSn9vJCek=', 'Content-Length': '0'} 2023-05-31T21:35:41.461 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:27:23 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f4d2ee4fb4efa063-006477ae3b-162e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:29:47 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:35:41.461 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/pklxkd-55/e 2023-05-31T21:35:41.461 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/pklxkd-55/e 2023-05-31T21:35:41.461 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/pklxkd-55/e?versionId=null 2023-05-31T21:35:41.462 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/pklxkd-55/e?versionId=null 2023-05-31T21:35:41.462 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-05-31T21:35:41.462 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /pklxkd-55/e?versionId=null 2023-05-31T21:35:41.462 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-05-31T21:35:41.462 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-05-31T21:35:41.463 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T21:35:41.463 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-05-31T21:35:41.463 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-05-31T21:35:41.463 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-05-31T21:35:41.464 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-05-31T21:35:41.464 INFO:tasks.rgw_multisite_tests:HEAD 2023-05-31T21:35:41.464 INFO:tasks.rgw_multisite_tests: 2023-05-31T21:35:41.464 INFO:tasks.rgw_multisite_tests: 2023-05-31T21:35:41.465 INFO:tasks.rgw_multisite_tests:Wed, 31 May 2023 20:29:47 GMT 2023-05-31T21:35:41.465 INFO:tasks.rgw_multisite_tests:/pklxkd-55/e?versionId=null 2023-05-31T21:35:41.465 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-05-31T21:35:41.465 INFO:tasks.rgw_multisite_tests:AWS Q80H11OLNU1S48PPYL26:EMisUW3qdqzH3u7tNtCSn9vJCek= 2023-05-31T21:35:41.465 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:29:47 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:EMisUW3qdqzH3u7tNtCSn9vJCek=', 'Content-Length': '0'} 2023-05-31T21:35:41.466 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:27:23 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000285f373464c67f62-006477ae3b-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:29:47 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:35:41.466 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: comparing key name=e 2023-05-31T21:35:41.466 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/pklxkd-55/e 2023-05-31T21:35:41.466 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/pklxkd-55/e 2023-05-31T21:35:41.467 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/pklxkd-55/e?versionId=null 2023-05-31T21:35:41.467 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/pklxkd-55/e?versionId=null 2023-05-31T21:35:41.467 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-05-31T21:35:41.467 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /pklxkd-55/e?versionId=null 2023-05-31T21:35:41.468 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-05-31T21:35:41.468 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-05-31T21:35:41.468 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T21:35:41.468 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-05-31T21:35:41.468 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-05-31T21:35:41.469 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-05-31T21:35:41.469 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-05-31T21:35:41.469 INFO:tasks.rgw_multisite_tests:GET 2023-05-31T21:35:41.469 INFO:tasks.rgw_multisite_tests: 2023-05-31T21:35:41.470 INFO:tasks.rgw_multisite_tests: 2023-05-31T21:35:41.470 INFO:tasks.rgw_multisite_tests:Wed, 31 May 2023 20:29:47 GMT 2023-05-31T21:35:41.470 INFO:tasks.rgw_multisite_tests:/pklxkd-55/e?versionId=null 2023-05-31T21:35:41.470 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-05-31T21:35:41.470 INFO:tasks.rgw_multisite_tests:AWS Q80H11OLNU1S48PPYL26:dbj3RRF7g23SIKkaPNS7dnpciek= 2023-05-31T21:35:41.471 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:29:47 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:dbj3RRF7g23SIKkaPNS7dnpciek=', 'Content-Length': '0'} 2023-05-31T21:35:41.471 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:27:23 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005f3bd3662479f769-006477ae3b-162e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:29:47 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:35:41.471 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/pklxkd-55/e 2023-05-31T21:35:41.471 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/pklxkd-55/e 2023-05-31T21:35:41.472 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/pklxkd-55/e?versionId=null 2023-05-31T21:35:41.472 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/pklxkd-55/e?versionId=null 2023-05-31T21:35:41.472 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-05-31T21:35:41.472 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /pklxkd-55/e?versionId=null 2023-05-31T21:35:41.473 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-05-31T21:35:41.473 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-05-31T21:35:41.473 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T21:35:41.473 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-05-31T21:35:41.474 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-05-31T21:35:41.474 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-05-31T21:35:41.474 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-05-31T21:35:41.474 INFO:tasks.rgw_multisite_tests:GET 2023-05-31T21:35:41.474 INFO:tasks.rgw_multisite_tests: 2023-05-31T21:35:41.475 INFO:tasks.rgw_multisite_tests: 2023-05-31T21:35:41.475 INFO:tasks.rgw_multisite_tests:Wed, 31 May 2023 20:29:47 GMT 2023-05-31T21:35:41.475 INFO:tasks.rgw_multisite_tests:/pklxkd-55/e?versionId=null 2023-05-31T21:35:41.475 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-05-31T21:35:41.476 INFO:tasks.rgw_multisite_tests:AWS Q80H11OLNU1S48PPYL26:dbj3RRF7g23SIKkaPNS7dnpciek= 2023-05-31T21:35:41.476 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:29:47 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:dbj3RRF7g23SIKkaPNS7dnpciek=', 'Content-Length': '0'} 2023-05-31T21:35:41.476 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:27:23 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000736a400a3b7da12a-006477ae3b-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:29:47 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:35:41.476 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/pklxkd-55/ 2023-05-31T21:35:41.476 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/pklxkd-55/ 2023-05-31T21:35:41.477 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/pklxkd-55/?prefix=e 2023-05-31T21:35:41.477 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/pklxkd-55/?prefix=e 2023-05-31T21:35:41.477 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-05-31T21:35:41.477 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /pklxkd-55/?prefix=e 2023-05-31T21:35:41.478 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-05-31T21:35:41.478 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-05-31T21:35:41.478 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T21:35:41.478 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-05-31T21:35:41.479 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-05-31T21:35:41.479 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-05-31T21:35:41.479 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-05-31T21:35:41.479 INFO:tasks.rgw_multisite_tests:GET 2023-05-31T21:35:41.479 INFO:tasks.rgw_multisite_tests: 2023-05-31T21:35:41.480 INFO:tasks.rgw_multisite_tests: 2023-05-31T21:35:41.480 INFO:tasks.rgw_multisite_tests:Wed, 31 May 2023 20:29:47 GMT 2023-05-31T21:35:41.480 INFO:tasks.rgw_multisite_tests:/pklxkd-55/ 2023-05-31T21:35:41.480 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-05-31T21:35:41.481 INFO:tasks.rgw_multisite_tests:AWS Q80H11OLNU1S48PPYL26:IBdSyRVlh8qg/aH9XxJ5KZfq+k4= 2023-05-31T21:35:41.481 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:29:47 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:IBdSyRVlh8qg/aH9XxJ5KZfq+k4=', 'Content-Length': '0'} 2023-05-31T21:35:41.481 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000002ab5d7da6605e3c4-006477ae3b-162e-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:29:47 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:35:41.481 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'pklxkd-55e1000falsee2023-05-31T20:27:23.955Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T21:35:41.482 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/pklxkd-55/ 2023-05-31T21:35:41.482 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/pklxkd-55/ 2023-05-31T21:35:41.482 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/pklxkd-55/?prefix=e 2023-05-31T21:35:41.482 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/pklxkd-55/?prefix=e 2023-05-31T21:35:41.483 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-05-31T21:35:41.483 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /pklxkd-55/?prefix=e 2023-05-31T21:35:41.483 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-05-31T21:35:41.483 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-05-31T21:35:41.483 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T21:35:41.484 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-05-31T21:35:41.484 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-05-31T21:35:41.484 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-05-31T21:35:41.484 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-05-31T21:35:41.485 INFO:tasks.rgw_multisite_tests:GET 2023-05-31T21:35:41.485 INFO:tasks.rgw_multisite_tests: 2023-05-31T21:35:41.485 INFO:tasks.rgw_multisite_tests: 2023-05-31T21:35:41.485 INFO:tasks.rgw_multisite_tests:Wed, 31 May 2023 20:29:47 GMT 2023-05-31T21:35:41.486 INFO:tasks.rgw_multisite_tests:/pklxkd-55/ 2023-05-31T21:35:41.486 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-05-31T21:35:41.486 INFO:tasks.rgw_multisite_tests:AWS Q80H11OLNU1S48PPYL26:IBdSyRVlh8qg/aH9XxJ5KZfq+k4= 2023-05-31T21:35:41.486 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:29:47 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:IBdSyRVlh8qg/aH9XxJ5KZfq+k4=', 'Content-Length': '0'} 2023-05-31T21:35:41.486 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000005505522a4c96dd97-006477ae3b-1630-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:29:47 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:35:41.487 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'pklxkd-55e1000falsee2023-05-31T20:27:23.955Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T21:35:41.487 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/pklxkd-55/e 2023-05-31T21:35:41.487 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/pklxkd-55/e 2023-05-31T21:35:41.487 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-05-31T21:35:41.488 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /pklxkd-55/e 2023-05-31T21:35:41.488 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-05-31T21:35:41.488 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-05-31T21:35:41.488 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T21:35:41.489 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-05-31T21:35:41.489 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-05-31T21:35:41.489 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-05-31T21:35:41.489 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-05-31T21:35:41.489 INFO:tasks.rgw_multisite_tests:HEAD 2023-05-31T21:35:41.490 INFO:tasks.rgw_multisite_tests: 2023-05-31T21:35:41.490 INFO:tasks.rgw_multisite_tests: 2023-05-31T21:35:41.490 INFO:tasks.rgw_multisite_tests:Wed, 31 May 2023 20:29:47 GMT 2023-05-31T21:35:41.490 INFO:tasks.rgw_multisite_tests:/pklxkd-55/e 2023-05-31T21:35:41.491 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-05-31T21:35:41.491 INFO:tasks.rgw_multisite_tests:AWS Q80H11OLNU1S48PPYL26:BH7veQe7nRdctzqYNJfbli6Qsqs= 2023-05-31T21:35:41.491 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:29:47 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:BH7veQe7nRdctzqYNJfbli6Qsqs=', 'Content-Length': '0'} 2023-05-31T21:35:41.491 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:27:23 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000013834ba2d3a89bc8-006477ae3b-162e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:29:47 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:35:41.492 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/pklxkd-55/e 2023-05-31T21:35:41.492 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/pklxkd-55/e 2023-05-31T21:35:41.492 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-05-31T21:35:41.492 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /pklxkd-55/e 2023-05-31T21:35:41.493 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-05-31T21:35:41.493 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-05-31T21:35:41.493 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T21:35:41.493 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-05-31T21:35:41.493 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-05-31T21:35:41.494 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-05-31T21:35:41.494 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-05-31T21:35:41.494 INFO:tasks.rgw_multisite_tests:HEAD 2023-05-31T21:35:41.494 INFO:tasks.rgw_multisite_tests: 2023-05-31T21:35:41.495 INFO:tasks.rgw_multisite_tests: 2023-05-31T21:35:41.495 INFO:tasks.rgw_multisite_tests:Wed, 31 May 2023 20:29:47 GMT 2023-05-31T21:35:41.495 INFO:tasks.rgw_multisite_tests:/pklxkd-55/e 2023-05-31T21:35:41.495 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-05-31T21:35:41.496 INFO:tasks.rgw_multisite_tests:AWS Q80H11OLNU1S48PPYL26:BH7veQe7nRdctzqYNJfbli6Qsqs= 2023-05-31T21:35:41.496 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:29:47 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:BH7veQe7nRdctzqYNJfbli6Qsqs=', 'Content-Length': '0'} 2023-05-31T21:35:41.496 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:27:23 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a6cd2b6b34890636-006477ae3b-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:29:47 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:35:41.496 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: comparing key name=e 2023-05-31T21:35:41.497 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/pklxkd-55/e 2023-05-31T21:35:41.497 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/pklxkd-55/e 2023-05-31T21:35:41.497 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-05-31T21:35:41.497 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /pklxkd-55/e 2023-05-31T21:35:41.497 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-05-31T21:35:41.498 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-05-31T21:35:41.498 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T21:35:41.498 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-05-31T21:35:41.498 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-05-31T21:35:41.499 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-05-31T21:35:41.499 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-05-31T21:35:41.499 INFO:tasks.rgw_multisite_tests:GET 2023-05-31T21:35:41.499 INFO:tasks.rgw_multisite_tests: 2023-05-31T21:35:41.500 INFO:tasks.rgw_multisite_tests: 2023-05-31T21:35:41.500 INFO:tasks.rgw_multisite_tests:Wed, 31 May 2023 20:29:47 GMT 2023-05-31T21:35:41.500 INFO:tasks.rgw_multisite_tests:/pklxkd-55/e 2023-05-31T21:35:41.500 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-05-31T21:35:41.500 INFO:tasks.rgw_multisite_tests:AWS Q80H11OLNU1S48PPYL26:lVX4sPK/Kzs95E32XjhOwYvD868= 2023-05-31T21:35:41.501 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:29:47 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:lVX4sPK/Kzs95E32XjhOwYvD868=', 'Content-Length': '0'} 2023-05-31T21:35:41.501 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:27:23 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003e5fd2a0107bc72c-006477ae3b-162e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:29:47 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:35:41.501 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/pklxkd-55/e 2023-05-31T21:35:41.501 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/pklxkd-55/e 2023-05-31T21:35:41.502 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-05-31T21:35:41.502 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /pklxkd-55/e 2023-05-31T21:35:41.502 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-05-31T21:35:41.502 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-05-31T21:35:41.503 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T21:35:41.503 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-05-31T21:35:41.503 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-05-31T21:35:41.503 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-05-31T21:35:41.503 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-05-31T21:35:41.504 INFO:tasks.rgw_multisite_tests:GET 2023-05-31T21:35:41.504 INFO:tasks.rgw_multisite_tests: 2023-05-31T21:35:41.504 INFO:tasks.rgw_multisite_tests: 2023-05-31T21:35:41.504 INFO:tasks.rgw_multisite_tests:Wed, 31 May 2023 20:29:47 GMT 2023-05-31T21:35:41.505 INFO:tasks.rgw_multisite_tests:/pklxkd-55/e 2023-05-31T21:35:41.505 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-05-31T21:35:41.505 INFO:tasks.rgw_multisite_tests:AWS Q80H11OLNU1S48PPYL26:lVX4sPK/Kzs95E32XjhOwYvD868= 2023-05-31T21:35:41.505 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:29:47 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:lVX4sPK/Kzs95E32XjhOwYvD868=', 'Content-Length': '0'} 2023-05-31T21:35:41.506 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:27:23 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005ccb1768bfb0e521-006477ae3b-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:29:47 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:35:41.506 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/pklxkd-55/ 2023-05-31T21:35:41.506 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/pklxkd-55/ 2023-05-31T21:35:41.506 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/pklxkd-55/?prefix=e 2023-05-31T21:35:41.507 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/pklxkd-55/?prefix=e 2023-05-31T21:35:41.507 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-05-31T21:35:41.507 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /pklxkd-55/?prefix=e 2023-05-31T21:35:41.507 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-05-31T21:35:41.507 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-05-31T21:35:41.508 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T21:35:41.508 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-05-31T21:35:41.508 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-05-31T21:35:41.508 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-05-31T21:35:41.509 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-05-31T21:35:41.509 INFO:tasks.rgw_multisite_tests:GET 2023-05-31T21:35:41.509 INFO:tasks.rgw_multisite_tests: 2023-05-31T21:35:41.509 INFO:tasks.rgw_multisite_tests: 2023-05-31T21:35:41.510 INFO:tasks.rgw_multisite_tests:Wed, 31 May 2023 20:29:47 GMT 2023-05-31T21:35:41.510 INFO:tasks.rgw_multisite_tests:/pklxkd-55/ 2023-05-31T21:35:41.510 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-05-31T21:35:41.510 INFO:tasks.rgw_multisite_tests:AWS Q80H11OLNU1S48PPYL26:IBdSyRVlh8qg/aH9XxJ5KZfq+k4= 2023-05-31T21:35:41.510 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:29:47 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:IBdSyRVlh8qg/aH9XxJ5KZfq+k4=', 'Content-Length': '0'} 2023-05-31T21:35:41.511 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e3d429776c8f9d39-006477ae3b-162e-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:29:47 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:35:41.511 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'pklxkd-55e1000falsee2023-05-31T20:27:23.955Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T21:35:41.511 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/pklxkd-55/ 2023-05-31T21:35:41.511 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/pklxkd-55/ 2023-05-31T21:35:41.512 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/pklxkd-55/?prefix=e 2023-05-31T21:35:41.512 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/pklxkd-55/?prefix=e 2023-05-31T21:35:41.512 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-05-31T21:35:41.512 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /pklxkd-55/?prefix=e 2023-05-31T21:35:41.513 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-05-31T21:35:41.513 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-05-31T21:35:41.513 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T21:35:41.513 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-05-31T21:35:41.513 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-05-31T21:35:41.514 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-05-31T21:35:41.514 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-05-31T21:35:41.514 INFO:tasks.rgw_multisite_tests:GET 2023-05-31T21:35:41.514 INFO:tasks.rgw_multisite_tests: 2023-05-31T21:35:41.515 INFO:tasks.rgw_multisite_tests: 2023-05-31T21:35:41.515 INFO:tasks.rgw_multisite_tests:Wed, 31 May 2023 20:29:47 GMT 2023-05-31T21:35:41.515 INFO:tasks.rgw_multisite_tests:/pklxkd-55/ 2023-05-31T21:35:41.515 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-05-31T21:35:41.516 INFO:tasks.rgw_multisite_tests:AWS Q80H11OLNU1S48PPYL26:IBdSyRVlh8qg/aH9XxJ5KZfq+k4= 2023-05-31T21:35:41.516 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:29:47 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:IBdSyRVlh8qg/aH9XxJ5KZfq+k4=', 'Content-Length': '0'} 2023-05-31T21:35:41.516 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000005b6984bd6b6ebba3-006477ae3b-1630-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:29:47 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:35:41.516 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'pklxkd-55e1000falsee2023-05-31T20:27:23.955Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T21:35:41.516 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: comparing key name=f 2023-05-31T21:35:41.517 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/pklxkd-55/f 2023-05-31T21:35:41.517 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/pklxkd-55/f 2023-05-31T21:35:41.517 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/pklxkd-55/f?versionId=null 2023-05-31T21:35:41.517 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/pklxkd-55/f?versionId=null 2023-05-31T21:35:41.518 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-05-31T21:35:41.518 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /pklxkd-55/f?versionId=null 2023-05-31T21:35:41.518 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-05-31T21:35:41.518 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-05-31T21:35:41.519 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T21:35:41.519 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-05-31T21:35:41.519 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-05-31T21:35:41.519 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-05-31T21:35:41.519 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-05-31T21:35:41.520 INFO:tasks.rgw_multisite_tests:GET 2023-05-31T21:35:41.520 INFO:tasks.rgw_multisite_tests: 2023-05-31T21:35:41.520 INFO:tasks.rgw_multisite_tests: 2023-05-31T21:35:41.520 INFO:tasks.rgw_multisite_tests:Wed, 31 May 2023 20:29:47 GMT 2023-05-31T21:35:41.521 INFO:tasks.rgw_multisite_tests:/pklxkd-55/f?versionId=null 2023-05-31T21:35:41.521 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-05-31T21:35:41.521 INFO:tasks.rgw_multisite_tests:AWS Q80H11OLNU1S48PPYL26:Mfy/ziqHy+KzjGSncy7KaRwMtxQ= 2023-05-31T21:35:41.521 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:29:47 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:Mfy/ziqHy+KzjGSncy7KaRwMtxQ=', 'Content-Length': '0'} 2023-05-31T21:35:41.522 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:27:23 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000058ff10fe3f6e86f3-006477ae3b-162e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:29:47 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:35:41.522 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/pklxkd-55/f 2023-05-31T21:35:41.522 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/pklxkd-55/f 2023-05-31T21:35:41.522 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/pklxkd-55/f?versionId=null 2023-05-31T21:35:41.522 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/pklxkd-55/f?versionId=null 2023-05-31T21:35:41.523 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-05-31T21:35:41.523 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /pklxkd-55/f?versionId=null 2023-05-31T21:35:41.523 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-05-31T21:35:41.523 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-05-31T21:35:41.524 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T21:35:41.524 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-05-31T21:35:41.524 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-05-31T21:35:41.524 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-05-31T21:35:41.525 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-05-31T21:35:41.525 INFO:tasks.rgw_multisite_tests:GET 2023-05-31T21:35:41.525 INFO:tasks.rgw_multisite_tests: 2023-05-31T21:35:41.525 INFO:tasks.rgw_multisite_tests: 2023-05-31T21:35:41.525 INFO:tasks.rgw_multisite_tests:Wed, 31 May 2023 20:29:47 GMT 2023-05-31T21:35:41.526 INFO:tasks.rgw_multisite_tests:/pklxkd-55/f?versionId=null 2023-05-31T21:35:41.526 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-05-31T21:35:41.526 INFO:tasks.rgw_multisite_tests:AWS Q80H11OLNU1S48PPYL26:Mfy/ziqHy+KzjGSncy7KaRwMtxQ= 2023-05-31T21:35:41.526 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:29:47 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:Mfy/ziqHy+KzjGSncy7KaRwMtxQ=', 'Content-Length': '0'} 2023-05-31T21:35:41.527 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:27:23 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000097d6e172d97d6b88-006477ae3b-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:29:47 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:35:41.527 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/pklxkd-55/f 2023-05-31T21:35:41.527 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/pklxkd-55/f 2023-05-31T21:35:41.527 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/pklxkd-55/f?versionId=null 2023-05-31T21:35:41.528 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/pklxkd-55/f?versionId=null 2023-05-31T21:35:41.528 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-05-31T21:35:41.528 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /pklxkd-55/f?versionId=null 2023-05-31T21:35:41.528 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-05-31T21:35:41.528 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-05-31T21:35:41.529 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T21:35:41.529 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-05-31T21:35:41.529 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-05-31T21:35:41.529 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-05-31T21:35:41.530 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-05-31T21:35:41.530 INFO:tasks.rgw_multisite_tests:HEAD 2023-05-31T21:35:41.530 INFO:tasks.rgw_multisite_tests: 2023-05-31T21:35:41.530 INFO:tasks.rgw_multisite_tests: 2023-05-31T21:35:41.531 INFO:tasks.rgw_multisite_tests:Wed, 31 May 2023 20:29:47 GMT 2023-05-31T21:35:41.531 INFO:tasks.rgw_multisite_tests:/pklxkd-55/f?versionId=null 2023-05-31T21:35:41.531 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-05-31T21:35:41.531 INFO:tasks.rgw_multisite_tests:AWS Q80H11OLNU1S48PPYL26:/pgwyH5Q7I5Kk2t5ntqQVa6/9es= 2023-05-31T21:35:41.531 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:29:47 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:/pgwyH5Q7I5Kk2t5ntqQVa6/9es=', 'Content-Length': '0'} 2023-05-31T21:35:41.532 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:27:23 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009518d7cecf9fb350-006477ae3b-162e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:29:47 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:35:41.532 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/pklxkd-55/f 2023-05-31T21:35:41.532 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/pklxkd-55/f 2023-05-31T21:35:41.532 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/pklxkd-55/f?versionId=null 2023-05-31T21:35:41.533 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/pklxkd-55/f?versionId=null 2023-05-31T21:35:41.533 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-05-31T21:35:41.533 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /pklxkd-55/f?versionId=null 2023-05-31T21:35:41.533 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-05-31T21:35:41.534 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-05-31T21:35:41.534 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T21:35:41.534 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-05-31T21:35:41.534 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-05-31T21:35:41.535 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-05-31T21:35:41.535 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-05-31T21:35:41.535 INFO:tasks.rgw_multisite_tests:HEAD 2023-05-31T21:35:41.535 INFO:tasks.rgw_multisite_tests: 2023-05-31T21:35:41.535 INFO:tasks.rgw_multisite_tests: 2023-05-31T21:35:41.536 INFO:tasks.rgw_multisite_tests:Wed, 31 May 2023 20:29:47 GMT 2023-05-31T21:35:41.536 INFO:tasks.rgw_multisite_tests:/pklxkd-55/f?versionId=null 2023-05-31T21:35:41.536 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-05-31T21:35:41.536 INFO:tasks.rgw_multisite_tests:AWS Q80H11OLNU1S48PPYL26:/pgwyH5Q7I5Kk2t5ntqQVa6/9es= 2023-05-31T21:35:41.537 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:29:47 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:/pgwyH5Q7I5Kk2t5ntqQVa6/9es=', 'Content-Length': '0'} 2023-05-31T21:35:41.537 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:27:23 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e3bd70b245716530-006477ae3b-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:29:47 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:35:41.537 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: comparing key name=f 2023-05-31T21:35:41.537 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/pklxkd-55/f 2023-05-31T21:35:41.538 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/pklxkd-55/f 2023-05-31T21:35:41.538 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/pklxkd-55/f?versionId=null 2023-05-31T21:35:41.538 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/pklxkd-55/f?versionId=null 2023-05-31T21:35:41.538 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-05-31T21:35:41.539 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /pklxkd-55/f?versionId=null 2023-05-31T21:35:41.539 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-05-31T21:35:41.539 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-05-31T21:35:41.539 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T21:35:41.539 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-05-31T21:35:41.540 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-05-31T21:35:41.540 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-05-31T21:35:41.540 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-05-31T21:35:41.540 INFO:tasks.rgw_multisite_tests:GET 2023-05-31T21:35:41.541 INFO:tasks.rgw_multisite_tests: 2023-05-31T21:35:41.541 INFO:tasks.rgw_multisite_tests: 2023-05-31T21:35:41.541 INFO:tasks.rgw_multisite_tests:Wed, 31 May 2023 20:29:47 GMT 2023-05-31T21:35:41.541 INFO:tasks.rgw_multisite_tests:/pklxkd-55/f?versionId=null 2023-05-31T21:35:41.542 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-05-31T21:35:41.542 INFO:tasks.rgw_multisite_tests:AWS Q80H11OLNU1S48PPYL26:Mfy/ziqHy+KzjGSncy7KaRwMtxQ= 2023-05-31T21:35:41.542 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:29:47 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:Mfy/ziqHy+KzjGSncy7KaRwMtxQ=', 'Content-Length': '0'} 2023-05-31T21:35:41.542 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:27:23 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000488fe1b89a19f9db-006477ae3b-162e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:29:47 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:35:41.543 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/pklxkd-55/f 2023-05-31T21:35:41.543 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/pklxkd-55/f 2023-05-31T21:35:41.543 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/pklxkd-55/f?versionId=null 2023-05-31T21:35:41.543 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/pklxkd-55/f?versionId=null 2023-05-31T21:35:41.543 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-05-31T21:35:41.544 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /pklxkd-55/f?versionId=null 2023-05-31T21:35:41.544 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-05-31T21:35:41.544 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-05-31T21:35:41.544 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T21:35:41.545 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-05-31T21:35:41.545 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-05-31T21:35:41.545 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-05-31T21:35:41.545 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-05-31T21:35:41.546 INFO:tasks.rgw_multisite_tests:GET 2023-05-31T21:35:41.546 INFO:tasks.rgw_multisite_tests: 2023-05-31T21:35:41.546 INFO:tasks.rgw_multisite_tests: 2023-05-31T21:35:41.546 INFO:tasks.rgw_multisite_tests:Wed, 31 May 2023 20:29:47 GMT 2023-05-31T21:35:41.547 INFO:tasks.rgw_multisite_tests:/pklxkd-55/f?versionId=null 2023-05-31T21:35:41.547 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-05-31T21:35:41.547 INFO:tasks.rgw_multisite_tests:AWS Q80H11OLNU1S48PPYL26:Mfy/ziqHy+KzjGSncy7KaRwMtxQ= 2023-05-31T21:35:41.547 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:29:47 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:Mfy/ziqHy+KzjGSncy7KaRwMtxQ=', 'Content-Length': '0'} 2023-05-31T21:35:41.547 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:27:23 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000848476d6171c2f54-006477ae3b-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:29:47 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:35:41.548 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/pklxkd-55/ 2023-05-31T21:35:41.548 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/pklxkd-55/ 2023-05-31T21:35:41.548 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/pklxkd-55/?prefix=f 2023-05-31T21:35:41.548 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/pklxkd-55/?prefix=f 2023-05-31T21:35:41.549 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-05-31T21:35:41.549 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /pklxkd-55/?prefix=f 2023-05-31T21:35:41.549 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-05-31T21:35:41.549 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-05-31T21:35:41.550 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T21:35:41.550 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-05-31T21:35:41.550 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-05-31T21:35:41.550 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-05-31T21:35:41.551 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-05-31T21:35:41.551 INFO:tasks.rgw_multisite_tests:GET 2023-05-31T21:35:41.551 INFO:tasks.rgw_multisite_tests: 2023-05-31T21:35:41.551 INFO:tasks.rgw_multisite_tests: 2023-05-31T21:35:41.551 INFO:tasks.rgw_multisite_tests:Wed, 31 May 2023 20:29:47 GMT 2023-05-31T21:35:41.552 INFO:tasks.rgw_multisite_tests:/pklxkd-55/ 2023-05-31T21:35:41.552 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-05-31T21:35:41.552 INFO:tasks.rgw_multisite_tests:AWS Q80H11OLNU1S48PPYL26:IBdSyRVlh8qg/aH9XxJ5KZfq+k4= 2023-05-31T21:35:41.552 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:29:47 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:IBdSyRVlh8qg/aH9XxJ5KZfq+k4=', 'Content-Length': '0'} 2023-05-31T21:35:41.553 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c36ed2d2b3ee014c-006477ae3b-162e-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:29:47 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:35:41.553 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'pklxkd-55f1000falsef2023-05-31T20:27:23.977Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T21:35:41.553 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/pklxkd-55/ 2023-05-31T21:35:41.553 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/pklxkd-55/ 2023-05-31T21:35:41.554 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/pklxkd-55/?prefix=f 2023-05-31T21:35:41.554 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/pklxkd-55/?prefix=f 2023-05-31T21:35:41.554 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-05-31T21:35:41.554 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /pklxkd-55/?prefix=f 2023-05-31T21:35:41.554 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-05-31T21:35:41.555 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-05-31T21:35:41.555 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T21:35:41.555 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-05-31T21:35:41.555 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-05-31T21:35:41.556 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-05-31T21:35:41.556 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-05-31T21:35:41.556 INFO:tasks.rgw_multisite_tests:GET 2023-05-31T21:35:41.556 INFO:tasks.rgw_multisite_tests: 2023-05-31T21:35:41.557 INFO:tasks.rgw_multisite_tests: 2023-05-31T21:35:41.557 INFO:tasks.rgw_multisite_tests:Wed, 31 May 2023 20:29:47 GMT 2023-05-31T21:35:41.557 INFO:tasks.rgw_multisite_tests:/pklxkd-55/ 2023-05-31T21:35:41.557 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-05-31T21:35:41.558 INFO:tasks.rgw_multisite_tests:AWS Q80H11OLNU1S48PPYL26:IBdSyRVlh8qg/aH9XxJ5KZfq+k4= 2023-05-31T21:35:41.558 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:29:47 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:IBdSyRVlh8qg/aH9XxJ5KZfq+k4=', 'Content-Length': '0'} 2023-05-31T21:35:41.558 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000015cce64da71492e9-006477ae3b-1630-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:29:47 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:35:41.558 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'pklxkd-55f1000falsef2023-05-31T20:27:23.977Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T21:35:41.558 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/pklxkd-55/f 2023-05-31T21:35:41.559 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/pklxkd-55/f 2023-05-31T21:35:41.559 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-05-31T21:35:41.559 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /pklxkd-55/f 2023-05-31T21:35:41.559 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-05-31T21:35:41.560 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-05-31T21:35:41.560 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T21:35:41.560 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-05-31T21:35:41.560 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-05-31T21:35:41.561 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-05-31T21:35:41.561 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-05-31T21:35:41.561 INFO:tasks.rgw_multisite_tests:HEAD 2023-05-31T21:35:41.561 INFO:tasks.rgw_multisite_tests: 2023-05-31T21:35:41.561 INFO:tasks.rgw_multisite_tests: 2023-05-31T21:35:41.562 INFO:tasks.rgw_multisite_tests:Wed, 31 May 2023 20:29:47 GMT 2023-05-31T21:35:41.562 INFO:tasks.rgw_multisite_tests:/pklxkd-55/f 2023-05-31T21:35:41.562 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-05-31T21:35:41.562 INFO:tasks.rgw_multisite_tests:AWS Q80H11OLNU1S48PPYL26:pWSXwnWTK74NrhdIBQRvMbeg5qs= 2023-05-31T21:35:41.563 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:29:47 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:pWSXwnWTK74NrhdIBQRvMbeg5qs=', 'Content-Length': '0'} 2023-05-31T21:35:41.563 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:27:23 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000060dda93cedae8c96-006477ae3b-162e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:29:47 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:35:41.563 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/pklxkd-55/f 2023-05-31T21:35:41.563 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/pklxkd-55/f 2023-05-31T21:35:41.564 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-05-31T21:35:41.564 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /pklxkd-55/f 2023-05-31T21:35:41.564 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-05-31T21:35:41.564 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-05-31T21:35:41.564 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T21:35:41.565 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-05-31T21:35:41.565 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-05-31T21:35:41.565 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-05-31T21:35:41.565 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-05-31T21:35:41.566 INFO:tasks.rgw_multisite_tests:HEAD 2023-05-31T21:35:41.566 INFO:tasks.rgw_multisite_tests: 2023-05-31T21:35:41.566 INFO:tasks.rgw_multisite_tests: 2023-05-31T21:35:41.566 INFO:tasks.rgw_multisite_tests:Wed, 31 May 2023 20:29:47 GMT 2023-05-31T21:35:41.567 INFO:tasks.rgw_multisite_tests:/pklxkd-55/f 2023-05-31T21:35:41.567 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-05-31T21:35:41.567 INFO:tasks.rgw_multisite_tests:AWS Q80H11OLNU1S48PPYL26:pWSXwnWTK74NrhdIBQRvMbeg5qs= 2023-05-31T21:35:41.567 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:29:47 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:pWSXwnWTK74NrhdIBQRvMbeg5qs=', 'Content-Length': '0'} 2023-05-31T21:35:41.567 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:27:23 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f5c9929bb24e0358-006477ae3b-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:29:47 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:35:41.568 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: comparing key name=f 2023-05-31T21:35:41.568 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/pklxkd-55/f 2023-05-31T21:35:41.568 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/pklxkd-55/f 2023-05-31T21:35:41.568 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-05-31T21:35:41.569 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /pklxkd-55/f 2023-05-31T21:35:41.569 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-05-31T21:35:41.569 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-05-31T21:35:41.569 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T21:35:41.570 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-05-31T21:35:41.570 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-05-31T21:35:41.570 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-05-31T21:35:41.570 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-05-31T21:35:41.570 INFO:tasks.rgw_multisite_tests:GET 2023-05-31T21:35:41.571 INFO:tasks.rgw_multisite_tests: 2023-05-31T21:35:41.571 INFO:tasks.rgw_multisite_tests: 2023-05-31T21:35:41.571 INFO:tasks.rgw_multisite_tests:Wed, 31 May 2023 20:29:47 GMT 2023-05-31T21:35:41.571 INFO:tasks.rgw_multisite_tests:/pklxkd-55/f 2023-05-31T21:35:41.572 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-05-31T21:35:41.572 INFO:tasks.rgw_multisite_tests:AWS Q80H11OLNU1S48PPYL26:ZqS8ujsq9YKWQDyX4J33P7BIN+Q= 2023-05-31T21:35:41.572 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:29:47 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:ZqS8ujsq9YKWQDyX4J33P7BIN+Q=', 'Content-Length': '0'} 2023-05-31T21:35:41.572 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:27:23 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000afb4b03dc06d2bb1-006477ae3b-162e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:29:47 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:35:41.573 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/pklxkd-55/f 2023-05-31T21:35:41.573 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/pklxkd-55/f 2023-05-31T21:35:41.573 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-05-31T21:35:41.573 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /pklxkd-55/f 2023-05-31T21:35:41.573 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-05-31T21:35:41.574 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-05-31T21:35:41.574 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T21:35:41.574 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-05-31T21:35:41.574 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-05-31T21:35:41.575 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-05-31T21:35:41.575 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-05-31T21:35:41.575 INFO:tasks.rgw_multisite_tests:GET 2023-05-31T21:35:41.575 INFO:tasks.rgw_multisite_tests: 2023-05-31T21:35:41.575 INFO:tasks.rgw_multisite_tests: 2023-05-31T21:35:41.576 INFO:tasks.rgw_multisite_tests:Wed, 31 May 2023 20:29:47 GMT 2023-05-31T21:35:41.576 INFO:tasks.rgw_multisite_tests:/pklxkd-55/f 2023-05-31T21:35:41.576 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-05-31T21:35:41.576 INFO:tasks.rgw_multisite_tests:AWS Q80H11OLNU1S48PPYL26:ZqS8ujsq9YKWQDyX4J33P7BIN+Q= 2023-05-31T21:35:41.577 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:29:47 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:ZqS8ujsq9YKWQDyX4J33P7BIN+Q=', 'Content-Length': '0'} 2023-05-31T21:35:41.577 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:27:23 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000035f989404b5e68ce-006477ae3b-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:29:47 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:35:41.577 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/pklxkd-55/ 2023-05-31T21:35:41.577 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/pklxkd-55/ 2023-05-31T21:35:41.578 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/pklxkd-55/?prefix=f 2023-05-31T21:35:41.578 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/pklxkd-55/?prefix=f 2023-05-31T21:35:41.578 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-05-31T21:35:41.578 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /pklxkd-55/?prefix=f 2023-05-31T21:35:41.578 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-05-31T21:35:41.579 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-05-31T21:35:41.579 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T21:35:41.579 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-05-31T21:35:41.579 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-05-31T21:35:41.580 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-05-31T21:35:41.580 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-05-31T21:35:41.580 INFO:tasks.rgw_multisite_tests:GET 2023-05-31T21:35:41.580 INFO:tasks.rgw_multisite_tests: 2023-05-31T21:35:41.580 INFO:tasks.rgw_multisite_tests: 2023-05-31T21:35:41.581 INFO:tasks.rgw_multisite_tests:Wed, 31 May 2023 20:29:47 GMT 2023-05-31T21:35:41.581 INFO:tasks.rgw_multisite_tests:/pklxkd-55/ 2023-05-31T21:35:41.581 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-05-31T21:35:41.581 INFO:tasks.rgw_multisite_tests:AWS Q80H11OLNU1S48PPYL26:IBdSyRVlh8qg/aH9XxJ5KZfq+k4= 2023-05-31T21:35:41.582 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:29:47 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:IBdSyRVlh8qg/aH9XxJ5KZfq+k4=', 'Content-Length': '0'} 2023-05-31T21:35:41.582 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c853f0a2dc6a1471-006477ae3b-162e-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:29:47 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:35:41.582 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'pklxkd-55f1000falsef2023-05-31T20:27:23.977Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T21:35:41.582 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/pklxkd-55/ 2023-05-31T21:35:41.583 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/pklxkd-55/ 2023-05-31T21:35:41.583 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/pklxkd-55/?prefix=f 2023-05-31T21:35:41.583 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/pklxkd-55/?prefix=f 2023-05-31T21:35:41.583 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-05-31T21:35:41.584 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /pklxkd-55/?prefix=f 2023-05-31T21:35:41.584 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-05-31T21:35:41.584 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-05-31T21:35:41.584 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T21:35:41.584 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-05-31T21:35:41.585 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-05-31T21:35:41.585 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-05-31T21:35:41.585 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-05-31T21:35:41.585 INFO:tasks.rgw_multisite_tests:GET 2023-05-31T21:35:41.586 INFO:tasks.rgw_multisite_tests: 2023-05-31T21:35:41.586 INFO:tasks.rgw_multisite_tests: 2023-05-31T21:35:41.586 INFO:tasks.rgw_multisite_tests:Wed, 31 May 2023 20:29:47 GMT 2023-05-31T21:35:41.586 INFO:tasks.rgw_multisite_tests:/pklxkd-55/ 2023-05-31T21:35:41.587 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-05-31T21:35:41.587 INFO:tasks.rgw_multisite_tests:AWS Q80H11OLNU1S48PPYL26:IBdSyRVlh8qg/aH9XxJ5KZfq+k4= 2023-05-31T21:35:41.587 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:29:47 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:IBdSyRVlh8qg/aH9XxJ5KZfq+k4=', 'Content-Length': '0'} 2023-05-31T21:35:41.587 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000688df95bdfa5dc15-006477ae3b-1630-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:29:47 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:35:41.588 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'pklxkd-55f1000falsef2023-05-31T20:27:23.977Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T21:35:41.588 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: comparing key name=g 2023-05-31T21:35:41.588 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/pklxkd-55/g 2023-05-31T21:35:41.588 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/pklxkd-55/g 2023-05-31T21:35:41.588 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/pklxkd-55/g?versionId=null 2023-05-31T21:35:41.589 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/pklxkd-55/g?versionId=null 2023-05-31T21:35:41.589 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-05-31T21:35:41.589 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /pklxkd-55/g?versionId=null 2023-05-31T21:35:41.589 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-05-31T21:35:41.590 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-05-31T21:35:41.590 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T21:35:41.590 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-05-31T21:35:41.590 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-05-31T21:35:41.590 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-05-31T21:35:41.591 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-05-31T21:35:41.591 INFO:tasks.rgw_multisite_tests:GET 2023-05-31T21:35:41.591 INFO:tasks.rgw_multisite_tests: 2023-05-31T21:35:41.591 INFO:tasks.rgw_multisite_tests: 2023-05-31T21:35:41.592 INFO:tasks.rgw_multisite_tests:Wed, 31 May 2023 20:29:47 GMT 2023-05-31T21:35:41.592 INFO:tasks.rgw_multisite_tests:/pklxkd-55/g?versionId=null 2023-05-31T21:35:41.592 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-05-31T21:35:41.592 INFO:tasks.rgw_multisite_tests:AWS Q80H11OLNU1S48PPYL26:8rwhJVFIeQEuHOFXFh01O4OeEEE= 2023-05-31T21:35:41.593 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:29:47 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:8rwhJVFIeQEuHOFXFh01O4OeEEE=', 'Content-Length': '0'} 2023-05-31T21:35:41.593 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:27:24 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005763b47dc83006a8-006477ae3b-162e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:29:47 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:35:41.593 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/pklxkd-55/g 2023-05-31T21:35:41.593 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/pklxkd-55/g 2023-05-31T21:35:41.594 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/pklxkd-55/g?versionId=null 2023-05-31T21:35:41.594 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/pklxkd-55/g?versionId=null 2023-05-31T21:35:41.594 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-05-31T21:35:41.594 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /pklxkd-55/g?versionId=null 2023-05-31T21:35:41.594 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-05-31T21:35:41.595 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-05-31T21:35:41.595 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T21:35:41.595 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-05-31T21:35:41.595 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-05-31T21:35:41.596 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-05-31T21:35:41.596 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-05-31T21:35:41.596 INFO:tasks.rgw_multisite_tests:GET 2023-05-31T21:35:41.596 INFO:tasks.rgw_multisite_tests: 2023-05-31T21:35:41.597 INFO:tasks.rgw_multisite_tests: 2023-05-31T21:35:41.597 INFO:tasks.rgw_multisite_tests:Wed, 31 May 2023 20:29:47 GMT 2023-05-31T21:35:41.597 INFO:tasks.rgw_multisite_tests:/pklxkd-55/g?versionId=null 2023-05-31T21:35:41.597 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-05-31T21:35:41.597 INFO:tasks.rgw_multisite_tests:AWS Q80H11OLNU1S48PPYL26:8rwhJVFIeQEuHOFXFh01O4OeEEE= 2023-05-31T21:35:41.598 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:29:47 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:8rwhJVFIeQEuHOFXFh01O4OeEEE=', 'Content-Length': '0'} 2023-05-31T21:35:41.598 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:27:24 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009fe6cc0cd3241251-006477ae3b-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:29:47 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:35:41.598 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/pklxkd-55/g 2023-05-31T21:35:41.598 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/pklxkd-55/g 2023-05-31T21:35:41.599 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/pklxkd-55/g?versionId=null 2023-05-31T21:35:41.599 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/pklxkd-55/g?versionId=null 2023-05-31T21:35:41.599 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-05-31T21:35:41.599 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /pklxkd-55/g?versionId=null 2023-05-31T21:35:41.600 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-05-31T21:35:41.600 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-05-31T21:35:41.600 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T21:35:41.600 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-05-31T21:35:41.600 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-05-31T21:35:41.601 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-05-31T21:35:41.601 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-05-31T21:35:41.601 INFO:tasks.rgw_multisite_tests:HEAD 2023-05-31T21:35:41.601 INFO:tasks.rgw_multisite_tests: 2023-05-31T21:35:41.602 INFO:tasks.rgw_multisite_tests: 2023-05-31T21:35:41.602 INFO:tasks.rgw_multisite_tests:Wed, 31 May 2023 20:29:47 GMT 2023-05-31T21:35:41.602 INFO:tasks.rgw_multisite_tests:/pklxkd-55/g?versionId=null 2023-05-31T21:35:41.602 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-05-31T21:35:41.602 INFO:tasks.rgw_multisite_tests:AWS Q80H11OLNU1S48PPYL26:EUQ90RtyUgav9p8DyBg2EaFCQJg= 2023-05-31T21:35:41.603 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:29:47 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:EUQ90RtyUgav9p8DyBg2EaFCQJg=', 'Content-Length': '0'} 2023-05-31T21:35:41.603 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:27:24 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000076f1a43908e465b9-006477ae3b-162e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:29:47 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:35:41.603 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/pklxkd-55/g 2023-05-31T21:35:41.603 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/pklxkd-55/g 2023-05-31T21:35:41.604 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/pklxkd-55/g?versionId=null 2023-05-31T21:35:41.604 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/pklxkd-55/g?versionId=null 2023-05-31T21:35:41.604 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-05-31T21:35:41.604 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /pklxkd-55/g?versionId=null 2023-05-31T21:35:41.605 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-05-31T21:35:41.605 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-05-31T21:35:41.605 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T21:35:41.605 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-05-31T21:35:41.606 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-05-31T21:35:41.606 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-05-31T21:35:41.606 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-05-31T21:35:41.606 INFO:tasks.rgw_multisite_tests:HEAD 2023-05-31T21:35:41.606 INFO:tasks.rgw_multisite_tests: 2023-05-31T21:35:41.607 INFO:tasks.rgw_multisite_tests: 2023-05-31T21:35:41.607 INFO:tasks.rgw_multisite_tests:Wed, 31 May 2023 20:29:47 GMT 2023-05-31T21:35:41.607 INFO:tasks.rgw_multisite_tests:/pklxkd-55/g?versionId=null 2023-05-31T21:35:41.607 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-05-31T21:35:41.608 INFO:tasks.rgw_multisite_tests:AWS Q80H11OLNU1S48PPYL26:EUQ90RtyUgav9p8DyBg2EaFCQJg= 2023-05-31T21:35:41.608 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:29:47 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:EUQ90RtyUgav9p8DyBg2EaFCQJg=', 'Content-Length': '0'} 2023-05-31T21:35:41.608 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:27:24 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000013227d712e296af0-006477ae3b-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:29:47 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:35:41.608 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: comparing key name=g 2023-05-31T21:35:41.609 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/pklxkd-55/g 2023-05-31T21:35:41.609 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/pklxkd-55/g 2023-05-31T21:35:41.609 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/pklxkd-55/g?versionId=null 2023-05-31T21:35:41.609 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/pklxkd-55/g?versionId=null 2023-05-31T21:35:41.609 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-05-31T21:35:41.610 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /pklxkd-55/g?versionId=null 2023-05-31T21:35:41.610 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-05-31T21:35:41.610 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-05-31T21:35:41.610 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T21:35:41.611 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-05-31T21:35:41.611 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-05-31T21:35:41.611 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-05-31T21:35:41.611 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-05-31T21:35:41.611 INFO:tasks.rgw_multisite_tests:GET 2023-05-31T21:35:41.612 INFO:tasks.rgw_multisite_tests: 2023-05-31T21:35:41.612 INFO:tasks.rgw_multisite_tests: 2023-05-31T21:35:41.612 INFO:tasks.rgw_multisite_tests:Wed, 31 May 2023 20:29:47 GMT 2023-05-31T21:35:41.612 INFO:tasks.rgw_multisite_tests:/pklxkd-55/g?versionId=null 2023-05-31T21:35:41.613 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-05-31T21:35:41.613 INFO:tasks.rgw_multisite_tests:AWS Q80H11OLNU1S48PPYL26:8rwhJVFIeQEuHOFXFh01O4OeEEE= 2023-05-31T21:35:41.613 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:29:47 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:8rwhJVFIeQEuHOFXFh01O4OeEEE=', 'Content-Length': '0'} 2023-05-31T21:35:41.613 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:27:24 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000194f3d665d538ebd-006477ae3b-162e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:29:47 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:35:41.614 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/pklxkd-55/g 2023-05-31T21:35:41.614 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/pklxkd-55/g 2023-05-31T21:35:41.614 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/pklxkd-55/g?versionId=null 2023-05-31T21:35:41.614 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/pklxkd-55/g?versionId=null 2023-05-31T21:35:41.614 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-05-31T21:35:41.615 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /pklxkd-55/g?versionId=null 2023-05-31T21:35:41.615 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-05-31T21:35:41.615 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-05-31T21:35:41.615 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T21:35:41.616 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-05-31T21:35:41.616 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-05-31T21:35:41.616 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-05-31T21:35:41.616 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-05-31T21:35:41.617 INFO:tasks.rgw_multisite_tests:GET 2023-05-31T21:35:41.617 INFO:tasks.rgw_multisite_tests: 2023-05-31T21:35:41.617 INFO:tasks.rgw_multisite_tests: 2023-05-31T21:35:41.617 INFO:tasks.rgw_multisite_tests:Wed, 31 May 2023 20:29:47 GMT 2023-05-31T21:35:41.617 INFO:tasks.rgw_multisite_tests:/pklxkd-55/g?versionId=null 2023-05-31T21:35:41.618 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-05-31T21:35:41.618 INFO:tasks.rgw_multisite_tests:AWS Q80H11OLNU1S48PPYL26:8rwhJVFIeQEuHOFXFh01O4OeEEE= 2023-05-31T21:35:41.618 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:29:47 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:8rwhJVFIeQEuHOFXFh01O4OeEEE=', 'Content-Length': '0'} 2023-05-31T21:35:41.618 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:27:24 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000097a43b7adfeb89a4-006477ae3b-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:29:47 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:35:41.619 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/pklxkd-55/ 2023-05-31T21:35:41.619 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/pklxkd-55/ 2023-05-31T21:35:41.619 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/pklxkd-55/?prefix=g 2023-05-31T21:35:41.619 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/pklxkd-55/?prefix=g 2023-05-31T21:35:41.620 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-05-31T21:35:41.620 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /pklxkd-55/?prefix=g 2023-05-31T21:35:41.620 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-05-31T21:35:41.620 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-05-31T21:35:41.620 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T21:35:41.621 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-05-31T21:35:41.621 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-05-31T21:35:41.621 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-05-31T21:35:41.621 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-05-31T21:35:41.622 INFO:tasks.rgw_multisite_tests:GET 2023-05-31T21:35:41.622 INFO:tasks.rgw_multisite_tests: 2023-05-31T21:35:41.622 INFO:tasks.rgw_multisite_tests: 2023-05-31T21:35:41.622 INFO:tasks.rgw_multisite_tests:Wed, 31 May 2023 20:29:47 GMT 2023-05-31T21:35:41.623 INFO:tasks.rgw_multisite_tests:/pklxkd-55/ 2023-05-31T21:35:41.623 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-05-31T21:35:41.623 INFO:tasks.rgw_multisite_tests:AWS Q80H11OLNU1S48PPYL26:IBdSyRVlh8qg/aH9XxJ5KZfq+k4= 2023-05-31T21:35:41.623 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:29:47 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:IBdSyRVlh8qg/aH9XxJ5KZfq+k4=', 'Content-Length': '0'} 2023-05-31T21:35:41.624 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000de6866aa36ba6168-006477ae3b-162e-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:29:47 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:35:41.624 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'pklxkd-55g1000falseg2023-05-31T20:27:24.035Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T21:35:41.624 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/pklxkd-55/ 2023-05-31T21:35:41.624 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/pklxkd-55/ 2023-05-31T21:35:41.625 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/pklxkd-55/?prefix=g 2023-05-31T21:35:41.625 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/pklxkd-55/?prefix=g 2023-05-31T21:35:41.625 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-05-31T21:35:41.625 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /pklxkd-55/?prefix=g 2023-05-31T21:35:41.625 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-05-31T21:35:41.626 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-05-31T21:35:41.626 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T21:35:41.626 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-05-31T21:35:41.626 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-05-31T21:35:41.627 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-05-31T21:35:41.627 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-05-31T21:35:41.627 INFO:tasks.rgw_multisite_tests:GET 2023-05-31T21:35:41.627 INFO:tasks.rgw_multisite_tests: 2023-05-31T21:35:41.627 INFO:tasks.rgw_multisite_tests: 2023-05-31T21:35:41.628 INFO:tasks.rgw_multisite_tests:Wed, 31 May 2023 20:29:47 GMT 2023-05-31T21:35:41.628 INFO:tasks.rgw_multisite_tests:/pklxkd-55/ 2023-05-31T21:35:41.628 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-05-31T21:35:41.628 INFO:tasks.rgw_multisite_tests:AWS Q80H11OLNU1S48PPYL26:IBdSyRVlh8qg/aH9XxJ5KZfq+k4= 2023-05-31T21:35:41.629 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:29:47 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:IBdSyRVlh8qg/aH9XxJ5KZfq+k4=', 'Content-Length': '0'} 2023-05-31T21:35:41.629 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e78fd15707b5c7ce-006477ae3b-1630-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:29:47 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:35:41.629 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'pklxkd-55g1000falseg2023-05-31T20:27:24.035Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T21:35:41.629 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/pklxkd-55/g 2023-05-31T21:35:41.630 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/pklxkd-55/g 2023-05-31T21:35:41.630 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-05-31T21:35:41.630 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /pklxkd-55/g 2023-05-31T21:35:41.630 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-05-31T21:35:41.630 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-05-31T21:35:41.631 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T21:35:41.631 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-05-31T21:35:41.631 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-05-31T21:35:41.631 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-05-31T21:35:41.632 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-05-31T21:35:41.632 INFO:tasks.rgw_multisite_tests:HEAD 2023-05-31T21:35:41.632 INFO:tasks.rgw_multisite_tests: 2023-05-31T21:35:41.632 INFO:tasks.rgw_multisite_tests: 2023-05-31T21:35:41.633 INFO:tasks.rgw_multisite_tests:Wed, 31 May 2023 20:29:47 GMT 2023-05-31T21:35:41.633 INFO:tasks.rgw_multisite_tests:/pklxkd-55/g 2023-05-31T21:35:41.633 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-05-31T21:35:41.633 INFO:tasks.rgw_multisite_tests:AWS Q80H11OLNU1S48PPYL26:zl7xf2Qpq84VG8Y7vB8gy2ps+ZM= 2023-05-31T21:35:41.633 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:29:47 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:zl7xf2Qpq84VG8Y7vB8gy2ps+ZM=', 'Content-Length': '0'} 2023-05-31T21:35:41.634 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:27:24 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000ca190d2d3d02499-006477ae3b-162e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:29:47 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:35:41.634 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/pklxkd-55/g 2023-05-31T21:35:41.634 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/pklxkd-55/g 2023-05-31T21:35:41.634 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-05-31T21:35:41.635 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /pklxkd-55/g 2023-05-31T21:35:41.635 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-05-31T21:35:41.635 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-05-31T21:35:41.635 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T21:35:41.636 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-05-31T21:35:41.636 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-05-31T21:35:41.636 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-05-31T21:35:41.636 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-05-31T21:35:41.637 INFO:tasks.rgw_multisite_tests:HEAD 2023-05-31T21:35:41.637 INFO:tasks.rgw_multisite_tests: 2023-05-31T21:35:41.637 INFO:tasks.rgw_multisite_tests: 2023-05-31T21:35:41.637 INFO:tasks.rgw_multisite_tests:Wed, 31 May 2023 20:29:47 GMT 2023-05-31T21:35:41.637 INFO:tasks.rgw_multisite_tests:/pklxkd-55/g 2023-05-31T21:35:41.638 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-05-31T21:35:41.638 INFO:tasks.rgw_multisite_tests:AWS Q80H11OLNU1S48PPYL26:zl7xf2Qpq84VG8Y7vB8gy2ps+ZM= 2023-05-31T21:35:41.638 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:29:47 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:zl7xf2Qpq84VG8Y7vB8gy2ps+ZM=', 'Content-Length': '0'} 2023-05-31T21:35:41.638 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:27:24 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000522253004e669a7d-006477ae3b-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:29:47 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:35:41.639 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: comparing key name=g 2023-05-31T21:35:41.639 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/pklxkd-55/g 2023-05-31T21:35:41.639 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/pklxkd-55/g 2023-05-31T21:35:41.639 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-05-31T21:35:41.640 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /pklxkd-55/g 2023-05-31T21:35:41.640 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-05-31T21:35:41.640 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-05-31T21:35:41.640 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T21:35:41.640 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-05-31T21:35:41.641 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-05-31T21:35:41.641 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-05-31T21:35:41.641 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-05-31T21:35:41.641 INFO:tasks.rgw_multisite_tests:GET 2023-05-31T21:35:41.642 INFO:tasks.rgw_multisite_tests: 2023-05-31T21:35:41.642 INFO:tasks.rgw_multisite_tests: 2023-05-31T21:35:41.642 INFO:tasks.rgw_multisite_tests:Wed, 31 May 2023 20:29:47 GMT 2023-05-31T21:35:41.642 INFO:tasks.rgw_multisite_tests:/pklxkd-55/g 2023-05-31T21:35:41.642 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-05-31T21:35:41.643 INFO:tasks.rgw_multisite_tests:AWS Q80H11OLNU1S48PPYL26:FF/Yavj/kG8A7H7WSv3UcpRfHvg= 2023-05-31T21:35:41.643 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:29:47 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:FF/Yavj/kG8A7H7WSv3UcpRfHvg=', 'Content-Length': '0'} 2023-05-31T21:35:41.643 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:27:24 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f58be5a110f5918a-006477ae3b-162e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:29:47 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:35:41.643 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/pklxkd-55/g 2023-05-31T21:35:41.644 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/pklxkd-55/g 2023-05-31T21:35:41.644 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-05-31T21:35:41.644 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /pklxkd-55/g 2023-05-31T21:35:41.644 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-05-31T21:35:41.645 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-05-31T21:35:41.645 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T21:35:41.645 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-05-31T21:35:41.645 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-05-31T21:35:41.646 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-05-31T21:35:41.646 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-05-31T21:35:41.646 INFO:tasks.rgw_multisite_tests:GET 2023-05-31T21:35:41.646 INFO:tasks.rgw_multisite_tests: 2023-05-31T21:35:41.646 INFO:tasks.rgw_multisite_tests: 2023-05-31T21:35:41.647 INFO:tasks.rgw_multisite_tests:Wed, 31 May 2023 20:29:47 GMT 2023-05-31T21:35:41.647 INFO:tasks.rgw_multisite_tests:/pklxkd-55/g 2023-05-31T21:35:41.647 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-05-31T21:35:41.647 INFO:tasks.rgw_multisite_tests:AWS Q80H11OLNU1S48PPYL26:FF/Yavj/kG8A7H7WSv3UcpRfHvg= 2023-05-31T21:35:41.648 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:29:47 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:FF/Yavj/kG8A7H7WSv3UcpRfHvg=', 'Content-Length': '0'} 2023-05-31T21:35:41.648 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:27:24 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001c904f5c9a53e6d5-006477ae3b-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:29:47 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:35:41.648 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/pklxkd-55/ 2023-05-31T21:35:41.648 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/pklxkd-55/ 2023-05-31T21:35:41.649 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/pklxkd-55/?prefix=g 2023-05-31T21:35:41.649 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/pklxkd-55/?prefix=g 2023-05-31T21:35:41.649 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-05-31T21:35:41.649 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /pklxkd-55/?prefix=g 2023-05-31T21:35:41.650 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-05-31T21:35:41.650 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-05-31T21:35:41.650 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T21:35:41.650 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-05-31T21:35:41.650 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-05-31T21:35:41.651 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-05-31T21:35:41.651 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-05-31T21:35:41.651 INFO:tasks.rgw_multisite_tests:GET 2023-05-31T21:35:41.651 INFO:tasks.rgw_multisite_tests: 2023-05-31T21:35:41.652 INFO:tasks.rgw_multisite_tests: 2023-05-31T21:35:41.652 INFO:tasks.rgw_multisite_tests:Wed, 31 May 2023 20:29:47 GMT 2023-05-31T21:35:41.652 INFO:tasks.rgw_multisite_tests:/pklxkd-55/ 2023-05-31T21:35:41.652 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-05-31T21:35:41.653 INFO:tasks.rgw_multisite_tests:AWS Q80H11OLNU1S48PPYL26:IBdSyRVlh8qg/aH9XxJ5KZfq+k4= 2023-05-31T21:35:41.653 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:29:47 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:IBdSyRVlh8qg/aH9XxJ5KZfq+k4=', 'Content-Length': '0'} 2023-05-31T21:35:41.653 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000003fe215c9a8816f37-006477ae3b-162e-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:29:47 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:35:41.653 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'pklxkd-55g1000falseg2023-05-31T20:27:24.035Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T21:35:41.653 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/pklxkd-55/ 2023-05-31T21:35:41.654 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/pklxkd-55/ 2023-05-31T21:35:41.654 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/pklxkd-55/?prefix=g 2023-05-31T21:35:41.654 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/pklxkd-55/?prefix=g 2023-05-31T21:35:41.654 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-05-31T21:35:41.655 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /pklxkd-55/?prefix=g 2023-05-31T21:35:41.655 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-05-31T21:35:41.655 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-05-31T21:35:41.655 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T21:35:41.656 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-05-31T21:35:41.656 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-05-31T21:35:41.656 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-05-31T21:35:41.656 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-05-31T21:35:41.657 INFO:tasks.rgw_multisite_tests:GET 2023-05-31T21:35:41.657 INFO:tasks.rgw_multisite_tests: 2023-05-31T21:35:41.657 INFO:tasks.rgw_multisite_tests: 2023-05-31T21:35:41.657 INFO:tasks.rgw_multisite_tests:Wed, 31 May 2023 20:29:47 GMT 2023-05-31T21:35:41.657 INFO:tasks.rgw_multisite_tests:/pklxkd-55/ 2023-05-31T21:35:41.658 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-05-31T21:35:41.658 INFO:tasks.rgw_multisite_tests:AWS Q80H11OLNU1S48PPYL26:IBdSyRVlh8qg/aH9XxJ5KZfq+k4= 2023-05-31T21:35:41.658 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:29:47 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:IBdSyRVlh8qg/aH9XxJ5KZfq+k4=', 'Content-Length': '0'} 2023-05-31T21:35:41.658 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f476bdc1ed70ecf8-006477ae3b-1630-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:29:47 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:35:41.659 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'pklxkd-55g1000falseg2023-05-31T20:27:24.035Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T21:35:41.659 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: comparing key name=h 2023-05-31T21:35:41.659 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/pklxkd-55/h 2023-05-31T21:35:41.659 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/pklxkd-55/h 2023-05-31T21:35:41.660 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/pklxkd-55/h?versionId=null 2023-05-31T21:35:41.660 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/pklxkd-55/h?versionId=null 2023-05-31T21:35:41.660 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-05-31T21:35:41.660 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /pklxkd-55/h?versionId=null 2023-05-31T21:35:41.660 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-05-31T21:35:41.661 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-05-31T21:35:41.661 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T21:35:41.661 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-05-31T21:35:41.661 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-05-31T21:35:41.662 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-05-31T21:35:41.662 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-05-31T21:35:41.662 INFO:tasks.rgw_multisite_tests:GET 2023-05-31T21:35:41.662 INFO:tasks.rgw_multisite_tests: 2023-05-31T21:35:41.663 INFO:tasks.rgw_multisite_tests: 2023-05-31T21:35:41.663 INFO:tasks.rgw_multisite_tests:Wed, 31 May 2023 20:29:47 GMT 2023-05-31T21:35:41.663 INFO:tasks.rgw_multisite_tests:/pklxkd-55/h?versionId=null 2023-05-31T21:35:41.663 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-05-31T21:35:41.663 INFO:tasks.rgw_multisite_tests:AWS Q80H11OLNU1S48PPYL26:FCLWQvP+WAiTc4EwDy/2WqLQaSI= 2023-05-31T21:35:41.664 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:29:47 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:FCLWQvP+WAiTc4EwDy/2WqLQaSI=', 'Content-Length': '0'} 2023-05-31T21:35:41.664 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:27:24 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000acc069d765ca9d2e-006477ae3b-162e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:29:47 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:35:41.664 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/pklxkd-55/h 2023-05-31T21:35:41.664 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/pklxkd-55/h 2023-05-31T21:35:41.665 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/pklxkd-55/h?versionId=null 2023-05-31T21:35:41.665 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/pklxkd-55/h?versionId=null 2023-05-31T21:35:41.665 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-05-31T21:35:41.665 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /pklxkd-55/h?versionId=null 2023-05-31T21:35:41.666 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-05-31T21:35:41.666 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-05-31T21:35:41.666 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T21:35:41.666 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-05-31T21:35:41.666 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-05-31T21:35:41.667 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-05-31T21:35:41.667 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-05-31T21:35:41.667 INFO:tasks.rgw_multisite_tests:GET 2023-05-31T21:35:41.667 INFO:tasks.rgw_multisite_tests: 2023-05-31T21:35:41.668 INFO:tasks.rgw_multisite_tests: 2023-05-31T21:35:41.668 INFO:tasks.rgw_multisite_tests:Wed, 31 May 2023 20:29:47 GMT 2023-05-31T21:35:41.668 INFO:tasks.rgw_multisite_tests:/pklxkd-55/h?versionId=null 2023-05-31T21:35:41.668 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-05-31T21:35:41.668 INFO:tasks.rgw_multisite_tests:AWS Q80H11OLNU1S48PPYL26:FCLWQvP+WAiTc4EwDy/2WqLQaSI= 2023-05-31T21:35:41.669 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:29:47 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:FCLWQvP+WAiTc4EwDy/2WqLQaSI=', 'Content-Length': '0'} 2023-05-31T21:35:41.669 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:27:24 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000ffd5844c36724ff-006477ae3b-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:29:47 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:35:41.669 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/pklxkd-55/h 2023-05-31T21:35:41.669 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/pklxkd-55/h 2023-05-31T21:35:41.670 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/pklxkd-55/h?versionId=null 2023-05-31T21:35:41.670 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/pklxkd-55/h?versionId=null 2023-05-31T21:35:41.670 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-05-31T21:35:41.670 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /pklxkd-55/h?versionId=null 2023-05-31T21:35:41.671 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-05-31T21:35:41.671 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-05-31T21:35:41.671 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T21:35:41.671 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-05-31T21:35:41.671 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-05-31T21:35:41.672 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-05-31T21:35:41.672 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-05-31T21:35:41.672 INFO:tasks.rgw_multisite_tests:HEAD 2023-05-31T21:35:41.672 INFO:tasks.rgw_multisite_tests: 2023-05-31T21:35:41.673 INFO:tasks.rgw_multisite_tests: 2023-05-31T21:35:41.673 INFO:tasks.rgw_multisite_tests:Wed, 31 May 2023 20:29:47 GMT 2023-05-31T21:35:41.673 INFO:tasks.rgw_multisite_tests:/pklxkd-55/h?versionId=null 2023-05-31T21:35:41.673 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-05-31T21:35:41.674 INFO:tasks.rgw_multisite_tests:AWS Q80H11OLNU1S48PPYL26:zHxzG1sH2HmxravJuhdLofvKy6I= 2023-05-31T21:35:41.674 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:29:47 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:zHxzG1sH2HmxravJuhdLofvKy6I=', 'Content-Length': '0'} 2023-05-31T21:35:41.674 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:27:24 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005364aace1e8225ae-006477ae3b-162e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:29:47 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:35:41.674 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/pklxkd-55/h 2023-05-31T21:35:41.674 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/pklxkd-55/h 2023-05-31T21:35:41.675 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/pklxkd-55/h?versionId=null 2023-05-31T21:35:41.675 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/pklxkd-55/h?versionId=null 2023-05-31T21:35:41.675 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-05-31T21:35:41.675 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /pklxkd-55/h?versionId=null 2023-05-31T21:35:41.676 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-05-31T21:35:41.676 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-05-31T21:35:41.676 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T21:35:41.676 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-05-31T21:35:41.677 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-05-31T21:35:41.677 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-05-31T21:35:41.677 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-05-31T21:35:41.677 INFO:tasks.rgw_multisite_tests:HEAD 2023-05-31T21:35:41.677 INFO:tasks.rgw_multisite_tests: 2023-05-31T21:35:41.678 INFO:tasks.rgw_multisite_tests: 2023-05-31T21:35:41.678 INFO:tasks.rgw_multisite_tests:Wed, 31 May 2023 20:29:47 GMT 2023-05-31T21:35:41.678 INFO:tasks.rgw_multisite_tests:/pklxkd-55/h?versionId=null 2023-05-31T21:35:41.678 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-05-31T21:35:41.679 INFO:tasks.rgw_multisite_tests:AWS Q80H11OLNU1S48PPYL26:zHxzG1sH2HmxravJuhdLofvKy6I= 2023-05-31T21:35:41.679 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:29:47 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:zHxzG1sH2HmxravJuhdLofvKy6I=', 'Content-Length': '0'} 2023-05-31T21:35:41.679 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:27:24 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000838ce741a113afe9-006477ae3b-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:29:47 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:35:41.679 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: comparing key name=h 2023-05-31T21:35:41.680 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/pklxkd-55/h 2023-05-31T21:35:41.680 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/pklxkd-55/h 2023-05-31T21:35:41.680 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/pklxkd-55/h?versionId=null 2023-05-31T21:35:41.680 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/pklxkd-55/h?versionId=null 2023-05-31T21:35:41.680 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-05-31T21:35:41.681 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /pklxkd-55/h?versionId=null 2023-05-31T21:35:41.681 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-05-31T21:35:41.681 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-05-31T21:35:41.681 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T21:35:41.682 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-05-31T21:35:41.682 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-05-31T21:35:41.682 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-05-31T21:35:41.682 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-05-31T21:35:41.683 INFO:tasks.rgw_multisite_tests:GET 2023-05-31T21:35:41.683 INFO:tasks.rgw_multisite_tests: 2023-05-31T21:35:41.683 INFO:tasks.rgw_multisite_tests: 2023-05-31T21:35:41.683 INFO:tasks.rgw_multisite_tests:Wed, 31 May 2023 20:29:47 GMT 2023-05-31T21:35:41.683 INFO:tasks.rgw_multisite_tests:/pklxkd-55/h?versionId=null 2023-05-31T21:35:41.684 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-05-31T21:35:41.684 INFO:tasks.rgw_multisite_tests:AWS Q80H11OLNU1S48PPYL26:FCLWQvP+WAiTc4EwDy/2WqLQaSI= 2023-05-31T21:35:41.684 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:29:47 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:FCLWQvP+WAiTc4EwDy/2WqLQaSI=', 'Content-Length': '0'} 2023-05-31T21:35:41.684 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:27:24 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004b23650bd4f1ad23-006477ae3b-162e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:29:47 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:35:41.685 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/pklxkd-55/h 2023-05-31T21:35:41.685 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/pklxkd-55/h 2023-05-31T21:35:41.685 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/pklxkd-55/h?versionId=null 2023-05-31T21:35:41.685 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/pklxkd-55/h?versionId=null 2023-05-31T21:35:41.686 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-05-31T21:35:41.686 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /pklxkd-55/h?versionId=null 2023-05-31T21:35:41.686 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-05-31T21:35:41.686 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-05-31T21:35:41.687 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T21:35:41.687 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-05-31T21:35:41.687 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-05-31T21:35:41.687 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-05-31T21:35:41.687 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-05-31T21:35:41.688 INFO:tasks.rgw_multisite_tests:GET 2023-05-31T21:35:41.688 INFO:tasks.rgw_multisite_tests: 2023-05-31T21:35:41.688 INFO:tasks.rgw_multisite_tests: 2023-05-31T21:35:41.688 INFO:tasks.rgw_multisite_tests:Wed, 31 May 2023 20:29:47 GMT 2023-05-31T21:35:41.689 INFO:tasks.rgw_multisite_tests:/pklxkd-55/h?versionId=null 2023-05-31T21:35:41.689 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-05-31T21:35:41.689 INFO:tasks.rgw_multisite_tests:AWS Q80H11OLNU1S48PPYL26:FCLWQvP+WAiTc4EwDy/2WqLQaSI= 2023-05-31T21:35:41.689 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:29:47 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:FCLWQvP+WAiTc4EwDy/2WqLQaSI=', 'Content-Length': '0'} 2023-05-31T21:35:41.690 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:27:24 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009a4d2cd898446b8b-006477ae3b-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:29:47 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:35:41.690 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/pklxkd-55/ 2023-05-31T21:35:41.690 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/pklxkd-55/ 2023-05-31T21:35:41.690 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/pklxkd-55/?prefix=h 2023-05-31T21:35:41.690 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/pklxkd-55/?prefix=h 2023-05-31T21:35:41.691 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-05-31T21:35:41.691 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /pklxkd-55/?prefix=h 2023-05-31T21:35:41.691 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-05-31T21:35:41.691 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-05-31T21:35:41.692 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T21:35:41.692 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-05-31T21:35:41.692 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-05-31T21:35:41.692 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-05-31T21:35:41.693 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-05-31T21:35:41.693 INFO:tasks.rgw_multisite_tests:GET 2023-05-31T21:35:41.693 INFO:tasks.rgw_multisite_tests: 2023-05-31T21:35:41.693 INFO:tasks.rgw_multisite_tests: 2023-05-31T21:35:41.693 INFO:tasks.rgw_multisite_tests:Wed, 31 May 2023 20:29:47 GMT 2023-05-31T21:35:41.694 INFO:tasks.rgw_multisite_tests:/pklxkd-55/ 2023-05-31T21:35:41.694 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-05-31T21:35:41.694 INFO:tasks.rgw_multisite_tests:AWS Q80H11OLNU1S48PPYL26:IBdSyRVlh8qg/aH9XxJ5KZfq+k4= 2023-05-31T21:35:41.694 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:29:47 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:IBdSyRVlh8qg/aH9XxJ5KZfq+k4=', 'Content-Length': '0'} 2023-05-31T21:35:41.695 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000064be796a973247dd-006477ae3b-162e-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:29:47 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:35:41.695 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'pklxkd-55h1000falseh2023-05-31T20:27:24.053Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T21:35:41.695 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/pklxkd-55/ 2023-05-31T21:35:41.695 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/pklxkd-55/ 2023-05-31T21:35:41.696 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/pklxkd-55/?prefix=h 2023-05-31T21:35:41.696 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/pklxkd-55/?prefix=h 2023-05-31T21:35:41.696 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-05-31T21:35:41.696 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /pklxkd-55/?prefix=h 2023-05-31T21:35:41.696 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-05-31T21:35:41.697 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-05-31T21:35:41.697 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T21:35:41.697 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-05-31T21:35:41.697 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-05-31T21:35:41.698 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-05-31T21:35:41.698 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-05-31T21:35:41.698 INFO:tasks.rgw_multisite_tests:GET 2023-05-31T21:35:41.698 INFO:tasks.rgw_multisite_tests: 2023-05-31T21:35:41.699 INFO:tasks.rgw_multisite_tests: 2023-05-31T21:35:41.699 INFO:tasks.rgw_multisite_tests:Wed, 31 May 2023 20:29:47 GMT 2023-05-31T21:35:41.699 INFO:tasks.rgw_multisite_tests:/pklxkd-55/ 2023-05-31T21:35:41.699 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-05-31T21:35:41.699 INFO:tasks.rgw_multisite_tests:AWS Q80H11OLNU1S48PPYL26:IBdSyRVlh8qg/aH9XxJ5KZfq+k4= 2023-05-31T21:35:41.700 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:29:47 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:IBdSyRVlh8qg/aH9XxJ5KZfq+k4=', 'Content-Length': '0'} 2023-05-31T21:35:41.700 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e092eb4798e151a8-006477ae3b-1630-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:29:47 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:35:41.700 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'pklxkd-55h1000falseh2023-05-31T20:27:24.053Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T21:35:41.700 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/pklxkd-55/h 2023-05-31T21:35:41.701 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/pklxkd-55/h 2023-05-31T21:35:41.701 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-05-31T21:35:41.701 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /pklxkd-55/h 2023-05-31T21:35:41.701 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-05-31T21:35:41.702 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-05-31T21:35:41.702 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T21:35:41.702 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-05-31T21:35:41.702 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-05-31T21:35:41.702 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-05-31T21:35:41.703 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-05-31T21:35:41.703 INFO:tasks.rgw_multisite_tests:HEAD 2023-05-31T21:35:41.703 INFO:tasks.rgw_multisite_tests: 2023-05-31T21:35:41.703 INFO:tasks.rgw_multisite_tests: 2023-05-31T21:35:41.704 INFO:tasks.rgw_multisite_tests:Wed, 31 May 2023 20:29:47 GMT 2023-05-31T21:35:41.704 INFO:tasks.rgw_multisite_tests:/pklxkd-55/h 2023-05-31T21:35:41.704 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-05-31T21:35:41.704 INFO:tasks.rgw_multisite_tests:AWS Q80H11OLNU1S48PPYL26:tc504Sy/vY0l3VhFKWmlDveznig= 2023-05-31T21:35:41.705 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:29:47 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:tc504Sy/vY0l3VhFKWmlDveznig=', 'Content-Length': '0'} 2023-05-31T21:35:41.705 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:27:24 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000be98e9b5d7ce814d-006477ae3b-162e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:29:47 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:35:41.705 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/pklxkd-55/h 2023-05-31T21:35:41.705 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/pklxkd-55/h 2023-05-31T21:35:41.705 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-05-31T21:35:41.706 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /pklxkd-55/h 2023-05-31T21:35:41.706 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-05-31T21:35:41.706 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-05-31T21:35:41.706 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T21:35:41.707 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-05-31T21:35:41.707 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-05-31T21:35:41.707 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-05-31T21:35:41.707 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-05-31T21:35:41.708 INFO:tasks.rgw_multisite_tests:HEAD 2023-05-31T21:35:41.708 INFO:tasks.rgw_multisite_tests: 2023-05-31T21:35:41.708 INFO:tasks.rgw_multisite_tests: 2023-05-31T21:35:41.708 INFO:tasks.rgw_multisite_tests:Wed, 31 May 2023 20:29:47 GMT 2023-05-31T21:35:41.709 INFO:tasks.rgw_multisite_tests:/pklxkd-55/h 2023-05-31T21:35:41.709 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-05-31T21:35:41.709 INFO:tasks.rgw_multisite_tests:AWS Q80H11OLNU1S48PPYL26:tc504Sy/vY0l3VhFKWmlDveznig= 2023-05-31T21:35:41.709 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:29:47 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:tc504Sy/vY0l3VhFKWmlDveznig=', 'Content-Length': '0'} 2023-05-31T21:35:41.709 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:27:24 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f0f5b4e2e29a3bea-006477ae3b-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:29:47 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:35:41.710 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: comparing key name=h 2023-05-31T21:35:41.710 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/pklxkd-55/h 2023-05-31T21:35:41.710 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/pklxkd-55/h 2023-05-31T21:35:41.710 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-05-31T21:35:41.711 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /pklxkd-55/h 2023-05-31T21:35:41.711 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-05-31T21:35:41.711 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-05-31T21:35:41.711 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T21:35:41.712 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-05-31T21:35:41.712 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-05-31T21:35:41.712 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-05-31T21:35:41.712 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-05-31T21:35:41.712 INFO:tasks.rgw_multisite_tests:GET 2023-05-31T21:35:41.713 INFO:tasks.rgw_multisite_tests: 2023-05-31T21:35:41.713 INFO:tasks.rgw_multisite_tests: 2023-05-31T21:35:41.713 INFO:tasks.rgw_multisite_tests:Wed, 31 May 2023 20:29:47 GMT 2023-05-31T21:35:41.713 INFO:tasks.rgw_multisite_tests:/pklxkd-55/h 2023-05-31T21:35:41.714 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-05-31T21:35:41.714 INFO:tasks.rgw_multisite_tests:AWS Q80H11OLNU1S48PPYL26:rw0y6VdqLvJVfvH8s+6UcVlWvH8= 2023-05-31T21:35:41.714 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:29:47 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:rw0y6VdqLvJVfvH8s+6UcVlWvH8=', 'Content-Length': '0'} 2023-05-31T21:35:41.714 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:27:24 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000bc3b6175cc0c5c60-006477ae3b-162e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:29:47 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:35:41.715 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/pklxkd-55/h 2023-05-31T21:35:41.715 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/pklxkd-55/h 2023-05-31T21:35:41.715 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-05-31T21:35:41.715 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /pklxkd-55/h 2023-05-31T21:35:41.715 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-05-31T21:35:41.716 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-05-31T21:35:41.716 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T21:35:41.716 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-05-31T21:35:41.716 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-05-31T21:35:41.717 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-05-31T21:35:41.717 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-05-31T21:35:41.717 INFO:tasks.rgw_multisite_tests:GET 2023-05-31T21:35:41.717 INFO:tasks.rgw_multisite_tests: 2023-05-31T21:35:41.718 INFO:tasks.rgw_multisite_tests: 2023-05-31T21:35:41.718 INFO:tasks.rgw_multisite_tests:Wed, 31 May 2023 20:29:47 GMT 2023-05-31T21:35:41.718 INFO:tasks.rgw_multisite_tests:/pklxkd-55/h 2023-05-31T21:35:41.718 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-05-31T21:35:41.718 INFO:tasks.rgw_multisite_tests:AWS Q80H11OLNU1S48PPYL26:rw0y6VdqLvJVfvH8s+6UcVlWvH8= 2023-05-31T21:35:41.719 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:29:47 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:rw0y6VdqLvJVfvH8s+6UcVlWvH8=', 'Content-Length': '0'} 2023-05-31T21:35:41.719 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 20:27:24 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000bef3ab43e3161eb6-006477ae3b-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 20:29:47 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:35:41.719 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/pklxkd-55/ 2023-05-31T21:35:41.719 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/pklxkd-55/ 2023-05-31T21:35:41.720 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/pklxkd-55/?prefix=h 2023-05-31T21:35:41.720 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/pklxkd-55/?prefix=h 2023-05-31T21:35:41.720 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-05-31T21:35:41.720 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /pklxkd-55/?prefix=h 2023-05-31T21:35:41.721 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-05-31T21:35:41.721 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-05-31T21:35:41.721 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T21:35:41.721 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-05-31T21:35:41.721 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-05-31T21:35:41.722 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-05-31T21:35:41.722 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-05-31T21:35:41.722 INFO:tasks.rgw_multisite_tests:GET 2023-05-31T21:35:41.722 INFO:tasks.rgw_multisite_tests: 2023-05-31T21:35:41.723 INFO:tasks.rgw_multisite_tests: 2023-05-31T21:35:41.723 INFO:tasks.rgw_multisite_tests:Wed, 31 May 2023 20:29:47 GMT 2023-05-31T21:35:41.723 INFO:tasks.rgw_multisite_tests:/pklxkd-55/ 2023-05-31T21:35:41.723 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-05-31T21:35:41.724 INFO:tasks.rgw_multisite_tests:AWS Q80H11OLNU1S48PPYL26:IBdSyRVlh8qg/aH9XxJ5KZfq+k4= 2023-05-31T21:35:41.724 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:29:47 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:IBdSyRVlh8qg/aH9XxJ5KZfq+k4=', 'Content-Length': '0'} 2023-05-31T21:35:41.724 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000050170fcbf3c14db7-006477ae3b-162e-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:29:47 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:35:41.724 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'pklxkd-55h1000falseh2023-05-31T20:27:24.053Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T21:35:41.725 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/pklxkd-55/ 2023-05-31T21:35:41.725 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/pklxkd-55/ 2023-05-31T21:35:41.725 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/pklxkd-55/?prefix=h 2023-05-31T21:35:41.725 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/pklxkd-55/?prefix=h 2023-05-31T21:35:41.725 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-05-31T21:35:41.726 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /pklxkd-55/?prefix=h 2023-05-31T21:35:41.726 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-05-31T21:35:41.726 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-05-31T21:35:41.726 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T21:35:41.727 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-05-31T21:35:41.727 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-05-31T21:35:41.727 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-05-31T21:35:41.727 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-05-31T21:35:41.728 INFO:tasks.rgw_multisite_tests:GET 2023-05-31T21:35:41.728 INFO:tasks.rgw_multisite_tests: 2023-05-31T21:35:41.728 INFO:tasks.rgw_multisite_tests: 2023-05-31T21:35:41.728 INFO:tasks.rgw_multisite_tests:Wed, 31 May 2023 20:29:47 GMT 2023-05-31T21:35:41.728 INFO:tasks.rgw_multisite_tests:/pklxkd-55/ 2023-05-31T21:35:41.729 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-05-31T21:35:41.729 INFO:tasks.rgw_multisite_tests:AWS Q80H11OLNU1S48PPYL26:IBdSyRVlh8qg/aH9XxJ5KZfq+k4= 2023-05-31T21:35:41.729 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 20:29:47 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:IBdSyRVlh8qg/aH9XxJ5KZfq+k4=', 'Content-Length': '0'} 2023-05-31T21:35:41.729 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000003ee255f8ff6c08c6-006477ae3b-1630-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 20:29:47 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:35:41.730 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'pklxkd-55h1000falseh2023-05-31T20:27:24.053Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T21:35:41.730 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: INFO: success, bucket identical: bucket=pklxkd-55 zones={a2, a1} 2023-05-31T21:35:41.730 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['bilog', 'autotrim', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-05-31T21:35:41.730 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'autotrim', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-05-31T21:35:41.731 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog autotrim --cluster c1 --debug-rgw 0 --debug-ms 0 2023-05-31T21:35:41.731 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: ignoring --setuser ceph since I am not root 2023-05-31T21:35:41.731 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-05-31T21:35:41.731 INFO:tasks.rgw_multisite_tests:tasks.rgw.c1.client.1.smithi099.stdout: INFO: 2023-05-31T20:29:48.097+0000 7f3b0914d700 -1 req 8445471696962185172 0.000999976s :get_bucket_index_log_status ERROR: rgw_read_bucket_full_sync_status() on pipe={s={b=pklxkd-19:3d163b1e-09c7-47e2-ae87-add4aace3f38.4368.19,z=3d163b1e-09c7-47e2-ae87-add4aace3f38,az=0},d={b=pklxkd-19:3d163b1e-09c7-47e2-ae87-add4aace3f38.4368.19,z=5d8acec7-7e57-4e56-8b32-e3d6b5036001,az=0}} returned ret=-2 2023-05-31T21:35:41.731 INFO:tasks.rgw_multisite_tests:tasks.rgw.c1.client.1.smithi099.stdout: INFO: 2023-05-31T20:29:48.097+0000 7f3afd135700 -1 req 1138760661668252224 0.000000000s :get_bucket_index_log_status ERROR: rgw_read_bucket_full_sync_status() on pipe={s={b=pklxkd-43:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.8,z=3d163b1e-09c7-47e2-ae87-add4aace3f38,az=0},d={b=pklxkd-43:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.8,z=5d8acec7-7e57-4e56-8b32-e3d6b5036001,az=0}} returned ret=-2 2023-05-31T21:35:41.732 INFO:tasks.rgw_multisite_tests:tasks.rgw.c1.client.1.smithi099.stdout: INFO: 2023-05-31T20:29:48.099+0000 7f3ad78ea700 -1 req 13455067496672773949 0.000999976s :get_bucket_index_log_status ERROR: rgw_read_bucket_full_sync_status() on pipe={s={b=pklxkd-6:3d163b1e-09c7-47e2-ae87-add4aace3f38.4368.6,z=3d163b1e-09c7-47e2-ae87-add4aace3f38,az=0},d={b=pklxkd-6:3d163b1e-09c7-47e2-ae87-add4aace3f38.4368.6,z=5d8acec7-7e57-4e56-8b32-e3d6b5036001,az=0}} returned ret=-2 2023-05-31T21:35:41.732 INFO:tasks.rgw_multisite_tests:tasks.rgw.c1.client.1.smithi099.stdout: INFO: 2023-05-31T20:29:48.104+0000 7f3af6928700 -1 req 3739170360535518042 0.000000000s :get_bucket_index_log_status ERROR: rgw_read_bucket_full_sync_status() on pipe={s={b=pklxkd-36:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.1,z=3d163b1e-09c7-47e2-ae87-add4aace3f38,az=0},d={b=pklxkd-36:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.1,z=5d8acec7-7e57-4e56-8b32-e3d6b5036001,az=0}} returned ret=-2 2023-05-31T21:35:41.732 INFO:tasks.rgw_multisite_tests:tasks.rgw.c1.client.1.smithi099.stdout: INFO: 2023-05-31T20:29:48.110+0000 7f3ace8d8700 -1 req 15092341274871871355 0.000000000s :get_bucket_index_log_status ERROR: rgw_read_bucket_full_sync_status() on pipe={s={b=pklxkd-45:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.10,z=3d163b1e-09c7-47e2-ae87-add4aace3f38,az=0},d={b=pklxkd-45:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.10,z=5d8acec7-7e57-4e56-8b32-e3d6b5036001,az=0}} returned ret=-2 2023-05-31T21:35:41.732 INFO:tasks.rgw_multisite_tests:tasks.rgw.c1.client.1.smithi099.stdout: INFO: 2023-05-31T20:29:48.145+0000 7f3abf8ba700 -1 req 16689726408984271522 0.000000000s :get_bucket_index_log_status ERROR: rgw_read_bucket_full_sync_status() on pipe={s={b=pklxkd-3:3d163b1e-09c7-47e2-ae87-add4aace3f38.4368.3,z=3d163b1e-09c7-47e2-ae87-add4aace3f38,az=0},d={b=pklxkd-3:3d163b1e-09c7-47e2-ae87-add4aace3f38.4368.3,z=5d8acec7-7e57-4e56-8b32-e3d6b5036001,az=0}} returned ret=-2 2023-05-31T21:35:41.733 INFO:tasks.rgw_multisite_tests:tasks.rgw.c1.client.1.smithi099.stdout: INFO: 2023-05-31T20:29:48.147+0000 7f3ac20bf700 -1 req 15881427886642416641 0.000000000s :get_bucket_index_log_status ERROR: rgw_read_bucket_full_sync_status() on pipe={s={b=pklxkd-1:3d163b1e-09c7-47e2-ae87-add4aace3f38.4368.1,z=3d163b1e-09c7-47e2-ae87-add4aace3f38,az=0},d={b=pklxkd-1:3d163b1e-09c7-47e2-ae87-add4aace3f38.4368.1,z=5d8acec7-7e57-4e56-8b32-e3d6b5036001,az=0}} returned ret=-2 2023-05-31T21:35:41.733 INFO:tasks.rgw_multisite_tests:tasks.rgw.c1.client.1.smithi099.stdout: INFO: 2023-05-31T20:29:48.189+0000 7f3abe0b7700 -1 req 7653350138766254582 0.000000000s :get_bucket_index_log_status ERROR: rgw_read_bucket_full_sync_status() on pipe={s={b=pklxkd-37:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.2,z=3d163b1e-09c7-47e2-ae87-add4aace3f38,az=0},d={b=pklxkd-37:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.2,z=5d8acec7-7e57-4e56-8b32-e3d6b5036001,az=0}} returned ret=-2 2023-05-31T21:35:41.733 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['bucket', 'layout', '--bucket', 'pklxkd-55', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-05-31T21:35:41.733 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'layout', '--bucket', 'pklxkd-55', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-05-31T21:35:41.734 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket layout --bucket pklxkd-55 --cluster c1 --debug-rgw 0 --debug-ms 0 2023-05-31T21:35:41.734 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: ignoring --setuser ceph since I am not root 2023-05-31T21:35:41.734 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-05-31T21:35:41.734 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2023-05-31T21:35:41.735 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "layout": { 2023-05-31T21:35:41.735 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "resharding": "None", 2023-05-31T21:35:41.735 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "current_index": { 2023-05-31T21:35:41.735 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "gen": 2, 2023-05-31T21:35:41.735 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "layout": { 2023-05-31T21:35:41.736 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "type": "Normal", 2023-05-31T21:35:41.736 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "normal": { 2023-05-31T21:35:41.736 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "num_shards": 3, 2023-05-31T21:35:41.736 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "hash_type": "Mod" 2023-05-31T21:35:41.737 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2023-05-31T21:35:41.737 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2023-05-31T21:35:41.737 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2023-05-31T21:35:41.737 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "logs": [ 2023-05-31T21:35:41.738 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2023-05-31T21:35:41.738 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "gen": 1, 2023-05-31T21:35:41.738 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "layout": { 2023-05-31T21:35:41.738 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "type": "InIndex", 2023-05-31T21:35:41.739 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "in_index": { 2023-05-31T21:35:41.739 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "gen": 1, 2023-05-31T21:35:41.739 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "layout": { 2023-05-31T21:35:41.739 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "num_shards": 3, 2023-05-31T21:35:41.739 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "hash_type": "Mod" 2023-05-31T21:35:41.740 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2023-05-31T21:35:41.740 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2023-05-31T21:35:41.740 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2023-05-31T21:35:41.740 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2023-05-31T21:35:41.741 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2023-05-31T21:35:41.741 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "gen": 2, 2023-05-31T21:35:41.741 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "layout": { 2023-05-31T21:35:41.741 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "type": "InIndex", 2023-05-31T21:35:41.742 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "in_index": { 2023-05-31T21:35:41.742 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "gen": 2, 2023-05-31T21:35:41.742 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "layout": { 2023-05-31T21:35:41.742 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "num_shards": 3, 2023-05-31T21:35:41.743 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "hash_type": "Mod" 2023-05-31T21:35:41.743 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2023-05-31T21:35:41.743 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2023-05-31T21:35:41.743 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2023-05-31T21:35:41.743 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2023-05-31T21:35:41.744 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: ] 2023-05-31T21:35:41.744 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2023-05-31T21:35:41.744 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2023-05-31T21:35:41.744 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['bilog', 'autotrim', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-05-31T21:35:41.745 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'autotrim', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-05-31T21:35:41.745 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog autotrim --cluster c1 --debug-rgw 0 --debug-ms 0 2023-05-31T21:35:41.745 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: ignoring --setuser ceph since I am not root 2023-05-31T21:35:41.745 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-05-31T21:35:41.746 INFO:tasks.rgw_multisite_tests:tasks.rgw.c1.client.1.smithi099.stdout: INFO: 2023-05-31T20:29:48.485+0000 7f3aae898700 -1 req 14616980191073695091 0.000000000s :get_bucket_index_log_status ERROR: rgw_read_bucket_full_sync_status() on pipe={s={b=pklxkd-4:3d163b1e-09c7-47e2-ae87-add4aace3f38.4368.4,z=3d163b1e-09c7-47e2-ae87-add4aace3f38,az=0},d={b=pklxkd-4:3d163b1e-09c7-47e2-ae87-add4aace3f38.4368.4,z=5d8acec7-7e57-4e56-8b32-e3d6b5036001,az=0}} returned ret=-2 2023-05-31T21:35:41.746 INFO:tasks.rgw_multisite_tests:tasks.rgw.c1.client.1.smithi099.stdout: INFO: 2023-05-31T20:29:48.485+0000 7f3aa808b700 -1 req 12861548945062890721 0.000000000s :get_bucket_index_log_status ERROR: rgw_read_bucket_full_sync_status() on pipe={s={b=pklxkd-2:3d163b1e-09c7-47e2-ae87-add4aace3f38.4368.2,z=3d163b1e-09c7-47e2-ae87-add4aace3f38,az=0},d={b=pklxkd-2:3d163b1e-09c7-47e2-ae87-add4aace3f38.4368.2,z=5d8acec7-7e57-4e56-8b32-e3d6b5036001,az=0}} returned ret=-2 2023-05-31T21:35:41.746 INFO:tasks.rgw_multisite_tests:tasks.rgw.c1.client.1.smithi099.stdout: INFO: 2023-05-31T20:29:48.485+0000 7f3ab80ab700 -1 req 2137930933412329992 0.000000000s :get_bucket_index_log_status ERROR: rgw_read_bucket_full_sync_status() on pipe={s={b=pklxkd-28:3d163b1e-09c7-47e2-ae87-add4aace3f38.4368.28,z=3d163b1e-09c7-47e2-ae87-add4aace3f38,az=0},d={b=pklxkd-28:3d163b1e-09c7-47e2-ae87-add4aace3f38.4368.28,z=5d8acec7-7e57-4e56-8b32-e3d6b5036001,az=0}} returned ret=-2 2023-05-31T21:35:41.746 INFO:tasks.rgw_multisite_tests:tasks.rgw.c1.client.1.smithi099.stdout: INFO: 2023-05-31T20:29:48.488+0000 7f3a8f85a700 -1 req 2531218361705108531 0.000999977s :get_bucket_index_log_status ERROR: rgw_read_bucket_full_sync_status() on pipe={s={b=pklxkd-40:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.5,z=3d163b1e-09c7-47e2-ae87-add4aace3f38,az=0},d={b=pklxkd-40:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.5,z=5d8acec7-7e57-4e56-8b32-e3d6b5036001,az=0}} returned ret=-2 2023-05-31T21:35:41.747 INFO:tasks.rgw_multisite_tests:tasks.rgw.c1.client.1.smithi099.stdout: INFO: 2023-05-31T20:29:48.488+0000 7f3a97069700 -1 req 4722288278995007716 0.000999977s :get_bucket_index_log_status ERROR: rgw_read_bucket_full_sync_status() on pipe={s={b=pklxkd-5:3d163b1e-09c7-47e2-ae87-add4aace3f38.4368.5,z=3d163b1e-09c7-47e2-ae87-add4aace3f38,az=0},d={b=pklxkd-5:3d163b1e-09c7-47e2-ae87-add4aace3f38.4368.5,z=5d8acec7-7e57-4e56-8b32-e3d6b5036001,az=0}} returned ret=-2 2023-05-31T21:35:41.747 INFO:tasks.rgw_multisite_tests:tasks.rgw.c1.client.1.smithi099.stdout: INFO: 2023-05-31T20:29:48.531+0000 7f3a94063700 -1 req 13620524963349987869 0.000999977s :get_bucket_index_log_status ERROR: rgw_read_bucket_full_sync_status() on pipe={s={b=pklxkd-29:3d163b1e-09c7-47e2-ae87-add4aace3f38.4368.29,z=3d163b1e-09c7-47e2-ae87-add4aace3f38,az=0},d={b=pklxkd-29:3d163b1e-09c7-47e2-ae87-add4aace3f38.4368.29,z=5d8acec7-7e57-4e56-8b32-e3d6b5036001,az=0}} returned ret=-2 2023-05-31T21:35:41.747 INFO:tasks.rgw_multisite_tests:tasks.rgw.c1.client.1.smithi099.stdout: INFO: 2023-05-31T20:29:48.535+0000 7f3b5a1ef700 -1 req 15780290087924430216 0.000000000s :get_bucket_index_log_status ERROR: rgw_read_bucket_full_sync_status() on pipe={s={b=pklxkd-30:3d163b1e-09c7-47e2-ae87-add4aace3f38.4368.30,z=3d163b1e-09c7-47e2-ae87-add4aace3f38,az=0},d={b=pklxkd-30:3d163b1e-09c7-47e2-ae87-add4aace3f38.4368.30,z=5d8acec7-7e57-4e56-8b32-e3d6b5036001,az=0}} returned ret=-2 2023-05-31T21:35:41.747 INFO:tasks.rgw_multisite_tests:tasks.rgw.c1.client.1.smithi099.stdout: INFO: 2023-05-31T20:29:48.573+0000 7f3a72820700 -1 req 15950283960328835668 0.000000000s :get_bucket_index_log_status ERROR: rgw_read_bucket_full_sync_status() on pipe={s={b=pklxkd-51:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.16,z=3d163b1e-09c7-47e2-ae87-add4aace3f38,az=0},d={b=pklxkd-51:3d163b1e-09c7-47e2-ae87-add4aace3f38.5684.16,z=5d8acec7-7e57-4e56-8b32-e3d6b5036001,az=0}} returned ret=-2 2023-05-31T21:35:41.748 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['bucket', 'layout', '--bucket', 'pklxkd-55', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-05-31T21:35:41.748 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'layout', '--bucket', 'pklxkd-55', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-05-31T21:35:41.748 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket layout --bucket pklxkd-55 --cluster c1 --debug-rgw 0 --debug-ms 0 2023-05-31T21:35:41.748 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: ignoring --setuser ceph since I am not root 2023-05-31T21:35:41.749 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-05-31T21:35:41.749 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2023-05-31T21:35:41.749 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "layout": { 2023-05-31T21:35:41.749 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "resharding": "None", 2023-05-31T21:35:41.749 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "current_index": { 2023-05-31T21:35:41.750 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "gen": 2, 2023-05-31T21:35:41.750 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "layout": { 2023-05-31T21:35:41.750 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "type": "Normal", 2023-05-31T21:35:41.750 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "normal": { 2023-05-31T21:35:41.751 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "num_shards": 3, 2023-05-31T21:35:41.751 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "hash_type": "Mod" 2023-05-31T21:35:41.751 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2023-05-31T21:35:41.751 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2023-05-31T21:35:41.752 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2023-05-31T21:35:41.752 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "logs": [ 2023-05-31T21:35:41.752 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2023-05-31T21:35:41.752 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "gen": 1, 2023-05-31T21:35:41.752 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "layout": { 2023-05-31T21:35:41.753 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "type": "InIndex", 2023-05-31T21:35:41.753 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "in_index": { 2023-05-31T21:35:41.753 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "gen": 1, 2023-05-31T21:35:41.753 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "layout": { 2023-05-31T21:35:41.754 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "num_shards": 3, 2023-05-31T21:35:41.754 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "hash_type": "Mod" 2023-05-31T21:35:41.754 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2023-05-31T21:35:41.754 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2023-05-31T21:35:41.755 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2023-05-31T21:35:41.755 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2023-05-31T21:35:41.755 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2023-05-31T21:35:41.755 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "gen": 2, 2023-05-31T21:35:41.756 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "layout": { 2023-05-31T21:35:41.756 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "type": "InIndex", 2023-05-31T21:35:41.756 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "in_index": { 2023-05-31T21:35:41.756 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "gen": 2, 2023-05-31T21:35:41.756 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "layout": { 2023-05-31T21:35:41.757 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "num_shards": 3, 2023-05-31T21:35:41.757 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "hash_type": "Mod" 2023-05-31T21:35:41.757 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2023-05-31T21:35:41.757 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2023-05-31T21:35:41.758 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2023-05-31T21:35:41.758 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2023-05-31T21:35:41.758 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: ] 2023-05-31T21:35:41.758 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2023-05-31T21:35:41.759 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2023-05-31T21:35:41.759 INFO:tasks.rgw_multisite_tests:--------------------- >> end captured logging << --------------------- 2023-05-31T21:35:41.759 INFO:tasks.rgw_multisite_tests: 2023-05-31T21:35:41.760 INFO:tasks.rgw_multisite_tests:====================================================================== 2023-05-31T21:35:41.760 INFO:tasks.rgw_multisite_tests:FAIL: rgw_multi.tests.test_resharded_bucket_incremental_sync_latest_after_data_sync_init 2023-05-31T21:35:41.760 INFO:tasks.rgw_multisite_tests:---------------------------------------------------------------------- 2023-05-31T21:35:41.761 INFO:tasks.rgw_multisite_tests:Traceback (most recent call last): 2023-05-31T21:35:41.761 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_teuthology_fa97ddd17d12e6d74b49196948f66e54fa009130/virtualenv/lib/python3.8/site-packages/nose/case.py", line 198, in runTest 2023-05-31T21:35:41.761 INFO:tasks.rgw_multisite_tests: self.test(*self.arg) 2023-05-31T21:35:41.762 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/github.com_ceph_ceph-c_2870bf456d3115490c153abce9d47ac5c043b65e/qa/../src/test/rgw/rgw_multi/tests.py", line 1842, in test_resharded_bucket_incremental_sync_latest_after_data_sync_init 2023-05-31T21:35:41.762 INFO:tasks.rgw_multisite_tests: zonegroup_bucket_checkpoint(zonegroup_conns, bucket.name) 2023-05-31T21:35:41.762 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/github.com_ceph_ceph-c_2870bf456d3115490c153abce9d47ac5c043b65e/qa/../src/test/rgw/rgw_multi/tests.py", line 395, in zonegroup_bucket_checkpoint 2023-05-31T21:35:41.762 INFO:tasks.rgw_multisite_tests: zone_bucket_checkpoint(target_conn.zone, source_conn.zone, bucket_name) 2023-05-31T21:35:41.762 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/github.com_ceph_ceph-c_2870bf456d3115490c153abce9d47ac5c043b65e/qa/../src/test/rgw/rgw_multi/tests.py", line 387, in zone_bucket_checkpoint 2023-05-31T21:35:41.763 INFO:tasks.rgw_multisite_tests: target_zone.cluster.admin(cmd, debug_rgw=1) 2023-05-31T21:35:41.763 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/github.com_ceph_ceph-c_2870bf456d3115490c153abce9d47ac5c043b65e/qa/tasks/rgw_multisite.py", line 197, in admin 2023-05-31T21:35:41.763 INFO:tasks.rgw_multisite_tests: assert r == 0 2023-05-31T21:35:41.763 INFO:tasks.rgw_multisite_tests:AssertionError: 2023-05-31T21:35:41.764 INFO:tasks.rgw_multisite_tests:-------------------- >> begin captured logging << -------------------- 2023-05-31T21:35:41.764 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/pklxkd-63/ 2023-05-31T21:35:41.764 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/pklxkd-63/ 2023-05-31T21:35:41.764 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2023-05-31T21:35:41.765 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /pklxkd-63/ 2023-05-31T21:35:41.765 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-05-31T21:35:41.765 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-05-31T21:35:41.765 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T21:35:41.766 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-05-31T21:35:41.766 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-05-31T21:35:41.766 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-05-31T21:35:41.766 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-05-31T21:35:41.766 INFO:tasks.rgw_multisite_tests:PUT 2023-05-31T21:35:41.767 INFO:tasks.rgw_multisite_tests: 2023-05-31T21:35:41.767 INFO:tasks.rgw_multisite_tests: 2023-05-31T21:35:41.767 INFO:tasks.rgw_multisite_tests:Wed, 31 May 2023 21:06:40 GMT 2023-05-31T21:35:41.767 INFO:tasks.rgw_multisite_tests:/pklxkd-63/ 2023-05-31T21:35:41.768 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-05-31T21:35:41.768 INFO:tasks.rgw_multisite_tests:AWS Q80H11OLNU1S48PPYL26:mTfmj5LSAzRFhEtl8LC5DrYrS74= 2023-05-31T21:35:41.768 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 21:06:40 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:mTfmj5LSAzRFhEtl8LC5DrYrS74=', 'Content-Length': '0'} 2023-05-31T21:35:41.768 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('x-amz-request-id', 'tx000003568f58ea5ffa0c5-006477b6e0-1630-a1'), ('Content-Length', '0'), ('Date', 'Wed, 31 May 2023 21:06:40 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:35:41.769 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: created bucket=pklxkd-63 2023-05-31T21:35:41.769 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-31T21:35:41.769 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-31T21:35:41.769 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-05-31T21:35:41.769 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: ignoring --setuser ceph since I am not root 2023-05-31T21:35:41.770 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-05-31T21:35:41.770 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: No --period given, using current period=2e6123d2-d695-4350-9a01-61a455bc782b 2023-05-31T21:35:41.770 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: [ 2023-05-31T21:35:41.770 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2023-05-31T21:35:41.771 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "marker": "1_1685566501.757183_260.1", 2023-05-31T21:35:41.771 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "last_update": "2023-05-31T20:55:01.757183Z" 2023-05-31T21:35:41.771 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2023-05-31T21:35:41.771 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2023-05-31T21:35:41.772 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "marker": "1_1685566501.735717_251.1", 2023-05-31T21:35:41.772 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "last_update": "2023-05-31T20:55:01.735717Z" 2023-05-31T21:35:41.772 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2023-05-31T21:35:41.772 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2023-05-31T21:35:41.773 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "marker": "1_1685562970.274572_93.1", 2023-05-31T21:35:41.773 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "last_update": "2023-05-31T19:56:10.274572Z" 2023-05-31T21:35:41.773 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2023-05-31T21:35:41.773 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2023-05-31T21:35:41.773 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "marker": "1_1685567200.773333_576.1", 2023-05-31T21:35:41.774 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "last_update": "2023-05-31T21:06:40.773333Z" 2023-05-31T21:35:41.774 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2023-05-31T21:35:41.774 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: ] 2023-05-31T21:35:41.774 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: master meta markers={0: '1_1685566501.757183_260.1', 1: '1_1685566501.735717_251.1', 2: '1_1685562970.274572_93.1', 3: '1_1685567200.773333_576.1'} 2023-05-31T21:35:41.775 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: INFO: starting meta checkpoint for zone=a2 2023-05-31T21:35:41.775 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-31T21:35:41.775 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-31T21:35:41.775 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-05-31T21:35:41.775 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: ignoring --setuser ceph since I am not root 2023-05-31T21:35:41.776 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-05-31T21:35:41.776 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: 2023-05-31T21:06:41.001+0000 7fbc594f4780 20 RGW-SYNC:meta: read sync status 2023-05-31T21:35:41.776 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: 2023-05-31T21:06:41.002+0000 7fbc594f4780 20 RGW-SYNC:meta: read sync status 2023-05-31T21:35:41.776 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2023-05-31T21:35:41.777 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "sync_status": { 2023-05-31T21:35:41.777 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "info": { 2023-05-31T21:35:41.777 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "status": "sync", 2023-05-31T21:35:41.777 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "num_shards": 4, 2023-05-31T21:35:41.778 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "period": "2e6123d2-d695-4350-9a01-61a455bc782b", 2023-05-31T21:35:41.778 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "realm_epoch": 2 2023-05-31T21:35:41.778 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2023-05-31T21:35:41.778 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "markers": [ 2023-05-31T21:35:41.778 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2023-05-31T21:35:41.779 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "key": 0, 2023-05-31T21:35:41.779 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "val": { 2023-05-31T21:35:41.779 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "state": 1, 2023-05-31T21:35:41.779 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "marker": "1_1685566501.757183_260.1", 2023-05-31T21:35:41.780 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "next_step_marker": "", 2023-05-31T21:35:41.780 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "total_entries": 1, 2023-05-31T21:35:41.780 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "pos": 0, 2023-05-31T21:35:41.780 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "timestamp": "2023-05-31T20:55:01.757183Z", 2023-05-31T21:35:41.781 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "realm_epoch": 2 2023-05-31T21:35:41.781 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2023-05-31T21:35:41.781 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2023-05-31T21:35:41.781 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2023-05-31T21:35:41.781 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "key": 1, 2023-05-31T21:35:41.782 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "val": { 2023-05-31T21:35:41.782 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "state": 1, 2023-05-31T21:35:41.782 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "marker": "1_1685566501.735717_251.1", 2023-05-31T21:35:41.783 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "next_step_marker": "", 2023-05-31T21:35:41.783 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "total_entries": 0, 2023-05-31T21:35:41.783 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "pos": 0, 2023-05-31T21:35:41.783 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "timestamp": "2023-05-31T20:55:01.735717Z", 2023-05-31T21:35:41.783 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "realm_epoch": 2 2023-05-31T21:35:41.784 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2023-05-31T21:35:41.784 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2023-05-31T21:35:41.784 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2023-05-31T21:35:41.784 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "key": 2, 2023-05-31T21:35:41.785 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "val": { 2023-05-31T21:35:41.785 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "state": 1, 2023-05-31T21:35:41.785 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "marker": "1_1685562970.274572_93.1", 2023-05-31T21:35:41.785 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "next_step_marker": "", 2023-05-31T21:35:41.786 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "total_entries": 0, 2023-05-31T21:35:41.786 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "pos": 0, 2023-05-31T21:35:41.786 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "timestamp": "2023-05-31T19:56:10.274572Z", 2023-05-31T21:35:41.786 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "realm_epoch": 2 2023-05-31T21:35:41.786 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2023-05-31T21:35:41.787 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2023-05-31T21:35:41.787 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2023-05-31T21:35:41.787 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "key": 3, 2023-05-31T21:35:41.787 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "val": { 2023-05-31T21:35:41.788 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "state": 1, 2023-05-31T21:35:41.788 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "marker": "1_1685567200.767436_573.1", 2023-05-31T21:35:41.788 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "next_step_marker": "", 2023-05-31T21:35:41.788 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "total_entries": 0, 2023-05-31T21:35:41.789 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "pos": 0, 2023-05-31T21:35:41.789 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "timestamp": "2023-05-31T21:06:40.767436Z", 2023-05-31T21:35:41.789 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "realm_epoch": 2 2023-05-31T21:35:41.789 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2023-05-31T21:35:41.789 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2023-05-31T21:35:41.790 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: ] 2023-05-31T21:35:41.790 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2023-05-31T21:35:41.790 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "full_sync": { 2023-05-31T21:35:41.790 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "total": 1, 2023-05-31T21:35:41.791 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "complete": 1 2023-05-31T21:35:41.791 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2023-05-31T21:35:41.791 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "current_time": "2023-05-31T21:06:41Z" 2023-05-31T21:35:41.791 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2023-05-31T21:35:41.792 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2023-05-31T21:35:41.792 INFO:tasks.rgw_multisite_tests: "sync_status": { 2023-05-31T21:35:41.792 INFO:tasks.rgw_multisite_tests: "info": { 2023-05-31T21:35:41.792 INFO:tasks.rgw_multisite_tests: "status": "sync", 2023-05-31T21:35:41.793 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2023-05-31T21:35:41.793 INFO:tasks.rgw_multisite_tests: "period": "2e6123d2-d695-4350-9a01-61a455bc782b", 2023-05-31T21:35:41.793 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-05-31T21:35:41.793 INFO:tasks.rgw_multisite_tests: }, 2023-05-31T21:35:41.793 INFO:tasks.rgw_multisite_tests: "markers": [ 2023-05-31T21:35:41.794 INFO:tasks.rgw_multisite_tests: { 2023-05-31T21:35:41.794 INFO:tasks.rgw_multisite_tests: "key": 0, 2023-05-31T21:35:41.794 INFO:tasks.rgw_multisite_tests: "val": { 2023-05-31T21:35:41.794 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-05-31T21:35:41.795 INFO:tasks.rgw_multisite_tests: "marker": "1_1685566501.757183_260.1", 2023-05-31T21:35:41.795 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-05-31T21:35:41.795 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2023-05-31T21:35:41.795 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-05-31T21:35:41.796 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-05-31T20:55:01.757183Z", 2023-05-31T21:35:41.796 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-05-31T21:35:41.796 INFO:tasks.rgw_multisite_tests: } 2023-05-31T21:35:41.796 INFO:tasks.rgw_multisite_tests: }, 2023-05-31T21:35:41.797 INFO:tasks.rgw_multisite_tests: { 2023-05-31T21:35:41.797 INFO:tasks.rgw_multisite_tests: "key": 1, 2023-05-31T21:35:41.797 INFO:tasks.rgw_multisite_tests: "val": { 2023-05-31T21:35:41.797 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-05-31T21:35:41.797 INFO:tasks.rgw_multisite_tests: "marker": "1_1685566501.735717_251.1", 2023-05-31T21:35:41.798 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-05-31T21:35:41.798 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-05-31T21:35:41.798 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-05-31T21:35:41.798 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-05-31T20:55:01.735717Z", 2023-05-31T21:35:41.799 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-05-31T21:35:41.799 INFO:tasks.rgw_multisite_tests: } 2023-05-31T21:35:41.799 INFO:tasks.rgw_multisite_tests: }, 2023-05-31T21:35:41.799 INFO:tasks.rgw_multisite_tests: { 2023-05-31T21:35:41.799 INFO:tasks.rgw_multisite_tests: "key": 2, 2023-05-31T21:35:41.800 INFO:tasks.rgw_multisite_tests: "val": { 2023-05-31T21:35:41.800 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-05-31T21:35:41.800 INFO:tasks.rgw_multisite_tests: "marker": "1_1685562970.274572_93.1", 2023-05-31T21:35:41.800 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-05-31T21:35:41.801 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-05-31T21:35:41.801 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-05-31T21:35:41.801 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-05-31T19:56:10.274572Z", 2023-05-31T21:35:41.801 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-05-31T21:35:41.802 INFO:tasks.rgw_multisite_tests: } 2023-05-31T21:35:41.802 INFO:tasks.rgw_multisite_tests: }, 2023-05-31T21:35:41.802 INFO:tasks.rgw_multisite_tests: { 2023-05-31T21:35:41.802 INFO:tasks.rgw_multisite_tests: "key": 3, 2023-05-31T21:35:41.802 INFO:tasks.rgw_multisite_tests: "val": { 2023-05-31T21:35:41.803 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-05-31T21:35:41.803 INFO:tasks.rgw_multisite_tests: "marker": "1_1685567200.767436_573.1", 2023-05-31T21:35:41.803 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-05-31T21:35:41.803 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-05-31T21:35:41.804 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-05-31T21:35:41.804 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-05-31T21:06:40.767436Z", 2023-05-31T21:35:41.804 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-05-31T21:35:41.804 INFO:tasks.rgw_multisite_tests: } 2023-05-31T21:35:41.805 INFO:tasks.rgw_multisite_tests: } 2023-05-31T21:35:41.805 INFO:tasks.rgw_multisite_tests: ] 2023-05-31T21:35:41.805 INFO:tasks.rgw_multisite_tests: }, 2023-05-31T21:35:41.805 INFO:tasks.rgw_multisite_tests: "full_sync": { 2023-05-31T21:35:41.805 INFO:tasks.rgw_multisite_tests: "total": 1, 2023-05-31T21:35:41.806 INFO:tasks.rgw_multisite_tests: "complete": 1 2023-05-31T21:35:41.806 INFO:tasks.rgw_multisite_tests: }, 2023-05-31T21:35:41.806 INFO:tasks.rgw_multisite_tests: "current_time": "2023-05-31T21:06:41Z" 2023-05-31T21:35:41.806 INFO:tasks.rgw_multisite_tests:} 2023-05-31T21:35:41.807 INFO:tasks.rgw_multisite_tests: 2023-05-31T21:35:41.807 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1685566501.757183_260.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-05-31T20:55:01.757183Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1685566501.735717_251.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-05-31T20:55:01.735717Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1685562970.274572_93.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-05-31T19:56:10.274572Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1685567200.767436_573.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-05-31T21:06:40.767436Z', 'realm_epoch': 2}}] 2023-05-31T21:35:41.807 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1685566501.757183_260.1', 1: '1_1685566501.735717_251.1', 2: '1_1685562970.274572_93.1', 3: '1_1685567200.773333_576.1'} 2023-05-31T21:35:41.807 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1685566501.757183_260.1', 1: '1_1685566501.735717_251.1', 2: '1_1685562970.274572_93.1', 3: '1_1685567200.767436_573.1'} 2023-05-31T21:35:41.808 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone a2 behind master: shard=3 master=1_1685567200.773333_576.1 target=1_1685567200.767436_573.1 2023-05-31T21:35:41.808 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-31T21:35:41.808 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-31T21:35:41.808 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-05-31T21:35:41.809 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: ignoring --setuser ceph since I am not root 2023-05-31T21:35:41.809 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-05-31T21:35:41.809 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: 2023-05-31T21:06:46.147+0000 7fe186130780 20 RGW-SYNC:meta: read sync status 2023-05-31T21:35:41.809 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: 2023-05-31T21:06:46.148+0000 7fe186130780 20 RGW-SYNC:meta: read sync status 2023-05-31T21:35:41.809 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2023-05-31T21:35:41.810 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "sync_status": { 2023-05-31T21:35:41.810 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "info": { 2023-05-31T21:35:41.810 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "status": "sync", 2023-05-31T21:35:41.810 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "num_shards": 4, 2023-05-31T21:35:41.811 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "period": "2e6123d2-d695-4350-9a01-61a455bc782b", 2023-05-31T21:35:41.811 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "realm_epoch": 2 2023-05-31T21:35:41.811 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2023-05-31T21:35:41.811 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "markers": [ 2023-05-31T21:35:41.811 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2023-05-31T21:35:41.812 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "key": 0, 2023-05-31T21:35:41.812 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "val": { 2023-05-31T21:35:41.812 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "state": 1, 2023-05-31T21:35:41.812 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "marker": "1_1685566501.757183_260.1", 2023-05-31T21:35:41.813 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "next_step_marker": "", 2023-05-31T21:35:41.813 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "total_entries": 1, 2023-05-31T21:35:41.813 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "pos": 0, 2023-05-31T21:35:41.813 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "timestamp": "2023-05-31T20:55:01.757183Z", 2023-05-31T21:35:41.814 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "realm_epoch": 2 2023-05-31T21:35:41.814 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2023-05-31T21:35:41.814 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2023-05-31T21:35:41.814 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2023-05-31T21:35:41.814 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "key": 1, 2023-05-31T21:35:41.815 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "val": { 2023-05-31T21:35:41.815 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "state": 1, 2023-05-31T21:35:41.815 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "marker": "1_1685566501.735717_251.1", 2023-05-31T21:35:41.815 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "next_step_marker": "", 2023-05-31T21:35:41.816 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "total_entries": 0, 2023-05-31T21:35:41.816 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "pos": 0, 2023-05-31T21:35:41.816 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "timestamp": "2023-05-31T20:55:01.735717Z", 2023-05-31T21:35:41.816 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "realm_epoch": 2 2023-05-31T21:35:41.817 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2023-05-31T21:35:41.817 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2023-05-31T21:35:41.817 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2023-05-31T21:35:41.817 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "key": 2, 2023-05-31T21:35:41.818 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "val": { 2023-05-31T21:35:41.818 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "state": 1, 2023-05-31T21:35:41.818 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "marker": "1_1685562970.274572_93.1", 2023-05-31T21:35:41.818 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "next_step_marker": "", 2023-05-31T21:35:41.818 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "total_entries": 0, 2023-05-31T21:35:41.819 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "pos": 0, 2023-05-31T21:35:41.819 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "timestamp": "2023-05-31T19:56:10.274572Z", 2023-05-31T21:35:41.819 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "realm_epoch": 2 2023-05-31T21:35:41.819 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2023-05-31T21:35:41.820 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2023-05-31T21:35:41.820 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2023-05-31T21:35:41.820 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "key": 3, 2023-05-31T21:35:41.820 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "val": { 2023-05-31T21:35:41.820 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "state": 1, 2023-05-31T21:35:41.821 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "marker": "1_1685567200.773333_576.1", 2023-05-31T21:35:41.821 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "next_step_marker": "", 2023-05-31T21:35:41.821 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "total_entries": 0, 2023-05-31T21:35:41.821 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "pos": 0, 2023-05-31T21:35:41.822 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "timestamp": "2023-05-31T21:06:40.773333Z", 2023-05-31T21:35:41.822 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "realm_epoch": 2 2023-05-31T21:35:41.822 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2023-05-31T21:35:41.822 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2023-05-31T21:35:41.823 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: ] 2023-05-31T21:35:41.823 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2023-05-31T21:35:41.823 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "full_sync": { 2023-05-31T21:35:41.823 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "total": 1, 2023-05-31T21:35:41.824 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "complete": 1 2023-05-31T21:35:41.824 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2023-05-31T21:35:41.824 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "current_time": "2023-05-31T21:06:46Z" 2023-05-31T21:35:41.824 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2023-05-31T21:35:41.825 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2023-05-31T21:35:41.825 INFO:tasks.rgw_multisite_tests: "sync_status": { 2023-05-31T21:35:41.825 INFO:tasks.rgw_multisite_tests: "info": { 2023-05-31T21:35:41.825 INFO:tasks.rgw_multisite_tests: "status": "sync", 2023-05-31T21:35:41.825 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2023-05-31T21:35:41.826 INFO:tasks.rgw_multisite_tests: "period": "2e6123d2-d695-4350-9a01-61a455bc782b", 2023-05-31T21:35:41.826 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-05-31T21:35:41.826 INFO:tasks.rgw_multisite_tests: }, 2023-05-31T21:35:41.826 INFO:tasks.rgw_multisite_tests: "markers": [ 2023-05-31T21:35:41.827 INFO:tasks.rgw_multisite_tests: { 2023-05-31T21:35:41.827 INFO:tasks.rgw_multisite_tests: "key": 0, 2023-05-31T21:35:41.827 INFO:tasks.rgw_multisite_tests: "val": { 2023-05-31T21:35:41.827 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-05-31T21:35:41.828 INFO:tasks.rgw_multisite_tests: "marker": "1_1685566501.757183_260.1", 2023-05-31T21:35:41.828 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-05-31T21:35:41.828 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2023-05-31T21:35:41.828 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-05-31T21:35:41.828 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-05-31T20:55:01.757183Z", 2023-05-31T21:35:41.829 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-05-31T21:35:41.829 INFO:tasks.rgw_multisite_tests: } 2023-05-31T21:35:41.829 INFO:tasks.rgw_multisite_tests: }, 2023-05-31T21:35:41.829 INFO:tasks.rgw_multisite_tests: { 2023-05-31T21:35:41.830 INFO:tasks.rgw_multisite_tests: "key": 1, 2023-05-31T21:35:41.830 INFO:tasks.rgw_multisite_tests: "val": { 2023-05-31T21:35:41.830 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-05-31T21:35:41.830 INFO:tasks.rgw_multisite_tests: "marker": "1_1685566501.735717_251.1", 2023-05-31T21:35:41.830 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-05-31T21:35:41.831 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-05-31T21:35:41.831 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-05-31T21:35:41.831 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-05-31T20:55:01.735717Z", 2023-05-31T21:35:41.831 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-05-31T21:35:41.832 INFO:tasks.rgw_multisite_tests: } 2023-05-31T21:35:41.832 INFO:tasks.rgw_multisite_tests: }, 2023-05-31T21:35:41.832 INFO:tasks.rgw_multisite_tests: { 2023-05-31T21:35:41.832 INFO:tasks.rgw_multisite_tests: "key": 2, 2023-05-31T21:35:41.833 INFO:tasks.rgw_multisite_tests: "val": { 2023-05-31T21:35:41.833 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-05-31T21:35:41.833 INFO:tasks.rgw_multisite_tests: "marker": "1_1685562970.274572_93.1", 2023-05-31T21:35:41.833 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-05-31T21:35:41.834 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-05-31T21:35:41.834 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-05-31T21:35:41.834 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-05-31T19:56:10.274572Z", 2023-05-31T21:35:41.834 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-05-31T21:35:41.834 INFO:tasks.rgw_multisite_tests: } 2023-05-31T21:35:41.835 INFO:tasks.rgw_multisite_tests: }, 2023-05-31T21:35:41.835 INFO:tasks.rgw_multisite_tests: { 2023-05-31T21:35:41.835 INFO:tasks.rgw_multisite_tests: "key": 3, 2023-05-31T21:35:41.835 INFO:tasks.rgw_multisite_tests: "val": { 2023-05-31T21:35:41.836 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-05-31T21:35:41.836 INFO:tasks.rgw_multisite_tests: "marker": "1_1685567200.773333_576.1", 2023-05-31T21:35:41.836 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-05-31T21:35:41.836 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-05-31T21:35:41.837 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-05-31T21:35:41.837 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-05-31T21:06:40.773333Z", 2023-05-31T21:35:41.837 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-05-31T21:35:41.837 INFO:tasks.rgw_multisite_tests: } 2023-05-31T21:35:41.837 INFO:tasks.rgw_multisite_tests: } 2023-05-31T21:35:41.838 INFO:tasks.rgw_multisite_tests: ] 2023-05-31T21:35:41.838 INFO:tasks.rgw_multisite_tests: }, 2023-05-31T21:35:41.838 INFO:tasks.rgw_multisite_tests: "full_sync": { 2023-05-31T21:35:41.838 INFO:tasks.rgw_multisite_tests: "total": 1, 2023-05-31T21:35:41.839 INFO:tasks.rgw_multisite_tests: "complete": 1 2023-05-31T21:35:41.839 INFO:tasks.rgw_multisite_tests: }, 2023-05-31T21:35:41.839 INFO:tasks.rgw_multisite_tests: "current_time": "2023-05-31T21:06:46Z" 2023-05-31T21:35:41.839 INFO:tasks.rgw_multisite_tests:} 2023-05-31T21:35:41.840 INFO:tasks.rgw_multisite_tests: 2023-05-31T21:35:41.840 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1685566501.757183_260.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-05-31T20:55:01.757183Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1685566501.735717_251.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-05-31T20:55:01.735717Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1685562970.274572_93.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-05-31T19:56:10.274572Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1685567200.773333_576.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-05-31T21:06:40.773333Z', 'realm_epoch': 2}}] 2023-05-31T21:35:41.840 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1685566501.757183_260.1', 1: '1_1685566501.735717_251.1', 2: '1_1685562970.274572_93.1', 3: '1_1685567200.773333_576.1'} 2023-05-31T21:35:41.840 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1685566501.757183_260.1', 1: '1_1685566501.735717_251.1', 2: '1_1685562970.274572_93.1', 3: '1_1685567200.773333_576.1'} 2023-05-31T21:35:41.840 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: INFO: finish meta checkpoint for zone=a2 2023-05-31T21:35:41.841 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/pklxkd-63/ 2023-05-31T21:35:41.841 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/pklxkd-63/ 2023-05-31T21:35:41.841 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-05-31T21:35:41.841 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /pklxkd-63/ 2023-05-31T21:35:41.842 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-05-31T21:35:41.842 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-05-31T21:35:41.842 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T21:35:41.842 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-05-31T21:35:41.843 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-05-31T21:35:41.843 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-05-31T21:35:41.843 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-05-31T21:35:41.843 INFO:tasks.rgw_multisite_tests:HEAD 2023-05-31T21:35:41.843 INFO:tasks.rgw_multisite_tests: 2023-05-31T21:35:41.844 INFO:tasks.rgw_multisite_tests: 2023-05-31T21:35:41.844 INFO:tasks.rgw_multisite_tests:Wed, 31 May 2023 21:06:46 GMT 2023-05-31T21:35:41.844 INFO:tasks.rgw_multisite_tests:/pklxkd-63/ 2023-05-31T21:35:41.844 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-05-31T21:35:41.845 INFO:tasks.rgw_multisite_tests:AWS Q80H11OLNU1S48PPYL26:bcNVcPZSg/FRj1VT8thTtJNmxUQ= 2023-05-31T21:35:41.845 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 21:06:46 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:bcNVcPZSg/FRj1VT8thTtJNmxUQ=', 'Content-Length': '0'} 2023-05-31T21:35:41.845 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000075c5c7474026f237-006477b6e6-1630-a1'), ('Content-Length', '0'), ('Date', 'Wed, 31 May 2023 21:06:46 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:35:41.845 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/pklxkd-63/obj11 2023-05-31T21:35:41.846 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/pklxkd-63/obj11 2023-05-31T21:35:41.846 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2023-05-31T21:35:41.846 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /pklxkd-63/obj11 2023-05-31T21:35:41.846 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-05-31T21:35:41.846 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-05-31T21:35:41.847 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T21:35:41.847 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-05-31T21:35:41.847 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-05-31T21:35:41.847 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-05-31T21:35:41.848 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-05-31T21:35:41.848 INFO:tasks.rgw_multisite_tests:PUT 2023-05-31T21:35:41.848 INFO:tasks.rgw_multisite_tests:rL0Y20zC+Fzt72VPzMSk2A== 2023-05-31T21:35:41.848 INFO:tasks.rgw_multisite_tests:application/octet-stream 2023-05-31T21:35:41.849 INFO:tasks.rgw_multisite_tests:Wed, 31 May 2023 21:06:46 GMT 2023-05-31T21:35:41.849 INFO:tasks.rgw_multisite_tests:/pklxkd-63/obj11 2023-05-31T21:35:41.849 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-05-31T21:35:41.849 INFO:tasks.rgw_multisite_tests:AWS Q80H11OLNU1S48PPYL26:GfZs+Vp+2YOiMXysi+n702nC2lY= 2023-05-31T21:35:41.849 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Wed, 31 May 2023 21:06:46 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:GfZs+Vp+2YOiMXysi+n702nC2lY='} 2023-05-31T21:35:41.850 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000abd08c606220b2d2-006477b6e6-1630-a1'), ('Date', 'Wed, 31 May 2023 21:06:46 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:35:41.850 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/pklxkd-63/ 2023-05-31T21:35:41.850 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/pklxkd-63/ 2023-05-31T21:35:41.850 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-05-31T21:35:41.851 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /pklxkd-63/ 2023-05-31T21:35:41.851 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-05-31T21:35:41.851 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-05-31T21:35:41.851 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T21:35:41.851 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-05-31T21:35:41.852 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-05-31T21:35:41.852 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-05-31T21:35:41.852 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-05-31T21:35:41.852 INFO:tasks.rgw_multisite_tests:HEAD 2023-05-31T21:35:41.853 INFO:tasks.rgw_multisite_tests: 2023-05-31T21:35:41.853 INFO:tasks.rgw_multisite_tests: 2023-05-31T21:35:41.853 INFO:tasks.rgw_multisite_tests:Wed, 31 May 2023 21:06:46 GMT 2023-05-31T21:35:41.853 INFO:tasks.rgw_multisite_tests:/pklxkd-63/ 2023-05-31T21:35:41.854 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-05-31T21:35:41.854 INFO:tasks.rgw_multisite_tests:AWS Q80H11OLNU1S48PPYL26:bcNVcPZSg/FRj1VT8thTtJNmxUQ= 2023-05-31T21:35:41.854 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 21:06:46 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:bcNVcPZSg/FRj1VT8thTtJNmxUQ=', 'Content-Length': '0'} 2023-05-31T21:35:41.854 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '3'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000008301ae5fbabc288-006477b6e6-1630-a1'), ('Content-Length', '0'), ('Date', 'Wed, 31 May 2023 21:06:46 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:35:41.854 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/pklxkd-63/obj22 2023-05-31T21:35:41.855 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/pklxkd-63/obj22 2023-05-31T21:35:41.855 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2023-05-31T21:35:41.855 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /pklxkd-63/obj22 2023-05-31T21:35:41.855 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-05-31T21:35:41.856 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-05-31T21:35:41.856 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T21:35:41.856 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-05-31T21:35:41.856 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-05-31T21:35:41.857 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-05-31T21:35:41.857 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-05-31T21:35:41.857 INFO:tasks.rgw_multisite_tests:PUT 2023-05-31T21:35:41.857 INFO:tasks.rgw_multisite_tests:rL0Y20zC+Fzt72VPzMSk2A== 2023-05-31T21:35:41.858 INFO:tasks.rgw_multisite_tests:application/octet-stream 2023-05-31T21:35:41.858 INFO:tasks.rgw_multisite_tests:Wed, 31 May 2023 21:06:46 GMT 2023-05-31T21:35:41.858 INFO:tasks.rgw_multisite_tests:/pklxkd-63/obj22 2023-05-31T21:35:41.858 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-05-31T21:35:41.858 INFO:tasks.rgw_multisite_tests:AWS Q80H11OLNU1S48PPYL26:PbPbXWKg3RYRYp11jInc+oIyhZk= 2023-05-31T21:35:41.859 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Wed, 31 May 2023 21:06:46 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:PbPbXWKg3RYRYp11jInc+oIyhZk='} 2023-05-31T21:35:41.859 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000005b28259f64dce2b4-006477b6e6-1630-a1'), ('Date', 'Wed, 31 May 2023 21:06:46 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:35:41.859 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/pklxkd-63/ 2023-05-31T21:35:41.859 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/pklxkd-63/ 2023-05-31T21:35:41.860 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-05-31T21:35:41.860 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /pklxkd-63/ 2023-05-31T21:35:41.860 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-05-31T21:35:41.860 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-05-31T21:35:41.861 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T21:35:41.861 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-05-31T21:35:41.861 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-05-31T21:35:41.861 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-05-31T21:35:41.861 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-05-31T21:35:41.862 INFO:tasks.rgw_multisite_tests:HEAD 2023-05-31T21:35:41.862 INFO:tasks.rgw_multisite_tests: 2023-05-31T21:35:41.862 INFO:tasks.rgw_multisite_tests: 2023-05-31T21:35:41.862 INFO:tasks.rgw_multisite_tests:Wed, 31 May 2023 21:06:46 GMT 2023-05-31T21:35:41.863 INFO:tasks.rgw_multisite_tests:/pklxkd-63/ 2023-05-31T21:35:41.863 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-05-31T21:35:41.863 INFO:tasks.rgw_multisite_tests:AWS Q80H11OLNU1S48PPYL26:bcNVcPZSg/FRj1VT8thTtJNmxUQ= 2023-05-31T21:35:41.863 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 21:06:46 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:bcNVcPZSg/FRj1VT8thTtJNmxUQ=', 'Content-Length': '0'} 2023-05-31T21:35:41.864 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '2'), ('X-RGW-Bytes-Used', '6'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000f604d18b14a281aa-006477b6e6-1630-a1'), ('Content-Length', '0'), ('Date', 'Wed, 31 May 2023 21:06:46 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:35:41.864 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/pklxkd-63/obj33 2023-05-31T21:35:41.864 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/pklxkd-63/obj33 2023-05-31T21:35:41.864 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2023-05-31T21:35:41.864 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /pklxkd-63/obj33 2023-05-31T21:35:41.865 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-05-31T21:35:41.865 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-05-31T21:35:41.865 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T21:35:41.865 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-05-31T21:35:41.866 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-05-31T21:35:41.866 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-05-31T21:35:41.866 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-05-31T21:35:41.866 INFO:tasks.rgw_multisite_tests:PUT 2023-05-31T21:35:41.867 INFO:tasks.rgw_multisite_tests:rL0Y20zC+Fzt72VPzMSk2A== 2023-05-31T21:35:41.867 INFO:tasks.rgw_multisite_tests:application/octet-stream 2023-05-31T21:35:41.867 INFO:tasks.rgw_multisite_tests:Wed, 31 May 2023 21:06:46 GMT 2023-05-31T21:35:41.867 INFO:tasks.rgw_multisite_tests:/pklxkd-63/obj33 2023-05-31T21:35:41.867 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-05-31T21:35:41.868 INFO:tasks.rgw_multisite_tests:AWS Q80H11OLNU1S48PPYL26:JMRCIKv5NREN7sNHZgQC2if6BJU= 2023-05-31T21:35:41.868 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Wed, 31 May 2023 21:06:46 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:JMRCIKv5NREN7sNHZgQC2if6BJU='} 2023-05-31T21:35:41.868 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000a4eacbb1521d2a83-006477b6e6-1630-a1'), ('Date', 'Wed, 31 May 2023 21:06:46 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:35:41.868 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/pklxkd-63/ 2023-05-31T21:35:41.869 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/pklxkd-63/ 2023-05-31T21:35:41.869 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-05-31T21:35:41.869 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /pklxkd-63/ 2023-05-31T21:35:41.869 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-05-31T21:35:41.869 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-05-31T21:35:41.870 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T21:35:41.870 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-05-31T21:35:41.870 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-05-31T21:35:41.870 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-05-31T21:35:41.871 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-05-31T21:35:41.871 INFO:tasks.rgw_multisite_tests:HEAD 2023-05-31T21:35:41.871 INFO:tasks.rgw_multisite_tests: 2023-05-31T21:35:41.871 INFO:tasks.rgw_multisite_tests: 2023-05-31T21:35:41.872 INFO:tasks.rgw_multisite_tests:Wed, 31 May 2023 21:06:46 GMT 2023-05-31T21:35:41.872 INFO:tasks.rgw_multisite_tests:/pklxkd-63/ 2023-05-31T21:35:41.872 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-05-31T21:35:41.872 INFO:tasks.rgw_multisite_tests:AWS Q80H11OLNU1S48PPYL26:bcNVcPZSg/FRj1VT8thTtJNmxUQ= 2023-05-31T21:35:41.872 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 21:06:46 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:bcNVcPZSg/FRj1VT8thTtJNmxUQ=', 'Content-Length': '0'} 2023-05-31T21:35:41.873 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '3'), ('X-RGW-Bytes-Used', '9'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000004b75a6d684635180-006477b6e6-1630-a1'), ('Content-Length', '0'), ('Date', 'Wed, 31 May 2023 21:06:46 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:35:41.873 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/pklxkd-63/obj44 2023-05-31T21:35:41.873 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/pklxkd-63/obj44 2023-05-31T21:35:41.873 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2023-05-31T21:35:41.874 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /pklxkd-63/obj44 2023-05-31T21:35:41.874 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-05-31T21:35:41.874 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-05-31T21:35:41.874 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T21:35:41.875 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-05-31T21:35:41.875 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-05-31T21:35:41.875 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-05-31T21:35:41.875 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-05-31T21:35:41.875 INFO:tasks.rgw_multisite_tests:PUT 2023-05-31T21:35:41.876 INFO:tasks.rgw_multisite_tests:rL0Y20zC+Fzt72VPzMSk2A== 2023-05-31T21:35:41.876 INFO:tasks.rgw_multisite_tests:application/octet-stream 2023-05-31T21:35:41.876 INFO:tasks.rgw_multisite_tests:Wed, 31 May 2023 21:06:46 GMT 2023-05-31T21:35:41.876 INFO:tasks.rgw_multisite_tests:/pklxkd-63/obj44 2023-05-31T21:35:41.877 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-05-31T21:35:41.877 INFO:tasks.rgw_multisite_tests:AWS Q80H11OLNU1S48PPYL26:wJtayVKGup7Jg2BoezAs2Co6FN4= 2023-05-31T21:35:41.877 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Wed, 31 May 2023 21:06:46 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:wJtayVKGup7Jg2BoezAs2Co6FN4='} 2023-05-31T21:35:41.877 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000007298f47ce55460ec-006477b6e6-1630-a1'), ('Date', 'Wed, 31 May 2023 21:06:46 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:35:41.878 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/pklxkd-63/ 2023-05-31T21:35:41.878 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/pklxkd-63/ 2023-05-31T21:35:41.878 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-05-31T21:35:41.878 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /pklxkd-63/ 2023-05-31T21:35:41.878 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-05-31T21:35:41.879 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-05-31T21:35:41.879 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T21:35:41.879 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-05-31T21:35:41.879 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-05-31T21:35:41.880 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-05-31T21:35:41.880 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-05-31T21:35:41.880 INFO:tasks.rgw_multisite_tests:HEAD 2023-05-31T21:35:41.880 INFO:tasks.rgw_multisite_tests: 2023-05-31T21:35:41.881 INFO:tasks.rgw_multisite_tests: 2023-05-31T21:35:41.881 INFO:tasks.rgw_multisite_tests:Wed, 31 May 2023 21:06:46 GMT 2023-05-31T21:35:41.881 INFO:tasks.rgw_multisite_tests:/pklxkd-63/ 2023-05-31T21:35:41.881 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-05-31T21:35:41.881 INFO:tasks.rgw_multisite_tests:AWS Q80H11OLNU1S48PPYL26:bcNVcPZSg/FRj1VT8thTtJNmxUQ= 2023-05-31T21:35:41.882 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 21:06:46 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:bcNVcPZSg/FRj1VT8thTtJNmxUQ=', 'Content-Length': '0'} 2023-05-31T21:35:41.882 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '4'), ('X-RGW-Bytes-Used', '12'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000001133cf367c031c8c-006477b6e6-1630-a1'), ('Content-Length', '0'), ('Date', 'Wed, 31 May 2023 21:06:46 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:35:41.882 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/pklxkd-63/obj55 2023-05-31T21:35:41.883 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/pklxkd-63/obj55 2023-05-31T21:35:41.883 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2023-05-31T21:35:41.883 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /pklxkd-63/obj55 2023-05-31T21:35:41.883 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-05-31T21:35:41.883 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-05-31T21:35:41.884 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T21:35:41.884 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-05-31T21:35:41.884 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-05-31T21:35:41.884 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-05-31T21:35:41.885 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-05-31T21:35:41.885 INFO:tasks.rgw_multisite_tests:PUT 2023-05-31T21:35:41.885 INFO:tasks.rgw_multisite_tests:rL0Y20zC+Fzt72VPzMSk2A== 2023-05-31T21:35:41.885 INFO:tasks.rgw_multisite_tests:application/octet-stream 2023-05-31T21:35:41.886 INFO:tasks.rgw_multisite_tests:Wed, 31 May 2023 21:06:46 GMT 2023-05-31T21:35:41.886 INFO:tasks.rgw_multisite_tests:/pklxkd-63/obj55 2023-05-31T21:35:41.886 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-05-31T21:35:41.886 INFO:tasks.rgw_multisite_tests:AWS Q80H11OLNU1S48PPYL26:XE45D3xSYM0zhAldN+S/Ukc6sLA= 2023-05-31T21:35:41.887 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Wed, 31 May 2023 21:06:46 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:XE45D3xSYM0zhAldN+S/Ukc6sLA='} 2023-05-31T21:35:41.887 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000003d86c94ac1f85316-006477b6e6-1630-a1'), ('Date', 'Wed, 31 May 2023 21:06:46 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:35:41.887 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['bucket', 'reshard', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'pklxkd-63', '--num-shards', '17', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-05-31T21:35:41.887 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'reshard', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'pklxkd-63', '--num-shards', '17', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-05-31T21:35:41.887 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket reshard --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --bucket pklxkd-63 --num-shards 17 --yes-i-really-mean-it --cluster c1 --debug-rgw 0 --debug-ms 0 2023-05-31T21:35:41.888 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: ignoring --setuser ceph since I am not root 2023-05-31T21:35:41.888 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-05-31T21:35:41.888 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: tenant: 2023-05-31T21:35:41.888 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: bucket name: pklxkd-63 2023-05-31T21:35:41.889 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: total entries: 5 2023-05-31T21:35:41.889 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/pklxkd-63/ 2023-05-31T21:35:41.889 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/pklxkd-63/ 2023-05-31T21:35:41.889 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-05-31T21:35:41.890 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /pklxkd-63/ 2023-05-31T21:35:41.890 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-05-31T21:35:41.890 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-05-31T21:35:41.890 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T21:35:41.890 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-05-31T21:35:41.891 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-05-31T21:35:41.891 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-05-31T21:35:41.891 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-05-31T21:35:41.891 INFO:tasks.rgw_multisite_tests:HEAD 2023-05-31T21:35:41.892 INFO:tasks.rgw_multisite_tests: 2023-05-31T21:35:41.892 INFO:tasks.rgw_multisite_tests: 2023-05-31T21:35:41.892 INFO:tasks.rgw_multisite_tests:Wed, 31 May 2023 21:06:46 GMT 2023-05-31T21:35:41.892 INFO:tasks.rgw_multisite_tests:/pklxkd-63/ 2023-05-31T21:35:41.893 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-05-31T21:35:41.893 INFO:tasks.rgw_multisite_tests:AWS Q80H11OLNU1S48PPYL26:bcNVcPZSg/FRj1VT8thTtJNmxUQ= 2023-05-31T21:35:41.893 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 21:06:46 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:bcNVcPZSg/FRj1VT8thTtJNmxUQ=', 'Content-Length': '0'} 2023-05-31T21:35:41.893 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '5'), ('X-RGW-Bytes-Used', '15'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000192ab699a1c9f1fb-006477b6e6-1630-a1'), ('Content-Length', '0'), ('Date', 'Wed, 31 May 2023 21:06:46 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:35:41.894 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/pklxkd-63/obj17 2023-05-31T21:35:41.894 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/pklxkd-63/obj17 2023-05-31T21:35:41.894 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2023-05-31T21:35:41.894 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /pklxkd-63/obj17 2023-05-31T21:35:41.894 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-05-31T21:35:41.895 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-05-31T21:35:41.895 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T21:35:41.895 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-05-31T21:35:41.895 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-05-31T21:35:41.896 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-05-31T21:35:41.896 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-05-31T21:35:41.896 INFO:tasks.rgw_multisite_tests:PUT 2023-05-31T21:35:41.896 INFO:tasks.rgw_multisite_tests:rL0Y20zC+Fzt72VPzMSk2A== 2023-05-31T21:35:41.897 INFO:tasks.rgw_multisite_tests:application/octet-stream 2023-05-31T21:35:41.897 INFO:tasks.rgw_multisite_tests:Wed, 31 May 2023 21:06:46 GMT 2023-05-31T21:35:41.897 INFO:tasks.rgw_multisite_tests:/pklxkd-63/obj17 2023-05-31T21:35:41.897 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-05-31T21:35:41.897 INFO:tasks.rgw_multisite_tests:AWS Q80H11OLNU1S48PPYL26:OrZ4mwcvMAeL5BQ2sQyjkRnOrAg= 2023-05-31T21:35:41.898 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Wed, 31 May 2023 21:06:46 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:OrZ4mwcvMAeL5BQ2sQyjkRnOrAg='} 2023-05-31T21:35:41.898 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000665ec0c114082c09-006477b6e6-1630-a1'), ('Date', 'Wed, 31 May 2023 21:06:46 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:35:41.898 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/pklxkd-63/ 2023-05-31T21:35:41.898 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/pklxkd-63/ 2023-05-31T21:35:41.899 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-05-31T21:35:41.899 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /pklxkd-63/ 2023-05-31T21:35:41.899 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-05-31T21:35:41.899 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-05-31T21:35:41.899 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T21:35:41.900 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-05-31T21:35:41.900 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-05-31T21:35:41.900 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-05-31T21:35:41.900 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-05-31T21:35:41.901 INFO:tasks.rgw_multisite_tests:HEAD 2023-05-31T21:35:41.901 INFO:tasks.rgw_multisite_tests: 2023-05-31T21:35:41.901 INFO:tasks.rgw_multisite_tests: 2023-05-31T21:35:41.901 INFO:tasks.rgw_multisite_tests:Wed, 31 May 2023 21:06:46 GMT 2023-05-31T21:35:41.902 INFO:tasks.rgw_multisite_tests:/pklxkd-63/ 2023-05-31T21:35:41.902 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-05-31T21:35:41.902 INFO:tasks.rgw_multisite_tests:AWS Q80H11OLNU1S48PPYL26:bcNVcPZSg/FRj1VT8thTtJNmxUQ= 2023-05-31T21:35:41.902 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 21:06:46 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:bcNVcPZSg/FRj1VT8thTtJNmxUQ=', 'Content-Length': '0'} 2023-05-31T21:35:41.902 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '6'), ('X-RGW-Bytes-Used', '18'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000008b4a5b6bd048c9fd-006477b6e6-1630-a1'), ('Content-Length', '0'), ('Date', 'Wed, 31 May 2023 21:06:46 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:35:41.903 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/pklxkd-63/obj34 2023-05-31T21:35:41.903 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/pklxkd-63/obj34 2023-05-31T21:35:41.903 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2023-05-31T21:35:41.903 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /pklxkd-63/obj34 2023-05-31T21:35:41.904 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-05-31T21:35:41.904 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-05-31T21:35:41.904 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T21:35:41.904 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-05-31T21:35:41.905 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-05-31T21:35:41.905 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-05-31T21:35:41.905 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-05-31T21:35:41.905 INFO:tasks.rgw_multisite_tests:PUT 2023-05-31T21:35:41.905 INFO:tasks.rgw_multisite_tests:rL0Y20zC+Fzt72VPzMSk2A== 2023-05-31T21:35:41.906 INFO:tasks.rgw_multisite_tests:application/octet-stream 2023-05-31T21:35:41.906 INFO:tasks.rgw_multisite_tests:Wed, 31 May 2023 21:06:46 GMT 2023-05-31T21:35:41.906 INFO:tasks.rgw_multisite_tests:/pklxkd-63/obj34 2023-05-31T21:35:41.906 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-05-31T21:35:41.907 INFO:tasks.rgw_multisite_tests:AWS Q80H11OLNU1S48PPYL26:lGZVgG1T2eZnI5etXzC2nXcRIDg= 2023-05-31T21:35:41.907 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Wed, 31 May 2023 21:06:46 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:lGZVgG1T2eZnI5etXzC2nXcRIDg='} 2023-05-31T21:35:41.907 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000006a0f79649399bce-006477b6e6-1630-a1'), ('Date', 'Wed, 31 May 2023 21:06:46 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:35:41.907 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/pklxkd-63/ 2023-05-31T21:35:41.907 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/pklxkd-63/ 2023-05-31T21:35:41.908 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-05-31T21:35:41.908 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /pklxkd-63/ 2023-05-31T21:35:41.908 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-05-31T21:35:41.908 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-05-31T21:35:41.909 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T21:35:41.909 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-05-31T21:35:41.909 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-05-31T21:35:41.909 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-05-31T21:35:41.910 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-05-31T21:35:41.910 INFO:tasks.rgw_multisite_tests:HEAD 2023-05-31T21:35:41.910 INFO:tasks.rgw_multisite_tests: 2023-05-31T21:35:41.910 INFO:tasks.rgw_multisite_tests: 2023-05-31T21:35:41.910 INFO:tasks.rgw_multisite_tests:Wed, 31 May 2023 21:06:46 GMT 2023-05-31T21:35:41.911 INFO:tasks.rgw_multisite_tests:/pklxkd-63/ 2023-05-31T21:35:41.911 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-05-31T21:35:41.911 INFO:tasks.rgw_multisite_tests:AWS Q80H11OLNU1S48PPYL26:bcNVcPZSg/FRj1VT8thTtJNmxUQ= 2023-05-31T21:35:41.911 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 21:06:46 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:bcNVcPZSg/FRj1VT8thTtJNmxUQ=', 'Content-Length': '0'} 2023-05-31T21:35:41.912 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '7'), ('X-RGW-Bytes-Used', '21'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000669d4202a23e0db8-006477b6e6-1630-a1'), ('Content-Length', '0'), ('Date', 'Wed, 31 May 2023 21:06:46 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:35:41.912 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/pklxkd-63/obj51 2023-05-31T21:35:41.912 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/pklxkd-63/obj51 2023-05-31T21:35:41.912 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2023-05-31T21:35:41.913 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /pklxkd-63/obj51 2023-05-31T21:35:41.913 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-05-31T21:35:41.913 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-05-31T21:35:41.913 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T21:35:41.913 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-05-31T21:35:41.914 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-05-31T21:35:41.914 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-05-31T21:35:41.914 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-05-31T21:35:41.914 INFO:tasks.rgw_multisite_tests:PUT 2023-05-31T21:35:41.915 INFO:tasks.rgw_multisite_tests:rL0Y20zC+Fzt72VPzMSk2A== 2023-05-31T21:35:41.915 INFO:tasks.rgw_multisite_tests:application/octet-stream 2023-05-31T21:35:41.915 INFO:tasks.rgw_multisite_tests:Wed, 31 May 2023 21:06:46 GMT 2023-05-31T21:35:41.915 INFO:tasks.rgw_multisite_tests:/pklxkd-63/obj51 2023-05-31T21:35:41.916 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-05-31T21:35:41.916 INFO:tasks.rgw_multisite_tests:AWS Q80H11OLNU1S48PPYL26:UFPzYWgZXJQ/N83OvcYNYPxz0WM= 2023-05-31T21:35:41.916 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Wed, 31 May 2023 21:06:46 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:UFPzYWgZXJQ/N83OvcYNYPxz0WM='} 2023-05-31T21:35:41.916 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000008900e0650d16e444-006477b6e6-1630-a1'), ('Date', 'Wed, 31 May 2023 21:06:46 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:35:41.916 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/pklxkd-63/ 2023-05-31T21:35:41.917 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/pklxkd-63/ 2023-05-31T21:35:41.917 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-05-31T21:35:41.917 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /pklxkd-63/ 2023-05-31T21:35:41.917 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-05-31T21:35:41.918 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-05-31T21:35:41.918 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T21:35:41.918 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-05-31T21:35:41.918 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-05-31T21:35:41.919 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-05-31T21:35:41.919 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-05-31T21:35:41.919 INFO:tasks.rgw_multisite_tests:HEAD 2023-05-31T21:35:41.919 INFO:tasks.rgw_multisite_tests: 2023-05-31T21:35:41.919 INFO:tasks.rgw_multisite_tests: 2023-05-31T21:35:41.920 INFO:tasks.rgw_multisite_tests:Wed, 31 May 2023 21:06:46 GMT 2023-05-31T21:35:41.920 INFO:tasks.rgw_multisite_tests:/pklxkd-63/ 2023-05-31T21:35:41.920 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-05-31T21:35:41.920 INFO:tasks.rgw_multisite_tests:AWS Q80H11OLNU1S48PPYL26:bcNVcPZSg/FRj1VT8thTtJNmxUQ= 2023-05-31T21:35:41.921 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 21:06:46 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:bcNVcPZSg/FRj1VT8thTtJNmxUQ=', 'Content-Length': '0'} 2023-05-31T21:35:41.921 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '8'), ('X-RGW-Bytes-Used', '24'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000077f9ac6df169e51a-006477b6e6-1630-a1'), ('Content-Length', '0'), ('Date', 'Wed, 31 May 2023 21:06:46 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:35:41.921 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/pklxkd-63/obj68 2023-05-31T21:35:41.921 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/pklxkd-63/obj68 2023-05-31T21:35:41.922 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2023-05-31T21:35:41.922 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /pklxkd-63/obj68 2023-05-31T21:35:41.922 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-05-31T21:35:41.922 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-05-31T21:35:41.922 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T21:35:41.923 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-05-31T21:35:41.923 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-05-31T21:35:41.923 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-05-31T21:35:41.923 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-05-31T21:35:41.924 INFO:tasks.rgw_multisite_tests:PUT 2023-05-31T21:35:41.924 INFO:tasks.rgw_multisite_tests:rL0Y20zC+Fzt72VPzMSk2A== 2023-05-31T21:35:41.924 INFO:tasks.rgw_multisite_tests:application/octet-stream 2023-05-31T21:35:41.924 INFO:tasks.rgw_multisite_tests:Wed, 31 May 2023 21:06:46 GMT 2023-05-31T21:35:41.925 INFO:tasks.rgw_multisite_tests:/pklxkd-63/obj68 2023-05-31T21:35:41.925 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-05-31T21:35:41.925 INFO:tasks.rgw_multisite_tests:AWS Q80H11OLNU1S48PPYL26:oSGOKxHThne7/WPrFjF94PPQApM= 2023-05-31T21:35:41.925 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Wed, 31 May 2023 21:06:46 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:oSGOKxHThne7/WPrFjF94PPQApM='} 2023-05-31T21:35:41.925 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000009d5571bab3e7fdfb-006477b6e6-1630-a1'), ('Date', 'Wed, 31 May 2023 21:06:46 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:35:41.926 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/pklxkd-63/ 2023-05-31T21:35:41.926 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/pklxkd-63/ 2023-05-31T21:35:41.926 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-05-31T21:35:41.926 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /pklxkd-63/ 2023-05-31T21:35:41.927 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-05-31T21:35:41.927 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-05-31T21:35:41.927 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T21:35:41.927 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-05-31T21:35:41.927 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-05-31T21:35:41.928 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-05-31T21:35:41.928 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-05-31T21:35:41.928 INFO:tasks.rgw_multisite_tests:HEAD 2023-05-31T21:35:41.928 INFO:tasks.rgw_multisite_tests: 2023-05-31T21:35:41.929 INFO:tasks.rgw_multisite_tests: 2023-05-31T21:35:41.929 INFO:tasks.rgw_multisite_tests:Wed, 31 May 2023 21:06:46 GMT 2023-05-31T21:35:41.929 INFO:tasks.rgw_multisite_tests:/pklxkd-63/ 2023-05-31T21:35:41.929 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-05-31T21:35:41.930 INFO:tasks.rgw_multisite_tests:AWS Q80H11OLNU1S48PPYL26:bcNVcPZSg/FRj1VT8thTtJNmxUQ= 2023-05-31T21:35:41.930 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 21:06:46 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:bcNVcPZSg/FRj1VT8thTtJNmxUQ=', 'Content-Length': '0'} 2023-05-31T21:35:41.930 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '9'), ('X-RGW-Bytes-Used', '27'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000ee935ba45e8171e7-006477b6e6-1630-a1'), ('Content-Length', '0'), ('Date', 'Wed, 31 May 2023 21:06:46 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:35:41.930 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/pklxkd-63/obj85 2023-05-31T21:35:41.930 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/pklxkd-63/obj85 2023-05-31T21:35:41.931 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2023-05-31T21:35:41.931 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /pklxkd-63/obj85 2023-05-31T21:35:41.931 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-05-31T21:35:41.931 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-05-31T21:35:41.932 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T21:35:41.932 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-05-31T21:35:41.932 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-05-31T21:35:41.932 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-05-31T21:35:41.933 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-05-31T21:35:41.933 INFO:tasks.rgw_multisite_tests:PUT 2023-05-31T21:35:41.933 INFO:tasks.rgw_multisite_tests:rL0Y20zC+Fzt72VPzMSk2A== 2023-05-31T21:35:41.933 INFO:tasks.rgw_multisite_tests:application/octet-stream 2023-05-31T21:35:41.933 INFO:tasks.rgw_multisite_tests:Wed, 31 May 2023 21:06:46 GMT 2023-05-31T21:35:41.934 INFO:tasks.rgw_multisite_tests:/pklxkd-63/obj85 2023-05-31T21:35:41.934 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-05-31T21:35:41.934 INFO:tasks.rgw_multisite_tests:AWS Q80H11OLNU1S48PPYL26:hSks8AIl9wGlA5rXTY6VJD/Yx9Q= 2023-05-31T21:35:41.934 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Wed, 31 May 2023 21:06:46 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:hSks8AIl9wGlA5rXTY6VJD/Yx9Q='} 2023-05-31T21:35:41.935 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000a403532f1c89ce02-006477b6e6-1630-a1'), ('Date', 'Wed, 31 May 2023 21:06:46 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:35:41.935 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['bucket', 'reshard', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'pklxkd-63', '--num-shards', '19', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-05-31T21:35:41.935 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'reshard', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'pklxkd-63', '--num-shards', '19', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-05-31T21:35:41.935 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket reshard --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --bucket pklxkd-63 --num-shards 19 --yes-i-really-mean-it --cluster c1 --debug-rgw 0 --debug-ms 0 2023-05-31T21:35:41.936 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: ignoring --setuser ceph since I am not root 2023-05-31T21:35:41.936 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-05-31T21:35:41.936 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: tenant: 2023-05-31T21:35:41.936 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: bucket name: pklxkd-63 2023-05-31T21:35:41.936 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: total entries: 10 2023-05-31T21:35:41.937 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/pklxkd-63/ 2023-05-31T21:35:41.937 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/pklxkd-63/ 2023-05-31T21:35:41.937 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-05-31T21:35:41.937 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /pklxkd-63/ 2023-05-31T21:35:41.938 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-05-31T21:35:41.938 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-05-31T21:35:41.938 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T21:35:41.938 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-05-31T21:35:41.939 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-05-31T21:35:41.939 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-05-31T21:35:41.939 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-05-31T21:35:41.939 INFO:tasks.rgw_multisite_tests:HEAD 2023-05-31T21:35:41.939 INFO:tasks.rgw_multisite_tests: 2023-05-31T21:35:41.940 INFO:tasks.rgw_multisite_tests: 2023-05-31T21:35:41.940 INFO:tasks.rgw_multisite_tests:Wed, 31 May 2023 21:06:46 GMT 2023-05-31T21:35:41.940 INFO:tasks.rgw_multisite_tests:/pklxkd-63/ 2023-05-31T21:35:41.940 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-05-31T21:35:41.941 INFO:tasks.rgw_multisite_tests:AWS Q80H11OLNU1S48PPYL26:bcNVcPZSg/FRj1VT8thTtJNmxUQ= 2023-05-31T21:35:41.941 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 21:06:46 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:bcNVcPZSg/FRj1VT8thTtJNmxUQ=', 'Content-Length': '0'} 2023-05-31T21:35:41.941 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '10'), ('X-RGW-Bytes-Used', '30'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000000bb7546a3333c47-006477b6e6-1630-a1'), ('Content-Length', '0'), ('Date', 'Wed, 31 May 2023 21:06:46 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:35:41.941 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/pklxkd-63/obj19 2023-05-31T21:35:41.942 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/pklxkd-63/obj19 2023-05-31T21:35:41.942 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2023-05-31T21:35:41.942 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /pklxkd-63/obj19 2023-05-31T21:35:41.942 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-05-31T21:35:41.943 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-05-31T21:35:41.943 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T21:35:41.943 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-05-31T21:35:41.943 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-05-31T21:35:41.943 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-05-31T21:35:41.944 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-05-31T21:35:41.944 INFO:tasks.rgw_multisite_tests:PUT 2023-05-31T21:35:41.944 INFO:tasks.rgw_multisite_tests:rL0Y20zC+Fzt72VPzMSk2A== 2023-05-31T21:35:41.944 INFO:tasks.rgw_multisite_tests:application/octet-stream 2023-05-31T21:35:41.945 INFO:tasks.rgw_multisite_tests:Wed, 31 May 2023 21:06:46 GMT 2023-05-31T21:35:41.945 INFO:tasks.rgw_multisite_tests:/pklxkd-63/obj19 2023-05-31T21:35:41.945 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-05-31T21:35:41.945 INFO:tasks.rgw_multisite_tests:AWS Q80H11OLNU1S48PPYL26:15k03jeZBvtfCMWs3WizVJ6VkpE= 2023-05-31T21:35:41.946 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Wed, 31 May 2023 21:06:46 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:15k03jeZBvtfCMWs3WizVJ6VkpE='} 2023-05-31T21:35:41.946 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000020343e4c0afd143-006477b6e6-1630-a1'), ('Date', 'Wed, 31 May 2023 21:06:46 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:35:41.946 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/pklxkd-63/ 2023-05-31T21:35:41.946 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/pklxkd-63/ 2023-05-31T21:35:41.947 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-05-31T21:35:41.947 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /pklxkd-63/ 2023-05-31T21:35:41.947 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-05-31T21:35:41.947 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-05-31T21:35:41.947 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T21:35:41.948 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-05-31T21:35:41.948 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-05-31T21:35:41.948 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-05-31T21:35:41.948 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-05-31T21:35:41.949 INFO:tasks.rgw_multisite_tests:HEAD 2023-05-31T21:35:41.949 INFO:tasks.rgw_multisite_tests: 2023-05-31T21:35:41.949 INFO:tasks.rgw_multisite_tests: 2023-05-31T21:35:41.949 INFO:tasks.rgw_multisite_tests:Wed, 31 May 2023 21:06:46 GMT 2023-05-31T21:35:41.950 INFO:tasks.rgw_multisite_tests:/pklxkd-63/ 2023-05-31T21:35:41.950 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-05-31T21:35:41.950 INFO:tasks.rgw_multisite_tests:AWS Q80H11OLNU1S48PPYL26:bcNVcPZSg/FRj1VT8thTtJNmxUQ= 2023-05-31T21:35:41.950 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 21:06:46 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:bcNVcPZSg/FRj1VT8thTtJNmxUQ=', 'Content-Length': '0'} 2023-05-31T21:35:41.950 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '11'), ('X-RGW-Bytes-Used', '33'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000082d23a8c64bce232-006477b6e6-1630-a1'), ('Content-Length', '0'), ('Date', 'Wed, 31 May 2023 21:06:46 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:35:41.951 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/pklxkd-63/obj38 2023-05-31T21:35:41.951 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/pklxkd-63/obj38 2023-05-31T21:35:41.951 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2023-05-31T21:35:41.951 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /pklxkd-63/obj38 2023-05-31T21:35:41.952 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-05-31T21:35:41.952 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-05-31T21:35:41.952 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T21:35:41.952 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-05-31T21:35:41.953 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-05-31T21:35:41.953 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-05-31T21:35:41.953 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-05-31T21:35:41.953 INFO:tasks.rgw_multisite_tests:PUT 2023-05-31T21:35:41.953 INFO:tasks.rgw_multisite_tests:rL0Y20zC+Fzt72VPzMSk2A== 2023-05-31T21:35:41.954 INFO:tasks.rgw_multisite_tests:application/octet-stream 2023-05-31T21:35:41.954 INFO:tasks.rgw_multisite_tests:Wed, 31 May 2023 21:06:46 GMT 2023-05-31T21:35:41.954 INFO:tasks.rgw_multisite_tests:/pklxkd-63/obj38 2023-05-31T21:35:41.954 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-05-31T21:35:41.955 INFO:tasks.rgw_multisite_tests:AWS Q80H11OLNU1S48PPYL26:4YeXx5JAAvSZJKDMoLkbmUUEDdg= 2023-05-31T21:35:41.955 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Wed, 31 May 2023 21:06:46 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:4YeXx5JAAvSZJKDMoLkbmUUEDdg='} 2023-05-31T21:35:41.955 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000007e5055c29f1d5d2d-006477b6e6-1630-a1'), ('Date', 'Wed, 31 May 2023 21:06:46 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:35:41.955 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/pklxkd-63/ 2023-05-31T21:35:41.956 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/pklxkd-63/ 2023-05-31T21:35:41.956 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-05-31T21:35:41.956 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /pklxkd-63/ 2023-05-31T21:35:41.956 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-05-31T21:35:41.956 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-05-31T21:35:41.957 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T21:35:41.957 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-05-31T21:35:41.957 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-05-31T21:35:41.957 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-05-31T21:35:41.958 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-05-31T21:35:41.958 INFO:tasks.rgw_multisite_tests:HEAD 2023-05-31T21:35:41.958 INFO:tasks.rgw_multisite_tests: 2023-05-31T21:35:41.958 INFO:tasks.rgw_multisite_tests: 2023-05-31T21:35:41.958 INFO:tasks.rgw_multisite_tests:Wed, 31 May 2023 21:06:46 GMT 2023-05-31T21:35:41.959 INFO:tasks.rgw_multisite_tests:/pklxkd-63/ 2023-05-31T21:35:41.959 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-05-31T21:35:41.959 INFO:tasks.rgw_multisite_tests:AWS Q80H11OLNU1S48PPYL26:bcNVcPZSg/FRj1VT8thTtJNmxUQ= 2023-05-31T21:35:41.959 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 21:06:46 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:bcNVcPZSg/FRj1VT8thTtJNmxUQ=', 'Content-Length': '0'} 2023-05-31T21:35:41.960 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '12'), ('X-RGW-Bytes-Used', '36'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000a6cff33ca196fd48-006477b6e6-1630-a1'), ('Content-Length', '0'), ('Date', 'Wed, 31 May 2023 21:06:46 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:35:41.960 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/pklxkd-63/obj57 2023-05-31T21:35:41.960 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/pklxkd-63/obj57 2023-05-31T21:35:41.960 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2023-05-31T21:35:41.961 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /pklxkd-63/obj57 2023-05-31T21:35:41.961 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-05-31T21:35:41.961 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-05-31T21:35:41.961 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T21:35:41.961 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-05-31T21:35:41.962 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-05-31T21:35:41.962 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-05-31T21:35:41.962 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-05-31T21:35:41.962 INFO:tasks.rgw_multisite_tests:PUT 2023-05-31T21:35:41.963 INFO:tasks.rgw_multisite_tests:rL0Y20zC+Fzt72VPzMSk2A== 2023-05-31T21:35:41.963 INFO:tasks.rgw_multisite_tests:application/octet-stream 2023-05-31T21:35:41.963 INFO:tasks.rgw_multisite_tests:Wed, 31 May 2023 21:06:46 GMT 2023-05-31T21:35:41.963 INFO:tasks.rgw_multisite_tests:/pklxkd-63/obj57 2023-05-31T21:35:41.964 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-05-31T21:35:41.964 INFO:tasks.rgw_multisite_tests:AWS Q80H11OLNU1S48PPYL26:ZM/wRTD2UVYv4UBQ9XZipWrNwYs= 2023-05-31T21:35:41.964 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Wed, 31 May 2023 21:06:46 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:ZM/wRTD2UVYv4UBQ9XZipWrNwYs='} 2023-05-31T21:35:41.964 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000007fd1d16cd0d31bdc-006477b6e6-1630-a1'), ('Date', 'Wed, 31 May 2023 21:06:46 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:35:41.964 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/pklxkd-63/ 2023-05-31T21:35:41.965 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/pklxkd-63/ 2023-05-31T21:35:41.965 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-05-31T21:35:41.965 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /pklxkd-63/ 2023-05-31T21:35:41.965 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-05-31T21:35:41.966 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-05-31T21:35:41.966 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T21:35:41.966 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-05-31T21:35:41.966 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-05-31T21:35:41.966 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-05-31T21:35:41.967 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-05-31T21:35:41.967 INFO:tasks.rgw_multisite_tests:HEAD 2023-05-31T21:35:41.967 INFO:tasks.rgw_multisite_tests: 2023-05-31T21:35:41.967 INFO:tasks.rgw_multisite_tests: 2023-05-31T21:35:41.968 INFO:tasks.rgw_multisite_tests:Wed, 31 May 2023 21:06:46 GMT 2023-05-31T21:35:41.968 INFO:tasks.rgw_multisite_tests:/pklxkd-63/ 2023-05-31T21:35:41.968 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-05-31T21:35:41.968 INFO:tasks.rgw_multisite_tests:AWS Q80H11OLNU1S48PPYL26:bcNVcPZSg/FRj1VT8thTtJNmxUQ= 2023-05-31T21:35:41.969 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 21:06:46 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:bcNVcPZSg/FRj1VT8thTtJNmxUQ=', 'Content-Length': '0'} 2023-05-31T21:35:41.969 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '13'), ('X-RGW-Bytes-Used', '39'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000cfd3303aa4bd4612-006477b6e6-1630-a1'), ('Content-Length', '0'), ('Date', 'Wed, 31 May 2023 21:06:46 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:35:41.969 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/pklxkd-63/obj76 2023-05-31T21:35:41.969 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/pklxkd-63/obj76 2023-05-31T21:35:41.969 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2023-05-31T21:35:41.970 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /pklxkd-63/obj76 2023-05-31T21:35:41.970 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-05-31T21:35:41.970 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-05-31T21:35:41.970 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T21:35:41.971 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-05-31T21:35:41.971 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-05-31T21:35:41.971 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-05-31T21:35:41.971 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-05-31T21:35:41.972 INFO:tasks.rgw_multisite_tests:PUT 2023-05-31T21:35:41.972 INFO:tasks.rgw_multisite_tests:rL0Y20zC+Fzt72VPzMSk2A== 2023-05-31T21:35:41.972 INFO:tasks.rgw_multisite_tests:application/octet-stream 2023-05-31T21:35:41.972 INFO:tasks.rgw_multisite_tests:Wed, 31 May 2023 21:06:46 GMT 2023-05-31T21:35:41.972 INFO:tasks.rgw_multisite_tests:/pklxkd-63/obj76 2023-05-31T21:35:41.973 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-05-31T21:35:41.973 INFO:tasks.rgw_multisite_tests:AWS Q80H11OLNU1S48PPYL26:4mtLbF5aErMMDq3GaxJKBI8NuSs= 2023-05-31T21:35:41.973 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Wed, 31 May 2023 21:06:46 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:4mtLbF5aErMMDq3GaxJKBI8NuSs='} 2023-05-31T21:35:41.973 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000054b1b5006afef11d-006477b6e6-1630-a1'), ('Date', 'Wed, 31 May 2023 21:06:46 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:35:41.974 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/pklxkd-63/ 2023-05-31T21:35:41.974 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/pklxkd-63/ 2023-05-31T21:35:41.974 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-05-31T21:35:41.974 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /pklxkd-63/ 2023-05-31T21:35:41.974 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-05-31T21:35:41.975 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-05-31T21:35:41.975 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T21:35:41.975 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-05-31T21:35:41.975 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-05-31T21:35:41.976 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-05-31T21:35:41.976 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-05-31T21:35:41.976 INFO:tasks.rgw_multisite_tests:HEAD 2023-05-31T21:35:41.976 INFO:tasks.rgw_multisite_tests: 2023-05-31T21:35:41.977 INFO:tasks.rgw_multisite_tests: 2023-05-31T21:35:41.977 INFO:tasks.rgw_multisite_tests:Wed, 31 May 2023 21:06:46 GMT 2023-05-31T21:35:41.977 INFO:tasks.rgw_multisite_tests:/pklxkd-63/ 2023-05-31T21:35:41.977 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-05-31T21:35:41.977 INFO:tasks.rgw_multisite_tests:AWS Q80H11OLNU1S48PPYL26:bcNVcPZSg/FRj1VT8thTtJNmxUQ= 2023-05-31T21:35:41.978 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 21:06:46 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:bcNVcPZSg/FRj1VT8thTtJNmxUQ=', 'Content-Length': '0'} 2023-05-31T21:35:41.978 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '14'), ('X-RGW-Bytes-Used', '42'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000037caabbad6771194-006477b6e6-1630-a1'), ('Content-Length', '0'), ('Date', 'Wed, 31 May 2023 21:06:46 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:35:41.978 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/pklxkd-63/obj95 2023-05-31T21:35:41.978 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/pklxkd-63/obj95 2023-05-31T21:35:41.979 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2023-05-31T21:35:41.979 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /pklxkd-63/obj95 2023-05-31T21:35:41.979 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-05-31T21:35:41.979 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-05-31T21:35:41.980 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T21:35:41.980 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-05-31T21:35:41.980 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-05-31T21:35:41.980 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-05-31T21:35:41.980 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-05-31T21:35:41.981 INFO:tasks.rgw_multisite_tests:PUT 2023-05-31T21:35:41.981 INFO:tasks.rgw_multisite_tests:rL0Y20zC+Fzt72VPzMSk2A== 2023-05-31T21:35:41.981 INFO:tasks.rgw_multisite_tests:application/octet-stream 2023-05-31T21:35:41.981 INFO:tasks.rgw_multisite_tests:Wed, 31 May 2023 21:06:46 GMT 2023-05-31T21:35:41.982 INFO:tasks.rgw_multisite_tests:/pklxkd-63/obj95 2023-05-31T21:35:41.982 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-05-31T21:35:41.982 INFO:tasks.rgw_multisite_tests:AWS Q80H11OLNU1S48PPYL26:K7yxuDgRGtzb7pIl5Z9my/8Hjvk= 2023-05-31T21:35:41.982 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Wed, 31 May 2023 21:06:46 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:K7yxuDgRGtzb7pIl5Z9my/8Hjvk='} 2023-05-31T21:35:41.983 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000047db55aa823e5316-006477b6e6-1630-a1'), ('Date', 'Wed, 31 May 2023 21:06:46 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:35:41.983 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['bucket', 'reshard', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'pklxkd-63', '--num-shards', '23', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-05-31T21:35:41.983 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'reshard', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'pklxkd-63', '--num-shards', '23', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-05-31T21:35:41.983 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket reshard --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --bucket pklxkd-63 --num-shards 23 --yes-i-really-mean-it --cluster c1 --debug-rgw 0 --debug-ms 0 2023-05-31T21:35:41.984 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: ignoring --setuser ceph since I am not root 2023-05-31T21:35:41.984 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-05-31T21:35:41.984 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: tenant: 2023-05-31T21:35:41.984 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: bucket name: pklxkd-63 2023-05-31T21:35:41.985 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: total entries: 15 2023-05-31T21:35:41.985 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/pklxkd-63/ 2023-05-31T21:35:41.985 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/pklxkd-63/ 2023-05-31T21:35:41.985 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-05-31T21:35:41.986 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /pklxkd-63/ 2023-05-31T21:35:41.986 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-05-31T21:35:41.986 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-05-31T21:35:41.986 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T21:35:41.986 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-05-31T21:35:41.987 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-05-31T21:35:41.987 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-05-31T21:35:41.987 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-05-31T21:35:41.987 INFO:tasks.rgw_multisite_tests:HEAD 2023-05-31T21:35:41.988 INFO:tasks.rgw_multisite_tests: 2023-05-31T21:35:41.988 INFO:tasks.rgw_multisite_tests: 2023-05-31T21:35:41.988 INFO:tasks.rgw_multisite_tests:Wed, 31 May 2023 21:06:47 GMT 2023-05-31T21:35:41.988 INFO:tasks.rgw_multisite_tests:/pklxkd-63/ 2023-05-31T21:35:41.989 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-05-31T21:35:41.989 INFO:tasks.rgw_multisite_tests:AWS Q80H11OLNU1S48PPYL26:PDWLJ6ztPB6x60LVcukbAfyzcxQ= 2023-05-31T21:35:41.989 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 21:06:47 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:PDWLJ6ztPB6x60LVcukbAfyzcxQ=', 'Content-Length': '0'} 2023-05-31T21:35:41.989 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '15'), ('X-RGW-Bytes-Used', '45'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000002f6aef316d52fee-006477b6e7-1630-a1'), ('Content-Length', '0'), ('Date', 'Wed, 31 May 2023 21:06:47 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:35:41.989 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/pklxkd-63/obj23 2023-05-31T21:35:41.990 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/pklxkd-63/obj23 2023-05-31T21:35:41.990 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2023-05-31T21:35:41.990 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /pklxkd-63/obj23 2023-05-31T21:35:41.990 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-05-31T21:35:41.991 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-05-31T21:35:41.991 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T21:35:41.991 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-05-31T21:35:41.991 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-05-31T21:35:41.991 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-05-31T21:35:41.992 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-05-31T21:35:41.992 INFO:tasks.rgw_multisite_tests:PUT 2023-05-31T21:35:41.992 INFO:tasks.rgw_multisite_tests:rL0Y20zC+Fzt72VPzMSk2A== 2023-05-31T21:35:41.992 INFO:tasks.rgw_multisite_tests:application/octet-stream 2023-05-31T21:35:41.993 INFO:tasks.rgw_multisite_tests:Wed, 31 May 2023 21:06:47 GMT 2023-05-31T21:35:41.993 INFO:tasks.rgw_multisite_tests:/pklxkd-63/obj23 2023-05-31T21:35:41.993 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-05-31T21:35:41.993 INFO:tasks.rgw_multisite_tests:AWS Q80H11OLNU1S48PPYL26:kLuJ+yf0EPxFUL+nN7YCIf06kSE= 2023-05-31T21:35:41.994 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Wed, 31 May 2023 21:06:47 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:kLuJ+yf0EPxFUL+nN7YCIf06kSE='} 2023-05-31T21:35:41.994 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000ca0c2dfd231121de-006477b6e7-1630-a1'), ('Date', 'Wed, 31 May 2023 21:06:47 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:35:41.994 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/pklxkd-63/ 2023-05-31T21:35:41.994 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/pklxkd-63/ 2023-05-31T21:35:41.995 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-05-31T21:35:41.995 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /pklxkd-63/ 2023-05-31T21:35:41.995 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-05-31T21:35:41.995 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-05-31T21:35:41.995 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T21:35:41.996 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-05-31T21:35:41.996 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-05-31T21:35:41.996 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-05-31T21:35:41.996 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-05-31T21:35:41.997 INFO:tasks.rgw_multisite_tests:HEAD 2023-05-31T21:35:41.997 INFO:tasks.rgw_multisite_tests: 2023-05-31T21:35:41.997 INFO:tasks.rgw_multisite_tests: 2023-05-31T21:35:41.997 INFO:tasks.rgw_multisite_tests:Wed, 31 May 2023 21:06:47 GMT 2023-05-31T21:35:41.998 INFO:tasks.rgw_multisite_tests:/pklxkd-63/ 2023-05-31T21:35:41.998 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-05-31T21:35:41.998 INFO:tasks.rgw_multisite_tests:AWS Q80H11OLNU1S48PPYL26:PDWLJ6ztPB6x60LVcukbAfyzcxQ= 2023-05-31T21:35:41.998 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 21:06:47 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:PDWLJ6ztPB6x60LVcukbAfyzcxQ=', 'Content-Length': '0'} 2023-05-31T21:35:41.998 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '16'), ('X-RGW-Bytes-Used', '48'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000006354af178c15516c-006477b6e7-1630-a1'), ('Content-Length', '0'), ('Date', 'Wed, 31 May 2023 21:06:47 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:35:41.999 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/pklxkd-63/obj46 2023-05-31T21:35:41.999 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/pklxkd-63/obj46 2023-05-31T21:35:41.999 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2023-05-31T21:35:41.999 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /pklxkd-63/obj46 2023-05-31T21:35:42.000 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-05-31T21:35:42.000 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-05-31T21:35:42.000 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T21:35:42.000 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-05-31T21:35:42.001 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-05-31T21:35:42.001 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-05-31T21:35:42.001 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-05-31T21:35:42.001 INFO:tasks.rgw_multisite_tests:PUT 2023-05-31T21:35:42.001 INFO:tasks.rgw_multisite_tests:rL0Y20zC+Fzt72VPzMSk2A== 2023-05-31T21:35:42.002 INFO:tasks.rgw_multisite_tests:application/octet-stream 2023-05-31T21:35:42.002 INFO:tasks.rgw_multisite_tests:Wed, 31 May 2023 21:06:47 GMT 2023-05-31T21:35:42.002 INFO:tasks.rgw_multisite_tests:/pklxkd-63/obj46 2023-05-31T21:35:42.002 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-05-31T21:35:42.003 INFO:tasks.rgw_multisite_tests:AWS Q80H11OLNU1S48PPYL26:UvYUQ/KnzWwSF2T7VbK9kWrVeIE= 2023-05-31T21:35:42.003 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Wed, 31 May 2023 21:06:47 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:UvYUQ/KnzWwSF2T7VbK9kWrVeIE='} 2023-05-31T21:35:42.003 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000715e05fb623f29ba-006477b6e7-1630-a1'), ('Date', 'Wed, 31 May 2023 21:06:47 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:35:42.003 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/pklxkd-63/ 2023-05-31T21:35:42.004 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/pklxkd-63/ 2023-05-31T21:35:42.004 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-05-31T21:35:42.004 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /pklxkd-63/ 2023-05-31T21:35:42.004 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-05-31T21:35:42.004 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-05-31T21:35:42.005 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T21:35:42.005 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-05-31T21:35:42.005 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-05-31T21:35:42.005 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-05-31T21:35:42.006 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-05-31T21:35:42.006 INFO:tasks.rgw_multisite_tests:HEAD 2023-05-31T21:35:42.006 INFO:tasks.rgw_multisite_tests: 2023-05-31T21:35:42.006 INFO:tasks.rgw_multisite_tests: 2023-05-31T21:35:42.007 INFO:tasks.rgw_multisite_tests:Wed, 31 May 2023 21:06:47 GMT 2023-05-31T21:35:42.007 INFO:tasks.rgw_multisite_tests:/pklxkd-63/ 2023-05-31T21:35:42.007 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-05-31T21:35:42.007 INFO:tasks.rgw_multisite_tests:AWS Q80H11OLNU1S48PPYL26:PDWLJ6ztPB6x60LVcukbAfyzcxQ= 2023-05-31T21:35:42.007 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 21:06:47 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:PDWLJ6ztPB6x60LVcukbAfyzcxQ=', 'Content-Length': '0'} 2023-05-31T21:35:42.008 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '17'), ('X-RGW-Bytes-Used', '51'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000e301ad907a50d841-006477b6e7-1630-a1'), ('Content-Length', '0'), ('Date', 'Wed, 31 May 2023 21:06:47 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:35:42.008 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/pklxkd-63/obj69 2023-05-31T21:35:42.008 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/pklxkd-63/obj69 2023-05-31T21:35:42.008 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2023-05-31T21:35:42.009 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /pklxkd-63/obj69 2023-05-31T21:35:42.009 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-05-31T21:35:42.009 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-05-31T21:35:42.009 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T21:35:42.010 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-05-31T21:35:42.010 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-05-31T21:35:42.010 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-05-31T21:35:42.010 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-05-31T21:35:42.010 INFO:tasks.rgw_multisite_tests:PUT 2023-05-31T21:35:42.011 INFO:tasks.rgw_multisite_tests:rL0Y20zC+Fzt72VPzMSk2A== 2023-05-31T21:35:42.011 INFO:tasks.rgw_multisite_tests:application/octet-stream 2023-05-31T21:35:42.011 INFO:tasks.rgw_multisite_tests:Wed, 31 May 2023 21:06:47 GMT 2023-05-31T21:35:42.011 INFO:tasks.rgw_multisite_tests:/pklxkd-63/obj69 2023-05-31T21:35:42.012 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-05-31T21:35:42.012 INFO:tasks.rgw_multisite_tests:AWS Q80H11OLNU1S48PPYL26:Gf7J2S29DgA6yqGpUVOv0uz0FcI= 2023-05-31T21:35:42.012 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Wed, 31 May 2023 21:06:47 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:Gf7J2S29DgA6yqGpUVOv0uz0FcI='} 2023-05-31T21:35:42.012 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000002d4da9ddd66065f0-006477b6e7-1630-a1'), ('Date', 'Wed, 31 May 2023 21:06:47 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:35:42.013 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/pklxkd-63/ 2023-05-31T21:35:42.013 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/pklxkd-63/ 2023-05-31T21:35:42.013 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-05-31T21:35:42.013 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /pklxkd-63/ 2023-05-31T21:35:42.013 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-05-31T21:35:42.014 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-05-31T21:35:42.014 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T21:35:42.014 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-05-31T21:35:42.014 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-05-31T21:35:42.015 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-05-31T21:35:42.015 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-05-31T21:35:42.015 INFO:tasks.rgw_multisite_tests:HEAD 2023-05-31T21:35:42.015 INFO:tasks.rgw_multisite_tests: 2023-05-31T21:35:42.016 INFO:tasks.rgw_multisite_tests: 2023-05-31T21:35:42.016 INFO:tasks.rgw_multisite_tests:Wed, 31 May 2023 21:06:47 GMT 2023-05-31T21:35:42.016 INFO:tasks.rgw_multisite_tests:/pklxkd-63/ 2023-05-31T21:35:42.016 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-05-31T21:35:42.016 INFO:tasks.rgw_multisite_tests:AWS Q80H11OLNU1S48PPYL26:PDWLJ6ztPB6x60LVcukbAfyzcxQ= 2023-05-31T21:35:42.017 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 21:06:47 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:PDWLJ6ztPB6x60LVcukbAfyzcxQ=', 'Content-Length': '0'} 2023-05-31T21:35:42.017 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '18'), ('X-RGW-Bytes-Used', '54'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000004aa7d88af8550c5a-006477b6e7-1630-a1'), ('Content-Length', '0'), ('Date', 'Wed, 31 May 2023 21:06:47 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:35:42.017 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/pklxkd-63/obj92 2023-05-31T21:35:42.017 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/pklxkd-63/obj92 2023-05-31T21:35:42.018 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2023-05-31T21:35:42.018 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /pklxkd-63/obj92 2023-05-31T21:35:42.018 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-05-31T21:35:42.018 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-05-31T21:35:42.019 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T21:35:42.019 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-05-31T21:35:42.019 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-05-31T21:35:42.019 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-05-31T21:35:42.019 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-05-31T21:35:42.020 INFO:tasks.rgw_multisite_tests:PUT 2023-05-31T21:35:42.020 INFO:tasks.rgw_multisite_tests:rL0Y20zC+Fzt72VPzMSk2A== 2023-05-31T21:35:42.020 INFO:tasks.rgw_multisite_tests:application/octet-stream 2023-05-31T21:35:42.020 INFO:tasks.rgw_multisite_tests:Wed, 31 May 2023 21:06:47 GMT 2023-05-31T21:35:42.021 INFO:tasks.rgw_multisite_tests:/pklxkd-63/obj92 2023-05-31T21:35:42.021 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-05-31T21:35:42.021 INFO:tasks.rgw_multisite_tests:AWS Q80H11OLNU1S48PPYL26:Vz62czs73e6agjkfVO37hHgeH40= 2023-05-31T21:35:42.021 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Wed, 31 May 2023 21:06:47 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:Vz62czs73e6agjkfVO37hHgeH40='} 2023-05-31T21:35:42.022 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000e11214b34fd54dbd-006477b6e7-1630-a1'), ('Date', 'Wed, 31 May 2023 21:06:47 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:35:42.022 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/pklxkd-63/ 2023-05-31T21:35:42.022 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/pklxkd-63/ 2023-05-31T21:35:42.022 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-05-31T21:35:42.023 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /pklxkd-63/ 2023-05-31T21:35:42.023 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-05-31T21:35:42.023 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-05-31T21:35:42.023 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T21:35:42.024 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-05-31T21:35:42.024 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-05-31T21:35:42.024 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-05-31T21:35:42.024 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-05-31T21:35:42.024 INFO:tasks.rgw_multisite_tests:HEAD 2023-05-31T21:35:42.025 INFO:tasks.rgw_multisite_tests: 2023-05-31T21:35:42.025 INFO:tasks.rgw_multisite_tests: 2023-05-31T21:35:42.025 INFO:tasks.rgw_multisite_tests:Wed, 31 May 2023 21:06:47 GMT 2023-05-31T21:35:42.025 INFO:tasks.rgw_multisite_tests:/pklxkd-63/ 2023-05-31T21:35:42.026 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-05-31T21:35:42.026 INFO:tasks.rgw_multisite_tests:AWS Q80H11OLNU1S48PPYL26:PDWLJ6ztPB6x60LVcukbAfyzcxQ= 2023-05-31T21:35:42.026 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 21:06:47 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:PDWLJ6ztPB6x60LVcukbAfyzcxQ=', 'Content-Length': '0'} 2023-05-31T21:35:42.026 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '19'), ('X-RGW-Bytes-Used', '57'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000008589207dc545f14a-006477b6e7-1630-a1'), ('Content-Length', '0'), ('Date', 'Wed, 31 May 2023 21:06:47 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:35:42.027 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/pklxkd-63/obj115 2023-05-31T21:35:42.027 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/pklxkd-63/obj115 2023-05-31T21:35:42.027 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2023-05-31T21:35:42.027 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /pklxkd-63/obj115 2023-05-31T21:35:42.027 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-05-31T21:35:42.028 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-05-31T21:35:42.028 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T21:35:42.028 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-05-31T21:35:42.028 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-05-31T21:35:42.029 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-05-31T21:35:42.029 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-05-31T21:35:42.029 INFO:tasks.rgw_multisite_tests:PUT 2023-05-31T21:35:42.029 INFO:tasks.rgw_multisite_tests:rL0Y20zC+Fzt72VPzMSk2A== 2023-05-31T21:35:42.030 INFO:tasks.rgw_multisite_tests:application/octet-stream 2023-05-31T21:35:42.030 INFO:tasks.rgw_multisite_tests:Wed, 31 May 2023 21:06:47 GMT 2023-05-31T21:35:42.030 INFO:tasks.rgw_multisite_tests:/pklxkd-63/obj115 2023-05-31T21:35:42.030 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-05-31T21:35:42.031 INFO:tasks.rgw_multisite_tests:AWS Q80H11OLNU1S48PPYL26:UnGd12vsbkgnzJ8sUj8XTUlw6Lc= 2023-05-31T21:35:42.031 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Wed, 31 May 2023 21:06:47 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:UnGd12vsbkgnzJ8sUj8XTUlw6Lc='} 2023-05-31T21:35:42.031 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000006ff8e05738bf193e-006477b6e7-1630-a1'), ('Date', 'Wed, 31 May 2023 21:06:47 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:35:42.031 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['bucket', 'reshard', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'pklxkd-63', '--num-shards', '29', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-05-31T21:35:42.031 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'reshard', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'pklxkd-63', '--num-shards', '29', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-05-31T21:35:42.032 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket reshard --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --bucket pklxkd-63 --num-shards 29 --yes-i-really-mean-it --cluster c1 --debug-rgw 0 --debug-ms 0 2023-05-31T21:35:42.032 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: ignoring --setuser ceph since I am not root 2023-05-31T21:35:42.032 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-05-31T21:35:42.032 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: tenant: 2023-05-31T21:35:42.033 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: bucket name: pklxkd-63 2023-05-31T21:35:42.033 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: total entries: 20 2023-05-31T21:35:42.033 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/pklxkd-63/ 2023-05-31T21:35:42.033 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/pklxkd-63/ 2023-05-31T21:35:42.034 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-05-31T21:35:42.034 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /pklxkd-63/ 2023-05-31T21:35:42.034 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-05-31T21:35:42.034 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-05-31T21:35:42.035 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T21:35:42.035 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-05-31T21:35:42.035 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-05-31T21:35:42.035 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-05-31T21:35:42.035 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-05-31T21:35:42.036 INFO:tasks.rgw_multisite_tests:HEAD 2023-05-31T21:35:42.036 INFO:tasks.rgw_multisite_tests: 2023-05-31T21:35:42.036 INFO:tasks.rgw_multisite_tests: 2023-05-31T21:35:42.036 INFO:tasks.rgw_multisite_tests:Wed, 31 May 2023 21:06:47 GMT 2023-05-31T21:35:42.037 INFO:tasks.rgw_multisite_tests:/pklxkd-63/ 2023-05-31T21:35:42.037 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-05-31T21:35:42.037 INFO:tasks.rgw_multisite_tests:AWS Q80H11OLNU1S48PPYL26:PDWLJ6ztPB6x60LVcukbAfyzcxQ= 2023-05-31T21:35:42.037 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 21:06:47 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:PDWLJ6ztPB6x60LVcukbAfyzcxQ=', 'Content-Length': '0'} 2023-05-31T21:35:42.038 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '20'), ('X-RGW-Bytes-Used', '60'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000dc77ed229906e7c0-006477b6e7-1630-a1'), ('Content-Length', '0'), ('Date', 'Wed, 31 May 2023 21:06:47 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:35:42.038 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/pklxkd-63/obj29 2023-05-31T21:35:42.038 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/pklxkd-63/obj29 2023-05-31T21:35:42.038 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2023-05-31T21:35:42.038 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /pklxkd-63/obj29 2023-05-31T21:35:42.039 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-05-31T21:35:42.039 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-05-31T21:35:42.039 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T21:35:42.039 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-05-31T21:35:42.040 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-05-31T21:35:42.040 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-05-31T21:35:42.040 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-05-31T21:35:42.040 INFO:tasks.rgw_multisite_tests:PUT 2023-05-31T21:35:42.041 INFO:tasks.rgw_multisite_tests:rL0Y20zC+Fzt72VPzMSk2A== 2023-05-31T21:35:42.041 INFO:tasks.rgw_multisite_tests:application/octet-stream 2023-05-31T21:35:42.041 INFO:tasks.rgw_multisite_tests:Wed, 31 May 2023 21:06:47 GMT 2023-05-31T21:35:42.041 INFO:tasks.rgw_multisite_tests:/pklxkd-63/obj29 2023-05-31T21:35:42.041 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-05-31T21:35:42.042 INFO:tasks.rgw_multisite_tests:AWS Q80H11OLNU1S48PPYL26:n12DIRivPPm+IGWVHMwiotYrp0s= 2023-05-31T21:35:42.042 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Wed, 31 May 2023 21:06:47 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:n12DIRivPPm+IGWVHMwiotYrp0s='} 2023-05-31T21:35:42.042 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000022fbc0d1f071983c-006477b6e7-1630-a1'), ('Date', 'Wed, 31 May 2023 21:06:47 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:35:42.042 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/pklxkd-63/ 2023-05-31T21:35:42.043 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/pklxkd-63/ 2023-05-31T21:35:42.043 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-05-31T21:35:42.043 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /pklxkd-63/ 2023-05-31T21:35:42.043 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-05-31T21:35:42.044 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-05-31T21:35:42.044 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T21:35:42.044 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-05-31T21:35:42.044 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-05-31T21:35:42.044 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-05-31T21:35:42.045 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-05-31T21:35:42.045 INFO:tasks.rgw_multisite_tests:HEAD 2023-05-31T21:35:42.045 INFO:tasks.rgw_multisite_tests: 2023-05-31T21:35:42.045 INFO:tasks.rgw_multisite_tests: 2023-05-31T21:35:42.046 INFO:tasks.rgw_multisite_tests:Wed, 31 May 2023 21:06:47 GMT 2023-05-31T21:35:42.046 INFO:tasks.rgw_multisite_tests:/pklxkd-63/ 2023-05-31T21:35:42.046 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-05-31T21:35:42.046 INFO:tasks.rgw_multisite_tests:AWS Q80H11OLNU1S48PPYL26:PDWLJ6ztPB6x60LVcukbAfyzcxQ= 2023-05-31T21:35:42.046 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 21:06:47 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:PDWLJ6ztPB6x60LVcukbAfyzcxQ=', 'Content-Length': '0'} 2023-05-31T21:35:42.047 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '21'), ('X-RGW-Bytes-Used', '63'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000006b07585032522d26-006477b6e7-1630-a1'), ('Content-Length', '0'), ('Date', 'Wed, 31 May 2023 21:06:47 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:35:42.047 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/pklxkd-63/obj58 2023-05-31T21:35:42.047 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/pklxkd-63/obj58 2023-05-31T21:35:42.047 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2023-05-31T21:35:42.048 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /pklxkd-63/obj58 2023-05-31T21:35:42.048 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-05-31T21:35:42.048 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-05-31T21:35:42.048 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T21:35:42.049 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-05-31T21:35:42.049 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-05-31T21:35:42.049 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-05-31T21:35:42.049 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-05-31T21:35:42.049 INFO:tasks.rgw_multisite_tests:PUT 2023-05-31T21:35:42.050 INFO:tasks.rgw_multisite_tests:rL0Y20zC+Fzt72VPzMSk2A== 2023-05-31T21:35:42.050 INFO:tasks.rgw_multisite_tests:application/octet-stream 2023-05-31T21:35:42.050 INFO:tasks.rgw_multisite_tests:Wed, 31 May 2023 21:06:47 GMT 2023-05-31T21:35:42.050 INFO:tasks.rgw_multisite_tests:/pklxkd-63/obj58 2023-05-31T21:35:42.051 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-05-31T21:35:42.051 INFO:tasks.rgw_multisite_tests:AWS Q80H11OLNU1S48PPYL26:Ag7/wh1FO6r0zI2o5UknOYdPlDs= 2023-05-31T21:35:42.051 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Wed, 31 May 2023 21:06:47 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:Ag7/wh1FO6r0zI2o5UknOYdPlDs='} 2023-05-31T21:35:42.051 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000094969ce405d469dc-006477b6e7-1630-a1'), ('Date', 'Wed, 31 May 2023 21:06:47 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:35:42.052 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/pklxkd-63/ 2023-05-31T21:35:42.052 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/pklxkd-63/ 2023-05-31T21:35:42.052 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-05-31T21:35:42.052 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /pklxkd-63/ 2023-05-31T21:35:42.052 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-05-31T21:35:42.053 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-05-31T21:35:42.053 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T21:35:42.053 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-05-31T21:35:42.053 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-05-31T21:35:42.054 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-05-31T21:35:42.054 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-05-31T21:35:42.054 INFO:tasks.rgw_multisite_tests:HEAD 2023-05-31T21:35:42.054 INFO:tasks.rgw_multisite_tests: 2023-05-31T21:35:42.055 INFO:tasks.rgw_multisite_tests: 2023-05-31T21:35:42.055 INFO:tasks.rgw_multisite_tests:Wed, 31 May 2023 21:06:47 GMT 2023-05-31T21:35:42.055 INFO:tasks.rgw_multisite_tests:/pklxkd-63/ 2023-05-31T21:35:42.055 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-05-31T21:35:42.055 INFO:tasks.rgw_multisite_tests:AWS Q80H11OLNU1S48PPYL26:PDWLJ6ztPB6x60LVcukbAfyzcxQ= 2023-05-31T21:35:42.056 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 21:06:47 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:PDWLJ6ztPB6x60LVcukbAfyzcxQ=', 'Content-Length': '0'} 2023-05-31T21:35:42.056 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '22'), ('X-RGW-Bytes-Used', '66'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000032600a2834f61e87-006477b6e7-1630-a1'), ('Content-Length', '0'), ('Date', 'Wed, 31 May 2023 21:06:47 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:35:42.056 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/pklxkd-63/obj87 2023-05-31T21:35:42.056 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/pklxkd-63/obj87 2023-05-31T21:35:42.057 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2023-05-31T21:35:42.057 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /pklxkd-63/obj87 2023-05-31T21:35:42.057 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-05-31T21:35:42.057 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-05-31T21:35:42.058 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T21:35:42.058 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-05-31T21:35:42.058 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-05-31T21:35:42.058 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-05-31T21:35:42.058 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-05-31T21:35:42.059 INFO:tasks.rgw_multisite_tests:PUT 2023-05-31T21:35:42.059 INFO:tasks.rgw_multisite_tests:rL0Y20zC+Fzt72VPzMSk2A== 2023-05-31T21:35:42.059 INFO:tasks.rgw_multisite_tests:application/octet-stream 2023-05-31T21:35:42.059 INFO:tasks.rgw_multisite_tests:Wed, 31 May 2023 21:06:47 GMT 2023-05-31T21:35:42.060 INFO:tasks.rgw_multisite_tests:/pklxkd-63/obj87 2023-05-31T21:35:42.060 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-05-31T21:35:42.060 INFO:tasks.rgw_multisite_tests:AWS Q80H11OLNU1S48PPYL26:DIqorMv0psvr2Biz6Mzu+2JQ5AU= 2023-05-31T21:35:42.060 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Wed, 31 May 2023 21:06:47 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:DIqorMv0psvr2Biz6Mzu+2JQ5AU='} 2023-05-31T21:35:42.060 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000cb7526c0c2b51653-006477b6e7-1630-a1'), ('Date', 'Wed, 31 May 2023 21:06:47 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:35:42.061 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/pklxkd-63/ 2023-05-31T21:35:42.061 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/pklxkd-63/ 2023-05-31T21:35:42.061 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-05-31T21:35:42.061 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /pklxkd-63/ 2023-05-31T21:35:42.062 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-05-31T21:35:42.062 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-05-31T21:35:42.062 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T21:35:42.062 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-05-31T21:35:42.063 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-05-31T21:35:42.063 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-05-31T21:35:42.063 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-05-31T21:35:42.063 INFO:tasks.rgw_multisite_tests:HEAD 2023-05-31T21:35:42.063 INFO:tasks.rgw_multisite_tests: 2023-05-31T21:35:42.064 INFO:tasks.rgw_multisite_tests: 2023-05-31T21:35:42.064 INFO:tasks.rgw_multisite_tests:Wed, 31 May 2023 21:06:47 GMT 2023-05-31T21:35:42.064 INFO:tasks.rgw_multisite_tests:/pklxkd-63/ 2023-05-31T21:35:42.064 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-05-31T21:35:42.065 INFO:tasks.rgw_multisite_tests:AWS Q80H11OLNU1S48PPYL26:PDWLJ6ztPB6x60LVcukbAfyzcxQ= 2023-05-31T21:35:42.065 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 21:06:47 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:PDWLJ6ztPB6x60LVcukbAfyzcxQ=', 'Content-Length': '0'} 2023-05-31T21:35:42.065 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '23'), ('X-RGW-Bytes-Used', '69'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000f0bf5f76852f04d5-006477b6e7-1630-a1'), ('Content-Length', '0'), ('Date', 'Wed, 31 May 2023 21:06:47 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:35:42.065 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/pklxkd-63/obj116 2023-05-31T21:35:42.066 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/pklxkd-63/obj116 2023-05-31T21:35:42.066 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2023-05-31T21:35:42.066 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /pklxkd-63/obj116 2023-05-31T21:35:42.066 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-05-31T21:35:42.066 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-05-31T21:35:42.067 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T21:35:42.067 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-05-31T21:35:42.067 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-05-31T21:35:42.067 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-05-31T21:35:42.068 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-05-31T21:35:42.068 INFO:tasks.rgw_multisite_tests:PUT 2023-05-31T21:35:42.068 INFO:tasks.rgw_multisite_tests:rL0Y20zC+Fzt72VPzMSk2A== 2023-05-31T21:35:42.068 INFO:tasks.rgw_multisite_tests:application/octet-stream 2023-05-31T21:35:42.069 INFO:tasks.rgw_multisite_tests:Wed, 31 May 2023 21:06:47 GMT 2023-05-31T21:35:42.069 INFO:tasks.rgw_multisite_tests:/pklxkd-63/obj116 2023-05-31T21:35:42.069 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-05-31T21:35:42.069 INFO:tasks.rgw_multisite_tests:AWS Q80H11OLNU1S48PPYL26:/l1LHH0C3O/ogvD6rYWqc4EHZm8= 2023-05-31T21:35:42.069 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Wed, 31 May 2023 21:06:47 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:/l1LHH0C3O/ogvD6rYWqc4EHZm8='} 2023-05-31T21:35:42.070 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000b47fd3bb164eab63-006477b6e7-1630-a1'), ('Date', 'Wed, 31 May 2023 21:06:47 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:35:42.070 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/pklxkd-63/ 2023-05-31T21:35:42.070 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/pklxkd-63/ 2023-05-31T21:35:42.070 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-05-31T21:35:42.071 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /pklxkd-63/ 2023-05-31T21:35:42.071 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-05-31T21:35:42.071 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-05-31T21:35:42.071 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T21:35:42.072 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-05-31T21:35:42.072 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-05-31T21:35:42.072 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-05-31T21:35:42.072 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-05-31T21:35:42.072 INFO:tasks.rgw_multisite_tests:HEAD 2023-05-31T21:35:42.073 INFO:tasks.rgw_multisite_tests: 2023-05-31T21:35:42.073 INFO:tasks.rgw_multisite_tests: 2023-05-31T21:35:42.073 INFO:tasks.rgw_multisite_tests:Wed, 31 May 2023 21:06:47 GMT 2023-05-31T21:35:42.073 INFO:tasks.rgw_multisite_tests:/pklxkd-63/ 2023-05-31T21:35:42.074 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-05-31T21:35:42.074 INFO:tasks.rgw_multisite_tests:AWS Q80H11OLNU1S48PPYL26:PDWLJ6ztPB6x60LVcukbAfyzcxQ= 2023-05-31T21:35:42.074 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 21:06:47 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:PDWLJ6ztPB6x60LVcukbAfyzcxQ=', 'Content-Length': '0'} 2023-05-31T21:35:42.074 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '24'), ('X-RGW-Bytes-Used', '72'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000007ba0f2e718d47479-006477b6e7-1630-a1'), ('Content-Length', '0'), ('Date', 'Wed, 31 May 2023 21:06:47 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:35:42.075 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/pklxkd-63/obj145 2023-05-31T21:35:42.075 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/pklxkd-63/obj145 2023-05-31T21:35:42.075 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2023-05-31T21:35:42.075 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /pklxkd-63/obj145 2023-05-31T21:35:42.075 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-05-31T21:35:42.076 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-05-31T21:35:42.076 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T21:35:42.076 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-05-31T21:35:42.076 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-05-31T21:35:42.077 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-05-31T21:35:42.077 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-05-31T21:35:42.077 INFO:tasks.rgw_multisite_tests:PUT 2023-05-31T21:35:42.077 INFO:tasks.rgw_multisite_tests:rL0Y20zC+Fzt72VPzMSk2A== 2023-05-31T21:35:42.078 INFO:tasks.rgw_multisite_tests:application/octet-stream 2023-05-31T21:35:42.078 INFO:tasks.rgw_multisite_tests:Wed, 31 May 2023 21:06:47 GMT 2023-05-31T21:35:42.078 INFO:tasks.rgw_multisite_tests:/pklxkd-63/obj145 2023-05-31T21:35:42.078 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-05-31T21:35:42.078 INFO:tasks.rgw_multisite_tests:AWS Q80H11OLNU1S48PPYL26:v6SA4aeiaMf1fHI0ciY61CF5XPo= 2023-05-31T21:35:42.079 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Wed, 31 May 2023 21:06:47 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:v6SA4aeiaMf1fHI0ciY61CF5XPo='} 2023-05-31T21:35:42.079 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000008cc5a189d2ccabd4-006477b6e7-1630-a1'), ('Date', 'Wed, 31 May 2023 21:06:47 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:35:42.079 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['bucket', 'reshard', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'pklxkd-63', '--num-shards', '31', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-05-31T21:35:42.079 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'reshard', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'pklxkd-63', '--num-shards', '31', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-05-31T21:35:42.080 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket reshard --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --bucket pklxkd-63 --num-shards 31 --yes-i-really-mean-it --cluster c1 --debug-rgw 0 --debug-ms 0 2023-05-31T21:35:42.080 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: ignoring --setuser ceph since I am not root 2023-05-31T21:35:42.080 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-05-31T21:35:42.080 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: tenant: 2023-05-31T21:35:42.081 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: bucket name: pklxkd-63 2023-05-31T21:35:42.081 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: total entries: 25 2023-05-31T21:35:42.081 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/pklxkd-63/ 2023-05-31T21:35:42.081 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/pklxkd-63/ 2023-05-31T21:35:42.081 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-05-31T21:35:42.082 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /pklxkd-63/ 2023-05-31T21:35:42.082 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-05-31T21:35:42.082 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-05-31T21:35:42.082 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T21:35:42.083 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-05-31T21:35:42.083 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-05-31T21:35:42.083 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-05-31T21:35:42.083 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-05-31T21:35:42.084 INFO:tasks.rgw_multisite_tests:HEAD 2023-05-31T21:35:42.084 INFO:tasks.rgw_multisite_tests: 2023-05-31T21:35:42.084 INFO:tasks.rgw_multisite_tests: 2023-05-31T21:35:42.084 INFO:tasks.rgw_multisite_tests:Wed, 31 May 2023 21:06:47 GMT 2023-05-31T21:35:42.085 INFO:tasks.rgw_multisite_tests:/pklxkd-63/ 2023-05-31T21:35:42.085 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-05-31T21:35:42.085 INFO:tasks.rgw_multisite_tests:AWS Q80H11OLNU1S48PPYL26:PDWLJ6ztPB6x60LVcukbAfyzcxQ= 2023-05-31T21:35:42.085 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 21:06:47 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:PDWLJ6ztPB6x60LVcukbAfyzcxQ=', 'Content-Length': '0'} 2023-05-31T21:35:42.086 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '25'), ('X-RGW-Bytes-Used', '75'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000001d107802d42c9954-006477b6e7-1630-a1'), ('Content-Length', '0'), ('Date', 'Wed, 31 May 2023 21:06:47 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:35:42.086 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/pklxkd-63/obj31 2023-05-31T21:35:42.086 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/pklxkd-63/obj31 2023-05-31T21:35:42.086 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2023-05-31T21:35:42.086 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /pklxkd-63/obj31 2023-05-31T21:35:42.087 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-05-31T21:35:42.087 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-05-31T21:35:42.087 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T21:35:42.087 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-05-31T21:35:42.088 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-05-31T21:35:42.088 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-05-31T21:35:42.088 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-05-31T21:35:42.088 INFO:tasks.rgw_multisite_tests:PUT 2023-05-31T21:35:42.089 INFO:tasks.rgw_multisite_tests:rL0Y20zC+Fzt72VPzMSk2A== 2023-05-31T21:35:42.089 INFO:tasks.rgw_multisite_tests:application/octet-stream 2023-05-31T21:35:42.089 INFO:tasks.rgw_multisite_tests:Wed, 31 May 2023 21:06:47 GMT 2023-05-31T21:35:42.089 INFO:tasks.rgw_multisite_tests:/pklxkd-63/obj31 2023-05-31T21:35:42.089 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-05-31T21:35:42.090 INFO:tasks.rgw_multisite_tests:AWS Q80H11OLNU1S48PPYL26:zrhkmI62RpvkHDJqCk77h0wa8hM= 2023-05-31T21:35:42.090 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Wed, 31 May 2023 21:06:47 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:zrhkmI62RpvkHDJqCk77h0wa8hM='} 2023-05-31T21:35:42.090 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000004cea157904a87577-006477b6e7-1630-a1'), ('Date', 'Wed, 31 May 2023 21:06:47 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:35:42.090 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/pklxkd-63/ 2023-05-31T21:35:42.091 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/pklxkd-63/ 2023-05-31T21:35:42.091 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-05-31T21:35:42.091 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /pklxkd-63/ 2023-05-31T21:35:42.091 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-05-31T21:35:42.092 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-05-31T21:35:42.092 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T21:35:42.092 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-05-31T21:35:42.092 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-05-31T21:35:42.092 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-05-31T21:35:42.093 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-05-31T21:35:42.093 INFO:tasks.rgw_multisite_tests:HEAD 2023-05-31T21:35:42.093 INFO:tasks.rgw_multisite_tests: 2023-05-31T21:35:42.093 INFO:tasks.rgw_multisite_tests: 2023-05-31T21:35:42.094 INFO:tasks.rgw_multisite_tests:Wed, 31 May 2023 21:06:47 GMT 2023-05-31T21:35:42.094 INFO:tasks.rgw_multisite_tests:/pklxkd-63/ 2023-05-31T21:35:42.094 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-05-31T21:35:42.094 INFO:tasks.rgw_multisite_tests:AWS Q80H11OLNU1S48PPYL26:PDWLJ6ztPB6x60LVcukbAfyzcxQ= 2023-05-31T21:35:42.095 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 21:06:47 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:PDWLJ6ztPB6x60LVcukbAfyzcxQ=', 'Content-Length': '0'} 2023-05-31T21:35:42.095 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '26'), ('X-RGW-Bytes-Used', '78'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000ed54f60914797f2d-006477b6e7-1630-a1'), ('Content-Length', '0'), ('Date', 'Wed, 31 May 2023 21:06:47 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:35:42.095 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/pklxkd-63/obj62 2023-05-31T21:35:42.095 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/pklxkd-63/obj62 2023-05-31T21:35:42.096 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2023-05-31T21:35:42.096 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /pklxkd-63/obj62 2023-05-31T21:35:42.096 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-05-31T21:35:42.096 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-05-31T21:35:42.097 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T21:35:42.097 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-05-31T21:35:42.097 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-05-31T21:35:42.097 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-05-31T21:35:42.097 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-05-31T21:35:42.098 INFO:tasks.rgw_multisite_tests:PUT 2023-05-31T21:35:42.098 INFO:tasks.rgw_multisite_tests:rL0Y20zC+Fzt72VPzMSk2A== 2023-05-31T21:35:42.098 INFO:tasks.rgw_multisite_tests:application/octet-stream 2023-05-31T21:35:42.098 INFO:tasks.rgw_multisite_tests:Wed, 31 May 2023 21:06:47 GMT 2023-05-31T21:35:42.099 INFO:tasks.rgw_multisite_tests:/pklxkd-63/obj62 2023-05-31T21:35:42.099 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-05-31T21:35:42.099 INFO:tasks.rgw_multisite_tests:AWS Q80H11OLNU1S48PPYL26:Uv+vO4o+BopQS9/BJ3j56lpa4L8= 2023-05-31T21:35:42.099 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Wed, 31 May 2023 21:06:47 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:Uv+vO4o+BopQS9/BJ3j56lpa4L8='} 2023-05-31T21:35:42.099 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000a8c66f26d43f8eef-006477b6e7-1630-a1'), ('Date', 'Wed, 31 May 2023 21:06:47 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:35:42.100 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/pklxkd-63/ 2023-05-31T21:35:42.100 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/pklxkd-63/ 2023-05-31T21:35:42.100 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-05-31T21:35:42.100 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /pklxkd-63/ 2023-05-31T21:35:42.101 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-05-31T21:35:42.101 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-05-31T21:35:42.101 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T21:35:42.101 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-05-31T21:35:42.102 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-05-31T21:35:42.102 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-05-31T21:35:42.102 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-05-31T21:35:42.102 INFO:tasks.rgw_multisite_tests:HEAD 2023-05-31T21:35:42.102 INFO:tasks.rgw_multisite_tests: 2023-05-31T21:35:42.103 INFO:tasks.rgw_multisite_tests: 2023-05-31T21:35:42.103 INFO:tasks.rgw_multisite_tests:Wed, 31 May 2023 21:06:47 GMT 2023-05-31T21:35:42.103 INFO:tasks.rgw_multisite_tests:/pklxkd-63/ 2023-05-31T21:35:42.103 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-05-31T21:35:42.104 INFO:tasks.rgw_multisite_tests:AWS Q80H11OLNU1S48PPYL26:PDWLJ6ztPB6x60LVcukbAfyzcxQ= 2023-05-31T21:35:42.104 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 21:06:47 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:PDWLJ6ztPB6x60LVcukbAfyzcxQ=', 'Content-Length': '0'} 2023-05-31T21:35:42.104 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '27'), ('X-RGW-Bytes-Used', '81'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000003f12ddc3e2f02aa6-006477b6e7-1630-a1'), ('Content-Length', '0'), ('Date', 'Wed, 31 May 2023 21:06:47 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:35:42.104 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/pklxkd-63/obj93 2023-05-31T21:35:42.105 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/pklxkd-63/obj93 2023-05-31T21:35:42.105 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2023-05-31T21:35:42.105 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /pklxkd-63/obj93 2023-05-31T21:35:42.105 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-05-31T21:35:42.105 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-05-31T21:35:42.106 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T21:35:42.106 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-05-31T21:35:42.106 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-05-31T21:35:42.106 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-05-31T21:35:42.107 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-05-31T21:35:42.107 INFO:tasks.rgw_multisite_tests:PUT 2023-05-31T21:35:42.107 INFO:tasks.rgw_multisite_tests:rL0Y20zC+Fzt72VPzMSk2A== 2023-05-31T21:35:42.107 INFO:tasks.rgw_multisite_tests:application/octet-stream 2023-05-31T21:35:42.108 INFO:tasks.rgw_multisite_tests:Wed, 31 May 2023 21:06:47 GMT 2023-05-31T21:35:42.108 INFO:tasks.rgw_multisite_tests:/pklxkd-63/obj93 2023-05-31T21:35:42.108 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-05-31T21:35:42.108 INFO:tasks.rgw_multisite_tests:AWS Q80H11OLNU1S48PPYL26:+c2QpvLlf0odtZW94vsbDts/7+0= 2023-05-31T21:35:42.108 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Wed, 31 May 2023 21:06:47 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:+c2QpvLlf0odtZW94vsbDts/7+0='} 2023-05-31T21:35:42.109 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000002c83ed27607e74e0-006477b6e7-1630-a1'), ('Date', 'Wed, 31 May 2023 21:06:47 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:35:42.109 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/pklxkd-63/ 2023-05-31T21:35:42.109 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/pklxkd-63/ 2023-05-31T21:35:42.109 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-05-31T21:35:42.110 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /pklxkd-63/ 2023-05-31T21:35:42.110 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-05-31T21:35:42.110 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-05-31T21:35:42.110 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T21:35:42.111 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-05-31T21:35:42.111 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-05-31T21:35:42.111 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-05-31T21:35:42.111 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-05-31T21:35:42.111 INFO:tasks.rgw_multisite_tests:HEAD 2023-05-31T21:35:42.112 INFO:tasks.rgw_multisite_tests: 2023-05-31T21:35:42.112 INFO:tasks.rgw_multisite_tests: 2023-05-31T21:35:42.112 INFO:tasks.rgw_multisite_tests:Wed, 31 May 2023 21:06:47 GMT 2023-05-31T21:35:42.112 INFO:tasks.rgw_multisite_tests:/pklxkd-63/ 2023-05-31T21:35:42.113 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-05-31T21:35:42.113 INFO:tasks.rgw_multisite_tests:AWS Q80H11OLNU1S48PPYL26:PDWLJ6ztPB6x60LVcukbAfyzcxQ= 2023-05-31T21:35:42.113 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 21:06:47 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:PDWLJ6ztPB6x60LVcukbAfyzcxQ=', 'Content-Length': '0'} 2023-05-31T21:35:42.113 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '28'), ('X-RGW-Bytes-Used', '84'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000009659443923b9493c-006477b6e7-1630-a1'), ('Content-Length', '0'), ('Date', 'Wed, 31 May 2023 21:06:47 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:35:42.113 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/pklxkd-63/obj124 2023-05-31T21:35:42.114 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/pklxkd-63/obj124 2023-05-31T21:35:42.114 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2023-05-31T21:35:42.114 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /pklxkd-63/obj124 2023-05-31T21:35:42.114 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-05-31T21:35:42.115 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-05-31T21:35:42.115 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T21:35:42.115 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-05-31T21:35:42.115 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-05-31T21:35:42.116 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-05-31T21:35:42.116 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-05-31T21:35:42.116 INFO:tasks.rgw_multisite_tests:PUT 2023-05-31T21:35:42.116 INFO:tasks.rgw_multisite_tests:rL0Y20zC+Fzt72VPzMSk2A== 2023-05-31T21:35:42.116 INFO:tasks.rgw_multisite_tests:application/octet-stream 2023-05-31T21:35:42.117 INFO:tasks.rgw_multisite_tests:Wed, 31 May 2023 21:06:47 GMT 2023-05-31T21:35:42.117 INFO:tasks.rgw_multisite_tests:/pklxkd-63/obj124 2023-05-31T21:35:42.117 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-05-31T21:35:42.117 INFO:tasks.rgw_multisite_tests:AWS Q80H11OLNU1S48PPYL26:SSajoq1ocXBjvYoK3elOWblnKqE= 2023-05-31T21:35:42.118 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Wed, 31 May 2023 21:06:47 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:SSajoq1ocXBjvYoK3elOWblnKqE='} 2023-05-31T21:35:42.118 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000b59fb54ab9f2902b-006477b6e7-1630-a1'), ('Date', 'Wed, 31 May 2023 21:06:48 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:35:42.118 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/pklxkd-63/ 2023-05-31T21:35:42.118 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/pklxkd-63/ 2023-05-31T21:35:42.119 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-05-31T21:35:42.119 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /pklxkd-63/ 2023-05-31T21:35:42.119 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-05-31T21:35:42.119 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-05-31T21:35:42.119 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T21:35:42.120 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-05-31T21:35:42.120 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-05-31T21:35:42.120 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-05-31T21:35:42.120 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-05-31T21:35:42.121 INFO:tasks.rgw_multisite_tests:HEAD 2023-05-31T21:35:42.121 INFO:tasks.rgw_multisite_tests: 2023-05-31T21:35:42.121 INFO:tasks.rgw_multisite_tests: 2023-05-31T21:35:42.121 INFO:tasks.rgw_multisite_tests:Wed, 31 May 2023 21:06:48 GMT 2023-05-31T21:35:42.121 INFO:tasks.rgw_multisite_tests:/pklxkd-63/ 2023-05-31T21:35:42.122 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-05-31T21:35:42.122 INFO:tasks.rgw_multisite_tests:AWS Q80H11OLNU1S48PPYL26:3B2mkaM+ObC4zRGdBhJqL6TchVg= 2023-05-31T21:35:42.122 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 21:06:48 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:3B2mkaM+ObC4zRGdBhJqL6TchVg=', 'Content-Length': '0'} 2023-05-31T21:35:42.122 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '29'), ('X-RGW-Bytes-Used', '87'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000e536e172573e3e33-006477b6e8-1630-a1'), ('Content-Length', '0'), ('Date', 'Wed, 31 May 2023 21:06:48 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:35:42.123 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/pklxkd-63/obj155 2023-05-31T21:35:42.123 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/pklxkd-63/obj155 2023-05-31T21:35:42.123 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2023-05-31T21:35:42.123 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /pklxkd-63/obj155 2023-05-31T21:35:42.124 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-05-31T21:35:42.124 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-05-31T21:35:42.124 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T21:35:42.124 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-05-31T21:35:42.124 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-05-31T21:35:42.125 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-05-31T21:35:42.125 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-05-31T21:35:42.125 INFO:tasks.rgw_multisite_tests:PUT 2023-05-31T21:35:42.125 INFO:tasks.rgw_multisite_tests:rL0Y20zC+Fzt72VPzMSk2A== 2023-05-31T21:35:42.126 INFO:tasks.rgw_multisite_tests:application/octet-stream 2023-05-31T21:35:42.126 INFO:tasks.rgw_multisite_tests:Wed, 31 May 2023 21:06:48 GMT 2023-05-31T21:35:42.126 INFO:tasks.rgw_multisite_tests:/pklxkd-63/obj155 2023-05-31T21:35:42.126 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-05-31T21:35:42.127 INFO:tasks.rgw_multisite_tests:AWS Q80H11OLNU1S48PPYL26:dcpNYhcIb8JZOiCJOlV6lv0aBkE= 2023-05-31T21:35:42.127 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Wed, 31 May 2023 21:06:48 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:dcpNYhcIb8JZOiCJOlV6lv0aBkE='} 2023-05-31T21:35:42.127 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000dd7ea80d569ef016-006477b6e8-1630-a1'), ('Date', 'Wed, 31 May 2023 21:06:48 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:35:42.127 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['bucket', 'reshard', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'pklxkd-63', '--num-shards', '37', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-05-31T21:35:42.127 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'reshard', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'pklxkd-63', '--num-shards', '37', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-05-31T21:35:42.128 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket reshard --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --bucket pklxkd-63 --num-shards 37 --yes-i-really-mean-it --cluster c1 --debug-rgw 0 --debug-ms 0 2023-05-31T21:35:42.128 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: ignoring --setuser ceph since I am not root 2023-05-31T21:35:42.128 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-05-31T21:35:42.128 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: tenant: 2023-05-31T21:35:42.129 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: bucket name: pklxkd-63 2023-05-31T21:35:42.129 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: total entries: 30 2023-05-31T21:35:42.129 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/pklxkd-63/ 2023-05-31T21:35:42.129 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/pklxkd-63/ 2023-05-31T21:35:42.130 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-05-31T21:35:42.130 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /pklxkd-63/ 2023-05-31T21:35:42.130 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-05-31T21:35:42.130 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-05-31T21:35:42.130 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T21:35:42.131 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-05-31T21:35:42.131 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-05-31T21:35:42.131 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-05-31T21:35:42.131 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-05-31T21:35:42.132 INFO:tasks.rgw_multisite_tests:HEAD 2023-05-31T21:35:42.132 INFO:tasks.rgw_multisite_tests: 2023-05-31T21:35:42.132 INFO:tasks.rgw_multisite_tests: 2023-05-31T21:35:42.132 INFO:tasks.rgw_multisite_tests:Wed, 31 May 2023 21:06:48 GMT 2023-05-31T21:35:42.133 INFO:tasks.rgw_multisite_tests:/pklxkd-63/ 2023-05-31T21:35:42.133 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-05-31T21:35:42.133 INFO:tasks.rgw_multisite_tests:AWS Q80H11OLNU1S48PPYL26:3B2mkaM+ObC4zRGdBhJqL6TchVg= 2023-05-31T21:35:42.133 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 21:06:48 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:3B2mkaM+ObC4zRGdBhJqL6TchVg=', 'Content-Length': '0'} 2023-05-31T21:35:42.133 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '30'), ('X-RGW-Bytes-Used', '90'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000e18f9d21b0ef49af-006477b6e8-1630-a1'), ('Content-Length', '0'), ('Date', 'Wed, 31 May 2023 21:06:48 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:35:42.134 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/pklxkd-63/obj37 2023-05-31T21:35:42.134 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/pklxkd-63/obj37 2023-05-31T21:35:42.134 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2023-05-31T21:35:42.134 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /pklxkd-63/obj37 2023-05-31T21:35:42.135 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-05-31T21:35:42.135 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-05-31T21:35:42.135 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T21:35:42.135 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-05-31T21:35:42.135 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-05-31T21:35:42.136 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-05-31T21:35:42.136 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-05-31T21:35:42.136 INFO:tasks.rgw_multisite_tests:PUT 2023-05-31T21:35:42.136 INFO:tasks.rgw_multisite_tests:rL0Y20zC+Fzt72VPzMSk2A== 2023-05-31T21:35:42.137 INFO:tasks.rgw_multisite_tests:application/octet-stream 2023-05-31T21:35:42.137 INFO:tasks.rgw_multisite_tests:Wed, 31 May 2023 21:06:48 GMT 2023-05-31T21:35:42.137 INFO:tasks.rgw_multisite_tests:/pklxkd-63/obj37 2023-05-31T21:35:42.137 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-05-31T21:35:42.138 INFO:tasks.rgw_multisite_tests:AWS Q80H11OLNU1S48PPYL26:+3UrOJNnZc3yQnfzzTyY32/f1g4= 2023-05-31T21:35:42.138 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Wed, 31 May 2023 21:06:48 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:+3UrOJNnZc3yQnfzzTyY32/f1g4='} 2023-05-31T21:35:42.138 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000ed0eba0a8f78ebd4-006477b6e8-1630-a1'), ('Date', 'Wed, 31 May 2023 21:06:48 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:35:42.138 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/pklxkd-63/ 2023-05-31T21:35:42.138 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/pklxkd-63/ 2023-05-31T21:35:42.139 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-05-31T21:35:42.139 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /pklxkd-63/ 2023-05-31T21:35:42.139 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-05-31T21:35:42.139 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-05-31T21:35:42.140 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T21:35:42.140 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-05-31T21:35:42.140 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-05-31T21:35:42.140 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-05-31T21:35:42.141 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-05-31T21:35:42.141 INFO:tasks.rgw_multisite_tests:HEAD 2023-05-31T21:35:42.141 INFO:tasks.rgw_multisite_tests: 2023-05-31T21:35:42.141 INFO:tasks.rgw_multisite_tests: 2023-05-31T21:35:42.141 INFO:tasks.rgw_multisite_tests:Wed, 31 May 2023 21:06:48 GMT 2023-05-31T21:35:42.142 INFO:tasks.rgw_multisite_tests:/pklxkd-63/ 2023-05-31T21:35:42.142 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-05-31T21:35:42.142 INFO:tasks.rgw_multisite_tests:AWS Q80H11OLNU1S48PPYL26:3B2mkaM+ObC4zRGdBhJqL6TchVg= 2023-05-31T21:35:42.142 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 21:06:48 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:3B2mkaM+ObC4zRGdBhJqL6TchVg=', 'Content-Length': '0'} 2023-05-31T21:35:42.143 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '31'), ('X-RGW-Bytes-Used', '93'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000011ac91a76a6b24b2-006477b6e8-1630-a1'), ('Content-Length', '0'), ('Date', 'Wed, 31 May 2023 21:06:48 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:35:42.143 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/pklxkd-63/obj74 2023-05-31T21:35:42.143 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/pklxkd-63/obj74 2023-05-31T21:35:42.143 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2023-05-31T21:35:42.143 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /pklxkd-63/obj74 2023-05-31T21:35:42.144 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-05-31T21:35:42.144 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-05-31T21:35:42.144 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T21:35:42.144 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-05-31T21:35:42.145 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-05-31T21:35:42.145 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-05-31T21:35:42.145 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-05-31T21:35:42.145 INFO:tasks.rgw_multisite_tests:PUT 2023-05-31T21:35:42.146 INFO:tasks.rgw_multisite_tests:rL0Y20zC+Fzt72VPzMSk2A== 2023-05-31T21:35:42.146 INFO:tasks.rgw_multisite_tests:application/octet-stream 2023-05-31T21:35:42.146 INFO:tasks.rgw_multisite_tests:Wed, 31 May 2023 21:06:48 GMT 2023-05-31T21:35:42.146 INFO:tasks.rgw_multisite_tests:/pklxkd-63/obj74 2023-05-31T21:35:42.147 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-05-31T21:35:42.147 INFO:tasks.rgw_multisite_tests:AWS Q80H11OLNU1S48PPYL26:pOkmsj/3yR+lv0vPx7yWqoR4EsA= 2023-05-31T21:35:42.147 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Wed, 31 May 2023 21:06:48 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:pOkmsj/3yR+lv0vPx7yWqoR4EsA='} 2023-05-31T21:35:42.147 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000a23e22eeb1d8d80c-006477b6e8-1630-a1'), ('Date', 'Wed, 31 May 2023 21:06:48 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:35:42.147 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/pklxkd-63/ 2023-05-31T21:35:42.148 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/pklxkd-63/ 2023-05-31T21:35:42.148 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-05-31T21:35:42.148 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /pklxkd-63/ 2023-05-31T21:35:42.148 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-05-31T21:35:42.149 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-05-31T21:35:42.149 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T21:35:42.149 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-05-31T21:35:42.149 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-05-31T21:35:42.150 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-05-31T21:35:42.150 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-05-31T21:35:42.150 INFO:tasks.rgw_multisite_tests:HEAD 2023-05-31T21:35:42.150 INFO:tasks.rgw_multisite_tests: 2023-05-31T21:35:42.150 INFO:tasks.rgw_multisite_tests: 2023-05-31T21:35:42.151 INFO:tasks.rgw_multisite_tests:Wed, 31 May 2023 21:06:48 GMT 2023-05-31T21:35:42.151 INFO:tasks.rgw_multisite_tests:/pklxkd-63/ 2023-05-31T21:35:42.151 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-05-31T21:35:42.151 INFO:tasks.rgw_multisite_tests:AWS Q80H11OLNU1S48PPYL26:3B2mkaM+ObC4zRGdBhJqL6TchVg= 2023-05-31T21:35:42.152 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 21:06:48 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:3B2mkaM+ObC4zRGdBhJqL6TchVg=', 'Content-Length': '0'} 2023-05-31T21:35:42.152 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '32'), ('X-RGW-Bytes-Used', '96'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000000f247c948c4c519b-006477b6e8-1630-a1'), ('Content-Length', '0'), ('Date', 'Wed, 31 May 2023 21:06:48 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:35:42.152 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/pklxkd-63/obj111 2023-05-31T21:35:42.152 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/pklxkd-63/obj111 2023-05-31T21:35:42.153 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2023-05-31T21:35:42.153 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /pklxkd-63/obj111 2023-05-31T21:35:42.153 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-05-31T21:35:42.153 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-05-31T21:35:42.153 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T21:35:42.154 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-05-31T21:35:42.154 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-05-31T21:35:42.154 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-05-31T21:35:42.154 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-05-31T21:35:42.155 INFO:tasks.rgw_multisite_tests:PUT 2023-05-31T21:35:42.155 INFO:tasks.rgw_multisite_tests:rL0Y20zC+Fzt72VPzMSk2A== 2023-05-31T21:35:42.155 INFO:tasks.rgw_multisite_tests:application/octet-stream 2023-05-31T21:35:42.155 INFO:tasks.rgw_multisite_tests:Wed, 31 May 2023 21:06:48 GMT 2023-05-31T21:35:42.155 INFO:tasks.rgw_multisite_tests:/pklxkd-63/obj111 2023-05-31T21:35:42.156 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-05-31T21:35:42.156 INFO:tasks.rgw_multisite_tests:AWS Q80H11OLNU1S48PPYL26:yu9VqeI6cTKyuU+Vh1K/vXT2aDg= 2023-05-31T21:35:42.156 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Wed, 31 May 2023 21:06:48 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:yu9VqeI6cTKyuU+Vh1K/vXT2aDg='} 2023-05-31T21:35:42.156 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000000191aba87ef31b77-006477b6e8-1630-a1'), ('Date', 'Wed, 31 May 2023 21:06:48 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:35:42.157 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/pklxkd-63/ 2023-05-31T21:35:42.157 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/pklxkd-63/ 2023-05-31T21:35:42.157 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-05-31T21:35:42.157 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /pklxkd-63/ 2023-05-31T21:35:42.158 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-05-31T21:35:42.158 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-05-31T21:35:42.158 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T21:35:42.158 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-05-31T21:35:42.159 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-05-31T21:35:42.159 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-05-31T21:35:42.159 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-05-31T21:35:42.159 INFO:tasks.rgw_multisite_tests:HEAD 2023-05-31T21:35:42.159 INFO:tasks.rgw_multisite_tests: 2023-05-31T21:35:42.160 INFO:tasks.rgw_multisite_tests: 2023-05-31T21:35:42.160 INFO:tasks.rgw_multisite_tests:Wed, 31 May 2023 21:06:48 GMT 2023-05-31T21:35:42.160 INFO:tasks.rgw_multisite_tests:/pklxkd-63/ 2023-05-31T21:35:42.160 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-05-31T21:35:42.161 INFO:tasks.rgw_multisite_tests:AWS Q80H11OLNU1S48PPYL26:3B2mkaM+ObC4zRGdBhJqL6TchVg= 2023-05-31T21:35:42.161 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 21:06:48 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:3B2mkaM+ObC4zRGdBhJqL6TchVg=', 'Content-Length': '0'} 2023-05-31T21:35:42.161 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '33'), ('X-RGW-Bytes-Used', '99'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000e9c4c11d41b56b4d-006477b6e8-1630-a1'), ('Content-Length', '0'), ('Date', 'Wed, 31 May 2023 21:06:48 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:35:42.161 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/pklxkd-63/obj148 2023-05-31T21:35:42.161 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/pklxkd-63/obj148 2023-05-31T21:35:42.162 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2023-05-31T21:35:42.162 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /pklxkd-63/obj148 2023-05-31T21:35:42.162 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-05-31T21:35:42.162 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-05-31T21:35:42.163 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T21:35:42.163 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-05-31T21:35:42.163 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-05-31T21:35:42.163 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-05-31T21:35:42.164 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-05-31T21:35:42.164 INFO:tasks.rgw_multisite_tests:PUT 2023-05-31T21:35:42.164 INFO:tasks.rgw_multisite_tests:rL0Y20zC+Fzt72VPzMSk2A== 2023-05-31T21:35:42.164 INFO:tasks.rgw_multisite_tests:application/octet-stream 2023-05-31T21:35:42.164 INFO:tasks.rgw_multisite_tests:Wed, 31 May 2023 21:06:48 GMT 2023-05-31T21:35:42.165 INFO:tasks.rgw_multisite_tests:/pklxkd-63/obj148 2023-05-31T21:35:42.165 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-05-31T21:35:42.165 INFO:tasks.rgw_multisite_tests:AWS Q80H11OLNU1S48PPYL26:4pS9svMLwTYBqPd7WZVhl3vZBK4= 2023-05-31T21:35:42.165 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Wed, 31 May 2023 21:06:48 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:4pS9svMLwTYBqPd7WZVhl3vZBK4='} 2023-05-31T21:35:42.166 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000d40f8aa5c0746ffe-006477b6e8-1630-a1'), ('Date', 'Wed, 31 May 2023 21:06:48 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:35:42.166 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/pklxkd-63/ 2023-05-31T21:35:42.166 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/pklxkd-63/ 2023-05-31T21:35:42.166 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-05-31T21:35:42.167 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /pklxkd-63/ 2023-05-31T21:35:42.167 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-05-31T21:35:42.167 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-05-31T21:35:42.167 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T21:35:42.167 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-05-31T21:35:42.168 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-05-31T21:35:42.168 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-05-31T21:35:42.168 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-05-31T21:35:42.168 INFO:tasks.rgw_multisite_tests:HEAD 2023-05-31T21:35:42.169 INFO:tasks.rgw_multisite_tests: 2023-05-31T21:35:42.169 INFO:tasks.rgw_multisite_tests: 2023-05-31T21:35:42.169 INFO:tasks.rgw_multisite_tests:Wed, 31 May 2023 21:06:48 GMT 2023-05-31T21:35:42.169 INFO:tasks.rgw_multisite_tests:/pklxkd-63/ 2023-05-31T21:35:42.170 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-05-31T21:35:42.170 INFO:tasks.rgw_multisite_tests:AWS Q80H11OLNU1S48PPYL26:3B2mkaM+ObC4zRGdBhJqL6TchVg= 2023-05-31T21:35:42.170 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 21:06:48 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:3B2mkaM+ObC4zRGdBhJqL6TchVg=', 'Content-Length': '0'} 2023-05-31T21:35:42.170 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '34'), ('X-RGW-Bytes-Used', '102'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000094aac4135bb50c2c-006477b6e8-1630-a1'), ('Content-Length', '0'), ('Date', 'Wed, 31 May 2023 21:06:48 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:35:42.170 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/pklxkd-63/obj185 2023-05-31T21:35:42.171 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/pklxkd-63/obj185 2023-05-31T21:35:42.171 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2023-05-31T21:35:42.171 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /pklxkd-63/obj185 2023-05-31T21:35:42.171 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-05-31T21:35:42.172 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-05-31T21:35:42.172 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T21:35:42.172 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-05-31T21:35:42.172 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-05-31T21:35:42.173 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-05-31T21:35:42.173 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-05-31T21:35:42.173 INFO:tasks.rgw_multisite_tests:PUT 2023-05-31T21:35:42.173 INFO:tasks.rgw_multisite_tests:rL0Y20zC+Fzt72VPzMSk2A== 2023-05-31T21:35:42.173 INFO:tasks.rgw_multisite_tests:application/octet-stream 2023-05-31T21:35:42.174 INFO:tasks.rgw_multisite_tests:Wed, 31 May 2023 21:06:48 GMT 2023-05-31T21:35:42.174 INFO:tasks.rgw_multisite_tests:/pklxkd-63/obj185 2023-05-31T21:35:42.174 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-05-31T21:35:42.174 INFO:tasks.rgw_multisite_tests:AWS Q80H11OLNU1S48PPYL26:tYJpToOQdPMRyxyyp7JpYZOL5RA= 2023-05-31T21:35:42.175 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Wed, 31 May 2023 21:06:48 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:tYJpToOQdPMRyxyyp7JpYZOL5RA='} 2023-05-31T21:35:42.175 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000000476126fb590a8f4-006477b6e8-1630-a1'), ('Date', 'Wed, 31 May 2023 21:06:48 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:35:42.175 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: bucket checkpoint: source=a1 target=a2 bucket=pklxkd-63 2023-05-31T21:35:42.175 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'pklxkd-63', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-05-31T21:35:42.175 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'pklxkd-63', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-05-31T21:35:42.176 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket pklxkd-63 --source-zone a1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2023-05-31T21:35:42.176 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: ignoring --setuser ceph since I am not root 2023-05-31T21:35:42.176 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-05-31T21:35:42.176 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: 2023-05-31T21:06:48.553+0000 7f11993d0780 1 waiting to reach incremental sync.. 2023-05-31T21:35:42.177 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: 2023-05-31T21:06:53.556+0000 7f11993d0780 1 bucket sync caught up with source: 2023-05-31T21:35:42.177 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: local status: [, , , , , , 00000000002.94.6, , , , , , 00000000002.85.6, , , , , 00000000002.85.6, , , , , , , , , , , , , , , 00000000001.110.6, , , , 00000000001.83.6] 2023-05-31T21:35:42.177 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: remote markers: [, , , , , , 00000000002.94.6, , , , , , 00000000002.85.6, , , , , 00000000002.85.6, , , , , , , , , , , , , , , 00000000001.110.6, , , , 00000000001.83.6] 2023-05-31T21:35:42.177 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: 2023-05-31T21:06:53.556+0000 7f11993d0780 0 bucket checkpoint complete 2023-05-31T21:35:42.178 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: bucket checkpoint: source=a2 target=a1 bucket=pklxkd-63 2023-05-31T21:35:42.178 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'pklxkd-63', '--source-zone', 'a2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-05-31T21:35:42.178 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'pklxkd-63', '--source-zone', 'a2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-05-31T21:35:42.178 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket pklxkd-63 --source-zone a2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2023-05-31T21:35:42.179 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: ignoring --setuser ceph since I am not root 2023-05-31T21:35:42.179 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-05-31T21:35:42.179 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: 2023-05-31T21:06:53.675+0000 7fcadbe43780 1 waiting to reach incremental sync.. 2023-05-31T21:35:42.179 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: 2023-05-31T21:06:58.676+0000 7fcadbe43780 1 waiting to reach incremental sync.. 2023-05-31T21:35:42.179 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: 2023-05-31T21:07:03.677+0000 7fcadbe43780 1 waiting to reach incremental sync.. 2023-05-31T21:35:42.180 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: 2023-05-31T21:07:08.678+0000 7fcadbe43780 1 waiting to reach incremental sync.. 2023-05-31T21:35:42.180 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: 2023-05-31T21:07:13.679+0000 7fcadbe43780 1 waiting to reach incremental sync.. 2023-05-31T21:35:42.180 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: 2023-05-31T21:07:18.679+0000 7fcadbe43780 1 waiting to reach incremental sync.. 2023-05-31T21:35:42.180 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: 2023-05-31T21:07:23.680+0000 7fcadbe43780 1 waiting to reach incremental sync.. 2023-05-31T21:35:42.181 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: 2023-05-31T21:07:28.680+0000 7fcadbe43780 1 waiting to reach incremental sync.. 2023-05-31T21:35:42.181 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: 2023-05-31T21:07:33.680+0000 7fcadbe43780 1 waiting to reach incremental sync.. 2023-05-31T21:35:42.181 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: 2023-05-31T21:07:38.681+0000 7fcadbe43780 1 waiting to reach incremental sync.. 2023-05-31T21:35:42.181 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: 2023-05-31T21:07:43.682+0000 7fcadbe43780 1 waiting to reach incremental sync.. 2023-05-31T21:35:42.182 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: 2023-05-31T21:07:48.683+0000 7fcadbe43780 1 waiting to reach incremental sync.. 2023-05-31T21:35:42.182 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: 2023-05-31T21:07:53.684+0000 7fcadbe43780 1 waiting to reach incremental sync.. 2023-05-31T21:35:42.182 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: 2023-05-31T21:07:58.685+0000 7fcadbe43780 1 waiting to reach incremental sync.. 2023-05-31T21:35:42.182 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: 2023-05-31T21:08:03.685+0000 7fcadbe43780 1 waiting to reach incremental sync.. 2023-05-31T21:35:42.182 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: 2023-05-31T21:08:08.685+0000 7fcadbe43780 1 waiting to reach incremental sync.. 2023-05-31T21:35:42.183 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: 2023-05-31T21:08:13.686+0000 7fcadbe43780 1 waiting to reach incremental sync.. 2023-05-31T21:35:42.183 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: 2023-05-31T21:08:18.687+0000 7fcadbe43780 1 waiting to reach incremental sync.. 2023-05-31T21:35:42.183 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: 2023-05-31T21:08:23.687+0000 7fcadbe43780 1 waiting to reach incremental sync.. 2023-05-31T21:35:42.184 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: 2023-05-31T21:08:28.689+0000 7fcadbe43780 1 waiting to reach incremental sync.. 2023-05-31T21:35:42.184 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: 2023-05-31T21:08:33.690+0000 7fcadbe43780 1 waiting to reach incremental sync.. 2023-05-31T21:35:42.184 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: 2023-05-31T21:08:38.691+0000 7fcadbe43780 1 waiting to reach incremental sync.. 2023-05-31T21:35:42.184 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: 2023-05-31T21:08:43.691+0000 7fcadbe43780 1 waiting to reach incremental sync.. 2023-05-31T21:35:42.185 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: 2023-05-31T21:08:48.691+0000 7fcadbe43780 1 waiting to reach incremental sync.. 2023-05-31T21:35:42.185 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: 2023-05-31T21:08:53.692+0000 7fcadbe43780 1 waiting to reach incremental sync.. 2023-05-31T21:35:42.185 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: 2023-05-31T21:08:58.692+0000 7fcadbe43780 1 waiting to reach incremental sync.. 2023-05-31T21:35:42.185 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: 2023-05-31T21:09:03.693+0000 7fcadbe43780 1 waiting to reach incremental sync.. 2023-05-31T21:35:42.185 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: 2023-05-31T21:09:08.694+0000 7fcadbe43780 1 waiting to reach incremental sync.. 2023-05-31T21:35:42.186 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: 2023-05-31T21:09:13.695+0000 7fcadbe43780 1 waiting to reach incremental sync.. 2023-05-31T21:35:42.186 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: 2023-05-31T21:09:18.696+0000 7fcadbe43780 1 waiting to reach incremental sync.. 2023-05-31T21:35:42.186 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: 2023-05-31T21:09:23.697+0000 7fcadbe43780 1 waiting to reach incremental sync.. 2023-05-31T21:35:42.186 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: 2023-05-31T21:09:28.697+0000 7fcadbe43780 1 waiting to reach incremental sync.. 2023-05-31T21:35:42.187 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: 2023-05-31T21:09:33.697+0000 7fcadbe43780 1 waiting to reach incremental sync.. 2023-05-31T21:35:42.187 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: 2023-05-31T21:09:38.698+0000 7fcadbe43780 1 waiting to reach incremental sync.. 2023-05-31T21:35:42.187 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: 2023-05-31T21:09:43.698+0000 7fcadbe43780 1 waiting to reach incremental sync.. 2023-05-31T21:35:42.187 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: 2023-05-31T21:09:48.699+0000 7fcadbe43780 1 waiting to reach incremental sync.. 2023-05-31T21:35:42.188 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: 2023-05-31T21:09:53.700+0000 7fcadbe43780 1 waiting to reach incremental sync.. 2023-05-31T21:35:42.188 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: 2023-05-31T21:09:58.701+0000 7fcadbe43780 1 waiting to reach incremental sync.. 2023-05-31T21:35:42.188 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: 2023-05-31T21:10:03.702+0000 7fcadbe43780 1 waiting to reach incremental sync.. 2023-05-31T21:35:42.188 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: 2023-05-31T21:10:08.702+0000 7fcadbe43780 1 waiting to reach incremental sync.. 2023-05-31T21:35:42.188 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: 2023-05-31T21:10:13.702+0000 7fcadbe43780 1 waiting to reach incremental sync.. 2023-05-31T21:35:42.189 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: 2023-05-31T21:10:18.702+0000 7fcadbe43780 1 waiting to reach incremental sync.. 2023-05-31T21:35:42.189 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: 2023-05-31T21:10:23.703+0000 7fcadbe43780 1 waiting to reach incremental sync.. 2023-05-31T21:35:42.189 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: 2023-05-31T21:10:28.704+0000 7fcadbe43780 1 waiting to reach incremental sync.. 2023-05-31T21:35:42.189 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: 2023-05-31T21:10:33.705+0000 7fcadbe43780 1 waiting to reach incremental sync.. 2023-05-31T21:35:42.190 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: 2023-05-31T21:10:38.706+0000 7fcadbe43780 1 waiting to reach incremental sync.. 2023-05-31T21:35:42.190 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: 2023-05-31T21:10:43.707+0000 7fcadbe43780 1 waiting to reach incremental sync.. 2023-05-31T21:35:42.190 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: 2023-05-31T21:10:48.707+0000 7fcadbe43780 1 waiting to reach incremental sync.. 2023-05-31T21:35:42.190 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: 2023-05-31T21:10:53.708+0000 7fcadbe43780 1 waiting to reach incremental sync.. 2023-05-31T21:35:42.190 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: 2023-05-31T21:10:58.707+0000 7fcadbe43780 1 waiting to reach incremental sync.. 2023-05-31T21:35:42.191 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: 2023-05-31T21:11:03.708+0000 7fcadbe43780 1 waiting to reach incremental sync.. 2023-05-31T21:35:42.191 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: 2023-05-31T21:11:08.709+0000 7fcadbe43780 1 waiting to reach incremental sync.. 2023-05-31T21:35:42.191 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: 2023-05-31T21:11:13.710+0000 7fcadbe43780 1 waiting to reach incremental sync.. 2023-05-31T21:35:42.191 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: 2023-05-31T21:11:18.711+0000 7fcadbe43780 1 waiting to reach incremental sync.. 2023-05-31T21:35:42.192 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: 2023-05-31T21:11:23.712+0000 7fcadbe43780 1 waiting to reach incremental sync.. 2023-05-31T21:35:42.192 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: 2023-05-31T21:11:28.712+0000 7fcadbe43780 1 waiting to reach incremental sync.. 2023-05-31T21:35:42.192 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: 2023-05-31T21:11:33.713+0000 7fcadbe43780 1 waiting to reach incremental sync.. 2023-05-31T21:35:42.192 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: 2023-05-31T21:11:38.713+0000 7fcadbe43780 1 waiting to reach incremental sync.. 2023-05-31T21:35:42.193 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: 2023-05-31T21:11:43.714+0000 7fcadbe43780 1 waiting to reach incremental sync.. 2023-05-31T21:35:42.193 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: 2023-05-31T21:11:48.714+0000 7fcadbe43780 -1 bucket checkpoint timed out waiting to reach incremental sync 2023-05-31T21:35:42.193 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: 2023-05-31T21:11:48.714+0000 7fcadbe43780 0 bucket sync checkpoint failed: (110) Connection timed out 2023-05-31T21:35:42.193 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: 2023-05-31T21:11:48.714+0000 7fcadbe43780 -1 bucket sync checkpoint failed: (110) Connection timed out 2023-05-31T21:35:42.194 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run: DEBUG: got remote process result: 110 2023-05-31T21:35:42.194 INFO:tasks.rgw_multisite_tests:--------------------- >> end captured logging << --------------------- 2023-05-31T21:35:42.194 INFO:tasks.rgw_multisite_tests: 2023-05-31T21:35:42.194 INFO:tasks.rgw_multisite_tests:====================================================================== 2023-05-31T21:35:42.195 INFO:tasks.rgw_multisite_tests:FAIL: rgw_multi.tests.test_resharded_bucket_incremental_sync_oldest_after_data_sync_init 2023-05-31T21:35:42.195 INFO:tasks.rgw_multisite_tests:---------------------------------------------------------------------- 2023-05-31T21:35:42.196 INFO:tasks.rgw_multisite_tests:Traceback (most recent call last): 2023-05-31T21:35:42.196 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_teuthology_fa97ddd17d12e6d74b49196948f66e54fa009130/virtualenv/lib/python3.8/site-packages/nose/case.py", line 198, in runTest 2023-05-31T21:35:42.196 INFO:tasks.rgw_multisite_tests: self.test(*self.arg) 2023-05-31T21:35:42.196 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/github.com_ceph_ceph-c_2870bf456d3115490c153abce9d47ac5c043b65e/qa/../src/test/rgw/rgw_multi/tests.py", line 1904, in test_resharded_bucket_incremental_sync_oldest_after_data_sync_init 2023-05-31T21:35:42.197 INFO:tasks.rgw_multisite_tests: zonegroup_bucket_checkpoint(zonegroup_conns, bucket.name) 2023-05-31T21:35:42.197 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/github.com_ceph_ceph-c_2870bf456d3115490c153abce9d47ac5c043b65e/qa/../src/test/rgw/rgw_multi/tests.py", line 395, in zonegroup_bucket_checkpoint 2023-05-31T21:35:42.197 INFO:tasks.rgw_multisite_tests: zone_bucket_checkpoint(target_conn.zone, source_conn.zone, bucket_name) 2023-05-31T21:35:42.197 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/github.com_ceph_ceph-c_2870bf456d3115490c153abce9d47ac5c043b65e/qa/../src/test/rgw/rgw_multi/tests.py", line 387, in zone_bucket_checkpoint 2023-05-31T21:35:42.198 INFO:tasks.rgw_multisite_tests: target_zone.cluster.admin(cmd, debug_rgw=1) 2023-05-31T21:35:42.198 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/github.com_ceph_ceph-c_2870bf456d3115490c153abce9d47ac5c043b65e/qa/tasks/rgw_multisite.py", line 197, in admin 2023-05-31T21:35:42.198 INFO:tasks.rgw_multisite_tests: assert r == 0 2023-05-31T21:35:42.198 INFO:tasks.rgw_multisite_tests:AssertionError: 2023-05-31T21:35:42.199 INFO:tasks.rgw_multisite_tests:-------------------- >> begin captured logging << -------------------- 2023-05-31T21:35:42.199 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/pklxkd-64/ 2023-05-31T21:35:42.199 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/pklxkd-64/ 2023-05-31T21:35:42.199 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2023-05-31T21:35:42.200 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /pklxkd-64/ 2023-05-31T21:35:42.200 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-05-31T21:35:42.200 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-05-31T21:35:42.200 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T21:35:42.200 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-05-31T21:35:42.201 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-05-31T21:35:42.201 INFO:tasks.rgw_multisite_tests:boto: DEBUG: establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2023-05-31T21:35:42.201 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-05-31T21:35:42.201 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-05-31T21:35:42.202 INFO:tasks.rgw_multisite_tests:PUT 2023-05-31T21:35:42.202 INFO:tasks.rgw_multisite_tests: 2023-05-31T21:35:42.202 INFO:tasks.rgw_multisite_tests: 2023-05-31T21:35:42.202 INFO:tasks.rgw_multisite_tests:Wed, 31 May 2023 21:11:48 GMT 2023-05-31T21:35:42.203 INFO:tasks.rgw_multisite_tests:/pklxkd-64/ 2023-05-31T21:35:42.203 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-05-31T21:35:42.203 INFO:tasks.rgw_multisite_tests:AWS Q80H11OLNU1S48PPYL26:u3CbdAOAJ5dYMZFBGrKRRaPxL5U= 2023-05-31T21:35:42.203 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 21:11:48 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:u3CbdAOAJ5dYMZFBGrKRRaPxL5U=', 'Content-Length': '0'} 2023-05-31T21:35:42.203 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('x-amz-request-id', 'tx00000ce71a9bd56ba6d86-006477b814-1630-a1'), ('Content-Length', '0'), ('Date', 'Wed, 31 May 2023 21:11:48 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:35:42.204 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: created bucket=pklxkd-64 2023-05-31T21:35:42.204 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-31T21:35:42.204 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-31T21:35:42.204 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-05-31T21:35:42.205 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: ignoring --setuser ceph since I am not root 2023-05-31T21:35:42.205 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-05-31T21:35:42.205 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: No --period given, using current period=2e6123d2-d695-4350-9a01-61a455bc782b 2023-05-31T21:35:42.205 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: [ 2023-05-31T21:35:42.206 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2023-05-31T21:35:42.206 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "marker": "1_1685566501.757183_260.1", 2023-05-31T21:35:42.206 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "last_update": "2023-05-31T20:55:01.757183Z" 2023-05-31T21:35:42.206 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2023-05-31T21:35:42.206 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2023-05-31T21:35:42.207 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "marker": "1_1685566501.735717_251.1", 2023-05-31T21:35:42.207 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "last_update": "2023-05-31T20:55:01.735717Z" 2023-05-31T21:35:42.207 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2023-05-31T21:35:42.207 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2023-05-31T21:35:42.208 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "marker": "1_1685562970.274572_93.1", 2023-05-31T21:35:42.208 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "last_update": "2023-05-31T19:56:10.274572Z" 2023-05-31T21:35:42.208 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2023-05-31T21:35:42.208 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2023-05-31T21:35:42.209 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "marker": "1_1685567508.772581_734.1", 2023-05-31T21:35:42.209 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "last_update": "2023-05-31T21:11:48.772581Z" 2023-05-31T21:35:42.209 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2023-05-31T21:35:42.209 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: ] 2023-05-31T21:35:42.209 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: master meta markers={0: '1_1685566501.757183_260.1', 1: '1_1685566501.735717_251.1', 2: '1_1685562970.274572_93.1', 3: '1_1685567508.772581_734.1'} 2023-05-31T21:35:42.210 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: INFO: starting meta checkpoint for zone=a2 2023-05-31T21:35:42.210 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-31T21:35:42.210 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-31T21:35:42.210 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-05-31T21:35:42.211 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: ignoring --setuser ceph since I am not root 2023-05-31T21:35:42.211 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-05-31T21:35:42.211 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: 2023-05-31T21:11:48.987+0000 7f5598d45780 20 RGW-SYNC:meta: read sync status 2023-05-31T21:35:42.211 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: 2023-05-31T21:11:48.987+0000 7f5598d45780 20 RGW-SYNC:meta: read sync status 2023-05-31T21:35:42.212 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2023-05-31T21:35:42.212 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "sync_status": { 2023-05-31T21:35:42.212 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "info": { 2023-05-31T21:35:42.212 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "status": "sync", 2023-05-31T21:35:42.212 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "num_shards": 4, 2023-05-31T21:35:42.213 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "period": "2e6123d2-d695-4350-9a01-61a455bc782b", 2023-05-31T21:35:42.213 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "realm_epoch": 2 2023-05-31T21:35:42.213 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2023-05-31T21:35:42.213 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "markers": [ 2023-05-31T21:35:42.214 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2023-05-31T21:35:42.214 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "key": 0, 2023-05-31T21:35:42.214 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "val": { 2023-05-31T21:35:42.214 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "state": 1, 2023-05-31T21:35:42.215 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "marker": "1_1685566501.757183_260.1", 2023-05-31T21:35:42.215 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "next_step_marker": "", 2023-05-31T21:35:42.215 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "total_entries": 1, 2023-05-31T21:35:42.215 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "pos": 0, 2023-05-31T21:35:42.215 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "timestamp": "2023-05-31T20:55:01.757183Z", 2023-05-31T21:35:42.216 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "realm_epoch": 2 2023-05-31T21:35:42.216 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2023-05-31T21:35:42.216 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2023-05-31T21:35:42.216 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2023-05-31T21:35:42.217 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "key": 1, 2023-05-31T21:35:42.217 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "val": { 2023-05-31T21:35:42.217 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "state": 1, 2023-05-31T21:35:42.217 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "marker": "1_1685566501.735717_251.1", 2023-05-31T21:35:42.218 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "next_step_marker": "", 2023-05-31T21:35:42.218 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "total_entries": 0, 2023-05-31T21:35:42.218 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "pos": 0, 2023-05-31T21:35:42.218 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "timestamp": "2023-05-31T20:55:01.735717Z", 2023-05-31T21:35:42.218 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "realm_epoch": 2 2023-05-31T21:35:42.219 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2023-05-31T21:35:42.219 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2023-05-31T21:35:42.219 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2023-05-31T21:35:42.219 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "key": 2, 2023-05-31T21:35:42.220 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "val": { 2023-05-31T21:35:42.220 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "state": 1, 2023-05-31T21:35:42.220 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "marker": "1_1685562970.274572_93.1", 2023-05-31T21:35:42.220 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "next_step_marker": "", 2023-05-31T21:35:42.221 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "total_entries": 0, 2023-05-31T21:35:42.221 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "pos": 0, 2023-05-31T21:35:42.221 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "timestamp": "2023-05-31T19:56:10.274572Z", 2023-05-31T21:35:42.221 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "realm_epoch": 2 2023-05-31T21:35:42.221 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2023-05-31T21:35:42.222 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2023-05-31T21:35:42.222 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2023-05-31T21:35:42.222 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "key": 3, 2023-05-31T21:35:42.223 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "val": { 2023-05-31T21:35:42.223 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "state": 1, 2023-05-31T21:35:42.223 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "marker": "1_1685567508.772581_734.1", 2023-05-31T21:35:42.223 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "next_step_marker": "", 2023-05-31T21:35:42.224 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "total_entries": 0, 2023-05-31T21:35:42.224 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "pos": 0, 2023-05-31T21:35:42.224 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "timestamp": "2023-05-31T21:11:48.772581Z", 2023-05-31T21:35:42.224 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "realm_epoch": 2 2023-05-31T21:35:42.224 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2023-05-31T21:35:42.225 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2023-05-31T21:35:42.225 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: ] 2023-05-31T21:35:42.225 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2023-05-31T21:35:42.225 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "full_sync": { 2023-05-31T21:35:42.226 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "total": 1, 2023-05-31T21:35:42.226 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "complete": 1 2023-05-31T21:35:42.226 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2023-05-31T21:35:42.226 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "current_time": "2023-05-31T21:11:48Z" 2023-05-31T21:35:42.227 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2023-05-31T21:35:42.227 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2023-05-31T21:35:42.227 INFO:tasks.rgw_multisite_tests: "sync_status": { 2023-05-31T21:35:42.227 INFO:tasks.rgw_multisite_tests: "info": { 2023-05-31T21:35:42.227 INFO:tasks.rgw_multisite_tests: "status": "sync", 2023-05-31T21:35:42.228 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2023-05-31T21:35:42.228 INFO:tasks.rgw_multisite_tests: "period": "2e6123d2-d695-4350-9a01-61a455bc782b", 2023-05-31T21:35:42.228 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-05-31T21:35:42.228 INFO:tasks.rgw_multisite_tests: }, 2023-05-31T21:35:42.229 INFO:tasks.rgw_multisite_tests: "markers": [ 2023-05-31T21:35:42.229 INFO:tasks.rgw_multisite_tests: { 2023-05-31T21:35:42.229 INFO:tasks.rgw_multisite_tests: "key": 0, 2023-05-31T21:35:42.229 INFO:tasks.rgw_multisite_tests: "val": { 2023-05-31T21:35:42.230 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-05-31T21:35:42.230 INFO:tasks.rgw_multisite_tests: "marker": "1_1685566501.757183_260.1", 2023-05-31T21:35:42.230 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-05-31T21:35:42.230 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2023-05-31T21:35:42.230 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-05-31T21:35:42.231 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-05-31T20:55:01.757183Z", 2023-05-31T21:35:42.231 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-05-31T21:35:42.231 INFO:tasks.rgw_multisite_tests: } 2023-05-31T21:35:42.231 INFO:tasks.rgw_multisite_tests: }, 2023-05-31T21:35:42.232 INFO:tasks.rgw_multisite_tests: { 2023-05-31T21:35:42.232 INFO:tasks.rgw_multisite_tests: "key": 1, 2023-05-31T21:35:42.232 INFO:tasks.rgw_multisite_tests: "val": { 2023-05-31T21:35:42.232 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-05-31T21:35:42.233 INFO:tasks.rgw_multisite_tests: "marker": "1_1685566501.735717_251.1", 2023-05-31T21:35:42.233 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-05-31T21:35:42.233 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-05-31T21:35:42.233 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-05-31T21:35:42.234 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-05-31T20:55:01.735717Z", 2023-05-31T21:35:42.234 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-05-31T21:35:42.234 INFO:tasks.rgw_multisite_tests: } 2023-05-31T21:35:42.234 INFO:tasks.rgw_multisite_tests: }, 2023-05-31T21:35:42.234 INFO:tasks.rgw_multisite_tests: { 2023-05-31T21:35:42.235 INFO:tasks.rgw_multisite_tests: "key": 2, 2023-05-31T21:35:42.235 INFO:tasks.rgw_multisite_tests: "val": { 2023-05-31T21:35:42.235 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-05-31T21:35:42.235 INFO:tasks.rgw_multisite_tests: "marker": "1_1685562970.274572_93.1", 2023-05-31T21:35:42.236 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-05-31T21:35:42.236 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-05-31T21:35:42.236 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-05-31T21:35:42.236 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-05-31T19:56:10.274572Z", 2023-05-31T21:35:42.237 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-05-31T21:35:42.237 INFO:tasks.rgw_multisite_tests: } 2023-05-31T21:35:42.237 INFO:tasks.rgw_multisite_tests: }, 2023-05-31T21:35:42.237 INFO:tasks.rgw_multisite_tests: { 2023-05-31T21:35:42.237 INFO:tasks.rgw_multisite_tests: "key": 3, 2023-05-31T21:35:42.238 INFO:tasks.rgw_multisite_tests: "val": { 2023-05-31T21:35:42.238 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-05-31T21:35:42.238 INFO:tasks.rgw_multisite_tests: "marker": "1_1685567508.772581_734.1", 2023-05-31T21:35:42.238 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-05-31T21:35:42.239 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-05-31T21:35:42.239 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-05-31T21:35:42.239 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-05-31T21:11:48.772581Z", 2023-05-31T21:35:42.239 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-05-31T21:35:42.240 INFO:tasks.rgw_multisite_tests: } 2023-05-31T21:35:42.240 INFO:tasks.rgw_multisite_tests: } 2023-05-31T21:35:42.240 INFO:tasks.rgw_multisite_tests: ] 2023-05-31T21:35:42.240 INFO:tasks.rgw_multisite_tests: }, 2023-05-31T21:35:42.240 INFO:tasks.rgw_multisite_tests: "full_sync": { 2023-05-31T21:35:42.241 INFO:tasks.rgw_multisite_tests: "total": 1, 2023-05-31T21:35:42.241 INFO:tasks.rgw_multisite_tests: "complete": 1 2023-05-31T21:35:42.241 INFO:tasks.rgw_multisite_tests: }, 2023-05-31T21:35:42.241 INFO:tasks.rgw_multisite_tests: "current_time": "2023-05-31T21:11:48Z" 2023-05-31T21:35:42.242 INFO:tasks.rgw_multisite_tests:} 2023-05-31T21:35:42.242 INFO:tasks.rgw_multisite_tests: 2023-05-31T21:35:42.242 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1685566501.757183_260.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-05-31T20:55:01.757183Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1685566501.735717_251.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-05-31T20:55:01.735717Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1685562970.274572_93.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-05-31T19:56:10.274572Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1685567508.772581_734.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-05-31T21:11:48.772581Z', 'realm_epoch': 2}}] 2023-05-31T21:35:42.242 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1685566501.757183_260.1', 1: '1_1685566501.735717_251.1', 2: '1_1685562970.274572_93.1', 3: '1_1685567508.772581_734.1'} 2023-05-31T21:35:42.243 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1685566501.757183_260.1', 1: '1_1685566501.735717_251.1', 2: '1_1685562970.274572_93.1', 3: '1_1685567508.772581_734.1'} 2023-05-31T21:35:42.243 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: INFO: finish meta checkpoint for zone=a2 2023-05-31T21:35:42.243 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/pklxkd-64/ 2023-05-31T21:35:42.243 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/pklxkd-64/ 2023-05-31T21:35:42.243 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-05-31T21:35:42.244 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /pklxkd-64/ 2023-05-31T21:35:42.244 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-05-31T21:35:42.244 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-05-31T21:35:42.244 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T21:35:42.245 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-05-31T21:35:42.245 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-05-31T21:35:42.245 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-05-31T21:35:42.245 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-05-31T21:35:42.246 INFO:tasks.rgw_multisite_tests:HEAD 2023-05-31T21:35:42.246 INFO:tasks.rgw_multisite_tests: 2023-05-31T21:35:42.246 INFO:tasks.rgw_multisite_tests: 2023-05-31T21:35:42.246 INFO:tasks.rgw_multisite_tests:Wed, 31 May 2023 21:11:49 GMT 2023-05-31T21:35:42.246 INFO:tasks.rgw_multisite_tests:/pklxkd-64/ 2023-05-31T21:35:42.247 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-05-31T21:35:42.247 INFO:tasks.rgw_multisite_tests:AWS Q80H11OLNU1S48PPYL26:WtQF3N8/UJADUP+Xy7BD1Fmkr/U= 2023-05-31T21:35:42.247 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 21:11:49 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:WtQF3N8/UJADUP+Xy7BD1Fmkr/U=', 'Content-Length': '0'} 2023-05-31T21:35:42.247 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000037c1cb85c92b2339-006477b815-1630-a1'), ('Content-Length', '0'), ('Date', 'Wed, 31 May 2023 21:11:49 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:35:42.248 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/pklxkd-64/obj11 2023-05-31T21:35:42.248 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/pklxkd-64/obj11 2023-05-31T21:35:42.248 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2023-05-31T21:35:42.248 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /pklxkd-64/obj11 2023-05-31T21:35:42.249 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-05-31T21:35:42.249 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-05-31T21:35:42.249 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T21:35:42.249 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-05-31T21:35:42.250 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-05-31T21:35:42.250 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-05-31T21:35:42.250 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-05-31T21:35:42.250 INFO:tasks.rgw_multisite_tests:PUT 2023-05-31T21:35:42.250 INFO:tasks.rgw_multisite_tests:rL0Y20zC+Fzt72VPzMSk2A== 2023-05-31T21:35:42.251 INFO:tasks.rgw_multisite_tests:application/octet-stream 2023-05-31T21:35:42.251 INFO:tasks.rgw_multisite_tests:Wed, 31 May 2023 21:11:49 GMT 2023-05-31T21:35:42.251 INFO:tasks.rgw_multisite_tests:/pklxkd-64/obj11 2023-05-31T21:35:42.251 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-05-31T21:35:42.252 INFO:tasks.rgw_multisite_tests:AWS Q80H11OLNU1S48PPYL26:tHsfNYdEXhvC6oJHxPZcU7ntuvA= 2023-05-31T21:35:42.252 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Wed, 31 May 2023 21:11:49 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:tHsfNYdEXhvC6oJHxPZcU7ntuvA='} 2023-05-31T21:35:42.252 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000c126649dfa9904a2-006477b815-1630-a1'), ('Date', 'Wed, 31 May 2023 21:11:49 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:35:42.252 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/pklxkd-64/ 2023-05-31T21:35:42.253 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/pklxkd-64/ 2023-05-31T21:35:42.253 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-05-31T21:35:42.253 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /pklxkd-64/ 2023-05-31T21:35:42.253 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-05-31T21:35:42.253 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-05-31T21:35:42.254 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T21:35:42.254 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-05-31T21:35:42.254 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-05-31T21:35:42.254 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-05-31T21:35:42.255 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-05-31T21:35:42.255 INFO:tasks.rgw_multisite_tests:HEAD 2023-05-31T21:35:42.255 INFO:tasks.rgw_multisite_tests: 2023-05-31T21:35:42.255 INFO:tasks.rgw_multisite_tests: 2023-05-31T21:35:42.255 INFO:tasks.rgw_multisite_tests:Wed, 31 May 2023 21:11:49 GMT 2023-05-31T21:35:42.256 INFO:tasks.rgw_multisite_tests:/pklxkd-64/ 2023-05-31T21:35:42.256 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-05-31T21:35:42.256 INFO:tasks.rgw_multisite_tests:AWS Q80H11OLNU1S48PPYL26:WtQF3N8/UJADUP+Xy7BD1Fmkr/U= 2023-05-31T21:35:42.256 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 21:11:49 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:WtQF3N8/UJADUP+Xy7BD1Fmkr/U=', 'Content-Length': '0'} 2023-05-31T21:35:42.257 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '3'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000013b39f071c4ce2e1-006477b815-1630-a1'), ('Content-Length', '0'), ('Date', 'Wed, 31 May 2023 21:11:49 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:35:42.257 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/pklxkd-64/obj22 2023-05-31T21:35:42.257 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/pklxkd-64/obj22 2023-05-31T21:35:42.257 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2023-05-31T21:35:42.258 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /pklxkd-64/obj22 2023-05-31T21:35:42.258 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-05-31T21:35:42.258 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-05-31T21:35:42.258 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T21:35:42.258 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-05-31T21:35:42.259 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-05-31T21:35:42.259 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-05-31T21:35:42.259 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-05-31T21:35:42.259 INFO:tasks.rgw_multisite_tests:PUT 2023-05-31T21:35:42.260 INFO:tasks.rgw_multisite_tests:rL0Y20zC+Fzt72VPzMSk2A== 2023-05-31T21:35:42.260 INFO:tasks.rgw_multisite_tests:application/octet-stream 2023-05-31T21:35:42.260 INFO:tasks.rgw_multisite_tests:Wed, 31 May 2023 21:11:49 GMT 2023-05-31T21:35:42.260 INFO:tasks.rgw_multisite_tests:/pklxkd-64/obj22 2023-05-31T21:35:42.261 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-05-31T21:35:42.261 INFO:tasks.rgw_multisite_tests:AWS Q80H11OLNU1S48PPYL26:kQd2RcTkswxSAzA+gLkxtE1EuHs= 2023-05-31T21:35:42.261 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Wed, 31 May 2023 21:11:49 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:kQd2RcTkswxSAzA+gLkxtE1EuHs='} 2023-05-31T21:35:42.261 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000031eb0e03361e4c5d-006477b815-1630-a1'), ('Date', 'Wed, 31 May 2023 21:11:49 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:35:42.261 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/pklxkd-64/ 2023-05-31T21:35:42.262 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/pklxkd-64/ 2023-05-31T21:35:42.262 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-05-31T21:35:42.262 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /pklxkd-64/ 2023-05-31T21:35:42.262 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-05-31T21:35:42.263 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-05-31T21:35:42.263 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T21:35:42.263 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-05-31T21:35:42.263 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-05-31T21:35:42.264 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-05-31T21:35:42.264 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-05-31T21:35:42.264 INFO:tasks.rgw_multisite_tests:HEAD 2023-05-31T21:35:42.264 INFO:tasks.rgw_multisite_tests: 2023-05-31T21:35:42.264 INFO:tasks.rgw_multisite_tests: 2023-05-31T21:35:42.265 INFO:tasks.rgw_multisite_tests:Wed, 31 May 2023 21:11:49 GMT 2023-05-31T21:35:42.265 INFO:tasks.rgw_multisite_tests:/pklxkd-64/ 2023-05-31T21:35:42.265 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-05-31T21:35:42.265 INFO:tasks.rgw_multisite_tests:AWS Q80H11OLNU1S48PPYL26:WtQF3N8/UJADUP+Xy7BD1Fmkr/U= 2023-05-31T21:35:42.266 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 21:11:49 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:WtQF3N8/UJADUP+Xy7BD1Fmkr/U=', 'Content-Length': '0'} 2023-05-31T21:35:42.266 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '2'), ('X-RGW-Bytes-Used', '6'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000b5ce5a33a5d1350f-006477b815-1630-a1'), ('Content-Length', '0'), ('Date', 'Wed, 31 May 2023 21:11:49 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:35:42.266 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/pklxkd-64/obj33 2023-05-31T21:35:42.266 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/pklxkd-64/obj33 2023-05-31T21:35:42.267 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2023-05-31T21:35:42.267 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /pklxkd-64/obj33 2023-05-31T21:35:42.267 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-05-31T21:35:42.267 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-05-31T21:35:42.267 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T21:35:42.268 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-05-31T21:35:42.268 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-05-31T21:35:42.268 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-05-31T21:35:42.268 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-05-31T21:35:42.269 INFO:tasks.rgw_multisite_tests:PUT 2023-05-31T21:35:42.269 INFO:tasks.rgw_multisite_tests:rL0Y20zC+Fzt72VPzMSk2A== 2023-05-31T21:35:42.269 INFO:tasks.rgw_multisite_tests:application/octet-stream 2023-05-31T21:35:42.269 INFO:tasks.rgw_multisite_tests:Wed, 31 May 2023 21:11:49 GMT 2023-05-31T21:35:42.270 INFO:tasks.rgw_multisite_tests:/pklxkd-64/obj33 2023-05-31T21:35:42.270 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-05-31T21:35:42.270 INFO:tasks.rgw_multisite_tests:AWS Q80H11OLNU1S48PPYL26:NXz3r2yvS8eyJWDa6bNsUS9Ln3o= 2023-05-31T21:35:42.270 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Wed, 31 May 2023 21:11:49 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:NXz3r2yvS8eyJWDa6bNsUS9Ln3o='} 2023-05-31T21:35:42.270 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000f24787bd2326be48-006477b815-1630-a1'), ('Date', 'Wed, 31 May 2023 21:11:49 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:35:42.271 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/pklxkd-64/ 2023-05-31T21:35:42.271 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/pklxkd-64/ 2023-05-31T21:35:42.271 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-05-31T21:35:42.271 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /pklxkd-64/ 2023-05-31T21:35:42.272 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-05-31T21:35:42.272 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-05-31T21:35:42.272 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T21:35:42.272 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-05-31T21:35:42.273 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-05-31T21:35:42.273 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-05-31T21:35:42.273 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-05-31T21:35:42.273 INFO:tasks.rgw_multisite_tests:HEAD 2023-05-31T21:35:42.273 INFO:tasks.rgw_multisite_tests: 2023-05-31T21:35:42.274 INFO:tasks.rgw_multisite_tests: 2023-05-31T21:35:42.274 INFO:tasks.rgw_multisite_tests:Wed, 31 May 2023 21:11:49 GMT 2023-05-31T21:35:42.274 INFO:tasks.rgw_multisite_tests:/pklxkd-64/ 2023-05-31T21:35:42.274 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-05-31T21:35:42.275 INFO:tasks.rgw_multisite_tests:AWS Q80H11OLNU1S48PPYL26:WtQF3N8/UJADUP+Xy7BD1Fmkr/U= 2023-05-31T21:35:42.275 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 21:11:49 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:WtQF3N8/UJADUP+Xy7BD1Fmkr/U=', 'Content-Length': '0'} 2023-05-31T21:35:42.275 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '3'), ('X-RGW-Bytes-Used', '9'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000093a7e10596af3f16-006477b815-1630-a1'), ('Content-Length', '0'), ('Date', 'Wed, 31 May 2023 21:11:49 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:35:42.275 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/pklxkd-64/obj44 2023-05-31T21:35:42.276 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/pklxkd-64/obj44 2023-05-31T21:35:42.276 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2023-05-31T21:35:42.276 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /pklxkd-64/obj44 2023-05-31T21:35:42.276 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-05-31T21:35:42.277 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-05-31T21:35:42.277 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T21:35:42.277 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-05-31T21:35:42.277 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-05-31T21:35:42.277 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-05-31T21:35:42.278 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-05-31T21:35:42.278 INFO:tasks.rgw_multisite_tests:PUT 2023-05-31T21:35:42.278 INFO:tasks.rgw_multisite_tests:rL0Y20zC+Fzt72VPzMSk2A== 2023-05-31T21:35:42.278 INFO:tasks.rgw_multisite_tests:application/octet-stream 2023-05-31T21:35:42.279 INFO:tasks.rgw_multisite_tests:Wed, 31 May 2023 21:11:49 GMT 2023-05-31T21:35:42.279 INFO:tasks.rgw_multisite_tests:/pklxkd-64/obj44 2023-05-31T21:35:42.279 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-05-31T21:35:42.279 INFO:tasks.rgw_multisite_tests:AWS Q80H11OLNU1S48PPYL26:o+D0PaoSF0fl0zqZh7AruwVEg0Y= 2023-05-31T21:35:42.279 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Wed, 31 May 2023 21:11:49 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:o+D0PaoSF0fl0zqZh7AruwVEg0Y='} 2023-05-31T21:35:42.280 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000008839e63d01901be8-006477b815-1630-a1'), ('Date', 'Wed, 31 May 2023 21:11:49 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:35:42.280 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/pklxkd-64/ 2023-05-31T21:35:42.280 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/pklxkd-64/ 2023-05-31T21:35:42.280 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-05-31T21:35:42.281 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /pklxkd-64/ 2023-05-31T21:35:42.281 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-05-31T21:35:42.281 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-05-31T21:35:42.281 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T21:35:42.282 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-05-31T21:35:42.282 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-05-31T21:35:42.282 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-05-31T21:35:42.282 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-05-31T21:35:42.282 INFO:tasks.rgw_multisite_tests:HEAD 2023-05-31T21:35:42.283 INFO:tasks.rgw_multisite_tests: 2023-05-31T21:35:42.283 INFO:tasks.rgw_multisite_tests: 2023-05-31T21:35:42.283 INFO:tasks.rgw_multisite_tests:Wed, 31 May 2023 21:11:49 GMT 2023-05-31T21:35:42.283 INFO:tasks.rgw_multisite_tests:/pklxkd-64/ 2023-05-31T21:35:42.284 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-05-31T21:35:42.284 INFO:tasks.rgw_multisite_tests:AWS Q80H11OLNU1S48PPYL26:WtQF3N8/UJADUP+Xy7BD1Fmkr/U= 2023-05-31T21:35:42.284 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 21:11:49 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:WtQF3N8/UJADUP+Xy7BD1Fmkr/U=', 'Content-Length': '0'} 2023-05-31T21:35:42.285 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '4'), ('X-RGW-Bytes-Used', '12'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000bca2f544eaa003a0-006477b815-1630-a1'), ('Content-Length', '0'), ('Date', 'Wed, 31 May 2023 21:11:49 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:35:42.285 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/pklxkd-64/obj55 2023-05-31T21:35:42.285 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/pklxkd-64/obj55 2023-05-31T21:35:42.285 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2023-05-31T21:35:42.285 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /pklxkd-64/obj55 2023-05-31T21:35:42.286 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-05-31T21:35:42.286 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-05-31T21:35:42.286 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T21:35:42.286 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-05-31T21:35:42.287 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-05-31T21:35:42.287 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-05-31T21:35:42.287 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-05-31T21:35:42.287 INFO:tasks.rgw_multisite_tests:PUT 2023-05-31T21:35:42.288 INFO:tasks.rgw_multisite_tests:rL0Y20zC+Fzt72VPzMSk2A== 2023-05-31T21:35:42.288 INFO:tasks.rgw_multisite_tests:application/octet-stream 2023-05-31T21:35:42.288 INFO:tasks.rgw_multisite_tests:Wed, 31 May 2023 21:11:49 GMT 2023-05-31T21:35:42.288 INFO:tasks.rgw_multisite_tests:/pklxkd-64/obj55 2023-05-31T21:35:42.288 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-05-31T21:35:42.289 INFO:tasks.rgw_multisite_tests:AWS Q80H11OLNU1S48PPYL26:EKkHwPtPXSjrCISLwfCrrx0vGpw= 2023-05-31T21:35:42.289 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Wed, 31 May 2023 21:11:49 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:EKkHwPtPXSjrCISLwfCrrx0vGpw='} 2023-05-31T21:35:42.289 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000087e07f1827f05bda-006477b815-1630-a1'), ('Date', 'Wed, 31 May 2023 21:11:49 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:35:42.289 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: bucket checkpoint: source=a1 target=a2 bucket=pklxkd-64 2023-05-31T21:35:42.290 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'pklxkd-64', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-05-31T21:35:42.290 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'pklxkd-64', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-05-31T21:35:42.290 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket pklxkd-64 --source-zone a1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2023-05-31T21:35:42.290 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: ignoring --setuser ceph since I am not root 2023-05-31T21:35:42.291 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-05-31T21:35:42.291 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: 2023-05-31T21:11:49.242+0000 7f1c4cdc1780 1 waiting to reach incremental sync.. 2023-05-31T21:35:42.291 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: 2023-05-31T21:11:54.243+0000 7f1c4cdc1780 1 waiting to reach incremental sync.. 2023-05-31T21:35:42.291 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: 2023-05-31T21:11:59.244+0000 7f1c4cdc1780 1 bucket sync caught up with source: 2023-05-31T21:35:42.292 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: local status: [, , , , , , , , 00000000001.133.6, 00000000002.139.6, 00000000002.101.6] 2023-05-31T21:35:42.292 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: remote markers: [, , , , , , , , 00000000001.133.6, 00000000002.139.6, 00000000002.101.6] 2023-05-31T21:35:42.292 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: 2023-05-31T21:11:59.244+0000 7f1c4cdc1780 0 bucket checkpoint complete 2023-05-31T21:35:42.292 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: bucket checkpoint: source=a2 target=a1 bucket=pklxkd-64 2023-05-31T21:35:42.292 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'pklxkd-64', '--source-zone', 'a2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-05-31T21:35:42.293 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'pklxkd-64', '--source-zone', 'a2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-05-31T21:35:42.293 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket pklxkd-64 --source-zone a2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2023-05-31T21:35:42.293 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: ignoring --setuser ceph since I am not root 2023-05-31T21:35:42.293 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-05-31T21:35:42.294 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: 2023-05-31T21:11:59.380+0000 7f229ef2c780 1 waiting for incremental sync to catch up: 2023-05-31T21:35:42.294 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: local status: [, , , , , , , , 00000000001.1365.6, 00000000002.3947.6, 00000000001.2023.6] 2023-05-31T21:35:42.294 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: remote markers: [, , , , , , , , 00000000001.1365.6, 00000000002.3947.6, 00000000002.2025.6] 2023-05-31T21:35:42.294 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: 2023-05-31T21:12:04.382+0000 7f229ef2c780 1 waiting for incremental sync to catch up: 2023-05-31T21:35:42.295 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: local status: [, , , , , , , , 00000000001.1365.6, 00000000002.3947.6, 00000000001.2023.6] 2023-05-31T21:35:42.295 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: remote markers: [, , , , , , , , 00000000001.1365.6, 00000000002.3947.6, 00000000002.2025.6] 2023-05-31T21:35:42.295 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: 2023-05-31T21:12:09.383+0000 7f229ef2c780 1 waiting for incremental sync to catch up: 2023-05-31T21:35:42.295 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: local status: [, , , , , , , , 00000000001.1365.6, 00000000002.3947.6, 00000000001.2023.6] 2023-05-31T21:35:42.296 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: remote markers: [, , , , , , , , 00000000001.1365.6, 00000000002.3947.6, 00000000002.2025.6] 2023-05-31T21:35:42.296 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: 2023-05-31T21:12:14.384+0000 7f229ef2c780 1 waiting for incremental sync to catch up: 2023-05-31T21:35:42.296 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: local status: [, , , , , , , , 00000000001.1365.6, 00000000002.3947.6, 00000000001.2023.6] 2023-05-31T21:35:42.296 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: remote markers: [, , , , , , , , 00000000001.1365.6, 00000000002.3947.6, 00000000002.2025.6] 2023-05-31T21:35:42.296 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: 2023-05-31T21:12:19.385+0000 7f229ef2c780 1 bucket sync caught up with source: 2023-05-31T21:35:42.297 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: local status: [, , , , , , , , 00000000001.1365.6, 00000000002.3947.6, 00000000002.2025.6] 2023-05-31T21:35:42.297 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: remote markers: [, , , , , , , , 00000000001.1365.6, 00000000002.3947.6, 00000000002.2025.6] 2023-05-31T21:35:42.297 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: 2023-05-31T21:12:19.385+0000 7f229ef2c780 0 bucket checkpoint complete 2023-05-31T21:35:42.297 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: INFO: comparing bucket=pklxkd-64 zones={a2, a1} 2023-05-31T21:35:42.298 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/pklxkd-64/ 2023-05-31T21:35:42.298 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/pklxkd-64/ 2023-05-31T21:35:42.298 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-05-31T21:35:42.298 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /pklxkd-64/ 2023-05-31T21:35:42.299 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-05-31T21:35:42.299 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-05-31T21:35:42.299 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T21:35:42.299 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-05-31T21:35:42.299 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-05-31T21:35:42.300 INFO:tasks.rgw_multisite_tests:boto: DEBUG: establishing HTTP connection: kwargs={'timeout': 70, 'port': 8001} 2023-05-31T21:35:42.300 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-05-31T21:35:42.300 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-05-31T21:35:42.300 INFO:tasks.rgw_multisite_tests:HEAD 2023-05-31T21:35:42.301 INFO:tasks.rgw_multisite_tests: 2023-05-31T21:35:42.301 INFO:tasks.rgw_multisite_tests: 2023-05-31T21:35:42.301 INFO:tasks.rgw_multisite_tests:Wed, 31 May 2023 21:12:19 GMT 2023-05-31T21:35:42.301 INFO:tasks.rgw_multisite_tests:/pklxkd-64/ 2023-05-31T21:35:42.301 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-05-31T21:35:42.302 INFO:tasks.rgw_multisite_tests:AWS Q80H11OLNU1S48PPYL26:KsfBBJmgO4AvTs8461Cw/7QVctI= 2023-05-31T21:35:42.302 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 21:12:19 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:KsfBBJmgO4AvTs8461Cw/7QVctI=', 'Content-Length': '0'} 2023-05-31T21:35:42.302 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '5'), ('X-RGW-Bytes-Used', '15'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000ea515b585d467391-006477b833-1fb6-a2'), ('Content-Length', '0'), ('Date', 'Wed, 31 May 2023 21:12:19 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:35:42.302 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/pklxkd-64/ 2023-05-31T21:35:42.303 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/pklxkd-64/ 2023-05-31T21:35:42.303 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-05-31T21:35:42.303 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /pklxkd-64/ 2023-05-31T21:35:42.303 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-05-31T21:35:42.304 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-05-31T21:35:42.304 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T21:35:42.304 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-05-31T21:35:42.304 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-05-31T21:35:42.304 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-05-31T21:35:42.305 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-05-31T21:35:42.305 INFO:tasks.rgw_multisite_tests:HEAD 2023-05-31T21:35:42.305 INFO:tasks.rgw_multisite_tests: 2023-05-31T21:35:42.305 INFO:tasks.rgw_multisite_tests: 2023-05-31T21:35:42.306 INFO:tasks.rgw_multisite_tests:Wed, 31 May 2023 21:12:19 GMT 2023-05-31T21:35:42.306 INFO:tasks.rgw_multisite_tests:/pklxkd-64/ 2023-05-31T21:35:42.306 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-05-31T21:35:42.306 INFO:tasks.rgw_multisite_tests:AWS Q80H11OLNU1S48PPYL26:KsfBBJmgO4AvTs8461Cw/7QVctI= 2023-05-31T21:35:42.307 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 21:12:19 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:KsfBBJmgO4AvTs8461Cw/7QVctI=', 'Content-Length': '0'} 2023-05-31T21:35:42.307 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '5'), ('X-RGW-Bytes-Used', '15'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000000be941d8ce1f1ff2-006477b833-1630-a1'), ('Content-Length', '0'), ('Date', 'Wed, 31 May 2023 21:12:19 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:35:42.307 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: bucket1 objects: 2023-05-31T21:35:42.307 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/pklxkd-64/ 2023-05-31T21:35:42.308 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/pklxkd-64/ 2023-05-31T21:35:42.308 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/pklxkd-64/?versions&max-keys=999 2023-05-31T21:35:42.308 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/pklxkd-64/?versions&max-keys=999 2023-05-31T21:35:42.308 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-05-31T21:35:42.309 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /pklxkd-64/?versions&max-keys=999 2023-05-31T21:35:42.309 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-05-31T21:35:42.309 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-05-31T21:35:42.309 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T21:35:42.309 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-05-31T21:35:42.310 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-05-31T21:35:42.310 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-05-31T21:35:42.310 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-05-31T21:35:42.310 INFO:tasks.rgw_multisite_tests:GET 2023-05-31T21:35:42.311 INFO:tasks.rgw_multisite_tests: 2023-05-31T21:35:42.311 INFO:tasks.rgw_multisite_tests: 2023-05-31T21:35:42.311 INFO:tasks.rgw_multisite_tests:Wed, 31 May 2023 21:12:19 GMT 2023-05-31T21:35:42.311 INFO:tasks.rgw_multisite_tests:/pklxkd-64/?versions 2023-05-31T21:35:42.312 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-05-31T21:35:42.312 INFO:tasks.rgw_multisite_tests:AWS Q80H11OLNU1S48PPYL26:Y+FNsnKkhxckLDmOpwOjktX94r0= 2023-05-31T21:35:42.312 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 21:12:19 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:Y+FNsnKkhxckLDmOpwOjktX94r0=', 'Content-Length': '0'} 2023-05-31T21:35:42.312 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000022f6e23df123856f-006477b833-1fb6-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 21:12:19 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:35:42.312 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'pklxkd-64999falseobj11nulltrue2023-05-31T21:11:49.036Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj22nulltrue2023-05-31T21:11:49.055Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj33nulltrue2023-05-31T21:11:49.072Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj44nulltrue2023-05-31T21:11:49.126Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj55nulltrue2023-05-31T21:11:49.145Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T21:35:42.313 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: o=obj11 2023-05-31T21:35:42.313 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: o=obj22 2023-05-31T21:35:42.313 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: o=obj33 2023-05-31T21:35:42.313 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: o=obj44 2023-05-31T21:35:42.314 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: o=obj55 2023-05-31T21:35:42.314 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: bucket2 objects: 2023-05-31T21:35:42.314 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/pklxkd-64/ 2023-05-31T21:35:42.314 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/pklxkd-64/ 2023-05-31T21:35:42.315 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/pklxkd-64/?versions&max-keys=999 2023-05-31T21:35:42.315 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/pklxkd-64/?versions&max-keys=999 2023-05-31T21:35:42.315 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-05-31T21:35:42.315 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /pklxkd-64/?versions&max-keys=999 2023-05-31T21:35:42.315 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-05-31T21:35:42.316 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-05-31T21:35:42.316 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T21:35:42.316 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-05-31T21:35:42.316 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-05-31T21:35:42.317 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-05-31T21:35:42.317 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-05-31T21:35:42.317 INFO:tasks.rgw_multisite_tests:GET 2023-05-31T21:35:42.317 INFO:tasks.rgw_multisite_tests: 2023-05-31T21:35:42.318 INFO:tasks.rgw_multisite_tests: 2023-05-31T21:35:42.318 INFO:tasks.rgw_multisite_tests:Wed, 31 May 2023 21:12:19 GMT 2023-05-31T21:35:42.318 INFO:tasks.rgw_multisite_tests:/pklxkd-64/?versions 2023-05-31T21:35:42.318 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-05-31T21:35:42.318 INFO:tasks.rgw_multisite_tests:AWS Q80H11OLNU1S48PPYL26:Y+FNsnKkhxckLDmOpwOjktX94r0= 2023-05-31T21:35:42.319 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 21:12:19 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:Y+FNsnKkhxckLDmOpwOjktX94r0=', 'Content-Length': '0'} 2023-05-31T21:35:42.319 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d7548dab02f2e1ca-006477b833-1630-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 21:12:19 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:35:42.319 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'pklxkd-64999falseobj11nulltrue2023-05-31T21:11:49.036Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj22nulltrue2023-05-31T21:11:49.055Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj33nulltrue2023-05-31T21:11:49.072Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj44nulltrue2023-05-31T21:11:49.126Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj55nulltrue2023-05-31T21:11:49.145Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T21:35:42.319 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: o=obj11 2023-05-31T21:35:42.320 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: o=obj22 2023-05-31T21:35:42.320 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: o=obj33 2023-05-31T21:35:42.320 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: o=obj44 2023-05-31T21:35:42.320 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: o=obj55 2023-05-31T21:35:42.321 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/pklxkd-64/ 2023-05-31T21:35:42.321 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/pklxkd-64/ 2023-05-31T21:35:42.321 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/pklxkd-64/?versions&max-keys=999 2023-05-31T21:35:42.321 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/pklxkd-64/?versions&max-keys=999 2023-05-31T21:35:42.321 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-05-31T21:35:42.322 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /pklxkd-64/?versions&max-keys=999 2023-05-31T21:35:42.322 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-05-31T21:35:42.322 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-05-31T21:35:42.322 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T21:35:42.323 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-05-31T21:35:42.323 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-05-31T21:35:42.323 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-05-31T21:35:42.323 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-05-31T21:35:42.324 INFO:tasks.rgw_multisite_tests:GET 2023-05-31T21:35:42.324 INFO:tasks.rgw_multisite_tests: 2023-05-31T21:35:42.324 INFO:tasks.rgw_multisite_tests: 2023-05-31T21:35:42.324 INFO:tasks.rgw_multisite_tests:Wed, 31 May 2023 21:12:19 GMT 2023-05-31T21:35:42.324 INFO:tasks.rgw_multisite_tests:/pklxkd-64/?versions 2023-05-31T21:35:42.325 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-05-31T21:35:42.325 INFO:tasks.rgw_multisite_tests:AWS Q80H11OLNU1S48PPYL26:Y+FNsnKkhxckLDmOpwOjktX94r0= 2023-05-31T21:35:42.325 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 21:12:19 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:Y+FNsnKkhxckLDmOpwOjktX94r0=', 'Content-Length': '0'} 2023-05-31T21:35:42.325 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000051318874d126aab3-006477b833-1fb6-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 21:12:19 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:35:42.326 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'pklxkd-64999falseobj11nulltrue2023-05-31T21:11:49.036Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj22nulltrue2023-05-31T21:11:49.055Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj33nulltrue2023-05-31T21:11:49.072Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj44nulltrue2023-05-31T21:11:49.126Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj55nulltrue2023-05-31T21:11:49.145Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T21:35:42.326 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/pklxkd-64/ 2023-05-31T21:35:42.326 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/pklxkd-64/ 2023-05-31T21:35:42.326 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/pklxkd-64/?versions&max-keys=999 2023-05-31T21:35:42.327 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/pklxkd-64/?versions&max-keys=999 2023-05-31T21:35:42.327 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-05-31T21:35:42.327 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /pklxkd-64/?versions&max-keys=999 2023-05-31T21:35:42.327 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-05-31T21:35:42.327 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-05-31T21:35:42.328 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T21:35:42.328 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-05-31T21:35:42.328 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-05-31T21:35:42.328 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-05-31T21:35:42.329 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-05-31T21:35:42.329 INFO:tasks.rgw_multisite_tests:GET 2023-05-31T21:35:42.329 INFO:tasks.rgw_multisite_tests: 2023-05-31T21:35:42.329 INFO:tasks.rgw_multisite_tests: 2023-05-31T21:35:42.330 INFO:tasks.rgw_multisite_tests:Wed, 31 May 2023 21:12:19 GMT 2023-05-31T21:35:42.330 INFO:tasks.rgw_multisite_tests:/pklxkd-64/?versions 2023-05-31T21:35:42.330 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-05-31T21:35:42.330 INFO:tasks.rgw_multisite_tests:AWS Q80H11OLNU1S48PPYL26:Y+FNsnKkhxckLDmOpwOjktX94r0= 2023-05-31T21:35:42.330 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 21:12:19 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:Y+FNsnKkhxckLDmOpwOjktX94r0=', 'Content-Length': '0'} 2023-05-31T21:35:42.331 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000082f04cd750ca8cff-006477b833-1630-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 21:12:19 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:35:42.331 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'pklxkd-64999falseobj11nulltrue2023-05-31T21:11:49.036Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj22nulltrue2023-05-31T21:11:49.055Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj33nulltrue2023-05-31T21:11:49.072Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj44nulltrue2023-05-31T21:11:49.126Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj55nulltrue2023-05-31T21:11:49.145Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T21:35:42.331 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: comparing key name=obj11 2023-05-31T21:35:42.331 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/pklxkd-64/obj11 2023-05-31T21:35:42.332 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/pklxkd-64/obj11 2023-05-31T21:35:42.332 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/pklxkd-64/obj11?versionId=null 2023-05-31T21:35:42.332 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/pklxkd-64/obj11?versionId=null 2023-05-31T21:35:42.332 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-05-31T21:35:42.333 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /pklxkd-64/obj11?versionId=null 2023-05-31T21:35:42.333 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-05-31T21:35:42.333 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-05-31T21:35:42.333 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T21:35:42.334 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-05-31T21:35:42.334 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-05-31T21:35:42.334 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-05-31T21:35:42.334 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-05-31T21:35:42.334 INFO:tasks.rgw_multisite_tests:GET 2023-05-31T21:35:42.335 INFO:tasks.rgw_multisite_tests: 2023-05-31T21:35:42.335 INFO:tasks.rgw_multisite_tests: 2023-05-31T21:35:42.335 INFO:tasks.rgw_multisite_tests:Wed, 31 May 2023 21:12:19 GMT 2023-05-31T21:35:42.335 INFO:tasks.rgw_multisite_tests:/pklxkd-64/obj11?versionId=null 2023-05-31T21:35:42.336 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-05-31T21:35:42.336 INFO:tasks.rgw_multisite_tests:AWS Q80H11OLNU1S48PPYL26:JoTqV5gdZipLpDgP6hOcJO5nUWg= 2023-05-31T21:35:42.336 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 21:12:19 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:JoTqV5gdZipLpDgP6hOcJO5nUWg=', 'Content-Length': '0'} 2023-05-31T21:35:42.336 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 21:11:49 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006e0d165b023ac06e-006477b833-1fb6-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 21:12:19 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:35:42.337 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/pklxkd-64/obj11 2023-05-31T21:35:42.337 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/pklxkd-64/obj11 2023-05-31T21:35:42.337 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/pklxkd-64/obj11?versionId=null 2023-05-31T21:35:42.337 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/pklxkd-64/obj11?versionId=null 2023-05-31T21:35:42.337 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-05-31T21:35:42.338 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /pklxkd-64/obj11?versionId=null 2023-05-31T21:35:42.338 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-05-31T21:35:42.338 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-05-31T21:35:42.338 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T21:35:42.339 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-05-31T21:35:42.339 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-05-31T21:35:42.339 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-05-31T21:35:42.339 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-05-31T21:35:42.340 INFO:tasks.rgw_multisite_tests:GET 2023-05-31T21:35:42.340 INFO:tasks.rgw_multisite_tests: 2023-05-31T21:35:42.340 INFO:tasks.rgw_multisite_tests: 2023-05-31T21:35:42.340 INFO:tasks.rgw_multisite_tests:Wed, 31 May 2023 21:12:19 GMT 2023-05-31T21:35:42.340 INFO:tasks.rgw_multisite_tests:/pklxkd-64/obj11?versionId=null 2023-05-31T21:35:42.341 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-05-31T21:35:42.341 INFO:tasks.rgw_multisite_tests:AWS Q80H11OLNU1S48PPYL26:JoTqV5gdZipLpDgP6hOcJO5nUWg= 2023-05-31T21:35:42.341 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 21:12:19 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:JoTqV5gdZipLpDgP6hOcJO5nUWg=', 'Content-Length': '0'} 2023-05-31T21:35:42.341 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 21:11:49 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000bd4530429db50bb3-006477b833-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 21:12:19 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:35:42.342 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/pklxkd-64/obj11 2023-05-31T21:35:42.342 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/pklxkd-64/obj11 2023-05-31T21:35:42.342 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/pklxkd-64/obj11?versionId=null 2023-05-31T21:35:42.342 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/pklxkd-64/obj11?versionId=null 2023-05-31T21:35:42.343 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-05-31T21:35:42.343 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /pklxkd-64/obj11?versionId=null 2023-05-31T21:35:42.343 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-05-31T21:35:42.343 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-05-31T21:35:42.343 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T21:35:42.344 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-05-31T21:35:42.344 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-05-31T21:35:42.344 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-05-31T21:35:42.344 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-05-31T21:35:42.345 INFO:tasks.rgw_multisite_tests:HEAD 2023-05-31T21:35:42.345 INFO:tasks.rgw_multisite_tests: 2023-05-31T21:35:42.345 INFO:tasks.rgw_multisite_tests: 2023-05-31T21:35:42.345 INFO:tasks.rgw_multisite_tests:Wed, 31 May 2023 21:12:19 GMT 2023-05-31T21:35:42.346 INFO:tasks.rgw_multisite_tests:/pklxkd-64/obj11?versionId=null 2023-05-31T21:35:42.346 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-05-31T21:35:42.346 INFO:tasks.rgw_multisite_tests:AWS Q80H11OLNU1S48PPYL26:ojWc3XEBka9c23WFsR2eI4DvBPE= 2023-05-31T21:35:42.346 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 21:12:19 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:ojWc3XEBka9c23WFsR2eI4DvBPE=', 'Content-Length': '0'} 2023-05-31T21:35:42.346 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 21:11:49 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c3c44ac98906140a-006477b833-1fb6-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 21:12:19 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:35:42.347 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/pklxkd-64/obj11 2023-05-31T21:35:42.347 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/pklxkd-64/obj11 2023-05-31T21:35:42.347 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/pklxkd-64/obj11?versionId=null 2023-05-31T21:35:42.347 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/pklxkd-64/obj11?versionId=null 2023-05-31T21:35:42.348 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-05-31T21:35:42.348 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /pklxkd-64/obj11?versionId=null 2023-05-31T21:35:42.348 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-05-31T21:35:42.348 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-05-31T21:35:42.349 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T21:35:42.349 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-05-31T21:35:42.349 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-05-31T21:35:42.349 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-05-31T21:35:42.350 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-05-31T21:35:42.350 INFO:tasks.rgw_multisite_tests:HEAD 2023-05-31T21:35:42.350 INFO:tasks.rgw_multisite_tests: 2023-05-31T21:35:42.350 INFO:tasks.rgw_multisite_tests: 2023-05-31T21:35:42.350 INFO:tasks.rgw_multisite_tests:Wed, 31 May 2023 21:12:19 GMT 2023-05-31T21:35:42.351 INFO:tasks.rgw_multisite_tests:/pklxkd-64/obj11?versionId=null 2023-05-31T21:35:42.351 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-05-31T21:35:42.351 INFO:tasks.rgw_multisite_tests:AWS Q80H11OLNU1S48PPYL26:ojWc3XEBka9c23WFsR2eI4DvBPE= 2023-05-31T21:35:42.351 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 21:12:19 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:ojWc3XEBka9c23WFsR2eI4DvBPE=', 'Content-Length': '0'} 2023-05-31T21:35:42.352 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 21:11:49 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000aeb09e39e0082b24-006477b833-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 21:12:19 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:35:42.352 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: comparing key name=obj11 2023-05-31T21:35:42.352 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/pklxkd-64/obj11 2023-05-31T21:35:42.352 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/pklxkd-64/obj11 2023-05-31T21:35:42.353 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/pklxkd-64/obj11?versionId=null 2023-05-31T21:35:42.353 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/pklxkd-64/obj11?versionId=null 2023-05-31T21:35:42.353 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-05-31T21:35:42.353 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /pklxkd-64/obj11?versionId=null 2023-05-31T21:35:42.354 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-05-31T21:35:42.354 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-05-31T21:35:42.354 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T21:35:42.354 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-05-31T21:35:42.354 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-05-31T21:35:42.355 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-05-31T21:35:42.355 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-05-31T21:35:42.355 INFO:tasks.rgw_multisite_tests:GET 2023-05-31T21:35:42.355 INFO:tasks.rgw_multisite_tests: 2023-05-31T21:35:42.356 INFO:tasks.rgw_multisite_tests: 2023-05-31T21:35:42.356 INFO:tasks.rgw_multisite_tests:Wed, 31 May 2023 21:12:19 GMT 2023-05-31T21:35:42.356 INFO:tasks.rgw_multisite_tests:/pklxkd-64/obj11?versionId=null 2023-05-31T21:35:42.356 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-05-31T21:35:42.356 INFO:tasks.rgw_multisite_tests:AWS Q80H11OLNU1S48PPYL26:JoTqV5gdZipLpDgP6hOcJO5nUWg= 2023-05-31T21:35:42.357 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 21:12:19 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:JoTqV5gdZipLpDgP6hOcJO5nUWg=', 'Content-Length': '0'} 2023-05-31T21:35:42.357 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 21:11:49 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007393a01af39bd799-006477b833-1fb6-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 21:12:19 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:35:42.357 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/pklxkd-64/obj11 2023-05-31T21:35:42.357 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/pklxkd-64/obj11 2023-05-31T21:35:42.358 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/pklxkd-64/obj11?versionId=null 2023-05-31T21:35:42.358 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/pklxkd-64/obj11?versionId=null 2023-05-31T21:35:42.358 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-05-31T21:35:42.358 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /pklxkd-64/obj11?versionId=null 2023-05-31T21:35:42.359 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-05-31T21:35:42.359 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-05-31T21:35:42.359 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T21:35:42.359 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-05-31T21:35:42.359 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-05-31T21:35:42.360 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-05-31T21:35:42.360 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-05-31T21:35:42.360 INFO:tasks.rgw_multisite_tests:GET 2023-05-31T21:35:42.360 INFO:tasks.rgw_multisite_tests: 2023-05-31T21:35:42.361 INFO:tasks.rgw_multisite_tests: 2023-05-31T21:35:42.361 INFO:tasks.rgw_multisite_tests:Wed, 31 May 2023 21:12:19 GMT 2023-05-31T21:35:42.361 INFO:tasks.rgw_multisite_tests:/pklxkd-64/obj11?versionId=null 2023-05-31T21:35:42.361 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-05-31T21:35:42.362 INFO:tasks.rgw_multisite_tests:AWS Q80H11OLNU1S48PPYL26:JoTqV5gdZipLpDgP6hOcJO5nUWg= 2023-05-31T21:35:42.362 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 21:12:19 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:JoTqV5gdZipLpDgP6hOcJO5nUWg=', 'Content-Length': '0'} 2023-05-31T21:35:42.362 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 21:11:49 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000930bf0409d66d684-006477b833-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 21:12:19 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:35:42.362 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/pklxkd-64/ 2023-05-31T21:35:42.362 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/pklxkd-64/ 2023-05-31T21:35:42.363 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/pklxkd-64/?prefix=obj11 2023-05-31T21:35:42.363 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/pklxkd-64/?prefix=obj11 2023-05-31T21:35:42.363 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-05-31T21:35:42.363 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /pklxkd-64/?prefix=obj11 2023-05-31T21:35:42.364 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-05-31T21:35:42.364 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-05-31T21:35:42.364 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T21:35:42.364 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-05-31T21:35:42.365 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-05-31T21:35:42.365 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-05-31T21:35:42.365 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-05-31T21:35:42.365 INFO:tasks.rgw_multisite_tests:GET 2023-05-31T21:35:42.365 INFO:tasks.rgw_multisite_tests: 2023-05-31T21:35:42.366 INFO:tasks.rgw_multisite_tests: 2023-05-31T21:35:42.366 INFO:tasks.rgw_multisite_tests:Wed, 31 May 2023 21:12:19 GMT 2023-05-31T21:35:42.366 INFO:tasks.rgw_multisite_tests:/pklxkd-64/ 2023-05-31T21:35:42.366 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-05-31T21:35:42.367 INFO:tasks.rgw_multisite_tests:AWS Q80H11OLNU1S48PPYL26:/ACajcjJ+wTjnzFp03OUBb/7FmU= 2023-05-31T21:35:42.367 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 21:12:19 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:/ACajcjJ+wTjnzFp03OUBb/7FmU=', 'Content-Length': '0'} 2023-05-31T21:35:42.367 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000091f9c45b81aff639-006477b833-1fb6-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 21:12:19 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:35:42.367 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'pklxkd-64obj111000falseobj112023-05-31T21:11:49.036Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T21:35:42.368 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/pklxkd-64/ 2023-05-31T21:35:42.368 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/pklxkd-64/ 2023-05-31T21:35:42.368 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/pklxkd-64/?prefix=obj11 2023-05-31T21:35:42.368 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/pklxkd-64/?prefix=obj11 2023-05-31T21:35:42.369 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-05-31T21:35:42.369 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /pklxkd-64/?prefix=obj11 2023-05-31T21:35:42.369 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-05-31T21:35:42.369 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-05-31T21:35:42.369 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T21:35:42.370 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-05-31T21:35:42.370 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-05-31T21:35:42.370 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-05-31T21:35:42.370 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-05-31T21:35:42.371 INFO:tasks.rgw_multisite_tests:GET 2023-05-31T21:35:42.371 INFO:tasks.rgw_multisite_tests: 2023-05-31T21:35:42.371 INFO:tasks.rgw_multisite_tests: 2023-05-31T21:35:42.371 INFO:tasks.rgw_multisite_tests:Wed, 31 May 2023 21:12:19 GMT 2023-05-31T21:35:42.371 INFO:tasks.rgw_multisite_tests:/pklxkd-64/ 2023-05-31T21:35:42.372 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-05-31T21:35:42.372 INFO:tasks.rgw_multisite_tests:AWS Q80H11OLNU1S48PPYL26:/ACajcjJ+wTjnzFp03OUBb/7FmU= 2023-05-31T21:35:42.372 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 21:12:19 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:/ACajcjJ+wTjnzFp03OUBb/7FmU=', 'Content-Length': '0'} 2023-05-31T21:35:42.372 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000008424732359768433-006477b833-1630-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 21:12:19 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:35:42.373 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'pklxkd-64obj111000falseobj112023-05-31T21:11:49.036Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T21:35:42.373 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/pklxkd-64/obj11 2023-05-31T21:35:42.373 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/pklxkd-64/obj11 2023-05-31T21:35:42.373 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-05-31T21:35:42.374 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /pklxkd-64/obj11 2023-05-31T21:35:42.374 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-05-31T21:35:42.374 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-05-31T21:35:42.374 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T21:35:42.374 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-05-31T21:35:42.375 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-05-31T21:35:42.375 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-05-31T21:35:42.375 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-05-31T21:35:42.375 INFO:tasks.rgw_multisite_tests:HEAD 2023-05-31T21:35:42.376 INFO:tasks.rgw_multisite_tests: 2023-05-31T21:35:42.376 INFO:tasks.rgw_multisite_tests: 2023-05-31T21:35:42.376 INFO:tasks.rgw_multisite_tests:Wed, 31 May 2023 21:12:19 GMT 2023-05-31T21:35:42.376 INFO:tasks.rgw_multisite_tests:/pklxkd-64/obj11 2023-05-31T21:35:42.377 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-05-31T21:35:42.377 INFO:tasks.rgw_multisite_tests:AWS Q80H11OLNU1S48PPYL26:mEI87qCToWPjfFwbwN26x4EB8q0= 2023-05-31T21:35:42.377 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 21:12:19 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:mEI87qCToWPjfFwbwN26x4EB8q0=', 'Content-Length': '0'} 2023-05-31T21:35:42.377 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 21:11:49 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a9edf60ee3b588d4-006477b833-1fb6-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 21:12:19 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:35:42.378 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/pklxkd-64/obj11 2023-05-31T21:35:42.378 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/pklxkd-64/obj11 2023-05-31T21:35:42.378 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-05-31T21:35:42.378 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /pklxkd-64/obj11 2023-05-31T21:35:42.378 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-05-31T21:35:42.379 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-05-31T21:35:42.379 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T21:35:42.379 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-05-31T21:35:42.379 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-05-31T21:35:42.380 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-05-31T21:35:42.380 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-05-31T21:35:42.380 INFO:tasks.rgw_multisite_tests:HEAD 2023-05-31T21:35:42.380 INFO:tasks.rgw_multisite_tests: 2023-05-31T21:35:42.381 INFO:tasks.rgw_multisite_tests: 2023-05-31T21:35:42.381 INFO:tasks.rgw_multisite_tests:Wed, 31 May 2023 21:12:19 GMT 2023-05-31T21:35:42.381 INFO:tasks.rgw_multisite_tests:/pklxkd-64/obj11 2023-05-31T21:35:42.381 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-05-31T21:35:42.381 INFO:tasks.rgw_multisite_tests:AWS Q80H11OLNU1S48PPYL26:mEI87qCToWPjfFwbwN26x4EB8q0= 2023-05-31T21:35:42.382 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 21:12:19 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:mEI87qCToWPjfFwbwN26x4EB8q0=', 'Content-Length': '0'} 2023-05-31T21:35:42.382 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 21:11:49 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ef2cfb8d27bda539-006477b833-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 21:12:19 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:35:42.382 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: comparing key name=obj11 2023-05-31T21:35:42.382 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/pklxkd-64/obj11 2023-05-31T21:35:42.383 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/pklxkd-64/obj11 2023-05-31T21:35:42.383 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-05-31T21:35:42.383 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /pklxkd-64/obj11 2023-05-31T21:35:42.383 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-05-31T21:35:42.384 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-05-31T21:35:42.384 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T21:35:42.384 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-05-31T21:35:42.384 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-05-31T21:35:42.385 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-05-31T21:35:42.385 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-05-31T21:35:42.385 INFO:tasks.rgw_multisite_tests:GET 2023-05-31T21:35:42.385 INFO:tasks.rgw_multisite_tests: 2023-05-31T21:35:42.386 INFO:tasks.rgw_multisite_tests: 2023-05-31T21:35:42.386 INFO:tasks.rgw_multisite_tests:Wed, 31 May 2023 21:12:19 GMT 2023-05-31T21:35:42.386 INFO:tasks.rgw_multisite_tests:/pklxkd-64/obj11 2023-05-31T21:35:42.386 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-05-31T21:35:42.386 INFO:tasks.rgw_multisite_tests:AWS Q80H11OLNU1S48PPYL26:iIB+1MW4PDp12ma/FQGl76unqys= 2023-05-31T21:35:42.387 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 21:12:19 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:iIB+1MW4PDp12ma/FQGl76unqys=', 'Content-Length': '0'} 2023-05-31T21:35:42.387 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 21:11:49 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006d6ce2b0d97bd2a6-006477b833-1fb6-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 21:12:19 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:35:42.387 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/pklxkd-64/obj11 2023-05-31T21:35:42.387 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/pklxkd-64/obj11 2023-05-31T21:35:42.388 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-05-31T21:35:42.388 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /pklxkd-64/obj11 2023-05-31T21:35:42.388 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-05-31T21:35:42.388 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-05-31T21:35:42.389 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T21:35:42.389 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-05-31T21:35:42.389 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-05-31T21:35:42.389 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-05-31T21:35:42.389 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-05-31T21:35:42.390 INFO:tasks.rgw_multisite_tests:GET 2023-05-31T21:35:42.390 INFO:tasks.rgw_multisite_tests: 2023-05-31T21:35:42.390 INFO:tasks.rgw_multisite_tests: 2023-05-31T21:35:42.390 INFO:tasks.rgw_multisite_tests:Wed, 31 May 2023 21:12:19 GMT 2023-05-31T21:35:42.391 INFO:tasks.rgw_multisite_tests:/pklxkd-64/obj11 2023-05-31T21:35:42.391 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-05-31T21:35:42.391 INFO:tasks.rgw_multisite_tests:AWS Q80H11OLNU1S48PPYL26:iIB+1MW4PDp12ma/FQGl76unqys= 2023-05-31T21:35:42.391 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 21:12:19 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:iIB+1MW4PDp12ma/FQGl76unqys=', 'Content-Length': '0'} 2023-05-31T21:35:42.392 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 21:11:49 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009904d83b360f5c33-006477b833-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 21:12:19 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:35:42.392 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/pklxkd-64/ 2023-05-31T21:35:42.392 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/pklxkd-64/ 2023-05-31T21:35:42.392 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/pklxkd-64/?prefix=obj11 2023-05-31T21:35:42.392 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/pklxkd-64/?prefix=obj11 2023-05-31T21:35:42.393 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-05-31T21:35:42.393 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /pklxkd-64/?prefix=obj11 2023-05-31T21:35:42.393 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-05-31T21:35:42.393 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-05-31T21:35:42.394 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T21:35:42.394 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-05-31T21:35:42.394 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-05-31T21:35:42.394 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-05-31T21:35:42.395 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-05-31T21:35:42.395 INFO:tasks.rgw_multisite_tests:GET 2023-05-31T21:35:42.395 INFO:tasks.rgw_multisite_tests: 2023-05-31T21:35:42.395 INFO:tasks.rgw_multisite_tests: 2023-05-31T21:35:42.396 INFO:tasks.rgw_multisite_tests:Wed, 31 May 2023 21:12:19 GMT 2023-05-31T21:35:42.396 INFO:tasks.rgw_multisite_tests:/pklxkd-64/ 2023-05-31T21:35:42.396 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-05-31T21:35:42.396 INFO:tasks.rgw_multisite_tests:AWS Q80H11OLNU1S48PPYL26:/ACajcjJ+wTjnzFp03OUBb/7FmU= 2023-05-31T21:35:42.397 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 21:12:19 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:/ACajcjJ+wTjnzFp03OUBb/7FmU=', 'Content-Length': '0'} 2023-05-31T21:35:42.397 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000032e3a758b471a239-006477b833-1fb6-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 21:12:19 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:35:42.397 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'pklxkd-64obj111000falseobj112023-05-31T21:11:49.036Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T21:35:42.397 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/pklxkd-64/ 2023-05-31T21:35:42.397 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/pklxkd-64/ 2023-05-31T21:35:42.398 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/pklxkd-64/?prefix=obj11 2023-05-31T21:35:42.398 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/pklxkd-64/?prefix=obj11 2023-05-31T21:35:42.398 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-05-31T21:35:42.398 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /pklxkd-64/?prefix=obj11 2023-05-31T21:35:42.399 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-05-31T21:35:42.399 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-05-31T21:35:42.399 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T21:35:42.399 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-05-31T21:35:42.400 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-05-31T21:35:42.400 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-05-31T21:35:42.400 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-05-31T21:35:42.400 INFO:tasks.rgw_multisite_tests:GET 2023-05-31T21:35:42.400 INFO:tasks.rgw_multisite_tests: 2023-05-31T21:35:42.401 INFO:tasks.rgw_multisite_tests: 2023-05-31T21:35:42.401 INFO:tasks.rgw_multisite_tests:Wed, 31 May 2023 21:12:19 GMT 2023-05-31T21:35:42.401 INFO:tasks.rgw_multisite_tests:/pklxkd-64/ 2023-05-31T21:35:42.401 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-05-31T21:35:42.402 INFO:tasks.rgw_multisite_tests:AWS Q80H11OLNU1S48PPYL26:/ACajcjJ+wTjnzFp03OUBb/7FmU= 2023-05-31T21:35:42.402 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 21:12:19 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:/ACajcjJ+wTjnzFp03OUBb/7FmU=', 'Content-Length': '0'} 2023-05-31T21:35:42.402 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000078b8c48d71fefd79-006477b833-1630-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 21:12:19 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:35:42.402 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'pklxkd-64obj111000falseobj112023-05-31T21:11:49.036Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T21:35:42.403 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: comparing key name=obj22 2023-05-31T21:35:42.403 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/pklxkd-64/obj22 2023-05-31T21:35:42.403 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/pklxkd-64/obj22 2023-05-31T21:35:42.403 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/pklxkd-64/obj22?versionId=null 2023-05-31T21:35:42.403 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/pklxkd-64/obj22?versionId=null 2023-05-31T21:35:42.404 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-05-31T21:35:42.404 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /pklxkd-64/obj22?versionId=null 2023-05-31T21:35:42.404 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-05-31T21:35:42.404 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-05-31T21:35:42.405 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T21:35:42.405 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-05-31T21:35:42.405 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-05-31T21:35:42.405 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-05-31T21:35:42.406 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-05-31T21:35:42.406 INFO:tasks.rgw_multisite_tests:GET 2023-05-31T21:35:42.406 INFO:tasks.rgw_multisite_tests: 2023-05-31T21:35:42.406 INFO:tasks.rgw_multisite_tests: 2023-05-31T21:35:42.406 INFO:tasks.rgw_multisite_tests:Wed, 31 May 2023 21:12:19 GMT 2023-05-31T21:35:42.407 INFO:tasks.rgw_multisite_tests:/pklxkd-64/obj22?versionId=null 2023-05-31T21:35:42.407 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-05-31T21:35:42.407 INFO:tasks.rgw_multisite_tests:AWS Q80H11OLNU1S48PPYL26:OZnIiljRkUdTAdQuQcOP60YYcNM= 2023-05-31T21:35:42.407 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 21:12:19 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:OZnIiljRkUdTAdQuQcOP60YYcNM=', 'Content-Length': '0'} 2023-05-31T21:35:42.408 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 21:11:49 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001a2a4c8db5672789-006477b833-1fb6-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 21:12:19 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:35:42.408 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/pklxkd-64/obj22 2023-05-31T21:35:42.408 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/pklxkd-64/obj22 2023-05-31T21:35:42.408 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/pklxkd-64/obj22?versionId=null 2023-05-31T21:35:42.409 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/pklxkd-64/obj22?versionId=null 2023-05-31T21:35:42.409 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-05-31T21:35:42.409 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /pklxkd-64/obj22?versionId=null 2023-05-31T21:35:42.409 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-05-31T21:35:42.409 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-05-31T21:35:42.410 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T21:35:42.410 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-05-31T21:35:42.410 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-05-31T21:35:42.410 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-05-31T21:35:42.411 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-05-31T21:35:42.411 INFO:tasks.rgw_multisite_tests:GET 2023-05-31T21:35:42.411 INFO:tasks.rgw_multisite_tests: 2023-05-31T21:35:42.411 INFO:tasks.rgw_multisite_tests: 2023-05-31T21:35:42.411 INFO:tasks.rgw_multisite_tests:Wed, 31 May 2023 21:12:19 GMT 2023-05-31T21:35:42.412 INFO:tasks.rgw_multisite_tests:/pklxkd-64/obj22?versionId=null 2023-05-31T21:35:42.412 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-05-31T21:35:42.412 INFO:tasks.rgw_multisite_tests:AWS Q80H11OLNU1S48PPYL26:OZnIiljRkUdTAdQuQcOP60YYcNM= 2023-05-31T21:35:42.412 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 21:12:19 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:OZnIiljRkUdTAdQuQcOP60YYcNM=', 'Content-Length': '0'} 2023-05-31T21:35:42.413 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 21:11:49 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003456431745452446-006477b833-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 21:12:19 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:35:42.413 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/pklxkd-64/obj22 2023-05-31T21:35:42.413 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/pklxkd-64/obj22 2023-05-31T21:35:42.413 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/pklxkd-64/obj22?versionId=null 2023-05-31T21:35:42.414 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/pklxkd-64/obj22?versionId=null 2023-05-31T21:35:42.414 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-05-31T21:35:42.414 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /pklxkd-64/obj22?versionId=null 2023-05-31T21:35:42.414 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-05-31T21:35:42.414 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-05-31T21:35:42.415 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T21:35:42.415 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-05-31T21:35:42.415 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-05-31T21:35:42.415 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-05-31T21:35:42.416 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-05-31T21:35:42.416 INFO:tasks.rgw_multisite_tests:HEAD 2023-05-31T21:35:42.416 INFO:tasks.rgw_multisite_tests: 2023-05-31T21:35:42.416 INFO:tasks.rgw_multisite_tests: 2023-05-31T21:35:42.417 INFO:tasks.rgw_multisite_tests:Wed, 31 May 2023 21:12:19 GMT 2023-05-31T21:35:42.417 INFO:tasks.rgw_multisite_tests:/pklxkd-64/obj22?versionId=null 2023-05-31T21:35:42.417 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-05-31T21:35:42.417 INFO:tasks.rgw_multisite_tests:AWS Q80H11OLNU1S48PPYL26:6GDQIzjluGv0gpA51bCI21tCHEs= 2023-05-31T21:35:42.417 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 21:12:19 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:6GDQIzjluGv0gpA51bCI21tCHEs=', 'Content-Length': '0'} 2023-05-31T21:35:42.418 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 21:11:49 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a8b2fd9edb88258b-006477b833-1fb6-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 21:12:19 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:35:42.418 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/pklxkd-64/obj22 2023-05-31T21:35:42.418 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/pklxkd-64/obj22 2023-05-31T21:35:42.418 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/pklxkd-64/obj22?versionId=null 2023-05-31T21:35:42.419 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/pklxkd-64/obj22?versionId=null 2023-05-31T21:35:42.419 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-05-31T21:35:42.419 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /pklxkd-64/obj22?versionId=null 2023-05-31T21:35:42.419 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-05-31T21:35:42.420 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-05-31T21:35:42.420 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T21:35:42.420 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-05-31T21:35:42.420 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-05-31T21:35:42.420 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-05-31T21:35:42.421 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-05-31T21:35:42.421 INFO:tasks.rgw_multisite_tests:HEAD 2023-05-31T21:35:42.421 INFO:tasks.rgw_multisite_tests: 2023-05-31T21:35:42.421 INFO:tasks.rgw_multisite_tests: 2023-05-31T21:35:42.422 INFO:tasks.rgw_multisite_tests:Wed, 31 May 2023 21:12:19 GMT 2023-05-31T21:35:42.422 INFO:tasks.rgw_multisite_tests:/pklxkd-64/obj22?versionId=null 2023-05-31T21:35:42.422 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-05-31T21:35:42.422 INFO:tasks.rgw_multisite_tests:AWS Q80H11OLNU1S48PPYL26:6GDQIzjluGv0gpA51bCI21tCHEs= 2023-05-31T21:35:42.423 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 21:12:19 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:6GDQIzjluGv0gpA51bCI21tCHEs=', 'Content-Length': '0'} 2023-05-31T21:35:42.423 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 21:11:49 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008a6c9877956b9292-006477b833-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 21:12:19 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:35:42.423 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: comparing key name=obj22 2023-05-31T21:35:42.423 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/pklxkd-64/obj22 2023-05-31T21:35:42.424 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/pklxkd-64/obj22 2023-05-31T21:35:42.424 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/pklxkd-64/obj22?versionId=null 2023-05-31T21:35:42.424 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/pklxkd-64/obj22?versionId=null 2023-05-31T21:35:42.424 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-05-31T21:35:42.425 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /pklxkd-64/obj22?versionId=null 2023-05-31T21:35:42.425 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-05-31T21:35:42.425 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-05-31T21:35:42.425 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T21:35:42.426 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-05-31T21:35:42.426 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-05-31T21:35:42.426 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-05-31T21:35:42.426 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-05-31T21:35:42.427 INFO:tasks.rgw_multisite_tests:GET 2023-05-31T21:35:42.427 INFO:tasks.rgw_multisite_tests: 2023-05-31T21:35:42.427 INFO:tasks.rgw_multisite_tests: 2023-05-31T21:35:42.427 INFO:tasks.rgw_multisite_tests:Wed, 31 May 2023 21:12:19 GMT 2023-05-31T21:35:42.427 INFO:tasks.rgw_multisite_tests:/pklxkd-64/obj22?versionId=null 2023-05-31T21:35:42.428 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-05-31T21:35:42.428 INFO:tasks.rgw_multisite_tests:AWS Q80H11OLNU1S48PPYL26:OZnIiljRkUdTAdQuQcOP60YYcNM= 2023-05-31T21:35:42.428 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 21:12:19 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:OZnIiljRkUdTAdQuQcOP60YYcNM=', 'Content-Length': '0'} 2023-05-31T21:35:42.428 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 21:11:49 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002d59a3b67ab06811-006477b833-1fb6-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 21:12:19 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:35:42.429 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/pklxkd-64/obj22 2023-05-31T21:35:42.429 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/pklxkd-64/obj22 2023-05-31T21:35:42.429 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/pklxkd-64/obj22?versionId=null 2023-05-31T21:35:42.429 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/pklxkd-64/obj22?versionId=null 2023-05-31T21:35:42.430 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-05-31T21:35:42.430 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /pklxkd-64/obj22?versionId=null 2023-05-31T21:35:42.430 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-05-31T21:35:42.430 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-05-31T21:35:42.430 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T21:35:42.431 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-05-31T21:35:42.431 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-05-31T21:35:42.431 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-05-31T21:35:42.431 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-05-31T21:35:42.432 INFO:tasks.rgw_multisite_tests:GET 2023-05-31T21:35:42.432 INFO:tasks.rgw_multisite_tests: 2023-05-31T21:35:42.432 INFO:tasks.rgw_multisite_tests: 2023-05-31T21:35:42.432 INFO:tasks.rgw_multisite_tests:Wed, 31 May 2023 21:12:19 GMT 2023-05-31T21:35:42.433 INFO:tasks.rgw_multisite_tests:/pklxkd-64/obj22?versionId=null 2023-05-31T21:35:42.433 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-05-31T21:35:42.433 INFO:tasks.rgw_multisite_tests:AWS Q80H11OLNU1S48PPYL26:OZnIiljRkUdTAdQuQcOP60YYcNM= 2023-05-31T21:35:42.433 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 21:12:19 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:OZnIiljRkUdTAdQuQcOP60YYcNM=', 'Content-Length': '0'} 2023-05-31T21:35:42.434 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 21:11:49 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e0ee5c04fe00e14d-006477b833-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 21:12:19 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:35:42.434 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/pklxkd-64/ 2023-05-31T21:35:42.434 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/pklxkd-64/ 2023-05-31T21:35:42.434 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/pklxkd-64/?prefix=obj22 2023-05-31T21:35:42.435 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/pklxkd-64/?prefix=obj22 2023-05-31T21:35:42.435 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-05-31T21:35:42.435 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /pklxkd-64/?prefix=obj22 2023-05-31T21:35:42.435 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-05-31T21:35:42.436 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-05-31T21:35:42.436 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T21:35:42.436 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-05-31T21:35:42.436 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-05-31T21:35:42.437 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-05-31T21:35:42.437 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-05-31T21:35:42.437 INFO:tasks.rgw_multisite_tests:GET 2023-05-31T21:35:42.437 INFO:tasks.rgw_multisite_tests: 2023-05-31T21:35:42.437 INFO:tasks.rgw_multisite_tests: 2023-05-31T21:35:42.438 INFO:tasks.rgw_multisite_tests:Wed, 31 May 2023 21:12:19 GMT 2023-05-31T21:35:42.438 INFO:tasks.rgw_multisite_tests:/pklxkd-64/ 2023-05-31T21:35:42.438 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-05-31T21:35:42.438 INFO:tasks.rgw_multisite_tests:AWS Q80H11OLNU1S48PPYL26:/ACajcjJ+wTjnzFp03OUBb/7FmU= 2023-05-31T21:35:42.439 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 21:12:19 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:/ACajcjJ+wTjnzFp03OUBb/7FmU=', 'Content-Length': '0'} 2023-05-31T21:35:42.439 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c9dc0401efa98e1f-006477b833-1fb6-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 21:12:19 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:35:42.439 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'pklxkd-64obj221000falseobj222023-05-31T21:11:49.055Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T21:35:42.439 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/pklxkd-64/ 2023-05-31T21:35:42.440 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/pklxkd-64/ 2023-05-31T21:35:42.440 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/pklxkd-64/?prefix=obj22 2023-05-31T21:35:42.440 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/pklxkd-64/?prefix=obj22 2023-05-31T21:35:42.440 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-05-31T21:35:42.440 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /pklxkd-64/?prefix=obj22 2023-05-31T21:35:42.441 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-05-31T21:35:42.441 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-05-31T21:35:42.441 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T21:35:42.441 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-05-31T21:35:42.442 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-05-31T21:35:42.442 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-05-31T21:35:42.442 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-05-31T21:35:42.442 INFO:tasks.rgw_multisite_tests:GET 2023-05-31T21:35:42.442 INFO:tasks.rgw_multisite_tests: 2023-05-31T21:35:42.443 INFO:tasks.rgw_multisite_tests: 2023-05-31T21:35:42.443 INFO:tasks.rgw_multisite_tests:Wed, 31 May 2023 21:12:19 GMT 2023-05-31T21:35:42.443 INFO:tasks.rgw_multisite_tests:/pklxkd-64/ 2023-05-31T21:35:42.443 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-05-31T21:35:42.444 INFO:tasks.rgw_multisite_tests:AWS Q80H11OLNU1S48PPYL26:/ACajcjJ+wTjnzFp03OUBb/7FmU= 2023-05-31T21:35:42.444 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 21:12:19 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:/ACajcjJ+wTjnzFp03OUBb/7FmU=', 'Content-Length': '0'} 2023-05-31T21:35:42.444 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000076226418bf2b7ba5-006477b833-1630-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 21:12:19 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:35:42.444 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'pklxkd-64obj221000falseobj222023-05-31T21:11:49.055Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T21:35:42.445 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/pklxkd-64/obj22 2023-05-31T21:35:42.445 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/pklxkd-64/obj22 2023-05-31T21:35:42.445 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-05-31T21:35:42.445 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /pklxkd-64/obj22 2023-05-31T21:35:42.446 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-05-31T21:35:42.446 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-05-31T21:35:42.446 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T21:35:42.446 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-05-31T21:35:42.447 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-05-31T21:35:42.447 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-05-31T21:35:42.447 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-05-31T21:35:42.447 INFO:tasks.rgw_multisite_tests:HEAD 2023-05-31T21:35:42.447 INFO:tasks.rgw_multisite_tests: 2023-05-31T21:35:42.448 INFO:tasks.rgw_multisite_tests: 2023-05-31T21:35:42.448 INFO:tasks.rgw_multisite_tests:Wed, 31 May 2023 21:12:19 GMT 2023-05-31T21:35:42.448 INFO:tasks.rgw_multisite_tests:/pklxkd-64/obj22 2023-05-31T21:35:42.448 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-05-31T21:35:42.449 INFO:tasks.rgw_multisite_tests:AWS Q80H11OLNU1S48PPYL26:xbUN1gny4Y3f+TfLN6Z/p45Qfa4= 2023-05-31T21:35:42.449 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 21:12:19 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:xbUN1gny4Y3f+TfLN6Z/p45Qfa4=', 'Content-Length': '0'} 2023-05-31T21:35:42.449 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 21:11:49 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000921218d1fbf10413-006477b833-1fb6-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 21:12:19 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:35:42.449 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/pklxkd-64/obj22 2023-05-31T21:35:42.450 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/pklxkd-64/obj22 2023-05-31T21:35:42.450 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-05-31T21:35:42.450 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /pklxkd-64/obj22 2023-05-31T21:35:42.450 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-05-31T21:35:42.450 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-05-31T21:35:42.451 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T21:35:42.451 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-05-31T21:35:42.451 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-05-31T21:35:42.451 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-05-31T21:35:42.452 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-05-31T21:35:42.452 INFO:tasks.rgw_multisite_tests:HEAD 2023-05-31T21:35:42.452 INFO:tasks.rgw_multisite_tests: 2023-05-31T21:35:42.452 INFO:tasks.rgw_multisite_tests: 2023-05-31T21:35:42.453 INFO:tasks.rgw_multisite_tests:Wed, 31 May 2023 21:12:19 GMT 2023-05-31T21:35:42.453 INFO:tasks.rgw_multisite_tests:/pklxkd-64/obj22 2023-05-31T21:35:42.453 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-05-31T21:35:42.453 INFO:tasks.rgw_multisite_tests:AWS Q80H11OLNU1S48PPYL26:xbUN1gny4Y3f+TfLN6Z/p45Qfa4= 2023-05-31T21:35:42.453 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 21:12:19 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:xbUN1gny4Y3f+TfLN6Z/p45Qfa4=', 'Content-Length': '0'} 2023-05-31T21:35:42.454 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 21:11:49 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008dc2e38ab4c8027c-006477b833-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 21:12:19 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:35:42.454 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: comparing key name=obj22 2023-05-31T21:35:42.454 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/pklxkd-64/obj22 2023-05-31T21:35:42.454 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/pklxkd-64/obj22 2023-05-31T21:35:42.455 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-05-31T21:35:42.455 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /pklxkd-64/obj22 2023-05-31T21:35:42.455 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-05-31T21:35:42.455 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-05-31T21:35:42.456 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T21:35:42.456 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-05-31T21:35:42.456 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-05-31T21:35:42.456 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-05-31T21:35:42.456 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-05-31T21:35:42.457 INFO:tasks.rgw_multisite_tests:GET 2023-05-31T21:35:42.457 INFO:tasks.rgw_multisite_tests: 2023-05-31T21:35:42.457 INFO:tasks.rgw_multisite_tests: 2023-05-31T21:35:42.457 INFO:tasks.rgw_multisite_tests:Wed, 31 May 2023 21:12:19 GMT 2023-05-31T21:35:42.458 INFO:tasks.rgw_multisite_tests:/pklxkd-64/obj22 2023-05-31T21:35:42.458 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-05-31T21:35:42.458 INFO:tasks.rgw_multisite_tests:AWS Q80H11OLNU1S48PPYL26:LqymWWd+I4aNdHpCpypwt512aNY= 2023-05-31T21:35:42.458 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 21:12:19 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:LqymWWd+I4aNdHpCpypwt512aNY=', 'Content-Length': '0'} 2023-05-31T21:35:42.459 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 21:11:49 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000034db1197e1549773-006477b833-1fb6-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 21:12:19 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:35:42.459 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/pklxkd-64/obj22 2023-05-31T21:35:42.459 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/pklxkd-64/obj22 2023-05-31T21:35:42.459 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-05-31T21:35:42.459 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /pklxkd-64/obj22 2023-05-31T21:35:42.460 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-05-31T21:35:42.460 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-05-31T21:35:42.460 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T21:35:42.460 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-05-31T21:35:42.461 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-05-31T21:35:42.461 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-05-31T21:35:42.461 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-05-31T21:35:42.461 INFO:tasks.rgw_multisite_tests:GET 2023-05-31T21:35:42.462 INFO:tasks.rgw_multisite_tests: 2023-05-31T21:35:42.462 INFO:tasks.rgw_multisite_tests: 2023-05-31T21:35:42.462 INFO:tasks.rgw_multisite_tests:Wed, 31 May 2023 21:12:19 GMT 2023-05-31T21:35:42.462 INFO:tasks.rgw_multisite_tests:/pklxkd-64/obj22 2023-05-31T21:35:42.462 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-05-31T21:35:42.463 INFO:tasks.rgw_multisite_tests:AWS Q80H11OLNU1S48PPYL26:LqymWWd+I4aNdHpCpypwt512aNY= 2023-05-31T21:35:42.463 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 21:12:19 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:LqymWWd+I4aNdHpCpypwt512aNY=', 'Content-Length': '0'} 2023-05-31T21:35:42.463 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 21:11:49 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000013d5f4c7b81b6fcf-006477b833-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 21:12:19 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:35:42.463 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/pklxkd-64/ 2023-05-31T21:35:42.464 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/pklxkd-64/ 2023-05-31T21:35:42.464 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/pklxkd-64/?prefix=obj22 2023-05-31T21:35:42.464 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/pklxkd-64/?prefix=obj22 2023-05-31T21:35:42.464 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-05-31T21:35:42.465 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /pklxkd-64/?prefix=obj22 2023-05-31T21:35:42.465 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-05-31T21:35:42.465 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-05-31T21:35:42.465 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T21:35:42.465 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-05-31T21:35:42.466 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-05-31T21:35:42.466 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-05-31T21:35:42.466 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-05-31T21:35:42.466 INFO:tasks.rgw_multisite_tests:GET 2023-05-31T21:35:42.467 INFO:tasks.rgw_multisite_tests: 2023-05-31T21:35:42.467 INFO:tasks.rgw_multisite_tests: 2023-05-31T21:35:42.467 INFO:tasks.rgw_multisite_tests:Wed, 31 May 2023 21:12:19 GMT 2023-05-31T21:35:42.467 INFO:tasks.rgw_multisite_tests:/pklxkd-64/ 2023-05-31T21:35:42.467 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-05-31T21:35:42.468 INFO:tasks.rgw_multisite_tests:AWS Q80H11OLNU1S48PPYL26:/ACajcjJ+wTjnzFp03OUBb/7FmU= 2023-05-31T21:35:42.468 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 21:12:19 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:/ACajcjJ+wTjnzFp03OUBb/7FmU=', 'Content-Length': '0'} 2023-05-31T21:35:42.468 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000498eece8668cc6c1-006477b833-1fb6-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 21:12:19 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:35:42.468 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'pklxkd-64obj221000falseobj222023-05-31T21:11:49.055Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T21:35:42.469 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/pklxkd-64/ 2023-05-31T21:35:42.469 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/pklxkd-64/ 2023-05-31T21:35:42.469 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/pklxkd-64/?prefix=obj22 2023-05-31T21:35:42.469 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/pklxkd-64/?prefix=obj22 2023-05-31T21:35:42.470 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-05-31T21:35:42.470 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /pklxkd-64/?prefix=obj22 2023-05-31T21:35:42.470 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-05-31T21:35:42.470 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-05-31T21:35:42.471 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T21:35:42.471 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-05-31T21:35:42.471 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-05-31T21:35:42.471 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-05-31T21:35:42.471 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-05-31T21:35:42.472 INFO:tasks.rgw_multisite_tests:GET 2023-05-31T21:35:42.472 INFO:tasks.rgw_multisite_tests: 2023-05-31T21:35:42.472 INFO:tasks.rgw_multisite_tests: 2023-05-31T21:35:42.472 INFO:tasks.rgw_multisite_tests:Wed, 31 May 2023 21:12:19 GMT 2023-05-31T21:35:42.473 INFO:tasks.rgw_multisite_tests:/pklxkd-64/ 2023-05-31T21:35:42.473 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-05-31T21:35:42.473 INFO:tasks.rgw_multisite_tests:AWS Q80H11OLNU1S48PPYL26:/ACajcjJ+wTjnzFp03OUBb/7FmU= 2023-05-31T21:35:42.473 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 21:12:19 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:/ACajcjJ+wTjnzFp03OUBb/7FmU=', 'Content-Length': '0'} 2023-05-31T21:35:42.474 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000009d7b945442a28b01-006477b833-1630-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 21:12:19 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:35:42.474 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'pklxkd-64obj221000falseobj222023-05-31T21:11:49.055Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T21:35:42.474 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: comparing key name=obj33 2023-05-31T21:35:42.474 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/pklxkd-64/obj33 2023-05-31T21:35:42.474 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/pklxkd-64/obj33 2023-05-31T21:35:42.475 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/pklxkd-64/obj33?versionId=null 2023-05-31T21:35:42.475 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/pklxkd-64/obj33?versionId=null 2023-05-31T21:35:42.475 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-05-31T21:35:42.475 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /pklxkd-64/obj33?versionId=null 2023-05-31T21:35:42.476 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-05-31T21:35:42.476 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-05-31T21:35:42.476 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T21:35:42.476 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-05-31T21:35:42.477 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-05-31T21:35:42.477 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-05-31T21:35:42.477 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-05-31T21:35:42.477 INFO:tasks.rgw_multisite_tests:GET 2023-05-31T21:35:42.477 INFO:tasks.rgw_multisite_tests: 2023-05-31T21:35:42.478 INFO:tasks.rgw_multisite_tests: 2023-05-31T21:35:42.478 INFO:tasks.rgw_multisite_tests:Wed, 31 May 2023 21:12:19 GMT 2023-05-31T21:35:42.478 INFO:tasks.rgw_multisite_tests:/pklxkd-64/obj33?versionId=null 2023-05-31T21:35:42.478 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-05-31T21:35:42.479 INFO:tasks.rgw_multisite_tests:AWS Q80H11OLNU1S48PPYL26:iYoF5oE6hcynDhEPAEuGmTfaklQ= 2023-05-31T21:35:42.479 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 21:12:19 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:iYoF5oE6hcynDhEPAEuGmTfaklQ=', 'Content-Length': '0'} 2023-05-31T21:35:42.479 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 21:11:49 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000063cd4400e3239841-006477b833-1fb6-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 21:12:19 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:35:42.479 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/pklxkd-64/obj33 2023-05-31T21:35:42.480 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/pklxkd-64/obj33 2023-05-31T21:35:42.480 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/pklxkd-64/obj33?versionId=null 2023-05-31T21:35:42.480 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/pklxkd-64/obj33?versionId=null 2023-05-31T21:35:42.480 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-05-31T21:35:42.481 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /pklxkd-64/obj33?versionId=null 2023-05-31T21:35:42.481 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-05-31T21:35:42.481 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-05-31T21:35:42.481 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T21:35:42.481 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-05-31T21:35:42.482 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-05-31T21:35:42.482 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-05-31T21:35:42.482 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-05-31T21:35:42.482 INFO:tasks.rgw_multisite_tests:GET 2023-05-31T21:35:42.483 INFO:tasks.rgw_multisite_tests: 2023-05-31T21:35:42.483 INFO:tasks.rgw_multisite_tests: 2023-05-31T21:35:42.483 INFO:tasks.rgw_multisite_tests:Wed, 31 May 2023 21:12:19 GMT 2023-05-31T21:35:42.483 INFO:tasks.rgw_multisite_tests:/pklxkd-64/obj33?versionId=null 2023-05-31T21:35:42.483 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-05-31T21:35:42.484 INFO:tasks.rgw_multisite_tests:AWS Q80H11OLNU1S48PPYL26:iYoF5oE6hcynDhEPAEuGmTfaklQ= 2023-05-31T21:35:42.484 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 21:12:19 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:iYoF5oE6hcynDhEPAEuGmTfaklQ=', 'Content-Length': '0'} 2023-05-31T21:35:42.484 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 21:11:49 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007d67214e85dc9510-006477b833-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 21:12:19 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:35:42.485 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/pklxkd-64/obj33 2023-05-31T21:35:42.485 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/pklxkd-64/obj33 2023-05-31T21:35:42.485 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/pklxkd-64/obj33?versionId=null 2023-05-31T21:35:42.485 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/pklxkd-64/obj33?versionId=null 2023-05-31T21:35:42.486 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-05-31T21:35:42.486 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /pklxkd-64/obj33?versionId=null 2023-05-31T21:35:42.486 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-05-31T21:35:42.486 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-05-31T21:35:42.487 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T21:35:42.487 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-05-31T21:35:42.487 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-05-31T21:35:42.487 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-05-31T21:35:42.487 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-05-31T21:35:42.488 INFO:tasks.rgw_multisite_tests:HEAD 2023-05-31T21:35:42.488 INFO:tasks.rgw_multisite_tests: 2023-05-31T21:35:42.488 INFO:tasks.rgw_multisite_tests: 2023-05-31T21:35:42.488 INFO:tasks.rgw_multisite_tests:Wed, 31 May 2023 21:12:19 GMT 2023-05-31T21:35:42.489 INFO:tasks.rgw_multisite_tests:/pklxkd-64/obj33?versionId=null 2023-05-31T21:35:42.489 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-05-31T21:35:42.489 INFO:tasks.rgw_multisite_tests:AWS Q80H11OLNU1S48PPYL26:TcU7F0hzqnrh05Rc4sPTJ3X8B6k= 2023-05-31T21:35:42.489 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 21:12:19 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:TcU7F0hzqnrh05Rc4sPTJ3X8B6k=', 'Content-Length': '0'} 2023-05-31T21:35:42.490 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 21:11:49 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009549219113534b95-006477b833-1fb6-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 21:12:19 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:35:42.490 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/pklxkd-64/obj33 2023-05-31T21:35:42.490 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/pklxkd-64/obj33 2023-05-31T21:35:42.490 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/pklxkd-64/obj33?versionId=null 2023-05-31T21:35:42.490 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/pklxkd-64/obj33?versionId=null 2023-05-31T21:35:42.491 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-05-31T21:35:42.491 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /pklxkd-64/obj33?versionId=null 2023-05-31T21:35:42.491 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-05-31T21:35:42.491 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-05-31T21:35:42.492 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T21:35:42.492 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-05-31T21:35:42.492 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-05-31T21:35:42.492 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-05-31T21:35:42.493 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-05-31T21:35:42.493 INFO:tasks.rgw_multisite_tests:HEAD 2023-05-31T21:35:42.493 INFO:tasks.rgw_multisite_tests: 2023-05-31T21:35:42.493 INFO:tasks.rgw_multisite_tests: 2023-05-31T21:35:42.494 INFO:tasks.rgw_multisite_tests:Wed, 31 May 2023 21:12:19 GMT 2023-05-31T21:35:42.494 INFO:tasks.rgw_multisite_tests:/pklxkd-64/obj33?versionId=null 2023-05-31T21:35:42.494 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-05-31T21:35:42.494 INFO:tasks.rgw_multisite_tests:AWS Q80H11OLNU1S48PPYL26:TcU7F0hzqnrh05Rc4sPTJ3X8B6k= 2023-05-31T21:35:42.494 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 21:12:19 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:TcU7F0hzqnrh05Rc4sPTJ3X8B6k=', 'Content-Length': '0'} 2023-05-31T21:35:42.495 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 21:11:49 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b39a0f55c0643646-006477b833-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 21:12:19 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:35:42.495 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: comparing key name=obj33 2023-05-31T21:35:42.495 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/pklxkd-64/obj33 2023-05-31T21:35:42.495 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/pklxkd-64/obj33 2023-05-31T21:35:42.496 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/pklxkd-64/obj33?versionId=null 2023-05-31T21:35:42.496 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/pklxkd-64/obj33?versionId=null 2023-05-31T21:35:42.496 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-05-31T21:35:42.496 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /pklxkd-64/obj33?versionId=null 2023-05-31T21:35:42.497 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-05-31T21:35:42.497 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-05-31T21:35:42.497 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T21:35:42.497 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-05-31T21:35:42.498 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-05-31T21:35:42.498 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-05-31T21:35:42.498 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-05-31T21:35:42.498 INFO:tasks.rgw_multisite_tests:GET 2023-05-31T21:35:42.498 INFO:tasks.rgw_multisite_tests: 2023-05-31T21:35:42.499 INFO:tasks.rgw_multisite_tests: 2023-05-31T21:35:42.499 INFO:tasks.rgw_multisite_tests:Wed, 31 May 2023 21:12:19 GMT 2023-05-31T21:35:42.499 INFO:tasks.rgw_multisite_tests:/pklxkd-64/obj33?versionId=null 2023-05-31T21:35:42.499 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-05-31T21:35:42.500 INFO:tasks.rgw_multisite_tests:AWS Q80H11OLNU1S48PPYL26:iYoF5oE6hcynDhEPAEuGmTfaklQ= 2023-05-31T21:35:42.500 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 21:12:19 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:iYoF5oE6hcynDhEPAEuGmTfaklQ=', 'Content-Length': '0'} 2023-05-31T21:35:42.500 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 21:11:49 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000f27337af0ad23bf-006477b833-1fb6-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 21:12:19 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:35:42.500 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/pklxkd-64/obj33 2023-05-31T21:35:42.501 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/pklxkd-64/obj33 2023-05-31T21:35:42.501 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/pklxkd-64/obj33?versionId=null 2023-05-31T21:35:42.501 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/pklxkd-64/obj33?versionId=null 2023-05-31T21:35:42.501 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-05-31T21:35:42.501 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /pklxkd-64/obj33?versionId=null 2023-05-31T21:35:42.502 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-05-31T21:35:42.502 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-05-31T21:35:42.502 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T21:35:42.502 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-05-31T21:35:42.503 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-05-31T21:35:42.503 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-05-31T21:35:42.503 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-05-31T21:35:42.503 INFO:tasks.rgw_multisite_tests:GET 2023-05-31T21:35:42.504 INFO:tasks.rgw_multisite_tests: 2023-05-31T21:35:42.504 INFO:tasks.rgw_multisite_tests: 2023-05-31T21:35:42.504 INFO:tasks.rgw_multisite_tests:Wed, 31 May 2023 21:12:19 GMT 2023-05-31T21:35:42.504 INFO:tasks.rgw_multisite_tests:/pklxkd-64/obj33?versionId=null 2023-05-31T21:35:42.504 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-05-31T21:35:42.505 INFO:tasks.rgw_multisite_tests:AWS Q80H11OLNU1S48PPYL26:iYoF5oE6hcynDhEPAEuGmTfaklQ= 2023-05-31T21:35:42.505 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 21:12:19 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:iYoF5oE6hcynDhEPAEuGmTfaklQ=', 'Content-Length': '0'} 2023-05-31T21:35:42.505 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 21:11:49 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004c9e36fe1be1349b-006477b833-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 21:12:19 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:35:42.505 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/pklxkd-64/ 2023-05-31T21:35:42.506 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/pklxkd-64/ 2023-05-31T21:35:42.506 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/pklxkd-64/?prefix=obj33 2023-05-31T21:35:42.506 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/pklxkd-64/?prefix=obj33 2023-05-31T21:35:42.506 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-05-31T21:35:42.507 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /pklxkd-64/?prefix=obj33 2023-05-31T21:35:42.507 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-05-31T21:35:42.507 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-05-31T21:35:42.507 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T21:35:42.507 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-05-31T21:35:42.508 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-05-31T21:35:42.508 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-05-31T21:35:42.508 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-05-31T21:35:42.508 INFO:tasks.rgw_multisite_tests:GET 2023-05-31T21:35:42.509 INFO:tasks.rgw_multisite_tests: 2023-05-31T21:35:42.509 INFO:tasks.rgw_multisite_tests: 2023-05-31T21:35:42.509 INFO:tasks.rgw_multisite_tests:Wed, 31 May 2023 21:12:19 GMT 2023-05-31T21:35:42.509 INFO:tasks.rgw_multisite_tests:/pklxkd-64/ 2023-05-31T21:35:42.510 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-05-31T21:35:42.510 INFO:tasks.rgw_multisite_tests:AWS Q80H11OLNU1S48PPYL26:/ACajcjJ+wTjnzFp03OUBb/7FmU= 2023-05-31T21:35:42.510 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 21:12:19 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:/ACajcjJ+wTjnzFp03OUBb/7FmU=', 'Content-Length': '0'} 2023-05-31T21:35:42.510 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c081e3df9d2a930e-006477b833-1fb6-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 21:12:19 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:35:42.510 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'pklxkd-64obj331000falseobj332023-05-31T21:11:49.072Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T21:35:42.511 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/pklxkd-64/ 2023-05-31T21:35:42.511 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/pklxkd-64/ 2023-05-31T21:35:42.511 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/pklxkd-64/?prefix=obj33 2023-05-31T21:35:42.511 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/pklxkd-64/?prefix=obj33 2023-05-31T21:35:42.512 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-05-31T21:35:42.512 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /pklxkd-64/?prefix=obj33 2023-05-31T21:35:42.512 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-05-31T21:35:42.512 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-05-31T21:35:42.513 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T21:35:42.513 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-05-31T21:35:42.513 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-05-31T21:35:42.513 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-05-31T21:35:42.513 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-05-31T21:35:42.514 INFO:tasks.rgw_multisite_tests:GET 2023-05-31T21:35:42.514 INFO:tasks.rgw_multisite_tests: 2023-05-31T21:35:42.514 INFO:tasks.rgw_multisite_tests: 2023-05-31T21:35:42.514 INFO:tasks.rgw_multisite_tests:Wed, 31 May 2023 21:12:19 GMT 2023-05-31T21:35:42.515 INFO:tasks.rgw_multisite_tests:/pklxkd-64/ 2023-05-31T21:35:42.515 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-05-31T21:35:42.515 INFO:tasks.rgw_multisite_tests:AWS Q80H11OLNU1S48PPYL26:/ACajcjJ+wTjnzFp03OUBb/7FmU= 2023-05-31T21:35:42.515 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 21:12:19 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:/ACajcjJ+wTjnzFp03OUBb/7FmU=', 'Content-Length': '0'} 2023-05-31T21:35:42.515 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000023995f2146040a59-006477b833-1630-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 21:12:19 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:35:42.516 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'pklxkd-64obj331000falseobj332023-05-31T21:11:49.072Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T21:35:42.516 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/pklxkd-64/obj33 2023-05-31T21:35:42.516 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/pklxkd-64/obj33 2023-05-31T21:35:42.516 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-05-31T21:35:42.517 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /pklxkd-64/obj33 2023-05-31T21:35:42.517 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-05-31T21:35:42.517 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-05-31T21:35:42.517 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T21:35:42.518 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-05-31T21:35:42.518 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-05-31T21:35:42.518 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-05-31T21:35:42.518 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-05-31T21:35:42.519 INFO:tasks.rgw_multisite_tests:HEAD 2023-05-31T21:35:42.519 INFO:tasks.rgw_multisite_tests: 2023-05-31T21:35:42.519 INFO:tasks.rgw_multisite_tests: 2023-05-31T21:35:42.519 INFO:tasks.rgw_multisite_tests:Wed, 31 May 2023 21:12:19 GMT 2023-05-31T21:35:42.519 INFO:tasks.rgw_multisite_tests:/pklxkd-64/obj33 2023-05-31T21:35:42.520 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-05-31T21:35:42.520 INFO:tasks.rgw_multisite_tests:AWS Q80H11OLNU1S48PPYL26:mRCtkdEvAE91nG83ueDl6lKeNdE= 2023-05-31T21:35:42.520 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 21:12:19 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:mRCtkdEvAE91nG83ueDl6lKeNdE=', 'Content-Length': '0'} 2023-05-31T21:35:42.520 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 21:11:49 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006efed62d7d44ebbf-006477b833-1fb6-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 21:12:19 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:35:42.521 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/pklxkd-64/obj33 2023-05-31T21:35:42.521 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/pklxkd-64/obj33 2023-05-31T21:35:42.521 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-05-31T21:35:42.521 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /pklxkd-64/obj33 2023-05-31T21:35:42.522 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-05-31T21:35:42.522 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-05-31T21:35:42.522 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T21:35:42.522 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-05-31T21:35:42.523 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-05-31T21:35:42.523 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-05-31T21:35:42.523 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-05-31T21:35:42.523 INFO:tasks.rgw_multisite_tests:HEAD 2023-05-31T21:35:42.523 INFO:tasks.rgw_multisite_tests: 2023-05-31T21:35:42.524 INFO:tasks.rgw_multisite_tests: 2023-05-31T21:35:42.524 INFO:tasks.rgw_multisite_tests:Wed, 31 May 2023 21:12:19 GMT 2023-05-31T21:35:42.524 INFO:tasks.rgw_multisite_tests:/pklxkd-64/obj33 2023-05-31T21:35:42.524 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-05-31T21:35:42.525 INFO:tasks.rgw_multisite_tests:AWS Q80H11OLNU1S48PPYL26:mRCtkdEvAE91nG83ueDl6lKeNdE= 2023-05-31T21:35:42.525 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 21:12:19 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:mRCtkdEvAE91nG83ueDl6lKeNdE=', 'Content-Length': '0'} 2023-05-31T21:35:42.525 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 21:11:49 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000575ad72a3c5e51fc-006477b833-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 21:12:19 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:35:42.525 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: comparing key name=obj33 2023-05-31T21:35:42.526 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/pklxkd-64/obj33 2023-05-31T21:35:42.526 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/pklxkd-64/obj33 2023-05-31T21:35:42.526 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-05-31T21:35:42.526 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /pklxkd-64/obj33 2023-05-31T21:35:42.526 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-05-31T21:35:42.527 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-05-31T21:35:42.527 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T21:35:42.527 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-05-31T21:35:42.527 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-05-31T21:35:42.528 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-05-31T21:35:42.528 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-05-31T21:35:42.528 INFO:tasks.rgw_multisite_tests:GET 2023-05-31T21:35:42.528 INFO:tasks.rgw_multisite_tests: 2023-05-31T21:35:42.529 INFO:tasks.rgw_multisite_tests: 2023-05-31T21:35:42.529 INFO:tasks.rgw_multisite_tests:Wed, 31 May 2023 21:12:19 GMT 2023-05-31T21:35:42.529 INFO:tasks.rgw_multisite_tests:/pklxkd-64/obj33 2023-05-31T21:35:42.529 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-05-31T21:35:42.529 INFO:tasks.rgw_multisite_tests:AWS Q80H11OLNU1S48PPYL26:5vzeOAOjDn24xq6OH9FzejuQMMc= 2023-05-31T21:35:42.530 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 21:12:19 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:5vzeOAOjDn24xq6OH9FzejuQMMc=', 'Content-Length': '0'} 2023-05-31T21:35:42.530 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 21:11:49 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000de4e4dfbd48df35c-006477b833-1fb6-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 21:12:19 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:35:42.530 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/pklxkd-64/obj33 2023-05-31T21:35:42.530 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/pklxkd-64/obj33 2023-05-31T21:35:42.531 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-05-31T21:35:42.531 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /pklxkd-64/obj33 2023-05-31T21:35:42.531 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-05-31T21:35:42.531 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-05-31T21:35:42.532 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T21:35:42.532 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-05-31T21:35:42.532 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-05-31T21:35:42.532 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-05-31T21:35:42.532 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-05-31T21:35:42.533 INFO:tasks.rgw_multisite_tests:GET 2023-05-31T21:35:42.533 INFO:tasks.rgw_multisite_tests: 2023-05-31T21:35:42.533 INFO:tasks.rgw_multisite_tests: 2023-05-31T21:35:42.533 INFO:tasks.rgw_multisite_tests:Wed, 31 May 2023 21:12:19 GMT 2023-05-31T21:35:42.534 INFO:tasks.rgw_multisite_tests:/pklxkd-64/obj33 2023-05-31T21:35:42.534 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-05-31T21:35:42.534 INFO:tasks.rgw_multisite_tests:AWS Q80H11OLNU1S48PPYL26:5vzeOAOjDn24xq6OH9FzejuQMMc= 2023-05-31T21:35:42.534 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 21:12:19 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:5vzeOAOjDn24xq6OH9FzejuQMMc=', 'Content-Length': '0'} 2023-05-31T21:35:42.535 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 21:11:49 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a1b8646e1ab404de-006477b833-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 21:12:19 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:35:42.535 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/pklxkd-64/ 2023-05-31T21:35:42.535 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/pklxkd-64/ 2023-05-31T21:35:42.535 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/pklxkd-64/?prefix=obj33 2023-05-31T21:35:42.535 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/pklxkd-64/?prefix=obj33 2023-05-31T21:35:42.536 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-05-31T21:35:42.536 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /pklxkd-64/?prefix=obj33 2023-05-31T21:35:42.536 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-05-31T21:35:42.536 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-05-31T21:35:42.537 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T21:35:42.537 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-05-31T21:35:42.537 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-05-31T21:35:42.537 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-05-31T21:35:42.538 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-05-31T21:35:42.538 INFO:tasks.rgw_multisite_tests:GET 2023-05-31T21:35:42.538 INFO:tasks.rgw_multisite_tests: 2023-05-31T21:35:42.538 INFO:tasks.rgw_multisite_tests: 2023-05-31T21:35:42.538 INFO:tasks.rgw_multisite_tests:Wed, 31 May 2023 21:12:19 GMT 2023-05-31T21:35:42.539 INFO:tasks.rgw_multisite_tests:/pklxkd-64/ 2023-05-31T21:35:42.539 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-05-31T21:35:42.539 INFO:tasks.rgw_multisite_tests:AWS Q80H11OLNU1S48PPYL26:/ACajcjJ+wTjnzFp03OUBb/7FmU= 2023-05-31T21:35:42.539 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 21:12:19 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:/ACajcjJ+wTjnzFp03OUBb/7FmU=', 'Content-Length': '0'} 2023-05-31T21:35:42.540 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000001ee9029dabb1fd07-006477b833-1fb6-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 21:12:19 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:35:42.540 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'pklxkd-64obj331000falseobj332023-05-31T21:11:49.072Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T21:35:42.540 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/pklxkd-64/ 2023-05-31T21:35:42.540 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/pklxkd-64/ 2023-05-31T21:35:42.541 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/pklxkd-64/?prefix=obj33 2023-05-31T21:35:42.541 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/pklxkd-64/?prefix=obj33 2023-05-31T21:35:42.541 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-05-31T21:35:42.541 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /pklxkd-64/?prefix=obj33 2023-05-31T21:35:42.541 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-05-31T21:35:42.542 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-05-31T21:35:42.542 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T21:35:42.542 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-05-31T21:35:42.542 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-05-31T21:35:42.543 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-05-31T21:35:42.543 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-05-31T21:35:42.543 INFO:tasks.rgw_multisite_tests:GET 2023-05-31T21:35:42.543 INFO:tasks.rgw_multisite_tests: 2023-05-31T21:35:42.543 INFO:tasks.rgw_multisite_tests: 2023-05-31T21:35:42.544 INFO:tasks.rgw_multisite_tests:Wed, 31 May 2023 21:12:19 GMT 2023-05-31T21:35:42.544 INFO:tasks.rgw_multisite_tests:/pklxkd-64/ 2023-05-31T21:35:42.544 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-05-31T21:35:42.544 INFO:tasks.rgw_multisite_tests:AWS Q80H11OLNU1S48PPYL26:/ACajcjJ+wTjnzFp03OUBb/7FmU= 2023-05-31T21:35:42.545 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 21:12:19 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:/ACajcjJ+wTjnzFp03OUBb/7FmU=', 'Content-Length': '0'} 2023-05-31T21:35:42.545 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a43dca39185a5522-006477b833-1630-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 21:12:19 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:35:42.545 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'pklxkd-64obj331000falseobj332023-05-31T21:11:49.072Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T21:35:42.545 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: comparing key name=obj44 2023-05-31T21:35:42.546 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/pklxkd-64/obj44 2023-05-31T21:35:42.546 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/pklxkd-64/obj44 2023-05-31T21:35:42.546 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/pklxkd-64/obj44?versionId=null 2023-05-31T21:35:42.546 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/pklxkd-64/obj44?versionId=null 2023-05-31T21:35:42.546 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-05-31T21:35:42.547 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /pklxkd-64/obj44?versionId=null 2023-05-31T21:35:42.547 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-05-31T21:35:42.547 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-05-31T21:35:42.547 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T21:35:42.548 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-05-31T21:35:42.548 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-05-31T21:35:42.548 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-05-31T21:35:42.548 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-05-31T21:35:42.549 INFO:tasks.rgw_multisite_tests:GET 2023-05-31T21:35:42.549 INFO:tasks.rgw_multisite_tests: 2023-05-31T21:35:42.549 INFO:tasks.rgw_multisite_tests: 2023-05-31T21:35:42.549 INFO:tasks.rgw_multisite_tests:Wed, 31 May 2023 21:12:19 GMT 2023-05-31T21:35:42.549 INFO:tasks.rgw_multisite_tests:/pklxkd-64/obj44?versionId=null 2023-05-31T21:35:42.550 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-05-31T21:35:42.550 INFO:tasks.rgw_multisite_tests:AWS Q80H11OLNU1S48PPYL26:z9fRV0/Yh/KKEef4iAzn+B+wCIQ= 2023-05-31T21:35:42.550 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 21:12:19 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:z9fRV0/Yh/KKEef4iAzn+B+wCIQ=', 'Content-Length': '0'} 2023-05-31T21:35:42.550 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 21:11:49 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003b0401011baccec4-006477b833-1fb6-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 21:12:19 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:35:42.551 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/pklxkd-64/obj44 2023-05-31T21:35:42.551 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/pklxkd-64/obj44 2023-05-31T21:35:42.551 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/pklxkd-64/obj44?versionId=null 2023-05-31T21:35:42.551 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/pklxkd-64/obj44?versionId=null 2023-05-31T21:35:42.552 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-05-31T21:35:42.552 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /pklxkd-64/obj44?versionId=null 2023-05-31T21:35:42.552 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-05-31T21:35:42.552 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-05-31T21:35:42.552 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T21:35:42.553 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-05-31T21:35:42.553 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-05-31T21:35:42.553 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-05-31T21:35:42.553 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-05-31T21:35:42.554 INFO:tasks.rgw_multisite_tests:GET 2023-05-31T21:35:42.554 INFO:tasks.rgw_multisite_tests: 2023-05-31T21:35:42.554 INFO:tasks.rgw_multisite_tests: 2023-05-31T21:35:42.554 INFO:tasks.rgw_multisite_tests:Wed, 31 May 2023 21:12:19 GMT 2023-05-31T21:35:42.555 INFO:tasks.rgw_multisite_tests:/pklxkd-64/obj44?versionId=null 2023-05-31T21:35:42.555 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-05-31T21:35:42.555 INFO:tasks.rgw_multisite_tests:AWS Q80H11OLNU1S48PPYL26:z9fRV0/Yh/KKEef4iAzn+B+wCIQ= 2023-05-31T21:35:42.555 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 21:12:19 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:z9fRV0/Yh/KKEef4iAzn+B+wCIQ=', 'Content-Length': '0'} 2023-05-31T21:35:42.556 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 21:11:49 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006b6d1e8ee42fb8c5-006477b833-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 21:12:19 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:35:42.556 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/pklxkd-64/obj44 2023-05-31T21:35:42.556 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/pklxkd-64/obj44 2023-05-31T21:35:42.556 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/pklxkd-64/obj44?versionId=null 2023-05-31T21:35:42.556 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/pklxkd-64/obj44?versionId=null 2023-05-31T21:35:42.557 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-05-31T21:35:42.557 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /pklxkd-64/obj44?versionId=null 2023-05-31T21:35:42.557 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-05-31T21:35:42.557 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-05-31T21:35:42.558 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T21:35:42.558 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-05-31T21:35:42.558 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-05-31T21:35:42.558 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-05-31T21:35:42.559 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-05-31T21:35:42.559 INFO:tasks.rgw_multisite_tests:HEAD 2023-05-31T21:35:42.559 INFO:tasks.rgw_multisite_tests: 2023-05-31T21:35:42.559 INFO:tasks.rgw_multisite_tests: 2023-05-31T21:35:42.559 INFO:tasks.rgw_multisite_tests:Wed, 31 May 2023 21:12:19 GMT 2023-05-31T21:35:42.560 INFO:tasks.rgw_multisite_tests:/pklxkd-64/obj44?versionId=null 2023-05-31T21:35:42.560 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-05-31T21:35:42.560 INFO:tasks.rgw_multisite_tests:AWS Q80H11OLNU1S48PPYL26:ibhi57mXKqH+2CYQztxin9SGqWg= 2023-05-31T21:35:42.560 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 21:12:19 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:ibhi57mXKqH+2CYQztxin9SGqWg=', 'Content-Length': '0'} 2023-05-31T21:35:42.561 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 21:11:49 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000059566caf0c9a59d1-006477b833-1fb6-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 21:12:19 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:35:42.561 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/pklxkd-64/obj44 2023-05-31T21:35:42.561 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/pklxkd-64/obj44 2023-05-31T21:35:42.561 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/pklxkd-64/obj44?versionId=null 2023-05-31T21:35:42.562 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/pklxkd-64/obj44?versionId=null 2023-05-31T21:35:42.562 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-05-31T21:35:42.562 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /pklxkd-64/obj44?versionId=null 2023-05-31T21:35:42.562 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-05-31T21:35:42.562 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-05-31T21:35:42.563 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T21:35:42.563 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-05-31T21:35:42.563 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-05-31T21:35:42.563 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-05-31T21:35:42.564 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-05-31T21:35:42.564 INFO:tasks.rgw_multisite_tests:HEAD 2023-05-31T21:35:42.564 INFO:tasks.rgw_multisite_tests: 2023-05-31T21:35:42.564 INFO:tasks.rgw_multisite_tests: 2023-05-31T21:35:42.564 INFO:tasks.rgw_multisite_tests:Wed, 31 May 2023 21:12:20 GMT 2023-05-31T21:35:42.565 INFO:tasks.rgw_multisite_tests:/pklxkd-64/obj44?versionId=null 2023-05-31T21:35:42.565 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-05-31T21:35:42.565 INFO:tasks.rgw_multisite_tests:AWS Q80H11OLNU1S48PPYL26:k3nwDZhgaR8EUg57JyJlccWnVeo= 2023-05-31T21:35:42.565 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 21:12:20 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:k3nwDZhgaR8EUg57JyJlccWnVeo=', 'Content-Length': '0'} 2023-05-31T21:35:42.566 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 21:11:49 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007874cd1e16bb6878-006477b833-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 21:12:20 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:35:42.566 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: comparing key name=obj44 2023-05-31T21:35:42.566 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/pklxkd-64/obj44 2023-05-31T21:35:42.566 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/pklxkd-64/obj44 2023-05-31T21:35:42.567 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/pklxkd-64/obj44?versionId=null 2023-05-31T21:35:42.567 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/pklxkd-64/obj44?versionId=null 2023-05-31T21:35:42.567 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-05-31T21:35:42.567 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /pklxkd-64/obj44?versionId=null 2023-05-31T21:35:42.567 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-05-31T21:35:42.568 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-05-31T21:35:42.568 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T21:35:42.568 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-05-31T21:35:42.568 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-05-31T21:35:42.569 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-05-31T21:35:42.569 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-05-31T21:35:42.569 INFO:tasks.rgw_multisite_tests:GET 2023-05-31T21:35:42.569 INFO:tasks.rgw_multisite_tests: 2023-05-31T21:35:42.570 INFO:tasks.rgw_multisite_tests: 2023-05-31T21:35:42.570 INFO:tasks.rgw_multisite_tests:Wed, 31 May 2023 21:12:20 GMT 2023-05-31T21:35:42.570 INFO:tasks.rgw_multisite_tests:/pklxkd-64/obj44?versionId=null 2023-05-31T21:35:42.570 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-05-31T21:35:42.570 INFO:tasks.rgw_multisite_tests:AWS Q80H11OLNU1S48PPYL26:u+nw0+IA9IGHYaKonVjMTqWl3Cw= 2023-05-31T21:35:42.571 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 21:12:20 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:u+nw0+IA9IGHYaKonVjMTqWl3Cw=', 'Content-Length': '0'} 2023-05-31T21:35:42.571 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 21:11:49 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000cecb2855bc60cdd1-006477b834-1fb6-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 21:12:20 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:35:42.571 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/pklxkd-64/obj44 2023-05-31T21:35:42.571 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/pklxkd-64/obj44 2023-05-31T21:35:42.572 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/pklxkd-64/obj44?versionId=null 2023-05-31T21:35:42.572 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/pklxkd-64/obj44?versionId=null 2023-05-31T21:35:42.572 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-05-31T21:35:42.572 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /pklxkd-64/obj44?versionId=null 2023-05-31T21:35:42.573 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-05-31T21:35:42.573 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-05-31T21:35:42.573 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T21:35:42.573 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-05-31T21:35:42.573 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-05-31T21:35:42.574 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-05-31T21:35:42.574 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-05-31T21:35:42.574 INFO:tasks.rgw_multisite_tests:GET 2023-05-31T21:35:42.574 INFO:tasks.rgw_multisite_tests: 2023-05-31T21:35:42.575 INFO:tasks.rgw_multisite_tests: 2023-05-31T21:35:42.575 INFO:tasks.rgw_multisite_tests:Wed, 31 May 2023 21:12:20 GMT 2023-05-31T21:35:42.575 INFO:tasks.rgw_multisite_tests:/pklxkd-64/obj44?versionId=null 2023-05-31T21:35:42.575 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-05-31T21:35:42.575 INFO:tasks.rgw_multisite_tests:AWS Q80H11OLNU1S48PPYL26:u+nw0+IA9IGHYaKonVjMTqWl3Cw= 2023-05-31T21:35:42.576 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 21:12:20 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:u+nw0+IA9IGHYaKonVjMTqWl3Cw=', 'Content-Length': '0'} 2023-05-31T21:35:42.576 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 21:11:49 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e88e1f67f32b0cbd-006477b834-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 21:12:20 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:35:42.576 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/pklxkd-64/ 2023-05-31T21:35:42.576 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/pklxkd-64/ 2023-05-31T21:35:42.577 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/pklxkd-64/?prefix=obj44 2023-05-31T21:35:42.577 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/pklxkd-64/?prefix=obj44 2023-05-31T21:35:42.577 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-05-31T21:35:42.577 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /pklxkd-64/?prefix=obj44 2023-05-31T21:35:42.578 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-05-31T21:35:42.578 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-05-31T21:35:42.578 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T21:35:42.578 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-05-31T21:35:42.579 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-05-31T21:35:42.579 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-05-31T21:35:42.579 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-05-31T21:35:42.579 INFO:tasks.rgw_multisite_tests:GET 2023-05-31T21:35:42.579 INFO:tasks.rgw_multisite_tests: 2023-05-31T21:35:42.580 INFO:tasks.rgw_multisite_tests: 2023-05-31T21:35:42.580 INFO:tasks.rgw_multisite_tests:Wed, 31 May 2023 21:12:20 GMT 2023-05-31T21:35:42.580 INFO:tasks.rgw_multisite_tests:/pklxkd-64/ 2023-05-31T21:35:42.580 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-05-31T21:35:42.581 INFO:tasks.rgw_multisite_tests:AWS Q80H11OLNU1S48PPYL26:eDdIGyOBSPGk4dT06OEG2smpfgM= 2023-05-31T21:35:42.581 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 21:12:20 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:eDdIGyOBSPGk4dT06OEG2smpfgM=', 'Content-Length': '0'} 2023-05-31T21:35:42.581 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e690366cf18d4124-006477b834-1fb6-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 21:12:20 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:35:42.581 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'pklxkd-64obj441000falseobj442023-05-31T21:11:49.126Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T21:35:42.582 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/pklxkd-64/ 2023-05-31T21:35:42.582 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/pklxkd-64/ 2023-05-31T21:35:42.582 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/pklxkd-64/?prefix=obj44 2023-05-31T21:35:42.582 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/pklxkd-64/?prefix=obj44 2023-05-31T21:35:42.582 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-05-31T21:35:42.583 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /pklxkd-64/?prefix=obj44 2023-05-31T21:35:42.583 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-05-31T21:35:42.583 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-05-31T21:35:42.583 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T21:35:42.584 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-05-31T21:35:42.584 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-05-31T21:35:42.584 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-05-31T21:35:42.584 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-05-31T21:35:42.585 INFO:tasks.rgw_multisite_tests:GET 2023-05-31T21:35:42.585 INFO:tasks.rgw_multisite_tests: 2023-05-31T21:35:42.585 INFO:tasks.rgw_multisite_tests: 2023-05-31T21:35:42.585 INFO:tasks.rgw_multisite_tests:Wed, 31 May 2023 21:12:20 GMT 2023-05-31T21:35:42.586 INFO:tasks.rgw_multisite_tests:/pklxkd-64/ 2023-05-31T21:35:42.586 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-05-31T21:35:42.586 INFO:tasks.rgw_multisite_tests:AWS Q80H11OLNU1S48PPYL26:eDdIGyOBSPGk4dT06OEG2smpfgM= 2023-05-31T21:35:42.586 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 21:12:20 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:eDdIGyOBSPGk4dT06OEG2smpfgM=', 'Content-Length': '0'} 2023-05-31T21:35:42.587 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f7e7658991dc86aa-006477b834-1630-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 21:12:20 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:35:42.587 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'pklxkd-64obj441000falseobj442023-05-31T21:11:49.126Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T21:35:42.587 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/pklxkd-64/obj44 2023-05-31T21:35:42.587 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/pklxkd-64/obj44 2023-05-31T21:35:42.588 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-05-31T21:35:42.588 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /pklxkd-64/obj44 2023-05-31T21:35:42.588 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-05-31T21:35:42.588 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-05-31T21:35:42.589 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T21:35:42.589 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-05-31T21:35:42.589 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-05-31T21:35:42.589 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-05-31T21:35:42.589 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-05-31T21:35:42.590 INFO:tasks.rgw_multisite_tests:HEAD 2023-05-31T21:35:42.590 INFO:tasks.rgw_multisite_tests: 2023-05-31T21:35:42.590 INFO:tasks.rgw_multisite_tests: 2023-05-31T21:35:42.590 INFO:tasks.rgw_multisite_tests:Wed, 31 May 2023 21:12:20 GMT 2023-05-31T21:35:42.591 INFO:tasks.rgw_multisite_tests:/pklxkd-64/obj44 2023-05-31T21:35:42.591 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-05-31T21:35:42.591 INFO:tasks.rgw_multisite_tests:AWS Q80H11OLNU1S48PPYL26:elrRkq62ZvBip1R1pOI1SUb3yyg= 2023-05-31T21:35:42.591 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 21:12:20 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:elrRkq62ZvBip1R1pOI1SUb3yyg=', 'Content-Length': '0'} 2023-05-31T21:35:42.591 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 21:11:49 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f6cd579eeb358040-006477b834-1fb6-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 21:12:20 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:35:42.592 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/pklxkd-64/obj44 2023-05-31T21:35:42.592 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/pklxkd-64/obj44 2023-05-31T21:35:42.592 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-05-31T21:35:42.592 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /pklxkd-64/obj44 2023-05-31T21:35:42.593 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-05-31T21:35:42.593 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-05-31T21:35:42.593 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T21:35:42.593 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-05-31T21:35:42.594 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-05-31T21:35:42.594 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-05-31T21:35:42.594 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-05-31T21:35:42.594 INFO:tasks.rgw_multisite_tests:HEAD 2023-05-31T21:35:42.594 INFO:tasks.rgw_multisite_tests: 2023-05-31T21:35:42.595 INFO:tasks.rgw_multisite_tests: 2023-05-31T21:35:42.595 INFO:tasks.rgw_multisite_tests:Wed, 31 May 2023 21:12:20 GMT 2023-05-31T21:35:42.595 INFO:tasks.rgw_multisite_tests:/pklxkd-64/obj44 2023-05-31T21:35:42.595 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-05-31T21:35:42.596 INFO:tasks.rgw_multisite_tests:AWS Q80H11OLNU1S48PPYL26:elrRkq62ZvBip1R1pOI1SUb3yyg= 2023-05-31T21:35:42.596 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 21:12:20 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:elrRkq62ZvBip1R1pOI1SUb3yyg=', 'Content-Length': '0'} 2023-05-31T21:35:42.596 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 21:11:49 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f16745f4a0aa1fed-006477b834-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 21:12:20 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:35:42.596 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: comparing key name=obj44 2023-05-31T21:35:42.597 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/pklxkd-64/obj44 2023-05-31T21:35:42.597 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/pklxkd-64/obj44 2023-05-31T21:35:42.597 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-05-31T21:35:42.597 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /pklxkd-64/obj44 2023-05-31T21:35:42.598 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-05-31T21:35:42.598 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-05-31T21:35:42.598 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T21:35:42.598 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-05-31T21:35:42.598 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-05-31T21:35:42.599 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-05-31T21:35:42.599 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-05-31T21:35:42.599 INFO:tasks.rgw_multisite_tests:GET 2023-05-31T21:35:42.599 INFO:tasks.rgw_multisite_tests: 2023-05-31T21:35:42.600 INFO:tasks.rgw_multisite_tests: 2023-05-31T21:35:42.600 INFO:tasks.rgw_multisite_tests:Wed, 31 May 2023 21:12:20 GMT 2023-05-31T21:35:42.600 INFO:tasks.rgw_multisite_tests:/pklxkd-64/obj44 2023-05-31T21:35:42.600 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-05-31T21:35:42.601 INFO:tasks.rgw_multisite_tests:AWS Q80H11OLNU1S48PPYL26:lLDDwRw4jQfWrAY9MrH2RMIZY3c= 2023-05-31T21:35:42.601 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 21:12:20 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:lLDDwRw4jQfWrAY9MrH2RMIZY3c=', 'Content-Length': '0'} 2023-05-31T21:35:42.601 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 21:11:49 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ec9f776c8038cfb6-006477b834-1fb6-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 21:12:20 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:35:42.601 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/pklxkd-64/obj44 2023-05-31T21:35:42.601 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/pklxkd-64/obj44 2023-05-31T21:35:42.602 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-05-31T21:35:42.602 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /pklxkd-64/obj44 2023-05-31T21:35:42.602 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-05-31T21:35:42.602 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-05-31T21:35:42.603 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T21:35:42.603 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-05-31T21:35:42.603 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-05-31T21:35:42.603 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-05-31T21:35:42.604 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-05-31T21:35:42.604 INFO:tasks.rgw_multisite_tests:GET 2023-05-31T21:35:42.604 INFO:tasks.rgw_multisite_tests: 2023-05-31T21:35:42.604 INFO:tasks.rgw_multisite_tests: 2023-05-31T21:35:42.604 INFO:tasks.rgw_multisite_tests:Wed, 31 May 2023 21:12:20 GMT 2023-05-31T21:35:42.605 INFO:tasks.rgw_multisite_tests:/pklxkd-64/obj44 2023-05-31T21:35:42.605 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-05-31T21:35:42.605 INFO:tasks.rgw_multisite_tests:AWS Q80H11OLNU1S48PPYL26:lLDDwRw4jQfWrAY9MrH2RMIZY3c= 2023-05-31T21:35:42.605 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 21:12:20 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:lLDDwRw4jQfWrAY9MrH2RMIZY3c=', 'Content-Length': '0'} 2023-05-31T21:35:42.606 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 21:11:49 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000cdc7915a9b14b739-006477b834-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 21:12:20 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:35:42.606 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/pklxkd-64/ 2023-05-31T21:35:42.606 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/pklxkd-64/ 2023-05-31T21:35:42.606 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/pklxkd-64/?prefix=obj44 2023-05-31T21:35:42.607 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/pklxkd-64/?prefix=obj44 2023-05-31T21:35:42.607 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-05-31T21:35:42.607 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /pklxkd-64/?prefix=obj44 2023-05-31T21:35:42.607 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-05-31T21:35:42.607 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-05-31T21:35:42.608 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T21:35:42.608 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-05-31T21:35:42.608 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-05-31T21:35:42.608 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-05-31T21:35:42.609 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-05-31T21:35:42.609 INFO:tasks.rgw_multisite_tests:GET 2023-05-31T21:35:42.609 INFO:tasks.rgw_multisite_tests: 2023-05-31T21:35:42.609 INFO:tasks.rgw_multisite_tests: 2023-05-31T21:35:42.610 INFO:tasks.rgw_multisite_tests:Wed, 31 May 2023 21:12:20 GMT 2023-05-31T21:35:42.610 INFO:tasks.rgw_multisite_tests:/pklxkd-64/ 2023-05-31T21:35:42.610 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-05-31T21:35:42.610 INFO:tasks.rgw_multisite_tests:AWS Q80H11OLNU1S48PPYL26:eDdIGyOBSPGk4dT06OEG2smpfgM= 2023-05-31T21:35:42.610 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 21:12:20 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:eDdIGyOBSPGk4dT06OEG2smpfgM=', 'Content-Length': '0'} 2023-05-31T21:35:42.611 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b08e34544de52037-006477b834-1fb6-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 21:12:20 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:35:42.611 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'pklxkd-64obj441000falseobj442023-05-31T21:11:49.126Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T21:35:42.611 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/pklxkd-64/ 2023-05-31T21:35:42.611 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/pklxkd-64/ 2023-05-31T21:35:42.612 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/pklxkd-64/?prefix=obj44 2023-05-31T21:35:42.612 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/pklxkd-64/?prefix=obj44 2023-05-31T21:35:42.612 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-05-31T21:35:42.612 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /pklxkd-64/?prefix=obj44 2023-05-31T21:35:42.613 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-05-31T21:35:42.613 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-05-31T21:35:42.613 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T21:35:42.613 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-05-31T21:35:42.613 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-05-31T21:35:42.614 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-05-31T21:35:42.614 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-05-31T21:35:42.614 INFO:tasks.rgw_multisite_tests:GET 2023-05-31T21:35:42.614 INFO:tasks.rgw_multisite_tests: 2023-05-31T21:35:42.615 INFO:tasks.rgw_multisite_tests: 2023-05-31T21:35:42.615 INFO:tasks.rgw_multisite_tests:Wed, 31 May 2023 21:12:20 GMT 2023-05-31T21:35:42.615 INFO:tasks.rgw_multisite_tests:/pklxkd-64/ 2023-05-31T21:35:42.615 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-05-31T21:35:42.616 INFO:tasks.rgw_multisite_tests:AWS Q80H11OLNU1S48PPYL26:eDdIGyOBSPGk4dT06OEG2smpfgM= 2023-05-31T21:35:42.616 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 21:12:20 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:eDdIGyOBSPGk4dT06OEG2smpfgM=', 'Content-Length': '0'} 2023-05-31T21:35:42.616 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000006fc5c4a082cab2c9-006477b834-1630-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 21:12:20 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:35:42.616 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'pklxkd-64obj441000falseobj442023-05-31T21:11:49.126Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T21:35:42.616 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: comparing key name=obj55 2023-05-31T21:35:42.617 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/pklxkd-64/obj55 2023-05-31T21:35:42.617 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/pklxkd-64/obj55 2023-05-31T21:35:42.617 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/pklxkd-64/obj55?versionId=null 2023-05-31T21:35:42.617 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/pklxkd-64/obj55?versionId=null 2023-05-31T21:35:42.618 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-05-31T21:35:42.618 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /pklxkd-64/obj55?versionId=null 2023-05-31T21:35:42.618 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-05-31T21:35:42.618 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-05-31T21:35:42.619 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T21:35:42.619 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-05-31T21:35:42.619 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-05-31T21:35:42.619 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-05-31T21:35:42.619 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-05-31T21:35:42.620 INFO:tasks.rgw_multisite_tests:GET 2023-05-31T21:35:42.620 INFO:tasks.rgw_multisite_tests: 2023-05-31T21:35:42.620 INFO:tasks.rgw_multisite_tests: 2023-05-31T21:35:42.620 INFO:tasks.rgw_multisite_tests:Wed, 31 May 2023 21:12:20 GMT 2023-05-31T21:35:42.621 INFO:tasks.rgw_multisite_tests:/pklxkd-64/obj55?versionId=null 2023-05-31T21:35:42.621 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-05-31T21:35:42.621 INFO:tasks.rgw_multisite_tests:AWS Q80H11OLNU1S48PPYL26:9yjL+gXeHFdgrQ4CunoYeEHYQFg= 2023-05-31T21:35:42.621 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 21:12:20 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:9yjL+gXeHFdgrQ4CunoYeEHYQFg=', 'Content-Length': '0'} 2023-05-31T21:35:42.622 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 21:11:49 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000923ad71e61192bda-006477b834-1fb6-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 21:12:20 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:35:42.622 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/pklxkd-64/obj55 2023-05-31T21:35:42.622 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/pklxkd-64/obj55 2023-05-31T21:35:42.622 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/pklxkd-64/obj55?versionId=null 2023-05-31T21:35:42.623 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/pklxkd-64/obj55?versionId=null 2023-05-31T21:35:42.623 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-05-31T21:35:42.623 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /pklxkd-64/obj55?versionId=null 2023-05-31T21:35:42.623 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-05-31T21:35:42.624 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-05-31T21:35:42.624 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T21:35:42.624 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-05-31T21:35:42.624 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-05-31T21:35:42.625 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-05-31T21:35:42.625 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-05-31T21:35:42.625 INFO:tasks.rgw_multisite_tests:GET 2023-05-31T21:35:42.625 INFO:tasks.rgw_multisite_tests: 2023-05-31T21:35:42.625 INFO:tasks.rgw_multisite_tests: 2023-05-31T21:35:42.626 INFO:tasks.rgw_multisite_tests:Wed, 31 May 2023 21:12:20 GMT 2023-05-31T21:35:42.626 INFO:tasks.rgw_multisite_tests:/pklxkd-64/obj55?versionId=null 2023-05-31T21:35:42.626 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-05-31T21:35:42.626 INFO:tasks.rgw_multisite_tests:AWS Q80H11OLNU1S48PPYL26:9yjL+gXeHFdgrQ4CunoYeEHYQFg= 2023-05-31T21:35:42.627 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 21:12:20 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:9yjL+gXeHFdgrQ4CunoYeEHYQFg=', 'Content-Length': '0'} 2023-05-31T21:35:42.627 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 21:11:49 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000aaf54489ecfb2a70-006477b834-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 21:12:20 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:35:42.627 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/pklxkd-64/obj55 2023-05-31T21:35:42.627 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/pklxkd-64/obj55 2023-05-31T21:35:42.627 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/pklxkd-64/obj55?versionId=null 2023-05-31T21:35:42.628 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/pklxkd-64/obj55?versionId=null 2023-05-31T21:35:42.628 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-05-31T21:35:42.628 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /pklxkd-64/obj55?versionId=null 2023-05-31T21:35:42.628 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-05-31T21:35:42.629 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-05-31T21:35:42.629 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T21:35:42.629 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-05-31T21:35:42.629 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-05-31T21:35:42.630 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-05-31T21:35:42.630 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-05-31T21:35:42.630 INFO:tasks.rgw_multisite_tests:HEAD 2023-05-31T21:35:42.630 INFO:tasks.rgw_multisite_tests: 2023-05-31T21:35:42.630 INFO:tasks.rgw_multisite_tests: 2023-05-31T21:35:42.631 INFO:tasks.rgw_multisite_tests:Wed, 31 May 2023 21:12:20 GMT 2023-05-31T21:35:42.631 INFO:tasks.rgw_multisite_tests:/pklxkd-64/obj55?versionId=null 2023-05-31T21:35:42.631 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-05-31T21:35:42.631 INFO:tasks.rgw_multisite_tests:AWS Q80H11OLNU1S48PPYL26:l+HBRB7ah6vPUB+HRib6vkbdDZU= 2023-05-31T21:35:42.632 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 21:12:20 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:l+HBRB7ah6vPUB+HRib6vkbdDZU=', 'Content-Length': '0'} 2023-05-31T21:35:42.632 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 21:11:49 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002addf74279b12ffd-006477b834-1fb6-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 21:12:20 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:35:42.632 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/pklxkd-64/obj55 2023-05-31T21:35:42.632 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/pklxkd-64/obj55 2023-05-31T21:35:42.633 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/pklxkd-64/obj55?versionId=null 2023-05-31T21:35:42.633 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/pklxkd-64/obj55?versionId=null 2023-05-31T21:35:42.633 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-05-31T21:35:42.633 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /pklxkd-64/obj55?versionId=null 2023-05-31T21:35:42.634 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-05-31T21:35:42.634 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-05-31T21:35:42.634 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T21:35:42.634 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-05-31T21:35:42.634 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-05-31T21:35:42.635 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-05-31T21:35:42.635 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-05-31T21:35:42.635 INFO:tasks.rgw_multisite_tests:HEAD 2023-05-31T21:35:42.635 INFO:tasks.rgw_multisite_tests: 2023-05-31T21:35:42.636 INFO:tasks.rgw_multisite_tests: 2023-05-31T21:35:42.636 INFO:tasks.rgw_multisite_tests:Wed, 31 May 2023 21:12:20 GMT 2023-05-31T21:35:42.636 INFO:tasks.rgw_multisite_tests:/pklxkd-64/obj55?versionId=null 2023-05-31T21:35:42.636 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-05-31T21:35:42.637 INFO:tasks.rgw_multisite_tests:AWS Q80H11OLNU1S48PPYL26:l+HBRB7ah6vPUB+HRib6vkbdDZU= 2023-05-31T21:35:42.637 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 21:12:20 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:l+HBRB7ah6vPUB+HRib6vkbdDZU=', 'Content-Length': '0'} 2023-05-31T21:35:42.637 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 21:11:49 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000080068e285ca5de6c-006477b834-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 21:12:20 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:35:42.637 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: comparing key name=obj55 2023-05-31T21:35:42.638 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/pklxkd-64/obj55 2023-05-31T21:35:42.638 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/pklxkd-64/obj55 2023-05-31T21:35:42.638 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/pklxkd-64/obj55?versionId=null 2023-05-31T21:35:42.638 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/pklxkd-64/obj55?versionId=null 2023-05-31T21:35:42.638 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-05-31T21:35:42.639 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /pklxkd-64/obj55?versionId=null 2023-05-31T21:35:42.639 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-05-31T21:35:42.639 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-05-31T21:35:42.639 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T21:35:42.640 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-05-31T21:35:42.640 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-05-31T21:35:42.640 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-05-31T21:35:42.640 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-05-31T21:35:42.641 INFO:tasks.rgw_multisite_tests:GET 2023-05-31T21:35:42.641 INFO:tasks.rgw_multisite_tests: 2023-05-31T21:35:42.641 INFO:tasks.rgw_multisite_tests: 2023-05-31T21:35:42.641 INFO:tasks.rgw_multisite_tests:Wed, 31 May 2023 21:12:20 GMT 2023-05-31T21:35:42.641 INFO:tasks.rgw_multisite_tests:/pklxkd-64/obj55?versionId=null 2023-05-31T21:35:42.642 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-05-31T21:35:42.642 INFO:tasks.rgw_multisite_tests:AWS Q80H11OLNU1S48PPYL26:9yjL+gXeHFdgrQ4CunoYeEHYQFg= 2023-05-31T21:35:42.642 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 21:12:20 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:9yjL+gXeHFdgrQ4CunoYeEHYQFg=', 'Content-Length': '0'} 2023-05-31T21:35:42.642 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 21:11:49 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b6c0fa901402e645-006477b834-1fb6-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 21:12:20 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:35:42.643 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/pklxkd-64/obj55 2023-05-31T21:35:42.643 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/pklxkd-64/obj55 2023-05-31T21:35:42.643 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/pklxkd-64/obj55?versionId=null 2023-05-31T21:35:42.643 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/pklxkd-64/obj55?versionId=null 2023-05-31T21:35:42.644 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-05-31T21:35:42.644 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /pklxkd-64/obj55?versionId=null 2023-05-31T21:35:42.644 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-05-31T21:35:42.644 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-05-31T21:35:42.644 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T21:35:42.645 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-05-31T21:35:42.645 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-05-31T21:35:42.645 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-05-31T21:35:42.645 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-05-31T21:35:42.646 INFO:tasks.rgw_multisite_tests:GET 2023-05-31T21:35:42.646 INFO:tasks.rgw_multisite_tests: 2023-05-31T21:35:42.646 INFO:tasks.rgw_multisite_tests: 2023-05-31T21:35:42.646 INFO:tasks.rgw_multisite_tests:Wed, 31 May 2023 21:12:20 GMT 2023-05-31T21:35:42.646 INFO:tasks.rgw_multisite_tests:/pklxkd-64/obj55?versionId=null 2023-05-31T21:35:42.647 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-05-31T21:35:42.647 INFO:tasks.rgw_multisite_tests:AWS Q80H11OLNU1S48PPYL26:9yjL+gXeHFdgrQ4CunoYeEHYQFg= 2023-05-31T21:35:42.647 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 21:12:20 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:9yjL+gXeHFdgrQ4CunoYeEHYQFg=', 'Content-Length': '0'} 2023-05-31T21:35:42.647 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 21:11:49 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002bb73188614ddb18-006477b834-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 21:12:20 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:35:42.648 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/pklxkd-64/ 2023-05-31T21:35:42.648 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/pklxkd-64/ 2023-05-31T21:35:42.648 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/pklxkd-64/?prefix=obj55 2023-05-31T21:35:42.648 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/pklxkd-64/?prefix=obj55 2023-05-31T21:35:42.649 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-05-31T21:35:42.649 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /pklxkd-64/?prefix=obj55 2023-05-31T21:35:42.649 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-05-31T21:35:42.649 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-05-31T21:35:42.650 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T21:35:42.650 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-05-31T21:35:42.650 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-05-31T21:35:42.650 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-05-31T21:35:42.650 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-05-31T21:35:42.651 INFO:tasks.rgw_multisite_tests:GET 2023-05-31T21:35:42.651 INFO:tasks.rgw_multisite_tests: 2023-05-31T21:35:42.651 INFO:tasks.rgw_multisite_tests: 2023-05-31T21:35:42.651 INFO:tasks.rgw_multisite_tests:Wed, 31 May 2023 21:12:20 GMT 2023-05-31T21:35:42.652 INFO:tasks.rgw_multisite_tests:/pklxkd-64/ 2023-05-31T21:35:42.652 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-05-31T21:35:42.652 INFO:tasks.rgw_multisite_tests:AWS Q80H11OLNU1S48PPYL26:eDdIGyOBSPGk4dT06OEG2smpfgM= 2023-05-31T21:35:42.652 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 21:12:20 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:eDdIGyOBSPGk4dT06OEG2smpfgM=', 'Content-Length': '0'} 2023-05-31T21:35:42.653 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000000bbe93386a6cc587-006477b834-1fb6-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 21:12:20 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:35:42.653 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'pklxkd-64obj551000falseobj552023-05-31T21:11:49.145Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T21:35:42.653 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/pklxkd-64/ 2023-05-31T21:35:42.653 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/pklxkd-64/ 2023-05-31T21:35:42.653 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/pklxkd-64/?prefix=obj55 2023-05-31T21:35:42.654 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/pklxkd-64/?prefix=obj55 2023-05-31T21:35:42.654 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-05-31T21:35:42.654 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /pklxkd-64/?prefix=obj55 2023-05-31T21:35:42.654 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-05-31T21:35:42.655 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-05-31T21:35:42.655 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T21:35:42.655 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-05-31T21:35:42.655 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-05-31T21:35:42.655 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-05-31T21:35:42.656 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-05-31T21:35:42.656 INFO:tasks.rgw_multisite_tests:GET 2023-05-31T21:35:42.656 INFO:tasks.rgw_multisite_tests: 2023-05-31T21:35:42.656 INFO:tasks.rgw_multisite_tests: 2023-05-31T21:35:42.657 INFO:tasks.rgw_multisite_tests:Wed, 31 May 2023 21:12:20 GMT 2023-05-31T21:35:42.657 INFO:tasks.rgw_multisite_tests:/pklxkd-64/ 2023-05-31T21:35:42.657 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-05-31T21:35:42.657 INFO:tasks.rgw_multisite_tests:AWS Q80H11OLNU1S48PPYL26:eDdIGyOBSPGk4dT06OEG2smpfgM= 2023-05-31T21:35:42.658 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 21:12:20 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:eDdIGyOBSPGk4dT06OEG2smpfgM=', 'Content-Length': '0'} 2023-05-31T21:35:42.658 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000feca934e4ff397ad-006477b834-1630-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 21:12:20 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:35:42.658 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'pklxkd-64obj551000falseobj552023-05-31T21:11:49.145Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T21:35:42.658 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/pklxkd-64/obj55 2023-05-31T21:35:42.659 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/pklxkd-64/obj55 2023-05-31T21:35:42.659 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-05-31T21:35:42.659 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /pklxkd-64/obj55 2023-05-31T21:35:42.659 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-05-31T21:35:42.659 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-05-31T21:35:42.660 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T21:35:42.660 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-05-31T21:35:42.660 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-05-31T21:35:42.660 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-05-31T21:35:42.661 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-05-31T21:35:42.661 INFO:tasks.rgw_multisite_tests:HEAD 2023-05-31T21:35:42.661 INFO:tasks.rgw_multisite_tests: 2023-05-31T21:35:42.661 INFO:tasks.rgw_multisite_tests: 2023-05-31T21:35:42.661 INFO:tasks.rgw_multisite_tests:Wed, 31 May 2023 21:12:20 GMT 2023-05-31T21:35:42.662 INFO:tasks.rgw_multisite_tests:/pklxkd-64/obj55 2023-05-31T21:35:42.662 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-05-31T21:35:42.662 INFO:tasks.rgw_multisite_tests:AWS Q80H11OLNU1S48PPYL26:fTtLSyxSzeH/WJVZLufS0POd02k= 2023-05-31T21:35:42.662 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 21:12:20 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:fTtLSyxSzeH/WJVZLufS0POd02k=', 'Content-Length': '0'} 2023-05-31T21:35:42.663 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 21:11:49 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000360849725bfb8d51-006477b834-1fb6-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 21:12:20 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:35:42.663 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/pklxkd-64/obj55 2023-05-31T21:35:42.663 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/pklxkd-64/obj55 2023-05-31T21:35:42.663 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-05-31T21:35:42.664 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /pklxkd-64/obj55 2023-05-31T21:35:42.664 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-05-31T21:35:42.664 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-05-31T21:35:42.664 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T21:35:42.664 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-05-31T21:35:42.665 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-05-31T21:35:42.665 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-05-31T21:35:42.665 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-05-31T21:35:42.665 INFO:tasks.rgw_multisite_tests:HEAD 2023-05-31T21:35:42.666 INFO:tasks.rgw_multisite_tests: 2023-05-31T21:35:42.666 INFO:tasks.rgw_multisite_tests: 2023-05-31T21:35:42.666 INFO:tasks.rgw_multisite_tests:Wed, 31 May 2023 21:12:20 GMT 2023-05-31T21:35:42.666 INFO:tasks.rgw_multisite_tests:/pklxkd-64/obj55 2023-05-31T21:35:42.667 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-05-31T21:35:42.667 INFO:tasks.rgw_multisite_tests:AWS Q80H11OLNU1S48PPYL26:fTtLSyxSzeH/WJVZLufS0POd02k= 2023-05-31T21:35:42.667 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 21:12:20 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:fTtLSyxSzeH/WJVZLufS0POd02k=', 'Content-Length': '0'} 2023-05-31T21:35:42.667 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 21:11:49 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f81aaf3a5f656709-006477b834-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 21:12:20 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:35:42.667 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: comparing key name=obj55 2023-05-31T21:35:42.668 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/pklxkd-64/obj55 2023-05-31T21:35:42.668 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/pklxkd-64/obj55 2023-05-31T21:35:42.668 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-05-31T21:35:42.668 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /pklxkd-64/obj55 2023-05-31T21:35:42.669 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-05-31T21:35:42.669 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-05-31T21:35:42.669 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T21:35:42.669 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-05-31T21:35:42.670 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-05-31T21:35:42.670 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-05-31T21:35:42.670 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-05-31T21:35:42.670 INFO:tasks.rgw_multisite_tests:GET 2023-05-31T21:35:42.670 INFO:tasks.rgw_multisite_tests: 2023-05-31T21:35:42.671 INFO:tasks.rgw_multisite_tests: 2023-05-31T21:35:42.671 INFO:tasks.rgw_multisite_tests:Wed, 31 May 2023 21:12:20 GMT 2023-05-31T21:35:42.671 INFO:tasks.rgw_multisite_tests:/pklxkd-64/obj55 2023-05-31T21:35:42.671 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-05-31T21:35:42.672 INFO:tasks.rgw_multisite_tests:AWS Q80H11OLNU1S48PPYL26:Zdlk1/S3LaNKscA8PK/PDYPMsZk= 2023-05-31T21:35:42.672 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 21:12:20 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:Zdlk1/S3LaNKscA8PK/PDYPMsZk=', 'Content-Length': '0'} 2023-05-31T21:35:42.672 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 21:11:49 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004769432281c95f51-006477b834-1fb6-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 21:12:20 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:35:42.672 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/pklxkd-64/obj55 2023-05-31T21:35:42.673 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/pklxkd-64/obj55 2023-05-31T21:35:42.673 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-05-31T21:35:42.673 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /pklxkd-64/obj55 2023-05-31T21:35:42.673 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-05-31T21:35:42.673 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-05-31T21:35:42.674 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T21:35:42.674 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-05-31T21:35:42.674 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-05-31T21:35:42.674 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-05-31T21:35:42.675 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-05-31T21:35:42.675 INFO:tasks.rgw_multisite_tests:GET 2023-05-31T21:35:42.675 INFO:tasks.rgw_multisite_tests: 2023-05-31T21:35:42.675 INFO:tasks.rgw_multisite_tests: 2023-05-31T21:35:42.676 INFO:tasks.rgw_multisite_tests:Wed, 31 May 2023 21:12:20 GMT 2023-05-31T21:35:42.676 INFO:tasks.rgw_multisite_tests:/pklxkd-64/obj55 2023-05-31T21:35:42.676 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-05-31T21:35:42.676 INFO:tasks.rgw_multisite_tests:AWS Q80H11OLNU1S48PPYL26:Zdlk1/S3LaNKscA8PK/PDYPMsZk= 2023-05-31T21:35:42.676 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 21:12:20 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:Zdlk1/S3LaNKscA8PK/PDYPMsZk=', 'Content-Length': '0'} 2023-05-31T21:35:42.677 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 21:11:49 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c58b03dabd2b47c6-006477b834-1630-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 21:12:20 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:35:42.677 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/pklxkd-64/ 2023-05-31T21:35:42.677 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/pklxkd-64/ 2023-05-31T21:35:42.677 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/pklxkd-64/?prefix=obj55 2023-05-31T21:35:42.678 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/pklxkd-64/?prefix=obj55 2023-05-31T21:35:42.678 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-05-31T21:35:42.678 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /pklxkd-64/?prefix=obj55 2023-05-31T21:35:42.678 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-05-31T21:35:42.678 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-05-31T21:35:42.679 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T21:35:42.679 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-05-31T21:35:42.679 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-05-31T21:35:42.679 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-05-31T21:35:42.680 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-05-31T21:35:42.680 INFO:tasks.rgw_multisite_tests:GET 2023-05-31T21:35:42.680 INFO:tasks.rgw_multisite_tests: 2023-05-31T21:35:42.680 INFO:tasks.rgw_multisite_tests: 2023-05-31T21:35:42.681 INFO:tasks.rgw_multisite_tests:Wed, 31 May 2023 21:12:20 GMT 2023-05-31T21:35:42.681 INFO:tasks.rgw_multisite_tests:/pklxkd-64/ 2023-05-31T21:35:42.681 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-05-31T21:35:42.681 INFO:tasks.rgw_multisite_tests:AWS Q80H11OLNU1S48PPYL26:eDdIGyOBSPGk4dT06OEG2smpfgM= 2023-05-31T21:35:42.681 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 21:12:20 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:eDdIGyOBSPGk4dT06OEG2smpfgM=', 'Content-Length': '0'} 2023-05-31T21:35:42.682 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000971df19e014e1838-006477b834-1fb6-a2'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 21:12:20 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:35:42.682 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'pklxkd-64obj551000falseobj552023-05-31T21:11:49.145Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T21:35:42.682 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/pklxkd-64/ 2023-05-31T21:35:42.682 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/pklxkd-64/ 2023-05-31T21:35:42.683 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/pklxkd-64/?prefix=obj55 2023-05-31T21:35:42.683 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/pklxkd-64/?prefix=obj55 2023-05-31T21:35:42.683 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-05-31T21:35:42.683 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /pklxkd-64/?prefix=obj55 2023-05-31T21:35:42.684 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-05-31T21:35:42.684 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-05-31T21:35:42.684 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T21:35:42.684 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-05-31T21:35:42.685 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-05-31T21:35:42.685 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-05-31T21:35:42.685 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-05-31T21:35:42.685 INFO:tasks.rgw_multisite_tests:GET 2023-05-31T21:35:42.686 INFO:tasks.rgw_multisite_tests: 2023-05-31T21:35:42.686 INFO:tasks.rgw_multisite_tests: 2023-05-31T21:35:42.686 INFO:tasks.rgw_multisite_tests:Wed, 31 May 2023 21:12:20 GMT 2023-05-31T21:35:42.686 INFO:tasks.rgw_multisite_tests:/pklxkd-64/ 2023-05-31T21:35:42.687 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-05-31T21:35:42.687 INFO:tasks.rgw_multisite_tests:AWS Q80H11OLNU1S48PPYL26:eDdIGyOBSPGk4dT06OEG2smpfgM= 2023-05-31T21:35:42.687 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 21:12:20 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:eDdIGyOBSPGk4dT06OEG2smpfgM=', 'Content-Length': '0'} 2023-05-31T21:35:42.687 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000085ae51b5583591fb-006477b834-1630-a1'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 21:12:20 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:35:42.687 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'pklxkd-64obj551000falseobj552023-05-31T21:11:49.145Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-05-31T21:35:42.688 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: INFO: success, bucket identical: bucket=pklxkd-64 zones={a2, a1} 2023-05-31T21:35:42.688 INFO:tasks.rgw_multisite_tests:tasks.rgw.c1.client.1: DEBUG: waiting for process to exit 2023-05-31T21:35:42.688 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run: INFO: waiting for 300 2023-05-31T21:35:42.688 INFO:tasks.rgw_multisite_tests:tasks.rgw.c1.client.1.smithi099.stdout: INFO: 2023-05-31T21:12:20.364+0000 7f33ee678700 -1 received signal: Terminated from /usr/bin/python3 /bin/daemon-helper term radosgw --rgw-frontends beast port=8001 -n client.1 --cluster c1 -k /etc/ceph/c1.client.1.keyring --log-file /var/log/ceph/rgw.c1.client.1.log --rgw_ops_log_socket_path /home/ubuntu/cephtest/rgw.opslog.c1.client.1.sock --foreground --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm (PID: 169194) UID: 0 2023-05-31T21:35:42.689 INFO:tasks.rgw_multisite_tests:tasks.rgw.c1.client.1.smithi099.stdout: INFO: 2023-05-31T21:12:20.364+0000 7f33f28e17c0 -1 shutting down 2023-05-31T21:35:42.689 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c1.rgw.client.1 is failed for ~0s 2023-05-31T21:35:42.689 INFO:tasks.rgw_multisite_tests:tasks.rgw.c1.client.1: INFO: Stopped 2023-05-31T21:35:42.689 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['bucket', 'reshard', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'pklxkd-64', '--num-shards', '17', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-05-31T21:35:42.690 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'reshard', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'pklxkd-64', '--num-shards', '17', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-05-31T21:35:42.690 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket reshard --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --bucket pklxkd-64 --num-shards 17 --yes-i-really-mean-it --cluster c1 --debug-rgw 0 --debug-ms 0 2023-05-31T21:35:42.690 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: ignoring --setuser ceph since I am not root 2023-05-31T21:35:42.690 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-05-31T21:35:42.691 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: tenant: 2023-05-31T21:35:42.691 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: bucket name: pklxkd-64 2023-05-31T21:35:42.691 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: total entries: 5 2023-05-31T21:35:42.691 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/pklxkd-64/ 2023-05-31T21:35:42.691 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/pklxkd-64/ 2023-05-31T21:35:42.692 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-05-31T21:35:42.692 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /pklxkd-64/ 2023-05-31T21:35:42.692 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-05-31T21:35:42.692 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-05-31T21:35:42.693 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T21:35:42.693 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-05-31T21:35:42.693 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-05-31T21:35:42.693 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-05-31T21:35:42.694 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-05-31T21:35:42.694 INFO:tasks.rgw_multisite_tests:HEAD 2023-05-31T21:35:42.694 INFO:tasks.rgw_multisite_tests: 2023-05-31T21:35:42.694 INFO:tasks.rgw_multisite_tests: 2023-05-31T21:35:42.694 INFO:tasks.rgw_multisite_tests:Wed, 31 May 2023 21:12:26 GMT 2023-05-31T21:35:42.695 INFO:tasks.rgw_multisite_tests:/pklxkd-64/ 2023-05-31T21:35:42.695 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-05-31T21:35:42.695 INFO:tasks.rgw_multisite_tests:AWS Q80H11OLNU1S48PPYL26:hP1uXxRwqPN9evN7pLR4dhogLN8= 2023-05-31T21:35:42.695 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 21:12:26 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:hP1uXxRwqPN9evN7pLR4dhogLN8=', 'Content-Length': '0'} 2023-05-31T21:35:42.696 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '5'), ('X-RGW-Bytes-Used', '15'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000fed9c66a1b7fb1d5-006477b83a-1630-a1'), ('Content-Length', '0'), ('Date', 'Wed, 31 May 2023 21:12:26 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:35:42.696 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/pklxkd-64/obj17 2023-05-31T21:35:42.696 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/pklxkd-64/obj17 2023-05-31T21:35:42.696 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2023-05-31T21:35:42.697 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /pklxkd-64/obj17 2023-05-31T21:35:42.697 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-05-31T21:35:42.697 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-05-31T21:35:42.697 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T21:35:42.698 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-05-31T21:35:42.698 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-05-31T21:35:42.698 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-05-31T21:35:42.698 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-05-31T21:35:42.699 INFO:tasks.rgw_multisite_tests:PUT 2023-05-31T21:35:42.699 INFO:tasks.rgw_multisite_tests:rL0Y20zC+Fzt72VPzMSk2A== 2023-05-31T21:35:42.699 INFO:tasks.rgw_multisite_tests:application/octet-stream 2023-05-31T21:35:42.699 INFO:tasks.rgw_multisite_tests:Wed, 31 May 2023 21:12:26 GMT 2023-05-31T21:35:42.699 INFO:tasks.rgw_multisite_tests:/pklxkd-64/obj17 2023-05-31T21:35:42.700 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-05-31T21:35:42.700 INFO:tasks.rgw_multisite_tests:AWS Q80H11OLNU1S48PPYL26:q9jUMGlyDGU2cYgC/i0bSOx9c0I= 2023-05-31T21:35:42.700 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Wed, 31 May 2023 21:12:26 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:q9jUMGlyDGU2cYgC/i0bSOx9c0I='} 2023-05-31T21:35:42.700 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000c52b5be32e911359-006477b83a-1630-a1'), ('Date', 'Wed, 31 May 2023 21:12:26 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:35:42.701 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/pklxkd-64/ 2023-05-31T21:35:42.701 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/pklxkd-64/ 2023-05-31T21:35:42.701 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-05-31T21:35:42.701 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /pklxkd-64/ 2023-05-31T21:35:42.702 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-05-31T21:35:42.702 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-05-31T21:35:42.702 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T21:35:42.702 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-05-31T21:35:42.702 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-05-31T21:35:42.703 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-05-31T21:35:42.703 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-05-31T21:35:42.703 INFO:tasks.rgw_multisite_tests:HEAD 2023-05-31T21:35:42.703 INFO:tasks.rgw_multisite_tests: 2023-05-31T21:35:42.704 INFO:tasks.rgw_multisite_tests: 2023-05-31T21:35:42.704 INFO:tasks.rgw_multisite_tests:Wed, 31 May 2023 21:12:26 GMT 2023-05-31T21:35:42.704 INFO:tasks.rgw_multisite_tests:/pklxkd-64/ 2023-05-31T21:35:42.704 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-05-31T21:35:42.705 INFO:tasks.rgw_multisite_tests:AWS Q80H11OLNU1S48PPYL26:hP1uXxRwqPN9evN7pLR4dhogLN8= 2023-05-31T21:35:42.705 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 21:12:26 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:hP1uXxRwqPN9evN7pLR4dhogLN8=', 'Content-Length': '0'} 2023-05-31T21:35:42.705 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '6'), ('X-RGW-Bytes-Used', '18'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000091128a856c40bb70-006477b83a-1630-a1'), ('Content-Length', '0'), ('Date', 'Wed, 31 May 2023 21:12:26 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:35:42.705 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/pklxkd-64/obj34 2023-05-31T21:35:42.705 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/pklxkd-64/obj34 2023-05-31T21:35:42.706 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2023-05-31T21:35:42.706 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /pklxkd-64/obj34 2023-05-31T21:35:42.706 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-05-31T21:35:42.706 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-05-31T21:35:42.707 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T21:35:42.707 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-05-31T21:35:42.707 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-05-31T21:35:42.707 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-05-31T21:35:42.708 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-05-31T21:35:42.708 INFO:tasks.rgw_multisite_tests:PUT 2023-05-31T21:35:42.708 INFO:tasks.rgw_multisite_tests:rL0Y20zC+Fzt72VPzMSk2A== 2023-05-31T21:35:42.708 INFO:tasks.rgw_multisite_tests:application/octet-stream 2023-05-31T21:35:42.708 INFO:tasks.rgw_multisite_tests:Wed, 31 May 2023 21:12:26 GMT 2023-05-31T21:35:42.709 INFO:tasks.rgw_multisite_tests:/pklxkd-64/obj34 2023-05-31T21:35:42.709 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-05-31T21:35:42.709 INFO:tasks.rgw_multisite_tests:AWS Q80H11OLNU1S48PPYL26:YL1eSekfJ5uppmX6M09RK1igy7M= 2023-05-31T21:35:42.709 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Wed, 31 May 2023 21:12:26 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:YL1eSekfJ5uppmX6M09RK1igy7M='} 2023-05-31T21:35:42.710 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000476c64a49cd694e8-006477b83a-1630-a1'), ('Date', 'Wed, 31 May 2023 21:12:26 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:35:42.710 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/pklxkd-64/ 2023-05-31T21:35:42.710 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/pklxkd-64/ 2023-05-31T21:35:42.710 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-05-31T21:35:42.711 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /pklxkd-64/ 2023-05-31T21:35:42.711 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-05-31T21:35:42.711 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-05-31T21:35:42.711 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T21:35:42.711 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-05-31T21:35:42.712 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-05-31T21:35:42.712 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-05-31T21:35:42.712 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-05-31T21:35:42.712 INFO:tasks.rgw_multisite_tests:HEAD 2023-05-31T21:35:42.713 INFO:tasks.rgw_multisite_tests: 2023-05-31T21:35:42.713 INFO:tasks.rgw_multisite_tests: 2023-05-31T21:35:42.713 INFO:tasks.rgw_multisite_tests:Wed, 31 May 2023 21:12:26 GMT 2023-05-31T21:35:42.713 INFO:tasks.rgw_multisite_tests:/pklxkd-64/ 2023-05-31T21:35:42.714 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-05-31T21:35:42.714 INFO:tasks.rgw_multisite_tests:AWS Q80H11OLNU1S48PPYL26:hP1uXxRwqPN9evN7pLR4dhogLN8= 2023-05-31T21:35:42.714 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 21:12:26 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:hP1uXxRwqPN9evN7pLR4dhogLN8=', 'Content-Length': '0'} 2023-05-31T21:35:42.714 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '7'), ('X-RGW-Bytes-Used', '21'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000004b21c357feb9d319-006477b83a-1630-a1'), ('Content-Length', '0'), ('Date', 'Wed, 31 May 2023 21:12:26 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:35:42.714 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/pklxkd-64/obj51 2023-05-31T21:35:42.715 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/pklxkd-64/obj51 2023-05-31T21:35:42.715 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2023-05-31T21:35:42.715 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /pklxkd-64/obj51 2023-05-31T21:35:42.715 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-05-31T21:35:42.716 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-05-31T21:35:42.716 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T21:35:42.716 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-05-31T21:35:42.716 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-05-31T21:35:42.716 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-05-31T21:35:42.717 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-05-31T21:35:42.717 INFO:tasks.rgw_multisite_tests:PUT 2023-05-31T21:35:42.717 INFO:tasks.rgw_multisite_tests:rL0Y20zC+Fzt72VPzMSk2A== 2023-05-31T21:35:42.717 INFO:tasks.rgw_multisite_tests:application/octet-stream 2023-05-31T21:35:42.718 INFO:tasks.rgw_multisite_tests:Wed, 31 May 2023 21:12:26 GMT 2023-05-31T21:35:42.718 INFO:tasks.rgw_multisite_tests:/pklxkd-64/obj51 2023-05-31T21:35:42.718 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-05-31T21:35:42.718 INFO:tasks.rgw_multisite_tests:AWS Q80H11OLNU1S48PPYL26:rEjT25Nwzn3abcaOepFslarWnWY= 2023-05-31T21:35:42.719 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Wed, 31 May 2023 21:12:26 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:rEjT25Nwzn3abcaOepFslarWnWY='} 2023-05-31T21:35:42.719 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000858170e1d4305a2a-006477b83a-1630-a1'), ('Date', 'Wed, 31 May 2023 21:12:26 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:35:42.719 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/pklxkd-64/ 2023-05-31T21:35:42.719 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/pklxkd-64/ 2023-05-31T21:35:42.719 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-05-31T21:35:42.720 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /pklxkd-64/ 2023-05-31T21:35:42.720 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-05-31T21:35:42.720 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-05-31T21:35:42.720 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T21:35:42.721 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-05-31T21:35:42.721 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-05-31T21:35:42.721 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-05-31T21:35:42.721 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-05-31T21:35:42.722 INFO:tasks.rgw_multisite_tests:HEAD 2023-05-31T21:35:42.722 INFO:tasks.rgw_multisite_tests: 2023-05-31T21:35:42.722 INFO:tasks.rgw_multisite_tests: 2023-05-31T21:35:42.722 INFO:tasks.rgw_multisite_tests:Wed, 31 May 2023 21:12:26 GMT 2023-05-31T21:35:42.722 INFO:tasks.rgw_multisite_tests:/pklxkd-64/ 2023-05-31T21:35:42.723 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-05-31T21:35:42.723 INFO:tasks.rgw_multisite_tests:AWS Q80H11OLNU1S48PPYL26:hP1uXxRwqPN9evN7pLR4dhogLN8= 2023-05-31T21:35:42.723 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 21:12:26 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:hP1uXxRwqPN9evN7pLR4dhogLN8=', 'Content-Length': '0'} 2023-05-31T21:35:42.723 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '8'), ('X-RGW-Bytes-Used', '24'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000dca3325349f0b971-006477b83a-1630-a1'), ('Content-Length', '0'), ('Date', 'Wed, 31 May 2023 21:12:26 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:35:42.724 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/pklxkd-64/obj68 2023-05-31T21:35:42.724 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/pklxkd-64/obj68 2023-05-31T21:35:42.724 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2023-05-31T21:35:42.724 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /pklxkd-64/obj68 2023-05-31T21:35:42.725 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-05-31T21:35:42.725 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-05-31T21:35:42.725 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T21:35:42.725 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-05-31T21:35:42.725 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-05-31T21:35:42.726 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-05-31T21:35:42.726 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-05-31T21:35:42.726 INFO:tasks.rgw_multisite_tests:PUT 2023-05-31T21:35:42.726 INFO:tasks.rgw_multisite_tests:rL0Y20zC+Fzt72VPzMSk2A== 2023-05-31T21:35:42.727 INFO:tasks.rgw_multisite_tests:application/octet-stream 2023-05-31T21:35:42.727 INFO:tasks.rgw_multisite_tests:Wed, 31 May 2023 21:12:26 GMT 2023-05-31T21:35:42.727 INFO:tasks.rgw_multisite_tests:/pklxkd-64/obj68 2023-05-31T21:35:42.727 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-05-31T21:35:42.728 INFO:tasks.rgw_multisite_tests:AWS Q80H11OLNU1S48PPYL26:RsEodSXQsrVbxryD3yyUqPyQLvU= 2023-05-31T21:35:42.728 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Wed, 31 May 2023 21:12:26 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:RsEodSXQsrVbxryD3yyUqPyQLvU='} 2023-05-31T21:35:42.728 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000037c652e39bbf1cc4-006477b83a-1630-a1'), ('Date', 'Wed, 31 May 2023 21:12:26 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:35:42.728 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/pklxkd-64/ 2023-05-31T21:35:42.728 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/pklxkd-64/ 2023-05-31T21:35:42.729 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-05-31T21:35:42.729 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /pklxkd-64/ 2023-05-31T21:35:42.729 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-05-31T21:35:42.729 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-05-31T21:35:42.730 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T21:35:42.730 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-05-31T21:35:42.730 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-05-31T21:35:42.730 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-05-31T21:35:42.730 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-05-31T21:35:42.731 INFO:tasks.rgw_multisite_tests:HEAD 2023-05-31T21:35:42.731 INFO:tasks.rgw_multisite_tests: 2023-05-31T21:35:42.731 INFO:tasks.rgw_multisite_tests: 2023-05-31T21:35:42.731 INFO:tasks.rgw_multisite_tests:Wed, 31 May 2023 21:12:26 GMT 2023-05-31T21:35:42.732 INFO:tasks.rgw_multisite_tests:/pklxkd-64/ 2023-05-31T21:35:42.732 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-05-31T21:35:42.732 INFO:tasks.rgw_multisite_tests:AWS Q80H11OLNU1S48PPYL26:hP1uXxRwqPN9evN7pLR4dhogLN8= 2023-05-31T21:35:42.732 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 21:12:26 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:hP1uXxRwqPN9evN7pLR4dhogLN8=', 'Content-Length': '0'} 2023-05-31T21:35:42.733 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '9'), ('X-RGW-Bytes-Used', '27'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000835a50cb2f758df4-006477b83a-1630-a1'), ('Content-Length', '0'), ('Date', 'Wed, 31 May 2023 21:12:26 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:35:42.733 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/pklxkd-64/obj85 2023-05-31T21:35:42.733 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/pklxkd-64/obj85 2023-05-31T21:35:42.733 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2023-05-31T21:35:42.733 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /pklxkd-64/obj85 2023-05-31T21:35:42.734 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-05-31T21:35:42.734 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-05-31T21:35:42.734 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T21:35:42.734 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-05-31T21:35:42.735 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-05-31T21:35:42.735 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-05-31T21:35:42.735 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-05-31T21:35:42.735 INFO:tasks.rgw_multisite_tests:PUT 2023-05-31T21:35:42.736 INFO:tasks.rgw_multisite_tests:rL0Y20zC+Fzt72VPzMSk2A== 2023-05-31T21:35:42.736 INFO:tasks.rgw_multisite_tests:application/octet-stream 2023-05-31T21:35:42.736 INFO:tasks.rgw_multisite_tests:Wed, 31 May 2023 21:12:26 GMT 2023-05-31T21:35:42.736 INFO:tasks.rgw_multisite_tests:/pklxkd-64/obj85 2023-05-31T21:35:42.736 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-05-31T21:35:42.737 INFO:tasks.rgw_multisite_tests:AWS Q80H11OLNU1S48PPYL26:keSx+iMcDOy4YwcK4PcsPrFnLiY= 2023-05-31T21:35:42.737 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Wed, 31 May 2023 21:12:26 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:keSx+iMcDOy4YwcK4PcsPrFnLiY='} 2023-05-31T21:35:42.737 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000047c73a5bb275012b-006477b83a-1630-a1'), ('Date', 'Wed, 31 May 2023 21:12:26 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:35:42.737 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['bucket', 'reshard', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'pklxkd-64', '--num-shards', '19', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-05-31T21:35:42.738 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'reshard', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'pklxkd-64', '--num-shards', '19', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-05-31T21:35:42.738 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket reshard --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --bucket pklxkd-64 --num-shards 19 --yes-i-really-mean-it --cluster c1 --debug-rgw 0 --debug-ms 0 2023-05-31T21:35:42.738 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: ignoring --setuser ceph since I am not root 2023-05-31T21:35:42.738 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-05-31T21:35:42.739 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: tenant: 2023-05-31T21:35:42.739 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: bucket name: pklxkd-64 2023-05-31T21:35:42.739 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: total entries: 10 2023-05-31T21:35:42.739 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/pklxkd-64/ 2023-05-31T21:35:42.739 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/pklxkd-64/ 2023-05-31T21:35:42.740 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-05-31T21:35:42.740 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /pklxkd-64/ 2023-05-31T21:35:42.740 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-05-31T21:35:42.740 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-05-31T21:35:42.741 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T21:35:42.741 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-05-31T21:35:42.741 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-05-31T21:35:42.741 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-05-31T21:35:42.742 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-05-31T21:35:42.742 INFO:tasks.rgw_multisite_tests:HEAD 2023-05-31T21:35:42.742 INFO:tasks.rgw_multisite_tests: 2023-05-31T21:35:42.742 INFO:tasks.rgw_multisite_tests: 2023-05-31T21:35:42.743 INFO:tasks.rgw_multisite_tests:Wed, 31 May 2023 21:12:27 GMT 2023-05-31T21:35:42.743 INFO:tasks.rgw_multisite_tests:/pklxkd-64/ 2023-05-31T21:35:42.743 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-05-31T21:35:42.743 INFO:tasks.rgw_multisite_tests:AWS Q80H11OLNU1S48PPYL26:5X+CZRYkJIw4yCbpLG+ZuT3v7Tg= 2023-05-31T21:35:42.743 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 21:12:27 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:5X+CZRYkJIw4yCbpLG+ZuT3v7Tg=', 'Content-Length': '0'} 2023-05-31T21:35:42.744 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '10'), ('X-RGW-Bytes-Used', '30'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000047597428e64ddea2-006477b83b-1630-a1'), ('Content-Length', '0'), ('Date', 'Wed, 31 May 2023 21:12:27 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:35:42.744 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/pklxkd-64/obj19 2023-05-31T21:35:42.744 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/pklxkd-64/obj19 2023-05-31T21:35:42.744 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2023-05-31T21:35:42.745 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /pklxkd-64/obj19 2023-05-31T21:35:42.745 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-05-31T21:35:42.745 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-05-31T21:35:42.745 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T21:35:42.746 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-05-31T21:35:42.746 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-05-31T21:35:42.746 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-05-31T21:35:42.746 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-05-31T21:35:42.747 INFO:tasks.rgw_multisite_tests:PUT 2023-05-31T21:35:42.747 INFO:tasks.rgw_multisite_tests:rL0Y20zC+Fzt72VPzMSk2A== 2023-05-31T21:35:42.747 INFO:tasks.rgw_multisite_tests:application/octet-stream 2023-05-31T21:35:42.747 INFO:tasks.rgw_multisite_tests:Wed, 31 May 2023 21:12:27 GMT 2023-05-31T21:35:42.748 INFO:tasks.rgw_multisite_tests:/pklxkd-64/obj19 2023-05-31T21:35:42.748 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-05-31T21:35:42.748 INFO:tasks.rgw_multisite_tests:AWS Q80H11OLNU1S48PPYL26:A2JdQ9/5ABu80cWWI9FSo44ZsjY= 2023-05-31T21:35:42.748 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Wed, 31 May 2023 21:12:27 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:A2JdQ9/5ABu80cWWI9FSo44ZsjY='} 2023-05-31T21:35:42.748 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000090c769d364ce31af-006477b83b-1630-a1'), ('Date', 'Wed, 31 May 2023 21:12:27 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:35:42.749 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/pklxkd-64/ 2023-05-31T21:35:42.749 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/pklxkd-64/ 2023-05-31T21:35:42.749 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-05-31T21:35:42.749 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /pklxkd-64/ 2023-05-31T21:35:42.750 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-05-31T21:35:42.750 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-05-31T21:35:42.750 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T21:35:42.750 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-05-31T21:35:42.751 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-05-31T21:35:42.751 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-05-31T21:35:42.751 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-05-31T21:35:42.751 INFO:tasks.rgw_multisite_tests:HEAD 2023-05-31T21:35:42.751 INFO:tasks.rgw_multisite_tests: 2023-05-31T21:35:42.752 INFO:tasks.rgw_multisite_tests: 2023-05-31T21:35:42.752 INFO:tasks.rgw_multisite_tests:Wed, 31 May 2023 21:12:27 GMT 2023-05-31T21:35:42.752 INFO:tasks.rgw_multisite_tests:/pklxkd-64/ 2023-05-31T21:35:42.752 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-05-31T21:35:42.753 INFO:tasks.rgw_multisite_tests:AWS Q80H11OLNU1S48PPYL26:5X+CZRYkJIw4yCbpLG+ZuT3v7Tg= 2023-05-31T21:35:42.753 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 21:12:27 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:5X+CZRYkJIw4yCbpLG+ZuT3v7Tg=', 'Content-Length': '0'} 2023-05-31T21:35:42.753 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '11'), ('X-RGW-Bytes-Used', '33'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000007c56c81ad97d7096-006477b83b-1630-a1'), ('Content-Length', '0'), ('Date', 'Wed, 31 May 2023 21:12:27 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:35:42.753 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/pklxkd-64/obj38 2023-05-31T21:35:42.754 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/pklxkd-64/obj38 2023-05-31T21:35:42.754 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2023-05-31T21:35:42.754 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /pklxkd-64/obj38 2023-05-31T21:35:42.754 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-05-31T21:35:42.754 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-05-31T21:35:42.755 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T21:35:42.755 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-05-31T21:35:42.755 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-05-31T21:35:42.755 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-05-31T21:35:42.756 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-05-31T21:35:42.756 INFO:tasks.rgw_multisite_tests:PUT 2023-05-31T21:35:42.756 INFO:tasks.rgw_multisite_tests:rL0Y20zC+Fzt72VPzMSk2A== 2023-05-31T21:35:42.756 INFO:tasks.rgw_multisite_tests:application/octet-stream 2023-05-31T21:35:42.757 INFO:tasks.rgw_multisite_tests:Wed, 31 May 2023 21:12:27 GMT 2023-05-31T21:35:42.757 INFO:tasks.rgw_multisite_tests:/pklxkd-64/obj38 2023-05-31T21:35:42.757 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-05-31T21:35:42.757 INFO:tasks.rgw_multisite_tests:AWS Q80H11OLNU1S48PPYL26:Vb9bEB0EerBFaVWfIMaOQieqPRY= 2023-05-31T21:35:42.758 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Wed, 31 May 2023 21:12:27 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:Vb9bEB0EerBFaVWfIMaOQieqPRY='} 2023-05-31T21:35:42.758 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000b8e4e43c12f7782f-006477b83b-1630-a1'), ('Date', 'Wed, 31 May 2023 21:12:27 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:35:42.758 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/pklxkd-64/ 2023-05-31T21:35:42.758 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/pklxkd-64/ 2023-05-31T21:35:42.759 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-05-31T21:35:42.759 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /pklxkd-64/ 2023-05-31T21:35:42.759 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-05-31T21:35:42.759 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-05-31T21:35:42.759 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T21:35:42.760 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-05-31T21:35:42.760 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-05-31T21:35:42.760 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-05-31T21:35:42.760 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-05-31T21:35:42.761 INFO:tasks.rgw_multisite_tests:HEAD 2023-05-31T21:35:42.761 INFO:tasks.rgw_multisite_tests: 2023-05-31T21:35:42.761 INFO:tasks.rgw_multisite_tests: 2023-05-31T21:35:42.761 INFO:tasks.rgw_multisite_tests:Wed, 31 May 2023 21:12:27 GMT 2023-05-31T21:35:42.762 INFO:tasks.rgw_multisite_tests:/pklxkd-64/ 2023-05-31T21:35:42.762 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-05-31T21:35:42.762 INFO:tasks.rgw_multisite_tests:AWS Q80H11OLNU1S48PPYL26:5X+CZRYkJIw4yCbpLG+ZuT3v7Tg= 2023-05-31T21:35:42.762 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 21:12:27 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:5X+CZRYkJIw4yCbpLG+ZuT3v7Tg=', 'Content-Length': '0'} 2023-05-31T21:35:42.762 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '12'), ('X-RGW-Bytes-Used', '36'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000f530210c80d6c0bf-006477b83b-1630-a1'), ('Content-Length', '0'), ('Date', 'Wed, 31 May 2023 21:12:27 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:35:42.763 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/pklxkd-64/obj57 2023-05-31T21:35:42.763 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/pklxkd-64/obj57 2023-05-31T21:35:42.763 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2023-05-31T21:35:42.763 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /pklxkd-64/obj57 2023-05-31T21:35:42.764 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-05-31T21:35:42.764 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-05-31T21:35:42.764 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T21:35:42.764 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-05-31T21:35:42.765 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-05-31T21:35:42.765 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-05-31T21:35:42.765 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-05-31T21:35:42.765 INFO:tasks.rgw_multisite_tests:PUT 2023-05-31T21:35:42.766 INFO:tasks.rgw_multisite_tests:rL0Y20zC+Fzt72VPzMSk2A== 2023-05-31T21:35:42.766 INFO:tasks.rgw_multisite_tests:application/octet-stream 2023-05-31T21:35:42.766 INFO:tasks.rgw_multisite_tests:Wed, 31 May 2023 21:12:27 GMT 2023-05-31T21:35:42.766 INFO:tasks.rgw_multisite_tests:/pklxkd-64/obj57 2023-05-31T21:35:42.766 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-05-31T21:35:42.767 INFO:tasks.rgw_multisite_tests:AWS Q80H11OLNU1S48PPYL26:T1Q6dGrai9KIojbw0x4os6xu57E= 2023-05-31T21:35:42.767 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Wed, 31 May 2023 21:12:27 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:T1Q6dGrai9KIojbw0x4os6xu57E='} 2023-05-31T21:35:42.767 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000fcaebd6c75751c2f-006477b83b-1630-a1'), ('Date', 'Wed, 31 May 2023 21:12:27 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:35:42.767 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/pklxkd-64/ 2023-05-31T21:35:42.768 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/pklxkd-64/ 2023-05-31T21:35:42.768 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-05-31T21:35:42.768 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /pklxkd-64/ 2023-05-31T21:35:42.768 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-05-31T21:35:42.769 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-05-31T21:35:42.769 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T21:35:42.769 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-05-31T21:35:42.769 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-05-31T21:35:42.769 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-05-31T21:35:42.770 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-05-31T21:35:42.770 INFO:tasks.rgw_multisite_tests:HEAD 2023-05-31T21:35:42.770 INFO:tasks.rgw_multisite_tests: 2023-05-31T21:35:42.770 INFO:tasks.rgw_multisite_tests: 2023-05-31T21:35:42.771 INFO:tasks.rgw_multisite_tests:Wed, 31 May 2023 21:12:27 GMT 2023-05-31T21:35:42.771 INFO:tasks.rgw_multisite_tests:/pklxkd-64/ 2023-05-31T21:35:42.771 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-05-31T21:35:42.771 INFO:tasks.rgw_multisite_tests:AWS Q80H11OLNU1S48PPYL26:5X+CZRYkJIw4yCbpLG+ZuT3v7Tg= 2023-05-31T21:35:42.772 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 21:12:27 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:5X+CZRYkJIw4yCbpLG+ZuT3v7Tg=', 'Content-Length': '0'} 2023-05-31T21:35:42.772 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '13'), ('X-RGW-Bytes-Used', '39'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000cb8a08ebf87ab97c-006477b83b-1630-a1'), ('Content-Length', '0'), ('Date', 'Wed, 31 May 2023 21:12:27 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:35:42.772 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/pklxkd-64/obj76 2023-05-31T21:35:42.772 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/pklxkd-64/obj76 2023-05-31T21:35:42.772 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2023-05-31T21:35:42.773 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /pklxkd-64/obj76 2023-05-31T21:35:42.773 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-05-31T21:35:42.773 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-05-31T21:35:42.773 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T21:35:42.774 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-05-31T21:35:42.774 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-05-31T21:35:42.774 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-05-31T21:35:42.774 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-05-31T21:35:42.775 INFO:tasks.rgw_multisite_tests:PUT 2023-05-31T21:35:42.775 INFO:tasks.rgw_multisite_tests:rL0Y20zC+Fzt72VPzMSk2A== 2023-05-31T21:35:42.775 INFO:tasks.rgw_multisite_tests:application/octet-stream 2023-05-31T21:35:42.775 INFO:tasks.rgw_multisite_tests:Wed, 31 May 2023 21:12:27 GMT 2023-05-31T21:35:42.775 INFO:tasks.rgw_multisite_tests:/pklxkd-64/obj76 2023-05-31T21:35:42.776 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-05-31T21:35:42.776 INFO:tasks.rgw_multisite_tests:AWS Q80H11OLNU1S48PPYL26:OnUg3LFa+4PHCkrdCam2Hzgfpd4= 2023-05-31T21:35:42.776 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Wed, 31 May 2023 21:12:27 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:OnUg3LFa+4PHCkrdCam2Hzgfpd4='} 2023-05-31T21:35:42.776 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000fcb4f66aa66bd147-006477b83b-1630-a1'), ('Date', 'Wed, 31 May 2023 21:12:27 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:35:42.777 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/pklxkd-64/ 2023-05-31T21:35:42.777 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/pklxkd-64/ 2023-05-31T21:35:42.777 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-05-31T21:35:42.777 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /pklxkd-64/ 2023-05-31T21:35:42.778 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-05-31T21:35:42.778 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-05-31T21:35:42.778 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T21:35:42.778 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-05-31T21:35:42.779 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-05-31T21:35:42.779 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-05-31T21:35:42.779 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-05-31T21:35:42.779 INFO:tasks.rgw_multisite_tests:HEAD 2023-05-31T21:35:42.779 INFO:tasks.rgw_multisite_tests: 2023-05-31T21:35:42.780 INFO:tasks.rgw_multisite_tests: 2023-05-31T21:35:42.780 INFO:tasks.rgw_multisite_tests:Wed, 31 May 2023 21:12:27 GMT 2023-05-31T21:35:42.780 INFO:tasks.rgw_multisite_tests:/pklxkd-64/ 2023-05-31T21:35:42.780 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-05-31T21:35:42.781 INFO:tasks.rgw_multisite_tests:AWS Q80H11OLNU1S48PPYL26:5X+CZRYkJIw4yCbpLG+ZuT3v7Tg= 2023-05-31T21:35:42.781 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 21:12:27 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:5X+CZRYkJIw4yCbpLG+ZuT3v7Tg=', 'Content-Length': '0'} 2023-05-31T21:35:42.781 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '14'), ('X-RGW-Bytes-Used', '42'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000bc9292d68385041e-006477b83b-1630-a1'), ('Content-Length', '0'), ('Date', 'Wed, 31 May 2023 21:12:27 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:35:42.781 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/pklxkd-64/obj95 2023-05-31T21:35:42.782 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/pklxkd-64/obj95 2023-05-31T21:35:42.782 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2023-05-31T21:35:42.782 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /pklxkd-64/obj95 2023-05-31T21:35:42.782 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-05-31T21:35:42.782 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-05-31T21:35:42.783 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T21:35:42.783 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-05-31T21:35:42.783 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-05-31T21:35:42.783 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-05-31T21:35:42.784 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-05-31T21:35:42.784 INFO:tasks.rgw_multisite_tests:PUT 2023-05-31T21:35:42.784 INFO:tasks.rgw_multisite_tests:rL0Y20zC+Fzt72VPzMSk2A== 2023-05-31T21:35:42.784 INFO:tasks.rgw_multisite_tests:application/octet-stream 2023-05-31T21:35:42.785 INFO:tasks.rgw_multisite_tests:Wed, 31 May 2023 21:12:27 GMT 2023-05-31T21:35:42.785 INFO:tasks.rgw_multisite_tests:/pklxkd-64/obj95 2023-05-31T21:35:42.785 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-05-31T21:35:42.785 INFO:tasks.rgw_multisite_tests:AWS Q80H11OLNU1S48PPYL26:b0WHs+gaxTAohHBUeJZEhCprsuw= 2023-05-31T21:35:42.786 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Wed, 31 May 2023 21:12:27 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:b0WHs+gaxTAohHBUeJZEhCprsuw='} 2023-05-31T21:35:42.786 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000007be561e131667703-006477b83b-1630-a1'), ('Date', 'Wed, 31 May 2023 21:12:27 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:35:42.786 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['bucket', 'reshard', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'pklxkd-64', '--num-shards', '23', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-05-31T21:35:42.786 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'reshard', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'pklxkd-64', '--num-shards', '23', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-05-31T21:35:42.787 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket reshard --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --bucket pklxkd-64 --num-shards 23 --yes-i-really-mean-it --cluster c1 --debug-rgw 0 --debug-ms 0 2023-05-31T21:35:42.787 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: ignoring --setuser ceph since I am not root 2023-05-31T21:35:42.787 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-05-31T21:35:42.787 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: tenant: 2023-05-31T21:35:42.788 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: bucket name: pklxkd-64 2023-05-31T21:35:42.788 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: total entries: 15 2023-05-31T21:35:42.788 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/pklxkd-64/ 2023-05-31T21:35:42.788 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/pklxkd-64/ 2023-05-31T21:35:42.789 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-05-31T21:35:42.789 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /pklxkd-64/ 2023-05-31T21:35:42.789 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-05-31T21:35:42.789 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-05-31T21:35:42.790 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T21:35:42.790 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-05-31T21:35:42.790 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-05-31T21:35:42.790 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-05-31T21:35:42.791 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-05-31T21:35:42.791 INFO:tasks.rgw_multisite_tests:HEAD 2023-05-31T21:35:42.791 INFO:tasks.rgw_multisite_tests: 2023-05-31T21:35:42.791 INFO:tasks.rgw_multisite_tests: 2023-05-31T21:35:42.792 INFO:tasks.rgw_multisite_tests:Wed, 31 May 2023 21:12:27 GMT 2023-05-31T21:35:42.792 INFO:tasks.rgw_multisite_tests:/pklxkd-64/ 2023-05-31T21:35:42.792 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-05-31T21:35:42.792 INFO:tasks.rgw_multisite_tests:AWS Q80H11OLNU1S48PPYL26:5X+CZRYkJIw4yCbpLG+ZuT3v7Tg= 2023-05-31T21:35:42.793 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 21:12:27 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:5X+CZRYkJIw4yCbpLG+ZuT3v7Tg=', 'Content-Length': '0'} 2023-05-31T21:35:42.793 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '15'), ('X-RGW-Bytes-Used', '45'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000d869f571c73db76d-006477b83b-1630-a1'), ('Content-Length', '0'), ('Date', 'Wed, 31 May 2023 21:12:27 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:35:42.793 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/pklxkd-64/obj23 2023-05-31T21:35:42.793 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/pklxkd-64/obj23 2023-05-31T21:35:42.793 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2023-05-31T21:35:42.794 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /pklxkd-64/obj23 2023-05-31T21:35:42.794 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-05-31T21:35:42.794 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-05-31T21:35:42.794 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T21:35:42.795 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-05-31T21:35:42.795 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-05-31T21:35:42.795 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-05-31T21:35:42.795 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-05-31T21:35:42.796 INFO:tasks.rgw_multisite_tests:PUT 2023-05-31T21:35:42.796 INFO:tasks.rgw_multisite_tests:rL0Y20zC+Fzt72VPzMSk2A== 2023-05-31T21:35:42.796 INFO:tasks.rgw_multisite_tests:application/octet-stream 2023-05-31T21:35:42.796 INFO:tasks.rgw_multisite_tests:Wed, 31 May 2023 21:12:27 GMT 2023-05-31T21:35:42.797 INFO:tasks.rgw_multisite_tests:/pklxkd-64/obj23 2023-05-31T21:35:42.797 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-05-31T21:35:42.797 INFO:tasks.rgw_multisite_tests:AWS Q80H11OLNU1S48PPYL26:W6y6Sx+q5uJUypBgeAdudmkJ1WY= 2023-05-31T21:35:42.797 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Wed, 31 May 2023 21:12:27 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:W6y6Sx+q5uJUypBgeAdudmkJ1WY='} 2023-05-31T21:35:42.797 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000003830f63fcc2aab6f-006477b83b-1630-a1'), ('Date', 'Wed, 31 May 2023 21:12:27 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:35:42.798 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/pklxkd-64/ 2023-05-31T21:35:42.798 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/pklxkd-64/ 2023-05-31T21:35:42.798 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-05-31T21:35:42.798 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /pklxkd-64/ 2023-05-31T21:35:42.799 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-05-31T21:35:42.799 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-05-31T21:35:42.799 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T21:35:42.799 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-05-31T21:35:42.800 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-05-31T21:35:42.800 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-05-31T21:35:42.800 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-05-31T21:35:42.800 INFO:tasks.rgw_multisite_tests:HEAD 2023-05-31T21:35:42.801 INFO:tasks.rgw_multisite_tests: 2023-05-31T21:35:42.801 INFO:tasks.rgw_multisite_tests: 2023-05-31T21:35:42.801 INFO:tasks.rgw_multisite_tests:Wed, 31 May 2023 21:12:27 GMT 2023-05-31T21:35:42.801 INFO:tasks.rgw_multisite_tests:/pklxkd-64/ 2023-05-31T21:35:42.801 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-05-31T21:35:42.802 INFO:tasks.rgw_multisite_tests:AWS Q80H11OLNU1S48PPYL26:5X+CZRYkJIw4yCbpLG+ZuT3v7Tg= 2023-05-31T21:35:42.802 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 21:12:27 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:5X+CZRYkJIw4yCbpLG+ZuT3v7Tg=', 'Content-Length': '0'} 2023-05-31T21:35:42.802 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '16'), ('X-RGW-Bytes-Used', '48'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000030a4be84dd1dd6a3-006477b83b-1630-a1'), ('Content-Length', '0'), ('Date', 'Wed, 31 May 2023 21:12:27 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:35:42.802 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/pklxkd-64/obj46 2023-05-31T21:35:42.803 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/pklxkd-64/obj46 2023-05-31T21:35:42.803 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2023-05-31T21:35:42.803 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /pklxkd-64/obj46 2023-05-31T21:35:42.803 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-05-31T21:35:42.804 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-05-31T21:35:42.804 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T21:35:42.804 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-05-31T21:35:42.804 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-05-31T21:35:42.804 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-05-31T21:35:42.805 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-05-31T21:35:42.805 INFO:tasks.rgw_multisite_tests:PUT 2023-05-31T21:35:42.805 INFO:tasks.rgw_multisite_tests:rL0Y20zC+Fzt72VPzMSk2A== 2023-05-31T21:35:42.805 INFO:tasks.rgw_multisite_tests:application/octet-stream 2023-05-31T21:35:42.806 INFO:tasks.rgw_multisite_tests:Wed, 31 May 2023 21:12:27 GMT 2023-05-31T21:35:42.806 INFO:tasks.rgw_multisite_tests:/pklxkd-64/obj46 2023-05-31T21:35:42.806 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-05-31T21:35:42.806 INFO:tasks.rgw_multisite_tests:AWS Q80H11OLNU1S48PPYL26:+v8b/mAzsWDWLlfFczRLwaAIU6s= 2023-05-31T21:35:42.807 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Wed, 31 May 2023 21:12:27 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:+v8b/mAzsWDWLlfFczRLwaAIU6s='} 2023-05-31T21:35:42.807 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000861cb24573876db2-006477b83b-1630-a1'), ('Date', 'Wed, 31 May 2023 21:12:27 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:35:42.807 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/pklxkd-64/ 2023-05-31T21:35:42.807 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/pklxkd-64/ 2023-05-31T21:35:42.807 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-05-31T21:35:42.808 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /pklxkd-64/ 2023-05-31T21:35:42.808 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-05-31T21:35:42.808 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-05-31T21:35:42.808 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T21:35:42.809 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-05-31T21:35:42.809 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-05-31T21:35:42.809 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-05-31T21:35:42.809 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-05-31T21:35:42.810 INFO:tasks.rgw_multisite_tests:HEAD 2023-05-31T21:35:42.810 INFO:tasks.rgw_multisite_tests: 2023-05-31T21:35:42.810 INFO:tasks.rgw_multisite_tests: 2023-05-31T21:35:42.810 INFO:tasks.rgw_multisite_tests:Wed, 31 May 2023 21:12:27 GMT 2023-05-31T21:35:42.810 INFO:tasks.rgw_multisite_tests:/pklxkd-64/ 2023-05-31T21:35:42.811 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-05-31T21:35:42.811 INFO:tasks.rgw_multisite_tests:AWS Q80H11OLNU1S48PPYL26:5X+CZRYkJIw4yCbpLG+ZuT3v7Tg= 2023-05-31T21:35:42.811 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 21:12:27 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:5X+CZRYkJIw4yCbpLG+ZuT3v7Tg=', 'Content-Length': '0'} 2023-05-31T21:35:42.811 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '17'), ('X-RGW-Bytes-Used', '51'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000f4eab07ae429c1e9-006477b83b-1630-a1'), ('Content-Length', '0'), ('Date', 'Wed, 31 May 2023 21:12:27 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:35:42.812 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/pklxkd-64/obj69 2023-05-31T21:35:42.812 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/pklxkd-64/obj69 2023-05-31T21:35:42.812 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2023-05-31T21:35:42.812 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /pklxkd-64/obj69 2023-05-31T21:35:42.813 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-05-31T21:35:42.813 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-05-31T21:35:42.813 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T21:35:42.813 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-05-31T21:35:42.813 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-05-31T21:35:42.814 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-05-31T21:35:42.814 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-05-31T21:35:42.814 INFO:tasks.rgw_multisite_tests:PUT 2023-05-31T21:35:42.814 INFO:tasks.rgw_multisite_tests:rL0Y20zC+Fzt72VPzMSk2A== 2023-05-31T21:35:42.815 INFO:tasks.rgw_multisite_tests:application/octet-stream 2023-05-31T21:35:42.815 INFO:tasks.rgw_multisite_tests:Wed, 31 May 2023 21:12:27 GMT 2023-05-31T21:35:42.815 INFO:tasks.rgw_multisite_tests:/pklxkd-64/obj69 2023-05-31T21:35:42.815 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-05-31T21:35:42.815 INFO:tasks.rgw_multisite_tests:AWS Q80H11OLNU1S48PPYL26:xGQkOVG17u+wkUcSi8jToV8jB88= 2023-05-31T21:35:42.816 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Wed, 31 May 2023 21:12:27 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:xGQkOVG17u+wkUcSi8jToV8jB88='} 2023-05-31T21:35:42.816 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000ba7a4a31fb53d4eb-006477b83b-1630-a1'), ('Date', 'Wed, 31 May 2023 21:12:27 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:35:42.816 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/pklxkd-64/ 2023-05-31T21:35:42.816 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/pklxkd-64/ 2023-05-31T21:35:42.817 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-05-31T21:35:42.817 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /pklxkd-64/ 2023-05-31T21:35:42.817 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-05-31T21:35:42.817 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-05-31T21:35:42.818 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T21:35:42.818 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-05-31T21:35:42.818 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-05-31T21:35:42.818 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-05-31T21:35:42.819 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-05-31T21:35:42.819 INFO:tasks.rgw_multisite_tests:HEAD 2023-05-31T21:35:42.819 INFO:tasks.rgw_multisite_tests: 2023-05-31T21:35:42.819 INFO:tasks.rgw_multisite_tests: 2023-05-31T21:35:42.819 INFO:tasks.rgw_multisite_tests:Wed, 31 May 2023 21:12:27 GMT 2023-05-31T21:35:42.820 INFO:tasks.rgw_multisite_tests:/pklxkd-64/ 2023-05-31T21:35:42.820 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-05-31T21:35:42.820 INFO:tasks.rgw_multisite_tests:AWS Q80H11OLNU1S48PPYL26:5X+CZRYkJIw4yCbpLG+ZuT3v7Tg= 2023-05-31T21:35:42.820 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 21:12:27 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:5X+CZRYkJIw4yCbpLG+ZuT3v7Tg=', 'Content-Length': '0'} 2023-05-31T21:35:42.821 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '18'), ('X-RGW-Bytes-Used', '54'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000957971fdda725249-006477b83b-1630-a1'), ('Content-Length', '0'), ('Date', 'Wed, 31 May 2023 21:12:27 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:35:42.821 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/pklxkd-64/obj92 2023-05-31T21:35:42.821 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/pklxkd-64/obj92 2023-05-31T21:35:42.821 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2023-05-31T21:35:42.822 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /pklxkd-64/obj92 2023-05-31T21:35:42.822 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-05-31T21:35:42.822 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-05-31T21:35:42.822 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T21:35:42.823 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-05-31T21:35:42.823 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-05-31T21:35:42.823 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-05-31T21:35:42.823 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-05-31T21:35:42.824 INFO:tasks.rgw_multisite_tests:PUT 2023-05-31T21:35:42.824 INFO:tasks.rgw_multisite_tests:rL0Y20zC+Fzt72VPzMSk2A== 2023-05-31T21:35:42.824 INFO:tasks.rgw_multisite_tests:application/octet-stream 2023-05-31T21:35:42.824 INFO:tasks.rgw_multisite_tests:Wed, 31 May 2023 21:12:27 GMT 2023-05-31T21:35:42.825 INFO:tasks.rgw_multisite_tests:/pklxkd-64/obj92 2023-05-31T21:35:42.825 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-05-31T21:35:42.825 INFO:tasks.rgw_multisite_tests:AWS Q80H11OLNU1S48PPYL26:ymh+SiqHVZO5Gjm728vj5qQJ5tI= 2023-05-31T21:35:42.825 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Wed, 31 May 2023 21:12:27 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:ymh+SiqHVZO5Gjm728vj5qQJ5tI='} 2023-05-31T21:35:42.825 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000aff85a7af419da3e-006477b83b-1630-a1'), ('Date', 'Wed, 31 May 2023 21:12:27 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:35:42.826 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/pklxkd-64/ 2023-05-31T21:35:42.826 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/pklxkd-64/ 2023-05-31T21:35:42.826 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-05-31T21:35:42.826 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /pklxkd-64/ 2023-05-31T21:35:42.827 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-05-31T21:35:42.827 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-05-31T21:35:42.827 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T21:35:42.827 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-05-31T21:35:42.828 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-05-31T21:35:42.828 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-05-31T21:35:42.828 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-05-31T21:35:42.828 INFO:tasks.rgw_multisite_tests:HEAD 2023-05-31T21:35:42.828 INFO:tasks.rgw_multisite_tests: 2023-05-31T21:35:42.829 INFO:tasks.rgw_multisite_tests: 2023-05-31T21:35:42.829 INFO:tasks.rgw_multisite_tests:Wed, 31 May 2023 21:12:27 GMT 2023-05-31T21:35:42.829 INFO:tasks.rgw_multisite_tests:/pklxkd-64/ 2023-05-31T21:35:42.829 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-05-31T21:35:42.830 INFO:tasks.rgw_multisite_tests:AWS Q80H11OLNU1S48PPYL26:5X+CZRYkJIw4yCbpLG+ZuT3v7Tg= 2023-05-31T21:35:42.830 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 21:12:27 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:5X+CZRYkJIw4yCbpLG+ZuT3v7Tg=', 'Content-Length': '0'} 2023-05-31T21:35:42.830 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '19'), ('X-RGW-Bytes-Used', '57'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000c07a17f39d8fdcce-006477b83b-1630-a1'), ('Content-Length', '0'), ('Date', 'Wed, 31 May 2023 21:12:27 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:35:42.830 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/pklxkd-64/obj115 2023-05-31T21:35:42.831 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/pklxkd-64/obj115 2023-05-31T21:35:42.831 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2023-05-31T21:35:42.831 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /pklxkd-64/obj115 2023-05-31T21:35:42.831 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-05-31T21:35:42.831 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-05-31T21:35:42.832 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T21:35:42.832 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-05-31T21:35:42.832 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-05-31T21:35:42.832 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-05-31T21:35:42.833 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-05-31T21:35:42.833 INFO:tasks.rgw_multisite_tests:PUT 2023-05-31T21:35:42.833 INFO:tasks.rgw_multisite_tests:rL0Y20zC+Fzt72VPzMSk2A== 2023-05-31T21:35:42.833 INFO:tasks.rgw_multisite_tests:application/octet-stream 2023-05-31T21:35:42.834 INFO:tasks.rgw_multisite_tests:Wed, 31 May 2023 21:12:27 GMT 2023-05-31T21:35:42.834 INFO:tasks.rgw_multisite_tests:/pklxkd-64/obj115 2023-05-31T21:35:42.834 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-05-31T21:35:42.834 INFO:tasks.rgw_multisite_tests:AWS Q80H11OLNU1S48PPYL26:ooz3ewEud8Vv3By4QWkU3r1snyI= 2023-05-31T21:35:42.835 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Wed, 31 May 2023 21:12:27 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:ooz3ewEud8Vv3By4QWkU3r1snyI='} 2023-05-31T21:35:42.835 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000009b862c5a1d7d8cbb-006477b83b-1630-a1'), ('Date', 'Wed, 31 May 2023 21:12:27 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:35:42.835 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['bucket', 'reshard', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'pklxkd-64', '--num-shards', '29', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-05-31T21:35:42.835 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'reshard', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'pklxkd-64', '--num-shards', '29', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-05-31T21:35:42.835 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket reshard --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --bucket pklxkd-64 --num-shards 29 --yes-i-really-mean-it --cluster c1 --debug-rgw 0 --debug-ms 0 2023-05-31T21:35:42.836 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: ignoring --setuser ceph since I am not root 2023-05-31T21:35:42.836 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-05-31T21:35:42.836 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: tenant: 2023-05-31T21:35:42.836 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: bucket name: pklxkd-64 2023-05-31T21:35:42.837 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: total entries: 20 2023-05-31T21:35:42.837 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/pklxkd-64/ 2023-05-31T21:35:42.837 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/pklxkd-64/ 2023-05-31T21:35:42.837 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-05-31T21:35:42.838 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /pklxkd-64/ 2023-05-31T21:35:42.838 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-05-31T21:35:42.838 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-05-31T21:35:42.838 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T21:35:42.838 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-05-31T21:35:42.839 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-05-31T21:35:42.839 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-05-31T21:35:42.839 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-05-31T21:35:42.839 INFO:tasks.rgw_multisite_tests:HEAD 2023-05-31T21:35:42.840 INFO:tasks.rgw_multisite_tests: 2023-05-31T21:35:42.840 INFO:tasks.rgw_multisite_tests: 2023-05-31T21:35:42.840 INFO:tasks.rgw_multisite_tests:Wed, 31 May 2023 21:12:27 GMT 2023-05-31T21:35:42.840 INFO:tasks.rgw_multisite_tests:/pklxkd-64/ 2023-05-31T21:35:42.841 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-05-31T21:35:42.841 INFO:tasks.rgw_multisite_tests:AWS Q80H11OLNU1S48PPYL26:5X+CZRYkJIw4yCbpLG+ZuT3v7Tg= 2023-05-31T21:35:42.841 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 21:12:27 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:5X+CZRYkJIw4yCbpLG+ZuT3v7Tg=', 'Content-Length': '0'} 2023-05-31T21:35:42.841 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '20'), ('X-RGW-Bytes-Used', '60'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000543e72c4cec958f4-006477b83b-1630-a1'), ('Content-Length', '0'), ('Date', 'Wed, 31 May 2023 21:12:27 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:35:42.841 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/pklxkd-64/obj29 2023-05-31T21:35:42.842 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/pklxkd-64/obj29 2023-05-31T21:35:42.842 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2023-05-31T21:35:42.842 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /pklxkd-64/obj29 2023-05-31T21:35:42.842 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-05-31T21:35:42.843 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-05-31T21:35:42.843 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T21:35:42.843 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-05-31T21:35:42.843 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-05-31T21:35:42.844 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-05-31T21:35:42.844 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-05-31T21:35:42.844 INFO:tasks.rgw_multisite_tests:PUT 2023-05-31T21:35:42.844 INFO:tasks.rgw_multisite_tests:rL0Y20zC+Fzt72VPzMSk2A== 2023-05-31T21:35:42.844 INFO:tasks.rgw_multisite_tests:application/octet-stream 2023-05-31T21:35:42.845 INFO:tasks.rgw_multisite_tests:Wed, 31 May 2023 21:12:27 GMT 2023-05-31T21:35:42.845 INFO:tasks.rgw_multisite_tests:/pklxkd-64/obj29 2023-05-31T21:35:42.845 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-05-31T21:35:42.846 INFO:tasks.rgw_multisite_tests:AWS Q80H11OLNU1S48PPYL26:t7QT1RnKQ1Uo0s5Xjhh5UkMsRUI= 2023-05-31T21:35:42.846 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Wed, 31 May 2023 21:12:27 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:t7QT1RnKQ1Uo0s5Xjhh5UkMsRUI='} 2023-05-31T21:35:42.846 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000073c5576572c5f2d5-006477b83b-1630-a1'), ('Date', 'Wed, 31 May 2023 21:12:27 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:35:42.846 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/pklxkd-64/ 2023-05-31T21:35:42.847 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/pklxkd-64/ 2023-05-31T21:35:42.847 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-05-31T21:35:42.847 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /pklxkd-64/ 2023-05-31T21:35:42.847 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-05-31T21:35:42.847 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-05-31T21:35:42.848 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T21:35:42.848 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-05-31T21:35:42.848 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-05-31T21:35:42.848 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-05-31T21:35:42.849 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-05-31T21:35:42.849 INFO:tasks.rgw_multisite_tests:HEAD 2023-05-31T21:35:42.849 INFO:tasks.rgw_multisite_tests: 2023-05-31T21:35:42.849 INFO:tasks.rgw_multisite_tests: 2023-05-31T21:35:42.850 INFO:tasks.rgw_multisite_tests:Wed, 31 May 2023 21:12:27 GMT 2023-05-31T21:35:42.850 INFO:tasks.rgw_multisite_tests:/pklxkd-64/ 2023-05-31T21:35:42.850 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-05-31T21:35:42.850 INFO:tasks.rgw_multisite_tests:AWS Q80H11OLNU1S48PPYL26:5X+CZRYkJIw4yCbpLG+ZuT3v7Tg= 2023-05-31T21:35:42.850 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 21:12:27 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:5X+CZRYkJIw4yCbpLG+ZuT3v7Tg=', 'Content-Length': '0'} 2023-05-31T21:35:42.851 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '21'), ('X-RGW-Bytes-Used', '63'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000006b0b7911504667f4-006477b83b-1630-a1'), ('Content-Length', '0'), ('Date', 'Wed, 31 May 2023 21:12:27 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:35:42.851 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/pklxkd-64/obj58 2023-05-31T21:35:42.851 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/pklxkd-64/obj58 2023-05-31T21:35:42.851 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2023-05-31T21:35:42.852 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /pklxkd-64/obj58 2023-05-31T21:35:42.852 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-05-31T21:35:42.852 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-05-31T21:35:42.852 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T21:35:42.853 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-05-31T21:35:42.853 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-05-31T21:35:42.853 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-05-31T21:35:42.853 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-05-31T21:35:42.853 INFO:tasks.rgw_multisite_tests:PUT 2023-05-31T21:35:42.854 INFO:tasks.rgw_multisite_tests:rL0Y20zC+Fzt72VPzMSk2A== 2023-05-31T21:35:42.854 INFO:tasks.rgw_multisite_tests:application/octet-stream 2023-05-31T21:35:42.854 INFO:tasks.rgw_multisite_tests:Wed, 31 May 2023 21:12:27 GMT 2023-05-31T21:35:42.854 INFO:tasks.rgw_multisite_tests:/pklxkd-64/obj58 2023-05-31T21:35:42.855 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-05-31T21:35:42.855 INFO:tasks.rgw_multisite_tests:AWS Q80H11OLNU1S48PPYL26:9I8FljyiHRtiyAl9DFT5eSkTJGI= 2023-05-31T21:35:42.855 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Wed, 31 May 2023 21:12:27 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:9I8FljyiHRtiyAl9DFT5eSkTJGI='} 2023-05-31T21:35:42.855 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000002306688ce75ac555-006477b83b-1630-a1'), ('Date', 'Wed, 31 May 2023 21:12:27 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:35:42.856 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/pklxkd-64/ 2023-05-31T21:35:42.856 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/pklxkd-64/ 2023-05-31T21:35:42.856 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-05-31T21:35:42.856 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /pklxkd-64/ 2023-05-31T21:35:42.857 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-05-31T21:35:42.857 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-05-31T21:35:42.857 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T21:35:42.857 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-05-31T21:35:42.858 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-05-31T21:35:42.858 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-05-31T21:35:42.858 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-05-31T21:35:42.858 INFO:tasks.rgw_multisite_tests:HEAD 2023-05-31T21:35:42.858 INFO:tasks.rgw_multisite_tests: 2023-05-31T21:35:42.859 INFO:tasks.rgw_multisite_tests: 2023-05-31T21:35:42.859 INFO:tasks.rgw_multisite_tests:Wed, 31 May 2023 21:12:27 GMT 2023-05-31T21:35:42.859 INFO:tasks.rgw_multisite_tests:/pklxkd-64/ 2023-05-31T21:35:42.859 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-05-31T21:35:42.860 INFO:tasks.rgw_multisite_tests:AWS Q80H11OLNU1S48PPYL26:5X+CZRYkJIw4yCbpLG+ZuT3v7Tg= 2023-05-31T21:35:42.860 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 21:12:27 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:5X+CZRYkJIw4yCbpLG+ZuT3v7Tg=', 'Content-Length': '0'} 2023-05-31T21:35:42.860 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '22'), ('X-RGW-Bytes-Used', '66'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000fbf3f5a8ab3e526d-006477b83b-1630-a1'), ('Content-Length', '0'), ('Date', 'Wed, 31 May 2023 21:12:27 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:35:42.860 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/pklxkd-64/obj87 2023-05-31T21:35:42.861 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/pklxkd-64/obj87 2023-05-31T21:35:42.861 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2023-05-31T21:35:42.861 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /pklxkd-64/obj87 2023-05-31T21:35:42.861 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-05-31T21:35:42.861 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-05-31T21:35:42.862 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T21:35:42.862 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-05-31T21:35:42.862 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-05-31T21:35:42.862 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-05-31T21:35:42.863 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-05-31T21:35:42.863 INFO:tasks.rgw_multisite_tests:PUT 2023-05-31T21:35:42.863 INFO:tasks.rgw_multisite_tests:rL0Y20zC+Fzt72VPzMSk2A== 2023-05-31T21:35:42.863 INFO:tasks.rgw_multisite_tests:application/octet-stream 2023-05-31T21:35:42.863 INFO:tasks.rgw_multisite_tests:Wed, 31 May 2023 21:12:27 GMT 2023-05-31T21:35:42.864 INFO:tasks.rgw_multisite_tests:/pklxkd-64/obj87 2023-05-31T21:35:42.864 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-05-31T21:35:42.864 INFO:tasks.rgw_multisite_tests:AWS Q80H11OLNU1S48PPYL26:K5xk9bbxgpZuDo7eiNcISG+mzQA= 2023-05-31T21:35:42.864 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Wed, 31 May 2023 21:12:27 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:K5xk9bbxgpZuDo7eiNcISG+mzQA='} 2023-05-31T21:35:42.865 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000e46a35ebe04ed500-006477b83b-1630-a1'), ('Date', 'Wed, 31 May 2023 21:12:27 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:35:42.865 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/pklxkd-64/ 2023-05-31T21:35:42.865 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/pklxkd-64/ 2023-05-31T21:35:42.865 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-05-31T21:35:42.866 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /pklxkd-64/ 2023-05-31T21:35:42.866 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-05-31T21:35:42.866 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-05-31T21:35:42.866 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T21:35:42.866 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-05-31T21:35:42.867 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-05-31T21:35:42.867 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-05-31T21:35:42.867 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-05-31T21:35:42.867 INFO:tasks.rgw_multisite_tests:HEAD 2023-05-31T21:35:42.868 INFO:tasks.rgw_multisite_tests: 2023-05-31T21:35:42.868 INFO:tasks.rgw_multisite_tests: 2023-05-31T21:35:42.868 INFO:tasks.rgw_multisite_tests:Wed, 31 May 2023 21:12:27 GMT 2023-05-31T21:35:42.868 INFO:tasks.rgw_multisite_tests:/pklxkd-64/ 2023-05-31T21:35:42.869 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-05-31T21:35:42.869 INFO:tasks.rgw_multisite_tests:AWS Q80H11OLNU1S48PPYL26:5X+CZRYkJIw4yCbpLG+ZuT3v7Tg= 2023-05-31T21:35:42.869 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 21:12:27 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:5X+CZRYkJIw4yCbpLG+ZuT3v7Tg=', 'Content-Length': '0'} 2023-05-31T21:35:42.869 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '23'), ('X-RGW-Bytes-Used', '69'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000003644d7c46a50a743-006477b83b-1630-a1'), ('Content-Length', '0'), ('Date', 'Wed, 31 May 2023 21:12:27 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:35:42.869 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/pklxkd-64/obj116 2023-05-31T21:35:42.870 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/pklxkd-64/obj116 2023-05-31T21:35:42.870 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2023-05-31T21:35:42.870 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /pklxkd-64/obj116 2023-05-31T21:35:42.870 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-05-31T21:35:42.871 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-05-31T21:35:42.871 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T21:35:42.871 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-05-31T21:35:42.871 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-05-31T21:35:42.872 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-05-31T21:35:42.872 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-05-31T21:35:42.872 INFO:tasks.rgw_multisite_tests:PUT 2023-05-31T21:35:42.872 INFO:tasks.rgw_multisite_tests:rL0Y20zC+Fzt72VPzMSk2A== 2023-05-31T21:35:42.872 INFO:tasks.rgw_multisite_tests:application/octet-stream 2023-05-31T21:35:42.873 INFO:tasks.rgw_multisite_tests:Wed, 31 May 2023 21:12:27 GMT 2023-05-31T21:35:42.873 INFO:tasks.rgw_multisite_tests:/pklxkd-64/obj116 2023-05-31T21:35:42.873 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-05-31T21:35:42.873 INFO:tasks.rgw_multisite_tests:AWS Q80H11OLNU1S48PPYL26:0GB6LjmEjDniQHpt1z6yvzOxfqA= 2023-05-31T21:35:42.874 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Wed, 31 May 2023 21:12:27 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:0GB6LjmEjDniQHpt1z6yvzOxfqA='} 2023-05-31T21:35:42.874 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000d02f82117d4df397-006477b83b-1630-a1'), ('Date', 'Wed, 31 May 2023 21:12:27 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:35:42.874 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/pklxkd-64/ 2023-05-31T21:35:42.874 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/pklxkd-64/ 2023-05-31T21:35:42.875 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-05-31T21:35:42.875 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /pklxkd-64/ 2023-05-31T21:35:42.875 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-05-31T21:35:42.875 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-05-31T21:35:42.875 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T21:35:42.876 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-05-31T21:35:42.876 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-05-31T21:35:42.876 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-05-31T21:35:42.876 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-05-31T21:35:42.877 INFO:tasks.rgw_multisite_tests:HEAD 2023-05-31T21:35:42.877 INFO:tasks.rgw_multisite_tests: 2023-05-31T21:35:42.877 INFO:tasks.rgw_multisite_tests: 2023-05-31T21:35:42.877 INFO:tasks.rgw_multisite_tests:Wed, 31 May 2023 21:12:27 GMT 2023-05-31T21:35:42.878 INFO:tasks.rgw_multisite_tests:/pklxkd-64/ 2023-05-31T21:35:42.878 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-05-31T21:35:42.878 INFO:tasks.rgw_multisite_tests:AWS Q80H11OLNU1S48PPYL26:5X+CZRYkJIw4yCbpLG+ZuT3v7Tg= 2023-05-31T21:35:42.878 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 21:12:27 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:5X+CZRYkJIw4yCbpLG+ZuT3v7Tg=', 'Content-Length': '0'} 2023-05-31T21:35:42.878 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '24'), ('X-RGW-Bytes-Used', '72'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000005d279c20b2596ab0-006477b83b-1630-a1'), ('Content-Length', '0'), ('Date', 'Wed, 31 May 2023 21:12:27 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:35:42.879 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/pklxkd-64/obj145 2023-05-31T21:35:42.879 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/pklxkd-64/obj145 2023-05-31T21:35:42.879 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2023-05-31T21:35:42.879 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /pklxkd-64/obj145 2023-05-31T21:35:42.880 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-05-31T21:35:42.880 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-05-31T21:35:42.880 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T21:35:42.880 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-05-31T21:35:42.881 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-05-31T21:35:42.881 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-05-31T21:35:42.881 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-05-31T21:35:42.881 INFO:tasks.rgw_multisite_tests:PUT 2023-05-31T21:35:42.881 INFO:tasks.rgw_multisite_tests:rL0Y20zC+Fzt72VPzMSk2A== 2023-05-31T21:35:42.882 INFO:tasks.rgw_multisite_tests:application/octet-stream 2023-05-31T21:35:42.882 INFO:tasks.rgw_multisite_tests:Wed, 31 May 2023 21:12:27 GMT 2023-05-31T21:35:42.882 INFO:tasks.rgw_multisite_tests:/pklxkd-64/obj145 2023-05-31T21:35:42.882 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-05-31T21:35:42.883 INFO:tasks.rgw_multisite_tests:AWS Q80H11OLNU1S48PPYL26:Ybsyo2W+0a7PkvpH4cKZS+1fMos= 2023-05-31T21:35:42.883 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Wed, 31 May 2023 21:12:27 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:Ybsyo2W+0a7PkvpH4cKZS+1fMos='} 2023-05-31T21:35:42.883 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000007eadb198801829c3-006477b83b-1630-a1'), ('Date', 'Wed, 31 May 2023 21:12:27 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:35:42.883 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['bucket', 'reshard', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'pklxkd-64', '--num-shards', '31', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-05-31T21:35:42.884 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'reshard', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'pklxkd-64', '--num-shards', '31', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-05-31T21:35:42.884 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket reshard --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --bucket pklxkd-64 --num-shards 31 --yes-i-really-mean-it --cluster c1 --debug-rgw 0 --debug-ms 0 2023-05-31T21:35:42.884 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: ignoring --setuser ceph since I am not root 2023-05-31T21:35:42.884 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-05-31T21:35:42.884 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: tenant: 2023-05-31T21:35:42.885 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: bucket name: pklxkd-64 2023-05-31T21:35:42.885 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: total entries: 25 2023-05-31T21:35:42.885 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/pklxkd-64/ 2023-05-31T21:35:42.885 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/pklxkd-64/ 2023-05-31T21:35:42.886 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-05-31T21:35:42.886 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /pklxkd-64/ 2023-05-31T21:35:42.886 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-05-31T21:35:42.886 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-05-31T21:35:42.887 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T21:35:42.887 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-05-31T21:35:42.887 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-05-31T21:35:42.887 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-05-31T21:35:42.888 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-05-31T21:35:42.888 INFO:tasks.rgw_multisite_tests:HEAD 2023-05-31T21:35:42.888 INFO:tasks.rgw_multisite_tests: 2023-05-31T21:35:42.888 INFO:tasks.rgw_multisite_tests: 2023-05-31T21:35:42.888 INFO:tasks.rgw_multisite_tests:Wed, 31 May 2023 21:12:28 GMT 2023-05-31T21:35:42.889 INFO:tasks.rgw_multisite_tests:/pklxkd-64/ 2023-05-31T21:35:42.889 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-05-31T21:35:42.889 INFO:tasks.rgw_multisite_tests:AWS Q80H11OLNU1S48PPYL26:wRm+Nzs6+vQGxm2DWUEut4l9HJU= 2023-05-31T21:35:42.889 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 21:12:28 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:wRm+Nzs6+vQGxm2DWUEut4l9HJU=', 'Content-Length': '0'} 2023-05-31T21:35:42.890 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '25'), ('X-RGW-Bytes-Used', '75'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000026e9078c7e3b079-006477b83c-1630-a1'), ('Content-Length', '0'), ('Date', 'Wed, 31 May 2023 21:12:28 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:35:42.890 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/pklxkd-64/obj31 2023-05-31T21:35:42.890 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/pklxkd-64/obj31 2023-05-31T21:35:42.890 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2023-05-31T21:35:42.891 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /pklxkd-64/obj31 2023-05-31T21:35:42.891 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-05-31T21:35:42.891 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-05-31T21:35:42.891 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T21:35:42.891 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-05-31T21:35:42.892 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-05-31T21:35:42.892 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-05-31T21:35:42.892 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-05-31T21:35:42.892 INFO:tasks.rgw_multisite_tests:PUT 2023-05-31T21:35:42.893 INFO:tasks.rgw_multisite_tests:rL0Y20zC+Fzt72VPzMSk2A== 2023-05-31T21:35:42.893 INFO:tasks.rgw_multisite_tests:application/octet-stream 2023-05-31T21:35:42.893 INFO:tasks.rgw_multisite_tests:Wed, 31 May 2023 21:12:28 GMT 2023-05-31T21:35:42.893 INFO:tasks.rgw_multisite_tests:/pklxkd-64/obj31 2023-05-31T21:35:42.894 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-05-31T21:35:42.894 INFO:tasks.rgw_multisite_tests:AWS Q80H11OLNU1S48PPYL26:xMsZxJtxILTfdzX032FxNs2P278= 2023-05-31T21:35:42.894 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Wed, 31 May 2023 21:12:28 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:xMsZxJtxILTfdzX032FxNs2P278='} 2023-05-31T21:35:42.894 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000dad4ea32b5673cb2-006477b83c-1630-a1'), ('Date', 'Wed, 31 May 2023 21:12:28 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:35:42.894 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/pklxkd-64/ 2023-05-31T21:35:42.895 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/pklxkd-64/ 2023-05-31T21:35:42.895 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-05-31T21:35:42.895 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /pklxkd-64/ 2023-05-31T21:35:42.895 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-05-31T21:35:42.896 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-05-31T21:35:42.896 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T21:35:42.896 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-05-31T21:35:42.896 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-05-31T21:35:42.897 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-05-31T21:35:42.897 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-05-31T21:35:42.897 INFO:tasks.rgw_multisite_tests:HEAD 2023-05-31T21:35:42.897 INFO:tasks.rgw_multisite_tests: 2023-05-31T21:35:42.898 INFO:tasks.rgw_multisite_tests: 2023-05-31T21:35:42.898 INFO:tasks.rgw_multisite_tests:Wed, 31 May 2023 21:12:28 GMT 2023-05-31T21:35:42.898 INFO:tasks.rgw_multisite_tests:/pklxkd-64/ 2023-05-31T21:35:42.898 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-05-31T21:35:42.898 INFO:tasks.rgw_multisite_tests:AWS Q80H11OLNU1S48PPYL26:wRm+Nzs6+vQGxm2DWUEut4l9HJU= 2023-05-31T21:35:42.899 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 21:12:28 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:wRm+Nzs6+vQGxm2DWUEut4l9HJU=', 'Content-Length': '0'} 2023-05-31T21:35:42.899 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '26'), ('X-RGW-Bytes-Used', '78'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000688681a10b388eff-006477b83c-1630-a1'), ('Content-Length', '0'), ('Date', 'Wed, 31 May 2023 21:12:28 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:35:42.899 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/pklxkd-64/obj62 2023-05-31T21:35:42.899 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/pklxkd-64/obj62 2023-05-31T21:35:42.900 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2023-05-31T21:35:42.900 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /pklxkd-64/obj62 2023-05-31T21:35:42.900 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-05-31T21:35:42.900 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-05-31T21:35:42.901 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T21:35:42.901 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-05-31T21:35:42.901 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-05-31T21:35:42.901 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-05-31T21:35:42.902 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-05-31T21:35:42.902 INFO:tasks.rgw_multisite_tests:PUT 2023-05-31T21:35:42.902 INFO:tasks.rgw_multisite_tests:rL0Y20zC+Fzt72VPzMSk2A== 2023-05-31T21:35:42.902 INFO:tasks.rgw_multisite_tests:application/octet-stream 2023-05-31T21:35:42.902 INFO:tasks.rgw_multisite_tests:Wed, 31 May 2023 21:12:28 GMT 2023-05-31T21:35:42.903 INFO:tasks.rgw_multisite_tests:/pklxkd-64/obj62 2023-05-31T21:35:42.903 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-05-31T21:35:42.903 INFO:tasks.rgw_multisite_tests:AWS Q80H11OLNU1S48PPYL26:d1DyCtPDHtuQTxUCfMZO0lvjABY= 2023-05-31T21:35:42.903 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Wed, 31 May 2023 21:12:28 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:d1DyCtPDHtuQTxUCfMZO0lvjABY='} 2023-05-31T21:35:42.904 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000cb80b6c2b0bdbcf3-006477b83c-1630-a1'), ('Date', 'Wed, 31 May 2023 21:12:28 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:35:42.904 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/pklxkd-64/ 2023-05-31T21:35:42.904 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/pklxkd-64/ 2023-05-31T21:35:42.904 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-05-31T21:35:42.905 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /pklxkd-64/ 2023-05-31T21:35:42.905 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-05-31T21:35:42.905 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-05-31T21:35:42.905 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T21:35:42.905 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-05-31T21:35:42.906 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-05-31T21:35:42.906 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-05-31T21:35:42.906 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-05-31T21:35:42.906 INFO:tasks.rgw_multisite_tests:HEAD 2023-05-31T21:35:42.907 INFO:tasks.rgw_multisite_tests: 2023-05-31T21:35:42.907 INFO:tasks.rgw_multisite_tests: 2023-05-31T21:35:42.907 INFO:tasks.rgw_multisite_tests:Wed, 31 May 2023 21:12:28 GMT 2023-05-31T21:35:42.907 INFO:tasks.rgw_multisite_tests:/pklxkd-64/ 2023-05-31T21:35:42.908 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-05-31T21:35:42.908 INFO:tasks.rgw_multisite_tests:AWS Q80H11OLNU1S48PPYL26:wRm+Nzs6+vQGxm2DWUEut4l9HJU= 2023-05-31T21:35:42.908 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 21:12:28 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:wRm+Nzs6+vQGxm2DWUEut4l9HJU=', 'Content-Length': '0'} 2023-05-31T21:35:42.908 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '27'), ('X-RGW-Bytes-Used', '81'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000011de4131e59e949-006477b83c-1630-a1'), ('Content-Length', '0'), ('Date', 'Wed, 31 May 2023 21:12:28 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:35:42.908 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/pklxkd-64/obj93 2023-05-31T21:35:42.909 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/pklxkd-64/obj93 2023-05-31T21:35:42.909 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2023-05-31T21:35:42.909 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /pklxkd-64/obj93 2023-05-31T21:35:42.909 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-05-31T21:35:42.910 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-05-31T21:35:42.910 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T21:35:42.910 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-05-31T21:35:42.910 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-05-31T21:35:42.911 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-05-31T21:35:42.911 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-05-31T21:35:42.911 INFO:tasks.rgw_multisite_tests:PUT 2023-05-31T21:35:42.911 INFO:tasks.rgw_multisite_tests:rL0Y20zC+Fzt72VPzMSk2A== 2023-05-31T21:35:42.911 INFO:tasks.rgw_multisite_tests:application/octet-stream 2023-05-31T21:35:42.912 INFO:tasks.rgw_multisite_tests:Wed, 31 May 2023 21:12:28 GMT 2023-05-31T21:35:42.912 INFO:tasks.rgw_multisite_tests:/pklxkd-64/obj93 2023-05-31T21:35:42.912 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-05-31T21:35:42.912 INFO:tasks.rgw_multisite_tests:AWS Q80H11OLNU1S48PPYL26:4DH1gTpDYbojyofOP8+LM6MvJhQ= 2023-05-31T21:35:42.913 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Wed, 31 May 2023 21:12:28 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:4DH1gTpDYbojyofOP8+LM6MvJhQ='} 2023-05-31T21:35:42.913 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000df6a8753180e4e4b-006477b83c-1630-a1'), ('Date', 'Wed, 31 May 2023 21:12:28 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:35:42.913 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/pklxkd-64/ 2023-05-31T21:35:42.913 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/pklxkd-64/ 2023-05-31T21:35:42.914 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-05-31T21:35:42.914 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /pklxkd-64/ 2023-05-31T21:35:42.914 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-05-31T21:35:42.914 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-05-31T21:35:42.914 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T21:35:42.915 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-05-31T21:35:42.915 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-05-31T21:35:42.915 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-05-31T21:35:42.915 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-05-31T21:35:42.916 INFO:tasks.rgw_multisite_tests:HEAD 2023-05-31T21:35:42.916 INFO:tasks.rgw_multisite_tests: 2023-05-31T21:35:42.916 INFO:tasks.rgw_multisite_tests: 2023-05-31T21:35:42.916 INFO:tasks.rgw_multisite_tests:Wed, 31 May 2023 21:12:28 GMT 2023-05-31T21:35:42.917 INFO:tasks.rgw_multisite_tests:/pklxkd-64/ 2023-05-31T21:35:42.917 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-05-31T21:35:42.917 INFO:tasks.rgw_multisite_tests:AWS Q80H11OLNU1S48PPYL26:wRm+Nzs6+vQGxm2DWUEut4l9HJU= 2023-05-31T21:35:42.917 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 21:12:28 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:wRm+Nzs6+vQGxm2DWUEut4l9HJU=', 'Content-Length': '0'} 2023-05-31T21:35:42.917 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '28'), ('X-RGW-Bytes-Used', '84'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000006498dacbcc2710c0-006477b83c-1630-a1'), ('Content-Length', '0'), ('Date', 'Wed, 31 May 2023 21:12:28 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:35:42.918 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/pklxkd-64/obj124 2023-05-31T21:35:42.918 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/pklxkd-64/obj124 2023-05-31T21:35:42.918 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2023-05-31T21:35:42.918 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /pklxkd-64/obj124 2023-05-31T21:35:42.919 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-05-31T21:35:42.919 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-05-31T21:35:42.919 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T21:35:42.919 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-05-31T21:35:42.919 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-05-31T21:35:42.920 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-05-31T21:35:42.920 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-05-31T21:35:42.920 INFO:tasks.rgw_multisite_tests:PUT 2023-05-31T21:35:42.920 INFO:tasks.rgw_multisite_tests:rL0Y20zC+Fzt72VPzMSk2A== 2023-05-31T21:35:42.921 INFO:tasks.rgw_multisite_tests:application/octet-stream 2023-05-31T21:35:42.921 INFO:tasks.rgw_multisite_tests:Wed, 31 May 2023 21:12:28 GMT 2023-05-31T21:35:42.921 INFO:tasks.rgw_multisite_tests:/pklxkd-64/obj124 2023-05-31T21:35:42.921 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-05-31T21:35:42.922 INFO:tasks.rgw_multisite_tests:AWS Q80H11OLNU1S48PPYL26:i9QnqSzgpLYMUplJoj3OIyIlFYg= 2023-05-31T21:35:42.922 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Wed, 31 May 2023 21:12:28 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:i9QnqSzgpLYMUplJoj3OIyIlFYg='} 2023-05-31T21:35:42.922 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000f74be47a51de8ab5-006477b83c-1630-a1'), ('Date', 'Wed, 31 May 2023 21:12:28 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:35:42.922 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/pklxkd-64/ 2023-05-31T21:35:42.922 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/pklxkd-64/ 2023-05-31T21:35:42.923 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-05-31T21:35:42.923 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /pklxkd-64/ 2023-05-31T21:35:42.923 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-05-31T21:35:42.923 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-05-31T21:35:42.924 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T21:35:42.924 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-05-31T21:35:42.924 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-05-31T21:35:42.924 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-05-31T21:35:42.925 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-05-31T21:35:42.925 INFO:tasks.rgw_multisite_tests:HEAD 2023-05-31T21:35:42.925 INFO:tasks.rgw_multisite_tests: 2023-05-31T21:35:42.925 INFO:tasks.rgw_multisite_tests: 2023-05-31T21:35:42.925 INFO:tasks.rgw_multisite_tests:Wed, 31 May 2023 21:12:28 GMT 2023-05-31T21:35:42.926 INFO:tasks.rgw_multisite_tests:/pklxkd-64/ 2023-05-31T21:35:42.926 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-05-31T21:35:42.926 INFO:tasks.rgw_multisite_tests:AWS Q80H11OLNU1S48PPYL26:wRm+Nzs6+vQGxm2DWUEut4l9HJU= 2023-05-31T21:35:42.926 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 21:12:28 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:wRm+Nzs6+vQGxm2DWUEut4l9HJU=', 'Content-Length': '0'} 2023-05-31T21:35:42.927 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '29'), ('X-RGW-Bytes-Used', '87'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000d726c76c09455665-006477b83c-1630-a1'), ('Content-Length', '0'), ('Date', 'Wed, 31 May 2023 21:12:28 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:35:42.927 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/pklxkd-64/obj155 2023-05-31T21:35:42.927 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/pklxkd-64/obj155 2023-05-31T21:35:42.927 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2023-05-31T21:35:42.928 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /pklxkd-64/obj155 2023-05-31T21:35:42.928 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-05-31T21:35:42.928 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-05-31T21:35:42.928 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T21:35:42.928 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-05-31T21:35:42.929 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-05-31T21:35:42.929 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-05-31T21:35:42.929 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-05-31T21:35:42.929 INFO:tasks.rgw_multisite_tests:PUT 2023-05-31T21:35:42.930 INFO:tasks.rgw_multisite_tests:rL0Y20zC+Fzt72VPzMSk2A== 2023-05-31T21:35:42.930 INFO:tasks.rgw_multisite_tests:application/octet-stream 2023-05-31T21:35:42.930 INFO:tasks.rgw_multisite_tests:Wed, 31 May 2023 21:12:28 GMT 2023-05-31T21:35:42.930 INFO:tasks.rgw_multisite_tests:/pklxkd-64/obj155 2023-05-31T21:35:42.930 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-05-31T21:35:42.931 INFO:tasks.rgw_multisite_tests:AWS Q80H11OLNU1S48PPYL26:1rueDycE+KWjtf+YsP6oWag+EL8= 2023-05-31T21:35:42.931 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Wed, 31 May 2023 21:12:28 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:1rueDycE+KWjtf+YsP6oWag+EL8='} 2023-05-31T21:35:42.931 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000035038da524831a5e-006477b83c-1630-a1'), ('Date', 'Wed, 31 May 2023 21:12:28 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:35:42.931 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['bucket', 'reshard', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'pklxkd-64', '--num-shards', '37', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-05-31T21:35:42.932 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'reshard', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'pklxkd-64', '--num-shards', '37', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-05-31T21:35:42.932 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket reshard --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --bucket pklxkd-64 --num-shards 37 --yes-i-really-mean-it --cluster c1 --debug-rgw 0 --debug-ms 0 2023-05-31T21:35:42.932 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: ignoring --setuser ceph since I am not root 2023-05-31T21:35:42.932 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-05-31T21:35:42.933 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: tenant: 2023-05-31T21:35:42.933 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: bucket name: pklxkd-64 2023-05-31T21:35:42.933 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: total entries: 30 2023-05-31T21:35:42.933 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/pklxkd-64/ 2023-05-31T21:35:42.933 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/pklxkd-64/ 2023-05-31T21:35:42.934 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-05-31T21:35:42.934 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /pklxkd-64/ 2023-05-31T21:35:42.934 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-05-31T21:35:42.934 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-05-31T21:35:42.935 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T21:35:42.935 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-05-31T21:35:42.935 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-05-31T21:35:42.935 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-05-31T21:35:42.936 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-05-31T21:35:42.936 INFO:tasks.rgw_multisite_tests:HEAD 2023-05-31T21:35:42.936 INFO:tasks.rgw_multisite_tests: 2023-05-31T21:35:42.936 INFO:tasks.rgw_multisite_tests: 2023-05-31T21:35:42.936 INFO:tasks.rgw_multisite_tests:Wed, 31 May 2023 21:12:28 GMT 2023-05-31T21:35:42.937 INFO:tasks.rgw_multisite_tests:/pklxkd-64/ 2023-05-31T21:35:42.937 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-05-31T21:35:42.937 INFO:tasks.rgw_multisite_tests:AWS Q80H11OLNU1S48PPYL26:wRm+Nzs6+vQGxm2DWUEut4l9HJU= 2023-05-31T21:35:42.937 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 21:12:28 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:wRm+Nzs6+vQGxm2DWUEut4l9HJU=', 'Content-Length': '0'} 2023-05-31T21:35:42.938 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '30'), ('X-RGW-Bytes-Used', '90'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000020b7b9eef9416f4b-006477b83c-1630-a1'), ('Content-Length', '0'), ('Date', 'Wed, 31 May 2023 21:12:28 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:35:42.938 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/pklxkd-64/obj37 2023-05-31T21:35:42.938 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/pklxkd-64/obj37 2023-05-31T21:35:42.938 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2023-05-31T21:35:42.939 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /pklxkd-64/obj37 2023-05-31T21:35:42.939 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-05-31T21:35:42.939 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-05-31T21:35:42.939 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T21:35:42.939 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-05-31T21:35:42.940 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-05-31T21:35:42.940 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-05-31T21:35:42.940 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-05-31T21:35:42.940 INFO:tasks.rgw_multisite_tests:PUT 2023-05-31T21:35:42.941 INFO:tasks.rgw_multisite_tests:rL0Y20zC+Fzt72VPzMSk2A== 2023-05-31T21:35:42.941 INFO:tasks.rgw_multisite_tests:application/octet-stream 2023-05-31T21:35:42.941 INFO:tasks.rgw_multisite_tests:Wed, 31 May 2023 21:12:28 GMT 2023-05-31T21:35:42.941 INFO:tasks.rgw_multisite_tests:/pklxkd-64/obj37 2023-05-31T21:35:42.942 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-05-31T21:35:42.942 INFO:tasks.rgw_multisite_tests:AWS Q80H11OLNU1S48PPYL26:NHXLlWggdAotn2hmkZiWiZKRvN4= 2023-05-31T21:35:42.942 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Wed, 31 May 2023 21:12:28 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:NHXLlWggdAotn2hmkZiWiZKRvN4='} 2023-05-31T21:35:42.942 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000002f914c6c1a1c0f4e-006477b83c-1630-a1'), ('Date', 'Wed, 31 May 2023 21:12:28 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:35:42.942 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/pklxkd-64/ 2023-05-31T21:35:42.943 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/pklxkd-64/ 2023-05-31T21:35:42.943 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-05-31T21:35:42.943 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /pklxkd-64/ 2023-05-31T21:35:42.943 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-05-31T21:35:42.944 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-05-31T21:35:42.944 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T21:35:42.944 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-05-31T21:35:42.944 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-05-31T21:35:42.945 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-05-31T21:35:42.945 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-05-31T21:35:42.945 INFO:tasks.rgw_multisite_tests:HEAD 2023-05-31T21:35:42.945 INFO:tasks.rgw_multisite_tests: 2023-05-31T21:35:42.945 INFO:tasks.rgw_multisite_tests: 2023-05-31T21:35:42.946 INFO:tasks.rgw_multisite_tests:Wed, 31 May 2023 21:12:28 GMT 2023-05-31T21:35:42.946 INFO:tasks.rgw_multisite_tests:/pklxkd-64/ 2023-05-31T21:35:42.946 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-05-31T21:35:42.946 INFO:tasks.rgw_multisite_tests:AWS Q80H11OLNU1S48PPYL26:wRm+Nzs6+vQGxm2DWUEut4l9HJU= 2023-05-31T21:35:42.947 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 21:12:28 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:wRm+Nzs6+vQGxm2DWUEut4l9HJU=', 'Content-Length': '0'} 2023-05-31T21:35:42.947 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '31'), ('X-RGW-Bytes-Used', '93'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000002d96a6b944dd8e88-006477b83c-1630-a1'), ('Content-Length', '0'), ('Date', 'Wed, 31 May 2023 21:12:28 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:35:42.947 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/pklxkd-64/obj74 2023-05-31T21:35:42.947 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/pklxkd-64/obj74 2023-05-31T21:35:42.947 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2023-05-31T21:35:42.948 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /pklxkd-64/obj74 2023-05-31T21:35:42.948 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-05-31T21:35:42.948 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-05-31T21:35:42.948 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T21:35:42.949 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-05-31T21:35:42.949 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-05-31T21:35:42.949 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-05-31T21:35:42.949 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-05-31T21:35:42.950 INFO:tasks.rgw_multisite_tests:PUT 2023-05-31T21:35:42.950 INFO:tasks.rgw_multisite_tests:rL0Y20zC+Fzt72VPzMSk2A== 2023-05-31T21:35:42.950 INFO:tasks.rgw_multisite_tests:application/octet-stream 2023-05-31T21:35:42.950 INFO:tasks.rgw_multisite_tests:Wed, 31 May 2023 21:12:28 GMT 2023-05-31T21:35:42.951 INFO:tasks.rgw_multisite_tests:/pklxkd-64/obj74 2023-05-31T21:35:42.951 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-05-31T21:35:42.951 INFO:tasks.rgw_multisite_tests:AWS Q80H11OLNU1S48PPYL26:JO1sVYCYBNr10jHyEESI41YkW4c= 2023-05-31T21:35:42.951 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Wed, 31 May 2023 21:12:28 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:JO1sVYCYBNr10jHyEESI41YkW4c='} 2023-05-31T21:35:42.951 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000089d7680e128596fa-006477b83c-1630-a1'), ('Date', 'Wed, 31 May 2023 21:12:28 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:35:42.952 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/pklxkd-64/ 2023-05-31T21:35:42.952 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/pklxkd-64/ 2023-05-31T21:35:42.952 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-05-31T21:35:42.952 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /pklxkd-64/ 2023-05-31T21:35:42.953 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-05-31T21:35:42.953 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-05-31T21:35:42.953 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T21:35:42.953 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-05-31T21:35:42.954 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-05-31T21:35:42.954 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-05-31T21:35:42.954 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-05-31T21:35:42.954 INFO:tasks.rgw_multisite_tests:HEAD 2023-05-31T21:35:42.954 INFO:tasks.rgw_multisite_tests: 2023-05-31T21:35:42.955 INFO:tasks.rgw_multisite_tests: 2023-05-31T21:35:42.955 INFO:tasks.rgw_multisite_tests:Wed, 31 May 2023 21:12:28 GMT 2023-05-31T21:35:42.955 INFO:tasks.rgw_multisite_tests:/pklxkd-64/ 2023-05-31T21:35:42.955 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-05-31T21:35:42.956 INFO:tasks.rgw_multisite_tests:AWS Q80H11OLNU1S48PPYL26:wRm+Nzs6+vQGxm2DWUEut4l9HJU= 2023-05-31T21:35:42.956 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 21:12:28 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:wRm+Nzs6+vQGxm2DWUEut4l9HJU=', 'Content-Length': '0'} 2023-05-31T21:35:42.956 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '32'), ('X-RGW-Bytes-Used', '96'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000177586ae6be493a7-006477b83c-1630-a1'), ('Content-Length', '0'), ('Date', 'Wed, 31 May 2023 21:12:28 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:35:42.956 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/pklxkd-64/obj111 2023-05-31T21:35:42.957 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/pklxkd-64/obj111 2023-05-31T21:35:42.957 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2023-05-31T21:35:42.957 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /pklxkd-64/obj111 2023-05-31T21:35:42.957 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-05-31T21:35:42.957 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-05-31T21:35:42.958 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T21:35:42.958 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-05-31T21:35:42.958 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-05-31T21:35:42.958 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-05-31T21:35:42.959 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-05-31T21:35:42.959 INFO:tasks.rgw_multisite_tests:PUT 2023-05-31T21:35:42.959 INFO:tasks.rgw_multisite_tests:rL0Y20zC+Fzt72VPzMSk2A== 2023-05-31T21:35:42.959 INFO:tasks.rgw_multisite_tests:application/octet-stream 2023-05-31T21:35:42.959 INFO:tasks.rgw_multisite_tests:Wed, 31 May 2023 21:12:28 GMT 2023-05-31T21:35:42.960 INFO:tasks.rgw_multisite_tests:/pklxkd-64/obj111 2023-05-31T21:35:42.960 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-05-31T21:35:42.960 INFO:tasks.rgw_multisite_tests:AWS Q80H11OLNU1S48PPYL26:PQYRTSWgOHNBCS+sq/d7wkr7tWE= 2023-05-31T21:35:42.960 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Wed, 31 May 2023 21:12:28 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:PQYRTSWgOHNBCS+sq/d7wkr7tWE='} 2023-05-31T21:35:42.961 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000000ab4af260ddb5f9d-006477b83c-1630-a1'), ('Date', 'Wed, 31 May 2023 21:12:28 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:35:42.961 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/pklxkd-64/ 2023-05-31T21:35:42.961 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/pklxkd-64/ 2023-05-31T21:35:42.961 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-05-31T21:35:42.962 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /pklxkd-64/ 2023-05-31T21:35:42.962 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-05-31T21:35:42.962 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-05-31T21:35:42.962 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T21:35:42.963 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-05-31T21:35:42.963 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-05-31T21:35:42.963 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-05-31T21:35:42.963 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-05-31T21:35:42.963 INFO:tasks.rgw_multisite_tests:HEAD 2023-05-31T21:35:42.964 INFO:tasks.rgw_multisite_tests: 2023-05-31T21:35:42.964 INFO:tasks.rgw_multisite_tests: 2023-05-31T21:35:42.964 INFO:tasks.rgw_multisite_tests:Wed, 31 May 2023 21:12:28 GMT 2023-05-31T21:35:42.964 INFO:tasks.rgw_multisite_tests:/pklxkd-64/ 2023-05-31T21:35:42.965 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-05-31T21:35:42.965 INFO:tasks.rgw_multisite_tests:AWS Q80H11OLNU1S48PPYL26:wRm+Nzs6+vQGxm2DWUEut4l9HJU= 2023-05-31T21:35:42.965 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 21:12:28 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:wRm+Nzs6+vQGxm2DWUEut4l9HJU=', 'Content-Length': '0'} 2023-05-31T21:35:42.965 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '33'), ('X-RGW-Bytes-Used', '99'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000c162c2186fe1b9c9-006477b83c-1630-a1'), ('Content-Length', '0'), ('Date', 'Wed, 31 May 2023 21:12:28 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:35:42.966 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/pklxkd-64/obj148 2023-05-31T21:35:42.966 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/pklxkd-64/obj148 2023-05-31T21:35:42.966 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2023-05-31T21:35:42.966 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /pklxkd-64/obj148 2023-05-31T21:35:42.966 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-05-31T21:35:42.967 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-05-31T21:35:42.967 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T21:35:42.967 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-05-31T21:35:42.967 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-05-31T21:35:42.968 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-05-31T21:35:42.968 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-05-31T21:35:42.968 INFO:tasks.rgw_multisite_tests:PUT 2023-05-31T21:35:42.968 INFO:tasks.rgw_multisite_tests:rL0Y20zC+Fzt72VPzMSk2A== 2023-05-31T21:35:42.969 INFO:tasks.rgw_multisite_tests:application/octet-stream 2023-05-31T21:35:42.969 INFO:tasks.rgw_multisite_tests:Wed, 31 May 2023 21:12:28 GMT 2023-05-31T21:35:42.969 INFO:tasks.rgw_multisite_tests:/pklxkd-64/obj148 2023-05-31T21:35:42.969 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-05-31T21:35:42.969 INFO:tasks.rgw_multisite_tests:AWS Q80H11OLNU1S48PPYL26:h5bJfi8JUN74fr/t6Qjb6Vk4sUM= 2023-05-31T21:35:42.970 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Wed, 31 May 2023 21:12:28 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:h5bJfi8JUN74fr/t6Qjb6Vk4sUM='} 2023-05-31T21:35:42.970 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000a5f187110ff17551-006477b83c-1630-a1'), ('Date', 'Wed, 31 May 2023 21:12:28 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:35:42.970 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/pklxkd-64/ 2023-05-31T21:35:42.970 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/pklxkd-64/ 2023-05-31T21:35:42.971 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-05-31T21:35:42.971 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /pklxkd-64/ 2023-05-31T21:35:42.971 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-05-31T21:35:42.971 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-05-31T21:35:42.972 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T21:35:42.972 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-05-31T21:35:42.972 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-05-31T21:35:42.972 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-05-31T21:35:42.972 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-05-31T21:35:42.973 INFO:tasks.rgw_multisite_tests:HEAD 2023-05-31T21:35:42.973 INFO:tasks.rgw_multisite_tests: 2023-05-31T21:35:42.973 INFO:tasks.rgw_multisite_tests: 2023-05-31T21:35:42.973 INFO:tasks.rgw_multisite_tests:Wed, 31 May 2023 21:12:28 GMT 2023-05-31T21:35:42.974 INFO:tasks.rgw_multisite_tests:/pklxkd-64/ 2023-05-31T21:35:42.974 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-05-31T21:35:42.974 INFO:tasks.rgw_multisite_tests:AWS Q80H11OLNU1S48PPYL26:wRm+Nzs6+vQGxm2DWUEut4l9HJU= 2023-05-31T21:35:42.974 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 21:12:28 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:wRm+Nzs6+vQGxm2DWUEut4l9HJU=', 'Content-Length': '0'} 2023-05-31T21:35:42.975 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '34'), ('X-RGW-Bytes-Used', '102'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000d2da6c89b8b166a9-006477b83c-1630-a1'), ('Content-Length', '0'), ('Date', 'Wed, 31 May 2023 21:12:28 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:35:42.975 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/pklxkd-64/obj185 2023-05-31T21:35:42.975 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/pklxkd-64/obj185 2023-05-31T21:35:42.975 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2023-05-31T21:35:42.975 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /pklxkd-64/obj185 2023-05-31T21:35:42.976 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-05-31T21:35:42.976 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-05-31T21:35:42.976 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T21:35:42.976 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-05-31T21:35:42.977 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-05-31T21:35:42.977 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-05-31T21:35:42.977 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-05-31T21:35:42.977 INFO:tasks.rgw_multisite_tests:PUT 2023-05-31T21:35:42.978 INFO:tasks.rgw_multisite_tests:rL0Y20zC+Fzt72VPzMSk2A== 2023-05-31T21:35:42.978 INFO:tasks.rgw_multisite_tests:application/octet-stream 2023-05-31T21:35:42.978 INFO:tasks.rgw_multisite_tests:Wed, 31 May 2023 21:12:28 GMT 2023-05-31T21:35:42.978 INFO:tasks.rgw_multisite_tests:/pklxkd-64/obj185 2023-05-31T21:35:42.978 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-05-31T21:35:42.979 INFO:tasks.rgw_multisite_tests:AWS Q80H11OLNU1S48PPYL26:Y2SdKSit78jNOubvBZare6NJX2I= 2023-05-31T21:35:42.979 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Wed, 31 May 2023 21:12:28 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:Y2SdKSit78jNOubvBZare6NJX2I='} 2023-05-31T21:35:42.979 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000f7b93d23e5d8e8a4-006477b83c-1630-a1'), ('Date', 'Wed, 31 May 2023 21:12:28 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:35:42.979 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['data', 'sync', 'init', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-05-31T21:35:42.980 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'init', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-05-31T21:35:42.980 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync init --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 2023-05-31T21:35:42.980 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: ignoring --setuser ceph since I am not root 2023-05-31T21:35:42.980 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-05-31T21:35:42.981 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: 2023-05-31T21:12:28.787+0000 7f823c58b780 5 RGW-SYNC:data: waiting on data sync status lease 2023-05-31T21:35:42.981 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: 2023-05-31T21:12:28.788+0000 7f823c58b780 5 RGW-SYNC:data: acquired data sync status lease 2023-05-31T21:35:42.981 INFO:tasks.rgw_multisite_tests:tasks.rgw.c1.client.1: INFO: Restarting daemon 2023-05-31T21:35:42.981 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099: DEBUG: > sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term radosgw --rgw-frontends 'beast port=8001' -n client.1 --cluster c1 -k /etc/ceph/c1.client.1.keyring --log-file /var/log/ceph/rgw.c1.client.1.log --rgw_ops_log_socket_path /home/ubuntu/cephtest/rgw.opslog.c1.client.1.sock --foreground --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm | sudo tee /var/log/ceph/rgw.c1.client.1.stdout 2>&1 2023-05-31T21:35:42.981 INFO:tasks.rgw_multisite_tests:tasks.rgw.c1.client.1: INFO: Started 2023-05-31T21:35:42.982 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099: DEBUG: > curl http://smithi099.front.sepia.ceph.com:8001 2023-05-31T21:35:42.982 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: % Total % Received % Xferd Average Speed Time Time Time Current 2023-05-31T21:35:42.982 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: Dload Upload Total Spent Left Speed 2023-05-31T21:35:42.982 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run: DEBUG: got remote process result: 7 2023-05-31T21:35:42.983 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0curl: (7) Failed to connect to smithi099.front.sepia.ceph.com port 8001: Connection refused 2023-05-31T21:35:42.983 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099: DEBUG: > curl http://smithi099.front.sepia.ceph.com:8001 2023-05-31T21:35:42.983 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: % Total % Received % Xferd Average Speed Time Time Time Current 2023-05-31T21:35:42.983 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: Dload Upload Total Spent Left Speed 2023-05-31T21:35:42.984 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 100 214 0 214 0 0 208k 0 --:--:-- --:--:-- --:--:-- 208k 2023-05-31T21:35:42.984 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: anonymous 2023-05-31T21:35:42.984 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: bucket checkpoint: source=a1 target=a2 bucket=pklxkd-64 2023-05-31T21:35:42.984 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'pklxkd-64', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-05-31T21:35:42.984 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'pklxkd-64', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-05-31T21:35:42.985 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket pklxkd-64 --source-zone a1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2023-05-31T21:35:42.985 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: ignoring --setuser ceph since I am not root 2023-05-31T21:35:42.985 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-05-31T21:35:42.985 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: 2023-05-31T21:12:30.020+0000 7f951157b780 1 waiting to reach latest gen 6, on 0.. 2023-05-31T21:35:42.986 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c1.rgw.client.1 has been restored 2023-05-31T21:35:42.986 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: 2023-05-31T21:12:35.020+0000 7f951157b780 1 waiting to reach latest gen 6, on 0.. 2023-05-31T21:35:42.986 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: 2023-05-31T21:12:40.021+0000 7f951157b780 1 waiting to reach latest gen 6, on 0.. 2023-05-31T21:35:42.986 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: 2023-05-31T21:12:45.022+0000 7f951157b780 1 waiting to reach latest gen 6, on 0.. 2023-05-31T21:35:42.987 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: 2023-05-31T21:12:50.023+0000 7f951157b780 1 waiting to reach latest gen 6, on 0.. 2023-05-31T21:35:42.987 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: 2023-05-31T21:12:55.024+0000 7f951157b780 1 waiting to reach latest gen 6, on 0.. 2023-05-31T21:35:42.987 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: 2023-05-31T21:13:00.024+0000 7f951157b780 1 waiting to reach latest gen 6, on 0.. 2023-05-31T21:35:42.987 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: 2023-05-31T21:13:05.024+0000 7f951157b780 1 waiting to reach latest gen 6, on 0.. 2023-05-31T21:35:42.988 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: 2023-05-31T21:13:10.024+0000 7f951157b780 1 waiting to reach latest gen 6, on 0.. 2023-05-31T21:35:42.988 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: 2023-05-31T21:13:15.025+0000 7f951157b780 1 waiting to reach latest gen 6, on 0.. 2023-05-31T21:35:42.988 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: 2023-05-31T21:13:20.026+0000 7f951157b780 1 waiting to reach latest gen 6, on 0.. 2023-05-31T21:35:42.988 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: 2023-05-31T21:13:25.027+0000 7f951157b780 1 waiting to reach latest gen 6, on 0.. 2023-05-31T21:35:42.989 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: 2023-05-31T21:13:30.028+0000 7f951157b780 1 waiting to reach latest gen 6, on 0.. 2023-05-31T21:35:42.989 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: 2023-05-31T21:13:35.029+0000 7f951157b780 1 waiting to reach latest gen 6, on 0.. 2023-05-31T21:35:42.989 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: 2023-05-31T21:13:40.030+0000 7f951157b780 1 waiting to reach latest gen 6, on 0.. 2023-05-31T21:35:42.989 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: 2023-05-31T21:13:45.030+0000 7f951157b780 1 waiting to reach latest gen 6, on 0.. 2023-05-31T21:35:42.990 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: 2023-05-31T21:13:50.030+0000 7f951157b780 1 waiting to reach latest gen 6, on 0.. 2023-05-31T21:35:42.990 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: 2023-05-31T21:13:55.030+0000 7f951157b780 1 waiting to reach latest gen 6, on 0.. 2023-05-31T21:35:42.990 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: 2023-05-31T21:14:00.031+0000 7f951157b780 1 waiting to reach latest gen 6, on 0.. 2023-05-31T21:35:42.990 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: 2023-05-31T21:14:05.032+0000 7f951157b780 1 waiting to reach latest gen 6, on 2.. 2023-05-31T21:35:42.990 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: 2023-05-31T21:14:10.033+0000 7f951157b780 1 waiting to reach latest gen 6, on 2.. 2023-05-31T21:35:42.991 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: 2023-05-31T21:14:15.034+0000 7f951157b780 1 waiting to reach latest gen 6, on 2.. 2023-05-31T21:35:42.991 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: 2023-05-31T21:14:20.035+0000 7f951157b780 1 waiting to reach latest gen 6, on 2.. 2023-05-31T21:35:42.991 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: 2023-05-31T21:14:25.035+0000 7f951157b780 1 waiting to reach latest gen 6, on 2.. 2023-05-31T21:35:42.991 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: 2023-05-31T21:14:30.035+0000 7f951157b780 1 waiting to reach latest gen 6, on 2.. 2023-05-31T21:35:42.992 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: 2023-05-31T21:14:35.036+0000 7f951157b780 1 waiting to reach latest gen 6, on 2.. 2023-05-31T21:35:42.992 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: 2023-05-31T21:14:40.036+0000 7f951157b780 1 waiting to reach latest gen 6, on 2.. 2023-05-31T21:35:42.992 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: 2023-05-31T21:14:45.037+0000 7f951157b780 1 waiting to reach latest gen 6, on 2.. 2023-05-31T21:35:42.992 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: 2023-05-31T21:14:50.039+0000 7f951157b780 1 waiting to reach latest gen 6, on 2.. 2023-05-31T21:35:42.993 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: 2023-05-31T21:14:55.039+0000 7f951157b780 1 waiting to reach latest gen 6, on 2.. 2023-05-31T21:35:42.993 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: 2023-05-31T21:15:00.040+0000 7f951157b780 1 waiting to reach latest gen 6, on 2.. 2023-05-31T21:35:42.993 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: 2023-05-31T21:15:05.040+0000 7f951157b780 1 waiting to reach latest gen 6, on 2.. 2023-05-31T21:35:42.993 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: 2023-05-31T21:15:10.040+0000 7f951157b780 1 waiting to reach latest gen 6, on 2.. 2023-05-31T21:35:42.993 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: 2023-05-31T21:15:15.040+0000 7f951157b780 1 waiting to reach latest gen 6, on 2.. 2023-05-31T21:35:42.994 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: 2023-05-31T21:15:20.041+0000 7f951157b780 1 waiting to reach latest gen 6, on 2.. 2023-05-31T21:35:42.994 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: 2023-05-31T21:15:25.042+0000 7f951157b780 1 waiting to reach latest gen 6, on 2.. 2023-05-31T21:35:42.994 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: 2023-05-31T21:15:30.043+0000 7f951157b780 1 waiting to reach latest gen 6, on 2.. 2023-05-31T21:35:42.994 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: 2023-05-31T21:15:35.044+0000 7f951157b780 1 waiting to reach latest gen 6, on 2.. 2023-05-31T21:35:42.995 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: 2023-05-31T21:15:40.045+0000 7f951157b780 1 waiting to reach latest gen 6, on 2.. 2023-05-31T21:35:42.995 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: 2023-05-31T21:15:45.046+0000 7f951157b780 1 waiting to reach latest gen 6, on 2.. 2023-05-31T21:35:42.995 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: 2023-05-31T21:15:50.046+0000 7f951157b780 1 waiting to reach latest gen 6, on 2.. 2023-05-31T21:35:42.995 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: 2023-05-31T21:15:55.046+0000 7f951157b780 1 waiting to reach latest gen 6, on 2.. 2023-05-31T21:35:42.995 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: 2023-05-31T21:16:00.047+0000 7f951157b780 1 waiting to reach latest gen 6, on 2.. 2023-05-31T21:35:42.996 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: 2023-05-31T21:16:05.048+0000 7f951157b780 1 waiting to reach latest gen 6, on 2.. 2023-05-31T21:35:42.996 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: 2023-05-31T21:16:10.049+0000 7f951157b780 1 waiting to reach latest gen 6, on 2.. 2023-05-31T21:35:42.996 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: 2023-05-31T21:16:15.050+0000 7f951157b780 1 waiting to reach latest gen 6, on 2.. 2023-05-31T21:35:42.996 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: 2023-05-31T21:16:20.051+0000 7f951157b780 1 waiting to reach latest gen 6, on 2.. 2023-05-31T21:35:42.997 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: 2023-05-31T21:16:25.052+0000 7f951157b780 1 waiting to reach latest gen 6, on 2.. 2023-05-31T21:35:42.997 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: 2023-05-31T21:16:30.052+0000 7f951157b780 1 waiting to reach latest gen 6, on 2.. 2023-05-31T21:35:42.997 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: 2023-05-31T21:16:35.052+0000 7f951157b780 1 waiting to reach latest gen 6, on 2.. 2023-05-31T21:35:42.997 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: 2023-05-31T21:16:40.053+0000 7f951157b780 1 waiting to reach latest gen 6, on 2.. 2023-05-31T21:35:42.998 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: 2023-05-31T21:16:45.054+0000 7f951157b780 1 waiting to reach latest gen 6, on 2.. 2023-05-31T21:35:42.998 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: 2023-05-31T21:16:50.054+0000 7f951157b780 1 waiting to reach latest gen 6, on 2.. 2023-05-31T21:35:42.998 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: 2023-05-31T21:16:55.055+0000 7f951157b780 1 waiting to reach latest gen 6, on 2.. 2023-05-31T21:35:42.998 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: 2023-05-31T21:17:00.056+0000 7f951157b780 1 waiting to reach latest gen 6, on 2.. 2023-05-31T21:35:42.999 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: 2023-05-31T21:17:05.057+0000 7f951157b780 1 waiting to reach latest gen 6, on 2.. 2023-05-31T21:35:42.999 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: 2023-05-31T21:17:10.057+0000 7f951157b780 1 waiting to reach latest gen 6, on 2.. 2023-05-31T21:35:42.999 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: 2023-05-31T21:17:15.058+0000 7f951157b780 1 waiting to reach latest gen 6, on 2.. 2023-05-31T21:35:42.999 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: 2023-05-31T21:17:20.058+0000 7f951157b780 1 waiting to reach latest gen 6, on 2.. 2023-05-31T21:35:42.999 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: 2023-05-31T21:17:25.059+0000 7f951157b780 -1 bucket checkpoint timed out waiting to reach latest generation 6 2023-05-31T21:35:43.000 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: 2023-05-31T21:17:25.059+0000 7f951157b780 0 bucket sync checkpoint failed: (110) Connection timed out 2023-05-31T21:35:43.000 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: 2023-05-31T21:17:25.059+0000 7f951157b780 -1 bucket sync checkpoint failed: (110) Connection timed out 2023-05-31T21:35:43.000 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run: DEBUG: got remote process result: 110 2023-05-31T21:35:43.000 INFO:tasks.rgw_multisite_tests:--------------------- >> end captured logging << --------------------- 2023-05-31T21:35:43.001 INFO:tasks.rgw_multisite_tests: 2023-05-31T21:35:43.001 INFO:tasks.rgw_multisite_tests:====================================================================== 2023-05-31T21:35:43.001 INFO:tasks.rgw_multisite_tests:FAIL: test_sync_single_bucket: 2023-05-31T21:35:43.002 INFO:tasks.rgw_multisite_tests:---------------------------------------------------------------------- 2023-05-31T21:35:43.003 INFO:tasks.rgw_multisite_tests:Traceback (most recent call last): 2023-05-31T21:35:43.003 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_teuthology_fa97ddd17d12e6d74b49196948f66e54fa009130/virtualenv/lib/python3.8/site-packages/nose/case.py", line 198, in runTest 2023-05-31T21:35:43.003 INFO:tasks.rgw_multisite_tests: self.test(*self.arg) 2023-05-31T21:35:43.003 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/github.com_ceph_ceph-c_2870bf456d3115490c153abce9d47ac5c043b65e/qa/../src/test/rgw/rgw_multi/tests.py", line 2432, in test_sync_single_bucket 2023-05-31T21:35:43.004 INFO:tasks.rgw_multisite_tests: check_objects_exist(bucket, objnames, content) 2023-05-31T21:35:43.004 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/github.com_ceph_ceph-c_2870bf456d3115490c153abce9d47ac5c043b65e/qa/../src/test/rgw/rgw_multi/tests.py", line 2047, in check_objects_exist 2023-05-31T21:35:43.004 INFO:tasks.rgw_multisite_tests: check_object_exists(bucket, objname, content) 2023-05-31T21:35:43.004 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/github.com_ceph_ceph-c_2870bf456d3115490c153abce9d47ac5c043b65e/qa/../src/test/rgw/rgw_multi/tests.py", line 2041, in check_object_exists 2023-05-31T21:35:43.005 INFO:tasks.rgw_multisite_tests: assert_not_equal(k, None) 2023-05-31T21:35:43.005 INFO:tasks.rgw_multisite_tests:AssertionError: None == None 2023-05-31T21:35:43.005 INFO:tasks.rgw_multisite_tests:-------------------- >> begin captured logging << -------------------- 2023-05-31T21:35:43.005 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-31T21:35:43.006 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-31T21:35:43.006 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-05-31T21:35:43.006 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: ignoring --setuser ceph since I am not root 2023-05-31T21:35:43.006 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-05-31T21:35:43.007 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: No --period given, using current period=2e6123d2-d695-4350-9a01-61a455bc782b 2023-05-31T21:35:43.007 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: [ 2023-05-31T21:35:43.007 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2023-05-31T21:35:43.007 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "marker": "1_1685566501.757183_260.1", 2023-05-31T21:35:43.007 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "last_update": "2023-05-31T20:55:01.757183Z" 2023-05-31T21:35:43.008 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2023-05-31T21:35:43.008 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2023-05-31T21:35:43.008 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "marker": "1_1685566501.735717_251.1", 2023-05-31T21:35:43.008 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "last_update": "2023-05-31T20:55:01.735717Z" 2023-05-31T21:35:43.009 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2023-05-31T21:35:43.009 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2023-05-31T21:35:43.009 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "marker": "1_1685562970.274572_93.1", 2023-05-31T21:35:43.009 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "last_update": "2023-05-31T19:56:10.274572Z" 2023-05-31T21:35:43.010 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2023-05-31T21:35:43.010 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2023-05-31T21:35:43.010 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "marker": "1_1685567943.999406_931.1", 2023-05-31T21:35:43.010 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "last_update": "2023-05-31T21:19:03.999406Z" 2023-05-31T21:35:43.010 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2023-05-31T21:35:43.011 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: ] 2023-05-31T21:35:43.011 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: master meta markers={0: '1_1685566501.757183_260.1', 1: '1_1685566501.735717_251.1', 2: '1_1685562970.274572_93.1', 3: '1_1685567943.999406_931.1'} 2023-05-31T21:35:43.011 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: INFO: starting meta checkpoint for zone=a2 2023-05-31T21:35:43.011 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-31T21:35:43.012 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-31T21:35:43.012 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-05-31T21:35:43.012 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: ignoring --setuser ceph since I am not root 2023-05-31T21:35:43.012 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-05-31T21:35:43.013 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: 2023-05-31T21:21:28.949+0000 7efd59a07780 20 RGW-SYNC:meta: read sync status 2023-05-31T21:35:43.013 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: 2023-05-31T21:21:28.950+0000 7efd59a07780 20 RGW-SYNC:meta: read sync status 2023-05-31T21:35:43.013 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2023-05-31T21:35:43.013 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "sync_status": { 2023-05-31T21:35:43.014 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "info": { 2023-05-31T21:35:43.014 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "status": "sync", 2023-05-31T21:35:43.014 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "num_shards": 4, 2023-05-31T21:35:43.014 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "period": "2e6123d2-d695-4350-9a01-61a455bc782b", 2023-05-31T21:35:43.014 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "realm_epoch": 2 2023-05-31T21:35:43.015 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2023-05-31T21:35:43.015 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "markers": [ 2023-05-31T21:35:43.015 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2023-05-31T21:35:43.015 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "key": 0, 2023-05-31T21:35:43.016 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "val": { 2023-05-31T21:35:43.016 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "state": 1, 2023-05-31T21:35:43.016 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "marker": "1_1685566501.757183_260.1", 2023-05-31T21:35:43.016 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "next_step_marker": "", 2023-05-31T21:35:43.017 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "total_entries": 1, 2023-05-31T21:35:43.017 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "pos": 0, 2023-05-31T21:35:43.017 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "timestamp": "2023-05-31T20:55:01.757183Z", 2023-05-31T21:35:43.017 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "realm_epoch": 2 2023-05-31T21:35:43.018 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2023-05-31T21:35:43.018 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2023-05-31T21:35:43.018 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2023-05-31T21:35:43.018 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "key": 1, 2023-05-31T21:35:43.019 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "val": { 2023-05-31T21:35:43.019 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "state": 1, 2023-05-31T21:35:43.019 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "marker": "1_1685566501.735717_251.1", 2023-05-31T21:35:43.019 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "next_step_marker": "", 2023-05-31T21:35:43.020 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "total_entries": 0, 2023-05-31T21:35:43.020 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "pos": 0, 2023-05-31T21:35:43.020 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "timestamp": "2023-05-31T20:55:01.735717Z", 2023-05-31T21:35:43.020 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "realm_epoch": 2 2023-05-31T21:35:43.021 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2023-05-31T21:35:43.021 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2023-05-31T21:35:43.021 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2023-05-31T21:35:43.021 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "key": 2, 2023-05-31T21:35:43.022 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "val": { 2023-05-31T21:35:43.022 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "state": 1, 2023-05-31T21:35:43.022 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "marker": "1_1685562970.274572_93.1", 2023-05-31T21:35:43.022 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "next_step_marker": "", 2023-05-31T21:35:43.023 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "total_entries": 0, 2023-05-31T21:35:43.023 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "pos": 0, 2023-05-31T21:35:43.023 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "timestamp": "2023-05-31T19:56:10.274572Z", 2023-05-31T21:35:43.023 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "realm_epoch": 2 2023-05-31T21:35:43.024 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2023-05-31T21:35:43.024 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2023-05-31T21:35:43.024 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2023-05-31T21:35:43.024 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "key": 3, 2023-05-31T21:35:43.025 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "val": { 2023-05-31T21:35:43.025 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "state": 1, 2023-05-31T21:35:43.025 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "marker": "1_1685567943.999406_931.1", 2023-05-31T21:35:43.025 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "next_step_marker": "", 2023-05-31T21:35:43.025 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "total_entries": 0, 2023-05-31T21:35:43.026 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "pos": 0, 2023-05-31T21:35:43.026 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "timestamp": "2023-05-31T21:19:03.999406Z", 2023-05-31T21:35:43.026 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "realm_epoch": 2 2023-05-31T21:35:43.026 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2023-05-31T21:35:43.027 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2023-05-31T21:35:43.027 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: ] 2023-05-31T21:35:43.027 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2023-05-31T21:35:43.027 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "full_sync": { 2023-05-31T21:35:43.028 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "total": 1, 2023-05-31T21:35:43.028 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "complete": 1 2023-05-31T21:35:43.028 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2023-05-31T21:35:43.028 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "current_time": "2023-05-31T21:21:28Z" 2023-05-31T21:35:43.029 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2023-05-31T21:35:43.029 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2023-05-31T21:35:43.029 INFO:tasks.rgw_multisite_tests: "sync_status": { 2023-05-31T21:35:43.029 INFO:tasks.rgw_multisite_tests: "info": { 2023-05-31T21:35:43.029 INFO:tasks.rgw_multisite_tests: "status": "sync", 2023-05-31T21:35:43.030 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2023-05-31T21:35:43.030 INFO:tasks.rgw_multisite_tests: "period": "2e6123d2-d695-4350-9a01-61a455bc782b", 2023-05-31T21:35:43.030 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-05-31T21:35:43.030 INFO:tasks.rgw_multisite_tests: }, 2023-05-31T21:35:43.031 INFO:tasks.rgw_multisite_tests: "markers": [ 2023-05-31T21:35:43.031 INFO:tasks.rgw_multisite_tests: { 2023-05-31T21:35:43.031 INFO:tasks.rgw_multisite_tests: "key": 0, 2023-05-31T21:35:43.031 INFO:tasks.rgw_multisite_tests: "val": { 2023-05-31T21:35:43.031 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-05-31T21:35:43.032 INFO:tasks.rgw_multisite_tests: "marker": "1_1685566501.757183_260.1", 2023-05-31T21:35:43.032 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-05-31T21:35:43.032 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2023-05-31T21:35:43.032 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-05-31T21:35:43.033 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-05-31T20:55:01.757183Z", 2023-05-31T21:35:43.033 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-05-31T21:35:43.033 INFO:tasks.rgw_multisite_tests: } 2023-05-31T21:35:43.033 INFO:tasks.rgw_multisite_tests: }, 2023-05-31T21:35:43.034 INFO:tasks.rgw_multisite_tests: { 2023-05-31T21:35:43.034 INFO:tasks.rgw_multisite_tests: "key": 1, 2023-05-31T21:35:43.034 INFO:tasks.rgw_multisite_tests: "val": { 2023-05-31T21:35:43.034 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-05-31T21:35:43.035 INFO:tasks.rgw_multisite_tests: "marker": "1_1685566501.735717_251.1", 2023-05-31T21:35:43.035 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-05-31T21:35:43.035 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-05-31T21:35:43.035 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-05-31T21:35:43.035 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-05-31T20:55:01.735717Z", 2023-05-31T21:35:43.036 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-05-31T21:35:43.036 INFO:tasks.rgw_multisite_tests: } 2023-05-31T21:35:43.036 INFO:tasks.rgw_multisite_tests: }, 2023-05-31T21:35:43.036 INFO:tasks.rgw_multisite_tests: { 2023-05-31T21:35:43.037 INFO:tasks.rgw_multisite_tests: "key": 2, 2023-05-31T21:35:43.037 INFO:tasks.rgw_multisite_tests: "val": { 2023-05-31T21:35:43.037 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-05-31T21:35:43.037 INFO:tasks.rgw_multisite_tests: "marker": "1_1685562970.274572_93.1", 2023-05-31T21:35:43.038 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-05-31T21:35:43.038 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-05-31T21:35:43.038 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-05-31T21:35:43.038 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-05-31T19:56:10.274572Z", 2023-05-31T21:35:43.038 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-05-31T21:35:43.039 INFO:tasks.rgw_multisite_tests: } 2023-05-31T21:35:43.039 INFO:tasks.rgw_multisite_tests: }, 2023-05-31T21:35:43.039 INFO:tasks.rgw_multisite_tests: { 2023-05-31T21:35:43.039 INFO:tasks.rgw_multisite_tests: "key": 3, 2023-05-31T21:35:43.040 INFO:tasks.rgw_multisite_tests: "val": { 2023-05-31T21:35:43.040 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-05-31T21:35:43.040 INFO:tasks.rgw_multisite_tests: "marker": "1_1685567943.999406_931.1", 2023-05-31T21:35:43.040 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-05-31T21:35:43.041 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-05-31T21:35:43.041 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-05-31T21:35:43.041 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-05-31T21:19:03.999406Z", 2023-05-31T21:35:43.041 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-05-31T21:35:43.041 INFO:tasks.rgw_multisite_tests: } 2023-05-31T21:35:43.042 INFO:tasks.rgw_multisite_tests: } 2023-05-31T21:35:43.042 INFO:tasks.rgw_multisite_tests: ] 2023-05-31T21:35:43.042 INFO:tasks.rgw_multisite_tests: }, 2023-05-31T21:35:43.042 INFO:tasks.rgw_multisite_tests: "full_sync": { 2023-05-31T21:35:43.043 INFO:tasks.rgw_multisite_tests: "total": 1, 2023-05-31T21:35:43.043 INFO:tasks.rgw_multisite_tests: "complete": 1 2023-05-31T21:35:43.043 INFO:tasks.rgw_multisite_tests: }, 2023-05-31T21:35:43.043 INFO:tasks.rgw_multisite_tests: "current_time": "2023-05-31T21:21:28Z" 2023-05-31T21:35:43.044 INFO:tasks.rgw_multisite_tests:} 2023-05-31T21:35:43.044 INFO:tasks.rgw_multisite_tests: 2023-05-31T21:35:43.044 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1685566501.757183_260.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-05-31T20:55:01.757183Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1685566501.735717_251.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-05-31T20:55:01.735717Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1685562970.274572_93.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-05-31T19:56:10.274572Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1685567943.999406_931.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-05-31T21:19:03.999406Z', 'realm_epoch': 2}}] 2023-05-31T21:35:43.044 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1685566501.757183_260.1', 1: '1_1685566501.735717_251.1', 2: '1_1685562970.274572_93.1', 3: '1_1685567943.999406_931.1'} 2023-05-31T21:35:43.045 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1685566501.757183_260.1', 1: '1_1685566501.735717_251.1', 2: '1_1685562970.274572_93.1', 3: '1_1685567943.999406_931.1'} 2023-05-31T21:35:43.045 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: INFO: finish meta checkpoint for zone=a2 2023-05-31T21:35:43.045 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['sync', 'policy', 'get', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-05-31T21:35:43.045 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'policy', 'get', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-05-31T21:35:43.045 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync policy get --cluster c1 --debug-rgw 0 --debug-ms 0 2023-05-31T21:35:43.046 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: ignoring --setuser ceph since I am not root 2023-05-31T21:35:43.046 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-05-31T21:35:43.046 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: {"groups":[]} 2023-05-31T21:35:43.046 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['sync', 'policy', 'get', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-05-31T21:35:43.047 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'policy', 'get', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-05-31T21:35:43.047 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync policy get --cluster c1 --debug-rgw 0 --debug-ms 0 2023-05-31T21:35:43.047 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: ignoring --setuser ceph since I am not root 2023-05-31T21:35:43.047 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-05-31T21:35:43.048 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: {"groups":[]} 2023-05-31T21:35:43.048 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: INFO: create bucket zone=a1 name=pklxkd-68 2023-05-31T21:35:43.048 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/pklxkd-68/ 2023-05-31T21:35:43.048 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/pklxkd-68/ 2023-05-31T21:35:43.048 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2023-05-31T21:35:43.049 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /pklxkd-68/ 2023-05-31T21:35:43.049 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-05-31T21:35:43.049 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-05-31T21:35:43.049 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T21:35:43.050 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-05-31T21:35:43.050 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-05-31T21:35:43.050 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-05-31T21:35:43.050 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-05-31T21:35:43.051 INFO:tasks.rgw_multisite_tests:PUT 2023-05-31T21:35:43.051 INFO:tasks.rgw_multisite_tests: 2023-05-31T21:35:43.051 INFO:tasks.rgw_multisite_tests: 2023-05-31T21:35:43.051 INFO:tasks.rgw_multisite_tests:Wed, 31 May 2023 21:21:29 GMT 2023-05-31T21:35:43.051 INFO:tasks.rgw_multisite_tests:/pklxkd-68/ 2023-05-31T21:35:43.052 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-05-31T21:35:43.052 INFO:tasks.rgw_multisite_tests:AWS Q80H11OLNU1S48PPYL26:vzOPE0/dSQUrwe/dBHBbvXm2k1c= 2023-05-31T21:35:43.052 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 21:21:29 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:vzOPE0/dSQUrwe/dBHBbvXm2k1c=', 'Content-Length': '0'} 2023-05-31T21:35:43.052 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('x-amz-request-id', 'tx00000b84acbfc8c03f6b6-006477ba59-22ab-a1'), ('Content-Length', '0'), ('Date', 'Wed, 31 May 2023 21:21:29 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:35:43.053 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: INFO: create bucket zone=a1 name=pklxkd-69 2023-05-31T21:35:43.053 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/pklxkd-69/ 2023-05-31T21:35:43.053 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/pklxkd-69/ 2023-05-31T21:35:43.053 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2023-05-31T21:35:43.054 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /pklxkd-69/ 2023-05-31T21:35:43.054 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-05-31T21:35:43.054 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-05-31T21:35:43.054 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T21:35:43.055 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-05-31T21:35:43.055 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-05-31T21:35:43.055 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-05-31T21:35:43.055 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-05-31T21:35:43.055 INFO:tasks.rgw_multisite_tests:PUT 2023-05-31T21:35:43.056 INFO:tasks.rgw_multisite_tests: 2023-05-31T21:35:43.056 INFO:tasks.rgw_multisite_tests: 2023-05-31T21:35:43.056 INFO:tasks.rgw_multisite_tests:Wed, 31 May 2023 21:21:29 GMT 2023-05-31T21:35:43.056 INFO:tasks.rgw_multisite_tests:/pklxkd-69/ 2023-05-31T21:35:43.057 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-05-31T21:35:43.057 INFO:tasks.rgw_multisite_tests:AWS Q80H11OLNU1S48PPYL26:GfXB6v/3aRB1HwWO9DeWd+HiK/o= 2023-05-31T21:35:43.057 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 21:21:29 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:GfXB6v/3aRB1HwWO9DeWd+HiK/o=', 'Content-Length': '0'} 2023-05-31T21:35:43.057 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('x-amz-request-id', 'tx0000005a68c715c02b31b-006477ba59-22ab-a1'), ('Content-Length', '0'), ('Date', 'Wed, 31 May 2023 21:21:29 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:35:43.058 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-31T21:35:43.058 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-31T21:35:43.058 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-05-31T21:35:43.058 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: ignoring --setuser ceph since I am not root 2023-05-31T21:35:43.059 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-05-31T21:35:43.059 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: No --period given, using current period=2e6123d2-d695-4350-9a01-61a455bc782b 2023-05-31T21:35:43.059 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: [ 2023-05-31T21:35:43.059 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2023-05-31T21:35:43.059 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "marker": "1_1685566501.757183_260.1", 2023-05-31T21:35:43.060 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "last_update": "2023-05-31T20:55:01.757183Z" 2023-05-31T21:35:43.060 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2023-05-31T21:35:43.060 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2023-05-31T21:35:43.060 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "marker": "1_1685566501.735717_251.1", 2023-05-31T21:35:43.061 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "last_update": "2023-05-31T20:55:01.735717Z" 2023-05-31T21:35:43.061 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2023-05-31T21:35:43.061 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2023-05-31T21:35:43.062 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "marker": "1_1685562970.274572_93.1", 2023-05-31T21:35:43.062 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "last_update": "2023-05-31T19:56:10.274572Z" 2023-05-31T21:35:43.062 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2023-05-31T21:35:43.062 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2023-05-31T21:35:43.063 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "marker": "1_1685568089.268814_960.1", 2023-05-31T21:35:43.063 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "last_update": "2023-05-31T21:21:29.268814Z" 2023-05-31T21:35:43.063 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2023-05-31T21:35:43.063 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: ] 2023-05-31T21:35:43.064 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: master meta markers={0: '1_1685566501.757183_260.1', 1: '1_1685566501.735717_251.1', 2: '1_1685562970.274572_93.1', 3: '1_1685568089.268814_960.1'} 2023-05-31T21:35:43.064 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: INFO: starting meta checkpoint for zone=a2 2023-05-31T21:35:43.064 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-31T21:35:43.064 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-31T21:35:43.065 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-05-31T21:35:43.065 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: ignoring --setuser ceph since I am not root 2023-05-31T21:35:43.065 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-05-31T21:35:43.065 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: 2023-05-31T21:21:29.504+0000 7ff1eb11f780 20 RGW-SYNC:meta: read sync status 2023-05-31T21:35:43.066 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: 2023-05-31T21:21:29.505+0000 7ff1eb11f780 20 RGW-SYNC:meta: read sync status 2023-05-31T21:35:43.066 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2023-05-31T21:35:43.066 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "sync_status": { 2023-05-31T21:35:43.066 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "info": { 2023-05-31T21:35:43.066 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "status": "sync", 2023-05-31T21:35:43.067 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "num_shards": 4, 2023-05-31T21:35:43.067 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "period": "2e6123d2-d695-4350-9a01-61a455bc782b", 2023-05-31T21:35:43.067 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "realm_epoch": 2 2023-05-31T21:35:43.067 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2023-05-31T21:35:43.068 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "markers": [ 2023-05-31T21:35:43.068 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2023-05-31T21:35:43.068 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "key": 0, 2023-05-31T21:35:43.068 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "val": { 2023-05-31T21:35:43.069 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "state": 1, 2023-05-31T21:35:43.069 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "marker": "1_1685566501.757183_260.1", 2023-05-31T21:35:43.069 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "next_step_marker": "", 2023-05-31T21:35:43.069 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "total_entries": 1, 2023-05-31T21:35:43.069 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "pos": 0, 2023-05-31T21:35:43.070 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "timestamp": "2023-05-31T20:55:01.757183Z", 2023-05-31T21:35:43.070 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "realm_epoch": 2 2023-05-31T21:35:43.070 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2023-05-31T21:35:43.070 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2023-05-31T21:35:43.071 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2023-05-31T21:35:43.071 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "key": 1, 2023-05-31T21:35:43.071 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "val": { 2023-05-31T21:35:43.071 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "state": 1, 2023-05-31T21:35:43.072 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "marker": "1_1685566501.735717_251.1", 2023-05-31T21:35:43.072 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "next_step_marker": "", 2023-05-31T21:35:43.072 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "total_entries": 0, 2023-05-31T21:35:43.072 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "pos": 0, 2023-05-31T21:35:43.073 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "timestamp": "2023-05-31T20:55:01.735717Z", 2023-05-31T21:35:43.073 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "realm_epoch": 2 2023-05-31T21:35:43.073 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2023-05-31T21:35:43.073 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2023-05-31T21:35:43.073 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2023-05-31T21:35:43.074 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "key": 2, 2023-05-31T21:35:43.074 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "val": { 2023-05-31T21:35:43.074 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "state": 1, 2023-05-31T21:35:43.074 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "marker": "1_1685562970.274572_93.1", 2023-05-31T21:35:43.075 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "next_step_marker": "", 2023-05-31T21:35:43.075 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "total_entries": 0, 2023-05-31T21:35:43.075 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "pos": 0, 2023-05-31T21:35:43.075 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "timestamp": "2023-05-31T19:56:10.274572Z", 2023-05-31T21:35:43.075 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "realm_epoch": 2 2023-05-31T21:35:43.076 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2023-05-31T21:35:43.076 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2023-05-31T21:35:43.076 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2023-05-31T21:35:43.076 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "key": 3, 2023-05-31T21:35:43.077 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "val": { 2023-05-31T21:35:43.077 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "state": 1, 2023-05-31T21:35:43.077 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "marker": "1_1685567943.999406_931.1", 2023-05-31T21:35:43.077 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "next_step_marker": "", 2023-05-31T21:35:43.078 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "total_entries": 0, 2023-05-31T21:35:43.078 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "pos": 0, 2023-05-31T21:35:43.078 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "timestamp": "2023-05-31T21:19:03.999406Z", 2023-05-31T21:35:43.078 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "realm_epoch": 2 2023-05-31T21:35:43.079 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2023-05-31T21:35:43.079 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2023-05-31T21:35:43.079 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: ] 2023-05-31T21:35:43.079 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2023-05-31T21:35:43.079 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "full_sync": { 2023-05-31T21:35:43.080 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "total": 1, 2023-05-31T21:35:43.080 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "complete": 1 2023-05-31T21:35:43.080 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2023-05-31T21:35:43.080 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "current_time": "2023-05-31T21:21:29Z" 2023-05-31T21:35:43.081 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2023-05-31T21:35:43.081 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2023-05-31T21:35:43.081 INFO:tasks.rgw_multisite_tests: "sync_status": { 2023-05-31T21:35:43.081 INFO:tasks.rgw_multisite_tests: "info": { 2023-05-31T21:35:43.082 INFO:tasks.rgw_multisite_tests: "status": "sync", 2023-05-31T21:35:43.082 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2023-05-31T21:35:43.082 INFO:tasks.rgw_multisite_tests: "period": "2e6123d2-d695-4350-9a01-61a455bc782b", 2023-05-31T21:35:43.082 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-05-31T21:35:43.082 INFO:tasks.rgw_multisite_tests: }, 2023-05-31T21:35:43.083 INFO:tasks.rgw_multisite_tests: "markers": [ 2023-05-31T21:35:43.083 INFO:tasks.rgw_multisite_tests: { 2023-05-31T21:35:43.083 INFO:tasks.rgw_multisite_tests: "key": 0, 2023-05-31T21:35:43.083 INFO:tasks.rgw_multisite_tests: "val": { 2023-05-31T21:35:43.084 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-05-31T21:35:43.084 INFO:tasks.rgw_multisite_tests: "marker": "1_1685566501.757183_260.1", 2023-05-31T21:35:43.084 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-05-31T21:35:43.084 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2023-05-31T21:35:43.085 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-05-31T21:35:43.085 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-05-31T20:55:01.757183Z", 2023-05-31T21:35:43.085 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-05-31T21:35:43.085 INFO:tasks.rgw_multisite_tests: } 2023-05-31T21:35:43.086 INFO:tasks.rgw_multisite_tests: }, 2023-05-31T21:35:43.086 INFO:tasks.rgw_multisite_tests: { 2023-05-31T21:35:43.086 INFO:tasks.rgw_multisite_tests: "key": 1, 2023-05-31T21:35:43.086 INFO:tasks.rgw_multisite_tests: "val": { 2023-05-31T21:35:43.087 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-05-31T21:35:43.087 INFO:tasks.rgw_multisite_tests: "marker": "1_1685566501.735717_251.1", 2023-05-31T21:35:43.087 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-05-31T21:35:43.087 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-05-31T21:35:43.088 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-05-31T21:35:43.088 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-05-31T20:55:01.735717Z", 2023-05-31T21:35:43.088 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-05-31T21:35:43.088 INFO:tasks.rgw_multisite_tests: } 2023-05-31T21:35:43.089 INFO:tasks.rgw_multisite_tests: }, 2023-05-31T21:35:43.089 INFO:tasks.rgw_multisite_tests: { 2023-05-31T21:35:43.089 INFO:tasks.rgw_multisite_tests: "key": 2, 2023-05-31T21:35:43.089 INFO:tasks.rgw_multisite_tests: "val": { 2023-05-31T21:35:43.090 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-05-31T21:35:43.090 INFO:tasks.rgw_multisite_tests: "marker": "1_1685562970.274572_93.1", 2023-05-31T21:35:43.090 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-05-31T21:35:43.090 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-05-31T21:35:43.090 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-05-31T21:35:43.091 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-05-31T19:56:10.274572Z", 2023-05-31T21:35:43.091 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-05-31T21:35:43.091 INFO:tasks.rgw_multisite_tests: } 2023-05-31T21:35:43.091 INFO:tasks.rgw_multisite_tests: }, 2023-05-31T21:35:43.092 INFO:tasks.rgw_multisite_tests: { 2023-05-31T21:35:43.092 INFO:tasks.rgw_multisite_tests: "key": 3, 2023-05-31T21:35:43.092 INFO:tasks.rgw_multisite_tests: "val": { 2023-05-31T21:35:43.092 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-05-31T21:35:43.093 INFO:tasks.rgw_multisite_tests: "marker": "1_1685567943.999406_931.1", 2023-05-31T21:35:43.093 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-05-31T21:35:43.093 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-05-31T21:35:43.093 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-05-31T21:35:43.093 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-05-31T21:19:03.999406Z", 2023-05-31T21:35:43.094 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-05-31T21:35:43.094 INFO:tasks.rgw_multisite_tests: } 2023-05-31T21:35:43.094 INFO:tasks.rgw_multisite_tests: } 2023-05-31T21:35:43.094 INFO:tasks.rgw_multisite_tests: ] 2023-05-31T21:35:43.095 INFO:tasks.rgw_multisite_tests: }, 2023-05-31T21:35:43.095 INFO:tasks.rgw_multisite_tests: "full_sync": { 2023-05-31T21:35:43.095 INFO:tasks.rgw_multisite_tests: "total": 1, 2023-05-31T21:35:43.095 INFO:tasks.rgw_multisite_tests: "complete": 1 2023-05-31T21:35:43.096 INFO:tasks.rgw_multisite_tests: }, 2023-05-31T21:35:43.096 INFO:tasks.rgw_multisite_tests: "current_time": "2023-05-31T21:21:29Z" 2023-05-31T21:35:43.096 INFO:tasks.rgw_multisite_tests:} 2023-05-31T21:35:43.096 INFO:tasks.rgw_multisite_tests: 2023-05-31T21:35:43.096 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1685566501.757183_260.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-05-31T20:55:01.757183Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1685566501.735717_251.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-05-31T20:55:01.735717Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1685562970.274572_93.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-05-31T19:56:10.274572Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1685567943.999406_931.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-05-31T21:19:03.999406Z', 'realm_epoch': 2}}] 2023-05-31T21:35:43.097 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1685566501.757183_260.1', 1: '1_1685566501.735717_251.1', 2: '1_1685562970.274572_93.1', 3: '1_1685568089.268814_960.1'} 2023-05-31T21:35:43.097 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1685566501.757183_260.1', 1: '1_1685566501.735717_251.1', 2: '1_1685562970.274572_93.1', 3: '1_1685567943.999406_931.1'} 2023-05-31T21:35:43.097 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone a2 behind master: shard=3 master=1_1685568089.268814_960.1 target=1_1685567943.999406_931.1 2023-05-31T21:35:43.097 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-31T21:35:43.098 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-31T21:35:43.098 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-05-31T21:35:43.098 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: ignoring --setuser ceph since I am not root 2023-05-31T21:35:43.098 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-05-31T21:35:43.099 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: 2023-05-31T21:21:34.642+0000 7f4312202780 20 RGW-SYNC:meta: read sync status 2023-05-31T21:35:43.099 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: 2023-05-31T21:21:34.643+0000 7f4312202780 20 RGW-SYNC:meta: read sync status 2023-05-31T21:35:43.099 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2023-05-31T21:35:43.099 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "sync_status": { 2023-05-31T21:35:43.100 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "info": { 2023-05-31T21:35:43.100 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "status": "sync", 2023-05-31T21:35:43.100 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "num_shards": 4, 2023-05-31T21:35:43.100 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "period": "2e6123d2-d695-4350-9a01-61a455bc782b", 2023-05-31T21:35:43.101 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "realm_epoch": 2 2023-05-31T21:35:43.101 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2023-05-31T21:35:43.101 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "markers": [ 2023-05-31T21:35:43.101 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2023-05-31T21:35:43.101 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "key": 0, 2023-05-31T21:35:43.102 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "val": { 2023-05-31T21:35:43.102 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "state": 1, 2023-05-31T21:35:43.102 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "marker": "1_1685566501.757183_260.1", 2023-05-31T21:35:43.102 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "next_step_marker": "", 2023-05-31T21:35:43.103 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "total_entries": 1, 2023-05-31T21:35:43.103 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "pos": 0, 2023-05-31T21:35:43.103 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "timestamp": "2023-05-31T20:55:01.757183Z", 2023-05-31T21:35:43.103 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "realm_epoch": 2 2023-05-31T21:35:43.104 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2023-05-31T21:35:43.104 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2023-05-31T21:35:43.104 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2023-05-31T21:35:43.104 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "key": 1, 2023-05-31T21:35:43.104 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "val": { 2023-05-31T21:35:43.105 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "state": 1, 2023-05-31T21:35:43.105 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "marker": "1_1685566501.735717_251.1", 2023-05-31T21:35:43.105 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "next_step_marker": "", 2023-05-31T21:35:43.105 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "total_entries": 0, 2023-05-31T21:35:43.106 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "pos": 0, 2023-05-31T21:35:43.106 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "timestamp": "2023-05-31T20:55:01.735717Z", 2023-05-31T21:35:43.106 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "realm_epoch": 2 2023-05-31T21:35:43.106 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2023-05-31T21:35:43.107 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2023-05-31T21:35:43.107 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2023-05-31T21:35:43.107 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "key": 2, 2023-05-31T21:35:43.107 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "val": { 2023-05-31T21:35:43.107 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "state": 1, 2023-05-31T21:35:43.108 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "marker": "1_1685562970.274572_93.1", 2023-05-31T21:35:43.108 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "next_step_marker": "", 2023-05-31T21:35:43.108 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "total_entries": 0, 2023-05-31T21:35:43.108 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "pos": 0, 2023-05-31T21:35:43.109 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "timestamp": "2023-05-31T19:56:10.274572Z", 2023-05-31T21:35:43.109 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "realm_epoch": 2 2023-05-31T21:35:43.109 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2023-05-31T21:35:43.109 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2023-05-31T21:35:43.110 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2023-05-31T21:35:43.110 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "key": 3, 2023-05-31T21:35:43.110 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "val": { 2023-05-31T21:35:43.110 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "state": 1, 2023-05-31T21:35:43.111 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "marker": "1_1685568089.268814_960.1", 2023-05-31T21:35:43.111 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "next_step_marker": "", 2023-05-31T21:35:43.111 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "total_entries": 0, 2023-05-31T21:35:43.111 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "pos": 0, 2023-05-31T21:35:43.111 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "timestamp": "2023-05-31T21:21:29.268814Z", 2023-05-31T21:35:43.112 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "realm_epoch": 2 2023-05-31T21:35:43.112 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2023-05-31T21:35:43.112 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2023-05-31T21:35:43.112 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: ] 2023-05-31T21:35:43.113 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2023-05-31T21:35:43.113 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "full_sync": { 2023-05-31T21:35:43.113 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "total": 1, 2023-05-31T21:35:43.113 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "complete": 1 2023-05-31T21:35:43.114 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2023-05-31T21:35:43.114 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "current_time": "2023-05-31T21:21:34Z" 2023-05-31T21:35:43.114 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2023-05-31T21:35:43.114 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2023-05-31T21:35:43.114 INFO:tasks.rgw_multisite_tests: "sync_status": { 2023-05-31T21:35:43.115 INFO:tasks.rgw_multisite_tests: "info": { 2023-05-31T21:35:43.115 INFO:tasks.rgw_multisite_tests: "status": "sync", 2023-05-31T21:35:43.115 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2023-05-31T21:35:43.115 INFO:tasks.rgw_multisite_tests: "period": "2e6123d2-d695-4350-9a01-61a455bc782b", 2023-05-31T21:35:43.116 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-05-31T21:35:43.116 INFO:tasks.rgw_multisite_tests: }, 2023-05-31T21:35:43.116 INFO:tasks.rgw_multisite_tests: "markers": [ 2023-05-31T21:35:43.116 INFO:tasks.rgw_multisite_tests: { 2023-05-31T21:35:43.117 INFO:tasks.rgw_multisite_tests: "key": 0, 2023-05-31T21:35:43.117 INFO:tasks.rgw_multisite_tests: "val": { 2023-05-31T21:35:43.117 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-05-31T21:35:43.117 INFO:tasks.rgw_multisite_tests: "marker": "1_1685566501.757183_260.1", 2023-05-31T21:35:43.117 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-05-31T21:35:43.118 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2023-05-31T21:35:43.118 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-05-31T21:35:43.118 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-05-31T20:55:01.757183Z", 2023-05-31T21:35:43.118 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-05-31T21:35:43.119 INFO:tasks.rgw_multisite_tests: } 2023-05-31T21:35:43.119 INFO:tasks.rgw_multisite_tests: }, 2023-05-31T21:35:43.119 INFO:tasks.rgw_multisite_tests: { 2023-05-31T21:35:43.119 INFO:tasks.rgw_multisite_tests: "key": 1, 2023-05-31T21:35:43.120 INFO:tasks.rgw_multisite_tests: "val": { 2023-05-31T21:35:43.120 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-05-31T21:35:43.120 INFO:tasks.rgw_multisite_tests: "marker": "1_1685566501.735717_251.1", 2023-05-31T21:35:43.120 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-05-31T21:35:43.120 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-05-31T21:35:43.121 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-05-31T21:35:43.121 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-05-31T20:55:01.735717Z", 2023-05-31T21:35:43.121 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-05-31T21:35:43.121 INFO:tasks.rgw_multisite_tests: } 2023-05-31T21:35:43.122 INFO:tasks.rgw_multisite_tests: }, 2023-05-31T21:35:43.123 INFO:tasks.rgw_multisite_tests: { 2023-05-31T21:35:43.124 INFO:tasks.rgw_multisite_tests: "key": 2, 2023-05-31T21:35:43.124 INFO:tasks.rgw_multisite_tests: "val": { 2023-05-31T21:35:43.124 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-05-31T21:35:43.124 INFO:tasks.rgw_multisite_tests: "marker": "1_1685562970.274572_93.1", 2023-05-31T21:35:43.125 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-05-31T21:35:43.125 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-05-31T21:35:43.125 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-05-31T21:35:43.125 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-05-31T19:56:10.274572Z", 2023-05-31T21:35:43.125 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-05-31T21:35:43.126 INFO:tasks.rgw_multisite_tests: } 2023-05-31T21:35:43.126 INFO:tasks.rgw_multisite_tests: }, 2023-05-31T21:35:43.126 INFO:tasks.rgw_multisite_tests: { 2023-05-31T21:35:43.126 INFO:tasks.rgw_multisite_tests: "key": 3, 2023-05-31T21:35:43.127 INFO:tasks.rgw_multisite_tests: "val": { 2023-05-31T21:35:43.127 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-05-31T21:35:43.127 INFO:tasks.rgw_multisite_tests: "marker": "1_1685568089.268814_960.1", 2023-05-31T21:35:43.127 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-05-31T21:35:43.128 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-05-31T21:35:43.128 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-05-31T21:35:43.128 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-05-31T21:21:29.268814Z", 2023-05-31T21:35:43.128 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-05-31T21:35:43.128 INFO:tasks.rgw_multisite_tests: } 2023-05-31T21:35:43.129 INFO:tasks.rgw_multisite_tests: } 2023-05-31T21:35:43.129 INFO:tasks.rgw_multisite_tests: ] 2023-05-31T21:35:43.129 INFO:tasks.rgw_multisite_tests: }, 2023-05-31T21:35:43.129 INFO:tasks.rgw_multisite_tests: "full_sync": { 2023-05-31T21:35:43.130 INFO:tasks.rgw_multisite_tests: "total": 1, 2023-05-31T21:35:43.130 INFO:tasks.rgw_multisite_tests: "complete": 1 2023-05-31T21:35:43.130 INFO:tasks.rgw_multisite_tests: }, 2023-05-31T21:35:43.130 INFO:tasks.rgw_multisite_tests: "current_time": "2023-05-31T21:21:34Z" 2023-05-31T21:35:43.131 INFO:tasks.rgw_multisite_tests:} 2023-05-31T21:35:43.131 INFO:tasks.rgw_multisite_tests: 2023-05-31T21:35:43.131 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1685566501.757183_260.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-05-31T20:55:01.757183Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1685566501.735717_251.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-05-31T20:55:01.735717Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1685562970.274572_93.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-05-31T19:56:10.274572Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1685568089.268814_960.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-05-31T21:21:29.268814Z', 'realm_epoch': 2}}] 2023-05-31T21:35:43.131 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1685566501.757183_260.1', 1: '1_1685566501.735717_251.1', 2: '1_1685562970.274572_93.1', 3: '1_1685568089.268814_960.1'} 2023-05-31T21:35:43.131 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1685566501.757183_260.1', 1: '1_1685566501.735717_251.1', 2: '1_1685562970.274572_93.1', 3: '1_1685568089.268814_960.1'} 2023-05-31T21:35:43.132 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: INFO: finish meta checkpoint for zone=a2 2023-05-31T21:35:43.132 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['sync', 'group', 'create', '--group-id', 'sync-group', '--status', 'allowed', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-05-31T21:35:43.132 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'group', 'create', '--group-id', 'sync-group', '--status', 'allowed', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-05-31T21:35:43.132 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync group create --group-id sync-group --status allowed --cluster c1 --debug-rgw 0 --debug-ms 0 2023-05-31T21:35:43.133 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: ignoring --setuser ceph since I am not root 2023-05-31T21:35:43.133 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-05-31T21:35:43.133 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: {"groups":[{"id":"sync-group","data_flow":{},"pipes":[],"status":"allowed"}]} 2023-05-31T21:35:43.133 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['sync', 'group', 'flow', 'create', '--group-id', 'sync-group', '--flow-id', 'sync-flow1', '--flow-type', 'symmetrical', '--zones=a1,a2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-05-31T21:35:43.134 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'group', 'flow', 'create', '--group-id', 'sync-group', '--flow-id', 'sync-flow1', '--flow-type', 'symmetrical', '--zones=a1,a2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-05-31T21:35:43.134 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync group flow create --group-id sync-group --flow-id sync-flow1 --flow-type symmetrical --zones=a1,a2 --cluster c1 --debug-rgw 0 --debug-ms 0 2023-05-31T21:35:43.134 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: ignoring --setuser ceph since I am not root 2023-05-31T21:35:43.134 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-05-31T21:35:43.135 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: {"groups":[{"id":"sync-group","data_flow":{"symmetrical":[{"id":"sync-flow1","zones":["a1","a2"]}]},"pipes":[],"status":"allowed"}]} 2023-05-31T21:35:43.135 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['sync', 'group', 'pipe', 'create', '--group-id', 'sync-group', '--pipe-id', 'sync-pipe', '--source-zones=a1,a2', '--dest-zones=a1,a2', '--source-bucket=pklxkd-68', '--dest-bucket=pklxkd-68', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-05-31T21:35:43.135 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'group', 'pipe', 'create', '--group-id', 'sync-group', '--pipe-id', 'sync-pipe', '--source-zones=a1,a2', '--dest-zones=a1,a2', '--source-bucket=pklxkd-68', '--dest-bucket=pklxkd-68', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-05-31T21:35:43.135 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync group pipe create --group-id sync-group --pipe-id sync-pipe --source-zones=a1,a2 --dest-zones=a1,a2 --source-bucket=pklxkd-68 --dest-bucket=pklxkd-68 --cluster c1 --debug-rgw 0 --debug-ms 0 2023-05-31T21:35:43.135 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: ignoring --setuser ceph since I am not root 2023-05-31T21:35:43.136 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-05-31T21:35:43.136 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: {"groups":[{"id":"sync-group","data_flow":{"symmetrical":[{"id":"sync-flow1","zones":["a1","a2"]}]},"pipes":[{"id":"sync-pipe","source":{"bucket":"pklxkd-68","zones":["a1","a2"]},"dest":{"bucket":"pklxkd-68","zones":["a1","a2"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":""}}],"status":"allowed"}]} 2023-05-31T21:35:43.136 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['sync', 'group', 'modify', '--group-id', 'sync-group', '--status', 'enabled', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-05-31T21:35:43.136 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'group', 'modify', '--group-id', 'sync-group', '--status', 'enabled', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-05-31T21:35:43.137 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync group modify --group-id sync-group --status enabled --cluster c1 --debug-rgw 0 --debug-ms 0 2023-05-31T21:35:43.137 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: ignoring --setuser ceph since I am not root 2023-05-31T21:35:43.137 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-05-31T21:35:43.137 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: {"groups":[{"id":"sync-group","data_flow":{"symmetrical":[{"id":"sync-flow1","zones":["a1","a2"]}]},"pipes":[{"id":"sync-pipe","source":{"bucket":"pklxkd-68","zones":["a1","a2"]},"dest":{"bucket":"pklxkd-68","zones":["a1","a2"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":""}}],"status":"enabled"}]} 2023-05-31T21:35:43.138 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['sync', 'policy', 'get', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-05-31T21:35:43.138 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'policy', 'get', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-05-31T21:35:43.138 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync policy get --cluster c1 --debug-rgw 0 --debug-ms 0 2023-05-31T21:35:43.138 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: ignoring --setuser ceph since I am not root 2023-05-31T21:35:43.139 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-05-31T21:35:43.139 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: {"groups":[{"id":"sync-group","data_flow":{"symmetrical":[{"id":"sync-flow1","zones":["a1","a2"]}]},"pipes":[{"id":"sync-pipe","source":{"bucket":"pklxkd-68","zones":["a1","a2"]},"dest":{"bucket":"pklxkd-68","zones":["a1","a2"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":""}}],"status":"enabled"}]} 2023-05-31T21:35:43.139 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['period', 'update', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-05-31T21:35:43.139 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'period', 'update', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-05-31T21:35:43.139 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 period update --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --commit --cluster c1 --debug-rgw 0 --debug-ms 0 2023-05-31T21:35:43.140 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: ignoring --setuser ceph since I am not root 2023-05-31T21:35:43.140 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-05-31T21:35:43.140 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2023-05-31T21:35:43.140 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "id": "2e6123d2-d695-4350-9a01-61a455bc782b", 2023-05-31T21:35:43.141 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "epoch": 16, 2023-05-31T21:35:43.141 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "predecessor_uuid": "629b9ced-80af-41ed-8a6c-f2cfc334cbb4", 2023-05-31T21:35:43.141 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "sync_status": [], 2023-05-31T21:35:43.141 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "period_map": { 2023-05-31T21:35:43.142 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "id": "2e6123d2-d695-4350-9a01-61a455bc782b", 2023-05-31T21:35:43.142 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "zonegroups": [ 2023-05-31T21:35:43.142 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2023-05-31T21:35:43.142 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "id": "a", 2023-05-31T21:35:43.142 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "name": "a", 2023-05-31T21:35:43.143 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "api_name": "", 2023-05-31T21:35:43.143 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "is_master": true, 2023-05-31T21:35:43.143 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "endpoints": [ 2023-05-31T21:35:43.143 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "http://smithi099.front.sepia.ceph.com:8000" 2023-05-31T21:35:43.144 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: ], 2023-05-31T21:35:43.144 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "hostnames": [], 2023-05-31T21:35:43.144 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "hostnames_s3website": [], 2023-05-31T21:35:43.144 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "master_zone": "3d163b1e-09c7-47e2-ae87-add4aace3f38", 2023-05-31T21:35:43.145 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "zones": [ 2023-05-31T21:35:43.145 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2023-05-31T21:35:43.145 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "id": "3d163b1e-09c7-47e2-ae87-add4aace3f38", 2023-05-31T21:35:43.145 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "name": "a1", 2023-05-31T21:35:43.145 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "endpoints": [ 2023-05-31T21:35:43.146 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "http://smithi099.front.sepia.ceph.com:8000" 2023-05-31T21:35:43.146 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: ], 2023-05-31T21:35:43.146 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "log_meta": false, 2023-05-31T21:35:43.146 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "log_data": true, 2023-05-31T21:35:43.147 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "bucket_index_max_shards": 11, 2023-05-31T21:35:43.147 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "read_only": false, 2023-05-31T21:35:43.147 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "tier_type": "", 2023-05-31T21:35:43.147 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "sync_from_all": true, 2023-05-31T21:35:43.148 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "sync_from": [], 2023-05-31T21:35:43.148 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "redirect_zone": "", 2023-05-31T21:35:43.148 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "supported_features": [ 2023-05-31T21:35:43.148 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "resharding" 2023-05-31T21:35:43.149 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: ] 2023-05-31T21:35:43.149 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2023-05-31T21:35:43.149 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2023-05-31T21:35:43.149 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "id": "5d8acec7-7e57-4e56-8b32-e3d6b5036001", 2023-05-31T21:35:43.149 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "name": "a2", 2023-05-31T21:35:43.150 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "endpoints": [ 2023-05-31T21:35:43.150 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "http://smithi099.front.sepia.ceph.com:8001" 2023-05-31T21:35:43.150 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: ], 2023-05-31T21:35:43.150 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "log_meta": false, 2023-05-31T21:35:43.151 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "log_data": true, 2023-05-31T21:35:43.151 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "bucket_index_max_shards": 11, 2023-05-31T21:35:43.151 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "read_only": false, 2023-05-31T21:35:43.151 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "tier_type": "", 2023-05-31T21:35:43.152 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "sync_from_all": true, 2023-05-31T21:35:43.152 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "sync_from": [], 2023-05-31T21:35:43.152 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "redirect_zone": "", 2023-05-31T21:35:43.152 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "supported_features": [ 2023-05-31T21:35:43.153 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "resharding" 2023-05-31T21:35:43.153 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: ] 2023-05-31T21:35:43.153 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2023-05-31T21:35:43.153 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: ], 2023-05-31T21:35:43.154 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "placement_targets": [ 2023-05-31T21:35:43.154 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2023-05-31T21:35:43.154 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "name": "default-placement", 2023-05-31T21:35:43.154 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "tags": [], 2023-05-31T21:35:43.155 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "storage_classes": [ 2023-05-31T21:35:43.155 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "STANDARD" 2023-05-31T21:35:43.155 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: ] 2023-05-31T21:35:43.155 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2023-05-31T21:35:43.156 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: ], 2023-05-31T21:35:43.156 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "default_placement": "default-placement", 2023-05-31T21:35:43.156 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "realm_id": "aaa33de6-515b-4cac-8f1b-3a1a30452117", 2023-05-31T21:35:43.156 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "sync_policy": { 2023-05-31T21:35:43.157 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "groups": [ 2023-05-31T21:35:43.157 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2023-05-31T21:35:43.157 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "id": "sync-group", 2023-05-31T21:35:43.158 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "data_flow": { 2023-05-31T21:35:43.158 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "symmetrical": [ 2023-05-31T21:35:43.158 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2023-05-31T21:35:43.158 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "id": "sync-flow1", 2023-05-31T21:35:43.159 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "zones": [ 2023-05-31T21:35:43.159 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "3d163b1e-09c7-47e2-ae87-add4aace3f38", 2023-05-31T21:35:43.159 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "5d8acec7-7e57-4e56-8b32-e3d6b5036001" 2023-05-31T21:35:43.159 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: ] 2023-05-31T21:35:43.160 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2023-05-31T21:35:43.160 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: ] 2023-05-31T21:35:43.160 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2023-05-31T21:35:43.160 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "pipes": [ 2023-05-31T21:35:43.161 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2023-05-31T21:35:43.161 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "id": "sync-pipe", 2023-05-31T21:35:43.161 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "source": { 2023-05-31T21:35:43.161 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "bucket": "pklxkd-68", 2023-05-31T21:35:43.161 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "zones": [ 2023-05-31T21:35:43.162 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "3d163b1e-09c7-47e2-ae87-add4aace3f38", 2023-05-31T21:35:43.162 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "5d8acec7-7e57-4e56-8b32-e3d6b5036001" 2023-05-31T21:35:43.162 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: ] 2023-05-31T21:35:43.162 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2023-05-31T21:35:43.163 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "dest": { 2023-05-31T21:35:43.163 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "bucket": "pklxkd-68", 2023-05-31T21:35:43.163 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "zones": [ 2023-05-31T21:35:43.163 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "3d163b1e-09c7-47e2-ae87-add4aace3f38", 2023-05-31T21:35:43.164 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "5d8acec7-7e57-4e56-8b32-e3d6b5036001" 2023-05-31T21:35:43.164 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: ] 2023-05-31T21:35:43.164 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2023-05-31T21:35:43.164 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "params": { 2023-05-31T21:35:43.165 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "source": { 2023-05-31T21:35:43.165 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "filter": { 2023-05-31T21:35:43.165 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "tags": [] 2023-05-31T21:35:43.165 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2023-05-31T21:35:43.165 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2023-05-31T21:35:43.166 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "dest": {}, 2023-05-31T21:35:43.166 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "priority": 0, 2023-05-31T21:35:43.166 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "mode": "system", 2023-05-31T21:35:43.166 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "user": "" 2023-05-31T21:35:43.167 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2023-05-31T21:35:43.167 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2023-05-31T21:35:43.167 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: ], 2023-05-31T21:35:43.167 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "status": "enabled" 2023-05-31T21:35:43.168 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2023-05-31T21:35:43.168 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: ] 2023-05-31T21:35:43.168 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2023-05-31T21:35:43.168 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "enabled_features": [ 2023-05-31T21:35:43.169 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "resharding" 2023-05-31T21:35:43.169 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: ] 2023-05-31T21:35:43.169 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2023-05-31T21:35:43.169 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2023-05-31T21:35:43.169 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "id": "b", 2023-05-31T21:35:43.170 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "name": "b", 2023-05-31T21:35:43.170 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "api_name": "", 2023-05-31T21:35:43.170 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "is_master": false, 2023-05-31T21:35:43.170 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "endpoints": [ 2023-05-31T21:35:43.171 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "http://smithi112.front.sepia.ceph.com:8000" 2023-05-31T21:35:43.171 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: ], 2023-05-31T21:35:43.171 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "hostnames": [], 2023-05-31T21:35:43.171 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "hostnames_s3website": [], 2023-05-31T21:35:43.172 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "master_zone": "f85d7236-2f27-4988-af26-9c66ab280b1b", 2023-05-31T21:35:43.172 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "zones": [ 2023-05-31T21:35:43.172 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2023-05-31T21:35:43.172 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "id": "a6a3aab5-bc52-44a4-8543-5c5ea6056acd", 2023-05-31T21:35:43.173 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "name": "b2", 2023-05-31T21:35:43.173 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "endpoints": [ 2023-05-31T21:35:43.173 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "http://smithi112.front.sepia.ceph.com:8001" 2023-05-31T21:35:43.173 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: ], 2023-05-31T21:35:43.173 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "log_meta": false, 2023-05-31T21:35:43.174 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "log_data": true, 2023-05-31T21:35:43.174 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "bucket_index_max_shards": 11, 2023-05-31T21:35:43.174 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "read_only": false, 2023-05-31T21:35:43.174 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "tier_type": "", 2023-05-31T21:35:43.175 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "sync_from_all": true, 2023-05-31T21:35:43.175 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "sync_from": [], 2023-05-31T21:35:43.175 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "redirect_zone": "", 2023-05-31T21:35:43.175 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "supported_features": [ 2023-05-31T21:35:43.176 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "resharding" 2023-05-31T21:35:43.176 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: ] 2023-05-31T21:35:43.176 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2023-05-31T21:35:43.176 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2023-05-31T21:35:43.177 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "id": "f85d7236-2f27-4988-af26-9c66ab280b1b", 2023-05-31T21:35:43.177 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "name": "b1", 2023-05-31T21:35:43.177 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "endpoints": [ 2023-05-31T21:35:43.177 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "http://smithi112.front.sepia.ceph.com:8000" 2023-05-31T21:35:43.177 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: ], 2023-05-31T21:35:43.178 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "log_meta": false, 2023-05-31T21:35:43.178 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "log_data": true, 2023-05-31T21:35:43.178 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "bucket_index_max_shards": 11, 2023-05-31T21:35:43.178 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "read_only": false, 2023-05-31T21:35:43.179 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "tier_type": "", 2023-05-31T21:35:43.179 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "sync_from_all": true, 2023-05-31T21:35:43.179 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "sync_from": [], 2023-05-31T21:35:43.179 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "redirect_zone": "", 2023-05-31T21:35:43.180 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "supported_features": [ 2023-05-31T21:35:43.180 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "resharding" 2023-05-31T21:35:43.180 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: ] 2023-05-31T21:35:43.180 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2023-05-31T21:35:43.181 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: ], 2023-05-31T21:35:43.181 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "placement_targets": [ 2023-05-31T21:35:43.181 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2023-05-31T21:35:43.181 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "name": "default-placement", 2023-05-31T21:35:43.181 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "tags": [], 2023-05-31T21:35:43.182 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "storage_classes": [ 2023-05-31T21:35:43.182 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "STANDARD" 2023-05-31T21:35:43.182 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: ] 2023-05-31T21:35:43.182 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2023-05-31T21:35:43.183 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: ], 2023-05-31T21:35:43.183 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "default_placement": "default-placement", 2023-05-31T21:35:43.183 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "realm_id": "aaa33de6-515b-4cac-8f1b-3a1a30452117", 2023-05-31T21:35:43.183 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "sync_policy": { 2023-05-31T21:35:43.184 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "groups": [] 2023-05-31T21:35:43.184 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2023-05-31T21:35:43.184 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "enabled_features": [ 2023-05-31T21:35:43.184 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "resharding" 2023-05-31T21:35:43.185 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: ] 2023-05-31T21:35:43.185 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2023-05-31T21:35:43.185 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: ], 2023-05-31T21:35:43.185 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "short_zone_ids": [ 2023-05-31T21:35:43.185 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2023-05-31T21:35:43.186 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "key": "3d163b1e-09c7-47e2-ae87-add4aace3f38", 2023-05-31T21:35:43.186 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "val": 3101833771 2023-05-31T21:35:43.186 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2023-05-31T21:35:43.186 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2023-05-31T21:35:43.187 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "key": "5d8acec7-7e57-4e56-8b32-e3d6b5036001", 2023-05-31T21:35:43.187 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "val": 4255530623 2023-05-31T21:35:43.187 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2023-05-31T21:35:43.187 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2023-05-31T21:35:43.188 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "key": "a6a3aab5-bc52-44a4-8543-5c5ea6056acd", 2023-05-31T21:35:43.188 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "val": 3708073710 2023-05-31T21:35:43.188 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2023-05-31T21:35:43.188 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2023-05-31T21:35:43.189 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "key": "f85d7236-2f27-4988-af26-9c66ab280b1b", 2023-05-31T21:35:43.189 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "val": 233395200 2023-05-31T21:35:43.189 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2023-05-31T21:35:43.189 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: ] 2023-05-31T21:35:43.189 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2023-05-31T21:35:43.190 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "master_zonegroup": "a", 2023-05-31T21:35:43.190 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "master_zone": "3d163b1e-09c7-47e2-ae87-add4aace3f38", 2023-05-31T21:35:43.190 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "period_config": { 2023-05-31T21:35:43.190 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "bucket_quota": { 2023-05-31T21:35:43.191 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "enabled": false, 2023-05-31T21:35:43.191 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "check_on_raw": false, 2023-05-31T21:35:43.191 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "max_size": -1, 2023-05-31T21:35:43.191 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "max_size_kb": 0, 2023-05-31T21:35:43.192 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "max_objects": -1 2023-05-31T21:35:43.192 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2023-05-31T21:35:43.192 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "user_quota": { 2023-05-31T21:35:43.192 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "enabled": false, 2023-05-31T21:35:43.192 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "check_on_raw": false, 2023-05-31T21:35:43.193 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "max_size": -1, 2023-05-31T21:35:43.193 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "max_size_kb": 0, 2023-05-31T21:35:43.193 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "max_objects": -1 2023-05-31T21:35:43.193 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2023-05-31T21:35:43.194 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "user_ratelimit": { 2023-05-31T21:35:43.194 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "max_read_ops": 0, 2023-05-31T21:35:43.194 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "max_write_ops": 0, 2023-05-31T21:35:43.194 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "max_read_bytes": 0, 2023-05-31T21:35:43.195 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "max_write_bytes": 0, 2023-05-31T21:35:43.195 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "enabled": false 2023-05-31T21:35:43.195 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2023-05-31T21:35:43.195 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "bucket_ratelimit": { 2023-05-31T21:35:43.195 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "max_read_ops": 0, 2023-05-31T21:35:43.196 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "max_write_ops": 0, 2023-05-31T21:35:43.196 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "max_read_bytes": 0, 2023-05-31T21:35:43.196 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "max_write_bytes": 0, 2023-05-31T21:35:43.196 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "enabled": false 2023-05-31T21:35:43.197 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2023-05-31T21:35:43.197 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "anonymous_ratelimit": { 2023-05-31T21:35:43.197 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "max_read_ops": 0, 2023-05-31T21:35:43.197 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "max_write_ops": 0, 2023-05-31T21:35:43.198 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "max_read_bytes": 0, 2023-05-31T21:35:43.198 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "max_write_bytes": 0, 2023-05-31T21:35:43.198 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "enabled": false 2023-05-31T21:35:43.198 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2023-05-31T21:35:43.199 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2023-05-31T21:35:43.199 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "realm_id": "aaa33de6-515b-4cac-8f1b-3a1a30452117", 2023-05-31T21:35:43.199 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "realm_name": "test-realm", 2023-05-31T21:35:43.199 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "realm_epoch": 2 2023-05-31T21:35:43.199 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2023-05-31T21:35:43.200 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['sync', 'info', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-05-31T21:35:43.200 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'info', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-05-31T21:35:43.200 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync info --cluster c1 --debug-rgw 0 --debug-ms 0 2023-05-31T21:35:43.200 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: ignoring --setuser ceph since I am not root 2023-05-31T21:35:43.201 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-05-31T21:35:43.201 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: {"sources":[{"id":"sync-pipe","source":{"zone":"a2","bucket":"pklxkd-68"},"dest":{"zone":"a1","bucket":"pklxkd-68"},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":""}}],"dests":[{"id":"sync-pipe","source":{"zone":"a1","bucket":"pklxkd-68"},"dest":{"zone":"a2","bucket":"pklxkd-68"},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":""}}],"hints":{"sources":[],"dests":[]},"resolved-hints-1":{"sources":[],"dests":[]},"resolved-hints":{"sources":[],"dests":[]}} 2023-05-31T21:35:43.201 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/pklxkd-68/ 2023-05-31T21:35:43.201 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/pklxkd-68/ 2023-05-31T21:35:43.202 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-05-31T21:35:43.202 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /pklxkd-68/ 2023-05-31T21:35:43.202 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-05-31T21:35:43.202 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-05-31T21:35:43.203 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T21:35:43.203 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-05-31T21:35:43.203 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-05-31T21:35:43.203 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-05-31T21:35:43.203 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-05-31T21:35:43.204 INFO:tasks.rgw_multisite_tests:HEAD 2023-05-31T21:35:43.204 INFO:tasks.rgw_multisite_tests: 2023-05-31T21:35:43.204 INFO:tasks.rgw_multisite_tests: 2023-05-31T21:35:43.204 INFO:tasks.rgw_multisite_tests:Wed, 31 May 2023 21:21:35 GMT 2023-05-31T21:35:43.205 INFO:tasks.rgw_multisite_tests:/pklxkd-68/ 2023-05-31T21:35:43.205 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-05-31T21:35:43.205 INFO:tasks.rgw_multisite_tests:AWS Q80H11OLNU1S48PPYL26:waphac2IPrTsrsEibWZ+aErNyv0= 2023-05-31T21:35:43.205 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 21:21:35 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:waphac2IPrTsrsEibWZ+aErNyv0=', 'Content-Length': '0'} 2023-05-31T21:35:43.206 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000f3a604afe26736ee-006477ba5f-242d-a1'), ('Content-Length', '0'), ('Date', 'Wed, 31 May 2023 21:21:35 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:35:43.206 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/pklxkd-68/obj1 2023-05-31T21:35:43.206 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/pklxkd-68/obj1 2023-05-31T21:35:43.206 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2023-05-31T21:35:43.207 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /pklxkd-68/obj1 2023-05-31T21:35:43.207 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-05-31T21:35:43.207 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2023-05-31T21:35:43.207 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T21:35:43.207 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-05-31T21:35:43.208 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-05-31T21:35:43.208 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-05-31T21:35:43.208 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-05-31T21:35:43.208 INFO:tasks.rgw_multisite_tests:PUT 2023-05-31T21:35:43.209 INFO:tasks.rgw_multisite_tests:qPXxZ/RPSWTmyZje6CcRDA== 2023-05-31T21:35:43.209 INFO:tasks.rgw_multisite_tests:application/octet-stream 2023-05-31T21:35:43.209 INFO:tasks.rgw_multisite_tests:Wed, 31 May 2023 21:21:35 GMT 2023-05-31T21:35:43.209 INFO:tasks.rgw_multisite_tests:/pklxkd-68/obj1 2023-05-31T21:35:43.210 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-05-31T21:35:43.210 INFO:tasks.rgw_multisite_tests:AWS Q80H11OLNU1S48PPYL26:mVI/r+9qbAxPZXIal0tVAHngjH8= 2023-05-31T21:35:43.210 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Wed, 31 May 2023 21:21:35 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:mVI/r+9qbAxPZXIal0tVAHngjH8='} 2023-05-31T21:35:43.210 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000c2e7b72ae9b90a31-006477ba5f-242d-a1'), ('Date', 'Wed, 31 May 2023 21:21:35 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:35:43.211 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/pklxkd-68/ 2023-05-31T21:35:43.211 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/pklxkd-68/ 2023-05-31T21:35:43.211 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-05-31T21:35:43.211 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /pklxkd-68/ 2023-05-31T21:35:43.211 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-05-31T21:35:43.212 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-05-31T21:35:43.212 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T21:35:43.212 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-05-31T21:35:43.212 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-05-31T21:35:43.213 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-05-31T21:35:43.213 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-05-31T21:35:43.213 INFO:tasks.rgw_multisite_tests:HEAD 2023-05-31T21:35:43.213 INFO:tasks.rgw_multisite_tests: 2023-05-31T21:35:43.213 INFO:tasks.rgw_multisite_tests: 2023-05-31T21:35:43.214 INFO:tasks.rgw_multisite_tests:Wed, 31 May 2023 21:21:35 GMT 2023-05-31T21:35:43.214 INFO:tasks.rgw_multisite_tests:/pklxkd-68/ 2023-05-31T21:35:43.214 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-05-31T21:35:43.214 INFO:tasks.rgw_multisite_tests:AWS Q80H11OLNU1S48PPYL26:waphac2IPrTsrsEibWZ+aErNyv0= 2023-05-31T21:35:43.215 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 21:21:35 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:waphac2IPrTsrsEibWZ+aErNyv0=', 'Content-Length': '0'} 2023-05-31T21:35:43.215 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '6'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000017f85e330601fcd-006477ba5f-242d-a1'), ('Content-Length', '0'), ('Date', 'Wed, 31 May 2023 21:21:35 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:35:43.215 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/pklxkd-68/obj2 2023-05-31T21:35:43.215 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/pklxkd-68/obj2 2023-05-31T21:35:43.216 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2023-05-31T21:35:43.216 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /pklxkd-68/obj2 2023-05-31T21:35:43.216 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-05-31T21:35:43.216 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2023-05-31T21:35:43.216 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T21:35:43.217 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-05-31T21:35:43.217 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-05-31T21:35:43.217 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-05-31T21:35:43.217 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-05-31T21:35:43.218 INFO:tasks.rgw_multisite_tests:PUT 2023-05-31T21:35:43.218 INFO:tasks.rgw_multisite_tests:qPXxZ/RPSWTmyZje6CcRDA== 2023-05-31T21:35:43.218 INFO:tasks.rgw_multisite_tests:application/octet-stream 2023-05-31T21:35:43.218 INFO:tasks.rgw_multisite_tests:Wed, 31 May 2023 21:21:35 GMT 2023-05-31T21:35:43.219 INFO:tasks.rgw_multisite_tests:/pklxkd-68/obj2 2023-05-31T21:35:43.219 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-05-31T21:35:43.219 INFO:tasks.rgw_multisite_tests:AWS Q80H11OLNU1S48PPYL26:aHKs/QEIand6BqZjck4e1wt+6Bk= 2023-05-31T21:35:43.219 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Wed, 31 May 2023 21:21:35 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:aHKs/QEIand6BqZjck4e1wt+6Bk='} 2023-05-31T21:35:43.219 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000584ac59c48bd95e6-006477ba5f-242d-a1'), ('Date', 'Wed, 31 May 2023 21:21:35 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:35:43.220 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/pklxkd-68/ 2023-05-31T21:35:43.220 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/pklxkd-68/ 2023-05-31T21:35:43.220 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-05-31T21:35:43.220 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /pklxkd-68/ 2023-05-31T21:35:43.221 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-05-31T21:35:43.221 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-05-31T21:35:43.221 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T21:35:43.221 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-05-31T21:35:43.222 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-05-31T21:35:43.222 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-05-31T21:35:43.222 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-05-31T21:35:43.222 INFO:tasks.rgw_multisite_tests:HEAD 2023-05-31T21:35:43.223 INFO:tasks.rgw_multisite_tests: 2023-05-31T21:35:43.223 INFO:tasks.rgw_multisite_tests: 2023-05-31T21:35:43.223 INFO:tasks.rgw_multisite_tests:Wed, 31 May 2023 21:21:35 GMT 2023-05-31T21:35:43.223 INFO:tasks.rgw_multisite_tests:/pklxkd-68/ 2023-05-31T21:35:43.223 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-05-31T21:35:43.224 INFO:tasks.rgw_multisite_tests:AWS Q80H11OLNU1S48PPYL26:waphac2IPrTsrsEibWZ+aErNyv0= 2023-05-31T21:35:43.224 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 21:21:35 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:waphac2IPrTsrsEibWZ+aErNyv0=', 'Content-Length': '0'} 2023-05-31T21:35:43.224 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '2'), ('X-RGW-Bytes-Used', '12'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000aa561205481c805a-006477ba5f-242d-a1'), ('Content-Length', '0'), ('Date', 'Wed, 31 May 2023 21:21:35 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:35:43.224 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/pklxkd-68/obj3 2023-05-31T21:35:43.225 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/pklxkd-68/obj3 2023-05-31T21:35:43.225 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2023-05-31T21:35:43.225 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /pklxkd-68/obj3 2023-05-31T21:35:43.225 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-05-31T21:35:43.226 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2023-05-31T21:35:43.226 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T21:35:43.226 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-05-31T21:35:43.226 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-05-31T21:35:43.226 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-05-31T21:35:43.227 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-05-31T21:35:43.227 INFO:tasks.rgw_multisite_tests:PUT 2023-05-31T21:35:43.227 INFO:tasks.rgw_multisite_tests:qPXxZ/RPSWTmyZje6CcRDA== 2023-05-31T21:35:43.227 INFO:tasks.rgw_multisite_tests:application/octet-stream 2023-05-31T21:35:43.228 INFO:tasks.rgw_multisite_tests:Wed, 31 May 2023 21:21:35 GMT 2023-05-31T21:35:43.228 INFO:tasks.rgw_multisite_tests:/pklxkd-68/obj3 2023-05-31T21:35:43.228 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-05-31T21:35:43.228 INFO:tasks.rgw_multisite_tests:AWS Q80H11OLNU1S48PPYL26:Bn8vqAG+9LvasOYLKU5JX0URbBk= 2023-05-31T21:35:43.229 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Wed, 31 May 2023 21:21:35 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:Bn8vqAG+9LvasOYLKU5JX0URbBk='} 2023-05-31T21:35:43.229 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000058cc7d734f4fbf91-006477ba5f-242d-a1'), ('Date', 'Wed, 31 May 2023 21:21:35 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:35:43.229 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/pklxkd-69/ 2023-05-31T21:35:43.229 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/pklxkd-69/ 2023-05-31T21:35:43.229 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-05-31T21:35:43.230 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /pklxkd-69/ 2023-05-31T21:35:43.230 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-05-31T21:35:43.230 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-05-31T21:35:43.230 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T21:35:43.231 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-05-31T21:35:43.231 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-05-31T21:35:43.231 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-05-31T21:35:43.231 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-05-31T21:35:43.232 INFO:tasks.rgw_multisite_tests:HEAD 2023-05-31T21:35:43.232 INFO:tasks.rgw_multisite_tests: 2023-05-31T21:35:43.232 INFO:tasks.rgw_multisite_tests: 2023-05-31T21:35:43.232 INFO:tasks.rgw_multisite_tests:Wed, 31 May 2023 21:21:35 GMT 2023-05-31T21:35:43.233 INFO:tasks.rgw_multisite_tests:/pklxkd-69/ 2023-05-31T21:35:43.233 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-05-31T21:35:43.233 INFO:tasks.rgw_multisite_tests:AWS Q80H11OLNU1S48PPYL26:N0bfzuM83NgwBSzsFSC64cfJmzI= 2023-05-31T21:35:43.233 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 21:21:35 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:N0bfzuM83NgwBSzsFSC64cfJmzI=', 'Content-Length': '0'} 2023-05-31T21:35:43.233 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000001f28a495cdf8b6f8-006477ba5f-242d-a1'), ('Content-Length', '0'), ('Date', 'Wed, 31 May 2023 21:21:35 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:35:43.234 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/pklxkd-69/%5B%27obj1%27%2C%20%27obj2%27%2C%20%27obj3%27%5D 2023-05-31T21:35:43.234 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/pklxkd-69/%5B%27obj1%27%2C%20%27obj2%27%2C%20%27obj3%27%5D 2023-05-31T21:35:43.234 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2023-05-31T21:35:43.234 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /pklxkd-69/%5B%27obj1%27%2C%20%27obj2%27%2C%20%27obj3%27%5D 2023-05-31T21:35:43.235 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-05-31T21:35:43.235 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2023-05-31T21:35:43.235 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T21:35:43.235 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-05-31T21:35:43.236 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-05-31T21:35:43.236 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-05-31T21:35:43.236 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-05-31T21:35:43.236 INFO:tasks.rgw_multisite_tests:PUT 2023-05-31T21:35:43.237 INFO:tasks.rgw_multisite_tests:qPXxZ/RPSWTmyZje6CcRDA== 2023-05-31T21:35:43.237 INFO:tasks.rgw_multisite_tests:application/octet-stream 2023-05-31T21:35:43.237 INFO:tasks.rgw_multisite_tests:Wed, 31 May 2023 21:21:35 GMT 2023-05-31T21:35:43.237 INFO:tasks.rgw_multisite_tests:/pklxkd-69/%5B%27obj1%27%2C%20%27obj2%27%2C%20%27obj3%27%5D 2023-05-31T21:35:43.237 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-05-31T21:35:43.238 INFO:tasks.rgw_multisite_tests:AWS Q80H11OLNU1S48PPYL26:zKqYD/2vHmuBRT+s3/OBgHjg4A8= 2023-05-31T21:35:43.238 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Wed, 31 May 2023 21:21:35 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:zKqYD/2vHmuBRT+s3/OBgHjg4A8='} 2023-05-31T21:35:43.238 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000004f731815dc999073-006477ba5f-242d-a1'), ('Date', 'Wed, 31 May 2023 21:21:35 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:35:43.238 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-31T21:35:43.239 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-31T21:35:43.239 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-05-31T21:35:43.239 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: ignoring --setuser ceph since I am not root 2023-05-31T21:35:43.239 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-05-31T21:35:43.240 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: No --period given, using current period=2e6123d2-d695-4350-9a01-61a455bc782b 2023-05-31T21:35:43.240 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: [ 2023-05-31T21:35:43.240 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2023-05-31T21:35:43.240 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "marker": "1_1685566501.757183_260.1", 2023-05-31T21:35:43.241 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "last_update": "2023-05-31T20:55:01.757183Z" 2023-05-31T21:35:43.241 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2023-05-31T21:35:43.241 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2023-05-31T21:35:43.241 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "marker": "1_1685566501.735717_251.1", 2023-05-31T21:35:43.241 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "last_update": "2023-05-31T20:55:01.735717Z" 2023-05-31T21:35:43.242 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2023-05-31T21:35:43.242 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2023-05-31T21:35:43.242 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "marker": "1_1685562970.274572_93.1", 2023-05-31T21:35:43.242 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "last_update": "2023-05-31T19:56:10.274572Z" 2023-05-31T21:35:43.243 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2023-05-31T21:35:43.243 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2023-05-31T21:35:43.243 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "marker": "1_1685568089.268814_960.1", 2023-05-31T21:35:43.243 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "last_update": "2023-05-31T21:21:29.268814Z" 2023-05-31T21:35:43.243 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2023-05-31T21:35:43.244 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: ] 2023-05-31T21:35:43.244 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: master meta markers={0: '1_1685566501.757183_260.1', 1: '1_1685566501.735717_251.1', 2: '1_1685562970.274572_93.1', 3: '1_1685568089.268814_960.1'} 2023-05-31T21:35:43.244 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: INFO: starting meta checkpoint for zone=a2 2023-05-31T21:35:43.244 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-31T21:35:43.245 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-31T21:35:43.245 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-05-31T21:35:43.245 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: ignoring --setuser ceph since I am not root 2023-05-31T21:35:43.245 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-05-31T21:35:43.246 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: 2023-05-31T21:21:36.261+0000 7f899ad53780 20 RGW-SYNC:meta: read sync status 2023-05-31T21:35:43.246 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: 2023-05-31T21:21:36.263+0000 7f899ad53780 20 RGW-SYNC:meta: read sync status 2023-05-31T21:35:43.246 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2023-05-31T21:35:43.246 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "sync_status": { 2023-05-31T21:35:43.247 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "info": { 2023-05-31T21:35:43.247 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "status": "sync", 2023-05-31T21:35:43.247 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "num_shards": 4, 2023-05-31T21:35:43.247 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "period": "2e6123d2-d695-4350-9a01-61a455bc782b", 2023-05-31T21:35:43.247 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "realm_epoch": 2 2023-05-31T21:35:43.248 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2023-05-31T21:35:43.248 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "markers": [ 2023-05-31T21:35:43.248 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2023-05-31T21:35:43.248 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "key": 0, 2023-05-31T21:35:43.249 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "val": { 2023-05-31T21:35:43.249 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "state": 1, 2023-05-31T21:35:43.249 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "marker": "1_1685566501.757183_260.1", 2023-05-31T21:35:43.249 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "next_step_marker": "", 2023-05-31T21:35:43.250 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "total_entries": 1, 2023-05-31T21:35:43.250 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "pos": 0, 2023-05-31T21:35:43.250 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "timestamp": "2023-05-31T20:55:01.757183Z", 2023-05-31T21:35:43.250 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "realm_epoch": 2 2023-05-31T21:35:43.250 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2023-05-31T21:35:43.251 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2023-05-31T21:35:43.251 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2023-05-31T21:35:43.251 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "key": 1, 2023-05-31T21:35:43.251 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "val": { 2023-05-31T21:35:43.252 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "state": 1, 2023-05-31T21:35:43.252 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "marker": "1_1685566501.735717_251.1", 2023-05-31T21:35:43.252 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "next_step_marker": "", 2023-05-31T21:35:43.252 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "total_entries": 0, 2023-05-31T21:35:43.253 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "pos": 0, 2023-05-31T21:35:43.253 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "timestamp": "2023-05-31T20:55:01.735717Z", 2023-05-31T21:35:43.253 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "realm_epoch": 2 2023-05-31T21:35:43.253 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2023-05-31T21:35:43.253 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2023-05-31T21:35:43.254 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2023-05-31T21:35:43.254 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "key": 2, 2023-05-31T21:35:43.254 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "val": { 2023-05-31T21:35:43.254 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "state": 1, 2023-05-31T21:35:43.255 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "marker": "1_1685562970.274572_93.1", 2023-05-31T21:35:43.255 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "next_step_marker": "", 2023-05-31T21:35:43.255 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "total_entries": 0, 2023-05-31T21:35:43.255 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "pos": 0, 2023-05-31T21:35:43.256 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "timestamp": "2023-05-31T19:56:10.274572Z", 2023-05-31T21:35:43.256 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "realm_epoch": 2 2023-05-31T21:35:43.256 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2023-05-31T21:35:43.256 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2023-05-31T21:35:43.256 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2023-05-31T21:35:43.257 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "key": 3, 2023-05-31T21:35:43.257 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "val": { 2023-05-31T21:35:43.257 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "state": 1, 2023-05-31T21:35:43.257 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "marker": "1_1685568089.268814_960.1", 2023-05-31T21:35:43.258 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "next_step_marker": "", 2023-05-31T21:35:43.258 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "total_entries": 0, 2023-05-31T21:35:43.258 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "pos": 0, 2023-05-31T21:35:43.258 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "timestamp": "2023-05-31T21:21:29.268814Z", 2023-05-31T21:35:43.258 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "realm_epoch": 2 2023-05-31T21:35:43.259 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2023-05-31T21:35:43.259 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2023-05-31T21:35:43.259 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: ] 2023-05-31T21:35:43.259 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2023-05-31T21:35:43.260 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "full_sync": { 2023-05-31T21:35:43.260 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "total": 1, 2023-05-31T21:35:43.260 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "complete": 1 2023-05-31T21:35:43.260 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2023-05-31T21:35:43.261 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "current_time": "2023-05-31T21:21:36Z" 2023-05-31T21:35:43.261 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2023-05-31T21:35:43.261 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2023-05-31T21:35:43.261 INFO:tasks.rgw_multisite_tests: "sync_status": { 2023-05-31T21:35:43.262 INFO:tasks.rgw_multisite_tests: "info": { 2023-05-31T21:35:43.262 INFO:tasks.rgw_multisite_tests: "status": "sync", 2023-05-31T21:35:43.262 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2023-05-31T21:35:43.262 INFO:tasks.rgw_multisite_tests: "period": "2e6123d2-d695-4350-9a01-61a455bc782b", 2023-05-31T21:35:43.262 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-05-31T21:35:43.263 INFO:tasks.rgw_multisite_tests: }, 2023-05-31T21:35:43.263 INFO:tasks.rgw_multisite_tests: "markers": [ 2023-05-31T21:35:43.263 INFO:tasks.rgw_multisite_tests: { 2023-05-31T21:35:43.263 INFO:tasks.rgw_multisite_tests: "key": 0, 2023-05-31T21:35:43.264 INFO:tasks.rgw_multisite_tests: "val": { 2023-05-31T21:35:43.264 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-05-31T21:35:43.264 INFO:tasks.rgw_multisite_tests: "marker": "1_1685566501.757183_260.1", 2023-05-31T21:35:43.264 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-05-31T21:35:43.265 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2023-05-31T21:35:43.265 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-05-31T21:35:43.265 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-05-31T20:55:01.757183Z", 2023-05-31T21:35:43.265 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-05-31T21:35:43.265 INFO:tasks.rgw_multisite_tests: } 2023-05-31T21:35:43.266 INFO:tasks.rgw_multisite_tests: }, 2023-05-31T21:35:43.266 INFO:tasks.rgw_multisite_tests: { 2023-05-31T21:35:43.266 INFO:tasks.rgw_multisite_tests: "key": 1, 2023-05-31T21:35:43.266 INFO:tasks.rgw_multisite_tests: "val": { 2023-05-31T21:35:43.267 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-05-31T21:35:43.267 INFO:tasks.rgw_multisite_tests: "marker": "1_1685566501.735717_251.1", 2023-05-31T21:35:43.267 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-05-31T21:35:43.267 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-05-31T21:35:43.268 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-05-31T21:35:43.268 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-05-31T20:55:01.735717Z", 2023-05-31T21:35:43.268 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-05-31T21:35:43.268 INFO:tasks.rgw_multisite_tests: } 2023-05-31T21:35:43.268 INFO:tasks.rgw_multisite_tests: }, 2023-05-31T21:35:43.269 INFO:tasks.rgw_multisite_tests: { 2023-05-31T21:35:43.269 INFO:tasks.rgw_multisite_tests: "key": 2, 2023-05-31T21:35:43.269 INFO:tasks.rgw_multisite_tests: "val": { 2023-05-31T21:35:43.269 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-05-31T21:35:43.270 INFO:tasks.rgw_multisite_tests: "marker": "1_1685562970.274572_93.1", 2023-05-31T21:35:43.270 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-05-31T21:35:43.270 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-05-31T21:35:43.270 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-05-31T21:35:43.271 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-05-31T19:56:10.274572Z", 2023-05-31T21:35:43.271 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-05-31T21:35:43.271 INFO:tasks.rgw_multisite_tests: } 2023-05-31T21:35:43.271 INFO:tasks.rgw_multisite_tests: }, 2023-05-31T21:35:43.271 INFO:tasks.rgw_multisite_tests: { 2023-05-31T21:35:43.272 INFO:tasks.rgw_multisite_tests: "key": 3, 2023-05-31T21:35:43.272 INFO:tasks.rgw_multisite_tests: "val": { 2023-05-31T21:35:43.272 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-05-31T21:35:43.272 INFO:tasks.rgw_multisite_tests: "marker": "1_1685568089.268814_960.1", 2023-05-31T21:35:43.273 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-05-31T21:35:43.273 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-05-31T21:35:43.273 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-05-31T21:35:43.273 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-05-31T21:21:29.268814Z", 2023-05-31T21:35:43.274 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-05-31T21:35:43.274 INFO:tasks.rgw_multisite_tests: } 2023-05-31T21:35:43.274 INFO:tasks.rgw_multisite_tests: } 2023-05-31T21:35:43.274 INFO:tasks.rgw_multisite_tests: ] 2023-05-31T21:35:43.274 INFO:tasks.rgw_multisite_tests: }, 2023-05-31T21:35:43.275 INFO:tasks.rgw_multisite_tests: "full_sync": { 2023-05-31T21:35:43.275 INFO:tasks.rgw_multisite_tests: "total": 1, 2023-05-31T21:35:43.275 INFO:tasks.rgw_multisite_tests: "complete": 1 2023-05-31T21:35:43.275 INFO:tasks.rgw_multisite_tests: }, 2023-05-31T21:35:43.276 INFO:tasks.rgw_multisite_tests: "current_time": "2023-05-31T21:21:36Z" 2023-05-31T21:35:43.276 INFO:tasks.rgw_multisite_tests:} 2023-05-31T21:35:43.276 INFO:tasks.rgw_multisite_tests: 2023-05-31T21:35:43.276 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1685566501.757183_260.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-05-31T20:55:01.757183Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1685566501.735717_251.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-05-31T20:55:01.735717Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1685562970.274572_93.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-05-31T19:56:10.274572Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1685568089.268814_960.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-05-31T21:21:29.268814Z', 'realm_epoch': 2}}] 2023-05-31T21:35:43.277 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1685566501.757183_260.1', 1: '1_1685566501.735717_251.1', 2: '1_1685562970.274572_93.1', 3: '1_1685568089.268814_960.1'} 2023-05-31T21:35:43.277 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1685566501.757183_260.1', 1: '1_1685566501.735717_251.1', 2: '1_1685562970.274572_93.1', 3: '1_1685568089.268814_960.1'} 2023-05-31T21:35:43.277 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: INFO: finish meta checkpoint for zone=a2 2023-05-31T21:35:43.277 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['datalog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-31T21:35:43.277 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'datalog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-31T21:35:43.278 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 datalog status --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-05-31T21:35:43.278 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: ignoring --setuser ceph since I am not root 2023-05-31T21:35:43.278 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-05-31T21:35:43.278 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: [ 2023-05-31T21:35:43.279 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2023-05-31T21:35:43.279 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "marker": "00000000000000000000:00000000000000044955", 2023-05-31T21:35:43.279 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "last_update": "2023-05-31T21:21:35.807414Z" 2023-05-31T21:35:43.279 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2023-05-31T21:35:43.280 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2023-05-31T21:35:43.280 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "marker": "00000000000000000000:00000000000000031499", 2023-05-31T21:35:43.280 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "last_update": "2023-05-31T21:12:40.667485Z" 2023-05-31T21:35:43.280 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2023-05-31T21:35:43.281 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2023-05-31T21:35:43.281 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "marker": "00000000000000000000:00000000000000051111", 2023-05-31T21:35:43.281 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "last_update": "2023-05-31T21:21:35.866368Z" 2023-05-31T21:35:43.281 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2023-05-31T21:35:43.281 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2023-05-31T21:35:43.282 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "marker": "00000000000000000000:00000000000000044038", 2023-05-31T21:35:43.282 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "last_update": "2023-05-31T21:21:35.912562Z" 2023-05-31T21:35:43.282 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2023-05-31T21:35:43.282 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: ] 2023-05-31T21:35:43.283 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: data markers for zone=a1 markers={0: '00000000000000000000:00000000000000044955', 1: '00000000000000000000:00000000000000031499', 2: '00000000000000000000:00000000000000051111', 3: '00000000000000000000:00000000000000044038'} 2023-05-31T21:35:43.283 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: INFO: starting data checkpoint for target_zone=a2 source_zone=a1 2023-05-31T21:35:43.283 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-31T21:35:43.283 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-31T21:35:43.284 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-05-31T21:35:43.284 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: ignoring --setuser ceph since I am not root 2023-05-31T21:35:43.284 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-05-31T21:35:43.284 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2023-05-31T21:35:43.284 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "sync_status": { 2023-05-31T21:35:43.285 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "info": { 2023-05-31T21:35:43.285 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "status": "sync", 2023-05-31T21:35:43.285 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "num_shards": 4, 2023-05-31T21:35:43.285 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "instance_id": 452136274422638824 2023-05-31T21:35:43.286 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2023-05-31T21:35:43.286 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "markers": [ 2023-05-31T21:35:43.286 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2023-05-31T21:35:43.286 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "key": 0, 2023-05-31T21:35:43.287 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "val": { 2023-05-31T21:35:43.287 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "status": "incremental-sync", 2023-05-31T21:35:43.287 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "marker": "00000000000000000000:00000000000000044815", 2023-05-31T21:35:43.287 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "next_step_marker": "", 2023-05-31T21:35:43.287 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "total_entries": 0, 2023-05-31T21:35:43.288 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "pos": 0, 2023-05-31T21:35:43.288 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "timestamp": "2023-05-31T21:21:05.078544Z" 2023-05-31T21:35:43.288 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2023-05-31T21:35:43.289 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2023-05-31T21:35:43.289 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2023-05-31T21:35:43.289 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "key": 1, 2023-05-31T21:35:43.289 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "val": { 2023-05-31T21:35:43.289 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "status": "incremental-sync", 2023-05-31T21:35:43.290 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "marker": "00000000000000000000:00000000000000031499", 2023-05-31T21:35:43.290 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "next_step_marker": "", 2023-05-31T21:35:43.290 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "total_entries": 0, 2023-05-31T21:35:43.290 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "pos": 0, 2023-05-31T21:35:43.291 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "timestamp": "2023-05-31T21:12:40.667485Z" 2023-05-31T21:35:43.291 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2023-05-31T21:35:43.291 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2023-05-31T21:35:43.291 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2023-05-31T21:35:43.292 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "key": 2, 2023-05-31T21:35:43.292 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "val": { 2023-05-31T21:35:43.292 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "status": "incremental-sync", 2023-05-31T21:35:43.292 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "marker": "00000000000000000000:00000000000000050831", 2023-05-31T21:35:43.293 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "next_step_marker": "", 2023-05-31T21:35:43.293 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "total_entries": 55, 2023-05-31T21:35:43.293 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "pos": 0, 2023-05-31T21:35:43.293 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "timestamp": "2023-05-31T21:19:03.876722Z" 2023-05-31T21:35:43.293 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2023-05-31T21:35:43.294 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2023-05-31T21:35:43.294 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2023-05-31T21:35:43.294 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "key": 3, 2023-05-31T21:35:43.294 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "val": { 2023-05-31T21:35:43.295 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "status": "incremental-sync", 2023-05-31T21:35:43.295 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "marker": "00000000000000000000:00000000000000043896", 2023-05-31T21:35:43.295 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "next_step_marker": "", 2023-05-31T21:35:43.295 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "total_entries": 9, 2023-05-31T21:35:43.296 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "pos": 0, 2023-05-31T21:35:43.296 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "timestamp": "2023-05-31T21:12:28.529956Z" 2023-05-31T21:35:43.296 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2023-05-31T21:35:43.296 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2023-05-31T21:35:43.296 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: ] 2023-05-31T21:35:43.297 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2023-05-31T21:35:43.297 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "full_sync": { 2023-05-31T21:35:43.297 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "total": 64, 2023-05-31T21:35:43.297 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "complete": 64 2023-05-31T21:35:43.298 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2023-05-31T21:35:43.298 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "current_time": "2023-05-31T21:21:36Z" 2023-05-31T21:35:43.298 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2023-05-31T21:35:43.298 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current data sync status={ 2023-05-31T21:35:43.299 INFO:tasks.rgw_multisite_tests: "sync_status": { 2023-05-31T21:35:43.299 INFO:tasks.rgw_multisite_tests: "info": { 2023-05-31T21:35:43.299 INFO:tasks.rgw_multisite_tests: "status": "sync", 2023-05-31T21:35:43.299 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2023-05-31T21:35:43.300 INFO:tasks.rgw_multisite_tests: "instance_id": 452136274422638824 2023-05-31T21:35:43.300 INFO:tasks.rgw_multisite_tests: }, 2023-05-31T21:35:43.300 INFO:tasks.rgw_multisite_tests: "markers": [ 2023-05-31T21:35:43.300 INFO:tasks.rgw_multisite_tests: { 2023-05-31T21:35:43.300 INFO:tasks.rgw_multisite_tests: "key": 0, 2023-05-31T21:35:43.301 INFO:tasks.rgw_multisite_tests: "val": { 2023-05-31T21:35:43.301 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2023-05-31T21:35:43.301 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000044815", 2023-05-31T21:35:43.301 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-05-31T21:35:43.302 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-05-31T21:35:43.302 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-05-31T21:35:43.302 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-05-31T21:21:05.078544Z" 2023-05-31T21:35:43.302 INFO:tasks.rgw_multisite_tests: } 2023-05-31T21:35:43.303 INFO:tasks.rgw_multisite_tests: }, 2023-05-31T21:35:43.303 INFO:tasks.rgw_multisite_tests: { 2023-05-31T21:35:43.303 INFO:tasks.rgw_multisite_tests: "key": 1, 2023-05-31T21:35:43.303 INFO:tasks.rgw_multisite_tests: "val": { 2023-05-31T21:35:43.303 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2023-05-31T21:35:43.304 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000031499", 2023-05-31T21:35:43.304 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-05-31T21:35:43.304 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-05-31T21:35:43.304 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-05-31T21:35:43.305 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-05-31T21:12:40.667485Z" 2023-05-31T21:35:43.305 INFO:tasks.rgw_multisite_tests: } 2023-05-31T21:35:43.305 INFO:tasks.rgw_multisite_tests: }, 2023-05-31T21:35:43.305 INFO:tasks.rgw_multisite_tests: { 2023-05-31T21:35:43.306 INFO:tasks.rgw_multisite_tests: "key": 2, 2023-05-31T21:35:43.306 INFO:tasks.rgw_multisite_tests: "val": { 2023-05-31T21:35:43.306 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2023-05-31T21:35:43.306 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000050831", 2023-05-31T21:35:43.306 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-05-31T21:35:43.307 INFO:tasks.rgw_multisite_tests: "total_entries": 55, 2023-05-31T21:35:43.307 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-05-31T21:35:43.307 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-05-31T21:19:03.876722Z" 2023-05-31T21:35:43.307 INFO:tasks.rgw_multisite_tests: } 2023-05-31T21:35:43.308 INFO:tasks.rgw_multisite_tests: }, 2023-05-31T21:35:43.308 INFO:tasks.rgw_multisite_tests: { 2023-05-31T21:35:43.308 INFO:tasks.rgw_multisite_tests: "key": 3, 2023-05-31T21:35:43.308 INFO:tasks.rgw_multisite_tests: "val": { 2023-05-31T21:35:43.309 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2023-05-31T21:35:43.309 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000043896", 2023-05-31T21:35:43.309 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-05-31T21:35:43.309 INFO:tasks.rgw_multisite_tests: "total_entries": 9, 2023-05-31T21:35:43.309 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-05-31T21:35:43.310 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-05-31T21:12:28.529956Z" 2023-05-31T21:35:43.310 INFO:tasks.rgw_multisite_tests: } 2023-05-31T21:35:43.310 INFO:tasks.rgw_multisite_tests: } 2023-05-31T21:35:43.310 INFO:tasks.rgw_multisite_tests: ] 2023-05-31T21:35:43.311 INFO:tasks.rgw_multisite_tests: }, 2023-05-31T21:35:43.311 INFO:tasks.rgw_multisite_tests: "full_sync": { 2023-05-31T21:35:43.311 INFO:tasks.rgw_multisite_tests: "total": 64, 2023-05-31T21:35:43.311 INFO:tasks.rgw_multisite_tests: "complete": 64 2023-05-31T21:35:43.312 INFO:tasks.rgw_multisite_tests: }, 2023-05-31T21:35:43.312 INFO:tasks.rgw_multisite_tests: "current_time": "2023-05-31T21:21:36Z" 2023-05-31T21:35:43.312 INFO:tasks.rgw_multisite_tests:} 2023-05-31T21:35:43.312 INFO:tasks.rgw_multisite_tests: 2023-05-31T21:35:43.312 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000044815', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-05-31T21:21:05.078544Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000031499', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-05-31T21:12:40.667485Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000050831', 'next_step_marker': '', 'total_entries': 55, 'pos': 0, 'timestamp': '2023-05-31T21:19:03.876722Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000043896', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2023-05-31T21:12:28.529956Z'}}] 2023-05-31T21:35:43.313 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '00000000000000000000:00000000000000044955', 1: '00000000000000000000:00000000000000031499', 2: '00000000000000000000:00000000000000051111', 3: '00000000000000000000:00000000000000044038'} 2023-05-31T21:35:43.313 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '00000000000000000000:00000000000000044815', 1: '00000000000000000000:00000000000000031499', 2: '00000000000000000000:00000000000000050831', 3: '00000000000000000000:00000000000000043896'} 2023-05-31T21:35:43.313 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: data of zone a2 behind zone a1: shard=0 master=00000000000000000000:00000000000000044955 target=00000000000000000000:00000000000000044815, shard=2 master=00000000000000000000:00000000000000051111 target=00000000000000000000:00000000000000050831, shard=3 master=00000000000000000000:00000000000000044038 target=00000000000000000000:00000000000000043896 2023-05-31T21:35:43.313 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-31T21:35:43.314 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-31T21:35:43.314 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-05-31T21:35:43.314 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: ignoring --setuser ceph since I am not root 2023-05-31T21:35:43.314 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-05-31T21:35:43.315 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2023-05-31T21:35:43.315 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "sync_status": { 2023-05-31T21:35:43.315 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "info": { 2023-05-31T21:35:43.315 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "status": "sync", 2023-05-31T21:35:43.315 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "num_shards": 4, 2023-05-31T21:35:43.316 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "instance_id": 452136274422638824 2023-05-31T21:35:43.316 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2023-05-31T21:35:43.316 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "markers": [ 2023-05-31T21:35:43.316 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2023-05-31T21:35:43.317 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "key": 0, 2023-05-31T21:35:43.317 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "val": { 2023-05-31T21:35:43.317 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "status": "incremental-sync", 2023-05-31T21:35:43.317 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "marker": "00000000000000000000:00000000000000044815", 2023-05-31T21:35:43.318 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "next_step_marker": "", 2023-05-31T21:35:43.318 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "total_entries": 0, 2023-05-31T21:35:43.318 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "pos": 0, 2023-05-31T21:35:43.318 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "timestamp": "2023-05-31T21:21:05.078544Z" 2023-05-31T21:35:43.319 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2023-05-31T21:35:43.319 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2023-05-31T21:35:43.319 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2023-05-31T21:35:43.319 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "key": 1, 2023-05-31T21:35:43.319 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "val": { 2023-05-31T21:35:43.320 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "status": "incremental-sync", 2023-05-31T21:35:43.320 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "marker": "00000000000000000000:00000000000000031499", 2023-05-31T21:35:43.320 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "next_step_marker": "", 2023-05-31T21:35:43.320 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "total_entries": 0, 2023-05-31T21:35:43.321 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "pos": 0, 2023-05-31T21:35:43.321 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "timestamp": "2023-05-31T21:12:40.667485Z" 2023-05-31T21:35:43.321 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2023-05-31T21:35:43.321 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2023-05-31T21:35:43.321 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2023-05-31T21:35:43.322 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "key": 2, 2023-05-31T21:35:43.322 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "val": { 2023-05-31T21:35:43.322 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "status": "incremental-sync", 2023-05-31T21:35:43.322 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "marker": "00000000000000000000:00000000000000050831", 2023-05-31T21:35:43.323 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "next_step_marker": "", 2023-05-31T21:35:43.323 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "total_entries": 55, 2023-05-31T21:35:43.323 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "pos": 0, 2023-05-31T21:35:43.323 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "timestamp": "2023-05-31T21:19:03.876722Z" 2023-05-31T21:35:43.324 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2023-05-31T21:35:43.324 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2023-05-31T21:35:43.324 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2023-05-31T21:35:43.324 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "key": 3, 2023-05-31T21:35:43.324 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "val": { 2023-05-31T21:35:43.325 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "status": "incremental-sync", 2023-05-31T21:35:43.325 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "marker": "00000000000000000000:00000000000000043896", 2023-05-31T21:35:43.325 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "next_step_marker": "", 2023-05-31T21:35:43.325 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "total_entries": 9, 2023-05-31T21:35:43.326 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "pos": 0, 2023-05-31T21:35:43.326 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "timestamp": "2023-05-31T21:12:28.529956Z" 2023-05-31T21:35:43.326 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2023-05-31T21:35:43.326 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2023-05-31T21:35:43.327 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: ] 2023-05-31T21:35:43.327 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2023-05-31T21:35:43.327 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "full_sync": { 2023-05-31T21:35:43.327 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "total": 64, 2023-05-31T21:35:43.328 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "complete": 64 2023-05-31T21:35:43.328 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2023-05-31T21:35:43.328 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "current_time": "2023-05-31T21:21:41Z" 2023-05-31T21:35:43.328 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2023-05-31T21:35:43.328 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current data sync status={ 2023-05-31T21:35:43.329 INFO:tasks.rgw_multisite_tests: "sync_status": { 2023-05-31T21:35:43.329 INFO:tasks.rgw_multisite_tests: "info": { 2023-05-31T21:35:43.329 INFO:tasks.rgw_multisite_tests: "status": "sync", 2023-05-31T21:35:43.329 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2023-05-31T21:35:43.330 INFO:tasks.rgw_multisite_tests: "instance_id": 452136274422638824 2023-05-31T21:35:43.330 INFO:tasks.rgw_multisite_tests: }, 2023-05-31T21:35:43.330 INFO:tasks.rgw_multisite_tests: "markers": [ 2023-05-31T21:35:43.330 INFO:tasks.rgw_multisite_tests: { 2023-05-31T21:35:43.330 INFO:tasks.rgw_multisite_tests: "key": 0, 2023-05-31T21:35:43.331 INFO:tasks.rgw_multisite_tests: "val": { 2023-05-31T21:35:43.331 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2023-05-31T21:35:43.331 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000044815", 2023-05-31T21:35:43.331 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-05-31T21:35:43.332 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-05-31T21:35:43.332 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-05-31T21:35:43.332 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-05-31T21:21:05.078544Z" 2023-05-31T21:35:43.332 INFO:tasks.rgw_multisite_tests: } 2023-05-31T21:35:43.333 INFO:tasks.rgw_multisite_tests: }, 2023-05-31T21:35:43.333 INFO:tasks.rgw_multisite_tests: { 2023-05-31T21:35:43.333 INFO:tasks.rgw_multisite_tests: "key": 1, 2023-05-31T21:35:43.333 INFO:tasks.rgw_multisite_tests: "val": { 2023-05-31T21:35:43.333 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2023-05-31T21:35:43.334 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000031499", 2023-05-31T21:35:43.334 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-05-31T21:35:43.334 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-05-31T21:35:43.334 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-05-31T21:35:43.335 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-05-31T21:12:40.667485Z" 2023-05-31T21:35:43.335 INFO:tasks.rgw_multisite_tests: } 2023-05-31T21:35:43.335 INFO:tasks.rgw_multisite_tests: }, 2023-05-31T21:35:43.335 INFO:tasks.rgw_multisite_tests: { 2023-05-31T21:35:43.336 INFO:tasks.rgw_multisite_tests: "key": 2, 2023-05-31T21:35:43.336 INFO:tasks.rgw_multisite_tests: "val": { 2023-05-31T21:35:43.336 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2023-05-31T21:35:43.336 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000050831", 2023-05-31T21:35:43.336 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-05-31T21:35:43.337 INFO:tasks.rgw_multisite_tests: "total_entries": 55, 2023-05-31T21:35:43.337 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-05-31T21:35:43.337 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-05-31T21:19:03.876722Z" 2023-05-31T21:35:43.337 INFO:tasks.rgw_multisite_tests: } 2023-05-31T21:35:43.338 INFO:tasks.rgw_multisite_tests: }, 2023-05-31T21:35:43.338 INFO:tasks.rgw_multisite_tests: { 2023-05-31T21:35:43.338 INFO:tasks.rgw_multisite_tests: "key": 3, 2023-05-31T21:35:43.338 INFO:tasks.rgw_multisite_tests: "val": { 2023-05-31T21:35:43.339 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2023-05-31T21:35:43.339 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000043896", 2023-05-31T21:35:43.339 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-05-31T21:35:43.339 INFO:tasks.rgw_multisite_tests: "total_entries": 9, 2023-05-31T21:35:43.339 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-05-31T21:35:43.340 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-05-31T21:12:28.529956Z" 2023-05-31T21:35:43.340 INFO:tasks.rgw_multisite_tests: } 2023-05-31T21:35:43.340 INFO:tasks.rgw_multisite_tests: } 2023-05-31T21:35:43.340 INFO:tasks.rgw_multisite_tests: ] 2023-05-31T21:35:43.341 INFO:tasks.rgw_multisite_tests: }, 2023-05-31T21:35:43.341 INFO:tasks.rgw_multisite_tests: "full_sync": { 2023-05-31T21:35:43.341 INFO:tasks.rgw_multisite_tests: "total": 64, 2023-05-31T21:35:43.341 INFO:tasks.rgw_multisite_tests: "complete": 64 2023-05-31T21:35:43.341 INFO:tasks.rgw_multisite_tests: }, 2023-05-31T21:35:43.342 INFO:tasks.rgw_multisite_tests: "current_time": "2023-05-31T21:21:41Z" 2023-05-31T21:35:43.342 INFO:tasks.rgw_multisite_tests:} 2023-05-31T21:35:43.342 INFO:tasks.rgw_multisite_tests: 2023-05-31T21:35:43.342 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000044815', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-05-31T21:21:05.078544Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000031499', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-05-31T21:12:40.667485Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000050831', 'next_step_marker': '', 'total_entries': 55, 'pos': 0, 'timestamp': '2023-05-31T21:19:03.876722Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000043896', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2023-05-31T21:12:28.529956Z'}}] 2023-05-31T21:35:43.343 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '00000000000000000000:00000000000000044955', 1: '00000000000000000000:00000000000000031499', 2: '00000000000000000000:00000000000000051111', 3: '00000000000000000000:00000000000000044038'} 2023-05-31T21:35:43.343 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '00000000000000000000:00000000000000044815', 1: '00000000000000000000:00000000000000031499', 2: '00000000000000000000:00000000000000050831', 3: '00000000000000000000:00000000000000043896'} 2023-05-31T21:35:43.343 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: data of zone a2 behind zone a1: shard=0 master=00000000000000000000:00000000000000044955 target=00000000000000000000:00000000000000044815, shard=2 master=00000000000000000000:00000000000000051111 target=00000000000000000000:00000000000000050831, shard=3 master=00000000000000000000:00000000000000044038 target=00000000000000000000:00000000000000043896 2023-05-31T21:35:43.343 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-31T21:35:43.344 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-31T21:35:43.344 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-05-31T21:35:43.344 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: ignoring --setuser ceph since I am not root 2023-05-31T21:35:43.344 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-05-31T21:35:43.345 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2023-05-31T21:35:43.345 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "sync_status": { 2023-05-31T21:35:43.345 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "info": { 2023-05-31T21:35:43.345 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "status": "sync", 2023-05-31T21:35:43.345 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "num_shards": 4, 2023-05-31T21:35:43.346 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "instance_id": 452136274422638824 2023-05-31T21:35:43.346 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2023-05-31T21:35:43.346 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "markers": [ 2023-05-31T21:35:43.346 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2023-05-31T21:35:43.347 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "key": 0, 2023-05-31T21:35:43.347 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "val": { 2023-05-31T21:35:43.347 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "status": "incremental-sync", 2023-05-31T21:35:43.347 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "marker": "00000000000000000000:00000000000000044815", 2023-05-31T21:35:43.348 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "next_step_marker": "", 2023-05-31T21:35:43.348 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "total_entries": 0, 2023-05-31T21:35:43.348 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "pos": 0, 2023-05-31T21:35:43.348 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "timestamp": "2023-05-31T21:21:05.078544Z" 2023-05-31T21:35:43.348 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2023-05-31T21:35:43.349 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2023-05-31T21:35:43.349 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2023-05-31T21:35:43.349 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "key": 1, 2023-05-31T21:35:43.349 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "val": { 2023-05-31T21:35:43.350 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "status": "incremental-sync", 2023-05-31T21:35:43.350 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "marker": "00000000000000000000:00000000000000031499", 2023-05-31T21:35:43.350 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "next_step_marker": "", 2023-05-31T21:35:43.350 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "total_entries": 0, 2023-05-31T21:35:43.351 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "pos": 0, 2023-05-31T21:35:43.351 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "timestamp": "2023-05-31T21:12:40.667485Z" 2023-05-31T21:35:43.351 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2023-05-31T21:35:43.351 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2023-05-31T21:35:43.351 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2023-05-31T21:35:43.352 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "key": 2, 2023-05-31T21:35:43.352 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "val": { 2023-05-31T21:35:43.352 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "status": "incremental-sync", 2023-05-31T21:35:43.352 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "marker": "00000000000000000000:00000000000000050831", 2023-05-31T21:35:43.353 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "next_step_marker": "", 2023-05-31T21:35:43.353 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "total_entries": 55, 2023-05-31T21:35:43.353 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "pos": 0, 2023-05-31T21:35:43.353 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "timestamp": "2023-05-31T21:19:03.876722Z" 2023-05-31T21:35:43.354 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2023-05-31T21:35:43.354 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2023-05-31T21:35:43.354 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2023-05-31T21:35:43.354 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "key": 3, 2023-05-31T21:35:43.354 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "val": { 2023-05-31T21:35:43.355 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "status": "incremental-sync", 2023-05-31T21:35:43.355 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "marker": "00000000000000000000:00000000000000043896", 2023-05-31T21:35:43.355 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "next_step_marker": "", 2023-05-31T21:35:43.355 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "total_entries": 9, 2023-05-31T21:35:43.356 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "pos": 0, 2023-05-31T21:35:43.356 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "timestamp": "2023-05-31T21:12:28.529956Z" 2023-05-31T21:35:43.356 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2023-05-31T21:35:43.356 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2023-05-31T21:35:43.357 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: ] 2023-05-31T21:35:43.357 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2023-05-31T21:35:43.357 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "full_sync": { 2023-05-31T21:35:43.357 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "total": 64, 2023-05-31T21:35:43.357 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "complete": 64 2023-05-31T21:35:43.358 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2023-05-31T21:35:43.358 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "current_time": "2023-05-31T21:21:46Z" 2023-05-31T21:35:43.358 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2023-05-31T21:35:43.358 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current data sync status={ 2023-05-31T21:35:43.359 INFO:tasks.rgw_multisite_tests: "sync_status": { 2023-05-31T21:35:43.359 INFO:tasks.rgw_multisite_tests: "info": { 2023-05-31T21:35:43.359 INFO:tasks.rgw_multisite_tests: "status": "sync", 2023-05-31T21:35:43.359 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2023-05-31T21:35:43.359 INFO:tasks.rgw_multisite_tests: "instance_id": 452136274422638824 2023-05-31T21:35:43.360 INFO:tasks.rgw_multisite_tests: }, 2023-05-31T21:35:43.360 INFO:tasks.rgw_multisite_tests: "markers": [ 2023-05-31T21:35:43.360 INFO:tasks.rgw_multisite_tests: { 2023-05-31T21:35:43.360 INFO:tasks.rgw_multisite_tests: "key": 0, 2023-05-31T21:35:43.361 INFO:tasks.rgw_multisite_tests: "val": { 2023-05-31T21:35:43.361 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2023-05-31T21:35:43.361 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000044815", 2023-05-31T21:35:43.361 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-05-31T21:35:43.362 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-05-31T21:35:43.362 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-05-31T21:35:43.362 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-05-31T21:21:05.078544Z" 2023-05-31T21:35:43.362 INFO:tasks.rgw_multisite_tests: } 2023-05-31T21:35:43.363 INFO:tasks.rgw_multisite_tests: }, 2023-05-31T21:35:43.363 INFO:tasks.rgw_multisite_tests: { 2023-05-31T21:35:43.363 INFO:tasks.rgw_multisite_tests: "key": 1, 2023-05-31T21:35:43.363 INFO:tasks.rgw_multisite_tests: "val": { 2023-05-31T21:35:43.363 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2023-05-31T21:35:43.364 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000031499", 2023-05-31T21:35:43.364 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-05-31T21:35:43.364 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-05-31T21:35:43.364 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-05-31T21:35:43.365 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-05-31T21:12:40.667485Z" 2023-05-31T21:35:43.365 INFO:tasks.rgw_multisite_tests: } 2023-05-31T21:35:43.365 INFO:tasks.rgw_multisite_tests: }, 2023-05-31T21:35:43.365 INFO:tasks.rgw_multisite_tests: { 2023-05-31T21:35:43.366 INFO:tasks.rgw_multisite_tests: "key": 2, 2023-05-31T21:35:43.366 INFO:tasks.rgw_multisite_tests: "val": { 2023-05-31T21:35:43.366 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2023-05-31T21:35:43.366 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000050831", 2023-05-31T21:35:43.367 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-05-31T21:35:43.367 INFO:tasks.rgw_multisite_tests: "total_entries": 55, 2023-05-31T21:35:43.367 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-05-31T21:35:43.367 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-05-31T21:19:03.876722Z" 2023-05-31T21:35:43.367 INFO:tasks.rgw_multisite_tests: } 2023-05-31T21:35:43.368 INFO:tasks.rgw_multisite_tests: }, 2023-05-31T21:35:43.368 INFO:tasks.rgw_multisite_tests: { 2023-05-31T21:35:43.368 INFO:tasks.rgw_multisite_tests: "key": 3, 2023-05-31T21:35:43.368 INFO:tasks.rgw_multisite_tests: "val": { 2023-05-31T21:35:43.369 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2023-05-31T21:35:43.369 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000043896", 2023-05-31T21:35:43.369 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-05-31T21:35:43.369 INFO:tasks.rgw_multisite_tests: "total_entries": 9, 2023-05-31T21:35:43.370 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-05-31T21:35:43.370 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-05-31T21:12:28.529956Z" 2023-05-31T21:35:43.370 INFO:tasks.rgw_multisite_tests: } 2023-05-31T21:35:43.370 INFO:tasks.rgw_multisite_tests: } 2023-05-31T21:35:43.371 INFO:tasks.rgw_multisite_tests: ] 2023-05-31T21:35:43.371 INFO:tasks.rgw_multisite_tests: }, 2023-05-31T21:35:43.371 INFO:tasks.rgw_multisite_tests: "full_sync": { 2023-05-31T21:35:43.371 INFO:tasks.rgw_multisite_tests: "total": 64, 2023-05-31T21:35:43.371 INFO:tasks.rgw_multisite_tests: "complete": 64 2023-05-31T21:35:43.372 INFO:tasks.rgw_multisite_tests: }, 2023-05-31T21:35:43.372 INFO:tasks.rgw_multisite_tests: "current_time": "2023-05-31T21:21:46Z" 2023-05-31T21:35:43.372 INFO:tasks.rgw_multisite_tests:} 2023-05-31T21:35:43.372 INFO:tasks.rgw_multisite_tests: 2023-05-31T21:35:43.373 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000044815', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-05-31T21:21:05.078544Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000031499', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-05-31T21:12:40.667485Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000050831', 'next_step_marker': '', 'total_entries': 55, 'pos': 0, 'timestamp': '2023-05-31T21:19:03.876722Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000043896', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2023-05-31T21:12:28.529956Z'}}] 2023-05-31T21:35:43.373 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '00000000000000000000:00000000000000044955', 1: '00000000000000000000:00000000000000031499', 2: '00000000000000000000:00000000000000051111', 3: '00000000000000000000:00000000000000044038'} 2023-05-31T21:35:43.373 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '00000000000000000000:00000000000000044815', 1: '00000000000000000000:00000000000000031499', 2: '00000000000000000000:00000000000000050831', 3: '00000000000000000000:00000000000000043896'} 2023-05-31T21:35:43.373 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: data of zone a2 behind zone a1: shard=0 master=00000000000000000000:00000000000000044955 target=00000000000000000000:00000000000000044815, shard=2 master=00000000000000000000:00000000000000051111 target=00000000000000000000:00000000000000050831, shard=3 master=00000000000000000000:00000000000000044038 target=00000000000000000000:00000000000000043896 2023-05-31T21:35:43.374 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-31T21:35:43.374 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-31T21:35:43.374 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-05-31T21:35:43.374 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: ignoring --setuser ceph since I am not root 2023-05-31T21:35:43.375 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-05-31T21:35:43.375 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2023-05-31T21:35:43.375 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "sync_status": { 2023-05-31T21:35:43.375 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "info": { 2023-05-31T21:35:43.375 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "status": "sync", 2023-05-31T21:35:43.376 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "num_shards": 4, 2023-05-31T21:35:43.376 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "instance_id": 452136274422638824 2023-05-31T21:35:43.376 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2023-05-31T21:35:43.376 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "markers": [ 2023-05-31T21:35:43.377 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2023-05-31T21:35:43.377 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "key": 0, 2023-05-31T21:35:43.377 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "val": { 2023-05-31T21:35:43.377 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "status": "incremental-sync", 2023-05-31T21:35:43.378 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "marker": "00000000000000000000:00000000000000044815", 2023-05-31T21:35:43.378 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "next_step_marker": "", 2023-05-31T21:35:43.378 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "total_entries": 0, 2023-05-31T21:35:43.378 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "pos": 0, 2023-05-31T21:35:43.378 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "timestamp": "2023-05-31T21:21:05.078544Z" 2023-05-31T21:35:43.379 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2023-05-31T21:35:43.379 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2023-05-31T21:35:43.379 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2023-05-31T21:35:43.379 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "key": 1, 2023-05-31T21:35:43.380 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "val": { 2023-05-31T21:35:43.380 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "status": "incremental-sync", 2023-05-31T21:35:43.380 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "marker": "00000000000000000000:00000000000000031499", 2023-05-31T21:35:43.380 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "next_step_marker": "", 2023-05-31T21:35:43.381 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "total_entries": 0, 2023-05-31T21:35:43.381 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "pos": 0, 2023-05-31T21:35:43.381 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "timestamp": "2023-05-31T21:12:40.667485Z" 2023-05-31T21:35:43.381 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2023-05-31T21:35:43.381 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2023-05-31T21:35:43.382 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2023-05-31T21:35:43.382 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "key": 2, 2023-05-31T21:35:43.382 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "val": { 2023-05-31T21:35:43.382 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "status": "incremental-sync", 2023-05-31T21:35:43.383 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "marker": "00000000000000000000:00000000000000050831", 2023-05-31T21:35:43.383 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "next_step_marker": "", 2023-05-31T21:35:43.383 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "total_entries": 55, 2023-05-31T21:35:43.383 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "pos": 0, 2023-05-31T21:35:43.384 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "timestamp": "2023-05-31T21:19:03.876722Z" 2023-05-31T21:35:43.384 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2023-05-31T21:35:43.384 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2023-05-31T21:35:43.384 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2023-05-31T21:35:43.384 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "key": 3, 2023-05-31T21:35:43.385 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "val": { 2023-05-31T21:35:43.385 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "status": "incremental-sync", 2023-05-31T21:35:43.385 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "marker": "00000000000000000000:00000000000000043896", 2023-05-31T21:35:43.385 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "next_step_marker": "", 2023-05-31T21:35:43.386 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "total_entries": 9, 2023-05-31T21:35:43.386 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "pos": 0, 2023-05-31T21:35:43.386 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "timestamp": "2023-05-31T21:12:28.529956Z" 2023-05-31T21:35:43.386 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2023-05-31T21:35:43.387 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2023-05-31T21:35:43.387 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: ] 2023-05-31T21:35:43.387 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2023-05-31T21:35:43.387 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "full_sync": { 2023-05-31T21:35:43.388 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "total": 64, 2023-05-31T21:35:43.388 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "complete": 64 2023-05-31T21:35:43.388 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2023-05-31T21:35:43.388 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "current_time": "2023-05-31T21:21:52Z" 2023-05-31T21:35:43.389 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2023-05-31T21:35:43.389 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current data sync status={ 2023-05-31T21:35:43.389 INFO:tasks.rgw_multisite_tests: "sync_status": { 2023-05-31T21:35:43.389 INFO:tasks.rgw_multisite_tests: "info": { 2023-05-31T21:35:43.389 INFO:tasks.rgw_multisite_tests: "status": "sync", 2023-05-31T21:35:43.390 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2023-05-31T21:35:43.390 INFO:tasks.rgw_multisite_tests: "instance_id": 452136274422638824 2023-05-31T21:35:43.390 INFO:tasks.rgw_multisite_tests: }, 2023-05-31T21:35:43.390 INFO:tasks.rgw_multisite_tests: "markers": [ 2023-05-31T21:35:43.391 INFO:tasks.rgw_multisite_tests: { 2023-05-31T21:35:43.391 INFO:tasks.rgw_multisite_tests: "key": 0, 2023-05-31T21:35:43.391 INFO:tasks.rgw_multisite_tests: "val": { 2023-05-31T21:35:43.391 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2023-05-31T21:35:43.392 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000044815", 2023-05-31T21:35:43.392 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-05-31T21:35:43.392 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-05-31T21:35:43.392 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-05-31T21:35:43.393 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-05-31T21:21:05.078544Z" 2023-05-31T21:35:43.393 INFO:tasks.rgw_multisite_tests: } 2023-05-31T21:35:43.393 INFO:tasks.rgw_multisite_tests: }, 2023-05-31T21:35:43.393 INFO:tasks.rgw_multisite_tests: { 2023-05-31T21:35:43.393 INFO:tasks.rgw_multisite_tests: "key": 1, 2023-05-31T21:35:43.394 INFO:tasks.rgw_multisite_tests: "val": { 2023-05-31T21:35:43.394 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2023-05-31T21:35:43.394 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000031499", 2023-05-31T21:35:43.394 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-05-31T21:35:43.395 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-05-31T21:35:43.395 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-05-31T21:35:43.395 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-05-31T21:12:40.667485Z" 2023-05-31T21:35:43.395 INFO:tasks.rgw_multisite_tests: } 2023-05-31T21:35:43.396 INFO:tasks.rgw_multisite_tests: }, 2023-05-31T21:35:43.396 INFO:tasks.rgw_multisite_tests: { 2023-05-31T21:35:43.396 INFO:tasks.rgw_multisite_tests: "key": 2, 2023-05-31T21:35:43.396 INFO:tasks.rgw_multisite_tests: "val": { 2023-05-31T21:35:43.396 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2023-05-31T21:35:43.397 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000050831", 2023-05-31T21:35:43.397 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-05-31T21:35:43.397 INFO:tasks.rgw_multisite_tests: "total_entries": 55, 2023-05-31T21:35:43.397 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-05-31T21:35:43.398 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-05-31T21:19:03.876722Z" 2023-05-31T21:35:43.398 INFO:tasks.rgw_multisite_tests: } 2023-05-31T21:35:43.398 INFO:tasks.rgw_multisite_tests: }, 2023-05-31T21:35:43.398 INFO:tasks.rgw_multisite_tests: { 2023-05-31T21:35:43.399 INFO:tasks.rgw_multisite_tests: "key": 3, 2023-05-31T21:35:43.399 INFO:tasks.rgw_multisite_tests: "val": { 2023-05-31T21:35:43.399 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2023-05-31T21:35:43.399 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000043896", 2023-05-31T21:35:43.400 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-05-31T21:35:43.400 INFO:tasks.rgw_multisite_tests: "total_entries": 9, 2023-05-31T21:35:43.400 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-05-31T21:35:43.400 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-05-31T21:12:28.529956Z" 2023-05-31T21:35:43.400 INFO:tasks.rgw_multisite_tests: } 2023-05-31T21:35:43.401 INFO:tasks.rgw_multisite_tests: } 2023-05-31T21:35:43.401 INFO:tasks.rgw_multisite_tests: ] 2023-05-31T21:35:43.401 INFO:tasks.rgw_multisite_tests: }, 2023-05-31T21:35:43.401 INFO:tasks.rgw_multisite_tests: "full_sync": { 2023-05-31T21:35:43.402 INFO:tasks.rgw_multisite_tests: "total": 64, 2023-05-31T21:35:43.402 INFO:tasks.rgw_multisite_tests: "complete": 64 2023-05-31T21:35:43.402 INFO:tasks.rgw_multisite_tests: }, 2023-05-31T21:35:43.402 INFO:tasks.rgw_multisite_tests: "current_time": "2023-05-31T21:21:52Z" 2023-05-31T21:35:43.403 INFO:tasks.rgw_multisite_tests:} 2023-05-31T21:35:43.403 INFO:tasks.rgw_multisite_tests: 2023-05-31T21:35:43.403 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000044815', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-05-31T21:21:05.078544Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000031499', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-05-31T21:12:40.667485Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000050831', 'next_step_marker': '', 'total_entries': 55, 'pos': 0, 'timestamp': '2023-05-31T21:19:03.876722Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000043896', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2023-05-31T21:12:28.529956Z'}}] 2023-05-31T21:35:43.403 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '00000000000000000000:00000000000000044955', 1: '00000000000000000000:00000000000000031499', 2: '00000000000000000000:00000000000000051111', 3: '00000000000000000000:00000000000000044038'} 2023-05-31T21:35:43.403 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '00000000000000000000:00000000000000044815', 1: '00000000000000000000:00000000000000031499', 2: '00000000000000000000:00000000000000050831', 3: '00000000000000000000:00000000000000043896'} 2023-05-31T21:35:43.404 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: data of zone a2 behind zone a1: shard=0 master=00000000000000000000:00000000000000044955 target=00000000000000000000:00000000000000044815, shard=2 master=00000000000000000000:00000000000000051111 target=00000000000000000000:00000000000000050831, shard=3 master=00000000000000000000:00000000000000044038 target=00000000000000000000:00000000000000043896 2023-05-31T21:35:43.404 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-31T21:35:43.404 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-31T21:35:43.404 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-05-31T21:35:43.405 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: ignoring --setuser ceph since I am not root 2023-05-31T21:35:43.405 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-05-31T21:35:43.405 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2023-05-31T21:35:43.405 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "sync_status": { 2023-05-31T21:35:43.406 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "info": { 2023-05-31T21:35:43.406 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "status": "sync", 2023-05-31T21:35:43.406 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "num_shards": 4, 2023-05-31T21:35:43.406 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "instance_id": 452136274422638824 2023-05-31T21:35:43.407 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2023-05-31T21:35:43.407 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "markers": [ 2023-05-31T21:35:43.407 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2023-05-31T21:35:43.407 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "key": 0, 2023-05-31T21:35:43.407 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "val": { 2023-05-31T21:35:43.408 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "status": "incremental-sync", 2023-05-31T21:35:43.408 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "marker": "00000000000000000000:00000000000000044815", 2023-05-31T21:35:43.408 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "next_step_marker": "", 2023-05-31T21:35:43.408 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "total_entries": 0, 2023-05-31T21:35:43.409 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "pos": 0, 2023-05-31T21:35:43.409 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "timestamp": "2023-05-31T21:21:05.078544Z" 2023-05-31T21:35:43.409 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2023-05-31T21:35:43.409 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2023-05-31T21:35:43.410 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2023-05-31T21:35:43.410 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "key": 1, 2023-05-31T21:35:43.410 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "val": { 2023-05-31T21:35:43.410 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "status": "incremental-sync", 2023-05-31T21:35:43.410 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "marker": "00000000000000000000:00000000000000031499", 2023-05-31T21:35:43.411 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "next_step_marker": "", 2023-05-31T21:35:43.411 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "total_entries": 0, 2023-05-31T21:35:43.411 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "pos": 0, 2023-05-31T21:35:43.411 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "timestamp": "2023-05-31T21:12:40.667485Z" 2023-05-31T21:35:43.412 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2023-05-31T21:35:43.412 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2023-05-31T21:35:43.412 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2023-05-31T21:35:43.412 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "key": 2, 2023-05-31T21:35:43.413 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "val": { 2023-05-31T21:35:43.413 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "status": "incremental-sync", 2023-05-31T21:35:43.413 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "marker": "00000000000000000000:00000000000000050831", 2023-05-31T21:35:43.413 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "next_step_marker": "", 2023-05-31T21:35:43.413 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "total_entries": 55, 2023-05-31T21:35:43.414 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "pos": 0, 2023-05-31T21:35:43.414 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "timestamp": "2023-05-31T21:19:03.876722Z" 2023-05-31T21:35:43.414 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2023-05-31T21:35:43.414 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2023-05-31T21:35:43.415 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2023-05-31T21:35:43.415 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "key": 3, 2023-05-31T21:35:43.415 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "val": { 2023-05-31T21:35:43.415 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "status": "incremental-sync", 2023-05-31T21:35:43.415 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "marker": "00000000000000000000:00000000000000043896", 2023-05-31T21:35:43.416 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "next_step_marker": "", 2023-05-31T21:35:43.416 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "total_entries": 9, 2023-05-31T21:35:43.416 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "pos": 0, 2023-05-31T21:35:43.416 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "timestamp": "2023-05-31T21:12:28.529956Z" 2023-05-31T21:35:43.417 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2023-05-31T21:35:43.417 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2023-05-31T21:35:43.417 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: ] 2023-05-31T21:35:43.417 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2023-05-31T21:35:43.418 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "full_sync": { 2023-05-31T21:35:43.418 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "total": 64, 2023-05-31T21:35:43.418 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "complete": 64 2023-05-31T21:35:43.418 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2023-05-31T21:35:43.419 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "current_time": "2023-05-31T21:21:57Z" 2023-05-31T21:35:43.419 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2023-05-31T21:35:43.419 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current data sync status={ 2023-05-31T21:35:43.419 INFO:tasks.rgw_multisite_tests: "sync_status": { 2023-05-31T21:35:43.419 INFO:tasks.rgw_multisite_tests: "info": { 2023-05-31T21:35:43.420 INFO:tasks.rgw_multisite_tests: "status": "sync", 2023-05-31T21:35:43.420 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2023-05-31T21:35:43.420 INFO:tasks.rgw_multisite_tests: "instance_id": 452136274422638824 2023-05-31T21:35:43.420 INFO:tasks.rgw_multisite_tests: }, 2023-05-31T21:35:43.421 INFO:tasks.rgw_multisite_tests: "markers": [ 2023-05-31T21:35:43.421 INFO:tasks.rgw_multisite_tests: { 2023-05-31T21:35:43.421 INFO:tasks.rgw_multisite_tests: "key": 0, 2023-05-31T21:35:43.421 INFO:tasks.rgw_multisite_tests: "val": { 2023-05-31T21:35:43.422 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2023-05-31T21:35:43.422 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000044815", 2023-05-31T21:35:43.422 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-05-31T21:35:43.422 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-05-31T21:35:43.423 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-05-31T21:35:43.423 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-05-31T21:21:05.078544Z" 2023-05-31T21:35:43.423 INFO:tasks.rgw_multisite_tests: } 2023-05-31T21:35:43.423 INFO:tasks.rgw_multisite_tests: }, 2023-05-31T21:35:43.424 INFO:tasks.rgw_multisite_tests: { 2023-05-31T21:35:43.424 INFO:tasks.rgw_multisite_tests: "key": 1, 2023-05-31T21:35:43.424 INFO:tasks.rgw_multisite_tests: "val": { 2023-05-31T21:35:43.424 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2023-05-31T21:35:43.424 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000031499", 2023-05-31T21:35:43.425 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-05-31T21:35:43.425 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-05-31T21:35:43.425 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-05-31T21:35:43.425 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-05-31T21:12:40.667485Z" 2023-05-31T21:35:43.426 INFO:tasks.rgw_multisite_tests: } 2023-05-31T21:35:43.426 INFO:tasks.rgw_multisite_tests: }, 2023-05-31T21:35:43.426 INFO:tasks.rgw_multisite_tests: { 2023-05-31T21:35:43.426 INFO:tasks.rgw_multisite_tests: "key": 2, 2023-05-31T21:35:43.427 INFO:tasks.rgw_multisite_tests: "val": { 2023-05-31T21:35:43.427 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2023-05-31T21:35:43.427 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000050831", 2023-05-31T21:35:43.427 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-05-31T21:35:43.427 INFO:tasks.rgw_multisite_tests: "total_entries": 55, 2023-05-31T21:35:43.428 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-05-31T21:35:43.428 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-05-31T21:19:03.876722Z" 2023-05-31T21:35:43.428 INFO:tasks.rgw_multisite_tests: } 2023-05-31T21:35:43.428 INFO:tasks.rgw_multisite_tests: }, 2023-05-31T21:35:43.429 INFO:tasks.rgw_multisite_tests: { 2023-05-31T21:35:43.429 INFO:tasks.rgw_multisite_tests: "key": 3, 2023-05-31T21:35:43.429 INFO:tasks.rgw_multisite_tests: "val": { 2023-05-31T21:35:43.429 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2023-05-31T21:35:43.430 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000043896", 2023-05-31T21:35:43.430 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-05-31T21:35:43.430 INFO:tasks.rgw_multisite_tests: "total_entries": 9, 2023-05-31T21:35:43.430 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-05-31T21:35:43.430 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-05-31T21:12:28.529956Z" 2023-05-31T21:35:43.431 INFO:tasks.rgw_multisite_tests: } 2023-05-31T21:35:43.431 INFO:tasks.rgw_multisite_tests: } 2023-05-31T21:35:43.431 INFO:tasks.rgw_multisite_tests: ] 2023-05-31T21:35:43.431 INFO:tasks.rgw_multisite_tests: }, 2023-05-31T21:35:43.432 INFO:tasks.rgw_multisite_tests: "full_sync": { 2023-05-31T21:35:43.432 INFO:tasks.rgw_multisite_tests: "total": 64, 2023-05-31T21:35:43.432 INFO:tasks.rgw_multisite_tests: "complete": 64 2023-05-31T21:35:43.432 INFO:tasks.rgw_multisite_tests: }, 2023-05-31T21:35:43.433 INFO:tasks.rgw_multisite_tests: "current_time": "2023-05-31T21:21:57Z" 2023-05-31T21:35:43.433 INFO:tasks.rgw_multisite_tests:} 2023-05-31T21:35:43.433 INFO:tasks.rgw_multisite_tests: 2023-05-31T21:35:43.433 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000044815', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-05-31T21:21:05.078544Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000031499', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-05-31T21:12:40.667485Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000050831', 'next_step_marker': '', 'total_entries': 55, 'pos': 0, 'timestamp': '2023-05-31T21:19:03.876722Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000043896', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2023-05-31T21:12:28.529956Z'}}] 2023-05-31T21:35:43.434 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '00000000000000000000:00000000000000044955', 1: '00000000000000000000:00000000000000031499', 2: '00000000000000000000:00000000000000051111', 3: '00000000000000000000:00000000000000044038'} 2023-05-31T21:35:43.434 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '00000000000000000000:00000000000000044815', 1: '00000000000000000000:00000000000000031499', 2: '00000000000000000000:00000000000000050831', 3: '00000000000000000000:00000000000000043896'} 2023-05-31T21:35:43.434 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: data of zone a2 behind zone a1: shard=0 master=00000000000000000000:00000000000000044955 target=00000000000000000000:00000000000000044815, shard=2 master=00000000000000000000:00000000000000051111 target=00000000000000000000:00000000000000050831, shard=3 master=00000000000000000000:00000000000000044038 target=00000000000000000000:00000000000000043896 2023-05-31T21:35:43.434 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-31T21:35:43.435 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-31T21:35:43.435 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-05-31T21:35:43.435 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: ignoring --setuser ceph since I am not root 2023-05-31T21:35:43.435 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-05-31T21:35:43.436 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2023-05-31T21:35:43.436 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "sync_status": { 2023-05-31T21:35:43.436 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "info": { 2023-05-31T21:35:43.436 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "status": "sync", 2023-05-31T21:35:43.437 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "num_shards": 4, 2023-05-31T21:35:43.437 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "instance_id": 452136274422638824 2023-05-31T21:35:43.437 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2023-05-31T21:35:43.437 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "markers": [ 2023-05-31T21:35:43.438 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2023-05-31T21:35:43.438 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "key": 0, 2023-05-31T21:35:43.438 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "val": { 2023-05-31T21:35:43.438 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "status": "incremental-sync", 2023-05-31T21:35:43.438 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "marker": "00000000000000000000:00000000000000044815", 2023-05-31T21:35:43.439 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "next_step_marker": "", 2023-05-31T21:35:43.439 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "total_entries": 0, 2023-05-31T21:35:43.439 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "pos": 0, 2023-05-31T21:35:43.439 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "timestamp": "2023-05-31T21:21:05.078544Z" 2023-05-31T21:35:43.440 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2023-05-31T21:35:43.440 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2023-05-31T21:35:43.440 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2023-05-31T21:35:43.440 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "key": 1, 2023-05-31T21:35:43.441 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "val": { 2023-05-31T21:35:43.441 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "status": "incremental-sync", 2023-05-31T21:35:43.441 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "marker": "00000000000000000000:00000000000000031499", 2023-05-31T21:35:43.441 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "next_step_marker": "", 2023-05-31T21:35:43.441 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "total_entries": 0, 2023-05-31T21:35:43.442 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "pos": 0, 2023-05-31T21:35:43.442 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "timestamp": "2023-05-31T21:12:40.667485Z" 2023-05-31T21:35:43.442 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2023-05-31T21:35:43.442 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2023-05-31T21:35:43.443 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2023-05-31T21:35:43.443 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "key": 2, 2023-05-31T21:35:43.443 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "val": { 2023-05-31T21:35:43.443 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "status": "incremental-sync", 2023-05-31T21:35:43.444 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "marker": "00000000000000000000:00000000000000050831", 2023-05-31T21:35:43.444 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "next_step_marker": "", 2023-05-31T21:35:43.444 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "total_entries": 55, 2023-05-31T21:35:43.444 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "pos": 0, 2023-05-31T21:35:43.445 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "timestamp": "2023-05-31T21:19:03.876722Z" 2023-05-31T21:35:43.445 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2023-05-31T21:35:43.445 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2023-05-31T21:35:43.445 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2023-05-31T21:35:43.446 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "key": 3, 2023-05-31T21:35:43.446 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "val": { 2023-05-31T21:35:43.446 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "status": "incremental-sync", 2023-05-31T21:35:43.446 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "marker": "00000000000000000000:00000000000000043896", 2023-05-31T21:35:43.446 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "next_step_marker": "", 2023-05-31T21:35:43.447 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "total_entries": 9, 2023-05-31T21:35:43.447 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "pos": 0, 2023-05-31T21:35:43.447 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "timestamp": "2023-05-31T21:12:28.529956Z" 2023-05-31T21:35:43.447 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2023-05-31T21:35:43.448 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2023-05-31T21:35:43.448 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: ] 2023-05-31T21:35:43.448 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2023-05-31T21:35:43.448 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "full_sync": { 2023-05-31T21:35:43.449 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "total": 64, 2023-05-31T21:35:43.449 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "complete": 64 2023-05-31T21:35:43.449 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2023-05-31T21:35:43.449 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "current_time": "2023-05-31T21:22:02Z" 2023-05-31T21:35:43.449 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2023-05-31T21:35:43.450 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current data sync status={ 2023-05-31T21:35:43.450 INFO:tasks.rgw_multisite_tests: "sync_status": { 2023-05-31T21:35:43.450 INFO:tasks.rgw_multisite_tests: "info": { 2023-05-31T21:35:43.450 INFO:tasks.rgw_multisite_tests: "status": "sync", 2023-05-31T21:35:43.451 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2023-05-31T21:35:43.451 INFO:tasks.rgw_multisite_tests: "instance_id": 452136274422638824 2023-05-31T21:35:43.451 INFO:tasks.rgw_multisite_tests: }, 2023-05-31T21:35:43.451 INFO:tasks.rgw_multisite_tests: "markers": [ 2023-05-31T21:35:43.452 INFO:tasks.rgw_multisite_tests: { 2023-05-31T21:35:43.452 INFO:tasks.rgw_multisite_tests: "key": 0, 2023-05-31T21:35:43.452 INFO:tasks.rgw_multisite_tests: "val": { 2023-05-31T21:35:43.452 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2023-05-31T21:35:43.452 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000044815", 2023-05-31T21:35:43.453 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-05-31T21:35:43.453 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-05-31T21:35:43.453 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-05-31T21:35:43.453 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-05-31T21:21:05.078544Z" 2023-05-31T21:35:43.454 INFO:tasks.rgw_multisite_tests: } 2023-05-31T21:35:43.454 INFO:tasks.rgw_multisite_tests: }, 2023-05-31T21:35:43.454 INFO:tasks.rgw_multisite_tests: { 2023-05-31T21:35:43.454 INFO:tasks.rgw_multisite_tests: "key": 1, 2023-05-31T21:35:43.455 INFO:tasks.rgw_multisite_tests: "val": { 2023-05-31T21:35:43.455 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2023-05-31T21:35:43.455 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000031499", 2023-05-31T21:35:43.455 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-05-31T21:35:43.455 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-05-31T21:35:43.456 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-05-31T21:35:43.456 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-05-31T21:12:40.667485Z" 2023-05-31T21:35:43.456 INFO:tasks.rgw_multisite_tests: } 2023-05-31T21:35:43.456 INFO:tasks.rgw_multisite_tests: }, 2023-05-31T21:35:43.457 INFO:tasks.rgw_multisite_tests: { 2023-05-31T21:35:43.457 INFO:tasks.rgw_multisite_tests: "key": 2, 2023-05-31T21:35:43.457 INFO:tasks.rgw_multisite_tests: "val": { 2023-05-31T21:35:43.457 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2023-05-31T21:35:43.458 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000050831", 2023-05-31T21:35:43.458 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-05-31T21:35:43.458 INFO:tasks.rgw_multisite_tests: "total_entries": 55, 2023-05-31T21:35:43.458 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-05-31T21:35:43.458 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-05-31T21:19:03.876722Z" 2023-05-31T21:35:43.459 INFO:tasks.rgw_multisite_tests: } 2023-05-31T21:35:43.459 INFO:tasks.rgw_multisite_tests: }, 2023-05-31T21:35:43.459 INFO:tasks.rgw_multisite_tests: { 2023-05-31T21:35:43.459 INFO:tasks.rgw_multisite_tests: "key": 3, 2023-05-31T21:35:43.460 INFO:tasks.rgw_multisite_tests: "val": { 2023-05-31T21:35:43.460 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2023-05-31T21:35:43.460 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000043896", 2023-05-31T21:35:43.460 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-05-31T21:35:43.461 INFO:tasks.rgw_multisite_tests: "total_entries": 9, 2023-05-31T21:35:43.461 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-05-31T21:35:43.461 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-05-31T21:12:28.529956Z" 2023-05-31T21:35:43.461 INFO:tasks.rgw_multisite_tests: } 2023-05-31T21:35:43.461 INFO:tasks.rgw_multisite_tests: } 2023-05-31T21:35:43.462 INFO:tasks.rgw_multisite_tests: ] 2023-05-31T21:35:43.462 INFO:tasks.rgw_multisite_tests: }, 2023-05-31T21:35:43.462 INFO:tasks.rgw_multisite_tests: "full_sync": { 2023-05-31T21:35:43.462 INFO:tasks.rgw_multisite_tests: "total": 64, 2023-05-31T21:35:43.463 INFO:tasks.rgw_multisite_tests: "complete": 64 2023-05-31T21:35:43.463 INFO:tasks.rgw_multisite_tests: }, 2023-05-31T21:35:43.463 INFO:tasks.rgw_multisite_tests: "current_time": "2023-05-31T21:22:02Z" 2023-05-31T21:35:43.463 INFO:tasks.rgw_multisite_tests:} 2023-05-31T21:35:43.464 INFO:tasks.rgw_multisite_tests: 2023-05-31T21:35:43.464 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000044815', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-05-31T21:21:05.078544Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000031499', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-05-31T21:12:40.667485Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000050831', 'next_step_marker': '', 'total_entries': 55, 'pos': 0, 'timestamp': '2023-05-31T21:19:03.876722Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000043896', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2023-05-31T21:12:28.529956Z'}}] 2023-05-31T21:35:43.464 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '00000000000000000000:00000000000000044955', 1: '00000000000000000000:00000000000000031499', 2: '00000000000000000000:00000000000000051111', 3: '00000000000000000000:00000000000000044038'} 2023-05-31T21:35:43.464 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '00000000000000000000:00000000000000044815', 1: '00000000000000000000:00000000000000031499', 2: '00000000000000000000:00000000000000050831', 3: '00000000000000000000:00000000000000043896'} 2023-05-31T21:35:43.464 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: data of zone a2 behind zone a1: shard=0 master=00000000000000000000:00000000000000044955 target=00000000000000000000:00000000000000044815, shard=2 master=00000000000000000000:00000000000000051111 target=00000000000000000000:00000000000000050831, shard=3 master=00000000000000000000:00000000000000044038 target=00000000000000000000:00000000000000043896 2023-05-31T21:35:43.465 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-31T21:35:43.465 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-31T21:35:43.465 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-05-31T21:35:43.465 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: ignoring --setuser ceph since I am not root 2023-05-31T21:35:43.466 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-05-31T21:35:43.466 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2023-05-31T21:35:43.466 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "sync_status": { 2023-05-31T21:35:43.466 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "info": { 2023-05-31T21:35:43.467 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "status": "sync", 2023-05-31T21:35:43.467 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "num_shards": 4, 2023-05-31T21:35:43.467 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "instance_id": 452136274422638824 2023-05-31T21:35:43.467 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2023-05-31T21:35:43.468 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "markers": [ 2023-05-31T21:35:43.468 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2023-05-31T21:35:43.468 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "key": 0, 2023-05-31T21:35:43.468 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "val": { 2023-05-31T21:35:43.468 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "status": "incremental-sync", 2023-05-31T21:35:43.469 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "marker": "00000000000000000000:00000000000000044815", 2023-05-31T21:35:43.469 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "next_step_marker": "", 2023-05-31T21:35:43.469 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "total_entries": 0, 2023-05-31T21:35:43.469 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "pos": 0, 2023-05-31T21:35:43.470 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "timestamp": "2023-05-31T21:21:05.078544Z" 2023-05-31T21:35:43.470 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2023-05-31T21:35:43.470 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2023-05-31T21:35:43.470 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2023-05-31T21:35:43.471 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "key": 1, 2023-05-31T21:35:43.471 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "val": { 2023-05-31T21:35:43.471 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "status": "incremental-sync", 2023-05-31T21:35:43.471 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "marker": "00000000000000000000:00000000000000031499", 2023-05-31T21:35:43.471 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "next_step_marker": "", 2023-05-31T21:35:43.472 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "total_entries": 0, 2023-05-31T21:35:43.472 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "pos": 0, 2023-05-31T21:35:43.472 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "timestamp": "2023-05-31T21:12:40.667485Z" 2023-05-31T21:35:43.472 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2023-05-31T21:35:43.473 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2023-05-31T21:35:43.473 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2023-05-31T21:35:43.473 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "key": 2, 2023-05-31T21:35:43.473 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "val": { 2023-05-31T21:35:43.474 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "status": "incremental-sync", 2023-05-31T21:35:43.474 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "marker": "00000000000000000000:00000000000000050831", 2023-05-31T21:35:43.474 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "next_step_marker": "", 2023-05-31T21:35:43.474 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "total_entries": 55, 2023-05-31T21:35:43.474 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "pos": 0, 2023-05-31T21:35:43.475 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "timestamp": "2023-05-31T21:19:03.876722Z" 2023-05-31T21:35:43.475 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2023-05-31T21:35:43.475 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2023-05-31T21:35:43.475 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2023-05-31T21:35:43.476 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "key": 3, 2023-05-31T21:35:43.476 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "val": { 2023-05-31T21:35:43.476 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "status": "incremental-sync", 2023-05-31T21:35:43.476 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "marker": "00000000000000000000:00000000000000043896", 2023-05-31T21:35:43.477 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "next_step_marker": "", 2023-05-31T21:35:43.477 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "total_entries": 9, 2023-05-31T21:35:43.477 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "pos": 0, 2023-05-31T21:35:43.477 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "timestamp": "2023-05-31T21:12:28.529956Z" 2023-05-31T21:35:43.478 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2023-05-31T21:35:43.478 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2023-05-31T21:35:43.478 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: ] 2023-05-31T21:35:43.478 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2023-05-31T21:35:43.478 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "full_sync": { 2023-05-31T21:35:43.479 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "total": 64, 2023-05-31T21:35:43.479 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "complete": 64 2023-05-31T21:35:43.479 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2023-05-31T21:35:43.479 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "current_time": "2023-05-31T21:22:07Z" 2023-05-31T21:35:43.480 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2023-05-31T21:35:43.480 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current data sync status={ 2023-05-31T21:35:43.480 INFO:tasks.rgw_multisite_tests: "sync_status": { 2023-05-31T21:35:43.480 INFO:tasks.rgw_multisite_tests: "info": { 2023-05-31T21:35:43.481 INFO:tasks.rgw_multisite_tests: "status": "sync", 2023-05-31T21:35:43.481 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2023-05-31T21:35:43.481 INFO:tasks.rgw_multisite_tests: "instance_id": 452136274422638824 2023-05-31T21:35:43.481 INFO:tasks.rgw_multisite_tests: }, 2023-05-31T21:35:43.481 INFO:tasks.rgw_multisite_tests: "markers": [ 2023-05-31T21:35:43.482 INFO:tasks.rgw_multisite_tests: { 2023-05-31T21:35:43.482 INFO:tasks.rgw_multisite_tests: "key": 0, 2023-05-31T21:35:43.482 INFO:tasks.rgw_multisite_tests: "val": { 2023-05-31T21:35:43.482 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2023-05-31T21:35:43.483 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000044815", 2023-05-31T21:35:43.483 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-05-31T21:35:43.483 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-05-31T21:35:43.483 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-05-31T21:35:43.483 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-05-31T21:21:05.078544Z" 2023-05-31T21:35:43.484 INFO:tasks.rgw_multisite_tests: } 2023-05-31T21:35:43.484 INFO:tasks.rgw_multisite_tests: }, 2023-05-31T21:35:43.484 INFO:tasks.rgw_multisite_tests: { 2023-05-31T21:35:43.484 INFO:tasks.rgw_multisite_tests: "key": 1, 2023-05-31T21:35:43.485 INFO:tasks.rgw_multisite_tests: "val": { 2023-05-31T21:35:43.485 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2023-05-31T21:35:43.485 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000031499", 2023-05-31T21:35:43.485 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-05-31T21:35:43.486 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-05-31T21:35:43.486 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-05-31T21:35:43.486 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-05-31T21:12:40.667485Z" 2023-05-31T21:35:43.486 INFO:tasks.rgw_multisite_tests: } 2023-05-31T21:35:43.487 INFO:tasks.rgw_multisite_tests: }, 2023-05-31T21:35:43.487 INFO:tasks.rgw_multisite_tests: { 2023-05-31T21:35:43.487 INFO:tasks.rgw_multisite_tests: "key": 2, 2023-05-31T21:35:43.487 INFO:tasks.rgw_multisite_tests: "val": { 2023-05-31T21:35:43.487 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2023-05-31T21:35:43.488 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000050831", 2023-05-31T21:35:43.488 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-05-31T21:35:43.488 INFO:tasks.rgw_multisite_tests: "total_entries": 55, 2023-05-31T21:35:43.488 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-05-31T21:35:43.489 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-05-31T21:19:03.876722Z" 2023-05-31T21:35:43.489 INFO:tasks.rgw_multisite_tests: } 2023-05-31T21:35:43.489 INFO:tasks.rgw_multisite_tests: }, 2023-05-31T21:35:43.490 INFO:tasks.rgw_multisite_tests: { 2023-05-31T21:35:43.490 INFO:tasks.rgw_multisite_tests: "key": 3, 2023-05-31T21:35:43.490 INFO:tasks.rgw_multisite_tests: "val": { 2023-05-31T21:35:43.490 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2023-05-31T21:35:43.490 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000043896", 2023-05-31T21:35:43.491 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-05-31T21:35:43.491 INFO:tasks.rgw_multisite_tests: "total_entries": 9, 2023-05-31T21:35:43.491 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-05-31T21:35:43.491 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-05-31T21:12:28.529956Z" 2023-05-31T21:35:43.492 INFO:tasks.rgw_multisite_tests: } 2023-05-31T21:35:43.492 INFO:tasks.rgw_multisite_tests: } 2023-05-31T21:35:43.492 INFO:tasks.rgw_multisite_tests: ] 2023-05-31T21:35:43.492 INFO:tasks.rgw_multisite_tests: }, 2023-05-31T21:35:43.493 INFO:tasks.rgw_multisite_tests: "full_sync": { 2023-05-31T21:35:43.493 INFO:tasks.rgw_multisite_tests: "total": 64, 2023-05-31T21:35:43.493 INFO:tasks.rgw_multisite_tests: "complete": 64 2023-05-31T21:35:43.493 INFO:tasks.rgw_multisite_tests: }, 2023-05-31T21:35:43.493 INFO:tasks.rgw_multisite_tests: "current_time": "2023-05-31T21:22:07Z" 2023-05-31T21:35:43.494 INFO:tasks.rgw_multisite_tests:} 2023-05-31T21:35:43.494 INFO:tasks.rgw_multisite_tests: 2023-05-31T21:35:43.494 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000044815', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-05-31T21:21:05.078544Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000031499', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-05-31T21:12:40.667485Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000050831', 'next_step_marker': '', 'total_entries': 55, 'pos': 0, 'timestamp': '2023-05-31T21:19:03.876722Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000043896', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2023-05-31T21:12:28.529956Z'}}] 2023-05-31T21:35:43.494 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '00000000000000000000:00000000000000044955', 1: '00000000000000000000:00000000000000031499', 2: '00000000000000000000:00000000000000051111', 3: '00000000000000000000:00000000000000044038'} 2023-05-31T21:35:43.495 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '00000000000000000000:00000000000000044815', 1: '00000000000000000000:00000000000000031499', 2: '00000000000000000000:00000000000000050831', 3: '00000000000000000000:00000000000000043896'} 2023-05-31T21:35:43.495 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: data of zone a2 behind zone a1: shard=0 master=00000000000000000000:00000000000000044955 target=00000000000000000000:00000000000000044815, shard=2 master=00000000000000000000:00000000000000051111 target=00000000000000000000:00000000000000050831, shard=3 master=00000000000000000000:00000000000000044038 target=00000000000000000000:00000000000000043896 2023-05-31T21:35:43.495 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-31T21:35:43.495 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-31T21:35:43.496 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-05-31T21:35:43.496 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: ignoring --setuser ceph since I am not root 2023-05-31T21:35:43.496 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-05-31T21:35:43.496 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2023-05-31T21:35:43.497 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "sync_status": { 2023-05-31T21:35:43.497 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "info": { 2023-05-31T21:35:43.497 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "status": "sync", 2023-05-31T21:35:43.497 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "num_shards": 4, 2023-05-31T21:35:43.497 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "instance_id": 452136274422638824 2023-05-31T21:35:43.498 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2023-05-31T21:35:43.498 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "markers": [ 2023-05-31T21:35:43.498 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2023-05-31T21:35:43.498 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "key": 0, 2023-05-31T21:35:43.499 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "val": { 2023-05-31T21:35:43.499 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "status": "incremental-sync", 2023-05-31T21:35:43.499 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "marker": "00000000000000000000:00000000000000044815", 2023-05-31T21:35:43.499 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "next_step_marker": "", 2023-05-31T21:35:43.500 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "total_entries": 0, 2023-05-31T21:35:43.500 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "pos": 0, 2023-05-31T21:35:43.500 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "timestamp": "2023-05-31T21:21:05.078544Z" 2023-05-31T21:35:43.500 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2023-05-31T21:35:43.501 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2023-05-31T21:35:43.501 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2023-05-31T21:35:43.501 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "key": 1, 2023-05-31T21:35:43.501 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "val": { 2023-05-31T21:35:43.501 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "status": "incremental-sync", 2023-05-31T21:35:43.502 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "marker": "00000000000000000000:00000000000000031499", 2023-05-31T21:35:43.502 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "next_step_marker": "", 2023-05-31T21:35:43.502 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "total_entries": 0, 2023-05-31T21:35:43.502 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "pos": 0, 2023-05-31T21:35:43.503 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "timestamp": "2023-05-31T21:12:40.667485Z" 2023-05-31T21:35:43.503 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2023-05-31T21:35:43.503 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2023-05-31T21:35:43.503 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2023-05-31T21:35:43.504 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "key": 2, 2023-05-31T21:35:43.504 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "val": { 2023-05-31T21:35:43.504 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "status": "incremental-sync", 2023-05-31T21:35:43.504 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "marker": "00000000000000000000:00000000000000050831", 2023-05-31T21:35:43.505 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "next_step_marker": "", 2023-05-31T21:35:43.505 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "total_entries": 55, 2023-05-31T21:35:43.505 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "pos": 0, 2023-05-31T21:35:43.505 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "timestamp": "2023-05-31T21:19:03.876722Z" 2023-05-31T21:35:43.505 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2023-05-31T21:35:43.506 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2023-05-31T21:35:43.506 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2023-05-31T21:35:43.506 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "key": 3, 2023-05-31T21:35:43.506 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "val": { 2023-05-31T21:35:43.507 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "status": "incremental-sync", 2023-05-31T21:35:43.507 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "marker": "00000000000000000000:00000000000000043896", 2023-05-31T21:35:43.507 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "next_step_marker": "", 2023-05-31T21:35:43.507 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "total_entries": 9, 2023-05-31T21:35:43.508 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "pos": 0, 2023-05-31T21:35:43.508 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "timestamp": "2023-05-31T21:12:28.529956Z" 2023-05-31T21:35:43.508 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2023-05-31T21:35:43.508 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2023-05-31T21:35:43.509 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: ] 2023-05-31T21:35:43.509 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2023-05-31T21:35:43.509 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "full_sync": { 2023-05-31T21:35:43.509 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "total": 64, 2023-05-31T21:35:43.509 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "complete": 64 2023-05-31T21:35:43.510 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2023-05-31T21:35:43.510 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "current_time": "2023-05-31T21:22:12Z" 2023-05-31T21:35:43.510 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2023-05-31T21:35:43.510 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current data sync status={ 2023-05-31T21:35:43.511 INFO:tasks.rgw_multisite_tests: "sync_status": { 2023-05-31T21:35:43.511 INFO:tasks.rgw_multisite_tests: "info": { 2023-05-31T21:35:43.511 INFO:tasks.rgw_multisite_tests: "status": "sync", 2023-05-31T21:35:43.511 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2023-05-31T21:35:43.512 INFO:tasks.rgw_multisite_tests: "instance_id": 452136274422638824 2023-05-31T21:35:43.512 INFO:tasks.rgw_multisite_tests: }, 2023-05-31T21:35:43.512 INFO:tasks.rgw_multisite_tests: "markers": [ 2023-05-31T21:35:43.512 INFO:tasks.rgw_multisite_tests: { 2023-05-31T21:35:43.512 INFO:tasks.rgw_multisite_tests: "key": 0, 2023-05-31T21:35:43.513 INFO:tasks.rgw_multisite_tests: "val": { 2023-05-31T21:35:43.513 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2023-05-31T21:35:43.513 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000044815", 2023-05-31T21:35:43.513 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-05-31T21:35:43.514 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-05-31T21:35:43.514 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-05-31T21:35:43.514 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-05-31T21:21:05.078544Z" 2023-05-31T21:35:43.514 INFO:tasks.rgw_multisite_tests: } 2023-05-31T21:35:43.515 INFO:tasks.rgw_multisite_tests: }, 2023-05-31T21:35:43.515 INFO:tasks.rgw_multisite_tests: { 2023-05-31T21:35:43.515 INFO:tasks.rgw_multisite_tests: "key": 1, 2023-05-31T21:35:43.515 INFO:tasks.rgw_multisite_tests: "val": { 2023-05-31T21:35:43.515 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2023-05-31T21:35:43.516 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000031499", 2023-05-31T21:35:43.516 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-05-31T21:35:43.516 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-05-31T21:35:43.516 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-05-31T21:35:43.517 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-05-31T21:12:40.667485Z" 2023-05-31T21:35:43.517 INFO:tasks.rgw_multisite_tests: } 2023-05-31T21:35:43.517 INFO:tasks.rgw_multisite_tests: }, 2023-05-31T21:35:43.517 INFO:tasks.rgw_multisite_tests: { 2023-05-31T21:35:43.518 INFO:tasks.rgw_multisite_tests: "key": 2, 2023-05-31T21:35:43.518 INFO:tasks.rgw_multisite_tests: "val": { 2023-05-31T21:35:43.518 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2023-05-31T21:35:43.518 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000050831", 2023-05-31T21:35:43.518 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-05-31T21:35:43.519 INFO:tasks.rgw_multisite_tests: "total_entries": 55, 2023-05-31T21:35:43.519 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-05-31T21:35:43.519 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-05-31T21:19:03.876722Z" 2023-05-31T21:35:43.519 INFO:tasks.rgw_multisite_tests: } 2023-05-31T21:35:43.520 INFO:tasks.rgw_multisite_tests: }, 2023-05-31T21:35:43.520 INFO:tasks.rgw_multisite_tests: { 2023-05-31T21:35:43.520 INFO:tasks.rgw_multisite_tests: "key": 3, 2023-05-31T21:35:43.520 INFO:tasks.rgw_multisite_tests: "val": { 2023-05-31T21:35:43.521 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2023-05-31T21:35:43.521 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000043896", 2023-05-31T21:35:43.521 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-05-31T21:35:43.521 INFO:tasks.rgw_multisite_tests: "total_entries": 9, 2023-05-31T21:35:43.521 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-05-31T21:35:43.522 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-05-31T21:12:28.529956Z" 2023-05-31T21:35:43.522 INFO:tasks.rgw_multisite_tests: } 2023-05-31T21:35:43.522 INFO:tasks.rgw_multisite_tests: } 2023-05-31T21:35:43.522 INFO:tasks.rgw_multisite_tests: ] 2023-05-31T21:35:43.523 INFO:tasks.rgw_multisite_tests: }, 2023-05-31T21:35:43.523 INFO:tasks.rgw_multisite_tests: "full_sync": { 2023-05-31T21:35:43.523 INFO:tasks.rgw_multisite_tests: "total": 64, 2023-05-31T21:35:43.523 INFO:tasks.rgw_multisite_tests: "complete": 64 2023-05-31T21:35:43.523 INFO:tasks.rgw_multisite_tests: }, 2023-05-31T21:35:43.524 INFO:tasks.rgw_multisite_tests: "current_time": "2023-05-31T21:22:12Z" 2023-05-31T21:35:43.524 INFO:tasks.rgw_multisite_tests:} 2023-05-31T21:35:43.524 INFO:tasks.rgw_multisite_tests: 2023-05-31T21:35:43.524 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000044815', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-05-31T21:21:05.078544Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000031499', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-05-31T21:12:40.667485Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000050831', 'next_step_marker': '', 'total_entries': 55, 'pos': 0, 'timestamp': '2023-05-31T21:19:03.876722Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000043896', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2023-05-31T21:12:28.529956Z'}}] 2023-05-31T21:35:43.525 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '00000000000000000000:00000000000000044955', 1: '00000000000000000000:00000000000000031499', 2: '00000000000000000000:00000000000000051111', 3: '00000000000000000000:00000000000000044038'} 2023-05-31T21:35:43.525 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '00000000000000000000:00000000000000044815', 1: '00000000000000000000:00000000000000031499', 2: '00000000000000000000:00000000000000050831', 3: '00000000000000000000:00000000000000043896'} 2023-05-31T21:35:43.525 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: data of zone a2 behind zone a1: shard=0 master=00000000000000000000:00000000000000044955 target=00000000000000000000:00000000000000044815, shard=2 master=00000000000000000000:00000000000000051111 target=00000000000000000000:00000000000000050831, shard=3 master=00000000000000000000:00000000000000044038 target=00000000000000000000:00000000000000043896 2023-05-31T21:35:43.525 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-31T21:35:43.526 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-31T21:35:43.526 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-05-31T21:35:43.526 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: ignoring --setuser ceph since I am not root 2023-05-31T21:35:43.527 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-05-31T21:35:43.527 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2023-05-31T21:35:43.527 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "sync_status": { 2023-05-31T21:35:43.527 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "info": { 2023-05-31T21:35:43.527 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "status": "sync", 2023-05-31T21:35:43.528 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "num_shards": 4, 2023-05-31T21:35:43.528 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "instance_id": 452136274422638824 2023-05-31T21:35:43.528 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2023-05-31T21:35:43.528 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "markers": [ 2023-05-31T21:35:43.529 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2023-05-31T21:35:43.529 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "key": 0, 2023-05-31T21:35:43.529 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "val": { 2023-05-31T21:35:43.529 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "status": "incremental-sync", 2023-05-31T21:35:43.530 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "marker": "00000000000000000000:00000000000000044815", 2023-05-31T21:35:43.530 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "next_step_marker": "", 2023-05-31T21:35:43.530 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "total_entries": 0, 2023-05-31T21:35:43.530 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "pos": 0, 2023-05-31T21:35:43.530 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "timestamp": "2023-05-31T21:21:05.078544Z" 2023-05-31T21:35:43.531 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2023-05-31T21:35:43.531 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2023-05-31T21:35:43.531 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2023-05-31T21:35:43.531 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "key": 1, 2023-05-31T21:35:43.532 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "val": { 2023-05-31T21:35:43.532 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "status": "incremental-sync", 2023-05-31T21:35:43.532 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "marker": "00000000000000000000:00000000000000031499", 2023-05-31T21:35:43.532 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "next_step_marker": "", 2023-05-31T21:35:43.533 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "total_entries": 0, 2023-05-31T21:35:43.533 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "pos": 0, 2023-05-31T21:35:43.533 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "timestamp": "2023-05-31T21:12:40.667485Z" 2023-05-31T21:35:43.533 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2023-05-31T21:35:43.533 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2023-05-31T21:35:43.534 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2023-05-31T21:35:43.534 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "key": 2, 2023-05-31T21:35:43.534 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "val": { 2023-05-31T21:35:43.534 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "status": "incremental-sync", 2023-05-31T21:35:43.535 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "marker": "00000000000000000000:00000000000000050831", 2023-05-31T21:35:43.535 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "next_step_marker": "", 2023-05-31T21:35:43.535 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "total_entries": 55, 2023-05-31T21:35:43.535 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "pos": 0, 2023-05-31T21:35:43.535 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "timestamp": "2023-05-31T21:19:03.876722Z" 2023-05-31T21:35:43.536 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2023-05-31T21:35:43.536 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2023-05-31T21:35:43.536 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2023-05-31T21:35:43.536 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "key": 3, 2023-05-31T21:35:43.537 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "val": { 2023-05-31T21:35:43.537 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "status": "incremental-sync", 2023-05-31T21:35:43.537 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "marker": "00000000000000000000:00000000000000043896", 2023-05-31T21:35:43.537 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "next_step_marker": "", 2023-05-31T21:35:43.538 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "total_entries": 9, 2023-05-31T21:35:43.538 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "pos": 0, 2023-05-31T21:35:43.538 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "timestamp": "2023-05-31T21:12:28.529956Z" 2023-05-31T21:35:43.538 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2023-05-31T21:35:43.539 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2023-05-31T21:35:43.539 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: ] 2023-05-31T21:35:43.539 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2023-05-31T21:35:43.539 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "full_sync": { 2023-05-31T21:35:43.539 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "total": 64, 2023-05-31T21:35:43.540 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "complete": 64 2023-05-31T21:35:43.540 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2023-05-31T21:35:43.540 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "current_time": "2023-05-31T21:22:17Z" 2023-05-31T21:35:43.540 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2023-05-31T21:35:43.541 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current data sync status={ 2023-05-31T21:35:43.541 INFO:tasks.rgw_multisite_tests: "sync_status": { 2023-05-31T21:35:43.541 INFO:tasks.rgw_multisite_tests: "info": { 2023-05-31T21:35:43.541 INFO:tasks.rgw_multisite_tests: "status": "sync", 2023-05-31T21:35:43.542 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2023-05-31T21:35:43.542 INFO:tasks.rgw_multisite_tests: "instance_id": 452136274422638824 2023-05-31T21:35:43.542 INFO:tasks.rgw_multisite_tests: }, 2023-05-31T21:35:43.542 INFO:tasks.rgw_multisite_tests: "markers": [ 2023-05-31T21:35:43.542 INFO:tasks.rgw_multisite_tests: { 2023-05-31T21:35:43.543 INFO:tasks.rgw_multisite_tests: "key": 0, 2023-05-31T21:35:43.543 INFO:tasks.rgw_multisite_tests: "val": { 2023-05-31T21:35:43.543 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2023-05-31T21:35:43.543 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000044815", 2023-05-31T21:35:43.544 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-05-31T21:35:43.544 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-05-31T21:35:43.544 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-05-31T21:35:43.544 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-05-31T21:21:05.078544Z" 2023-05-31T21:35:43.545 INFO:tasks.rgw_multisite_tests: } 2023-05-31T21:35:43.545 INFO:tasks.rgw_multisite_tests: }, 2023-05-31T21:35:43.545 INFO:tasks.rgw_multisite_tests: { 2023-05-31T21:35:43.545 INFO:tasks.rgw_multisite_tests: "key": 1, 2023-05-31T21:35:43.545 INFO:tasks.rgw_multisite_tests: "val": { 2023-05-31T21:35:43.546 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2023-05-31T21:35:43.546 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000031499", 2023-05-31T21:35:43.546 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-05-31T21:35:43.546 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-05-31T21:35:43.547 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-05-31T21:35:43.547 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-05-31T21:12:40.667485Z" 2023-05-31T21:35:43.547 INFO:tasks.rgw_multisite_tests: } 2023-05-31T21:35:43.547 INFO:tasks.rgw_multisite_tests: }, 2023-05-31T21:35:43.547 INFO:tasks.rgw_multisite_tests: { 2023-05-31T21:35:43.548 INFO:tasks.rgw_multisite_tests: "key": 2, 2023-05-31T21:35:43.548 INFO:tasks.rgw_multisite_tests: "val": { 2023-05-31T21:35:43.548 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2023-05-31T21:35:43.548 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000050831", 2023-05-31T21:35:43.549 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-05-31T21:35:43.549 INFO:tasks.rgw_multisite_tests: "total_entries": 55, 2023-05-31T21:35:43.549 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-05-31T21:35:43.549 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-05-31T21:19:03.876722Z" 2023-05-31T21:35:43.550 INFO:tasks.rgw_multisite_tests: } 2023-05-31T21:35:43.550 INFO:tasks.rgw_multisite_tests: }, 2023-05-31T21:35:43.550 INFO:tasks.rgw_multisite_tests: { 2023-05-31T21:35:43.550 INFO:tasks.rgw_multisite_tests: "key": 3, 2023-05-31T21:35:43.550 INFO:tasks.rgw_multisite_tests: "val": { 2023-05-31T21:35:43.551 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2023-05-31T21:35:43.551 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000043896", 2023-05-31T21:35:43.551 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-05-31T21:35:43.551 INFO:tasks.rgw_multisite_tests: "total_entries": 9, 2023-05-31T21:35:43.552 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-05-31T21:35:43.552 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-05-31T21:12:28.529956Z" 2023-05-31T21:35:43.552 INFO:tasks.rgw_multisite_tests: } 2023-05-31T21:35:43.552 INFO:tasks.rgw_multisite_tests: } 2023-05-31T21:35:43.553 INFO:tasks.rgw_multisite_tests: ] 2023-05-31T21:35:43.553 INFO:tasks.rgw_multisite_tests: }, 2023-05-31T21:35:43.553 INFO:tasks.rgw_multisite_tests: "full_sync": { 2023-05-31T21:35:43.553 INFO:tasks.rgw_multisite_tests: "total": 64, 2023-05-31T21:35:43.553 INFO:tasks.rgw_multisite_tests: "complete": 64 2023-05-31T21:35:43.554 INFO:tasks.rgw_multisite_tests: }, 2023-05-31T21:35:43.554 INFO:tasks.rgw_multisite_tests: "current_time": "2023-05-31T21:22:17Z" 2023-05-31T21:35:43.554 INFO:tasks.rgw_multisite_tests:} 2023-05-31T21:35:43.554 INFO:tasks.rgw_multisite_tests: 2023-05-31T21:35:43.555 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000044815', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-05-31T21:21:05.078544Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000031499', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-05-31T21:12:40.667485Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000050831', 'next_step_marker': '', 'total_entries': 55, 'pos': 0, 'timestamp': '2023-05-31T21:19:03.876722Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000043896', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2023-05-31T21:12:28.529956Z'}}] 2023-05-31T21:35:43.555 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '00000000000000000000:00000000000000044955', 1: '00000000000000000000:00000000000000031499', 2: '00000000000000000000:00000000000000051111', 3: '00000000000000000000:00000000000000044038'} 2023-05-31T21:35:43.555 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '00000000000000000000:00000000000000044815', 1: '00000000000000000000:00000000000000031499', 2: '00000000000000000000:00000000000000050831', 3: '00000000000000000000:00000000000000043896'} 2023-05-31T21:35:43.555 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: data of zone a2 behind zone a1: shard=0 master=00000000000000000000:00000000000000044955 target=00000000000000000000:00000000000000044815, shard=2 master=00000000000000000000:00000000000000051111 target=00000000000000000000:00000000000000050831, shard=3 master=00000000000000000000:00000000000000044038 target=00000000000000000000:00000000000000043896 2023-05-31T21:35:43.556 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-31T21:35:43.556 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-31T21:35:43.556 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-05-31T21:35:43.556 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: ignoring --setuser ceph since I am not root 2023-05-31T21:35:43.557 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-05-31T21:35:43.557 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2023-05-31T21:35:43.557 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "sync_status": { 2023-05-31T21:35:43.557 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "info": { 2023-05-31T21:35:43.557 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "status": "sync", 2023-05-31T21:35:43.558 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "num_shards": 4, 2023-05-31T21:35:43.558 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "instance_id": 452136274422638824 2023-05-31T21:35:43.558 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2023-05-31T21:35:43.558 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "markers": [ 2023-05-31T21:35:43.559 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2023-05-31T21:35:43.559 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "key": 0, 2023-05-31T21:35:43.559 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "val": { 2023-05-31T21:35:43.559 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "status": "incremental-sync", 2023-05-31T21:35:43.560 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "marker": "00000000000000000000:00000000000000044815", 2023-05-31T21:35:43.560 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "next_step_marker": "", 2023-05-31T21:35:43.560 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "total_entries": 0, 2023-05-31T21:35:43.560 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "pos": 0, 2023-05-31T21:35:43.560 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "timestamp": "2023-05-31T21:21:05.078544Z" 2023-05-31T21:35:43.561 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2023-05-31T21:35:43.561 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2023-05-31T21:35:43.561 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2023-05-31T21:35:43.561 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "key": 1, 2023-05-31T21:35:43.562 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "val": { 2023-05-31T21:35:43.562 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "status": "incremental-sync", 2023-05-31T21:35:43.562 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "marker": "00000000000000000000:00000000000000031499", 2023-05-31T21:35:43.562 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "next_step_marker": "", 2023-05-31T21:35:43.563 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "total_entries": 0, 2023-05-31T21:35:43.563 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "pos": 0, 2023-05-31T21:35:43.563 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "timestamp": "2023-05-31T21:12:40.667485Z" 2023-05-31T21:35:43.563 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2023-05-31T21:35:43.563 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2023-05-31T21:35:43.564 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2023-05-31T21:35:43.564 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "key": 2, 2023-05-31T21:35:43.564 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "val": { 2023-05-31T21:35:43.564 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "status": "incremental-sync", 2023-05-31T21:35:43.565 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "marker": "00000000000000000000:00000000000000050831", 2023-05-31T21:35:43.565 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "next_step_marker": "", 2023-05-31T21:35:43.565 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "total_entries": 55, 2023-05-31T21:35:43.565 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "pos": 0, 2023-05-31T21:35:43.565 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "timestamp": "2023-05-31T21:19:03.876722Z" 2023-05-31T21:35:43.566 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2023-05-31T21:35:43.566 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2023-05-31T21:35:43.566 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2023-05-31T21:35:43.566 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "key": 3, 2023-05-31T21:35:43.567 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "val": { 2023-05-31T21:35:43.567 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "status": "incremental-sync", 2023-05-31T21:35:43.567 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "marker": "00000000000000000000:00000000000000043896", 2023-05-31T21:35:43.567 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "next_step_marker": "", 2023-05-31T21:35:43.568 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "total_entries": 9, 2023-05-31T21:35:43.568 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "pos": 0, 2023-05-31T21:35:43.568 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "timestamp": "2023-05-31T21:12:28.529956Z" 2023-05-31T21:35:43.568 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2023-05-31T21:35:43.569 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2023-05-31T21:35:43.569 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: ] 2023-05-31T21:35:43.569 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2023-05-31T21:35:43.569 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "full_sync": { 2023-05-31T21:35:43.570 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "total": 64, 2023-05-31T21:35:43.570 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "complete": 64 2023-05-31T21:35:43.570 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2023-05-31T21:35:43.570 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "current_time": "2023-05-31T21:22:22Z" 2023-05-31T21:35:43.570 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2023-05-31T21:35:43.571 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current data sync status={ 2023-05-31T21:35:43.571 INFO:tasks.rgw_multisite_tests: "sync_status": { 2023-05-31T21:35:43.571 INFO:tasks.rgw_multisite_tests: "info": { 2023-05-31T21:35:43.571 INFO:tasks.rgw_multisite_tests: "status": "sync", 2023-05-31T21:35:43.572 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2023-05-31T21:35:43.572 INFO:tasks.rgw_multisite_tests: "instance_id": 452136274422638824 2023-05-31T21:35:43.572 INFO:tasks.rgw_multisite_tests: }, 2023-05-31T21:35:43.572 INFO:tasks.rgw_multisite_tests: "markers": [ 2023-05-31T21:35:43.573 INFO:tasks.rgw_multisite_tests: { 2023-05-31T21:35:43.573 INFO:tasks.rgw_multisite_tests: "key": 0, 2023-05-31T21:35:43.573 INFO:tasks.rgw_multisite_tests: "val": { 2023-05-31T21:35:43.573 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2023-05-31T21:35:43.574 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000044815", 2023-05-31T21:35:43.574 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-05-31T21:35:43.574 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-05-31T21:35:43.574 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-05-31T21:35:43.575 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-05-31T21:21:05.078544Z" 2023-05-31T21:35:43.575 INFO:tasks.rgw_multisite_tests: } 2023-05-31T21:35:43.575 INFO:tasks.rgw_multisite_tests: }, 2023-05-31T21:35:43.575 INFO:tasks.rgw_multisite_tests: { 2023-05-31T21:35:43.575 INFO:tasks.rgw_multisite_tests: "key": 1, 2023-05-31T21:35:43.576 INFO:tasks.rgw_multisite_tests: "val": { 2023-05-31T21:35:43.576 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2023-05-31T21:35:43.576 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000031499", 2023-05-31T21:35:43.576 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-05-31T21:35:43.577 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-05-31T21:35:43.577 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-05-31T21:35:43.577 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-05-31T21:12:40.667485Z" 2023-05-31T21:35:43.577 INFO:tasks.rgw_multisite_tests: } 2023-05-31T21:35:43.578 INFO:tasks.rgw_multisite_tests: }, 2023-05-31T21:35:43.578 INFO:tasks.rgw_multisite_tests: { 2023-05-31T21:35:43.578 INFO:tasks.rgw_multisite_tests: "key": 2, 2023-05-31T21:35:43.578 INFO:tasks.rgw_multisite_tests: "val": { 2023-05-31T21:35:43.579 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2023-05-31T21:35:43.579 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000050831", 2023-05-31T21:35:43.579 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-05-31T21:35:43.579 INFO:tasks.rgw_multisite_tests: "total_entries": 55, 2023-05-31T21:35:43.579 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-05-31T21:35:43.580 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-05-31T21:19:03.876722Z" 2023-05-31T21:35:43.580 INFO:tasks.rgw_multisite_tests: } 2023-05-31T21:35:43.580 INFO:tasks.rgw_multisite_tests: }, 2023-05-31T21:35:43.580 INFO:tasks.rgw_multisite_tests: { 2023-05-31T21:35:43.581 INFO:tasks.rgw_multisite_tests: "key": 3, 2023-05-31T21:35:43.581 INFO:tasks.rgw_multisite_tests: "val": { 2023-05-31T21:35:43.581 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2023-05-31T21:35:43.581 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000043896", 2023-05-31T21:35:43.582 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-05-31T21:35:43.582 INFO:tasks.rgw_multisite_tests: "total_entries": 9, 2023-05-31T21:35:43.582 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-05-31T21:35:43.582 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-05-31T21:12:28.529956Z" 2023-05-31T21:35:43.582 INFO:tasks.rgw_multisite_tests: } 2023-05-31T21:35:43.583 INFO:tasks.rgw_multisite_tests: } 2023-05-31T21:35:43.583 INFO:tasks.rgw_multisite_tests: ] 2023-05-31T21:35:43.583 INFO:tasks.rgw_multisite_tests: }, 2023-05-31T21:35:43.583 INFO:tasks.rgw_multisite_tests: "full_sync": { 2023-05-31T21:35:43.584 INFO:tasks.rgw_multisite_tests: "total": 64, 2023-05-31T21:35:43.584 INFO:tasks.rgw_multisite_tests: "complete": 64 2023-05-31T21:35:43.584 INFO:tasks.rgw_multisite_tests: }, 2023-05-31T21:35:43.584 INFO:tasks.rgw_multisite_tests: "current_time": "2023-05-31T21:22:22Z" 2023-05-31T21:35:43.585 INFO:tasks.rgw_multisite_tests:} 2023-05-31T21:35:43.585 INFO:tasks.rgw_multisite_tests: 2023-05-31T21:35:43.585 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000044815', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-05-31T21:21:05.078544Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000031499', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-05-31T21:12:40.667485Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000050831', 'next_step_marker': '', 'total_entries': 55, 'pos': 0, 'timestamp': '2023-05-31T21:19:03.876722Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000043896', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2023-05-31T21:12:28.529956Z'}}] 2023-05-31T21:35:43.585 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '00000000000000000000:00000000000000044955', 1: '00000000000000000000:00000000000000031499', 2: '00000000000000000000:00000000000000051111', 3: '00000000000000000000:00000000000000044038'} 2023-05-31T21:35:43.585 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '00000000000000000000:00000000000000044815', 1: '00000000000000000000:00000000000000031499', 2: '00000000000000000000:00000000000000050831', 3: '00000000000000000000:00000000000000043896'} 2023-05-31T21:35:43.586 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: data of zone a2 behind zone a1: shard=0 master=00000000000000000000:00000000000000044955 target=00000000000000000000:00000000000000044815, shard=2 master=00000000000000000000:00000000000000051111 target=00000000000000000000:00000000000000050831, shard=3 master=00000000000000000000:00000000000000044038 target=00000000000000000000:00000000000000043896 2023-05-31T21:35:43.586 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-31T21:35:43.586 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-31T21:35:43.586 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-05-31T21:35:43.587 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: ignoring --setuser ceph since I am not root 2023-05-31T21:35:43.587 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-05-31T21:35:43.587 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2023-05-31T21:35:43.587 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "sync_status": { 2023-05-31T21:35:43.588 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "info": { 2023-05-31T21:35:43.588 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "status": "sync", 2023-05-31T21:35:43.588 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "num_shards": 4, 2023-05-31T21:35:43.588 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "instance_id": 452136274422638824 2023-05-31T21:35:43.589 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2023-05-31T21:35:43.589 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "markers": [ 2023-05-31T21:35:43.589 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2023-05-31T21:35:43.589 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "key": 0, 2023-05-31T21:35:43.590 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "val": { 2023-05-31T21:35:43.590 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "status": "incremental-sync", 2023-05-31T21:35:43.590 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "marker": "00000000000000000000:00000000000000044815", 2023-05-31T21:35:43.590 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "next_step_marker": "", 2023-05-31T21:35:43.591 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "total_entries": 0, 2023-05-31T21:35:43.591 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "pos": 0, 2023-05-31T21:35:43.591 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "timestamp": "2023-05-31T21:21:05.078544Z" 2023-05-31T21:35:43.591 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2023-05-31T21:35:43.592 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2023-05-31T21:35:43.592 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2023-05-31T21:35:43.592 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "key": 1, 2023-05-31T21:35:43.592 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "val": { 2023-05-31T21:35:43.593 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "status": "incremental-sync", 2023-05-31T21:35:43.593 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "marker": "00000000000000000000:00000000000000031499", 2023-05-31T21:35:43.593 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "next_step_marker": "", 2023-05-31T21:35:43.593 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "total_entries": 0, 2023-05-31T21:35:43.593 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "pos": 0, 2023-05-31T21:35:43.594 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "timestamp": "2023-05-31T21:12:40.667485Z" 2023-05-31T21:35:43.594 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2023-05-31T21:35:43.594 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2023-05-31T21:35:43.594 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2023-05-31T21:35:43.595 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "key": 2, 2023-05-31T21:35:43.595 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "val": { 2023-05-31T21:35:43.595 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "status": "incremental-sync", 2023-05-31T21:35:43.595 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "marker": "00000000000000000000:00000000000000050831", 2023-05-31T21:35:43.596 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "next_step_marker": "", 2023-05-31T21:35:43.596 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "total_entries": 55, 2023-05-31T21:35:43.596 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "pos": 0, 2023-05-31T21:35:43.596 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "timestamp": "2023-05-31T21:19:03.876722Z" 2023-05-31T21:35:43.596 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2023-05-31T21:35:43.597 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2023-05-31T21:35:43.597 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2023-05-31T21:35:43.597 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "key": 3, 2023-05-31T21:35:43.597 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "val": { 2023-05-31T21:35:43.598 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "status": "incremental-sync", 2023-05-31T21:35:43.598 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "marker": "00000000000000000000:00000000000000043896", 2023-05-31T21:35:43.598 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "next_step_marker": "", 2023-05-31T21:35:43.598 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "total_entries": 9, 2023-05-31T21:35:43.599 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "pos": 0, 2023-05-31T21:35:43.599 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "timestamp": "2023-05-31T21:12:28.529956Z" 2023-05-31T21:35:43.599 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2023-05-31T21:35:43.599 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2023-05-31T21:35:43.600 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: ] 2023-05-31T21:35:43.600 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2023-05-31T21:35:43.600 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "full_sync": { 2023-05-31T21:35:43.600 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "total": 64, 2023-05-31T21:35:43.600 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "complete": 64 2023-05-31T21:35:43.601 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2023-05-31T21:35:43.601 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "current_time": "2023-05-31T21:22:28Z" 2023-05-31T21:35:43.601 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2023-05-31T21:35:43.601 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current data sync status={ 2023-05-31T21:35:43.602 INFO:tasks.rgw_multisite_tests: "sync_status": { 2023-05-31T21:35:43.602 INFO:tasks.rgw_multisite_tests: "info": { 2023-05-31T21:35:43.602 INFO:tasks.rgw_multisite_tests: "status": "sync", 2023-05-31T21:35:43.602 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2023-05-31T21:35:43.603 INFO:tasks.rgw_multisite_tests: "instance_id": 452136274422638824 2023-05-31T21:35:43.603 INFO:tasks.rgw_multisite_tests: }, 2023-05-31T21:35:43.603 INFO:tasks.rgw_multisite_tests: "markers": [ 2023-05-31T21:35:43.603 INFO:tasks.rgw_multisite_tests: { 2023-05-31T21:35:43.603 INFO:tasks.rgw_multisite_tests: "key": 0, 2023-05-31T21:35:43.604 INFO:tasks.rgw_multisite_tests: "val": { 2023-05-31T21:35:43.604 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2023-05-31T21:35:43.604 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000044815", 2023-05-31T21:35:43.604 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-05-31T21:35:43.605 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-05-31T21:35:43.605 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-05-31T21:35:43.605 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-05-31T21:21:05.078544Z" 2023-05-31T21:35:43.605 INFO:tasks.rgw_multisite_tests: } 2023-05-31T21:35:43.606 INFO:tasks.rgw_multisite_tests: }, 2023-05-31T21:35:43.606 INFO:tasks.rgw_multisite_tests: { 2023-05-31T21:35:43.606 INFO:tasks.rgw_multisite_tests: "key": 1, 2023-05-31T21:35:43.606 INFO:tasks.rgw_multisite_tests: "val": { 2023-05-31T21:35:43.606 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2023-05-31T21:35:43.607 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000031499", 2023-05-31T21:35:43.607 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-05-31T21:35:43.607 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-05-31T21:35:43.607 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-05-31T21:35:43.608 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-05-31T21:12:40.667485Z" 2023-05-31T21:35:43.608 INFO:tasks.rgw_multisite_tests: } 2023-05-31T21:35:43.608 INFO:tasks.rgw_multisite_tests: }, 2023-05-31T21:35:43.608 INFO:tasks.rgw_multisite_tests: { 2023-05-31T21:35:43.609 INFO:tasks.rgw_multisite_tests: "key": 2, 2023-05-31T21:35:43.609 INFO:tasks.rgw_multisite_tests: "val": { 2023-05-31T21:35:43.609 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2023-05-31T21:35:43.609 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000050831", 2023-05-31T21:35:43.609 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-05-31T21:35:43.610 INFO:tasks.rgw_multisite_tests: "total_entries": 55, 2023-05-31T21:35:43.610 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-05-31T21:35:43.610 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-05-31T21:19:03.876722Z" 2023-05-31T21:35:43.610 INFO:tasks.rgw_multisite_tests: } 2023-05-31T21:35:43.611 INFO:tasks.rgw_multisite_tests: }, 2023-05-31T21:35:43.611 INFO:tasks.rgw_multisite_tests: { 2023-05-31T21:35:43.611 INFO:tasks.rgw_multisite_tests: "key": 3, 2023-05-31T21:35:43.611 INFO:tasks.rgw_multisite_tests: "val": { 2023-05-31T21:35:43.612 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2023-05-31T21:35:43.612 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000043896", 2023-05-31T21:35:43.612 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-05-31T21:35:43.612 INFO:tasks.rgw_multisite_tests: "total_entries": 9, 2023-05-31T21:35:43.612 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-05-31T21:35:43.613 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-05-31T21:12:28.529956Z" 2023-05-31T21:35:43.613 INFO:tasks.rgw_multisite_tests: } 2023-05-31T21:35:43.613 INFO:tasks.rgw_multisite_tests: } 2023-05-31T21:35:43.613 INFO:tasks.rgw_multisite_tests: ] 2023-05-31T21:35:43.614 INFO:tasks.rgw_multisite_tests: }, 2023-05-31T21:35:43.614 INFO:tasks.rgw_multisite_tests: "full_sync": { 2023-05-31T21:35:43.614 INFO:tasks.rgw_multisite_tests: "total": 64, 2023-05-31T21:35:43.614 INFO:tasks.rgw_multisite_tests: "complete": 64 2023-05-31T21:35:43.615 INFO:tasks.rgw_multisite_tests: }, 2023-05-31T21:35:43.615 INFO:tasks.rgw_multisite_tests: "current_time": "2023-05-31T21:22:28Z" 2023-05-31T21:35:43.615 INFO:tasks.rgw_multisite_tests:} 2023-05-31T21:35:43.615 INFO:tasks.rgw_multisite_tests: 2023-05-31T21:35:43.615 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000044815', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-05-31T21:21:05.078544Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000031499', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-05-31T21:12:40.667485Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000050831', 'next_step_marker': '', 'total_entries': 55, 'pos': 0, 'timestamp': '2023-05-31T21:19:03.876722Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000043896', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2023-05-31T21:12:28.529956Z'}}] 2023-05-31T21:35:43.616 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '00000000000000000000:00000000000000044955', 1: '00000000000000000000:00000000000000031499', 2: '00000000000000000000:00000000000000051111', 3: '00000000000000000000:00000000000000044038'} 2023-05-31T21:35:43.616 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '00000000000000000000:00000000000000044815', 1: '00000000000000000000:00000000000000031499', 2: '00000000000000000000:00000000000000050831', 3: '00000000000000000000:00000000000000043896'} 2023-05-31T21:35:43.616 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: data of zone a2 behind zone a1: shard=0 master=00000000000000000000:00000000000000044955 target=00000000000000000000:00000000000000044815, shard=2 master=00000000000000000000:00000000000000051111 target=00000000000000000000:00000000000000050831, shard=3 master=00000000000000000000:00000000000000044038 target=00000000000000000000:00000000000000043896 2023-05-31T21:35:43.616 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-31T21:35:43.617 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-31T21:35:43.617 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-05-31T21:35:43.617 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: ignoring --setuser ceph since I am not root 2023-05-31T21:35:43.617 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-05-31T21:35:43.618 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2023-05-31T21:35:43.618 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "sync_status": { 2023-05-31T21:35:43.618 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "info": { 2023-05-31T21:35:43.618 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "status": "sync", 2023-05-31T21:35:43.618 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "num_shards": 4, 2023-05-31T21:35:43.619 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "instance_id": 452136274422638824 2023-05-31T21:35:43.619 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2023-05-31T21:35:43.619 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "markers": [ 2023-05-31T21:35:43.619 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2023-05-31T21:35:43.620 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "key": 0, 2023-05-31T21:35:43.620 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "val": { 2023-05-31T21:35:43.620 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "status": "incremental-sync", 2023-05-31T21:35:43.620 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "marker": "00000000000000000000:00000000000000044815", 2023-05-31T21:35:43.621 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "next_step_marker": "", 2023-05-31T21:35:43.621 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "total_entries": 0, 2023-05-31T21:35:43.621 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "pos": 0, 2023-05-31T21:35:43.621 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "timestamp": "2023-05-31T21:21:05.078544Z" 2023-05-31T21:35:43.622 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2023-05-31T21:35:43.622 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2023-05-31T21:35:43.622 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2023-05-31T21:35:43.622 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "key": 1, 2023-05-31T21:35:43.623 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "val": { 2023-05-31T21:35:43.623 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "status": "incremental-sync", 2023-05-31T21:35:43.623 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "marker": "00000000000000000000:00000000000000031499", 2023-05-31T21:35:43.623 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "next_step_marker": "", 2023-05-31T21:35:43.624 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "total_entries": 0, 2023-05-31T21:35:43.624 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "pos": 0, 2023-05-31T21:35:43.624 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "timestamp": "2023-05-31T21:12:40.667485Z" 2023-05-31T21:35:43.624 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2023-05-31T21:35:43.625 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2023-05-31T21:35:43.625 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2023-05-31T21:35:43.625 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "key": 2, 2023-05-31T21:35:43.625 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "val": { 2023-05-31T21:35:43.625 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "status": "incremental-sync", 2023-05-31T21:35:43.626 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "marker": "00000000000000000000:00000000000000050831", 2023-05-31T21:35:43.626 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "next_step_marker": "", 2023-05-31T21:35:43.626 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "total_entries": 55, 2023-05-31T21:35:43.626 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "pos": 0, 2023-05-31T21:35:43.627 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "timestamp": "2023-05-31T21:19:03.876722Z" 2023-05-31T21:35:43.627 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2023-05-31T21:35:43.627 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2023-05-31T21:35:43.627 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2023-05-31T21:35:43.628 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "key": 3, 2023-05-31T21:35:43.628 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "val": { 2023-05-31T21:35:43.628 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "status": "incremental-sync", 2023-05-31T21:35:43.628 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "marker": "00000000000000000000:00000000000000043896", 2023-05-31T21:35:43.629 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "next_step_marker": "", 2023-05-31T21:35:43.629 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "total_entries": 9, 2023-05-31T21:35:43.629 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "pos": 0, 2023-05-31T21:35:43.629 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "timestamp": "2023-05-31T21:12:28.529956Z" 2023-05-31T21:35:43.629 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2023-05-31T21:35:43.630 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2023-05-31T21:35:43.630 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: ] 2023-05-31T21:35:43.630 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2023-05-31T21:35:43.630 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "full_sync": { 2023-05-31T21:35:43.631 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "total": 64, 2023-05-31T21:35:43.631 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "complete": 64 2023-05-31T21:35:43.631 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2023-05-31T21:35:43.631 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "current_time": "2023-05-31T21:22:33Z" 2023-05-31T21:35:43.631 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2023-05-31T21:35:43.632 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current data sync status={ 2023-05-31T21:35:43.632 INFO:tasks.rgw_multisite_tests: "sync_status": { 2023-05-31T21:35:43.632 INFO:tasks.rgw_multisite_tests: "info": { 2023-05-31T21:35:43.633 INFO:tasks.rgw_multisite_tests: "status": "sync", 2023-05-31T21:35:43.633 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2023-05-31T21:35:43.633 INFO:tasks.rgw_multisite_tests: "instance_id": 452136274422638824 2023-05-31T21:35:43.633 INFO:tasks.rgw_multisite_tests: }, 2023-05-31T21:35:43.633 INFO:tasks.rgw_multisite_tests: "markers": [ 2023-05-31T21:35:43.634 INFO:tasks.rgw_multisite_tests: { 2023-05-31T21:35:43.634 INFO:tasks.rgw_multisite_tests: "key": 0, 2023-05-31T21:35:43.634 INFO:tasks.rgw_multisite_tests: "val": { 2023-05-31T21:35:43.634 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2023-05-31T21:35:43.635 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000044815", 2023-05-31T21:35:43.635 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-05-31T21:35:43.635 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-05-31T21:35:43.635 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-05-31T21:35:43.635 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-05-31T21:21:05.078544Z" 2023-05-31T21:35:43.636 INFO:tasks.rgw_multisite_tests: } 2023-05-31T21:35:43.636 INFO:tasks.rgw_multisite_tests: }, 2023-05-31T21:35:43.636 INFO:tasks.rgw_multisite_tests: { 2023-05-31T21:35:43.636 INFO:tasks.rgw_multisite_tests: "key": 1, 2023-05-31T21:35:43.637 INFO:tasks.rgw_multisite_tests: "val": { 2023-05-31T21:35:43.637 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2023-05-31T21:35:43.637 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000031499", 2023-05-31T21:35:43.637 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-05-31T21:35:43.638 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-05-31T21:35:43.638 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-05-31T21:35:43.638 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-05-31T21:12:40.667485Z" 2023-05-31T21:35:43.638 INFO:tasks.rgw_multisite_tests: } 2023-05-31T21:35:43.638 INFO:tasks.rgw_multisite_tests: }, 2023-05-31T21:35:43.639 INFO:tasks.rgw_multisite_tests: { 2023-05-31T21:35:43.639 INFO:tasks.rgw_multisite_tests: "key": 2, 2023-05-31T21:35:43.639 INFO:tasks.rgw_multisite_tests: "val": { 2023-05-31T21:35:43.639 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2023-05-31T21:35:43.640 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000050831", 2023-05-31T21:35:43.640 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-05-31T21:35:43.640 INFO:tasks.rgw_multisite_tests: "total_entries": 55, 2023-05-31T21:35:43.640 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-05-31T21:35:43.641 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-05-31T21:19:03.876722Z" 2023-05-31T21:35:43.641 INFO:tasks.rgw_multisite_tests: } 2023-05-31T21:35:43.641 INFO:tasks.rgw_multisite_tests: }, 2023-05-31T21:35:43.641 INFO:tasks.rgw_multisite_tests: { 2023-05-31T21:35:43.641 INFO:tasks.rgw_multisite_tests: "key": 3, 2023-05-31T21:35:43.642 INFO:tasks.rgw_multisite_tests: "val": { 2023-05-31T21:35:43.642 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2023-05-31T21:35:43.642 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000043896", 2023-05-31T21:35:43.642 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-05-31T21:35:43.643 INFO:tasks.rgw_multisite_tests: "total_entries": 9, 2023-05-31T21:35:43.643 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-05-31T21:35:43.643 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-05-31T21:12:28.529956Z" 2023-05-31T21:35:43.643 INFO:tasks.rgw_multisite_tests: } 2023-05-31T21:35:43.644 INFO:tasks.rgw_multisite_tests: } 2023-05-31T21:35:43.644 INFO:tasks.rgw_multisite_tests: ] 2023-05-31T21:35:43.644 INFO:tasks.rgw_multisite_tests: }, 2023-05-31T21:35:43.644 INFO:tasks.rgw_multisite_tests: "full_sync": { 2023-05-31T21:35:43.644 INFO:tasks.rgw_multisite_tests: "total": 64, 2023-05-31T21:35:43.645 INFO:tasks.rgw_multisite_tests: "complete": 64 2023-05-31T21:35:43.645 INFO:tasks.rgw_multisite_tests: }, 2023-05-31T21:35:43.645 INFO:tasks.rgw_multisite_tests: "current_time": "2023-05-31T21:22:33Z" 2023-05-31T21:35:43.645 INFO:tasks.rgw_multisite_tests:} 2023-05-31T21:35:43.646 INFO:tasks.rgw_multisite_tests: 2023-05-31T21:35:43.646 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000044815', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-05-31T21:21:05.078544Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000031499', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-05-31T21:12:40.667485Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000050831', 'next_step_marker': '', 'total_entries': 55, 'pos': 0, 'timestamp': '2023-05-31T21:19:03.876722Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000043896', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2023-05-31T21:12:28.529956Z'}}] 2023-05-31T21:35:43.646 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '00000000000000000000:00000000000000044955', 1: '00000000000000000000:00000000000000031499', 2: '00000000000000000000:00000000000000051111', 3: '00000000000000000000:00000000000000044038'} 2023-05-31T21:35:43.646 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '00000000000000000000:00000000000000044815', 1: '00000000000000000000:00000000000000031499', 2: '00000000000000000000:00000000000000050831', 3: '00000000000000000000:00000000000000043896'} 2023-05-31T21:35:43.647 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: data of zone a2 behind zone a1: shard=0 master=00000000000000000000:00000000000000044955 target=00000000000000000000:00000000000000044815, shard=2 master=00000000000000000000:00000000000000051111 target=00000000000000000000:00000000000000050831, shard=3 master=00000000000000000000:00000000000000044038 target=00000000000000000000:00000000000000043896 2023-05-31T21:35:43.647 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-31T21:35:43.647 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-31T21:35:43.647 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-05-31T21:35:43.648 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: ignoring --setuser ceph since I am not root 2023-05-31T21:35:43.648 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-05-31T21:35:43.648 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2023-05-31T21:35:43.648 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "sync_status": { 2023-05-31T21:35:43.648 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "info": { 2023-05-31T21:35:43.649 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "status": "sync", 2023-05-31T21:35:43.649 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "num_shards": 4, 2023-05-31T21:35:43.649 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "instance_id": 452136274422638824 2023-05-31T21:35:43.649 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2023-05-31T21:35:43.650 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "markers": [ 2023-05-31T21:35:43.650 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2023-05-31T21:35:43.650 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "key": 0, 2023-05-31T21:35:43.650 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "val": { 2023-05-31T21:35:43.650 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "status": "incremental-sync", 2023-05-31T21:35:43.651 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "marker": "00000000000000000000:00000000000000044815", 2023-05-31T21:35:43.651 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "next_step_marker": "", 2023-05-31T21:35:43.651 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "total_entries": 0, 2023-05-31T21:35:43.651 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "pos": 0, 2023-05-31T21:35:43.652 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "timestamp": "2023-05-31T21:21:05.078544Z" 2023-05-31T21:35:43.652 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2023-05-31T21:35:43.652 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2023-05-31T21:35:43.652 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2023-05-31T21:35:43.653 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "key": 1, 2023-05-31T21:35:43.653 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "val": { 2023-05-31T21:35:43.653 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "status": "incremental-sync", 2023-05-31T21:35:43.653 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "marker": "00000000000000000000:00000000000000031499", 2023-05-31T21:35:43.654 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "next_step_marker": "", 2023-05-31T21:35:43.654 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "total_entries": 0, 2023-05-31T21:35:43.654 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "pos": 0, 2023-05-31T21:35:43.654 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "timestamp": "2023-05-31T21:12:40.667485Z" 2023-05-31T21:35:43.654 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2023-05-31T21:35:43.655 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2023-05-31T21:35:43.655 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2023-05-31T21:35:43.655 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "key": 2, 2023-05-31T21:35:43.655 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "val": { 2023-05-31T21:35:43.656 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "status": "incremental-sync", 2023-05-31T21:35:43.656 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "marker": "00000000000000000000:00000000000000050831", 2023-05-31T21:35:43.656 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "next_step_marker": "", 2023-05-31T21:35:43.656 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "total_entries": 55, 2023-05-31T21:35:43.657 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "pos": 0, 2023-05-31T21:35:43.657 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "timestamp": "2023-05-31T21:19:03.876722Z" 2023-05-31T21:35:43.657 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2023-05-31T21:35:43.657 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2023-05-31T21:35:43.657 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2023-05-31T21:35:43.658 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "key": 3, 2023-05-31T21:35:43.658 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "val": { 2023-05-31T21:35:43.658 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "status": "incremental-sync", 2023-05-31T21:35:43.658 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "marker": "00000000000000000000:00000000000000043896", 2023-05-31T21:35:43.659 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "next_step_marker": "", 2023-05-31T21:35:43.659 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "total_entries": 9, 2023-05-31T21:35:43.659 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "pos": 0, 2023-05-31T21:35:43.659 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "timestamp": "2023-05-31T21:12:28.529956Z" 2023-05-31T21:35:43.660 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2023-05-31T21:35:43.660 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2023-05-31T21:35:43.660 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: ] 2023-05-31T21:35:43.660 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2023-05-31T21:35:43.661 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "full_sync": { 2023-05-31T21:35:43.661 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "total": 64, 2023-05-31T21:35:43.661 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "complete": 64 2023-05-31T21:35:43.661 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2023-05-31T21:35:43.661 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "current_time": "2023-05-31T21:22:38Z" 2023-05-31T21:35:43.662 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2023-05-31T21:35:43.662 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current data sync status={ 2023-05-31T21:35:43.662 INFO:tasks.rgw_multisite_tests: "sync_status": { 2023-05-31T21:35:43.662 INFO:tasks.rgw_multisite_tests: "info": { 2023-05-31T21:35:43.663 INFO:tasks.rgw_multisite_tests: "status": "sync", 2023-05-31T21:35:43.663 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2023-05-31T21:35:43.663 INFO:tasks.rgw_multisite_tests: "instance_id": 452136274422638824 2023-05-31T21:35:43.663 INFO:tasks.rgw_multisite_tests: }, 2023-05-31T21:35:43.663 INFO:tasks.rgw_multisite_tests: "markers": [ 2023-05-31T21:35:43.664 INFO:tasks.rgw_multisite_tests: { 2023-05-31T21:35:43.664 INFO:tasks.rgw_multisite_tests: "key": 0, 2023-05-31T21:35:43.664 INFO:tasks.rgw_multisite_tests: "val": { 2023-05-31T21:35:43.664 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2023-05-31T21:35:43.665 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000044815", 2023-05-31T21:35:43.665 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-05-31T21:35:43.665 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-05-31T21:35:43.665 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-05-31T21:35:43.666 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-05-31T21:21:05.078544Z" 2023-05-31T21:35:43.666 INFO:tasks.rgw_multisite_tests: } 2023-05-31T21:35:43.666 INFO:tasks.rgw_multisite_tests: }, 2023-05-31T21:35:43.666 INFO:tasks.rgw_multisite_tests: { 2023-05-31T21:35:43.667 INFO:tasks.rgw_multisite_tests: "key": 1, 2023-05-31T21:35:43.667 INFO:tasks.rgw_multisite_tests: "val": { 2023-05-31T21:35:43.667 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2023-05-31T21:35:43.667 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000031499", 2023-05-31T21:35:43.667 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-05-31T21:35:43.668 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-05-31T21:35:43.668 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-05-31T21:35:43.668 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-05-31T21:12:40.667485Z" 2023-05-31T21:35:43.668 INFO:tasks.rgw_multisite_tests: } 2023-05-31T21:35:43.669 INFO:tasks.rgw_multisite_tests: }, 2023-05-31T21:35:43.669 INFO:tasks.rgw_multisite_tests: { 2023-05-31T21:35:43.669 INFO:tasks.rgw_multisite_tests: "key": 2, 2023-05-31T21:35:43.669 INFO:tasks.rgw_multisite_tests: "val": { 2023-05-31T21:35:43.670 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2023-05-31T21:35:43.670 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000050831", 2023-05-31T21:35:43.670 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-05-31T21:35:43.670 INFO:tasks.rgw_multisite_tests: "total_entries": 55, 2023-05-31T21:35:43.670 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-05-31T21:35:43.671 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-05-31T21:19:03.876722Z" 2023-05-31T21:35:43.671 INFO:tasks.rgw_multisite_tests: } 2023-05-31T21:35:43.671 INFO:tasks.rgw_multisite_tests: }, 2023-05-31T21:35:43.671 INFO:tasks.rgw_multisite_tests: { 2023-05-31T21:35:43.672 INFO:tasks.rgw_multisite_tests: "key": 3, 2023-05-31T21:35:43.672 INFO:tasks.rgw_multisite_tests: "val": { 2023-05-31T21:35:43.672 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2023-05-31T21:35:43.672 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000043896", 2023-05-31T21:35:43.673 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-05-31T21:35:43.673 INFO:tasks.rgw_multisite_tests: "total_entries": 9, 2023-05-31T21:35:43.673 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-05-31T21:35:43.673 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-05-31T21:12:28.529956Z" 2023-05-31T21:35:43.673 INFO:tasks.rgw_multisite_tests: } 2023-05-31T21:35:43.674 INFO:tasks.rgw_multisite_tests: } 2023-05-31T21:35:43.674 INFO:tasks.rgw_multisite_tests: ] 2023-05-31T21:35:43.674 INFO:tasks.rgw_multisite_tests: }, 2023-05-31T21:35:43.674 INFO:tasks.rgw_multisite_tests: "full_sync": { 2023-05-31T21:35:43.675 INFO:tasks.rgw_multisite_tests: "total": 64, 2023-05-31T21:35:43.675 INFO:tasks.rgw_multisite_tests: "complete": 64 2023-05-31T21:35:43.675 INFO:tasks.rgw_multisite_tests: }, 2023-05-31T21:35:43.675 INFO:tasks.rgw_multisite_tests: "current_time": "2023-05-31T21:22:38Z" 2023-05-31T21:35:43.676 INFO:tasks.rgw_multisite_tests:} 2023-05-31T21:35:43.676 INFO:tasks.rgw_multisite_tests: 2023-05-31T21:35:43.676 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000044815', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-05-31T21:21:05.078544Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000031499', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-05-31T21:12:40.667485Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000050831', 'next_step_marker': '', 'total_entries': 55, 'pos': 0, 'timestamp': '2023-05-31T21:19:03.876722Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000043896', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2023-05-31T21:12:28.529956Z'}}] 2023-05-31T21:35:43.676 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '00000000000000000000:00000000000000044955', 1: '00000000000000000000:00000000000000031499', 2: '00000000000000000000:00000000000000051111', 3: '00000000000000000000:00000000000000044038'} 2023-05-31T21:35:43.676 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '00000000000000000000:00000000000000044815', 1: '00000000000000000000:00000000000000031499', 2: '00000000000000000000:00000000000000050831', 3: '00000000000000000000:00000000000000043896'} 2023-05-31T21:35:43.677 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: data of zone a2 behind zone a1: shard=0 master=00000000000000000000:00000000000000044955 target=00000000000000000000:00000000000000044815, shard=2 master=00000000000000000000:00000000000000051111 target=00000000000000000000:00000000000000050831, shard=3 master=00000000000000000000:00000000000000044038 target=00000000000000000000:00000000000000043896 2023-05-31T21:35:43.677 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-31T21:35:43.677 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-31T21:35:43.677 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-05-31T21:35:43.678 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: ignoring --setuser ceph since I am not root 2023-05-31T21:35:43.678 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-05-31T21:35:43.678 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2023-05-31T21:35:43.678 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "sync_status": { 2023-05-31T21:35:43.679 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "info": { 2023-05-31T21:35:43.679 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "status": "sync", 2023-05-31T21:35:43.679 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "num_shards": 4, 2023-05-31T21:35:43.679 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "instance_id": 452136274422638824 2023-05-31T21:35:43.680 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2023-05-31T21:35:43.680 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "markers": [ 2023-05-31T21:35:43.680 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2023-05-31T21:35:43.680 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "key": 0, 2023-05-31T21:35:43.680 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "val": { 2023-05-31T21:35:43.681 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "status": "incremental-sync", 2023-05-31T21:35:43.681 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "marker": "00000000000000000000:00000000000000044815", 2023-05-31T21:35:43.681 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "next_step_marker": "", 2023-05-31T21:35:43.681 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "total_entries": 0, 2023-05-31T21:35:43.682 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "pos": 0, 2023-05-31T21:35:43.682 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "timestamp": "2023-05-31T21:21:05.078544Z" 2023-05-31T21:35:43.682 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2023-05-31T21:35:43.682 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2023-05-31T21:35:43.683 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2023-05-31T21:35:43.683 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "key": 1, 2023-05-31T21:35:43.683 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "val": { 2023-05-31T21:35:43.683 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "status": "incremental-sync", 2023-05-31T21:35:43.683 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "marker": "00000000000000000000:00000000000000031499", 2023-05-31T21:35:43.684 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "next_step_marker": "", 2023-05-31T21:35:43.684 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "total_entries": 0, 2023-05-31T21:35:43.684 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "pos": 0, 2023-05-31T21:35:43.684 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "timestamp": "2023-05-31T21:12:40.667485Z" 2023-05-31T21:35:43.685 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2023-05-31T21:35:43.685 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2023-05-31T21:35:43.685 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2023-05-31T21:35:43.685 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "key": 2, 2023-05-31T21:35:43.686 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "val": { 2023-05-31T21:35:43.686 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "status": "incremental-sync", 2023-05-31T21:35:43.686 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "marker": "00000000000000000000:00000000000000050831", 2023-05-31T21:35:43.686 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "next_step_marker": "", 2023-05-31T21:35:43.686 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "total_entries": 55, 2023-05-31T21:35:43.687 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "pos": 0, 2023-05-31T21:35:43.687 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "timestamp": "2023-05-31T21:19:03.876722Z" 2023-05-31T21:35:43.687 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2023-05-31T21:35:43.687 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2023-05-31T21:35:43.688 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2023-05-31T21:35:43.688 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "key": 3, 2023-05-31T21:35:43.688 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "val": { 2023-05-31T21:35:43.688 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "status": "incremental-sync", 2023-05-31T21:35:43.689 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "marker": "00000000000000000000:00000000000000043896", 2023-05-31T21:35:43.689 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "next_step_marker": "", 2023-05-31T21:35:43.689 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "total_entries": 9, 2023-05-31T21:35:43.689 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "pos": 0, 2023-05-31T21:35:43.690 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "timestamp": "2023-05-31T21:12:28.529956Z" 2023-05-31T21:35:43.690 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2023-05-31T21:35:43.690 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2023-05-31T21:35:43.690 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: ] 2023-05-31T21:35:43.691 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2023-05-31T21:35:43.691 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "full_sync": { 2023-05-31T21:35:43.691 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "total": 64, 2023-05-31T21:35:43.691 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "complete": 64 2023-05-31T21:35:43.692 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2023-05-31T21:35:43.692 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "current_time": "2023-05-31T21:22:43Z" 2023-05-31T21:35:43.692 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2023-05-31T21:35:43.692 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current data sync status={ 2023-05-31T21:35:43.693 INFO:tasks.rgw_multisite_tests: "sync_status": { 2023-05-31T21:35:43.693 INFO:tasks.rgw_multisite_tests: "info": { 2023-05-31T21:35:43.693 INFO:tasks.rgw_multisite_tests: "status": "sync", 2023-05-31T21:35:43.693 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2023-05-31T21:35:43.693 INFO:tasks.rgw_multisite_tests: "instance_id": 452136274422638824 2023-05-31T21:35:43.694 INFO:tasks.rgw_multisite_tests: }, 2023-05-31T21:35:43.694 INFO:tasks.rgw_multisite_tests: "markers": [ 2023-05-31T21:35:43.694 INFO:tasks.rgw_multisite_tests: { 2023-05-31T21:35:43.694 INFO:tasks.rgw_multisite_tests: "key": 0, 2023-05-31T21:35:43.695 INFO:tasks.rgw_multisite_tests: "val": { 2023-05-31T21:35:43.695 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2023-05-31T21:35:43.695 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000044815", 2023-05-31T21:35:43.695 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-05-31T21:35:43.696 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-05-31T21:35:43.696 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-05-31T21:35:43.696 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-05-31T21:21:05.078544Z" 2023-05-31T21:35:43.696 INFO:tasks.rgw_multisite_tests: } 2023-05-31T21:35:43.696 INFO:tasks.rgw_multisite_tests: }, 2023-05-31T21:35:43.697 INFO:tasks.rgw_multisite_tests: { 2023-05-31T21:35:43.697 INFO:tasks.rgw_multisite_tests: "key": 1, 2023-05-31T21:35:43.697 INFO:tasks.rgw_multisite_tests: "val": { 2023-05-31T21:35:43.697 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2023-05-31T21:35:43.698 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000031499", 2023-05-31T21:35:43.698 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-05-31T21:35:43.698 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-05-31T21:35:43.698 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-05-31T21:35:43.699 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-05-31T21:12:40.667485Z" 2023-05-31T21:35:43.699 INFO:tasks.rgw_multisite_tests: } 2023-05-31T21:35:43.699 INFO:tasks.rgw_multisite_tests: }, 2023-05-31T21:35:43.699 INFO:tasks.rgw_multisite_tests: { 2023-05-31T21:35:43.699 INFO:tasks.rgw_multisite_tests: "key": 2, 2023-05-31T21:35:43.700 INFO:tasks.rgw_multisite_tests: "val": { 2023-05-31T21:35:43.700 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2023-05-31T21:35:43.700 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000050831", 2023-05-31T21:35:43.700 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-05-31T21:35:43.701 INFO:tasks.rgw_multisite_tests: "total_entries": 55, 2023-05-31T21:35:43.701 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-05-31T21:35:43.701 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-05-31T21:19:03.876722Z" 2023-05-31T21:35:43.701 INFO:tasks.rgw_multisite_tests: } 2023-05-31T21:35:43.702 INFO:tasks.rgw_multisite_tests: }, 2023-05-31T21:35:43.702 INFO:tasks.rgw_multisite_tests: { 2023-05-31T21:35:43.702 INFO:tasks.rgw_multisite_tests: "key": 3, 2023-05-31T21:35:43.702 INFO:tasks.rgw_multisite_tests: "val": { 2023-05-31T21:35:43.703 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2023-05-31T21:35:43.703 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000043896", 2023-05-31T21:35:43.703 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-05-31T21:35:43.703 INFO:tasks.rgw_multisite_tests: "total_entries": 9, 2023-05-31T21:35:43.703 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-05-31T21:35:43.704 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-05-31T21:12:28.529956Z" 2023-05-31T21:35:43.704 INFO:tasks.rgw_multisite_tests: } 2023-05-31T21:35:43.704 INFO:tasks.rgw_multisite_tests: } 2023-05-31T21:35:43.704 INFO:tasks.rgw_multisite_tests: ] 2023-05-31T21:35:43.705 INFO:tasks.rgw_multisite_tests: }, 2023-05-31T21:35:43.705 INFO:tasks.rgw_multisite_tests: "full_sync": { 2023-05-31T21:35:43.705 INFO:tasks.rgw_multisite_tests: "total": 64, 2023-05-31T21:35:43.705 INFO:tasks.rgw_multisite_tests: "complete": 64 2023-05-31T21:35:43.706 INFO:tasks.rgw_multisite_tests: }, 2023-05-31T21:35:43.706 INFO:tasks.rgw_multisite_tests: "current_time": "2023-05-31T21:22:43Z" 2023-05-31T21:35:43.706 INFO:tasks.rgw_multisite_tests:} 2023-05-31T21:35:43.706 INFO:tasks.rgw_multisite_tests: 2023-05-31T21:35:43.706 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000044815', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-05-31T21:21:05.078544Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000031499', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-05-31T21:12:40.667485Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000050831', 'next_step_marker': '', 'total_entries': 55, 'pos': 0, 'timestamp': '2023-05-31T21:19:03.876722Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000043896', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2023-05-31T21:12:28.529956Z'}}] 2023-05-31T21:35:43.707 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '00000000000000000000:00000000000000044955', 1: '00000000000000000000:00000000000000031499', 2: '00000000000000000000:00000000000000051111', 3: '00000000000000000000:00000000000000044038'} 2023-05-31T21:35:43.707 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '00000000000000000000:00000000000000044815', 1: '00000000000000000000:00000000000000031499', 2: '00000000000000000000:00000000000000050831', 3: '00000000000000000000:00000000000000043896'} 2023-05-31T21:35:43.707 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: data of zone a2 behind zone a1: shard=0 master=00000000000000000000:00000000000000044955 target=00000000000000000000:00000000000000044815, shard=2 master=00000000000000000000:00000000000000051111 target=00000000000000000000:00000000000000050831, shard=3 master=00000000000000000000:00000000000000044038 target=00000000000000000000:00000000000000043896 2023-05-31T21:35:43.707 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-31T21:35:43.708 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-31T21:35:43.708 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-05-31T21:35:43.708 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: ignoring --setuser ceph since I am not root 2023-05-31T21:35:43.708 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-05-31T21:35:43.709 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2023-05-31T21:35:43.709 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "sync_status": { 2023-05-31T21:35:43.709 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "info": { 2023-05-31T21:35:43.709 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "status": "sync", 2023-05-31T21:35:43.710 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "num_shards": 4, 2023-05-31T21:35:43.710 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "instance_id": 452136274422638824 2023-05-31T21:35:43.710 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2023-05-31T21:35:43.710 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "markers": [ 2023-05-31T21:35:43.710 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2023-05-31T21:35:43.711 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "key": 0, 2023-05-31T21:35:43.711 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "val": { 2023-05-31T21:35:43.711 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "status": "incremental-sync", 2023-05-31T21:35:43.711 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "marker": "00000000000000000000:00000000000000044815", 2023-05-31T21:35:43.712 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "next_step_marker": "", 2023-05-31T21:35:43.712 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "total_entries": 0, 2023-05-31T21:35:43.712 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "pos": 0, 2023-05-31T21:35:43.712 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "timestamp": "2023-05-31T21:21:05.078544Z" 2023-05-31T21:35:43.713 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2023-05-31T21:35:43.713 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2023-05-31T21:35:43.713 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2023-05-31T21:35:43.713 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "key": 1, 2023-05-31T21:35:43.713 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "val": { 2023-05-31T21:35:43.714 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "status": "incremental-sync", 2023-05-31T21:35:43.714 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "marker": "00000000000000000000:00000000000000031499", 2023-05-31T21:35:43.714 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "next_step_marker": "", 2023-05-31T21:35:43.714 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "total_entries": 0, 2023-05-31T21:35:43.715 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "pos": 0, 2023-05-31T21:35:43.715 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "timestamp": "2023-05-31T21:12:40.667485Z" 2023-05-31T21:35:43.715 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2023-05-31T21:35:43.715 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2023-05-31T21:35:43.716 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2023-05-31T21:35:43.716 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "key": 2, 2023-05-31T21:35:43.716 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "val": { 2023-05-31T21:35:43.716 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "status": "incremental-sync", 2023-05-31T21:35:43.717 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "marker": "00000000000000000000:00000000000000050831", 2023-05-31T21:35:43.717 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "next_step_marker": "", 2023-05-31T21:35:43.717 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "total_entries": 55, 2023-05-31T21:35:43.717 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "pos": 0, 2023-05-31T21:35:43.717 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "timestamp": "2023-05-31T21:19:03.876722Z" 2023-05-31T21:35:43.718 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2023-05-31T21:35:43.718 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2023-05-31T21:35:43.718 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2023-05-31T21:35:43.718 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "key": 3, 2023-05-31T21:35:43.719 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "val": { 2023-05-31T21:35:43.719 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "status": "incremental-sync", 2023-05-31T21:35:43.719 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "marker": "00000000000000000000:00000000000000043896", 2023-05-31T21:35:43.719 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "next_step_marker": "", 2023-05-31T21:35:43.719 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "total_entries": 9, 2023-05-31T21:35:43.720 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "pos": 0, 2023-05-31T21:35:43.720 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "timestamp": "2023-05-31T21:12:28.529956Z" 2023-05-31T21:35:43.720 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2023-05-31T21:35:43.720 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2023-05-31T21:35:43.721 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: ] 2023-05-31T21:35:43.721 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2023-05-31T21:35:43.721 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "full_sync": { 2023-05-31T21:35:43.721 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "total": 64, 2023-05-31T21:35:43.722 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "complete": 64 2023-05-31T21:35:43.722 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2023-05-31T21:35:43.722 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "current_time": "2023-05-31T21:22:48Z" 2023-05-31T21:35:43.722 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2023-05-31T21:35:43.722 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current data sync status={ 2023-05-31T21:35:43.723 INFO:tasks.rgw_multisite_tests: "sync_status": { 2023-05-31T21:35:43.723 INFO:tasks.rgw_multisite_tests: "info": { 2023-05-31T21:35:43.723 INFO:tasks.rgw_multisite_tests: "status": "sync", 2023-05-31T21:35:43.723 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2023-05-31T21:35:43.724 INFO:tasks.rgw_multisite_tests: "instance_id": 452136274422638824 2023-05-31T21:35:43.724 INFO:tasks.rgw_multisite_tests: }, 2023-05-31T21:35:43.724 INFO:tasks.rgw_multisite_tests: "markers": [ 2023-05-31T21:35:43.724 INFO:tasks.rgw_multisite_tests: { 2023-05-31T21:35:43.725 INFO:tasks.rgw_multisite_tests: "key": 0, 2023-05-31T21:35:43.725 INFO:tasks.rgw_multisite_tests: "val": { 2023-05-31T21:35:43.725 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2023-05-31T21:35:43.725 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000044815", 2023-05-31T21:35:43.725 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-05-31T21:35:43.726 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-05-31T21:35:43.726 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-05-31T21:35:43.726 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-05-31T21:21:05.078544Z" 2023-05-31T21:35:43.726 INFO:tasks.rgw_multisite_tests: } 2023-05-31T21:35:43.727 INFO:tasks.rgw_multisite_tests: }, 2023-05-31T21:35:43.727 INFO:tasks.rgw_multisite_tests: { 2023-05-31T21:35:43.727 INFO:tasks.rgw_multisite_tests: "key": 1, 2023-05-31T21:35:43.727 INFO:tasks.rgw_multisite_tests: "val": { 2023-05-31T21:35:43.728 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2023-05-31T21:35:43.728 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000031499", 2023-05-31T21:35:43.728 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-05-31T21:35:43.728 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-05-31T21:35:43.729 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-05-31T21:35:43.729 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-05-31T21:12:40.667485Z" 2023-05-31T21:35:43.729 INFO:tasks.rgw_multisite_tests: } 2023-05-31T21:35:43.729 INFO:tasks.rgw_multisite_tests: }, 2023-05-31T21:35:43.729 INFO:tasks.rgw_multisite_tests: { 2023-05-31T21:35:43.730 INFO:tasks.rgw_multisite_tests: "key": 2, 2023-05-31T21:35:43.730 INFO:tasks.rgw_multisite_tests: "val": { 2023-05-31T21:35:43.730 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2023-05-31T21:35:43.730 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000050831", 2023-05-31T21:35:43.731 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-05-31T21:35:43.731 INFO:tasks.rgw_multisite_tests: "total_entries": 55, 2023-05-31T21:35:43.731 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-05-31T21:35:43.731 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-05-31T21:19:03.876722Z" 2023-05-31T21:35:43.731 INFO:tasks.rgw_multisite_tests: } 2023-05-31T21:35:43.732 INFO:tasks.rgw_multisite_tests: }, 2023-05-31T21:35:43.732 INFO:tasks.rgw_multisite_tests: { 2023-05-31T21:35:43.732 INFO:tasks.rgw_multisite_tests: "key": 3, 2023-05-31T21:35:43.732 INFO:tasks.rgw_multisite_tests: "val": { 2023-05-31T21:35:43.733 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2023-05-31T21:35:43.733 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000043896", 2023-05-31T21:35:43.733 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-05-31T21:35:43.733 INFO:tasks.rgw_multisite_tests: "total_entries": 9, 2023-05-31T21:35:43.734 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-05-31T21:35:43.734 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-05-31T21:12:28.529956Z" 2023-05-31T21:35:43.734 INFO:tasks.rgw_multisite_tests: } 2023-05-31T21:35:43.734 INFO:tasks.rgw_multisite_tests: } 2023-05-31T21:35:43.735 INFO:tasks.rgw_multisite_tests: ] 2023-05-31T21:35:43.735 INFO:tasks.rgw_multisite_tests: }, 2023-05-31T21:35:43.735 INFO:tasks.rgw_multisite_tests: "full_sync": { 2023-05-31T21:35:43.735 INFO:tasks.rgw_multisite_tests: "total": 64, 2023-05-31T21:35:43.735 INFO:tasks.rgw_multisite_tests: "complete": 64 2023-05-31T21:35:43.736 INFO:tasks.rgw_multisite_tests: }, 2023-05-31T21:35:43.736 INFO:tasks.rgw_multisite_tests: "current_time": "2023-05-31T21:22:48Z" 2023-05-31T21:35:43.736 INFO:tasks.rgw_multisite_tests:} 2023-05-31T21:35:43.736 INFO:tasks.rgw_multisite_tests: 2023-05-31T21:35:43.737 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000044815', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-05-31T21:21:05.078544Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000031499', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-05-31T21:12:40.667485Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000050831', 'next_step_marker': '', 'total_entries': 55, 'pos': 0, 'timestamp': '2023-05-31T21:19:03.876722Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000043896', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2023-05-31T21:12:28.529956Z'}}] 2023-05-31T21:35:43.737 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '00000000000000000000:00000000000000044955', 1: '00000000000000000000:00000000000000031499', 2: '00000000000000000000:00000000000000051111', 3: '00000000000000000000:00000000000000044038'} 2023-05-31T21:35:43.737 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '00000000000000000000:00000000000000044815', 1: '00000000000000000000:00000000000000031499', 2: '00000000000000000000:00000000000000050831', 3: '00000000000000000000:00000000000000043896'} 2023-05-31T21:35:43.737 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: data of zone a2 behind zone a1: shard=0 master=00000000000000000000:00000000000000044955 target=00000000000000000000:00000000000000044815, shard=2 master=00000000000000000000:00000000000000051111 target=00000000000000000000:00000000000000050831, shard=3 master=00000000000000000000:00000000000000044038 target=00000000000000000000:00000000000000043896 2023-05-31T21:35:43.738 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-31T21:35:43.738 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-31T21:35:43.738 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-05-31T21:35:43.738 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: ignoring --setuser ceph since I am not root 2023-05-31T21:35:43.739 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-05-31T21:35:43.739 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2023-05-31T21:35:43.739 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "sync_status": { 2023-05-31T21:35:43.739 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "info": { 2023-05-31T21:35:43.739 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "status": "sync", 2023-05-31T21:35:43.740 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "num_shards": 4, 2023-05-31T21:35:43.740 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "instance_id": 452136274422638824 2023-05-31T21:35:43.740 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2023-05-31T21:35:43.740 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "markers": [ 2023-05-31T21:35:43.741 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2023-05-31T21:35:43.741 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "key": 0, 2023-05-31T21:35:43.741 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "val": { 2023-05-31T21:35:43.741 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "status": "incremental-sync", 2023-05-31T21:35:43.742 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "marker": "00000000000000000000:00000000000000044815", 2023-05-31T21:35:43.742 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "next_step_marker": "", 2023-05-31T21:35:43.742 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "total_entries": 0, 2023-05-31T21:35:43.742 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "pos": 0, 2023-05-31T21:35:43.742 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "timestamp": "2023-05-31T21:21:05.078544Z" 2023-05-31T21:35:43.743 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2023-05-31T21:35:43.743 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2023-05-31T21:35:43.743 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2023-05-31T21:35:43.743 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "key": 1, 2023-05-31T21:35:43.744 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "val": { 2023-05-31T21:35:43.744 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "status": "incremental-sync", 2023-05-31T21:35:43.744 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "marker": "00000000000000000000:00000000000000031499", 2023-05-31T21:35:43.744 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "next_step_marker": "", 2023-05-31T21:35:43.745 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "total_entries": 0, 2023-05-31T21:35:43.745 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "pos": 0, 2023-05-31T21:35:43.745 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "timestamp": "2023-05-31T21:12:40.667485Z" 2023-05-31T21:35:43.745 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2023-05-31T21:35:43.746 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2023-05-31T21:35:43.746 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2023-05-31T21:35:43.746 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "key": 2, 2023-05-31T21:35:43.746 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "val": { 2023-05-31T21:35:43.746 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "status": "incremental-sync", 2023-05-31T21:35:43.747 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "marker": "00000000000000000000:00000000000000050831", 2023-05-31T21:35:43.747 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "next_step_marker": "", 2023-05-31T21:35:43.747 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "total_entries": 55, 2023-05-31T21:35:43.747 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "pos": 0, 2023-05-31T21:35:43.748 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "timestamp": "2023-05-31T21:19:03.876722Z" 2023-05-31T21:35:43.748 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2023-05-31T21:35:43.748 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2023-05-31T21:35:43.748 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2023-05-31T21:35:43.749 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "key": 3, 2023-05-31T21:35:43.749 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "val": { 2023-05-31T21:35:43.749 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "status": "incremental-sync", 2023-05-31T21:35:43.749 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "marker": "00000000000000000000:00000000000000043896", 2023-05-31T21:35:43.749 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "next_step_marker": "", 2023-05-31T21:35:43.750 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "total_entries": 9, 2023-05-31T21:35:43.750 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "pos": 0, 2023-05-31T21:35:43.750 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "timestamp": "2023-05-31T21:12:28.529956Z" 2023-05-31T21:35:43.750 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2023-05-31T21:35:43.751 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2023-05-31T21:35:43.751 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: ] 2023-05-31T21:35:43.751 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2023-05-31T21:35:43.751 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "full_sync": { 2023-05-31T21:35:43.752 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "total": 64, 2023-05-31T21:35:43.752 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "complete": 64 2023-05-31T21:35:43.752 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2023-05-31T21:35:43.752 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "current_time": "2023-05-31T21:22:53Z" 2023-05-31T21:35:43.753 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2023-05-31T21:35:43.753 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current data sync status={ 2023-05-31T21:35:43.753 INFO:tasks.rgw_multisite_tests: "sync_status": { 2023-05-31T21:35:43.753 INFO:tasks.rgw_multisite_tests: "info": { 2023-05-31T21:35:43.753 INFO:tasks.rgw_multisite_tests: "status": "sync", 2023-05-31T21:35:43.754 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2023-05-31T21:35:43.754 INFO:tasks.rgw_multisite_tests: "instance_id": 452136274422638824 2023-05-31T21:35:43.754 INFO:tasks.rgw_multisite_tests: }, 2023-05-31T21:35:43.754 INFO:tasks.rgw_multisite_tests: "markers": [ 2023-05-31T21:35:43.755 INFO:tasks.rgw_multisite_tests: { 2023-05-31T21:35:43.755 INFO:tasks.rgw_multisite_tests: "key": 0, 2023-05-31T21:35:43.755 INFO:tasks.rgw_multisite_tests: "val": { 2023-05-31T21:35:43.755 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2023-05-31T21:35:43.756 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000044815", 2023-05-31T21:35:43.756 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-05-31T21:35:43.756 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-05-31T21:35:43.756 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-05-31T21:35:43.756 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-05-31T21:21:05.078544Z" 2023-05-31T21:35:43.757 INFO:tasks.rgw_multisite_tests: } 2023-05-31T21:35:43.757 INFO:tasks.rgw_multisite_tests: }, 2023-05-31T21:35:43.757 INFO:tasks.rgw_multisite_tests: { 2023-05-31T21:35:43.757 INFO:tasks.rgw_multisite_tests: "key": 1, 2023-05-31T21:35:43.758 INFO:tasks.rgw_multisite_tests: "val": { 2023-05-31T21:35:43.758 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2023-05-31T21:35:43.758 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000031499", 2023-05-31T21:35:43.758 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-05-31T21:35:43.759 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-05-31T21:35:43.759 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-05-31T21:35:43.759 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-05-31T21:12:40.667485Z" 2023-05-31T21:35:43.759 INFO:tasks.rgw_multisite_tests: } 2023-05-31T21:35:43.759 INFO:tasks.rgw_multisite_tests: }, 2023-05-31T21:35:43.760 INFO:tasks.rgw_multisite_tests: { 2023-05-31T21:35:43.760 INFO:tasks.rgw_multisite_tests: "key": 2, 2023-05-31T21:35:43.760 INFO:tasks.rgw_multisite_tests: "val": { 2023-05-31T21:35:43.760 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2023-05-31T21:35:43.761 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000050831", 2023-05-31T21:35:43.761 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-05-31T21:35:43.761 INFO:tasks.rgw_multisite_tests: "total_entries": 55, 2023-05-31T21:35:43.761 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-05-31T21:35:43.762 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-05-31T21:19:03.876722Z" 2023-05-31T21:35:43.762 INFO:tasks.rgw_multisite_tests: } 2023-05-31T21:35:43.762 INFO:tasks.rgw_multisite_tests: }, 2023-05-31T21:35:43.762 INFO:tasks.rgw_multisite_tests: { 2023-05-31T21:35:43.762 INFO:tasks.rgw_multisite_tests: "key": 3, 2023-05-31T21:35:43.763 INFO:tasks.rgw_multisite_tests: "val": { 2023-05-31T21:35:43.763 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2023-05-31T21:35:43.763 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000043896", 2023-05-31T21:35:43.763 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-05-31T21:35:43.764 INFO:tasks.rgw_multisite_tests: "total_entries": 9, 2023-05-31T21:35:43.764 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-05-31T21:35:43.764 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-05-31T21:12:28.529956Z" 2023-05-31T21:35:43.764 INFO:tasks.rgw_multisite_tests: } 2023-05-31T21:35:43.765 INFO:tasks.rgw_multisite_tests: } 2023-05-31T21:35:43.765 INFO:tasks.rgw_multisite_tests: ] 2023-05-31T21:35:43.765 INFO:tasks.rgw_multisite_tests: }, 2023-05-31T21:35:43.765 INFO:tasks.rgw_multisite_tests: "full_sync": { 2023-05-31T21:35:43.765 INFO:tasks.rgw_multisite_tests: "total": 64, 2023-05-31T21:35:43.766 INFO:tasks.rgw_multisite_tests: "complete": 64 2023-05-31T21:35:43.766 INFO:tasks.rgw_multisite_tests: }, 2023-05-31T21:35:43.766 INFO:tasks.rgw_multisite_tests: "current_time": "2023-05-31T21:22:53Z" 2023-05-31T21:35:43.766 INFO:tasks.rgw_multisite_tests:} 2023-05-31T21:35:43.767 INFO:tasks.rgw_multisite_tests: 2023-05-31T21:35:43.767 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000044815', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-05-31T21:21:05.078544Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000031499', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-05-31T21:12:40.667485Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000050831', 'next_step_marker': '', 'total_entries': 55, 'pos': 0, 'timestamp': '2023-05-31T21:19:03.876722Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000043896', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2023-05-31T21:12:28.529956Z'}}] 2023-05-31T21:35:43.767 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '00000000000000000000:00000000000000044955', 1: '00000000000000000000:00000000000000031499', 2: '00000000000000000000:00000000000000051111', 3: '00000000000000000000:00000000000000044038'} 2023-05-31T21:35:43.767 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '00000000000000000000:00000000000000044815', 1: '00000000000000000000:00000000000000031499', 2: '00000000000000000000:00000000000000050831', 3: '00000000000000000000:00000000000000043896'} 2023-05-31T21:35:43.768 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: data of zone a2 behind zone a1: shard=0 master=00000000000000000000:00000000000000044955 target=00000000000000000000:00000000000000044815, shard=2 master=00000000000000000000:00000000000000051111 target=00000000000000000000:00000000000000050831, shard=3 master=00000000000000000000:00000000000000044038 target=00000000000000000000:00000000000000043896 2023-05-31T21:35:43.768 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-31T21:35:43.768 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-31T21:35:43.768 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-05-31T21:35:43.769 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: ignoring --setuser ceph since I am not root 2023-05-31T21:35:43.769 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-05-31T21:35:43.769 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2023-05-31T21:35:43.769 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "sync_status": { 2023-05-31T21:35:43.769 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "info": { 2023-05-31T21:35:43.770 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "status": "sync", 2023-05-31T21:35:43.770 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "num_shards": 4, 2023-05-31T21:35:43.770 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "instance_id": 452136274422638824 2023-05-31T21:35:43.770 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2023-05-31T21:35:43.771 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "markers": [ 2023-05-31T21:35:43.771 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2023-05-31T21:35:43.771 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "key": 0, 2023-05-31T21:35:43.771 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "val": { 2023-05-31T21:35:43.771 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "status": "incremental-sync", 2023-05-31T21:35:43.772 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "marker": "00000000000000000000:00000000000000044815", 2023-05-31T21:35:43.772 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "next_step_marker": "", 2023-05-31T21:35:43.772 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "total_entries": 0, 2023-05-31T21:35:43.772 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "pos": 0, 2023-05-31T21:35:43.773 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "timestamp": "2023-05-31T21:21:05.078544Z" 2023-05-31T21:35:43.773 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2023-05-31T21:35:43.773 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2023-05-31T21:35:43.773 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2023-05-31T21:35:43.774 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "key": 1, 2023-05-31T21:35:43.774 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "val": { 2023-05-31T21:35:43.774 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "status": "incremental-sync", 2023-05-31T21:35:43.774 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "marker": "00000000000000000000:00000000000000031499", 2023-05-31T21:35:43.775 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "next_step_marker": "", 2023-05-31T21:35:43.775 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "total_entries": 0, 2023-05-31T21:35:43.775 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "pos": 0, 2023-05-31T21:35:43.775 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "timestamp": "2023-05-31T21:12:40.667485Z" 2023-05-31T21:35:43.775 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2023-05-31T21:35:43.776 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2023-05-31T21:35:43.776 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2023-05-31T21:35:43.776 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "key": 2, 2023-05-31T21:35:43.776 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "val": { 2023-05-31T21:35:43.777 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "status": "incremental-sync", 2023-05-31T21:35:43.777 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "marker": "00000000000000000000:00000000000000050831", 2023-05-31T21:35:43.777 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "next_step_marker": "", 2023-05-31T21:35:43.777 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "total_entries": 55, 2023-05-31T21:35:43.778 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "pos": 0, 2023-05-31T21:35:43.778 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "timestamp": "2023-05-31T21:19:03.876722Z" 2023-05-31T21:35:43.778 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2023-05-31T21:35:43.778 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2023-05-31T21:35:43.779 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2023-05-31T21:35:43.779 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "key": 3, 2023-05-31T21:35:43.779 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "val": { 2023-05-31T21:35:43.779 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "status": "incremental-sync", 2023-05-31T21:35:43.779 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "marker": "00000000000000000000:00000000000000043896", 2023-05-31T21:35:43.780 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "next_step_marker": "", 2023-05-31T21:35:43.780 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "total_entries": 9, 2023-05-31T21:35:43.780 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "pos": 0, 2023-05-31T21:35:43.780 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "timestamp": "2023-05-31T21:12:28.529956Z" 2023-05-31T21:35:43.781 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2023-05-31T21:35:43.781 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2023-05-31T21:35:43.781 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: ] 2023-05-31T21:35:43.781 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2023-05-31T21:35:43.782 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "full_sync": { 2023-05-31T21:35:43.782 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "total": 64, 2023-05-31T21:35:43.782 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "complete": 64 2023-05-31T21:35:43.782 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2023-05-31T21:35:43.782 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "current_time": "2023-05-31T21:22:58Z" 2023-05-31T21:35:43.783 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2023-05-31T21:35:43.783 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current data sync status={ 2023-05-31T21:35:43.783 INFO:tasks.rgw_multisite_tests: "sync_status": { 2023-05-31T21:35:43.783 INFO:tasks.rgw_multisite_tests: "info": { 2023-05-31T21:35:43.784 INFO:tasks.rgw_multisite_tests: "status": "sync", 2023-05-31T21:35:43.784 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2023-05-31T21:35:43.784 INFO:tasks.rgw_multisite_tests: "instance_id": 452136274422638824 2023-05-31T21:35:43.784 INFO:tasks.rgw_multisite_tests: }, 2023-05-31T21:35:43.785 INFO:tasks.rgw_multisite_tests: "markers": [ 2023-05-31T21:35:43.785 INFO:tasks.rgw_multisite_tests: { 2023-05-31T21:35:43.785 INFO:tasks.rgw_multisite_tests: "key": 0, 2023-05-31T21:35:43.785 INFO:tasks.rgw_multisite_tests: "val": { 2023-05-31T21:35:43.785 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2023-05-31T21:35:43.786 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000044815", 2023-05-31T21:35:43.786 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-05-31T21:35:43.786 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-05-31T21:35:43.786 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-05-31T21:35:43.787 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-05-31T21:21:05.078544Z" 2023-05-31T21:35:43.787 INFO:tasks.rgw_multisite_tests: } 2023-05-31T21:35:43.787 INFO:tasks.rgw_multisite_tests: }, 2023-05-31T21:35:43.787 INFO:tasks.rgw_multisite_tests: { 2023-05-31T21:35:43.788 INFO:tasks.rgw_multisite_tests: "key": 1, 2023-05-31T21:35:43.788 INFO:tasks.rgw_multisite_tests: "val": { 2023-05-31T21:35:43.788 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2023-05-31T21:35:43.788 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000031499", 2023-05-31T21:35:43.788 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-05-31T21:35:43.789 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-05-31T21:35:43.789 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-05-31T21:35:43.789 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-05-31T21:12:40.667485Z" 2023-05-31T21:35:43.789 INFO:tasks.rgw_multisite_tests: } 2023-05-31T21:35:43.790 INFO:tasks.rgw_multisite_tests: }, 2023-05-31T21:35:43.790 INFO:tasks.rgw_multisite_tests: { 2023-05-31T21:35:43.790 INFO:tasks.rgw_multisite_tests: "key": 2, 2023-05-31T21:35:43.791 INFO:tasks.rgw_multisite_tests: "val": { 2023-05-31T21:35:43.791 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2023-05-31T21:35:43.791 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000050831", 2023-05-31T21:35:43.791 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-05-31T21:35:43.792 INFO:tasks.rgw_multisite_tests: "total_entries": 55, 2023-05-31T21:35:43.792 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-05-31T21:35:43.792 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-05-31T21:19:03.876722Z" 2023-05-31T21:35:43.792 INFO:tasks.rgw_multisite_tests: } 2023-05-31T21:35:43.792 INFO:tasks.rgw_multisite_tests: }, 2023-05-31T21:35:43.793 INFO:tasks.rgw_multisite_tests: { 2023-05-31T21:35:43.793 INFO:tasks.rgw_multisite_tests: "key": 3, 2023-05-31T21:35:43.793 INFO:tasks.rgw_multisite_tests: "val": { 2023-05-31T21:35:43.793 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2023-05-31T21:35:43.794 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000043896", 2023-05-31T21:35:43.794 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-05-31T21:35:43.794 INFO:tasks.rgw_multisite_tests: "total_entries": 9, 2023-05-31T21:35:43.794 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-05-31T21:35:43.795 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-05-31T21:12:28.529956Z" 2023-05-31T21:35:43.795 INFO:tasks.rgw_multisite_tests: } 2023-05-31T21:35:43.795 INFO:tasks.rgw_multisite_tests: } 2023-05-31T21:35:43.795 INFO:tasks.rgw_multisite_tests: ] 2023-05-31T21:35:43.795 INFO:tasks.rgw_multisite_tests: }, 2023-05-31T21:35:43.796 INFO:tasks.rgw_multisite_tests: "full_sync": { 2023-05-31T21:35:43.796 INFO:tasks.rgw_multisite_tests: "total": 64, 2023-05-31T21:35:43.796 INFO:tasks.rgw_multisite_tests: "complete": 64 2023-05-31T21:35:43.796 INFO:tasks.rgw_multisite_tests: }, 2023-05-31T21:35:43.797 INFO:tasks.rgw_multisite_tests: "current_time": "2023-05-31T21:22:58Z" 2023-05-31T21:35:43.797 INFO:tasks.rgw_multisite_tests:} 2023-05-31T21:35:43.797 INFO:tasks.rgw_multisite_tests: 2023-05-31T21:35:43.797 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000044815', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-05-31T21:21:05.078544Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000031499', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-05-31T21:12:40.667485Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000050831', 'next_step_marker': '', 'total_entries': 55, 'pos': 0, 'timestamp': '2023-05-31T21:19:03.876722Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000043896', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2023-05-31T21:12:28.529956Z'}}] 2023-05-31T21:35:43.798 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '00000000000000000000:00000000000000044955', 1: '00000000000000000000:00000000000000031499', 2: '00000000000000000000:00000000000000051111', 3: '00000000000000000000:00000000000000044038'} 2023-05-31T21:35:43.798 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '00000000000000000000:00000000000000044815', 1: '00000000000000000000:00000000000000031499', 2: '00000000000000000000:00000000000000050831', 3: '00000000000000000000:00000000000000043896'} 2023-05-31T21:35:43.798 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: data of zone a2 behind zone a1: shard=0 master=00000000000000000000:00000000000000044955 target=00000000000000000000:00000000000000044815, shard=2 master=00000000000000000000:00000000000000051111 target=00000000000000000000:00000000000000050831, shard=3 master=00000000000000000000:00000000000000044038 target=00000000000000000000:00000000000000043896 2023-05-31T21:35:43.798 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-31T21:35:43.798 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-31T21:35:43.799 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-05-31T21:35:43.799 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: ignoring --setuser ceph since I am not root 2023-05-31T21:35:43.799 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-05-31T21:35:43.799 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2023-05-31T21:35:43.800 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "sync_status": { 2023-05-31T21:35:43.800 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "info": { 2023-05-31T21:35:43.800 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "status": "sync", 2023-05-31T21:35:43.800 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "num_shards": 4, 2023-05-31T21:35:43.801 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "instance_id": 452136274422638824 2023-05-31T21:35:43.801 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2023-05-31T21:35:43.801 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "markers": [ 2023-05-31T21:35:43.801 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2023-05-31T21:35:43.802 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "key": 0, 2023-05-31T21:35:43.802 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "val": { 2023-05-31T21:35:43.802 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "status": "incremental-sync", 2023-05-31T21:35:43.802 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "marker": "00000000000000000000:00000000000000044815", 2023-05-31T21:35:43.803 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "next_step_marker": "", 2023-05-31T21:35:43.803 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "total_entries": 0, 2023-05-31T21:35:43.803 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "pos": 0, 2023-05-31T21:35:43.803 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "timestamp": "2023-05-31T21:21:05.078544Z" 2023-05-31T21:35:43.803 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2023-05-31T21:35:43.804 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2023-05-31T21:35:43.804 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2023-05-31T21:35:43.804 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "key": 1, 2023-05-31T21:35:43.804 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "val": { 2023-05-31T21:35:43.805 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "status": "incremental-sync", 2023-05-31T21:35:43.805 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "marker": "00000000000000000000:00000000000000031499", 2023-05-31T21:35:43.805 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "next_step_marker": "", 2023-05-31T21:35:43.805 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "total_entries": 0, 2023-05-31T21:35:43.806 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "pos": 0, 2023-05-31T21:35:43.806 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "timestamp": "2023-05-31T21:12:40.667485Z" 2023-05-31T21:35:43.806 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2023-05-31T21:35:43.806 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2023-05-31T21:35:43.807 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2023-05-31T21:35:43.807 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "key": 2, 2023-05-31T21:35:43.807 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "val": { 2023-05-31T21:35:43.807 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "status": "incremental-sync", 2023-05-31T21:35:43.807 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "marker": "00000000000000000000:00000000000000050831", 2023-05-31T21:35:43.808 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "next_step_marker": "", 2023-05-31T21:35:43.808 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "total_entries": 55, 2023-05-31T21:35:43.808 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "pos": 0, 2023-05-31T21:35:43.808 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "timestamp": "2023-05-31T21:19:03.876722Z" 2023-05-31T21:35:43.809 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2023-05-31T21:35:43.809 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2023-05-31T21:35:43.809 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2023-05-31T21:35:43.809 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "key": 3, 2023-05-31T21:35:43.810 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "val": { 2023-05-31T21:35:43.810 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "status": "incremental-sync", 2023-05-31T21:35:43.810 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "marker": "00000000000000000000:00000000000000043896", 2023-05-31T21:35:43.810 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "next_step_marker": "", 2023-05-31T21:35:43.810 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "total_entries": 9, 2023-05-31T21:35:43.811 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "pos": 0, 2023-05-31T21:35:43.811 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "timestamp": "2023-05-31T21:12:28.529956Z" 2023-05-31T21:35:43.811 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2023-05-31T21:35:43.811 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2023-05-31T21:35:43.812 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: ] 2023-05-31T21:35:43.812 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2023-05-31T21:35:43.812 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "full_sync": { 2023-05-31T21:35:43.812 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "total": 64, 2023-05-31T21:35:43.813 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "complete": 64 2023-05-31T21:35:43.813 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2023-05-31T21:35:43.813 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "current_time": "2023-05-31T21:23:04Z" 2023-05-31T21:35:43.813 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2023-05-31T21:35:43.813 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current data sync status={ 2023-05-31T21:35:43.814 INFO:tasks.rgw_multisite_tests: "sync_status": { 2023-05-31T21:35:43.814 INFO:tasks.rgw_multisite_tests: "info": { 2023-05-31T21:35:43.814 INFO:tasks.rgw_multisite_tests: "status": "sync", 2023-05-31T21:35:43.814 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2023-05-31T21:35:43.815 INFO:tasks.rgw_multisite_tests: "instance_id": 452136274422638824 2023-05-31T21:35:43.815 INFO:tasks.rgw_multisite_tests: }, 2023-05-31T21:35:43.815 INFO:tasks.rgw_multisite_tests: "markers": [ 2023-05-31T21:35:43.815 INFO:tasks.rgw_multisite_tests: { 2023-05-31T21:35:43.816 INFO:tasks.rgw_multisite_tests: "key": 0, 2023-05-31T21:35:43.816 INFO:tasks.rgw_multisite_tests: "val": { 2023-05-31T21:35:43.816 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2023-05-31T21:35:43.816 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000044815", 2023-05-31T21:35:43.816 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-05-31T21:35:43.817 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-05-31T21:35:43.817 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-05-31T21:35:43.817 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-05-31T21:21:05.078544Z" 2023-05-31T21:35:43.817 INFO:tasks.rgw_multisite_tests: } 2023-05-31T21:35:43.818 INFO:tasks.rgw_multisite_tests: }, 2023-05-31T21:35:43.818 INFO:tasks.rgw_multisite_tests: { 2023-05-31T21:35:43.818 INFO:tasks.rgw_multisite_tests: "key": 1, 2023-05-31T21:35:43.818 INFO:tasks.rgw_multisite_tests: "val": { 2023-05-31T21:35:43.819 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2023-05-31T21:35:43.819 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000031499", 2023-05-31T21:35:43.819 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-05-31T21:35:43.819 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-05-31T21:35:43.819 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-05-31T21:35:43.820 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-05-31T21:12:40.667485Z" 2023-05-31T21:35:43.820 INFO:tasks.rgw_multisite_tests: } 2023-05-31T21:35:43.820 INFO:tasks.rgw_multisite_tests: }, 2023-05-31T21:35:43.820 INFO:tasks.rgw_multisite_tests: { 2023-05-31T21:35:43.821 INFO:tasks.rgw_multisite_tests: "key": 2, 2023-05-31T21:35:43.821 INFO:tasks.rgw_multisite_tests: "val": { 2023-05-31T21:35:43.821 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2023-05-31T21:35:43.821 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000050831", 2023-05-31T21:35:43.822 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-05-31T21:35:43.822 INFO:tasks.rgw_multisite_tests: "total_entries": 55, 2023-05-31T21:35:43.822 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-05-31T21:35:43.822 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-05-31T21:19:03.876722Z" 2023-05-31T21:35:43.823 INFO:tasks.rgw_multisite_tests: } 2023-05-31T21:35:43.823 INFO:tasks.rgw_multisite_tests: }, 2023-05-31T21:35:43.823 INFO:tasks.rgw_multisite_tests: { 2023-05-31T21:35:43.824 INFO:tasks.rgw_multisite_tests: "key": 3, 2023-05-31T21:35:43.824 INFO:tasks.rgw_multisite_tests: "val": { 2023-05-31T21:35:43.824 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2023-05-31T21:35:43.824 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000043896", 2023-05-31T21:35:43.825 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-05-31T21:35:43.825 INFO:tasks.rgw_multisite_tests: "total_entries": 9, 2023-05-31T21:35:43.825 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-05-31T21:35:43.825 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-05-31T21:12:28.529956Z" 2023-05-31T21:35:43.826 INFO:tasks.rgw_multisite_tests: } 2023-05-31T21:35:43.826 INFO:tasks.rgw_multisite_tests: } 2023-05-31T21:35:43.826 INFO:tasks.rgw_multisite_tests: ] 2023-05-31T21:35:43.826 INFO:tasks.rgw_multisite_tests: }, 2023-05-31T21:35:43.827 INFO:tasks.rgw_multisite_tests: "full_sync": { 2023-05-31T21:35:43.827 INFO:tasks.rgw_multisite_tests: "total": 64, 2023-05-31T21:35:43.827 INFO:tasks.rgw_multisite_tests: "complete": 64 2023-05-31T21:35:43.827 INFO:tasks.rgw_multisite_tests: }, 2023-05-31T21:35:43.827 INFO:tasks.rgw_multisite_tests: "current_time": "2023-05-31T21:23:04Z" 2023-05-31T21:35:43.828 INFO:tasks.rgw_multisite_tests:} 2023-05-31T21:35:43.828 INFO:tasks.rgw_multisite_tests: 2023-05-31T21:35:43.828 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000044815', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-05-31T21:21:05.078544Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000031499', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-05-31T21:12:40.667485Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000050831', 'next_step_marker': '', 'total_entries': 55, 'pos': 0, 'timestamp': '2023-05-31T21:19:03.876722Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000043896', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2023-05-31T21:12:28.529956Z'}}] 2023-05-31T21:35:43.828 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '00000000000000000000:00000000000000044955', 1: '00000000000000000000:00000000000000031499', 2: '00000000000000000000:00000000000000051111', 3: '00000000000000000000:00000000000000044038'} 2023-05-31T21:35:43.829 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '00000000000000000000:00000000000000044815', 1: '00000000000000000000:00000000000000031499', 2: '00000000000000000000:00000000000000050831', 3: '00000000000000000000:00000000000000043896'} 2023-05-31T21:35:43.829 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: data of zone a2 behind zone a1: shard=0 master=00000000000000000000:00000000000000044955 target=00000000000000000000:00000000000000044815, shard=2 master=00000000000000000000:00000000000000051111 target=00000000000000000000:00000000000000050831, shard=3 master=00000000000000000000:00000000000000044038 target=00000000000000000000:00000000000000043896 2023-05-31T21:35:43.829 INFO:tasks.rgw_multisite_tests:tasks.rgw.c2.client.1.smithi112.stdout: INFO: 2023-05-31T21:23:07.840+0000 7fecb0821700 -1 WARNING: The OSD cluster is overloaded and struggling to complete ops. You need more capacity to serve this level of demand. 2023-05-31T21:35:43.829 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-31T21:35:43.830 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-31T21:35:43.830 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-05-31T21:35:43.830 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: ignoring --setuser ceph since I am not root 2023-05-31T21:35:43.830 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-05-31T21:35:43.831 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2023-05-31T21:35:43.831 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "sync_status": { 2023-05-31T21:35:43.831 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "info": { 2023-05-31T21:35:43.831 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "status": "sync", 2023-05-31T21:35:43.832 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "num_shards": 4, 2023-05-31T21:35:43.832 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "instance_id": 452136274422638824 2023-05-31T21:35:43.832 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2023-05-31T21:35:43.832 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "markers": [ 2023-05-31T21:35:43.833 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2023-05-31T21:35:43.833 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "key": 0, 2023-05-31T21:35:43.833 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "val": { 2023-05-31T21:35:43.833 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "status": "incremental-sync", 2023-05-31T21:35:43.833 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "marker": "00000000000000000000:00000000000000044955", 2023-05-31T21:35:43.834 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "next_step_marker": "", 2023-05-31T21:35:43.834 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "total_entries": 0, 2023-05-31T21:35:43.834 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "pos": 0, 2023-05-31T21:35:43.834 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "timestamp": "2023-05-31T21:21:35.807414Z" 2023-05-31T21:35:43.835 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2023-05-31T21:35:43.835 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2023-05-31T21:35:43.835 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2023-05-31T21:35:43.835 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "key": 1, 2023-05-31T21:35:43.836 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "val": { 2023-05-31T21:35:43.836 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "status": "incremental-sync", 2023-05-31T21:35:43.836 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "marker": "00000000000000000000:00000000000000031499", 2023-05-31T21:35:43.836 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "next_step_marker": "", 2023-05-31T21:35:43.837 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "total_entries": 0, 2023-05-31T21:35:43.837 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "pos": 0, 2023-05-31T21:35:43.837 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "timestamp": "2023-05-31T21:12:40.667485Z" 2023-05-31T21:35:43.837 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2023-05-31T21:35:43.837 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2023-05-31T21:35:43.838 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2023-05-31T21:35:43.838 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "key": 2, 2023-05-31T21:35:43.838 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "val": { 2023-05-31T21:35:43.838 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "status": "incremental-sync", 2023-05-31T21:35:43.839 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "marker": "00000000000000000000:00000000000000051111", 2023-05-31T21:35:43.839 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "next_step_marker": "", 2023-05-31T21:35:43.839 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "total_entries": 55, 2023-05-31T21:35:43.839 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "pos": 0, 2023-05-31T21:35:43.840 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "timestamp": "2023-05-31T21:21:35.866368Z" 2023-05-31T21:35:43.840 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2023-05-31T21:35:43.840 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2023-05-31T21:35:43.840 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2023-05-31T21:35:43.840 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "key": 3, 2023-05-31T21:35:43.841 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "val": { 2023-05-31T21:35:43.841 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "status": "incremental-sync", 2023-05-31T21:35:43.841 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "marker": "00000000000000000000:00000000000000044038", 2023-05-31T21:35:43.841 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "next_step_marker": "", 2023-05-31T21:35:43.842 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "total_entries": 9, 2023-05-31T21:35:43.842 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "pos": 0, 2023-05-31T21:35:43.842 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "timestamp": "2023-05-31T21:21:35.912562Z" 2023-05-31T21:35:43.842 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2023-05-31T21:35:43.843 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2023-05-31T21:35:43.843 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: ] 2023-05-31T21:35:43.843 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2023-05-31T21:35:43.843 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "full_sync": { 2023-05-31T21:35:43.843 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "total": 64, 2023-05-31T21:35:43.844 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "complete": 64 2023-05-31T21:35:43.844 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2023-05-31T21:35:43.844 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "current_time": "2023-05-31T21:23:09Z" 2023-05-31T21:35:43.844 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2023-05-31T21:35:43.845 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current data sync status={ 2023-05-31T21:35:43.845 INFO:tasks.rgw_multisite_tests: "sync_status": { 2023-05-31T21:35:43.845 INFO:tasks.rgw_multisite_tests: "info": { 2023-05-31T21:35:43.845 INFO:tasks.rgw_multisite_tests: "status": "sync", 2023-05-31T21:35:43.845 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2023-05-31T21:35:43.846 INFO:tasks.rgw_multisite_tests: "instance_id": 452136274422638824 2023-05-31T21:35:43.846 INFO:tasks.rgw_multisite_tests: }, 2023-05-31T21:35:43.846 INFO:tasks.rgw_multisite_tests: "markers": [ 2023-05-31T21:35:43.846 INFO:tasks.rgw_multisite_tests: { 2023-05-31T21:35:43.847 INFO:tasks.rgw_multisite_tests: "key": 0, 2023-05-31T21:35:43.847 INFO:tasks.rgw_multisite_tests: "val": { 2023-05-31T21:35:43.847 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2023-05-31T21:35:43.847 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000044955", 2023-05-31T21:35:43.848 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-05-31T21:35:43.848 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-05-31T21:35:43.848 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-05-31T21:35:43.848 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-05-31T21:21:35.807414Z" 2023-05-31T21:35:43.849 INFO:tasks.rgw_multisite_tests: } 2023-05-31T21:35:43.849 INFO:tasks.rgw_multisite_tests: }, 2023-05-31T21:35:43.849 INFO:tasks.rgw_multisite_tests: { 2023-05-31T21:35:43.849 INFO:tasks.rgw_multisite_tests: "key": 1, 2023-05-31T21:35:43.849 INFO:tasks.rgw_multisite_tests: "val": { 2023-05-31T21:35:43.850 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2023-05-31T21:35:43.850 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000031499", 2023-05-31T21:35:43.850 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-05-31T21:35:43.850 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-05-31T21:35:43.851 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-05-31T21:35:43.851 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-05-31T21:12:40.667485Z" 2023-05-31T21:35:43.851 INFO:tasks.rgw_multisite_tests: } 2023-05-31T21:35:43.851 INFO:tasks.rgw_multisite_tests: }, 2023-05-31T21:35:43.852 INFO:tasks.rgw_multisite_tests: { 2023-05-31T21:35:43.852 INFO:tasks.rgw_multisite_tests: "key": 2, 2023-05-31T21:35:43.852 INFO:tasks.rgw_multisite_tests: "val": { 2023-05-31T21:35:43.852 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2023-05-31T21:35:43.852 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000051111", 2023-05-31T21:35:43.853 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-05-31T21:35:43.853 INFO:tasks.rgw_multisite_tests: "total_entries": 55, 2023-05-31T21:35:43.853 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-05-31T21:35:43.853 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-05-31T21:21:35.866368Z" 2023-05-31T21:35:43.854 INFO:tasks.rgw_multisite_tests: } 2023-05-31T21:35:43.854 INFO:tasks.rgw_multisite_tests: }, 2023-05-31T21:35:43.854 INFO:tasks.rgw_multisite_tests: { 2023-05-31T21:35:43.854 INFO:tasks.rgw_multisite_tests: "key": 3, 2023-05-31T21:35:43.855 INFO:tasks.rgw_multisite_tests: "val": { 2023-05-31T21:35:43.855 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2023-05-31T21:35:43.855 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000044038", 2023-05-31T21:35:43.855 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-05-31T21:35:43.855 INFO:tasks.rgw_multisite_tests: "total_entries": 9, 2023-05-31T21:35:43.856 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-05-31T21:35:43.856 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-05-31T21:21:35.912562Z" 2023-05-31T21:35:43.856 INFO:tasks.rgw_multisite_tests: } 2023-05-31T21:35:43.856 INFO:tasks.rgw_multisite_tests: } 2023-05-31T21:35:43.857 INFO:tasks.rgw_multisite_tests: ] 2023-05-31T21:35:43.857 INFO:tasks.rgw_multisite_tests: }, 2023-05-31T21:35:43.857 INFO:tasks.rgw_multisite_tests: "full_sync": { 2023-05-31T21:35:43.857 INFO:tasks.rgw_multisite_tests: "total": 64, 2023-05-31T21:35:43.858 INFO:tasks.rgw_multisite_tests: "complete": 64 2023-05-31T21:35:43.858 INFO:tasks.rgw_multisite_tests: }, 2023-05-31T21:35:43.858 INFO:tasks.rgw_multisite_tests: "current_time": "2023-05-31T21:23:09Z" 2023-05-31T21:35:43.858 INFO:tasks.rgw_multisite_tests:} 2023-05-31T21:35:43.858 INFO:tasks.rgw_multisite_tests: 2023-05-31T21:35:43.859 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000044955', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-05-31T21:21:35.807414Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000031499', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-05-31T21:12:40.667485Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000051111', 'next_step_marker': '', 'total_entries': 55, 'pos': 0, 'timestamp': '2023-05-31T21:21:35.866368Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000044038', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2023-05-31T21:21:35.912562Z'}}] 2023-05-31T21:35:43.859 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '00000000000000000000:00000000000000044955', 1: '00000000000000000000:00000000000000031499', 2: '00000000000000000000:00000000000000051111', 3: '00000000000000000000:00000000000000044038'} 2023-05-31T21:35:43.859 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '00000000000000000000:00000000000000044955', 1: '00000000000000000000:00000000000000031499', 2: '00000000000000000000:00000000000000051111', 3: '00000000000000000000:00000000000000044038'} 2023-05-31T21:35:43.859 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: INFO: finished data checkpoint for target_zone=a2 source_zone=a1 2023-05-31T21:35:43.860 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/pklxkd-68/ 2023-05-31T21:35:43.860 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/pklxkd-68/ 2023-05-31T21:35:43.860 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-05-31T21:35:43.860 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /pklxkd-68/ 2023-05-31T21:35:43.861 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-05-31T21:35:43.861 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-05-31T21:35:43.861 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T21:35:43.861 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-05-31T21:35:43.862 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-05-31T21:35:43.862 INFO:tasks.rgw_multisite_tests:boto: DEBUG: establishing HTTP connection: kwargs={'timeout': 70, 'port': 8001} 2023-05-31T21:35:43.862 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-05-31T21:35:43.862 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-05-31T21:35:43.862 INFO:tasks.rgw_multisite_tests:HEAD 2023-05-31T21:35:43.863 INFO:tasks.rgw_multisite_tests: 2023-05-31T21:35:43.863 INFO:tasks.rgw_multisite_tests: 2023-05-31T21:35:43.863 INFO:tasks.rgw_multisite_tests:Wed, 31 May 2023 21:23:09 GMT 2023-05-31T21:35:43.863 INFO:tasks.rgw_multisite_tests:/pklxkd-68/ 2023-05-31T21:35:43.864 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-05-31T21:35:43.864 INFO:tasks.rgw_multisite_tests:AWS Q80H11OLNU1S48PPYL26:VSZdDti3zILUSUUi9mHLWMzr1/w= 2023-05-31T21:35:43.864 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 21:23:09 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:VSZdDti3zILUSUUi9mHLWMzr1/w=', 'Content-Length': '0'} 2023-05-31T21:35:43.864 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '2'), ('X-RGW-Bytes-Used', '12'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000c55aa408ee276086-006477babd-2423-a2'), ('Content-Length', '0'), ('Date', 'Wed, 31 May 2023 21:23:09 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:35:43.865 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/pklxkd-68/obj1 2023-05-31T21:35:43.865 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/pklxkd-68/obj1 2023-05-31T21:35:43.865 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-05-31T21:35:43.865 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /pklxkd-68/obj1 2023-05-31T21:35:43.866 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-05-31T21:35:43.866 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-05-31T21:35:43.866 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T21:35:43.866 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-05-31T21:35:43.866 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-05-31T21:35:43.867 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-05-31T21:35:43.867 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-05-31T21:35:43.867 INFO:tasks.rgw_multisite_tests:HEAD 2023-05-31T21:35:43.867 INFO:tasks.rgw_multisite_tests: 2023-05-31T21:35:43.868 INFO:tasks.rgw_multisite_tests: 2023-05-31T21:35:43.868 INFO:tasks.rgw_multisite_tests:Wed, 31 May 2023 21:23:09 GMT 2023-05-31T21:35:43.868 INFO:tasks.rgw_multisite_tests:/pklxkd-68/obj1 2023-05-31T21:35:43.868 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-05-31T21:35:43.869 INFO:tasks.rgw_multisite_tests:AWS Q80H11OLNU1S48PPYL26:gNfHIUa1ogwkHtXeBd5lOWoE1Cc= 2023-05-31T21:35:43.869 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 21:23:09 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:gNfHIUa1ogwkHtXeBd5lOWoE1Cc=', 'Content-Length': '0'} 2023-05-31T21:35:43.869 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 21:21:35 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000064f451063416ebe-006477babd-2423-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 21:23:09 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:35:43.869 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/pklxkd-68/obj1 2023-05-31T21:35:43.869 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/pklxkd-68/obj1 2023-05-31T21:35:43.870 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-05-31T21:35:43.870 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /pklxkd-68/obj1 2023-05-31T21:35:43.870 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-05-31T21:35:43.870 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-05-31T21:35:43.871 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T21:35:43.871 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-05-31T21:35:43.871 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-05-31T21:35:43.871 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-05-31T21:35:43.872 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-05-31T21:35:43.872 INFO:tasks.rgw_multisite_tests:GET 2023-05-31T21:35:43.872 INFO:tasks.rgw_multisite_tests: 2023-05-31T21:35:43.872 INFO:tasks.rgw_multisite_tests: 2023-05-31T21:35:43.872 INFO:tasks.rgw_multisite_tests:Wed, 31 May 2023 21:23:09 GMT 2023-05-31T21:35:43.873 INFO:tasks.rgw_multisite_tests:/pklxkd-68/obj1 2023-05-31T21:35:43.873 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-05-31T21:35:43.873 INFO:tasks.rgw_multisite_tests:AWS Q80H11OLNU1S48PPYL26:ffZk0/JGsZwpTv7KaXovK4n7aGA= 2023-05-31T21:35:43.873 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 21:23:09 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:ffZk0/JGsZwpTv7KaXovK4n7aGA=', 'Content-Length': '0'} 2023-05-31T21:35:43.874 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 21:21:35 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000bd9595fb58ffdcda-006477babd-2423-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 21:23:09 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:35:43.874 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/pklxkd-68/obj2 2023-05-31T21:35:43.874 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/pklxkd-68/obj2 2023-05-31T21:35:43.874 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-05-31T21:35:43.875 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /pklxkd-68/obj2 2023-05-31T21:35:43.875 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-05-31T21:35:43.875 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-05-31T21:35:43.875 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T21:35:43.875 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-05-31T21:35:43.876 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-05-31T21:35:43.876 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-05-31T21:35:43.876 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-05-31T21:35:43.876 INFO:tasks.rgw_multisite_tests:HEAD 2023-05-31T21:35:43.877 INFO:tasks.rgw_multisite_tests: 2023-05-31T21:35:43.877 INFO:tasks.rgw_multisite_tests: 2023-05-31T21:35:43.877 INFO:tasks.rgw_multisite_tests:Wed, 31 May 2023 21:23:09 GMT 2023-05-31T21:35:43.877 INFO:tasks.rgw_multisite_tests:/pklxkd-68/obj2 2023-05-31T21:35:43.877 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-05-31T21:35:43.878 INFO:tasks.rgw_multisite_tests:AWS Q80H11OLNU1S48PPYL26:lBtr+2sQG/Fk8BzkadV/5iEzqEI= 2023-05-31T21:35:43.878 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 21:23:09 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:lBtr+2sQG/Fk8BzkadV/5iEzqEI=', 'Content-Length': '0'} 2023-05-31T21:35:43.878 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '219'), ('x-amz-request-id', 'tx0000091a16af7b267f2f6-006477babd-2423-a2'), ('Accept-Ranges', 'bytes'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 21:23:09 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:35:43.878 INFO:tasks.rgw_multisite_tests:--------------------- >> end captured logging << --------------------- 2023-05-31T21:35:43.879 INFO:tasks.rgw_multisite_tests: 2023-05-31T21:35:43.879 INFO:tasks.rgw_multisite_tests:====================================================================== 2023-05-31T21:35:43.879 INFO:tasks.rgw_multisite_tests:FAIL: test_sync_multiple_buckets_to_single: 2023-05-31T21:35:43.880 INFO:tasks.rgw_multisite_tests:---------------------------------------------------------------------- 2023-05-31T21:35:43.882 INFO:tasks.rgw_multisite_tests:Traceback (most recent call last): 2023-05-31T21:35:43.882 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_teuthology_fa97ddd17d12e6d74b49196948f66e54fa009130/virtualenv/lib/python3.8/site-packages/nose/case.py", line 198, in runTest 2023-05-31T21:35:43.882 INFO:tasks.rgw_multisite_tests: self.test(*self.arg) 2023-05-31T21:35:43.882 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/github.com_ceph_ceph-c_2870bf456d3115490c153abce9d47ac5c043b65e/qa/../src/test/rgw/rgw_multi/tests.py", line 2745, in test_sync_multiple_buckets_to_single 2023-05-31T21:35:43.883 INFO:tasks.rgw_multisite_tests: check_objects_exist(bucket, objnamesD, content) 2023-05-31T21:35:43.883 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/github.com_ceph_ceph-c_2870bf456d3115490c153abce9d47ac5c043b65e/qa/../src/test/rgw/rgw_multi/tests.py", line 2047, in check_objects_exist 2023-05-31T21:35:43.883 INFO:tasks.rgw_multisite_tests: check_object_exists(bucket, objname, content) 2023-05-31T21:35:43.883 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/github.com_ceph_ceph-c_2870bf456d3115490c153abce9d47ac5c043b65e/qa/../src/test/rgw/rgw_multi/tests.py", line 2041, in check_object_exists 2023-05-31T21:35:43.884 INFO:tasks.rgw_multisite_tests: assert_not_equal(k, None) 2023-05-31T21:35:43.884 INFO:tasks.rgw_multisite_tests:AssertionError: None == None 2023-05-31T21:35:43.884 INFO:tasks.rgw_multisite_tests:-------------------- >> begin captured logging << -------------------- 2023-05-31T21:35:43.884 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-31T21:35:43.885 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-31T21:35:43.885 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-05-31T21:35:43.885 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: ignoring --setuser ceph since I am not root 2023-05-31T21:35:43.885 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-05-31T21:35:43.885 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: No --period given, using current period=2e6123d2-d695-4350-9a01-61a455bc782b 2023-05-31T21:35:43.886 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: [ 2023-05-31T21:35:43.886 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2023-05-31T21:35:43.886 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "marker": "1_1685566501.757183_260.1", 2023-05-31T21:35:43.886 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "last_update": "2023-05-31T20:55:01.757183Z" 2023-05-31T21:35:43.887 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2023-05-31T21:35:43.887 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2023-05-31T21:35:43.887 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "marker": "1_1685566501.735717_251.1", 2023-05-31T21:35:43.887 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "last_update": "2023-05-31T20:55:01.735717Z" 2023-05-31T21:35:43.888 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2023-05-31T21:35:43.888 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2023-05-31T21:35:43.888 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "marker": "1_1685562970.274572_93.1", 2023-05-31T21:35:43.888 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "last_update": "2023-05-31T19:56:10.274572Z" 2023-05-31T21:35:43.889 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2023-05-31T21:35:43.889 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2023-05-31T21:35:43.889 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "marker": "1_1685568473.966785_1006.1", 2023-05-31T21:35:43.889 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "last_update": "2023-05-31T21:27:53.966785Z" 2023-05-31T21:35:43.889 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2023-05-31T21:35:43.890 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: ] 2023-05-31T21:35:43.890 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: master meta markers={0: '1_1685566501.757183_260.1', 1: '1_1685566501.735717_251.1', 2: '1_1685562970.274572_93.1', 3: '1_1685568473.966785_1006.1'} 2023-05-31T21:35:43.890 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: INFO: starting meta checkpoint for zone=a2 2023-05-31T21:35:43.890 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-31T21:35:43.891 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-31T21:35:43.891 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-05-31T21:35:43.891 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: ignoring --setuser ceph since I am not root 2023-05-31T21:35:43.891 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-05-31T21:35:43.892 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: 2023-05-31T21:27:54.338+0000 7f3a387d8780 20 RGW-SYNC:meta: read sync status 2023-05-31T21:35:43.892 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: 2023-05-31T21:27:54.338+0000 7f3a387d8780 20 RGW-SYNC:meta: read sync status 2023-05-31T21:35:43.892 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2023-05-31T21:35:43.892 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "sync_status": { 2023-05-31T21:35:43.893 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "info": { 2023-05-31T21:35:43.893 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "status": "sync", 2023-05-31T21:35:43.893 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "num_shards": 4, 2023-05-31T21:35:43.893 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "period": "2e6123d2-d695-4350-9a01-61a455bc782b", 2023-05-31T21:35:43.894 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "realm_epoch": 2 2023-05-31T21:35:43.894 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2023-05-31T21:35:43.894 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "markers": [ 2023-05-31T21:35:43.894 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2023-05-31T21:35:43.894 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "key": 0, 2023-05-31T21:35:43.895 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "val": { 2023-05-31T21:35:43.895 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "state": 1, 2023-05-31T21:35:43.895 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "marker": "1_1685566501.757183_260.1", 2023-05-31T21:35:43.895 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "next_step_marker": "", 2023-05-31T21:35:43.896 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "total_entries": 1, 2023-05-31T21:35:43.896 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "pos": 0, 2023-05-31T21:35:43.896 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "timestamp": "2023-05-31T20:55:01.757183Z", 2023-05-31T21:35:43.896 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "realm_epoch": 2 2023-05-31T21:35:43.897 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2023-05-31T21:35:43.897 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2023-05-31T21:35:43.897 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2023-05-31T21:35:43.897 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "key": 1, 2023-05-31T21:35:43.898 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "val": { 2023-05-31T21:35:43.898 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "state": 1, 2023-05-31T21:35:43.898 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "marker": "1_1685566501.735717_251.1", 2023-05-31T21:35:43.898 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "next_step_marker": "", 2023-05-31T21:35:43.898 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "total_entries": 0, 2023-05-31T21:35:43.899 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "pos": 0, 2023-05-31T21:35:43.899 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "timestamp": "2023-05-31T20:55:01.735717Z", 2023-05-31T21:35:43.899 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "realm_epoch": 2 2023-05-31T21:35:43.899 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2023-05-31T21:35:43.900 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2023-05-31T21:35:43.900 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2023-05-31T21:35:43.900 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "key": 2, 2023-05-31T21:35:43.900 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "val": { 2023-05-31T21:35:43.901 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "state": 1, 2023-05-31T21:35:43.901 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "marker": "1_1685562970.274572_93.1", 2023-05-31T21:35:43.901 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "next_step_marker": "", 2023-05-31T21:35:43.901 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "total_entries": 0, 2023-05-31T21:35:43.901 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "pos": 0, 2023-05-31T21:35:43.902 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "timestamp": "2023-05-31T19:56:10.274572Z", 2023-05-31T21:35:43.902 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "realm_epoch": 2 2023-05-31T21:35:43.902 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2023-05-31T21:35:43.902 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2023-05-31T21:35:43.903 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2023-05-31T21:35:43.903 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "key": 3, 2023-05-31T21:35:43.903 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "val": { 2023-05-31T21:35:43.903 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "state": 1, 2023-05-31T21:35:43.904 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "marker": "1_1685568452.432716_1004.1", 2023-05-31T21:35:43.904 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "next_step_marker": "", 2023-05-31T21:35:43.904 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "total_entries": 0, 2023-05-31T21:35:43.904 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "pos": 0, 2023-05-31T21:35:43.905 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "timestamp": "2023-05-31T21:27:32.432716Z", 2023-05-31T21:35:43.905 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "realm_epoch": 2 2023-05-31T21:35:43.905 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2023-05-31T21:35:43.905 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2023-05-31T21:35:43.905 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: ] 2023-05-31T21:35:43.906 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2023-05-31T21:35:43.906 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "full_sync": { 2023-05-31T21:35:43.906 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "total": 1, 2023-05-31T21:35:43.906 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "complete": 1 2023-05-31T21:35:43.907 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2023-05-31T21:35:43.907 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "current_time": "2023-05-31T21:27:54Z" 2023-05-31T21:35:43.907 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2023-05-31T21:35:43.907 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2023-05-31T21:35:43.907 INFO:tasks.rgw_multisite_tests: "sync_status": { 2023-05-31T21:35:43.908 INFO:tasks.rgw_multisite_tests: "info": { 2023-05-31T21:35:43.908 INFO:tasks.rgw_multisite_tests: "status": "sync", 2023-05-31T21:35:43.908 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2023-05-31T21:35:43.908 INFO:tasks.rgw_multisite_tests: "period": "2e6123d2-d695-4350-9a01-61a455bc782b", 2023-05-31T21:35:43.909 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-05-31T21:35:43.909 INFO:tasks.rgw_multisite_tests: }, 2023-05-31T21:35:43.909 INFO:tasks.rgw_multisite_tests: "markers": [ 2023-05-31T21:35:43.909 INFO:tasks.rgw_multisite_tests: { 2023-05-31T21:35:43.910 INFO:tasks.rgw_multisite_tests: "key": 0, 2023-05-31T21:35:43.910 INFO:tasks.rgw_multisite_tests: "val": { 2023-05-31T21:35:43.910 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-05-31T21:35:43.910 INFO:tasks.rgw_multisite_tests: "marker": "1_1685566501.757183_260.1", 2023-05-31T21:35:43.910 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-05-31T21:35:43.911 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2023-05-31T21:35:43.911 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-05-31T21:35:43.911 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-05-31T20:55:01.757183Z", 2023-05-31T21:35:43.911 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-05-31T21:35:43.912 INFO:tasks.rgw_multisite_tests: } 2023-05-31T21:35:43.912 INFO:tasks.rgw_multisite_tests: }, 2023-05-31T21:35:43.912 INFO:tasks.rgw_multisite_tests: { 2023-05-31T21:35:43.912 INFO:tasks.rgw_multisite_tests: "key": 1, 2023-05-31T21:35:43.913 INFO:tasks.rgw_multisite_tests: "val": { 2023-05-31T21:35:43.913 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-05-31T21:35:43.913 INFO:tasks.rgw_multisite_tests: "marker": "1_1685566501.735717_251.1", 2023-05-31T21:35:43.913 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-05-31T21:35:43.913 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-05-31T21:35:43.914 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-05-31T21:35:43.914 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-05-31T20:55:01.735717Z", 2023-05-31T21:35:43.914 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-05-31T21:35:43.914 INFO:tasks.rgw_multisite_tests: } 2023-05-31T21:35:43.915 INFO:tasks.rgw_multisite_tests: }, 2023-05-31T21:35:43.915 INFO:tasks.rgw_multisite_tests: { 2023-05-31T21:35:43.915 INFO:tasks.rgw_multisite_tests: "key": 2, 2023-05-31T21:35:43.915 INFO:tasks.rgw_multisite_tests: "val": { 2023-05-31T21:35:43.916 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-05-31T21:35:43.916 INFO:tasks.rgw_multisite_tests: "marker": "1_1685562970.274572_93.1", 2023-05-31T21:35:43.916 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-05-31T21:35:43.916 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-05-31T21:35:43.916 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-05-31T21:35:43.917 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-05-31T19:56:10.274572Z", 2023-05-31T21:35:43.917 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-05-31T21:35:43.917 INFO:tasks.rgw_multisite_tests: } 2023-05-31T21:35:43.917 INFO:tasks.rgw_multisite_tests: }, 2023-05-31T21:35:43.918 INFO:tasks.rgw_multisite_tests: { 2023-05-31T21:35:43.918 INFO:tasks.rgw_multisite_tests: "key": 3, 2023-05-31T21:35:43.918 INFO:tasks.rgw_multisite_tests: "val": { 2023-05-31T21:35:43.918 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-05-31T21:35:43.919 INFO:tasks.rgw_multisite_tests: "marker": "1_1685568452.432716_1004.1", 2023-05-31T21:35:43.919 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-05-31T21:35:43.919 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-05-31T21:35:43.919 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-05-31T21:35:43.919 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-05-31T21:27:32.432716Z", 2023-05-31T21:35:43.920 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-05-31T21:35:43.920 INFO:tasks.rgw_multisite_tests: } 2023-05-31T21:35:43.920 INFO:tasks.rgw_multisite_tests: } 2023-05-31T21:35:43.920 INFO:tasks.rgw_multisite_tests: ] 2023-05-31T21:35:43.921 INFO:tasks.rgw_multisite_tests: }, 2023-05-31T21:35:43.921 INFO:tasks.rgw_multisite_tests: "full_sync": { 2023-05-31T21:35:43.921 INFO:tasks.rgw_multisite_tests: "total": 1, 2023-05-31T21:35:43.921 INFO:tasks.rgw_multisite_tests: "complete": 1 2023-05-31T21:35:43.921 INFO:tasks.rgw_multisite_tests: }, 2023-05-31T21:35:43.922 INFO:tasks.rgw_multisite_tests: "current_time": "2023-05-31T21:27:54Z" 2023-05-31T21:35:43.922 INFO:tasks.rgw_multisite_tests:} 2023-05-31T21:35:43.922 INFO:tasks.rgw_multisite_tests: 2023-05-31T21:35:43.922 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1685566501.757183_260.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-05-31T20:55:01.757183Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1685566501.735717_251.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-05-31T20:55:01.735717Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1685562970.274572_93.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-05-31T19:56:10.274572Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1685568452.432716_1004.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-05-31T21:27:32.432716Z', 'realm_epoch': 2}}] 2023-05-31T21:35:43.923 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1685566501.757183_260.1', 1: '1_1685566501.735717_251.1', 2: '1_1685562970.274572_93.1', 3: '1_1685568473.966785_1006.1'} 2023-05-31T21:35:43.923 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1685566501.757183_260.1', 1: '1_1685566501.735717_251.1', 2: '1_1685562970.274572_93.1', 3: '1_1685568452.432716_1004.1'} 2023-05-31T21:35:43.923 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone a2 behind master: shard=3 master=1_1685568473.966785_1006.1 target=1_1685568452.432716_1004.1 2023-05-31T21:35:43.923 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-31T21:35:43.924 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-31T21:35:43.924 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-05-31T21:35:43.924 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: ignoring --setuser ceph since I am not root 2023-05-31T21:35:43.924 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-05-31T21:35:43.925 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: 2023-05-31T21:27:59.475+0000 7f71a0c90780 20 RGW-SYNC:meta: read sync status 2023-05-31T21:35:43.925 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: 2023-05-31T21:27:59.476+0000 7f71a0c90780 20 RGW-SYNC:meta: read sync status 2023-05-31T21:35:43.925 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2023-05-31T21:35:43.925 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "sync_status": { 2023-05-31T21:35:43.925 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "info": { 2023-05-31T21:35:43.926 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "status": "sync", 2023-05-31T21:35:43.926 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "num_shards": 4, 2023-05-31T21:35:43.926 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "period": "2e6123d2-d695-4350-9a01-61a455bc782b", 2023-05-31T21:35:43.926 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "realm_epoch": 2 2023-05-31T21:35:43.927 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2023-05-31T21:35:43.927 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "markers": [ 2023-05-31T21:35:43.927 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2023-05-31T21:35:43.927 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "key": 0, 2023-05-31T21:35:43.928 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "val": { 2023-05-31T21:35:43.928 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "state": 1, 2023-05-31T21:35:43.928 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "marker": "1_1685566501.757183_260.1", 2023-05-31T21:35:43.928 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "next_step_marker": "", 2023-05-31T21:35:43.928 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "total_entries": 1, 2023-05-31T21:35:43.929 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "pos": 0, 2023-05-31T21:35:43.929 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "timestamp": "2023-05-31T20:55:01.757183Z", 2023-05-31T21:35:43.929 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "realm_epoch": 2 2023-05-31T21:35:43.929 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2023-05-31T21:35:43.930 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2023-05-31T21:35:43.930 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2023-05-31T21:35:43.930 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "key": 1, 2023-05-31T21:35:43.930 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "val": { 2023-05-31T21:35:43.931 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "state": 1, 2023-05-31T21:35:43.931 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "marker": "1_1685566501.735717_251.1", 2023-05-31T21:35:43.931 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "next_step_marker": "", 2023-05-31T21:35:43.931 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "total_entries": 0, 2023-05-31T21:35:43.931 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "pos": 0, 2023-05-31T21:35:43.932 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "timestamp": "2023-05-31T20:55:01.735717Z", 2023-05-31T21:35:43.932 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "realm_epoch": 2 2023-05-31T21:35:43.932 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2023-05-31T21:35:43.932 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2023-05-31T21:35:43.933 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2023-05-31T21:35:43.933 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "key": 2, 2023-05-31T21:35:43.933 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "val": { 2023-05-31T21:35:43.933 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "state": 1, 2023-05-31T21:35:43.934 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "marker": "1_1685562970.274572_93.1", 2023-05-31T21:35:43.934 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "next_step_marker": "", 2023-05-31T21:35:43.934 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "total_entries": 0, 2023-05-31T21:35:43.934 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "pos": 0, 2023-05-31T21:35:43.934 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "timestamp": "2023-05-31T19:56:10.274572Z", 2023-05-31T21:35:43.935 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "realm_epoch": 2 2023-05-31T21:35:43.935 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2023-05-31T21:35:43.935 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2023-05-31T21:35:43.935 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2023-05-31T21:35:43.936 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "key": 3, 2023-05-31T21:35:43.936 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "val": { 2023-05-31T21:35:43.936 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "state": 1, 2023-05-31T21:35:43.936 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "marker": "1_1685568452.432716_1004.1", 2023-05-31T21:35:43.937 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "next_step_marker": "", 2023-05-31T21:35:43.937 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "total_entries": 0, 2023-05-31T21:35:43.937 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "pos": 0, 2023-05-31T21:35:43.937 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "timestamp": "2023-05-31T21:27:32.432716Z", 2023-05-31T21:35:43.937 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "realm_epoch": 2 2023-05-31T21:35:43.938 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2023-05-31T21:35:43.938 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2023-05-31T21:35:43.938 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: ] 2023-05-31T21:35:43.938 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2023-05-31T21:35:43.939 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "full_sync": { 2023-05-31T21:35:43.939 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "total": 1, 2023-05-31T21:35:43.939 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "complete": 1 2023-05-31T21:35:43.939 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2023-05-31T21:35:43.939 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "current_time": "2023-05-31T21:27:59Z" 2023-05-31T21:35:43.940 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2023-05-31T21:35:43.940 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2023-05-31T21:35:43.940 INFO:tasks.rgw_multisite_tests: "sync_status": { 2023-05-31T21:35:43.940 INFO:tasks.rgw_multisite_tests: "info": { 2023-05-31T21:35:43.941 INFO:tasks.rgw_multisite_tests: "status": "sync", 2023-05-31T21:35:43.941 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2023-05-31T21:35:43.941 INFO:tasks.rgw_multisite_tests: "period": "2e6123d2-d695-4350-9a01-61a455bc782b", 2023-05-31T21:35:43.941 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-05-31T21:35:43.942 INFO:tasks.rgw_multisite_tests: }, 2023-05-31T21:35:43.942 INFO:tasks.rgw_multisite_tests: "markers": [ 2023-05-31T21:35:43.942 INFO:tasks.rgw_multisite_tests: { 2023-05-31T21:35:43.942 INFO:tasks.rgw_multisite_tests: "key": 0, 2023-05-31T21:35:43.942 INFO:tasks.rgw_multisite_tests: "val": { 2023-05-31T21:35:43.943 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-05-31T21:35:43.943 INFO:tasks.rgw_multisite_tests: "marker": "1_1685566501.757183_260.1", 2023-05-31T21:35:43.943 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-05-31T21:35:43.943 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2023-05-31T21:35:43.944 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-05-31T21:35:43.944 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-05-31T20:55:01.757183Z", 2023-05-31T21:35:43.944 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-05-31T21:35:43.944 INFO:tasks.rgw_multisite_tests: } 2023-05-31T21:35:43.945 INFO:tasks.rgw_multisite_tests: }, 2023-05-31T21:35:43.945 INFO:tasks.rgw_multisite_tests: { 2023-05-31T21:35:43.945 INFO:tasks.rgw_multisite_tests: "key": 1, 2023-05-31T21:35:43.945 INFO:tasks.rgw_multisite_tests: "val": { 2023-05-31T21:35:43.945 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-05-31T21:35:43.946 INFO:tasks.rgw_multisite_tests: "marker": "1_1685566501.735717_251.1", 2023-05-31T21:35:43.946 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-05-31T21:35:43.946 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-05-31T21:35:43.946 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-05-31T21:35:43.947 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-05-31T20:55:01.735717Z", 2023-05-31T21:35:43.947 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-05-31T21:35:43.947 INFO:tasks.rgw_multisite_tests: } 2023-05-31T21:35:43.947 INFO:tasks.rgw_multisite_tests: }, 2023-05-31T21:35:43.948 INFO:tasks.rgw_multisite_tests: { 2023-05-31T21:35:43.948 INFO:tasks.rgw_multisite_tests: "key": 2, 2023-05-31T21:35:43.948 INFO:tasks.rgw_multisite_tests: "val": { 2023-05-31T21:35:43.948 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-05-31T21:35:43.948 INFO:tasks.rgw_multisite_tests: "marker": "1_1685562970.274572_93.1", 2023-05-31T21:35:43.949 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-05-31T21:35:43.949 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-05-31T21:35:43.949 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-05-31T21:35:43.949 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-05-31T19:56:10.274572Z", 2023-05-31T21:35:43.950 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-05-31T21:35:43.950 INFO:tasks.rgw_multisite_tests: } 2023-05-31T21:35:43.950 INFO:tasks.rgw_multisite_tests: }, 2023-05-31T21:35:43.950 INFO:tasks.rgw_multisite_tests: { 2023-05-31T21:35:43.951 INFO:tasks.rgw_multisite_tests: "key": 3, 2023-05-31T21:35:43.951 INFO:tasks.rgw_multisite_tests: "val": { 2023-05-31T21:35:43.951 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-05-31T21:35:43.951 INFO:tasks.rgw_multisite_tests: "marker": "1_1685568452.432716_1004.1", 2023-05-31T21:35:43.951 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-05-31T21:35:43.952 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-05-31T21:35:43.952 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-05-31T21:35:43.952 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-05-31T21:27:32.432716Z", 2023-05-31T21:35:43.952 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-05-31T21:35:43.953 INFO:tasks.rgw_multisite_tests: } 2023-05-31T21:35:43.953 INFO:tasks.rgw_multisite_tests: } 2023-05-31T21:35:43.953 INFO:tasks.rgw_multisite_tests: ] 2023-05-31T21:35:43.953 INFO:tasks.rgw_multisite_tests: }, 2023-05-31T21:35:43.954 INFO:tasks.rgw_multisite_tests: "full_sync": { 2023-05-31T21:35:43.954 INFO:tasks.rgw_multisite_tests: "total": 1, 2023-05-31T21:35:43.954 INFO:tasks.rgw_multisite_tests: "complete": 1 2023-05-31T21:35:43.954 INFO:tasks.rgw_multisite_tests: }, 2023-05-31T21:35:43.955 INFO:tasks.rgw_multisite_tests: "current_time": "2023-05-31T21:27:59Z" 2023-05-31T21:35:43.955 INFO:tasks.rgw_multisite_tests:} 2023-05-31T21:35:43.955 INFO:tasks.rgw_multisite_tests: 2023-05-31T21:35:43.955 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1685566501.757183_260.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-05-31T20:55:01.757183Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1685566501.735717_251.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-05-31T20:55:01.735717Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1685562970.274572_93.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-05-31T19:56:10.274572Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1685568452.432716_1004.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-05-31T21:27:32.432716Z', 'realm_epoch': 2}}] 2023-05-31T21:35:43.955 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1685566501.757183_260.1', 1: '1_1685566501.735717_251.1', 2: '1_1685562970.274572_93.1', 3: '1_1685568473.966785_1006.1'} 2023-05-31T21:35:43.956 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1685566501.757183_260.1', 1: '1_1685566501.735717_251.1', 2: '1_1685562970.274572_93.1', 3: '1_1685568452.432716_1004.1'} 2023-05-31T21:35:43.956 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone a2 behind master: shard=3 master=1_1685568473.966785_1006.1 target=1_1685568452.432716_1004.1 2023-05-31T21:35:43.956 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-31T21:35:43.956 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-31T21:35:43.957 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-05-31T21:35:43.957 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: ignoring --setuser ceph since I am not root 2023-05-31T21:35:43.957 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-05-31T21:35:43.957 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: 2023-05-31T21:28:04.615+0000 7f81db60d780 20 RGW-SYNC:meta: read sync status 2023-05-31T21:35:43.958 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: 2023-05-31T21:28:04.616+0000 7f81db60d780 20 RGW-SYNC:meta: read sync status 2023-05-31T21:35:43.958 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2023-05-31T21:35:43.958 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "sync_status": { 2023-05-31T21:35:43.958 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "info": { 2023-05-31T21:35:43.959 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "status": "sync", 2023-05-31T21:35:43.959 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "num_shards": 4, 2023-05-31T21:35:43.959 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "period": "2e6123d2-d695-4350-9a01-61a455bc782b", 2023-05-31T21:35:43.959 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "realm_epoch": 2 2023-05-31T21:35:43.959 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2023-05-31T21:35:43.960 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "markers": [ 2023-05-31T21:35:43.960 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2023-05-31T21:35:43.960 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "key": 0, 2023-05-31T21:35:43.960 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "val": { 2023-05-31T21:35:43.961 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "state": 1, 2023-05-31T21:35:43.961 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "marker": "1_1685566501.757183_260.1", 2023-05-31T21:35:43.961 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "next_step_marker": "", 2023-05-31T21:35:43.961 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "total_entries": 1, 2023-05-31T21:35:43.962 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "pos": 0, 2023-05-31T21:35:43.962 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "timestamp": "2023-05-31T20:55:01.757183Z", 2023-05-31T21:35:43.962 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "realm_epoch": 2 2023-05-31T21:35:43.962 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2023-05-31T21:35:43.962 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2023-05-31T21:35:43.963 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2023-05-31T21:35:43.963 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "key": 1, 2023-05-31T21:35:43.963 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "val": { 2023-05-31T21:35:43.963 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "state": 1, 2023-05-31T21:35:43.964 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "marker": "1_1685566501.735717_251.1", 2023-05-31T21:35:43.964 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "next_step_marker": "", 2023-05-31T21:35:43.964 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "total_entries": 0, 2023-05-31T21:35:43.964 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "pos": 0, 2023-05-31T21:35:43.965 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "timestamp": "2023-05-31T20:55:01.735717Z", 2023-05-31T21:35:43.965 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "realm_epoch": 2 2023-05-31T21:35:43.965 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2023-05-31T21:35:43.965 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2023-05-31T21:35:43.965 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2023-05-31T21:35:43.966 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "key": 2, 2023-05-31T21:35:43.966 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "val": { 2023-05-31T21:35:43.966 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "state": 1, 2023-05-31T21:35:43.966 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "marker": "1_1685562970.274572_93.1", 2023-05-31T21:35:43.967 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "next_step_marker": "", 2023-05-31T21:35:43.967 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "total_entries": 0, 2023-05-31T21:35:43.967 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "pos": 0, 2023-05-31T21:35:43.967 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "timestamp": "2023-05-31T19:56:10.274572Z", 2023-05-31T21:35:43.967 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "realm_epoch": 2 2023-05-31T21:35:43.968 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2023-05-31T21:35:43.968 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2023-05-31T21:35:43.968 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2023-05-31T21:35:43.968 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "key": 3, 2023-05-31T21:35:43.969 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "val": { 2023-05-31T21:35:43.969 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "state": 1, 2023-05-31T21:35:43.969 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "marker": "1_1685568452.432716_1004.1", 2023-05-31T21:35:43.969 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "next_step_marker": "", 2023-05-31T21:35:43.970 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "total_entries": 0, 2023-05-31T21:35:43.970 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "pos": 0, 2023-05-31T21:35:43.970 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "timestamp": "2023-05-31T21:27:32.432716Z", 2023-05-31T21:35:43.970 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "realm_epoch": 2 2023-05-31T21:35:43.970 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2023-05-31T21:35:43.971 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2023-05-31T21:35:43.971 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: ] 2023-05-31T21:35:43.971 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2023-05-31T21:35:43.971 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "full_sync": { 2023-05-31T21:35:43.972 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "total": 1, 2023-05-31T21:35:43.972 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "complete": 1 2023-05-31T21:35:43.972 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2023-05-31T21:35:43.972 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "current_time": "2023-05-31T21:28:04Z" 2023-05-31T21:35:43.973 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2023-05-31T21:35:43.973 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2023-05-31T21:35:43.973 INFO:tasks.rgw_multisite_tests: "sync_status": { 2023-05-31T21:35:43.973 INFO:tasks.rgw_multisite_tests: "info": { 2023-05-31T21:35:43.973 INFO:tasks.rgw_multisite_tests: "status": "sync", 2023-05-31T21:35:43.974 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2023-05-31T21:35:43.974 INFO:tasks.rgw_multisite_tests: "period": "2e6123d2-d695-4350-9a01-61a455bc782b", 2023-05-31T21:35:43.974 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-05-31T21:35:43.974 INFO:tasks.rgw_multisite_tests: }, 2023-05-31T21:35:43.975 INFO:tasks.rgw_multisite_tests: "markers": [ 2023-05-31T21:35:43.975 INFO:tasks.rgw_multisite_tests: { 2023-05-31T21:35:43.975 INFO:tasks.rgw_multisite_tests: "key": 0, 2023-05-31T21:35:43.975 INFO:tasks.rgw_multisite_tests: "val": { 2023-05-31T21:35:43.976 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-05-31T21:35:43.976 INFO:tasks.rgw_multisite_tests: "marker": "1_1685566501.757183_260.1", 2023-05-31T21:35:43.976 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-05-31T21:35:43.976 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2023-05-31T21:35:43.976 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-05-31T21:35:43.977 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-05-31T20:55:01.757183Z", 2023-05-31T21:35:43.977 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-05-31T21:35:43.977 INFO:tasks.rgw_multisite_tests: } 2023-05-31T21:35:43.977 INFO:tasks.rgw_multisite_tests: }, 2023-05-31T21:35:43.978 INFO:tasks.rgw_multisite_tests: { 2023-05-31T21:35:43.978 INFO:tasks.rgw_multisite_tests: "key": 1, 2023-05-31T21:35:43.978 INFO:tasks.rgw_multisite_tests: "val": { 2023-05-31T21:35:43.978 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-05-31T21:35:43.979 INFO:tasks.rgw_multisite_tests: "marker": "1_1685566501.735717_251.1", 2023-05-31T21:35:43.979 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-05-31T21:35:43.979 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-05-31T21:35:43.979 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-05-31T21:35:43.979 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-05-31T20:55:01.735717Z", 2023-05-31T21:35:43.980 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-05-31T21:35:43.980 INFO:tasks.rgw_multisite_tests: } 2023-05-31T21:35:43.980 INFO:tasks.rgw_multisite_tests: }, 2023-05-31T21:35:43.980 INFO:tasks.rgw_multisite_tests: { 2023-05-31T21:35:43.981 INFO:tasks.rgw_multisite_tests: "key": 2, 2023-05-31T21:35:43.981 INFO:tasks.rgw_multisite_tests: "val": { 2023-05-31T21:35:43.981 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-05-31T21:35:43.981 INFO:tasks.rgw_multisite_tests: "marker": "1_1685562970.274572_93.1", 2023-05-31T21:35:43.982 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-05-31T21:35:43.982 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-05-31T21:35:43.982 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-05-31T21:35:43.982 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-05-31T19:56:10.274572Z", 2023-05-31T21:35:43.983 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-05-31T21:35:43.983 INFO:tasks.rgw_multisite_tests: } 2023-05-31T21:35:43.983 INFO:tasks.rgw_multisite_tests: }, 2023-05-31T21:35:43.983 INFO:tasks.rgw_multisite_tests: { 2023-05-31T21:35:43.983 INFO:tasks.rgw_multisite_tests: "key": 3, 2023-05-31T21:35:43.984 INFO:tasks.rgw_multisite_tests: "val": { 2023-05-31T21:35:43.984 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-05-31T21:35:43.984 INFO:tasks.rgw_multisite_tests: "marker": "1_1685568452.432716_1004.1", 2023-05-31T21:35:43.984 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-05-31T21:35:43.985 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-05-31T21:35:43.985 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-05-31T21:35:43.985 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-05-31T21:27:32.432716Z", 2023-05-31T21:35:43.985 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-05-31T21:35:43.986 INFO:tasks.rgw_multisite_tests: } 2023-05-31T21:35:43.986 INFO:tasks.rgw_multisite_tests: } 2023-05-31T21:35:43.986 INFO:tasks.rgw_multisite_tests: ] 2023-05-31T21:35:43.986 INFO:tasks.rgw_multisite_tests: }, 2023-05-31T21:35:43.987 INFO:tasks.rgw_multisite_tests: "full_sync": { 2023-05-31T21:35:43.987 INFO:tasks.rgw_multisite_tests: "total": 1, 2023-05-31T21:35:43.987 INFO:tasks.rgw_multisite_tests: "complete": 1 2023-05-31T21:35:43.987 INFO:tasks.rgw_multisite_tests: }, 2023-05-31T21:35:43.987 INFO:tasks.rgw_multisite_tests: "current_time": "2023-05-31T21:28:04Z" 2023-05-31T21:35:43.988 INFO:tasks.rgw_multisite_tests:} 2023-05-31T21:35:43.988 INFO:tasks.rgw_multisite_tests: 2023-05-31T21:35:43.988 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1685566501.757183_260.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-05-31T20:55:01.757183Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1685566501.735717_251.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-05-31T20:55:01.735717Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1685562970.274572_93.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-05-31T19:56:10.274572Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1685568452.432716_1004.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-05-31T21:27:32.432716Z', 'realm_epoch': 2}}] 2023-05-31T21:35:43.988 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1685566501.757183_260.1', 1: '1_1685566501.735717_251.1', 2: '1_1685562970.274572_93.1', 3: '1_1685568473.966785_1006.1'} 2023-05-31T21:35:43.989 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1685566501.757183_260.1', 1: '1_1685566501.735717_251.1', 2: '1_1685562970.274572_93.1', 3: '1_1685568452.432716_1004.1'} 2023-05-31T21:35:43.989 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone a2 behind master: shard=3 master=1_1685568473.966785_1006.1 target=1_1685568452.432716_1004.1 2023-05-31T21:35:43.989 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-31T21:35:43.989 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-31T21:35:43.990 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-05-31T21:35:43.990 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: ignoring --setuser ceph since I am not root 2023-05-31T21:35:43.990 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-05-31T21:35:43.990 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: 2023-05-31T21:28:09.755+0000 7fdf7277f780 20 RGW-SYNC:meta: read sync status 2023-05-31T21:35:43.991 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: 2023-05-31T21:28:09.756+0000 7fdf7277f780 20 RGW-SYNC:meta: read sync status 2023-05-31T21:35:43.991 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2023-05-31T21:35:43.991 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "sync_status": { 2023-05-31T21:35:43.991 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "info": { 2023-05-31T21:35:43.992 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "status": "sync", 2023-05-31T21:35:43.992 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "num_shards": 4, 2023-05-31T21:35:43.992 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "period": "2e6123d2-d695-4350-9a01-61a455bc782b", 2023-05-31T21:35:43.992 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "realm_epoch": 2 2023-05-31T21:35:43.993 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2023-05-31T21:35:43.993 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "markers": [ 2023-05-31T21:35:43.993 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2023-05-31T21:35:43.993 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "key": 0, 2023-05-31T21:35:43.993 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "val": { 2023-05-31T21:35:43.994 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "state": 1, 2023-05-31T21:35:43.994 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "marker": "1_1685566501.757183_260.1", 2023-05-31T21:35:43.994 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "next_step_marker": "", 2023-05-31T21:35:43.994 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "total_entries": 1, 2023-05-31T21:35:43.995 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "pos": 0, 2023-05-31T21:35:43.995 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "timestamp": "2023-05-31T20:55:01.757183Z", 2023-05-31T21:35:43.995 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "realm_epoch": 2 2023-05-31T21:35:43.995 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2023-05-31T21:35:43.995 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2023-05-31T21:35:43.996 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2023-05-31T21:35:43.996 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "key": 1, 2023-05-31T21:35:43.996 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "val": { 2023-05-31T21:35:43.996 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "state": 1, 2023-05-31T21:35:43.997 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "marker": "1_1685566501.735717_251.1", 2023-05-31T21:35:43.997 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "next_step_marker": "", 2023-05-31T21:35:43.997 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "total_entries": 0, 2023-05-31T21:35:43.997 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "pos": 0, 2023-05-31T21:35:43.998 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "timestamp": "2023-05-31T20:55:01.735717Z", 2023-05-31T21:35:43.998 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "realm_epoch": 2 2023-05-31T21:35:43.998 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2023-05-31T21:35:43.998 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2023-05-31T21:35:43.998 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2023-05-31T21:35:43.999 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "key": 2, 2023-05-31T21:35:43.999 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "val": { 2023-05-31T21:35:43.999 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "state": 1, 2023-05-31T21:35:43.999 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "marker": "1_1685562970.274572_93.1", 2023-05-31T21:35:44.000 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "next_step_marker": "", 2023-05-31T21:35:44.000 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "total_entries": 0, 2023-05-31T21:35:44.000 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "pos": 0, 2023-05-31T21:35:44.000 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "timestamp": "2023-05-31T19:56:10.274572Z", 2023-05-31T21:35:44.001 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "realm_epoch": 2 2023-05-31T21:35:44.001 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2023-05-31T21:35:44.001 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2023-05-31T21:35:44.001 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2023-05-31T21:35:44.002 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "key": 3, 2023-05-31T21:35:44.002 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "val": { 2023-05-31T21:35:44.002 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "state": 1, 2023-05-31T21:35:44.002 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "marker": "1_1685568473.966785_1006.1", 2023-05-31T21:35:44.003 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "next_step_marker": "", 2023-05-31T21:35:44.003 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "total_entries": 0, 2023-05-31T21:35:44.003 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "pos": 0, 2023-05-31T21:35:44.003 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "timestamp": "2023-05-31T21:27:53.966785Z", 2023-05-31T21:35:44.003 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "realm_epoch": 2 2023-05-31T21:35:44.004 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2023-05-31T21:35:44.004 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2023-05-31T21:35:44.004 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: ] 2023-05-31T21:35:44.004 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2023-05-31T21:35:44.005 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "full_sync": { 2023-05-31T21:35:44.005 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "total": 1, 2023-05-31T21:35:44.005 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "complete": 1 2023-05-31T21:35:44.005 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2023-05-31T21:35:44.006 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "current_time": "2023-05-31T21:28:09Z" 2023-05-31T21:35:44.006 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2023-05-31T21:35:44.006 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2023-05-31T21:35:44.006 INFO:tasks.rgw_multisite_tests: "sync_status": { 2023-05-31T21:35:44.006 INFO:tasks.rgw_multisite_tests: "info": { 2023-05-31T21:35:44.007 INFO:tasks.rgw_multisite_tests: "status": "sync", 2023-05-31T21:35:44.007 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2023-05-31T21:35:44.007 INFO:tasks.rgw_multisite_tests: "period": "2e6123d2-d695-4350-9a01-61a455bc782b", 2023-05-31T21:35:44.007 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-05-31T21:35:44.008 INFO:tasks.rgw_multisite_tests: }, 2023-05-31T21:35:44.008 INFO:tasks.rgw_multisite_tests: "markers": [ 2023-05-31T21:35:44.008 INFO:tasks.rgw_multisite_tests: { 2023-05-31T21:35:44.008 INFO:tasks.rgw_multisite_tests: "key": 0, 2023-05-31T21:35:44.009 INFO:tasks.rgw_multisite_tests: "val": { 2023-05-31T21:35:44.009 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-05-31T21:35:44.009 INFO:tasks.rgw_multisite_tests: "marker": "1_1685566501.757183_260.1", 2023-05-31T21:35:44.009 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-05-31T21:35:44.010 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2023-05-31T21:35:44.010 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-05-31T21:35:44.010 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-05-31T20:55:01.757183Z", 2023-05-31T21:35:44.011 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-05-31T21:35:44.011 INFO:tasks.rgw_multisite_tests: } 2023-05-31T21:35:44.011 INFO:tasks.rgw_multisite_tests: }, 2023-05-31T21:35:44.011 INFO:tasks.rgw_multisite_tests: { 2023-05-31T21:35:44.012 INFO:tasks.rgw_multisite_tests: "key": 1, 2023-05-31T21:35:44.012 INFO:tasks.rgw_multisite_tests: "val": { 2023-05-31T21:35:44.012 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-05-31T21:35:44.013 INFO:tasks.rgw_multisite_tests: "marker": "1_1685566501.735717_251.1", 2023-05-31T21:35:44.013 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-05-31T21:35:44.013 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-05-31T21:35:44.013 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-05-31T21:35:44.014 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-05-31T20:55:01.735717Z", 2023-05-31T21:35:44.014 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-05-31T21:35:44.014 INFO:tasks.rgw_multisite_tests: } 2023-05-31T21:35:44.014 INFO:tasks.rgw_multisite_tests: }, 2023-05-31T21:35:44.015 INFO:tasks.rgw_multisite_tests: { 2023-05-31T21:35:44.015 INFO:tasks.rgw_multisite_tests: "key": 2, 2023-05-31T21:35:44.015 INFO:tasks.rgw_multisite_tests: "val": { 2023-05-31T21:35:44.015 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-05-31T21:35:44.015 INFO:tasks.rgw_multisite_tests: "marker": "1_1685562970.274572_93.1", 2023-05-31T21:35:44.016 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-05-31T21:35:44.016 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-05-31T21:35:44.016 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-05-31T21:35:44.016 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-05-31T19:56:10.274572Z", 2023-05-31T21:35:44.017 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-05-31T21:35:44.017 INFO:tasks.rgw_multisite_tests: } 2023-05-31T21:35:44.017 INFO:tasks.rgw_multisite_tests: }, 2023-05-31T21:35:44.017 INFO:tasks.rgw_multisite_tests: { 2023-05-31T21:35:44.018 INFO:tasks.rgw_multisite_tests: "key": 3, 2023-05-31T21:35:44.018 INFO:tasks.rgw_multisite_tests: "val": { 2023-05-31T21:35:44.018 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-05-31T21:35:44.018 INFO:tasks.rgw_multisite_tests: "marker": "1_1685568473.966785_1006.1", 2023-05-31T21:35:44.018 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-05-31T21:35:44.019 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-05-31T21:35:44.019 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-05-31T21:35:44.019 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-05-31T21:27:53.966785Z", 2023-05-31T21:35:44.019 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-05-31T21:35:44.020 INFO:tasks.rgw_multisite_tests: } 2023-05-31T21:35:44.020 INFO:tasks.rgw_multisite_tests: } 2023-05-31T21:35:44.020 INFO:tasks.rgw_multisite_tests: ] 2023-05-31T21:35:44.020 INFO:tasks.rgw_multisite_tests: }, 2023-05-31T21:35:44.021 INFO:tasks.rgw_multisite_tests: "full_sync": { 2023-05-31T21:35:44.021 INFO:tasks.rgw_multisite_tests: "total": 1, 2023-05-31T21:35:44.021 INFO:tasks.rgw_multisite_tests: "complete": 1 2023-05-31T21:35:44.021 INFO:tasks.rgw_multisite_tests: }, 2023-05-31T21:35:44.021 INFO:tasks.rgw_multisite_tests: "current_time": "2023-05-31T21:28:09Z" 2023-05-31T21:35:44.022 INFO:tasks.rgw_multisite_tests:} 2023-05-31T21:35:44.022 INFO:tasks.rgw_multisite_tests: 2023-05-31T21:35:44.022 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1685566501.757183_260.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-05-31T20:55:01.757183Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1685566501.735717_251.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-05-31T20:55:01.735717Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1685562970.274572_93.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-05-31T19:56:10.274572Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1685568473.966785_1006.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-05-31T21:27:53.966785Z', 'realm_epoch': 2}}] 2023-05-31T21:35:44.022 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1685566501.757183_260.1', 1: '1_1685566501.735717_251.1', 2: '1_1685562970.274572_93.1', 3: '1_1685568473.966785_1006.1'} 2023-05-31T21:35:44.023 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1685566501.757183_260.1', 1: '1_1685566501.735717_251.1', 2: '1_1685562970.274572_93.1', 3: '1_1685568473.966785_1006.1'} 2023-05-31T21:35:44.023 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: INFO: finish meta checkpoint for zone=a2 2023-05-31T21:35:44.023 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['sync', 'policy', 'get', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-05-31T21:35:44.023 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'policy', 'get', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-05-31T21:35:44.024 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync policy get --cluster c1 --debug-rgw 0 --debug-ms 0 2023-05-31T21:35:44.024 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: ignoring --setuser ceph since I am not root 2023-05-31T21:35:44.024 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-05-31T21:35:44.024 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: {"groups":[]} 2023-05-31T21:35:44.025 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: INFO: create bucket zone=a1 name=pklxkd-72 2023-05-31T21:35:44.025 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/pklxkd-72/ 2023-05-31T21:35:44.025 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/pklxkd-72/ 2023-05-31T21:35:44.025 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2023-05-31T21:35:44.026 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /pklxkd-72/ 2023-05-31T21:35:44.026 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-05-31T21:35:44.026 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-05-31T21:35:44.026 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T21:35:44.026 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-05-31T21:35:44.027 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-05-31T21:35:44.027 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-05-31T21:35:44.027 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-05-31T21:35:44.027 INFO:tasks.rgw_multisite_tests:PUT 2023-05-31T21:35:44.028 INFO:tasks.rgw_multisite_tests: 2023-05-31T21:35:44.028 INFO:tasks.rgw_multisite_tests: 2023-05-31T21:35:44.028 INFO:tasks.rgw_multisite_tests:Wed, 31 May 2023 21:28:09 GMT 2023-05-31T21:35:44.028 INFO:tasks.rgw_multisite_tests:/pklxkd-72/ 2023-05-31T21:35:44.029 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-05-31T21:35:44.029 INFO:tasks.rgw_multisite_tests:AWS Q80H11OLNU1S48PPYL26:Cy1XqspsHBkkNuz52tx5tVjAXPo= 2023-05-31T21:35:44.029 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 21:28:09 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:Cy1XqspsHBkkNuz52tx5tVjAXPo=', 'Content-Length': '0'} 2023-05-31T21:35:44.029 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('x-amz-request-id', 'tx000009f02d8f9ebdd14e9-006477bbe9-264e-a1'), ('Content-Length', '0'), ('Date', 'Wed, 31 May 2023 21:28:09 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:35:44.030 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: INFO: create bucket zone=a1 name=pklxkd-73 2023-05-31T21:35:44.030 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/pklxkd-73/ 2023-05-31T21:35:44.030 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/pklxkd-73/ 2023-05-31T21:35:44.030 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2023-05-31T21:35:44.030 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /pklxkd-73/ 2023-05-31T21:35:44.031 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-05-31T21:35:44.031 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-05-31T21:35:44.031 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T21:35:44.031 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-05-31T21:35:44.032 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-05-31T21:35:44.032 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-05-31T21:35:44.032 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-05-31T21:35:44.032 INFO:tasks.rgw_multisite_tests:PUT 2023-05-31T21:35:44.033 INFO:tasks.rgw_multisite_tests: 2023-05-31T21:35:44.033 INFO:tasks.rgw_multisite_tests: 2023-05-31T21:35:44.033 INFO:tasks.rgw_multisite_tests:Wed, 31 May 2023 21:28:09 GMT 2023-05-31T21:35:44.033 INFO:tasks.rgw_multisite_tests:/pklxkd-73/ 2023-05-31T21:35:44.034 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-05-31T21:35:44.034 INFO:tasks.rgw_multisite_tests:AWS Q80H11OLNU1S48PPYL26:qC8hvvNRTXfWcvcI0mIBxQl74n0= 2023-05-31T21:35:44.034 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 21:28:09 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:qC8hvvNRTXfWcvcI0mIBxQl74n0=', 'Content-Length': '0'} 2023-05-31T21:35:44.034 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('x-amz-request-id', 'tx00000ff187ef5c6106ce8-006477bbe9-264e-a1'), ('Content-Length', '0'), ('Date', 'Wed, 31 May 2023 21:28:09 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:35:44.034 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-31T21:35:44.035 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-31T21:35:44.035 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-05-31T21:35:44.035 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: ignoring --setuser ceph since I am not root 2023-05-31T21:35:44.035 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-05-31T21:35:44.036 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: No --period given, using current period=2e6123d2-d695-4350-9a01-61a455bc782b 2023-05-31T21:35:44.036 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: [ 2023-05-31T21:35:44.036 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2023-05-31T21:35:44.036 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "marker": "1_1685566501.757183_260.1", 2023-05-31T21:35:44.037 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "last_update": "2023-05-31T20:55:01.757183Z" 2023-05-31T21:35:44.037 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2023-05-31T21:35:44.037 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2023-05-31T21:35:44.037 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "marker": "1_1685566501.735717_251.1", 2023-05-31T21:35:44.037 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "last_update": "2023-05-31T20:55:01.735717Z" 2023-05-31T21:35:44.038 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2023-05-31T21:35:44.038 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2023-05-31T21:35:44.038 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "marker": "1_1685562970.274572_93.1", 2023-05-31T21:35:44.038 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "last_update": "2023-05-31T19:56:10.274572Z" 2023-05-31T21:35:44.039 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2023-05-31T21:35:44.039 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2023-05-31T21:35:44.039 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "marker": "1_1685568489.974220_1014.1", 2023-05-31T21:35:44.039 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "last_update": "2023-05-31T21:28:09.974220Z" 2023-05-31T21:35:44.040 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2023-05-31T21:35:44.040 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: ] 2023-05-31T21:35:44.040 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: master meta markers={0: '1_1685566501.757183_260.1', 1: '1_1685566501.735717_251.1', 2: '1_1685562970.274572_93.1', 3: '1_1685568489.974220_1014.1'} 2023-05-31T21:35:44.040 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: INFO: starting meta checkpoint for zone=a2 2023-05-31T21:35:44.040 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-31T21:35:44.041 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-31T21:35:44.041 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-05-31T21:35:44.041 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: ignoring --setuser ceph since I am not root 2023-05-31T21:35:44.041 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-05-31T21:35:44.042 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: 2023-05-31T21:28:10.193+0000 7fed542cb780 20 RGW-SYNC:meta: read sync status 2023-05-31T21:35:44.042 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: 2023-05-31T21:28:10.194+0000 7fed542cb780 20 RGW-SYNC:meta: read sync status 2023-05-31T21:35:44.042 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2023-05-31T21:35:44.042 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "sync_status": { 2023-05-31T21:35:44.043 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "info": { 2023-05-31T21:35:44.043 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "status": "sync", 2023-05-31T21:35:44.043 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "num_shards": 4, 2023-05-31T21:35:44.043 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "period": "2e6123d2-d695-4350-9a01-61a455bc782b", 2023-05-31T21:35:44.043 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "realm_epoch": 2 2023-05-31T21:35:44.044 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2023-05-31T21:35:44.044 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "markers": [ 2023-05-31T21:35:44.044 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2023-05-31T21:35:44.044 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "key": 0, 2023-05-31T21:35:44.045 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "val": { 2023-05-31T21:35:44.045 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "state": 1, 2023-05-31T21:35:44.045 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "marker": "1_1685566501.757183_260.1", 2023-05-31T21:35:44.045 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "next_step_marker": "", 2023-05-31T21:35:44.046 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "total_entries": 1, 2023-05-31T21:35:44.046 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "pos": 0, 2023-05-31T21:35:44.046 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "timestamp": "2023-05-31T20:55:01.757183Z", 2023-05-31T21:35:44.046 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "realm_epoch": 2 2023-05-31T21:35:44.047 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2023-05-31T21:35:44.047 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2023-05-31T21:35:44.047 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2023-05-31T21:35:44.047 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "key": 1, 2023-05-31T21:35:44.047 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "val": { 2023-05-31T21:35:44.048 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "state": 1, 2023-05-31T21:35:44.048 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "marker": "1_1685566501.735717_251.1", 2023-05-31T21:35:44.048 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "next_step_marker": "", 2023-05-31T21:35:44.048 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "total_entries": 0, 2023-05-31T21:35:44.049 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "pos": 0, 2023-05-31T21:35:44.049 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "timestamp": "2023-05-31T20:55:01.735717Z", 2023-05-31T21:35:44.049 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "realm_epoch": 2 2023-05-31T21:35:44.049 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2023-05-31T21:35:44.050 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2023-05-31T21:35:44.050 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2023-05-31T21:35:44.050 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "key": 2, 2023-05-31T21:35:44.050 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "val": { 2023-05-31T21:35:44.050 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "state": 1, 2023-05-31T21:35:44.051 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "marker": "1_1685562970.274572_93.1", 2023-05-31T21:35:44.051 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "next_step_marker": "", 2023-05-31T21:35:44.051 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "total_entries": 0, 2023-05-31T21:35:44.051 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "pos": 0, 2023-05-31T21:35:44.052 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "timestamp": "2023-05-31T19:56:10.274572Z", 2023-05-31T21:35:44.052 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "realm_epoch": 2 2023-05-31T21:35:44.052 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2023-05-31T21:35:44.052 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2023-05-31T21:35:44.053 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2023-05-31T21:35:44.053 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "key": 3, 2023-05-31T21:35:44.053 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "val": { 2023-05-31T21:35:44.053 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "state": 1, 2023-05-31T21:35:44.054 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "marker": "1_1685568489.956971_1011.1", 2023-05-31T21:35:44.054 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "next_step_marker": "", 2023-05-31T21:35:44.054 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "total_entries": 0, 2023-05-31T21:35:44.054 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "pos": 0, 2023-05-31T21:35:44.054 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "timestamp": "2023-05-31T21:28:09.956971Z", 2023-05-31T21:35:44.055 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "realm_epoch": 2 2023-05-31T21:35:44.055 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2023-05-31T21:35:44.055 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2023-05-31T21:35:44.055 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: ] 2023-05-31T21:35:44.056 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2023-05-31T21:35:44.056 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "full_sync": { 2023-05-31T21:35:44.056 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "total": 1, 2023-05-31T21:35:44.056 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "complete": 1 2023-05-31T21:35:44.056 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2023-05-31T21:35:44.057 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "current_time": "2023-05-31T21:28:10Z" 2023-05-31T21:35:44.057 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2023-05-31T21:35:44.057 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2023-05-31T21:35:44.057 INFO:tasks.rgw_multisite_tests: "sync_status": { 2023-05-31T21:35:44.058 INFO:tasks.rgw_multisite_tests: "info": { 2023-05-31T21:35:44.058 INFO:tasks.rgw_multisite_tests: "status": "sync", 2023-05-31T21:35:44.058 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2023-05-31T21:35:44.058 INFO:tasks.rgw_multisite_tests: "period": "2e6123d2-d695-4350-9a01-61a455bc782b", 2023-05-31T21:35:44.059 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-05-31T21:35:44.059 INFO:tasks.rgw_multisite_tests: }, 2023-05-31T21:35:44.059 INFO:tasks.rgw_multisite_tests: "markers": [ 2023-05-31T21:35:44.059 INFO:tasks.rgw_multisite_tests: { 2023-05-31T21:35:44.059 INFO:tasks.rgw_multisite_tests: "key": 0, 2023-05-31T21:35:44.060 INFO:tasks.rgw_multisite_tests: "val": { 2023-05-31T21:35:44.060 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-05-31T21:35:44.060 INFO:tasks.rgw_multisite_tests: "marker": "1_1685566501.757183_260.1", 2023-05-31T21:35:44.060 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-05-31T21:35:44.061 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2023-05-31T21:35:44.061 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-05-31T21:35:44.061 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-05-31T20:55:01.757183Z", 2023-05-31T21:35:44.061 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-05-31T21:35:44.062 INFO:tasks.rgw_multisite_tests: } 2023-05-31T21:35:44.062 INFO:tasks.rgw_multisite_tests: }, 2023-05-31T21:35:44.062 INFO:tasks.rgw_multisite_tests: { 2023-05-31T21:35:44.062 INFO:tasks.rgw_multisite_tests: "key": 1, 2023-05-31T21:35:44.062 INFO:tasks.rgw_multisite_tests: "val": { 2023-05-31T21:35:44.063 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-05-31T21:35:44.063 INFO:tasks.rgw_multisite_tests: "marker": "1_1685566501.735717_251.1", 2023-05-31T21:35:44.063 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-05-31T21:35:44.063 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-05-31T21:35:44.064 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-05-31T21:35:44.064 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-05-31T20:55:01.735717Z", 2023-05-31T21:35:44.064 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-05-31T21:35:44.064 INFO:tasks.rgw_multisite_tests: } 2023-05-31T21:35:44.064 INFO:tasks.rgw_multisite_tests: }, 2023-05-31T21:35:44.065 INFO:tasks.rgw_multisite_tests: { 2023-05-31T21:35:44.065 INFO:tasks.rgw_multisite_tests: "key": 2, 2023-05-31T21:35:44.065 INFO:tasks.rgw_multisite_tests: "val": { 2023-05-31T21:35:44.065 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-05-31T21:35:44.066 INFO:tasks.rgw_multisite_tests: "marker": "1_1685562970.274572_93.1", 2023-05-31T21:35:44.066 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-05-31T21:35:44.066 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-05-31T21:35:44.066 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-05-31T21:35:44.067 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-05-31T19:56:10.274572Z", 2023-05-31T21:35:44.067 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-05-31T21:35:44.067 INFO:tasks.rgw_multisite_tests: } 2023-05-31T21:35:44.067 INFO:tasks.rgw_multisite_tests: }, 2023-05-31T21:35:44.067 INFO:tasks.rgw_multisite_tests: { 2023-05-31T21:35:44.068 INFO:tasks.rgw_multisite_tests: "key": 3, 2023-05-31T21:35:44.068 INFO:tasks.rgw_multisite_tests: "val": { 2023-05-31T21:35:44.068 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-05-31T21:35:44.068 INFO:tasks.rgw_multisite_tests: "marker": "1_1685568489.956971_1011.1", 2023-05-31T21:35:44.069 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-05-31T21:35:44.069 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-05-31T21:35:44.069 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-05-31T21:35:44.069 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-05-31T21:28:09.956971Z", 2023-05-31T21:35:44.070 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-05-31T21:35:44.070 INFO:tasks.rgw_multisite_tests: } 2023-05-31T21:35:44.070 INFO:tasks.rgw_multisite_tests: } 2023-05-31T21:35:44.070 INFO:tasks.rgw_multisite_tests: ] 2023-05-31T21:35:44.070 INFO:tasks.rgw_multisite_tests: }, 2023-05-31T21:35:44.071 INFO:tasks.rgw_multisite_tests: "full_sync": { 2023-05-31T21:35:44.071 INFO:tasks.rgw_multisite_tests: "total": 1, 2023-05-31T21:35:44.071 INFO:tasks.rgw_multisite_tests: "complete": 1 2023-05-31T21:35:44.071 INFO:tasks.rgw_multisite_tests: }, 2023-05-31T21:35:44.072 INFO:tasks.rgw_multisite_tests: "current_time": "2023-05-31T21:28:10Z" 2023-05-31T21:35:44.072 INFO:tasks.rgw_multisite_tests:} 2023-05-31T21:35:44.072 INFO:tasks.rgw_multisite_tests: 2023-05-31T21:35:44.072 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1685566501.757183_260.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-05-31T20:55:01.757183Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1685566501.735717_251.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-05-31T20:55:01.735717Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1685562970.274572_93.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-05-31T19:56:10.274572Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1685568489.956971_1011.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-05-31T21:28:09.956971Z', 'realm_epoch': 2}}] 2023-05-31T21:35:44.073 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1685566501.757183_260.1', 1: '1_1685566501.735717_251.1', 2: '1_1685562970.274572_93.1', 3: '1_1685568489.974220_1014.1'} 2023-05-31T21:35:44.073 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1685566501.757183_260.1', 1: '1_1685566501.735717_251.1', 2: '1_1685562970.274572_93.1', 3: '1_1685568489.956971_1011.1'} 2023-05-31T21:35:44.073 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone a2 behind master: shard=3 master=1_1685568489.974220_1014.1 target=1_1685568489.956971_1011.1 2023-05-31T21:35:44.073 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-31T21:35:44.073 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-31T21:35:44.074 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-05-31T21:35:44.074 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: ignoring --setuser ceph since I am not root 2023-05-31T21:35:44.074 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-05-31T21:35:44.074 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: 2023-05-31T21:28:15.338+0000 7f1eaef38780 20 RGW-SYNC:meta: read sync status 2023-05-31T21:35:44.075 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: 2023-05-31T21:28:15.339+0000 7f1eaef38780 20 RGW-SYNC:meta: read sync status 2023-05-31T21:35:44.075 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2023-05-31T21:35:44.075 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "sync_status": { 2023-05-31T21:35:44.075 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "info": { 2023-05-31T21:35:44.076 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "status": "sync", 2023-05-31T21:35:44.076 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "num_shards": 4, 2023-05-31T21:35:44.076 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "period": "2e6123d2-d695-4350-9a01-61a455bc782b", 2023-05-31T21:35:44.076 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "realm_epoch": 2 2023-05-31T21:35:44.077 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2023-05-31T21:35:44.077 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "markers": [ 2023-05-31T21:35:44.077 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2023-05-31T21:35:44.077 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "key": 0, 2023-05-31T21:35:44.078 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "val": { 2023-05-31T21:35:44.078 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "state": 1, 2023-05-31T21:35:44.078 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "marker": "1_1685566501.757183_260.1", 2023-05-31T21:35:44.078 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "next_step_marker": "", 2023-05-31T21:35:44.078 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "total_entries": 1, 2023-05-31T21:35:44.079 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "pos": 0, 2023-05-31T21:35:44.079 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "timestamp": "2023-05-31T20:55:01.757183Z", 2023-05-31T21:35:44.079 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "realm_epoch": 2 2023-05-31T21:35:44.079 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2023-05-31T21:35:44.080 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2023-05-31T21:35:44.080 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2023-05-31T21:35:44.080 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "key": 1, 2023-05-31T21:35:44.080 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "val": { 2023-05-31T21:35:44.081 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "state": 1, 2023-05-31T21:35:44.081 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "marker": "1_1685566501.735717_251.1", 2023-05-31T21:35:44.081 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "next_step_marker": "", 2023-05-31T21:35:44.081 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "total_entries": 0, 2023-05-31T21:35:44.081 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "pos": 0, 2023-05-31T21:35:44.082 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "timestamp": "2023-05-31T20:55:01.735717Z", 2023-05-31T21:35:44.082 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "realm_epoch": 2 2023-05-31T21:35:44.082 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2023-05-31T21:35:44.082 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2023-05-31T21:35:44.083 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2023-05-31T21:35:44.083 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "key": 2, 2023-05-31T21:35:44.083 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "val": { 2023-05-31T21:35:44.083 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "state": 1, 2023-05-31T21:35:44.084 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "marker": "1_1685562970.274572_93.1", 2023-05-31T21:35:44.084 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "next_step_marker": "", 2023-05-31T21:35:44.084 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "total_entries": 0, 2023-05-31T21:35:44.084 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "pos": 0, 2023-05-31T21:35:44.084 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "timestamp": "2023-05-31T19:56:10.274572Z", 2023-05-31T21:35:44.085 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "realm_epoch": 2 2023-05-31T21:35:44.085 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2023-05-31T21:35:44.085 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2023-05-31T21:35:44.085 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2023-05-31T21:35:44.086 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "key": 3, 2023-05-31T21:35:44.086 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "val": { 2023-05-31T21:35:44.086 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "state": 1, 2023-05-31T21:35:44.086 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "marker": "1_1685568489.974220_1014.1", 2023-05-31T21:35:44.087 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "next_step_marker": "", 2023-05-31T21:35:44.087 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "total_entries": 0, 2023-05-31T21:35:44.087 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "pos": 0, 2023-05-31T21:35:44.087 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "timestamp": "2023-05-31T21:28:09.974220Z", 2023-05-31T21:35:44.087 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "realm_epoch": 2 2023-05-31T21:35:44.088 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2023-05-31T21:35:44.088 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2023-05-31T21:35:44.088 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: ] 2023-05-31T21:35:44.088 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2023-05-31T21:35:44.089 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "full_sync": { 2023-05-31T21:35:44.089 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "total": 1, 2023-05-31T21:35:44.089 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "complete": 1 2023-05-31T21:35:44.089 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2023-05-31T21:35:44.090 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "current_time": "2023-05-31T21:28:15Z" 2023-05-31T21:35:44.090 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2023-05-31T21:35:44.090 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2023-05-31T21:35:44.090 INFO:tasks.rgw_multisite_tests: "sync_status": { 2023-05-31T21:35:44.090 INFO:tasks.rgw_multisite_tests: "info": { 2023-05-31T21:35:44.091 INFO:tasks.rgw_multisite_tests: "status": "sync", 2023-05-31T21:35:44.091 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2023-05-31T21:35:44.091 INFO:tasks.rgw_multisite_tests: "period": "2e6123d2-d695-4350-9a01-61a455bc782b", 2023-05-31T21:35:44.091 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-05-31T21:35:44.092 INFO:tasks.rgw_multisite_tests: }, 2023-05-31T21:35:44.092 INFO:tasks.rgw_multisite_tests: "markers": [ 2023-05-31T21:35:44.092 INFO:tasks.rgw_multisite_tests: { 2023-05-31T21:35:44.092 INFO:tasks.rgw_multisite_tests: "key": 0, 2023-05-31T21:35:44.093 INFO:tasks.rgw_multisite_tests: "val": { 2023-05-31T21:35:44.093 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-05-31T21:35:44.093 INFO:tasks.rgw_multisite_tests: "marker": "1_1685566501.757183_260.1", 2023-05-31T21:35:44.093 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-05-31T21:35:44.094 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2023-05-31T21:35:44.094 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-05-31T21:35:44.094 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-05-31T20:55:01.757183Z", 2023-05-31T21:35:44.094 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-05-31T21:35:44.094 INFO:tasks.rgw_multisite_tests: } 2023-05-31T21:35:44.095 INFO:tasks.rgw_multisite_tests: }, 2023-05-31T21:35:44.095 INFO:tasks.rgw_multisite_tests: { 2023-05-31T21:35:44.095 INFO:tasks.rgw_multisite_tests: "key": 1, 2023-05-31T21:35:44.095 INFO:tasks.rgw_multisite_tests: "val": { 2023-05-31T21:35:44.096 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-05-31T21:35:44.096 INFO:tasks.rgw_multisite_tests: "marker": "1_1685566501.735717_251.1", 2023-05-31T21:35:44.096 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-05-31T21:35:44.096 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-05-31T21:35:44.097 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-05-31T21:35:44.097 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-05-31T20:55:01.735717Z", 2023-05-31T21:35:44.097 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-05-31T21:35:44.097 INFO:tasks.rgw_multisite_tests: } 2023-05-31T21:35:44.097 INFO:tasks.rgw_multisite_tests: }, 2023-05-31T21:35:44.098 INFO:tasks.rgw_multisite_tests: { 2023-05-31T21:35:44.098 INFO:tasks.rgw_multisite_tests: "key": 2, 2023-05-31T21:35:44.098 INFO:tasks.rgw_multisite_tests: "val": { 2023-05-31T21:35:44.098 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-05-31T21:35:44.099 INFO:tasks.rgw_multisite_tests: "marker": "1_1685562970.274572_93.1", 2023-05-31T21:35:44.099 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-05-31T21:35:44.099 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-05-31T21:35:44.099 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-05-31T21:35:44.100 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-05-31T19:56:10.274572Z", 2023-05-31T21:35:44.100 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-05-31T21:35:44.100 INFO:tasks.rgw_multisite_tests: } 2023-05-31T21:35:44.100 INFO:tasks.rgw_multisite_tests: }, 2023-05-31T21:35:44.100 INFO:tasks.rgw_multisite_tests: { 2023-05-31T21:35:44.101 INFO:tasks.rgw_multisite_tests: "key": 3, 2023-05-31T21:35:44.101 INFO:tasks.rgw_multisite_tests: "val": { 2023-05-31T21:35:44.101 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-05-31T21:35:44.101 INFO:tasks.rgw_multisite_tests: "marker": "1_1685568489.974220_1014.1", 2023-05-31T21:35:44.102 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-05-31T21:35:44.102 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-05-31T21:35:44.102 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-05-31T21:35:44.102 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-05-31T21:28:09.974220Z", 2023-05-31T21:35:44.103 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-05-31T21:35:44.103 INFO:tasks.rgw_multisite_tests: } 2023-05-31T21:35:44.103 INFO:tasks.rgw_multisite_tests: } 2023-05-31T21:35:44.103 INFO:tasks.rgw_multisite_tests: ] 2023-05-31T21:35:44.104 INFO:tasks.rgw_multisite_tests: }, 2023-05-31T21:35:44.104 INFO:tasks.rgw_multisite_tests: "full_sync": { 2023-05-31T21:35:44.104 INFO:tasks.rgw_multisite_tests: "total": 1, 2023-05-31T21:35:44.104 INFO:tasks.rgw_multisite_tests: "complete": 1 2023-05-31T21:35:44.104 INFO:tasks.rgw_multisite_tests: }, 2023-05-31T21:35:44.105 INFO:tasks.rgw_multisite_tests: "current_time": "2023-05-31T21:28:15Z" 2023-05-31T21:35:44.105 INFO:tasks.rgw_multisite_tests:} 2023-05-31T21:35:44.105 INFO:tasks.rgw_multisite_tests: 2023-05-31T21:35:44.105 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1685566501.757183_260.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-05-31T20:55:01.757183Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1685566501.735717_251.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-05-31T20:55:01.735717Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1685562970.274572_93.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-05-31T19:56:10.274572Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1685568489.974220_1014.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-05-31T21:28:09.974220Z', 'realm_epoch': 2}}] 2023-05-31T21:35:44.106 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1685566501.757183_260.1', 1: '1_1685566501.735717_251.1', 2: '1_1685562970.274572_93.1', 3: '1_1685568489.974220_1014.1'} 2023-05-31T21:35:44.106 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1685566501.757183_260.1', 1: '1_1685566501.735717_251.1', 2: '1_1685562970.274572_93.1', 3: '1_1685568489.974220_1014.1'} 2023-05-31T21:35:44.106 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: INFO: finish meta checkpoint for zone=a2 2023-05-31T21:35:44.106 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['sync', 'group', 'create', '--group-id', 'sync-group', '--status', 'allowed', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-05-31T21:35:44.107 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'group', 'create', '--group-id', 'sync-group', '--status', 'allowed', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-05-31T21:35:44.107 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync group create --group-id sync-group --status allowed --cluster c1 --debug-rgw 0 --debug-ms 0 2023-05-31T21:35:44.107 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: ignoring --setuser ceph since I am not root 2023-05-31T21:35:44.107 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-05-31T21:35:44.108 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: {"groups":[{"id":"sync-group","data_flow":{},"pipes":[],"status":"allowed"}]} 2023-05-31T21:35:44.108 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['sync', 'group', 'flow', 'create', '--group-id', 'sync-group', '--flow-id', 'sync-flow', '--flow-type', 'directional', '--source-zone=a1', '--dest-zone=a2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-05-31T21:35:44.108 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'group', 'flow', 'create', '--group-id', 'sync-group', '--flow-id', 'sync-flow', '--flow-type', 'directional', '--source-zone=a1', '--dest-zone=a2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-05-31T21:35:44.108 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync group flow create --group-id sync-group --flow-id sync-flow --flow-type directional --source-zone=a1 --dest-zone=a2 --cluster c1 --debug-rgw 0 --debug-ms 0 2023-05-31T21:35:44.109 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: ignoring --setuser ceph since I am not root 2023-05-31T21:35:44.109 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-05-31T21:35:44.109 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: {"groups":[{"id":"sync-group","data_flow":{"directional":[{"source_zone":"a1","dest_zone":"a2"}]},"pipes":[],"status":"allowed"}]} 2023-05-31T21:35:44.109 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['sync', 'group', 'pipe', 'create', '--group-id', 'sync-group', '--pipe-id', 'sync-pipe-pklxkd-72', '--source-zones=a1', '--dest-zones=a2', '--source-bucket=pklxkd-72', '--dest-bucket=pklxkd-73', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-05-31T21:35:44.109 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'group', 'pipe', 'create', '--group-id', 'sync-group', '--pipe-id', 'sync-pipe-pklxkd-72', '--source-zones=a1', '--dest-zones=a2', '--source-bucket=pklxkd-72', '--dest-bucket=pklxkd-73', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-05-31T21:35:44.110 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync group pipe create --group-id sync-group --pipe-id sync-pipe-pklxkd-72 --source-zones=a1 --dest-zones=a2 --source-bucket=pklxkd-72 --dest-bucket=pklxkd-73 --cluster c1 --debug-rgw 0 --debug-ms 0 2023-05-31T21:35:44.110 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: ignoring --setuser ceph since I am not root 2023-05-31T21:35:44.110 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-05-31T21:35:44.110 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: {"groups":[{"id":"sync-group","data_flow":{"directional":[{"source_zone":"a1","dest_zone":"a2"}]},"pipes":[{"id":"sync-pipe-pklxkd-72","source":{"bucket":"pklxkd-72","zones":["a1"]},"dest":{"bucket":"pklxkd-73","zones":["a2"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":""}}],"status":"allowed"}]} 2023-05-31T21:35:44.111 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['sync', 'group', 'pipe', 'create', '--group-id', 'sync-group', '--pipe-id', 'sync-pipe-pklxkd-73', '--source-zones=a1', '--dest-zones=a2', '--source-bucket=pklxkd-73', '--dest-bucket=pklxkd-73', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-05-31T21:35:44.111 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'group', 'pipe', 'create', '--group-id', 'sync-group', '--pipe-id', 'sync-pipe-pklxkd-73', '--source-zones=a1', '--dest-zones=a2', '--source-bucket=pklxkd-73', '--dest-bucket=pklxkd-73', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-05-31T21:35:44.111 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync group pipe create --group-id sync-group --pipe-id sync-pipe-pklxkd-73 --source-zones=a1 --dest-zones=a2 --source-bucket=pklxkd-73 --dest-bucket=pklxkd-73 --cluster c1 --debug-rgw 0 --debug-ms 0 2023-05-31T21:35:44.111 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: ignoring --setuser ceph since I am not root 2023-05-31T21:35:44.112 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-05-31T21:35:44.112 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: {"groups":[{"id":"sync-group","data_flow":{"directional":[{"source_zone":"a1","dest_zone":"a2"}]},"pipes":[{"id":"sync-pipe-pklxkd-72","source":{"bucket":"pklxkd-72","zones":["a1"]},"dest":{"bucket":"pklxkd-73","zones":["a2"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":""}},{"id":"sync-pipe-pklxkd-73","source":{"bucket":"pklxkd-73","zones":["a1"]},"dest":{"bucket":"pklxkd-73","zones":["a2"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":""}}],"status":"allowed"}]} 2023-05-31T21:35:44.112 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['sync', 'group', 'modify', '--group-id', 'sync-group', '--status', 'enabled', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-05-31T21:35:44.112 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'group', 'modify', '--group-id', 'sync-group', '--status', 'enabled', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-05-31T21:35:44.113 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync group modify --group-id sync-group --status enabled --cluster c1 --debug-rgw 0 --debug-ms 0 2023-05-31T21:35:44.113 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: ignoring --setuser ceph since I am not root 2023-05-31T21:35:44.113 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-05-31T21:35:44.113 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: {"groups":[{"id":"sync-group","data_flow":{"directional":[{"source_zone":"a1","dest_zone":"a2"}]},"pipes":[{"id":"sync-pipe-pklxkd-72","source":{"bucket":"pklxkd-72","zones":["a1"]},"dest":{"bucket":"pklxkd-73","zones":["a2"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":""}},{"id":"sync-pipe-pklxkd-73","source":{"bucket":"pklxkd-73","zones":["a1"]},"dest":{"bucket":"pklxkd-73","zones":["a2"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":""}}],"status":"enabled"}]} 2023-05-31T21:35:44.113 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['period', 'update', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-05-31T21:35:44.114 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'period', 'update', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-05-31T21:35:44.114 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 period update --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --commit --cluster c1 --debug-rgw 0 --debug-ms 0 2023-05-31T21:35:44.114 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: ignoring --setuser ceph since I am not root 2023-05-31T21:35:44.114 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-05-31T21:35:44.115 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2023-05-31T21:35:44.115 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "id": "2e6123d2-d695-4350-9a01-61a455bc782b", 2023-05-31T21:35:44.115 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "epoch": 19, 2023-05-31T21:35:44.115 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "predecessor_uuid": "629b9ced-80af-41ed-8a6c-f2cfc334cbb4", 2023-05-31T21:35:44.116 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "sync_status": [], 2023-05-31T21:35:44.116 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "period_map": { 2023-05-31T21:35:44.116 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "id": "2e6123d2-d695-4350-9a01-61a455bc782b", 2023-05-31T21:35:44.116 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "zonegroups": [ 2023-05-31T21:35:44.117 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2023-05-31T21:35:44.117 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "id": "a", 2023-05-31T21:35:44.117 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "name": "a", 2023-05-31T21:35:44.117 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "api_name": "", 2023-05-31T21:35:44.117 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "is_master": true, 2023-05-31T21:35:44.118 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "endpoints": [ 2023-05-31T21:35:44.118 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "http://smithi099.front.sepia.ceph.com:8000" 2023-05-31T21:35:44.118 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: ], 2023-05-31T21:35:44.118 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "hostnames": [], 2023-05-31T21:35:44.119 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "hostnames_s3website": [], 2023-05-31T21:35:44.119 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "master_zone": "3d163b1e-09c7-47e2-ae87-add4aace3f38", 2023-05-31T21:35:44.119 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "zones": [ 2023-05-31T21:35:44.119 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2023-05-31T21:35:44.119 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "id": "3d163b1e-09c7-47e2-ae87-add4aace3f38", 2023-05-31T21:35:44.120 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "name": "a1", 2023-05-31T21:35:44.120 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "endpoints": [ 2023-05-31T21:35:44.120 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "http://smithi099.front.sepia.ceph.com:8000" 2023-05-31T21:35:44.120 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: ], 2023-05-31T21:35:44.121 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "log_meta": false, 2023-05-31T21:35:44.121 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "log_data": true, 2023-05-31T21:35:44.121 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "bucket_index_max_shards": 11, 2023-05-31T21:35:44.121 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "read_only": false, 2023-05-31T21:35:44.122 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "tier_type": "", 2023-05-31T21:35:44.122 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "sync_from_all": true, 2023-05-31T21:35:44.122 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "sync_from": [], 2023-05-31T21:35:44.122 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "redirect_zone": "", 2023-05-31T21:35:44.123 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "supported_features": [ 2023-05-31T21:35:44.123 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "resharding" 2023-05-31T21:35:44.123 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: ] 2023-05-31T21:35:44.123 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2023-05-31T21:35:44.123 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2023-05-31T21:35:44.124 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "id": "5d8acec7-7e57-4e56-8b32-e3d6b5036001", 2023-05-31T21:35:44.124 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "name": "a2", 2023-05-31T21:35:44.124 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "endpoints": [ 2023-05-31T21:35:44.124 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "http://smithi099.front.sepia.ceph.com:8001" 2023-05-31T21:35:44.125 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: ], 2023-05-31T21:35:44.125 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "log_meta": false, 2023-05-31T21:35:44.125 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "log_data": true, 2023-05-31T21:35:44.125 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "bucket_index_max_shards": 11, 2023-05-31T21:35:44.125 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "read_only": false, 2023-05-31T21:35:44.126 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "tier_type": "", 2023-05-31T21:35:44.126 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "sync_from_all": true, 2023-05-31T21:35:44.126 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "sync_from": [], 2023-05-31T21:35:44.126 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "redirect_zone": "", 2023-05-31T21:35:44.127 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "supported_features": [ 2023-05-31T21:35:44.127 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "resharding" 2023-05-31T21:35:44.127 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: ] 2023-05-31T21:35:44.127 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2023-05-31T21:35:44.128 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: ], 2023-05-31T21:35:44.128 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "placement_targets": [ 2023-05-31T21:35:44.128 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2023-05-31T21:35:44.128 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "name": "default-placement", 2023-05-31T21:35:44.128 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "tags": [], 2023-05-31T21:35:44.129 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "storage_classes": [ 2023-05-31T21:35:44.129 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "STANDARD" 2023-05-31T21:35:44.129 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: ] 2023-05-31T21:35:44.129 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2023-05-31T21:35:44.130 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: ], 2023-05-31T21:35:44.130 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "default_placement": "default-placement", 2023-05-31T21:35:44.130 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "realm_id": "aaa33de6-515b-4cac-8f1b-3a1a30452117", 2023-05-31T21:35:44.130 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "sync_policy": { 2023-05-31T21:35:44.131 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "groups": [ 2023-05-31T21:35:44.131 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2023-05-31T21:35:44.131 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "id": "sync-group", 2023-05-31T21:35:44.131 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "data_flow": { 2023-05-31T21:35:44.131 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "directional": [ 2023-05-31T21:35:44.132 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2023-05-31T21:35:44.132 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "source_zone": "3d163b1e-09c7-47e2-ae87-add4aace3f38", 2023-05-31T21:35:44.132 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "dest_zone": "5d8acec7-7e57-4e56-8b32-e3d6b5036001" 2023-05-31T21:35:44.132 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2023-05-31T21:35:44.133 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: ] 2023-05-31T21:35:44.133 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2023-05-31T21:35:44.133 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "pipes": [ 2023-05-31T21:35:44.133 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2023-05-31T21:35:44.134 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "id": "sync-pipe-pklxkd-72", 2023-05-31T21:35:44.134 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "source": { 2023-05-31T21:35:44.134 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "bucket": "pklxkd-72", 2023-05-31T21:35:44.134 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "zones": [ 2023-05-31T21:35:44.134 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "3d163b1e-09c7-47e2-ae87-add4aace3f38" 2023-05-31T21:35:44.135 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: ] 2023-05-31T21:35:44.135 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2023-05-31T21:35:44.135 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "dest": { 2023-05-31T21:35:44.135 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "bucket": "pklxkd-73", 2023-05-31T21:35:44.136 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "zones": [ 2023-05-31T21:35:44.136 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "5d8acec7-7e57-4e56-8b32-e3d6b5036001" 2023-05-31T21:35:44.136 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: ] 2023-05-31T21:35:44.136 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2023-05-31T21:35:44.137 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "params": { 2023-05-31T21:35:44.137 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "source": { 2023-05-31T21:35:44.137 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "filter": { 2023-05-31T21:35:44.137 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "tags": [] 2023-05-31T21:35:44.138 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2023-05-31T21:35:44.138 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2023-05-31T21:35:44.138 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "dest": {}, 2023-05-31T21:35:44.138 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "priority": 0, 2023-05-31T21:35:44.138 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "mode": "system", 2023-05-31T21:35:44.139 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "user": "" 2023-05-31T21:35:44.139 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2023-05-31T21:35:44.139 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2023-05-31T21:35:44.139 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2023-05-31T21:35:44.140 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "id": "sync-pipe-pklxkd-73", 2023-05-31T21:35:44.140 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "source": { 2023-05-31T21:35:44.140 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "bucket": "pklxkd-73", 2023-05-31T21:35:44.140 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "zones": [ 2023-05-31T21:35:44.141 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "3d163b1e-09c7-47e2-ae87-add4aace3f38" 2023-05-31T21:35:44.141 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: ] 2023-05-31T21:35:44.141 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2023-05-31T21:35:44.141 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "dest": { 2023-05-31T21:35:44.141 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "bucket": "pklxkd-73", 2023-05-31T21:35:44.142 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "zones": [ 2023-05-31T21:35:44.142 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "5d8acec7-7e57-4e56-8b32-e3d6b5036001" 2023-05-31T21:35:44.142 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: ] 2023-05-31T21:35:44.142 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2023-05-31T21:35:44.143 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "params": { 2023-05-31T21:35:44.143 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "source": { 2023-05-31T21:35:44.143 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "filter": { 2023-05-31T21:35:44.143 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "tags": [] 2023-05-31T21:35:44.144 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2023-05-31T21:35:44.144 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2023-05-31T21:35:44.144 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "dest": {}, 2023-05-31T21:35:44.144 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "priority": 0, 2023-05-31T21:35:44.144 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "mode": "system", 2023-05-31T21:35:44.145 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "user": "" 2023-05-31T21:35:44.145 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2023-05-31T21:35:44.145 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2023-05-31T21:35:44.145 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: ], 2023-05-31T21:35:44.146 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "status": "enabled" 2023-05-31T21:35:44.146 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2023-05-31T21:35:44.146 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: ] 2023-05-31T21:35:44.146 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2023-05-31T21:35:44.146 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "enabled_features": [ 2023-05-31T21:35:44.147 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "resharding" 2023-05-31T21:35:44.147 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: ] 2023-05-31T21:35:44.147 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2023-05-31T21:35:44.147 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2023-05-31T21:35:44.148 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "id": "b", 2023-05-31T21:35:44.148 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "name": "b", 2023-05-31T21:35:44.148 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "api_name": "", 2023-05-31T21:35:44.148 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "is_master": false, 2023-05-31T21:35:44.149 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "endpoints": [ 2023-05-31T21:35:44.149 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "http://smithi112.front.sepia.ceph.com:8000" 2023-05-31T21:35:44.149 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: ], 2023-05-31T21:35:44.149 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "hostnames": [], 2023-05-31T21:35:44.149 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "hostnames_s3website": [], 2023-05-31T21:35:44.150 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "master_zone": "f85d7236-2f27-4988-af26-9c66ab280b1b", 2023-05-31T21:35:44.150 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "zones": [ 2023-05-31T21:35:44.150 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2023-05-31T21:35:44.150 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "id": "a6a3aab5-bc52-44a4-8543-5c5ea6056acd", 2023-05-31T21:35:44.151 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "name": "b2", 2023-05-31T21:35:44.151 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "endpoints": [ 2023-05-31T21:35:44.151 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "http://smithi112.front.sepia.ceph.com:8001" 2023-05-31T21:35:44.151 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: ], 2023-05-31T21:35:44.152 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "log_meta": false, 2023-05-31T21:35:44.152 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "log_data": true, 2023-05-31T21:35:44.152 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "bucket_index_max_shards": 11, 2023-05-31T21:35:44.152 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "read_only": false, 2023-05-31T21:35:44.152 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "tier_type": "", 2023-05-31T21:35:44.153 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "sync_from_all": true, 2023-05-31T21:35:44.153 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "sync_from": [], 2023-05-31T21:35:44.153 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "redirect_zone": "", 2023-05-31T21:35:44.153 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "supported_features": [ 2023-05-31T21:35:44.154 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "resharding" 2023-05-31T21:35:44.154 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: ] 2023-05-31T21:35:44.154 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2023-05-31T21:35:44.154 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2023-05-31T21:35:44.155 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "id": "f85d7236-2f27-4988-af26-9c66ab280b1b", 2023-05-31T21:35:44.155 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "name": "b1", 2023-05-31T21:35:44.155 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "endpoints": [ 2023-05-31T21:35:44.155 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "http://smithi112.front.sepia.ceph.com:8000" 2023-05-31T21:35:44.155 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: ], 2023-05-31T21:35:44.156 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "log_meta": false, 2023-05-31T21:35:44.156 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "log_data": true, 2023-05-31T21:35:44.156 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "bucket_index_max_shards": 11, 2023-05-31T21:35:44.156 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "read_only": false, 2023-05-31T21:35:44.157 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "tier_type": "", 2023-05-31T21:35:44.157 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "sync_from_all": true, 2023-05-31T21:35:44.157 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "sync_from": [], 2023-05-31T21:35:44.157 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "redirect_zone": "", 2023-05-31T21:35:44.158 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "supported_features": [ 2023-05-31T21:35:44.158 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "resharding" 2023-05-31T21:35:44.158 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: ] 2023-05-31T21:35:44.158 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2023-05-31T21:35:44.158 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: ], 2023-05-31T21:35:44.159 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "placement_targets": [ 2023-05-31T21:35:44.159 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2023-05-31T21:35:44.159 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "name": "default-placement", 2023-05-31T21:35:44.159 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "tags": [], 2023-05-31T21:35:44.160 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "storage_classes": [ 2023-05-31T21:35:44.160 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "STANDARD" 2023-05-31T21:35:44.160 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: ] 2023-05-31T21:35:44.160 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2023-05-31T21:35:44.161 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: ], 2023-05-31T21:35:44.161 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "default_placement": "default-placement", 2023-05-31T21:35:44.161 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "realm_id": "aaa33de6-515b-4cac-8f1b-3a1a30452117", 2023-05-31T21:35:44.161 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "sync_policy": { 2023-05-31T21:35:44.161 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "groups": [] 2023-05-31T21:35:44.162 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2023-05-31T21:35:44.162 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "enabled_features": [ 2023-05-31T21:35:44.162 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "resharding" 2023-05-31T21:35:44.162 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: ] 2023-05-31T21:35:44.163 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2023-05-31T21:35:44.163 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: ], 2023-05-31T21:35:44.163 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "short_zone_ids": [ 2023-05-31T21:35:44.163 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2023-05-31T21:35:44.163 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "key": "3d163b1e-09c7-47e2-ae87-add4aace3f38", 2023-05-31T21:35:44.164 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "val": 3101833771 2023-05-31T21:35:44.164 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2023-05-31T21:35:44.164 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2023-05-31T21:35:44.164 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "key": "5d8acec7-7e57-4e56-8b32-e3d6b5036001", 2023-05-31T21:35:44.165 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "val": 4255530623 2023-05-31T21:35:44.165 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2023-05-31T21:35:44.165 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2023-05-31T21:35:44.165 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "key": "a6a3aab5-bc52-44a4-8543-5c5ea6056acd", 2023-05-31T21:35:44.166 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "val": 3708073710 2023-05-31T21:35:44.166 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2023-05-31T21:35:44.166 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2023-05-31T21:35:44.166 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "key": "f85d7236-2f27-4988-af26-9c66ab280b1b", 2023-05-31T21:35:44.167 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "val": 233395200 2023-05-31T21:35:44.167 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2023-05-31T21:35:44.167 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: ] 2023-05-31T21:35:44.167 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2023-05-31T21:35:44.167 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "master_zonegroup": "a", 2023-05-31T21:35:44.168 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "master_zone": "3d163b1e-09c7-47e2-ae87-add4aace3f38", 2023-05-31T21:35:44.168 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "period_config": { 2023-05-31T21:35:44.168 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "bucket_quota": { 2023-05-31T21:35:44.168 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "enabled": false, 2023-05-31T21:35:44.169 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "check_on_raw": false, 2023-05-31T21:35:44.169 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "max_size": -1, 2023-05-31T21:35:44.169 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "max_size_kb": 0, 2023-05-31T21:35:44.169 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "max_objects": -1 2023-05-31T21:35:44.170 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2023-05-31T21:35:44.170 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "user_quota": { 2023-05-31T21:35:44.170 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "enabled": false, 2023-05-31T21:35:44.170 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "check_on_raw": false, 2023-05-31T21:35:44.170 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "max_size": -1, 2023-05-31T21:35:44.171 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "max_size_kb": 0, 2023-05-31T21:35:44.171 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "max_objects": -1 2023-05-31T21:35:44.171 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2023-05-31T21:35:44.171 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "user_ratelimit": { 2023-05-31T21:35:44.172 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "max_read_ops": 0, 2023-05-31T21:35:44.172 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "max_write_ops": 0, 2023-05-31T21:35:44.172 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "max_read_bytes": 0, 2023-05-31T21:35:44.172 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "max_write_bytes": 0, 2023-05-31T21:35:44.172 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "enabled": false 2023-05-31T21:35:44.173 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2023-05-31T21:35:44.173 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "bucket_ratelimit": { 2023-05-31T21:35:44.173 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "max_read_ops": 0, 2023-05-31T21:35:44.173 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "max_write_ops": 0, 2023-05-31T21:35:44.174 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "max_read_bytes": 0, 2023-05-31T21:35:44.174 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "max_write_bytes": 0, 2023-05-31T21:35:44.174 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "enabled": false 2023-05-31T21:35:44.174 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2023-05-31T21:35:44.175 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "anonymous_ratelimit": { 2023-05-31T21:35:44.175 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "max_read_ops": 0, 2023-05-31T21:35:44.175 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "max_write_ops": 0, 2023-05-31T21:35:44.175 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "max_read_bytes": 0, 2023-05-31T21:35:44.175 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "max_write_bytes": 0, 2023-05-31T21:35:44.176 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "enabled": false 2023-05-31T21:35:44.176 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2023-05-31T21:35:44.176 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2023-05-31T21:35:44.176 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "realm_id": "aaa33de6-515b-4cac-8f1b-3a1a30452117", 2023-05-31T21:35:44.177 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "realm_name": "test-realm", 2023-05-31T21:35:44.177 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "realm_epoch": 2 2023-05-31T21:35:44.177 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2023-05-31T21:35:44.177 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['sync', 'policy', 'get', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-05-31T21:35:44.178 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'policy', 'get', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-05-31T21:35:44.178 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync policy get --cluster c1 --debug-rgw 0 --debug-ms 0 2023-05-31T21:35:44.178 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: ignoring --setuser ceph since I am not root 2023-05-31T21:35:44.178 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-05-31T21:35:44.178 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: {"groups":[{"id":"sync-group","data_flow":{"directional":[{"source_zone":"a1","dest_zone":"a2"}]},"pipes":[{"id":"sync-pipe-pklxkd-72","source":{"bucket":"pklxkd-72","zones":["a1"]},"dest":{"bucket":"pklxkd-73","zones":["a2"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":""}},{"id":"sync-pipe-pklxkd-73","source":{"bucket":"pklxkd-73","zones":["a1"]},"dest":{"bucket":"pklxkd-73","zones":["a2"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":""}}],"status":"enabled"}]} 2023-05-31T21:35:44.179 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/pklxkd-72/ 2023-05-31T21:35:44.179 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/pklxkd-72/ 2023-05-31T21:35:44.179 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-05-31T21:35:44.179 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /pklxkd-72/ 2023-05-31T21:35:44.180 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-05-31T21:35:44.180 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-05-31T21:35:44.180 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T21:35:44.181 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-05-31T21:35:44.181 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-05-31T21:35:44.181 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-05-31T21:35:44.182 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-05-31T21:35:44.182 INFO:tasks.rgw_multisite_tests:HEAD 2023-05-31T21:35:44.182 INFO:tasks.rgw_multisite_tests: 2023-05-31T21:35:44.182 INFO:tasks.rgw_multisite_tests: 2023-05-31T21:35:44.182 INFO:tasks.rgw_multisite_tests:Wed, 31 May 2023 21:28:16 GMT 2023-05-31T21:35:44.183 INFO:tasks.rgw_multisite_tests:/pklxkd-72/ 2023-05-31T21:35:44.183 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-05-31T21:35:44.183 INFO:tasks.rgw_multisite_tests:AWS Q80H11OLNU1S48PPYL26:9PtYFtThg5hPPy0zS71/BqB2Dq0= 2023-05-31T21:35:44.183 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 21:28:16 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:9PtYFtThg5hPPy0zS71/BqB2Dq0=', 'Content-Length': '0'} 2023-05-31T21:35:44.184 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000010c7bc910f3b07c6-006477bbf0-2889-a1'), ('Content-Length', '0'), ('Date', 'Wed, 31 May 2023 21:28:16 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:35:44.184 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/pklxkd-72/obj1 2023-05-31T21:35:44.184 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/pklxkd-72/obj1 2023-05-31T21:35:44.184 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2023-05-31T21:35:44.185 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /pklxkd-72/obj1 2023-05-31T21:35:44.185 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-05-31T21:35:44.185 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2023-05-31T21:35:44.185 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T21:35:44.186 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-05-31T21:35:44.186 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-05-31T21:35:44.186 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-05-31T21:35:44.186 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-05-31T21:35:44.186 INFO:tasks.rgw_multisite_tests:PUT 2023-05-31T21:35:44.187 INFO:tasks.rgw_multisite_tests:qPXxZ/RPSWTmyZje6CcRDA== 2023-05-31T21:35:44.187 INFO:tasks.rgw_multisite_tests:application/octet-stream 2023-05-31T21:35:44.187 INFO:tasks.rgw_multisite_tests:Wed, 31 May 2023 21:28:16 GMT 2023-05-31T21:35:44.187 INFO:tasks.rgw_multisite_tests:/pklxkd-72/obj1 2023-05-31T21:35:44.188 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-05-31T21:35:44.188 INFO:tasks.rgw_multisite_tests:AWS Q80H11OLNU1S48PPYL26:FO1bXcUKeUfkhg7Qx9jopHBdlhA= 2023-05-31T21:35:44.188 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Wed, 31 May 2023 21:28:16 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:FO1bXcUKeUfkhg7Qx9jopHBdlhA='} 2023-05-31T21:35:44.188 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000008423f3ae98190f06-006477bbf0-2889-a1'), ('Date', 'Wed, 31 May 2023 21:28:16 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:35:44.189 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/pklxkd-72/ 2023-05-31T21:35:44.189 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/pklxkd-72/ 2023-05-31T21:35:44.189 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-05-31T21:35:44.189 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /pklxkd-72/ 2023-05-31T21:35:44.190 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-05-31T21:35:44.190 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-05-31T21:35:44.190 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T21:35:44.190 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-05-31T21:35:44.190 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-05-31T21:35:44.191 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-05-31T21:35:44.191 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-05-31T21:35:44.191 INFO:tasks.rgw_multisite_tests:HEAD 2023-05-31T21:35:44.191 INFO:tasks.rgw_multisite_tests: 2023-05-31T21:35:44.192 INFO:tasks.rgw_multisite_tests: 2023-05-31T21:35:44.192 INFO:tasks.rgw_multisite_tests:Wed, 31 May 2023 21:28:16 GMT 2023-05-31T21:35:44.192 INFO:tasks.rgw_multisite_tests:/pklxkd-72/ 2023-05-31T21:35:44.192 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-05-31T21:35:44.193 INFO:tasks.rgw_multisite_tests:AWS Q80H11OLNU1S48PPYL26:9PtYFtThg5hPPy0zS71/BqB2Dq0= 2023-05-31T21:35:44.193 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 21:28:16 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:9PtYFtThg5hPPy0zS71/BqB2Dq0=', 'Content-Length': '0'} 2023-05-31T21:35:44.193 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '6'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000012439cbc60c0185f-006477bbf0-2889-a1'), ('Content-Length', '0'), ('Date', 'Wed, 31 May 2023 21:28:16 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:35:44.193 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/pklxkd-72/obj2 2023-05-31T21:35:44.193 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/pklxkd-72/obj2 2023-05-31T21:35:44.194 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2023-05-31T21:35:44.194 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /pklxkd-72/obj2 2023-05-31T21:35:44.194 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-05-31T21:35:44.194 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2023-05-31T21:35:44.195 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T21:35:44.195 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-05-31T21:35:44.195 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-05-31T21:35:44.195 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-05-31T21:35:44.196 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-05-31T21:35:44.196 INFO:tasks.rgw_multisite_tests:PUT 2023-05-31T21:35:44.196 INFO:tasks.rgw_multisite_tests:qPXxZ/RPSWTmyZje6CcRDA== 2023-05-31T21:35:44.196 INFO:tasks.rgw_multisite_tests:application/octet-stream 2023-05-31T21:35:44.197 INFO:tasks.rgw_multisite_tests:Wed, 31 May 2023 21:28:16 GMT 2023-05-31T21:35:44.197 INFO:tasks.rgw_multisite_tests:/pklxkd-72/obj2 2023-05-31T21:35:44.197 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-05-31T21:35:44.197 INFO:tasks.rgw_multisite_tests:AWS Q80H11OLNU1S48PPYL26:PeN9keHJFLO7RITqn4MZMHXEpAo= 2023-05-31T21:35:44.197 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Wed, 31 May 2023 21:28:16 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:PeN9keHJFLO7RITqn4MZMHXEpAo='} 2023-05-31T21:35:44.198 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000005b9889f488b162b9-006477bbf0-2889-a1'), ('Date', 'Wed, 31 May 2023 21:28:16 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:35:44.198 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/pklxkd-73/ 2023-05-31T21:35:44.198 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/pklxkd-73/ 2023-05-31T21:35:44.198 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-05-31T21:35:44.199 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /pklxkd-73/ 2023-05-31T21:35:44.199 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-05-31T21:35:44.199 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-05-31T21:35:44.199 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T21:35:44.200 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-05-31T21:35:44.200 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-05-31T21:35:44.200 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-05-31T21:35:44.200 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-05-31T21:35:44.200 INFO:tasks.rgw_multisite_tests:HEAD 2023-05-31T21:35:44.201 INFO:tasks.rgw_multisite_tests: 2023-05-31T21:35:44.201 INFO:tasks.rgw_multisite_tests: 2023-05-31T21:35:44.201 INFO:tasks.rgw_multisite_tests:Wed, 31 May 2023 21:28:16 GMT 2023-05-31T21:35:44.201 INFO:tasks.rgw_multisite_tests:/pklxkd-73/ 2023-05-31T21:35:44.202 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-05-31T21:35:44.202 INFO:tasks.rgw_multisite_tests:AWS Q80H11OLNU1S48PPYL26:j+nV5H6YldvybXmh5I9m+woTyoE= 2023-05-31T21:35:44.202 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 21:28:16 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:j+nV5H6YldvybXmh5I9m+woTyoE=', 'Content-Length': '0'} 2023-05-31T21:35:44.202 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000fd0936e3dbc45f97-006477bbf0-2889-a1'), ('Content-Length', '0'), ('Date', 'Wed, 31 May 2023 21:28:16 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:35:44.203 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/pklxkd-73/obj3 2023-05-31T21:35:44.203 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/pklxkd-73/obj3 2023-05-31T21:35:44.203 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2023-05-31T21:35:44.203 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /pklxkd-73/obj3 2023-05-31T21:35:44.203 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-05-31T21:35:44.204 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2023-05-31T21:35:44.204 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T21:35:44.204 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-05-31T21:35:44.204 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-05-31T21:35:44.205 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-05-31T21:35:44.205 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-05-31T21:35:44.205 INFO:tasks.rgw_multisite_tests:PUT 2023-05-31T21:35:44.205 INFO:tasks.rgw_multisite_tests:qPXxZ/RPSWTmyZje6CcRDA== 2023-05-31T21:35:44.206 INFO:tasks.rgw_multisite_tests:application/octet-stream 2023-05-31T21:35:44.206 INFO:tasks.rgw_multisite_tests:Wed, 31 May 2023 21:28:16 GMT 2023-05-31T21:35:44.206 INFO:tasks.rgw_multisite_tests:/pklxkd-73/obj3 2023-05-31T21:35:44.206 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-05-31T21:35:44.206 INFO:tasks.rgw_multisite_tests:AWS Q80H11OLNU1S48PPYL26:2IMH2+VthMq4o9Rpi0wpKGsElT8= 2023-05-31T21:35:44.207 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Wed, 31 May 2023 21:28:16 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:2IMH2+VthMq4o9Rpi0wpKGsElT8='} 2023-05-31T21:35:44.207 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000cd410b439d942f09-006477bbf0-2889-a1'), ('Date', 'Wed, 31 May 2023 21:28:16 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:35:44.207 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/pklxkd-73/ 2023-05-31T21:35:44.207 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/pklxkd-73/ 2023-05-31T21:35:44.208 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-05-31T21:35:44.208 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /pklxkd-73/ 2023-05-31T21:35:44.208 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-05-31T21:35:44.208 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-05-31T21:35:44.209 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T21:35:44.209 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-05-31T21:35:44.209 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-05-31T21:35:44.209 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-05-31T21:35:44.209 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-05-31T21:35:44.210 INFO:tasks.rgw_multisite_tests:HEAD 2023-05-31T21:35:44.210 INFO:tasks.rgw_multisite_tests: 2023-05-31T21:35:44.210 INFO:tasks.rgw_multisite_tests: 2023-05-31T21:35:44.210 INFO:tasks.rgw_multisite_tests:Wed, 31 May 2023 21:28:16 GMT 2023-05-31T21:35:44.211 INFO:tasks.rgw_multisite_tests:/pklxkd-73/ 2023-05-31T21:35:44.211 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-05-31T21:35:44.211 INFO:tasks.rgw_multisite_tests:AWS Q80H11OLNU1S48PPYL26:j+nV5H6YldvybXmh5I9m+woTyoE= 2023-05-31T21:35:44.211 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 21:28:16 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:j+nV5H6YldvybXmh5I9m+woTyoE=', 'Content-Length': '0'} 2023-05-31T21:35:44.212 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '6'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000046de10161e6d37a5-006477bbf0-2889-a1'), ('Content-Length', '0'), ('Date', 'Wed, 31 May 2023 21:28:16 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:35:44.212 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/pklxkd-73/obj4 2023-05-31T21:35:44.212 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/pklxkd-73/obj4 2023-05-31T21:35:44.212 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2023-05-31T21:35:44.212 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /pklxkd-73/obj4 2023-05-31T21:35:44.213 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-05-31T21:35:44.213 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2023-05-31T21:35:44.213 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T21:35:44.213 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-05-31T21:35:44.214 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-05-31T21:35:44.214 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-05-31T21:35:44.214 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-05-31T21:35:44.214 INFO:tasks.rgw_multisite_tests:PUT 2023-05-31T21:35:44.215 INFO:tasks.rgw_multisite_tests:qPXxZ/RPSWTmyZje6CcRDA== 2023-05-31T21:35:44.215 INFO:tasks.rgw_multisite_tests:application/octet-stream 2023-05-31T21:35:44.215 INFO:tasks.rgw_multisite_tests:Wed, 31 May 2023 21:28:16 GMT 2023-05-31T21:35:44.215 INFO:tasks.rgw_multisite_tests:/pklxkd-73/obj4 2023-05-31T21:35:44.215 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-05-31T21:35:44.216 INFO:tasks.rgw_multisite_tests:AWS Q80H11OLNU1S48PPYL26:jmIrMmstM8992AAbTygYZtWzN9c= 2023-05-31T21:35:44.216 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Wed, 31 May 2023 21:28:16 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:jmIrMmstM8992AAbTygYZtWzN9c='} 2023-05-31T21:35:44.216 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000c933cad929c90c98-006477bbf0-2889-a1'), ('Date', 'Wed, 31 May 2023 21:28:16 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:35:44.216 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-31T21:35:44.217 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-31T21:35:44.217 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-05-31T21:35:44.217 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: ignoring --setuser ceph since I am not root 2023-05-31T21:35:44.217 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-05-31T21:35:44.218 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: No --period given, using current period=2e6123d2-d695-4350-9a01-61a455bc782b 2023-05-31T21:35:44.218 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: [ 2023-05-31T21:35:44.218 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2023-05-31T21:35:44.218 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "marker": "1_1685566501.757183_260.1", 2023-05-31T21:35:44.219 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "last_update": "2023-05-31T20:55:01.757183Z" 2023-05-31T21:35:44.219 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2023-05-31T21:35:44.219 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2023-05-31T21:35:44.219 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "marker": "1_1685566501.735717_251.1", 2023-05-31T21:35:44.219 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "last_update": "2023-05-31T20:55:01.735717Z" 2023-05-31T21:35:44.220 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2023-05-31T21:35:44.220 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2023-05-31T21:35:44.220 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "marker": "1_1685562970.274572_93.1", 2023-05-31T21:35:44.220 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "last_update": "2023-05-31T19:56:10.274572Z" 2023-05-31T21:35:44.221 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2023-05-31T21:35:44.221 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2023-05-31T21:35:44.221 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "marker": "1_1685568489.974220_1014.1", 2023-05-31T21:35:44.221 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "last_update": "2023-05-31T21:28:09.974220Z" 2023-05-31T21:35:44.222 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2023-05-31T21:35:44.222 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: ] 2023-05-31T21:35:44.222 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: master meta markers={0: '1_1685566501.757183_260.1', 1: '1_1685566501.735717_251.1', 2: '1_1685562970.274572_93.1', 3: '1_1685568489.974220_1014.1'} 2023-05-31T21:35:44.222 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: INFO: starting meta checkpoint for zone=a2 2023-05-31T21:35:44.222 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-31T21:35:44.223 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-31T21:35:44.223 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-05-31T21:35:44.223 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: ignoring --setuser ceph since I am not root 2023-05-31T21:35:44.223 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-05-31T21:35:44.224 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: 2023-05-31T21:28:16.904+0000 7fc4af060780 20 RGW-SYNC:meta: read sync status 2023-05-31T21:35:44.224 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: 2023-05-31T21:28:16.905+0000 7fc4af060780 20 RGW-SYNC:meta: read sync status 2023-05-31T21:35:44.224 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2023-05-31T21:35:44.224 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "sync_status": { 2023-05-31T21:35:44.225 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "info": { 2023-05-31T21:35:44.225 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "status": "sync", 2023-05-31T21:35:44.225 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "num_shards": 4, 2023-05-31T21:35:44.225 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "period": "2e6123d2-d695-4350-9a01-61a455bc782b", 2023-05-31T21:35:44.225 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "realm_epoch": 2 2023-05-31T21:35:44.226 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2023-05-31T21:35:44.226 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "markers": [ 2023-05-31T21:35:44.226 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2023-05-31T21:35:44.226 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "key": 0, 2023-05-31T21:35:44.227 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "val": { 2023-05-31T21:35:44.227 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "state": 1, 2023-05-31T21:35:44.227 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "marker": "1_1685566501.757183_260.1", 2023-05-31T21:35:44.227 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "next_step_marker": "", 2023-05-31T21:35:44.228 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "total_entries": 1, 2023-05-31T21:35:44.228 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "pos": 0, 2023-05-31T21:35:44.228 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "timestamp": "2023-05-31T20:55:01.757183Z", 2023-05-31T21:35:44.228 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "realm_epoch": 2 2023-05-31T21:35:44.229 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2023-05-31T21:35:44.229 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2023-05-31T21:35:44.229 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2023-05-31T21:35:44.229 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "key": 1, 2023-05-31T21:35:44.229 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "val": { 2023-05-31T21:35:44.230 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "state": 1, 2023-05-31T21:35:44.230 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "marker": "1_1685566501.735717_251.1", 2023-05-31T21:35:44.230 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "next_step_marker": "", 2023-05-31T21:35:44.230 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "total_entries": 0, 2023-05-31T21:35:44.231 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "pos": 0, 2023-05-31T21:35:44.231 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "timestamp": "2023-05-31T20:55:01.735717Z", 2023-05-31T21:35:44.231 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "realm_epoch": 2 2023-05-31T21:35:44.231 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2023-05-31T21:35:44.231 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2023-05-31T21:35:44.232 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2023-05-31T21:35:44.232 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "key": 2, 2023-05-31T21:35:44.232 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "val": { 2023-05-31T21:35:44.232 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "state": 1, 2023-05-31T21:35:44.233 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "marker": "1_1685562970.274572_93.1", 2023-05-31T21:35:44.233 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "next_step_marker": "", 2023-05-31T21:35:44.233 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "total_entries": 0, 2023-05-31T21:35:44.233 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "pos": 0, 2023-05-31T21:35:44.234 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "timestamp": "2023-05-31T19:56:10.274572Z", 2023-05-31T21:35:44.234 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "realm_epoch": 2 2023-05-31T21:35:44.234 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2023-05-31T21:35:44.234 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2023-05-31T21:35:44.234 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2023-05-31T21:35:44.235 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "key": 3, 2023-05-31T21:35:44.235 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "val": { 2023-05-31T21:35:44.235 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "state": 1, 2023-05-31T21:35:44.235 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "marker": "1_1685568489.974220_1014.1", 2023-05-31T21:35:44.236 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "next_step_marker": "", 2023-05-31T21:35:44.236 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "total_entries": 0, 2023-05-31T21:35:44.236 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "pos": 0, 2023-05-31T21:35:44.236 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "timestamp": "2023-05-31T21:28:09.974220Z", 2023-05-31T21:35:44.237 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "realm_epoch": 2 2023-05-31T21:35:44.237 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2023-05-31T21:35:44.237 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2023-05-31T21:35:44.237 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: ] 2023-05-31T21:35:44.237 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2023-05-31T21:35:44.238 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "full_sync": { 2023-05-31T21:35:44.238 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "total": 1, 2023-05-31T21:35:44.238 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "complete": 1 2023-05-31T21:35:44.238 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2023-05-31T21:35:44.239 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "current_time": "2023-05-31T21:28:16Z" 2023-05-31T21:35:44.239 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2023-05-31T21:35:44.239 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2023-05-31T21:35:44.239 INFO:tasks.rgw_multisite_tests: "sync_status": { 2023-05-31T21:35:44.240 INFO:tasks.rgw_multisite_tests: "info": { 2023-05-31T21:35:44.240 INFO:tasks.rgw_multisite_tests: "status": "sync", 2023-05-31T21:35:44.240 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2023-05-31T21:35:44.240 INFO:tasks.rgw_multisite_tests: "period": "2e6123d2-d695-4350-9a01-61a455bc782b", 2023-05-31T21:35:44.240 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-05-31T21:35:44.241 INFO:tasks.rgw_multisite_tests: }, 2023-05-31T21:35:44.241 INFO:tasks.rgw_multisite_tests: "markers": [ 2023-05-31T21:35:44.241 INFO:tasks.rgw_multisite_tests: { 2023-05-31T21:35:44.241 INFO:tasks.rgw_multisite_tests: "key": 0, 2023-05-31T21:35:44.242 INFO:tasks.rgw_multisite_tests: "val": { 2023-05-31T21:35:44.242 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-05-31T21:35:44.242 INFO:tasks.rgw_multisite_tests: "marker": "1_1685566501.757183_260.1", 2023-05-31T21:35:44.242 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-05-31T21:35:44.243 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2023-05-31T21:35:44.243 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-05-31T21:35:44.243 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-05-31T20:55:01.757183Z", 2023-05-31T21:35:44.243 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-05-31T21:35:44.243 INFO:tasks.rgw_multisite_tests: } 2023-05-31T21:35:44.244 INFO:tasks.rgw_multisite_tests: }, 2023-05-31T21:35:44.244 INFO:tasks.rgw_multisite_tests: { 2023-05-31T21:35:44.244 INFO:tasks.rgw_multisite_tests: "key": 1, 2023-05-31T21:35:44.244 INFO:tasks.rgw_multisite_tests: "val": { 2023-05-31T21:35:44.245 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-05-31T21:35:44.245 INFO:tasks.rgw_multisite_tests: "marker": "1_1685566501.735717_251.1", 2023-05-31T21:35:44.245 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-05-31T21:35:44.245 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-05-31T21:35:44.246 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-05-31T21:35:44.246 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-05-31T20:55:01.735717Z", 2023-05-31T21:35:44.246 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-05-31T21:35:44.246 INFO:tasks.rgw_multisite_tests: } 2023-05-31T21:35:44.246 INFO:tasks.rgw_multisite_tests: }, 2023-05-31T21:35:44.247 INFO:tasks.rgw_multisite_tests: { 2023-05-31T21:35:44.247 INFO:tasks.rgw_multisite_tests: "key": 2, 2023-05-31T21:35:44.247 INFO:tasks.rgw_multisite_tests: "val": { 2023-05-31T21:35:44.247 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-05-31T21:35:44.248 INFO:tasks.rgw_multisite_tests: "marker": "1_1685562970.274572_93.1", 2023-05-31T21:35:44.248 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-05-31T21:35:44.248 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-05-31T21:35:44.248 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-05-31T21:35:44.249 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-05-31T19:56:10.274572Z", 2023-05-31T21:35:44.249 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-05-31T21:35:44.249 INFO:tasks.rgw_multisite_tests: } 2023-05-31T21:35:44.249 INFO:tasks.rgw_multisite_tests: }, 2023-05-31T21:35:44.249 INFO:tasks.rgw_multisite_tests: { 2023-05-31T21:35:44.250 INFO:tasks.rgw_multisite_tests: "key": 3, 2023-05-31T21:35:44.250 INFO:tasks.rgw_multisite_tests: "val": { 2023-05-31T21:35:44.250 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-05-31T21:35:44.250 INFO:tasks.rgw_multisite_tests: "marker": "1_1685568489.974220_1014.1", 2023-05-31T21:35:44.251 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-05-31T21:35:44.251 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-05-31T21:35:44.251 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-05-31T21:35:44.251 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-05-31T21:28:09.974220Z", 2023-05-31T21:35:44.251 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-05-31T21:35:44.252 INFO:tasks.rgw_multisite_tests: } 2023-05-31T21:35:44.252 INFO:tasks.rgw_multisite_tests: } 2023-05-31T21:35:44.252 INFO:tasks.rgw_multisite_tests: ] 2023-05-31T21:35:44.252 INFO:tasks.rgw_multisite_tests: }, 2023-05-31T21:35:44.253 INFO:tasks.rgw_multisite_tests: "full_sync": { 2023-05-31T21:35:44.253 INFO:tasks.rgw_multisite_tests: "total": 1, 2023-05-31T21:35:44.253 INFO:tasks.rgw_multisite_tests: "complete": 1 2023-05-31T21:35:44.253 INFO:tasks.rgw_multisite_tests: }, 2023-05-31T21:35:44.254 INFO:tasks.rgw_multisite_tests: "current_time": "2023-05-31T21:28:16Z" 2023-05-31T21:35:44.254 INFO:tasks.rgw_multisite_tests:} 2023-05-31T21:35:44.254 INFO:tasks.rgw_multisite_tests: 2023-05-31T21:35:44.254 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1685566501.757183_260.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-05-31T20:55:01.757183Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1685566501.735717_251.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-05-31T20:55:01.735717Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1685562970.274572_93.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-05-31T19:56:10.274572Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1685568489.974220_1014.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-05-31T21:28:09.974220Z', 'realm_epoch': 2}}] 2023-05-31T21:35:44.255 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1685566501.757183_260.1', 1: '1_1685566501.735717_251.1', 2: '1_1685562970.274572_93.1', 3: '1_1685568489.974220_1014.1'} 2023-05-31T21:35:44.255 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1685566501.757183_260.1', 1: '1_1685566501.735717_251.1', 2: '1_1685562970.274572_93.1', 3: '1_1685568489.974220_1014.1'} 2023-05-31T21:35:44.255 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: INFO: finish meta checkpoint for zone=a2 2023-05-31T21:35:44.255 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['datalog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-31T21:35:44.255 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'datalog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-31T21:35:44.256 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 datalog status --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-05-31T21:35:44.256 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: ignoring --setuser ceph since I am not root 2023-05-31T21:35:44.256 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-05-31T21:35:44.256 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: [ 2023-05-31T21:35:44.257 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2023-05-31T21:35:44.257 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "marker": "00000000000000000000:00000000000000045375", 2023-05-31T21:35:44.257 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "last_update": "2023-05-31T21:28:16.422581Z" 2023-05-31T21:35:44.257 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2023-05-31T21:35:44.258 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2023-05-31T21:35:44.258 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "marker": "00000000000000000000:00000000000000031780", 2023-05-31T21:35:44.258 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "last_update": "2023-05-31T21:28:16.501293Z" 2023-05-31T21:35:44.258 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2023-05-31T21:35:44.259 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2023-05-31T21:35:44.259 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "marker": "00000000000000000000:00000000000000051671", 2023-05-31T21:35:44.259 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "last_update": "2023-05-31T21:28:16.476487Z" 2023-05-31T21:35:44.259 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2023-05-31T21:35:44.259 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2023-05-31T21:35:44.260 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "marker": "00000000000000000000:00000000000000044178", 2023-05-31T21:35:44.260 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "last_update": "2023-05-31T21:27:37.215905Z" 2023-05-31T21:35:44.260 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2023-05-31T21:35:44.260 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: ] 2023-05-31T21:35:44.261 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: data markers for zone=a1 markers={0: '00000000000000000000:00000000000000045375', 1: '00000000000000000000:00000000000000031780', 2: '00000000000000000000:00000000000000051671', 3: '00000000000000000000:00000000000000044178'} 2023-05-31T21:35:44.261 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: INFO: starting data checkpoint for target_zone=a2 source_zone=a1 2023-05-31T21:35:44.261 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-31T21:35:44.261 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-31T21:35:44.262 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-05-31T21:35:44.262 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: ignoring --setuser ceph since I am not root 2023-05-31T21:35:44.262 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-05-31T21:35:44.262 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2023-05-31T21:35:44.262 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "sync_status": { 2023-05-31T21:35:44.263 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "info": { 2023-05-31T21:35:44.263 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "status": "sync", 2023-05-31T21:35:44.263 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "num_shards": 4, 2023-05-31T21:35:44.263 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "instance_id": 452136274422638824 2023-05-31T21:35:44.264 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2023-05-31T21:35:44.264 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "markers": [ 2023-05-31T21:35:44.264 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2023-05-31T21:35:44.264 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "key": 0, 2023-05-31T21:35:44.265 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "val": { 2023-05-31T21:35:44.265 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "status": "incremental-sync", 2023-05-31T21:35:44.265 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "marker": "00000000000000000000:00000000000000045235", 2023-05-31T21:35:44.265 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "next_step_marker": "", 2023-05-31T21:35:44.265 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "total_entries": 0, 2023-05-31T21:35:44.266 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "pos": 0, 2023-05-31T21:35:44.266 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "timestamp": "2023-05-31T21:25:53.171643Z" 2023-05-31T21:35:44.266 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2023-05-31T21:35:44.266 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2023-05-31T21:35:44.267 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2023-05-31T21:35:44.267 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "key": 1, 2023-05-31T21:35:44.267 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "val": { 2023-05-31T21:35:44.267 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "status": "incremental-sync", 2023-05-31T21:35:44.268 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "marker": "00000000000000000000:00000000000000031640", 2023-05-31T21:35:44.268 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "next_step_marker": "", 2023-05-31T21:35:44.268 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "total_entries": 0, 2023-05-31T21:35:44.268 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "pos": 0, 2023-05-31T21:35:44.268 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "timestamp": "2023-05-31T21:27:37.214586Z" 2023-05-31T21:35:44.269 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2023-05-31T21:35:44.269 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2023-05-31T21:35:44.269 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2023-05-31T21:35:44.269 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "key": 2, 2023-05-31T21:35:44.270 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "val": { 2023-05-31T21:35:44.270 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "status": "incremental-sync", 2023-05-31T21:35:44.270 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "marker": "00000000000000000000:00000000000000051391", 2023-05-31T21:35:44.270 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "next_step_marker": "", 2023-05-31T21:35:44.270 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "total_entries": 55, 2023-05-31T21:35:44.271 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "pos": 0, 2023-05-31T21:35:44.271 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "timestamp": "2023-05-31T21:25:53.141700Z" 2023-05-31T21:35:44.271 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2023-05-31T21:35:44.271 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2023-05-31T21:35:44.272 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2023-05-31T21:35:44.272 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "key": 3, 2023-05-31T21:35:44.272 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "val": { 2023-05-31T21:35:44.272 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "status": "incremental-sync", 2023-05-31T21:35:44.273 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "marker": "00000000000000000000:00000000000000044178", 2023-05-31T21:35:44.273 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "next_step_marker": "", 2023-05-31T21:35:44.273 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "total_entries": 9, 2023-05-31T21:35:44.273 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "pos": 0, 2023-05-31T21:35:44.274 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "timestamp": "2023-05-31T21:27:37.215905Z" 2023-05-31T21:35:44.274 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2023-05-31T21:35:44.274 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2023-05-31T21:35:44.274 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: ] 2023-05-31T21:35:44.274 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2023-05-31T21:35:44.275 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "full_sync": { 2023-05-31T21:35:44.275 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "total": 64, 2023-05-31T21:35:44.275 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "complete": 64 2023-05-31T21:35:44.275 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2023-05-31T21:35:44.276 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "current_time": "2023-05-31T21:28:17Z" 2023-05-31T21:35:44.276 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2023-05-31T21:35:44.276 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current data sync status={ 2023-05-31T21:35:44.276 INFO:tasks.rgw_multisite_tests: "sync_status": { 2023-05-31T21:35:44.277 INFO:tasks.rgw_multisite_tests: "info": { 2023-05-31T21:35:44.277 INFO:tasks.rgw_multisite_tests: "status": "sync", 2023-05-31T21:35:44.277 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2023-05-31T21:35:44.277 INFO:tasks.rgw_multisite_tests: "instance_id": 452136274422638824 2023-05-31T21:35:44.277 INFO:tasks.rgw_multisite_tests: }, 2023-05-31T21:35:44.278 INFO:tasks.rgw_multisite_tests: "markers": [ 2023-05-31T21:35:44.278 INFO:tasks.rgw_multisite_tests: { 2023-05-31T21:35:44.278 INFO:tasks.rgw_multisite_tests: "key": 0, 2023-05-31T21:35:44.278 INFO:tasks.rgw_multisite_tests: "val": { 2023-05-31T21:35:44.279 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2023-05-31T21:35:44.279 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000045235", 2023-05-31T21:35:44.279 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-05-31T21:35:44.279 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-05-31T21:35:44.279 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-05-31T21:35:44.280 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-05-31T21:25:53.171643Z" 2023-05-31T21:35:44.280 INFO:tasks.rgw_multisite_tests: } 2023-05-31T21:35:44.280 INFO:tasks.rgw_multisite_tests: }, 2023-05-31T21:35:44.280 INFO:tasks.rgw_multisite_tests: { 2023-05-31T21:35:44.281 INFO:tasks.rgw_multisite_tests: "key": 1, 2023-05-31T21:35:44.281 INFO:tasks.rgw_multisite_tests: "val": { 2023-05-31T21:35:44.281 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2023-05-31T21:35:44.281 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000031640", 2023-05-31T21:35:44.282 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-05-31T21:35:44.282 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-05-31T21:35:44.282 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-05-31T21:35:44.282 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-05-31T21:27:37.214586Z" 2023-05-31T21:35:44.282 INFO:tasks.rgw_multisite_tests: } 2023-05-31T21:35:44.283 INFO:tasks.rgw_multisite_tests: }, 2023-05-31T21:35:44.283 INFO:tasks.rgw_multisite_tests: { 2023-05-31T21:35:44.283 INFO:tasks.rgw_multisite_tests: "key": 2, 2023-05-31T21:35:44.283 INFO:tasks.rgw_multisite_tests: "val": { 2023-05-31T21:35:44.284 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2023-05-31T21:35:44.284 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000051391", 2023-05-31T21:35:44.284 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-05-31T21:35:44.284 INFO:tasks.rgw_multisite_tests: "total_entries": 55, 2023-05-31T21:35:44.285 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-05-31T21:35:44.285 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-05-31T21:25:53.141700Z" 2023-05-31T21:35:44.285 INFO:tasks.rgw_multisite_tests: } 2023-05-31T21:35:44.285 INFO:tasks.rgw_multisite_tests: }, 2023-05-31T21:35:44.285 INFO:tasks.rgw_multisite_tests: { 2023-05-31T21:35:44.286 INFO:tasks.rgw_multisite_tests: "key": 3, 2023-05-31T21:35:44.286 INFO:tasks.rgw_multisite_tests: "val": { 2023-05-31T21:35:44.286 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2023-05-31T21:35:44.286 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000044178", 2023-05-31T21:35:44.287 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-05-31T21:35:44.287 INFO:tasks.rgw_multisite_tests: "total_entries": 9, 2023-05-31T21:35:44.287 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-05-31T21:35:44.287 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-05-31T21:27:37.215905Z" 2023-05-31T21:35:44.288 INFO:tasks.rgw_multisite_tests: } 2023-05-31T21:35:44.288 INFO:tasks.rgw_multisite_tests: } 2023-05-31T21:35:44.288 INFO:tasks.rgw_multisite_tests: ] 2023-05-31T21:35:44.288 INFO:tasks.rgw_multisite_tests: }, 2023-05-31T21:35:44.289 INFO:tasks.rgw_multisite_tests: "full_sync": { 2023-05-31T21:35:44.289 INFO:tasks.rgw_multisite_tests: "total": 64, 2023-05-31T21:35:44.289 INFO:tasks.rgw_multisite_tests: "complete": 64 2023-05-31T21:35:44.289 INFO:tasks.rgw_multisite_tests: }, 2023-05-31T21:35:44.289 INFO:tasks.rgw_multisite_tests: "current_time": "2023-05-31T21:28:17Z" 2023-05-31T21:35:44.290 INFO:tasks.rgw_multisite_tests:} 2023-05-31T21:35:44.290 INFO:tasks.rgw_multisite_tests: 2023-05-31T21:35:44.290 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000045235', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-05-31T21:25:53.171643Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000031640', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-05-31T21:27:37.214586Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000051391', 'next_step_marker': '', 'total_entries': 55, 'pos': 0, 'timestamp': '2023-05-31T21:25:53.141700Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000044178', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2023-05-31T21:27:37.215905Z'}}] 2023-05-31T21:35:44.290 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '00000000000000000000:00000000000000045375', 1: '00000000000000000000:00000000000000031780', 2: '00000000000000000000:00000000000000051671', 3: '00000000000000000000:00000000000000044178'} 2023-05-31T21:35:44.291 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '00000000000000000000:00000000000000045235', 1: '00000000000000000000:00000000000000031640', 2: '00000000000000000000:00000000000000051391', 3: '00000000000000000000:00000000000000044178'} 2023-05-31T21:35:44.291 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: data of zone a2 behind zone a1: shard=0 master=00000000000000000000:00000000000000045375 target=00000000000000000000:00000000000000045235, shard=1 master=00000000000000000000:00000000000000031780 target=00000000000000000000:00000000000000031640, shard=2 master=00000000000000000000:00000000000000051671 target=00000000000000000000:00000000000000051391 2023-05-31T21:35:44.291 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-31T21:35:44.291 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-31T21:35:44.292 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-05-31T21:35:44.292 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: ignoring --setuser ceph since I am not root 2023-05-31T21:35:44.292 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-05-31T21:35:44.292 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2023-05-31T21:35:44.293 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "sync_status": { 2023-05-31T21:35:44.293 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "info": { 2023-05-31T21:35:44.293 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "status": "sync", 2023-05-31T21:35:44.293 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "num_shards": 4, 2023-05-31T21:35:44.293 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "instance_id": 452136274422638824 2023-05-31T21:35:44.294 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2023-05-31T21:35:44.294 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "markers": [ 2023-05-31T21:35:44.294 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2023-05-31T21:35:44.294 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "key": 0, 2023-05-31T21:35:44.295 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "val": { 2023-05-31T21:35:44.295 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "status": "incremental-sync", 2023-05-31T21:35:44.295 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "marker": "00000000000000000000:00000000000000045235", 2023-05-31T21:35:44.295 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "next_step_marker": "", 2023-05-31T21:35:44.296 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "total_entries": 0, 2023-05-31T21:35:44.296 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "pos": 0, 2023-05-31T21:35:44.296 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "timestamp": "2023-05-31T21:25:53.171643Z" 2023-05-31T21:35:44.296 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2023-05-31T21:35:44.296 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2023-05-31T21:35:44.297 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2023-05-31T21:35:44.297 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "key": 1, 2023-05-31T21:35:44.297 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "val": { 2023-05-31T21:35:44.297 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "status": "incremental-sync", 2023-05-31T21:35:44.298 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "marker": "00000000000000000000:00000000000000031640", 2023-05-31T21:35:44.298 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "next_step_marker": "", 2023-05-31T21:35:44.298 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "total_entries": 0, 2023-05-31T21:35:44.298 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "pos": 0, 2023-05-31T21:35:44.299 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "timestamp": "2023-05-31T21:27:37.214586Z" 2023-05-31T21:35:44.299 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2023-05-31T21:35:44.299 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2023-05-31T21:35:44.299 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2023-05-31T21:35:44.299 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "key": 2, 2023-05-31T21:35:44.300 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "val": { 2023-05-31T21:35:44.300 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "status": "incremental-sync", 2023-05-31T21:35:44.300 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "marker": "00000000000000000000:00000000000000051391", 2023-05-31T21:35:44.300 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "next_step_marker": "", 2023-05-31T21:35:44.301 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "total_entries": 55, 2023-05-31T21:35:44.301 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "pos": 0, 2023-05-31T21:35:44.301 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "timestamp": "2023-05-31T21:25:53.141700Z" 2023-05-31T21:35:44.301 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2023-05-31T21:35:44.302 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2023-05-31T21:35:44.302 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2023-05-31T21:35:44.302 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "key": 3, 2023-05-31T21:35:44.302 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "val": { 2023-05-31T21:35:44.302 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "status": "incremental-sync", 2023-05-31T21:35:44.303 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "marker": "00000000000000000000:00000000000000044178", 2023-05-31T21:35:44.303 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "next_step_marker": "", 2023-05-31T21:35:44.303 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "total_entries": 9, 2023-05-31T21:35:44.303 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "pos": 0, 2023-05-31T21:35:44.304 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "timestamp": "2023-05-31T21:27:37.215905Z" 2023-05-31T21:35:44.304 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2023-05-31T21:35:44.304 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2023-05-31T21:35:44.304 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: ] 2023-05-31T21:35:44.305 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2023-05-31T21:35:44.305 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "full_sync": { 2023-05-31T21:35:44.305 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "total": 64, 2023-05-31T21:35:44.305 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "complete": 64 2023-05-31T21:35:44.305 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2023-05-31T21:35:44.306 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "current_time": "2023-05-31T21:28:22Z" 2023-05-31T21:35:44.306 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2023-05-31T21:35:44.306 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current data sync status={ 2023-05-31T21:35:44.306 INFO:tasks.rgw_multisite_tests: "sync_status": { 2023-05-31T21:35:44.307 INFO:tasks.rgw_multisite_tests: "info": { 2023-05-31T21:35:44.307 INFO:tasks.rgw_multisite_tests: "status": "sync", 2023-05-31T21:35:44.307 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2023-05-31T21:35:44.307 INFO:tasks.rgw_multisite_tests: "instance_id": 452136274422638824 2023-05-31T21:35:44.307 INFO:tasks.rgw_multisite_tests: }, 2023-05-31T21:35:44.308 INFO:tasks.rgw_multisite_tests: "markers": [ 2023-05-31T21:35:44.308 INFO:tasks.rgw_multisite_tests: { 2023-05-31T21:35:44.308 INFO:tasks.rgw_multisite_tests: "key": 0, 2023-05-31T21:35:44.308 INFO:tasks.rgw_multisite_tests: "val": { 2023-05-31T21:35:44.309 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2023-05-31T21:35:44.309 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000045235", 2023-05-31T21:35:44.309 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-05-31T21:35:44.309 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-05-31T21:35:44.310 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-05-31T21:35:44.310 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-05-31T21:25:53.171643Z" 2023-05-31T21:35:44.310 INFO:tasks.rgw_multisite_tests: } 2023-05-31T21:35:44.310 INFO:tasks.rgw_multisite_tests: }, 2023-05-31T21:35:44.310 INFO:tasks.rgw_multisite_tests: { 2023-05-31T21:35:44.311 INFO:tasks.rgw_multisite_tests: "key": 1, 2023-05-31T21:35:44.311 INFO:tasks.rgw_multisite_tests: "val": { 2023-05-31T21:35:44.311 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2023-05-31T21:35:44.311 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000031640", 2023-05-31T21:35:44.312 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-05-31T21:35:44.312 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-05-31T21:35:44.312 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-05-31T21:35:44.312 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-05-31T21:27:37.214586Z" 2023-05-31T21:35:44.313 INFO:tasks.rgw_multisite_tests: } 2023-05-31T21:35:44.313 INFO:tasks.rgw_multisite_tests: }, 2023-05-31T21:35:44.313 INFO:tasks.rgw_multisite_tests: { 2023-05-31T21:35:44.313 INFO:tasks.rgw_multisite_tests: "key": 2, 2023-05-31T21:35:44.313 INFO:tasks.rgw_multisite_tests: "val": { 2023-05-31T21:35:44.314 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2023-05-31T21:35:44.314 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000051391", 2023-05-31T21:35:44.314 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-05-31T21:35:44.314 INFO:tasks.rgw_multisite_tests: "total_entries": 55, 2023-05-31T21:35:44.315 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-05-31T21:35:44.315 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-05-31T21:25:53.141700Z" 2023-05-31T21:35:44.315 INFO:tasks.rgw_multisite_tests: } 2023-05-31T21:35:44.315 INFO:tasks.rgw_multisite_tests: }, 2023-05-31T21:35:44.316 INFO:tasks.rgw_multisite_tests: { 2023-05-31T21:35:44.316 INFO:tasks.rgw_multisite_tests: "key": 3, 2023-05-31T21:35:44.316 INFO:tasks.rgw_multisite_tests: "val": { 2023-05-31T21:35:44.316 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2023-05-31T21:35:44.316 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000044178", 2023-05-31T21:35:44.317 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-05-31T21:35:44.317 INFO:tasks.rgw_multisite_tests: "total_entries": 9, 2023-05-31T21:35:44.317 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-05-31T21:35:44.317 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-05-31T21:27:37.215905Z" 2023-05-31T21:35:44.318 INFO:tasks.rgw_multisite_tests: } 2023-05-31T21:35:44.318 INFO:tasks.rgw_multisite_tests: } 2023-05-31T21:35:44.318 INFO:tasks.rgw_multisite_tests: ] 2023-05-31T21:35:44.318 INFO:tasks.rgw_multisite_tests: }, 2023-05-31T21:35:44.319 INFO:tasks.rgw_multisite_tests: "full_sync": { 2023-05-31T21:35:44.319 INFO:tasks.rgw_multisite_tests: "total": 64, 2023-05-31T21:35:44.319 INFO:tasks.rgw_multisite_tests: "complete": 64 2023-05-31T21:35:44.319 INFO:tasks.rgw_multisite_tests: }, 2023-05-31T21:35:44.319 INFO:tasks.rgw_multisite_tests: "current_time": "2023-05-31T21:28:22Z" 2023-05-31T21:35:44.320 INFO:tasks.rgw_multisite_tests:} 2023-05-31T21:35:44.320 INFO:tasks.rgw_multisite_tests: 2023-05-31T21:35:44.320 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000045235', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-05-31T21:25:53.171643Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000031640', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-05-31T21:27:37.214586Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000051391', 'next_step_marker': '', 'total_entries': 55, 'pos': 0, 'timestamp': '2023-05-31T21:25:53.141700Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000044178', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2023-05-31T21:27:37.215905Z'}}] 2023-05-31T21:35:44.320 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '00000000000000000000:00000000000000045375', 1: '00000000000000000000:00000000000000031780', 2: '00000000000000000000:00000000000000051671', 3: '00000000000000000000:00000000000000044178'} 2023-05-31T21:35:44.321 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '00000000000000000000:00000000000000045235', 1: '00000000000000000000:00000000000000031640', 2: '00000000000000000000:00000000000000051391', 3: '00000000000000000000:00000000000000044178'} 2023-05-31T21:35:44.321 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: data of zone a2 behind zone a1: shard=0 master=00000000000000000000:00000000000000045375 target=00000000000000000000:00000000000000045235, shard=1 master=00000000000000000000:00000000000000031780 target=00000000000000000000:00000000000000031640, shard=2 master=00000000000000000000:00000000000000051671 target=00000000000000000000:00000000000000051391 2023-05-31T21:35:44.321 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-31T21:35:44.321 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-31T21:35:44.322 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-05-31T21:35:44.322 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: ignoring --setuser ceph since I am not root 2023-05-31T21:35:44.322 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-05-31T21:35:44.322 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2023-05-31T21:35:44.323 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "sync_status": { 2023-05-31T21:35:44.323 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "info": { 2023-05-31T21:35:44.323 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "status": "sync", 2023-05-31T21:35:44.323 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "num_shards": 4, 2023-05-31T21:35:44.323 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "instance_id": 452136274422638824 2023-05-31T21:35:44.324 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2023-05-31T21:35:44.324 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "markers": [ 2023-05-31T21:35:44.324 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2023-05-31T21:35:44.324 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "key": 0, 2023-05-31T21:35:44.325 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "val": { 2023-05-31T21:35:44.325 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "status": "incremental-sync", 2023-05-31T21:35:44.325 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "marker": "00000000000000000000:00000000000000045235", 2023-05-31T21:35:44.325 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "next_step_marker": "", 2023-05-31T21:35:44.326 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "total_entries": 0, 2023-05-31T21:35:44.326 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "pos": 0, 2023-05-31T21:35:44.326 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "timestamp": "2023-05-31T21:25:53.171643Z" 2023-05-31T21:35:44.326 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2023-05-31T21:35:44.326 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2023-05-31T21:35:44.327 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2023-05-31T21:35:44.327 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "key": 1, 2023-05-31T21:35:44.327 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "val": { 2023-05-31T21:35:44.327 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "status": "incremental-sync", 2023-05-31T21:35:44.328 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "marker": "00000000000000000000:00000000000000031640", 2023-05-31T21:35:44.328 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "next_step_marker": "", 2023-05-31T21:35:44.328 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "total_entries": 0, 2023-05-31T21:35:44.328 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "pos": 0, 2023-05-31T21:35:44.329 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "timestamp": "2023-05-31T21:27:37.214586Z" 2023-05-31T21:35:44.329 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2023-05-31T21:35:44.329 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2023-05-31T21:35:44.329 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2023-05-31T21:35:44.330 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "key": 2, 2023-05-31T21:35:44.330 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "val": { 2023-05-31T21:35:44.330 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "status": "incremental-sync", 2023-05-31T21:35:44.330 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "marker": "00000000000000000000:00000000000000051391", 2023-05-31T21:35:44.330 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "next_step_marker": "", 2023-05-31T21:35:44.331 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "total_entries": 55, 2023-05-31T21:35:44.331 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "pos": 0, 2023-05-31T21:35:44.331 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "timestamp": "2023-05-31T21:25:53.141700Z" 2023-05-31T21:35:44.331 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2023-05-31T21:35:44.332 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2023-05-31T21:35:44.332 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2023-05-31T21:35:44.332 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "key": 3, 2023-05-31T21:35:44.332 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "val": { 2023-05-31T21:35:44.333 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "status": "incremental-sync", 2023-05-31T21:35:44.333 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "marker": "00000000000000000000:00000000000000044178", 2023-05-31T21:35:44.333 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "next_step_marker": "", 2023-05-31T21:35:44.333 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "total_entries": 9, 2023-05-31T21:35:44.333 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "pos": 0, 2023-05-31T21:35:44.334 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "timestamp": "2023-05-31T21:27:37.215905Z" 2023-05-31T21:35:44.334 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2023-05-31T21:35:44.334 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2023-05-31T21:35:44.334 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: ] 2023-05-31T21:35:44.335 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2023-05-31T21:35:44.335 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "full_sync": { 2023-05-31T21:35:44.335 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "total": 64, 2023-05-31T21:35:44.335 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "complete": 64 2023-05-31T21:35:44.335 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2023-05-31T21:35:44.336 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "current_time": "2023-05-31T21:28:27Z" 2023-05-31T21:35:44.336 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2023-05-31T21:35:44.336 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current data sync status={ 2023-05-31T21:35:44.336 INFO:tasks.rgw_multisite_tests: "sync_status": { 2023-05-31T21:35:44.337 INFO:tasks.rgw_multisite_tests: "info": { 2023-05-31T21:35:44.337 INFO:tasks.rgw_multisite_tests: "status": "sync", 2023-05-31T21:35:44.337 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2023-05-31T21:35:44.337 INFO:tasks.rgw_multisite_tests: "instance_id": 452136274422638824 2023-05-31T21:35:44.338 INFO:tasks.rgw_multisite_tests: }, 2023-05-31T21:35:44.338 INFO:tasks.rgw_multisite_tests: "markers": [ 2023-05-31T21:35:44.338 INFO:tasks.rgw_multisite_tests: { 2023-05-31T21:35:44.338 INFO:tasks.rgw_multisite_tests: "key": 0, 2023-05-31T21:35:44.338 INFO:tasks.rgw_multisite_tests: "val": { 2023-05-31T21:35:44.339 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2023-05-31T21:35:44.339 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000045235", 2023-05-31T21:35:44.339 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-05-31T21:35:44.339 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-05-31T21:35:44.340 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-05-31T21:35:44.340 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-05-31T21:25:53.171643Z" 2023-05-31T21:35:44.340 INFO:tasks.rgw_multisite_tests: } 2023-05-31T21:35:44.340 INFO:tasks.rgw_multisite_tests: }, 2023-05-31T21:35:44.341 INFO:tasks.rgw_multisite_tests: { 2023-05-31T21:35:44.341 INFO:tasks.rgw_multisite_tests: "key": 1, 2023-05-31T21:35:44.341 INFO:tasks.rgw_multisite_tests: "val": { 2023-05-31T21:35:44.341 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2023-05-31T21:35:44.341 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000031640", 2023-05-31T21:35:44.342 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-05-31T21:35:44.342 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-05-31T21:35:44.342 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-05-31T21:35:44.342 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-05-31T21:27:37.214586Z" 2023-05-31T21:35:44.343 INFO:tasks.rgw_multisite_tests: } 2023-05-31T21:35:44.343 INFO:tasks.rgw_multisite_tests: }, 2023-05-31T21:35:44.343 INFO:tasks.rgw_multisite_tests: { 2023-05-31T21:35:44.343 INFO:tasks.rgw_multisite_tests: "key": 2, 2023-05-31T21:35:44.344 INFO:tasks.rgw_multisite_tests: "val": { 2023-05-31T21:35:44.344 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2023-05-31T21:35:44.344 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000051391", 2023-05-31T21:35:44.344 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-05-31T21:35:44.344 INFO:tasks.rgw_multisite_tests: "total_entries": 55, 2023-05-31T21:35:44.345 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-05-31T21:35:44.345 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-05-31T21:25:53.141700Z" 2023-05-31T21:35:44.345 INFO:tasks.rgw_multisite_tests: } 2023-05-31T21:35:44.345 INFO:tasks.rgw_multisite_tests: }, 2023-05-31T21:35:44.346 INFO:tasks.rgw_multisite_tests: { 2023-05-31T21:35:44.346 INFO:tasks.rgw_multisite_tests: "key": 3, 2023-05-31T21:35:44.346 INFO:tasks.rgw_multisite_tests: "val": { 2023-05-31T21:35:44.346 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2023-05-31T21:35:44.347 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000044178", 2023-05-31T21:35:44.347 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-05-31T21:35:44.347 INFO:tasks.rgw_multisite_tests: "total_entries": 9, 2023-05-31T21:35:44.347 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-05-31T21:35:44.347 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-05-31T21:27:37.215905Z" 2023-05-31T21:35:44.348 INFO:tasks.rgw_multisite_tests: } 2023-05-31T21:35:44.348 INFO:tasks.rgw_multisite_tests: } 2023-05-31T21:35:44.348 INFO:tasks.rgw_multisite_tests: ] 2023-05-31T21:35:44.348 INFO:tasks.rgw_multisite_tests: }, 2023-05-31T21:35:44.349 INFO:tasks.rgw_multisite_tests: "full_sync": { 2023-05-31T21:35:44.349 INFO:tasks.rgw_multisite_tests: "total": 64, 2023-05-31T21:35:44.349 INFO:tasks.rgw_multisite_tests: "complete": 64 2023-05-31T21:35:44.349 INFO:tasks.rgw_multisite_tests: }, 2023-05-31T21:35:44.350 INFO:tasks.rgw_multisite_tests: "current_time": "2023-05-31T21:28:27Z" 2023-05-31T21:35:44.350 INFO:tasks.rgw_multisite_tests:} 2023-05-31T21:35:44.350 INFO:tasks.rgw_multisite_tests: 2023-05-31T21:35:44.350 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000045235', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-05-31T21:25:53.171643Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000031640', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-05-31T21:27:37.214586Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000051391', 'next_step_marker': '', 'total_entries': 55, 'pos': 0, 'timestamp': '2023-05-31T21:25:53.141700Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000044178', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2023-05-31T21:27:37.215905Z'}}] 2023-05-31T21:35:44.350 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '00000000000000000000:00000000000000045375', 1: '00000000000000000000:00000000000000031780', 2: '00000000000000000000:00000000000000051671', 3: '00000000000000000000:00000000000000044178'} 2023-05-31T21:35:44.351 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '00000000000000000000:00000000000000045235', 1: '00000000000000000000:00000000000000031640', 2: '00000000000000000000:00000000000000051391', 3: '00000000000000000000:00000000000000044178'} 2023-05-31T21:35:44.351 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: data of zone a2 behind zone a1: shard=0 master=00000000000000000000:00000000000000045375 target=00000000000000000000:00000000000000045235, shard=1 master=00000000000000000000:00000000000000031780 target=00000000000000000000:00000000000000031640, shard=2 master=00000000000000000000:00000000000000051671 target=00000000000000000000:00000000000000051391 2023-05-31T21:35:44.351 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-31T21:35:44.351 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-31T21:35:44.352 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-05-31T21:35:44.352 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: ignoring --setuser ceph since I am not root 2023-05-31T21:35:44.352 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-05-31T21:35:44.352 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2023-05-31T21:35:44.353 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "sync_status": { 2023-05-31T21:35:44.353 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "info": { 2023-05-31T21:35:44.353 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "status": "sync", 2023-05-31T21:35:44.353 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "num_shards": 4, 2023-05-31T21:35:44.354 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "instance_id": 452136274422638824 2023-05-31T21:35:44.354 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2023-05-31T21:35:44.354 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "markers": [ 2023-05-31T21:35:44.354 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2023-05-31T21:35:44.354 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "key": 0, 2023-05-31T21:35:44.355 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "val": { 2023-05-31T21:35:44.355 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "status": "incremental-sync", 2023-05-31T21:35:44.355 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "marker": "00000000000000000000:00000000000000045235", 2023-05-31T21:35:44.355 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "next_step_marker": "", 2023-05-31T21:35:44.356 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "total_entries": 0, 2023-05-31T21:35:44.356 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "pos": 0, 2023-05-31T21:35:44.356 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "timestamp": "2023-05-31T21:25:53.171643Z" 2023-05-31T21:35:44.356 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2023-05-31T21:35:44.356 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2023-05-31T21:35:44.357 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2023-05-31T21:35:44.357 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "key": 1, 2023-05-31T21:35:44.357 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "val": { 2023-05-31T21:35:44.357 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "status": "incremental-sync", 2023-05-31T21:35:44.358 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "marker": "00000000000000000000:00000000000000031640", 2023-05-31T21:35:44.358 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "next_step_marker": "", 2023-05-31T21:35:44.358 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "total_entries": 0, 2023-05-31T21:35:44.358 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "pos": 0, 2023-05-31T21:35:44.359 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "timestamp": "2023-05-31T21:27:37.214586Z" 2023-05-31T21:35:44.359 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2023-05-31T21:35:44.359 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2023-05-31T21:35:44.359 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2023-05-31T21:35:44.359 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "key": 2, 2023-05-31T21:35:44.360 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "val": { 2023-05-31T21:35:44.360 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "status": "incremental-sync", 2023-05-31T21:35:44.360 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "marker": "00000000000000000000:00000000000000051391", 2023-05-31T21:35:44.360 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "next_step_marker": "", 2023-05-31T21:35:44.361 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "total_entries": 55, 2023-05-31T21:35:44.361 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "pos": 0, 2023-05-31T21:35:44.361 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "timestamp": "2023-05-31T21:25:53.141700Z" 2023-05-31T21:35:44.361 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2023-05-31T21:35:44.362 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2023-05-31T21:35:44.362 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2023-05-31T21:35:44.362 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "key": 3, 2023-05-31T21:35:44.362 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "val": { 2023-05-31T21:35:44.362 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "status": "incremental-sync", 2023-05-31T21:35:44.363 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "marker": "00000000000000000000:00000000000000044178", 2023-05-31T21:35:44.363 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "next_step_marker": "", 2023-05-31T21:35:44.363 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "total_entries": 9, 2023-05-31T21:35:44.363 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "pos": 0, 2023-05-31T21:35:44.364 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "timestamp": "2023-05-31T21:27:37.215905Z" 2023-05-31T21:35:44.364 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2023-05-31T21:35:44.364 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2023-05-31T21:35:44.364 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: ] 2023-05-31T21:35:44.365 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2023-05-31T21:35:44.365 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "full_sync": { 2023-05-31T21:35:44.365 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "total": 64, 2023-05-31T21:35:44.365 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "complete": 64 2023-05-31T21:35:44.366 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2023-05-31T21:35:44.366 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "current_time": "2023-05-31T21:28:32Z" 2023-05-31T21:35:44.366 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2023-05-31T21:35:44.366 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current data sync status={ 2023-05-31T21:35:44.367 INFO:tasks.rgw_multisite_tests: "sync_status": { 2023-05-31T21:35:44.367 INFO:tasks.rgw_multisite_tests: "info": { 2023-05-31T21:35:44.367 INFO:tasks.rgw_multisite_tests: "status": "sync", 2023-05-31T21:35:44.367 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2023-05-31T21:35:44.367 INFO:tasks.rgw_multisite_tests: "instance_id": 452136274422638824 2023-05-31T21:35:44.368 INFO:tasks.rgw_multisite_tests: }, 2023-05-31T21:35:44.368 INFO:tasks.rgw_multisite_tests: "markers": [ 2023-05-31T21:35:44.368 INFO:tasks.rgw_multisite_tests: { 2023-05-31T21:35:44.368 INFO:tasks.rgw_multisite_tests: "key": 0, 2023-05-31T21:35:44.369 INFO:tasks.rgw_multisite_tests: "val": { 2023-05-31T21:35:44.369 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2023-05-31T21:35:44.369 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000045235", 2023-05-31T21:35:44.369 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-05-31T21:35:44.370 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-05-31T21:35:44.370 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-05-31T21:35:44.370 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-05-31T21:25:53.171643Z" 2023-05-31T21:35:44.370 INFO:tasks.rgw_multisite_tests: } 2023-05-31T21:35:44.370 INFO:tasks.rgw_multisite_tests: }, 2023-05-31T21:35:44.371 INFO:tasks.rgw_multisite_tests: { 2023-05-31T21:35:44.371 INFO:tasks.rgw_multisite_tests: "key": 1, 2023-05-31T21:35:44.371 INFO:tasks.rgw_multisite_tests: "val": { 2023-05-31T21:35:44.371 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2023-05-31T21:35:44.372 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000031640", 2023-05-31T21:35:44.372 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-05-31T21:35:44.372 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-05-31T21:35:44.372 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-05-31T21:35:44.373 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-05-31T21:27:37.214586Z" 2023-05-31T21:35:44.373 INFO:tasks.rgw_multisite_tests: } 2023-05-31T21:35:44.373 INFO:tasks.rgw_multisite_tests: }, 2023-05-31T21:35:44.373 INFO:tasks.rgw_multisite_tests: { 2023-05-31T21:35:44.373 INFO:tasks.rgw_multisite_tests: "key": 2, 2023-05-31T21:35:44.374 INFO:tasks.rgw_multisite_tests: "val": { 2023-05-31T21:35:44.374 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2023-05-31T21:35:44.374 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000051391", 2023-05-31T21:35:44.374 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-05-31T21:35:44.375 INFO:tasks.rgw_multisite_tests: "total_entries": 55, 2023-05-31T21:35:44.375 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-05-31T21:35:44.375 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-05-31T21:25:53.141700Z" 2023-05-31T21:35:44.375 INFO:tasks.rgw_multisite_tests: } 2023-05-31T21:35:44.376 INFO:tasks.rgw_multisite_tests: }, 2023-05-31T21:35:44.376 INFO:tasks.rgw_multisite_tests: { 2023-05-31T21:35:44.376 INFO:tasks.rgw_multisite_tests: "key": 3, 2023-05-31T21:35:44.376 INFO:tasks.rgw_multisite_tests: "val": { 2023-05-31T21:35:44.377 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2023-05-31T21:35:44.377 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000044178", 2023-05-31T21:35:44.377 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-05-31T21:35:44.377 INFO:tasks.rgw_multisite_tests: "total_entries": 9, 2023-05-31T21:35:44.377 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-05-31T21:35:44.378 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-05-31T21:27:37.215905Z" 2023-05-31T21:35:44.378 INFO:tasks.rgw_multisite_tests: } 2023-05-31T21:35:44.378 INFO:tasks.rgw_multisite_tests: } 2023-05-31T21:35:44.378 INFO:tasks.rgw_multisite_tests: ] 2023-05-31T21:35:44.379 INFO:tasks.rgw_multisite_tests: }, 2023-05-31T21:35:44.379 INFO:tasks.rgw_multisite_tests: "full_sync": { 2023-05-31T21:35:44.379 INFO:tasks.rgw_multisite_tests: "total": 64, 2023-05-31T21:35:44.379 INFO:tasks.rgw_multisite_tests: "complete": 64 2023-05-31T21:35:44.380 INFO:tasks.rgw_multisite_tests: }, 2023-05-31T21:35:44.380 INFO:tasks.rgw_multisite_tests: "current_time": "2023-05-31T21:28:32Z" 2023-05-31T21:35:44.380 INFO:tasks.rgw_multisite_tests:} 2023-05-31T21:35:44.380 INFO:tasks.rgw_multisite_tests: 2023-05-31T21:35:44.380 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000045235', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-05-31T21:25:53.171643Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000031640', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-05-31T21:27:37.214586Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000051391', 'next_step_marker': '', 'total_entries': 55, 'pos': 0, 'timestamp': '2023-05-31T21:25:53.141700Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000044178', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2023-05-31T21:27:37.215905Z'}}] 2023-05-31T21:35:44.381 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '00000000000000000000:00000000000000045375', 1: '00000000000000000000:00000000000000031780', 2: '00000000000000000000:00000000000000051671', 3: '00000000000000000000:00000000000000044178'} 2023-05-31T21:35:44.381 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '00000000000000000000:00000000000000045235', 1: '00000000000000000000:00000000000000031640', 2: '00000000000000000000:00000000000000051391', 3: '00000000000000000000:00000000000000044178'} 2023-05-31T21:35:44.381 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: data of zone a2 behind zone a1: shard=0 master=00000000000000000000:00000000000000045375 target=00000000000000000000:00000000000000045235, shard=1 master=00000000000000000000:00000000000000031780 target=00000000000000000000:00000000000000031640, shard=2 master=00000000000000000000:00000000000000051671 target=00000000000000000000:00000000000000051391 2023-05-31T21:35:44.381 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-31T21:35:44.382 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-31T21:35:44.382 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-05-31T21:35:44.382 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: ignoring --setuser ceph since I am not root 2023-05-31T21:35:44.382 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-05-31T21:35:44.383 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2023-05-31T21:35:44.383 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "sync_status": { 2023-05-31T21:35:44.383 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "info": { 2023-05-31T21:35:44.383 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "status": "sync", 2023-05-31T21:35:44.383 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "num_shards": 4, 2023-05-31T21:35:44.384 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "instance_id": 452136274422638824 2023-05-31T21:35:44.384 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2023-05-31T21:35:44.384 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "markers": [ 2023-05-31T21:35:44.384 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2023-05-31T21:35:44.385 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "key": 0, 2023-05-31T21:35:44.385 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "val": { 2023-05-31T21:35:44.385 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "status": "incremental-sync", 2023-05-31T21:35:44.385 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "marker": "00000000000000000000:00000000000000045235", 2023-05-31T21:35:44.386 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "next_step_marker": "", 2023-05-31T21:35:44.386 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "total_entries": 0, 2023-05-31T21:35:44.386 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "pos": 0, 2023-05-31T21:35:44.386 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "timestamp": "2023-05-31T21:25:53.171643Z" 2023-05-31T21:35:44.387 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2023-05-31T21:35:44.387 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2023-05-31T21:35:44.387 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2023-05-31T21:35:44.387 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "key": 1, 2023-05-31T21:35:44.387 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "val": { 2023-05-31T21:35:44.388 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "status": "incremental-sync", 2023-05-31T21:35:44.388 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "marker": "00000000000000000000:00000000000000031640", 2023-05-31T21:35:44.388 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "next_step_marker": "", 2023-05-31T21:35:44.388 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "total_entries": 0, 2023-05-31T21:35:44.389 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "pos": 0, 2023-05-31T21:35:44.389 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "timestamp": "2023-05-31T21:27:37.214586Z" 2023-05-31T21:35:44.389 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2023-05-31T21:35:44.389 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2023-05-31T21:35:44.390 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2023-05-31T21:35:44.390 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "key": 2, 2023-05-31T21:35:44.390 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "val": { 2023-05-31T21:35:44.390 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "status": "incremental-sync", 2023-05-31T21:35:44.390 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "marker": "00000000000000000000:00000000000000051391", 2023-05-31T21:35:44.391 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "next_step_marker": "", 2023-05-31T21:35:44.391 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "total_entries": 55, 2023-05-31T21:35:44.391 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "pos": 0, 2023-05-31T21:35:44.391 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "timestamp": "2023-05-31T21:25:53.141700Z" 2023-05-31T21:35:44.392 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2023-05-31T21:35:44.392 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2023-05-31T21:35:44.392 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2023-05-31T21:35:44.392 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "key": 3, 2023-05-31T21:35:44.393 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "val": { 2023-05-31T21:35:44.393 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "status": "incremental-sync", 2023-05-31T21:35:44.393 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "marker": "00000000000000000000:00000000000000044178", 2023-05-31T21:35:44.393 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "next_step_marker": "", 2023-05-31T21:35:44.393 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "total_entries": 9, 2023-05-31T21:35:44.394 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "pos": 0, 2023-05-31T21:35:44.394 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "timestamp": "2023-05-31T21:27:37.215905Z" 2023-05-31T21:35:44.394 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2023-05-31T21:35:44.394 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2023-05-31T21:35:44.395 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: ] 2023-05-31T21:35:44.395 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2023-05-31T21:35:44.395 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "full_sync": { 2023-05-31T21:35:44.395 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "total": 64, 2023-05-31T21:35:44.396 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "complete": 64 2023-05-31T21:35:44.396 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2023-05-31T21:35:44.396 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "current_time": "2023-05-31T21:28:37Z" 2023-05-31T21:35:44.396 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2023-05-31T21:35:44.396 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current data sync status={ 2023-05-31T21:35:44.397 INFO:tasks.rgw_multisite_tests: "sync_status": { 2023-05-31T21:35:44.397 INFO:tasks.rgw_multisite_tests: "info": { 2023-05-31T21:35:44.397 INFO:tasks.rgw_multisite_tests: "status": "sync", 2023-05-31T21:35:44.397 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2023-05-31T21:35:44.398 INFO:tasks.rgw_multisite_tests: "instance_id": 452136274422638824 2023-05-31T21:35:44.398 INFO:tasks.rgw_multisite_tests: }, 2023-05-31T21:35:44.398 INFO:tasks.rgw_multisite_tests: "markers": [ 2023-05-31T21:35:44.398 INFO:tasks.rgw_multisite_tests: { 2023-05-31T21:35:44.399 INFO:tasks.rgw_multisite_tests: "key": 0, 2023-05-31T21:35:44.399 INFO:tasks.rgw_multisite_tests: "val": { 2023-05-31T21:35:44.399 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2023-05-31T21:35:44.399 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000045235", 2023-05-31T21:35:44.399 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-05-31T21:35:44.400 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-05-31T21:35:44.400 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-05-31T21:35:44.400 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-05-31T21:25:53.171643Z" 2023-05-31T21:35:44.400 INFO:tasks.rgw_multisite_tests: } 2023-05-31T21:35:44.401 INFO:tasks.rgw_multisite_tests: }, 2023-05-31T21:35:44.401 INFO:tasks.rgw_multisite_tests: { 2023-05-31T21:35:44.401 INFO:tasks.rgw_multisite_tests: "key": 1, 2023-05-31T21:35:44.401 INFO:tasks.rgw_multisite_tests: "val": { 2023-05-31T21:35:44.402 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2023-05-31T21:35:44.402 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000031640", 2023-05-31T21:35:44.402 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-05-31T21:35:44.402 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-05-31T21:35:44.402 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-05-31T21:35:44.403 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-05-31T21:27:37.214586Z" 2023-05-31T21:35:44.403 INFO:tasks.rgw_multisite_tests: } 2023-05-31T21:35:44.403 INFO:tasks.rgw_multisite_tests: }, 2023-05-31T21:35:44.403 INFO:tasks.rgw_multisite_tests: { 2023-05-31T21:35:44.404 INFO:tasks.rgw_multisite_tests: "key": 2, 2023-05-31T21:35:44.404 INFO:tasks.rgw_multisite_tests: "val": { 2023-05-31T21:35:44.404 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2023-05-31T21:35:44.404 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000051391", 2023-05-31T21:35:44.405 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-05-31T21:35:44.405 INFO:tasks.rgw_multisite_tests: "total_entries": 55, 2023-05-31T21:35:44.405 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-05-31T21:35:44.405 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-05-31T21:25:53.141700Z" 2023-05-31T21:35:44.405 INFO:tasks.rgw_multisite_tests: } 2023-05-31T21:35:44.406 INFO:tasks.rgw_multisite_tests: }, 2023-05-31T21:35:44.406 INFO:tasks.rgw_multisite_tests: { 2023-05-31T21:35:44.406 INFO:tasks.rgw_multisite_tests: "key": 3, 2023-05-31T21:35:44.406 INFO:tasks.rgw_multisite_tests: "val": { 2023-05-31T21:35:44.407 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2023-05-31T21:35:44.407 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000044178", 2023-05-31T21:35:44.407 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-05-31T21:35:44.407 INFO:tasks.rgw_multisite_tests: "total_entries": 9, 2023-05-31T21:35:44.407 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-05-31T21:35:44.408 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-05-31T21:27:37.215905Z" 2023-05-31T21:35:44.408 INFO:tasks.rgw_multisite_tests: } 2023-05-31T21:35:44.408 INFO:tasks.rgw_multisite_tests: } 2023-05-31T21:35:44.408 INFO:tasks.rgw_multisite_tests: ] 2023-05-31T21:35:44.409 INFO:tasks.rgw_multisite_tests: }, 2023-05-31T21:35:44.409 INFO:tasks.rgw_multisite_tests: "full_sync": { 2023-05-31T21:35:44.409 INFO:tasks.rgw_multisite_tests: "total": 64, 2023-05-31T21:35:44.409 INFO:tasks.rgw_multisite_tests: "complete": 64 2023-05-31T21:35:44.410 INFO:tasks.rgw_multisite_tests: }, 2023-05-31T21:35:44.410 INFO:tasks.rgw_multisite_tests: "current_time": "2023-05-31T21:28:37Z" 2023-05-31T21:35:44.410 INFO:tasks.rgw_multisite_tests:} 2023-05-31T21:35:44.410 INFO:tasks.rgw_multisite_tests: 2023-05-31T21:35:44.410 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000045235', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-05-31T21:25:53.171643Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000031640', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-05-31T21:27:37.214586Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000051391', 'next_step_marker': '', 'total_entries': 55, 'pos': 0, 'timestamp': '2023-05-31T21:25:53.141700Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000044178', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2023-05-31T21:27:37.215905Z'}}] 2023-05-31T21:35:44.411 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '00000000000000000000:00000000000000045375', 1: '00000000000000000000:00000000000000031780', 2: '00000000000000000000:00000000000000051671', 3: '00000000000000000000:00000000000000044178'} 2023-05-31T21:35:44.411 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '00000000000000000000:00000000000000045235', 1: '00000000000000000000:00000000000000031640', 2: '00000000000000000000:00000000000000051391', 3: '00000000000000000000:00000000000000044178'} 2023-05-31T21:35:44.411 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: data of zone a2 behind zone a1: shard=0 master=00000000000000000000:00000000000000045375 target=00000000000000000000:00000000000000045235, shard=1 master=00000000000000000000:00000000000000031780 target=00000000000000000000:00000000000000031640, shard=2 master=00000000000000000000:00000000000000051671 target=00000000000000000000:00000000000000051391 2023-05-31T21:35:44.411 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-31T21:35:44.412 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-31T21:35:44.412 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-05-31T21:35:44.412 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: ignoring --setuser ceph since I am not root 2023-05-31T21:35:44.412 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-05-31T21:35:44.413 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2023-05-31T21:35:44.413 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "sync_status": { 2023-05-31T21:35:44.413 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "info": { 2023-05-31T21:35:44.413 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "status": "sync", 2023-05-31T21:35:44.414 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "num_shards": 4, 2023-05-31T21:35:44.414 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "instance_id": 452136274422638824 2023-05-31T21:35:44.414 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2023-05-31T21:35:44.414 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "markers": [ 2023-05-31T21:35:44.414 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2023-05-31T21:35:44.415 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "key": 0, 2023-05-31T21:35:44.415 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "val": { 2023-05-31T21:35:44.415 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "status": "incremental-sync", 2023-05-31T21:35:44.415 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "marker": "00000000000000000000:00000000000000045235", 2023-05-31T21:35:44.416 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "next_step_marker": "", 2023-05-31T21:35:44.416 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "total_entries": 0, 2023-05-31T21:35:44.416 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "pos": 0, 2023-05-31T21:35:44.416 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "timestamp": "2023-05-31T21:25:53.171643Z" 2023-05-31T21:35:44.417 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2023-05-31T21:35:44.417 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2023-05-31T21:35:44.417 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2023-05-31T21:35:44.417 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "key": 1, 2023-05-31T21:35:44.418 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "val": { 2023-05-31T21:35:44.418 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "status": "incremental-sync", 2023-05-31T21:35:44.418 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "marker": "00000000000000000000:00000000000000031640", 2023-05-31T21:35:44.418 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "next_step_marker": "", 2023-05-31T21:35:44.418 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "total_entries": 0, 2023-05-31T21:35:44.419 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "pos": 0, 2023-05-31T21:35:44.419 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "timestamp": "2023-05-31T21:27:37.214586Z" 2023-05-31T21:35:44.419 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2023-05-31T21:35:44.419 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2023-05-31T21:35:44.420 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2023-05-31T21:35:44.420 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "key": 2, 2023-05-31T21:35:44.420 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "val": { 2023-05-31T21:35:44.420 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "status": "incremental-sync", 2023-05-31T21:35:44.421 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "marker": "00000000000000000000:00000000000000051391", 2023-05-31T21:35:44.421 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "next_step_marker": "", 2023-05-31T21:35:44.421 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "total_entries": 55, 2023-05-31T21:35:44.421 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "pos": 0, 2023-05-31T21:35:44.421 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "timestamp": "2023-05-31T21:25:53.141700Z" 2023-05-31T21:35:44.422 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2023-05-31T21:35:44.422 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2023-05-31T21:35:44.422 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2023-05-31T21:35:44.422 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "key": 3, 2023-05-31T21:35:44.423 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "val": { 2023-05-31T21:35:44.423 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "status": "incremental-sync", 2023-05-31T21:35:44.423 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "marker": "00000000000000000000:00000000000000044178", 2023-05-31T21:35:44.423 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "next_step_marker": "", 2023-05-31T21:35:44.424 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "total_entries": 9, 2023-05-31T21:35:44.424 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "pos": 0, 2023-05-31T21:35:44.424 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "timestamp": "2023-05-31T21:27:37.215905Z" 2023-05-31T21:35:44.424 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2023-05-31T21:35:44.425 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2023-05-31T21:35:44.425 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: ] 2023-05-31T21:35:44.425 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2023-05-31T21:35:44.425 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "full_sync": { 2023-05-31T21:35:44.425 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "total": 64, 2023-05-31T21:35:44.426 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "complete": 64 2023-05-31T21:35:44.426 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2023-05-31T21:35:44.426 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "current_time": "2023-05-31T21:28:43Z" 2023-05-31T21:35:44.426 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2023-05-31T21:35:44.427 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current data sync status={ 2023-05-31T21:35:44.427 INFO:tasks.rgw_multisite_tests: "sync_status": { 2023-05-31T21:35:44.427 INFO:tasks.rgw_multisite_tests: "info": { 2023-05-31T21:35:44.427 INFO:tasks.rgw_multisite_tests: "status": "sync", 2023-05-31T21:35:44.428 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2023-05-31T21:35:44.428 INFO:tasks.rgw_multisite_tests: "instance_id": 452136274422638824 2023-05-31T21:35:44.428 INFO:tasks.rgw_multisite_tests: }, 2023-05-31T21:35:44.428 INFO:tasks.rgw_multisite_tests: "markers": [ 2023-05-31T21:35:44.428 INFO:tasks.rgw_multisite_tests: { 2023-05-31T21:35:44.429 INFO:tasks.rgw_multisite_tests: "key": 0, 2023-05-31T21:35:44.429 INFO:tasks.rgw_multisite_tests: "val": { 2023-05-31T21:35:44.429 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2023-05-31T21:35:44.429 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000045235", 2023-05-31T21:35:44.430 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-05-31T21:35:44.430 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-05-31T21:35:44.430 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-05-31T21:35:44.430 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-05-31T21:25:53.171643Z" 2023-05-31T21:35:44.431 INFO:tasks.rgw_multisite_tests: } 2023-05-31T21:35:44.431 INFO:tasks.rgw_multisite_tests: }, 2023-05-31T21:35:44.431 INFO:tasks.rgw_multisite_tests: { 2023-05-31T21:35:44.431 INFO:tasks.rgw_multisite_tests: "key": 1, 2023-05-31T21:35:44.431 INFO:tasks.rgw_multisite_tests: "val": { 2023-05-31T21:35:44.432 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2023-05-31T21:35:44.432 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000031640", 2023-05-31T21:35:44.432 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-05-31T21:35:44.432 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-05-31T21:35:44.433 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-05-31T21:35:44.433 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-05-31T21:27:37.214586Z" 2023-05-31T21:35:44.433 INFO:tasks.rgw_multisite_tests: } 2023-05-31T21:35:44.433 INFO:tasks.rgw_multisite_tests: }, 2023-05-31T21:35:44.434 INFO:tasks.rgw_multisite_tests: { 2023-05-31T21:35:44.434 INFO:tasks.rgw_multisite_tests: "key": 2, 2023-05-31T21:35:44.434 INFO:tasks.rgw_multisite_tests: "val": { 2023-05-31T21:35:44.434 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2023-05-31T21:35:44.434 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000051391", 2023-05-31T21:35:44.435 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-05-31T21:35:44.435 INFO:tasks.rgw_multisite_tests: "total_entries": 55, 2023-05-31T21:35:44.435 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-05-31T21:35:44.435 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-05-31T21:25:53.141700Z" 2023-05-31T21:35:44.436 INFO:tasks.rgw_multisite_tests: } 2023-05-31T21:35:44.436 INFO:tasks.rgw_multisite_tests: }, 2023-05-31T21:35:44.436 INFO:tasks.rgw_multisite_tests: { 2023-05-31T21:35:44.436 INFO:tasks.rgw_multisite_tests: "key": 3, 2023-05-31T21:35:44.437 INFO:tasks.rgw_multisite_tests: "val": { 2023-05-31T21:35:44.437 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2023-05-31T21:35:44.437 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000044178", 2023-05-31T21:35:44.437 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-05-31T21:35:44.437 INFO:tasks.rgw_multisite_tests: "total_entries": 9, 2023-05-31T21:35:44.438 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-05-31T21:35:44.438 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-05-31T21:27:37.215905Z" 2023-05-31T21:35:44.438 INFO:tasks.rgw_multisite_tests: } 2023-05-31T21:35:44.438 INFO:tasks.rgw_multisite_tests: } 2023-05-31T21:35:44.439 INFO:tasks.rgw_multisite_tests: ] 2023-05-31T21:35:44.439 INFO:tasks.rgw_multisite_tests: }, 2023-05-31T21:35:44.439 INFO:tasks.rgw_multisite_tests: "full_sync": { 2023-05-31T21:35:44.439 INFO:tasks.rgw_multisite_tests: "total": 64, 2023-05-31T21:35:44.440 INFO:tasks.rgw_multisite_tests: "complete": 64 2023-05-31T21:35:44.440 INFO:tasks.rgw_multisite_tests: }, 2023-05-31T21:35:44.440 INFO:tasks.rgw_multisite_tests: "current_time": "2023-05-31T21:28:43Z" 2023-05-31T21:35:44.440 INFO:tasks.rgw_multisite_tests:} 2023-05-31T21:35:44.440 INFO:tasks.rgw_multisite_tests: 2023-05-31T21:35:44.441 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000045235', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-05-31T21:25:53.171643Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000031640', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-05-31T21:27:37.214586Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000051391', 'next_step_marker': '', 'total_entries': 55, 'pos': 0, 'timestamp': '2023-05-31T21:25:53.141700Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000044178', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2023-05-31T21:27:37.215905Z'}}] 2023-05-31T21:35:44.441 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '00000000000000000000:00000000000000045375', 1: '00000000000000000000:00000000000000031780', 2: '00000000000000000000:00000000000000051671', 3: '00000000000000000000:00000000000000044178'} 2023-05-31T21:35:44.441 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '00000000000000000000:00000000000000045235', 1: '00000000000000000000:00000000000000031640', 2: '00000000000000000000:00000000000000051391', 3: '00000000000000000000:00000000000000044178'} 2023-05-31T21:35:44.441 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: data of zone a2 behind zone a1: shard=0 master=00000000000000000000:00000000000000045375 target=00000000000000000000:00000000000000045235, shard=1 master=00000000000000000000:00000000000000031780 target=00000000000000000000:00000000000000031640, shard=2 master=00000000000000000000:00000000000000051671 target=00000000000000000000:00000000000000051391 2023-05-31T21:35:44.442 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-31T21:35:44.442 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-31T21:35:44.442 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-05-31T21:35:44.442 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: ignoring --setuser ceph since I am not root 2023-05-31T21:35:44.443 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-05-31T21:35:44.443 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2023-05-31T21:35:44.443 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "sync_status": { 2023-05-31T21:35:44.443 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "info": { 2023-05-31T21:35:44.443 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "status": "sync", 2023-05-31T21:35:44.444 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "num_shards": 4, 2023-05-31T21:35:44.444 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "instance_id": 452136274422638824 2023-05-31T21:35:44.444 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2023-05-31T21:35:44.444 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "markers": [ 2023-05-31T21:35:44.445 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2023-05-31T21:35:44.445 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "key": 0, 2023-05-31T21:35:44.445 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "val": { 2023-05-31T21:35:44.445 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "status": "incremental-sync", 2023-05-31T21:35:44.446 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "marker": "00000000000000000000:00000000000000045235", 2023-05-31T21:35:44.446 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "next_step_marker": "", 2023-05-31T21:35:44.446 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "total_entries": 0, 2023-05-31T21:35:44.446 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "pos": 0, 2023-05-31T21:35:44.447 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "timestamp": "2023-05-31T21:25:53.171643Z" 2023-05-31T21:35:44.447 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2023-05-31T21:35:44.447 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2023-05-31T21:35:44.447 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2023-05-31T21:35:44.447 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "key": 1, 2023-05-31T21:35:44.448 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "val": { 2023-05-31T21:35:44.448 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "status": "incremental-sync", 2023-05-31T21:35:44.448 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "marker": "00000000000000000000:00000000000000031640", 2023-05-31T21:35:44.448 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "next_step_marker": "", 2023-05-31T21:35:44.449 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "total_entries": 0, 2023-05-31T21:35:44.449 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "pos": 0, 2023-05-31T21:35:44.449 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "timestamp": "2023-05-31T21:27:37.214586Z" 2023-05-31T21:35:44.449 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2023-05-31T21:35:44.450 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2023-05-31T21:35:44.450 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2023-05-31T21:35:44.450 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "key": 2, 2023-05-31T21:35:44.450 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "val": { 2023-05-31T21:35:44.450 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "status": "incremental-sync", 2023-05-31T21:35:44.451 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "marker": "00000000000000000000:00000000000000051391", 2023-05-31T21:35:44.451 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "next_step_marker": "", 2023-05-31T21:35:44.451 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "total_entries": 55, 2023-05-31T21:35:44.451 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "pos": 0, 2023-05-31T21:35:44.452 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "timestamp": "2023-05-31T21:25:53.141700Z" 2023-05-31T21:35:44.452 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2023-05-31T21:35:44.452 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2023-05-31T21:35:44.452 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2023-05-31T21:35:44.453 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "key": 3, 2023-05-31T21:35:44.453 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "val": { 2023-05-31T21:35:44.453 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "status": "incremental-sync", 2023-05-31T21:35:44.453 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "marker": "00000000000000000000:00000000000000044178", 2023-05-31T21:35:44.453 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "next_step_marker": "", 2023-05-31T21:35:44.454 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "total_entries": 9, 2023-05-31T21:35:44.454 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "pos": 0, 2023-05-31T21:35:44.454 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "timestamp": "2023-05-31T21:27:37.215905Z" 2023-05-31T21:35:44.454 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2023-05-31T21:35:44.455 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2023-05-31T21:35:44.455 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: ] 2023-05-31T21:35:44.455 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2023-05-31T21:35:44.455 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "full_sync": { 2023-05-31T21:35:44.456 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "total": 64, 2023-05-31T21:35:44.456 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "complete": 64 2023-05-31T21:35:44.456 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2023-05-31T21:35:44.456 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "current_time": "2023-05-31T21:28:48Z" 2023-05-31T21:35:44.456 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2023-05-31T21:35:44.457 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current data sync status={ 2023-05-31T21:35:44.457 INFO:tasks.rgw_multisite_tests: "sync_status": { 2023-05-31T21:35:44.457 INFO:tasks.rgw_multisite_tests: "info": { 2023-05-31T21:35:44.457 INFO:tasks.rgw_multisite_tests: "status": "sync", 2023-05-31T21:35:44.458 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2023-05-31T21:35:44.458 INFO:tasks.rgw_multisite_tests: "instance_id": 452136274422638824 2023-05-31T21:35:44.458 INFO:tasks.rgw_multisite_tests: }, 2023-05-31T21:35:44.458 INFO:tasks.rgw_multisite_tests: "markers": [ 2023-05-31T21:35:44.459 INFO:tasks.rgw_multisite_tests: { 2023-05-31T21:35:44.459 INFO:tasks.rgw_multisite_tests: "key": 0, 2023-05-31T21:35:44.459 INFO:tasks.rgw_multisite_tests: "val": { 2023-05-31T21:35:44.459 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2023-05-31T21:35:44.459 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000045235", 2023-05-31T21:35:44.460 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-05-31T21:35:44.460 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-05-31T21:35:44.460 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-05-31T21:35:44.460 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-05-31T21:25:53.171643Z" 2023-05-31T21:35:44.461 INFO:tasks.rgw_multisite_tests: } 2023-05-31T21:35:44.461 INFO:tasks.rgw_multisite_tests: }, 2023-05-31T21:35:44.461 INFO:tasks.rgw_multisite_tests: { 2023-05-31T21:35:44.461 INFO:tasks.rgw_multisite_tests: "key": 1, 2023-05-31T21:35:44.462 INFO:tasks.rgw_multisite_tests: "val": { 2023-05-31T21:35:44.462 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2023-05-31T21:35:44.462 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000031640", 2023-05-31T21:35:44.462 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-05-31T21:35:44.462 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-05-31T21:35:44.463 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-05-31T21:35:44.463 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-05-31T21:27:37.214586Z" 2023-05-31T21:35:44.463 INFO:tasks.rgw_multisite_tests: } 2023-05-31T21:35:44.463 INFO:tasks.rgw_multisite_tests: }, 2023-05-31T21:35:44.464 INFO:tasks.rgw_multisite_tests: { 2023-05-31T21:35:44.464 INFO:tasks.rgw_multisite_tests: "key": 2, 2023-05-31T21:35:44.464 INFO:tasks.rgw_multisite_tests: "val": { 2023-05-31T21:35:44.464 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2023-05-31T21:35:44.465 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000051391", 2023-05-31T21:35:44.465 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-05-31T21:35:44.465 INFO:tasks.rgw_multisite_tests: "total_entries": 55, 2023-05-31T21:35:44.465 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-05-31T21:35:44.466 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-05-31T21:25:53.141700Z" 2023-05-31T21:35:44.466 INFO:tasks.rgw_multisite_tests: } 2023-05-31T21:35:44.466 INFO:tasks.rgw_multisite_tests: }, 2023-05-31T21:35:44.466 INFO:tasks.rgw_multisite_tests: { 2023-05-31T21:35:44.466 INFO:tasks.rgw_multisite_tests: "key": 3, 2023-05-31T21:35:44.467 INFO:tasks.rgw_multisite_tests: "val": { 2023-05-31T21:35:44.467 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2023-05-31T21:35:44.467 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000044178", 2023-05-31T21:35:44.467 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-05-31T21:35:44.468 INFO:tasks.rgw_multisite_tests: "total_entries": 9, 2023-05-31T21:35:44.468 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-05-31T21:35:44.468 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-05-31T21:27:37.215905Z" 2023-05-31T21:35:44.468 INFO:tasks.rgw_multisite_tests: } 2023-05-31T21:35:44.469 INFO:tasks.rgw_multisite_tests: } 2023-05-31T21:35:44.469 INFO:tasks.rgw_multisite_tests: ] 2023-05-31T21:35:44.469 INFO:tasks.rgw_multisite_tests: }, 2023-05-31T21:35:44.469 INFO:tasks.rgw_multisite_tests: "full_sync": { 2023-05-31T21:35:44.469 INFO:tasks.rgw_multisite_tests: "total": 64, 2023-05-31T21:35:44.470 INFO:tasks.rgw_multisite_tests: "complete": 64 2023-05-31T21:35:44.470 INFO:tasks.rgw_multisite_tests: }, 2023-05-31T21:35:44.470 INFO:tasks.rgw_multisite_tests: "current_time": "2023-05-31T21:28:48Z" 2023-05-31T21:35:44.470 INFO:tasks.rgw_multisite_tests:} 2023-05-31T21:35:44.471 INFO:tasks.rgw_multisite_tests: 2023-05-31T21:35:44.471 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000045235', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-05-31T21:25:53.171643Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000031640', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-05-31T21:27:37.214586Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000051391', 'next_step_marker': '', 'total_entries': 55, 'pos': 0, 'timestamp': '2023-05-31T21:25:53.141700Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000044178', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2023-05-31T21:27:37.215905Z'}}] 2023-05-31T21:35:44.471 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '00000000000000000000:00000000000000045375', 1: '00000000000000000000:00000000000000031780', 2: '00000000000000000000:00000000000000051671', 3: '00000000000000000000:00000000000000044178'} 2023-05-31T21:35:44.471 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '00000000000000000000:00000000000000045235', 1: '00000000000000000000:00000000000000031640', 2: '00000000000000000000:00000000000000051391', 3: '00000000000000000000:00000000000000044178'} 2023-05-31T21:35:44.472 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: data of zone a2 behind zone a1: shard=0 master=00000000000000000000:00000000000000045375 target=00000000000000000000:00000000000000045235, shard=1 master=00000000000000000000:00000000000000031780 target=00000000000000000000:00000000000000031640, shard=2 master=00000000000000000000:00000000000000051671 target=00000000000000000000:00000000000000051391 2023-05-31T21:35:44.472 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-31T21:35:44.472 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-31T21:35:44.472 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-05-31T21:35:44.473 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: ignoring --setuser ceph since I am not root 2023-05-31T21:35:44.473 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-05-31T21:35:44.473 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2023-05-31T21:35:44.473 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "sync_status": { 2023-05-31T21:35:44.473 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "info": { 2023-05-31T21:35:44.474 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "status": "sync", 2023-05-31T21:35:44.474 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "num_shards": 4, 2023-05-31T21:35:44.474 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "instance_id": 452136274422638824 2023-05-31T21:35:44.474 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2023-05-31T21:35:44.475 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "markers": [ 2023-05-31T21:35:44.475 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2023-05-31T21:35:44.475 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "key": 0, 2023-05-31T21:35:44.475 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "val": { 2023-05-31T21:35:44.476 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "status": "incremental-sync", 2023-05-31T21:35:44.476 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "marker": "00000000000000000000:00000000000000045235", 2023-05-31T21:35:44.476 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "next_step_marker": "", 2023-05-31T21:35:44.476 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "total_entries": 0, 2023-05-31T21:35:44.477 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "pos": 0, 2023-05-31T21:35:44.477 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "timestamp": "2023-05-31T21:25:53.171643Z" 2023-05-31T21:35:44.477 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2023-05-31T21:35:44.477 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2023-05-31T21:35:44.477 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2023-05-31T21:35:44.478 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "key": 1, 2023-05-31T21:35:44.478 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "val": { 2023-05-31T21:35:44.478 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "status": "incremental-sync", 2023-05-31T21:35:44.478 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "marker": "00000000000000000000:00000000000000031640", 2023-05-31T21:35:44.479 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "next_step_marker": "", 2023-05-31T21:35:44.479 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "total_entries": 0, 2023-05-31T21:35:44.479 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "pos": 0, 2023-05-31T21:35:44.479 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "timestamp": "2023-05-31T21:27:37.214586Z" 2023-05-31T21:35:44.480 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2023-05-31T21:35:44.480 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2023-05-31T21:35:44.480 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2023-05-31T21:35:44.480 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "key": 2, 2023-05-31T21:35:44.480 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "val": { 2023-05-31T21:35:44.481 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "status": "incremental-sync", 2023-05-31T21:35:44.481 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "marker": "00000000000000000000:00000000000000051391", 2023-05-31T21:35:44.481 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "next_step_marker": "", 2023-05-31T21:35:44.481 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "total_entries": 55, 2023-05-31T21:35:44.482 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "pos": 0, 2023-05-31T21:35:44.482 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "timestamp": "2023-05-31T21:25:53.141700Z" 2023-05-31T21:35:44.482 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2023-05-31T21:35:44.482 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2023-05-31T21:35:44.483 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2023-05-31T21:35:44.483 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "key": 3, 2023-05-31T21:35:44.483 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "val": { 2023-05-31T21:35:44.483 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "status": "incremental-sync", 2023-05-31T21:35:44.483 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "marker": "00000000000000000000:00000000000000044178", 2023-05-31T21:35:44.484 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "next_step_marker": "", 2023-05-31T21:35:44.484 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "total_entries": 9, 2023-05-31T21:35:44.484 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "pos": 0, 2023-05-31T21:35:44.484 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "timestamp": "2023-05-31T21:27:37.215905Z" 2023-05-31T21:35:44.485 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2023-05-31T21:35:44.485 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2023-05-31T21:35:44.485 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: ] 2023-05-31T21:35:44.485 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2023-05-31T21:35:44.486 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "full_sync": { 2023-05-31T21:35:44.486 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "total": 64, 2023-05-31T21:35:44.486 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "complete": 64 2023-05-31T21:35:44.486 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2023-05-31T21:35:44.486 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "current_time": "2023-05-31T21:28:53Z" 2023-05-31T21:35:44.487 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2023-05-31T21:35:44.487 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current data sync status={ 2023-05-31T21:35:44.487 INFO:tasks.rgw_multisite_tests: "sync_status": { 2023-05-31T21:35:44.487 INFO:tasks.rgw_multisite_tests: "info": { 2023-05-31T21:35:44.488 INFO:tasks.rgw_multisite_tests: "status": "sync", 2023-05-31T21:35:44.488 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2023-05-31T21:35:44.488 INFO:tasks.rgw_multisite_tests: "instance_id": 452136274422638824 2023-05-31T21:35:44.488 INFO:tasks.rgw_multisite_tests: }, 2023-05-31T21:35:44.489 INFO:tasks.rgw_multisite_tests: "markers": [ 2023-05-31T21:35:44.489 INFO:tasks.rgw_multisite_tests: { 2023-05-31T21:35:44.489 INFO:tasks.rgw_multisite_tests: "key": 0, 2023-05-31T21:35:44.489 INFO:tasks.rgw_multisite_tests: "val": { 2023-05-31T21:35:44.489 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2023-05-31T21:35:44.490 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000045235", 2023-05-31T21:35:44.490 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-05-31T21:35:44.490 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-05-31T21:35:44.490 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-05-31T21:35:44.491 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-05-31T21:25:53.171643Z" 2023-05-31T21:35:44.491 INFO:tasks.rgw_multisite_tests: } 2023-05-31T21:35:44.491 INFO:tasks.rgw_multisite_tests: }, 2023-05-31T21:35:44.491 INFO:tasks.rgw_multisite_tests: { 2023-05-31T21:35:44.492 INFO:tasks.rgw_multisite_tests: "key": 1, 2023-05-31T21:35:44.492 INFO:tasks.rgw_multisite_tests: "val": { 2023-05-31T21:35:44.492 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2023-05-31T21:35:44.492 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000031640", 2023-05-31T21:35:44.492 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-05-31T21:35:44.493 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-05-31T21:35:44.493 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-05-31T21:35:44.493 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-05-31T21:27:37.214586Z" 2023-05-31T21:35:44.493 INFO:tasks.rgw_multisite_tests: } 2023-05-31T21:35:44.494 INFO:tasks.rgw_multisite_tests: }, 2023-05-31T21:35:44.494 INFO:tasks.rgw_multisite_tests: { 2023-05-31T21:35:44.494 INFO:tasks.rgw_multisite_tests: "key": 2, 2023-05-31T21:35:44.494 INFO:tasks.rgw_multisite_tests: "val": { 2023-05-31T21:35:44.495 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2023-05-31T21:35:44.495 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000051391", 2023-05-31T21:35:44.495 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-05-31T21:35:44.495 INFO:tasks.rgw_multisite_tests: "total_entries": 55, 2023-05-31T21:35:44.495 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-05-31T21:35:44.496 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-05-31T21:25:53.141700Z" 2023-05-31T21:35:44.496 INFO:tasks.rgw_multisite_tests: } 2023-05-31T21:35:44.496 INFO:tasks.rgw_multisite_tests: }, 2023-05-31T21:35:44.496 INFO:tasks.rgw_multisite_tests: { 2023-05-31T21:35:44.497 INFO:tasks.rgw_multisite_tests: "key": 3, 2023-05-31T21:35:44.497 INFO:tasks.rgw_multisite_tests: "val": { 2023-05-31T21:35:44.497 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2023-05-31T21:35:44.497 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000044178", 2023-05-31T21:35:44.498 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-05-31T21:35:44.498 INFO:tasks.rgw_multisite_tests: "total_entries": 9, 2023-05-31T21:35:44.498 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-05-31T21:35:44.498 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-05-31T21:27:37.215905Z" 2023-05-31T21:35:44.498 INFO:tasks.rgw_multisite_tests: } 2023-05-31T21:35:44.499 INFO:tasks.rgw_multisite_tests: } 2023-05-31T21:35:44.499 INFO:tasks.rgw_multisite_tests: ] 2023-05-31T21:35:44.499 INFO:tasks.rgw_multisite_tests: }, 2023-05-31T21:35:44.499 INFO:tasks.rgw_multisite_tests: "full_sync": { 2023-05-31T21:35:44.500 INFO:tasks.rgw_multisite_tests: "total": 64, 2023-05-31T21:35:44.500 INFO:tasks.rgw_multisite_tests: "complete": 64 2023-05-31T21:35:44.500 INFO:tasks.rgw_multisite_tests: }, 2023-05-31T21:35:44.500 INFO:tasks.rgw_multisite_tests: "current_time": "2023-05-31T21:28:53Z" 2023-05-31T21:35:44.501 INFO:tasks.rgw_multisite_tests:} 2023-05-31T21:35:44.501 INFO:tasks.rgw_multisite_tests: 2023-05-31T21:35:44.501 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000045235', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-05-31T21:25:53.171643Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000031640', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-05-31T21:27:37.214586Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000051391', 'next_step_marker': '', 'total_entries': 55, 'pos': 0, 'timestamp': '2023-05-31T21:25:53.141700Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000044178', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2023-05-31T21:27:37.215905Z'}}] 2023-05-31T21:35:44.501 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '00000000000000000000:00000000000000045375', 1: '00000000000000000000:00000000000000031780', 2: '00000000000000000000:00000000000000051671', 3: '00000000000000000000:00000000000000044178'} 2023-05-31T21:35:44.501 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '00000000000000000000:00000000000000045235', 1: '00000000000000000000:00000000000000031640', 2: '00000000000000000000:00000000000000051391', 3: '00000000000000000000:00000000000000044178'} 2023-05-31T21:35:44.502 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: data of zone a2 behind zone a1: shard=0 master=00000000000000000000:00000000000000045375 target=00000000000000000000:00000000000000045235, shard=1 master=00000000000000000000:00000000000000031780 target=00000000000000000000:00000000000000031640, shard=2 master=00000000000000000000:00000000000000051671 target=00000000000000000000:00000000000000051391 2023-05-31T21:35:44.502 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-31T21:35:44.502 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-31T21:35:44.502 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-05-31T21:35:44.503 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: ignoring --setuser ceph since I am not root 2023-05-31T21:35:44.503 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-05-31T21:35:44.503 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2023-05-31T21:35:44.503 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "sync_status": { 2023-05-31T21:35:44.504 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "info": { 2023-05-31T21:35:44.504 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "status": "sync", 2023-05-31T21:35:44.504 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "num_shards": 4, 2023-05-31T21:35:44.504 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "instance_id": 452136274422638824 2023-05-31T21:35:44.505 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2023-05-31T21:35:44.505 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "markers": [ 2023-05-31T21:35:44.505 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2023-05-31T21:35:44.505 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "key": 0, 2023-05-31T21:35:44.505 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "val": { 2023-05-31T21:35:44.506 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "status": "incremental-sync", 2023-05-31T21:35:44.506 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "marker": "00000000000000000000:00000000000000045235", 2023-05-31T21:35:44.506 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "next_step_marker": "", 2023-05-31T21:35:44.506 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "total_entries": 0, 2023-05-31T21:35:44.507 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "pos": 0, 2023-05-31T21:35:44.507 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "timestamp": "2023-05-31T21:25:53.171643Z" 2023-05-31T21:35:44.507 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2023-05-31T21:35:44.507 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2023-05-31T21:35:44.508 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2023-05-31T21:35:44.508 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "key": 1, 2023-05-31T21:35:44.508 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "val": { 2023-05-31T21:35:44.508 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "status": "incremental-sync", 2023-05-31T21:35:44.508 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "marker": "00000000000000000000:00000000000000031640", 2023-05-31T21:35:44.509 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "next_step_marker": "", 2023-05-31T21:35:44.509 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "total_entries": 0, 2023-05-31T21:35:44.509 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "pos": 0, 2023-05-31T21:35:44.509 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "timestamp": "2023-05-31T21:27:37.214586Z" 2023-05-31T21:35:44.510 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2023-05-31T21:35:44.510 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2023-05-31T21:35:44.510 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2023-05-31T21:35:44.510 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "key": 2, 2023-05-31T21:35:44.511 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "val": { 2023-05-31T21:35:44.511 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "status": "incremental-sync", 2023-05-31T21:35:44.511 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "marker": "00000000000000000000:00000000000000051391", 2023-05-31T21:35:44.511 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "next_step_marker": "", 2023-05-31T21:35:44.511 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "total_entries": 55, 2023-05-31T21:35:44.512 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "pos": 0, 2023-05-31T21:35:44.512 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "timestamp": "2023-05-31T21:25:53.141700Z" 2023-05-31T21:35:44.512 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2023-05-31T21:35:44.512 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2023-05-31T21:35:44.513 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2023-05-31T21:35:44.513 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "key": 3, 2023-05-31T21:35:44.513 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "val": { 2023-05-31T21:35:44.513 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "status": "incremental-sync", 2023-05-31T21:35:44.514 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "marker": "00000000000000000000:00000000000000044178", 2023-05-31T21:35:44.514 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "next_step_marker": "", 2023-05-31T21:35:44.514 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "total_entries": 9, 2023-05-31T21:35:44.514 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "pos": 0, 2023-05-31T21:35:44.514 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "timestamp": "2023-05-31T21:27:37.215905Z" 2023-05-31T21:35:44.515 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2023-05-31T21:35:44.515 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2023-05-31T21:35:44.515 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: ] 2023-05-31T21:35:44.515 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2023-05-31T21:35:44.516 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "full_sync": { 2023-05-31T21:35:44.516 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "total": 64, 2023-05-31T21:35:44.516 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "complete": 64 2023-05-31T21:35:44.516 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2023-05-31T21:35:44.517 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "current_time": "2023-05-31T21:28:58Z" 2023-05-31T21:35:44.517 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2023-05-31T21:35:44.517 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current data sync status={ 2023-05-31T21:35:44.517 INFO:tasks.rgw_multisite_tests: "sync_status": { 2023-05-31T21:35:44.517 INFO:tasks.rgw_multisite_tests: "info": { 2023-05-31T21:35:44.518 INFO:tasks.rgw_multisite_tests: "status": "sync", 2023-05-31T21:35:44.518 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2023-05-31T21:35:44.518 INFO:tasks.rgw_multisite_tests: "instance_id": 452136274422638824 2023-05-31T21:35:44.518 INFO:tasks.rgw_multisite_tests: }, 2023-05-31T21:35:44.519 INFO:tasks.rgw_multisite_tests: "markers": [ 2023-05-31T21:35:44.519 INFO:tasks.rgw_multisite_tests: { 2023-05-31T21:35:44.519 INFO:tasks.rgw_multisite_tests: "key": 0, 2023-05-31T21:35:44.519 INFO:tasks.rgw_multisite_tests: "val": { 2023-05-31T21:35:44.520 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2023-05-31T21:35:44.520 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000045235", 2023-05-31T21:35:44.520 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-05-31T21:35:44.520 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-05-31T21:35:44.520 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-05-31T21:35:44.521 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-05-31T21:25:53.171643Z" 2023-05-31T21:35:44.521 INFO:tasks.rgw_multisite_tests: } 2023-05-31T21:35:44.521 INFO:tasks.rgw_multisite_tests: }, 2023-05-31T21:35:44.521 INFO:tasks.rgw_multisite_tests: { 2023-05-31T21:35:44.522 INFO:tasks.rgw_multisite_tests: "key": 1, 2023-05-31T21:35:44.522 INFO:tasks.rgw_multisite_tests: "val": { 2023-05-31T21:35:44.522 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2023-05-31T21:35:44.522 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000031640", 2023-05-31T21:35:44.523 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-05-31T21:35:44.523 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-05-31T21:35:44.523 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-05-31T21:35:44.523 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-05-31T21:27:37.214586Z" 2023-05-31T21:35:44.523 INFO:tasks.rgw_multisite_tests: } 2023-05-31T21:35:44.524 INFO:tasks.rgw_multisite_tests: }, 2023-05-31T21:35:44.524 INFO:tasks.rgw_multisite_tests: { 2023-05-31T21:35:44.524 INFO:tasks.rgw_multisite_tests: "key": 2, 2023-05-31T21:35:44.524 INFO:tasks.rgw_multisite_tests: "val": { 2023-05-31T21:35:44.525 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2023-05-31T21:35:44.525 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000051391", 2023-05-31T21:35:44.525 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-05-31T21:35:44.525 INFO:tasks.rgw_multisite_tests: "total_entries": 55, 2023-05-31T21:35:44.526 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-05-31T21:35:44.526 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-05-31T21:25:53.141700Z" 2023-05-31T21:35:44.526 INFO:tasks.rgw_multisite_tests: } 2023-05-31T21:35:44.526 INFO:tasks.rgw_multisite_tests: }, 2023-05-31T21:35:44.526 INFO:tasks.rgw_multisite_tests: { 2023-05-31T21:35:44.527 INFO:tasks.rgw_multisite_tests: "key": 3, 2023-05-31T21:35:44.527 INFO:tasks.rgw_multisite_tests: "val": { 2023-05-31T21:35:44.527 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2023-05-31T21:35:44.527 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000044178", 2023-05-31T21:35:44.528 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-05-31T21:35:44.528 INFO:tasks.rgw_multisite_tests: "total_entries": 9, 2023-05-31T21:35:44.528 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-05-31T21:35:44.528 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-05-31T21:27:37.215905Z" 2023-05-31T21:35:44.529 INFO:tasks.rgw_multisite_tests: } 2023-05-31T21:35:44.529 INFO:tasks.rgw_multisite_tests: } 2023-05-31T21:35:44.529 INFO:tasks.rgw_multisite_tests: ] 2023-05-31T21:35:44.529 INFO:tasks.rgw_multisite_tests: }, 2023-05-31T21:35:44.529 INFO:tasks.rgw_multisite_tests: "full_sync": { 2023-05-31T21:35:44.530 INFO:tasks.rgw_multisite_tests: "total": 64, 2023-05-31T21:35:44.530 INFO:tasks.rgw_multisite_tests: "complete": 64 2023-05-31T21:35:44.530 INFO:tasks.rgw_multisite_tests: }, 2023-05-31T21:35:44.530 INFO:tasks.rgw_multisite_tests: "current_time": "2023-05-31T21:28:58Z" 2023-05-31T21:35:44.531 INFO:tasks.rgw_multisite_tests:} 2023-05-31T21:35:44.531 INFO:tasks.rgw_multisite_tests: 2023-05-31T21:35:44.531 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000045235', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-05-31T21:25:53.171643Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000031640', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-05-31T21:27:37.214586Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000051391', 'next_step_marker': '', 'total_entries': 55, 'pos': 0, 'timestamp': '2023-05-31T21:25:53.141700Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000044178', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2023-05-31T21:27:37.215905Z'}}] 2023-05-31T21:35:44.531 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '00000000000000000000:00000000000000045375', 1: '00000000000000000000:00000000000000031780', 2: '00000000000000000000:00000000000000051671', 3: '00000000000000000000:00000000000000044178'} 2023-05-31T21:35:44.532 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '00000000000000000000:00000000000000045235', 1: '00000000000000000000:00000000000000031640', 2: '00000000000000000000:00000000000000051391', 3: '00000000000000000000:00000000000000044178'} 2023-05-31T21:35:44.532 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: data of zone a2 behind zone a1: shard=0 master=00000000000000000000:00000000000000045375 target=00000000000000000000:00000000000000045235, shard=1 master=00000000000000000000:00000000000000031780 target=00000000000000000000:00000000000000031640, shard=2 master=00000000000000000000:00000000000000051671 target=00000000000000000000:00000000000000051391 2023-05-31T21:35:44.532 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-31T21:35:44.532 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-31T21:35:44.533 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-05-31T21:35:44.533 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: ignoring --setuser ceph since I am not root 2023-05-31T21:35:44.533 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-05-31T21:35:44.533 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2023-05-31T21:35:44.533 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "sync_status": { 2023-05-31T21:35:44.534 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "info": { 2023-05-31T21:35:44.534 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "status": "sync", 2023-05-31T21:35:44.534 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "num_shards": 4, 2023-05-31T21:35:44.534 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "instance_id": 452136274422638824 2023-05-31T21:35:44.535 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2023-05-31T21:35:44.535 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "markers": [ 2023-05-31T21:35:44.535 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2023-05-31T21:35:44.535 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "key": 0, 2023-05-31T21:35:44.535 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "val": { 2023-05-31T21:35:44.536 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "status": "incremental-sync", 2023-05-31T21:35:44.536 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "marker": "00000000000000000000:00000000000000045235", 2023-05-31T21:35:44.536 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "next_step_marker": "", 2023-05-31T21:35:44.536 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "total_entries": 0, 2023-05-31T21:35:44.537 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "pos": 0, 2023-05-31T21:35:44.537 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "timestamp": "2023-05-31T21:25:53.171643Z" 2023-05-31T21:35:44.537 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2023-05-31T21:35:44.537 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2023-05-31T21:35:44.538 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2023-05-31T21:35:44.538 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "key": 1, 2023-05-31T21:35:44.538 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "val": { 2023-05-31T21:35:44.538 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "status": "incremental-sync", 2023-05-31T21:35:44.539 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "marker": "00000000000000000000:00000000000000031640", 2023-05-31T21:35:44.539 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "next_step_marker": "", 2023-05-31T21:35:44.539 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "total_entries": 0, 2023-05-31T21:35:44.539 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "pos": 0, 2023-05-31T21:35:44.539 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "timestamp": "2023-05-31T21:27:37.214586Z" 2023-05-31T21:35:44.540 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2023-05-31T21:35:44.540 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2023-05-31T21:35:44.540 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2023-05-31T21:35:44.540 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "key": 2, 2023-05-31T21:35:44.541 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "val": { 2023-05-31T21:35:44.541 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "status": "incremental-sync", 2023-05-31T21:35:44.541 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "marker": "00000000000000000000:00000000000000051391", 2023-05-31T21:35:44.541 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "next_step_marker": "", 2023-05-31T21:35:44.542 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "total_entries": 55, 2023-05-31T21:35:44.542 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "pos": 0, 2023-05-31T21:35:44.542 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "timestamp": "2023-05-31T21:25:53.141700Z" 2023-05-31T21:35:44.542 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2023-05-31T21:35:44.542 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2023-05-31T21:35:44.543 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2023-05-31T21:35:44.543 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "key": 3, 2023-05-31T21:35:44.543 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "val": { 2023-05-31T21:35:44.543 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "status": "incremental-sync", 2023-05-31T21:35:44.544 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "marker": "00000000000000000000:00000000000000044178", 2023-05-31T21:35:44.544 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "next_step_marker": "", 2023-05-31T21:35:44.544 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "total_entries": 9, 2023-05-31T21:35:44.544 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "pos": 0, 2023-05-31T21:35:44.545 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "timestamp": "2023-05-31T21:27:37.215905Z" 2023-05-31T21:35:44.545 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2023-05-31T21:35:44.545 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2023-05-31T21:35:44.545 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: ] 2023-05-31T21:35:44.545 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2023-05-31T21:35:44.546 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "full_sync": { 2023-05-31T21:35:44.546 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "total": 64, 2023-05-31T21:35:44.546 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "complete": 64 2023-05-31T21:35:44.546 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2023-05-31T21:35:44.547 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "current_time": "2023-05-31T21:29:03Z" 2023-05-31T21:35:44.547 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2023-05-31T21:35:44.547 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current data sync status={ 2023-05-31T21:35:44.547 INFO:tasks.rgw_multisite_tests: "sync_status": { 2023-05-31T21:35:44.548 INFO:tasks.rgw_multisite_tests: "info": { 2023-05-31T21:35:44.548 INFO:tasks.rgw_multisite_tests: "status": "sync", 2023-05-31T21:35:44.548 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2023-05-31T21:35:44.548 INFO:tasks.rgw_multisite_tests: "instance_id": 452136274422638824 2023-05-31T21:35:44.548 INFO:tasks.rgw_multisite_tests: }, 2023-05-31T21:35:44.549 INFO:tasks.rgw_multisite_tests: "markers": [ 2023-05-31T21:35:44.549 INFO:tasks.rgw_multisite_tests: { 2023-05-31T21:35:44.549 INFO:tasks.rgw_multisite_tests: "key": 0, 2023-05-31T21:35:44.549 INFO:tasks.rgw_multisite_tests: "val": { 2023-05-31T21:35:44.550 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2023-05-31T21:35:44.550 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000045235", 2023-05-31T21:35:44.550 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-05-31T21:35:44.550 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-05-31T21:35:44.551 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-05-31T21:35:44.551 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-05-31T21:25:53.171643Z" 2023-05-31T21:35:44.551 INFO:tasks.rgw_multisite_tests: } 2023-05-31T21:35:44.551 INFO:tasks.rgw_multisite_tests: }, 2023-05-31T21:35:44.551 INFO:tasks.rgw_multisite_tests: { 2023-05-31T21:35:44.552 INFO:tasks.rgw_multisite_tests: "key": 1, 2023-05-31T21:35:44.552 INFO:tasks.rgw_multisite_tests: "val": { 2023-05-31T21:35:44.552 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2023-05-31T21:35:44.552 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000031640", 2023-05-31T21:35:44.553 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-05-31T21:35:44.553 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-05-31T21:35:44.553 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-05-31T21:35:44.553 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-05-31T21:27:37.214586Z" 2023-05-31T21:35:44.554 INFO:tasks.rgw_multisite_tests: } 2023-05-31T21:35:44.554 INFO:tasks.rgw_multisite_tests: }, 2023-05-31T21:35:44.554 INFO:tasks.rgw_multisite_tests: { 2023-05-31T21:35:44.554 INFO:tasks.rgw_multisite_tests: "key": 2, 2023-05-31T21:35:44.554 INFO:tasks.rgw_multisite_tests: "val": { 2023-05-31T21:35:44.555 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2023-05-31T21:35:44.555 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000051391", 2023-05-31T21:35:44.555 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-05-31T21:35:44.555 INFO:tasks.rgw_multisite_tests: "total_entries": 55, 2023-05-31T21:35:44.556 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-05-31T21:35:44.556 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-05-31T21:25:53.141700Z" 2023-05-31T21:35:44.556 INFO:tasks.rgw_multisite_tests: } 2023-05-31T21:35:44.556 INFO:tasks.rgw_multisite_tests: }, 2023-05-31T21:35:44.557 INFO:tasks.rgw_multisite_tests: { 2023-05-31T21:35:44.557 INFO:tasks.rgw_multisite_tests: "key": 3, 2023-05-31T21:35:44.557 INFO:tasks.rgw_multisite_tests: "val": { 2023-05-31T21:35:44.557 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2023-05-31T21:35:44.557 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000044178", 2023-05-31T21:35:44.558 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-05-31T21:35:44.558 INFO:tasks.rgw_multisite_tests: "total_entries": 9, 2023-05-31T21:35:44.558 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-05-31T21:35:44.558 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-05-31T21:27:37.215905Z" 2023-05-31T21:35:44.559 INFO:tasks.rgw_multisite_tests: } 2023-05-31T21:35:44.559 INFO:tasks.rgw_multisite_tests: } 2023-05-31T21:35:44.559 INFO:tasks.rgw_multisite_tests: ] 2023-05-31T21:35:44.559 INFO:tasks.rgw_multisite_tests: }, 2023-05-31T21:35:44.560 INFO:tasks.rgw_multisite_tests: "full_sync": { 2023-05-31T21:35:44.560 INFO:tasks.rgw_multisite_tests: "total": 64, 2023-05-31T21:35:44.560 INFO:tasks.rgw_multisite_tests: "complete": 64 2023-05-31T21:35:44.560 INFO:tasks.rgw_multisite_tests: }, 2023-05-31T21:35:44.560 INFO:tasks.rgw_multisite_tests: "current_time": "2023-05-31T21:29:03Z" 2023-05-31T21:35:44.561 INFO:tasks.rgw_multisite_tests:} 2023-05-31T21:35:44.561 INFO:tasks.rgw_multisite_tests: 2023-05-31T21:35:44.561 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000045235', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-05-31T21:25:53.171643Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000031640', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-05-31T21:27:37.214586Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000051391', 'next_step_marker': '', 'total_entries': 55, 'pos': 0, 'timestamp': '2023-05-31T21:25:53.141700Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000044178', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2023-05-31T21:27:37.215905Z'}}] 2023-05-31T21:35:44.561 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '00000000000000000000:00000000000000045375', 1: '00000000000000000000:00000000000000031780', 2: '00000000000000000000:00000000000000051671', 3: '00000000000000000000:00000000000000044178'} 2023-05-31T21:35:44.562 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '00000000000000000000:00000000000000045235', 1: '00000000000000000000:00000000000000031640', 2: '00000000000000000000:00000000000000051391', 3: '00000000000000000000:00000000000000044178'} 2023-05-31T21:35:44.562 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: data of zone a2 behind zone a1: shard=0 master=00000000000000000000:00000000000000045375 target=00000000000000000000:00000000000000045235, shard=1 master=00000000000000000000:00000000000000031780 target=00000000000000000000:00000000000000031640, shard=2 master=00000000000000000000:00000000000000051671 target=00000000000000000000:00000000000000051391 2023-05-31T21:35:44.562 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-31T21:35:44.562 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-31T21:35:44.563 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-05-31T21:35:44.563 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: ignoring --setuser ceph since I am not root 2023-05-31T21:35:44.563 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-05-31T21:35:44.563 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2023-05-31T21:35:44.563 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "sync_status": { 2023-05-31T21:35:44.564 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "info": { 2023-05-31T21:35:44.564 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "status": "sync", 2023-05-31T21:35:44.564 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "num_shards": 4, 2023-05-31T21:35:44.564 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "instance_id": 452136274422638824 2023-05-31T21:35:44.565 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2023-05-31T21:35:44.565 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "markers": [ 2023-05-31T21:35:44.565 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2023-05-31T21:35:44.565 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "key": 0, 2023-05-31T21:35:44.566 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "val": { 2023-05-31T21:35:44.566 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "status": "incremental-sync", 2023-05-31T21:35:44.566 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "marker": "00000000000000000000:00000000000000045235", 2023-05-31T21:35:44.566 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "next_step_marker": "", 2023-05-31T21:35:44.567 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "total_entries": 0, 2023-05-31T21:35:44.567 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "pos": 0, 2023-05-31T21:35:44.567 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "timestamp": "2023-05-31T21:25:53.171643Z" 2023-05-31T21:35:44.567 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2023-05-31T21:35:44.567 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2023-05-31T21:35:44.568 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2023-05-31T21:35:44.568 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "key": 1, 2023-05-31T21:35:44.568 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "val": { 2023-05-31T21:35:44.568 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "status": "incremental-sync", 2023-05-31T21:35:44.569 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "marker": "00000000000000000000:00000000000000031640", 2023-05-31T21:35:44.569 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "next_step_marker": "", 2023-05-31T21:35:44.569 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "total_entries": 0, 2023-05-31T21:35:44.569 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "pos": 0, 2023-05-31T21:35:44.570 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "timestamp": "2023-05-31T21:27:37.214586Z" 2023-05-31T21:35:44.570 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2023-05-31T21:35:44.570 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2023-05-31T21:35:44.570 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2023-05-31T21:35:44.570 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "key": 2, 2023-05-31T21:35:44.571 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "val": { 2023-05-31T21:35:44.571 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "status": "incremental-sync", 2023-05-31T21:35:44.571 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "marker": "00000000000000000000:00000000000000051391", 2023-05-31T21:35:44.571 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "next_step_marker": "", 2023-05-31T21:35:44.572 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "total_entries": 55, 2023-05-31T21:35:44.572 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "pos": 0, 2023-05-31T21:35:44.572 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "timestamp": "2023-05-31T21:25:53.141700Z" 2023-05-31T21:35:44.572 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2023-05-31T21:35:44.573 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2023-05-31T21:35:44.573 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2023-05-31T21:35:44.573 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "key": 3, 2023-05-31T21:35:44.573 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "val": { 2023-05-31T21:35:44.573 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "status": "incremental-sync", 2023-05-31T21:35:44.574 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "marker": "00000000000000000000:00000000000000044178", 2023-05-31T21:35:44.574 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "next_step_marker": "", 2023-05-31T21:35:44.574 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "total_entries": 9, 2023-05-31T21:35:44.574 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "pos": 0, 2023-05-31T21:35:44.575 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "timestamp": "2023-05-31T21:27:37.215905Z" 2023-05-31T21:35:44.575 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2023-05-31T21:35:44.575 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2023-05-31T21:35:44.575 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: ] 2023-05-31T21:35:44.576 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2023-05-31T21:35:44.576 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "full_sync": { 2023-05-31T21:35:44.576 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "total": 64, 2023-05-31T21:35:44.576 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "complete": 64 2023-05-31T21:35:44.576 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2023-05-31T21:35:44.577 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "current_time": "2023-05-31T21:29:08Z" 2023-05-31T21:35:44.577 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2023-05-31T21:35:44.577 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current data sync status={ 2023-05-31T21:35:44.577 INFO:tasks.rgw_multisite_tests: "sync_status": { 2023-05-31T21:35:44.578 INFO:tasks.rgw_multisite_tests: "info": { 2023-05-31T21:35:44.578 INFO:tasks.rgw_multisite_tests: "status": "sync", 2023-05-31T21:35:44.578 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2023-05-31T21:35:44.578 INFO:tasks.rgw_multisite_tests: "instance_id": 452136274422638824 2023-05-31T21:35:44.579 INFO:tasks.rgw_multisite_tests: }, 2023-05-31T21:35:44.579 INFO:tasks.rgw_multisite_tests: "markers": [ 2023-05-31T21:35:44.579 INFO:tasks.rgw_multisite_tests: { 2023-05-31T21:35:44.579 INFO:tasks.rgw_multisite_tests: "key": 0, 2023-05-31T21:35:44.579 INFO:tasks.rgw_multisite_tests: "val": { 2023-05-31T21:35:44.580 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2023-05-31T21:35:44.580 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000045235", 2023-05-31T21:35:44.580 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-05-31T21:35:44.580 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-05-31T21:35:44.581 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-05-31T21:35:44.581 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-05-31T21:25:53.171643Z" 2023-05-31T21:35:44.581 INFO:tasks.rgw_multisite_tests: } 2023-05-31T21:35:44.581 INFO:tasks.rgw_multisite_tests: }, 2023-05-31T21:35:44.582 INFO:tasks.rgw_multisite_tests: { 2023-05-31T21:35:44.582 INFO:tasks.rgw_multisite_tests: "key": 1, 2023-05-31T21:35:44.582 INFO:tasks.rgw_multisite_tests: "val": { 2023-05-31T21:35:44.582 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2023-05-31T21:35:44.582 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000031640", 2023-05-31T21:35:44.583 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-05-31T21:35:44.583 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-05-31T21:35:44.583 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-05-31T21:35:44.583 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-05-31T21:27:37.214586Z" 2023-05-31T21:35:44.584 INFO:tasks.rgw_multisite_tests: } 2023-05-31T21:35:44.584 INFO:tasks.rgw_multisite_tests: }, 2023-05-31T21:35:44.584 INFO:tasks.rgw_multisite_tests: { 2023-05-31T21:35:44.584 INFO:tasks.rgw_multisite_tests: "key": 2, 2023-05-31T21:35:44.585 INFO:tasks.rgw_multisite_tests: "val": { 2023-05-31T21:35:44.585 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2023-05-31T21:35:44.585 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000051391", 2023-05-31T21:35:44.585 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-05-31T21:35:44.585 INFO:tasks.rgw_multisite_tests: "total_entries": 55, 2023-05-31T21:35:44.586 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-05-31T21:35:44.586 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-05-31T21:25:53.141700Z" 2023-05-31T21:35:44.586 INFO:tasks.rgw_multisite_tests: } 2023-05-31T21:35:44.586 INFO:tasks.rgw_multisite_tests: }, 2023-05-31T21:35:44.587 INFO:tasks.rgw_multisite_tests: { 2023-05-31T21:35:44.587 INFO:tasks.rgw_multisite_tests: "key": 3, 2023-05-31T21:35:44.587 INFO:tasks.rgw_multisite_tests: "val": { 2023-05-31T21:35:44.587 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2023-05-31T21:35:44.587 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000044178", 2023-05-31T21:35:44.588 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-05-31T21:35:44.588 INFO:tasks.rgw_multisite_tests: "total_entries": 9, 2023-05-31T21:35:44.588 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-05-31T21:35:44.588 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-05-31T21:27:37.215905Z" 2023-05-31T21:35:44.589 INFO:tasks.rgw_multisite_tests: } 2023-05-31T21:35:44.589 INFO:tasks.rgw_multisite_tests: } 2023-05-31T21:35:44.589 INFO:tasks.rgw_multisite_tests: ] 2023-05-31T21:35:44.589 INFO:tasks.rgw_multisite_tests: }, 2023-05-31T21:35:44.590 INFO:tasks.rgw_multisite_tests: "full_sync": { 2023-05-31T21:35:44.590 INFO:tasks.rgw_multisite_tests: "total": 64, 2023-05-31T21:35:44.590 INFO:tasks.rgw_multisite_tests: "complete": 64 2023-05-31T21:35:44.590 INFO:tasks.rgw_multisite_tests: }, 2023-05-31T21:35:44.590 INFO:tasks.rgw_multisite_tests: "current_time": "2023-05-31T21:29:08Z" 2023-05-31T21:35:44.591 INFO:tasks.rgw_multisite_tests:} 2023-05-31T21:35:44.591 INFO:tasks.rgw_multisite_tests: 2023-05-31T21:35:44.591 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000045235', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-05-31T21:25:53.171643Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000031640', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-05-31T21:27:37.214586Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000051391', 'next_step_marker': '', 'total_entries': 55, 'pos': 0, 'timestamp': '2023-05-31T21:25:53.141700Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000044178', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2023-05-31T21:27:37.215905Z'}}] 2023-05-31T21:35:44.591 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '00000000000000000000:00000000000000045375', 1: '00000000000000000000:00000000000000031780', 2: '00000000000000000000:00000000000000051671', 3: '00000000000000000000:00000000000000044178'} 2023-05-31T21:35:44.592 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '00000000000000000000:00000000000000045235', 1: '00000000000000000000:00000000000000031640', 2: '00000000000000000000:00000000000000051391', 3: '00000000000000000000:00000000000000044178'} 2023-05-31T21:35:44.592 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: data of zone a2 behind zone a1: shard=0 master=00000000000000000000:00000000000000045375 target=00000000000000000000:00000000000000045235, shard=1 master=00000000000000000000:00000000000000031780 target=00000000000000000000:00000000000000031640, shard=2 master=00000000000000000000:00000000000000051671 target=00000000000000000000:00000000000000051391 2023-05-31T21:35:44.592 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-31T21:35:44.592 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-31T21:35:44.593 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-05-31T21:35:44.593 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: ignoring --setuser ceph since I am not root 2023-05-31T21:35:44.593 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-05-31T21:35:44.593 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2023-05-31T21:35:44.594 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "sync_status": { 2023-05-31T21:35:44.594 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "info": { 2023-05-31T21:35:44.594 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "status": "sync", 2023-05-31T21:35:44.594 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "num_shards": 4, 2023-05-31T21:35:44.595 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "instance_id": 452136274422638824 2023-05-31T21:35:44.595 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2023-05-31T21:35:44.595 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "markers": [ 2023-05-31T21:35:44.595 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2023-05-31T21:35:44.595 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "key": 0, 2023-05-31T21:35:44.596 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "val": { 2023-05-31T21:35:44.596 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "status": "incremental-sync", 2023-05-31T21:35:44.596 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "marker": "00000000000000000000:00000000000000045235", 2023-05-31T21:35:44.596 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "next_step_marker": "", 2023-05-31T21:35:44.597 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "total_entries": 0, 2023-05-31T21:35:44.597 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "pos": 0, 2023-05-31T21:35:44.597 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "timestamp": "2023-05-31T21:25:53.171643Z" 2023-05-31T21:35:44.597 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2023-05-31T21:35:44.598 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2023-05-31T21:35:44.598 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2023-05-31T21:35:44.598 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "key": 1, 2023-05-31T21:35:44.598 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "val": { 2023-05-31T21:35:44.598 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "status": "incremental-sync", 2023-05-31T21:35:44.599 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "marker": "00000000000000000000:00000000000000031640", 2023-05-31T21:35:44.599 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "next_step_marker": "", 2023-05-31T21:35:44.599 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "total_entries": 0, 2023-05-31T21:35:44.599 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "pos": 0, 2023-05-31T21:35:44.600 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "timestamp": "2023-05-31T21:27:37.214586Z" 2023-05-31T21:35:44.600 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2023-05-31T21:35:44.600 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2023-05-31T21:35:44.600 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2023-05-31T21:35:44.601 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "key": 2, 2023-05-31T21:35:44.601 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "val": { 2023-05-31T21:35:44.601 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "status": "incremental-sync", 2023-05-31T21:35:44.601 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "marker": "00000000000000000000:00000000000000051391", 2023-05-31T21:35:44.601 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "next_step_marker": "", 2023-05-31T21:35:44.602 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "total_entries": 55, 2023-05-31T21:35:44.602 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "pos": 0, 2023-05-31T21:35:44.602 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "timestamp": "2023-05-31T21:25:53.141700Z" 2023-05-31T21:35:44.602 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2023-05-31T21:35:44.603 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2023-05-31T21:35:44.603 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2023-05-31T21:35:44.603 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "key": 3, 2023-05-31T21:35:44.603 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "val": { 2023-05-31T21:35:44.603 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "status": "incremental-sync", 2023-05-31T21:35:44.604 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "marker": "00000000000000000000:00000000000000044178", 2023-05-31T21:35:44.604 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "next_step_marker": "", 2023-05-31T21:35:44.604 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "total_entries": 9, 2023-05-31T21:35:44.604 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "pos": 0, 2023-05-31T21:35:44.605 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "timestamp": "2023-05-31T21:27:37.215905Z" 2023-05-31T21:35:44.605 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2023-05-31T21:35:44.605 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2023-05-31T21:35:44.605 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: ] 2023-05-31T21:35:44.606 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2023-05-31T21:35:44.606 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "full_sync": { 2023-05-31T21:35:44.606 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "total": 64, 2023-05-31T21:35:44.606 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "complete": 64 2023-05-31T21:35:44.607 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2023-05-31T21:35:44.607 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "current_time": "2023-05-31T21:29:13Z" 2023-05-31T21:35:44.607 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2023-05-31T21:35:44.607 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current data sync status={ 2023-05-31T21:35:44.607 INFO:tasks.rgw_multisite_tests: "sync_status": { 2023-05-31T21:35:44.608 INFO:tasks.rgw_multisite_tests: "info": { 2023-05-31T21:35:44.608 INFO:tasks.rgw_multisite_tests: "status": "sync", 2023-05-31T21:35:44.608 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2023-05-31T21:35:44.608 INFO:tasks.rgw_multisite_tests: "instance_id": 452136274422638824 2023-05-31T21:35:44.609 INFO:tasks.rgw_multisite_tests: }, 2023-05-31T21:35:44.609 INFO:tasks.rgw_multisite_tests: "markers": [ 2023-05-31T21:35:44.609 INFO:tasks.rgw_multisite_tests: { 2023-05-31T21:35:44.609 INFO:tasks.rgw_multisite_tests: "key": 0, 2023-05-31T21:35:44.610 INFO:tasks.rgw_multisite_tests: "val": { 2023-05-31T21:35:44.610 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2023-05-31T21:35:44.610 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000045235", 2023-05-31T21:35:44.610 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-05-31T21:35:44.610 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-05-31T21:35:44.611 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-05-31T21:35:44.611 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-05-31T21:25:53.171643Z" 2023-05-31T21:35:44.611 INFO:tasks.rgw_multisite_tests: } 2023-05-31T21:35:44.611 INFO:tasks.rgw_multisite_tests: }, 2023-05-31T21:35:44.612 INFO:tasks.rgw_multisite_tests: { 2023-05-31T21:35:44.612 INFO:tasks.rgw_multisite_tests: "key": 1, 2023-05-31T21:35:44.612 INFO:tasks.rgw_multisite_tests: "val": { 2023-05-31T21:35:44.612 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2023-05-31T21:35:44.613 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000031640", 2023-05-31T21:35:44.613 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-05-31T21:35:44.613 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-05-31T21:35:44.613 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-05-31T21:35:44.613 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-05-31T21:27:37.214586Z" 2023-05-31T21:35:44.614 INFO:tasks.rgw_multisite_tests: } 2023-05-31T21:35:44.614 INFO:tasks.rgw_multisite_tests: }, 2023-05-31T21:35:44.614 INFO:tasks.rgw_multisite_tests: { 2023-05-31T21:35:44.614 INFO:tasks.rgw_multisite_tests: "key": 2, 2023-05-31T21:35:44.615 INFO:tasks.rgw_multisite_tests: "val": { 2023-05-31T21:35:44.615 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2023-05-31T21:35:44.615 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000051391", 2023-05-31T21:35:44.615 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-05-31T21:35:44.616 INFO:tasks.rgw_multisite_tests: "total_entries": 55, 2023-05-31T21:35:44.616 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-05-31T21:35:44.616 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-05-31T21:25:53.141700Z" 2023-05-31T21:35:44.616 INFO:tasks.rgw_multisite_tests: } 2023-05-31T21:35:44.616 INFO:tasks.rgw_multisite_tests: }, 2023-05-31T21:35:44.617 INFO:tasks.rgw_multisite_tests: { 2023-05-31T21:35:44.617 INFO:tasks.rgw_multisite_tests: "key": 3, 2023-05-31T21:35:44.617 INFO:tasks.rgw_multisite_tests: "val": { 2023-05-31T21:35:44.617 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2023-05-31T21:35:44.618 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000044178", 2023-05-31T21:35:44.618 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-05-31T21:35:44.618 INFO:tasks.rgw_multisite_tests: "total_entries": 9, 2023-05-31T21:35:44.618 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-05-31T21:35:44.619 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-05-31T21:27:37.215905Z" 2023-05-31T21:35:44.619 INFO:tasks.rgw_multisite_tests: } 2023-05-31T21:35:44.619 INFO:tasks.rgw_multisite_tests: } 2023-05-31T21:35:44.619 INFO:tasks.rgw_multisite_tests: ] 2023-05-31T21:35:44.619 INFO:tasks.rgw_multisite_tests: }, 2023-05-31T21:35:44.620 INFO:tasks.rgw_multisite_tests: "full_sync": { 2023-05-31T21:35:44.620 INFO:tasks.rgw_multisite_tests: "total": 64, 2023-05-31T21:35:44.620 INFO:tasks.rgw_multisite_tests: "complete": 64 2023-05-31T21:35:44.620 INFO:tasks.rgw_multisite_tests: }, 2023-05-31T21:35:44.621 INFO:tasks.rgw_multisite_tests: "current_time": "2023-05-31T21:29:13Z" 2023-05-31T21:35:44.621 INFO:tasks.rgw_multisite_tests:} 2023-05-31T21:35:44.621 INFO:tasks.rgw_multisite_tests: 2023-05-31T21:35:44.621 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000045235', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-05-31T21:25:53.171643Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000031640', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-05-31T21:27:37.214586Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000051391', 'next_step_marker': '', 'total_entries': 55, 'pos': 0, 'timestamp': '2023-05-31T21:25:53.141700Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000044178', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2023-05-31T21:27:37.215905Z'}}] 2023-05-31T21:35:44.622 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '00000000000000000000:00000000000000045375', 1: '00000000000000000000:00000000000000031780', 2: '00000000000000000000:00000000000000051671', 3: '00000000000000000000:00000000000000044178'} 2023-05-31T21:35:44.622 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '00000000000000000000:00000000000000045235', 1: '00000000000000000000:00000000000000031640', 2: '00000000000000000000:00000000000000051391', 3: '00000000000000000000:00000000000000044178'} 2023-05-31T21:35:44.622 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: data of zone a2 behind zone a1: shard=0 master=00000000000000000000:00000000000000045375 target=00000000000000000000:00000000000000045235, shard=1 master=00000000000000000000:00000000000000031780 target=00000000000000000000:00000000000000031640, shard=2 master=00000000000000000000:00000000000000051671 target=00000000000000000000:00000000000000051391 2023-05-31T21:35:44.622 INFO:tasks.rgw_multisite_tests:tasks.rgw.c2.client.1.smithi112.stdout: INFO: 2023-05-31T21:29:17.714+0000 7fecb1022700 -1 WARNING: The OSD cluster is overloaded and struggling to complete ops. You need more capacity to serve this level of demand. 2023-05-31T21:35:44.622 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-31T21:35:44.623 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-31T21:35:44.623 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-05-31T21:35:44.623 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: ignoring --setuser ceph since I am not root 2023-05-31T21:35:44.623 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-05-31T21:35:44.624 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2023-05-31T21:35:44.624 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "sync_status": { 2023-05-31T21:35:44.624 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "info": { 2023-05-31T21:35:44.624 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "status": "sync", 2023-05-31T21:35:44.625 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "num_shards": 4, 2023-05-31T21:35:44.625 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "instance_id": 452136274422638824 2023-05-31T21:35:44.625 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2023-05-31T21:35:44.625 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "markers": [ 2023-05-31T21:35:44.625 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2023-05-31T21:35:44.626 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "key": 0, 2023-05-31T21:35:44.626 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "val": { 2023-05-31T21:35:44.626 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "status": "incremental-sync", 2023-05-31T21:35:44.626 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "marker": "00000000000000000000:00000000000000045375", 2023-05-31T21:35:44.627 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "next_step_marker": "", 2023-05-31T21:35:44.627 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "total_entries": 0, 2023-05-31T21:35:44.627 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "pos": 0, 2023-05-31T21:35:44.627 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "timestamp": "2023-05-31T21:28:16.422581Z" 2023-05-31T21:35:44.628 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2023-05-31T21:35:44.628 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2023-05-31T21:35:44.628 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2023-05-31T21:35:44.628 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "key": 1, 2023-05-31T21:35:44.629 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "val": { 2023-05-31T21:35:44.629 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "status": "incremental-sync", 2023-05-31T21:35:44.629 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "marker": "00000000000000000000:00000000000000031780", 2023-05-31T21:35:44.629 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "next_step_marker": "", 2023-05-31T21:35:44.629 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "total_entries": 0, 2023-05-31T21:35:44.630 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "pos": 0, 2023-05-31T21:35:44.630 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "timestamp": "2023-05-31T21:28:16.501293Z" 2023-05-31T21:35:44.630 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2023-05-31T21:35:44.630 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2023-05-31T21:35:44.631 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2023-05-31T21:35:44.631 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "key": 2, 2023-05-31T21:35:44.631 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "val": { 2023-05-31T21:35:44.631 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "status": "incremental-sync", 2023-05-31T21:35:44.631 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "marker": "00000000000000000000:00000000000000051671", 2023-05-31T21:35:44.632 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "next_step_marker": "", 2023-05-31T21:35:44.632 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "total_entries": 55, 2023-05-31T21:35:44.632 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "pos": 0, 2023-05-31T21:35:44.632 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "timestamp": "2023-05-31T21:28:16.476487Z" 2023-05-31T21:35:44.633 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2023-05-31T21:35:44.633 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2023-05-31T21:35:44.633 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2023-05-31T21:35:44.633 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "key": 3, 2023-05-31T21:35:44.634 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "val": { 2023-05-31T21:35:44.634 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "status": "incremental-sync", 2023-05-31T21:35:44.634 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "marker": "00000000000000000000:00000000000000044178", 2023-05-31T21:35:44.634 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "next_step_marker": "", 2023-05-31T21:35:44.634 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "total_entries": 9, 2023-05-31T21:35:44.635 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "pos": 0, 2023-05-31T21:35:44.635 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "timestamp": "2023-05-31T21:27:37.215905Z" 2023-05-31T21:35:44.635 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2023-05-31T21:35:44.635 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2023-05-31T21:35:44.636 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: ] 2023-05-31T21:35:44.636 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2023-05-31T21:35:44.636 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "full_sync": { 2023-05-31T21:35:44.636 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "total": 64, 2023-05-31T21:35:44.637 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "complete": 64 2023-05-31T21:35:44.637 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2023-05-31T21:35:44.637 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "current_time": "2023-05-31T21:29:19Z" 2023-05-31T21:35:44.637 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2023-05-31T21:35:44.637 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current data sync status={ 2023-05-31T21:35:44.638 INFO:tasks.rgw_multisite_tests: "sync_status": { 2023-05-31T21:35:44.638 INFO:tasks.rgw_multisite_tests: "info": { 2023-05-31T21:35:44.638 INFO:tasks.rgw_multisite_tests: "status": "sync", 2023-05-31T21:35:44.638 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2023-05-31T21:35:44.639 INFO:tasks.rgw_multisite_tests: "instance_id": 452136274422638824 2023-05-31T21:35:44.639 INFO:tasks.rgw_multisite_tests: }, 2023-05-31T21:35:44.639 INFO:tasks.rgw_multisite_tests: "markers": [ 2023-05-31T21:35:44.639 INFO:tasks.rgw_multisite_tests: { 2023-05-31T21:35:44.640 INFO:tasks.rgw_multisite_tests: "key": 0, 2023-05-31T21:35:44.640 INFO:tasks.rgw_multisite_tests: "val": { 2023-05-31T21:35:44.640 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2023-05-31T21:35:44.640 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000045375", 2023-05-31T21:35:44.640 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-05-31T21:35:44.641 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-05-31T21:35:44.641 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-05-31T21:35:44.641 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-05-31T21:28:16.422581Z" 2023-05-31T21:35:44.641 INFO:tasks.rgw_multisite_tests: } 2023-05-31T21:35:44.642 INFO:tasks.rgw_multisite_tests: }, 2023-05-31T21:35:44.642 INFO:tasks.rgw_multisite_tests: { 2023-05-31T21:35:44.642 INFO:tasks.rgw_multisite_tests: "key": 1, 2023-05-31T21:35:44.642 INFO:tasks.rgw_multisite_tests: "val": { 2023-05-31T21:35:44.643 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2023-05-31T21:35:44.643 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000031780", 2023-05-31T21:35:44.643 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-05-31T21:35:44.643 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-05-31T21:35:44.643 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-05-31T21:35:44.644 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-05-31T21:28:16.501293Z" 2023-05-31T21:35:44.644 INFO:tasks.rgw_multisite_tests: } 2023-05-31T21:35:44.644 INFO:tasks.rgw_multisite_tests: }, 2023-05-31T21:35:44.644 INFO:tasks.rgw_multisite_tests: { 2023-05-31T21:35:44.645 INFO:tasks.rgw_multisite_tests: "key": 2, 2023-05-31T21:35:44.645 INFO:tasks.rgw_multisite_tests: "val": { 2023-05-31T21:35:44.645 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2023-05-31T21:35:44.645 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000051671", 2023-05-31T21:35:44.646 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-05-31T21:35:44.646 INFO:tasks.rgw_multisite_tests: "total_entries": 55, 2023-05-31T21:35:44.646 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-05-31T21:35:44.646 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-05-31T21:28:16.476487Z" 2023-05-31T21:35:44.646 INFO:tasks.rgw_multisite_tests: } 2023-05-31T21:35:44.647 INFO:tasks.rgw_multisite_tests: }, 2023-05-31T21:35:44.647 INFO:tasks.rgw_multisite_tests: { 2023-05-31T21:35:44.647 INFO:tasks.rgw_multisite_tests: "key": 3, 2023-05-31T21:35:44.647 INFO:tasks.rgw_multisite_tests: "val": { 2023-05-31T21:35:44.648 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2023-05-31T21:35:44.648 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000044178", 2023-05-31T21:35:44.648 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-05-31T21:35:44.648 INFO:tasks.rgw_multisite_tests: "total_entries": 9, 2023-05-31T21:35:44.649 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-05-31T21:35:44.649 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-05-31T21:27:37.215905Z" 2023-05-31T21:35:44.649 INFO:tasks.rgw_multisite_tests: } 2023-05-31T21:35:44.649 INFO:tasks.rgw_multisite_tests: } 2023-05-31T21:35:44.649 INFO:tasks.rgw_multisite_tests: ] 2023-05-31T21:35:44.650 INFO:tasks.rgw_multisite_tests: }, 2023-05-31T21:35:44.650 INFO:tasks.rgw_multisite_tests: "full_sync": { 2023-05-31T21:35:44.650 INFO:tasks.rgw_multisite_tests: "total": 64, 2023-05-31T21:35:44.650 INFO:tasks.rgw_multisite_tests: "complete": 64 2023-05-31T21:35:44.651 INFO:tasks.rgw_multisite_tests: }, 2023-05-31T21:35:44.651 INFO:tasks.rgw_multisite_tests: "current_time": "2023-05-31T21:29:19Z" 2023-05-31T21:35:44.651 INFO:tasks.rgw_multisite_tests:} 2023-05-31T21:35:44.651 INFO:tasks.rgw_multisite_tests: 2023-05-31T21:35:44.652 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000045375', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-05-31T21:28:16.422581Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000031780', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-05-31T21:28:16.501293Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000051671', 'next_step_marker': '', 'total_entries': 55, 'pos': 0, 'timestamp': '2023-05-31T21:28:16.476487Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000044178', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2023-05-31T21:27:37.215905Z'}}] 2023-05-31T21:35:44.652 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '00000000000000000000:00000000000000045375', 1: '00000000000000000000:00000000000000031780', 2: '00000000000000000000:00000000000000051671', 3: '00000000000000000000:00000000000000044178'} 2023-05-31T21:35:44.652 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '00000000000000000000:00000000000000045375', 1: '00000000000000000000:00000000000000031780', 2: '00000000000000000000:00000000000000051671', 3: '00000000000000000000:00000000000000044178'} 2023-05-31T21:35:44.652 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: INFO: finished data checkpoint for target_zone=a2 source_zone=a1 2023-05-31T21:35:44.653 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/pklxkd-72/ 2023-05-31T21:35:44.653 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/pklxkd-72/ 2023-05-31T21:35:44.653 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-05-31T21:35:44.653 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /pklxkd-72/ 2023-05-31T21:35:44.653 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-05-31T21:35:44.654 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-05-31T21:35:44.654 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T21:35:44.654 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-05-31T21:35:44.654 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-05-31T21:35:44.655 INFO:tasks.rgw_multisite_tests:boto: DEBUG: establishing HTTP connection: kwargs={'timeout': 70, 'port': 8001} 2023-05-31T21:35:44.655 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-05-31T21:35:44.655 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-05-31T21:35:44.655 INFO:tasks.rgw_multisite_tests:HEAD 2023-05-31T21:35:44.656 INFO:tasks.rgw_multisite_tests: 2023-05-31T21:35:44.656 INFO:tasks.rgw_multisite_tests: 2023-05-31T21:35:44.656 INFO:tasks.rgw_multisite_tests:Wed, 31 May 2023 21:29:19 GMT 2023-05-31T21:35:44.656 INFO:tasks.rgw_multisite_tests:/pklxkd-72/ 2023-05-31T21:35:44.656 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-05-31T21:35:44.657 INFO:tasks.rgw_multisite_tests:AWS Q80H11OLNU1S48PPYL26:BFJJo4xAtyS1PNeIId1usN44Ywg= 2023-05-31T21:35:44.657 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 21:29:19 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:BFJJo4xAtyS1PNeIId1usN44Ywg=', 'Content-Length': '0'} 2023-05-31T21:35:44.657 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000005b7b42cf84d2e2e9-006477bc2f-2895-a2'), ('Content-Length', '0'), ('Date', 'Wed, 31 May 2023 21:29:19 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:35:44.657 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/pklxkd-72/obj1 2023-05-31T21:35:44.658 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/pklxkd-72/obj1 2023-05-31T21:35:44.658 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-05-31T21:35:44.658 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /pklxkd-72/obj1 2023-05-31T21:35:44.658 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-05-31T21:35:44.659 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-05-31T21:35:44.659 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T21:35:44.659 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-05-31T21:35:44.659 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-05-31T21:35:44.659 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-05-31T21:35:44.660 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-05-31T21:35:44.660 INFO:tasks.rgw_multisite_tests:HEAD 2023-05-31T21:35:44.660 INFO:tasks.rgw_multisite_tests: 2023-05-31T21:35:44.660 INFO:tasks.rgw_multisite_tests: 2023-05-31T21:35:44.661 INFO:tasks.rgw_multisite_tests:Wed, 31 May 2023 21:29:19 GMT 2023-05-31T21:35:44.661 INFO:tasks.rgw_multisite_tests:/pklxkd-72/obj1 2023-05-31T21:35:44.661 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-05-31T21:35:44.661 INFO:tasks.rgw_multisite_tests:AWS Q80H11OLNU1S48PPYL26:r+A6pzN1ALBdR8XWTn8ykL4OXHk= 2023-05-31T21:35:44.662 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 21:29:19 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:r+A6pzN1ALBdR8XWTn8ykL4OXHk=', 'Content-Length': '0'} 2023-05-31T21:35:44.662 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '219'), ('x-amz-request-id', 'tx00000181d0e0a5695c8eb-006477bc2f-2895-a2'), ('Accept-Ranges', 'bytes'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 21:29:19 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:35:44.662 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/pklxkd-72/obj2 2023-05-31T21:35:44.662 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/pklxkd-72/obj2 2023-05-31T21:35:44.662 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-05-31T21:35:44.663 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /pklxkd-72/obj2 2023-05-31T21:35:44.663 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-05-31T21:35:44.663 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-05-31T21:35:44.663 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T21:35:44.664 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-05-31T21:35:44.664 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-05-31T21:35:44.664 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-05-31T21:35:44.664 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-05-31T21:35:44.665 INFO:tasks.rgw_multisite_tests:HEAD 2023-05-31T21:35:44.665 INFO:tasks.rgw_multisite_tests: 2023-05-31T21:35:44.665 INFO:tasks.rgw_multisite_tests: 2023-05-31T21:35:44.665 INFO:tasks.rgw_multisite_tests:Wed, 31 May 2023 21:29:19 GMT 2023-05-31T21:35:44.665 INFO:tasks.rgw_multisite_tests:/pklxkd-72/obj2 2023-05-31T21:35:44.666 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-05-31T21:35:44.666 INFO:tasks.rgw_multisite_tests:AWS Q80H11OLNU1S48PPYL26:P4IzF3tySTbOEpWmEoDYBOjVvIQ= 2023-05-31T21:35:44.666 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 21:29:19 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:P4IzF3tySTbOEpWmEoDYBOjVvIQ=', 'Content-Length': '0'} 2023-05-31T21:35:44.666 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '219'), ('x-amz-request-id', 'tx000004d0aec115bf6872e-006477bc2f-2895-a2'), ('Accept-Ranges', 'bytes'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 21:29:19 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:35:44.667 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/pklxkd-72/obj3 2023-05-31T21:35:44.667 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/pklxkd-72/obj3 2023-05-31T21:35:44.667 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-05-31T21:35:44.667 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /pklxkd-72/obj3 2023-05-31T21:35:44.668 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-05-31T21:35:44.668 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-05-31T21:35:44.668 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T21:35:44.668 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-05-31T21:35:44.668 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-05-31T21:35:44.669 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-05-31T21:35:44.669 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-05-31T21:35:44.669 INFO:tasks.rgw_multisite_tests:HEAD 2023-05-31T21:35:44.669 INFO:tasks.rgw_multisite_tests: 2023-05-31T21:35:44.670 INFO:tasks.rgw_multisite_tests: 2023-05-31T21:35:44.670 INFO:tasks.rgw_multisite_tests:Wed, 31 May 2023 21:29:19 GMT 2023-05-31T21:35:44.670 INFO:tasks.rgw_multisite_tests:/pklxkd-72/obj3 2023-05-31T21:35:44.670 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-05-31T21:35:44.671 INFO:tasks.rgw_multisite_tests:AWS Q80H11OLNU1S48PPYL26:S1TArTQ8AaqFEWTbMqpujSd91X8= 2023-05-31T21:35:44.671 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 21:29:19 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:S1TArTQ8AaqFEWTbMqpujSd91X8=', 'Content-Length': '0'} 2023-05-31T21:35:44.671 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '219'), ('x-amz-request-id', 'tx00000e6950a6c6b6e4784-006477bc2f-2895-a2'), ('Accept-Ranges', 'bytes'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 21:29:19 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:35:44.671 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/pklxkd-72/obj4 2023-05-31T21:35:44.671 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/pklxkd-72/obj4 2023-05-31T21:35:44.672 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-05-31T21:35:44.672 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /pklxkd-72/obj4 2023-05-31T21:35:44.672 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-05-31T21:35:44.672 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-05-31T21:35:44.673 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T21:35:44.673 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-05-31T21:35:44.673 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-05-31T21:35:44.673 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-05-31T21:35:44.674 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-05-31T21:35:44.674 INFO:tasks.rgw_multisite_tests:HEAD 2023-05-31T21:35:44.674 INFO:tasks.rgw_multisite_tests: 2023-05-31T21:35:44.674 INFO:tasks.rgw_multisite_tests: 2023-05-31T21:35:44.674 INFO:tasks.rgw_multisite_tests:Wed, 31 May 2023 21:29:19 GMT 2023-05-31T21:35:44.675 INFO:tasks.rgw_multisite_tests:/pklxkd-72/obj4 2023-05-31T21:35:44.675 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-05-31T21:35:44.675 INFO:tasks.rgw_multisite_tests:AWS Q80H11OLNU1S48PPYL26:zUGPQgsgcD1bptrIsSIh2SkGxjE= 2023-05-31T21:35:44.675 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 21:29:19 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:zUGPQgsgcD1bptrIsSIh2SkGxjE=', 'Content-Length': '0'} 2023-05-31T21:35:44.676 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '219'), ('x-amz-request-id', 'tx0000079c98e2558020c81-006477bc2f-2895-a2'), ('Accept-Ranges', 'bytes'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 21:29:19 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:35:44.676 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/pklxkd-73/ 2023-05-31T21:35:44.676 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/pklxkd-73/ 2023-05-31T21:35:44.676 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-05-31T21:35:44.676 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /pklxkd-73/ 2023-05-31T21:35:44.677 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-05-31T21:35:44.677 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-05-31T21:35:44.677 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T21:35:44.677 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-05-31T21:35:44.678 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-05-31T21:35:44.678 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-05-31T21:35:44.678 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-05-31T21:35:44.678 INFO:tasks.rgw_multisite_tests:HEAD 2023-05-31T21:35:44.679 INFO:tasks.rgw_multisite_tests: 2023-05-31T21:35:44.679 INFO:tasks.rgw_multisite_tests: 2023-05-31T21:35:44.679 INFO:tasks.rgw_multisite_tests:Wed, 31 May 2023 21:29:19 GMT 2023-05-31T21:35:44.679 INFO:tasks.rgw_multisite_tests:/pklxkd-73/ 2023-05-31T21:35:44.679 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-05-31T21:35:44.680 INFO:tasks.rgw_multisite_tests:AWS Q80H11OLNU1S48PPYL26:sJckmyz/H1LJxlU3uaTHyo7egJI= 2023-05-31T21:35:44.680 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 21:29:19 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:sJckmyz/H1LJxlU3uaTHyo7egJI=', 'Content-Length': '0'} 2023-05-31T21:35:44.680 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '4'), ('X-RGW-Bytes-Used', '24'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000004c67f428c7fea22c-006477bc2f-2895-a2'), ('Content-Length', '0'), ('Date', 'Wed, 31 May 2023 21:29:19 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:35:44.680 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/pklxkd-73/obj1 2023-05-31T21:35:44.681 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/pklxkd-73/obj1 2023-05-31T21:35:44.681 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-05-31T21:35:44.681 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /pklxkd-73/obj1 2023-05-31T21:35:44.681 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-05-31T21:35:44.682 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-05-31T21:35:44.682 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T21:35:44.682 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-05-31T21:35:44.682 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-05-31T21:35:44.682 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-05-31T21:35:44.683 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-05-31T21:35:44.683 INFO:tasks.rgw_multisite_tests:HEAD 2023-05-31T21:35:44.683 INFO:tasks.rgw_multisite_tests: 2023-05-31T21:35:44.683 INFO:tasks.rgw_multisite_tests: 2023-05-31T21:35:44.684 INFO:tasks.rgw_multisite_tests:Wed, 31 May 2023 21:29:19 GMT 2023-05-31T21:35:44.684 INFO:tasks.rgw_multisite_tests:/pklxkd-73/obj1 2023-05-31T21:35:44.684 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-05-31T21:35:44.684 INFO:tasks.rgw_multisite_tests:AWS Q80H11OLNU1S48PPYL26:phEb9kqw3ArZ6zfq4zWwft5SFQ0= 2023-05-31T21:35:44.685 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 21:29:19 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:phEb9kqw3ArZ6zfq4zWwft5SFQ0=', 'Content-Length': '0'} 2023-05-31T21:35:44.685 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 21:28:16 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000009557cea09a20409f-006477bc2f-2895-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 21:29:19 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:35:44.685 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/pklxkd-73/obj1 2023-05-31T21:35:44.685 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/pklxkd-73/obj1 2023-05-31T21:35:44.685 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-05-31T21:35:44.686 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /pklxkd-73/obj1 2023-05-31T21:35:44.686 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-05-31T21:35:44.686 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-05-31T21:35:44.686 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T21:35:44.687 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-05-31T21:35:44.687 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-05-31T21:35:44.687 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-05-31T21:35:44.687 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-05-31T21:35:44.688 INFO:tasks.rgw_multisite_tests:GET 2023-05-31T21:35:44.688 INFO:tasks.rgw_multisite_tests: 2023-05-31T21:35:44.688 INFO:tasks.rgw_multisite_tests: 2023-05-31T21:35:44.688 INFO:tasks.rgw_multisite_tests:Wed, 31 May 2023 21:29:19 GMT 2023-05-31T21:35:44.688 INFO:tasks.rgw_multisite_tests:/pklxkd-73/obj1 2023-05-31T21:35:44.689 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-05-31T21:35:44.689 INFO:tasks.rgw_multisite_tests:AWS Q80H11OLNU1S48PPYL26:qYzcv4wNIUPtK73on70Hg1uRCiI= 2023-05-31T21:35:44.689 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 21:29:19 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:qYzcv4wNIUPtK73on70Hg1uRCiI=', 'Content-Length': '0'} 2023-05-31T21:35:44.689 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 21:28:16 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000af3dca5f466e1f0c-006477bc2f-2895-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 21:29:19 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:35:44.690 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/pklxkd-73/obj2 2023-05-31T21:35:44.690 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/pklxkd-73/obj2 2023-05-31T21:35:44.690 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-05-31T21:35:44.690 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /pklxkd-73/obj2 2023-05-31T21:35:44.691 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-05-31T21:35:44.691 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-05-31T21:35:44.691 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T21:35:44.691 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-05-31T21:35:44.691 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-05-31T21:35:44.692 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-05-31T21:35:44.692 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-05-31T21:35:44.692 INFO:tasks.rgw_multisite_tests:HEAD 2023-05-31T21:35:44.692 INFO:tasks.rgw_multisite_tests: 2023-05-31T21:35:44.693 INFO:tasks.rgw_multisite_tests: 2023-05-31T21:35:44.693 INFO:tasks.rgw_multisite_tests:Wed, 31 May 2023 21:29:19 GMT 2023-05-31T21:35:44.693 INFO:tasks.rgw_multisite_tests:/pklxkd-73/obj2 2023-05-31T21:35:44.693 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-05-31T21:35:44.694 INFO:tasks.rgw_multisite_tests:AWS Q80H11OLNU1S48PPYL26:ItvQ3HbDpzjecotJU0Jbr2FsyyI= 2023-05-31T21:35:44.694 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 21:29:19 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:ItvQ3HbDpzjecotJU0Jbr2FsyyI=', 'Content-Length': '0'} 2023-05-31T21:35:44.694 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 21:28:16 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000040dc51eefd9e4536-006477bc2f-2895-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 21:29:19 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:35:44.694 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/pklxkd-73/obj2 2023-05-31T21:35:44.694 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/pklxkd-73/obj2 2023-05-31T21:35:44.695 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-05-31T21:35:44.695 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /pklxkd-73/obj2 2023-05-31T21:35:44.695 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-05-31T21:35:44.695 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-05-31T21:35:44.696 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T21:35:44.696 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-05-31T21:35:44.696 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-05-31T21:35:44.696 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-05-31T21:35:44.697 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-05-31T21:35:44.697 INFO:tasks.rgw_multisite_tests:GET 2023-05-31T21:35:44.697 INFO:tasks.rgw_multisite_tests: 2023-05-31T21:35:44.697 INFO:tasks.rgw_multisite_tests: 2023-05-31T21:35:44.697 INFO:tasks.rgw_multisite_tests:Wed, 31 May 2023 21:29:19 GMT 2023-05-31T21:35:44.698 INFO:tasks.rgw_multisite_tests:/pklxkd-73/obj2 2023-05-31T21:35:44.698 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-05-31T21:35:44.698 INFO:tasks.rgw_multisite_tests:AWS Q80H11OLNU1S48PPYL26:2XfO5WHUyQj4hijiDmsxKcAN9Q0= 2023-05-31T21:35:44.698 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 21:29:19 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:2XfO5WHUyQj4hijiDmsxKcAN9Q0=', 'Content-Length': '0'} 2023-05-31T21:35:44.699 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 21:28:16 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000025fac1bc4b9fde98-006477bc2f-2895-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 21:29:19 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:35:44.699 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/pklxkd-73/obj3 2023-05-31T21:35:44.699 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/pklxkd-73/obj3 2023-05-31T21:35:44.699 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-05-31T21:35:44.699 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /pklxkd-73/obj3 2023-05-31T21:35:44.700 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-05-31T21:35:44.700 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-05-31T21:35:44.700 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T21:35:44.700 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-05-31T21:35:44.701 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-05-31T21:35:44.701 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-05-31T21:35:44.701 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-05-31T21:35:44.701 INFO:tasks.rgw_multisite_tests:HEAD 2023-05-31T21:35:44.702 INFO:tasks.rgw_multisite_tests: 2023-05-31T21:35:44.702 INFO:tasks.rgw_multisite_tests: 2023-05-31T21:35:44.702 INFO:tasks.rgw_multisite_tests:Wed, 31 May 2023 21:29:19 GMT 2023-05-31T21:35:44.702 INFO:tasks.rgw_multisite_tests:/pklxkd-73/obj3 2023-05-31T21:35:44.702 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-05-31T21:35:44.703 INFO:tasks.rgw_multisite_tests:AWS Q80H11OLNU1S48PPYL26:o7cXevH62cF5/L8I7QK4EnEwUWw= 2023-05-31T21:35:44.703 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 21:29:19 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:o7cXevH62cF5/L8I7QK4EnEwUWw=', 'Content-Length': '0'} 2023-05-31T21:35:44.703 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 21:28:16 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000fe882a74c119fc0a-006477bc2f-2895-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 21:29:19 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:35:44.703 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/pklxkd-73/obj3 2023-05-31T21:35:44.704 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/pklxkd-73/obj3 2023-05-31T21:35:44.704 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-05-31T21:35:44.704 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /pklxkd-73/obj3 2023-05-31T21:35:44.704 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-05-31T21:35:44.705 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-05-31T21:35:44.705 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T21:35:44.705 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-05-31T21:35:44.705 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-05-31T21:35:44.706 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-05-31T21:35:44.706 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-05-31T21:35:44.706 INFO:tasks.rgw_multisite_tests:GET 2023-05-31T21:35:44.706 INFO:tasks.rgw_multisite_tests: 2023-05-31T21:35:44.706 INFO:tasks.rgw_multisite_tests: 2023-05-31T21:35:44.707 INFO:tasks.rgw_multisite_tests:Wed, 31 May 2023 21:29:19 GMT 2023-05-31T21:35:44.707 INFO:tasks.rgw_multisite_tests:/pklxkd-73/obj3 2023-05-31T21:35:44.707 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-05-31T21:35:44.707 INFO:tasks.rgw_multisite_tests:AWS Q80H11OLNU1S48PPYL26:1GAZM9EFqP0vtRu3pnD4RZOM9sw= 2023-05-31T21:35:44.708 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 21:29:19 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:1GAZM9EFqP0vtRu3pnD4RZOM9sw=', 'Content-Length': '0'} 2023-05-31T21:35:44.708 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 21:28:16 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000191f35953b715922-006477bc2f-2895-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 21:29:19 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:35:44.708 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/pklxkd-73/obj4 2023-05-31T21:35:44.708 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/pklxkd-73/obj4 2023-05-31T21:35:44.709 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-05-31T21:35:44.709 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /pklxkd-73/obj4 2023-05-31T21:35:44.709 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-05-31T21:35:44.709 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-05-31T21:35:44.709 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T21:35:44.710 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-05-31T21:35:44.710 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-05-31T21:35:44.710 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-05-31T21:35:44.710 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-05-31T21:35:44.711 INFO:tasks.rgw_multisite_tests:HEAD 2023-05-31T21:35:44.711 INFO:tasks.rgw_multisite_tests: 2023-05-31T21:35:44.711 INFO:tasks.rgw_multisite_tests: 2023-05-31T21:35:44.711 INFO:tasks.rgw_multisite_tests:Wed, 31 May 2023 21:29:19 GMT 2023-05-31T21:35:44.711 INFO:tasks.rgw_multisite_tests:/pklxkd-73/obj4 2023-05-31T21:35:44.712 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-05-31T21:35:44.712 INFO:tasks.rgw_multisite_tests:AWS Q80H11OLNU1S48PPYL26:iiUaAKfJIaMhzvz7bs9BGIRJU4I= 2023-05-31T21:35:44.712 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 21:29:19 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:iiUaAKfJIaMhzvz7bs9BGIRJU4I=', 'Content-Length': '0'} 2023-05-31T21:35:44.712 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 21:28:16 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000e73a54e5b051ba27-006477bc2f-2895-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 21:29:19 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:35:44.713 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/pklxkd-73/obj4 2023-05-31T21:35:44.713 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/pklxkd-73/obj4 2023-05-31T21:35:44.713 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-05-31T21:35:44.713 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /pklxkd-73/obj4 2023-05-31T21:35:44.714 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-05-31T21:35:44.714 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-05-31T21:35:44.714 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T21:35:44.714 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-05-31T21:35:44.715 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-05-31T21:35:44.715 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-05-31T21:35:44.715 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-05-31T21:35:44.715 INFO:tasks.rgw_multisite_tests:GET 2023-05-31T21:35:44.715 INFO:tasks.rgw_multisite_tests: 2023-05-31T21:35:44.716 INFO:tasks.rgw_multisite_tests: 2023-05-31T21:35:44.716 INFO:tasks.rgw_multisite_tests:Wed, 31 May 2023 21:29:19 GMT 2023-05-31T21:35:44.716 INFO:tasks.rgw_multisite_tests:/pklxkd-73/obj4 2023-05-31T21:35:44.716 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-05-31T21:35:44.717 INFO:tasks.rgw_multisite_tests:AWS Q80H11OLNU1S48PPYL26:ylYFT2rgy+tBVS+mJt8Wl5FOJKY= 2023-05-31T21:35:44.717 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 21:29:19 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:ylYFT2rgy+tBVS+mJt8Wl5FOJKY=', 'Content-Length': '0'} 2023-05-31T21:35:44.717 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 21:28:16 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000c9e70b9197b82c25-006477bc2f-2895-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 21:29:19 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:35:44.717 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['sync', 'group', 'pipe', 'remove', '--group-id', 'sync-group', '--pipe-id', 'sync-pipe-pklxkd-72', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-05-31T21:35:44.718 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'group', 'pipe', 'remove', '--group-id', 'sync-group', '--pipe-id', 'sync-pipe-pklxkd-72', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-05-31T21:35:44.718 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync group pipe remove --group-id sync-group --pipe-id sync-pipe-pklxkd-72 --cluster c1 --debug-rgw 0 --debug-ms 0 2023-05-31T21:35:44.718 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: ignoring --setuser ceph since I am not root 2023-05-31T21:35:44.718 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-05-31T21:35:44.718 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: {"groups":[{"id":"sync-group","data_flow":{"directional":[{"source_zone":"a1","dest_zone":"a2"}]},"pipes":[{"id":"sync-pipe-pklxkd-73","source":{"bucket":"pklxkd-73","zones":["a1"]},"dest":{"bucket":"pklxkd-73","zones":["a2"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":""}}],"status":"enabled"}]} 2023-05-31T21:35:44.719 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['sync', 'group', 'pipe', 'remove', '--group-id', 'sync-group', '--pipe-id', 'sync-pipe-pklxkd-73', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-05-31T21:35:44.719 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'group', 'pipe', 'remove', '--group-id', 'sync-group', '--pipe-id', 'sync-pipe-pklxkd-73', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-05-31T21:35:44.719 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync group pipe remove --group-id sync-group --pipe-id sync-pipe-pklxkd-73 --cluster c1 --debug-rgw 0 --debug-ms 0 2023-05-31T21:35:44.719 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: ignoring --setuser ceph since I am not root 2023-05-31T21:35:44.720 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-05-31T21:35:44.720 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: {"groups":[{"id":"sync-group","data_flow":{"directional":[{"source_zone":"a1","dest_zone":"a2"}]},"pipes":[],"status":"enabled"}]} 2023-05-31T21:35:44.720 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['sync', 'group', 'flow', 'remove', '--group-id', 'sync-group', '--flow-id', 'sync-flow', '--flow-type', 'directional', '--source-zone=a1', '--dest-zone=a2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-05-31T21:35:44.720 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'group', 'flow', 'remove', '--group-id', 'sync-group', '--flow-id', 'sync-flow', '--flow-type', 'directional', '--source-zone=a1', '--dest-zone=a2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-05-31T21:35:44.721 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync group flow remove --group-id sync-group --flow-id sync-flow --flow-type directional --source-zone=a1 --dest-zone=a2 --cluster c1 --debug-rgw 0 --debug-ms 0 2023-05-31T21:35:44.721 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: ignoring --setuser ceph since I am not root 2023-05-31T21:35:44.721 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-05-31T21:35:44.721 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: {"groups":[{"id":"sync-group","data_flow":{},"pipes":[],"status":"enabled"}]} 2023-05-31T21:35:44.722 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['sync', 'group', 'flow', 'create', '--group-id', 'sync-group', '--flow-id', 'sync-flow1', '--flow-type', 'symmetrical', '--zones=a1,a2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-05-31T21:35:44.722 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'group', 'flow', 'create', '--group-id', 'sync-group', '--flow-id', 'sync-flow1', '--flow-type', 'symmetrical', '--zones=a1,a2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-05-31T21:35:44.722 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync group flow create --group-id sync-group --flow-id sync-flow1 --flow-type symmetrical --zones=a1,a2 --cluster c1 --debug-rgw 0 --debug-ms 0 2023-05-31T21:35:44.722 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: ignoring --setuser ceph since I am not root 2023-05-31T21:35:44.722 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-05-31T21:35:44.723 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: {"groups":[{"id":"sync-group","data_flow":{"symmetrical":[{"id":"sync-flow1","zones":["a1","a2"]}]},"pipes":[],"status":"enabled"}]} 2023-05-31T21:35:44.723 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['sync', 'group', 'pipe', 'create', '--group-id', 'sync-group', '--pipe-id', 'sync-pipe', '--source-zones=a1,a2', '--dest-zones=a1,a2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-05-31T21:35:44.723 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'group', 'pipe', 'create', '--group-id', 'sync-group', '--pipe-id', 'sync-pipe', '--source-zones=a1,a2', '--dest-zones=a1,a2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-05-31T21:35:44.723 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync group pipe create --group-id sync-group --pipe-id sync-pipe --source-zones=a1,a2 --dest-zones=a1,a2 --cluster c1 --debug-rgw 0 --debug-ms 0 2023-05-31T21:35:44.724 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: ignoring --setuser ceph since I am not root 2023-05-31T21:35:44.724 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-05-31T21:35:44.724 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: {"groups":[{"id":"sync-group","data_flow":{"symmetrical":[{"id":"sync-flow1","zones":["a1","a2"]}]},"pipes":[{"id":"sync-pipe","source":{"bucket":"*","zones":["a1","a2"]},"dest":{"bucket":"*","zones":["a1","a2"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":""}}],"status":"enabled"}]} 2023-05-31T21:35:44.724 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['sync', 'group', 'modify', '--group-id', 'sync-group', '--status', 'allowed', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-05-31T21:35:44.725 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'group', 'modify', '--group-id', 'sync-group', '--status', 'allowed', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-05-31T21:35:44.725 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync group modify --group-id sync-group --status allowed --cluster c1 --debug-rgw 0 --debug-ms 0 2023-05-31T21:35:44.725 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: ignoring --setuser ceph since I am not root 2023-05-31T21:35:44.725 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-05-31T21:35:44.726 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: {"groups":[{"id":"sync-group","data_flow":{"symmetrical":[{"id":"sync-flow1","zones":["a1","a2"]}]},"pipes":[{"id":"sync-pipe","source":{"bucket":"*","zones":["a1","a2"]},"dest":{"bucket":"*","zones":["a1","a2"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":""}}],"status":"allowed"}]} 2023-05-31T21:35:44.726 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['period', 'update', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-05-31T21:35:44.726 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'period', 'update', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-05-31T21:35:44.726 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 period update --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --commit --cluster c1 --debug-rgw 0 --debug-ms 0 2023-05-31T21:35:44.726 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: ignoring --setuser ceph since I am not root 2023-05-31T21:35:44.727 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-05-31T21:35:44.727 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2023-05-31T21:35:44.727 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "id": "2e6123d2-d695-4350-9a01-61a455bc782b", 2023-05-31T21:35:44.727 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "epoch": 20, 2023-05-31T21:35:44.728 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "predecessor_uuid": "629b9ced-80af-41ed-8a6c-f2cfc334cbb4", 2023-05-31T21:35:44.728 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "sync_status": [], 2023-05-31T21:35:44.728 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "period_map": { 2023-05-31T21:35:44.728 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "id": "2e6123d2-d695-4350-9a01-61a455bc782b", 2023-05-31T21:35:44.729 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "zonegroups": [ 2023-05-31T21:35:44.729 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2023-05-31T21:35:44.729 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "id": "a", 2023-05-31T21:35:44.729 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "name": "a", 2023-05-31T21:35:44.729 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "api_name": "", 2023-05-31T21:35:44.730 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "is_master": true, 2023-05-31T21:35:44.730 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "endpoints": [ 2023-05-31T21:35:44.730 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "http://smithi099.front.sepia.ceph.com:8000" 2023-05-31T21:35:44.730 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: ], 2023-05-31T21:35:44.731 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "hostnames": [], 2023-05-31T21:35:44.731 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "hostnames_s3website": [], 2023-05-31T21:35:44.731 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "master_zone": "3d163b1e-09c7-47e2-ae87-add4aace3f38", 2023-05-31T21:35:44.731 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "zones": [ 2023-05-31T21:35:44.732 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2023-05-31T21:35:44.732 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "id": "3d163b1e-09c7-47e2-ae87-add4aace3f38", 2023-05-31T21:35:44.732 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "name": "a1", 2023-05-31T21:35:44.732 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "endpoints": [ 2023-05-31T21:35:44.732 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "http://smithi099.front.sepia.ceph.com:8000" 2023-05-31T21:35:44.733 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: ], 2023-05-31T21:35:44.733 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "log_meta": false, 2023-05-31T21:35:44.733 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "log_data": true, 2023-05-31T21:35:44.733 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "bucket_index_max_shards": 11, 2023-05-31T21:35:44.734 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "read_only": false, 2023-05-31T21:35:44.734 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "tier_type": "", 2023-05-31T21:35:44.734 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "sync_from_all": true, 2023-05-31T21:35:44.734 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "sync_from": [], 2023-05-31T21:35:44.735 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "redirect_zone": "", 2023-05-31T21:35:44.735 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "supported_features": [ 2023-05-31T21:35:44.735 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "resharding" 2023-05-31T21:35:44.735 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: ] 2023-05-31T21:35:44.735 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2023-05-31T21:35:44.736 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2023-05-31T21:35:44.736 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "id": "5d8acec7-7e57-4e56-8b32-e3d6b5036001", 2023-05-31T21:35:44.736 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "name": "a2", 2023-05-31T21:35:44.736 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "endpoints": [ 2023-05-31T21:35:44.737 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "http://smithi099.front.sepia.ceph.com:8001" 2023-05-31T21:35:44.737 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: ], 2023-05-31T21:35:44.737 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "log_meta": false, 2023-05-31T21:35:44.737 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "log_data": true, 2023-05-31T21:35:44.738 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "bucket_index_max_shards": 11, 2023-05-31T21:35:44.738 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "read_only": false, 2023-05-31T21:35:44.738 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "tier_type": "", 2023-05-31T21:35:44.738 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "sync_from_all": true, 2023-05-31T21:35:44.739 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "sync_from": [], 2023-05-31T21:35:44.739 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "redirect_zone": "", 2023-05-31T21:35:44.739 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "supported_features": [ 2023-05-31T21:35:44.739 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "resharding" 2023-05-31T21:35:44.740 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: ] 2023-05-31T21:35:44.740 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2023-05-31T21:35:44.740 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: ], 2023-05-31T21:35:44.740 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "placement_targets": [ 2023-05-31T21:35:44.740 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2023-05-31T21:35:44.741 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "name": "default-placement", 2023-05-31T21:35:44.741 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "tags": [], 2023-05-31T21:35:44.741 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "storage_classes": [ 2023-05-31T21:35:44.741 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "STANDARD" 2023-05-31T21:35:44.742 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: ] 2023-05-31T21:35:44.742 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2023-05-31T21:35:44.742 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: ], 2023-05-31T21:35:44.742 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "default_placement": "default-placement", 2023-05-31T21:35:44.742 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "realm_id": "aaa33de6-515b-4cac-8f1b-3a1a30452117", 2023-05-31T21:35:44.743 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "sync_policy": { 2023-05-31T21:35:44.743 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "groups": [ 2023-05-31T21:35:44.743 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2023-05-31T21:35:44.743 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "id": "sync-group", 2023-05-31T21:35:44.744 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "data_flow": { 2023-05-31T21:35:44.744 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "symmetrical": [ 2023-05-31T21:35:44.744 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2023-05-31T21:35:44.744 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "id": "sync-flow1", 2023-05-31T21:35:44.745 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "zones": [ 2023-05-31T21:35:44.745 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "3d163b1e-09c7-47e2-ae87-add4aace3f38", 2023-05-31T21:35:44.745 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "5d8acec7-7e57-4e56-8b32-e3d6b5036001" 2023-05-31T21:35:44.745 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: ] 2023-05-31T21:35:44.746 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2023-05-31T21:35:44.746 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: ] 2023-05-31T21:35:44.746 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2023-05-31T21:35:44.746 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "pipes": [ 2023-05-31T21:35:44.746 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2023-05-31T21:35:44.747 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "id": "sync-pipe", 2023-05-31T21:35:44.747 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "source": { 2023-05-31T21:35:44.747 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "bucket": "*", 2023-05-31T21:35:44.747 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "zones": [ 2023-05-31T21:35:44.748 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "3d163b1e-09c7-47e2-ae87-add4aace3f38", 2023-05-31T21:35:44.748 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "5d8acec7-7e57-4e56-8b32-e3d6b5036001" 2023-05-31T21:35:44.748 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: ] 2023-05-31T21:35:44.748 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2023-05-31T21:35:44.749 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "dest": { 2023-05-31T21:35:44.749 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "bucket": "*", 2023-05-31T21:35:44.749 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "zones": [ 2023-05-31T21:35:44.749 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "3d163b1e-09c7-47e2-ae87-add4aace3f38", 2023-05-31T21:35:44.749 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "5d8acec7-7e57-4e56-8b32-e3d6b5036001" 2023-05-31T21:35:44.750 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: ] 2023-05-31T21:35:44.750 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2023-05-31T21:35:44.750 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "params": { 2023-05-31T21:35:44.750 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "source": { 2023-05-31T21:35:44.751 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "filter": { 2023-05-31T21:35:44.751 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "tags": [] 2023-05-31T21:35:44.751 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2023-05-31T21:35:44.751 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2023-05-31T21:35:44.752 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "dest": {}, 2023-05-31T21:35:44.752 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "priority": 0, 2023-05-31T21:35:44.752 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "mode": "system", 2023-05-31T21:35:44.752 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "user": "" 2023-05-31T21:35:44.752 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2023-05-31T21:35:44.753 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2023-05-31T21:35:44.753 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: ], 2023-05-31T21:35:44.753 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "status": "allowed" 2023-05-31T21:35:44.753 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2023-05-31T21:35:44.754 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: ] 2023-05-31T21:35:44.754 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2023-05-31T21:35:44.754 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "enabled_features": [ 2023-05-31T21:35:44.754 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "resharding" 2023-05-31T21:35:44.755 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: ] 2023-05-31T21:35:44.755 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2023-05-31T21:35:44.755 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2023-05-31T21:35:44.755 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "id": "b", 2023-05-31T21:35:44.756 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "name": "b", 2023-05-31T21:35:44.756 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "api_name": "", 2023-05-31T21:35:44.756 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "is_master": false, 2023-05-31T21:35:44.756 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "endpoints": [ 2023-05-31T21:35:44.756 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "http://smithi112.front.sepia.ceph.com:8000" 2023-05-31T21:35:44.757 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: ], 2023-05-31T21:35:44.757 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "hostnames": [], 2023-05-31T21:35:44.757 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "hostnames_s3website": [], 2023-05-31T21:35:44.757 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "master_zone": "f85d7236-2f27-4988-af26-9c66ab280b1b", 2023-05-31T21:35:44.758 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "zones": [ 2023-05-31T21:35:44.758 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2023-05-31T21:35:44.758 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "id": "a6a3aab5-bc52-44a4-8543-5c5ea6056acd", 2023-05-31T21:35:44.758 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "name": "b2", 2023-05-31T21:35:44.759 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "endpoints": [ 2023-05-31T21:35:44.759 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "http://smithi112.front.sepia.ceph.com:8001" 2023-05-31T21:35:44.759 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: ], 2023-05-31T21:35:44.759 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "log_meta": false, 2023-05-31T21:35:44.759 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "log_data": true, 2023-05-31T21:35:44.760 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "bucket_index_max_shards": 11, 2023-05-31T21:35:44.760 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "read_only": false, 2023-05-31T21:35:44.760 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "tier_type": "", 2023-05-31T21:35:44.760 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "sync_from_all": true, 2023-05-31T21:35:44.761 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "sync_from": [], 2023-05-31T21:35:44.761 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "redirect_zone": "", 2023-05-31T21:35:44.761 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "supported_features": [ 2023-05-31T21:35:44.761 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "resharding" 2023-05-31T21:35:44.762 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: ] 2023-05-31T21:35:44.762 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2023-05-31T21:35:44.762 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2023-05-31T21:35:44.762 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "id": "f85d7236-2f27-4988-af26-9c66ab280b1b", 2023-05-31T21:35:44.762 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "name": "b1", 2023-05-31T21:35:44.763 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "endpoints": [ 2023-05-31T21:35:44.763 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "http://smithi112.front.sepia.ceph.com:8000" 2023-05-31T21:35:44.763 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: ], 2023-05-31T21:35:44.763 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "log_meta": false, 2023-05-31T21:35:44.764 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "log_data": true, 2023-05-31T21:35:44.764 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "bucket_index_max_shards": 11, 2023-05-31T21:35:44.764 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "read_only": false, 2023-05-31T21:35:44.764 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "tier_type": "", 2023-05-31T21:35:44.765 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "sync_from_all": true, 2023-05-31T21:35:44.765 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "sync_from": [], 2023-05-31T21:35:44.765 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "redirect_zone": "", 2023-05-31T21:35:44.765 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "supported_features": [ 2023-05-31T21:35:44.765 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "resharding" 2023-05-31T21:35:44.766 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: ] 2023-05-31T21:35:44.766 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2023-05-31T21:35:44.766 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: ], 2023-05-31T21:35:44.766 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "placement_targets": [ 2023-05-31T21:35:44.767 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2023-05-31T21:35:44.767 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "name": "default-placement", 2023-05-31T21:35:44.767 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "tags": [], 2023-05-31T21:35:44.767 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "storage_classes": [ 2023-05-31T21:35:44.768 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "STANDARD" 2023-05-31T21:35:44.768 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: ] 2023-05-31T21:35:44.768 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2023-05-31T21:35:44.768 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: ], 2023-05-31T21:35:44.768 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "default_placement": "default-placement", 2023-05-31T21:35:44.769 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "realm_id": "aaa33de6-515b-4cac-8f1b-3a1a30452117", 2023-05-31T21:35:44.769 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "sync_policy": { 2023-05-31T21:35:44.769 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "groups": [] 2023-05-31T21:35:44.769 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2023-05-31T21:35:44.770 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "enabled_features": [ 2023-05-31T21:35:44.770 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "resharding" 2023-05-31T21:35:44.770 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: ] 2023-05-31T21:35:44.770 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2023-05-31T21:35:44.770 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: ], 2023-05-31T21:35:44.771 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "short_zone_ids": [ 2023-05-31T21:35:44.771 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2023-05-31T21:35:44.771 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "key": "3d163b1e-09c7-47e2-ae87-add4aace3f38", 2023-05-31T21:35:44.771 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "val": 3101833771 2023-05-31T21:35:44.772 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2023-05-31T21:35:44.772 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2023-05-31T21:35:44.772 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "key": "5d8acec7-7e57-4e56-8b32-e3d6b5036001", 2023-05-31T21:35:44.772 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "val": 4255530623 2023-05-31T21:35:44.773 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2023-05-31T21:35:44.773 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2023-05-31T21:35:44.773 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "key": "a6a3aab5-bc52-44a4-8543-5c5ea6056acd", 2023-05-31T21:35:44.773 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "val": 3708073710 2023-05-31T21:35:44.774 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2023-05-31T21:35:44.774 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2023-05-31T21:35:44.774 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "key": "f85d7236-2f27-4988-af26-9c66ab280b1b", 2023-05-31T21:35:44.774 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "val": 233395200 2023-05-31T21:35:44.774 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2023-05-31T21:35:44.775 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: ] 2023-05-31T21:35:44.775 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2023-05-31T21:35:44.775 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "master_zonegroup": "a", 2023-05-31T21:35:44.775 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "master_zone": "3d163b1e-09c7-47e2-ae87-add4aace3f38", 2023-05-31T21:35:44.776 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "period_config": { 2023-05-31T21:35:44.776 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "bucket_quota": { 2023-05-31T21:35:44.776 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "enabled": false, 2023-05-31T21:35:44.776 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "check_on_raw": false, 2023-05-31T21:35:44.777 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "max_size": -1, 2023-05-31T21:35:44.777 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "max_size_kb": 0, 2023-05-31T21:35:44.777 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "max_objects": -1 2023-05-31T21:35:44.777 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2023-05-31T21:35:44.777 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "user_quota": { 2023-05-31T21:35:44.778 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "enabled": false, 2023-05-31T21:35:44.778 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "check_on_raw": false, 2023-05-31T21:35:44.778 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "max_size": -1, 2023-05-31T21:35:44.778 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "max_size_kb": 0, 2023-05-31T21:35:44.779 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "max_objects": -1 2023-05-31T21:35:44.779 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2023-05-31T21:35:44.779 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "user_ratelimit": { 2023-05-31T21:35:44.779 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "max_read_ops": 0, 2023-05-31T21:35:44.779 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "max_write_ops": 0, 2023-05-31T21:35:44.780 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "max_read_bytes": 0, 2023-05-31T21:35:44.780 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "max_write_bytes": 0, 2023-05-31T21:35:44.780 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "enabled": false 2023-05-31T21:35:44.780 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2023-05-31T21:35:44.781 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "bucket_ratelimit": { 2023-05-31T21:35:44.781 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "max_read_ops": 0, 2023-05-31T21:35:44.781 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "max_write_ops": 0, 2023-05-31T21:35:44.781 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "max_read_bytes": 0, 2023-05-31T21:35:44.782 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "max_write_bytes": 0, 2023-05-31T21:35:44.782 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "enabled": false 2023-05-31T21:35:44.782 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2023-05-31T21:35:44.782 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "anonymous_ratelimit": { 2023-05-31T21:35:44.782 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "max_read_ops": 0, 2023-05-31T21:35:44.783 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "max_write_ops": 0, 2023-05-31T21:35:44.783 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "max_read_bytes": 0, 2023-05-31T21:35:44.783 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "max_write_bytes": 0, 2023-05-31T21:35:44.783 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "enabled": false 2023-05-31T21:35:44.784 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2023-05-31T21:35:44.784 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2023-05-31T21:35:44.784 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "realm_id": "aaa33de6-515b-4cac-8f1b-3a1a30452117", 2023-05-31T21:35:44.784 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "realm_name": "test-realm", 2023-05-31T21:35:44.785 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "realm_epoch": 2 2023-05-31T21:35:44.785 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2023-05-31T21:35:44.785 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['sync', 'policy', 'get', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-05-31T21:35:44.785 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'policy', 'get', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-05-31T21:35:44.786 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync policy get --cluster c1 --debug-rgw 0 --debug-ms 0 2023-05-31T21:35:44.786 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: ignoring --setuser ceph since I am not root 2023-05-31T21:35:44.786 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-05-31T21:35:44.786 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: {"groups":[{"id":"sync-group","data_flow":{"symmetrical":[{"id":"sync-flow1","zones":["a1","a2"]}]},"pipes":[{"id":"sync-pipe","source":{"bucket":"*","zones":["a1","a2"]},"dest":{"bucket":"*","zones":["a1","a2"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":""}}],"status":"allowed"}]} 2023-05-31T21:35:44.786 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['sync', 'group', 'create', '--group-id', 'sync-bucket', '--status', 'allowed', '--bucket', 'pklxkd-72', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-05-31T21:35:44.787 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'group', 'create', '--group-id', 'sync-bucket', '--status', 'allowed', '--bucket', 'pklxkd-72', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-05-31T21:35:44.787 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync group create --group-id sync-bucket --status allowed --bucket pklxkd-72 --cluster c1 --debug-rgw 0 --debug-ms 0 2023-05-31T21:35:44.787 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: ignoring --setuser ceph since I am not root 2023-05-31T21:35:44.787 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-05-31T21:35:44.788 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: {"groups":[{"id":"sync-bucket","data_flow":{},"pipes":[],"status":"allowed"}]} 2023-05-31T21:35:44.788 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['sync', 'group', 'flow', 'create', '--group-id', 'sync-bucket', '--flow-id', 'sync-flowA', '--flow-type', 'symmetrical', '--zones=a1,a2', '--bucket', 'pklxkd-72', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-05-31T21:35:44.788 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'group', 'flow', 'create', '--group-id', 'sync-bucket', '--flow-id', 'sync-flowA', '--flow-type', 'symmetrical', '--zones=a1,a2', '--bucket', 'pklxkd-72', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-05-31T21:35:44.788 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync group flow create --group-id sync-bucket --flow-id sync-flowA --flow-type symmetrical --zones=a1,a2 --bucket pklxkd-72 --cluster c1 --debug-rgw 0 --debug-ms 0 2023-05-31T21:35:44.789 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: ignoring --setuser ceph since I am not root 2023-05-31T21:35:44.789 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-05-31T21:35:44.789 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: {"groups":[{"id":"sync-bucket","data_flow":{"symmetrical":[{"id":"sync-flowA","zones":["a1","a2"]}]},"pipes":[],"status":"allowed"}]} 2023-05-31T21:35:44.789 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['sync', 'group', 'pipe', 'create', '--group-id', 'sync-bucket', '--pipe-id', 'sync-pipe-pklxkd-72', '--source-zones=a1', '--dest-zones=a2', '--bucket=pklxkd-72', '--source-bucket=pklxkd-72', '--dest-bucket=*', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-05-31T21:35:44.790 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'group', 'pipe', 'create', '--group-id', 'sync-bucket', '--pipe-id', 'sync-pipe-pklxkd-72', '--source-zones=a1', '--dest-zones=a2', '--bucket=pklxkd-72', '--source-bucket=pklxkd-72', '--dest-bucket=*', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-05-31T21:35:44.790 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync group pipe create --group-id sync-bucket --pipe-id sync-pipe-pklxkd-72 --source-zones=a1 --dest-zones=a2 --bucket=pklxkd-72 --source-bucket=pklxkd-72 '--dest-bucket=*' --cluster c1 --debug-rgw 0 --debug-ms 0 2023-05-31T21:35:44.790 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: ignoring --setuser ceph since I am not root 2023-05-31T21:35:44.790 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-05-31T21:35:44.790 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: {"groups":[{"id":"sync-bucket","data_flow":{"symmetrical":[{"id":"sync-flowA","zones":["a1","a2"]}]},"pipes":[{"id":"sync-pipe-pklxkd-72","source":{"bucket":"pklxkd-72","zones":["a1"]},"dest":{"bucket":"*","zones":["a2"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":"rgw-multisite-test-user"}}],"status":"allowed"}]} 2023-05-31T21:35:44.791 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['sync', 'group', 'pipe', 'create', '--group-id', 'sync-bucket', '--pipe-id', 'sync-pipe-pklxkd-73', '--source-zones=a1', '--dest-zones=a2', '--bucket=pklxkd-72', '--source-bucket=pklxkd-73', '--dest-bucket=*', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-05-31T21:35:44.791 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'group', 'pipe', 'create', '--group-id', 'sync-bucket', '--pipe-id', 'sync-pipe-pklxkd-73', '--source-zones=a1', '--dest-zones=a2', '--bucket=pklxkd-72', '--source-bucket=pklxkd-73', '--dest-bucket=*', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-05-31T21:35:44.791 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync group pipe create --group-id sync-bucket --pipe-id sync-pipe-pklxkd-73 --source-zones=a1 --dest-zones=a2 --bucket=pklxkd-72 --source-bucket=pklxkd-73 '--dest-bucket=*' --cluster c1 --debug-rgw 0 --debug-ms 0 2023-05-31T21:35:44.791 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: ignoring --setuser ceph since I am not root 2023-05-31T21:35:44.792 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-05-31T21:35:44.792 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: {"groups":[{"id":"sync-bucket","data_flow":{"symmetrical":[{"id":"sync-flowA","zones":["a1","a2"]}]},"pipes":[{"id":"sync-pipe-pklxkd-72","source":{"bucket":"pklxkd-72","zones":["a1"]},"dest":{"bucket":"*","zones":["a2"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":"rgw-multisite-test-user"}},{"id":"sync-pipe-pklxkd-73","source":{"bucket":"pklxkd-73","zones":["a1"]},"dest":{"bucket":"*","zones":["a2"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":"rgw-multisite-test-user"}}],"status":"allowed"}]} 2023-05-31T21:35:44.792 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['sync', 'group', 'modify', '--group-id', 'sync-bucket', '--status', 'enabled', '--bucket', 'pklxkd-72', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-05-31T21:35:44.792 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'group', 'modify', '--group-id', 'sync-bucket', '--status', 'enabled', '--bucket', 'pklxkd-72', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-05-31T21:35:44.793 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync group modify --group-id sync-bucket --status enabled --bucket pklxkd-72 --cluster c1 --debug-rgw 0 --debug-ms 0 2023-05-31T21:35:44.793 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: ignoring --setuser ceph since I am not root 2023-05-31T21:35:44.793 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-05-31T21:35:44.793 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: {"groups":[{"id":"sync-bucket","data_flow":{"symmetrical":[{"id":"sync-flowA","zones":["a1","a2"]}]},"pipes":[{"id":"sync-pipe-pklxkd-72","source":{"bucket":"pklxkd-72","zones":["a1"]},"dest":{"bucket":"*","zones":["a2"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":"rgw-multisite-test-user"}},{"id":"sync-pipe-pklxkd-73","source":{"bucket":"pklxkd-73","zones":["a1"]},"dest":{"bucket":"*","zones":["a2"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":"rgw-multisite-test-user"}}],"status":"enabled"}]} 2023-05-31T21:35:44.794 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['sync', 'policy', 'get', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-05-31T21:35:44.794 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'policy', 'get', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-05-31T21:35:44.794 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync policy get --cluster c1 --debug-rgw 0 --debug-ms 0 2023-05-31T21:35:44.794 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: ignoring --setuser ceph since I am not root 2023-05-31T21:35:44.795 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-05-31T21:35:44.795 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: {"groups":[{"id":"sync-group","data_flow":{"symmetrical":[{"id":"sync-flow1","zones":["a1","a2"]}]},"pipes":[{"id":"sync-pipe","source":{"bucket":"*","zones":["a1","a2"]},"dest":{"bucket":"*","zones":["a1","a2"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":""}}],"status":"allowed"}]} 2023-05-31T21:35:44.795 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-31T21:35:44.795 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-31T21:35:44.795 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-05-31T21:35:44.796 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: ignoring --setuser ceph since I am not root 2023-05-31T21:35:44.796 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-05-31T21:35:44.796 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: No --period given, using current period=2e6123d2-d695-4350-9a01-61a455bc782b 2023-05-31T21:35:44.796 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: [ 2023-05-31T21:35:44.797 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2023-05-31T21:35:44.797 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "marker": "1_1685566501.757183_260.1", 2023-05-31T21:35:44.797 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "last_update": "2023-05-31T20:55:01.757183Z" 2023-05-31T21:35:44.797 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2023-05-31T21:35:44.798 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2023-05-31T21:35:44.798 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "marker": "1_1685566501.735717_251.1", 2023-05-31T21:35:44.798 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "last_update": "2023-05-31T20:55:01.735717Z" 2023-05-31T21:35:44.798 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2023-05-31T21:35:44.798 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2023-05-31T21:35:44.799 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "marker": "1_1685562970.274572_93.1", 2023-05-31T21:35:44.799 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "last_update": "2023-05-31T19:56:10.274572Z" 2023-05-31T21:35:44.799 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2023-05-31T21:35:44.799 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2023-05-31T21:35:44.800 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "marker": "1_1685568561.112279_1036.1", 2023-05-31T21:35:44.800 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "last_update": "2023-05-31T21:29:21.112279Z" 2023-05-31T21:35:44.800 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2023-05-31T21:35:44.800 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: ] 2023-05-31T21:35:44.801 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: master meta markers={0: '1_1685566501.757183_260.1', 1: '1_1685566501.735717_251.1', 2: '1_1685562970.274572_93.1', 3: '1_1685568561.112279_1036.1'} 2023-05-31T21:35:44.801 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: INFO: starting meta checkpoint for zone=a2 2023-05-31T21:35:44.801 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-31T21:35:44.801 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-31T21:35:44.802 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-05-31T21:35:44.802 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: ignoring --setuser ceph since I am not root 2023-05-31T21:35:44.802 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-05-31T21:35:44.802 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: 2023-05-31T21:29:21.449+0000 7f8f0cd91780 20 RGW-SYNC:meta: read sync status 2023-05-31T21:35:44.802 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: 2023-05-31T21:29:21.450+0000 7f8f0cd91780 20 RGW-SYNC:meta: read sync status 2023-05-31T21:35:44.803 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2023-05-31T21:35:44.803 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "sync_status": { 2023-05-31T21:35:44.803 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "info": { 2023-05-31T21:35:44.803 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "status": "sync", 2023-05-31T21:35:44.804 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "num_shards": 4, 2023-05-31T21:35:44.804 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "period": "2e6123d2-d695-4350-9a01-61a455bc782b", 2023-05-31T21:35:44.804 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "realm_epoch": 2 2023-05-31T21:35:44.804 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2023-05-31T21:35:44.805 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "markers": [ 2023-05-31T21:35:44.805 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2023-05-31T21:35:44.805 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "key": 0, 2023-05-31T21:35:44.805 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "val": { 2023-05-31T21:35:44.805 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "state": 1, 2023-05-31T21:35:44.806 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "marker": "1_1685566501.757183_260.1", 2023-05-31T21:35:44.806 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "next_step_marker": "", 2023-05-31T21:35:44.806 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "total_entries": 1, 2023-05-31T21:35:44.806 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "pos": 0, 2023-05-31T21:35:44.807 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "timestamp": "2023-05-31T20:55:01.757183Z", 2023-05-31T21:35:44.807 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "realm_epoch": 2 2023-05-31T21:35:44.807 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2023-05-31T21:35:44.807 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2023-05-31T21:35:44.808 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2023-05-31T21:35:44.808 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "key": 1, 2023-05-31T21:35:44.808 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "val": { 2023-05-31T21:35:44.808 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "state": 1, 2023-05-31T21:35:44.809 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "marker": "1_1685566501.735717_251.1", 2023-05-31T21:35:44.809 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "next_step_marker": "", 2023-05-31T21:35:44.809 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "total_entries": 0, 2023-05-31T21:35:44.809 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "pos": 0, 2023-05-31T21:35:44.809 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "timestamp": "2023-05-31T20:55:01.735717Z", 2023-05-31T21:35:44.810 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "realm_epoch": 2 2023-05-31T21:35:44.810 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2023-05-31T21:35:44.810 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2023-05-31T21:35:44.810 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2023-05-31T21:35:44.811 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "key": 2, 2023-05-31T21:35:44.811 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "val": { 2023-05-31T21:35:44.811 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "state": 1, 2023-05-31T21:35:44.811 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "marker": "1_1685562970.274572_93.1", 2023-05-31T21:35:44.811 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "next_step_marker": "", 2023-05-31T21:35:44.812 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "total_entries": 0, 2023-05-31T21:35:44.812 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "pos": 0, 2023-05-31T21:35:44.812 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "timestamp": "2023-05-31T19:56:10.274572Z", 2023-05-31T21:35:44.812 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "realm_epoch": 2 2023-05-31T21:35:44.813 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2023-05-31T21:35:44.813 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2023-05-31T21:35:44.813 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2023-05-31T21:35:44.813 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "key": 3, 2023-05-31T21:35:44.814 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "val": { 2023-05-31T21:35:44.814 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "state": 1, 2023-05-31T21:35:44.814 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "marker": "1_1685568489.974220_1014.1", 2023-05-31T21:35:44.814 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "next_step_marker": "", 2023-05-31T21:35:44.814 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "total_entries": 0, 2023-05-31T21:35:44.815 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "pos": 0, 2023-05-31T21:35:44.815 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "timestamp": "2023-05-31T21:28:09.974220Z", 2023-05-31T21:35:44.815 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "realm_epoch": 2 2023-05-31T21:35:44.815 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2023-05-31T21:35:44.816 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2023-05-31T21:35:44.816 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: ] 2023-05-31T21:35:44.816 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2023-05-31T21:35:44.816 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "full_sync": { 2023-05-31T21:35:44.817 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "total": 1, 2023-05-31T21:35:44.817 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "complete": 1 2023-05-31T21:35:44.817 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2023-05-31T21:35:44.817 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "current_time": "2023-05-31T21:29:21Z" 2023-05-31T21:35:44.817 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2023-05-31T21:35:44.818 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2023-05-31T21:35:44.818 INFO:tasks.rgw_multisite_tests: "sync_status": { 2023-05-31T21:35:44.818 INFO:tasks.rgw_multisite_tests: "info": { 2023-05-31T21:35:44.818 INFO:tasks.rgw_multisite_tests: "status": "sync", 2023-05-31T21:35:44.819 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2023-05-31T21:35:44.819 INFO:tasks.rgw_multisite_tests: "period": "2e6123d2-d695-4350-9a01-61a455bc782b", 2023-05-31T21:35:44.819 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-05-31T21:35:44.819 INFO:tasks.rgw_multisite_tests: }, 2023-05-31T21:35:44.820 INFO:tasks.rgw_multisite_tests: "markers": [ 2023-05-31T21:35:44.820 INFO:tasks.rgw_multisite_tests: { 2023-05-31T21:35:44.820 INFO:tasks.rgw_multisite_tests: "key": 0, 2023-05-31T21:35:44.820 INFO:tasks.rgw_multisite_tests: "val": { 2023-05-31T21:35:44.820 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-05-31T21:35:44.821 INFO:tasks.rgw_multisite_tests: "marker": "1_1685566501.757183_260.1", 2023-05-31T21:35:44.821 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-05-31T21:35:44.821 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2023-05-31T21:35:44.821 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-05-31T21:35:44.822 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-05-31T20:55:01.757183Z", 2023-05-31T21:35:44.822 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-05-31T21:35:44.822 INFO:tasks.rgw_multisite_tests: } 2023-05-31T21:35:44.822 INFO:tasks.rgw_multisite_tests: }, 2023-05-31T21:35:44.823 INFO:tasks.rgw_multisite_tests: { 2023-05-31T21:35:44.823 INFO:tasks.rgw_multisite_tests: "key": 1, 2023-05-31T21:35:44.823 INFO:tasks.rgw_multisite_tests: "val": { 2023-05-31T21:35:44.823 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-05-31T21:35:44.823 INFO:tasks.rgw_multisite_tests: "marker": "1_1685566501.735717_251.1", 2023-05-31T21:35:44.824 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-05-31T21:35:44.824 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-05-31T21:35:44.824 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-05-31T21:35:44.824 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-05-31T20:55:01.735717Z", 2023-05-31T21:35:44.825 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-05-31T21:35:44.825 INFO:tasks.rgw_multisite_tests: } 2023-05-31T21:35:44.825 INFO:tasks.rgw_multisite_tests: }, 2023-05-31T21:35:44.825 INFO:tasks.rgw_multisite_tests: { 2023-05-31T21:35:44.826 INFO:tasks.rgw_multisite_tests: "key": 2, 2023-05-31T21:35:44.826 INFO:tasks.rgw_multisite_tests: "val": { 2023-05-31T21:35:44.826 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-05-31T21:35:44.826 INFO:tasks.rgw_multisite_tests: "marker": "1_1685562970.274572_93.1", 2023-05-31T21:35:44.826 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-05-31T21:35:44.827 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-05-31T21:35:44.827 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-05-31T21:35:44.827 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-05-31T19:56:10.274572Z", 2023-05-31T21:35:44.827 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-05-31T21:35:44.828 INFO:tasks.rgw_multisite_tests: } 2023-05-31T21:35:44.828 INFO:tasks.rgw_multisite_tests: }, 2023-05-31T21:35:44.828 INFO:tasks.rgw_multisite_tests: { 2023-05-31T21:35:44.828 INFO:tasks.rgw_multisite_tests: "key": 3, 2023-05-31T21:35:44.829 INFO:tasks.rgw_multisite_tests: "val": { 2023-05-31T21:35:44.829 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-05-31T21:35:44.829 INFO:tasks.rgw_multisite_tests: "marker": "1_1685568489.974220_1014.1", 2023-05-31T21:35:44.829 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-05-31T21:35:44.829 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-05-31T21:35:44.830 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-05-31T21:35:44.830 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-05-31T21:28:09.974220Z", 2023-05-31T21:35:44.830 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-05-31T21:35:44.830 INFO:tasks.rgw_multisite_tests: } 2023-05-31T21:35:44.831 INFO:tasks.rgw_multisite_tests: } 2023-05-31T21:35:44.831 INFO:tasks.rgw_multisite_tests: ] 2023-05-31T21:35:44.831 INFO:tasks.rgw_multisite_tests: }, 2023-05-31T21:35:44.831 INFO:tasks.rgw_multisite_tests: "full_sync": { 2023-05-31T21:35:44.831 INFO:tasks.rgw_multisite_tests: "total": 1, 2023-05-31T21:35:44.832 INFO:tasks.rgw_multisite_tests: "complete": 1 2023-05-31T21:35:44.832 INFO:tasks.rgw_multisite_tests: }, 2023-05-31T21:35:44.832 INFO:tasks.rgw_multisite_tests: "current_time": "2023-05-31T21:29:21Z" 2023-05-31T21:35:44.832 INFO:tasks.rgw_multisite_tests:} 2023-05-31T21:35:44.833 INFO:tasks.rgw_multisite_tests: 2023-05-31T21:35:44.833 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1685566501.757183_260.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-05-31T20:55:01.757183Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1685566501.735717_251.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-05-31T20:55:01.735717Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1685562970.274572_93.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-05-31T19:56:10.274572Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1685568489.974220_1014.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-05-31T21:28:09.974220Z', 'realm_epoch': 2}}] 2023-05-31T21:35:44.833 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1685566501.757183_260.1', 1: '1_1685566501.735717_251.1', 2: '1_1685562970.274572_93.1', 3: '1_1685568561.112279_1036.1'} 2023-05-31T21:35:44.833 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1685566501.757183_260.1', 1: '1_1685566501.735717_251.1', 2: '1_1685562970.274572_93.1', 3: '1_1685568489.974220_1014.1'} 2023-05-31T21:35:44.834 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone a2 behind master: shard=3 master=1_1685568561.112279_1036.1 target=1_1685568489.974220_1014.1 2023-05-31T21:35:44.834 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-31T21:35:44.834 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-31T21:35:44.834 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-05-31T21:35:44.835 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: ignoring --setuser ceph since I am not root 2023-05-31T21:35:44.835 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-05-31T21:35:44.835 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: 2023-05-31T21:29:26.577+0000 7f3d9e706780 20 RGW-SYNC:meta: read sync status 2023-05-31T21:35:44.835 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: 2023-05-31T21:29:26.578+0000 7f3d9e706780 20 RGW-SYNC:meta: read sync status 2023-05-31T21:35:44.835 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2023-05-31T21:35:44.836 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "sync_status": { 2023-05-31T21:35:44.836 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "info": { 2023-05-31T21:35:44.836 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "status": "sync", 2023-05-31T21:35:44.836 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "num_shards": 4, 2023-05-31T21:35:44.837 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "period": "2e6123d2-d695-4350-9a01-61a455bc782b", 2023-05-31T21:35:44.837 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "realm_epoch": 2 2023-05-31T21:35:44.837 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2023-05-31T21:35:44.837 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "markers": [ 2023-05-31T21:35:44.838 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2023-05-31T21:35:44.838 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "key": 0, 2023-05-31T21:35:44.838 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "val": { 2023-05-31T21:35:44.838 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "state": 1, 2023-05-31T21:35:44.838 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "marker": "1_1685566501.757183_260.1", 2023-05-31T21:35:44.839 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "next_step_marker": "", 2023-05-31T21:35:44.839 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "total_entries": 1, 2023-05-31T21:35:44.839 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "pos": 0, 2023-05-31T21:35:44.839 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "timestamp": "2023-05-31T20:55:01.757183Z", 2023-05-31T21:35:44.840 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "realm_epoch": 2 2023-05-31T21:35:44.840 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2023-05-31T21:35:44.840 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2023-05-31T21:35:44.840 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2023-05-31T21:35:44.841 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "key": 1, 2023-05-31T21:35:44.841 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "val": { 2023-05-31T21:35:44.841 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "state": 1, 2023-05-31T21:35:44.841 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "marker": "1_1685566501.735717_251.1", 2023-05-31T21:35:44.841 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "next_step_marker": "", 2023-05-31T21:35:44.842 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "total_entries": 0, 2023-05-31T21:35:44.842 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "pos": 0, 2023-05-31T21:35:44.842 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "timestamp": "2023-05-31T20:55:01.735717Z", 2023-05-31T21:35:44.842 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "realm_epoch": 2 2023-05-31T21:35:44.843 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2023-05-31T21:35:44.843 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2023-05-31T21:35:44.843 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2023-05-31T21:35:44.843 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "key": 2, 2023-05-31T21:35:44.844 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "val": { 2023-05-31T21:35:44.844 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "state": 1, 2023-05-31T21:35:44.844 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "marker": "1_1685562970.274572_93.1", 2023-05-31T21:35:44.844 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "next_step_marker": "", 2023-05-31T21:35:44.845 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "total_entries": 0, 2023-05-31T21:35:44.845 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "pos": 0, 2023-05-31T21:35:44.845 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "timestamp": "2023-05-31T19:56:10.274572Z", 2023-05-31T21:35:44.845 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "realm_epoch": 2 2023-05-31T21:35:44.845 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2023-05-31T21:35:44.846 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2023-05-31T21:35:44.846 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2023-05-31T21:35:44.846 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "key": 3, 2023-05-31T21:35:44.846 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "val": { 2023-05-31T21:35:44.847 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "state": 1, 2023-05-31T21:35:44.847 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "marker": "1_1685568489.974220_1014.1", 2023-05-31T21:35:44.847 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "next_step_marker": "", 2023-05-31T21:35:44.847 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "total_entries": 0, 2023-05-31T21:35:44.847 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "pos": 0, 2023-05-31T21:35:44.848 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "timestamp": "2023-05-31T21:28:09.974220Z", 2023-05-31T21:35:44.848 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "realm_epoch": 2 2023-05-31T21:35:44.848 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2023-05-31T21:35:44.848 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2023-05-31T21:35:44.849 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: ] 2023-05-31T21:35:44.849 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2023-05-31T21:35:44.849 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "full_sync": { 2023-05-31T21:35:44.849 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "total": 1, 2023-05-31T21:35:44.850 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "complete": 1 2023-05-31T21:35:44.850 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2023-05-31T21:35:44.850 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "current_time": "2023-05-31T21:29:26Z" 2023-05-31T21:35:44.850 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2023-05-31T21:35:44.850 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2023-05-31T21:35:44.851 INFO:tasks.rgw_multisite_tests: "sync_status": { 2023-05-31T21:35:44.851 INFO:tasks.rgw_multisite_tests: "info": { 2023-05-31T21:35:44.851 INFO:tasks.rgw_multisite_tests: "status": "sync", 2023-05-31T21:35:44.851 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2023-05-31T21:35:44.852 INFO:tasks.rgw_multisite_tests: "period": "2e6123d2-d695-4350-9a01-61a455bc782b", 2023-05-31T21:35:44.852 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-05-31T21:35:44.852 INFO:tasks.rgw_multisite_tests: }, 2023-05-31T21:35:44.852 INFO:tasks.rgw_multisite_tests: "markers": [ 2023-05-31T21:35:44.853 INFO:tasks.rgw_multisite_tests: { 2023-05-31T21:35:44.853 INFO:tasks.rgw_multisite_tests: "key": 0, 2023-05-31T21:35:44.853 INFO:tasks.rgw_multisite_tests: "val": { 2023-05-31T21:35:44.853 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-05-31T21:35:44.854 INFO:tasks.rgw_multisite_tests: "marker": "1_1685566501.757183_260.1", 2023-05-31T21:35:44.854 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-05-31T21:35:44.854 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2023-05-31T21:35:44.854 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-05-31T21:35:44.854 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-05-31T20:55:01.757183Z", 2023-05-31T21:35:44.855 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-05-31T21:35:44.855 INFO:tasks.rgw_multisite_tests: } 2023-05-31T21:35:44.855 INFO:tasks.rgw_multisite_tests: }, 2023-05-31T21:35:44.855 INFO:tasks.rgw_multisite_tests: { 2023-05-31T21:35:44.856 INFO:tasks.rgw_multisite_tests: "key": 1, 2023-05-31T21:35:44.856 INFO:tasks.rgw_multisite_tests: "val": { 2023-05-31T21:35:44.856 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-05-31T21:35:44.856 INFO:tasks.rgw_multisite_tests: "marker": "1_1685566501.735717_251.1", 2023-05-31T21:35:44.856 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-05-31T21:35:44.857 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-05-31T21:35:44.857 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-05-31T21:35:44.857 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-05-31T20:55:01.735717Z", 2023-05-31T21:35:44.857 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-05-31T21:35:44.858 INFO:tasks.rgw_multisite_tests: } 2023-05-31T21:35:44.858 INFO:tasks.rgw_multisite_tests: }, 2023-05-31T21:35:44.858 INFO:tasks.rgw_multisite_tests: { 2023-05-31T21:35:44.858 INFO:tasks.rgw_multisite_tests: "key": 2, 2023-05-31T21:35:44.859 INFO:tasks.rgw_multisite_tests: "val": { 2023-05-31T21:35:44.859 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-05-31T21:35:44.859 INFO:tasks.rgw_multisite_tests: "marker": "1_1685562970.274572_93.1", 2023-05-31T21:35:44.859 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-05-31T21:35:44.859 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-05-31T21:35:44.860 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-05-31T21:35:44.860 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-05-31T19:56:10.274572Z", 2023-05-31T21:35:44.860 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-05-31T21:35:44.860 INFO:tasks.rgw_multisite_tests: } 2023-05-31T21:35:44.861 INFO:tasks.rgw_multisite_tests: }, 2023-05-31T21:35:44.861 INFO:tasks.rgw_multisite_tests: { 2023-05-31T21:35:44.861 INFO:tasks.rgw_multisite_tests: "key": 3, 2023-05-31T21:35:44.861 INFO:tasks.rgw_multisite_tests: "val": { 2023-05-31T21:35:44.862 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-05-31T21:35:44.862 INFO:tasks.rgw_multisite_tests: "marker": "1_1685568489.974220_1014.1", 2023-05-31T21:35:44.862 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-05-31T21:35:44.862 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-05-31T21:35:44.862 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-05-31T21:35:44.863 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-05-31T21:28:09.974220Z", 2023-05-31T21:35:44.863 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-05-31T21:35:44.863 INFO:tasks.rgw_multisite_tests: } 2023-05-31T21:35:44.863 INFO:tasks.rgw_multisite_tests: } 2023-05-31T21:35:44.864 INFO:tasks.rgw_multisite_tests: ] 2023-05-31T21:35:44.864 INFO:tasks.rgw_multisite_tests: }, 2023-05-31T21:35:44.864 INFO:tasks.rgw_multisite_tests: "full_sync": { 2023-05-31T21:35:44.864 INFO:tasks.rgw_multisite_tests: "total": 1, 2023-05-31T21:35:44.865 INFO:tasks.rgw_multisite_tests: "complete": 1 2023-05-31T21:35:44.865 INFO:tasks.rgw_multisite_tests: }, 2023-05-31T21:35:44.865 INFO:tasks.rgw_multisite_tests: "current_time": "2023-05-31T21:29:26Z" 2023-05-31T21:35:44.865 INFO:tasks.rgw_multisite_tests:} 2023-05-31T21:35:44.865 INFO:tasks.rgw_multisite_tests: 2023-05-31T21:35:44.866 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1685566501.757183_260.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-05-31T20:55:01.757183Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1685566501.735717_251.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-05-31T20:55:01.735717Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1685562970.274572_93.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-05-31T19:56:10.274572Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1685568489.974220_1014.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-05-31T21:28:09.974220Z', 'realm_epoch': 2}}] 2023-05-31T21:35:44.866 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1685566501.757183_260.1', 1: '1_1685566501.735717_251.1', 2: '1_1685562970.274572_93.1', 3: '1_1685568561.112279_1036.1'} 2023-05-31T21:35:44.866 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1685566501.757183_260.1', 1: '1_1685566501.735717_251.1', 2: '1_1685562970.274572_93.1', 3: '1_1685568489.974220_1014.1'} 2023-05-31T21:35:44.866 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone a2 behind master: shard=3 master=1_1685568561.112279_1036.1 target=1_1685568489.974220_1014.1 2023-05-31T21:35:44.867 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-31T21:35:44.867 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-31T21:35:44.867 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-05-31T21:35:44.867 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: ignoring --setuser ceph since I am not root 2023-05-31T21:35:44.868 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-05-31T21:35:44.868 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: 2023-05-31T21:29:31.706+0000 7f66ace9a780 20 RGW-SYNC:meta: read sync status 2023-05-31T21:35:44.868 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: 2023-05-31T21:29:31.707+0000 7f66ace9a780 20 RGW-SYNC:meta: read sync status 2023-05-31T21:35:44.868 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2023-05-31T21:35:44.869 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "sync_status": { 2023-05-31T21:35:44.869 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "info": { 2023-05-31T21:35:44.869 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "status": "sync", 2023-05-31T21:35:44.869 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "num_shards": 4, 2023-05-31T21:35:44.869 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "period": "2e6123d2-d695-4350-9a01-61a455bc782b", 2023-05-31T21:35:44.870 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "realm_epoch": 2 2023-05-31T21:35:44.870 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2023-05-31T21:35:44.870 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "markers": [ 2023-05-31T21:35:44.870 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2023-05-31T21:35:44.871 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "key": 0, 2023-05-31T21:35:44.871 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "val": { 2023-05-31T21:35:44.871 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "state": 1, 2023-05-31T21:35:44.871 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "marker": "1_1685566501.757183_260.1", 2023-05-31T21:35:44.871 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "next_step_marker": "", 2023-05-31T21:35:44.872 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "total_entries": 1, 2023-05-31T21:35:44.872 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "pos": 0, 2023-05-31T21:35:44.872 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "timestamp": "2023-05-31T20:55:01.757183Z", 2023-05-31T21:35:44.872 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "realm_epoch": 2 2023-05-31T21:35:44.873 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2023-05-31T21:35:44.873 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2023-05-31T21:35:44.873 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2023-05-31T21:35:44.873 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "key": 1, 2023-05-31T21:35:44.874 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "val": { 2023-05-31T21:35:44.874 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "state": 1, 2023-05-31T21:35:44.874 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "marker": "1_1685566501.735717_251.1", 2023-05-31T21:35:44.874 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "next_step_marker": "", 2023-05-31T21:35:44.874 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "total_entries": 0, 2023-05-31T21:35:44.875 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "pos": 0, 2023-05-31T21:35:44.875 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "timestamp": "2023-05-31T20:55:01.735717Z", 2023-05-31T21:35:44.875 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "realm_epoch": 2 2023-05-31T21:35:44.875 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2023-05-31T21:35:44.876 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2023-05-31T21:35:44.876 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2023-05-31T21:35:44.876 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "key": 2, 2023-05-31T21:35:44.876 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "val": { 2023-05-31T21:35:44.877 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "state": 1, 2023-05-31T21:35:44.877 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "marker": "1_1685562970.274572_93.1", 2023-05-31T21:35:44.877 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "next_step_marker": "", 2023-05-31T21:35:44.877 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "total_entries": 0, 2023-05-31T21:35:44.877 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "pos": 0, 2023-05-31T21:35:44.878 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "timestamp": "2023-05-31T19:56:10.274572Z", 2023-05-31T21:35:44.878 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "realm_epoch": 2 2023-05-31T21:35:44.878 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2023-05-31T21:35:44.878 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2023-05-31T21:35:44.879 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2023-05-31T21:35:44.879 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "key": 3, 2023-05-31T21:35:44.879 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "val": { 2023-05-31T21:35:44.879 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "state": 1, 2023-05-31T21:35:44.880 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "marker": "1_1685568489.974220_1014.1", 2023-05-31T21:35:44.880 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "next_step_marker": "", 2023-05-31T21:35:44.880 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "total_entries": 0, 2023-05-31T21:35:44.880 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "pos": 0, 2023-05-31T21:35:44.880 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "timestamp": "2023-05-31T21:28:09.974220Z", 2023-05-31T21:35:44.881 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "realm_epoch": 2 2023-05-31T21:35:44.881 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2023-05-31T21:35:44.881 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2023-05-31T21:35:44.881 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: ] 2023-05-31T21:35:44.882 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2023-05-31T21:35:44.882 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "full_sync": { 2023-05-31T21:35:44.882 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "total": 1, 2023-05-31T21:35:44.882 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "complete": 1 2023-05-31T21:35:44.883 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2023-05-31T21:35:44.883 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "current_time": "2023-05-31T21:29:31Z" 2023-05-31T21:35:44.883 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2023-05-31T21:35:44.883 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2023-05-31T21:35:44.883 INFO:tasks.rgw_multisite_tests: "sync_status": { 2023-05-31T21:35:44.884 INFO:tasks.rgw_multisite_tests: "info": { 2023-05-31T21:35:44.884 INFO:tasks.rgw_multisite_tests: "status": "sync", 2023-05-31T21:35:44.884 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2023-05-31T21:35:44.884 INFO:tasks.rgw_multisite_tests: "period": "2e6123d2-d695-4350-9a01-61a455bc782b", 2023-05-31T21:35:44.885 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-05-31T21:35:44.885 INFO:tasks.rgw_multisite_tests: }, 2023-05-31T21:35:44.885 INFO:tasks.rgw_multisite_tests: "markers": [ 2023-05-31T21:35:44.885 INFO:tasks.rgw_multisite_tests: { 2023-05-31T21:35:44.886 INFO:tasks.rgw_multisite_tests: "key": 0, 2023-05-31T21:35:44.886 INFO:tasks.rgw_multisite_tests: "val": { 2023-05-31T21:35:44.886 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-05-31T21:35:44.886 INFO:tasks.rgw_multisite_tests: "marker": "1_1685566501.757183_260.1", 2023-05-31T21:35:44.886 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-05-31T21:35:44.887 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2023-05-31T21:35:44.887 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-05-31T21:35:44.887 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-05-31T20:55:01.757183Z", 2023-05-31T21:35:44.887 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-05-31T21:35:44.888 INFO:tasks.rgw_multisite_tests: } 2023-05-31T21:35:44.888 INFO:tasks.rgw_multisite_tests: }, 2023-05-31T21:35:44.888 INFO:tasks.rgw_multisite_tests: { 2023-05-31T21:35:44.888 INFO:tasks.rgw_multisite_tests: "key": 1, 2023-05-31T21:35:44.889 INFO:tasks.rgw_multisite_tests: "val": { 2023-05-31T21:35:44.889 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-05-31T21:35:44.889 INFO:tasks.rgw_multisite_tests: "marker": "1_1685566501.735717_251.1", 2023-05-31T21:35:44.889 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-05-31T21:35:44.889 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-05-31T21:35:44.890 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-05-31T21:35:44.890 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-05-31T20:55:01.735717Z", 2023-05-31T21:35:44.890 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-05-31T21:35:44.890 INFO:tasks.rgw_multisite_tests: } 2023-05-31T21:35:44.891 INFO:tasks.rgw_multisite_tests: }, 2023-05-31T21:35:44.891 INFO:tasks.rgw_multisite_tests: { 2023-05-31T21:35:44.891 INFO:tasks.rgw_multisite_tests: "key": 2, 2023-05-31T21:35:44.891 INFO:tasks.rgw_multisite_tests: "val": { 2023-05-31T21:35:44.892 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-05-31T21:35:44.892 INFO:tasks.rgw_multisite_tests: "marker": "1_1685562970.274572_93.1", 2023-05-31T21:35:44.892 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-05-31T21:35:44.892 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-05-31T21:35:44.892 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-05-31T21:35:44.893 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-05-31T19:56:10.274572Z", 2023-05-31T21:35:44.893 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-05-31T21:35:44.893 INFO:tasks.rgw_multisite_tests: } 2023-05-31T21:35:44.893 INFO:tasks.rgw_multisite_tests: }, 2023-05-31T21:35:44.894 INFO:tasks.rgw_multisite_tests: { 2023-05-31T21:35:44.894 INFO:tasks.rgw_multisite_tests: "key": 3, 2023-05-31T21:35:44.894 INFO:tasks.rgw_multisite_tests: "val": { 2023-05-31T21:35:44.894 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-05-31T21:35:44.895 INFO:tasks.rgw_multisite_tests: "marker": "1_1685568489.974220_1014.1", 2023-05-31T21:35:44.895 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-05-31T21:35:44.895 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-05-31T21:35:44.895 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-05-31T21:35:44.895 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-05-31T21:28:09.974220Z", 2023-05-31T21:35:44.896 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-05-31T21:35:44.896 INFO:tasks.rgw_multisite_tests: } 2023-05-31T21:35:44.896 INFO:tasks.rgw_multisite_tests: } 2023-05-31T21:35:44.896 INFO:tasks.rgw_multisite_tests: ] 2023-05-31T21:35:44.897 INFO:tasks.rgw_multisite_tests: }, 2023-05-31T21:35:44.897 INFO:tasks.rgw_multisite_tests: "full_sync": { 2023-05-31T21:35:44.897 INFO:tasks.rgw_multisite_tests: "total": 1, 2023-05-31T21:35:44.897 INFO:tasks.rgw_multisite_tests: "complete": 1 2023-05-31T21:35:44.897 INFO:tasks.rgw_multisite_tests: }, 2023-05-31T21:35:44.898 INFO:tasks.rgw_multisite_tests: "current_time": "2023-05-31T21:29:31Z" 2023-05-31T21:35:44.898 INFO:tasks.rgw_multisite_tests:} 2023-05-31T21:35:44.898 INFO:tasks.rgw_multisite_tests: 2023-05-31T21:35:44.898 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1685566501.757183_260.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-05-31T20:55:01.757183Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1685566501.735717_251.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-05-31T20:55:01.735717Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1685562970.274572_93.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-05-31T19:56:10.274572Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1685568489.974220_1014.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-05-31T21:28:09.974220Z', 'realm_epoch': 2}}] 2023-05-31T21:35:44.899 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1685566501.757183_260.1', 1: '1_1685566501.735717_251.1', 2: '1_1685562970.274572_93.1', 3: '1_1685568561.112279_1036.1'} 2023-05-31T21:35:44.899 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1685566501.757183_260.1', 1: '1_1685566501.735717_251.1', 2: '1_1685562970.274572_93.1', 3: '1_1685568489.974220_1014.1'} 2023-05-31T21:35:44.899 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone a2 behind master: shard=3 master=1_1685568561.112279_1036.1 target=1_1685568489.974220_1014.1 2023-05-31T21:35:44.899 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-31T21:35:44.900 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-31T21:35:44.900 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-05-31T21:35:44.900 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: ignoring --setuser ceph since I am not root 2023-05-31T21:35:44.900 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-05-31T21:35:44.901 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: 2023-05-31T21:29:36.864+0000 7f2380a93780 20 RGW-SYNC:meta: read sync status 2023-05-31T21:35:44.901 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: 2023-05-31T21:29:36.865+0000 7f2380a93780 20 RGW-SYNC:meta: read sync status 2023-05-31T21:35:44.901 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2023-05-31T21:35:44.901 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "sync_status": { 2023-05-31T21:35:44.901 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "info": { 2023-05-31T21:35:44.902 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "status": "sync", 2023-05-31T21:35:44.902 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "num_shards": 4, 2023-05-31T21:35:44.902 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "period": "2e6123d2-d695-4350-9a01-61a455bc782b", 2023-05-31T21:35:44.902 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "realm_epoch": 2 2023-05-31T21:35:44.903 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2023-05-31T21:35:44.903 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "markers": [ 2023-05-31T21:35:44.903 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2023-05-31T21:35:44.903 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "key": 0, 2023-05-31T21:35:44.904 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "val": { 2023-05-31T21:35:44.904 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "state": 1, 2023-05-31T21:35:44.904 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "marker": "1_1685566501.757183_260.1", 2023-05-31T21:35:44.904 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "next_step_marker": "", 2023-05-31T21:35:44.904 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "total_entries": 1, 2023-05-31T21:35:44.905 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "pos": 0, 2023-05-31T21:35:44.905 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "timestamp": "2023-05-31T20:55:01.757183Z", 2023-05-31T21:35:44.905 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "realm_epoch": 2 2023-05-31T21:35:44.905 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2023-05-31T21:35:44.906 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2023-05-31T21:35:44.906 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2023-05-31T21:35:44.906 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "key": 1, 2023-05-31T21:35:44.906 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "val": { 2023-05-31T21:35:44.907 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "state": 1, 2023-05-31T21:35:44.907 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "marker": "1_1685566501.735717_251.1", 2023-05-31T21:35:44.907 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "next_step_marker": "", 2023-05-31T21:35:44.907 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "total_entries": 0, 2023-05-31T21:35:44.907 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "pos": 0, 2023-05-31T21:35:44.908 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "timestamp": "2023-05-31T20:55:01.735717Z", 2023-05-31T21:35:44.908 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "realm_epoch": 2 2023-05-31T21:35:44.908 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2023-05-31T21:35:44.908 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2023-05-31T21:35:44.909 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2023-05-31T21:35:44.909 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "key": 2, 2023-05-31T21:35:44.909 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "val": { 2023-05-31T21:35:44.909 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "state": 1, 2023-05-31T21:35:44.910 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "marker": "1_1685562970.274572_93.1", 2023-05-31T21:35:44.910 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "next_step_marker": "", 2023-05-31T21:35:44.910 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "total_entries": 0, 2023-05-31T21:35:44.910 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "pos": 0, 2023-05-31T21:35:44.910 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "timestamp": "2023-05-31T19:56:10.274572Z", 2023-05-31T21:35:44.911 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "realm_epoch": 2 2023-05-31T21:35:44.911 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2023-05-31T21:35:44.911 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2023-05-31T21:35:44.911 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2023-05-31T21:35:44.912 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "key": 3, 2023-05-31T21:35:44.912 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "val": { 2023-05-31T21:35:44.912 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "state": 1, 2023-05-31T21:35:44.912 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "marker": "1_1685568489.974220_1014.1", 2023-05-31T21:35:44.913 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "next_step_marker": "", 2023-05-31T21:35:44.913 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "total_entries": 0, 2023-05-31T21:35:44.913 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "pos": 0, 2023-05-31T21:35:44.913 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "timestamp": "2023-05-31T21:28:09.974220Z", 2023-05-31T21:35:44.913 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "realm_epoch": 2 2023-05-31T21:35:44.914 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2023-05-31T21:35:44.914 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2023-05-31T21:35:44.914 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: ] 2023-05-31T21:35:44.914 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2023-05-31T21:35:44.915 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "full_sync": { 2023-05-31T21:35:44.915 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "total": 1, 2023-05-31T21:35:44.915 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "complete": 1 2023-05-31T21:35:44.915 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2023-05-31T21:35:44.916 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "current_time": "2023-05-31T21:29:36Z" 2023-05-31T21:35:44.916 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2023-05-31T21:35:44.916 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2023-05-31T21:35:44.916 INFO:tasks.rgw_multisite_tests: "sync_status": { 2023-05-31T21:35:44.916 INFO:tasks.rgw_multisite_tests: "info": { 2023-05-31T21:35:44.917 INFO:tasks.rgw_multisite_tests: "status": "sync", 2023-05-31T21:35:44.917 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2023-05-31T21:35:44.917 INFO:tasks.rgw_multisite_tests: "period": "2e6123d2-d695-4350-9a01-61a455bc782b", 2023-05-31T21:35:44.917 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-05-31T21:35:44.918 INFO:tasks.rgw_multisite_tests: }, 2023-05-31T21:35:44.918 INFO:tasks.rgw_multisite_tests: "markers": [ 2023-05-31T21:35:44.918 INFO:tasks.rgw_multisite_tests: { 2023-05-31T21:35:44.918 INFO:tasks.rgw_multisite_tests: "key": 0, 2023-05-31T21:35:44.919 INFO:tasks.rgw_multisite_tests: "val": { 2023-05-31T21:35:44.919 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-05-31T21:35:44.919 INFO:tasks.rgw_multisite_tests: "marker": "1_1685566501.757183_260.1", 2023-05-31T21:35:44.919 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-05-31T21:35:44.919 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2023-05-31T21:35:44.920 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-05-31T21:35:44.920 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-05-31T20:55:01.757183Z", 2023-05-31T21:35:44.920 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-05-31T21:35:44.920 INFO:tasks.rgw_multisite_tests: } 2023-05-31T21:35:44.921 INFO:tasks.rgw_multisite_tests: }, 2023-05-31T21:35:44.921 INFO:tasks.rgw_multisite_tests: { 2023-05-31T21:35:44.921 INFO:tasks.rgw_multisite_tests: "key": 1, 2023-05-31T21:35:44.921 INFO:tasks.rgw_multisite_tests: "val": { 2023-05-31T21:35:44.922 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-05-31T21:35:44.922 INFO:tasks.rgw_multisite_tests: "marker": "1_1685566501.735717_251.1", 2023-05-31T21:35:44.922 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-05-31T21:35:44.922 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-05-31T21:35:44.922 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-05-31T21:35:44.923 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-05-31T20:55:01.735717Z", 2023-05-31T21:35:44.923 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-05-31T21:35:44.923 INFO:tasks.rgw_multisite_tests: } 2023-05-31T21:35:44.923 INFO:tasks.rgw_multisite_tests: }, 2023-05-31T21:35:44.924 INFO:tasks.rgw_multisite_tests: { 2023-05-31T21:35:44.924 INFO:tasks.rgw_multisite_tests: "key": 2, 2023-05-31T21:35:44.924 INFO:tasks.rgw_multisite_tests: "val": { 2023-05-31T21:35:44.924 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-05-31T21:35:44.924 INFO:tasks.rgw_multisite_tests: "marker": "1_1685562970.274572_93.1", 2023-05-31T21:35:44.925 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-05-31T21:35:44.925 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-05-31T21:35:44.925 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-05-31T21:35:44.925 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-05-31T19:56:10.274572Z", 2023-05-31T21:35:44.926 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-05-31T21:35:44.926 INFO:tasks.rgw_multisite_tests: } 2023-05-31T21:35:44.926 INFO:tasks.rgw_multisite_tests: }, 2023-05-31T21:35:44.926 INFO:tasks.rgw_multisite_tests: { 2023-05-31T21:35:44.927 INFO:tasks.rgw_multisite_tests: "key": 3, 2023-05-31T21:35:44.927 INFO:tasks.rgw_multisite_tests: "val": { 2023-05-31T21:35:44.927 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-05-31T21:35:44.927 INFO:tasks.rgw_multisite_tests: "marker": "1_1685568489.974220_1014.1", 2023-05-31T21:35:44.927 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-05-31T21:35:44.928 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-05-31T21:35:44.928 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-05-31T21:35:44.928 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-05-31T21:28:09.974220Z", 2023-05-31T21:35:44.928 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-05-31T21:35:44.929 INFO:tasks.rgw_multisite_tests: } 2023-05-31T21:35:44.929 INFO:tasks.rgw_multisite_tests: } 2023-05-31T21:35:44.929 INFO:tasks.rgw_multisite_tests: ] 2023-05-31T21:35:44.929 INFO:tasks.rgw_multisite_tests: }, 2023-05-31T21:35:44.930 INFO:tasks.rgw_multisite_tests: "full_sync": { 2023-05-31T21:35:44.930 INFO:tasks.rgw_multisite_tests: "total": 1, 2023-05-31T21:35:44.930 INFO:tasks.rgw_multisite_tests: "complete": 1 2023-05-31T21:35:44.930 INFO:tasks.rgw_multisite_tests: }, 2023-05-31T21:35:44.930 INFO:tasks.rgw_multisite_tests: "current_time": "2023-05-31T21:29:36Z" 2023-05-31T21:35:44.931 INFO:tasks.rgw_multisite_tests:} 2023-05-31T21:35:44.931 INFO:tasks.rgw_multisite_tests: 2023-05-31T21:35:44.931 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1685566501.757183_260.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-05-31T20:55:01.757183Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1685566501.735717_251.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-05-31T20:55:01.735717Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1685562970.274572_93.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-05-31T19:56:10.274572Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1685568489.974220_1014.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-05-31T21:28:09.974220Z', 'realm_epoch': 2}}] 2023-05-31T21:35:44.931 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1685566501.757183_260.1', 1: '1_1685566501.735717_251.1', 2: '1_1685562970.274572_93.1', 3: '1_1685568561.112279_1036.1'} 2023-05-31T21:35:44.932 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1685566501.757183_260.1', 1: '1_1685566501.735717_251.1', 2: '1_1685562970.274572_93.1', 3: '1_1685568489.974220_1014.1'} 2023-05-31T21:35:44.932 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone a2 behind master: shard=3 master=1_1685568561.112279_1036.1 target=1_1685568489.974220_1014.1 2023-05-31T21:35:44.932 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-31T21:35:44.932 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-31T21:35:44.933 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-05-31T21:35:44.933 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: ignoring --setuser ceph since I am not root 2023-05-31T21:35:44.933 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-05-31T21:35:44.933 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: 2023-05-31T21:29:41.992+0000 7f5637efe780 20 RGW-SYNC:meta: read sync status 2023-05-31T21:35:44.933 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: 2023-05-31T21:29:41.993+0000 7f5637efe780 20 RGW-SYNC:meta: read sync status 2023-05-31T21:35:44.934 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2023-05-31T21:35:44.934 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "sync_status": { 2023-05-31T21:35:44.934 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "info": { 2023-05-31T21:35:44.934 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "status": "sync", 2023-05-31T21:35:44.935 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "num_shards": 4, 2023-05-31T21:35:44.935 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "period": "2e6123d2-d695-4350-9a01-61a455bc782b", 2023-05-31T21:35:44.935 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "realm_epoch": 2 2023-05-31T21:35:44.935 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2023-05-31T21:35:44.936 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "markers": [ 2023-05-31T21:35:44.936 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2023-05-31T21:35:44.936 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "key": 0, 2023-05-31T21:35:44.936 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "val": { 2023-05-31T21:35:44.936 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "state": 1, 2023-05-31T21:35:44.937 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "marker": "1_1685566501.757183_260.1", 2023-05-31T21:35:44.937 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "next_step_marker": "", 2023-05-31T21:35:44.937 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "total_entries": 1, 2023-05-31T21:35:44.937 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "pos": 0, 2023-05-31T21:35:44.938 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "timestamp": "2023-05-31T20:55:01.757183Z", 2023-05-31T21:35:44.938 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "realm_epoch": 2 2023-05-31T21:35:44.938 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2023-05-31T21:35:44.938 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2023-05-31T21:35:44.939 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2023-05-31T21:35:44.939 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "key": 1, 2023-05-31T21:35:44.939 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "val": { 2023-05-31T21:35:44.939 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "state": 1, 2023-05-31T21:35:44.939 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "marker": "1_1685566501.735717_251.1", 2023-05-31T21:35:44.940 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "next_step_marker": "", 2023-05-31T21:35:44.940 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "total_entries": 0, 2023-05-31T21:35:44.940 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "pos": 0, 2023-05-31T21:35:44.940 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "timestamp": "2023-05-31T20:55:01.735717Z", 2023-05-31T21:35:44.941 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "realm_epoch": 2 2023-05-31T21:35:44.941 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2023-05-31T21:35:44.941 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2023-05-31T21:35:44.941 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2023-05-31T21:35:44.942 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "key": 2, 2023-05-31T21:35:44.942 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "val": { 2023-05-31T21:35:44.942 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "state": 1, 2023-05-31T21:35:44.942 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "marker": "1_1685562970.274572_93.1", 2023-05-31T21:35:44.942 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "next_step_marker": "", 2023-05-31T21:35:44.943 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "total_entries": 0, 2023-05-31T21:35:44.943 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "pos": 0, 2023-05-31T21:35:44.943 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "timestamp": "2023-05-31T19:56:10.274572Z", 2023-05-31T21:35:44.943 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "realm_epoch": 2 2023-05-31T21:35:44.944 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2023-05-31T21:35:44.944 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2023-05-31T21:35:44.944 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2023-05-31T21:35:44.944 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "key": 3, 2023-05-31T21:35:44.945 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "val": { 2023-05-31T21:35:44.945 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "state": 1, 2023-05-31T21:35:44.945 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "marker": "1_1685568489.974220_1014.1", 2023-05-31T21:35:44.945 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "next_step_marker": "", 2023-05-31T21:35:44.945 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "total_entries": 0, 2023-05-31T21:35:44.946 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "pos": 0, 2023-05-31T21:35:44.946 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "timestamp": "2023-05-31T21:28:09.974220Z", 2023-05-31T21:35:44.946 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "realm_epoch": 2 2023-05-31T21:35:44.946 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2023-05-31T21:35:44.947 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2023-05-31T21:35:44.947 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: ] 2023-05-31T21:35:44.947 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2023-05-31T21:35:44.947 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "full_sync": { 2023-05-31T21:35:44.948 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "total": 1, 2023-05-31T21:35:44.948 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "complete": 1 2023-05-31T21:35:44.948 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2023-05-31T21:35:44.948 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "current_time": "2023-05-31T21:29:41Z" 2023-05-31T21:35:44.948 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2023-05-31T21:35:44.949 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2023-05-31T21:35:44.949 INFO:tasks.rgw_multisite_tests: "sync_status": { 2023-05-31T21:35:44.949 INFO:tasks.rgw_multisite_tests: "info": { 2023-05-31T21:35:44.949 INFO:tasks.rgw_multisite_tests: "status": "sync", 2023-05-31T21:35:44.950 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2023-05-31T21:35:44.950 INFO:tasks.rgw_multisite_tests: "period": "2e6123d2-d695-4350-9a01-61a455bc782b", 2023-05-31T21:35:44.950 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-05-31T21:35:44.950 INFO:tasks.rgw_multisite_tests: }, 2023-05-31T21:35:44.951 INFO:tasks.rgw_multisite_tests: "markers": [ 2023-05-31T21:35:44.951 INFO:tasks.rgw_multisite_tests: { 2023-05-31T21:35:44.951 INFO:tasks.rgw_multisite_tests: "key": 0, 2023-05-31T21:35:44.951 INFO:tasks.rgw_multisite_tests: "val": { 2023-05-31T21:35:44.951 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-05-31T21:35:44.952 INFO:tasks.rgw_multisite_tests: "marker": "1_1685566501.757183_260.1", 2023-05-31T21:35:44.952 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-05-31T21:35:44.952 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2023-05-31T21:35:44.952 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-05-31T21:35:44.953 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-05-31T20:55:01.757183Z", 2023-05-31T21:35:44.953 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-05-31T21:35:44.953 INFO:tasks.rgw_multisite_tests: } 2023-05-31T21:35:44.953 INFO:tasks.rgw_multisite_tests: }, 2023-05-31T21:35:44.954 INFO:tasks.rgw_multisite_tests: { 2023-05-31T21:35:44.954 INFO:tasks.rgw_multisite_tests: "key": 1, 2023-05-31T21:35:44.954 INFO:tasks.rgw_multisite_tests: "val": { 2023-05-31T21:35:44.954 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-05-31T21:35:44.954 INFO:tasks.rgw_multisite_tests: "marker": "1_1685566501.735717_251.1", 2023-05-31T21:35:44.955 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-05-31T21:35:44.955 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-05-31T21:35:44.955 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-05-31T21:35:44.955 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-05-31T20:55:01.735717Z", 2023-05-31T21:35:44.956 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-05-31T21:35:44.956 INFO:tasks.rgw_multisite_tests: } 2023-05-31T21:35:44.956 INFO:tasks.rgw_multisite_tests: }, 2023-05-31T21:35:44.956 INFO:tasks.rgw_multisite_tests: { 2023-05-31T21:35:44.957 INFO:tasks.rgw_multisite_tests: "key": 2, 2023-05-31T21:35:44.957 INFO:tasks.rgw_multisite_tests: "val": { 2023-05-31T21:35:44.957 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-05-31T21:35:44.957 INFO:tasks.rgw_multisite_tests: "marker": "1_1685562970.274572_93.1", 2023-05-31T21:35:44.957 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-05-31T21:35:44.958 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-05-31T21:35:44.958 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-05-31T21:35:44.958 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-05-31T19:56:10.274572Z", 2023-05-31T21:35:44.958 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-05-31T21:35:44.959 INFO:tasks.rgw_multisite_tests: } 2023-05-31T21:35:44.959 INFO:tasks.rgw_multisite_tests: }, 2023-05-31T21:35:44.959 INFO:tasks.rgw_multisite_tests: { 2023-05-31T21:35:44.959 INFO:tasks.rgw_multisite_tests: "key": 3, 2023-05-31T21:35:44.959 INFO:tasks.rgw_multisite_tests: "val": { 2023-05-31T21:35:44.960 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-05-31T21:35:44.960 INFO:tasks.rgw_multisite_tests: "marker": "1_1685568489.974220_1014.1", 2023-05-31T21:35:44.960 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-05-31T21:35:44.960 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-05-31T21:35:44.961 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-05-31T21:35:44.961 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-05-31T21:28:09.974220Z", 2023-05-31T21:35:44.961 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-05-31T21:35:44.961 INFO:tasks.rgw_multisite_tests: } 2023-05-31T21:35:44.962 INFO:tasks.rgw_multisite_tests: } 2023-05-31T21:35:44.962 INFO:tasks.rgw_multisite_tests: ] 2023-05-31T21:35:44.962 INFO:tasks.rgw_multisite_tests: }, 2023-05-31T21:35:44.962 INFO:tasks.rgw_multisite_tests: "full_sync": { 2023-05-31T21:35:44.962 INFO:tasks.rgw_multisite_tests: "total": 1, 2023-05-31T21:35:44.963 INFO:tasks.rgw_multisite_tests: "complete": 1 2023-05-31T21:35:44.963 INFO:tasks.rgw_multisite_tests: }, 2023-05-31T21:35:44.963 INFO:tasks.rgw_multisite_tests: "current_time": "2023-05-31T21:29:41Z" 2023-05-31T21:35:44.963 INFO:tasks.rgw_multisite_tests:} 2023-05-31T21:35:44.964 INFO:tasks.rgw_multisite_tests: 2023-05-31T21:35:44.964 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1685566501.757183_260.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-05-31T20:55:01.757183Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1685566501.735717_251.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-05-31T20:55:01.735717Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1685562970.274572_93.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-05-31T19:56:10.274572Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1685568489.974220_1014.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-05-31T21:28:09.974220Z', 'realm_epoch': 2}}] 2023-05-31T21:35:44.964 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1685566501.757183_260.1', 1: '1_1685566501.735717_251.1', 2: '1_1685562970.274572_93.1', 3: '1_1685568561.112279_1036.1'} 2023-05-31T21:35:44.964 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1685566501.757183_260.1', 1: '1_1685566501.735717_251.1', 2: '1_1685562970.274572_93.1', 3: '1_1685568489.974220_1014.1'} 2023-05-31T21:35:44.965 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone a2 behind master: shard=3 master=1_1685568561.112279_1036.1 target=1_1685568489.974220_1014.1 2023-05-31T21:35:44.965 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-31T21:35:44.965 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-31T21:35:44.965 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-05-31T21:35:44.966 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: ignoring --setuser ceph since I am not root 2023-05-31T21:35:44.966 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-05-31T21:35:44.966 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: 2023-05-31T21:29:47.132+0000 7f5934670780 20 RGW-SYNC:meta: read sync status 2023-05-31T21:35:44.966 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: 2023-05-31T21:29:47.133+0000 7f5934670780 20 RGW-SYNC:meta: read sync status 2023-05-31T21:35:44.966 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2023-05-31T21:35:44.967 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "sync_status": { 2023-05-31T21:35:44.967 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "info": { 2023-05-31T21:35:44.967 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "status": "sync", 2023-05-31T21:35:44.967 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "num_shards": 4, 2023-05-31T21:35:44.968 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "period": "2e6123d2-d695-4350-9a01-61a455bc782b", 2023-05-31T21:35:44.968 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "realm_epoch": 2 2023-05-31T21:35:44.968 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2023-05-31T21:35:44.968 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "markers": [ 2023-05-31T21:35:44.969 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2023-05-31T21:35:44.969 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "key": 0, 2023-05-31T21:35:44.969 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "val": { 2023-05-31T21:35:44.969 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "state": 1, 2023-05-31T21:35:44.969 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "marker": "1_1685566501.757183_260.1", 2023-05-31T21:35:44.970 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "next_step_marker": "", 2023-05-31T21:35:44.970 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "total_entries": 1, 2023-05-31T21:35:44.970 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "pos": 0, 2023-05-31T21:35:44.970 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "timestamp": "2023-05-31T20:55:01.757183Z", 2023-05-31T21:35:44.971 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "realm_epoch": 2 2023-05-31T21:35:44.971 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2023-05-31T21:35:44.971 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2023-05-31T21:35:44.971 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2023-05-31T21:35:44.972 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "key": 1, 2023-05-31T21:35:44.972 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "val": { 2023-05-31T21:35:44.972 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "state": 1, 2023-05-31T21:35:44.972 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "marker": "1_1685566501.735717_251.1", 2023-05-31T21:35:44.972 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "next_step_marker": "", 2023-05-31T21:35:44.973 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "total_entries": 0, 2023-05-31T21:35:44.973 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "pos": 0, 2023-05-31T21:35:44.973 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "timestamp": "2023-05-31T20:55:01.735717Z", 2023-05-31T21:35:44.973 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "realm_epoch": 2 2023-05-31T21:35:44.974 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2023-05-31T21:35:44.974 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2023-05-31T21:35:44.974 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2023-05-31T21:35:44.974 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "key": 2, 2023-05-31T21:35:44.975 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "val": { 2023-05-31T21:35:44.975 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "state": 1, 2023-05-31T21:35:44.975 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "marker": "1_1685562970.274572_93.1", 2023-05-31T21:35:44.975 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "next_step_marker": "", 2023-05-31T21:35:44.975 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "total_entries": 0, 2023-05-31T21:35:44.976 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "pos": 0, 2023-05-31T21:35:44.976 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "timestamp": "2023-05-31T19:56:10.274572Z", 2023-05-31T21:35:44.976 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "realm_epoch": 2 2023-05-31T21:35:44.976 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2023-05-31T21:35:44.977 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2023-05-31T21:35:44.977 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2023-05-31T21:35:44.977 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "key": 3, 2023-05-31T21:35:44.977 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "val": { 2023-05-31T21:35:44.978 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "state": 1, 2023-05-31T21:35:44.978 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "marker": "1_1685568489.974220_1014.1", 2023-05-31T21:35:44.978 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "next_step_marker": "", 2023-05-31T21:35:44.978 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "total_entries": 0, 2023-05-31T21:35:44.978 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "pos": 0, 2023-05-31T21:35:44.979 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "timestamp": "2023-05-31T21:28:09.974220Z", 2023-05-31T21:35:44.979 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "realm_epoch": 2 2023-05-31T21:35:44.979 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2023-05-31T21:35:44.979 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2023-05-31T21:35:44.980 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: ] 2023-05-31T21:35:44.980 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2023-05-31T21:35:44.980 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "full_sync": { 2023-05-31T21:35:44.980 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "total": 1, 2023-05-31T21:35:44.981 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "complete": 1 2023-05-31T21:35:44.981 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2023-05-31T21:35:44.981 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "current_time": "2023-05-31T21:29:47Z" 2023-05-31T21:35:44.981 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2023-05-31T21:35:44.981 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2023-05-31T21:35:44.982 INFO:tasks.rgw_multisite_tests: "sync_status": { 2023-05-31T21:35:44.982 INFO:tasks.rgw_multisite_tests: "info": { 2023-05-31T21:35:44.982 INFO:tasks.rgw_multisite_tests: "status": "sync", 2023-05-31T21:35:44.982 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2023-05-31T21:35:44.983 INFO:tasks.rgw_multisite_tests: "period": "2e6123d2-d695-4350-9a01-61a455bc782b", 2023-05-31T21:35:44.983 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-05-31T21:35:44.983 INFO:tasks.rgw_multisite_tests: }, 2023-05-31T21:35:44.983 INFO:tasks.rgw_multisite_tests: "markers": [ 2023-05-31T21:35:44.984 INFO:tasks.rgw_multisite_tests: { 2023-05-31T21:35:44.984 INFO:tasks.rgw_multisite_tests: "key": 0, 2023-05-31T21:35:44.984 INFO:tasks.rgw_multisite_tests: "val": { 2023-05-31T21:35:44.984 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-05-31T21:35:44.984 INFO:tasks.rgw_multisite_tests: "marker": "1_1685566501.757183_260.1", 2023-05-31T21:35:44.985 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-05-31T21:35:44.985 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2023-05-31T21:35:44.985 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-05-31T21:35:44.985 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-05-31T20:55:01.757183Z", 2023-05-31T21:35:44.986 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-05-31T21:35:44.986 INFO:tasks.rgw_multisite_tests: } 2023-05-31T21:35:44.986 INFO:tasks.rgw_multisite_tests: }, 2023-05-31T21:35:44.986 INFO:tasks.rgw_multisite_tests: { 2023-05-31T21:35:44.987 INFO:tasks.rgw_multisite_tests: "key": 1, 2023-05-31T21:35:44.987 INFO:tasks.rgw_multisite_tests: "val": { 2023-05-31T21:35:44.987 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-05-31T21:35:44.987 INFO:tasks.rgw_multisite_tests: "marker": "1_1685566501.735717_251.1", 2023-05-31T21:35:44.988 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-05-31T21:35:44.988 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-05-31T21:35:44.988 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-05-31T21:35:44.988 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-05-31T20:55:01.735717Z", 2023-05-31T21:35:44.988 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-05-31T21:35:44.989 INFO:tasks.rgw_multisite_tests: } 2023-05-31T21:35:44.989 INFO:tasks.rgw_multisite_tests: }, 2023-05-31T21:35:44.989 INFO:tasks.rgw_multisite_tests: { 2023-05-31T21:35:44.989 INFO:tasks.rgw_multisite_tests: "key": 2, 2023-05-31T21:35:44.990 INFO:tasks.rgw_multisite_tests: "val": { 2023-05-31T21:35:44.990 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-05-31T21:35:44.990 INFO:tasks.rgw_multisite_tests: "marker": "1_1685562970.274572_93.1", 2023-05-31T21:35:44.990 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-05-31T21:35:44.991 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-05-31T21:35:44.991 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-05-31T21:35:44.991 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-05-31T19:56:10.274572Z", 2023-05-31T21:35:44.991 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-05-31T21:35:44.991 INFO:tasks.rgw_multisite_tests: } 2023-05-31T21:35:44.992 INFO:tasks.rgw_multisite_tests: }, 2023-05-31T21:35:44.992 INFO:tasks.rgw_multisite_tests: { 2023-05-31T21:35:44.992 INFO:tasks.rgw_multisite_tests: "key": 3, 2023-05-31T21:35:44.992 INFO:tasks.rgw_multisite_tests: "val": { 2023-05-31T21:35:44.993 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-05-31T21:35:44.993 INFO:tasks.rgw_multisite_tests: "marker": "1_1685568489.974220_1014.1", 2023-05-31T21:35:44.993 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-05-31T21:35:44.993 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-05-31T21:35:44.994 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-05-31T21:35:44.994 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-05-31T21:28:09.974220Z", 2023-05-31T21:35:44.994 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-05-31T21:35:44.994 INFO:tasks.rgw_multisite_tests: } 2023-05-31T21:35:44.994 INFO:tasks.rgw_multisite_tests: } 2023-05-31T21:35:44.995 INFO:tasks.rgw_multisite_tests: ] 2023-05-31T21:35:44.995 INFO:tasks.rgw_multisite_tests: }, 2023-05-31T21:35:44.995 INFO:tasks.rgw_multisite_tests: "full_sync": { 2023-05-31T21:35:44.995 INFO:tasks.rgw_multisite_tests: "total": 1, 2023-05-31T21:35:44.996 INFO:tasks.rgw_multisite_tests: "complete": 1 2023-05-31T21:35:44.996 INFO:tasks.rgw_multisite_tests: }, 2023-05-31T21:35:44.996 INFO:tasks.rgw_multisite_tests: "current_time": "2023-05-31T21:29:47Z" 2023-05-31T21:35:44.996 INFO:tasks.rgw_multisite_tests:} 2023-05-31T21:35:44.996 INFO:tasks.rgw_multisite_tests: 2023-05-31T21:35:44.997 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1685566501.757183_260.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-05-31T20:55:01.757183Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1685566501.735717_251.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-05-31T20:55:01.735717Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1685562970.274572_93.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-05-31T19:56:10.274572Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1685568489.974220_1014.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-05-31T21:28:09.974220Z', 'realm_epoch': 2}}] 2023-05-31T21:35:44.997 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1685566501.757183_260.1', 1: '1_1685566501.735717_251.1', 2: '1_1685562970.274572_93.1', 3: '1_1685568561.112279_1036.1'} 2023-05-31T21:35:44.997 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1685566501.757183_260.1', 1: '1_1685566501.735717_251.1', 2: '1_1685562970.274572_93.1', 3: '1_1685568489.974220_1014.1'} 2023-05-31T21:35:44.997 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone a2 behind master: shard=3 master=1_1685568561.112279_1036.1 target=1_1685568489.974220_1014.1 2023-05-31T21:35:44.998 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-31T21:35:44.998 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-31T21:35:44.998 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-05-31T21:35:44.998 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: ignoring --setuser ceph since I am not root 2023-05-31T21:35:44.999 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-05-31T21:35:44.999 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: 2023-05-31T21:29:52.272+0000 7ff01e920780 20 RGW-SYNC:meta: read sync status 2023-05-31T21:35:44.999 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: 2023-05-31T21:29:52.273+0000 7ff01e920780 20 RGW-SYNC:meta: read sync status 2023-05-31T21:35:44.999 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2023-05-31T21:35:45.000 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "sync_status": { 2023-05-31T21:35:45.000 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "info": { 2023-05-31T21:35:45.000 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "status": "sync", 2023-05-31T21:35:45.000 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "num_shards": 4, 2023-05-31T21:35:45.000 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "period": "2e6123d2-d695-4350-9a01-61a455bc782b", 2023-05-31T21:35:45.001 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "realm_epoch": 2 2023-05-31T21:35:45.001 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2023-05-31T21:35:45.001 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "markers": [ 2023-05-31T21:35:45.001 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2023-05-31T21:35:45.002 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "key": 0, 2023-05-31T21:35:45.002 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "val": { 2023-05-31T21:35:45.002 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "state": 1, 2023-05-31T21:35:45.002 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "marker": "1_1685566501.757183_260.1", 2023-05-31T21:35:45.003 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "next_step_marker": "", 2023-05-31T21:35:45.003 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "total_entries": 1, 2023-05-31T21:35:45.003 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "pos": 0, 2023-05-31T21:35:45.003 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "timestamp": "2023-05-31T20:55:01.757183Z", 2023-05-31T21:35:45.003 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "realm_epoch": 2 2023-05-31T21:35:45.004 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2023-05-31T21:35:45.004 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2023-05-31T21:35:45.004 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2023-05-31T21:35:45.004 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "key": 1, 2023-05-31T21:35:45.005 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "val": { 2023-05-31T21:35:45.005 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "state": 1, 2023-05-31T21:35:45.005 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "marker": "1_1685566501.735717_251.1", 2023-05-31T21:35:45.005 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "next_step_marker": "", 2023-05-31T21:35:45.006 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "total_entries": 0, 2023-05-31T21:35:45.006 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "pos": 0, 2023-05-31T21:35:45.006 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "timestamp": "2023-05-31T20:55:01.735717Z", 2023-05-31T21:35:45.006 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "realm_epoch": 2 2023-05-31T21:35:45.006 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2023-05-31T21:35:45.007 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2023-05-31T21:35:45.007 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2023-05-31T21:35:45.007 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "key": 2, 2023-05-31T21:35:45.007 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "val": { 2023-05-31T21:35:45.008 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "state": 1, 2023-05-31T21:35:45.008 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "marker": "1_1685562970.274572_93.1", 2023-05-31T21:35:45.008 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "next_step_marker": "", 2023-05-31T21:35:45.008 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "total_entries": 0, 2023-05-31T21:35:45.009 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "pos": 0, 2023-05-31T21:35:45.009 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "timestamp": "2023-05-31T19:56:10.274572Z", 2023-05-31T21:35:45.009 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "realm_epoch": 2 2023-05-31T21:35:45.009 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2023-05-31T21:35:45.009 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2023-05-31T21:35:45.010 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2023-05-31T21:35:45.010 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "key": 3, 2023-05-31T21:35:45.010 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "val": { 2023-05-31T21:35:45.010 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "state": 1, 2023-05-31T21:35:45.011 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "marker": "1_1685568560.368414_1023.1", 2023-05-31T21:35:45.011 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "next_step_marker": "", 2023-05-31T21:35:45.011 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "total_entries": 0, 2023-05-31T21:35:45.011 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "pos": 0, 2023-05-31T21:35:45.012 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "timestamp": "2023-05-31T21:29:20.368414Z", 2023-05-31T21:35:45.012 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "realm_epoch": 2 2023-05-31T21:35:45.012 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2023-05-31T21:35:45.012 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2023-05-31T21:35:45.013 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: ] 2023-05-31T21:35:45.013 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2023-05-31T21:35:45.013 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "full_sync": { 2023-05-31T21:35:45.013 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "total": 1, 2023-05-31T21:35:45.013 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "complete": 1 2023-05-31T21:35:45.014 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2023-05-31T21:35:45.014 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "current_time": "2023-05-31T21:29:52Z" 2023-05-31T21:35:45.014 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2023-05-31T21:35:45.014 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2023-05-31T21:35:45.015 INFO:tasks.rgw_multisite_tests: "sync_status": { 2023-05-31T21:35:45.015 INFO:tasks.rgw_multisite_tests: "info": { 2023-05-31T21:35:45.015 INFO:tasks.rgw_multisite_tests: "status": "sync", 2023-05-31T21:35:45.015 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2023-05-31T21:35:45.016 INFO:tasks.rgw_multisite_tests: "period": "2e6123d2-d695-4350-9a01-61a455bc782b", 2023-05-31T21:35:45.016 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-05-31T21:35:45.016 INFO:tasks.rgw_multisite_tests: }, 2023-05-31T21:35:45.016 INFO:tasks.rgw_multisite_tests: "markers": [ 2023-05-31T21:35:45.016 INFO:tasks.rgw_multisite_tests: { 2023-05-31T21:35:45.017 INFO:tasks.rgw_multisite_tests: "key": 0, 2023-05-31T21:35:45.017 INFO:tasks.rgw_multisite_tests: "val": { 2023-05-31T21:35:45.017 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-05-31T21:35:45.017 INFO:tasks.rgw_multisite_tests: "marker": "1_1685566501.757183_260.1", 2023-05-31T21:35:45.018 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-05-31T21:35:45.018 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2023-05-31T21:35:45.018 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-05-31T21:35:45.018 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-05-31T20:55:01.757183Z", 2023-05-31T21:35:45.019 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-05-31T21:35:45.019 INFO:tasks.rgw_multisite_tests: } 2023-05-31T21:35:45.019 INFO:tasks.rgw_multisite_tests: }, 2023-05-31T21:35:45.019 INFO:tasks.rgw_multisite_tests: { 2023-05-31T21:35:45.019 INFO:tasks.rgw_multisite_tests: "key": 1, 2023-05-31T21:35:45.020 INFO:tasks.rgw_multisite_tests: "val": { 2023-05-31T21:35:45.020 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-05-31T21:35:45.020 INFO:tasks.rgw_multisite_tests: "marker": "1_1685566501.735717_251.1", 2023-05-31T21:35:45.020 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-05-31T21:35:45.021 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-05-31T21:35:45.021 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-05-31T21:35:45.021 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-05-31T20:55:01.735717Z", 2023-05-31T21:35:45.021 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-05-31T21:35:45.022 INFO:tasks.rgw_multisite_tests: } 2023-05-31T21:35:45.022 INFO:tasks.rgw_multisite_tests: }, 2023-05-31T21:35:45.022 INFO:tasks.rgw_multisite_tests: { 2023-05-31T21:35:45.022 INFO:tasks.rgw_multisite_tests: "key": 2, 2023-05-31T21:35:45.022 INFO:tasks.rgw_multisite_tests: "val": { 2023-05-31T21:35:45.023 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-05-31T21:35:45.023 INFO:tasks.rgw_multisite_tests: "marker": "1_1685562970.274572_93.1", 2023-05-31T21:35:45.023 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-05-31T21:35:45.023 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-05-31T21:35:45.024 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-05-31T21:35:45.024 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-05-31T19:56:10.274572Z", 2023-05-31T21:35:45.024 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-05-31T21:35:45.024 INFO:tasks.rgw_multisite_tests: } 2023-05-31T21:35:45.024 INFO:tasks.rgw_multisite_tests: }, 2023-05-31T21:35:45.025 INFO:tasks.rgw_multisite_tests: { 2023-05-31T21:35:45.025 INFO:tasks.rgw_multisite_tests: "key": 3, 2023-05-31T21:35:45.025 INFO:tasks.rgw_multisite_tests: "val": { 2023-05-31T21:35:45.025 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-05-31T21:35:45.026 INFO:tasks.rgw_multisite_tests: "marker": "1_1685568560.368414_1023.1", 2023-05-31T21:35:45.026 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-05-31T21:35:45.026 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-05-31T21:35:45.026 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-05-31T21:35:45.027 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-05-31T21:29:20.368414Z", 2023-05-31T21:35:45.027 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-05-31T21:35:45.027 INFO:tasks.rgw_multisite_tests: } 2023-05-31T21:35:45.027 INFO:tasks.rgw_multisite_tests: } 2023-05-31T21:35:45.027 INFO:tasks.rgw_multisite_tests: ] 2023-05-31T21:35:45.028 INFO:tasks.rgw_multisite_tests: }, 2023-05-31T21:35:45.028 INFO:tasks.rgw_multisite_tests: "full_sync": { 2023-05-31T21:35:45.028 INFO:tasks.rgw_multisite_tests: "total": 1, 2023-05-31T21:35:45.028 INFO:tasks.rgw_multisite_tests: "complete": 1 2023-05-31T21:35:45.029 INFO:tasks.rgw_multisite_tests: }, 2023-05-31T21:35:45.029 INFO:tasks.rgw_multisite_tests: "current_time": "2023-05-31T21:29:52Z" 2023-05-31T21:35:45.029 INFO:tasks.rgw_multisite_tests:} 2023-05-31T21:35:45.029 INFO:tasks.rgw_multisite_tests: 2023-05-31T21:35:45.030 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1685566501.757183_260.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-05-31T20:55:01.757183Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1685566501.735717_251.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-05-31T20:55:01.735717Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1685562970.274572_93.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-05-31T19:56:10.274572Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1685568560.368414_1023.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-05-31T21:29:20.368414Z', 'realm_epoch': 2}}] 2023-05-31T21:35:45.030 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1685566501.757183_260.1', 1: '1_1685566501.735717_251.1', 2: '1_1685562970.274572_93.1', 3: '1_1685568561.112279_1036.1'} 2023-05-31T21:35:45.030 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1685566501.757183_260.1', 1: '1_1685566501.735717_251.1', 2: '1_1685562970.274572_93.1', 3: '1_1685568560.368414_1023.1'} 2023-05-31T21:35:45.030 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone a2 behind master: shard=3 master=1_1685568561.112279_1036.1 target=1_1685568560.368414_1023.1 2023-05-31T21:35:45.031 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-31T21:35:45.031 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-31T21:35:45.031 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-05-31T21:35:45.031 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: ignoring --setuser ceph since I am not root 2023-05-31T21:35:45.031 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-05-31T21:35:45.032 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: 2023-05-31T21:29:57.418+0000 7f93752f2780 20 RGW-SYNC:meta: read sync status 2023-05-31T21:35:45.032 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: 2023-05-31T21:29:57.419+0000 7f93752f2780 20 RGW-SYNC:meta: read sync status 2023-05-31T21:35:45.032 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2023-05-31T21:35:45.032 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "sync_status": { 2023-05-31T21:35:45.033 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "info": { 2023-05-31T21:35:45.033 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "status": "sync", 2023-05-31T21:35:45.033 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "num_shards": 4, 2023-05-31T21:35:45.033 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "period": "2e6123d2-d695-4350-9a01-61a455bc782b", 2023-05-31T21:35:45.034 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "realm_epoch": 2 2023-05-31T21:35:45.034 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2023-05-31T21:35:45.034 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "markers": [ 2023-05-31T21:35:45.034 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2023-05-31T21:35:45.035 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "key": 0, 2023-05-31T21:35:45.035 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "val": { 2023-05-31T21:35:45.035 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "state": 1, 2023-05-31T21:35:45.035 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "marker": "1_1685566501.757183_260.1", 2023-05-31T21:35:45.035 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "next_step_marker": "", 2023-05-31T21:35:45.036 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "total_entries": 1, 2023-05-31T21:35:45.036 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "pos": 0, 2023-05-31T21:35:45.036 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "timestamp": "2023-05-31T20:55:01.757183Z", 2023-05-31T21:35:45.036 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "realm_epoch": 2 2023-05-31T21:35:45.037 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2023-05-31T21:35:45.037 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2023-05-31T21:35:45.037 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2023-05-31T21:35:45.037 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "key": 1, 2023-05-31T21:35:45.038 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "val": { 2023-05-31T21:35:45.038 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "state": 1, 2023-05-31T21:35:45.038 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "marker": "1_1685566501.735717_251.1", 2023-05-31T21:35:45.038 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "next_step_marker": "", 2023-05-31T21:35:45.038 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "total_entries": 0, 2023-05-31T21:35:45.039 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "pos": 0, 2023-05-31T21:35:45.039 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "timestamp": "2023-05-31T20:55:01.735717Z", 2023-05-31T21:35:45.039 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "realm_epoch": 2 2023-05-31T21:35:45.039 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2023-05-31T21:35:45.040 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2023-05-31T21:35:45.040 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2023-05-31T21:35:45.040 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "key": 2, 2023-05-31T21:35:45.040 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "val": { 2023-05-31T21:35:45.041 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "state": 1, 2023-05-31T21:35:45.041 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "marker": "1_1685562970.274572_93.1", 2023-05-31T21:35:45.041 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "next_step_marker": "", 2023-05-31T21:35:45.041 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "total_entries": 0, 2023-05-31T21:35:45.041 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "pos": 0, 2023-05-31T21:35:45.042 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "timestamp": "2023-05-31T19:56:10.274572Z", 2023-05-31T21:35:45.042 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "realm_epoch": 2 2023-05-31T21:35:45.042 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2023-05-31T21:35:45.042 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2023-05-31T21:35:45.043 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2023-05-31T21:35:45.043 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "key": 3, 2023-05-31T21:35:45.043 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "val": { 2023-05-31T21:35:45.043 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "state": 1, 2023-05-31T21:35:45.043 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "marker": "1_1685568561.112279_1036.1", 2023-05-31T21:35:45.044 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "next_step_marker": "", 2023-05-31T21:35:45.044 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "total_entries": 0, 2023-05-31T21:35:45.044 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "pos": 0, 2023-05-31T21:35:45.044 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "timestamp": "2023-05-31T21:29:21.112279Z", 2023-05-31T21:35:45.045 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "realm_epoch": 2 2023-05-31T21:35:45.045 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2023-05-31T21:35:45.045 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2023-05-31T21:35:45.045 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: ] 2023-05-31T21:35:45.046 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2023-05-31T21:35:45.046 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "full_sync": { 2023-05-31T21:35:45.046 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "total": 1, 2023-05-31T21:35:45.046 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "complete": 1 2023-05-31T21:35:45.047 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2023-05-31T21:35:45.047 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "current_time": "2023-05-31T21:29:57Z" 2023-05-31T21:35:45.047 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2023-05-31T21:35:45.047 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2023-05-31T21:35:45.047 INFO:tasks.rgw_multisite_tests: "sync_status": { 2023-05-31T21:35:45.048 INFO:tasks.rgw_multisite_tests: "info": { 2023-05-31T21:35:45.048 INFO:tasks.rgw_multisite_tests: "status": "sync", 2023-05-31T21:35:45.048 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2023-05-31T21:35:45.048 INFO:tasks.rgw_multisite_tests: "period": "2e6123d2-d695-4350-9a01-61a455bc782b", 2023-05-31T21:35:45.049 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-05-31T21:35:45.049 INFO:tasks.rgw_multisite_tests: }, 2023-05-31T21:35:45.049 INFO:tasks.rgw_multisite_tests: "markers": [ 2023-05-31T21:35:45.049 INFO:tasks.rgw_multisite_tests: { 2023-05-31T21:35:45.050 INFO:tasks.rgw_multisite_tests: "key": 0, 2023-05-31T21:35:45.050 INFO:tasks.rgw_multisite_tests: "val": { 2023-05-31T21:35:45.050 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-05-31T21:35:45.050 INFO:tasks.rgw_multisite_tests: "marker": "1_1685566501.757183_260.1", 2023-05-31T21:35:45.050 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-05-31T21:35:45.051 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2023-05-31T21:35:45.051 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-05-31T21:35:45.051 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-05-31T20:55:01.757183Z", 2023-05-31T21:35:45.051 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-05-31T21:35:45.052 INFO:tasks.rgw_multisite_tests: } 2023-05-31T21:35:45.052 INFO:tasks.rgw_multisite_tests: }, 2023-05-31T21:35:45.052 INFO:tasks.rgw_multisite_tests: { 2023-05-31T21:35:45.052 INFO:tasks.rgw_multisite_tests: "key": 1, 2023-05-31T21:35:45.053 INFO:tasks.rgw_multisite_tests: "val": { 2023-05-31T21:35:45.053 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-05-31T21:35:45.053 INFO:tasks.rgw_multisite_tests: "marker": "1_1685566501.735717_251.1", 2023-05-31T21:35:45.053 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-05-31T21:35:45.053 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-05-31T21:35:45.054 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-05-31T21:35:45.054 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-05-31T20:55:01.735717Z", 2023-05-31T21:35:45.054 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-05-31T21:35:45.054 INFO:tasks.rgw_multisite_tests: } 2023-05-31T21:35:45.055 INFO:tasks.rgw_multisite_tests: }, 2023-05-31T21:35:45.055 INFO:tasks.rgw_multisite_tests: { 2023-05-31T21:35:45.055 INFO:tasks.rgw_multisite_tests: "key": 2, 2023-05-31T21:35:45.055 INFO:tasks.rgw_multisite_tests: "val": { 2023-05-31T21:35:45.055 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-05-31T21:35:45.056 INFO:tasks.rgw_multisite_tests: "marker": "1_1685562970.274572_93.1", 2023-05-31T21:35:45.056 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-05-31T21:35:45.056 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-05-31T21:35:45.056 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-05-31T21:35:45.057 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-05-31T19:56:10.274572Z", 2023-05-31T21:35:45.057 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-05-31T21:35:45.057 INFO:tasks.rgw_multisite_tests: } 2023-05-31T21:35:45.057 INFO:tasks.rgw_multisite_tests: }, 2023-05-31T21:35:45.058 INFO:tasks.rgw_multisite_tests: { 2023-05-31T21:35:45.058 INFO:tasks.rgw_multisite_tests: "key": 3, 2023-05-31T21:35:45.058 INFO:tasks.rgw_multisite_tests: "val": { 2023-05-31T21:35:45.058 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-05-31T21:35:45.058 INFO:tasks.rgw_multisite_tests: "marker": "1_1685568561.112279_1036.1", 2023-05-31T21:35:45.059 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-05-31T21:35:45.059 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-05-31T21:35:45.059 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-05-31T21:35:45.059 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-05-31T21:29:21.112279Z", 2023-05-31T21:35:45.060 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-05-31T21:35:45.060 INFO:tasks.rgw_multisite_tests: } 2023-05-31T21:35:45.060 INFO:tasks.rgw_multisite_tests: } 2023-05-31T21:35:45.060 INFO:tasks.rgw_multisite_tests: ] 2023-05-31T21:35:45.061 INFO:tasks.rgw_multisite_tests: }, 2023-05-31T21:35:45.061 INFO:tasks.rgw_multisite_tests: "full_sync": { 2023-05-31T21:35:45.061 INFO:tasks.rgw_multisite_tests: "total": 1, 2023-05-31T21:35:45.061 INFO:tasks.rgw_multisite_tests: "complete": 1 2023-05-31T21:35:45.061 INFO:tasks.rgw_multisite_tests: }, 2023-05-31T21:35:45.062 INFO:tasks.rgw_multisite_tests: "current_time": "2023-05-31T21:29:57Z" 2023-05-31T21:35:45.062 INFO:tasks.rgw_multisite_tests:} 2023-05-31T21:35:45.062 INFO:tasks.rgw_multisite_tests: 2023-05-31T21:35:45.062 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1685566501.757183_260.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-05-31T20:55:01.757183Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1685566501.735717_251.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-05-31T20:55:01.735717Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1685562970.274572_93.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-05-31T19:56:10.274572Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1685568561.112279_1036.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-05-31T21:29:21.112279Z', 'realm_epoch': 2}}] 2023-05-31T21:35:45.063 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1685566501.757183_260.1', 1: '1_1685566501.735717_251.1', 2: '1_1685562970.274572_93.1', 3: '1_1685568561.112279_1036.1'} 2023-05-31T21:35:45.063 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1685566501.757183_260.1', 1: '1_1685566501.735717_251.1', 2: '1_1685562970.274572_93.1', 3: '1_1685568561.112279_1036.1'} 2023-05-31T21:35:45.063 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: INFO: finish meta checkpoint for zone=a2 2023-05-31T21:35:45.063 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/pklxkd-72/ 2023-05-31T21:35:45.064 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/pklxkd-72/ 2023-05-31T21:35:45.064 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-05-31T21:35:45.064 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /pklxkd-72/ 2023-05-31T21:35:45.064 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-05-31T21:35:45.064 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-05-31T21:35:45.065 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T21:35:45.065 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-05-31T21:35:45.065 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-05-31T21:35:45.065 INFO:tasks.rgw_multisite_tests:boto: DEBUG: establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2023-05-31T21:35:45.066 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-05-31T21:35:45.066 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-05-31T21:35:45.066 INFO:tasks.rgw_multisite_tests:HEAD 2023-05-31T21:35:45.066 INFO:tasks.rgw_multisite_tests: 2023-05-31T21:35:45.067 INFO:tasks.rgw_multisite_tests: 2023-05-31T21:35:45.067 INFO:tasks.rgw_multisite_tests:Wed, 31 May 2023 21:29:57 GMT 2023-05-31T21:35:45.067 INFO:tasks.rgw_multisite_tests:/pklxkd-72/ 2023-05-31T21:35:45.067 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-05-31T21:35:45.067 INFO:tasks.rgw_multisite_tests:AWS Q80H11OLNU1S48PPYL26:F21Pds4gJbdmbGiZ1XnS+wcZrLw= 2023-05-31T21:35:45.068 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 21:29:57 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:F21Pds4gJbdmbGiZ1XnS+wcZrLw=', 'Content-Length': '0'} 2023-05-31T21:35:45.068 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '2'), ('X-RGW-Bytes-Used', '12'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000083c10bed5eea0c31-006477bc55-2958-a1'), ('Content-Length', '0'), ('Date', 'Wed, 31 May 2023 21:29:57 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:35:45.068 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/pklxkd-72/obj5 2023-05-31T21:35:45.068 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/pklxkd-72/obj5 2023-05-31T21:35:45.069 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2023-05-31T21:35:45.069 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /pklxkd-72/obj5 2023-05-31T21:35:45.069 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-05-31T21:35:45.069 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2023-05-31T21:35:45.070 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T21:35:45.070 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-05-31T21:35:45.070 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-05-31T21:35:45.070 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-05-31T21:35:45.071 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-05-31T21:35:45.071 INFO:tasks.rgw_multisite_tests:PUT 2023-05-31T21:35:45.071 INFO:tasks.rgw_multisite_tests:qPXxZ/RPSWTmyZje6CcRDA== 2023-05-31T21:35:45.071 INFO:tasks.rgw_multisite_tests:application/octet-stream 2023-05-31T21:35:45.071 INFO:tasks.rgw_multisite_tests:Wed, 31 May 2023 21:29:57 GMT 2023-05-31T21:35:45.072 INFO:tasks.rgw_multisite_tests:/pklxkd-72/obj5 2023-05-31T21:35:45.072 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-05-31T21:35:45.072 INFO:tasks.rgw_multisite_tests:AWS Q80H11OLNU1S48PPYL26:rXaM+mUYb/nryX39GzigpcwXV84= 2023-05-31T21:35:45.072 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Wed, 31 May 2023 21:29:57 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:rXaM+mUYb/nryX39GzigpcwXV84='} 2023-05-31T21:35:45.073 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000815722a9f9ef11b4-006477bc55-2958-a1'), ('Date', 'Wed, 31 May 2023 21:29:57 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:35:45.073 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/pklxkd-72/ 2023-05-31T21:35:45.073 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/pklxkd-72/ 2023-05-31T21:35:45.073 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-05-31T21:35:45.074 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /pklxkd-72/ 2023-05-31T21:35:45.074 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-05-31T21:35:45.074 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-05-31T21:35:45.074 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T21:35:45.074 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-05-31T21:35:45.075 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-05-31T21:35:45.075 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-05-31T21:35:45.075 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-05-31T21:35:45.075 INFO:tasks.rgw_multisite_tests:HEAD 2023-05-31T21:35:45.076 INFO:tasks.rgw_multisite_tests: 2023-05-31T21:35:45.076 INFO:tasks.rgw_multisite_tests: 2023-05-31T21:35:45.076 INFO:tasks.rgw_multisite_tests:Wed, 31 May 2023 21:29:57 GMT 2023-05-31T21:35:45.076 INFO:tasks.rgw_multisite_tests:/pklxkd-72/ 2023-05-31T21:35:45.077 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-05-31T21:35:45.077 INFO:tasks.rgw_multisite_tests:AWS Q80H11OLNU1S48PPYL26:F21Pds4gJbdmbGiZ1XnS+wcZrLw= 2023-05-31T21:35:45.077 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 21:29:57 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:F21Pds4gJbdmbGiZ1XnS+wcZrLw=', 'Content-Length': '0'} 2023-05-31T21:35:45.077 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '3'), ('X-RGW-Bytes-Used', '18'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000e7dba73aec154804-006477bc55-2958-a1'), ('Content-Length', '0'), ('Date', 'Wed, 31 May 2023 21:29:57 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:35:45.077 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/pklxkd-72/obj6 2023-05-31T21:35:45.078 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/pklxkd-72/obj6 2023-05-31T21:35:45.078 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2023-05-31T21:35:45.078 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /pklxkd-72/obj6 2023-05-31T21:35:45.078 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-05-31T21:35:45.079 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2023-05-31T21:35:45.079 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T21:35:45.079 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-05-31T21:35:45.079 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-05-31T21:35:45.080 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-05-31T21:35:45.080 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-05-31T21:35:45.080 INFO:tasks.rgw_multisite_tests:PUT 2023-05-31T21:35:45.080 INFO:tasks.rgw_multisite_tests:qPXxZ/RPSWTmyZje6CcRDA== 2023-05-31T21:35:45.080 INFO:tasks.rgw_multisite_tests:application/octet-stream 2023-05-31T21:35:45.081 INFO:tasks.rgw_multisite_tests:Wed, 31 May 2023 21:29:57 GMT 2023-05-31T21:35:45.081 INFO:tasks.rgw_multisite_tests:/pklxkd-72/obj6 2023-05-31T21:35:45.081 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-05-31T21:35:45.081 INFO:tasks.rgw_multisite_tests:AWS Q80H11OLNU1S48PPYL26:MPGBl8jm7DzCPiqXtoDDJMIO98s= 2023-05-31T21:35:45.082 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Wed, 31 May 2023 21:29:57 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:MPGBl8jm7DzCPiqXtoDDJMIO98s='} 2023-05-31T21:35:45.082 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000a6f82f98c41b8409-006477bc55-2958-a1'), ('Date', 'Wed, 31 May 2023 21:29:57 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:35:45.082 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/pklxkd-73/ 2023-05-31T21:35:45.082 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/pklxkd-73/ 2023-05-31T21:35:45.083 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-05-31T21:35:45.083 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /pklxkd-73/ 2023-05-31T21:35:45.083 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-05-31T21:35:45.083 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-05-31T21:35:45.083 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T21:35:45.084 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-05-31T21:35:45.084 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-05-31T21:35:45.084 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-05-31T21:35:45.084 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-05-31T21:35:45.085 INFO:tasks.rgw_multisite_tests:HEAD 2023-05-31T21:35:45.085 INFO:tasks.rgw_multisite_tests: 2023-05-31T21:35:45.085 INFO:tasks.rgw_multisite_tests: 2023-05-31T21:35:45.085 INFO:tasks.rgw_multisite_tests:Wed, 31 May 2023 21:29:57 GMT 2023-05-31T21:35:45.086 INFO:tasks.rgw_multisite_tests:/pklxkd-73/ 2023-05-31T21:35:45.086 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-05-31T21:35:45.086 INFO:tasks.rgw_multisite_tests:AWS Q80H11OLNU1S48PPYL26:5+tH0aHlFpzrhDHewn/fGUEnCaY= 2023-05-31T21:35:45.086 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 21:29:57 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:5+tH0aHlFpzrhDHewn/fGUEnCaY=', 'Content-Length': '0'} 2023-05-31T21:35:45.087 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '2'), ('X-RGW-Bytes-Used', '12'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000da9fadc96819c97c-006477bc55-2958-a1'), ('Content-Length', '0'), ('Date', 'Wed, 31 May 2023 21:29:57 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:35:45.087 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/pklxkd-73/obj7 2023-05-31T21:35:45.087 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/pklxkd-73/obj7 2023-05-31T21:35:45.087 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2023-05-31T21:35:45.087 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /pklxkd-73/obj7 2023-05-31T21:35:45.088 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-05-31T21:35:45.088 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2023-05-31T21:35:45.088 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T21:35:45.089 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-05-31T21:35:45.089 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-05-31T21:35:45.089 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-05-31T21:35:45.089 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-05-31T21:35:45.090 INFO:tasks.rgw_multisite_tests:PUT 2023-05-31T21:35:45.090 INFO:tasks.rgw_multisite_tests:qPXxZ/RPSWTmyZje6CcRDA== 2023-05-31T21:35:45.090 INFO:tasks.rgw_multisite_tests:application/octet-stream 2023-05-31T21:35:45.090 INFO:tasks.rgw_multisite_tests:Wed, 31 May 2023 21:29:57 GMT 2023-05-31T21:35:45.091 INFO:tasks.rgw_multisite_tests:/pklxkd-73/obj7 2023-05-31T21:35:45.091 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-05-31T21:35:45.091 INFO:tasks.rgw_multisite_tests:AWS Q80H11OLNU1S48PPYL26:9M7Z2y8F+6b09LscCSHaB8ticgY= 2023-05-31T21:35:45.091 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Wed, 31 May 2023 21:29:57 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:9M7Z2y8F+6b09LscCSHaB8ticgY='} 2023-05-31T21:35:45.092 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000c32e435997d99d10-006477bc55-2958-a1'), ('Date', 'Wed, 31 May 2023 21:29:57 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:35:45.092 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/pklxkd-73/ 2023-05-31T21:35:45.092 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/pklxkd-73/ 2023-05-31T21:35:45.092 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-05-31T21:35:45.093 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /pklxkd-73/ 2023-05-31T21:35:45.093 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-05-31T21:35:45.093 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-05-31T21:35:45.093 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T21:35:45.094 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-05-31T21:35:45.094 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-05-31T21:35:45.094 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-05-31T21:35:45.095 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-05-31T21:35:45.095 INFO:tasks.rgw_multisite_tests:HEAD 2023-05-31T21:35:45.095 INFO:tasks.rgw_multisite_tests: 2023-05-31T21:35:45.095 INFO:tasks.rgw_multisite_tests: 2023-05-31T21:35:45.096 INFO:tasks.rgw_multisite_tests:Wed, 31 May 2023 21:29:57 GMT 2023-05-31T21:35:45.096 INFO:tasks.rgw_multisite_tests:/pklxkd-73/ 2023-05-31T21:35:45.096 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-05-31T21:35:45.096 INFO:tasks.rgw_multisite_tests:AWS Q80H11OLNU1S48PPYL26:5+tH0aHlFpzrhDHewn/fGUEnCaY= 2023-05-31T21:35:45.097 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 21:29:57 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:5+tH0aHlFpzrhDHewn/fGUEnCaY=', 'Content-Length': '0'} 2023-05-31T21:35:45.097 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '3'), ('X-RGW-Bytes-Used', '18'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000595b708d9f08954d-006477bc55-2958-a1'), ('Content-Length', '0'), ('Date', 'Wed, 31 May 2023 21:29:57 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:35:45.097 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/pklxkd-73/obj8 2023-05-31T21:35:45.097 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/pklxkd-73/obj8 2023-05-31T21:35:45.098 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2023-05-31T21:35:45.098 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /pklxkd-73/obj8 2023-05-31T21:35:45.098 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-05-31T21:35:45.099 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2023-05-31T21:35:45.099 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi099.front.sepia.ceph.com:8000 2023-05-31T21:35:45.099 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-05-31T21:35:45.100 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-05-31T21:35:45.100 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-05-31T21:35:45.100 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-05-31T21:35:45.100 INFO:tasks.rgw_multisite_tests:PUT 2023-05-31T21:35:45.100 INFO:tasks.rgw_multisite_tests:qPXxZ/RPSWTmyZje6CcRDA== 2023-05-31T21:35:45.101 INFO:tasks.rgw_multisite_tests:application/octet-stream 2023-05-31T21:35:45.101 INFO:tasks.rgw_multisite_tests:Wed, 31 May 2023 21:29:57 GMT 2023-05-31T21:35:45.101 INFO:tasks.rgw_multisite_tests:/pklxkd-73/obj8 2023-05-31T21:35:45.101 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-05-31T21:35:45.102 INFO:tasks.rgw_multisite_tests:AWS Q80H11OLNU1S48PPYL26:LHqecPLTHuDIH4WYPs0xOKo39S8= 2023-05-31T21:35:45.102 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Wed, 31 May 2023 21:29:57 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:LHqecPLTHuDIH4WYPs0xOKo39S8='} 2023-05-31T21:35:45.102 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000048692692bd253a4-006477bc55-2958-a1'), ('Date', 'Wed, 31 May 2023 21:29:57 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:35:45.102 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-31T21:35:45.103 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-31T21:35:45.103 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-05-31T21:35:45.103 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: ignoring --setuser ceph since I am not root 2023-05-31T21:35:45.103 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-05-31T21:35:45.103 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: No --period given, using current period=2e6123d2-d695-4350-9a01-61a455bc782b 2023-05-31T21:35:45.104 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: [ 2023-05-31T21:35:45.104 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2023-05-31T21:35:45.104 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "marker": "1_1685566501.757183_260.1", 2023-05-31T21:35:45.104 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "last_update": "2023-05-31T20:55:01.757183Z" 2023-05-31T21:35:45.105 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2023-05-31T21:35:45.105 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2023-05-31T21:35:45.105 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "marker": "1_1685566501.735717_251.1", 2023-05-31T21:35:45.105 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "last_update": "2023-05-31T20:55:01.735717Z" 2023-05-31T21:35:45.106 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2023-05-31T21:35:45.106 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2023-05-31T21:35:45.106 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "marker": "1_1685562970.274572_93.1", 2023-05-31T21:35:45.106 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "last_update": "2023-05-31T19:56:10.274572Z" 2023-05-31T21:35:45.106 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2023-05-31T21:35:45.107 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2023-05-31T21:35:45.107 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "marker": "1_1685568561.112279_1036.1", 2023-05-31T21:35:45.107 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "last_update": "2023-05-31T21:29:21.112279Z" 2023-05-31T21:35:45.107 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2023-05-31T21:35:45.108 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: ] 2023-05-31T21:35:45.108 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: master meta markers={0: '1_1685566501.757183_260.1', 1: '1_1685566501.735717_251.1', 2: '1_1685562970.274572_93.1', 3: '1_1685568561.112279_1036.1'} 2023-05-31T21:35:45.108 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: INFO: starting meta checkpoint for zone=a2 2023-05-31T21:35:45.108 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-31T21:35:45.109 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-31T21:35:45.109 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-05-31T21:35:45.109 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: ignoring --setuser ceph since I am not root 2023-05-31T21:35:45.109 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-05-31T21:35:45.110 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: 2023-05-31T21:29:57.776+0000 7fa2bb2e1780 20 RGW-SYNC:meta: read sync status 2023-05-31T21:35:45.110 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: 2023-05-31T21:29:57.777+0000 7fa2bb2e1780 20 RGW-SYNC:meta: read sync status 2023-05-31T21:35:45.110 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2023-05-31T21:35:45.110 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "sync_status": { 2023-05-31T21:35:45.110 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "info": { 2023-05-31T21:35:45.111 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "status": "sync", 2023-05-31T21:35:45.111 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "num_shards": 4, 2023-05-31T21:35:45.111 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "period": "2e6123d2-d695-4350-9a01-61a455bc782b", 2023-05-31T21:35:45.111 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "realm_epoch": 2 2023-05-31T21:35:45.112 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2023-05-31T21:35:45.112 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "markers": [ 2023-05-31T21:35:45.112 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2023-05-31T21:35:45.112 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "key": 0, 2023-05-31T21:35:45.113 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "val": { 2023-05-31T21:35:45.113 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "state": 1, 2023-05-31T21:35:45.113 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "marker": "1_1685566501.757183_260.1", 2023-05-31T21:35:45.113 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "next_step_marker": "", 2023-05-31T21:35:45.113 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "total_entries": 1, 2023-05-31T21:35:45.114 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "pos": 0, 2023-05-31T21:35:45.114 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "timestamp": "2023-05-31T20:55:01.757183Z", 2023-05-31T21:35:45.114 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "realm_epoch": 2 2023-05-31T21:35:45.114 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2023-05-31T21:35:45.115 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2023-05-31T21:35:45.115 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2023-05-31T21:35:45.115 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "key": 1, 2023-05-31T21:35:45.115 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "val": { 2023-05-31T21:35:45.116 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "state": 1, 2023-05-31T21:35:45.116 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "marker": "1_1685566501.735717_251.1", 2023-05-31T21:35:45.116 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "next_step_marker": "", 2023-05-31T21:35:45.116 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "total_entries": 0, 2023-05-31T21:35:45.116 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "pos": 0, 2023-05-31T21:35:45.117 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "timestamp": "2023-05-31T20:55:01.735717Z", 2023-05-31T21:35:45.117 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "realm_epoch": 2 2023-05-31T21:35:45.117 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2023-05-31T21:35:45.117 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2023-05-31T21:35:45.118 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2023-05-31T21:35:45.118 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "key": 2, 2023-05-31T21:35:45.118 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "val": { 2023-05-31T21:35:45.118 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "state": 1, 2023-05-31T21:35:45.119 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "marker": "1_1685562970.274572_93.1", 2023-05-31T21:35:45.119 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "next_step_marker": "", 2023-05-31T21:35:45.119 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "total_entries": 0, 2023-05-31T21:35:45.119 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "pos": 0, 2023-05-31T21:35:45.119 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "timestamp": "2023-05-31T19:56:10.274572Z", 2023-05-31T21:35:45.120 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "realm_epoch": 2 2023-05-31T21:35:45.120 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2023-05-31T21:35:45.120 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2023-05-31T21:35:45.120 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2023-05-31T21:35:45.121 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "key": 3, 2023-05-31T21:35:45.121 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "val": { 2023-05-31T21:35:45.121 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "state": 1, 2023-05-31T21:35:45.121 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "marker": "1_1685568561.112279_1036.1", 2023-05-31T21:35:45.122 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "next_step_marker": "", 2023-05-31T21:35:45.122 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "total_entries": 0, 2023-05-31T21:35:45.122 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "pos": 0, 2023-05-31T21:35:45.122 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "timestamp": "2023-05-31T21:29:21.112279Z", 2023-05-31T21:35:45.122 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "realm_epoch": 2 2023-05-31T21:35:45.123 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2023-05-31T21:35:45.123 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2023-05-31T21:35:45.123 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: ] 2023-05-31T21:35:45.123 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2023-05-31T21:35:45.124 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "full_sync": { 2023-05-31T21:35:45.124 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "total": 1, 2023-05-31T21:35:45.124 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "complete": 1 2023-05-31T21:35:45.124 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2023-05-31T21:35:45.125 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "current_time": "2023-05-31T21:29:57Z" 2023-05-31T21:35:45.125 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2023-05-31T21:35:45.125 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2023-05-31T21:35:45.125 INFO:tasks.rgw_multisite_tests: "sync_status": { 2023-05-31T21:35:45.125 INFO:tasks.rgw_multisite_tests: "info": { 2023-05-31T21:35:45.126 INFO:tasks.rgw_multisite_tests: "status": "sync", 2023-05-31T21:35:45.126 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2023-05-31T21:35:45.126 INFO:tasks.rgw_multisite_tests: "period": "2e6123d2-d695-4350-9a01-61a455bc782b", 2023-05-31T21:35:45.126 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-05-31T21:35:45.127 INFO:tasks.rgw_multisite_tests: }, 2023-05-31T21:35:45.127 INFO:tasks.rgw_multisite_tests: "markers": [ 2023-05-31T21:35:45.127 INFO:tasks.rgw_multisite_tests: { 2023-05-31T21:35:45.127 INFO:tasks.rgw_multisite_tests: "key": 0, 2023-05-31T21:35:45.128 INFO:tasks.rgw_multisite_tests: "val": { 2023-05-31T21:35:45.128 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-05-31T21:35:45.128 INFO:tasks.rgw_multisite_tests: "marker": "1_1685566501.757183_260.1", 2023-05-31T21:35:45.128 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-05-31T21:35:45.128 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2023-05-31T21:35:45.129 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-05-31T21:35:45.129 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-05-31T20:55:01.757183Z", 2023-05-31T21:35:45.129 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-05-31T21:35:45.129 INFO:tasks.rgw_multisite_tests: } 2023-05-31T21:35:45.130 INFO:tasks.rgw_multisite_tests: }, 2023-05-31T21:35:45.130 INFO:tasks.rgw_multisite_tests: { 2023-05-31T21:35:45.130 INFO:tasks.rgw_multisite_tests: "key": 1, 2023-05-31T21:35:45.130 INFO:tasks.rgw_multisite_tests: "val": { 2023-05-31T21:35:45.130 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-05-31T21:35:45.131 INFO:tasks.rgw_multisite_tests: "marker": "1_1685566501.735717_251.1", 2023-05-31T21:35:45.131 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-05-31T21:35:45.131 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-05-31T21:35:45.131 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-05-31T21:35:45.132 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-05-31T20:55:01.735717Z", 2023-05-31T21:35:45.132 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-05-31T21:35:45.132 INFO:tasks.rgw_multisite_tests: } 2023-05-31T21:35:45.132 INFO:tasks.rgw_multisite_tests: }, 2023-05-31T21:35:45.133 INFO:tasks.rgw_multisite_tests: { 2023-05-31T21:35:45.133 INFO:tasks.rgw_multisite_tests: "key": 2, 2023-05-31T21:35:45.133 INFO:tasks.rgw_multisite_tests: "val": { 2023-05-31T21:35:45.133 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-05-31T21:35:45.133 INFO:tasks.rgw_multisite_tests: "marker": "1_1685562970.274572_93.1", 2023-05-31T21:35:45.134 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-05-31T21:35:45.134 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-05-31T21:35:45.134 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-05-31T21:35:45.134 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-05-31T19:56:10.274572Z", 2023-05-31T21:35:45.135 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-05-31T21:35:45.135 INFO:tasks.rgw_multisite_tests: } 2023-05-31T21:35:45.135 INFO:tasks.rgw_multisite_tests: }, 2023-05-31T21:35:45.135 INFO:tasks.rgw_multisite_tests: { 2023-05-31T21:35:45.135 INFO:tasks.rgw_multisite_tests: "key": 3, 2023-05-31T21:35:45.136 INFO:tasks.rgw_multisite_tests: "val": { 2023-05-31T21:35:45.136 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-05-31T21:35:45.136 INFO:tasks.rgw_multisite_tests: "marker": "1_1685568561.112279_1036.1", 2023-05-31T21:35:45.136 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-05-31T21:35:45.137 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-05-31T21:35:45.137 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-05-31T21:35:45.137 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-05-31T21:29:21.112279Z", 2023-05-31T21:35:45.139 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-05-31T21:35:45.140 INFO:tasks.rgw_multisite_tests: } 2023-05-31T21:35:45.140 INFO:tasks.rgw_multisite_tests: } 2023-05-31T21:35:45.140 INFO:tasks.rgw_multisite_tests: ] 2023-05-31T21:35:45.140 INFO:tasks.rgw_multisite_tests: }, 2023-05-31T21:35:45.140 INFO:tasks.rgw_multisite_tests: "full_sync": { 2023-05-31T21:35:45.141 INFO:tasks.rgw_multisite_tests: "total": 1, 2023-05-31T21:35:45.141 INFO:tasks.rgw_multisite_tests: "complete": 1 2023-05-31T21:35:45.141 INFO:tasks.rgw_multisite_tests: }, 2023-05-31T21:35:45.141 INFO:tasks.rgw_multisite_tests: "current_time": "2023-05-31T21:29:57Z" 2023-05-31T21:35:45.142 INFO:tasks.rgw_multisite_tests:} 2023-05-31T21:35:45.142 INFO:tasks.rgw_multisite_tests: 2023-05-31T21:35:45.142 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1685566501.757183_260.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-05-31T20:55:01.757183Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1685566501.735717_251.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-05-31T20:55:01.735717Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1685562970.274572_93.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-05-31T19:56:10.274572Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1685568561.112279_1036.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-05-31T21:29:21.112279Z', 'realm_epoch': 2}}] 2023-05-31T21:35:45.142 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1685566501.757183_260.1', 1: '1_1685566501.735717_251.1', 2: '1_1685562970.274572_93.1', 3: '1_1685568561.112279_1036.1'} 2023-05-31T21:35:45.143 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1685566501.757183_260.1', 1: '1_1685566501.735717_251.1', 2: '1_1685562970.274572_93.1', 3: '1_1685568561.112279_1036.1'} 2023-05-31T21:35:45.143 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: INFO: finish meta checkpoint for zone=a2 2023-05-31T21:35:45.143 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['datalog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-31T21:35:45.143 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'datalog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-31T21:35:45.143 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 datalog status --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-05-31T21:35:45.144 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: ignoring --setuser ceph since I am not root 2023-05-31T21:35:45.144 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-05-31T21:35:45.144 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: [ 2023-05-31T21:35:45.144 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2023-05-31T21:35:45.145 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "marker": "00000000000000000000:00000000000000045515", 2023-05-31T21:35:45.145 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "last_update": "2023-05-31T21:29:57.506834Z" 2023-05-31T21:35:45.145 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2023-05-31T21:35:45.145 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2023-05-31T21:35:45.146 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "marker": "00000000000000000000:00000000000000031920", 2023-05-31T21:35:45.146 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "last_update": "2023-05-31T21:29:57.583636Z" 2023-05-31T21:35:45.146 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2023-05-31T21:35:45.146 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2023-05-31T21:35:45.147 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "marker": "00000000000000000000:00000000000000051951", 2023-05-31T21:35:45.147 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "last_update": "2023-05-31T21:29:57.526899Z" 2023-05-31T21:35:45.147 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2023-05-31T21:35:45.147 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2023-05-31T21:35:45.147 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "marker": "00000000000000000000:00000000000000044178", 2023-05-31T21:35:45.148 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "last_update": "2023-05-31T21:27:37.215905Z" 2023-05-31T21:35:45.148 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2023-05-31T21:35:45.148 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: ] 2023-05-31T21:35:45.148 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: data markers for zone=a1 markers={0: '00000000000000000000:00000000000000045515', 1: '00000000000000000000:00000000000000031920', 2: '00000000000000000000:00000000000000051951', 3: '00000000000000000000:00000000000000044178'} 2023-05-31T21:35:45.149 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: INFO: starting data checkpoint for target_zone=a2 source_zone=a1 2023-05-31T21:35:45.149 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-31T21:35:45.149 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-31T21:35:45.149 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-05-31T21:35:45.150 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: ignoring --setuser ceph since I am not root 2023-05-31T21:35:45.150 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-05-31T21:35:45.150 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2023-05-31T21:35:45.150 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "sync_status": { 2023-05-31T21:35:45.150 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "info": { 2023-05-31T21:35:45.151 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "status": "sync", 2023-05-31T21:35:45.151 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "num_shards": 4, 2023-05-31T21:35:45.151 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "instance_id": 452136274422638824 2023-05-31T21:35:45.151 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2023-05-31T21:35:45.152 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "markers": [ 2023-05-31T21:35:45.152 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2023-05-31T21:35:45.152 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "key": 0, 2023-05-31T21:35:45.152 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "val": { 2023-05-31T21:35:45.153 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "status": "incremental-sync", 2023-05-31T21:35:45.153 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "marker": "00000000000000000000:00000000000000045375", 2023-05-31T21:35:45.153 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "next_step_marker": "", 2023-05-31T21:35:45.153 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "total_entries": 0, 2023-05-31T21:35:45.153 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "pos": 0, 2023-05-31T21:35:45.154 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "timestamp": "2023-05-31T21:28:16.422581Z" 2023-05-31T21:35:45.154 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2023-05-31T21:35:45.154 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2023-05-31T21:35:45.154 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2023-05-31T21:35:45.155 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "key": 1, 2023-05-31T21:35:45.155 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "val": { 2023-05-31T21:35:45.155 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "status": "incremental-sync", 2023-05-31T21:35:45.155 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "marker": "00000000000000000000:00000000000000031780", 2023-05-31T21:35:45.156 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "next_step_marker": "", 2023-05-31T21:35:45.156 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "total_entries": 0, 2023-05-31T21:35:45.156 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "pos": 0, 2023-05-31T21:35:45.156 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "timestamp": "2023-05-31T21:28:16.501293Z" 2023-05-31T21:35:45.156 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2023-05-31T21:35:45.157 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2023-05-31T21:35:45.157 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2023-05-31T21:35:45.157 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "key": 2, 2023-05-31T21:35:45.157 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "val": { 2023-05-31T21:35:45.158 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "status": "incremental-sync", 2023-05-31T21:35:45.158 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "marker": "00000000000000000000:00000000000000051671", 2023-05-31T21:35:45.158 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "next_step_marker": "", 2023-05-31T21:35:45.158 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "total_entries": 55, 2023-05-31T21:35:45.159 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "pos": 0, 2023-05-31T21:35:45.159 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "timestamp": "2023-05-31T21:28:16.476487Z" 2023-05-31T21:35:45.159 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2023-05-31T21:35:45.159 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2023-05-31T21:35:45.159 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2023-05-31T21:35:45.160 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "key": 3, 2023-05-31T21:35:45.160 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "val": { 2023-05-31T21:35:45.160 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "status": "incremental-sync", 2023-05-31T21:35:45.160 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "marker": "00000000000000000000:00000000000000044178", 2023-05-31T21:35:45.161 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "next_step_marker": "", 2023-05-31T21:35:45.161 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "total_entries": 9, 2023-05-31T21:35:45.161 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "pos": 0, 2023-05-31T21:35:45.161 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "timestamp": "2023-05-31T21:27:37.215905Z" 2023-05-31T21:35:45.162 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2023-05-31T21:35:45.162 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2023-05-31T21:35:45.162 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: ] 2023-05-31T21:35:45.162 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2023-05-31T21:35:45.162 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "full_sync": { 2023-05-31T21:35:45.163 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "total": 64, 2023-05-31T21:35:45.163 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "complete": 64 2023-05-31T21:35:45.163 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2023-05-31T21:35:45.163 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "current_time": "2023-05-31T21:29:58Z" 2023-05-31T21:35:45.164 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2023-05-31T21:35:45.164 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current data sync status={ 2023-05-31T21:35:45.164 INFO:tasks.rgw_multisite_tests: "sync_status": { 2023-05-31T21:35:45.164 INFO:tasks.rgw_multisite_tests: "info": { 2023-05-31T21:35:45.165 INFO:tasks.rgw_multisite_tests: "status": "sync", 2023-05-31T21:35:45.165 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2023-05-31T21:35:45.165 INFO:tasks.rgw_multisite_tests: "instance_id": 452136274422638824 2023-05-31T21:35:45.165 INFO:tasks.rgw_multisite_tests: }, 2023-05-31T21:35:45.165 INFO:tasks.rgw_multisite_tests: "markers": [ 2023-05-31T21:35:45.166 INFO:tasks.rgw_multisite_tests: { 2023-05-31T21:35:45.166 INFO:tasks.rgw_multisite_tests: "key": 0, 2023-05-31T21:35:45.166 INFO:tasks.rgw_multisite_tests: "val": { 2023-05-31T21:35:45.166 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2023-05-31T21:35:45.167 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000045375", 2023-05-31T21:35:45.167 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-05-31T21:35:45.167 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-05-31T21:35:45.167 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-05-31T21:35:45.168 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-05-31T21:28:16.422581Z" 2023-05-31T21:35:45.168 INFO:tasks.rgw_multisite_tests: } 2023-05-31T21:35:45.168 INFO:tasks.rgw_multisite_tests: }, 2023-05-31T21:35:45.168 INFO:tasks.rgw_multisite_tests: { 2023-05-31T21:35:45.168 INFO:tasks.rgw_multisite_tests: "key": 1, 2023-05-31T21:35:45.169 INFO:tasks.rgw_multisite_tests: "val": { 2023-05-31T21:35:45.169 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2023-05-31T21:35:45.169 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000031780", 2023-05-31T21:35:45.169 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-05-31T21:35:45.170 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-05-31T21:35:45.170 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-05-31T21:35:45.170 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-05-31T21:28:16.501293Z" 2023-05-31T21:35:45.170 INFO:tasks.rgw_multisite_tests: } 2023-05-31T21:35:45.170 INFO:tasks.rgw_multisite_tests: }, 2023-05-31T21:35:45.171 INFO:tasks.rgw_multisite_tests: { 2023-05-31T21:35:45.171 INFO:tasks.rgw_multisite_tests: "key": 2, 2023-05-31T21:35:45.171 INFO:tasks.rgw_multisite_tests: "val": { 2023-05-31T21:35:45.171 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2023-05-31T21:35:45.172 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000051671", 2023-05-31T21:35:45.172 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-05-31T21:35:45.172 INFO:tasks.rgw_multisite_tests: "total_entries": 55, 2023-05-31T21:35:45.172 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-05-31T21:35:45.173 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-05-31T21:28:16.476487Z" 2023-05-31T21:35:45.173 INFO:tasks.rgw_multisite_tests: } 2023-05-31T21:35:45.173 INFO:tasks.rgw_multisite_tests: }, 2023-05-31T21:35:45.173 INFO:tasks.rgw_multisite_tests: { 2023-05-31T21:35:45.173 INFO:tasks.rgw_multisite_tests: "key": 3, 2023-05-31T21:35:45.174 INFO:tasks.rgw_multisite_tests: "val": { 2023-05-31T21:35:45.174 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2023-05-31T21:35:45.174 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000044178", 2023-05-31T21:35:45.174 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-05-31T21:35:45.175 INFO:tasks.rgw_multisite_tests: "total_entries": 9, 2023-05-31T21:35:45.175 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-05-31T21:35:45.175 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-05-31T21:27:37.215905Z" 2023-05-31T21:35:45.175 INFO:tasks.rgw_multisite_tests: } 2023-05-31T21:35:45.176 INFO:tasks.rgw_multisite_tests: } 2023-05-31T21:35:45.176 INFO:tasks.rgw_multisite_tests: ] 2023-05-31T21:35:45.176 INFO:tasks.rgw_multisite_tests: }, 2023-05-31T21:35:45.176 INFO:tasks.rgw_multisite_tests: "full_sync": { 2023-05-31T21:35:45.176 INFO:tasks.rgw_multisite_tests: "total": 64, 2023-05-31T21:35:45.177 INFO:tasks.rgw_multisite_tests: "complete": 64 2023-05-31T21:35:45.177 INFO:tasks.rgw_multisite_tests: }, 2023-05-31T21:35:45.177 INFO:tasks.rgw_multisite_tests: "current_time": "2023-05-31T21:29:58Z" 2023-05-31T21:35:45.177 INFO:tasks.rgw_multisite_tests:} 2023-05-31T21:35:45.178 INFO:tasks.rgw_multisite_tests: 2023-05-31T21:35:45.178 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000045375', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-05-31T21:28:16.422581Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000031780', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-05-31T21:28:16.501293Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000051671', 'next_step_marker': '', 'total_entries': 55, 'pos': 0, 'timestamp': '2023-05-31T21:28:16.476487Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000044178', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2023-05-31T21:27:37.215905Z'}}] 2023-05-31T21:35:45.178 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '00000000000000000000:00000000000000045515', 1: '00000000000000000000:00000000000000031920', 2: '00000000000000000000:00000000000000051951', 3: '00000000000000000000:00000000000000044178'} 2023-05-31T21:35:45.179 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '00000000000000000000:00000000000000045375', 1: '00000000000000000000:00000000000000031780', 2: '00000000000000000000:00000000000000051671', 3: '00000000000000000000:00000000000000044178'} 2023-05-31T21:35:45.179 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: data of zone a2 behind zone a1: shard=0 master=00000000000000000000:00000000000000045515 target=00000000000000000000:00000000000000045375, shard=1 master=00000000000000000000:00000000000000031920 target=00000000000000000000:00000000000000031780, shard=2 master=00000000000000000000:00000000000000051951 target=00000000000000000000:00000000000000051671 2023-05-31T21:35:45.179 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-31T21:35:45.180 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-31T21:35:45.180 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-05-31T21:35:45.180 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: ignoring --setuser ceph since I am not root 2023-05-31T21:35:45.180 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-05-31T21:35:45.181 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2023-05-31T21:35:45.181 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "sync_status": { 2023-05-31T21:35:45.181 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "info": { 2023-05-31T21:35:45.181 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "status": "sync", 2023-05-31T21:35:45.182 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "num_shards": 4, 2023-05-31T21:35:45.182 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "instance_id": 452136274422638824 2023-05-31T21:35:45.182 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2023-05-31T21:35:45.182 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "markers": [ 2023-05-31T21:35:45.182 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2023-05-31T21:35:45.183 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "key": 0, 2023-05-31T21:35:45.183 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "val": { 2023-05-31T21:35:45.183 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "status": "incremental-sync", 2023-05-31T21:35:45.183 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "marker": "00000000000000000000:00000000000000045375", 2023-05-31T21:35:45.184 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "next_step_marker": "", 2023-05-31T21:35:45.184 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "total_entries": 0, 2023-05-31T21:35:45.184 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "pos": 0, 2023-05-31T21:35:45.184 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "timestamp": "2023-05-31T21:28:16.422581Z" 2023-05-31T21:35:45.185 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2023-05-31T21:35:45.185 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2023-05-31T21:35:45.185 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2023-05-31T21:35:45.185 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "key": 1, 2023-05-31T21:35:45.186 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "val": { 2023-05-31T21:35:45.186 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "status": "incremental-sync", 2023-05-31T21:35:45.186 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "marker": "00000000000000000000:00000000000000031780", 2023-05-31T21:35:45.186 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "next_step_marker": "", 2023-05-31T21:35:45.187 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "total_entries": 0, 2023-05-31T21:35:45.187 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "pos": 0, 2023-05-31T21:35:45.187 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "timestamp": "2023-05-31T21:28:16.501293Z" 2023-05-31T21:35:45.187 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2023-05-31T21:35:45.188 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2023-05-31T21:35:45.188 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2023-05-31T21:35:45.188 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "key": 2, 2023-05-31T21:35:45.188 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "val": { 2023-05-31T21:35:45.188 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "status": "incremental-sync", 2023-05-31T21:35:45.189 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "marker": "00000000000000000000:00000000000000051671", 2023-05-31T21:35:45.189 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "next_step_marker": "", 2023-05-31T21:35:45.189 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "total_entries": 55, 2023-05-31T21:35:45.189 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "pos": 0, 2023-05-31T21:35:45.190 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "timestamp": "2023-05-31T21:28:16.476487Z" 2023-05-31T21:35:45.190 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2023-05-31T21:35:45.190 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2023-05-31T21:35:45.190 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2023-05-31T21:35:45.191 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "key": 3, 2023-05-31T21:35:45.191 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "val": { 2023-05-31T21:35:45.191 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "status": "incremental-sync", 2023-05-31T21:35:45.191 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "marker": "00000000000000000000:00000000000000044178", 2023-05-31T21:35:45.191 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "next_step_marker": "", 2023-05-31T21:35:45.192 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "total_entries": 9, 2023-05-31T21:35:45.192 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "pos": 0, 2023-05-31T21:35:45.192 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "timestamp": "2023-05-31T21:27:37.215905Z" 2023-05-31T21:35:45.192 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2023-05-31T21:35:45.193 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2023-05-31T21:35:45.193 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: ] 2023-05-31T21:35:45.193 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2023-05-31T21:35:45.193 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "full_sync": { 2023-05-31T21:35:45.194 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "total": 64, 2023-05-31T21:35:45.194 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "complete": 64 2023-05-31T21:35:45.194 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2023-05-31T21:35:45.194 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "current_time": "2023-05-31T21:30:03Z" 2023-05-31T21:35:45.195 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2023-05-31T21:35:45.195 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current data sync status={ 2023-05-31T21:35:45.195 INFO:tasks.rgw_multisite_tests: "sync_status": { 2023-05-31T21:35:45.195 INFO:tasks.rgw_multisite_tests: "info": { 2023-05-31T21:35:45.196 INFO:tasks.rgw_multisite_tests: "status": "sync", 2023-05-31T21:35:45.196 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2023-05-31T21:35:45.196 INFO:tasks.rgw_multisite_tests: "instance_id": 452136274422638824 2023-05-31T21:35:45.196 INFO:tasks.rgw_multisite_tests: }, 2023-05-31T21:35:45.197 INFO:tasks.rgw_multisite_tests: "markers": [ 2023-05-31T21:35:45.197 INFO:tasks.rgw_multisite_tests: { 2023-05-31T21:35:45.197 INFO:tasks.rgw_multisite_tests: "key": 0, 2023-05-31T21:35:45.197 INFO:tasks.rgw_multisite_tests: "val": { 2023-05-31T21:35:45.197 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2023-05-31T21:35:45.198 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000045375", 2023-05-31T21:35:45.198 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-05-31T21:35:45.198 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-05-31T21:35:45.198 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-05-31T21:35:45.199 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-05-31T21:28:16.422581Z" 2023-05-31T21:35:45.199 INFO:tasks.rgw_multisite_tests: } 2023-05-31T21:35:45.199 INFO:tasks.rgw_multisite_tests: }, 2023-05-31T21:35:45.199 INFO:tasks.rgw_multisite_tests: { 2023-05-31T21:35:45.200 INFO:tasks.rgw_multisite_tests: "key": 1, 2023-05-31T21:35:45.200 INFO:tasks.rgw_multisite_tests: "val": { 2023-05-31T21:35:45.200 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2023-05-31T21:35:45.200 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000031780", 2023-05-31T21:35:45.201 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-05-31T21:35:45.201 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-05-31T21:35:45.201 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-05-31T21:35:45.201 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-05-31T21:28:16.501293Z" 2023-05-31T21:35:45.201 INFO:tasks.rgw_multisite_tests: } 2023-05-31T21:35:45.202 INFO:tasks.rgw_multisite_tests: }, 2023-05-31T21:35:45.202 INFO:tasks.rgw_multisite_tests: { 2023-05-31T21:35:45.202 INFO:tasks.rgw_multisite_tests: "key": 2, 2023-05-31T21:35:45.202 INFO:tasks.rgw_multisite_tests: "val": { 2023-05-31T21:35:45.203 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2023-05-31T21:35:45.203 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000051671", 2023-05-31T21:35:45.203 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-05-31T21:35:45.203 INFO:tasks.rgw_multisite_tests: "total_entries": 55, 2023-05-31T21:35:45.204 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-05-31T21:35:45.204 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-05-31T21:28:16.476487Z" 2023-05-31T21:35:45.204 INFO:tasks.rgw_multisite_tests: } 2023-05-31T21:35:45.204 INFO:tasks.rgw_multisite_tests: }, 2023-05-31T21:35:45.205 INFO:tasks.rgw_multisite_tests: { 2023-05-31T21:35:45.205 INFO:tasks.rgw_multisite_tests: "key": 3, 2023-05-31T21:35:45.205 INFO:tasks.rgw_multisite_tests: "val": { 2023-05-31T21:35:45.205 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2023-05-31T21:35:45.205 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000044178", 2023-05-31T21:35:45.206 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-05-31T21:35:45.206 INFO:tasks.rgw_multisite_tests: "total_entries": 9, 2023-05-31T21:35:45.206 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-05-31T21:35:45.206 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-05-31T21:27:37.215905Z" 2023-05-31T21:35:45.207 INFO:tasks.rgw_multisite_tests: } 2023-05-31T21:35:45.207 INFO:tasks.rgw_multisite_tests: } 2023-05-31T21:35:45.207 INFO:tasks.rgw_multisite_tests: ] 2023-05-31T21:35:45.207 INFO:tasks.rgw_multisite_tests: }, 2023-05-31T21:35:45.208 INFO:tasks.rgw_multisite_tests: "full_sync": { 2023-05-31T21:35:45.208 INFO:tasks.rgw_multisite_tests: "total": 64, 2023-05-31T21:35:45.208 INFO:tasks.rgw_multisite_tests: "complete": 64 2023-05-31T21:35:45.208 INFO:tasks.rgw_multisite_tests: }, 2023-05-31T21:35:45.208 INFO:tasks.rgw_multisite_tests: "current_time": "2023-05-31T21:30:03Z" 2023-05-31T21:35:45.209 INFO:tasks.rgw_multisite_tests:} 2023-05-31T21:35:45.209 INFO:tasks.rgw_multisite_tests: 2023-05-31T21:35:45.209 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000045375', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-05-31T21:28:16.422581Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000031780', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-05-31T21:28:16.501293Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000051671', 'next_step_marker': '', 'total_entries': 55, 'pos': 0, 'timestamp': '2023-05-31T21:28:16.476487Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000044178', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2023-05-31T21:27:37.215905Z'}}] 2023-05-31T21:35:45.209 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '00000000000000000000:00000000000000045515', 1: '00000000000000000000:00000000000000031920', 2: '00000000000000000000:00000000000000051951', 3: '00000000000000000000:00000000000000044178'} 2023-05-31T21:35:45.210 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '00000000000000000000:00000000000000045375', 1: '00000000000000000000:00000000000000031780', 2: '00000000000000000000:00000000000000051671', 3: '00000000000000000000:00000000000000044178'} 2023-05-31T21:35:45.210 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: data of zone a2 behind zone a1: shard=0 master=00000000000000000000:00000000000000045515 target=00000000000000000000:00000000000000045375, shard=1 master=00000000000000000000:00000000000000031920 target=00000000000000000000:00000000000000031780, shard=2 master=00000000000000000000:00000000000000051951 target=00000000000000000000:00000000000000051671 2023-05-31T21:35:45.210 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-31T21:35:45.210 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-31T21:35:45.211 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-05-31T21:35:45.211 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: ignoring --setuser ceph since I am not root 2023-05-31T21:35:45.211 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-05-31T21:35:45.211 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2023-05-31T21:35:45.212 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "sync_status": { 2023-05-31T21:35:45.212 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "info": { 2023-05-31T21:35:45.212 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "status": "sync", 2023-05-31T21:35:45.212 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "num_shards": 4, 2023-05-31T21:35:45.213 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "instance_id": 452136274422638824 2023-05-31T21:35:45.213 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2023-05-31T21:35:45.213 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "markers": [ 2023-05-31T21:35:45.213 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2023-05-31T21:35:45.213 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "key": 0, 2023-05-31T21:35:45.214 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "val": { 2023-05-31T21:35:45.214 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "status": "incremental-sync", 2023-05-31T21:35:45.214 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "marker": "00000000000000000000:00000000000000045375", 2023-05-31T21:35:45.214 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "next_step_marker": "", 2023-05-31T21:35:45.215 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "total_entries": 0, 2023-05-31T21:35:45.215 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "pos": 0, 2023-05-31T21:35:45.215 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "timestamp": "2023-05-31T21:28:16.422581Z" 2023-05-31T21:35:45.215 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2023-05-31T21:35:45.216 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2023-05-31T21:35:45.216 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2023-05-31T21:35:45.216 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "key": 1, 2023-05-31T21:35:45.216 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "val": { 2023-05-31T21:35:45.216 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "status": "incremental-sync", 2023-05-31T21:35:45.217 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "marker": "00000000000000000000:00000000000000031780", 2023-05-31T21:35:45.217 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "next_step_marker": "", 2023-05-31T21:35:45.217 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "total_entries": 0, 2023-05-31T21:35:45.217 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "pos": 0, 2023-05-31T21:35:45.218 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "timestamp": "2023-05-31T21:28:16.501293Z" 2023-05-31T21:35:45.218 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2023-05-31T21:35:45.218 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2023-05-31T21:35:45.218 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2023-05-31T21:35:45.219 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "key": 2, 2023-05-31T21:35:45.219 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "val": { 2023-05-31T21:35:45.219 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "status": "incremental-sync", 2023-05-31T21:35:45.219 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "marker": "00000000000000000000:00000000000000051671", 2023-05-31T21:35:45.219 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "next_step_marker": "", 2023-05-31T21:35:45.220 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "total_entries": 55, 2023-05-31T21:35:45.220 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "pos": 0, 2023-05-31T21:35:45.220 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "timestamp": "2023-05-31T21:28:16.476487Z" 2023-05-31T21:35:45.220 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2023-05-31T21:35:45.221 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2023-05-31T21:35:45.221 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2023-05-31T21:35:45.221 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "key": 3, 2023-05-31T21:35:45.221 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "val": { 2023-05-31T21:35:45.222 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "status": "incremental-sync", 2023-05-31T21:35:45.222 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "marker": "00000000000000000000:00000000000000044178", 2023-05-31T21:35:45.222 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "next_step_marker": "", 2023-05-31T21:35:45.222 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "total_entries": 9, 2023-05-31T21:35:45.222 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "pos": 0, 2023-05-31T21:35:45.223 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "timestamp": "2023-05-31T21:27:37.215905Z" 2023-05-31T21:35:45.223 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2023-05-31T21:35:45.223 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2023-05-31T21:35:45.223 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: ] 2023-05-31T21:35:45.224 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2023-05-31T21:35:45.224 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "full_sync": { 2023-05-31T21:35:45.224 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "total": 64, 2023-05-31T21:35:45.224 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "complete": 64 2023-05-31T21:35:45.225 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2023-05-31T21:35:45.225 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "current_time": "2023-05-31T21:30:08Z" 2023-05-31T21:35:45.225 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2023-05-31T21:35:45.225 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current data sync status={ 2023-05-31T21:35:45.226 INFO:tasks.rgw_multisite_tests: "sync_status": { 2023-05-31T21:35:45.226 INFO:tasks.rgw_multisite_tests: "info": { 2023-05-31T21:35:45.226 INFO:tasks.rgw_multisite_tests: "status": "sync", 2023-05-31T21:35:45.226 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2023-05-31T21:35:45.226 INFO:tasks.rgw_multisite_tests: "instance_id": 452136274422638824 2023-05-31T21:35:45.227 INFO:tasks.rgw_multisite_tests: }, 2023-05-31T21:35:45.227 INFO:tasks.rgw_multisite_tests: "markers": [ 2023-05-31T21:35:45.227 INFO:tasks.rgw_multisite_tests: { 2023-05-31T21:35:45.227 INFO:tasks.rgw_multisite_tests: "key": 0, 2023-05-31T21:35:45.228 INFO:tasks.rgw_multisite_tests: "val": { 2023-05-31T21:35:45.228 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2023-05-31T21:35:45.228 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000045375", 2023-05-31T21:35:45.228 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-05-31T21:35:45.229 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-05-31T21:35:45.229 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-05-31T21:35:45.229 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-05-31T21:28:16.422581Z" 2023-05-31T21:35:45.229 INFO:tasks.rgw_multisite_tests: } 2023-05-31T21:35:45.229 INFO:tasks.rgw_multisite_tests: }, 2023-05-31T21:35:45.230 INFO:tasks.rgw_multisite_tests: { 2023-05-31T21:35:45.230 INFO:tasks.rgw_multisite_tests: "key": 1, 2023-05-31T21:35:45.230 INFO:tasks.rgw_multisite_tests: "val": { 2023-05-31T21:35:45.230 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2023-05-31T21:35:45.231 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000031780", 2023-05-31T21:35:45.231 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-05-31T21:35:45.231 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-05-31T21:35:45.231 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-05-31T21:35:45.232 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-05-31T21:28:16.501293Z" 2023-05-31T21:35:45.232 INFO:tasks.rgw_multisite_tests: } 2023-05-31T21:35:45.232 INFO:tasks.rgw_multisite_tests: }, 2023-05-31T21:35:45.232 INFO:tasks.rgw_multisite_tests: { 2023-05-31T21:35:45.232 INFO:tasks.rgw_multisite_tests: "key": 2, 2023-05-31T21:35:45.233 INFO:tasks.rgw_multisite_tests: "val": { 2023-05-31T21:35:45.233 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2023-05-31T21:35:45.233 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000051671", 2023-05-31T21:35:45.233 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-05-31T21:35:45.234 INFO:tasks.rgw_multisite_tests: "total_entries": 55, 2023-05-31T21:35:45.234 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-05-31T21:35:45.234 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-05-31T21:28:16.476487Z" 2023-05-31T21:35:45.234 INFO:tasks.rgw_multisite_tests: } 2023-05-31T21:35:45.235 INFO:tasks.rgw_multisite_tests: }, 2023-05-31T21:35:45.235 INFO:tasks.rgw_multisite_tests: { 2023-05-31T21:35:45.235 INFO:tasks.rgw_multisite_tests: "key": 3, 2023-05-31T21:35:45.235 INFO:tasks.rgw_multisite_tests: "val": { 2023-05-31T21:35:45.235 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2023-05-31T21:35:45.236 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000044178", 2023-05-31T21:35:45.236 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-05-31T21:35:45.236 INFO:tasks.rgw_multisite_tests: "total_entries": 9, 2023-05-31T21:35:45.236 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-05-31T21:35:45.237 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-05-31T21:27:37.215905Z" 2023-05-31T21:35:45.237 INFO:tasks.rgw_multisite_tests: } 2023-05-31T21:35:45.237 INFO:tasks.rgw_multisite_tests: } 2023-05-31T21:35:45.237 INFO:tasks.rgw_multisite_tests: ] 2023-05-31T21:35:45.238 INFO:tasks.rgw_multisite_tests: }, 2023-05-31T21:35:45.238 INFO:tasks.rgw_multisite_tests: "full_sync": { 2023-05-31T21:35:45.238 INFO:tasks.rgw_multisite_tests: "total": 64, 2023-05-31T21:35:45.238 INFO:tasks.rgw_multisite_tests: "complete": 64 2023-05-31T21:35:45.239 INFO:tasks.rgw_multisite_tests: }, 2023-05-31T21:35:45.239 INFO:tasks.rgw_multisite_tests: "current_time": "2023-05-31T21:30:08Z" 2023-05-31T21:35:45.239 INFO:tasks.rgw_multisite_tests:} 2023-05-31T21:35:45.239 INFO:tasks.rgw_multisite_tests: 2023-05-31T21:35:45.240 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000045375', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-05-31T21:28:16.422581Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000031780', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-05-31T21:28:16.501293Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000051671', 'next_step_marker': '', 'total_entries': 55, 'pos': 0, 'timestamp': '2023-05-31T21:28:16.476487Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000044178', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2023-05-31T21:27:37.215905Z'}}] 2023-05-31T21:35:45.240 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '00000000000000000000:00000000000000045515', 1: '00000000000000000000:00000000000000031920', 2: '00000000000000000000:00000000000000051951', 3: '00000000000000000000:00000000000000044178'} 2023-05-31T21:35:45.240 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '00000000000000000000:00000000000000045375', 1: '00000000000000000000:00000000000000031780', 2: '00000000000000000000:00000000000000051671', 3: '00000000000000000000:00000000000000044178'} 2023-05-31T21:35:45.240 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: data of zone a2 behind zone a1: shard=0 master=00000000000000000000:00000000000000045515 target=00000000000000000000:00000000000000045375, shard=1 master=00000000000000000000:00000000000000031920 target=00000000000000000000:00000000000000031780, shard=2 master=00000000000000000000:00000000000000051951 target=00000000000000000000:00000000000000051671 2023-05-31T21:35:45.241 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-31T21:35:45.241 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-31T21:35:45.241 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-05-31T21:35:45.241 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: ignoring --setuser ceph since I am not root 2023-05-31T21:35:45.242 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-05-31T21:35:45.242 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2023-05-31T21:35:45.242 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "sync_status": { 2023-05-31T21:35:45.242 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "info": { 2023-05-31T21:35:45.243 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "status": "sync", 2023-05-31T21:35:45.243 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "num_shards": 4, 2023-05-31T21:35:45.243 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "instance_id": 452136274422638824 2023-05-31T21:35:45.243 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2023-05-31T21:35:45.244 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "markers": [ 2023-05-31T21:35:45.244 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2023-05-31T21:35:45.244 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "key": 0, 2023-05-31T21:35:45.244 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "val": { 2023-05-31T21:35:45.245 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "status": "incremental-sync", 2023-05-31T21:35:45.245 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "marker": "00000000000000000000:00000000000000045375", 2023-05-31T21:35:45.245 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "next_step_marker": "", 2023-05-31T21:35:45.245 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "total_entries": 0, 2023-05-31T21:35:45.246 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "pos": 0, 2023-05-31T21:35:45.246 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "timestamp": "2023-05-31T21:28:16.422581Z" 2023-05-31T21:35:45.246 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2023-05-31T21:35:45.246 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2023-05-31T21:35:45.247 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2023-05-31T21:35:45.247 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "key": 1, 2023-05-31T21:35:45.247 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "val": { 2023-05-31T21:35:45.247 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "status": "incremental-sync", 2023-05-31T21:35:45.248 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "marker": "00000000000000000000:00000000000000031780", 2023-05-31T21:35:45.248 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "next_step_marker": "", 2023-05-31T21:35:45.248 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "total_entries": 0, 2023-05-31T21:35:45.248 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "pos": 0, 2023-05-31T21:35:45.249 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "timestamp": "2023-05-31T21:28:16.501293Z" 2023-05-31T21:35:45.249 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2023-05-31T21:35:45.249 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2023-05-31T21:35:45.249 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2023-05-31T21:35:45.249 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "key": 2, 2023-05-31T21:35:45.250 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "val": { 2023-05-31T21:35:45.250 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "status": "incremental-sync", 2023-05-31T21:35:45.250 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "marker": "00000000000000000000:00000000000000051671", 2023-05-31T21:35:45.251 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "next_step_marker": "", 2023-05-31T21:35:45.251 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "total_entries": 55, 2023-05-31T21:35:45.251 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "pos": 0, 2023-05-31T21:35:45.251 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "timestamp": "2023-05-31T21:28:16.476487Z" 2023-05-31T21:35:45.252 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2023-05-31T21:35:45.252 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2023-05-31T21:35:45.252 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2023-05-31T21:35:45.253 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "key": 3, 2023-05-31T21:35:45.253 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "val": { 2023-05-31T21:35:45.253 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "status": "incremental-sync", 2023-05-31T21:35:45.253 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "marker": "00000000000000000000:00000000000000044178", 2023-05-31T21:35:45.254 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "next_step_marker": "", 2023-05-31T21:35:45.254 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "total_entries": 9, 2023-05-31T21:35:45.254 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "pos": 0, 2023-05-31T21:35:45.254 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "timestamp": "2023-05-31T21:27:37.215905Z" 2023-05-31T21:35:45.254 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2023-05-31T21:35:45.255 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2023-05-31T21:35:45.255 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: ] 2023-05-31T21:35:45.255 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2023-05-31T21:35:45.255 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "full_sync": { 2023-05-31T21:35:45.256 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "total": 64, 2023-05-31T21:35:45.256 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "complete": 64 2023-05-31T21:35:45.256 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2023-05-31T21:35:45.256 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "current_time": "2023-05-31T21:30:13Z" 2023-05-31T21:35:45.257 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2023-05-31T21:35:45.257 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current data sync status={ 2023-05-31T21:35:45.257 INFO:tasks.rgw_multisite_tests: "sync_status": { 2023-05-31T21:35:45.257 INFO:tasks.rgw_multisite_tests: "info": { 2023-05-31T21:35:45.257 INFO:tasks.rgw_multisite_tests: "status": "sync", 2023-05-31T21:35:45.258 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2023-05-31T21:35:45.258 INFO:tasks.rgw_multisite_tests: "instance_id": 452136274422638824 2023-05-31T21:35:45.258 INFO:tasks.rgw_multisite_tests: }, 2023-05-31T21:35:45.258 INFO:tasks.rgw_multisite_tests: "markers": [ 2023-05-31T21:35:45.259 INFO:tasks.rgw_multisite_tests: { 2023-05-31T21:35:45.259 INFO:tasks.rgw_multisite_tests: "key": 0, 2023-05-31T21:35:45.259 INFO:tasks.rgw_multisite_tests: "val": { 2023-05-31T21:35:45.259 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2023-05-31T21:35:45.260 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000045375", 2023-05-31T21:35:45.260 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-05-31T21:35:45.260 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-05-31T21:35:45.260 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-05-31T21:35:45.261 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-05-31T21:28:16.422581Z" 2023-05-31T21:35:45.261 INFO:tasks.rgw_multisite_tests: } 2023-05-31T21:35:45.261 INFO:tasks.rgw_multisite_tests: }, 2023-05-31T21:35:45.261 INFO:tasks.rgw_multisite_tests: { 2023-05-31T21:35:45.261 INFO:tasks.rgw_multisite_tests: "key": 1, 2023-05-31T21:35:45.262 INFO:tasks.rgw_multisite_tests: "val": { 2023-05-31T21:35:45.262 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2023-05-31T21:35:45.262 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000031780", 2023-05-31T21:35:45.262 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-05-31T21:35:45.263 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-05-31T21:35:45.263 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-05-31T21:35:45.263 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-05-31T21:28:16.501293Z" 2023-05-31T21:35:45.263 INFO:tasks.rgw_multisite_tests: } 2023-05-31T21:35:45.264 INFO:tasks.rgw_multisite_tests: }, 2023-05-31T21:35:45.264 INFO:tasks.rgw_multisite_tests: { 2023-05-31T21:35:45.264 INFO:tasks.rgw_multisite_tests: "key": 2, 2023-05-31T21:35:45.264 INFO:tasks.rgw_multisite_tests: "val": { 2023-05-31T21:35:45.264 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2023-05-31T21:35:45.265 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000051671", 2023-05-31T21:35:45.265 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-05-31T21:35:45.265 INFO:tasks.rgw_multisite_tests: "total_entries": 55, 2023-05-31T21:35:45.265 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-05-31T21:35:45.266 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-05-31T21:28:16.476487Z" 2023-05-31T21:35:45.266 INFO:tasks.rgw_multisite_tests: } 2023-05-31T21:35:45.266 INFO:tasks.rgw_multisite_tests: }, 2023-05-31T21:35:45.266 INFO:tasks.rgw_multisite_tests: { 2023-05-31T21:35:45.267 INFO:tasks.rgw_multisite_tests: "key": 3, 2023-05-31T21:35:45.267 INFO:tasks.rgw_multisite_tests: "val": { 2023-05-31T21:35:45.267 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2023-05-31T21:35:45.267 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000044178", 2023-05-31T21:35:45.267 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-05-31T21:35:45.268 INFO:tasks.rgw_multisite_tests: "total_entries": 9, 2023-05-31T21:35:45.268 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-05-31T21:35:45.268 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-05-31T21:27:37.215905Z" 2023-05-31T21:35:45.268 INFO:tasks.rgw_multisite_tests: } 2023-05-31T21:35:45.269 INFO:tasks.rgw_multisite_tests: } 2023-05-31T21:35:45.269 INFO:tasks.rgw_multisite_tests: ] 2023-05-31T21:35:45.269 INFO:tasks.rgw_multisite_tests: }, 2023-05-31T21:35:45.269 INFO:tasks.rgw_multisite_tests: "full_sync": { 2023-05-31T21:35:45.270 INFO:tasks.rgw_multisite_tests: "total": 64, 2023-05-31T21:35:45.270 INFO:tasks.rgw_multisite_tests: "complete": 64 2023-05-31T21:35:45.270 INFO:tasks.rgw_multisite_tests: }, 2023-05-31T21:35:45.270 INFO:tasks.rgw_multisite_tests: "current_time": "2023-05-31T21:30:13Z" 2023-05-31T21:35:45.271 INFO:tasks.rgw_multisite_tests:} 2023-05-31T21:35:45.271 INFO:tasks.rgw_multisite_tests: 2023-05-31T21:35:45.271 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000045375', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-05-31T21:28:16.422581Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000031780', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-05-31T21:28:16.501293Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000051671', 'next_step_marker': '', 'total_entries': 55, 'pos': 0, 'timestamp': '2023-05-31T21:28:16.476487Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000044178', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2023-05-31T21:27:37.215905Z'}}] 2023-05-31T21:35:45.271 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '00000000000000000000:00000000000000045515', 1: '00000000000000000000:00000000000000031920', 2: '00000000000000000000:00000000000000051951', 3: '00000000000000000000:00000000000000044178'} 2023-05-31T21:35:45.272 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '00000000000000000000:00000000000000045375', 1: '00000000000000000000:00000000000000031780', 2: '00000000000000000000:00000000000000051671', 3: '00000000000000000000:00000000000000044178'} 2023-05-31T21:35:45.272 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: data of zone a2 behind zone a1: shard=0 master=00000000000000000000:00000000000000045515 target=00000000000000000000:00000000000000045375, shard=1 master=00000000000000000000:00000000000000031920 target=00000000000000000000:00000000000000031780, shard=2 master=00000000000000000000:00000000000000051951 target=00000000000000000000:00000000000000051671 2023-05-31T21:35:45.272 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-31T21:35:45.272 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-31T21:35:45.272 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-05-31T21:35:45.273 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: ignoring --setuser ceph since I am not root 2023-05-31T21:35:45.273 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-05-31T21:35:45.273 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2023-05-31T21:35:45.273 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "sync_status": { 2023-05-31T21:35:45.274 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "info": { 2023-05-31T21:35:45.274 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "status": "sync", 2023-05-31T21:35:45.274 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "num_shards": 4, 2023-05-31T21:35:45.274 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "instance_id": 452136274422638824 2023-05-31T21:35:45.275 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2023-05-31T21:35:45.275 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "markers": [ 2023-05-31T21:35:45.275 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2023-05-31T21:35:45.275 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "key": 0, 2023-05-31T21:35:45.275 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "val": { 2023-05-31T21:35:45.276 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "status": "incremental-sync", 2023-05-31T21:35:45.276 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "marker": "00000000000000000000:00000000000000045375", 2023-05-31T21:35:45.276 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "next_step_marker": "", 2023-05-31T21:35:45.276 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "total_entries": 0, 2023-05-31T21:35:45.277 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "pos": 0, 2023-05-31T21:35:45.277 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "timestamp": "2023-05-31T21:28:16.422581Z" 2023-05-31T21:35:45.277 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2023-05-31T21:35:45.277 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2023-05-31T21:35:45.278 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2023-05-31T21:35:45.278 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "key": 1, 2023-05-31T21:35:45.278 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "val": { 2023-05-31T21:35:45.278 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "status": "incremental-sync", 2023-05-31T21:35:45.279 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "marker": "00000000000000000000:00000000000000031780", 2023-05-31T21:35:45.279 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "next_step_marker": "", 2023-05-31T21:35:45.279 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "total_entries": 0, 2023-05-31T21:35:45.279 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "pos": 0, 2023-05-31T21:35:45.279 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "timestamp": "2023-05-31T21:28:16.501293Z" 2023-05-31T21:35:45.280 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2023-05-31T21:35:45.280 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2023-05-31T21:35:45.280 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2023-05-31T21:35:45.280 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "key": 2, 2023-05-31T21:35:45.281 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "val": { 2023-05-31T21:35:45.281 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "status": "incremental-sync", 2023-05-31T21:35:45.281 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "marker": "00000000000000000000:00000000000000051671", 2023-05-31T21:35:45.281 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "next_step_marker": "", 2023-05-31T21:35:45.282 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "total_entries": 55, 2023-05-31T21:35:45.282 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "pos": 0, 2023-05-31T21:35:45.282 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "timestamp": "2023-05-31T21:28:16.476487Z" 2023-05-31T21:35:45.282 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2023-05-31T21:35:45.283 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2023-05-31T21:35:45.283 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2023-05-31T21:35:45.283 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "key": 3, 2023-05-31T21:35:45.283 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "val": { 2023-05-31T21:35:45.283 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "status": "incremental-sync", 2023-05-31T21:35:45.284 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "marker": "00000000000000000000:00000000000000044178", 2023-05-31T21:35:45.284 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "next_step_marker": "", 2023-05-31T21:35:45.284 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "total_entries": 9, 2023-05-31T21:35:45.284 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "pos": 0, 2023-05-31T21:35:45.285 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "timestamp": "2023-05-31T21:27:37.215905Z" 2023-05-31T21:35:45.285 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2023-05-31T21:35:45.285 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2023-05-31T21:35:45.285 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: ] 2023-05-31T21:35:45.286 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2023-05-31T21:35:45.286 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "full_sync": { 2023-05-31T21:35:45.286 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "total": 64, 2023-05-31T21:35:45.286 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "complete": 64 2023-05-31T21:35:45.287 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2023-05-31T21:35:45.287 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "current_time": "2023-05-31T21:30:18Z" 2023-05-31T21:35:45.287 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2023-05-31T21:35:45.287 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current data sync status={ 2023-05-31T21:35:45.287 INFO:tasks.rgw_multisite_tests: "sync_status": { 2023-05-31T21:35:45.288 INFO:tasks.rgw_multisite_tests: "info": { 2023-05-31T21:35:45.288 INFO:tasks.rgw_multisite_tests: "status": "sync", 2023-05-31T21:35:45.288 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2023-05-31T21:35:45.288 INFO:tasks.rgw_multisite_tests: "instance_id": 452136274422638824 2023-05-31T21:35:45.289 INFO:tasks.rgw_multisite_tests: }, 2023-05-31T21:35:45.289 INFO:tasks.rgw_multisite_tests: "markers": [ 2023-05-31T21:35:45.289 INFO:tasks.rgw_multisite_tests: { 2023-05-31T21:35:45.289 INFO:tasks.rgw_multisite_tests: "key": 0, 2023-05-31T21:35:45.290 INFO:tasks.rgw_multisite_tests: "val": { 2023-05-31T21:35:45.290 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2023-05-31T21:35:45.290 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000045375", 2023-05-31T21:35:45.290 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-05-31T21:35:45.290 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-05-31T21:35:45.291 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-05-31T21:35:45.291 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-05-31T21:28:16.422581Z" 2023-05-31T21:35:45.291 INFO:tasks.rgw_multisite_tests: } 2023-05-31T21:35:45.291 INFO:tasks.rgw_multisite_tests: }, 2023-05-31T21:35:45.292 INFO:tasks.rgw_multisite_tests: { 2023-05-31T21:35:45.292 INFO:tasks.rgw_multisite_tests: "key": 1, 2023-05-31T21:35:45.292 INFO:tasks.rgw_multisite_tests: "val": { 2023-05-31T21:35:45.293 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2023-05-31T21:35:45.293 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000031780", 2023-05-31T21:35:45.293 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-05-31T21:35:45.293 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-05-31T21:35:45.294 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-05-31T21:35:45.294 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-05-31T21:28:16.501293Z" 2023-05-31T21:35:45.294 INFO:tasks.rgw_multisite_tests: } 2023-05-31T21:35:45.294 INFO:tasks.rgw_multisite_tests: }, 2023-05-31T21:35:45.295 INFO:tasks.rgw_multisite_tests: { 2023-05-31T21:35:45.295 INFO:tasks.rgw_multisite_tests: "key": 2, 2023-05-31T21:35:45.295 INFO:tasks.rgw_multisite_tests: "val": { 2023-05-31T21:35:45.295 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2023-05-31T21:35:45.296 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000051671", 2023-05-31T21:35:45.296 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-05-31T21:35:45.296 INFO:tasks.rgw_multisite_tests: "total_entries": 55, 2023-05-31T21:35:45.296 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-05-31T21:35:45.296 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-05-31T21:28:16.476487Z" 2023-05-31T21:35:45.297 INFO:tasks.rgw_multisite_tests: } 2023-05-31T21:35:45.297 INFO:tasks.rgw_multisite_tests: }, 2023-05-31T21:35:45.297 INFO:tasks.rgw_multisite_tests: { 2023-05-31T21:35:45.297 INFO:tasks.rgw_multisite_tests: "key": 3, 2023-05-31T21:35:45.298 INFO:tasks.rgw_multisite_tests: "val": { 2023-05-31T21:35:45.298 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2023-05-31T21:35:45.298 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000044178", 2023-05-31T21:35:45.298 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-05-31T21:35:45.299 INFO:tasks.rgw_multisite_tests: "total_entries": 9, 2023-05-31T21:35:45.299 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-05-31T21:35:45.299 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-05-31T21:27:37.215905Z" 2023-05-31T21:35:45.299 INFO:tasks.rgw_multisite_tests: } 2023-05-31T21:35:45.299 INFO:tasks.rgw_multisite_tests: } 2023-05-31T21:35:45.300 INFO:tasks.rgw_multisite_tests: ] 2023-05-31T21:35:45.300 INFO:tasks.rgw_multisite_tests: }, 2023-05-31T21:35:45.300 INFO:tasks.rgw_multisite_tests: "full_sync": { 2023-05-31T21:35:45.300 INFO:tasks.rgw_multisite_tests: "total": 64, 2023-05-31T21:35:45.301 INFO:tasks.rgw_multisite_tests: "complete": 64 2023-05-31T21:35:45.301 INFO:tasks.rgw_multisite_tests: }, 2023-05-31T21:35:45.301 INFO:tasks.rgw_multisite_tests: "current_time": "2023-05-31T21:30:18Z" 2023-05-31T21:35:45.301 INFO:tasks.rgw_multisite_tests:} 2023-05-31T21:35:45.302 INFO:tasks.rgw_multisite_tests: 2023-05-31T21:35:45.302 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000045375', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-05-31T21:28:16.422581Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000031780', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-05-31T21:28:16.501293Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000051671', 'next_step_marker': '', 'total_entries': 55, 'pos': 0, 'timestamp': '2023-05-31T21:28:16.476487Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000044178', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2023-05-31T21:27:37.215905Z'}}] 2023-05-31T21:35:45.302 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '00000000000000000000:00000000000000045515', 1: '00000000000000000000:00000000000000031920', 2: '00000000000000000000:00000000000000051951', 3: '00000000000000000000:00000000000000044178'} 2023-05-31T21:35:45.302 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '00000000000000000000:00000000000000045375', 1: '00000000000000000000:00000000000000031780', 2: '00000000000000000000:00000000000000051671', 3: '00000000000000000000:00000000000000044178'} 2023-05-31T21:35:45.303 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: data of zone a2 behind zone a1: shard=0 master=00000000000000000000:00000000000000045515 target=00000000000000000000:00000000000000045375, shard=1 master=00000000000000000000:00000000000000031920 target=00000000000000000000:00000000000000031780, shard=2 master=00000000000000000000:00000000000000051951 target=00000000000000000000:00000000000000051671 2023-05-31T21:35:45.303 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-31T21:35:45.303 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-31T21:35:45.303 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-05-31T21:35:45.303 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: ignoring --setuser ceph since I am not root 2023-05-31T21:35:45.304 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-05-31T21:35:45.304 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2023-05-31T21:35:45.304 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "sync_status": { 2023-05-31T21:35:45.304 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "info": { 2023-05-31T21:35:45.305 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "status": "sync", 2023-05-31T21:35:45.305 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "num_shards": 4, 2023-05-31T21:35:45.305 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "instance_id": 452136274422638824 2023-05-31T21:35:45.305 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2023-05-31T21:35:45.306 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "markers": [ 2023-05-31T21:35:45.306 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2023-05-31T21:35:45.306 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "key": 0, 2023-05-31T21:35:45.306 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "val": { 2023-05-31T21:35:45.307 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "status": "incremental-sync", 2023-05-31T21:35:45.307 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "marker": "00000000000000000000:00000000000000045375", 2023-05-31T21:35:45.307 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "next_step_marker": "", 2023-05-31T21:35:45.307 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "total_entries": 0, 2023-05-31T21:35:45.307 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "pos": 0, 2023-05-31T21:35:45.308 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "timestamp": "2023-05-31T21:28:16.422581Z" 2023-05-31T21:35:45.308 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2023-05-31T21:35:45.308 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2023-05-31T21:35:45.308 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2023-05-31T21:35:45.309 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "key": 1, 2023-05-31T21:35:45.309 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "val": { 2023-05-31T21:35:45.309 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "status": "incremental-sync", 2023-05-31T21:35:45.309 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "marker": "00000000000000000000:00000000000000031780", 2023-05-31T21:35:45.310 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "next_step_marker": "", 2023-05-31T21:35:45.310 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "total_entries": 0, 2023-05-31T21:35:45.310 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "pos": 0, 2023-05-31T21:35:45.310 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "timestamp": "2023-05-31T21:28:16.501293Z" 2023-05-31T21:35:45.310 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2023-05-31T21:35:45.311 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2023-05-31T21:35:45.311 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2023-05-31T21:35:45.311 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "key": 2, 2023-05-31T21:35:45.311 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "val": { 2023-05-31T21:35:45.312 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "status": "incremental-sync", 2023-05-31T21:35:45.312 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "marker": "00000000000000000000:00000000000000051671", 2023-05-31T21:35:45.312 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "next_step_marker": "", 2023-05-31T21:35:45.312 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "total_entries": 55, 2023-05-31T21:35:45.313 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "pos": 0, 2023-05-31T21:35:45.313 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "timestamp": "2023-05-31T21:28:16.476487Z" 2023-05-31T21:35:45.313 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2023-05-31T21:35:45.313 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2023-05-31T21:35:45.313 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2023-05-31T21:35:45.314 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "key": 3, 2023-05-31T21:35:45.314 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "val": { 2023-05-31T21:35:45.314 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "status": "incremental-sync", 2023-05-31T21:35:45.314 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "marker": "00000000000000000000:00000000000000044178", 2023-05-31T21:35:45.315 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "next_step_marker": "", 2023-05-31T21:35:45.315 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "total_entries": 9, 2023-05-31T21:35:45.315 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "pos": 0, 2023-05-31T21:35:45.315 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "timestamp": "2023-05-31T21:27:37.215905Z" 2023-05-31T21:35:45.316 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2023-05-31T21:35:45.316 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2023-05-31T21:35:45.316 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: ] 2023-05-31T21:35:45.316 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2023-05-31T21:35:45.317 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "full_sync": { 2023-05-31T21:35:45.317 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "total": 64, 2023-05-31T21:35:45.317 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "complete": 64 2023-05-31T21:35:45.317 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2023-05-31T21:35:45.317 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "current_time": "2023-05-31T21:30:23Z" 2023-05-31T21:35:45.318 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2023-05-31T21:35:45.318 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current data sync status={ 2023-05-31T21:35:45.318 INFO:tasks.rgw_multisite_tests: "sync_status": { 2023-05-31T21:35:45.318 INFO:tasks.rgw_multisite_tests: "info": { 2023-05-31T21:35:45.319 INFO:tasks.rgw_multisite_tests: "status": "sync", 2023-05-31T21:35:45.319 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2023-05-31T21:35:45.319 INFO:tasks.rgw_multisite_tests: "instance_id": 452136274422638824 2023-05-31T21:35:45.319 INFO:tasks.rgw_multisite_tests: }, 2023-05-31T21:35:45.319 INFO:tasks.rgw_multisite_tests: "markers": [ 2023-05-31T21:35:45.320 INFO:tasks.rgw_multisite_tests: { 2023-05-31T21:35:45.320 INFO:tasks.rgw_multisite_tests: "key": 0, 2023-05-31T21:35:45.320 INFO:tasks.rgw_multisite_tests: "val": { 2023-05-31T21:35:45.320 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2023-05-31T21:35:45.321 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000045375", 2023-05-31T21:35:45.321 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-05-31T21:35:45.321 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-05-31T21:35:45.321 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-05-31T21:35:45.322 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-05-31T21:28:16.422581Z" 2023-05-31T21:35:45.322 INFO:tasks.rgw_multisite_tests: } 2023-05-31T21:35:45.322 INFO:tasks.rgw_multisite_tests: }, 2023-05-31T21:35:45.322 INFO:tasks.rgw_multisite_tests: { 2023-05-31T21:35:45.323 INFO:tasks.rgw_multisite_tests: "key": 1, 2023-05-31T21:35:45.323 INFO:tasks.rgw_multisite_tests: "val": { 2023-05-31T21:35:45.323 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2023-05-31T21:35:45.323 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000031780", 2023-05-31T21:35:45.323 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-05-31T21:35:45.324 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-05-31T21:35:45.324 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-05-31T21:35:45.324 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-05-31T21:28:16.501293Z" 2023-05-31T21:35:45.324 INFO:tasks.rgw_multisite_tests: } 2023-05-31T21:35:45.325 INFO:tasks.rgw_multisite_tests: }, 2023-05-31T21:35:45.325 INFO:tasks.rgw_multisite_tests: { 2023-05-31T21:35:45.325 INFO:tasks.rgw_multisite_tests: "key": 2, 2023-05-31T21:35:45.325 INFO:tasks.rgw_multisite_tests: "val": { 2023-05-31T21:35:45.326 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2023-05-31T21:35:45.326 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000051671", 2023-05-31T21:35:45.326 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-05-31T21:35:45.326 INFO:tasks.rgw_multisite_tests: "total_entries": 55, 2023-05-31T21:35:45.326 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-05-31T21:35:45.327 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-05-31T21:28:16.476487Z" 2023-05-31T21:35:45.327 INFO:tasks.rgw_multisite_tests: } 2023-05-31T21:35:45.327 INFO:tasks.rgw_multisite_tests: }, 2023-05-31T21:35:45.327 INFO:tasks.rgw_multisite_tests: { 2023-05-31T21:35:45.328 INFO:tasks.rgw_multisite_tests: "key": 3, 2023-05-31T21:35:45.328 INFO:tasks.rgw_multisite_tests: "val": { 2023-05-31T21:35:45.328 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2023-05-31T21:35:45.328 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000044178", 2023-05-31T21:35:45.329 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-05-31T21:35:45.329 INFO:tasks.rgw_multisite_tests: "total_entries": 9, 2023-05-31T21:35:45.329 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-05-31T21:35:45.329 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-05-31T21:27:37.215905Z" 2023-05-31T21:35:45.329 INFO:tasks.rgw_multisite_tests: } 2023-05-31T21:35:45.330 INFO:tasks.rgw_multisite_tests: } 2023-05-31T21:35:45.330 INFO:tasks.rgw_multisite_tests: ] 2023-05-31T21:35:45.330 INFO:tasks.rgw_multisite_tests: }, 2023-05-31T21:35:45.330 INFO:tasks.rgw_multisite_tests: "full_sync": { 2023-05-31T21:35:45.331 INFO:tasks.rgw_multisite_tests: "total": 64, 2023-05-31T21:35:45.331 INFO:tasks.rgw_multisite_tests: "complete": 64 2023-05-31T21:35:45.331 INFO:tasks.rgw_multisite_tests: }, 2023-05-31T21:35:45.331 INFO:tasks.rgw_multisite_tests: "current_time": "2023-05-31T21:30:23Z" 2023-05-31T21:35:45.332 INFO:tasks.rgw_multisite_tests:} 2023-05-31T21:35:45.332 INFO:tasks.rgw_multisite_tests: 2023-05-31T21:35:45.332 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000045375', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-05-31T21:28:16.422581Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000031780', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-05-31T21:28:16.501293Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000051671', 'next_step_marker': '', 'total_entries': 55, 'pos': 0, 'timestamp': '2023-05-31T21:28:16.476487Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000044178', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2023-05-31T21:27:37.215905Z'}}] 2023-05-31T21:35:45.332 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '00000000000000000000:00000000000000045515', 1: '00000000000000000000:00000000000000031920', 2: '00000000000000000000:00000000000000051951', 3: '00000000000000000000:00000000000000044178'} 2023-05-31T21:35:45.333 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '00000000000000000000:00000000000000045375', 1: '00000000000000000000:00000000000000031780', 2: '00000000000000000000:00000000000000051671', 3: '00000000000000000000:00000000000000044178'} 2023-05-31T21:35:45.333 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: data of zone a2 behind zone a1: shard=0 master=00000000000000000000:00000000000000045515 target=00000000000000000000:00000000000000045375, shard=1 master=00000000000000000000:00000000000000031920 target=00000000000000000000:00000000000000031780, shard=2 master=00000000000000000000:00000000000000051951 target=00000000000000000000:00000000000000051671 2023-05-31T21:35:45.333 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-31T21:35:45.333 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-31T21:35:45.333 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-05-31T21:35:45.334 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: ignoring --setuser ceph since I am not root 2023-05-31T21:35:45.334 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-05-31T21:35:45.334 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2023-05-31T21:35:45.334 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "sync_status": { 2023-05-31T21:35:45.335 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "info": { 2023-05-31T21:35:45.335 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "status": "sync", 2023-05-31T21:35:45.335 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "num_shards": 4, 2023-05-31T21:35:45.335 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "instance_id": 452136274422638824 2023-05-31T21:35:45.336 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2023-05-31T21:35:45.336 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "markers": [ 2023-05-31T21:35:45.336 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2023-05-31T21:35:45.336 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "key": 0, 2023-05-31T21:35:45.337 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "val": { 2023-05-31T21:35:45.337 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "status": "incremental-sync", 2023-05-31T21:35:45.337 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "marker": "00000000000000000000:00000000000000045375", 2023-05-31T21:35:45.337 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "next_step_marker": "", 2023-05-31T21:35:45.337 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "total_entries": 0, 2023-05-31T21:35:45.338 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "pos": 0, 2023-05-31T21:35:45.338 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "timestamp": "2023-05-31T21:28:16.422581Z" 2023-05-31T21:35:45.338 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2023-05-31T21:35:45.338 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2023-05-31T21:35:45.339 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2023-05-31T21:35:45.339 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "key": 1, 2023-05-31T21:35:45.339 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "val": { 2023-05-31T21:35:45.339 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "status": "incremental-sync", 2023-05-31T21:35:45.340 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "marker": "00000000000000000000:00000000000000031780", 2023-05-31T21:35:45.340 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "next_step_marker": "", 2023-05-31T21:35:45.340 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "total_entries": 0, 2023-05-31T21:35:45.340 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "pos": 0, 2023-05-31T21:35:45.340 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "timestamp": "2023-05-31T21:28:16.501293Z" 2023-05-31T21:35:45.341 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2023-05-31T21:35:45.341 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2023-05-31T21:35:45.341 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2023-05-31T21:35:45.341 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "key": 2, 2023-05-31T21:35:45.342 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "val": { 2023-05-31T21:35:45.342 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "status": "incremental-sync", 2023-05-31T21:35:45.342 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "marker": "00000000000000000000:00000000000000051671", 2023-05-31T21:35:45.342 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "next_step_marker": "", 2023-05-31T21:35:45.343 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "total_entries": 55, 2023-05-31T21:35:45.343 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "pos": 0, 2023-05-31T21:35:45.343 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "timestamp": "2023-05-31T21:28:16.476487Z" 2023-05-31T21:35:45.343 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2023-05-31T21:35:45.344 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2023-05-31T21:35:45.344 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2023-05-31T21:35:45.344 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "key": 3, 2023-05-31T21:35:45.344 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "val": { 2023-05-31T21:35:45.344 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "status": "incremental-sync", 2023-05-31T21:35:45.345 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "marker": "00000000000000000000:00000000000000044178", 2023-05-31T21:35:45.345 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "next_step_marker": "", 2023-05-31T21:35:45.345 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "total_entries": 9, 2023-05-31T21:35:45.345 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "pos": 0, 2023-05-31T21:35:45.346 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "timestamp": "2023-05-31T21:27:37.215905Z" 2023-05-31T21:35:45.346 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2023-05-31T21:35:45.346 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2023-05-31T21:35:45.346 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: ] 2023-05-31T21:35:45.347 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2023-05-31T21:35:45.347 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "full_sync": { 2023-05-31T21:35:45.347 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "total": 64, 2023-05-31T21:35:45.347 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "complete": 64 2023-05-31T21:35:45.347 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2023-05-31T21:35:45.348 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "current_time": "2023-05-31T21:30:28Z" 2023-05-31T21:35:45.348 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2023-05-31T21:35:45.348 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current data sync status={ 2023-05-31T21:35:45.348 INFO:tasks.rgw_multisite_tests: "sync_status": { 2023-05-31T21:35:45.349 INFO:tasks.rgw_multisite_tests: "info": { 2023-05-31T21:35:45.349 INFO:tasks.rgw_multisite_tests: "status": "sync", 2023-05-31T21:35:45.349 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2023-05-31T21:35:45.349 INFO:tasks.rgw_multisite_tests: "instance_id": 452136274422638824 2023-05-31T21:35:45.350 INFO:tasks.rgw_multisite_tests: }, 2023-05-31T21:35:45.350 INFO:tasks.rgw_multisite_tests: "markers": [ 2023-05-31T21:35:45.350 INFO:tasks.rgw_multisite_tests: { 2023-05-31T21:35:45.350 INFO:tasks.rgw_multisite_tests: "key": 0, 2023-05-31T21:35:45.351 INFO:tasks.rgw_multisite_tests: "val": { 2023-05-31T21:35:45.351 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2023-05-31T21:35:45.351 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000045375", 2023-05-31T21:35:45.351 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-05-31T21:35:45.351 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-05-31T21:35:45.352 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-05-31T21:35:45.352 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-05-31T21:28:16.422581Z" 2023-05-31T21:35:45.352 INFO:tasks.rgw_multisite_tests: } 2023-05-31T21:35:45.353 INFO:tasks.rgw_multisite_tests: }, 2023-05-31T21:35:45.353 INFO:tasks.rgw_multisite_tests: { 2023-05-31T21:35:45.353 INFO:tasks.rgw_multisite_tests: "key": 1, 2023-05-31T21:35:45.353 INFO:tasks.rgw_multisite_tests: "val": { 2023-05-31T21:35:45.353 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2023-05-31T21:35:45.354 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000031780", 2023-05-31T21:35:45.354 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-05-31T21:35:45.354 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-05-31T21:35:45.354 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-05-31T21:35:45.355 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-05-31T21:28:16.501293Z" 2023-05-31T21:35:45.355 INFO:tasks.rgw_multisite_tests: } 2023-05-31T21:35:45.355 INFO:tasks.rgw_multisite_tests: }, 2023-05-31T21:35:45.355 INFO:tasks.rgw_multisite_tests: { 2023-05-31T21:35:45.356 INFO:tasks.rgw_multisite_tests: "key": 2, 2023-05-31T21:35:45.356 INFO:tasks.rgw_multisite_tests: "val": { 2023-05-31T21:35:45.356 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2023-05-31T21:35:45.356 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000051671", 2023-05-31T21:35:45.356 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-05-31T21:35:45.357 INFO:tasks.rgw_multisite_tests: "total_entries": 55, 2023-05-31T21:35:45.357 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-05-31T21:35:45.357 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-05-31T21:28:16.476487Z" 2023-05-31T21:35:45.357 INFO:tasks.rgw_multisite_tests: } 2023-05-31T21:35:45.358 INFO:tasks.rgw_multisite_tests: }, 2023-05-31T21:35:45.358 INFO:tasks.rgw_multisite_tests: { 2023-05-31T21:35:45.358 INFO:tasks.rgw_multisite_tests: "key": 3, 2023-05-31T21:35:45.358 INFO:tasks.rgw_multisite_tests: "val": { 2023-05-31T21:35:45.359 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2023-05-31T21:35:45.359 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000044178", 2023-05-31T21:35:45.359 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-05-31T21:35:45.359 INFO:tasks.rgw_multisite_tests: "total_entries": 9, 2023-05-31T21:35:45.359 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-05-31T21:35:45.360 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-05-31T21:27:37.215905Z" 2023-05-31T21:35:45.360 INFO:tasks.rgw_multisite_tests: } 2023-05-31T21:35:45.360 INFO:tasks.rgw_multisite_tests: } 2023-05-31T21:35:45.360 INFO:tasks.rgw_multisite_tests: ] 2023-05-31T21:35:45.361 INFO:tasks.rgw_multisite_tests: }, 2023-05-31T21:35:45.361 INFO:tasks.rgw_multisite_tests: "full_sync": { 2023-05-31T21:35:45.361 INFO:tasks.rgw_multisite_tests: "total": 64, 2023-05-31T21:35:45.361 INFO:tasks.rgw_multisite_tests: "complete": 64 2023-05-31T21:35:45.362 INFO:tasks.rgw_multisite_tests: }, 2023-05-31T21:35:45.362 INFO:tasks.rgw_multisite_tests: "current_time": "2023-05-31T21:30:28Z" 2023-05-31T21:35:45.362 INFO:tasks.rgw_multisite_tests:} 2023-05-31T21:35:45.362 INFO:tasks.rgw_multisite_tests: 2023-05-31T21:35:45.363 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000045375', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-05-31T21:28:16.422581Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000031780', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-05-31T21:28:16.501293Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000051671', 'next_step_marker': '', 'total_entries': 55, 'pos': 0, 'timestamp': '2023-05-31T21:28:16.476487Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000044178', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2023-05-31T21:27:37.215905Z'}}] 2023-05-31T21:35:45.363 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '00000000000000000000:00000000000000045515', 1: '00000000000000000000:00000000000000031920', 2: '00000000000000000000:00000000000000051951', 3: '00000000000000000000:00000000000000044178'} 2023-05-31T21:35:45.363 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '00000000000000000000:00000000000000045375', 1: '00000000000000000000:00000000000000031780', 2: '00000000000000000000:00000000000000051671', 3: '00000000000000000000:00000000000000044178'} 2023-05-31T21:35:45.363 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: data of zone a2 behind zone a1: shard=0 master=00000000000000000000:00000000000000045515 target=00000000000000000000:00000000000000045375, shard=1 master=00000000000000000000:00000000000000031920 target=00000000000000000000:00000000000000031780, shard=2 master=00000000000000000000:00000000000000051951 target=00000000000000000000:00000000000000051671 2023-05-31T21:35:45.363 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-31T21:35:45.364 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-31T21:35:45.364 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-05-31T21:35:45.364 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: ignoring --setuser ceph since I am not root 2023-05-31T21:35:45.364 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-05-31T21:35:45.365 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2023-05-31T21:35:45.365 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "sync_status": { 2023-05-31T21:35:45.365 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "info": { 2023-05-31T21:35:45.365 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "status": "sync", 2023-05-31T21:35:45.366 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "num_shards": 4, 2023-05-31T21:35:45.366 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "instance_id": 452136274422638824 2023-05-31T21:35:45.366 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2023-05-31T21:35:45.366 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "markers": [ 2023-05-31T21:35:45.367 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2023-05-31T21:35:45.367 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "key": 0, 2023-05-31T21:35:45.367 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "val": { 2023-05-31T21:35:45.367 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "status": "incremental-sync", 2023-05-31T21:35:45.368 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "marker": "00000000000000000000:00000000000000045375", 2023-05-31T21:35:45.368 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "next_step_marker": "", 2023-05-31T21:35:45.368 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "total_entries": 0, 2023-05-31T21:35:45.368 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "pos": 0, 2023-05-31T21:35:45.368 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "timestamp": "2023-05-31T21:28:16.422581Z" 2023-05-31T21:35:45.369 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2023-05-31T21:35:45.369 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2023-05-31T21:35:45.369 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2023-05-31T21:35:45.369 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "key": 1, 2023-05-31T21:35:45.370 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "val": { 2023-05-31T21:35:45.370 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "status": "incremental-sync", 2023-05-31T21:35:45.370 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "marker": "00000000000000000000:00000000000000031780", 2023-05-31T21:35:45.370 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "next_step_marker": "", 2023-05-31T21:35:45.371 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "total_entries": 0, 2023-05-31T21:35:45.371 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "pos": 0, 2023-05-31T21:35:45.371 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "timestamp": "2023-05-31T21:28:16.501293Z" 2023-05-31T21:35:45.371 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2023-05-31T21:35:45.371 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2023-05-31T21:35:45.372 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2023-05-31T21:35:45.372 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "key": 2, 2023-05-31T21:35:45.372 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "val": { 2023-05-31T21:35:45.372 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "status": "incremental-sync", 2023-05-31T21:35:45.373 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "marker": "00000000000000000000:00000000000000051671", 2023-05-31T21:35:45.373 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "next_step_marker": "", 2023-05-31T21:35:45.373 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "total_entries": 55, 2023-05-31T21:35:45.373 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "pos": 0, 2023-05-31T21:35:45.374 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "timestamp": "2023-05-31T21:28:16.476487Z" 2023-05-31T21:35:45.374 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2023-05-31T21:35:45.374 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2023-05-31T21:35:45.374 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2023-05-31T21:35:45.375 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "key": 3, 2023-05-31T21:35:45.375 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "val": { 2023-05-31T21:35:45.375 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "status": "incremental-sync", 2023-05-31T21:35:45.375 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "marker": "00000000000000000000:00000000000000044178", 2023-05-31T21:35:45.375 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "next_step_marker": "", 2023-05-31T21:35:45.376 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "total_entries": 9, 2023-05-31T21:35:45.376 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "pos": 0, 2023-05-31T21:35:45.376 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "timestamp": "2023-05-31T21:27:37.215905Z" 2023-05-31T21:35:45.376 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2023-05-31T21:35:45.377 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2023-05-31T21:35:45.377 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: ] 2023-05-31T21:35:45.377 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2023-05-31T21:35:45.377 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "full_sync": { 2023-05-31T21:35:45.378 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "total": 64, 2023-05-31T21:35:45.378 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "complete": 64 2023-05-31T21:35:45.378 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2023-05-31T21:35:45.378 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "current_time": "2023-05-31T21:30:33Z" 2023-05-31T21:35:45.378 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2023-05-31T21:35:45.379 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current data sync status={ 2023-05-31T21:35:45.379 INFO:tasks.rgw_multisite_tests: "sync_status": { 2023-05-31T21:35:45.379 INFO:tasks.rgw_multisite_tests: "info": { 2023-05-31T21:35:45.379 INFO:tasks.rgw_multisite_tests: "status": "sync", 2023-05-31T21:35:45.380 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2023-05-31T21:35:45.380 INFO:tasks.rgw_multisite_tests: "instance_id": 452136274422638824 2023-05-31T21:35:45.380 INFO:tasks.rgw_multisite_tests: }, 2023-05-31T21:35:45.380 INFO:tasks.rgw_multisite_tests: "markers": [ 2023-05-31T21:35:45.381 INFO:tasks.rgw_multisite_tests: { 2023-05-31T21:35:45.381 INFO:tasks.rgw_multisite_tests: "key": 0, 2023-05-31T21:35:45.381 INFO:tasks.rgw_multisite_tests: "val": { 2023-05-31T21:35:45.381 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2023-05-31T21:35:45.382 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000045375", 2023-05-31T21:35:45.382 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-05-31T21:35:45.382 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-05-31T21:35:45.382 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-05-31T21:35:45.382 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-05-31T21:28:16.422581Z" 2023-05-31T21:35:45.383 INFO:tasks.rgw_multisite_tests: } 2023-05-31T21:35:45.383 INFO:tasks.rgw_multisite_tests: }, 2023-05-31T21:35:45.383 INFO:tasks.rgw_multisite_tests: { 2023-05-31T21:35:45.383 INFO:tasks.rgw_multisite_tests: "key": 1, 2023-05-31T21:35:45.384 INFO:tasks.rgw_multisite_tests: "val": { 2023-05-31T21:35:45.384 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2023-05-31T21:35:45.384 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000031780", 2023-05-31T21:35:45.384 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-05-31T21:35:45.385 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-05-31T21:35:45.385 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-05-31T21:35:45.385 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-05-31T21:28:16.501293Z" 2023-05-31T21:35:45.385 INFO:tasks.rgw_multisite_tests: } 2023-05-31T21:35:45.385 INFO:tasks.rgw_multisite_tests: }, 2023-05-31T21:35:45.386 INFO:tasks.rgw_multisite_tests: { 2023-05-31T21:35:45.386 INFO:tasks.rgw_multisite_tests: "key": 2, 2023-05-31T21:35:45.386 INFO:tasks.rgw_multisite_tests: "val": { 2023-05-31T21:35:45.386 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2023-05-31T21:35:45.387 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000051671", 2023-05-31T21:35:45.387 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-05-31T21:35:45.387 INFO:tasks.rgw_multisite_tests: "total_entries": 55, 2023-05-31T21:35:45.387 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-05-31T21:35:45.388 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-05-31T21:28:16.476487Z" 2023-05-31T21:35:45.388 INFO:tasks.rgw_multisite_tests: } 2023-05-31T21:35:45.388 INFO:tasks.rgw_multisite_tests: }, 2023-05-31T21:35:45.388 INFO:tasks.rgw_multisite_tests: { 2023-05-31T21:35:45.389 INFO:tasks.rgw_multisite_tests: "key": 3, 2023-05-31T21:35:45.389 INFO:tasks.rgw_multisite_tests: "val": { 2023-05-31T21:35:45.389 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2023-05-31T21:35:45.389 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000044178", 2023-05-31T21:35:45.389 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-05-31T21:35:45.390 INFO:tasks.rgw_multisite_tests: "total_entries": 9, 2023-05-31T21:35:45.390 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-05-31T21:35:45.390 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-05-31T21:27:37.215905Z" 2023-05-31T21:35:45.390 INFO:tasks.rgw_multisite_tests: } 2023-05-31T21:35:45.391 INFO:tasks.rgw_multisite_tests: } 2023-05-31T21:35:45.391 INFO:tasks.rgw_multisite_tests: ] 2023-05-31T21:35:45.391 INFO:tasks.rgw_multisite_tests: }, 2023-05-31T21:35:45.391 INFO:tasks.rgw_multisite_tests: "full_sync": { 2023-05-31T21:35:45.392 INFO:tasks.rgw_multisite_tests: "total": 64, 2023-05-31T21:35:45.392 INFO:tasks.rgw_multisite_tests: "complete": 64 2023-05-31T21:35:45.392 INFO:tasks.rgw_multisite_tests: }, 2023-05-31T21:35:45.392 INFO:tasks.rgw_multisite_tests: "current_time": "2023-05-31T21:30:33Z" 2023-05-31T21:35:45.392 INFO:tasks.rgw_multisite_tests:} 2023-05-31T21:35:45.393 INFO:tasks.rgw_multisite_tests: 2023-05-31T21:35:45.393 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000045375', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-05-31T21:28:16.422581Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000031780', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-05-31T21:28:16.501293Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000051671', 'next_step_marker': '', 'total_entries': 55, 'pos': 0, 'timestamp': '2023-05-31T21:28:16.476487Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000044178', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2023-05-31T21:27:37.215905Z'}}] 2023-05-31T21:35:45.393 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '00000000000000000000:00000000000000045515', 1: '00000000000000000000:00000000000000031920', 2: '00000000000000000000:00000000000000051951', 3: '00000000000000000000:00000000000000044178'} 2023-05-31T21:35:45.393 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '00000000000000000000:00000000000000045375', 1: '00000000000000000000:00000000000000031780', 2: '00000000000000000000:00000000000000051671', 3: '00000000000000000000:00000000000000044178'} 2023-05-31T21:35:45.394 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: data of zone a2 behind zone a1: shard=0 master=00000000000000000000:00000000000000045515 target=00000000000000000000:00000000000000045375, shard=1 master=00000000000000000000:00000000000000031920 target=00000000000000000000:00000000000000031780, shard=2 master=00000000000000000000:00000000000000051951 target=00000000000000000000:00000000000000051671 2023-05-31T21:35:45.394 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-31T21:35:45.394 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-31T21:35:45.394 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-05-31T21:35:45.395 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: ignoring --setuser ceph since I am not root 2023-05-31T21:35:45.395 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-05-31T21:35:45.395 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2023-05-31T21:35:45.395 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "sync_status": { 2023-05-31T21:35:45.396 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "info": { 2023-05-31T21:35:45.396 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "status": "sync", 2023-05-31T21:35:45.396 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "num_shards": 4, 2023-05-31T21:35:45.396 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "instance_id": 452136274422638824 2023-05-31T21:35:45.397 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2023-05-31T21:35:45.397 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "markers": [ 2023-05-31T21:35:45.397 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2023-05-31T21:35:45.397 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "key": 0, 2023-05-31T21:35:45.397 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "val": { 2023-05-31T21:35:45.398 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "status": "incremental-sync", 2023-05-31T21:35:45.398 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "marker": "00000000000000000000:00000000000000045375", 2023-05-31T21:35:45.398 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "next_step_marker": "", 2023-05-31T21:35:45.398 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "total_entries": 0, 2023-05-31T21:35:45.399 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "pos": 0, 2023-05-31T21:35:45.399 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "timestamp": "2023-05-31T21:28:16.422581Z" 2023-05-31T21:35:45.399 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2023-05-31T21:35:45.399 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2023-05-31T21:35:45.400 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2023-05-31T21:35:45.400 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "key": 1, 2023-05-31T21:35:45.400 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "val": { 2023-05-31T21:35:45.400 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "status": "incremental-sync", 2023-05-31T21:35:45.400 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "marker": "00000000000000000000:00000000000000031780", 2023-05-31T21:35:45.401 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "next_step_marker": "", 2023-05-31T21:35:45.401 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "total_entries": 0, 2023-05-31T21:35:45.401 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "pos": 0, 2023-05-31T21:35:45.401 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "timestamp": "2023-05-31T21:28:16.501293Z" 2023-05-31T21:35:45.402 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2023-05-31T21:35:45.402 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2023-05-31T21:35:45.402 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2023-05-31T21:35:45.402 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "key": 2, 2023-05-31T21:35:45.403 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "val": { 2023-05-31T21:35:45.403 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "status": "incremental-sync", 2023-05-31T21:35:45.403 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "marker": "00000000000000000000:00000000000000051671", 2023-05-31T21:35:45.403 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "next_step_marker": "", 2023-05-31T21:35:45.404 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "total_entries": 55, 2023-05-31T21:35:45.404 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "pos": 0, 2023-05-31T21:35:45.404 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "timestamp": "2023-05-31T21:28:16.476487Z" 2023-05-31T21:35:45.404 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2023-05-31T21:35:45.404 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2023-05-31T21:35:45.405 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2023-05-31T21:35:45.405 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "key": 3, 2023-05-31T21:35:45.405 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "val": { 2023-05-31T21:35:45.405 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "status": "incremental-sync", 2023-05-31T21:35:45.406 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "marker": "00000000000000000000:00000000000000044178", 2023-05-31T21:35:45.406 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "next_step_marker": "", 2023-05-31T21:35:45.406 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "total_entries": 9, 2023-05-31T21:35:45.406 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "pos": 0, 2023-05-31T21:35:45.407 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "timestamp": "2023-05-31T21:27:37.215905Z" 2023-05-31T21:35:45.407 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2023-05-31T21:35:45.407 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2023-05-31T21:35:45.407 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: ] 2023-05-31T21:35:45.407 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2023-05-31T21:35:45.408 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "full_sync": { 2023-05-31T21:35:45.408 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "total": 64, 2023-05-31T21:35:45.408 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "complete": 64 2023-05-31T21:35:45.408 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2023-05-31T21:35:45.409 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "current_time": "2023-05-31T21:30:39Z" 2023-05-31T21:35:45.409 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2023-05-31T21:35:45.409 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current data sync status={ 2023-05-31T21:35:45.409 INFO:tasks.rgw_multisite_tests: "sync_status": { 2023-05-31T21:35:45.410 INFO:tasks.rgw_multisite_tests: "info": { 2023-05-31T21:35:45.410 INFO:tasks.rgw_multisite_tests: "status": "sync", 2023-05-31T21:35:45.410 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2023-05-31T21:35:45.410 INFO:tasks.rgw_multisite_tests: "instance_id": 452136274422638824 2023-05-31T21:35:45.411 INFO:tasks.rgw_multisite_tests: }, 2023-05-31T21:35:45.411 INFO:tasks.rgw_multisite_tests: "markers": [ 2023-05-31T21:35:45.411 INFO:tasks.rgw_multisite_tests: { 2023-05-31T21:35:45.411 INFO:tasks.rgw_multisite_tests: "key": 0, 2023-05-31T21:35:45.411 INFO:tasks.rgw_multisite_tests: "val": { 2023-05-31T21:35:45.412 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2023-05-31T21:35:45.412 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000045375", 2023-05-31T21:35:45.412 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-05-31T21:35:45.412 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-05-31T21:35:45.413 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-05-31T21:35:45.413 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-05-31T21:28:16.422581Z" 2023-05-31T21:35:45.413 INFO:tasks.rgw_multisite_tests: } 2023-05-31T21:35:45.413 INFO:tasks.rgw_multisite_tests: }, 2023-05-31T21:35:45.414 INFO:tasks.rgw_multisite_tests: { 2023-05-31T21:35:45.414 INFO:tasks.rgw_multisite_tests: "key": 1, 2023-05-31T21:35:45.414 INFO:tasks.rgw_multisite_tests: "val": { 2023-05-31T21:35:45.414 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2023-05-31T21:35:45.415 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000031780", 2023-05-31T21:35:45.415 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-05-31T21:35:45.415 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-05-31T21:35:45.415 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-05-31T21:35:45.415 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-05-31T21:28:16.501293Z" 2023-05-31T21:35:45.416 INFO:tasks.rgw_multisite_tests: } 2023-05-31T21:35:45.416 INFO:tasks.rgw_multisite_tests: }, 2023-05-31T21:35:45.416 INFO:tasks.rgw_multisite_tests: { 2023-05-31T21:35:45.416 INFO:tasks.rgw_multisite_tests: "key": 2, 2023-05-31T21:35:45.417 INFO:tasks.rgw_multisite_tests: "val": { 2023-05-31T21:35:45.417 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2023-05-31T21:35:45.417 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000051671", 2023-05-31T21:35:45.417 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-05-31T21:35:45.418 INFO:tasks.rgw_multisite_tests: "total_entries": 55, 2023-05-31T21:35:45.418 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-05-31T21:35:45.418 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-05-31T21:28:16.476487Z" 2023-05-31T21:35:45.418 INFO:tasks.rgw_multisite_tests: } 2023-05-31T21:35:45.418 INFO:tasks.rgw_multisite_tests: }, 2023-05-31T21:35:45.419 INFO:tasks.rgw_multisite_tests: { 2023-05-31T21:35:45.419 INFO:tasks.rgw_multisite_tests: "key": 3, 2023-05-31T21:35:45.419 INFO:tasks.rgw_multisite_tests: "val": { 2023-05-31T21:35:45.419 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2023-05-31T21:35:45.420 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000044178", 2023-05-31T21:35:45.420 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-05-31T21:35:45.420 INFO:tasks.rgw_multisite_tests: "total_entries": 9, 2023-05-31T21:35:45.420 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-05-31T21:35:45.421 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-05-31T21:27:37.215905Z" 2023-05-31T21:35:45.421 INFO:tasks.rgw_multisite_tests: } 2023-05-31T21:35:45.421 INFO:tasks.rgw_multisite_tests: } 2023-05-31T21:35:45.421 INFO:tasks.rgw_multisite_tests: ] 2023-05-31T21:35:45.421 INFO:tasks.rgw_multisite_tests: }, 2023-05-31T21:35:45.422 INFO:tasks.rgw_multisite_tests: "full_sync": { 2023-05-31T21:35:45.422 INFO:tasks.rgw_multisite_tests: "total": 64, 2023-05-31T21:35:45.422 INFO:tasks.rgw_multisite_tests: "complete": 64 2023-05-31T21:35:45.422 INFO:tasks.rgw_multisite_tests: }, 2023-05-31T21:35:45.423 INFO:tasks.rgw_multisite_tests: "current_time": "2023-05-31T21:30:39Z" 2023-05-31T21:35:45.423 INFO:tasks.rgw_multisite_tests:} 2023-05-31T21:35:45.423 INFO:tasks.rgw_multisite_tests: 2023-05-31T21:35:45.423 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000045375', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-05-31T21:28:16.422581Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000031780', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-05-31T21:28:16.501293Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000051671', 'next_step_marker': '', 'total_entries': 55, 'pos': 0, 'timestamp': '2023-05-31T21:28:16.476487Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000044178', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2023-05-31T21:27:37.215905Z'}}] 2023-05-31T21:35:45.424 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '00000000000000000000:00000000000000045515', 1: '00000000000000000000:00000000000000031920', 2: '00000000000000000000:00000000000000051951', 3: '00000000000000000000:00000000000000044178'} 2023-05-31T21:35:45.424 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '00000000000000000000:00000000000000045375', 1: '00000000000000000000:00000000000000031780', 2: '00000000000000000000:00000000000000051671', 3: '00000000000000000000:00000000000000044178'} 2023-05-31T21:35:45.424 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: data of zone a2 behind zone a1: shard=0 master=00000000000000000000:00000000000000045515 target=00000000000000000000:00000000000000045375, shard=1 master=00000000000000000000:00000000000000031920 target=00000000000000000000:00000000000000031780, shard=2 master=00000000000000000000:00000000000000051951 target=00000000000000000000:00000000000000051671 2023-05-31T21:35:45.424 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-31T21:35:45.425 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-31T21:35:45.425 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-05-31T21:35:45.425 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: ignoring --setuser ceph since I am not root 2023-05-31T21:35:45.425 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-05-31T21:35:45.425 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2023-05-31T21:35:45.426 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "sync_status": { 2023-05-31T21:35:45.426 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "info": { 2023-05-31T21:35:45.426 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "status": "sync", 2023-05-31T21:35:45.426 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "num_shards": 4, 2023-05-31T21:35:45.427 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "instance_id": 452136274422638824 2023-05-31T21:35:45.427 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2023-05-31T21:35:45.427 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "markers": [ 2023-05-31T21:35:45.427 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2023-05-31T21:35:45.428 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "key": 0, 2023-05-31T21:35:45.428 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "val": { 2023-05-31T21:35:45.428 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "status": "incremental-sync", 2023-05-31T21:35:45.428 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "marker": "00000000000000000000:00000000000000045375", 2023-05-31T21:35:45.428 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "next_step_marker": "", 2023-05-31T21:35:45.429 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "total_entries": 0, 2023-05-31T21:35:45.429 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "pos": 0, 2023-05-31T21:35:45.429 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "timestamp": "2023-05-31T21:28:16.422581Z" 2023-05-31T21:35:45.429 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2023-05-31T21:35:45.430 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2023-05-31T21:35:45.430 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2023-05-31T21:35:45.430 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "key": 1, 2023-05-31T21:35:45.430 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "val": { 2023-05-31T21:35:45.431 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "status": "incremental-sync", 2023-05-31T21:35:45.431 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "marker": "00000000000000000000:00000000000000031780", 2023-05-31T21:35:45.431 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "next_step_marker": "", 2023-05-31T21:35:45.431 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "total_entries": 0, 2023-05-31T21:35:45.432 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "pos": 0, 2023-05-31T21:35:45.432 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "timestamp": "2023-05-31T21:28:16.501293Z" 2023-05-31T21:35:45.432 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2023-05-31T21:35:45.432 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2023-05-31T21:35:45.432 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2023-05-31T21:35:45.433 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "key": 2, 2023-05-31T21:35:45.433 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "val": { 2023-05-31T21:35:45.433 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "status": "incremental-sync", 2023-05-31T21:35:45.433 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "marker": "00000000000000000000:00000000000000051671", 2023-05-31T21:35:45.434 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "next_step_marker": "", 2023-05-31T21:35:45.434 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "total_entries": 55, 2023-05-31T21:35:45.434 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "pos": 0, 2023-05-31T21:35:45.434 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "timestamp": "2023-05-31T21:28:16.476487Z" 2023-05-31T21:35:45.435 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2023-05-31T21:35:45.435 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2023-05-31T21:35:45.435 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2023-05-31T21:35:45.435 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "key": 3, 2023-05-31T21:35:45.435 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "val": { 2023-05-31T21:35:45.436 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "status": "incremental-sync", 2023-05-31T21:35:45.436 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "marker": "00000000000000000000:00000000000000044178", 2023-05-31T21:35:45.436 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "next_step_marker": "", 2023-05-31T21:35:45.436 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "total_entries": 9, 2023-05-31T21:35:45.437 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "pos": 0, 2023-05-31T21:35:45.437 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "timestamp": "2023-05-31T21:27:37.215905Z" 2023-05-31T21:35:45.437 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2023-05-31T21:35:45.437 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2023-05-31T21:35:45.438 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: ] 2023-05-31T21:35:45.438 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2023-05-31T21:35:45.438 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "full_sync": { 2023-05-31T21:35:45.438 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "total": 64, 2023-05-31T21:35:45.439 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "complete": 64 2023-05-31T21:35:45.439 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2023-05-31T21:35:45.439 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "current_time": "2023-05-31T21:30:44Z" 2023-05-31T21:35:45.439 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2023-05-31T21:35:45.439 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current data sync status={ 2023-05-31T21:35:45.440 INFO:tasks.rgw_multisite_tests: "sync_status": { 2023-05-31T21:35:45.440 INFO:tasks.rgw_multisite_tests: "info": { 2023-05-31T21:35:45.440 INFO:tasks.rgw_multisite_tests: "status": "sync", 2023-05-31T21:35:45.440 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2023-05-31T21:35:45.441 INFO:tasks.rgw_multisite_tests: "instance_id": 452136274422638824 2023-05-31T21:35:45.441 INFO:tasks.rgw_multisite_tests: }, 2023-05-31T21:35:45.441 INFO:tasks.rgw_multisite_tests: "markers": [ 2023-05-31T21:35:45.441 INFO:tasks.rgw_multisite_tests: { 2023-05-31T21:35:45.442 INFO:tasks.rgw_multisite_tests: "key": 0, 2023-05-31T21:35:45.442 INFO:tasks.rgw_multisite_tests: "val": { 2023-05-31T21:35:45.442 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2023-05-31T21:35:45.442 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000045375", 2023-05-31T21:35:45.442 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-05-31T21:35:45.443 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-05-31T21:35:45.443 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-05-31T21:35:45.443 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-05-31T21:28:16.422581Z" 2023-05-31T21:35:45.443 INFO:tasks.rgw_multisite_tests: } 2023-05-31T21:35:45.444 INFO:tasks.rgw_multisite_tests: }, 2023-05-31T21:35:45.444 INFO:tasks.rgw_multisite_tests: { 2023-05-31T21:35:45.444 INFO:tasks.rgw_multisite_tests: "key": 1, 2023-05-31T21:35:45.444 INFO:tasks.rgw_multisite_tests: "val": { 2023-05-31T21:35:45.445 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2023-05-31T21:35:45.445 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000031780", 2023-05-31T21:35:45.445 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-05-31T21:35:45.445 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-05-31T21:35:45.445 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-05-31T21:35:45.446 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-05-31T21:28:16.501293Z" 2023-05-31T21:35:45.446 INFO:tasks.rgw_multisite_tests: } 2023-05-31T21:35:45.446 INFO:tasks.rgw_multisite_tests: }, 2023-05-31T21:35:45.446 INFO:tasks.rgw_multisite_tests: { 2023-05-31T21:35:45.447 INFO:tasks.rgw_multisite_tests: "key": 2, 2023-05-31T21:35:45.447 INFO:tasks.rgw_multisite_tests: "val": { 2023-05-31T21:35:45.447 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2023-05-31T21:35:45.447 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000051671", 2023-05-31T21:35:45.448 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-05-31T21:35:45.448 INFO:tasks.rgw_multisite_tests: "total_entries": 55, 2023-05-31T21:35:45.448 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-05-31T21:35:45.448 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-05-31T21:28:16.476487Z" 2023-05-31T21:35:45.449 INFO:tasks.rgw_multisite_tests: } 2023-05-31T21:35:45.449 INFO:tasks.rgw_multisite_tests: }, 2023-05-31T21:35:45.449 INFO:tasks.rgw_multisite_tests: { 2023-05-31T21:35:45.449 INFO:tasks.rgw_multisite_tests: "key": 3, 2023-05-31T21:35:45.449 INFO:tasks.rgw_multisite_tests: "val": { 2023-05-31T21:35:45.450 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2023-05-31T21:35:45.450 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000044178", 2023-05-31T21:35:45.450 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-05-31T21:35:45.450 INFO:tasks.rgw_multisite_tests: "total_entries": 9, 2023-05-31T21:35:45.451 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-05-31T21:35:45.451 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-05-31T21:27:37.215905Z" 2023-05-31T21:35:45.451 INFO:tasks.rgw_multisite_tests: } 2023-05-31T21:35:45.451 INFO:tasks.rgw_multisite_tests: } 2023-05-31T21:35:45.452 INFO:tasks.rgw_multisite_tests: ] 2023-05-31T21:35:45.452 INFO:tasks.rgw_multisite_tests: }, 2023-05-31T21:35:45.452 INFO:tasks.rgw_multisite_tests: "full_sync": { 2023-05-31T21:35:45.452 INFO:tasks.rgw_multisite_tests: "total": 64, 2023-05-31T21:35:45.452 INFO:tasks.rgw_multisite_tests: "complete": 64 2023-05-31T21:35:45.453 INFO:tasks.rgw_multisite_tests: }, 2023-05-31T21:35:45.453 INFO:tasks.rgw_multisite_tests: "current_time": "2023-05-31T21:30:44Z" 2023-05-31T21:35:45.453 INFO:tasks.rgw_multisite_tests:} 2023-05-31T21:35:45.453 INFO:tasks.rgw_multisite_tests: 2023-05-31T21:35:45.454 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000045375', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-05-31T21:28:16.422581Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000031780', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-05-31T21:28:16.501293Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000051671', 'next_step_marker': '', 'total_entries': 55, 'pos': 0, 'timestamp': '2023-05-31T21:28:16.476487Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000044178', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2023-05-31T21:27:37.215905Z'}}] 2023-05-31T21:35:45.454 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '00000000000000000000:00000000000000045515', 1: '00000000000000000000:00000000000000031920', 2: '00000000000000000000:00000000000000051951', 3: '00000000000000000000:00000000000000044178'} 2023-05-31T21:35:45.454 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '00000000000000000000:00000000000000045375', 1: '00000000000000000000:00000000000000031780', 2: '00000000000000000000:00000000000000051671', 3: '00000000000000000000:00000000000000044178'} 2023-05-31T21:35:45.454 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: data of zone a2 behind zone a1: shard=0 master=00000000000000000000:00000000000000045515 target=00000000000000000000:00000000000000045375, shard=1 master=00000000000000000000:00000000000000031920 target=00000000000000000000:00000000000000031780, shard=2 master=00000000000000000000:00000000000000051951 target=00000000000000000000:00000000000000051671 2023-05-31T21:35:45.455 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-31T21:35:45.455 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-31T21:35:45.455 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-05-31T21:35:45.455 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: ignoring --setuser ceph since I am not root 2023-05-31T21:35:45.456 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-05-31T21:35:45.456 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2023-05-31T21:35:45.456 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "sync_status": { 2023-05-31T21:35:45.456 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "info": { 2023-05-31T21:35:45.457 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "status": "sync", 2023-05-31T21:35:45.457 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "num_shards": 4, 2023-05-31T21:35:45.457 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "instance_id": 452136274422638824 2023-05-31T21:35:45.457 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2023-05-31T21:35:45.457 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "markers": [ 2023-05-31T21:35:45.458 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2023-05-31T21:35:45.458 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "key": 0, 2023-05-31T21:35:45.458 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "val": { 2023-05-31T21:35:45.458 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "status": "incremental-sync", 2023-05-31T21:35:45.459 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "marker": "00000000000000000000:00000000000000045375", 2023-05-31T21:35:45.459 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "next_step_marker": "", 2023-05-31T21:35:45.459 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "total_entries": 0, 2023-05-31T21:35:45.459 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "pos": 0, 2023-05-31T21:35:45.460 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "timestamp": "2023-05-31T21:28:16.422581Z" 2023-05-31T21:35:45.460 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2023-05-31T21:35:45.460 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2023-05-31T21:35:45.460 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2023-05-31T21:35:45.461 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "key": 1, 2023-05-31T21:35:45.461 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "val": { 2023-05-31T21:35:45.461 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "status": "incremental-sync", 2023-05-31T21:35:45.462 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "marker": "00000000000000000000:00000000000000031780", 2023-05-31T21:35:45.462 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "next_step_marker": "", 2023-05-31T21:35:45.462 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "total_entries": 0, 2023-05-31T21:35:45.462 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "pos": 0, 2023-05-31T21:35:45.463 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "timestamp": "2023-05-31T21:28:16.501293Z" 2023-05-31T21:35:45.463 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2023-05-31T21:35:45.463 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2023-05-31T21:35:45.463 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2023-05-31T21:35:45.464 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "key": 2, 2023-05-31T21:35:45.464 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "val": { 2023-05-31T21:35:45.464 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "status": "incremental-sync", 2023-05-31T21:35:45.464 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "marker": "00000000000000000000:00000000000000051671", 2023-05-31T21:35:45.465 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "next_step_marker": "", 2023-05-31T21:35:45.465 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "total_entries": 55, 2023-05-31T21:35:45.465 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "pos": 0, 2023-05-31T21:35:45.466 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "timestamp": "2023-05-31T21:28:16.476487Z" 2023-05-31T21:35:45.466 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2023-05-31T21:35:45.466 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2023-05-31T21:35:45.466 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2023-05-31T21:35:45.467 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "key": 3, 2023-05-31T21:35:45.467 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "val": { 2023-05-31T21:35:45.467 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "status": "incremental-sync", 2023-05-31T21:35:45.468 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "marker": "00000000000000000000:00000000000000044178", 2023-05-31T21:35:45.468 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "next_step_marker": "", 2023-05-31T21:35:45.468 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "total_entries": 9, 2023-05-31T21:35:45.468 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "pos": 0, 2023-05-31T21:35:45.469 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "timestamp": "2023-05-31T21:27:37.215905Z" 2023-05-31T21:35:45.469 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2023-05-31T21:35:45.469 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2023-05-31T21:35:45.469 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: ] 2023-05-31T21:35:45.470 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2023-05-31T21:35:45.470 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "full_sync": { 2023-05-31T21:35:45.470 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "total": 64, 2023-05-31T21:35:45.471 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "complete": 64 2023-05-31T21:35:45.471 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2023-05-31T21:35:45.471 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "current_time": "2023-05-31T21:30:49Z" 2023-05-31T21:35:45.471 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2023-05-31T21:35:45.472 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current data sync status={ 2023-05-31T21:35:45.472 INFO:tasks.rgw_multisite_tests: "sync_status": { 2023-05-31T21:35:45.472 INFO:tasks.rgw_multisite_tests: "info": { 2023-05-31T21:35:45.472 INFO:tasks.rgw_multisite_tests: "status": "sync", 2023-05-31T21:35:45.473 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2023-05-31T21:35:45.473 INFO:tasks.rgw_multisite_tests: "instance_id": 452136274422638824 2023-05-31T21:35:45.473 INFO:tasks.rgw_multisite_tests: }, 2023-05-31T21:35:45.473 INFO:tasks.rgw_multisite_tests: "markers": [ 2023-05-31T21:35:45.474 INFO:tasks.rgw_multisite_tests: { 2023-05-31T21:35:45.474 INFO:tasks.rgw_multisite_tests: "key": 0, 2023-05-31T21:35:45.474 INFO:tasks.rgw_multisite_tests: "val": { 2023-05-31T21:35:45.475 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2023-05-31T21:35:45.475 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000045375", 2023-05-31T21:35:45.475 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-05-31T21:35:45.475 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-05-31T21:35:45.476 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-05-31T21:35:45.476 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-05-31T21:28:16.422581Z" 2023-05-31T21:35:45.476 INFO:tasks.rgw_multisite_tests: } 2023-05-31T21:35:45.476 INFO:tasks.rgw_multisite_tests: }, 2023-05-31T21:35:45.477 INFO:tasks.rgw_multisite_tests: { 2023-05-31T21:35:45.477 INFO:tasks.rgw_multisite_tests: "key": 1, 2023-05-31T21:35:45.477 INFO:tasks.rgw_multisite_tests: "val": { 2023-05-31T21:35:45.477 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2023-05-31T21:35:45.478 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000031780", 2023-05-31T21:35:45.478 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-05-31T21:35:45.478 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-05-31T21:35:45.478 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-05-31T21:35:45.479 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-05-31T21:28:16.501293Z" 2023-05-31T21:35:45.479 INFO:tasks.rgw_multisite_tests: } 2023-05-31T21:35:45.479 INFO:tasks.rgw_multisite_tests: }, 2023-05-31T21:35:45.479 INFO:tasks.rgw_multisite_tests: { 2023-05-31T21:35:45.479 INFO:tasks.rgw_multisite_tests: "key": 2, 2023-05-31T21:35:45.480 INFO:tasks.rgw_multisite_tests: "val": { 2023-05-31T21:35:45.480 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2023-05-31T21:35:45.480 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000051671", 2023-05-31T21:35:45.480 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-05-31T21:35:45.481 INFO:tasks.rgw_multisite_tests: "total_entries": 55, 2023-05-31T21:35:45.481 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-05-31T21:35:45.481 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-05-31T21:28:16.476487Z" 2023-05-31T21:35:45.481 INFO:tasks.rgw_multisite_tests: } 2023-05-31T21:35:45.482 INFO:tasks.rgw_multisite_tests: }, 2023-05-31T21:35:45.482 INFO:tasks.rgw_multisite_tests: { 2023-05-31T21:35:45.482 INFO:tasks.rgw_multisite_tests: "key": 3, 2023-05-31T21:35:45.482 INFO:tasks.rgw_multisite_tests: "val": { 2023-05-31T21:35:45.483 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2023-05-31T21:35:45.483 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000044178", 2023-05-31T21:35:45.483 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-05-31T21:35:45.483 INFO:tasks.rgw_multisite_tests: "total_entries": 9, 2023-05-31T21:35:45.483 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-05-31T21:35:45.484 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-05-31T21:27:37.215905Z" 2023-05-31T21:35:45.484 INFO:tasks.rgw_multisite_tests: } 2023-05-31T21:35:45.484 INFO:tasks.rgw_multisite_tests: } 2023-05-31T21:35:45.484 INFO:tasks.rgw_multisite_tests: ] 2023-05-31T21:35:45.485 INFO:tasks.rgw_multisite_tests: }, 2023-05-31T21:35:45.485 INFO:tasks.rgw_multisite_tests: "full_sync": { 2023-05-31T21:35:45.485 INFO:tasks.rgw_multisite_tests: "total": 64, 2023-05-31T21:35:45.485 INFO:tasks.rgw_multisite_tests: "complete": 64 2023-05-31T21:35:45.486 INFO:tasks.rgw_multisite_tests: }, 2023-05-31T21:35:45.486 INFO:tasks.rgw_multisite_tests: "current_time": "2023-05-31T21:30:49Z" 2023-05-31T21:35:45.486 INFO:tasks.rgw_multisite_tests:} 2023-05-31T21:35:45.486 INFO:tasks.rgw_multisite_tests: 2023-05-31T21:35:45.486 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000045375', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-05-31T21:28:16.422581Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000031780', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-05-31T21:28:16.501293Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000051671', 'next_step_marker': '', 'total_entries': 55, 'pos': 0, 'timestamp': '2023-05-31T21:28:16.476487Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000044178', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2023-05-31T21:27:37.215905Z'}}] 2023-05-31T21:35:45.487 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '00000000000000000000:00000000000000045515', 1: '00000000000000000000:00000000000000031920', 2: '00000000000000000000:00000000000000051951', 3: '00000000000000000000:00000000000000044178'} 2023-05-31T21:35:45.487 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '00000000000000000000:00000000000000045375', 1: '00000000000000000000:00000000000000031780', 2: '00000000000000000000:00000000000000051671', 3: '00000000000000000000:00000000000000044178'} 2023-05-31T21:35:45.487 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: data of zone a2 behind zone a1: shard=0 master=00000000000000000000:00000000000000045515 target=00000000000000000000:00000000000000045375, shard=1 master=00000000000000000000:00000000000000031920 target=00000000000000000000:00000000000000031780, shard=2 master=00000000000000000000:00000000000000051951 target=00000000000000000000:00000000000000051671 2023-05-31T21:35:45.487 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-31T21:35:45.488 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-31T21:35:45.488 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-05-31T21:35:45.488 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: ignoring --setuser ceph since I am not root 2023-05-31T21:35:45.488 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-05-31T21:35:45.489 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2023-05-31T21:35:45.489 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "sync_status": { 2023-05-31T21:35:45.489 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "info": { 2023-05-31T21:35:45.489 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "status": "sync", 2023-05-31T21:35:45.490 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "num_shards": 4, 2023-05-31T21:35:45.490 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "instance_id": 452136274422638824 2023-05-31T21:35:45.490 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2023-05-31T21:35:45.490 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "markers": [ 2023-05-31T21:35:45.490 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2023-05-31T21:35:45.491 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "key": 0, 2023-05-31T21:35:45.491 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "val": { 2023-05-31T21:35:45.491 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "status": "incremental-sync", 2023-05-31T21:35:45.491 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "marker": "00000000000000000000:00000000000000045375", 2023-05-31T21:35:45.492 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "next_step_marker": "", 2023-05-31T21:35:45.492 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "total_entries": 0, 2023-05-31T21:35:45.492 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "pos": 0, 2023-05-31T21:35:45.492 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "timestamp": "2023-05-31T21:28:16.422581Z" 2023-05-31T21:35:45.493 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2023-05-31T21:35:45.493 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2023-05-31T21:35:45.493 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2023-05-31T21:35:45.493 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "key": 1, 2023-05-31T21:35:45.493 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "val": { 2023-05-31T21:35:45.494 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "status": "incremental-sync", 2023-05-31T21:35:45.494 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "marker": "00000000000000000000:00000000000000031780", 2023-05-31T21:35:45.494 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "next_step_marker": "", 2023-05-31T21:35:45.494 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "total_entries": 0, 2023-05-31T21:35:45.495 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "pos": 0, 2023-05-31T21:35:45.495 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "timestamp": "2023-05-31T21:28:16.501293Z" 2023-05-31T21:35:45.495 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2023-05-31T21:35:45.495 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2023-05-31T21:35:45.496 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2023-05-31T21:35:45.496 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "key": 2, 2023-05-31T21:35:45.496 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "val": { 2023-05-31T21:35:45.496 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "status": "incremental-sync", 2023-05-31T21:35:45.497 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "marker": "00000000000000000000:00000000000000051671", 2023-05-31T21:35:45.497 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "next_step_marker": "", 2023-05-31T21:35:45.497 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "total_entries": 55, 2023-05-31T21:35:45.497 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "pos": 0, 2023-05-31T21:35:45.497 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "timestamp": "2023-05-31T21:28:16.476487Z" 2023-05-31T21:35:45.498 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2023-05-31T21:35:45.498 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2023-05-31T21:35:45.498 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2023-05-31T21:35:45.498 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "key": 3, 2023-05-31T21:35:45.499 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "val": { 2023-05-31T21:35:45.499 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "status": "incremental-sync", 2023-05-31T21:35:45.499 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "marker": "00000000000000000000:00000000000000044178", 2023-05-31T21:35:45.499 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "next_step_marker": "", 2023-05-31T21:35:45.500 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "total_entries": 9, 2023-05-31T21:35:45.500 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "pos": 0, 2023-05-31T21:35:45.500 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "timestamp": "2023-05-31T21:27:37.215905Z" 2023-05-31T21:35:45.500 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2023-05-31T21:35:45.500 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2023-05-31T21:35:45.501 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: ] 2023-05-31T21:35:45.501 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2023-05-31T21:35:45.501 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "full_sync": { 2023-05-31T21:35:45.501 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "total": 64, 2023-05-31T21:35:45.502 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "complete": 64 2023-05-31T21:35:45.502 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2023-05-31T21:35:45.502 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "current_time": "2023-05-31T21:30:54Z" 2023-05-31T21:35:45.502 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2023-05-31T21:35:45.503 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current data sync status={ 2023-05-31T21:35:45.503 INFO:tasks.rgw_multisite_tests: "sync_status": { 2023-05-31T21:35:45.503 INFO:tasks.rgw_multisite_tests: "info": { 2023-05-31T21:35:45.503 INFO:tasks.rgw_multisite_tests: "status": "sync", 2023-05-31T21:35:45.503 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2023-05-31T21:35:45.504 INFO:tasks.rgw_multisite_tests: "instance_id": 452136274422638824 2023-05-31T21:35:45.504 INFO:tasks.rgw_multisite_tests: }, 2023-05-31T21:35:45.504 INFO:tasks.rgw_multisite_tests: "markers": [ 2023-05-31T21:35:45.504 INFO:tasks.rgw_multisite_tests: { 2023-05-31T21:35:45.505 INFO:tasks.rgw_multisite_tests: "key": 0, 2023-05-31T21:35:45.505 INFO:tasks.rgw_multisite_tests: "val": { 2023-05-31T21:35:45.505 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2023-05-31T21:35:45.505 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000045375", 2023-05-31T21:35:45.506 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-05-31T21:35:45.506 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-05-31T21:35:45.506 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-05-31T21:35:45.506 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-05-31T21:28:16.422581Z" 2023-05-31T21:35:45.506 INFO:tasks.rgw_multisite_tests: } 2023-05-31T21:35:45.507 INFO:tasks.rgw_multisite_tests: }, 2023-05-31T21:35:45.507 INFO:tasks.rgw_multisite_tests: { 2023-05-31T21:35:45.507 INFO:tasks.rgw_multisite_tests: "key": 1, 2023-05-31T21:35:45.507 INFO:tasks.rgw_multisite_tests: "val": { 2023-05-31T21:35:45.508 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2023-05-31T21:35:45.508 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000031780", 2023-05-31T21:35:45.508 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-05-31T21:35:45.508 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-05-31T21:35:45.509 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-05-31T21:35:45.509 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-05-31T21:28:16.501293Z" 2023-05-31T21:35:45.509 INFO:tasks.rgw_multisite_tests: } 2023-05-31T21:35:45.509 INFO:tasks.rgw_multisite_tests: }, 2023-05-31T21:35:45.509 INFO:tasks.rgw_multisite_tests: { 2023-05-31T21:35:45.510 INFO:tasks.rgw_multisite_tests: "key": 2, 2023-05-31T21:35:45.510 INFO:tasks.rgw_multisite_tests: "val": { 2023-05-31T21:35:45.510 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2023-05-31T21:35:45.510 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000051671", 2023-05-31T21:35:45.511 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-05-31T21:35:45.511 INFO:tasks.rgw_multisite_tests: "total_entries": 55, 2023-05-31T21:35:45.511 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-05-31T21:35:45.511 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-05-31T21:28:16.476487Z" 2023-05-31T21:35:45.512 INFO:tasks.rgw_multisite_tests: } 2023-05-31T21:35:45.512 INFO:tasks.rgw_multisite_tests: }, 2023-05-31T21:35:45.512 INFO:tasks.rgw_multisite_tests: { 2023-05-31T21:35:45.512 INFO:tasks.rgw_multisite_tests: "key": 3, 2023-05-31T21:35:45.512 INFO:tasks.rgw_multisite_tests: "val": { 2023-05-31T21:35:45.513 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2023-05-31T21:35:45.513 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000044178", 2023-05-31T21:35:45.513 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-05-31T21:35:45.513 INFO:tasks.rgw_multisite_tests: "total_entries": 9, 2023-05-31T21:35:45.514 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-05-31T21:35:45.514 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-05-31T21:27:37.215905Z" 2023-05-31T21:35:45.514 INFO:tasks.rgw_multisite_tests: } 2023-05-31T21:35:45.514 INFO:tasks.rgw_multisite_tests: } 2023-05-31T21:35:45.515 INFO:tasks.rgw_multisite_tests: ] 2023-05-31T21:35:45.515 INFO:tasks.rgw_multisite_tests: }, 2023-05-31T21:35:45.515 INFO:tasks.rgw_multisite_tests: "full_sync": { 2023-05-31T21:35:45.515 INFO:tasks.rgw_multisite_tests: "total": 64, 2023-05-31T21:35:45.515 INFO:tasks.rgw_multisite_tests: "complete": 64 2023-05-31T21:35:45.516 INFO:tasks.rgw_multisite_tests: }, 2023-05-31T21:35:45.516 INFO:tasks.rgw_multisite_tests: "current_time": "2023-05-31T21:30:54Z" 2023-05-31T21:35:45.516 INFO:tasks.rgw_multisite_tests:} 2023-05-31T21:35:45.516 INFO:tasks.rgw_multisite_tests: 2023-05-31T21:35:45.517 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000045375', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-05-31T21:28:16.422581Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000031780', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-05-31T21:28:16.501293Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000051671', 'next_step_marker': '', 'total_entries': 55, 'pos': 0, 'timestamp': '2023-05-31T21:28:16.476487Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000044178', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2023-05-31T21:27:37.215905Z'}}] 2023-05-31T21:35:45.517 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '00000000000000000000:00000000000000045515', 1: '00000000000000000000:00000000000000031920', 2: '00000000000000000000:00000000000000051951', 3: '00000000000000000000:00000000000000044178'} 2023-05-31T21:35:45.517 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '00000000000000000000:00000000000000045375', 1: '00000000000000000000:00000000000000031780', 2: '00000000000000000000:00000000000000051671', 3: '00000000000000000000:00000000000000044178'} 2023-05-31T21:35:45.517 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: data of zone a2 behind zone a1: shard=0 master=00000000000000000000:00000000000000045515 target=00000000000000000000:00000000000000045375, shard=1 master=00000000000000000000:00000000000000031920 target=00000000000000000000:00000000000000031780, shard=2 master=00000000000000000000:00000000000000051951 target=00000000000000000000:00000000000000051671 2023-05-31T21:35:45.518 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-31T21:35:45.518 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-31T21:35:45.518 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-05-31T21:35:45.518 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: ignoring --setuser ceph since I am not root 2023-05-31T21:35:45.519 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-05-31T21:35:45.519 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2023-05-31T21:35:45.519 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "sync_status": { 2023-05-31T21:35:45.519 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "info": { 2023-05-31T21:35:45.520 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "status": "sync", 2023-05-31T21:35:45.520 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "num_shards": 4, 2023-05-31T21:35:45.520 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "instance_id": 452136274422638824 2023-05-31T21:35:45.520 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2023-05-31T21:35:45.520 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "markers": [ 2023-05-31T21:35:45.521 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2023-05-31T21:35:45.521 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "key": 0, 2023-05-31T21:35:45.521 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "val": { 2023-05-31T21:35:45.521 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "status": "incremental-sync", 2023-05-31T21:35:45.522 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "marker": "00000000000000000000:00000000000000045375", 2023-05-31T21:35:45.522 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "next_step_marker": "", 2023-05-31T21:35:45.522 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "total_entries": 0, 2023-05-31T21:35:45.522 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "pos": 0, 2023-05-31T21:35:45.523 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "timestamp": "2023-05-31T21:28:16.422581Z" 2023-05-31T21:35:45.523 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2023-05-31T21:35:45.523 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2023-05-31T21:35:45.523 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2023-05-31T21:35:45.523 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "key": 1, 2023-05-31T21:35:45.524 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "val": { 2023-05-31T21:35:45.524 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "status": "incremental-sync", 2023-05-31T21:35:45.524 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "marker": "00000000000000000000:00000000000000031780", 2023-05-31T21:35:45.524 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "next_step_marker": "", 2023-05-31T21:35:45.525 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "total_entries": 0, 2023-05-31T21:35:45.525 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "pos": 0, 2023-05-31T21:35:45.525 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "timestamp": "2023-05-31T21:28:16.501293Z" 2023-05-31T21:35:45.525 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2023-05-31T21:35:45.526 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2023-05-31T21:35:45.526 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2023-05-31T21:35:45.526 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "key": 2, 2023-05-31T21:35:45.526 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "val": { 2023-05-31T21:35:45.526 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "status": "incremental-sync", 2023-05-31T21:35:45.527 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "marker": "00000000000000000000:00000000000000051671", 2023-05-31T21:35:45.527 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "next_step_marker": "", 2023-05-31T21:35:45.527 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "total_entries": 55, 2023-05-31T21:35:45.527 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "pos": 0, 2023-05-31T21:35:45.528 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "timestamp": "2023-05-31T21:28:16.476487Z" 2023-05-31T21:35:45.528 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2023-05-31T21:35:45.528 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2023-05-31T21:35:45.528 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2023-05-31T21:35:45.529 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "key": 3, 2023-05-31T21:35:45.529 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "val": { 2023-05-31T21:35:45.529 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "status": "incremental-sync", 2023-05-31T21:35:45.529 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "marker": "00000000000000000000:00000000000000044178", 2023-05-31T21:35:45.529 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "next_step_marker": "", 2023-05-31T21:35:45.530 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "total_entries": 9, 2023-05-31T21:35:45.530 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "pos": 0, 2023-05-31T21:35:45.530 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "timestamp": "2023-05-31T21:27:37.215905Z" 2023-05-31T21:35:45.530 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2023-05-31T21:35:45.531 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2023-05-31T21:35:45.531 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: ] 2023-05-31T21:35:45.531 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2023-05-31T21:35:45.531 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "full_sync": { 2023-05-31T21:35:45.532 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "total": 64, 2023-05-31T21:35:45.532 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "complete": 64 2023-05-31T21:35:45.532 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2023-05-31T21:35:45.532 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "current_time": "2023-05-31T21:30:59Z" 2023-05-31T21:35:45.532 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2023-05-31T21:35:45.533 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current data sync status={ 2023-05-31T21:35:45.533 INFO:tasks.rgw_multisite_tests: "sync_status": { 2023-05-31T21:35:45.533 INFO:tasks.rgw_multisite_tests: "info": { 2023-05-31T21:35:45.533 INFO:tasks.rgw_multisite_tests: "status": "sync", 2023-05-31T21:35:45.534 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2023-05-31T21:35:45.534 INFO:tasks.rgw_multisite_tests: "instance_id": 452136274422638824 2023-05-31T21:35:45.534 INFO:tasks.rgw_multisite_tests: }, 2023-05-31T21:35:45.534 INFO:tasks.rgw_multisite_tests: "markers": [ 2023-05-31T21:35:45.535 INFO:tasks.rgw_multisite_tests: { 2023-05-31T21:35:45.535 INFO:tasks.rgw_multisite_tests: "key": 0, 2023-05-31T21:35:45.535 INFO:tasks.rgw_multisite_tests: "val": { 2023-05-31T21:35:45.535 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2023-05-31T21:35:45.535 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000045375", 2023-05-31T21:35:45.536 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-05-31T21:35:45.536 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-05-31T21:35:45.536 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-05-31T21:35:45.536 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-05-31T21:28:16.422581Z" 2023-05-31T21:35:45.537 INFO:tasks.rgw_multisite_tests: } 2023-05-31T21:35:45.537 INFO:tasks.rgw_multisite_tests: }, 2023-05-31T21:35:45.537 INFO:tasks.rgw_multisite_tests: { 2023-05-31T21:35:45.537 INFO:tasks.rgw_multisite_tests: "key": 1, 2023-05-31T21:35:45.538 INFO:tasks.rgw_multisite_tests: "val": { 2023-05-31T21:35:45.538 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2023-05-31T21:35:45.538 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000031780", 2023-05-31T21:35:45.538 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-05-31T21:35:45.539 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-05-31T21:35:45.539 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-05-31T21:35:45.539 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-05-31T21:28:16.501293Z" 2023-05-31T21:35:45.539 INFO:tasks.rgw_multisite_tests: } 2023-05-31T21:35:45.539 INFO:tasks.rgw_multisite_tests: }, 2023-05-31T21:35:45.540 INFO:tasks.rgw_multisite_tests: { 2023-05-31T21:35:45.540 INFO:tasks.rgw_multisite_tests: "key": 2, 2023-05-31T21:35:45.540 INFO:tasks.rgw_multisite_tests: "val": { 2023-05-31T21:35:45.540 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2023-05-31T21:35:45.541 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000051671", 2023-05-31T21:35:45.541 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-05-31T21:35:45.541 INFO:tasks.rgw_multisite_tests: "total_entries": 55, 2023-05-31T21:35:45.541 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-05-31T21:35:45.542 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-05-31T21:28:16.476487Z" 2023-05-31T21:35:45.542 INFO:tasks.rgw_multisite_tests: } 2023-05-31T21:35:45.542 INFO:tasks.rgw_multisite_tests: }, 2023-05-31T21:35:45.542 INFO:tasks.rgw_multisite_tests: { 2023-05-31T21:35:45.542 INFO:tasks.rgw_multisite_tests: "key": 3, 2023-05-31T21:35:45.543 INFO:tasks.rgw_multisite_tests: "val": { 2023-05-31T21:35:45.543 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2023-05-31T21:35:45.543 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000044178", 2023-05-31T21:35:45.543 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-05-31T21:35:45.544 INFO:tasks.rgw_multisite_tests: "total_entries": 9, 2023-05-31T21:35:45.544 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-05-31T21:35:45.544 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-05-31T21:27:37.215905Z" 2023-05-31T21:35:45.544 INFO:tasks.rgw_multisite_tests: } 2023-05-31T21:35:45.545 INFO:tasks.rgw_multisite_tests: } 2023-05-31T21:35:45.545 INFO:tasks.rgw_multisite_tests: ] 2023-05-31T21:35:45.545 INFO:tasks.rgw_multisite_tests: }, 2023-05-31T21:35:45.545 INFO:tasks.rgw_multisite_tests: "full_sync": { 2023-05-31T21:35:45.545 INFO:tasks.rgw_multisite_tests: "total": 64, 2023-05-31T21:35:45.546 INFO:tasks.rgw_multisite_tests: "complete": 64 2023-05-31T21:35:45.546 INFO:tasks.rgw_multisite_tests: }, 2023-05-31T21:35:45.546 INFO:tasks.rgw_multisite_tests: "current_time": "2023-05-31T21:30:59Z" 2023-05-31T21:35:45.546 INFO:tasks.rgw_multisite_tests:} 2023-05-31T21:35:45.547 INFO:tasks.rgw_multisite_tests: 2023-05-31T21:35:45.547 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000045375', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-05-31T21:28:16.422581Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000031780', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-05-31T21:28:16.501293Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000051671', 'next_step_marker': '', 'total_entries': 55, 'pos': 0, 'timestamp': '2023-05-31T21:28:16.476487Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000044178', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2023-05-31T21:27:37.215905Z'}}] 2023-05-31T21:35:45.547 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '00000000000000000000:00000000000000045515', 1: '00000000000000000000:00000000000000031920', 2: '00000000000000000000:00000000000000051951', 3: '00000000000000000000:00000000000000044178'} 2023-05-31T21:35:45.547 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '00000000000000000000:00000000000000045375', 1: '00000000000000000000:00000000000000031780', 2: '00000000000000000000:00000000000000051671', 3: '00000000000000000000:00000000000000044178'} 2023-05-31T21:35:45.548 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: data of zone a2 behind zone a1: shard=0 master=00000000000000000000:00000000000000045515 target=00000000000000000000:00000000000000045375, shard=1 master=00000000000000000000:00000000000000031920 target=00000000000000000000:00000000000000031780, shard=2 master=00000000000000000000:00000000000000051951 target=00000000000000000000:00000000000000051671 2023-05-31T21:35:45.548 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-31T21:35:45.548 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-31T21:35:45.548 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-05-31T21:35:45.549 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: ignoring --setuser ceph since I am not root 2023-05-31T21:35:45.549 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-05-31T21:35:45.549 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2023-05-31T21:35:45.549 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "sync_status": { 2023-05-31T21:35:45.550 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "info": { 2023-05-31T21:35:45.550 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "status": "sync", 2023-05-31T21:35:45.550 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "num_shards": 4, 2023-05-31T21:35:45.550 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "instance_id": 452136274422638824 2023-05-31T21:35:45.550 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2023-05-31T21:35:45.551 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "markers": [ 2023-05-31T21:35:45.551 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2023-05-31T21:35:45.551 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "key": 0, 2023-05-31T21:35:45.551 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "val": { 2023-05-31T21:35:45.552 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "status": "incremental-sync", 2023-05-31T21:35:45.552 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "marker": "00000000000000000000:00000000000000045375", 2023-05-31T21:35:45.552 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "next_step_marker": "", 2023-05-31T21:35:45.552 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "total_entries": 0, 2023-05-31T21:35:45.553 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "pos": 0, 2023-05-31T21:35:45.553 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "timestamp": "2023-05-31T21:28:16.422581Z" 2023-05-31T21:35:45.553 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2023-05-31T21:35:45.553 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2023-05-31T21:35:45.553 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2023-05-31T21:35:45.554 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "key": 1, 2023-05-31T21:35:45.554 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "val": { 2023-05-31T21:35:45.554 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "status": "incremental-sync", 2023-05-31T21:35:45.554 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "marker": "00000000000000000000:00000000000000031780", 2023-05-31T21:35:45.555 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "next_step_marker": "", 2023-05-31T21:35:45.555 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "total_entries": 0, 2023-05-31T21:35:45.555 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "pos": 0, 2023-05-31T21:35:45.555 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "timestamp": "2023-05-31T21:28:16.501293Z" 2023-05-31T21:35:45.556 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2023-05-31T21:35:45.556 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2023-05-31T21:35:45.556 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2023-05-31T21:35:45.556 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "key": 2, 2023-05-31T21:35:45.556 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "val": { 2023-05-31T21:35:45.557 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "status": "incremental-sync", 2023-05-31T21:35:45.557 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "marker": "00000000000000000000:00000000000000051671", 2023-05-31T21:35:45.557 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "next_step_marker": "", 2023-05-31T21:35:45.557 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "total_entries": 55, 2023-05-31T21:35:45.558 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "pos": 0, 2023-05-31T21:35:45.558 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "timestamp": "2023-05-31T21:28:16.476487Z" 2023-05-31T21:35:45.558 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2023-05-31T21:35:45.558 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2023-05-31T21:35:45.559 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2023-05-31T21:35:45.559 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "key": 3, 2023-05-31T21:35:45.559 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "val": { 2023-05-31T21:35:45.559 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "status": "incremental-sync", 2023-05-31T21:35:45.560 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "marker": "00000000000000000000:00000000000000044178", 2023-05-31T21:35:45.560 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "next_step_marker": "", 2023-05-31T21:35:45.560 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "total_entries": 9, 2023-05-31T21:35:45.560 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "pos": 0, 2023-05-31T21:35:45.560 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "timestamp": "2023-05-31T21:27:37.215905Z" 2023-05-31T21:35:45.561 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2023-05-31T21:35:45.561 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2023-05-31T21:35:45.561 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: ] 2023-05-31T21:35:45.561 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2023-05-31T21:35:45.562 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "full_sync": { 2023-05-31T21:35:45.562 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "total": 64, 2023-05-31T21:35:45.562 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "complete": 64 2023-05-31T21:35:45.562 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2023-05-31T21:35:45.563 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "current_time": "2023-05-31T21:31:04Z" 2023-05-31T21:35:45.563 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2023-05-31T21:35:45.563 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current data sync status={ 2023-05-31T21:35:45.563 INFO:tasks.rgw_multisite_tests: "sync_status": { 2023-05-31T21:35:45.563 INFO:tasks.rgw_multisite_tests: "info": { 2023-05-31T21:35:45.564 INFO:tasks.rgw_multisite_tests: "status": "sync", 2023-05-31T21:35:45.564 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2023-05-31T21:35:45.564 INFO:tasks.rgw_multisite_tests: "instance_id": 452136274422638824 2023-05-31T21:35:45.564 INFO:tasks.rgw_multisite_tests: }, 2023-05-31T21:35:45.565 INFO:tasks.rgw_multisite_tests: "markers": [ 2023-05-31T21:35:45.565 INFO:tasks.rgw_multisite_tests: { 2023-05-31T21:35:45.565 INFO:tasks.rgw_multisite_tests: "key": 0, 2023-05-31T21:35:45.565 INFO:tasks.rgw_multisite_tests: "val": { 2023-05-31T21:35:45.566 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2023-05-31T21:35:45.566 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000045375", 2023-05-31T21:35:45.566 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-05-31T21:35:45.566 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-05-31T21:35:45.566 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-05-31T21:35:45.567 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-05-31T21:28:16.422581Z" 2023-05-31T21:35:45.567 INFO:tasks.rgw_multisite_tests: } 2023-05-31T21:35:45.567 INFO:tasks.rgw_multisite_tests: }, 2023-05-31T21:35:45.567 INFO:tasks.rgw_multisite_tests: { 2023-05-31T21:35:45.568 INFO:tasks.rgw_multisite_tests: "key": 1, 2023-05-31T21:35:45.568 INFO:tasks.rgw_multisite_tests: "val": { 2023-05-31T21:35:45.568 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2023-05-31T21:35:45.568 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000031780", 2023-05-31T21:35:45.569 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-05-31T21:35:45.569 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-05-31T21:35:45.569 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-05-31T21:35:45.569 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-05-31T21:28:16.501293Z" 2023-05-31T21:35:45.570 INFO:tasks.rgw_multisite_tests: } 2023-05-31T21:35:45.570 INFO:tasks.rgw_multisite_tests: }, 2023-05-31T21:35:45.570 INFO:tasks.rgw_multisite_tests: { 2023-05-31T21:35:45.570 INFO:tasks.rgw_multisite_tests: "key": 2, 2023-05-31T21:35:45.570 INFO:tasks.rgw_multisite_tests: "val": { 2023-05-31T21:35:45.571 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2023-05-31T21:35:45.571 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000051671", 2023-05-31T21:35:45.571 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-05-31T21:35:45.571 INFO:tasks.rgw_multisite_tests: "total_entries": 55, 2023-05-31T21:35:45.572 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-05-31T21:35:45.572 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-05-31T21:28:16.476487Z" 2023-05-31T21:35:45.572 INFO:tasks.rgw_multisite_tests: } 2023-05-31T21:35:45.572 INFO:tasks.rgw_multisite_tests: }, 2023-05-31T21:35:45.573 INFO:tasks.rgw_multisite_tests: { 2023-05-31T21:35:45.573 INFO:tasks.rgw_multisite_tests: "key": 3, 2023-05-31T21:35:45.573 INFO:tasks.rgw_multisite_tests: "val": { 2023-05-31T21:35:45.573 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2023-05-31T21:35:45.573 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000044178", 2023-05-31T21:35:45.574 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-05-31T21:35:45.574 INFO:tasks.rgw_multisite_tests: "total_entries": 9, 2023-05-31T21:35:45.574 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-05-31T21:35:45.574 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-05-31T21:27:37.215905Z" 2023-05-31T21:35:45.575 INFO:tasks.rgw_multisite_tests: } 2023-05-31T21:35:45.575 INFO:tasks.rgw_multisite_tests: } 2023-05-31T21:35:45.575 INFO:tasks.rgw_multisite_tests: ] 2023-05-31T21:35:45.575 INFO:tasks.rgw_multisite_tests: }, 2023-05-31T21:35:45.576 INFO:tasks.rgw_multisite_tests: "full_sync": { 2023-05-31T21:35:45.576 INFO:tasks.rgw_multisite_tests: "total": 64, 2023-05-31T21:35:45.576 INFO:tasks.rgw_multisite_tests: "complete": 64 2023-05-31T21:35:45.576 INFO:tasks.rgw_multisite_tests: }, 2023-05-31T21:35:45.577 INFO:tasks.rgw_multisite_tests: "current_time": "2023-05-31T21:31:04Z" 2023-05-31T21:35:45.577 INFO:tasks.rgw_multisite_tests:} 2023-05-31T21:35:45.577 INFO:tasks.rgw_multisite_tests: 2023-05-31T21:35:45.577 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000045375', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-05-31T21:28:16.422581Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000031780', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-05-31T21:28:16.501293Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000051671', 'next_step_marker': '', 'total_entries': 55, 'pos': 0, 'timestamp': '2023-05-31T21:28:16.476487Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000044178', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2023-05-31T21:27:37.215905Z'}}] 2023-05-31T21:35:45.577 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '00000000000000000000:00000000000000045515', 1: '00000000000000000000:00000000000000031920', 2: '00000000000000000000:00000000000000051951', 3: '00000000000000000000:00000000000000044178'} 2023-05-31T21:35:45.578 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '00000000000000000000:00000000000000045375', 1: '00000000000000000000:00000000000000031780', 2: '00000000000000000000:00000000000000051671', 3: '00000000000000000000:00000000000000044178'} 2023-05-31T21:35:45.578 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: data of zone a2 behind zone a1: shard=0 master=00000000000000000000:00000000000000045515 target=00000000000000000000:00000000000000045375, shard=1 master=00000000000000000000:00000000000000031920 target=00000000000000000000:00000000000000031780, shard=2 master=00000000000000000000:00000000000000051951 target=00000000000000000000:00000000000000051671 2023-05-31T21:35:45.578 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-31T21:35:45.578 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-31T21:35:45.579 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-05-31T21:35:45.579 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: ignoring --setuser ceph since I am not root 2023-05-31T21:35:45.579 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-05-31T21:35:45.579 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2023-05-31T21:35:45.580 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "sync_status": { 2023-05-31T21:35:45.580 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "info": { 2023-05-31T21:35:45.580 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "status": "sync", 2023-05-31T21:35:45.580 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "num_shards": 4, 2023-05-31T21:35:45.581 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "instance_id": 452136274422638824 2023-05-31T21:35:45.581 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2023-05-31T21:35:45.581 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "markers": [ 2023-05-31T21:35:45.581 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2023-05-31T21:35:45.581 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "key": 0, 2023-05-31T21:35:45.582 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "val": { 2023-05-31T21:35:45.582 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "status": "incremental-sync", 2023-05-31T21:35:45.582 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "marker": "00000000000000000000:00000000000000045375", 2023-05-31T21:35:45.582 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "next_step_marker": "", 2023-05-31T21:35:45.583 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "total_entries": 0, 2023-05-31T21:35:45.583 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "pos": 0, 2023-05-31T21:35:45.583 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "timestamp": "2023-05-31T21:28:16.422581Z" 2023-05-31T21:35:45.583 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2023-05-31T21:35:45.584 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2023-05-31T21:35:45.584 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2023-05-31T21:35:45.584 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "key": 1, 2023-05-31T21:35:45.584 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "val": { 2023-05-31T21:35:45.585 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "status": "incremental-sync", 2023-05-31T21:35:45.585 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "marker": "00000000000000000000:00000000000000031780", 2023-05-31T21:35:45.585 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "next_step_marker": "", 2023-05-31T21:35:45.585 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "total_entries": 0, 2023-05-31T21:35:45.585 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "pos": 0, 2023-05-31T21:35:45.586 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "timestamp": "2023-05-31T21:28:16.501293Z" 2023-05-31T21:35:45.586 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2023-05-31T21:35:45.586 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2023-05-31T21:35:45.586 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2023-05-31T21:35:45.587 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "key": 2, 2023-05-31T21:35:45.587 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "val": { 2023-05-31T21:35:45.587 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "status": "incremental-sync", 2023-05-31T21:35:45.587 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "marker": "00000000000000000000:00000000000000051671", 2023-05-31T21:35:45.587 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "next_step_marker": "", 2023-05-31T21:35:45.588 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "total_entries": 55, 2023-05-31T21:35:45.588 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "pos": 0, 2023-05-31T21:35:45.588 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "timestamp": "2023-05-31T21:28:16.476487Z" 2023-05-31T21:35:45.588 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2023-05-31T21:35:45.589 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2023-05-31T21:35:45.589 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2023-05-31T21:35:45.589 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "key": 3, 2023-05-31T21:35:45.589 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "val": { 2023-05-31T21:35:45.590 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "status": "incremental-sync", 2023-05-31T21:35:45.590 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "marker": "00000000000000000000:00000000000000044178", 2023-05-31T21:35:45.590 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "next_step_marker": "", 2023-05-31T21:35:45.590 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "total_entries": 9, 2023-05-31T21:35:45.591 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "pos": 0, 2023-05-31T21:35:45.591 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "timestamp": "2023-05-31T21:27:37.215905Z" 2023-05-31T21:35:45.591 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2023-05-31T21:35:45.591 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2023-05-31T21:35:45.591 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: ] 2023-05-31T21:35:45.592 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2023-05-31T21:35:45.592 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "full_sync": { 2023-05-31T21:35:45.592 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "total": 64, 2023-05-31T21:35:45.592 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "complete": 64 2023-05-31T21:35:45.593 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2023-05-31T21:35:45.593 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "current_time": "2023-05-31T21:31:09Z" 2023-05-31T21:35:45.593 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2023-05-31T21:35:45.593 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current data sync status={ 2023-05-31T21:35:45.594 INFO:tasks.rgw_multisite_tests: "sync_status": { 2023-05-31T21:35:45.594 INFO:tasks.rgw_multisite_tests: "info": { 2023-05-31T21:35:45.594 INFO:tasks.rgw_multisite_tests: "status": "sync", 2023-05-31T21:35:45.594 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2023-05-31T21:35:45.594 INFO:tasks.rgw_multisite_tests: "instance_id": 452136274422638824 2023-05-31T21:35:45.595 INFO:tasks.rgw_multisite_tests: }, 2023-05-31T21:35:45.595 INFO:tasks.rgw_multisite_tests: "markers": [ 2023-05-31T21:35:45.595 INFO:tasks.rgw_multisite_tests: { 2023-05-31T21:35:45.595 INFO:tasks.rgw_multisite_tests: "key": 0, 2023-05-31T21:35:45.596 INFO:tasks.rgw_multisite_tests: "val": { 2023-05-31T21:35:45.596 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2023-05-31T21:35:45.596 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000045375", 2023-05-31T21:35:45.596 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-05-31T21:35:45.596 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-05-31T21:35:45.597 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-05-31T21:35:45.597 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-05-31T21:28:16.422581Z" 2023-05-31T21:35:45.597 INFO:tasks.rgw_multisite_tests: } 2023-05-31T21:35:45.597 INFO:tasks.rgw_multisite_tests: }, 2023-05-31T21:35:45.598 INFO:tasks.rgw_multisite_tests: { 2023-05-31T21:35:45.598 INFO:tasks.rgw_multisite_tests: "key": 1, 2023-05-31T21:35:45.598 INFO:tasks.rgw_multisite_tests: "val": { 2023-05-31T21:35:45.598 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2023-05-31T21:35:45.599 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000031780", 2023-05-31T21:35:45.599 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-05-31T21:35:45.599 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-05-31T21:35:45.599 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-05-31T21:35:45.600 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-05-31T21:28:16.501293Z" 2023-05-31T21:35:45.600 INFO:tasks.rgw_multisite_tests: } 2023-05-31T21:35:45.600 INFO:tasks.rgw_multisite_tests: }, 2023-05-31T21:35:45.600 INFO:tasks.rgw_multisite_tests: { 2023-05-31T21:35:45.600 INFO:tasks.rgw_multisite_tests: "key": 2, 2023-05-31T21:35:45.601 INFO:tasks.rgw_multisite_tests: "val": { 2023-05-31T21:35:45.601 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2023-05-31T21:35:45.601 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000051671", 2023-05-31T21:35:45.601 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-05-31T21:35:45.602 INFO:tasks.rgw_multisite_tests: "total_entries": 55, 2023-05-31T21:35:45.602 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-05-31T21:35:45.602 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-05-31T21:28:16.476487Z" 2023-05-31T21:35:45.602 INFO:tasks.rgw_multisite_tests: } 2023-05-31T21:35:45.603 INFO:tasks.rgw_multisite_tests: }, 2023-05-31T21:35:45.603 INFO:tasks.rgw_multisite_tests: { 2023-05-31T21:35:45.603 INFO:tasks.rgw_multisite_tests: "key": 3, 2023-05-31T21:35:45.603 INFO:tasks.rgw_multisite_tests: "val": { 2023-05-31T21:35:45.603 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2023-05-31T21:35:45.604 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000044178", 2023-05-31T21:35:45.604 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-05-31T21:35:45.604 INFO:tasks.rgw_multisite_tests: "total_entries": 9, 2023-05-31T21:35:45.604 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-05-31T21:35:45.605 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-05-31T21:27:37.215905Z" 2023-05-31T21:35:45.605 INFO:tasks.rgw_multisite_tests: } 2023-05-31T21:35:45.605 INFO:tasks.rgw_multisite_tests: } 2023-05-31T21:35:45.605 INFO:tasks.rgw_multisite_tests: ] 2023-05-31T21:35:45.606 INFO:tasks.rgw_multisite_tests: }, 2023-05-31T21:35:45.606 INFO:tasks.rgw_multisite_tests: "full_sync": { 2023-05-31T21:35:45.606 INFO:tasks.rgw_multisite_tests: "total": 64, 2023-05-31T21:35:45.606 INFO:tasks.rgw_multisite_tests: "complete": 64 2023-05-31T21:35:45.606 INFO:tasks.rgw_multisite_tests: }, 2023-05-31T21:35:45.607 INFO:tasks.rgw_multisite_tests: "current_time": "2023-05-31T21:31:09Z" 2023-05-31T21:35:45.607 INFO:tasks.rgw_multisite_tests:} 2023-05-31T21:35:45.607 INFO:tasks.rgw_multisite_tests: 2023-05-31T21:35:45.607 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000045375', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-05-31T21:28:16.422581Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000031780', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-05-31T21:28:16.501293Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000051671', 'next_step_marker': '', 'total_entries': 55, 'pos': 0, 'timestamp': '2023-05-31T21:28:16.476487Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000044178', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2023-05-31T21:27:37.215905Z'}}] 2023-05-31T21:35:45.608 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '00000000000000000000:00000000000000045515', 1: '00000000000000000000:00000000000000031920', 2: '00000000000000000000:00000000000000051951', 3: '00000000000000000000:00000000000000044178'} 2023-05-31T21:35:45.608 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '00000000000000000000:00000000000000045375', 1: '00000000000000000000:00000000000000031780', 2: '00000000000000000000:00000000000000051671', 3: '00000000000000000000:00000000000000044178'} 2023-05-31T21:35:45.608 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: data of zone a2 behind zone a1: shard=0 master=00000000000000000000:00000000000000045515 target=00000000000000000000:00000000000000045375, shard=1 master=00000000000000000000:00000000000000031920 target=00000000000000000000:00000000000000031780, shard=2 master=00000000000000000000:00000000000000051951 target=00000000000000000000:00000000000000051671 2023-05-31T21:35:45.608 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-31T21:35:45.609 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-31T21:35:45.609 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-05-31T21:35:45.609 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: ignoring --setuser ceph since I am not root 2023-05-31T21:35:45.609 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-05-31T21:35:45.610 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2023-05-31T21:35:45.610 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "sync_status": { 2023-05-31T21:35:45.610 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "info": { 2023-05-31T21:35:45.610 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "status": "sync", 2023-05-31T21:35:45.610 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "num_shards": 4, 2023-05-31T21:35:45.611 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "instance_id": 452136274422638824 2023-05-31T21:35:45.611 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2023-05-31T21:35:45.611 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "markers": [ 2023-05-31T21:35:45.611 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2023-05-31T21:35:45.612 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "key": 0, 2023-05-31T21:35:45.612 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "val": { 2023-05-31T21:35:45.612 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "status": "incremental-sync", 2023-05-31T21:35:45.612 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "marker": "00000000000000000000:00000000000000045375", 2023-05-31T21:35:45.613 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "next_step_marker": "", 2023-05-31T21:35:45.613 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "total_entries": 0, 2023-05-31T21:35:45.613 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "pos": 0, 2023-05-31T21:35:45.613 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "timestamp": "2023-05-31T21:28:16.422581Z" 2023-05-31T21:35:45.614 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2023-05-31T21:35:45.614 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2023-05-31T21:35:45.614 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2023-05-31T21:35:45.614 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "key": 1, 2023-05-31T21:35:45.614 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "val": { 2023-05-31T21:35:45.615 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "status": "incremental-sync", 2023-05-31T21:35:45.615 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "marker": "00000000000000000000:00000000000000031780", 2023-05-31T21:35:45.615 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "next_step_marker": "", 2023-05-31T21:35:45.615 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "total_entries": 0, 2023-05-31T21:35:45.616 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "pos": 0, 2023-05-31T21:35:45.616 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "timestamp": "2023-05-31T21:28:16.501293Z" 2023-05-31T21:35:45.616 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2023-05-31T21:35:45.616 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2023-05-31T21:35:45.617 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2023-05-31T21:35:45.617 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "key": 2, 2023-05-31T21:35:45.617 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "val": { 2023-05-31T21:35:45.617 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "status": "incremental-sync", 2023-05-31T21:35:45.617 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "marker": "00000000000000000000:00000000000000051671", 2023-05-31T21:35:45.618 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "next_step_marker": "", 2023-05-31T21:35:45.618 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "total_entries": 55, 2023-05-31T21:35:45.618 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "pos": 0, 2023-05-31T21:35:45.618 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "timestamp": "2023-05-31T21:28:16.476487Z" 2023-05-31T21:35:45.619 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2023-05-31T21:35:45.619 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2023-05-31T21:35:45.619 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2023-05-31T21:35:45.619 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "key": 3, 2023-05-31T21:35:45.620 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "val": { 2023-05-31T21:35:45.620 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "status": "incremental-sync", 2023-05-31T21:35:45.620 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "marker": "00000000000000000000:00000000000000044178", 2023-05-31T21:35:45.620 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "next_step_marker": "", 2023-05-31T21:35:45.620 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "total_entries": 9, 2023-05-31T21:35:45.621 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "pos": 0, 2023-05-31T21:35:45.621 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "timestamp": "2023-05-31T21:27:37.215905Z" 2023-05-31T21:35:45.621 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2023-05-31T21:35:45.621 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2023-05-31T21:35:45.622 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: ] 2023-05-31T21:35:45.622 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2023-05-31T21:35:45.622 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "full_sync": { 2023-05-31T21:35:45.622 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "total": 64, 2023-05-31T21:35:45.623 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "complete": 64 2023-05-31T21:35:45.623 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2023-05-31T21:35:45.623 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "current_time": "2023-05-31T21:31:15Z" 2023-05-31T21:35:45.623 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2023-05-31T21:35:45.623 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current data sync status={ 2023-05-31T21:35:45.624 INFO:tasks.rgw_multisite_tests: "sync_status": { 2023-05-31T21:35:45.624 INFO:tasks.rgw_multisite_tests: "info": { 2023-05-31T21:35:45.624 INFO:tasks.rgw_multisite_tests: "status": "sync", 2023-05-31T21:35:45.624 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2023-05-31T21:35:45.625 INFO:tasks.rgw_multisite_tests: "instance_id": 452136274422638824 2023-05-31T21:35:45.625 INFO:tasks.rgw_multisite_tests: }, 2023-05-31T21:35:45.625 INFO:tasks.rgw_multisite_tests: "markers": [ 2023-05-31T21:35:45.625 INFO:tasks.rgw_multisite_tests: { 2023-05-31T21:35:45.626 INFO:tasks.rgw_multisite_tests: "key": 0, 2023-05-31T21:35:45.626 INFO:tasks.rgw_multisite_tests: "val": { 2023-05-31T21:35:45.626 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2023-05-31T21:35:45.626 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000045375", 2023-05-31T21:35:45.626 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-05-31T21:35:45.627 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-05-31T21:35:45.627 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-05-31T21:35:45.627 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-05-31T21:28:16.422581Z" 2023-05-31T21:35:45.627 INFO:tasks.rgw_multisite_tests: } 2023-05-31T21:35:45.628 INFO:tasks.rgw_multisite_tests: }, 2023-05-31T21:35:45.628 INFO:tasks.rgw_multisite_tests: { 2023-05-31T21:35:45.628 INFO:tasks.rgw_multisite_tests: "key": 1, 2023-05-31T21:35:45.628 INFO:tasks.rgw_multisite_tests: "val": { 2023-05-31T21:35:45.629 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2023-05-31T21:35:45.629 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000031780", 2023-05-31T21:35:45.629 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-05-31T21:35:45.629 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-05-31T21:35:45.629 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-05-31T21:35:45.630 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-05-31T21:28:16.501293Z" 2023-05-31T21:35:45.630 INFO:tasks.rgw_multisite_tests: } 2023-05-31T21:35:45.630 INFO:tasks.rgw_multisite_tests: }, 2023-05-31T21:35:45.630 INFO:tasks.rgw_multisite_tests: { 2023-05-31T21:35:45.631 INFO:tasks.rgw_multisite_tests: "key": 2, 2023-05-31T21:35:45.631 INFO:tasks.rgw_multisite_tests: "val": { 2023-05-31T21:35:45.631 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2023-05-31T21:35:45.631 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000051671", 2023-05-31T21:35:45.632 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-05-31T21:35:45.632 INFO:tasks.rgw_multisite_tests: "total_entries": 55, 2023-05-31T21:35:45.632 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-05-31T21:35:45.632 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-05-31T21:28:16.476487Z" 2023-05-31T21:35:45.632 INFO:tasks.rgw_multisite_tests: } 2023-05-31T21:35:45.633 INFO:tasks.rgw_multisite_tests: }, 2023-05-31T21:35:45.633 INFO:tasks.rgw_multisite_tests: { 2023-05-31T21:35:45.633 INFO:tasks.rgw_multisite_tests: "key": 3, 2023-05-31T21:35:45.633 INFO:tasks.rgw_multisite_tests: "val": { 2023-05-31T21:35:45.634 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2023-05-31T21:35:45.634 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000044178", 2023-05-31T21:35:45.634 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-05-31T21:35:45.634 INFO:tasks.rgw_multisite_tests: "total_entries": 9, 2023-05-31T21:35:45.635 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-05-31T21:35:45.635 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-05-31T21:27:37.215905Z" 2023-05-31T21:35:45.635 INFO:tasks.rgw_multisite_tests: } 2023-05-31T21:35:45.635 INFO:tasks.rgw_multisite_tests: } 2023-05-31T21:35:45.635 INFO:tasks.rgw_multisite_tests: ] 2023-05-31T21:35:45.636 INFO:tasks.rgw_multisite_tests: }, 2023-05-31T21:35:45.636 INFO:tasks.rgw_multisite_tests: "full_sync": { 2023-05-31T21:35:45.636 INFO:tasks.rgw_multisite_tests: "total": 64, 2023-05-31T21:35:45.636 INFO:tasks.rgw_multisite_tests: "complete": 64 2023-05-31T21:35:45.637 INFO:tasks.rgw_multisite_tests: }, 2023-05-31T21:35:45.637 INFO:tasks.rgw_multisite_tests: "current_time": "2023-05-31T21:31:15Z" 2023-05-31T21:35:45.637 INFO:tasks.rgw_multisite_tests:} 2023-05-31T21:35:45.637 INFO:tasks.rgw_multisite_tests: 2023-05-31T21:35:45.638 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000045375', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-05-31T21:28:16.422581Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000031780', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-05-31T21:28:16.501293Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000051671', 'next_step_marker': '', 'total_entries': 55, 'pos': 0, 'timestamp': '2023-05-31T21:28:16.476487Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000044178', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2023-05-31T21:27:37.215905Z'}}] 2023-05-31T21:35:45.638 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '00000000000000000000:00000000000000045515', 1: '00000000000000000000:00000000000000031920', 2: '00000000000000000000:00000000000000051951', 3: '00000000000000000000:00000000000000044178'} 2023-05-31T21:35:45.638 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '00000000000000000000:00000000000000045375', 1: '00000000000000000000:00000000000000031780', 2: '00000000000000000000:00000000000000051671', 3: '00000000000000000000:00000000000000044178'} 2023-05-31T21:35:45.638 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: data of zone a2 behind zone a1: shard=0 master=00000000000000000000:00000000000000045515 target=00000000000000000000:00000000000000045375, shard=1 master=00000000000000000000:00000000000000031920 target=00000000000000000000:00000000000000031780, shard=2 master=00000000000000000000:00000000000000051951 target=00000000000000000000:00000000000000051671 2023-05-31T21:35:45.638 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-31T21:35:45.639 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-31T21:35:45.639 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-05-31T21:35:45.639 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: ignoring --setuser ceph since I am not root 2023-05-31T21:35:45.639 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-05-31T21:35:45.640 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2023-05-31T21:35:45.640 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "sync_status": { 2023-05-31T21:35:45.640 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "info": { 2023-05-31T21:35:45.640 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "status": "sync", 2023-05-31T21:35:45.641 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "num_shards": 4, 2023-05-31T21:35:45.641 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "instance_id": 452136274422638824 2023-05-31T21:35:45.641 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2023-05-31T21:35:45.641 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "markers": [ 2023-05-31T21:35:45.642 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2023-05-31T21:35:45.642 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "key": 0, 2023-05-31T21:35:45.642 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "val": { 2023-05-31T21:35:45.642 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "status": "incremental-sync", 2023-05-31T21:35:45.643 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "marker": "00000000000000000000:00000000000000045375", 2023-05-31T21:35:45.643 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "next_step_marker": "", 2023-05-31T21:35:45.643 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "total_entries": 0, 2023-05-31T21:35:45.643 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "pos": 0, 2023-05-31T21:35:45.643 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "timestamp": "2023-05-31T21:28:16.422581Z" 2023-05-31T21:35:45.644 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2023-05-31T21:35:45.644 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2023-05-31T21:35:45.644 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2023-05-31T21:35:45.644 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "key": 1, 2023-05-31T21:35:45.645 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "val": { 2023-05-31T21:35:45.645 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "status": "incremental-sync", 2023-05-31T21:35:45.645 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "marker": "00000000000000000000:00000000000000031780", 2023-05-31T21:35:45.645 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "next_step_marker": "", 2023-05-31T21:35:45.646 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "total_entries": 0, 2023-05-31T21:35:45.646 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "pos": 0, 2023-05-31T21:35:45.646 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "timestamp": "2023-05-31T21:28:16.501293Z" 2023-05-31T21:35:45.646 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2023-05-31T21:35:45.646 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2023-05-31T21:35:45.647 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2023-05-31T21:35:45.647 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "key": 2, 2023-05-31T21:35:45.647 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "val": { 2023-05-31T21:35:45.647 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "status": "incremental-sync", 2023-05-31T21:35:45.648 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "marker": "00000000000000000000:00000000000000051671", 2023-05-31T21:35:45.648 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "next_step_marker": "", 2023-05-31T21:35:45.648 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "total_entries": 55, 2023-05-31T21:35:45.648 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "pos": 0, 2023-05-31T21:35:45.649 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "timestamp": "2023-05-31T21:28:16.476487Z" 2023-05-31T21:35:45.649 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2023-05-31T21:35:45.649 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2023-05-31T21:35:45.649 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2023-05-31T21:35:45.649 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "key": 3, 2023-05-31T21:35:45.650 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "val": { 2023-05-31T21:35:45.650 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "status": "incremental-sync", 2023-05-31T21:35:45.650 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "marker": "00000000000000000000:00000000000000044178", 2023-05-31T21:35:45.650 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "next_step_marker": "", 2023-05-31T21:35:45.651 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "total_entries": 9, 2023-05-31T21:35:45.651 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "pos": 0, 2023-05-31T21:35:45.651 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "timestamp": "2023-05-31T21:27:37.215905Z" 2023-05-31T21:35:45.651 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2023-05-31T21:35:45.652 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2023-05-31T21:35:45.652 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: ] 2023-05-31T21:35:45.652 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2023-05-31T21:35:45.652 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "full_sync": { 2023-05-31T21:35:45.653 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "total": 64, 2023-05-31T21:35:45.653 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "complete": 64 2023-05-31T21:35:45.653 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2023-05-31T21:35:45.653 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "current_time": "2023-05-31T21:31:20Z" 2023-05-31T21:35:45.654 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2023-05-31T21:35:45.654 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current data sync status={ 2023-05-31T21:35:45.654 INFO:tasks.rgw_multisite_tests: "sync_status": { 2023-05-31T21:35:45.654 INFO:tasks.rgw_multisite_tests: "info": { 2023-05-31T21:35:45.654 INFO:tasks.rgw_multisite_tests: "status": "sync", 2023-05-31T21:35:45.655 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2023-05-31T21:35:45.655 INFO:tasks.rgw_multisite_tests: "instance_id": 452136274422638824 2023-05-31T21:35:45.655 INFO:tasks.rgw_multisite_tests: }, 2023-05-31T21:35:45.655 INFO:tasks.rgw_multisite_tests: "markers": [ 2023-05-31T21:35:45.656 INFO:tasks.rgw_multisite_tests: { 2023-05-31T21:35:45.656 INFO:tasks.rgw_multisite_tests: "key": 0, 2023-05-31T21:35:45.656 INFO:tasks.rgw_multisite_tests: "val": { 2023-05-31T21:35:45.656 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2023-05-31T21:35:45.657 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000045375", 2023-05-31T21:35:45.657 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-05-31T21:35:45.657 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-05-31T21:35:45.657 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-05-31T21:35:45.658 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-05-31T21:28:16.422581Z" 2023-05-31T21:35:45.658 INFO:tasks.rgw_multisite_tests: } 2023-05-31T21:35:45.658 INFO:tasks.rgw_multisite_tests: }, 2023-05-31T21:35:45.658 INFO:tasks.rgw_multisite_tests: { 2023-05-31T21:35:45.658 INFO:tasks.rgw_multisite_tests: "key": 1, 2023-05-31T21:35:45.659 INFO:tasks.rgw_multisite_tests: "val": { 2023-05-31T21:35:45.659 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2023-05-31T21:35:45.659 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000031780", 2023-05-31T21:35:45.659 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-05-31T21:35:45.660 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-05-31T21:35:45.660 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-05-31T21:35:45.660 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-05-31T21:28:16.501293Z" 2023-05-31T21:35:45.661 INFO:tasks.rgw_multisite_tests: } 2023-05-31T21:35:45.661 INFO:tasks.rgw_multisite_tests: }, 2023-05-31T21:35:45.661 INFO:tasks.rgw_multisite_tests: { 2023-05-31T21:35:45.661 INFO:tasks.rgw_multisite_tests: "key": 2, 2023-05-31T21:35:45.662 INFO:tasks.rgw_multisite_tests: "val": { 2023-05-31T21:35:45.662 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2023-05-31T21:35:45.662 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000051671", 2023-05-31T21:35:45.662 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-05-31T21:35:45.662 INFO:tasks.rgw_multisite_tests: "total_entries": 55, 2023-05-31T21:35:45.663 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-05-31T21:35:45.663 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-05-31T21:28:16.476487Z" 2023-05-31T21:35:45.663 INFO:tasks.rgw_multisite_tests: } 2023-05-31T21:35:45.663 INFO:tasks.rgw_multisite_tests: }, 2023-05-31T21:35:45.664 INFO:tasks.rgw_multisite_tests: { 2023-05-31T21:35:45.664 INFO:tasks.rgw_multisite_tests: "key": 3, 2023-05-31T21:35:45.664 INFO:tasks.rgw_multisite_tests: "val": { 2023-05-31T21:35:45.664 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2023-05-31T21:35:45.665 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000044178", 2023-05-31T21:35:45.665 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-05-31T21:35:45.665 INFO:tasks.rgw_multisite_tests: "total_entries": 9, 2023-05-31T21:35:45.665 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-05-31T21:35:45.666 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-05-31T21:27:37.215905Z" 2023-05-31T21:35:45.666 INFO:tasks.rgw_multisite_tests: } 2023-05-31T21:35:45.666 INFO:tasks.rgw_multisite_tests: } 2023-05-31T21:35:45.666 INFO:tasks.rgw_multisite_tests: ] 2023-05-31T21:35:45.666 INFO:tasks.rgw_multisite_tests: }, 2023-05-31T21:35:45.667 INFO:tasks.rgw_multisite_tests: "full_sync": { 2023-05-31T21:35:45.667 INFO:tasks.rgw_multisite_tests: "total": 64, 2023-05-31T21:35:45.667 INFO:tasks.rgw_multisite_tests: "complete": 64 2023-05-31T21:35:45.667 INFO:tasks.rgw_multisite_tests: }, 2023-05-31T21:35:45.668 INFO:tasks.rgw_multisite_tests: "current_time": "2023-05-31T21:31:20Z" 2023-05-31T21:35:45.668 INFO:tasks.rgw_multisite_tests:} 2023-05-31T21:35:45.668 INFO:tasks.rgw_multisite_tests: 2023-05-31T21:35:45.668 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000045375', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-05-31T21:28:16.422581Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000031780', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-05-31T21:28:16.501293Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000051671', 'next_step_marker': '', 'total_entries': 55, 'pos': 0, 'timestamp': '2023-05-31T21:28:16.476487Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000044178', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2023-05-31T21:27:37.215905Z'}}] 2023-05-31T21:35:45.669 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '00000000000000000000:00000000000000045515', 1: '00000000000000000000:00000000000000031920', 2: '00000000000000000000:00000000000000051951', 3: '00000000000000000000:00000000000000044178'} 2023-05-31T21:35:45.669 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '00000000000000000000:00000000000000045375', 1: '00000000000000000000:00000000000000031780', 2: '00000000000000000000:00000000000000051671', 3: '00000000000000000000:00000000000000044178'} 2023-05-31T21:35:45.669 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: data of zone a2 behind zone a1: shard=0 master=00000000000000000000:00000000000000045515 target=00000000000000000000:00000000000000045375, shard=1 master=00000000000000000000:00000000000000031920 target=00000000000000000000:00000000000000031780, shard=2 master=00000000000000000000:00000000000000051951 target=00000000000000000000:00000000000000051671 2023-05-31T21:35:45.669 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-31T21:35:45.670 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-05-31T21:35:45.670 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-05-31T21:35:45.670 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: ignoring --setuser ceph since I am not root 2023-05-31T21:35:45.670 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-05-31T21:35:45.670 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2023-05-31T21:35:45.671 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "sync_status": { 2023-05-31T21:35:45.671 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "info": { 2023-05-31T21:35:45.671 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "status": "sync", 2023-05-31T21:35:45.671 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "num_shards": 4, 2023-05-31T21:35:45.672 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "instance_id": 452136274422638824 2023-05-31T21:35:45.672 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2023-05-31T21:35:45.672 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "markers": [ 2023-05-31T21:35:45.672 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2023-05-31T21:35:45.673 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "key": 0, 2023-05-31T21:35:45.673 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "val": { 2023-05-31T21:35:45.673 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "status": "incremental-sync", 2023-05-31T21:35:45.673 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "marker": "00000000000000000000:00000000000000045515", 2023-05-31T21:35:45.674 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "next_step_marker": "", 2023-05-31T21:35:45.674 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "total_entries": 0, 2023-05-31T21:35:45.674 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "pos": 0, 2023-05-31T21:35:45.674 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "timestamp": "2023-05-31T21:29:57.506834Z" 2023-05-31T21:35:45.674 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2023-05-31T21:35:45.675 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2023-05-31T21:35:45.675 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2023-05-31T21:35:45.675 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "key": 1, 2023-05-31T21:35:45.675 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "val": { 2023-05-31T21:35:45.676 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "status": "incremental-sync", 2023-05-31T21:35:45.676 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "marker": "00000000000000000000:00000000000000031920", 2023-05-31T21:35:45.676 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "next_step_marker": "", 2023-05-31T21:35:45.676 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "total_entries": 0, 2023-05-31T21:35:45.677 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "pos": 0, 2023-05-31T21:35:45.677 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "timestamp": "2023-05-31T21:29:57.583636Z" 2023-05-31T21:35:45.677 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2023-05-31T21:35:45.677 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2023-05-31T21:35:45.677 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2023-05-31T21:35:45.678 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "key": 2, 2023-05-31T21:35:45.678 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "val": { 2023-05-31T21:35:45.678 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "status": "incremental-sync", 2023-05-31T21:35:45.678 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "marker": "00000000000000000000:00000000000000051951", 2023-05-31T21:35:45.679 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "next_step_marker": "", 2023-05-31T21:35:45.679 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "total_entries": 55, 2023-05-31T21:35:45.679 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "pos": 0, 2023-05-31T21:35:45.679 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "timestamp": "2023-05-31T21:29:57.526899Z" 2023-05-31T21:35:45.680 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2023-05-31T21:35:45.680 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2023-05-31T21:35:45.680 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: { 2023-05-31T21:35:45.680 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "key": 3, 2023-05-31T21:35:45.680 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "val": { 2023-05-31T21:35:45.681 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "status": "incremental-sync", 2023-05-31T21:35:45.681 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "marker": "00000000000000000000:00000000000000044178", 2023-05-31T21:35:45.681 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "next_step_marker": "", 2023-05-31T21:35:45.681 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "total_entries": 9, 2023-05-31T21:35:45.682 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "pos": 0, 2023-05-31T21:35:45.682 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "timestamp": "2023-05-31T21:27:37.215905Z" 2023-05-31T21:35:45.682 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2023-05-31T21:35:45.682 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2023-05-31T21:35:45.683 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: ] 2023-05-31T21:35:45.683 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2023-05-31T21:35:45.683 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "full_sync": { 2023-05-31T21:35:45.683 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "total": 64, 2023-05-31T21:35:45.683 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "complete": 64 2023-05-31T21:35:45.684 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: }, 2023-05-31T21:35:45.684 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: "current_time": "2023-05-31T21:31:25Z" 2023-05-31T21:35:45.684 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi099.stdout: INFO: } 2023-05-31T21:35:45.684 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current data sync status={ 2023-05-31T21:35:45.685 INFO:tasks.rgw_multisite_tests: "sync_status": { 2023-05-31T21:35:45.685 INFO:tasks.rgw_multisite_tests: "info": { 2023-05-31T21:35:45.685 INFO:tasks.rgw_multisite_tests: "status": "sync", 2023-05-31T21:35:45.685 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2023-05-31T21:35:45.686 INFO:tasks.rgw_multisite_tests: "instance_id": 452136274422638824 2023-05-31T21:35:45.686 INFO:tasks.rgw_multisite_tests: }, 2023-05-31T21:35:45.686 INFO:tasks.rgw_multisite_tests: "markers": [ 2023-05-31T21:35:45.686 INFO:tasks.rgw_multisite_tests: { 2023-05-31T21:35:45.686 INFO:tasks.rgw_multisite_tests: "key": 0, 2023-05-31T21:35:45.687 INFO:tasks.rgw_multisite_tests: "val": { 2023-05-31T21:35:45.687 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2023-05-31T21:35:45.687 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000045515", 2023-05-31T21:35:45.687 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-05-31T21:35:45.688 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-05-31T21:35:45.688 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-05-31T21:35:45.688 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-05-31T21:29:57.506834Z" 2023-05-31T21:35:45.688 INFO:tasks.rgw_multisite_tests: } 2023-05-31T21:35:45.689 INFO:tasks.rgw_multisite_tests: }, 2023-05-31T21:35:45.689 INFO:tasks.rgw_multisite_tests: { 2023-05-31T21:35:45.689 INFO:tasks.rgw_multisite_tests: "key": 1, 2023-05-31T21:35:45.689 INFO:tasks.rgw_multisite_tests: "val": { 2023-05-31T21:35:45.689 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2023-05-31T21:35:45.690 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000031920", 2023-05-31T21:35:45.690 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-05-31T21:35:45.690 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-05-31T21:35:45.690 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-05-31T21:35:45.691 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-05-31T21:29:57.583636Z" 2023-05-31T21:35:45.691 INFO:tasks.rgw_multisite_tests: } 2023-05-31T21:35:45.691 INFO:tasks.rgw_multisite_tests: }, 2023-05-31T21:35:45.691 INFO:tasks.rgw_multisite_tests: { 2023-05-31T21:35:45.692 INFO:tasks.rgw_multisite_tests: "key": 2, 2023-05-31T21:35:45.692 INFO:tasks.rgw_multisite_tests: "val": { 2023-05-31T21:35:45.692 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2023-05-31T21:35:45.692 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000051951", 2023-05-31T21:35:45.692 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-05-31T21:35:45.693 INFO:tasks.rgw_multisite_tests: "total_entries": 55, 2023-05-31T21:35:45.693 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-05-31T21:35:45.693 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-05-31T21:29:57.526899Z" 2023-05-31T21:35:45.693 INFO:tasks.rgw_multisite_tests: } 2023-05-31T21:35:45.694 INFO:tasks.rgw_multisite_tests: }, 2023-05-31T21:35:45.694 INFO:tasks.rgw_multisite_tests: { 2023-05-31T21:35:45.694 INFO:tasks.rgw_multisite_tests: "key": 3, 2023-05-31T21:35:45.694 INFO:tasks.rgw_multisite_tests: "val": { 2023-05-31T21:35:45.695 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2023-05-31T21:35:45.695 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000044178", 2023-05-31T21:35:45.695 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-05-31T21:35:45.695 INFO:tasks.rgw_multisite_tests: "total_entries": 9, 2023-05-31T21:35:45.695 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-05-31T21:35:45.696 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-05-31T21:27:37.215905Z" 2023-05-31T21:35:45.696 INFO:tasks.rgw_multisite_tests: } 2023-05-31T21:35:45.696 INFO:tasks.rgw_multisite_tests: } 2023-05-31T21:35:45.696 INFO:tasks.rgw_multisite_tests: ] 2023-05-31T21:35:45.697 INFO:tasks.rgw_multisite_tests: }, 2023-05-31T21:35:45.697 INFO:tasks.rgw_multisite_tests: "full_sync": { 2023-05-31T21:35:45.697 INFO:tasks.rgw_multisite_tests: "total": 64, 2023-05-31T21:35:45.697 INFO:tasks.rgw_multisite_tests: "complete": 64 2023-05-31T21:35:45.698 INFO:tasks.rgw_multisite_tests: }, 2023-05-31T21:35:45.698 INFO:tasks.rgw_multisite_tests: "current_time": "2023-05-31T21:31:25Z" 2023-05-31T21:35:45.698 INFO:tasks.rgw_multisite_tests:} 2023-05-31T21:35:45.698 INFO:tasks.rgw_multisite_tests: 2023-05-31T21:35:45.699 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000045515', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-05-31T21:29:57.506834Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000031920', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-05-31T21:29:57.583636Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000051951', 'next_step_marker': '', 'total_entries': 55, 'pos': 0, 'timestamp': '2023-05-31T21:29:57.526899Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000044178', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2023-05-31T21:27:37.215905Z'}}] 2023-05-31T21:35:45.699 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '00000000000000000000:00000000000000045515', 1: '00000000000000000000:00000000000000031920', 2: '00000000000000000000:00000000000000051951', 3: '00000000000000000000:00000000000000044178'} 2023-05-31T21:35:45.699 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '00000000000000000000:00000000000000045515', 1: '00000000000000000000:00000000000000031920', 2: '00000000000000000000:00000000000000051951', 3: '00000000000000000000:00000000000000044178'} 2023-05-31T21:35:45.699 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: INFO: finished data checkpoint for target_zone=a2 source_zone=a1 2023-05-31T21:35:45.700 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/pklxkd-73/ 2023-05-31T21:35:45.700 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/pklxkd-73/ 2023-05-31T21:35:45.700 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-05-31T21:35:45.700 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /pklxkd-73/ 2023-05-31T21:35:45.700 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-05-31T21:35:45.701 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-05-31T21:35:45.701 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T21:35:45.701 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-05-31T21:35:45.701 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-05-31T21:35:45.702 INFO:tasks.rgw_multisite_tests:boto: DEBUG: establishing HTTP connection: kwargs={'timeout': 70, 'port': 8001} 2023-05-31T21:35:45.702 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-05-31T21:35:45.702 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-05-31T21:35:45.702 INFO:tasks.rgw_multisite_tests:HEAD 2023-05-31T21:35:45.703 INFO:tasks.rgw_multisite_tests: 2023-05-31T21:35:45.703 INFO:tasks.rgw_multisite_tests: 2023-05-31T21:35:45.703 INFO:tasks.rgw_multisite_tests:Wed, 31 May 2023 21:31:25 GMT 2023-05-31T21:35:45.703 INFO:tasks.rgw_multisite_tests:/pklxkd-73/ 2023-05-31T21:35:45.703 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-05-31T21:35:45.704 INFO:tasks.rgw_multisite_tests:AWS Q80H11OLNU1S48PPYL26:cYK3RC61Sar3CdxyPNRXpbeWjvE= 2023-05-31T21:35:45.704 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 21:31:25 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:cYK3RC61Sar3CdxyPNRXpbeWjvE=', 'Content-Length': '0'} 2023-05-31T21:35:45.704 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '4'), ('X-RGW-Bytes-Used', '24'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000bc957d57825cd06b-006477bcad-295a-a2'), ('Content-Length', '0'), ('Date', 'Wed, 31 May 2023 21:31:25 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:35:45.704 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/pklxkd-73/obj5 2023-05-31T21:35:45.705 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/pklxkd-73/obj5 2023-05-31T21:35:45.705 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-05-31T21:35:45.705 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /pklxkd-73/obj5 2023-05-31T21:35:45.705 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-05-31T21:35:45.706 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-05-31T21:35:45.706 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T21:35:45.706 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-05-31T21:35:45.706 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-05-31T21:35:45.706 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-05-31T21:35:45.707 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-05-31T21:35:45.707 INFO:tasks.rgw_multisite_tests:HEAD 2023-05-31T21:35:45.707 INFO:tasks.rgw_multisite_tests: 2023-05-31T21:35:45.707 INFO:tasks.rgw_multisite_tests: 2023-05-31T21:35:45.708 INFO:tasks.rgw_multisite_tests:Wed, 31 May 2023 21:31:25 GMT 2023-05-31T21:35:45.708 INFO:tasks.rgw_multisite_tests:/pklxkd-73/obj5 2023-05-31T21:35:45.708 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-05-31T21:35:45.708 INFO:tasks.rgw_multisite_tests:AWS Q80H11OLNU1S48PPYL26:N6FKCXiG2bbfLpMfh4xP9yHtt1M= 2023-05-31T21:35:45.709 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 21:31:25 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:N6FKCXiG2bbfLpMfh4xP9yHtt1M=', 'Content-Length': '0'} 2023-05-31T21:35:45.709 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '219'), ('x-amz-request-id', 'tx000002a03875ef34b6cda-006477bcad-295a-a2'), ('Accept-Ranges', 'bytes'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 21:31:25 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:35:45.709 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/pklxkd-73/obj6 2023-05-31T21:35:45.709 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/pklxkd-73/obj6 2023-05-31T21:35:45.709 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-05-31T21:35:45.710 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /pklxkd-73/obj6 2023-05-31T21:35:45.710 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-05-31T21:35:45.710 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-05-31T21:35:45.710 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T21:35:45.711 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-05-31T21:35:45.711 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-05-31T21:35:45.711 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-05-31T21:35:45.711 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-05-31T21:35:45.712 INFO:tasks.rgw_multisite_tests:HEAD 2023-05-31T21:35:45.712 INFO:tasks.rgw_multisite_tests: 2023-05-31T21:35:45.712 INFO:tasks.rgw_multisite_tests: 2023-05-31T21:35:45.712 INFO:tasks.rgw_multisite_tests:Wed, 31 May 2023 21:31:25 GMT 2023-05-31T21:35:45.712 INFO:tasks.rgw_multisite_tests:/pklxkd-73/obj6 2023-05-31T21:35:45.713 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-05-31T21:35:45.713 INFO:tasks.rgw_multisite_tests:AWS Q80H11OLNU1S48PPYL26:WgJD09mmQ3RaO79aaziMNsPqJ4s= 2023-05-31T21:35:45.713 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 21:31:25 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:WgJD09mmQ3RaO79aaziMNsPqJ4s=', 'Content-Length': '0'} 2023-05-31T21:35:45.713 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '219'), ('x-amz-request-id', 'tx00000f681d5f2b207003b-006477bcad-295a-a2'), ('Accept-Ranges', 'bytes'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 21:31:25 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:35:45.714 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/pklxkd-73/obj7 2023-05-31T21:35:45.714 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/pklxkd-73/obj7 2023-05-31T21:35:45.714 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-05-31T21:35:45.714 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /pklxkd-73/obj7 2023-05-31T21:35:45.715 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-05-31T21:35:45.715 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-05-31T21:35:45.715 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T21:35:45.715 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-05-31T21:35:45.715 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-05-31T21:35:45.716 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-05-31T21:35:45.716 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-05-31T21:35:45.716 INFO:tasks.rgw_multisite_tests:HEAD 2023-05-31T21:35:45.716 INFO:tasks.rgw_multisite_tests: 2023-05-31T21:35:45.717 INFO:tasks.rgw_multisite_tests: 2023-05-31T21:35:45.717 INFO:tasks.rgw_multisite_tests:Wed, 31 May 2023 21:31:25 GMT 2023-05-31T21:35:45.717 INFO:tasks.rgw_multisite_tests:/pklxkd-73/obj7 2023-05-31T21:35:45.717 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-05-31T21:35:45.718 INFO:tasks.rgw_multisite_tests:AWS Q80H11OLNU1S48PPYL26:ooZUcfP99buMy0+sJJzFhS8cZ2U= 2023-05-31T21:35:45.718 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 21:31:25 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:ooZUcfP99buMy0+sJJzFhS8cZ2U=', 'Content-Length': '0'} 2023-05-31T21:35:45.718 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '219'), ('x-amz-request-id', 'tx00000ad3c678afc74ce10-006477bcad-295a-a2'), ('Accept-Ranges', 'bytes'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 21:31:25 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:35:45.718 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/pklxkd-73/obj8 2023-05-31T21:35:45.719 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/pklxkd-73/obj8 2023-05-31T21:35:45.719 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-05-31T21:35:45.719 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /pklxkd-73/obj8 2023-05-31T21:35:45.719 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-05-31T21:35:45.719 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-05-31T21:35:45.720 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T21:35:45.720 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-05-31T21:35:45.720 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-05-31T21:35:45.720 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-05-31T21:35:45.721 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-05-31T21:35:45.721 INFO:tasks.rgw_multisite_tests:HEAD 2023-05-31T21:35:45.721 INFO:tasks.rgw_multisite_tests: 2023-05-31T21:35:45.721 INFO:tasks.rgw_multisite_tests: 2023-05-31T21:35:45.722 INFO:tasks.rgw_multisite_tests:Wed, 31 May 2023 21:31:25 GMT 2023-05-31T21:35:45.722 INFO:tasks.rgw_multisite_tests:/pklxkd-73/obj8 2023-05-31T21:35:45.722 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-05-31T21:35:45.722 INFO:tasks.rgw_multisite_tests:AWS Q80H11OLNU1S48PPYL26:Lk9j/QesqAD3pRQVIyHpc6MdWXo= 2023-05-31T21:35:45.722 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 21:31:25 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:Lk9j/QesqAD3pRQVIyHpc6MdWXo=', 'Content-Length': '0'} 2023-05-31T21:35:45.723 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '219'), ('x-amz-request-id', 'tx0000069f425a68fd8ca8d-006477bcad-295a-a2'), ('Accept-Ranges', 'bytes'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 21:31:25 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:35:45.723 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/pklxkd-72/ 2023-05-31T21:35:45.723 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/pklxkd-72/ 2023-05-31T21:35:45.723 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-05-31T21:35:45.724 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /pklxkd-72/ 2023-05-31T21:35:45.724 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-05-31T21:35:45.724 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-05-31T21:35:45.724 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T21:35:45.725 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-05-31T21:35:45.725 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-05-31T21:35:45.725 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-05-31T21:35:45.725 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-05-31T21:35:45.725 INFO:tasks.rgw_multisite_tests:HEAD 2023-05-31T21:35:45.726 INFO:tasks.rgw_multisite_tests: 2023-05-31T21:35:45.726 INFO:tasks.rgw_multisite_tests: 2023-05-31T21:35:45.726 INFO:tasks.rgw_multisite_tests:Wed, 31 May 2023 21:31:25 GMT 2023-05-31T21:35:45.726 INFO:tasks.rgw_multisite_tests:/pklxkd-72/ 2023-05-31T21:35:45.727 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-05-31T21:35:45.727 INFO:tasks.rgw_multisite_tests:AWS Q80H11OLNU1S48PPYL26:Xe9FH9UqMdZbBaujXAKWuxKWVTE= 2023-05-31T21:35:45.727 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 21:31:25 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:Xe9FH9UqMdZbBaujXAKWuxKWVTE=', 'Content-Length': '0'} 2023-05-31T21:35:45.727 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '7'), ('X-RGW-Bytes-Used', '42'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000006c7d2125e423be82-006477bcad-295a-a2'), ('Content-Length', '0'), ('Date', 'Wed, 31 May 2023 21:31:25 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:35:45.728 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/pklxkd-72/obj7 2023-05-31T21:35:45.728 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/pklxkd-72/obj7 2023-05-31T21:35:45.728 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-05-31T21:35:45.728 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /pklxkd-72/obj7 2023-05-31T21:35:45.728 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-05-31T21:35:45.729 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-05-31T21:35:45.729 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T21:35:45.729 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-05-31T21:35:45.729 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-05-31T21:35:45.730 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-05-31T21:35:45.730 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-05-31T21:35:45.730 INFO:tasks.rgw_multisite_tests:HEAD 2023-05-31T21:35:45.730 INFO:tasks.rgw_multisite_tests: 2023-05-31T21:35:45.731 INFO:tasks.rgw_multisite_tests: 2023-05-31T21:35:45.731 INFO:tasks.rgw_multisite_tests:Wed, 31 May 2023 21:31:25 GMT 2023-05-31T21:35:45.731 INFO:tasks.rgw_multisite_tests:/pklxkd-72/obj7 2023-05-31T21:35:45.731 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-05-31T21:35:45.731 INFO:tasks.rgw_multisite_tests:AWS Q80H11OLNU1S48PPYL26:Z68UmYLQOCkNnU1WTAXLIMzn71g= 2023-05-31T21:35:45.732 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 21:31:25 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:Z68UmYLQOCkNnU1WTAXLIMzn71g=', 'Content-Length': '0'} 2023-05-31T21:35:45.732 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 21:29:57 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000cecdf4b02bc22f7d-006477bcad-295a-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 21:31:25 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:35:45.732 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/pklxkd-72/obj7 2023-05-31T21:35:45.732 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/pklxkd-72/obj7 2023-05-31T21:35:45.733 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-05-31T21:35:45.733 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /pklxkd-72/obj7 2023-05-31T21:35:45.733 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-05-31T21:35:45.733 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-05-31T21:35:45.734 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T21:35:45.734 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-05-31T21:35:45.734 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-05-31T21:35:45.734 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-05-31T21:35:45.735 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-05-31T21:35:45.735 INFO:tasks.rgw_multisite_tests:GET 2023-05-31T21:35:45.735 INFO:tasks.rgw_multisite_tests: 2023-05-31T21:35:45.735 INFO:tasks.rgw_multisite_tests: 2023-05-31T21:35:45.735 INFO:tasks.rgw_multisite_tests:Wed, 31 May 2023 21:31:25 GMT 2023-05-31T21:35:45.736 INFO:tasks.rgw_multisite_tests:/pklxkd-72/obj7 2023-05-31T21:35:45.736 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-05-31T21:35:45.736 INFO:tasks.rgw_multisite_tests:AWS Q80H11OLNU1S48PPYL26:Cl7zD9Lr0agN3NIzRfSnFdJitNk= 2023-05-31T21:35:45.736 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 21:31:25 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:Cl7zD9Lr0agN3NIzRfSnFdJitNk=', 'Content-Length': '0'} 2023-05-31T21:35:45.737 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 31 May 2023 21:29:57 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000004095f79069098877-006477bcad-295a-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 31 May 2023 21:31:25 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:35:45.737 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/pklxkd-72/obj8 2023-05-31T21:35:45.737 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/pklxkd-72/obj8 2023-05-31T21:35:45.737 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-05-31T21:35:45.738 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /pklxkd-72/obj8 2023-05-31T21:35:45.738 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-05-31T21:35:45.738 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-05-31T21:35:45.738 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi099.front.sepia.ceph.com:8001 2023-05-31T21:35:45.739 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-05-31T21:35:45.739 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-05-31T21:35:45.739 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-05-31T21:35:45.739 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-05-31T21:35:45.739 INFO:tasks.rgw_multisite_tests:HEAD 2023-05-31T21:35:45.740 INFO:tasks.rgw_multisite_tests: 2023-05-31T21:35:45.740 INFO:tasks.rgw_multisite_tests: 2023-05-31T21:35:45.740 INFO:tasks.rgw_multisite_tests:Wed, 31 May 2023 21:31:25 GMT 2023-05-31T21:35:45.740 INFO:tasks.rgw_multisite_tests:/pklxkd-72/obj8 2023-05-31T21:35:45.741 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-05-31T21:35:45.741 INFO:tasks.rgw_multisite_tests:AWS Q80H11OLNU1S48PPYL26:9gkSEfJQ+mw6AjehtS35Kk9cpxo= 2023-05-31T21:35:45.741 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Wed, 31 May 2023 21:31:25 GMT', 'Authorization': 'AWS Q80H11OLNU1S48PPYL26:9gkSEfJQ+mw6AjehtS35Kk9cpxo=', 'Content-Length': '0'} 2023-05-31T21:35:45.741 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '219'), ('x-amz-request-id', 'tx000009a9277fda1c01384-006477bcad-295a-a2'), ('Accept-Ranges', 'bytes'), ('Content-Type', 'application/xml'), ('Date', 'Wed, 31 May 2023 21:31:25 GMT'), ('Connection', 'Keep-Alive')] 2023-05-31T21:35:45.742 INFO:tasks.rgw_multisite_tests:--------------------- >> end captured logging << --------------------- 2023-05-31T21:35:45.742 INFO:tasks.rgw_multisite_tests: 2023-05-31T21:35:45.743 INFO:tasks.rgw_multisite_tests:---------------------------------------------------------------------- 2023-05-31T21:35:45.743 INFO:tasks.rgw_multisite_tests:Ran 47 tests in 6187.236s 2023-05-31T21:35:45.743 INFO:tasks.rgw_multisite_tests: 2023-05-31T21:35:45.743 INFO:tasks.rgw_multisite_tests:FAILED (SKIP=4, failures=7) 2023-05-31T21:35:45.747 ERROR:teuthology.run_tasks:Saw exception from tasks. Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_fa97ddd17d12e6d74b49196948f66e54fa009130/teuthology/run_tasks.py", line 109, in run_tasks manager.__enter__() File "/home/teuthworker/src/git.ceph.com_teuthology_fa97ddd17d12e6d74b49196948f66e54fa009130/teuthology/task/__init__.py", line 123, in __enter__ self.begin() File "/home/teuthworker/src/github.com_ceph_ceph-c_2870bf456d3115490c153abce9d47ac5c043b65e/qa/tasks/rgw_multisite_tests.py", line 95, in begin assert nose.run(argv=argv, config=conf), 'rgw multisite test failures' AssertionError: rgw multisite test failures 2023-05-31T21:35:45.819 ERROR:teuthology.run_tasks: Sentry event: https://sentry.ceph.com/organizations/ceph/?query=8d201981081f4c40857626305f95bc8e Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_fa97ddd17d12e6d74b49196948f66e54fa009130/teuthology/run_tasks.py", line 109, in run_tasks manager.__enter__() File "/home/teuthworker/src/git.ceph.com_teuthology_fa97ddd17d12e6d74b49196948f66e54fa009130/teuthology/task/__init__.py", line 123, in __enter__ self.begin() File "/home/teuthworker/src/github.com_ceph_ceph-c_2870bf456d3115490c153abce9d47ac5c043b65e/qa/tasks/rgw_multisite_tests.py", line 95, in begin assert nose.run(argv=argv, config=conf), 'rgw multisite test failures' AssertionError: rgw multisite test failures 2023-05-31T21:35:45.821 DEBUG:teuthology.run_tasks:Unwinding manager rgw-multisite-tests 2023-05-31T21:35:45.831 DEBUG:teuthology.run_tasks:Unwinding manager rgw-multisite 2023-05-31T21:35:45.840 DEBUG:teuthology.run_tasks:Unwinding manager rgw 2023-05-31T21:35:45.849 DEBUG:tasks.rgw.c1.client.0:waiting for process to exit 2023-05-31T21:35:45.849 INFO:teuthology.orchestra.run:waiting for 300 2023-05-31T21:35:45.852 INFO:tasks.rgw.c1.client.0.smithi099.stdout:2023-05-31T21:35:45.846+0000 7f3b15c07700 -1 received signal: Terminated from /usr/bin/python3 /bin/daemon-helper term radosgw --rgw-frontends beast port=8000 -n client.0 --cluster c1 -k /etc/ceph/c1.client.0.keyring --log-file /var/log/ceph/rgw.c1.client.0.log --rgw_ops_log_socket_path /home/ubuntu/cephtest/rgw.opslog.c1.client.0.sock --foreground --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm (PID: 112618) UID: 0 2023-05-31T21:35:45.853 INFO:tasks.rgw.c1.client.0.smithi099.stdout:2023-05-31T21:35:45.846+0000 7f3b19e707c0 -1 shutting down 2023-05-31T21:35:46.455 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.0 is failed for ~0s 2023-05-31T21:35:51.951 INFO:tasks.rgw.c1.client.0:Stopped 2023-05-31T21:35:51.952 DEBUG:teuthology.orchestra.run.smithi099:> rm -f /home/ubuntu/cephtest/rgw.opslog.c1.client.0.sock 2023-05-31T21:35:51.975 DEBUG:teuthology.orchestra.run.smithi099:> sudo rm -f /etc/ceph/vault-root-token 2023-05-31T21:35:52.048 DEBUG:tasks.rgw.c1.client.1:waiting for process to exit 2023-05-31T21:35:52.048 INFO:teuthology.orchestra.run:waiting for 300 2023-05-31T21:35:52.060 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.0 is failed for ~6s 2023-05-31T21:35:52.089 INFO:tasks.rgw.c1.client.1.smithi099.stdout:2023-05-31T21:35:52.085+0000 7f6c43a31700 -1 received signal: Terminated from /usr/bin/python3 /bin/daemon-helper term radosgw --rgw-frontends beast port=8001 -n client.1 --cluster c1 -k /etc/ceph/c1.client.1.keyring --log-file /var/log/ceph/rgw.c1.client.1.log --rgw_ops_log_socket_path /home/ubuntu/cephtest/rgw.opslog.c1.client.1.sock --foreground --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm (PID: 173001) UID: 0 2023-05-31T21:35:52.089 INFO:tasks.rgw.c1.client.1.smithi099.stdout:2023-05-31T21:35:52.085+0000 7f6c47c9a7c0 -1 shutting down 2023-05-31T21:35:57.562 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.1 is failed for ~0s 2023-05-31T21:35:57.563 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.0 has been restored 2023-05-31T21:35:58.150 INFO:tasks.rgw.c1.client.1:Stopped 2023-05-31T21:35:58.151 DEBUG:teuthology.orchestra.run.smithi099:> rm -f /home/ubuntu/cephtest/rgw.opslog.c1.client.1.sock 2023-05-31T21:35:58.173 DEBUG:teuthology.orchestra.run.smithi099:> sudo rm -f /etc/ceph/vault-root-token 2023-05-31T21:35:58.244 DEBUG:tasks.rgw.c2.client.0:waiting for process to exit 2023-05-31T21:35:58.244 INFO:teuthology.orchestra.run:waiting for 300 2023-05-31T21:35:58.245 INFO:tasks.rgw.c2.client.0.smithi112.stdout:2023-05-31T21:35:58.241+0000 7f4456c12700 -1 received signal: Terminated from /usr/bin/python3 /bin/daemon-helper term radosgw --rgw-frontends beast port=8000 -n client.0 --cluster c2 -k /etc/ceph/c2.client.0.keyring --log-file /var/log/ceph/rgw.c2.client.0.log --rgw_ops_log_socket_path /home/ubuntu/cephtest/rgw.opslog.c2.client.0.sock --foreground --rgw-zone b1 --rgw-zonegroup b --rgw-realm test-realm (PID: 112484) UID: 0 2023-05-31T21:35:58.246 INFO:tasks.rgw.c2.client.0.smithi112.stdout:2023-05-31T21:35:58.241+0000 7f445ae7b7c0 -1 shutting down 2023-05-31T21:36:03.067 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.1 has been restored 2023-05-31T21:36:03.568 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.rgw.client.0 is failed for ~0s 2023-05-31T21:36:04.346 INFO:tasks.rgw.c2.client.0:Stopped 2023-05-31T21:36:04.346 DEBUG:teuthology.orchestra.run.smithi112:> rm -f /home/ubuntu/cephtest/rgw.opslog.c2.client.0.sock 2023-05-31T21:36:04.370 DEBUG:teuthology.orchestra.run.smithi112:> sudo rm -f /etc/ceph/vault-root-token 2023-05-31T21:36:04.444 DEBUG:tasks.rgw.c2.client.1:waiting for process to exit 2023-05-31T21:36:04.445 INFO:teuthology.orchestra.run:waiting for 300 2023-05-31T21:36:04.485 INFO:tasks.rgw.c2.client.1.smithi112.stdout:2023-05-31T21:36:04.482+0000 7fecce05c700 -1 received signal: Terminated from /usr/bin/python3 /bin/daemon-helper term radosgw --rgw-frontends beast port=8001 -n client.1 --cluster c2 -k /etc/ceph/c2.client.1.keyring --log-file /var/log/ceph/rgw.c2.client.1.log --rgw_ops_log_socket_path /home/ubuntu/cephtest/rgw.opslog.c2.client.1.sock --foreground --rgw-zone b2 --rgw-zonegroup b --rgw-realm test-realm (PID: 113779) UID: 0 2023-05-31T21:36:04.486 INFO:tasks.rgw.c2.client.1.smithi112.stdout:2023-05-31T21:36:04.482+0000 7fecd22c57c0 -1 shutting down 2023-05-31T21:36:09.071 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.rgw.client.1 is failed for ~0s 2023-05-31T21:36:09.072 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.rgw.client.0 has been restored 2023-05-31T21:36:10.547 INFO:tasks.rgw.c2.client.1:Stopped 2023-05-31T21:36:10.547 DEBUG:teuthology.orchestra.run.smithi112:> rm -f /home/ubuntu/cephtest/rgw.opslog.c2.client.1.sock 2023-05-31T21:36:10.570 DEBUG:teuthology.orchestra.run.smithi112:> sudo rm -f /etc/ceph/vault-root-token 2023-05-31T21:36:10.641 DEBUG:teuthology.run_tasks:Unwinding manager ceph 2023-05-31T21:36:10.650 DEBUG:teuthology.orchestra.run.smithi112:> sudo ceph --cluster c2 config set global mon_health_to_clog false 2023-05-31T21:36:11.017 INFO:teuthology.misc:Shutting down mds daemons... 2023-05-31T21:36:11.018 INFO:teuthology.misc:Shutting down osd daemons... 2023-05-31T21:36:11.018 DEBUG:tasks.ceph.c2.osd.0:waiting for process to exit 2023-05-31T21:36:11.019 INFO:teuthology.orchestra.run:waiting for 300 2023-05-31T21:36:11.060 INFO:tasks.ceph.c2.osd.0:Stopped 2023-05-31T21:36:11.060 DEBUG:tasks.ceph.c2.osd.1:waiting for process to exit 2023-05-31T21:36:11.060 INFO:teuthology.orchestra.run:waiting for 300 2023-05-31T21:36:11.101 INFO:tasks.ceph.c2.osd.1:Stopped 2023-05-31T21:36:11.101 DEBUG:tasks.ceph.c2.osd.2:waiting for process to exit 2023-05-31T21:36:11.102 INFO:teuthology.orchestra.run:waiting for 300 2023-05-31T21:36:11.153 INFO:tasks.ceph.c2.osd.2:Stopped 2023-05-31T21:36:11.154 INFO:teuthology.misc:Shutting down mgr daemons... 2023-05-31T21:36:11.154 DEBUG:tasks.ceph.c2.mgr.x:waiting for process to exit 2023-05-31T21:36:11.154 INFO:teuthology.orchestra.run:waiting for 300 2023-05-31T21:36:11.177 INFO:tasks.ceph.c2.mgr.x:Stopped 2023-05-31T21:36:11.178 INFO:teuthology.misc:Shutting down mon daemons... 2023-05-31T21:36:11.178 DEBUG:tasks.ceph.c2.mon.a:waiting for process to exit 2023-05-31T21:36:11.179 INFO:teuthology.orchestra.run:waiting for 300 2023-05-31T21:36:11.235 INFO:tasks.ceph.c2.mon.a:Stopped 2023-05-31T21:36:11.235 INFO:tasks.ceph:Checking cluster log for badness... 2023-05-31T21:36:11.236 DEBUG:teuthology.orchestra.run.smithi112:> sudo egrep '\[ERR\]|\[WRN\]|\[SEC\]' /var/log/ceph/c2.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v '\(PG_AVAILABILITY\)' | egrep -v '\(PG_DEGRADED\)' | egrep -v '\(POOL_APP_NOT_ENABLED\)' | head -n 1 2023-05-31T21:36:11.304 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/c2-0 on ubuntu@smithi112.front.sepia.ceph.com 2023-05-31T21:36:11.305 DEBUG:teuthology.orchestra.run.smithi112:> sync && sudo umount -f /var/lib/ceph/osd/c2-0 2023-05-31T21:36:11.721 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/c2-1 on ubuntu@smithi112.front.sepia.ceph.com 2023-05-31T21:36:11.722 DEBUG:teuthology.orchestra.run.smithi112:> sync && sudo umount -f /var/lib/ceph/osd/c2-1 2023-05-31T21:36:11.796 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/c2-2 on ubuntu@smithi112.front.sepia.ceph.com 2023-05-31T21:36:11.797 DEBUG:teuthology.orchestra.run.smithi112:> sync && sudo umount -f /var/lib/ceph/osd/c2-2 2023-05-31T21:36:11.864 INFO:tasks.ceph:Archiving mon data... 2023-05-31T21:36:11.866 DEBUG:teuthology.misc:Transferring archived files from smithi112:/var/lib/ceph/mon/c2-a to /home/teuthworker/archive/yuriw-2023-05-31_19:25:20-rgw-reef-release-distro-default-smithi/7292285/data/c2.mon.a.tgz 2023-05-31T21:36:11.866 DEBUG:teuthology.orchestra.run.smithi112:> mktemp 2023-05-31T21:36:11.884 INFO:teuthology.orchestra.run.smithi112.stdout:/tmp/tmp.eQZRWKZPai 2023-05-31T21:36:11.885 DEBUG:teuthology.orchestra.run.smithi112:> sudo tar cz -f - -C /var/lib/ceph/mon/c2-a -- . > /tmp/tmp.eQZRWKZPai 2023-05-31T21:36:12.064 DEBUG:teuthology.orchestra.run.smithi112:> sudo chmod 0666 /tmp/tmp.eQZRWKZPai 2023-05-31T21:36:12.151 DEBUG:teuthology.orchestra.remote:smithi112:/tmp/tmp.eQZRWKZPai is 404KB 2023-05-31T21:36:12.245 DEBUG:teuthology.orchestra.run.smithi112:> rm -fr /tmp/tmp.eQZRWKZPai 2023-05-31T21:36:12.262 INFO:tasks.ceph:Cleaning ceph cluster... 2023-05-31T21:36:12.262 DEBUG:teuthology.orchestra.run.smithi099:> sudo rm -rf -- /etc/ceph/c2.conf /etc/ceph/c2.keyring /home/ubuntu/cephtest/c2.data /home/ubuntu/cephtest/c2.monmap /home/ubuntu/cephtest/../*.pid 2023-05-31T21:36:12.265 DEBUG:teuthology.orchestra.run.smithi112:> sudo rm -rf -- /etc/ceph/c2.conf /etc/ceph/c2.keyring /home/ubuntu/cephtest/c2.data /home/ubuntu/cephtest/c2.monmap /home/ubuntu/cephtest/../*.pid 2023-05-31T21:36:12.327 DEBUG:teuthology.run_tasks:Unwinding manager ceph 2023-05-31T21:36:12.338 DEBUG:teuthology.orchestra.run.smithi099:> sudo ceph --cluster c1 config set global mon_health_to_clog false 2023-05-31T21:36:12.677 INFO:teuthology.misc:Shutting down mds daemons... 2023-05-31T21:36:12.677 INFO:teuthology.misc:Shutting down osd daemons... 2023-05-31T21:36:12.678 DEBUG:tasks.ceph.c1.osd.0:waiting for process to exit 2023-05-31T21:36:12.678 INFO:teuthology.orchestra.run:waiting for 300 2023-05-31T21:36:12.726 INFO:tasks.ceph.c1.osd.0:Stopped 2023-05-31T21:36:12.726 DEBUG:tasks.ceph.c1.osd.1:waiting for process to exit 2023-05-31T21:36:12.727 INFO:teuthology.orchestra.run:waiting for 300 2023-05-31T21:36:12.775 INFO:tasks.ceph.c1.osd.1:Stopped 2023-05-31T21:36:12.775 DEBUG:tasks.ceph.c1.osd.2:waiting for process to exit 2023-05-31T21:36:12.776 INFO:teuthology.orchestra.run:waiting for 300 2023-05-31T21:36:12.830 INFO:tasks.ceph.c1.osd.2:Stopped 2023-05-31T21:36:12.830 INFO:teuthology.misc:Shutting down mgr daemons... 2023-05-31T21:36:12.831 DEBUG:tasks.ceph.c1.mgr.x:waiting for process to exit 2023-05-31T21:36:12.831 INFO:teuthology.orchestra.run:waiting for 300 2023-05-31T21:36:12.854 INFO:tasks.ceph.c1.mgr.x:Stopped 2023-05-31T21:36:12.855 INFO:teuthology.misc:Shutting down mon daemons... 2023-05-31T21:36:12.855 DEBUG:tasks.ceph.c1.mon.a:waiting for process to exit 2023-05-31T21:36:12.856 INFO:teuthology.orchestra.run:waiting for 300 2023-05-31T21:36:12.915 INFO:tasks.ceph.c1.mon.a:Stopped 2023-05-31T21:36:12.915 INFO:tasks.ceph:Checking cluster log for badness... 2023-05-31T21:36:12.916 DEBUG:teuthology.orchestra.run.smithi099:> sudo egrep '\[ERR\]|\[WRN\]|\[SEC\]' /var/log/ceph/c1.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v '\(PG_AVAILABILITY\)' | egrep -v '\(PG_DEGRADED\)' | egrep -v '\(POOL_APP_NOT_ENABLED\)' | head -n 1 2023-05-31T21:36:12.984 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/c1-0 on ubuntu@smithi099.front.sepia.ceph.com 2023-05-31T21:36:12.984 DEBUG:teuthology.orchestra.run.smithi099:> sync && sudo umount -f /var/lib/ceph/osd/c1-0 2023-05-31T21:36:13.373 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/c1-1 on ubuntu@smithi099.front.sepia.ceph.com 2023-05-31T21:36:13.374 DEBUG:teuthology.orchestra.run.smithi099:> sync && sudo umount -f /var/lib/ceph/osd/c1-1 2023-05-31T21:36:13.483 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/c1-2 on ubuntu@smithi099.front.sepia.ceph.com 2023-05-31T21:36:13.484 DEBUG:teuthology.orchestra.run.smithi099:> sync && sudo umount -f /var/lib/ceph/osd/c1-2 2023-05-31T21:36:13.557 INFO:tasks.ceph:Archiving mon data... 2023-05-31T21:36:13.557 DEBUG:teuthology.misc:Transferring archived files from smithi099:/var/lib/ceph/mon/c1-a to /home/teuthworker/archive/yuriw-2023-05-31_19:25:20-rgw-reef-release-distro-default-smithi/7292285/data/c1.mon.a.tgz 2023-05-31T21:36:13.558 DEBUG:teuthology.orchestra.run.smithi099:> mktemp 2023-05-31T21:36:13.575 INFO:teuthology.orchestra.run.smithi099.stdout:/tmp/tmp.CFjiwxv0m7 2023-05-31T21:36:13.576 DEBUG:teuthology.orchestra.run.smithi099:> sudo tar cz -f - -C /var/lib/ceph/mon/c1-a -- . > /tmp/tmp.CFjiwxv0m7 2023-05-31T21:36:13.775 DEBUG:teuthology.orchestra.run.smithi099:> sudo chmod 0666 /tmp/tmp.CFjiwxv0m7 2023-05-31T21:36:13.861 DEBUG:teuthology.orchestra.remote:smithi099:/tmp/tmp.CFjiwxv0m7 is 480KB 2023-05-31T21:36:13.955 DEBUG:teuthology.orchestra.run.smithi099:> rm -fr /tmp/tmp.CFjiwxv0m7 2023-05-31T21:36:13.973 INFO:tasks.ceph:Cleaning ceph cluster... 2023-05-31T21:36:13.973 DEBUG:teuthology.orchestra.run.smithi099:> sudo rm -rf -- /etc/ceph/c1.conf /etc/ceph/c1.keyring /home/ubuntu/cephtest/c1.data /home/ubuntu/cephtest/c1.monmap /home/ubuntu/cephtest/../*.pid 2023-05-31T21:36:14.015 DEBUG:teuthology.orchestra.run.smithi112:> sudo rm -rf -- /etc/ceph/c1.conf /etc/ceph/c1.keyring /home/ubuntu/cephtest/c1.data /home/ubuntu/cephtest/c1.monmap /home/ubuntu/cephtest/../*.pid 2023-05-31T21:36:14.048 INFO:tasks.ceph:Checking for errors in any valgrind logs... 2023-05-31T21:36:14.048 DEBUG:teuthology.orchestra.run.smithi099:> sudo zgrep '' /var/log/ceph/valgrind/* /dev/null | sort | uniq 2023-05-31T21:36:14.072 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.rgw.client.1 has been restored 2023-05-31T21:36:14.081 DEBUG:teuthology.orchestra.run.smithi112:> sudo zgrep '' /var/log/ceph/valgrind/* /dev/null | sort | uniq 2023-05-31T21:36:14.106 INFO:teuthology.orchestra.run.smithi099.stderr:gzip: /var/log/ceph/valgrind/*.gz: No such file or directory 2023-05-31T21:36:14.119 INFO:teuthology.orchestra.run.smithi112.stderr:gzip: /var/log/ceph/valgrind/*.gz: No such file or directory 2023-05-31T21:36:14.126 INFO:tasks.ceph:Archiving crash dumps... 2023-05-31T21:36:14.127 DEBUG:teuthology.misc:Transferring archived files from smithi099:/var/lib/ceph/crash to /home/teuthworker/archive/yuriw-2023-05-31_19:25:20-rgw-reef-release-distro-default-smithi/7292285/remote/smithi099/crash 2023-05-31T21:36:14.128 DEBUG:teuthology.orchestra.run.smithi099:> sudo tar c -f - -C /var/lib/ceph/crash -- . 2023-05-31T21:36:14.179 DEBUG:teuthology.misc:Transferring archived files from smithi112:/var/lib/ceph/crash to /home/teuthworker/archive/yuriw-2023-05-31_19:25:20-rgw-reef-release-distro-default-smithi/7292285/remote/smithi112/crash 2023-05-31T21:36:14.180 DEBUG:teuthology.orchestra.run.smithi112:> sudo tar c -f - -C /var/lib/ceph/crash -- . 2023-05-31T21:36:14.209 INFO:tasks.ceph:Compressing logs... 2023-05-31T21:36:14.210 DEBUG:teuthology.orchestra.run.smithi099:> sudo find /var/log/ceph -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2023-05-31T21:36:14.223 DEBUG:teuthology.orchestra.run.smithi112:> sudo find /var/log/ceph -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2023-05-31T21:38:37.450 INFO:tasks.ceph:Archiving logs... 2023-05-31T21:38:37.451 DEBUG:teuthology.misc:Transferring archived files from smithi099:/var/log/ceph to /home/teuthworker/archive/yuriw-2023-05-31_19:25:20-rgw-reef-release-distro-default-smithi/7292285/remote/smithi099/log 2023-05-31T21:38:37.453 DEBUG:teuthology.orchestra.run.smithi099:> sudo tar c -f - -C /var/log/ceph -- . 2023-05-31T21:38:51.157 DEBUG:teuthology.misc:Transferring archived files from smithi112:/var/log/ceph to /home/teuthworker/archive/yuriw-2023-05-31_19:25:20-rgw-reef-release-distro-default-smithi/7292285/remote/smithi112/log 2023-05-31T21:38:51.158 DEBUG:teuthology.orchestra.run.smithi112:> sudo tar c -f - -C /var/log/ceph -- . 2023-05-31T21:38:59.748 DEBUG:teuthology.run_tasks:Unwinding manager install 2023-05-31T21:38:59.758 INFO:teuthology.task.install.util:Removing shipped files: /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits... 2023-05-31T21:38:59.759 DEBUG:teuthology.orchestra.run.smithi099:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits 2023-05-31T21:38:59.761 DEBUG:teuthology.orchestra.run.smithi112:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits 2023-05-31T21:38:59.854 DEBUG:teuthology.run_tasks:Unwinding manager clock 2023-05-31T21:38:59.863 INFO:teuthology.task.clock:Checking final clock skew... 2023-05-31T21:38:59.864 DEBUG:teuthology.orchestra.run.smithi099:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2023-05-31T21:38:59.866 DEBUG:teuthology.orchestra.run.smithi112:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2023-05-31T21:38:59.883 INFO:teuthology.orchestra.run.smithi112.stderr:bash: ntpq: command not found 2023-05-31T21:38:59.884 INFO:teuthology.orchestra.run.smithi099.stderr:bash: ntpq: command not found 2023-05-31T21:38:59.887 INFO:teuthology.orchestra.run.smithi112.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2023-05-31T21:38:59.888 INFO:teuthology.orchestra.run.smithi112.stdout:=============================================================================== 2023-05-31T21:38:59.888 INFO:teuthology.orchestra.run.smithi112.stdout:^+ hv01.front.sepia.ceph.com 2 6 377 10 -687us[ -702us] +/- 89ms 2023-05-31T21:38:59.888 INFO:teuthology.orchestra.run.smithi112.stdout:^* hv02.front.sepia.ceph.com 3 8 377 6 +293us[ +279us] +/- 102ms 2023-05-31T21:38:59.888 INFO:teuthology.orchestra.run.smithi112.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 5 +330us[ +330us] +/- 77ms 2023-05-31T21:38:59.889 INFO:teuthology.orchestra.run.smithi112.stdout:^? hv04.front.sepia.ceph.com 0 10 0 - +0ns[ +0ns] +/- 0ns 2023-05-31T21:38:59.890 INFO:teuthology.orchestra.run.smithi099.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2023-05-31T21:38:59.890 INFO:teuthology.orchestra.run.smithi099.stdout:=============================================================================== 2023-05-31T21:38:59.890 INFO:teuthology.orchestra.run.smithi099.stdout:^+ hv01.front.sepia.ceph.com 2 6 377 14 -697us[ -733us] +/- 89ms 2023-05-31T21:38:59.891 INFO:teuthology.orchestra.run.smithi099.stdout:^+ hv02.front.sepia.ceph.com 3 8 377 13 +277us[ +242us] +/- 102ms 2023-05-31T21:38:59.891 INFO:teuthology.orchestra.run.smithi099.stdout:^* hv03.front.sepia.ceph.com 3 6 377 2 +332us[ +296us] +/- 77ms 2023-05-31T21:38:59.891 INFO:teuthology.orchestra.run.smithi099.stdout:^? hv04.front.sepia.ceph.com 0 10 0 - +0ns[ +0ns] +/- 0ns 2023-05-31T21:38:59.891 DEBUG:teuthology.run_tasks:Unwinding manager ansible.cephlab 2023-05-31T21:38:59.902 INFO:teuthology.task.ansible:Skipping ansible cleanup... 2023-05-31T21:38:59.902 DEBUG:teuthology.run_tasks:Unwinding manager selinux 2023-05-31T21:38:59.912 DEBUG:teuthology.orchestra.run.smithi099:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2023-05-31T21:39:00.028 DEBUG:teuthology.orchestra.run.smithi112:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2023-05-31T21:39:00.116 DEBUG:teuthology.orchestra.run.smithi099:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2023-05-31T21:39:00.149 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-05-31T21:39:00.150 DEBUG:teuthology.orchestra.run.smithi112:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2023-05-31T21:39:00.181 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-05-31T21:39:00.182 DEBUG:teuthology.run_tasks:Unwinding manager pcp 2023-05-31T21:39:00.192 DEBUG:teuthology.run_tasks:Unwinding manager internal.timer 2023-05-31T21:39:00.206 INFO:teuthology.task.internal:Duration was 7400.890781 seconds 2023-05-31T21:39:00.207 DEBUG:teuthology.run_tasks:Unwinding manager internal.syslog 2023-05-31T21:39:00.221 INFO:teuthology.task.internal.syslog:Shutting down syslog monitoring... 2023-05-31T21:39:00.222 DEBUG:teuthology.orchestra.run.smithi099:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2023-05-31T21:39:00.224 DEBUG:teuthology.orchestra.run.smithi112:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2023-05-31T21:39:00.270 INFO:teuthology.orchestra.run.smithi099.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2023-05-31T21:39:00.271 INFO:teuthology.orchestra.run.smithi112.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2023-05-31T21:39:00.555 INFO:teuthology.task.internal.syslog:Checking logs for errors... 2023-05-31T21:39:00.555 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi099.front.sepia.ceph.com 2023-05-31T21:39:00.556 DEBUG:teuthology.orchestra.run.smithi099:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2023-05-31T21:39:00.579 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi112.front.sepia.ceph.com 2023-05-31T21:39:00.579 DEBUG:teuthology.orchestra.run.smithi112:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2023-05-31T21:39:00.602 INFO:teuthology.task.internal.syslog:Compressing syslogs... 2023-05-31T21:39:00.602 DEBUG:teuthology.orchestra.run.smithi099:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2023-05-31T21:39:00.621 DEBUG:teuthology.orchestra.run.smithi112:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2023-05-31T21:39:00.679 INFO:teuthology.task.internal.syslog:Gathering journactl -b0... 2023-05-31T21:39:00.680 DEBUG:teuthology.orchestra.run.smithi099:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2023-05-31T21:39:00.699 DEBUG:teuthology.orchestra.run.smithi112:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2023-05-31T21:39:00.795 DEBUG:teuthology.run_tasks:Unwinding manager internal.sudo 2023-05-31T21:39:00.805 INFO:teuthology.task.internal:Restoring /etc/sudoers... 2023-05-31T21:39:00.806 DEBUG:teuthology.orchestra.run.smithi099:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2023-05-31T21:39:00.833 DEBUG:teuthology.orchestra.run.smithi112:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2023-05-31T21:39:00.860 DEBUG:teuthology.run_tasks:Unwinding manager internal.coredump 2023-05-31T21:39:00.870 DEBUG:teuthology.orchestra.run.smithi099:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2023-05-31T21:39:00.875 DEBUG:teuthology.orchestra.run.smithi112:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2023-05-31T21:39:00.899 INFO:teuthology.orchestra.run.smithi099.stdout:kernel.core_pattern = core 2023-05-31T21:39:00.927 INFO:teuthology.orchestra.run.smithi112.stdout:kernel.core_pattern = core 2023-05-31T21:39:00.977 DEBUG:teuthology.orchestra.run.smithi099:> test -e /home/ubuntu/cephtest/archive/coredump 2023-05-31T21:39:00.995 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-05-31T21:39:00.995 DEBUG:teuthology.orchestra.run.smithi112:> test -e /home/ubuntu/cephtest/archive/coredump 2023-05-31T21:39:01.034 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-05-31T21:39:01.035 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive 2023-05-31T21:39:01.045 INFO:teuthology.task.internal:Transferring archived files... 2023-05-31T21:39:01.046 DEBUG:teuthology.misc:Transferring archived files from smithi099:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/yuriw-2023-05-31_19:25:20-rgw-reef-release-distro-default-smithi/7292285/remote/smithi099 2023-05-31T21:39:01.046 DEBUG:teuthology.orchestra.run.smithi099:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2023-05-31T21:39:01.082 DEBUG:teuthology.misc:Transferring archived files from smithi112:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/yuriw-2023-05-31_19:25:20-rgw-reef-release-distro-default-smithi/7292285/remote/smithi112 2023-05-31T21:39:01.082 DEBUG:teuthology.orchestra.run.smithi112:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2023-05-31T21:39:01.116 INFO:teuthology.task.internal:Removing archive directory... 2023-05-31T21:39:01.117 DEBUG:teuthology.orchestra.run.smithi099:> rm -rf -- /home/ubuntu/cephtest/archive 2023-05-31T21:39:01.119 DEBUG:teuthology.orchestra.run.smithi112:> rm -rf -- /home/ubuntu/cephtest/archive 2023-05-31T21:39:01.169 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive_upload 2023-05-31T21:39:01.181 INFO:teuthology.task.internal:Not uploading archives. 2023-05-31T21:39:01.182 DEBUG:teuthology.run_tasks:Unwinding manager internal.base 2023-05-31T21:39:01.196 INFO:teuthology.task.internal:Tidying up after the test... 2023-05-31T21:39:01.196 DEBUG:teuthology.orchestra.run.smithi099:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2023-05-31T21:39:01.199 DEBUG:teuthology.orchestra.run.smithi112:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2023-05-31T21:39:01.214 INFO:teuthology.orchestra.run.smithi099.stdout: 393224 4 drwxr-xr-x 2 ubuntu ubuntu 4096 May 31 21:39 /home/ubuntu/cephtest 2023-05-31T21:39:01.226 INFO:teuthology.orchestra.run.smithi112.stdout: 393224 4 drwxr-xr-x 2 ubuntu ubuntu 4096 May 31 21:39 /home/ubuntu/cephtest 2023-05-31T21:39:01.228 DEBUG:teuthology.run_tasks:Unwinding manager kernel 2023-05-31T21:39:01.241 DEBUG:teuthology.run_tasks:Unwinding manager console_log 2023-05-31T21:39:01.272 INFO:teuthology.nuke:Checking targets against current locks 2023-05-31T21:39:01.300 DEBUG:teuthology.nuke:shortname: smithi099 2023-05-31T21:39:01.301 INFO:teuthology.task.internal.check_lock:Checking locks... 2023-05-31T21:39:01.328 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi099.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/yuriw-2023-05-31_19:25:20-rgw-reef-release-distro-default-smithi/7292285', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '8', 'arch': 'x86_64', 'locked': True, 'locked_since': '2023-05-31 19:27:13.562019', 'locked_by': 'scheduled_yuriw@teuthology', 'mac_address': None, 'ssh_pub_key': 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDox97V1ZbGiXYfIam0QsAKCOossAxUlL++9Y+qMtFgf7WbqpNcOYHYWWtH1ov+9qROYmTCRNQIFX+NkdHU8Q2Wbwg5LFP43nv89monL09OSkXGOUGTF6MF+MDeWU/gCRrcDxxs4QnoWTTbcd4HtqMzUprCFR5KLeO4XJSkTCxXlVJC1I2F9x9uGkv5VM4gB0ZVUeoVsATmS79+GLwNLH2DhXh4JQJL9HKgC9FN03CerL+Ka6cdcu3Wc6rSelCxMx0Ead4Vsre8il1J8PfCPvcko7HAkfp2XPu3avJXTKUTKYrF05s3uZq027l5T6VrlavgVtfGsyB8Ez4YDolVQOoGGfF2I99aVh9LAYc+J8KOY1Qh5KaILCCmrMI74XRPrqPrg7phDNIDNwjAxPcmoKpNDR6ALert5o+mM8onphQ/yRxisjwMH/bNXE63CdbjgkIOm2VJ2g/k1Ncf4lBxUPSx3eF+ERlKFdboN8S567n9hWRVwbSdGmzQf3AsOb38Hlk='} 2023-05-31T21:39:01.331 DEBUG:teuthology.nuke:shortname: smithi112 2023-05-31T21:39:01.331 INFO:teuthology.task.internal.check_lock:Checking locks... 2023-05-31T21:39:01.356 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi112.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/yuriw-2023-05-31_19:25:20-rgw-reef-release-distro-default-smithi/7292285', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '8', 'arch': 'x86_64', 'locked': True, 'locked_since': '2023-05-31 19:27:13.563506', 'locked_by': 'scheduled_yuriw@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBFfCJhk9QMgDL70wZsKQnRvWaVVb4spsBOWpIa4TPw/+clTUBIAM2haG88c+/+tNQhpaYNEiHo4bsY1a+mmIK9A='} 2023-05-31T21:39:01.392 INFO:teuthology.orchestra.console:Power off smithi099 2023-05-31T21:39:01.393 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi099.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2023-05-31T21:39:01.416 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi099.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-05-31T21:39:01.440 INFO:teuthology.orchestra.console:Power off smithi112 2023-05-31T21:39:01.441 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi112.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2023-05-31T21:39:01.464 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi112.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-05-31T21:39:05.453 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi099.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-05-31T21:39:05.490 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi112.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-05-31T21:39:09.567 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi099.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-05-31T21:39:09.603 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi112.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-05-31T21:39:09.816 INFO:teuthology.orchestra.console:Power off for smithi112 completed 2023-05-31T21:39:13.681 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi099.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-05-31T21:39:13.894 INFO:teuthology.orchestra.console:Power off for smithi099 completed 2023-05-31T21:39:14.000 INFO:teuthology.run:Summary data: description: rgw/multisite/{clusters frontend/beast ignore-pg-availability notify omap_limits overrides realms/two-zonegroup supported-random-distro$/{centos_8} tasks/test_multi} duration: 7400.890780925751 failure_reason: rgw multisite test failures flavor: default owner: scheduled_yuriw@teuthology sentry_event: https://sentry.ceph.com/organizations/ceph/?query=8d201981081f4c40857626305f95bc8e status: fail success: false 2023-05-31T21:39:14.000 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2023-05-31T21:39:14.001 DEBUG:urllib3.util.retry:Converted retries value: 10 -> Retry(total=10, connect=None, read=None, redirect=None, status=None) 2023-05-31T21:39:14.093 INFO:teuthology.run:FAIL